[==========] Running 8 tests from 2 test suites.
[----------] Global test environment set-up.
[----------] 6 tests from LeaderRebalancerTest
[ RUN      ] LeaderRebalancerTest.FunctionalTestForDivided
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20260709 12:57:58.763638   431 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.107.254:34493
I20260709 12:57:58.765586   431 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260709 12:57:58.766858   431 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:57:58.793641   438 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:57:58.793708   440 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:57:58.796836   437 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:58:00.122130   439 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1328 milliseconds
I20260709 12:58:00.122272   431 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260709 12:58:00.123921   431 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:58:00.124167   431 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:58:00.124385   431 hybrid_clock.cc:648] HybridClock initialized: now 1783601880124349 us; error 0 us; skew 500 ppm
I20260709 12:58:00.125247   431 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:58:00.134073   431 webserver.cc:533] Webserver started at http://127.0.107.254:34835/ using document root <none> and password file <none>
I20260709 12:58:00.135782   431 fs_manager.cc:362] Metadata directory not provided
I20260709 12:58:00.136062   431 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:58:00.136677   431 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:58:00.144536   431 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskakPCWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783601878574137-431-0/minicluster-data/master-0-root/instance:
uuid: "8c2299d3af364ee0a759958e5da33fa1"
format_stamp: "Formatted at 2026-07-09 12:58:00 on dist-test-slave-nvvq"
I20260709 12:58:00.155995   431 fs_manager.cc:696] Time spent creating directory manager: real 0.010s	user 0.011s	sys 0.001s
I20260709 12:58:00.163401   446 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:58:00.164749   431 fs_manager.cc:730] Time spent opening block manager: real 0.005s	user 0.001s	sys 0.003s
I20260709 12:58:00.165112   431 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskakPCWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783601878574137-431-0/minicluster-data/master-0-root
uuid: "8c2299d3af364ee0a759958e5da33fa1"
format_stamp: "Formatted at 2026-07-09 12:58:00 on dist-test-slave-nvvq"
I20260709 12:58:00.165470   431 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskakPCWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783601878574137-431-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskakPCWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783601878574137-431-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskakPCWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783601878574137-431-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:58:00.248708   431 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:58:00.251196   431 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260709 12:58:00.251829   431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:58:00.443946   431 rpc_server.cc:307] RPC server started. Bound to: 127.0.107.254:34493
I20260709 12:58:00.444116   497 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.107.254:34493 every 8 connection(s)
I20260709 12:58:00.457526   498 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:00.491667   498 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c2299d3af364ee0a759958e5da33fa1: Bootstrap starting.
I20260709 12:58:00.502903   498 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8c2299d3af364ee0a759958e5da33fa1: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:00.506326   498 log.cc:826] T 00000000000000000000000000000000 P 8c2299d3af364ee0a759958e5da33fa1: Log is configured to *not* fsync() on all Append() calls
I20260709 12:58:00.514945   498 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c2299d3af364ee0a759958e5da33fa1: No bootstrap required, opened a new log
I20260709 12:58:00.546094   498 raft_consensus.cc:359] T 00000000000000000000000000000000 P 8c2299d3af364ee0a759958e5da33fa1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c2299d3af364ee0a759958e5da33fa1" member_type: VOTER }
I20260709 12:58:00.546972   498 raft_consensus.cc:385] T 00000000000000000000000000000000 P 8c2299d3af364ee0a759958e5da33fa1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:00.547276   498 raft_consensus.cc:740] T 00000000000000000000000000000000 P 8c2299d3af364ee0a759958e5da33fa1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c2299d3af364ee0a759958e5da33fa1, State: Initialized, Role: FOLLOWER
I20260709 12:58:00.548707   498 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8c2299d3af364ee0a759958e5da33fa1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c2299d3af364ee0a759958e5da33fa1" member_type: VOTER }
I20260709 12:58:00.549403   498 raft_consensus.cc:399] T 00000000000000000000000000000000 P 8c2299d3af364ee0a759958e5da33fa1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260709 12:58:00.549748   498 raft_consensus.cc:493] T 00000000000000000000000000000000 P 8c2299d3af364ee0a759958e5da33fa1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260709 12:58:00.550175   498 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 8c2299d3af364ee0a759958e5da33fa1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:00.557256   498 raft_consensus.cc:515] T 00000000000000000000000000000000 P 8c2299d3af364ee0a759958e5da33fa1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c2299d3af364ee0a759958e5da33fa1" member_type: VOTER }
I20260709 12:58:00.558216   498 leader_election.cc:304] T 00000000000000000000000000000000 P 8c2299d3af364ee0a759958e5da33fa1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8c2299d3af364ee0a759958e5da33fa1; no voters: 
I20260709 12:58:00.562247   498 leader_election.cc:290] T 00000000000000000000000000000000 P 8c2299d3af364ee0a759958e5da33fa1 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20260709 12:58:00.564260   501 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 8c2299d3af364ee0a759958e5da33fa1 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:58:00.571014   501 raft_consensus.cc:697] T 00000000000000000000000000000000 P 8c2299d3af364ee0a759958e5da33fa1 [term 1 LEADER]: Becoming Leader. State: Replica: 8c2299d3af364ee0a759958e5da33fa1, State: Running, Role: LEADER
I20260709 12:58:00.572727   501 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8c2299d3af364ee0a759958e5da33fa1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c2299d3af364ee0a759958e5da33fa1" member_type: VOTER }
I20260709 12:58:00.573897   498 sys_catalog.cc:565] T 00000000000000000000000000000000 P 8c2299d3af364ee0a759958e5da33fa1 [sys.catalog]: configured and running, proceeding with master startup.
I20260709 12:58:00.598819   502 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c2299d3af364ee0a759958e5da33fa1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8c2299d3af364ee0a759958e5da33fa1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c2299d3af364ee0a759958e5da33fa1" member_type: VOTER } }
I20260709 12:58:00.600602   502 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c2299d3af364ee0a759958e5da33fa1 [sys.catalog]: This master's current role is: LEADER
I20260709 12:58:00.598839   503 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c2299d3af364ee0a759958e5da33fa1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8c2299d3af364ee0a759958e5da33fa1. Latest consensus state: current_term: 1 leader_uuid: "8c2299d3af364ee0a759958e5da33fa1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c2299d3af364ee0a759958e5da33fa1" member_type: VOTER } }
I20260709 12:58:00.603611   503 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c2299d3af364ee0a759958e5da33fa1 [sys.catalog]: This master's current role is: LEADER
I20260709 12:58:00.611955   510 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260709 12:58:00.634805   510 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260709 12:58:00.644816   431 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260709 12:58:00.667014   510 catalog_manager.cc:1383] Generated new cluster ID: 88655b2ea5704729887ad930ba43ce76
I20260709 12:58:00.667460   510 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260709 12:58:00.710381   510 catalog_manager.cc:1406] Generated new certificate authority record
I20260709 12:58:00.713158   510 catalog_manager.cc:1540] Loading token signing keys...
I20260709 12:58:00.739486   510 catalog_manager.cc:6070] T 00000000000000000000000000000000 P 8c2299d3af364ee0a759958e5da33fa1: Generated new TSK 0
I20260709 12:58:00.740978   510 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260709 12:58:00.778991   431 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:58:00.793577   520 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:58:00.797842   521 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:58:00.807276   523 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:58:00.810923   431 server_base.cc:1061] running on GCE node
I20260709 12:58:00.811770   431 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:58:00.812013   431 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:58:00.812269   431 hybrid_clock.cc:648] HybridClock initialized: now 1783601880812223 us; error 0 us; skew 500 ppm
I20260709 12:58:00.812945   431 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:58:00.821517   431 webserver.cc:533] Webserver started at http://127.0.107.193:38879/ using document root <none> and password file <none>
I20260709 12:58:00.822203   431 fs_manager.cc:362] Metadata directory not provided
I20260709 12:58:00.822510   431 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:58:00.822855   431 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:58:00.824297   431 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskakPCWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783601878574137-431-0/minicluster-data/ts-0-root/instance:
uuid: "b5e46fc5aac9426a945bb34207d634d4"
format_stamp: "Formatted at 2026-07-09 12:58:00 on dist-test-slave-nvvq"
I20260709 12:58:00.833271   431 fs_manager.cc:696] Time spent creating directory manager: real 0.008s	user 0.003s	sys 0.007s
I20260709 12:58:00.841022   528 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:58:00.842108   431 fs_manager.cc:730] Time spent opening block manager: real 0.007s	user 0.000s	sys 0.006s
I20260709 12:58:00.842522   431 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskakPCWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783601878574137-431-0/minicluster-data/ts-0-root
uuid: "b5e46fc5aac9426a945bb34207d634d4"
format_stamp: "Formatted at 2026-07-09 12:58:00 on dist-test-slave-nvvq"
I20260709 12:58:00.842839   431 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskakPCWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783601878574137-431-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskakPCWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783601878574137-431-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskakPCWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783601878574137-431-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:58:00.872433   431 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:58:00.874903   431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:58:00.881038   431 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:58:00.889424   431 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:58:00.889729   431 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:58:00.890084   431 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:58:00.890246   431 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:58:01.156411   431 rpc_server.cc:307] RPC server started. Bound to: 127.0.107.193:38425
I20260709 12:58:01.158553   590 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.107.193:38425 every 8 connection(s)
I20260709 12:58:01.171573   431 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:58:01.199112   595 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:58:01.205613   596 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:58:01.227385   599 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:58:01.228214   431 server_base.cc:1061] running on GCE node
I20260709 12:58:01.229772   431 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:58:01.230072   431 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:58:01.230301   431 hybrid_clock.cc:648] HybridClock initialized: now 1783601881230280 us; error 0 us; skew 500 ppm
I20260709 12:58:01.231205   431 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:58:01.234143   591 heartbeater.cc:344] Connected to a master server at 127.0.107.254:34493
I20260709 12:58:01.234786   591 heartbeater.cc:461] Registering TS with master...
I20260709 12:58:01.235080   431 webserver.cc:533] Webserver started at http://127.0.107.194:35491/ using document root <none> and password file <none>
I20260709 12:58:01.235855   431 fs_manager.cc:362] Metadata directory not provided
I20260709 12:58:01.236178   431 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:58:01.236547   431 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:58:01.236534   591 heartbeater.cc:507] Master 127.0.107.254:34493 requested a full tablet report, sending...
I20260709 12:58:01.238425   431 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskakPCWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783601878574137-431-0/minicluster-data/ts-1-root/instance:
uuid: "263986885d354f7ba08739e419a6cded"
format_stamp: "Formatted at 2026-07-09 12:58:01 on dist-test-slave-nvvq"
I20260709 12:58:01.240620   463 ts_manager.cc:194] Registered new tserver with Master: b5e46fc5aac9426a945bb34207d634d4 (127.0.107.193:38425)
I20260709 12:58:01.243019   463 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:44476
I20260709 12:58:01.247120   431 fs_manager.cc:696] Time spent creating directory manager: real 0.008s	user 0.006s	sys 0.000s
I20260709 12:58:01.251588   603 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:58:01.252473   431 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
I20260709 12:58:01.252791   431 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskakPCWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783601878574137-431-0/minicluster-data/ts-1-root
uuid: "263986885d354f7ba08739e419a6cded"
format_stamp: "Formatted at 2026-07-09 12:58:01 on dist-test-slave-nvvq"
I20260709 12:58:01.253149   431 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskakPCWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783601878574137-431-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskakPCWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783601878574137-431-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskakPCWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783601878574137-431-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:58:01.276768   431 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:58:01.279402   431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:58:01.282584   431 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:58:01.287012   431 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:58:01.287297   431 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:58:01.287577   431 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:58:01.287779   431 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:58:01.569949   431 rpc_server.cc:307] RPC server started. Bound to: 127.0.107.194:45491
I20260709 12:58:01.570087   665 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.107.194:45491 every 8 connection(s)
I20260709 12:58:01.588711   431 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:58:01.607355   669 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:58:01.611927   666 heartbeater.cc:344] Connected to a master server at 127.0.107.254:34493
I20260709 12:58:01.612587   666 heartbeater.cc:461] Registering TS with master...
I20260709 12:58:01.614625   666 heartbeater.cc:507] Master 127.0.107.254:34493 requested a full tablet report, sending...
I20260709 12:58:01.623489   463 ts_manager.cc:194] Registered new tserver with Master: 263986885d354f7ba08739e419a6cded (127.0.107.194:45491)
I20260709 12:58:01.631544   463 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:44482
I20260709 12:58:01.647675   431 server_base.cc:1061] running on GCE node
W20260709 12:58:01.647547   670 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:58:01.648850   672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:58:01.651301   431 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:58:01.651618   431 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:58:01.651883   431 hybrid_clock.cc:648] HybridClock initialized: now 1783601881651863 us; error 0 us; skew 500 ppm
I20260709 12:58:01.652726   431 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:58:01.656701   431 webserver.cc:533] Webserver started at http://127.0.107.195:45591/ using document root <none> and password file <none>
I20260709 12:58:01.657442   431 fs_manager.cc:362] Metadata directory not provided
I20260709 12:58:01.657807   431 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:58:01.658242   431 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:58:01.660046   431 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskakPCWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783601878574137-431-0/minicluster-data/ts-2-root/instance:
uuid: "1bcd68d29a7b41a9bf114ac103d4c649"
format_stamp: "Formatted at 2026-07-09 12:58:01 on dist-test-slave-nvvq"
I20260709 12:58:01.666028   431 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.007s	sys 0.000s
I20260709 12:58:01.669981   677 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:58:01.670997   431 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20260709 12:58:01.671331   431 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskakPCWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783601878574137-431-0/minicluster-data/ts-2-root
uuid: "1bcd68d29a7b41a9bf114ac103d4c649"
format_stamp: "Formatted at 2026-07-09 12:58:01 on dist-test-slave-nvvq"
I20260709 12:58:01.671665   431 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskakPCWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783601878574137-431-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskakPCWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783601878574137-431-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskakPCWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783601878574137-431-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:58:01.700187   431 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:58:01.702052   431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:58:01.704981   431 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:58:01.708925   431 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:58:01.709187   431 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:58:01.709473   431 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:58:01.709681   431 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:58:01.960691   431 rpc_server.cc:307] RPC server started. Bound to: 127.0.107.195:46365
I20260709 12:58:01.961414   739 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.107.195:46365 every 8 connection(s)
I20260709 12:58:01.993494   740 heartbeater.cc:344] Connected to a master server at 127.0.107.254:34493
I20260709 12:58:01.994000   740 heartbeater.cc:461] Registering TS with master...
I20260709 12:58:01.995332   740 heartbeater.cc:507] Master 127.0.107.254:34493 requested a full tablet report, sending...
I20260709 12:58:01.998380   463 ts_manager.cc:194] Registered new tserver with Master: 1bcd68d29a7b41a9bf114ac103d4c649 (127.0.107.195:46365)
I20260709 12:58:01.999328   431 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.020430389s
I20260709 12:58:02.000780   463 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:44494
I20260709 12:58:02.104019   463 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44506:
name: "test-workload"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
  rows: "\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q""\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q"
  indirect_data: """"
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W20260709 12:58:02.111824   463 catalog_manager.cc:7077] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260709 12:58:02.247350   591 heartbeater.cc:499] Master 127.0.107.254:34493 was elected leader, sending a full tablet report...
I20260709 12:58:02.332736   630 tablet_service.cc:1511] Processing CreateTablet for tablet 2e14183cc6064b1cbe6530eb1c8c899c (DEFAULT_TABLE table=test-workload [id=7d68ac1df47f4cda927735119830900e]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 12:58:02.336994   627 tablet_service.cc:1511] Processing CreateTablet for tablet 59112618a86046e1af899ce6d3c85203 (DEFAULT_TABLE table=test-workload [id=7d68ac1df47f4cda927735119830900e]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 12:58:02.340525   627 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59112618a86046e1af899ce6d3c85203. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:02.335178   628 tablet_service.cc:1511] Processing CreateTablet for tablet d8b33c9315094794a876a4b62755efaf (DEFAULT_TABLE table=test-workload [id=7d68ac1df47f4cda927735119830900e]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 12:58:02.335440   629 tablet_service.cc:1511] Processing CreateTablet for tablet d441098f471f4b0084938676c20981e5 (DEFAULT_TABLE table=test-workload [id=7d68ac1df47f4cda927735119830900e]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 12:58:02.342825   630 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e14183cc6064b1cbe6530eb1c8c899c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:02.330044   631 tablet_service.cc:1511] Processing CreateTablet for tablet 793bafcf38ca4ffb9a9aac9374ea2094 (DEFAULT_TABLE table=test-workload [id=7d68ac1df47f4cda927735119830900e]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 12:58:02.347332   629 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d441098f471f4b0084938676c20981e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:02.354764   628 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8b33c9315094794a876a4b62755efaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:02.356532   626 tablet_service.cc:1511] Processing CreateTablet for tablet e6aabcfcdeb34e6c94ec0bb089deabda (DEFAULT_TABLE table=test-workload [id=7d68ac1df47f4cda927735119830900e]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 12:58:02.363722   631 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 793bafcf38ca4ffb9a9aac9374ea2094. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:02.371693   705 tablet_service.cc:1511] Processing CreateTablet for tablet 793bafcf38ca4ffb9a9aac9374ea2094 (DEFAULT_TABLE table=test-workload [id=7d68ac1df47f4cda927735119830900e]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 12:58:02.382516   626 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e6aabcfcdeb34e6c94ec0bb089deabda. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:02.379290   704 tablet_service.cc:1511] Processing CreateTablet for tablet 2e14183cc6064b1cbe6530eb1c8c899c (DEFAULT_TABLE table=test-workload [id=7d68ac1df47f4cda927735119830900e]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 12:58:02.392352   701 tablet_service.cc:1511] Processing CreateTablet for tablet 59112618a86046e1af899ce6d3c85203 (DEFAULT_TABLE table=test-workload [id=7d68ac1df47f4cda927735119830900e]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 12:58:02.379168   703 tablet_service.cc:1511] Processing CreateTablet for tablet d441098f471f4b0084938676c20981e5 (DEFAULT_TABLE table=test-workload [id=7d68ac1df47f4cda927735119830900e]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 12:58:02.407861   555 tablet_service.cc:1511] Processing CreateTablet for tablet 2e14183cc6064b1cbe6530eb1c8c899c (DEFAULT_TABLE table=test-workload [id=7d68ac1df47f4cda927735119830900e]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 12:58:02.404058   556 tablet_service.cc:1511] Processing CreateTablet for tablet 793bafcf38ca4ffb9a9aac9374ea2094 (DEFAULT_TABLE table=test-workload [id=7d68ac1df47f4cda927735119830900e]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 12:58:02.418552   705 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 793bafcf38ca4ffb9a9aac9374ea2094. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:02.415241   554 tablet_service.cc:1511] Processing CreateTablet for tablet d441098f471f4b0084938676c20981e5 (DEFAULT_TABLE table=test-workload [id=7d68ac1df47f4cda927735119830900e]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 12:58:02.423695   552 tablet_service.cc:1511] Processing CreateTablet for tablet 59112618a86046e1af899ce6d3c85203 (DEFAULT_TABLE table=test-workload [id=7d68ac1df47f4cda927735119830900e]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 12:58:02.423607   699 tablet_service.cc:1511] Processing CreateTablet for tablet d4d26ba9fb084e748bddb93f7547621f (DEFAULT_TABLE table=test-workload [id=7d68ac1df47f4cda927735119830900e]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 12:58:02.406775   702 tablet_service.cc:1511] Processing CreateTablet for tablet d8b33c9315094794a876a4b62755efaf (DEFAULT_TABLE table=test-workload [id=7d68ac1df47f4cda927735119830900e]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 12:58:02.410094   700 tablet_service.cc:1511] Processing CreateTablet for tablet e6aabcfcdeb34e6c94ec0bb089deabda (DEFAULT_TABLE table=test-workload [id=7d68ac1df47f4cda927735119830900e]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 12:58:02.409473   625 tablet_service.cc:1511] Processing CreateTablet for tablet d4d26ba9fb084e748bddb93f7547621f (DEFAULT_TABLE table=test-workload [id=7d68ac1df47f4cda927735119830900e]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 12:58:02.429137   553 tablet_service.cc:1511] Processing CreateTablet for tablet d8b33c9315094794a876a4b62755efaf (DEFAULT_TABLE table=test-workload [id=7d68ac1df47f4cda927735119830900e]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 12:58:02.434687   555 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e14183cc6064b1cbe6530eb1c8c899c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:02.436395   625 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d4d26ba9fb084e748bddb93f7547621f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:02.434674   704 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e14183cc6064b1cbe6530eb1c8c899c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:02.431236   551 tablet_service.cc:1511] Processing CreateTablet for tablet e6aabcfcdeb34e6c94ec0bb089deabda (DEFAULT_TABLE table=test-workload [id=7d68ac1df47f4cda927735119830900e]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 12:58:02.449064   554 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d441098f471f4b0084938676c20981e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:02.450116   550 tablet_service.cc:1511] Processing CreateTablet for tablet d4d26ba9fb084e748bddb93f7547621f (DEFAULT_TABLE table=test-workload [id=7d68ac1df47f4cda927735119830900e]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 12:58:02.454478   549 tablet_service.cc:1511] Processing CreateTablet for tablet 61c59a4c19754dedb9d385e2ecfe145b (DEFAULT_TABLE table=test-workload [id=7d68ac1df47f4cda927735119830900e]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 12:58:02.457752   551 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e6aabcfcdeb34e6c94ec0bb089deabda. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:02.459775   549 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 61c59a4c19754dedb9d385e2ecfe145b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:02.463222   550 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d4d26ba9fb084e748bddb93f7547621f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:02.473523   553 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8b33c9315094794a876a4b62755efaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:02.470849   624 tablet_service.cc:1511] Processing CreateTablet for tablet 61c59a4c19754dedb9d385e2ecfe145b (DEFAULT_TABLE table=test-workload [id=7d68ac1df47f4cda927735119830900e]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 12:58:02.479627   701 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59112618a86046e1af899ce6d3c85203. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:02.489941   698 tablet_service.cc:1511] Processing CreateTablet for tablet 61c59a4c19754dedb9d385e2ecfe145b (DEFAULT_TABLE table=test-workload [id=7d68ac1df47f4cda927735119830900e]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 12:58:02.486827   702 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8b33c9315094794a876a4b62755efaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:02.495407   700 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e6aabcfcdeb34e6c94ec0bb089deabda. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:02.496944   624 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 61c59a4c19754dedb9d385e2ecfe145b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:02.494719   552 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59112618a86046e1af899ce6d3c85203. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:02.500051   698 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 61c59a4c19754dedb9d385e2ecfe145b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:02.510200   699 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d4d26ba9fb084e748bddb93f7547621f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:02.518850   703 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d441098f471f4b0084938676c20981e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:02.521296   556 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 793bafcf38ca4ffb9a9aac9374ea2094. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:02.528009   548 tablet_service.cc:1511] Processing CreateTablet for tablet dd06089bfde94412b12705a116ecf3d5 (DEFAULT_TABLE table=test-workload [id=7d68ac1df47f4cda927735119830900e]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 12:58:02.530244   548 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd06089bfde94412b12705a116ecf3d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:02.559747   623 tablet_service.cc:1511] Processing CreateTablet for tablet dd06089bfde94412b12705a116ecf3d5 (DEFAULT_TABLE table=test-workload [id=7d68ac1df47f4cda927735119830900e]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 12:58:02.643908   666 heartbeater.cc:499] Master 127.0.107.254:34493 was elected leader, sending a full tablet report...
I20260709 12:58:02.647753   623 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd06089bfde94412b12705a116ecf3d5. 1 dirs total, 0 dirs full, 0 dirs failed
W20260709 12:58:02.648698   511 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:58:02.654846   697 tablet_service.cc:1511] Processing CreateTablet for tablet dd06089bfde94412b12705a116ecf3d5 (DEFAULT_TABLE table=test-workload [id=7d68ac1df47f4cda927735119830900e]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 12:58:02.662822   697 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd06089bfde94412b12705a116ecf3d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:02.677220   759 tablet_bootstrap.cc:492] T d441098f471f4b0084938676c20981e5 P 263986885d354f7ba08739e419a6cded: Bootstrap starting.
I20260709 12:58:02.785279   759 tablet_bootstrap.cc:654] T d441098f471f4b0084938676c20981e5 P 263986885d354f7ba08739e419a6cded: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:02.810776   760 tablet_bootstrap.cc:492] T 793bafcf38ca4ffb9a9aac9374ea2094 P 1bcd68d29a7b41a9bf114ac103d4c649: Bootstrap starting.
I20260709 12:58:02.885741   761 tablet_bootstrap.cc:492] T dd06089bfde94412b12705a116ecf3d5 P b5e46fc5aac9426a945bb34207d634d4: Bootstrap starting.
I20260709 12:58:02.915721   760 tablet_bootstrap.cc:654] T 793bafcf38ca4ffb9a9aac9374ea2094 P 1bcd68d29a7b41a9bf114ac103d4c649: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:02.924010   761 tablet_bootstrap.cc:654] T dd06089bfde94412b12705a116ecf3d5 P b5e46fc5aac9426a945bb34207d634d4: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:02.923009   759 tablet_bootstrap.cc:492] T d441098f471f4b0084938676c20981e5 P 263986885d354f7ba08739e419a6cded: No bootstrap required, opened a new log
I20260709 12:58:02.928046   759 ts_tablet_manager.cc:1403] T d441098f471f4b0084938676c20981e5 P 263986885d354f7ba08739e419a6cded: Time spent bootstrapping tablet: real 0.253s	user 0.019s	sys 0.019s
I20260709 12:58:02.955585   760 tablet_bootstrap.cc:492] T 793bafcf38ca4ffb9a9aac9374ea2094 P 1bcd68d29a7b41a9bf114ac103d4c649: No bootstrap required, opened a new log
I20260709 12:58:02.956432   760 ts_tablet_manager.cc:1403] T 793bafcf38ca4ffb9a9aac9374ea2094 P 1bcd68d29a7b41a9bf114ac103d4c649: Time spent bootstrapping tablet: real 0.148s	user 0.007s	sys 0.024s
I20260709 12:58:02.959415   759 raft_consensus.cc:359] T d441098f471f4b0084938676c20981e5 P 263986885d354f7ba08739e419a6cded [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } }
I20260709 12:58:02.960515   759 raft_consensus.cc:385] T d441098f471f4b0084938676c20981e5 P 263986885d354f7ba08739e419a6cded [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:02.960903   759 raft_consensus.cc:740] T d441098f471f4b0084938676c20981e5 P 263986885d354f7ba08739e419a6cded [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 263986885d354f7ba08739e419a6cded, State: Initialized, Role: FOLLOWER
I20260709 12:58:02.961910   760 raft_consensus.cc:359] T 793bafcf38ca4ffb9a9aac9374ea2094 P 1bcd68d29a7b41a9bf114ac103d4c649 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } } peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } }
I20260709 12:58:02.961894   759 consensus_queue.cc:260] T d441098f471f4b0084938676c20981e5 P 263986885d354f7ba08739e419a6cded [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } }
I20260709 12:58:02.963271   760 raft_consensus.cc:385] T 793bafcf38ca4ffb9a9aac9374ea2094 P 1bcd68d29a7b41a9bf114ac103d4c649 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:02.963867   760 raft_consensus.cc:740] T 793bafcf38ca4ffb9a9aac9374ea2094 P 1bcd68d29a7b41a9bf114ac103d4c649 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bcd68d29a7b41a9bf114ac103d4c649, State: Initialized, Role: FOLLOWER
I20260709 12:58:02.965957   760 consensus_queue.cc:260] T 793bafcf38ca4ffb9a9aac9374ea2094 P 1bcd68d29a7b41a9bf114ac103d4c649 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } } peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } }
I20260709 12:58:02.968708   759 ts_tablet_manager.cc:1434] T d441098f471f4b0084938676c20981e5 P 263986885d354f7ba08739e419a6cded: Time spent starting tablet: real 0.040s	user 0.014s	sys 0.000s
I20260709 12:58:02.972604   759 tablet_bootstrap.cc:492] T 2e14183cc6064b1cbe6530eb1c8c899c P 263986885d354f7ba08739e419a6cded: Bootstrap starting.
I20260709 12:58:02.981257   740 heartbeater.cc:499] Master 127.0.107.254:34493 was elected leader, sending a full tablet report...
I20260709 12:58:02.982609   760 ts_tablet_manager.cc:1434] T 793bafcf38ca4ffb9a9aac9374ea2094 P 1bcd68d29a7b41a9bf114ac103d4c649: Time spent starting tablet: real 0.026s	user 0.009s	sys 0.012s
W20260709 12:58:02.984171   741 tablet.cc:2406] T 793bafcf38ca4ffb9a9aac9374ea2094 P 1bcd68d29a7b41a9bf114ac103d4c649: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260709 12:58:02.984292   760 tablet_bootstrap.cc:492] T d441098f471f4b0084938676c20981e5 P 1bcd68d29a7b41a9bf114ac103d4c649: Bootstrap starting.
I20260709 12:58:02.993496   759 tablet_bootstrap.cc:654] T 2e14183cc6064b1cbe6530eb1c8c899c P 263986885d354f7ba08739e419a6cded: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:03.006078   760 tablet_bootstrap.cc:654] T d441098f471f4b0084938676c20981e5 P 1bcd68d29a7b41a9bf114ac103d4c649: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:03.021699   759 tablet_bootstrap.cc:492] T 2e14183cc6064b1cbe6530eb1c8c899c P 263986885d354f7ba08739e419a6cded: No bootstrap required, opened a new log
I20260709 12:58:03.023141   759 ts_tablet_manager.cc:1403] T 2e14183cc6064b1cbe6530eb1c8c899c P 263986885d354f7ba08739e419a6cded: Time spent bootstrapping tablet: real 0.051s	user 0.016s	sys 0.011s
I20260709 12:58:03.029232   760 tablet_bootstrap.cc:492] T d441098f471f4b0084938676c20981e5 P 1bcd68d29a7b41a9bf114ac103d4c649: No bootstrap required, opened a new log
I20260709 12:58:03.029950   760 ts_tablet_manager.cc:1403] T d441098f471f4b0084938676c20981e5 P 1bcd68d29a7b41a9bf114ac103d4c649: Time spent bootstrapping tablet: real 0.046s	user 0.013s	sys 0.014s
I20260709 12:58:03.031250   759 raft_consensus.cc:359] T 2e14183cc6064b1cbe6530eb1c8c899c P 263986885d354f7ba08739e419a6cded [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } }
I20260709 12:58:03.032357   759 raft_consensus.cc:385] T 2e14183cc6064b1cbe6530eb1c8c899c P 263986885d354f7ba08739e419a6cded [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:03.032768   759 raft_consensus.cc:740] T 2e14183cc6064b1cbe6530eb1c8c899c P 263986885d354f7ba08739e419a6cded [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 263986885d354f7ba08739e419a6cded, State: Initialized, Role: FOLLOWER
I20260709 12:58:03.033913   759 consensus_queue.cc:260] T 2e14183cc6064b1cbe6530eb1c8c899c P 263986885d354f7ba08739e419a6cded [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } }
I20260709 12:58:03.037643   759 ts_tablet_manager.cc:1434] T 2e14183cc6064b1cbe6530eb1c8c899c P 263986885d354f7ba08739e419a6cded: Time spent starting tablet: real 0.014s	user 0.002s	sys 0.008s
I20260709 12:58:03.037384   760 raft_consensus.cc:359] T d441098f471f4b0084938676c20981e5 P 1bcd68d29a7b41a9bf114ac103d4c649 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } }
I20260709 12:58:03.038504   760 raft_consensus.cc:385] T d441098f471f4b0084938676c20981e5 P 1bcd68d29a7b41a9bf114ac103d4c649 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:03.039011   759 tablet_bootstrap.cc:492] T e6aabcfcdeb34e6c94ec0bb089deabda P 263986885d354f7ba08739e419a6cded: Bootstrap starting.
I20260709 12:58:03.039029   760 raft_consensus.cc:740] T d441098f471f4b0084938676c20981e5 P 1bcd68d29a7b41a9bf114ac103d4c649 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bcd68d29a7b41a9bf114ac103d4c649, State: Initialized, Role: FOLLOWER
I20260709 12:58:03.040488   760 consensus_queue.cc:260] T d441098f471f4b0084938676c20981e5 P 1bcd68d29a7b41a9bf114ac103d4c649 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } }
I20260709 12:58:03.044016   760 ts_tablet_manager.cc:1434] T d441098f471f4b0084938676c20981e5 P 1bcd68d29a7b41a9bf114ac103d4c649: Time spent starting tablet: real 0.014s	user 0.004s	sys 0.011s
I20260709 12:58:03.045382   760 tablet_bootstrap.cc:492] T 59112618a86046e1af899ce6d3c85203 P 1bcd68d29a7b41a9bf114ac103d4c649: Bootstrap starting.
I20260709 12:58:03.057382   760 tablet_bootstrap.cc:654] T 59112618a86046e1af899ce6d3c85203 P 1bcd68d29a7b41a9bf114ac103d4c649: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:03.059388   759 tablet_bootstrap.cc:654] T e6aabcfcdeb34e6c94ec0bb089deabda P 263986885d354f7ba08739e419a6cded: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:03.099418   766 raft_consensus.cc:493] T d441098f471f4b0084938676c20981e5 P 1bcd68d29a7b41a9bf114ac103d4c649 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:03.100567   766 raft_consensus.cc:515] T d441098f471f4b0084938676c20981e5 P 1bcd68d29a7b41a9bf114ac103d4c649 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } }
I20260709 12:58:03.107123   766 leader_election.cc:290] T d441098f471f4b0084938676c20981e5 P 1bcd68d29a7b41a9bf114ac103d4c649 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b5e46fc5aac9426a945bb34207d634d4 (127.0.107.193:38425), 263986885d354f7ba08739e419a6cded (127.0.107.194:45491)
I20260709 12:58:03.185755   761 tablet_bootstrap.cc:492] T dd06089bfde94412b12705a116ecf3d5 P b5e46fc5aac9426a945bb34207d634d4: No bootstrap required, opened a new log
I20260709 12:58:03.187353   766 raft_consensus.cc:493] T 793bafcf38ca4ffb9a9aac9374ea2094 P 1bcd68d29a7b41a9bf114ac103d4c649 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:03.188155   766 raft_consensus.cc:515] T 793bafcf38ca4ffb9a9aac9374ea2094 P 1bcd68d29a7b41a9bf114ac103d4c649 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } } peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } }
I20260709 12:58:03.190784   761 ts_tablet_manager.cc:1403] T dd06089bfde94412b12705a116ecf3d5 P b5e46fc5aac9426a945bb34207d634d4: Time spent bootstrapping tablet: real 0.308s	user 0.019s	sys 0.014s
W20260709 12:58:03.249337   450 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.0.107.193:38425, user_credentials={real_user=slave}} blocked reactor thread for 77120.3us
I20260709 12:58:03.259008   761 raft_consensus.cc:359] T dd06089bfde94412b12705a116ecf3d5 P b5e46fc5aac9426a945bb34207d634d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } }
I20260709 12:58:03.277067   761 raft_consensus.cc:385] T dd06089bfde94412b12705a116ecf3d5 P b5e46fc5aac9426a945bb34207d634d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:03.277170   766 leader_election.cc:290] T 793bafcf38ca4ffb9a9aac9374ea2094 P 1bcd68d29a7b41a9bf114ac103d4c649 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 263986885d354f7ba08739e419a6cded (127.0.107.194:45491), b5e46fc5aac9426a945bb34207d634d4 (127.0.107.193:38425)
I20260709 12:58:03.277724   761 raft_consensus.cc:740] T dd06089bfde94412b12705a116ecf3d5 P b5e46fc5aac9426a945bb34207d634d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5e46fc5aac9426a945bb34207d634d4, State: Initialized, Role: FOLLOWER
I20260709 12:58:03.278924   761 consensus_queue.cc:260] T dd06089bfde94412b12705a116ecf3d5 P b5e46fc5aac9426a945bb34207d634d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } }
I20260709 12:58:03.367115   761 ts_tablet_manager.cc:1434] T dd06089bfde94412b12705a116ecf3d5 P b5e46fc5aac9426a945bb34207d634d4: Time spent starting tablet: real 0.175s	user 0.035s	sys 0.040s
I20260709 12:58:03.368928   761 tablet_bootstrap.cc:492] T 793bafcf38ca4ffb9a9aac9374ea2094 P b5e46fc5aac9426a945bb34207d634d4: Bootstrap starting.
I20260709 12:58:03.379617   641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d441098f471f4b0084938676c20981e5" candidate_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "263986885d354f7ba08739e419a6cded" is_pre_election: true
I20260709 12:58:03.380061   640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "793bafcf38ca4ffb9a9aac9374ea2094" candidate_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "263986885d354f7ba08739e419a6cded" is_pre_election: true
I20260709 12:58:03.382690   759 tablet_bootstrap.cc:492] T e6aabcfcdeb34e6c94ec0bb089deabda P 263986885d354f7ba08739e419a6cded: No bootstrap required, opened a new log
I20260709 12:58:03.383620   759 ts_tablet_manager.cc:1403] T e6aabcfcdeb34e6c94ec0bb089deabda P 263986885d354f7ba08739e419a6cded: Time spent bootstrapping tablet: real 0.345s	user 0.052s	sys 0.111s
I20260709 12:58:03.381844   641 raft_consensus.cc:2468] T d441098f471f4b0084938676c20981e5 P 263986885d354f7ba08739e419a6cded [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bcd68d29a7b41a9bf114ac103d4c649 in term 0.
I20260709 12:58:03.384433   760 tablet_bootstrap.cc:492] T 59112618a86046e1af899ce6d3c85203 P 1bcd68d29a7b41a9bf114ac103d4c649: No bootstrap required, opened a new log
I20260709 12:58:03.385456   760 ts_tablet_manager.cc:1403] T 59112618a86046e1af899ce6d3c85203 P 1bcd68d29a7b41a9bf114ac103d4c649: Time spent bootstrapping tablet: real 0.340s	user 0.101s	sys 0.175s
W20260709 12:58:03.388672   680 leader_election.cc:343] T 793bafcf38ca4ffb9a9aac9374ea2094 P 1bcd68d29a7b41a9bf114ac103d4c649 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 263986885d354f7ba08739e419a6cded (127.0.107.194:45491): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:03.390269   680 leader_election.cc:304] T d441098f471f4b0084938676c20981e5 P 1bcd68d29a7b41a9bf114ac103d4c649 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bcd68d29a7b41a9bf114ac103d4c649, 263986885d354f7ba08739e419a6cded; no voters: 
I20260709 12:58:03.391918   766 raft_consensus.cc:2804] T d441098f471f4b0084938676c20981e5 P 1bcd68d29a7b41a9bf114ac103d4c649 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:58:03.392472   766 raft_consensus.cc:493] T d441098f471f4b0084938676c20981e5 P 1bcd68d29a7b41a9bf114ac103d4c649 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:58:03.393033   766 raft_consensus.cc:3060] T d441098f471f4b0084938676c20981e5 P 1bcd68d29a7b41a9bf114ac103d4c649 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:03.399644   761 tablet_bootstrap.cc:654] T 793bafcf38ca4ffb9a9aac9374ea2094 P b5e46fc5aac9426a945bb34207d634d4: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:03.394053   760 raft_consensus.cc:359] T 59112618a86046e1af899ce6d3c85203 P 1bcd68d29a7b41a9bf114ac103d4c649 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } }
I20260709 12:58:03.404192   760 raft_consensus.cc:385] T 59112618a86046e1af899ce6d3c85203 P 1bcd68d29a7b41a9bf114ac103d4c649 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:03.405059   760 raft_consensus.cc:740] T 59112618a86046e1af899ce6d3c85203 P 1bcd68d29a7b41a9bf114ac103d4c649 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bcd68d29a7b41a9bf114ac103d4c649, State: Initialized, Role: FOLLOWER
I20260709 12:58:03.406684   760 consensus_queue.cc:260] T 59112618a86046e1af899ce6d3c85203 P 1bcd68d29a7b41a9bf114ac103d4c649 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } }
I20260709 12:58:03.410214   759 raft_consensus.cc:359] T e6aabcfcdeb34e6c94ec0bb089deabda P 263986885d354f7ba08739e419a6cded [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } }
I20260709 12:58:03.411414   759 raft_consensus.cc:385] T e6aabcfcdeb34e6c94ec0bb089deabda P 263986885d354f7ba08739e419a6cded [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:03.411928   759 raft_consensus.cc:740] T e6aabcfcdeb34e6c94ec0bb089deabda P 263986885d354f7ba08739e419a6cded [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 263986885d354f7ba08739e419a6cded, State: Initialized, Role: FOLLOWER
I20260709 12:58:03.416472   759 consensus_queue.cc:260] T e6aabcfcdeb34e6c94ec0bb089deabda P 263986885d354f7ba08739e419a6cded [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } }
I20260709 12:58:03.418327   761 tablet_bootstrap.cc:492] T 793bafcf38ca4ffb9a9aac9374ea2094 P b5e46fc5aac9426a945bb34207d634d4: No bootstrap required, opened a new log
I20260709 12:58:03.419164   566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d441098f471f4b0084938676c20981e5" candidate_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5e46fc5aac9426a945bb34207d634d4" is_pre_election: true
I20260709 12:58:03.420403   761 ts_tablet_manager.cc:1403] T 793bafcf38ca4ffb9a9aac9374ea2094 P b5e46fc5aac9426a945bb34207d634d4: Time spent bootstrapping tablet: real 0.052s	user 0.011s	sys 0.019s
I20260709 12:58:03.419598   565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "793bafcf38ca4ffb9a9aac9374ea2094" candidate_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5e46fc5aac9426a945bb34207d634d4" is_pre_election: true
I20260709 12:58:03.418745   766 raft_consensus.cc:515] T d441098f471f4b0084938676c20981e5 P 1bcd68d29a7b41a9bf114ac103d4c649 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } }
I20260709 12:58:03.422472   759 ts_tablet_manager.cc:1434] T e6aabcfcdeb34e6c94ec0bb089deabda P 263986885d354f7ba08739e419a6cded: Time spent starting tablet: real 0.037s	user 0.002s	sys 0.012s
I20260709 12:58:03.424178   759 tablet_bootstrap.cc:492] T 793bafcf38ca4ffb9a9aac9374ea2094 P 263986885d354f7ba08739e419a6cded: Bootstrap starting.
I20260709 12:58:03.430295   761 raft_consensus.cc:359] T 793bafcf38ca4ffb9a9aac9374ea2094 P b5e46fc5aac9426a945bb34207d634d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } } peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } }
I20260709 12:58:03.431519   761 raft_consensus.cc:385] T 793bafcf38ca4ffb9a9aac9374ea2094 P b5e46fc5aac9426a945bb34207d634d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:03.431906   761 raft_consensus.cc:740] T 793bafcf38ca4ffb9a9aac9374ea2094 P b5e46fc5aac9426a945bb34207d634d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5e46fc5aac9426a945bb34207d634d4, State: Initialized, Role: FOLLOWER
I20260709 12:58:03.433063   761 consensus_queue.cc:260] T 793bafcf38ca4ffb9a9aac9374ea2094 P b5e46fc5aac9426a945bb34207d634d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } } peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } }
W20260709 12:58:03.439152   681 leader_election.cc:343] T d441098f471f4b0084938676c20981e5 P 1bcd68d29a7b41a9bf114ac103d4c649 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b5e46fc5aac9426a945bb34207d634d4 (127.0.107.193:38425): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:03.441578   774 raft_consensus.cc:493] T dd06089bfde94412b12705a116ecf3d5 P b5e46fc5aac9426a945bb34207d634d4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:03.442405   774 raft_consensus.cc:515] T dd06089bfde94412b12705a116ecf3d5 P b5e46fc5aac9426a945bb34207d634d4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } }
I20260709 12:58:03.447501   761 ts_tablet_manager.cc:1434] T 793bafcf38ca4ffb9a9aac9374ea2094 P b5e46fc5aac9426a945bb34207d634d4: Time spent starting tablet: real 0.026s	user 0.007s	sys 0.005s
I20260709 12:58:03.444455   566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d441098f471f4b0084938676c20981e5" candidate_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5e46fc5aac9426a945bb34207d634d4"
I20260709 12:58:03.449298   761 tablet_bootstrap.cc:492] T d4d26ba9fb084e748bddb93f7547621f P b5e46fc5aac9426a945bb34207d634d4: Bootstrap starting.
W20260709 12:58:03.449497   681 leader_election.cc:343] T d441098f471f4b0084938676c20981e5 P 1bcd68d29a7b41a9bf114ac103d4c649 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b5e46fc5aac9426a945bb34207d634d4 (127.0.107.193:38425): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:03.443807   565 raft_consensus.cc:2468] T 793bafcf38ca4ffb9a9aac9374ea2094 P b5e46fc5aac9426a945bb34207d634d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bcd68d29a7b41a9bf114ac103d4c649 in term 0.
I20260709 12:58:03.453856   774 leader_election.cc:290] T dd06089bfde94412b12705a116ecf3d5 P b5e46fc5aac9426a945bb34207d634d4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 263986885d354f7ba08739e419a6cded (127.0.107.194:45491), 1bcd68d29a7b41a9bf114ac103d4c649 (127.0.107.195:46365)
I20260709 12:58:03.460496   640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d441098f471f4b0084938676c20981e5" candidate_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "263986885d354f7ba08739e419a6cded"
I20260709 12:58:03.461421   640 raft_consensus.cc:3060] T d441098f471f4b0084938676c20981e5 P 263986885d354f7ba08739e419a6cded [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:03.451982   766 leader_election.cc:290] T d441098f471f4b0084938676c20981e5 P 1bcd68d29a7b41a9bf114ac103d4c649 [CANDIDATE]: Term 1 election: Requested vote from peers b5e46fc5aac9426a945bb34207d634d4 (127.0.107.193:38425), 263986885d354f7ba08739e419a6cded (127.0.107.194:45491)
I20260709 12:58:03.466673   681 leader_election.cc:304] T 793bafcf38ca4ffb9a9aac9374ea2094 P 1bcd68d29a7b41a9bf114ac103d4c649 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1bcd68d29a7b41a9bf114ac103d4c649, b5e46fc5aac9426a945bb34207d634d4; no voters: 263986885d354f7ba08739e419a6cded
I20260709 12:58:03.468775   766 raft_consensus.cc:2804] T 793bafcf38ca4ffb9a9aac9374ea2094 P 1bcd68d29a7b41a9bf114ac103d4c649 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:58:03.469386   766 raft_consensus.cc:493] T 793bafcf38ca4ffb9a9aac9374ea2094 P 1bcd68d29a7b41a9bf114ac103d4c649 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:58:03.469928   766 raft_consensus.cc:3060] T 793bafcf38ca4ffb9a9aac9374ea2094 P 1bcd68d29a7b41a9bf114ac103d4c649 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:03.475224   759 tablet_bootstrap.cc:654] T 793bafcf38ca4ffb9a9aac9374ea2094 P 263986885d354f7ba08739e419a6cded: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:03.477372   760 ts_tablet_manager.cc:1434] T 59112618a86046e1af899ce6d3c85203 P 1bcd68d29a7b41a9bf114ac103d4c649: Time spent starting tablet: real 0.091s	user 0.030s	sys 0.013s
I20260709 12:58:03.485776   760 tablet_bootstrap.cc:492] T e6aabcfcdeb34e6c94ec0bb089deabda P 1bcd68d29a7b41a9bf114ac103d4c649: Bootstrap starting.
I20260709 12:58:03.493374   761 tablet_bootstrap.cc:654] T d4d26ba9fb084e748bddb93f7547621f P b5e46fc5aac9426a945bb34207d634d4: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:03.506067   640 raft_consensus.cc:2468] T d441098f471f4b0084938676c20981e5 P 263986885d354f7ba08739e419a6cded [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1bcd68d29a7b41a9bf114ac103d4c649 in term 1.
I20260709 12:58:03.513932   680 leader_election.cc:304] T d441098f471f4b0084938676c20981e5 P 1bcd68d29a7b41a9bf114ac103d4c649 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1bcd68d29a7b41a9bf114ac103d4c649, 263986885d354f7ba08739e419a6cded; no voters: b5e46fc5aac9426a945bb34207d634d4
I20260709 12:58:03.517311   777 raft_consensus.cc:2804] T d441098f471f4b0084938676c20981e5 P 1bcd68d29a7b41a9bf114ac103d4c649 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:58:03.530516   765 raft_consensus.cc:493] T 2e14183cc6064b1cbe6530eb1c8c899c P 263986885d354f7ba08739e419a6cded [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:03.531149   765 raft_consensus.cc:515] T 2e14183cc6064b1cbe6530eb1c8c899c P 263986885d354f7ba08739e419a6cded [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } }
I20260709 12:58:03.535629   777 raft_consensus.cc:697] T d441098f471f4b0084938676c20981e5 P 1bcd68d29a7b41a9bf114ac103d4c649 [term 1 LEADER]: Becoming Leader. State: Replica: 1bcd68d29a7b41a9bf114ac103d4c649, State: Running, Role: LEADER
I20260709 12:58:03.541599   760 tablet_bootstrap.cc:654] T e6aabcfcdeb34e6c94ec0bb089deabda P 1bcd68d29a7b41a9bf114ac103d4c649: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:03.553004   759 tablet_bootstrap.cc:492] T 793bafcf38ca4ffb9a9aac9374ea2094 P 263986885d354f7ba08739e419a6cded: No bootstrap required, opened a new log
I20260709 12:58:03.553650   759 ts_tablet_manager.cc:1403] T 793bafcf38ca4ffb9a9aac9374ea2094 P 263986885d354f7ba08739e419a6cded: Time spent bootstrapping tablet: real 0.130s	user 0.021s	sys 0.031s
I20260709 12:58:03.553787   766 raft_consensus.cc:515] T 793bafcf38ca4ffb9a9aac9374ea2094 P 1bcd68d29a7b41a9bf114ac103d4c649 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } } peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } }
I20260709 12:58:03.546393   765 leader_election.cc:290] T 2e14183cc6064b1cbe6530eb1c8c899c P 263986885d354f7ba08739e419a6cded [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b5e46fc5aac9426a945bb34207d634d4 (127.0.107.193:38425), 1bcd68d29a7b41a9bf114ac103d4c649 (127.0.107.195:46365)
I20260709 12:58:03.577427   766 leader_election.cc:290] T 793bafcf38ca4ffb9a9aac9374ea2094 P 1bcd68d29a7b41a9bf114ac103d4c649 [CANDIDATE]: Term 1 election: Requested vote from peers 263986885d354f7ba08739e419a6cded (127.0.107.194:45491), b5e46fc5aac9426a945bb34207d634d4 (127.0.107.193:38425)
I20260709 12:58:03.581960   640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "793bafcf38ca4ffb9a9aac9374ea2094" candidate_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "263986885d354f7ba08739e419a6cded"
W20260709 12:58:03.593150   680 leader_election.cc:343] T 793bafcf38ca4ffb9a9aac9374ea2094 P 1bcd68d29a7b41a9bf114ac103d4c649 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 263986885d354f7ba08739e419a6cded (127.0.107.194:45491): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:03.544605   777 consensus_queue.cc:237] T d441098f471f4b0084938676c20981e5 P 1bcd68d29a7b41a9bf114ac103d4c649 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } }
I20260709 12:58:03.593824   759 raft_consensus.cc:359] T 793bafcf38ca4ffb9a9aac9374ea2094 P 263986885d354f7ba08739e419a6cded [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } } peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } }
I20260709 12:58:03.597035   759 raft_consensus.cc:385] T 793bafcf38ca4ffb9a9aac9374ea2094 P 263986885d354f7ba08739e419a6cded [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:03.599918   565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "793bafcf38ca4ffb9a9aac9374ea2094" candidate_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5e46fc5aac9426a945bb34207d634d4"
I20260709 12:58:03.601071   565 raft_consensus.cc:3060] T 793bafcf38ca4ffb9a9aac9374ea2094 P b5e46fc5aac9426a945bb34207d634d4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:03.597534   759 raft_consensus.cc:740] T 793bafcf38ca4ffb9a9aac9374ea2094 P 263986885d354f7ba08739e419a6cded [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 263986885d354f7ba08739e419a6cded, State: Initialized, Role: FOLLOWER
I20260709 12:58:03.607610   759 consensus_queue.cc:260] T 793bafcf38ca4ffb9a9aac9374ea2094 P 263986885d354f7ba08739e419a6cded [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } } peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } }
I20260709 12:58:03.619078   640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd06089bfde94412b12705a116ecf3d5" candidate_uuid: "b5e46fc5aac9426a945bb34207d634d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "263986885d354f7ba08739e419a6cded" is_pre_election: true
W20260709 12:58:03.627372   531 leader_election.cc:343] T dd06089bfde94412b12705a116ecf3d5 P b5e46fc5aac9426a945bb34207d634d4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 263986885d354f7ba08739e419a6cded (127.0.107.194:45491): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:03.630334   759 ts_tablet_manager.cc:1434] T 793bafcf38ca4ffb9a9aac9374ea2094 P 263986885d354f7ba08739e419a6cded: Time spent starting tablet: real 0.076s	user 0.002s	sys 0.013s
I20260709 12:58:03.635775   759 tablet_bootstrap.cc:492] T d8b33c9315094794a876a4b62755efaf P 263986885d354f7ba08739e419a6cded: Bootstrap starting.
I20260709 12:58:03.645969   760 tablet_bootstrap.cc:492] T e6aabcfcdeb34e6c94ec0bb089deabda P 1bcd68d29a7b41a9bf114ac103d4c649: No bootstrap required, opened a new log
I20260709 12:58:03.645818   765 raft_consensus.cc:493] T 793bafcf38ca4ffb9a9aac9374ea2094 P 263986885d354f7ba08739e419a6cded [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:03.646997   760 ts_tablet_manager.cc:1403] T e6aabcfcdeb34e6c94ec0bb089deabda P 1bcd68d29a7b41a9bf114ac103d4c649: Time spent bootstrapping tablet: real 0.162s	user 0.015s	sys 0.018s
I20260709 12:58:03.647004   765 raft_consensus.cc:515] T 793bafcf38ca4ffb9a9aac9374ea2094 P 263986885d354f7ba08739e419a6cded [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } } peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } }
I20260709 12:58:03.650610   761 tablet_bootstrap.cc:492] T d4d26ba9fb084e748bddb93f7547621f P b5e46fc5aac9426a945bb34207d634d4: No bootstrap required, opened a new log
I20260709 12:58:03.651336   761 ts_tablet_manager.cc:1403] T d4d26ba9fb084e748bddb93f7547621f P b5e46fc5aac9426a945bb34207d634d4: Time spent bootstrapping tablet: real 0.202s	user 0.028s	sys 0.040s
W20260709 12:58:03.660815   511 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:58:03.661619   765 leader_election.cc:290] T 793bafcf38ca4ffb9a9aac9374ea2094 P 263986885d354f7ba08739e419a6cded [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1bcd68d29a7b41a9bf114ac103d4c649 (127.0.107.195:46365), b5e46fc5aac9426a945bb34207d634d4 (127.0.107.193:38425)
I20260709 12:58:03.669664   715 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd06089bfde94412b12705a116ecf3d5" candidate_uuid: "b5e46fc5aac9426a945bb34207d634d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" is_pre_election: true
W20260709 12:58:03.675232   529 leader_election.cc:343] T dd06089bfde94412b12705a116ecf3d5 P b5e46fc5aac9426a945bb34207d634d4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1bcd68d29a7b41a9bf114ac103d4c649 (127.0.107.195:46365): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:03.676330   529 leader_election.cc:304] T dd06089bfde94412b12705a116ecf3d5 P b5e46fc5aac9426a945bb34207d634d4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b5e46fc5aac9426a945bb34207d634d4; no voters: 1bcd68d29a7b41a9bf114ac103d4c649, 263986885d354f7ba08739e419a6cded
I20260709 12:58:03.678925   774 raft_consensus.cc:2749] T dd06089bfde94412b12705a116ecf3d5 P b5e46fc5aac9426a945bb34207d634d4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:58:03.682186   566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e14183cc6064b1cbe6530eb1c8c899c" candidate_uuid: "263986885d354f7ba08739e419a6cded" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5e46fc5aac9426a945bb34207d634d4" is_pre_election: true
I20260709 12:58:03.685086   564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "793bafcf38ca4ffb9a9aac9374ea2094" candidate_uuid: "263986885d354f7ba08739e419a6cded" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5e46fc5aac9426a945bb34207d634d4" is_pre_election: true
I20260709 12:58:03.670246   760 raft_consensus.cc:359] T e6aabcfcdeb34e6c94ec0bb089deabda P 1bcd68d29a7b41a9bf114ac103d4c649 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } }
I20260709 12:58:03.682783   759 tablet_bootstrap.cc:654] T d8b33c9315094794a876a4b62755efaf P 263986885d354f7ba08739e419a6cded: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:03.689064   760 raft_consensus.cc:385] T e6aabcfcdeb34e6c94ec0bb089deabda P 1bcd68d29a7b41a9bf114ac103d4c649 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260709 12:58:03.692873   607 leader_election.cc:343] T 2e14183cc6064b1cbe6530eb1c8c899c P 263986885d354f7ba08739e419a6cded [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b5e46fc5aac9426a945bb34207d634d4 (127.0.107.193:38425): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:03.689587   760 raft_consensus.cc:740] T e6aabcfcdeb34e6c94ec0bb089deabda P 1bcd68d29a7b41a9bf114ac103d4c649 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bcd68d29a7b41a9bf114ac103d4c649, State: Initialized, Role: FOLLOWER
I20260709 12:58:03.698158   565 raft_consensus.cc:2468] T 793bafcf38ca4ffb9a9aac9374ea2094 P b5e46fc5aac9426a945bb34207d634d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1bcd68d29a7b41a9bf114ac103d4c649 in term 1.
I20260709 12:58:03.700331   681 leader_election.cc:304] T 793bafcf38ca4ffb9a9aac9374ea2094 P 1bcd68d29a7b41a9bf114ac103d4c649 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1bcd68d29a7b41a9bf114ac103d4c649, b5e46fc5aac9426a945bb34207d634d4; no voters: 263986885d354f7ba08739e419a6cded
I20260709 12:58:03.697216   760 consensus_queue.cc:260] T e6aabcfcdeb34e6c94ec0bb089deabda P 1bcd68d29a7b41a9bf114ac103d4c649 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } }
I20260709 12:58:03.702876   766 raft_consensus.cc:493] T e6aabcfcdeb34e6c94ec0bb089deabda P 1bcd68d29a7b41a9bf114ac103d4c649 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:03.703646   766 raft_consensus.cc:515] T e6aabcfcdeb34e6c94ec0bb089deabda P 1bcd68d29a7b41a9bf114ac103d4c649 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } }
I20260709 12:58:03.702842   782 raft_consensus.cc:2804] T 793bafcf38ca4ffb9a9aac9374ea2094 P 1bcd68d29a7b41a9bf114ac103d4c649 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:58:03.701233   761 raft_consensus.cc:359] T d4d26ba9fb084e748bddb93f7547621f P b5e46fc5aac9426a945bb34207d634d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } }
I20260709 12:58:03.709354   761 raft_consensus.cc:385] T d4d26ba9fb084e748bddb93f7547621f P b5e46fc5aac9426a945bb34207d634d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:03.709542   782 raft_consensus.cc:697] T 793bafcf38ca4ffb9a9aac9374ea2094 P 1bcd68d29a7b41a9bf114ac103d4c649 [term 1 LEADER]: Becoming Leader. State: Replica: 1bcd68d29a7b41a9bf114ac103d4c649, State: Running, Role: LEADER
I20260709 12:58:03.710080   761 raft_consensus.cc:740] T d4d26ba9fb084e748bddb93f7547621f P b5e46fc5aac9426a945bb34207d634d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5e46fc5aac9426a945bb34207d634d4, State: Initialized, Role: FOLLOWER
I20260709 12:58:03.711941   715 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e14183cc6064b1cbe6530eb1c8c899c" candidate_uuid: "263986885d354f7ba08739e419a6cded" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" is_pre_election: true
I20260709 12:58:03.711217   782 consensus_queue.cc:237] T 793bafcf38ca4ffb9a9aac9374ea2094 P 1bcd68d29a7b41a9bf114ac103d4c649 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } } peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } }
I20260709 12:58:03.714628   766 leader_election.cc:290] T e6aabcfcdeb34e6c94ec0bb089deabda P 1bcd68d29a7b41a9bf114ac103d4c649 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b5e46fc5aac9426a945bb34207d634d4 (127.0.107.193:38425), 263986885d354f7ba08739e419a6cded (127.0.107.194:45491)
I20260709 12:58:03.715448   714 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "793bafcf38ca4ffb9a9aac9374ea2094" candidate_uuid: "263986885d354f7ba08739e419a6cded" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" is_pre_election: true
I20260709 12:58:03.711669   761 consensus_queue.cc:260] T d4d26ba9fb084e748bddb93f7547621f P b5e46fc5aac9426a945bb34207d634d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } }
I20260709 12:58:03.717427   565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e6aabcfcdeb34e6c94ec0bb089deabda" candidate_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5e46fc5aac9426a945bb34207d634d4" is_pre_election: true
I20260709 12:58:03.719485   640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e6aabcfcdeb34e6c94ec0bb089deabda" candidate_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "263986885d354f7ba08739e419a6cded" is_pre_election: true
W20260709 12:58:03.720171   604 leader_election.cc:343] T 2e14183cc6064b1cbe6530eb1c8c899c P 263986885d354f7ba08739e419a6cded [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1bcd68d29a7b41a9bf114ac103d4c649 (127.0.107.195:46365): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:03.720477   640 raft_consensus.cc:2468] T e6aabcfcdeb34e6c94ec0bb089deabda P 263986885d354f7ba08739e419a6cded [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bcd68d29a7b41a9bf114ac103d4c649 in term 0.
I20260709 12:58:03.720973   604 leader_election.cc:304] T 2e14183cc6064b1cbe6530eb1c8c899c P 263986885d354f7ba08739e419a6cded [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 263986885d354f7ba08739e419a6cded; no voters: 1bcd68d29a7b41a9bf114ac103d4c649, b5e46fc5aac9426a945bb34207d634d4
I20260709 12:58:03.722141   680 leader_election.cc:304] T e6aabcfcdeb34e6c94ec0bb089deabda P 1bcd68d29a7b41a9bf114ac103d4c649 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bcd68d29a7b41a9bf114ac103d4c649, 263986885d354f7ba08739e419a6cded; no voters: 
I20260709 12:58:03.722982   765 raft_consensus.cc:2749] T 2e14183cc6064b1cbe6530eb1c8c899c P 263986885d354f7ba08739e419a6cded [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:58:03.723897   766 raft_consensus.cc:2804] T e6aabcfcdeb34e6c94ec0bb089deabda P 1bcd68d29a7b41a9bf114ac103d4c649 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:58:03.726022   766 raft_consensus.cc:493] T e6aabcfcdeb34e6c94ec0bb089deabda P 1bcd68d29a7b41a9bf114ac103d4c649 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:58:03.723896   604 leader_election.cc:304] T 793bafcf38ca4ffb9a9aac9374ea2094 P 263986885d354f7ba08739e419a6cded [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 263986885d354f7ba08739e419a6cded; no voters: 1bcd68d29a7b41a9bf114ac103d4c649, b5e46fc5aac9426a945bb34207d634d4
I20260709 12:58:03.727064   765 raft_consensus.cc:2749] T 793bafcf38ca4ffb9a9aac9374ea2094 P 263986885d354f7ba08739e419a6cded [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:58:03.726616   766 raft_consensus.cc:3060] T e6aabcfcdeb34e6c94ec0bb089deabda P 1bcd68d29a7b41a9bf114ac103d4c649 [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:58:03.725750   681 leader_election.cc:343] T e6aabcfcdeb34e6c94ec0bb089deabda P 1bcd68d29a7b41a9bf114ac103d4c649 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b5e46fc5aac9426a945bb34207d634d4 (127.0.107.193:38425): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:03.715502   463 catalog_manager.cc:5697] T d441098f471f4b0084938676c20981e5 P 1bcd68d29a7b41a9bf114ac103d4c649 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1bcd68d29a7b41a9bf114ac103d4c649 (127.0.107.195). New cstate: current_term: 1 leader_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } health_report { overall_health: HEALTHY } } }
I20260709 12:58:03.741329   777 raft_consensus.cc:493] T 59112618a86046e1af899ce6d3c85203 P 1bcd68d29a7b41a9bf114ac103d4c649 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:03.743639   777 raft_consensus.cc:515] T 59112618a86046e1af899ce6d3c85203 P 1bcd68d29a7b41a9bf114ac103d4c649 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } }
I20260709 12:58:03.744472   760 ts_tablet_manager.cc:1434] T e6aabcfcdeb34e6c94ec0bb089deabda P 1bcd68d29a7b41a9bf114ac103d4c649: Time spent starting tablet: real 0.097s	user 0.010s	sys 0.006s
I20260709 12:58:03.749775   565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59112618a86046e1af899ce6d3c85203" candidate_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5e46fc5aac9426a945bb34207d634d4" is_pre_election: true
I20260709 12:58:03.750025   640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59112618a86046e1af899ce6d3c85203" candidate_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "263986885d354f7ba08739e419a6cded" is_pre_election: true
W20260709 12:58:03.751976   680 leader_election.cc:343] T 59112618a86046e1af899ce6d3c85203 P 1bcd68d29a7b41a9bf114ac103d4c649 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 263986885d354f7ba08739e419a6cded (127.0.107.194:45491): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:58:03.752672   681 leader_election.cc:343] T 59112618a86046e1af899ce6d3c85203 P 1bcd68d29a7b41a9bf114ac103d4c649 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b5e46fc5aac9426a945bb34207d634d4 (127.0.107.193:38425): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:03.753540   681 leader_election.cc:304] T 59112618a86046e1af899ce6d3c85203 P 1bcd68d29a7b41a9bf114ac103d4c649 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1bcd68d29a7b41a9bf114ac103d4c649; no voters: 263986885d354f7ba08739e419a6cded, b5e46fc5aac9426a945bb34207d634d4
I20260709 12:58:03.752739   777 leader_election.cc:290] T 59112618a86046e1af899ce6d3c85203 P 1bcd68d29a7b41a9bf114ac103d4c649 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b5e46fc5aac9426a945bb34207d634d4 (127.0.107.193:38425), 263986885d354f7ba08739e419a6cded (127.0.107.194:45491)
I20260709 12:58:03.754639   761 ts_tablet_manager.cc:1434] T d4d26ba9fb084e748bddb93f7547621f P b5e46fc5aac9426a945bb34207d634d4: Time spent starting tablet: real 0.103s	user 0.004s	sys 0.010s
I20260709 12:58:03.746295   760 tablet_bootstrap.cc:492] T 2e14183cc6064b1cbe6530eb1c8c899c P 1bcd68d29a7b41a9bf114ac103d4c649: Bootstrap starting.
I20260709 12:58:03.762807   761 tablet_bootstrap.cc:492] T 59112618a86046e1af899ce6d3c85203 P b5e46fc5aac9426a945bb34207d634d4: Bootstrap starting.
I20260709 12:58:03.763492   777 raft_consensus.cc:2749] T 59112618a86046e1af899ce6d3c85203 P 1bcd68d29a7b41a9bf114ac103d4c649 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:58:03.781848   759 tablet_bootstrap.cc:492] T d8b33c9315094794a876a4b62755efaf P 263986885d354f7ba08739e419a6cded: No bootstrap required, opened a new log
I20260709 12:58:03.782938   759 ts_tablet_manager.cc:1403] T d8b33c9315094794a876a4b62755efaf P 263986885d354f7ba08739e419a6cded: Time spent bootstrapping tablet: real 0.149s	user 0.016s	sys 0.017s
I20260709 12:58:03.787586   766 raft_consensus.cc:515] T e6aabcfcdeb34e6c94ec0bb089deabda P 1bcd68d29a7b41a9bf114ac103d4c649 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } }
I20260709 12:58:03.789263   759 raft_consensus.cc:359] T d8b33c9315094794a876a4b62755efaf P 263986885d354f7ba08739e419a6cded [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } }
I20260709 12:58:03.790524   759 raft_consensus.cc:385] T d8b33c9315094794a876a4b62755efaf P 263986885d354f7ba08739e419a6cded [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:03.790967   759 raft_consensus.cc:740] T d8b33c9315094794a876a4b62755efaf P 263986885d354f7ba08739e419a6cded [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 263986885d354f7ba08739e419a6cded, State: Initialized, Role: FOLLOWER
I20260709 12:58:03.793426   640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e6aabcfcdeb34e6c94ec0bb089deabda" candidate_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "263986885d354f7ba08739e419a6cded"
I20260709 12:58:03.797925   640 raft_consensus.cc:3060] T e6aabcfcdeb34e6c94ec0bb089deabda P 263986885d354f7ba08739e419a6cded [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:58:03.802134   681 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:38425, user_credentials={real_user=slave}} blocked reactor thread for 50121us
I20260709 12:58:03.799567   766 leader_election.cc:290] T e6aabcfcdeb34e6c94ec0bb089deabda P 1bcd68d29a7b41a9bf114ac103d4c649 [CANDIDATE]: Term 1 election: Requested vote from peers b5e46fc5aac9426a945bb34207d634d4 (127.0.107.193:38425), 263986885d354f7ba08739e419a6cded (127.0.107.194:45491)
I20260709 12:58:03.807834   759 consensus_queue.cc:260] T d8b33c9315094794a876a4b62755efaf P 263986885d354f7ba08739e419a6cded [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } }
I20260709 12:58:03.809106   565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e6aabcfcdeb34e6c94ec0bb089deabda" candidate_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5e46fc5aac9426a945bb34207d634d4"
I20260709 12:58:03.813661   761 tablet_bootstrap.cc:654] T 59112618a86046e1af899ce6d3c85203 P b5e46fc5aac9426a945bb34207d634d4: Neither blocks nor log segments found. Creating new log.
W20260709 12:58:03.822073   681 leader_election.cc:343] T e6aabcfcdeb34e6c94ec0bb089deabda P 1bcd68d29a7b41a9bf114ac103d4c649 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b5e46fc5aac9426a945bb34207d634d4 (127.0.107.193:38425): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:03.826609   640 raft_consensus.cc:2468] T e6aabcfcdeb34e6c94ec0bb089deabda P 263986885d354f7ba08739e419a6cded [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1bcd68d29a7b41a9bf114ac103d4c649 in term 1.
I20260709 12:58:03.827145   759 ts_tablet_manager.cc:1434] T d8b33c9315094794a876a4b62755efaf P 263986885d354f7ba08739e419a6cded: Time spent starting tablet: real 0.044s	user 0.006s	sys 0.011s
I20260709 12:58:03.828656   680 leader_election.cc:304] T e6aabcfcdeb34e6c94ec0bb089deabda P 1bcd68d29a7b41a9bf114ac103d4c649 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1bcd68d29a7b41a9bf114ac103d4c649, 263986885d354f7ba08739e419a6cded; no voters: b5e46fc5aac9426a945bb34207d634d4
I20260709 12:58:03.831257   792 raft_consensus.cc:2804] T e6aabcfcdeb34e6c94ec0bb089deabda P 1bcd68d29a7b41a9bf114ac103d4c649 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:58:03.831945   792 raft_consensus.cc:697] T e6aabcfcdeb34e6c94ec0bb089deabda P 1bcd68d29a7b41a9bf114ac103d4c649 [term 1 LEADER]: Becoming Leader. State: Replica: 1bcd68d29a7b41a9bf114ac103d4c649, State: Running, Role: LEADER
I20260709 12:58:03.830332   759 tablet_bootstrap.cc:492] T d4d26ba9fb084e748bddb93f7547621f P 263986885d354f7ba08739e419a6cded: Bootstrap starting.
I20260709 12:58:03.833767   792 consensus_queue.cc:237] T e6aabcfcdeb34e6c94ec0bb089deabda P 1bcd68d29a7b41a9bf114ac103d4c649 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } }
I20260709 12:58:03.836943   760 tablet_bootstrap.cc:654] T 2e14183cc6064b1cbe6530eb1c8c899c P 1bcd68d29a7b41a9bf114ac103d4c649: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:03.857372   761 tablet_bootstrap.cc:492] T 59112618a86046e1af899ce6d3c85203 P b5e46fc5aac9426a945bb34207d634d4: No bootstrap required, opened a new log
I20260709 12:58:03.858381   761 ts_tablet_manager.cc:1403] T 59112618a86046e1af899ce6d3c85203 P b5e46fc5aac9426a945bb34207d634d4: Time spent bootstrapping tablet: real 0.096s	user 0.005s	sys 0.027s
I20260709 12:58:03.867383   761 raft_consensus.cc:359] T 59112618a86046e1af899ce6d3c85203 P b5e46fc5aac9426a945bb34207d634d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } }
I20260709 12:58:03.868762   761 raft_consensus.cc:385] T 59112618a86046e1af899ce6d3c85203 P b5e46fc5aac9426a945bb34207d634d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:03.869339   761 raft_consensus.cc:740] T 59112618a86046e1af899ce6d3c85203 P b5e46fc5aac9426a945bb34207d634d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5e46fc5aac9426a945bb34207d634d4, State: Initialized, Role: FOLLOWER
I20260709 12:58:03.870885   761 consensus_queue.cc:260] T 59112618a86046e1af899ce6d3c85203 P b5e46fc5aac9426a945bb34207d634d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } }
I20260709 12:58:03.866371   759 tablet_bootstrap.cc:654] T d4d26ba9fb084e748bddb93f7547621f P 263986885d354f7ba08739e419a6cded: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:03.881935   761 ts_tablet_manager.cc:1434] T 59112618a86046e1af899ce6d3c85203 P b5e46fc5aac9426a945bb34207d634d4: Time spent starting tablet: real 0.022s	user 0.006s	sys 0.008s
I20260709 12:58:03.883217   761 tablet_bootstrap.cc:492] T d8b33c9315094794a876a4b62755efaf P b5e46fc5aac9426a945bb34207d634d4: Bootstrap starting.
I20260709 12:58:03.890612   760 tablet_bootstrap.cc:492] T 2e14183cc6064b1cbe6530eb1c8c899c P 1bcd68d29a7b41a9bf114ac103d4c649: No bootstrap required, opened a new log
I20260709 12:58:03.891134   760 ts_tablet_manager.cc:1403] T 2e14183cc6064b1cbe6530eb1c8c899c P 1bcd68d29a7b41a9bf114ac103d4c649: Time spent bootstrapping tablet: real 0.145s	user 0.016s	sys 0.019s
I20260709 12:58:03.903160   760 raft_consensus.cc:359] T 2e14183cc6064b1cbe6530eb1c8c899c P 1bcd68d29a7b41a9bf114ac103d4c649 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } }
I20260709 12:58:03.906509   760 raft_consensus.cc:385] T 2e14183cc6064b1cbe6530eb1c8c899c P 1bcd68d29a7b41a9bf114ac103d4c649 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:03.907027   760 raft_consensus.cc:740] T 2e14183cc6064b1cbe6530eb1c8c899c P 1bcd68d29a7b41a9bf114ac103d4c649 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bcd68d29a7b41a9bf114ac103d4c649, State: Initialized, Role: FOLLOWER
I20260709 12:58:03.908990   761 tablet_bootstrap.cc:654] T d8b33c9315094794a876a4b62755efaf P b5e46fc5aac9426a945bb34207d634d4: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:03.912549   759 tablet_bootstrap.cc:492] T d4d26ba9fb084e748bddb93f7547621f P 263986885d354f7ba08739e419a6cded: No bootstrap required, opened a new log
I20260709 12:58:03.913254   759 ts_tablet_manager.cc:1403] T d4d26ba9fb084e748bddb93f7547621f P 263986885d354f7ba08739e419a6cded: Time spent bootstrapping tablet: real 0.083s	user 0.014s	sys 0.015s
I20260709 12:58:03.908325   760 consensus_queue.cc:260] T 2e14183cc6064b1cbe6530eb1c8c899c P 1bcd68d29a7b41a9bf114ac103d4c649 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } }
I20260709 12:58:03.917426   760 ts_tablet_manager.cc:1434] T 2e14183cc6064b1cbe6530eb1c8c899c P 1bcd68d29a7b41a9bf114ac103d4c649: Time spent starting tablet: real 0.026s	user 0.013s	sys 0.000s
I20260709 12:58:03.922040   760 tablet_bootstrap.cc:492] T d8b33c9315094794a876a4b62755efaf P 1bcd68d29a7b41a9bf114ac103d4c649: Bootstrap starting.
I20260709 12:58:03.930230   774 raft_consensus.cc:493] T 59112618a86046e1af899ce6d3c85203 P b5e46fc5aac9426a945bb34207d634d4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:03.922618   759 raft_consensus.cc:359] T d4d26ba9fb084e748bddb93f7547621f P 263986885d354f7ba08739e419a6cded [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } }
I20260709 12:58:03.931957   759 raft_consensus.cc:385] T d4d26ba9fb084e748bddb93f7547621f P 263986885d354f7ba08739e419a6cded [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:03.931380   774 raft_consensus.cc:515] T 59112618a86046e1af899ce6d3c85203 P b5e46fc5aac9426a945bb34207d634d4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } }
I20260709 12:58:03.932384   759 raft_consensus.cc:740] T d4d26ba9fb084e748bddb93f7547621f P 263986885d354f7ba08739e419a6cded [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 263986885d354f7ba08739e419a6cded, State: Initialized, Role: FOLLOWER
I20260709 12:58:03.933561   759 consensus_queue.cc:260] T d4d26ba9fb084e748bddb93f7547621f P 263986885d354f7ba08739e419a6cded [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } }
I20260709 12:58:03.935803   640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59112618a86046e1af899ce6d3c85203" candidate_uuid: "b5e46fc5aac9426a945bb34207d634d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "263986885d354f7ba08739e419a6cded" is_pre_election: true
I20260709 12:58:03.936039   774 leader_election.cc:290] T 59112618a86046e1af899ce6d3c85203 P b5e46fc5aac9426a945bb34207d634d4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 263986885d354f7ba08739e419a6cded (127.0.107.194:45491), 1bcd68d29a7b41a9bf114ac103d4c649 (127.0.107.195:46365)
I20260709 12:58:03.939028   714 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59112618a86046e1af899ce6d3c85203" candidate_uuid: "b5e46fc5aac9426a945bb34207d634d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" is_pre_election: true
I20260709 12:58:03.940026   714 raft_consensus.cc:2468] T 59112618a86046e1af899ce6d3c85203 P 1bcd68d29a7b41a9bf114ac103d4c649 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5e46fc5aac9426a945bb34207d634d4 in term 0.
I20260709 12:58:03.940464   759 ts_tablet_manager.cc:1434] T d4d26ba9fb084e748bddb93f7547621f P 263986885d354f7ba08739e419a6cded: Time spent starting tablet: real 0.027s	user 0.009s	sys 0.004s
I20260709 12:58:03.941920   759 tablet_bootstrap.cc:492] T 59112618a86046e1af899ce6d3c85203 P 263986885d354f7ba08739e419a6cded: Bootstrap starting.
W20260709 12:58:03.942126   531 leader_election.cc:343] T 59112618a86046e1af899ce6d3c85203 P b5e46fc5aac9426a945bb34207d634d4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 263986885d354f7ba08739e419a6cded (127.0.107.194:45491): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:03.943002   760 tablet_bootstrap.cc:654] T d8b33c9315094794a876a4b62755efaf P 1bcd68d29a7b41a9bf114ac103d4c649: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:03.946892   529 leader_election.cc:304] T 59112618a86046e1af899ce6d3c85203 P b5e46fc5aac9426a945bb34207d634d4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1bcd68d29a7b41a9bf114ac103d4c649, b5e46fc5aac9426a945bb34207d634d4; no voters: 263986885d354f7ba08739e419a6cded
I20260709 12:58:03.947891   774 raft_consensus.cc:2804] T 59112618a86046e1af899ce6d3c85203 P b5e46fc5aac9426a945bb34207d634d4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:58:03.948453   774 raft_consensus.cc:493] T 59112618a86046e1af899ce6d3c85203 P b5e46fc5aac9426a945bb34207d634d4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:58:03.949076   774 raft_consensus.cc:3060] T 59112618a86046e1af899ce6d3c85203 P b5e46fc5aac9426a945bb34207d634d4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:03.952019   461 catalog_manager.cc:5697] T 793bafcf38ca4ffb9a9aac9374ea2094 P 1bcd68d29a7b41a9bf114ac103d4c649 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1bcd68d29a7b41a9bf114ac103d4c649 (127.0.107.195). New cstate: current_term: 1 leader_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } health_report { overall_health: UNKNOWN } } }
I20260709 12:58:03.954664   461 catalog_manager.cc:5697] T e6aabcfcdeb34e6c94ec0bb089deabda P 1bcd68d29a7b41a9bf114ac103d4c649 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1bcd68d29a7b41a9bf114ac103d4c649 (127.0.107.195). New cstate: current_term: 1 leader_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } health_report { overall_health: HEALTHY } } }
I20260709 12:58:03.968065   761 tablet_bootstrap.cc:492] T d8b33c9315094794a876a4b62755efaf P b5e46fc5aac9426a945bb34207d634d4: No bootstrap required, opened a new log
I20260709 12:58:03.968909   761 ts_tablet_manager.cc:1403] T d8b33c9315094794a876a4b62755efaf P b5e46fc5aac9426a945bb34207d634d4: Time spent bootstrapping tablet: real 0.086s	user 0.006s	sys 0.019s
I20260709 12:58:03.975680   761 raft_consensus.cc:359] T d8b33c9315094794a876a4b62755efaf P b5e46fc5aac9426a945bb34207d634d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } }
I20260709 12:58:03.977916   761 raft_consensus.cc:385] T d8b33c9315094794a876a4b62755efaf P b5e46fc5aac9426a945bb34207d634d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:03.978374   761 raft_consensus.cc:740] T d8b33c9315094794a876a4b62755efaf P b5e46fc5aac9426a945bb34207d634d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5e46fc5aac9426a945bb34207d634d4, State: Initialized, Role: FOLLOWER
I20260709 12:58:03.978230   774 raft_consensus.cc:515] T 59112618a86046e1af899ce6d3c85203 P b5e46fc5aac9426a945bb34207d634d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } }
I20260709 12:58:03.979687   761 consensus_queue.cc:260] T d8b33c9315094794a876a4b62755efaf P b5e46fc5aac9426a945bb34207d634d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } }
I20260709 12:58:03.982401   774 leader_election.cc:290] T 59112618a86046e1af899ce6d3c85203 P b5e46fc5aac9426a945bb34207d634d4 [CANDIDATE]: Term 1 election: Requested vote from peers 263986885d354f7ba08739e419a6cded (127.0.107.194:45491), 1bcd68d29a7b41a9bf114ac103d4c649 (127.0.107.195:46365)
I20260709 12:58:03.982945   640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59112618a86046e1af899ce6d3c85203" candidate_uuid: "b5e46fc5aac9426a945bb34207d634d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "263986885d354f7ba08739e419a6cded"
W20260709 12:58:03.985198   531 leader_election.cc:343] T 59112618a86046e1af899ce6d3c85203 P b5e46fc5aac9426a945bb34207d634d4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 263986885d354f7ba08739e419a6cded (127.0.107.194:45491): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:03.986613   714 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59112618a86046e1af899ce6d3c85203" candidate_uuid: "b5e46fc5aac9426a945bb34207d634d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1bcd68d29a7b41a9bf114ac103d4c649"
I20260709 12:58:03.987325   714 raft_consensus.cc:3060] T 59112618a86046e1af899ce6d3c85203 P 1bcd68d29a7b41a9bf114ac103d4c649 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:03.992784   759 tablet_bootstrap.cc:654] T 59112618a86046e1af899ce6d3c85203 P 263986885d354f7ba08739e419a6cded: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:03.996829   761 ts_tablet_manager.cc:1434] T d8b33c9315094794a876a4b62755efaf P b5e46fc5aac9426a945bb34207d634d4: Time spent starting tablet: real 0.027s	user 0.011s	sys 0.008s
I20260709 12:58:03.997170   760 tablet_bootstrap.cc:492] T d8b33c9315094794a876a4b62755efaf P 1bcd68d29a7b41a9bf114ac103d4c649: No bootstrap required, opened a new log
I20260709 12:58:03.998242   760 ts_tablet_manager.cc:1403] T d8b33c9315094794a876a4b62755efaf P 1bcd68d29a7b41a9bf114ac103d4c649: Time spent bootstrapping tablet: real 0.077s	user 0.010s	sys 0.018s
I20260709 12:58:03.999303   761 tablet_bootstrap.cc:492] T e6aabcfcdeb34e6c94ec0bb089deabda P b5e46fc5aac9426a945bb34207d634d4: Bootstrap starting.
I20260709 12:58:04.002991   760 raft_consensus.cc:359] T d8b33c9315094794a876a4b62755efaf P 1bcd68d29a7b41a9bf114ac103d4c649 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } }
I20260709 12:58:04.003961   760 raft_consensus.cc:385] T d8b33c9315094794a876a4b62755efaf P 1bcd68d29a7b41a9bf114ac103d4c649 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:04.004410   760 raft_consensus.cc:740] T d8b33c9315094794a876a4b62755efaf P 1bcd68d29a7b41a9bf114ac103d4c649 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bcd68d29a7b41a9bf114ac103d4c649, State: Initialized, Role: FOLLOWER
I20260709 12:58:04.005779   714 raft_consensus.cc:2468] T 59112618a86046e1af899ce6d3c85203 P 1bcd68d29a7b41a9bf114ac103d4c649 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5e46fc5aac9426a945bb34207d634d4 in term 1.
I20260709 12:58:04.008932   529 leader_election.cc:304] T 59112618a86046e1af899ce6d3c85203 P b5e46fc5aac9426a945bb34207d634d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1bcd68d29a7b41a9bf114ac103d4c649, b5e46fc5aac9426a945bb34207d634d4; no voters: 263986885d354f7ba08739e419a6cded
I20260709 12:58:04.013345   802 raft_consensus.cc:2804] T 59112618a86046e1af899ce6d3c85203 P b5e46fc5aac9426a945bb34207d634d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:58:04.008658   760 consensus_queue.cc:260] T d8b33c9315094794a876a4b62755efaf P 1bcd68d29a7b41a9bf114ac103d4c649 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } }
I20260709 12:58:04.016122   802 raft_consensus.cc:697] T 59112618a86046e1af899ce6d3c85203 P b5e46fc5aac9426a945bb34207d634d4 [term 1 LEADER]: Becoming Leader. State: Replica: b5e46fc5aac9426a945bb34207d634d4, State: Running, Role: LEADER
I20260709 12:58:04.021255   761 tablet_bootstrap.cc:654] T e6aabcfcdeb34e6c94ec0bb089deabda P b5e46fc5aac9426a945bb34207d634d4: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:04.020117   802 consensus_queue.cc:237] T 59112618a86046e1af899ce6d3c85203 P b5e46fc5aac9426a945bb34207d634d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } }
I20260709 12:58:04.026157   759 tablet_bootstrap.cc:492] T 59112618a86046e1af899ce6d3c85203 P 263986885d354f7ba08739e419a6cded: No bootstrap required, opened a new log
I20260709 12:58:04.026801   759 ts_tablet_manager.cc:1403] T 59112618a86046e1af899ce6d3c85203 P 263986885d354f7ba08739e419a6cded: Time spent bootstrapping tablet: real 0.085s	user 0.020s	sys 0.016s
W20260709 12:58:04.029186   681 consensus_peers.cc:597] T d441098f471f4b0084938676c20981e5 P 1bcd68d29a7b41a9bf114ac103d4c649 -> Peer b5e46fc5aac9426a945bb34207d634d4 (127.0.107.193:38425): Couldn't send request to peer b5e46fc5aac9426a945bb34207d634d4. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:58:04.032691   760 ts_tablet_manager.cc:1434] T d8b33c9315094794a876a4b62755efaf P 1bcd68d29a7b41a9bf114ac103d4c649: Time spent starting tablet: real 0.034s	user 0.010s	sys 0.000s
I20260709 12:58:04.034401   760 tablet_bootstrap.cc:492] T dd06089bfde94412b12705a116ecf3d5 P 1bcd68d29a7b41a9bf114ac103d4c649: Bootstrap starting.
I20260709 12:58:04.036610   759 raft_consensus.cc:359] T 59112618a86046e1af899ce6d3c85203 P 263986885d354f7ba08739e419a6cded [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } }
I20260709 12:58:04.037595   759 raft_consensus.cc:385] T 59112618a86046e1af899ce6d3c85203 P 263986885d354f7ba08739e419a6cded [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:04.044525   759 raft_consensus.cc:740] T 59112618a86046e1af899ce6d3c85203 P 263986885d354f7ba08739e419a6cded [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 263986885d354f7ba08739e419a6cded, State: Initialized, Role: FOLLOWER
I20260709 12:58:04.046576   759 consensus_queue.cc:260] T 59112618a86046e1af899ce6d3c85203 P 263986885d354f7ba08739e419a6cded [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } }
I20260709 12:58:04.049823   759 ts_tablet_manager.cc:1434] T 59112618a86046e1af899ce6d3c85203 P 263986885d354f7ba08739e419a6cded: Time spent starting tablet: real 0.023s	user 0.009s	sys 0.000s
I20260709 12:58:04.055948   760 tablet_bootstrap.cc:654] T dd06089bfde94412b12705a116ecf3d5 P 1bcd68d29a7b41a9bf114ac103d4c649: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:04.059489   759 tablet_bootstrap.cc:492] T 61c59a4c19754dedb9d385e2ecfe145b P 263986885d354f7ba08739e419a6cded: Bootstrap starting.
I20260709 12:58:04.062613   761 tablet_bootstrap.cc:492] T e6aabcfcdeb34e6c94ec0bb089deabda P b5e46fc5aac9426a945bb34207d634d4: No bootstrap required, opened a new log
I20260709 12:58:04.064209   761 ts_tablet_manager.cc:1403] T e6aabcfcdeb34e6c94ec0bb089deabda P b5e46fc5aac9426a945bb34207d634d4: Time spent bootstrapping tablet: real 0.066s	user 0.009s	sys 0.031s
I20260709 12:58:04.071015   761 raft_consensus.cc:359] T e6aabcfcdeb34e6c94ec0bb089deabda P b5e46fc5aac9426a945bb34207d634d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } }
I20260709 12:58:04.075395   761 raft_consensus.cc:385] T e6aabcfcdeb34e6c94ec0bb089deabda P b5e46fc5aac9426a945bb34207d634d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:04.075749   761 raft_consensus.cc:740] T e6aabcfcdeb34e6c94ec0bb089deabda P b5e46fc5aac9426a945bb34207d634d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5e46fc5aac9426a945bb34207d634d4, State: Initialized, Role: FOLLOWER
I20260709 12:58:04.068063   463 catalog_manager.cc:5697] T 59112618a86046e1af899ce6d3c85203 P b5e46fc5aac9426a945bb34207d634d4 reported cstate change: term changed from 0 to 1, leader changed from <none> to b5e46fc5aac9426a945bb34207d634d4 (127.0.107.193). New cstate: current_term: 1 leader_uuid: "b5e46fc5aac9426a945bb34207d634d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } health_report { overall_health: UNKNOWN } } }
I20260709 12:58:04.076900   761 consensus_queue.cc:260] T e6aabcfcdeb34e6c94ec0bb089deabda P b5e46fc5aac9426a945bb34207d634d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } }
I20260709 12:58:04.079742   761 ts_tablet_manager.cc:1434] T e6aabcfcdeb34e6c94ec0bb089deabda P b5e46fc5aac9426a945bb34207d634d4: Time spent starting tablet: real 0.015s	user 0.009s	sys 0.004s
I20260709 12:58:04.081030   761 tablet_bootstrap.cc:492] T 2e14183cc6064b1cbe6530eb1c8c899c P b5e46fc5aac9426a945bb34207d634d4: Bootstrap starting.
I20260709 12:58:04.083909   759 tablet_bootstrap.cc:654] T 61c59a4c19754dedb9d385e2ecfe145b P 263986885d354f7ba08739e419a6cded: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:04.103287   760 tablet_bootstrap.cc:492] T dd06089bfde94412b12705a116ecf3d5 P 1bcd68d29a7b41a9bf114ac103d4c649: No bootstrap required, opened a new log
I20260709 12:58:04.104041   760 ts_tablet_manager.cc:1403] T dd06089bfde94412b12705a116ecf3d5 P 1bcd68d29a7b41a9bf114ac103d4c649: Time spent bootstrapping tablet: real 0.070s	user 0.013s	sys 0.022s
I20260709 12:58:04.107872   759 tablet_bootstrap.cc:492] T 61c59a4c19754dedb9d385e2ecfe145b P 263986885d354f7ba08739e419a6cded: No bootstrap required, opened a new log
I20260709 12:58:04.108553   759 ts_tablet_manager.cc:1403] T 61c59a4c19754dedb9d385e2ecfe145b P 263986885d354f7ba08739e419a6cded: Time spent bootstrapping tablet: real 0.049s	user 0.019s	sys 0.015s
I20260709 12:58:04.111490   761 tablet_bootstrap.cc:654] T 2e14183cc6064b1cbe6530eb1c8c899c P b5e46fc5aac9426a945bb34207d634d4: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:04.112928   759 raft_consensus.cc:359] T 61c59a4c19754dedb9d385e2ecfe145b P 263986885d354f7ba08739e419a6cded [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } }
I20260709 12:58:04.113844   759 raft_consensus.cc:385] T 61c59a4c19754dedb9d385e2ecfe145b P 263986885d354f7ba08739e419a6cded [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:04.114239   759 raft_consensus.cc:740] T 61c59a4c19754dedb9d385e2ecfe145b P 263986885d354f7ba08739e419a6cded [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 263986885d354f7ba08739e419a6cded, State: Initialized, Role: FOLLOWER
I20260709 12:58:04.114003   760 raft_consensus.cc:359] T dd06089bfde94412b12705a116ecf3d5 P 1bcd68d29a7b41a9bf114ac103d4c649 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } }
I20260709 12:58:04.115054   760 raft_consensus.cc:385] T dd06089bfde94412b12705a116ecf3d5 P 1bcd68d29a7b41a9bf114ac103d4c649 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:04.115478   760 raft_consensus.cc:740] T dd06089bfde94412b12705a116ecf3d5 P 1bcd68d29a7b41a9bf114ac103d4c649 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bcd68d29a7b41a9bf114ac103d4c649, State: Initialized, Role: FOLLOWER
I20260709 12:58:04.115315   759 consensus_queue.cc:260] T 61c59a4c19754dedb9d385e2ecfe145b P 263986885d354f7ba08739e419a6cded [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } }
I20260709 12:58:04.116524   760 consensus_queue.cc:260] T dd06089bfde94412b12705a116ecf3d5 P 1bcd68d29a7b41a9bf114ac103d4c649 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } }
I20260709 12:58:04.122679   760 ts_tablet_manager.cc:1434] T dd06089bfde94412b12705a116ecf3d5 P 1bcd68d29a7b41a9bf114ac103d4c649: Time spent starting tablet: real 0.018s	user 0.011s	sys 0.005s
I20260709 12:58:04.124955   759 ts_tablet_manager.cc:1434] T 61c59a4c19754dedb9d385e2ecfe145b P 263986885d354f7ba08739e419a6cded: Time spent starting tablet: real 0.016s	user 0.008s	sys 0.000s
I20260709 12:58:04.125965   759 tablet_bootstrap.cc:492] T dd06089bfde94412b12705a116ecf3d5 P 263986885d354f7ba08739e419a6cded: Bootstrap starting.
I20260709 12:58:04.135079   792 consensus_queue.cc:1048] T d441098f471f4b0084938676c20981e5 P 1bcd68d29a7b41a9bf114ac103d4c649 [LEADER]: Connected to new peer: Peer: permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:58:04.138710   761 tablet_bootstrap.cc:492] T 2e14183cc6064b1cbe6530eb1c8c899c P b5e46fc5aac9426a945bb34207d634d4: No bootstrap required, opened a new log
I20260709 12:58:04.139626   761 ts_tablet_manager.cc:1403] T 2e14183cc6064b1cbe6530eb1c8c899c P b5e46fc5aac9426a945bb34207d634d4: Time spent bootstrapping tablet: real 0.059s	user 0.011s	sys 0.017s
I20260709 12:58:04.139580   802 raft_consensus.cc:493] T d8b33c9315094794a876a4b62755efaf P b5e46fc5aac9426a945bb34207d634d4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:04.140266   802 raft_consensus.cc:515] T d8b33c9315094794a876a4b62755efaf P b5e46fc5aac9426a945bb34207d634d4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } }
I20260709 12:58:04.142730   774 raft_consensus.cc:493] T d4d26ba9fb084e748bddb93f7547621f P b5e46fc5aac9426a945bb34207d634d4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:04.129014   760 tablet_bootstrap.cc:492] T 61c59a4c19754dedb9d385e2ecfe145b P 1bcd68d29a7b41a9bf114ac103d4c649: Bootstrap starting.
I20260709 12:58:04.144273   774 raft_consensus.cc:515] T d4d26ba9fb084e748bddb93f7547621f P b5e46fc5aac9426a945bb34207d634d4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } }
I20260709 12:58:04.146394   641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d8b33c9315094794a876a4b62755efaf" candidate_uuid: "b5e46fc5aac9426a945bb34207d634d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "263986885d354f7ba08739e419a6cded" is_pre_election: true
I20260709 12:58:04.147204   641 raft_consensus.cc:2468] T d8b33c9315094794a876a4b62755efaf P 263986885d354f7ba08739e419a6cded [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5e46fc5aac9426a945bb34207d634d4 in term 0.
I20260709 12:58:04.148062   774 leader_election.cc:290] T d4d26ba9fb084e748bddb93f7547621f P b5e46fc5aac9426a945bb34207d634d4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 263986885d354f7ba08739e419a6cded (127.0.107.194:45491), 1bcd68d29a7b41a9bf114ac103d4c649 (127.0.107.195:46365)
I20260709 12:58:04.148136   802 leader_election.cc:290] T d8b33c9315094794a876a4b62755efaf P b5e46fc5aac9426a945bb34207d634d4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 263986885d354f7ba08739e419a6cded (127.0.107.194:45491), 1bcd68d29a7b41a9bf114ac103d4c649 (127.0.107.195:46365)
I20260709 12:58:04.149139   531 leader_election.cc:304] T d8b33c9315094794a876a4b62755efaf P b5e46fc5aac9426a945bb34207d634d4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 263986885d354f7ba08739e419a6cded, b5e46fc5aac9426a945bb34207d634d4; no voters: 
I20260709 12:58:04.150144   802 raft_consensus.cc:2804] T d8b33c9315094794a876a4b62755efaf P b5e46fc5aac9426a945bb34207d634d4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:58:04.150501   641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d26ba9fb084e748bddb93f7547621f" candidate_uuid: "b5e46fc5aac9426a945bb34207d634d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "263986885d354f7ba08739e419a6cded" is_pre_election: true
I20260709 12:58:04.150731   802 raft_consensus.cc:493] T d8b33c9315094794a876a4b62755efaf P b5e46fc5aac9426a945bb34207d634d4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:58:04.150862   714 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d8b33c9315094794a876a4b62755efaf" candidate_uuid: "b5e46fc5aac9426a945bb34207d634d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" is_pre_election: true
I20260709 12:58:04.151243   641 raft_consensus.cc:2468] T d4d26ba9fb084e748bddb93f7547621f P 263986885d354f7ba08739e419a6cded [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5e46fc5aac9426a945bb34207d634d4 in term 0.
I20260709 12:58:04.151408   802 raft_consensus.cc:3060] T d8b33c9315094794a876a4b62755efaf P b5e46fc5aac9426a945bb34207d634d4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:04.151693   714 raft_consensus.cc:2468] T d8b33c9315094794a876a4b62755efaf P 1bcd68d29a7b41a9bf114ac103d4c649 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5e46fc5aac9426a945bb34207d634d4 in term 0.
I20260709 12:58:04.152688   531 leader_election.cc:304] T d4d26ba9fb084e748bddb93f7547621f P b5e46fc5aac9426a945bb34207d634d4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 263986885d354f7ba08739e419a6cded, b5e46fc5aac9426a945bb34207d634d4; no voters: 
I20260709 12:58:04.153199   715 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d26ba9fb084e748bddb93f7547621f" candidate_uuid: "b5e46fc5aac9426a945bb34207d634d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" is_pre_election: true
I20260709 12:58:04.153669   774 raft_consensus.cc:2804] T d4d26ba9fb084e748bddb93f7547621f P b5e46fc5aac9426a945bb34207d634d4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:58:04.154062   774 raft_consensus.cc:493] T d4d26ba9fb084e748bddb93f7547621f P b5e46fc5aac9426a945bb34207d634d4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:58:04.154484   774 raft_consensus.cc:3060] T d4d26ba9fb084e748bddb93f7547621f P b5e46fc5aac9426a945bb34207d634d4 [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:58:04.155853   529 leader_election.cc:343] T d4d26ba9fb084e748bddb93f7547621f P b5e46fc5aac9426a945bb34207d634d4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1bcd68d29a7b41a9bf114ac103d4c649 (127.0.107.195:46365): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:04.155846   761 raft_consensus.cc:359] T 2e14183cc6064b1cbe6530eb1c8c899c P b5e46fc5aac9426a945bb34207d634d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } }
I20260709 12:58:04.159453   761 raft_consensus.cc:385] T 2e14183cc6064b1cbe6530eb1c8c899c P b5e46fc5aac9426a945bb34207d634d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:04.159889   761 raft_consensus.cc:740] T 2e14183cc6064b1cbe6530eb1c8c899c P b5e46fc5aac9426a945bb34207d634d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5e46fc5aac9426a945bb34207d634d4, State: Initialized, Role: FOLLOWER
I20260709 12:58:04.161090   761 consensus_queue.cc:260] T 2e14183cc6064b1cbe6530eb1c8c899c P b5e46fc5aac9426a945bb34207d634d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } }
I20260709 12:58:04.166229   759 tablet_bootstrap.cc:654] T dd06089bfde94412b12705a116ecf3d5 P 263986885d354f7ba08739e419a6cded: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:04.181849   774 raft_consensus.cc:515] T d4d26ba9fb084e748bddb93f7547621f P b5e46fc5aac9426a945bb34207d634d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } }
I20260709 12:58:04.187877   774 leader_election.cc:290] T d4d26ba9fb084e748bddb93f7547621f P b5e46fc5aac9426a945bb34207d634d4 [CANDIDATE]: Term 1 election: Requested vote from peers 263986885d354f7ba08739e419a6cded (127.0.107.194:45491), 1bcd68d29a7b41a9bf114ac103d4c649 (127.0.107.195:46365)
I20260709 12:58:04.187566   802 raft_consensus.cc:515] T d8b33c9315094794a876a4b62755efaf P b5e46fc5aac9426a945bb34207d634d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } }
I20260709 12:58:04.191290   761 ts_tablet_manager.cc:1434] T 2e14183cc6064b1cbe6530eb1c8c899c P b5e46fc5aac9426a945bb34207d634d4: Time spent starting tablet: real 0.051s	user 0.013s	sys 0.004s
I20260709 12:58:04.191545   715 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d26ba9fb084e748bddb93f7547621f" candidate_uuid: "b5e46fc5aac9426a945bb34207d634d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1bcd68d29a7b41a9bf114ac103d4c649"
I20260709 12:58:04.192886   760 tablet_bootstrap.cc:654] T 61c59a4c19754dedb9d385e2ecfe145b P 1bcd68d29a7b41a9bf114ac103d4c649: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:04.192844   761 tablet_bootstrap.cc:492] T d441098f471f4b0084938676c20981e5 P b5e46fc5aac9426a945bb34207d634d4: Bootstrap starting.
W20260709 12:58:04.194818   529 leader_election.cc:343] T d4d26ba9fb084e748bddb93f7547621f P b5e46fc5aac9426a945bb34207d634d4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1bcd68d29a7b41a9bf114ac103d4c649 (127.0.107.195:46365): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:58:04.203325   662 debug-util.cc:398] Leaking SignalData structure 0x7b0800125060 after lost signal to thread 434
W20260709 12:58:04.204550   662 debug-util.cc:398] Leaking SignalData structure 0x7b0800125740 after lost signal to thread 497
I20260709 12:58:04.205109   715 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d8b33c9315094794a876a4b62755efaf" candidate_uuid: "b5e46fc5aac9426a945bb34207d634d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1bcd68d29a7b41a9bf114ac103d4c649"
I20260709 12:58:04.206362   639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d8b33c9315094794a876a4b62755efaf" candidate_uuid: "b5e46fc5aac9426a945bb34207d634d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "263986885d354f7ba08739e419a6cded"
I20260709 12:58:04.206584   715 raft_consensus.cc:3060] T d8b33c9315094794a876a4b62755efaf P 1bcd68d29a7b41a9bf114ac103d4c649 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:04.207171   639 raft_consensus.cc:3060] T d8b33c9315094794a876a4b62755efaf P 263986885d354f7ba08739e419a6cded [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:04.213860   640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d26ba9fb084e748bddb93f7547621f" candidate_uuid: "b5e46fc5aac9426a945bb34207d634d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "263986885d354f7ba08739e419a6cded"
I20260709 12:58:04.217447   792 raft_consensus.cc:493] T 2e14183cc6064b1cbe6530eb1c8c899c P 1bcd68d29a7b41a9bf114ac103d4c649 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:04.217900   640 raft_consensus.cc:3060] T d4d26ba9fb084e748bddb93f7547621f P 263986885d354f7ba08739e419a6cded [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:04.218103   792 raft_consensus.cc:515] T 2e14183cc6064b1cbe6530eb1c8c899c P 1bcd68d29a7b41a9bf114ac103d4c649 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } }
I20260709 12:58:04.213011   802 leader_election.cc:290] T d8b33c9315094794a876a4b62755efaf P b5e46fc5aac9426a945bb34207d634d4 [CANDIDATE]: Term 1 election: Requested vote from peers 263986885d354f7ba08739e419a6cded (127.0.107.194:45491), 1bcd68d29a7b41a9bf114ac103d4c649 (127.0.107.195:46365)
I20260709 12:58:04.222564   565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e14183cc6064b1cbe6530eb1c8c899c" candidate_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5e46fc5aac9426a945bb34207d634d4" is_pre_election: true
I20260709 12:58:04.231336   638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e14183cc6064b1cbe6530eb1c8c899c" candidate_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "263986885d354f7ba08739e419a6cded" is_pre_election: true
I20260709 12:58:04.231702   565 raft_consensus.cc:2468] T 2e14183cc6064b1cbe6530eb1c8c899c P b5e46fc5aac9426a945bb34207d634d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bcd68d29a7b41a9bf114ac103d4c649 in term 0.
I20260709 12:58:04.232259   638 raft_consensus.cc:2468] T 2e14183cc6064b1cbe6530eb1c8c899c P 263986885d354f7ba08739e419a6cded [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bcd68d29a7b41a9bf114ac103d4c649 in term 0.
I20260709 12:58:04.233703   681 leader_election.cc:304] T 2e14183cc6064b1cbe6530eb1c8c899c P 1bcd68d29a7b41a9bf114ac103d4c649 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bcd68d29a7b41a9bf114ac103d4c649, b5e46fc5aac9426a945bb34207d634d4; no voters: 
I20260709 12:58:04.235195   782 raft_consensus.cc:2804] T 2e14183cc6064b1cbe6530eb1c8c899c P 1bcd68d29a7b41a9bf114ac103d4c649 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:58:04.235750   782 raft_consensus.cc:493] T 2e14183cc6064b1cbe6530eb1c8c899c P 1bcd68d29a7b41a9bf114ac103d4c649 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:58:04.236323   782 raft_consensus.cc:3060] T 2e14183cc6064b1cbe6530eb1c8c899c P 1bcd68d29a7b41a9bf114ac103d4c649 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:04.237454   715 raft_consensus.cc:2468] T d8b33c9315094794a876a4b62755efaf P 1bcd68d29a7b41a9bf114ac103d4c649 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5e46fc5aac9426a945bb34207d634d4 in term 1.
I20260709 12:58:04.241449   529 leader_election.cc:304] T d8b33c9315094794a876a4b62755efaf P b5e46fc5aac9426a945bb34207d634d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bcd68d29a7b41a9bf114ac103d4c649, b5e46fc5aac9426a945bb34207d634d4; no voters: 
I20260709 12:58:04.227718   792 leader_election.cc:290] T 2e14183cc6064b1cbe6530eb1c8c899c P 1bcd68d29a7b41a9bf114ac103d4c649 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b5e46fc5aac9426a945bb34207d634d4 (127.0.107.193:38425), 263986885d354f7ba08739e419a6cded (127.0.107.194:45491)
I20260709 12:58:04.243749   802 raft_consensus.cc:2804] T d8b33c9315094794a876a4b62755efaf P b5e46fc5aac9426a945bb34207d634d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:58:04.244519   802 raft_consensus.cc:697] T d8b33c9315094794a876a4b62755efaf P b5e46fc5aac9426a945bb34207d634d4 [term 1 LEADER]: Becoming Leader. State: Replica: b5e46fc5aac9426a945bb34207d634d4, State: Running, Role: LEADER
I20260709 12:58:04.246982   759 tablet_bootstrap.cc:492] T dd06089bfde94412b12705a116ecf3d5 P 263986885d354f7ba08739e419a6cded: No bootstrap required, opened a new log
I20260709 12:58:04.247606   759 ts_tablet_manager.cc:1403] T dd06089bfde94412b12705a116ecf3d5 P 263986885d354f7ba08739e419a6cded: Time spent bootstrapping tablet: real 0.122s	user 0.013s	sys 0.012s
I20260709 12:58:04.251579   761 tablet_bootstrap.cc:654] T d441098f471f4b0084938676c20981e5 P b5e46fc5aac9426a945bb34207d634d4: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:04.251199   802 consensus_queue.cc:237] T d8b33c9315094794a876a4b62755efaf P b5e46fc5aac9426a945bb34207d634d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } }
I20260709 12:58:04.251937   640 raft_consensus.cc:2468] T d4d26ba9fb084e748bddb93f7547621f P 263986885d354f7ba08739e419a6cded [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5e46fc5aac9426a945bb34207d634d4 in term 1.
I20260709 12:58:04.242497   639 raft_consensus.cc:2468] T d8b33c9315094794a876a4b62755efaf P 263986885d354f7ba08739e419a6cded [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5e46fc5aac9426a945bb34207d634d4 in term 1.
I20260709 12:58:04.254312   531 leader_election.cc:304] T d4d26ba9fb084e748bddb93f7547621f P b5e46fc5aac9426a945bb34207d634d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 263986885d354f7ba08739e419a6cded, b5e46fc5aac9426a945bb34207d634d4; no voters: 1bcd68d29a7b41a9bf114ac103d4c649
I20260709 12:58:04.256153   759 raft_consensus.cc:359] T dd06089bfde94412b12705a116ecf3d5 P 263986885d354f7ba08739e419a6cded [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } }
I20260709 12:58:04.260118   565 raft_consensus.cc:3060] T e6aabcfcdeb34e6c94ec0bb089deabda P b5e46fc5aac9426a945bb34207d634d4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:04.263145   759 raft_consensus.cc:385] T dd06089bfde94412b12705a116ecf3d5 P 263986885d354f7ba08739e419a6cded [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:04.263509   759 raft_consensus.cc:740] T dd06089bfde94412b12705a116ecf3d5 P 263986885d354f7ba08739e419a6cded [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 263986885d354f7ba08739e419a6cded, State: Initialized, Role: FOLLOWER
I20260709 12:58:04.264690   774 raft_consensus.cc:2804] T d4d26ba9fb084e748bddb93f7547621f P b5e46fc5aac9426a945bb34207d634d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:58:04.264350   759 consensus_queue.cc:260] T dd06089bfde94412b12705a116ecf3d5 P 263986885d354f7ba08739e419a6cded [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } }
I20260709 12:58:04.265309   774 raft_consensus.cc:697] T d4d26ba9fb084e748bddb93f7547621f P b5e46fc5aac9426a945bb34207d634d4 [term 1 LEADER]: Becoming Leader. State: Replica: b5e46fc5aac9426a945bb34207d634d4, State: Running, Role: LEADER
I20260709 12:58:04.262506   640 raft_consensus.cc:3060] T 793bafcf38ca4ffb9a9aac9374ea2094 P 263986885d354f7ba08739e419a6cded [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:04.272998   759 ts_tablet_manager.cc:1434] T dd06089bfde94412b12705a116ecf3d5 P 263986885d354f7ba08739e419a6cded: Time spent starting tablet: real 0.025s	user 0.004s	sys 0.004s
I20260709 12:58:04.274326   774 consensus_queue.cc:237] T d4d26ba9fb084e748bddb93f7547621f P b5e46fc5aac9426a945bb34207d634d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } }
I20260709 12:58:04.277132   782 raft_consensus.cc:515] T 2e14183cc6064b1cbe6530eb1c8c899c P 1bcd68d29a7b41a9bf114ac103d4c649 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } }
I20260709 12:58:04.286242   639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e14183cc6064b1cbe6530eb1c8c899c" candidate_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "263986885d354f7ba08739e419a6cded"
I20260709 12:58:04.287093   639 raft_consensus.cc:3060] T 2e14183cc6064b1cbe6530eb1c8c899c P 263986885d354f7ba08739e419a6cded [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:04.295143   564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e14183cc6064b1cbe6530eb1c8c899c" candidate_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5e46fc5aac9426a945bb34207d634d4"
I20260709 12:58:04.296110   564 raft_consensus.cc:3060] T 2e14183cc6064b1cbe6530eb1c8c899c P b5e46fc5aac9426a945bb34207d634d4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:04.314096   782 leader_election.cc:290] T 2e14183cc6064b1cbe6530eb1c8c899c P 1bcd68d29a7b41a9bf114ac103d4c649 [CANDIDATE]: Term 1 election: Requested vote from peers b5e46fc5aac9426a945bb34207d634d4 (127.0.107.193:38425), 263986885d354f7ba08739e419a6cded (127.0.107.194:45491)
I20260709 12:58:04.319651   639 raft_consensus.cc:2468] T 2e14183cc6064b1cbe6530eb1c8c899c P 263986885d354f7ba08739e419a6cded [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1bcd68d29a7b41a9bf114ac103d4c649 in term 1.
I20260709 12:58:04.321237   680 leader_election.cc:304] T 2e14183cc6064b1cbe6530eb1c8c899c P 1bcd68d29a7b41a9bf114ac103d4c649 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bcd68d29a7b41a9bf114ac103d4c649, 263986885d354f7ba08739e419a6cded; no voters: 
I20260709 12:58:04.322427   782 raft_consensus.cc:2804] T 2e14183cc6064b1cbe6530eb1c8c899c P 1bcd68d29a7b41a9bf114ac103d4c649 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:58:04.324609   782 raft_consensus.cc:697] T 2e14183cc6064b1cbe6530eb1c8c899c P 1bcd68d29a7b41a9bf114ac103d4c649 [term 1 LEADER]: Becoming Leader. State: Replica: 1bcd68d29a7b41a9bf114ac103d4c649, State: Running, Role: LEADER
I20260709 12:58:04.326176   782 consensus_queue.cc:237] T 2e14183cc6064b1cbe6530eb1c8c899c P 1bcd68d29a7b41a9bf114ac103d4c649 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } }
I20260709 12:58:04.334409   792 consensus_queue.cc:1048] T 793bafcf38ca4ffb9a9aac9374ea2094 P 1bcd68d29a7b41a9bf114ac103d4c649 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:58:04.339704   792 consensus_queue.cc:1048] T e6aabcfcdeb34e6c94ec0bb089deabda P 1bcd68d29a7b41a9bf114ac103d4c649 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:58:04.346276   564 raft_consensus.cc:2468] T 2e14183cc6064b1cbe6530eb1c8c899c P b5e46fc5aac9426a945bb34207d634d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1bcd68d29a7b41a9bf114ac103d4c649 in term 1.
I20260709 12:58:04.348608   761 tablet_bootstrap.cc:492] T d441098f471f4b0084938676c20981e5 P b5e46fc5aac9426a945bb34207d634d4: No bootstrap required, opened a new log
I20260709 12:58:04.349176   761 ts_tablet_manager.cc:1403] T d441098f471f4b0084938676c20981e5 P b5e46fc5aac9426a945bb34207d634d4: Time spent bootstrapping tablet: real 0.157s	user 0.011s	sys 0.015s
I20260709 12:58:04.350298   760 tablet_bootstrap.cc:492] T 61c59a4c19754dedb9d385e2ecfe145b P 1bcd68d29a7b41a9bf114ac103d4c649: No bootstrap required, opened a new log
I20260709 12:58:04.350826   760 ts_tablet_manager.cc:1403] T 61c59a4c19754dedb9d385e2ecfe145b P 1bcd68d29a7b41a9bf114ac103d4c649: Time spent bootstrapping tablet: real 0.224s	user 0.018s	sys 0.009s
I20260709 12:58:04.352581   765 raft_consensus.cc:493] T 61c59a4c19754dedb9d385e2ecfe145b P 263986885d354f7ba08739e419a6cded [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:04.353214   765 raft_consensus.cc:515] T 61c59a4c19754dedb9d385e2ecfe145b P 263986885d354f7ba08739e419a6cded [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } }
I20260709 12:58:04.357913   765 leader_election.cc:290] T 61c59a4c19754dedb9d385e2ecfe145b P 263986885d354f7ba08739e419a6cded [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b5e46fc5aac9426a945bb34207d634d4 (127.0.107.193:38425), 1bcd68d29a7b41a9bf114ac103d4c649 (127.0.107.195:46365)
I20260709 12:58:04.359319   565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "61c59a4c19754dedb9d385e2ecfe145b" candidate_uuid: "263986885d354f7ba08739e419a6cded" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5e46fc5aac9426a945bb34207d634d4" is_pre_election: true
I20260709 12:58:04.359606   715 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "61c59a4c19754dedb9d385e2ecfe145b" candidate_uuid: "263986885d354f7ba08739e419a6cded" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" is_pre_election: true
I20260709 12:58:04.360076   462 catalog_manager.cc:5697] T d8b33c9315094794a876a4b62755efaf P b5e46fc5aac9426a945bb34207d634d4 reported cstate change: term changed from 0 to 1, leader changed from <none> to b5e46fc5aac9426a945bb34207d634d4 (127.0.107.193). New cstate: current_term: 1 leader_uuid: "b5e46fc5aac9426a945bb34207d634d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } health_report { overall_health: UNKNOWN } } }
W20260709 12:58:04.362044   607 leader_election.cc:343] T 61c59a4c19754dedb9d385e2ecfe145b P 263986885d354f7ba08739e419a6cded [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b5e46fc5aac9426a945bb34207d634d4 (127.0.107.193:38425): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:04.362807   761 raft_consensus.cc:359] T d441098f471f4b0084938676c20981e5 P b5e46fc5aac9426a945bb34207d634d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } }
I20260709 12:58:04.368242   761 raft_consensus.cc:385] T d441098f471f4b0084938676c20981e5 P b5e46fc5aac9426a945bb34207d634d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:04.368610   761 raft_consensus.cc:740] T d441098f471f4b0084938676c20981e5 P b5e46fc5aac9426a945bb34207d634d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5e46fc5aac9426a945bb34207d634d4, State: Initialized, Role: FOLLOWER
I20260709 12:58:04.369606   761 consensus_queue.cc:260] T d441098f471f4b0084938676c20981e5 P b5e46fc5aac9426a945bb34207d634d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } }
I20260709 12:58:04.370883   792 raft_consensus.cc:493] T dd06089bfde94412b12705a116ecf3d5 P 1bcd68d29a7b41a9bf114ac103d4c649 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:04.372325   792 raft_consensus.cc:515] T dd06089bfde94412b12705a116ecf3d5 P 1bcd68d29a7b41a9bf114ac103d4c649 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } }
I20260709 12:58:04.376392   766 consensus_queue.cc:1048] T 793bafcf38ca4ffb9a9aac9374ea2094 P 1bcd68d29a7b41a9bf114ac103d4c649 [LEADER]: Connected to new peer: Peer: permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260709 12:58:04.378907   604 leader_election.cc:343] T 61c59a4c19754dedb9d385e2ecfe145b P 263986885d354f7ba08739e419a6cded [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1bcd68d29a7b41a9bf114ac103d4c649 (127.0.107.195:46365): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:04.379626   604 leader_election.cc:304] T 61c59a4c19754dedb9d385e2ecfe145b P 263986885d354f7ba08739e419a6cded [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 263986885d354f7ba08739e419a6cded; no voters: 1bcd68d29a7b41a9bf114ac103d4c649, b5e46fc5aac9426a945bb34207d634d4
I20260709 12:58:04.381466   765 raft_consensus.cc:2749] T 61c59a4c19754dedb9d385e2ecfe145b P 263986885d354f7ba08739e419a6cded [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:58:04.383800   761 ts_tablet_manager.cc:1434] T d441098f471f4b0084938676c20981e5 P b5e46fc5aac9426a945bb34207d634d4: Time spent starting tablet: real 0.034s	user 0.007s	sys 0.004s
I20260709 12:58:04.387416   792 leader_election.cc:290] T dd06089bfde94412b12705a116ecf3d5 P 1bcd68d29a7b41a9bf114ac103d4c649 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b5e46fc5aac9426a945bb34207d634d4 (127.0.107.193:38425), 263986885d354f7ba08739e419a6cded (127.0.107.194:45491)
I20260709 12:58:04.390854   565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd06089bfde94412b12705a116ecf3d5" candidate_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5e46fc5aac9426a945bb34207d634d4" is_pre_election: true
I20260709 12:58:04.391515   640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd06089bfde94412b12705a116ecf3d5" candidate_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "263986885d354f7ba08739e419a6cded" is_pre_election: true
I20260709 12:58:04.391767   565 raft_consensus.cc:2468] T dd06089bfde94412b12705a116ecf3d5 P b5e46fc5aac9426a945bb34207d634d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bcd68d29a7b41a9bf114ac103d4c649 in term 0.
I20260709 12:58:04.392356   640 raft_consensus.cc:2468] T dd06089bfde94412b12705a116ecf3d5 P 263986885d354f7ba08739e419a6cded [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bcd68d29a7b41a9bf114ac103d4c649 in term 0.
I20260709 12:58:04.393518   681 leader_election.cc:304] T dd06089bfde94412b12705a116ecf3d5 P 1bcd68d29a7b41a9bf114ac103d4c649 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bcd68d29a7b41a9bf114ac103d4c649, b5e46fc5aac9426a945bb34207d634d4; no voters: 
I20260709 12:58:04.395431   792 raft_consensus.cc:2804] T dd06089bfde94412b12705a116ecf3d5 P 1bcd68d29a7b41a9bf114ac103d4c649 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:58:04.396044   792 raft_consensus.cc:493] T dd06089bfde94412b12705a116ecf3d5 P 1bcd68d29a7b41a9bf114ac103d4c649 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:58:04.396577   792 raft_consensus.cc:3060] T dd06089bfde94412b12705a116ecf3d5 P 1bcd68d29a7b41a9bf114ac103d4c649 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:04.400195   765 raft_consensus.cc:493] T 59112618a86046e1af899ce6d3c85203 P 263986885d354f7ba08739e419a6cded [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:04.400643   760 raft_consensus.cc:359] T 61c59a4c19754dedb9d385e2ecfe145b P 1bcd68d29a7b41a9bf114ac103d4c649 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } }
I20260709 12:58:04.400820   765 raft_consensus.cc:515] T 59112618a86046e1af899ce6d3c85203 P 263986885d354f7ba08739e419a6cded [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } }
I20260709 12:58:04.401471   760 raft_consensus.cc:385] T 61c59a4c19754dedb9d385e2ecfe145b P 1bcd68d29a7b41a9bf114ac103d4c649 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:04.401831   760 raft_consensus.cc:740] T 61c59a4c19754dedb9d385e2ecfe145b P 1bcd68d29a7b41a9bf114ac103d4c649 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bcd68d29a7b41a9bf114ac103d4c649, State: Initialized, Role: FOLLOWER
I20260709 12:58:04.402873   760 consensus_queue.cc:260] T 61c59a4c19754dedb9d385e2ecfe145b P 1bcd68d29a7b41a9bf114ac103d4c649 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } }
I20260709 12:58:04.404183   765 leader_election.cc:290] T 59112618a86046e1af899ce6d3c85203 P 263986885d354f7ba08739e419a6cded [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b5e46fc5aac9426a945bb34207d634d4 (127.0.107.193:38425), 1bcd68d29a7b41a9bf114ac103d4c649 (127.0.107.195:46365)
I20260709 12:58:04.406028   715 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59112618a86046e1af899ce6d3c85203" candidate_uuid: "263986885d354f7ba08739e419a6cded" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" is_pre_election: true
I20260709 12:58:04.406720   565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59112618a86046e1af899ce6d3c85203" candidate_uuid: "263986885d354f7ba08739e419a6cded" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5e46fc5aac9426a945bb34207d634d4" is_pre_election: true
I20260709 12:58:04.407409   715 raft_consensus.cc:2393] T 59112618a86046e1af899ce6d3c85203 P 1bcd68d29a7b41a9bf114ac103d4c649 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 263986885d354f7ba08739e419a6cded in current term 1: Already voted for candidate b5e46fc5aac9426a945bb34207d634d4 in this term.
I20260709 12:58:04.410301   607 leader_election.cc:304] T 59112618a86046e1af899ce6d3c85203 P 263986885d354f7ba08739e419a6cded [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 263986885d354f7ba08739e419a6cded; no voters: 1bcd68d29a7b41a9bf114ac103d4c649, b5e46fc5aac9426a945bb34207d634d4
I20260709 12:58:04.411950   761 tablet_bootstrap.cc:492] T 61c59a4c19754dedb9d385e2ecfe145b P b5e46fc5aac9426a945bb34207d634d4: Bootstrap starting.
I20260709 12:58:04.422262   765 raft_consensus.cc:3060] T 59112618a86046e1af899ce6d3c85203 P 263986885d354f7ba08739e419a6cded [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:04.428776   760 ts_tablet_manager.cc:1434] T 61c59a4c19754dedb9d385e2ecfe145b P 1bcd68d29a7b41a9bf114ac103d4c649: Time spent starting tablet: real 0.078s	user 0.007s	sys 0.002s
I20260709 12:58:04.430068   760 tablet_bootstrap.cc:492] T d4d26ba9fb084e748bddb93f7547621f P 1bcd68d29a7b41a9bf114ac103d4c649: Bootstrap starting.
I20260709 12:58:04.436412   463 catalog_manager.cc:5697] T 2e14183cc6064b1cbe6530eb1c8c899c P 1bcd68d29a7b41a9bf114ac103d4c649 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1bcd68d29a7b41a9bf114ac103d4c649 (127.0.107.195). New cstate: current_term: 1 leader_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } health_report { overall_health: HEALTHY } } }
I20260709 12:58:04.457834   782 consensus_queue.cc:1048] T e6aabcfcdeb34e6c94ec0bb089deabda P 1bcd68d29a7b41a9bf114ac103d4c649 [LEADER]: Connected to new peer: Peer: permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:58:04.464210   765 raft_consensus.cc:2749] T 59112618a86046e1af899ce6d3c85203 P 263986885d354f7ba08739e419a6cded [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:58:04.492038   565 raft_consensus.cc:3060] T d441098f471f4b0084938676c20981e5 P b5e46fc5aac9426a945bb34207d634d4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:04.502631   463 catalog_manager.cc:5697] T d4d26ba9fb084e748bddb93f7547621f P b5e46fc5aac9426a945bb34207d634d4 reported cstate change: term changed from 0 to 1, leader changed from <none> to b5e46fc5aac9426a945bb34207d634d4 (127.0.107.193). New cstate: current_term: 1 leader_uuid: "b5e46fc5aac9426a945bb34207d634d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } health_report { overall_health: UNKNOWN } } }
I20260709 12:58:04.507511   761 tablet_bootstrap.cc:654] T 61c59a4c19754dedb9d385e2ecfe145b P b5e46fc5aac9426a945bb34207d634d4: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:04.531646   774 consensus_queue.cc:1048] T 59112618a86046e1af899ce6d3c85203 P b5e46fc5aac9426a945bb34207d634d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:58:04.510670   792 raft_consensus.cc:515] T dd06089bfde94412b12705a116ecf3d5 P 1bcd68d29a7b41a9bf114ac103d4c649 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } }
I20260709 12:58:04.546904   563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd06089bfde94412b12705a116ecf3d5" candidate_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5e46fc5aac9426a945bb34207d634d4"
I20260709 12:58:04.550755   563 raft_consensus.cc:3060] T dd06089bfde94412b12705a116ecf3d5 P b5e46fc5aac9426a945bb34207d634d4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:04.553756   641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd06089bfde94412b12705a116ecf3d5" candidate_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "263986885d354f7ba08739e419a6cded"
I20260709 12:58:04.554567   641 raft_consensus.cc:3060] T dd06089bfde94412b12705a116ecf3d5 P 263986885d354f7ba08739e419a6cded [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:04.552600   792 leader_election.cc:290] T dd06089bfde94412b12705a116ecf3d5 P 1bcd68d29a7b41a9bf114ac103d4c649 [CANDIDATE]: Term 1 election: Requested vote from peers b5e46fc5aac9426a945bb34207d634d4 (127.0.107.193:38425), 263986885d354f7ba08739e419a6cded (127.0.107.194:45491)
I20260709 12:58:04.565649   760 tablet_bootstrap.cc:654] T d4d26ba9fb084e748bddb93f7547621f P 1bcd68d29a7b41a9bf114ac103d4c649: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:04.590864   774 consensus_queue.cc:1048] T 59112618a86046e1af899ce6d3c85203 P b5e46fc5aac9426a945bb34207d634d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:58:04.601590   761 tablet_bootstrap.cc:492] T 61c59a4c19754dedb9d385e2ecfe145b P b5e46fc5aac9426a945bb34207d634d4: No bootstrap required, opened a new log
I20260709 12:58:04.602144   761 ts_tablet_manager.cc:1403] T 61c59a4c19754dedb9d385e2ecfe145b P b5e46fc5aac9426a945bb34207d634d4: Time spent bootstrapping tablet: real 0.191s	user 0.023s	sys 0.007s
I20260709 12:58:04.605991   641 raft_consensus.cc:2468] T dd06089bfde94412b12705a116ecf3d5 P 263986885d354f7ba08739e419a6cded [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1bcd68d29a7b41a9bf114ac103d4c649 in term 1.
I20260709 12:58:04.607971   680 leader_election.cc:304] T dd06089bfde94412b12705a116ecf3d5 P 1bcd68d29a7b41a9bf114ac103d4c649 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bcd68d29a7b41a9bf114ac103d4c649, 263986885d354f7ba08739e419a6cded; no voters: 
I20260709 12:58:04.609241   792 raft_consensus.cc:2804] T dd06089bfde94412b12705a116ecf3d5 P 1bcd68d29a7b41a9bf114ac103d4c649 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:58:04.610044   792 raft_consensus.cc:697] T dd06089bfde94412b12705a116ecf3d5 P 1bcd68d29a7b41a9bf114ac103d4c649 [term 1 LEADER]: Becoming Leader. State: Replica: 1bcd68d29a7b41a9bf114ac103d4c649, State: Running, Role: LEADER
I20260709 12:58:04.611370   761 raft_consensus.cc:359] T 61c59a4c19754dedb9d385e2ecfe145b P b5e46fc5aac9426a945bb34207d634d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } }
I20260709 12:58:04.611389   792 consensus_queue.cc:237] T dd06089bfde94412b12705a116ecf3d5 P 1bcd68d29a7b41a9bf114ac103d4c649 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } }
I20260709 12:58:04.614775   761 raft_consensus.cc:385] T 61c59a4c19754dedb9d385e2ecfe145b P b5e46fc5aac9426a945bb34207d634d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:04.618067   761 raft_consensus.cc:740] T 61c59a4c19754dedb9d385e2ecfe145b P b5e46fc5aac9426a945bb34207d634d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5e46fc5aac9426a945bb34207d634d4, State: Initialized, Role: FOLLOWER
I20260709 12:58:04.626013   761 consensus_queue.cc:260] T 61c59a4c19754dedb9d385e2ecfe145b P b5e46fc5aac9426a945bb34207d634d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } }
I20260709 12:58:04.642556   761 ts_tablet_manager.cc:1434] T 61c59a4c19754dedb9d385e2ecfe145b P b5e46fc5aac9426a945bb34207d634d4: Time spent starting tablet: real 0.040s	user 0.011s	sys 0.000s
I20260709 12:58:04.627349   563 raft_consensus.cc:2468] T dd06089bfde94412b12705a116ecf3d5 P b5e46fc5aac9426a945bb34207d634d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1bcd68d29a7b41a9bf114ac103d4c649 in term 1.
I20260709 12:58:04.707968   760 tablet_bootstrap.cc:492] T d4d26ba9fb084e748bddb93f7547621f P 1bcd68d29a7b41a9bf114ac103d4c649: No bootstrap required, opened a new log
I20260709 12:58:04.708921   760 ts_tablet_manager.cc:1403] T d4d26ba9fb084e748bddb93f7547621f P 1bcd68d29a7b41a9bf114ac103d4c649: Time spent bootstrapping tablet: real 0.279s	user 0.021s	sys 0.011s
I20260709 12:58:04.713027   760 raft_consensus.cc:359] T d4d26ba9fb084e748bddb93f7547621f P 1bcd68d29a7b41a9bf114ac103d4c649 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } }
I20260709 12:58:04.713737   760 raft_consensus.cc:385] T d4d26ba9fb084e748bddb93f7547621f P 1bcd68d29a7b41a9bf114ac103d4c649 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:04.713999   760 raft_consensus.cc:740] T d4d26ba9fb084e748bddb93f7547621f P 1bcd68d29a7b41a9bf114ac103d4c649 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bcd68d29a7b41a9bf114ac103d4c649, State: Initialized, Role: FOLLOWER
I20260709 12:58:04.714814   760 consensus_queue.cc:260] T d4d26ba9fb084e748bddb93f7547621f P 1bcd68d29a7b41a9bf114ac103d4c649 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } }
I20260709 12:58:04.722474   766 raft_consensus.cc:493] T 61c59a4c19754dedb9d385e2ecfe145b P 1bcd68d29a7b41a9bf114ac103d4c649 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:04.723217   766 raft_consensus.cc:515] T 61c59a4c19754dedb9d385e2ecfe145b P 1bcd68d29a7b41a9bf114ac103d4c649 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } }
W20260709 12:58:04.726850   511 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:58:04.729542   766 leader_election.cc:290] T 61c59a4c19754dedb9d385e2ecfe145b P 1bcd68d29a7b41a9bf114ac103d4c649 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b5e46fc5aac9426a945bb34207d634d4 (127.0.107.193:38425), 263986885d354f7ba08739e419a6cded (127.0.107.194:45491)
I20260709 12:58:04.729418   565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "61c59a4c19754dedb9d385e2ecfe145b" candidate_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5e46fc5aac9426a945bb34207d634d4" is_pre_election: true
I20260709 12:58:04.730621   565 raft_consensus.cc:2468] T 61c59a4c19754dedb9d385e2ecfe145b P b5e46fc5aac9426a945bb34207d634d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bcd68d29a7b41a9bf114ac103d4c649 in term 0.
I20260709 12:58:04.732401   640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "61c59a4c19754dedb9d385e2ecfe145b" candidate_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "263986885d354f7ba08739e419a6cded" is_pre_election: true
I20260709 12:58:04.732841   681 leader_election.cc:304] T 61c59a4c19754dedb9d385e2ecfe145b P 1bcd68d29a7b41a9bf114ac103d4c649 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bcd68d29a7b41a9bf114ac103d4c649, b5e46fc5aac9426a945bb34207d634d4; no voters: 
I20260709 12:58:04.734479   766 raft_consensus.cc:2804] T 61c59a4c19754dedb9d385e2ecfe145b P 1bcd68d29a7b41a9bf114ac103d4c649 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:58:04.735095   766 raft_consensus.cc:493] T 61c59a4c19754dedb9d385e2ecfe145b P 1bcd68d29a7b41a9bf114ac103d4c649 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:58:04.735705   766 raft_consensus.cc:3060] T 61c59a4c19754dedb9d385e2ecfe145b P 1bcd68d29a7b41a9bf114ac103d4c649 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:04.735764   640 raft_consensus.cc:2468] T 61c59a4c19754dedb9d385e2ecfe145b P 263986885d354f7ba08739e419a6cded [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bcd68d29a7b41a9bf114ac103d4c649 in term 0.
I20260709 12:58:04.736457   461 catalog_manager.cc:5697] T dd06089bfde94412b12705a116ecf3d5 P 1bcd68d29a7b41a9bf114ac103d4c649 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1bcd68d29a7b41a9bf114ac103d4c649 (127.0.107.195). New cstate: current_term: 1 leader_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } health_report { overall_health: HEALTHY } } }
I20260709 12:58:04.743408   760 ts_tablet_manager.cc:1434] T d4d26ba9fb084e748bddb93f7547621f P 1bcd68d29a7b41a9bf114ac103d4c649: Time spent starting tablet: real 0.034s	user 0.002s	sys 0.005s
I20260709 12:58:04.751094   774 consensus_queue.cc:1048] T d8b33c9315094794a876a4b62755efaf P b5e46fc5aac9426a945bb34207d634d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20260709 12:58:04.756623   766 raft_consensus.cc:515] T 61c59a4c19754dedb9d385e2ecfe145b P 1bcd68d29a7b41a9bf114ac103d4c649 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } }
I20260709 12:58:04.768963   565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "61c59a4c19754dedb9d385e2ecfe145b" candidate_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5e46fc5aac9426a945bb34207d634d4"
I20260709 12:58:04.769869   565 raft_consensus.cc:3060] T 61c59a4c19754dedb9d385e2ecfe145b P b5e46fc5aac9426a945bb34207d634d4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:04.777261   766 leader_election.cc:290] T 61c59a4c19754dedb9d385e2ecfe145b P 1bcd68d29a7b41a9bf114ac103d4c649 [CANDIDATE]: Term 1 election: Requested vote from peers b5e46fc5aac9426a945bb34207d634d4 (127.0.107.193:38425), 263986885d354f7ba08739e419a6cded (127.0.107.194:45491)
I20260709 12:58:04.778975   639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "61c59a4c19754dedb9d385e2ecfe145b" candidate_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "263986885d354f7ba08739e419a6cded"
I20260709 12:58:04.779642   639 raft_consensus.cc:3060] T 61c59a4c19754dedb9d385e2ecfe145b P 263986885d354f7ba08739e419a6cded [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:04.785004   792 consensus_queue.cc:1048] T 2e14183cc6064b1cbe6530eb1c8c899c P 1bcd68d29a7b41a9bf114ac103d4c649 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:58:04.794327   639 raft_consensus.cc:2468] T 61c59a4c19754dedb9d385e2ecfe145b P 263986885d354f7ba08739e419a6cded [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1bcd68d29a7b41a9bf114ac103d4c649 in term 1.
I20260709 12:58:04.800874   774 consensus_queue.cc:1048] T d8b33c9315094794a876a4b62755efaf P b5e46fc5aac9426a945bb34207d634d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:58:04.802944   680 leader_election.cc:304] T 61c59a4c19754dedb9d385e2ecfe145b P 1bcd68d29a7b41a9bf114ac103d4c649 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bcd68d29a7b41a9bf114ac103d4c649, 263986885d354f7ba08739e419a6cded; no voters: 
I20260709 12:58:04.803560   565 raft_consensus.cc:2468] T 61c59a4c19754dedb9d385e2ecfe145b P b5e46fc5aac9426a945bb34207d634d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1bcd68d29a7b41a9bf114ac103d4c649 in term 1.
I20260709 12:58:04.804158   766 raft_consensus.cc:2804] T 61c59a4c19754dedb9d385e2ecfe145b P 1bcd68d29a7b41a9bf114ac103d4c649 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:58:04.805100   766 raft_consensus.cc:697] T 61c59a4c19754dedb9d385e2ecfe145b P 1bcd68d29a7b41a9bf114ac103d4c649 [term 1 LEADER]: Becoming Leader. State: Replica: 1bcd68d29a7b41a9bf114ac103d4c649, State: Running, Role: LEADER
I20260709 12:58:04.818843   766 consensus_queue.cc:237] T 61c59a4c19754dedb9d385e2ecfe145b P 1bcd68d29a7b41a9bf114ac103d4c649 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } }
I20260709 12:58:04.859202   792 consensus_queue.cc:1048] T 2e14183cc6064b1cbe6530eb1c8c899c P 1bcd68d29a7b41a9bf114ac103d4c649 [LEADER]: Connected to new peer: Peer: permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:58:04.861474   774 consensus_queue.cc:1048] T d4d26ba9fb084e748bddb93f7547621f P b5e46fc5aac9426a945bb34207d634d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:58:04.893345   461 catalog_manager.cc:5697] T 61c59a4c19754dedb9d385e2ecfe145b P 1bcd68d29a7b41a9bf114ac103d4c649 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1bcd68d29a7b41a9bf114ac103d4c649 (127.0.107.195). New cstate: current_term: 1 leader_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } health_report { overall_health: HEALTHY } } }
I20260709 12:58:04.903026   715 raft_consensus.cc:3060] T d4d26ba9fb084e748bddb93f7547621f P 1bcd68d29a7b41a9bf114ac103d4c649 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:04.941870   774 consensus_queue.cc:1048] T d4d26ba9fb084e748bddb93f7547621f P b5e46fc5aac9426a945bb34207d634d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:58:05.037369   782 consensus_queue.cc:1048] T dd06089bfde94412b12705a116ecf3d5 P 1bcd68d29a7b41a9bf114ac103d4c649 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:58:05.065690   782 consensus_queue.cc:1048] T dd06089bfde94412b12705a116ecf3d5 P 1bcd68d29a7b41a9bf114ac103d4c649 [LEADER]: Connected to new peer: Peer: permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:58:05.349711   782 consensus_queue.cc:1048] T 61c59a4c19754dedb9d385e2ecfe145b P 1bcd68d29a7b41a9bf114ac103d4c649 [LEADER]: Connected to new peer: Peer: permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:58:05.366120   766 consensus_queue.cc:1048] T 61c59a4c19754dedb9d385e2ecfe145b P 1bcd68d29a7b41a9bf114ac103d4c649 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:58:05.441205   715 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "793bafcf38ca4ffb9a9aac9374ea2094"
dest_uuid: "1bcd68d29a7b41a9bf114ac103d4c649"
mode: GRACEFUL
new_leader_uuid: "b5e46fc5aac9426a945bb34207d634d4"
 from {username='slave'} at 127.0.0.1:56664
I20260709 12:58:05.442305   715 raft_consensus.cc:606] T 793bafcf38ca4ffb9a9aac9374ea2094 P 1bcd68d29a7b41a9bf114ac103d4c649 [term 1 LEADER]: Received request to transfer leadership to TS b5e46fc5aac9426a945bb34207d634d4
I20260709 12:58:05.449564   715 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "2e14183cc6064b1cbe6530eb1c8c899c"
dest_uuid: "1bcd68d29a7b41a9bf114ac103d4c649"
mode: GRACEFUL
new_leader_uuid: "b5e46fc5aac9426a945bb34207d634d4"
 from {username='slave'} at 127.0.0.1:56664
I20260709 12:58:05.450312   715 raft_consensus.cc:606] T 2e14183cc6064b1cbe6530eb1c8c899c P 1bcd68d29a7b41a9bf114ac103d4c649 [term 1 LEADER]: Received request to transfer leadership to TS b5e46fc5aac9426a945bb34207d634d4
I20260709 12:58:05.456192   715 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "d441098f471f4b0084938676c20981e5"
dest_uuid: "1bcd68d29a7b41a9bf114ac103d4c649"
mode: GRACEFUL
new_leader_uuid: "b5e46fc5aac9426a945bb34207d634d4"
 from {username='slave'} at 127.0.0.1:56664
I20260709 12:58:05.456815   715 raft_consensus.cc:606] T d441098f471f4b0084938676c20981e5 P 1bcd68d29a7b41a9bf114ac103d4c649 [term 1 LEADER]: Received request to transfer leadership to TS b5e46fc5aac9426a945bb34207d634d4
I20260709 12:58:05.476647   563 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "59112618a86046e1af899ce6d3c85203"
dest_uuid: "b5e46fc5aac9426a945bb34207d634d4"
mode: GRACEFUL
new_leader_uuid: "263986885d354f7ba08739e419a6cded"
 from {username='slave'} at 127.0.0.1:45358
I20260709 12:58:05.477362   563 raft_consensus.cc:606] T 59112618a86046e1af899ce6d3c85203 P b5e46fc5aac9426a945bb34207d634d4 [term 1 LEADER]: Received request to transfer leadership to TS 263986885d354f7ba08739e419a6cded
I20260709 12:58:05.484007   715 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "e6aabcfcdeb34e6c94ec0bb089deabda"
dest_uuid: "1bcd68d29a7b41a9bf114ac103d4c649"
mode: GRACEFUL
new_leader_uuid: "263986885d354f7ba08739e419a6cded"
 from {username='slave'} at 127.0.0.1:56664
I20260709 12:58:05.484656   715 raft_consensus.cc:606] T e6aabcfcdeb34e6c94ec0bb089deabda P 1bcd68d29a7b41a9bf114ac103d4c649 [term 1 LEADER]: Received request to transfer leadership to TS 263986885d354f7ba08739e419a6cded
I20260709 12:58:05.489907   563 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "d4d26ba9fb084e748bddb93f7547621f"
dest_uuid: "b5e46fc5aac9426a945bb34207d634d4"
mode: GRACEFUL
new_leader_uuid: "263986885d354f7ba08739e419a6cded"
 from {username='slave'} at 127.0.0.1:45358
I20260709 12:58:05.490628   563 raft_consensus.cc:606] T d4d26ba9fb084e748bddb93f7547621f P b5e46fc5aac9426a945bb34207d634d4 [term 1 LEADER]: Received request to transfer leadership to TS 263986885d354f7ba08739e419a6cded
I20260709 12:58:05.496906   715 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "61c59a4c19754dedb9d385e2ecfe145b"
dest_uuid: "1bcd68d29a7b41a9bf114ac103d4c649"
mode: GRACEFUL
new_leader_uuid: "263986885d354f7ba08739e419a6cded"
 from {username='slave'} at 127.0.0.1:56664
I20260709 12:58:05.497427   715 raft_consensus.cc:606] T 61c59a4c19754dedb9d385e2ecfe145b P 1bcd68d29a7b41a9bf114ac103d4c649 [term 1 LEADER]: Received request to transfer leadership to TS 263986885d354f7ba08739e419a6cded
I20260709 12:58:05.543490   849 raft_consensus.cc:993] T d4d26ba9fb084e748bddb93f7547621f P b5e46fc5aac9426a945bb34207d634d4: : Instructing follower 263986885d354f7ba08739e419a6cded to start an election
I20260709 12:58:05.544003   810 raft_consensus.cc:1081] T d4d26ba9fb084e748bddb93f7547621f P b5e46fc5aac9426a945bb34207d634d4 [term 1 LEADER]: Signalling peer 263986885d354f7ba08739e419a6cded to start an election
I20260709 12:58:05.545922   640 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "d4d26ba9fb084e748bddb93f7547621f"
dest_uuid: "263986885d354f7ba08739e419a6cded"
 from {username='slave'} at 127.0.0.1:60338
I20260709 12:58:05.546551   640 raft_consensus.cc:493] T d4d26ba9fb084e748bddb93f7547621f P 263986885d354f7ba08739e419a6cded [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:58:05.546908   640 raft_consensus.cc:3060] T d4d26ba9fb084e748bddb93f7547621f P 263986885d354f7ba08739e419a6cded [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:58:05.552896   640 raft_consensus.cc:515] T d4d26ba9fb084e748bddb93f7547621f P 263986885d354f7ba08739e419a6cded [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } }
I20260709 12:58:05.554736   640 leader_election.cc:290] T d4d26ba9fb084e748bddb93f7547621f P 263986885d354f7ba08739e419a6cded [CANDIDATE]: Term 2 election: Requested vote from peers b5e46fc5aac9426a945bb34207d634d4 (127.0.107.193:38425), 1bcd68d29a7b41a9bf114ac103d4c649 (127.0.107.195:46365)
I20260709 12:58:05.555684   563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d26ba9fb084e748bddb93f7547621f" candidate_uuid: "263986885d354f7ba08739e419a6cded" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "b5e46fc5aac9426a945bb34207d634d4"
I20260709 12:58:05.556077   715 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d26ba9fb084e748bddb93f7547621f" candidate_uuid: "263986885d354f7ba08739e419a6cded" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1bcd68d29a7b41a9bf114ac103d4c649"
I20260709 12:58:05.556445   563 raft_consensus.cc:3055] T d4d26ba9fb084e748bddb93f7547621f P b5e46fc5aac9426a945bb34207d634d4 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:58:05.556716   715 raft_consensus.cc:3060] T d4d26ba9fb084e748bddb93f7547621f P 1bcd68d29a7b41a9bf114ac103d4c649 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:58:05.556819   563 raft_consensus.cc:740] T d4d26ba9fb084e748bddb93f7547621f P b5e46fc5aac9426a945bb34207d634d4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b5e46fc5aac9426a945bb34207d634d4, State: Running, Role: LEADER
I20260709 12:58:05.558166   563 consensus_queue.cc:260] T d4d26ba9fb084e748bddb93f7547621f P b5e46fc5aac9426a945bb34207d634d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } }
I20260709 12:58:05.559803   563 raft_consensus.cc:3060] T d4d26ba9fb084e748bddb93f7547621f P b5e46fc5aac9426a945bb34207d634d4 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:58:05.561954   792 raft_consensus.cc:993] T d441098f471f4b0084938676c20981e5 P 1bcd68d29a7b41a9bf114ac103d4c649: : Instructing follower b5e46fc5aac9426a945bb34207d634d4 to start an election
I20260709 12:58:05.562409   792 raft_consensus.cc:1081] T d441098f471f4b0084938676c20981e5 P 1bcd68d29a7b41a9bf114ac103d4c649 [term 1 LEADER]: Signalling peer b5e46fc5aac9426a945bb34207d634d4 to start an election
I20260709 12:58:05.564294   565 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "d441098f471f4b0084938676c20981e5"
dest_uuid: "b5e46fc5aac9426a945bb34207d634d4"
 from {username='slave'} at 127.0.0.1:45340
I20260709 12:58:05.564689   715 raft_consensus.cc:2468] T d4d26ba9fb084e748bddb93f7547621f P 1bcd68d29a7b41a9bf114ac103d4c649 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 263986885d354f7ba08739e419a6cded in term 2.
I20260709 12:58:05.564875   565 raft_consensus.cc:493] T d441098f471f4b0084938676c20981e5 P b5e46fc5aac9426a945bb34207d634d4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:58:05.565250   565 raft_consensus.cc:3060] T d441098f471f4b0084938676c20981e5 P b5e46fc5aac9426a945bb34207d634d4 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:58:05.566362   604 leader_election.cc:304] T d4d26ba9fb084e748bddb93f7547621f P 263986885d354f7ba08739e419a6cded [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bcd68d29a7b41a9bf114ac103d4c649, 263986885d354f7ba08739e419a6cded; no voters: 
I20260709 12:58:05.567214   827 raft_consensus.cc:2804] T d4d26ba9fb084e748bddb93f7547621f P 263986885d354f7ba08739e419a6cded [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:58:05.568496   563 raft_consensus.cc:2468] T d4d26ba9fb084e748bddb93f7547621f P b5e46fc5aac9426a945bb34207d634d4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 263986885d354f7ba08739e419a6cded in term 2.
I20260709 12:58:05.570081   827 raft_consensus.cc:697] T d4d26ba9fb084e748bddb93f7547621f P 263986885d354f7ba08739e419a6cded [term 2 LEADER]: Becoming Leader. State: Replica: 263986885d354f7ba08739e419a6cded, State: Running, Role: LEADER
I20260709 12:58:05.571147   827 consensus_queue.cc:237] T d4d26ba9fb084e748bddb93f7547621f P 263986885d354f7ba08739e419a6cded [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } }
I20260709 12:58:05.572420   565 raft_consensus.cc:515] T d441098f471f4b0084938676c20981e5 P b5e46fc5aac9426a945bb34207d634d4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } }
I20260709 12:58:05.575024   640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d441098f471f4b0084938676c20981e5" candidate_uuid: "b5e46fc5aac9426a945bb34207d634d4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "263986885d354f7ba08739e419a6cded"
I20260709 12:58:05.575588   640 raft_consensus.cc:3060] T d441098f471f4b0084938676c20981e5 P 263986885d354f7ba08739e419a6cded [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:58:05.575922   715 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d441098f471f4b0084938676c20981e5" candidate_uuid: "b5e46fc5aac9426a945bb34207d634d4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1bcd68d29a7b41a9bf114ac103d4c649"
I20260709 12:58:05.576488   715 raft_consensus.cc:3055] T d441098f471f4b0084938676c20981e5 P 1bcd68d29a7b41a9bf114ac103d4c649 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:58:05.576792   715 raft_consensus.cc:740] T d441098f471f4b0084938676c20981e5 P 1bcd68d29a7b41a9bf114ac103d4c649 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1bcd68d29a7b41a9bf114ac103d4c649, State: Running, Role: LEADER
I20260709 12:58:05.577566   715 consensus_queue.cc:260] T d441098f471f4b0084938676c20981e5 P 1bcd68d29a7b41a9bf114ac103d4c649 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } }
I20260709 12:58:05.578694   715 raft_consensus.cc:3060] T d441098f471f4b0084938676c20981e5 P 1bcd68d29a7b41a9bf114ac103d4c649 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:58:05.582530   640 raft_consensus.cc:2468] T d441098f471f4b0084938676c20981e5 P 263986885d354f7ba08739e419a6cded [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5e46fc5aac9426a945bb34207d634d4 in term 2.
I20260709 12:58:05.582149   462 catalog_manager.cc:5697] T d4d26ba9fb084e748bddb93f7547621f P 263986885d354f7ba08739e419a6cded reported cstate change: term changed from 1 to 2, leader changed from b5e46fc5aac9426a945bb34207d634d4 (127.0.107.193) to 263986885d354f7ba08739e419a6cded (127.0.107.194). New cstate: current_term: 2 leader_uuid: "263986885d354f7ba08739e419a6cded" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } health_report { overall_health: UNKNOWN } } }
I20260709 12:58:05.583451   565 leader_election.cc:290] T d441098f471f4b0084938676c20981e5 P b5e46fc5aac9426a945bb34207d634d4 [CANDIDATE]: Term 2 election: Requested vote from peers 263986885d354f7ba08739e419a6cded (127.0.107.194:45491), 1bcd68d29a7b41a9bf114ac103d4c649 (127.0.107.195:46365)
I20260709 12:58:05.584936   531 leader_election.cc:304] T d441098f471f4b0084938676c20981e5 P b5e46fc5aac9426a945bb34207d634d4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 263986885d354f7ba08739e419a6cded, b5e46fc5aac9426a945bb34207d634d4; no voters: 
I20260709 12:58:05.585800   810 raft_consensus.cc:2804] T d441098f471f4b0084938676c20981e5 P b5e46fc5aac9426a945bb34207d634d4 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:58:05.586292   810 raft_consensus.cc:697] T d441098f471f4b0084938676c20981e5 P b5e46fc5aac9426a945bb34207d634d4 [term 2 LEADER]: Becoming Leader. State: Replica: b5e46fc5aac9426a945bb34207d634d4, State: Running, Role: LEADER
I20260709 12:58:05.587085   715 raft_consensus.cc:2468] T d441098f471f4b0084938676c20981e5 P 1bcd68d29a7b41a9bf114ac103d4c649 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5e46fc5aac9426a945bb34207d634d4 in term 2.
I20260709 12:58:05.587204   810 consensus_queue.cc:237] T d441098f471f4b0084938676c20981e5 P b5e46fc5aac9426a945bb34207d634d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } }
I20260709 12:58:05.596478   461 catalog_manager.cc:5697] T d441098f471f4b0084938676c20981e5 P b5e46fc5aac9426a945bb34207d634d4 reported cstate change: term changed from 1 to 2, leader changed from 1bcd68d29a7b41a9bf114ac103d4c649 (127.0.107.195) to b5e46fc5aac9426a945bb34207d634d4 (127.0.107.193). New cstate: current_term: 2 leader_uuid: "b5e46fc5aac9426a945bb34207d634d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } health_report { overall_health: UNKNOWN } } }
I20260709 12:58:05.617512   766 raft_consensus.cc:993] T e6aabcfcdeb34e6c94ec0bb089deabda P 1bcd68d29a7b41a9bf114ac103d4c649: : Instructing follower 263986885d354f7ba08739e419a6cded to start an election
I20260709 12:58:05.618053   766 raft_consensus.cc:1081] T e6aabcfcdeb34e6c94ec0bb089deabda P 1bcd68d29a7b41a9bf114ac103d4c649 [term 1 LEADER]: Signalling peer 263986885d354f7ba08739e419a6cded to start an election
I20260709 12:58:05.620254   640 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "e6aabcfcdeb34e6c94ec0bb089deabda"
dest_uuid: "263986885d354f7ba08739e419a6cded"
 from {username='slave'} at 127.0.0.1:60330
I20260709 12:58:05.620963   640 raft_consensus.cc:493] T e6aabcfcdeb34e6c94ec0bb089deabda P 263986885d354f7ba08739e419a6cded [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:58:05.621371   640 raft_consensus.cc:3060] T e6aabcfcdeb34e6c94ec0bb089deabda P 263986885d354f7ba08739e419a6cded [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:58:05.627312   849 raft_consensus.cc:993] T 59112618a86046e1af899ce6d3c85203 P b5e46fc5aac9426a945bb34207d634d4: : Instructing follower 263986885d354f7ba08739e419a6cded to start an election
I20260709 12:58:05.627761   810 raft_consensus.cc:1081] T 59112618a86046e1af899ce6d3c85203 P b5e46fc5aac9426a945bb34207d634d4 [term 1 LEADER]: Signalling peer 263986885d354f7ba08739e419a6cded to start an election
I20260709 12:58:05.629303   641 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "59112618a86046e1af899ce6d3c85203"
dest_uuid: "263986885d354f7ba08739e419a6cded"
 from {username='slave'} at 127.0.0.1:60338
I20260709 12:58:05.629890   641 raft_consensus.cc:493] T 59112618a86046e1af899ce6d3c85203 P 263986885d354f7ba08739e419a6cded [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:58:05.629464   640 raft_consensus.cc:515] T e6aabcfcdeb34e6c94ec0bb089deabda P 263986885d354f7ba08739e419a6cded [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } }
I20260709 12:58:05.630254   641 raft_consensus.cc:3060] T 59112618a86046e1af899ce6d3c85203 P 263986885d354f7ba08739e419a6cded [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:58:05.631866   640 leader_election.cc:290] T e6aabcfcdeb34e6c94ec0bb089deabda P 263986885d354f7ba08739e419a6cded [CANDIDATE]: Term 2 election: Requested vote from peers b5e46fc5aac9426a945bb34207d634d4 (127.0.107.193:38425), 1bcd68d29a7b41a9bf114ac103d4c649 (127.0.107.195:46365)
I20260709 12:58:05.633240   565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e6aabcfcdeb34e6c94ec0bb089deabda" candidate_uuid: "263986885d354f7ba08739e419a6cded" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "b5e46fc5aac9426a945bb34207d634d4"
I20260709 12:58:05.633476   715 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e6aabcfcdeb34e6c94ec0bb089deabda" candidate_uuid: "263986885d354f7ba08739e419a6cded" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1bcd68d29a7b41a9bf114ac103d4c649"
I20260709 12:58:05.633854   565 raft_consensus.cc:3060] T e6aabcfcdeb34e6c94ec0bb089deabda P b5e46fc5aac9426a945bb34207d634d4 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:58:05.634107   715 raft_consensus.cc:3055] T e6aabcfcdeb34e6c94ec0bb089deabda P 1bcd68d29a7b41a9bf114ac103d4c649 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:58:05.634502   715 raft_consensus.cc:740] T e6aabcfcdeb34e6c94ec0bb089deabda P 1bcd68d29a7b41a9bf114ac103d4c649 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1bcd68d29a7b41a9bf114ac103d4c649, State: Running, Role: LEADER
I20260709 12:58:05.635486   715 consensus_queue.cc:260] T e6aabcfcdeb34e6c94ec0bb089deabda P 1bcd68d29a7b41a9bf114ac103d4c649 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } }
I20260709 12:58:05.636929   715 raft_consensus.cc:3060] T e6aabcfcdeb34e6c94ec0bb089deabda P 1bcd68d29a7b41a9bf114ac103d4c649 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:58:05.637756   641 raft_consensus.cc:515] T 59112618a86046e1af899ce6d3c85203 P 263986885d354f7ba08739e419a6cded [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } }
I20260709 12:58:05.640430   641 leader_election.cc:290] T 59112618a86046e1af899ce6d3c85203 P 263986885d354f7ba08739e419a6cded [CANDIDATE]: Term 2 election: Requested vote from peers b5e46fc5aac9426a945bb34207d634d4 (127.0.107.193:38425), 1bcd68d29a7b41a9bf114ac103d4c649 (127.0.107.195:46365)
I20260709 12:58:05.642011   565 raft_consensus.cc:2468] T e6aabcfcdeb34e6c94ec0bb089deabda P b5e46fc5aac9426a945bb34207d634d4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 263986885d354f7ba08739e419a6cded in term 2.
I20260709 12:58:05.642172   714 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59112618a86046e1af899ce6d3c85203" candidate_uuid: "263986885d354f7ba08739e419a6cded" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1bcd68d29a7b41a9bf114ac103d4c649"
I20260709 12:58:05.642129   563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59112618a86046e1af899ce6d3c85203" candidate_uuid: "263986885d354f7ba08739e419a6cded" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "b5e46fc5aac9426a945bb34207d634d4"
I20260709 12:58:05.643011   714 raft_consensus.cc:3060] T 59112618a86046e1af899ce6d3c85203 P 1bcd68d29a7b41a9bf114ac103d4c649 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:58:05.643501   563 raft_consensus.cc:3055] T 59112618a86046e1af899ce6d3c85203 P b5e46fc5aac9426a945bb34207d634d4 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:58:05.643558   607 leader_election.cc:304] T e6aabcfcdeb34e6c94ec0bb089deabda P 263986885d354f7ba08739e419a6cded [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 263986885d354f7ba08739e419a6cded, b5e46fc5aac9426a945bb34207d634d4; no voters: 
I20260709 12:58:05.643956   563 raft_consensus.cc:740] T 59112618a86046e1af899ce6d3c85203 P b5e46fc5aac9426a945bb34207d634d4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b5e46fc5aac9426a945bb34207d634d4, State: Running, Role: LEADER
I20260709 12:58:05.644701   827 raft_consensus.cc:2804] T e6aabcfcdeb34e6c94ec0bb089deabda P 263986885d354f7ba08739e419a6cded [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:58:05.644816   563 consensus_queue.cc:260] T 59112618a86046e1af899ce6d3c85203 P b5e46fc5aac9426a945bb34207d634d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } }
I20260709 12:58:05.645321   827 raft_consensus.cc:697] T e6aabcfcdeb34e6c94ec0bb089deabda P 263986885d354f7ba08739e419a6cded [term 2 LEADER]: Becoming Leader. State: Replica: 263986885d354f7ba08739e419a6cded, State: Running, Role: LEADER
I20260709 12:58:05.646194   563 raft_consensus.cc:3060] T 59112618a86046e1af899ce6d3c85203 P b5e46fc5aac9426a945bb34207d634d4 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:58:05.646296   827 consensus_queue.cc:237] T e6aabcfcdeb34e6c94ec0bb089deabda P 263986885d354f7ba08739e419a6cded [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } }
I20260709 12:58:05.650712   714 raft_consensus.cc:2468] T 59112618a86046e1af899ce6d3c85203 P 1bcd68d29a7b41a9bf114ac103d4c649 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 263986885d354f7ba08739e419a6cded in term 2.
I20260709 12:58:05.652236   604 leader_election.cc:304] T 59112618a86046e1af899ce6d3c85203 P 263986885d354f7ba08739e419a6cded [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bcd68d29a7b41a9bf114ac103d4c649, 263986885d354f7ba08739e419a6cded; no voters: 
I20260709 12:58:05.654706   850 raft_consensus.cc:2804] T 59112618a86046e1af899ce6d3c85203 P 263986885d354f7ba08739e419a6cded [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:58:05.656358   563 raft_consensus.cc:2468] T 59112618a86046e1af899ce6d3c85203 P b5e46fc5aac9426a945bb34207d634d4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 263986885d354f7ba08739e419a6cded in term 2.
I20260709 12:58:05.658427   715 raft_consensus.cc:2468] T e6aabcfcdeb34e6c94ec0bb089deabda P 1bcd68d29a7b41a9bf114ac103d4c649 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 263986885d354f7ba08739e419a6cded in term 2.
I20260709 12:58:05.660318   850 raft_consensus.cc:697] T 59112618a86046e1af899ce6d3c85203 P 263986885d354f7ba08739e419a6cded [term 2 LEADER]: Becoming Leader. State: Replica: 263986885d354f7ba08739e419a6cded, State: Running, Role: LEADER
I20260709 12:58:05.659353   461 catalog_manager.cc:5697] T e6aabcfcdeb34e6c94ec0bb089deabda P 263986885d354f7ba08739e419a6cded reported cstate change: term changed from 1 to 2, leader changed from 1bcd68d29a7b41a9bf114ac103d4c649 (127.0.107.195) to 263986885d354f7ba08739e419a6cded (127.0.107.194). New cstate: current_term: 2 leader_uuid: "263986885d354f7ba08739e419a6cded" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } health_report { overall_health: UNKNOWN } } }
I20260709 12:58:05.661352   850 consensus_queue.cc:237] T 59112618a86046e1af899ce6d3c85203 P 263986885d354f7ba08739e419a6cded [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } }
I20260709 12:58:05.674125   461 catalog_manager.cc:5697] T 59112618a86046e1af899ce6d3c85203 P 263986885d354f7ba08739e419a6cded reported cstate change: term changed from 1 to 2, leader changed from b5e46fc5aac9426a945bb34207d634d4 (127.0.107.193) to 263986885d354f7ba08739e419a6cded (127.0.107.194). New cstate: current_term: 2 leader_uuid: "263986885d354f7ba08739e419a6cded" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } health_report { overall_health: UNKNOWN } } }
I20260709 12:58:05.690649   792 raft_consensus.cc:993] T 2e14183cc6064b1cbe6530eb1c8c899c P 1bcd68d29a7b41a9bf114ac103d4c649: : Instructing follower b5e46fc5aac9426a945bb34207d634d4 to start an election
I20260709 12:58:05.691215   766 raft_consensus.cc:1081] T 2e14183cc6064b1cbe6530eb1c8c899c P 1bcd68d29a7b41a9bf114ac103d4c649 [term 1 LEADER]: Signalling peer b5e46fc5aac9426a945bb34207d634d4 to start an election
I20260709 12:58:05.693177   563 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "2e14183cc6064b1cbe6530eb1c8c899c"
dest_uuid: "b5e46fc5aac9426a945bb34207d634d4"
 from {username='slave'} at 127.0.0.1:45340
I20260709 12:58:05.693768   563 raft_consensus.cc:493] T 2e14183cc6064b1cbe6530eb1c8c899c P b5e46fc5aac9426a945bb34207d634d4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:58:05.694142   563 raft_consensus.cc:3060] T 2e14183cc6064b1cbe6530eb1c8c899c P b5e46fc5aac9426a945bb34207d634d4 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:58:05.703392   563 raft_consensus.cc:515] T 2e14183cc6064b1cbe6530eb1c8c899c P b5e46fc5aac9426a945bb34207d634d4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } }
I20260709 12:58:05.706935   563 leader_election.cc:290] T 2e14183cc6064b1cbe6530eb1c8c899c P b5e46fc5aac9426a945bb34207d634d4 [CANDIDATE]: Term 2 election: Requested vote from peers 263986885d354f7ba08739e419a6cded (127.0.107.194:45491), 1bcd68d29a7b41a9bf114ac103d4c649 (127.0.107.195:46365)
I20260709 12:58:05.706933   641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e14183cc6064b1cbe6530eb1c8c899c" candidate_uuid: "b5e46fc5aac9426a945bb34207d634d4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "263986885d354f7ba08739e419a6cded"
I20260709 12:58:05.707278   715 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e14183cc6064b1cbe6530eb1c8c899c" candidate_uuid: "b5e46fc5aac9426a945bb34207d634d4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1bcd68d29a7b41a9bf114ac103d4c649"
I20260709 12:58:05.707933   641 raft_consensus.cc:3060] T 2e14183cc6064b1cbe6530eb1c8c899c P 263986885d354f7ba08739e419a6cded [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:58:05.708241   715 raft_consensus.cc:3055] T 2e14183cc6064b1cbe6530eb1c8c899c P 1bcd68d29a7b41a9bf114ac103d4c649 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:58:05.708829   715 raft_consensus.cc:740] T 2e14183cc6064b1cbe6530eb1c8c899c P 1bcd68d29a7b41a9bf114ac103d4c649 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1bcd68d29a7b41a9bf114ac103d4c649, State: Running, Role: LEADER
I20260709 12:58:05.710316   715 consensus_queue.cc:260] T 2e14183cc6064b1cbe6530eb1c8c899c P 1bcd68d29a7b41a9bf114ac103d4c649 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } }
I20260709 12:58:05.712237   715 raft_consensus.cc:3060] T 2e14183cc6064b1cbe6530eb1c8c899c P 1bcd68d29a7b41a9bf114ac103d4c649 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:58:05.716246   641 raft_consensus.cc:2468] T 2e14183cc6064b1cbe6530eb1c8c899c P 263986885d354f7ba08739e419a6cded [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5e46fc5aac9426a945bb34207d634d4 in term 2.
I20260709 12:58:05.717633   531 leader_election.cc:304] T 2e14183cc6064b1cbe6530eb1c8c899c P b5e46fc5aac9426a945bb34207d634d4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 263986885d354f7ba08739e419a6cded, b5e46fc5aac9426a945bb34207d634d4; no voters: 
I20260709 12:58:05.722234   810 raft_consensus.cc:2804] T 2e14183cc6064b1cbe6530eb1c8c899c P b5e46fc5aac9426a945bb34207d634d4 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:58:05.722543   715 raft_consensus.cc:2468] T 2e14183cc6064b1cbe6530eb1c8c899c P 1bcd68d29a7b41a9bf114ac103d4c649 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5e46fc5aac9426a945bb34207d634d4 in term 2.
I20260709 12:58:05.722952   810 raft_consensus.cc:697] T 2e14183cc6064b1cbe6530eb1c8c899c P b5e46fc5aac9426a945bb34207d634d4 [term 2 LEADER]: Becoming Leader. State: Replica: b5e46fc5aac9426a945bb34207d634d4, State: Running, Role: LEADER
I20260709 12:58:05.725167   810 consensus_queue.cc:237] T 2e14183cc6064b1cbe6530eb1c8c899c P b5e46fc5aac9426a945bb34207d634d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } }
I20260709 12:58:05.736392   461 catalog_manager.cc:5697] T 2e14183cc6064b1cbe6530eb1c8c899c P b5e46fc5aac9426a945bb34207d634d4 reported cstate change: term changed from 1 to 2, leader changed from 1bcd68d29a7b41a9bf114ac103d4c649 (127.0.107.195) to b5e46fc5aac9426a945bb34207d634d4 (127.0.107.193). New cstate: current_term: 2 leader_uuid: "b5e46fc5aac9426a945bb34207d634d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } health_report { overall_health: UNKNOWN } } }
I20260709 12:58:05.796954   792 raft_consensus.cc:993] T 793bafcf38ca4ffb9a9aac9374ea2094 P 1bcd68d29a7b41a9bf114ac103d4c649: : Instructing follower b5e46fc5aac9426a945bb34207d634d4 to start an election
I20260709 12:58:05.797559   766 raft_consensus.cc:1081] T 793bafcf38ca4ffb9a9aac9374ea2094 P 1bcd68d29a7b41a9bf114ac103d4c649 [term 1 LEADER]: Signalling peer b5e46fc5aac9426a945bb34207d634d4 to start an election
I20260709 12:58:05.799736   563 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "793bafcf38ca4ffb9a9aac9374ea2094"
dest_uuid: "b5e46fc5aac9426a945bb34207d634d4"
 from {username='slave'} at 127.0.0.1:45340
I20260709 12:58:05.800390   563 raft_consensus.cc:493] T 793bafcf38ca4ffb9a9aac9374ea2094 P b5e46fc5aac9426a945bb34207d634d4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:58:05.800810   563 raft_consensus.cc:3060] T 793bafcf38ca4ffb9a9aac9374ea2094 P b5e46fc5aac9426a945bb34207d634d4 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:58:05.807677   563 raft_consensus.cc:515] T 793bafcf38ca4ffb9a9aac9374ea2094 P b5e46fc5aac9426a945bb34207d634d4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } } peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } }
I20260709 12:58:05.810876   563 leader_election.cc:290] T 793bafcf38ca4ffb9a9aac9374ea2094 P b5e46fc5aac9426a945bb34207d634d4 [CANDIDATE]: Term 2 election: Requested vote from peers 263986885d354f7ba08739e419a6cded (127.0.107.194:45491), 1bcd68d29a7b41a9bf114ac103d4c649 (127.0.107.195:46365)
I20260709 12:58:05.812120   641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "793bafcf38ca4ffb9a9aac9374ea2094" candidate_uuid: "b5e46fc5aac9426a945bb34207d634d4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "263986885d354f7ba08739e419a6cded"
I20260709 12:58:05.812446   714 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "793bafcf38ca4ffb9a9aac9374ea2094" candidate_uuid: "b5e46fc5aac9426a945bb34207d634d4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1bcd68d29a7b41a9bf114ac103d4c649"
I20260709 12:58:05.813030   641 raft_consensus.cc:3060] T 793bafcf38ca4ffb9a9aac9374ea2094 P 263986885d354f7ba08739e419a6cded [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:58:05.813292   714 raft_consensus.cc:3055] T 793bafcf38ca4ffb9a9aac9374ea2094 P 1bcd68d29a7b41a9bf114ac103d4c649 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:58:05.813817   714 raft_consensus.cc:740] T 793bafcf38ca4ffb9a9aac9374ea2094 P 1bcd68d29a7b41a9bf114ac103d4c649 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1bcd68d29a7b41a9bf114ac103d4c649, State: Running, Role: LEADER
I20260709 12:58:05.815021   714 consensus_queue.cc:260] T 793bafcf38ca4ffb9a9aac9374ea2094 P 1bcd68d29a7b41a9bf114ac103d4c649 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } } peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } }
I20260709 12:58:05.816814   714 raft_consensus.cc:3060] T 793bafcf38ca4ffb9a9aac9374ea2094 P 1bcd68d29a7b41a9bf114ac103d4c649 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:58:05.820676   641 raft_consensus.cc:2468] T 793bafcf38ca4ffb9a9aac9374ea2094 P 263986885d354f7ba08739e419a6cded [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5e46fc5aac9426a945bb34207d634d4 in term 2.
I20260709 12:58:05.821914   531 leader_election.cc:304] T 793bafcf38ca4ffb9a9aac9374ea2094 P b5e46fc5aac9426a945bb34207d634d4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 263986885d354f7ba08739e419a6cded, b5e46fc5aac9426a945bb34207d634d4; no voters: 
I20260709 12:58:05.822862   810 raft_consensus.cc:2804] T 793bafcf38ca4ffb9a9aac9374ea2094 P b5e46fc5aac9426a945bb34207d634d4 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:58:05.823299   810 raft_consensus.cc:697] T 793bafcf38ca4ffb9a9aac9374ea2094 P b5e46fc5aac9426a945bb34207d634d4 [term 2 LEADER]: Becoming Leader. State: Replica: b5e46fc5aac9426a945bb34207d634d4, State: Running, Role: LEADER
I20260709 12:58:05.824091   810 consensus_queue.cc:237] T 793bafcf38ca4ffb9a9aac9374ea2094 P b5e46fc5aac9426a945bb34207d634d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } } peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } }
I20260709 12:58:05.827283   714 raft_consensus.cc:2468] T 793bafcf38ca4ffb9a9aac9374ea2094 P 1bcd68d29a7b41a9bf114ac103d4c649 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5e46fc5aac9426a945bb34207d634d4 in term 2.
I20260709 12:58:05.834949   461 catalog_manager.cc:5697] T 793bafcf38ca4ffb9a9aac9374ea2094 P b5e46fc5aac9426a945bb34207d634d4 reported cstate change: term changed from 1 to 2, leader changed from 1bcd68d29a7b41a9bf114ac103d4c649 (127.0.107.195) to b5e46fc5aac9426a945bb34207d634d4 (127.0.107.193). New cstate: current_term: 2 leader_uuid: "b5e46fc5aac9426a945bb34207d634d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } health_report { overall_health: HEALTHY } } }
I20260709 12:58:05.892534   782 raft_consensus.cc:993] T 61c59a4c19754dedb9d385e2ecfe145b P 1bcd68d29a7b41a9bf114ac103d4c649: : Instructing follower 263986885d354f7ba08739e419a6cded to start an election
I20260709 12:58:05.893105   766 raft_consensus.cc:1081] T 61c59a4c19754dedb9d385e2ecfe145b P 1bcd68d29a7b41a9bf114ac103d4c649 [term 1 LEADER]: Signalling peer 263986885d354f7ba08739e419a6cded to start an election
I20260709 12:58:05.894876   641 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "61c59a4c19754dedb9d385e2ecfe145b"
dest_uuid: "263986885d354f7ba08739e419a6cded"
 from {username='slave'} at 127.0.0.1:60330
I20260709 12:58:05.895391   641 raft_consensus.cc:493] T 61c59a4c19754dedb9d385e2ecfe145b P 263986885d354f7ba08739e419a6cded [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:58:05.895733   641 raft_consensus.cc:3060] T 61c59a4c19754dedb9d385e2ecfe145b P 263986885d354f7ba08739e419a6cded [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:58:05.900936   641 raft_consensus.cc:515] T 61c59a4c19754dedb9d385e2ecfe145b P 263986885d354f7ba08739e419a6cded [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } }
I20260709 12:58:05.902716   641 leader_election.cc:290] T 61c59a4c19754dedb9d385e2ecfe145b P 263986885d354f7ba08739e419a6cded [CANDIDATE]: Term 2 election: Requested vote from peers b5e46fc5aac9426a945bb34207d634d4 (127.0.107.193:38425), 1bcd68d29a7b41a9bf114ac103d4c649 (127.0.107.195:46365)
I20260709 12:58:05.904007   714 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "61c59a4c19754dedb9d385e2ecfe145b" candidate_uuid: "263986885d354f7ba08739e419a6cded" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1bcd68d29a7b41a9bf114ac103d4c649"
I20260709 12:58:05.904733   714 raft_consensus.cc:3055] T 61c59a4c19754dedb9d385e2ecfe145b P 1bcd68d29a7b41a9bf114ac103d4c649 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:58:05.903991   563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "61c59a4c19754dedb9d385e2ecfe145b" candidate_uuid: "263986885d354f7ba08739e419a6cded" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "b5e46fc5aac9426a945bb34207d634d4"
I20260709 12:58:05.905007   714 raft_consensus.cc:740] T 61c59a4c19754dedb9d385e2ecfe145b P 1bcd68d29a7b41a9bf114ac103d4c649 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1bcd68d29a7b41a9bf114ac103d4c649, State: Running, Role: LEADER
I20260709 12:58:05.905234   563 raft_consensus.cc:3060] T 61c59a4c19754dedb9d385e2ecfe145b P b5e46fc5aac9426a945bb34207d634d4 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:58:05.905910   714 consensus_queue.cc:260] T 61c59a4c19754dedb9d385e2ecfe145b P 1bcd68d29a7b41a9bf114ac103d4c649 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } }
I20260709 12:58:05.907212   714 raft_consensus.cc:3060] T 61c59a4c19754dedb9d385e2ecfe145b P 1bcd68d29a7b41a9bf114ac103d4c649 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:58:05.913029   563 raft_consensus.cc:2468] T 61c59a4c19754dedb9d385e2ecfe145b P b5e46fc5aac9426a945bb34207d634d4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 263986885d354f7ba08739e419a6cded in term 2.
I20260709 12:58:05.914475   607 leader_election.cc:304] T 61c59a4c19754dedb9d385e2ecfe145b P 263986885d354f7ba08739e419a6cded [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 263986885d354f7ba08739e419a6cded, b5e46fc5aac9426a945bb34207d634d4; no voters: 
I20260709 12:58:05.914809   714 raft_consensus.cc:2468] T 61c59a4c19754dedb9d385e2ecfe145b P 1bcd68d29a7b41a9bf114ac103d4c649 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 263986885d354f7ba08739e419a6cded in term 2.
I20260709 12:58:05.915467   850 raft_consensus.cc:2804] T 61c59a4c19754dedb9d385e2ecfe145b P 263986885d354f7ba08739e419a6cded [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:58:05.916008   850 raft_consensus.cc:697] T 61c59a4c19754dedb9d385e2ecfe145b P 263986885d354f7ba08739e419a6cded [term 2 LEADER]: Becoming Leader. State: Replica: 263986885d354f7ba08739e419a6cded, State: Running, Role: LEADER
I20260709 12:58:05.916774   850 consensus_queue.cc:237] T 61c59a4c19754dedb9d385e2ecfe145b P 263986885d354f7ba08739e419a6cded [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } }
I20260709 12:58:05.924180   462 catalog_manager.cc:5697] T 61c59a4c19754dedb9d385e2ecfe145b P 263986885d354f7ba08739e419a6cded reported cstate change: term changed from 1 to 2, leader changed from 1bcd68d29a7b41a9bf114ac103d4c649 (127.0.107.195) to 263986885d354f7ba08739e419a6cded (127.0.107.194). New cstate: current_term: 2 leader_uuid: "263986885d354f7ba08739e419a6cded" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } health_report { overall_health: UNKNOWN } } }
I20260709 12:58:05.970721   641 raft_consensus.cc:1275] T d441098f471f4b0084938676c20981e5 P 263986885d354f7ba08739e419a6cded [term 2 FOLLOWER]: Refusing update from remote peer b5e46fc5aac9426a945bb34207d634d4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:58:05.972163   810 consensus_queue.cc:1048] T d441098f471f4b0084938676c20981e5 P b5e46fc5aac9426a945bb34207d634d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:58:05.989207   714 raft_consensus.cc:1275] T d441098f471f4b0084938676c20981e5 P 1bcd68d29a7b41a9bf114ac103d4c649 [term 2 FOLLOWER]: Refusing update from remote peer b5e46fc5aac9426a945bb34207d634d4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:58:05.990659   860 consensus_queue.cc:1048] T d441098f471f4b0084938676c20981e5 P b5e46fc5aac9426a945bb34207d634d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:58:06.091413   563 raft_consensus.cc:1275] T e6aabcfcdeb34e6c94ec0bb089deabda P b5e46fc5aac9426a945bb34207d634d4 [term 2 FOLLOWER]: Refusing update from remote peer 263986885d354f7ba08739e419a6cded: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:58:06.093111   850 consensus_queue.cc:1048] T e6aabcfcdeb34e6c94ec0bb089deabda P 263986885d354f7ba08739e419a6cded [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:58:06.113714   565 raft_consensus.cc:1275] T d4d26ba9fb084e748bddb93f7547621f P b5e46fc5aac9426a945bb34207d634d4 [term 2 FOLLOWER]: Refusing update from remote peer 263986885d354f7ba08739e419a6cded: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:58:06.115233   850 consensus_queue.cc:1048] T d4d26ba9fb084e748bddb93f7547621f P 263986885d354f7ba08739e419a6cded [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:58:06.121953   714 raft_consensus.cc:1275] T e6aabcfcdeb34e6c94ec0bb089deabda P 1bcd68d29a7b41a9bf114ac103d4c649 [term 2 FOLLOWER]: Refusing update from remote peer 263986885d354f7ba08739e419a6cded: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:58:06.124034   855 consensus_queue.cc:1048] T e6aabcfcdeb34e6c94ec0bb089deabda P 263986885d354f7ba08739e419a6cded [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:58:06.124737   714 raft_consensus.cc:1275] T 59112618a86046e1af899ce6d3c85203 P 1bcd68d29a7b41a9bf114ac103d4c649 [term 2 FOLLOWER]: Refusing update from remote peer 263986885d354f7ba08739e419a6cded: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:58:06.128088   855 consensus_queue.cc:1048] T 59112618a86046e1af899ce6d3c85203 P 263986885d354f7ba08739e419a6cded [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:58:06.146760   715 raft_consensus.cc:1275] T d4d26ba9fb084e748bddb93f7547621f P 1bcd68d29a7b41a9bf114ac103d4c649 [term 2 FOLLOWER]: Refusing update from remote peer 263986885d354f7ba08739e419a6cded: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:58:06.149140   855 consensus_queue.cc:1048] T d4d26ba9fb084e748bddb93f7547621f P 263986885d354f7ba08739e419a6cded [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:58:06.153723   563 raft_consensus.cc:1275] T 59112618a86046e1af899ce6d3c85203 P b5e46fc5aac9426a945bb34207d634d4 [term 2 FOLLOWER]: Refusing update from remote peer 263986885d354f7ba08739e419a6cded: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:58:06.157750   855 consensus_queue.cc:1048] T 59112618a86046e1af899ce6d3c85203 P 263986885d354f7ba08739e419a6cded [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:58:06.305220   715 raft_consensus.cc:1275] T 2e14183cc6064b1cbe6530eb1c8c899c P 1bcd68d29a7b41a9bf114ac103d4c649 [term 2 FOLLOWER]: Refusing update from remote peer b5e46fc5aac9426a945bb34207d634d4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:58:06.306641   860 consensus_queue.cc:1048] T 2e14183cc6064b1cbe6530eb1c8c899c P b5e46fc5aac9426a945bb34207d634d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:58:06.309798   714 raft_consensus.cc:1275] T 793bafcf38ca4ffb9a9aac9374ea2094 P 1bcd68d29a7b41a9bf114ac103d4c649 [term 2 FOLLOWER]: Refusing update from remote peer b5e46fc5aac9426a945bb34207d634d4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:58:06.311321   860 consensus_queue.cc:1048] T 793bafcf38ca4ffb9a9aac9374ea2094 P b5e46fc5aac9426a945bb34207d634d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:58:06.318904   641 raft_consensus.cc:1275] T 2e14183cc6064b1cbe6530eb1c8c899c P 263986885d354f7ba08739e419a6cded [term 2 FOLLOWER]: Refusing update from remote peer b5e46fc5aac9426a945bb34207d634d4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:58:06.321044   849 consensus_queue.cc:1048] T 2e14183cc6064b1cbe6530eb1c8c899c P b5e46fc5aac9426a945bb34207d634d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:58:06.329216   715 raft_consensus.cc:1275] T 61c59a4c19754dedb9d385e2ecfe145b P 1bcd68d29a7b41a9bf114ac103d4c649 [term 2 FOLLOWER]: Refusing update from remote peer 263986885d354f7ba08739e419a6cded: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:58:06.330989   855 consensus_queue.cc:1048] T 61c59a4c19754dedb9d385e2ecfe145b P 263986885d354f7ba08739e419a6cded [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:58:06.342305   641 raft_consensus.cc:1275] T 793bafcf38ca4ffb9a9aac9374ea2094 P 263986885d354f7ba08739e419a6cded [term 2 FOLLOWER]: Refusing update from remote peer b5e46fc5aac9426a945bb34207d634d4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:58:06.345806   563 raft_consensus.cc:1275] T 61c59a4c19754dedb9d385e2ecfe145b P b5e46fc5aac9426a945bb34207d634d4 [term 2 FOLLOWER]: Refusing update from remote peer 263986885d354f7ba08739e419a6cded: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:58:06.353549   849 consensus_queue.cc:1048] T 793bafcf38ca4ffb9a9aac9374ea2094 P b5e46fc5aac9426a945bb34207d634d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:58:06.353997   855 consensus_queue.cc:1048] T 61c59a4c19754dedb9d385e2ecfe145b P 263986885d354f7ba08739e419a6cded [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:58:08.541651   431 auto_leader_rebalancer-test.cc:316] The leader distribution is 
I20260709 12:58:08.543226   431 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:58:08.571051   640 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "59112618a86046e1af899ce6d3c85203"
dest_uuid: "263986885d354f7ba08739e419a6cded"
mode: GRACEFUL
new_leader_uuid: "1bcd68d29a7b41a9bf114ac103d4c649"
 from {username='slave'} at 127.0.0.1:60340
I20260709 12:58:08.572135   640 raft_consensus.cc:606] T 59112618a86046e1af899ce6d3c85203 P 263986885d354f7ba08739e419a6cded [term 2 LEADER]: Received request to transfer leadership to TS 1bcd68d29a7b41a9bf114ac103d4c649
I20260709 12:58:08.588555   563 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "d8b33c9315094794a876a4b62755efaf"
dest_uuid: "b5e46fc5aac9426a945bb34207d634d4"
mode: GRACEFUL
new_leader_uuid: "1bcd68d29a7b41a9bf114ac103d4c649"
 from {username='slave'} at 127.0.0.1:45360
I20260709 12:58:08.589267   563 raft_consensus.cc:606] T d8b33c9315094794a876a4b62755efaf P b5e46fc5aac9426a945bb34207d634d4 [term 1 LEADER]: Received request to transfer leadership to TS 1bcd68d29a7b41a9bf114ac103d4c649
I20260709 12:58:08.590930   431 auto_leader_rebalancer.cc:452] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260709 12:58:08.591490   431 auto_leader_rebalancer.cc:527] All tables' leader rebalancing finished this round
I20260709 12:58:08.666815   873 raft_consensus.cc:993] T 59112618a86046e1af899ce6d3c85203 P 263986885d354f7ba08739e419a6cded: : Instructing follower 1bcd68d29a7b41a9bf114ac103d4c649 to start an election
I20260709 12:58:08.667819   882 raft_consensus.cc:1081] T 59112618a86046e1af899ce6d3c85203 P 263986885d354f7ba08739e419a6cded [term 2 LEADER]: Signalling peer 1bcd68d29a7b41a9bf114ac103d4c649 to start an election
I20260709 12:58:08.671530   715 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "59112618a86046e1af899ce6d3c85203"
dest_uuid: "1bcd68d29a7b41a9bf114ac103d4c649"
 from {username='slave'} at 127.0.0.1:56652
I20260709 12:58:08.672437   715 raft_consensus.cc:493] T 59112618a86046e1af899ce6d3c85203 P 1bcd68d29a7b41a9bf114ac103d4c649 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:58:08.672992   715 raft_consensus.cc:3060] T 59112618a86046e1af899ce6d3c85203 P 1bcd68d29a7b41a9bf114ac103d4c649 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:58:08.680559   715 raft_consensus.cc:515] T 59112618a86046e1af899ce6d3c85203 P 1bcd68d29a7b41a9bf114ac103d4c649 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } }
I20260709 12:58:08.682897   715 leader_election.cc:290] T 59112618a86046e1af899ce6d3c85203 P 1bcd68d29a7b41a9bf114ac103d4c649 [CANDIDATE]: Term 3 election: Requested vote from peers b5e46fc5aac9426a945bb34207d634d4 (127.0.107.193:38425), 263986885d354f7ba08739e419a6cded (127.0.107.194:45491)
I20260709 12:58:08.684232   563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59112618a86046e1af899ce6d3c85203" candidate_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "b5e46fc5aac9426a945bb34207d634d4"
I20260709 12:58:08.685297   563 raft_consensus.cc:3060] T 59112618a86046e1af899ce6d3c85203 P b5e46fc5aac9426a945bb34207d634d4 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:58:08.686510   640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59112618a86046e1af899ce6d3c85203" candidate_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "263986885d354f7ba08739e419a6cded"
I20260709 12:58:08.687168   640 raft_consensus.cc:3055] T 59112618a86046e1af899ce6d3c85203 P 263986885d354f7ba08739e419a6cded [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:58:08.687500   640 raft_consensus.cc:740] T 59112618a86046e1af899ce6d3c85203 P 263986885d354f7ba08739e419a6cded [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 263986885d354f7ba08739e419a6cded, State: Running, Role: LEADER
I20260709 12:58:08.688349   640 consensus_queue.cc:260] T 59112618a86046e1af899ce6d3c85203 P 263986885d354f7ba08739e419a6cded [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } }
I20260709 12:58:08.689601   640 raft_consensus.cc:3060] T 59112618a86046e1af899ce6d3c85203 P 263986885d354f7ba08739e419a6cded [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:58:08.697299   563 raft_consensus.cc:2468] T 59112618a86046e1af899ce6d3c85203 P b5e46fc5aac9426a945bb34207d634d4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1bcd68d29a7b41a9bf114ac103d4c649 in term 3.
I20260709 12:58:08.697842   640 raft_consensus.cc:2468] T 59112618a86046e1af899ce6d3c85203 P 263986885d354f7ba08739e419a6cded [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1bcd68d29a7b41a9bf114ac103d4c649 in term 3.
I20260709 12:58:08.699419   681 leader_election.cc:304] T 59112618a86046e1af899ce6d3c85203 P 1bcd68d29a7b41a9bf114ac103d4c649 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bcd68d29a7b41a9bf114ac103d4c649, b5e46fc5aac9426a945bb34207d634d4; no voters: 
I20260709 12:58:08.700663   766 raft_consensus.cc:2804] T 59112618a86046e1af899ce6d3c85203 P 1bcd68d29a7b41a9bf114ac103d4c649 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:58:08.703410   766 raft_consensus.cc:697] T 59112618a86046e1af899ce6d3c85203 P 1bcd68d29a7b41a9bf114ac103d4c649 [term 3 LEADER]: Becoming Leader. State: Replica: 1bcd68d29a7b41a9bf114ac103d4c649, State: Running, Role: LEADER
I20260709 12:58:08.704897   766 consensus_queue.cc:237] T 59112618a86046e1af899ce6d3c85203 P 1bcd68d29a7b41a9bf114ac103d4c649 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } }
I20260709 12:58:08.716104   461 catalog_manager.cc:5697] T 59112618a86046e1af899ce6d3c85203 P 1bcd68d29a7b41a9bf114ac103d4c649 reported cstate change: term changed from 2 to 3, leader changed from 263986885d354f7ba08739e419a6cded (127.0.107.194) to 1bcd68d29a7b41a9bf114ac103d4c649 (127.0.107.195). New cstate: current_term: 3 leader_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } health_report { overall_health: HEALTHY } } }
I20260709 12:58:08.947671   876 raft_consensus.cc:993] T d8b33c9315094794a876a4b62755efaf P b5e46fc5aac9426a945bb34207d634d4: : Instructing follower 1bcd68d29a7b41a9bf114ac103d4c649 to start an election
I20260709 12:58:08.948608   891 raft_consensus.cc:1081] T d8b33c9315094794a876a4b62755efaf P b5e46fc5aac9426a945bb34207d634d4 [term 1 LEADER]: Signalling peer 1bcd68d29a7b41a9bf114ac103d4c649 to start an election
I20260709 12:58:08.950815   715 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "d8b33c9315094794a876a4b62755efaf"
dest_uuid: "1bcd68d29a7b41a9bf114ac103d4c649"
 from {username='slave'} at 127.0.0.1:56638
I20260709 12:58:08.951591   715 raft_consensus.cc:493] T d8b33c9315094794a876a4b62755efaf P 1bcd68d29a7b41a9bf114ac103d4c649 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:58:08.952018   715 raft_consensus.cc:3060] T d8b33c9315094794a876a4b62755efaf P 1bcd68d29a7b41a9bf114ac103d4c649 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:58:08.958815   715 raft_consensus.cc:515] T d8b33c9315094794a876a4b62755efaf P 1bcd68d29a7b41a9bf114ac103d4c649 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } }
I20260709 12:58:08.961114   715 leader_election.cc:290] T d8b33c9315094794a876a4b62755efaf P 1bcd68d29a7b41a9bf114ac103d4c649 [CANDIDATE]: Term 2 election: Requested vote from peers b5e46fc5aac9426a945bb34207d634d4 (127.0.107.193:38425), 263986885d354f7ba08739e419a6cded (127.0.107.194:45491)
I20260709 12:58:08.962631   641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d8b33c9315094794a876a4b62755efaf" candidate_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "263986885d354f7ba08739e419a6cded"
I20260709 12:58:08.962457   565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d8b33c9315094794a876a4b62755efaf" candidate_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "b5e46fc5aac9426a945bb34207d634d4"
I20260709 12:58:08.963575   641 raft_consensus.cc:3060] T d8b33c9315094794a876a4b62755efaf P 263986885d354f7ba08739e419a6cded [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:58:08.963610   565 raft_consensus.cc:3055] T d8b33c9315094794a876a4b62755efaf P b5e46fc5aac9426a945bb34207d634d4 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:58:08.964058   565 raft_consensus.cc:740] T d8b33c9315094794a876a4b62755efaf P b5e46fc5aac9426a945bb34207d634d4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b5e46fc5aac9426a945bb34207d634d4, State: Running, Role: LEADER
I20260709 12:58:08.965071   565 consensus_queue.cc:260] T d8b33c9315094794a876a4b62755efaf P b5e46fc5aac9426a945bb34207d634d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } }
I20260709 12:58:08.966835   565 raft_consensus.cc:3060] T d8b33c9315094794a876a4b62755efaf P b5e46fc5aac9426a945bb34207d634d4 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:58:08.970300   641 raft_consensus.cc:2468] T d8b33c9315094794a876a4b62755efaf P 263986885d354f7ba08739e419a6cded [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1bcd68d29a7b41a9bf114ac103d4c649 in term 2.
I20260709 12:58:08.971748   680 leader_election.cc:304] T d8b33c9315094794a876a4b62755efaf P 1bcd68d29a7b41a9bf114ac103d4c649 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bcd68d29a7b41a9bf114ac103d4c649, 263986885d354f7ba08739e419a6cded; no voters: 
I20260709 12:58:08.972678   766 raft_consensus.cc:2804] T d8b33c9315094794a876a4b62755efaf P 1bcd68d29a7b41a9bf114ac103d4c649 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:58:08.973132   766 raft_consensus.cc:697] T d8b33c9315094794a876a4b62755efaf P 1bcd68d29a7b41a9bf114ac103d4c649 [term 2 LEADER]: Becoming Leader. State: Replica: 1bcd68d29a7b41a9bf114ac103d4c649, State: Running, Role: LEADER
I20260709 12:58:08.974306   766 consensus_queue.cc:237] T d8b33c9315094794a876a4b62755efaf P 1bcd68d29a7b41a9bf114ac103d4c649 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } }
I20260709 12:58:08.975528   565 raft_consensus.cc:2468] T d8b33c9315094794a876a4b62755efaf P b5e46fc5aac9426a945bb34207d634d4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1bcd68d29a7b41a9bf114ac103d4c649 in term 2.
I20260709 12:58:08.986334   462 catalog_manager.cc:5697] T d8b33c9315094794a876a4b62755efaf P 1bcd68d29a7b41a9bf114ac103d4c649 reported cstate change: term changed from 1 to 2, leader changed from b5e46fc5aac9426a945bb34207d634d4 (127.0.107.193) to 1bcd68d29a7b41a9bf114ac103d4c649 (127.0.107.195). New cstate: current_term: 2 leader_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } health_report { overall_health: HEALTHY } } }
I20260709 12:58:09.093982   641 raft_consensus.cc:1275] T 59112618a86046e1af899ce6d3c85203 P 263986885d354f7ba08739e419a6cded [term 3 FOLLOWER]: Refusing update from remote peer 1bcd68d29a7b41a9bf114ac103d4c649: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:58:09.095484   766 consensus_queue.cc:1048] T 59112618a86046e1af899ce6d3c85203 P 1bcd68d29a7b41a9bf114ac103d4c649 [LEADER]: Connected to new peer: Peer: permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:58:09.110277   565 raft_consensus.cc:1275] T 59112618a86046e1af899ce6d3c85203 P b5e46fc5aac9426a945bb34207d634d4 [term 3 FOLLOWER]: Refusing update from remote peer 1bcd68d29a7b41a9bf114ac103d4c649: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:58:09.113962   766 consensus_queue.cc:1048] T 59112618a86046e1af899ce6d3c85203 P 1bcd68d29a7b41a9bf114ac103d4c649 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:58:09.427002   565 raft_consensus.cc:1275] T d8b33c9315094794a876a4b62755efaf P b5e46fc5aac9426a945bb34207d634d4 [term 2 FOLLOWER]: Refusing update from remote peer 1bcd68d29a7b41a9bf114ac103d4c649: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:58:09.428380   896 consensus_queue.cc:1048] T d8b33c9315094794a876a4b62755efaf P 1bcd68d29a7b41a9bf114ac103d4c649 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:58:09.438870   641 raft_consensus.cc:1275] T d8b33c9315094794a876a4b62755efaf P 263986885d354f7ba08739e419a6cded [term 2 FOLLOWER]: Refusing update from remote peer 1bcd68d29a7b41a9bf114ac103d4c649: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:58:09.444355   884 consensus_queue.cc:1048] T d8b33c9315094794a876a4b62755efaf P 1bcd68d29a7b41a9bf114ac103d4c649 [LEADER]: Connected to new peer: Peer: permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:58:09.592746   431 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:58:09.604146   565 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "793bafcf38ca4ffb9a9aac9374ea2094"
dest_uuid: "b5e46fc5aac9426a945bb34207d634d4"
mode: GRACEFUL
new_leader_uuid: "263986885d354f7ba08739e419a6cded"
 from {username='slave'} at 127.0.0.1:45358
I20260709 12:58:09.604928   565 raft_consensus.cc:606] T 793bafcf38ca4ffb9a9aac9374ea2094 P b5e46fc5aac9426a945bb34207d634d4 [term 2 LEADER]: Received request to transfer leadership to TS 263986885d354f7ba08739e419a6cded
I20260709 12:58:09.610167   565 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "2e14183cc6064b1cbe6530eb1c8c899c"
dest_uuid: "b5e46fc5aac9426a945bb34207d634d4"
mode: GRACEFUL
new_leader_uuid: "263986885d354f7ba08739e419a6cded"
 from {username='slave'} at 127.0.0.1:45358
I20260709 12:58:09.610767   565 raft_consensus.cc:606] T 2e14183cc6064b1cbe6530eb1c8c899c P b5e46fc5aac9426a945bb34207d634d4 [term 2 LEADER]: Received request to transfer leadership to TS 263986885d354f7ba08739e419a6cded
I20260709 12:58:09.615423   565 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "d441098f471f4b0084938676c20981e5"
dest_uuid: "b5e46fc5aac9426a945bb34207d634d4"
mode: GRACEFUL
new_leader_uuid: "263986885d354f7ba08739e419a6cded"
 from {username='slave'} at 127.0.0.1:45358
I20260709 12:58:09.615924   565 raft_consensus.cc:606] T d441098f471f4b0084938676c20981e5 P b5e46fc5aac9426a945bb34207d634d4 [term 2 LEADER]: Received request to transfer leadership to TS 263986885d354f7ba08739e419a6cded
I20260709 12:58:09.620389   715 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "d8b33c9315094794a876a4b62755efaf"
dest_uuid: "1bcd68d29a7b41a9bf114ac103d4c649"
mode: GRACEFUL
new_leader_uuid: "263986885d354f7ba08739e419a6cded"
 from {username='slave'} at 127.0.0.1:56664
I20260709 12:58:09.620934   715 raft_consensus.cc:606] T d8b33c9315094794a876a4b62755efaf P 1bcd68d29a7b41a9bf114ac103d4c649 [term 2 LEADER]: Received request to transfer leadership to TS 263986885d354f7ba08739e419a6cded
I20260709 12:58:09.625459   715 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "59112618a86046e1af899ce6d3c85203"
dest_uuid: "1bcd68d29a7b41a9bf114ac103d4c649"
mode: GRACEFUL
new_leader_uuid: "263986885d354f7ba08739e419a6cded"
 from {username='slave'} at 127.0.0.1:56664
I20260709 12:58:09.625965   715 raft_consensus.cc:606] T 59112618a86046e1af899ce6d3c85203 P 1bcd68d29a7b41a9bf114ac103d4c649 [term 3 LEADER]: Received request to transfer leadership to TS 263986885d354f7ba08739e419a6cded
I20260709 12:58:09.642413   884 raft_consensus.cc:993] T 59112618a86046e1af899ce6d3c85203 P 1bcd68d29a7b41a9bf114ac103d4c649: : Instructing follower 263986885d354f7ba08739e419a6cded to start an election
I20260709 12:58:09.642879   766 raft_consensus.cc:1081] T 59112618a86046e1af899ce6d3c85203 P 1bcd68d29a7b41a9bf114ac103d4c649 [term 3 LEADER]: Signalling peer 263986885d354f7ba08739e419a6cded to start an election
I20260709 12:58:09.644275   641 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "59112618a86046e1af899ce6d3c85203"
dest_uuid: "263986885d354f7ba08739e419a6cded"
 from {username='slave'} at 127.0.0.1:60330
I20260709 12:58:09.644693   641 raft_consensus.cc:493] T 59112618a86046e1af899ce6d3c85203 P 263986885d354f7ba08739e419a6cded [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:58:09.644953   641 raft_consensus.cc:3060] T 59112618a86046e1af899ce6d3c85203 P 263986885d354f7ba08739e419a6cded [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:58:09.650234   641 raft_consensus.cc:515] T 59112618a86046e1af899ce6d3c85203 P 263986885d354f7ba08739e419a6cded [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } }
I20260709 12:58:09.652513   641 leader_election.cc:290] T 59112618a86046e1af899ce6d3c85203 P 263986885d354f7ba08739e419a6cded [CANDIDATE]: Term 4 election: Requested vote from peers b5e46fc5aac9426a945bb34207d634d4 (127.0.107.193:38425), 1bcd68d29a7b41a9bf114ac103d4c649 (127.0.107.195:46365)
I20260709 12:58:09.653517   565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59112618a86046e1af899ce6d3c85203" candidate_uuid: "263986885d354f7ba08739e419a6cded" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "b5e46fc5aac9426a945bb34207d634d4"
I20260709 12:58:09.653762   715 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59112618a86046e1af899ce6d3c85203" candidate_uuid: "263986885d354f7ba08739e419a6cded" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "1bcd68d29a7b41a9bf114ac103d4c649"
I20260709 12:58:09.654173   565 raft_consensus.cc:3060] T 59112618a86046e1af899ce6d3c85203 P b5e46fc5aac9426a945bb34207d634d4 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:58:09.654314   715 raft_consensus.cc:3055] T 59112618a86046e1af899ce6d3c85203 P 1bcd68d29a7b41a9bf114ac103d4c649 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:58:09.654784   715 raft_consensus.cc:740] T 59112618a86046e1af899ce6d3c85203 P 1bcd68d29a7b41a9bf114ac103d4c649 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1bcd68d29a7b41a9bf114ac103d4c649, State: Running, Role: LEADER
I20260709 12:58:09.655696   715 consensus_queue.cc:260] T 59112618a86046e1af899ce6d3c85203 P 1bcd68d29a7b41a9bf114ac103d4c649 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } }
I20260709 12:58:09.657078   715 raft_consensus.cc:3060] T 59112618a86046e1af899ce6d3c85203 P 1bcd68d29a7b41a9bf114ac103d4c649 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:58:09.659349   891 raft_consensus.cc:993] T 793bafcf38ca4ffb9a9aac9374ea2094 P b5e46fc5aac9426a945bb34207d634d4: : Instructing follower 263986885d354f7ba08739e419a6cded to start an election
I20260709 12:58:09.659787   876 raft_consensus.cc:1081] T 793bafcf38ca4ffb9a9aac9374ea2094 P b5e46fc5aac9426a945bb34207d634d4 [term 2 LEADER]: Signalling peer 263986885d354f7ba08739e419a6cded to start an election
I20260709 12:58:09.661250   641 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "793bafcf38ca4ffb9a9aac9374ea2094"
dest_uuid: "263986885d354f7ba08739e419a6cded"
 from {username='slave'} at 127.0.0.1:60338
I20260709 12:58:09.661708   641 raft_consensus.cc:493] T 793bafcf38ca4ffb9a9aac9374ea2094 P 263986885d354f7ba08739e419a6cded [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:58:09.661999   641 raft_consensus.cc:3060] T 793bafcf38ca4ffb9a9aac9374ea2094 P 263986885d354f7ba08739e419a6cded [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:58:09.663106   565 raft_consensus.cc:2468] T 59112618a86046e1af899ce6d3c85203 P b5e46fc5aac9426a945bb34207d634d4 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 263986885d354f7ba08739e419a6cded in term 4.
I20260709 12:58:09.664379   607 leader_election.cc:304] T 59112618a86046e1af899ce6d3c85203 P 263986885d354f7ba08739e419a6cded [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 263986885d354f7ba08739e419a6cded, b5e46fc5aac9426a945bb34207d634d4; no voters: 
I20260709 12:58:09.664892   715 raft_consensus.cc:2468] T 59112618a86046e1af899ce6d3c85203 P 1bcd68d29a7b41a9bf114ac103d4c649 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 263986885d354f7ba08739e419a6cded in term 4.
I20260709 12:58:09.665395   899 raft_consensus.cc:2804] T 59112618a86046e1af899ce6d3c85203 P 263986885d354f7ba08739e419a6cded [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:58:09.665923   899 raft_consensus.cc:697] T 59112618a86046e1af899ce6d3c85203 P 263986885d354f7ba08739e419a6cded [term 4 LEADER]: Becoming Leader. State: Replica: 263986885d354f7ba08739e419a6cded, State: Running, Role: LEADER
I20260709 12:58:09.667019   899 consensus_queue.cc:237] T 59112618a86046e1af899ce6d3c85203 P 263986885d354f7ba08739e419a6cded [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } }
I20260709 12:58:09.668035   641 raft_consensus.cc:515] T 793bafcf38ca4ffb9a9aac9374ea2094 P 263986885d354f7ba08739e419a6cded [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } } peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } }
I20260709 12:58:09.670145   641 leader_election.cc:290] T 793bafcf38ca4ffb9a9aac9374ea2094 P 263986885d354f7ba08739e419a6cded [CANDIDATE]: Term 3 election: Requested vote from peers 1bcd68d29a7b41a9bf114ac103d4c649 (127.0.107.195:46365), b5e46fc5aac9426a945bb34207d634d4 (127.0.107.193:38425)
I20260709 12:58:09.671479   715 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "793bafcf38ca4ffb9a9aac9374ea2094" candidate_uuid: "263986885d354f7ba08739e419a6cded" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "1bcd68d29a7b41a9bf114ac103d4c649"
I20260709 12:58:09.672075   715 raft_consensus.cc:3060] T 793bafcf38ca4ffb9a9aac9374ea2094 P 1bcd68d29a7b41a9bf114ac103d4c649 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:58:09.672122   565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "793bafcf38ca4ffb9a9aac9374ea2094" candidate_uuid: "263986885d354f7ba08739e419a6cded" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "b5e46fc5aac9426a945bb34207d634d4"
I20260709 12:58:09.672777   565 raft_consensus.cc:3055] T 793bafcf38ca4ffb9a9aac9374ea2094 P b5e46fc5aac9426a945bb34207d634d4 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:58:09.673126   565 raft_consensus.cc:740] T 793bafcf38ca4ffb9a9aac9374ea2094 P b5e46fc5aac9426a945bb34207d634d4 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: b5e46fc5aac9426a945bb34207d634d4, State: Running, Role: LEADER
I20260709 12:58:09.674053   565 consensus_queue.cc:260] T 793bafcf38ca4ffb9a9aac9374ea2094 P b5e46fc5aac9426a945bb34207d634d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } } peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } }
I20260709 12:58:09.674309   462 catalog_manager.cc:5697] T 59112618a86046e1af899ce6d3c85203 P 263986885d354f7ba08739e419a6cded reported cstate change: term changed from 3 to 4, leader changed from 1bcd68d29a7b41a9bf114ac103d4c649 (127.0.107.195) to 263986885d354f7ba08739e419a6cded (127.0.107.194). New cstate: current_term: 4 leader_uuid: "263986885d354f7ba08739e419a6cded" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } health_report { overall_health: UNKNOWN } } }
I20260709 12:58:09.675526   565 raft_consensus.cc:3060] T 793bafcf38ca4ffb9a9aac9374ea2094 P b5e46fc5aac9426a945bb34207d634d4 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:58:09.678920   715 raft_consensus.cc:2468] T 793bafcf38ca4ffb9a9aac9374ea2094 P 1bcd68d29a7b41a9bf114ac103d4c649 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 263986885d354f7ba08739e419a6cded in term 3.
I20260709 12:58:09.680320   604 leader_election.cc:304] T 793bafcf38ca4ffb9a9aac9374ea2094 P 263986885d354f7ba08739e419a6cded [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bcd68d29a7b41a9bf114ac103d4c649, 263986885d354f7ba08739e419a6cded; no voters: 
I20260709 12:58:09.681186   899 raft_consensus.cc:2804] T 793bafcf38ca4ffb9a9aac9374ea2094 P 263986885d354f7ba08739e419a6cded [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:58:09.681697   899 raft_consensus.cc:697] T 793bafcf38ca4ffb9a9aac9374ea2094 P 263986885d354f7ba08739e419a6cded [term 3 LEADER]: Becoming Leader. State: Replica: 263986885d354f7ba08739e419a6cded, State: Running, Role: LEADER
I20260709 12:58:09.682401   565 raft_consensus.cc:2468] T 793bafcf38ca4ffb9a9aac9374ea2094 P b5e46fc5aac9426a945bb34207d634d4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 263986885d354f7ba08739e419a6cded in term 3.
I20260709 12:58:09.682687   899 consensus_queue.cc:237] T 793bafcf38ca4ffb9a9aac9374ea2094 P 263986885d354f7ba08739e419a6cded [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } } peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } }
I20260709 12:58:09.691625   462 catalog_manager.cc:5697] T 793bafcf38ca4ffb9a9aac9374ea2094 P 263986885d354f7ba08739e419a6cded reported cstate change: term changed from 2 to 3, leader changed from b5e46fc5aac9426a945bb34207d634d4 (127.0.107.193) to 263986885d354f7ba08739e419a6cded (127.0.107.194). New cstate: current_term: 3 leader_uuid: "263986885d354f7ba08739e419a6cded" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } health_report { overall_health: UNKNOWN } } }
I20260709 12:58:09.834355   891 raft_consensus.cc:993] T 2e14183cc6064b1cbe6530eb1c8c899c P b5e46fc5aac9426a945bb34207d634d4: : Instructing follower 263986885d354f7ba08739e419a6cded to start an election
I20260709 12:58:09.834860   876 raft_consensus.cc:1081] T 2e14183cc6064b1cbe6530eb1c8c899c P b5e46fc5aac9426a945bb34207d634d4 [term 2 LEADER]: Signalling peer 263986885d354f7ba08739e419a6cded to start an election
I20260709 12:58:09.836298   641 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "2e14183cc6064b1cbe6530eb1c8c899c"
dest_uuid: "263986885d354f7ba08739e419a6cded"
 from {username='slave'} at 127.0.0.1:60338
I20260709 12:58:09.836760   641 raft_consensus.cc:493] T 2e14183cc6064b1cbe6530eb1c8c899c P 263986885d354f7ba08739e419a6cded [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:58:09.837013   641 raft_consensus.cc:3060] T 2e14183cc6064b1cbe6530eb1c8c899c P 263986885d354f7ba08739e419a6cded [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:58:09.842182   641 raft_consensus.cc:515] T 2e14183cc6064b1cbe6530eb1c8c899c P 263986885d354f7ba08739e419a6cded [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } }
I20260709 12:58:09.843840   641 leader_election.cc:290] T 2e14183cc6064b1cbe6530eb1c8c899c P 263986885d354f7ba08739e419a6cded [CANDIDATE]: Term 3 election: Requested vote from peers b5e46fc5aac9426a945bb34207d634d4 (127.0.107.193:38425), 1bcd68d29a7b41a9bf114ac103d4c649 (127.0.107.195:46365)
I20260709 12:58:09.844923   715 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e14183cc6064b1cbe6530eb1c8c899c" candidate_uuid: "263986885d354f7ba08739e419a6cded" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "1bcd68d29a7b41a9bf114ac103d4c649"
I20260709 12:58:09.844955   565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e14183cc6064b1cbe6530eb1c8c899c" candidate_uuid: "263986885d354f7ba08739e419a6cded" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "b5e46fc5aac9426a945bb34207d634d4"
I20260709 12:58:09.845633   715 raft_consensus.cc:3060] T 2e14183cc6064b1cbe6530eb1c8c899c P 1bcd68d29a7b41a9bf114ac103d4c649 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:58:09.845727   565 raft_consensus.cc:3055] T 2e14183cc6064b1cbe6530eb1c8c899c P b5e46fc5aac9426a945bb34207d634d4 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:58:09.846082   565 raft_consensus.cc:740] T 2e14183cc6064b1cbe6530eb1c8c899c P b5e46fc5aac9426a945bb34207d634d4 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: b5e46fc5aac9426a945bb34207d634d4, State: Running, Role: LEADER
I20260709 12:58:09.846896   565 consensus_queue.cc:260] T 2e14183cc6064b1cbe6530eb1c8c899c P b5e46fc5aac9426a945bb34207d634d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } }
I20260709 12:58:09.848075   565 raft_consensus.cc:3060] T 2e14183cc6064b1cbe6530eb1c8c899c P b5e46fc5aac9426a945bb34207d634d4 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:58:09.850054   715 raft_consensus.cc:2468] T 2e14183cc6064b1cbe6530eb1c8c899c P 1bcd68d29a7b41a9bf114ac103d4c649 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 263986885d354f7ba08739e419a6cded in term 3.
I20260709 12:58:09.851083   604 leader_election.cc:304] T 2e14183cc6064b1cbe6530eb1c8c899c P 263986885d354f7ba08739e419a6cded [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bcd68d29a7b41a9bf114ac103d4c649, 263986885d354f7ba08739e419a6cded; no voters: 
I20260709 12:58:09.851799   899 raft_consensus.cc:2804] T 2e14183cc6064b1cbe6530eb1c8c899c P 263986885d354f7ba08739e419a6cded [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:58:09.852234   899 raft_consensus.cc:697] T 2e14183cc6064b1cbe6530eb1c8c899c P 263986885d354f7ba08739e419a6cded [term 3 LEADER]: Becoming Leader. State: Replica: 263986885d354f7ba08739e419a6cded, State: Running, Role: LEADER
I20260709 12:58:09.853195   899 consensus_queue.cc:237] T 2e14183cc6064b1cbe6530eb1c8c899c P 263986885d354f7ba08739e419a6cded [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } }
I20260709 12:58:09.855160   565 raft_consensus.cc:2468] T 2e14183cc6064b1cbe6530eb1c8c899c P b5e46fc5aac9426a945bb34207d634d4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 263986885d354f7ba08739e419a6cded in term 3.
I20260709 12:58:09.860991   462 catalog_manager.cc:5697] T 2e14183cc6064b1cbe6530eb1c8c899c P 263986885d354f7ba08739e419a6cded reported cstate change: term changed from 2 to 3, leader changed from b5e46fc5aac9426a945bb34207d634d4 (127.0.107.193) to 263986885d354f7ba08739e419a6cded (127.0.107.194). New cstate: current_term: 3 leader_uuid: "263986885d354f7ba08739e419a6cded" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } health_report { overall_health: UNKNOWN } } }
I20260709 12:58:10.036190   884 raft_consensus.cc:993] T d8b33c9315094794a876a4b62755efaf P 1bcd68d29a7b41a9bf114ac103d4c649: : Instructing follower 263986885d354f7ba08739e419a6cded to start an election
I20260709 12:58:10.036831   884 raft_consensus.cc:1081] T d8b33c9315094794a876a4b62755efaf P 1bcd68d29a7b41a9bf114ac103d4c649 [term 2 LEADER]: Signalling peer 263986885d354f7ba08739e419a6cded to start an election
I20260709 12:58:10.121894   641 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "d8b33c9315094794a876a4b62755efaf"
dest_uuid: "263986885d354f7ba08739e419a6cded"
 from {username='slave'} at 127.0.0.1:60330
I20260709 12:58:10.123035   641 raft_consensus.cc:493] T d8b33c9315094794a876a4b62755efaf P 263986885d354f7ba08739e419a6cded [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:58:10.123667   641 raft_consensus.cc:3060] T d8b33c9315094794a876a4b62755efaf P 263986885d354f7ba08739e419a6cded [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:58:10.174085   713 raft_consensus.cc:1275] T 793bafcf38ca4ffb9a9aac9374ea2094 P 1bcd68d29a7b41a9bf114ac103d4c649 [term 3 FOLLOWER]: Refusing update from remote peer 263986885d354f7ba08739e419a6cded: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:58:10.208243   881 consensus_queue.cc:1048] T 793bafcf38ca4ffb9a9aac9374ea2094 P 263986885d354f7ba08739e419a6cded [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:58:10.239848   715 raft_consensus.cc:1275] T 59112618a86046e1af899ce6d3c85203 P 1bcd68d29a7b41a9bf114ac103d4c649 [term 4 FOLLOWER]: Refusing update from remote peer 263986885d354f7ba08739e419a6cded: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:58:10.253702   899 consensus_queue.cc:1048] T 59112618a86046e1af899ce6d3c85203 P 263986885d354f7ba08739e419a6cded [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:58:10.263446   641 raft_consensus.cc:515] T d8b33c9315094794a876a4b62755efaf P 263986885d354f7ba08739e419a6cded [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } }
I20260709 12:58:10.265940   641 leader_election.cc:290] T d8b33c9315094794a876a4b62755efaf P 263986885d354f7ba08739e419a6cded [CANDIDATE]: Term 3 election: Requested vote from peers b5e46fc5aac9426a945bb34207d634d4 (127.0.107.193:38425), 1bcd68d29a7b41a9bf114ac103d4c649 (127.0.107.195:46365)
I20260709 12:58:10.267743   714 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d8b33c9315094794a876a4b62755efaf" candidate_uuid: "263986885d354f7ba08739e419a6cded" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "1bcd68d29a7b41a9bf114ac103d4c649"
I20260709 12:58:10.268527   714 raft_consensus.cc:3055] T d8b33c9315094794a876a4b62755efaf P 1bcd68d29a7b41a9bf114ac103d4c649 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:58:10.268857   714 raft_consensus.cc:740] T d8b33c9315094794a876a4b62755efaf P 1bcd68d29a7b41a9bf114ac103d4c649 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1bcd68d29a7b41a9bf114ac103d4c649, State: Running, Role: LEADER
I20260709 12:58:10.269812   714 consensus_queue.cc:260] T d8b33c9315094794a876a4b62755efaf P 1bcd68d29a7b41a9bf114ac103d4c649 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } }
I20260709 12:58:10.272836   714 raft_consensus.cc:3060] T d8b33c9315094794a876a4b62755efaf P 1bcd68d29a7b41a9bf114ac103d4c649 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:58:10.283905   714 raft_consensus.cc:2468] T d8b33c9315094794a876a4b62755efaf P 1bcd68d29a7b41a9bf114ac103d4c649 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 263986885d354f7ba08739e419a6cded in term 3.
I20260709 12:58:10.285027   604 leader_election.cc:304] T d8b33c9315094794a876a4b62755efaf P 263986885d354f7ba08739e419a6cded [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bcd68d29a7b41a9bf114ac103d4c649, 263986885d354f7ba08739e419a6cded; no voters: 
I20260709 12:58:10.286226   911 raft_consensus.cc:2804] T d8b33c9315094794a876a4b62755efaf P 263986885d354f7ba08739e419a6cded [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:58:10.286796   911 raft_consensus.cc:697] T d8b33c9315094794a876a4b62755efaf P 263986885d354f7ba08739e419a6cded [term 3 LEADER]: Becoming Leader. State: Replica: 263986885d354f7ba08739e419a6cded, State: Running, Role: LEADER
I20260709 12:58:10.287774   911 consensus_queue.cc:237] T d8b33c9315094794a876a4b62755efaf P 263986885d354f7ba08739e419a6cded [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } }
I20260709 12:58:10.374085   714 raft_consensus.cc:1275] T 2e14183cc6064b1cbe6530eb1c8c899c P 1bcd68d29a7b41a9bf114ac103d4c649 [term 3 FOLLOWER]: Refusing update from remote peer 263986885d354f7ba08739e419a6cded: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
W20260709 12:58:10.444089   680 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:58:10.465220   911 consensus_queue.cc:1048] T 2e14183cc6064b1cbe6530eb1c8c899c P 263986885d354f7ba08739e419a6cded [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:58:10.602173   563 raft_consensus.cc:1275] T 59112618a86046e1af899ce6d3c85203 P b5e46fc5aac9426a945bb34207d634d4 [term 4 FOLLOWER]: Refusing update from remote peer 263986885d354f7ba08739e419a6cded: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:58:10.604174   562 raft_consensus.cc:1275] T 2e14183cc6064b1cbe6530eb1c8c899c P b5e46fc5aac9426a945bb34207d634d4 [term 3 FOLLOWER]: Refusing update from remote peer 263986885d354f7ba08739e419a6cded: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:58:10.606251   565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d8b33c9315094794a876a4b62755efaf" candidate_uuid: "263986885d354f7ba08739e419a6cded" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "b5e46fc5aac9426a945bb34207d634d4"
I20260709 12:58:10.607039   565 raft_consensus.cc:3060] T d8b33c9315094794a876a4b62755efaf P b5e46fc5aac9426a945bb34207d634d4 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:58:10.617856   565 raft_consensus.cc:2468] T d8b33c9315094794a876a4b62755efaf P b5e46fc5aac9426a945bb34207d634d4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 263986885d354f7ba08739e419a6cded in term 3.
I20260709 12:58:10.823722   564 raft_consensus.cc:1275] T 793bafcf38ca4ffb9a9aac9374ea2094 P b5e46fc5aac9426a945bb34207d634d4 [term 3 FOLLOWER]: Refusing update from remote peer 263986885d354f7ba08739e419a6cded: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:58:10.857509   914 raft_consensus.cc:993] T d441098f471f4b0084938676c20981e5 P b5e46fc5aac9426a945bb34207d634d4: : Instructing follower 263986885d354f7ba08739e419a6cded to start an election
I20260709 12:58:10.858119   914 raft_consensus.cc:1081] T d441098f471f4b0084938676c20981e5 P b5e46fc5aac9426a945bb34207d634d4 [term 2 LEADER]: Signalling peer 263986885d354f7ba08739e419a6cded to start an election
I20260709 12:58:10.859941   641 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "d441098f471f4b0084938676c20981e5"
dest_uuid: "263986885d354f7ba08739e419a6cded"
 from {username='slave'} at 127.0.0.1:60338
I20260709 12:58:10.860424   641 raft_consensus.cc:493] T d441098f471f4b0084938676c20981e5 P 263986885d354f7ba08739e419a6cded [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:58:10.860757   641 raft_consensus.cc:3060] T d441098f471f4b0084938676c20981e5 P 263986885d354f7ba08739e419a6cded [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:58:10.866899   641 raft_consensus.cc:515] T d441098f471f4b0084938676c20981e5 P 263986885d354f7ba08739e419a6cded [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } }
I20260709 12:58:10.870819   641 leader_election.cc:290] T d441098f471f4b0084938676c20981e5 P 263986885d354f7ba08739e419a6cded [CANDIDATE]: Term 3 election: Requested vote from peers b5e46fc5aac9426a945bb34207d634d4 (127.0.107.193:38425), 1bcd68d29a7b41a9bf114ac103d4c649 (127.0.107.195:46365)
I20260709 12:58:10.872272   713 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d441098f471f4b0084938676c20981e5" candidate_uuid: "263986885d354f7ba08739e419a6cded" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "1bcd68d29a7b41a9bf114ac103d4c649"
I20260709 12:58:10.872921   713 raft_consensus.cc:3060] T d441098f471f4b0084938676c20981e5 P 1bcd68d29a7b41a9bf114ac103d4c649 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:58:10.880847   713 raft_consensus.cc:2468] T d441098f471f4b0084938676c20981e5 P 1bcd68d29a7b41a9bf114ac103d4c649 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 263986885d354f7ba08739e419a6cded in term 3.
I20260709 12:58:10.882035   604 leader_election.cc:304] T d441098f471f4b0084938676c20981e5 P 263986885d354f7ba08739e419a6cded [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bcd68d29a7b41a9bf114ac103d4c649, 263986885d354f7ba08739e419a6cded; no voters: 
I20260709 12:58:10.882962   881 raft_consensus.cc:2804] T d441098f471f4b0084938676c20981e5 P 263986885d354f7ba08739e419a6cded [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:58:10.883769   881 raft_consensus.cc:697] T d441098f471f4b0084938676c20981e5 P 263986885d354f7ba08739e419a6cded [term 3 LEADER]: Becoming Leader. State: Replica: 263986885d354f7ba08739e419a6cded, State: Running, Role: LEADER
I20260709 12:58:10.884814   881 consensus_queue.cc:237] T d441098f471f4b0084938676c20981e5 P 263986885d354f7ba08739e419a6cded [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } }
I20260709 12:58:10.887547   714 raft_consensus.cc:1275] T d8b33c9315094794a876a4b62755efaf P 1bcd68d29a7b41a9bf114ac103d4c649 [term 3 FOLLOWER]: Refusing update from remote peer 263986885d354f7ba08739e419a6cded: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:58:10.889039   881 consensus_queue.cc:1048] T d8b33c9315094794a876a4b62755efaf P 263986885d354f7ba08739e419a6cded [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:58:10.892802   911 consensus_queue.cc:1048] T 59112618a86046e1af899ce6d3c85203 P 263986885d354f7ba08739e419a6cded [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:58:10.893203   564 raft_consensus.cc:1275] T d8b33c9315094794a876a4b62755efaf P b5e46fc5aac9426a945bb34207d634d4 [term 3 FOLLOWER]: Refusing update from remote peer 263986885d354f7ba08739e419a6cded: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:58:10.895687   565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d441098f471f4b0084938676c20981e5" candidate_uuid: "263986885d354f7ba08739e419a6cded" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "b5e46fc5aac9426a945bb34207d634d4"
I20260709 12:58:10.896284   565 raft_consensus.cc:3055] T d441098f471f4b0084938676c20981e5 P b5e46fc5aac9426a945bb34207d634d4 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:58:10.896621   565 raft_consensus.cc:740] T d441098f471f4b0084938676c20981e5 P b5e46fc5aac9426a945bb34207d634d4 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: b5e46fc5aac9426a945bb34207d634d4, State: Running, Role: LEADER
I20260709 12:58:10.897782   565 consensus_queue.cc:260] T d441098f471f4b0084938676c20981e5 P b5e46fc5aac9426a945bb34207d634d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } }
I20260709 12:58:10.898887   565 raft_consensus.cc:3060] T d441098f471f4b0084938676c20981e5 P b5e46fc5aac9426a945bb34207d634d4 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:58:10.907938   715 raft_consensus.cc:1240] T d441098f471f4b0084938676c20981e5 P 1bcd68d29a7b41a9bf114ac103d4c649 [term 3 FOLLOWER]: Rejecting Update request from peer b5e46fc5aac9426a945bb34207d634d4 for earlier term 2. Current term is 3. Ops: []
I20260709 12:58:10.914266   565 raft_consensus.cc:2468] T d441098f471f4b0084938676c20981e5 P b5e46fc5aac9426a945bb34207d634d4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 263986885d354f7ba08739e419a6cded in term 3.
I20260709 12:58:10.917024   899 consensus_queue.cc:1048] T 2e14183cc6064b1cbe6530eb1c8c899c P 263986885d354f7ba08739e419a6cded [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:58:10.917508   911 consensus_queue.cc:1048] T 793bafcf38ca4ffb9a9aac9374ea2094 P 263986885d354f7ba08739e419a6cded [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:58:10.917793   881 consensus_queue.cc:1048] T d8b33c9315094794a876a4b62755efaf P 263986885d354f7ba08739e419a6cded [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:58:10.915685   462 catalog_manager.cc:5697] T d8b33c9315094794a876a4b62755efaf P 263986885d354f7ba08739e419a6cded reported cstate change: term changed from 2 to 3, leader changed from 1bcd68d29a7b41a9bf114ac103d4c649 (127.0.107.195) to 263986885d354f7ba08739e419a6cded (127.0.107.194). New cstate: current_term: 3 leader_uuid: "263986885d354f7ba08739e419a6cded" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } health_report { overall_health: UNKNOWN } } }
I20260709 12:58:10.919480   462 catalog_manager.cc:5697] T d441098f471f4b0084938676c20981e5 P 263986885d354f7ba08739e419a6cded reported cstate change: term changed from 2 to 3, leader changed from b5e46fc5aac9426a945bb34207d634d4 (127.0.107.193) to 263986885d354f7ba08739e419a6cded (127.0.107.194). New cstate: current_term: 3 leader_uuid: "263986885d354f7ba08739e419a6cded" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } health_report { overall_health: UNKNOWN } } }
I20260709 12:58:11.339788   563 raft_consensus.cc:1275] T d441098f471f4b0084938676c20981e5 P b5e46fc5aac9426a945bb34207d634d4 [term 3 FOLLOWER]: Refusing update from remote peer 263986885d354f7ba08739e419a6cded: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:58:11.341301   916 consensus_queue.cc:1048] T d441098f471f4b0084938676c20981e5 P 263986885d354f7ba08739e419a6cded [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:58:11.353658   714 raft_consensus.cc:1275] T d441098f471f4b0084938676c20981e5 P 1bcd68d29a7b41a9bf114ac103d4c649 [term 3 FOLLOWER]: Refusing update from remote peer 263986885d354f7ba08739e419a6cded: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:58:11.355669   911 consensus_queue.cc:1048] T d441098f471f4b0084938676c20981e5 P 263986885d354f7ba08739e419a6cded [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20260709 12:58:11.595993   587 debug-util.cc:398] Leaking SignalData structure 0x7b0800066000 after lost signal to thread 434
W20260709 12:58:11.597237   587 debug-util.cc:398] Leaking SignalData structure 0x7b0800085ac0 after lost signal to thread 497
W20260709 12:58:11.598274   587 debug-util.cc:398] Leaking SignalData structure 0x7b08001e2360 after lost signal to thread 590
W20260709 12:58:11.599406   587 debug-util.cc:398] Leaking SignalData structure 0x7b0800178900 after lost signal to thread 665
W20260709 12:58:11.600346   587 debug-util.cc:398] Leaking SignalData structure 0x7b0800125b60 after lost signal to thread 739
I20260709 12:58:13.169206   431 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:58:13.178167   641 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "2e14183cc6064b1cbe6530eb1c8c899c"
dest_uuid: "263986885d354f7ba08739e419a6cded"
mode: GRACEFUL
new_leader_uuid: "b5e46fc5aac9426a945bb34207d634d4"
 from {username='slave'} at 127.0.0.1:60340
I20260709 12:58:13.178735   641 raft_consensus.cc:606] T 2e14183cc6064b1cbe6530eb1c8c899c P 263986885d354f7ba08739e419a6cded [term 3 LEADER]: Received request to transfer leadership to TS b5e46fc5aac9426a945bb34207d634d4
I20260709 12:58:13.181173   641 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "59112618a86046e1af899ce6d3c85203"
dest_uuid: "263986885d354f7ba08739e419a6cded"
mode: GRACEFUL
new_leader_uuid: "b5e46fc5aac9426a945bb34207d634d4"
 from {username='slave'} at 127.0.0.1:60340
I20260709 12:58:13.181607   641 raft_consensus.cc:606] T 59112618a86046e1af899ce6d3c85203 P 263986885d354f7ba08739e419a6cded [term 4 LEADER]: Received request to transfer leadership to TS b5e46fc5aac9426a945bb34207d634d4
I20260709 12:58:13.184062   641 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "61c59a4c19754dedb9d385e2ecfe145b"
dest_uuid: "263986885d354f7ba08739e419a6cded"
mode: GRACEFUL
new_leader_uuid: "1bcd68d29a7b41a9bf114ac103d4c649"
 from {username='slave'} at 127.0.0.1:60340
I20260709 12:58:13.184518   641 raft_consensus.cc:606] T 61c59a4c19754dedb9d385e2ecfe145b P 263986885d354f7ba08739e419a6cded [term 2 LEADER]: Received request to transfer leadership to TS 1bcd68d29a7b41a9bf114ac103d4c649
I20260709 12:58:13.187323   641 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "d4d26ba9fb084e748bddb93f7547621f"
dest_uuid: "263986885d354f7ba08739e419a6cded"
mode: GRACEFUL
new_leader_uuid: "b5e46fc5aac9426a945bb34207d634d4"
 from {username='slave'} at 127.0.0.1:60340
I20260709 12:58:13.187938   641 raft_consensus.cc:606] T d4d26ba9fb084e748bddb93f7547621f P 263986885d354f7ba08739e419a6cded [term 2 LEADER]: Received request to transfer leadership to TS b5e46fc5aac9426a945bb34207d634d4
I20260709 12:58:13.190779   641 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "e6aabcfcdeb34e6c94ec0bb089deabda"
dest_uuid: "263986885d354f7ba08739e419a6cded"
mode: GRACEFUL
new_leader_uuid: "1bcd68d29a7b41a9bf114ac103d4c649"
 from {username='slave'} at 127.0.0.1:60340
I20260709 12:58:13.191331   641 raft_consensus.cc:606] T e6aabcfcdeb34e6c94ec0bb089deabda P 263986885d354f7ba08739e419a6cded [term 2 LEADER]: Received request to transfer leadership to TS 1bcd68d29a7b41a9bf114ac103d4c649
I20260709 12:58:13.192454   431 auto_leader_rebalancer.cc:452] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 12:58:13.192921   431 auto_leader_rebalancer.cc:527] All tables' leader rebalancing finished this round
I20260709 12:58:13.393270   881 raft_consensus.cc:993] T 2e14183cc6064b1cbe6530eb1c8c899c P 263986885d354f7ba08739e419a6cded: : Instructing follower b5e46fc5aac9426a945bb34207d634d4 to start an election
I20260709 12:58:13.393831   934 raft_consensus.cc:1081] T 2e14183cc6064b1cbe6530eb1c8c899c P 263986885d354f7ba08739e419a6cded [term 3 LEADER]: Signalling peer b5e46fc5aac9426a945bb34207d634d4 to start an election
I20260709 12:58:13.395470   563 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "2e14183cc6064b1cbe6530eb1c8c899c"
dest_uuid: "b5e46fc5aac9426a945bb34207d634d4"
 from {username='slave'} at 127.0.0.1:45346
I20260709 12:58:13.395952   563 raft_consensus.cc:493] T 2e14183cc6064b1cbe6530eb1c8c899c P b5e46fc5aac9426a945bb34207d634d4 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:58:13.396248   563 raft_consensus.cc:3060] T 2e14183cc6064b1cbe6530eb1c8c899c P b5e46fc5aac9426a945bb34207d634d4 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:58:13.401302   563 raft_consensus.cc:515] T 2e14183cc6064b1cbe6530eb1c8c899c P b5e46fc5aac9426a945bb34207d634d4 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } }
I20260709 12:58:13.403028   563 leader_election.cc:290] T 2e14183cc6064b1cbe6530eb1c8c899c P b5e46fc5aac9426a945bb34207d634d4 [CANDIDATE]: Term 4 election: Requested vote from peers 263986885d354f7ba08739e419a6cded (127.0.107.194:45491), 1bcd68d29a7b41a9bf114ac103d4c649 (127.0.107.195:46365)
I20260709 12:58:13.404197   714 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e14183cc6064b1cbe6530eb1c8c899c" candidate_uuid: "b5e46fc5aac9426a945bb34207d634d4" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "1bcd68d29a7b41a9bf114ac103d4c649"
I20260709 12:58:13.404270   641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e14183cc6064b1cbe6530eb1c8c899c" candidate_uuid: "b5e46fc5aac9426a945bb34207d634d4" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "263986885d354f7ba08739e419a6cded"
I20260709 12:58:13.404912   714 raft_consensus.cc:3060] T 2e14183cc6064b1cbe6530eb1c8c899c P 1bcd68d29a7b41a9bf114ac103d4c649 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:58:13.404982   641 raft_consensus.cc:3055] T 2e14183cc6064b1cbe6530eb1c8c899c P 263986885d354f7ba08739e419a6cded [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:58:13.405359   641 raft_consensus.cc:740] T 2e14183cc6064b1cbe6530eb1c8c899c P 263986885d354f7ba08739e419a6cded [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 263986885d354f7ba08739e419a6cded, State: Running, Role: LEADER
I20260709 12:58:13.406175   641 consensus_queue.cc:260] T 2e14183cc6064b1cbe6530eb1c8c899c P 263986885d354f7ba08739e419a6cded [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } }
I20260709 12:58:13.407619   641 raft_consensus.cc:3060] T 2e14183cc6064b1cbe6530eb1c8c899c P 263986885d354f7ba08739e419a6cded [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:58:13.410791   714 raft_consensus.cc:2468] T 2e14183cc6064b1cbe6530eb1c8c899c P 1bcd68d29a7b41a9bf114ac103d4c649 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5e46fc5aac9426a945bb34207d634d4 in term 4.
I20260709 12:58:13.412030   529 leader_election.cc:304] T 2e14183cc6064b1cbe6530eb1c8c899c P b5e46fc5aac9426a945bb34207d634d4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bcd68d29a7b41a9bf114ac103d4c649, b5e46fc5aac9426a945bb34207d634d4; no voters: 
I20260709 12:58:13.413609   935 raft_consensus.cc:2804] T 2e14183cc6064b1cbe6530eb1c8c899c P b5e46fc5aac9426a945bb34207d634d4 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:58:13.414784   935 raft_consensus.cc:697] T 2e14183cc6064b1cbe6530eb1c8c899c P b5e46fc5aac9426a945bb34207d634d4 [term 4 LEADER]: Becoming Leader. State: Replica: b5e46fc5aac9426a945bb34207d634d4, State: Running, Role: LEADER
I20260709 12:58:13.414963   641 raft_consensus.cc:2468] T 2e14183cc6064b1cbe6530eb1c8c899c P 263986885d354f7ba08739e419a6cded [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5e46fc5aac9426a945bb34207d634d4 in term 4.
I20260709 12:58:13.415684   935 consensus_queue.cc:237] T 2e14183cc6064b1cbe6530eb1c8c899c P b5e46fc5aac9426a945bb34207d634d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } }
I20260709 12:58:13.424355   463 catalog_manager.cc:5697] T 2e14183cc6064b1cbe6530eb1c8c899c P b5e46fc5aac9426a945bb34207d634d4 reported cstate change: term changed from 3 to 4, leader changed from 263986885d354f7ba08739e419a6cded (127.0.107.194) to b5e46fc5aac9426a945bb34207d634d4 (127.0.107.193). New cstate: current_term: 4 leader_uuid: "b5e46fc5aac9426a945bb34207d634d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } health_report { overall_health: UNKNOWN } } }
I20260709 12:58:13.563607   881 raft_consensus.cc:993] T 59112618a86046e1af899ce6d3c85203 P 263986885d354f7ba08739e419a6cded: : Instructing follower b5e46fc5aac9426a945bb34207d634d4 to start an election
I20260709 12:58:13.564167   934 raft_consensus.cc:1081] T 59112618a86046e1af899ce6d3c85203 P 263986885d354f7ba08739e419a6cded [term 4 LEADER]: Signalling peer b5e46fc5aac9426a945bb34207d634d4 to start an election
I20260709 12:58:13.565974   563 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "59112618a86046e1af899ce6d3c85203"
dest_uuid: "b5e46fc5aac9426a945bb34207d634d4"
 from {username='slave'} at 127.0.0.1:45346
I20260709 12:58:13.566519   563 raft_consensus.cc:493] T 59112618a86046e1af899ce6d3c85203 P b5e46fc5aac9426a945bb34207d634d4 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:58:13.566835   563 raft_consensus.cc:3060] T 59112618a86046e1af899ce6d3c85203 P b5e46fc5aac9426a945bb34207d634d4 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:58:13.571503   563 raft_consensus.cc:515] T 59112618a86046e1af899ce6d3c85203 P b5e46fc5aac9426a945bb34207d634d4 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } }
I20260709 12:58:13.573601   563 leader_election.cc:290] T 59112618a86046e1af899ce6d3c85203 P b5e46fc5aac9426a945bb34207d634d4 [CANDIDATE]: Term 5 election: Requested vote from peers 263986885d354f7ba08739e419a6cded (127.0.107.194:45491), 1bcd68d29a7b41a9bf114ac103d4c649 (127.0.107.195:46365)
I20260709 12:58:13.574012   641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59112618a86046e1af899ce6d3c85203" candidate_uuid: "b5e46fc5aac9426a945bb34207d634d4" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "263986885d354f7ba08739e419a6cded"
I20260709 12:58:13.574280   714 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59112618a86046e1af899ce6d3c85203" candidate_uuid: "b5e46fc5aac9426a945bb34207d634d4" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "1bcd68d29a7b41a9bf114ac103d4c649"
I20260709 12:58:13.574673   641 raft_consensus.cc:3055] T 59112618a86046e1af899ce6d3c85203 P 263986885d354f7ba08739e419a6cded [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:58:13.574801   714 raft_consensus.cc:3060] T 59112618a86046e1af899ce6d3c85203 P 1bcd68d29a7b41a9bf114ac103d4c649 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:58:13.575095   641 raft_consensus.cc:740] T 59112618a86046e1af899ce6d3c85203 P 263986885d354f7ba08739e419a6cded [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 263986885d354f7ba08739e419a6cded, State: Running, Role: LEADER
I20260709 12:58:13.575938   641 consensus_queue.cc:260] T 59112618a86046e1af899ce6d3c85203 P 263986885d354f7ba08739e419a6cded [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } }
I20260709 12:58:13.577047   641 raft_consensus.cc:3060] T 59112618a86046e1af899ce6d3c85203 P 263986885d354f7ba08739e419a6cded [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:58:13.579936   714 raft_consensus.cc:2468] T 59112618a86046e1af899ce6d3c85203 P 1bcd68d29a7b41a9bf114ac103d4c649 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5e46fc5aac9426a945bb34207d634d4 in term 5.
I20260709 12:58:13.581212   529 leader_election.cc:304] T 59112618a86046e1af899ce6d3c85203 P b5e46fc5aac9426a945bb34207d634d4 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bcd68d29a7b41a9bf114ac103d4c649, b5e46fc5aac9426a945bb34207d634d4; no voters: 
I20260709 12:58:13.582088   935 raft_consensus.cc:2804] T 59112618a86046e1af899ce6d3c85203 P b5e46fc5aac9426a945bb34207d634d4 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:58:13.582583   935 raft_consensus.cc:697] T 59112618a86046e1af899ce6d3c85203 P b5e46fc5aac9426a945bb34207d634d4 [term 5 LEADER]: Becoming Leader. State: Replica: b5e46fc5aac9426a945bb34207d634d4, State: Running, Role: LEADER
I20260709 12:58:13.583503   935 consensus_queue.cc:237] T 59112618a86046e1af899ce6d3c85203 P b5e46fc5aac9426a945bb34207d634d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } }
I20260709 12:58:13.584507   641 raft_consensus.cc:2468] T 59112618a86046e1af899ce6d3c85203 P 263986885d354f7ba08739e419a6cded [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5e46fc5aac9426a945bb34207d634d4 in term 5.
I20260709 12:58:13.590720   934 raft_consensus.cc:993] T d4d26ba9fb084e748bddb93f7547621f P 263986885d354f7ba08739e419a6cded: : Instructing follower b5e46fc5aac9426a945bb34207d634d4 to start an election
I20260709 12:58:13.591200   934 raft_consensus.cc:1081] T d4d26ba9fb084e748bddb93f7547621f P 263986885d354f7ba08739e419a6cded [term 2 LEADER]: Signalling peer b5e46fc5aac9426a945bb34207d634d4 to start an election
I20260709 12:58:13.592902   563 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "d4d26ba9fb084e748bddb93f7547621f"
dest_uuid: "b5e46fc5aac9426a945bb34207d634d4"
 from {username='slave'} at 127.0.0.1:45346
I20260709 12:58:13.593371   563 raft_consensus.cc:493] T d4d26ba9fb084e748bddb93f7547621f P b5e46fc5aac9426a945bb34207d634d4 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:58:13.593741   563 raft_consensus.cc:3060] T d4d26ba9fb084e748bddb93f7547621f P b5e46fc5aac9426a945bb34207d634d4 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:58:13.595227   463 catalog_manager.cc:5697] T 59112618a86046e1af899ce6d3c85203 P b5e46fc5aac9426a945bb34207d634d4 reported cstate change: term changed from 4 to 5, leader changed from 263986885d354f7ba08739e419a6cded (127.0.107.194) to b5e46fc5aac9426a945bb34207d634d4 (127.0.107.193). New cstate: current_term: 5 leader_uuid: "b5e46fc5aac9426a945bb34207d634d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } health_report { overall_health: UNKNOWN } } }
I20260709 12:58:13.599892   563 raft_consensus.cc:515] T d4d26ba9fb084e748bddb93f7547621f P b5e46fc5aac9426a945bb34207d634d4 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } }
I20260709 12:58:13.602705   641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d26ba9fb084e748bddb93f7547621f" candidate_uuid: "b5e46fc5aac9426a945bb34207d634d4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "263986885d354f7ba08739e419a6cded"
I20260709 12:58:13.602937   714 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4d26ba9fb084e748bddb93f7547621f" candidate_uuid: "b5e46fc5aac9426a945bb34207d634d4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "1bcd68d29a7b41a9bf114ac103d4c649"
I20260709 12:58:13.603287   641 raft_consensus.cc:3055] T d4d26ba9fb084e748bddb93f7547621f P 263986885d354f7ba08739e419a6cded [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:58:13.603554   714 raft_consensus.cc:3060] T d4d26ba9fb084e748bddb93f7547621f P 1bcd68d29a7b41a9bf114ac103d4c649 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:58:13.603637   641 raft_consensus.cc:740] T d4d26ba9fb084e748bddb93f7547621f P 263986885d354f7ba08739e419a6cded [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 263986885d354f7ba08739e419a6cded, State: Running, Role: LEADER
I20260709 12:58:13.604552   641 consensus_queue.cc:260] T d4d26ba9fb084e748bddb93f7547621f P 263986885d354f7ba08739e419a6cded [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } }
I20260709 12:58:13.605669   641 raft_consensus.cc:3060] T d4d26ba9fb084e748bddb93f7547621f P 263986885d354f7ba08739e419a6cded [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:58:13.607801   563 leader_election.cc:290] T d4d26ba9fb084e748bddb93f7547621f P b5e46fc5aac9426a945bb34207d634d4 [CANDIDATE]: Term 3 election: Requested vote from peers 263986885d354f7ba08739e419a6cded (127.0.107.194:45491), 1bcd68d29a7b41a9bf114ac103d4c649 (127.0.107.195:46365)
I20260709 12:58:13.610929   714 raft_consensus.cc:2468] T d4d26ba9fb084e748bddb93f7547621f P 1bcd68d29a7b41a9bf114ac103d4c649 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5e46fc5aac9426a945bb34207d634d4 in term 3.
I20260709 12:58:13.611915   529 leader_election.cc:304] T d4d26ba9fb084e748bddb93f7547621f P b5e46fc5aac9426a945bb34207d634d4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bcd68d29a7b41a9bf114ac103d4c649, b5e46fc5aac9426a945bb34207d634d4; no voters: 
I20260709 12:58:13.613111   935 raft_consensus.cc:2804] T d4d26ba9fb084e748bddb93f7547621f P b5e46fc5aac9426a945bb34207d634d4 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:58:13.613624   935 raft_consensus.cc:697] T d4d26ba9fb084e748bddb93f7547621f P b5e46fc5aac9426a945bb34207d634d4 [term 3 LEADER]: Becoming Leader. State: Replica: b5e46fc5aac9426a945bb34207d634d4, State: Running, Role: LEADER
I20260709 12:58:13.614476   641 raft_consensus.cc:2468] T d4d26ba9fb084e748bddb93f7547621f P 263986885d354f7ba08739e419a6cded [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5e46fc5aac9426a945bb34207d634d4 in term 3.
I20260709 12:58:13.615091   935 consensus_queue.cc:237] T d4d26ba9fb084e748bddb93f7547621f P b5e46fc5aac9426a945bb34207d634d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } }
I20260709 12:58:13.625751   881 raft_consensus.cc:993] T 61c59a4c19754dedb9d385e2ecfe145b P 263986885d354f7ba08739e419a6cded: : Instructing follower 1bcd68d29a7b41a9bf114ac103d4c649 to start an election
I20260709 12:58:13.626274   934 raft_consensus.cc:1081] T 61c59a4c19754dedb9d385e2ecfe145b P 263986885d354f7ba08739e419a6cded [term 2 LEADER]: Signalling peer 1bcd68d29a7b41a9bf114ac103d4c649 to start an election
I20260709 12:58:13.625506   463 catalog_manager.cc:5697] T d4d26ba9fb084e748bddb93f7547621f P b5e46fc5aac9426a945bb34207d634d4 reported cstate change: term changed from 2 to 3, leader changed from 263986885d354f7ba08739e419a6cded (127.0.107.194) to b5e46fc5aac9426a945bb34207d634d4 (127.0.107.193). New cstate: current_term: 3 leader_uuid: "b5e46fc5aac9426a945bb34207d634d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } health_report { overall_health: UNKNOWN } } }
I20260709 12:58:13.627790   714 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "61c59a4c19754dedb9d385e2ecfe145b"
dest_uuid: "1bcd68d29a7b41a9bf114ac103d4c649"
 from {username='slave'} at 127.0.0.1:56652
I20260709 12:58:13.628283   714 raft_consensus.cc:493] T 61c59a4c19754dedb9d385e2ecfe145b P 1bcd68d29a7b41a9bf114ac103d4c649 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:58:13.628629   714 raft_consensus.cc:3060] T 61c59a4c19754dedb9d385e2ecfe145b P 1bcd68d29a7b41a9bf114ac103d4c649 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:58:13.636411   714 raft_consensus.cc:515] T 61c59a4c19754dedb9d385e2ecfe145b P 1bcd68d29a7b41a9bf114ac103d4c649 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } }
I20260709 12:58:13.638715   714 leader_election.cc:290] T 61c59a4c19754dedb9d385e2ecfe145b P 1bcd68d29a7b41a9bf114ac103d4c649 [CANDIDATE]: Term 3 election: Requested vote from peers b5e46fc5aac9426a945bb34207d634d4 (127.0.107.193:38425), 263986885d354f7ba08739e419a6cded (127.0.107.194:45491)
I20260709 12:58:13.639760   563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "61c59a4c19754dedb9d385e2ecfe145b" candidate_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "b5e46fc5aac9426a945bb34207d634d4"
I20260709 12:58:13.640373   563 raft_consensus.cc:3060] T 61c59a4c19754dedb9d385e2ecfe145b P b5e46fc5aac9426a945bb34207d634d4 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:58:13.641927   641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "61c59a4c19754dedb9d385e2ecfe145b" candidate_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "263986885d354f7ba08739e419a6cded"
I20260709 12:58:13.642509   641 raft_consensus.cc:3055] T 61c59a4c19754dedb9d385e2ecfe145b P 263986885d354f7ba08739e419a6cded [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:58:13.642799   641 raft_consensus.cc:740] T 61c59a4c19754dedb9d385e2ecfe145b P 263986885d354f7ba08739e419a6cded [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 263986885d354f7ba08739e419a6cded, State: Running, Role: LEADER
I20260709 12:58:13.643564   641 consensus_queue.cc:260] T 61c59a4c19754dedb9d385e2ecfe145b P 263986885d354f7ba08739e419a6cded [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } }
I20260709 12:58:13.644721   641 raft_consensus.cc:3060] T 61c59a4c19754dedb9d385e2ecfe145b P 263986885d354f7ba08739e419a6cded [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:58:13.647055   563 raft_consensus.cc:2468] T 61c59a4c19754dedb9d385e2ecfe145b P b5e46fc5aac9426a945bb34207d634d4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1bcd68d29a7b41a9bf114ac103d4c649 in term 3.
I20260709 12:58:13.649487   681 leader_election.cc:304] T 61c59a4c19754dedb9d385e2ecfe145b P 1bcd68d29a7b41a9bf114ac103d4c649 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bcd68d29a7b41a9bf114ac103d4c649, b5e46fc5aac9426a945bb34207d634d4; no voters: 
I20260709 12:58:13.650501   933 raft_consensus.cc:2804] T 61c59a4c19754dedb9d385e2ecfe145b P 1bcd68d29a7b41a9bf114ac103d4c649 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:58:13.650887   641 raft_consensus.cc:2468] T 61c59a4c19754dedb9d385e2ecfe145b P 263986885d354f7ba08739e419a6cded [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1bcd68d29a7b41a9bf114ac103d4c649 in term 3.
I20260709 12:58:13.651512   934 raft_consensus.cc:993] T e6aabcfcdeb34e6c94ec0bb089deabda P 263986885d354f7ba08739e419a6cded: : Instructing follower 1bcd68d29a7b41a9bf114ac103d4c649 to start an election
I20260709 12:58:13.652045   942 raft_consensus.cc:1081] T e6aabcfcdeb34e6c94ec0bb089deabda P 263986885d354f7ba08739e419a6cded [term 2 LEADER]: Signalling peer 1bcd68d29a7b41a9bf114ac103d4c649 to start an election
I20260709 12:58:13.654184   714 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "e6aabcfcdeb34e6c94ec0bb089deabda"
dest_uuid: "1bcd68d29a7b41a9bf114ac103d4c649"
 from {username='slave'} at 127.0.0.1:56652
I20260709 12:58:13.654400   933 raft_consensus.cc:697] T 61c59a4c19754dedb9d385e2ecfe145b P 1bcd68d29a7b41a9bf114ac103d4c649 [term 3 LEADER]: Becoming Leader. State: Replica: 1bcd68d29a7b41a9bf114ac103d4c649, State: Running, Role: LEADER
I20260709 12:58:13.654918   714 raft_consensus.cc:493] T e6aabcfcdeb34e6c94ec0bb089deabda P 1bcd68d29a7b41a9bf114ac103d4c649 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:58:13.655367   714 raft_consensus.cc:3060] T e6aabcfcdeb34e6c94ec0bb089deabda P 1bcd68d29a7b41a9bf114ac103d4c649 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:58:13.655577   933 consensus_queue.cc:237] T 61c59a4c19754dedb9d385e2ecfe145b P 1bcd68d29a7b41a9bf114ac103d4c649 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } }
I20260709 12:58:13.661864   714 raft_consensus.cc:515] T e6aabcfcdeb34e6c94ec0bb089deabda P 1bcd68d29a7b41a9bf114ac103d4c649 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } }
I20260709 12:58:13.663792   714 leader_election.cc:290] T e6aabcfcdeb34e6c94ec0bb089deabda P 1bcd68d29a7b41a9bf114ac103d4c649 [CANDIDATE]: Term 3 election: Requested vote from peers b5e46fc5aac9426a945bb34207d634d4 (127.0.107.193:38425), 263986885d354f7ba08739e419a6cded (127.0.107.194:45491)
I20260709 12:58:13.665403   641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e6aabcfcdeb34e6c94ec0bb089deabda" candidate_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "263986885d354f7ba08739e419a6cded"
I20260709 12:58:13.665933   641 raft_consensus.cc:3055] T e6aabcfcdeb34e6c94ec0bb089deabda P 263986885d354f7ba08739e419a6cded [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:58:13.665223   463 catalog_manager.cc:5697] T 61c59a4c19754dedb9d385e2ecfe145b P 1bcd68d29a7b41a9bf114ac103d4c649 reported cstate change: term changed from 2 to 3, leader changed from 263986885d354f7ba08739e419a6cded (127.0.107.194) to 1bcd68d29a7b41a9bf114ac103d4c649 (127.0.107.195). New cstate: current_term: 3 leader_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } health_report { overall_health: HEALTHY } } }
I20260709 12:58:13.666218   641 raft_consensus.cc:740] T e6aabcfcdeb34e6c94ec0bb089deabda P 263986885d354f7ba08739e419a6cded [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 263986885d354f7ba08739e419a6cded, State: Running, Role: LEADER
I20260709 12:58:13.667052   641 consensus_queue.cc:260] T e6aabcfcdeb34e6c94ec0bb089deabda P 263986885d354f7ba08739e419a6cded [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } }
I20260709 12:58:13.668181   641 raft_consensus.cc:3060] T e6aabcfcdeb34e6c94ec0bb089deabda P 263986885d354f7ba08739e419a6cded [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:58:13.668524   563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e6aabcfcdeb34e6c94ec0bb089deabda" candidate_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "b5e46fc5aac9426a945bb34207d634d4"
I20260709 12:58:13.669126   563 raft_consensus.cc:3060] T e6aabcfcdeb34e6c94ec0bb089deabda P b5e46fc5aac9426a945bb34207d634d4 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:58:13.674774   641 raft_consensus.cc:2468] T e6aabcfcdeb34e6c94ec0bb089deabda P 263986885d354f7ba08739e419a6cded [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1bcd68d29a7b41a9bf114ac103d4c649 in term 3.
I20260709 12:58:13.675997   680 leader_election.cc:304] T e6aabcfcdeb34e6c94ec0bb089deabda P 1bcd68d29a7b41a9bf114ac103d4c649 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bcd68d29a7b41a9bf114ac103d4c649, 263986885d354f7ba08739e419a6cded; no voters: 
I20260709 12:58:13.676883   945 raft_consensus.cc:2804] T e6aabcfcdeb34e6c94ec0bb089deabda P 1bcd68d29a7b41a9bf114ac103d4c649 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:58:13.678526   563 raft_consensus.cc:2468] T e6aabcfcdeb34e6c94ec0bb089deabda P b5e46fc5aac9426a945bb34207d634d4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1bcd68d29a7b41a9bf114ac103d4c649 in term 3.
I20260709 12:58:13.680068   945 raft_consensus.cc:697] T e6aabcfcdeb34e6c94ec0bb089deabda P 1bcd68d29a7b41a9bf114ac103d4c649 [term 3 LEADER]: Becoming Leader. State: Replica: 1bcd68d29a7b41a9bf114ac103d4c649, State: Running, Role: LEADER
I20260709 12:58:13.681094   945 consensus_queue.cc:237] T e6aabcfcdeb34e6c94ec0bb089deabda P 1bcd68d29a7b41a9bf114ac103d4c649 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } }
I20260709 12:58:13.689309   463 catalog_manager.cc:5697] T e6aabcfcdeb34e6c94ec0bb089deabda P 1bcd68d29a7b41a9bf114ac103d4c649 reported cstate change: term changed from 2 to 3, leader changed from 263986885d354f7ba08739e419a6cded (127.0.107.194) to 1bcd68d29a7b41a9bf114ac103d4c649 (127.0.107.195). New cstate: current_term: 3 leader_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 } health_report { overall_health: HEALTHY } } }
I20260709 12:58:13.904876   714 raft_consensus.cc:1275] T 2e14183cc6064b1cbe6530eb1c8c899c P 1bcd68d29a7b41a9bf114ac103d4c649 [term 4 FOLLOWER]: Refusing update from remote peer b5e46fc5aac9426a945bb34207d634d4: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:58:13.906196   935 consensus_queue.cc:1048] T 2e14183cc6064b1cbe6530eb1c8c899c P b5e46fc5aac9426a945bb34207d634d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:58:13.921996   641 raft_consensus.cc:1275] T 2e14183cc6064b1cbe6530eb1c8c899c P 263986885d354f7ba08739e419a6cded [term 4 FOLLOWER]: Refusing update from remote peer b5e46fc5aac9426a945bb34207d634d4: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:58:13.923605   935 consensus_queue.cc:1048] T 2e14183cc6064b1cbe6530eb1c8c899c P b5e46fc5aac9426a945bb34207d634d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:58:13.976974   714 raft_consensus.cc:1275] T 59112618a86046e1af899ce6d3c85203 P 1bcd68d29a7b41a9bf114ac103d4c649 [term 5 FOLLOWER]: Refusing update from remote peer b5e46fc5aac9426a945bb34207d634d4: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260709 12:58:13.978586   952 consensus_queue.cc:1048] T 59112618a86046e1af899ce6d3c85203 P b5e46fc5aac9426a945bb34207d634d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:58:13.991055   641 raft_consensus.cc:1275] T 59112618a86046e1af899ce6d3c85203 P 263986885d354f7ba08739e419a6cded [term 5 FOLLOWER]: Refusing update from remote peer b5e46fc5aac9426a945bb34207d634d4: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260709 12:58:13.997665   714 raft_consensus.cc:1275] T d4d26ba9fb084e748bddb93f7547621f P 1bcd68d29a7b41a9bf114ac103d4c649 [term 3 FOLLOWER]: Refusing update from remote peer b5e46fc5aac9426a945bb34207d634d4: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:58:13.998025   936 consensus_queue.cc:1048] T 59112618a86046e1af899ce6d3c85203 P b5e46fc5aac9426a945bb34207d634d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:58:14.000701   936 consensus_queue.cc:1048] T d4d26ba9fb084e748bddb93f7547621f P b5e46fc5aac9426a945bb34207d634d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bcd68d29a7b41a9bf114ac103d4c649" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:58:14.020043   641 raft_consensus.cc:1275] T d4d26ba9fb084e748bddb93f7547621f P 263986885d354f7ba08739e419a6cded [term 3 FOLLOWER]: Refusing update from remote peer b5e46fc5aac9426a945bb34207d634d4: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:58:14.021673   936 consensus_queue.cc:1048] T d4d26ba9fb084e748bddb93f7547621f P b5e46fc5aac9426a945bb34207d634d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:58:14.086704   563 raft_consensus.cc:1275] T 61c59a4c19754dedb9d385e2ecfe145b P b5e46fc5aac9426a945bb34207d634d4 [term 3 FOLLOWER]: Refusing update from remote peer 1bcd68d29a7b41a9bf114ac103d4c649: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:58:14.088264   945 consensus_queue.cc:1048] T 61c59a4c19754dedb9d385e2ecfe145b P 1bcd68d29a7b41a9bf114ac103d4c649 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:58:14.102420   641 raft_consensus.cc:1275] T 61c59a4c19754dedb9d385e2ecfe145b P 263986885d354f7ba08739e419a6cded [term 3 FOLLOWER]: Refusing update from remote peer 1bcd68d29a7b41a9bf114ac103d4c649: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:58:14.105372   947 consensus_queue.cc:1048] T 61c59a4c19754dedb9d385e2ecfe145b P 1bcd68d29a7b41a9bf114ac103d4c649 [LEADER]: Connected to new peer: Peer: permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:58:14.110409   563 raft_consensus.cc:1275] T e6aabcfcdeb34e6c94ec0bb089deabda P b5e46fc5aac9426a945bb34207d634d4 [term 3 FOLLOWER]: Refusing update from remote peer 1bcd68d29a7b41a9bf114ac103d4c649: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:58:14.112097   947 consensus_queue.cc:1048] T e6aabcfcdeb34e6c94ec0bb089deabda P 1bcd68d29a7b41a9bf114ac103d4c649 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5e46fc5aac9426a945bb34207d634d4" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:58:14.133641   641 raft_consensus.cc:1275] T e6aabcfcdeb34e6c94ec0bb089deabda P 263986885d354f7ba08739e419a6cded [term 3 FOLLOWER]: Refusing update from remote peer 1bcd68d29a7b41a9bf114ac103d4c649: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:58:14.136112   947 consensus_queue.cc:1048] T e6aabcfcdeb34e6c94ec0bb089deabda P 1bcd68d29a7b41a9bf114ac103d4c649 [LEADER]: Connected to new peer: Peer: permanent_uuid: "263986885d354f7ba08739e419a6cded" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:58:14.193909   431 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:58:14.201805   431 tablet_server.cc:179] TabletServer@127.0.107.193:0 shutting down...
I20260709 12:58:14.241930   431 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:58:14.243093   431 tablet_replica.cc:333] T 61c59a4c19754dedb9d385e2ecfe145b P b5e46fc5aac9426a945bb34207d634d4: stopping tablet replica
I20260709 12:58:14.244243   431 raft_consensus.cc:2243] T 61c59a4c19754dedb9d385e2ecfe145b P b5e46fc5aac9426a945bb34207d634d4 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:58:14.244976   431 raft_consensus.cc:2272] T 61c59a4c19754dedb9d385e2ecfe145b P b5e46fc5aac9426a945bb34207d634d4 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:58:14.249276   431 tablet_replica.cc:333] T 2e14183cc6064b1cbe6530eb1c8c899c P b5e46fc5aac9426a945bb34207d634d4: stopping tablet replica
I20260709 12:58:14.250151   431 raft_consensus.cc:2243] T 2e14183cc6064b1cbe6530eb1c8c899c P b5e46fc5aac9426a945bb34207d634d4 [term 4 LEADER]: Raft consensus shutting down.
I20260709 12:58:14.251353   431 raft_consensus.cc:2272] T 2e14183cc6064b1cbe6530eb1c8c899c P b5e46fc5aac9426a945bb34207d634d4 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:58:14.254236   431 tablet_replica.cc:333] T e6aabcfcdeb34e6c94ec0bb089deabda P b5e46fc5aac9426a945bb34207d634d4: stopping tablet replica
I20260709 12:58:14.255045   431 raft_consensus.cc:2243] T e6aabcfcdeb34e6c94ec0bb089deabda P b5e46fc5aac9426a945bb34207d634d4 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:58:14.255640   431 raft_consensus.cc:2272] T e6aabcfcdeb34e6c94ec0bb089deabda P b5e46fc5aac9426a945bb34207d634d4 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:58:14.258306   431 tablet_replica.cc:333] T d8b33c9315094794a876a4b62755efaf P b5e46fc5aac9426a945bb34207d634d4: stopping tablet replica
I20260709 12:58:14.259114   431 raft_consensus.cc:2243] T d8b33c9315094794a876a4b62755efaf P b5e46fc5aac9426a945bb34207d634d4 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:58:14.259687   431 raft_consensus.cc:2272] T d8b33c9315094794a876a4b62755efaf P b5e46fc5aac9426a945bb34207d634d4 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:58:14.262014   431 tablet_replica.cc:333] T 793bafcf38ca4ffb9a9aac9374ea2094 P b5e46fc5aac9426a945bb34207d634d4: stopping tablet replica
I20260709 12:58:14.262754   431 raft_consensus.cc:2243] T 793bafcf38ca4ffb9a9aac9374ea2094 P b5e46fc5aac9426a945bb34207d634d4 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:58:14.263268   431 raft_consensus.cc:2272] T 793bafcf38ca4ffb9a9aac9374ea2094 P b5e46fc5aac9426a945bb34207d634d4 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:58:14.265690   431 tablet_replica.cc:333] T d441098f471f4b0084938676c20981e5 P b5e46fc5aac9426a945bb34207d634d4: stopping tablet replica
I20260709 12:58:14.266458   431 raft_consensus.cc:2243] T d441098f471f4b0084938676c20981e5 P b5e46fc5aac9426a945bb34207d634d4 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:58:14.266978   431 raft_consensus.cc:2272] T d441098f471f4b0084938676c20981e5 P b5e46fc5aac9426a945bb34207d634d4 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:58:14.269289   431 tablet_replica.cc:333] T 59112618a86046e1af899ce6d3c85203 P b5e46fc5aac9426a945bb34207d634d4: stopping tablet replica
I20260709 12:58:14.269979   431 raft_consensus.cc:2243] T 59112618a86046e1af899ce6d3c85203 P b5e46fc5aac9426a945bb34207d634d4 [term 5 LEADER]: Raft consensus shutting down.
I20260709 12:58:14.271082   431 raft_consensus.cc:2272] T 59112618a86046e1af899ce6d3c85203 P b5e46fc5aac9426a945bb34207d634d4 [term 5 FOLLOWER]: Raft consensus is shut down!
I20260709 12:58:14.273697   431 tablet_replica.cc:333] T d4d26ba9fb084e748bddb93f7547621f P b5e46fc5aac9426a945bb34207d634d4: stopping tablet replica
I20260709 12:58:14.274482   431 raft_consensus.cc:2243] T d4d26ba9fb084e748bddb93f7547621f P b5e46fc5aac9426a945bb34207d634d4 [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:58:14.275569   431 raft_consensus.cc:2272] T d4d26ba9fb084e748bddb93f7547621f P b5e46fc5aac9426a945bb34207d634d4 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:58:14.278052   431 tablet_replica.cc:333] T dd06089bfde94412b12705a116ecf3d5 P b5e46fc5aac9426a945bb34207d634d4: stopping tablet replica
I20260709 12:58:14.278843   431 raft_consensus.cc:2243] T dd06089bfde94412b12705a116ecf3d5 P b5e46fc5aac9426a945bb34207d634d4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:58:14.279398   431 raft_consensus.cc:2272] T dd06089bfde94412b12705a116ecf3d5 P b5e46fc5aac9426a945bb34207d634d4 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 12:58:14.554428   607 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.107.193:38425: connect: Connection refused (error 111)
W20260709 12:58:14.559109   607 consensus_peers.cc:597] T 793bafcf38ca4ffb9a9aac9374ea2094 P 263986885d354f7ba08739e419a6cded -> Peer b5e46fc5aac9426a945bb34207d634d4 (127.0.107.193:38425): Couldn't send request to peer b5e46fc5aac9426a945bb34207d634d4. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:38425: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:58:14.559675   607 consensus_peers.cc:597] T d8b33c9315094794a876a4b62755efaf P 263986885d354f7ba08739e419a6cded -> Peer b5e46fc5aac9426a945bb34207d634d4 (127.0.107.193:38425): Couldn't send request to peer b5e46fc5aac9426a945bb34207d634d4. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:38425: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:58:14.565040   607 consensus_peers.cc:597] T d441098f471f4b0084938676c20981e5 P 263986885d354f7ba08739e419a6cded -> Peer b5e46fc5aac9426a945bb34207d634d4 (127.0.107.193:38425): Couldn't send request to peer b5e46fc5aac9426a945bb34207d634d4. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:38425: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:58:14.590163   431 tablet_server.cc:196] TabletServer@127.0.107.193:0 shutdown complete.
I20260709 12:58:14.611361   431 tablet_server.cc:179] TabletServer@127.0.107.194:0 shutting down...
I20260709 12:58:14.636874   431 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:58:14.637689   431 tablet_replica.cc:333] T dd06089bfde94412b12705a116ecf3d5 P 263986885d354f7ba08739e419a6cded: stopping tablet replica
I20260709 12:58:14.638409   431 raft_consensus.cc:2243] T dd06089bfde94412b12705a116ecf3d5 P 263986885d354f7ba08739e419a6cded [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:58:14.638983   431 raft_consensus.cc:2272] T dd06089bfde94412b12705a116ecf3d5 P 263986885d354f7ba08739e419a6cded [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:58:14.640893   431 tablet_replica.cc:333] T 61c59a4c19754dedb9d385e2ecfe145b P 263986885d354f7ba08739e419a6cded: stopping tablet replica
I20260709 12:58:14.641448   431 raft_consensus.cc:2243] T 61c59a4c19754dedb9d385e2ecfe145b P 263986885d354f7ba08739e419a6cded [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:58:14.641876   431 raft_consensus.cc:2272] T 61c59a4c19754dedb9d385e2ecfe145b P 263986885d354f7ba08739e419a6cded [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:58:14.644342   431 tablet_replica.cc:333] T d4d26ba9fb084e748bddb93f7547621f P 263986885d354f7ba08739e419a6cded: stopping tablet replica
I20260709 12:58:14.645117   431 raft_consensus.cc:2243] T d4d26ba9fb084e748bddb93f7547621f P 263986885d354f7ba08739e419a6cded [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:58:14.645674   431 raft_consensus.cc:2272] T d4d26ba9fb084e748bddb93f7547621f P 263986885d354f7ba08739e419a6cded [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:58:14.648092   431 tablet_replica.cc:333] T d8b33c9315094794a876a4b62755efaf P 263986885d354f7ba08739e419a6cded: stopping tablet replica
I20260709 12:58:14.648610   431 raft_consensus.cc:2243] T d8b33c9315094794a876a4b62755efaf P 263986885d354f7ba08739e419a6cded [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:58:14.649629   431 raft_consensus.cc:2272] T d8b33c9315094794a876a4b62755efaf P 263986885d354f7ba08739e419a6cded [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:58:14.651552   431 tablet_replica.cc:333] T 793bafcf38ca4ffb9a9aac9374ea2094 P 263986885d354f7ba08739e419a6cded: stopping tablet replica
W20260709 12:58:14.651528   681 consensus_peers.cc:597] T e6aabcfcdeb34e6c94ec0bb089deabda P 1bcd68d29a7b41a9bf114ac103d4c649 -> Peer b5e46fc5aac9426a945bb34207d634d4 (127.0.107.193:38425): Couldn't send request to peer b5e46fc5aac9426a945bb34207d634d4. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:38425: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:58:14.652292   431 raft_consensus.cc:2243] T 793bafcf38ca4ffb9a9aac9374ea2094 P 263986885d354f7ba08739e419a6cded [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:58:14.653030   431 raft_consensus.cc:2272] T 793bafcf38ca4ffb9a9aac9374ea2094 P 263986885d354f7ba08739e419a6cded [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:58:14.654723   431 tablet_replica.cc:333] T e6aabcfcdeb34e6c94ec0bb089deabda P 263986885d354f7ba08739e419a6cded: stopping tablet replica
I20260709 12:58:14.655263   431 raft_consensus.cc:2243] T e6aabcfcdeb34e6c94ec0bb089deabda P 263986885d354f7ba08739e419a6cded [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:58:14.655669   431 raft_consensus.cc:2272] T e6aabcfcdeb34e6c94ec0bb089deabda P 263986885d354f7ba08739e419a6cded [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:58:14.658948   431 tablet_replica.cc:333] T 59112618a86046e1af899ce6d3c85203 P 263986885d354f7ba08739e419a6cded: stopping tablet replica
W20260709 12:58:14.659312   681 consensus_peers.cc:597] T dd06089bfde94412b12705a116ecf3d5 P 1bcd68d29a7b41a9bf114ac103d4c649 -> Peer b5e46fc5aac9426a945bb34207d634d4 (127.0.107.193:38425): Couldn't send request to peer b5e46fc5aac9426a945bb34207d634d4. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:38425: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:58:14.659457   431 raft_consensus.cc:2243] T 59112618a86046e1af899ce6d3c85203 P 263986885d354f7ba08739e419a6cded [term 5 FOLLOWER]: Raft consensus shutting down.
I20260709 12:58:14.659933   431 raft_consensus.cc:2272] T 59112618a86046e1af899ce6d3c85203 P 263986885d354f7ba08739e419a6cded [term 5 FOLLOWER]: Raft consensus is shut down!
I20260709 12:58:14.661739   431 tablet_replica.cc:333] T d441098f471f4b0084938676c20981e5 P 263986885d354f7ba08739e419a6cded: stopping tablet replica
I20260709 12:58:14.662281   431 raft_consensus.cc:2243] T d441098f471f4b0084938676c20981e5 P 263986885d354f7ba08739e419a6cded [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:58:14.663028   431 raft_consensus.cc:2272] T d441098f471f4b0084938676c20981e5 P 263986885d354f7ba08739e419a6cded [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:58:14.664743   431 tablet_replica.cc:333] T 2e14183cc6064b1cbe6530eb1c8c899c P 263986885d354f7ba08739e419a6cded: stopping tablet replica
I20260709 12:58:14.665149   431 raft_consensus.cc:2243] T 2e14183cc6064b1cbe6530eb1c8c899c P 263986885d354f7ba08739e419a6cded [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 12:58:14.665571   431 raft_consensus.cc:2272] T 2e14183cc6064b1cbe6530eb1c8c899c P 263986885d354f7ba08739e419a6cded [term 4 FOLLOWER]: Raft consensus is shut down!
W20260709 12:58:14.678233   681 consensus_peers.cc:597] T 61c59a4c19754dedb9d385e2ecfe145b P 1bcd68d29a7b41a9bf114ac103d4c649 -> Peer b5e46fc5aac9426a945bb34207d634d4 (127.0.107.193:38425): Couldn't send request to peer b5e46fc5aac9426a945bb34207d634d4. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:38425: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:58:14.722213   431 tablet_server.cc:196] TabletServer@127.0.107.194:0 shutdown complete.
I20260709 12:58:14.741243   431 tablet_server.cc:179] TabletServer@127.0.107.195:0 shutting down...
W20260709 12:58:14.763183   680 consensus_peers.cc:597] T dd06089bfde94412b12705a116ecf3d5 P 1bcd68d29a7b41a9bf114ac103d4c649 -> Peer 263986885d354f7ba08739e419a6cded (127.0.107.194:45491): Couldn't send request to peer 263986885d354f7ba08739e419a6cded. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:45491: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:58:14.766472   680 consensus_peers.cc:597] T e6aabcfcdeb34e6c94ec0bb089deabda P 1bcd68d29a7b41a9bf114ac103d4c649 -> Peer 263986885d354f7ba08739e419a6cded (127.0.107.194:45491): Couldn't send request to peer 263986885d354f7ba08739e419a6cded. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:45491: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:58:14.771999   431 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:58:14.772652   431 tablet_replica.cc:333] T d4d26ba9fb084e748bddb93f7547621f P 1bcd68d29a7b41a9bf114ac103d4c649: stopping tablet replica
I20260709 12:58:14.773286   431 raft_consensus.cc:2243] T d4d26ba9fb084e748bddb93f7547621f P 1bcd68d29a7b41a9bf114ac103d4c649 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:58:14.773748   431 raft_consensus.cc:2272] T d4d26ba9fb084e748bddb93f7547621f P 1bcd68d29a7b41a9bf114ac103d4c649 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:58:14.775918   431 tablet_replica.cc:333] T 61c59a4c19754dedb9d385e2ecfe145b P 1bcd68d29a7b41a9bf114ac103d4c649: stopping tablet replica
I20260709 12:58:14.776461   431 raft_consensus.cc:2243] T 61c59a4c19754dedb9d385e2ecfe145b P 1bcd68d29a7b41a9bf114ac103d4c649 [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:58:14.777199   431 raft_consensus.cc:2272] T 61c59a4c19754dedb9d385e2ecfe145b P 1bcd68d29a7b41a9bf114ac103d4c649 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:58:14.779124   431 tablet_replica.cc:333] T dd06089bfde94412b12705a116ecf3d5 P 1bcd68d29a7b41a9bf114ac103d4c649: stopping tablet replica
I20260709 12:58:14.779685   431 raft_consensus.cc:2243] T dd06089bfde94412b12705a116ecf3d5 P 1bcd68d29a7b41a9bf114ac103d4c649 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:58:14.780423   431 raft_consensus.cc:2272] T dd06089bfde94412b12705a116ecf3d5 P 1bcd68d29a7b41a9bf114ac103d4c649 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:58:14.782110   431 tablet_replica.cc:333] T d8b33c9315094794a876a4b62755efaf P 1bcd68d29a7b41a9bf114ac103d4c649: stopping tablet replica
I20260709 12:58:14.782648   431 raft_consensus.cc:2243] T d8b33c9315094794a876a4b62755efaf P 1bcd68d29a7b41a9bf114ac103d4c649 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:58:14.783037   431 raft_consensus.cc:2272] T d8b33c9315094794a876a4b62755efaf P 1bcd68d29a7b41a9bf114ac103d4c649 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:58:14.784549   431 tablet_replica.cc:333] T 2e14183cc6064b1cbe6530eb1c8c899c P 1bcd68d29a7b41a9bf114ac103d4c649: stopping tablet replica
I20260709 12:58:14.785027   431 raft_consensus.cc:2243] T 2e14183cc6064b1cbe6530eb1c8c899c P 1bcd68d29a7b41a9bf114ac103d4c649 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 12:58:14.785454   431 raft_consensus.cc:2272] T 2e14183cc6064b1cbe6530eb1c8c899c P 1bcd68d29a7b41a9bf114ac103d4c649 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:58:14.787407   431 tablet_replica.cc:333] T d441098f471f4b0084938676c20981e5 P 1bcd68d29a7b41a9bf114ac103d4c649: stopping tablet replica
I20260709 12:58:14.787897   431 raft_consensus.cc:2243] T d441098f471f4b0084938676c20981e5 P 1bcd68d29a7b41a9bf114ac103d4c649 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:58:14.788318   431 raft_consensus.cc:2272] T d441098f471f4b0084938676c20981e5 P 1bcd68d29a7b41a9bf114ac103d4c649 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:58:14.789901   431 tablet_replica.cc:333] T 59112618a86046e1af899ce6d3c85203 P 1bcd68d29a7b41a9bf114ac103d4c649: stopping tablet replica
I20260709 12:58:14.790359   431 raft_consensus.cc:2243] T 59112618a86046e1af899ce6d3c85203 P 1bcd68d29a7b41a9bf114ac103d4c649 [term 5 FOLLOWER]: Raft consensus shutting down.
I20260709 12:58:14.790809   431 raft_consensus.cc:2272] T 59112618a86046e1af899ce6d3c85203 P 1bcd68d29a7b41a9bf114ac103d4c649 [term 5 FOLLOWER]: Raft consensus is shut down!
I20260709 12:58:14.792685   431 tablet_replica.cc:333] T e6aabcfcdeb34e6c94ec0bb089deabda P 1bcd68d29a7b41a9bf114ac103d4c649: stopping tablet replica
I20260709 12:58:14.793164   431 raft_consensus.cc:2243] T e6aabcfcdeb34e6c94ec0bb089deabda P 1bcd68d29a7b41a9bf114ac103d4c649 [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:58:14.793902   431 raft_consensus.cc:2272] T e6aabcfcdeb34e6c94ec0bb089deabda P 1bcd68d29a7b41a9bf114ac103d4c649 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:58:14.795775   431 tablet_replica.cc:333] T 793bafcf38ca4ffb9a9aac9374ea2094 P 1bcd68d29a7b41a9bf114ac103d4c649: stopping tablet replica
I20260709 12:58:14.796232   431 raft_consensus.cc:2243] T 793bafcf38ca4ffb9a9aac9374ea2094 P 1bcd68d29a7b41a9bf114ac103d4c649 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:58:14.796645   431 raft_consensus.cc:2272] T 793bafcf38ca4ffb9a9aac9374ea2094 P 1bcd68d29a7b41a9bf114ac103d4c649 [term 3 FOLLOWER]: Raft consensus is shut down!
W20260709 12:58:15.564756   736 debug-util.cc:398] Leaking SignalData structure 0x7b0800166060 after lost signal to thread 434
W20260709 12:58:15.565784   736 debug-util.cc:398] Leaking SignalData structure 0x7b0800166740 after lost signal to thread 497
W20260709 12:58:15.566855   736 debug-util.cc:398] Leaking SignalData structure 0x7b08001eff20 after lost signal to thread 665
W20260709 12:58:15.567706   736 debug-util.cc:398] Leaking SignalData structure 0x7b08001b0660 after lost signal to thread 739
W20260709 12:58:15.821053   431 thread.cc:527] Waited for 1000ms trying to join with diag-logger (tid 736)
W20260709 12:58:16.821599   431 thread.cc:527] Waited for 2000ms trying to join with diag-logger (tid 736)
W20260709 12:58:17.822070   431 thread.cc:527] Waited for 3000ms trying to join with diag-logger (tid 736)
I20260709 12:58:18.227671   431 tablet_server.cc:196] TabletServer@127.0.107.195:0 shutdown complete.
I20260709 12:58:18.247097   431 master.cc:562] Master@127.0.107.254:34493 shutting down...
I20260709 12:58:18.367810   431 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 8c2299d3af364ee0a759958e5da33fa1 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:58:18.368552   431 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 8c2299d3af364ee0a759958e5da33fa1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:58:18.369107   431 tablet_replica.cc:333] T 00000000000000000000000000000000 P 8c2299d3af364ee0a759958e5da33fa1: stopping tablet replica
I20260709 12:58:18.389618   431 master.cc:584] Master@127.0.107.254:34493 shutdown complete.
1bcd68d29a7b41a9bf114ac103d4c649  1
263986885d354f7ba08739e419a6cded  4
b5e46fc5aac9426a945bb34207d634d4  4
[       OK ] LeaderRebalancerTest.FunctionalTestForDivided (19676 ms)
[ RUN      ] LeaderRebalancerTest.FunctionalTestForNotDivided
I20260709 12:58:18.436436   431 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.107.254:33267
I20260709 12:58:18.437744   431 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:58:18.444114   968 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:58:18.444628   969 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:58:18.445369   971 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:58:18.445417   431 server_base.cc:1061] running on GCE node
I20260709 12:58:18.446866   431 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:58:18.447094   431 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:58:18.447250   431 hybrid_clock.cc:648] HybridClock initialized: now 1783601898447230 us; error 0 us; skew 500 ppm
I20260709 12:58:18.447782   431 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:58:18.450318   431 webserver.cc:533] Webserver started at http://127.0.107.254:37565/ using document root <none> and password file <none>
I20260709 12:58:18.450955   431 fs_manager.cc:362] Metadata directory not provided
I20260709 12:58:18.451165   431 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:58:18.451462   431 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:58:18.452683   431 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskakPCWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783601878574137-431-0/minicluster-data/master-0-root/instance:
uuid: "851f14c22aa944d2ac3a9e0031eea552"
format_stamp: "Formatted at 2026-07-09 12:58:18 on dist-test-slave-nvvq"
I20260709 12:58:18.457394   431 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.006s	sys 0.000s
I20260709 12:58:18.460758   976 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:58:18.461431   431 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.000s
I20260709 12:58:18.461825   431 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskakPCWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783601878574137-431-0/minicluster-data/master-0-root
uuid: "851f14c22aa944d2ac3a9e0031eea552"
format_stamp: "Formatted at 2026-07-09 12:58:18 on dist-test-slave-nvvq"
I20260709 12:58:18.462092   431 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskakPCWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783601878574137-431-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskakPCWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783601878574137-431-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskakPCWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783601878574137-431-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:58:18.472142   431 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:58:18.473333   431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:58:18.517180   431 rpc_server.cc:307] RPC server started. Bound to: 127.0.107.254:33267
I20260709 12:58:18.517299  1027 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.107.254:33267 every 8 connection(s)
I20260709 12:58:18.521533  1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:18.531985  1028 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 851f14c22aa944d2ac3a9e0031eea552: Bootstrap starting.
I20260709 12:58:18.537534  1028 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 851f14c22aa944d2ac3a9e0031eea552: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:18.542539  1028 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 851f14c22aa944d2ac3a9e0031eea552: No bootstrap required, opened a new log
I20260709 12:58:18.544978  1028 raft_consensus.cc:359] T 00000000000000000000000000000000 P 851f14c22aa944d2ac3a9e0031eea552 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "851f14c22aa944d2ac3a9e0031eea552" member_type: VOTER }
I20260709 12:58:18.545392  1028 raft_consensus.cc:385] T 00000000000000000000000000000000 P 851f14c22aa944d2ac3a9e0031eea552 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:18.545702  1028 raft_consensus.cc:740] T 00000000000000000000000000000000 P 851f14c22aa944d2ac3a9e0031eea552 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 851f14c22aa944d2ac3a9e0031eea552, State: Initialized, Role: FOLLOWER
I20260709 12:58:18.546343  1028 consensus_queue.cc:260] T 00000000000000000000000000000000 P 851f14c22aa944d2ac3a9e0031eea552 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "851f14c22aa944d2ac3a9e0031eea552" member_type: VOTER }
I20260709 12:58:18.546849  1028 raft_consensus.cc:399] T 00000000000000000000000000000000 P 851f14c22aa944d2ac3a9e0031eea552 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260709 12:58:18.547088  1028 raft_consensus.cc:493] T 00000000000000000000000000000000 P 851f14c22aa944d2ac3a9e0031eea552 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260709 12:58:18.547371  1028 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 851f14c22aa944d2ac3a9e0031eea552 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:18.551924  1028 raft_consensus.cc:515] T 00000000000000000000000000000000 P 851f14c22aa944d2ac3a9e0031eea552 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "851f14c22aa944d2ac3a9e0031eea552" member_type: VOTER }
I20260709 12:58:18.552469  1028 leader_election.cc:304] T 00000000000000000000000000000000 P 851f14c22aa944d2ac3a9e0031eea552 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 851f14c22aa944d2ac3a9e0031eea552; no voters: 
I20260709 12:58:18.553689  1028 leader_election.cc:290] T 00000000000000000000000000000000 P 851f14c22aa944d2ac3a9e0031eea552 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20260709 12:58:18.554186  1031 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 851f14c22aa944d2ac3a9e0031eea552 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:58:18.555711  1031 raft_consensus.cc:697] T 00000000000000000000000000000000 P 851f14c22aa944d2ac3a9e0031eea552 [term 1 LEADER]: Becoming Leader. State: Replica: 851f14c22aa944d2ac3a9e0031eea552, State: Running, Role: LEADER
I20260709 12:58:18.556439  1031 consensus_queue.cc:237] T 00000000000000000000000000000000 P 851f14c22aa944d2ac3a9e0031eea552 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "851f14c22aa944d2ac3a9e0031eea552" member_type: VOTER }
I20260709 12:58:18.556901  1028 sys_catalog.cc:565] T 00000000000000000000000000000000 P 851f14c22aa944d2ac3a9e0031eea552 [sys.catalog]: configured and running, proceeding with master startup.
I20260709 12:58:18.561960  1033 sys_catalog.cc:455] T 00000000000000000000000000000000 P 851f14c22aa944d2ac3a9e0031eea552 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 851f14c22aa944d2ac3a9e0031eea552. Latest consensus state: current_term: 1 leader_uuid: "851f14c22aa944d2ac3a9e0031eea552" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "851f14c22aa944d2ac3a9e0031eea552" member_type: VOTER } }
I20260709 12:58:18.562312  1032 sys_catalog.cc:455] T 00000000000000000000000000000000 P 851f14c22aa944d2ac3a9e0031eea552 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "851f14c22aa944d2ac3a9e0031eea552" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "851f14c22aa944d2ac3a9e0031eea552" member_type: VOTER } }
I20260709 12:58:18.562824  1033 sys_catalog.cc:458] T 00000000000000000000000000000000 P 851f14c22aa944d2ac3a9e0031eea552 [sys.catalog]: This master's current role is: LEADER
I20260709 12:58:18.563110  1032 sys_catalog.cc:458] T 00000000000000000000000000000000 P 851f14c22aa944d2ac3a9e0031eea552 [sys.catalog]: This master's current role is: LEADER
I20260709 12:58:18.566638  1039 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260709 12:58:18.571971  1039 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260709 12:58:18.574358   431 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260709 12:58:18.581063  1039 catalog_manager.cc:1383] Generated new cluster ID: 540e220d09ef440c8d62fe0c33b5d38a
I20260709 12:58:18.581338  1039 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260709 12:58:18.607647  1039 catalog_manager.cc:1406] Generated new certificate authority record
I20260709 12:58:18.608929  1039 catalog_manager.cc:1540] Loading token signing keys...
I20260709 12:58:18.632376  1039 catalog_manager.cc:6070] T 00000000000000000000000000000000 P 851f14c22aa944d2ac3a9e0031eea552: Generated new TSK 0
I20260709 12:58:18.633052  1039 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260709 12:58:18.640594   431 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:58:18.647157  1049 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:58:18.648111  1050 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:58:18.650647   431 server_base.cc:1061] running on GCE node
W20260709 12:58:18.650974  1052 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:58:18.651938   431 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:58:18.652173   431 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:58:18.652359   431 hybrid_clock.cc:648] HybridClock initialized: now 1783601898652337 us; error 0 us; skew 500 ppm
I20260709 12:58:18.652951   431 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:58:18.655475   431 webserver.cc:533] Webserver started at http://127.0.107.193:38847/ using document root <none> and password file <none>
I20260709 12:58:18.656000   431 fs_manager.cc:362] Metadata directory not provided
I20260709 12:58:18.656224   431 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:58:18.656523   431 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:58:18.657691   431 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskakPCWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783601878574137-431-0/minicluster-data/ts-0-root/instance:
uuid: "d88b316b7ecd49f29219068c5fe51a26"
format_stamp: "Formatted at 2026-07-09 12:58:18 on dist-test-slave-nvvq"
I20260709 12:58:18.662197   431 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.006s	sys 0.000s
I20260709 12:58:18.665537  1057 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:58:18.666194   431 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.003s	sys 0.000s
I20260709 12:58:18.666637   431 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskakPCWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783601878574137-431-0/minicluster-data/ts-0-root
uuid: "d88b316b7ecd49f29219068c5fe51a26"
format_stamp: "Formatted at 2026-07-09 12:58:18 on dist-test-slave-nvvq"
I20260709 12:58:18.666921   431 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskakPCWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783601878574137-431-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskakPCWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783601878574137-431-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskakPCWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783601878574137-431-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:58:18.682124   431 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:58:18.683431   431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:58:18.684968   431 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:58:18.692430   431 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:58:18.692656   431 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:58:18.692941   431 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:58:18.693176   431 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:58:18.739231   431 rpc_server.cc:307] RPC server started. Bound to: 127.0.107.193:38591
I20260709 12:58:18.739347  1119 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.107.193:38591 every 8 connection(s)
I20260709 12:58:18.744652   431 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:58:18.751662  1124 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:58:18.752691  1125 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:58:18.756184  1127 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:58:18.758119   431 server_base.cc:1061] running on GCE node
I20260709 12:58:18.759351   431 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:58:18.759577   431 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:58:18.759765   431 hybrid_clock.cc:648] HybridClock initialized: now 1783601898759744 us; error 0 us; skew 500 ppm
I20260709 12:58:18.760416   431 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:58:18.761844  1120 heartbeater.cc:344] Connected to a master server at 127.0.107.254:33267
I20260709 12:58:18.762209  1120 heartbeater.cc:461] Registering TS with master...
I20260709 12:58:18.763058  1120 heartbeater.cc:507] Master 127.0.107.254:33267 requested a full tablet report, sending...
I20260709 12:58:18.763401   431 webserver.cc:533] Webserver started at http://127.0.107.194:41627/ using document root <none> and password file <none>
I20260709 12:58:18.764032   431 fs_manager.cc:362] Metadata directory not provided
I20260709 12:58:18.764319   431 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:58:18.764691   431 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:58:18.765779   993 ts_manager.cc:194] Registered new tserver with Master: d88b316b7ecd49f29219068c5fe51a26 (127.0.107.193:38591)
I20260709 12:58:18.766460   431 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskakPCWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783601878574137-431-0/minicluster-data/ts-1-root/instance:
uuid: "b3afae27e8744ee6921cd67f805e3e4a"
format_stamp: "Formatted at 2026-07-09 12:58:18 on dist-test-slave-nvvq"
I20260709 12:58:18.768385   993 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:52564
I20260709 12:58:18.772415   431 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.005s	sys 0.000s
I20260709 12:58:18.775902  1132 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:58:18.776635   431 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.003s	sys 0.000s
I20260709 12:58:18.776913   431 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskakPCWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783601878574137-431-0/minicluster-data/ts-1-root
uuid: "b3afae27e8744ee6921cd67f805e3e4a"
format_stamp: "Formatted at 2026-07-09 12:58:18 on dist-test-slave-nvvq"
I20260709 12:58:18.777185   431 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskakPCWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783601878574137-431-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskakPCWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783601878574137-431-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskakPCWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783601878574137-431-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:58:18.812264   431 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:58:18.813730   431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:58:18.815429   431 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:58:18.817790   431 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:58:18.818013   431 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:58:18.818259   431 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:58:18.818430   431 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:58:18.889566   431 rpc_server.cc:307] RPC server started. Bound to: 127.0.107.194:36785
I20260709 12:58:18.889652  1195 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.107.194:36785 every 8 connection(s)
I20260709 12:58:18.896721   431 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:58:18.906383  1199 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:58:18.908239  1200 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:58:18.909786   431 server_base.cc:1061] running on GCE node
W20260709 12:58:18.910099  1202 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:58:18.911432   431 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:58:18.911733   431 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:58:18.912043   431 hybrid_clock.cc:648] HybridClock initialized: now 1783601898912026 us; error 0 us; skew 500 ppm
I20260709 12:58:18.912776   431 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:58:18.913264  1196 heartbeater.cc:344] Connected to a master server at 127.0.107.254:33267
I20260709 12:58:18.913668  1196 heartbeater.cc:461] Registering TS with master...
I20260709 12:58:18.914711  1196 heartbeater.cc:507] Master 127.0.107.254:33267 requested a full tablet report, sending...
I20260709 12:58:18.916020   431 webserver.cc:533] Webserver started at http://127.0.107.195:41257/ using document root <none> and password file <none>
I20260709 12:58:18.916564   431 fs_manager.cc:362] Metadata directory not provided
I20260709 12:58:18.916802   431 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:58:18.917171   431 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:58:18.917367   993 ts_manager.cc:194] Registered new tserver with Master: b3afae27e8744ee6921cd67f805e3e4a (127.0.107.194:36785)
I20260709 12:58:18.918843   431 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskakPCWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783601878574137-431-0/minicluster-data/ts-2-root/instance:
uuid: "d0e95822debe474ba6a700c661aa262f"
format_stamp: "Formatted at 2026-07-09 12:58:18 on dist-test-slave-nvvq"
I20260709 12:58:18.919340   993 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:52566
I20260709 12:58:18.924196   431 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.005s	sys 0.000s
I20260709 12:58:18.927757  1207 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:58:18.928498   431 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.003s	sys 0.000s
I20260709 12:58:18.928789   431 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskakPCWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783601878574137-431-0/minicluster-data/ts-2-root
uuid: "d0e95822debe474ba6a700c661aa262f"
format_stamp: "Formatted at 2026-07-09 12:58:18 on dist-test-slave-nvvq"
I20260709 12:58:18.929076   431 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskakPCWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783601878574137-431-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskakPCWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783601878574137-431-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskakPCWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783601878574137-431-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:58:18.957235   431 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:58:18.958882   431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:58:18.960973   431 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:58:18.963385   431 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:58:18.963689   431 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:58:18.963946   431 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:58:18.964140   431 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:58:19.010211   431 rpc_server.cc:307] RPC server started. Bound to: 127.0.107.195:40863
I20260709 12:58:19.010322  1269 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.107.195:40863 every 8 connection(s)
I20260709 12:58:19.025408  1270 heartbeater.cc:344] Connected to a master server at 127.0.107.254:33267
I20260709 12:58:19.025820  1270 heartbeater.cc:461] Registering TS with master...
I20260709 12:58:19.026713  1270 heartbeater.cc:507] Master 127.0.107.254:33267 requested a full tablet report, sending...
I20260709 12:58:19.028712   993 ts_manager.cc:194] Registered new tserver with Master: d0e95822debe474ba6a700c661aa262f (127.0.107.195:40863)
I20260709 12:58:19.029435   431 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.015722177s
I20260709 12:58:19.030262   993 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:52578
I20260709 12:58:19.057658   993 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52592:
name: "test-workload"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
  rows: "\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s""\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s"
  indirect_data: """"
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W20260709 12:58:19.061210   993 catalog_manager.cc:7077] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260709 12:58:19.154680  1160 tablet_service.cc:1511] Processing CreateTablet for tablet 759e30de7d7b4bc6980dbf8c4ba8031f (DEFAULT_TABLE table=test-workload [id=222f68d8d9af46b5ba25d626416da003]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260709 12:58:19.157007  1160 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 759e30de7d7b4bc6980dbf8c4ba8031f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:19.159901  1158 tablet_service.cc:1511] Processing CreateTablet for tablet 1935c9164755416a965cccf80900d941 (DEFAULT_TABLE table=test-workload [id=222f68d8d9af46b5ba25d626416da003]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260709 12:58:19.160310  1235 tablet_service.cc:1511] Processing CreateTablet for tablet 6198a4398a3140e8ad9f1a158956fe3e (DEFAULT_TABLE table=test-workload [id=222f68d8d9af46b5ba25d626416da003]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260709 12:58:19.161405  1158 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1935c9164755416a965cccf80900d941. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:19.161605  1235 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6198a4398a3140e8ad9f1a158956fe3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:19.161252  1233 tablet_service.cc:1511] Processing CreateTablet for tablet 73e09365f8cd4513a4600030950af811 (DEFAULT_TABLE table=test-workload [id=222f68d8d9af46b5ba25d626416da003]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260709 12:58:19.162777  1233 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 73e09365f8cd4513a4600030950af811. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:19.163720  1161 tablet_service.cc:1511] Processing CreateTablet for tablet 6198a4398a3140e8ad9f1a158956fe3e (DEFAULT_TABLE table=test-workload [id=222f68d8d9af46b5ba25d626416da003]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260709 12:58:19.165105  1161 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6198a4398a3140e8ad9f1a158956fe3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:19.167580  1159 tablet_service.cc:1511] Processing CreateTablet for tablet 73e09365f8cd4513a4600030950af811 (DEFAULT_TABLE table=test-workload [id=222f68d8d9af46b5ba25d626416da003]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260709 12:58:19.167821  1234 tablet_service.cc:1511] Processing CreateTablet for tablet 759e30de7d7b4bc6980dbf8c4ba8031f (DEFAULT_TABLE table=test-workload [id=222f68d8d9af46b5ba25d626416da003]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260709 12:58:19.168987  1159 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 73e09365f8cd4513a4600030950af811. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:19.169354  1234 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 759e30de7d7b4bc6980dbf8c4ba8031f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:19.178301  1232 tablet_service.cc:1511] Processing CreateTablet for tablet 1935c9164755416a965cccf80900d941 (DEFAULT_TABLE table=test-workload [id=222f68d8d9af46b5ba25d626416da003]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260709 12:58:19.179783  1232 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1935c9164755416a965cccf80900d941. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:19.180470  1157 tablet_service.cc:1511] Processing CreateTablet for tablet 8ee1d07b2c9444f8ad28921d69d297bf (DEFAULT_TABLE table=test-workload [id=222f68d8d9af46b5ba25d626416da003]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260709 12:58:19.181865  1157 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ee1d07b2c9444f8ad28921d69d297bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:19.182053  1156 tablet_service.cc:1511] Processing CreateTablet for tablet 2cde8b0b8c5c46e78f405744f084c7c6 (DEFAULT_TABLE table=test-workload [id=222f68d8d9af46b5ba25d626416da003]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260709 12:58:19.183534  1156 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2cde8b0b8c5c46e78f405744f084c7c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:19.183763  1228 tablet_service.cc:1511] Processing CreateTablet for tablet d87ed5c1a25e40f8b7aa82d33ca30bf1 (DEFAULT_TABLE table=test-workload [id=222f68d8d9af46b5ba25d626416da003]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260709 12:58:19.185232  1228 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d87ed5c1a25e40f8b7aa82d33ca30bf1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:19.185803  1230 tablet_service.cc:1511] Processing CreateTablet for tablet 2cde8b0b8c5c46e78f405744f084c7c6 (DEFAULT_TABLE table=test-workload [id=222f68d8d9af46b5ba25d626416da003]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260709 12:58:19.187183  1230 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2cde8b0b8c5c46e78f405744f084c7c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:19.189555  1227 tablet_service.cc:1511] Processing CreateTablet for tablet c6480f4fe1b54658b39e73baa7c719fd (DEFAULT_TABLE table=test-workload [id=222f68d8d9af46b5ba25d626416da003]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260709 12:58:19.190143  1155 tablet_service.cc:1511] Processing CreateTablet for tablet 2685bf6c4fd043a481d1d05136dbbd5e (DEFAULT_TABLE table=test-workload [id=222f68d8d9af46b5ba25d626416da003]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260709 12:58:19.190994  1227 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6480f4fe1b54658b39e73baa7c719fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:19.191782  1229 tablet_service.cc:1511] Processing CreateTablet for tablet 2685bf6c4fd043a481d1d05136dbbd5e (DEFAULT_TABLE table=test-workload [id=222f68d8d9af46b5ba25d626416da003]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260709 12:58:19.191642  1155 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2685bf6c4fd043a481d1d05136dbbd5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:19.193183  1229 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2685bf6c4fd043a481d1d05136dbbd5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:19.192986  1151 tablet_service.cc:1511] Processing CreateTablet for tablet df94d2062ca84496bc46321e6e4012fc (DEFAULT_TABLE table=test-workload [id=222f68d8d9af46b5ba25d626416da003]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260709 12:58:19.196023  1152 tablet_service.cc:1511] Processing CreateTablet for tablet c6480f4fe1b54658b39e73baa7c719fd (DEFAULT_TABLE table=test-workload [id=222f68d8d9af46b5ba25d626416da003]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260709 12:58:19.196205  1226 tablet_service.cc:1511] Processing CreateTablet for tablet df94d2062ca84496bc46321e6e4012fc (DEFAULT_TABLE table=test-workload [id=222f68d8d9af46b5ba25d626416da003]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260709 12:58:19.197682  1226 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df94d2062ca84496bc46321e6e4012fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:19.198630  1154 tablet_service.cc:1511] Processing CreateTablet for tablet d87ed5c1a25e40f8b7aa82d33ca30bf1 (DEFAULT_TABLE table=test-workload [id=222f68d8d9af46b5ba25d626416da003]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260709 12:58:19.183753  1231 tablet_service.cc:1511] Processing CreateTablet for tablet 8ee1d07b2c9444f8ad28921d69d297bf (DEFAULT_TABLE table=test-workload [id=222f68d8d9af46b5ba25d626416da003]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260709 12:58:19.194343  1151 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df94d2062ca84496bc46321e6e4012fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:19.205448  1231 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ee1d07b2c9444f8ad28921d69d297bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:19.205438  1152 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6480f4fe1b54658b39e73baa7c719fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:19.215296  1154 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d87ed5c1a25e40f8b7aa82d33ca30bf1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:19.280006  1084 tablet_service.cc:1511] Processing CreateTablet for tablet 759e30de7d7b4bc6980dbf8c4ba8031f (DEFAULT_TABLE table=test-workload [id=222f68d8d9af46b5ba25d626416da003]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260709 12:58:19.292044  1084 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 759e30de7d7b4bc6980dbf8c4ba8031f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:19.281840  1083 tablet_service.cc:1511] Processing CreateTablet for tablet 73e09365f8cd4513a4600030950af811 (DEFAULT_TABLE table=test-workload [id=222f68d8d9af46b5ba25d626416da003]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260709 12:58:19.295868  1083 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 73e09365f8cd4513a4600030950af811. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:19.283242  1082 tablet_service.cc:1511] Processing CreateTablet for tablet 1935c9164755416a965cccf80900d941 (DEFAULT_TABLE table=test-workload [id=222f68d8d9af46b5ba25d626416da003]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260709 12:58:19.299868  1082 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1935c9164755416a965cccf80900d941. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:19.278366  1085 tablet_service.cc:1511] Processing CreateTablet for tablet 6198a4398a3140e8ad9f1a158956fe3e (DEFAULT_TABLE table=test-workload [id=222f68d8d9af46b5ba25d626416da003]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260709 12:58:19.303568  1085 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6198a4398a3140e8ad9f1a158956fe3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:19.308439  1076 tablet_service.cc:1511] Processing CreateTablet for tablet df94d2062ca84496bc46321e6e4012fc (DEFAULT_TABLE table=test-workload [id=222f68d8d9af46b5ba25d626416da003]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260709 12:58:19.309768  1076 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df94d2062ca84496bc46321e6e4012fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:19.285868  1080 tablet_service.cc:1511] Processing CreateTablet for tablet 2cde8b0b8c5c46e78f405744f084c7c6 (DEFAULT_TABLE table=test-workload [id=222f68d8d9af46b5ba25d626416da003]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260709 12:58:19.284579  1081 tablet_service.cc:1511] Processing CreateTablet for tablet 8ee1d07b2c9444f8ad28921d69d297bf (DEFAULT_TABLE table=test-workload [id=222f68d8d9af46b5ba25d626416da003]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260709 12:58:19.290113  1077 tablet_service.cc:1511] Processing CreateTablet for tablet c6480f4fe1b54658b39e73baa7c719fd (DEFAULT_TABLE table=test-workload [id=222f68d8d9af46b5ba25d626416da003]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260709 12:58:19.332891  1080 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2cde8b0b8c5c46e78f405744f084c7c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:19.333709  1077 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6480f4fe1b54658b39e73baa7c719fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:19.287715  1078 tablet_service.cc:1511] Processing CreateTablet for tablet d87ed5c1a25e40f8b7aa82d33ca30bf1 (DEFAULT_TABLE table=test-workload [id=222f68d8d9af46b5ba25d626416da003]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260709 12:58:19.335330  1078 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d87ed5c1a25e40f8b7aa82d33ca30bf1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:19.335988  1286 tablet_bootstrap.cc:492] T 2685bf6c4fd043a481d1d05136dbbd5e P b3afae27e8744ee6921cd67f805e3e4a: Bootstrap starting.
I20260709 12:58:19.339000  1081 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ee1d07b2c9444f8ad28921d69d297bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:19.288820  1079 tablet_service.cc:1511] Processing CreateTablet for tablet 2685bf6c4fd043a481d1d05136dbbd5e (DEFAULT_TABLE table=test-workload [id=222f68d8d9af46b5ba25d626416da003]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260709 12:58:19.340201  1079 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2685bf6c4fd043a481d1d05136dbbd5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:19.355644  1286 tablet_bootstrap.cc:654] T 2685bf6c4fd043a481d1d05136dbbd5e P b3afae27e8744ee6921cd67f805e3e4a: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:19.283902  1285 tablet_bootstrap.cc:492] T 759e30de7d7b4bc6980dbf8c4ba8031f P d0e95822debe474ba6a700c661aa262f: Bootstrap starting.
I20260709 12:58:19.368695  1287 tablet_bootstrap.cc:492] T d87ed5c1a25e40f8b7aa82d33ca30bf1 P d88b316b7ecd49f29219068c5fe51a26: Bootstrap starting.
I20260709 12:58:19.376614  1287 tablet_bootstrap.cc:654] T d87ed5c1a25e40f8b7aa82d33ca30bf1 P d88b316b7ecd49f29219068c5fe51a26: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:19.383482  1285 tablet_bootstrap.cc:654] T 759e30de7d7b4bc6980dbf8c4ba8031f P d0e95822debe474ba6a700c661aa262f: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:19.407116  1286 tablet_bootstrap.cc:492] T 2685bf6c4fd043a481d1d05136dbbd5e P b3afae27e8744ee6921cd67f805e3e4a: No bootstrap required, opened a new log
I20260709 12:58:19.407747  1286 ts_tablet_manager.cc:1403] T 2685bf6c4fd043a481d1d05136dbbd5e P b3afae27e8744ee6921cd67f805e3e4a: Time spent bootstrapping tablet: real 0.072s	user 0.018s	sys 0.014s
I20260709 12:58:19.408766  1287 tablet_bootstrap.cc:492] T d87ed5c1a25e40f8b7aa82d33ca30bf1 P d88b316b7ecd49f29219068c5fe51a26: No bootstrap required, opened a new log
I20260709 12:58:19.408771  1285 tablet_bootstrap.cc:492] T 759e30de7d7b4bc6980dbf8c4ba8031f P d0e95822debe474ba6a700c661aa262f: No bootstrap required, opened a new log
I20260709 12:58:19.409312  1287 ts_tablet_manager.cc:1403] T d87ed5c1a25e40f8b7aa82d33ca30bf1 P d88b316b7ecd49f29219068c5fe51a26: Time spent bootstrapping tablet: real 0.041s	user 0.006s	sys 0.009s
I20260709 12:58:19.409400  1285 ts_tablet_manager.cc:1403] T 759e30de7d7b4bc6980dbf8c4ba8031f P d0e95822debe474ba6a700c661aa262f: Time spent bootstrapping tablet: real 0.126s	user 0.012s	sys 0.004s
I20260709 12:58:19.411074  1286 raft_consensus.cc:359] T 2685bf6c4fd043a481d1d05136dbbd5e P b3afae27e8744ee6921cd67f805e3e4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:19.412173  1286 raft_consensus.cc:385] T 2685bf6c4fd043a481d1d05136dbbd5e P b3afae27e8744ee6921cd67f805e3e4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:19.411947  1287 raft_consensus.cc:359] T d87ed5c1a25e40f8b7aa82d33ca30bf1 P d88b316b7ecd49f29219068c5fe51a26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:19.412550  1286 raft_consensus.cc:740] T 2685bf6c4fd043a481d1d05136dbbd5e P b3afae27e8744ee6921cd67f805e3e4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3afae27e8744ee6921cd67f805e3e4a, State: Initialized, Role: FOLLOWER
I20260709 12:58:19.412603  1285 raft_consensus.cc:359] T 759e30de7d7b4bc6980dbf8c4ba8031f P d0e95822debe474ba6a700c661aa262f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:19.413468  1285 raft_consensus.cc:385] T 759e30de7d7b4bc6980dbf8c4ba8031f P d0e95822debe474ba6a700c661aa262f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:19.413836  1285 raft_consensus.cc:740] T 759e30de7d7b4bc6980dbf8c4ba8031f P d0e95822debe474ba6a700c661aa262f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0e95822debe474ba6a700c661aa262f, State: Initialized, Role: FOLLOWER
I20260709 12:58:19.413586  1286 consensus_queue.cc:260] T 2685bf6c4fd043a481d1d05136dbbd5e P b3afae27e8744ee6921cd67f805e3e4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:19.412719  1287 raft_consensus.cc:385] T d87ed5c1a25e40f8b7aa82d33ca30bf1 P d88b316b7ecd49f29219068c5fe51a26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:19.414616  1287 raft_consensus.cc:740] T d87ed5c1a25e40f8b7aa82d33ca30bf1 P d88b316b7ecd49f29219068c5fe51a26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d88b316b7ecd49f29219068c5fe51a26, State: Initialized, Role: FOLLOWER
I20260709 12:58:19.414796  1285 consensus_queue.cc:260] T 759e30de7d7b4bc6980dbf8c4ba8031f P d0e95822debe474ba6a700c661aa262f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:19.415555  1287 consensus_queue.cc:260] T d87ed5c1a25e40f8b7aa82d33ca30bf1 P d88b316b7ecd49f29219068c5fe51a26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:19.416033  1196 heartbeater.cc:499] Master 127.0.107.254:33267 was elected leader, sending a full tablet report...
I20260709 12:58:19.416991  1286 ts_tablet_manager.cc:1434] T 2685bf6c4fd043a481d1d05136dbbd5e P b3afae27e8744ee6921cd67f805e3e4a: Time spent starting tablet: real 0.009s	user 0.009s	sys 0.000s
I20260709 12:58:19.418187  1286 tablet_bootstrap.cc:492] T 73e09365f8cd4513a4600030950af811 P b3afae27e8744ee6921cd67f805e3e4a: Bootstrap starting.
I20260709 12:58:19.424039  1120 heartbeater.cc:499] Master 127.0.107.254:33267 was elected leader, sending a full tablet report...
I20260709 12:58:19.424278  1287 ts_tablet_manager.cc:1434] T d87ed5c1a25e40f8b7aa82d33ca30bf1 P d88b316b7ecd49f29219068c5fe51a26: Time spent starting tablet: real 0.015s	user 0.010s	sys 0.004s
I20260709 12:58:19.425209  1287 tablet_bootstrap.cc:492] T 1935c9164755416a965cccf80900d941 P d88b316b7ecd49f29219068c5fe51a26: Bootstrap starting.
I20260709 12:58:19.426024  1286 tablet_bootstrap.cc:654] T 73e09365f8cd4513a4600030950af811 P b3afae27e8744ee6921cd67f805e3e4a: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:19.433015  1287 tablet_bootstrap.cc:654] T 1935c9164755416a965cccf80900d941 P d88b316b7ecd49f29219068c5fe51a26: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:19.435505  1285 ts_tablet_manager.cc:1434] T 759e30de7d7b4bc6980dbf8c4ba8031f P d0e95822debe474ba6a700c661aa262f: Time spent starting tablet: real 0.026s	user 0.002s	sys 0.011s
I20260709 12:58:19.438346  1286 tablet_bootstrap.cc:492] T 73e09365f8cd4513a4600030950af811 P b3afae27e8744ee6921cd67f805e3e4a: No bootstrap required, opened a new log
I20260709 12:58:19.438902  1286 ts_tablet_manager.cc:1403] T 73e09365f8cd4513a4600030950af811 P b3afae27e8744ee6921cd67f805e3e4a: Time spent bootstrapping tablet: real 0.021s	user 0.017s	sys 0.003s
I20260709 12:58:19.439085  1270 heartbeater.cc:499] Master 127.0.107.254:33267 was elected leader, sending a full tablet report...
I20260709 12:58:19.442075  1286 raft_consensus.cc:359] T 73e09365f8cd4513a4600030950af811 P b3afae27e8744ee6921cd67f805e3e4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:19.442870  1286 raft_consensus.cc:385] T 73e09365f8cd4513a4600030950af811 P b3afae27e8744ee6921cd67f805e3e4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:19.443185  1286 raft_consensus.cc:740] T 73e09365f8cd4513a4600030950af811 P b3afae27e8744ee6921cd67f805e3e4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3afae27e8744ee6921cd67f805e3e4a, State: Initialized, Role: FOLLOWER
I20260709 12:58:19.444197  1286 consensus_queue.cc:260] T 73e09365f8cd4513a4600030950af811 P b3afae27e8744ee6921cd67f805e3e4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:19.445075  1287 tablet_bootstrap.cc:492] T 1935c9164755416a965cccf80900d941 P d88b316b7ecd49f29219068c5fe51a26: No bootstrap required, opened a new log
I20260709 12:58:19.445518  1287 ts_tablet_manager.cc:1403] T 1935c9164755416a965cccf80900d941 P d88b316b7ecd49f29219068c5fe51a26: Time spent bootstrapping tablet: real 0.021s	user 0.008s	sys 0.009s
I20260709 12:58:19.446769  1286 ts_tablet_manager.cc:1434] T 73e09365f8cd4513a4600030950af811 P b3afae27e8744ee6921cd67f805e3e4a: Time spent starting tablet: real 0.008s	user 0.008s	sys 0.000s
I20260709 12:58:19.447754  1286 tablet_bootstrap.cc:492] T 8ee1d07b2c9444f8ad28921d69d297bf P b3afae27e8744ee6921cd67f805e3e4a: Bootstrap starting.
I20260709 12:58:19.448215  1285 tablet_bootstrap.cc:492] T 6198a4398a3140e8ad9f1a158956fe3e P d0e95822debe474ba6a700c661aa262f: Bootstrap starting.
I20260709 12:58:19.448608  1287 raft_consensus.cc:359] T 1935c9164755416a965cccf80900d941 P d88b316b7ecd49f29219068c5fe51a26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:19.449126  1287 raft_consensus.cc:385] T 1935c9164755416a965cccf80900d941 P d88b316b7ecd49f29219068c5fe51a26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:19.449303  1287 raft_consensus.cc:740] T 1935c9164755416a965cccf80900d941 P d88b316b7ecd49f29219068c5fe51a26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d88b316b7ecd49f29219068c5fe51a26, State: Initialized, Role: FOLLOWER
I20260709 12:58:19.449890  1287 consensus_queue.cc:260] T 1935c9164755416a965cccf80900d941 P d88b316b7ecd49f29219068c5fe51a26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:19.451668  1287 ts_tablet_manager.cc:1434] T 1935c9164755416a965cccf80900d941 P d88b316b7ecd49f29219068c5fe51a26: Time spent starting tablet: real 0.006s	user 0.007s	sys 0.000s
I20260709 12:58:19.452417  1287 tablet_bootstrap.cc:492] T 6198a4398a3140e8ad9f1a158956fe3e P d88b316b7ecd49f29219068c5fe51a26: Bootstrap starting.
I20260709 12:58:19.455245  1286 tablet_bootstrap.cc:654] T 8ee1d07b2c9444f8ad28921d69d297bf P b3afae27e8744ee6921cd67f805e3e4a: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:19.457099  1285 tablet_bootstrap.cc:654] T 6198a4398a3140e8ad9f1a158956fe3e P d0e95822debe474ba6a700c661aa262f: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:19.458338  1287 tablet_bootstrap.cc:654] T 6198a4398a3140e8ad9f1a158956fe3e P d88b316b7ecd49f29219068c5fe51a26: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:19.467772  1286 tablet_bootstrap.cc:492] T 8ee1d07b2c9444f8ad28921d69d297bf P b3afae27e8744ee6921cd67f805e3e4a: No bootstrap required, opened a new log
I20260709 12:58:19.468230  1286 ts_tablet_manager.cc:1403] T 8ee1d07b2c9444f8ad28921d69d297bf P b3afae27e8744ee6921cd67f805e3e4a: Time spent bootstrapping tablet: real 0.021s	user 0.015s	sys 0.004s
I20260709 12:58:19.471875  1286 raft_consensus.cc:359] T 8ee1d07b2c9444f8ad28921d69d297bf P b3afae27e8744ee6921cd67f805e3e4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:19.472836  1286 raft_consensus.cc:385] T 8ee1d07b2c9444f8ad28921d69d297bf P b3afae27e8744ee6921cd67f805e3e4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:19.473215  1286 raft_consensus.cc:740] T 8ee1d07b2c9444f8ad28921d69d297bf P b3afae27e8744ee6921cd67f805e3e4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3afae27e8744ee6921cd67f805e3e4a, State: Initialized, Role: FOLLOWER
I20260709 12:58:19.475685  1286 consensus_queue.cc:260] T 8ee1d07b2c9444f8ad28921d69d297bf P b3afae27e8744ee6921cd67f805e3e4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:19.477176  1287 tablet_bootstrap.cc:492] T 6198a4398a3140e8ad9f1a158956fe3e P d88b316b7ecd49f29219068c5fe51a26: No bootstrap required, opened a new log
I20260709 12:58:19.477734  1287 ts_tablet_manager.cc:1403] T 6198a4398a3140e8ad9f1a158956fe3e P d88b316b7ecd49f29219068c5fe51a26: Time spent bootstrapping tablet: real 0.025s	user 0.009s	sys 0.013s
I20260709 12:58:19.478219  1286 ts_tablet_manager.cc:1434] T 8ee1d07b2c9444f8ad28921d69d297bf P b3afae27e8744ee6921cd67f805e3e4a: Time spent starting tablet: real 0.010s	user 0.007s	sys 0.000s
I20260709 12:58:19.479985  1286 tablet_bootstrap.cc:492] T 6198a4398a3140e8ad9f1a158956fe3e P b3afae27e8744ee6921cd67f805e3e4a: Bootstrap starting.
I20260709 12:58:19.481307  1285 tablet_bootstrap.cc:492] T 6198a4398a3140e8ad9f1a158956fe3e P d0e95822debe474ba6a700c661aa262f: No bootstrap required, opened a new log
I20260709 12:58:19.480929  1287 raft_consensus.cc:359] T 6198a4398a3140e8ad9f1a158956fe3e P d88b316b7ecd49f29219068c5fe51a26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:19.481696  1287 raft_consensus.cc:385] T 6198a4398a3140e8ad9f1a158956fe3e P d88b316b7ecd49f29219068c5fe51a26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:19.481818  1285 ts_tablet_manager.cc:1403] T 6198a4398a3140e8ad9f1a158956fe3e P d0e95822debe474ba6a700c661aa262f: Time spent bootstrapping tablet: real 0.034s	user 0.009s	sys 0.010s
I20260709 12:58:19.482029  1287 raft_consensus.cc:740] T 6198a4398a3140e8ad9f1a158956fe3e P d88b316b7ecd49f29219068c5fe51a26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d88b316b7ecd49f29219068c5fe51a26, State: Initialized, Role: FOLLOWER
I20260709 12:58:19.483103  1287 consensus_queue.cc:260] T 6198a4398a3140e8ad9f1a158956fe3e P d88b316b7ecd49f29219068c5fe51a26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:19.484993  1285 raft_consensus.cc:359] T 6198a4398a3140e8ad9f1a158956fe3e P d0e95822debe474ba6a700c661aa262f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:19.485713  1285 raft_consensus.cc:385] T 6198a4398a3140e8ad9f1a158956fe3e P d0e95822debe474ba6a700c661aa262f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:19.486011  1285 raft_consensus.cc:740] T 6198a4398a3140e8ad9f1a158956fe3e P d0e95822debe474ba6a700c661aa262f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0e95822debe474ba6a700c661aa262f, State: Initialized, Role: FOLLOWER
I20260709 12:58:19.486903  1285 consensus_queue.cc:260] T 6198a4398a3140e8ad9f1a158956fe3e P d0e95822debe474ba6a700c661aa262f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:19.487323  1286 tablet_bootstrap.cc:654] T 6198a4398a3140e8ad9f1a158956fe3e P b3afae27e8744ee6921cd67f805e3e4a: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:19.487349  1287 ts_tablet_manager.cc:1434] T 6198a4398a3140e8ad9f1a158956fe3e P d88b316b7ecd49f29219068c5fe51a26: Time spent starting tablet: real 0.009s	user 0.002s	sys 0.007s
I20260709 12:58:19.488471  1287 tablet_bootstrap.cc:492] T df94d2062ca84496bc46321e6e4012fc P d88b316b7ecd49f29219068c5fe51a26: Bootstrap starting.
I20260709 12:58:19.489341  1285 ts_tablet_manager.cc:1434] T 6198a4398a3140e8ad9f1a158956fe3e P d0e95822debe474ba6a700c661aa262f: Time spent starting tablet: real 0.007s	user 0.004s	sys 0.004s
I20260709 12:58:19.490300  1285 tablet_bootstrap.cc:492] T 2685bf6c4fd043a481d1d05136dbbd5e P d0e95822debe474ba6a700c661aa262f: Bootstrap starting.
I20260709 12:58:19.496646  1287 tablet_bootstrap.cc:654] T df94d2062ca84496bc46321e6e4012fc P d88b316b7ecd49f29219068c5fe51a26: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:19.499995  1285 tablet_bootstrap.cc:654] T 2685bf6c4fd043a481d1d05136dbbd5e P d0e95822debe474ba6a700c661aa262f: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:19.517726  1291 raft_consensus.cc:493] T 8ee1d07b2c9444f8ad28921d69d297bf P b3afae27e8744ee6921cd67f805e3e4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:19.518308  1291 raft_consensus.cc:515] T 8ee1d07b2c9444f8ad28921d69d297bf P b3afae27e8744ee6921cd67f805e3e4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:19.523304  1291 leader_election.cc:290] T 8ee1d07b2c9444f8ad28921d69d297bf P b3afae27e8744ee6921cd67f805e3e4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d88b316b7ecd49f29219068c5fe51a26 (127.0.107.193:38591), d0e95822debe474ba6a700c661aa262f (127.0.107.195:40863)
I20260709 12:58:19.533332  1292 raft_consensus.cc:493] T 6198a4398a3140e8ad9f1a158956fe3e P d0e95822debe474ba6a700c661aa262f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:19.533874  1292 raft_consensus.cc:515] T 6198a4398a3140e8ad9f1a158956fe3e P d0e95822debe474ba6a700c661aa262f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:19.536751  1292 leader_election.cc:290] T 6198a4398a3140e8ad9f1a158956fe3e P d0e95822debe474ba6a700c661aa262f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b3afae27e8744ee6921cd67f805e3e4a (127.0.107.194:36785), d88b316b7ecd49f29219068c5fe51a26 (127.0.107.193:38591)
W20260709 12:58:19.569564  1041 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:58:19.581377  1293 raft_consensus.cc:493] T 1935c9164755416a965cccf80900d941 P d88b316b7ecd49f29219068c5fe51a26 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:19.582217  1293 raft_consensus.cc:515] T 1935c9164755416a965cccf80900d941 P d88b316b7ecd49f29219068c5fe51a26 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:19.588395  1286 tablet_bootstrap.cc:492] T 6198a4398a3140e8ad9f1a158956fe3e P b3afae27e8744ee6921cd67f805e3e4a: No bootstrap required, opened a new log
I20260709 12:58:19.589150  1286 ts_tablet_manager.cc:1403] T 6198a4398a3140e8ad9f1a158956fe3e P b3afae27e8744ee6921cd67f805e3e4a: Time spent bootstrapping tablet: real 0.109s	user 0.041s	sys 0.056s
I20260709 12:58:19.593423  1286 raft_consensus.cc:359] T 6198a4398a3140e8ad9f1a158956fe3e P b3afae27e8744ee6921cd67f805e3e4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:19.603128  1293 leader_election.cc:290] T 1935c9164755416a965cccf80900d941 P d88b316b7ecd49f29219068c5fe51a26 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b3afae27e8744ee6921cd67f805e3e4a (127.0.107.194:36785), d0e95822debe474ba6a700c661aa262f (127.0.107.195:40863)
I20260709 12:58:19.596975  1286 raft_consensus.cc:385] T 6198a4398a3140e8ad9f1a158956fe3e P b3afae27e8744ee6921cd67f805e3e4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:19.604823  1286 raft_consensus.cc:740] T 6198a4398a3140e8ad9f1a158956fe3e P b3afae27e8744ee6921cd67f805e3e4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3afae27e8744ee6921cd67f805e3e4a, State: Initialized, Role: FOLLOWER
I20260709 12:58:19.607810  1285 tablet_bootstrap.cc:492] T 2685bf6c4fd043a481d1d05136dbbd5e P d0e95822debe474ba6a700c661aa262f: No bootstrap required, opened a new log
I20260709 12:58:19.610860  1285 ts_tablet_manager.cc:1403] T 2685bf6c4fd043a481d1d05136dbbd5e P d0e95822debe474ba6a700c661aa262f: Time spent bootstrapping tablet: real 0.121s	user 0.009s	sys 0.032s
I20260709 12:58:19.612296  1286 consensus_queue.cc:260] T 6198a4398a3140e8ad9f1a158956fe3e P b3afae27e8744ee6921cd67f805e3e4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:19.618381  1285 raft_consensus.cc:359] T 2685bf6c4fd043a481d1d05136dbbd5e P d0e95822debe474ba6a700c661aa262f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:19.618660  1245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ee1d07b2c9444f8ad28921d69d297bf" candidate_uuid: "b3afae27e8744ee6921cd67f805e3e4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0e95822debe474ba6a700c661aa262f" is_pre_election: true
I20260709 12:58:19.619987  1285 raft_consensus.cc:385] T 2685bf6c4fd043a481d1d05136dbbd5e P d0e95822debe474ba6a700c661aa262f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260709 12:58:19.622381  1136 leader_election.cc:343] T 8ee1d07b2c9444f8ad28921d69d297bf P b3afae27e8744ee6921cd67f805e3e4a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d0e95822debe474ba6a700c661aa262f (127.0.107.195:40863): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:19.622504  1285 raft_consensus.cc:740] T 2685bf6c4fd043a481d1d05136dbbd5e P d0e95822debe474ba6a700c661aa262f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0e95822debe474ba6a700c661aa262f, State: Initialized, Role: FOLLOWER
I20260709 12:58:19.623958  1285 consensus_queue.cc:260] T 2685bf6c4fd043a481d1d05136dbbd5e P d0e95822debe474ba6a700c661aa262f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:19.622370  1286 ts_tablet_manager.cc:1434] T 6198a4398a3140e8ad9f1a158956fe3e P b3afae27e8744ee6921cd67f805e3e4a: Time spent starting tablet: real 0.033s	user 0.009s	sys 0.004s
I20260709 12:58:19.632689  1286 tablet_bootstrap.cc:492] T 1935c9164755416a965cccf80900d941 P b3afae27e8744ee6921cd67f805e3e4a: Bootstrap starting.
I20260709 12:58:19.627952  1285 ts_tablet_manager.cc:1434] T 2685bf6c4fd043a481d1d05136dbbd5e P d0e95822debe474ba6a700c661aa262f: Time spent starting tablet: real 0.017s	user 0.008s	sys 0.000s
I20260709 12:58:19.635210  1285 tablet_bootstrap.cc:492] T 2cde8b0b8c5c46e78f405744f084c7c6 P d0e95822debe474ba6a700c661aa262f: Bootstrap starting.
I20260709 12:58:19.635339  1095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ee1d07b2c9444f8ad28921d69d297bf" candidate_uuid: "b3afae27e8744ee6921cd67f805e3e4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d88b316b7ecd49f29219068c5fe51a26" is_pre_election: true
I20260709 12:58:19.628059  1287 tablet_bootstrap.cc:492] T df94d2062ca84496bc46321e6e4012fc P d88b316b7ecd49f29219068c5fe51a26: No bootstrap required, opened a new log
W20260709 12:58:19.637629  1133 leader_election.cc:343] T 8ee1d07b2c9444f8ad28921d69d297bf P b3afae27e8744ee6921cd67f805e3e4a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d88b316b7ecd49f29219068c5fe51a26 (127.0.107.193:38591): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:19.638113  1133 leader_election.cc:304] T 8ee1d07b2c9444f8ad28921d69d297bf P b3afae27e8744ee6921cd67f805e3e4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b3afae27e8744ee6921cd67f805e3e4a; no voters: d0e95822debe474ba6a700c661aa262f, d88b316b7ecd49f29219068c5fe51a26
I20260709 12:58:19.639894  1287 ts_tablet_manager.cc:1403] T df94d2062ca84496bc46321e6e4012fc P d88b316b7ecd49f29219068c5fe51a26: Time spent bootstrapping tablet: real 0.152s	user 0.039s	sys 0.076s
I20260709 12:58:19.639808  1291 raft_consensus.cc:2749] T 8ee1d07b2c9444f8ad28921d69d297bf P b3afae27e8744ee6921cd67f805e3e4a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:58:19.641957  1095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6198a4398a3140e8ad9f1a158956fe3e" candidate_uuid: "d0e95822debe474ba6a700c661aa262f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d88b316b7ecd49f29219068c5fe51a26" is_pre_election: true
I20260709 12:58:19.642673  1095 raft_consensus.cc:2468] T 6198a4398a3140e8ad9f1a158956fe3e P d88b316b7ecd49f29219068c5fe51a26 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0e95822debe474ba6a700c661aa262f in term 0.
I20260709 12:58:19.645604  1208 leader_election.cc:304] T 6198a4398a3140e8ad9f1a158956fe3e P d0e95822debe474ba6a700c661aa262f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d0e95822debe474ba6a700c661aa262f, d88b316b7ecd49f29219068c5fe51a26; no voters: 
I20260709 12:58:19.646606  1285 tablet_bootstrap.cc:654] T 2cde8b0b8c5c46e78f405744f084c7c6 P d0e95822debe474ba6a700c661aa262f: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:19.646622  1286 tablet_bootstrap.cc:654] T 1935c9164755416a965cccf80900d941 P b3afae27e8744ee6921cd67f805e3e4a: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:19.647850  1292 raft_consensus.cc:2804] T 6198a4398a3140e8ad9f1a158956fe3e P d0e95822debe474ba6a700c661aa262f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:58:19.648233  1292 raft_consensus.cc:493] T 6198a4398a3140e8ad9f1a158956fe3e P d0e95822debe474ba6a700c661aa262f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:58:19.648566  1292 raft_consensus.cc:3060] T 6198a4398a3140e8ad9f1a158956fe3e P d0e95822debe474ba6a700c661aa262f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:19.650621  1171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6198a4398a3140e8ad9f1a158956fe3e" candidate_uuid: "d0e95822debe474ba6a700c661aa262f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3afae27e8744ee6921cd67f805e3e4a" is_pre_election: true
I20260709 12:58:19.651358  1171 raft_consensus.cc:2468] T 6198a4398a3140e8ad9f1a158956fe3e P b3afae27e8744ee6921cd67f805e3e4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0e95822debe474ba6a700c661aa262f in term 0.
I20260709 12:58:19.652717  1287 raft_consensus.cc:359] T df94d2062ca84496bc46321e6e4012fc P d88b316b7ecd49f29219068c5fe51a26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:19.662684  1245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1935c9164755416a965cccf80900d941" candidate_uuid: "d88b316b7ecd49f29219068c5fe51a26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0e95822debe474ba6a700c661aa262f" is_pre_election: true
I20260709 12:58:19.662495  1292 raft_consensus.cc:515] T 6198a4398a3140e8ad9f1a158956fe3e P d0e95822debe474ba6a700c661aa262f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:19.656980  1287 raft_consensus.cc:385] T df94d2062ca84496bc46321e6e4012fc P d88b316b7ecd49f29219068c5fe51a26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:19.664063  1287 raft_consensus.cc:740] T df94d2062ca84496bc46321e6e4012fc P d88b316b7ecd49f29219068c5fe51a26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d88b316b7ecd49f29219068c5fe51a26, State: Initialized, Role: FOLLOWER
W20260709 12:58:19.665881  1061 leader_election.cc:343] T 1935c9164755416a965cccf80900d941 P d88b316b7ecd49f29219068c5fe51a26 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d0e95822debe474ba6a700c661aa262f (127.0.107.195:40863): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:19.666232  1292 leader_election.cc:290] T 6198a4398a3140e8ad9f1a158956fe3e P d0e95822debe474ba6a700c661aa262f [CANDIDATE]: Term 1 election: Requested vote from peers b3afae27e8744ee6921cd67f805e3e4a (127.0.107.194:36785), d88b316b7ecd49f29219068c5fe51a26 (127.0.107.193:38591)
I20260709 12:58:19.667815  1171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6198a4398a3140e8ad9f1a158956fe3e" candidate_uuid: "d0e95822debe474ba6a700c661aa262f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3afae27e8744ee6921cd67f805e3e4a"
I20260709 12:58:19.668543  1171 raft_consensus.cc:3060] T 6198a4398a3140e8ad9f1a158956fe3e P b3afae27e8744ee6921cd67f805e3e4a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:19.665406  1287 consensus_queue.cc:260] T df94d2062ca84496bc46321e6e4012fc P d88b316b7ecd49f29219068c5fe51a26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:19.669412  1286 tablet_bootstrap.cc:492] T 1935c9164755416a965cccf80900d941 P b3afae27e8744ee6921cd67f805e3e4a: No bootstrap required, opened a new log
I20260709 12:58:19.670063  1286 ts_tablet_manager.cc:1403] T 1935c9164755416a965cccf80900d941 P b3afae27e8744ee6921cd67f805e3e4a: Time spent bootstrapping tablet: real 0.038s	user 0.011s	sys 0.009s
I20260709 12:58:19.670957  1287 ts_tablet_manager.cc:1434] T df94d2062ca84496bc46321e6e4012fc P d88b316b7ecd49f29219068c5fe51a26: Time spent starting tablet: real 0.031s	user 0.006s	sys 0.003s
I20260709 12:58:19.671993  1287 tablet_bootstrap.cc:492] T 2685bf6c4fd043a481d1d05136dbbd5e P d88b316b7ecd49f29219068c5fe51a26: Bootstrap starting.
I20260709 12:58:19.667783  1095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6198a4398a3140e8ad9f1a158956fe3e" candidate_uuid: "d0e95822debe474ba6a700c661aa262f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d88b316b7ecd49f29219068c5fe51a26"
I20260709 12:58:19.673836  1095 raft_consensus.cc:3060] T 6198a4398a3140e8ad9f1a158956fe3e P d88b316b7ecd49f29219068c5fe51a26 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:19.674242  1286 raft_consensus.cc:359] T 1935c9164755416a965cccf80900d941 P b3afae27e8744ee6921cd67f805e3e4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:19.675232  1286 raft_consensus.cc:385] T 1935c9164755416a965cccf80900d941 P b3afae27e8744ee6921cd67f805e3e4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:19.675592  1286 raft_consensus.cc:740] T 1935c9164755416a965cccf80900d941 P b3afae27e8744ee6921cd67f805e3e4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3afae27e8744ee6921cd67f805e3e4a, State: Initialized, Role: FOLLOWER
I20260709 12:58:19.676585  1286 consensus_queue.cc:260] T 1935c9164755416a965cccf80900d941 P b3afae27e8744ee6921cd67f805e3e4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:19.679311  1287 tablet_bootstrap.cc:654] T 2685bf6c4fd043a481d1d05136dbbd5e P d88b316b7ecd49f29219068c5fe51a26: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:19.693025  1095 raft_consensus.cc:2468] T 6198a4398a3140e8ad9f1a158956fe3e P d88b316b7ecd49f29219068c5fe51a26 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0e95822debe474ba6a700c661aa262f in term 1.
I20260709 12:58:19.694058  1171 raft_consensus.cc:2468] T 6198a4398a3140e8ad9f1a158956fe3e P b3afae27e8744ee6921cd67f805e3e4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0e95822debe474ba6a700c661aa262f in term 1.
I20260709 12:58:19.698774  1291 raft_consensus.cc:493] T 73e09365f8cd4513a4600030950af811 P b3afae27e8744ee6921cd67f805e3e4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:19.695083  1286 ts_tablet_manager.cc:1434] T 1935c9164755416a965cccf80900d941 P b3afae27e8744ee6921cd67f805e3e4a: Time spent starting tablet: real 0.025s	user 0.005s	sys 0.006s
I20260709 12:58:19.694761  1208 leader_election.cc:304] T 6198a4398a3140e8ad9f1a158956fe3e P d0e95822debe474ba6a700c661aa262f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d0e95822debe474ba6a700c661aa262f, d88b316b7ecd49f29219068c5fe51a26; no voters: 
I20260709 12:58:19.695518  1170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1935c9164755416a965cccf80900d941" candidate_uuid: "d88b316b7ecd49f29219068c5fe51a26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3afae27e8744ee6921cd67f805e3e4a" is_pre_election: true
I20260709 12:58:19.699302  1291 raft_consensus.cc:515] T 73e09365f8cd4513a4600030950af811 P b3afae27e8744ee6921cd67f805e3e4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:19.699715  1170 raft_consensus.cc:2468] T 1935c9164755416a965cccf80900d941 P b3afae27e8744ee6921cd67f805e3e4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d88b316b7ecd49f29219068c5fe51a26 in term 0.
I20260709 12:58:19.700770  1293 raft_consensus.cc:493] T d87ed5c1a25e40f8b7aa82d33ca30bf1 P d88b316b7ecd49f29219068c5fe51a26 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:19.701215  1286 tablet_bootstrap.cc:492] T 759e30de7d7b4bc6980dbf8c4ba8031f P b3afae27e8744ee6921cd67f805e3e4a: Bootstrap starting.
I20260709 12:58:19.701612  1291 leader_election.cc:290] T 73e09365f8cd4513a4600030950af811 P b3afae27e8744ee6921cd67f805e3e4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d88b316b7ecd49f29219068c5fe51a26 (127.0.107.193:38591), d0e95822debe474ba6a700c661aa262f (127.0.107.195:40863)
I20260709 12:58:19.702194  1292 raft_consensus.cc:2804] T 6198a4398a3140e8ad9f1a158956fe3e P d0e95822debe474ba6a700c661aa262f [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:58:19.701254  1293 raft_consensus.cc:515] T d87ed5c1a25e40f8b7aa82d33ca30bf1 P d88b316b7ecd49f29219068c5fe51a26 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:19.705713  1293 leader_election.cc:290] T d87ed5c1a25e40f8b7aa82d33ca30bf1 P d88b316b7ecd49f29219068c5fe51a26 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b3afae27e8744ee6921cd67f805e3e4a (127.0.107.194:36785), d0e95822debe474ba6a700c661aa262f (127.0.107.195:40863)
I20260709 12:58:19.707813  1095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73e09365f8cd4513a4600030950af811" candidate_uuid: "b3afae27e8744ee6921cd67f805e3e4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d88b316b7ecd49f29219068c5fe51a26" is_pre_election: true
I20260709 12:58:19.708284  1245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d87ed5c1a25e40f8b7aa82d33ca30bf1" candidate_uuid: "d88b316b7ecd49f29219068c5fe51a26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0e95822debe474ba6a700c661aa262f" is_pre_election: true
W20260709 12:58:19.710008  1133 leader_election.cc:343] T 73e09365f8cd4513a4600030950af811 P b3afae27e8744ee6921cd67f805e3e4a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d88b316b7ecd49f29219068c5fe51a26 (127.0.107.193:38591): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:58:19.710532  1061 leader_election.cc:343] T d87ed5c1a25e40f8b7aa82d33ca30bf1 P d88b316b7ecd49f29219068c5fe51a26 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d0e95822debe474ba6a700c661aa262f (127.0.107.195:40863): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:19.712236  1286 tablet_bootstrap.cc:654] T 759e30de7d7b4bc6980dbf8c4ba8031f P b3afae27e8744ee6921cd67f805e3e4a: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:19.712388  1245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73e09365f8cd4513a4600030950af811" candidate_uuid: "b3afae27e8744ee6921cd67f805e3e4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0e95822debe474ba6a700c661aa262f" is_pre_election: true
I20260709 12:58:19.714016  1059 leader_election.cc:304] T 1935c9164755416a965cccf80900d941 P d88b316b7ecd49f29219068c5fe51a26 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b3afae27e8744ee6921cd67f805e3e4a, d88b316b7ecd49f29219068c5fe51a26; no voters: d0e95822debe474ba6a700c661aa262f
I20260709 12:58:19.698239  1287 tablet_bootstrap.cc:492] T 2685bf6c4fd043a481d1d05136dbbd5e P d88b316b7ecd49f29219068c5fe51a26: No bootstrap required, opened a new log
I20260709 12:58:19.715308  1293 raft_consensus.cc:2804] T 1935c9164755416a965cccf80900d941 P d88b316b7ecd49f29219068c5fe51a26 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20260709 12:58:19.716715  1136 leader_election.cc:343] T 73e09365f8cd4513a4600030950af811 P b3afae27e8744ee6921cd67f805e3e4a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d0e95822debe474ba6a700c661aa262f (127.0.107.195:40863): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:19.716801  1171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d87ed5c1a25e40f8b7aa82d33ca30bf1" candidate_uuid: "d88b316b7ecd49f29219068c5fe51a26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3afae27e8744ee6921cd67f805e3e4a" is_pre_election: true
I20260709 12:58:19.718291  1136 leader_election.cc:304] T 73e09365f8cd4513a4600030950af811 P b3afae27e8744ee6921cd67f805e3e4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b3afae27e8744ee6921cd67f805e3e4a; no voters: d0e95822debe474ba6a700c661aa262f, d88b316b7ecd49f29219068c5fe51a26
I20260709 12:58:19.717614  1293 raft_consensus.cc:493] T 1935c9164755416a965cccf80900d941 P d88b316b7ecd49f29219068c5fe51a26 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:58:19.719388  1291 raft_consensus.cc:2749] T 73e09365f8cd4513a4600030950af811 P b3afae27e8744ee6921cd67f805e3e4a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:58:19.719696  1293 raft_consensus.cc:3060] T 1935c9164755416a965cccf80900d941 P d88b316b7ecd49f29219068c5fe51a26 [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:58:19.721153  1059 leader_election.cc:343] T d87ed5c1a25e40f8b7aa82d33ca30bf1 P d88b316b7ecd49f29219068c5fe51a26 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b3afae27e8744ee6921cd67f805e3e4a (127.0.107.194:36785): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:19.721733  1059 leader_election.cc:304] T d87ed5c1a25e40f8b7aa82d33ca30bf1 P d88b316b7ecd49f29219068c5fe51a26 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d88b316b7ecd49f29219068c5fe51a26; no voters: b3afae27e8744ee6921cd67f805e3e4a, d0e95822debe474ba6a700c661aa262f
I20260709 12:58:19.722528  1287 ts_tablet_manager.cc:1403] T 2685bf6c4fd043a481d1d05136dbbd5e P d88b316b7ecd49f29219068c5fe51a26: Time spent bootstrapping tablet: real 0.051s	user 0.016s	sys 0.004s
I20260709 12:58:19.731271  1285 tablet_bootstrap.cc:492] T 2cde8b0b8c5c46e78f405744f084c7c6 P d0e95822debe474ba6a700c661aa262f: No bootstrap required, opened a new log
I20260709 12:58:19.733135  1285 ts_tablet_manager.cc:1403] T 2cde8b0b8c5c46e78f405744f084c7c6 P d0e95822debe474ba6a700c661aa262f: Time spent bootstrapping tablet: real 0.098s	user 0.036s	sys 0.034s
I20260709 12:58:19.735733  1293 raft_consensus.cc:515] T 1935c9164755416a965cccf80900d941 P d88b316b7ecd49f29219068c5fe51a26 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:19.730193  1314 raft_consensus.cc:2749] T d87ed5c1a25e40f8b7aa82d33ca30bf1 P d88b316b7ecd49f29219068c5fe51a26 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:58:19.738161  1285 raft_consensus.cc:359] T 2cde8b0b8c5c46e78f405744f084c7c6 P d0e95822debe474ba6a700c661aa262f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:19.739153  1171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1935c9164755416a965cccf80900d941" candidate_uuid: "d88b316b7ecd49f29219068c5fe51a26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3afae27e8744ee6921cd67f805e3e4a"
I20260709 12:58:19.739470  1285 raft_consensus.cc:385] T 2cde8b0b8c5c46e78f405744f084c7c6 P d0e95822debe474ba6a700c661aa262f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:19.739763  1171 raft_consensus.cc:3060] T 1935c9164755416a965cccf80900d941 P b3afae27e8744ee6921cd67f805e3e4a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:19.740067  1285 raft_consensus.cc:740] T 2cde8b0b8c5c46e78f405744f084c7c6 P d0e95822debe474ba6a700c661aa262f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0e95822debe474ba6a700c661aa262f, State: Initialized, Role: FOLLOWER
I20260709 12:58:19.740242  1287 raft_consensus.cc:359] T 2685bf6c4fd043a481d1d05136dbbd5e P d88b316b7ecd49f29219068c5fe51a26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:19.741051  1287 raft_consensus.cc:385] T 2685bf6c4fd043a481d1d05136dbbd5e P d88b316b7ecd49f29219068c5fe51a26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:19.741360  1287 raft_consensus.cc:740] T 2685bf6c4fd043a481d1d05136dbbd5e P d88b316b7ecd49f29219068c5fe51a26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d88b316b7ecd49f29219068c5fe51a26, State: Initialized, Role: FOLLOWER
I20260709 12:58:19.741374  1285 consensus_queue.cc:260] T 2cde8b0b8c5c46e78f405744f084c7c6 P d0e95822debe474ba6a700c661aa262f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:19.742471  1292 raft_consensus.cc:697] T 6198a4398a3140e8ad9f1a158956fe3e P d0e95822debe474ba6a700c661aa262f [term 1 LEADER]: Becoming Leader. State: Replica: d0e95822debe474ba6a700c661aa262f, State: Running, Role: LEADER
I20260709 12:58:19.743108  1293 leader_election.cc:290] T 1935c9164755416a965cccf80900d941 P d88b316b7ecd49f29219068c5fe51a26 [CANDIDATE]: Term 1 election: Requested vote from peers b3afae27e8744ee6921cd67f805e3e4a (127.0.107.194:36785), d0e95822debe474ba6a700c661aa262f (127.0.107.195:40863)
I20260709 12:58:19.743526  1292 consensus_queue.cc:237] T 6198a4398a3140e8ad9f1a158956fe3e P d0e95822debe474ba6a700c661aa262f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:19.744292  1245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1935c9164755416a965cccf80900d941" candidate_uuid: "d88b316b7ecd49f29219068c5fe51a26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0e95822debe474ba6a700c661aa262f"
W20260709 12:58:19.745828  1061 leader_election.cc:343] T 1935c9164755416a965cccf80900d941 P d88b316b7ecd49f29219068c5fe51a26 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d0e95822debe474ba6a700c661aa262f (127.0.107.195:40863): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:19.746493  1287 consensus_queue.cc:260] T 2685bf6c4fd043a481d1d05136dbbd5e P d88b316b7ecd49f29219068c5fe51a26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:19.753319  1285 ts_tablet_manager.cc:1434] T 2cde8b0b8c5c46e78f405744f084c7c6 P d0e95822debe474ba6a700c661aa262f: Time spent starting tablet: real 0.020s	user 0.008s	sys 0.008s
I20260709 12:58:19.753547   990 catalog_manager.cc:5697] T 6198a4398a3140e8ad9f1a158956fe3e P d0e95822debe474ba6a700c661aa262f reported cstate change: term changed from 0 to 1, leader changed from <none> to d0e95822debe474ba6a700c661aa262f (127.0.107.195). New cstate: current_term: 1 leader_uuid: "d0e95822debe474ba6a700c661aa262f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } health_report { overall_health: HEALTHY } } }
I20260709 12:58:19.754412  1285 tablet_bootstrap.cc:492] T 1935c9164755416a965cccf80900d941 P d0e95822debe474ba6a700c661aa262f: Bootstrap starting.
I20260709 12:58:19.756254  1287 ts_tablet_manager.cc:1434] T 2685bf6c4fd043a481d1d05136dbbd5e P d88b316b7ecd49f29219068c5fe51a26: Time spent starting tablet: real 0.032s	user 0.002s	sys 0.005s
I20260709 12:58:19.757328  1287 tablet_bootstrap.cc:492] T 2cde8b0b8c5c46e78f405744f084c7c6 P d88b316b7ecd49f29219068c5fe51a26: Bootstrap starting.
I20260709 12:58:19.761977  1285 tablet_bootstrap.cc:654] T 1935c9164755416a965cccf80900d941 P d0e95822debe474ba6a700c661aa262f: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:19.764592  1171 raft_consensus.cc:2468] T 1935c9164755416a965cccf80900d941 P b3afae27e8744ee6921cd67f805e3e4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d88b316b7ecd49f29219068c5fe51a26 in term 1.
I20260709 12:58:19.766106  1059 leader_election.cc:304] T 1935c9164755416a965cccf80900d941 P d88b316b7ecd49f29219068c5fe51a26 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b3afae27e8744ee6921cd67f805e3e4a, d88b316b7ecd49f29219068c5fe51a26; no voters: d0e95822debe474ba6a700c661aa262f
I20260709 12:58:19.766911  1286 tablet_bootstrap.cc:492] T 759e30de7d7b4bc6980dbf8c4ba8031f P b3afae27e8744ee6921cd67f805e3e4a: No bootstrap required, opened a new log
I20260709 12:58:19.767488  1286 ts_tablet_manager.cc:1403] T 759e30de7d7b4bc6980dbf8c4ba8031f P b3afae27e8744ee6921cd67f805e3e4a: Time spent bootstrapping tablet: real 0.067s	user 0.015s	sys 0.002s
I20260709 12:58:19.767458  1293 raft_consensus.cc:2804] T 1935c9164755416a965cccf80900d941 P d88b316b7ecd49f29219068c5fe51a26 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:58:19.768246  1293 raft_consensus.cc:697] T 1935c9164755416a965cccf80900d941 P d88b316b7ecd49f29219068c5fe51a26 [term 1 LEADER]: Becoming Leader. State: Replica: d88b316b7ecd49f29219068c5fe51a26, State: Running, Role: LEADER
I20260709 12:58:19.769234  1293 consensus_queue.cc:237] T 1935c9164755416a965cccf80900d941 P d88b316b7ecd49f29219068c5fe51a26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:19.770700  1287 tablet_bootstrap.cc:654] T 2cde8b0b8c5c46e78f405744f084c7c6 P d88b316b7ecd49f29219068c5fe51a26: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:19.771406  1286 raft_consensus.cc:359] T 759e30de7d7b4bc6980dbf8c4ba8031f P b3afae27e8744ee6921cd67f805e3e4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:19.772537  1286 raft_consensus.cc:385] T 759e30de7d7b4bc6980dbf8c4ba8031f P b3afae27e8744ee6921cd67f805e3e4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:19.773046  1286 raft_consensus.cc:740] T 759e30de7d7b4bc6980dbf8c4ba8031f P b3afae27e8744ee6921cd67f805e3e4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3afae27e8744ee6921cd67f805e3e4a, State: Initialized, Role: FOLLOWER
I20260709 12:58:19.774186  1286 consensus_queue.cc:260] T 759e30de7d7b4bc6980dbf8c4ba8031f P b3afae27e8744ee6921cd67f805e3e4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:19.776916  1286 ts_tablet_manager.cc:1434] T 759e30de7d7b4bc6980dbf8c4ba8031f P b3afae27e8744ee6921cd67f805e3e4a: Time spent starting tablet: real 0.009s	user 0.006s	sys 0.000s
I20260709 12:58:19.778026  1286 tablet_bootstrap.cc:492] T 2cde8b0b8c5c46e78f405744f084c7c6 P b3afae27e8744ee6921cd67f805e3e4a: Bootstrap starting.
I20260709 12:58:19.784211  1287 tablet_bootstrap.cc:492] T 2cde8b0b8c5c46e78f405744f084c7c6 P d88b316b7ecd49f29219068c5fe51a26: No bootstrap required, opened a new log
I20260709 12:58:19.784900  1287 ts_tablet_manager.cc:1403] T 2cde8b0b8c5c46e78f405744f084c7c6 P d88b316b7ecd49f29219068c5fe51a26: Time spent bootstrapping tablet: real 0.028s	user 0.005s	sys 0.010s
I20260709 12:58:19.786983  1286 tablet_bootstrap.cc:654] T 2cde8b0b8c5c46e78f405744f084c7c6 P b3afae27e8744ee6921cd67f805e3e4a: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:19.789060  1287 raft_consensus.cc:359] T 2cde8b0b8c5c46e78f405744f084c7c6 P d88b316b7ecd49f29219068c5fe51a26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:19.790020  1287 raft_consensus.cc:385] T 2cde8b0b8c5c46e78f405744f084c7c6 P d88b316b7ecd49f29219068c5fe51a26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:19.790390  1287 raft_consensus.cc:740] T 2cde8b0b8c5c46e78f405744f084c7c6 P d88b316b7ecd49f29219068c5fe51a26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d88b316b7ecd49f29219068c5fe51a26, State: Initialized, Role: FOLLOWER
I20260709 12:58:19.791678  1287 consensus_queue.cc:260] T 2cde8b0b8c5c46e78f405744f084c7c6 P d88b316b7ecd49f29219068c5fe51a26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:19.792830   992 catalog_manager.cc:5697] T 1935c9164755416a965cccf80900d941 P d88b316b7ecd49f29219068c5fe51a26 reported cstate change: term changed from 0 to 1, leader changed from <none> to d88b316b7ecd49f29219068c5fe51a26 (127.0.107.193). New cstate: current_term: 1 leader_uuid: "d88b316b7ecd49f29219068c5fe51a26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } health_report { overall_health: UNKNOWN } } }
I20260709 12:58:19.795300  1287 ts_tablet_manager.cc:1434] T 2cde8b0b8c5c46e78f405744f084c7c6 P d88b316b7ecd49f29219068c5fe51a26: Time spent starting tablet: real 0.010s	user 0.009s	sys 0.003s
I20260709 12:58:19.796715  1287 tablet_bootstrap.cc:492] T 73e09365f8cd4513a4600030950af811 P d88b316b7ecd49f29219068c5fe51a26: Bootstrap starting.
I20260709 12:58:19.799506  1286 tablet_bootstrap.cc:492] T 2cde8b0b8c5c46e78f405744f084c7c6 P b3afae27e8744ee6921cd67f805e3e4a: No bootstrap required, opened a new log
I20260709 12:58:19.799831  1293 raft_consensus.cc:493] T 2685bf6c4fd043a481d1d05136dbbd5e P d88b316b7ecd49f29219068c5fe51a26 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:19.800091  1286 ts_tablet_manager.cc:1403] T 2cde8b0b8c5c46e78f405744f084c7c6 P b3afae27e8744ee6921cd67f805e3e4a: Time spent bootstrapping tablet: real 0.022s	user 0.011s	sys 0.008s
I20260709 12:58:19.800313  1293 raft_consensus.cc:515] T 2685bf6c4fd043a481d1d05136dbbd5e P d88b316b7ecd49f29219068c5fe51a26 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:19.802687  1293 leader_election.cc:290] T 2685bf6c4fd043a481d1d05136dbbd5e P d88b316b7ecd49f29219068c5fe51a26 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b3afae27e8744ee6921cd67f805e3e4a (127.0.107.194:36785), d0e95822debe474ba6a700c661aa262f (127.0.107.195:40863)
I20260709 12:58:19.803572  1171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2685bf6c4fd043a481d1d05136dbbd5e" candidate_uuid: "d88b316b7ecd49f29219068c5fe51a26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3afae27e8744ee6921cd67f805e3e4a" is_pre_election: true
I20260709 12:58:19.804210  1171 raft_consensus.cc:2468] T 2685bf6c4fd043a481d1d05136dbbd5e P b3afae27e8744ee6921cd67f805e3e4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d88b316b7ecd49f29219068c5fe51a26 in term 0.
I20260709 12:58:19.804306  1245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2685bf6c4fd043a481d1d05136dbbd5e" candidate_uuid: "d88b316b7ecd49f29219068c5fe51a26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0e95822debe474ba6a700c661aa262f" is_pre_election: true
I20260709 12:58:19.804891  1245 raft_consensus.cc:2468] T 2685bf6c4fd043a481d1d05136dbbd5e P d0e95822debe474ba6a700c661aa262f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d88b316b7ecd49f29219068c5fe51a26 in term 0.
I20260709 12:58:19.805649  1059 leader_election.cc:304] T 2685bf6c4fd043a481d1d05136dbbd5e P d88b316b7ecd49f29219068c5fe51a26 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3afae27e8744ee6921cd67f805e3e4a, d88b316b7ecd49f29219068c5fe51a26; no voters: 
I20260709 12:58:19.806638  1287 tablet_bootstrap.cc:654] T 73e09365f8cd4513a4600030950af811 P d88b316b7ecd49f29219068c5fe51a26: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:19.806835  1293 raft_consensus.cc:2804] T 2685bf6c4fd043a481d1d05136dbbd5e P d88b316b7ecd49f29219068c5fe51a26 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:58:19.807313  1293 raft_consensus.cc:493] T 2685bf6c4fd043a481d1d05136dbbd5e P d88b316b7ecd49f29219068c5fe51a26 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:58:19.807691  1293 raft_consensus.cc:3060] T 2685bf6c4fd043a481d1d05136dbbd5e P d88b316b7ecd49f29219068c5fe51a26 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:19.815424  1286 raft_consensus.cc:359] T 2cde8b0b8c5c46e78f405744f084c7c6 P b3afae27e8744ee6921cd67f805e3e4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:19.817417  1293 raft_consensus.cc:515] T 2685bf6c4fd043a481d1d05136dbbd5e P d88b316b7ecd49f29219068c5fe51a26 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:19.818006  1286 raft_consensus.cc:385] T 2cde8b0b8c5c46e78f405744f084c7c6 P b3afae27e8744ee6921cd67f805e3e4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:19.818368  1286 raft_consensus.cc:740] T 2cde8b0b8c5c46e78f405744f084c7c6 P b3afae27e8744ee6921cd67f805e3e4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3afae27e8744ee6921cd67f805e3e4a, State: Initialized, Role: FOLLOWER
I20260709 12:58:19.818605  1285 tablet_bootstrap.cc:492] T 1935c9164755416a965cccf80900d941 P d0e95822debe474ba6a700c661aa262f: No bootstrap required, opened a new log
I20260709 12:58:19.819239  1285 ts_tablet_manager.cc:1403] T 1935c9164755416a965cccf80900d941 P d0e95822debe474ba6a700c661aa262f: Time spent bootstrapping tablet: real 0.065s	user 0.021s	sys 0.031s
I20260709 12:58:19.820298  1171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2685bf6c4fd043a481d1d05136dbbd5e" candidate_uuid: "d88b316b7ecd49f29219068c5fe51a26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3afae27e8744ee6921cd67f805e3e4a"
I20260709 12:58:19.820529  1245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2685bf6c4fd043a481d1d05136dbbd5e" candidate_uuid: "d88b316b7ecd49f29219068c5fe51a26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0e95822debe474ba6a700c661aa262f"
I20260709 12:58:19.820861  1171 raft_consensus.cc:3060] T 2685bf6c4fd043a481d1d05136dbbd5e P b3afae27e8744ee6921cd67f805e3e4a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:19.821045  1245 raft_consensus.cc:3060] T 2685bf6c4fd043a481d1d05136dbbd5e P d0e95822debe474ba6a700c661aa262f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:19.823354  1285 raft_consensus.cc:359] T 1935c9164755416a965cccf80900d941 P d0e95822debe474ba6a700c661aa262f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:19.824277  1285 raft_consensus.cc:385] T 1935c9164755416a965cccf80900d941 P d0e95822debe474ba6a700c661aa262f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:19.824620  1285 raft_consensus.cc:740] T 1935c9164755416a965cccf80900d941 P d0e95822debe474ba6a700c661aa262f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0e95822debe474ba6a700c661aa262f, State: Initialized, Role: FOLLOWER
I20260709 12:58:19.826891  1291 raft_consensus.cc:493] T 759e30de7d7b4bc6980dbf8c4ba8031f P b3afae27e8744ee6921cd67f805e3e4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:19.827369  1291 raft_consensus.cc:515] T 759e30de7d7b4bc6980dbf8c4ba8031f P b3afae27e8744ee6921cd67f805e3e4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:19.828109  1245 raft_consensus.cc:2468] T 2685bf6c4fd043a481d1d05136dbbd5e P d0e95822debe474ba6a700c661aa262f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d88b316b7ecd49f29219068c5fe51a26 in term 1.
I20260709 12:58:19.829388  1061 leader_election.cc:304] T 2685bf6c4fd043a481d1d05136dbbd5e P d88b316b7ecd49f29219068c5fe51a26 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d0e95822debe474ba6a700c661aa262f, d88b316b7ecd49f29219068c5fe51a26; no voters: 
I20260709 12:58:19.829557  1291 leader_election.cc:290] T 759e30de7d7b4bc6980dbf8c4ba8031f P b3afae27e8744ee6921cd67f805e3e4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d88b316b7ecd49f29219068c5fe51a26 (127.0.107.193:38591), d0e95822debe474ba6a700c661aa262f (127.0.107.195:40863)
I20260709 12:58:19.829643  1285 consensus_queue.cc:260] T 1935c9164755416a965cccf80900d941 P d0e95822debe474ba6a700c661aa262f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:19.830313  1314 raft_consensus.cc:2804] T 2685bf6c4fd043a481d1d05136dbbd5e P d88b316b7ecd49f29219068c5fe51a26 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:58:19.831971  1095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "759e30de7d7b4bc6980dbf8c4ba8031f" candidate_uuid: "b3afae27e8744ee6921cd67f805e3e4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d88b316b7ecd49f29219068c5fe51a26" is_pre_election: true
I20260709 12:58:19.833041  1285 ts_tablet_manager.cc:1434] T 1935c9164755416a965cccf80900d941 P d0e95822debe474ba6a700c661aa262f: Time spent starting tablet: real 0.013s	user 0.003s	sys 0.005s
W20260709 12:58:19.833389  1133 leader_election.cc:343] T 759e30de7d7b4bc6980dbf8c4ba8031f P b3afae27e8744ee6921cd67f805e3e4a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d88b316b7ecd49f29219068c5fe51a26 (127.0.107.193:38591): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:19.834368  1285 tablet_bootstrap.cc:492] T d87ed5c1a25e40f8b7aa82d33ca30bf1 P d0e95822debe474ba6a700c661aa262f: Bootstrap starting.
I20260709 12:58:19.836472  1171 raft_consensus.cc:2468] T 2685bf6c4fd043a481d1d05136dbbd5e P b3afae27e8744ee6921cd67f805e3e4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d88b316b7ecd49f29219068c5fe51a26 in term 1.
I20260709 12:58:19.838871  1293 leader_election.cc:290] T 2685bf6c4fd043a481d1d05136dbbd5e P d88b316b7ecd49f29219068c5fe51a26 [CANDIDATE]: Term 1 election: Requested vote from peers b3afae27e8744ee6921cd67f805e3e4a (127.0.107.194:36785), d0e95822debe474ba6a700c661aa262f (127.0.107.195:40863)
I20260709 12:58:19.840284  1286 consensus_queue.cc:260] T 2cde8b0b8c5c46e78f405744f084c7c6 P b3afae27e8744ee6921cd67f805e3e4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:19.838798  1245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "759e30de7d7b4bc6980dbf8c4ba8031f" candidate_uuid: "b3afae27e8744ee6921cd67f805e3e4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0e95822debe474ba6a700c661aa262f" is_pre_election: true
I20260709 12:58:19.842106  1285 tablet_bootstrap.cc:654] T d87ed5c1a25e40f8b7aa82d33ca30bf1 P d0e95822debe474ba6a700c661aa262f: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:19.843250  1286 ts_tablet_manager.cc:1434] T 2cde8b0b8c5c46e78f405744f084c7c6 P b3afae27e8744ee6921cd67f805e3e4a: Time spent starting tablet: real 0.043s	user 0.007s	sys 0.000s
I20260709 12:58:19.844398  1286 tablet_bootstrap.cc:492] T d87ed5c1a25e40f8b7aa82d33ca30bf1 P b3afae27e8744ee6921cd67f805e3e4a: Bootstrap starting.
I20260709 12:58:19.850052  1287 tablet_bootstrap.cc:492] T 73e09365f8cd4513a4600030950af811 P d88b316b7ecd49f29219068c5fe51a26: No bootstrap required, opened a new log
I20260709 12:58:19.850729  1287 ts_tablet_manager.cc:1403] T 73e09365f8cd4513a4600030950af811 P d88b316b7ecd49f29219068c5fe51a26: Time spent bootstrapping tablet: real 0.054s	user 0.009s	sys 0.026s
I20260709 12:58:19.854779  1245 raft_consensus.cc:2468] T 759e30de7d7b4bc6980dbf8c4ba8031f P d0e95822debe474ba6a700c661aa262f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3afae27e8744ee6921cd67f805e3e4a in term 0.
I20260709 12:58:19.855027  1287 raft_consensus.cc:359] T 73e09365f8cd4513a4600030950af811 P d88b316b7ecd49f29219068c5fe51a26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:19.856123  1287 raft_consensus.cc:385] T 73e09365f8cd4513a4600030950af811 P d88b316b7ecd49f29219068c5fe51a26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:19.855998  1136 leader_election.cc:304] T 759e30de7d7b4bc6980dbf8c4ba8031f P b3afae27e8744ee6921cd67f805e3e4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b3afae27e8744ee6921cd67f805e3e4a, d0e95822debe474ba6a700c661aa262f; no voters: d88b316b7ecd49f29219068c5fe51a26
I20260709 12:58:19.856594  1287 raft_consensus.cc:740] T 73e09365f8cd4513a4600030950af811 P d88b316b7ecd49f29219068c5fe51a26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d88b316b7ecd49f29219068c5fe51a26, State: Initialized, Role: FOLLOWER
I20260709 12:58:19.857497  1291 raft_consensus.cc:2804] T 759e30de7d7b4bc6980dbf8c4ba8031f P b3afae27e8744ee6921cd67f805e3e4a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:58:19.858028  1291 raft_consensus.cc:493] T 759e30de7d7b4bc6980dbf8c4ba8031f P b3afae27e8744ee6921cd67f805e3e4a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:58:19.857568  1287 consensus_queue.cc:260] T 73e09365f8cd4513a4600030950af811 P d88b316b7ecd49f29219068c5fe51a26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:19.858469  1314 raft_consensus.cc:697] T 2685bf6c4fd043a481d1d05136dbbd5e P d88b316b7ecd49f29219068c5fe51a26 [term 1 LEADER]: Becoming Leader. State: Replica: d88b316b7ecd49f29219068c5fe51a26, State: Running, Role: LEADER
I20260709 12:58:19.858630  1291 raft_consensus.cc:3060] T 759e30de7d7b4bc6980dbf8c4ba8031f P b3afae27e8744ee6921cd67f805e3e4a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:19.859611  1285 tablet_bootstrap.cc:492] T d87ed5c1a25e40f8b7aa82d33ca30bf1 P d0e95822debe474ba6a700c661aa262f: No bootstrap required, opened a new log
I20260709 12:58:19.859431  1314 consensus_queue.cc:237] T 2685bf6c4fd043a481d1d05136dbbd5e P d88b316b7ecd49f29219068c5fe51a26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:19.860109  1285 ts_tablet_manager.cc:1403] T d87ed5c1a25e40f8b7aa82d33ca30bf1 P d0e95822debe474ba6a700c661aa262f: Time spent bootstrapping tablet: real 0.026s	user 0.017s	sys 0.004s
I20260709 12:58:19.860857  1287 ts_tablet_manager.cc:1434] T 73e09365f8cd4513a4600030950af811 P d88b316b7ecd49f29219068c5fe51a26: Time spent starting tablet: real 0.010s	user 0.009s	sys 0.000s
I20260709 12:58:19.863375  1287 tablet_bootstrap.cc:492] T 8ee1d07b2c9444f8ad28921d69d297bf P d88b316b7ecd49f29219068c5fe51a26: Bootstrap starting.
I20260709 12:58:19.863631  1285 raft_consensus.cc:359] T d87ed5c1a25e40f8b7aa82d33ca30bf1 P d0e95822debe474ba6a700c661aa262f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:19.864547  1285 raft_consensus.cc:385] T d87ed5c1a25e40f8b7aa82d33ca30bf1 P d0e95822debe474ba6a700c661aa262f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:19.864936  1285 raft_consensus.cc:740] T d87ed5c1a25e40f8b7aa82d33ca30bf1 P d0e95822debe474ba6a700c661aa262f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0e95822debe474ba6a700c661aa262f, State: Initialized, Role: FOLLOWER
I20260709 12:58:19.866106  1285 consensus_queue.cc:260] T d87ed5c1a25e40f8b7aa82d33ca30bf1 P d0e95822debe474ba6a700c661aa262f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:19.869079  1285 ts_tablet_manager.cc:1434] T d87ed5c1a25e40f8b7aa82d33ca30bf1 P d0e95822debe474ba6a700c661aa262f: Time spent starting tablet: real 0.009s	user 0.010s	sys 0.000s
I20260709 12:58:19.870077  1285 tablet_bootstrap.cc:492] T 73e09365f8cd4513a4600030950af811 P d0e95822debe474ba6a700c661aa262f: Bootstrap starting.
I20260709 12:58:19.871302  1287 tablet_bootstrap.cc:654] T 8ee1d07b2c9444f8ad28921d69d297bf P d88b316b7ecd49f29219068c5fe51a26: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:19.869961  1291 raft_consensus.cc:515] T 759e30de7d7b4bc6980dbf8c4ba8031f P b3afae27e8744ee6921cd67f805e3e4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:19.875325  1326 raft_consensus.cc:493] T df94d2062ca84496bc46321e6e4012fc P d88b316b7ecd49f29219068c5fe51a26 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:19.875820  1326 raft_consensus.cc:515] T df94d2062ca84496bc46321e6e4012fc P d88b316b7ecd49f29219068c5fe51a26 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:19.878113  1286 tablet_bootstrap.cc:654] T d87ed5c1a25e40f8b7aa82d33ca30bf1 P b3afae27e8744ee6921cd67f805e3e4a: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:19.878291  1326 leader_election.cc:290] T df94d2062ca84496bc46321e6e4012fc P d88b316b7ecd49f29219068c5fe51a26 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b3afae27e8744ee6921cd67f805e3e4a (127.0.107.194:36785), d0e95822debe474ba6a700c661aa262f (127.0.107.195:40863)
I20260709 12:58:19.879748  1171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df94d2062ca84496bc46321e6e4012fc" candidate_uuid: "d88b316b7ecd49f29219068c5fe51a26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3afae27e8744ee6921cd67f805e3e4a" is_pre_election: true
I20260709 12:58:19.881816  1245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df94d2062ca84496bc46321e6e4012fc" candidate_uuid: "d88b316b7ecd49f29219068c5fe51a26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0e95822debe474ba6a700c661aa262f" is_pre_election: true
I20260709 12:58:19.882599  1285 tablet_bootstrap.cc:654] T 73e09365f8cd4513a4600030950af811 P d0e95822debe474ba6a700c661aa262f: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:19.882568  1095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "759e30de7d7b4bc6980dbf8c4ba8031f" candidate_uuid: "b3afae27e8744ee6921cd67f805e3e4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d88b316b7ecd49f29219068c5fe51a26"
W20260709 12:58:19.884615  1061 leader_election.cc:343] T df94d2062ca84496bc46321e6e4012fc P d88b316b7ecd49f29219068c5fe51a26 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d0e95822debe474ba6a700c661aa262f (127.0.107.195:40863): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:58:19.885732  1059 leader_election.cc:343] T df94d2062ca84496bc46321e6e4012fc P d88b316b7ecd49f29219068c5fe51a26 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b3afae27e8744ee6921cd67f805e3e4a (127.0.107.194:36785): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:19.886241  1059 leader_election.cc:304] T df94d2062ca84496bc46321e6e4012fc P d88b316b7ecd49f29219068c5fe51a26 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d88b316b7ecd49f29219068c5fe51a26; no voters: b3afae27e8744ee6921cd67f805e3e4a, d0e95822debe474ba6a700c661aa262f
I20260709 12:58:19.887813  1326 raft_consensus.cc:2749] T df94d2062ca84496bc46321e6e4012fc P d88b316b7ecd49f29219068c5fe51a26 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260709 12:58:19.888478  1133 leader_election.cc:343] T 759e30de7d7b4bc6980dbf8c4ba8031f P b3afae27e8744ee6921cd67f805e3e4a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d88b316b7ecd49f29219068c5fe51a26 (127.0.107.193:38591): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:19.889609  1291 leader_election.cc:290] T 759e30de7d7b4bc6980dbf8c4ba8031f P b3afae27e8744ee6921cd67f805e3e4a [CANDIDATE]: Term 1 election: Requested vote from peers d88b316b7ecd49f29219068c5fe51a26 (127.0.107.193:38591), d0e95822debe474ba6a700c661aa262f (127.0.107.195:40863)
I20260709 12:58:19.894332  1245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "759e30de7d7b4bc6980dbf8c4ba8031f" candidate_uuid: "b3afae27e8744ee6921cd67f805e3e4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0e95822debe474ba6a700c661aa262f"
I20260709 12:58:19.895196  1245 raft_consensus.cc:3060] T 759e30de7d7b4bc6980dbf8c4ba8031f P d0e95822debe474ba6a700c661aa262f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:19.902395   990 catalog_manager.cc:5697] T 2685bf6c4fd043a481d1d05136dbbd5e P d88b316b7ecd49f29219068c5fe51a26 reported cstate change: term changed from 0 to 1, leader changed from <none> to d88b316b7ecd49f29219068c5fe51a26 (127.0.107.193). New cstate: current_term: 1 leader_uuid: "d88b316b7ecd49f29219068c5fe51a26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } health_report { overall_health: UNKNOWN } } }
I20260709 12:58:19.909523  1245 raft_consensus.cc:2468] T 759e30de7d7b4bc6980dbf8c4ba8031f P d0e95822debe474ba6a700c661aa262f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3afae27e8744ee6921cd67f805e3e4a in term 1.
I20260709 12:58:19.911893  1136 leader_election.cc:304] T 759e30de7d7b4bc6980dbf8c4ba8031f P b3afae27e8744ee6921cd67f805e3e4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b3afae27e8744ee6921cd67f805e3e4a, d0e95822debe474ba6a700c661aa262f; no voters: d88b316b7ecd49f29219068c5fe51a26
I20260709 12:58:19.913453  1291 raft_consensus.cc:2804] T 759e30de7d7b4bc6980dbf8c4ba8031f P b3afae27e8744ee6921cd67f805e3e4a [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:58:19.916909  1287 tablet_bootstrap.cc:492] T 8ee1d07b2c9444f8ad28921d69d297bf P d88b316b7ecd49f29219068c5fe51a26: No bootstrap required, opened a new log
I20260709 12:58:19.917666  1287 ts_tablet_manager.cc:1403] T 8ee1d07b2c9444f8ad28921d69d297bf P d88b316b7ecd49f29219068c5fe51a26: Time spent bootstrapping tablet: real 0.056s	user 0.024s	sys 0.022s
I20260709 12:58:19.921406  1287 raft_consensus.cc:359] T 8ee1d07b2c9444f8ad28921d69d297bf P d88b316b7ecd49f29219068c5fe51a26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:19.925271  1285 tablet_bootstrap.cc:492] T 73e09365f8cd4513a4600030950af811 P d0e95822debe474ba6a700c661aa262f: No bootstrap required, opened a new log
I20260709 12:58:19.924145  1286 tablet_bootstrap.cc:492] T d87ed5c1a25e40f8b7aa82d33ca30bf1 P b3afae27e8744ee6921cd67f805e3e4a: No bootstrap required, opened a new log
I20260709 12:58:19.925261  1287 raft_consensus.cc:385] T 8ee1d07b2c9444f8ad28921d69d297bf P d88b316b7ecd49f29219068c5fe51a26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:19.925863  1287 raft_consensus.cc:740] T 8ee1d07b2c9444f8ad28921d69d297bf P d88b316b7ecd49f29219068c5fe51a26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d88b316b7ecd49f29219068c5fe51a26, State: Initialized, Role: FOLLOWER
I20260709 12:58:19.925967  1286 ts_tablet_manager.cc:1403] T d87ed5c1a25e40f8b7aa82d33ca30bf1 P b3afae27e8744ee6921cd67f805e3e4a: Time spent bootstrapping tablet: real 0.082s	user 0.016s	sys 0.017s
I20260709 12:58:19.925993  1285 ts_tablet_manager.cc:1403] T 73e09365f8cd4513a4600030950af811 P d0e95822debe474ba6a700c661aa262f: Time spent bootstrapping tablet: real 0.056s	user 0.027s	sys 0.000s
I20260709 12:58:19.926759  1287 consensus_queue.cc:260] T 8ee1d07b2c9444f8ad28921d69d297bf P d88b316b7ecd49f29219068c5fe51a26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:19.929262  1287 ts_tablet_manager.cc:1434] T 8ee1d07b2c9444f8ad28921d69d297bf P d88b316b7ecd49f29219068c5fe51a26: Time spent starting tablet: real 0.011s	user 0.004s	sys 0.002s
I20260709 12:58:19.929212  1286 raft_consensus.cc:359] T d87ed5c1a25e40f8b7aa82d33ca30bf1 P b3afae27e8744ee6921cd67f805e3e4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:19.929523  1285 raft_consensus.cc:359] T 73e09365f8cd4513a4600030950af811 P d0e95822debe474ba6a700c661aa262f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:19.924767  1291 raft_consensus.cc:697] T 759e30de7d7b4bc6980dbf8c4ba8031f P b3afae27e8744ee6921cd67f805e3e4a [term 1 LEADER]: Becoming Leader. State: Replica: b3afae27e8744ee6921cd67f805e3e4a, State: Running, Role: LEADER
I20260709 12:58:19.930358  1285 raft_consensus.cc:385] T 73e09365f8cd4513a4600030950af811 P d0e95822debe474ba6a700c661aa262f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:19.930697  1287 tablet_bootstrap.cc:492] T 759e30de7d7b4bc6980dbf8c4ba8031f P d88b316b7ecd49f29219068c5fe51a26: Bootstrap starting.
I20260709 12:58:19.930732  1285 raft_consensus.cc:740] T 73e09365f8cd4513a4600030950af811 P d0e95822debe474ba6a700c661aa262f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0e95822debe474ba6a700c661aa262f, State: Initialized, Role: FOLLOWER
I20260709 12:58:19.931072  1291 consensus_queue.cc:237] T 759e30de7d7b4bc6980dbf8c4ba8031f P b3afae27e8744ee6921cd67f805e3e4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:19.931766  1285 consensus_queue.cc:260] T 73e09365f8cd4513a4600030950af811 P d0e95822debe474ba6a700c661aa262f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:19.934232  1285 ts_tablet_manager.cc:1434] T 73e09365f8cd4513a4600030950af811 P d0e95822debe474ba6a700c661aa262f: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.000s
I20260709 12:58:19.935398  1285 tablet_bootstrap.cc:492] T c6480f4fe1b54658b39e73baa7c719fd P d0e95822debe474ba6a700c661aa262f: Bootstrap starting.
I20260709 12:58:19.938381  1287 tablet_bootstrap.cc:654] T 759e30de7d7b4bc6980dbf8c4ba8031f P d88b316b7ecd49f29219068c5fe51a26: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:19.942939  1285 tablet_bootstrap.cc:654] T c6480f4fe1b54658b39e73baa7c719fd P d0e95822debe474ba6a700c661aa262f: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:19.930022  1286 raft_consensus.cc:385] T d87ed5c1a25e40f8b7aa82d33ca30bf1 P b3afae27e8744ee6921cd67f805e3e4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:19.943953  1286 raft_consensus.cc:740] T d87ed5c1a25e40f8b7aa82d33ca30bf1 P b3afae27e8744ee6921cd67f805e3e4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3afae27e8744ee6921cd67f805e3e4a, State: Initialized, Role: FOLLOWER
I20260709 12:58:19.945925  1286 consensus_queue.cc:260] T d87ed5c1a25e40f8b7aa82d33ca30bf1 P b3afae27e8744ee6921cd67f805e3e4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:19.948939  1286 ts_tablet_manager.cc:1434] T d87ed5c1a25e40f8b7aa82d33ca30bf1 P b3afae27e8744ee6921cd67f805e3e4a: Time spent starting tablet: real 0.022s	user 0.004s	sys 0.001s
I20260709 12:58:19.949659  1286 tablet_bootstrap.cc:492] T c6480f4fe1b54658b39e73baa7c719fd P b3afae27e8744ee6921cd67f805e3e4a: Bootstrap starting.
I20260709 12:58:19.950152  1287 tablet_bootstrap.cc:492] T 759e30de7d7b4bc6980dbf8c4ba8031f P d88b316b7ecd49f29219068c5fe51a26: No bootstrap required, opened a new log
I20260709 12:58:19.950745  1287 ts_tablet_manager.cc:1403] T 759e30de7d7b4bc6980dbf8c4ba8031f P d88b316b7ecd49f29219068c5fe51a26: Time spent bootstrapping tablet: real 0.021s	user 0.008s	sys 0.010s
I20260709 12:58:19.954048  1286 tablet_bootstrap.cc:654] T c6480f4fe1b54658b39e73baa7c719fd P b3afae27e8744ee6921cd67f805e3e4a: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:19.954427  1287 raft_consensus.cc:359] T 759e30de7d7b4bc6980dbf8c4ba8031f P d88b316b7ecd49f29219068c5fe51a26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:19.955209  1287 raft_consensus.cc:385] T 759e30de7d7b4bc6980dbf8c4ba8031f P d88b316b7ecd49f29219068c5fe51a26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:19.955457  1287 raft_consensus.cc:740] T 759e30de7d7b4bc6980dbf8c4ba8031f P d88b316b7ecd49f29219068c5fe51a26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d88b316b7ecd49f29219068c5fe51a26, State: Initialized, Role: FOLLOWER
I20260709 12:58:19.956197  1287 consensus_queue.cc:260] T 759e30de7d7b4bc6980dbf8c4ba8031f P d88b316b7ecd49f29219068c5fe51a26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:19.957108  1285 tablet_bootstrap.cc:492] T c6480f4fe1b54658b39e73baa7c719fd P d0e95822debe474ba6a700c661aa262f: No bootstrap required, opened a new log
I20260709 12:58:19.957639  1285 ts_tablet_manager.cc:1403] T c6480f4fe1b54658b39e73baa7c719fd P d0e95822debe474ba6a700c661aa262f: Time spent bootstrapping tablet: real 0.022s	user 0.012s	sys 0.009s
I20260709 12:58:19.960598  1285 raft_consensus.cc:359] T c6480f4fe1b54658b39e73baa7c719fd P d0e95822debe474ba6a700c661aa262f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:19.961309  1285 raft_consensus.cc:385] T c6480f4fe1b54658b39e73baa7c719fd P d0e95822debe474ba6a700c661aa262f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:19.961650  1285 raft_consensus.cc:740] T c6480f4fe1b54658b39e73baa7c719fd P d0e95822debe474ba6a700c661aa262f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0e95822debe474ba6a700c661aa262f, State: Initialized, Role: FOLLOWER
I20260709 12:58:19.962579  1285 consensus_queue.cc:260] T c6480f4fe1b54658b39e73baa7c719fd P d0e95822debe474ba6a700c661aa262f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:19.964659   992 catalog_manager.cc:5697] T 759e30de7d7b4bc6980dbf8c4ba8031f P b3afae27e8744ee6921cd67f805e3e4a reported cstate change: term changed from 0 to 1, leader changed from <none> to b3afae27e8744ee6921cd67f805e3e4a (127.0.107.194). New cstate: current_term: 1 leader_uuid: "b3afae27e8744ee6921cd67f805e3e4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } health_report { overall_health: UNKNOWN } } }
I20260709 12:58:19.965883  1287 ts_tablet_manager.cc:1434] T 759e30de7d7b4bc6980dbf8c4ba8031f P d88b316b7ecd49f29219068c5fe51a26: Time spent starting tablet: real 0.015s	user 0.006s	sys 0.000s
I20260709 12:58:19.967283  1285 ts_tablet_manager.cc:1434] T c6480f4fe1b54658b39e73baa7c719fd P d0e95822debe474ba6a700c661aa262f: Time spent starting tablet: real 0.009s	user 0.006s	sys 0.000s
I20260709 12:58:19.968035  1286 tablet_bootstrap.cc:492] T c6480f4fe1b54658b39e73baa7c719fd P b3afae27e8744ee6921cd67f805e3e4a: No bootstrap required, opened a new log
I20260709 12:58:19.968338  1285 tablet_bootstrap.cc:492] T df94d2062ca84496bc46321e6e4012fc P d0e95822debe474ba6a700c661aa262f: Bootstrap starting.
I20260709 12:58:19.968487  1286 ts_tablet_manager.cc:1403] T c6480f4fe1b54658b39e73baa7c719fd P b3afae27e8744ee6921cd67f805e3e4a: Time spent bootstrapping tablet: real 0.019s	user 0.004s	sys 0.008s
I20260709 12:58:19.969609  1287 tablet_bootstrap.cc:492] T c6480f4fe1b54658b39e73baa7c719fd P d88b316b7ecd49f29219068c5fe51a26: Bootstrap starting.
I20260709 12:58:19.972057  1286 raft_consensus.cc:359] T c6480f4fe1b54658b39e73baa7c719fd P b3afae27e8744ee6921cd67f805e3e4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:19.972721  1286 raft_consensus.cc:385] T c6480f4fe1b54658b39e73baa7c719fd P b3afae27e8744ee6921cd67f805e3e4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:19.972996  1286 raft_consensus.cc:740] T c6480f4fe1b54658b39e73baa7c719fd P b3afae27e8744ee6921cd67f805e3e4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3afae27e8744ee6921cd67f805e3e4a, State: Initialized, Role: FOLLOWER
I20260709 12:58:19.973688  1286 consensus_queue.cc:260] T c6480f4fe1b54658b39e73baa7c719fd P b3afae27e8744ee6921cd67f805e3e4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:19.975988  1287 tablet_bootstrap.cc:654] T c6480f4fe1b54658b39e73baa7c719fd P d88b316b7ecd49f29219068c5fe51a26: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:19.975988  1285 tablet_bootstrap.cc:654] T df94d2062ca84496bc46321e6e4012fc P d0e95822debe474ba6a700c661aa262f: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:19.976114  1286 ts_tablet_manager.cc:1434] T c6480f4fe1b54658b39e73baa7c719fd P b3afae27e8744ee6921cd67f805e3e4a: Time spent starting tablet: real 0.007s	user 0.006s	sys 0.000s
I20260709 12:58:19.977501  1286 tablet_bootstrap.cc:492] T df94d2062ca84496bc46321e6e4012fc P b3afae27e8744ee6921cd67f805e3e4a: Bootstrap starting.
I20260709 12:58:19.983551  1286 tablet_bootstrap.cc:654] T df94d2062ca84496bc46321e6e4012fc P b3afae27e8744ee6921cd67f805e3e4a: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:19.998862  1287 tablet_bootstrap.cc:492] T c6480f4fe1b54658b39e73baa7c719fd P d88b316b7ecd49f29219068c5fe51a26: No bootstrap required, opened a new log
I20260709 12:58:19.999377  1287 ts_tablet_manager.cc:1403] T c6480f4fe1b54658b39e73baa7c719fd P d88b316b7ecd49f29219068c5fe51a26: Time spent bootstrapping tablet: real 0.030s	user 0.016s	sys 0.010s
I20260709 12:58:20.000437  1285 tablet_bootstrap.cc:492] T df94d2062ca84496bc46321e6e4012fc P d0e95822debe474ba6a700c661aa262f: No bootstrap required, opened a new log
I20260709 12:58:20.000439  1286 tablet_bootstrap.cc:492] T df94d2062ca84496bc46321e6e4012fc P b3afae27e8744ee6921cd67f805e3e4a: No bootstrap required, opened a new log
I20260709 12:58:20.001014  1285 ts_tablet_manager.cc:1403] T df94d2062ca84496bc46321e6e4012fc P d0e95822debe474ba6a700c661aa262f: Time spent bootstrapping tablet: real 0.033s	user 0.015s	sys 0.010s
I20260709 12:58:20.001122  1286 ts_tablet_manager.cc:1403] T df94d2062ca84496bc46321e6e4012fc P b3afae27e8744ee6921cd67f805e3e4a: Time spent bootstrapping tablet: real 0.024s	user 0.011s	sys 0.012s
I20260709 12:58:20.002266  1287 raft_consensus.cc:359] T c6480f4fe1b54658b39e73baa7c719fd P d88b316b7ecd49f29219068c5fe51a26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:20.002905  1287 raft_consensus.cc:385] T c6480f4fe1b54658b39e73baa7c719fd P d88b316b7ecd49f29219068c5fe51a26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:20.003144  1287 raft_consensus.cc:740] T c6480f4fe1b54658b39e73baa7c719fd P d88b316b7ecd49f29219068c5fe51a26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d88b316b7ecd49f29219068c5fe51a26, State: Initialized, Role: FOLLOWER
I20260709 12:58:20.003777  1287 consensus_queue.cc:260] T c6480f4fe1b54658b39e73baa7c719fd P d88b316b7ecd49f29219068c5fe51a26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:20.003885  1285 raft_consensus.cc:359] T df94d2062ca84496bc46321e6e4012fc P d0e95822debe474ba6a700c661aa262f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:20.004525  1285 raft_consensus.cc:385] T df94d2062ca84496bc46321e6e4012fc P d0e95822debe474ba6a700c661aa262f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:20.004788  1285 raft_consensus.cc:740] T df94d2062ca84496bc46321e6e4012fc P d0e95822debe474ba6a700c661aa262f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0e95822debe474ba6a700c661aa262f, State: Initialized, Role: FOLLOWER
I20260709 12:58:20.005177  1287 ts_tablet_manager.cc:1434] T c6480f4fe1b54658b39e73baa7c719fd P d88b316b7ecd49f29219068c5fe51a26: Time spent starting tablet: real 0.006s	user 0.004s	sys 0.000s
I20260709 12:58:20.005460  1285 consensus_queue.cc:260] T df94d2062ca84496bc46321e6e4012fc P d0e95822debe474ba6a700c661aa262f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:20.007422  1285 ts_tablet_manager.cc:1434] T df94d2062ca84496bc46321e6e4012fc P d0e95822debe474ba6a700c661aa262f: Time spent starting tablet: real 0.006s	user 0.007s	sys 0.000s
I20260709 12:58:20.007961  1286 raft_consensus.cc:359] T df94d2062ca84496bc46321e6e4012fc P b3afae27e8744ee6921cd67f805e3e4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:20.008399  1286 raft_consensus.cc:385] T df94d2062ca84496bc46321e6e4012fc P b3afae27e8744ee6921cd67f805e3e4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:20.008618  1286 raft_consensus.cc:740] T df94d2062ca84496bc46321e6e4012fc P b3afae27e8744ee6921cd67f805e3e4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3afae27e8744ee6921cd67f805e3e4a, State: Initialized, Role: FOLLOWER
I20260709 12:58:20.009207  1286 consensus_queue.cc:260] T df94d2062ca84496bc46321e6e4012fc P b3afae27e8744ee6921cd67f805e3e4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:20.011243  1286 ts_tablet_manager.cc:1434] T df94d2062ca84496bc46321e6e4012fc P b3afae27e8744ee6921cd67f805e3e4a: Time spent starting tablet: real 0.010s	user 0.005s	sys 0.000s
I20260709 12:58:20.012035  1285 tablet_bootstrap.cc:492] T 8ee1d07b2c9444f8ad28921d69d297bf P d0e95822debe474ba6a700c661aa262f: Bootstrap starting.
I20260709 12:58:20.018965  1285 tablet_bootstrap.cc:654] T 8ee1d07b2c9444f8ad28921d69d297bf P d0e95822debe474ba6a700c661aa262f: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:20.020535  1291 raft_consensus.cc:493] T df94d2062ca84496bc46321e6e4012fc P b3afae27e8744ee6921cd67f805e3e4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:20.021050  1291 raft_consensus.cc:515] T df94d2062ca84496bc46321e6e4012fc P b3afae27e8744ee6921cd67f805e3e4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:20.023013  1314 raft_consensus.cc:493] T 2cde8b0b8c5c46e78f405744f084c7c6 P d88b316b7ecd49f29219068c5fe51a26 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:20.023450  1314 raft_consensus.cc:515] T 2cde8b0b8c5c46e78f405744f084c7c6 P d88b316b7ecd49f29219068c5fe51a26 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:20.024667  1095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df94d2062ca84496bc46321e6e4012fc" candidate_uuid: "b3afae27e8744ee6921cd67f805e3e4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d88b316b7ecd49f29219068c5fe51a26" is_pre_election: true
I20260709 12:58:20.025353  1314 leader_election.cc:290] T 2cde8b0b8c5c46e78f405744f084c7c6 P d88b316b7ecd49f29219068c5fe51a26 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b3afae27e8744ee6921cd67f805e3e4a (127.0.107.194:36785), d0e95822debe474ba6a700c661aa262f (127.0.107.195:40863)
I20260709 12:58:20.025507  1095 raft_consensus.cc:2468] T df94d2062ca84496bc46321e6e4012fc P d88b316b7ecd49f29219068c5fe51a26 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3afae27e8744ee6921cd67f805e3e4a in term 0.
I20260709 12:58:20.026782  1245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cde8b0b8c5c46e78f405744f084c7c6" candidate_uuid: "d88b316b7ecd49f29219068c5fe51a26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0e95822debe474ba6a700c661aa262f" is_pre_election: true
I20260709 12:58:20.026952  1171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cde8b0b8c5c46e78f405744f084c7c6" candidate_uuid: "d88b316b7ecd49f29219068c5fe51a26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3afae27e8744ee6921cd67f805e3e4a" is_pre_election: true
I20260709 12:58:20.027315  1245 raft_consensus.cc:2468] T 2cde8b0b8c5c46e78f405744f084c7c6 P d0e95822debe474ba6a700c661aa262f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d88b316b7ecd49f29219068c5fe51a26 in term 0.
I20260709 12:58:20.027807  1171 raft_consensus.cc:2468] T 2cde8b0b8c5c46e78f405744f084c7c6 P b3afae27e8744ee6921cd67f805e3e4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d88b316b7ecd49f29219068c5fe51a26 in term 0.
I20260709 12:58:20.028486  1061 leader_election.cc:304] T 2cde8b0b8c5c46e78f405744f084c7c6 P d88b316b7ecd49f29219068c5fe51a26 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d0e95822debe474ba6a700c661aa262f, d88b316b7ecd49f29219068c5fe51a26; no voters: 
I20260709 12:58:20.030547  1133 leader_election.cc:304] T df94d2062ca84496bc46321e6e4012fc P b3afae27e8744ee6921cd67f805e3e4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3afae27e8744ee6921cd67f805e3e4a, d88b316b7ecd49f29219068c5fe51a26; no voters: 
I20260709 12:58:20.030910  1314 raft_consensus.cc:2804] T 2cde8b0b8c5c46e78f405744f084c7c6 P d88b316b7ecd49f29219068c5fe51a26 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:58:20.031322  1314 raft_consensus.cc:493] T 2cde8b0b8c5c46e78f405744f084c7c6 P d88b316b7ecd49f29219068c5fe51a26 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:58:20.031740  1314 raft_consensus.cc:3060] T 2cde8b0b8c5c46e78f405744f084c7c6 P d88b316b7ecd49f29219068c5fe51a26 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:20.032545  1291 leader_election.cc:290] T df94d2062ca84496bc46321e6e4012fc P b3afae27e8744ee6921cd67f805e3e4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d88b316b7ecd49f29219068c5fe51a26 (127.0.107.193:38591), d0e95822debe474ba6a700c661aa262f (127.0.107.195:40863)
I20260709 12:58:20.032758  1245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df94d2062ca84496bc46321e6e4012fc" candidate_uuid: "b3afae27e8744ee6921cd67f805e3e4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0e95822debe474ba6a700c661aa262f" is_pre_election: true
I20260709 12:58:20.033412  1245 raft_consensus.cc:2468] T df94d2062ca84496bc46321e6e4012fc P d0e95822debe474ba6a700c661aa262f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3afae27e8744ee6921cd67f805e3e4a in term 0.
I20260709 12:58:20.031731  1332 raft_consensus.cc:2804] T df94d2062ca84496bc46321e6e4012fc P b3afae27e8744ee6921cd67f805e3e4a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:58:20.034828  1332 raft_consensus.cc:493] T df94d2062ca84496bc46321e6e4012fc P b3afae27e8744ee6921cd67f805e3e4a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:58:20.035126  1332 raft_consensus.cc:3060] T df94d2062ca84496bc46321e6e4012fc P b3afae27e8744ee6921cd67f805e3e4a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:20.036705  1326 raft_consensus.cc:493] T c6480f4fe1b54658b39e73baa7c719fd P d88b316b7ecd49f29219068c5fe51a26 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:20.037339  1326 raft_consensus.cc:515] T c6480f4fe1b54658b39e73baa7c719fd P d88b316b7ecd49f29219068c5fe51a26 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:20.040063  1326 leader_election.cc:290] T c6480f4fe1b54658b39e73baa7c719fd P d88b316b7ecd49f29219068c5fe51a26 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b3afae27e8744ee6921cd67f805e3e4a (127.0.107.194:36785), d0e95822debe474ba6a700c661aa262f (127.0.107.195:40863)
I20260709 12:58:20.041425  1332 raft_consensus.cc:515] T df94d2062ca84496bc46321e6e4012fc P b3afae27e8744ee6921cd67f805e3e4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:20.042064  1245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6480f4fe1b54658b39e73baa7c719fd" candidate_uuid: "d88b316b7ecd49f29219068c5fe51a26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0e95822debe474ba6a700c661aa262f" is_pre_election: true
I20260709 12:58:20.043335  1171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6480f4fe1b54658b39e73baa7c719fd" candidate_uuid: "d88b316b7ecd49f29219068c5fe51a26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3afae27e8744ee6921cd67f805e3e4a" is_pre_election: true
I20260709 12:58:20.044133  1171 raft_consensus.cc:2468] T c6480f4fe1b54658b39e73baa7c719fd P b3afae27e8744ee6921cd67f805e3e4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d88b316b7ecd49f29219068c5fe51a26 in term 0.
I20260709 12:58:20.044535  1095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df94d2062ca84496bc46321e6e4012fc" candidate_uuid: "b3afae27e8744ee6921cd67f805e3e4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d88b316b7ecd49f29219068c5fe51a26"
I20260709 12:58:20.045109  1095 raft_consensus.cc:3060] T df94d2062ca84496bc46321e6e4012fc P d88b316b7ecd49f29219068c5fe51a26 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:20.045482  1245 raft_consensus.cc:2468] T c6480f4fe1b54658b39e73baa7c719fd P d0e95822debe474ba6a700c661aa262f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d88b316b7ecd49f29219068c5fe51a26 in term 0.
I20260709 12:58:20.045593  1314 raft_consensus.cc:515] T 2cde8b0b8c5c46e78f405744f084c7c6 P d88b316b7ecd49f29219068c5fe51a26 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:20.046767  1059 leader_election.cc:304] T c6480f4fe1b54658b39e73baa7c719fd P d88b316b7ecd49f29219068c5fe51a26 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3afae27e8744ee6921cd67f805e3e4a, d88b316b7ecd49f29219068c5fe51a26; no voters: 
I20260709 12:58:20.047686  1326 raft_consensus.cc:2804] T c6480f4fe1b54658b39e73baa7c719fd P d88b316b7ecd49f29219068c5fe51a26 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:58:20.048069  1326 raft_consensus.cc:493] T c6480f4fe1b54658b39e73baa7c719fd P d88b316b7ecd49f29219068c5fe51a26 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:58:20.048437  1326 raft_consensus.cc:3060] T c6480f4fe1b54658b39e73baa7c719fd P d88b316b7ecd49f29219068c5fe51a26 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:20.048647  1314 leader_election.cc:290] T 2cde8b0b8c5c46e78f405744f084c7c6 P d88b316b7ecd49f29219068c5fe51a26 [CANDIDATE]: Term 1 election: Requested vote from peers b3afae27e8744ee6921cd67f805e3e4a (127.0.107.194:36785), d0e95822debe474ba6a700c661aa262f (127.0.107.195:40863)
I20260709 12:58:20.050173  1245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cde8b0b8c5c46e78f405744f084c7c6" candidate_uuid: "d88b316b7ecd49f29219068c5fe51a26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0e95822debe474ba6a700c661aa262f"
I20260709 12:58:20.051530  1245 raft_consensus.cc:3060] T 2cde8b0b8c5c46e78f405744f084c7c6 P d0e95822debe474ba6a700c661aa262f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:20.052831  1244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df94d2062ca84496bc46321e6e4012fc" candidate_uuid: "b3afae27e8744ee6921cd67f805e3e4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0e95822debe474ba6a700c661aa262f"
I20260709 12:58:20.053455  1244 raft_consensus.cc:3060] T df94d2062ca84496bc46321e6e4012fc P d0e95822debe474ba6a700c661aa262f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:20.054576  1095 raft_consensus.cc:2468] T df94d2062ca84496bc46321e6e4012fc P d88b316b7ecd49f29219068c5fe51a26 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3afae27e8744ee6921cd67f805e3e4a in term 1.
I20260709 12:58:20.050803  1285 tablet_bootstrap.cc:492] T 8ee1d07b2c9444f8ad28921d69d297bf P d0e95822debe474ba6a700c661aa262f: No bootstrap required, opened a new log
I20260709 12:58:20.050971  1171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cde8b0b8c5c46e78f405744f084c7c6" candidate_uuid: "d88b316b7ecd49f29219068c5fe51a26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3afae27e8744ee6921cd67f805e3e4a"
I20260709 12:58:20.055673  1133 leader_election.cc:304] T df94d2062ca84496bc46321e6e4012fc P b3afae27e8744ee6921cd67f805e3e4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3afae27e8744ee6921cd67f805e3e4a, d88b316b7ecd49f29219068c5fe51a26; no voters: 
I20260709 12:58:20.055969  1171 raft_consensus.cc:3060] T 2cde8b0b8c5c46e78f405744f084c7c6 P b3afae27e8744ee6921cd67f805e3e4a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:20.056581  1291 raft_consensus.cc:2804] T df94d2062ca84496bc46321e6e4012fc P b3afae27e8744ee6921cd67f805e3e4a [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:58:20.057780  1332 leader_election.cc:290] T df94d2062ca84496bc46321e6e4012fc P b3afae27e8744ee6921cd67f805e3e4a [CANDIDATE]: Term 1 election: Requested vote from peers d88b316b7ecd49f29219068c5fe51a26 (127.0.107.193:38591), d0e95822debe474ba6a700c661aa262f (127.0.107.195:40863)
I20260709 12:58:20.058840  1285 ts_tablet_manager.cc:1403] T 8ee1d07b2c9444f8ad28921d69d297bf P d0e95822debe474ba6a700c661aa262f: Time spent bootstrapping tablet: real 0.047s	user 0.022s	sys 0.016s
I20260709 12:58:20.059945  1291 raft_consensus.cc:697] T df94d2062ca84496bc46321e6e4012fc P b3afae27e8744ee6921cd67f805e3e4a [term 1 LEADER]: Becoming Leader. State: Replica: b3afae27e8744ee6921cd67f805e3e4a, State: Running, Role: LEADER
I20260709 12:58:20.059746  1326 raft_consensus.cc:515] T c6480f4fe1b54658b39e73baa7c719fd P d88b316b7ecd49f29219068c5fe51a26 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:20.060911  1291 consensus_queue.cc:237] T df94d2062ca84496bc46321e6e4012fc P b3afae27e8744ee6921cd67f805e3e4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:20.062739  1170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6480f4fe1b54658b39e73baa7c719fd" candidate_uuid: "d88b316b7ecd49f29219068c5fe51a26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3afae27e8744ee6921cd67f805e3e4a"
I20260709 12:58:20.062846  1285 raft_consensus.cc:359] T 8ee1d07b2c9444f8ad28921d69d297bf P d0e95822debe474ba6a700c661aa262f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:20.063413  1170 raft_consensus.cc:3060] T c6480f4fe1b54658b39e73baa7c719fd P b3afae27e8744ee6921cd67f805e3e4a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:20.063606  1285 raft_consensus.cc:385] T 8ee1d07b2c9444f8ad28921d69d297bf P d0e95822debe474ba6a700c661aa262f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:20.063146  1244 raft_consensus.cc:2468] T df94d2062ca84496bc46321e6e4012fc P d0e95822debe474ba6a700c661aa262f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3afae27e8744ee6921cd67f805e3e4a in term 1.
I20260709 12:58:20.065732  1285 raft_consensus.cc:740] T 8ee1d07b2c9444f8ad28921d69d297bf P d0e95822debe474ba6a700c661aa262f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0e95822debe474ba6a700c661aa262f, State: Initialized, Role: FOLLOWER
I20260709 12:58:20.065377  1245 raft_consensus.cc:2468] T 2cde8b0b8c5c46e78f405744f084c7c6 P d0e95822debe474ba6a700c661aa262f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d88b316b7ecd49f29219068c5fe51a26 in term 1.
I20260709 12:58:20.066644  1285 consensus_queue.cc:260] T 8ee1d07b2c9444f8ad28921d69d297bf P d0e95822debe474ba6a700c661aa262f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:20.066946  1326 leader_election.cc:290] T c6480f4fe1b54658b39e73baa7c719fd P d88b316b7ecd49f29219068c5fe51a26 [CANDIDATE]: Term 1 election: Requested vote from peers b3afae27e8744ee6921cd67f805e3e4a (127.0.107.194:36785), d0e95822debe474ba6a700c661aa262f (127.0.107.195:40863)
I20260709 12:58:20.066982  1061 leader_election.cc:304] T 2cde8b0b8c5c46e78f405744f084c7c6 P d88b316b7ecd49f29219068c5fe51a26 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d0e95822debe474ba6a700c661aa262f, d88b316b7ecd49f29219068c5fe51a26; no voters: 
I20260709 12:58:20.068981  1285 ts_tablet_manager.cc:1434] T 8ee1d07b2c9444f8ad28921d69d297bf P d0e95822debe474ba6a700c661aa262f: Time spent starting tablet: real 0.010s	user 0.006s	sys 0.000s
I20260709 12:58:20.069399  1243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6480f4fe1b54658b39e73baa7c719fd" candidate_uuid: "d88b316b7ecd49f29219068c5fe51a26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0e95822debe474ba6a700c661aa262f"
I20260709 12:58:20.069826  1326 raft_consensus.cc:2804] T 2cde8b0b8c5c46e78f405744f084c7c6 P d88b316b7ecd49f29219068c5fe51a26 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:58:20.070005  1243 raft_consensus.cc:3060] T c6480f4fe1b54658b39e73baa7c719fd P d0e95822debe474ba6a700c661aa262f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:20.071508  1326 raft_consensus.cc:697] T 2cde8b0b8c5c46e78f405744f084c7c6 P d88b316b7ecd49f29219068c5fe51a26 [term 1 LEADER]: Becoming Leader. State: Replica: d88b316b7ecd49f29219068c5fe51a26, State: Running, Role: LEADER
I20260709 12:58:20.072335  1171 raft_consensus.cc:2468] T 2cde8b0b8c5c46e78f405744f084c7c6 P b3afae27e8744ee6921cd67f805e3e4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d88b316b7ecd49f29219068c5fe51a26 in term 1.
I20260709 12:58:20.072562  1326 consensus_queue.cc:237] T 2cde8b0b8c5c46e78f405744f084c7c6 P d88b316b7ecd49f29219068c5fe51a26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:20.080544  1243 raft_consensus.cc:2468] T c6480f4fe1b54658b39e73baa7c719fd P d0e95822debe474ba6a700c661aa262f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d88b316b7ecd49f29219068c5fe51a26 in term 1.
I20260709 12:58:20.080853  1170 raft_consensus.cc:2468] T c6480f4fe1b54658b39e73baa7c719fd P b3afae27e8744ee6921cd67f805e3e4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d88b316b7ecd49f29219068c5fe51a26 in term 1.
I20260709 12:58:20.082552  1059 leader_election.cc:304] T c6480f4fe1b54658b39e73baa7c719fd P d88b316b7ecd49f29219068c5fe51a26 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3afae27e8744ee6921cd67f805e3e4a, d88b316b7ecd49f29219068c5fe51a26; no voters: 
I20260709 12:58:20.083731  1314 raft_consensus.cc:2804] T c6480f4fe1b54658b39e73baa7c719fd P d88b316b7ecd49f29219068c5fe51a26 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:58:20.084326  1314 raft_consensus.cc:697] T c6480f4fe1b54658b39e73baa7c719fd P d88b316b7ecd49f29219068c5fe51a26 [term 1 LEADER]: Becoming Leader. State: Replica: d88b316b7ecd49f29219068c5fe51a26, State: Running, Role: LEADER
I20260709 12:58:20.085366  1314 consensus_queue.cc:237] T c6480f4fe1b54658b39e73baa7c719fd P d88b316b7ecd49f29219068c5fe51a26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:20.090358   989 catalog_manager.cc:5697] T df94d2062ca84496bc46321e6e4012fc P b3afae27e8744ee6921cd67f805e3e4a reported cstate change: term changed from 0 to 1, leader changed from <none> to b3afae27e8744ee6921cd67f805e3e4a (127.0.107.194). New cstate: current_term: 1 leader_uuid: "b3afae27e8744ee6921cd67f805e3e4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } health_report { overall_health: UNKNOWN } } }
I20260709 12:58:20.093132   990 catalog_manager.cc:5697] T 2cde8b0b8c5c46e78f405744f084c7c6 P d88b316b7ecd49f29219068c5fe51a26 reported cstate change: term changed from 0 to 1, leader changed from <none> to d88b316b7ecd49f29219068c5fe51a26 (127.0.107.193). New cstate: current_term: 1 leader_uuid: "d88b316b7ecd49f29219068c5fe51a26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } health_report { overall_health: UNKNOWN } } }
I20260709 12:58:20.095320  1313 raft_consensus.cc:493] T 1935c9164755416a965cccf80900d941 P d0e95822debe474ba6a700c661aa262f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:20.095741  1313 raft_consensus.cc:515] T 1935c9164755416a965cccf80900d941 P d0e95822debe474ba6a700c661aa262f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:20.097990  1313 leader_election.cc:290] T 1935c9164755416a965cccf80900d941 P d0e95822debe474ba6a700c661aa262f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d88b316b7ecd49f29219068c5fe51a26 (127.0.107.193:38591), b3afae27e8744ee6921cd67f805e3e4a (127.0.107.194:36785)
I20260709 12:58:20.099207  1095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1935c9164755416a965cccf80900d941" candidate_uuid: "d0e95822debe474ba6a700c661aa262f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d88b316b7ecd49f29219068c5fe51a26" is_pre_election: true
I20260709 12:58:20.100291  1170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1935c9164755416a965cccf80900d941" candidate_uuid: "d0e95822debe474ba6a700c661aa262f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3afae27e8744ee6921cd67f805e3e4a" is_pre_election: true
I20260709 12:58:20.101136  1170 raft_consensus.cc:2393] T 1935c9164755416a965cccf80900d941 P b3afae27e8744ee6921cd67f805e3e4a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0e95822debe474ba6a700c661aa262f in current term 1: Already voted for candidate d88b316b7ecd49f29219068c5fe51a26 in this term.
I20260709 12:58:20.102285  1209 leader_election.cc:304] T 1935c9164755416a965cccf80900d941 P d0e95822debe474ba6a700c661aa262f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0e95822debe474ba6a700c661aa262f; no voters: b3afae27e8744ee6921cd67f805e3e4a, d88b316b7ecd49f29219068c5fe51a26
I20260709 12:58:20.103113  1313 raft_consensus.cc:3060] T 1935c9164755416a965cccf80900d941 P d0e95822debe474ba6a700c661aa262f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:20.108469   990 catalog_manager.cc:5697] T c6480f4fe1b54658b39e73baa7c719fd P d88b316b7ecd49f29219068c5fe51a26 reported cstate change: term changed from 0 to 1, leader changed from <none> to d88b316b7ecd49f29219068c5fe51a26 (127.0.107.193). New cstate: current_term: 1 leader_uuid: "d88b316b7ecd49f29219068c5fe51a26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } health_report { overall_health: UNKNOWN } } }
I20260709 12:58:20.110502  1313 raft_consensus.cc:2749] T 1935c9164755416a965cccf80900d941 P d0e95822debe474ba6a700c661aa262f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:58:20.116876  1314 raft_consensus.cc:493] T 73e09365f8cd4513a4600030950af811 P d88b316b7ecd49f29219068c5fe51a26 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:20.117244  1314 raft_consensus.cc:515] T 73e09365f8cd4513a4600030950af811 P d88b316b7ecd49f29219068c5fe51a26 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:20.118865  1314 leader_election.cc:290] T 73e09365f8cd4513a4600030950af811 P d88b316b7ecd49f29219068c5fe51a26 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b3afae27e8744ee6921cd67f805e3e4a (127.0.107.194:36785), d0e95822debe474ba6a700c661aa262f (127.0.107.195:40863)
I20260709 12:58:20.119525  1170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73e09365f8cd4513a4600030950af811" candidate_uuid: "d88b316b7ecd49f29219068c5fe51a26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3afae27e8744ee6921cd67f805e3e4a" is_pre_election: true
I20260709 12:58:20.119863  1243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73e09365f8cd4513a4600030950af811" candidate_uuid: "d88b316b7ecd49f29219068c5fe51a26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0e95822debe474ba6a700c661aa262f" is_pre_election: true
I20260709 12:58:20.120217  1170 raft_consensus.cc:2468] T 73e09365f8cd4513a4600030950af811 P b3afae27e8744ee6921cd67f805e3e4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d88b316b7ecd49f29219068c5fe51a26 in term 0.
I20260709 12:58:20.120505  1243 raft_consensus.cc:2468] T 73e09365f8cd4513a4600030950af811 P d0e95822debe474ba6a700c661aa262f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d88b316b7ecd49f29219068c5fe51a26 in term 0.
I20260709 12:58:20.121387  1059 leader_election.cc:304] T 73e09365f8cd4513a4600030950af811 P d88b316b7ecd49f29219068c5fe51a26 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3afae27e8744ee6921cd67f805e3e4a, d88b316b7ecd49f29219068c5fe51a26; no voters: 
I20260709 12:58:20.122082  1314 raft_consensus.cc:2804] T 73e09365f8cd4513a4600030950af811 P d88b316b7ecd49f29219068c5fe51a26 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:58:20.122423  1314 raft_consensus.cc:493] T 73e09365f8cd4513a4600030950af811 P d88b316b7ecd49f29219068c5fe51a26 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:58:20.122728  1314 raft_consensus.cc:3060] T 73e09365f8cd4513a4600030950af811 P d88b316b7ecd49f29219068c5fe51a26 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:20.127516  1314 raft_consensus.cc:515] T 73e09365f8cd4513a4600030950af811 P d88b316b7ecd49f29219068c5fe51a26 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:20.129130  1314 leader_election.cc:290] T 73e09365f8cd4513a4600030950af811 P d88b316b7ecd49f29219068c5fe51a26 [CANDIDATE]: Term 1 election: Requested vote from peers b3afae27e8744ee6921cd67f805e3e4a (127.0.107.194:36785), d0e95822debe474ba6a700c661aa262f (127.0.107.195:40863)
I20260709 12:58:20.129791  1170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73e09365f8cd4513a4600030950af811" candidate_uuid: "d88b316b7ecd49f29219068c5fe51a26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3afae27e8744ee6921cd67f805e3e4a"
I20260709 12:58:20.130172  1243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73e09365f8cd4513a4600030950af811" candidate_uuid: "d88b316b7ecd49f29219068c5fe51a26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0e95822debe474ba6a700c661aa262f"
I20260709 12:58:20.130401  1170 raft_consensus.cc:3060] T 73e09365f8cd4513a4600030950af811 P b3afae27e8744ee6921cd67f805e3e4a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:20.130780  1243 raft_consensus.cc:3060] T 73e09365f8cd4513a4600030950af811 P d0e95822debe474ba6a700c661aa262f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:20.137681  1243 raft_consensus.cc:2468] T 73e09365f8cd4513a4600030950af811 P d0e95822debe474ba6a700c661aa262f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d88b316b7ecd49f29219068c5fe51a26 in term 1.
I20260709 12:58:20.137681  1170 raft_consensus.cc:2468] T 73e09365f8cd4513a4600030950af811 P b3afae27e8744ee6921cd67f805e3e4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d88b316b7ecd49f29219068c5fe51a26 in term 1.
I20260709 12:58:20.138978  1061 leader_election.cc:304] T 73e09365f8cd4513a4600030950af811 P d88b316b7ecd49f29219068c5fe51a26 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d0e95822debe474ba6a700c661aa262f, d88b316b7ecd49f29219068c5fe51a26; no voters: 
I20260709 12:58:20.139648  1314 raft_consensus.cc:2804] T 73e09365f8cd4513a4600030950af811 P d88b316b7ecd49f29219068c5fe51a26 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:58:20.140189  1314 raft_consensus.cc:697] T 73e09365f8cd4513a4600030950af811 P d88b316b7ecd49f29219068c5fe51a26 [term 1 LEADER]: Becoming Leader. State: Replica: d88b316b7ecd49f29219068c5fe51a26, State: Running, Role: LEADER
I20260709 12:58:20.140945  1314 consensus_queue.cc:237] T 73e09365f8cd4513a4600030950af811 P d88b316b7ecd49f29219068c5fe51a26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:20.148029   990 catalog_manager.cc:5697] T 73e09365f8cd4513a4600030950af811 P d88b316b7ecd49f29219068c5fe51a26 reported cstate change: term changed from 0 to 1, leader changed from <none> to d88b316b7ecd49f29219068c5fe51a26 (127.0.107.193). New cstate: current_term: 1 leader_uuid: "d88b316b7ecd49f29219068c5fe51a26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } health_report { overall_health: UNKNOWN } } }
I20260709 12:58:20.152218  1314 raft_consensus.cc:493] T 759e30de7d7b4bc6980dbf8c4ba8031f P d88b316b7ecd49f29219068c5fe51a26 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:20.153848  1313 consensus_queue.cc:1048] T 6198a4398a3140e8ad9f1a158956fe3e P d0e95822debe474ba6a700c661aa262f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:58:20.153980  1314 raft_consensus.cc:515] T 759e30de7d7b4bc6980dbf8c4ba8031f P d88b316b7ecd49f29219068c5fe51a26 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:20.156899  1171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "759e30de7d7b4bc6980dbf8c4ba8031f" candidate_uuid: "d88b316b7ecd49f29219068c5fe51a26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3afae27e8744ee6921cd67f805e3e4a" is_pre_election: true
I20260709 12:58:20.157621  1314 leader_election.cc:290] T 759e30de7d7b4bc6980dbf8c4ba8031f P d88b316b7ecd49f29219068c5fe51a26 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b3afae27e8744ee6921cd67f805e3e4a (127.0.107.194:36785), d0e95822debe474ba6a700c661aa262f (127.0.107.195:40863)
I20260709 12:58:20.157821  1243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "759e30de7d7b4bc6980dbf8c4ba8031f" candidate_uuid: "d88b316b7ecd49f29219068c5fe51a26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0e95822debe474ba6a700c661aa262f" is_pre_election: true
I20260709 12:58:20.158660  1243 raft_consensus.cc:2393] T 759e30de7d7b4bc6980dbf8c4ba8031f P d0e95822debe474ba6a700c661aa262f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d88b316b7ecd49f29219068c5fe51a26 in current term 1: Already voted for candidate b3afae27e8744ee6921cd67f805e3e4a in this term.
I20260709 12:58:20.159729  1061 leader_election.cc:304] T 759e30de7d7b4bc6980dbf8c4ba8031f P d88b316b7ecd49f29219068c5fe51a26 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d88b316b7ecd49f29219068c5fe51a26; no voters: b3afae27e8744ee6921cd67f805e3e4a, d0e95822debe474ba6a700c661aa262f
I20260709 12:58:20.160616  1314 raft_consensus.cc:3060] T 759e30de7d7b4bc6980dbf8c4ba8031f P d88b316b7ecd49f29219068c5fe51a26 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:20.170176  1314 raft_consensus.cc:2749] T 759e30de7d7b4bc6980dbf8c4ba8031f P d88b316b7ecd49f29219068c5fe51a26 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:58:20.173795  1315 consensus_queue.cc:1048] T 6198a4398a3140e8ad9f1a158956fe3e P d0e95822debe474ba6a700c661aa262f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:58:20.218197  1315 raft_consensus.cc:493] T d87ed5c1a25e40f8b7aa82d33ca30bf1 P d0e95822debe474ba6a700c661aa262f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:20.218828  1315 raft_consensus.cc:515] T d87ed5c1a25e40f8b7aa82d33ca30bf1 P d0e95822debe474ba6a700c661aa262f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:20.220763  1315 leader_election.cc:290] T d87ed5c1a25e40f8b7aa82d33ca30bf1 P d0e95822debe474ba6a700c661aa262f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d88b316b7ecd49f29219068c5fe51a26 (127.0.107.193:38591), b3afae27e8744ee6921cd67f805e3e4a (127.0.107.194:36785)
I20260709 12:58:20.221722  1095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d87ed5c1a25e40f8b7aa82d33ca30bf1" candidate_uuid: "d0e95822debe474ba6a700c661aa262f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d88b316b7ecd49f29219068c5fe51a26" is_pre_election: true
I20260709 12:58:20.221889  1170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d87ed5c1a25e40f8b7aa82d33ca30bf1" candidate_uuid: "d0e95822debe474ba6a700c661aa262f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3afae27e8744ee6921cd67f805e3e4a" is_pre_election: true
I20260709 12:58:20.222474  1095 raft_consensus.cc:2468] T d87ed5c1a25e40f8b7aa82d33ca30bf1 P d88b316b7ecd49f29219068c5fe51a26 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0e95822debe474ba6a700c661aa262f in term 0.
I20260709 12:58:20.222481  1170 raft_consensus.cc:2468] T d87ed5c1a25e40f8b7aa82d33ca30bf1 P b3afae27e8744ee6921cd67f805e3e4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0e95822debe474ba6a700c661aa262f in term 0.
I20260709 12:58:20.223729  1209 leader_election.cc:304] T d87ed5c1a25e40f8b7aa82d33ca30bf1 P d0e95822debe474ba6a700c661aa262f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3afae27e8744ee6921cd67f805e3e4a, d0e95822debe474ba6a700c661aa262f; no voters: 
I20260709 12:58:20.224383  1315 raft_consensus.cc:2804] T d87ed5c1a25e40f8b7aa82d33ca30bf1 P d0e95822debe474ba6a700c661aa262f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:58:20.224722  1315 raft_consensus.cc:493] T d87ed5c1a25e40f8b7aa82d33ca30bf1 P d0e95822debe474ba6a700c661aa262f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:58:20.224961  1315 raft_consensus.cc:3060] T d87ed5c1a25e40f8b7aa82d33ca30bf1 P d0e95822debe474ba6a700c661aa262f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:20.230222  1315 raft_consensus.cc:515] T d87ed5c1a25e40f8b7aa82d33ca30bf1 P d0e95822debe474ba6a700c661aa262f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:20.231877  1315 leader_election.cc:290] T d87ed5c1a25e40f8b7aa82d33ca30bf1 P d0e95822debe474ba6a700c661aa262f [CANDIDATE]: Term 1 election: Requested vote from peers d88b316b7ecd49f29219068c5fe51a26 (127.0.107.193:38591), b3afae27e8744ee6921cd67f805e3e4a (127.0.107.194:36785)
I20260709 12:58:20.232633  1095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d87ed5c1a25e40f8b7aa82d33ca30bf1" candidate_uuid: "d0e95822debe474ba6a700c661aa262f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d88b316b7ecd49f29219068c5fe51a26"
I20260709 12:58:20.232982  1170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d87ed5c1a25e40f8b7aa82d33ca30bf1" candidate_uuid: "d0e95822debe474ba6a700c661aa262f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3afae27e8744ee6921cd67f805e3e4a"
I20260709 12:58:20.233265  1095 raft_consensus.cc:3060] T d87ed5c1a25e40f8b7aa82d33ca30bf1 P d88b316b7ecd49f29219068c5fe51a26 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:20.233546  1170 raft_consensus.cc:3060] T d87ed5c1a25e40f8b7aa82d33ca30bf1 P b3afae27e8744ee6921cd67f805e3e4a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:20.240732  1170 raft_consensus.cc:2468] T d87ed5c1a25e40f8b7aa82d33ca30bf1 P b3afae27e8744ee6921cd67f805e3e4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0e95822debe474ba6a700c661aa262f in term 1.
I20260709 12:58:20.240732  1095 raft_consensus.cc:2468] T d87ed5c1a25e40f8b7aa82d33ca30bf1 P d88b316b7ecd49f29219068c5fe51a26 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0e95822debe474ba6a700c661aa262f in term 1.
I20260709 12:58:20.242026  1209 leader_election.cc:304] T d87ed5c1a25e40f8b7aa82d33ca30bf1 P d0e95822debe474ba6a700c661aa262f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3afae27e8744ee6921cd67f805e3e4a, d0e95822debe474ba6a700c661aa262f; no voters: 
I20260709 12:58:20.242764  1315 raft_consensus.cc:2804] T d87ed5c1a25e40f8b7aa82d33ca30bf1 P d0e95822debe474ba6a700c661aa262f [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:58:20.243197  1315 raft_consensus.cc:697] T d87ed5c1a25e40f8b7aa82d33ca30bf1 P d0e95822debe474ba6a700c661aa262f [term 1 LEADER]: Becoming Leader. State: Replica: d0e95822debe474ba6a700c661aa262f, State: Running, Role: LEADER
I20260709 12:58:20.243894  1315 consensus_queue.cc:237] T d87ed5c1a25e40f8b7aa82d33ca30bf1 P d0e95822debe474ba6a700c661aa262f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:20.251642   990 catalog_manager.cc:5697] T d87ed5c1a25e40f8b7aa82d33ca30bf1 P d0e95822debe474ba6a700c661aa262f reported cstate change: term changed from 0 to 1, leader changed from <none> to d0e95822debe474ba6a700c661aa262f (127.0.107.195). New cstate: current_term: 1 leader_uuid: "d0e95822debe474ba6a700c661aa262f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } health_report { overall_health: HEALTHY } } }
I20260709 12:58:20.303802  1314 consensus_queue.cc:1048] T 1935c9164755416a965cccf80900d941 P d88b316b7ecd49f29219068c5fe51a26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:58:20.315665  1326 consensus_queue.cc:1048] T 1935c9164755416a965cccf80900d941 P d88b316b7ecd49f29219068c5fe51a26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:58:20.333983  1291 consensus_queue.cc:1048] T 759e30de7d7b4bc6980dbf8c4ba8031f P b3afae27e8744ee6921cd67f805e3e4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:58:20.345824  1341 consensus_queue.cc:1048] T 759e30de7d7b4bc6980dbf8c4ba8031f P b3afae27e8744ee6921cd67f805e3e4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:58:20.347874  1326 consensus_queue.cc:1048] T 2685bf6c4fd043a481d1d05136dbbd5e P d88b316b7ecd49f29219068c5fe51a26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:58:20.360085  1293 consensus_queue.cc:1048] T 2685bf6c4fd043a481d1d05136dbbd5e P d88b316b7ecd49f29219068c5fe51a26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:58:20.375068  1293 raft_consensus.cc:493] T 8ee1d07b2c9444f8ad28921d69d297bf P d88b316b7ecd49f29219068c5fe51a26 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:20.375665  1293 raft_consensus.cc:515] T 8ee1d07b2c9444f8ad28921d69d297bf P d88b316b7ecd49f29219068c5fe51a26 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:20.378325  1293 leader_election.cc:290] T 8ee1d07b2c9444f8ad28921d69d297bf P d88b316b7ecd49f29219068c5fe51a26 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b3afae27e8744ee6921cd67f805e3e4a (127.0.107.194:36785), d0e95822debe474ba6a700c661aa262f (127.0.107.195:40863)
I20260709 12:58:20.379096  1170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ee1d07b2c9444f8ad28921d69d297bf" candidate_uuid: "d88b316b7ecd49f29219068c5fe51a26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3afae27e8744ee6921cd67f805e3e4a" is_pre_election: true
I20260709 12:58:20.379462  1243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ee1d07b2c9444f8ad28921d69d297bf" candidate_uuid: "d88b316b7ecd49f29219068c5fe51a26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0e95822debe474ba6a700c661aa262f" is_pre_election: true
I20260709 12:58:20.379941  1170 raft_consensus.cc:2468] T 8ee1d07b2c9444f8ad28921d69d297bf P b3afae27e8744ee6921cd67f805e3e4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d88b316b7ecd49f29219068c5fe51a26 in term 0.
I20260709 12:58:20.380136  1243 raft_consensus.cc:2468] T 8ee1d07b2c9444f8ad28921d69d297bf P d0e95822debe474ba6a700c661aa262f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d88b316b7ecd49f29219068c5fe51a26 in term 0.
I20260709 12:58:20.380988  1059 leader_election.cc:304] T 8ee1d07b2c9444f8ad28921d69d297bf P d88b316b7ecd49f29219068c5fe51a26 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3afae27e8744ee6921cd67f805e3e4a, d88b316b7ecd49f29219068c5fe51a26; no voters: 
I20260709 12:58:20.381758  1293 raft_consensus.cc:2804] T 8ee1d07b2c9444f8ad28921d69d297bf P d88b316b7ecd49f29219068c5fe51a26 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:58:20.382081  1293 raft_consensus.cc:493] T 8ee1d07b2c9444f8ad28921d69d297bf P d88b316b7ecd49f29219068c5fe51a26 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:58:20.382369  1293 raft_consensus.cc:3060] T 8ee1d07b2c9444f8ad28921d69d297bf P d88b316b7ecd49f29219068c5fe51a26 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:20.387444  1293 raft_consensus.cc:515] T 8ee1d07b2c9444f8ad28921d69d297bf P d88b316b7ecd49f29219068c5fe51a26 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:20.389001  1293 leader_election.cc:290] T 8ee1d07b2c9444f8ad28921d69d297bf P d88b316b7ecd49f29219068c5fe51a26 [CANDIDATE]: Term 1 election: Requested vote from peers b3afae27e8744ee6921cd67f805e3e4a (127.0.107.194:36785), d0e95822debe474ba6a700c661aa262f (127.0.107.195:40863)
I20260709 12:58:20.389912  1170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ee1d07b2c9444f8ad28921d69d297bf" candidate_uuid: "d88b316b7ecd49f29219068c5fe51a26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3afae27e8744ee6921cd67f805e3e4a"
I20260709 12:58:20.390153  1243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ee1d07b2c9444f8ad28921d69d297bf" candidate_uuid: "d88b316b7ecd49f29219068c5fe51a26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0e95822debe474ba6a700c661aa262f"
I20260709 12:58:20.390568  1170 raft_consensus.cc:3060] T 8ee1d07b2c9444f8ad28921d69d297bf P b3afae27e8744ee6921cd67f805e3e4a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:20.390727  1243 raft_consensus.cc:3060] T 8ee1d07b2c9444f8ad28921d69d297bf P d0e95822debe474ba6a700c661aa262f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:20.395604  1243 raft_consensus.cc:2468] T 8ee1d07b2c9444f8ad28921d69d297bf P d0e95822debe474ba6a700c661aa262f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d88b316b7ecd49f29219068c5fe51a26 in term 1.
I20260709 12:58:20.395725  1170 raft_consensus.cc:2468] T 8ee1d07b2c9444f8ad28921d69d297bf P b3afae27e8744ee6921cd67f805e3e4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d88b316b7ecd49f29219068c5fe51a26 in term 1.
I20260709 12:58:20.396724  1061 leader_election.cc:304] T 8ee1d07b2c9444f8ad28921d69d297bf P d88b316b7ecd49f29219068c5fe51a26 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d0e95822debe474ba6a700c661aa262f, d88b316b7ecd49f29219068c5fe51a26; no voters: 
I20260709 12:58:20.397410  1293 raft_consensus.cc:2804] T 8ee1d07b2c9444f8ad28921d69d297bf P d88b316b7ecd49f29219068c5fe51a26 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:58:20.397760  1293 raft_consensus.cc:697] T 8ee1d07b2c9444f8ad28921d69d297bf P d88b316b7ecd49f29219068c5fe51a26 [term 1 LEADER]: Becoming Leader. State: Replica: d88b316b7ecd49f29219068c5fe51a26, State: Running, Role: LEADER
I20260709 12:58:20.398476  1293 consensus_queue.cc:237] T 8ee1d07b2c9444f8ad28921d69d297bf P d88b316b7ecd49f29219068c5fe51a26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:20.406316   989 catalog_manager.cc:5697] T 8ee1d07b2c9444f8ad28921d69d297bf P d88b316b7ecd49f29219068c5fe51a26 reported cstate change: term changed from 0 to 1, leader changed from <none> to d88b316b7ecd49f29219068c5fe51a26 (127.0.107.193). New cstate: current_term: 1 leader_uuid: "d88b316b7ecd49f29219068c5fe51a26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } health_report { overall_health: UNKNOWN } } }
I20260709 12:58:20.522810  1341 consensus_queue.cc:1048] T df94d2062ca84496bc46321e6e4012fc P b3afae27e8744ee6921cd67f805e3e4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:58:20.534258  1341 consensus_queue.cc:1048] T df94d2062ca84496bc46321e6e4012fc P b3afae27e8744ee6921cd67f805e3e4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:58:20.572175  1293 consensus_queue.cc:1048] T c6480f4fe1b54658b39e73baa7c719fd P d88b316b7ecd49f29219068c5fe51a26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:58:20.583832  1353 consensus_queue.cc:1048] T c6480f4fe1b54658b39e73baa7c719fd P d88b316b7ecd49f29219068c5fe51a26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:58:20.616451  1353 consensus_queue.cc:1048] T 73e09365f8cd4513a4600030950af811 P d88b316b7ecd49f29219068c5fe51a26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:58:20.628232  1293 consensus_queue.cc:1048] T 73e09365f8cd4513a4600030950af811 P d88b316b7ecd49f29219068c5fe51a26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:58:20.629415  1313 consensus_queue.cc:1048] T d87ed5c1a25e40f8b7aa82d33ca30bf1 P d0e95822debe474ba6a700c661aa262f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:58:20.634269  1314 consensus_queue.cc:1048] T 2cde8b0b8c5c46e78f405744f084c7c6 P d88b316b7ecd49f29219068c5fe51a26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:58:20.650404  1314 consensus_queue.cc:1048] T 2cde8b0b8c5c46e78f405744f084c7c6 P d88b316b7ecd49f29219068c5fe51a26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:58:20.657279  1313 consensus_queue.cc:1048] T d87ed5c1a25e40f8b7aa82d33ca30bf1 P d0e95822debe474ba6a700c661aa262f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:58:20.689086  1244 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "6198a4398a3140e8ad9f1a158956fe3e"
dest_uuid: "d0e95822debe474ba6a700c661aa262f"
mode: GRACEFUL
new_leader_uuid: "d88b316b7ecd49f29219068c5fe51a26"
 from {username='slave'} at 127.0.0.1:33220
I20260709 12:58:20.689900  1244 raft_consensus.cc:606] T 6198a4398a3140e8ad9f1a158956fe3e P d0e95822debe474ba6a700c661aa262f [term 1 LEADER]: Received request to transfer leadership to TS d88b316b7ecd49f29219068c5fe51a26
I20260709 12:58:20.707134  1171 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "759e30de7d7b4bc6980dbf8c4ba8031f"
dest_uuid: "b3afae27e8744ee6921cd67f805e3e4a"
mode: GRACEFUL
new_leader_uuid: "d88b316b7ecd49f29219068c5fe51a26"
 from {username='slave'} at 127.0.0.1:42552
I20260709 12:58:20.707794  1171 raft_consensus.cc:606] T 759e30de7d7b4bc6980dbf8c4ba8031f P b3afae27e8744ee6921cd67f805e3e4a [term 1 LEADER]: Received request to transfer leadership to TS d88b316b7ecd49f29219068c5fe51a26
I20260709 12:58:20.728994  1095 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "2cde8b0b8c5c46e78f405744f084c7c6"
dest_uuid: "d88b316b7ecd49f29219068c5fe51a26"
mode: GRACEFUL
new_leader_uuid: "b3afae27e8744ee6921cd67f805e3e4a"
 from {username='slave'} at 127.0.0.1:35578
I20260709 12:58:20.729493  1095 raft_consensus.cc:606] T 2cde8b0b8c5c46e78f405744f084c7c6 P d88b316b7ecd49f29219068c5fe51a26 [term 1 LEADER]: Received request to transfer leadership to TS b3afae27e8744ee6921cd67f805e3e4a
I20260709 12:58:20.734861  1095 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "2685bf6c4fd043a481d1d05136dbbd5e"
dest_uuid: "d88b316b7ecd49f29219068c5fe51a26"
mode: GRACEFUL
new_leader_uuid: "b3afae27e8744ee6921cd67f805e3e4a"
 from {username='slave'} at 127.0.0.1:35578
I20260709 12:58:20.735381  1095 raft_consensus.cc:606] T 2685bf6c4fd043a481d1d05136dbbd5e P d88b316b7ecd49f29219068c5fe51a26 [term 1 LEADER]: Received request to transfer leadership to TS b3afae27e8744ee6921cd67f805e3e4a
I20260709 12:58:20.740082  1244 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "d87ed5c1a25e40f8b7aa82d33ca30bf1"
dest_uuid: "d0e95822debe474ba6a700c661aa262f"
mode: GRACEFUL
new_leader_uuid: "b3afae27e8744ee6921cd67f805e3e4a"
 from {username='slave'} at 127.0.0.1:33220
I20260709 12:58:20.740573  1244 raft_consensus.cc:606] T d87ed5c1a25e40f8b7aa82d33ca30bf1 P d0e95822debe474ba6a700c661aa262f [term 1 LEADER]: Received request to transfer leadership to TS b3afae27e8744ee6921cd67f805e3e4a
I20260709 12:58:20.745141  1095 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "c6480f4fe1b54658b39e73baa7c719fd"
dest_uuid: "d88b316b7ecd49f29219068c5fe51a26"
mode: GRACEFUL
new_leader_uuid: "b3afae27e8744ee6921cd67f805e3e4a"
 from {username='slave'} at 127.0.0.1:35578
I20260709 12:58:20.745604  1095 raft_consensus.cc:606] T c6480f4fe1b54658b39e73baa7c719fd P d88b316b7ecd49f29219068c5fe51a26 [term 1 LEADER]: Received request to transfer leadership to TS b3afae27e8744ee6921cd67f805e3e4a
I20260709 12:58:20.750136  1171 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "df94d2062ca84496bc46321e6e4012fc"
dest_uuid: "b3afae27e8744ee6921cd67f805e3e4a"
mode: GRACEFUL
new_leader_uuid: "d0e95822debe474ba6a700c661aa262f"
 from {username='slave'} at 127.0.0.1:42552
I20260709 12:58:20.750687  1171 raft_consensus.cc:606] T df94d2062ca84496bc46321e6e4012fc P b3afae27e8744ee6921cd67f805e3e4a [term 1 LEADER]: Received request to transfer leadership to TS d0e95822debe474ba6a700c661aa262f
I20260709 12:58:20.871852  1314 raft_consensus.cc:993] T 2685bf6c4fd043a481d1d05136dbbd5e P d88b316b7ecd49f29219068c5fe51a26: : Instructing follower b3afae27e8744ee6921cd67f805e3e4a to start an election
I20260709 12:58:20.872427  1326 raft_consensus.cc:1081] T 2685bf6c4fd043a481d1d05136dbbd5e P d88b316b7ecd49f29219068c5fe51a26 [term 1 LEADER]: Signalling peer b3afae27e8744ee6921cd67f805e3e4a to start an election
I20260709 12:58:20.874044  1171 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "2685bf6c4fd043a481d1d05136dbbd5e"
dest_uuid: "b3afae27e8744ee6921cd67f805e3e4a"
 from {username='slave'} at 127.0.0.1:34394
I20260709 12:58:20.874663  1171 raft_consensus.cc:493] T 2685bf6c4fd043a481d1d05136dbbd5e P b3afae27e8744ee6921cd67f805e3e4a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:58:20.875010  1171 raft_consensus.cc:3060] T 2685bf6c4fd043a481d1d05136dbbd5e P b3afae27e8744ee6921cd67f805e3e4a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:58:20.880589  1171 raft_consensus.cc:515] T 2685bf6c4fd043a481d1d05136dbbd5e P b3afae27e8744ee6921cd67f805e3e4a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:20.882179  1171 leader_election.cc:290] T 2685bf6c4fd043a481d1d05136dbbd5e P b3afae27e8744ee6921cd67f805e3e4a [CANDIDATE]: Term 2 election: Requested vote from peers d88b316b7ecd49f29219068c5fe51a26 (127.0.107.193:38591), d0e95822debe474ba6a700c661aa262f (127.0.107.195:40863)
I20260709 12:58:20.883141  1095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2685bf6c4fd043a481d1d05136dbbd5e" candidate_uuid: "b3afae27e8744ee6921cd67f805e3e4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d88b316b7ecd49f29219068c5fe51a26"
I20260709 12:58:20.883325  1244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2685bf6c4fd043a481d1d05136dbbd5e" candidate_uuid: "b3afae27e8744ee6921cd67f805e3e4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d0e95822debe474ba6a700c661aa262f"
I20260709 12:58:20.883827  1244 raft_consensus.cc:3060] T 2685bf6c4fd043a481d1d05136dbbd5e P d0e95822debe474ba6a700c661aa262f [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:58:20.883795  1095 raft_consensus.cc:3055] T 2685bf6c4fd043a481d1d05136dbbd5e P d88b316b7ecd49f29219068c5fe51a26 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:58:20.884332  1095 raft_consensus.cc:740] T 2685bf6c4fd043a481d1d05136dbbd5e P d88b316b7ecd49f29219068c5fe51a26 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d88b316b7ecd49f29219068c5fe51a26, State: Running, Role: LEADER
I20260709 12:58:20.885176  1095 consensus_queue.cc:260] T 2685bf6c4fd043a481d1d05136dbbd5e P d88b316b7ecd49f29219068c5fe51a26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:20.886389  1095 raft_consensus.cc:3060] T 2685bf6c4fd043a481d1d05136dbbd5e P d88b316b7ecd49f29219068c5fe51a26 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:58:20.891456  1244 raft_consensus.cc:2468] T 2685bf6c4fd043a481d1d05136dbbd5e P d0e95822debe474ba6a700c661aa262f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3afae27e8744ee6921cd67f805e3e4a in term 2.
I20260709 12:58:20.892812  1136 leader_election.cc:304] T 2685bf6c4fd043a481d1d05136dbbd5e P b3afae27e8744ee6921cd67f805e3e4a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3afae27e8744ee6921cd67f805e3e4a, d0e95822debe474ba6a700c661aa262f; no voters: 
I20260709 12:58:20.893554  1341 raft_consensus.cc:2804] T 2685bf6c4fd043a481d1d05136dbbd5e P b3afae27e8744ee6921cd67f805e3e4a [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:58:20.893468  1095 raft_consensus.cc:2468] T 2685bf6c4fd043a481d1d05136dbbd5e P d88b316b7ecd49f29219068c5fe51a26 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3afae27e8744ee6921cd67f805e3e4a in term 2.
I20260709 12:58:20.894131  1341 raft_consensus.cc:697] T 2685bf6c4fd043a481d1d05136dbbd5e P b3afae27e8744ee6921cd67f805e3e4a [term 2 LEADER]: Becoming Leader. State: Replica: b3afae27e8744ee6921cd67f805e3e4a, State: Running, Role: LEADER
I20260709 12:58:20.895049  1341 consensus_queue.cc:237] T 2685bf6c4fd043a481d1d05136dbbd5e P b3afae27e8744ee6921cd67f805e3e4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:20.902105   989 catalog_manager.cc:5697] T 2685bf6c4fd043a481d1d05136dbbd5e P b3afae27e8744ee6921cd67f805e3e4a reported cstate change: term changed from 1 to 2, leader changed from d88b316b7ecd49f29219068c5fe51a26 (127.0.107.193) to b3afae27e8744ee6921cd67f805e3e4a (127.0.107.194). New cstate: current_term: 2 leader_uuid: "b3afae27e8744ee6921cd67f805e3e4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } health_report { overall_health: UNKNOWN } } }
I20260709 12:58:20.928336  1326 consensus_queue.cc:1048] T 8ee1d07b2c9444f8ad28921d69d297bf P d88b316b7ecd49f29219068c5fe51a26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:58:20.929899  1326 consensus_queue.cc:1048] T 8ee1d07b2c9444f8ad28921d69d297bf P d88b316b7ecd49f29219068c5fe51a26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:58:20.963500  1291 raft_consensus.cc:993] T df94d2062ca84496bc46321e6e4012fc P b3afae27e8744ee6921cd67f805e3e4a: : Instructing follower d0e95822debe474ba6a700c661aa262f to start an election
I20260709 12:58:20.964051  1291 raft_consensus.cc:1081] T df94d2062ca84496bc46321e6e4012fc P b3afae27e8744ee6921cd67f805e3e4a [term 1 LEADER]: Signalling peer d0e95822debe474ba6a700c661aa262f to start an election
I20260709 12:58:20.965811  1244 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "df94d2062ca84496bc46321e6e4012fc"
dest_uuid: "d0e95822debe474ba6a700c661aa262f"
 from {username='slave'} at 127.0.0.1:33242
I20260709 12:58:20.966485  1244 raft_consensus.cc:493] T df94d2062ca84496bc46321e6e4012fc P d0e95822debe474ba6a700c661aa262f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:58:20.966892  1244 raft_consensus.cc:3060] T df94d2062ca84496bc46321e6e4012fc P d0e95822debe474ba6a700c661aa262f [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:58:20.972730  1332 raft_consensus.cc:993] T 759e30de7d7b4bc6980dbf8c4ba8031f P b3afae27e8744ee6921cd67f805e3e4a: : Instructing follower d88b316b7ecd49f29219068c5fe51a26 to start an election
I20260709 12:58:20.973233  1291 raft_consensus.cc:1081] T 759e30de7d7b4bc6980dbf8c4ba8031f P b3afae27e8744ee6921cd67f805e3e4a [term 1 LEADER]: Signalling peer d88b316b7ecd49f29219068c5fe51a26 to start an election
I20260709 12:58:20.974797  1095 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "759e30de7d7b4bc6980dbf8c4ba8031f"
dest_uuid: "d88b316b7ecd49f29219068c5fe51a26"
 from {username='slave'} at 127.0.0.1:49984
I20260709 12:58:20.975317  1095 raft_consensus.cc:493] T 759e30de7d7b4bc6980dbf8c4ba8031f P d88b316b7ecd49f29219068c5fe51a26 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:58:20.975160  1244 raft_consensus.cc:515] T df94d2062ca84496bc46321e6e4012fc P d0e95822debe474ba6a700c661aa262f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:20.975708  1095 raft_consensus.cc:3060] T 759e30de7d7b4bc6980dbf8c4ba8031f P d88b316b7ecd49f29219068c5fe51a26 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:58:20.978317  1094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df94d2062ca84496bc46321e6e4012fc" candidate_uuid: "d0e95822debe474ba6a700c661aa262f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d88b316b7ecd49f29219068c5fe51a26"
I20260709 12:58:20.978979  1094 raft_consensus.cc:3060] T df94d2062ca84496bc46321e6e4012fc P d88b316b7ecd49f29219068c5fe51a26 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:58:20.981153  1314 raft_consensus.cc:993] T c6480f4fe1b54658b39e73baa7c719fd P d88b316b7ecd49f29219068c5fe51a26: : Instructing follower b3afae27e8744ee6921cd67f805e3e4a to start an election
I20260709 12:58:20.981614  1314 raft_consensus.cc:1081] T c6480f4fe1b54658b39e73baa7c719fd P d88b316b7ecd49f29219068c5fe51a26 [term 1 LEADER]: Signalling peer b3afae27e8744ee6921cd67f805e3e4a to start an election
I20260709 12:58:20.981190  1170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df94d2062ca84496bc46321e6e4012fc" candidate_uuid: "d0e95822debe474ba6a700c661aa262f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "b3afae27e8744ee6921cd67f805e3e4a"
I20260709 12:58:20.982618  1170 raft_consensus.cc:3055] T df94d2062ca84496bc46321e6e4012fc P b3afae27e8744ee6921cd67f805e3e4a [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:58:20.982937  1170 raft_consensus.cc:740] T df94d2062ca84496bc46321e6e4012fc P b3afae27e8744ee6921cd67f805e3e4a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b3afae27e8744ee6921cd67f805e3e4a, State: Running, Role: LEADER
I20260709 12:58:20.983598  1169 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "c6480f4fe1b54658b39e73baa7c719fd"
dest_uuid: "b3afae27e8744ee6921cd67f805e3e4a"
 from {username='slave'} at 127.0.0.1:34394
I20260709 12:58:20.983393  1095 raft_consensus.cc:515] T 759e30de7d7b4bc6980dbf8c4ba8031f P d88b316b7ecd49f29219068c5fe51a26 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:20.984187  1169 raft_consensus.cc:493] T c6480f4fe1b54658b39e73baa7c719fd P b3afae27e8744ee6921cd67f805e3e4a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:58:20.983836  1170 consensus_queue.cc:260] T df94d2062ca84496bc46321e6e4012fc P b3afae27e8744ee6921cd67f805e3e4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:20.984552  1169 raft_consensus.cc:3060] T c6480f4fe1b54658b39e73baa7c719fd P b3afae27e8744ee6921cd67f805e3e4a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:58:20.985113  1170 raft_consensus.cc:3060] T df94d2062ca84496bc46321e6e4012fc P b3afae27e8744ee6921cd67f805e3e4a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:58:20.985621  1095 leader_election.cc:290] T 759e30de7d7b4bc6980dbf8c4ba8031f P d88b316b7ecd49f29219068c5fe51a26 [CANDIDATE]: Term 2 election: Requested vote from peers b3afae27e8744ee6921cd67f805e3e4a (127.0.107.194:36785), d0e95822debe474ba6a700c661aa262f (127.0.107.195:40863)
I20260709 12:58:20.987154  1094 raft_consensus.cc:2468] T df94d2062ca84496bc46321e6e4012fc P d88b316b7ecd49f29219068c5fe51a26 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0e95822debe474ba6a700c661aa262f in term 2.
I20260709 12:58:20.991261  1243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "759e30de7d7b4bc6980dbf8c4ba8031f" candidate_uuid: "d88b316b7ecd49f29219068c5fe51a26" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d0e95822debe474ba6a700c661aa262f"
I20260709 12:58:20.991870  1243 raft_consensus.cc:3060] T 759e30de7d7b4bc6980dbf8c4ba8031f P d0e95822debe474ba6a700c661aa262f [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:58:20.991792  1171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "759e30de7d7b4bc6980dbf8c4ba8031f" candidate_uuid: "d88b316b7ecd49f29219068c5fe51a26" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "b3afae27e8744ee6921cd67f805e3e4a"
I20260709 12:58:20.990845  1169 raft_consensus.cc:515] T c6480f4fe1b54658b39e73baa7c719fd P b3afae27e8744ee6921cd67f805e3e4a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:20.992972  1208 leader_election.cc:304] T df94d2062ca84496bc46321e6e4012fc P d0e95822debe474ba6a700c661aa262f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d0e95822debe474ba6a700c661aa262f, d88b316b7ecd49f29219068c5fe51a26; no voters: 
I20260709 12:58:20.995927  1169 leader_election.cc:290] T c6480f4fe1b54658b39e73baa7c719fd P b3afae27e8744ee6921cd67f805e3e4a [CANDIDATE]: Term 2 election: Requested vote from peers d88b316b7ecd49f29219068c5fe51a26 (127.0.107.193:38591), d0e95822debe474ba6a700c661aa262f (127.0.107.195:40863)
I20260709 12:58:20.996517  1313 raft_consensus.cc:2804] T df94d2062ca84496bc46321e6e4012fc P d0e95822debe474ba6a700c661aa262f [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:58:20.993134  1244 leader_election.cc:290] T df94d2062ca84496bc46321e6e4012fc P d0e95822debe474ba6a700c661aa262f [CANDIDATE]: Term 2 election: Requested vote from peers d88b316b7ecd49f29219068c5fe51a26 (127.0.107.193:38591), b3afae27e8744ee6921cd67f805e3e4a (127.0.107.194:36785)
I20260709 12:58:20.997219  1094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6480f4fe1b54658b39e73baa7c719fd" candidate_uuid: "b3afae27e8744ee6921cd67f805e3e4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d88b316b7ecd49f29219068c5fe51a26"
I20260709 12:58:20.997982  1094 raft_consensus.cc:3055] T c6480f4fe1b54658b39e73baa7c719fd P d88b316b7ecd49f29219068c5fe51a26 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:58:20.998349  1094 raft_consensus.cc:740] T c6480f4fe1b54658b39e73baa7c719fd P d88b316b7ecd49f29219068c5fe51a26 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d88b316b7ecd49f29219068c5fe51a26, State: Running, Role: LEADER
I20260709 12:58:20.994782  1170 raft_consensus.cc:2468] T df94d2062ca84496bc46321e6e4012fc P b3afae27e8744ee6921cd67f805e3e4a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0e95822debe474ba6a700c661aa262f in term 2.
I20260709 12:58:20.993112  1171 raft_consensus.cc:3055] T 759e30de7d7b4bc6980dbf8c4ba8031f P b3afae27e8744ee6921cd67f805e3e4a [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:58:20.997025  1313 raft_consensus.cc:697] T df94d2062ca84496bc46321e6e4012fc P d0e95822debe474ba6a700c661aa262f [term 2 LEADER]: Becoming Leader. State: Replica: d0e95822debe474ba6a700c661aa262f, State: Running, Role: LEADER
I20260709 12:58:20.999221  1094 consensus_queue.cc:260] T c6480f4fe1b54658b39e73baa7c719fd P d88b316b7ecd49f29219068c5fe51a26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:21.000123  1313 consensus_queue.cc:237] T df94d2062ca84496bc46321e6e4012fc P d0e95822debe474ba6a700c661aa262f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:21.000630  1094 raft_consensus.cc:3060] T c6480f4fe1b54658b39e73baa7c719fd P d88b316b7ecd49f29219068c5fe51a26 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:58:20.997219  1245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6480f4fe1b54658b39e73baa7c719fd" candidate_uuid: "b3afae27e8744ee6921cd67f805e3e4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d0e95822debe474ba6a700c661aa262f"
I20260709 12:58:21.002740  1245 raft_consensus.cc:3060] T c6480f4fe1b54658b39e73baa7c719fd P d0e95822debe474ba6a700c661aa262f [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:58:21.003995  1243 raft_consensus.cc:2468] T 759e30de7d7b4bc6980dbf8c4ba8031f P d0e95822debe474ba6a700c661aa262f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d88b316b7ecd49f29219068c5fe51a26 in term 2.
I20260709 12:58:21.005301  1061 leader_election.cc:304] T 759e30de7d7b4bc6980dbf8c4ba8031f P d88b316b7ecd49f29219068c5fe51a26 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d0e95822debe474ba6a700c661aa262f, d88b316b7ecd49f29219068c5fe51a26; no voters: 
I20260709 12:58:21.006218  1293 raft_consensus.cc:2804] T 759e30de7d7b4bc6980dbf8c4ba8031f P d88b316b7ecd49f29219068c5fe51a26 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:58:21.006726  1293 raft_consensus.cc:697] T 759e30de7d7b4bc6980dbf8c4ba8031f P d88b316b7ecd49f29219068c5fe51a26 [term 2 LEADER]: Becoming Leader. State: Replica: d88b316b7ecd49f29219068c5fe51a26, State: Running, Role: LEADER
I20260709 12:58:21.007561  1293 consensus_queue.cc:237] T 759e30de7d7b4bc6980dbf8c4ba8031f P d88b316b7ecd49f29219068c5fe51a26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:21.008283  1094 raft_consensus.cc:2468] T c6480f4fe1b54658b39e73baa7c719fd P d88b316b7ecd49f29219068c5fe51a26 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3afae27e8744ee6921cd67f805e3e4a in term 2.
I20260709 12:58:21.009389  1171 raft_consensus.cc:740] T 759e30de7d7b4bc6980dbf8c4ba8031f P b3afae27e8744ee6921cd67f805e3e4a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b3afae27e8744ee6921cd67f805e3e4a, State: Running, Role: LEADER
I20260709 12:58:21.009600  1133 leader_election.cc:304] T c6480f4fe1b54658b39e73baa7c719fd P b3afae27e8744ee6921cd67f805e3e4a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3afae27e8744ee6921cd67f805e3e4a, d88b316b7ecd49f29219068c5fe51a26; no voters: 
I20260709 12:58:21.010605  1291 raft_consensus.cc:2804] T c6480f4fe1b54658b39e73baa7c719fd P b3afae27e8744ee6921cd67f805e3e4a [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:58:21.009965   989 catalog_manager.cc:5697] T df94d2062ca84496bc46321e6e4012fc P d0e95822debe474ba6a700c661aa262f reported cstate change: term changed from 1 to 2, leader changed from b3afae27e8744ee6921cd67f805e3e4a (127.0.107.194) to d0e95822debe474ba6a700c661aa262f (127.0.107.195). New cstate: current_term: 2 leader_uuid: "d0e95822debe474ba6a700c661aa262f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } health_report { overall_health: HEALTHY } } }
I20260709 12:58:21.011971  1291 raft_consensus.cc:697] T c6480f4fe1b54658b39e73baa7c719fd P b3afae27e8744ee6921cd67f805e3e4a [term 2 LEADER]: Becoming Leader. State: Replica: b3afae27e8744ee6921cd67f805e3e4a, State: Running, Role: LEADER
I20260709 12:58:21.012447  1245 raft_consensus.cc:2468] T c6480f4fe1b54658b39e73baa7c719fd P d0e95822debe474ba6a700c661aa262f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3afae27e8744ee6921cd67f805e3e4a in term 2.
I20260709 12:58:21.010257  1171 consensus_queue.cc:260] T 759e30de7d7b4bc6980dbf8c4ba8031f P b3afae27e8744ee6921cd67f805e3e4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:21.013013  1291 consensus_queue.cc:237] T c6480f4fe1b54658b39e73baa7c719fd P b3afae27e8744ee6921cd67f805e3e4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:21.013849  1171 raft_consensus.cc:3060] T 759e30de7d7b4bc6980dbf8c4ba8031f P b3afae27e8744ee6921cd67f805e3e4a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:58:21.020949  1171 raft_consensus.cc:2468] T 759e30de7d7b4bc6980dbf8c4ba8031f P b3afae27e8744ee6921cd67f805e3e4a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d88b316b7ecd49f29219068c5fe51a26 in term 2.
I20260709 12:58:21.023658  1315 raft_consensus.cc:993] T 6198a4398a3140e8ad9f1a158956fe3e P d0e95822debe474ba6a700c661aa262f: : Instructing follower d88b316b7ecd49f29219068c5fe51a26 to start an election
I20260709 12:58:21.024164  1315 raft_consensus.cc:1081] T 6198a4398a3140e8ad9f1a158956fe3e P d0e95822debe474ba6a700c661aa262f [term 1 LEADER]: Signalling peer d88b316b7ecd49f29219068c5fe51a26 to start an election
I20260709 12:58:21.022078   993 catalog_manager.cc:5697] T 759e30de7d7b4bc6980dbf8c4ba8031f P d88b316b7ecd49f29219068c5fe51a26 reported cstate change: term changed from 1 to 2, leader changed from b3afae27e8744ee6921cd67f805e3e4a (127.0.107.194) to d88b316b7ecd49f29219068c5fe51a26 (127.0.107.193). New cstate: current_term: 2 leader_uuid: "d88b316b7ecd49f29219068c5fe51a26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } health_report { overall_health: UNKNOWN } } }
I20260709 12:58:21.026105  1094 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "6198a4398a3140e8ad9f1a158956fe3e"
dest_uuid: "d88b316b7ecd49f29219068c5fe51a26"
 from {username='slave'} at 127.0.0.1:49986
I20260709 12:58:21.025842   992 catalog_manager.cc:5697] T c6480f4fe1b54658b39e73baa7c719fd P b3afae27e8744ee6921cd67f805e3e4a reported cstate change: term changed from 1 to 2, leader changed from d88b316b7ecd49f29219068c5fe51a26 (127.0.107.193) to b3afae27e8744ee6921cd67f805e3e4a (127.0.107.194). New cstate: current_term: 2 leader_uuid: "b3afae27e8744ee6921cd67f805e3e4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } health_report { overall_health: UNKNOWN } } }
I20260709 12:58:21.026773  1094 raft_consensus.cc:493] T 6198a4398a3140e8ad9f1a158956fe3e P d88b316b7ecd49f29219068c5fe51a26 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:58:21.027328  1094 raft_consensus.cc:3060] T 6198a4398a3140e8ad9f1a158956fe3e P d88b316b7ecd49f29219068c5fe51a26 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:58:21.034904  1094 raft_consensus.cc:515] T 6198a4398a3140e8ad9f1a158956fe3e P d88b316b7ecd49f29219068c5fe51a26 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:21.036798  1094 leader_election.cc:290] T 6198a4398a3140e8ad9f1a158956fe3e P d88b316b7ecd49f29219068c5fe51a26 [CANDIDATE]: Term 2 election: Requested vote from peers b3afae27e8744ee6921cd67f805e3e4a (127.0.107.194:36785), d0e95822debe474ba6a700c661aa262f (127.0.107.195:40863)
I20260709 12:58:21.037563  1171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6198a4398a3140e8ad9f1a158956fe3e" candidate_uuid: "d88b316b7ecd49f29219068c5fe51a26" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "b3afae27e8744ee6921cd67f805e3e4a"
I20260709 12:58:21.037876  1245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6198a4398a3140e8ad9f1a158956fe3e" candidate_uuid: "d88b316b7ecd49f29219068c5fe51a26" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d0e95822debe474ba6a700c661aa262f"
I20260709 12:58:21.038208  1171 raft_consensus.cc:3060] T 6198a4398a3140e8ad9f1a158956fe3e P b3afae27e8744ee6921cd67f805e3e4a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:58:21.038452  1245 raft_consensus.cc:3055] T 6198a4398a3140e8ad9f1a158956fe3e P d0e95822debe474ba6a700c661aa262f [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:58:21.038807  1245 raft_consensus.cc:740] T 6198a4398a3140e8ad9f1a158956fe3e P d0e95822debe474ba6a700c661aa262f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d0e95822debe474ba6a700c661aa262f, State: Running, Role: LEADER
I20260709 12:58:21.039662  1245 consensus_queue.cc:260] T 6198a4398a3140e8ad9f1a158956fe3e P d0e95822debe474ba6a700c661aa262f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:21.041056  1245 raft_consensus.cc:3060] T 6198a4398a3140e8ad9f1a158956fe3e P d0e95822debe474ba6a700c661aa262f [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:58:21.047132  1171 raft_consensus.cc:2468] T 6198a4398a3140e8ad9f1a158956fe3e P b3afae27e8744ee6921cd67f805e3e4a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d88b316b7ecd49f29219068c5fe51a26 in term 2.
I20260709 12:58:21.048720  1059 leader_election.cc:304] T 6198a4398a3140e8ad9f1a158956fe3e P d88b316b7ecd49f29219068c5fe51a26 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3afae27e8744ee6921cd67f805e3e4a, d88b316b7ecd49f29219068c5fe51a26; no voters: 
I20260709 12:58:21.049886  1293 raft_consensus.cc:2804] T 6198a4398a3140e8ad9f1a158956fe3e P d88b316b7ecd49f29219068c5fe51a26 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:58:21.050773  1293 raft_consensus.cc:697] T 6198a4398a3140e8ad9f1a158956fe3e P d88b316b7ecd49f29219068c5fe51a26 [term 2 LEADER]: Becoming Leader. State: Replica: d88b316b7ecd49f29219068c5fe51a26, State: Running, Role: LEADER
I20260709 12:58:21.051055  1245 raft_consensus.cc:2468] T 6198a4398a3140e8ad9f1a158956fe3e P d0e95822debe474ba6a700c661aa262f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d88b316b7ecd49f29219068c5fe51a26 in term 2.
I20260709 12:58:21.052332  1293 consensus_queue.cc:237] T 6198a4398a3140e8ad9f1a158956fe3e P d88b316b7ecd49f29219068c5fe51a26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:21.062633   992 catalog_manager.cc:5697] T 6198a4398a3140e8ad9f1a158956fe3e P d88b316b7ecd49f29219068c5fe51a26 reported cstate change: term changed from 1 to 2, leader changed from d0e95822debe474ba6a700c661aa262f (127.0.107.195) to d88b316b7ecd49f29219068c5fe51a26 (127.0.107.193). New cstate: current_term: 2 leader_uuid: "d88b316b7ecd49f29219068c5fe51a26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } health_report { overall_health: UNKNOWN } } }
I20260709 12:58:21.100467  1292 raft_consensus.cc:993] T d87ed5c1a25e40f8b7aa82d33ca30bf1 P d0e95822debe474ba6a700c661aa262f: : Instructing follower b3afae27e8744ee6921cd67f805e3e4a to start an election
I20260709 12:58:21.101938  1313 raft_consensus.cc:1081] T d87ed5c1a25e40f8b7aa82d33ca30bf1 P d0e95822debe474ba6a700c661aa262f [term 1 LEADER]: Signalling peer b3afae27e8744ee6921cd67f805e3e4a to start an election
I20260709 12:58:21.104903  1171 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "d87ed5c1a25e40f8b7aa82d33ca30bf1"
dest_uuid: "b3afae27e8744ee6921cd67f805e3e4a"
 from {username='slave'} at 127.0.0.1:34386
I20260709 12:58:21.105654  1171 raft_consensus.cc:493] T d87ed5c1a25e40f8b7aa82d33ca30bf1 P b3afae27e8744ee6921cd67f805e3e4a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:58:21.106266  1171 raft_consensus.cc:3060] T d87ed5c1a25e40f8b7aa82d33ca30bf1 P b3afae27e8744ee6921cd67f805e3e4a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:58:21.115569  1171 raft_consensus.cc:515] T d87ed5c1a25e40f8b7aa82d33ca30bf1 P b3afae27e8744ee6921cd67f805e3e4a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:21.118465  1171 leader_election.cc:290] T d87ed5c1a25e40f8b7aa82d33ca30bf1 P b3afae27e8744ee6921cd67f805e3e4a [CANDIDATE]: Term 2 election: Requested vote from peers d88b316b7ecd49f29219068c5fe51a26 (127.0.107.193:38591), d0e95822debe474ba6a700c661aa262f (127.0.107.195:40863)
I20260709 12:58:21.120791  1094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d87ed5c1a25e40f8b7aa82d33ca30bf1" candidate_uuid: "b3afae27e8744ee6921cd67f805e3e4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d88b316b7ecd49f29219068c5fe51a26"
I20260709 12:58:21.120816  1245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d87ed5c1a25e40f8b7aa82d33ca30bf1" candidate_uuid: "b3afae27e8744ee6921cd67f805e3e4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d0e95822debe474ba6a700c661aa262f"
I20260709 12:58:21.121925  1094 raft_consensus.cc:3060] T d87ed5c1a25e40f8b7aa82d33ca30bf1 P d88b316b7ecd49f29219068c5fe51a26 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:58:21.122349  1245 raft_consensus.cc:3055] T d87ed5c1a25e40f8b7aa82d33ca30bf1 P d0e95822debe474ba6a700c661aa262f [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:58:21.123410  1245 raft_consensus.cc:740] T d87ed5c1a25e40f8b7aa82d33ca30bf1 P d0e95822debe474ba6a700c661aa262f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d0e95822debe474ba6a700c661aa262f, State: Running, Role: LEADER
I20260709 12:58:21.125373  1245 consensus_queue.cc:260] T d87ed5c1a25e40f8b7aa82d33ca30bf1 P d0e95822debe474ba6a700c661aa262f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:21.128296  1245 raft_consensus.cc:3060] T d87ed5c1a25e40f8b7aa82d33ca30bf1 P d0e95822debe474ba6a700c661aa262f [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:58:21.138671  1245 raft_consensus.cc:2468] T d87ed5c1a25e40f8b7aa82d33ca30bf1 P d0e95822debe474ba6a700c661aa262f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3afae27e8744ee6921cd67f805e3e4a in term 2.
I20260709 12:58:21.139035  1094 raft_consensus.cc:2468] T d87ed5c1a25e40f8b7aa82d33ca30bf1 P d88b316b7ecd49f29219068c5fe51a26 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3afae27e8744ee6921cd67f805e3e4a in term 2.
I20260709 12:58:21.140106  1095 raft_consensus.cc:1240] T d87ed5c1a25e40f8b7aa82d33ca30bf1 P d88b316b7ecd49f29219068c5fe51a26 [term 2 FOLLOWER]: Rejecting Update request from peer d0e95822debe474ba6a700c661aa262f for earlier term 1. Current term is 2. Ops: []
I20260709 12:58:21.140184  1136 leader_election.cc:304] T d87ed5c1a25e40f8b7aa82d33ca30bf1 P b3afae27e8744ee6921cd67f805e3e4a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3afae27e8744ee6921cd67f805e3e4a, d0e95822debe474ba6a700c661aa262f; no voters: 
I20260709 12:58:21.141507  1291 raft_consensus.cc:2804] T d87ed5c1a25e40f8b7aa82d33ca30bf1 P b3afae27e8744ee6921cd67f805e3e4a [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:58:21.142373  1291 raft_consensus.cc:697] T d87ed5c1a25e40f8b7aa82d33ca30bf1 P b3afae27e8744ee6921cd67f805e3e4a [term 2 LEADER]: Becoming Leader. State: Replica: b3afae27e8744ee6921cd67f805e3e4a, State: Running, Role: LEADER
I20260709 12:58:21.143828  1291 consensus_queue.cc:237] T d87ed5c1a25e40f8b7aa82d33ca30bf1 P b3afae27e8744ee6921cd67f805e3e4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:21.156924   992 catalog_manager.cc:5697] T d87ed5c1a25e40f8b7aa82d33ca30bf1 P b3afae27e8744ee6921cd67f805e3e4a reported cstate change: term changed from 1 to 2, leader changed from d0e95822debe474ba6a700c661aa262f (127.0.107.195) to b3afae27e8744ee6921cd67f805e3e4a (127.0.107.194). New cstate: current_term: 2 leader_uuid: "b3afae27e8744ee6921cd67f805e3e4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } health_report { overall_health: UNKNOWN } } }
I20260709 12:58:21.233372  1326 raft_consensus.cc:993] T 2cde8b0b8c5c46e78f405744f084c7c6 P d88b316b7ecd49f29219068c5fe51a26: : Instructing follower b3afae27e8744ee6921cd67f805e3e4a to start an election
I20260709 12:58:21.234205  1293 raft_consensus.cc:1081] T 2cde8b0b8c5c46e78f405744f084c7c6 P d88b316b7ecd49f29219068c5fe51a26 [term 1 LEADER]: Signalling peer b3afae27e8744ee6921cd67f805e3e4a to start an election
I20260709 12:58:21.236752  1171 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "2cde8b0b8c5c46e78f405744f084c7c6"
dest_uuid: "b3afae27e8744ee6921cd67f805e3e4a"
 from {username='slave'} at 127.0.0.1:34394
I20260709 12:58:21.237726  1171 raft_consensus.cc:493] T 2cde8b0b8c5c46e78f405744f084c7c6 P b3afae27e8744ee6921cd67f805e3e4a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:58:21.238327  1171 raft_consensus.cc:3060] T 2cde8b0b8c5c46e78f405744f084c7c6 P b3afae27e8744ee6921cd67f805e3e4a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:58:21.248939  1171 raft_consensus.cc:515] T 2cde8b0b8c5c46e78f405744f084c7c6 P b3afae27e8744ee6921cd67f805e3e4a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:21.252594  1171 leader_election.cc:290] T 2cde8b0b8c5c46e78f405744f084c7c6 P b3afae27e8744ee6921cd67f805e3e4a [CANDIDATE]: Term 2 election: Requested vote from peers d88b316b7ecd49f29219068c5fe51a26 (127.0.107.193:38591), d0e95822debe474ba6a700c661aa262f (127.0.107.195:40863)
I20260709 12:58:21.254209  1095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cde8b0b8c5c46e78f405744f084c7c6" candidate_uuid: "b3afae27e8744ee6921cd67f805e3e4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d88b316b7ecd49f29219068c5fe51a26"
I20260709 12:58:21.255285  1095 raft_consensus.cc:3055] T 2cde8b0b8c5c46e78f405744f084c7c6 P d88b316b7ecd49f29219068c5fe51a26 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:58:21.255796  1095 raft_consensus.cc:740] T 2cde8b0b8c5c46e78f405744f084c7c6 P d88b316b7ecd49f29219068c5fe51a26 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d88b316b7ecd49f29219068c5fe51a26, State: Running, Role: LEADER
I20260709 12:58:21.255978  1245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cde8b0b8c5c46e78f405744f084c7c6" candidate_uuid: "b3afae27e8744ee6921cd67f805e3e4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d0e95822debe474ba6a700c661aa262f"
I20260709 12:58:21.256883  1245 raft_consensus.cc:3060] T 2cde8b0b8c5c46e78f405744f084c7c6 P d0e95822debe474ba6a700c661aa262f [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:58:21.257205  1095 consensus_queue.cc:260] T 2cde8b0b8c5c46e78f405744f084c7c6 P d88b316b7ecd49f29219068c5fe51a26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:21.260378  1095 raft_consensus.cc:3060] T 2cde8b0b8c5c46e78f405744f084c7c6 P d88b316b7ecd49f29219068c5fe51a26 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:58:21.272783  1245 raft_consensus.cc:2468] T 2cde8b0b8c5c46e78f405744f084c7c6 P d0e95822debe474ba6a700c661aa262f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3afae27e8744ee6921cd67f805e3e4a in term 2.
I20260709 12:58:21.275108  1095 raft_consensus.cc:2468] T 2cde8b0b8c5c46e78f405744f084c7c6 P d88b316b7ecd49f29219068c5fe51a26 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3afae27e8744ee6921cd67f805e3e4a in term 2.
I20260709 12:58:21.275875  1136 leader_election.cc:304] T 2cde8b0b8c5c46e78f405744f084c7c6 P b3afae27e8744ee6921cd67f805e3e4a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3afae27e8744ee6921cd67f805e3e4a, d0e95822debe474ba6a700c661aa262f; no voters: 
I20260709 12:58:21.284336  1291 raft_consensus.cc:2804] T 2cde8b0b8c5c46e78f405744f084c7c6 P b3afae27e8744ee6921cd67f805e3e4a [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:58:21.285422  1291 raft_consensus.cc:697] T 2cde8b0b8c5c46e78f405744f084c7c6 P b3afae27e8744ee6921cd67f805e3e4a [term 2 LEADER]: Becoming Leader. State: Replica: b3afae27e8744ee6921cd67f805e3e4a, State: Running, Role: LEADER
I20260709 12:58:21.288410  1291 consensus_queue.cc:237] T 2cde8b0b8c5c46e78f405744f084c7c6 P b3afae27e8744ee6921cd67f805e3e4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:21.290714  1095 raft_consensus.cc:1275] T 2685bf6c4fd043a481d1d05136dbbd5e P d88b316b7ecd49f29219068c5fe51a26 [term 2 FOLLOWER]: Refusing update from remote peer b3afae27e8744ee6921cd67f805e3e4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:58:21.294602  1332 consensus_queue.cc:1048] T 2685bf6c4fd043a481d1d05136dbbd5e P b3afae27e8744ee6921cd67f805e3e4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:58:21.313567   992 catalog_manager.cc:5697] T 2cde8b0b8c5c46e78f405744f084c7c6 P b3afae27e8744ee6921cd67f805e3e4a reported cstate change: term changed from 1 to 2, leader changed from d88b316b7ecd49f29219068c5fe51a26 (127.0.107.193) to b3afae27e8744ee6921cd67f805e3e4a (127.0.107.194). New cstate: current_term: 2 leader_uuid: "b3afae27e8744ee6921cd67f805e3e4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } health_report { overall_health: UNKNOWN } } }
I20260709 12:58:21.315927  1245 raft_consensus.cc:1275] T 2685bf6c4fd043a481d1d05136dbbd5e P d0e95822debe474ba6a700c661aa262f [term 2 FOLLOWER]: Refusing update from remote peer b3afae27e8744ee6921cd67f805e3e4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:58:21.321694  1291 consensus_queue.cc:1048] T 2685bf6c4fd043a481d1d05136dbbd5e P b3afae27e8744ee6921cd67f805e3e4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:58:21.426488  1245 raft_consensus.cc:1275] T 759e30de7d7b4bc6980dbf8c4ba8031f P d0e95822debe474ba6a700c661aa262f [term 2 FOLLOWER]: Refusing update from remote peer d88b316b7ecd49f29219068c5fe51a26: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:58:21.429929  1293 consensus_queue.cc:1048] T 759e30de7d7b4bc6980dbf8c4ba8031f P d88b316b7ecd49f29219068c5fe51a26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:58:21.454658  1171 raft_consensus.cc:1275] T 759e30de7d7b4bc6980dbf8c4ba8031f P b3afae27e8744ee6921cd67f805e3e4a [term 2 FOLLOWER]: Refusing update from remote peer d88b316b7ecd49f29219068c5fe51a26: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:58:21.459687  1326 consensus_queue.cc:1048] T 759e30de7d7b4bc6980dbf8c4ba8031f P d88b316b7ecd49f29219068c5fe51a26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:58:21.483659  1168 raft_consensus.cc:1275] T df94d2062ca84496bc46321e6e4012fc P b3afae27e8744ee6921cd67f805e3e4a [term 2 FOLLOWER]: Refusing update from remote peer d0e95822debe474ba6a700c661aa262f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:58:21.488149  1313 consensus_queue.cc:1048] T df94d2062ca84496bc46321e6e4012fc P d0e95822debe474ba6a700c661aa262f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:58:21.510051  1095 raft_consensus.cc:1275] T df94d2062ca84496bc46321e6e4012fc P d88b316b7ecd49f29219068c5fe51a26 [term 2 FOLLOWER]: Refusing update from remote peer d0e95822debe474ba6a700c661aa262f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:58:21.514014  1292 consensus_queue.cc:1048] T df94d2062ca84496bc46321e6e4012fc P d0e95822debe474ba6a700c661aa262f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:58:21.522104  1095 raft_consensus.cc:1275] T c6480f4fe1b54658b39e73baa7c719fd P d88b316b7ecd49f29219068c5fe51a26 [term 2 FOLLOWER]: Refusing update from remote peer b3afae27e8744ee6921cd67f805e3e4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:58:21.527395  1291 consensus_queue.cc:1048] T c6480f4fe1b54658b39e73baa7c719fd P b3afae27e8744ee6921cd67f805e3e4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:58:21.536245  1171 raft_consensus.cc:1275] T 6198a4398a3140e8ad9f1a158956fe3e P b3afae27e8744ee6921cd67f805e3e4a [term 2 FOLLOWER]: Refusing update from remote peer d88b316b7ecd49f29219068c5fe51a26: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:58:21.544754  1293 consensus_queue.cc:1048] T 6198a4398a3140e8ad9f1a158956fe3e P d88b316b7ecd49f29219068c5fe51a26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:58:21.567032  1245 raft_consensus.cc:1275] T c6480f4fe1b54658b39e73baa7c719fd P d0e95822debe474ba6a700c661aa262f [term 2 FOLLOWER]: Refusing update from remote peer b3afae27e8744ee6921cd67f805e3e4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:58:21.571950  1332 consensus_queue.cc:1048] T c6480f4fe1b54658b39e73baa7c719fd P b3afae27e8744ee6921cd67f805e3e4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:58:21.608377  1093 raft_consensus.cc:1275] T d87ed5c1a25e40f8b7aa82d33ca30bf1 P d88b316b7ecd49f29219068c5fe51a26 [term 2 FOLLOWER]: Refusing update from remote peer b3afae27e8744ee6921cd67f805e3e4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:58:21.608747  1245 raft_consensus.cc:1275] T 6198a4398a3140e8ad9f1a158956fe3e P d0e95822debe474ba6a700c661aa262f [term 2 FOLLOWER]: Refusing update from remote peer d88b316b7ecd49f29219068c5fe51a26: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:58:21.614513  1314 consensus_queue.cc:1048] T 6198a4398a3140e8ad9f1a158956fe3e P d88b316b7ecd49f29219068c5fe51a26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
I20260709 12:58:21.619585  1332 consensus_queue.cc:1048] T d87ed5c1a25e40f8b7aa82d33ca30bf1 P b3afae27e8744ee6921cd67f805e3e4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
I20260709 12:58:21.672282  1244 raft_consensus.cc:1275] T d87ed5c1a25e40f8b7aa82d33ca30bf1 P d0e95822debe474ba6a700c661aa262f [term 2 FOLLOWER]: Refusing update from remote peer b3afae27e8744ee6921cd67f805e3e4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:58:21.698556  1341 consensus_queue.cc:1048] T d87ed5c1a25e40f8b7aa82d33ca30bf1 P b3afae27e8744ee6921cd67f805e3e4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
I20260709 12:58:21.747778  1244 raft_consensus.cc:1275] T 2cde8b0b8c5c46e78f405744f084c7c6 P d0e95822debe474ba6a700c661aa262f [term 2 FOLLOWER]: Refusing update from remote peer b3afae27e8744ee6921cd67f805e3e4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:58:21.753474  1341 consensus_queue.cc:1048] T 2cde8b0b8c5c46e78f405744f084c7c6 P b3afae27e8744ee6921cd67f805e3e4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
I20260709 12:58:21.781327  1095 raft_consensus.cc:1275] T 2cde8b0b8c5c46e78f405744f084c7c6 P d88b316b7ecd49f29219068c5fe51a26 [term 2 FOLLOWER]: Refusing update from remote peer b3afae27e8744ee6921cd67f805e3e4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:58:21.805421  1341 consensus_queue.cc:1048] T 2cde8b0b8c5c46e78f405744f084c7c6 P b3afae27e8744ee6921cd67f805e3e4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.014s
I20260709 12:58:23.783087   431 auto_leader_rebalancer-test.cc:348] The leader distribution is 
I20260709 12:58:23.783862   431 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:58:23.804432  1093 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "1935c9164755416a965cccf80900d941"
dest_uuid: "d88b316b7ecd49f29219068c5fe51a26"
mode: GRACEFUL
new_leader_uuid: "d0e95822debe474ba6a700c661aa262f"
 from {username='slave'} at 127.0.0.1:35586
I20260709 12:58:23.805022  1093 raft_consensus.cc:606] T 1935c9164755416a965cccf80900d941 P d88b316b7ecd49f29219068c5fe51a26 [term 1 LEADER]: Received request to transfer leadership to TS d0e95822debe474ba6a700c661aa262f
I20260709 12:58:23.807535  1093 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "73e09365f8cd4513a4600030950af811"
dest_uuid: "d88b316b7ecd49f29219068c5fe51a26"
mode: GRACEFUL
new_leader_uuid: "d0e95822debe474ba6a700c661aa262f"
 from {username='slave'} at 127.0.0.1:35586
I20260709 12:58:23.808033  1093 raft_consensus.cc:606] T 73e09365f8cd4513a4600030950af811 P d88b316b7ecd49f29219068c5fe51a26 [term 1 LEADER]: Received request to transfer leadership to TS d0e95822debe474ba6a700c661aa262f
I20260709 12:58:23.809048   431 auto_leader_rebalancer.cc:452] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260709 12:58:23.809545   431 auto_leader_rebalancer.cc:527] All tables' leader rebalancing finished this round
I20260709 12:58:23.891681  1293 raft_consensus.cc:993] T 73e09365f8cd4513a4600030950af811 P d88b316b7ecd49f29219068c5fe51a26: : Instructing follower d0e95822debe474ba6a700c661aa262f to start an election
I20260709 12:58:23.892207  1386 raft_consensus.cc:1081] T 73e09365f8cd4513a4600030950af811 P d88b316b7ecd49f29219068c5fe51a26 [term 1 LEADER]: Signalling peer d0e95822debe474ba6a700c661aa262f to start an election
I20260709 12:58:23.893816  1243 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "73e09365f8cd4513a4600030950af811"
dest_uuid: "d0e95822debe474ba6a700c661aa262f"
 from {username='slave'} at 127.0.0.1:33256
I20260709 12:58:23.894284  1243 raft_consensus.cc:493] T 73e09365f8cd4513a4600030950af811 P d0e95822debe474ba6a700c661aa262f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:58:23.894630  1243 raft_consensus.cc:3060] T 73e09365f8cd4513a4600030950af811 P d0e95822debe474ba6a700c661aa262f [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:58:23.935273  1243 raft_consensus.cc:515] T 73e09365f8cd4513a4600030950af811 P d0e95822debe474ba6a700c661aa262f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:23.937587  1243 leader_election.cc:290] T 73e09365f8cd4513a4600030950af811 P d0e95822debe474ba6a700c661aa262f [CANDIDATE]: Term 2 election: Requested vote from peers d88b316b7ecd49f29219068c5fe51a26 (127.0.107.193:38591), b3afae27e8744ee6921cd67f805e3e4a (127.0.107.194:36785)
I20260709 12:58:23.938402  1093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73e09365f8cd4513a4600030950af811" candidate_uuid: "d0e95822debe474ba6a700c661aa262f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d88b316b7ecd49f29219068c5fe51a26"
I20260709 12:58:23.938925  1168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73e09365f8cd4513a4600030950af811" candidate_uuid: "d0e95822debe474ba6a700c661aa262f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "b3afae27e8744ee6921cd67f805e3e4a"
I20260709 12:58:23.939208  1093 raft_consensus.cc:3055] T 73e09365f8cd4513a4600030950af811 P d88b316b7ecd49f29219068c5fe51a26 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:58:23.939608  1168 raft_consensus.cc:3060] T 73e09365f8cd4513a4600030950af811 P b3afae27e8744ee6921cd67f805e3e4a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:58:23.939616  1093 raft_consensus.cc:740] T 73e09365f8cd4513a4600030950af811 P d88b316b7ecd49f29219068c5fe51a26 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d88b316b7ecd49f29219068c5fe51a26, State: Running, Role: LEADER
I20260709 12:58:23.940799  1093 consensus_queue.cc:260] T 73e09365f8cd4513a4600030950af811 P d88b316b7ecd49f29219068c5fe51a26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:23.941833  1093 raft_consensus.cc:3060] T 73e09365f8cd4513a4600030950af811 P d88b316b7ecd49f29219068c5fe51a26 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:58:24.037958  1168 raft_consensus.cc:2468] T 73e09365f8cd4513a4600030950af811 P b3afae27e8744ee6921cd67f805e3e4a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0e95822debe474ba6a700c661aa262f in term 2.
I20260709 12:58:24.037958  1093 raft_consensus.cc:2468] T 73e09365f8cd4513a4600030950af811 P d88b316b7ecd49f29219068c5fe51a26 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0e95822debe474ba6a700c661aa262f in term 2.
I20260709 12:58:24.039512  1209 leader_election.cc:304] T 73e09365f8cd4513a4600030950af811 P d0e95822debe474ba6a700c661aa262f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3afae27e8744ee6921cd67f805e3e4a, d0e95822debe474ba6a700c661aa262f; no voters: 
I20260709 12:58:24.040558  1315 raft_consensus.cc:2804] T 73e09365f8cd4513a4600030950af811 P d0e95822debe474ba6a700c661aa262f [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:58:24.041672  1315 raft_consensus.cc:697] T 73e09365f8cd4513a4600030950af811 P d0e95822debe474ba6a700c661aa262f [term 2 LEADER]: Becoming Leader. State: Replica: d0e95822debe474ba6a700c661aa262f, State: Running, Role: LEADER
I20260709 12:58:24.042724  1315 consensus_queue.cc:237] T 73e09365f8cd4513a4600030950af811 P d0e95822debe474ba6a700c661aa262f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:24.053934   990 catalog_manager.cc:5697] T 73e09365f8cd4513a4600030950af811 P d0e95822debe474ba6a700c661aa262f reported cstate change: term changed from 1 to 2, leader changed from d88b316b7ecd49f29219068c5fe51a26 (127.0.107.193) to d0e95822debe474ba6a700c661aa262f (127.0.107.195). New cstate: current_term: 2 leader_uuid: "d0e95822debe474ba6a700c661aa262f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } health_report { overall_health: HEALTHY } } }
I20260709 12:58:24.133371  1293 raft_consensus.cc:993] T 1935c9164755416a965cccf80900d941 P d88b316b7ecd49f29219068c5fe51a26: : Instructing follower d0e95822debe474ba6a700c661aa262f to start an election
I20260709 12:58:24.133889  1386 raft_consensus.cc:1081] T 1935c9164755416a965cccf80900d941 P d88b316b7ecd49f29219068c5fe51a26 [term 1 LEADER]: Signalling peer d0e95822debe474ba6a700c661aa262f to start an election
I20260709 12:58:24.136255  1243 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "1935c9164755416a965cccf80900d941"
dest_uuid: "d0e95822debe474ba6a700c661aa262f"
 from {username='slave'} at 127.0.0.1:33256
I20260709 12:58:24.136847  1243 raft_consensus.cc:493] T 1935c9164755416a965cccf80900d941 P d0e95822debe474ba6a700c661aa262f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:58:24.137183  1243 raft_consensus.cc:3060] T 1935c9164755416a965cccf80900d941 P d0e95822debe474ba6a700c661aa262f [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:58:24.142777  1243 raft_consensus.cc:515] T 1935c9164755416a965cccf80900d941 P d0e95822debe474ba6a700c661aa262f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:24.144614  1243 leader_election.cc:290] T 1935c9164755416a965cccf80900d941 P d0e95822debe474ba6a700c661aa262f [CANDIDATE]: Term 2 election: Requested vote from peers d88b316b7ecd49f29219068c5fe51a26 (127.0.107.193:38591), b3afae27e8744ee6921cd67f805e3e4a (127.0.107.194:36785)
I20260709 12:58:24.145427  1093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1935c9164755416a965cccf80900d941" candidate_uuid: "d0e95822debe474ba6a700c661aa262f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d88b316b7ecd49f29219068c5fe51a26"
I20260709 12:58:24.145820  1168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1935c9164755416a965cccf80900d941" candidate_uuid: "d0e95822debe474ba6a700c661aa262f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "b3afae27e8744ee6921cd67f805e3e4a"
I20260709 12:58:24.146035  1093 raft_consensus.cc:3055] T 1935c9164755416a965cccf80900d941 P d88b316b7ecd49f29219068c5fe51a26 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:58:24.146413  1093 raft_consensus.cc:740] T 1935c9164755416a965cccf80900d941 P d88b316b7ecd49f29219068c5fe51a26 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d88b316b7ecd49f29219068c5fe51a26, State: Running, Role: LEADER
I20260709 12:58:24.146484  1168 raft_consensus.cc:3060] T 1935c9164755416a965cccf80900d941 P b3afae27e8744ee6921cd67f805e3e4a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:58:24.147228  1093 consensus_queue.cc:260] T 1935c9164755416a965cccf80900d941 P d88b316b7ecd49f29219068c5fe51a26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:24.148206  1093 raft_consensus.cc:3060] T 1935c9164755416a965cccf80900d941 P d88b316b7ecd49f29219068c5fe51a26 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:58:24.241750  1168 raft_consensus.cc:2468] T 1935c9164755416a965cccf80900d941 P b3afae27e8744ee6921cd67f805e3e4a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0e95822debe474ba6a700c661aa262f in term 2.
I20260709 12:58:24.241755  1093 raft_consensus.cc:2468] T 1935c9164755416a965cccf80900d941 P d88b316b7ecd49f29219068c5fe51a26 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0e95822debe474ba6a700c661aa262f in term 2.
I20260709 12:58:24.243412  1209 leader_election.cc:304] T 1935c9164755416a965cccf80900d941 P d0e95822debe474ba6a700c661aa262f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3afae27e8744ee6921cd67f805e3e4a, d0e95822debe474ba6a700c661aa262f; no voters: 
I20260709 12:58:24.244364  1315 raft_consensus.cc:2804] T 1935c9164755416a965cccf80900d941 P d0e95822debe474ba6a700c661aa262f [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:58:24.244845  1315 raft_consensus.cc:697] T 1935c9164755416a965cccf80900d941 P d0e95822debe474ba6a700c661aa262f [term 2 LEADER]: Becoming Leader. State: Replica: d0e95822debe474ba6a700c661aa262f, State: Running, Role: LEADER
I20260709 12:58:24.245720  1315 consensus_queue.cc:237] T 1935c9164755416a965cccf80900d941 P d0e95822debe474ba6a700c661aa262f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:24.254987   992 catalog_manager.cc:5697] T 1935c9164755416a965cccf80900d941 P d0e95822debe474ba6a700c661aa262f reported cstate change: term changed from 1 to 2, leader changed from d88b316b7ecd49f29219068c5fe51a26 (127.0.107.193) to d0e95822debe474ba6a700c661aa262f (127.0.107.195). New cstate: current_term: 2 leader_uuid: "d0e95822debe474ba6a700c661aa262f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } health_report { overall_health: HEALTHY } } }
I20260709 12:58:24.574923  1093 raft_consensus.cc:1275] T 73e09365f8cd4513a4600030950af811 P d88b316b7ecd49f29219068c5fe51a26 [term 2 FOLLOWER]: Refusing update from remote peer d0e95822debe474ba6a700c661aa262f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:58:24.576620  1315 consensus_queue.cc:1048] T 73e09365f8cd4513a4600030950af811 P d0e95822debe474ba6a700c661aa262f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:58:24.590027  1168 raft_consensus.cc:1275] T 73e09365f8cd4513a4600030950af811 P b3afae27e8744ee6921cd67f805e3e4a [term 2 FOLLOWER]: Refusing update from remote peer d0e95822debe474ba6a700c661aa262f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:58:24.592741  1387 consensus_queue.cc:1048] T 73e09365f8cd4513a4600030950af811 P d0e95822debe474ba6a700c661aa262f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:58:24.708860  1168 raft_consensus.cc:1275] T 1935c9164755416a965cccf80900d941 P b3afae27e8744ee6921cd67f805e3e4a [term 2 FOLLOWER]: Refusing update from remote peer d0e95822debe474ba6a700c661aa262f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:58:24.710511  1396 consensus_queue.cc:1048] T 1935c9164755416a965cccf80900d941 P d0e95822debe474ba6a700c661aa262f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:58:24.722942  1093 raft_consensus.cc:1275] T 1935c9164755416a965cccf80900d941 P d88b316b7ecd49f29219068c5fe51a26 [term 2 FOLLOWER]: Refusing update from remote peer d0e95822debe474ba6a700c661aa262f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:58:24.725556  1396 consensus_queue.cc:1048] T 1935c9164755416a965cccf80900d941 P d0e95822debe474ba6a700c661aa262f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:58:24.810791   431 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:58:24.827896  1243 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "73e09365f8cd4513a4600030950af811"
dest_uuid: "d0e95822debe474ba6a700c661aa262f"
mode: GRACEFUL
new_leader_uuid: "d88b316b7ecd49f29219068c5fe51a26"
 from {username='slave'} at 127.0.0.1:33220
I20260709 12:58:24.828711  1243 raft_consensus.cc:606] T 73e09365f8cd4513a4600030950af811 P d0e95822debe474ba6a700c661aa262f [term 2 LEADER]: Received request to transfer leadership to TS d88b316b7ecd49f29219068c5fe51a26
I20260709 12:58:24.834519  1243 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "1935c9164755416a965cccf80900d941"
dest_uuid: "d0e95822debe474ba6a700c661aa262f"
mode: GRACEFUL
new_leader_uuid: "d88b316b7ecd49f29219068c5fe51a26"
 from {username='slave'} at 127.0.0.1:33220
I20260709 12:58:24.835239  1243 raft_consensus.cc:606] T 1935c9164755416a965cccf80900d941 P d0e95822debe474ba6a700c661aa262f [term 2 LEADER]: Received request to transfer leadership to TS d88b316b7ecd49f29219068c5fe51a26
I20260709 12:58:24.844333  1168 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "2cde8b0b8c5c46e78f405744f084c7c6"
dest_uuid: "b3afae27e8744ee6921cd67f805e3e4a"
mode: GRACEFUL
new_leader_uuid: "d88b316b7ecd49f29219068c5fe51a26"
 from {username='slave'} at 127.0.0.1:42552
I20260709 12:58:24.844986  1168 raft_consensus.cc:606] T 2cde8b0b8c5c46e78f405744f084c7c6 P b3afae27e8744ee6921cd67f805e3e4a [term 2 LEADER]: Received request to transfer leadership to TS d88b316b7ecd49f29219068c5fe51a26
I20260709 12:58:24.850725  1168 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "2685bf6c4fd043a481d1d05136dbbd5e"
dest_uuid: "b3afae27e8744ee6921cd67f805e3e4a"
mode: GRACEFUL
new_leader_uuid: "d88b316b7ecd49f29219068c5fe51a26"
 from {username='slave'} at 127.0.0.1:42552
I20260709 12:58:24.851303  1168 raft_consensus.cc:606] T 2685bf6c4fd043a481d1d05136dbbd5e P b3afae27e8744ee6921cd67f805e3e4a [term 2 LEADER]: Received request to transfer leadership to TS d88b316b7ecd49f29219068c5fe51a26
I20260709 12:58:24.856431  1168 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "d87ed5c1a25e40f8b7aa82d33ca30bf1"
dest_uuid: "b3afae27e8744ee6921cd67f805e3e4a"
mode: GRACEFUL
new_leader_uuid: "d88b316b7ecd49f29219068c5fe51a26"
 from {username='slave'} at 127.0.0.1:42552
I20260709 12:58:24.856978  1168 raft_consensus.cc:606] T d87ed5c1a25e40f8b7aa82d33ca30bf1 P b3afae27e8744ee6921cd67f805e3e4a [term 2 LEADER]: Received request to transfer leadership to TS d88b316b7ecd49f29219068c5fe51a26
I20260709 12:58:25.055276  1404 raft_consensus.cc:993] T 2cde8b0b8c5c46e78f405744f084c7c6 P b3afae27e8744ee6921cd67f805e3e4a: : Instructing follower d88b316b7ecd49f29219068c5fe51a26 to start an election
I20260709 12:58:25.055961  1291 raft_consensus.cc:1081] T 2cde8b0b8c5c46e78f405744f084c7c6 P b3afae27e8744ee6921cd67f805e3e4a [term 2 LEADER]: Signalling peer d88b316b7ecd49f29219068c5fe51a26 to start an election
I20260709 12:58:25.058101  1093 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "2cde8b0b8c5c46e78f405744f084c7c6"
dest_uuid: "d88b316b7ecd49f29219068c5fe51a26"
 from {username='slave'} at 127.0.0.1:49984
I20260709 12:58:25.058853  1093 raft_consensus.cc:493] T 2cde8b0b8c5c46e78f405744f084c7c6 P d88b316b7ecd49f29219068c5fe51a26 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:58:25.059309  1093 raft_consensus.cc:3060] T 2cde8b0b8c5c46e78f405744f084c7c6 P d88b316b7ecd49f29219068c5fe51a26 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:58:25.066926  1093 raft_consensus.cc:515] T 2cde8b0b8c5c46e78f405744f084c7c6 P d88b316b7ecd49f29219068c5fe51a26 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:25.069461  1093 leader_election.cc:290] T 2cde8b0b8c5c46e78f405744f084c7c6 P d88b316b7ecd49f29219068c5fe51a26 [CANDIDATE]: Term 3 election: Requested vote from peers b3afae27e8744ee6921cd67f805e3e4a (127.0.107.194:36785), d0e95822debe474ba6a700c661aa262f (127.0.107.195:40863)
I20260709 12:58:25.070580  1168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cde8b0b8c5c46e78f405744f084c7c6" candidate_uuid: "d88b316b7ecd49f29219068c5fe51a26" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "b3afae27e8744ee6921cd67f805e3e4a"
I20260709 12:58:25.071020  1243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cde8b0b8c5c46e78f405744f084c7c6" candidate_uuid: "d88b316b7ecd49f29219068c5fe51a26" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "d0e95822debe474ba6a700c661aa262f"
I20260709 12:58:25.071306  1168 raft_consensus.cc:3055] T 2cde8b0b8c5c46e78f405744f084c7c6 P b3afae27e8744ee6921cd67f805e3e4a [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:58:25.071722  1168 raft_consensus.cc:740] T 2cde8b0b8c5c46e78f405744f084c7c6 P b3afae27e8744ee6921cd67f805e3e4a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: b3afae27e8744ee6921cd67f805e3e4a, State: Running, Role: LEADER
I20260709 12:58:25.071794  1243 raft_consensus.cc:3060] T 2cde8b0b8c5c46e78f405744f084c7c6 P d0e95822debe474ba6a700c661aa262f [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:58:25.072734  1168 consensus_queue.cc:260] T 2cde8b0b8c5c46e78f405744f084c7c6 P b3afae27e8744ee6921cd67f805e3e4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:25.074038  1168 raft_consensus.cc:3060] T 2cde8b0b8c5c46e78f405744f084c7c6 P b3afae27e8744ee6921cd67f805e3e4a [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:58:25.079531  1243 raft_consensus.cc:2468] T 2cde8b0b8c5c46e78f405744f084c7c6 P d0e95822debe474ba6a700c661aa262f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d88b316b7ecd49f29219068c5fe51a26 in term 3.
I20260709 12:58:25.081100  1061 leader_election.cc:304] T 2cde8b0b8c5c46e78f405744f084c7c6 P d88b316b7ecd49f29219068c5fe51a26 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d0e95822debe474ba6a700c661aa262f, d88b316b7ecd49f29219068c5fe51a26; no voters: 
I20260709 12:58:25.082050  1168 raft_consensus.cc:2468] T 2cde8b0b8c5c46e78f405744f084c7c6 P b3afae27e8744ee6921cd67f805e3e4a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d88b316b7ecd49f29219068c5fe51a26 in term 3.
I20260709 12:58:25.082090  1403 raft_consensus.cc:2804] T 2cde8b0b8c5c46e78f405744f084c7c6 P d88b316b7ecd49f29219068c5fe51a26 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:58:25.082862  1403 raft_consensus.cc:697] T 2cde8b0b8c5c46e78f405744f084c7c6 P d88b316b7ecd49f29219068c5fe51a26 [term 3 LEADER]: Becoming Leader. State: Replica: d88b316b7ecd49f29219068c5fe51a26, State: Running, Role: LEADER
I20260709 12:58:25.084031  1403 consensus_queue.cc:237] T 2cde8b0b8c5c46e78f405744f084c7c6 P d88b316b7ecd49f29219068c5fe51a26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:25.093185   990 catalog_manager.cc:5697] T 2cde8b0b8c5c46e78f405744f084c7c6 P d88b316b7ecd49f29219068c5fe51a26 reported cstate change: term changed from 2 to 3, leader changed from b3afae27e8744ee6921cd67f805e3e4a (127.0.107.194) to d88b316b7ecd49f29219068c5fe51a26 (127.0.107.193). New cstate: current_term: 3 leader_uuid: "d88b316b7ecd49f29219068c5fe51a26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } health_report { overall_health: UNKNOWN } } }
I20260709 12:58:25.100171  1396 raft_consensus.cc:993] T 73e09365f8cd4513a4600030950af811 P d0e95822debe474ba6a700c661aa262f: : Instructing follower d88b316b7ecd49f29219068c5fe51a26 to start an election
I20260709 12:58:25.100769  1315 raft_consensus.cc:1081] T 73e09365f8cd4513a4600030950af811 P d0e95822debe474ba6a700c661aa262f [term 2 LEADER]: Signalling peer d88b316b7ecd49f29219068c5fe51a26 to start an election
I20260709 12:58:25.102828  1093 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "73e09365f8cd4513a4600030950af811"
dest_uuid: "d88b316b7ecd49f29219068c5fe51a26"
 from {username='slave'} at 127.0.0.1:49986
I20260709 12:58:25.103427  1093 raft_consensus.cc:493] T 73e09365f8cd4513a4600030950af811 P d88b316b7ecd49f29219068c5fe51a26 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:58:25.103832  1093 raft_consensus.cc:3060] T 73e09365f8cd4513a4600030950af811 P d88b316b7ecd49f29219068c5fe51a26 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:58:25.112165  1093 raft_consensus.cc:515] T 73e09365f8cd4513a4600030950af811 P d88b316b7ecd49f29219068c5fe51a26 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:25.114737  1093 leader_election.cc:290] T 73e09365f8cd4513a4600030950af811 P d88b316b7ecd49f29219068c5fe51a26 [CANDIDATE]: Term 3 election: Requested vote from peers b3afae27e8744ee6921cd67f805e3e4a (127.0.107.194:36785), d0e95822debe474ba6a700c661aa262f (127.0.107.195:40863)
I20260709 12:58:25.115772  1168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73e09365f8cd4513a4600030950af811" candidate_uuid: "d88b316b7ecd49f29219068c5fe51a26" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "b3afae27e8744ee6921cd67f805e3e4a"
I20260709 12:58:25.116391  1243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73e09365f8cd4513a4600030950af811" candidate_uuid: "d88b316b7ecd49f29219068c5fe51a26" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "d0e95822debe474ba6a700c661aa262f"
I20260709 12:58:25.116756  1168 raft_consensus.cc:3060] T 73e09365f8cd4513a4600030950af811 P b3afae27e8744ee6921cd67f805e3e4a [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:58:25.117183  1243 raft_consensus.cc:3055] T 73e09365f8cd4513a4600030950af811 P d0e95822debe474ba6a700c661aa262f [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:58:25.117681  1243 raft_consensus.cc:740] T 73e09365f8cd4513a4600030950af811 P d0e95822debe474ba6a700c661aa262f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d0e95822debe474ba6a700c661aa262f, State: Running, Role: LEADER
I20260709 12:58:25.118948  1243 consensus_queue.cc:260] T 73e09365f8cd4513a4600030950af811 P d0e95822debe474ba6a700c661aa262f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:25.120720  1243 raft_consensus.cc:3060] T 73e09365f8cd4513a4600030950af811 P d0e95822debe474ba6a700c661aa262f [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:58:25.127089  1168 raft_consensus.cc:2468] T 73e09365f8cd4513a4600030950af811 P b3afae27e8744ee6921cd67f805e3e4a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d88b316b7ecd49f29219068c5fe51a26 in term 3.
I20260709 12:58:25.129034  1059 leader_election.cc:304] T 73e09365f8cd4513a4600030950af811 P d88b316b7ecd49f29219068c5fe51a26 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3afae27e8744ee6921cd67f805e3e4a, d88b316b7ecd49f29219068c5fe51a26; no voters: 
I20260709 12:58:25.129838  1404 raft_consensus.cc:993] T d87ed5c1a25e40f8b7aa82d33ca30bf1 P b3afae27e8744ee6921cd67f805e3e4a: : Instructing follower d88b316b7ecd49f29219068c5fe51a26 to start an election
I20260709 12:58:25.130209  1403 raft_consensus.cc:2804] T 73e09365f8cd4513a4600030950af811 P d88b316b7ecd49f29219068c5fe51a26 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:58:25.130465  1404 raft_consensus.cc:1081] T d87ed5c1a25e40f8b7aa82d33ca30bf1 P b3afae27e8744ee6921cd67f805e3e4a [term 2 LEADER]: Signalling peer d88b316b7ecd49f29219068c5fe51a26 to start an election
I20260709 12:58:25.130239  1243 raft_consensus.cc:2468] T 73e09365f8cd4513a4600030950af811 P d0e95822debe474ba6a700c661aa262f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d88b316b7ecd49f29219068c5fe51a26 in term 3.
I20260709 12:58:25.131179  1403 raft_consensus.cc:697] T 73e09365f8cd4513a4600030950af811 P d88b316b7ecd49f29219068c5fe51a26 [term 3 LEADER]: Becoming Leader. State: Replica: d88b316b7ecd49f29219068c5fe51a26, State: Running, Role: LEADER
I20260709 12:58:25.132607  1403 consensus_queue.cc:237] T 73e09365f8cd4513a4600030950af811 P d88b316b7ecd49f29219068c5fe51a26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:25.134500  1093 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "d87ed5c1a25e40f8b7aa82d33ca30bf1"
dest_uuid: "d88b316b7ecd49f29219068c5fe51a26"
 from {username='slave'} at 127.0.0.1:49984
I20260709 12:58:25.135113  1093 raft_consensus.cc:493] T d87ed5c1a25e40f8b7aa82d33ca30bf1 P d88b316b7ecd49f29219068c5fe51a26 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:58:25.135535  1093 raft_consensus.cc:3060] T d87ed5c1a25e40f8b7aa82d33ca30bf1 P d88b316b7ecd49f29219068c5fe51a26 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:58:25.143335   990 catalog_manager.cc:5697] T 73e09365f8cd4513a4600030950af811 P d88b316b7ecd49f29219068c5fe51a26 reported cstate change: term changed from 2 to 3, leader changed from d0e95822debe474ba6a700c661aa262f (127.0.107.195) to d88b316b7ecd49f29219068c5fe51a26 (127.0.107.193). New cstate: current_term: 3 leader_uuid: "d88b316b7ecd49f29219068c5fe51a26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } health_report { overall_health: UNKNOWN } } }
I20260709 12:58:25.145491  1093 raft_consensus.cc:515] T d87ed5c1a25e40f8b7aa82d33ca30bf1 P d88b316b7ecd49f29219068c5fe51a26 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:25.150399  1093 leader_election.cc:290] T d87ed5c1a25e40f8b7aa82d33ca30bf1 P d88b316b7ecd49f29219068c5fe51a26 [CANDIDATE]: Term 3 election: Requested vote from peers b3afae27e8744ee6921cd67f805e3e4a (127.0.107.194:36785), d0e95822debe474ba6a700c661aa262f (127.0.107.195:40863)
I20260709 12:58:25.150627  1243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d87ed5c1a25e40f8b7aa82d33ca30bf1" candidate_uuid: "d88b316b7ecd49f29219068c5fe51a26" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "d0e95822debe474ba6a700c661aa262f"
I20260709 12:58:25.151280  1243 raft_consensus.cc:3060] T d87ed5c1a25e40f8b7aa82d33ca30bf1 P d0e95822debe474ba6a700c661aa262f [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:58:25.152854  1168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d87ed5c1a25e40f8b7aa82d33ca30bf1" candidate_uuid: "d88b316b7ecd49f29219068c5fe51a26" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "b3afae27e8744ee6921cd67f805e3e4a"
I20260709 12:58:25.153692  1168 raft_consensus.cc:3055] T d87ed5c1a25e40f8b7aa82d33ca30bf1 P b3afae27e8744ee6921cd67f805e3e4a [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:58:25.154115  1168 raft_consensus.cc:740] T d87ed5c1a25e40f8b7aa82d33ca30bf1 P b3afae27e8744ee6921cd67f805e3e4a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: b3afae27e8744ee6921cd67f805e3e4a, State: Running, Role: LEADER
I20260709 12:58:25.155278  1168 consensus_queue.cc:260] T d87ed5c1a25e40f8b7aa82d33ca30bf1 P b3afae27e8744ee6921cd67f805e3e4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:25.156909  1168 raft_consensus.cc:3060] T d87ed5c1a25e40f8b7aa82d33ca30bf1 P b3afae27e8744ee6921cd67f805e3e4a [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:58:25.160763  1243 raft_consensus.cc:2468] T d87ed5c1a25e40f8b7aa82d33ca30bf1 P d0e95822debe474ba6a700c661aa262f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d88b316b7ecd49f29219068c5fe51a26 in term 3.
I20260709 12:58:25.162122  1061 leader_election.cc:304] T d87ed5c1a25e40f8b7aa82d33ca30bf1 P d88b316b7ecd49f29219068c5fe51a26 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d0e95822debe474ba6a700c661aa262f, d88b316b7ecd49f29219068c5fe51a26; no voters: 
I20260709 12:58:25.163460  1403 raft_consensus.cc:2804] T d87ed5c1a25e40f8b7aa82d33ca30bf1 P d88b316b7ecd49f29219068c5fe51a26 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:58:25.163957  1403 raft_consensus.cc:697] T d87ed5c1a25e40f8b7aa82d33ca30bf1 P d88b316b7ecd49f29219068c5fe51a26 [term 3 LEADER]: Becoming Leader. State: Replica: d88b316b7ecd49f29219068c5fe51a26, State: Running, Role: LEADER
I20260709 12:58:25.165030  1403 consensus_queue.cc:237] T d87ed5c1a25e40f8b7aa82d33ca30bf1 P d88b316b7ecd49f29219068c5fe51a26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:25.166194  1168 raft_consensus.cc:2468] T d87ed5c1a25e40f8b7aa82d33ca30bf1 P b3afae27e8744ee6921cd67f805e3e4a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d88b316b7ecd49f29219068c5fe51a26 in term 3.
I20260709 12:58:25.173949   990 catalog_manager.cc:5697] T d87ed5c1a25e40f8b7aa82d33ca30bf1 P d88b316b7ecd49f29219068c5fe51a26 reported cstate change: term changed from 2 to 3, leader changed from b3afae27e8744ee6921cd67f805e3e4a (127.0.107.194) to d88b316b7ecd49f29219068c5fe51a26 (127.0.107.193). New cstate: current_term: 3 leader_uuid: "d88b316b7ecd49f29219068c5fe51a26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } health_report { overall_health: UNKNOWN } } }
I20260709 12:58:25.195765  1291 raft_consensus.cc:993] T 2685bf6c4fd043a481d1d05136dbbd5e P b3afae27e8744ee6921cd67f805e3e4a: : Instructing follower d88b316b7ecd49f29219068c5fe51a26 to start an election
I20260709 12:58:25.196380  1404 raft_consensus.cc:1081] T 2685bf6c4fd043a481d1d05136dbbd5e P b3afae27e8744ee6921cd67f805e3e4a [term 2 LEADER]: Signalling peer d88b316b7ecd49f29219068c5fe51a26 to start an election
I20260709 12:58:25.198261  1093 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "2685bf6c4fd043a481d1d05136dbbd5e"
dest_uuid: "d88b316b7ecd49f29219068c5fe51a26"
 from {username='slave'} at 127.0.0.1:49984
I20260709 12:58:25.199128  1093 raft_consensus.cc:493] T 2685bf6c4fd043a481d1d05136dbbd5e P d88b316b7ecd49f29219068c5fe51a26 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:58:25.199579  1093 raft_consensus.cc:3060] T 2685bf6c4fd043a481d1d05136dbbd5e P d88b316b7ecd49f29219068c5fe51a26 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:58:25.208482  1093 raft_consensus.cc:515] T 2685bf6c4fd043a481d1d05136dbbd5e P d88b316b7ecd49f29219068c5fe51a26 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:25.210943  1093 leader_election.cc:290] T 2685bf6c4fd043a481d1d05136dbbd5e P d88b316b7ecd49f29219068c5fe51a26 [CANDIDATE]: Term 3 election: Requested vote from peers b3afae27e8744ee6921cd67f805e3e4a (127.0.107.194:36785), d0e95822debe474ba6a700c661aa262f (127.0.107.195:40863)
I20260709 12:58:25.211855  1168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2685bf6c4fd043a481d1d05136dbbd5e" candidate_uuid: "d88b316b7ecd49f29219068c5fe51a26" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "b3afae27e8744ee6921cd67f805e3e4a"
I20260709 12:58:25.212760  1168 raft_consensus.cc:3055] T 2685bf6c4fd043a481d1d05136dbbd5e P b3afae27e8744ee6921cd67f805e3e4a [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:58:25.212924  1243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2685bf6c4fd043a481d1d05136dbbd5e" candidate_uuid: "d88b316b7ecd49f29219068c5fe51a26" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "d0e95822debe474ba6a700c661aa262f"
I20260709 12:58:25.213261  1168 raft_consensus.cc:740] T 2685bf6c4fd043a481d1d05136dbbd5e P b3afae27e8744ee6921cd67f805e3e4a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: b3afae27e8744ee6921cd67f805e3e4a, State: Running, Role: LEADER
I20260709 12:58:25.213568  1243 raft_consensus.cc:3060] T 2685bf6c4fd043a481d1d05136dbbd5e P d0e95822debe474ba6a700c661aa262f [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:58:25.214495  1168 consensus_queue.cc:260] T 2685bf6c4fd043a481d1d05136dbbd5e P b3afae27e8744ee6921cd67f805e3e4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:25.216251  1168 raft_consensus.cc:3060] T 2685bf6c4fd043a481d1d05136dbbd5e P b3afae27e8744ee6921cd67f805e3e4a [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:58:25.222738  1243 raft_consensus.cc:2468] T 2685bf6c4fd043a481d1d05136dbbd5e P d0e95822debe474ba6a700c661aa262f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d88b316b7ecd49f29219068c5fe51a26 in term 3.
I20260709 12:58:25.224751  1061 leader_election.cc:304] T 2685bf6c4fd043a481d1d05136dbbd5e P d88b316b7ecd49f29219068c5fe51a26 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d0e95822debe474ba6a700c661aa262f, d88b316b7ecd49f29219068c5fe51a26; no voters: 
I20260709 12:58:25.225687  1403 raft_consensus.cc:2804] T 2685bf6c4fd043a481d1d05136dbbd5e P d88b316b7ecd49f29219068c5fe51a26 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:58:25.226511  1168 raft_consensus.cc:2468] T 2685bf6c4fd043a481d1d05136dbbd5e P b3afae27e8744ee6921cd67f805e3e4a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d88b316b7ecd49f29219068c5fe51a26 in term 3.
I20260709 12:58:25.228127  1403 raft_consensus.cc:697] T 2685bf6c4fd043a481d1d05136dbbd5e P d88b316b7ecd49f29219068c5fe51a26 [term 3 LEADER]: Becoming Leader. State: Replica: d88b316b7ecd49f29219068c5fe51a26, State: Running, Role: LEADER
I20260709 12:58:25.229218  1403 consensus_queue.cc:237] T 2685bf6c4fd043a481d1d05136dbbd5e P d88b316b7ecd49f29219068c5fe51a26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:25.236835   990 catalog_manager.cc:5697] T 2685bf6c4fd043a481d1d05136dbbd5e P d88b316b7ecd49f29219068c5fe51a26 reported cstate change: term changed from 2 to 3, leader changed from b3afae27e8744ee6921cd67f805e3e4a (127.0.107.194) to d88b316b7ecd49f29219068c5fe51a26 (127.0.107.193). New cstate: current_term: 3 leader_uuid: "d88b316b7ecd49f29219068c5fe51a26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } health_report { overall_health: UNKNOWN } } }
I20260709 12:58:25.342492  1396 raft_consensus.cc:993] T 1935c9164755416a965cccf80900d941 P d0e95822debe474ba6a700c661aa262f: : Instructing follower d88b316b7ecd49f29219068c5fe51a26 to start an election
I20260709 12:58:25.343034  1412 raft_consensus.cc:1081] T 1935c9164755416a965cccf80900d941 P d0e95822debe474ba6a700c661aa262f [term 2 LEADER]: Signalling peer d88b316b7ecd49f29219068c5fe51a26 to start an election
I20260709 12:58:25.344676  1093 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "1935c9164755416a965cccf80900d941"
dest_uuid: "d88b316b7ecd49f29219068c5fe51a26"
 from {username='slave'} at 127.0.0.1:49986
I20260709 12:58:25.345263  1093 raft_consensus.cc:493] T 1935c9164755416a965cccf80900d941 P d88b316b7ecd49f29219068c5fe51a26 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:58:25.345790  1093 raft_consensus.cc:3060] T 1935c9164755416a965cccf80900d941 P d88b316b7ecd49f29219068c5fe51a26 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:58:25.351632  1093 raft_consensus.cc:515] T 1935c9164755416a965cccf80900d941 P d88b316b7ecd49f29219068c5fe51a26 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:25.353365  1093 leader_election.cc:290] T 1935c9164755416a965cccf80900d941 P d88b316b7ecd49f29219068c5fe51a26 [CANDIDATE]: Term 3 election: Requested vote from peers b3afae27e8744ee6921cd67f805e3e4a (127.0.107.194:36785), d0e95822debe474ba6a700c661aa262f (127.0.107.195:40863)
I20260709 12:58:25.354153  1168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1935c9164755416a965cccf80900d941" candidate_uuid: "d88b316b7ecd49f29219068c5fe51a26" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "b3afae27e8744ee6921cd67f805e3e4a"
I20260709 12:58:25.354529  1243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1935c9164755416a965cccf80900d941" candidate_uuid: "d88b316b7ecd49f29219068c5fe51a26" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "d0e95822debe474ba6a700c661aa262f"
I20260709 12:58:25.354926  1168 raft_consensus.cc:3060] T 1935c9164755416a965cccf80900d941 P b3afae27e8744ee6921cd67f805e3e4a [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:58:25.355077  1243 raft_consensus.cc:3055] T 1935c9164755416a965cccf80900d941 P d0e95822debe474ba6a700c661aa262f [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:58:25.355427  1243 raft_consensus.cc:740] T 1935c9164755416a965cccf80900d941 P d0e95822debe474ba6a700c661aa262f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d0e95822debe474ba6a700c661aa262f, State: Running, Role: LEADER
I20260709 12:58:25.356045  1243 consensus_queue.cc:260] T 1935c9164755416a965cccf80900d941 P d0e95822debe474ba6a700c661aa262f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:25.356959  1243 raft_consensus.cc:3060] T 1935c9164755416a965cccf80900d941 P d0e95822debe474ba6a700c661aa262f [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:58:25.360814  1168 raft_consensus.cc:2468] T 1935c9164755416a965cccf80900d941 P b3afae27e8744ee6921cd67f805e3e4a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d88b316b7ecd49f29219068c5fe51a26 in term 3.
I20260709 12:58:25.361850  1243 raft_consensus.cc:2468] T 1935c9164755416a965cccf80900d941 P d0e95822debe474ba6a700c661aa262f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d88b316b7ecd49f29219068c5fe51a26 in term 3.
I20260709 12:58:25.361945  1059 leader_election.cc:304] T 1935c9164755416a965cccf80900d941 P d88b316b7ecd49f29219068c5fe51a26 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3afae27e8744ee6921cd67f805e3e4a, d88b316b7ecd49f29219068c5fe51a26; no voters: 
I20260709 12:58:25.362848  1403 raft_consensus.cc:2804] T 1935c9164755416a965cccf80900d941 P d88b316b7ecd49f29219068c5fe51a26 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:58:25.363343  1403 raft_consensus.cc:697] T 1935c9164755416a965cccf80900d941 P d88b316b7ecd49f29219068c5fe51a26 [term 3 LEADER]: Becoming Leader. State: Replica: d88b316b7ecd49f29219068c5fe51a26, State: Running, Role: LEADER
I20260709 12:58:25.364158  1403 consensus_queue.cc:237] T 1935c9164755416a965cccf80900d941 P d88b316b7ecd49f29219068c5fe51a26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:25.370529   990 catalog_manager.cc:5697] T 1935c9164755416a965cccf80900d941 P d88b316b7ecd49f29219068c5fe51a26 reported cstate change: term changed from 2 to 3, leader changed from d0e95822debe474ba6a700c661aa262f (127.0.107.195) to d88b316b7ecd49f29219068c5fe51a26 (127.0.107.193). New cstate: current_term: 3 leader_uuid: "d88b316b7ecd49f29219068c5fe51a26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } health_report { overall_health: UNKNOWN } } }
I20260709 12:58:25.527594  1168 raft_consensus.cc:1275] T 73e09365f8cd4513a4600030950af811 P b3afae27e8744ee6921cd67f805e3e4a [term 3 FOLLOWER]: Refusing update from remote peer d88b316b7ecd49f29219068c5fe51a26: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:58:25.529352  1403 consensus_queue.cc:1048] T 73e09365f8cd4513a4600030950af811 P d88b316b7ecd49f29219068c5fe51a26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:58:25.541258  1243 raft_consensus.cc:1275] T 73e09365f8cd4513a4600030950af811 P d0e95822debe474ba6a700c661aa262f [term 3 FOLLOWER]: Refusing update from remote peer d88b316b7ecd49f29219068c5fe51a26: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:58:25.543335  1386 consensus_queue.cc:1048] T 73e09365f8cd4513a4600030950af811 P d88b316b7ecd49f29219068c5fe51a26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:58:25.560655  1243 raft_consensus.cc:1275] T 2cde8b0b8c5c46e78f405744f084c7c6 P d0e95822debe474ba6a700c661aa262f [term 3 FOLLOWER]: Refusing update from remote peer d88b316b7ecd49f29219068c5fe51a26: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:58:25.562031  1410 consensus_queue.cc:1048] T 2cde8b0b8c5c46e78f405744f084c7c6 P d88b316b7ecd49f29219068c5fe51a26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:58:25.575583  1168 raft_consensus.cc:1275] T 2cde8b0b8c5c46e78f405744f084c7c6 P b3afae27e8744ee6921cd67f805e3e4a [term 3 FOLLOWER]: Refusing update from remote peer d88b316b7ecd49f29219068c5fe51a26: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:58:25.577479  1410 consensus_queue.cc:1048] T 2cde8b0b8c5c46e78f405744f084c7c6 P d88b316b7ecd49f29219068c5fe51a26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:58:25.694326  1243 raft_consensus.cc:1275] T d87ed5c1a25e40f8b7aa82d33ca30bf1 P d0e95822debe474ba6a700c661aa262f [term 3 FOLLOWER]: Refusing update from remote peer d88b316b7ecd49f29219068c5fe51a26: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:58:25.695730  1410 consensus_queue.cc:1048] T d87ed5c1a25e40f8b7aa82d33ca30bf1 P d88b316b7ecd49f29219068c5fe51a26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:58:25.709093  1168 raft_consensus.cc:1275] T d87ed5c1a25e40f8b7aa82d33ca30bf1 P b3afae27e8744ee6921cd67f805e3e4a [term 3 FOLLOWER]: Refusing update from remote peer d88b316b7ecd49f29219068c5fe51a26: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:58:25.710893  1410 consensus_queue.cc:1048] T d87ed5c1a25e40f8b7aa82d33ca30bf1 P d88b316b7ecd49f29219068c5fe51a26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:58:25.727561  1243 raft_consensus.cc:1275] T 2685bf6c4fd043a481d1d05136dbbd5e P d0e95822debe474ba6a700c661aa262f [term 3 FOLLOWER]: Refusing update from remote peer d88b316b7ecd49f29219068c5fe51a26: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:58:25.729199  1417 consensus_queue.cc:1048] T 2685bf6c4fd043a481d1d05136dbbd5e P d88b316b7ecd49f29219068c5fe51a26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:58:25.742357  1168 raft_consensus.cc:1275] T 2685bf6c4fd043a481d1d05136dbbd5e P b3afae27e8744ee6921cd67f805e3e4a [term 3 FOLLOWER]: Refusing update from remote peer d88b316b7ecd49f29219068c5fe51a26: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:58:25.745126  1417 consensus_queue.cc:1048] T 2685bf6c4fd043a481d1d05136dbbd5e P d88b316b7ecd49f29219068c5fe51a26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:58:25.839160  1243 raft_consensus.cc:1275] T 1935c9164755416a965cccf80900d941 P d0e95822debe474ba6a700c661aa262f [term 3 FOLLOWER]: Refusing update from remote peer d88b316b7ecd49f29219068c5fe51a26: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:58:25.840804  1417 consensus_queue.cc:1048] T 1935c9164755416a965cccf80900d941 P d88b316b7ecd49f29219068c5fe51a26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:58:25.851616  1168 raft_consensus.cc:1275] T 1935c9164755416a965cccf80900d941 P b3afae27e8744ee6921cd67f805e3e4a [term 3 FOLLOWER]: Refusing update from remote peer d88b316b7ecd49f29219068c5fe51a26: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:58:25.853551  1386 consensus_queue.cc:1048] T 1935c9164755416a965cccf80900d941 P d88b316b7ecd49f29219068c5fe51a26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:58:28.039047   431 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:58:28.059587  1093 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "2cde8b0b8c5c46e78f405744f084c7c6"
dest_uuid: "d88b316b7ecd49f29219068c5fe51a26"
mode: GRACEFUL
new_leader_uuid: "d0e95822debe474ba6a700c661aa262f"
 from {username='slave'} at 127.0.0.1:35586
I20260709 12:58:28.060346  1093 raft_consensus.cc:606] T 2cde8b0b8c5c46e78f405744f084c7c6 P d88b316b7ecd49f29219068c5fe51a26 [term 3 LEADER]: Received request to transfer leadership to TS d0e95822debe474ba6a700c661aa262f
I20260709 12:58:28.063663  1093 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "6198a4398a3140e8ad9f1a158956fe3e"
dest_uuid: "d88b316b7ecd49f29219068c5fe51a26"
mode: GRACEFUL
new_leader_uuid: "d0e95822debe474ba6a700c661aa262f"
 from {username='slave'} at 127.0.0.1:35586
I20260709 12:58:28.064324  1093 raft_consensus.cc:606] T 6198a4398a3140e8ad9f1a158956fe3e P d88b316b7ecd49f29219068c5fe51a26 [term 2 LEADER]: Received request to transfer leadership to TS d0e95822debe474ba6a700c661aa262f
I20260709 12:58:28.066953  1093 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "759e30de7d7b4bc6980dbf8c4ba8031f"
dest_uuid: "d88b316b7ecd49f29219068c5fe51a26"
mode: GRACEFUL
new_leader_uuid: "b3afae27e8744ee6921cd67f805e3e4a"
 from {username='slave'} at 127.0.0.1:35586
I20260709 12:58:28.067624  1093 raft_consensus.cc:606] T 759e30de7d7b4bc6980dbf8c4ba8031f P d88b316b7ecd49f29219068c5fe51a26 [term 2 LEADER]: Received request to transfer leadership to TS b3afae27e8744ee6921cd67f805e3e4a
I20260709 12:58:28.070770  1093 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "d87ed5c1a25e40f8b7aa82d33ca30bf1"
dest_uuid: "d88b316b7ecd49f29219068c5fe51a26"
mode: GRACEFUL
new_leader_uuid: "b3afae27e8744ee6921cd67f805e3e4a"
 from {username='slave'} at 127.0.0.1:35586
I20260709 12:58:28.071525  1093 raft_consensus.cc:606] T d87ed5c1a25e40f8b7aa82d33ca30bf1 P d88b316b7ecd49f29219068c5fe51a26 [term 3 LEADER]: Received request to transfer leadership to TS b3afae27e8744ee6921cd67f805e3e4a
I20260709 12:58:28.072738   431 auto_leader_rebalancer.cc:452] table: test-workload, leader rebalance finish, leader transfer count: 4
I20260709 12:58:28.073272   431 auto_leader_rebalancer.cc:527] All tables' leader rebalancing finished this round
I20260709 12:58:28.133761  1410 raft_consensus.cc:993] T 759e30de7d7b4bc6980dbf8c4ba8031f P d88b316b7ecd49f29219068c5fe51a26: : Instructing follower b3afae27e8744ee6921cd67f805e3e4a to start an election
I20260709 12:58:28.134514  1428 raft_consensus.cc:1081] T 759e30de7d7b4bc6980dbf8c4ba8031f P d88b316b7ecd49f29219068c5fe51a26 [term 2 LEADER]: Signalling peer b3afae27e8744ee6921cd67f805e3e4a to start an election
I20260709 12:58:28.136648  1170 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "759e30de7d7b4bc6980dbf8c4ba8031f"
dest_uuid: "b3afae27e8744ee6921cd67f805e3e4a"
 from {username='slave'} at 127.0.0.1:34394
I20260709 12:58:28.137423  1170 raft_consensus.cc:493] T 759e30de7d7b4bc6980dbf8c4ba8031f P b3afae27e8744ee6921cd67f805e3e4a [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:58:28.137892  1170 raft_consensus.cc:3060] T 759e30de7d7b4bc6980dbf8c4ba8031f P b3afae27e8744ee6921cd67f805e3e4a [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:58:28.145170  1170 raft_consensus.cc:515] T 759e30de7d7b4bc6980dbf8c4ba8031f P b3afae27e8744ee6921cd67f805e3e4a [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:28.147684  1170 leader_election.cc:290] T 759e30de7d7b4bc6980dbf8c4ba8031f P b3afae27e8744ee6921cd67f805e3e4a [CANDIDATE]: Term 3 election: Requested vote from peers d88b316b7ecd49f29219068c5fe51a26 (127.0.107.193:38591), d0e95822debe474ba6a700c661aa262f (127.0.107.195:40863)
I20260709 12:58:28.148382  1093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "759e30de7d7b4bc6980dbf8c4ba8031f" candidate_uuid: "b3afae27e8744ee6921cd67f805e3e4a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "d88b316b7ecd49f29219068c5fe51a26"
I20260709 12:58:28.149067  1245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "759e30de7d7b4bc6980dbf8c4ba8031f" candidate_uuid: "b3afae27e8744ee6921cd67f805e3e4a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "d0e95822debe474ba6a700c661aa262f"
I20260709 12:58:28.149221  1093 raft_consensus.cc:3055] T 759e30de7d7b4bc6980dbf8c4ba8031f P d88b316b7ecd49f29219068c5fe51a26 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:58:28.149664  1093 raft_consensus.cc:740] T 759e30de7d7b4bc6980dbf8c4ba8031f P d88b316b7ecd49f29219068c5fe51a26 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d88b316b7ecd49f29219068c5fe51a26, State: Running, Role: LEADER
I20260709 12:58:28.149724  1245 raft_consensus.cc:3060] T 759e30de7d7b4bc6980dbf8c4ba8031f P d0e95822debe474ba6a700c661aa262f [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:58:28.150789  1093 consensus_queue.cc:260] T 759e30de7d7b4bc6980dbf8c4ba8031f P d88b316b7ecd49f29219068c5fe51a26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:28.152060  1093 raft_consensus.cc:3060] T 759e30de7d7b4bc6980dbf8c4ba8031f P d88b316b7ecd49f29219068c5fe51a26 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:58:28.156904  1093 raft_consensus.cc:2468] T 759e30de7d7b4bc6980dbf8c4ba8031f P d88b316b7ecd49f29219068c5fe51a26 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3afae27e8744ee6921cd67f805e3e4a in term 3.
I20260709 12:58:28.156949  1245 raft_consensus.cc:2468] T 759e30de7d7b4bc6980dbf8c4ba8031f P d0e95822debe474ba6a700c661aa262f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3afae27e8744ee6921cd67f805e3e4a in term 3.
I20260709 12:58:28.158022  1133 leader_election.cc:304] T 759e30de7d7b4bc6980dbf8c4ba8031f P b3afae27e8744ee6921cd67f805e3e4a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3afae27e8744ee6921cd67f805e3e4a, d88b316b7ecd49f29219068c5fe51a26; no voters: 
I20260709 12:58:28.159402  1426 raft_consensus.cc:2804] T 759e30de7d7b4bc6980dbf8c4ba8031f P b3afae27e8744ee6921cd67f805e3e4a [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:58:28.160550  1426 raft_consensus.cc:697] T 759e30de7d7b4bc6980dbf8c4ba8031f P b3afae27e8744ee6921cd67f805e3e4a [term 3 LEADER]: Becoming Leader. State: Replica: b3afae27e8744ee6921cd67f805e3e4a, State: Running, Role: LEADER
I20260709 12:58:28.161741  1426 consensus_queue.cc:237] T 759e30de7d7b4bc6980dbf8c4ba8031f P b3afae27e8744ee6921cd67f805e3e4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:28.173079   992 catalog_manager.cc:5697] T 759e30de7d7b4bc6980dbf8c4ba8031f P b3afae27e8744ee6921cd67f805e3e4a reported cstate change: term changed from 2 to 3, leader changed from d88b316b7ecd49f29219068c5fe51a26 (127.0.107.193) to b3afae27e8744ee6921cd67f805e3e4a (127.0.107.194). New cstate: current_term: 3 leader_uuid: "b3afae27e8744ee6921cd67f805e3e4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } health_report { overall_health: UNKNOWN } } }
I20260709 12:58:28.341085  1427 raft_consensus.cc:993] T 6198a4398a3140e8ad9f1a158956fe3e P d88b316b7ecd49f29219068c5fe51a26: : Instructing follower d0e95822debe474ba6a700c661aa262f to start an election
I20260709 12:58:28.341778  1428 raft_consensus.cc:1081] T 6198a4398a3140e8ad9f1a158956fe3e P d88b316b7ecd49f29219068c5fe51a26 [term 2 LEADER]: Signalling peer d0e95822debe474ba6a700c661aa262f to start an election
I20260709 12:58:28.343634  1243 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "6198a4398a3140e8ad9f1a158956fe3e"
dest_uuid: "d0e95822debe474ba6a700c661aa262f"
 from {username='slave'} at 127.0.0.1:33256
I20260709 12:58:28.344079  1243 raft_consensus.cc:493] T 6198a4398a3140e8ad9f1a158956fe3e P d0e95822debe474ba6a700c661aa262f [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:58:28.344314  1243 raft_consensus.cc:3060] T 6198a4398a3140e8ad9f1a158956fe3e P d0e95822debe474ba6a700c661aa262f [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:58:28.350968  1243 raft_consensus.cc:515] T 6198a4398a3140e8ad9f1a158956fe3e P d0e95822debe474ba6a700c661aa262f [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:28.352826  1243 leader_election.cc:290] T 6198a4398a3140e8ad9f1a158956fe3e P d0e95822debe474ba6a700c661aa262f [CANDIDATE]: Term 3 election: Requested vote from peers b3afae27e8744ee6921cd67f805e3e4a (127.0.107.194:36785), d88b316b7ecd49f29219068c5fe51a26 (127.0.107.193:38591)
I20260709 12:58:28.353989  1170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6198a4398a3140e8ad9f1a158956fe3e" candidate_uuid: "d0e95822debe474ba6a700c661aa262f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "b3afae27e8744ee6921cd67f805e3e4a"
I20260709 12:58:28.354468  1093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6198a4398a3140e8ad9f1a158956fe3e" candidate_uuid: "d0e95822debe474ba6a700c661aa262f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "d88b316b7ecd49f29219068c5fe51a26"
I20260709 12:58:28.354846  1170 raft_consensus.cc:3060] T 6198a4398a3140e8ad9f1a158956fe3e P b3afae27e8744ee6921cd67f805e3e4a [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:58:28.355165  1093 raft_consensus.cc:3055] T 6198a4398a3140e8ad9f1a158956fe3e P d88b316b7ecd49f29219068c5fe51a26 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:58:28.355525  1093 raft_consensus.cc:740] T 6198a4398a3140e8ad9f1a158956fe3e P d88b316b7ecd49f29219068c5fe51a26 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d88b316b7ecd49f29219068c5fe51a26, State: Running, Role: LEADER
I20260709 12:58:28.356459  1093 consensus_queue.cc:260] T 6198a4398a3140e8ad9f1a158956fe3e P d88b316b7ecd49f29219068c5fe51a26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:28.357800  1093 raft_consensus.cc:3060] T 6198a4398a3140e8ad9f1a158956fe3e P d88b316b7ecd49f29219068c5fe51a26 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:58:28.364287  1170 raft_consensus.cc:2468] T 6198a4398a3140e8ad9f1a158956fe3e P b3afae27e8744ee6921cd67f805e3e4a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0e95822debe474ba6a700c661aa262f in term 3.
I20260709 12:58:28.366067  1093 raft_consensus.cc:2468] T 6198a4398a3140e8ad9f1a158956fe3e P d88b316b7ecd49f29219068c5fe51a26 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0e95822debe474ba6a700c661aa262f in term 3.
I20260709 12:58:28.365924  1209 leader_election.cc:304] T 6198a4398a3140e8ad9f1a158956fe3e P d0e95822debe474ba6a700c661aa262f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3afae27e8744ee6921cd67f805e3e4a, d0e95822debe474ba6a700c661aa262f; no voters: 
I20260709 12:58:28.367012  1412 raft_consensus.cc:2804] T 6198a4398a3140e8ad9f1a158956fe3e P d0e95822debe474ba6a700c661aa262f [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:58:28.369060  1412 raft_consensus.cc:697] T 6198a4398a3140e8ad9f1a158956fe3e P d0e95822debe474ba6a700c661aa262f [term 3 LEADER]: Becoming Leader. State: Replica: d0e95822debe474ba6a700c661aa262f, State: Running, Role: LEADER
I20260709 12:58:28.370239  1412 consensus_queue.cc:237] T 6198a4398a3140e8ad9f1a158956fe3e P d0e95822debe474ba6a700c661aa262f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:28.379405   992 catalog_manager.cc:5697] T 6198a4398a3140e8ad9f1a158956fe3e P d0e95822debe474ba6a700c661aa262f reported cstate change: term changed from 2 to 3, leader changed from d88b316b7ecd49f29219068c5fe51a26 (127.0.107.193) to d0e95822debe474ba6a700c661aa262f (127.0.107.195). New cstate: current_term: 3 leader_uuid: "d0e95822debe474ba6a700c661aa262f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } health_report { overall_health: HEALTHY } } }
I20260709 12:58:28.553395  1428 raft_consensus.cc:993] T 2cde8b0b8c5c46e78f405744f084c7c6 P d88b316b7ecd49f29219068c5fe51a26: : Instructing follower d0e95822debe474ba6a700c661aa262f to start an election
I20260709 12:58:28.554214  1428 raft_consensus.cc:1081] T 2cde8b0b8c5c46e78f405744f084c7c6 P d88b316b7ecd49f29219068c5fe51a26 [term 3 LEADER]: Signalling peer d0e95822debe474ba6a700c661aa262f to start an election
I20260709 12:58:28.556172  1243 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "2cde8b0b8c5c46e78f405744f084c7c6"
dest_uuid: "d0e95822debe474ba6a700c661aa262f"
 from {username='slave'} at 127.0.0.1:33256
I20260709 12:58:28.556298  1093 raft_consensus.cc:1275] T 759e30de7d7b4bc6980dbf8c4ba8031f P d88b316b7ecd49f29219068c5fe51a26 [term 3 FOLLOWER]: Refusing update from remote peer b3afae27e8744ee6921cd67f805e3e4a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:58:28.556715  1243 raft_consensus.cc:493] T 2cde8b0b8c5c46e78f405744f084c7c6 P d0e95822debe474ba6a700c661aa262f [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:58:28.557063  1243 raft_consensus.cc:3060] T 2cde8b0b8c5c46e78f405744f084c7c6 P d0e95822debe474ba6a700c661aa262f [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:58:28.557984  1426 consensus_queue.cc:1048] T 759e30de7d7b4bc6980dbf8c4ba8031f P b3afae27e8744ee6921cd67f805e3e4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:58:28.563621  1243 raft_consensus.cc:515] T 2cde8b0b8c5c46e78f405744f084c7c6 P d0e95822debe474ba6a700c661aa262f [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:28.567727  1095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cde8b0b8c5c46e78f405744f084c7c6" candidate_uuid: "d0e95822debe474ba6a700c661aa262f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "d88b316b7ecd49f29219068c5fe51a26"
I20260709 12:58:28.568545  1095 raft_consensus.cc:3055] T 2cde8b0b8c5c46e78f405744f084c7c6 P d88b316b7ecd49f29219068c5fe51a26 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:58:28.568995  1095 raft_consensus.cc:740] T 2cde8b0b8c5c46e78f405744f084c7c6 P d88b316b7ecd49f29219068c5fe51a26 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d88b316b7ecd49f29219068c5fe51a26, State: Running, Role: LEADER
I20260709 12:58:28.570281  1427 raft_consensus.cc:993] T d87ed5c1a25e40f8b7aa82d33ca30bf1 P d88b316b7ecd49f29219068c5fe51a26: : Instructing follower b3afae27e8744ee6921cd67f805e3e4a to start an election
I20260709 12:58:28.571218  1243 leader_election.cc:290] T 2cde8b0b8c5c46e78f405744f084c7c6 P d0e95822debe474ba6a700c661aa262f [CANDIDATE]: Term 4 election: Requested vote from peers d88b316b7ecd49f29219068c5fe51a26 (127.0.107.193:38591), b3afae27e8744ee6921cd67f805e3e4a (127.0.107.194:36785)
I20260709 12:58:28.570936  1410 raft_consensus.cc:1081] T d87ed5c1a25e40f8b7aa82d33ca30bf1 P d88b316b7ecd49f29219068c5fe51a26 [term 3 LEADER]: Signalling peer b3afae27e8744ee6921cd67f805e3e4a to start an election
I20260709 12:58:28.570298  1095 consensus_queue.cc:260] T 2cde8b0b8c5c46e78f405744f084c7c6 P d88b316b7ecd49f29219068c5fe51a26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:28.572703  1095 raft_consensus.cc:3060] T 2cde8b0b8c5c46e78f405744f084c7c6 P d88b316b7ecd49f29219068c5fe51a26 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:58:28.576049  1170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cde8b0b8c5c46e78f405744f084c7c6" candidate_uuid: "d0e95822debe474ba6a700c661aa262f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "b3afae27e8744ee6921cd67f805e3e4a"
I20260709 12:58:28.576828  1170 raft_consensus.cc:3060] T 2cde8b0b8c5c46e78f405744f084c7c6 P b3afae27e8744ee6921cd67f805e3e4a [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:58:28.582749  1168 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "d87ed5c1a25e40f8b7aa82d33ca30bf1"
dest_uuid: "b3afae27e8744ee6921cd67f805e3e4a"
 from {username='slave'} at 127.0.0.1:34394
I20260709 12:58:28.583455  1168 raft_consensus.cc:493] T d87ed5c1a25e40f8b7aa82d33ca30bf1 P b3afae27e8744ee6921cd67f805e3e4a [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:58:28.583861  1168 raft_consensus.cc:3060] T d87ed5c1a25e40f8b7aa82d33ca30bf1 P b3afae27e8744ee6921cd67f805e3e4a [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:58:28.601678  1170 raft_consensus.cc:2468] T 2cde8b0b8c5c46e78f405744f084c7c6 P b3afae27e8744ee6921cd67f805e3e4a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0e95822debe474ba6a700c661aa262f in term 4.
I20260709 12:58:28.603515  1209 leader_election.cc:304] T 2cde8b0b8c5c46e78f405744f084c7c6 P d0e95822debe474ba6a700c661aa262f [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3afae27e8744ee6921cd67f805e3e4a, d0e95822debe474ba6a700c661aa262f; no voters: 
I20260709 12:58:28.604511  1412 raft_consensus.cc:2804] T 2cde8b0b8c5c46e78f405744f084c7c6 P d0e95822debe474ba6a700c661aa262f [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:58:28.605293  1412 raft_consensus.cc:697] T 2cde8b0b8c5c46e78f405744f084c7c6 P d0e95822debe474ba6a700c661aa262f [term 4 LEADER]: Becoming Leader. State: Replica: d0e95822debe474ba6a700c661aa262f, State: Running, Role: LEADER
I20260709 12:58:28.605381  1168 raft_consensus.cc:515] T d87ed5c1a25e40f8b7aa82d33ca30bf1 P b3afae27e8744ee6921cd67f805e3e4a [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:28.606825  1412 consensus_queue.cc:237] T 2cde8b0b8c5c46e78f405744f084c7c6 P d0e95822debe474ba6a700c661aa262f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:28.608994  1093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d87ed5c1a25e40f8b7aa82d33ca30bf1" candidate_uuid: "b3afae27e8744ee6921cd67f805e3e4a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "d88b316b7ecd49f29219068c5fe51a26"
I20260709 12:58:28.609707  1093 raft_consensus.cc:3055] T d87ed5c1a25e40f8b7aa82d33ca30bf1 P d88b316b7ecd49f29219068c5fe51a26 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:58:28.610064  1093 raft_consensus.cc:740] T d87ed5c1a25e40f8b7aa82d33ca30bf1 P d88b316b7ecd49f29219068c5fe51a26 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d88b316b7ecd49f29219068c5fe51a26, State: Running, Role: LEADER
I20260709 12:58:28.611145  1093 consensus_queue.cc:260] T d87ed5c1a25e40f8b7aa82d33ca30bf1 P d88b316b7ecd49f29219068c5fe51a26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:28.612601  1093 raft_consensus.cc:3060] T d87ed5c1a25e40f8b7aa82d33ca30bf1 P d88b316b7ecd49f29219068c5fe51a26 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:58:28.616052  1168 leader_election.cc:290] T d87ed5c1a25e40f8b7aa82d33ca30bf1 P b3afae27e8744ee6921cd67f805e3e4a [CANDIDATE]: Term 4 election: Requested vote from peers d88b316b7ecd49f29219068c5fe51a26 (127.0.107.193:38591), d0e95822debe474ba6a700c661aa262f (127.0.107.195:40863)
I20260709 12:58:28.618741  1243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d87ed5c1a25e40f8b7aa82d33ca30bf1" candidate_uuid: "b3afae27e8744ee6921cd67f805e3e4a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "d0e95822debe474ba6a700c661aa262f"
I20260709 12:58:28.619338  1243 raft_consensus.cc:3060] T d87ed5c1a25e40f8b7aa82d33ca30bf1 P d0e95822debe474ba6a700c661aa262f [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:58:28.621546  1093 raft_consensus.cc:2468] T d87ed5c1a25e40f8b7aa82d33ca30bf1 P d88b316b7ecd49f29219068c5fe51a26 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3afae27e8744ee6921cd67f805e3e4a in term 4.
I20260709 12:58:28.622973  1133 leader_election.cc:304] T d87ed5c1a25e40f8b7aa82d33ca30bf1 P b3afae27e8744ee6921cd67f805e3e4a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3afae27e8744ee6921cd67f805e3e4a, d88b316b7ecd49f29219068c5fe51a26; no voters: 
I20260709 12:58:28.627637  1243 raft_consensus.cc:2468] T d87ed5c1a25e40f8b7aa82d33ca30bf1 P d0e95822debe474ba6a700c661aa262f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3afae27e8744ee6921cd67f805e3e4a in term 4.
I20260709 12:58:28.626076  1440 raft_consensus.cc:2804] T d87ed5c1a25e40f8b7aa82d33ca30bf1 P b3afae27e8744ee6921cd67f805e3e4a [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:58:28.632030  1095 raft_consensus.cc:2468] T 2cde8b0b8c5c46e78f405744f084c7c6 P d88b316b7ecd49f29219068c5fe51a26 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0e95822debe474ba6a700c661aa262f in term 4.
I20260709 12:58:28.636970   992 catalog_manager.cc:5697] T 2cde8b0b8c5c46e78f405744f084c7c6 P d0e95822debe474ba6a700c661aa262f reported cstate change: term changed from 3 to 4, leader changed from d88b316b7ecd49f29219068c5fe51a26 (127.0.107.193) to d0e95822debe474ba6a700c661aa262f (127.0.107.195). New cstate: current_term: 4 leader_uuid: "d0e95822debe474ba6a700c661aa262f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } health_report { overall_health: HEALTHY } } }
I20260709 12:58:28.639133  1243 raft_consensus.cc:1275] T 759e30de7d7b4bc6980dbf8c4ba8031f P d0e95822debe474ba6a700c661aa262f [term 3 FOLLOWER]: Refusing update from remote peer b3afae27e8744ee6921cd67f805e3e4a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:58:28.640954  1440 raft_consensus.cc:697] T d87ed5c1a25e40f8b7aa82d33ca30bf1 P b3afae27e8744ee6921cd67f805e3e4a [term 4 LEADER]: Becoming Leader. State: Replica: b3afae27e8744ee6921cd67f805e3e4a, State: Running, Role: LEADER
I20260709 12:58:28.642485  1440 consensus_queue.cc:237] T d87ed5c1a25e40f8b7aa82d33ca30bf1 P b3afae27e8744ee6921cd67f805e3e4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:28.643576  1443 consensus_queue.cc:1048] T 759e30de7d7b4bc6980dbf8c4ba8031f P b3afae27e8744ee6921cd67f805e3e4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:58:28.657749   992 catalog_manager.cc:5697] T d87ed5c1a25e40f8b7aa82d33ca30bf1 P b3afae27e8744ee6921cd67f805e3e4a reported cstate change: term changed from 3 to 4, leader changed from d88b316b7ecd49f29219068c5fe51a26 (127.0.107.193) to b3afae27e8744ee6921cd67f805e3e4a (127.0.107.194). New cstate: current_term: 4 leader_uuid: "b3afae27e8744ee6921cd67f805e3e4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } health_report { overall_health: UNKNOWN } } }
I20260709 12:58:28.879451  1095 raft_consensus.cc:1275] T 6198a4398a3140e8ad9f1a158956fe3e P d88b316b7ecd49f29219068c5fe51a26 [term 3 FOLLOWER]: Refusing update from remote peer d0e95822debe474ba6a700c661aa262f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:58:28.881552  1412 consensus_queue.cc:1048] T 6198a4398a3140e8ad9f1a158956fe3e P d0e95822debe474ba6a700c661aa262f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:58:28.934832  1168 raft_consensus.cc:1275] T 6198a4398a3140e8ad9f1a158956fe3e P b3afae27e8744ee6921cd67f805e3e4a [term 3 FOLLOWER]: Refusing update from remote peer d0e95822debe474ba6a700c661aa262f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:58:28.938129  1412 consensus_queue.cc:1048] T 6198a4398a3140e8ad9f1a158956fe3e P d0e95822debe474ba6a700c661aa262f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20260709 12:58:28.947973  1116 debug-util.cc:398] Leaking SignalData structure 0x7b08001c1e40 after lost signal to thread 434
W20260709 12:58:28.949470  1116 debug-util.cc:398] Leaking SignalData structure 0x7b08001dcae0 after lost signal to thread 1027
W20260709 12:58:28.951097  1116 debug-util.cc:398] Leaking SignalData structure 0x7b0800129560 after lost signal to thread 1119
W20260709 12:58:28.952368  1116 debug-util.cc:398] Leaking SignalData structure 0x7b08000a40c0 after lost signal to thread 1195
W20260709 12:58:28.953945  1116 debug-util.cc:398] Leaking SignalData structure 0x7b08000ceec0 after lost signal to thread 1269
I20260709 12:58:29.074661   431 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:58:29.085911   431 tablet_server.cc:179] TabletServer@127.0.107.193:0 shutting down...
W20260709 12:58:29.104213  1133 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 7 similar messages]
W20260709 12:58:29.112380  1133 consensus_peers.cc:597] T 759e30de7d7b4bc6980dbf8c4ba8031f P b3afae27e8744ee6921cd67f805e3e4a -> Peer d88b316b7ecd49f29219068c5fe51a26 (127.0.107.193:38591): Couldn't send request to peer d88b316b7ecd49f29219068c5fe51a26. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:58:29.137617   431 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:58:29.139834  1168 raft_consensus.cc:1275] T 2cde8b0b8c5c46e78f405744f084c7c6 P b3afae27e8744ee6921cd67f805e3e4a [term 4 FOLLOWER]: Refusing update from remote peer d0e95822debe474ba6a700c661aa262f: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:58:29.140408   431 tablet_replica.cc:333] T 759e30de7d7b4bc6980dbf8c4ba8031f P d88b316b7ecd49f29219068c5fe51a26: stopping tablet replica
W20260709 12:58:29.141214  1208 consensus_peers.cc:597] T df94d2062ca84496bc46321e6e4012fc P d0e95822debe474ba6a700c661aa262f -> Peer d88b316b7ecd49f29219068c5fe51a26 (127.0.107.193:38591): Couldn't send request to peer d88b316b7ecd49f29219068c5fe51a26. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:58:29.140503  1243 raft_consensus.cc:1275] T d87ed5c1a25e40f8b7aa82d33ca30bf1 P d0e95822debe474ba6a700c661aa262f [term 4 FOLLOWER]: Refusing update from remote peer b3afae27e8744ee6921cd67f805e3e4a: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:58:29.141570  1435 consensus_queue.cc:1048] T 2cde8b0b8c5c46e78f405744f084c7c6 P d0e95822debe474ba6a700c661aa262f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:58:29.142060   431 raft_consensus.cc:2243] T 759e30de7d7b4bc6980dbf8c4ba8031f P d88b316b7ecd49f29219068c5fe51a26 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:58:29.142941   431 raft_consensus.cc:2272] T 759e30de7d7b4bc6980dbf8c4ba8031f P d88b316b7ecd49f29219068c5fe51a26 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:58:29.143383  1440 consensus_queue.cc:1048] T d87ed5c1a25e40f8b7aa82d33ca30bf1 P b3afae27e8744ee6921cd67f805e3e4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:58:29.149220   431 tablet_replica.cc:333] T c6480f4fe1b54658b39e73baa7c719fd P d88b316b7ecd49f29219068c5fe51a26: stopping tablet replica
I20260709 12:58:29.150558   431 raft_consensus.cc:2243] T c6480f4fe1b54658b39e73baa7c719fd P d88b316b7ecd49f29219068c5fe51a26 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:58:29.151316   431 raft_consensus.cc:2272] T c6480f4fe1b54658b39e73baa7c719fd P d88b316b7ecd49f29219068c5fe51a26 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:58:29.155238   431 tablet_replica.cc:333] T 8ee1d07b2c9444f8ad28921d69d297bf P d88b316b7ecd49f29219068c5fe51a26: stopping tablet replica
I20260709 12:58:29.156271   431 raft_consensus.cc:2243] T 8ee1d07b2c9444f8ad28921d69d297bf P d88b316b7ecd49f29219068c5fe51a26 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:58:29.158069   431 raft_consensus.cc:2272] T 8ee1d07b2c9444f8ad28921d69d297bf P d88b316b7ecd49f29219068c5fe51a26 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:58:29.167115   431 tablet_replica.cc:333] T 73e09365f8cd4513a4600030950af811 P d88b316b7ecd49f29219068c5fe51a26: stopping tablet replica
I20260709 12:58:29.168874   431 raft_consensus.cc:2243] T 73e09365f8cd4513a4600030950af811 P d88b316b7ecd49f29219068c5fe51a26 [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:58:29.170173   431 raft_consensus.cc:2272] T 73e09365f8cd4513a4600030950af811 P d88b316b7ecd49f29219068c5fe51a26 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:58:29.173332   431 tablet_replica.cc:333] T 2cde8b0b8c5c46e78f405744f084c7c6 P d88b316b7ecd49f29219068c5fe51a26: stopping tablet replica
I20260709 12:58:29.174561   431 raft_consensus.cc:2243] T 2cde8b0b8c5c46e78f405744f084c7c6 P d88b316b7ecd49f29219068c5fe51a26 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 12:58:29.175218   431 raft_consensus.cc:2272] T 2cde8b0b8c5c46e78f405744f084c7c6 P d88b316b7ecd49f29219068c5fe51a26 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:58:29.178655   431 tablet_replica.cc:333] T df94d2062ca84496bc46321e6e4012fc P d88b316b7ecd49f29219068c5fe51a26: stopping tablet replica
I20260709 12:58:29.179683   431 raft_consensus.cc:2243] T df94d2062ca84496bc46321e6e4012fc P d88b316b7ecd49f29219068c5fe51a26 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:58:29.180651   431 raft_consensus.cc:2272] T df94d2062ca84496bc46321e6e4012fc P d88b316b7ecd49f29219068c5fe51a26 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:58:29.184543   431 tablet_replica.cc:333] T 6198a4398a3140e8ad9f1a158956fe3e P d88b316b7ecd49f29219068c5fe51a26: stopping tablet replica
I20260709 12:58:29.185352   431 raft_consensus.cc:2243] T 6198a4398a3140e8ad9f1a158956fe3e P d88b316b7ecd49f29219068c5fe51a26 [term 3 FOLLOWER]: Raft consensus shutting down.
W20260709 12:58:29.185547  1133 consensus_peers.cc:597] T d87ed5c1a25e40f8b7aa82d33ca30bf1 P b3afae27e8744ee6921cd67f805e3e4a -> Peer d88b316b7ecd49f29219068c5fe51a26 (127.0.107.193:38591): Couldn't send request to peer d88b316b7ecd49f29219068c5fe51a26. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:58:29.186853   431 raft_consensus.cc:2272] T 6198a4398a3140e8ad9f1a158956fe3e P d88b316b7ecd49f29219068c5fe51a26 [term 3 FOLLOWER]: Raft consensus is shut down!
W20260709 12:58:29.187783  1208 consensus_peers.cc:597] T 2cde8b0b8c5c46e78f405744f084c7c6 P d0e95822debe474ba6a700c661aa262f -> Peer d88b316b7ecd49f29219068c5fe51a26 (127.0.107.193:38591): Couldn't send request to peer d88b316b7ecd49f29219068c5fe51a26. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:58:29.189433   431 tablet_replica.cc:333] T 2685bf6c4fd043a481d1d05136dbbd5e P d88b316b7ecd49f29219068c5fe51a26: stopping tablet replica
I20260709 12:58:29.190217   431 raft_consensus.cc:2243] T 2685bf6c4fd043a481d1d05136dbbd5e P d88b316b7ecd49f29219068c5fe51a26 [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:58:29.191414   431 raft_consensus.cc:2272] T 2685bf6c4fd043a481d1d05136dbbd5e P d88b316b7ecd49f29219068c5fe51a26 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:58:29.193616   431 tablet_replica.cc:333] T 1935c9164755416a965cccf80900d941 P d88b316b7ecd49f29219068c5fe51a26: stopping tablet replica
I20260709 12:58:29.194116   431 raft_consensus.cc:2243] T 1935c9164755416a965cccf80900d941 P d88b316b7ecd49f29219068c5fe51a26 [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:58:29.195238   431 raft_consensus.cc:2272] T 1935c9164755416a965cccf80900d941 P d88b316b7ecd49f29219068c5fe51a26 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:58:29.198014   431 tablet_replica.cc:333] T d87ed5c1a25e40f8b7aa82d33ca30bf1 P d88b316b7ecd49f29219068c5fe51a26: stopping tablet replica
I20260709 12:58:29.198673   431 raft_consensus.cc:2243] T d87ed5c1a25e40f8b7aa82d33ca30bf1 P d88b316b7ecd49f29219068c5fe51a26 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 12:58:29.199301   431 raft_consensus.cc:2272] T d87ed5c1a25e40f8b7aa82d33ca30bf1 P d88b316b7ecd49f29219068c5fe51a26 [term 4 FOLLOWER]: Raft consensus is shut down!
W20260709 12:58:29.471380  1133 consensus_peers.cc:597] T c6480f4fe1b54658b39e73baa7c719fd P b3afae27e8744ee6921cd67f805e3e4a -> Peer d88b316b7ecd49f29219068c5fe51a26 (127.0.107.193:38591): Couldn't send request to peer d88b316b7ecd49f29219068c5fe51a26. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:38591: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:58:29.511420  1208 consensus_peers.cc:597] T 6198a4398a3140e8ad9f1a158956fe3e P d0e95822debe474ba6a700c661aa262f -> Peer d88b316b7ecd49f29219068c5fe51a26 (127.0.107.193:38591): Couldn't send request to peer d88b316b7ecd49f29219068c5fe51a26. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:38591: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:58:30.120314  1435 raft_consensus.cc:493] T 8ee1d07b2c9444f8ad28921d69d297bf P d0e95822debe474ba6a700c661aa262f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d88b316b7ecd49f29219068c5fe51a26)
I20260709 12:58:30.121294  1435 raft_consensus.cc:515] T 8ee1d07b2c9444f8ad28921d69d297bf P d0e95822debe474ba6a700c661aa262f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:30.123809  1435 leader_election.cc:290] T 8ee1d07b2c9444f8ad28921d69d297bf P d0e95822debe474ba6a700c661aa262f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d88b316b7ecd49f29219068c5fe51a26 (127.0.107.193:38591), b3afae27e8744ee6921cd67f805e3e4a (127.0.107.194:36785)
I20260709 12:58:30.125195  1168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ee1d07b2c9444f8ad28921d69d297bf" candidate_uuid: "d0e95822debe474ba6a700c661aa262f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b3afae27e8744ee6921cd67f805e3e4a" is_pre_election: true
W20260709 12:58:30.127528  1208 leader_election.cc:336] T 8ee1d07b2c9444f8ad28921d69d297bf P d0e95822debe474ba6a700c661aa262f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d88b316b7ecd49f29219068c5fe51a26 (127.0.107.193:38591): Network error: Client connection negotiation failed: client connection to 127.0.107.193:38591: connect: Connection refused (error 111)
I20260709 12:58:30.128026  1208 leader_election.cc:304] T 8ee1d07b2c9444f8ad28921d69d297bf P d0e95822debe474ba6a700c661aa262f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0e95822debe474ba6a700c661aa262f; no voters: b3afae27e8744ee6921cd67f805e3e4a, d88b316b7ecd49f29219068c5fe51a26
I20260709 12:58:30.128968  1435 raft_consensus.cc:2749] T 8ee1d07b2c9444f8ad28921d69d297bf P d0e95822debe474ba6a700c661aa262f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:58:30.173750  1435 raft_consensus.cc:493] T 1935c9164755416a965cccf80900d941 P d0e95822debe474ba6a700c661aa262f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d88b316b7ecd49f29219068c5fe51a26)
I20260709 12:58:30.174654  1435 raft_consensus.cc:515] T 1935c9164755416a965cccf80900d941 P d0e95822debe474ba6a700c661aa262f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:30.177069  1435 leader_election.cc:290] T 1935c9164755416a965cccf80900d941 P d0e95822debe474ba6a700c661aa262f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d88b316b7ecd49f29219068c5fe51a26 (127.0.107.193:38591), b3afae27e8744ee6921cd67f805e3e4a (127.0.107.194:36785)
I20260709 12:58:30.178275  1168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1935c9164755416a965cccf80900d941" candidate_uuid: "d0e95822debe474ba6a700c661aa262f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "b3afae27e8744ee6921cd67f805e3e4a" is_pre_election: true
W20260709 12:58:30.181345  1208 leader_election.cc:336] T 1935c9164755416a965cccf80900d941 P d0e95822debe474ba6a700c661aa262f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer d88b316b7ecd49f29219068c5fe51a26 (127.0.107.193:38591): Network error: Client connection negotiation failed: client connection to 127.0.107.193:38591: connect: Connection refused (error 111)
I20260709 12:58:30.181774  1208 leader_election.cc:304] T 1935c9164755416a965cccf80900d941 P d0e95822debe474ba6a700c661aa262f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0e95822debe474ba6a700c661aa262f; no voters: b3afae27e8744ee6921cd67f805e3e4a, d88b316b7ecd49f29219068c5fe51a26
I20260709 12:58:30.182456  1435 raft_consensus.cc:2749] T 1935c9164755416a965cccf80900d941 P d0e95822debe474ba6a700c661aa262f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260709 12:58:30.224478   431 thread.cc:527] Waited for 1000ms trying to join with diag-logger (tid 1116)
I20260709 12:58:30.426563  1435 raft_consensus.cc:493] T 2685bf6c4fd043a481d1d05136dbbd5e P d0e95822debe474ba6a700c661aa262f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d88b316b7ecd49f29219068c5fe51a26)
I20260709 12:58:30.427397  1435 raft_consensus.cc:515] T 2685bf6c4fd043a481d1d05136dbbd5e P d0e95822debe474ba6a700c661aa262f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:30.430141  1435 leader_election.cc:290] T 2685bf6c4fd043a481d1d05136dbbd5e P d0e95822debe474ba6a700c661aa262f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d88b316b7ecd49f29219068c5fe51a26 (127.0.107.193:38591), b3afae27e8744ee6921cd67f805e3e4a (127.0.107.194:36785)
I20260709 12:58:30.431631  1168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2685bf6c4fd043a481d1d05136dbbd5e" candidate_uuid: "d0e95822debe474ba6a700c661aa262f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "b3afae27e8744ee6921cd67f805e3e4a" is_pre_election: true
W20260709 12:58:30.434968  1208 leader_election.cc:336] T 2685bf6c4fd043a481d1d05136dbbd5e P d0e95822debe474ba6a700c661aa262f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer d88b316b7ecd49f29219068c5fe51a26 (127.0.107.193:38591): Network error: Client connection negotiation failed: client connection to 127.0.107.193:38591: connect: Connection refused (error 111)
I20260709 12:58:30.435403  1208 leader_election.cc:304] T 2685bf6c4fd043a481d1d05136dbbd5e P d0e95822debe474ba6a700c661aa262f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0e95822debe474ba6a700c661aa262f; no voters: b3afae27e8744ee6921cd67f805e3e4a, d88b316b7ecd49f29219068c5fe51a26
I20260709 12:58:30.436134  1435 raft_consensus.cc:2749] T 2685bf6c4fd043a481d1d05136dbbd5e P d0e95822debe474ba6a700c661aa262f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:58:30.506860  1440 raft_consensus.cc:493] T 8ee1d07b2c9444f8ad28921d69d297bf P b3afae27e8744ee6921cd67f805e3e4a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d88b316b7ecd49f29219068c5fe51a26)
I20260709 12:58:30.507526  1440 raft_consensus.cc:515] T 8ee1d07b2c9444f8ad28921d69d297bf P b3afae27e8744ee6921cd67f805e3e4a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:30.510519  1440 leader_election.cc:290] T 8ee1d07b2c9444f8ad28921d69d297bf P b3afae27e8744ee6921cd67f805e3e4a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d88b316b7ecd49f29219068c5fe51a26 (127.0.107.193:38591), d0e95822debe474ba6a700c661aa262f (127.0.107.195:40863)
I20260709 12:58:30.511036  1440 raft_consensus.cc:493] T 2685bf6c4fd043a481d1d05136dbbd5e P b3afae27e8744ee6921cd67f805e3e4a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d88b316b7ecd49f29219068c5fe51a26)
I20260709 12:58:30.511551  1440 raft_consensus.cc:515] T 2685bf6c4fd043a481d1d05136dbbd5e P b3afae27e8744ee6921cd67f805e3e4a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:30.514217  1243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ee1d07b2c9444f8ad28921d69d297bf" candidate_uuid: "b3afae27e8744ee6921cd67f805e3e4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d0e95822debe474ba6a700c661aa262f" is_pre_election: true
I20260709 12:58:30.514488  1440 leader_election.cc:290] T 2685bf6c4fd043a481d1d05136dbbd5e P b3afae27e8744ee6921cd67f805e3e4a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d88b316b7ecd49f29219068c5fe51a26 (127.0.107.193:38591), d0e95822debe474ba6a700c661aa262f (127.0.107.195:40863)
I20260709 12:58:30.514943  1243 raft_consensus.cc:2468] T 8ee1d07b2c9444f8ad28921d69d297bf P d0e95822debe474ba6a700c661aa262f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3afae27e8744ee6921cd67f805e3e4a in term 1.
I20260709 12:58:30.517233  1136 leader_election.cc:304] T 8ee1d07b2c9444f8ad28921d69d297bf P b3afae27e8744ee6921cd67f805e3e4a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3afae27e8744ee6921cd67f805e3e4a, d0e95822debe474ba6a700c661aa262f; no voters: 
I20260709 12:58:30.518376  1458 raft_consensus.cc:2804] T 8ee1d07b2c9444f8ad28921d69d297bf P b3afae27e8744ee6921cd67f805e3e4a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:58:30.518675  1243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2685bf6c4fd043a481d1d05136dbbd5e" candidate_uuid: "b3afae27e8744ee6921cd67f805e3e4a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d0e95822debe474ba6a700c661aa262f" is_pre_election: true
I20260709 12:58:30.518891  1458 raft_consensus.cc:493] T 8ee1d07b2c9444f8ad28921d69d297bf P b3afae27e8744ee6921cd67f805e3e4a [term 1 FOLLOWER]: Starting leader election (detected failure of leader d88b316b7ecd49f29219068c5fe51a26)
I20260709 12:58:30.519328  1458 raft_consensus.cc:3060] T 8ee1d07b2c9444f8ad28921d69d297bf P b3afae27e8744ee6921cd67f805e3e4a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:58:30.519373  1243 raft_consensus.cc:2468] T 2685bf6c4fd043a481d1d05136dbbd5e P d0e95822debe474ba6a700c661aa262f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3afae27e8744ee6921cd67f805e3e4a in term 3.
I20260709 12:58:30.521054  1136 leader_election.cc:304] T 2685bf6c4fd043a481d1d05136dbbd5e P b3afae27e8744ee6921cd67f805e3e4a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3afae27e8744ee6921cd67f805e3e4a, d0e95822debe474ba6a700c661aa262f; no voters: 
I20260709 12:58:30.522040  1440 raft_consensus.cc:2804] T 2685bf6c4fd043a481d1d05136dbbd5e P b3afae27e8744ee6921cd67f805e3e4a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:58:30.522550  1440 raft_consensus.cc:493] T 2685bf6c4fd043a481d1d05136dbbd5e P b3afae27e8744ee6921cd67f805e3e4a [term 3 FOLLOWER]: Starting leader election (detected failure of leader d88b316b7ecd49f29219068c5fe51a26)
I20260709 12:58:30.523000  1440 raft_consensus.cc:3060] T 2685bf6c4fd043a481d1d05136dbbd5e P b3afae27e8744ee6921cd67f805e3e4a [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:58:30.532289  1440 raft_consensus.cc:515] T 2685bf6c4fd043a481d1d05136dbbd5e P b3afae27e8744ee6921cd67f805e3e4a [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:30.532035  1458 raft_consensus.cc:515] T 8ee1d07b2c9444f8ad28921d69d297bf P b3afae27e8744ee6921cd67f805e3e4a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:30.535503  1458 leader_election.cc:290] T 8ee1d07b2c9444f8ad28921d69d297bf P b3afae27e8744ee6921cd67f805e3e4a [CANDIDATE]: Term 2 election: Requested vote from peers d88b316b7ecd49f29219068c5fe51a26 (127.0.107.193:38591), d0e95822debe474ba6a700c661aa262f (127.0.107.195:40863)
I20260709 12:58:30.536391  1458 raft_consensus.cc:493] T 73e09365f8cd4513a4600030950af811 P b3afae27e8744ee6921cd67f805e3e4a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d88b316b7ecd49f29219068c5fe51a26)
I20260709 12:58:30.537119  1243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2685bf6c4fd043a481d1d05136dbbd5e" candidate_uuid: "b3afae27e8744ee6921cd67f805e3e4a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d0e95822debe474ba6a700c661aa262f"
I20260709 12:58:30.537063  1458 raft_consensus.cc:515] T 73e09365f8cd4513a4600030950af811 P b3afae27e8744ee6921cd67f805e3e4a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:30.537837  1243 raft_consensus.cc:3060] T 2685bf6c4fd043a481d1d05136dbbd5e P d0e95822debe474ba6a700c661aa262f [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:58:30.540098  1458 leader_election.cc:290] T 73e09365f8cd4513a4600030950af811 P b3afae27e8744ee6921cd67f805e3e4a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d88b316b7ecd49f29219068c5fe51a26 (127.0.107.193:38591), d0e95822debe474ba6a700c661aa262f (127.0.107.195:40863)
W20260709 12:58:30.541327  1133 leader_election.cc:336] T 8ee1d07b2c9444f8ad28921d69d297bf P b3afae27e8744ee6921cd67f805e3e4a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d88b316b7ecd49f29219068c5fe51a26 (127.0.107.193:38591): Network error: Client connection negotiation failed: client connection to 127.0.107.193:38591: connect: Connection refused (error 111)
W20260709 12:58:30.542169  1133 leader_election.cc:336] T 2685bf6c4fd043a481d1d05136dbbd5e P b3afae27e8744ee6921cd67f805e3e4a [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer d88b316b7ecd49f29219068c5fe51a26 (127.0.107.193:38591): Network error: Client connection negotiation failed: client connection to 127.0.107.193:38591: connect: Connection refused (error 111)
I20260709 12:58:30.542688  1245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ee1d07b2c9444f8ad28921d69d297bf" candidate_uuid: "b3afae27e8744ee6921cd67f805e3e4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d0e95822debe474ba6a700c661aa262f"
I20260709 12:58:30.542948  1244 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73e09365f8cd4513a4600030950af811" candidate_uuid: "b3afae27e8744ee6921cd67f805e3e4a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d0e95822debe474ba6a700c661aa262f" is_pre_election: true
I20260709 12:58:30.543489  1245 raft_consensus.cc:3060] T 8ee1d07b2c9444f8ad28921d69d297bf P d0e95822debe474ba6a700c661aa262f [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:58:30.545847  1435 raft_consensus.cc:493] T 73e09365f8cd4513a4600030950af811 P d0e95822debe474ba6a700c661aa262f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d88b316b7ecd49f29219068c5fe51a26)
I20260709 12:58:30.546397  1435 raft_consensus.cc:515] T 73e09365f8cd4513a4600030950af811 P d0e95822debe474ba6a700c661aa262f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:30.551781  1440 leader_election.cc:290] T 2685bf6c4fd043a481d1d05136dbbd5e P b3afae27e8744ee6921cd67f805e3e4a [CANDIDATE]: Term 4 election: Requested vote from peers d88b316b7ecd49f29219068c5fe51a26 (127.0.107.193:38591), d0e95822debe474ba6a700c661aa262f (127.0.107.195:40863)
I20260709 12:58:30.554620  1245 raft_consensus.cc:2468] T 8ee1d07b2c9444f8ad28921d69d297bf P d0e95822debe474ba6a700c661aa262f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3afae27e8744ee6921cd67f805e3e4a in term 2.
I20260709 12:58:30.554705  1168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73e09365f8cd4513a4600030950af811" candidate_uuid: "d0e95822debe474ba6a700c661aa262f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "b3afae27e8744ee6921cd67f805e3e4a" is_pre_election: true
I20260709 12:58:30.555356  1168 raft_consensus.cc:2468] T 73e09365f8cd4513a4600030950af811 P b3afae27e8744ee6921cd67f805e3e4a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0e95822debe474ba6a700c661aa262f in term 3.
I20260709 12:58:30.556092  1136 leader_election.cc:304] T 8ee1d07b2c9444f8ad28921d69d297bf P b3afae27e8744ee6921cd67f805e3e4a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3afae27e8744ee6921cd67f805e3e4a, d0e95822debe474ba6a700c661aa262f; no voters: 
I20260709 12:58:30.556496  1435 leader_election.cc:290] T 73e09365f8cd4513a4600030950af811 P d0e95822debe474ba6a700c661aa262f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d88b316b7ecd49f29219068c5fe51a26 (127.0.107.193:38591), b3afae27e8744ee6921cd67f805e3e4a (127.0.107.194:36785)
I20260709 12:58:30.556778  1209 leader_election.cc:304] T 73e09365f8cd4513a4600030950af811 P d0e95822debe474ba6a700c661aa262f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3afae27e8744ee6921cd67f805e3e4a, d0e95822debe474ba6a700c661aa262f; no voters: 
W20260709 12:58:30.559137  1208 leader_election.cc:336] T 73e09365f8cd4513a4600030950af811 P d0e95822debe474ba6a700c661aa262f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer d88b316b7ecd49f29219068c5fe51a26 (127.0.107.193:38591): Network error: Client connection negotiation failed: client connection to 127.0.107.193:38591: connect: Connection refused (error 111)
I20260709 12:58:30.559711  1435 raft_consensus.cc:2804] T 73e09365f8cd4513a4600030950af811 P d0e95822debe474ba6a700c661aa262f [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:58:30.560114  1435 raft_consensus.cc:493] T 73e09365f8cd4513a4600030950af811 P d0e95822debe474ba6a700c661aa262f [term 3 FOLLOWER]: Starting leader election (detected failure of leader d88b316b7ecd49f29219068c5fe51a26)
I20260709 12:58:30.560231  1440 raft_consensus.cc:2804] T 8ee1d07b2c9444f8ad28921d69d297bf P b3afae27e8744ee6921cd67f805e3e4a [term 2 FOLLOWER]: Leader election won for term 2
W20260709 12:58:30.560616  1133 leader_election.cc:336] T 2685bf6c4fd043a481d1d05136dbbd5e P b3afae27e8744ee6921cd67f805e3e4a [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer d88b316b7ecd49f29219068c5fe51a26 (127.0.107.193:38591): Network error: Client connection negotiation failed: client connection to 127.0.107.193:38591: connect: Connection refused (error 111)
I20260709 12:58:30.560858  1440 raft_consensus.cc:697] T 8ee1d07b2c9444f8ad28921d69d297bf P b3afae27e8744ee6921cd67f805e3e4a [term 2 LEADER]: Becoming Leader. State: Replica: b3afae27e8744ee6921cd67f805e3e4a, State: Running, Role: LEADER
W20260709 12:58:30.561396  1133 leader_election.cc:336] T 8ee1d07b2c9444f8ad28921d69d297bf P b3afae27e8744ee6921cd67f805e3e4a [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer d88b316b7ecd49f29219068c5fe51a26 (127.0.107.193:38591): Network error: Client connection negotiation failed: client connection to 127.0.107.193:38591: connect: Connection refused (error 111)
I20260709 12:58:30.560561  1435 raft_consensus.cc:3060] T 73e09365f8cd4513a4600030950af811 P d0e95822debe474ba6a700c661aa262f [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:58:30.562237  1133 leader_election.cc:336] T 73e09365f8cd4513a4600030950af811 P b3afae27e8744ee6921cd67f805e3e4a [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer d88b316b7ecd49f29219068c5fe51a26 (127.0.107.193:38591): Network error: Client connection negotiation failed: client connection to 127.0.107.193:38591: connect: Connection refused (error 111)
I20260709 12:58:30.562047  1440 consensus_queue.cc:237] T 8ee1d07b2c9444f8ad28921d69d297bf P b3afae27e8744ee6921cd67f805e3e4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:30.562812  1133 leader_election.cc:304] T 73e09365f8cd4513a4600030950af811 P b3afae27e8744ee6921cd67f805e3e4a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b3afae27e8744ee6921cd67f805e3e4a; no voters: d0e95822debe474ba6a700c661aa262f, d88b316b7ecd49f29219068c5fe51a26
I20260709 12:58:30.563848  1459 raft_consensus.cc:2749] T 73e09365f8cd4513a4600030950af811 P b3afae27e8744ee6921cd67f805e3e4a [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:58:30.567641  1243 raft_consensus.cc:2468] T 2685bf6c4fd043a481d1d05136dbbd5e P d0e95822debe474ba6a700c661aa262f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3afae27e8744ee6921cd67f805e3e4a in term 4.
I20260709 12:58:30.568985  1136 leader_election.cc:304] T 2685bf6c4fd043a481d1d05136dbbd5e P b3afae27e8744ee6921cd67f805e3e4a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b3afae27e8744ee6921cd67f805e3e4a, d0e95822debe474ba6a700c661aa262f; no voters: d88b316b7ecd49f29219068c5fe51a26
I20260709 12:58:30.569844  1459 raft_consensus.cc:2804] T 2685bf6c4fd043a481d1d05136dbbd5e P b3afae27e8744ee6921cd67f805e3e4a [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:58:30.570379  1459 raft_consensus.cc:697] T 2685bf6c4fd043a481d1d05136dbbd5e P b3afae27e8744ee6921cd67f805e3e4a [term 4 LEADER]: Becoming Leader. State: Replica: b3afae27e8744ee6921cd67f805e3e4a, State: Running, Role: LEADER
I20260709 12:58:30.570640  1435 raft_consensus.cc:515] T 73e09365f8cd4513a4600030950af811 P d0e95822debe474ba6a700c661aa262f [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:30.571391  1459 consensus_queue.cc:237] T 2685bf6c4fd043a481d1d05136dbbd5e P b3afae27e8744ee6921cd67f805e3e4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:30.579103   990 catalog_manager.cc:5697] T 8ee1d07b2c9444f8ad28921d69d297bf P b3afae27e8744ee6921cd67f805e3e4a reported cstate change: term changed from 1 to 2, leader changed from d88b316b7ecd49f29219068c5fe51a26 (127.0.107.193) to b3afae27e8744ee6921cd67f805e3e4a (127.0.107.194). New cstate: current_term: 2 leader_uuid: "b3afae27e8744ee6921cd67f805e3e4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } health_report { overall_health: UNKNOWN } } }
W20260709 12:58:30.580515  1208 leader_election.cc:336] T 73e09365f8cd4513a4600030950af811 P d0e95822debe474ba6a700c661aa262f [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer d88b316b7ecd49f29219068c5fe51a26 (127.0.107.193:38591): Network error: Client connection negotiation failed: client connection to 127.0.107.193:38591: connect: Connection refused (error 111)
I20260709 12:58:30.581475  1435 leader_election.cc:290] T 73e09365f8cd4513a4600030950af811 P d0e95822debe474ba6a700c661aa262f [CANDIDATE]: Term 4 election: Requested vote from peers d88b316b7ecd49f29219068c5fe51a26 (127.0.107.193:38591), b3afae27e8744ee6921cd67f805e3e4a (127.0.107.194:36785)
I20260709 12:58:30.583189  1168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73e09365f8cd4513a4600030950af811" candidate_uuid: "d0e95822debe474ba6a700c661aa262f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "b3afae27e8744ee6921cd67f805e3e4a"
I20260709 12:58:30.584090  1168 raft_consensus.cc:3060] T 73e09365f8cd4513a4600030950af811 P b3afae27e8744ee6921cd67f805e3e4a [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:58:30.590926  1440 raft_consensus.cc:493] T 1935c9164755416a965cccf80900d941 P b3afae27e8744ee6921cd67f805e3e4a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d88b316b7ecd49f29219068c5fe51a26)
I20260709 12:58:30.591668  1440 raft_consensus.cc:515] T 1935c9164755416a965cccf80900d941 P b3afae27e8744ee6921cd67f805e3e4a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:30.594395  1440 leader_election.cc:290] T 1935c9164755416a965cccf80900d941 P b3afae27e8744ee6921cd67f805e3e4a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d88b316b7ecd49f29219068c5fe51a26 (127.0.107.193:38591), d0e95822debe474ba6a700c661aa262f (127.0.107.195:40863)
I20260709 12:58:30.595752  1243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1935c9164755416a965cccf80900d941" candidate_uuid: "b3afae27e8744ee6921cd67f805e3e4a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d0e95822debe474ba6a700c661aa262f" is_pre_election: true
I20260709 12:58:30.596469  1243 raft_consensus.cc:2468] T 1935c9164755416a965cccf80900d941 P d0e95822debe474ba6a700c661aa262f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3afae27e8744ee6921cd67f805e3e4a in term 3.
I20260709 12:58:30.596748  1168 raft_consensus.cc:2468] T 73e09365f8cd4513a4600030950af811 P b3afae27e8744ee6921cd67f805e3e4a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0e95822debe474ba6a700c661aa262f in term 4.
I20260709 12:58:30.597858  1136 leader_election.cc:304] T 1935c9164755416a965cccf80900d941 P b3afae27e8744ee6921cd67f805e3e4a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3afae27e8744ee6921cd67f805e3e4a, d0e95822debe474ba6a700c661aa262f; no voters: 
I20260709 12:58:30.599466  1209 leader_election.cc:304] T 73e09365f8cd4513a4600030950af811 P d0e95822debe474ba6a700c661aa262f [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b3afae27e8744ee6921cd67f805e3e4a, d0e95822debe474ba6a700c661aa262f; no voters: d88b316b7ecd49f29219068c5fe51a26
I20260709 12:58:30.600739  1435 raft_consensus.cc:2804] T 73e09365f8cd4513a4600030950af811 P d0e95822debe474ba6a700c661aa262f [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:58:30.602957  1435 raft_consensus.cc:697] T 73e09365f8cd4513a4600030950af811 P d0e95822debe474ba6a700c661aa262f [term 4 LEADER]: Becoming Leader. State: Replica: d0e95822debe474ba6a700c661aa262f, State: Running, Role: LEADER
I20260709 12:58:30.605273  1440 raft_consensus.cc:2804] T 1935c9164755416a965cccf80900d941 P b3afae27e8744ee6921cd67f805e3e4a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:58:30.605708  1440 raft_consensus.cc:493] T 1935c9164755416a965cccf80900d941 P b3afae27e8744ee6921cd67f805e3e4a [term 3 FOLLOWER]: Starting leader election (detected failure of leader d88b316b7ecd49f29219068c5fe51a26)
I20260709 12:58:30.606132  1440 raft_consensus.cc:3060] T 1935c9164755416a965cccf80900d941 P b3afae27e8744ee6921cd67f805e3e4a [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:58:30.604372  1435 consensus_queue.cc:237] T 73e09365f8cd4513a4600030950af811 P d0e95822debe474ba6a700c661aa262f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:30.618682  1440 raft_consensus.cc:515] T 1935c9164755416a965cccf80900d941 P b3afae27e8744ee6921cd67f805e3e4a [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
W20260709 12:58:30.619155  1133 leader_election.cc:336] T 1935c9164755416a965cccf80900d941 P b3afae27e8744ee6921cd67f805e3e4a [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer d88b316b7ecd49f29219068c5fe51a26 (127.0.107.193:38591): Network error: Client connection negotiation failed: client connection to 127.0.107.193:38591: connect: Connection refused (error 111)
I20260709 12:58:30.620996  1440 leader_election.cc:290] T 1935c9164755416a965cccf80900d941 P b3afae27e8744ee6921cd67f805e3e4a [CANDIDATE]: Term 4 election: Requested vote from peers d88b316b7ecd49f29219068c5fe51a26 (127.0.107.193:38591), d0e95822debe474ba6a700c661aa262f (127.0.107.195:40863)
I20260709 12:58:30.622761  1243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1935c9164755416a965cccf80900d941" candidate_uuid: "b3afae27e8744ee6921cd67f805e3e4a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d0e95822debe474ba6a700c661aa262f"
I20260709 12:58:30.623457  1243 raft_consensus.cc:3060] T 1935c9164755416a965cccf80900d941 P d0e95822debe474ba6a700c661aa262f [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:58:30.628844  1133 leader_election.cc:336] T 1935c9164755416a965cccf80900d941 P b3afae27e8744ee6921cd67f805e3e4a [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer d88b316b7ecd49f29219068c5fe51a26 (127.0.107.193:38591): Network error: Client connection negotiation failed: client connection to 127.0.107.193:38591: connect: Connection refused (error 111)
I20260709 12:58:30.632401   992 catalog_manager.cc:5697] T 73e09365f8cd4513a4600030950af811 P d0e95822debe474ba6a700c661aa262f reported cstate change: term changed from 3 to 4, leader changed from d88b316b7ecd49f29219068c5fe51a26 (127.0.107.193) to d0e95822debe474ba6a700c661aa262f (127.0.107.195). New cstate: current_term: 4 leader_uuid: "d0e95822debe474ba6a700c661aa262f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } health_report { overall_health: HEALTHY } } }
I20260709 12:58:30.635120  1243 raft_consensus.cc:2468] T 1935c9164755416a965cccf80900d941 P d0e95822debe474ba6a700c661aa262f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3afae27e8744ee6921cd67f805e3e4a in term 4.
I20260709 12:58:30.636713  1136 leader_election.cc:304] T 1935c9164755416a965cccf80900d941 P b3afae27e8744ee6921cd67f805e3e4a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b3afae27e8744ee6921cd67f805e3e4a, d0e95822debe474ba6a700c661aa262f; no voters: d88b316b7ecd49f29219068c5fe51a26
I20260709 12:58:30.637699  1440 raft_consensus.cc:2804] T 1935c9164755416a965cccf80900d941 P b3afae27e8744ee6921cd67f805e3e4a [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:58:30.638191  1440 raft_consensus.cc:697] T 1935c9164755416a965cccf80900d941 P b3afae27e8744ee6921cd67f805e3e4a [term 4 LEADER]: Becoming Leader. State: Replica: b3afae27e8744ee6921cd67f805e3e4a, State: Running, Role: LEADER
I20260709 12:58:30.639379  1440 consensus_queue.cc:237] T 1935c9164755416a965cccf80900d941 P b3afae27e8744ee6921cd67f805e3e4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } }
I20260709 12:58:30.641233   990 catalog_manager.cc:5697] T 2685bf6c4fd043a481d1d05136dbbd5e P b3afae27e8744ee6921cd67f805e3e4a reported cstate change: term changed from 3 to 4, leader changed from d88b316b7ecd49f29219068c5fe51a26 (127.0.107.193) to b3afae27e8744ee6921cd67f805e3e4a (127.0.107.194). New cstate: current_term: 4 leader_uuid: "b3afae27e8744ee6921cd67f805e3e4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } health_report { overall_health: UNKNOWN } } }
I20260709 12:58:30.656896   990 catalog_manager.cc:5697] T 1935c9164755416a965cccf80900d941 P b3afae27e8744ee6921cd67f805e3e4a reported cstate change: term changed from 3 to 4, leader changed from d88b316b7ecd49f29219068c5fe51a26 (127.0.107.193) to b3afae27e8744ee6921cd67f805e3e4a (127.0.107.194). New cstate: current_term: 4 leader_uuid: "b3afae27e8744ee6921cd67f805e3e4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88b316b7ecd49f29219068c5fe51a26" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 } health_report { overall_health: UNKNOWN } } }
W20260709 12:58:30.973304  1133 consensus_peers.cc:597] T 8ee1d07b2c9444f8ad28921d69d297bf P b3afae27e8744ee6921cd67f805e3e4a -> Peer d88b316b7ecd49f29219068c5fe51a26 (127.0.107.193:38591): Couldn't send request to peer d88b316b7ecd49f29219068c5fe51a26. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:38591: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:58:31.012604  1168 raft_consensus.cc:1275] T 73e09365f8cd4513a4600030950af811 P b3afae27e8744ee6921cd67f805e3e4a [term 4 FOLLOWER]: Refusing update from remote peer d0e95822debe474ba6a700c661aa262f: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:58:31.014055  1412 consensus_queue.cc:1048] T 73e09365f8cd4513a4600030950af811 P d0e95822debe474ba6a700c661aa262f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3afae27e8744ee6921cd67f805e3e4a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 36785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
W20260709 12:58:31.035534  1208 consensus_peers.cc:597] T 73e09365f8cd4513a4600030950af811 P d0e95822debe474ba6a700c661aa262f -> Peer d88b316b7ecd49f29219068c5fe51a26 (127.0.107.193:38591): Couldn't send request to peer d88b316b7ecd49f29219068c5fe51a26. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:38591: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:58:31.084117  1133 consensus_peers.cc:597] T 1935c9164755416a965cccf80900d941 P b3afae27e8744ee6921cd67f805e3e4a -> Peer d88b316b7ecd49f29219068c5fe51a26 (127.0.107.193:38591): Couldn't send request to peer d88b316b7ecd49f29219068c5fe51a26. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:38591: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:58:31.085173  1243 raft_consensus.cc:1275] T 2685bf6c4fd043a481d1d05136dbbd5e P d0e95822debe474ba6a700c661aa262f [term 4 FOLLOWER]: Refusing update from remote peer b3afae27e8744ee6921cd67f805e3e4a: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:58:31.086472  1459 consensus_queue.cc:1048] T 2685bf6c4fd043a481d1d05136dbbd5e P b3afae27e8744ee6921cd67f805e3e4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
W20260709 12:58:31.100373  1133 consensus_peers.cc:597] T 2685bf6c4fd043a481d1d05136dbbd5e P b3afae27e8744ee6921cd67f805e3e4a -> Peer d88b316b7ecd49f29219068c5fe51a26 (127.0.107.193:38591): Couldn't send request to peer d88b316b7ecd49f29219068c5fe51a26. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:38591: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:58:31.151190  1243 raft_consensus.cc:1275] T 1935c9164755416a965cccf80900d941 P d0e95822debe474ba6a700c661aa262f [term 4 FOLLOWER]: Refusing update from remote peer b3afae27e8744ee6921cd67f805e3e4a: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:58:31.151602  1245 raft_consensus.cc:1275] T 8ee1d07b2c9444f8ad28921d69d297bf P d0e95822debe474ba6a700c661aa262f [term 2 FOLLOWER]: Refusing update from remote peer b3afae27e8744ee6921cd67f805e3e4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:58:31.152681  1473 consensus_queue.cc:1048] T 1935c9164755416a965cccf80900d941 P b3afae27e8744ee6921cd67f805e3e4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:58:31.153059  1459 consensus_queue.cc:1048] T 8ee1d07b2c9444f8ad28921d69d297bf P b3afae27e8744ee6921cd67f805e3e4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0e95822debe474ba6a700c661aa262f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260709 12:58:31.225010   431 thread.cc:527] Waited for 2000ms trying to join with diag-logger (tid 1116)
W20260709 12:58:31.521062  1208 consensus_peers.cc:597] T df94d2062ca84496bc46321e6e4012fc P d0e95822debe474ba6a700c661aa262f -> Peer d88b316b7ecd49f29219068c5fe51a26 (127.0.107.193:38591): Couldn't send request to peer d88b316b7ecd49f29219068c5fe51a26. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:38591: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:58:31.542481  1133 consensus_peers.cc:597] T 759e30de7d7b4bc6980dbf8c4ba8031f P b3afae27e8744ee6921cd67f805e3e4a -> Peer d88b316b7ecd49f29219068c5fe51a26 (127.0.107.193:38591): Couldn't send request to peer d88b316b7ecd49f29219068c5fe51a26. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:38591: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:58:31.712018  1208 consensus_peers.cc:597] T 2cde8b0b8c5c46e78f405744f084c7c6 P d0e95822debe474ba6a700c661aa262f -> Peer d88b316b7ecd49f29219068c5fe51a26 (127.0.107.193:38591): Couldn't send request to peer d88b316b7ecd49f29219068c5fe51a26. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:38591: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:58:31.757681   431 tablet_server.cc:196] TabletServer@127.0.107.193:0 shutdown complete.
I20260709 12:58:31.784040   431 tablet_server.cc:179] TabletServer@127.0.107.194:0 shutting down...
I20260709 12:58:31.811606   431 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:58:31.812337   431 tablet_replica.cc:333] T df94d2062ca84496bc46321e6e4012fc P b3afae27e8744ee6921cd67f805e3e4a: stopping tablet replica
I20260709 12:58:31.813028   431 raft_consensus.cc:2243] T df94d2062ca84496bc46321e6e4012fc P b3afae27e8744ee6921cd67f805e3e4a [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:58:31.813515   431 raft_consensus.cc:2272] T df94d2062ca84496bc46321e6e4012fc P b3afae27e8744ee6921cd67f805e3e4a [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:58:31.815621   431 tablet_replica.cc:333] T d87ed5c1a25e40f8b7aa82d33ca30bf1 P b3afae27e8744ee6921cd67f805e3e4a: stopping tablet replica
I20260709 12:58:31.816123   431 raft_consensus.cc:2243] T d87ed5c1a25e40f8b7aa82d33ca30bf1 P b3afae27e8744ee6921cd67f805e3e4a [term 4 LEADER]: Raft consensus shutting down.
I20260709 12:58:31.816912   431 raft_consensus.cc:2272] T d87ed5c1a25e40f8b7aa82d33ca30bf1 P b3afae27e8744ee6921cd67f805e3e4a [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:58:31.818750   431 tablet_replica.cc:333] T 2cde8b0b8c5c46e78f405744f084c7c6 P b3afae27e8744ee6921cd67f805e3e4a: stopping tablet replica
I20260709 12:58:31.819224   431 raft_consensus.cc:2243] T 2cde8b0b8c5c46e78f405744f084c7c6 P b3afae27e8744ee6921cd67f805e3e4a [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 12:58:31.819608   431 raft_consensus.cc:2272] T 2cde8b0b8c5c46e78f405744f084c7c6 P b3afae27e8744ee6921cd67f805e3e4a [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:58:31.821303   431 tablet_replica.cc:333] T 759e30de7d7b4bc6980dbf8c4ba8031f P b3afae27e8744ee6921cd67f805e3e4a: stopping tablet replica
I20260709 12:58:31.821756   431 raft_consensus.cc:2243] T 759e30de7d7b4bc6980dbf8c4ba8031f P b3afae27e8744ee6921cd67f805e3e4a [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:58:31.822557   431 raft_consensus.cc:2272] T 759e30de7d7b4bc6980dbf8c4ba8031f P b3afae27e8744ee6921cd67f805e3e4a [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:58:31.824309   431 tablet_replica.cc:333] T 2685bf6c4fd043a481d1d05136dbbd5e P b3afae27e8744ee6921cd67f805e3e4a: stopping tablet replica
I20260709 12:58:31.824779   431 raft_consensus.cc:2243] T 2685bf6c4fd043a481d1d05136dbbd5e P b3afae27e8744ee6921cd67f805e3e4a [term 4 LEADER]: Raft consensus shutting down.
I20260709 12:58:31.825480   431 raft_consensus.cc:2272] T 2685bf6c4fd043a481d1d05136dbbd5e P b3afae27e8744ee6921cd67f805e3e4a [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:58:31.827723   431 tablet_replica.cc:333] T 1935c9164755416a965cccf80900d941 P b3afae27e8744ee6921cd67f805e3e4a: stopping tablet replica
I20260709 12:58:31.828246   431 raft_consensus.cc:2243] T 1935c9164755416a965cccf80900d941 P b3afae27e8744ee6921cd67f805e3e4a [term 4 LEADER]: Raft consensus shutting down.
I20260709 12:58:31.829093   431 raft_consensus.cc:2272] T 1935c9164755416a965cccf80900d941 P b3afae27e8744ee6921cd67f805e3e4a [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:58:31.831758   431 tablet_replica.cc:333] T 6198a4398a3140e8ad9f1a158956fe3e P b3afae27e8744ee6921cd67f805e3e4a: stopping tablet replica
I20260709 12:58:31.832264   431 raft_consensus.cc:2243] T 6198a4398a3140e8ad9f1a158956fe3e P b3afae27e8744ee6921cd67f805e3e4a [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:58:31.832682   431 raft_consensus.cc:2272] T 6198a4398a3140e8ad9f1a158956fe3e P b3afae27e8744ee6921cd67f805e3e4a [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:58:31.834406   431 tablet_replica.cc:333] T c6480f4fe1b54658b39e73baa7c719fd P b3afae27e8744ee6921cd67f805e3e4a: stopping tablet replica
I20260709 12:58:31.834961   431 raft_consensus.cc:2243] T c6480f4fe1b54658b39e73baa7c719fd P b3afae27e8744ee6921cd67f805e3e4a [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:58:31.835717   431 raft_consensus.cc:2272] T c6480f4fe1b54658b39e73baa7c719fd P b3afae27e8744ee6921cd67f805e3e4a [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:58:31.837438   431 tablet_replica.cc:333] T 8ee1d07b2c9444f8ad28921d69d297bf P b3afae27e8744ee6921cd67f805e3e4a: stopping tablet replica
I20260709 12:58:31.837888   431 raft_consensus.cc:2243] T 8ee1d07b2c9444f8ad28921d69d297bf P b3afae27e8744ee6921cd67f805e3e4a [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:58:31.838716   431 raft_consensus.cc:2272] T 8ee1d07b2c9444f8ad28921d69d297bf P b3afae27e8744ee6921cd67f805e3e4a [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:58:31.840579   431 tablet_replica.cc:333] T 73e09365f8cd4513a4600030950af811 P b3afae27e8744ee6921cd67f805e3e4a: stopping tablet replica
I20260709 12:58:31.841027   431 raft_consensus.cc:2243] T 73e09365f8cd4513a4600030950af811 P b3afae27e8744ee6921cd67f805e3e4a [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 12:58:31.841459   431 raft_consensus.cc:2272] T 73e09365f8cd4513a4600030950af811 P b3afae27e8744ee6921cd67f805e3e4a [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:58:31.872593   431 tablet_server.cc:196] TabletServer@127.0.107.194:0 shutdown complete.
I20260709 12:58:31.895264   431 tablet_server.cc:179] TabletServer@127.0.107.195:0 shutting down...
I20260709 12:58:31.930593   431 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:58:31.931571   431 tablet_replica.cc:333] T df94d2062ca84496bc46321e6e4012fc P d0e95822debe474ba6a700c661aa262f: stopping tablet replica
I20260709 12:58:31.932303   431 raft_consensus.cc:2243] T df94d2062ca84496bc46321e6e4012fc P d0e95822debe474ba6a700c661aa262f [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:58:31.933660   431 raft_consensus.cc:2272] T df94d2062ca84496bc46321e6e4012fc P d0e95822debe474ba6a700c661aa262f [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:58:31.937047   431 tablet_replica.cc:333] T 8ee1d07b2c9444f8ad28921d69d297bf P d0e95822debe474ba6a700c661aa262f: stopping tablet replica
I20260709 12:58:31.937896   431 raft_consensus.cc:2243] T 8ee1d07b2c9444f8ad28921d69d297bf P d0e95822debe474ba6a700c661aa262f [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:58:31.938537   431 raft_consensus.cc:2272] T 8ee1d07b2c9444f8ad28921d69d297bf P d0e95822debe474ba6a700c661aa262f [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:58:31.941124   431 tablet_replica.cc:333] T c6480f4fe1b54658b39e73baa7c719fd P d0e95822debe474ba6a700c661aa262f: stopping tablet replica
I20260709 12:58:31.941843   431 raft_consensus.cc:2243] T c6480f4fe1b54658b39e73baa7c719fd P d0e95822debe474ba6a700c661aa262f [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:58:31.942384   431 raft_consensus.cc:2272] T c6480f4fe1b54658b39e73baa7c719fd P d0e95822debe474ba6a700c661aa262f [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:58:31.944762   431 tablet_replica.cc:333] T 73e09365f8cd4513a4600030950af811 P d0e95822debe474ba6a700c661aa262f: stopping tablet replica
I20260709 12:58:31.945479   431 raft_consensus.cc:2243] T 73e09365f8cd4513a4600030950af811 P d0e95822debe474ba6a700c661aa262f [term 4 LEADER]: Raft consensus shutting down.
I20260709 12:58:31.946511   431 raft_consensus.cc:2272] T 73e09365f8cd4513a4600030950af811 P d0e95822debe474ba6a700c661aa262f [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:58:31.949496   431 tablet_replica.cc:333] T d87ed5c1a25e40f8b7aa82d33ca30bf1 P d0e95822debe474ba6a700c661aa262f: stopping tablet replica
I20260709 12:58:31.950268   431 raft_consensus.cc:2243] T d87ed5c1a25e40f8b7aa82d33ca30bf1 P d0e95822debe474ba6a700c661aa262f [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 12:58:31.950836   431 raft_consensus.cc:2272] T d87ed5c1a25e40f8b7aa82d33ca30bf1 P d0e95822debe474ba6a700c661aa262f [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:58:31.953493   431 tablet_replica.cc:333] T 2cde8b0b8c5c46e78f405744f084c7c6 P d0e95822debe474ba6a700c661aa262f: stopping tablet replica
I20260709 12:58:31.954147   431 raft_consensus.cc:2243] T 2cde8b0b8c5c46e78f405744f084c7c6 P d0e95822debe474ba6a700c661aa262f [term 4 LEADER]: Raft consensus shutting down.
I20260709 12:58:31.955082   431 raft_consensus.cc:2272] T 2cde8b0b8c5c46e78f405744f084c7c6 P d0e95822debe474ba6a700c661aa262f [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:58:31.956835   431 tablet_replica.cc:333] T 6198a4398a3140e8ad9f1a158956fe3e P d0e95822debe474ba6a700c661aa262f: stopping tablet replica
I20260709 12:58:31.957352   431 raft_consensus.cc:2243] T 6198a4398a3140e8ad9f1a158956fe3e P d0e95822debe474ba6a700c661aa262f [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:58:31.958144   431 raft_consensus.cc:2272] T 6198a4398a3140e8ad9f1a158956fe3e P d0e95822debe474ba6a700c661aa262f [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:58:31.960119   431 tablet_replica.cc:333] T 1935c9164755416a965cccf80900d941 P d0e95822debe474ba6a700c661aa262f: stopping tablet replica
I20260709 12:58:31.960660   431 raft_consensus.cc:2243] T 1935c9164755416a965cccf80900d941 P d0e95822debe474ba6a700c661aa262f [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 12:58:31.961110   431 raft_consensus.cc:2272] T 1935c9164755416a965cccf80900d941 P d0e95822debe474ba6a700c661aa262f [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:58:31.963553   431 tablet_replica.cc:333] T 2685bf6c4fd043a481d1d05136dbbd5e P d0e95822debe474ba6a700c661aa262f: stopping tablet replica
I20260709 12:58:31.964100   431 raft_consensus.cc:2243] T 2685bf6c4fd043a481d1d05136dbbd5e P d0e95822debe474ba6a700c661aa262f [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 12:58:31.964676   431 raft_consensus.cc:2272] T 2685bf6c4fd043a481d1d05136dbbd5e P d0e95822debe474ba6a700c661aa262f [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:58:31.968943   431 tablet_replica.cc:333] T 759e30de7d7b4bc6980dbf8c4ba8031f P d0e95822debe474ba6a700c661aa262f: stopping tablet replica
I20260709 12:58:31.969605   431 raft_consensus.cc:2243] T 759e30de7d7b4bc6980dbf8c4ba8031f P d0e95822debe474ba6a700c661aa262f [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:58:31.970204   431 raft_consensus.cc:2272] T 759e30de7d7b4bc6980dbf8c4ba8031f P d0e95822debe474ba6a700c661aa262f [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:58:32.001322   431 tablet_server.cc:196] TabletServer@127.0.107.195:0 shutdown complete.
I20260709 12:58:32.023254   431 master.cc:562] Master@127.0.107.254:33267 shutting down...
I20260709 12:58:32.046701   431 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 851f14c22aa944d2ac3a9e0031eea552 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:58:32.047358   431 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 851f14c22aa944d2ac3a9e0031eea552 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:58:32.047744   431 tablet_replica.cc:333] T 00000000000000000000000000000000 P 851f14c22aa944d2ac3a9e0031eea552: stopping tablet replica
I20260709 12:58:32.067989   431 master.cc:584] Master@127.0.107.254:33267 shutdown complete.
b3afae27e8744ee6921cd67f805e3e4a  4
d0e95822debe474ba6a700c661aa262f  1
d88b316b7ecd49f29219068c5fe51a26  5
[       OK ] LeaderRebalancerTest.FunctionalTestForNotDivided (13678 ms)
[ RUN      ] LeaderRebalancerTest.AddTserver
I20260709 12:58:32.114934   431 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.107.254:33643
I20260709 12:58:32.116259   431 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:58:32.122985  1478 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:58:32.123064  1479 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:58:32.123286  1481 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:58:32.125450   431 server_base.cc:1061] running on GCE node
I20260709 12:58:32.126428   431 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:58:32.126703   431 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:58:32.126871   431 hybrid_clock.cc:648] HybridClock initialized: now 1783601912126851 us; error 0 us; skew 500 ppm
I20260709 12:58:32.127424   431 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:58:32.129952   431 webserver.cc:533] Webserver started at http://127.0.107.254:45713/ using document root <none> and password file <none>
I20260709 12:58:32.130529   431 fs_manager.cc:362] Metadata directory not provided
I20260709 12:58:32.130749   431 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:58:32.131060   431 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:58:32.132308   431 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskakPCWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783601878574137-431-0/minicluster-data/master-0-root/instance:
uuid: "919f6392d4f44fd68987e2c5e7ec37ca"
format_stamp: "Formatted at 2026-07-09 12:58:32 on dist-test-slave-nvvq"
I20260709 12:58:32.136868   431 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.002s	sys 0.003s
I20260709 12:58:32.140329  1486 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:58:32.141086   431 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.001s	sys 0.000s
I20260709 12:58:32.141374   431 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskakPCWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783601878574137-431-0/minicluster-data/master-0-root
uuid: "919f6392d4f44fd68987e2c5e7ec37ca"
format_stamp: "Formatted at 2026-07-09 12:58:32 on dist-test-slave-nvvq"
I20260709 12:58:32.141685   431 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskakPCWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783601878574137-431-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskakPCWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783601878574137-431-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskakPCWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783601878574137-431-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:58:32.169219   431 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:58:32.170841   431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:58:32.215049   431 rpc_server.cc:307] RPC server started. Bound to: 127.0.107.254:33643
I20260709 12:58:32.215137  1537 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.107.254:33643 every 8 connection(s)
I20260709 12:58:32.219617  1538 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:32.230818  1538 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 919f6392d4f44fd68987e2c5e7ec37ca: Bootstrap starting.
I20260709 12:58:32.236825  1538 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 919f6392d4f44fd68987e2c5e7ec37ca: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:32.242096  1538 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 919f6392d4f44fd68987e2c5e7ec37ca: No bootstrap required, opened a new log
I20260709 12:58:32.244812  1538 raft_consensus.cc:359] T 00000000000000000000000000000000 P 919f6392d4f44fd68987e2c5e7ec37ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "919f6392d4f44fd68987e2c5e7ec37ca" member_type: VOTER }
I20260709 12:58:32.245263  1538 raft_consensus.cc:385] T 00000000000000000000000000000000 P 919f6392d4f44fd68987e2c5e7ec37ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:32.245533  1538 raft_consensus.cc:740] T 00000000000000000000000000000000 P 919f6392d4f44fd68987e2c5e7ec37ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 919f6392d4f44fd68987e2c5e7ec37ca, State: Initialized, Role: FOLLOWER
I20260709 12:58:32.246196  1538 consensus_queue.cc:260] T 00000000000000000000000000000000 P 919f6392d4f44fd68987e2c5e7ec37ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "919f6392d4f44fd68987e2c5e7ec37ca" member_type: VOTER }
I20260709 12:58:32.246766  1538 raft_consensus.cc:399] T 00000000000000000000000000000000 P 919f6392d4f44fd68987e2c5e7ec37ca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260709 12:58:32.247099  1538 raft_consensus.cc:493] T 00000000000000000000000000000000 P 919f6392d4f44fd68987e2c5e7ec37ca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260709 12:58:32.247398  1538 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 919f6392d4f44fd68987e2c5e7ec37ca [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:32.252080  1538 raft_consensus.cc:515] T 00000000000000000000000000000000 P 919f6392d4f44fd68987e2c5e7ec37ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "919f6392d4f44fd68987e2c5e7ec37ca" member_type: VOTER }
I20260709 12:58:32.252596  1538 leader_election.cc:304] T 00000000000000000000000000000000 P 919f6392d4f44fd68987e2c5e7ec37ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 919f6392d4f44fd68987e2c5e7ec37ca; no voters: 
I20260709 12:58:32.253856  1538 leader_election.cc:290] T 00000000000000000000000000000000 P 919f6392d4f44fd68987e2c5e7ec37ca [CANDIDATE]: Term 1 election: Requested vote from peers 
I20260709 12:58:32.254220  1541 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 919f6392d4f44fd68987e2c5e7ec37ca [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:58:32.255774  1541 raft_consensus.cc:697] T 00000000000000000000000000000000 P 919f6392d4f44fd68987e2c5e7ec37ca [term 1 LEADER]: Becoming Leader. State: Replica: 919f6392d4f44fd68987e2c5e7ec37ca, State: Running, Role: LEADER
I20260709 12:58:32.256505  1541 consensus_queue.cc:237] T 00000000000000000000000000000000 P 919f6392d4f44fd68987e2c5e7ec37ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "919f6392d4f44fd68987e2c5e7ec37ca" member_type: VOTER }
I20260709 12:58:32.257201  1538 sys_catalog.cc:565] T 00000000000000000000000000000000 P 919f6392d4f44fd68987e2c5e7ec37ca [sys.catalog]: configured and running, proceeding with master startup.
I20260709 12:58:32.261781  1543 sys_catalog.cc:455] T 00000000000000000000000000000000 P 919f6392d4f44fd68987e2c5e7ec37ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader 919f6392d4f44fd68987e2c5e7ec37ca. Latest consensus state: current_term: 1 leader_uuid: "919f6392d4f44fd68987e2c5e7ec37ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "919f6392d4f44fd68987e2c5e7ec37ca" member_type: VOTER } }
I20260709 12:58:32.261916  1542 sys_catalog.cc:455] T 00000000000000000000000000000000 P 919f6392d4f44fd68987e2c5e7ec37ca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "919f6392d4f44fd68987e2c5e7ec37ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "919f6392d4f44fd68987e2c5e7ec37ca" member_type: VOTER } }
I20260709 12:58:32.262420  1543 sys_catalog.cc:458] T 00000000000000000000000000000000 P 919f6392d4f44fd68987e2c5e7ec37ca [sys.catalog]: This master's current role is: LEADER
I20260709 12:58:32.262560  1542 sys_catalog.cc:458] T 00000000000000000000000000000000 P 919f6392d4f44fd68987e2c5e7ec37ca [sys.catalog]: This master's current role is: LEADER
I20260709 12:58:32.265542  1548 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260709 12:58:32.270941  1548 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260709 12:58:32.276652   431 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260709 12:58:32.281102  1548 catalog_manager.cc:1383] Generated new cluster ID: a5667b6c58bc46109aac67a1f102ef0d
I20260709 12:58:32.281378  1548 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260709 12:58:32.302347  1548 catalog_manager.cc:1406] Generated new certificate authority record
I20260709 12:58:32.303889  1548 catalog_manager.cc:1540] Loading token signing keys...
I20260709 12:58:32.324340  1548 catalog_manager.cc:6070] T 00000000000000000000000000000000 P 919f6392d4f44fd68987e2c5e7ec37ca: Generated new TSK 0
I20260709 12:58:32.325078  1548 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260709 12:58:32.346310   431 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:58:32.353703  1559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:58:32.354959  1560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:58:32.357443   431 server_base.cc:1061] running on GCE node
W20260709 12:58:32.357611  1562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:58:32.358814   431 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:58:32.359055   431 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:58:32.359239   431 hybrid_clock.cc:648] HybridClock initialized: now 1783601912359216 us; error 0 us; skew 500 ppm
I20260709 12:58:32.359836   431 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:58:32.362550   431 webserver.cc:533] Webserver started at http://127.0.107.193:42923/ using document root <none> and password file <none>
I20260709 12:58:32.363090   431 fs_manager.cc:362] Metadata directory not provided
I20260709 12:58:32.363306   431 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:58:32.363602   431 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:58:32.364879   431 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskakPCWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783601878574137-431-0/minicluster-data/ts-0-root/instance:
uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
format_stamp: "Formatted at 2026-07-09 12:58:32 on dist-test-slave-nvvq"
I20260709 12:58:32.369897   431 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.007s	sys 0.000s
I20260709 12:58:32.373705  1567 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:58:32.374601   431 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.003s	sys 0.000s
I20260709 12:58:32.374923   431 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskakPCWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783601878574137-431-0/minicluster-data/ts-0-root
uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
format_stamp: "Formatted at 2026-07-09 12:58:32 on dist-test-slave-nvvq"
I20260709 12:58:32.375263   431 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskakPCWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783601878574137-431-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskakPCWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783601878574137-431-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskakPCWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783601878574137-431-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:58:32.398183   431 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:58:32.399879   431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:58:32.401675   431 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:58:32.404245   431 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:58:32.404471   431 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:58:32.404719   431 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:58:32.404915   431 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:58:32.455142   431 rpc_server.cc:307] RPC server started. Bound to: 127.0.107.193:46445
I20260709 12:58:32.455250  1629 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.107.193:46445 every 8 connection(s)
I20260709 12:58:32.460978   431 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:58:32.468958  1634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:58:32.473398  1635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:58:32.475247   431 server_base.cc:1061] running on GCE node
W20260709 12:58:32.476485  1637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:58:32.477897   431 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:58:32.478125   431 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:58:32.478374   431 hybrid_clock.cc:648] HybridClock initialized: now 1783601912478344 us; error 0 us; skew 500 ppm
I20260709 12:58:32.478724  1630 heartbeater.cc:344] Connected to a master server at 127.0.107.254:33643
I20260709 12:58:32.479142  1630 heartbeater.cc:461] Registering TS with master...
I20260709 12:58:32.479156   431 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:58:32.480305  1630 heartbeater.cc:507] Master 127.0.107.254:33643 requested a full tablet report, sending...
I20260709 12:58:32.482565   431 webserver.cc:533] Webserver started at http://127.0.107.194:36931/ using document root <none> and password file <none>
I20260709 12:58:32.483310   431 fs_manager.cc:362] Metadata directory not provided
I20260709 12:58:32.483626   431 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:58:32.483443  1503 ts_manager.cc:194] Registered new tserver with Master: 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445)
I20260709 12:58:32.484076   431 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:58:32.485904   431 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskakPCWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783601878574137-431-0/minicluster-data/ts-1-root/instance:
uuid: "816fadcea8fc4893882f5f9b9f22a03d"
format_stamp: "Formatted at 2026-07-09 12:58:32 on dist-test-slave-nvvq"
I20260709 12:58:32.486481  1503 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:35444
I20260709 12:58:32.491496   431 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.004s	sys 0.000s
I20260709 12:58:32.495289  1642 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:58:32.496109   431 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.001s
I20260709 12:58:32.496412   431 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskakPCWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783601878574137-431-0/minicluster-data/ts-1-root
uuid: "816fadcea8fc4893882f5f9b9f22a03d"
format_stamp: "Formatted at 2026-07-09 12:58:32 on dist-test-slave-nvvq"
I20260709 12:58:32.496717   431 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskakPCWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783601878574137-431-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskakPCWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783601878574137-431-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskakPCWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783601878574137-431-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:58:32.513121   431 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:58:32.514717   431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:58:32.516467   431 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:58:32.519035   431 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:58:32.519269   431 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:58:32.519533   431 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:58:32.519721   431 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:58:32.568675   431 rpc_server.cc:307] RPC server started. Bound to: 127.0.107.194:46261
I20260709 12:58:32.568816  1704 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.107.194:46261 every 8 connection(s)
I20260709 12:58:32.574468   431 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:58:32.582018  1708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:58:32.583966  1709 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:58:32.585289  1711 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:58:32.587621   431 server_base.cc:1061] running on GCE node
I20260709 12:58:32.588238  1705 heartbeater.cc:344] Connected to a master server at 127.0.107.254:33643
I20260709 12:58:32.588647  1705 heartbeater.cc:461] Registering TS with master...
I20260709 12:58:32.588738   431 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:58:32.589031   431 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:58:32.589214   431 hybrid_clock.cc:648] HybridClock initialized: now 1783601912589188 us; error 0 us; skew 500 ppm
I20260709 12:58:32.589622  1705 heartbeater.cc:507] Master 127.0.107.254:33643 requested a full tablet report, sending...
I20260709 12:58:32.589792   431 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:58:32.592132  1503 ts_manager.cc:194] Registered new tserver with Master: 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:58:32.592835   431 webserver.cc:533] Webserver started at http://127.0.107.195:42169/ using document root <none> and password file <none>
I20260709 12:58:32.593397   431 fs_manager.cc:362] Metadata directory not provided
I20260709 12:58:32.593633   431 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:58:32.593940   431 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:58:32.593909  1503 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:35458
I20260709 12:58:32.595659   431 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskakPCWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783601878574137-431-0/minicluster-data/ts-2-root/instance:
uuid: "af520b67945d443b901bbeca37cc5701"
format_stamp: "Formatted at 2026-07-09 12:58:32 on dist-test-slave-nvvq"
I20260709 12:58:32.600545   431 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.001s	sys 0.004s
I20260709 12:58:32.603924  1716 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:58:32.604820   431 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.003s	sys 0.000s
I20260709 12:58:32.605087   431 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskakPCWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783601878574137-431-0/minicluster-data/ts-2-root
uuid: "af520b67945d443b901bbeca37cc5701"
format_stamp: "Formatted at 2026-07-09 12:58:32 on dist-test-slave-nvvq"
I20260709 12:58:32.605332   431 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskakPCWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783601878574137-431-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskakPCWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783601878574137-431-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskakPCWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783601878574137-431-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:58:32.645031   431 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:58:32.646413   431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:58:32.648015   431 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:58:32.650579   431 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:58:32.650790   431 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:58:32.651079   431 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:58:32.651248   431 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:58:32.708487   431 rpc_server.cc:307] RPC server started. Bound to: 127.0.107.195:40693
I20260709 12:58:32.708690  1778 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.107.195:40693 every 8 connection(s)
I20260709 12:58:32.726536  1779 heartbeater.cc:344] Connected to a master server at 127.0.107.254:33643
I20260709 12:58:32.726980  1779 heartbeater.cc:461] Registering TS with master...
I20260709 12:58:32.727726  1779 heartbeater.cc:507] Master 127.0.107.254:33643 requested a full tablet report, sending...
I20260709 12:58:32.729776  1503 ts_manager.cc:194] Registered new tserver with Master: af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:58:32.730234   431 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.017497572s
I20260709 12:58:32.731405  1503 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:35464
I20260709 12:58:32.761227  1503 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35470:
name: "test-workload"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
  rows: "\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}""\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}"
  indirect_data: """"
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W20260709 12:58:32.768611  1503 catalog_manager.cc:7077] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260709 12:58:33.001932  1744 tablet_service.cc:1511] Processing CreateTablet for tablet 13031f9e955e4f5fb289c79ee2c46108 (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260709 12:58:33.002942  1666 tablet_service.cc:1511] Processing CreateTablet for tablet d21485c094584689917b2a97b2d3c624 (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260709 12:58:33.003726  1744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 13031f9e955e4f5fb289c79ee2c46108. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:33.004516  1666 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d21485c094584689917b2a97b2d3c624. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:33.004861  1595 tablet_service.cc:1511] Processing CreateTablet for tablet 13031f9e955e4f5fb289c79ee2c46108 (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260709 12:58:33.007786  1668 tablet_service.cc:1511] Processing CreateTablet for tablet 592fa647ae2545e09ad5ee25f8cb658e (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260709 12:58:33.009454  1668 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 592fa647ae2545e09ad5ee25f8cb658e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:33.009344  1739 tablet_service.cc:1511] Processing CreateTablet for tablet 0615de9bff034ecb9ed6173433128ff5 (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260709 12:58:33.009933  1593 tablet_service.cc:1511] Processing CreateTablet for tablet 592fa647ae2545e09ad5ee25f8cb658e (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260709 12:58:33.010785  1739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0615de9bff034ecb9ed6173433128ff5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:33.011771  1740 tablet_service.cc:1511] Processing CreateTablet for tablet d21485c094584689917b2a97b2d3c624 (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260709 12:58:33.012593  1592 tablet_service.cc:1511] Processing CreateTablet for tablet 7ebbb6d7c6c4478e95972de2f78d3f80 (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260709 12:58:33.013186  1740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d21485c094584689917b2a97b2d3c624. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:33.015298  1591 tablet_service.cc:1511] Processing CreateTablet for tablet d21485c094584689917b2a97b2d3c624 (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260709 12:58:33.015793  1669 tablet_service.cc:1511] Processing CreateTablet for tablet 65285571995c4ed5b593041598056a22 (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260709 12:58:33.017311  1669 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65285571995c4ed5b593041598056a22. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:33.017145  1738 tablet_service.cc:1511] Processing CreateTablet for tablet 32dc176efd53437ab7735ea8c6ac4a0d (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260709 12:58:33.017774  1590 tablet_service.cc:1511] Processing CreateTablet for tablet 0615de9bff034ecb9ed6173433128ff5 (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260709 12:58:33.018534  1738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32dc176efd53437ab7735ea8c6ac4a0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:33.019591  1741 tablet_service.cc:1511] Processing CreateTablet for tablet 7ebbb6d7c6c4478e95972de2f78d3f80 (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260709 12:58:33.020970  1741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ebbb6d7c6c4478e95972de2f78d3f80. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:33.007855  1594 tablet_service.cc:1511] Processing CreateTablet for tablet 65285571995c4ed5b593041598056a22 (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260709 12:58:33.023655  1670 tablet_service.cc:1511] Processing CreateTablet for tablet 13031f9e955e4f5fb289c79ee2c46108 (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260709 12:58:33.025043  1670 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 13031f9e955e4f5fb289c79ee2c46108. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:33.025318  1737 tablet_service.cc:1511] Processing CreateTablet for tablet 84e78b66ceec4f759ae9e101a5469e4c (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260709 12:58:33.026710  1737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 84e78b66ceec4f759ae9e101a5469e4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:33.026610  1589 tablet_service.cc:1511] Processing CreateTablet for tablet 32dc176efd53437ab7735ea8c6ac4a0d (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260709 12:58:33.027721  1742 tablet_service.cc:1511] Processing CreateTablet for tablet 592fa647ae2545e09ad5ee25f8cb658e (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260709 12:58:33.029055  1742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 592fa647ae2545e09ad5ee25f8cb658e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:33.029699  1588 tablet_service.cc:1511] Processing CreateTablet for tablet 84e78b66ceec4f759ae9e101a5469e4c (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260709 12:58:33.031674  1743 tablet_service.cc:1511] Processing CreateTablet for tablet 65285571995c4ed5b593041598056a22 (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260709 12:58:33.033099  1743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65285571995c4ed5b593041598056a22. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:33.032716  1736 tablet_service.cc:1511] Processing CreateTablet for tablet e4f4b860aab3464383ce8f9d990f4a4a (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260709 12:58:33.032809  1587 tablet_service.cc:1511] Processing CreateTablet for tablet e4f4b860aab3464383ce8f9d990f4a4a (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260709 12:58:33.034122  1736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e4f4b860aab3464383ce8f9d990f4a4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:33.035938  1586 tablet_service.cc:1511] Processing CreateTablet for tablet 63cf5ab930f24f239fde96a128656333 (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260709 12:58:33.037367  1667 tablet_service.cc:1511] Processing CreateTablet for tablet 7ebbb6d7c6c4478e95972de2f78d3f80 (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260709 12:58:33.038796  1667 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ebbb6d7c6c4478e95972de2f78d3f80. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:33.040624  1595 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 13031f9e955e4f5fb289c79ee2c46108. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:33.042261  1735 tablet_service.cc:1511] Processing CreateTablet for tablet 63cf5ab930f24f239fde96a128656333 (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260709 12:58:33.043684  1735 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 63cf5ab930f24f239fde96a128656333. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:33.044020  1587 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e4f4b860aab3464383ce8f9d990f4a4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:33.045042  1664 tablet_service.cc:1511] Processing CreateTablet for tablet 32dc176efd53437ab7735ea8c6ac4a0d (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260709 12:58:33.047067  1589 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32dc176efd53437ab7735ea8c6ac4a0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:33.049222  1588 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 84e78b66ceec4f759ae9e101a5469e4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:33.048238  1663 tablet_service.cc:1511] Processing CreateTablet for tablet 84e78b66ceec4f759ae9e101a5469e4c (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260709 12:58:33.049944  1665 tablet_service.cc:1511] Processing CreateTablet for tablet 0615de9bff034ecb9ed6173433128ff5 (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260709 12:58:33.052515  1664 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32dc176efd53437ab7735ea8c6ac4a0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:33.054842  1665 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0615de9bff034ecb9ed6173433128ff5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:33.054951  1593 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 592fa647ae2545e09ad5ee25f8cb658e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:33.057014  1663 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 84e78b66ceec4f759ae9e101a5469e4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:33.057466  1594 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65285571995c4ed5b593041598056a22. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:33.060426  1662 tablet_service.cc:1511] Processing CreateTablet for tablet e4f4b860aab3464383ce8f9d990f4a4a (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260709 12:58:33.062741  1662 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e4f4b860aab3464383ce8f9d990f4a4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:33.065248  1592 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ebbb6d7c6c4478e95972de2f78d3f80. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:33.067754  1661 tablet_service.cc:1511] Processing CreateTablet for tablet 63cf5ab930f24f239fde96a128656333 (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260709 12:58:33.069336  1661 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 63cf5ab930f24f239fde96a128656333. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:33.070978  1586 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 63cf5ab930f24f239fde96a128656333. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:33.074190  1591 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d21485c094584689917b2a97b2d3c624. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:33.077828  1590 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0615de9bff034ecb9ed6173433128ff5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:33.181624  1666 tablet_service.cc:1511] Processing CreateTablet for tablet 388afdca3ce84bf0a8ce3331ed0d9b2f (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260709 12:58:33.192999  1735 tablet_service.cc:1511] Processing CreateTablet for tablet 388afdca3ce84bf0a8ce3331ed0d9b2f (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260709 12:58:33.207631  1735 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 388afdca3ce84bf0a8ce3331ed0d9b2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:33.209146  1743 tablet_service.cc:1511] Processing CreateTablet for tablet fe7e5e9266614676a9c05c398cf7ba9e (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260709 12:58:33.198654  1664 tablet_service.cc:1511] Processing CreateTablet for tablet e26344297fff44bb8d2099b8e64d7df2 (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260709 12:58:33.210822  1743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe7e5e9266614676a9c05c398cf7ba9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:33.211280  1664 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e26344297fff44bb8d2099b8e64d7df2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:33.202700  1592 tablet_service.cc:1511] Processing CreateTablet for tablet e26344297fff44bb8d2099b8e64d7df2 (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260709 12:58:33.214636  1666 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 388afdca3ce84bf0a8ce3331ed0d9b2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:33.209215  1590 tablet_service.cc:1511] Processing CreateTablet for tablet 2a521fb6c3e7489d9694fd81cadeb33f (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260709 12:58:33.224790  1742 tablet_service.cc:1511] Processing CreateTablet for tablet cd44a2ebdc764e898f25ad5390f4bffd (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260709 12:58:33.233251  1742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd44a2ebdc764e898f25ad5390f4bffd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:33.220960  1736 tablet_service.cc:1511] Processing CreateTablet for tablet e26344297fff44bb8d2099b8e64d7df2 (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260709 12:58:33.235296  1736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e26344297fff44bb8d2099b8e64d7df2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:33.224978  1737 tablet_service.cc:1511] Processing CreateTablet for tablet b917a1e9c9aa40a0afce5a12c343eacd (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260709 12:58:33.238873  1794 tablet_bootstrap.cc:492] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d: Bootstrap starting.
I20260709 12:58:33.239319  1737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b917a1e9c9aa40a0afce5a12c343eacd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:33.244230  1663 tablet_service.cc:1511] Processing CreateTablet for tablet fe7e5e9266614676a9c05c398cf7ba9e (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260709 12:58:33.195521  1593 tablet_service.cc:1511] Processing CreateTablet for tablet 388afdca3ce84bf0a8ce3331ed0d9b2f (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260709 12:58:33.228083  1740 tablet_service.cc:1511] Processing CreateTablet for tablet 83226095f79a46dab445849b1febd425 (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260709 12:58:33.245977  1663 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe7e5e9266614676a9c05c398cf7ba9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:33.225411  1738 tablet_service.cc:1511] Processing CreateTablet for tablet 2a521fb6c3e7489d9694fd81cadeb33f (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260709 12:58:33.246420  1740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 83226095f79a46dab445849b1febd425. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:33.250392  1665 tablet_service.cc:1511] Processing CreateTablet for tablet cd44a2ebdc764e898f25ad5390f4bffd (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260709 12:58:33.247038  1670 tablet_service.cc:1511] Processing CreateTablet for tablet 2a521fb6c3e7489d9694fd81cadeb33f (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260709 12:58:33.254638  1738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2a521fb6c3e7489d9694fd81cadeb33f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:33.227389  1741 tablet_service.cc:1511] Processing CreateTablet for tablet e527450b1d6c4cb79e9f6d3c87d4ad23 (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260709 12:58:33.261132  1667 tablet_service.cc:1511] Processing CreateTablet for tablet b917a1e9c9aa40a0afce5a12c343eacd (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260709 12:58:33.270808  1667 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b917a1e9c9aa40a0afce5a12c343eacd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:33.272915  1735 tablet_service.cc:1511] Processing CreateTablet for tablet 77d66b92688a42e3ad231959a644db56 (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260709 12:58:33.275908  1735 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 77d66b92688a42e3ad231959a644db56. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:33.280122  1595 tablet_service.cc:1511] Processing CreateTablet for tablet fe7e5e9266614676a9c05c398cf7ba9e (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260709 12:58:33.265514  1794 tablet_bootstrap.cc:654] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:33.268291  1662 tablet_service.cc:1511] Processing CreateTablet for tablet 03daa4b346f54f1da820d1ff6cd1ff86 (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
W20260709 12:58:33.272074  1551 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:58:33.283988  1662 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03daa4b346f54f1da820d1ff6cd1ff86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:33.287364  1592 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e26344297fff44bb8d2099b8e64d7df2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:33.263425  1669 tablet_service.cc:1511] Processing CreateTablet for tablet e527450b1d6c4cb79e9f6d3c87d4ad23 (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260709 12:58:33.288866  1669 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e527450b1d6c4cb79e9f6d3c87d4ad23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:33.288786  1587 tablet_service.cc:1511] Processing CreateTablet for tablet cd44a2ebdc764e898f25ad5390f4bffd (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260709 12:58:33.290258  1587 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd44a2ebdc764e898f25ad5390f4bffd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:33.293184  1588 tablet_service.cc:1511] Processing CreateTablet for tablet b917a1e9c9aa40a0afce5a12c343eacd (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260709 12:58:33.293247  1663 tablet_service.cc:1511] Processing CreateTablet for tablet 79ed08f2339a4b1ead1e70d78eaec2ec (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260709 12:58:33.272670  1739 tablet_service.cc:1511] Processing CreateTablet for tablet dd3f391a609743de886c525de483fc29 (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260709 12:58:33.302419  1795 tablet_bootstrap.cc:492] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701: Bootstrap starting.
I20260709 12:58:33.303478  1796 tablet_bootstrap.cc:492] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a: Bootstrap starting.
I20260709 12:58:33.306569  1665 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd44a2ebdc764e898f25ad5390f4bffd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:33.306771  1590 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2a521fb6c3e7489d9694fd81cadeb33f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:33.309247  1593 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 388afdca3ce84bf0a8ce3331ed0d9b2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:33.310811  1670 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2a521fb6c3e7489d9694fd81cadeb33f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:33.311143  1591 tablet_service.cc:1511] Processing CreateTablet for tablet 83226095f79a46dab445849b1febd425 (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260709 12:58:33.315786  1796 tablet_bootstrap.cc:654] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:33.315550  1589 tablet_service.cc:1511] Processing CreateTablet for tablet e527450b1d6c4cb79e9f6d3c87d4ad23 (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260709 12:58:33.316438  1741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e527450b1d6c4cb79e9f6d3c87d4ad23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:33.323450  1668 tablet_service.cc:1511] Processing CreateTablet for tablet 83226095f79a46dab445849b1febd425 (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260709 12:58:33.325035  1668 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 83226095f79a46dab445849b1febd425. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:33.324150  1795 tablet_bootstrap.cc:654] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:33.347210  1739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd3f391a609743de886c525de483fc29. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:33.363150  1663 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 79ed08f2339a4b1ead1e70d78eaec2ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:33.367000  1744 tablet_service.cc:1511] Processing CreateTablet for tablet 513043a9b96c4f70adad53ac4b5d01dc (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260709 12:58:33.376683  1744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 513043a9b96c4f70adad53ac4b5d01dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:33.370167  1595 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe7e5e9266614676a9c05c398cf7ba9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:33.376801  1594 tablet_service.cc:1511] Processing CreateTablet for tablet dd3f391a609743de886c525de483fc29 (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260709 12:58:33.377694  1588 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b917a1e9c9aa40a0afce5a12c343eacd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:33.371805  1670 tablet_service.cc:1511] Processing CreateTablet for tablet dd3f391a609743de886c525de483fc29 (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260709 12:58:33.372020  1586 tablet_service.cc:1511] Processing CreateTablet for tablet 77d66b92688a42e3ad231959a644db56 (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260709 12:58:33.378371  1591 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 83226095f79a46dab445849b1febd425. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:33.379423  1586 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 77d66b92688a42e3ad231959a644db56. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:33.372985  1666 tablet_service.cc:1511] Processing CreateTablet for tablet a40e423ed583480280352b261913276c (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260709 12:58:33.368947  1742 tablet_service.cc:1511] Processing CreateTablet for tablet 79ed08f2339a4b1ead1e70d78eaec2ec (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260709 12:58:33.386999  1670 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd3f391a609743de886c525de483fc29. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:33.393463  1738 tablet_service.cc:1511] Processing CreateTablet for tablet 03daa4b346f54f1da820d1ff6cd1ff86 (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260709 12:58:33.394696  1590 tablet_service.cc:1511] Processing CreateTablet for tablet a40e423ed583480280352b261913276c (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260709 12:58:33.395967  1796 tablet_bootstrap.cc:492] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a: No bootstrap required, opened a new log
I20260709 12:58:33.396509  1796 ts_tablet_manager.cc:1403] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent bootstrapping tablet: real 0.093s	user 0.013s	sys 0.009s
I20260709 12:58:33.402734  1796 raft_consensus.cc:359] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:33.403489  1796 raft_consensus.cc:385] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:33.403806  1796 raft_consensus.cc:740] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Initialized, Role: FOLLOWER
I20260709 12:58:33.406538  1796 consensus_queue.cc:260] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:33.409825  1661 tablet_service.cc:1511] Processing CreateTablet for tablet 77d66b92688a42e3ad231959a644db56 (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260709 12:58:33.415191  1795 tablet_bootstrap.cc:492] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701: No bootstrap required, opened a new log
I20260709 12:58:33.417675  1795 ts_tablet_manager.cc:1403] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701: Time spent bootstrapping tablet: real 0.115s	user 0.026s	sys 0.010s
I20260709 12:58:33.397001  1594 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd3f391a609743de886c525de483fc29. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:33.430940  1740 tablet_service.cc:1511] Processing CreateTablet for tablet 58d4fdd02c6d4667aa718fb146a76505 (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260709 12:58:33.435201  1666 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a40e423ed583480280352b261913276c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:33.442864  1589 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e527450b1d6c4cb79e9f6d3c87d4ad23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:33.444257  1661 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 77d66b92688a42e3ad231959a644db56. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:33.444809  1590 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a40e423ed583480280352b261913276c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:33.358362  1737 tablet_service.cc:1511] Processing CreateTablet for tablet a40e423ed583480280352b261913276c (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260709 12:58:33.452677  1664 tablet_service.cc:1511] Processing CreateTablet for tablet 513043a9b96c4f70adad53ac4b5d01dc (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260709 12:58:33.454809  1742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 79ed08f2339a4b1ead1e70d78eaec2ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:33.470000  1738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03daa4b346f54f1da820d1ff6cd1ff86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:33.476210  1668 tablet_service.cc:1511] Processing CreateTablet for tablet 58d4fdd02c6d4667aa718fb146a76505 (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260709 12:58:33.434660  1593 tablet_service.cc:1511] Processing CreateTablet for tablet 513043a9b96c4f70adad53ac4b5d01dc (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260709 12:58:33.488853  1740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 58d4fdd02c6d4667aa718fb146a76505. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:33.489310  1593 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 513043a9b96c4f70adad53ac4b5d01dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:33.517653  1737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a40e423ed583480280352b261913276c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:33.500777  1630 heartbeater.cc:499] Master 127.0.107.254:33643 was elected leader, sending a full tablet report...
I20260709 12:58:33.434943  1591 tablet_service.cc:1511] Processing CreateTablet for tablet 79ed08f2339a4b1ead1e70d78eaec2ec (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260709 12:58:33.501067  1664 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 513043a9b96c4f70adad53ac4b5d01dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:33.528443  1668 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 58d4fdd02c6d4667aa718fb146a76505. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:33.530957  1801 raft_consensus.cc:493] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:33.535766  1801 raft_consensus.cc:515] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 12:58:33.546550  1488 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 99776.8us
I20260709 12:58:33.551578  1796 ts_tablet_manager.cc:1434] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent starting tablet: real 0.155s	user 0.015s	sys 0.013s
I20260709 12:58:33.552870  1796 tablet_bootstrap.cc:492] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a: Bootstrap starting.
I20260709 12:58:33.559219  1741 tablet_service.cc:1511] Processing CreateTablet for tablet bb255e8a61e7458babbbdf0183773a81 (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260709 12:58:33.559659  1795 raft_consensus.cc:359] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } }
I20260709 12:58:33.560643  1795 raft_consensus.cc:385] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:33.560710  1741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb255e8a61e7458babbbdf0183773a81. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:33.561226  1591 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 79ed08f2339a4b1ead1e70d78eaec2ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:33.630117  1795 raft_consensus.cc:740] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Initialized, Role: FOLLOWER
I20260709 12:58:33.624147  1794 tablet_bootstrap.cc:492] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d: No bootstrap required, opened a new log
I20260709 12:58:33.645905  1794 ts_tablet_manager.cc:1403] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d: Time spent bootstrapping tablet: real 0.407s	user 0.015s	sys 0.016s
I20260709 12:58:33.649361  1794 raft_consensus.cc:359] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:33.650154  1794 raft_consensus.cc:385] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:33.650823  1592 tablet_service.cc:1511] Processing CreateTablet for tablet 03daa4b346f54f1da820d1ff6cd1ff86 (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260709 12:58:33.570648  1743 tablet_service.cc:1511] Processing CreateTablet for tablet 87b86d46388a41cd90374c203cb27914 (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260709 12:58:33.640662  1662 tablet_service.cc:1511] Processing CreateTablet for tablet bb255e8a61e7458babbbdf0183773a81 (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260709 12:58:33.655298  1662 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb255e8a61e7458babbbdf0183773a81. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:33.643748  1588 tablet_service.cc:1511] Processing CreateTablet for tablet 58d4fdd02c6d4667aa718fb146a76505 (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260709 12:58:33.659159  1794 raft_consensus.cc:740] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Initialized, Role: FOLLOWER
I20260709 12:58:33.660567  1794 consensus_queue.cc:260] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:33.582751  1801 leader_election.cc:290] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:58:33.563443  1665 tablet_service.cc:1511] Processing CreateTablet for tablet 87b86d46388a41cd90374c203cb27914 (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260709 12:58:33.678067  1665 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 87b86d46388a41cd90374c203cb27914. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:33.638880  1705 heartbeater.cc:499] Master 127.0.107.254:33643 was elected leader, sending a full tablet report...
I20260709 12:58:33.683207  1795 consensus_queue.cc:260] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } }
I20260709 12:58:33.684072  1587 tablet_service.cc:1511] Processing CreateTablet for tablet bb255e8a61e7458babbbdf0183773a81 (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260709 12:58:33.678187  1667 tablet_service.cc:1511] Processing CreateTablet for tablet b9bb5648969b4299a776843d72043448 (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260709 12:58:33.698864  1667 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b9bb5648969b4299a776843d72043448. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:33.693482  1595 tablet_service.cc:1511] Processing CreateTablet for tablet 87b86d46388a41cd90374c203cb27914 (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260709 12:58:33.695842  1743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 87b86d46388a41cd90374c203cb27914. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:33.707824  1669 tablet_service.cc:1511] Processing CreateTablet for tablet 127630d60f944b899508c92ee18fdddf (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260709 12:58:33.709206  1669 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 127630d60f944b899508c92ee18fdddf. 1 dirs total, 0 dirs full, 0 dirs failed
W20260709 12:58:33.713719  1631 tablet.cc:2406] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260709 12:58:33.716259  1670 tablet_service.cc:1511] Processing CreateTablet for tablet 5108457796a949b7b753ab22b575b7a0 (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260709 12:58:33.717515  1670 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5108457796a949b7b753ab22b575b7a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:33.723114  1587 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb255e8a61e7458babbbdf0183773a81. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:33.727272  1592 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03daa4b346f54f1da820d1ff6cd1ff86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:33.710057  1736 tablet_service.cc:1511] Processing CreateTablet for tablet b9bb5648969b4299a776843d72043448 (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260709 12:58:33.733912  1595 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 87b86d46388a41cd90374c203cb27914. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:33.734747  1779 heartbeater.cc:499] Master 127.0.107.254:33643 was elected leader, sending a full tablet report...
I20260709 12:58:33.738149  1593 tablet_service.cc:1511] Processing CreateTablet for tablet b9bb5648969b4299a776843d72043448 (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260709 12:58:33.738761  1588 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 58d4fdd02c6d4667aa718fb146a76505. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:33.767165  1807 raft_consensus.cc:493] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:33.767776  1807 raft_consensus.cc:515] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } }
I20260709 12:58:33.768322  1806 raft_consensus.cc:493] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:33.768879  1806 raft_consensus.cc:515] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:33.771714  1736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b9bb5648969b4299a776843d72043448. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:33.783103  1593 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b9bb5648969b4299a776843d72043448. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:33.764432  1794 ts_tablet_manager.cc:1434] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d: Time spent starting tablet: real 0.118s	user 0.005s	sys 0.016s
I20260709 12:58:33.792537  1794 tablet_bootstrap.cc:492] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d: Bootstrap starting.
I20260709 12:58:33.823173  1796 tablet_bootstrap.cc:654] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:33.823882  1735 tablet_service.cc:1511] Processing CreateTablet for tablet 5108457796a949b7b753ab22b575b7a0 (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260709 12:58:33.825421  1735 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5108457796a949b7b753ab22b575b7a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:33.797008  1806 leader_election.cc:290] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:58:33.803434  1586 tablet_service.cc:1511] Processing CreateTablet for tablet 127630d60f944b899508c92ee18fdddf (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260709 12:58:33.845736  1586 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 127630d60f944b899508c92ee18fdddf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:33.864275  1795 ts_tablet_manager.cc:1434] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701: Time spent starting tablet: real 0.441s	user 0.014s	sys 0.025s
I20260709 12:58:33.866012  1795 tablet_bootstrap.cc:492] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701: Bootstrap starting.
I20260709 12:58:33.871651  1739 tablet_service.cc:1511] Processing CreateTablet for tablet c6ab3300be1e4a59b64b14d50da193ca (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260709 12:58:33.801967  1737 tablet_service.cc:1511] Processing CreateTablet for tablet 127630d60f944b899508c92ee18fdddf (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260709 12:58:33.899549  1663 tablet_service.cc:1511] Processing CreateTablet for tablet 1809d674329a492dbea0b60a4703fe51 (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260709 12:58:33.901074  1663 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1809d674329a492dbea0b60a4703fe51. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:33.901682  1594 tablet_service.cc:1511] Processing CreateTablet for tablet 72f3cdc0abae4f74aad5e4a15cb02e1b (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260709 12:58:33.902976  1594 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 72f3cdc0abae4f74aad5e4a15cb02e1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:33.906642  1740 tablet_service.cc:1511] Processing CreateTablet for tablet 983621a6fc5b4c9ea0159a648dfe7a36 (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260709 12:58:33.907423  1590 tablet_service.cc:1511] Processing CreateTablet for tablet 5108457796a949b7b753ab22b575b7a0 (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260709 12:58:33.908736  1590 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5108457796a949b7b753ab22b575b7a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:33.871515  1744 tablet_service.cc:1511] Processing CreateTablet for tablet 1809d674329a492dbea0b60a4703fe51 (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260709 12:58:33.895471  1739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6ab3300be1e4a59b64b14d50da193ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:33.916965  1807 leader_election.cc:290] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:58:33.939908  1668 tablet_service.cc:1511] Processing CreateTablet for tablet c6ab3300be1e4a59b64b14d50da193ca (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260709 12:58:33.941905  1668 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6ab3300be1e4a59b64b14d50da193ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:33.943697  1740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 983621a6fc5b4c9ea0159a648dfe7a36. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:33.946856  1737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 127630d60f944b899508c92ee18fdddf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:33.962884  1744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1809d674329a492dbea0b60a4703fe51. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:33.931973  1738 tablet_service.cc:1511] Processing CreateTablet for tablet 72f3cdc0abae4f74aad5e4a15cb02e1b (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260709 12:58:33.971493  1738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 72f3cdc0abae4f74aad5e4a15cb02e1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:33.920675  1666 tablet_service.cc:1511] Processing CreateTablet for tablet 72f3cdc0abae4f74aad5e4a15cb02e1b (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260709 12:58:33.978399  1666 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 72f3cdc0abae4f74aad5e4a15cb02e1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:34.004079  1662 tablet_service.cc:1511] Processing CreateTablet for tablet dce8db7565da4e13a82ffbb9c39fde78 (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260709 12:58:34.007441  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 12:58:34.010268  1569 leader_election.cc:343] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:33.954341  1742 tablet_service.cc:1511] Processing CreateTablet for tablet 86abe5695d5643f888190301b88dacef (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260709 12:58:34.018955  1742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 86abe5695d5643f888190301b88dacef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:34.020538  1661 tablet_service.cc:1511] Processing CreateTablet for tablet 86abe5695d5643f888190301b88dacef (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260709 12:58:34.006274  1662 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dce8db7565da4e13a82ffbb9c39fde78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:34.040891  1590 tablet_service.cc:1511] Processing CreateTablet for tablet 1809d674329a492dbea0b60a4703fe51 (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260709 12:58:34.047690  1795 tablet_bootstrap.cc:654] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:34.048549  1794 tablet_bootstrap.cc:654] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:34.052551  1796 tablet_bootstrap.cc:492] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a: No bootstrap required, opened a new log
I20260709 12:58:34.043687  1590 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1809d674329a492dbea0b60a4703fe51. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:34.056051  1661 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 86abe5695d5643f888190301b88dacef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:34.054332  1796 ts_tablet_manager.cc:1403] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent bootstrapping tablet: real 0.502s	user 0.018s	sys 0.015s
I20260709 12:58:34.060879  1796 raft_consensus.cc:359] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:34.061664  1796 raft_consensus.cc:385] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:34.062008  1796 raft_consensus.cc:740] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Initialized, Role: FOLLOWER
I20260709 12:58:34.063803  1669 tablet_service.cc:1511] Processing CreateTablet for tablet 03e37c14b11849208b5c9c51f32e1702 (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260709 12:58:34.065258  1669 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03e37c14b11849208b5c9c51f32e1702. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:34.074201  1663 tablet_service.cc:1511] Processing CreateTablet for tablet 344b4152823f4d618345b14ffd579400 (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260709 12:58:34.075950  1663 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 344b4152823f4d618345b14ffd579400. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:34.078954  1743 tablet_service.cc:1511] Processing CreateTablet for tablet dce8db7565da4e13a82ffbb9c39fde78 (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260709 12:58:34.081373  1589 tablet_service.cc:1511] Processing CreateTablet for tablet c6ab3300be1e4a59b64b14d50da193ca (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260709 12:58:34.096719  1664 tablet_service.cc:1511] Processing CreateTablet for tablet 983621a6fc5b4c9ea0159a648dfe7a36 (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260709 12:58:34.102141  1589 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6ab3300be1e4a59b64b14d50da193ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:34.096807  1796 consensus_queue.cc:260] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:34.104352  1796 ts_tablet_manager.cc:1434] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent starting tablet: real 0.047s	user 0.008s	sys 0.000s
I20260709 12:58:34.105461  1796 tablet_bootstrap.cc:492] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a: Bootstrap starting.
I20260709 12:58:34.109298  1667 tablet_service.cc:1511] Processing CreateTablet for tablet c81785db7d654e4288fcaeacf4db2b49 (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260709 12:58:34.111147  1592 tablet_service.cc:1511] Processing CreateTablet for tablet 86abe5695d5643f888190301b88dacef (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260709 12:58:34.112985  1588 tablet_service.cc:1511] Processing CreateTablet for tablet 03e37c14b11849208b5c9c51f32e1702 (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260709 12:58:34.123518  1591 tablet_service.cc:1511] Processing CreateTablet for tablet dce8db7565da4e13a82ffbb9c39fde78 (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260709 12:58:34.125193  1743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dce8db7565da4e13a82ffbb9c39fde78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:34.136355  1588 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03e37c14b11849208b5c9c51f32e1702. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:34.121698  1595 tablet_service.cc:1511] Processing CreateTablet for tablet 344b4152823f4d618345b14ffd579400 (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260709 12:58:34.149340  1664 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 983621a6fc5b4c9ea0159a648dfe7a36. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:34.157121  1667 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c81785db7d654e4288fcaeacf4db2b49. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:34.161751  1744 tablet_service.cc:1511] Processing CreateTablet for tablet 344b4152823f4d618345b14ffd579400 (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260709 12:58:34.163231  1744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 344b4152823f4d618345b14ffd579400. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:34.165680  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:58:34.166317  1794 tablet_bootstrap.cc:492] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d: No bootstrap required, opened a new log
I20260709 12:58:34.166887  1794 ts_tablet_manager.cc:1403] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent bootstrapping tablet: real 0.375s	user 0.011s	sys 0.040s
I20260709 12:58:34.172433  1592 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 86abe5695d5643f888190301b88dacef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:34.173959  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:58:34.175938  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 12:58:34.176626  1717 leader_election.cc:343] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:34.177538  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 12:58:34.179330  1646 leader_election.cc:343] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:34.180004  1586 tablet_service.cc:1511] Processing CreateTablet for tablet 453513ba44f34adb8b436305ee51fe3d (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
W20260709 12:58:34.180550  1571 leader_election.cc:343] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:34.181059  1571 leader_election.cc:304] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 12:58:34.181599  1586 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 453513ba44f34adb8b436305ee51fe3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:34.182406  1801 raft_consensus.cc:2749] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:58:34.183930  1595 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 344b4152823f4d618345b14ffd579400. 1 dirs total, 0 dirs full, 0 dirs failed
W20260709 12:58:34.185881  1718 leader_election.cc:343] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:34.193436  1591 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dce8db7565da4e13a82ffbb9c39fde78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:34.159659  1670 tablet_service.cc:1511] Processing CreateTablet for tablet 453513ba44f34adb8b436305ee51fe3d (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260709 12:58:34.195978  1670 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 453513ba44f34adb8b436305ee51fe3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:34.198581  1718 leader_election.cc:304] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:58:34.199586  1807 raft_consensus.cc:2749] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:58:34.217527  1593 tablet_service.cc:1511] Processing CreateTablet for tablet 983621a6fc5b4c9ea0159a648dfe7a36 (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260709 12:58:34.219024  1593 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 983621a6fc5b4c9ea0159a648dfe7a36. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:34.217285  1665 tablet_service.cc:1511] Processing CreateTablet for tablet e947690dcb3849cb9f9731123d363b91 (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260709 12:58:34.220222  1665 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e947690dcb3849cb9f9731123d363b91. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:34.224709  1740 tablet_service.cc:1511] Processing CreateTablet for tablet 03e37c14b11849208b5c9c51f32e1702 (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260709 12:58:34.226087  1740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03e37c14b11849208b5c9c51f32e1702. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:34.227128  1795 tablet_bootstrap.cc:492] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701: No bootstrap required, opened a new log
I20260709 12:58:34.227656  1795 ts_tablet_manager.cc:1403] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701: Time spent bootstrapping tablet: real 0.362s	user 0.020s	sys 0.016s
I20260709 12:58:34.228809  1744 tablet_service.cc:1511] Processing CreateTablet for tablet c81785db7d654e4288fcaeacf4db2b49 (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260709 12:58:34.231727  1741 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:55838 (request call id 26) took 1011 ms. Trace:
I20260709 12:58:34.231849  1587 tablet_service.cc:1511] Processing CreateTablet for tablet c81785db7d654e4288fcaeacf4db2b49 (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260709 12:58:34.233201  1587 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c81785db7d654e4288fcaeacf4db2b49. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:34.232249  1741 rpcz_store.cc:276] 0709 12:58:33.220794 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:58:33.558121 (+337327us) service_pool.cc:224] Handling call
0709 12:58:33.560185 (+  2064us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:58:33.560293 (+   108us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:58:34.065093 (+504800us) tablet_metadata.cc:665] Metadata flushed
0709 12:58:34.231190 (+166097us) tablet_replica.cc:206] Creating consensus instance
0709 12:58:34.231644 (+   454us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":29162}
I20260709 12:58:34.234175  1736 tablet_service.cc:1511] Processing CreateTablet for tablet 453513ba44f34adb8b436305ee51fe3d (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260709 12:58:34.239933  1594 tablet_service.cc:1511] Processing CreateTablet for tablet e947690dcb3849cb9f9731123d363b91 (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260709 12:58:34.241277  1594 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e947690dcb3849cb9f9731123d363b91. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:34.244513  1796 tablet_bootstrap.cc:654] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:34.246119  1736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 453513ba44f34adb8b436305ee51fe3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:34.222522  1794 raft_consensus.cc:359] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:34.250854  1794 raft_consensus.cc:385] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:34.251204  1794 raft_consensus.cc:740] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Initialized, Role: FOLLOWER
I20260709 12:58:34.252060  1794 consensus_queue.cc:260] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:34.250319  1743 tablet_service.cc:1511] Processing CreateTablet for tablet e947690dcb3849cb9f9731123d363b91 (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260709 12:58:34.257393  1794 ts_tablet_manager.cc:1434] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent starting tablet: real 0.090s	user 0.006s	sys 0.001s
I20260709 12:58:34.271883  1795 raft_consensus.cc:359] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:34.272784  1795 raft_consensus.cc:385] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:34.273106  1795 raft_consensus.cc:740] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Initialized, Role: FOLLOWER
I20260709 12:58:34.275707  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:58:34.275745  1794 tablet_bootstrap.cc:492] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d: Bootstrap starting.
W20260709 12:58:34.277367  1643 leader_election.cc:343] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:34.277916  1643 leader_election.cc:304] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 12:58:34.278961  1808 raft_consensus.cc:2749] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260709 12:58:34.285307  1551 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:58:34.286888  1743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e947690dcb3849cb9f9731123d363b91. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:34.294860  1744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c81785db7d654e4288fcaeacf4db2b49. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:34.300014  1668 tablet_service.cc:1511] Processing CreateTablet for tablet 14588fa8cfd6467e989746e326a0ed31 (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260709 12:58:34.301496  1668 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 14588fa8cfd6467e989746e326a0ed31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:34.311672  1590 tablet_service.cc:1511] Processing CreateTablet for tablet 14588fa8cfd6467e989746e326a0ed31 (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260709 12:58:34.312940  1590 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 14588fa8cfd6467e989746e326a0ed31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:34.274003  1795 consensus_queue.cc:260] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:34.316160  1737 tablet_service.cc:1511] Processing CreateTablet for tablet 14588fa8cfd6467e989746e326a0ed31 (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260709 12:58:34.324600  1737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 14588fa8cfd6467e989746e326a0ed31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:34.327898  1586 tablet_service.cc:1511] Processing CreateTablet for tablet 2ebc67196b4f4119981cdb499e3350cc (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260709 12:58:34.329362  1586 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ebc67196b4f4119981cdb499e3350cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:34.331391  1662 tablet_service.cc:1511] Processing CreateTablet for tablet 2ebc67196b4f4119981cdb499e3350cc (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260709 12:58:34.332695  1662 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ebc67196b4f4119981cdb499e3350cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:34.343094  1738 tablet_service.cc:1511] Processing CreateTablet for tablet 2ebc67196b4f4119981cdb499e3350cc (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260709 12:58:34.344576  1738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ebc67196b4f4119981cdb499e3350cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:34.345603  1796 tablet_bootstrap.cc:492] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a: No bootstrap required, opened a new log
I20260709 12:58:34.346194  1796 ts_tablet_manager.cc:1403] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent bootstrapping tablet: real 0.241s	user 0.011s	sys 0.000s
I20260709 12:58:34.351135  1796 raft_consensus.cc:359] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:34.351922  1796 raft_consensus.cc:385] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:34.352257  1796 raft_consensus.cc:740] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Initialized, Role: FOLLOWER
I20260709 12:58:34.353006  1796 consensus_queue.cc:260] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:34.358739  1796 ts_tablet_manager.cc:1434] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent starting tablet: real 0.012s	user 0.007s	sys 0.000s
I20260709 12:58:34.359921  1796 tablet_bootstrap.cc:492] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a: Bootstrap starting.
I20260709 12:58:34.354351  1794 tablet_bootstrap.cc:654] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:34.388661  1795 ts_tablet_manager.cc:1434] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701: Time spent starting tablet: real 0.161s	user 0.005s	sys 0.005s
I20260709 12:58:34.390005  1795 tablet_bootstrap.cc:492] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701: Bootstrap starting.
I20260709 12:58:34.403980  1794 tablet_bootstrap.cc:492] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d: No bootstrap required, opened a new log
I20260709 12:58:34.404646  1794 ts_tablet_manager.cc:1403] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d: Time spent bootstrapping tablet: real 0.146s	user 0.013s	sys 0.000s
I20260709 12:58:34.405690  1796 tablet_bootstrap.cc:654] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:34.413254  1742 tablet_service.cc:1511] Processing CreateTablet for tablet 1c2f5fa72dee4ec58979ace307d2acc3 (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260709 12:58:34.415580  1742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c2f5fa72dee4ec58979ace307d2acc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:34.416823  1669 tablet_service.cc:1511] Processing CreateTablet for tablet 1c2f5fa72dee4ec58979ace307d2acc3 (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260709 12:58:34.418174  1794 raft_consensus.cc:359] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:34.419149  1590 tablet_service.cc:1511] Processing CreateTablet for tablet 73dc0a9696be41f78e22f1f77a6eee72 (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260709 12:58:34.421428  1794 raft_consensus.cc:385] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:34.422534  1794 raft_consensus.cc:740] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Initialized, Role: FOLLOWER
I20260709 12:58:34.423278  1669 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c2f5fa72dee4ec58979ace307d2acc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:34.423384  1794 consensus_queue.cc:260] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:34.426851  1794 ts_tablet_manager.cc:1434] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d: Time spent starting tablet: real 0.022s	user 0.010s	sys 0.000s
I20260709 12:58:34.429101  1590 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 73dc0a9696be41f78e22f1f77a6eee72. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:34.431720  1794 tablet_bootstrap.cc:492] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d: Bootstrap starting.
I20260709 12:58:34.433529  1744 tablet_service.cc:1511] Processing CreateTablet for tablet 73dc0a9696be41f78e22f1f77a6eee72 (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260709 12:58:34.435011  1744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 73dc0a9696be41f78e22f1f77a6eee72. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:34.437808  1795 tablet_bootstrap.cc:654] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:34.431721  1662 tablet_service.cc:1511] Processing CreateTablet for tablet 73dc0a9696be41f78e22f1f77a6eee72 (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260709 12:58:34.443442  1662 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 73dc0a9696be41f78e22f1f77a6eee72. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:34.447126  1796 tablet_bootstrap.cc:492] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a: No bootstrap required, opened a new log
I20260709 12:58:34.424458  1594 tablet_service.cc:1511] Processing CreateTablet for tablet 1c2f5fa72dee4ec58979ace307d2acc3 (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260709 12:58:34.443578  1670 tablet_service.cc:1511] Processing CreateTablet for tablet 4d4ba8d6f4f947fabf8a2609af29c3d3 (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260709 12:58:34.448359  1594 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c2f5fa72dee4ec58979ace307d2acc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:34.448632  1670 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d4ba8d6f4f947fabf8a2609af29c3d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:34.444800  1740 tablet_service.cc:1511] Processing CreateTablet for tablet 4d4ba8d6f4f947fabf8a2609af29c3d3 (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260709 12:58:34.447674  1796 ts_tablet_manager.cc:1403] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent bootstrapping tablet: real 0.088s	user 0.008s	sys 0.004s
I20260709 12:58:34.454965  1794 tablet_bootstrap.cc:654] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:34.455324  1740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d4ba8d6f4f947fabf8a2609af29c3d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:34.457813  1796 raft_consensus.cc:359] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:34.458212  1592 tablet_service.cc:1511] Processing CreateTablet for tablet 1f1329a6c1e442bcba5945a2d1e65e85 (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260709 12:58:34.461954  1592 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1f1329a6c1e442bcba5945a2d1e65e85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:34.470126  1796 raft_consensus.cc:385] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:34.470511  1796 raft_consensus.cc:740] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Initialized, Role: FOLLOWER
I20260709 12:58:34.471997  1591 tablet_service.cc:1511] Processing CreateTablet for tablet 4d4ba8d6f4f947fabf8a2609af29c3d3 (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260709 12:58:34.473470  1591 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d4ba8d6f4f947fabf8a2609af29c3d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:34.476621  1668 tablet_service.cc:1511] Processing CreateTablet for tablet 1f1329a6c1e442bcba5945a2d1e65e85 (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260709 12:58:34.477952  1668 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1f1329a6c1e442bcba5945a2d1e65e85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:34.500324  1794 tablet_bootstrap.cc:492] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d: No bootstrap required, opened a new log
I20260709 12:58:34.500995  1794 ts_tablet_manager.cc:1403] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d: Time spent bootstrapping tablet: real 0.070s	user 0.011s	sys 0.000s
I20260709 12:58:34.503144  1742 tablet_service.cc:1511] Processing CreateTablet for tablet 1f1329a6c1e442bcba5945a2d1e65e85 (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260709 12:58:34.504577  1742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1f1329a6c1e442bcba5945a2d1e65e85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:34.503141  1796 consensus_queue.cc:260] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:34.509459  1794 raft_consensus.cc:359] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:34.510244  1794 raft_consensus.cc:385] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:34.510628  1794 raft_consensus.cc:740] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Initialized, Role: FOLLOWER
I20260709 12:58:34.510918  1796 ts_tablet_manager.cc:1434] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent starting tablet: real 0.056s	user 0.008s	sys 0.000s
I20260709 12:58:34.513887  1796 tablet_bootstrap.cc:492] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a: Bootstrap starting.
I20260709 12:58:34.513556  1591 tablet_service.cc:1511] Processing CreateTablet for tablet 6f6cc48def184bcc907c18c98277187f (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260709 12:58:34.515442  1591 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f6cc48def184bcc907c18c98277187f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:34.516794  1795 tablet_bootstrap.cc:492] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701: No bootstrap required, opened a new log
I20260709 12:58:34.517490  1801 raft_consensus.cc:493] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:34.517972  1801 raft_consensus.cc:515] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:34.519356  1590 tablet_service.cc:1511] Processing CreateTablet for tablet 19ce8a0cd7c54eafbb0e61fc6e305b62 (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260709 12:58:34.520144  1801 leader_election.cc:290] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:58:34.520642  1590 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19ce8a0cd7c54eafbb0e61fc6e305b62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:34.520817  1740 tablet_service.cc:1511] Processing CreateTablet for tablet 6f6cc48def184bcc907c18c98277187f (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260709 12:58:34.522101  1740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f6cc48def184bcc907c18c98277187f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:34.522424  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:58:34.524530  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 12:58:34.525638  1571 leader_election.cc:343] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:58:34.526775  1569 leader_election.cc:343] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:34.527237  1569 leader_election.cc:304] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 12:58:34.531509  1669 tablet_service.cc:1511] Processing CreateTablet for tablet 6f6cc48def184bcc907c18c98277187f (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260709 12:58:34.532826  1669 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f6cc48def184bcc907c18c98277187f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:34.535193  1801 raft_consensus.cc:2749] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:58:34.537267  1795 ts_tablet_manager.cc:1403] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701: Time spent bootstrapping tablet: real 0.148s	user 0.026s	sys 0.018s
I20260709 12:58:34.540894  1795 raft_consensus.cc:359] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:34.541669  1795 raft_consensus.cc:385] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:34.541957  1795 raft_consensus.cc:740] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Initialized, Role: FOLLOWER
I20260709 12:58:34.535233  1794 consensus_queue.cc:260] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:34.546196  1662 tablet_service.cc:1511] Processing CreateTablet for tablet 19ce8a0cd7c54eafbb0e61fc6e305b62 (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260709 12:58:34.547132  1737 tablet_service.cc:1511] Processing CreateTablet for tablet 19ce8a0cd7c54eafbb0e61fc6e305b62 (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260709 12:58:34.547665  1662 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19ce8a0cd7c54eafbb0e61fc6e305b62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:34.542913  1795 consensus_queue.cc:260] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:34.548545  1737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19ce8a0cd7c54eafbb0e61fc6e305b62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:34.549968  1795 ts_tablet_manager.cc:1434] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701: Time spent starting tablet: real 0.012s	user 0.003s	sys 0.003s
I20260709 12:58:34.551137  1795 tablet_bootstrap.cc:492] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701: Bootstrap starting.
I20260709 12:58:34.557336  1794 ts_tablet_manager.cc:1434] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d: Time spent starting tablet: real 0.056s	user 0.009s	sys 0.000s
I20260709 12:58:34.558030  1795 tablet_bootstrap.cc:654] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:34.565956  1591 tablet_service.cc:1511] Processing CreateTablet for tablet a93d4b974fee4d02afbaa26d69bfeb07 (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260709 12:58:34.567348  1794 tablet_bootstrap.cc:492] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d: Bootstrap starting.
I20260709 12:58:34.567430  1591 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a93d4b974fee4d02afbaa26d69bfeb07. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:34.567657  1808 raft_consensus.cc:493] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:34.568327  1808 raft_consensus.cc:515] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:34.572350  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 12:58:34.574330  1643 leader_election.cc:343] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:34.577262  1670 tablet_service.cc:1511] Processing CreateTablet for tablet a93d4b974fee4d02afbaa26d69bfeb07 (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260709 12:58:34.578024  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:58:34.578624  1670 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a93d4b974fee4d02afbaa26d69bfeb07. 1 dirs total, 0 dirs full, 0 dirs failed
W20260709 12:58:34.579545  1646 leader_election.cc:343] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:34.579990  1646 leader_election.cc:304] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 12:58:34.580186  1744 tablet_service.cc:1511] Processing CreateTablet for tablet a93d4b974fee4d02afbaa26d69bfeb07 (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260709 12:58:34.581066  1808 leader_election.cc:290] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:58:34.581508  1744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a93d4b974fee4d02afbaa26d69bfeb07. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:34.581591  1808 raft_consensus.cc:2749] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:58:34.547657  1796 tablet_bootstrap.cc:654] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:34.594106  1592 tablet_service.cc:1511] Processing CreateTablet for tablet f6028a708cb344fbb08ab8056921d9e7 (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260709 12:58:34.595757  1592 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f6028a708cb344fbb08ab8056921d9e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:34.599556  1808 raft_consensus.cc:493] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:34.599640  1667 tablet_service.cc:1511] Processing CreateTablet for tablet f6028a708cb344fbb08ab8056921d9e7 (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260709 12:58:34.600150  1808 raft_consensus.cc:515] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:34.601034  1667 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f6028a708cb344fbb08ab8056921d9e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:34.602283  1794 tablet_bootstrap.cc:654] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:34.603327  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 12:58:34.604875  1643 leader_election.cc:343] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:34.605870  1808 leader_election.cc:290] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:58:34.607975  1796 tablet_bootstrap.cc:492] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a: No bootstrap required, opened a new log
I20260709 12:58:34.608506  1796 ts_tablet_manager.cc:1403] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent bootstrapping tablet: real 0.095s	user 0.006s	sys 0.007s
I20260709 12:58:34.613965  1807 raft_consensus.cc:493] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:34.614555  1807 raft_consensus.cc:515] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:34.616752  1807 leader_election.cc:290] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:58:34.618492  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:58:34.618848  1794 tablet_bootstrap.cc:492] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d: No bootstrap required, opened a new log
I20260709 12:58:34.619405  1794 ts_tablet_manager.cc:1403] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent bootstrapping tablet: real 0.061s	user 0.006s	sys 0.006s
I20260709 12:58:34.620388  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:58:34.622125  1795 tablet_bootstrap.cc:492] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701: No bootstrap required, opened a new log
W20260709 12:58:34.621980  1717 leader_election.cc:343] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:58:34.622961  1718 leader_election.cc:343] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:58:34.623101  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 43588.8us
I20260709 12:58:34.623636  1718 leader_election.cc:304] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:58:34.623924  1742 tablet_service.cc:1511] Processing CreateTablet for tablet f6028a708cb344fbb08ab8056921d9e7 (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260709 12:58:34.622999  1795 ts_tablet_manager.cc:1403] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701: Time spent bootstrapping tablet: real 0.072s	user 0.018s	sys 0.008s
I20260709 12:58:34.624732  1807 raft_consensus.cc:2749] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:58:34.625142  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:58:34.627333  1742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f6028a708cb344fbb08ab8056921d9e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:34.628610  1796 raft_consensus.cc:359] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } }
I20260709 12:58:34.629336  1796 raft_consensus.cc:385] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:34.629630  1796 raft_consensus.cc:740] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Initialized, Role: FOLLOWER
I20260709 12:58:34.633163  1668 tablet_service.cc:1511] Processing CreateTablet for tablet 1d9ac76dc2c445edb5a25cf9087998bb (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260709 12:58:34.630558  1795 raft_consensus.cc:359] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:34.634770  1668 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d9ac76dc2c445edb5a25cf9087998bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:34.634369  1738 tablet_service.cc:1511] Processing CreateTablet for tablet 1d9ac76dc2c445edb5a25cf9087998bb (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260709 12:58:34.635823  1738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d9ac76dc2c445edb5a25cf9087998bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:34.636197  1591 tablet_service.cc:1511] Processing CreateTablet for tablet 1d9ac76dc2c445edb5a25cf9087998bb (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260709 12:58:34.634732  1795 raft_consensus.cc:385] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260709 12:58:34.638541  1646 leader_election.cc:343] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:34.641959  1795 raft_consensus.cc:740] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Initialized, Role: FOLLOWER
I20260709 12:58:34.639415  1794 raft_consensus.cc:359] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } }
I20260709 12:58:34.648955  1794 raft_consensus.cc:385] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:34.649271  1794 raft_consensus.cc:740] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Initialized, Role: FOLLOWER
I20260709 12:58:34.649415  1795 consensus_queue.cc:260] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:34.650038  1794 consensus_queue.cc:260] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } }
I20260709 12:58:34.652504  1794 ts_tablet_manager.cc:1434] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent starting tablet: real 0.033s	user 0.007s	sys 0.000s
I20260709 12:58:34.642057  1646 leader_election.cc:304] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 12:58:34.638983  1591 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d9ac76dc2c445edb5a25cf9087998bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:34.644062  1590 tablet_service.cc:1511] Processing CreateTablet for tablet a3accbb733bb4cafa69428319664f239 (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260709 12:58:34.659217  1590 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3accbb733bb4cafa69428319664f239. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:34.658958  1594 tablet_service.cc:1511] Processing CreateTablet for tablet 41166f283aea41b0b3f5be60e8234443 (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260709 12:58:34.660226  1594 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 41166f283aea41b0b3f5be60e8234443. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:34.660545  1826 raft_consensus.cc:2749] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:58:34.646485  1665 tablet_service.cc:1511] Processing CreateTablet for tablet a3accbb733bb4cafa69428319664f239 (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260709 12:58:34.660277  1795 ts_tablet_manager.cc:1434] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701: Time spent starting tablet: real 0.035s	user 0.008s	sys 0.000s
I20260709 12:58:34.630554  1796 consensus_queue.cc:260] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } }
I20260709 12:58:34.661784  1665 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3accbb733bb4cafa69428319664f239. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:34.647612  1737 tablet_service.cc:1511] Processing CreateTablet for tablet a3accbb733bb4cafa69428319664f239 (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260709 12:58:34.679402  1669 tablet_service.cc:1511] Processing CreateTablet for tablet 41166f283aea41b0b3f5be60e8234443 (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260709 12:58:34.683620  1740 tablet_service.cc:1511] Processing CreateTablet for tablet 41166f283aea41b0b3f5be60e8234443 (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260709 12:58:34.684823  1669 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 41166f283aea41b0b3f5be60e8234443. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:34.686064  1794 tablet_bootstrap.cc:492] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d: Bootstrap starting.
I20260709 12:58:34.691831  1794 tablet_bootstrap.cc:654] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:34.695858  1592 tablet_service.cc:1511] Processing CreateTablet for tablet a8c04bcdff52470cb06667e46356e653 (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260709 12:58:34.718659  1737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3accbb733bb4cafa69428319664f239. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:34.727160  1795 tablet_bootstrap.cc:492] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701: Bootstrap starting.
I20260709 12:58:34.727960  1738 tablet_service.cc:1511] Processing CreateTablet for tablet a8c04bcdff52470cb06667e46356e653 (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260709 12:58:34.729308  1738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8c04bcdff52470cb06667e46356e653. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:34.729629  1796 ts_tablet_manager.cc:1434] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent starting tablet: real 0.121s	user 0.010s	sys 0.000s
I20260709 12:58:34.730038  1591 tablet_service.cc:1511] Processing CreateTablet for tablet f8ab62ffb73246dc9bcf39f7bee51147 (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260709 12:58:34.731741  1669 tablet_service.cc:1511] Processing CreateTablet for tablet a8c04bcdff52470cb06667e46356e653 (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260709 12:58:34.732131  1591 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f8ab62ffb73246dc9bcf39f7bee51147. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:34.732909  1669 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8c04bcdff52470cb06667e46356e653. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:34.738906  1740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 41166f283aea41b0b3f5be60e8234443. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:34.739804  1801 raft_consensus.cc:493] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:34.740324  1801 raft_consensus.cc:515] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } }
I20260709 12:58:34.759469  1796 tablet_bootstrap.cc:492] T e4f4b860aab3464383ce8f9d990f4a4a P 83595ad0fd5940c0a2bfc7c61acd969a: Bootstrap starting.
I20260709 12:58:34.764393  1592 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8c04bcdff52470cb06667e46356e653. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:34.771592  1807 raft_consensus.cc:493] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:34.772280  1807 raft_consensus.cc:515] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:34.783185  1794 tablet_bootstrap.cc:492] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d: No bootstrap required, opened a new log
I20260709 12:58:34.784806  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:58:34.793762  1670 tablet_service.cc:1511] Processing CreateTablet for tablet 78cabb03415041ca9c1a5a400e506131 (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260709 12:58:34.794683  1605 raft_consensus.cc:2468] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 0.
I20260709 12:58:34.795286  1670 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 78cabb03415041ca9c1a5a400e506131. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:34.785817  1807 leader_election.cc:290] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:58:34.785951  1667 tablet_service.cc:1511] Processing CreateTablet for tablet f8ab62ffb73246dc9bcf39f7bee51147 (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260709 12:58:34.787264  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:58:34.796764  1667 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f8ab62ffb73246dc9bcf39f7bee51147. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:34.796558  1717 leader_election.cc:304] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:58:34.796976  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:58:34.797688  1679 raft_consensus.cc:2468] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 0.
I20260709 12:58:34.798007  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:58:34.798650  1753 raft_consensus.cc:2468] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 0.
I20260709 12:58:34.789937  1594 tablet_service.cc:1511] Processing CreateTablet for tablet 78cabb03415041ca9c1a5a400e506131 (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260709 12:58:34.793819  1808 raft_consensus.cc:493] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:34.799335  1808 raft_consensus.cc:515] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:34.800060  1571 leader_election.cc:304] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:58:34.802210  1808 leader_election.cc:290] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:58:34.800359  1807 raft_consensus.cc:2804] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:58:34.803256  1807 raft_consensus.cc:493] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:58:34.803779  1807 raft_consensus.cc:3060] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:34.804087  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:58:34.804533  1594 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 78cabb03415041ca9c1a5a400e506131. 1 dirs total, 0 dirs full, 0 dirs failed
W20260709 12:58:34.805636  1646 leader_election.cc:343] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:34.806700  1801 leader_election.cc:290] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:58:34.807283  1801 raft_consensus.cc:493] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:34.788784  1742 tablet_service.cc:1511] Processing CreateTablet for tablet f8ab62ffb73246dc9bcf39f7bee51147 (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260709 12:58:34.807698  1801 raft_consensus.cc:515] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:34.808696  1742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f8ab62ffb73246dc9bcf39f7bee51147. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:34.809836  1801 leader_election.cc:290] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 12:58:34.809876  1718 leader_election.cc:343] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:34.818890  1794 ts_tablet_manager.cc:1403] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent bootstrapping tablet: real 0.133s	user 0.025s	sys 0.021s
I20260709 12:58:34.822094  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:58:34.822296  1744 tablet_service.cc:1511] Processing CreateTablet for tablet 78cabb03415041ca9c1a5a400e506131 (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260709 12:58:34.823737  1744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 78cabb03415041ca9c1a5a400e506131. 1 dirs total, 0 dirs full, 0 dirs failed
W20260709 12:58:34.823822  1569 leader_election.cc:343] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:34.824007  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:58:34.826028  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:58:34.826728  1605 raft_consensus.cc:2468] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 0.
I20260709 12:58:34.827901  1643 leader_election.cc:304] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 12:58:34.829119  1808 raft_consensus.cc:2804] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:58:34.829537  1808 raft_consensus.cc:493] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:58:34.829910  1808 raft_consensus.cc:3060] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:34.835623  1796 tablet_bootstrap.cc:654] T e4f4b860aab3464383ce8f9d990f4a4a P 83595ad0fd5940c0a2bfc7c61acd969a: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:34.837479  1808 raft_consensus.cc:515] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:34.819626  1831 raft_consensus.cc:2804] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:58:34.825866  1794 raft_consensus.cc:359] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:34.859603  1808 leader_election.cc:290] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:58:34.859956  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
W20260709 12:58:34.861527  1646 leader_election.cc:343] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:34.862003  1795 tablet_bootstrap.cc:654] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:34.847420  1794 raft_consensus.cc:385] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:34.863353  1794 raft_consensus.cc:740] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Initialized, Role: FOLLOWER
I20260709 12:58:34.864210  1794 consensus_queue.cc:260] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:34.855822  1831 raft_consensus.cc:493] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:58:34.829299  1807 raft_consensus.cc:515] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:34.879220  1591 tablet_service.cc:1511] Processing CreateTablet for tablet a92588e9316e4b4aa1691e8669c839d3 (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260709 12:58:34.881063  1591 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a92588e9316e4b4aa1691e8669c839d3. 1 dirs total, 0 dirs full, 0 dirs failed
W20260709 12:58:34.873142  1571 leader_election.cc:343] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:34.864605  1665 tablet_service.cc:1511] Processing CreateTablet for tablet a92588e9316e4b4aa1691e8669c839d3 (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260709 12:58:34.883100  1669 tablet_service.cc:1511] Processing CreateTablet for tablet ef5897d71124435db68c88060b095421 (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260709 12:58:34.891227  1669 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef5897d71124435db68c88060b095421. 1 dirs total, 0 dirs full, 0 dirs failed
W20260709 12:58:34.878361  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 81900.5us
I20260709 12:58:34.894372  1744 tablet_service.cc:1511] Processing CreateTablet for tablet a92588e9316e4b4aa1691e8669c839d3 (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260709 12:58:34.894680  1740 tablet_service.cc:1511] Processing CreateTablet for tablet ef5897d71124435db68c88060b095421 (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260709 12:58:34.895848  1744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a92588e9316e4b4aa1691e8669c839d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:34.879858  1831 raft_consensus.cc:3060] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:34.901021  1807 leader_election.cc:290] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:58:34.902872  1794 ts_tablet_manager.cc:1434] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent starting tablet: real 0.080s	user 0.006s	sys 0.001s
I20260709 12:58:34.904034  1794 tablet_bootstrap.cc:492] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d: Bootstrap starting.
I20260709 12:58:34.911383  1796 tablet_bootstrap.cc:492] T e4f4b860aab3464383ce8f9d990f4a4a P 83595ad0fd5940c0a2bfc7c61acd969a: No bootstrap required, opened a new log
I20260709 12:58:34.912081  1796 ts_tablet_manager.cc:1403] T e4f4b860aab3464383ce8f9d990f4a4a P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent bootstrapping tablet: real 0.154s	user 0.010s	sys 0.028s
I20260709 12:58:34.912863  1665 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a92588e9316e4b4aa1691e8669c839d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:34.912712  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:58:34.913432  1679 raft_consensus.cc:3060] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:34.891517  1571 leader_election.cc:304] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 12:58:34.915414  1801 raft_consensus.cc:2749] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260709 12:58:34.915609  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 48358.3us
I20260709 12:58:34.916929  1796 raft_consensus.cc:359] T e4f4b860aab3464383ce8f9d990f4a4a P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:34.917433  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:58:34.917739  1796 raft_consensus.cc:385] T e4f4b860aab3464383ce8f9d990f4a4a P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:34.918095  1605 raft_consensus.cc:3060] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:34.918160  1796 raft_consensus.cc:740] T e4f4b860aab3464383ce8f9d990f4a4a P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Initialized, Role: FOLLOWER
I20260709 12:58:34.919282  1796 consensus_queue.cc:260] T e4f4b860aab3464383ce8f9d990f4a4a P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:34.924809  1796 ts_tablet_manager.cc:1434] T e4f4b860aab3464383ce8f9d990f4a4a P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent starting tablet: real 0.012s	user 0.006s	sys 0.000s
I20260709 12:58:34.925988  1796 tablet_bootstrap.cc:492] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a: Bootstrap starting.
I20260709 12:58:34.926371  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:58:34.928246  1807 raft_consensus.cc:493] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:34.927088  1604 raft_consensus.cc:3060] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:34.931191  1807 raft_consensus.cc:515] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:34.933729  1807 leader_election.cc:290] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:58:34.934594  1796 tablet_bootstrap.cc:654] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:34.935014  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:58:34.935698  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:58:34.936484  1603 raft_consensus.cc:2468] T e4f4b860aab3464383ce8f9d990f4a4a P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 0.
W20260709 12:58:34.936671  1718 leader_election.cc:343] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:34.939324  1717 leader_election.cc:304] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 12:58:34.942258  1592 tablet_service.cc:1511] Processing CreateTablet for tablet ef5897d71124435db68c88060b095421 (DEFAULT_TABLE table=test-workload [id=75d77b57b3f34786b9c215c4f11bd727]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260709 12:58:34.942694  1605 raft_consensus.cc:2468] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 1.
I20260709 12:58:34.943580  1740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef5897d71124435db68c88060b095421. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:34.944173  1679 raft_consensus.cc:2468] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 1.
I20260709 12:58:34.944583  1717 leader_election.cc:304] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:58:34.930068  1831 raft_consensus.cc:515] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } }
I20260709 12:58:34.946605  1830 raft_consensus.cc:2804] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:58:34.947033  1830 raft_consensus.cc:493] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:58:34.947374  1830 raft_consensus.cc:3060] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:34.947527  1794 tablet_bootstrap.cc:654] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:34.949434  1592 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef5897d71124435db68c88060b095421. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:34.951848  1796 tablet_bootstrap.cc:492] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a: No bootstrap required, opened a new log
I20260709 12:58:34.952358  1796 ts_tablet_manager.cc:1403] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent bootstrapping tablet: real 0.027s	user 0.010s	sys 0.007s
I20260709 12:58:34.955716  1796 raft_consensus.cc:359] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:34.956459  1796 raft_consensus.cc:385] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:34.956763  1796 raft_consensus.cc:740] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Initialized, Role: FOLLOWER
I20260709 12:58:34.957520  1796 consensus_queue.cc:260] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:34.960350  1796 ts_tablet_manager.cc:1434] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent starting tablet: real 0.008s	user 0.008s	sys 0.000s
I20260709 12:58:34.961382  1796 tablet_bootstrap.cc:492] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a: Bootstrap starting.
I20260709 12:58:34.946609  1807 raft_consensus.cc:2804] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:58:34.963954  1831 leader_election.cc:290] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:58:34.967299  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:58:34.967931  1753 raft_consensus.cc:3060] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:34.968350  1796 tablet_bootstrap.cc:654] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:34.975612  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:58:34.976236  1679 raft_consensus.cc:3060] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:34.975236  1604 raft_consensus.cc:2468] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 1.
I20260709 12:58:34.983178  1807 raft_consensus.cc:697] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 1 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 12:58:34.983628  1643 leader_election.cc:304] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 12:58:34.984340  1807 consensus_queue.cc:237] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:34.984903  1808 raft_consensus.cc:2804] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:58:34.985463  1808 raft_consensus.cc:697] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 1 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 12:58:34.986334  1808 consensus_queue.cc:237] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:34.988281  1679 raft_consensus.cc:2468] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 1.
I20260709 12:58:34.989605  1569 leader_election.cc:304] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 12:58:34.990532  1831 raft_consensus.cc:2804] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:58:34.991041  1753 raft_consensus.cc:2468] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 1.
I20260709 12:58:34.991053  1831 raft_consensus.cc:697] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 12:58:34.992177  1831 consensus_queue.cc:237] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } }
I20260709 12:58:34.995208  1795 tablet_bootstrap.cc:492] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701: No bootstrap required, opened a new log
I20260709 12:58:34.995796  1795 ts_tablet_manager.cc:1403] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701: Time spent bootstrapping tablet: real 0.281s	user 0.022s	sys 0.012s
I20260709 12:58:35.005043  1794 tablet_bootstrap.cc:492] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d: No bootstrap required, opened a new log
I20260709 12:58:35.005744  1794 ts_tablet_manager.cc:1403] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent bootstrapping tablet: real 0.102s	user 0.004s	sys 0.011s
I20260709 12:58:35.000792  1795 raft_consensus.cc:359] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:35.007122  1795 raft_consensus.cc:385] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:35.007602  1795 raft_consensus.cc:740] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Initialized, Role: FOLLOWER
I20260709 12:58:35.008664  1795 consensus_queue.cc:260] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:35.011847  1795 ts_tablet_manager.cc:1434] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701: Time spent starting tablet: real 0.015s	user 0.007s	sys 0.001s
I20260709 12:58:35.013448  1795 tablet_bootstrap.cc:492] T 592fa647ae2545e09ad5ee25f8cb658e P af520b67945d443b901bbeca37cc5701: Bootstrap starting.
I20260709 12:58:35.007099  1830 raft_consensus.cc:515] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:35.015978  1794 raft_consensus.cc:359] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:35.024260  1794 raft_consensus.cc:385] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:35.024534  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:58:35.024690  1794 raft_consensus.cc:740] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Initialized, Role: FOLLOWER
I20260709 12:58:35.025245  1604 raft_consensus.cc:3060] T e4f4b860aab3464383ce8f9d990f4a4a P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:35.025388  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
W20260709 12:58:35.027015  1718 leader_election.cc:343] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:35.027613  1830 leader_election.cc:290] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:58:35.030719  1794 consensus_queue.cc:260] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:35.040247  1604 raft_consensus.cc:2468] T e4f4b860aab3464383ce8f9d990f4a4a P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 1.
I20260709 12:58:35.043555  1503 catalog_manager.cc:5697] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 reported cstate change: term changed from 0 to 1, leader changed from <none> to af520b67945d443b901bbeca37cc5701 (127.0.107.195). New cstate: current_term: 1 leader_uuid: "af520b67945d443b901bbeca37cc5701" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: HEALTHY } } }
I20260709 12:58:35.045333  1795 tablet_bootstrap.cc:654] T 592fa647ae2545e09ad5ee25f8cb658e P af520b67945d443b901bbeca37cc5701: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:35.049150  1794 ts_tablet_manager.cc:1434] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent starting tablet: real 0.043s	user 0.010s	sys 0.000s
I20260709 12:58:35.048823  1502 catalog_manager.cc:5697] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d reported cstate change: term changed from 0 to 1, leader changed from <none> to 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194). New cstate: current_term: 1 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } }
I20260709 12:58:35.050643  1794 tablet_bootstrap.cc:492] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d: Bootstrap starting.
I20260709 12:58:35.057600  1717 leader_election.cc:304] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 12:58:35.058801  1807 raft_consensus.cc:2804] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:58:35.059334  1807 raft_consensus.cc:697] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 1 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 12:58:35.060240  1807 consensus_queue.cc:237] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:35.061076  1794 tablet_bootstrap.cc:654] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:35.064647  1796 tablet_bootstrap.cc:492] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a: No bootstrap required, opened a new log
I20260709 12:58:35.065081  1796 ts_tablet_manager.cc:1403] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent bootstrapping tablet: real 0.104s	user 0.025s	sys 0.024s
I20260709 12:58:35.068003  1826 raft_consensus.cc:493] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:35.069020  1826 raft_consensus.cc:515] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:35.067941  1796 raft_consensus.cc:359] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:35.070194  1796 raft_consensus.cc:385] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:35.070523  1796 raft_consensus.cc:740] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Initialized, Role: FOLLOWER
I20260709 12:58:35.071386  1796 consensus_queue.cc:260] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:35.073988  1796 ts_tablet_manager.cc:1434] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent starting tablet: real 0.009s	user 0.008s	sys 0.000s
I20260709 12:58:35.075178  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:58:35.074215  1501 catalog_manager.cc:5697] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 0 to 1, leader changed from <none> to 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193). New cstate: current_term: 1 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: UNKNOWN } } }
I20260709 12:58:35.076584  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:58:35.077227  1753 raft_consensus.cc:2468] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 0.
I20260709 12:58:35.078382  1646 leader_election.cc:304] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:58:35.080477  1808 raft_consensus.cc:2804] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:58:35.080950  1808 raft_consensus.cc:493] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:58:35.081437  1808 raft_consensus.cc:3060] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:35.083161  1796 tablet_bootstrap.cc:492] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a: Bootstrap starting.
I20260709 12:58:35.085788  1795 tablet_bootstrap.cc:492] T 592fa647ae2545e09ad5ee25f8cb658e P af520b67945d443b901bbeca37cc5701: No bootstrap required, opened a new log
W20260709 12:58:35.083050  1643 leader_election.cc:343] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:35.090515  1795 ts_tablet_manager.cc:1403] T 592fa647ae2545e09ad5ee25f8cb658e P af520b67945d443b901bbeca37cc5701: Time spent bootstrapping tablet: real 0.073s	user 0.014s	sys 0.013s
I20260709 12:58:35.078802  1826 leader_election.cc:290] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:58:35.094226  1796 tablet_bootstrap.cc:654] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:35.096653  1795 raft_consensus.cc:359] T 592fa647ae2545e09ad5ee25f8cb658e P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:35.097397  1795 raft_consensus.cc:385] T 592fa647ae2545e09ad5ee25f8cb658e P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:35.097725  1795 raft_consensus.cc:740] T 592fa647ae2545e09ad5ee25f8cb658e P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Initialized, Role: FOLLOWER
I20260709 12:58:35.106323  1794 tablet_bootstrap.cc:492] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d: No bootstrap required, opened a new log
I20260709 12:58:35.107138  1794 ts_tablet_manager.cc:1403] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent bootstrapping tablet: real 0.057s	user 0.022s	sys 0.009s
I20260709 12:58:35.101415  1795 consensus_queue.cc:260] T 592fa647ae2545e09ad5ee25f8cb658e P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:35.111528  1794 raft_consensus.cc:359] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:35.112862  1794 raft_consensus.cc:385] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:35.113216  1794 raft_consensus.cc:740] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Initialized, Role: FOLLOWER
I20260709 12:58:35.112339  1808 raft_consensus.cc:515] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:35.114250  1795 ts_tablet_manager.cc:1434] T 592fa647ae2545e09ad5ee25f8cb658e P af520b67945d443b901bbeca37cc5701: Time spent starting tablet: real 0.023s	user 0.007s	sys 0.004s
I20260709 12:58:35.114176  1794 consensus_queue.cc:260] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:35.115419  1795 tablet_bootstrap.cc:492] T 84e78b66ceec4f759ae9e101a5469e4c P af520b67945d443b901bbeca37cc5701: Bootstrap starting.
I20260709 12:58:35.116839  1794 ts_tablet_manager.cc:1434] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent starting tablet: real 0.009s	user 0.007s	sys 0.000s
I20260709 12:58:35.117951  1794 tablet_bootstrap.cc:492] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d: Bootstrap starting.
I20260709 12:58:35.118063  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
W20260709 12:58:35.120735  1643 leader_election.cc:343] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:35.124574  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:58:35.125523  1753 raft_consensus.cc:3060] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:35.128252  1794 tablet_bootstrap.cc:654] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:35.129732  1795 tablet_bootstrap.cc:654] T 84e78b66ceec4f759ae9e101a5469e4c P af520b67945d443b901bbeca37cc5701: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:35.134591  1808 leader_election.cc:290] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:58:35.150676  1796 tablet_bootstrap.cc:492] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a: No bootstrap required, opened a new log
I20260709 12:58:35.151361  1796 ts_tablet_manager.cc:1403] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent bootstrapping tablet: real 0.068s	user 0.012s	sys 0.020s
I20260709 12:58:35.152951  1795 tablet_bootstrap.cc:492] T 84e78b66ceec4f759ae9e101a5469e4c P af520b67945d443b901bbeca37cc5701: No bootstrap required, opened a new log
I20260709 12:58:35.153790  1795 ts_tablet_manager.cc:1403] T 84e78b66ceec4f759ae9e101a5469e4c P af520b67945d443b901bbeca37cc5701: Time spent bootstrapping tablet: real 0.039s	user 0.008s	sys 0.012s
I20260709 12:58:35.156682  1796 raft_consensus.cc:359] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:35.158301  1796 raft_consensus.cc:385] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:35.158686  1796 raft_consensus.cc:740] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Initialized, Role: FOLLOWER
I20260709 12:58:35.159659  1795 raft_consensus.cc:359] T 84e78b66ceec4f759ae9e101a5469e4c P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:35.146001  1753 raft_consensus.cc:2468] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 1.
I20260709 12:58:35.160899  1795 raft_consensus.cc:385] T 84e78b66ceec4f759ae9e101a5469e4c P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:35.159970  1796 consensus_queue.cc:260] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:35.165483  1796 ts_tablet_manager.cc:1434] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent starting tablet: real 0.014s	user 0.008s	sys 0.000s
I20260709 12:58:35.162858  1646 leader_election.cc:304] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:58:35.167551  1796 tablet_bootstrap.cc:492] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a: Bootstrap starting.
I20260709 12:58:35.167641  1808 raft_consensus.cc:2804] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:58:35.168467  1808 raft_consensus.cc:697] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 12:58:35.169828  1808 consensus_queue.cc:237] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:35.161487  1795 raft_consensus.cc:740] T 84e78b66ceec4f759ae9e101a5469e4c P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Initialized, Role: FOLLOWER
I20260709 12:58:35.175918  1795 consensus_queue.cc:260] T 84e78b66ceec4f759ae9e101a5469e4c P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:35.179158  1796 tablet_bootstrap.cc:654] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:35.188680  1794 tablet_bootstrap.cc:492] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d: No bootstrap required, opened a new log
I20260709 12:58:35.189294  1794 ts_tablet_manager.cc:1403] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent bootstrapping tablet: real 0.072s	user 0.025s	sys 0.034s
I20260709 12:58:35.193058  1794 raft_consensus.cc:359] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:35.193828  1794 raft_consensus.cc:385] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:35.194221  1794 raft_consensus.cc:740] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Initialized, Role: FOLLOWER
I20260709 12:58:35.191115  1795 ts_tablet_manager.cc:1434] T 84e78b66ceec4f759ae9e101a5469e4c P af520b67945d443b901bbeca37cc5701: Time spent starting tablet: real 0.037s	user 0.000s	sys 0.013s
I20260709 12:58:35.196053  1795 tablet_bootstrap.cc:492] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701: Bootstrap starting.
I20260709 12:58:35.195240  1794 consensus_queue.cc:260] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:35.199839  1794 ts_tablet_manager.cc:1434] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent starting tablet: real 0.010s	user 0.003s	sys 0.003s
I20260709 12:58:35.198405  1503 catalog_manager.cc:5697] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 reported cstate change: term changed from 0 to 1, leader changed from <none> to af520b67945d443b901bbeca37cc5701 (127.0.107.195). New cstate: current_term: 1 leader_uuid: "af520b67945d443b901bbeca37cc5701" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: HEALTHY } } }
I20260709 12:58:35.201547  1794 tablet_bootstrap.cc:492] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d: Bootstrap starting.
I20260709 12:58:35.206295  1808 raft_consensus.cc:493] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:35.207173  1808 raft_consensus.cc:515] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:35.213935  1794 tablet_bootstrap.cc:654] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:35.214597  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:58:35.215600  1604 raft_consensus.cc:2468] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 0.
I20260709 12:58:35.217161  1643 leader_election.cc:304] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 12:58:35.218915  1826 raft_consensus.cc:2804] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:58:35.219594  1826 raft_consensus.cc:493] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:58:35.220095  1826 raft_consensus.cc:3060] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:35.221410  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 12:58:35.224573  1646 leader_election.cc:343] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:35.231848  1502 catalog_manager.cc:5697] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d reported cstate change: term changed from 0 to 1, leader changed from <none> to 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194). New cstate: current_term: 1 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } }
I20260709 12:58:35.234524  1808 leader_election.cc:290] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:58:35.241132  1808 raft_consensus.cc:493] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:35.241952  1808 raft_consensus.cc:515] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:35.243806  1795 tablet_bootstrap.cc:654] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:35.250103  1796 tablet_bootstrap.cc:492] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a: No bootstrap required, opened a new log
I20260709 12:58:35.250629  1796 ts_tablet_manager.cc:1403] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent bootstrapping tablet: real 0.083s	user 0.024s	sys 0.006s
I20260709 12:58:35.253388  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:58:35.254403  1604 raft_consensus.cc:2468] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 0.
I20260709 12:58:35.256232  1796 raft_consensus.cc:359] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:35.255702  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:58:35.257159  1796 raft_consensus.cc:385] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:35.257551  1796 raft_consensus.cc:740] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Initialized, Role: FOLLOWER
I20260709 12:58:35.260198  1796 consensus_queue.cc:260] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:35.262573  1808 leader_election.cc:290] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:58:35.264848  1643 leader_election.cc:304] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
W20260709 12:58:35.266458  1646 leader_election.cc:343] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:35.269225  1808 raft_consensus.cc:2804] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:58:35.269873  1808 raft_consensus.cc:493] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:58:35.271867  1796 ts_tablet_manager.cc:1434] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent starting tablet: real 0.021s	user 0.008s	sys 0.001s
I20260709 12:58:35.272251  1808 raft_consensus.cc:3060] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:35.275785  1796 tablet_bootstrap.cc:492] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a: Bootstrap starting.
I20260709 12:58:35.274394  1826 raft_consensus.cc:515] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:35.279917  1795 tablet_bootstrap.cc:492] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701: No bootstrap required, opened a new log
I20260709 12:58:35.280205  1794 tablet_bootstrap.cc:492] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d: No bootstrap required, opened a new log
I20260709 12:58:35.283756  1794 ts_tablet_manager.cc:1403] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d: Time spent bootstrapping tablet: real 0.082s	user 0.017s	sys 0.035s
I20260709 12:58:35.284272  1831 raft_consensus.cc:493] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:35.284910  1831 raft_consensus.cc:515] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:35.286201  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:58:35.286988  1604 raft_consensus.cc:3060] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:35.287941  1794 raft_consensus.cc:359] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:35.288682  1794 raft_consensus.cc:385] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:35.288969  1794 raft_consensus.cc:740] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Initialized, Role: FOLLOWER
I20260709 12:58:35.289428  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:58:35.289759  1794 consensus_queue.cc:260] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:35.291366  1795 ts_tablet_manager.cc:1403] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701: Time spent bootstrapping tablet: real 0.096s	user 0.006s	sys 0.024s
I20260709 12:58:35.290355  1679 raft_consensus.cc:2468] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 0.
I20260709 12:58:35.293335  1569 leader_election.cc:304] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 12:58:35.294780  1801 raft_consensus.cc:2804] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:58:35.295333  1801 raft_consensus.cc:493] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:58:35.295854  1801 raft_consensus.cc:3060] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:35.295625  1795 raft_consensus.cc:359] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:35.296521  1795 raft_consensus.cc:385] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:35.296875  1795 raft_consensus.cc:740] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Initialized, Role: FOLLOWER
I20260709 12:58:35.297737  1795 consensus_queue.cc:260] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:35.299451  1831 leader_election.cc:290] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:58:35.301882  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:58:35.281591  1796 tablet_bootstrap.cc:654] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:35.304615  1795 ts_tablet_manager.cc:1434] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701: Time spent starting tablet: real 0.013s	user 0.008s	sys 0.000s
W20260709 12:58:35.306545  1571 leader_election.cc:343] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:58:35.315701  1551 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:58:35.317209  1826 leader_election.cc:290] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:58:35.321329  1826 raft_consensus.cc:493] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:35.322045  1826 raft_consensus.cc:515] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:35.324342  1795 tablet_bootstrap.cc:492] T d21485c094584689917b2a97b2d3c624 P af520b67945d443b901bbeca37cc5701: Bootstrap starting.
I20260709 12:58:35.326629  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:58:35.327062  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:58:35.328118  1605 raft_consensus.cc:2393] T e4f4b860aab3464383ce8f9d990f4a4a P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 1: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
W20260709 12:58:35.329298  1646 leader_election.cc:343] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:35.330545  1807 raft_consensus.cc:493] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:35.331230  1807 raft_consensus.cc:515] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:35.332074  1826 leader_election.cc:290] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:58:35.325525  1604 raft_consensus.cc:2468] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 1.
I20260709 12:58:35.337956  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:58:35.338721  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:58:35.316998  1801 raft_consensus.cc:515] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:35.339715  1605 raft_consensus.cc:2468] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 0.
I20260709 12:58:35.340518  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:58:35.341313  1794 ts_tablet_manager.cc:1434] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d: Time spent starting tablet: real 0.057s	user 0.018s	sys 0.027s
I20260709 12:58:35.341569  1643 leader_election.cc:304] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 12:58:35.342620  1794 tablet_bootstrap.cc:492] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d: Bootstrap starting.
I20260709 12:58:35.342785  1646 leader_election.cc:304] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 12:58:35.343402  1717 leader_election.cc:304] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:58:35.344136  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:58:35.344643  1830 raft_consensus.cc:2804] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:58:35.344244  1808 raft_consensus.cc:515] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:35.345003  1680 raft_consensus.cc:3060] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:35.345144  1830 raft_consensus.cc:493] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:58:35.345721  1830 raft_consensus.cc:3060] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:35.346037  1679 raft_consensus.cc:2393] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 1: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 12:58:35.350803  1794 tablet_bootstrap.cc:654] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:35.354175  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:58:35.355093  1605 raft_consensus.cc:3060] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:35.355687  1857 raft_consensus.cc:2804] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:58:35.356917  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
W20260709 12:58:35.359375  1571 leader_election.cc:343] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:35.357169  1826 raft_consensus.cc:3060] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:35.338284  1807 leader_election.cc:290] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:58:35.363164  1801 leader_election.cc:290] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:58:35.364037  1796 tablet_bootstrap.cc:492] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a: No bootstrap required, opened a new log
I20260709 12:58:35.364656  1796 ts_tablet_manager.cc:1403] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent bootstrapping tablet: real 0.089s	user 0.017s	sys 0.002s
I20260709 12:58:35.366354  1830 raft_consensus.cc:515] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:35.375944  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:58:35.377203  1753 raft_consensus.cc:2393] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 1: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 12:58:35.383226  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:58:35.385380  1680 raft_consensus.cc:2468] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 1.
I20260709 12:58:35.387157  1569 leader_election.cc:304] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 12:58:35.388918  1808 leader_election.cc:290] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:58:35.369931  1605 raft_consensus.cc:2468] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 1.
I20260709 12:58:35.388994  1801 raft_consensus.cc:2804] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:58:35.392072  1807 raft_consensus.cc:493] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:35.369037  1796 raft_consensus.cc:359] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:35.392855  1796 raft_consensus.cc:385] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:35.393128  1795 tablet_bootstrap.cc:654] T d21485c094584689917b2a97b2d3c624 P af520b67945d443b901bbeca37cc5701: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:35.398748  1831 consensus_queue.cc:1048] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:58:35.400578  1794 tablet_bootstrap.cc:492] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d: No bootstrap required, opened a new log
I20260709 12:58:35.392251  1830 leader_election.cc:290] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:58:35.393252  1796 raft_consensus.cc:740] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Initialized, Role: FOLLOWER
I20260709 12:58:35.401381  1831 raft_consensus.cc:493] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:35.401824  1794 ts_tablet_manager.cc:1403] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d: Time spent bootstrapping tablet: real 0.059s	user 0.029s	sys 0.022s
I20260709 12:58:35.393795  1857 raft_consensus.cc:697] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 12:58:35.393822  1801 raft_consensus.cc:697] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 12:58:35.403903  1801 consensus_queue.cc:237] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:35.403708  1857 consensus_queue.cc:237] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:35.394312  1754 raft_consensus.cc:3060] T 592fa647ae2545e09ad5ee25f8cb658e P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:35.406323  1794 raft_consensus.cc:359] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:35.407478  1794 raft_consensus.cc:385] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:35.407930  1794 raft_consensus.cc:740] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Initialized, Role: FOLLOWER
I20260709 12:58:35.409097  1794 consensus_queue.cc:260] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:35.402155  1831 raft_consensus.cc:515] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:35.412823  1794 ts_tablet_manager.cc:1434] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d: Time spent starting tablet: real 0.010s	user 0.008s	sys 0.000s
I20260709 12:58:35.392859  1807 raft_consensus.cc:515] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:35.402120  1796 consensus_queue.cc:260] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:35.420131  1796 ts_tablet_manager.cc:1434] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent starting tablet: real 0.055s	user 0.008s	sys 0.002s
I20260709 12:58:35.417574  1807 leader_election.cc:290] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:58:35.423020  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:58:35.424959  1605 raft_consensus.cc:2468] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 0.
I20260709 12:58:35.414247  1831 leader_election.cc:290] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:58:35.414215  1794 tablet_bootstrap.cc:492] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d: Bootstrap starting.
I20260709 12:58:35.434638  1717 leader_election.cc:304] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:58:35.437427  1796 tablet_bootstrap.cc:492] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a: Bootstrap starting.
I20260709 12:58:35.438593  1830 raft_consensus.cc:2804] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:58:35.439141  1830 raft_consensus.cc:493] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:58:35.443650  1826 raft_consensus.cc:2749] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:58:35.443982  1830 raft_consensus.cc:3060] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:35.447242  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:58:35.454373  1831 raft_consensus.cc:493] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:35.475063  1751 raft_consensus.cc:2393] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 1: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 12:58:35.478181  1836 raft_consensus.cc:493] T 84e78b66ceec4f759ae9e101a5469e4c P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:35.478868  1836 raft_consensus.cc:515] T 84e78b66ceec4f759ae9e101a5469e4c P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:35.480106  1796 tablet_bootstrap.cc:654] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:35.484009  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:58:35.486358  1836 leader_election.cc:290] T 84e78b66ceec4f759ae9e101a5469e4c P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:58:35.486639  1831 raft_consensus.cc:515] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:35.488443  1605 raft_consensus.cc:2468] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 0.
I20260709 12:58:35.498981  1643 leader_election.cc:304] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 12:58:35.501690  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:58:35.503510  1794 tablet_bootstrap.cc:654] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:35.490335  1831 leader_election.cc:290] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:58:35.491765  1717 leader_election.cc:304] T 84e78b66ceec4f759ae9e101a5469e4c P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:58:35.508442  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:58:35.506701  1857 raft_consensus.cc:2804] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:58:35.509132  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:58:35.520709  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:58:35.521775  1678 raft_consensus.cc:2468] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 0.
I20260709 12:58:35.518741  1836 raft_consensus.cc:2804] T 84e78b66ceec4f759ae9e101a5469e4c P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:58:35.522857  1836 raft_consensus.cc:493] T 84e78b66ceec4f759ae9e101a5469e4c P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:58:35.524168  1836 raft_consensus.cc:3060] T 84e78b66ceec4f759ae9e101a5469e4c P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:35.524606  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:58:35.526679  1676 raft_consensus.cc:2468] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 0.
I20260709 12:58:35.528739  1569 leader_election.cc:304] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 12:58:35.530211  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:58:35.533751  1677 raft_consensus.cc:2468] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 0.
I20260709 12:58:35.523279  1857 raft_consensus.cc:697] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 12:58:35.537618  1569 leader_election.cc:304] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 12:58:35.539611  1857 consensus_queue.cc:237] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:35.533008  1801 raft_consensus.cc:2804] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:58:35.541271  1826 consensus_queue.cc:1048] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:58:35.541749  1862 raft_consensus.cc:3060] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:35.543254  1801 raft_consensus.cc:493] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:58:35.544674  1801 raft_consensus.cc:3060] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:35.546532  1807 consensus_queue.cc:1048] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20260709 12:58:35.533949  1830 raft_consensus.cc:515] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:35.552304  1795 tablet_bootstrap.cc:492] T d21485c094584689917b2a97b2d3c624 P af520b67945d443b901bbeca37cc5701: No bootstrap required, opened a new log
I20260709 12:58:35.552971  1795 ts_tablet_manager.cc:1403] T d21485c094584689917b2a97b2d3c624 P af520b67945d443b901bbeca37cc5701: Time spent bootstrapping tablet: real 0.229s	user 0.034s	sys 0.052s
I20260709 12:58:35.557477  1830 leader_election.cc:290] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:58:35.561434  1861 consensus_queue.cc:1048] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.015s
I20260709 12:58:35.574961  1796 tablet_bootstrap.cc:492] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a: No bootstrap required, opened a new log
I20260709 12:58:35.575510  1796 ts_tablet_manager.cc:1403] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent bootstrapping tablet: real 0.139s	user 0.020s	sys 0.012s
I20260709 12:58:35.583956  1831 consensus_queue.cc:1048] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:58:35.585633  1795 raft_consensus.cc:359] T d21485c094584689917b2a97b2d3c624 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:35.586225  1795 raft_consensus.cc:385] T d21485c094584689917b2a97b2d3c624 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:35.586510  1795 raft_consensus.cc:740] T d21485c094584689917b2a97b2d3c624 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Initialized, Role: FOLLOWER
I20260709 12:58:35.588681  1795 consensus_queue.cc:260] T d21485c094584689917b2a97b2d3c624 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:35.590233  1795 ts_tablet_manager.cc:1434] T d21485c094584689917b2a97b2d3c624 P af520b67945d443b901bbeca37cc5701: Time spent starting tablet: real 0.037s	user 0.008s	sys 0.000s
I20260709 12:58:35.591118  1795 tablet_bootstrap.cc:492] T 388afdca3ce84bf0a8ce3331ed0d9b2f P af520b67945d443b901bbeca37cc5701: Bootstrap starting.
I20260709 12:58:35.595283  1862 raft_consensus.cc:2749] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:58:35.595113  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:58:35.595990  1678 raft_consensus.cc:3060] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:35.603840  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:58:35.604772  1604 raft_consensus.cc:3060] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:35.610858  1796 raft_consensus.cc:359] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:35.611850  1796 raft_consensus.cc:385] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:35.612077  1796 raft_consensus.cc:740] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Initialized, Role: FOLLOWER
I20260709 12:58:35.622849  1831 raft_consensus.cc:493] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:35.660910  1861 raft_consensus.cc:493] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:35.661827  1861 raft_consensus.cc:515] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:35.647578  1831 raft_consensus.cc:515] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:35.593806  1502 catalog_manager.cc:5697] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d reported cstate change: term changed from 0 to 1, leader changed from <none> to 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194). New cstate: current_term: 1 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } }
I20260709 12:58:35.682425  1796 consensus_queue.cc:260] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:35.693293  1796 ts_tablet_manager.cc:1434] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent starting tablet: real 0.117s	user 0.011s	sys 0.000s
I20260709 12:58:35.694605  1796 tablet_bootstrap.cc:492] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a: Bootstrap starting.
I20260709 12:58:35.696086  1718 leader_election.cc:304] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:58:35.686489  1861 leader_election.cc:290] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:58:35.717057  1869 raft_consensus.cc:2749] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 12:58:35.716954  1873 consensus_queue.cc:1048] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.022s
I20260709 12:58:35.722826  1868 consensus_queue.cc:1048] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.050s
I20260709 12:58:35.706729  1807 raft_consensus.cc:493] T 592fa647ae2545e09ad5ee25f8cb658e P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 12:58:35.730247  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:58:35.730782  1807 raft_consensus.cc:515] T 592fa647ae2545e09ad5ee25f8cb658e P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:35.733170  1831 leader_election.cc:290] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 12:58:35.734489  1646 leader_election.cc:343] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:58:35.713682  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 90420.9us
I20260709 12:58:35.837702  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:58:35.857226  1503 catalog_manager.cc:5697] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 0 to 1, leader changed from <none> to 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193). New cstate: current_term: 1 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } }
I20260709 12:58:35.861341  1826 consensus_queue.cc:1048] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260709 12:58:35.866761  1872 log.cc:927] Time spent T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.092s	user 0.000s	sys 0.000s
I20260709 12:58:35.902859  1794 tablet_bootstrap.cc:492] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d: No bootstrap required, opened a new log
I20260709 12:58:35.903301  1794 ts_tablet_manager.cc:1403] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d: Time spent bootstrapping tablet: real 0.489s	user 0.018s	sys 0.029s
W20260709 12:58:35.904147  1871 log.cc:927] Time spent T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.169s	user 0.004s	sys 0.002s
I20260709 12:58:35.891434  1836 raft_consensus.cc:515] T 84e78b66ceec4f759ae9e101a5469e4c P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:35.892062  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:58:35.885730  1801 raft_consensus.cc:515] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:35.916987  1751 raft_consensus.cc:2393] T 84e78b66ceec4f759ae9e101a5469e4c P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 1: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
W20260709 12:58:35.926914  1569 leader_election.cc:343] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:35.931033  1801 leader_election.cc:290] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:58:35.938954  1604 raft_consensus.cc:2468] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 1.
I20260709 12:58:35.942065  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:58:35.943077  1680 raft_consensus.cc:3060] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:35.951126  1678 raft_consensus.cc:2468] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 1.
W20260709 12:58:35.964475  1646 consensus_peers.cc:597] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d -> Peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Couldn't send request to peer af520b67945d443b901bbeca37cc5701. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:58:35.959519  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:58:35.969313  1836 leader_election.cc:290] T 84e78b66ceec4f759ae9e101a5469e4c P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:58:35.970508  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:58:35.962181  1807 leader_election.cc:290] T 592fa647ae2545e09ad5ee25f8cb658e P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:58:36.073220  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:58:36.078475  1795 tablet_bootstrap.cc:654] T 388afdca3ce84bf0a8ce3331ed0d9b2f P af520b67945d443b901bbeca37cc5701: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:36.001474  1680 raft_consensus.cc:2468] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 1.
W20260709 12:58:36.044503  1571 leader_election.cc:343] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:35.999146  1836 consensus_queue.cc:1048] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.092s
I20260709 12:58:35.997638  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:58:36.037356  1801 raft_consensus.cc:493] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:36.093345  1794 raft_consensus.cc:359] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:36.095408  1794 raft_consensus.cc:385] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:35.977542  1717 leader_election.cc:304] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:58:36.088575  1807 raft_consensus.cc:493] T d21485c094584689917b2a97b2d3c624 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:36.095880  1794 raft_consensus.cc:740] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Initialized, Role: FOLLOWER
I20260709 12:58:36.044967  1874 consensus_queue.cc:1048] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.051s
I20260709 12:58:36.092208  1569 leader_election.cc:304] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
W20260709 12:58:36.098538  1643 leader_election.cc:343] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:36.101322  1794 consensus_queue.cc:260] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:36.101843  1796 tablet_bootstrap.cc:654] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:36.104513  1794 ts_tablet_manager.cc:1434] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d: Time spent starting tablet: real 0.201s	user 0.011s	sys 0.013s
I20260709 12:58:36.100731  1807 raft_consensus.cc:515] T d21485c094584689917b2a97b2d3c624 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:36.107509  1877 raft_consensus.cc:2804] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:58:36.108211  1877 raft_consensus.cc:697] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 12:58:36.103513  1751 raft_consensus.cc:2393] T 84e78b66ceec4f759ae9e101a5469e4c P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 1: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 12:58:36.101155  1826 consensus_queue.cc:1048] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.011s
I20260709 12:58:36.105145  1643 leader_election.cc:304] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 12:58:36.112994  1861 raft_consensus.cc:2749] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:58:36.113255  1807 leader_election.cc:290] T d21485c094584689917b2a97b2d3c624 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:58:36.094153  1571 leader_election.cc:304] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 12:58:36.095244  1801 raft_consensus.cc:515] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:36.103314  1836 raft_consensus.cc:2804] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:58:36.116359  1836 raft_consensus.cc:697] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 1 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
W20260709 12:58:36.102428  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 125032us
I20260709 12:58:36.117115  1861 consensus_queue.cc:1048] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:58:36.109627  1877 consensus_queue.cc:237] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:36.121766  1862 raft_consensus.cc:2749] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:58:36.121889  1836 consensus_queue.cc:237] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 12:58:36.128916  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 84844.8us
I20260709 12:58:36.171543  1794 tablet_bootstrap.cc:492] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d: Bootstrap starting.
W20260709 12:58:36.173323  1883 log.cc:927] Time spent T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.094s	user 0.002s	sys 0.002s
W20260709 12:58:36.173451  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 82491us
I20260709 12:58:36.174719  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:58:36.176209  1674 raft_consensus.cc:2468] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 0.
W20260709 12:58:36.179189  1881 log.cc:927] Time spent T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.152s	user 0.004s	sys 0.005s
I20260709 12:58:36.181929  1801 leader_election.cc:290] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:58:36.185812  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:58:36.187999  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:58:36.189951  1796 tablet_bootstrap.cc:492] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a: No bootstrap required, opened a new log
I20260709 12:58:36.190742  1796 ts_tablet_manager.cc:1403] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent bootstrapping tablet: real 0.496s	user 0.010s	sys 0.028s
I20260709 12:58:36.198975  1569 leader_election.cc:304] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 12:58:36.200444  1862 raft_consensus.cc:2804] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:58:36.201006  1862 raft_consensus.cc:493] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:58:36.201581  1862 raft_consensus.cc:3060] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:36.210659  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:58:36.239612  1751 raft_consensus.cc:3060] T d21485c094584689917b2a97b2d3c624 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:36.239764  1717 leader_election.cc:304] T 592fa647ae2545e09ad5ee25f8cb658e P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:58:36.241518  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:58:36.244474  1836 raft_consensus.cc:2749] T 592fa647ae2545e09ad5ee25f8cb658e P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:58:36.245507  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:58:36.248355  1826 consensus_queue.cc:1048] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20260709 12:58:36.258534  1717 leader_election.cc:304] T 84e78b66ceec4f759ae9e101a5469e4c P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:58:36.258080  1796 raft_consensus.cc:359] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:36.260289  1796 raft_consensus.cc:385] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:36.260499  1796 raft_consensus.cc:740] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Initialized, Role: FOLLOWER
W20260709 12:58:36.263527  1571 leader_election.cc:343] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:36.263438  1717 leader_election.cc:304] T d21485c094584689917b2a97b2d3c624 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:58:36.265935  1796 consensus_queue.cc:260] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:36.274688  1857 consensus_queue.cc:1048] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:58:36.265491  1807 raft_consensus.cc:2749] T 84e78b66ceec4f759ae9e101a5469e4c P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 12:58:36.277248  1862 raft_consensus.cc:515] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:36.303972  1795 tablet_bootstrap.cc:492] T 388afdca3ce84bf0a8ce3331ed0d9b2f P af520b67945d443b901bbeca37cc5701: No bootstrap required, opened a new log
I20260709 12:58:36.306903  1795 ts_tablet_manager.cc:1403] T 388afdca3ce84bf0a8ce3331ed0d9b2f P af520b67945d443b901bbeca37cc5701: Time spent bootstrapping tablet: real 0.716s	user 0.029s	sys 0.041s
I20260709 12:58:36.308459  1796 ts_tablet_manager.cc:1434] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent starting tablet: real 0.117s	user 0.007s	sys 0.004s
I20260709 12:58:36.309844  1796 tablet_bootstrap.cc:492] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a: Bootstrap starting.
W20260709 12:58:36.312224  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 48978.9us
I20260709 12:58:36.325875  1794 tablet_bootstrap.cc:654] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:36.336457  1862 leader_election.cc:290] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:58:36.337172  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:58:36.338734  1676 raft_consensus.cc:3060] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:36.347839  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:58:36.348361  1796 tablet_bootstrap.cc:654] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:36.350301  1873 raft_consensus.cc:2749] T d21485c094584689917b2a97b2d3c624 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:58:36.366411  1801 consensus_queue.cc:1048] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260709 12:58:36.370191  1551 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:58:36.382030  1795 raft_consensus.cc:359] T 388afdca3ce84bf0a8ce3331ed0d9b2f P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:36.383040  1795 raft_consensus.cc:385] T 388afdca3ce84bf0a8ce3331ed0d9b2f P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:36.383332  1795 raft_consensus.cc:740] T 388afdca3ce84bf0a8ce3331ed0d9b2f P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Initialized, Role: FOLLOWER
I20260709 12:58:36.394464  1795 consensus_queue.cc:260] T 388afdca3ce84bf0a8ce3331ed0d9b2f P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:36.397629  1753 raft_consensus.cc:3060] T 388afdca3ce84bf0a8ce3331ed0d9b2f P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:36.416595  1795 ts_tablet_manager.cc:1434] T 388afdca3ce84bf0a8ce3331ed0d9b2f P af520b67945d443b901bbeca37cc5701: Time spent starting tablet: real 0.109s	user 0.011s	sys 0.004s
I20260709 12:58:36.418284  1795 tablet_bootstrap.cc:492] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701: Bootstrap starting.
I20260709 12:58:36.428658  1501 catalog_manager.cc:5697] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d reported cstate change: term changed from 0 to 1, leader changed from <none> to 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194). New cstate: current_term: 1 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } }
I20260709 12:58:36.436527  1676 raft_consensus.cc:2468] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 1.
I20260709 12:58:36.438288  1794 tablet_bootstrap.cc:492] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d: No bootstrap required, opened a new log
I20260709 12:58:36.438233  1569 leader_election.cc:304] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 12:58:36.439028  1794 ts_tablet_manager.cc:1403] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d: Time spent bootstrapping tablet: real 0.268s	user 0.014s	sys 0.018s
I20260709 12:58:36.439502  1877 raft_consensus.cc:2804] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:58:36.440327  1877 raft_consensus.cc:697] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 12:58:36.442032  1877 consensus_queue.cc:237] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:36.442960  1796 tablet_bootstrap.cc:492] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a: No bootstrap required, opened a new log
I20260709 12:58:36.443779  1796 ts_tablet_manager.cc:1403] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent bootstrapping tablet: real 0.134s	user 0.015s	sys 0.012s
I20260709 12:58:36.441176  1753 raft_consensus.cc:2468] T 388afdca3ce84bf0a8ce3331ed0d9b2f P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 1.
I20260709 12:58:36.448801  1795 tablet_bootstrap.cc:654] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:36.449028  1794 raft_consensus.cc:359] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:36.449923  1794 raft_consensus.cc:385] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:36.450289  1794 raft_consensus.cc:740] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Initialized, Role: FOLLOWER
I20260709 12:58:36.453086  1796 raft_consensus.cc:359] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:36.453648  1794 consensus_queue.cc:260] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:36.454351  1796 raft_consensus.cc:385] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:36.454945  1796 raft_consensus.cc:740] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Initialized, Role: FOLLOWER
I20260709 12:58:36.456912  1794 ts_tablet_manager.cc:1434] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d: Time spent starting tablet: real 0.018s	user 0.009s	sys 0.003s
I20260709 12:58:36.456316  1796 consensus_queue.cc:260] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:36.458156  1794 tablet_bootstrap.cc:492] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d: Bootstrap starting.
I20260709 12:58:36.460477  1796 ts_tablet_manager.cc:1434] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent starting tablet: real 0.016s	user 0.008s	sys 0.001s
I20260709 12:58:36.461802  1796 tablet_bootstrap.cc:492] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a: Bootstrap starting.
I20260709 12:58:36.469413  1794 tablet_bootstrap.cc:654] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:36.473492  1796 tablet_bootstrap.cc:654] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:36.494886  1795 tablet_bootstrap.cc:492] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701: No bootstrap required, opened a new log
I20260709 12:58:36.495494  1795 ts_tablet_manager.cc:1403] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701: Time spent bootstrapping tablet: real 0.078s	user 0.022s	sys 0.028s
I20260709 12:58:36.495733  1503 catalog_manager.cc:5697] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 0 to 1, leader changed from <none> to 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193). New cstate: current_term: 1 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } }
I20260709 12:58:36.499764  1795 raft_consensus.cc:359] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:36.500828  1795 raft_consensus.cc:385] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:36.501176  1795 raft_consensus.cc:740] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Initialized, Role: FOLLOWER
I20260709 12:58:36.502899  1795 consensus_queue.cc:260] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:36.506217  1795 ts_tablet_manager.cc:1434] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701: Time spent starting tablet: real 0.010s	user 0.008s	sys 0.001s
I20260709 12:58:36.507396  1795 tablet_bootstrap.cc:492] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701: Bootstrap starting.
I20260709 12:58:36.528187  1795 tablet_bootstrap.cc:654] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:36.530906  1873 consensus_queue.cc:1048] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:58:36.532300  1796 tablet_bootstrap.cc:492] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a: No bootstrap required, opened a new log
I20260709 12:58:36.532944  1796 ts_tablet_manager.cc:1403] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent bootstrapping tablet: real 0.071s	user 0.015s	sys 0.048s
I20260709 12:58:36.538070  1796 raft_consensus.cc:359] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:36.539145  1796 raft_consensus.cc:385] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:36.539563  1796 raft_consensus.cc:740] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Initialized, Role: FOLLOWER
I20260709 12:58:36.540618  1796 consensus_queue.cc:260] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:36.544888  1794 tablet_bootstrap.cc:492] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d: No bootstrap required, opened a new log
I20260709 12:58:36.545574  1794 ts_tablet_manager.cc:1403] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent bootstrapping tablet: real 0.088s	user 0.012s	sys 0.026s
I20260709 12:58:36.547119  1796 ts_tablet_manager.cc:1434] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent starting tablet: real 0.014s	user 0.003s	sys 0.009s
I20260709 12:58:36.547938  1795 tablet_bootstrap.cc:492] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701: No bootstrap required, opened a new log
I20260709 12:58:36.548314  1796 tablet_bootstrap.cc:492] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a: Bootstrap starting.
I20260709 12:58:36.548539  1795 ts_tablet_manager.cc:1403] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701: Time spent bootstrapping tablet: real 0.041s	user 0.019s	sys 0.008s
I20260709 12:58:36.555456  1794 raft_consensus.cc:359] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:36.556274  1794 raft_consensus.cc:385] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:36.556538  1794 raft_consensus.cc:740] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Initialized, Role: FOLLOWER
I20260709 12:58:36.562553  1794 consensus_queue.cc:260] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:36.561046  1502 catalog_manager.cc:5697] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 reported cstate change: term changed from 0 to 1, leader changed from <none> to af520b67945d443b901bbeca37cc5701 (127.0.107.195). New cstate: current_term: 1 leader_uuid: "af520b67945d443b901bbeca37cc5701" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: HEALTHY } } }
I20260709 12:58:36.565888  1794 ts_tablet_manager.cc:1434] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent starting tablet: real 0.020s	user 0.005s	sys 0.004s
I20260709 12:58:36.559262  1795 raft_consensus.cc:359] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:36.567080  1795 raft_consensus.cc:385] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:36.567471  1795 raft_consensus.cc:740] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Initialized, Role: FOLLOWER
I20260709 12:58:36.568567  1795 consensus_queue.cc:260] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:36.571745  1794 tablet_bootstrap.cc:492] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d: Bootstrap starting.
I20260709 12:58:36.571909  1795 ts_tablet_manager.cc:1434] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701: Time spent starting tablet: real 0.023s	user 0.008s	sys 0.001s
I20260709 12:58:36.577157  1795 tablet_bootstrap.cc:492] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701: Bootstrap starting.
I20260709 12:58:36.587998  1796 tablet_bootstrap.cc:654] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:36.597687  1795 tablet_bootstrap.cc:654] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:36.609877  1830 raft_consensus.cc:493] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:36.610623  1830 raft_consensus.cc:515] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:36.613596  1830 leader_election.cc:290] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:58:36.613866  1794 tablet_bootstrap.cc:654] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:36.614320  1830 consensus_queue.cc:1048] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20260709 12:58:36.620093  1796 tablet_bootstrap.cc:492] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a: No bootstrap required, opened a new log
I20260709 12:58:36.620791  1796 ts_tablet_manager.cc:1403] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent bootstrapping tablet: real 0.073s	user 0.014s	sys 0.016s
I20260709 12:58:36.623277  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:58:36.627082  1877 consensus_queue.cc:1048] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:58:36.628312  1501 catalog_manager.cc:5697] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 0 to 1, leader changed from <none> to 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193). New cstate: current_term: 1 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } }
I20260709 12:58:36.623302  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:58:36.635568  1795 tablet_bootstrap.cc:492] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701: No bootstrap required, opened a new log
I20260709 12:58:36.636271  1795 ts_tablet_manager.cc:1403] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701: Time spent bootstrapping tablet: real 0.063s	user 0.008s	sys 0.010s
W20260709 12:58:36.640193  1717 leader_election.cc:343] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:58:36.640736  1718 leader_election.cc:343] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:36.641202  1718 leader_election.cc:304] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:58:36.642357  1830 raft_consensus.cc:2749] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:58:36.644551  1796 raft_consensus.cc:359] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:36.645538  1796 raft_consensus.cc:385] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:36.645879  1796 raft_consensus.cc:740] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Initialized, Role: FOLLOWER
I20260709 12:58:36.646682  1796 consensus_queue.cc:260] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:36.642679  1795 raft_consensus.cc:359] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:36.647535  1795 raft_consensus.cc:385] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:36.647869  1795 raft_consensus.cc:740] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Initialized, Role: FOLLOWER
I20260709 12:58:36.648656  1795 consensus_queue.cc:260] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:36.651052  1795 ts_tablet_manager.cc:1434] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701: Time spent starting tablet: real 0.014s	user 0.008s	sys 0.000s
I20260709 12:58:36.653702  1862 raft_consensus.cc:493] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:36.655213  1794 tablet_bootstrap.cc:492] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d: No bootstrap required, opened a new log
I20260709 12:58:36.655807  1794 ts_tablet_manager.cc:1403] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent bootstrapping tablet: real 0.084s	user 0.011s	sys 0.017s
I20260709 12:58:36.657725  1796 ts_tablet_manager.cc:1434] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent starting tablet: real 0.037s	user 0.005s	sys 0.004s
I20260709 12:58:36.658895  1796 tablet_bootstrap.cc:492] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a: Bootstrap starting.
I20260709 12:58:36.663199  1795 tablet_bootstrap.cc:492] T 2a521fb6c3e7489d9694fd81cadeb33f P af520b67945d443b901bbeca37cc5701: Bootstrap starting.
I20260709 12:58:36.662362  1862 raft_consensus.cc:515] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:36.667966  1877 raft_consensus.cc:493] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:36.668309  1794 raft_consensus.cc:359] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:36.668998  1794 raft_consensus.cc:385] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:36.668663  1877 raft_consensus.cc:515] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:36.669248  1794 raft_consensus.cc:740] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Initialized, Role: FOLLOWER
I20260709 12:58:36.671234  1862 leader_election.cc:290] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:58:36.671447  1794 consensus_queue.cc:260] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:36.673161  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:58:36.677807  1794 ts_tablet_manager.cc:1434] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent starting tablet: real 0.022s	user 0.009s	sys 0.000s
I20260709 12:58:36.678869  1794 tablet_bootstrap.cc:492] T cd44a2ebdc764e898f25ad5390f4bffd P 816fadcea8fc4893882f5f9b9f22a03d: Bootstrap starting.
W20260709 12:58:36.679890  1571 leader_election.cc:343] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:36.681813  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:58:36.683310  1676 raft_consensus.cc:2468] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 0.
I20260709 12:58:36.685267  1795 tablet_bootstrap.cc:654] T 2a521fb6c3e7489d9694fd81cadeb33f P af520b67945d443b901bbeca37cc5701: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:36.685587  1857 raft_consensus.cc:493] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:36.690830  1569 leader_election.cc:304] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 12:58:36.691219  1857 raft_consensus.cc:515] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:36.692414  1831 raft_consensus.cc:2804] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:58:36.692907  1831 raft_consensus.cc:493] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:58:36.693365  1831 raft_consensus.cc:3060] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:36.697304  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:58:36.698552  1676 raft_consensus.cc:2468] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 0.
I20260709 12:58:36.702997  1877 leader_election.cc:290] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:58:36.704133  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:58:36.705104  1605 raft_consensus.cc:2468] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 0.
I20260709 12:58:36.705493  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:58:36.705543  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:58:36.707165  1643 leader_election.cc:304] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
W20260709 12:58:36.708103  1646 leader_election.cc:343] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:36.708688  1902 raft_consensus.cc:2804] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:58:36.709285  1902 raft_consensus.cc:493] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:58:36.709813  1902 raft_consensus.cc:3060] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:36.710073  1796 tablet_bootstrap.cc:654] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:36.701387  1569 leader_election.cc:304] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
W20260709 12:58:36.711483  1571 leader_election.cc:343] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:36.712491  1862 raft_consensus.cc:2804] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:58:36.713161  1862 raft_consensus.cc:493] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:58:36.714489  1862 raft_consensus.cc:3060] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:36.710402  1857 leader_election.cc:290] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:58:36.726606  1877 consensus_queue.cc:1048] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:58:36.734488  1794 tablet_bootstrap.cc:654] T cd44a2ebdc764e898f25ad5390f4bffd P 816fadcea8fc4893882f5f9b9f22a03d: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:36.737056  1862 raft_consensus.cc:515] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:36.744702  1831 raft_consensus.cc:515] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:36.751521  1902 raft_consensus.cc:515] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:36.753158  1877 raft_consensus.cc:493] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:36.764303  1830 raft_consensus.cc:493] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:36.770962  1830 raft_consensus.cc:515] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:36.767992  1877 raft_consensus.cc:515] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:36.783089  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:58:36.784094  1605 raft_consensus.cc:2468] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 0.
I20260709 12:58:36.783521  1831 leader_election.cc:290] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:58:36.790191  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:58:36.795897  1795 tablet_bootstrap.cc:492] T 2a521fb6c3e7489d9694fd81cadeb33f P af520b67945d443b901bbeca37cc5701: No bootstrap required, opened a new log
I20260709 12:58:36.796082  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:58:36.796525  1795 ts_tablet_manager.cc:1403] T 2a521fb6c3e7489d9694fd81cadeb33f P af520b67945d443b901bbeca37cc5701: Time spent bootstrapping tablet: real 0.133s	user 0.008s	sys 0.013s
I20260709 12:58:36.796659  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:58:36.796962  1676 raft_consensus.cc:3060] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:36.798236  1877 leader_election.cc:290] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:58:36.800578  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:58:36.802914  1717 leader_election.cc:304] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:58:36.804280  1873 raft_consensus.cc:2804] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:58:36.804839  1873 raft_consensus.cc:493] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:58:36.805341  1873 raft_consensus.cc:3060] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:36.790602  1862 leader_election.cc:290] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 12:58:36.807526  1569 leader_election.cc:343] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:36.807569  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:58:36.806753  1902 leader_election.cc:290] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:58:36.797539  1677 raft_consensus.cc:3060] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:36.808872  1830 leader_election.cc:290] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:58:36.813946  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:58:36.815238  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:58:36.815763  1795 raft_consensus.cc:359] T 2a521fb6c3e7489d9694fd81cadeb33f P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:36.816577  1795 raft_consensus.cc:385] T 2a521fb6c3e7489d9694fd81cadeb33f P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:36.816906  1795 raft_consensus.cc:740] T 2a521fb6c3e7489d9694fd81cadeb33f P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Initialized, Role: FOLLOWER
W20260709 12:58:36.817162  1718 leader_election.cc:343] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:36.821779  1796 tablet_bootstrap.cc:492] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a: No bootstrap required, opened a new log
I20260709 12:58:36.821823  1795 consensus_queue.cc:260] T 2a521fb6c3e7489d9694fd81cadeb33f P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:36.822422  1796 ts_tablet_manager.cc:1403] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent bootstrapping tablet: real 0.164s	user 0.020s	sys 0.012s
I20260709 12:58:36.824779  1795 ts_tablet_manager.cc:1434] T 2a521fb6c3e7489d9694fd81cadeb33f P af520b67945d443b901bbeca37cc5701: Time spent starting tablet: real 0.028s	user 0.015s	sys 0.001s
I20260709 12:58:36.825769  1795 tablet_bootstrap.cc:492] T 83226095f79a46dab445849b1febd425 P af520b67945d443b901bbeca37cc5701: Bootstrap starting.
I20260709 12:58:36.826095  1794 tablet_bootstrap.cc:492] T cd44a2ebdc764e898f25ad5390f4bffd P 816fadcea8fc4893882f5f9b9f22a03d: No bootstrap required, opened a new log
I20260709 12:58:36.827847  1754 raft_consensus.cc:3060] T 2a521fb6c3e7489d9694fd81cadeb33f P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:36.829023  1794 ts_tablet_manager.cc:1403] T cd44a2ebdc764e898f25ad5390f4bffd P 816fadcea8fc4893882f5f9b9f22a03d: Time spent bootstrapping tablet: real 0.150s	user 0.031s	sys 0.005s
W20260709 12:58:36.814242  1571 leader_election.cc:343] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:36.833004  1796 raft_consensus.cc:359] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:36.833897  1796 raft_consensus.cc:385] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:36.834332  1796 raft_consensus.cc:740] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Initialized, Role: FOLLOWER
I20260709 12:58:36.835626  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:58:36.835376  1796 consensus_queue.cc:260] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:36.837754  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:58:36.838037  1796 ts_tablet_manager.cc:1434] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent starting tablet: real 0.015s	user 0.007s	sys 0.000s
I20260709 12:58:36.838867  1676 raft_consensus.cc:2468] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 1.
I20260709 12:58:36.839444  1796 tablet_bootstrap.cc:492] T 513043a9b96c4f70adad53ac4b5d01dc P 83595ad0fd5940c0a2bfc7c61acd969a: Bootstrap starting.
I20260709 12:58:36.840013  1605 raft_consensus.cc:3060] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:36.840936  1569 leader_election.cc:304] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 12:58:36.840106  1794 raft_consensus.cc:359] T cd44a2ebdc764e898f25ad5390f4bffd P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:36.846589  1794 raft_consensus.cc:385] T cd44a2ebdc764e898f25ad5390f4bffd P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:36.846973  1794 raft_consensus.cc:740] T cd44a2ebdc764e898f25ad5390f4bffd P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Initialized, Role: FOLLOWER
I20260709 12:58:36.850642  1862 raft_consensus.cc:2804] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:58:36.851357  1796 tablet_bootstrap.cc:654] T 513043a9b96c4f70adad53ac4b5d01dc P 83595ad0fd5940c0a2bfc7c61acd969a: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:36.851368  1862 raft_consensus.cc:697] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 12:58:36.852764  1754 raft_consensus.cc:2468] T 2a521fb6c3e7489d9694fd81cadeb33f P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 1.
I20260709 12:58:36.853173  1873 raft_consensus.cc:515] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:36.855046  1753 raft_consensus.cc:2393] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 1: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 12:58:36.854683  1646 leader_election.cc:304] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:58:36.856061  1902 raft_consensus.cc:2804] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:58:36.857645  1794 consensus_queue.cc:260] T cd44a2ebdc764e898f25ad5390f4bffd P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:36.857178  1902 raft_consensus.cc:697] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 1 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 12:58:36.865386  1873 leader_election.cc:290] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:58:36.857930  1862 consensus_queue.cc:237] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:36.867056  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
W20260709 12:58:36.869171  1718 leader_election.cc:343] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:36.866359  1902 consensus_queue.cc:237] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:36.881381  1877 raft_consensus.cc:493] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:36.881943  1795 tablet_bootstrap.cc:654] T 83226095f79a46dab445849b1febd425 P af520b67945d443b901bbeca37cc5701: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:36.882180  1877 raft_consensus.cc:515] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:36.890347  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:58:36.887684  1877 leader_election.cc:290] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:58:36.891410  1676 raft_consensus.cc:2468] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 0.
I20260709 12:58:36.894397  1794 ts_tablet_manager.cc:1434] T cd44a2ebdc764e898f25ad5390f4bffd P 816fadcea8fc4893882f5f9b9f22a03d: Time spent starting tablet: real 0.065s	user 0.006s	sys 0.002s
I20260709 12:58:36.895744  1794 tablet_bootstrap.cc:492] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d: Bootstrap starting.
I20260709 12:58:36.903952  1569 leader_election.cc:304] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 12:58:36.905130  1877 raft_consensus.cc:2804] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:58:36.905658  1877 raft_consensus.cc:493] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:58:36.906170  1877 raft_consensus.cc:3060] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:36.902691  1677 raft_consensus.cc:2468] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 1.
W20260709 12:58:36.908957  1571 leader_election.cc:343] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:36.913075  1571 leader_election.cc:304] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 12:58:36.913640  1873 raft_consensus.cc:493] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:36.913952  1569 leader_election.cc:304] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 12:58:36.914330  1873 raft_consensus.cc:515] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:36.916455  1831 raft_consensus.cc:3060] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:36.916989  1796 tablet_bootstrap.cc:492] T 513043a9b96c4f70adad53ac4b5d01dc P 83595ad0fd5940c0a2bfc7c61acd969a: No bootstrap required, opened a new log
I20260709 12:58:36.917441  1754 raft_consensus.cc:3060] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:36.917750  1796 ts_tablet_manager.cc:1403] T 513043a9b96c4f70adad53ac4b5d01dc P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent bootstrapping tablet: real 0.079s	user 0.027s	sys 0.033s
I20260709 12:58:36.917889  1801 raft_consensus.cc:2804] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:58:36.919436  1801 raft_consensus.cc:697] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 12:58:36.920843  1801 consensus_queue.cc:237] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:36.921556  1873 leader_election.cc:290] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:58:36.923167  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:58:36.926175  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:58:36.930675  1605 raft_consensus.cc:2468] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 1.
I20260709 12:58:36.947644  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:58:36.949213  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:58:36.958791  1796 raft_consensus.cc:359] T 513043a9b96c4f70adad53ac4b5d01dc P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:36.963012  1796 raft_consensus.cc:385] T 513043a9b96c4f70adad53ac4b5d01dc P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:36.970651  1796 raft_consensus.cc:740] T 513043a9b96c4f70adad53ac4b5d01dc P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Initialized, Role: FOLLOWER
I20260709 12:58:36.968698  1503 catalog_manager.cc:5697] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d reported cstate change: term changed from 0 to 1, leader changed from <none> to 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194). New cstate: current_term: 1 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } }
I20260709 12:58:36.971547  1796 consensus_queue.cc:260] T 513043a9b96c4f70adad53ac4b5d01dc P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:36.975325  1794 tablet_bootstrap.cc:654] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:36.987524  1796 ts_tablet_manager.cc:1434] T 513043a9b96c4f70adad53ac4b5d01dc P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent starting tablet: real 0.069s	user 0.007s	sys 0.003s
I20260709 12:58:36.986547  1831 raft_consensus.cc:2749] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:58:36.990521  1928 raft_consensus.cc:493] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:36.991333  1928 raft_consensus.cc:515] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:36.996764  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:58:36.989187  1796 tablet_bootstrap.cc:492] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a: Bootstrap starting.
I20260709 12:58:36.997782  1677 raft_consensus.cc:2468] T cd44a2ebdc764e898f25ad5390f4bffd P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 0.
I20260709 12:58:36.980374  1877 raft_consensus.cc:515] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:36.960619  1874 consensus_queue.cc:1048] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.044s
I20260709 12:58:37.007727  1569 leader_election.cc:304] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
W20260709 12:58:37.021211  1927 log.cc:927] Time spent T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.061s	user 0.001s	sys 0.000s
I20260709 12:58:37.015738  1862 raft_consensus.cc:2804] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:58:37.022042  1862 raft_consensus.cc:493] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:58:37.022672  1862 raft_consensus.cc:3060] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:58:37.043987  1571 leader_election.cc:343] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:37.041183  1928 leader_election.cc:290] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:58:37.091620  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:58:37.092332  1676 raft_consensus.cc:3060] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:37.092897  1605 raft_consensus.cc:2468] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 1.
W20260709 12:58:37.087725  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 52934us
I20260709 12:58:37.097775  1877 leader_election.cc:290] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:58:37.104461  1795 tablet_bootstrap.cc:492] T 83226095f79a46dab445849b1febd425 P af520b67945d443b901bbeca37cc5701: No bootstrap required, opened a new log
I20260709 12:58:37.097999  1862 raft_consensus.cc:515] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:37.105049  1795 ts_tablet_manager.cc:1403] T 83226095f79a46dab445849b1febd425 P af520b67945d443b901bbeca37cc5701: Time spent bootstrapping tablet: real 0.279s	user 0.019s	sys 0.005s
I20260709 12:58:37.107774  1862 leader_election.cc:290] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:58:37.110211  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:58:37.105053  1717 leader_election.cc:304] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:58:37.111292  1680 raft_consensus.cc:3060] T cd44a2ebdc764e898f25ad5390f4bffd P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:37.112874  1717 leader_election.cc:304] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 12:58:37.113955  1830 raft_consensus.cc:2749] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:58:37.115329  1830 raft_consensus.cc:2804] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:58:37.116215  1830 raft_consensus.cc:697] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 1 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 12:58:37.116365  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:58:37.115089  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:58:37.118839  1753 raft_consensus.cc:2468] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 0.
I20260709 12:58:37.122141  1794 tablet_bootstrap.cc:492] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d: No bootstrap required, opened a new log
I20260709 12:58:37.122781  1794 ts_tablet_manager.cc:1403] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent bootstrapping tablet: real 0.227s	user 0.024s	sys 0.006s
I20260709 12:58:37.112695  1930 consensus_queue.cc:1048] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.017s
I20260709 12:58:37.117847  1830 consensus_queue.cc:237] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:37.118243  1877 raft_consensus.cc:493] T 513043a9b96c4f70adad53ac4b5d01dc P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:37.113848  1502 catalog_manager.cc:5697] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 0 to 1, leader changed from <none> to 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193). New cstate: current_term: 1 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } }
I20260709 12:58:37.137092  1795 raft_consensus.cc:359] T 83226095f79a46dab445849b1febd425 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:37.137655  1795 raft_consensus.cc:385] T 83226095f79a46dab445849b1febd425 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:37.137885  1795 raft_consensus.cc:740] T 83226095f79a46dab445849b1febd425 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Initialized, Role: FOLLOWER
I20260709 12:58:37.139137  1795 consensus_queue.cc:260] T 83226095f79a46dab445849b1febd425 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:37.140683  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:58:37.142614  1754 raft_consensus.cc:3060] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:58:37.134999  1571 leader_election.cc:343] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:37.144024  1795 ts_tablet_manager.cc:1434] T 83226095f79a46dab445849b1febd425 P af520b67945d443b901bbeca37cc5701: Time spent starting tablet: real 0.039s	user 0.004s	sys 0.003s
I20260709 12:58:37.145311  1795 tablet_bootstrap.cc:492] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701: Bootstrap starting.
I20260709 12:58:37.147198  1676 raft_consensus.cc:2468] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 1.
I20260709 12:58:37.149971  1569 leader_election.cc:304] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 12:58:37.151994  1928 raft_consensus.cc:2804] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:58:37.152752  1928 raft_consensus.cc:697] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 12:58:37.154235  1928 consensus_queue.cc:237] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:37.136723  1877 raft_consensus.cc:515] T 513043a9b96c4f70adad53ac4b5d01dc P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:37.161082  1877 leader_election.cc:290] T 513043a9b96c4f70adad53ac4b5d01dc P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:58:37.167236  1796 tablet_bootstrap.cc:654] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:37.180861  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:58:37.164819  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:58:37.186494  1680 raft_consensus.cc:2468] T cd44a2ebdc764e898f25ad5390f4bffd P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 1.
W20260709 12:58:37.190856  1569 leader_election.cc:343] T 513043a9b96c4f70adad53ac4b5d01dc P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:37.192739  1569 leader_election.cc:304] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 12:58:37.195084  1877 raft_consensus.cc:2804] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:58:37.204233  1877 raft_consensus.cc:697] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 12:58:37.205443  1794 raft_consensus.cc:359] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:37.206331  1794 raft_consensus.cc:385] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:37.206696  1794 raft_consensus.cc:740] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Initialized, Role: FOLLOWER
I20260709 12:58:37.208142  1794 consensus_queue.cc:260] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:37.211269  1794 ts_tablet_manager.cc:1434] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent starting tablet: real 0.088s	user 0.008s	sys 0.001s
I20260709 12:58:37.212612  1794 tablet_bootstrap.cc:492] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d: Bootstrap starting.
I20260709 12:58:37.218616  1795 tablet_bootstrap.cc:654] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:37.205610  1877 consensus_queue.cc:237] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 12:58:37.211755  1571 leader_election.cc:343] T 513043a9b96c4f70adad53ac4b5d01dc P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:37.228926  1571 leader_election.cc:304] T 513043a9b96c4f70adad53ac4b5d01dc P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 12:58:37.232769  1928 raft_consensus.cc:2749] T 513043a9b96c4f70adad53ac4b5d01dc P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:58:37.248168  1502 catalog_manager.cc:5697] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 reported cstate change: term changed from 0 to 1, leader changed from <none> to af520b67945d443b901bbeca37cc5701 (127.0.107.195). New cstate: current_term: 1 leader_uuid: "af520b67945d443b901bbeca37cc5701" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: HEALTHY } } }
I20260709 12:58:37.267769  1794 tablet_bootstrap.cc:654] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d: Neither blocks nor log segments found. Creating new log.
W20260709 12:58:37.268400  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 60925us
I20260709 12:58:37.274537  1754 raft_consensus.cc:2468] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 1.
I20260709 12:58:37.294495  1796 tablet_bootstrap.cc:492] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a: No bootstrap required, opened a new log
I20260709 12:58:37.295253  1796 ts_tablet_manager.cc:1403] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent bootstrapping tablet: real 0.306s	user 0.021s	sys 0.037s
I20260709 12:58:37.318111  1795 tablet_bootstrap.cc:492] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701: No bootstrap required, opened a new log
I20260709 12:58:37.318832  1795 ts_tablet_manager.cc:1403] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701: Time spent bootstrapping tablet: real 0.174s	user 0.020s	sys 0.034s
I20260709 12:58:37.320834  1794 tablet_bootstrap.cc:492] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d: No bootstrap required, opened a new log
I20260709 12:58:37.323658  1794 ts_tablet_manager.cc:1403] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d: Time spent bootstrapping tablet: real 0.111s	user 0.028s	sys 0.022s
I20260709 12:58:37.323259  1795 raft_consensus.cc:359] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:37.323410  1826 consensus_queue.cc:1048] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:58:37.324237  1795 raft_consensus.cc:385] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:37.324612  1795 raft_consensus.cc:740] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Initialized, Role: FOLLOWER
I20260709 12:58:37.322455  1796 raft_consensus.cc:359] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:37.329131  1796 raft_consensus.cc:385] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:37.325445  1795 consensus_queue.cc:260] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:37.329551  1796 raft_consensus.cc:740] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Initialized, Role: FOLLOWER
I20260709 12:58:37.330534  1796 consensus_queue.cc:260] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:37.331619  1795 ts_tablet_manager.cc:1434] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701: Time spent starting tablet: real 0.012s	user 0.007s	sys 0.002s
I20260709 12:58:37.328233  1794 raft_consensus.cc:359] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:37.332181  1794 raft_consensus.cc:385] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:37.332593  1794 raft_consensus.cc:740] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Initialized, Role: FOLLOWER
I20260709 12:58:37.333693  1795 tablet_bootstrap.cc:492] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701: Bootstrap starting.
I20260709 12:58:37.334192  1794 consensus_queue.cc:260] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:37.333659  1796 ts_tablet_manager.cc:1434] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent starting tablet: real 0.038s	user 0.008s	sys 0.002s
I20260709 12:58:37.338824  1794 ts_tablet_manager.cc:1434] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d: Time spent starting tablet: real 0.015s	user 0.008s	sys 0.000s
I20260709 12:58:37.340416  1794 tablet_bootstrap.cc:492] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 816fadcea8fc4893882f5f9b9f22a03d: Bootstrap starting.
I20260709 12:58:37.342969  1796 tablet_bootstrap.cc:492] T a40e423ed583480280352b261913276c P 83595ad0fd5940c0a2bfc7c61acd969a: Bootstrap starting.
I20260709 12:58:37.349452  1795 tablet_bootstrap.cc:654] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:37.352711  1794 tablet_bootstrap.cc:654] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 816fadcea8fc4893882f5f9b9f22a03d: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:37.358939  1826 raft_consensus.cc:493] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:37.359786  1826 raft_consensus.cc:515] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:37.368136  1796 tablet_bootstrap.cc:654] T a40e423ed583480280352b261913276c P 83595ad0fd5940c0a2bfc7c61acd969a: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:37.372361  1830 raft_consensus.cc:493] T 83226095f79a46dab445849b1febd425 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:37.373010  1830 raft_consensus.cc:515] T 83226095f79a46dab445849b1febd425 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:37.374117  1794 tablet_bootstrap.cc:492] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 816fadcea8fc4893882f5f9b9f22a03d: No bootstrap required, opened a new log
I20260709 12:58:37.374831  1794 ts_tablet_manager.cc:1403] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent bootstrapping tablet: real 0.035s	user 0.014s	sys 0.011s
I20260709 12:58:37.375423  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:58:37.379805  1794 raft_consensus.cc:359] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:37.380822  1794 raft_consensus.cc:385] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:37.381209  1794 raft_consensus.cc:740] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Initialized, Role: FOLLOWER
I20260709 12:58:37.382495  1830 leader_election.cc:290] T 83226095f79a46dab445849b1febd425 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:58:37.383229  1826 leader_election.cc:290] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:58:37.382977  1794 consensus_queue.cc:260] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:37.384091  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:58:37.385093  1676 raft_consensus.cc:2393] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 1: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 12:58:37.385777  1794 ts_tablet_manager.cc:1434] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent starting tablet: real 0.011s	user 0.007s	sys 0.000s
I20260709 12:58:37.386911  1794 tablet_bootstrap.cc:492] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d: Bootstrap starting.
I20260709 12:58:37.388392  1754 raft_consensus.cc:3060] T 83226095f79a46dab445849b1febd425 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:37.388257  1928 consensus_queue.cc:1048] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:58:37.390856  1830 raft_consensus.cc:493] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:37.391494  1830 raft_consensus.cc:515] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:37.397812  1830 leader_election.cc:290] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:58:37.398175  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:58:37.399520  1680 raft_consensus.cc:2393] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 1: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 12:58:37.410250  1794 tablet_bootstrap.cc:654] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:37.414088  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:58:37.417289  1601 raft_consensus.cc:2393] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 1: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 12:58:37.418887  1928 consensus_queue.cc:1048] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:58:37.435732  1643 leader_election.cc:304] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 12:58:37.438644  1861 raft_consensus.cc:3060] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:37.461674  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:58:37.463975  1826 raft_consensus.cc:493] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:37.465613  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:58:37.465745  1826 raft_consensus.cc:515] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:37.470253  1826 leader_election.cc:290] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:58:37.476351  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:58:37.477417  1826 consensus_queue.cc:1048] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:58:37.505152  1717 leader_election.cc:304] T 83226095f79a46dab445849b1febd425 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:58:37.506035  1861 raft_consensus.cc:2749] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260709 12:58:37.509518  1646 leader_election.cc:343] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:37.523887  1717 leader_election.cc:304] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:58:37.524641  1830 raft_consensus.cc:2749] T 83226095f79a46dab445849b1febd425 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:58:37.525679  1873 raft_consensus.cc:3060] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:37.526600  1795 tablet_bootstrap.cc:492] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701: No bootstrap required, opened a new log
I20260709 12:58:37.526599  1794 tablet_bootstrap.cc:492] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d: No bootstrap required, opened a new log
I20260709 12:58:37.527202  1794 ts_tablet_manager.cc:1403] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent bootstrapping tablet: real 0.141s	user 0.025s	sys 0.076s
I20260709 12:58:37.527202  1795 ts_tablet_manager.cc:1403] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701: Time spent bootstrapping tablet: real 0.194s	user 0.039s	sys 0.091s
I20260709 12:58:37.531199  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 12:58:37.534252  1571 consensus_peers.cc:597] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a -> Peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Couldn't send request to peer af520b67945d443b901bbeca37cc5701. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:58:37.536636  1862 consensus_queue.cc:1048] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:58:37.518263  1500 catalog_manager.cc:5697] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 0 to 1, leader changed from <none> to 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193). New cstate: current_term: 1 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } }
I20260709 12:58:37.538810  1873 raft_consensus.cc:2749] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260709 12:58:37.538939  1551 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
W20260709 12:58:37.540257  1643 leader_election.cc:343] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:37.540939  1643 leader_election.cc:304] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 12:58:37.542080  1950 raft_consensus.cc:2749] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:58:37.541589  1795 raft_consensus.cc:359] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:37.542990  1795 raft_consensus.cc:385] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:37.541409  1500 catalog_manager.cc:5697] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 0 to 1, leader changed from <none> to 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193). New cstate: current_term: 1 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } }
I20260709 12:58:37.543361  1795 raft_consensus.cc:740] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Initialized, Role: FOLLOWER
I20260709 12:58:37.544137  1795 consensus_queue.cc:260] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:37.546066  1795 ts_tablet_manager.cc:1434] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701: Time spent starting tablet: real 0.018s	user 0.004s	sys 0.002s
I20260709 12:58:37.547210  1795 tablet_bootstrap.cc:492] T e26344297fff44bb8d2099b8e64d7df2 P af520b67945d443b901bbeca37cc5701: Bootstrap starting.
I20260709 12:58:37.548563  1794 raft_consensus.cc:359] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:37.549381  1794 raft_consensus.cc:385] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:37.549747  1794 raft_consensus.cc:740] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Initialized, Role: FOLLOWER
I20260709 12:58:37.504333  1796 tablet_bootstrap.cc:492] T a40e423ed583480280352b261913276c P 83595ad0fd5940c0a2bfc7c61acd969a: No bootstrap required, opened a new log
I20260709 12:58:37.550875  1796 ts_tablet_manager.cc:1403] T a40e423ed583480280352b261913276c P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent bootstrapping tablet: real 0.208s	user 0.015s	sys 0.010s
I20260709 12:58:37.554338  1794 consensus_queue.cc:260] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:37.556965  1794 ts_tablet_manager.cc:1434] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent starting tablet: real 0.029s	user 0.007s	sys 0.000s
I20260709 12:58:37.557066  1796 raft_consensus.cc:359] T a40e423ed583480280352b261913276c P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:37.557703  1796 raft_consensus.cc:385] T a40e423ed583480280352b261913276c P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:37.557757  1794 tablet_bootstrap.cc:492] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d: Bootstrap starting.
I20260709 12:58:37.557960  1796 raft_consensus.cc:740] T a40e423ed583480280352b261913276c P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Initialized, Role: FOLLOWER
I20260709 12:58:37.558830  1796 consensus_queue.cc:260] T a40e423ed583480280352b261913276c P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:37.561995  1796 ts_tablet_manager.cc:1434] T a40e423ed583480280352b261913276c P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent starting tablet: real 0.011s	user 0.007s	sys 0.001s
I20260709 12:58:37.563232  1796 tablet_bootstrap.cc:492] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a: Bootstrap starting.
I20260709 12:58:37.565788  1950 raft_consensus.cc:493] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:37.565287  1500 catalog_manager.cc:5697] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 0 to 1, leader changed from <none> to 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193). New cstate: current_term: 1 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } }
I20260709 12:58:37.566519  1950 raft_consensus.cc:515] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:37.575546  1794 tablet_bootstrap.cc:654] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:37.576222  1795 tablet_bootstrap.cc:654] T e26344297fff44bb8d2099b8e64d7df2 P af520b67945d443b901bbeca37cc5701: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:37.579939  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 12:58:37.582147  1643 leader_election.cc:343] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:37.582729  1950 leader_election.cc:290] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:58:37.583753  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 12:58:37.585664  1646 leader_election.cc:343] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:37.586289  1646 leader_election.cc:304] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 12:58:37.588408  1826 raft_consensus.cc:2749] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:58:37.600574  1873 consensus_queue.cc:1048] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:58:37.615406  1873 raft_consensus.cc:493] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:37.616194  1873 raft_consensus.cc:515] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:37.618713  1796 tablet_bootstrap.cc:654] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:37.619465  1928 raft_consensus.cc:493] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:37.620537  1928 raft_consensus.cc:515] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:37.623914  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:58:37.624285  1873 leader_election.cc:290] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:58:37.625299  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:58:37.625998  1677 raft_consensus.cc:2468] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 0.
I20260709 12:58:37.626579  1928 leader_election.cc:290] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:58:37.627688  1718 leader_election.cc:304] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:58:37.631702  1873 raft_consensus.cc:2804] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:58:37.632243  1873 raft_consensus.cc:493] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:58:37.632751  1873 raft_consensus.cc:3060] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:37.633450  1794 tablet_bootstrap.cc:492] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d: No bootstrap required, opened a new log
I20260709 12:58:37.633962  1794 ts_tablet_manager.cc:1403] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d: Time spent bootstrapping tablet: real 0.076s	user 0.016s	sys 0.007s
I20260709 12:58:37.636107  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:58:37.636940  1795 tablet_bootstrap.cc:492] T e26344297fff44bb8d2099b8e64d7df2 P af520b67945d443b901bbeca37cc5701: No bootstrap required, opened a new log
W20260709 12:58:37.637071  1717 leader_election.cc:343] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:37.637547  1795 ts_tablet_manager.cc:1403] T e26344297fff44bb8d2099b8e64d7df2 P af520b67945d443b901bbeca37cc5701: Time spent bootstrapping tablet: real 0.091s	user 0.011s	sys 0.017s
I20260709 12:58:37.638217  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:58:37.638424  1794 raft_consensus.cc:359] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:37.639173  1794 raft_consensus.cc:385] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260709 12:58:37.639216  1571 leader_election.cc:343] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:37.639540  1794 raft_consensus.cc:740] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Initialized, Role: FOLLOWER
W20260709 12:58:37.640297  1569 leader_election.cc:343] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:37.640434  1794 consensus_queue.cc:260] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:37.640906  1569 leader_election.cc:304] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 12:58:37.641489  1795 raft_consensus.cc:359] T e26344297fff44bb8d2099b8e64d7df2 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:37.642006  1928 raft_consensus.cc:2749] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:58:37.643575  1794 ts_tablet_manager.cc:1434] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d: Time spent starting tablet: real 0.009s	user 0.005s	sys 0.005s
I20260709 12:58:37.644863  1794 tablet_bootstrap.cc:492] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d: Bootstrap starting.
I20260709 12:58:37.642292  1795 raft_consensus.cc:385] T e26344297fff44bb8d2099b8e64d7df2 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:37.648463  1795 raft_consensus.cc:740] T e26344297fff44bb8d2099b8e64d7df2 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Initialized, Role: FOLLOWER
I20260709 12:58:37.649343  1795 consensus_queue.cc:260] T e26344297fff44bb8d2099b8e64d7df2 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:37.651008  1862 consensus_queue.cc:1048] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:58:37.658471  1873 raft_consensus.cc:515] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:37.660241  1795 ts_tablet_manager.cc:1434] T e26344297fff44bb8d2099b8e64d7df2 P af520b67945d443b901bbeca37cc5701: Time spent starting tablet: real 0.022s	user 0.011s	sys 0.000s
I20260709 12:58:37.661763  1795 tablet_bootstrap.cc:492] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701: Bootstrap starting.
I20260709 12:58:37.665370  1862 consensus_queue.cc:1048] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260709 12:58:37.670895  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:58:37.673589  1794 tablet_bootstrap.cc:654] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:37.683640  1873 leader_election.cc:290] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:58:37.683969  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:58:37.684924  1676 raft_consensus.cc:3060] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:58:37.695925  1717 leader_election.cc:343] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:37.697575  1796 tablet_bootstrap.cc:492] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a: No bootstrap required, opened a new log
I20260709 12:58:37.698242  1796 ts_tablet_manager.cc:1403] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent bootstrapping tablet: real 0.135s	user 0.016s	sys 0.021s
I20260709 12:58:37.699528  1873 consensus_queue.cc:1048] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:58:37.704332  1796 raft_consensus.cc:359] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:37.705542  1796 raft_consensus.cc:385] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:37.706014  1796 raft_consensus.cc:740] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Initialized, Role: FOLLOWER
I20260709 12:58:37.707370  1796 consensus_queue.cc:260] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:37.737458  1950 raft_consensus.cc:493] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:37.738348  1950 raft_consensus.cc:515] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:37.710835  1796 ts_tablet_manager.cc:1434] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent starting tablet: real 0.012s	user 0.007s	sys 0.004s
I20260709 12:58:37.744959  1902 raft_consensus.cc:493] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:37.746090  1902 raft_consensus.cc:515] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:37.746585  1796 tablet_bootstrap.cc:492] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a: Bootstrap starting.
I20260709 12:58:37.760751  1676 raft_consensus.cc:2468] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 1.
I20260709 12:58:37.763033  1796 tablet_bootstrap.cc:654] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:37.763489  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:58:37.764830  1950 leader_election.cc:290] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:58:37.768765  1794 tablet_bootstrap.cc:492] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d: No bootstrap required, opened a new log
I20260709 12:58:37.769433  1794 ts_tablet_manager.cc:1403] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent bootstrapping tablet: real 0.125s	user 0.019s	sys 0.033s
W20260709 12:58:37.768335  1646 leader_election.cc:343] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:37.766789  1902 leader_election.cc:290] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:58:37.775019  1718 leader_election.cc:304] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:58:37.775449  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:58:37.777194  1605 raft_consensus.cc:2468] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 0.
I20260709 12:58:37.779268  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:58:37.783401  1873 raft_consensus.cc:2804] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:58:37.784319  1873 raft_consensus.cc:697] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 1 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
W20260709 12:58:37.784503  1646 leader_election.cc:343] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:37.794332  1643 leader_election.cc:304] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 12:58:37.795650  1902 raft_consensus.cc:2804] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:58:37.796286  1902 raft_consensus.cc:493] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:58:37.796819  1902 raft_consensus.cc:3060] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:37.808193  1795 tablet_bootstrap.cc:654] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:37.786058  1873 consensus_queue.cc:237] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:37.834733  1902 raft_consensus.cc:515] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:37.855643  1902 leader_election.cc:290] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:58:37.869081  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:58:37.874181  1605 raft_consensus.cc:2468] T a40e423ed583480280352b261913276c P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 0.
I20260709 12:58:37.879515  1794 raft_consensus.cc:359] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:37.880599  1794 raft_consensus.cc:385] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:37.880913  1794 raft_consensus.cc:740] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Initialized, Role: FOLLOWER
I20260709 12:58:37.882882  1794 consensus_queue.cc:260] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 12:58:37.879830  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 85610.5us
I20260709 12:58:37.892783  1796 tablet_bootstrap.cc:492] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a: No bootstrap required, opened a new log
I20260709 12:58:37.892302  1963 consensus_queue.cc:1048] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:58:37.893435  1796 ts_tablet_manager.cc:1403] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent bootstrapping tablet: real 0.147s	user 0.011s	sys 0.009s
I20260709 12:58:37.896996  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:58:37.902282  1794 ts_tablet_manager.cc:1434] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent starting tablet: real 0.132s	user 0.020s	sys 0.008s
I20260709 12:58:37.903606  1794 tablet_bootstrap.cc:492] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d: Bootstrap starting.
I20260709 12:58:37.913185  1643 leader_election.cc:304] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 12:58:37.914290  1861 raft_consensus.cc:2804] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:58:37.914847  1861 raft_consensus.cc:493] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:58:37.915326  1861 raft_consensus.cc:3060] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:37.914057  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:58:37.918807  1605 raft_consensus.cc:3060] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:37.936851  1605 raft_consensus.cc:2468] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 1.
I20260709 12:58:37.938993  1796 raft_consensus.cc:359] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:37.939893  1796 raft_consensus.cc:385] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:37.940246  1796 raft_consensus.cc:740] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Initialized, Role: FOLLOWER
W20260709 12:58:37.935089  1646 leader_election.cc:343] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:37.941115  1796 consensus_queue.cc:260] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:37.943691  1963 consensus_queue.cc:1048] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:58:37.947160  1830 raft_consensus.cc:493] T e26344297fff44bb8d2099b8e64d7df2 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:37.947767  1830 raft_consensus.cc:515] T e26344297fff44bb8d2099b8e64d7df2 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:37.950532  1830 leader_election.cc:290] T e26344297fff44bb8d2099b8e64d7df2 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:58:37.952183  1794 tablet_bootstrap.cc:654] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:37.955235  1796 ts_tablet_manager.cc:1434] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent starting tablet: real 0.061s	user 0.005s	sys 0.004s
I20260709 12:58:37.958519  1796 tablet_bootstrap.cc:492] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a: Bootstrap starting.
I20260709 12:58:37.962486  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:58:37.967607  1500 catalog_manager.cc:5697] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 reported cstate change: term changed from 0 to 1, leader changed from <none> to af520b67945d443b901bbeca37cc5701 (127.0.107.195). New cstate: current_term: 1 leader_uuid: "af520b67945d443b901bbeca37cc5701" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: HEALTHY } } }
I20260709 12:58:37.970176  1861 raft_consensus.cc:515] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:37.973542  1861 leader_election.cc:290] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:58:37.977761  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:58:37.980149  1643 leader_election.cc:304] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
W20260709 12:58:37.981824  1646 leader_election.cc:343] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:37.982075  1902 raft_consensus.cc:2804] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:58:37.982807  1902 raft_consensus.cc:697] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 12:58:37.984091  1902 consensus_queue.cc:237] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:37.992964  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:58:38.001694  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:58:38.001886  1718 leader_election.cc:304] T e26344297fff44bb8d2099b8e64d7df2 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:58:38.002851  1604 raft_consensus.cc:3060] T a40e423ed583480280352b261913276c P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:38.004806  1873 raft_consensus.cc:2749] T e26344297fff44bb8d2099b8e64d7df2 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:58:38.023670  1796 tablet_bootstrap.cc:654] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:38.030735  1795 tablet_bootstrap.cc:492] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701: No bootstrap required, opened a new log
I20260709 12:58:38.031289  1795 ts_tablet_manager.cc:1403] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701: Time spent bootstrapping tablet: real 0.370s	user 0.022s	sys 0.004s
I20260709 12:58:38.041188  1795 raft_consensus.cc:359] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:38.041819  1930 raft_consensus.cc:493] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:38.042398  1930 raft_consensus.cc:515] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:38.043042  1604 raft_consensus.cc:2468] T a40e423ed583480280352b261913276c P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 1.
I20260709 12:58:38.041988  1795 raft_consensus.cc:385] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:38.043648  1795 raft_consensus.cc:740] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Initialized, Role: FOLLOWER
I20260709 12:58:38.044601  1930 leader_election.cc:290] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:58:38.044431  1795 consensus_queue.cc:260] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:38.046537  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:58:38.046654  1643 leader_election.cc:304] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
W20260709 12:58:38.048240  1569 leader_election.cc:343] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:38.048563  1902 raft_consensus.cc:2804] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:58:38.049100  1902 raft_consensus.cc:697] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 1 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 12:58:38.049338  1502 catalog_manager.cc:5697] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d reported cstate change: term changed from 0 to 1, leader changed from <none> to 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194). New cstate: current_term: 1 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } }
I20260709 12:58:38.049966  1902 consensus_queue.cc:237] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:38.052570  1795 ts_tablet_manager.cc:1434] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701: Time spent starting tablet: real 0.021s	user 0.005s	sys 0.002s
I20260709 12:58:38.053565  1795 tablet_bootstrap.cc:492] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701: Bootstrap starting.
I20260709 12:58:38.055887  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 12:58:38.057186  1571 leader_election.cc:343] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:38.057618  1571 leader_election.cc:304] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 12:58:38.065837  1930 raft_consensus.cc:2749] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:58:38.073087  1794 tablet_bootstrap.cc:492] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d: No bootstrap required, opened a new log
I20260709 12:58:38.073638  1794 ts_tablet_manager.cc:1403] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d: Time spent bootstrapping tablet: real 0.170s	user 0.020s	sys 0.021s
I20260709 12:58:38.080096  1795 tablet_bootstrap.cc:654] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:38.080050  1500 catalog_manager.cc:5697] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d reported cstate change: term changed from 0 to 1, leader changed from <none> to 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194). New cstate: current_term: 1 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } }
I20260709 12:58:38.083429  1794 raft_consensus.cc:359] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:38.084313  1794 raft_consensus.cc:385] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:38.084671  1794 raft_consensus.cc:740] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Initialized, Role: FOLLOWER
I20260709 12:58:38.085647  1794 consensus_queue.cc:260] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:38.088786  1794 ts_tablet_manager.cc:1434] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d: Time spent starting tablet: real 0.015s	user 0.009s	sys 0.000s
I20260709 12:58:38.090024  1794 tablet_bootstrap.cc:492] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d: Bootstrap starting.
I20260709 12:58:38.098526  1795 tablet_bootstrap.cc:492] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701: No bootstrap required, opened a new log
I20260709 12:58:38.099104  1795 ts_tablet_manager.cc:1403] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701: Time spent bootstrapping tablet: real 0.046s	user 0.012s	sys 0.011s
I20260709 12:58:38.100415  1794 tablet_bootstrap.cc:654] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:38.101728  1795 raft_consensus.cc:359] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:38.102259  1795 raft_consensus.cc:385] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:38.102582  1795 raft_consensus.cc:740] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Initialized, Role: FOLLOWER
I20260709 12:58:38.103382  1795 consensus_queue.cc:260] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:38.104554  1796 tablet_bootstrap.cc:492] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a: No bootstrap required, opened a new log
I20260709 12:58:38.105119  1796 ts_tablet_manager.cc:1403] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent bootstrapping tablet: real 0.149s	user 0.025s	sys 0.022s
I20260709 12:58:38.109400  1753 raft_consensus.cc:3060] T e26344297fff44bb8d2099b8e64d7df2 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:38.112319  1795 ts_tablet_manager.cc:1434] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701: Time spent starting tablet: real 0.013s	user 0.007s	sys 0.000s
I20260709 12:58:38.113610  1795 tablet_bootstrap.cc:492] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701: Bootstrap starting.
I20260709 12:58:38.127130  1796 raft_consensus.cc:359] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:38.127753  1796 raft_consensus.cc:385] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:38.127996  1796 raft_consensus.cc:740] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Initialized, Role: FOLLOWER
I20260709 12:58:38.128654  1796 consensus_queue.cc:260] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:38.130811  1796 ts_tablet_manager.cc:1434] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent starting tablet: real 0.025s	user 0.008s	sys 0.000s
I20260709 12:58:38.132851  1796 tablet_bootstrap.cc:492] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a: Bootstrap starting.
I20260709 12:58:38.133806  1795 tablet_bootstrap.cc:654] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:38.139400  1794 tablet_bootstrap.cc:492] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d: No bootstrap required, opened a new log
I20260709 12:58:38.139952  1794 ts_tablet_manager.cc:1403] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d: Time spent bootstrapping tablet: real 0.050s	user 0.013s	sys 0.021s
I20260709 12:58:38.145649  1794 raft_consensus.cc:359] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:38.146577  1794 raft_consensus.cc:385] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:38.146920  1794 raft_consensus.cc:740] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Initialized, Role: FOLLOWER
I20260709 12:58:38.147818  1794 consensus_queue.cc:260] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:38.148717  1796 tablet_bootstrap.cc:654] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:38.152520  1794 ts_tablet_manager.cc:1434] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d: Time spent starting tablet: real 0.012s	user 0.008s	sys 0.003s
I20260709 12:58:38.154479  1794 tablet_bootstrap.cc:492] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d: Bootstrap starting.
I20260709 12:58:38.159852  1795 tablet_bootstrap.cc:492] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701: No bootstrap required, opened a new log
I20260709 12:58:38.160346  1795 ts_tablet_manager.cc:1403] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701: Time spent bootstrapping tablet: real 0.047s	user 0.010s	sys 0.008s
I20260709 12:58:38.164325  1795 raft_consensus.cc:359] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:38.165045  1795 raft_consensus.cc:385] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:38.165362  1795 raft_consensus.cc:740] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Initialized, Role: FOLLOWER
I20260709 12:58:38.165472  1963 raft_consensus.cc:493] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:38.166071  1795 consensus_queue.cc:260] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:38.166142  1963 raft_consensus.cc:515] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:38.171772  1795 ts_tablet_manager.cc:1434] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701: Time spent starting tablet: real 0.011s	user 0.008s	sys 0.000s
I20260709 12:58:38.172475  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:58:38.172916  1795 tablet_bootstrap.cc:492] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701: Bootstrap starting.
I20260709 12:58:38.173547  1680 raft_consensus.cc:2393] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 1: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 12:58:38.189826  1963 leader_election.cc:290] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:58:38.192909  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:58:38.194835  1571 leader_election.cc:304] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 12:58:38.197319  1963 raft_consensus.cc:3060] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:38.198464  1794 tablet_bootstrap.cc:654] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:38.205124  1795 tablet_bootstrap.cc:654] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:38.220288  1796 tablet_bootstrap.cc:492] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a: No bootstrap required, opened a new log
I20260709 12:58:38.220965  1796 ts_tablet_manager.cc:1403] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent bootstrapping tablet: real 0.088s	user 0.024s	sys 0.023s
I20260709 12:58:38.228350  1795 tablet_bootstrap.cc:492] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701: No bootstrap required, opened a new log
I20260709 12:58:38.228955  1795 ts_tablet_manager.cc:1403] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701: Time spent bootstrapping tablet: real 0.056s	user 0.009s	sys 0.007s
I20260709 12:58:38.225690  1796 raft_consensus.cc:359] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:38.229765  1796 raft_consensus.cc:385] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:38.230110  1796 raft_consensus.cc:740] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Initialized, Role: FOLLOWER
I20260709 12:58:38.231022  1796 consensus_queue.cc:260] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:38.233165  1796 ts_tablet_manager.cc:1434] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent starting tablet: real 0.012s	user 0.008s	sys 0.001s
I20260709 12:58:38.234094  1796 tablet_bootstrap.cc:492] T 79ed08f2339a4b1ead1e70d78eaec2ec P 83595ad0fd5940c0a2bfc7c61acd969a: Bootstrap starting.
I20260709 12:58:38.233543  1795 raft_consensus.cc:359] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:38.238947  1795 raft_consensus.cc:385] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:38.239310  1795 raft_consensus.cc:740] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Initialized, Role: FOLLOWER
I20260709 12:58:38.240181  1795 consensus_queue.cc:260] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:38.238233  1963 raft_consensus.cc:2749] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:58:38.244457  1795 ts_tablet_manager.cc:1434] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701: Time spent starting tablet: real 0.015s	user 0.007s	sys 0.003s
I20260709 12:58:38.244592  1902 raft_consensus.cc:493] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:38.245721  1902 raft_consensus.cc:515] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:38.246707  1795 tablet_bootstrap.cc:492] T dd3f391a609743de886c525de483fc29 P af520b67945d443b901bbeca37cc5701: Bootstrap starting.
I20260709 12:58:38.249763  1796 tablet_bootstrap.cc:654] T 79ed08f2339a4b1ead1e70d78eaec2ec P 83595ad0fd5940c0a2bfc7c61acd969a: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:38.251995  1794 tablet_bootstrap.cc:492] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d: No bootstrap required, opened a new log
I20260709 12:58:38.252864  1794 ts_tablet_manager.cc:1403] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent bootstrapping tablet: real 0.099s	user 0.036s	sys 0.010s
I20260709 12:58:38.261443  1794 raft_consensus.cc:359] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:38.262743  1794 raft_consensus.cc:385] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:38.263106  1794 raft_consensus.cc:740] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Initialized, Role: FOLLOWER
I20260709 12:58:38.264140  1794 consensus_queue.cc:260] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:38.267442  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:58:38.268254  1794 ts_tablet_manager.cc:1434] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent starting tablet: real 0.015s	user 0.012s	sys 0.000s
I20260709 12:58:38.269579  1794 tablet_bootstrap.cc:492] T 77d66b92688a42e3ad231959a644db56 P 816fadcea8fc4893882f5f9b9f22a03d: Bootstrap starting.
W20260709 12:58:38.269378  1646 leader_election.cc:343] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:38.269941  1902 leader_election.cc:290] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:58:38.276803  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:58:38.281546  1794 tablet_bootstrap.cc:654] T 77d66b92688a42e3ad231959a644db56 P 816fadcea8fc4893882f5f9b9f22a03d: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:38.284313  1795 tablet_bootstrap.cc:654] T dd3f391a609743de886c525de483fc29 P af520b67945d443b901bbeca37cc5701: Neither blocks nor log segments found. Creating new log.
W20260709 12:58:38.284090  1643 leader_election.cc:343] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:38.284842  1643 leader_election.cc:304] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 12:58:38.287102  1902 raft_consensus.cc:2749] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:58:38.302079  1979 consensus_queue.cc:1048] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:58:38.330166  1796 tablet_bootstrap.cc:492] T 79ed08f2339a4b1ead1e70d78eaec2ec P 83595ad0fd5940c0a2bfc7c61acd969a: No bootstrap required, opened a new log
I20260709 12:58:38.334584  1796 ts_tablet_manager.cc:1403] T 79ed08f2339a4b1ead1e70d78eaec2ec P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent bootstrapping tablet: real 0.101s	user 0.033s	sys 0.035s
I20260709 12:58:38.337054  1795 tablet_bootstrap.cc:492] T dd3f391a609743de886c525de483fc29 P af520b67945d443b901bbeca37cc5701: No bootstrap required, opened a new log
I20260709 12:58:38.337863  1795 ts_tablet_manager.cc:1403] T dd3f391a609743de886c525de483fc29 P af520b67945d443b901bbeca37cc5701: Time spent bootstrapping tablet: real 0.091s	user 0.026s	sys 0.026s
I20260709 12:58:38.349689  1795 raft_consensus.cc:359] T dd3f391a609743de886c525de483fc29 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:38.350548  1795 raft_consensus.cc:385] T dd3f391a609743de886c525de483fc29 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:38.350884  1795 raft_consensus.cc:740] T dd3f391a609743de886c525de483fc29 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Initialized, Role: FOLLOWER
I20260709 12:58:38.357126  1873 consensus_queue.cc:1048] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:58:38.361796  1902 raft_consensus.cc:493] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:38.363274  1794 tablet_bootstrap.cc:492] T 77d66b92688a42e3ad231959a644db56 P 816fadcea8fc4893882f5f9b9f22a03d: No bootstrap required, opened a new log
I20260709 12:58:38.362579  1902 raft_consensus.cc:515] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:38.363773  1794 ts_tablet_manager.cc:1403] T 77d66b92688a42e3ad231959a644db56 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent bootstrapping tablet: real 0.094s	user 0.033s	sys 0.008s
I20260709 12:58:38.365923  1980 raft_consensus.cc:493] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:38.361054  1796 raft_consensus.cc:359] T 79ed08f2339a4b1ead1e70d78eaec2ec P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:38.351930  1795 consensus_queue.cc:260] T dd3f391a609743de886c525de483fc29 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:38.366827  1980 raft_consensus.cc:515] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:38.367736  1794 raft_consensus.cc:359] T 77d66b92688a42e3ad231959a644db56 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:38.368589  1794 raft_consensus.cc:385] T 77d66b92688a42e3ad231959a644db56 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:38.368917  1794 raft_consensus.cc:740] T 77d66b92688a42e3ad231959a644db56 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Initialized, Role: FOLLOWER
I20260709 12:58:38.370014  1794 consensus_queue.cc:260] T 77d66b92688a42e3ad231959a644db56 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:38.371503  1979 raft_consensus.cc:493] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:38.367897  1796 raft_consensus.cc:385] T 79ed08f2339a4b1ead1e70d78eaec2ec P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:38.373797  1796 raft_consensus.cc:740] T 79ed08f2339a4b1ead1e70d78eaec2ec P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Initialized, Role: FOLLOWER
I20260709 12:58:38.372254  1979 raft_consensus.cc:515] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:38.374706  1796 consensus_queue.cc:260] T 79ed08f2339a4b1ead1e70d78eaec2ec P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:38.375674  1794 ts_tablet_manager.cc:1434] T 77d66b92688a42e3ad231959a644db56 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent starting tablet: real 0.012s	user 0.010s	sys 0.000s
I20260709 12:58:38.372828  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:58:38.377715  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:58:38.372495  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:58:38.378672  1601 raft_consensus.cc:2468] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 0.
W20260709 12:58:38.379020  1717 leader_election.cc:343] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:38.379343  1680 raft_consensus.cc:2468] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 0.
I20260709 12:58:38.380290  1717 leader_election.cc:304] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:58:38.376890  1979 leader_election.cc:290] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:58:38.381181  1718 leader_election.cc:304] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:58:38.381604  1830 raft_consensus.cc:2804] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:58:38.382164  1830 raft_consensus.cc:493] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:58:38.382239  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:58:38.382783  1830 raft_consensus.cc:3060] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:38.383387  1604 raft_consensus.cc:2468] T 513043a9b96c4f70adad53ac4b5d01dc P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 0.
I20260709 12:58:38.385315  1643 leader_election.cc:304] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 12:58:38.383457  1979 raft_consensus.cc:2804] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:58:38.378180  1794 tablet_bootstrap.cc:492] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d: Bootstrap starting.
I20260709 12:58:38.384371  1796 ts_tablet_manager.cc:1434] T 79ed08f2339a4b1ead1e70d78eaec2ec P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent starting tablet: real 0.049s	user 0.009s	sys 0.000s
I20260709 12:58:38.392558  1796 tablet_bootstrap.cc:492] T 03daa4b346f54f1da820d1ff6cd1ff86 P 83595ad0fd5940c0a2bfc7c61acd969a: Bootstrap starting.
I20260709 12:58:38.389016  1795 ts_tablet_manager.cc:1434] T dd3f391a609743de886c525de483fc29 P af520b67945d443b901bbeca37cc5701: Time spent starting tablet: real 0.051s	user 0.009s	sys 0.001s
I20260709 12:58:38.388870  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:58:38.387375  1980 leader_election.cc:290] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:58:38.397462  1754 raft_consensus.cc:2468] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 0.
I20260709 12:58:38.389605  1861 raft_consensus.cc:2804] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:58:38.390005  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:58:38.386632  1902 leader_election.cc:290] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:58:38.396839  1795 tablet_bootstrap.cc:492] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701: Bootstrap starting.
I20260709 12:58:38.398562  1928 raft_consensus.cc:493] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:38.391113  1979 raft_consensus.cc:493] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:58:38.398943  1861 raft_consensus.cc:493] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:58:38.408038  1861 raft_consensus.cc:3060] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:38.404006  1676 raft_consensus.cc:2468] T 77d66b92688a42e3ad231959a644db56 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 0.
I20260709 12:58:38.430928  1980 raft_consensus.cc:493] T dd3f391a609743de886c525de483fc29 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:38.431876  1980 raft_consensus.cc:515] T dd3f391a609743de886c525de483fc29 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:38.435930  1980 leader_election.cc:290] T dd3f391a609743de886c525de483fc29 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:58:38.438057  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:58:38.446228  1980 raft_consensus.cc:493] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:38.446398  1861 raft_consensus.cc:515] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:38.447422  1980 raft_consensus.cc:515] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:38.422509  1928 raft_consensus.cc:515] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:38.403723  1979 raft_consensus.cc:3060] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:38.450409  1830 raft_consensus.cc:515] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:38.451913  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:58:38.453833  1830 leader_election.cc:290] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:58:38.455485  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:58:38.455813  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:58:38.456568  1861 leader_election.cc:290] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:58:38.456789  1601 raft_consensus.cc:2393] T a40e423ed583480280352b261913276c P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 1: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 12:58:38.456930  1754 raft_consensus.cc:3060] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:38.459473  1794 tablet_bootstrap.cc:654] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:38.459363  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:58:38.460428  1601 raft_consensus.cc:3060] T 513043a9b96c4f70adad53ac4b5d01dc P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:38.446178  1604 raft_consensus.cc:2393] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 1: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 12:58:38.464826  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:58:38.465045  1717 leader_election.cc:304] T dd3f391a609743de886c525de483fc29 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:58:38.459277  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:58:38.471647  1796 tablet_bootstrap.cc:654] T 03daa4b346f54f1da820d1ff6cd1ff86 P 83595ad0fd5940c0a2bfc7c61acd969a: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:38.474560  1605 raft_consensus.cc:3060] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:38.477383  1830 raft_consensus.cc:3060] T dd3f391a609743de886c525de483fc29 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:38.478380  1795 tablet_bootstrap.cc:654] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:38.480679  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:58:38.465711  1676 raft_consensus.cc:3060] T 77d66b92688a42e3ad231959a644db56 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:38.477895  1928 leader_election.cc:290] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 12:58:38.495568  1569 leader_election.cc:343] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:38.498797  1861 raft_consensus.cc:493] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:38.498991  1601 raft_consensus.cc:2468] T 513043a9b96c4f70adad53ac4b5d01dc P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 1.
I20260709 12:58:38.499554  1861 raft_consensus.cc:515] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:38.500427  1980 leader_election.cc:290] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:58:38.505134  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:58:38.508772  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:58:38.509876  1985 consensus_queue.cc:1048] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260709 12:58:38.511268  1646 leader_election.cc:343] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:38.505491  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:58:38.513125  1643 leader_election.cc:304] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 12:58:38.513903  1861 leader_election.cc:290] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:58:38.514837  1718 leader_election.cc:304] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 12:58:38.516062  1571 leader_election.cc:343] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:38.514812  1985 raft_consensus.cc:2804] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:58:38.516742  1571 leader_election.cc:304] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 12:58:38.518787  1985 raft_consensus.cc:697] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 1 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 12:58:38.520126  1985 consensus_queue.cc:237] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:38.514007  1605 raft_consensus.cc:2468] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 1.
I20260709 12:58:38.527055  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:58:38.528002  1601 raft_consensus.cc:2468] T 79ed08f2339a4b1ead1e70d78eaec2ec P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 0.
I20260709 12:58:38.524010  1980 raft_consensus.cc:3060] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:38.524576  1963 raft_consensus.cc:2749] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:58:38.540071  1717 leader_election.cc:304] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:58:38.544304  1873 raft_consensus.cc:2804] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:58:38.544837  1676 raft_consensus.cc:2468] T 77d66b92688a42e3ad231959a644db56 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 1.
I20260709 12:58:38.553700  1794 tablet_bootstrap.cc:492] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d: No bootstrap required, opened a new log
I20260709 12:58:38.554126  1794 ts_tablet_manager.cc:1403] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d: Time spent bootstrapping tablet: real 0.176s	user 0.018s	sys 0.013s
I20260709 12:58:38.559158  1830 raft_consensus.cc:2749] T dd3f391a609743de886c525de483fc29 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:58:38.559710  1979 raft_consensus.cc:515] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:38.555728  1643 leader_election.cc:304] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 12:58:38.562682  1861 raft_consensus.cc:2804] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:58:38.563058  1950 consensus_queue.cc:1048] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:58:38.563073  1861 raft_consensus.cc:493] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:58:38.564911  1861 raft_consensus.cc:3060] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:38.576326  1796 tablet_bootstrap.cc:492] T 03daa4b346f54f1da820d1ff6cd1ff86 P 83595ad0fd5940c0a2bfc7c61acd969a: No bootstrap required, opened a new log
I20260709 12:58:38.576996  1796 ts_tablet_manager.cc:1403] T 03daa4b346f54f1da820d1ff6cd1ff86 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent bootstrapping tablet: real 0.185s	user 0.026s	sys 0.053s
I20260709 12:58:38.577479  1795 tablet_bootstrap.cc:492] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701: No bootstrap required, opened a new log
I20260709 12:58:38.578028  1795 ts_tablet_manager.cc:1403] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701: Time spent bootstrapping tablet: real 0.181s	user 0.022s	sys 0.010s
I20260709 12:58:38.576498  1794 raft_consensus.cc:359] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:38.578403  1794 raft_consensus.cc:385] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:38.578754  1794 raft_consensus.cc:740] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Initialized, Role: FOLLOWER
I20260709 12:58:38.587494  1794 consensus_queue.cc:260] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:38.572577  1873 raft_consensus.cc:697] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 1 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 12:58:38.593705  1794 ts_tablet_manager.cc:1434] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d: Time spent starting tablet: real 0.039s	user 0.008s	sys 0.000s
I20260709 12:58:38.594620  1794 tablet_bootstrap.cc:492] T 1809d674329a492dbea0b60a4703fe51 P 816fadcea8fc4893882f5f9b9f22a03d: Bootstrap starting.
I20260709 12:58:38.589457  1754 raft_consensus.cc:2468] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 1.
I20260709 12:58:38.610419  1873 consensus_queue.cc:237] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:38.605669  1979 leader_election.cc:290] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:58:38.618621  1795 raft_consensus.cc:359] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:38.619354  1795 raft_consensus.cc:385] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:38.619644  1795 raft_consensus.cc:740] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Initialized, Role: FOLLOWER
I20260709 12:58:38.620363  1795 consensus_queue.cc:260] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:38.622753  1950 raft_consensus.cc:493] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:38.623308  1950 raft_consensus.cc:515] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 12:58:38.624804  1551 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:58:38.625528  1950 leader_election.cc:290] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:58:38.629438  1795 ts_tablet_manager.cc:1434] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701: Time spent starting tablet: real 0.051s	user 0.008s	sys 0.000s
I20260709 12:58:38.630561  1795 tablet_bootstrap.cc:492] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701: Bootstrap starting.
I20260709 12:58:38.631464  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:58:38.632112  1680 raft_consensus.cc:3060] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:38.632761  1902 consensus_queue.cc:1048] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:58:38.627782  1980 raft_consensus.cc:2749] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:58:38.630491  1861 raft_consensus.cc:515] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:38.648772  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:58:38.649747  1680 raft_consensus.cc:2468] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 1.
W20260709 12:58:38.652712  1646 leader_election.cc:343] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:38.653491  1902 consensus_queue.cc:1048] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:58:38.663597  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:58:38.664239  1796 raft_consensus.cc:359] T 03daa4b346f54f1da820d1ff6cd1ff86 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:38.664937  1796 raft_consensus.cc:385] T 03daa4b346f54f1da820d1ff6cd1ff86 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:38.665202  1796 raft_consensus.cc:740] T 03daa4b346f54f1da820d1ff6cd1ff86 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Initialized, Role: FOLLOWER
I20260709 12:58:38.666267  1794 tablet_bootstrap.cc:654] T 1809d674329a492dbea0b60a4703fe51 P 816fadcea8fc4893882f5f9b9f22a03d: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:38.667900  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:58:38.668931  1502 catalog_manager.cc:5697] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d reported cstate change: term changed from 0 to 1, leader changed from <none> to 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194). New cstate: current_term: 1 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } }
I20260709 12:58:38.672315  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:58:38.672446  1718 leader_election.cc:304] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:58:38.678838  1795 tablet_bootstrap.cc:654] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701: Neither blocks nor log segments found. Creating new log.
W20260709 12:58:38.679391  1646 leader_election.cc:343] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:38.669682  1861 leader_election.cc:290] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:58:38.679706  1980 raft_consensus.cc:2804] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Leader election won for term 1
W20260709 12:58:38.679733  1717 leader_election.cc:343] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:38.680576  1980 raft_consensus.cc:697] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 1 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 12:58:38.681509  1980 consensus_queue.cc:237] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:38.733060  1796 consensus_queue.cc:260] T 03daa4b346f54f1da820d1ff6cd1ff86 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:38.750540  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:58:38.751475  1602 raft_consensus.cc:3060] T 79ed08f2339a4b1ead1e70d78eaec2ec P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:38.752727  1796 ts_tablet_manager.cc:1434] T 03daa4b346f54f1da820d1ff6cd1ff86 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent starting tablet: real 0.175s	user 0.008s	sys 0.000s
I20260709 12:58:38.754696  1796 tablet_bootstrap.cc:492] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a: Bootstrap starting.
W20260709 12:58:38.752079  1643 leader_election.cc:343] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:38.762380  1643 leader_election.cc:304] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 12:58:38.781278  1993 raft_consensus.cc:2749] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:58:38.799124  1795 tablet_bootstrap.cc:492] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701: No bootstrap required, opened a new log
I20260709 12:58:38.799882  1795 ts_tablet_manager.cc:1403] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701: Time spent bootstrapping tablet: real 0.170s	user 0.012s	sys 0.007s
I20260709 12:58:38.801698  1794 tablet_bootstrap.cc:492] T 1809d674329a492dbea0b60a4703fe51 P 816fadcea8fc4893882f5f9b9f22a03d: No bootstrap required, opened a new log
I20260709 12:58:38.802359  1794 ts_tablet_manager.cc:1403] T 1809d674329a492dbea0b60a4703fe51 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent bootstrapping tablet: real 0.208s	user 0.018s	sys 0.006s
I20260709 12:58:38.806874  1794 raft_consensus.cc:359] T 1809d674329a492dbea0b60a4703fe51 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:38.807711  1794 raft_consensus.cc:385] T 1809d674329a492dbea0b60a4703fe51 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:38.807899  1602 raft_consensus.cc:2468] T 79ed08f2339a4b1ead1e70d78eaec2ec P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 1.
I20260709 12:58:38.808085  1794 raft_consensus.cc:740] T 1809d674329a492dbea0b60a4703fe51 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Initialized, Role: FOLLOWER
I20260709 12:58:38.809000  1794 consensus_queue.cc:260] T 1809d674329a492dbea0b60a4703fe51 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:38.811002  1795 raft_consensus.cc:359] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:38.811533  1795 raft_consensus.cc:385] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:38.811746  1795 raft_consensus.cc:740] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Initialized, Role: FOLLOWER
I20260709 12:58:38.812273  1795 consensus_queue.cc:260] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:38.822748  1795 ts_tablet_manager.cc:1434] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701: Time spent starting tablet: real 0.022s	user 0.008s	sys 0.000s
I20260709 12:58:38.818979  1500 catalog_manager.cc:5697] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 reported cstate change: term changed from 0 to 1, leader changed from <none> to af520b67945d443b901bbeca37cc5701 (127.0.107.195). New cstate: current_term: 1 leader_uuid: "af520b67945d443b901bbeca37cc5701" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: HEALTHY } } }
I20260709 12:58:38.823921  1795 tablet_bootstrap.cc:492] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701: Bootstrap starting.
I20260709 12:58:38.826201  1794 ts_tablet_manager.cc:1434] T 1809d674329a492dbea0b60a4703fe51 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent starting tablet: real 0.023s	user 0.008s	sys 0.001s
I20260709 12:58:38.827214  1796 tablet_bootstrap.cc:654] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:38.827319  1794 tablet_bootstrap.cc:492] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d: Bootstrap starting.
I20260709 12:58:38.831372  1643 leader_election.cc:304] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 12:58:38.832584  1993 raft_consensus.cc:2804] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:58:38.834528  1993 raft_consensus.cc:697] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 1 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 12:58:38.835527  1993 consensus_queue.cc:237] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:38.838177  1991 raft_consensus.cc:493] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:38.838718  1991 raft_consensus.cc:515] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:38.841346  1991 leader_election.cc:290] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:58:38.843495  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:58:38.854507  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:58:38.856072  1605 raft_consensus.cc:2393] T 79ed08f2339a4b1ead1e70d78eaec2ec P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 1: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 12:58:38.863096  1795 tablet_bootstrap.cc:654] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:38.864663  1796 tablet_bootstrap.cc:492] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a: No bootstrap required, opened a new log
I20260709 12:58:38.865585  1796 ts_tablet_manager.cc:1403] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent bootstrapping tablet: real 0.112s	user 0.021s	sys 0.048s
I20260709 12:58:38.869999  1717 leader_election.cc:304] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:58:38.871277  1991 raft_consensus.cc:3060] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:38.873832  1796 raft_consensus.cc:359] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:38.877136  1796 raft_consensus.cc:385] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:38.877568  1796 raft_consensus.cc:740] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Initialized, Role: FOLLOWER
I20260709 12:58:38.878530  1796 consensus_queue.cc:260] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:38.881494  1796 ts_tablet_manager.cc:1434] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent starting tablet: real 0.016s	user 0.005s	sys 0.002s
I20260709 12:58:38.875521  1794 tablet_bootstrap.cc:654] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:38.885859  1991 raft_consensus.cc:2749] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:58:38.888656  1796 tablet_bootstrap.cc:492] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a: Bootstrap starting.
I20260709 12:58:38.901479  1795 tablet_bootstrap.cc:492] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701: No bootstrap required, opened a new log
I20260709 12:58:38.902123  1795 ts_tablet_manager.cc:1403] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701: Time spent bootstrapping tablet: real 0.078s	user 0.010s	sys 0.026s
I20260709 12:58:38.905656  1794 tablet_bootstrap.cc:492] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d: No bootstrap required, opened a new log
I20260709 12:58:38.906137  1794 ts_tablet_manager.cc:1403] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent bootstrapping tablet: real 0.079s	user 0.011s	sys 0.006s
I20260709 12:58:38.909932  1796 tablet_bootstrap.cc:654] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:38.910408  1794 raft_consensus.cc:359] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:38.911707  1794 raft_consensus.cc:385] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:38.911937  1794 raft_consensus.cc:740] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Initialized, Role: FOLLOWER
I20260709 12:58:38.915969  1794 consensus_queue.cc:260] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:38.920172  1794 ts_tablet_manager.cc:1434] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent starting tablet: real 0.014s	user 0.009s	sys 0.000s
I20260709 12:58:38.921258  1794 tablet_bootstrap.cc:492] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d: Bootstrap starting.
I20260709 12:58:38.926970  1795 raft_consensus.cc:359] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:38.935537  1795 raft_consensus.cc:385] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:38.935889  1795 raft_consensus.cc:740] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Initialized, Role: FOLLOWER
I20260709 12:58:38.936589  1795 consensus_queue.cc:260] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:38.939079  1795 ts_tablet_manager.cc:1434] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701: Time spent starting tablet: real 0.037s	user 0.008s	sys 0.006s
I20260709 12:58:38.940085  1795 tablet_bootstrap.cc:492] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701: Bootstrap starting.
I20260709 12:58:38.945080  1794 tablet_bootstrap.cc:654] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:38.948580  1795 tablet_bootstrap.cc:654] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:38.927536  1500 catalog_manager.cc:5697] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d reported cstate change: term changed from 0 to 1, leader changed from <none> to 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194). New cstate: current_term: 1 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } }
I20260709 12:58:38.974860  1993 consensus_queue.cc:1048] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:58:38.994817  1993 consensus_queue.cc:1048] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:58:39.004463  1993 raft_consensus.cc:493] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:39.005821  1993 raft_consensus.cc:515] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:39.002234  1501 catalog_manager.cc:5697] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 reported cstate change: term changed from 0 to 1, leader changed from <none> to af520b67945d443b901bbeca37cc5701 (127.0.107.195). New cstate: current_term: 1 leader_uuid: "af520b67945d443b901bbeca37cc5701" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: HEALTHY } } }
I20260709 12:58:39.019814  1993 leader_election.cc:290] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:58:39.023445  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 12:58:39.029604  1643 leader_election.cc:343] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:39.048067  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:58:39.056875  1794 tablet_bootstrap.cc:492] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d: No bootstrap required, opened a new log
I20260709 12:58:39.057389  1795 tablet_bootstrap.cc:492] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701: No bootstrap required, opened a new log
I20260709 12:58:39.057458  1794 ts_tablet_manager.cc:1403] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent bootstrapping tablet: real 0.136s	user 0.024s	sys 0.036s
I20260709 12:58:39.058020  1795 ts_tablet_manager.cc:1403] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701: Time spent bootstrapping tablet: real 0.118s	user 0.021s	sys 0.035s
I20260709 12:58:39.059780  1991 consensus_queue.cc:1048] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:58:39.062336  1795 raft_consensus.cc:359] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:39.062664  1794 raft_consensus.cc:359] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:39.063158  1795 raft_consensus.cc:385] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:39.063333  1794 raft_consensus.cc:385] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:39.063488  1795 raft_consensus.cc:740] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Initialized, Role: FOLLOWER
I20260709 12:58:39.063642  1794 raft_consensus.cc:740] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Initialized, Role: FOLLOWER
I20260709 12:58:39.064271  1795 consensus_queue.cc:260] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:39.064316  1794 consensus_queue.cc:260] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 12:58:39.065925  1646 leader_election.cc:343] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:39.066725  1794 ts_tablet_manager.cc:1434] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent starting tablet: real 0.009s	user 0.004s	sys 0.003s
I20260709 12:58:39.067083  1795 ts_tablet_manager.cc:1434] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.001s
I20260709 12:58:39.066665  1646 leader_election.cc:304] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 12:58:39.067852  1794 tablet_bootstrap.cc:492] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d: Bootstrap starting.
I20260709 12:58:39.067924  1795 tablet_bootstrap.cc:492] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701: Bootstrap starting.
I20260709 12:58:39.068897  1950 raft_consensus.cc:2749] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:58:39.080305  1795 tablet_bootstrap.cc:654] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:39.080926  1794 tablet_bootstrap.cc:654] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:39.084564  1796 tablet_bootstrap.cc:492] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a: No bootstrap required, opened a new log
I20260709 12:58:39.085081  1796 ts_tablet_manager.cc:1403] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent bootstrapping tablet: real 0.197s	user 0.014s	sys 0.055s
I20260709 12:58:39.088964  1796 raft_consensus.cc:359] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:39.089829  1796 raft_consensus.cc:385] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:39.090188  1796 raft_consensus.cc:740] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Initialized, Role: FOLLOWER
I20260709 12:58:39.091136  1796 consensus_queue.cc:260] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:39.094106  1796 ts_tablet_manager.cc:1434] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent starting tablet: real 0.009s	user 0.005s	sys 0.000s
I20260709 12:58:39.094836  1950 raft_consensus.cc:493] T 1809d674329a492dbea0b60a4703fe51 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:39.095498  1950 raft_consensus.cc:515] T 1809d674329a492dbea0b60a4703fe51 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:39.100216  1796 tablet_bootstrap.cc:492] T 87b86d46388a41cd90374c203cb27914 P 83595ad0fd5940c0a2bfc7c61acd969a: Bootstrap starting.
I20260709 12:58:39.105038  1980 consensus_queue.cc:1048] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:58:39.107776  1980 raft_consensus.cc:493] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:39.108561  1980 raft_consensus.cc:515] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:39.115374  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 12:58:39.117775  1646 leader_election.cc:343] T 1809d674329a492dbea0b60a4703fe51 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:39.119462  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 12:58:39.121449  1643 leader_election.cc:343] T 1809d674329a492dbea0b60a4703fe51 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:39.122148  1643 leader_election.cc:304] T 1809d674329a492dbea0b60a4703fe51 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 12:58:39.125094  1993 raft_consensus.cc:2749] T 1809d674329a492dbea0b60a4703fe51 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:58:39.130273  1794 tablet_bootstrap.cc:492] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d: No bootstrap required, opened a new log
I20260709 12:58:39.130925  1794 ts_tablet_manager.cc:1403] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent bootstrapping tablet: real 0.063s	user 0.021s	sys 0.035s
I20260709 12:58:39.132817  1980 leader_election.cc:290] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:58:39.131500  1950 leader_election.cc:290] T 1809d674329a492dbea0b60a4703fe51 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:58:39.191509  1928 raft_consensus.cc:493] T 03daa4b346f54f1da820d1ff6cd1ff86 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:39.203940  1794 raft_consensus.cc:359] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:39.204761  1928 raft_consensus.cc:515] T 03daa4b346f54f1da820d1ff6cd1ff86 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:39.205811  1794 raft_consensus.cc:385] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:39.206259  1794 raft_consensus.cc:740] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Initialized, Role: FOLLOWER
I20260709 12:58:39.190004  1795 tablet_bootstrap.cc:492] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701: No bootstrap required, opened a new log
I20260709 12:58:39.207520  1794 consensus_queue.cc:260] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:39.211452  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:58:39.213006  1676 raft_consensus.cc:2468] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 0.
I20260709 12:58:39.213272  1795 ts_tablet_manager.cc:1403] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701: Time spent bootstrapping tablet: real 0.146s	user 0.018s	sys 0.054s
I20260709 12:58:39.217610  1985 raft_consensus.cc:493] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:39.218580  1985 raft_consensus.cc:515] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:39.224224  1928 leader_election.cc:290] T 03daa4b346f54f1da820d1ff6cd1ff86 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:58:39.227080  1718 leader_election.cc:304] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:58:39.213910  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:58:39.230144  1605 raft_consensus.cc:2468] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 0.
I20260709 12:58:39.228951  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:58:39.222123  1985 leader_election.cc:290] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:58:39.232391  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:58:39.233268  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:58:39.234131  1674 raft_consensus.cc:2393] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 1: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 12:58:39.235417  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:58:39.236510  1605 raft_consensus.cc:2468] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 0.
I20260709 12:58:39.245221  1991 raft_consensus.cc:2804] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:58:39.245779  1991 raft_consensus.cc:493] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:58:39.246169  1991 raft_consensus.cc:3060] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:39.251113  1571 leader_election.cc:304] T 03daa4b346f54f1da820d1ff6cd1ff86 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 12:58:39.253305  1962 raft_consensus.cc:3060] T 03daa4b346f54f1da820d1ff6cd1ff86 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:39.258738  1980 consensus_queue.cc:1048] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:58:39.260284  1794 ts_tablet_manager.cc:1434] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent starting tablet: real 0.129s	user 0.005s	sys 0.006s
I20260709 12:58:39.264158  1794 tablet_bootstrap.cc:492] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 816fadcea8fc4893882f5f9b9f22a03d: Bootstrap starting.
I20260709 12:58:39.262908  1796 tablet_bootstrap.cc:654] T 87b86d46388a41cd90374c203cb27914 P 83595ad0fd5940c0a2bfc7c61acd969a: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:39.278326  1643 leader_election.cc:304] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
W20260709 12:58:39.279762  1646 leader_election.cc:343] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:39.280438  1985 raft_consensus.cc:2804] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:58:39.280920  1985 raft_consensus.cc:493] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:58:39.281275  1985 raft_consensus.cc:3060] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:39.260985  1795 raft_consensus.cc:359] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:39.283679  1795 raft_consensus.cc:385] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:39.284056  1795 raft_consensus.cc:740] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Initialized, Role: FOLLOWER
I20260709 12:58:39.284864  1795 consensus_queue.cc:260] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:39.293083  1795 ts_tablet_manager.cc:1434] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701: Time spent starting tablet: real 0.079s	user 0.005s	sys 0.005s
I20260709 12:58:39.293507  1962 raft_consensus.cc:2749] T 03daa4b346f54f1da820d1ff6cd1ff86 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:58:39.301923  1985 raft_consensus.cc:515] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:39.308429  1991 raft_consensus.cc:515] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:39.311717  1985 leader_election.cc:290] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:58:39.313407  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:58:39.314050  1604 raft_consensus.cc:3060] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:39.315766  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:58:39.316521  1601 raft_consensus.cc:3060] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:39.312490  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
W20260709 12:58:39.320423  1646 leader_election.cc:343] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:39.321283  1902 raft_consensus.cc:493] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:39.332949  1993 raft_consensus.cc:493] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:39.336109  1902 raft_consensus.cc:515] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:39.321235  1873 raft_consensus.cc:493] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:39.337585  1873 raft_consensus.cc:515] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:39.340111  1873 leader_election.cc:290] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:58:39.339984  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:58:39.341320  1873 consensus_queue.cc:1048] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260709 12:58:39.342308  1643 leader_election.cc:343] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:39.347178  1902 leader_election.cc:290] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:58:39.350132  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:58:39.336109  1993 raft_consensus.cc:515] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:39.353811  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:58:39.355244  1993 leader_election.cc:290] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:58:39.350132  1979 raft_consensus.cc:493] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:39.356163  1979 raft_consensus.cc:515] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:39.356729  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:58:39.357193  1750 raft_consensus.cc:2468] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 0.
I20260709 12:58:39.357285  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:58:39.358107  1602 raft_consensus.cc:2468] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 0.
W20260709 12:58:39.358748  1717 leader_election.cc:343] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:39.359452  1643 leader_election.cc:304] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
W20260709 12:58:39.365068  1646 leader_election.cc:343] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:39.365410  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:58:39.364188  1991 leader_election.cc:290] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:58:39.361572  1993 raft_consensus.cc:2804] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:58:39.367797  1993 raft_consensus.cc:493] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:58:39.368219  1993 raft_consensus.cc:3060] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:39.369554  1795 tablet_bootstrap.cc:492] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701: Bootstrap starting.
I20260709 12:58:39.370636  1902 consensus_queue.cc:1048] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:58:39.359452  1979 leader_election.cc:290] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:58:39.370321  1796 tablet_bootstrap.cc:492] T 87b86d46388a41cd90374c203cb27914 P 83595ad0fd5940c0a2bfc7c61acd969a: No bootstrap required, opened a new log
I20260709 12:58:39.381016  1796 ts_tablet_manager.cc:1403] T 87b86d46388a41cd90374c203cb27914 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent bootstrapping tablet: real 0.281s	user 0.023s	sys 0.007s
I20260709 12:58:39.384799  1604 raft_consensus.cc:2468] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 1.
W20260709 12:58:39.385679  1717 leader_election.cc:343] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:39.386518  1601 raft_consensus.cc:2468] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 1.
I20260709 12:58:39.386855  1717 leader_election.cc:304] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:58:39.388320  1643 leader_election.cc:304] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 12:58:39.389432  2021 raft_consensus.cc:2804] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:58:39.390054  2021 raft_consensus.cc:697] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 1 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 12:58:39.391175  2021 consensus_queue.cc:237] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:39.392314  1794 tablet_bootstrap.cc:654] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 816fadcea8fc4893882f5f9b9f22a03d: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:39.394419  1902 raft_consensus.cc:2804] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:58:39.395013  1902 raft_consensus.cc:697] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 12:58:39.396116  1902 consensus_queue.cc:237] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:39.411820  1646 leader_election.cc:304] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:58:39.412776  1985 raft_consensus.cc:2804] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:58:39.413190  1985 raft_consensus.cc:493] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:58:39.413528  1985 raft_consensus.cc:3060] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:39.414710  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:58:39.415184  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 12:58:39.436861  1718 leader_election.cc:343] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:39.437491  1718 leader_election.cc:304] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:58:39.447911  1873 raft_consensus.cc:2749] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:58:39.451509  1962 raft_consensus.cc:493] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:39.452095  1962 raft_consensus.cc:515] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:39.452785  1796 raft_consensus.cc:359] T 87b86d46388a41cd90374c203cb27914 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:39.453732  1796 raft_consensus.cc:385] T 87b86d46388a41cd90374c203cb27914 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:39.454058  1796 raft_consensus.cc:740] T 87b86d46388a41cd90374c203cb27914 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Initialized, Role: FOLLOWER
I20260709 12:58:39.454663  1962 leader_election.cc:290] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:58:39.453282  1502 catalog_manager.cc:5697] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 reported cstate change: term changed from 0 to 1, leader changed from <none> to af520b67945d443b901bbeca37cc5701 (127.0.107.195). New cstate: current_term: 1 leader_uuid: "af520b67945d443b901bbeca37cc5701" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: HEALTHY } } }
I20260709 12:58:39.454919  1796 consensus_queue.cc:260] T 87b86d46388a41cd90374c203cb27914 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:39.458500  1796 ts_tablet_manager.cc:1434] T 87b86d46388a41cd90374c203cb27914 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent starting tablet: real 0.077s	user 0.006s	sys 0.003s
I20260709 12:58:39.458350  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:58:39.451661  1985 raft_consensus.cc:515] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:39.459209  1795 tablet_bootstrap.cc:654] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:39.459144  1679 raft_consensus.cc:2468] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 0.
I20260709 12:58:39.460068  1796 tablet_bootstrap.cc:492] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a: Bootstrap starting.
I20260709 12:58:39.469918  1677 raft_consensus.cc:2393] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 1: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 12:58:39.476787  1794 tablet_bootstrap.cc:492] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 816fadcea8fc4893882f5f9b9f22a03d: No bootstrap required, opened a new log
I20260709 12:58:39.477272  1794 ts_tablet_manager.cc:1403] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 816fadcea8fc4893882f5f9b9f22a03d: Time spent bootstrapping tablet: real 0.214s	user 0.008s	sys 0.023s
I20260709 12:58:39.480558  1569 leader_election.cc:304] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 12:58:39.483089  1794 raft_consensus.cc:359] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:39.483843  1794 raft_consensus.cc:385] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:39.484115  1794 raft_consensus.cc:740] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Initialized, Role: FOLLOWER
I20260709 12:58:39.484833  1794 consensus_queue.cc:260] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:39.487457  1794 ts_tablet_manager.cc:1434] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 816fadcea8fc4893882f5f9b9f22a03d: Time spent starting tablet: real 0.010s	user 0.009s	sys 0.000s
I20260709 12:58:39.490906  1796 tablet_bootstrap.cc:654] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:39.491519  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:58:39.492515  1963 raft_consensus.cc:2804] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:58:39.493295  1963 raft_consensus.cc:493] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:58:39.493780  1963 raft_consensus.cc:3060] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:39.494604  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:58:39.495558  1676 raft_consensus.cc:3060] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:58:39.498976  1571 leader_election.cc:343] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:39.500370  1501 catalog_manager.cc:5697] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d reported cstate change: term changed from 0 to 1, leader changed from <none> to 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194). New cstate: current_term: 1 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } }
I20260709 12:58:39.470939  1993 raft_consensus.cc:515] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:39.506793  1676 raft_consensus.cc:2468] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 1.
I20260709 12:58:39.508040  1993 leader_election.cc:290] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:58:39.512836  1985 leader_election.cc:290] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:58:39.511000  1993 consensus_queue.cc:1048] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:58:39.516389  2022 raft_consensus.cc:493] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:39.517014  2022 raft_consensus.cc:515] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:39.519023  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:58:39.515731  1718 leader_election.cc:304] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:58:39.520061  1750 raft_consensus.cc:3060] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:39.519094  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:58:39.530319  2022 leader_election.cc:290] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:58:39.530596  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:58:39.532358  1794 tablet_bootstrap.cc:492] T c6ab3300be1e4a59b64b14d50da193ca P 816fadcea8fc4893882f5f9b9f22a03d: Bootstrap starting.
W20260709 12:58:39.535034  1717 leader_election.cc:343] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:39.547369  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:58:39.548029  1601 raft_consensus.cc:3060] T 87b86d46388a41cd90374c203cb27914 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:39.549484  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:58:39.550112  1604 raft_consensus.cc:3060] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:39.550851  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:58:39.551571  1680 raft_consensus.cc:2468] T 1809d674329a492dbea0b60a4703fe51 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 0.
I20260709 12:58:39.553133  1963 raft_consensus.cc:515] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 12:58:39.546751  1646 leader_election.cc:343] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:39.560624  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:58:39.561255  1680 raft_consensus.cc:3060] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:39.563046  1795 tablet_bootstrap.cc:492] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701: No bootstrap required, opened a new log
I20260709 12:58:39.563746  1601 raft_consensus.cc:2468] T 87b86d46388a41cd90374c203cb27914 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 1.
I20260709 12:58:39.563737  1795 ts_tablet_manager.cc:1403] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701: Time spent bootstrapping tablet: real 0.195s	user 0.011s	sys 0.015s
I20260709 12:58:39.565253  1643 leader_election.cc:304] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 12:58:39.566527  1902 raft_consensus.cc:2804] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:58:39.567044  1902 raft_consensus.cc:697] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 12:58:39.567978  1902 consensus_queue.cc:237] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:39.571069  1680 raft_consensus.cc:2468] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 1.
I20260709 12:58:39.572193  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:58:39.574010  1569 leader_election.cc:304] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
W20260709 12:58:39.574690  1571 leader_election.cc:343] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:39.571184  1750 raft_consensus.cc:2468] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 1.
I20260709 12:58:39.579569  1604 raft_consensus.cc:2468] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 1.
I20260709 12:58:39.580148  1718 leader_election.cc:304] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:58:39.580811  1643 leader_election.cc:304] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 12:58:39.581009  2022 raft_consensus.cc:2804] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:58:39.581416  2022 raft_consensus.cc:493] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:58:39.581761  2022 raft_consensus.cc:3060] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:39.582415  1993 raft_consensus.cc:2804] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:58:39.566536  1963 leader_election.cc:290] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:58:39.576926  1962 raft_consensus.cc:2804] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:58:39.583047  1873 raft_consensus.cc:2749] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:58:39.586539  1993 raft_consensus.cc:697] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 12:58:39.590603  1795 raft_consensus.cc:359] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:39.591369  1795 raft_consensus.cc:385] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:39.591887  1795 raft_consensus.cc:740] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Initialized, Role: FOLLOWER
I20260709 12:58:39.591264  1993 consensus_queue.cc:237] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:39.615949  1795 consensus_queue.cc:260] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:39.615114  1962 raft_consensus.cc:697] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 12:58:39.621165  1795 ts_tablet_manager.cc:1434] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701: Time spent starting tablet: real 0.056s	user 0.008s	sys 0.000s
I20260709 12:58:39.622058  1962 consensus_queue.cc:237] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:39.633878  1794 tablet_bootstrap.cc:654] T c6ab3300be1e4a59b64b14d50da193ca P 816fadcea8fc4893882f5f9b9f22a03d: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:39.632622  2022 raft_consensus.cc:515] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:39.642956  1795 tablet_bootstrap.cc:492] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701: Bootstrap starting.
I20260709 12:58:39.645380  1950 raft_consensus.cc:493] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:39.645888  1950 raft_consensus.cc:515] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:39.646574  1796 tablet_bootstrap.cc:492] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a: No bootstrap required, opened a new log
I20260709 12:58:39.646744  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:58:39.647070  1796 ts_tablet_manager.cc:1403] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent bootstrapping tablet: real 0.187s	user 0.016s	sys 0.017s
I20260709 12:58:39.649225  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:58:39.649829  1680 raft_consensus.cc:3060] T 1809d674329a492dbea0b60a4703fe51 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:39.651131  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:58:39.651335  1950 leader_election.cc:290] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:58:39.654103  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:58:39.650087  1796 raft_consensus.cc:359] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 12:58:39.656150  1717 leader_election.cc:343] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:39.656487  1796 raft_consensus.cc:385] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260709 12:58:39.656150  1643 leader_election.cc:343] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:39.656908  1796 raft_consensus.cc:740] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Initialized, Role: FOLLOWER
I20260709 12:58:39.657725  1796 consensus_queue.cc:260] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:39.655768  2022 leader_election.cc:290] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 12:58:39.665843  1646 leader_election.cc:343] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:39.671964  1646 leader_election.cc:304] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 12:58:39.672792  1950 raft_consensus.cc:2749] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:58:39.676286  1796 ts_tablet_manager.cc:1434] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent starting tablet: real 0.029s	user 0.008s	sys 0.000s
I20260709 12:58:39.677505  1796 tablet_bootstrap.cc:492] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a: Bootstrap starting.
I20260709 12:58:39.684000  2034 raft_consensus.cc:493] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:39.687027  2034 raft_consensus.cc:515] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:39.689991  2034 leader_election.cc:290] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:58:39.692124  1795 tablet_bootstrap.cc:654] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:39.696314  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:58:39.696408  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:58:39.698374  1676 raft_consensus.cc:2468] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 0.
W20260709 12:58:39.699455  1571 leader_election.cc:343] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:39.702777  1569 leader_election.cc:304] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 12:58:39.706071  2034 raft_consensus.cc:2804] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:58:39.707161  2034 raft_consensus.cc:493] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:58:39.707594  2034 raft_consensus.cc:3060] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:39.707924  1503 catalog_manager.cc:5697] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d reported cstate change: term changed from 0 to 1, leader changed from <none> to 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194). New cstate: current_term: 1 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } }
I20260709 12:58:39.714623  1794 tablet_bootstrap.cc:492] T c6ab3300be1e4a59b64b14d50da193ca P 816fadcea8fc4893882f5f9b9f22a03d: No bootstrap required, opened a new log
I20260709 12:58:39.714157  1503 catalog_manager.cc:5697] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d reported cstate change: term changed from 0 to 1, leader changed from <none> to 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194). New cstate: current_term: 1 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } }
I20260709 12:58:39.715176  1794 ts_tablet_manager.cc:1403] T c6ab3300be1e4a59b64b14d50da193ca P 816fadcea8fc4893882f5f9b9f22a03d: Time spent bootstrapping tablet: real 0.192s	user 0.015s	sys 0.009s
I20260709 12:58:39.715431  1500 catalog_manager.cc:5697] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 0 to 1, leader changed from <none> to 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193). New cstate: current_term: 1 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } }
I20260709 12:58:39.714917  1680 raft_consensus.cc:2468] T 1809d674329a492dbea0b60a4703fe51 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 1.
I20260709 12:58:39.718534  1794 raft_consensus.cc:359] T c6ab3300be1e4a59b64b14d50da193ca P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:39.720221  1794 raft_consensus.cc:385] T c6ab3300be1e4a59b64b14d50da193ca P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:39.720533  1794 raft_consensus.cc:740] T c6ab3300be1e4a59b64b14d50da193ca P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Initialized, Role: FOLLOWER
I20260709 12:58:39.720863  1718 leader_election.cc:304] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:58:39.721290  1794 consensus_queue.cc:260] T c6ab3300be1e4a59b64b14d50da193ca P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:39.722154  1873 raft_consensus.cc:2804] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:58:39.722714  1873 raft_consensus.cc:697] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 1 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 12:58:39.723578  1873 consensus_queue.cc:237] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:39.725216  2034 raft_consensus.cc:515] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:39.727051  1796 tablet_bootstrap.cc:654] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:39.732100  1795 tablet_bootstrap.cc:492] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701: No bootstrap required, opened a new log
I20260709 12:58:39.732467  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:58:39.732579  1795 ts_tablet_manager.cc:1403] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701: Time spent bootstrapping tablet: real 0.093s	user 0.009s	sys 0.005s
I20260709 12:58:39.732646  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:58:39.733065  1676 raft_consensus.cc:3060] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:58:39.734872  1571 leader_election.cc:343] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:39.736013  1795 raft_consensus.cc:359] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:39.736696  1795 raft_consensus.cc:385] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:39.737020  1795 raft_consensus.cc:740] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Initialized, Role: FOLLOWER
I20260709 12:58:39.736932  2034 leader_election.cc:290] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:58:39.737952  1795 consensus_queue.cc:260] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:39.740788  1795 ts_tablet_manager.cc:1434] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.000s
I20260709 12:58:39.741957  1795 tablet_bootstrap.cc:492] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701: Bootstrap starting.
I20260709 12:58:39.743950  1796 tablet_bootstrap.cc:492] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a: No bootstrap required, opened a new log
I20260709 12:58:39.744426  1796 ts_tablet_manager.cc:1403] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent bootstrapping tablet: real 0.067s	user 0.015s	sys 0.000s
I20260709 12:58:39.747388  1794 ts_tablet_manager.cc:1434] T c6ab3300be1e4a59b64b14d50da193ca P 816fadcea8fc4893882f5f9b9f22a03d: Time spent starting tablet: real 0.032s	user 0.007s	sys 0.000s
I20260709 12:58:39.747931  1676 raft_consensus.cc:2468] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 1.
I20260709 12:58:39.749125  1569 leader_election.cc:304] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 12:58:39.749945  1928 raft_consensus.cc:2804] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:58:39.750466  1928 raft_consensus.cc:697] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 12:58:39.751628  1796 raft_consensus.cc:359] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:39.751279  1928 consensus_queue.cc:237] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:39.752470  1796 raft_consensus.cc:385] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:39.752750  1796 raft_consensus.cc:740] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Initialized, Role: FOLLOWER
I20260709 12:58:39.753436  1796 consensus_queue.cc:260] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:39.757261  1794 tablet_bootstrap.cc:492] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d: Bootstrap starting.
I20260709 12:58:39.757973  1796 ts_tablet_manager.cc:1434] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent starting tablet: real 0.013s	user 0.007s	sys 0.000s
I20260709 12:58:39.759178  1796 tablet_bootstrap.cc:492] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a: Bootstrap starting.
I20260709 12:58:39.766582  1795 tablet_bootstrap.cc:654] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:39.767316  1796 tablet_bootstrap.cc:654] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:39.785049  1794 tablet_bootstrap.cc:654] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:39.787832  1796 tablet_bootstrap.cc:492] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a: No bootstrap required, opened a new log
I20260709 12:58:39.788383  1796 ts_tablet_manager.cc:1403] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent bootstrapping tablet: real 0.029s	user 0.007s	sys 0.009s
I20260709 12:58:39.791464  1795 tablet_bootstrap.cc:492] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701: No bootstrap required, opened a new log
I20260709 12:58:39.791534  1796 raft_consensus.cc:359] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:39.792032  1795 ts_tablet_manager.cc:1403] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701: Time spent bootstrapping tablet: real 0.050s	user 0.008s	sys 0.003s
I20260709 12:58:39.792320  1796 raft_consensus.cc:385] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:39.792626  1796 raft_consensus.cc:740] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Initialized, Role: FOLLOWER
I20260709 12:58:39.794860  1796 consensus_queue.cc:260] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:39.795559  1795 raft_consensus.cc:359] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:39.796530  1795 raft_consensus.cc:385] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:39.797840  1796 ts_tablet_manager.cc:1434] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent starting tablet: real 0.009s	user 0.004s	sys 0.004s
I20260709 12:58:39.799253  1796 tablet_bootstrap.cc:492] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a: Bootstrap starting.
I20260709 12:58:39.797705  1795 raft_consensus.cc:740] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Initialized, Role: FOLLOWER
I20260709 12:58:39.800925  1795 consensus_queue.cc:260] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:39.803289  1795 ts_tablet_manager.cc:1434] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701: Time spent starting tablet: real 0.011s	user 0.007s	sys 0.000s
I20260709 12:58:39.805819  1794 tablet_bootstrap.cc:492] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d: No bootstrap required, opened a new log
I20260709 12:58:39.806315  1794 ts_tablet_manager.cc:1403] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d: Time spent bootstrapping tablet: real 0.049s	user 0.009s	sys 0.003s
W20260709 12:58:39.786096  1551 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:58:39.808224  1502 catalog_manager.cc:5697] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 reported cstate change: term changed from 0 to 1, leader changed from <none> to af520b67945d443b901bbeca37cc5701 (127.0.107.195). New cstate: current_term: 1 leader_uuid: "af520b67945d443b901bbeca37cc5701" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: HEALTHY } } }
I20260709 12:58:39.809106  1794 raft_consensus.cc:359] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:39.809778  1794 raft_consensus.cc:385] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:39.810057  1794 raft_consensus.cc:740] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Initialized, Role: FOLLOWER
I20260709 12:58:39.811239  1795 tablet_bootstrap.cc:492] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701: Bootstrap starting.
I20260709 12:58:39.811066  1794 consensus_queue.cc:260] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:39.813063  1796 tablet_bootstrap.cc:654] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:39.812733  1500 catalog_manager.cc:5697] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 0 to 1, leader changed from <none> to 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193). New cstate: current_term: 1 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } }
I20260709 12:58:39.819023  1795 tablet_bootstrap.cc:654] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:39.820914  1794 ts_tablet_manager.cc:1434] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d: Time spent starting tablet: real 0.014s	user 0.006s	sys 0.000s
I20260709 12:58:39.824007  1950 raft_consensus.cc:493] T c6ab3300be1e4a59b64b14d50da193ca P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:39.825623  1794 tablet_bootstrap.cc:492] T dce8db7565da4e13a82ffbb9c39fde78 P 816fadcea8fc4893882f5f9b9f22a03d: Bootstrap starting.
I20260709 12:58:39.838061  1794 tablet_bootstrap.cc:654] T dce8db7565da4e13a82ffbb9c39fde78 P 816fadcea8fc4893882f5f9b9f22a03d: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:39.825624  1950 raft_consensus.cc:515] T c6ab3300be1e4a59b64b14d50da193ca P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:39.847447  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:58:39.849669  1794 tablet_bootstrap.cc:492] T dce8db7565da4e13a82ffbb9c39fde78 P 816fadcea8fc4893882f5f9b9f22a03d: No bootstrap required, opened a new log
I20260709 12:58:39.849830  1950 leader_election.cc:290] T c6ab3300be1e4a59b64b14d50da193ca P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:58:39.850170  1794 ts_tablet_manager.cc:1403] T dce8db7565da4e13a82ffbb9c39fde78 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent bootstrapping tablet: real 0.025s	user 0.011s	sys 0.000s
W20260709 12:58:39.850692  1646 leader_election.cc:343] T c6ab3300be1e4a59b64b14d50da193ca P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:39.851599  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 12:58:39.852998  1643 leader_election.cc:343] T c6ab3300be1e4a59b64b14d50da193ca P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:39.853149  1794 raft_consensus.cc:359] T dce8db7565da4e13a82ffbb9c39fde78 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:39.853449  1643 leader_election.cc:304] T c6ab3300be1e4a59b64b14d50da193ca P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 12:58:39.853794  1794 raft_consensus.cc:385] T dce8db7565da4e13a82ffbb9c39fde78 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:39.854086  1794 raft_consensus.cc:740] T dce8db7565da4e13a82ffbb9c39fde78 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Initialized, Role: FOLLOWER
I20260709 12:58:39.860019  1794 consensus_queue.cc:260] T dce8db7565da4e13a82ffbb9c39fde78 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:39.862084  1902 raft_consensus.cc:2749] T c6ab3300be1e4a59b64b14d50da193ca P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:58:39.864226  1794 ts_tablet_manager.cc:1434] T dce8db7565da4e13a82ffbb9c39fde78 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent starting tablet: real 0.014s	user 0.007s	sys 0.000s
I20260709 12:58:39.865291  1794 tablet_bootstrap.cc:492] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d: Bootstrap starting.
I20260709 12:58:39.875303  1796 tablet_bootstrap.cc:492] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a: No bootstrap required, opened a new log
I20260709 12:58:39.875818  1796 ts_tablet_manager.cc:1403] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent bootstrapping tablet: real 0.077s	user 0.017s	sys 0.013s
I20260709 12:58:39.876681  1794 tablet_bootstrap.cc:654] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:39.875353  1795 tablet_bootstrap.cc:492] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701: No bootstrap required, opened a new log
I20260709 12:58:39.878393  1795 ts_tablet_manager.cc:1403] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701: Time spent bootstrapping tablet: real 0.067s	user 0.038s	sys 0.018s
I20260709 12:58:39.879125  1796 raft_consensus.cc:359] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:39.879801  1796 raft_consensus.cc:385] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:39.880069  1796 raft_consensus.cc:740] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Initialized, Role: FOLLOWER
I20260709 12:58:39.880695  1796 consensus_queue.cc:260] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:39.882366  1796 ts_tablet_manager.cc:1434] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent starting tablet: real 0.006s	user 0.002s	sys 0.000s
I20260709 12:58:39.883518  1873 raft_consensus.cc:493] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:39.883970  1873 raft_consensus.cc:515] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:39.884276  1795 raft_consensus.cc:359] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:39.884912  1795 raft_consensus.cc:385] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:39.885167  1795 raft_consensus.cc:740] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Initialized, Role: FOLLOWER
I20260709 12:58:39.885847  1795 consensus_queue.cc:260] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:39.887177  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:58:39.888067  1873 leader_election.cc:290] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:58:39.888324  1795 ts_tablet_manager.cc:1434] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701: Time spent starting tablet: real 0.010s	user 0.006s	sys 0.000s
W20260709 12:58:39.888783  1718 leader_election.cc:343] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:39.889003  1796 tablet_bootstrap.cc:492] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a: Bootstrap starting.
I20260709 12:58:39.889719  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:58:39.891077  1795 tablet_bootstrap.cc:492] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701: Bootstrap starting.
W20260709 12:58:39.891289  1717 leader_election.cc:343] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:39.891882  1717 leader_election.cc:304] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:58:39.892699  1873 raft_consensus.cc:2749] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:58:39.897696  1796 tablet_bootstrap.cc:654] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:39.898839  1795 tablet_bootstrap.cc:654] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:39.906993  1928 raft_consensus.cc:493] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:39.907562  1928 raft_consensus.cc:515] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:39.909770  1928 leader_election.cc:290] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:58:39.910302  1796 tablet_bootstrap.cc:492] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a: No bootstrap required, opened a new log
I20260709 12:58:39.910554  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:58:39.910804  1796 ts_tablet_manager.cc:1403] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent bootstrapping tablet: real 0.022s	user 0.004s	sys 0.015s
I20260709 12:58:39.910892  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:58:39.911273  1676 raft_consensus.cc:2468] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 0.
I20260709 12:58:39.912932  1794 tablet_bootstrap.cc:492] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d: No bootstrap required, opened a new log
I20260709 12:58:39.912873  1569 leader_election.cc:304] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 12:58:39.913478  1794 ts_tablet_manager.cc:1403] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent bootstrapping tablet: real 0.048s	user 0.010s	sys 0.006s
W20260709 12:58:39.913542  1571 leader_election.cc:343] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:39.913791  1928 raft_consensus.cc:2804] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:58:39.914269  1928 raft_consensus.cc:493] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:58:39.914693  1928 raft_consensus.cc:3060] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:39.916545  1794 raft_consensus.cc:359] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:39.917287  1794 raft_consensus.cc:385] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:39.914081  1796 raft_consensus.cc:359] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:39.917606  1794 raft_consensus.cc:740] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Initialized, Role: FOLLOWER
I20260709 12:58:39.917804  1796 raft_consensus.cc:385] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:39.917001  1795 tablet_bootstrap.cc:492] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701: No bootstrap required, opened a new log
I20260709 12:58:39.918253  1796 raft_consensus.cc:740] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Initialized, Role: FOLLOWER
I20260709 12:58:39.918293  1794 consensus_queue.cc:260] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:39.919060  1796 consensus_queue.cc:260] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:39.920670  1902 raft_consensus.cc:493] T dce8db7565da4e13a82ffbb9c39fde78 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:39.921166  1794 ts_tablet_manager.cc:1434] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent starting tablet: real 0.007s	user 0.008s	sys 0.000s
I20260709 12:58:39.921378  1796 ts_tablet_manager.cc:1434] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent starting tablet: real 0.010s	user 0.003s	sys 0.002s
I20260709 12:58:39.921133  1902 raft_consensus.cc:515] T dce8db7565da4e13a82ffbb9c39fde78 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:39.922549  1796 tablet_bootstrap.cc:492] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a: Bootstrap starting.
I20260709 12:58:39.923434  1902 leader_election.cc:290] T dce8db7565da4e13a82ffbb9c39fde78 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:58:39.924978  1795 ts_tablet_manager.cc:1403] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701: Time spent bootstrapping tablet: real 0.034s	user 0.012s	sys 0.002s
I20260709 12:58:39.927932  1794 tablet_bootstrap.cc:492] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d: Bootstrap starting.
I20260709 12:58:39.928859  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:58:39.930094  1928 raft_consensus.cc:515] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 12:58:39.930809  1646 leader_election.cc:343] T dce8db7565da4e13a82ffbb9c39fde78 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:39.931881  1751 raft_consensus.cc:3060] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:39.932616  1928 leader_election.cc:290] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:58:39.933267  1795 raft_consensus.cc:359] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:39.933606  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:58:39.933995  1795 raft_consensus.cc:385] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:39.934198  1676 raft_consensus.cc:3060] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:39.934320  1795 raft_consensus.cc:740] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Initialized, Role: FOLLOWER
I20260709 12:58:39.936076  1902 raft_consensus.cc:493] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:39.936228  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:58:39.936565  1902 raft_consensus.cc:515] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 12:58:39.938834  1643 leader_election.cc:343] T dce8db7565da4e13a82ffbb9c39fde78 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:39.939167  1902 leader_election.cc:290] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:58:39.939273  1643 leader_election.cc:304] T dce8db7565da4e13a82ffbb9c39fde78 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 12:58:39.939981  1795 consensus_queue.cc:260] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:39.942260  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:58:39.947161  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 12:58:39.948642  1646 leader_election.cc:343] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:58:39.950088  1643 leader_election.cc:343] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:39.950601  1643 leader_election.cc:304] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 12:58:39.951565  1993 raft_consensus.cc:2749] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:58:39.943425  1902 raft_consensus.cc:2749] T dce8db7565da4e13a82ffbb9c39fde78 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:58:39.956672  1795 ts_tablet_manager.cc:1434] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701: Time spent starting tablet: real 0.031s	user 0.009s	sys 0.001s
I20260709 12:58:39.957728  1795 tablet_bootstrap.cc:492] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701: Bootstrap starting.
I20260709 12:58:39.944238  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:58:39.965224  1796 tablet_bootstrap.cc:654] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a: Neither blocks nor log segments found. Creating new log.
W20260709 12:58:39.966276  1571 leader_election.cc:343] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:39.967693  1979 raft_consensus.cc:493] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:39.968168  1979 raft_consensus.cc:515] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:39.968389  1902 consensus_queue.cc:1048] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:58:39.970327  1979 leader_election.cc:290] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:58:39.972131  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:58:39.972957  1680 raft_consensus.cc:2393] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 1: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 12:58:39.975306  1795 tablet_bootstrap.cc:654] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:39.975592  1676 raft_consensus.cc:2468] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 1.
I20260709 12:58:39.976199  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:58:39.977241  1569 leader_election.cc:304] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 12:58:39.977716  1717 leader_election.cc:304] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:58:39.978451  2034 raft_consensus.cc:2804] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:58:39.978899  2034 raft_consensus.cc:697] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 12:58:39.979189  2022 raft_consensus.cc:3060] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:39.979764  2034 consensus_queue.cc:237] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:39.982604  1794 tablet_bootstrap.cc:654] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:39.986395  2022 raft_consensus.cc:2749] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:58:39.985829  1979 consensus_queue.cc:1048] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260709 12:58:40.001808  1794 tablet_bootstrap.cc:492] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d: No bootstrap required, opened a new log
I20260709 12:58:40.002462  1794 ts_tablet_manager.cc:1403] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent bootstrapping tablet: real 0.075s	user 0.009s	sys 0.005s
I20260709 12:58:40.003782  1796 tablet_bootstrap.cc:492] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a: No bootstrap required, opened a new log
I20260709 12:58:40.004194  1796 ts_tablet_manager.cc:1403] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent bootstrapping tablet: real 0.082s	user 0.016s	sys 0.007s
I20260709 12:58:40.005870  1795 tablet_bootstrap.cc:492] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701: No bootstrap required, opened a new log
I20260709 12:58:40.006405  1795 ts_tablet_manager.cc:1403] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701: Time spent bootstrapping tablet: real 0.049s	user 0.015s	sys 0.001s
I20260709 12:58:40.009378  1979 raft_consensus.cc:493] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:40.009369  1796 raft_consensus.cc:359] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:40.009933  1794 raft_consensus.cc:359] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:40.010041  1979 raft_consensus.cc:515] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:40.010342  1796 raft_consensus.cc:385] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:40.011023  1794 raft_consensus.cc:385] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:40.010980  1796 raft_consensus.cc:740] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Initialized, Role: FOLLOWER
I20260709 12:58:40.011379  1794 raft_consensus.cc:740] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Initialized, Role: FOLLOWER
I20260709 12:58:40.012101  1794 consensus_queue.cc:260] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:40.013962  1794 ts_tablet_manager.cc:1434] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent starting tablet: real 0.011s	user 0.003s	sys 0.000s
I20260709 12:58:40.014084  1979 leader_election.cc:290] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:58:40.006801  1902 consensus_queue.cc:1048] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20260709 12:58:40.015894  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:58:40.016541  1604 raft_consensus.cc:2468] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 0.
I20260709 12:58:40.017400  1795 raft_consensus.cc:359] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:40.018042  1795 raft_consensus.cc:385] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:40.018347  1717 leader_election.cc:304] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:58:40.019295  2022 raft_consensus.cc:2804] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:58:40.019749  2022 raft_consensus.cc:493] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:58:40.020133  2022 raft_consensus.cc:3060] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:40.020741  1794 tablet_bootstrap.cc:492] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d: Bootstrap starting.
I20260709 12:58:40.021066  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:58:40.021759  1680 raft_consensus.cc:2468] T c6ab3300be1e4a59b64b14d50da193ca P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 0.
I20260709 12:58:40.018285  1795 raft_consensus.cc:740] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Initialized, Role: FOLLOWER
I20260709 12:58:40.024351  1503 catalog_manager.cc:5697] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 0 to 1, leader changed from <none> to 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193). New cstate: current_term: 1 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } }
I20260709 12:58:40.025533  1795 consensus_queue.cc:260] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:40.027913  1795 ts_tablet_manager.cc:1434] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701: Time spent starting tablet: real 0.021s	user 0.005s	sys 0.000s
I20260709 12:58:40.029165  1795 tablet_bootstrap.cc:492] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701: Bootstrap starting.
I20260709 12:58:40.032397  1796 consensus_queue.cc:260] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:40.047175  1795 tablet_bootstrap.cc:654] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:40.048548  1794 tablet_bootstrap.cc:654] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:40.054507  1950 raft_consensus.cc:493] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:40.055076  1950 raft_consensus.cc:515] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:40.057271  1950 leader_election.cc:290] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:58:40.058235  1796 ts_tablet_manager.cc:1434] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent starting tablet: real 0.054s	user 0.010s	sys 0.001s
I20260709 12:58:40.068697  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:58:40.069113  1962 raft_consensus.cc:493] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:40.073031  1962 raft_consensus.cc:515] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:40.073565  1796 tablet_bootstrap.cc:492] T 344b4152823f4d618345b14ffd579400 P 83595ad0fd5940c0a2bfc7c61acd969a: Bootstrap starting.
I20260709 12:58:40.075255  1962 leader_election.cc:290] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:58:40.076213  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:58:40.062209  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:58:40.077715  1676 raft_consensus.cc:2393] T 1809d674329a492dbea0b60a4703fe51 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 1: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 12:58:40.080242  1979 consensus_queue.cc:1048] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:58:40.082150  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:58:40.082571  1950 consensus_queue.cc:1048] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260709 12:58:40.090318  1643 leader_election.cc:343] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:40.085852  1985 consensus_queue.cc:1048] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.007s
I20260709 12:58:40.092311  1571 leader_election.cc:304] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 12:58:40.094692  1928 raft_consensus.cc:3060] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:40.085445  1754 raft_consensus.cc:2468] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 0.
I20260709 12:58:40.095626  2022 raft_consensus.cc:515] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:40.099150  2034 consensus_queue.cc:1048] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.008s
I20260709 12:58:40.112072  1962 raft_consensus.cc:493] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:40.112581  1962 raft_consensus.cc:515] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:40.114712  2022 leader_election.cc:290] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:58:40.116415  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:58:40.116668  1962 leader_election.cc:290] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:58:40.116988  1680 raft_consensus.cc:3060] T c6ab3300be1e4a59b64b14d50da193ca P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:40.117980  1795 tablet_bootstrap.cc:492] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701: No bootstrap required, opened a new log
I20260709 12:58:40.118350  1795 ts_tablet_manager.cc:1403] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701: Time spent bootstrapping tablet: real 0.090s	user 0.005s	sys 0.017s
I20260709 12:58:40.125581  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:58:40.126871  1646 leader_election.cc:304] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:58:40.129989  1794 tablet_bootstrap.cc:492] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d: No bootstrap required, opened a new log
I20260709 12:58:40.130589  1794 ts_tablet_manager.cc:1403] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent bootstrapping tablet: real 0.110s	user 0.011s	sys 0.005s
I20260709 12:58:40.120636  1979 raft_consensus.cc:493] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:40.168766  1950 raft_consensus.cc:493] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:58:40.169384  1569 leader_election.cc:343] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:40.170866  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:58:40.173049  1601 raft_consensus.cc:3060] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:40.173115  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:58:40.176659  1962 raft_consensus.cc:493] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:40.195901  1795 raft_consensus.cc:359] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:40.197405  1796 tablet_bootstrap.cc:654] T 344b4152823f4d618345b14ffd579400 P 83595ad0fd5940c0a2bfc7c61acd969a: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:40.198863  1795 raft_consensus.cc:385] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:40.199242  1795 raft_consensus.cc:740] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Initialized, Role: FOLLOWER
I20260709 12:58:40.169384  1950 raft_consensus.cc:515] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:40.194042  1985 raft_consensus.cc:2804] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:58:40.172494  1979 raft_consensus.cc:515] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 12:58:40.222116  1571 leader_election.cc:343] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:40.237488  1571 leader_election.cc:304] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
W20260709 12:58:40.199443  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 73053.7us
I20260709 12:58:40.243494  1795 consensus_queue.cc:260] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:40.211287  1985 raft_consensus.cc:493] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:58:40.244750  1985 raft_consensus.cc:3060] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:40.245286  1795 ts_tablet_manager.cc:1434] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701: Time spent starting tablet: real 0.127s	user 0.009s	sys 0.006s
I20260709 12:58:40.246335  1795 tablet_bootstrap.cc:492] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701: Bootstrap starting.
I20260709 12:58:40.247856  1794 raft_consensus.cc:359] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:40.251631  1794 raft_consensus.cc:385] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:40.251935  1794 raft_consensus.cc:740] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Initialized, Role: FOLLOWER
I20260709 12:58:40.252642  1794 consensus_queue.cc:260] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:40.254187  1794 ts_tablet_manager.cc:1434] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent starting tablet: real 0.123s	user 0.007s	sys 0.009s
I20260709 12:58:40.255192  1794 tablet_bootstrap.cc:492] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d: Bootstrap starting.
I20260709 12:58:40.205700  1962 raft_consensus.cc:515] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:40.242303  1601 raft_consensus.cc:2468] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 1.
I20260709 12:58:40.297581  1680 raft_consensus.cc:2468] T c6ab3300be1e4a59b64b14d50da193ca P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 1.
I20260709 12:58:40.299196  1962 leader_election.cc:290] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:58:40.301143  1950 leader_election.cc:290] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:58:40.301244  1962 raft_consensus.cc:2749] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:58:40.299831  1979 leader_election.cc:290] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:58:40.327114  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:58:40.327973  1676 raft_consensus.cc:2468] T dce8db7565da4e13a82ffbb9c39fde78 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 0.
W20260709 12:58:40.395716  2063 log.cc:927] Time spent T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.086s	user 0.000s	sys 0.000s
W20260709 12:58:40.411304  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 189184us
I20260709 12:58:40.452695  2067 raft_consensus.cc:493] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:40.530037  1928 raft_consensus.cc:2749] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:58:40.581223  1794 tablet_bootstrap.cc:654] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:40.528810  2067 raft_consensus.cc:515] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:40.624253  2069 raft_consensus.cc:493] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:40.625028  2069 raft_consensus.cc:515] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:40.585867  1985 raft_consensus.cc:515] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:40.634958  1985 leader_election.cc:290] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:58:40.639216  2067 leader_election.cc:290] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:58:40.721172  2069 leader_election.cc:290] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:58:40.733203  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:58:40.738464  1754 raft_consensus.cc:3060] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:40.741695  1795 tablet_bootstrap.cc:654] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:40.733250  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:58:40.747876  1796 tablet_bootstrap.cc:492] T 344b4152823f4d618345b14ffd579400 P 83595ad0fd5940c0a2bfc7c61acd969a: No bootstrap required, opened a new log
I20260709 12:58:40.748306  1796 ts_tablet_manager.cc:1403] T 344b4152823f4d618345b14ffd579400 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent bootstrapping tablet: real 0.675s	user 0.021s	sys 0.037s
I20260709 12:58:40.746742  2067 raft_consensus.cc:493] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:40.749872  2067 raft_consensus.cc:515] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:40.759335  1796 raft_consensus.cc:359] T 344b4152823f4d618345b14ffd579400 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:40.760061  1796 raft_consensus.cc:385] T 344b4152823f4d618345b14ffd579400 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:40.760329  1796 raft_consensus.cc:740] T 344b4152823f4d618345b14ffd579400 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Initialized, Role: FOLLOWER
I20260709 12:58:40.761475  2067 leader_election.cc:290] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:58:40.767668  1796 consensus_queue.cc:260] T 344b4152823f4d618345b14ffd579400 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:40.770897  1796 ts_tablet_manager.cc:1434] T 344b4152823f4d618345b14ffd579400 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent starting tablet: real 0.022s	user 0.007s	sys 0.000s
I20260709 12:58:40.774797  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:58:40.786662  1796 tablet_bootstrap.cc:492] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a: Bootstrap starting.
W20260709 12:58:40.786484  1646 leader_election.cc:343] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:40.792402  2073 raft_consensus.cc:493] T 344b4152823f4d618345b14ffd579400 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:58:40.787562  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 220242us
I20260709 12:58:40.793009  2073 raft_consensus.cc:515] T 344b4152823f4d618345b14ffd579400 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:40.806008  2073 leader_election.cc:290] T 344b4152823f4d618345b14ffd579400 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:58:40.815536  2066 consensus_queue.cc:1048] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
W20260709 12:58:40.820699  1718 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.504s	user 0.026s	sys 0.026s
W20260709 12:58:40.821131  1718 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.505s	user 0.026s	sys 0.026s
I20260709 12:58:40.825188  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:58:40.825451  1718 leader_election.cc:304] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:58:40.825958  1749 raft_consensus.cc:2468] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 0.
I20260709 12:58:40.826529  1718 leader_election.cc:304] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:58:40.827522  1646 leader_election.cc:304] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:58:40.829353  1794 tablet_bootstrap.cc:492] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d: No bootstrap required, opened a new log
I20260709 12:58:40.829784  1794 ts_tablet_manager.cc:1403] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent bootstrapping tablet: real 0.575s	user 0.024s	sys 0.021s
I20260709 12:58:40.830490  2080 raft_consensus.cc:2804] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:58:40.830411  1873 raft_consensus.cc:2804] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:58:40.830979  2080 raft_consensus.cc:493] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:58:40.831182  1873 raft_consensus.cc:697] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 1 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 12:58:40.831358  2080 raft_consensus.cc:3060] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:40.832136  1873 consensus_queue.cc:237] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:40.834599  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:58:40.839274  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:58:40.842206  1902 raft_consensus.cc:2804] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:58:40.848687  1754 raft_consensus.cc:2468] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 1.
I20260709 12:58:40.848876  1902 raft_consensus.cc:493] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:58:40.849517  1902 raft_consensus.cc:3060] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:40.851969  1750 raft_consensus.cc:3060] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:40.861398  1796 tablet_bootstrap.cc:654] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:40.885329  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:58:40.909418  2077 consensus_queue.cc:1048] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20260709 12:58:40.913277  1794 raft_consensus.cc:359] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:40.914070  1794 raft_consensus.cc:385] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260709 12:58:40.869735  1569 leader_election.cc:343] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:58:40.916471  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 47454.5us
I20260709 12:58:40.917760  1795 tablet_bootstrap.cc:492] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701: No bootstrap required, opened a new log
I20260709 12:58:40.917873  2034 consensus_queue.cc:1048] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:58:40.918330  1795 ts_tablet_manager.cc:1403] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701: Time spent bootstrapping tablet: real 0.672s	user 0.017s	sys 0.025s
I20260709 12:58:40.914346  1794 raft_consensus.cc:740] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Initialized, Role: FOLLOWER
I20260709 12:58:40.922247  1794 consensus_queue.cc:260] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:40.926604  1646 leader_election.cc:304] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:58:40.930171  2025 raft_consensus.cc:2804] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:58:40.930862  2025 raft_consensus.cc:697] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 12:58:40.931972  2025 consensus_queue.cc:237] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:40.934906  1794 ts_tablet_manager.cc:1434] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent starting tablet: real 0.105s	user 0.009s	sys 0.000s
I20260709 12:58:40.949656  1795 raft_consensus.cc:359] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:40.995198  1795 raft_consensus.cc:385] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:40.995609  1795 raft_consensus.cc:740] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Initialized, Role: FOLLOWER
I20260709 12:58:40.997783  1795 consensus_queue.cc:260] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:40.999455  1794 tablet_bootstrap.cc:492] T 14588fa8cfd6467e989746e326a0ed31 P 816fadcea8fc4893882f5f9b9f22a03d: Bootstrap starting.
I20260709 12:58:40.993423  1993 consensus_queue.cc:1048] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.009s
I20260709 12:58:41.005072  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:58:41.010265  2066 consensus_queue.cc:1048] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:58:41.012208  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:58:41.012923  1795 ts_tablet_manager.cc:1434] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701: Time spent starting tablet: real 0.094s	user 0.006s	sys 0.002s
I20260709 12:58:41.014025  1795 tablet_bootstrap.cc:492] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701: Bootstrap starting.
W20260709 12:58:40.938364  1718 leader_election.cc:343] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:40.974335  2080 raft_consensus.cc:515] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:41.035269  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 12:58:41.033501  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 95166.9us
I20260709 12:58:41.048902  1796 tablet_bootstrap.cc:492] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a: No bootstrap required, opened a new log
I20260709 12:58:41.049427  1796 ts_tablet_manager.cc:1403] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent bootstrapping tablet: real 0.263s	user 0.009s	sys 0.017s
I20260709 12:58:41.040659  1602 raft_consensus.cc:2468] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 0.
I20260709 12:58:41.054757  1796 raft_consensus.cc:359] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:41.055718  1796 raft_consensus.cc:385] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:41.059881  1796 raft_consensus.cc:740] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Initialized, Role: FOLLOWER
I20260709 12:58:41.060708  1796 consensus_queue.cc:260] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:41.068657  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:58:41.073689  1795 tablet_bootstrap.cc:654] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:41.077872  1794 tablet_bootstrap.cc:654] T 14588fa8cfd6467e989746e326a0ed31 P 816fadcea8fc4893882f5f9b9f22a03d: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:41.079635  1796 ts_tablet_manager.cc:1434] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent starting tablet: real 0.030s	user 0.008s	sys 0.001s
I20260709 12:58:41.064083  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:58:41.071346  1601 raft_consensus.cc:2468] T 344b4152823f4d618345b14ffd579400 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 0.
I20260709 12:58:41.102656  1754 raft_consensus.cc:2468] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 0.
I20260709 12:58:41.172892  1796 tablet_bootstrap.cc:492] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a: Bootstrap starting.
I20260709 12:58:41.183597  1500 catalog_manager.cc:5697] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 reported cstate change: term changed from 0 to 1, leader changed from <none> to af520b67945d443b901bbeca37cc5701 (127.0.107.195). New cstate: current_term: 1 leader_uuid: "af520b67945d443b901bbeca37cc5701" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: HEALTHY } } }
W20260709 12:58:41.132872  2087 log.cc:927] Time spent T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.069s	user 0.001s	sys 0.000s
I20260709 12:58:41.126892  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:58:41.171836  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:58:41.278540  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:58:41.279486  1746 raft_consensus.cc:2468] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 0.
I20260709 12:58:41.157852  2080 leader_election.cc:290] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:58:41.290077  1993 raft_consensus.cc:493] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:41.290227  1747 raft_consensus.cc:3060] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:41.290781  1993 raft_consensus.cc:515] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:41.293152  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:58:41.294703  1993 leader_election.cc:290] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:58:41.158318  1902 raft_consensus.cc:515] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 12:58:41.299391  1717 leader_election.cc:343] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:41.299633  1985 consensus_queue.cc:1048] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:58:41.300858  1902 leader_election.cc:290] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:58:41.301270  1677 raft_consensus.cc:2393] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 1: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
W20260709 12:58:41.303296  2088 log.cc:927] Time spent T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.066s	user 0.000s	sys 0.000s
I20260709 12:58:41.271054  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:58:41.335949  1571 leader_election.cc:304] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 12:58:41.292335  1604 raft_consensus.cc:3060] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:58:41.351353  1571 consensus_peers.cc:597] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a -> Peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Couldn't send request to peer af520b67945d443b901bbeca37cc5701. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:58:41.352331  1571 leader_election.cc:304] T 344b4152823f4d618345b14ffd579400 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:58:41.339419  1928 raft_consensus.cc:2804] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:58:41.340191  1674 raft_consensus.cc:3060] T dce8db7565da4e13a82ffbb9c39fde78 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:58:41.358659  1551 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:58:41.334448  2077 consensus_queue.cc:1048] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.017s
I20260709 12:58:41.358162  1928 raft_consensus.cc:493] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:58:41.398706  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 52856.4us
I20260709 12:58:41.372218  1962 raft_consensus.cc:2804] T 344b4152823f4d618345b14ffd579400 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Leader pre-election won for term 1
W20260709 12:58:41.404264  1643 leader_election.cc:343] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:41.405426  1643 leader_election.cc:304] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 12:58:41.390923  1928 raft_consensus.cc:3060] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:58:41.408941  2092 log.cc:927] Time spent T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.176s	user 0.004s	sys 0.000s
I20260709 12:58:41.404024  1962 raft_consensus.cc:493] T 344b4152823f4d618345b14ffd579400 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:58:41.406322  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 59201.5us
I20260709 12:58:41.415598  2079 raft_consensus.cc:2749] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:58:41.416146  1962 raft_consensus.cc:3060] T 344b4152823f4d618345b14ffd579400 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:41.424460  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:58:41.431499  1795 tablet_bootstrap.cc:492] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701: No bootstrap required, opened a new log
I20260709 12:58:41.432193  1795 ts_tablet_manager.cc:1403] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701: Time spent bootstrapping tablet: real 0.418s	user 0.031s	sys 0.018s
I20260709 12:58:41.435000  1717 leader_election.cc:304] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 12:58:41.426141  2097 log.cc:927] Time spent T 72f3cdc0abae4f74aad5e4a15cb02e1b P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.074s	user 0.001s	sys 0.000s
I20260709 12:58:41.423035  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:58:41.427546  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:58:41.453433  2105 raft_consensus.cc:2749] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:58:41.459618  1796 tablet_bootstrap.cc:654] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a: Neither blocks nor log segments found. Creating new log.
W20260709 12:58:41.481807  1643 leader_election.cc:343] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:41.487461  1794 tablet_bootstrap.cc:492] T 14588fa8cfd6467e989746e326a0ed31 P 816fadcea8fc4893882f5f9b9f22a03d: No bootstrap required, opened a new log
I20260709 12:58:41.494148  1794 ts_tablet_manager.cc:1403] T 14588fa8cfd6467e989746e326a0ed31 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent bootstrapping tablet: real 0.495s	user 0.018s	sys 0.036s
I20260709 12:58:41.497937  1502 catalog_manager.cc:5697] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d reported cstate change: term changed from 0 to 1, leader changed from <none> to 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194). New cstate: current_term: 1 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } }
I20260709 12:58:41.502115  1795 raft_consensus.cc:359] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:41.502800  1795 raft_consensus.cc:385] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:41.503049  1795 raft_consensus.cc:740] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Initialized, Role: FOLLOWER
I20260709 12:58:41.504916  1795 consensus_queue.cc:260] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:41.516356  1604 raft_consensus.cc:2468] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 1.
I20260709 12:58:41.531721  1795 ts_tablet_manager.cc:1434] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701: Time spent starting tablet: real 0.099s	user 0.008s	sys 0.002s
I20260709 12:58:41.496500  1717 leader_election.cc:304] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 12:58:41.500698  1928 raft_consensus.cc:515] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:41.544229  1794 raft_consensus.cc:359] T 14588fa8cfd6467e989746e326a0ed31 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:41.545145  1794 raft_consensus.cc:385] T 14588fa8cfd6467e989746e326a0ed31 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:41.541462  1674 raft_consensus.cc:2468] T dce8db7565da4e13a82ffbb9c39fde78 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 1.
I20260709 12:58:41.545498  1794 raft_consensus.cc:740] T 14588fa8cfd6467e989746e326a0ed31 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Initialized, Role: FOLLOWER
I20260709 12:58:41.547127  2077 raft_consensus.cc:2804] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:58:41.549886  2077 raft_consensus.cc:493] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:58:41.550470  2077 raft_consensus.cc:3060] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:41.549155  1794 consensus_queue.cc:260] T 14588fa8cfd6467e989746e326a0ed31 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:41.552408  1795 tablet_bootstrap.cc:492] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701: Bootstrap starting.
W20260709 12:58:41.552845  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 68840.3us
W20260709 12:58:41.554749  1717 leader_election.cc:343] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:41.547127  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:58:41.557224  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:58:41.558089  1745 raft_consensus.cc:3060] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:41.553063  1962 raft_consensus.cc:515] T 344b4152823f4d618345b14ffd579400 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:41.569872  1605 raft_consensus.cc:2393] T 344b4152823f4d618345b14ffd579400 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 1: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 12:58:41.576795  1928 leader_election.cc:290] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 12:58:41.585728  1569 leader_election.cc:343] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:41.594027  1794 ts_tablet_manager.cc:1434] T 14588fa8cfd6467e989746e326a0ed31 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent starting tablet: real 0.099s	user 0.005s	sys 0.004s
I20260709 12:58:41.600538  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:58:41.603812  1794 tablet_bootstrap.cc:492] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d: Bootstrap starting.
I20260709 12:58:41.604985  2103 raft_consensus.cc:493] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:41.605767  2103 raft_consensus.cc:515] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 12:58:41.608817  1646 leader_election.cc:343] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:41.609300  1646 leader_election.cc:304] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 12:58:41.610605  2069 raft_consensus.cc:2749] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:58:41.603111  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:58:41.625960  1928 consensus_queue.cc:1048] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260709 12:58:41.637507  1674 raft_consensus.cc:2393] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 1: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 12:58:41.643896  1571 leader_election.cc:304] T 344b4152823f4d618345b14ffd579400 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 12:58:41.644740  1928 raft_consensus.cc:2749] T 344b4152823f4d618345b14ffd579400 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 12:58:41.634697  1718 leader_election.cc:304] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:58:41.647672  2105 raft_consensus.cc:2804] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:58:41.648726  2105 raft_consensus.cc:697] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 1 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 12:58:41.650347  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:58:41.659310  1747 raft_consensus.cc:3060] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:58:41.657156  2108 log.cc:927] Time spent T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.051s	user 0.001s	sys 0.000s
I20260709 12:58:41.662520  1962 leader_election.cc:290] T 344b4152823f4d618345b14ffd579400 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:58:41.663527  1796 tablet_bootstrap.cc:492] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a: No bootstrap required, opened a new log
I20260709 12:58:41.666369  2022 consensus_queue.cc:1048] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:58:41.667518  1796 ts_tablet_manager.cc:1403] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent bootstrapping tablet: real 0.562s	user 0.025s	sys 0.027s
I20260709 12:58:41.672514  1902 consensus_queue.cc:1048] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:58:41.650220  2105 consensus_queue.cc:237] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:41.685417  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:58:41.686661  1795 tablet_bootstrap.cc:654] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:41.685396  2103 leader_election.cc:290] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:58:41.691112  1794 tablet_bootstrap.cc:654] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:41.693353  2078 consensus_queue.cc:1048] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.015s
I20260709 12:58:41.717320  1796 raft_consensus.cc:359] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:41.747201  1796 raft_consensus.cc:385] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:41.747663  1796 raft_consensus.cc:740] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Initialized, Role: FOLLOWER
I20260709 12:58:41.748656  1796 consensus_queue.cc:260] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:41.741380  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:58:41.757875  2069 raft_consensus.cc:493] T 14588fa8cfd6467e989746e326a0ed31 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:41.758941  2069 raft_consensus.cc:515] T 14588fa8cfd6467e989746e326a0ed31 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:41.762748  2080 consensus_queue.cc:1048] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:58:41.762980  1718 leader_election.cc:304] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:58:41.763947  2069 leader_election.cc:290] T 14588fa8cfd6467e989746e326a0ed31 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:58:41.767213  1796 ts_tablet_manager.cc:1434] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent starting tablet: real 0.099s	user 0.012s	sys 0.002s
I20260709 12:58:41.769234  1796 tablet_bootstrap.cc:492] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a: Bootstrap starting.
I20260709 12:58:41.771384  1745 raft_consensus.cc:2468] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 1.
I20260709 12:58:41.774497  1646 leader_election.cc:304] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:58:41.777765  2078 raft_consensus.cc:2804] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:58:41.783641  2078 raft_consensus.cc:697] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 12:58:41.782900  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:58:41.794809  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 12:58:41.804847  1646 leader_election.cc:343] T 14588fa8cfd6467e989746e326a0ed31 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:58:41.805701  1643 leader_election.cc:343] T 14588fa8cfd6467e989746e326a0ed31 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:41.806252  1643 leader_election.cc:304] T 14588fa8cfd6467e989746e326a0ed31 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 12:58:41.811962  2102 raft_consensus.cc:2749] T 14588fa8cfd6467e989746e326a0ed31 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:58:41.813115  2106 raft_consensus.cc:2749] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:58:41.827009  2075 raft_consensus.cc:493] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:41.824246  2077 raft_consensus.cc:515] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:41.849813  2077 leader_election.cc:290] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:58:41.862183  1753 raft_consensus.cc:2393] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 1: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 12:58:41.827816  2075 raft_consensus.cc:515] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:41.804100  2078 consensus_queue.cc:237] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:41.912863  2075 leader_election.cc:290] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:58:41.884750  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:58:41.919850  1603 raft_consensus.cc:2393] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 1: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
W20260709 12:58:41.920127  2118 log.cc:927] Time spent T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.070s	user 0.001s	sys 0.003s
W20260709 12:58:41.926815  2120 log.cc:927] Time spent T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.122s	user 0.000s	sys 0.001s
I20260709 12:58:41.945390  1796 tablet_bootstrap.cc:654] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:41.958564  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:58:41.964011  1795 tablet_bootstrap.cc:492] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701: No bootstrap required, opened a new log
I20260709 12:58:41.964596  1795 ts_tablet_manager.cc:1403] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701: Time spent bootstrapping tablet: real 0.418s	user 0.009s	sys 0.012s
I20260709 12:58:41.968307  1795 raft_consensus.cc:359] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:41.969074  1795 raft_consensus.cc:385] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:41.969537  1795 raft_consensus.cc:740] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Initialized, Role: FOLLOWER
I20260709 12:58:41.977154  1796 tablet_bootstrap.cc:492] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a: No bootstrap required, opened a new log
I20260709 12:58:41.977830  1796 ts_tablet_manager.cc:1403] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent bootstrapping tablet: real 0.209s	user 0.007s	sys 0.011s
I20260709 12:58:41.981341  1796 raft_consensus.cc:359] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:41.982187  1796 raft_consensus.cc:385] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:41.982580  1796 raft_consensus.cc:740] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Initialized, Role: FOLLOWER
I20260709 12:58:41.983585  1796 consensus_queue.cc:260] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:41.987092  1796 ts_tablet_manager.cc:1434] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent starting tablet: real 0.009s	user 0.004s	sys 0.003s
I20260709 12:58:41.988287  1796 tablet_bootstrap.cc:492] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a: Bootstrap starting.
I20260709 12:58:41.995115  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:58:41.970286  1795 consensus_queue.cc:260] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 12:58:41.997489  1718 leader_election.cc:343] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:41.997659  1571 leader_election.cc:304] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 12:58:41.998126  1718 leader_election.cc:304] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:58:41.999230  2105 raft_consensus.cc:2749] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
W20260709 12:58:41.999656  1646 outbound_call.cc:321] RPC callback for RPC call kudu.master.MasterService.TSHeartbeat -> {remote=127.0.107.254:33643, user_credentials={real_user=slave}} blocked reactor thread for 48082.7us
I20260709 12:58:42.005085  1796 tablet_bootstrap.cc:654] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:42.005254  1502 catalog_manager.cc:5697] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 reported cstate change: term changed from 0 to 1, leader changed from <none> to af520b67945d443b901bbeca37cc5701 (127.0.107.195). New cstate: current_term: 1 leader_uuid: "af520b67945d443b901bbeca37cc5701" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: HEALTHY } } }
I20260709 12:58:42.006415  1795 ts_tablet_manager.cc:1434] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701: Time spent starting tablet: real 0.041s	user 0.008s	sys 0.001s
I20260709 12:58:42.008121  1795 tablet_bootstrap.cc:492] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701: Bootstrap starting.
I20260709 12:58:41.982563  1794 tablet_bootstrap.cc:492] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d: No bootstrap required, opened a new log
I20260709 12:58:42.010390  1794 ts_tablet_manager.cc:1403] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent bootstrapping tablet: real 0.407s	user 0.014s	sys 0.020s
I20260709 12:58:42.014566  1794 raft_consensus.cc:359] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:42.015349  1794 raft_consensus.cc:385] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:42.015653  1794 raft_consensus.cc:740] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Initialized, Role: FOLLOWER
I20260709 12:58:42.016399  1794 consensus_queue.cc:260] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:42.017562  1962 raft_consensus.cc:493] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:42.020213  1962 raft_consensus.cc:515] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:42.023312  1962 leader_election.cc:290] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:58:42.018090  2101 raft_consensus.cc:2749] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 12:58:42.026866  1794 ts_tablet_manager.cc:1434] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent starting tablet: real 0.016s	user 0.007s	sys 0.000s
I20260709 12:58:42.027734  2105 raft_consensus.cc:493] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:42.028393  2105 raft_consensus.cc:515] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 12:58:42.028977  1646 leader_election.cc:343] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:42.029870  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:58:42.032716  2105 leader_election.cc:290] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:58:42.034634  1794 tablet_bootstrap.cc:492] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d: Bootstrap starting.
I20260709 12:58:42.040633  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:58:42.042362  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:58:42.057192  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 12:58:42.060263  1569 leader_election.cc:343] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:58:42.063894  1717 leader_election.cc:343] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:42.074216  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:58:42.075443  1676 raft_consensus.cc:2468] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 0.
W20260709 12:58:42.075740  1571 leader_election.cc:343] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:42.076375  1571 leader_election.cc:304] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 12:58:42.077689  2073 raft_consensus.cc:2749] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:58:42.079746  1795 tablet_bootstrap.cc:654] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:42.085031  1794 tablet_bootstrap.cc:654] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:42.019034  1796 tablet_bootstrap.cc:492] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a: No bootstrap required, opened a new log
I20260709 12:58:42.089812  1796 ts_tablet_manager.cc:1403] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent bootstrapping tablet: real 0.102s	user 0.004s	sys 0.019s
I20260709 12:58:42.097110  1718 leader_election.cc:304] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:58:42.098407  1795 tablet_bootstrap.cc:492] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701: No bootstrap required, opened a new log
I20260709 12:58:42.099014  1795 ts_tablet_manager.cc:1403] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701: Time spent bootstrapping tablet: real 0.091s	user 0.008s	sys 0.014s
I20260709 12:58:42.103117  2105 raft_consensus.cc:2804] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:58:42.103294  1795 raft_consensus.cc:359] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:42.103886  2105 raft_consensus.cc:493] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:58:42.109040  1794 tablet_bootstrap.cc:492] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d: No bootstrap required, opened a new log
I20260709 12:58:42.109541  1794 ts_tablet_manager.cc:1403] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d: Time spent bootstrapping tablet: real 0.075s	user 0.012s	sys 0.005s
I20260709 12:58:42.113667  1794 raft_consensus.cc:359] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:42.105547  2105 raft_consensus.cc:3060] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:42.115633  1796 raft_consensus.cc:359] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:42.116396  1796 raft_consensus.cc:385] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:42.116667  1796 raft_consensus.cc:740] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Initialized, Role: FOLLOWER
I20260709 12:58:42.104130  1795 raft_consensus.cc:385] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:42.118539  1794 raft_consensus.cc:385] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:42.120864  1795 raft_consensus.cc:740] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Initialized, Role: FOLLOWER
I20260709 12:58:42.125126  1795 consensus_queue.cc:260] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:42.125571  2105 raft_consensus.cc:515] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:42.121490  1794 raft_consensus.cc:740] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Initialized, Role: FOLLOWER
W20260709 12:58:42.127974  1643 leader_election.cc:343] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:42.131361  1794 consensus_queue.cc:260] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:42.131938  1643 leader_election.cc:304] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 12:58:42.133358  2102 raft_consensus.cc:2749] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:58:42.134235  1794 ts_tablet_manager.cc:1434] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d: Time spent starting tablet: real 0.024s	user 0.003s	sys 0.003s
I20260709 12:58:42.128970  2105 leader_election.cc:290] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:58:42.130512  1795 ts_tablet_manager.cc:1434] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701: Time spent starting tablet: real 0.031s	user 0.008s	sys 0.000s
I20260709 12:58:42.136395  1795 tablet_bootstrap.cc:492] T c81785db7d654e4288fcaeacf4db2b49 P af520b67945d443b901bbeca37cc5701: Bootstrap starting.
I20260709 12:58:42.141430  1794 tablet_bootstrap.cc:492] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d: Bootstrap starting.
I20260709 12:58:42.145123  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
W20260709 12:58:42.147449  1717 leader_election.cc:343] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:42.150730  1796 consensus_queue.cc:260] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:42.152662  1796 ts_tablet_manager.cc:1434] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent starting tablet: real 0.062s	user 0.005s	sys 0.000s
I20260709 12:58:42.157943  1794 tablet_bootstrap.cc:654] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:42.152709  1795 tablet_bootstrap.cc:654] T c81785db7d654e4288fcaeacf4db2b49 P af520b67945d443b901bbeca37cc5701: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:42.163534  1796 tablet_bootstrap.cc:492] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a: Bootstrap starting.
I20260709 12:58:42.157658  1503 catalog_manager.cc:5697] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d reported cstate change: term changed from 0 to 1, leader changed from <none> to 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194). New cstate: current_term: 1 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } }
I20260709 12:58:42.186620  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:58:42.187435  1674 raft_consensus.cc:3060] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:42.200384  1674 raft_consensus.cc:2468] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 1.
I20260709 12:58:42.202896  2073 raft_consensus.cc:493] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:42.203811  2073 raft_consensus.cc:515] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:42.215624  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:58:42.217216  1747 raft_consensus.cc:2468] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 0.
I20260709 12:58:42.220163  1571 leader_election.cc:304] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:58:42.221103  1962 raft_consensus.cc:2804] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:58:42.221721  1962 raft_consensus.cc:493] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:58:42.222213  1962 raft_consensus.cc:3060] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:42.223634  2073 leader_election.cc:290] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:58:42.224555  1718 leader_election.cc:304] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:58:42.225533  1795 tablet_bootstrap.cc:492] T c81785db7d654e4288fcaeacf4db2b49 P af520b67945d443b901bbeca37cc5701: No bootstrap required, opened a new log
I20260709 12:58:42.226296  1795 ts_tablet_manager.cc:1403] T c81785db7d654e4288fcaeacf4db2b49 P af520b67945d443b901bbeca37cc5701: Time spent bootstrapping tablet: real 0.090s	user 0.012s	sys 0.006s
I20260709 12:58:42.227430  2106 raft_consensus.cc:2804] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:58:42.228194  2106 raft_consensus.cc:697] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 1 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 12:58:42.229758  2106 consensus_queue.cc:237] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:42.233675  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:58:42.234947  1674 raft_consensus.cc:2468] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 0.
I20260709 12:58:42.238129  1962 raft_consensus.cc:515] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:42.249014  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:58:42.249765  1676 raft_consensus.cc:3060] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:42.257584  1962 leader_election.cc:290] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:58:42.259320  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:58:42.260313  1747 raft_consensus.cc:3060] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:42.267360  1795 raft_consensus.cc:359] T c81785db7d654e4288fcaeacf4db2b49 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:42.268066  1795 raft_consensus.cc:385] T c81785db7d654e4288fcaeacf4db2b49 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:42.270004  1795 raft_consensus.cc:740] T c81785db7d654e4288fcaeacf4db2b49 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Initialized, Role: FOLLOWER
I20260709 12:58:42.270886  1795 consensus_queue.cc:260] T c81785db7d654e4288fcaeacf4db2b49 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:42.276566  2075 raft_consensus.cc:493] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:42.276858  1747 raft_consensus.cc:2468] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 1.
I20260709 12:58:42.285223  1571 leader_election.cc:304] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:58:42.277364  2075 raft_consensus.cc:515] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:42.287734  2073 raft_consensus.cc:2804] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:58:42.277799  1676 raft_consensus.cc:2468] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 1.
I20260709 12:58:42.290346  2073 raft_consensus.cc:697] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 12:58:42.292059  2073 consensus_queue.cc:237] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:42.296996  2075 leader_election.cc:290] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:58:42.298676  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:58:42.300665  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 12:58:42.300907  1646 leader_election.cc:343] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:42.301584  1599 raft_consensus.cc:2468] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 0.
I20260709 12:58:42.305544  1643 leader_election.cc:304] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 12:58:42.306978  2102 raft_consensus.cc:2804] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:58:42.307403  1794 tablet_bootstrap.cc:492] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d: No bootstrap required, opened a new log
W20260709 12:58:42.307535  1717 consensus_peers.cc:597] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:58:42.308005  1794 ts_tablet_manager.cc:1403] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d: Time spent bootstrapping tablet: real 0.167s	user 0.028s	sys 0.060s
I20260709 12:58:42.307518  2102 raft_consensus.cc:493] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:58:42.310146  2102 raft_consensus.cc:3060] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:42.316239  1794 raft_consensus.cc:359] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:42.317260  1794 raft_consensus.cc:385] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:42.317605  1794 raft_consensus.cc:740] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Initialized, Role: FOLLOWER
I20260709 12:58:42.318496  1794 consensus_queue.cc:260] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:42.321698  1794 ts_tablet_manager.cc:1434] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d: Time spent starting tablet: real 0.013s	user 0.011s	sys 0.000s
I20260709 12:58:42.323807  1796 tablet_bootstrap.cc:654] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:42.323194  1503 catalog_manager.cc:5697] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 reported cstate change: term changed from 0 to 1, leader changed from <none> to af520b67945d443b901bbeca37cc5701 (127.0.107.195). New cstate: current_term: 1 leader_uuid: "af520b67945d443b901bbeca37cc5701" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: HEALTHY } } }
I20260709 12:58:42.326951  1794 tablet_bootstrap.cc:492] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 816fadcea8fc4893882f5f9b9f22a03d: Bootstrap starting.
I20260709 12:58:42.348994  1500 catalog_manager.cc:5697] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 0 to 1, leader changed from <none> to 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193). New cstate: current_term: 1 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } }
I20260709 12:58:42.350926  2106 consensus_queue.cc:1048] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:58:42.352360  1794 tablet_bootstrap.cc:654] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 816fadcea8fc4893882f5f9b9f22a03d: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:42.357110  1795 ts_tablet_manager.cc:1434] T c81785db7d654e4288fcaeacf4db2b49 P af520b67945d443b901bbeca37cc5701: Time spent starting tablet: real 0.130s	user 0.010s	sys 0.000s
I20260709 12:58:42.372256  1962 raft_consensus.cc:493] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:42.373234  1962 raft_consensus.cc:515] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:42.377693  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:58:42.378763  1674 raft_consensus.cc:2468] T 14588fa8cfd6467e989746e326a0ed31 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 0.
I20260709 12:58:42.380677  1569 leader_election.cc:304] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 12:58:42.382144  1928 raft_consensus.cc:2804] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:58:42.382839  1928 raft_consensus.cc:493] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:58:42.383322  1928 raft_consensus.cc:3060] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:42.390017  2078 consensus_queue.cc:1048] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:58:42.402022  1795 tablet_bootstrap.cc:492] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701: Bootstrap starting.
W20260709 12:58:42.444640  1551 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:58:42.444839  1962 leader_election.cc:290] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:58:42.445698  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:58:42.446683  1794 tablet_bootstrap.cc:492] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 816fadcea8fc4893882f5f9b9f22a03d: No bootstrap required, opened a new log
W20260709 12:58:42.447082  1571 leader_election.cc:343] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:42.447189  1794 ts_tablet_manager.cc:1403] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent bootstrapping tablet: real 0.121s	user 0.016s	sys 0.013s
I20260709 12:58:42.452172  1794 raft_consensus.cc:359] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:42.453016  1794 raft_consensus.cc:385] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:42.453369  1794 raft_consensus.cc:740] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Initialized, Role: FOLLOWER
I20260709 12:58:42.454296  1794 consensus_queue.cc:260] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:42.458509  2106 raft_consensus.cc:493] T c81785db7d654e4288fcaeacf4db2b49 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:42.460745  1962 raft_consensus.cc:493] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:42.461346  1962 raft_consensus.cc:515] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:42.463865  1962 leader_election.cc:290] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:58:42.468297  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:58:42.471014  1794 ts_tablet_manager.cc:1434] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent starting tablet: real 0.023s	user 0.008s	sys 0.000s
I20260709 12:58:42.471416  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:58:42.472082  1674 raft_consensus.cc:2468] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 0.
I20260709 12:58:42.472258  1794 tablet_bootstrap.cc:492] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d: Bootstrap starting.
I20260709 12:58:42.474071  1569 leader_election.cc:304] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 12:58:42.459529  2106 raft_consensus.cc:515] T c81785db7d654e4288fcaeacf4db2b49 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:42.477031  1747 raft_consensus.cc:2468] T c81785db7d654e4288fcaeacf4db2b49 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 0.
I20260709 12:58:42.477310  2066 raft_consensus.cc:2804] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:58:42.477788  2066 raft_consensus.cc:493] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:58:42.478168  2066 raft_consensus.cc:3060] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:42.451560  2102 raft_consensus.cc:515] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:42.486625  2102 leader_election.cc:290] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:58:42.490850  2078 consensus_queue.cc:1048] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:58:42.498348  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:58:42.499441  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:58:42.509927  1928 raft_consensus.cc:515] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:42.516571  2078 raft_consensus.cc:493] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:42.517284  2078 raft_consensus.cc:515] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:42.557068  2106 leader_election.cc:290] T c81785db7d654e4288fcaeacf4db2b49 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 12:58:42.575290  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 95900.8us
I20260709 12:58:42.578688  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:58:42.579120  1796 tablet_bootstrap.cc:492] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a: No bootstrap required, opened a new log
I20260709 12:58:42.579430  1676 raft_consensus.cc:3060] T 14588fa8cfd6467e989746e326a0ed31 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:42.579672  1796 ts_tablet_manager.cc:1403] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent bootstrapping tablet: real 0.416s	user 0.008s	sys 0.016s
I20260709 12:58:42.581081  1794 tablet_bootstrap.cc:654] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:42.581081  1795 tablet_bootstrap.cc:654] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701: Neither blocks nor log segments found. Creating new log.
W20260709 12:58:42.586325  1646 leader_election.cc:343] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:42.589362  2078 leader_election.cc:290] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:58:42.590067  1928 leader_election.cc:290] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:58:42.592123  1796 raft_consensus.cc:359] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:42.592722  1796 raft_consensus.cc:385] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:42.592954  1796 raft_consensus.cc:740] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Initialized, Role: FOLLOWER
I20260709 12:58:42.593531  1796 consensus_queue.cc:260] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:42.601401  1676 raft_consensus.cc:2468] T 14588fa8cfd6467e989746e326a0ed31 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 1.
I20260709 12:58:42.602418  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:58:42.603652  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:58:42.604234  1599 raft_consensus.cc:3060] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:42.604367  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 12:58:42.610838  1643 leader_election.cc:343] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:42.596071  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:58:42.639448  1569 leader_election.cc:304] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 12:58:42.657063  1962 raft_consensus.cc:2804] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:58:42.657793  1962 raft_consensus.cc:697] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 12:58:42.642555  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:58:42.661118  1679 raft_consensus.cc:2468] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 0.
I20260709 12:58:42.665912  1796 ts_tablet_manager.cc:1434] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent starting tablet: real 0.086s	user 0.005s	sys 0.002s
I20260709 12:58:42.656492  2066 raft_consensus.cc:515] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:42.667671  1962 consensus_queue.cc:237] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:42.671007  1600 raft_consensus.cc:2393] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 1: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 12:58:42.679189  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
W20260709 12:58:42.673316  1646 leader_election.cc:343] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:42.675031  1718 leader_election.cc:304] T c81785db7d654e4288fcaeacf4db2b49 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:58:42.679867  1679 raft_consensus.cc:3060] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:42.679991  1646 leader_election.cc:304] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 12:58:42.680575  2105 raft_consensus.cc:2804] T c81785db7d654e4288fcaeacf4db2b49 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:58:42.681082  2105 raft_consensus.cc:493] T c81785db7d654e4288fcaeacf4db2b49 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:58:42.681422  2105 raft_consensus.cc:3060] T c81785db7d654e4288fcaeacf4db2b49 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:42.681672  2102 raft_consensus.cc:2749] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260709 12:58:42.678984  2139 log.cc:927] Time spent T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.097s	user 0.000s	sys 0.001s
I20260709 12:58:42.699684  1796 tablet_bootstrap.cc:492] T 86abe5695d5643f888190301b88dacef P 83595ad0fd5940c0a2bfc7c61acd969a: Bootstrap starting.
I20260709 12:58:42.708690  1794 tablet_bootstrap.cc:492] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d: No bootstrap required, opened a new log
I20260709 12:58:42.709390  1794 ts_tablet_manager.cc:1403] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent bootstrapping tablet: real 0.237s	user 0.014s	sys 0.001s
I20260709 12:58:42.712291  1794 raft_consensus.cc:359] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:42.731593  1796 tablet_bootstrap.cc:654] T 86abe5695d5643f888190301b88dacef P 83595ad0fd5940c0a2bfc7c61acd969a: Neither blocks nor log segments found. Creating new log.
W20260709 12:58:42.715456  1571 leader_election.cc:343] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:42.731734  1679 raft_consensus.cc:2468] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 1.
I20260709 12:58:42.725435  2066 leader_election.cc:290] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:58:42.728716  1599 raft_consensus.cc:2468] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 1.
I20260709 12:58:42.730484  2105 raft_consensus.cc:515] T c81785db7d654e4288fcaeacf4db2b49 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:42.740147  2105 leader_election.cc:290] T c81785db7d654e4288fcaeacf4db2b49 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:58:42.733282  1795 tablet_bootstrap.cc:492] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701: No bootstrap required, opened a new log
I20260709 12:58:42.741245  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:58:42.744164  1795 ts_tablet_manager.cc:1403] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701: Time spent bootstrapping tablet: real 0.345s	user 0.018s	sys 0.005s
I20260709 12:58:42.744637  1679 raft_consensus.cc:2393] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 1: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 12:58:42.747215  1795 raft_consensus.cc:359] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:42.747921  1795 raft_consensus.cc:385] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:42.748163  1795 raft_consensus.cc:740] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Initialized, Role: FOLLOWER
I20260709 12:58:42.748860  1795 consensus_queue.cc:260] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:42.749650  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:58:42.744170  1796 tablet_bootstrap.cc:492] T 86abe5695d5643f888190301b88dacef P 83595ad0fd5940c0a2bfc7c61acd969a: No bootstrap required, opened a new log
I20260709 12:58:42.750466  1747 raft_consensus.cc:2393] T c81785db7d654e4288fcaeacf4db2b49 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 1: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 12:58:42.750962  1796 ts_tablet_manager.cc:1403] T 86abe5695d5643f888190301b88dacef P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent bootstrapping tablet: real 0.052s	user 0.013s	sys 0.000s
I20260709 12:58:42.748312  1794 raft_consensus.cc:385] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:42.752122  1794 raft_consensus.cc:740] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Initialized, Role: FOLLOWER
I20260709 12:58:42.752810  1794 consensus_queue.cc:260] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:42.753885  1796 raft_consensus.cc:359] T 86abe5695d5643f888190301b88dacef P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:42.755203  1794 ts_tablet_manager.cc:1434] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent starting tablet: real 0.045s	user 0.006s	sys 0.000s
I20260709 12:58:42.755854  1795 ts_tablet_manager.cc:1434] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701: Time spent starting tablet: real 0.011s	user 0.006s	sys 0.000s
I20260709 12:58:42.756354  1794 tablet_bootstrap.cc:492] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d: Bootstrap starting.
I20260709 12:58:42.756898  1795 tablet_bootstrap.cc:492] T 2ebc67196b4f4119981cdb499e3350cc P af520b67945d443b901bbeca37cc5701: Bootstrap starting.
I20260709 12:58:42.754572  1796 raft_consensus.cc:385] T 86abe5695d5643f888190301b88dacef P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:42.758726  1796 raft_consensus.cc:740] T 86abe5695d5643f888190301b88dacef P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Initialized, Role: FOLLOWER
W20260709 12:58:42.754706  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 151740us
I20260709 12:58:42.760571  1569 leader_election.cc:304] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 12:58:42.761508  1962 raft_consensus.cc:2804] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:58:42.762029  1962 raft_consensus.cc:697] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 12:58:42.762502  1643 leader_election.cc:304] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 12:58:42.763263  2025 raft_consensus.cc:2804] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:58:42.763048  1962 consensus_queue.cc:237] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:42.763783  2025 raft_consensus.cc:697] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 1 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 12:58:42.764731  2025 consensus_queue.cc:237] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:42.759670  1796 consensus_queue.cc:260] T 86abe5695d5643f888190301b88dacef P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:42.768167  1796 ts_tablet_manager.cc:1434] T 86abe5695d5643f888190301b88dacef P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent starting tablet: real 0.017s	user 0.005s	sys 0.001s
I20260709 12:58:42.769230  1796 tablet_bootstrap.cc:492] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a: Bootstrap starting.
I20260709 12:58:42.771157  2077 consensus_queue.cc:1048] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:58:42.775909  2066 raft_consensus.cc:493] T 86abe5695d5643f888190301b88dacef P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:42.766397  1795 tablet_bootstrap.cc:654] T 2ebc67196b4f4119981cdb499e3350cc P af520b67945d443b901bbeca37cc5701: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:42.776434  2066 raft_consensus.cc:515] T 86abe5695d5643f888190301b88dacef P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:42.780725  1796 tablet_bootstrap.cc:654] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:42.784682  2066 leader_election.cc:290] T 86abe5695d5643f888190301b88dacef P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:58:42.788111  2102 raft_consensus.cc:493] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:42.788633  2102 raft_consensus.cc:515] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:42.789537  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:58:42.792121  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:58:42.792979  1747 raft_consensus.cc:2393] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 1: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 12:58:42.793922  2102 leader_election.cc:290] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:58:42.796428  1796 tablet_bootstrap.cc:492] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a: No bootstrap required, opened a new log
I20260709 12:58:42.796957  1796 ts_tablet_manager.cc:1403] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent bootstrapping tablet: real 0.028s	user 0.004s	sys 0.007s
I20260709 12:58:42.797109  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:58:42.800968  1795 tablet_bootstrap.cc:492] T 2ebc67196b4f4119981cdb499e3350cc P af520b67945d443b901bbeca37cc5701: No bootstrap required, opened a new log
I20260709 12:58:42.800813  1796 raft_consensus.cc:359] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:42.801486  1795 ts_tablet_manager.cc:1403] T 2ebc67196b4f4119981cdb499e3350cc P af520b67945d443b901bbeca37cc5701: Time spent bootstrapping tablet: real 0.045s	user 0.010s	sys 0.003s
I20260709 12:58:42.801519  1796 raft_consensus.cc:385] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:42.801999  1796 raft_consensus.cc:740] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Initialized, Role: FOLLOWER
I20260709 12:58:42.802736  1796 consensus_queue.cc:260] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:42.804988  1796 ts_tablet_manager.cc:1434] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.000s
I20260709 12:58:42.805312  1795 raft_consensus.cc:359] T 2ebc67196b4f4119981cdb499e3350cc P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:42.805955  1796 tablet_bootstrap.cc:492] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a: Bootstrap starting.
I20260709 12:58:42.805946  1795 raft_consensus.cc:385] T 2ebc67196b4f4119981cdb499e3350cc P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:42.806384  1795 raft_consensus.cc:740] T 2ebc67196b4f4119981cdb499e3350cc P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Initialized, Role: FOLLOWER
I20260709 12:58:42.807156  1795 consensus_queue.cc:260] T 2ebc67196b4f4119981cdb499e3350cc P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:42.809814  1795 ts_tablet_manager.cc:1434] T 2ebc67196b4f4119981cdb499e3350cc P af520b67945d443b901bbeca37cc5701: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.001s
I20260709 12:58:42.813016  1796 tablet_bootstrap.cc:654] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:42.813071  1599 raft_consensus.cc:3060] T 86abe5695d5643f888190301b88dacef P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:42.821585  2066 consensus_queue.cc:1048] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:58:42.824435  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:58:42.827118  1796 tablet_bootstrap.cc:492] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a: No bootstrap required, opened a new log
I20260709 12:58:42.827626  1796 ts_tablet_manager.cc:1403] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent bootstrapping tablet: real 0.022s	user 0.008s	sys 0.011s
I20260709 12:58:42.828040  1571 leader_election.cc:304] T 86abe5695d5643f888190301b88dacef P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 12:58:42.828953  1928 raft_consensus.cc:2749] T 86abe5695d5643f888190301b88dacef P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:58:42.830340  1796 raft_consensus.cc:359] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:42.831216  1796 raft_consensus.cc:385] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:42.831480  1796 raft_consensus.cc:740] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Initialized, Role: FOLLOWER
I20260709 12:58:42.832008  2105 consensus_queue.cc:1048] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:58:42.832373  1796 consensus_queue.cc:260] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:42.834686  1795 tablet_bootstrap.cc:492] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701: Bootstrap starting.
I20260709 12:58:42.834723  1796 ts_tablet_manager.cc:1434] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent starting tablet: real 0.007s	user 0.008s	sys 0.000s
I20260709 12:58:42.836074  1796 tablet_bootstrap.cc:492] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a: Bootstrap starting.
I20260709 12:58:42.828253  1600 raft_consensus.cc:3060] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:42.846737  1796 tablet_bootstrap.cc:654] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:42.846930  1717 leader_election.cc:304] T c81785db7d654e4288fcaeacf4db2b49 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:58:42.853449  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:58:42.855079  1598 raft_consensus.cc:2393] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 1: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 12:58:42.850749  2142 raft_consensus.cc:2749] T c81785db7d654e4288fcaeacf4db2b49 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 12:58:42.859982  1795 tablet_bootstrap.cc:654] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:42.862049  2066 consensus_queue.cc:1048] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:58:42.863842  1794 tablet_bootstrap.cc:654] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:42.865942  1643 leader_election.cc:304] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 12:58:42.872623  2102 raft_consensus.cc:3060] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:42.905007  1500 catalog_manager.cc:5697] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d reported cstate change: term changed from 0 to 1, leader changed from <none> to 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194). New cstate: current_term: 1 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } }
I20260709 12:58:42.918608  2105 raft_consensus.cc:493] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:42.919401  2105 raft_consensus.cc:515] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:42.920987  1928 raft_consensus.cc:493] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:42.947671  1928 raft_consensus.cc:515] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:42.974067  1928 leader_election.cc:290] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:58:42.979820  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:58:42.980630  1599 raft_consensus.cc:2468] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 0.
I20260709 12:58:42.980769  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:58:42.983652  2102 raft_consensus.cc:2749] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:58:42.984380  1674 raft_consensus.cc:2468] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 0.
I20260709 12:58:42.980333  1796 tablet_bootstrap.cc:492] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a: No bootstrap required, opened a new log
I20260709 12:58:42.987017  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:58:42.987887  2105 leader_election.cc:290] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:58:42.988135  1752 raft_consensus.cc:2468] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 0.
I20260709 12:58:42.990209  1796 ts_tablet_manager.cc:1403] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent bootstrapping tablet: real 0.154s	user 0.019s	sys 0.024s
I20260709 12:58:42.995579  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:58:42.996012  1569 leader_election.cc:304] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 12:58:42.996438  1674 raft_consensus.cc:2468] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 0.
I20260709 12:58:43.001662  1718 leader_election.cc:304] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:58:42.997426  2066 raft_consensus.cc:2804] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:58:43.003001  2066 raft_consensus.cc:493] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:58:43.003487  2066 raft_consensus.cc:3060] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:43.007689  2106 raft_consensus.cc:2804] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:58:43.008190  2106 raft_consensus.cc:493] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:58:43.008734  2106 raft_consensus.cc:3060] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:43.020839  2077 raft_consensus.cc:493] T 2ebc67196b4f4119981cdb499e3350cc P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:43.021492  2077 raft_consensus.cc:515] T 2ebc67196b4f4119981cdb499e3350cc P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:43.024590  2077 leader_election.cc:290] T 2ebc67196b4f4119981cdb499e3350cc P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:58:43.026358  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:58:43.018312  1796 raft_consensus.cc:359] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:43.030099  1796 raft_consensus.cc:385] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:43.030448  1796 raft_consensus.cc:740] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Initialized, Role: FOLLOWER
I20260709 12:58:43.032958  1796 consensus_queue.cc:260] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:43.038323  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:58:43.039255  1605 raft_consensus.cc:2393] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 1: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 12:58:43.040999  1796 ts_tablet_manager.cc:1434] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent starting tablet: real 0.045s	user 0.008s	sys 0.000s
I20260709 12:58:43.042578  1796 tablet_bootstrap.cc:492] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a: Bootstrap starting.
I20260709 12:58:43.052768  1717 leader_election.cc:304] T 2ebc67196b4f4119981cdb499e3350cc P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:58:43.054612  2077 raft_consensus.cc:3060] T 2ebc67196b4f4119981cdb499e3350cc P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:43.092878  1794 tablet_bootstrap.cc:492] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d: No bootstrap required, opened a new log
I20260709 12:58:43.093427  1794 ts_tablet_manager.cc:1403] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent bootstrapping tablet: real 0.337s	user 0.011s	sys 0.014s
I20260709 12:58:43.109333  1928 raft_consensus.cc:493] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:43.109941  1928 raft_consensus.cc:515] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:43.114495  1928 leader_election.cc:290] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:58:43.114665  2106 raft_consensus.cc:515] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:43.115190  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 12:58:43.117126  1569 leader_election.cc:343] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:43.117450  2077 raft_consensus.cc:2749] T 2ebc67196b4f4119981cdb499e3350cc P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260709 12:58:43.118268  1571 consensus_peers.cc:597] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a -> Peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Couldn't send request to peer af520b67945d443b901bbeca37cc5701. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:58:43.108558  2066 raft_consensus.cc:515] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:43.120085  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:58:43.121542  1598 raft_consensus.cc:2393] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 1: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 12:58:43.122049  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:58:43.122545  2066 leader_election.cc:290] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 12:58:43.124106  1571 leader_election.cc:343] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:43.124677  1571 leader_election.cc:304] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 12:58:43.119963  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:58:43.125929  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:58:43.126103  2066 raft_consensus.cc:2749] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:58:43.126353  1674 raft_consensus.cc:3060] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:43.124316  1795 tablet_bootstrap.cc:492] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701: No bootstrap required, opened a new log
I20260709 12:58:43.127130  1747 raft_consensus.cc:2393] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 1: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 12:58:43.127692  1795 ts_tablet_manager.cc:1403] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701: Time spent bootstrapping tablet: real 0.293s	user 0.054s	sys 0.084s
I20260709 12:58:43.118373  2106 leader_election.cc:290] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:58:43.125946  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:58:43.130753  1569 leader_election.cc:304] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 12:58:43.131316  1794 raft_consensus.cc:359] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:43.132225  1794 raft_consensus.cc:385] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:43.131933  2066 raft_consensus.cc:2749] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 12:58:43.132644  1794 raft_consensus.cc:740] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Initialized, Role: FOLLOWER
I20260709 12:58:43.135643  1796 tablet_bootstrap.cc:654] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:43.140656  2025 raft_consensus.cc:493] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:43.141338  2025 raft_consensus.cc:515] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:43.142083  1795 raft_consensus.cc:359] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:43.142982  1795 raft_consensus.cc:385] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:43.143321  1795 raft_consensus.cc:740] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Initialized, Role: FOLLOWER
I20260709 12:58:43.145249  1794 consensus_queue.cc:260] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:43.147261  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:58:43.150030  1794 ts_tablet_manager.cc:1434] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent starting tablet: real 0.056s	user 0.007s	sys 0.002s
I20260709 12:58:43.151505  1794 tablet_bootstrap.cc:492] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d: Bootstrap starting.
W20260709 12:58:43.153236  1643 leader_election.cc:343] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:43.160391  1795 consensus_queue.cc:260] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:43.161072  2025 leader_election.cc:290] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:58:43.161816  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 12:58:43.172804  1646 leader_election.cc:343] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:43.173076  1795 ts_tablet_manager.cc:1434] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701: Time spent starting tablet: real 0.045s	user 0.011s	sys 0.006s
I20260709 12:58:43.173203  1646 leader_election.cc:304] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 12:58:43.172922  1674 raft_consensus.cc:2468] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 1.
I20260709 12:58:43.174230  2075 raft_consensus.cc:2749] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:58:43.175639  1718 leader_election.cc:304] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:58:43.176868  2106 raft_consensus.cc:2804] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:58:43.177666  2106 raft_consensus.cc:697] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 1 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 12:58:43.179737  2106 consensus_queue.cc:237] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:43.180224  1795 tablet_bootstrap.cc:492] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701: Bootstrap starting.
I20260709 12:58:43.208166  1928 consensus_queue.cc:1048] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:58:43.219157  1503 catalog_manager.cc:5697] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 0 to 1, leader changed from <none> to 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193). New cstate: current_term: 1 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } }
I20260709 12:58:43.221236  1503 catalog_manager.cc:5697] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 0 to 1, leader changed from <none> to 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193). New cstate: current_term: 1 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } }
I20260709 12:58:43.226502  1794 tablet_bootstrap.cc:654] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:43.228305  1796 tablet_bootstrap.cc:492] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a: No bootstrap required, opened a new log
I20260709 12:58:43.237735  1796 ts_tablet_manager.cc:1403] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent bootstrapping tablet: real 0.196s	user 0.031s	sys 0.013s
I20260709 12:58:43.240906  2075 consensus_queue.cc:1048] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:58:43.246379  1796 raft_consensus.cc:359] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:43.247298  1796 raft_consensus.cc:385] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:43.249632  1796 raft_consensus.cc:740] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Initialized, Role: FOLLOWER
I20260709 12:58:43.253700  1795 tablet_bootstrap.cc:654] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:43.256388  2073 consensus_queue.cc:1048] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:58:43.282318  1795 tablet_bootstrap.cc:492] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701: No bootstrap required, opened a new log
I20260709 12:58:43.282972  1795 ts_tablet_manager.cc:1403] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701: Time spent bootstrapping tablet: real 0.103s	user 0.024s	sys 0.009s
I20260709 12:58:43.287483  1795 raft_consensus.cc:359] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:43.288407  1795 raft_consensus.cc:385] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:43.288759  1795 raft_consensus.cc:740] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Initialized, Role: FOLLOWER
I20260709 12:58:43.289913  1795 consensus_queue.cc:260] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:43.294070  1795 ts_tablet_manager.cc:1434] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701: Time spent starting tablet: real 0.011s	user 0.010s	sys 0.000s
I20260709 12:58:43.295372  1795 tablet_bootstrap.cc:492] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701: Bootstrap starting.
I20260709 12:58:43.305265  2078 consensus_queue.cc:1048] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:58:43.304602  1501 catalog_manager.cc:5697] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 reported cstate change: term changed from 0 to 1, leader changed from <none> to af520b67945d443b901bbeca37cc5701 (127.0.107.195). New cstate: current_term: 1 leader_uuid: "af520b67945d443b901bbeca37cc5701" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: HEALTHY } } }
I20260709 12:58:43.250608  1796 consensus_queue.cc:260] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:43.324198  1795 tablet_bootstrap.cc:654] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:43.325233  1794 tablet_bootstrap.cc:492] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d: No bootstrap required, opened a new log
I20260709 12:58:43.326179  1794 ts_tablet_manager.cc:1403] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent bootstrapping tablet: real 0.175s	user 0.026s	sys 0.024s
I20260709 12:58:43.331576  1794 raft_consensus.cc:359] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:43.332464  1794 raft_consensus.cc:385] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:43.332793  1794 raft_consensus.cc:740] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Initialized, Role: FOLLOWER
I20260709 12:58:43.333791  1794 consensus_queue.cc:260] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:43.336299  1796 ts_tablet_manager.cc:1434] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent starting tablet: real 0.098s	user 0.006s	sys 0.003s
I20260709 12:58:43.337607  1796 tablet_bootstrap.cc:492] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a: Bootstrap starting.
I20260709 12:58:43.349125  1794 ts_tablet_manager.cc:1434] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent starting tablet: real 0.022s	user 0.007s	sys 0.000s
I20260709 12:58:43.349572  2073 consensus_queue.cc:1048] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:58:43.350353  1794 tablet_bootstrap.cc:492] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d: Bootstrap starting.
I20260709 12:58:43.355715  2077 raft_consensus.cc:493] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:43.356359  2077 raft_consensus.cc:515] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:43.356918  1795 tablet_bootstrap.cc:492] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701: No bootstrap required, opened a new log
I20260709 12:58:43.357579  1795 ts_tablet_manager.cc:1403] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701: Time spent bootstrapping tablet: real 0.062s	user 0.010s	sys 0.011s
I20260709 12:58:43.360131  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:58:43.361264  1795 raft_consensus.cc:359] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:43.361970  1795 raft_consensus.cc:385] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:43.362294  1795 raft_consensus.cc:740] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Initialized, Role: FOLLOWER
I20260709 12:58:43.362282  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:58:43.363200  1674 raft_consensus.cc:2468] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 0.
W20260709 12:58:43.363471  1717 leader_election.cc:343] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:43.363250  1795 consensus_queue.cc:260] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:43.365237  1718 leader_election.cc:304] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:58:43.365878  1795 ts_tablet_manager.cc:1434] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701: Time spent starting tablet: real 0.008s	user 0.005s	sys 0.000s
I20260709 12:58:43.366148  2105 raft_consensus.cc:2804] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:58:43.366726  2105 raft_consensus.cc:493] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:58:43.367162  1795 tablet_bootstrap.cc:492] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701: Bootstrap starting.
I20260709 12:58:43.367168  2105 raft_consensus.cc:3060] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:43.359498  2077 leader_election.cc:290] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:58:43.413002  1796 tablet_bootstrap.cc:654] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:43.433097  2105 raft_consensus.cc:515] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:43.440706  1795 tablet_bootstrap.cc:654] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:43.440237  1794 tablet_bootstrap.cc:654] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:43.443702  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
W20260709 12:58:43.445410  1717 leader_election.cc:343] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:43.448038  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:58:43.448829  1674 raft_consensus.cc:3060] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:43.457509  2105 leader_election.cc:290] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:58:43.467128  2105 raft_consensus.cc:493] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:43.467617  1795 tablet_bootstrap.cc:492] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701: No bootstrap required, opened a new log
I20260709 12:58:43.468252  1795 ts_tablet_manager.cc:1403] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701: Time spent bootstrapping tablet: real 0.101s	user 0.019s	sys 0.003s
I20260709 12:58:43.467680  2105 raft_consensus.cc:515] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:43.471084  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:58:43.471457  2025 raft_consensus.cc:493] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:43.472188  2025 raft_consensus.cc:515] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:43.471581  2105 leader_election.cc:290] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:58:43.474980  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:58:43.477209  1718 leader_election.cc:304] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:58:43.476058  1674 raft_consensus.cc:2468] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 1.
I20260709 12:58:43.478585  2105 raft_consensus.cc:2749] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:58:43.478662  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:58:43.479755  1604 raft_consensus.cc:2468] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 0.
I20260709 12:58:43.479930  1718 leader_election.cc:304] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:58:43.478770  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:58:43.481254  1643 leader_election.cc:304] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 12:58:43.482048  2102 raft_consensus.cc:2804] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Leader pre-election won for term 1
W20260709 12:58:43.482105  1646 leader_election.cc:343] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:43.482239  1796 tablet_bootstrap.cc:492] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a: No bootstrap required, opened a new log
I20260709 12:58:43.482545  2102 raft_consensus.cc:493] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:58:43.477846  2025 leader_election.cc:290] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:58:43.482961  2102 raft_consensus.cc:3060] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:43.483188  1796 ts_tablet_manager.cc:1403] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent bootstrapping tablet: real 0.146s	user 0.021s	sys 0.012s
I20260709 12:58:43.483381  2105 raft_consensus.cc:2804] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:58:43.484138  2105 raft_consensus.cc:697] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 1 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 12:58:43.485404  2105 consensus_queue.cc:237] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:43.491689  1796 raft_consensus.cc:359] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:43.493055  1796 raft_consensus.cc:385] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:43.493396  1796 raft_consensus.cc:740] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Initialized, Role: FOLLOWER
I20260709 12:58:43.492373  1795 raft_consensus.cc:359] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:43.494282  1795 raft_consensus.cc:385] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:43.494643  1795 raft_consensus.cc:740] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Initialized, Role: FOLLOWER
I20260709 12:58:43.495476  1795 consensus_queue.cc:260] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:43.496732  1794 tablet_bootstrap.cc:492] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d: No bootstrap required, opened a new log
I20260709 12:58:43.497524  1794 ts_tablet_manager.cc:1403] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent bootstrapping tablet: real 0.147s	user 0.014s	sys 0.013s
I20260709 12:58:43.501174  1794 raft_consensus.cc:359] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:43.501780  1794 raft_consensus.cc:385] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:43.502023  1794 raft_consensus.cc:740] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Initialized, Role: FOLLOWER
I20260709 12:58:43.496410  1796 consensus_queue.cc:260] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:43.503398  1795 ts_tablet_manager.cc:1434] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701: Time spent starting tablet: real 0.035s	user 0.010s	sys 0.000s
I20260709 12:58:43.503126  2066 raft_consensus.cc:493] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:43.504081  2066 raft_consensus.cc:515] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:43.504905  1795 tablet_bootstrap.cc:492] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701: Bootstrap starting.
I20260709 12:58:43.511974  2102 raft_consensus.cc:515] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:43.515046  2102 leader_election.cc:290] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:58:43.516258  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:58:43.517026  1752 raft_consensus.cc:3060] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:43.519199  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:58:43.519909  1604 raft_consensus.cc:3060] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:58:43.520185  1551 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:58:43.520068  2066 leader_election.cc:290] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:58:43.520488  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:58:43.520955  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:58:43.521175  2066 raft_consensus.cc:493] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:58:43.522369  1571 leader_election.cc:343] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:43.502853  1794 consensus_queue.cc:260] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 12:58:43.530220  1569 leader_election.cc:343] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:43.530656  1794 ts_tablet_manager.cc:1434] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent starting tablet: real 0.033s	user 0.007s	sys 0.001s
I20260709 12:58:43.532148  1794 tablet_bootstrap.cc:492] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d: Bootstrap starting.
I20260709 12:58:43.533004  1569 leader_election.cc:304] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 12:58:43.534277  2073 raft_consensus.cc:493] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:43.534905  2073 raft_consensus.cc:515] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:43.535669  1928 raft_consensus.cc:2749] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:58:43.538589  2073 leader_election.cc:290] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:58:43.539794  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:58:43.541103  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:58:43.541891  1674 raft_consensus.cc:2468] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 0.
I20260709 12:58:43.543907  2102 raft_consensus.cc:493] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:43.544828  1569 leader_election.cc:304] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 12:58:43.548352  2073 raft_consensus.cc:2804] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:58:43.548686  1796 ts_tablet_manager.cc:1434] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent starting tablet: real 0.065s	user 0.008s	sys 0.003s
I20260709 12:58:43.548873  2073 raft_consensus.cc:493] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:58:43.549527  2078 raft_consensus.cc:493] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:43.550226  2078 raft_consensus.cc:515] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:43.549817  2102 raft_consensus.cc:515] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:43.553539  1604 raft_consensus.cc:2468] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 1.
I20260709 12:58:43.554539  2102 leader_election.cc:290] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:58:43.556157  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:58:43.556771  2078 leader_election.cc:290] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:58:43.549588  2073 raft_consensus.cc:3060] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:58:43.559593  1646 leader_election.cc:343] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:43.561677  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 12:58:43.564828  1646 leader_election.cc:343] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:43.565805  1643 leader_election.cc:304] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
W20260709 12:58:43.569597  1571 leader_election.cc:343] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:43.567967  2078 raft_consensus.cc:2804] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:58:43.573180  2078 raft_consensus.cc:697] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 12:58:43.577630  2078 consensus_queue.cc:237] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:43.584482  1794 tablet_bootstrap.cc:654] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:43.537992  2066 raft_consensus.cc:515] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:43.592935  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:58:43.593909  1674 raft_consensus.cc:2468] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 0.
I20260709 12:58:43.595320  1569 leader_election.cc:304] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 12:58:43.596351  1928 raft_consensus.cc:2804] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:58:43.596681  1928 raft_consensus.cc:493] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:58:43.596957  1928 raft_consensus.cc:3060] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:43.598834  2142 raft_consensus.cc:493] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:43.621475  1794 tablet_bootstrap.cc:492] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d: No bootstrap required, opened a new log
I20260709 12:58:43.621999  1794 ts_tablet_manager.cc:1403] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent bootstrapping tablet: real 0.090s	user 0.014s	sys 0.013s
I20260709 12:58:43.631038  2080 consensus_queue.cc:1048] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:58:43.621281  2142 raft_consensus.cc:515] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:43.636157  2142 leader_election.cc:290] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:58:43.637929  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:58:43.639007  1679 raft_consensus.cc:2468] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 1.
I20260709 12:58:43.640614  1718 leader_election.cc:304] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:58:43.642925  2142 raft_consensus.cc:2804] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:58:43.643564  2142 raft_consensus.cc:493] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:58:43.644037  2142 raft_consensus.cc:3060] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:58:43.647433  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:58:43.653780  1795 tablet_bootstrap.cc:654] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:43.655273  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:58:43.656277  1603 raft_consensus.cc:2468] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 1.
I20260709 12:58:43.655414  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:58:43.658946  1796 tablet_bootstrap.cc:492] T 1c2f5fa72dee4ec58979ace307d2acc3 P 83595ad0fd5940c0a2bfc7c61acd969a: Bootstrap starting.
I20260709 12:58:43.668298  1928 raft_consensus.cc:515] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:43.651436  2066 leader_election.cc:290] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:58:43.678013  1928 leader_election.cc:290] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:58:43.679184  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:58:43.680066  1674 raft_consensus.cc:3060] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:58:43.682449  1643 leader_election.cc:343] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:43.685603  1643 leader_election.cc:304] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 12:58:43.687667  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:58:43.688606  1751 raft_consensus.cc:2468] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 0.
I20260709 12:58:43.699365  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:58:43.700150  1748 raft_consensus.cc:3060] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:43.693167  1752 raft_consensus.cc:2468] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 1.
I20260709 12:58:43.692921  1902 raft_consensus.cc:2749] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:58:43.704746  2105 consensus_queue.cc:1048] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.015s
I20260709 12:58:43.727449  2142 raft_consensus.cc:515] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:43.732432  1752 raft_consensus.cc:3060] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:43.735533  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:58:43.720660  1794 raft_consensus.cc:359] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:43.736922  1794 raft_consensus.cc:385] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:43.737262  1794 raft_consensus.cc:740] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Initialized, Role: FOLLOWER
I20260709 12:58:43.737720  1603 raft_consensus.cc:3060] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:58:43.738021  1794 consensus_queue.cc:260] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:43.741205  1794 ts_tablet_manager.cc:1434] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent starting tablet: real 0.119s	user 0.006s	sys 0.006s
I20260709 12:58:43.740470  1501 catalog_manager.cc:5697] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d reported cstate change: term changed from 0 to 1, leader changed from <none> to 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194). New cstate: current_term: 1 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } }
I20260709 12:58:43.745126  1794 tablet_bootstrap.cc:492] T 6f6cc48def184bcc907c18c98277187f P 816fadcea8fc4893882f5f9b9f22a03d: Bootstrap starting.
I20260709 12:58:43.756924  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:58:43.757946  1680 raft_consensus.cc:3060] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:58:43.761765  2142 leader_election.cc:290] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 2 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:58:43.771356  1795 tablet_bootstrap.cc:492] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701: No bootstrap required, opened a new log
I20260709 12:58:43.772255  1795 ts_tablet_manager.cc:1403] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701: Time spent bootstrapping tablet: real 0.268s	user 0.019s	sys 0.012s
I20260709 12:58:43.776353  1796 tablet_bootstrap.cc:654] T 1c2f5fa72dee4ec58979ace307d2acc3 P 83595ad0fd5940c0a2bfc7c61acd969a: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:43.780062  1503 catalog_manager.cc:5697] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 reported cstate change: term changed from 0 to 1, leader changed from <none> to af520b67945d443b901bbeca37cc5701 (127.0.107.195). New cstate: current_term: 1 leader_uuid: "af520b67945d443b901bbeca37cc5701" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: HEALTHY } } }
I20260709 12:58:43.827169  1680 raft_consensus.cc:2468] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 2.
I20260709 12:58:43.853308  1902 raft_consensus.cc:493] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:43.854215  1902 raft_consensus.cc:515] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:43.857813  1902 leader_election.cc:290] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:58:43.859827  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:58:43.862311  1718 leader_election.cc:304] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:58:43.840466  1603 raft_consensus.cc:2468] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 2.
I20260709 12:58:43.869774  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:58:43.867063  2080 raft_consensus.cc:2804] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader election won for term 2
W20260709 12:58:43.877503  1646 leader_election.cc:343] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:43.878402  2073 raft_consensus.cc:515] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:43.872501  2080 raft_consensus.cc:697] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 2 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 12:58:43.880244  1598 raft_consensus.cc:2393] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 1: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 12:58:43.880877  2080 consensus_queue.cc:237] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:43.886924  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:58:43.889525  1680 raft_consensus.cc:3060] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:43.899026  1643 leader_election.cc:304] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 12:58:43.901099  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
W20260709 12:58:43.906535  1643 leader_election.cc:343] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:43.906993  1643 leader_election.cc:304] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 12:58:43.908946  2025 raft_consensus.cc:2749] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:58:43.911669  1748 raft_consensus.cc:2468] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 1.
I20260709 12:58:43.907090  1795 raft_consensus.cc:359] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:43.913502  1795 raft_consensus.cc:385] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:43.913826  1795 raft_consensus.cc:740] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Initialized, Role: FOLLOWER
I20260709 12:58:43.914546  1795 consensus_queue.cc:260] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:43.918215  1795 ts_tablet_manager.cc:1434] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701: Time spent starting tablet: real 0.142s	user 0.007s	sys 0.001s
I20260709 12:58:43.919514  1795 tablet_bootstrap.cc:492] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701: Bootstrap starting.
I20260709 12:58:43.923943  2073 leader_election.cc:290] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 12:58:43.931499  1571 leader_election.cc:343] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:43.936362  1571 leader_election.cc:304] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:58:43.932137  1680 raft_consensus.cc:2468] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 1.
I20260709 12:58:43.937598  2075 raft_consensus.cc:2749] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:58:43.944684  1569 leader_election.cc:304] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 12:58:43.946311  2101 raft_consensus.cc:2804] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:58:43.947152  2101 raft_consensus.cc:697] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 12:58:43.948973  2066 raft_consensus.cc:2804] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:58:43.949802  2066 raft_consensus.cc:697] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 12:58:43.948786  2101 consensus_queue.cc:237] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:43.951198  2066 consensus_queue.cc:237] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:43.961457  2142 consensus_queue.cc:1048] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:58:43.954250  1674 raft_consensus.cc:2468] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 1.
I20260709 12:58:44.002508  1794 tablet_bootstrap.cc:654] T 6f6cc48def184bcc907c18c98277187f P 816fadcea8fc4893882f5f9b9f22a03d: Neither blocks nor log segments found. Creating new log.
W20260709 12:58:44.035838  1717 consensus_peers.cc:597] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:58:44.040859  1795 tablet_bootstrap.cc:654] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:44.045655  2078 consensus_queue.cc:1048] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:58:44.052306  1501 catalog_manager.cc:5697] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 reported cstate change: term changed from 0 to 2, leader changed from <none> to af520b67945d443b901bbeca37cc5701 (127.0.107.195). New cstate: current_term: 2 leader_uuid: "af520b67945d443b901bbeca37cc5701" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: HEALTHY } } }
I20260709 12:58:44.056303  1796 tablet_bootstrap.cc:492] T 1c2f5fa72dee4ec58979ace307d2acc3 P 83595ad0fd5940c0a2bfc7c61acd969a: No bootstrap required, opened a new log
I20260709 12:58:44.057014  1796 ts_tablet_manager.cc:1403] T 1c2f5fa72dee4ec58979ace307d2acc3 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent bootstrapping tablet: real 0.399s	user 0.027s	sys 0.020s
I20260709 12:58:44.060796  1794 tablet_bootstrap.cc:492] T 6f6cc48def184bcc907c18c98277187f P 816fadcea8fc4893882f5f9b9f22a03d: No bootstrap required, opened a new log
I20260709 12:58:44.061321  1794 ts_tablet_manager.cc:1403] T 6f6cc48def184bcc907c18c98277187f P 816fadcea8fc4893882f5f9b9f22a03d: Time spent bootstrapping tablet: real 0.317s	user 0.013s	sys 0.009s
I20260709 12:58:44.061781  1796 raft_consensus.cc:359] T 1c2f5fa72dee4ec58979ace307d2acc3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:44.062614  1796 raft_consensus.cc:385] T 1c2f5fa72dee4ec58979ace307d2acc3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:44.062965  1796 raft_consensus.cc:740] T 1c2f5fa72dee4ec58979ace307d2acc3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Initialized, Role: FOLLOWER
I20260709 12:58:44.064586  1796 consensus_queue.cc:260] T 1c2f5fa72dee4ec58979ace307d2acc3 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:44.066196  1795 tablet_bootstrap.cc:492] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701: No bootstrap required, opened a new log
I20260709 12:58:44.066709  1795 ts_tablet_manager.cc:1403] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701: Time spent bootstrapping tablet: real 0.148s	user 0.015s	sys 0.000s
I20260709 12:58:44.066977  1796 ts_tablet_manager.cc:1434] T 1c2f5fa72dee4ec58979ace307d2acc3 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent starting tablet: real 0.010s	user 0.009s	sys 0.001s
I20260709 12:58:44.066601  1503 catalog_manager.cc:5697] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 0 to 1, leader changed from <none> to 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193). New cstate: current_term: 1 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } }
I20260709 12:58:44.068071  1796 tablet_bootstrap.cc:492] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a: Bootstrap starting.
I20260709 12:58:44.069705  1795 raft_consensus.cc:359] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:44.070459  1795 raft_consensus.cc:385] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:44.070731  1795 raft_consensus.cc:740] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Initialized, Role: FOLLOWER
I20260709 12:58:44.072228  2102 consensus_queue.cc:1048] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:58:44.075311  1796 tablet_bootstrap.cc:654] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:44.068462  1503 catalog_manager.cc:5697] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 0 to 1, leader changed from <none> to 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193). New cstate: current_term: 1 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } }
I20260709 12:58:44.077999  1795 consensus_queue.cc:260] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:44.076972  1794 raft_consensus.cc:359] T 6f6cc48def184bcc907c18c98277187f P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:44.079595  1794 raft_consensus.cc:385] T 6f6cc48def184bcc907c18c98277187f P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:44.080003  1794 raft_consensus.cc:740] T 6f6cc48def184bcc907c18c98277187f P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Initialized, Role: FOLLOWER
I20260709 12:58:44.080746  1794 consensus_queue.cc:260] T 6f6cc48def184bcc907c18c98277187f P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:44.082957  1794 ts_tablet_manager.cc:1434] T 6f6cc48def184bcc907c18c98277187f P 816fadcea8fc4893882f5f9b9f22a03d: Time spent starting tablet: real 0.021s	user 0.007s	sys 0.001s
I20260709 12:58:44.086027  1795 ts_tablet_manager.cc:1434] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701: Time spent starting tablet: real 0.019s	user 0.006s	sys 0.000s
I20260709 12:58:44.088482  1796 tablet_bootstrap.cc:492] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a: No bootstrap required, opened a new log
I20260709 12:58:44.088984  1796 ts_tablet_manager.cc:1403] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent bootstrapping tablet: real 0.021s	user 0.011s	sys 0.003s
I20260709 12:58:44.089641  1794 tablet_bootstrap.cc:492] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d: Bootstrap starting.
I20260709 12:58:44.091610  1795 tablet_bootstrap.cc:492] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701: Bootstrap starting.
I20260709 12:58:44.095705  2101 raft_consensus.cc:493] T 1c2f5fa72dee4ec58979ace307d2acc3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:44.097997  2101 raft_consensus.cc:515] T 1c2f5fa72dee4ec58979ace307d2acc3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:44.100991  2101 leader_election.cc:290] T 1c2f5fa72dee4ec58979ace307d2acc3 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:58:44.101923  1796 raft_consensus.cc:359] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:44.110924  1796 raft_consensus.cc:385] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:44.111342  1796 raft_consensus.cc:740] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Initialized, Role: FOLLOWER
I20260709 12:58:44.112099  1794 tablet_bootstrap.cc:654] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:44.112198  1796 consensus_queue.cc:260] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:44.113272  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:58:44.115913  1796 ts_tablet_manager.cc:1434] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent starting tablet: real 0.027s	user 0.004s	sys 0.004s
I20260709 12:58:44.120111  1796 tablet_bootstrap.cc:492] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a: Bootstrap starting.
I20260709 12:58:44.118837  1795 tablet_bootstrap.cc:654] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:44.124289  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:58:44.139189  1571 leader_election.cc:304] T 1c2f5fa72dee4ec58979ace307d2acc3 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 12:58:44.140036  2101 raft_consensus.cc:2749] T 1c2f5fa72dee4ec58979ace307d2acc3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:58:44.141029  1796 tablet_bootstrap.cc:654] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:44.141029  1794 tablet_bootstrap.cc:492] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d: No bootstrap required, opened a new log
I20260709 12:58:44.141863  1794 ts_tablet_manager.cc:1403] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent bootstrapping tablet: real 0.052s	user 0.017s	sys 0.011s
I20260709 12:58:44.147435  1794 raft_consensus.cc:359] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:44.156251  1796 tablet_bootstrap.cc:492] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a: No bootstrap required, opened a new log
I20260709 12:58:44.164634  1794 raft_consensus.cc:385] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:44.164970  1796 ts_tablet_manager.cc:1403] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent bootstrapping tablet: real 0.045s	user 0.012s	sys 0.007s
I20260709 12:58:44.165030  1794 raft_consensus.cc:740] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Initialized, Role: FOLLOWER
I20260709 12:58:44.166465  1794 consensus_queue.cc:260] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:44.168326  1794 ts_tablet_manager.cc:1434] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent starting tablet: real 0.026s	user 0.008s	sys 0.000s
I20260709 12:58:44.168795  1796 raft_consensus.cc:359] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:44.169480  1794 tablet_bootstrap.cc:492] T a3accbb733bb4cafa69428319664f239 P 816fadcea8fc4893882f5f9b9f22a03d: Bootstrap starting.
I20260709 12:58:44.169600  1796 raft_consensus.cc:385] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:44.170125  1796 raft_consensus.cc:740] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Initialized, Role: FOLLOWER
I20260709 12:58:44.170997  1796 consensus_queue.cc:260] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:44.173508  1796 ts_tablet_manager.cc:1434] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent starting tablet: real 0.008s	user 0.005s	sys 0.001s
I20260709 12:58:44.179137  1796 tablet_bootstrap.cc:492] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a: Bootstrap starting.
I20260709 12:58:44.183385  2080 raft_consensus.cc:493] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:44.184504  2080 raft_consensus.cc:515] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:44.187057  1795 tablet_bootstrap.cc:492] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701: No bootstrap required, opened a new log
I20260709 12:58:44.187638  1795 ts_tablet_manager.cc:1403] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701: Time spent bootstrapping tablet: real 0.096s	user 0.019s	sys 0.017s
I20260709 12:58:44.189280  2080 leader_election.cc:290] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:58:44.189343  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:58:44.191684  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:58:44.192992  1680 raft_consensus.cc:2393] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 1: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 12:58:44.193802  1795 raft_consensus.cc:359] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:44.194644  1795 raft_consensus.cc:385] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:44.194878  1718 leader_election.cc:304] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:58:44.195264  1795 raft_consensus.cc:740] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Initialized, Role: FOLLOWER
I20260709 12:58:44.196748  2080 raft_consensus.cc:3060] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:44.196269  1795 consensus_queue.cc:260] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:44.200099  1795 ts_tablet_manager.cc:1434] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701: Time spent starting tablet: real 0.012s	user 0.006s	sys 0.003s
I20260709 12:58:44.208637  1794 tablet_bootstrap.cc:654] T a3accbb733bb4cafa69428319664f239 P 816fadcea8fc4893882f5f9b9f22a03d: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:44.210935  1795 tablet_bootstrap.cc:492] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701: Bootstrap starting.
I20260709 12:58:44.221305  2080 raft_consensus.cc:2749] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:58:44.222362  1796 tablet_bootstrap.cc:654] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:44.224481  1795 tablet_bootstrap.cc:654] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:44.226073  1794 tablet_bootstrap.cc:492] T a3accbb733bb4cafa69428319664f239 P 816fadcea8fc4893882f5f9b9f22a03d: No bootstrap required, opened a new log
I20260709 12:58:44.226997  1794 ts_tablet_manager.cc:1403] T a3accbb733bb4cafa69428319664f239 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent bootstrapping tablet: real 0.058s	user 0.004s	sys 0.025s
I20260709 12:58:44.230967  2101 raft_consensus.cc:493] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:44.231617  2101 raft_consensus.cc:515] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:44.236451  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 12:58:44.238220  1569 leader_election.cc:343] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:44.240873  2101 leader_election.cc:290] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:58:44.243019  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:58:44.243986  1747 raft_consensus.cc:2468] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 0.
I20260709 12:58:44.245582  1794 raft_consensus.cc:359] T a3accbb733bb4cafa69428319664f239 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:44.246665  1794 raft_consensus.cc:385] T a3accbb733bb4cafa69428319664f239 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:44.247041  1794 raft_consensus.cc:740] T a3accbb733bb4cafa69428319664f239 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Initialized, Role: FOLLOWER
I20260709 12:58:44.248095  1794 consensus_queue.cc:260] T a3accbb733bb4cafa69428319664f239 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:44.254980  1794 ts_tablet_manager.cc:1434] T a3accbb733bb4cafa69428319664f239 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent starting tablet: real 0.028s	user 0.009s	sys 0.002s
I20260709 12:58:44.261536  1794 tablet_bootstrap.cc:492] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d: Bootstrap starting.
I20260709 12:58:44.246198  1571 leader_election.cc:304] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 12:58:44.271842  2101 raft_consensus.cc:2804] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:58:44.272411  2101 raft_consensus.cc:493] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:58:44.272831  2101 raft_consensus.cc:3060] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:44.273854  1794 tablet_bootstrap.cc:654] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:44.289489  1794 tablet_bootstrap.cc:492] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d: No bootstrap required, opened a new log
I20260709 12:58:44.289474  1674 raft_consensus.cc:1275] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260709 12:58:44.290460  1794 ts_tablet_manager.cc:1403] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent bootstrapping tablet: real 0.030s	user 0.021s	sys 0.004s
W20260709 12:58:44.290333  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 44244.7us
I20260709 12:58:44.291832  2142 consensus_queue.cc:1048] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:58:44.295344  1794 raft_consensus.cc:359] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:44.296754  1794 raft_consensus.cc:385] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:44.297266  1794 raft_consensus.cc:740] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Initialized, Role: FOLLOWER
I20260709 12:58:44.298903  1794 consensus_queue.cc:260] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:44.301013  2142 raft_consensus.cc:493] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:44.301685  2142 raft_consensus.cc:515] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:44.302707  1794 ts_tablet_manager.cc:1434] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent starting tablet: real 0.012s	user 0.010s	sys 0.000s
I20260709 12:58:44.307126  2142 leader_election.cc:290] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:58:44.307433  1794 tablet_bootstrap.cc:492] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d: Bootstrap starting.
I20260709 12:58:44.309583  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:58:44.319067  1676 raft_consensus.cc:2468] T 6f6cc48def184bcc907c18c98277187f P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 0.
I20260709 12:58:44.321241  1718 leader_election.cc:304] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:58:44.322372  2142 raft_consensus.cc:2804] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:58:44.322216  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:58:44.323004  2142 raft_consensus.cc:493] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:58:44.323369  1603 raft_consensus.cc:2468] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 0.
I20260709 12:58:44.324661  2075 raft_consensus.cc:493] T a3accbb733bb4cafa69428319664f239 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:44.325353  2075 raft_consensus.cc:515] T a3accbb733bb4cafa69428319664f239 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:44.311715  2101 raft_consensus.cc:515] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:44.323531  2142 raft_consensus.cc:3060] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:44.333410  2075 leader_election.cc:290] T a3accbb733bb4cafa69428319664f239 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:58:44.338177  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 12:58:44.350425  1646 leader_election.cc:343] T a3accbb733bb4cafa69428319664f239 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:44.343039  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:58:44.347285  2101 leader_election.cc:290] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:58:44.347522  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 12:58:44.353947  1569 leader_election.cc:343] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:44.375890  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:58:44.376760  1747 raft_consensus.cc:3060] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:44.385214  1928 consensus_queue.cc:1048] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260709 12:58:44.389765  1643 leader_election.cc:343] T a3accbb733bb4cafa69428319664f239 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:44.390412  1643 leader_election.cc:304] T a3accbb733bb4cafa69428319664f239 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 12:58:44.390681  1599 raft_consensus.cc:1275] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260709 12:58:44.394013  1902 raft_consensus.cc:2749] T a3accbb733bb4cafa69428319664f239 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:58:44.400760  2101 raft_consensus.cc:493] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:44.401371  2101 raft_consensus.cc:515] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:44.404558  1794 tablet_bootstrap.cc:654] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:44.404934  2106 consensus_queue.cc:1048] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:58:44.405958  2101 leader_election.cc:290] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:58:44.409458  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:58:44.410885  2142 raft_consensus.cc:515] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:44.398114  1747 raft_consensus.cc:2468] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 1.
I20260709 12:58:44.418378  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:58:44.400295  1795 tablet_bootstrap.cc:492] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701: No bootstrap required, opened a new log
I20260709 12:58:44.421339  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:58:44.422024  1680 raft_consensus.cc:3060] T 6f6cc48def184bcc907c18c98277187f P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:44.419291  1679 raft_consensus.cc:2468] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 0.
I20260709 12:58:44.430366  2075 raft_consensus.cc:493] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:44.431171  2075 raft_consensus.cc:515] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:44.414166  2142 leader_election.cc:290] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:58:44.448065  1680 raft_consensus.cc:2468] T 6f6cc48def184bcc907c18c98277187f P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 1.
I20260709 12:58:44.474195  1796 tablet_bootstrap.cc:492] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a: No bootstrap required, opened a new log
W20260709 12:58:44.492817  1571 leader_election.cc:343] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:44.493458  1796 ts_tablet_manager.cc:1403] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent bootstrapping tablet: real 0.319s	user 0.039s	sys 0.029s
I20260709 12:58:44.493996  1571 leader_election.cc:304] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 12:58:44.495251  1928 raft_consensus.cc:2804] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:58:44.495743  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:58:44.495757  1928 raft_consensus.cc:697] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 12:58:44.496433  1605 raft_consensus.cc:3060] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:44.496923  1928 consensus_queue.cc:237] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:44.492550  1795 ts_tablet_manager.cc:1403] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701: Time spent bootstrapping tablet: real 0.282s	user 0.054s	sys 0.096s
I20260709 12:58:44.492599  2075 leader_election.cc:290] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:58:44.506079  1795 raft_consensus.cc:359] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:44.506775  1795 raft_consensus.cc:385] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:44.507055  1795 raft_consensus.cc:740] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Initialized, Role: FOLLOWER
I20260709 12:58:44.502018  1796 raft_consensus.cc:359] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:44.508083  1796 raft_consensus.cc:385] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:44.508301  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:58:44.509346  1795 consensus_queue.cc:260] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:44.512221  1795 ts_tablet_manager.cc:1434] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701: Time spent starting tablet: real 0.014s	user 0.006s	sys 0.000s
W20260709 12:58:44.513188  1646 leader_election.cc:343] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:44.513415  1605 raft_consensus.cc:2468] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 1.
I20260709 12:58:44.515447  1718 leader_election.cc:304] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:58:44.516764  2142 raft_consensus.cc:2804] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:58:44.517547  2142 raft_consensus.cc:697] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 1 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 12:58:44.519534  1599 raft_consensus.cc:3060] T 1c2f5fa72dee4ec58979ace307d2acc3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:44.518965  2142 consensus_queue.cc:237] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:44.523716  1569 leader_election.cc:304] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 12:58:44.524986  2066 raft_consensus.cc:2804] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:58:44.525419  2066 raft_consensus.cc:493] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:58:44.526257  2066 raft_consensus.cc:3060] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:44.539737  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:58:44.508966  1796 raft_consensus.cc:740] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Initialized, Role: FOLLOWER
I20260709 12:58:44.556466  1796 consensus_queue.cc:260] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:44.560866  1796 ts_tablet_manager.cc:1434] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent starting tablet: real 0.067s	user 0.004s	sys 0.004s
I20260709 12:58:44.561018  1928 consensus_queue.cc:1048] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:58:44.562825  1796 tablet_bootstrap.cc:492] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a: Bootstrap starting.
W20260709 12:58:44.571065  1643 leader_election.cc:343] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:44.571525  1643 leader_election.cc:304] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 12:58:44.591248  1795 tablet_bootstrap.cc:492] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701: Bootstrap starting.
I20260709 12:58:44.602029  2025 raft_consensus.cc:493] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:44.602761  2025 raft_consensus.cc:515] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:44.605860  2025 leader_election.cc:290] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 12:58:44.613035  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 43589.7us
I20260709 12:58:44.598522  1902 raft_consensus.cc:2749] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:58:44.631198  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:58:44.655803  1796 tablet_bootstrap.cc:654] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:44.665925  1794 tablet_bootstrap.cc:492] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d: No bootstrap required, opened a new log
I20260709 12:58:44.666567  1794 ts_tablet_manager.cc:1403] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent bootstrapping tablet: real 0.362s	user 0.032s	sys 0.032s
I20260709 12:58:44.672003  1503 catalog_manager.cc:5697] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 0 to 1, leader changed from <none> to 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193). New cstate: current_term: 1 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } }
I20260709 12:58:44.692451  1501 catalog_manager.cc:5697] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 reported cstate change: term changed from 0 to 1, leader changed from <none> to af520b67945d443b901bbeca37cc5701 (127.0.107.195). New cstate: current_term: 1 leader_uuid: "af520b67945d443b901bbeca37cc5701" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: HEALTHY } } }
W20260709 12:58:44.700655  1551 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:58:44.730988  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 12:58:44.733494  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 122099us
I20260709 12:58:44.748915  1794 raft_consensus.cc:359] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:44.750110  1794 raft_consensus.cc:385] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:44.750551  1794 raft_consensus.cc:740] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Initialized, Role: FOLLOWER
I20260709 12:58:44.753672  1794 consensus_queue.cc:260] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:44.724156  2066 raft_consensus.cc:515] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 12:58:44.768890  1643 leader_election.cc:343] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:44.770017  2215 consensus_queue.cc:1048] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:58:44.770579  1928 consensus_queue.cc:1048] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260709 12:58:44.775938  1646 leader_election.cc:343] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:44.788717  1795 tablet_bootstrap.cc:654] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:44.791944  2066 leader_election.cc:290] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:58:44.795135  1646 leader_election.cc:304] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 12:58:44.797159  2210 raft_consensus.cc:2749] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:58:44.807353  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:58:44.808300  1679 raft_consensus.cc:3060] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:44.823822  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:58:44.828673  1794 ts_tablet_manager.cc:1434] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent starting tablet: real 0.162s	user 0.009s	sys 0.000s
W20260709 12:58:44.832350  1571 leader_election.cc:343] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:44.860368  1794 tablet_bootstrap.cc:492] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d: Bootstrap starting.
I20260709 12:58:44.877602  1796 tablet_bootstrap.cc:492] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a: No bootstrap required, opened a new log
I20260709 12:58:44.878134  1796 ts_tablet_manager.cc:1403] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent bootstrapping tablet: real 0.316s	user 0.012s	sys 0.007s
I20260709 12:58:44.912118  1796 raft_consensus.cc:359] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:44.912945  1796 raft_consensus.cc:385] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:44.913226  1796 raft_consensus.cc:740] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Initialized, Role: FOLLOWER
I20260709 12:58:44.919569  1796 consensus_queue.cc:260] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:44.928758  1796 ts_tablet_manager.cc:1434] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent starting tablet: real 0.050s	user 0.008s	sys 0.001s
I20260709 12:58:44.921479  1679 raft_consensus.cc:2468] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 1.
I20260709 12:58:44.945003  1796 tablet_bootstrap.cc:492] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a: Bootstrap starting.
W20260709 12:58:44.946171  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 76277.2us
I20260709 12:58:44.952168  1569 leader_election.cc:304] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 12:58:44.953512  2215 raft_consensus.cc:2804] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:58:44.950301  2105 raft_consensus.cc:493] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:44.954188  2215 raft_consensus.cc:697] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 12:58:44.954923  2077 consensus_queue.cc:1048] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:58:44.955636  2215 consensus_queue.cc:237] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:44.954659  2105 raft_consensus.cc:515] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:44.966483  2066 raft_consensus.cc:493] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:44.993992  2066 raft_consensus.cc:515] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:44.994585  2105 leader_election.cc:290] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:58:44.995820  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:58:44.997445  2066 leader_election.cc:290] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 12:58:45.002348  1718 leader_election.cc:343] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:45.025250  1794 tablet_bootstrap.cc:654] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:45.025419  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:58:45.026257  1753 raft_consensus.cc:2468] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 0.
I20260709 12:58:45.031093  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:58:45.031876  1795 tablet_bootstrap.cc:492] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701: No bootstrap required, opened a new log
I20260709 12:58:45.032557  1795 ts_tablet_manager.cc:1403] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701: Time spent bootstrapping tablet: real 0.460s	user 0.021s	sys 0.003s
I20260709 12:58:45.029307  1571 leader_election.cc:304] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:58:45.066110  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:58:45.067459  2215 raft_consensus.cc:2804] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:58:45.068147  2215 raft_consensus.cc:493] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:58:45.068670  2215 raft_consensus.cc:3060] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:58:45.071990  1569 leader_election.cc:343] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:45.075032  2209 raft_consensus.cc:493] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:45.087563  2209 raft_consensus.cc:515] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:45.091099  2209 leader_election.cc:290] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:58:45.086030  2106 consensus_queue.cc:1048] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.006s
I20260709 12:58:45.075783  2212 consensus_queue.cc:1048] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.018s
I20260709 12:58:45.101204  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 12:58:45.102847  1571 leader_election.cc:343] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:45.108652  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:58:45.109361  1674 raft_consensus.cc:2468] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 0.
I20260709 12:58:45.111132  1795 raft_consensus.cc:359] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:45.112282  1795 raft_consensus.cc:385] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:45.116401  1796 tablet_bootstrap.cc:654] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:45.117399  1795 raft_consensus.cc:740] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Initialized, Role: FOLLOWER
I20260709 12:58:45.118147  1795 consensus_queue.cc:260] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:45.131387  1569 leader_election.cc:304] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 12:58:45.154256  1795 ts_tablet_manager.cc:1434] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701: Time spent starting tablet: real 0.121s	user 0.008s	sys 0.001s
W20260709 12:58:45.158114  1569 consensus_peers.cc:597] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:58:45.153652  2207 raft_consensus.cc:2804] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:58:45.168322  1795 tablet_bootstrap.cc:492] T f6028a708cb344fbb08ab8056921d9e7 P af520b67945d443b901bbeca37cc5701: Bootstrap starting.
I20260709 12:58:45.165973  2207 raft_consensus.cc:493] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:58:45.194898  2207 raft_consensus.cc:3060] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:45.180917  2210 raft_consensus.cc:493] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:45.203364  2210 raft_consensus.cc:515] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 12:58:45.249033  2234 log.cc:927] Time spent T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.073s	user 0.001s	sys 0.000s
I20260709 12:58:45.288736  1502 catalog_manager.cc:5697] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 0 to 1, leader changed from <none> to 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193). New cstate: current_term: 1 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } }
I20260709 12:58:45.328523  1794 tablet_bootstrap.cc:492] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d: No bootstrap required, opened a new log
I20260709 12:58:45.329180  1794 ts_tablet_manager.cc:1403] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent bootstrapping tablet: real 0.489s	user 0.026s	sys 0.073s
I20260709 12:58:45.339912  1796 tablet_bootstrap.cc:492] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a: No bootstrap required, opened a new log
I20260709 12:58:45.340543  1796 ts_tablet_manager.cc:1403] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent bootstrapping tablet: real 0.396s	user 0.015s	sys 0.012s
I20260709 12:58:45.344828  1794 raft_consensus.cc:359] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:45.290432  2215 raft_consensus.cc:515] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 12:58:45.391088  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 49462.6us
I20260709 12:58:45.391078  1794 raft_consensus.cc:385] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:45.392566  1794 raft_consensus.cc:740] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Initialized, Role: FOLLOWER
I20260709 12:58:45.393271  1794 consensus_queue.cc:260] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:45.393924  2215 leader_election.cc:290] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:58:45.396543  2210 leader_election.cc:290] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:58:45.397342  1599 raft_consensus.cc:2393] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 1: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 12:58:45.399170  1796 raft_consensus.cc:359] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:45.400038  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:58:45.400581  1796 raft_consensus.cc:385] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:45.401104  1796 raft_consensus.cc:740] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Initialized, Role: FOLLOWER
W20260709 12:58:45.404477  1569 leader_election.cc:343] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:45.420629  1794 ts_tablet_manager.cc:1434] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent starting tablet: real 0.090s	user 0.007s	sys 0.000s
I20260709 12:58:45.405325  1795 tablet_bootstrap.cc:654] T f6028a708cb344fbb08ab8056921d9e7 P af520b67945d443b901bbeca37cc5701: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:45.402354  1796 consensus_queue.cc:260] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:45.423487  1794 tablet_bootstrap.cc:492] T 1d9ac76dc2c445edb5a25cf9087998bb P 816fadcea8fc4893882f5f9b9f22a03d: Bootstrap starting.
I20260709 12:58:45.337046  2207 raft_consensus.cc:515] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:45.435317  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:58:45.436740  2207 leader_election.cc:290] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:58:45.437418  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:58:45.438225  1745 raft_consensus.cc:3060] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:45.438862  1796 ts_tablet_manager.cc:1434] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent starting tablet: real 0.098s	user 0.010s	sys 0.000s
I20260709 12:58:45.440666  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:58:45.441366  1680 raft_consensus.cc:3060] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:58:45.449505  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 107378us
I20260709 12:58:45.470731  1717 leader_election.cc:304] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:58:45.446388  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:58:45.500015  1745 raft_consensus.cc:2468] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 1.
W20260709 12:58:45.507201  1646 leader_election.cc:343] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:45.512729  1680 raft_consensus.cc:2468] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 1.
I20260709 12:58:45.521503  1569 leader_election.cc:304] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 12:58:45.503334  2106 raft_consensus.cc:2749] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:58:45.529729  2236 raft_consensus.cc:2804] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:58:45.530596  2236 raft_consensus.cc:697] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 12:58:45.525888  1796 tablet_bootstrap.cc:492] T 41166f283aea41b0b3f5be60e8234443 P 83595ad0fd5940c0a2bfc7c61acd969a: Bootstrap starting.
I20260709 12:58:45.522667  1571 leader_election.cc:304] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 12:58:45.534572  1680 raft_consensus.cc:3060] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:45.541555  2215 raft_consensus.cc:2804] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:58:45.542214  2215 raft_consensus.cc:697] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 12:58:45.543511  2215 consensus_queue.cc:237] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 12:58:45.547389  1643 leader_election.cc:343] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:45.548026  1643 leader_election.cc:304] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 12:58:45.551019  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:58:45.553052  1794 tablet_bootstrap.cc:654] T 1d9ac76dc2c445edb5a25cf9087998bb P 816fadcea8fc4893882f5f9b9f22a03d: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:45.552896  2236 consensus_queue.cc:237] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:45.558115  2025 raft_consensus.cc:2749] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:58:45.590374  1795 tablet_bootstrap.cc:492] T f6028a708cb344fbb08ab8056921d9e7 P af520b67945d443b901bbeca37cc5701: No bootstrap required, opened a new log
I20260709 12:58:45.591058  1795 ts_tablet_manager.cc:1403] T f6028a708cb344fbb08ab8056921d9e7 P af520b67945d443b901bbeca37cc5701: Time spent bootstrapping tablet: real 0.433s	user 0.027s	sys 0.038s
I20260709 12:58:45.598086  1795 raft_consensus.cc:359] T f6028a708cb344fbb08ab8056921d9e7 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:45.598918  1795 raft_consensus.cc:385] T f6028a708cb344fbb08ab8056921d9e7 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:45.599242  1795 raft_consensus.cc:740] T f6028a708cb344fbb08ab8056921d9e7 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Initialized, Role: FOLLOWER
I20260709 12:58:45.600039  1795 consensus_queue.cc:260] T f6028a708cb344fbb08ab8056921d9e7 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:45.603235  2106 raft_consensus.cc:493] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:45.619812  1795 ts_tablet_manager.cc:1434] T f6028a708cb344fbb08ab8056921d9e7 P af520b67945d443b901bbeca37cc5701: Time spent starting tablet: real 0.028s	user 0.008s	sys 0.000s
I20260709 12:58:45.621083  2106 raft_consensus.cc:515] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:45.629676  2106 leader_election.cc:290] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:58:45.631553  1795 tablet_bootstrap.cc:492] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701: Bootstrap starting.
I20260709 12:58:45.637487  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:58:45.638996  1677 raft_consensus.cc:2393] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 1: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 12:58:45.655690  1748 raft_consensus.cc:3060] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:45.659004  1796 tablet_bootstrap.cc:654] T 41166f283aea41b0b3f5be60e8234443 P 83595ad0fd5940c0a2bfc7c61acd969a: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:45.660532  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 12:58:45.668354  1571 leader_election.cc:343] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:45.690717  2212 consensus_queue.cc:1048] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:58:45.732167  1717 leader_election.cc:304] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:58:45.734513  2105 raft_consensus.cc:2749] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:58:45.739271  1794 tablet_bootstrap.cc:492] T 1d9ac76dc2c445edb5a25cf9087998bb P 816fadcea8fc4893882f5f9b9f22a03d: No bootstrap required, opened a new log
I20260709 12:58:45.739984  1794 ts_tablet_manager.cc:1403] T 1d9ac76dc2c445edb5a25cf9087998bb P 816fadcea8fc4893882f5f9b9f22a03d: Time spent bootstrapping tablet: real 0.317s	user 0.031s	sys 0.019s
W20260709 12:58:45.746120  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 65632.7us
I20260709 12:58:45.752254  2236 raft_consensus.cc:493] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:45.753042  1794 raft_consensus.cc:359] T 1d9ac76dc2c445edb5a25cf9087998bb P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:45.752966  2236 raft_consensus.cc:515] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:45.753912  1794 raft_consensus.cc:385] T 1d9ac76dc2c445edb5a25cf9087998bb P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:45.754217  1794 raft_consensus.cc:740] T 1d9ac76dc2c445edb5a25cf9087998bb P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Initialized, Role: FOLLOWER
I20260709 12:58:45.754423  1796 tablet_bootstrap.cc:492] T 41166f283aea41b0b3f5be60e8234443 P 83595ad0fd5940c0a2bfc7c61acd969a: No bootstrap required, opened a new log
I20260709 12:58:45.754930  1796 ts_tablet_manager.cc:1403] T 41166f283aea41b0b3f5be60e8234443 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent bootstrapping tablet: real 0.229s	user 0.019s	sys 0.036s
I20260709 12:58:45.755019  1794 consensus_queue.cc:260] T 1d9ac76dc2c445edb5a25cf9087998bb P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:45.756946  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:58:45.757597  1676 raft_consensus.cc:2468] T a3accbb733bb4cafa69428319664f239 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 0.
I20260709 12:58:45.758839  1569 leader_election.cc:304] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 12:58:45.759987  2212 raft_consensus.cc:2804] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:58:45.760511  2212 raft_consensus.cc:493] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:58:45.761075  2212 raft_consensus.cc:3060] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:45.762389  1503 catalog_manager.cc:5697] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 0 to 1, leader changed from <none> to 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193). New cstate: current_term: 1 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } }
I20260709 12:58:45.771955  1503 catalog_manager.cc:5697] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 0 to 1, leader changed from <none> to 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193). New cstate: current_term: 1 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } }
I20260709 12:58:45.774330  1796 raft_consensus.cc:359] T 41166f283aea41b0b3f5be60e8234443 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:45.775092  1796 raft_consensus.cc:385] T 41166f283aea41b0b3f5be60e8234443 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:45.775408  1796 raft_consensus.cc:740] T 41166f283aea41b0b3f5be60e8234443 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Initialized, Role: FOLLOWER
I20260709 12:58:45.775174  2101 consensus_queue.cc:1048] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:58:45.776137  1796 consensus_queue.cc:260] T 41166f283aea41b0b3f5be60e8234443 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:45.777319  2236 leader_election.cc:290] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:58:45.780483  1794 ts_tablet_manager.cc:1434] T 1d9ac76dc2c445edb5a25cf9087998bb P 816fadcea8fc4893882f5f9b9f22a03d: Time spent starting tablet: real 0.040s	user 0.005s	sys 0.006s
I20260709 12:58:45.781960  1796 ts_tablet_manager.cc:1434] T 41166f283aea41b0b3f5be60e8234443 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent starting tablet: real 0.027s	user 0.008s	sys 0.000s
I20260709 12:58:45.792949  1794 tablet_bootstrap.cc:492] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d: Bootstrap starting.
I20260709 12:58:45.793707  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 12:58:45.804701  1571 leader_election.cc:343] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:45.806980  1796 tablet_bootstrap.cc:492] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a: Bootstrap starting.
I20260709 12:58:45.814430  1795 tablet_bootstrap.cc:654] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701: Neither blocks nor log segments found. Creating new log.
W20260709 12:58:45.830833  1551 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:58:45.836328  2212 raft_consensus.cc:515] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:45.862715  1796 tablet_bootstrap.cc:654] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:45.871511  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:58:45.873876  1794 tablet_bootstrap.cc:654] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d: Neither blocks nor log segments found. Creating new log.
W20260709 12:58:45.873795  1571 leader_election.cc:343] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:45.872378  2212 leader_election.cc:290] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:58:45.880872  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:58:45.881835  1677 raft_consensus.cc:3060] T a3accbb733bb4cafa69428319664f239 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:45.908903  1677 raft_consensus.cc:2468] T a3accbb733bb4cafa69428319664f239 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 1.
I20260709 12:58:45.910127  1569 leader_election.cc:304] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 12:58:45.911310  2212 raft_consensus.cc:2804] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:58:45.912034  2212 raft_consensus.cc:697] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 12:58:45.912874  2212 consensus_queue.cc:237] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:45.916538  1795 tablet_bootstrap.cc:492] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701: No bootstrap required, opened a new log
I20260709 12:58:45.917268  1795 ts_tablet_manager.cc:1403] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701: Time spent bootstrapping tablet: real 0.291s	user 0.017s	sys 0.012s
I20260709 12:58:45.921028  1795 raft_consensus.cc:359] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:45.921810  1795 raft_consensus.cc:385] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:45.922140  1795 raft_consensus.cc:740] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Initialized, Role: FOLLOWER
I20260709 12:58:45.916538  1796 tablet_bootstrap.cc:492] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a: No bootstrap required, opened a new log
I20260709 12:58:45.923578  1796 ts_tablet_manager.cc:1403] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent bootstrapping tablet: real 0.117s	user 0.009s	sys 0.016s
I20260709 12:58:45.926678  1796 raft_consensus.cc:359] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:45.927353  1796 raft_consensus.cc:385] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:45.927603  1796 raft_consensus.cc:740] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Initialized, Role: FOLLOWER
I20260709 12:58:45.928241  1795 consensus_queue.cc:260] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:45.929782  1794 tablet_bootstrap.cc:492] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d: No bootstrap required, opened a new log
I20260709 12:58:45.930261  1794 ts_tablet_manager.cc:1403] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent bootstrapping tablet: real 0.138s	user 0.023s	sys 0.000s
I20260709 12:58:45.930308  1795 ts_tablet_manager.cc:1434] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701: Time spent starting tablet: real 0.013s	user 0.006s	sys 0.000s
I20260709 12:58:45.928292  1796 consensus_queue.cc:260] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:45.932250  1795 tablet_bootstrap.cc:492] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701: Bootstrap starting.
I20260709 12:58:45.933142  1794 raft_consensus.cc:359] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:45.939034  1794 raft_consensus.cc:385] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:45.939707  1794 raft_consensus.cc:740] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Initialized, Role: FOLLOWER
I20260709 12:58:45.940613  1794 consensus_queue.cc:260] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:45.940442  1503 catalog_manager.cc:5697] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 0 to 1, leader changed from <none> to 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193). New cstate: current_term: 1 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } }
I20260709 12:58:45.937346  1796 ts_tablet_manager.cc:1434] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent starting tablet: real 0.013s	user 0.005s	sys 0.001s
I20260709 12:58:45.944937  1796 tablet_bootstrap.cc:492] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a: Bootstrap starting.
I20260709 12:58:45.943508  1794 ts_tablet_manager.cc:1434] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent starting tablet: real 0.013s	user 0.005s	sys 0.002s
I20260709 12:58:45.947817  1794 tablet_bootstrap.cc:492] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d: Bootstrap starting.
I20260709 12:58:45.949572  1795 tablet_bootstrap.cc:654] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:45.954658  1794 tablet_bootstrap.cc:654] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:45.961086  2142 raft_consensus.cc:493] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:45.959247  1796 tablet_bootstrap.cc:654] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:45.961603  2142 raft_consensus.cc:515] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:45.965659  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:58:45.966084  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:58:45.966360  1677 raft_consensus.cc:2468] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 0.
I20260709 12:58:45.966776  1603 raft_consensus.cc:2468] T 41166f283aea41b0b3f5be60e8234443 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 0.
I20260709 12:58:45.967634  2142 leader_election.cc:290] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:58:45.967662  1718 leader_election.cc:304] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:58:45.972894  2245 raft_consensus.cc:2804] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:58:45.973394  2245 raft_consensus.cc:493] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:58:45.973853  2105 raft_consensus.cc:493] T f6028a708cb344fbb08ab8056921d9e7 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:45.973874  2245 raft_consensus.cc:3060] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:45.974375  2105 raft_consensus.cc:515] T f6028a708cb344fbb08ab8056921d9e7 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:45.982172  2105 leader_election.cc:290] T f6028a708cb344fbb08ab8056921d9e7 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:58:45.983863  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:58:45.974092  2212 raft_consensus.cc:493] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:45.984920  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:58:45.985121  2212 raft_consensus.cc:515] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:45.986056  1677 raft_consensus.cc:2393] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 1: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 12:58:45.987401  2212 leader_election.cc:290] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:58:45.990180  1717 leader_election.cc:304] T f6028a708cb344fbb08ab8056921d9e7 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:58:45.991241  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:58:45.991982  2142 raft_consensus.cc:3060] T f6028a708cb344fbb08ab8056921d9e7 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:45.992477  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 12:58:45.994172  1569 leader_election.cc:343] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:58:46.010515  1571 leader_election.cc:343] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:46.011332  2142 raft_consensus.cc:2749] T f6028a708cb344fbb08ab8056921d9e7 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:58:46.012431  1571 leader_election.cc:304] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 12:58:46.016705  2212 raft_consensus.cc:2749] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:58:46.019338  1794 tablet_bootstrap.cc:492] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d: No bootstrap required, opened a new log
I20260709 12:58:46.022938  1795 tablet_bootstrap.cc:492] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701: No bootstrap required, opened a new log
I20260709 12:58:46.023556  1795 ts_tablet_manager.cc:1403] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701: Time spent bootstrapping tablet: real 0.092s	user 0.030s	sys 0.017s
I20260709 12:58:46.023298  2245 raft_consensus.cc:515] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:46.027580  1795 raft_consensus.cc:359] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:46.028358  1795 raft_consensus.cc:385] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:46.028676  1795 raft_consensus.cc:740] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Initialized, Role: FOLLOWER
I20260709 12:58:46.029363  1795 consensus_queue.cc:260] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:46.029618  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:58:46.030303  1676 raft_consensus.cc:3060] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:46.031087  1795 ts_tablet_manager.cc:1434] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701: Time spent starting tablet: real 0.007s	user 0.004s	sys 0.003s
I20260709 12:58:46.032016  1680 raft_consensus.cc:3060] T 1d9ac76dc2c445edb5a25cf9087998bb P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:46.032153  1795 tablet_bootstrap.cc:492] T f8ab62ffb73246dc9bcf39f7bee51147 P af520b67945d443b901bbeca37cc5701: Bootstrap starting.
I20260709 12:58:46.034482  1794 ts_tablet_manager.cc:1403] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent bootstrapping tablet: real 0.072s	user 0.021s	sys 0.021s
I20260709 12:58:46.038546  1794 raft_consensus.cc:359] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:46.039222  1794 raft_consensus.cc:385] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:46.039490  1794 raft_consensus.cc:740] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Initialized, Role: FOLLOWER
I20260709 12:58:46.040096  1794 consensus_queue.cc:260] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:46.044492  1794 ts_tablet_manager.cc:1434] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent starting tablet: real 0.010s	user 0.008s	sys 0.000s
I20260709 12:58:46.047907  2212 consensus_queue.cc:1048] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:58:46.048596  2078 raft_consensus.cc:493] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:46.049319  2078 raft_consensus.cc:515] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:46.051829  1795 tablet_bootstrap.cc:654] T f8ab62ffb73246dc9bcf39f7bee51147 P af520b67945d443b901bbeca37cc5701: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:46.052448  2078 leader_election.cc:290] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:58:46.057895  1676 raft_consensus.cc:2468] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 1.
I20260709 12:58:46.060936  1718 leader_election.cc:304] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:58:46.061672  1794 tablet_bootstrap.cc:492] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d: Bootstrap starting.
I20260709 12:58:46.061599  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:58:46.063143  2225 raft_consensus.cc:2804] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:58:46.063820  2225 raft_consensus.cc:697] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 1 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 12:58:46.063637  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 12:58:46.066259  1646 leader_election.cc:343] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:46.069579  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
W20260709 12:58:46.071170  1643 leader_election.cc:343] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:46.071673  1643 leader_election.cc:304] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 12:58:46.070745  1603 raft_consensus.cc:3060] T 41166f283aea41b0b3f5be60e8234443 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:46.072060  2211 raft_consensus.cc:493] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:46.072335  2225 consensus_queue.cc:237] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:46.075659  1928 consensus_queue.cc:1048] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:58:46.066514  2245 leader_election.cc:290] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:58:46.072723  2211 raft_consensus.cc:515] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:46.093927  2236 consensus_queue.cc:1048] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:58:46.096494  1796 tablet_bootstrap.cc:492] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a: No bootstrap required, opened a new log
I20260709 12:58:46.078362  2078 raft_consensus.cc:2749] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:58:46.097034  1796 ts_tablet_manager.cc:1403] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent bootstrapping tablet: real 0.152s	user 0.011s	sys 0.003s
I20260709 12:58:46.102782  1796 raft_consensus.cc:359] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:46.103623  1796 raft_consensus.cc:385] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:46.104218  1796 raft_consensus.cc:740] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Initialized, Role: FOLLOWER
I20260709 12:58:46.107398  1796 consensus_queue.cc:260] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:46.112044  1796 ts_tablet_manager.cc:1434] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent starting tablet: real 0.015s	user 0.004s	sys 0.004s
I20260709 12:58:46.113269  1796 tablet_bootstrap.cc:492] T 78cabb03415041ca9c1a5a400e506131 P 83595ad0fd5940c0a2bfc7c61acd969a: Bootstrap starting.
I20260709 12:58:46.121615  1794 tablet_bootstrap.cc:654] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:46.122076  2211 leader_election.cc:290] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:58:46.125422  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:58:46.143059  2236 consensus_queue.cc:1048] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:58:46.147863  1603 raft_consensus.cc:2468] T 41166f283aea41b0b3f5be60e8234443 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 1.
I20260709 12:58:46.148741  1796 tablet_bootstrap.cc:654] T 78cabb03415041ca9c1a5a400e506131 P 83595ad0fd5940c0a2bfc7c61acd969a: Neither blocks nor log segments found. Creating new log.
W20260709 12:58:46.159641  1646 leader_election.cc:343] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:46.183622  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:58:46.184540  1603 raft_consensus.cc:2468] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 0.
I20260709 12:58:46.183266  1502 catalog_manager.cc:5697] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 reported cstate change: term changed from 0 to 1, leader changed from <none> to af520b67945d443b901bbeca37cc5701 (127.0.107.195). New cstate: current_term: 1 leader_uuid: "af520b67945d443b901bbeca37cc5701" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: HEALTHY } } }
I20260709 12:58:46.194394  1795 tablet_bootstrap.cc:492] T f8ab62ffb73246dc9bcf39f7bee51147 P af520b67945d443b901bbeca37cc5701: No bootstrap required, opened a new log
I20260709 12:58:46.195046  1795 ts_tablet_manager.cc:1403] T f8ab62ffb73246dc9bcf39f7bee51147 P af520b67945d443b901bbeca37cc5701: Time spent bootstrapping tablet: real 0.163s	user 0.005s	sys 0.027s
I20260709 12:58:46.229663  1795 raft_consensus.cc:359] T f8ab62ffb73246dc9bcf39f7bee51147 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:46.230569  1795 raft_consensus.cc:385] T f8ab62ffb73246dc9bcf39f7bee51147 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:46.230847  1795 raft_consensus.cc:740] T f8ab62ffb73246dc9bcf39f7bee51147 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Initialized, Role: FOLLOWER
I20260709 12:58:46.231496  1795 consensus_queue.cc:260] T f8ab62ffb73246dc9bcf39f7bee51147 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:46.233781  1795 ts_tablet_manager.cc:1434] T f8ab62ffb73246dc9bcf39f7bee51147 P af520b67945d443b901bbeca37cc5701: Time spent starting tablet: real 0.038s	user 0.004s	sys 0.002s
I20260709 12:58:46.243866  1794 tablet_bootstrap.cc:492] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d: No bootstrap required, opened a new log
I20260709 12:58:46.246351  1794 ts_tablet_manager.cc:1403] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent bootstrapping tablet: real 0.185s	user 0.024s	sys 0.022s
I20260709 12:58:46.226153  1643 leader_election.cc:304] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 12:58:46.249742  2240 raft_consensus.cc:2804] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:58:46.250147  2240 raft_consensus.cc:493] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:58:46.249938  1794 raft_consensus.cc:359] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:46.250563  2240 raft_consensus.cc:3060] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:46.250751  1794 raft_consensus.cc:385] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:46.256724  1795 tablet_bootstrap.cc:492] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701: Bootstrap starting.
I20260709 12:58:46.256906  1794 raft_consensus.cc:740] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Initialized, Role: FOLLOWER
I20260709 12:58:46.257627  1794 consensus_queue.cc:260] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:46.259804  1794 ts_tablet_manager.cc:1434] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent starting tablet: real 0.013s	user 0.006s	sys 0.000s
I20260709 12:58:46.281525  1796 tablet_bootstrap.cc:492] T 78cabb03415041ca9c1a5a400e506131 P 83595ad0fd5940c0a2bfc7c61acd969a: No bootstrap required, opened a new log
I20260709 12:58:46.282275  1796 ts_tablet_manager.cc:1403] T 78cabb03415041ca9c1a5a400e506131 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent bootstrapping tablet: real 0.169s	user 0.026s	sys 0.026s
I20260709 12:58:46.284623  1795 tablet_bootstrap.cc:654] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:46.290727  1796 raft_consensus.cc:359] T 78cabb03415041ca9c1a5a400e506131 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:46.298977  1796 raft_consensus.cc:385] T 78cabb03415041ca9c1a5a400e506131 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:46.299265  1796 raft_consensus.cc:740] T 78cabb03415041ca9c1a5a400e506131 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Initialized, Role: FOLLOWER
I20260709 12:58:46.288815  2240 raft_consensus.cc:515] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:46.300293  1796 consensus_queue.cc:260] T 78cabb03415041ca9c1a5a400e506131 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:46.303480  1796 ts_tablet_manager.cc:1434] T 78cabb03415041ca9c1a5a400e506131 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent starting tablet: real 0.021s	user 0.003s	sys 0.005s
I20260709 12:58:46.303632  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:58:46.304472  1601 raft_consensus.cc:3060] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:46.306263  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:58:46.307086  1748 raft_consensus.cc:3060] T f8ab62ffb73246dc9bcf39f7bee51147 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:58:46.319860  1571 consensus_peers.cc:597] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a -> Peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Couldn't send request to peer af520b67945d443b901bbeca37cc5701. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:58:46.320920  1902 raft_consensus.cc:493] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:46.321444  1902 raft_consensus.cc:515] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:46.304734  1796 tablet_bootstrap.cc:492] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a: Bootstrap starting.
I20260709 12:58:46.323994  1902 leader_election.cc:290] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:58:46.324493  2240 leader_election.cc:290] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:58:46.328969  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:58:46.330129  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:58:46.331089  1603 raft_consensus.cc:2468] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 0.
I20260709 12:58:46.331218  1748 raft_consensus.cc:2468] T f8ab62ffb73246dc9bcf39f7bee51147 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 1.
I20260709 12:58:46.332046  2212 consensus_queue.cc:1048] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260709 12:58:46.333214  1646 leader_election.cc:343] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:46.334049  1646 leader_election.cc:304] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:58:46.336055  1643 leader_election.cc:304] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 12:58:46.336783  2240 raft_consensus.cc:2804] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:58:46.337695  2210 raft_consensus.cc:2804] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:58:46.338166  2210 raft_consensus.cc:493] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:58:46.338660  2210 raft_consensus.cc:3060] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:46.342835  1796 tablet_bootstrap.cc:654] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:46.355391  2240 raft_consensus.cc:697] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 12:58:46.357192  2240 consensus_queue.cc:237] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:46.364816  1795 tablet_bootstrap.cc:492] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701: No bootstrap required, opened a new log
I20260709 12:58:46.365559  1795 ts_tablet_manager.cc:1403] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701: Time spent bootstrapping tablet: real 0.109s	user 0.012s	sys 0.004s
I20260709 12:58:46.372408  1601 raft_consensus.cc:2468] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 1.
I20260709 12:58:46.374789  2210 raft_consensus.cc:515] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:46.377044  1795 raft_consensus.cc:359] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:46.377923  1795 raft_consensus.cc:385] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:46.380386  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:58:46.381582  1601 raft_consensus.cc:3060] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:46.378192  1795 raft_consensus.cc:740] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Initialized, Role: FOLLOWER
I20260709 12:58:46.383390  1795 consensus_queue.cc:260] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:46.386176  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:58:46.387002  1751 raft_consensus.cc:3060] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:46.385875  2210 leader_election.cc:290] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:58:46.390964  1795 ts_tablet_manager.cc:1434] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701: Time spent starting tablet: real 0.025s	user 0.006s	sys 0.005s
I20260709 12:58:46.392043  1795 tablet_bootstrap.cc:492] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701: Bootstrap starting.
I20260709 12:58:46.411882  1751 raft_consensus.cc:2468] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 1.
I20260709 12:58:46.412153  1601 raft_consensus.cc:2468] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 1.
I20260709 12:58:46.419071  1646 leader_election.cc:304] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:58:46.430043  2240 raft_consensus.cc:2804] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:58:46.432830  2240 raft_consensus.cc:697] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 12:58:46.433748  1796 tablet_bootstrap.cc:492] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a: No bootstrap required, opened a new log
I20260709 12:58:46.434300  1796 ts_tablet_manager.cc:1403] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent bootstrapping tablet: real 0.130s	user 0.009s	sys 0.019s
I20260709 12:58:46.433777  2240 consensus_queue.cc:237] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:46.437235  1796 raft_consensus.cc:359] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:46.437937  1796 raft_consensus.cc:385] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:46.439342  1796 raft_consensus.cc:740] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Initialized, Role: FOLLOWER
I20260709 12:58:46.438925  1503 catalog_manager.cc:5697] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d reported cstate change: term changed from 0 to 1, leader changed from <none> to 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194). New cstate: current_term: 1 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } }
I20260709 12:58:46.440021  1796 consensus_queue.cc:260] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:46.439015  1795 tablet_bootstrap.cc:654] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:46.442517  2210 raft_consensus.cc:493] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:46.442997  2210 raft_consensus.cc:515] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:46.445753  2210 leader_election.cc:290] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:58:46.448561  1796 ts_tablet_manager.cc:1434] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent starting tablet: real 0.014s	user 0.005s	sys 0.001s
I20260709 12:58:46.448688  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:58:46.449357  1603 raft_consensus.cc:2468] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 0.
I20260709 12:58:46.451836  1643 leader_election.cc:304] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 12:58:46.452365  1796 tablet_bootstrap.cc:492] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a: Bootstrap starting.
I20260709 12:58:46.453825  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 12:58:46.458334  1646 leader_election.cc:343] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:46.458549  2210 raft_consensus.cc:2804] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:58:46.458904  2210 raft_consensus.cc:493] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:58:46.459228  2210 raft_consensus.cc:3060] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:46.475175  2210 raft_consensus.cc:515] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:46.478331  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:58:46.478989  1603 raft_consensus.cc:3060] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:46.479342  1796 tablet_bootstrap.cc:654] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:46.482200  2210 leader_election.cc:290] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:58:46.481712  1503 catalog_manager.cc:5697] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d reported cstate change: term changed from 0 to 1, leader changed from <none> to 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194). New cstate: current_term: 1 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } }
I20260709 12:58:46.484786  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
W20260709 12:58:46.486341  1646 leader_election.cc:343] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:58:46.486907  1796 tablet_bootstrap.cc:492] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a: No bootstrap required, opened a new log
I20260709 12:58:46.487403  1796 ts_tablet_manager.cc:1403] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent bootstrapping tablet: real 0.036s	user 0.012s	sys 0.001s
I20260709 12:58:46.495309  1796 raft_consensus.cc:359] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:46.496210  1796 raft_consensus.cc:385] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:46.496485  1796 raft_consensus.cc:740] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Initialized, Role: FOLLOWER
I20260709 12:58:46.497294  1796 consensus_queue.cc:260] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:46.498793  1795 tablet_bootstrap.cc:492] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701: No bootstrap required, opened a new log
I20260709 12:58:46.499312  1795 ts_tablet_manager.cc:1403] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701: Time spent bootstrapping tablet: real 0.107s	user 0.010s	sys 0.004s
I20260709 12:58:46.502094  2225 consensus_queue.cc:1048] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:58:46.503327  1796 ts_tablet_manager.cc:1434] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent starting tablet: real 0.016s	user 0.009s	sys 0.001s
I20260709 12:58:46.504523  1795 raft_consensus.cc:359] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:46.506242  1795 raft_consensus.cc:385] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:46.506680  1795 raft_consensus.cc:740] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Initialized, Role: FOLLOWER
I20260709 12:58:46.507689  1795 consensus_queue.cc:260] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:46.508404  1603 raft_consensus.cc:2468] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 1.
I20260709 12:58:46.509630  1643 leader_election.cc:304] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 12:58:46.510372  2240 raft_consensus.cc:2804] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:58:46.511957  2240 raft_consensus.cc:697] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 12:58:46.512930  2240 consensus_queue.cc:237] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:46.521552  2105 raft_consensus.cc:493] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:46.522714  1795 ts_tablet_manager.cc:1434] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701: Time spent starting tablet: real 0.023s	user 0.009s	sys 0.000s
I20260709 12:58:46.522327  2105 raft_consensus.cc:515] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:46.526790  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:58:46.527691  1603 raft_consensus.cc:2468] T 78cabb03415041ca9c1a5a400e506131 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 0.
I20260709 12:58:46.529760  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:58:46.530355  1676 raft_consensus.cc:2468] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 0.
I20260709 12:58:46.532510  1718 leader_election.cc:304] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:58:46.524011  1795 tablet_bootstrap.cc:492] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701: Bootstrap starting.
I20260709 12:58:46.535164  2105 leader_election.cc:290] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:58:46.536759  2142 raft_consensus.cc:2804] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:58:46.550804  2142 raft_consensus.cc:493] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:58:46.551391  2142 raft_consensus.cc:3060] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:46.564529  2224 consensus_queue.cc:1048] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:58:46.579217  1500 catalog_manager.cc:5697] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d reported cstate change: term changed from 0 to 1, leader changed from <none> to 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194). New cstate: current_term: 1 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } }
I20260709 12:58:46.593719  2142 raft_consensus.cc:515] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:46.598994  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:58:46.599912  1602 raft_consensus.cc:3060] T 78cabb03415041ca9c1a5a400e506131 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:46.603058  1795 tablet_bootstrap.cc:654] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:46.616432  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:58:46.617092  1680 raft_consensus.cc:3060] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:46.618014  2142 leader_election.cc:290] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:58:46.628098  1680 raft_consensus.cc:2468] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 1.
I20260709 12:58:46.633395  1718 leader_election.cc:304] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:58:46.633948  1602 raft_consensus.cc:2468] T 78cabb03415041ca9c1a5a400e506131 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 1.
I20260709 12:58:46.634302  2224 raft_consensus.cc:2804] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:58:46.634680  1795 tablet_bootstrap.cc:492] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701: No bootstrap required, opened a new log
I20260709 12:58:46.634899  2224 raft_consensus.cc:697] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 1 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 12:58:46.635349  1795 ts_tablet_manager.cc:1403] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701: Time spent bootstrapping tablet: real 0.112s	user 0.020s	sys 0.000s
I20260709 12:58:46.635915  2224 consensus_queue.cc:237] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:46.639467  1795 raft_consensus.cc:359] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:46.643033  1795 raft_consensus.cc:385] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:46.643807  1795 raft_consensus.cc:740] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Initialized, Role: FOLLOWER
I20260709 12:58:46.644500  1795 consensus_queue.cc:260] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:46.646641  1795 ts_tablet_manager.cc:1434] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701: Time spent starting tablet: real 0.010s	user 0.006s	sys 0.001s
I20260709 12:58:46.647626  1795 tablet_bootstrap.cc:492] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701: Bootstrap starting.
I20260709 12:58:46.653484  1795 tablet_bootstrap.cc:654] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701: Neither blocks nor log segments found. Creating new log.
I20260709 12:58:46.656128  1501 catalog_manager.cc:5697] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 reported cstate change: term changed from 0 to 1, leader changed from <none> to af520b67945d443b901bbeca37cc5701 (127.0.107.195). New cstate: current_term: 1 leader_uuid: "af520b67945d443b901bbeca37cc5701" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: HEALTHY } } }
I20260709 12:58:46.665000  1795 tablet_bootstrap.cc:492] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701: No bootstrap required, opened a new log
I20260709 12:58:46.665515  1795 ts_tablet_manager.cc:1403] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701: Time spent bootstrapping tablet: real 0.018s	user 0.008s	sys 0.008s
I20260709 12:58:46.667876  1795 raft_consensus.cc:359] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:46.668391  1795 raft_consensus.cc:385] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:58:46.668618  1795 raft_consensus.cc:740] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Initialized, Role: FOLLOWER
I20260709 12:58:46.669188  1795 consensus_queue.cc:260] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:46.671387  1795 ts_tablet_manager.cc:1434] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701: Time spent starting tablet: real 0.006s	user 0.006s	sys 0.001s
I20260709 12:58:46.762452  2207 raft_consensus.cc:493] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:46.762995  2207 raft_consensus.cc:515] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:46.764906  2207 leader_election.cc:290] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:58:46.765493  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:58:46.765933  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:58:46.766324  1679 raft_consensus.cc:2468] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 0.
I20260709 12:58:46.766758  1751 raft_consensus.cc:2468] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 0.
I20260709 12:58:46.767613  1569 leader_election.cc:304] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 12:58:46.768359  2207 raft_consensus.cc:2804] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:58:46.768750  2207 raft_consensus.cc:493] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:58:46.769059  2207 raft_consensus.cc:3060] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:46.778921  2207 raft_consensus.cc:515] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:46.782671  2207 leader_election.cc:290] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:58:46.783914  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:58:46.784250  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:58:46.784595  1751 raft_consensus.cc:3060] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:46.784911  1679 raft_consensus.cc:3060] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:46.785866  1748 raft_consensus.cc:3060] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:46.793282  1679 raft_consensus.cc:2468] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 1.
I20260709 12:58:46.794602  1569 leader_election.cc:304] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 12:58:46.793855  1751 raft_consensus.cc:2468] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 1.
I20260709 12:58:46.795482  2207 raft_consensus.cc:2804] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:58:46.796012  2207 raft_consensus.cc:697] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 12:58:46.796900  2207 consensus_queue.cc:237] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:46.810705  1501 catalog_manager.cc:5697] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 0 to 1, leader changed from <none> to 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193). New cstate: current_term: 1 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } }
I20260709 12:58:46.848311  2240 consensus_queue.cc:1048] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:58:46.854336  2240 consensus_queue.cc:1048] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:58:46.876762  2211 consensus_queue.cc:1048] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:58:46.887646  2240 consensus_queue.cc:1048] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:58:46.926527  2142 raft_consensus.cc:493] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:58:46.929107  2142 raft_consensus.cc:515] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:58:46.946461  2142 leader_election.cc:290] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:58:46.952390  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:58:46.960769  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:58:46.962008  1602 raft_consensus.cc:2393] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 1: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 12:58:46.972354  1717 leader_election.cc:304] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:58:46.973486  2142 raft_consensus.cc:3060] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:58:46.988710  2142 raft_consensus.cc:2749] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:58:47.018249  2210 consensus_queue.cc:1048] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:58:47.032722  2210 consensus_queue.cc:1048] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:58:47.185333  2105 consensus_queue.cc:1048] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:58:47.195942   431 file_cache.cc:504] Constructed file cache file cache with capacity 419430
I20260709 12:58:47.208843  2224 consensus_queue.cc:1048] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260709 12:58:47.226104  2289 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:58:47.226835  2287 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:58:47.249121  2292 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:58:47.253353   431 server_base.cc:1061] running on GCE node
I20260709 12:58:47.256819   431 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:58:47.257174   431 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:58:47.259311   431 hybrid_clock.cc:648] HybridClock initialized: now 1783601927259288 us; error 0 us; skew 500 ppm
I20260709 12:58:47.260258   431 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:58:47.264293   431 webserver.cc:533] Webserver started at http://127.0.107.196:44307/ using document root <none> and password file <none>
I20260709 12:58:47.264964   431 fs_manager.cc:362] Metadata directory not provided
I20260709 12:58:47.265233   431 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:58:47.265637   431 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:58:47.267141   431 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskakPCWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783601878574137-431-0/minicluster-data/ts-3-root/instance:
uuid: "3b879799899a499c8d409ee9f1277cd1"
format_stamp: "Formatted at 2026-07-09 12:58:47 on dist-test-slave-nvvq"
I20260709 12:58:47.272783   431 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.005s	sys 0.001s
I20260709 12:58:47.276432  2297 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:58:47.277253   431 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.001s	sys 0.000s
I20260709 12:58:47.277556   431 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskakPCWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783601878574137-431-0/minicluster-data/ts-3-root
uuid: "3b879799899a499c8d409ee9f1277cd1"
format_stamp: "Formatted at 2026-07-09 12:58:47 on dist-test-slave-nvvq"
I20260709 12:58:47.277868   431 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskakPCWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783601878574137-431-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskakPCWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783601878574137-431-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskakPCWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783601878574137-431-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:58:47.293638   431 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:58:47.295332   431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:58:47.296999   431 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:58:47.299705   431 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:58:47.299930   431 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:58:47.300294   431 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:58:47.300558   431 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:58:47.304139  2212 consensus_queue.cc:1048] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:58:47.334746  2207 consensus_queue.cc:1048] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:58:47.476248   431 rpc_server.cc:307] RPC server started. Bound to: 127.0.107.196:46625
I20260709 12:58:47.476352  2361 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.107.196:46625 every 8 connection(s)
I20260709 12:58:47.484660   431 auto_leader_rebalancer-test.cc:384] add a tserver: 3b879799899a499c8d409ee9f1277cd1
I20260709 12:58:47.506392  2362 heartbeater.cc:344] Connected to a master server at 127.0.107.254:33643
I20260709 12:58:47.506953  2362 heartbeater.cc:461] Registering TS with master...
I20260709 12:58:47.507746  2362 heartbeater.cc:507] Master 127.0.107.254:33643 requested a full tablet report, sending...
I20260709 12:58:47.509713  1501 ts_manager.cc:194] Registered new tserver with Master: 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:58:47.511159  1501 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43476
I20260709 12:58:48.079205  1680 consensus_queue.cc:237] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } }
I20260709 12:58:48.087349  1745 raft_consensus.cc:1275] T 592fa647ae2545e09ad5ee25f8cb658e P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:58:48.087982  1601 raft_consensus.cc:1275] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:58:48.091127  2281 consensus_queue.cc:1048] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:58:48.092525  2210 consensus_queue.cc:1048] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:58:48.125039  2211 raft_consensus.cc:2955] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } } }
I20260709 12:58:48.127656  1601 raft_consensus.cc:2955] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } } }
W20260709 12:58:48.131158  1644 consensus_peers.cc:597] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d -> Peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625): Couldn't send request to peer 3b879799899a499c8d409ee9f1277cd1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 592fa647ae2545e09ad5ee25f8cb658e. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:58:48.133497  1748 raft_consensus.cc:2955] T 592fa647ae2545e09ad5ee25f8cb658e P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } } }
I20260709 12:58:48.157725  1680 consensus_queue.cc:237] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } }
W20260709 12:58:48.181978  1644 consensus_peers.cc:597] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d -> Peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625): Couldn't send request to peer 3b879799899a499c8d409ee9f1277cd1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 87b86d46388a41cd90374c203cb27914. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:58:48.187021  1748 raft_consensus.cc:1275] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:58:48.187099  1503 catalog_manager.cc:5697] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: config changed from index -1 to 2, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New cstate: current_term: 1 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } } }
I20260709 12:58:48.189021  1603 raft_consensus.cc:1275] T 87b86d46388a41cd90374c203cb27914 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:58:48.190485  2375 consensus_queue.cc:1048] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:58:48.191991  2371 consensus_queue.cc:1048] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:58:48.207300  2210 raft_consensus.cc:2955] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } } }
I20260709 12:58:48.209215  1603 raft_consensus.cc:2955] T 87b86d46388a41cd90374c203cb27914 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } } }
I20260709 12:58:48.219897  1751 raft_consensus.cc:2955] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } } }
I20260709 12:58:48.247479  1503 catalog_manager.cc:5697] T 87b86d46388a41cd90374c203cb27914 P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: config changed from index -1 to 2, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New cstate: current_term: 1 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } } }
I20260709 12:58:48.254278  1751 consensus_queue.cc:237] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } }
I20260709 12:58:48.272563  1603 raft_consensus.cc:1275] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:58:48.274080  2224 consensus_queue.cc:1048] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:58:48.284758  1680 raft_consensus.cc:1275] T 77d66b92688a42e3ad231959a644db56 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:58:48.284359  2245 raft_consensus.cc:2955] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } } }
I20260709 12:58:48.286649  1603 raft_consensus.cc:2955] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } } }
I20260709 12:58:48.294250  2224 consensus_queue.cc:1048] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:58:48.300997  1680 raft_consensus.cc:2955] T 77d66b92688a42e3ad231959a644db56 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } } }
W20260709 12:58:48.306778  1718 consensus_peers.cc:597] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 -> Peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625): Couldn't send request to peer 3b879799899a499c8d409ee9f1277cd1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 77d66b92688a42e3ad231959a644db56. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:58:48.311869  1500 catalog_manager.cc:5697] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 reported cstate change: config changed from index -1 to 2, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New cstate: current_term: 1 leader_uuid: "af520b67945d443b901bbeca37cc5701" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260709 12:58:48.314278  1751 consensus_queue.cc:237] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } }
I20260709 12:58:48.357658  1679 raft_consensus.cc:1275] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:58:48.358464  1603 raft_consensus.cc:1275] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:58:48.359217  2245 consensus_queue.cc:1048] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:58:48.360966  2224 consensus_queue.cc:1048] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260709 12:58:48.362063  1718 consensus_peers.cc:597] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 -> Peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625): Couldn't send request to peer 3b879799899a499c8d409ee9f1277cd1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e527450b1d6c4cb79e9f6d3c87d4ad23. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:58:48.370679  2142 raft_consensus.cc:2955] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } } }
I20260709 12:58:48.372797  1603 raft_consensus.cc:2955] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } } }
I20260709 12:58:48.395866  1679 raft_consensus.cc:2955] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } } }
I20260709 12:58:48.412441  1503 catalog_manager.cc:5697] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: config changed from index -1 to 2, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New cstate: current_term: 1 leader_uuid: "af520b67945d443b901bbeca37cc5701" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } } }
I20260709 12:58:48.513895  2362 heartbeater.cc:499] Master 127.0.107.254:33643 was elected leader, sending a full tablet report...
I20260709 12:58:48.555445  2393 ts_tablet_manager.cc:933] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1: Initiating tablet copy from peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:58:48.560882  2393 tablet_copy_client.cc:323] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Beginning tablet copy session from remote peer at address 127.0.107.194:46261
I20260709 12:58:48.577471  1690 tablet_copy_service.cc:140] P 816fadcea8fc4893882f5f9b9f22a03d: Received BeginTabletCopySession request for tablet 592fa647ae2545e09ad5ee25f8cb658e from peer 3b879799899a499c8d409ee9f1277cd1 ({username='slave'} at 127.0.0.1:57092)
I20260709 12:58:48.578209  1690 tablet_copy_service.cc:161] P 816fadcea8fc4893882f5f9b9f22a03d: Beginning new tablet copy session on tablet 592fa647ae2545e09ad5ee25f8cb658e from peer 3b879799899a499c8d409ee9f1277cd1 at {username='slave'} at 127.0.0.1:57092: session id = 3b879799899a499c8d409ee9f1277cd1-592fa647ae2545e09ad5ee25f8cb658e
I20260709 12:58:48.585417  1690 tablet_copy_source_session.cc:215] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:58:48.589851  2393 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 592fa647ae2545e09ad5ee25f8cb658e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:48.607198  2393 tablet_copy_client.cc:806] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Starting download of 0 data blocks...
I20260709 12:58:48.607950  2393 tablet_copy_client.cc:670] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Starting download of 1 WAL segments...
I20260709 12:58:48.612820  2393 tablet_copy_client.cc:538] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:58:48.623039  2393 tablet_bootstrap.cc:492] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1: Bootstrap starting.
I20260709 12:58:48.656574  2393 tablet_bootstrap.cc:492] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:58:48.657797  2393 tablet_bootstrap.cc:492] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1: Bootstrap complete.
I20260709 12:58:48.658618  2393 ts_tablet_manager.cc:1403] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1: Time spent bootstrapping tablet: real 0.036s	user 0.019s	sys 0.015s
I20260709 12:58:48.662129  2393 raft_consensus.cc:359] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } }
I20260709 12:58:48.663125  2393 raft_consensus.cc:740] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Initialized, Role: LEARNER
I20260709 12:58:48.663852  2393 consensus_queue.cc:260] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } }
I20260709 12:58:48.667826  2393 ts_tablet_manager.cc:1434] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1: Time spent starting tablet: real 0.009s	user 0.006s	sys 0.001s
I20260709 12:58:48.670805  1690 tablet_copy_service.cc:342] P 816fadcea8fc4893882f5f9b9f22a03d: Request end of tablet copy session 3b879799899a499c8d409ee9f1277cd1-592fa647ae2545e09ad5ee25f8cb658e received from {username='slave'} at 127.0.0.1:57092
I20260709 12:58:48.671396  1690 tablet_copy_service.cc:434] P 816fadcea8fc4893882f5f9b9f22a03d: ending tablet copy session 3b879799899a499c8d409ee9f1277cd1-592fa647ae2545e09ad5ee25f8cb658e on tablet 592fa647ae2545e09ad5ee25f8cb658e with peer 3b879799899a499c8d409ee9f1277cd1
I20260709 12:58:48.723546  2393 ts_tablet_manager.cc:933] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1: Initiating tablet copy from peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:58:48.727286  2393 tablet_copy_client.cc:323] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Beginning tablet copy session from remote peer at address 127.0.107.195:40693
I20260709 12:58:48.742527  1764 tablet_copy_service.cc:140] P af520b67945d443b901bbeca37cc5701: Received BeginTabletCopySession request for tablet 77d66b92688a42e3ad231959a644db56 from peer 3b879799899a499c8d409ee9f1277cd1 ({username='slave'} at 127.0.0.1:43156)
I20260709 12:58:48.743211  1764 tablet_copy_service.cc:161] P af520b67945d443b901bbeca37cc5701: Beginning new tablet copy session on tablet 77d66b92688a42e3ad231959a644db56 from peer 3b879799899a499c8d409ee9f1277cd1 at {username='slave'} at 127.0.0.1:43156: session id = 3b879799899a499c8d409ee9f1277cd1-77d66b92688a42e3ad231959a644db56
I20260709 12:58:48.752753  1764 tablet_copy_source_session.cc:215] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:58:48.755236  2399 ts_tablet_manager.cc:933] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1: Initiating tablet copy from peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:58:48.760429  2393 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 77d66b92688a42e3ad231959a644db56. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:48.763607  2399 tablet_copy_client.cc:323] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Beginning tablet copy session from remote peer at address 127.0.107.194:46261
I20260709 12:58:48.766039  1690 tablet_copy_service.cc:140] P 816fadcea8fc4893882f5f9b9f22a03d: Received BeginTabletCopySession request for tablet 87b86d46388a41cd90374c203cb27914 from peer 3b879799899a499c8d409ee9f1277cd1 ({username='slave'} at 127.0.0.1:57092)
I20260709 12:58:48.766884  1690 tablet_copy_service.cc:161] P 816fadcea8fc4893882f5f9b9f22a03d: Beginning new tablet copy session on tablet 87b86d46388a41cd90374c203cb27914 from peer 3b879799899a499c8d409ee9f1277cd1 at {username='slave'} at 127.0.0.1:57092: session id = 3b879799899a499c8d409ee9f1277cd1-87b86d46388a41cd90374c203cb27914
I20260709 12:58:48.785157  1690 tablet_copy_source_session.cc:215] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:58:48.796463  2393 tablet_copy_client.cc:806] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Starting download of 0 data blocks...
I20260709 12:58:48.797001  2399 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 87b86d46388a41cd90374c203cb27914. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:48.797878  2393 tablet_copy_client.cc:670] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Starting download of 1 WAL segments...
I20260709 12:58:48.806726  2393 tablet_copy_client.cc:538] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:58:48.836930  2401 ts_tablet_manager.cc:933] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1: Initiating tablet copy from peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:58:48.839162  2393 tablet_bootstrap.cc:492] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1: Bootstrap starting.
I20260709 12:58:48.848446  2401 tablet_copy_client.cc:323] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Beginning tablet copy session from remote peer at address 127.0.107.195:40693
I20260709 12:58:48.853277  2399 tablet_copy_client.cc:806] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Starting download of 0 data blocks...
I20260709 12:58:48.854872  2399 tablet_copy_client.cc:670] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Starting download of 1 WAL segments...
I20260709 12:58:48.854852  1764 tablet_copy_service.cc:140] P af520b67945d443b901bbeca37cc5701: Received BeginTabletCopySession request for tablet e527450b1d6c4cb79e9f6d3c87d4ad23 from peer 3b879799899a499c8d409ee9f1277cd1 ({username='slave'} at 127.0.0.1:43156)
I20260709 12:58:48.855948  1764 tablet_copy_service.cc:161] P af520b67945d443b901bbeca37cc5701: Beginning new tablet copy session on tablet e527450b1d6c4cb79e9f6d3c87d4ad23 from peer 3b879799899a499c8d409ee9f1277cd1 at {username='slave'} at 127.0.0.1:43156: session id = 3b879799899a499c8d409ee9f1277cd1-e527450b1d6c4cb79e9f6d3c87d4ad23
I20260709 12:58:48.872309  1764 tablet_copy_source_session.cc:215] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:58:48.876464  2399 tablet_copy_client.cc:538] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:58:48.878964  2401 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e527450b1d6c4cb79e9f6d3c87d4ad23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:48.906890  2399 tablet_bootstrap.cc:492] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1: Bootstrap starting.
I20260709 12:58:48.930045  2401 tablet_copy_client.cc:806] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Starting download of 0 data blocks...
I20260709 12:58:48.931537  2401 tablet_copy_client.cc:670] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Starting download of 1 WAL segments...
I20260709 12:58:48.943734  2401 tablet_copy_client.cc:538] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:58:48.948462  2393 tablet_bootstrap.cc:492] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:58:48.949947  2393 tablet_bootstrap.cc:492] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1: Bootstrap complete.
I20260709 12:58:48.951928  2393 ts_tablet_manager.cc:1403] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1: Time spent bootstrapping tablet: real 0.113s	user 0.023s	sys 0.017s
I20260709 12:58:48.958528  2399 tablet_bootstrap.cc:492] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:58:48.957805  2393 raft_consensus.cc:359] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } }
I20260709 12:58:48.959415  2393 raft_consensus.cc:740] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Initialized, Role: LEARNER
I20260709 12:58:48.959985  2399 tablet_bootstrap.cc:492] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1: Bootstrap complete.
I20260709 12:58:48.962913  2399 ts_tablet_manager.cc:1403] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1: Time spent bootstrapping tablet: real 0.057s	user 0.013s	sys 0.032s
I20260709 12:58:48.968652  2399 raft_consensus.cc:359] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } }
I20260709 12:58:48.969679  2399 raft_consensus.cc:740] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Initialized, Role: LEARNER
I20260709 12:58:48.970651  2399 consensus_queue.cc:260] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } }
I20260709 12:58:48.960639  2393 consensus_queue.cc:260] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } }
I20260709 12:58:48.980213  2399 ts_tablet_manager.cc:1434] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1: Time spent starting tablet: real 0.017s	user 0.005s	sys 0.005s
I20260709 12:58:48.983007  1690 tablet_copy_service.cc:342] P 816fadcea8fc4893882f5f9b9f22a03d: Request end of tablet copy session 3b879799899a499c8d409ee9f1277cd1-87b86d46388a41cd90374c203cb27914 received from {username='slave'} at 127.0.0.1:57092
I20260709 12:58:48.983738  1690 tablet_copy_service.cc:434] P 816fadcea8fc4893882f5f9b9f22a03d: ending tablet copy session 3b879799899a499c8d409ee9f1277cd1-87b86d46388a41cd90374c203cb27914 on tablet 87b86d46388a41cd90374c203cb27914 with peer 3b879799899a499c8d409ee9f1277cd1
I20260709 12:58:49.000551  2393 ts_tablet_manager.cc:1434] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1: Time spent starting tablet: real 0.048s	user 0.004s	sys 0.026s
I20260709 12:58:49.003157  1764 tablet_copy_service.cc:342] P af520b67945d443b901bbeca37cc5701: Request end of tablet copy session 3b879799899a499c8d409ee9f1277cd1-77d66b92688a42e3ad231959a644db56 received from {username='slave'} at 127.0.0.1:43156
I20260709 12:58:49.003784  1764 tablet_copy_service.cc:434] P af520b67945d443b901bbeca37cc5701: ending tablet copy session 3b879799899a499c8d409ee9f1277cd1-77d66b92688a42e3ad231959a644db56 on tablet 77d66b92688a42e3ad231959a644db56 with peer 3b879799899a499c8d409ee9f1277cd1
I20260709 12:58:49.017688  2401 tablet_bootstrap.cc:492] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1: Bootstrap starting.
I20260709 12:58:49.048717  2401 tablet_bootstrap.cc:492] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:58:49.049993  2401 tablet_bootstrap.cc:492] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1: Bootstrap complete.
I20260709 12:58:49.050959  2401 ts_tablet_manager.cc:1403] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1: Time spent bootstrapping tablet: real 0.034s	user 0.019s	sys 0.015s
I20260709 12:58:49.056133  2401 raft_consensus.cc:359] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } }
I20260709 12:58:49.057163  2401 raft_consensus.cc:740] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Initialized, Role: LEARNER
I20260709 12:58:49.057906  2401 consensus_queue.cc:260] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } }
I20260709 12:58:49.063273  2401 ts_tablet_manager.cc:1434] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1: Time spent starting tablet: real 0.012s	user 0.008s	sys 0.002s
I20260709 12:58:49.065857  1764 tablet_copy_service.cc:342] P af520b67945d443b901bbeca37cc5701: Request end of tablet copy session 3b879799899a499c8d409ee9f1277cd1-e527450b1d6c4cb79e9f6d3c87d4ad23 received from {username='slave'} at 127.0.0.1:43156
I20260709 12:58:49.066401  1764 tablet_copy_service.cc:434] P af520b67945d443b901bbeca37cc5701: ending tablet copy session 3b879799899a499c8d409ee9f1277cd1-e527450b1d6c4cb79e9f6d3c87d4ad23 on tablet e527450b1d6c4cb79e9f6d3c87d4ad23 with peer 3b879799899a499c8d409ee9f1277cd1
I20260709 12:58:49.146613  2337 raft_consensus.cc:1217] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:58:49.218918  2337 raft_consensus.cc:1217] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:58:49.238642  2337 raft_consensus.cc:1217] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:58:49.402763  2337 raft_consensus.cc:1217] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:58:49.621013  2210 raft_consensus.cc:1064] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d: attempting to promote NON_VOTER 3b879799899a499c8d409ee9f1277cd1 to VOTER
I20260709 12:58:49.623878  2210 consensus_queue.cc:237] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:58:49.631726  1751 raft_consensus.cc:1275] T 592fa647ae2545e09ad5ee25f8cb658e P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:58:49.631923  2337 raft_consensus.cc:1275] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 1 LEARNER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:58:49.632609  1601 raft_consensus.cc:1275] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:58:49.633791  2407 consensus_queue.cc:1048] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:58:49.634724  2210 consensus_queue.cc:1048] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:58:49.635663  2371 consensus_queue.cc:1048] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:58:49.653304  2210 raft_consensus.cc:2955] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:58:49.655351  2337 raft_consensus.cc:2955] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:58:49.655602  1751 raft_consensus.cc:2955] T 592fa647ae2545e09ad5ee25f8cb658e P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:58:49.662743  2371 raft_consensus.cc:1064] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d: attempting to promote NON_VOTER 3b879799899a499c8d409ee9f1277cd1 to VOTER
I20260709 12:58:49.670800  1601 raft_consensus.cc:2955] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:58:49.674506  2371 consensus_queue.cc:237] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:58:49.680429  1500 catalog_manager.cc:5697] T 592fa647ae2545e09ad5ee25f8cb658e P af520b67945d443b901bbeca37cc5701 reported cstate change: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:58:49.771302  2337 raft_consensus.cc:1275] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 1 LEARNER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:58:49.773916  2371 consensus_queue.cc:1048] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:58:49.778055  1751 raft_consensus.cc:1275] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:58:49.802301  2371 consensus_queue.cc:1048] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:58:49.817945  1602 raft_consensus.cc:1275] T 87b86d46388a41cd90374c203cb27914 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:58:49.863729  2409 raft_consensus.cc:1064] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701: attempting to promote NON_VOTER 3b879799899a499c8d409ee9f1277cd1 to VOTER
I20260709 12:58:49.890407  2409 consensus_queue.cc:237] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:58:49.936373  2418 consensus_queue.cc:1048] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:58:49.959095  1680 consensus_queue.cc:237] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:58:49.969012  2336 raft_consensus.cc:1275] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:58:49.971117  2418 consensus_queue.cc:1048] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:58:49.974771  2337 raft_consensus.cc:1275] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 1 LEARNER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:58:49.975466  2418 raft_consensus.cc:2955] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:58:49.979499  2142 raft_consensus.cc:1064] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701: attempting to promote NON_VOTER 3b879799899a499c8d409ee9f1277cd1 to VOTER
I20260709 12:58:49.980636  1747 raft_consensus.cc:2955] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:58:49.985489  1677 raft_consensus.cc:1275] T 77d66b92688a42e3ad231959a644db56 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:58:49.988641  2142 consensus_queue.cc:237] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:58:50.011302  2392 consensus_queue.cc:1048] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:58:50.018355  2407 raft_consensus.cc:2955] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER af520b67945d443b901bbeca37cc5701 (127.0.107.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:58:50.020212  2337 raft_consensus.cc:2955] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER af520b67945d443b901bbeca37cc5701 (127.0.107.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:58:50.021796  2392 consensus_queue.cc:1048] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:58:50.029990  1488 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 592fa647ae2545e09ad5ee25f8cb658e with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:58:50.037927  1502 catalog_manager.cc:5697] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 reported cstate change: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:58:50.041388  1603 raft_consensus.cc:1275] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:58:50.064934  1503 catalog_manager.cc:5697] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 reported cstate change: config changed from index 3 to 4, VOTER af520b67945d443b901bbeca37cc5701 (127.0.107.195) evicted. New cstate: current_term: 1 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:58:50.067354  1601 raft_consensus.cc:1275] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:58:50.069407  2337 raft_consensus.cc:2955] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:58:50.071738  2336 raft_consensus.cc:1275] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 1 LEARNER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:58:50.080371  1677 raft_consensus.cc:1275] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:58:50.088294  2142 consensus_queue.cc:1048] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:58:50.099916  1740 tablet_service.cc:1558] Processing DeleteTablet for tablet 592fa647ae2545e09ad5ee25f8cb658e with delete_type TABLET_DATA_TOMBSTONED (TS af520b67945d443b901bbeca37cc5701 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:55838
I20260709 12:58:50.120342  2142 consensus_queue.cc:1048] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20260709 12:58:50.147716  2428 log.cc:927] Time spent T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.086s	user 0.001s	sys 0.000s
I20260709 12:58:50.156500  1602 raft_consensus.cc:1275] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:58:50.157919  2434 tablet_replica.cc:333] T 592fa647ae2545e09ad5ee25f8cb658e P af520b67945d443b901bbeca37cc5701: stopping tablet replica
I20260709 12:58:50.161549  2391 consensus_queue.cc:1048] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:58:50.167757  2434 raft_consensus.cc:2243] T 592fa647ae2545e09ad5ee25f8cb658e P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:58:50.172232  2418 consensus_queue.cc:1048] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:58:50.176931  2434 raft_consensus.cc:2272] T 592fa647ae2545e09ad5ee25f8cb658e P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:58:50.180452  2245 consensus_queue.cc:1048] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:58:50.184677  2409 raft_consensus.cc:2955] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:58:50.189189  1677 raft_consensus.cc:2955] T 77d66b92688a42e3ad231959a644db56 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:58:50.189877  2142 raft_consensus.cc:2955] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:58:50.190673  1679 consensus_queue.cc:237] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:58:50.205017  2337 raft_consensus.cc:1275] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:58:50.199529  2336 raft_consensus.cc:2955] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:58:50.206775  2418 consensus_queue.cc:1048] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:58:50.207937  1748 raft_consensus.cc:1275] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:58:50.208168  2434 ts_tablet_manager.cc:1916] T 592fa647ae2545e09ad5ee25f8cb658e P af520b67945d443b901bbeca37cc5701: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:58:50.209543  2407 consensus_queue.cc:1048] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:58:50.218390  2418 raft_consensus.cc:2955] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:58:50.220162  1748 raft_consensus.cc:2955] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:58:50.226058  2335 raft_consensus.cc:2955] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:58:50.234346  2434 ts_tablet_manager.cc:1929] T 592fa647ae2545e09ad5ee25f8cb658e P af520b67945d443b901bbeca37cc5701: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:58:50.235257  2434 log.cc:1199] T 592fa647ae2545e09ad5ee25f8cb658e P af520b67945d443b901bbeca37cc5701: Deleting WAL directory at /tmp/dist-test-taskakPCWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783601878574137-431-0/minicluster-data/ts-2-root/wals/592fa647ae2545e09ad5ee25f8cb658e
I20260709 12:58:50.237560  1490 catalog_manager.cc:5028] TS af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): tablet 592fa647ae2545e09ad5ee25f8cb658e (table test-workload [id=75d77b57b3f34786b9c215c4f11bd727]) successfully deleted
I20260709 12:58:50.240662  1503 catalog_manager.cc:5697] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 reported cstate change: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "af520b67945d443b901bbeca37cc5701" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:58:50.247022  1602 raft_consensus.cc:2955] T 87b86d46388a41cd90374c203cb27914 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:58:50.248178  1603 raft_consensus.cc:2955] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER af520b67945d443b901bbeca37cc5701 (127.0.107.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:58:50.260593  1679 raft_consensus.cc:2955] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:58:50.262087  1503 catalog_manager.cc:5697] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 reported cstate change: config changed from index 3 to 4, VOTER 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) evicted. New cstate: current_term: 1 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:58:50.267529  1501 catalog_manager.cc:5697] T 77d66b92688a42e3ad231959a644db56 P 816fadcea8fc4893882f5f9b9f22a03d reported cstate change: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "af520b67945d443b901bbeca37cc5701" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:58:50.292893  2335 raft_consensus.cc:2955] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:58:50.297096  1592 tablet_service.cc:1558] Processing DeleteTablet for tablet 87b86d46388a41cd90374c203cb27914 with delete_type TABLET_DATA_TOMBSTONED (TS 83595ad0fd5940c0a2bfc7c61acd969a not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:40138
I20260709 12:58:50.311399  1488 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 87b86d46388a41cd90374c203cb27914 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:58:50.351912  2439 tablet_replica.cc:333] T 87b86d46388a41cd90374c203cb27914 P 83595ad0fd5940c0a2bfc7c61acd969a: stopping tablet replica
I20260709 12:58:50.353345  2439 raft_consensus.cc:2243] T 87b86d46388a41cd90374c203cb27914 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:58:50.354250  2439 raft_consensus.cc:2272] T 87b86d46388a41cd90374c203cb27914 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:58:50.358741  2439 ts_tablet_manager.cc:1916] T 87b86d46388a41cd90374c203cb27914 P 83595ad0fd5940c0a2bfc7c61acd969a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:58:50.408670  1603 raft_consensus.cc:2955] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:58:50.409132  1600 raft_consensus.cc:2955] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:58:50.416321  1747 consensus_queue.cc:237] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:58:50.433576  2335 raft_consensus.cc:1275] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:58:50.435269  2437 consensus_queue.cc:1048] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:58:50.442960  2439 ts_tablet_manager.cc:1929] T 87b86d46388a41cd90374c203cb27914 P 83595ad0fd5940c0a2bfc7c61acd969a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:58:50.443079  2245 raft_consensus.cc:2955] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:58:50.443763  2439 log.cc:1199] T 87b86d46388a41cd90374c203cb27914 P 83595ad0fd5940c0a2bfc7c61acd969a: Deleting WAL directory at /tmp/dist-test-taskakPCWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783601878574137-431-0/minicluster-data/ts-0-root/wals/87b86d46388a41cd90374c203cb27914
I20260709 12:58:50.452294  2335 raft_consensus.cc:2955] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:58:50.453287  1487 catalog_manager.cc:5028] TS 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): tablet 87b86d46388a41cd90374c203cb27914 (table test-workload [id=75d77b57b3f34786b9c215c4f11bd727]) successfully deleted
I20260709 12:58:50.463069  1490 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet e527450b1d6c4cb79e9f6d3c87d4ad23 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:58:50.466451  1603 raft_consensus.cc:1275] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:58:50.466459  1747 consensus_queue.cc:237] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:58:50.469612  1501 catalog_manager.cc:5697] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 reported cstate change: config changed from index 3 to 4, VOTER 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194) evicted. New cstate: current_term: 1 leader_uuid: "af520b67945d443b901bbeca37cc5701" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:58:50.471989  2436 consensus_queue.cc:1048] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:58:50.478353  1600 raft_consensus.cc:2955] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:58:50.483825  1665 tablet_service.cc:1558] Processing DeleteTablet for tablet e527450b1d6c4cb79e9f6d3c87d4ad23 with delete_type TABLET_DATA_TOMBSTONED (TS 816fadcea8fc4893882f5f9b9f22a03d not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:52760
I20260709 12:58:50.484759  1603 raft_consensus.cc:1275] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:58:50.485738  2335 raft_consensus.cc:1275] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:58:50.487057  2245 consensus_queue.cc:1048] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:58:50.488010  2142 consensus_queue.cc:1048] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:58:50.497831  2442 tablet_replica.cc:333] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 816fadcea8fc4893882f5f9b9f22a03d: stopping tablet replica
I20260709 12:58:50.499233  2442 raft_consensus.cc:2243] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:58:50.500161  2442 raft_consensus.cc:2272] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:58:50.503144  2436 raft_consensus.cc:2955] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:58:50.504961  2335 raft_consensus.cc:2955] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:58:50.515111  1490 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 77d66b92688a42e3ad231959a644db56 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:58:50.515358  1603 raft_consensus.cc:2955] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:58:50.516471  2442 ts_tablet_manager.cc:1916] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 816fadcea8fc4893882f5f9b9f22a03d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:58:50.536924  1501 catalog_manager.cc:5697] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 reported cstate change: config changed from index 3 to 4, VOTER 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194) evicted. New cstate: current_term: 1 leader_uuid: "af520b67945d443b901bbeca37cc5701" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:58:50.557766  2442 ts_tablet_manager.cc:1929] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 816fadcea8fc4893882f5f9b9f22a03d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:58:50.559835  2442 log.cc:1199] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 816fadcea8fc4893882f5f9b9f22a03d: Deleting WAL directory at /tmp/dist-test-taskakPCWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783601878574137-431-0/minicluster-data/ts-1-root/wals/e527450b1d6c4cb79e9f6d3c87d4ad23
I20260709 12:58:50.566689  1488 catalog_manager.cc:5028] TS 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): tablet e527450b1d6c4cb79e9f6d3c87d4ad23 (table test-workload [id=75d77b57b3f34786b9c215c4f11bd727]) successfully deleted
I20260709 12:58:50.566140  1665 tablet_service.cc:1558] Processing DeleteTablet for tablet 77d66b92688a42e3ad231959a644db56 with delete_type TABLET_DATA_TOMBSTONED (TS 816fadcea8fc4893882f5f9b9f22a03d not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:52760
I20260709 12:58:50.568326  2442 tablet_replica.cc:333] T 77d66b92688a42e3ad231959a644db56 P 816fadcea8fc4893882f5f9b9f22a03d: stopping tablet replica
I20260709 12:58:50.569413  2442 raft_consensus.cc:2243] T 77d66b92688a42e3ad231959a644db56 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:58:50.570585  2442 raft_consensus.cc:2272] T 77d66b92688a42e3ad231959a644db56 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:58:50.579464  2442 ts_tablet_manager.cc:1916] T 77d66b92688a42e3ad231959a644db56 P 816fadcea8fc4893882f5f9b9f22a03d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:58:50.593276  2442 ts_tablet_manager.cc:1929] T 77d66b92688a42e3ad231959a644db56 P 816fadcea8fc4893882f5f9b9f22a03d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:58:50.593590  2442 log.cc:1199] T 77d66b92688a42e3ad231959a644db56 P 816fadcea8fc4893882f5f9b9f22a03d: Deleting WAL directory at /tmp/dist-test-taskakPCWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783601878574137-431-0/minicluster-data/ts-1-root/wals/77d66b92688a42e3ad231959a644db56
I20260709 12:58:50.595074  1488 catalog_manager.cc:5028] TS 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): tablet 77d66b92688a42e3ad231959a644db56 (table test-workload [id=75d77b57b3f34786b9c215c4f11bd727]) successfully deleted
I20260709 12:58:51.654286  1602 consensus_queue.cc:237] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } }
I20260709 12:58:51.665115  1677 raft_consensus.cc:1275] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:58:51.666524  1745 raft_consensus.cc:1275] T c81785db7d654e4288fcaeacf4db2b49 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:58:51.667284  2412 consensus_queue.cc:1048] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:58:51.676860  2412 raft_consensus.cc:2955] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } } }
I20260709 12:58:51.680609  2207 consensus_queue.cc:1048] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:58:51.686497  1745 raft_consensus.cc:2955] T c81785db7d654e4288fcaeacf4db2b49 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } } }
I20260709 12:58:51.687536  1677 raft_consensus.cc:2955] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } } }
I20260709 12:58:51.707459  1502 catalog_manager.cc:5697] T c81785db7d654e4288fcaeacf4db2b49 P af520b67945d443b901bbeca37cc5701 reported cstate change: config changed from index -1 to 2, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New cstate: current_term: 1 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } } }
I20260709 12:58:51.708276  1677 consensus_queue.cc:237] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } }
W20260709 12:58:51.702952  1569 consensus_peers.cc:597] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a -> Peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625): Couldn't send request to peer 3b879799899a499c8d409ee9f1277cd1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c81785db7d654e4288fcaeacf4db2b49. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:58:51.719388  1602 raft_consensus.cc:1275] T a40e423ed583480280352b261913276c P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:58:51.719851  1745 raft_consensus.cc:1275] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:58:51.720907  2443 consensus_queue.cc:1048] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260709 12:58:51.722028  1644 consensus_peers.cc:597] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d -> Peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625): Couldn't send request to peer 3b879799899a499c8d409ee9f1277cd1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a40e423ed583480280352b261913276c. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:58:51.722548  2418 consensus_queue.cc:1048] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:58:51.746399  2418 raft_consensus.cc:2955] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } } }
I20260709 12:58:51.751752  1602 raft_consensus.cc:2955] T a40e423ed583480280352b261913276c P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } } }
I20260709 12:58:51.780946  1677 consensus_queue.cc:237] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } }
I20260709 12:58:51.791517  1752 raft_consensus.cc:2955] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } } }
I20260709 12:58:51.801011  1503 catalog_manager.cc:5697] T a40e423ed583480280352b261913276c P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: config changed from index -1 to 2, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New cstate: current_term: 1 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } } }
W20260709 12:58:51.823024  1644 consensus_peers.cc:597] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d -> Peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625): Couldn't send request to peer 3b879799899a499c8d409ee9f1277cd1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dd3f391a609743de886c525de483fc29. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:58:51.825119  1748 raft_consensus.cc:1275] T dd3f391a609743de886c525de483fc29 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:58:51.825327  1603 raft_consensus.cc:1275] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:58:51.826786  2460 consensus_queue.cc:1048] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:58:51.828075  2459 consensus_queue.cc:1048] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:58:51.843889  2459 raft_consensus.cc:2955] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } } }
I20260709 12:58:51.846402  1748 raft_consensus.cc:2955] T dd3f391a609743de886c525de483fc29 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } } }
I20260709 12:58:51.849656  1603 raft_consensus.cc:2955] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } } }
I20260709 12:58:51.860390  1745 consensus_queue.cc:237] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } }
I20260709 12:58:51.867154  1503 catalog_manager.cc:5697] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d reported cstate change: config changed from index -1 to 2, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New cstate: current_term: 1 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260709 12:58:51.879868  1603 raft_consensus.cc:1275] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:58:51.880918  1677 raft_consensus.cc:1275] T c6ab3300be1e4a59b64b14d50da193ca P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:58:51.882390  2436 consensus_queue.cc:1048] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260709 12:58:51.883460  1718 consensus_peers.cc:597] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 -> Peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625): Couldn't send request to peer 3b879799899a499c8d409ee9f1277cd1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c6ab3300be1e4a59b64b14d50da193ca. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:58:51.884433  2245 consensus_queue.cc:1048] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:58:51.904703  2245 raft_consensus.cc:2955] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } } }
I20260709 12:58:51.909008  1603 raft_consensus.cc:2955] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } } }
I20260709 12:58:51.906221  1677 raft_consensus.cc:2955] T c6ab3300be1e4a59b64b14d50da193ca P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } } }
I20260709 12:58:51.924007  1503 catalog_manager.cc:5697] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 reported cstate change: config changed from index -1 to 2, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New cstate: current_term: 1 leader_uuid: "af520b67945d443b901bbeca37cc5701" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260709 12:58:52.275184  2470 ts_tablet_manager.cc:933] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1: Initiating tablet copy from peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445)
I20260709 12:58:52.280084  2470 tablet_copy_client.cc:323] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Beginning tablet copy session from remote peer at address 127.0.107.193:46445
I20260709 12:58:52.311218  1615 tablet_copy_service.cc:140] P 83595ad0fd5940c0a2bfc7c61acd969a: Received BeginTabletCopySession request for tablet c81785db7d654e4288fcaeacf4db2b49 from peer 3b879799899a499c8d409ee9f1277cd1 ({username='slave'} at 127.0.0.1:36462)
I20260709 12:58:52.311900  1615 tablet_copy_service.cc:161] P 83595ad0fd5940c0a2bfc7c61acd969a: Beginning new tablet copy session on tablet c81785db7d654e4288fcaeacf4db2b49 from peer 3b879799899a499c8d409ee9f1277cd1 at {username='slave'} at 127.0.0.1:36462: session id = 3b879799899a499c8d409ee9f1277cd1-c81785db7d654e4288fcaeacf4db2b49
I20260709 12:58:52.321941  1615 tablet_copy_source_session.cc:215] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:58:52.326006  2470 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c81785db7d654e4288fcaeacf4db2b49. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:52.343583  2474 ts_tablet_manager.cc:933] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1: Initiating tablet copy from peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:58:52.347512  2474 tablet_copy_client.cc:323] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Beginning tablet copy session from remote peer at address 127.0.107.194:46261
I20260709 12:58:52.349635  1690 tablet_copy_service.cc:140] P 816fadcea8fc4893882f5f9b9f22a03d: Received BeginTabletCopySession request for tablet a40e423ed583480280352b261913276c from peer 3b879799899a499c8d409ee9f1277cd1 ({username='slave'} at 127.0.0.1:57092)
I20260709 12:58:52.350005  2470 tablet_copy_client.cc:806] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Starting download of 0 data blocks...
I20260709 12:58:52.350188  1690 tablet_copy_service.cc:161] P 816fadcea8fc4893882f5f9b9f22a03d: Beginning new tablet copy session on tablet a40e423ed583480280352b261913276c from peer 3b879799899a499c8d409ee9f1277cd1 at {username='slave'} at 127.0.0.1:57092: session id = 3b879799899a499c8d409ee9f1277cd1-a40e423ed583480280352b261913276c
I20260709 12:58:52.351824  2470 tablet_copy_client.cc:670] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Starting download of 1 WAL segments...
I20260709 12:58:52.357142  2470 tablet_copy_client.cc:538] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:58:52.359279  1690 tablet_copy_source_session.cc:215] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:58:52.370096  2470 tablet_bootstrap.cc:492] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1: Bootstrap starting.
I20260709 12:58:52.372150  2474 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a40e423ed583480280352b261913276c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:52.407292  2470 tablet_bootstrap.cc:492] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:58:52.408833  2470 tablet_bootstrap.cc:492] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1: Bootstrap complete.
I20260709 12:58:52.409801  2470 ts_tablet_manager.cc:1403] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1: Time spent bootstrapping tablet: real 0.040s	user 0.024s	sys 0.011s
I20260709 12:58:52.412959  2474 tablet_copy_client.cc:806] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Starting download of 0 data blocks...
I20260709 12:58:52.414698  2474 tablet_copy_client.cc:670] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Starting download of 1 WAL segments...
I20260709 12:58:52.413748  2470 raft_consensus.cc:359] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } }
I20260709 12:58:52.416066  2470 raft_consensus.cc:740] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Initialized, Role: LEARNER
I20260709 12:58:52.416793  2470 consensus_queue.cc:260] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } }
I20260709 12:58:52.419430  2474 tablet_copy_client.cc:538] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:58:52.428637  2470 ts_tablet_manager.cc:1434] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1: Time spent starting tablet: real 0.018s	user 0.005s	sys 0.004s
I20260709 12:58:52.438679  1615 tablet_copy_service.cc:342] P 83595ad0fd5940c0a2bfc7c61acd969a: Request end of tablet copy session 3b879799899a499c8d409ee9f1277cd1-c81785db7d654e4288fcaeacf4db2b49 received from {username='slave'} at 127.0.0.1:36462
I20260709 12:58:52.447618  1615 tablet_copy_service.cc:434] P 83595ad0fd5940c0a2bfc7c61acd969a: ending tablet copy session 3b879799899a499c8d409ee9f1277cd1-c81785db7d654e4288fcaeacf4db2b49 on tablet c81785db7d654e4288fcaeacf4db2b49 with peer 3b879799899a499c8d409ee9f1277cd1
I20260709 12:58:52.464849  2470 ts_tablet_manager.cc:933] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1: Initiating tablet copy from peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:58:52.471462  2470 tablet_copy_client.cc:323] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Beginning tablet copy session from remote peer at address 127.0.107.194:46261
I20260709 12:58:52.476962  1690 tablet_copy_service.cc:140] P 816fadcea8fc4893882f5f9b9f22a03d: Received BeginTabletCopySession request for tablet dd3f391a609743de886c525de483fc29 from peer 3b879799899a499c8d409ee9f1277cd1 ({username='slave'} at 127.0.0.1:57092)
I20260709 12:58:52.477767  1690 tablet_copy_service.cc:161] P 816fadcea8fc4893882f5f9b9f22a03d: Beginning new tablet copy session on tablet dd3f391a609743de886c525de483fc29 from peer 3b879799899a499c8d409ee9f1277cd1 at {username='slave'} at 127.0.0.1:57092: session id = 3b879799899a499c8d409ee9f1277cd1-dd3f391a609743de886c525de483fc29
I20260709 12:58:52.492959  2477 ts_tablet_manager.cc:933] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1: Initiating tablet copy from peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:58:52.495652  1690 tablet_copy_source_session.cc:215] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:58:52.492969  2474 tablet_bootstrap.cc:492] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1: Bootstrap starting.
I20260709 12:58:52.507478  2477 tablet_copy_client.cc:323] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Beginning tablet copy session from remote peer at address 127.0.107.195:40693
I20260709 12:58:52.512398  2470 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd3f391a609743de886c525de483fc29. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:52.514395  1764 tablet_copy_service.cc:140] P af520b67945d443b901bbeca37cc5701: Received BeginTabletCopySession request for tablet c6ab3300be1e4a59b64b14d50da193ca from peer 3b879799899a499c8d409ee9f1277cd1 ({username='slave'} at 127.0.0.1:43156)
I20260709 12:58:52.515086  1764 tablet_copy_service.cc:161] P af520b67945d443b901bbeca37cc5701: Beginning new tablet copy session on tablet c6ab3300be1e4a59b64b14d50da193ca from peer 3b879799899a499c8d409ee9f1277cd1 at {username='slave'} at 127.0.0.1:43156: session id = 3b879799899a499c8d409ee9f1277cd1-c6ab3300be1e4a59b64b14d50da193ca
I20260709 12:58:52.524518  1764 tablet_copy_source_session.cc:215] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:58:52.529330  2477 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6ab3300be1e4a59b64b14d50da193ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:52.548352  2477 tablet_copy_client.cc:806] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Starting download of 0 data blocks...
I20260709 12:58:52.550019  2477 tablet_copy_client.cc:670] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Starting download of 1 WAL segments...
I20260709 12:58:52.550132  2470 tablet_copy_client.cc:806] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Starting download of 0 data blocks...
I20260709 12:58:52.552896  2470 tablet_copy_client.cc:670] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Starting download of 1 WAL segments...
I20260709 12:58:52.557353  2477 tablet_copy_client.cc:538] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:58:52.558349  2474 tablet_bootstrap.cc:492] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:58:52.559336  2474 tablet_bootstrap.cc:492] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1: Bootstrap complete.
I20260709 12:58:52.562633  2474 ts_tablet_manager.cc:1403] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1: Time spent bootstrapping tablet: real 0.071s	user 0.018s	sys 0.016s
I20260709 12:58:52.567215  2477 tablet_bootstrap.cc:492] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1: Bootstrap starting.
I20260709 12:58:52.567593  2470 tablet_copy_client.cc:538] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:58:52.568656  2474 raft_consensus.cc:359] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } }
I20260709 12:58:52.569547  2474 raft_consensus.cc:740] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Initialized, Role: LEARNER
I20260709 12:58:52.570215  2474 consensus_queue.cc:260] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } }
I20260709 12:58:52.578277  2470 tablet_bootstrap.cc:492] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1: Bootstrap starting.
I20260709 12:58:52.585384  2474 ts_tablet_manager.cc:1434] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1: Time spent starting tablet: real 0.022s	user 0.003s	sys 0.003s
I20260709 12:58:52.602123  1690 tablet_copy_service.cc:342] P 816fadcea8fc4893882f5f9b9f22a03d: Request end of tablet copy session 3b879799899a499c8d409ee9f1277cd1-a40e423ed583480280352b261913276c received from {username='slave'} at 127.0.0.1:57092
I20260709 12:58:52.602677  1690 tablet_copy_service.cc:434] P 816fadcea8fc4893882f5f9b9f22a03d: ending tablet copy session 3b879799899a499c8d409ee9f1277cd1-a40e423ed583480280352b261913276c on tablet a40e423ed583480280352b261913276c with peer 3b879799899a499c8d409ee9f1277cd1
I20260709 12:58:52.610263  2477 tablet_bootstrap.cc:492] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:58:52.611454  2477 tablet_bootstrap.cc:492] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1: Bootstrap complete.
I20260709 12:58:52.612478  2477 ts_tablet_manager.cc:1403] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1: Time spent bootstrapping tablet: real 0.046s	user 0.017s	sys 0.016s
I20260709 12:58:52.615972  2477 raft_consensus.cc:359] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } }
I20260709 12:58:52.616988  2477 raft_consensus.cc:740] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Initialized, Role: LEARNER
I20260709 12:58:52.618006  2477 consensus_queue.cc:260] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } }
I20260709 12:58:52.618964  2470 tablet_bootstrap.cc:492] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:58:52.619946  2470 tablet_bootstrap.cc:492] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1: Bootstrap complete.
I20260709 12:58:52.620741  2470 ts_tablet_manager.cc:1403] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1: Time spent bootstrapping tablet: real 0.043s	user 0.008s	sys 0.022s
I20260709 12:58:52.623215  2477 ts_tablet_manager.cc:1434] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1: Time spent starting tablet: real 0.010s	user 0.001s	sys 0.007s
I20260709 12:58:52.625159  1764 tablet_copy_service.cc:342] P af520b67945d443b901bbeca37cc5701: Request end of tablet copy session 3b879799899a499c8d409ee9f1277cd1-c6ab3300be1e4a59b64b14d50da193ca received from {username='slave'} at 127.0.0.1:43156
I20260709 12:58:52.625670  1764 tablet_copy_service.cc:434] P af520b67945d443b901bbeca37cc5701: ending tablet copy session 3b879799899a499c8d409ee9f1277cd1-c6ab3300be1e4a59b64b14d50da193ca on tablet c6ab3300be1e4a59b64b14d50da193ca with peer 3b879799899a499c8d409ee9f1277cd1
I20260709 12:58:52.625056  2470 raft_consensus.cc:359] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } }
I20260709 12:58:52.628006  2470 raft_consensus.cc:740] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Initialized, Role: LEARNER
I20260709 12:58:52.628810  2470 consensus_queue.cc:260] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } }
I20260709 12:58:52.633036  2470 ts_tablet_manager.cc:1434] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1: Time spent starting tablet: real 0.011s	user 0.001s	sys 0.005s
I20260709 12:58:52.636894  1690 tablet_copy_service.cc:342] P 816fadcea8fc4893882f5f9b9f22a03d: Request end of tablet copy session 3b879799899a499c8d409ee9f1277cd1-dd3f391a609743de886c525de483fc29 received from {username='slave'} at 127.0.0.1:57092
I20260709 12:58:52.637360  1690 tablet_copy_service.cc:434] P 816fadcea8fc4893882f5f9b9f22a03d: ending tablet copy session 3b879799899a499c8d409ee9f1277cd1-dd3f391a609743de886c525de483fc29 on tablet dd3f391a609743de886c525de483fc29 with peer 3b879799899a499c8d409ee9f1277cd1
I20260709 12:58:52.745389  2335 raft_consensus.cc:1217] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:58:52.869333  2335 raft_consensus.cc:1217] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:58:52.968637  2335 raft_consensus.cc:1217] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:58:53.001780  2335 raft_consensus.cc:1217] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:58:53.368234  2486 raft_consensus.cc:1064] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a: attempting to promote NON_VOTER 3b879799899a499c8d409ee9f1277cd1 to VOTER
I20260709 12:58:53.371631  2486 consensus_queue.cc:237] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:58:53.380038  2335 raft_consensus.cc:1275] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 1 LEARNER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:58:53.380360  1748 raft_consensus.cc:1275] T c81785db7d654e4288fcaeacf4db2b49 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:58:53.380959  1680 raft_consensus.cc:1275] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:58:53.382191  2207 consensus_queue.cc:1048] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:58:53.383153  2486 consensus_queue.cc:1048] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:58:53.385046  2451 consensus_queue.cc:1048] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:58:53.413931  2451 raft_consensus.cc:2955] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:58:53.420439  1680 raft_consensus.cc:2955] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:58:53.427402  2462 raft_consensus.cc:1064] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d: attempting to promote NON_VOTER 3b879799899a499c8d409ee9f1277cd1 to VOTER
I20260709 12:58:53.417447  1748 raft_consensus.cc:2955] T c81785db7d654e4288fcaeacf4db2b49 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:58:53.443876  2479 raft_consensus.cc:1064] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701: attempting to promote NON_VOTER 3b879799899a499c8d409ee9f1277cd1 to VOTER
I20260709 12:58:53.447690  2479 consensus_queue.cc:237] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:58:53.462124  2335 raft_consensus.cc:2955] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:58:53.463099  2462 consensus_queue.cc:237] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:58:53.475574  2336 raft_consensus.cc:1275] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 1 LEARNER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:58:53.477384  2493 consensus_queue.cc:1048] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:58:53.497843  1748 raft_consensus.cc:1275] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:58:53.498845  1603 raft_consensus.cc:1275] T a40e423ed583480280352b261913276c P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:58:53.502519  2462 consensus_queue.cc:1048] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.008s
I20260709 12:58:53.511086  2493 consensus_queue.cc:1048] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:58:53.508266  1501 catalog_manager.cc:5697] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:58:53.593464  2336 raft_consensus.cc:1275] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 1 LEARNER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:58:53.609395  2436 consensus_queue.cc:1048] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:58:53.623311  2491 raft_consensus.cc:1064] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d: attempting to promote NON_VOTER 3b879799899a499c8d409ee9f1277cd1 to VOTER
I20260709 12:58:53.635087  2491 consensus_queue.cc:237] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:58:53.655050  1680 raft_consensus.cc:1275] T c6ab3300be1e4a59b64b14d50da193ca P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:58:53.664472  2485 consensus_queue.cc:1048] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20260709 12:58:53.675917  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 94271.1us
I20260709 12:58:53.680610  1602 raft_consensus.cc:1275] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:58:53.685237  2491 consensus_queue.cc:1048] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:58:53.694063  2335 raft_consensus.cc:1275] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 1 LEARNER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
W20260709 12:58:53.767680  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 200373us
W20260709 12:58:53.774024  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 88002.8us
I20260709 12:58:53.797827  1752 raft_consensus.cc:1275] T dd3f391a609743de886c525de483fc29 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:58:53.770218  2462 raft_consensus.cc:2955] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:58:53.809610  1602 raft_consensus.cc:1275] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:58:53.856829  2499 consensus_queue.cc:1048] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:58:53.855997  1748 raft_consensus.cc:2955] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:58:53.798125  2514 consensus_queue.cc:1048] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.073s
I20260709 12:58:53.879206  2336 raft_consensus.cc:2955] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:58:53.884738  2457 consensus_queue.cc:1048] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:58:53.913938  2499 raft_consensus.cc:2955] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:58:53.916409  1605 raft_consensus.cc:2955] T a40e423ed583480280352b261913276c P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:58:53.936342  1604 raft_consensus.cc:2955] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:58:53.955224  2336 raft_consensus.cc:2955] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:58:53.972163  1677 raft_consensus.cc:2955] T c6ab3300be1e4a59b64b14d50da193ca P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:58:53.991314  2520 raft_consensus.cc:2955] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:58:53.999188  1752 raft_consensus.cc:2955] T dd3f391a609743de886c525de483fc29 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:58:54.011040  1601 raft_consensus.cc:2955] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:58:54.053903  2336 raft_consensus.cc:2955] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:58:54.054672  1605 consensus_queue.cc:237] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:58:54.070204  1500 catalog_manager.cc:5697] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 reported cstate change: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "af520b67945d443b901bbeca37cc5701" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:58:54.081931  1500 catalog_manager.cc:5697] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 reported cstate change: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:58:54.083055  2335 raft_consensus.cc:1275] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:58:54.084728  2512 consensus_queue.cc:1048] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:58:54.091758  1677 raft_consensus.cc:1275] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:58:54.093603  2512 consensus_queue.cc:1048] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:58:54.093012  2500 raft_consensus.cc:2955] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER af520b67945d443b901bbeca37cc5701 (127.0.107.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:58:54.095582  2335 raft_consensus.cc:2955] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER af520b67945d443b901bbeca37cc5701 (127.0.107.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:58:54.099565  1677 raft_consensus.cc:2955] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER af520b67945d443b901bbeca37cc5701 (127.0.107.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:58:54.105060  1487 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet c81785db7d654e4288fcaeacf4db2b49 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:58:54.108984  1503 catalog_manager.cc:5697] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d reported cstate change: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:58:54.125135  1501 catalog_manager.cc:5697] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 reported cstate change: config changed from index 3 to 4, VOTER af520b67945d443b901bbeca37cc5701 (127.0.107.195) evicted. New cstate: current_term: 1 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:58:54.137737  1680 consensus_queue.cc:237] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:58:54.143457  1740 tablet_service.cc:1558] Processing DeleteTablet for tablet c81785db7d654e4288fcaeacf4db2b49 with delete_type TABLET_DATA_TOMBSTONED (TS af520b67945d443b901bbeca37cc5701 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:55838
I20260709 12:58:54.145725  1745 consensus_queue.cc:237] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:58:54.157850  2335 raft_consensus.cc:1275] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:58:54.160411  2335 raft_consensus.cc:1275] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:58:54.162653  2499 consensus_queue.cc:1048] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:58:54.168349  2457 consensus_queue.cc:1048] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:58:54.171517  2519 raft_consensus.cc:2955] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:58:54.179834  1605 raft_consensus.cc:1275] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:58:54.185309  1677 consensus_queue.cc:237] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:58:54.186707  2336 raft_consensus.cc:2955] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:58:54.189630  2526 tablet_replica.cc:333] T c81785db7d654e4288fcaeacf4db2b49 P af520b67945d443b901bbeca37cc5701: stopping tablet replica
I20260709 12:58:54.203596  1500 catalog_manager.cc:5697] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 reported cstate change: config changed from index 3 to 4, VOTER 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194) evicted. New cstate: current_term: 1 leader_uuid: "af520b67945d443b901bbeca37cc5701" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:58:54.208602  2526 raft_consensus.cc:2243] T c81785db7d654e4288fcaeacf4db2b49 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:58:54.209468  2526 raft_consensus.cc:2272] T c81785db7d654e4288fcaeacf4db2b49 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:58:54.209887  2337 raft_consensus.cc:1275] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:58:54.212534  1605 raft_consensus.cc:1275] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:58:54.214562  2515 consensus_queue.cc:1048] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:58:54.206372  2520 raft_consensus.cc:2955] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER af520b67945d443b901bbeca37cc5701 (127.0.107.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:58:54.218321  2492 consensus_queue.cc:1048] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:58:54.219240  1745 raft_consensus.cc:1275] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:58:54.221624  2518 consensus_queue.cc:1048] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:58:54.222033  2515 consensus_queue.cc:1048] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:58:54.225607  2526 ts_tablet_manager.cc:1916] T c81785db7d654e4288fcaeacf4db2b49 P af520b67945d443b901bbeca37cc5701: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:58:54.209614  2335 raft_consensus.cc:2955] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER af520b67945d443b901bbeca37cc5701 (127.0.107.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:58:54.235194  2462 raft_consensus.cc:2955] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:58:54.232775  1600 raft_consensus.cc:2955] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER af520b67945d443b901bbeca37cc5701 (127.0.107.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:58:54.242409  1605 raft_consensus.cc:2955] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:58:54.241134  1745 raft_consensus.cc:2955] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:58:54.247793  1665 tablet_service.cc:1558] Processing DeleteTablet for tablet c6ab3300be1e4a59b64b14d50da193ca with delete_type TABLET_DATA_TOMBSTONED (TS 816fadcea8fc4893882f5f9b9f22a03d not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:52760
I20260709 12:58:54.263368  1488 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet dd3f391a609743de886c525de483fc29 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:58:54.390456  1500 catalog_manager.cc:5697] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d reported cstate change: config changed from index 3 to 4, VOTER af520b67945d443b901bbeca37cc5701 (127.0.107.195) evicted. New cstate: current_term: 1 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:58:54.398774  1490 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet c6ab3300be1e4a59b64b14d50da193ca with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20260709 12:58:54.402100  2502 log.cc:927] Time spent T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.091s	user 0.000s	sys 0.008s
I20260709 12:58:54.411872  2527 tablet_replica.cc:333] T c6ab3300be1e4a59b64b14d50da193ca P 816fadcea8fc4893882f5f9b9f22a03d: stopping tablet replica
I20260709 12:58:54.413507  2527 raft_consensus.cc:2243] T c6ab3300be1e4a59b64b14d50da193ca P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Raft consensus shutting down.
W20260709 12:58:54.416860  2523 log.cc:927] Time spent T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.098s	user 0.001s	sys 0.007s
I20260709 12:58:54.416761  1488 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet a40e423ed583480280352b261913276c with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:58:54.449913  2527 raft_consensus.cc:2272] T c6ab3300be1e4a59b64b14d50da193ca P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:58:54.457041  2527 ts_tablet_manager.cc:1916] T c6ab3300be1e4a59b64b14d50da193ca P 816fadcea8fc4893882f5f9b9f22a03d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:58:54.476323  1740 tablet_service.cc:1558] Processing DeleteTablet for tablet dd3f391a609743de886c525de483fc29 with delete_type TABLET_DATA_TOMBSTONED (TS af520b67945d443b901bbeca37cc5701 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:55838
I20260709 12:58:54.493276  2526 ts_tablet_manager.cc:1929] T c81785db7d654e4288fcaeacf4db2b49 P af520b67945d443b901bbeca37cc5701: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:58:54.494045  2526 log.cc:1199] T c81785db7d654e4288fcaeacf4db2b49 P af520b67945d443b901bbeca37cc5701: Deleting WAL directory at /tmp/dist-test-taskakPCWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783601878574137-431-0/minicluster-data/ts-2-root/wals/c81785db7d654e4288fcaeacf4db2b49
I20260709 12:58:54.518615  1490 catalog_manager.cc:5028] TS af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): tablet c81785db7d654e4288fcaeacf4db2b49 (table test-workload [id=75d77b57b3f34786b9c215c4f11bd727]) successfully deleted
I20260709 12:58:54.519630  2526 tablet_replica.cc:333] T dd3f391a609743de886c525de483fc29 P af520b67945d443b901bbeca37cc5701: stopping tablet replica
I20260709 12:58:54.539479  1503 catalog_manager.cc:5697] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 reported cstate change: config changed from index 3 to 4, VOTER 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) evicted. New cstate: current_term: 1 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:58:54.524222  2336 raft_consensus.cc:2955] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:58:54.583424  2526 raft_consensus.cc:2243] T dd3f391a609743de886c525de483fc29 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:58:54.586598  2527 ts_tablet_manager.cc:1929] T c6ab3300be1e4a59b64b14d50da193ca P 816fadcea8fc4893882f5f9b9f22a03d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:58:54.587301  2527 log.cc:1199] T c6ab3300be1e4a59b64b14d50da193ca P 816fadcea8fc4893882f5f9b9f22a03d: Deleting WAL directory at /tmp/dist-test-taskakPCWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783601878574137-431-0/minicluster-data/ts-1-root/wals/c6ab3300be1e4a59b64b14d50da193ca
I20260709 12:58:54.599637  1592 tablet_service.cc:1558] Processing DeleteTablet for tablet a40e423ed583480280352b261913276c with delete_type TABLET_DATA_TOMBSTONED (TS 83595ad0fd5940c0a2bfc7c61acd969a not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:40138
I20260709 12:58:54.584749  2526 raft_consensus.cc:2272] T dd3f391a609743de886c525de483fc29 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:58:54.605314  1488 catalog_manager.cc:5028] TS 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): tablet c6ab3300be1e4a59b64b14d50da193ca (table test-workload [id=75d77b57b3f34786b9c215c4f11bd727]) successfully deleted
I20260709 12:58:54.614404  2526 ts_tablet_manager.cc:1916] T dd3f391a609743de886c525de483fc29 P af520b67945d443b901bbeca37cc5701: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:58:54.622263  2529 tablet_replica.cc:333] T a40e423ed583480280352b261913276c P 83595ad0fd5940c0a2bfc7c61acd969a: stopping tablet replica
I20260709 12:58:54.623488  2529 raft_consensus.cc:2243] T a40e423ed583480280352b261913276c P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:58:54.624414  2529 raft_consensus.cc:2272] T a40e423ed583480280352b261913276c P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:58:54.628026  2529 ts_tablet_manager.cc:1916] T a40e423ed583480280352b261913276c P 83595ad0fd5940c0a2bfc7c61acd969a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:58:54.649083  2526 ts_tablet_manager.cc:1929] T dd3f391a609743de886c525de483fc29 P af520b67945d443b901bbeca37cc5701: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:58:54.649588  2526 log.cc:1199] T dd3f391a609743de886c525de483fc29 P af520b67945d443b901bbeca37cc5701: Deleting WAL directory at /tmp/dist-test-taskakPCWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783601878574137-431-0/minicluster-data/ts-2-root/wals/dd3f391a609743de886c525de483fc29
I20260709 12:58:54.651003  2529 ts_tablet_manager.cc:1929] T a40e423ed583480280352b261913276c P 83595ad0fd5940c0a2bfc7c61acd969a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:58:54.651484  2529 log.cc:1199] T a40e423ed583480280352b261913276c P 83595ad0fd5940c0a2bfc7c61acd969a: Deleting WAL directory at /tmp/dist-test-taskakPCWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783601878574137-431-0/minicluster-data/ts-0-root/wals/a40e423ed583480280352b261913276c
I20260709 12:58:54.651943  1490 catalog_manager.cc:5028] TS af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): tablet dd3f391a609743de886c525de483fc29 (table test-workload [id=75d77b57b3f34786b9c215c4f11bd727]) successfully deleted
I20260709 12:58:54.653627  1487 catalog_manager.cc:5028] TS 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): tablet a40e423ed583480280352b261913276c (table test-workload [id=75d77b57b3f34786b9c215c4f11bd727]) successfully deleted
I20260709 12:58:55.704967  1677 consensus_queue.cc:237] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } }
W20260709 12:58:55.711350  1644 consensus_peers.cc:597] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d -> Peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625): Couldn't send request to peer 3b879799899a499c8d409ee9f1277cd1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 344b4152823f4d618345b14ffd579400. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:58:55.711515  1748 raft_consensus.cc:1275] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:58:55.712095  1600 raft_consensus.cc:1275] T 344b4152823f4d618345b14ffd579400 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:58:55.712855  2492 consensus_queue.cc:1048] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:58:55.713591  2531 consensus_queue.cc:1048] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:58:55.721822  2515 raft_consensus.cc:2955] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } } }
I20260709 12:58:55.723529  1751 raft_consensus.cc:2955] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } } }
I20260709 12:58:55.723670  1600 raft_consensus.cc:2955] T 344b4152823f4d618345b14ffd579400 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } } }
I20260709 12:58:55.743427  1600 consensus_queue.cc:237] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } }
I20260709 12:58:55.741592  1503 catalog_manager.cc:5697] T 344b4152823f4d618345b14ffd579400 P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: config changed from index -1 to 2, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New cstate: current_term: 1 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } } }
W20260709 12:58:55.757246  1569 consensus_peers.cc:597] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a -> Peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625): Couldn't send request to peer 3b879799899a499c8d409ee9f1277cd1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 14588fa8cfd6467e989746e326a0ed31. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:58:55.759150  1751 raft_consensus.cc:1275] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:58:55.759202  1677 raft_consensus.cc:1275] T 14588fa8cfd6467e989746e326a0ed31 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:58:55.760799  2500 consensus_queue.cc:1048] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:58:55.761878  2528 consensus_queue.cc:1048] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:58:55.770671  2528 raft_consensus.cc:2955] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } } }
I20260709 12:58:55.772259  1677 raft_consensus.cc:2955] T 14588fa8cfd6467e989746e326a0ed31 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } } }
I20260709 12:58:55.782410  1748 raft_consensus.cc:2955] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } } }
I20260709 12:58:55.794837  1677 consensus_queue.cc:237] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } }
I20260709 12:58:55.797935  1500 catalog_manager.cc:5697] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: config changed from index -1 to 2, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New cstate: current_term: 1 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20260709 12:58:55.804196  1644 consensus_peers.cc:597] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d -> Peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625): Couldn't send request to peer 3b879799899a499c8d409ee9f1277cd1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 513043a9b96c4f70adad53ac4b5d01dc. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:58:55.806154  1748 raft_consensus.cc:1275] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:58:55.807096  1600 raft_consensus.cc:1275] T 513043a9b96c4f70adad53ac4b5d01dc P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:58:55.807651  2531 consensus_queue.cc:1048] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:58:55.808970  2492 consensus_queue.cc:1048] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:58:55.819634  2531 raft_consensus.cc:2955] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } } }
I20260709 12:58:55.821656  1600 raft_consensus.cc:2955] T 513043a9b96c4f70adad53ac4b5d01dc P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } } }
I20260709 12:58:55.838406  1677 consensus_queue.cc:237] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } }
I20260709 12:58:55.840750  1748 raft_consensus.cc:2955] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } } }
I20260709 12:58:55.840886  1501 catalog_manager.cc:5697] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d reported cstate change: config changed from index -1 to 2, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New cstate: current_term: 1 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260709 12:58:55.853230  1751 raft_consensus.cc:1275] T f8ab62ffb73246dc9bcf39f7bee51147 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260709 12:58:55.854250  1644 consensus_peers.cc:597] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d -> Peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625): Couldn't send request to peer 3b879799899a499c8d409ee9f1277cd1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f8ab62ffb73246dc9bcf39f7bee51147. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:58:55.855201  2492 consensus_queue.cc:1048] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:58:55.856005  1600 raft_consensus.cc:1275] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:58:55.858219  2492 consensus_queue.cc:1048] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.004s
I20260709 12:58:55.870208  2531 raft_consensus.cc:2955] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } } }
I20260709 12:58:55.874715  1751 raft_consensus.cc:2955] T f8ab62ffb73246dc9bcf39f7bee51147 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } } }
I20260709 12:58:55.887459  1600 raft_consensus.cc:2955] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } } }
I20260709 12:58:55.893911  1503 catalog_manager.cc:5697] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d reported cstate change: config changed from index -1 to 2, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New cstate: current_term: 1 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260709 12:58:56.141939  2552 ts_tablet_manager.cc:933] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1: Initiating tablet copy from peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:58:56.145730  2552 tablet_copy_client.cc:323] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Beginning tablet copy session from remote peer at address 127.0.107.194:46261
I20260709 12:58:56.148067  1690 tablet_copy_service.cc:140] P 816fadcea8fc4893882f5f9b9f22a03d: Received BeginTabletCopySession request for tablet 344b4152823f4d618345b14ffd579400 from peer 3b879799899a499c8d409ee9f1277cd1 ({username='slave'} at 127.0.0.1:57092)
I20260709 12:58:56.148656  1690 tablet_copy_service.cc:161] P 816fadcea8fc4893882f5f9b9f22a03d: Beginning new tablet copy session on tablet 344b4152823f4d618345b14ffd579400 from peer 3b879799899a499c8d409ee9f1277cd1 at {username='slave'} at 127.0.0.1:57092: session id = 3b879799899a499c8d409ee9f1277cd1-344b4152823f4d618345b14ffd579400
I20260709 12:58:56.150866  2554 ts_tablet_manager.cc:933] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1: Initiating tablet copy from peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445)
I20260709 12:58:56.155583  2554 tablet_copy_client.cc:323] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Beginning tablet copy session from remote peer at address 127.0.107.193:46445
I20260709 12:58:56.157135  1615 tablet_copy_service.cc:140] P 83595ad0fd5940c0a2bfc7c61acd969a: Received BeginTabletCopySession request for tablet 14588fa8cfd6467e989746e326a0ed31 from peer 3b879799899a499c8d409ee9f1277cd1 ({username='slave'} at 127.0.0.1:36462)
I20260709 12:58:56.157666  1615 tablet_copy_service.cc:161] P 83595ad0fd5940c0a2bfc7c61acd969a: Beginning new tablet copy session on tablet 14588fa8cfd6467e989746e326a0ed31 from peer 3b879799899a499c8d409ee9f1277cd1 at {username='slave'} at 127.0.0.1:36462: session id = 3b879799899a499c8d409ee9f1277cd1-14588fa8cfd6467e989746e326a0ed31
I20260709 12:58:56.159440  1690 tablet_copy_source_session.cc:215] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:58:56.163010  2552 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 344b4152823f4d618345b14ffd579400. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:56.164093  1615 tablet_copy_source_session.cc:215] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:58:56.167420  2554 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 14588fa8cfd6467e989746e326a0ed31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:56.188344  2552 tablet_copy_client.cc:806] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Starting download of 0 data blocks...
I20260709 12:58:56.189045  2552 tablet_copy_client.cc:670] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Starting download of 1 WAL segments...
I20260709 12:58:56.198259  2552 tablet_copy_client.cc:538] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:58:56.209070  2552 tablet_bootstrap.cc:492] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1: Bootstrap starting.
I20260709 12:58:56.235421  2554 tablet_copy_client.cc:806] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Starting download of 0 data blocks...
I20260709 12:58:56.236218  2554 tablet_copy_client.cc:670] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Starting download of 1 WAL segments...
I20260709 12:58:56.240422  2554 tablet_copy_client.cc:538] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:58:56.250327  2552 tablet_bootstrap.cc:492] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:58:56.251466  2552 tablet_bootstrap.cc:492] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1: Bootstrap complete.
I20260709 12:58:56.251999  2554 tablet_bootstrap.cc:492] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1: Bootstrap starting.
I20260709 12:58:56.252287  2552 ts_tablet_manager.cc:1403] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1: Time spent bootstrapping tablet: real 0.043s	user 0.014s	sys 0.012s
I20260709 12:58:56.262583  2552 raft_consensus.cc:359] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } }
I20260709 12:58:56.263609  2552 raft_consensus.cc:740] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Initialized, Role: LEARNER
I20260709 12:58:56.264362  2552 consensus_queue.cc:260] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } }
I20260709 12:58:56.280942  2552 ts_tablet_manager.cc:1434] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1: Time spent starting tablet: real 0.028s	user 0.010s	sys 0.014s
I20260709 12:58:56.283618  1690 tablet_copy_service.cc:342] P 816fadcea8fc4893882f5f9b9f22a03d: Request end of tablet copy session 3b879799899a499c8d409ee9f1277cd1-344b4152823f4d618345b14ffd579400 received from {username='slave'} at 127.0.0.1:57092
I20260709 12:58:56.284087  1690 tablet_copy_service.cc:434] P 816fadcea8fc4893882f5f9b9f22a03d: ending tablet copy session 3b879799899a499c8d409ee9f1277cd1-344b4152823f4d618345b14ffd579400 on tablet 344b4152823f4d618345b14ffd579400 with peer 3b879799899a499c8d409ee9f1277cd1
I20260709 12:58:56.291334  2554 tablet_bootstrap.cc:492] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:58:56.292393  2554 tablet_bootstrap.cc:492] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1: Bootstrap complete.
I20260709 12:58:56.293210  2554 ts_tablet_manager.cc:1403] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1: Time spent bootstrapping tablet: real 0.041s	user 0.023s	sys 0.009s
I20260709 12:58:56.293658  2552 ts_tablet_manager.cc:933] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1: Initiating tablet copy from peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:58:56.296702  2554 raft_consensus.cc:359] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } }
I20260709 12:58:56.297672  2554 raft_consensus.cc:740] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Initialized, Role: LEARNER
I20260709 12:58:56.298419  2554 consensus_queue.cc:260] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } }
I20260709 12:58:56.302592  2554 ts_tablet_manager.cc:1434] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1: Time spent starting tablet: real 0.009s	user 0.008s	sys 0.000s
I20260709 12:58:56.304454  1615 tablet_copy_service.cc:342] P 83595ad0fd5940c0a2bfc7c61acd969a: Request end of tablet copy session 3b879799899a499c8d409ee9f1277cd1-14588fa8cfd6467e989746e326a0ed31 received from {username='slave'} at 127.0.0.1:36462
I20260709 12:58:56.304888  1615 tablet_copy_service.cc:434] P 83595ad0fd5940c0a2bfc7c61acd969a: ending tablet copy session 3b879799899a499c8d409ee9f1277cd1-14588fa8cfd6467e989746e326a0ed31 on tablet 14588fa8cfd6467e989746e326a0ed31 with peer 3b879799899a499c8d409ee9f1277cd1
I20260709 12:58:56.307811  2552 tablet_copy_client.cc:323] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Beginning tablet copy session from remote peer at address 127.0.107.194:46261
I20260709 12:58:56.309536  1690 tablet_copy_service.cc:140] P 816fadcea8fc4893882f5f9b9f22a03d: Received BeginTabletCopySession request for tablet 513043a9b96c4f70adad53ac4b5d01dc from peer 3b879799899a499c8d409ee9f1277cd1 ({username='slave'} at 127.0.0.1:57092)
I20260709 12:58:56.310048  1690 tablet_copy_service.cc:161] P 816fadcea8fc4893882f5f9b9f22a03d: Beginning new tablet copy session on tablet 513043a9b96c4f70adad53ac4b5d01dc from peer 3b879799899a499c8d409ee9f1277cd1 at {username='slave'} at 127.0.0.1:57092: session id = 3b879799899a499c8d409ee9f1277cd1-513043a9b96c4f70adad53ac4b5d01dc
I20260709 12:58:56.322041  1690 tablet_copy_source_session.cc:215] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:58:56.325506  2552 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 513043a9b96c4f70adad53ac4b5d01dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:56.342237  2552 tablet_copy_client.cc:806] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Starting download of 0 data blocks...
I20260709 12:58:56.342872  2552 tablet_copy_client.cc:670] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Starting download of 1 WAL segments...
I20260709 12:58:56.346516  2552 tablet_copy_client.cc:538] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:58:56.353145  2559 ts_tablet_manager.cc:933] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1: Initiating tablet copy from peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:58:56.358064  2552 tablet_bootstrap.cc:492] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1: Bootstrap starting.
I20260709 12:58:56.363924  2559 tablet_copy_client.cc:323] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Beginning tablet copy session from remote peer at address 127.0.107.194:46261
I20260709 12:58:56.365854  1690 tablet_copy_service.cc:140] P 816fadcea8fc4893882f5f9b9f22a03d: Received BeginTabletCopySession request for tablet f8ab62ffb73246dc9bcf39f7bee51147 from peer 3b879799899a499c8d409ee9f1277cd1 ({username='slave'} at 127.0.0.1:57092)
I20260709 12:58:56.366403  1690 tablet_copy_service.cc:161] P 816fadcea8fc4893882f5f9b9f22a03d: Beginning new tablet copy session on tablet f8ab62ffb73246dc9bcf39f7bee51147 from peer 3b879799899a499c8d409ee9f1277cd1 at {username='slave'} at 127.0.0.1:57092: session id = 3b879799899a499c8d409ee9f1277cd1-f8ab62ffb73246dc9bcf39f7bee51147
I20260709 12:58:56.376009  1690 tablet_copy_source_session.cc:215] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:58:56.382016  2559 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f8ab62ffb73246dc9bcf39f7bee51147. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:56.396557  2552 tablet_bootstrap.cc:492] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:58:56.397683  2552 tablet_bootstrap.cc:492] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1: Bootstrap complete.
I20260709 12:58:56.398507  2552 ts_tablet_manager.cc:1403] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1: Time spent bootstrapping tablet: real 0.042s	user 0.025s	sys 0.011s
I20260709 12:58:56.405891  2552 raft_consensus.cc:359] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } }
I20260709 12:58:56.406876  2552 raft_consensus.cc:740] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Initialized, Role: LEARNER
I20260709 12:58:56.407649  2552 consensus_queue.cc:260] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } }
I20260709 12:58:56.410488  2559 tablet_copy_client.cc:806] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Starting download of 0 data blocks...
I20260709 12:58:56.417537  2552 ts_tablet_manager.cc:1434] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1: Time spent starting tablet: real 0.019s	user 0.006s	sys 0.001s
I20260709 12:58:56.417798  2559 tablet_copy_client.cc:670] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Starting download of 1 WAL segments...
I20260709 12:58:56.419586  1690 tablet_copy_service.cc:342] P 816fadcea8fc4893882f5f9b9f22a03d: Request end of tablet copy session 3b879799899a499c8d409ee9f1277cd1-513043a9b96c4f70adad53ac4b5d01dc received from {username='slave'} at 127.0.0.1:57092
I20260709 12:58:56.420034  1690 tablet_copy_service.cc:434] P 816fadcea8fc4893882f5f9b9f22a03d: ending tablet copy session 3b879799899a499c8d409ee9f1277cd1-513043a9b96c4f70adad53ac4b5d01dc on tablet 513043a9b96c4f70adad53ac4b5d01dc with peer 3b879799899a499c8d409ee9f1277cd1
I20260709 12:58:56.423501  2559 tablet_copy_client.cc:538] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:58:56.433988  2559 tablet_bootstrap.cc:492] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1: Bootstrap starting.
I20260709 12:58:56.454674  2559 tablet_bootstrap.cc:492] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:58:56.455808  2559 tablet_bootstrap.cc:492] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1: Bootstrap complete.
I20260709 12:58:56.456595  2559 ts_tablet_manager.cc:1403] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1: Time spent bootstrapping tablet: real 0.023s	user 0.025s	sys 0.000s
I20260709 12:58:56.459841  2559 raft_consensus.cc:359] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } }
I20260709 12:58:56.460754  2559 raft_consensus.cc:740] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Initialized, Role: LEARNER
I20260709 12:58:56.461493  2559 consensus_queue.cc:260] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } }
I20260709 12:58:56.464071  2559 ts_tablet_manager.cc:1434] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1: Time spent starting tablet: real 0.007s	user 0.002s	sys 0.004s
I20260709 12:58:56.467851  1689 tablet_copy_service.cc:342] P 816fadcea8fc4893882f5f9b9f22a03d: Request end of tablet copy session 3b879799899a499c8d409ee9f1277cd1-f8ab62ffb73246dc9bcf39f7bee51147 received from {username='slave'} at 127.0.0.1:57092
I20260709 12:58:56.468279  1689 tablet_copy_service.cc:434] P 816fadcea8fc4893882f5f9b9f22a03d: ending tablet copy session 3b879799899a499c8d409ee9f1277cd1-f8ab62ffb73246dc9bcf39f7bee51147 on tablet f8ab62ffb73246dc9bcf39f7bee51147 with peer 3b879799899a499c8d409ee9f1277cd1
I20260709 12:58:56.554481  2335 raft_consensus.cc:1217] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:58:56.657235  2336 raft_consensus.cc:1217] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:58:56.750406  2336 raft_consensus.cc:1217] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:58:56.848304  2336 raft_consensus.cc:1217] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:58:57.107084  2515 raft_consensus.cc:1064] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d: attempting to promote NON_VOTER 3b879799899a499c8d409ee9f1277cd1 to VOTER
I20260709 12:58:57.110278  2515 consensus_queue.cc:237] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:58:57.117193  2336 raft_consensus.cc:1275] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 1 LEARNER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:58:57.119791  1748 raft_consensus.cc:1275] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:58:57.119627  2492 consensus_queue.cc:1048] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:58:57.120186  1601 raft_consensus.cc:1275] T 344b4152823f4d618345b14ffd579400 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:58:57.121673  2531 consensus_queue.cc:1048] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:58:57.125831  2515 consensus_queue.cc:1048] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:58:57.133335  2531 raft_consensus.cc:2955] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:58:57.139703  1601 raft_consensus.cc:2955] T 344b4152823f4d618345b14ffd579400 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:58:57.142271  1748 raft_consensus.cc:2955] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:58:57.147938  2335 raft_consensus.cc:2955] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:58:57.151108  2500 raft_consensus.cc:1064] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a: attempting to promote NON_VOTER 3b879799899a499c8d409ee9f1277cd1 to VOTER
I20260709 12:58:57.153769  2500 consensus_queue.cc:237] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:58:57.157797  1501 catalog_manager.cc:5697] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d reported cstate change: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:58:57.193150  2492 raft_consensus.cc:1064] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d: attempting to promote NON_VOTER 3b879799899a499c8d409ee9f1277cd1 to VOTER
I20260709 12:58:57.219242  1677 raft_consensus.cc:1275] T 14588fa8cfd6467e989746e326a0ed31 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:58:57.223343  2336 raft_consensus.cc:1275] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 1 LEARNER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:58:57.233536  2532 consensus_queue.cc:1048] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:58:57.235632  2573 consensus_queue.cc:1048] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:58:57.196341  2492 consensus_queue.cc:237] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:58:57.255322  1748 raft_consensus.cc:1275] T f8ab62ffb73246dc9bcf39f7bee51147 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:58:57.257220  2492 consensus_queue.cc:1048] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:58:57.258049  2336 raft_consensus.cc:1275] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 1 LEARNER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:58:57.260434  2515 consensus_queue.cc:1048] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20260709 12:58:57.293078  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 132393us
W20260709 12:58:57.267536  2572 log.cc:927] Time spent T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.073s	user 0.001s	sys 0.000s
I20260709 12:58:57.311439  1751 raft_consensus.cc:1275] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:58:57.313256  2515 raft_consensus.cc:1064] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d: attempting to promote NON_VOTER 3b879799899a499c8d409ee9f1277cd1 to VOTER
I20260709 12:58:57.316951  2515 consensus_queue.cc:237] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:58:57.363201  1601 raft_consensus.cc:1275] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:58:57.372284  2573 consensus_queue.cc:1048] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:58:57.394727  2334 raft_consensus.cc:1275] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 1 LEARNER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:58:57.396744  2531 consensus_queue.cc:1048] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:58:57.414013  2531 consensus_queue.cc:1048] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:58:57.445022  1602 raft_consensus.cc:1275] T 513043a9b96c4f70adad53ac4b5d01dc P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:58:57.460696  2492 consensus_queue.cc:1048] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20260709 12:58:57.472822  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 46317.1us
I20260709 12:58:57.481454  2492 raft_consensus.cc:2955] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:58:57.487234  2500 raft_consensus.cc:2955] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:58:57.511933  1752 raft_consensus.cc:2955] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:58:57.517220  1748 raft_consensus.cc:1275] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:58:57.510742  2334 raft_consensus.cc:2955] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:58:57.535383  1676 raft_consensus.cc:2955] T 14588fa8cfd6467e989746e326a0ed31 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:58:57.547608  1752 raft_consensus.cc:2955] T f8ab62ffb73246dc9bcf39f7bee51147 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:58:57.550549  2492 consensus_queue.cc:1048] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:58:57.556010  1604 raft_consensus.cc:2955] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:58:57.574674  2334 raft_consensus.cc:2955] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:58:57.596647  2584 raft_consensus.cc:2955] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:58:57.603912  1751 raft_consensus.cc:2955] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:58:57.617413  1605 raft_consensus.cc:2955] T 513043a9b96c4f70adad53ac4b5d01dc P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:58:57.619179  1676 consensus_queue.cc:237] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:58:57.618356  2334 raft_consensus.cc:2955] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:58:57.640463  2336 raft_consensus.cc:1275] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:58:57.647141  2492 consensus_queue.cc:1048] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:58:57.647610  1503 catalog_manager.cc:5697] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 reported cstate change: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:58:57.663170  1751 raft_consensus.cc:1275] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:58:57.661923  1503 catalog_manager.cc:5697] T f8ab62ffb73246dc9bcf39f7bee51147 P af520b67945d443b901bbeca37cc5701 reported cstate change: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:58:57.665253  2515 consensus_queue.cc:1048] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:58:57.669675  2492 raft_consensus.cc:2955] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:58:57.671300  2336 raft_consensus.cc:2955] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:58:57.682595  1488 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 344b4152823f4d618345b14ffd579400 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:58:57.693183  1751 raft_consensus.cc:2955] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:58:57.696337  1605 consensus_queue.cc:237] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:58:57.708144  1502 catalog_manager.cc:5697] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d reported cstate change: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:58:57.717519  1751 raft_consensus.cc:1275] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:58:57.720067  2336 raft_consensus.cc:1275] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:58:57.720651  2500 consensus_queue.cc:1048] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:58:57.721805  2532 consensus_queue.cc:1048] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:58:57.735674  2571 raft_consensus.cc:2955] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:58:57.738881  2336 raft_consensus.cc:2955] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:58:57.741737  1748 raft_consensus.cc:2955] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:58:57.761149  1487 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 14588fa8cfd6467e989746e326a0ed31 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:58:57.764988  1503 catalog_manager.cc:5697] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 reported cstate change: config changed from index 3 to 4, VOTER 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) evicted. New cstate: current_term: 1 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:58:57.777108  1676 consensus_queue.cc:237] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:58:57.780015  1501 catalog_manager.cc:5697] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 reported cstate change: config changed from index 3 to 4, VOTER 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194) evicted. New cstate: current_term: 1 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:58:57.784786  2336 raft_consensus.cc:1275] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:58:57.786731  1592 tablet_service.cc:1558] Processing DeleteTablet for tablet 344b4152823f4d618345b14ffd579400 with delete_type TABLET_DATA_TOMBSTONED (TS 83595ad0fd5940c0a2bfc7c61acd969a not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:40138
I20260709 12:58:57.787981  1676 consensus_queue.cc:237] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:58:57.790218  1598 raft_consensus.cc:1275] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:58:57.791312  2585 consensus_queue.cc:1048] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:58:57.806175  1665 tablet_service.cc:1558] Processing DeleteTablet for tablet 14588fa8cfd6467e989746e326a0ed31 with delete_type TABLET_DATA_TOMBSTONED (TS 816fadcea8fc4893882f5f9b9f22a03d not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:52760
I20260709 12:58:57.820819  2593 tablet_replica.cc:333] T 344b4152823f4d618345b14ffd579400 P 83595ad0fd5940c0a2bfc7c61acd969a: stopping tablet replica
I20260709 12:58:57.823995  2515 raft_consensus.cc:2955] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER af520b67945d443b901bbeca37cc5701 (127.0.107.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:58:57.831893  2593 raft_consensus.cc:2243] T 344b4152823f4d618345b14ffd579400 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:58:57.833037  2593 raft_consensus.cc:2272] T 344b4152823f4d618345b14ffd579400 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:58:57.839169  2593 ts_tablet_manager.cc:1916] T 344b4152823f4d618345b14ffd579400 P 83595ad0fd5940c0a2bfc7c61acd969a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:58:57.840268  2336 raft_consensus.cc:2955] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER af520b67945d443b901bbeca37cc5701 (127.0.107.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:58:57.847329  2334 raft_consensus.cc:1275] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:58:57.849732  1751 raft_consensus.cc:1275] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:58:57.852252  2584 consensus_queue.cc:1048] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:58:57.849788  2594 tablet_replica.cc:333] T 14588fa8cfd6467e989746e326a0ed31 P 816fadcea8fc4893882f5f9b9f22a03d: stopping tablet replica
I20260709 12:58:57.860411  2492 consensus_queue.cc:1048] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:58:57.864027  2594 raft_consensus.cc:2243] T 14588fa8cfd6467e989746e326a0ed31 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:58:57.864609  2531 consensus_queue.cc:1048] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:58:57.865155  2594 raft_consensus.cc:2272] T 14588fa8cfd6467e989746e326a0ed31 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:58:57.877810  2594 ts_tablet_manager.cc:1916] T 14588fa8cfd6467e989746e326a0ed31 P 816fadcea8fc4893882f5f9b9f22a03d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:58:57.882896  2531 raft_consensus.cc:2955] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:58:57.887260  2334 raft_consensus.cc:2955] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:58:57.895552  1598 raft_consensus.cc:2955] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER af520b67945d443b901bbeca37cc5701 (127.0.107.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:58:57.920243  1748 raft_consensus.cc:2955] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:58:57.937050  2594 ts_tablet_manager.cc:1929] T 14588fa8cfd6467e989746e326a0ed31 P 816fadcea8fc4893882f5f9b9f22a03d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:58:57.937743  2594 log.cc:1199] T 14588fa8cfd6467e989746e326a0ed31 P 816fadcea8fc4893882f5f9b9f22a03d: Deleting WAL directory at /tmp/dist-test-taskakPCWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783601878574137-431-0/minicluster-data/ts-1-root/wals/14588fa8cfd6467e989746e326a0ed31
I20260709 12:58:57.921075  1500 catalog_manager.cc:5697] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 reported cstate change: config changed from index 3 to 4, VOTER 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) evicted. New cstate: current_term: 1 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:58:57.928509  2593 ts_tablet_manager.cc:1929] T 344b4152823f4d618345b14ffd579400 P 83595ad0fd5940c0a2bfc7c61acd969a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:58:57.939359  2593 log.cc:1199] T 344b4152823f4d618345b14ffd579400 P 83595ad0fd5940c0a2bfc7c61acd969a: Deleting WAL directory at /tmp/dist-test-taskakPCWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783601878574137-431-0/minicluster-data/ts-0-root/wals/344b4152823f4d618345b14ffd579400
I20260709 12:58:57.932564  1488 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet f8ab62ffb73246dc9bcf39f7bee51147 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:58:57.941461  1487 catalog_manager.cc:5028] TS 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): tablet 344b4152823f4d618345b14ffd579400 (table test-workload [id=75d77b57b3f34786b9c215c4f11bd727]) successfully deleted
I20260709 12:58:57.944708  1488 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 513043a9b96c4f70adad53ac4b5d01dc with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:58:57.956961  1501 catalog_manager.cc:5697] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d reported cstate change: config changed from index 3 to 4, VOTER af520b67945d443b901bbeca37cc5701 (127.0.107.195) evicted. New cstate: current_term: 1 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:58:57.962738  1488 catalog_manager.cc:5028] TS 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): tablet 14588fa8cfd6467e989746e326a0ed31 (table test-workload [id=75d77b57b3f34786b9c215c4f11bd727]) successfully deleted
I20260709 12:58:57.983518  1740 tablet_service.cc:1558] Processing DeleteTablet for tablet f8ab62ffb73246dc9bcf39f7bee51147 with delete_type TABLET_DATA_TOMBSTONED (TS af520b67945d443b901bbeca37cc5701 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:55838
I20260709 12:58:57.984895  1592 tablet_service.cc:1558] Processing DeleteTablet for tablet 513043a9b96c4f70adad53ac4b5d01dc with delete_type TABLET_DATA_TOMBSTONED (TS 83595ad0fd5940c0a2bfc7c61acd969a not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:40138
I20260709 12:58:57.994743  2593 tablet_replica.cc:333] T 513043a9b96c4f70adad53ac4b5d01dc P 83595ad0fd5940c0a2bfc7c61acd969a: stopping tablet replica
I20260709 12:58:58.002183  2593 raft_consensus.cc:2243] T 513043a9b96c4f70adad53ac4b5d01dc P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:58:58.004123  2593 raft_consensus.cc:2272] T 513043a9b96c4f70adad53ac4b5d01dc P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:58:58.012857  2596 tablet_replica.cc:333] T f8ab62ffb73246dc9bcf39f7bee51147 P af520b67945d443b901bbeca37cc5701: stopping tablet replica
I20260709 12:58:58.014127  2593 ts_tablet_manager.cc:1916] T 513043a9b96c4f70adad53ac4b5d01dc P 83595ad0fd5940c0a2bfc7c61acd969a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:58:58.014329  2596 raft_consensus.cc:2243] T f8ab62ffb73246dc9bcf39f7bee51147 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:58:58.016371  2596 raft_consensus.cc:2272] T f8ab62ffb73246dc9bcf39f7bee51147 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:58:58.020536  2596 ts_tablet_manager.cc:1916] T f8ab62ffb73246dc9bcf39f7bee51147 P af520b67945d443b901bbeca37cc5701: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:58:58.030812  2593 ts_tablet_manager.cc:1929] T 513043a9b96c4f70adad53ac4b5d01dc P 83595ad0fd5940c0a2bfc7c61acd969a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:58:58.031356  2593 log.cc:1199] T 513043a9b96c4f70adad53ac4b5d01dc P 83595ad0fd5940c0a2bfc7c61acd969a: Deleting WAL directory at /tmp/dist-test-taskakPCWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783601878574137-431-0/minicluster-data/ts-0-root/wals/513043a9b96c4f70adad53ac4b5d01dc
I20260709 12:58:58.032840  1487 catalog_manager.cc:5028] TS 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): tablet 513043a9b96c4f70adad53ac4b5d01dc (table test-workload [id=75d77b57b3f34786b9c215c4f11bd727]) successfully deleted
I20260709 12:58:58.034145  2596 ts_tablet_manager.cc:1929] T f8ab62ffb73246dc9bcf39f7bee51147 P af520b67945d443b901bbeca37cc5701: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:58:58.034586  2596 log.cc:1199] T f8ab62ffb73246dc9bcf39f7bee51147 P af520b67945d443b901bbeca37cc5701: Deleting WAL directory at /tmp/dist-test-taskakPCWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783601878574137-431-0/minicluster-data/ts-2-root/wals/f8ab62ffb73246dc9bcf39f7bee51147
I20260709 12:58:58.035909  1490 catalog_manager.cc:5028] TS af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): tablet f8ab62ffb73246dc9bcf39f7bee51147 (table test-workload [id=75d77b57b3f34786b9c215c4f11bd727]) successfully deleted
I20260709 12:58:59.078188  1598 consensus_queue.cc:237] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } }
I20260709 12:58:59.087006  1751 raft_consensus.cc:1275] T e26344297fff44bb8d2099b8e64d7df2 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:58:59.089396  1677 raft_consensus.cc:1275] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260709 12:58:59.089818  1569 consensus_peers.cc:597] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a -> Peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625): Couldn't send request to peer 3b879799899a499c8d409ee9f1277cd1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e26344297fff44bb8d2099b8e64d7df2. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:58:59.091324  2571 consensus_queue.cc:1048] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:58:59.092818  2500 consensus_queue.cc:1048] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:58:59.108021  1752 raft_consensus.cc:2955] T e26344297fff44bb8d2099b8e64d7df2 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } } }
I20260709 12:58:59.106060  2599 raft_consensus.cc:2955] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } } }
I20260709 12:58:59.109074  1677 raft_consensus.cc:2955] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } } }
I20260709 12:58:59.129760  1598 consensus_queue.cc:237] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } }
I20260709 12:58:59.132500  1503 catalog_manager.cc:5697] T e26344297fff44bb8d2099b8e64d7df2 P af520b67945d443b901bbeca37cc5701 reported cstate change: config changed from index -1 to 2, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New cstate: current_term: 1 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } } }
I20260709 12:58:59.138562  1751 raft_consensus.cc:1275] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:58:59.139701  1676 raft_consensus.cc:1275] T a3accbb733bb4cafa69428319664f239 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:58:59.141726  2599 consensus_queue.cc:1048] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:58:59.142768  2500 consensus_queue.cc:1048] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260709 12:58:59.153443  1569 consensus_peers.cc:597] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a -> Peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625): Couldn't send request to peer 3b879799899a499c8d409ee9f1277cd1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a3accbb733bb4cafa69428319664f239. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:58:59.165005  2500 raft_consensus.cc:2955] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } } }
I20260709 12:58:59.166101  1751 raft_consensus.cc:2955] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } } }
I20260709 12:58:59.171156  1676 raft_consensus.cc:2955] T a3accbb733bb4cafa69428319664f239 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } } }
I20260709 12:58:59.201716  1751 consensus_queue.cc:237] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } }
I20260709 12:58:59.218024  1502 catalog_manager.cc:5697] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: config changed from index -1 to 2, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New cstate: current_term: 1 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260709 12:58:59.223510  1604 raft_consensus.cc:1275] T e4f4b860aab3464383ce8f9d990f4a4a P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:58:59.223768  1677 raft_consensus.cc:1275] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260709 12:58:59.227541  1718 consensus_peers.cc:597] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 -> Peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625): Couldn't send request to peer 3b879799899a499c8d409ee9f1277cd1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e4f4b860aab3464383ce8f9d990f4a4a. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:58:59.230501  2581 consensus_queue.cc:1048] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:58:59.240505  2611 consensus_queue.cc:1048] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:58:59.253757  2581 raft_consensus.cc:2955] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } } }
I20260709 12:58:59.255645  1677 raft_consensus.cc:2955] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } } }
I20260709 12:58:59.255707  1604 raft_consensus.cc:2955] T e4f4b860aab3464383ce8f9d990f4a4a P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } } }
I20260709 12:58:59.269927  1751 consensus_queue.cc:237] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } }
I20260709 12:58:59.275364  1501 catalog_manager.cc:5697] T e4f4b860aab3464383ce8f9d990f4a4a P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: config changed from index -1 to 2, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New cstate: current_term: 1 leader_uuid: "af520b67945d443b901bbeca37cc5701" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } } }
I20260709 12:58:59.289441  1677 raft_consensus.cc:1275] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260709 12:58:59.290757  1718 consensus_peers.cc:597] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 -> Peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625): Couldn't send request to peer 3b879799899a499c8d409ee9f1277cd1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 41166f283aea41b0b3f5be60e8234443. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:58:59.296365  2581 consensus_queue.cc:1048] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:58:59.298005  1604 raft_consensus.cc:1275] T 41166f283aea41b0b3f5be60e8234443 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:58:59.299651  2612 consensus_queue.cc:1048] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:58:59.306540  2612 raft_consensus.cc:2955] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } } }
I20260709 12:58:59.308148  1677 raft_consensus.cc:2955] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } } }
I20260709 12:58:59.309284  1604 raft_consensus.cc:2955] T 41166f283aea41b0b3f5be60e8234443 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } } }
I20260709 12:58:59.338837  1503 catalog_manager.cc:5697] T 41166f283aea41b0b3f5be60e8234443 P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: config changed from index -1 to 2, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New cstate: current_term: 1 leader_uuid: "af520b67945d443b901bbeca37cc5701" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } } }
I20260709 12:58:59.614383  2620 ts_tablet_manager.cc:933] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1: Initiating tablet copy from peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445)
I20260709 12:58:59.620460  2620 tablet_copy_client.cc:323] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Beginning tablet copy session from remote peer at address 127.0.107.193:46445
I20260709 12:58:59.623307  1615 tablet_copy_service.cc:140] P 83595ad0fd5940c0a2bfc7c61acd969a: Received BeginTabletCopySession request for tablet e26344297fff44bb8d2099b8e64d7df2 from peer 3b879799899a499c8d409ee9f1277cd1 ({username='slave'} at 127.0.0.1:36462)
I20260709 12:58:59.624217  1615 tablet_copy_service.cc:161] P 83595ad0fd5940c0a2bfc7c61acd969a: Beginning new tablet copy session on tablet e26344297fff44bb8d2099b8e64d7df2 from peer 3b879799899a499c8d409ee9f1277cd1 at {username='slave'} at 127.0.0.1:36462: session id = 3b879799899a499c8d409ee9f1277cd1-e26344297fff44bb8d2099b8e64d7df2
I20260709 12:58:59.645749  1615 tablet_copy_source_session.cc:215] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:58:59.652228  2620 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e26344297fff44bb8d2099b8e64d7df2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:59.674155  2620 tablet_copy_client.cc:806] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Starting download of 0 data blocks...
I20260709 12:58:59.674878  2620 tablet_copy_client.cc:670] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Starting download of 1 WAL segments...
I20260709 12:58:59.680157  2620 tablet_copy_client.cc:538] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:58:59.690514  2620 tablet_bootstrap.cc:492] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1: Bootstrap starting.
I20260709 12:58:59.693131  2622 ts_tablet_manager.cc:933] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1: Initiating tablet copy from peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445)
I20260709 12:58:59.695523  2622 tablet_copy_client.cc:323] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Beginning tablet copy session from remote peer at address 127.0.107.193:46445
I20260709 12:58:59.697365  1615 tablet_copy_service.cc:140] P 83595ad0fd5940c0a2bfc7c61acd969a: Received BeginTabletCopySession request for tablet a3accbb733bb4cafa69428319664f239 from peer 3b879799899a499c8d409ee9f1277cd1 ({username='slave'} at 127.0.0.1:36462)
I20260709 12:58:59.697961  1615 tablet_copy_service.cc:161] P 83595ad0fd5940c0a2bfc7c61acd969a: Beginning new tablet copy session on tablet a3accbb733bb4cafa69428319664f239 from peer 3b879799899a499c8d409ee9f1277cd1 at {username='slave'} at 127.0.0.1:36462: session id = 3b879799899a499c8d409ee9f1277cd1-a3accbb733bb4cafa69428319664f239
I20260709 12:58:59.725380  1615 tablet_copy_source_session.cc:215] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:58:59.737565  2622 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3accbb733bb4cafa69428319664f239. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:59.758826  2620 tablet_bootstrap.cc:492] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:58:59.761322  2620 tablet_bootstrap.cc:492] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1: Bootstrap complete.
I20260709 12:58:59.762153  2620 ts_tablet_manager.cc:1403] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1: Time spent bootstrapping tablet: real 0.072s	user 0.020s	sys 0.043s
I20260709 12:58:59.763954  2622 tablet_copy_client.cc:806] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Starting download of 0 data blocks...
I20260709 12:58:59.764622  2622 tablet_copy_client.cc:670] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Starting download of 1 WAL segments...
I20260709 12:58:59.766283  2620 raft_consensus.cc:359] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } }
I20260709 12:58:59.767738  2620 raft_consensus.cc:740] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Initialized, Role: LEARNER
I20260709 12:58:59.768534  2620 consensus_queue.cc:260] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } }
I20260709 12:58:59.785152  2620 ts_tablet_manager.cc:1434] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1: Time spent starting tablet: real 0.023s	user 0.007s	sys 0.013s
I20260709 12:58:59.793975  1614 tablet_copy_service.cc:342] P 83595ad0fd5940c0a2bfc7c61acd969a: Request end of tablet copy session 3b879799899a499c8d409ee9f1277cd1-e26344297fff44bb8d2099b8e64d7df2 received from {username='slave'} at 127.0.0.1:36462
I20260709 12:58:59.794643  1614 tablet_copy_service.cc:434] P 83595ad0fd5940c0a2bfc7c61acd969a: ending tablet copy session 3b879799899a499c8d409ee9f1277cd1-e26344297fff44bb8d2099b8e64d7df2 on tablet e26344297fff44bb8d2099b8e64d7df2 with peer 3b879799899a499c8d409ee9f1277cd1
I20260709 12:58:59.797847  2620 ts_tablet_manager.cc:933] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1: Initiating tablet copy from peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:58:59.823591  2622 tablet_copy_client.cc:538] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:58:59.831516  2620 tablet_copy_client.cc:323] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Beginning tablet copy session from remote peer at address 127.0.107.195:40693
I20260709 12:58:59.834121  1764 tablet_copy_service.cc:140] P af520b67945d443b901bbeca37cc5701: Received BeginTabletCopySession request for tablet e4f4b860aab3464383ce8f9d990f4a4a from peer 3b879799899a499c8d409ee9f1277cd1 ({username='slave'} at 127.0.0.1:43156)
I20260709 12:58:59.834798  1764 tablet_copy_service.cc:161] P af520b67945d443b901bbeca37cc5701: Beginning new tablet copy session on tablet e4f4b860aab3464383ce8f9d990f4a4a from peer 3b879799899a499c8d409ee9f1277cd1 at {username='slave'} at 127.0.0.1:43156: session id = 3b879799899a499c8d409ee9f1277cd1-e4f4b860aab3464383ce8f9d990f4a4a
I20260709 12:58:59.844039  2622 tablet_bootstrap.cc:492] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1: Bootstrap starting.
I20260709 12:58:59.854391  1764 tablet_copy_source_session.cc:215] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:58:59.858278  2620 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e4f4b860aab3464383ce8f9d990f4a4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:59.870040  2622 tablet_bootstrap.cc:492] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:58:59.871016  2622 tablet_bootstrap.cc:492] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1: Bootstrap complete.
I20260709 12:58:59.871616  2622 ts_tablet_manager.cc:1403] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1: Time spent bootstrapping tablet: real 0.028s	user 0.022s	sys 0.004s
I20260709 12:58:59.874919  2622 raft_consensus.cc:359] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } }
I20260709 12:58:59.875808  2622 raft_consensus.cc:740] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Initialized, Role: LEARNER
I20260709 12:58:59.876523  2622 consensus_queue.cc:260] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } }
I20260709 12:58:59.878659  2620 tablet_copy_client.cc:806] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Starting download of 0 data blocks...
I20260709 12:58:59.879293  2620 tablet_copy_client.cc:670] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Starting download of 1 WAL segments...
I20260709 12:58:59.881954  2622 ts_tablet_manager.cc:1434] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1: Time spent starting tablet: real 0.010s	user 0.008s	sys 0.000s
I20260709 12:58:59.883817  1615 tablet_copy_service.cc:342] P 83595ad0fd5940c0a2bfc7c61acd969a: Request end of tablet copy session 3b879799899a499c8d409ee9f1277cd1-a3accbb733bb4cafa69428319664f239 received from {username='slave'} at 127.0.0.1:36462
I20260709 12:58:59.883882  2620 tablet_copy_client.cc:538] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:58:59.884230  1615 tablet_copy_service.cc:434] P 83595ad0fd5940c0a2bfc7c61acd969a: ending tablet copy session 3b879799899a499c8d409ee9f1277cd1-a3accbb733bb4cafa69428319664f239 on tablet a3accbb733bb4cafa69428319664f239 with peer 3b879799899a499c8d409ee9f1277cd1
I20260709 12:58:59.895006  2620 tablet_bootstrap.cc:492] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1: Bootstrap starting.
I20260709 12:58:59.908329  2622 ts_tablet_manager.cc:933] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1: Initiating tablet copy from peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:58:59.917313  2622 tablet_copy_client.cc:323] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Beginning tablet copy session from remote peer at address 127.0.107.195:40693
I20260709 12:58:59.920408  1764 tablet_copy_service.cc:140] P af520b67945d443b901bbeca37cc5701: Received BeginTabletCopySession request for tablet 41166f283aea41b0b3f5be60e8234443 from peer 3b879799899a499c8d409ee9f1277cd1 ({username='slave'} at 127.0.0.1:43156)
I20260709 12:58:59.921010  1764 tablet_copy_service.cc:161] P af520b67945d443b901bbeca37cc5701: Beginning new tablet copy session on tablet 41166f283aea41b0b3f5be60e8234443 from peer 3b879799899a499c8d409ee9f1277cd1 at {username='slave'} at 127.0.0.1:43156: session id = 3b879799899a499c8d409ee9f1277cd1-41166f283aea41b0b3f5be60e8234443
I20260709 12:58:59.936661  2620 tablet_bootstrap.cc:492] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:58:59.937862  2620 tablet_bootstrap.cc:492] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1: Bootstrap complete.
I20260709 12:58:59.938673  2620 ts_tablet_manager.cc:1403] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1: Time spent bootstrapping tablet: real 0.044s	user 0.020s	sys 0.010s
I20260709 12:58:59.942042  2620 raft_consensus.cc:359] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } }
I20260709 12:58:59.942970  2620 raft_consensus.cc:740] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Initialized, Role: LEARNER
I20260709 12:58:59.943763  2620 consensus_queue.cc:260] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } }
I20260709 12:58:59.946708  2620 ts_tablet_manager.cc:1434] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1: Time spent starting tablet: real 0.008s	user 0.005s	sys 0.002s
I20260709 12:58:59.948817  1763 tablet_copy_service.cc:342] P af520b67945d443b901bbeca37cc5701: Request end of tablet copy session 3b879799899a499c8d409ee9f1277cd1-e4f4b860aab3464383ce8f9d990f4a4a received from {username='slave'} at 127.0.0.1:43156
I20260709 12:58:59.949252  1763 tablet_copy_service.cc:434] P af520b67945d443b901bbeca37cc5701: ending tablet copy session 3b879799899a499c8d409ee9f1277cd1-e4f4b860aab3464383ce8f9d990f4a4a on tablet e4f4b860aab3464383ce8f9d990f4a4a with peer 3b879799899a499c8d409ee9f1277cd1
I20260709 12:58:59.955423  1764 tablet_copy_source_session.cc:215] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:58:59.962579  2622 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 41166f283aea41b0b3f5be60e8234443. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:58:59.977456  2622 tablet_copy_client.cc:806] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Starting download of 0 data blocks...
I20260709 12:58:59.978087  2622 tablet_copy_client.cc:670] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Starting download of 1 WAL segments...
I20260709 12:58:59.981817  2622 tablet_copy_client.cc:538] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:58:59.991767  2622 tablet_bootstrap.cc:492] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1: Bootstrap starting.
I20260709 12:59:00.014039  2622 tablet_bootstrap.cc:492] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:59:00.015013  2622 tablet_bootstrap.cc:492] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1: Bootstrap complete.
I20260709 12:59:00.015722  2622 ts_tablet_manager.cc:1403] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1: Time spent bootstrapping tablet: real 0.024s	user 0.012s	sys 0.011s
I20260709 12:59:00.019048  2622 raft_consensus.cc:359] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } }
I20260709 12:59:00.019933  2622 raft_consensus.cc:740] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Initialized, Role: LEARNER
I20260709 12:59:00.020604  2622 consensus_queue.cc:260] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } }
I20260709 12:59:00.023335  2622 ts_tablet_manager.cc:1434] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1: Time spent starting tablet: real 0.007s	user 0.008s	sys 0.000s
I20260709 12:59:00.025269  1764 tablet_copy_service.cc:342] P af520b67945d443b901bbeca37cc5701: Request end of tablet copy session 3b879799899a499c8d409ee9f1277cd1-41166f283aea41b0b3f5be60e8234443 received from {username='slave'} at 127.0.0.1:43156
I20260709 12:59:00.025695  1764 tablet_copy_service.cc:434] P af520b67945d443b901bbeca37cc5701: ending tablet copy session 3b879799899a499c8d409ee9f1277cd1-41166f283aea41b0b3f5be60e8234443 on tablet 41166f283aea41b0b3f5be60e8234443 with peer 3b879799899a499c8d409ee9f1277cd1
I20260709 12:59:00.123797  2334 raft_consensus.cc:1217] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:59:00.143939  2334 raft_consensus.cc:1217] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:59:00.335919  2334 raft_consensus.cc:1217] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:59:00.513409  2334 raft_consensus.cc:1217] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:59:00.527210  2500 raft_consensus.cc:1064] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a: attempting to promote NON_VOTER 3b879799899a499c8d409ee9f1277cd1 to VOTER
I20260709 12:59:00.530136  2500 consensus_queue.cc:237] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:00.541092  2334 raft_consensus.cc:1275] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 1 LEARNER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:59:00.543123  1751 raft_consensus.cc:1275] T e26344297fff44bb8d2099b8e64d7df2 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:59:00.544677  1677 raft_consensus.cc:1275] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:59:00.543144  2602 consensus_queue.cc:1048] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20260709 12:59:00.546561  2602 consensus_queue.cc:1048] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:59:00.548352  2602 consensus_queue.cc:1048] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:59:00.571379  2634 raft_consensus.cc:2955] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:00.574054  1751 raft_consensus.cc:2955] T e26344297fff44bb8d2099b8e64d7df2 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:00.577611  1680 raft_consensus.cc:2955] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:00.598091  2334 raft_consensus.cc:2955] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:00.613648  1502 catalog_manager.cc:5697] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d reported cstate change: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:00.654700  2634 raft_consensus.cc:1064] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a: attempting to promote NON_VOTER 3b879799899a499c8d409ee9f1277cd1 to VOTER
I20260709 12:59:00.657631  2634 consensus_queue.cc:237] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:00.672576  1605 consensus_queue.cc:237] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:00.676143  1751 raft_consensus.cc:1275] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:59:00.673972  2334 raft_consensus.cc:1275] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 1 LEARNER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:59:00.677718  1677 raft_consensus.cc:1275] T a3accbb733bb4cafa69428319664f239 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:59:00.681896  2500 consensus_queue.cc:1048] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:59:00.682898  2634 consensus_queue.cc:1048] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:59:00.684558  2571 consensus_queue.cc:1048] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:59:00.695830  1680 raft_consensus.cc:1275] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:59:00.696054  2334 raft_consensus.cc:1275] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:59:00.703014  2571 consensus_queue.cc:1048] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.003s
I20260709 12:59:00.707626  2634 consensus_queue.cc:1048] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:59:00.714010  2500 raft_consensus.cc:2955] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER af520b67945d443b901bbeca37cc5701 (127.0.107.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:00.716300  1680 raft_consensus.cc:2955] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER af520b67945d443b901bbeca37cc5701 (127.0.107.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:00.737493  2646 raft_consensus.cc:2955] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:00.751101  1487 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet e26344297fff44bb8d2099b8e64d7df2 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:59:00.752187  2334 raft_consensus.cc:2955] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:00.764768  1500 catalog_manager.cc:5697] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: config changed from index 3 to 4, VOTER af520b67945d443b901bbeca37cc5701 (127.0.107.195) evicted. New cstate: current_term: 1 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:59:00.765390  1751 raft_consensus.cc:2955] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:00.781708  2335 raft_consensus.cc:2955] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER af520b67945d443b901bbeca37cc5701 (127.0.107.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:00.784341  1740 tablet_service.cc:1558] Processing DeleteTablet for tablet e26344297fff44bb8d2099b8e64d7df2 with delete_type TABLET_DATA_TOMBSTONED (TS af520b67945d443b901bbeca37cc5701 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:55838
I20260709 12:59:00.809959  2648 tablet_replica.cc:333] T e26344297fff44bb8d2099b8e64d7df2 P af520b67945d443b901bbeca37cc5701: stopping tablet replica
I20260709 12:59:00.811405  2648 raft_consensus.cc:2243] T e26344297fff44bb8d2099b8e64d7df2 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:59:00.812166  2648 raft_consensus.cc:2272] T e26344297fff44bb8d2099b8e64d7df2 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:59:00.811014  1680 raft_consensus.cc:2955] T a3accbb733bb4cafa69428319664f239 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:00.813755  1503 catalog_manager.cc:5697] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 reported cstate change: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:00.816650  2648 ts_tablet_manager.cc:1916] T e26344297fff44bb8d2099b8e64d7df2 P af520b67945d443b901bbeca37cc5701: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:59:00.835835  2648 ts_tablet_manager.cc:1929] T e26344297fff44bb8d2099b8e64d7df2 P af520b67945d443b901bbeca37cc5701: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:59:00.836772  2648 log.cc:1199] T e26344297fff44bb8d2099b8e64d7df2 P af520b67945d443b901bbeca37cc5701: Deleting WAL directory at /tmp/dist-test-taskakPCWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783601878574137-431-0/minicluster-data/ts-2-root/wals/e26344297fff44bb8d2099b8e64d7df2
I20260709 12:59:00.838838  1490 catalog_manager.cc:5028] TS af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): tablet e26344297fff44bb8d2099b8e64d7df2 (table test-workload [id=75d77b57b3f34786b9c215c4f11bd727]) successfully deleted
I20260709 12:59:00.856220  1605 consensus_queue.cc:237] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:00.860574  1751 raft_consensus.cc:1275] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:59:00.860750  2334 raft_consensus.cc:1275] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:59:00.862263  2602 consensus_queue.cc:1048] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:59:00.864259  2647 consensus_queue.cc:1048] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:59:00.872239  2602 raft_consensus.cc:2955] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:00.877379  2334 raft_consensus.cc:2955] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:00.891991  1751 raft_consensus.cc:2955] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:00.896641  2550 raft_consensus.cc:1064] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701: attempting to promote NON_VOTER 3b879799899a499c8d409ee9f1277cd1 to VOTER
I20260709 12:59:00.903148  2550 consensus_queue.cc:237] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:00.916280  1487 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet a3accbb733bb4cafa69428319664f239 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:59:00.919734  1501 catalog_manager.cc:5697] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 reported cstate change: config changed from index 3 to 4, VOTER 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194) evicted. New cstate: current_term: 1 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:00.943431  2581 raft_consensus.cc:1064] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701: attempting to promote NON_VOTER 3b879799899a499c8d409ee9f1277cd1 to VOTER
I20260709 12:59:00.943707  2334 raft_consensus.cc:1275] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 1 LEARNER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:59:00.944859  1605 raft_consensus.cc:1275] T e4f4b860aab3464383ce8f9d990f4a4a P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:59:00.946058  1665 tablet_service.cc:1558] Processing DeleteTablet for tablet a3accbb733bb4cafa69428319664f239 with delete_type TABLET_DATA_TOMBSTONED (TS 816fadcea8fc4893882f5f9b9f22a03d not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:52760
I20260709 12:59:00.948206  2581 consensus_queue.cc:237] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:00.950907  2651 tablet_replica.cc:333] T a3accbb733bb4cafa69428319664f239 P 816fadcea8fc4893882f5f9b9f22a03d: stopping tablet replica
I20260709 12:59:00.951268  2632 consensus_queue.cc:1048] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:59:00.952299  2651 raft_consensus.cc:2243] T a3accbb733bb4cafa69428319664f239 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:59:00.953161  2651 raft_consensus.cc:2272] T a3accbb733bb4cafa69428319664f239 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:59:00.953733  1677 raft_consensus.cc:1275] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:59:00.954015  2550 consensus_queue.cc:1048] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:59:00.956899  2650 consensus_queue.cc:1048] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:59:00.959520  2651 ts_tablet_manager.cc:1916] T a3accbb733bb4cafa69428319664f239 P 816fadcea8fc4893882f5f9b9f22a03d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:59:00.970067  2335 raft_consensus.cc:1275] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 1 LEARNER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:59:00.970533  1680 raft_consensus.cc:1275] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:59:00.972807  2632 consensus_queue.cc:1048] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:59:00.975114  1604 raft_consensus.cc:1275] T 41166f283aea41b0b3f5be60e8234443 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:59:00.977298  2550 consensus_queue.cc:1048] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:59:00.986649  2652 consensus_queue.cc:1048] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.009s
I20260709 12:59:00.994721  2651 ts_tablet_manager.cc:1929] T a3accbb733bb4cafa69428319664f239 P 816fadcea8fc4893882f5f9b9f22a03d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:59:01.008342  2550 raft_consensus.cc:2955] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:01.020169  2651 log.cc:1199] T a3accbb733bb4cafa69428319664f239 P 816fadcea8fc4893882f5f9b9f22a03d: Deleting WAL directory at /tmp/dist-test-taskakPCWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783601878574137-431-0/minicluster-data/ts-1-root/wals/a3accbb733bb4cafa69428319664f239
I20260709 12:59:01.013407  1604 raft_consensus.cc:2955] T e4f4b860aab3464383ce8f9d990f4a4a P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:01.022426  1488 catalog_manager.cc:5028] TS 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): tablet a3accbb733bb4cafa69428319664f239 (table test-workload [id=75d77b57b3f34786b9c215c4f11bd727]) successfully deleted
I20260709 12:59:01.021553  2632 raft_consensus.cc:2955] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:01.024724  2335 raft_consensus.cc:2955] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:01.030695  1677 raft_consensus.cc:2955] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:01.044615  1503 catalog_manager.cc:5697] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 reported cstate change: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "af520b67945d443b901bbeca37cc5701" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:01.045737  1680 raft_consensus.cc:2955] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:01.046463  1598 raft_consensus.cc:2955] T 41166f283aea41b0b3f5be60e8234443 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:01.061429  1500 catalog_manager.cc:5697] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d reported cstate change: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "af520b67945d443b901bbeca37cc5701" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:01.063898  2335 raft_consensus.cc:2955] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:01.073079  1751 consensus_queue.cc:237] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:01.083256  2334 raft_consensus.cc:1275] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:59:01.087535  2632 consensus_queue.cc:1048] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:59:01.099720  2650 raft_consensus.cc:2955] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:01.106204  2335 raft_consensus.cc:2955] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:01.117436  1680 raft_consensus.cc:1275] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:59:01.118978  2550 consensus_queue.cc:1048] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:59:01.125444  1677 raft_consensus.cc:2955] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:01.134276  1490 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 41166f283aea41b0b3f5be60e8234443 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:59:01.137493  1500 catalog_manager.cc:5697] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 reported cstate change: config changed from index 3 to 4, VOTER 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) evicted. New cstate: current_term: 1 leader_uuid: "af520b67945d443b901bbeca37cc5701" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:01.140276  1752 consensus_queue.cc:237] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:01.145678  2335 raft_consensus.cc:1275] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:59:01.147341  1677 raft_consensus.cc:1275] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:59:01.148634  2650 consensus_queue.cc:1048] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:59:01.154579  2650 consensus_queue.cc:1048] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:59:01.163149  2652 raft_consensus.cc:2955] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:01.169167  1592 tablet_service.cc:1558] Processing DeleteTablet for tablet 41166f283aea41b0b3f5be60e8234443 with delete_type TABLET_DATA_TOMBSTONED (Replica with old config index 3 (current committed config index is 4)) from {username='slave'} at 127.0.0.1:40138
I20260709 12:59:01.187968  2335 raft_consensus.cc:2955] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:01.187968  1677 raft_consensus.cc:2955] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:01.193859  2660 tablet_replica.cc:333] T 41166f283aea41b0b3f5be60e8234443 P 83595ad0fd5940c0a2bfc7c61acd969a: stopping tablet replica
I20260709 12:59:01.195749  2660 raft_consensus.cc:2243] T 41166f283aea41b0b3f5be60e8234443 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:59:01.205641  1490 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet e4f4b860aab3464383ce8f9d990f4a4a with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:59:01.211328  1503 catalog_manager.cc:5697] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d reported cstate change: config changed from index 3 to 4, VOTER 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) evicted. New cstate: current_term: 1 leader_uuid: "af520b67945d443b901bbeca37cc5701" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:01.248911  2660 raft_consensus.cc:2272] T 41166f283aea41b0b3f5be60e8234443 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:59:01.258810  2660 ts_tablet_manager.cc:1916] T 41166f283aea41b0b3f5be60e8234443 P 83595ad0fd5940c0a2bfc7c61acd969a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:59:01.263643  1592 tablet_service.cc:1558] Processing DeleteTablet for tablet e4f4b860aab3464383ce8f9d990f4a4a with delete_type TABLET_DATA_TOMBSTONED (TS 83595ad0fd5940c0a2bfc7c61acd969a not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:40138
I20260709 12:59:01.282582  2660 ts_tablet_manager.cc:1929] T 41166f283aea41b0b3f5be60e8234443 P 83595ad0fd5940c0a2bfc7c61acd969a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:59:01.283253  2660 log.cc:1199] T 41166f283aea41b0b3f5be60e8234443 P 83595ad0fd5940c0a2bfc7c61acd969a: Deleting WAL directory at /tmp/dist-test-taskakPCWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783601878574137-431-0/minicluster-data/ts-0-root/wals/41166f283aea41b0b3f5be60e8234443
I20260709 12:59:01.285560  1487 catalog_manager.cc:5028] TS 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): tablet 41166f283aea41b0b3f5be60e8234443 (table test-workload [id=75d77b57b3f34786b9c215c4f11bd727]) successfully deleted
I20260709 12:59:01.285909  2660 tablet_replica.cc:333] T e4f4b860aab3464383ce8f9d990f4a4a P 83595ad0fd5940c0a2bfc7c61acd969a: stopping tablet replica
I20260709 12:59:01.287684  2660 raft_consensus.cc:2243] T e4f4b860aab3464383ce8f9d990f4a4a P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:59:01.288534  2660 raft_consensus.cc:2272] T e4f4b860aab3464383ce8f9d990f4a4a P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:59:01.292033  2660 ts_tablet_manager.cc:1916] T e4f4b860aab3464383ce8f9d990f4a4a P 83595ad0fd5940c0a2bfc7c61acd969a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:59:01.311269  2660 ts_tablet_manager.cc:1929] T e4f4b860aab3464383ce8f9d990f4a4a P 83595ad0fd5940c0a2bfc7c61acd969a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:59:01.311689  2660 log.cc:1199] T e4f4b860aab3464383ce8f9d990f4a4a P 83595ad0fd5940c0a2bfc7c61acd969a: Deleting WAL directory at /tmp/dist-test-taskakPCWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783601878574137-431-0/minicluster-data/ts-0-root/wals/e4f4b860aab3464383ce8f9d990f4a4a
I20260709 12:59:01.313246  1487 catalog_manager.cc:5028] TS 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): tablet e4f4b860aab3464383ce8f9d990f4a4a (table test-workload [id=75d77b57b3f34786b9c215c4f11bd727]) successfully deleted
I20260709 12:59:02.628681  1745 consensus_queue.cc:237] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } }
W20260709 12:59:02.645152  1718 consensus_peers.cc:597] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 -> Peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625): Couldn't send request to peer 3b879799899a499c8d409ee9f1277cd1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1809d674329a492dbea0b60a4703fe51. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:59:02.647409  1673 raft_consensus.cc:1275] T 1809d674329a492dbea0b60a4703fe51 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:59:02.654717  2673 consensus_queue.cc:1048] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:59:02.678514  2550 raft_consensus.cc:2955] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } } }
I20260709 12:59:02.687333  1673 raft_consensus.cc:2955] T 1809d674329a492dbea0b60a4703fe51 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } } }
I20260709 12:59:02.695384  1600 raft_consensus.cc:1275] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:59:02.704581  1598 consensus_queue.cc:237] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } }
I20260709 12:59:02.705318  1502 catalog_manager.cc:5697] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 reported cstate change: config changed from index -1 to 2, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New cstate: current_term: 1 leader_uuid: "af520b67945d443b901bbeca37cc5701" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260709 12:59:02.715828  2670 consensus_queue.cc:1048] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260709 12:59:02.720489  1569 consensus_peers.cc:597] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a -> Peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625): Couldn't send request to peer 3b879799899a499c8d409ee9f1277cd1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d21485c094584689917b2a97b2d3c624. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:59:02.729619  1678 raft_consensus.cc:1275] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:59:02.731417  2662 consensus_queue.cc:1048] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
I20260709 12:59:02.733222  1751 raft_consensus.cc:1275] T d21485c094584689917b2a97b2d3c624 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:59:02.734915  1600 raft_consensus.cc:2955] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } } }
I20260709 12:59:02.737108  2661 consensus_queue.cc:1048] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:59:02.754089  2571 raft_consensus.cc:2955] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } } }
I20260709 12:59:02.756333  1678 raft_consensus.cc:2955] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } } }
I20260709 12:59:02.765748  1751 raft_consensus.cc:2955] T d21485c094584689917b2a97b2d3c624 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } } }
I20260709 12:59:02.782420  1502 catalog_manager.cc:5697] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d reported cstate change: config changed from index -1 to 2, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New cstate: current_term: 1 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } } }
I20260709 12:59:02.786201  1604 consensus_queue.cc:237] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } }
I20260709 12:59:02.809752  1751 raft_consensus.cc:1275] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:59:02.810680  1673 raft_consensus.cc:1275] T cd44a2ebdc764e898f25ad5390f4bffd P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260709 12:59:02.810838  1569 consensus_peers.cc:597] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a -> Peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625): Couldn't send request to peer 3b879799899a499c8d409ee9f1277cd1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cd44a2ebdc764e898f25ad5390f4bffd. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:59:02.812723  2661 consensus_queue.cc:1048] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:59:02.815477  2571 consensus_queue.cc:1048] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:59:02.824338  2663 raft_consensus.cc:2955] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } } }
I20260709 12:59:02.828208  1745 raft_consensus.cc:2955] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } } }
I20260709 12:59:02.828691  1673 raft_consensus.cc:2955] T cd44a2ebdc764e898f25ad5390f4bffd P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } } }
I20260709 12:59:02.841226  1745 consensus_queue.cc:237] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } }
I20260709 12:59:02.845240  1502 catalog_manager.cc:5697] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 reported cstate change: config changed from index -1 to 2, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New cstate: current_term: 1 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } } }
I20260709 12:59:02.858302  1604 raft_consensus.cc:1275] T 1c2f5fa72dee4ec58979ace307d2acc3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:59:02.859153  1673 raft_consensus.cc:1275] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260709 12:59:02.860247  1718 consensus_peers.cc:597] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 -> Peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625): Couldn't send request to peer 3b879799899a499c8d409ee9f1277cd1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1c2f5fa72dee4ec58979ace307d2acc3. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:59:02.860982  2673 consensus_queue.cc:1048] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:59:02.861832  2550 consensus_queue.cc:1048] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:59:02.882162  2673 raft_consensus.cc:2955] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } } }
I20260709 12:59:02.884712  1678 raft_consensus.cc:2955] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } } }
I20260709 12:59:02.885067  1604 raft_consensus.cc:2955] T 1c2f5fa72dee4ec58979ace307d2acc3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } } }
I20260709 12:59:02.907306  1502 catalog_manager.cc:5697] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 reported cstate change: config changed from index -1 to 2, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New cstate: current_term: 1 leader_uuid: "af520b67945d443b901bbeca37cc5701" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260709 12:59:03.140404  2690 ts_tablet_manager.cc:933] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1: Initiating tablet copy from peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:03.144374  2690 tablet_copy_client.cc:323] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Beginning tablet copy session from remote peer at address 127.0.107.195:40693
I20260709 12:59:03.146765  1764 tablet_copy_service.cc:140] P af520b67945d443b901bbeca37cc5701: Received BeginTabletCopySession request for tablet 1809d674329a492dbea0b60a4703fe51 from peer 3b879799899a499c8d409ee9f1277cd1 ({username='slave'} at 127.0.0.1:43156)
I20260709 12:59:03.147291  1764 tablet_copy_service.cc:161] P af520b67945d443b901bbeca37cc5701: Beginning new tablet copy session on tablet 1809d674329a492dbea0b60a4703fe51 from peer 3b879799899a499c8d409ee9f1277cd1 at {username='slave'} at 127.0.0.1:43156: session id = 3b879799899a499c8d409ee9f1277cd1-1809d674329a492dbea0b60a4703fe51
I20260709 12:59:03.157398  1764 tablet_copy_source_session.cc:215] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:59:03.161782  2690 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1809d674329a492dbea0b60a4703fe51. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:59:03.179720  2690 tablet_copy_client.cc:806] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Starting download of 0 data blocks...
I20260709 12:59:03.180394  2690 tablet_copy_client.cc:670] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Starting download of 1 WAL segments...
I20260709 12:59:03.183977  2690 tablet_copy_client.cc:538] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:59:03.192811  2690 tablet_bootstrap.cc:492] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1: Bootstrap starting.
I20260709 12:59:03.237277  2690 tablet_bootstrap.cc:492] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:59:03.238709  2690 tablet_bootstrap.cc:492] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1: Bootstrap complete.
I20260709 12:59:03.239727  2690 ts_tablet_manager.cc:1403] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1: Time spent bootstrapping tablet: real 0.047s	user 0.022s	sys 0.018s
I20260709 12:59:03.244721  2690 raft_consensus.cc:359] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } }
I20260709 12:59:03.245848  2690 raft_consensus.cc:740] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Initialized, Role: LEARNER
I20260709 12:59:03.246846  2690 consensus_queue.cc:260] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } }
I20260709 12:59:03.254717  2690 ts_tablet_manager.cc:1434] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1: Time spent starting tablet: real 0.015s	user 0.009s	sys 0.005s
I20260709 12:59:03.257979  1764 tablet_copy_service.cc:342] P af520b67945d443b901bbeca37cc5701: Request end of tablet copy session 3b879799899a499c8d409ee9f1277cd1-1809d674329a492dbea0b60a4703fe51 received from {username='slave'} at 127.0.0.1:43156
I20260709 12:59:03.258502  1764 tablet_copy_service.cc:434] P af520b67945d443b901bbeca37cc5701: ending tablet copy session 3b879799899a499c8d409ee9f1277cd1-1809d674329a492dbea0b60a4703fe51 on tablet 1809d674329a492dbea0b60a4703fe51 with peer 3b879799899a499c8d409ee9f1277cd1
I20260709 12:59:03.282766  2690 ts_tablet_manager.cc:933] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1: Initiating tablet copy from peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445)
I20260709 12:59:03.285241  2690 tablet_copy_client.cc:323] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Beginning tablet copy session from remote peer at address 127.0.107.193:46445
I20260709 12:59:03.287434  1615 tablet_copy_service.cc:140] P 83595ad0fd5940c0a2bfc7c61acd969a: Received BeginTabletCopySession request for tablet d21485c094584689917b2a97b2d3c624 from peer 3b879799899a499c8d409ee9f1277cd1 ({username='slave'} at 127.0.0.1:36462)
I20260709 12:59:03.288172  1615 tablet_copy_service.cc:161] P 83595ad0fd5940c0a2bfc7c61acd969a: Beginning new tablet copy session on tablet d21485c094584689917b2a97b2d3c624 from peer 3b879799899a499c8d409ee9f1277cd1 at {username='slave'} at 127.0.0.1:36462: session id = 3b879799899a499c8d409ee9f1277cd1-d21485c094584689917b2a97b2d3c624
I20260709 12:59:03.300252  1615 tablet_copy_source_session.cc:215] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:59:03.304301  2695 ts_tablet_manager.cc:933] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1: Initiating tablet copy from peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:03.310627  2690 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d21485c094584689917b2a97b2d3c624. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:59:03.319736  2695 tablet_copy_client.cc:323] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Beginning tablet copy session from remote peer at address 127.0.107.195:40693
I20260709 12:59:03.333606  1764 tablet_copy_service.cc:140] P af520b67945d443b901bbeca37cc5701: Received BeginTabletCopySession request for tablet 1c2f5fa72dee4ec58979ace307d2acc3 from peer 3b879799899a499c8d409ee9f1277cd1 ({username='slave'} at 127.0.0.1:43156)
I20260709 12:59:03.334391  1764 tablet_copy_service.cc:161] P af520b67945d443b901bbeca37cc5701: Beginning new tablet copy session on tablet 1c2f5fa72dee4ec58979ace307d2acc3 from peer 3b879799899a499c8d409ee9f1277cd1 at {username='slave'} at 127.0.0.1:43156: session id = 3b879799899a499c8d409ee9f1277cd1-1c2f5fa72dee4ec58979ace307d2acc3
I20260709 12:59:03.342475  2690 tablet_copy_client.cc:806] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Starting download of 0 data blocks...
I20260709 12:59:03.344683  2690 tablet_copy_client.cc:670] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Starting download of 1 WAL segments...
I20260709 12:59:03.344923  1764 tablet_copy_source_session.cc:215] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:59:03.351788  2690 tablet_copy_client.cc:538] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:59:03.353385  2695 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c2f5fa72dee4ec58979ace307d2acc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:59:03.366428  2690 tablet_bootstrap.cc:492] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1: Bootstrap starting.
I20260709 12:59:03.394586  2695 tablet_copy_client.cc:806] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Starting download of 0 data blocks...
I20260709 12:59:03.395742  2695 tablet_copy_client.cc:670] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Starting download of 1 WAL segments...
I20260709 12:59:03.399712  2690 tablet_bootstrap.cc:492] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:59:03.401216  2690 tablet_bootstrap.cc:492] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1: Bootstrap complete.
I20260709 12:59:03.402061  2690 ts_tablet_manager.cc:1403] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1: Time spent bootstrapping tablet: real 0.036s	user 0.020s	sys 0.017s
I20260709 12:59:03.402829  2695 tablet_copy_client.cc:538] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:59:03.411000  2690 raft_consensus.cc:359] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } }
I20260709 12:59:03.412317  2690 raft_consensus.cc:740] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Initialized, Role: LEARNER
I20260709 12:59:03.446406  2690 consensus_queue.cc:260] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } }
W20260709 12:59:03.457680  1626 debug-util.cc:398] Leaking SignalData structure 0x7b08000bc980 after lost signal to thread 434
W20260709 12:59:03.459165  1626 debug-util.cc:398] Leaking SignalData structure 0x7b08001d18a0 after lost signal to thread 1537
W20260709 12:59:03.460721  1626 debug-util.cc:398] Leaking SignalData structure 0x7b08002a2780 after lost signal to thread 1629
I20260709 12:59:03.462370  2695 tablet_bootstrap.cc:492] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1: Bootstrap starting.
W20260709 12:59:03.473006  1626 debug-util.cc:398] Leaking SignalData structure 0x7b08000face0 after lost signal to thread 1704
W20260709 12:59:03.487723  1626 debug-util.cc:398] Leaking SignalData structure 0x7b0800302a00 after lost signal to thread 1778
W20260709 12:59:03.490479  1626 debug-util.cc:398] Leaking SignalData structure 0x7b08001b5620 after lost signal to thread 2361
I20260709 12:59:03.490578  2690 ts_tablet_manager.cc:1434] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1: Time spent starting tablet: real 0.088s	user 0.009s	sys 0.004s
I20260709 12:59:03.495085  1615 tablet_copy_service.cc:342] P 83595ad0fd5940c0a2bfc7c61acd969a: Request end of tablet copy session 3b879799899a499c8d409ee9f1277cd1-d21485c094584689917b2a97b2d3c624 received from {username='slave'} at 127.0.0.1:36462
I20260709 12:59:03.495779  1615 tablet_copy_service.cc:434] P 83595ad0fd5940c0a2bfc7c61acd969a: ending tablet copy session 3b879799899a499c8d409ee9f1277cd1-d21485c094584689917b2a97b2d3c624 on tablet d21485c094584689917b2a97b2d3c624 with peer 3b879799899a499c8d409ee9f1277cd1
I20260709 12:59:03.499590  2690 ts_tablet_manager.cc:933] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1: Initiating tablet copy from peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445)
I20260709 12:59:03.517102  2690 tablet_copy_client.cc:323] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Beginning tablet copy session from remote peer at address 127.0.107.193:46445
I20260709 12:59:03.520578  1615 tablet_copy_service.cc:140] P 83595ad0fd5940c0a2bfc7c61acd969a: Received BeginTabletCopySession request for tablet cd44a2ebdc764e898f25ad5390f4bffd from peer 3b879799899a499c8d409ee9f1277cd1 ({username='slave'} at 127.0.0.1:36462)
I20260709 12:59:03.521440  1615 tablet_copy_service.cc:161] P 83595ad0fd5940c0a2bfc7c61acd969a: Beginning new tablet copy session on tablet cd44a2ebdc764e898f25ad5390f4bffd from peer 3b879799899a499c8d409ee9f1277cd1 at {username='slave'} at 127.0.0.1:36462: session id = 3b879799899a499c8d409ee9f1277cd1-cd44a2ebdc764e898f25ad5390f4bffd
I20260709 12:59:03.546890  2695 tablet_bootstrap.cc:492] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:59:03.548425  2695 tablet_bootstrap.cc:492] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1: Bootstrap complete.
I20260709 12:59:03.549558  2695 ts_tablet_manager.cc:1403] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1: Time spent bootstrapping tablet: real 0.088s	user 0.024s	sys 0.014s
I20260709 12:59:03.658370  1615 tablet_copy_source_session.cc:215] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:59:03.667802  2695 raft_consensus.cc:359] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } }
I20260709 12:59:03.669309  2695 raft_consensus.cc:740] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Initialized, Role: LEARNER
I20260709 12:59:03.670280  2695 consensus_queue.cc:260] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } }
I20260709 12:59:03.699386  2695 ts_tablet_manager.cc:1434] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1: Time spent starting tablet: real 0.149s	user 0.023s	sys 0.003s
I20260709 12:59:03.719991  2690 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd44a2ebdc764e898f25ad5390f4bffd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:59:03.780054  2334 raft_consensus.cc:1217] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:59:03.771343  1764 tablet_copy_service.cc:342] P af520b67945d443b901bbeca37cc5701: Request end of tablet copy session 3b879799899a499c8d409ee9f1277cd1-1c2f5fa72dee4ec58979ace307d2acc3 received from {username='slave'} at 127.0.0.1:43156
I20260709 12:59:03.829614  2333 raft_consensus.cc:1217] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
W20260709 12:59:03.835595  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 283197us
I20260709 12:59:03.827231  1764 tablet_copy_service.cc:434] P af520b67945d443b901bbeca37cc5701: ending tablet copy session 3b879799899a499c8d409ee9f1277cd1-1c2f5fa72dee4ec58979ace307d2acc3 on tablet 1c2f5fa72dee4ec58979ace307d2acc3 with peer 3b879799899a499c8d409ee9f1277cd1
I20260709 12:59:03.864059  2333 raft_consensus.cc:1217] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
W20260709 12:59:03.913487  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 124294us
W20260709 12:59:03.942704  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 87856.5us
W20260709 12:59:03.958498  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 141919us
I20260709 12:59:04.039743  2690 tablet_copy_client.cc:806] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Starting download of 0 data blocks...
I20260709 12:59:04.040462  2690 tablet_copy_client.cc:670] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Starting download of 1 WAL segments...
I20260709 12:59:04.047122  2690 tablet_copy_client.cc:538] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:59:04.070122  2690 tablet_bootstrap.cc:492] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1: Bootstrap starting.
I20260709 12:59:04.100546  2690 tablet_bootstrap.cc:492] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:59:04.101651  2690 tablet_bootstrap.cc:492] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1: Bootstrap complete.
I20260709 12:59:04.102531  2690 ts_tablet_manager.cc:1403] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1: Time spent bootstrapping tablet: real 0.036s	user 0.027s	sys 0.005s
I20260709 12:59:04.105693  2690 raft_consensus.cc:359] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } }
I20260709 12:59:04.106557  2690 raft_consensus.cc:740] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Initialized, Role: LEARNER
I20260709 12:59:04.107290  2690 consensus_queue.cc:260] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } }
I20260709 12:59:04.118961  2690 ts_tablet_manager.cc:1434] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1: Time spent starting tablet: real 0.016s	user 0.003s	sys 0.003s
I20260709 12:59:04.124959  1615 tablet_copy_service.cc:342] P 83595ad0fd5940c0a2bfc7c61acd969a: Request end of tablet copy session 3b879799899a499c8d409ee9f1277cd1-cd44a2ebdc764e898f25ad5390f4bffd received from {username='slave'} at 127.0.0.1:36462
I20260709 12:59:04.125424  1615 tablet_copy_service.cc:434] P 83595ad0fd5940c0a2bfc7c61acd969a: ending tablet copy session 3b879799899a499c8d409ee9f1277cd1-cd44a2ebdc764e898f25ad5390f4bffd on tablet cd44a2ebdc764e898f25ad5390f4bffd with peer 3b879799899a499c8d409ee9f1277cd1
I20260709 12:59:04.289031  2708 raft_consensus.cc:1064] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701: attempting to promote NON_VOTER 3b879799899a499c8d409ee9f1277cd1 to VOTER
I20260709 12:59:04.292335  2708 consensus_queue.cc:237] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:04.357249  2650 raft_consensus.cc:1064] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701: attempting to promote NON_VOTER 3b879799899a499c8d409ee9f1277cd1 to VOTER
I20260709 12:59:04.360455  2650 consensus_queue.cc:237] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:04.370611  2333 raft_consensus.cc:1275] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 1 LEARNER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:59:04.373739  1674 raft_consensus.cc:1275] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:59:04.375694  1598 raft_consensus.cc:1275] T 1c2f5fa72dee4ec58979ace307d2acc3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:59:04.375610  2708 consensus_queue.cc:1048] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:59:04.379487  2711 consensus_queue.cc:1048] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:59:04.385453  2708 consensus_queue.cc:1048] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:59:04.394789  2721 raft_consensus.cc:1064] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a: attempting to promote NON_VOTER 3b879799899a499c8d409ee9f1277cd1 to VOTER
I20260709 12:59:04.398136  2721 consensus_queue.cc:237] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:04.443341  2337 raft_consensus.cc:1275] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 1 LEARNER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:59:04.445662  2673 raft_consensus.cc:2955] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:04.445660  2708 consensus_queue.cc:1048] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.008s
I20260709 12:59:04.452847  1674 raft_consensus.cc:2955] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:04.455456  2333 raft_consensus.cc:2955] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:04.463932  1677 raft_consensus.cc:1275] T 1809d674329a492dbea0b60a4703fe51 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:59:04.466539  1745 raft_consensus.cc:1275] T d21485c094584689917b2a97b2d3c624 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:59:04.468586  1678 raft_consensus.cc:1275] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:59:04.471807  2337 raft_consensus.cc:1275] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 1 LEARNER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:59:04.472191  2713 consensus_queue.cc:1048] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:59:04.476773  2698 consensus_queue.cc:1048] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:59:04.482254  2713 consensus_queue.cc:1048] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:59:04.488415  2711 consensus_queue.cc:1048] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:59:04.553546  1599 raft_consensus.cc:1275] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:59:04.555894  2711 consensus_queue.cc:1048] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:59:04.567554  2734 raft_consensus.cc:2955] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:04.569974  1599 raft_consensus.cc:2955] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:04.589557  1598 raft_consensus.cc:2955] T 1c2f5fa72dee4ec58979ace307d2acc3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:04.592552  2335 raft_consensus.cc:1217] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:59:04.594774  2334 raft_consensus.cc:2955] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:04.599184  1673 raft_consensus.cc:2955] T 1809d674329a492dbea0b60a4703fe51 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:04.632344  1500 catalog_manager.cc:5697] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 reported cstate change: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "af520b67945d443b901bbeca37cc5701" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:04.640120  2721 raft_consensus.cc:2955] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:04.659401  1751 raft_consensus.cc:2955] T d21485c094584689917b2a97b2d3c624 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:04.667105  1674 raft_consensus.cc:2955] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:04.689126  2334 raft_consensus.cc:2955] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:04.787045  1503 catalog_manager.cc:5697] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 reported cstate change: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "af520b67945d443b901bbeca37cc5701" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:59:04.819566  1501 catalog_manager.cc:5697] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d reported cstate change: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:04.828802  1751 consensus_queue.cc:237] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:04.833161  1745 consensus_queue.cc:237] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:04.835268  2337 raft_consensus.cc:1275] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:59:04.837998  1599 raft_consensus.cc:1275] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:59:04.842283  2708 consensus_queue.cc:1048] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:59:04.847584  2708 consensus_queue.cc:1048] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:59:04.849265  2334 raft_consensus.cc:1275] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:59:04.852263  2650 consensus_queue.cc:1048] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:59:04.866080  1673 raft_consensus.cc:1275] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:59:04.873749  2711 raft_consensus.cc:2955] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:04.882496  1599 consensus_queue.cc:237] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:04.886632  2337 raft_consensus.cc:2955] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:04.875501  2650 raft_consensus.cc:2955] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:04.892594  1602 raft_consensus.cc:2955] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:04.907618  2708 consensus_queue.cc:1048] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:59:04.917419  2333 raft_consensus.cc:1275] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:59:04.917825  1673 raft_consensus.cc:1275] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:59:04.919994  2721 consensus_queue.cc:1048] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:59:04.923862  2720 consensus_queue.cc:1048] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:59:04.942081  1678 raft_consensus.cc:2955] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:04.954660  2720 raft_consensus.cc:2955] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER af520b67945d443b901bbeca37cc5701 (127.0.107.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:04.971508  2333 raft_consensus.cc:2955] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER af520b67945d443b901bbeca37cc5701 (127.0.107.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:04.887233  2334 raft_consensus.cc:2955] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:05.004890  1501 catalog_manager.cc:5697] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 reported cstate change: config changed from index 3 to 4, VOTER 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) evicted. New cstate: current_term: 1 leader_uuid: "af520b67945d443b901bbeca37cc5701" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:59:04.992936  1673 raft_consensus.cc:2955] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER af520b67945d443b901bbeca37cc5701 (127.0.107.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:05.022806  1592 tablet_service.cc:1558] Processing DeleteTablet for tablet 1c2f5fa72dee4ec58979ace307d2acc3 with delete_type TABLET_DATA_TOMBSTONED (TS 83595ad0fd5940c0a2bfc7c61acd969a not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:40138
I20260709 12:59:05.032657  1490 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 1c2f5fa72dee4ec58979ace307d2acc3 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:59:05.097620  1490 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 1809d674329a492dbea0b60a4703fe51 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:59:05.090579  1500 catalog_manager.cc:5697] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 reported cstate change: config changed from index 3 to 4, VOTER 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194) evicted. New cstate: current_term: 1 leader_uuid: "af520b67945d443b901bbeca37cc5701" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:05.114929  2740 tablet_replica.cc:333] T 1c2f5fa72dee4ec58979ace307d2acc3 P 83595ad0fd5940c0a2bfc7c61acd969a: stopping tablet replica
I20260709 12:59:05.118818  1487 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet d21485c094584689917b2a97b2d3c624 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:59:05.142910  1665 tablet_service.cc:1558] Processing DeleteTablet for tablet 1809d674329a492dbea0b60a4703fe51 with delete_type TABLET_DATA_TOMBSTONED (TS 816fadcea8fc4893882f5f9b9f22a03d not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:52760
I20260709 12:59:05.165483  2740 raft_consensus.cc:2243] T 1c2f5fa72dee4ec58979ace307d2acc3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:59:05.166841  2740 raft_consensus.cc:2272] T 1c2f5fa72dee4ec58979ace307d2acc3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:59:05.170300  2741 tablet_replica.cc:333] T 1809d674329a492dbea0b60a4703fe51 P 816fadcea8fc4893882f5f9b9f22a03d: stopping tablet replica
I20260709 12:59:05.185544  2740 ts_tablet_manager.cc:1916] T 1c2f5fa72dee4ec58979ace307d2acc3 P 83595ad0fd5940c0a2bfc7c61acd969a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:59:05.161564  1500 catalog_manager.cc:5697] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 reported cstate change: config changed from index 3 to 4, VOTER af520b67945d443b901bbeca37cc5701 (127.0.107.195) evicted. New cstate: current_term: 1 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:05.196013  2741 raft_consensus.cc:2243] T 1809d674329a492dbea0b60a4703fe51 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:59:05.199870  2741 raft_consensus.cc:2272] T 1809d674329a492dbea0b60a4703fe51 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:59:05.207863  2713 raft_consensus.cc:1064] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a: attempting to promote NON_VOTER 3b879799899a499c8d409ee9f1277cd1 to VOTER
I20260709 12:59:05.208454  2741 ts_tablet_manager.cc:1916] T 1809d674329a492dbea0b60a4703fe51 P 816fadcea8fc4893882f5f9b9f22a03d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:59:05.211051  2713 consensus_queue.cc:237] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:05.223955  1740 tablet_service.cc:1558] Processing DeleteTablet for tablet d21485c094584689917b2a97b2d3c624 with delete_type TABLET_DATA_TOMBSTONED (TS af520b67945d443b901bbeca37cc5701 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:55838
I20260709 12:59:05.229900  2333 raft_consensus.cc:1275] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 1 LEARNER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:59:05.231585  2720 consensus_queue.cc:1048] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:59:05.237197  1672 raft_consensus.cc:1275] T cd44a2ebdc764e898f25ad5390f4bffd P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:59:05.238947  2713 consensus_queue.cc:1048] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:59:05.257007  1752 raft_consensus.cc:1275] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:59:05.265165  2721 consensus_queue.cc:1048] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20260709 12:59:05.319034  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 47528.3us
I20260709 12:59:05.348016  2720 raft_consensus.cc:2955] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:05.350948  2743 tablet_replica.cc:333] T d21485c094584689917b2a97b2d3c624 P af520b67945d443b901bbeca37cc5701: stopping tablet replica
I20260709 12:59:05.351024  1752 raft_consensus.cc:2955] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:05.369159  2333 raft_consensus.cc:2955] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:05.387790  1501 catalog_manager.cc:5697] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:59:05.376744  2743 raft_consensus.cc:2243] T d21485c094584689917b2a97b2d3c624 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:59:05.404431  2743 raft_consensus.cc:2272] T d21485c094584689917b2a97b2d3c624 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:59:05.412925  2740 ts_tablet_manager.cc:1929] T 1c2f5fa72dee4ec58979ace307d2acc3 P 83595ad0fd5940c0a2bfc7c61acd969a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:59:05.414523  2740 log.cc:1199] T 1c2f5fa72dee4ec58979ace307d2acc3 P 83595ad0fd5940c0a2bfc7c61acd969a: Deleting WAL directory at /tmp/dist-test-taskakPCWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783601878574137-431-0/minicluster-data/ts-0-root/wals/1c2f5fa72dee4ec58979ace307d2acc3
I20260709 12:59:05.413973  2741 ts_tablet_manager.cc:1929] T 1809d674329a492dbea0b60a4703fe51 P 816fadcea8fc4893882f5f9b9f22a03d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:59:05.418040  1487 catalog_manager.cc:5028] TS 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): tablet 1c2f5fa72dee4ec58979ace307d2acc3 (table test-workload [id=75d77b57b3f34786b9c215c4f11bd727]) successfully deleted
I20260709 12:59:05.418270  2741 log.cc:1199] T 1809d674329a492dbea0b60a4703fe51 P 816fadcea8fc4893882f5f9b9f22a03d: Deleting WAL directory at /tmp/dist-test-taskakPCWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783601878574137-431-0/minicluster-data/ts-1-root/wals/1809d674329a492dbea0b60a4703fe51
I20260709 12:59:05.423831  1488 catalog_manager.cc:5028] TS 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): tablet 1809d674329a492dbea0b60a4703fe51 (table test-workload [id=75d77b57b3f34786b9c215c4f11bd727]) successfully deleted
I20260709 12:59:05.427670  1672 raft_consensus.cc:2955] T cd44a2ebdc764e898f25ad5390f4bffd P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:05.437402  1599 consensus_queue.cc:237] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:05.451920  2337 raft_consensus.cc:1275] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:59:05.453934  2713 consensus_queue.cc:1048] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:59:05.467671  1751 raft_consensus.cc:1275] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:59:05.436944  2743 ts_tablet_manager.cc:1916] T d21485c094584689917b2a97b2d3c624 P af520b67945d443b901bbeca37cc5701: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:59:05.472539  2713 consensus_queue.cc:1048] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:59:05.476430  2698 raft_consensus.cc:2955] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:05.492057  1751 raft_consensus.cc:2955] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:05.521015  1502 catalog_manager.cc:5697] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 reported cstate change: config changed from index 3 to 4, VOTER 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194) evicted. New cstate: current_term: 1 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:05.532099  2743 ts_tablet_manager.cc:1929] T d21485c094584689917b2a97b2d3c624 P af520b67945d443b901bbeca37cc5701: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:59:05.532886  2743 log.cc:1199] T d21485c094584689917b2a97b2d3c624 P af520b67945d443b901bbeca37cc5701: Deleting WAL directory at /tmp/dist-test-taskakPCWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783601878574137-431-0/minicluster-data/ts-2-root/wals/d21485c094584689917b2a97b2d3c624
I20260709 12:59:05.536161  1490 catalog_manager.cc:5028] TS af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): tablet d21485c094584689917b2a97b2d3c624 (table test-workload [id=75d77b57b3f34786b9c215c4f11bd727]) successfully deleted
I20260709 12:59:05.539013  1665 tablet_service.cc:1558] Processing DeleteTablet for tablet cd44a2ebdc764e898f25ad5390f4bffd with delete_type TABLET_DATA_TOMBSTONED (TS 816fadcea8fc4893882f5f9b9f22a03d not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:52760
I20260709 12:59:05.541590  2741 tablet_replica.cc:333] T cd44a2ebdc764e898f25ad5390f4bffd P 816fadcea8fc4893882f5f9b9f22a03d: stopping tablet replica
I20260709 12:59:05.543081  2741 raft_consensus.cc:2243] T cd44a2ebdc764e898f25ad5390f4bffd P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:59:05.543993  2741 raft_consensus.cc:2272] T cd44a2ebdc764e898f25ad5390f4bffd P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:59:05.549955  2741 ts_tablet_manager.cc:1916] T cd44a2ebdc764e898f25ad5390f4bffd P 816fadcea8fc4893882f5f9b9f22a03d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:59:05.562882  2333 raft_consensus.cc:2955] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:05.596083  1487 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet cd44a2ebdc764e898f25ad5390f4bffd with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:59:05.621654  2741 ts_tablet_manager.cc:1929] T cd44a2ebdc764e898f25ad5390f4bffd P 816fadcea8fc4893882f5f9b9f22a03d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:59:05.622285  2741 log.cc:1199] T cd44a2ebdc764e898f25ad5390f4bffd P 816fadcea8fc4893882f5f9b9f22a03d: Deleting WAL directory at /tmp/dist-test-taskakPCWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783601878574137-431-0/minicluster-data/ts-1-root/wals/cd44a2ebdc764e898f25ad5390f4bffd
I20260709 12:59:05.625101  1488 catalog_manager.cc:5028] TS 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): tablet cd44a2ebdc764e898f25ad5390f4bffd (table test-workload [id=75d77b57b3f34786b9c215c4f11bd727]) successfully deleted
I20260709 12:59:06.635190  1672 consensus_queue.cc:237] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } }
I20260709 12:59:06.647287  1751 raft_consensus.cc:1275] T 2ebc67196b4f4119981cdb499e3350cc P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:59:06.648816  2665 consensus_queue.cc:1048] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260709 12:59:06.649959  1644 consensus_peers.cc:597] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d -> Peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625): Couldn't send request to peer 3b879799899a499c8d409ee9f1277cd1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2ebc67196b4f4119981cdb499e3350cc. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:59:06.655045  1598 raft_consensus.cc:1275] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:59:06.664472  2702 consensus_queue.cc:1048] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:59:06.679378  2702 raft_consensus.cc:2955] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } } }
I20260709 12:59:06.683303  1751 raft_consensus.cc:2955] T 2ebc67196b4f4119981cdb499e3350cc P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } } }
I20260709 12:59:06.726884  1502 catalog_manager.cc:5697] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d reported cstate change: config changed from index -1 to 2, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New cstate: current_term: 1 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260709 12:59:06.729429  1602 raft_consensus.cc:2955] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } } }
I20260709 12:59:06.746042  1678 consensus_queue.cc:237] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } }
W20260709 12:59:06.769294  1644 consensus_peers.cc:597] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d -> Peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625): Couldn't send request to peer 3b879799899a499c8d409ee9f1277cd1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2a521fb6c3e7489d9694fd81cadeb33f. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:59:06.770411  1604 raft_consensus.cc:1275] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:59:06.773577  1751 raft_consensus.cc:1275] T 2a521fb6c3e7489d9694fd81cadeb33f P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:59:06.775184  2671 consensus_queue.cc:1048] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:59:06.782312  2702 consensus_queue.cc:1048] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:59:06.794616  2671 raft_consensus.cc:2955] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } } }
I20260709 12:59:06.797945  1604 raft_consensus.cc:2955] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } } }
I20260709 12:59:06.802697  1751 raft_consensus.cc:2955] T 2a521fb6c3e7489d9694fd81cadeb33f P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } } }
I20260709 12:59:06.824782  1745 consensus_queue.cc:237] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } }
I20260709 12:59:06.833746  1502 catalog_manager.cc:5697] T 2a521fb6c3e7489d9694fd81cadeb33f P af520b67945d443b901bbeca37cc5701 reported cstate change: config changed from index -1 to 2, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New cstate: current_term: 1 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } } }
W20260709 12:59:06.848771  1718 consensus_peers.cc:597] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 -> Peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625): Couldn't send request to peer 3b879799899a499c8d409ee9f1277cd1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 86abe5695d5643f888190301b88dacef. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:59:06.849730  1678 raft_consensus.cc:1275] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:59:06.851888  1604 raft_consensus.cc:1275] T 86abe5695d5643f888190301b88dacef P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:59:06.853139  2673 consensus_queue.cc:1048] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:59:06.854043  2670 consensus_queue.cc:1048] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:59:06.874534  2670 raft_consensus.cc:2955] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } } }
I20260709 12:59:06.874779  1678 raft_consensus.cc:2955] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } } }
I20260709 12:59:06.896809  1604 raft_consensus.cc:2955] T 86abe5695d5643f888190301b88dacef P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } } }
I20260709 12:59:06.905016  1502 catalog_manager.cc:5697] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d reported cstate change: config changed from index -1 to 2, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New cstate: current_term: 1 leader_uuid: "af520b67945d443b901bbeca37cc5701" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } } }
I20260709 12:59:06.949074  1599 consensus_queue.cc:237] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } }
W20260709 12:59:06.972303  1569 consensus_peers.cc:597] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a -> Peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625): Couldn't send request to peer 3b879799899a499c8d409ee9f1277cd1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1d9ac76dc2c445edb5a25cf9087998bb. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:59:06.972307  1678 raft_consensus.cc:1275] T 1d9ac76dc2c445edb5a25cf9087998bb P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:59:06.974150  2751 consensus_queue.cc:1048] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:59:06.978832  1751 raft_consensus.cc:1275] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:59:06.982658  2720 consensus_queue.cc:1048] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:59:07.016948  2751 raft_consensus.cc:2955] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } } }
I20260709 12:59:07.031096  1672 raft_consensus.cc:2955] T 1d9ac76dc2c445edb5a25cf9087998bb P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } } }
I20260709 12:59:07.087512  2769 ts_tablet_manager.cc:933] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1: Initiating tablet copy from peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:07.098382  1745 raft_consensus.cc:2955] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } } }
W20260709 12:59:07.144558  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 67133.7us
I20260709 12:59:07.174073  1500 catalog_manager.cc:5697] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: config changed from index -1 to 2, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New cstate: current_term: 1 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260709 12:59:07.191941  2769 tablet_copy_client.cc:323] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Beginning tablet copy session from remote peer at address 127.0.107.194:46261
I20260709 12:59:07.194882  1689 tablet_copy_service.cc:140] P 816fadcea8fc4893882f5f9b9f22a03d: Received BeginTabletCopySession request for tablet 2ebc67196b4f4119981cdb499e3350cc from peer 3b879799899a499c8d409ee9f1277cd1 ({username='slave'} at 127.0.0.1:57092)
I20260709 12:59:07.195626  1689 tablet_copy_service.cc:161] P 816fadcea8fc4893882f5f9b9f22a03d: Beginning new tablet copy session on tablet 2ebc67196b4f4119981cdb499e3350cc from peer 3b879799899a499c8d409ee9f1277cd1 at {username='slave'} at 127.0.0.1:57092: session id = 3b879799899a499c8d409ee9f1277cd1-2ebc67196b4f4119981cdb499e3350cc
I20260709 12:59:07.219668  2773 ts_tablet_manager.cc:933] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1: Initiating tablet copy from peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:07.251941  2773 tablet_copy_client.cc:323] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Beginning tablet copy session from remote peer at address 127.0.107.194:46261
I20260709 12:59:07.255038  1690 tablet_copy_service.cc:140] P 816fadcea8fc4893882f5f9b9f22a03d: Received BeginTabletCopySession request for tablet 2a521fb6c3e7489d9694fd81cadeb33f from peer 3b879799899a499c8d409ee9f1277cd1 ({username='slave'} at 127.0.0.1:57092)
I20260709 12:59:07.255913  1690 tablet_copy_service.cc:161] P 816fadcea8fc4893882f5f9b9f22a03d: Beginning new tablet copy session on tablet 2a521fb6c3e7489d9694fd81cadeb33f from peer 3b879799899a499c8d409ee9f1277cd1 at {username='slave'} at 127.0.0.1:57092: session id = 3b879799899a499c8d409ee9f1277cd1-2a521fb6c3e7489d9694fd81cadeb33f
I20260709 12:59:07.270319  1689 tablet_copy_source_session.cc:215] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:59:07.274760  1690 tablet_copy_source_session.cc:215] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:59:07.280769  2769 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ebc67196b4f4119981cdb499e3350cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:59:07.285068  2773 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2a521fb6c3e7489d9694fd81cadeb33f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:59:07.318094  2773 tablet_copy_client.cc:806] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Starting download of 0 data blocks...
I20260709 12:59:07.319013  2773 tablet_copy_client.cc:670] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Starting download of 1 WAL segments...
I20260709 12:59:07.336438  2773 tablet_copy_client.cc:538] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:59:07.345469  2776 ts_tablet_manager.cc:933] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1: Initiating tablet copy from peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:07.363567  2773 tablet_bootstrap.cc:492] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1: Bootstrap starting.
I20260709 12:59:07.359467  2769 tablet_copy_client.cc:806] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Starting download of 0 data blocks...
I20260709 12:59:07.364836  2769 tablet_copy_client.cc:670] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Starting download of 1 WAL segments...
I20260709 12:59:07.366380  2776 tablet_copy_client.cc:323] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Beginning tablet copy session from remote peer at address 127.0.107.195:40693
I20260709 12:59:07.368638  1764 tablet_copy_service.cc:140] P af520b67945d443b901bbeca37cc5701: Received BeginTabletCopySession request for tablet 86abe5695d5643f888190301b88dacef from peer 3b879799899a499c8d409ee9f1277cd1 ({username='slave'} at 127.0.0.1:43156)
I20260709 12:59:07.369445  1764 tablet_copy_service.cc:161] P af520b67945d443b901bbeca37cc5701: Beginning new tablet copy session on tablet 86abe5695d5643f888190301b88dacef from peer 3b879799899a499c8d409ee9f1277cd1 at {username='slave'} at 127.0.0.1:43156: session id = 3b879799899a499c8d409ee9f1277cd1-86abe5695d5643f888190301b88dacef
I20260709 12:59:07.377368  2769 tablet_copy_client.cc:538] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:59:07.385893  1764 tablet_copy_source_session.cc:215] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:59:07.403010  2776 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 86abe5695d5643f888190301b88dacef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:59:07.444595  2769 tablet_bootstrap.cc:492] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1: Bootstrap starting.
I20260709 12:59:07.481668  2773 tablet_bootstrap.cc:492] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:59:07.483367  2773 tablet_bootstrap.cc:492] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1: Bootstrap complete.
I20260709 12:59:07.484481  2773 ts_tablet_manager.cc:1403] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1: Time spent bootstrapping tablet: real 0.121s	user 0.026s	sys 0.017s
I20260709 12:59:07.490415  2776 tablet_copy_client.cc:806] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Starting download of 0 data blocks...
I20260709 12:59:07.499455  2776 tablet_copy_client.cc:670] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Starting download of 1 WAL segments...
I20260709 12:59:07.490428  2773 raft_consensus.cc:359] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } }
I20260709 12:59:07.504294  2773 raft_consensus.cc:740] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Initialized, Role: LEARNER
I20260709 12:59:07.505609  2773 consensus_queue.cc:260] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } }
I20260709 12:59:07.509440  2780 ts_tablet_manager.cc:933] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1: Initiating tablet copy from peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445)
I20260709 12:59:07.515977  2776 tablet_copy_client.cc:538] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:59:07.531258  2773 ts_tablet_manager.cc:1434] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1: Time spent starting tablet: real 0.046s	user 0.011s	sys 0.005s
I20260709 12:59:07.546478  1690 tablet_copy_service.cc:342] P 816fadcea8fc4893882f5f9b9f22a03d: Request end of tablet copy session 3b879799899a499c8d409ee9f1277cd1-2a521fb6c3e7489d9694fd81cadeb33f received from {username='slave'} at 127.0.0.1:57092
I20260709 12:59:07.547405  1690 tablet_copy_service.cc:434] P 816fadcea8fc4893882f5f9b9f22a03d: ending tablet copy session 3b879799899a499c8d409ee9f1277cd1-2a521fb6c3e7489d9694fd81cadeb33f on tablet 2a521fb6c3e7489d9694fd81cadeb33f with peer 3b879799899a499c8d409ee9f1277cd1
I20260709 12:59:07.548660  2780 tablet_copy_client.cc:323] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Beginning tablet copy session from remote peer at address 127.0.107.193:46445
I20260709 12:59:07.551194  1615 tablet_copy_service.cc:140] P 83595ad0fd5940c0a2bfc7c61acd969a: Received BeginTabletCopySession request for tablet 1d9ac76dc2c445edb5a25cf9087998bb from peer 3b879799899a499c8d409ee9f1277cd1 ({username='slave'} at 127.0.0.1:36462)
I20260709 12:59:07.552850  1615 tablet_copy_service.cc:161] P 83595ad0fd5940c0a2bfc7c61acd969a: Beginning new tablet copy session on tablet 1d9ac76dc2c445edb5a25cf9087998bb from peer 3b879799899a499c8d409ee9f1277cd1 at {username='slave'} at 127.0.0.1:36462: session id = 3b879799899a499c8d409ee9f1277cd1-1d9ac76dc2c445edb5a25cf9087998bb
I20260709 12:59:07.562402  2776 tablet_bootstrap.cc:492] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1: Bootstrap starting.
I20260709 12:59:07.563683  1615 tablet_copy_source_session.cc:215] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:59:07.573519  2780 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d9ac76dc2c445edb5a25cf9087998bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:59:07.577343   431 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:59:07.578212  2769 tablet_bootstrap.cc:492] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:59:07.579270  2769 tablet_bootstrap.cc:492] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1: Bootstrap complete.
I20260709 12:59:07.580194  2769 ts_tablet_manager.cc:1403] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1: Time spent bootstrapping tablet: real 0.136s	user 0.017s	sys 0.036s
I20260709 12:59:07.591499  2769 raft_consensus.cc:359] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } }
I20260709 12:59:07.592344  2769 raft_consensus.cc:740] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Initialized, Role: LEARNER
I20260709 12:59:07.600889  2780 tablet_copy_client.cc:806] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Starting download of 0 data blocks...
I20260709 12:59:07.602196  2780 tablet_copy_client.cc:670] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Starting download of 1 WAL segments...
I20260709 12:59:07.612972  2333 raft_consensus.cc:1217] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:59:07.593055  2769 consensus_queue.cc:260] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } }
I20260709 12:59:07.623499  2780 tablet_copy_client.cc:538] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:59:07.629228  2769 ts_tablet_manager.cc:1434] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1: Time spent starting tablet: real 0.049s	user 0.006s	sys 0.005s
I20260709 12:59:07.639854  1690 tablet_copy_service.cc:342] P 816fadcea8fc4893882f5f9b9f22a03d: Request end of tablet copy session 3b879799899a499c8d409ee9f1277cd1-2ebc67196b4f4119981cdb499e3350cc received from {username='slave'} at 127.0.0.1:57092
I20260709 12:59:07.640497  1690 tablet_copy_service.cc:434] P 816fadcea8fc4893882f5f9b9f22a03d: ending tablet copy session 3b879799899a499c8d409ee9f1277cd1-2ebc67196b4f4119981cdb499e3350cc on tablet 2ebc67196b4f4119981cdb499e3350cc with peer 3b879799899a499c8d409ee9f1277cd1
I20260709 12:59:07.673979  2780 tablet_bootstrap.cc:492] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1: Bootstrap starting.
I20260709 12:59:07.688925  2776 tablet_bootstrap.cc:492] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:59:07.691336  2776 tablet_bootstrap.cc:492] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1: Bootstrap complete.
I20260709 12:59:07.692296  2776 ts_tablet_manager.cc:1403] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1: Time spent bootstrapping tablet: real 0.130s	user 0.023s	sys 0.022s
I20260709 12:59:07.708366  2776 raft_consensus.cc:359] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } }
I20260709 12:59:07.718274  2776 raft_consensus.cc:740] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Initialized, Role: LEARNER
I20260709 12:59:07.719087  2776 consensus_queue.cc:260] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } }
I20260709 12:59:07.724814  2776 ts_tablet_manager.cc:1434] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1: Time spent starting tablet: real 0.032s	user 0.010s	sys 0.002s
I20260709 12:59:07.728044  1764 tablet_copy_service.cc:342] P af520b67945d443b901bbeca37cc5701: Request end of tablet copy session 3b879799899a499c8d409ee9f1277cd1-86abe5695d5643f888190301b88dacef received from {username='slave'} at 127.0.0.1:43156
I20260709 12:59:07.728482  1764 tablet_copy_service.cc:434] P af520b67945d443b901bbeca37cc5701: ending tablet copy session 3b879799899a499c8d409ee9f1277cd1-86abe5695d5643f888190301b88dacef on tablet 86abe5695d5643f888190301b88dacef with peer 3b879799899a499c8d409ee9f1277cd1
I20260709 12:59:07.765200  2780 tablet_bootstrap.cc:492] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:59:07.766292  2780 tablet_bootstrap.cc:492] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1: Bootstrap complete.
I20260709 12:59:07.767167  2780 ts_tablet_manager.cc:1403] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1: Time spent bootstrapping tablet: real 0.094s	user 0.017s	sys 0.020s
I20260709 12:59:07.770773  2780 raft_consensus.cc:359] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } }
I20260709 12:59:07.771662  2780 raft_consensus.cc:740] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Initialized, Role: LEARNER
I20260709 12:59:07.772336  2780 consensus_queue.cc:260] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } }
I20260709 12:59:07.775122  2780 ts_tablet_manager.cc:1434] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.001s
I20260709 12:59:07.777118  1615 tablet_copy_service.cc:342] P 83595ad0fd5940c0a2bfc7c61acd969a: Request end of tablet copy session 3b879799899a499c8d409ee9f1277cd1-1d9ac76dc2c445edb5a25cf9087998bb received from {username='slave'} at 127.0.0.1:36462
I20260709 12:59:07.777665  1615 tablet_copy_service.cc:434] P 83595ad0fd5940c0a2bfc7c61acd969a: ending tablet copy session 3b879799899a499c8d409ee9f1277cd1-1d9ac76dc2c445edb5a25cf9087998bb on tablet 1d9ac76dc2c445edb5a25cf9087998bb with peer 3b879799899a499c8d409ee9f1277cd1
I20260709 12:59:07.811570  1677 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72"
dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
mode: GRACEFUL
new_leader_uuid: "af520b67945d443b901bbeca37cc5701"
 from {username='slave'} at 127.0.0.1:43266
I20260709 12:59:07.812374  1677 raft_consensus.cc:606] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 LEADER]: Received request to transfer leadership to TS af520b67945d443b901bbeca37cc5701
I20260709 12:59:07.816293  1677 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "87b86d46388a41cd90374c203cb27914"
dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
mode: GRACEFUL
new_leader_uuid: "3b879799899a499c8d409ee9f1277cd1"
 from {username='slave'} at 127.0.0.1:43266
I20260709 12:59:07.816880  1677 raft_consensus.cc:606] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 LEADER]: Received request to transfer leadership to TS 3b879799899a499c8d409ee9f1277cd1
I20260709 12:59:07.820529  1677 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "a40e423ed583480280352b261913276c"
dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
mode: GRACEFUL
new_leader_uuid: "3b879799899a499c8d409ee9f1277cd1"
 from {username='slave'} at 127.0.0.1:43266
I20260709 12:59:07.821031  1677 raft_consensus.cc:606] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 1 LEADER]: Received request to transfer leadership to TS 3b879799899a499c8d409ee9f1277cd1
I20260709 12:59:07.842635  1599 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81"
dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
mode: GRACEFUL
new_leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
 from {username='slave'} at 127.0.0.1:47752
I20260709 12:59:07.843191  1599 raft_consensus.cc:606] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 LEADER]: Received request to transfer leadership to TS 816fadcea8fc4893882f5f9b9f22a03d
I20260709 12:59:07.867300  1752 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a"
dest_uuid: "af520b67945d443b901bbeca37cc5701"
mode: GRACEFUL
new_leader_uuid: "3b879799899a499c8d409ee9f1277cd1"
 from {username='slave'} at 127.0.0.1:51270
I20260709 12:59:07.867947  1752 raft_consensus.cc:606] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 1 LEADER]: Received request to transfer leadership to TS 3b879799899a499c8d409ee9f1277cd1
I20260709 12:59:07.871526  1672 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147"
dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
mode: GRACEFUL
new_leader_uuid: "3b879799899a499c8d409ee9f1277cd1"
 from {username='slave'} at 127.0.0.1:43266
I20260709 12:59:07.872192  1672 raft_consensus.cc:606] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 LEADER]: Received request to transfer leadership to TS 3b879799899a499c8d409ee9f1277cd1
I20260709 12:59:07.873695   431 auto_leader_rebalancer.cc:452] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260709 12:59:07.875937   431 auto_leader_rebalancer.cc:527] All tables' leader rebalancing finished this round
I20260709 12:59:07.883598  2333 raft_consensus.cc:1217] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:59:07.887955  2333 raft_consensus.cc:1217] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:59:08.009876  2775 raft_consensus.cc:993] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d: : Instructing follower af520b67945d443b901bbeca37cc5701 to start an election
I20260709 12:59:08.010586  2775 raft_consensus.cc:1081] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 LEADER]: Signalling peer af520b67945d443b901bbeca37cc5701 to start an election
I20260709 12:59:08.012918  1752 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72"
dest_uuid: "af520b67945d443b901bbeca37cc5701"
 from {username='slave'} at 127.0.0.1:55848
I20260709 12:59:08.013669  1752 raft_consensus.cc:493] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:59:08.014032  1752 raft_consensus.cc:3060] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:08.030170  1752 raft_consensus.cc:515] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:08.036671  1752 leader_election.cc:290] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 2 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:08.043258  2337 raft_consensus.cc:1217] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:59:08.057571  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:59:08.058220  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:59:08.058467  1677 raft_consensus.cc:3055] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:59:08.058928  1599 raft_consensus.cc:3060] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:08.058893  1677 raft_consensus.cc:740] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 12:59:08.062597  1677 consensus_queue.cc:260] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:08.063967  1677 raft_consensus.cc:3060] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:08.071278  1599 raft_consensus.cc:2468] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 2.
I20260709 12:59:08.077353  1717 leader_election.cc:304] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:59:08.078279  2670 raft_consensus.cc:2804] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:59:08.079005  2670 raft_consensus.cc:697] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 2 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 12:59:08.079803  2670 consensus_queue.cc:237] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:08.082370  1677 raft_consensus.cc:2468] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 2.
I20260709 12:59:08.102973  1500 catalog_manager.cc:5697] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 reported cstate change: term changed from 1 to 2, leader changed from 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194) to af520b67945d443b901bbeca37cc5701 (127.0.107.195). New cstate: current_term: 2 leader_uuid: "af520b67945d443b901bbeca37cc5701" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: HEALTHY } } }
I20260709 12:59:08.176200  2720 raft_consensus.cc:993] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a: : Instructing follower 816fadcea8fc4893882f5f9b9f22a03d to start an election
I20260709 12:59:08.176824  2721 raft_consensus.cc:1081] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 LEADER]: Signalling peer 816fadcea8fc4893882f5f9b9f22a03d to start an election
I20260709 12:59:08.178771  1677 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81"
dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
 from {username='slave'} at 127.0.0.1:52766
I20260709 12:59:08.179387  1677 raft_consensus.cc:493] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:59:08.179755  1677 raft_consensus.cc:3060] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:08.191190  1677 raft_consensus.cc:515] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:08.194954  1677 leader_election.cc:290] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 2 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:08.201344  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:59:08.202013  1599 raft_consensus.cc:3055] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:59:08.202330  1599 raft_consensus.cc:740] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 12:59:08.202369  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:08.203161  1745 raft_consensus.cc:3060] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:08.203118  1599 consensus_queue.cc:260] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:08.204648  1599 raft_consensus.cc:3060] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:08.215204  2671 raft_consensus.cc:1064] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d: attempting to promote NON_VOTER 3b879799899a499c8d409ee9f1277cd1 to VOTER
I20260709 12:59:08.226087  1599 raft_consensus.cc:2468] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 2.
I20260709 12:59:08.227258  1643 leader_election.cc:304] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 12:59:08.227502  2775 raft_consensus.cc:993] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d: : Instructing follower 3b879799899a499c8d409ee9f1277cd1 to start an election
I20260709 12:59:08.231366  2702 raft_consensus.cc:2804] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:59:08.231971  2702 raft_consensus.cc:697] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 12:59:08.233014  2702 consensus_queue.cc:237] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:08.235312  2795 raft_consensus.cc:1081] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 LEADER]: Signalling peer 3b879799899a499c8d409ee9f1277cd1 to start an election
I20260709 12:59:08.217877  2671 consensus_queue.cc:237] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:08.237591  2337 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147"
dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
 from {username='slave'} at 127.0.0.1:39726
I20260709 12:59:08.238154  2337 raft_consensus.cc:493] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:59:08.238529  2337 raft_consensus.cc:3060] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:08.247673  2337 raft_consensus.cc:515] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:08.251894  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:59:08.251993  2337 leader_election.cc:290] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 2 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:08.252234  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:59:08.252538  1599 raft_consensus.cc:3060] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:08.252954  1672 raft_consensus.cc:3055] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:59:08.253329  1672 raft_consensus.cc:740] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 12:59:08.254199  1672 consensus_queue.cc:260] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:08.255661  1672 raft_consensus.cc:3060] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:08.265612  1599 raft_consensus.cc:2468] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 2.
I20260709 12:59:08.266825  2298 leader_election.cc:304] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 12:59:08.268206  2781 raft_consensus.cc:2804] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:59:08.286154  2670 raft_consensus.cc:1064] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701: attempting to promote NON_VOTER 3b879799899a499c8d409ee9f1277cd1 to VOTER
I20260709 12:59:08.288784  2670 consensus_queue.cc:237] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:08.305539  2337 raft_consensus.cc:1275] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 1 LEARNER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:59:08.306649  1599 raft_consensus.cc:1275] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:59:08.307767  2793 consensus_queue.cc:1048] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:59:08.311662  2795 consensus_queue.cc:1048] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:59:08.332267  1745 raft_consensus.cc:2468] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 2.
I20260709 12:59:08.403272  1752 raft_consensus.cc:1275] T 2a521fb6c3e7489d9694fd81cadeb33f P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:59:08.433202  1604 raft_consensus.cc:1275] T 86abe5695d5643f888190301b88dacef P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
W20260709 12:59:08.463127  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 175717us
I20260709 12:59:08.433471  2781 raft_consensus.cc:697] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 2 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
W20260709 12:59:08.471822  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 70183.1us
I20260709 12:59:08.468849  2764 consensus_queue.cc:1048] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:59:08.469189  1672 raft_consensus.cc:2468] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 2.
I20260709 12:59:08.490041  2337 raft_consensus.cc:1275] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 1 LEARNER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:59:08.491582  1500 catalog_manager.cc:5697] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d reported cstate change: term changed from 1 to 2, leader changed from 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) to 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194). New cstate: current_term: 2 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } }
I20260709 12:59:08.493359  2795 consensus_queue.cc:1048] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:59:08.495445  2806 raft_consensus.cc:993] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d: : Instructing follower 3b879799899a499c8d409ee9f1277cd1 to start an election
I20260709 12:59:08.495925  2806 raft_consensus.cc:1081] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 LEADER]: Signalling peer 3b879799899a499c8d409ee9f1277cd1 to start an election
I20260709 12:59:08.504343  2781 consensus_queue.cc:237] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:08.566359  2802 consensus_queue.cc:1048] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20260709 12:59:08.571521  1678 raft_consensus.cc:1275] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:59:08.573128  1672 raft_consensus.cc:1275] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:59:08.583982  2772 raft_consensus.cc:1064] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a: attempting to promote NON_VOTER 3b879799899a499c8d409ee9f1277cd1 to VOTER
I20260709 12:59:08.586977  2772 consensus_queue.cc:237] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:08.599248  2806 raft_consensus.cc:2955] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:08.601843  2673 raft_consensus.cc:993] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701: : Instructing follower 3b879799899a499c8d409ee9f1277cd1 to start an election
I20260709 12:59:08.602388  2673 raft_consensus.cc:1081] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 1 LEADER]: Signalling peer 3b879799899a499c8d409ee9f1277cd1 to start an election
I20260709 12:59:08.606879  1745 raft_consensus.cc:2955] T 2a521fb6c3e7489d9694fd81cadeb33f P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:08.616643  1500 catalog_manager.cc:5697] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 reported cstate change: term changed from 1 to 2, leader changed from 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194) to 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196). New cstate: current_term: 2 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:59:08.611446  1604 raft_consensus.cc:2955] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:08.624938  2335 raft_consensus.cc:1275] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 1 LEARNER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:59:08.627528  2721 consensus_queue.cc:1048] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:59:08.638712  2334 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a"
dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
 from {username='slave'} at 127.0.0.1:39738
I20260709 12:59:08.639343  2334 raft_consensus.cc:493] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:59:08.639735  2334 raft_consensus.cc:3060] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:08.660748  2334 raft_consensus.cc:515] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:08.682927  1677 raft_consensus.cc:1275] T 1d9ac76dc2c445edb5a25cf9087998bb P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:59:08.689221  2334 leader_election.cc:290] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 2 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:08.693784  1502 catalog_manager.cc:5697] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d reported cstate change: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:59:08.697646  2720 consensus_queue.cc:1048] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:59:08.699381  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:08.700021  1745 raft_consensus.cc:3055] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:59:08.700316  1745 raft_consensus.cc:740] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 12:59:08.700709  2670 consensus_queue.cc:1048] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:59:08.701226  1745 consensus_queue.cc:260] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:08.714121  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:59:08.715176  1672 raft_consensus.cc:3060] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:08.698637  2764 consensus_queue.cc:1048] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.018s
I20260709 12:59:08.727175  1672 raft_consensus.cc:2468] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 2.
W20260709 12:59:08.727572  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 289407us
I20260709 12:59:08.730676  2333 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "87b86d46388a41cd90374c203cb27914"
dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
 from {username='slave'} at 127.0.0.1:39726
I20260709 12:59:08.731269  2333 raft_consensus.cc:493] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:59:08.731676  2333 raft_consensus.cc:3060] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:08.737767  1745 raft_consensus.cc:3060] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:08.739630  2333 raft_consensus.cc:515] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:08.743392  2299 leader_election.cc:304] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 12:59:08.745308  2781 raft_consensus.cc:2804] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:59:08.745748  2781 raft_consensus.cc:697] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 2 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 12:59:08.747972  1752 raft_consensus.cc:1275] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:59:08.749188  1751 raft_consensus.cc:1275] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:59:08.749023  2781 consensus_queue.cc:237] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:08.749625  2793 consensus_queue.cc:1048] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:59:08.757589  1602 raft_consensus.cc:1275] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:59:08.758857  1745 raft_consensus.cc:2468] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 2.
I20260709 12:59:08.759125  1598 raft_consensus.cc:1275] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260709 12:59:08.767877  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 56606.1us
I20260709 12:59:08.771143  2334 raft_consensus.cc:2955] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:08.799896  2333 leader_election.cc:290] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 2 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:08.801775  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:08.816835  1677 consensus_queue.cc:237] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 12:59:08.820863  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 55843.6us
I20260709 12:59:08.865914  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:59:08.866667  1674 raft_consensus.cc:3055] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:59:08.837415  2820 consensus_queue.cc:1048] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.019s
I20260709 12:59:08.877250   431 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:59:08.846735  2775 raft_consensus.cc:1064] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d: attempting to promote NON_VOTER 3b879799899a499c8d409ee9f1277cd1 to VOTER
I20260709 12:59:08.850739  1502 catalog_manager.cc:5697] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 reported cstate change: term changed from 1 to 2, leader changed from af520b67945d443b901bbeca37cc5701 (127.0.107.195) to 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196). New cstate: current_term: 2 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:59:08.867010  1674 raft_consensus.cc:740] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 12:59:08.900550  1674 consensus_queue.cc:260] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:08.862932  1751 raft_consensus.cc:3060] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:08.867822  2670 consensus_queue.cc:1048] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:59:08.946345  1674 raft_consensus.cc:3060] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:59:08.949460  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 52533.5us
W20260709 12:59:08.950481  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 61529.3us
W20260709 12:59:08.960747  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 63013.4us
W20260709 12:59:09.008910  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RunLeaderElection -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 48469.9us
I20260709 12:59:09.022174  2333 raft_consensus.cc:1275] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:59:09.061156  1751 raft_consensus.cc:2468] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 2.
I20260709 12:59:09.054452  2814 consensus_queue.cc:1048] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.067s
W20260709 12:59:09.067731  2818 log.cc:927] Time spent T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.178s	user 0.000s	sys 0.002s
I20260709 12:59:09.072211  1674 raft_consensus.cc:2468] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 2.
I20260709 12:59:09.073240  2807 raft_consensus.cc:993] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d: : Instructing follower 3b879799899a499c8d409ee9f1277cd1 to start an election
I20260709 12:59:09.107455  2301 leader_election.cc:304] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:59:09.108493  2781 raft_consensus.cc:2804] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:59:09.110637  2807 raft_consensus.cc:1081] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 1 LEADER]: Signalling peer 3b879799899a499c8d409ee9f1277cd1 to start an election
I20260709 12:59:09.119063  2337 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "a40e423ed583480280352b261913276c"
dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
 from {username='slave'} at 127.0.0.1:39726
I20260709 12:59:09.119903  2337 raft_consensus.cc:493] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:59:09.120362  2337 raft_consensus.cc:3060] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:09.121500  1674 raft_consensus.cc:1275] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:59:09.119083  2808 consensus_queue.cc:1048] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.023s
I20260709 12:59:09.128038  2796 consensus_queue.cc:1048] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.002s
I20260709 12:59:09.134706  2337 raft_consensus.cc:515] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:09.137513  2337 leader_election.cc:290] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 2 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:09.139266  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:09.140793  1751 raft_consensus.cc:3060] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:09.150606  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:59:09.151249  1678 raft_consensus.cc:3055] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:59:09.151592  1678 raft_consensus.cc:740] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 12:59:09.153132  1678 consensus_queue.cc:260] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:09.155601  1678 raft_consensus.cc:3060] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:09.086163  2775 consensus_queue.cc:237] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:09.169771  2337 raft_consensus.cc:2955] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER af520b67945d443b901bbeca37cc5701 (127.0.107.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:09.165637  2805 raft_consensus.cc:2955] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER af520b67945d443b901bbeca37cc5701 (127.0.107.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:09.181481  1598 raft_consensus.cc:1275] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:59:09.182659  1751 raft_consensus.cc:2468] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 2.
I20260709 12:59:09.184475  2301 leader_election.cc:304] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:59:09.194316  1678 raft_consensus.cc:2468] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 2.
I20260709 12:59:09.207947  2796 raft_consensus.cc:2804] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:59:09.209437  1745 raft_consensus.cc:1275] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:59:09.336428  1488 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2a521fb6c3e7489d9694fd81cadeb33f with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20260709 12:59:09.335925  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 337096us
I20260709 12:59:09.342306  2781 raft_consensus.cc:697] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 2 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 12:59:09.342856  2835 consensus_queue.cc:1048] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:59:09.343532  2781 consensus_queue.cc:237] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:09.345085  2835 consensus_queue.cc:1048] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20260709 12:59:09.366497  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 368680us
W20260709 12:59:09.367739  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 404487us
W20260709 12:59:09.374001  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 139500us
W20260709 12:59:09.472851  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 53574.2us
W20260709 12:59:09.471403  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 90444.1us
I20260709 12:59:09.487663  1602 raft_consensus.cc:1275] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:59:09.491094  1674 raft_consensus.cc:1275] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:59:09.463045  2796 raft_consensus.cc:697] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 2 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 12:59:09.519937  2337 raft_consensus.cc:2955] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
W20260709 12:59:09.561122  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 49402.3us
W20260709 12:59:09.564733  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 54066.6us
I20260709 12:59:09.566143  2334 raft_consensus.cc:1275] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 1 LEARNER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:59:09.465631  2764 raft_consensus.cc:2955] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
W20260709 12:59:09.676570  2833 log.cc:927] Time spent T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.128s	user 0.000s	sys 0.000s
I20260709 12:59:09.665939  2794 consensus_queue.cc:1048] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.052s
I20260709 12:59:09.653154  2839 consensus_queue.cc:1048] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.072s
I20260709 12:59:09.642331  2796 consensus_queue.cc:237] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:09.653384  2819 raft_consensus.cc:2955] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:09.783171  1502 catalog_manager.cc:5697] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 reported cstate change: term changed from 1 to 2, leader changed from 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194) to 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196). New cstate: current_term: 2 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:59:09.800189   431 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
W20260709 12:59:09.810961  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 211143us
I20260709 12:59:09.901013  1751 raft_consensus.cc:1275] T 2ebc67196b4f4119981cdb499e3350cc P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:59:09.944355  2796 raft_consensus.cc:493] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 12:59:09.945634  2796 raft_consensus.cc:515] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:09.951166  2796 leader_election.cc:290] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:09.953238  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:09.953291  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:09.959489  2299 leader_election.cc:304] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:09.964016  2796 raft_consensus.cc:2749] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 12:59:09.972206  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 380314us
I20260709 12:59:09.987414  2794 consensus_queue.cc:1048] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20260709 12:59:09.990538  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 239610us
W20260709 12:59:09.991690  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 50241.9us
I20260709 12:59:09.995090  2721 raft_consensus.cc:493] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 12:59:09.996052  2721 raft_consensus.cc:515] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:10.002075  2772 raft_consensus.cc:493] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 12:59:10.003060  2772 raft_consensus.cc:515] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:10.004858  2846 raft_consensus.cc:493] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 12:59:10.005897  2772 leader_election.cc:290] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:10.005630  2846 raft_consensus.cc:515] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:10.008193  2846 leader_election.cc:290] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:10.033146  1673 raft_consensus.cc:2955] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:10.036528  1678 raft_consensus.cc:1275] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:59:10.042068  2721 leader_election.cc:290] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:10.071895  1745 raft_consensus.cc:1275] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:59:10.074903  2841 consensus_queue.cc:1048] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.028s
W20260709 12:59:10.091780  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 49085.1us
I20260709 12:59:10.113235  2837 consensus_queue.cc:1048] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.009s
I20260709 12:59:10.120390  2829 raft_consensus.cc:493] T 03daa4b346f54f1da820d1ff6cd1ff86 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 12:59:10.121308  2829 raft_consensus.cc:515] T 03daa4b346f54f1da820d1ff6cd1ff86 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:10.124234  2829 leader_election.cc:290] T 03daa4b346f54f1da820d1ff6cd1ff86 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 12:59:10.135102  2858 log.cc:927] Time spent T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.054s	user 0.000s	sys 0.000s
I20260709 12:59:10.136332  2846 raft_consensus.cc:493] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 12:59:10.137074  2846 raft_consensus.cc:515] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:10.150056  2798 raft_consensus.cc:493] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 12:59:10.150746  2798 raft_consensus.cc:515] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:10.130983  2840 raft_consensus.cc:493] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 12:59:10.149717  1676 raft_consensus.cc:2955] T 1d9ac76dc2c445edb5a25cf9087998bb P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:10.161844  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:10.171213  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:10.171525  2852 raft_consensus.cc:493] T 86abe5695d5643f888190301b88dacef P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 12:59:10.172480  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:10.173604  2720 raft_consensus.cc:493] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 12:59:10.177488  2821 raft_consensus.cc:493] T 2a521fb6c3e7489d9694fd81cadeb33f P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 12:59:10.181376  2821 raft_consensus.cc:515] T 2a521fb6c3e7489d9694fd81cadeb33f P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:10.176035  2720 raft_consensus.cc:515] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:10.152789  2840 raft_consensus.cc:515] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:10.214740  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:10.172514  2852 raft_consensus.cc:515] T 86abe5695d5643f888190301b88dacef P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:10.212381  2798 leader_election.cc:290] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:10.245126  2846 leader_election.cc:290] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:10.262691  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:10.272800  2720 leader_election.cc:290] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:10.205401  2840 leader_election.cc:290] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:10.285027  2721 raft_consensus.cc:493] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 12:59:10.298965  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:10.310376  2721 raft_consensus.cc:515] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 12:59:10.319789  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 202777us
I20260709 12:59:10.294050  2821 leader_election.cc:290] T 2a521fb6c3e7489d9694fd81cadeb33f P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:10.357223  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:10.360553  2301 leader_election.cc:304] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 12:59:10.361649  2840 raft_consensus.cc:2749] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:59:10.369645  2865 raft_consensus.cc:493] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 12:59:10.370502  2865 raft_consensus.cc:515] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 12:59:10.371337  2819 consensus_meta.cc:287] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.648s	user 0.010s	sys 0.014s
I20260709 12:59:10.375272  2720 raft_consensus.cc:493] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 12:59:10.376097  2720 raft_consensus.cc:515] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:10.339241  2852 leader_election.cc:290] T 86abe5695d5643f888190301b88dacef P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:10.380442  2721 leader_election.cc:290] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:10.384362  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:10.391243  1674 raft_consensus.cc:1275] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:59:10.402541  1747 raft_consensus.cc:1275] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:59:10.348142  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:10.408272  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:10.415846  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:10.416345  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:10.416839  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:10.419457  1749 raft_consensus.cc:2955] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:10.416872  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:10.402477  2840 consensus_queue.cc:1048] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.012s
I20260709 12:59:10.411185  2852 raft_consensus.cc:493] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 12:59:10.430591  2852 raft_consensus.cc:515] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:10.414894  2720 leader_election.cc:290] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:10.404850  2772 raft_consensus.cc:493] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 12:59:10.439476  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:10.415035  2846 raft_consensus.cc:493] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 12:59:10.411827  2794 consensus_queue.cc:1048] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.034s
I20260709 12:59:10.416373  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:10.431406  2332 raft_consensus.cc:2955] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:10.446885  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:10.462522  2837 consensus_queue.cc:1048] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:59:10.462785  2846 raft_consensus.cc:515] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:10.465246  2846 leader_election.cc:290] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:10.465937  2846 raft_consensus.cc:493] T 78cabb03415041ca9c1a5a400e506131 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 12:59:10.466614  2846 raft_consensus.cc:515] T 78cabb03415041ca9c1a5a400e506131 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:10.468768  2852 leader_election.cc:290] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:10.484754  2865 leader_election.cc:290] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:10.488036  2334 raft_consensus.cc:2955] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:10.513821  1605 raft_consensus.cc:1275] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:59:10.514262  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:10.563045  2720 raft_consensus.cc:493] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 12:59:10.540831  2831 raft_consensus.cc:493] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 12:59:10.508106  2852 raft_consensus.cc:493] T 79ed08f2339a4b1ead1e70d78eaec2ec P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
W20260709 12:59:10.562763  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 52248us
I20260709 12:59:10.566218  2831 raft_consensus.cc:515] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } }
I20260709 12:59:10.577258  2720 raft_consensus.cc:515] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:10.452232  2814 raft_consensus.cc:2955] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:10.500659  2772 raft_consensus.cc:515] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:10.555308  2846 leader_election.cc:290] T 78cabb03415041ca9c1a5a400e506131 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:10.610343  2720 leader_election.cc:290] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:10.589176  2852 raft_consensus.cc:515] T 79ed08f2339a4b1ead1e70d78eaec2ec P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:10.579523  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 12:59:10.667798  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 64950.2us
I20260709 12:59:10.647126  1597 raft_consensus.cc:2468] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 1.
W20260709 12:59:10.670105  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 87187.2us
W20260709 12:59:10.669057  2869 log.cc:927] Time spent T 1d9ac76dc2c445edb5a25cf9087998bb P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.239s	user 0.001s	sys 0.000s
I20260709 12:59:10.683616  2852 leader_election.cc:290] T 79ed08f2339a4b1ead1e70d78eaec2ec P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 12:59:10.693212  2871 log.cc:927] Time spent T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.088s	user 0.000s	sys 0.001s
I20260709 12:59:10.693393  2772 leader_election.cc:290] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:10.712925  1718 leader_election.cc:304] T 2a521fb6c3e7489d9694fd81cadeb33f P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 12:59:10.727252  2831 leader_election.cc:290] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:10.728055  2831 raft_consensus.cc:2749] T 2a521fb6c3e7489d9694fd81cadeb33f P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:59:10.730635  2808 consensus_queue.cc:1048] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.013s
I20260709 12:59:10.739030  1571 leader_election.cc:304] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
W20260709 12:59:10.749047  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 62582.6us
I20260709 12:59:10.751235  1717 leader_election.cc:304] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:10.754531  2798 raft_consensus.cc:2749] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:59:10.755811  2772 raft_consensus.cc:2749] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:59:10.756248  1571 leader_election.cc:304] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 12:59:10.758293  2819 raft_consensus.cc:2749] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:59:10.758404  1571 leader_election.cc:304] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 12:59:10.759814  2865 raft_consensus.cc:2749] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:59:10.780617  1502 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:43476 (request call id 111) took 1012 ms. Trace:
I20260709 12:59:10.786886  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:10.790700  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:10.799401  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:10.801368  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:10.803846  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:10.815523  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 12:59:10.817207  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 64720.5us
W20260709 12:59:10.841177  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 122882us
I20260709 12:59:10.846062  1501 catalog_manager.cc:5697] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 reported cstate change: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "af520b67945d443b901bbeca37cc5701" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:59:10.847947  1569 leader_election.cc:304] T 86abe5695d5643f888190301b88dacef P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 12:59:10.849167  2852 raft_consensus.cc:2749] T 86abe5695d5643f888190301b88dacef P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:59:10.860936  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:10.864967  1571 leader_election.cc:304] T 03daa4b346f54f1da820d1ff6cd1ff86 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 12:59:10.877581  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:10.881989  2846 raft_consensus.cc:2749] T 03daa4b346f54f1da820d1ff6cd1ff86 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:59:10.783826  1502 rpcz_store.cc:276] 0709 12:59:09.765269 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:09.766011 (+   742us) service_pool.cc:224] Handling call
0709 12:59:09.776022 (+ 10011us) spinlock_profiling.cc:243] Waited 3.61 ms on lock 0x7b5400002490. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f426303db71 00007f426303e0b2 00007f426bfb7a89 00007f426bf4fda6 00007f426bf21088 00007f426c01f37e 00007f426a4e02a4 00007f426a4e0232 00007f426a4e0161 00007f426a4e00dc
0709 12:59:10.777680 (+1001658us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:59:10.079843 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:59:10.080865 (+  1022us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:59:10.080936 (+    71us) write_op.cc:435] Acquired schema lock
0709 12:59:10.080991 (+    55us) tablet.cc:662] Decoding operations
0709 12:59:10.081493 (+   502us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:59:10.081594 (+   101us) write_op.cc:641] Partition lock acquired for write op
0709 12:59:10.081629 (+    35us) tablet.cc:685] Acquiring locks for 1 operations
0709 12:59:10.082037 (+   408us) tablet.cc:701] Row locks acquired
0709 12:59:10.082090 (+    53us) write_op.cc:260] PREPARE: finished
0709 12:59:10.082273 (+   183us) write_op.cc:270] Start()
0709 12:59:10.082475 (+   202us) write_op.cc:276] Timestamp: P: 1783601950082214 usec, L: 0
0709 12:59:10.082519 (+    44us) op_driver.cc:348] REPLICATION: starting
0709 12:59:10.083409 (+   890us) log.cc:844] Serialized 535 byte log entry
0709 12:59:10.321431 (+238022us) op_driver.cc:464] REPLICATION: finished
0709 12:59:10.450781 (+129350us) write_op.cc:301] APPLY: starting
0709 12:59:10.450907 (+   126us) tablet.cc:1367] starting BulkCheckPresence
0709 12:59:10.453057 (+  2150us) tablet.cc:1370] finished BulkCheckPresence
0709 12:59:10.453200 (+   143us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:59:10.566603 (+113403us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:59:10.566948 (+   345us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0709 12:59:10.585597 (+ 18649us) write_op.cc:312] APPLY: finished
0709 12:59:10.616090 (+ 30493us) log.cc:844] Serialized 22 byte log entry
0709 12:59:10.622162 (+  6072us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:59:10.622301 (+   139us) write_op.cc:454] Released schema lock
0709 12:59:10.622372 (+    71us) write_op.cc:341] FINISH: Updating metrics
0709 12:59:10.625674 (+  3302us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:59:10.625723 (+    49us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":1,"spinlock_wait_cycles":8382464,"child_traces":[["op",{"apply.queue_time_us":121173,"apply.run_cpu_time_us":3533,"apply.run_wall_time_us":174801,"num_ops":1,"prepare.queue_time_us":173437,"prepare.run_cpu_time_us":4143,"prepare.run_wall_time_us":54658,"replication_time_us":221059,"thread_start_us":70952,"threads_started":2}]]}
I20260709 12:59:10.929858  1596 raft_consensus.cc:2955] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:10.956204  2875 raft_consensus.cc:493] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 12:59:10.964365  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:10.971249  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 12:59:10.975162  2879 log.cc:927] Time spent T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.129s	user 0.001s	sys 0.000s
I20260709 12:59:10.980719  2876 raft_consensus.cc:493] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 12:59:10.922871  1599 raft_consensus.cc:2955] T 86abe5695d5643f888190301b88dacef P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:10.981673  2876 raft_consensus.cc:515] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:10.957165  2875 raft_consensus.cc:515] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:10.999228  2801 raft_consensus.cc:493] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 12:59:11.006898  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:11.007167  2854 raft_consensus.cc:493] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 12:59:11.008358  2852 raft_consensus.cc:493] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 12:59:11.014173  2865 raft_consensus.cc:493] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 12:59:11.016955  2865 raft_consensus.cc:515] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:11.015280  2852 raft_consensus.cc:515] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:11.020731  2852 leader_election.cc:290] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:11.001181  2878 raft_consensus.cc:493] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 12:59:11.023825  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:11.024947  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:11.000120  2801 raft_consensus.cc:515] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:11.032481  2865 leader_election.cc:290] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:11.008003  2854 raft_consensus.cc:515] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 12:59:11.087533  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 210324us
I20260709 12:59:11.091763  2875 leader_election.cc:290] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:11.102626  2820 raft_consensus.cc:493] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 12:59:11.106654  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:11.103408  2820 raft_consensus.cc:515] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:10.909457  1597 raft_consensus.cc:2955] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER af520b67945d443b901bbeca37cc5701 (127.0.107.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:11.088858  2878 raft_consensus.cc:515] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:11.111765  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 12:59:11.118669  2887 log.cc:927] Time spent T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.153s	user 0.002s	sys 0.000s
I20260709 12:59:11.130841  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:11.128218  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:11.155751  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:11.161831  2798 raft_consensus.cc:493] T 84e78b66ceec4f759ae9e101a5469e4c P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 12:59:11.164551  2893 raft_consensus.cc:493] T 83226095f79a46dab445849b1febd425 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 12:59:11.179371  2876 leader_election.cc:290] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:11.187139  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:11.178369  2854 leader_election.cc:290] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:11.188555  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:11.189383  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:11.194131  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:11.208405  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:11.214829  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:11.212487  1717 leader_election.cc:304] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:11.217698  2820 leader_election.cc:290] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:11.195195  2893 raft_consensus.cc:515] T 83226095f79a46dab445849b1febd425 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:11.192659  2798 raft_consensus.cc:515] T 84e78b66ceec4f759ae9e101a5469e4c P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:11.222363  2801 leader_election.cc:290] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:11.222992  2798 leader_election.cc:290] T 84e78b66ceec4f759ae9e101a5469e4c P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:11.226159  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:11.226982  1745 raft_consensus.cc:2468] T 2a521fb6c3e7489d9694fd81cadeb33f P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 1.
I20260709 12:59:11.228790  2829 raft_consensus.cc:493] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 12:59:11.229449  2829 raft_consensus.cc:515] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:11.229465  2854 raft_consensus.cc:493] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 12:59:11.231491  2854 raft_consensus.cc:515] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:11.235126  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:11.235402  2851 raft_consensus.cc:2749] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:59:11.239169  2878 leader_election.cc:290] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:11.242188  2893 leader_election.cc:290] T 83226095f79a46dab445849b1febd425 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:11.242933  2829 leader_election.cc:290] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:11.247256  2854 leader_election.cc:290] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:11.249976  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:11.250720  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:11.251344  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:11.252027  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:11.255738  1569 leader_election.cc:304] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
W20260709 12:59:11.264729  2892 log.cc:927] Time spent T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.053s	user 0.000s	sys 0.000s
I20260709 12:59:11.267275  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:11.271826  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:11.273334  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:11.274324  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 12:59:11.275492  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 63118.6us
I20260709 12:59:11.290548  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:11.290591  1569 leader_election.cc:304] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 12:59:11.292790  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:11.294111  2846 raft_consensus.cc:2749] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:59:11.298475  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:11.301118  2854 raft_consensus.cc:2749] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:59:11.304596  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:11.311120  1502 catalog_manager.cc:5697] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 reported cstate change: config changed from index 3 to 4, VOTER af520b67945d443b901bbeca37cc5701 (127.0.107.195) evicted. New cstate: current_term: 1 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:11.311525  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:11.313292  1502 catalog_manager.cc:5697] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 reported cstate change: term changed from 1 to 2, leader changed from 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194) to 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196). New cstate: current_term: 2 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:59:11.314922  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:11.317857  1502 catalog_manager.cc:5697] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 reported cstate change: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:11.344606  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:11.349695  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:11.358306  1500 catalog_manager.cc:5697] T 1d9ac76dc2c445edb5a25cf9087998bb P 816fadcea8fc4893882f5f9b9f22a03d reported cstate change: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:11.359258  1752 consensus_queue.cc:237] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:11.359678  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:11.361428  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:11.362074  1747 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d"
dest_uuid: "af520b67945d443b901bbeca37cc5701"
mode: GRACEFUL
new_leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
 from {username='slave'} at 127.0.0.1:51270
I20260709 12:59:11.362996  1747 raft_consensus.cc:606] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 2 LEADER]: Received request to transfer leadership to TS 816fadcea8fc4893882f5f9b9f22a03d
I20260709 12:59:11.373196  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:11.375562  1673 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80"
dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
mode: GRACEFUL
new_leader_uuid: "af520b67945d443b901bbeca37cc5701"
 from {username='slave'} at 127.0.0.1:43266
I20260709 12:59:11.391064  1673 raft_consensus.cc:606] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 LEADER]: Received request to transfer leadership to TS af520b67945d443b901bbeca37cc5701
I20260709 12:59:11.392752  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:11.395725  1752 raft_consensus.cc:2955] T 2ebc67196b4f4119981cdb499e3350cc P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:11.410902  1740 tablet_service.cc:1558] Processing DeleteTablet for tablet 2a521fb6c3e7489d9694fd81cadeb33f with delete_type TABLET_DATA_TOMBSTONED (TS af520b67945d443b901bbeca37cc5701 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:55838
I20260709 12:59:11.412317  2332 raft_consensus.cc:1275] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:59:11.415877  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:11.418156  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:11.419227  1599 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c"
dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
mode: GRACEFUL
new_leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
 from {username='slave'} at 127.0.0.1:47752
I20260709 12:59:11.419821  1599 raft_consensus.cc:606] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 LEADER]: Received request to transfer leadership to TS 816fadcea8fc4893882f5f9b9f22a03d
I20260709 12:59:11.427448  1673 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "87b86d46388a41cd90374c203cb27914"
dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
mode: GRACEFUL
new_leader_uuid: "3b879799899a499c8d409ee9f1277cd1"
 from {username='slave'} at 127.0.0.1:43266
I20260709 12:59:11.429564  1673 raft_consensus.cc:606] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Received request to transfer leadership to TS 3b879799899a499c8d409ee9f1277cd1
I20260709 12:59:11.430025  1673 raft_consensus.cc:614] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:59:11.433470  1680 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "a40e423ed583480280352b261913276c"
dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
mode: GRACEFUL
new_leader_uuid: "3b879799899a499c8d409ee9f1277cd1"
 from {username='slave'} at 127.0.0.1:43266
I20260709 12:59:11.434016  1680 raft_consensus.cc:606] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Received request to transfer leadership to TS 3b879799899a499c8d409ee9f1277cd1
I20260709 12:59:11.434326  1680 raft_consensus.cc:614] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:59:11.441625  1571 leader_election.cc:304] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 12:59:11.443481  1599 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "ef5897d71124435db68c88060b095421"
dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
mode: GRACEFUL
new_leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
 from {username='slave'} at 127.0.0.1:47752
I20260709 12:59:11.444562  1599 raft_consensus.cc:606] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 LEADER]: Received request to transfer leadership to TS 816fadcea8fc4893882f5f9b9f22a03d
I20260709 12:59:11.447710  2897 consensus_queue.cc:1048] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:59:11.448588  1646 leader_election.cc:304] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 12:59:11.450780   431 auto_leader_rebalancer.cc:452] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260709 12:59:11.452387  2775 raft_consensus.cc:2749] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:59:11.436769  1717 leader_election.cc:304] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:11.453053  2814 raft_consensus.cc:993] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d: : Instructing follower af520b67945d443b901bbeca37cc5701 to start an election
I20260709 12:59:11.454252  1680 raft_consensus.cc:1275] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:59:11.455197  1717 leader_election.cc:304] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:11.454015  2893 raft_consensus.cc:2749] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:59:11.453994  2814 raft_consensus.cc:1081] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 LEADER]: Signalling peer af520b67945d443b901bbeca37cc5701 to start an election
I20260709 12:59:11.461059  1646 leader_election.cc:304] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 12:59:11.462380  2807 raft_consensus.cc:2749] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:59:11.464080  1599 consensus_queue.cc:237] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:11.453111   431 auto_leader_rebalancer.cc:527] All tables' leader rebalancing finished this round
I20260709 12:59:11.469344  1717 leader_election.cc:304] T 84e78b66ceec4f759ae9e101a5469e4c P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:11.472249  1717 leader_election.cc:304] T 83226095f79a46dab445849b1febd425 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:11.472837  2900 tablet_replica.cc:333] T 2a521fb6c3e7489d9694fd81cadeb33f P af520b67945d443b901bbeca37cc5701: stopping tablet replica
I20260709 12:59:11.475117  2889 raft_consensus.cc:2749] T 84e78b66ceec4f759ae9e101a5469e4c P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:59:11.468317  2893 raft_consensus.cc:2749] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:59:11.468755  1751 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80"
dest_uuid: "af520b67945d443b901bbeca37cc5701"
 from {username='slave'} at 127.0.0.1:55848
I20260709 12:59:11.476174  1751 raft_consensus.cc:493] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:59:11.476522  1751 raft_consensus.cc:3060] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:11.471163  2846 raft_consensus.cc:2749] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:59:11.471782  2878 consensus_queue.cc:1048] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.008s
I20260709 12:59:11.480104  2335 raft_consensus.cc:1275] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:59:11.482914  2900 raft_consensus.cc:2243] T 2a521fb6c3e7489d9694fd81cadeb33f P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:59:11.488531  2900 raft_consensus.cc:2272] T 2a521fb6c3e7489d9694fd81cadeb33f P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:59:11.493666  2878 raft_consensus.cc:2749] T 83226095f79a46dab445849b1febd425 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:59:11.508824  2772 consensus_queue.cc:1048] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
W20260709 12:59:11.517629  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 46975.3us
I20260709 12:59:11.523092  1571 leader_election.cc:304] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 12:59:11.525121  1571 leader_election.cc:304] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 12:59:11.517086  1751 raft_consensus.cc:515] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:11.529654  1571 leader_election.cc:304] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 12:59:11.532089  1571 leader_election.cc:304] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 12:59:11.533576  1751 leader_election.cc:290] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 2 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:11.526871  2820 raft_consensus.cc:2749] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:59:11.528205  2865 raft_consensus.cc:2749] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:59:11.533895  2877 raft_consensus.cc:2749] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:59:11.537317  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:59:11.542234  1673 raft_consensus.cc:3055] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:59:11.542600  1673 raft_consensus.cc:740] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 12:59:11.543514  1673 consensus_queue.cc:260] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:11.544983  1673 raft_consensus.cc:3060] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:11.541178  2846 raft_consensus.cc:2749] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:59:11.551784  2877 raft_consensus.cc:2955] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:11.556387  2337 raft_consensus.cc:2955] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:11.563519  2897 raft_consensus.cc:2955] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:11.568328  1680 raft_consensus.cc:2955] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:11.571254  2332 raft_consensus.cc:2955] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:11.568351  2900 ts_tablet_manager.cc:1916] T 2a521fb6c3e7489d9694fd81cadeb33f P af520b67945d443b901bbeca37cc5701: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:59:11.598881  1673 raft_consensus.cc:2468] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 2.
I20260709 12:59:11.601234  1718 leader_election.cc:304] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:59:11.603205  1487 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 1d9ac76dc2c445edb5a25cf9087998bb with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:59:11.606801  2878 raft_consensus.cc:2804] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:59:11.607652  2878 raft_consensus.cc:697] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 2 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 12:59:11.608989  2878 consensus_queue.cc:237] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 12:59:11.631049  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 99012.7us
I20260709 12:59:11.626456  1501 catalog_manager.cc:5697] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 reported cstate change: config changed from index 3 to 4, VOTER 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194) evicted. New cstate: current_term: 1 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:11.632030  1571 leader_election.cc:304] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 12:59:11.632400  1501 catalog_manager.cc:5697] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 reported cstate change: config changed from index 3 to 4, VOTER 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) evicted. New cstate: current_term: 1 leader_uuid: "af520b67945d443b901bbeca37cc5701" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:11.633982  1490 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 86abe5695d5643f888190301b88dacef with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:59:11.634680  2877 raft_consensus.cc:2749] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:59:11.639279  1571 leader_election.cc:304] T 78cabb03415041ca9c1a5a400e506131 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 12:59:11.640103  2865 raft_consensus.cc:2749] T 78cabb03415041ca9c1a5a400e506131 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:59:11.643759  1571 leader_election.cc:304] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 12:59:11.645202  1571 leader_election.cc:304] T 79ed08f2339a4b1ead1e70d78eaec2ec P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 12:59:11.646842  1571 leader_election.cc:304] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 12:59:11.650658  2846 raft_consensus.cc:2749] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:59:11.648411  2865 raft_consensus.cc:2749] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:59:11.648978  2877 raft_consensus.cc:2749] T 79ed08f2339a4b1ead1e70d78eaec2ec P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:59:11.668424  1571 leader_election.cc:304] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 12:59:11.669332  2877 raft_consensus.cc:2749] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:59:11.670322  1571 leader_election.cc:304] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 12:59:11.682711  2877 raft_consensus.cc:2749] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:59:11.682670  1503 catalog_manager.cc:5697] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 reported cstate change: term changed from 1 to 2, leader changed from 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194) to af520b67945d443b901bbeca37cc5701 (127.0.107.195). New cstate: current_term: 2 leader_uuid: "af520b67945d443b901bbeca37cc5701" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: HEALTHY } } }
I20260709 12:59:11.687918  1571 leader_election.cc:304] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 12:59:11.688783  2877 raft_consensus.cc:2749] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:59:11.696110  1571 leader_election.cc:304] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 12:59:11.698168  2891 raft_consensus.cc:993] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701: : Instructing follower 816fadcea8fc4893882f5f9b9f22a03d to start an election
I20260709 12:59:11.698413  2900 ts_tablet_manager.cc:1929] T 2a521fb6c3e7489d9694fd81cadeb33f P af520b67945d443b901bbeca37cc5701: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:59:11.698813  2891 raft_consensus.cc:1081] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 2 LEADER]: Signalling peer 816fadcea8fc4893882f5f9b9f22a03d to start an election
I20260709 12:59:11.699273  2846 raft_consensus.cc:2749] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:59:11.701658  1680 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d"
dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
 from {username='slave'} at 127.0.0.1:52778
I20260709 12:59:11.702378  1592 tablet_service.cc:1558] Processing DeleteTablet for tablet 86abe5695d5643f888190301b88dacef with delete_type TABLET_DATA_TOMBSTONED (TS 83595ad0fd5940c0a2bfc7c61acd969a not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:40138
I20260709 12:59:11.702389  1680 raft_consensus.cc:493] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:59:11.703667  1680 raft_consensus.cc:3060] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:59:11.703987  1665 tablet_service.cc:1558] Processing DeleteTablet for tablet 1d9ac76dc2c445edb5a25cf9087998bb with delete_type TABLET_DATA_TOMBSTONED (TS 816fadcea8fc4893882f5f9b9f22a03d not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:52760
I20260709 12:59:11.711169  1571 leader_election.cc:304] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 12:59:11.714185  2877 raft_consensus.cc:2749] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:59:11.716881  1751 raft_consensus.cc:1275] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:59:11.738695  2900 log.cc:1199] T 2a521fb6c3e7489d9694fd81cadeb33f P af520b67945d443b901bbeca37cc5701: Deleting WAL directory at /tmp/dist-test-taskakPCWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783601878574137-431-0/minicluster-data/ts-2-root/wals/2a521fb6c3e7489d9694fd81cadeb33f
I20260709 12:59:11.746982  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:59:11.747845  1597 raft_consensus.cc:3060] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:11.752213  1490 catalog_manager.cc:5028] TS af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): tablet 2a521fb6c3e7489d9694fd81cadeb33f (table test-workload [id=75d77b57b3f34786b9c215c4f11bd727]) successfully deleted
I20260709 12:59:11.753021  2877 raft_consensus.cc:993] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a: : Instructing follower 816fadcea8fc4893882f5f9b9f22a03d to start an election
I20260709 12:59:11.753553  2865 raft_consensus.cc:1081] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 LEADER]: Signalling peer 816fadcea8fc4893882f5f9b9f22a03d to start an election
I20260709 12:59:11.762780  1673 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "ef5897d71124435db68c88060b095421"
dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
 from {username='slave'} at 127.0.0.1:52766
I20260709 12:59:11.763473  1673 raft_consensus.cc:493] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:59:11.763859  1673 raft_consensus.cc:3060] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:11.771288  2907 tablet_replica.cc:333] T 86abe5695d5643f888190301b88dacef P 83595ad0fd5940c0a2bfc7c61acd969a: stopping tablet replica
I20260709 12:59:11.771906  2908 tablet_replica.cc:333] T 1d9ac76dc2c445edb5a25cf9087998bb P 816fadcea8fc4893882f5f9b9f22a03d: stopping tablet replica
I20260709 12:59:11.771574  1671 consensus_queue.cc:237] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:11.784039  2908 raft_consensus.cc:2243] T 1d9ac76dc2c445edb5a25cf9087998bb P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:59:11.785144  2908 raft_consensus.cc:2272] T 1d9ac76dc2c445edb5a25cf9087998bb P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:59:11.788836  2908 ts_tablet_manager.cc:1916] T 1d9ac76dc2c445edb5a25cf9087998bb P 816fadcea8fc4893882f5f9b9f22a03d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:59:11.792719  2332 raft_consensus.cc:1275] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:59:11.794677  2807 consensus_queue.cc:1048] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:59:11.780807  1680 raft_consensus.cc:515] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:11.803800  2807 raft_consensus.cc:2955] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER af520b67945d443b901bbeca37cc5701 (127.0.107.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:11.809126  2337 raft_consensus.cc:2955] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER af520b67945d443b901bbeca37cc5701 (127.0.107.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:11.810527  1680 leader_election.cc:290] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 3 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:11.818765  2907 raft_consensus.cc:2243] T 86abe5695d5643f888190301b88dacef P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:59:11.819674  2907 raft_consensus.cc:2272] T 86abe5695d5643f888190301b88dacef P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:59:11.820180  1673 raft_consensus.cc:515] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:11.828745  2907 ts_tablet_manager.cc:1916] T 86abe5695d5643f888190301b88dacef P 83595ad0fd5940c0a2bfc7c61acd969a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:59:11.829753  1673 leader_election.cc:290] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 2 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:11.835054  1597 raft_consensus.cc:2468] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 2.
I20260709 12:59:11.851127  2877 consensus_queue.cc:1048] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:59:11.861231  2908 ts_tablet_manager.cc:1929] T 1d9ac76dc2c445edb5a25cf9087998bb P 816fadcea8fc4893882f5f9b9f22a03d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:59:11.861930  2908 log.cc:1199] T 1d9ac76dc2c445edb5a25cf9087998bb P 816fadcea8fc4893882f5f9b9f22a03d: Deleting WAL directory at /tmp/dist-test-taskakPCWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783601878574137-431-0/minicluster-data/ts-1-root/wals/1d9ac76dc2c445edb5a25cf9087998bb
I20260709 12:59:11.863656  1597 raft_consensus.cc:1275] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:59:11.864876  1488 catalog_manager.cc:5028] TS 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): tablet 1d9ac76dc2c445edb5a25cf9087998bb (table test-workload [id=75d77b57b3f34786b9c215c4f11bd727]) successfully deleted
I20260709 12:59:11.868217  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: true dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:59:11.869297  1597 raft_consensus.cc:3060] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:59:11.875252  1488 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2ebc67196b4f4119981cdb499e3350cc with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:59:11.876830  1501 catalog_manager.cc:5697] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 reported cstate change: config changed from index 3 to 4, VOTER af520b67945d443b901bbeca37cc5701 (127.0.107.195) evicted. New cstate: current_term: 1 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:11.882704  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:59:11.883379  1604 raft_consensus.cc:3055] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:59:11.883682  1604 raft_consensus.cc:740] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 12:59:11.890486  1604 consensus_queue.cc:260] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:11.891906  1604 raft_consensus.cc:3060] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:11.895359  2876 consensus_queue.cc:1048] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:59:11.896766  2907 ts_tablet_manager.cc:1929] T 86abe5695d5643f888190301b88dacef P 83595ad0fd5940c0a2bfc7c61acd969a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:59:11.902242  2907 log.cc:1199] T 86abe5695d5643f888190301b88dacef P 83595ad0fd5940c0a2bfc7c61acd969a: Deleting WAL directory at /tmp/dist-test-taskakPCWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783601878574137-431-0/minicluster-data/ts-0-root/wals/86abe5695d5643f888190301b88dacef
I20260709 12:59:11.904824  1487 catalog_manager.cc:5028] TS 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): tablet 86abe5695d5643f888190301b88dacef (table test-workload [id=75d77b57b3f34786b9c215c4f11bd727]) successfully deleted
I20260709 12:59:11.905386  1740 tablet_service.cc:1558] Processing DeleteTablet for tablet 2ebc67196b4f4119981cdb499e3350cc with delete_type TABLET_DATA_TOMBSTONED (TS af520b67945d443b901bbeca37cc5701 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:55838
I20260709 12:59:11.905812  1597 raft_consensus.cc:2468] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 3.
I20260709 12:59:11.906536  2900 tablet_replica.cc:333] T 2ebc67196b4f4119981cdb499e3350cc P af520b67945d443b901bbeca37cc5701: stopping tablet replica
I20260709 12:59:11.907722  2900 raft_consensus.cc:2243] T 2ebc67196b4f4119981cdb499e3350cc P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:59:11.908573  2900 raft_consensus.cc:2272] T 2ebc67196b4f4119981cdb499e3350cc P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:59:11.912441  2900 ts_tablet_manager.cc:1916] T 2ebc67196b4f4119981cdb499e3350cc P af520b67945d443b901bbeca37cc5701: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:59:11.901486  1604 raft_consensus.cc:2468] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 2.
I20260709 12:59:11.930253  1599 raft_consensus.cc:2955] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER af520b67945d443b901bbeca37cc5701 (127.0.107.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:11.936358  2846 raft_consensus.cc:993] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a: : Instructing follower 816fadcea8fc4893882f5f9b9f22a03d to start an election
I20260709 12:59:11.937052  2846 raft_consensus.cc:1081] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 LEADER]: Signalling peer 816fadcea8fc4893882f5f9b9f22a03d to start an election
I20260709 12:59:11.938727  1671 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c"
dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
 from {username='slave'} at 127.0.0.1:52766
I20260709 12:59:11.939301  1671 raft_consensus.cc:493] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:59:11.939659  1671 raft_consensus.cc:3060] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:11.943056  1643 leader_election.cc:304] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 12:59:11.945840  1643 leader_election.cc:304] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 12:59:11.947253  2794 raft_consensus.cc:2804] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:59:11.947774  2794 raft_consensus.cc:697] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 3 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 12:59:11.948221  2881 raft_consensus.cc:2804] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:59:11.948925  2881 raft_consensus.cc:697] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 12:59:11.948690  2794 consensus_queue.cc:237] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:11.949991  2881 consensus_queue.cc:237] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:11.970934  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: true dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:11.971673  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:11.972441  1751 raft_consensus.cc:3060] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:11.971669  1752 raft_consensus.cc:3055] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:59:11.974895  1752 raft_consensus.cc:740] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 12:59:11.970326  1671 raft_consensus.cc:515] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:11.990327  1752 consensus_queue.cc:260] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:11.992100  1751 raft_consensus.cc:2468] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 2.
I20260709 12:59:11.992347  1752 raft_consensus.cc:3060] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:59:11.999938  1671 leader_election.cc:290] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 2 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:12.006979  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:12.007694  1754 raft_consensus.cc:3060] T 84e78b66ceec4f759ae9e101a5469e4c P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:12.018281  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:59:12.018666  1752 raft_consensus.cc:2468] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 3.
I20260709 12:59:12.019116  1605 raft_consensus.cc:3055] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:59:12.019471  1605 raft_consensus.cc:740] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 12:59:12.020358  1605 consensus_queue.cc:260] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:12.021611  1605 raft_consensus.cc:3060] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:12.051750  1754 raft_consensus.cc:2468] T 84e78b66ceec4f759ae9e101a5469e4c P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 2.
I20260709 12:59:12.059947  1605 raft_consensus.cc:2468] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 2.
I20260709 12:59:12.064198  1750 raft_consensus.cc:2955] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:12.061877  1646 leader_election.cc:304] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:59:12.067449  1502 catalog_manager.cc:5697] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d reported cstate change: term changed from 2 to 3, leader changed from af520b67945d443b901bbeca37cc5701 (127.0.107.195) to 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194). New cstate: current_term: 3 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } }
I20260709 12:59:12.070863  2794 raft_consensus.cc:2804] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:59:12.071527  2794 raft_consensus.cc:697] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 2 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 12:59:12.070818  1502 catalog_manager.cc:5697] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d reported cstate change: term changed from 1 to 2, leader changed from 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) to 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194). New cstate: current_term: 2 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } }
I20260709 12:59:12.072583  2794 consensus_queue.cc:237] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:12.078464  2900 ts_tablet_manager.cc:1929] T 2ebc67196b4f4119981cdb499e3350cc P af520b67945d443b901bbeca37cc5701: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:59:12.082713  2900 log.cc:1199] T 2ebc67196b4f4119981cdb499e3350cc P af520b67945d443b901bbeca37cc5701: Deleting WAL directory at /tmp/dist-test-taskakPCWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783601878574137-431-0/minicluster-data/ts-2-root/wals/2ebc67196b4f4119981cdb499e3350cc
I20260709 12:59:12.087234  1605 raft_consensus.cc:1275] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:59:12.087949  1490 catalog_manager.cc:5028] TS af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): tablet 2ebc67196b4f4119981cdb499e3350cc (table test-workload [id=75d77b57b3f34786b9c215c4f11bd727]) successfully deleted
I20260709 12:59:12.090094  2897 consensus_queue.cc:1048] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:59:12.109246  1502 catalog_manager.cc:5697] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d reported cstate change: term changed from 1 to 2, leader changed from 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) to 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194). New cstate: current_term: 2 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } }
I20260709 12:59:12.157117  1680 raft_consensus.cc:1275] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:59:12.159941  2909 consensus_queue.cc:1048] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:59:12.470331   431 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:59:12.484088  1751 raft_consensus.cc:1275] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 12:59:12.486079  2794 consensus_queue.cc:1048] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:59:12.543545   431 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:59:12.556463  1605 raft_consensus.cc:1275] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 12:59:12.563227  2881 consensus_queue.cc:1048] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:59:12.565769  1750 raft_consensus.cc:1275] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:59:12.572131  2881 consensus_queue.cc:1048] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:59:12.582098  1596 raft_consensus.cc:1275] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:59:12.582525  1599 raft_consensus.cc:1275] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:59:12.584647  2794 consensus_queue.cc:1048] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:59:12.586233  2794 consensus_queue.cc:1048] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:59:12.643505  1605 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31"
dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
mode: GRACEFUL
new_leader_uuid: "3b879799899a499c8d409ee9f1277cd1"
 from {username='slave'} at 127.0.0.1:47752
I20260709 12:59:12.644199  1605 raft_consensus.cc:606] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 LEADER]: Received request to transfer leadership to TS 3b879799899a499c8d409ee9f1277cd1
I20260709 12:59:12.648193  1671 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f"
dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
mode: GRACEFUL
new_leader_uuid: "3b879799899a499c8d409ee9f1277cd1"
 from {username='slave'} at 127.0.0.1:43266
I20260709 12:59:12.648825  1671 raft_consensus.cc:606] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 1 LEADER]: Received request to transfer leadership to TS 3b879799899a499c8d409ee9f1277cd1
I20260709 12:59:12.653564  1605 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "83226095f79a46dab445849b1febd425"
dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
mode: GRACEFUL
new_leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
 from {username='slave'} at 127.0.0.1:47752
I20260709 12:59:12.654124  1605 raft_consensus.cc:606] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 LEADER]: Received request to transfer leadership to TS 816fadcea8fc4893882f5f9b9f22a03d
I20260709 12:59:12.658337  1671 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3"
dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
mode: GRACEFUL
new_leader_uuid: "af520b67945d443b901bbeca37cc5701"
 from {username='slave'} at 127.0.0.1:43266
I20260709 12:59:12.659121  1671 raft_consensus.cc:606] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 LEADER]: Received request to transfer leadership to TS af520b67945d443b901bbeca37cc5701
I20260709 12:59:12.664716  1680 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "dd3f391a609743de886c525de483fc29"
dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
mode: GRACEFUL
new_leader_uuid: "3b879799899a499c8d409ee9f1277cd1"
 from {username='slave'} at 127.0.0.1:43266
I20260709 12:59:12.665354  1680 raft_consensus.cc:606] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 LEADER]: Received request to transfer leadership to TS 3b879799899a499c8d409ee9f1277cd1
I20260709 12:59:12.669174  1754 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91"
dest_uuid: "af520b67945d443b901bbeca37cc5701"
mode: GRACEFUL
new_leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
 from {username='slave'} at 127.0.0.1:51270
I20260709 12:59:12.669760  1754 raft_consensus.cc:606] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 1 LEADER]: Received request to transfer leadership to TS 816fadcea8fc4893882f5f9b9f22a03d
I20260709 12:59:12.675099   431 auto_leader_rebalancer.cc:452] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260709 12:59:12.697774   431 auto_leader_rebalancer.cc:527] All tables' leader rebalancing finished this round
I20260709 12:59:12.698321  2807 raft_consensus.cc:993] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d: : Instructing follower af520b67945d443b901bbeca37cc5701 to start an election
I20260709 12:59:12.713792  2807 raft_consensus.cc:1081] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 LEADER]: Signalling peer af520b67945d443b901bbeca37cc5701 to start an election
I20260709 12:59:12.712252  2877 raft_consensus.cc:993] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a: : Instructing follower 3b879799899a499c8d409ee9f1277cd1 to start an election
I20260709 12:59:12.718299  2877 raft_consensus.cc:1081] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 LEADER]: Signalling peer 3b879799899a499c8d409ee9f1277cd1 to start an election
I20260709 12:59:12.720546  2332 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31"
dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
 from {username='slave'} at 127.0.0.1:49082
I20260709 12:59:12.722540  2332 raft_consensus.cc:493] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:59:12.723248  2332 raft_consensus.cc:3060] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:12.709324  2925 raft_consensus.cc:993] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d: : Instructing follower 3b879799899a499c8d409ee9f1277cd1 to start an election
I20260709 12:59:12.723886  2925 raft_consensus.cc:1081] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 LEADER]: Signalling peer 3b879799899a499c8d409ee9f1277cd1 to start an election
I20260709 12:59:12.725764  2335 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "dd3f391a609743de886c525de483fc29"
dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
 from {username='slave'} at 127.0.0.1:39726
I20260709 12:59:12.726313  2335 raft_consensus.cc:493] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:59:12.726682  2335 raft_consensus.cc:3060] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:12.740224  2335 raft_consensus.cc:515] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:12.741822  1752 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3"
dest_uuid: "af520b67945d443b901bbeca37cc5701"
 from {username='slave'} at 127.0.0.1:55848
I20260709 12:59:12.748657  1750 raft_consensus.cc:1275] T 84e78b66ceec4f759ae9e101a5469e4c P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:59:12.751356  1752 raft_consensus.cc:493] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:59:12.751842  1752 raft_consensus.cc:3060] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:59:12.754753  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 58938.3us
I20260709 12:59:12.785794  2332 raft_consensus.cc:515] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:12.790165  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:12.791159  1751 raft_consensus.cc:3060] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:12.794390  2925 consensus_queue.cc:1048] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:59:12.794571  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:59:12.796015  1596 raft_consensus.cc:3055] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:59:12.796402  1596 raft_consensus.cc:740] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 12:59:12.797739  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:59:12.798378  1603 raft_consensus.cc:3060] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:12.798799  1596 consensus_queue.cc:260] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:12.800073  1596 raft_consensus.cc:3060] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:12.808080  2335 leader_election.cc:290] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 2 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:12.809157  1752 raft_consensus.cc:515] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:12.817183  1603 raft_consensus.cc:2468] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 2.
I20260709 12:59:12.819283  1752 leader_election.cc:290] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 2 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:12.821449  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:59:12.822002  1603 raft_consensus.cc:3060] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:12.790992  2332 leader_election.cc:290] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 2 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:12.818619  2298 leader_election.cc:304] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 12:59:12.824934  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:59:12.825546  1680 raft_consensus.cc:3055] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:59:12.825891  1680 raft_consensus.cc:740] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 12:59:12.826704  2913 raft_consensus.cc:2804] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:59:12.830456  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:59:12.831594  1671 raft_consensus.cc:3055] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:59:12.832818  1671 raft_consensus.cc:740] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 12:59:12.835858  1671 consensus_queue.cc:260] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:12.837065  1603 raft_consensus.cc:2468] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 2.
I20260709 12:59:12.828541  1680 consensus_queue.cc:260] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:12.841993  1717 leader_election.cc:304] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:59:12.842800  1751 raft_consensus.cc:2468] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 2.
I20260709 12:59:12.844105  2301 leader_election.cc:304] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:59:12.845979  1680 raft_consensus.cc:3060] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:12.847736  2889 raft_consensus.cc:2804] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:59:12.860031  1680 raft_consensus.cc:2468] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 2.
I20260709 12:59:12.866586  2889 raft_consensus.cc:697] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 2 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 12:59:12.868098  2889 consensus_queue.cc:237] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:12.863268  1671 raft_consensus.cc:3060] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:12.884068  2913 raft_consensus.cc:697] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 2 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 12:59:12.885272  2913 consensus_queue.cc:237] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:12.869494  2933 raft_consensus.cc:2804] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Leader election won for term 2
W20260709 12:59:12.913645  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 69607.8us
I20260709 12:59:12.908442  2933 raft_consensus.cc:697] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 2 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 12:59:12.928545  2933 consensus_queue.cc:237] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 12:59:12.943989  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 84010.4us
I20260709 12:59:13.003929  1596 raft_consensus.cc:2468] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 2.
I20260709 12:59:13.004565  2807 raft_consensus.cc:993] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d: : Instructing follower 3b879799899a499c8d409ee9f1277cd1 to start an election
I20260709 12:59:13.005167  2930 raft_consensus.cc:1081] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 1 LEADER]: Signalling peer 3b879799899a499c8d409ee9f1277cd1 to start an election
I20260709 12:59:13.007036  2335 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f"
dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
 from {username='slave'} at 127.0.0.1:39726
I20260709 12:59:13.007618  2335 raft_consensus.cc:493] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:59:13.007957  2335 raft_consensus.cc:3060] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:13.026996  2877 raft_consensus.cc:993] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a: : Instructing follower 816fadcea8fc4893882f5f9b9f22a03d to start an election
I20260709 12:59:13.027012  1671 raft_consensus.cc:2468] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 2.
I20260709 12:59:13.057555  2335 raft_consensus.cc:515] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:13.073873  2939 raft_consensus.cc:1081] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 LEADER]: Signalling peer 816fadcea8fc4893882f5f9b9f22a03d to start an election
I20260709 12:59:13.080905  1672 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "83226095f79a46dab445849b1febd425"
dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
 from {username='slave'} at 127.0.0.1:52766
I20260709 12:59:13.081571  1672 raft_consensus.cc:493] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:59:13.081959  1672 raft_consensus.cc:3060] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:13.086033  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:59:13.086746  1596 raft_consensus.cc:3060] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:13.088925  2335 leader_election.cc:290] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 2 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:13.090193  1672 raft_consensus.cc:515] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:13.096323  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:59:13.098546  1671 raft_consensus.cc:3055] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:59:13.098903  1671 raft_consensus.cc:740] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 12:59:13.099809  1671 consensus_queue.cc:260] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 12:59:13.101053  1646 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:59:13.101157  1671 raft_consensus.cc:3060] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:13.107996  1672 leader_election.cc:290] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 2 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:13.108887  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:13.109534  1750 raft_consensus.cc:3060] T 83226095f79a46dab445849b1febd425 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:13.111606  1502 catalog_manager.cc:5697] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 reported cstate change: term changed from 1 to 2, leader changed from 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194) to 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196). New cstate: current_term: 2 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:59:13.113850  1671 raft_consensus.cc:2468] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 2.
I20260709 12:59:13.115199  2299 leader_election.cc:304] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 12:59:13.116070  2913 raft_consensus.cc:2804] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:59:13.115834  1501 catalog_manager.cc:5697] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 reported cstate change: term changed from 1 to 2, leader changed from 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194) to af520b67945d443b901bbeca37cc5701 (127.0.107.195). New cstate: current_term: 2 leader_uuid: "af520b67945d443b901bbeca37cc5701" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: HEALTHY } } }
I20260709 12:59:13.116636  2913 raft_consensus.cc:697] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 2 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 12:59:13.117671  2913 consensus_queue.cc:237] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:13.119004  1750 raft_consensus.cc:2468] T 83226095f79a46dab445849b1febd425 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 2.
I20260709 12:59:13.120185  1646 leader_election.cc:304] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:59:13.122581  2925 raft_consensus.cc:2804] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:59:13.123059  2925 raft_consensus.cc:697] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 12:59:13.123965  2925 consensus_queue.cc:237] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:13.130465  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:59:13.131042  1599 raft_consensus.cc:3055] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:59:13.131351  1599 raft_consensus.cc:740] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 12:59:13.132194  1599 consensus_queue.cc:260] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:13.135362  1599 raft_consensus.cc:3060] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:13.136070  1596 raft_consensus.cc:2468] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 2.
I20260709 12:59:13.156039  1599 raft_consensus.cc:2468] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 2.
I20260709 12:59:13.174243  1672 consensus_queue.cc:237] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } }
W20260709 12:59:13.187665  1644 consensus_peers.cc:597] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d -> Peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625): Couldn't send request to peer 3b879799899a499c8d409ee9f1277cd1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 84e78b66ceec4f759ae9e101a5469e4c. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:59:13.189944  1750 raft_consensus.cc:1275] T 84e78b66ceec4f759ae9e101a5469e4c P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260709 12:59:13.191584  2925 consensus_queue.cc:1048] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:59:13.192020  1599 raft_consensus.cc:1275] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260709 12:59:13.193750  2925 consensus_queue.cc:1048] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:59:13.202621  2807 raft_consensus.cc:2955] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } } }
I20260709 12:59:13.205957  1753 raft_consensus.cc:2955] T 84e78b66ceec4f759ae9e101a5469e4c P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } } }
I20260709 12:59:13.215014  1503 catalog_manager.cc:5697] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d reported cstate change: term changed from 1 to 2, leader changed from 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) to 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194). New cstate: current_term: 2 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } }
I20260709 12:59:13.219156  2942 raft_consensus.cc:993] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701: : Instructing follower 816fadcea8fc4893882f5f9b9f22a03d to start an election
I20260709 12:59:13.219735  2942 raft_consensus.cc:1081] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 1 LEADER]: Signalling peer 816fadcea8fc4893882f5f9b9f22a03d to start an election
I20260709 12:59:13.221745  1672 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91"
dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
 from {username='slave'} at 127.0.0.1:52778
I20260709 12:59:13.222569  1672 raft_consensus.cc:493] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:59:13.222963  1672 raft_consensus.cc:3060] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:13.233131  1599 raft_consensus.cc:2955] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } } }
I20260709 12:59:13.243278  1672 raft_consensus.cc:515] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:13.246423  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:59:13.247206  1596 raft_consensus.cc:3060] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:13.247484  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:13.248132  1750 raft_consensus.cc:3055] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:59:13.248430  1750 raft_consensus.cc:740] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 12:59:13.249290  1750 consensus_queue.cc:260] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:13.255870  1750 raft_consensus.cc:3060] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:13.260098  1672 leader_election.cc:290] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 2 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:13.297578  1596 raft_consensus.cc:2468] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 2.
I20260709 12:59:13.309535  1643 leader_election.cc:304] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 12:59:13.310478  2927 raft_consensus.cc:2804] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:59:13.311179  2927 raft_consensus.cc:697] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 12:59:13.312237  2927 consensus_queue.cc:237] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:13.313175  1750 raft_consensus.cc:2468] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 2.
W20260709 12:59:13.317920  1551 auto_rebalancer.cc:629] Failed to schedule move for tablet e947690dcb3849cb9f9731123d363b91: Illegal state: Replica af520b67945d443b901bbeca37cc5701 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } }
I20260709 12:59:13.324314  1671 raft_consensus.cc:1275] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:59:13.325340  1596 raft_consensus.cc:1275] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:59:13.327205  2942 consensus_queue.cc:1048] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:59:13.328154  1750 consensus_queue.cc:237] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } }
I20260709 12:59:13.330610  2913 consensus_queue.cc:1048] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20260709 12:59:13.346953  1718 consensus_peers.cc:597] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 -> Peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625): Couldn't send request to peer 3b879799899a499c8d409ee9f1277cd1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 03daa4b346f54f1da820d1ff6cd1ff86. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:59:13.349617  1672 raft_consensus.cc:1275] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:59:13.351145  1603 raft_consensus.cc:1275] T 03daa4b346f54f1da820d1ff6cd1ff86 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:59:13.351572  2942 consensus_queue.cc:1048] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:59:13.353680  2942 consensus_queue.cc:1048] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:59:13.351145  1596 raft_consensus.cc:1275] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:59:13.358796  2893 consensus_queue.cc:1048] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.013s
I20260709 12:59:13.394640  1754 raft_consensus.cc:1275] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:59:13.396798  2933 consensus_queue.cc:1048] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.002s
I20260709 12:59:13.412693  1500 catalog_manager.cc:5697] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: config changed from index -1 to 3, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New cstate: current_term: 2 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } } }
I20260709 12:59:13.452189  1605 raft_consensus.cc:1275] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:59:13.451848  2893 raft_consensus.cc:2955] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } } }
I20260709 12:59:13.453754  2933 consensus_queue.cc:1048] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:59:13.455703  1680 raft_consensus.cc:2955] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } } }
I20260709 12:59:13.469748  1596 raft_consensus.cc:2955] T 03daa4b346f54f1da820d1ff6cd1ff86 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } } }
I20260709 12:59:13.473047  1502 catalog_manager.cc:5697] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 reported cstate change: term changed from 1 to 2, leader changed from 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) to 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196). New cstate: current_term: 2 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:59:13.474984  1502 catalog_manager.cc:5697] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 reported cstate change: term changed from 1 to 2, leader changed from 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194) to 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196). New cstate: current_term: 2 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:59:13.496269  1680 consensus_queue.cc:237] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } }
I20260709 12:59:13.499580  1500 catalog_manager.cc:5697] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d reported cstate change: term changed from 1 to 2, leader changed from af520b67945d443b901bbeca37cc5701 (127.0.107.195) to 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194). New cstate: current_term: 2 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } }
I20260709 12:59:13.508526  1672 raft_consensus.cc:1275] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:59:13.520731  2913 consensus_queue.cc:1048] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:59:13.505821  1500 catalog_manager.cc:5697] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d reported cstate change: config changed from index -1 to 2, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New cstate: current_term: 1 leader_uuid: "af520b67945d443b901bbeca37cc5701" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } } }
I20260709 12:59:13.547861  1605 raft_consensus.cc:1275] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:59:13.550583  2913 consensus_queue.cc:1048] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20260709 12:59:13.563573  1644 consensus_peers.cc:597] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d -> Peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625): Couldn't send request to peer 3b879799899a499c8d409ee9f1277cd1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 79ed08f2339a4b1ead1e70d78eaec2ec. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:59:13.568404  1605 raft_consensus.cc:1275] T 79ed08f2339a4b1ead1e70d78eaec2ec P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:59:13.590512  1754 raft_consensus.cc:1275] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:59:13.603231  1671 raft_consensus.cc:1275] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:59:13.605960  2913 consensus_queue.cc:1048] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.004s
I20260709 12:59:13.582752  2927 consensus_queue.cc:1048] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.037s
I20260709 12:59:13.625945  1605 raft_consensus.cc:1275] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:59:13.628301  2928 consensus_queue.cc:1048] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:59:13.629623  1754 raft_consensus.cc:1275] T 83226095f79a46dab445849b1febd425 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:59:13.625979  2807 consensus_queue.cc:1048] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.007s
I20260709 12:59:13.632788  2929 consensus_queue.cc:1048] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:59:13.699186   431 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:59:13.702057  2928 raft_consensus.cc:2955] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } } }
I20260709 12:59:13.711503  1604 raft_consensus.cc:2955] T 79ed08f2339a4b1ead1e70d78eaec2ec P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } } }
I20260709 12:59:13.749332  1754 raft_consensus.cc:1275] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:59:13.751637  2928 consensus_queue.cc:1048] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:59:13.807467  2980 ts_tablet_manager.cc:933] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1: Initiating tablet copy from peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:13.828158  1753 raft_consensus.cc:2955] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } } }
I20260709 12:59:13.837306  1501 catalog_manager.cc:5697] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d reported cstate change: config changed from index -1 to 2, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New cstate: current_term: 1 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260709 12:59:13.882747  2980 tablet_copy_client.cc:323] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Beginning tablet copy session from remote peer at address 127.0.107.194:46261
I20260709 12:59:13.887212  1690 tablet_copy_service.cc:140] P 816fadcea8fc4893882f5f9b9f22a03d: Received BeginTabletCopySession request for tablet 84e78b66ceec4f759ae9e101a5469e4c from peer 3b879799899a499c8d409ee9f1277cd1 ({username='slave'} at 127.0.0.1:57092)
I20260709 12:59:13.887812  1690 tablet_copy_service.cc:161] P 816fadcea8fc4893882f5f9b9f22a03d: Beginning new tablet copy session on tablet 84e78b66ceec4f759ae9e101a5469e4c from peer 3b879799899a499c8d409ee9f1277cd1 at {username='slave'} at 127.0.0.1:57092: session id = 3b879799899a499c8d409ee9f1277cd1-84e78b66ceec4f759ae9e101a5469e4c
W20260709 12:59:13.926483  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 81850.3us
I20260709 12:59:13.928515  1596 raft_consensus.cc:1275] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:59:13.933616  1690 tablet_copy_source_session.cc:215] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:59:13.934634  2928 consensus_queue.cc:1048] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:59:13.966285  2980 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 84e78b66ceec4f759ae9e101a5469e4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:59:13.988190  2983 ts_tablet_manager.cc:933] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1: Initiating tablet copy from peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:14.005991  2983 tablet_copy_client.cc:323] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Beginning tablet copy session from remote peer at address 127.0.107.195:40693
I20260709 12:59:14.008476  1764 tablet_copy_service.cc:140] P af520b67945d443b901bbeca37cc5701: Received BeginTabletCopySession request for tablet 03daa4b346f54f1da820d1ff6cd1ff86 from peer 3b879799899a499c8d409ee9f1277cd1 ({username='slave'} at 127.0.0.1:43156)
I20260709 12:59:14.009145  1764 tablet_copy_service.cc:161] P af520b67945d443b901bbeca37cc5701: Beginning new tablet copy session on tablet 03daa4b346f54f1da820d1ff6cd1ff86 from peer 3b879799899a499c8d409ee9f1277cd1 at {username='slave'} at 127.0.0.1:43156: session id = 3b879799899a499c8d409ee9f1277cd1-03daa4b346f54f1da820d1ff6cd1ff86
I20260709 12:59:14.009789  2980 tablet_copy_client.cc:806] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Starting download of 0 data blocks...
I20260709 12:59:14.011340  2980 tablet_copy_client.cc:670] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Starting download of 1 WAL segments...
I20260709 12:59:14.015519  2980 tablet_copy_client.cc:538] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:59:14.020377  1764 tablet_copy_source_session.cc:215] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:59:14.024513  2983 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03daa4b346f54f1da820d1ff6cd1ff86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:59:14.038884  2980 tablet_bootstrap.cc:492] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1: Bootstrap starting.
I20260709 12:59:14.045073  2983 tablet_copy_client.cc:806] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Starting download of 0 data blocks...
I20260709 12:59:14.045789  2983 tablet_copy_client.cc:670] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Starting download of 1 WAL segments...
I20260709 12:59:14.051594  2983 tablet_copy_client.cc:538] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:59:14.059878  2985 ts_tablet_manager.cc:933] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1: Initiating tablet copy from peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:14.063462  2983 tablet_bootstrap.cc:492] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1: Bootstrap starting.
I20260709 12:59:14.075408  2985 tablet_copy_client.cc:323] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Beginning tablet copy session from remote peer at address 127.0.107.194:46261
I20260709 12:59:14.077802  1690 tablet_copy_service.cc:140] P 816fadcea8fc4893882f5f9b9f22a03d: Received BeginTabletCopySession request for tablet 79ed08f2339a4b1ead1e70d78eaec2ec from peer 3b879799899a499c8d409ee9f1277cd1 ({username='slave'} at 127.0.0.1:57092)
I20260709 12:59:14.078295  1690 tablet_copy_service.cc:161] P 816fadcea8fc4893882f5f9b9f22a03d: Beginning new tablet copy session on tablet 79ed08f2339a4b1ead1e70d78eaec2ec from peer 3b879799899a499c8d409ee9f1277cd1 at {username='slave'} at 127.0.0.1:57092: session id = 3b879799899a499c8d409ee9f1277cd1-79ed08f2339a4b1ead1e70d78eaec2ec
I20260709 12:59:14.085537   431 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:59:14.087878  1690 tablet_copy_source_session.cc:215] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:59:14.091781  2985 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 79ed08f2339a4b1ead1e70d78eaec2ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:59:14.093475  2980 tablet_bootstrap.cc:492] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:59:14.099674  2980 tablet_bootstrap.cc:492] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1: Bootstrap complete.
I20260709 12:59:14.100464  2980 ts_tablet_manager.cc:1403] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1: Time spent bootstrapping tablet: real 0.062s	user 0.018s	sys 0.015s
I20260709 12:59:14.104106  2980 raft_consensus.cc:359] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } }
I20260709 12:59:14.105005  2980 raft_consensus.cc:740] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Initialized, Role: LEARNER
I20260709 12:59:14.105715  2980 consensus_queue.cc:260] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } }
I20260709 12:59:14.112776  2980 ts_tablet_manager.cc:1434] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1: Time spent starting tablet: real 0.012s	user 0.008s	sys 0.000s
I20260709 12:59:14.116426  1689 tablet_copy_service.cc:342] P 816fadcea8fc4893882f5f9b9f22a03d: Request end of tablet copy session 3b879799899a499c8d409ee9f1277cd1-84e78b66ceec4f759ae9e101a5469e4c received from {username='slave'} at 127.0.0.1:57092
I20260709 12:59:14.116925  1689 tablet_copy_service.cc:434] P 816fadcea8fc4893882f5f9b9f22a03d: ending tablet copy session 3b879799899a499c8d409ee9f1277cd1-84e78b66ceec4f759ae9e101a5469e4c on tablet 84e78b66ceec4f759ae9e101a5469e4c with peer 3b879799899a499c8d409ee9f1277cd1
I20260709 12:59:14.122522  2983 tablet_bootstrap.cc:492] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:59:14.127381  2983 tablet_bootstrap.cc:492] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1: Bootstrap complete.
I20260709 12:59:14.128192  2983 ts_tablet_manager.cc:1403] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1: Time spent bootstrapping tablet: real 0.065s	user 0.023s	sys 0.015s
I20260709 12:59:14.128381  2985 tablet_copy_client.cc:806] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Starting download of 0 data blocks...
I20260709 12:59:14.129009  2985 tablet_copy_client.cc:670] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Starting download of 1 WAL segments...
I20260709 12:59:14.137523  2985 tablet_copy_client.cc:538] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:59:14.142073  2983 raft_consensus.cc:359] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } }
I20260709 12:59:14.143065  2983 raft_consensus.cc:740] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Initialized, Role: LEARNER
I20260709 12:59:14.143798  2983 consensus_queue.cc:260] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } }
I20260709 12:59:14.146854  2983 ts_tablet_manager.cc:1434] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1: Time spent starting tablet: real 0.018s	user 0.007s	sys 0.000s
I20260709 12:59:14.151516  2985 tablet_bootstrap.cc:492] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1: Bootstrap starting.
I20260709 12:59:14.153752  1764 tablet_copy_service.cc:342] P af520b67945d443b901bbeca37cc5701: Request end of tablet copy session 3b879799899a499c8d409ee9f1277cd1-03daa4b346f54f1da820d1ff6cd1ff86 received from {username='slave'} at 127.0.0.1:43156
I20260709 12:59:14.154286  1764 tablet_copy_service.cc:434] P af520b67945d443b901bbeca37cc5701: ending tablet copy session 3b879799899a499c8d409ee9f1277cd1-03daa4b346f54f1da820d1ff6cd1ff86 on tablet 03daa4b346f54f1da820d1ff6cd1ff86 with peer 3b879799899a499c8d409ee9f1277cd1
I20260709 12:59:14.187283  2985 tablet_bootstrap.cc:492] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:59:14.188303  2985 tablet_bootstrap.cc:492] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1: Bootstrap complete.
I20260709 12:59:14.189074  2985 ts_tablet_manager.cc:1403] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1: Time spent bootstrapping tablet: real 0.038s	user 0.025s	sys 0.000s
I20260709 12:59:14.194674  2985 raft_consensus.cc:359] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } }
I20260709 12:59:14.199427  2985 raft_consensus.cc:740] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Initialized, Role: LEARNER
I20260709 12:59:14.204173  2337 raft_consensus.cc:1217] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.2->[2.3-2.3]   Dedup: 2.3->[]
I20260709 12:59:14.200133  2985 consensus_queue.cc:260] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } }
I20260709 12:59:14.213691  2985 ts_tablet_manager.cc:1434] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1: Time spent starting tablet: real 0.024s	user 0.009s	sys 0.000s
I20260709 12:59:14.219269  1690 tablet_copy_service.cc:342] P 816fadcea8fc4893882f5f9b9f22a03d: Request end of tablet copy session 3b879799899a499c8d409ee9f1277cd1-79ed08f2339a4b1ead1e70d78eaec2ec received from {username='slave'} at 127.0.0.1:57092
I20260709 12:59:14.219725  1690 tablet_copy_service.cc:434] P 816fadcea8fc4893882f5f9b9f22a03d: ending tablet copy session 3b879799899a499c8d409ee9f1277cd1-79ed08f2339a4b1ead1e70d78eaec2ec on tablet 79ed08f2339a4b1ead1e70d78eaec2ec with peer 3b879799899a499c8d409ee9f1277cd1
I20260709 12:59:14.224027  1676 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "344b4152823f4d618345b14ffd579400"
dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
mode: GRACEFUL
new_leader_uuid: "3b879799899a499c8d409ee9f1277cd1"
 from {username='slave'} at 127.0.0.1:43266
I20260709 12:59:14.224701  1676 raft_consensus.cc:606] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 LEADER]: Received request to transfer leadership to TS 3b879799899a499c8d409ee9f1277cd1
I20260709 12:59:14.231083  1676 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81"
dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
mode: GRACEFUL
new_leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
 from {username='slave'} at 127.0.0.1:43266
I20260709 12:59:14.231637  1676 raft_consensus.cc:606] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 LEADER]: Received request to transfer leadership to TS 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:14.234423  1603 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "d21485c094584689917b2a97b2d3c624"
dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
mode: GRACEFUL
new_leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
 from {username='slave'} at 127.0.0.1:47752
I20260709 12:59:14.235011  1603 raft_consensus.cc:606] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 LEADER]: Received request to transfer leadership to TS 816fadcea8fc4893882f5f9b9f22a03d
I20260709 12:59:14.238379  1754 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23"
dest_uuid: "af520b67945d443b901bbeca37cc5701"
mode: GRACEFUL
new_leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
 from {username='slave'} at 127.0.0.1:51270
I20260709 12:59:14.238973  1754 raft_consensus.cc:606] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 1 LEADER]: Received request to transfer leadership to TS 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:14.242383  1596 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7"
dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
mode: GRACEFUL
new_leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
 from {username='slave'} at 127.0.0.1:47752
I20260709 12:59:14.242959  1596 raft_consensus.cc:606] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 LEADER]: Received request to transfer leadership to TS 816fadcea8fc4893882f5f9b9f22a03d
I20260709 12:59:14.244045   431 auto_leader_rebalancer.cc:452] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 12:59:14.245380   431 auto_leader_rebalancer.cc:527] All tables' leader rebalancing finished this round
I20260709 12:59:14.348591  2337 raft_consensus.cc:1217] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:59:14.362802  2928 raft_consensus.cc:993] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d: : Instructing follower 83595ad0fd5940c0a2bfc7c61acd969a to start an election
I20260709 12:59:14.363324  2928 raft_consensus.cc:1081] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 LEADER]: Signalling peer 83595ad0fd5940c0a2bfc7c61acd969a to start an election
I20260709 12:59:14.365024  1596 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81"
dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
 from {username='slave'} at 127.0.0.1:40166
I20260709 12:59:14.365720  1596 raft_consensus.cc:493] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:59:14.366168  1596 raft_consensus.cc:3060] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:59:14.375569  1596 raft_consensus.cc:515] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:14.377595  1596 leader_election.cc:290] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 3 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:14.378129  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:59:14.378894  1676 raft_consensus.cc:3055] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:59:14.379269  1676 raft_consensus.cc:740] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 12:59:14.379905  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:14.380057  1676 consensus_queue.cc:260] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:14.380591  1754 raft_consensus.cc:3060] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:59:14.381487  1676 raft_consensus.cc:3060] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:59:14.388646  1754 raft_consensus.cc:2468] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 3.
I20260709 12:59:14.388646  1676 raft_consensus.cc:2468] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 3.
I20260709 12:59:14.391589  1569 leader_election.cc:304] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 12:59:14.392472  2820 raft_consensus.cc:2804] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:59:14.392957  2820 raft_consensus.cc:697] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 12:59:14.393869  2820 consensus_queue.cc:237] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:14.409891  1501 catalog_manager.cc:5697] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 2 to 3, leader changed from 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194) to 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193). New cstate: current_term: 3 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } }
I20260709 12:59:14.511667  2940 raft_consensus.cc:993] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701: : Instructing follower 83595ad0fd5940c0a2bfc7c61acd969a to start an election
I20260709 12:59:14.512240  2940 raft_consensus.cc:1081] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 1 LEADER]: Signalling peer 83595ad0fd5940c0a2bfc7c61acd969a to start an election
I20260709 12:59:14.514667  1596 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23"
dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
 from {username='slave'} at 127.0.0.1:40154
I20260709 12:59:14.515276  1596 raft_consensus.cc:493] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:59:14.515681  1596 raft_consensus.cc:3060] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:14.522059  1596 raft_consensus.cc:515] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:14.523875  1596 leader_election.cc:290] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 2 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:14.524757  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:14.524998  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 12:59:14.525446  1754 raft_consensus.cc:3055] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:59:14.525619  2337 raft_consensus.cc:3060] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:14.525868  1754 raft_consensus.cc:740] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 12:59:14.526726  1754 consensus_queue.cc:260] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:14.527916  1754 raft_consensus.cc:3060] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:14.536283  2337 raft_consensus.cc:2468] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 2.
I20260709 12:59:14.537631  1569 leader_election.cc:304] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 12:59:14.538709  2820 raft_consensus.cc:2804] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:59:14.539196  2820 raft_consensus.cc:697] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 12:59:14.540141  2820 consensus_queue.cc:237] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:14.540843  1754 raft_consensus.cc:2468] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 2.
I20260709 12:59:14.554083  2930 raft_consensus.cc:993] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d: : Instructing follower 3b879799899a499c8d409ee9f1277cd1 to start an election
I20260709 12:59:14.554639  2930 raft_consensus.cc:1081] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 LEADER]: Signalling peer 3b879799899a499c8d409ee9f1277cd1 to start an election
I20260709 12:59:14.556304  1501 catalog_manager.cc:5697] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 1 to 2, leader changed from af520b67945d443b901bbeca37cc5701 (127.0.107.195) to 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193). New cstate: current_term: 2 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:59:14.563688  2332 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "344b4152823f4d618345b14ffd579400"
dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
 from {username='slave'} at 127.0.0.1:39726
I20260709 12:59:14.564368  2332 raft_consensus.cc:493] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:59:14.564774  2332 raft_consensus.cc:3060] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:14.567272  2337 raft_consensus.cc:1217] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:59:14.578408  2332 raft_consensus.cc:515] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:14.581532  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:59:14.582171  1676 raft_consensus.cc:3055] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:59:14.582481  1676 raft_consensus.cc:740] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 12:59:14.583222  2332 leader_election.cc:290] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 2 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:14.583343  1676 consensus_queue.cc:260] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:14.584028  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:14.584667  1754 raft_consensus.cc:3060] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:14.584530  1676 raft_consensus.cc:3060] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:14.594789  1676 raft_consensus.cc:2468] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 2.
I20260709 12:59:14.596004  2299 leader_election.cc:304] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 12:59:14.597142  2933 raft_consensus.cc:2804] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:59:14.597630  2933 raft_consensus.cc:697] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 2 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 12:59:14.598562  2933 consensus_queue.cc:237] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:14.600525  1754 raft_consensus.cc:2468] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 2.
I20260709 12:59:14.615842  1501 catalog_manager.cc:5697] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 reported cstate change: term changed from 1 to 2, leader changed from 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194) to 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196). New cstate: current_term: 2 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:59:14.640712  2939 raft_consensus.cc:993] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a: : Instructing follower 816fadcea8fc4893882f5f9b9f22a03d to start an election
I20260709 12:59:14.641235  2820 raft_consensus.cc:1081] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 LEADER]: Signalling peer 816fadcea8fc4893882f5f9b9f22a03d to start an election
I20260709 12:59:14.643728  1676 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "d21485c094584689917b2a97b2d3c624"
dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
 from {username='slave'} at 127.0.0.1:52766
I20260709 12:59:14.644291  1676 raft_consensus.cc:493] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:59:14.644658  1676 raft_consensus.cc:3060] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:14.654094  1676 raft_consensus.cc:515] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:14.656929  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:59:14.657549  1596 raft_consensus.cc:3055] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:59:14.657829  1596 raft_consensus.cc:740] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 12:59:14.658715  1596 consensus_queue.cc:260] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:14.659884  1596 raft_consensus.cc:3060] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:14.667111  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 12:59:14.667846  2332 raft_consensus.cc:3060] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:14.676542  2332 raft_consensus.cc:2468] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 2.
I20260709 12:59:14.677924  1644 leader_election.cc:304] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 12:59:14.678858  2807 raft_consensus.cc:2804] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:59:14.678903  1676 leader_election.cc:290] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 2 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:14.680274  2807 raft_consensus.cc:697] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 12:59:14.681417  2807 consensus_queue.cc:237] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:14.684489  1596 raft_consensus.cc:2468] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 2.
I20260709 12:59:14.709350  1501 catalog_manager.cc:5697] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d reported cstate change: term changed from 1 to 2, leader changed from 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) to 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194). New cstate: current_term: 2 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:59:14.743280  2928 raft_consensus.cc:1064] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d: attempting to promote NON_VOTER 3b879799899a499c8d409ee9f1277cd1 to VOTER
I20260709 12:59:14.746428  2928 consensus_queue.cc:237] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 1, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:14.753789  2332 raft_consensus.cc:1275] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [term 2 LEARNER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260709 12:59:14.754830  1754 raft_consensus.cc:1275] T 84e78b66ceec4f759ae9e101a5469e4c P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260709 12:59:14.755594  1596 raft_consensus.cc:1275] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260709 12:59:14.756552  2807 consensus_queue.cc:1048] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:59:14.757445  2930 consensus_queue.cc:1048] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:59:14.758473  2928 consensus_queue.cc:1048] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:59:14.772296  2928 raft_consensus.cc:2955] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:14.777431  2332 raft_consensus.cc:2955] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:14.777814  1596 raft_consensus.cc:2955] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:14.795110  2820 raft_consensus.cc:993] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a: : Instructing follower 816fadcea8fc4893882f5f9b9f22a03d to start an election
I20260709 12:59:14.796736  2939 raft_consensus.cc:1081] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 LEADER]: Signalling peer 816fadcea8fc4893882f5f9b9f22a03d to start an election
I20260709 12:59:14.797693  1754 raft_consensus.cc:2955] T 84e78b66ceec4f759ae9e101a5469e4c P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:14.808336  1671 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7"
dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
 from {username='slave'} at 127.0.0.1:52766
I20260709 12:59:14.809015  1671 raft_consensus.cc:493] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:59:14.809429  1671 raft_consensus.cc:3060] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:14.815806  1501 catalog_manager.cc:5697] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d reported cstate change: config changed from index 3 to 4, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:59:14.826326  1671 raft_consensus.cc:515] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:14.829344  1754 raft_consensus.cc:1275] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:59:14.835105  2893 raft_consensus.cc:1064] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701: attempting to promote NON_VOTER 3b879799899a499c8d409ee9f1277cd1 to VOTER
I20260709 12:59:14.837682  2893 consensus_queue.cc:237] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:14.829672  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:59:14.839143  3002 consensus_queue.cc:1048] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:59:14.839069  1596 raft_consensus.cc:3055] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:59:14.843215  1596 raft_consensus.cc:740] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 12:59:14.844103  1596 consensus_queue.cc:260] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:14.845320  1596 raft_consensus.cc:3060] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:14.858028  1596 raft_consensus.cc:2468] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 2.
I20260709 12:59:14.830811  1671 leader_election.cc:290] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 2 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:14.831445  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:14.858767  1750 raft_consensus.cc:3060] T f6028a708cb344fbb08ab8056921d9e7 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:14.862870  2337 raft_consensus.cc:1275] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 1 LEARNER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:59:14.865649  1676 raft_consensus.cc:1275] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:59:14.866406  1750 raft_consensus.cc:2468] T f6028a708cb344fbb08ab8056921d9e7 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 2.
I20260709 12:59:14.867723  1646 leader_election.cc:304] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:59:14.868599  2930 raft_consensus.cc:2804] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:59:14.869073  2930 raft_consensus.cc:697] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 12:59:14.869922  2930 consensus_queue.cc:237] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:14.874598  1596 raft_consensus.cc:1275] T 03daa4b346f54f1da820d1ff6cd1ff86 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:59:14.878925  2936 consensus_queue.cc:1048] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:59:14.849920  1672 consensus_queue.cc:237] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:14.882634  2940 consensus_queue.cc:1048] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:59:14.890041  2337 raft_consensus.cc:1275] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:59:14.893520  2807 consensus_queue.cc:1048] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:59:14.897423  1676 raft_consensus.cc:1275] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:59:14.938544  2337 raft_consensus.cc:2955] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER af520b67945d443b901bbeca37cc5701 (127.0.107.195) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:14.927671  3002 consensus_queue.cc:1048] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.017s
I20260709 12:59:14.915283  2928 raft_consensus.cc:2955] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER af520b67945d443b901bbeca37cc5701 (127.0.107.195) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:15.001246  1754 raft_consensus.cc:1275] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:59:15.005107  2932 consensus_queue.cc:1048] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:59:15.006155  1488 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 84e78b66ceec4f759ae9e101a5469e4c with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20260709 12:59:15.035403  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 53948us
I20260709 12:59:15.036108  1503 catalog_manager.cc:5697] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d reported cstate change: term changed from 1 to 2, leader changed from 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) to 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194). New cstate: current_term: 2 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } }
W20260709 12:59:15.169950  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 297127us
I20260709 12:59:15.185745  1604 raft_consensus.cc:1275] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:59:15.202365  3019 consensus_queue.cc:1048] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:59:15.225375  2332 raft_consensus.cc:1275] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:59:15.230549  2939 consensus_queue.cc:1048] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:59:15.246397   431 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:59:15.249402  1501 catalog_manager.cc:5697] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 reported cstate change: config changed from index 4 to 5, VOTER af520b67945d443b901bbeca37cc5701 (127.0.107.195) evicted. New cstate: current_term: 2 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:15.254189  3019 raft_consensus.cc:2955] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:15.278252  1740 tablet_service.cc:1558] Processing DeleteTablet for tablet 84e78b66ceec4f759ae9e101a5469e4c with delete_type TABLET_DATA_TOMBSTONED (TS af520b67945d443b901bbeca37cc5701 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:55838
I20260709 12:59:15.298357  2332 raft_consensus.cc:2955] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:15.312387  1671 raft_consensus.cc:2955] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:15.379396  1750 raft_consensus.cc:1275] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:59:15.384806  3032 tablet_replica.cc:333] T 84e78b66ceec4f759ae9e101a5469e4c P af520b67945d443b901bbeca37cc5701: stopping tablet replica
I20260709 12:59:15.409719  2930 consensus_queue.cc:1048] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.014s
W20260709 12:59:15.391791  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 48106.2us
I20260709 12:59:15.401237  1680 raft_consensus.cc:1275] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:59:15.427063  1596 raft_consensus.cc:1275] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:59:15.402526  1503 catalog_manager.cc:5697] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 reported cstate change: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "af520b67945d443b901bbeca37cc5701" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:59:15.439395  3008 consensus_queue.cc:1048] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:59:15.456593  2807 consensus_queue.cc:1048] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:59:15.465372  2332 raft_consensus.cc:1275] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:59:15.468601  1750 raft_consensus.cc:1275] T f6028a708cb344fbb08ab8056921d9e7 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:59:15.470677  2998 consensus_queue.cc:1048] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:59:15.471578  2930 consensus_queue.cc:1048] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:59:15.476533  3024 raft_consensus.cc:1064] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d: attempting to promote NON_VOTER 3b879799899a499c8d409ee9f1277cd1 to VOTER
I20260709 12:59:15.492702  3032 raft_consensus.cc:2243] T 84e78b66ceec4f759ae9e101a5469e4c P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:59:15.494030  3032 raft_consensus.cc:2272] T 84e78b66ceec4f759ae9e101a5469e4c P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:59:15.499526  3032 ts_tablet_manager.cc:1916] T 84e78b66ceec4f759ae9e101a5469e4c P af520b67945d443b901bbeca37cc5701: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:59:15.481045  3024 consensus_queue.cc:237] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:15.536535  1753 consensus_queue.cc:237] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:15.543025  2337 raft_consensus.cc:1275] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:59:15.545231  3020 consensus_queue.cc:1048] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:59:15.563060  2337 raft_consensus.cc:2955] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:15.574924  1680 raft_consensus.cc:1275] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:59:15.586285  2932 consensus_queue.cc:1048] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:59:15.586819  3020 consensus_queue.cc:1048] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.003s
I20260709 12:59:15.558615  3029 raft_consensus.cc:2955] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:15.601990  1672 raft_consensus.cc:2955] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
W20260709 12:59:15.617275  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 180589us
I20260709 12:59:15.640046  1596 raft_consensus.cc:2955] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER af520b67945d443b901bbeca37cc5701 (127.0.107.195) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:15.658624  1598 raft_consensus.cc:2955] T 03daa4b346f54f1da820d1ff6cd1ff86 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:15.660862  1604 raft_consensus.cc:1275] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:59:15.663391   431 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:59:15.690706  1503 catalog_manager.cc:5697] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 reported cstate change: config changed from index 3 to 4, VOTER 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) evicted. New cstate: current_term: 1 leader_uuid: "af520b67945d443b901bbeca37cc5701" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:15.700278  2807 consensus_queue.cc:1048] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:59:15.723218  1490 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 03daa4b346f54f1da820d1ff6cd1ff86 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:59:15.718531  1592 tablet_service.cc:1558] Processing DeleteTablet for tablet 03daa4b346f54f1da820d1ff6cd1ff86 with delete_type TABLET_DATA_TOMBSTONED (TS 83595ad0fd5940c0a2bfc7c61acd969a not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:40138
I20260709 12:59:15.819821  3045 tablet_replica.cc:333] T 03daa4b346f54f1da820d1ff6cd1ff86 P 83595ad0fd5940c0a2bfc7c61acd969a: stopping tablet replica
I20260709 12:59:15.846786  2335 raft_consensus.cc:1275] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 1 LEARNER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:59:15.852350  2928 consensus_queue.cc:1048] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:59:15.868644  3032 ts_tablet_manager.cc:1929] T 84e78b66ceec4f759ae9e101a5469e4c P af520b67945d443b901bbeca37cc5701: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20260709 12:59:15.869478  3032 log.cc:1199] T 84e78b66ceec4f759ae9e101a5469e4c P af520b67945d443b901bbeca37cc5701: Deleting WAL directory at /tmp/dist-test-taskakPCWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783601878574137-431-0/minicluster-data/ts-2-root/wals/84e78b66ceec4f759ae9e101a5469e4c
I20260709 12:59:15.872749  1490 catalog_manager.cc:5028] TS af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): tablet 84e78b66ceec4f759ae9e101a5469e4c (table test-workload [id=75d77b57b3f34786b9c215c4f11bd727]) successfully deleted
I20260709 12:59:15.874539  1753 raft_consensus.cc:1275] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:59:15.887228  2807 consensus_queue.cc:1048] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:59:15.899505  1605 raft_consensus.cc:1275] T 79ed08f2339a4b1ead1e70d78eaec2ec P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:59:15.909386  2928 consensus_queue.cc:1048] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:59:15.948221  3045 raft_consensus.cc:2243] T 03daa4b346f54f1da820d1ff6cd1ff86 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:59:15.949115  3045 raft_consensus.cc:2272] T 03daa4b346f54f1da820d1ff6cd1ff86 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:59:15.952795  3024 raft_consensus.cc:2955] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:15.956696  3045 ts_tablet_manager.cc:1916] T 03daa4b346f54f1da820d1ff6cd1ff86 P 83595ad0fd5940c0a2bfc7c61acd969a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:59:15.956135  1605 raft_consensus.cc:2955] T 79ed08f2339a4b1ead1e70d78eaec2ec P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:15.965931  1603 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85"
dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
mode: GRACEFUL
new_leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
 from {username='slave'} at 127.0.0.1:47752
I20260709 12:59:15.966624  1603 raft_consensus.cc:606] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 LEADER]: Received request to transfer leadership to TS 816fadcea8fc4893882f5f9b9f22a03d
I20260709 12:59:15.975152  2335 raft_consensus.cc:2955] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:15.984218  1754 raft_consensus.cc:2955] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:15.990419  1750 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72"
dest_uuid: "af520b67945d443b901bbeca37cc5701"
mode: GRACEFUL
new_leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
 from {username='slave'} at 127.0.0.1:51270
I20260709 12:59:15.993220  1750 raft_consensus.cc:606] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 2 LEADER]: Received request to transfer leadership to TS 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:15.998469  1753 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "86abe5695d5643f888190301b88dacef"
dest_uuid: "af520b67945d443b901bbeca37cc5701"
mode: GRACEFUL
new_leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
 from {username='slave'} at 127.0.0.1:51270
I20260709 12:59:15.999069  1753 raft_consensus.cc:606] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 1 LEADER]: Received request to transfer leadership to TS 816fadcea8fc4893882f5f9b9f22a03d
I20260709 12:59:16.003232  1672 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "d21485c094584689917b2a97b2d3c624"
dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
mode: GRACEFUL
new_leader_uuid: "3b879799899a499c8d409ee9f1277cd1"
 from {username='slave'} at 127.0.0.1:43266
I20260709 12:59:16.003777  1672 raft_consensus.cc:606] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 LEADER]: Received request to transfer leadership to TS 3b879799899a499c8d409ee9f1277cd1
I20260709 12:59:16.003989  1501 catalog_manager.cc:5697] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 reported cstate change: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:16.007354   431 auto_leader_rebalancer.cc:452] table: test-workload, leader rebalance finish, leader transfer count: 4
I20260709 12:59:16.009539   431 auto_leader_rebalancer.cc:527] All tables' leader rebalancing finished this round
I20260709 12:59:16.012117  3045 ts_tablet_manager.cc:1929] T 03daa4b346f54f1da820d1ff6cd1ff86 P 83595ad0fd5940c0a2bfc7c61acd969a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:59:16.012656  3045 log.cc:1199] T 03daa4b346f54f1da820d1ff6cd1ff86 P 83595ad0fd5940c0a2bfc7c61acd969a: Deleting WAL directory at /tmp/dist-test-taskakPCWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783601878574137-431-0/minicluster-data/ts-0-root/wals/03daa4b346f54f1da820d1ff6cd1ff86
I20260709 12:59:16.014546  1487 catalog_manager.cc:5028] TS 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): tablet 03daa4b346f54f1da820d1ff6cd1ff86 (table test-workload [id=75d77b57b3f34786b9c215c4f11bd727]) successfully deleted
I20260709 12:59:16.043485  2940 raft_consensus.cc:993] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701: : Instructing follower 83595ad0fd5940c0a2bfc7c61acd969a to start an election
I20260709 12:59:16.043677  2998 raft_consensus.cc:993] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d: : Instructing follower 3b879799899a499c8d409ee9f1277cd1 to start an election
I20260709 12:59:16.044062  2940 raft_consensus.cc:1081] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 2 LEADER]: Signalling peer 83595ad0fd5940c0a2bfc7c61acd969a to start an election
I20260709 12:59:16.044204  2998 raft_consensus.cc:1081] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 LEADER]: Signalling peer 3b879799899a499c8d409ee9f1277cd1 to start an election
I20260709 12:59:16.046018  2335 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "d21485c094584689917b2a97b2d3c624"
dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
 from {username='slave'} at 127.0.0.1:39726
I20260709 12:59:16.046612  2335 raft_consensus.cc:493] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:59:16.046963  2335 raft_consensus.cc:3060] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:59:16.047259  1605 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72"
dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
 from {username='slave'} at 127.0.0.1:40154
I20260709 12:59:16.047860  1605 raft_consensus.cc:493] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:59:16.048202  1605 raft_consensus.cc:3060] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:59:16.053778  1672 consensus_queue.cc:237] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:16.058604  1605 raft_consensus.cc:515] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:16.063948  1605 leader_election.cc:290] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 3 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:16.065192  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:16.065842  1754 raft_consensus.cc:3055] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:59:16.066144  1754 raft_consensus.cc:740] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 12:59:16.066949  1754 consensus_queue.cc:260] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:16.068066  1754 raft_consensus.cc:3060] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:59:16.075770  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:59:16.076512  1677 raft_consensus.cc:3060] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:59:16.089689  2335 raft_consensus.cc:515] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:16.097383  1754 raft_consensus.cc:2468] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 3.
I20260709 12:59:16.099722  1571 leader_election.cc:304] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:59:16.100560  3013 raft_consensus.cc:2804] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:59:16.101045  3013 raft_consensus.cc:697] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 12:59:16.101899  3013 consensus_queue.cc:237] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:16.103829  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:59:16.104450  1605 raft_consensus.cc:3060] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:59:16.106056  2337 raft_consensus.cc:1275] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:59:16.105269  1754 raft_consensus.cc:1275] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:59:16.107396  3024 consensus_queue.cc:1048] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:59:16.110289  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:59:16.110954  1680 raft_consensus.cc:3055] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:59:16.111251  1680 raft_consensus.cc:740] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 12:59:16.112020  1680 consensus_queue.cc:260] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:16.113162  1680 raft_consensus.cc:3060] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:59:16.115602  1605 raft_consensus.cc:2468] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 3.
I20260709 12:59:16.117276  2928 raft_consensus.cc:2955] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:16.122709  2998 consensus_queue.cc:1048] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:59:16.104353  1677 raft_consensus.cc:2468] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 3.
I20260709 12:59:16.130115  2337 raft_consensus.cc:2955] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:16.131119  2298 leader_election.cc:304] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 12:59:16.139431  1488 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 79ed08f2339a4b1ead1e70d78eaec2ec with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:59:16.136281  1754 raft_consensus.cc:2955] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:16.144611  2933 raft_consensus.cc:2804] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:59:16.146426  2933 raft_consensus.cc:697] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 3 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 12:59:16.147527  2933 consensus_queue.cc:237] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:16.148949  2335 leader_election.cc:290] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 3 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:16.152796  1503 catalog_manager.cc:5697] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d reported cstate change: config changed from index 3 to 4, VOTER 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) evicted. New cstate: current_term: 1 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:59:16.159245  1502 catalog_manager.cc:5697] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 reported cstate change: term changed from 2 to 3, leader changed from 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194) to 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196). New cstate: current_term: 3 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:59:16.140532  1680 raft_consensus.cc:2468] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 3.
I20260709 12:59:16.176175  1592 tablet_service.cc:1558] Processing DeleteTablet for tablet 79ed08f2339a4b1ead1e70d78eaec2ec with delete_type TABLET_DATA_TOMBSTONED (TS 83595ad0fd5940c0a2bfc7c61acd969a not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:40138
I20260709 12:59:16.177167  3045 tablet_replica.cc:333] T 79ed08f2339a4b1ead1e70d78eaec2ec P 83595ad0fd5940c0a2bfc7c61acd969a: stopping tablet replica
I20260709 12:59:16.178308  3045 raft_consensus.cc:2243] T 79ed08f2339a4b1ead1e70d78eaec2ec P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:59:16.179206  3045 raft_consensus.cc:2272] T 79ed08f2339a4b1ead1e70d78eaec2ec P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:59:16.182757  1501 catalog_manager.cc:5697] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 2 to 3, leader changed from af520b67945d443b901bbeca37cc5701 (127.0.107.195) to 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193). New cstate: current_term: 3 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } }
I20260709 12:59:16.218101  3045 ts_tablet_manager.cc:1916] T 79ed08f2339a4b1ead1e70d78eaec2ec P 83595ad0fd5940c0a2bfc7c61acd969a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:59:16.239069  3045 ts_tablet_manager.cc:1929] T 79ed08f2339a4b1ead1e70d78eaec2ec P 83595ad0fd5940c0a2bfc7c61acd969a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:59:16.239713  3045 log.cc:1199] T 79ed08f2339a4b1ead1e70d78eaec2ec P 83595ad0fd5940c0a2bfc7c61acd969a: Deleting WAL directory at /tmp/dist-test-taskakPCWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783601878574137-431-0/minicluster-data/ts-0-root/wals/79ed08f2339a4b1ead1e70d78eaec2ec
I20260709 12:59:16.248831  1487 catalog_manager.cc:5028] TS 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): tablet 79ed08f2339a4b1ead1e70d78eaec2ec (table test-workload [id=75d77b57b3f34786b9c215c4f11bd727]) successfully deleted
I20260709 12:59:16.370364  3019 raft_consensus.cc:993] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701: : Instructing follower 816fadcea8fc4893882f5f9b9f22a03d to start an election
I20260709 12:59:16.370990  3029 raft_consensus.cc:1081] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 1 LEADER]: Signalling peer 816fadcea8fc4893882f5f9b9f22a03d to start an election
I20260709 12:59:16.372720  1672 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "86abe5695d5643f888190301b88dacef"
dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
 from {username='slave'} at 127.0.0.1:52778
I20260709 12:59:16.373247  1672 raft_consensus.cc:493] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:59:16.373561  1672 raft_consensus.cc:3060] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:16.379590  1672 raft_consensus.cc:515] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:16.381404  1672 leader_election.cc:290] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 2 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:16.382215  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:16.382503  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 12:59:16.382910  1753 raft_consensus.cc:3055] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:59:16.383116  2337 raft_consensus.cc:3060] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:16.383253  1753 raft_consensus.cc:740] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 12:59:16.384194  1753 consensus_queue.cc:260] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:16.385205  1753 raft_consensus.cc:3060] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:16.389822  2337 raft_consensus.cc:2468] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 2.
I20260709 12:59:16.391039  1644 leader_election.cc:304] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 12:59:16.391777  1753 raft_consensus.cc:2468] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 2.
I20260709 12:59:16.391870  3024 raft_consensus.cc:2804] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:59:16.392481  3024 raft_consensus.cc:697] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 2 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 12:59:16.393395  3024 consensus_queue.cc:237] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:16.403517  1503 catalog_manager.cc:5697] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d reported cstate change: term changed from 1 to 2, leader changed from af520b67945d443b901bbeca37cc5701 (127.0.107.195) to 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194). New cstate: current_term: 2 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:59:16.424471  3026 raft_consensus.cc:993] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a: : Instructing follower 816fadcea8fc4893882f5f9b9f22a03d to start an election
I20260709 12:59:16.425036  3013 raft_consensus.cc:1081] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 LEADER]: Signalling peer 816fadcea8fc4893882f5f9b9f22a03d to start an election
I20260709 12:59:16.426419  1672 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85"
dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
 from {username='slave'} at 127.0.0.1:52766
I20260709 12:59:16.426857  1672 raft_consensus.cc:493] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:59:16.427136  1672 raft_consensus.cc:3060] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:16.434252  1672 raft_consensus.cc:515] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:16.436012  1672 leader_election.cc:290] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 2 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:16.436661  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:59:16.436906  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:16.437353  1603 raft_consensus.cc:3055] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:59:16.437541  1753 raft_consensus.cc:3060] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:16.437652  1603 raft_consensus.cc:740] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 12:59:16.438622  1603 consensus_queue.cc:260] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:16.439803  1603 raft_consensus.cc:3060] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:16.444154  1753 raft_consensus.cc:2468] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 2.
I20260709 12:59:16.445690  1603 raft_consensus.cc:2468] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 2.
I20260709 12:59:16.446141  1646 leader_election.cc:304] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:59:16.447010  3024 raft_consensus.cc:2804] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:59:16.450280  3024 raft_consensus.cc:697] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 12:59:16.451252  3024 consensus_queue.cc:237] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:16.462203  1503 catalog_manager.cc:5697] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d reported cstate change: term changed from 1 to 2, leader changed from 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) to 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194). New cstate: current_term: 2 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } }
I20260709 12:59:16.503055  1672 raft_consensus.cc:1275] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:59:16.504587  3013 consensus_queue.cc:1048] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:59:16.515120  1753 raft_consensus.cc:1275] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:59:16.519546  3026 consensus_queue.cc:1048] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:59:16.598721  1672 raft_consensus.cc:1275] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260709 12:59:16.600131  2933 consensus_queue.cc:1048] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:59:16.611662  1603 raft_consensus.cc:1275] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260709 12:59:16.613673  2933 consensus_queue.cc:1048] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:59:16.940838  2337 raft_consensus.cc:1275] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:59:16.942087  2928 consensus_queue.cc:1048] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:59:16.955192  1754 raft_consensus.cc:1275] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:59:16.956487  3024 consensus_queue.cc:1048] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:59:16.986037  1754 raft_consensus.cc:1275] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:59:16.987236  3024 consensus_queue.cc:1048] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:59:16.997742  1605 raft_consensus.cc:1275] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:59:17.001541  3024 consensus_queue.cc:1048] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:59:17.010674   431 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:59:17.078152   431 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:59:17.115408  1672 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5"
dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
mode: GRACEFUL
new_leader_uuid: "af520b67945d443b901bbeca37cc5701"
 from {username='slave'} at 127.0.0.1:43266
I20260709 12:59:17.117784  1672 raft_consensus.cc:606] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 LEADER]: Received request to transfer leadership to TS af520b67945d443b901bbeca37cc5701
I20260709 12:59:17.120703  1603 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49"
dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
mode: GRACEFUL
new_leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
 from {username='slave'} at 127.0.0.1:47752
I20260709 12:59:17.121309  1603 raft_consensus.cc:606] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 LEADER]: Received request to transfer leadership to TS 816fadcea8fc4893882f5f9b9f22a03d
I20260709 12:59:17.124148  1672 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91"
dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
mode: GRACEFUL
new_leader_uuid: "af520b67945d443b901bbeca37cc5701"
 from {username='slave'} at 127.0.0.1:43266
I20260709 12:59:17.124692  1672 raft_consensus.cc:606] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 LEADER]: Received request to transfer leadership to TS af520b67945d443b901bbeca37cc5701
I20260709 12:59:17.125901   431 auto_leader_rebalancer.cc:452] table: test-workload, leader rebalance finish, leader transfer count: 3
I20260709 12:59:17.127471   431 auto_leader_rebalancer.cc:527] All tables' leader rebalancing finished this round
I20260709 12:59:17.209410  3024 raft_consensus.cc:993] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d: : Instructing follower af520b67945d443b901bbeca37cc5701 to start an election
I20260709 12:59:17.210162  2928 raft_consensus.cc:1081] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 LEADER]: Signalling peer af520b67945d443b901bbeca37cc5701 to start an election
I20260709 12:59:17.211875  1754 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91"
dest_uuid: "af520b67945d443b901bbeca37cc5701"
 from {username='slave'} at 127.0.0.1:55848
I20260709 12:59:17.212411  1754 raft_consensus.cc:493] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:59:17.212738  1754 raft_consensus.cc:3060] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:59:17.219643  1754 raft_consensus.cc:515] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:17.221732  1754 leader_election.cc:290] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:17.222568  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:59:17.224509  1603 raft_consensus.cc:3060] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:59:17.223847  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:59:17.227149  1677 raft_consensus.cc:3055] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:59:17.227463  1677 raft_consensus.cc:740] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 12:59:17.228219  1677 consensus_queue.cc:260] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:17.229250  1677 raft_consensus.cc:3060] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:59:17.241750  1677 raft_consensus.cc:2468] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 3.
I20260709 12:59:17.242578  1603 raft_consensus.cc:2468] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 3.
I20260709 12:59:17.243364  1718 leader_election.cc:304] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:59:17.244320  2893 raft_consensus.cc:2804] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:59:17.244881  2893 raft_consensus.cc:697] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 3 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 12:59:17.245755  2893 consensus_queue.cc:237] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:17.264729  1501 catalog_manager.cc:5697] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 reported cstate change: term changed from 2 to 3, leader changed from 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194) to af520b67945d443b901bbeca37cc5701 (127.0.107.195). New cstate: current_term: 3 leader_uuid: "af520b67945d443b901bbeca37cc5701" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: HEALTHY } } }
I20260709 12:59:17.308266  1603 consensus_queue.cc:237] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } }
I20260709 12:59:17.317762  1753 raft_consensus.cc:1275] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260709 12:59:17.320041  1569 consensus_peers.cc:597] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a -> Peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625): Couldn't send request to peer 3b879799899a499c8d409ee9f1277cd1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4d4ba8d6f4f947fabf8a2609af29c3d3. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:59:17.320302  3026 consensus_queue.cc:1048] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.003s
I20260709 12:59:17.325515  1677 raft_consensus.cc:1275] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:59:17.327858  3026 consensus_queue.cc:1048] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:59:17.335124  3013 raft_consensus.cc:2955] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } } }
I20260709 12:59:17.338071  1753 raft_consensus.cc:2955] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } } }
I20260709 12:59:17.340617  1677 raft_consensus.cc:2955] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } } }
I20260709 12:59:17.352705  1596 consensus_queue.cc:237] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } }
I20260709 12:59:17.357239  1501 catalog_manager.cc:5697] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 reported cstate change: config changed from index -1 to 2, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New cstate: current_term: 1 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } } }
I20260709 12:59:17.363164  1754 raft_consensus.cc:1275] T 388afdca3ce84bf0a8ce3331ed0d9b2f P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260709 12:59:17.365406  1569 consensus_peers.cc:597] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a -> Peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625): Couldn't send request to peer 3b879799899a499c8d409ee9f1277cd1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 388afdca3ce84bf0a8ce3331ed0d9b2f. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:59:17.365916  3026 consensus_queue.cc:1048] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:59:17.370824  1672 raft_consensus.cc:1275] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:59:17.377570  3006 consensus_queue.cc:1048] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:59:17.399322  3075 raft_consensus.cc:2955] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } } }
I20260709 12:59:17.402359  1672 raft_consensus.cc:2955] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } } }
I20260709 12:59:17.415366  1754 raft_consensus.cc:2955] T 388afdca3ce84bf0a8ce3331ed0d9b2f P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } } }
I20260709 12:59:17.422138  1752 consensus_queue.cc:237] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } }
I20260709 12:59:17.431684  1502 catalog_manager.cc:5697] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: config changed from index -1 to 2, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New cstate: current_term: 1 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20260709 12:59:17.438834  1718 consensus_peers.cc:597] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 -> Peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625): Couldn't send request to peer 3b879799899a499c8d409ee9f1277cd1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dce8db7565da4e13a82ffbb9c39fde78. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:59:17.439396  1596 raft_consensus.cc:1275] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:59:17.440088  1677 raft_consensus.cc:1275] T dce8db7565da4e13a82ffbb9c39fde78 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:59:17.440868  2893 consensus_queue.cc:1048] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:59:17.442304  3067 consensus_queue.cc:1048] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:59:17.456434  2893 raft_consensus.cc:2955] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } } }
I20260709 12:59:17.458009  1596 raft_consensus.cc:2955] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } } }
I20260709 12:59:17.460637  1677 raft_consensus.cc:2955] T dce8db7565da4e13a82ffbb9c39fde78 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } } }
I20260709 12:59:17.473526  1754 consensus_queue.cc:237] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } }
I20260709 12:59:17.480082  3075 raft_consensus.cc:993] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a: : Instructing follower 816fadcea8fc4893882f5f9b9f22a03d to start an election
I20260709 12:59:17.480674  3075 raft_consensus.cc:1081] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 LEADER]: Signalling peer 816fadcea8fc4893882f5f9b9f22a03d to start an election
I20260709 12:59:17.482254  1672 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49"
dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
 from {username='slave'} at 127.0.0.1:52766
I20260709 12:59:17.482791  1672 raft_consensus.cc:493] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:59:17.483150  1672 raft_consensus.cc:3060] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:17.484856  1502 catalog_manager.cc:5697] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: config changed from index -1 to 2, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New cstate: current_term: 1 leader_uuid: "af520b67945d443b901bbeca37cc5701" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } } }
I20260709 12:59:17.491353  1672 raft_consensus.cc:515] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:17.493300  1672 leader_election.cc:290] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 2 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:17.496904  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:59:17.497511  1596 raft_consensus.cc:3055] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:59:17.497687  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 12:59:17.498385  2337 raft_consensus.cc:3060] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:17.497803  1596 raft_consensus.cc:740] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 12:59:17.503364  1596 consensus_queue.cc:260] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:17.504748  1596 raft_consensus.cc:3060] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:17.507341  1603 raft_consensus.cc:1275] T 78cabb03415041ca9c1a5a400e506131 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:59:17.509241  3067 consensus_queue.cc:1048] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:59:17.523386  2337 raft_consensus.cc:2468] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 2.
I20260709 12:59:17.524942  1644 leader_election.cc:304] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 12:59:17.526014  3024 raft_consensus.cc:2804] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:59:17.526593  3024 raft_consensus.cc:697] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 12:59:17.527029  1596 raft_consensus.cc:2468] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 2.
I20260709 12:59:17.527884  3024 consensus_queue.cc:237] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:17.533581  1677 raft_consensus.cc:1275] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:59:17.538324  3083 raft_consensus.cc:2955] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } } }
I20260709 12:59:17.540050  3029 consensus_queue.cc:1048] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260709 12:59:17.545951  1718 consensus_peers.cc:597] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 -> Peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625): Couldn't send request to peer 3b879799899a499c8d409ee9f1277cd1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 78cabb03415041ca9c1a5a400e506131. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:59:17.546118  1596 raft_consensus.cc:2955] T 78cabb03415041ca9c1a5a400e506131 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } } }
I20260709 12:59:17.552047  1501 catalog_manager.cc:5697] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d reported cstate change: term changed from 1 to 2, leader changed from 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) to 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194). New cstate: current_term: 2 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:59:17.571910  1502 catalog_manager.cc:5697] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 reported cstate change: config changed from index -1 to 2, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New cstate: current_term: 1 leader_uuid: "af520b67945d443b901bbeca37cc5701" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260709 12:59:17.572329  1672 raft_consensus.cc:2955] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } } }
I20260709 12:59:17.698704  3063 raft_consensus.cc:993] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d: : Instructing follower af520b67945d443b901bbeca37cc5701 to start an election
I20260709 12:59:17.699296  3024 raft_consensus.cc:1081] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 LEADER]: Signalling peer af520b67945d443b901bbeca37cc5701 to start an election
I20260709 12:59:17.700878  1754 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5"
dest_uuid: "af520b67945d443b901bbeca37cc5701"
 from {username='slave'} at 127.0.0.1:55848
I20260709 12:59:17.701403  1754 raft_consensus.cc:493] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:59:17.701704  1754 raft_consensus.cc:3060] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:17.709973  1754 raft_consensus.cc:515] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:17.712870  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:59:17.713558  1596 raft_consensus.cc:3060] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:17.713430  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:59:17.714679  1672 raft_consensus.cc:3055] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:59:17.715063  1672 raft_consensus.cc:740] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 12:59:17.715955  1672 consensus_queue.cc:260] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:17.717151  1672 raft_consensus.cc:3060] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:17.712870  1754 leader_election.cc:290] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 2 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:17.723706  1596 raft_consensus.cc:2468] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 2.
I20260709 12:59:17.725445  1717 leader_election.cc:304] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:59:17.726270  3083 raft_consensus.cc:2804] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:59:17.726796  3083 raft_consensus.cc:697] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 2 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 12:59:17.727785  3083 consensus_queue.cc:237] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:17.727746  1672 raft_consensus.cc:2468] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 2.
I20260709 12:59:17.742514  1501 catalog_manager.cc:5697] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 reported cstate change: term changed from 1 to 2, leader changed from 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194) to af520b67945d443b901bbeca37cc5701 (127.0.107.195). New cstate: current_term: 2 leader_uuid: "af520b67945d443b901bbeca37cc5701" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: HEALTHY } } }
I20260709 12:59:17.757047  1596 raft_consensus.cc:1275] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:59:17.758486  3083 consensus_queue.cc:1048] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:59:17.817168  1680 raft_consensus.cc:1275] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:59:17.822336  3083 consensus_queue.cc:1048] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:59:17.866865  3093 ts_tablet_manager.cc:933] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1: Initiating tablet copy from peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445)
I20260709 12:59:17.898020  3092 ts_tablet_manager.cc:933] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1: Initiating tablet copy from peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445)
I20260709 12:59:17.935789  3093 tablet_copy_client.cc:323] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Beginning tablet copy session from remote peer at address 127.0.107.193:46445
I20260709 12:59:17.938032  1615 tablet_copy_service.cc:140] P 83595ad0fd5940c0a2bfc7c61acd969a: Received BeginTabletCopySession request for tablet 4d4ba8d6f4f947fabf8a2609af29c3d3 from peer 3b879799899a499c8d409ee9f1277cd1 ({username='slave'} at 127.0.0.1:36462)
I20260709 12:59:17.938630  1615 tablet_copy_service.cc:161] P 83595ad0fd5940c0a2bfc7c61acd969a: Beginning new tablet copy session on tablet 4d4ba8d6f4f947fabf8a2609af29c3d3 from peer 3b879799899a499c8d409ee9f1277cd1 at {username='slave'} at 127.0.0.1:36462: session id = 3b879799899a499c8d409ee9f1277cd1-4d4ba8d6f4f947fabf8a2609af29c3d3
I20260709 12:59:17.941043  1596 raft_consensus.cc:1275] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:59:17.945290  3096 consensus_queue.cc:1048] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:59:17.951110  1615 tablet_copy_source_session.cc:215] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:59:17.956931  3093 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d4ba8d6f4f947fabf8a2609af29c3d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:59:17.962090  3094 ts_tablet_manager.cc:933] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1: Initiating tablet copy from peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:17.970954  3094 tablet_copy_client.cc:323] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Beginning tablet copy session from remote peer at address 127.0.107.195:40693
I20260709 12:59:17.979219  2332 raft_consensus.cc:1275] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:59:17.987522  2928 consensus_queue.cc:1048] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:59:18.011826  3092 tablet_copy_client.cc:323] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Beginning tablet copy session from remote peer at address 127.0.107.193:46445
I20260709 12:59:18.018528  1615 tablet_copy_service.cc:140] P 83595ad0fd5940c0a2bfc7c61acd969a: Received BeginTabletCopySession request for tablet 388afdca3ce84bf0a8ce3331ed0d9b2f from peer 3b879799899a499c8d409ee9f1277cd1 ({username='slave'} at 127.0.0.1:36462)
I20260709 12:59:18.019227  1615 tablet_copy_service.cc:161] P 83595ad0fd5940c0a2bfc7c61acd969a: Beginning new tablet copy session on tablet 388afdca3ce84bf0a8ce3331ed0d9b2f from peer 3b879799899a499c8d409ee9f1277cd1 at {username='slave'} at 127.0.0.1:36462: session id = 3b879799899a499c8d409ee9f1277cd1-388afdca3ce84bf0a8ce3331ed0d9b2f
I20260709 12:59:18.033704  3093 tablet_copy_client.cc:806] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Starting download of 0 data blocks...
I20260709 12:59:18.034554  3093 tablet_copy_client.cc:670] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Starting download of 1 WAL segments...
I20260709 12:59:18.039038  3093 tablet_copy_client.cc:538] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:59:18.040234  1615 tablet_copy_source_session.cc:215] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:59:18.045234  1764 tablet_copy_service.cc:140] P af520b67945d443b901bbeca37cc5701: Received BeginTabletCopySession request for tablet dce8db7565da4e13a82ffbb9c39fde78 from peer 3b879799899a499c8d409ee9f1277cd1 ({username='slave'} at 127.0.0.1:43156)
I20260709 12:59:18.047183  1764 tablet_copy_service.cc:161] P af520b67945d443b901bbeca37cc5701: Beginning new tablet copy session on tablet dce8db7565da4e13a82ffbb9c39fde78 from peer 3b879799899a499c8d409ee9f1277cd1 at {username='slave'} at 127.0.0.1:43156: session id = 3b879799899a499c8d409ee9f1277cd1-dce8db7565da4e13a82ffbb9c39fde78
I20260709 12:59:18.054495  3093 tablet_bootstrap.cc:492] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1: Bootstrap starting.
I20260709 12:59:18.055671  1764 tablet_copy_source_session.cc:215] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:59:18.060191  3094 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dce8db7565da4e13a82ffbb9c39fde78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:59:18.060910  3092 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 388afdca3ce84bf0a8ce3331ed0d9b2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:59:18.088032  3103 ts_tablet_manager.cc:933] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1: Initiating tablet copy from peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:18.096024  3092 tablet_copy_client.cc:806] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Starting download of 0 data blocks...
I20260709 12:59:18.097400  3092 tablet_copy_client.cc:670] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Starting download of 1 WAL segments...
I20260709 12:59:18.098804  3093 tablet_bootstrap.cc:492] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:59:18.099823  3093 tablet_bootstrap.cc:492] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1: Bootstrap complete.
I20260709 12:59:18.100641  3093 ts_tablet_manager.cc:1403] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1: Time spent bootstrapping tablet: real 0.046s	user 0.021s	sys 0.015s
I20260709 12:59:18.105129  3092 tablet_copy_client.cc:538] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:59:18.105247  3093 raft_consensus.cc:359] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } }
I20260709 12:59:18.106734  3093 raft_consensus.cc:740] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Initialized, Role: LEARNER
I20260709 12:59:18.107553  3093 consensus_queue.cc:260] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } }
I20260709 12:59:18.111868  3103 tablet_copy_client.cc:323] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Beginning tablet copy session from remote peer at address 127.0.107.195:40693
I20260709 12:59:18.113706  1764 tablet_copy_service.cc:140] P af520b67945d443b901bbeca37cc5701: Received BeginTabletCopySession request for tablet 78cabb03415041ca9c1a5a400e506131 from peer 3b879799899a499c8d409ee9f1277cd1 ({username='slave'} at 127.0.0.1:43156)
I20260709 12:59:18.114472  1764 tablet_copy_service.cc:161] P af520b67945d443b901bbeca37cc5701: Beginning new tablet copy session on tablet 78cabb03415041ca9c1a5a400e506131 from peer 3b879799899a499c8d409ee9f1277cd1 at {username='slave'} at 127.0.0.1:43156: session id = 3b879799899a499c8d409ee9f1277cd1-78cabb03415041ca9c1a5a400e506131
I20260709 12:59:18.122282  3094 tablet_copy_client.cc:806] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Starting download of 0 data blocks...
I20260709 12:59:18.123199  3094 tablet_copy_client.cc:670] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Starting download of 1 WAL segments...
I20260709 12:59:18.125048  3093 ts_tablet_manager.cc:1434] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1: Time spent starting tablet: real 0.023s	user 0.009s	sys 0.000s
I20260709 12:59:18.128994  3094 tablet_copy_client.cc:538] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:59:18.129783   431 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:59:18.132110  1615 tablet_copy_service.cc:342] P 83595ad0fd5940c0a2bfc7c61acd969a: Request end of tablet copy session 3b879799899a499c8d409ee9f1277cd1-4d4ba8d6f4f947fabf8a2609af29c3d3 received from {username='slave'} at 127.0.0.1:36462
I20260709 12:59:18.132682  1615 tablet_copy_service.cc:434] P 83595ad0fd5940c0a2bfc7c61acd969a: ending tablet copy session 3b879799899a499c8d409ee9f1277cd1-4d4ba8d6f4f947fabf8a2609af29c3d3 on tablet 4d4ba8d6f4f947fabf8a2609af29c3d3 with peer 3b879799899a499c8d409ee9f1277cd1
I20260709 12:59:18.126585  1764 tablet_copy_source_session.cc:215] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:59:18.143640  3094 tablet_bootstrap.cc:492] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1: Bootstrap starting.
I20260709 12:59:18.151391  3103 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 78cabb03415041ca9c1a5a400e506131. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:59:18.175604  3092 tablet_bootstrap.cc:492] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1: Bootstrap starting.
I20260709 12:59:18.186785  3103 tablet_copy_client.cc:806] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Starting download of 0 data blocks...
I20260709 12:59:18.187922  3103 tablet_copy_client.cc:670] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Starting download of 1 WAL segments...
I20260709 12:59:18.189659  1596 raft_consensus.cc:1275] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:59:18.195042  3103 tablet_copy_client.cc:538] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:59:18.202358  3029 consensus_queue.cc:1048] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:59:18.232688  3103 tablet_bootstrap.cc:492] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1: Bootstrap starting.
I20260709 12:59:18.245034  1672 raft_consensus.cc:1275] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:59:18.263906  2893 consensus_queue.cc:1048] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
I20260709 12:59:18.310868  3094 tablet_bootstrap.cc:492] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:59:18.327953  3094 tablet_bootstrap.cc:492] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1: Bootstrap complete.
I20260709 12:59:18.333153  3094 ts_tablet_manager.cc:1403] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1: Time spent bootstrapping tablet: real 0.190s	user 0.016s	sys 0.032s
I20260709 12:59:18.333236  2337 raft_consensus.cc:1217] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:59:18.354315  3092 tablet_bootstrap.cc:492] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:59:18.359412  3094 raft_consensus.cc:359] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } }
I20260709 12:59:18.360404  3094 raft_consensus.cc:740] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Initialized, Role: LEARNER
I20260709 12:59:18.361248  3092 tablet_bootstrap.cc:492] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1: Bootstrap complete.
I20260709 12:59:18.361102  3094 consensus_queue.cc:260] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } }
I20260709 12:59:18.362257  3092 ts_tablet_manager.cc:1403] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1: Time spent bootstrapping tablet: real 0.187s	user 0.050s	sys 0.033s
I20260709 12:59:18.365150   431 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:59:18.401799  3092 raft_consensus.cc:359] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } }
I20260709 12:59:18.413502  3092 raft_consensus.cc:740] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Initialized, Role: LEARNER
I20260709 12:59:18.417514  3092 consensus_queue.cc:260] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } }
I20260709 12:59:18.422654  3094 ts_tablet_manager.cc:1434] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1: Time spent starting tablet: real 0.082s	user 0.008s	sys 0.002s
I20260709 12:59:18.429195  3092 ts_tablet_manager.cc:1434] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1: Time spent starting tablet: real 0.066s	user 0.011s	sys 0.007s
I20260709 12:59:18.430421  1764 tablet_copy_service.cc:342] P af520b67945d443b901bbeca37cc5701: Request end of tablet copy session 3b879799899a499c8d409ee9f1277cd1-dce8db7565da4e13a82ffbb9c39fde78 received from {username='slave'} at 127.0.0.1:43156
I20260709 12:59:18.431087  1764 tablet_copy_service.cc:434] P af520b67945d443b901bbeca37cc5701: ending tablet copy session 3b879799899a499c8d409ee9f1277cd1-dce8db7565da4e13a82ffbb9c39fde78 on tablet dce8db7565da4e13a82ffbb9c39fde78 with peer 3b879799899a499c8d409ee9f1277cd1
I20260709 12:59:18.432554  1615 tablet_copy_service.cc:342] P 83595ad0fd5940c0a2bfc7c61acd969a: Request end of tablet copy session 3b879799899a499c8d409ee9f1277cd1-388afdca3ce84bf0a8ce3331ed0d9b2f received from {username='slave'} at 127.0.0.1:36462
I20260709 12:59:18.433020  1615 tablet_copy_service.cc:434] P 83595ad0fd5940c0a2bfc7c61acd969a: ending tablet copy session 3b879799899a499c8d409ee9f1277cd1-388afdca3ce84bf0a8ce3331ed0d9b2f on tablet 388afdca3ce84bf0a8ce3331ed0d9b2f with peer 3b879799899a499c8d409ee9f1277cd1
I20260709 12:59:18.462675  3103 tablet_bootstrap.cc:492] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:59:18.463867  3103 tablet_bootstrap.cc:492] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1: Bootstrap complete.
I20260709 12:59:18.464759  3103 ts_tablet_manager.cc:1403] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1: Time spent bootstrapping tablet: real 0.232s	user 0.019s	sys 0.020s
I20260709 12:59:18.468549  3103 raft_consensus.cc:359] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } }
I20260709 12:59:18.469432  3103 raft_consensus.cc:740] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Initialized, Role: LEARNER
I20260709 12:59:18.470144  3103 consensus_queue.cc:260] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } }
I20260709 12:59:18.472039  2337 raft_consensus.cc:1217] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:59:18.477828  3103 ts_tablet_manager.cc:1434] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1: Time spent starting tablet: real 0.013s	user 0.006s	sys 0.000s
I20260709 12:59:18.483237  2335 raft_consensus.cc:1217] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:59:18.486711  1764 tablet_copy_service.cc:342] P af520b67945d443b901bbeca37cc5701: Request end of tablet copy session 3b879799899a499c8d409ee9f1277cd1-78cabb03415041ca9c1a5a400e506131 received from {username='slave'} at 127.0.0.1:43156
I20260709 12:59:18.487208  1764 tablet_copy_service.cc:434] P af520b67945d443b901bbeca37cc5701: ending tablet copy session 3b879799899a499c8d409ee9f1277cd1-78cabb03415041ca9c1a5a400e506131 on tablet 78cabb03415041ca9c1a5a400e506131 with peer 3b879799899a499c8d409ee9f1277cd1
I20260709 12:59:18.487705  1605 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb"
dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
mode: GRACEFUL
new_leader_uuid: "3b879799899a499c8d409ee9f1277cd1"
 from {username='slave'} at 127.0.0.1:47752
I20260709 12:59:18.488389  1605 raft_consensus.cc:606] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 LEADER]: Received request to transfer leadership to TS 3b879799899a499c8d409ee9f1277cd1
I20260709 12:59:18.498519  1672 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "65285571995c4ed5b593041598056a22"
dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
mode: GRACEFUL
new_leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
 from {username='slave'} at 127.0.0.1:43266
I20260709 12:59:18.499099  1672 raft_consensus.cc:606] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 LEADER]: Received request to transfer leadership to TS 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:18.508056  1754 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78"
dest_uuid: "af520b67945d443b901bbeca37cc5701"
mode: GRACEFUL
new_leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
 from {username='slave'} at 127.0.0.1:51270
I20260709 12:59:18.508700  1754 raft_consensus.cc:606] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 1 LEADER]: Received request to transfer leadership to TS 816fadcea8fc4893882f5f9b9f22a03d
I20260709 12:59:18.515252   431 auto_leader_rebalancer.cc:452] table: test-workload, leader rebalance finish, leader transfer count: 3
I20260709 12:59:18.531072   431 auto_leader_rebalancer.cc:527] All tables' leader rebalancing finished this round
I20260709 12:59:18.649091  2332 raft_consensus.cc:1217] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:59:18.723457  3006 raft_consensus.cc:993] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a: : Instructing follower 3b879799899a499c8d409ee9f1277cd1 to start an election
I20260709 12:59:18.724067  3006 raft_consensus.cc:1081] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 LEADER]: Signalling peer 3b879799899a499c8d409ee9f1277cd1 to start an election
I20260709 12:59:18.731112  2332 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb"
dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
 from {username='slave'} at 127.0.0.1:49082
I20260709 12:59:18.731823  2332 raft_consensus.cc:493] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:59:18.732261  2332 raft_consensus.cc:3060] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:18.742203  2332 raft_consensus.cc:515] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:18.745527  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:59:18.746150  1605 raft_consensus.cc:3055] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:59:18.746466  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:18.746718  1605 raft_consensus.cc:740] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 12:59:18.747076  1754 raft_consensus.cc:3060] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:18.747517  1605 consensus_queue.cc:260] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:18.748997  1605 raft_consensus.cc:3060] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:18.755012  1754 raft_consensus.cc:2468] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 2.
I20260709 12:59:18.745836  2332 leader_election.cc:290] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 2 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:18.756186  2301 leader_election.cc:304] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:59:18.757059  3065 raft_consensus.cc:2804] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:59:18.757550  3065 raft_consensus.cc:697] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 2 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 12:59:18.758478  3065 consensus_queue.cc:237] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:18.759979  1605 raft_consensus.cc:2468] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 2.
I20260709 12:59:18.776130  1503 catalog_manager.cc:5697] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 reported cstate change: term changed from 1 to 2, leader changed from 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) to 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196). New cstate: current_term: 2 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:59:19.029837  3029 raft_consensus.cc:993] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701: : Instructing follower 816fadcea8fc4893882f5f9b9f22a03d to start an election
I20260709 12:59:19.030638  3029 raft_consensus.cc:1081] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 1 LEADER]: Signalling peer 816fadcea8fc4893882f5f9b9f22a03d to start an election
W20260709 12:59:19.045552  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 66827.5us
I20260709 12:59:19.056666  3122 raft_consensus.cc:1064] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a: attempting to promote NON_VOTER 3b879799899a499c8d409ee9f1277cd1 to VOTER
I20260709 12:59:19.058420  3006 raft_consensus.cc:1064] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a: attempting to promote NON_VOTER 3b879799899a499c8d409ee9f1277cd1 to VOTER
W20260709 12:59:19.064838  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 57975.3us
I20260709 12:59:19.136626  3006 consensus_queue.cc:237] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:19.082783  3115 raft_consensus.cc:993] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d: : Instructing follower 83595ad0fd5940c0a2bfc7c61acd969a to start an election
I20260709 12:59:19.147511  3115 raft_consensus.cc:1081] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 LEADER]: Signalling peer 83595ad0fd5940c0a2bfc7c61acd969a to start an election
W20260709 12:59:19.131826  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 103725us
I20260709 12:59:19.150640  3122 consensus_queue.cc:237] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:19.156081  1753 raft_consensus.cc:1275] T 388afdca3ce84bf0a8ce3331ed0d9b2f P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:59:19.165773  3110 raft_consensus.cc:1064] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701: attempting to promote NON_VOTER 3b879799899a499c8d409ee9f1277cd1 to VOTER
W20260709 12:59:19.166299  3110 raft_consensus.cc:1068] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701: Unable to promote non-voter 3b879799899a499c8d409ee9f1277cd1: Service unavailable: leader transfer in progress
I20260709 12:59:19.167443  1596 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "65285571995c4ed5b593041598056a22"
dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
 from {username='slave'} at 127.0.0.1:40166
I20260709 12:59:19.168144  1596 raft_consensus.cc:493] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:59:19.168540  1596 raft_consensus.cc:3060] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:19.182799  1596 raft_consensus.cc:515] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:19.185107  1596 leader_election.cc:290] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 2 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:19.189611  1680 raft_consensus.cc:1275] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:59:19.190541  1672 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78"
dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
 from {username='slave'} at 127.0.0.1:52778
I20260709 12:59:19.191190  1672 raft_consensus.cc:493] T dce8db7565da4e13a82ffbb9c39fde78 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:59:19.191406  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:59:19.192098  1680 raft_consensus.cc:3055] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:59:19.192175  2337 raft_consensus.cc:1275] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 1 LEARNER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:59:19.192514  1680 raft_consensus.cc:740] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 12:59:19.193506  1680 consensus_queue.cc:260] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:19.195037  1680 raft_consensus.cc:3060] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:19.191543  1672 raft_consensus.cc:3060] T dce8db7565da4e13a82ffbb9c39fde78 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:19.207653  3006 consensus_queue.cc:1048] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:59:19.210305  3127 consensus_queue.cc:1048] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:59:19.212500  3127 consensus_queue.cc:1048] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:59:19.217361  1672 raft_consensus.cc:515] T dce8db7565da4e13a82ffbb9c39fde78 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } }
I20260709 12:59:19.223718  1672 leader_election.cc:290] T dce8db7565da4e13a82ffbb9c39fde78 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 2 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:19.226805  1680 raft_consensus.cc:2468] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 2.
I20260709 12:59:19.232283  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:19.232910  1750 raft_consensus.cc:3055] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:59:19.233242  1750 raft_consensus.cc:740] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 12:59:19.233502  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:19.234115  1751 raft_consensus.cc:3060] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:19.236156  1752 raft_consensus.cc:1275] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:59:19.234073  1750 consensus_queue.cc:260] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } }
I20260709 12:59:19.240068  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:59:19.240693  1598 raft_consensus.cc:3060] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:19.242033  1750 raft_consensus.cc:3060] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:19.252717  3075 consensus_queue.cc:1048] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:59:19.260842  3110 raft_consensus.cc:1064] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701: attempting to promote NON_VOTER 3b879799899a499c8d409ee9f1277cd1 to VOTER
I20260709 12:59:19.266173  3110 consensus_queue.cc:237] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:19.268733  1598 raft_consensus.cc:2468] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 2.
I20260709 12:59:19.275871  1751 raft_consensus.cc:2468] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 2.
I20260709 12:59:19.278174  1643 leader_election.cc:304] T dce8db7565da4e13a82ffbb9c39fde78 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 12:59:19.295367  2332 raft_consensus.cc:1275] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 1 LEARNER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:59:19.295892  1680 raft_consensus.cc:1275] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:59:19.295395  1569 leader_election.cc:304] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 12:59:19.309552  1750 raft_consensus.cc:2468] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 2.
I20260709 12:59:19.313551  3124 raft_consensus.cc:2804] T dce8db7565da4e13a82ffbb9c39fde78 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:59:19.337829  3124 raft_consensus.cc:697] T dce8db7565da4e13a82ffbb9c39fde78 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 12:59:19.340921  3124 consensus_queue.cc:237] T dce8db7565da4e13a82ffbb9c39fde78 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } }
I20260709 12:59:19.344449  2332 raft_consensus.cc:1275] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 1 LEARNER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:59:19.325304  3075 raft_consensus.cc:2804] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:59:19.345909  3075 raft_consensus.cc:697] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 12:59:19.331070  3006 consensus_queue.cc:1048] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.022s
I20260709 12:59:19.347337  1680 raft_consensus.cc:1275] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:59:19.348660  3127 consensus_queue.cc:1048] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:59:19.350359  3075 consensus_queue.cc:237] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:19.354571  3029 consensus_queue.cc:1048] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:59:19.362447  3112 consensus_queue.cc:1048] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:59:19.380957  1596 raft_consensus.cc:1275] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:59:19.384258  3126 consensus_queue.cc:1048] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:59:19.388737  1753 raft_consensus.cc:1275] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:59:19.404418  3113 consensus_queue.cc:1048] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.008s
W20260709 12:59:19.415068  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 80569.3us
I20260709 12:59:19.436784  3135 raft_consensus.cc:2955] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:19.450328  2333 raft_consensus.cc:2955] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:19.492902  1602 raft_consensus.cc:1275] T 78cabb03415041ca9c1a5a400e506131 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:59:19.500699  1680 raft_consensus.cc:2955] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
W20260709 12:59:19.534454  3137 log.cc:927] Time spent T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.088s	user 0.000s	sys 0.001s
I20260709 12:59:19.535341  1501 catalog_manager.cc:5697] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 1 to 2, leader changed from 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194) to 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193). New cstate: current_term: 2 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } }
W20260709 12:59:19.546617  3138 log.cc:927] Time spent T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.120s	user 0.000s	sys 0.002s
I20260709 12:59:19.551703   431 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:59:19.554051  1502 catalog_manager.cc:5697] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 reported cstate change: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
W20260709 12:59:19.560985  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 47736.1us
W20260709 12:59:19.572412  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 74885.3us
I20260709 12:59:19.665710  1677 raft_consensus.cc:2955] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
W20260709 12:59:19.705757  3139 log.cc:927] Time spent T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.097s	user 0.001s	sys 0.000s
I20260709 12:59:19.661412  3110 raft_consensus.cc:2955] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
W20260709 12:59:19.736508  3116 log.cc:927] Time spent T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.113s	user 0.000s	sys 0.000s
I20260709 12:59:19.758738  2332 raft_consensus.cc:2955] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:19.754246  1677 raft_consensus.cc:2955] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
W20260709 12:59:19.782346  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 220384us
I20260709 12:59:19.761140  3149 raft_consensus.cc:2955] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:19.830775  1754 raft_consensus.cc:2955] T 388afdca3ce84bf0a8ce3331ed0d9b2f P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:19.847553  2332 raft_consensus.cc:3060] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 1 LEARNER]: Advancing to term 2
I20260709 12:59:19.859360  2332 raft_consensus.cc:1275] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 2 LEARNER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260709 12:59:19.831082  3123 consensus_queue.cc:1048] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.022s
I20260709 12:59:19.864763  1503 catalog_manager.cc:5697] T dce8db7565da4e13a82ffbb9c39fde78 P 816fadcea8fc4893882f5f9b9f22a03d reported cstate change: term changed from 1 to 2, leader changed from af520b67945d443b901bbeca37cc5701 (127.0.107.195) to 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194). New cstate: current_term: 2 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20260709 12:59:19.883472  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 52164.7us
I20260709 12:59:19.884891  3124 consensus_queue.cc:1048] T dce8db7565da4e13a82ffbb9c39fde78 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.015s
I20260709 12:59:19.900446  1677 raft_consensus.cc:1275] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:59:19.904819  3075 consensus_queue.cc:1048] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:59:19.976554  2337 raft_consensus.cc:2955] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
W20260709 12:59:20.035972  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 62250us
W20260709 12:59:20.043578  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 158755us
W20260709 12:59:20.066645  1775 debug-util.cc:398] Leaking SignalData structure 0x7b080012fda0 after lost signal to thread 434
W20260709 12:59:20.069051  1775 debug-util.cc:398] Leaking SignalData structure 0x7b080012fcc0 after lost signal to thread 1537
I20260709 12:59:20.074348  1754 raft_consensus.cc:1275] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
W20260709 12:59:20.074133  1775 debug-util.cc:398] Leaking SignalData structure 0x7b08003298c0 after lost signal to thread 1629
I20260709 12:59:20.069970  1751 raft_consensus.cc:2955] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
W20260709 12:59:20.081518  1775 debug-util.cc:398] Leaking SignalData structure 0x7b080018e9e0 after lost signal to thread 1704
I20260709 12:59:20.106841  1602 raft_consensus.cc:2955] T 78cabb03415041ca9c1a5a400e506131 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:20.129323  3024 raft_consensus.cc:1064] T dce8db7565da4e13a82ffbb9c39fde78 P 816fadcea8fc4893882f5f9b9f22a03d: attempting to promote NON_VOTER 3b879799899a499c8d409ee9f1277cd1 to VOTER
W20260709 12:59:20.130137  3024 raft_consensus.cc:1068] T dce8db7565da4e13a82ffbb9c39fde78 P 816fadcea8fc4893882f5f9b9f22a03d: Unable to promote non-voter 3b879799899a499c8d409ee9f1277cd1: Illegal state: Leader has not yet committed an operation in its own term
W20260709 12:59:20.184122  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 67622.5us
I20260709 12:59:20.184584  3157 consensus_queue.cc:1048] T dce8db7565da4e13a82ffbb9c39fde78 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:59:20.189476  1745 raft_consensus.cc:1275] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260709 12:59:20.193261  1775 debug-util.cc:398] Leaking SignalData structure 0x7b080033c9e0 after lost signal to thread 1778
W20260709 12:59:20.195405  1775 debug-util.cc:398] Leaking SignalData structure 0x7b08002beb80 after lost signal to thread 2361
W20260709 12:59:20.240526  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 101077us
I20260709 12:59:20.244755  3162 consensus_queue.cc:1048] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:59:20.352896  1501 catalog_manager.cc:5697] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 reported cstate change: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:20.392117  3168 raft_consensus.cc:1064] T dce8db7565da4e13a82ffbb9c39fde78 P 816fadcea8fc4893882f5f9b9f22a03d: attempting to promote NON_VOTER 3b879799899a499c8d409ee9f1277cd1 to VOTER
I20260709 12:59:20.404908  3168 consensus_queue.cc:237] T dce8db7565da4e13a82ffbb9c39fde78 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 2, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:20.454739  1502 catalog_manager.cc:5697] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d reported cstate change: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "af520b67945d443b901bbeca37cc5701" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:20.476927  1599 consensus_queue.cc:237] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:20.487848  2335 raft_consensus.cc:1275] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:59:20.490177  1602 consensus_queue.cc:237] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:20.501654  1602 raft_consensus.cc:1275] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260709 12:59:20.504052  3167 consensus_queue.cc:1048] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:59:20.507036  2335 raft_consensus.cc:1275] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:59:20.509019  3024 consensus_queue.cc:1048] T dce8db7565da4e13a82ffbb9c39fde78 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:59:20.512885  3162 consensus_queue.cc:1048] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:59:20.513231  1676 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78"
dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:52760
I20260709 12:59:20.513978  1676 raft_consensus.cc:606] T dce8db7565da4e13a82ffbb9c39fde78 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 LEADER]: Received request to transfer leadership
I20260709 12:59:20.529217  1753 consensus_queue.cc:237] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:20.537371  2332 raft_consensus.cc:1275] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 2 LEARNER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260709 12:59:20.541391  2333 raft_consensus.cc:1275] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:59:20.543061  3123 consensus_queue.cc:1048] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:59:20.544039  1752 raft_consensus.cc:1275] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260709 12:59:20.543172  3122 raft_consensus.cc:2955] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:20.546034  3024 consensus_queue.cc:1048] T dce8db7565da4e13a82ffbb9c39fde78 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:59:20.545483  1750 raft_consensus.cc:1275] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:59:20.553184  1676 raft_consensus.cc:1275] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:59:20.556269  1671 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78"
dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:52760
I20260709 12:59:20.556886  1671 raft_consensus.cc:606] T dce8db7565da4e13a82ffbb9c39fde78 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 LEADER]: Received request to transfer leadership
I20260709 12:59:20.558418  3024 consensus_queue.cc:1048] T dce8db7565da4e13a82ffbb9c39fde78 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:59:20.567341  1672 raft_consensus.cc:1275] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20260709 12:59:20.572050  1488 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: leadership transfer for dce8db7565da4e13a82ffbb9c39fde78 already in progress [suppressed 52 similar messages]
I20260709 12:59:20.572225  3110 consensus_queue.cc:1048] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:59:20.569523  2335 raft_consensus.cc:2955] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:20.576915  2333 raft_consensus.cc:2955] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER af520b67945d443b901bbeca37cc5701 (127.0.107.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:20.559262  3162 raft_consensus.cc:2955] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER af520b67945d443b901bbeca37cc5701 (127.0.107.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:20.596630  3172 consensus_queue.cc:1048] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.011s
I20260709 12:59:20.574608  3167 consensus_queue.cc:1048] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.020s
I20260709 12:59:20.614050  3123 raft_consensus.cc:2955] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
W20260709 12:59:20.625555  1488 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.LeaderStepDown -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 58460.3us
I20260709 12:59:20.631356  3169 raft_consensus.cc:993] T dce8db7565da4e13a82ffbb9c39fde78 P 816fadcea8fc4893882f5f9b9f22a03d: : Instructing follower 83595ad0fd5940c0a2bfc7c61acd969a to start an election
I20260709 12:59:20.618824  2332 raft_consensus.cc:2955] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:20.634030  1753 raft_consensus.cc:2955] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
W20260709 12:59:20.730095  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 70924.3us
I20260709 12:59:20.648325  2336 raft_consensus.cc:2955] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
W20260709 12:59:20.738794  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 70152.8us
I20260709 12:59:20.700960  1677 raft_consensus.cc:2955] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:20.648325  1745 raft_consensus.cc:2955] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:20.760463  3169 raft_consensus.cc:1081] T dce8db7565da4e13a82ffbb9c39fde78 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 LEADER]: Signalling peer 83595ad0fd5940c0a2bfc7c61acd969a to start an election
I20260709 12:59:20.789604  1672 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78"
dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:52760
I20260709 12:59:20.761950  3124 raft_consensus.cc:2955] T dce8db7565da4e13a82ffbb9c39fde78 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 2 to 4, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:20.832417  1671 raft_consensus.cc:2955] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER af520b67945d443b901bbeca37cc5701 (127.0.107.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:20.838613  1487 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 4d4ba8d6f4f947fabf8a2609af29c3d3 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:59:20.857543  1598 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78"
dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
 from {username='slave'} at 127.0.0.1:40166
I20260709 12:59:20.858328  1598 raft_consensus.cc:493] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:59:20.858788  1598 raft_consensus.cc:3060] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:59:20.885143  1487 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 388afdca3ce84bf0a8ce3331ed0d9b2f with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:59:20.891192  1490 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 78cabb03415041ca9c1a5a400e506131 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:59:20.898208  1598 raft_consensus.cc:515] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:20.902100  1500 catalog_manager.cc:5697] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 reported cstate change: config changed from index 2 to 4, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:20.905251  1598 leader_election.cc:290] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 3 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:20.906983  1672 raft_consensus.cc:606] T dce8db7565da4e13a82ffbb9c39fde78 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 LEADER]: Received request to transfer leadership
I20260709 12:59:20.904001  1500 catalog_manager.cc:5697] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 reported cstate change: config changed from index 3 to 4, VOTER af520b67945d443b901bbeca37cc5701 (127.0.107.195) evicted. New cstate: current_term: 1 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:20.908305  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:59:20.908218  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 12:59:20.908155  1500 catalog_manager.cc:5697] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 reported cstate change: config changed from index 3 to 4, VOTER 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194) evicted. New cstate: current_term: 1 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:20.909166  2332 raft_consensus.cc:3060] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:59:20.909860  1500 catalog_manager.cc:5697] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 reported cstate change: config changed from index 3 to 4, VOTER 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) evicted. New cstate: current_term: 1 leader_uuid: "af520b67945d443b901bbeca37cc5701" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:20.911381  1605 raft_consensus.cc:1240] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 2. Current term is 3. Ops: []
I20260709 12:59:20.912810  3024 consensus_queue.cc:1059] T dce8db7565da4e13a82ffbb9c39fde78 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 2.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:59:20.900839   431 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:59:20.936244  2332 raft_consensus.cc:2468] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 3.
I20260709 12:59:20.936973  1677 raft_consensus.cc:3055] T dce8db7565da4e13a82ffbb9c39fde78 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 LEADER]: Stepping down as leader of term 2
W20260709 12:59:20.941080  1551 auto_rebalancer.cc:893] Removing replace marker failed: Service unavailable: leader transfer in progress
I20260709 12:59:20.941886  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:20.942541  1753 raft_consensus.cc:3060] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:59:20.941607  1551 auto_rebalancer.cc:901] Could not move replica: Remote error: Service unavailable: leadership transfer for dce8db7565da4e13a82ffbb9c39fde78 already in progress
W20260709 12:59:20.944952  1551 auto_rebalancer.cc:330] scheduled replica move failed to complete: Remote error: Service unavailable: leadership transfer for dce8db7565da4e13a82ffbb9c39fde78 already in progress
I20260709 12:59:20.963296  1753 raft_consensus.cc:2468] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 3.
I20260709 12:59:20.937325  1677 raft_consensus.cc:740] T dce8db7565da4e13a82ffbb9c39fde78 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 12:59:20.966565  1677 consensus_queue.cc:260] T dce8db7565da4e13a82ffbb9c39fde78 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:20.968021  1677 raft_consensus.cc:3060] T dce8db7565da4e13a82ffbb9c39fde78 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:59:20.971982  1571 leader_election.cc:304] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:59:20.980472  3127 raft_consensus.cc:2804] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:59:20.980988  3127 raft_consensus.cc:697] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 12:59:20.981962  3127 consensus_queue.cc:237] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:20.989933  1740 tablet_service.cc:1558] Processing DeleteTablet for tablet 388afdca3ce84bf0a8ce3331ed0d9b2f with delete_type TABLET_DATA_TOMBSTONED (TS af520b67945d443b901bbeca37cc5701 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:55838
I20260709 12:59:20.991577  1665 tablet_service.cc:1558] Processing DeleteTablet for tablet 4d4ba8d6f4f947fabf8a2609af29c3d3 with delete_type TABLET_DATA_TOMBSTONED (TS 816fadcea8fc4893882f5f9b9f22a03d not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:52760
I20260709 12:59:21.016958  1677 raft_consensus.cc:2468] T dce8db7565da4e13a82ffbb9c39fde78 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 3.
I20260709 12:59:21.019856  3185 tablet_replica.cc:333] T 388afdca3ce84bf0a8ce3331ed0d9b2f P af520b67945d443b901bbeca37cc5701: stopping tablet replica
I20260709 12:59:21.026878  3185 raft_consensus.cc:2243] T 388afdca3ce84bf0a8ce3331ed0d9b2f P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:59:21.027846  3185 raft_consensus.cc:2272] T 388afdca3ce84bf0a8ce3331ed0d9b2f P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:59:21.025648  3186 tablet_replica.cc:333] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 816fadcea8fc4893882f5f9b9f22a03d: stopping tablet replica
I20260709 12:59:21.029713  1592 tablet_service.cc:1558] Processing DeleteTablet for tablet 78cabb03415041ca9c1a5a400e506131 with delete_type TABLET_DATA_TOMBSTONED (TS 83595ad0fd5940c0a2bfc7c61acd969a not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:40138
I20260709 12:59:21.033159  3185 ts_tablet_manager.cc:1916] T 388afdca3ce84bf0a8ce3331ed0d9b2f P af520b67945d443b901bbeca37cc5701: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:59:21.035703  3187 tablet_replica.cc:333] T 78cabb03415041ca9c1a5a400e506131 P 83595ad0fd5940c0a2bfc7c61acd969a: stopping tablet replica
W20260709 12:59:21.047304  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 43905.3us
I20260709 12:59:21.049768  3187 raft_consensus.cc:2243] T 78cabb03415041ca9c1a5a400e506131 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:59:21.050539  3187 raft_consensus.cc:2272] T 78cabb03415041ca9c1a5a400e506131 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:59:21.060153  3187 ts_tablet_manager.cc:1916] T 78cabb03415041ca9c1a5a400e506131 P 83595ad0fd5940c0a2bfc7c61acd969a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:59:21.063540  1502 catalog_manager.cc:5697] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: config changed from index 4 to 2, term changed from 2 to 3, leader changed from 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194) to 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from VOTER to NON_VOTER, now has a pending config: VOTER 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193), VOTER 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194), VOTER af520b67945d443b901bbeca37cc5701 (127.0.107.195), VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196). New cstate: current_term: 3 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } health_report { overall_health: UNKNOWN } } } pending_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:21.095129  3186 raft_consensus.cc:2243] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:59:21.096271  3185 ts_tablet_manager.cc:1929] T 388afdca3ce84bf0a8ce3331ed0d9b2f P af520b67945d443b901bbeca37cc5701: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:59:21.096262  3186 raft_consensus.cc:2272] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:59:21.097105  3185 log.cc:1199] T 388afdca3ce84bf0a8ce3331ed0d9b2f P af520b67945d443b901bbeca37cc5701: Deleting WAL directory at /tmp/dist-test-taskakPCWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783601878574137-431-0/minicluster-data/ts-2-root/wals/388afdca3ce84bf0a8ce3331ed0d9b2f
I20260709 12:59:21.100730  3186 ts_tablet_manager.cc:1916] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 816fadcea8fc4893882f5f9b9f22a03d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:59:21.107177  1490 catalog_manager.cc:5028] TS af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): tablet 388afdca3ce84bf0a8ce3331ed0d9b2f (table test-workload [id=75d77b57b3f34786b9c215c4f11bd727]) successfully deleted
I20260709 12:59:21.107725  1598 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f"
dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
mode: GRACEFUL
new_leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
 from {username='slave'} at 127.0.0.1:47752
I20260709 12:59:21.108314  1598 raft_consensus.cc:606] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 LEADER]: Received request to transfer leadership to TS 816fadcea8fc4893882f5f9b9f22a03d
I20260709 12:59:21.112473  1677 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36"
dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
mode: GRACEFUL
new_leader_uuid: "af520b67945d443b901bbeca37cc5701"
 from {username='slave'} at 127.0.0.1:43266
I20260709 12:59:21.113072  1677 raft_consensus.cc:606] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 LEADER]: Received request to transfer leadership to TS af520b67945d443b901bbeca37cc5701
I20260709 12:59:21.116572  1753 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91"
dest_uuid: "af520b67945d443b901bbeca37cc5701"
mode: GRACEFUL
new_leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
 from {username='slave'} at 127.0.0.1:51270
I20260709 12:59:21.117115  1753 raft_consensus.cc:606] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 3 LEADER]: Received request to transfer leadership to TS 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:21.119490   431 auto_leader_rebalancer.cc:452] table: test-workload, leader rebalance finish, leader transfer count: 3
I20260709 12:59:21.121553   431 auto_leader_rebalancer.cc:527] All tables' leader rebalancing finished this round
I20260709 12:59:21.137607  3187 ts_tablet_manager.cc:1929] T 78cabb03415041ca9c1a5a400e506131 P 83595ad0fd5940c0a2bfc7c61acd969a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:59:21.138258  3187 log.cc:1199] T 78cabb03415041ca9c1a5a400e506131 P 83595ad0fd5940c0a2bfc7c61acd969a: Deleting WAL directory at /tmp/dist-test-taskakPCWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783601878574137-431-0/minicluster-data/ts-0-root/wals/78cabb03415041ca9c1a5a400e506131
I20260709 12:59:21.140166  3186 ts_tablet_manager.cc:1929] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 816fadcea8fc4893882f5f9b9f22a03d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:59:21.140734  3186 log.cc:1199] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 816fadcea8fc4893882f5f9b9f22a03d: Deleting WAL directory at /tmp/dist-test-taskakPCWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783601878574137-431-0/minicluster-data/ts-1-root/wals/4d4ba8d6f4f947fabf8a2609af29c3d3
I20260709 12:59:21.140626  1487 catalog_manager.cc:5028] TS 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): tablet 78cabb03415041ca9c1a5a400e506131 (table test-workload [id=75d77b57b3f34786b9c215c4f11bd727]) successfully deleted
I20260709 12:59:21.142889  1488 catalog_manager.cc:5028] TS 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): tablet 4d4ba8d6f4f947fabf8a2609af29c3d3 (table test-workload [id=75d77b57b3f34786b9c215c4f11bd727]) successfully deleted
I20260709 12:59:21.241861  3167 raft_consensus.cc:993] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a: : Instructing follower 816fadcea8fc4893882f5f9b9f22a03d to start an election
I20260709 12:59:21.242741  3127 raft_consensus.cc:1081] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 LEADER]: Signalling peer 816fadcea8fc4893882f5f9b9f22a03d to start an election
I20260709 12:59:21.248322  1677 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f"
dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
 from {username='slave'} at 127.0.0.1:52766
I20260709 12:59:21.249028  1677 raft_consensus.cc:493] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:59:21.249434  1677 raft_consensus.cc:3060] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:21.262756  1677 raft_consensus.cc:515] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:21.266891  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 12:59:21.267585  2336 raft_consensus.cc:3060] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:21.266520  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:59:21.272866  1605 raft_consensus.cc:3055] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:59:21.273319  1605 raft_consensus.cc:740] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 12:59:21.275030  1605 consensus_queue.cc:260] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:21.276633  1605 raft_consensus.cc:3060] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:21.278367  2336 raft_consensus.cc:2468] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 2.
I20260709 12:59:21.280251  1644 leader_election.cc:304] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 12:59:21.282198  3024 raft_consensus.cc:2804] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:59:21.282701  3024 raft_consensus.cc:697] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 2 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 12:59:21.285506  1605 raft_consensus.cc:2468] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 2.
I20260709 12:59:21.283622  3024 consensus_queue.cc:237] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:21.278556  1677 leader_election.cc:290] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 2 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:21.314730  1502 catalog_manager.cc:5697] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d reported cstate change: term changed from 1 to 2, leader changed from 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) to 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194). New cstate: current_term: 2 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:59:21.380088  3146 raft_consensus.cc:993] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d: : Instructing follower af520b67945d443b901bbeca37cc5701 to start an election
I20260709 12:59:21.380719  3124 raft_consensus.cc:1081] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 LEADER]: Signalling peer af520b67945d443b901bbeca37cc5701 to start an election
I20260709 12:59:21.382534  1745 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36"
dest_uuid: "af520b67945d443b901bbeca37cc5701"
 from {username='slave'} at 127.0.0.1:55848
I20260709 12:59:21.383152  1745 raft_consensus.cc:493] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:59:21.383551  1745 raft_consensus.cc:3060] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:21.393885  1745 raft_consensus.cc:515] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:21.397907  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:59:21.398680  1598 raft_consensus.cc:3060] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:21.398567  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:59:21.399359  1673 raft_consensus.cc:3055] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:59:21.399482  1745 leader_election.cc:290] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 2 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:21.399729  1673 raft_consensus.cc:740] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 12:59:21.400635  1673 consensus_queue.cc:260] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:21.401901  1673 raft_consensus.cc:3060] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:21.409813  1673 raft_consensus.cc:2468] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 2.
I20260709 12:59:21.410764  1598 raft_consensus.cc:2468] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 2.
I20260709 12:59:21.411886  1717 leader_election.cc:304] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:59:21.412787  3158 raft_consensus.cc:2804] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:59:21.413311  3158 raft_consensus.cc:697] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 2 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 12:59:21.414330  3158 consensus_queue.cc:237] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:21.426545  1502 catalog_manager.cc:5697] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 reported cstate change: term changed from 1 to 2, leader changed from 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194) to af520b67945d443b901bbeca37cc5701 (127.0.107.195). New cstate: current_term: 2 leader_uuid: "af520b67945d443b901bbeca37cc5701" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: HEALTHY } } }
I20260709 12:59:21.434809  1745 raft_consensus.cc:1275] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 12:59:21.447114  3127 consensus_queue.cc:1048] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:59:21.457345  1502 catalog_manager.cc:5697] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 reported cstate change: config changed from index 2 to 4, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER, no longer has a pending config: VOTER 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193), VOTER 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194), VOTER af520b67945d443b901bbeca37cc5701 (127.0.107.195), VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196). New cstate: current_term: 3 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:21.501856  1673 raft_consensus.cc:1275] T dce8db7565da4e13a82ffbb9c39fde78 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 12:59:21.503093  3127 consensus_queue.cc:1048] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:59:21.512234  3167 raft_consensus.cc:2955] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 LEADER]: Committing config change with OpId 2.4: config changed from index 2 to 4, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:21.538206  2336 raft_consensus.cc:1275] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 12:59:21.571627  3127 consensus_queue.cc:1048] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:59:21.612514  3110 raft_consensus.cc:993] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701: : Instructing follower 83595ad0fd5940c0a2bfc7c61acd969a to start an election
I20260709 12:59:21.613114  3110 raft_consensus.cc:1081] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 3 LEADER]: Signalling peer 83595ad0fd5940c0a2bfc7c61acd969a to start an election
I20260709 12:59:21.615614  1605 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91"
dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
 from {username='slave'} at 127.0.0.1:40154
I20260709 12:59:21.616200  1605 raft_consensus.cc:493] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:59:21.616562  1605 raft_consensus.cc:3060] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:59:21.623778  1605 raft_consensus.cc:515] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:21.625379  1605 leader_election.cc:290] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 4 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:21.625993  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:59:21.626307  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:21.626737  1673 raft_consensus.cc:3060] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:59:21.626940  1745 raft_consensus.cc:3055] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:59:21.627329  1745 raft_consensus.cc:740] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 12:59:21.628103  1745 consensus_queue.cc:260] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:21.629205  1745 raft_consensus.cc:3060] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:59:21.634693  1673 raft_consensus.cc:2468] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 4.
I20260709 12:59:21.636189  1569 leader_election.cc:304] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 12:59:21.635480  1745 raft_consensus.cc:2468] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 4.
I20260709 12:59:21.637189  3127 raft_consensus.cc:2804] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:59:21.637946  3127 raft_consensus.cc:697] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 12:59:21.639904  3127 consensus_queue.cc:237] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:21.658015  1502 catalog_manager.cc:5697] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 3 to 4, leader changed from af520b67945d443b901bbeca37cc5701 (127.0.107.195) to 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193). New cstate: current_term: 4 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } }
I20260709 12:59:21.708819  2336 raft_consensus.cc:1275] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:59:21.710098  3146 consensus_queue.cc:1048] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:59:21.724859  1605 raft_consensus.cc:1275] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:59:21.726111  3146 consensus_queue.cc:1048] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:59:21.806717  1673 raft_consensus.cc:1275] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:59:21.808334  3158 consensus_queue.cc:1048] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:59:21.814270  1605 raft_consensus.cc:1275] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:59:21.815878  3158 consensus_queue.cc:1048] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:59:21.980618  1605 consensus_queue.cc:237] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:21.987244  2336 raft_consensus.cc:1275] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260709 12:59:21.987556  1753 raft_consensus.cc:1275] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260709 12:59:21.988798  3162 consensus_queue.cc:1048] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:59:21.991530  3162 consensus_queue.cc:1048] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:59:21.997884  3162 raft_consensus.cc:2955] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 LEADER]: Committing config change with OpId 3.6: config changed from index 4 to 6, VOTER 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:22.001942  1753 raft_consensus.cc:2955] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Committing config change with OpId 3.6: config changed from index 4 to 6, VOTER 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:22.006686  2336 raft_consensus.cc:2955] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Committing config change with OpId 3.6: config changed from index 4 to 6, VOTER 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:22.023006  1487 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet dce8db7565da4e13a82ffbb9c39fde78 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:59:22.027715  1502 catalog_manager.cc:5697] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: config changed from index 4 to 6, VOTER 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194) evicted. New cstate: current_term: 3 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:59:22.044737  1665 tablet_service.cc:1558] Processing DeleteTablet for tablet dce8db7565da4e13a82ffbb9c39fde78 with delete_type TABLET_DATA_TOMBSTONED (TS 816fadcea8fc4893882f5f9b9f22a03d not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:52760
I20260709 12:59:22.051033  1677 raft_consensus.cc:1275] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:59:22.051846  3200 tablet_replica.cc:333] T dce8db7565da4e13a82ffbb9c39fde78 P 816fadcea8fc4893882f5f9b9f22a03d: stopping tablet replica
I20260709 12:59:22.053083  3200 raft_consensus.cc:2243] T dce8db7565da4e13a82ffbb9c39fde78 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:59:22.053758  3200 raft_consensus.cc:2272] T dce8db7565da4e13a82ffbb9c39fde78 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:59:22.053611  3127 consensus_queue.cc:1048] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:59:22.061869  3200 ts_tablet_manager.cc:1916] T dce8db7565da4e13a82ffbb9c39fde78 P 816fadcea8fc4893882f5f9b9f22a03d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:59:22.105099  3200 ts_tablet_manager.cc:1929] T dce8db7565da4e13a82ffbb9c39fde78 P 816fadcea8fc4893882f5f9b9f22a03d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.5
I20260709 12:59:22.105958  3200 log.cc:1199] T dce8db7565da4e13a82ffbb9c39fde78 P 816fadcea8fc4893882f5f9b9f22a03d: Deleting WAL directory at /tmp/dist-test-taskakPCWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783601878574137-431-0/minicluster-data/ts-1-root/wals/dce8db7565da4e13a82ffbb9c39fde78
I20260709 12:59:22.108465  1488 catalog_manager.cc:5028] TS 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): tablet dce8db7565da4e13a82ffbb9c39fde78 (table test-workload [id=75d77b57b3f34786b9c215c4f11bd727]) successfully deleted
I20260709 12:59:22.118275  1753 raft_consensus.cc:1275] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:59:22.122498  3162 consensus_queue.cc:1048] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:59:22.122877   431 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:59:22.267764  1672 consensus_queue.cc:237] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } }
W20260709 12:59:22.285683  1644 consensus_peers.cc:597] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d -> Peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625): Couldn't send request to peer 3b879799899a499c8d409ee9f1277cd1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 83226095f79a46dab445849b1febd425. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:59:22.286795  1598 raft_consensus.cc:1275] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260709 12:59:22.286803  1753 raft_consensus.cc:1275] T 83226095f79a46dab445849b1febd425 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260709 12:59:22.288414  3124 consensus_queue.cc:1048] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:59:22.289958  3024 consensus_queue.cc:1048] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:59:22.308187  3124 raft_consensus.cc:2955] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } } }
I20260709 12:59:22.311115  1753 raft_consensus.cc:2955] T 83226095f79a46dab445849b1febd425 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } } }
I20260709 12:59:22.312523  1605 raft_consensus.cc:2955] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } } }
I20260709 12:59:22.329835  1745 consensus_queue.cc:237] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } }
I20260709 12:59:22.337527  1502 catalog_manager.cc:5697] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d reported cstate change: config changed from index -1 to 3, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New cstate: current_term: 2 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260709 12:59:22.360462  1605 raft_consensus.cc:1275] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260709 12:59:22.360983  1718 consensus_peers.cc:597] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 -> Peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625): Couldn't send request to peer 3b879799899a499c8d409ee9f1277cd1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6f6cc48def184bcc907c18c98277187f. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:59:22.362272  1672 raft_consensus.cc:1275] T 6f6cc48def184bcc907c18c98277187f P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:59:22.362088  3110 consensus_queue.cc:1048] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:59:22.369427  3203 consensus_queue.cc:1048] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:59:22.385201   431 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:59:22.390350  3029 raft_consensus.cc:2955] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } } }
I20260709 12:59:22.393124  1677 raft_consensus.cc:2955] T 6f6cc48def184bcc907c18c98277187f P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } } }
I20260709 12:59:22.402498  1598 raft_consensus.cc:2955] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } } }
I20260709 12:59:22.422263  1605 consensus_queue.cc:237] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } }
I20260709 12:59:22.417199  1499 catalog_manager.cc:5697] T 6f6cc48def184bcc907c18c98277187f P 816fadcea8fc4893882f5f9b9f22a03d reported cstate change: config changed from index -1 to 2, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New cstate: current_term: 1 leader_uuid: "af520b67945d443b901bbeca37cc5701" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } } }
I20260709 12:59:22.442011  1745 raft_consensus.cc:1275] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:59:22.447204  3162 consensus_queue.cc:1048] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260709 12:59:22.450094  1569 consensus_peers.cc:597] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a -> Peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625): Couldn't send request to peer 3b879799899a499c8d409ee9f1277cd1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 72f3cdc0abae4f74aad5e4a15cb02e1b. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:59:22.456483  1677 raft_consensus.cc:1275] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:59:22.458259  3192 consensus_queue.cc:1048] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:59:22.477928  1605 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62"
dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
mode: GRACEFUL
new_leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
 from {username='slave'} at 127.0.0.1:47752
I20260709 12:59:22.481482  1605 raft_consensus.cc:606] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 LEADER]: Received request to transfer leadership to TS 816fadcea8fc4893882f5f9b9f22a03d
I20260709 12:59:22.495127  1753 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3"
dest_uuid: "af520b67945d443b901bbeca37cc5701"
mode: GRACEFUL
new_leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
 from {username='slave'} at 127.0.0.1:51270
I20260709 12:59:22.498620  1753 raft_consensus.cc:606] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 1 LEADER]: Received request to transfer leadership to TS 816fadcea8fc4893882f5f9b9f22a03d
I20260709 12:59:22.509194  1753 raft_consensus.cc:2955] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } } }
I20260709 12:59:22.506238  3192 raft_consensus.cc:2955] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } } }
I20260709 12:59:22.525051  1677 raft_consensus.cc:2955] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } } }
I20260709 12:59:22.532207  1672 consensus_queue.cc:237] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } }
I20260709 12:59:22.535714  1605 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3"
dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
mode: GRACEFUL
new_leader_uuid: "af520b67945d443b901bbeca37cc5701"
 from {username='slave'} at 127.0.0.1:47752
I20260709 12:59:22.536376  1605 raft_consensus.cc:606] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 LEADER]: Received request to transfer leadership to TS af520b67945d443b901bbeca37cc5701
I20260709 12:59:22.549173  1502 catalog_manager.cc:5697] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 reported cstate change: config changed from index -1 to 2, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New cstate: current_term: 1 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } } }
I20260709 12:59:22.551043  1677 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec"
dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
mode: GRACEFUL
new_leader_uuid: "3b879799899a499c8d409ee9f1277cd1"
 from {username='slave'} at 127.0.0.1:43266
I20260709 12:59:22.551728  1677 raft_consensus.cc:606] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 1 LEADER]: Received request to transfer leadership to TS 3b879799899a499c8d409ee9f1277cd1
I20260709 12:59:22.553458   431 auto_leader_rebalancer.cc:452] table: test-workload, leader rebalance finish, leader transfer count: 4
I20260709 12:59:22.555720   431 auto_leader_rebalancer.cc:527] All tables' leader rebalancing finished this round
I20260709 12:59:22.571797  1753 raft_consensus.cc:1275] T f6028a708cb344fbb08ab8056921d9e7 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260709 12:59:22.572546  1605 raft_consensus.cc:1275] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260709 12:59:22.573298  3124 consensus_queue.cc:1048] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:59:22.574633  3024 consensus_queue.cc:1048] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:59:22.577720  3029 raft_consensus.cc:993] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701: : Instructing follower 816fadcea8fc4893882f5f9b9f22a03d to start an election
I20260709 12:59:22.578383  3029 raft_consensus.cc:1081] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 1 LEADER]: Signalling peer 816fadcea8fc4893882f5f9b9f22a03d to start an election
W20260709 12:59:22.571666  1644 consensus_peers.cc:597] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d -> Peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625): Couldn't send request to peer 3b879799899a499c8d409ee9f1277cd1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f6028a708cb344fbb08ab8056921d9e7. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:59:22.580521  1677 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3"
dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
 from {username='slave'} at 127.0.0.1:52778
I20260709 12:59:22.581118  1677 raft_consensus.cc:493] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:59:22.581485  1677 raft_consensus.cc:3060] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:22.590476  3214 raft_consensus.cc:993] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a: : Instructing follower 816fadcea8fc4893882f5f9b9f22a03d to start an election
I20260709 12:59:22.591058  3214 raft_consensus.cc:1081] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 LEADER]: Signalling peer 816fadcea8fc4893882f5f9b9f22a03d to start an election
I20260709 12:59:22.592523  1677 raft_consensus.cc:515] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:22.593542  1672 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62"
dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
 from {username='slave'} at 127.0.0.1:52766
I20260709 12:59:22.594681  1672 raft_consensus.cc:493] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:59:22.595183  1672 raft_consensus.cc:3060] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:22.595423  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:22.596091  1751 raft_consensus.cc:3055] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:59:22.596398  1751 raft_consensus.cc:740] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 12:59:22.596745  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 12:59:22.597347  2336 raft_consensus.cc:3060] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:22.597237  1751 consensus_queue.cc:260] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:22.598659  1751 raft_consensus.cc:3060] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:22.607317  1677 leader_election.cc:290] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 2 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:22.611593  2336 raft_consensus.cc:2468] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 2.
I20260709 12:59:22.629925  3127 raft_consensus.cc:993] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a: : Instructing follower af520b67945d443b901bbeca37cc5701 to start an election
I20260709 12:59:22.649410  3127 raft_consensus.cc:1081] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 LEADER]: Signalling peer af520b67945d443b901bbeca37cc5701 to start an election
I20260709 12:59:22.611963  1751 raft_consensus.cc:2468] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 2.
I20260709 12:59:22.613137  1644 leader_election.cc:304] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
W20260709 12:59:22.699967  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 86912.1us
I20260709 12:59:22.700078  3124 raft_consensus.cc:2804] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:59:22.700770  3124 raft_consensus.cc:697] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 12:59:22.701745  3124 consensus_queue.cc:237] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:22.706058  3024 raft_consensus.cc:993] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d: : Instructing follower 3b879799899a499c8d409ee9f1277cd1 to start an election
I20260709 12:59:22.707137  3024 raft_consensus.cc:1081] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 1 LEADER]: Signalling peer 3b879799899a499c8d409ee9f1277cd1 to start an election
I20260709 12:59:22.711203  2332 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec"
dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
 from {username='slave'} at 127.0.0.1:39726
I20260709 12:59:22.711833  2332 raft_consensus.cc:493] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:59:22.712236  2332 raft_consensus.cc:3060] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:22.725450  3218 raft_consensus.cc:2955] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } } }
I20260709 12:59:22.740682  2332 raft_consensus.cc:515] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:22.752688  1753 raft_consensus.cc:2955] T f6028a708cb344fbb08ab8056921d9e7 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } } }
I20260709 12:59:22.761214  2332 leader_election.cc:290] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 2 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:22.770773  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:22.764844  1745 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3"
dest_uuid: "af520b67945d443b901bbeca37cc5701"
 from {username='slave'} at 127.0.0.1:55840
I20260709 12:59:22.771449  1747 raft_consensus.cc:3060] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:22.771704  1745 raft_consensus.cc:493] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:59:22.773355  1745 raft_consensus.cc:3060] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:22.783151  1745 raft_consensus.cc:515] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:22.785117  1502 catalog_manager.cc:5697] T f6028a708cb344fbb08ab8056921d9e7 P af520b67945d443b901bbeca37cc5701 reported cstate change: config changed from index -1 to 3, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New cstate: current_term: 2 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } } }
I20260709 12:59:22.786159  1745 leader_election.cc:290] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 2 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:22.786791  1598 raft_consensus.cc:2955] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } } }
I20260709 12:59:22.796682  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:59:22.797279  1673 raft_consensus.cc:3055] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:59:22.797608  1673 raft_consensus.cc:740] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 12:59:22.798390  1673 consensus_queue.cc:260] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:22.799762  1673 raft_consensus.cc:3060] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:22.803175  1672 raft_consensus.cc:515] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 12:59:22.806665  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 51976.6us
I20260709 12:59:22.819725  1672 leader_election.cc:290] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 2 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:22.821530  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 12:59:22.822188  2332 raft_consensus.cc:3060] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:22.824193  1747 raft_consensus.cc:2468] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 2.
I20260709 12:59:22.825615  1673 raft_consensus.cc:2468] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 2.
I20260709 12:59:22.826440  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:22.829200  1745 raft_consensus.cc:3060] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:22.846649  2332 raft_consensus.cc:2468] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 2.
I20260709 12:59:22.847450  2336 raft_consensus.cc:1240] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 1. Current term is 2. Ops: []
I20260709 12:59:22.856132  1745 raft_consensus.cc:2468] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 2.
I20260709 12:59:22.857121  1646 leader_election.cc:304] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:59:22.866214  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:59:22.866808  1605 raft_consensus.cc:3055] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:59:22.867091  1605 raft_consensus.cc:740] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 12:59:22.867820  1605 consensus_queue.cc:260] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:22.868623  3219 raft_consensus.cc:2804] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:59:22.868927  1605 raft_consensus.cc:3060] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:22.869203  3219 raft_consensus.cc:697] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 12:59:22.871872  3219 consensus_queue.cc:237] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:22.875265  1502 catalog_manager.cc:5697] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d reported cstate change: term changed from 1 to 2, leader changed from af520b67945d443b901bbeca37cc5701 (127.0.107.195) to 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194). New cstate: current_term: 2 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:59:22.886052  1605 raft_consensus.cc:2468] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 2.
I20260709 12:59:22.886889  2299 leader_election.cc:304] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
W20260709 12:59:22.899456  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 58604.5us
W20260709 12:59:22.903584  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 56412.6us
I20260709 12:59:22.904084  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:59:22.904759  1599 raft_consensus.cc:3055] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:59:22.905036  1599 raft_consensus.cc:740] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 12:59:22.905853  1599 consensus_queue.cc:260] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:22.907241  1599 raft_consensus.cc:3060] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:22.911110  3222 raft_consensus.cc:2804] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:59:22.911651  3222 raft_consensus.cc:697] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 2 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 12:59:22.912155  1717 leader_election.cc:304] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:59:22.913056  3223 raft_consensus.cc:2804] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:59:22.913596  3223 raft_consensus.cc:697] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 2 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 12:59:22.914479  3223 consensus_queue.cc:237] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:22.918502  1599 raft_consensus.cc:2468] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 2.
I20260709 12:59:22.912562  3222 consensus_queue.cc:237] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:22.921999  1502 catalog_manager.cc:5697] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d reported cstate change: term changed from 1 to 2, leader changed from 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) to 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194). New cstate: current_term: 2 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } }
I20260709 12:59:22.985492  1500 catalog_manager.cc:5697] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 reported cstate change: term changed from 1 to 2, leader changed from 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) to af520b67945d443b901bbeca37cc5701 (127.0.107.195). New cstate: current_term: 2 leader_uuid: "af520b67945d443b901bbeca37cc5701" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:59:22.998075  3226 ts_tablet_manager.cc:933] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1: Initiating tablet copy from peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:23.002557  3229 ts_tablet_manager.cc:933] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1: Initiating tablet copy from peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:23.010982  1502 catalog_manager.cc:5697] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 reported cstate change: term changed from 1 to 2, leader changed from 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194) to 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196). New cstate: current_term: 2 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:59:23.031100  3229 tablet_copy_client.cc:323] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Beginning tablet copy session from remote peer at address 127.0.107.194:46261
I20260709 12:59:23.033443  1690 tablet_copy_service.cc:140] P 816fadcea8fc4893882f5f9b9f22a03d: Received BeginTabletCopySession request for tablet 83226095f79a46dab445849b1febd425 from peer 3b879799899a499c8d409ee9f1277cd1 ({username='slave'} at 127.0.0.1:57092)
I20260709 12:59:23.033962  3234 ts_tablet_manager.cc:933] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1: Initiating tablet copy from peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:23.034071  1690 tablet_copy_service.cc:161] P 816fadcea8fc4893882f5f9b9f22a03d: Beginning new tablet copy session on tablet 83226095f79a46dab445849b1febd425 from peer 3b879799899a499c8d409ee9f1277cd1 at {username='slave'} at 127.0.0.1:57092: session id = 3b879799899a499c8d409ee9f1277cd1-83226095f79a46dab445849b1febd425
I20260709 12:59:23.050375  3236 ts_tablet_manager.cc:933] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1: Initiating tablet copy from peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445)
I20260709 12:59:23.052363  1690 tablet_copy_source_session.cc:215] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:59:23.052716  3226 tablet_copy_client.cc:323] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Beginning tablet copy session from remote peer at address 127.0.107.195:40693
I20260709 12:59:23.054899  1764 tablet_copy_service.cc:140] P af520b67945d443b901bbeca37cc5701: Received BeginTabletCopySession request for tablet 6f6cc48def184bcc907c18c98277187f from peer 3b879799899a499c8d409ee9f1277cd1 ({username='slave'} at 127.0.0.1:43156)
I20260709 12:59:23.055665  1764 tablet_copy_service.cc:161] P af520b67945d443b901bbeca37cc5701: Beginning new tablet copy session on tablet 6f6cc48def184bcc907c18c98277187f from peer 3b879799899a499c8d409ee9f1277cd1 at {username='slave'} at 127.0.0.1:43156: session id = 3b879799899a499c8d409ee9f1277cd1-6f6cc48def184bcc907c18c98277187f
I20260709 12:59:23.059049  3236 tablet_copy_client.cc:323] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Beginning tablet copy session from remote peer at address 127.0.107.193:46445
I20260709 12:59:23.061072  1615 tablet_copy_service.cc:140] P 83595ad0fd5940c0a2bfc7c61acd969a: Received BeginTabletCopySession request for tablet 72f3cdc0abae4f74aad5e4a15cb02e1b from peer 3b879799899a499c8d409ee9f1277cd1 ({username='slave'} at 127.0.0.1:36462)
I20260709 12:59:23.061651  1615 tablet_copy_service.cc:161] P 83595ad0fd5940c0a2bfc7c61acd969a: Beginning new tablet copy session on tablet 72f3cdc0abae4f74aad5e4a15cb02e1b from peer 3b879799899a499c8d409ee9f1277cd1 at {username='slave'} at 127.0.0.1:36462: session id = 3b879799899a499c8d409ee9f1277cd1-72f3cdc0abae4f74aad5e4a15cb02e1b
I20260709 12:59:23.061997  3229 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 83226095f79a46dab445849b1febd425. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:59:23.067595  1764 tablet_copy_source_session.cc:215] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:59:23.070122  1615 tablet_copy_source_session.cc:215] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:59:23.070374  3234 tablet_copy_client.cc:323] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Beginning tablet copy session from remote peer at address 127.0.107.194:46261
I20260709 12:59:23.072117  1690 tablet_copy_service.cc:140] P 816fadcea8fc4893882f5f9b9f22a03d: Received BeginTabletCopySession request for tablet f6028a708cb344fbb08ab8056921d9e7 from peer 3b879799899a499c8d409ee9f1277cd1 ({username='slave'} at 127.0.0.1:57092)
I20260709 12:59:23.072616  1690 tablet_copy_service.cc:161] P 816fadcea8fc4893882f5f9b9f22a03d: Beginning new tablet copy session on tablet f6028a708cb344fbb08ab8056921d9e7 from peer 3b879799899a499c8d409ee9f1277cd1 at {username='slave'} at 127.0.0.1:57092: session id = 3b879799899a499c8d409ee9f1277cd1-f6028a708cb344fbb08ab8056921d9e7
I20260709 12:59:23.075958  3226 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f6cc48def184bcc907c18c98277187f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:59:23.092435  2332 raft_consensus.cc:1275] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:59:23.094281  3236 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 72f3cdc0abae4f74aad5e4a15cb02e1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:59:23.094106  3124 consensus_queue.cc:1048] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:59:23.092597  1690 tablet_copy_source_session.cc:215] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:59:23.102497  3234 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f6028a708cb344fbb08ab8056921d9e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:59:23.120177  3234 tablet_copy_client.cc:806] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Starting download of 0 data blocks...
I20260709 12:59:23.120795  3234 tablet_copy_client.cc:670] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Starting download of 1 WAL segments...
I20260709 12:59:23.127228  3229 tablet_copy_client.cc:806] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Starting download of 0 data blocks...
I20260709 12:59:23.142603  3229 tablet_copy_client.cc:670] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Starting download of 1 WAL segments...
I20260709 12:59:23.133236  3236 tablet_copy_client.cc:806] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Starting download of 0 data blocks...
I20260709 12:59:23.143639  3236 tablet_copy_client.cc:670] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Starting download of 1 WAL segments...
I20260709 12:59:23.131206  3234 tablet_copy_client.cc:538] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:59:23.150235  3236 tablet_copy_client.cc:538] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:59:23.153736  1745 raft_consensus.cc:1275] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:59:23.156153  3218 consensus_queue.cc:1048] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:59:23.163053  3229 tablet_copy_client.cc:538] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:59:23.184630  3226 tablet_copy_client.cc:806] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Starting download of 0 data blocks...
I20260709 12:59:23.185366  3226 tablet_copy_client.cc:670] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Starting download of 1 WAL segments...
I20260709 12:59:23.190299  3226 tablet_copy_client.cc:538] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:59:23.208655  3234 tablet_bootstrap.cc:492] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1: Bootstrap starting.
I20260709 12:59:23.220568  3229 tablet_bootstrap.cc:492] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1: Bootstrap starting.
I20260709 12:59:23.229110  3226 tablet_bootstrap.cc:492] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1: Bootstrap starting.
I20260709 12:59:23.236341  3236 tablet_bootstrap.cc:492] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1: Bootstrap starting.
I20260709 12:59:23.270239  1745 raft_consensus.cc:1275] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:59:23.271802  3218 consensus_queue.cc:1048] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:59:23.300112  2332 raft_consensus.cc:1275] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:59:23.308974  3158 consensus_queue.cc:1048] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:59:23.311489  1673 raft_consensus.cc:1275] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:59:23.313496  3221 consensus_queue.cc:1048] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:59:23.323012  1599 raft_consensus.cc:1275] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:59:23.379908  3124 consensus_queue.cc:1048] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:59:23.441672  1753 raft_consensus.cc:1275] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:59:23.477188  3229 tablet_bootstrap.cc:492] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:59:23.503757  1598 raft_consensus.cc:1275] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:59:23.497356  3229 tablet_bootstrap.cc:492] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1: Bootstrap complete.
W20260709 12:59:23.551071  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 45362.2us
I20260709 12:59:23.540633  3229 ts_tablet_manager.cc:1403] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1: Time spent bootstrapping tablet: real 0.320s	user 0.021s	sys 0.029s
W20260709 12:59:23.591478  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 58256us
W20260709 12:59:23.597455  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 50517.5us
I20260709 12:59:23.575621  3226 tablet_bootstrap.cc:492] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:59:23.576756   431 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:59:23.566756  3153 consensus_queue.cc:1048] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.142s
W20260709 12:59:23.634730  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 67356.4us
I20260709 12:59:23.593429  3223 consensus_queue.cc:1048] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.033s
I20260709 12:59:23.641461  3226 tablet_bootstrap.cc:492] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1: Bootstrap complete.
I20260709 12:59:23.642871  3226 ts_tablet_manager.cc:1403] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1: Time spent bootstrapping tablet: real 0.414s	user 0.084s	sys 0.090s
I20260709 12:59:23.655164  3229 raft_consensus.cc:359] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } }
I20260709 12:59:23.659446  3236 tablet_bootstrap.cc:492] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:59:23.683041  3236 tablet_bootstrap.cc:492] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1: Bootstrap complete.
I20260709 12:59:23.684437  3236 ts_tablet_manager.cc:1403] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1: Time spent bootstrapping tablet: real 0.448s	user 0.043s	sys 0.018s
I20260709 12:59:23.667743  3226 raft_consensus.cc:359] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } }
I20260709 12:59:23.686806  3226 raft_consensus.cc:740] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Initialized, Role: LEARNER
I20260709 12:59:23.714222  3226 consensus_queue.cc:260] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } }
I20260709 12:59:23.681011  3229 raft_consensus.cc:740] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Initialized, Role: LEARNER
W20260709 12:59:23.737041  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 135903us
I20260709 12:59:23.766508  3236 raft_consensus.cc:359] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } }
I20260709 12:59:23.773018  3236 raft_consensus.cc:740] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Initialized, Role: LEARNER
W20260709 12:59:23.830188  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 198564us
I20260709 12:59:23.870476  3236 consensus_queue.cc:260] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } }
I20260709 12:59:23.812788  3229 consensus_queue.cc:260] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } }
I20260709 12:59:23.929533  3234 tablet_bootstrap.cc:492] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:59:23.930537  3226 ts_tablet_manager.cc:1434] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1: Time spent starting tablet: real 0.287s	user 0.002s	sys 0.010s
I20260709 12:59:23.932843  3234 tablet_bootstrap.cc:492] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1: Bootstrap complete.
I20260709 12:59:23.933650  1764 tablet_copy_service.cc:342] P af520b67945d443b901bbeca37cc5701: Request end of tablet copy session 3b879799899a499c8d409ee9f1277cd1-6f6cc48def184bcc907c18c98277187f received from {username='slave'} at 127.0.0.1:43156
I20260709 12:59:23.933756  3234 ts_tablet_manager.cc:1403] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1: Time spent bootstrapping tablet: real 0.725s	user 0.021s	sys 0.025s
I20260709 12:59:23.936338  1764 tablet_copy_service.cc:434] P af520b67945d443b901bbeca37cc5701: ending tablet copy session 3b879799899a499c8d409ee9f1277cd1-6f6cc48def184bcc907c18c98277187f on tablet 6f6cc48def184bcc907c18c98277187f with peer 3b879799899a499c8d409ee9f1277cd1
I20260709 12:59:23.975742  3236 ts_tablet_manager.cc:1434] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1: Time spent starting tablet: real 0.290s	user 0.007s	sys 0.004s
W20260709 12:59:23.978497  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 348928us
I20260709 12:59:24.052376  3234 raft_consensus.cc:359] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } }
I20260709 12:59:24.053393  3234 raft_consensus.cc:740] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Initialized, Role: LEARNER
I20260709 12:59:24.056326  1615 tablet_copy_service.cc:342] P 83595ad0fd5940c0a2bfc7c61acd969a: Request end of tablet copy session 3b879799899a499c8d409ee9f1277cd1-72f3cdc0abae4f74aad5e4a15cb02e1b received from {username='slave'} at 127.0.0.1:36462
I20260709 12:59:24.056881  1615 tablet_copy_service.cc:434] P 83595ad0fd5940c0a2bfc7c61acd969a: ending tablet copy session 3b879799899a499c8d409ee9f1277cd1-72f3cdc0abae4f74aad5e4a15cb02e1b on tablet 72f3cdc0abae4f74aad5e4a15cb02e1b with peer 3b879799899a499c8d409ee9f1277cd1
I20260709 12:59:24.054132  3234 consensus_queue.cc:260] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } }
I20260709 12:59:24.092509  3234 ts_tablet_manager.cc:1434] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1: Time spent starting tablet: real 0.157s	user 0.006s	sys 0.001s
I20260709 12:59:24.128371  2336 raft_consensus.cc:1217] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:59:24.129684  1690 tablet_copy_service.cc:342] P 816fadcea8fc4893882f5f9b9f22a03d: Request end of tablet copy session 3b879799899a499c8d409ee9f1277cd1-f6028a708cb344fbb08ab8056921d9e7 received from {username='slave'} at 127.0.0.1:57092
I20260709 12:59:24.130275  1690 tablet_copy_service.cc:434] P 816fadcea8fc4893882f5f9b9f22a03d: ending tablet copy session 3b879799899a499c8d409ee9f1277cd1-f6028a708cb344fbb08ab8056921d9e7 on tablet f6028a708cb344fbb08ab8056921d9e7 with peer 3b879799899a499c8d409ee9f1277cd1
I20260709 12:59:24.135910  3229 ts_tablet_manager.cc:1434] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1: Time spent starting tablet: real 0.565s	user 0.015s	sys 0.000s
W20260709 12:59:24.136322  3229 ts_tablet_manager.cc:1467] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1: Tablet startup took 569ms
W20260709 12:59:24.136638  3229 ts_tablet_manager.cc:1469] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1: Trace:
0709 12:59:22.915852 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:22.916636 (+   784us) service_pool.cc:224] Handling call
0709 12:59:23.003035 (+ 86399us) ts_tablet_manager.cc:934] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1: Initiating tablet copy from peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
0709 12:59:23.060496 (+ 57461us) tablet_copy_client.cc:368] Tablet copy session begun
0709 12:59:23.106080 (+ 45584us) tablet_metadata.cc:665] Metadata flushed
0709 12:59:23.106208 (+   128us) tablet_copy_client.cc:497] Wrote new tablet metadata
0709 12:59:23.114210 (+  8002us) tablet_copy_client.cc:418] Wrote new consensus metadata
0709 12:59:23.117752 (+  3542us) tablet_replica.cc:206] Creating consensus instance
0709 12:59:23.118112 (+   360us) ts_tablet_manager.cc:989] Replica registered.
0709 12:59:23.118181 (+    69us) inbound_call.cc:177] Queueing success response
0709 12:59:23.220166 (+101985us) ts_tablet_manager.cc:1379] Bootstrapping tablet
0709 12:59:23.570586 (+350420us) ts_tablet_manager.cc:1435] Starting tablet replica
0709 12:59:23.634091 (+ 63505us) tablet_replica.cc:261] Starting instrumentation
0709 12:59:23.652622 (+ 18531us) tablet_replica.cc:280] Starting consensus
0709 12:59:24.134346 (+481724us) spinlock_profiling.cc:243] Waited 11.6 ms on lock 0x7b5400358e70. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f42630c6bec 00007f4269b5a623 00007f426b176e04 00007f426b17548d 00007f426b1a6f4e 00007f426b18bfdd 00007f426b18bf89 00007f426b18bf19 00007f426b18bee1 00007f426b18b10d
Metrics: {"dns_us":1127,"mutex_wait_us":34716,"raft.queue_time_us":33972,"raft.run_cpu_time_us":233,"raft.run_wall_time_us":27406,"spinlock_wait_cycles":26709760,"tablet-copy.queue_time_us":69728,"tablet-download-pool.queue_time_us":132,"tablet-download-pool.run_cpu_time_us":1095,"tablet-download-pool.run_wall_time_us":5383,"thread_start_us":82169,"threads_started":2}
I20260709 12:59:24.155299  2336 raft_consensus.cc:1217] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.2->[2.3-2.3]   Dedup: 2.3->[]
I20260709 12:59:24.225992  1690 tablet_copy_service.cc:342] P 816fadcea8fc4893882f5f9b9f22a03d: Request end of tablet copy session 3b879799899a499c8d409ee9f1277cd1-83226095f79a46dab445849b1febd425 received from {username='slave'} at 127.0.0.1:57092
I20260709 12:59:24.228808  1690 tablet_copy_service.cc:434] P 816fadcea8fc4893882f5f9b9f22a03d: ending tablet copy session 3b879799899a499c8d409ee9f1277cd1-83226095f79a46dab445849b1febd425 on tablet 83226095f79a46dab445849b1febd425 with peer 3b879799899a499c8d409ee9f1277cd1
I20260709 12:59:24.244163  2335 raft_consensus.cc:1217] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:59:24.133339   431 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:59:24.312311  2332 raft_consensus.cc:1217] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.2->[2.3-2.3]   Dedup: 2.3->[]
I20260709 12:59:24.451084  1674 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62"
dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
mode: GRACEFUL
new_leader_uuid: "af520b67945d443b901bbeca37cc5701"
 from {username='slave'} at 127.0.0.1:43266
I20260709 12:59:24.452029  1674 raft_consensus.cc:606] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 LEADER]: Received request to transfer leadership to TS af520b67945d443b901bbeca37cc5701
I20260709 12:59:24.461987  1677 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc"
dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
mode: GRACEFUL
new_leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
 from {username='slave'} at 127.0.0.1:43266
I20260709 12:59:24.462841  1677 raft_consensus.cc:606] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 1 LEADER]: Received request to transfer leadership to TS 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:24.466272  1745 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3"
dest_uuid: "af520b67945d443b901bbeca37cc5701"
mode: GRACEFUL
new_leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
 from {username='slave'} at 127.0.0.1:51270
I20260709 12:59:24.466936  1745 raft_consensus.cc:606] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 2 LEADER]: Received request to transfer leadership to TS 816fadcea8fc4893882f5f9b9f22a03d
I20260709 12:59:24.469640   431 auto_leader_rebalancer.cc:452] table: test-workload, leader rebalance finish, leader transfer count: 3
I20260709 12:59:24.471992   431 auto_leader_rebalancer.cc:527] All tables' leader rebalancing finished this round
I20260709 12:59:24.473783  3250 raft_consensus.cc:993] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d: : Instructing follower 83595ad0fd5940c0a2bfc7c61acd969a to start an election
I20260709 12:59:24.474495  3250 raft_consensus.cc:1081] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 1 LEADER]: Signalling peer 83595ad0fd5940c0a2bfc7c61acd969a to start an election
I20260709 12:59:24.481779  1605 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc"
dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
 from {username='slave'} at 127.0.0.1:40166
I20260709 12:59:24.482367  1605 raft_consensus.cc:493] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:59:24.483394  1605 raft_consensus.cc:3060] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:24.559800  1605 raft_consensus.cc:515] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:24.564344  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:59:24.565045  1677 raft_consensus.cc:3055] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:59:24.565311  1677 raft_consensus.cc:740] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 12:59:24.562330  1605 leader_election.cc:290] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 2 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:24.566079  1677 consensus_queue.cc:260] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:24.567273  1677 raft_consensus.cc:3060] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:24.569412  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 12:59:24.570098  2336 raft_consensus.cc:3060] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:24.576339  1677 raft_consensus.cc:2468] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 2.
I20260709 12:59:24.577859  1569 leader_election.cc:304] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 12:59:24.593431  2336 raft_consensus.cc:2468] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 2.
I20260709 12:59:24.596366  3275 raft_consensus.cc:2804] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:59:24.604516  3275 raft_consensus.cc:697] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 12:59:24.635041  3252 raft_consensus.cc:1064] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d: attempting to promote NON_VOTER 3b879799899a499c8d409ee9f1277cd1 to VOTER
I20260709 12:59:24.653609  3275 consensus_queue.cc:237] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:24.669184  3276 raft_consensus.cc:993] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d: : Instructing follower af520b67945d443b901bbeca37cc5701 to start an election
I20260709 12:59:24.669806  3276 raft_consensus.cc:1081] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 LEADER]: Signalling peer af520b67945d443b901bbeca37cc5701 to start an election
I20260709 12:59:24.672190  1745 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62"
dest_uuid: "af520b67945d443b901bbeca37cc5701"
 from {username='slave'} at 127.0.0.1:55848
I20260709 12:59:24.672811  1745 raft_consensus.cc:493] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:59:24.673247  1745 raft_consensus.cc:3060] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:59:24.694190  3268 raft_consensus.cc:1064] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a: attempting to promote NON_VOTER 3b879799899a499c8d409ee9f1277cd1 to VOTER
I20260709 12:59:24.698784  1745 raft_consensus.cc:515] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:24.708396  1745 leader_election.cc:290] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:24.712725  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:59:24.713451  1605 raft_consensus.cc:3060] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:59:24.734598  3276 raft_consensus.cc:1064] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d: attempting to promote NON_VOTER 3b879799899a499c8d409ee9f1277cd1 to VOTER
I20260709 12:59:24.738050  3276 consensus_queue.cc:237] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 1, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:24.710489  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:59:24.747022  1674 raft_consensus.cc:3055] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:59:24.747421  1674 raft_consensus.cc:740] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 12:59:24.754722  1605 raft_consensus.cc:2468] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 3.
I20260709 12:59:24.778417  1717 leader_election.cc:304] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:59:24.755944  1674 consensus_queue.cc:260] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:24.783002  1674 raft_consensus.cc:3060] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:59:24.788830  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 182454us
I20260709 12:59:24.800163  3287 raft_consensus.cc:2804] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:59:24.801448  3287 raft_consensus.cc:697] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 3 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 12:59:24.797798  3262 raft_consensus.cc:993] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701: : Instructing follower 816fadcea8fc4893882f5f9b9f22a03d to start an election
I20260709 12:59:24.802511  3287 consensus_queue.cc:237] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:24.807173  3252 consensus_queue.cc:237] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 1, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:24.818233  3255 raft_consensus.cc:1081] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 2 LEADER]: Signalling peer 816fadcea8fc4893882f5f9b9f22a03d to start an election
I20260709 12:59:24.838757  3268 consensus_queue.cc:237] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:24.856550  1674 raft_consensus.cc:2468] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 3.
W20260709 12:59:24.869782  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 76925.7us
I20260709 12:59:24.898299  1674 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3"
dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
 from {username='slave'} at 127.0.0.1:52778
I20260709 12:59:24.899027  1674 raft_consensus.cc:493] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:59:24.899452  1674 raft_consensus.cc:3060] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:59:24.903441  3288 raft_consensus.cc:1064] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701: attempting to promote NON_VOTER 3b879799899a499c8d409ee9f1277cd1 to VOTER
I20260709 12:59:24.908886  1674 raft_consensus.cc:515] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:24.918249  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:24.918921  1745 raft_consensus.cc:3055] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:59:24.919237  1745 raft_consensus.cc:740] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 12:59:24.921080  1745 consensus_queue.cc:260] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:24.922243  1745 raft_consensus.cc:3060] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:59:24.925443  1674 leader_election.cc:290] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 3 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:24.926786  2332 raft_consensus.cc:1275] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 2 LEARNER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260709 12:59:24.931612  3218 consensus_queue.cc:1048] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:59:24.936288  1745 raft_consensus.cc:2468] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 3.
I20260709 12:59:24.938449  1753 raft_consensus.cc:1275] T f6028a708cb344fbb08ab8056921d9e7 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260709 12:59:24.940910  1646 leader_election.cc:304] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
W20260709 12:59:24.957578  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 49570.9us
I20260709 12:59:24.958057  1753 raft_consensus.cc:1275] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:59:24.960551  1502 catalog_manager.cc:5697] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 1 to 2, leader changed from 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194) to 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193). New cstate: current_term: 2 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:59:24.971834  2335 raft_consensus.cc:1275] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 1 LEARNER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:59:24.947834  3250 consensus_queue.cc:1048] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.014s
I20260709 12:59:24.975593  1674 raft_consensus.cc:1275] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:59:24.968149  3218 raft_consensus.cc:2804] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:59:25.018743  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:59:25.045259  1601 raft_consensus.cc:3060] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:59:25.066354  3218 raft_consensus.cc:697] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 12:59:25.114728  3288 consensus_queue.cc:237] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:25.120015  3218 consensus_queue.cc:237] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:25.086701  3303 consensus_queue.cc:1048] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.099s
I20260709 12:59:25.190352  1601 raft_consensus.cc:2468] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 3.
I20260709 12:59:25.226804  1500 catalog_manager.cc:5697] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 reported cstate change: term changed from 2 to 3, leader changed from 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194) to af520b67945d443b901bbeca37cc5701 (127.0.107.195). New cstate: current_term: 3 leader_uuid: "af520b67945d443b901bbeca37cc5701" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: HEALTHY } } }
I20260709 12:59:25.250177  2335 raft_consensus.cc:1275] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 1 LEARNER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:59:25.252233  3223 consensus_queue.cc:1048] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:59:25.279101  1599 raft_consensus.cc:1275] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
W20260709 12:59:25.320210  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 348323us
I20260709 12:59:25.343811  3283 consensus_queue.cc:1048] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:59:25.349486  3295 consensus_queue.cc:1048] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20260709 12:59:25.354861  1672 raft_consensus.cc:1275] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:59:25.360428  1674 raft_consensus.cc:1275] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:59:25.365573  2332 raft_consensus.cc:1275] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 2 LEARNER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260709 12:59:25.380823  3300 consensus_queue.cc:1048] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:59:25.385452  1599 raft_consensus.cc:1275] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260709 12:59:25.388267  1598 raft_consensus.cc:1275] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:59:25.408933  1747 raft_consensus.cc:1275] T 83226095f79a46dab445849b1febd425 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260709 12:59:25.412324  1674 raft_consensus.cc:1275] T 6f6cc48def184bcc907c18c98277187f P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:59:25.419128  2336 raft_consensus.cc:1275] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:59:25.417644  3283 consensus_queue.cc:1048] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.011s
I20260709 12:59:25.416697  3288 consensus_queue.cc:1048] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.009s
I20260709 12:59:25.463610  3295 raft_consensus.cc:2955] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:25.420989  3296 consensus_queue.cc:1048] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.028s
I20260709 12:59:25.471668  3308 consensus_queue.cc:1048] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
W20260709 12:59:25.486673  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 43643.2us
W20260709 12:59:25.508744  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 53135.9us
I20260709 12:59:25.522694   431 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:59:25.529979  1605 raft_consensus.cc:1275] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260709 12:59:25.531630  3316 consensus_queue.cc:1048] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.005s
W20260709 12:59:25.535967  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 91992.1us
I20260709 12:59:25.541488  3252 consensus_queue.cc:1048] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:59:25.543865  3286 raft_consensus.cc:2955] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:25.561853  1598 raft_consensus.cc:2955] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:25.568570  1674 raft_consensus.cc:2955] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:25.588068  3308 raft_consensus.cc:2955] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:25.598165  1753 raft_consensus.cc:2955] T f6028a708cb344fbb08ab8056921d9e7 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:25.562783  3287 consensus_queue.cc:1048] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.021s
I20260709 12:59:25.588789  3273 consensus_queue.cc:1048] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.062s
I20260709 12:59:25.584831  3303 consensus_queue.cc:1048] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.036s
I20260709 12:59:25.716917  1747 raft_consensus.cc:2955] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:25.735124  2332 raft_consensus.cc:2955] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:25.749571  1751 raft_consensus.cc:1275] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:59:25.732647  1499 catalog_manager.cc:5697] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d reported cstate change: term changed from 2 to 3, leader changed from af520b67945d443b901bbeca37cc5701 (127.0.107.195) to 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194). New cstate: current_term: 3 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } }
I20260709 12:59:25.758939  3314 consensus_queue.cc:1048] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.004s
W20260709 12:59:25.764827  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 202932us
W20260709 12:59:25.911007  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 348842us
I20260709 12:59:25.918073  1601 raft_consensus.cc:1275] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
W20260709 12:59:25.979180  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 88691.9us
W20260709 12:59:26.061560  1605 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.502s	user 0.015s	sys 0.023s
W20260709 12:59:26.062178  1605 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.503s	user 0.015s	sys 0.023s
I20260709 12:59:26.166793  2333 raft_consensus.cc:2955] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:26.218250  1674 raft_consensus.cc:2955] T 6f6cc48def184bcc907c18c98277187f P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
W20260709 12:59:26.226117  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 299269us
I20260709 12:59:26.221172  1502 catalog_manager.cc:5697] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:59:26.244359  1502 catalog_manager.cc:5697] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: config changed from index 2 to 3, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "af520b67945d443b901bbeca37cc5701" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:26.271935  2332 raft_consensus.cc:2955] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
W20260709 12:59:26.285382  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 59900.5us
W20260709 12:59:26.347513  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 93263.4us
W20260709 12:59:26.401657  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 67644.9us
W20260709 12:59:26.401770  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 52976.3us
W20260709 12:59:26.457124  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 48742.9us
W20260709 12:59:26.429448  3283 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.533s	user 0.007s	sys 0.030s
I20260709 12:59:26.365418  3317 consensus_queue.cc:1048] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.091s
W20260709 12:59:26.503793  3283 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.608s	user 0.012s	sys 0.030s
W20260709 12:59:26.492851  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 63483.9us
W20260709 12:59:26.476385  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 172443us
W20260709 12:59:26.562502  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 54880.4us
W20260709 12:59:26.588414  3349 log.cc:927] Time spent T 6f6cc48def184bcc907c18c98277187f P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.259s	user 0.000s	sys 0.001s
W20260709 12:59:26.591178  3353 log.cc:927] Time spent T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.169s	user 0.001s	sys 0.000s
W20260709 12:59:26.588426  3336 log.cc:927] Time spent T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.079s	user 0.001s	sys 0.000s
W20260709 12:59:26.593513  3256 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.545s	user 0.075s	sys 0.086s
W20260709 12:59:26.593918  3256 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.589s	user 0.075s	sys 0.086s
W20260709 12:59:26.595160  3341 log.cc:927] Time spent T 83226095f79a46dab445849b1febd425 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.316s	user 0.000s	sys 0.001s
I20260709 12:59:26.596444  1752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55848 (request call id 1562) took 1030 ms. Trace:
I20260709 12:59:26.596875  1752 rpcz_store.cc:276] 0709 12:59:25.566087 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:25.566415 (+   328us) service_pool.cc:224] Handling call
0709 12:59:25.567004 (+   589us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:59:25.567356 (+   352us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:59:25.567380 (+    24us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:59:25.574829 (+  7449us) log.cc:844] Serialized 564 byte log entry
0709 12:59:26.070908 (+496079us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:59:26.070971 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:26.071078 (+   107us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:59:26.596198 (+525120us) raft_consensus.cc:1685] finished
0709 12:59:26.596223 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:59:26.596354 (+   131us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":495682,"threads_started":1,"wal-append.queue_time_us":640098}
W20260709 12:59:26.659942  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 94728.6us
W20260709 12:59:26.711890  1747 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.776s	user 0.000s	sys 0.046s
W20260709 12:59:26.712157  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 253823us
I20260709 12:59:26.723647  3296 raft_consensus.cc:2955] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
W20260709 12:59:26.712455  1747 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.777s	user 0.000s	sys 0.046s
I20260709 12:59:26.763859  1599 raft_consensus.cc:2955] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:26.775863  2332 raft_consensus.cc:2955] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:26.888073  1499 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35458 (request call id 292) took 1466 ms. Trace:
I20260709 12:59:26.888768  1499 rpcz_store.cc:276] 0709 12:59:25.421904 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:25.425815 (+  3911us) service_pool.cc:224] Handling call
0709 12:59:26.887913 (+1462098us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:59:26.285990 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:59:26.584921 (+298931us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:59:26.594629 (+  9708us) write_op.cc:435] Acquired schema lock
0709 12:59:26.594654 (+    25us) tablet.cc:662] Decoding operations
0709 12:59:26.595138 (+   484us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:59:26.595204 (+    66us) write_op.cc:641] Partition lock acquired for write op
0709 12:59:26.595249 (+    45us) tablet.cc:685] Acquiring locks for 1 operations
0709 12:59:26.595460 (+   211us) tablet.cc:701] Row locks acquired
0709 12:59:26.595497 (+    37us) write_op.cc:260] PREPARE: finished
0709 12:59:26.595663 (+   166us) write_op.cc:270] Start()
0709 12:59:26.595803 (+   140us) write_op.cc:276] Timestamp: P: 1783601966595618 usec, L: 0
0709 12:59:26.595826 (+    23us) op_driver.cc:348] REPLICATION: starting
0709 12:59:26.596679 (+   853us) log.cc:844] Serialized 541 byte log entry
0709 12:59:26.732548 (+135869us) spinlock_profiling.cc:243] Waited 1.87 ms on lock 0x7b540041d0a0. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cc090 00007f42660d96b8 00007f42660d92fa 00007f426615d965 00007f426615de09 00007f4269b98581 00007f4269b97d58 00007f4269b9c9f1 00007f4269b9c9a9 00007f4269b9c939 00007f4269b9c901
0709 12:59:26.738065 (+  5517us) op_driver.cc:464] REPLICATION: finished
0709 12:59:26.882946 (+144881us) write_op.cc:301] APPLY: starting
0709 12:59:26.883100 (+   154us) tablet.cc:1367] starting BulkCheckPresence
0709 12:59:26.883270 (+   170us) tablet.cc:1370] finished BulkCheckPresence
0709 12:59:26.883305 (+    35us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:59:26.883749 (+   444us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:59:26.884001 (+   252us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0709 12:59:26.884082 (+    81us) write_op.cc:312] APPLY: finished
0709 12:59:26.884380 (+   298us) log.cc:844] Serialized 23 byte log entry
0709 12:59:26.884674 (+   294us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:59:26.884794 (+   120us) write_op.cc:454] Released schema lock
0709 12:59:26.884856 (+    62us) write_op.cc:341] FINISH: Updating metrics
0709 12:59:26.885364 (+   508us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:59:26.885404 (+    40us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":1,"child_traces":[["op",{"apply.queue_time_us":144492,"apply.run_cpu_time_us":2414,"apply.run_wall_time_us":2420,"num_ops":1,"prepare.queue_time_us":438622,"prepare.run_cpu_time_us":8816,"prepare.run_wall_time_us":466344,"raft.queue_time_us":4496,"raft.run_cpu_time_us":12859,"raft.run_wall_time_us":120662,"replication_time_us":142057,"spinlock_wait_cycles":4289920,"thread_start_us":365310,"threads_started":4,"wal-append.queue_time_us":119467}]]}
W20260709 12:59:26.975318  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 99303.8us
I20260709 12:59:26.975363  1754 raft_consensus.cc:2955] T 83226095f79a46dab445849b1febd425 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:27.025661  1599 raft_consensus.cc:2955] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:27.118315   431 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:59:27.159125  1502 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35444 (request call id 247) took 1418 ms. Trace:
I20260709 12:59:27.154729  1500 catalog_manager.cc:5697] T f6028a708cb344fbb08ab8056921d9e7 P af520b67945d443b901bbeca37cc5701 reported cstate change: config changed from index 3 to 4, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:27.167079  1599 consensus_queue.cc:237] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:27.159683  1502 rpcz_store.cc:276] 0709 12:59:25.740838 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:25.741655 (+   817us) service_pool.cc:224] Handling call
0709 12:59:25.980748 (+239093us) spinlock_profiling.cc:243] Waited 1.46 ms on lock 0x7b5400002490. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f426303db71 00007f426303e0b2 00007f426bfb7a89 00007f426bf4fda6 00007f426bf21088 00007f426c01f37e 00007f426a4e02a4 00007f426a4e0232 00007f426a4e0161 00007f426a4e00dc
0709 12:59:27.159036 (+1178288us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:59:26.757859 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:59:26.758539 (+   680us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:59:26.758584 (+    45us) write_op.cc:435] Acquired schema lock
0709 12:59:26.758601 (+    17us) tablet.cc:662] Decoding operations
0709 12:59:26.759123 (+   522us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:59:26.759170 (+    47us) write_op.cc:641] Partition lock acquired for write op
0709 12:59:26.759199 (+    29us) tablet.cc:685] Acquiring locks for 2 operations
0709 12:59:26.763242 (+  4043us) tablet.cc:701] Row locks acquired
0709 12:59:26.767284 (+  4042us) write_op.cc:260] PREPARE: finished
0709 12:59:26.795727 (+ 28443us) write_op.cc:270] Start()
0709 12:59:26.829065 (+ 33338us) write_op.cc:276] Timestamp: P: 1783601966791288 usec, L: 0
0709 12:59:26.830077 (+  1012us) op_driver.cc:348] REPLICATION: starting
0709 12:59:26.889966 (+ 59889us) log.cc:844] Serialized 1045 byte log entry
0709 12:59:26.891732 (+  1766us) spinlock_profiling.cc:243] Waited 31 ms on lock 0x7b540041cb54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f426615de28 00007f4269b98581 00007f4269b97d58 00007f4269b9c9f1 00007f4269b9c9a9 00007f4269b9c939 00007f4269b9c901 00007f4269b9bb2d 000000000038f354 000000000038f199
0709 12:59:26.937184 (+ 45452us) op_driver.cc:464] REPLICATION: finished
0709 12:59:26.974686 (+ 37502us) write_op.cc:301] APPLY: starting
0709 12:59:26.980496 (+  5810us) tablet.cc:1367] starting BulkCheckPresence
0709 12:59:26.994874 (+ 14378us) tablet.cc:1370] finished BulkCheckPresence
0709 12:59:26.997270 (+  2396us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:59:27.005389 (+  8119us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:59:27.013126 (+  7737us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0709 12:59:27.013200 (+    74us) write_op.cc:312] APPLY: finished
0709 12:59:27.027194 (+ 13994us) log.cc:844] Serialized 29 byte log entry
0709 12:59:27.072926 (+ 45732us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:59:27.075252 (+  2326us) write_op.cc:454] Released schema lock
0709 12:59:27.075321 (+    69us) write_op.cc:341] FINISH: Updating metrics
0709 12:59:27.075761 (+   440us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:59:27.075803 (+    42us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":3,"spinlock_wait_cycles":3361024,"child_traces":[["op",{"apply.queue_time_us":33834,"apply.run_cpu_time_us":2602,"apply.run_wall_time_us":101022,"num_ops":2,"prepare.queue_time_us":498223,"prepare.run_cpu_time_us":5678,"prepare.run_wall_time_us":134089,"replication_time_us":106912,"spinlock_wait_cycles":71331200,"thread_start_us":73950,"threads_started":1}]]}
W20260709 12:59:27.197695  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 45332.1us
I20260709 12:59:27.285597  2332 raft_consensus.cc:1275] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:59:27.287530  3268 consensus_queue.cc:1048] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:59:27.289652  1499 catalog_manager.cc:5697] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 reported cstate change: config changed from index 3 to 4, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:27.328078  1747 consensus_queue.cc:237] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:27.412271  1748 raft_consensus.cc:1275] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:59:27.475592  3162 consensus_queue.cc:1048] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:59:27.517699  1673 consensus_queue.cc:237] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 12:59:27.528416  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 62550.4us
I20260709 12:59:27.538618  2332 raft_consensus.cc:1275] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:59:27.538702  1599 raft_consensus.cc:1275] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:59:27.560331  3283 raft_consensus.cc:2955] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:27.567803  3256 consensus_queue.cc:1048] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:59:27.568534  2333 raft_consensus.cc:2955] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:27.573107  3358 consensus_queue.cc:1048] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:59:27.577316  2332 raft_consensus.cc:1275] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:59:27.580684  1677 consensus_queue.cc:237] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:27.582231  3361 consensus_queue.cc:1048] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.012s
I20260709 12:59:27.642601  1604 raft_consensus.cc:1275] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:59:27.647876  1747 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62"
dest_uuid: "af520b67945d443b901bbeca37cc5701"
mode: GRACEFUL
new_leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
 from {username='slave'} at 127.0.0.1:51270
I20260709 12:59:27.648610  1747 raft_consensus.cc:606] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 3 LEADER]: Received request to transfer leadership to TS 816fadcea8fc4893882f5f9b9f22a03d
I20260709 12:59:27.648901  1500 catalog_manager.cc:5697] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 reported cstate change: config changed from index 3 to 4, VOTER 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194) evicted. New cstate: current_term: 1 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:27.658938  1676 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "b9bb5648969b4299a776843d72043448"
dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
mode: GRACEFUL
new_leader_uuid: "af520b67945d443b901bbeca37cc5701"
 from {username='slave'} at 127.0.0.1:43266
I20260709 12:59:27.659699  1676 raft_consensus.cc:606] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 LEADER]: Received request to transfer leadership to TS af520b67945d443b901bbeca37cc5701
I20260709 12:59:27.666778  1604 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78"
dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
mode: GRACEFUL
new_leader_uuid: "3b879799899a499c8d409ee9f1277cd1"
 from {username='slave'} at 127.0.0.1:47752
I20260709 12:59:27.667526  1604 raft_consensus.cc:606] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 LEADER]: Received request to transfer leadership to TS 3b879799899a499c8d409ee9f1277cd1
I20260709 12:59:27.670171   431 auto_leader_rebalancer.cc:452] table: test-workload, leader rebalance finish, leader transfer count: 3
I20260709 12:59:27.649789  3370 consensus_queue.cc:1048] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.025s
I20260709 12:59:27.695930   431 auto_leader_rebalancer.cc:527] All tables' leader rebalancing finished this round
I20260709 12:59:27.705986  2333 raft_consensus.cc:1275] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:59:27.724444  1603 raft_consensus.cc:1275] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:59:27.724159  3256 raft_consensus.cc:2955] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:27.732036  1599 raft_consensus.cc:2955] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:27.737689  1490 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 6f6cc48def184bcc907c18c98277187f with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:59:27.742354  2333 raft_consensus.cc:2955] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:27.741791  1753 raft_consensus.cc:2955] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:27.745294  1665 tablet_service.cc:1558] Processing DeleteTablet for tablet 72f3cdc0abae4f74aad5e4a15cb02e1b with delete_type TABLET_DATA_TOMBSTONED (TS 816fadcea8fc4893882f5f9b9f22a03d not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:52760
I20260709 12:59:27.737689  3307 consensus_queue.cc:1048] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.015s
I20260709 12:59:27.761354  3370 consensus_queue.cc:1048] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:59:27.815260  1487 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 72f3cdc0abae4f74aad5e4a15cb02e1b with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:59:27.836768  3314 raft_consensus.cc:2955] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER af520b67945d443b901bbeca37cc5701 (127.0.107.195) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:27.839560  1500 catalog_manager.cc:5697] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: config changed from index 3 to 4, VOTER 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194) evicted. New cstate: current_term: 1 leader_uuid: "af520b67945d443b901bbeca37cc5701" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:27.852849  3248 raft_consensus.cc:993] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d: : Instructing follower af520b67945d443b901bbeca37cc5701 to start an election
I20260709 12:59:27.853583  3248 raft_consensus.cc:1081] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 LEADER]: Signalling peer af520b67945d443b901bbeca37cc5701 to start an election
I20260709 12:59:27.856570  1752 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "b9bb5648969b4299a776843d72043448"
dest_uuid: "af520b67945d443b901bbeca37cc5701"
 from {username='slave'} at 127.0.0.1:55848
I20260709 12:59:27.857323  1752 raft_consensus.cc:493] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:59:27.857775  1752 raft_consensus.cc:3060] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:27.841964  1604 raft_consensus.cc:2955] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER af520b67945d443b901bbeca37cc5701 (127.0.107.195) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:27.865454  3390 tablet_replica.cc:333] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 816fadcea8fc4893882f5f9b9f22a03d: stopping tablet replica
I20260709 12:59:27.890699  1488 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 83226095f79a46dab445849b1febd425 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:59:27.896494  2332 raft_consensus.cc:2955] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER af520b67945d443b901bbeca37cc5701 (127.0.107.195) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:27.898861  1501 catalog_manager.cc:5697] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d reported cstate change: config changed from index 4 to 5, VOTER af520b67945d443b901bbeca37cc5701 (127.0.107.195) evicted. New cstate: current_term: 2 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:59:27.912909  1665 tablet_service.cc:1558] Processing DeleteTablet for tablet 6f6cc48def184bcc907c18c98277187f with delete_type TABLET_DATA_TOMBSTONED (TS 816fadcea8fc4893882f5f9b9f22a03d not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:52760
I20260709 12:59:27.925763  1740 tablet_service.cc:1558] Processing DeleteTablet for tablet 83226095f79a46dab445849b1febd425 with delete_type TABLET_DATA_TOMBSTONED (TS af520b67945d443b901bbeca37cc5701 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:55838
I20260709 12:59:27.953751  3297 raft_consensus.cc:993] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701: : Instructing follower 816fadcea8fc4893882f5f9b9f22a03d to start an election
I20260709 12:59:27.952539  3248 raft_consensus.cc:2955] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER af520b67945d443b901bbeca37cc5701 (127.0.107.195) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:27.954370  3297 raft_consensus.cc:1081] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 3 LEADER]: Signalling peer 816fadcea8fc4893882f5f9b9f22a03d to start an election
I20260709 12:59:27.954821  2333 raft_consensus.cc:2955] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER af520b67945d443b901bbeca37cc5701 (127.0.107.195) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:27.957175  3392 tablet_replica.cc:333] T 83226095f79a46dab445849b1febd425 P af520b67945d443b901bbeca37cc5701: stopping tablet replica
I20260709 12:59:27.961663  1676 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62"
dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
 from {username='slave'} at 127.0.0.1:52778
I20260709 12:59:27.959870  3392 raft_consensus.cc:2243] T 83226095f79a46dab445849b1febd425 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:59:27.962283  1676 raft_consensus.cc:493] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:59:27.962715  1676 raft_consensus.cc:3060] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:59:27.963004  3392 raft_consensus.cc:2272] T 83226095f79a46dab445849b1febd425 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:59:27.967499  3390 raft_consensus.cc:2243] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:59:27.968462  3390 raft_consensus.cc:2272] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:59:27.971753  3390 ts_tablet_manager.cc:1916] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 816fadcea8fc4893882f5f9b9f22a03d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:59:27.982697  1752 raft_consensus.cc:515] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:27.983387  1488 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet f6028a708cb344fbb08ab8056921d9e7 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:59:27.992414  1676 raft_consensus.cc:515] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:27.994115  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:59:27.995465  1599 raft_consensus.cc:3060] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:27.996712  1502 catalog_manager.cc:5697] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 reported cstate change: config changed from index 4 to 5, VOTER af520b67945d443b901bbeca37cc5701 (127.0.107.195) evicted. New cstate: current_term: 2 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:27.999348  1752 leader_election.cc:290] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 2 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:28.002823  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:59:28.003509  1677 raft_consensus.cc:3055] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:59:28.003906  1677 raft_consensus.cc:740] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 12:59:28.008368  1677 consensus_queue.cc:260] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:28.013262  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:59:28.014047  1604 raft_consensus.cc:3060] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:59:28.019958  1599 raft_consensus.cc:2468] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 2.
I20260709 12:59:28.023474  3392 ts_tablet_manager.cc:1916] T 83226095f79a46dab445849b1febd425 P af520b67945d443b901bbeca37cc5701: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:59:28.039825  1717 leader_election.cc:304] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:59:28.040966  1677 raft_consensus.cc:3060] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:28.042349  1676 leader_election.cc:290] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 4 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:28.028343  1604 raft_consensus.cc:2468] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 4.
I20260709 12:59:28.044044  1643 leader_election.cc:304] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 12:59:28.045893  3361 raft_consensus.cc:2804] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:59:28.046420  3361 raft_consensus.cc:697] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 12:59:28.047369  3361 consensus_queue.cc:237] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:28.050482  3297 raft_consensus.cc:2804] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:59:28.051076  3297 raft_consensus.cc:697] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 2 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 12:59:28.051985  3297 consensus_queue.cc:237] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:28.076809  1740 tablet_service.cc:1558] Processing DeleteTablet for tablet f6028a708cb344fbb08ab8056921d9e7 with delete_type TABLET_DATA_TOMBSTONED (TS af520b67945d443b901bbeca37cc5701 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:55838
I20260709 12:59:28.074617  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:28.078791  1750 raft_consensus.cc:3055] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:59:28.079166  1750 raft_consensus.cc:740] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 12:59:28.081326  1677 raft_consensus.cc:2468] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 2.
I20260709 12:59:28.079941  1750 consensus_queue.cc:260] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:28.085642  1750 raft_consensus.cc:3060] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:59:28.109009  1604 raft_consensus.cc:2955] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER af520b67945d443b901bbeca37cc5701 (127.0.107.195) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:28.141537  3347 raft_consensus.cc:993] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a: : Instructing follower 3b879799899a499c8d409ee9f1277cd1 to start an election
I20260709 12:59:28.142158  3347 raft_consensus.cc:1081] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 LEADER]: Signalling peer 3b879799899a499c8d409ee9f1277cd1 to start an election
I20260709 12:59:28.143241  3392 ts_tablet_manager.cc:1929] T 83226095f79a46dab445849b1febd425 P af520b67945d443b901bbeca37cc5701: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20260709 12:59:28.143793  3392 log.cc:1199] T 83226095f79a46dab445849b1febd425 P af520b67945d443b901bbeca37cc5701: Deleting WAL directory at /tmp/dist-test-taskakPCWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783601878574137-431-0/minicluster-data/ts-2-root/wals/83226095f79a46dab445849b1febd425
I20260709 12:59:28.142891  1502 catalog_manager.cc:5697] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 reported cstate change: term changed from 1 to 2, leader changed from 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194) to af520b67945d443b901bbeca37cc5701 (127.0.107.195). New cstate: current_term: 2 leader_uuid: "af520b67945d443b901bbeca37cc5701" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: HEALTHY } } }
I20260709 12:59:28.147534  2332 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78"
dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
 from {username='slave'} at 127.0.0.1:49082
I20260709 12:59:28.148214  2332 raft_consensus.cc:493] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:59:28.148620  2332 raft_consensus.cc:3060] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:59:28.155010  1501 catalog_manager.cc:5697] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d reported cstate change: term changed from 3 to 4, leader changed from af520b67945d443b901bbeca37cc5701 (127.0.107.195) to 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194). New cstate: current_term: 4 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } }
I20260709 12:59:28.165004  1490 catalog_manager.cc:5028] TS af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): tablet 83226095f79a46dab445849b1febd425 (table test-workload [id=75d77b57b3f34786b9c215c4f11bd727]) successfully deleted
I20260709 12:59:28.169198  1750 raft_consensus.cc:2468] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 4.
I20260709 12:59:28.167389  3392 tablet_replica.cc:333] T f6028a708cb344fbb08ab8056921d9e7 P af520b67945d443b901bbeca37cc5701: stopping tablet replica
I20260709 12:59:28.170748  2332 raft_consensus.cc:515] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:28.175316  3390 ts_tablet_manager.cc:1929] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 816fadcea8fc4893882f5f9b9f22a03d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:59:28.175853  3390 log.cc:1199] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 816fadcea8fc4893882f5f9b9f22a03d: Deleting WAL directory at /tmp/dist-test-taskakPCWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783601878574137-431-0/minicluster-data/ts-1-root/wals/72f3cdc0abae4f74aad5e4a15cb02e1b
I20260709 12:59:28.177791  1488 catalog_manager.cc:5028] TS 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): tablet 72f3cdc0abae4f74aad5e4a15cb02e1b (table test-workload [id=75d77b57b3f34786b9c215c4f11bd727]) successfully deleted
I20260709 12:59:28.178156  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: true dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:59:28.179003  1604 raft_consensus.cc:3055] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:59:28.179451  1604 raft_consensus.cc:740] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 12:59:28.178161  3392 raft_consensus.cc:2243] T f6028a708cb344fbb08ab8056921d9e7 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:59:28.180315  1604 consensus_queue.cc:260] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:28.180999  3392 raft_consensus.cc:2272] T f6028a708cb344fbb08ab8056921d9e7 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:59:28.181576  1604 raft_consensus.cc:3060] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:59:28.187292  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: true dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:28.187979  1750 raft_consensus.cc:3060] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:59:28.188467  2332 leader_election.cc:290] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 4 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:28.189442  3390 tablet_replica.cc:333] T 6f6cc48def184bcc907c18c98277187f P 816fadcea8fc4893882f5f9b9f22a03d: stopping tablet replica
I20260709 12:59:28.192644  3390 raft_consensus.cc:2243] T 6f6cc48def184bcc907c18c98277187f P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:59:28.193533  3390 raft_consensus.cc:2272] T 6f6cc48def184bcc907c18c98277187f P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:59:28.194370  1604 raft_consensus.cc:2468] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 4.
I20260709 12:59:28.191491  3392 ts_tablet_manager.cc:1916] T f6028a708cb344fbb08ab8056921d9e7 P af520b67945d443b901bbeca37cc5701: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:59:28.196009  2298 leader_election.cc:304] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 12:59:28.196465  1750 raft_consensus.cc:2468] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 4.
I20260709 12:59:28.197031  3207 raft_consensus.cc:2804] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:59:28.197607  3207 raft_consensus.cc:697] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 4 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 12:59:28.198652  3207 consensus_queue.cc:237] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:28.199652  3390 ts_tablet_manager.cc:1916] T 6f6cc48def184bcc907c18c98277187f P 816fadcea8fc4893882f5f9b9f22a03d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:59:28.217447  3392 ts_tablet_manager.cc:1929] T f6028a708cb344fbb08ab8056921d9e7 P af520b67945d443b901bbeca37cc5701: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20260709 12:59:28.219007  3392 log.cc:1199] T f6028a708cb344fbb08ab8056921d9e7 P af520b67945d443b901bbeca37cc5701: Deleting WAL directory at /tmp/dist-test-taskakPCWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783601878574137-431-0/minicluster-data/ts-2-root/wals/f6028a708cb344fbb08ab8056921d9e7
I20260709 12:59:28.221036  3390 ts_tablet_manager.cc:1929] T 6f6cc48def184bcc907c18c98277187f P 816fadcea8fc4893882f5f9b9f22a03d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:59:28.221439  1490 catalog_manager.cc:5028] TS af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): tablet f6028a708cb344fbb08ab8056921d9e7 (table test-workload [id=75d77b57b3f34786b9c215c4f11bd727]) successfully deleted
I20260709 12:59:28.221890  3390 log.cc:1199] T 6f6cc48def184bcc907c18c98277187f P 816fadcea8fc4893882f5f9b9f22a03d: Deleting WAL directory at /tmp/dist-test-taskakPCWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783601878574137-431-0/minicluster-data/ts-1-root/wals/6f6cc48def184bcc907c18c98277187f
I20260709 12:59:28.225293  1488 catalog_manager.cc:5028] TS 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): tablet 6f6cc48def184bcc907c18c98277187f (table test-workload [id=75d77b57b3f34786b9c215c4f11bd727]) successfully deleted
I20260709 12:59:28.245673  1501 catalog_manager.cc:5697] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 reported cstate change: term changed from 3 to 4, leader changed from 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) to 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196). New cstate: current_term: 4 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:59:28.716712   431 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
W20260709 12:59:29.375434  1534 debug-util.cc:398] Leaking SignalData structure 0x7b0800182f20 after lost signal to thread 434
W20260709 12:59:29.380851  1534 debug-util.cc:398] Leaking SignalData structure 0x7b08001b8460 after lost signal to thread 1537
W20260709 12:59:29.416473  1571 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.619s	user 0.000s	sys 0.024s
W20260709 12:59:29.417214  1571 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.620s	user 0.000s	sys 0.024s
W20260709 12:59:29.426661  1570 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.612s	user 0.005s	sys 0.007s
W20260709 12:59:29.427273  1570 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.612s	user 0.006s	sys 0.007s
I20260709 12:59:29.446599  3398 raft_consensus.cc:493] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 12:59:29.447976  3398 raft_consensus.cc:515] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:29.452195  3295 raft_consensus.cc:493] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 12:59:29.455031  3295 raft_consensus.cc:515] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:29.458968  3295 leader_election.cc:290] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:29.456653  3398 leader_election.cc:290] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:29.480619  3355 raft_consensus.cc:493] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 12:59:29.481338  3355 raft_consensus.cc:515] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:29.485451  3355 leader_election.cc:290] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:29.491880  3355 raft_consensus.cc:493] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 12:59:29.492425  3355 raft_consensus.cc:515] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:29.494884  3355 leader_election.cc:290] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 12:59:29.511179  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 1.16003e+06us
I20260709 12:59:29.524691  3329 raft_consensus.cc:493] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 12:59:29.525439  3329 raft_consensus.cc:515] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:29.513115  3347 raft_consensus.cc:493] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 12:59:29.534680  3268 raft_consensus.cc:493] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 12:59:29.512552  3397 raft_consensus.cc:493] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 12:59:29.611521  3162 raft_consensus.cc:493] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 12:59:29.612200  3162 raft_consensus.cc:515] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:29.614902  3162 leader_election.cc:290] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:29.615607  3162 raft_consensus.cc:493] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 12:59:29.616217  3162 raft_consensus.cc:515] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:29.609876  3408 raft_consensus.cc:493] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 12:59:29.610759  3295 raft_consensus.cc:493] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 12:59:29.534816  3347 raft_consensus.cc:515] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 12:59:29.629158  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 1.27539e+06us
I20260709 12:59:29.630872  3162 leader_election.cc:290] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:29.535334  3268 raft_consensus.cc:515] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:29.656448  3162 raft_consensus.cc:493] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 12:59:29.661000  3412 raft_consensus.cc:493] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 12:59:29.661862  3412 raft_consensus.cc:515] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:29.665196  3412 leader_election.cc:290] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:29.670640  3398 raft_consensus.cc:493] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 12:59:29.671020  3214 raft_consensus.cc:493] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 12:59:29.671659  3214 raft_consensus.cc:515] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:29.671465  3398 raft_consensus.cc:515] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:29.674026  3214 leader_election.cc:290] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:29.674625  3214 raft_consensus.cc:493] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 12:59:29.675135  3214 raft_consensus.cc:515] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:29.675799  3398 leader_election.cc:290] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:29.670323  3404 raft_consensus.cc:493] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 12:59:29.657375  3162 raft_consensus.cc:515] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:29.680116  3406 raft_consensus.cc:493] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 12:59:29.691335  3162 leader_election.cc:290] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:29.541641  3397 raft_consensus.cc:515] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:29.696988  3400 raft_consensus.cc:493] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 12:59:29.633546  3402 raft_consensus.cc:493] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 12:59:29.651121  3370 raft_consensus.cc:493] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 12:59:29.697777  3400 raft_consensus.cc:515] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } }
I20260709 12:59:29.707114  3162 raft_consensus.cc:493] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 12:59:29.709882  3162 raft_consensus.cc:515] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:29.635948  3295 raft_consensus.cc:515] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:29.631098  3408 raft_consensus.cc:515] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:29.709403  3402 raft_consensus.cc:515] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:29.719591  3402 leader_election.cc:290] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:29.720041  3268 leader_election.cc:290] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:29.713899  3407 raft_consensus.cc:493] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 12:59:29.721560  3407 raft_consensus.cc:515] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:29.705371  3329 leader_election.cc:290] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:29.710485  3405 raft_consensus.cc:493] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 12:59:29.691220  3404 raft_consensus.cc:515] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:29.703949  3406 raft_consensus.cc:515] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:29.795738  3268 raft_consensus.cc:493] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 12:59:29.712745  3370 raft_consensus.cc:515] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:29.805613  3268 raft_consensus.cc:515] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:29.808123  3268 leader_election.cc:290] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:29.812929  3417 raft_consensus.cc:493] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 12:59:29.815274  3408 leader_election.cc:290] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:29.816016  3408 raft_consensus.cc:493] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 12:59:29.816623  3408 raft_consensus.cc:515] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:29.819145  3410 raft_consensus.cc:493] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 12:59:29.819931  3400 leader_election.cc:290] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:29.813074  3397 leader_election.cc:290] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:29.789577  3411 raft_consensus.cc:493] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 12:59:29.792548  3329 raft_consensus.cc:493] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 12:59:29.845234  3162 leader_election.cc:290] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:29.802878  3398 raft_consensus.cc:493] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 12:59:29.821401  3374 raft_consensus.cc:493] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 12:59:29.890736  3162 raft_consensus.cc:493] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:29.819851  3410 raft_consensus.cc:515] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 12:59:29.860848  1534 debug-util.cc:398] Leaking SignalData structure 0x7b080037e180 after lost signal to thread 1629
I20260709 12:59:29.891729  3162 raft_consensus.cc:515] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:29.854151  3268 raft_consensus.cc:493] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 12:59:29.854490  3307 raft_consensus.cc:493] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 12:59:29.813815  3417 raft_consensus.cc:515] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:29.926425  3162 leader_election.cc:290] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:29.883870  3397 raft_consensus.cc:493] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 12:59:29.895218  3214 leader_election.cc:290] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:29.935251  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:29.935922  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:29.935940  1677 raft_consensus.cc:2468] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 2.
I20260709 12:59:29.863036  3329 raft_consensus.cc:515] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:29.938932  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:29.943001  3415 raft_consensus.cc:493] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 12:59:29.944088  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:29.944653  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:29.919538  3248 raft_consensus.cc:493] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 12:59:29.945116  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:29.947584  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:29.802879  3405 raft_consensus.cc:515] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:29.918983  3402 raft_consensus.cc:493] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 12:59:29.952013  3428 raft_consensus.cc:493] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 12:59:29.952637  3423 raft_consensus.cc:493] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 12:59:29.950623  1671 raft_consensus.cc:2468] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 2.
I20260709 12:59:29.955956  3162 raft_consensus.cc:493] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 12:59:29.960204  3162 raft_consensus.cc:515] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:29.962296  3162 leader_election.cc:290] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:29.854064  3411 raft_consensus.cc:515] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:29.930557  3307 raft_consensus.cc:515] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:29.963089  1750 consensus_queue.cc:237] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } }
W20260709 12:59:29.955797  1534 debug-util.cc:398] Leaking SignalData structure 0x7b080037d960 after lost signal to thread 1704
I20260709 12:59:29.967592  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:29.968149  3162 raft_consensus.cc:493] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 12:59:29.918898  3398 raft_consensus.cc:515] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:29.945140  3421 raft_consensus.cc:493] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 12:59:29.950989  3407 leader_election.cc:290] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:29.940578  3397 raft_consensus.cc:515] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:29.933188  3268 raft_consensus.cc:515] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:29.982540  3268 leader_election.cc:290] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:29.982612  3397 leader_election.cc:290] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:29.975706  3162 raft_consensus.cc:515] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:29.985255  3162 leader_election.cc:290] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:29.954537  3404 leader_election.cc:290] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:29.975351  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:29.986296  1678 raft_consensus.cc:2468] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 2.
I20260709 12:59:29.965451  3214 raft_consensus.cc:493] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 12:59:29.987082  3357 raft_consensus.cc:493] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 12:59:29.987335  3426 raft_consensus.cc:493] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 12:59:29.964818  3411 leader_election.cc:290] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:29.988410  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:29.975813  3423 raft_consensus.cc:515] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:29.966182  3297 raft_consensus.cc:493] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 12:59:29.897538  3374 raft_consensus.cc:515] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:30.027029  3297 raft_consensus.cc:515] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:30.027629  3419 raft_consensus.cc:493] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 12:59:30.028069  3370 leader_election.cc:290] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:30.029369  3297 leader_election.cc:290] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:30.026320  3329 leader_election.cc:290] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:29.956921  3248 raft_consensus.cc:515] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:30.031625  3248 leader_election.cc:290] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:30.032125  3398 leader_election.cc:290] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:30.032473  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:30.033142  1679 raft_consensus.cc:2468] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 1.
I20260709 12:59:30.034111  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:30.034214  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:30.029321  3374 leader_election.cc:290] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:30.024564  3268 raft_consensus.cc:493] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:30.034847  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:29.978266  3295 leader_election.cc:290] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:30.035159  3268 raft_consensus.cc:515] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:29.964852  3307 leader_election.cc:290] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:30.034996  1680 raft_consensus.cc:2468] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 1.
I20260709 12:59:30.038894  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:30.043781  1672 raft_consensus.cc:2468] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 1.
I20260709 12:59:30.045840  1569 leader_election.cc:304] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 12:59:29.987903  3357 raft_consensus.cc:515] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:30.049773  3357 leader_election.cc:290] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:30.056250  3162 raft_consensus.cc:2804] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:59:30.057030  3162 raft_consensus.cc:493] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Starting leader election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 12:59:30.057564  3162 raft_consensus.cc:3060] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:59:29.973418  3355 raft_consensus.cc:493] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 12:59:30.058801  3351 raft_consensus.cc:493] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 12:59:30.058943  3355 raft_consensus.cc:515] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:30.059711  3351 raft_consensus.cc:515] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:29.959514  3402 raft_consensus.cc:515] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:30.066390  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:30.067281  1678 raft_consensus.cc:2468] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 1.
I20260709 12:59:30.017427  3407 raft_consensus.cc:493] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 12:59:30.055490  1569 leader_election.cc:304] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 12:59:30.064590  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:30.017163  3426 raft_consensus.cc:515] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 12:59:30.102945  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 47555.2us
I20260709 12:59:30.105298  3426 leader_election.cc:290] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:30.106155  3402 leader_election.cc:290] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:30.106103  3426 raft_consensus.cc:493] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
W20260709 12:59:30.067458  1534 debug-util.cc:398] Leaking SignalData structure 0x7b080036e380 after lost signal to thread 1778
I20260709 12:59:29.987149  3214 raft_consensus.cc:515] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:30.044179  3411 raft_consensus.cc:493] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 12:59:30.124357  3351 leader_election.cc:290] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:30.109742  3426 raft_consensus.cc:515] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 12:59:30.132198  1534 debug-util.cc:398] Leaking SignalData structure 0x7b080038fe60 after lost signal to thread 2361
I20260709 12:59:30.124161  3162 raft_consensus.cc:515] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:30.101609  3408 leader_election.cc:290] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:30.098018  3374 raft_consensus.cc:493] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 12:59:30.164255  3406 leader_election.cc:290] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:29.988958  3421 raft_consensus.cc:515] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:30.101111  3407 raft_consensus.cc:515] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:30.068992  3420 raft_consensus.cc:493] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 12:59:30.028358  3419 raft_consensus.cc:515] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:29.959815  3415 raft_consensus.cc:515] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:29.973469  3428 raft_consensus.cc:515] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:30.195544  3422 raft_consensus.cc:493] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 12:59:30.195559  3425 raft_consensus.cc:493] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 12:59:30.198386  3162 leader_election.cc:290] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 3 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 12:59:30.198354  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 92950.1us
I20260709 12:59:30.133989  3411 raft_consensus.cc:515] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:30.187419  3397 raft_consensus.cc:2804] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:59:30.187076  3400 raft_consensus.cc:493] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 12:59:30.256824  1569 leader_election.cc:304] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 12:59:30.201272  3429 raft_consensus.cc:493] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:30.196144  3410 leader_election.cc:290] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:30.189652  3351 raft_consensus.cc:493] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 12:59:30.199409  3307 raft_consensus.cc:493] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 12:59:30.287106  3427 raft_consensus.cc:493] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 12:59:30.182505  3402 raft_consensus.cc:493] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 12:59:30.311241  1604 raft_consensus.cc:1275] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
W20260709 12:59:30.315249  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 58525.5us
I20260709 12:59:30.258124  3423 leader_election.cc:290] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:30.350292  3297 raft_consensus.cc:493] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 12:59:30.331099  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:30.334611  3400 raft_consensus.cc:515] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:30.332116  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:30.332180  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:30.367951  2332 raft_consensus.cc:2468] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 2.
I20260709 12:59:30.312358  3357 raft_consensus.cc:493] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 12:59:30.238933  3403 raft_consensus.cc:493] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:30.239351  3440 raft_consensus.cc:493] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 12:59:30.288089  3433 raft_consensus.cc:493] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 12:59:30.354213  3443 raft_consensus.cc:493] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
W20260709 12:59:30.358204  2300 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.532s	user 0.001s	sys 0.025s
I20260709 12:59:30.365329  1569 leader_election.cc:304] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 12:59:30.381322  3268 leader_election.cc:290] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:30.381930  3268 raft_consensus.cc:2804] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:59:30.382012  1569 leader_election.cc:304] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 12:59:30.382369  3268 raft_consensus.cc:493] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Starting leader election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 12:59:30.365518  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:30.382962  3419 leader_election.cc:290] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:30.383350  1604 raft_consensus.cc:2468] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 2.
I20260709 12:59:30.385406  3162 raft_consensus.cc:2804] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:59:30.385566  3428 leader_election.cc:290] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:30.385954  3162 raft_consensus.cc:493] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Starting leader election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 12:59:30.198590  3374 raft_consensus.cc:515] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:30.311959  3397 raft_consensus.cc:493] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 12:59:30.355151  3358 raft_consensus.cc:493] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 12:59:30.393810  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:30.399323  2333 raft_consensus.cc:2468] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 2.
I20260709 12:59:30.356019  3410 raft_consensus.cc:493] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 12:59:30.334754  3295 raft_consensus.cc:2804] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:59:30.379346  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:30.367060  3297 raft_consensus.cc:515] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:30.377687  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:30.382802  3268 raft_consensus.cc:3060] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:30.386504  3162 raft_consensus.cc:3060] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:30.351912  3404 raft_consensus.cc:493] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 12:59:30.312709  3329 raft_consensus.cc:493] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 12:59:30.406885  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:30.411239  3438 raft_consensus.cc:493] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 12:59:30.376729  3347 leader_election.cc:290] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:30.419888  3415 leader_election.cc:290] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:30.371447  3432 raft_consensus.cc:493] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
W20260709 12:59:30.404374  2300 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.609s	user 0.002s	sys 0.025s
I20260709 12:59:30.408715  3295 raft_consensus.cc:493] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Starting leader election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 12:59:30.399022  3397 raft_consensus.cc:3060] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:59:30.386986  3409 raft_consensus.cc:493] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 12:59:30.379856  3417 leader_election.cc:290] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:30.367424  3307 raft_consensus.cc:515] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:30.371534  3351 raft_consensus.cc:515] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:30.427774  2332 raft_consensus.cc:2468] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 1.
I20260709 12:59:30.437244  3434 raft_consensus.cc:493] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 12:59:30.437172  3439 raft_consensus.cc:493] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 12:59:30.437812  3441 raft_consensus.cc:493] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 12:59:30.438269  3446 raft_consensus.cc:493] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 12:59:30.386590  3402 raft_consensus.cc:515] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:30.421350  3426 leader_election.cc:290] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:30.389915  3357 raft_consensus.cc:515] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:30.431355  3355 leader_election.cc:290] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:30.202476  3420 raft_consensus.cc:515] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:30.430816  3405 leader_election.cc:290] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:30.433985  3399 raft_consensus.cc:493] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:30.370829  3429 raft_consensus.cc:515] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:30.415143  3410 raft_consensus.cc:515] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:30.315774  3422 raft_consensus.cc:515] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:30.431562  3404 raft_consensus.cc:515] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:30.446100  3400 leader_election.cc:290] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:30.416023  3358 raft_consensus.cc:515] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:30.315475  3425 raft_consensus.cc:515] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:30.387532  3427 raft_consensus.cc:515] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:30.414881  3443 raft_consensus.cc:515] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:30.404192  3440 raft_consensus.cc:515] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:30.458421  3407 leader_election.cc:290] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:30.449164  3432 raft_consensus.cc:515] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:30.412048  3433 raft_consensus.cc:515] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:30.457202  3434 raft_consensus.cc:515] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:30.456422  3441 raft_consensus.cc:515] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:30.453265  3439 raft_consensus.cc:515] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:30.455140  3446 raft_consensus.cc:515] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:30.458112  3409 raft_consensus.cc:515] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:30.442880  3438 raft_consensus.cc:515] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:30.383008  1569 leader_election.cc:304] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 12:59:30.399940  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:30.443938  3214 leader_election.cc:290] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:30.442739  3295 raft_consensus.cc:3060] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:59:30.583959  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:30.433087  3329 raft_consensus.cc:515] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:30.651448  3446 leader_election.cc:290] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:30.653326  3440 leader_election.cc:290] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:30.655133  3432 leader_election.cc:290] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:30.608280  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:30.608834  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:30.617720  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:30.554312  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:30.575467  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:30.607412  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:30.557747  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:30.481673  3420 leader_election.cc:290] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:30.555536  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:30.587244  2335 raft_consensus.cc:2468] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 1.
I20260709 12:59:30.677211  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:30.602780  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:30.403851  3403 raft_consensus.cc:515] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:30.458798  3421 leader_election.cc:290] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 12:59:30.768469  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 334526us
I20260709 12:59:30.702674  3314 raft_consensus.cc:493] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 12:59:30.584017  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:30.704085   431 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:59:30.705978  3429 leader_election.cc:290] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:30.457011  3442 raft_consensus.cc:493] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 12:59:30.632051  3162 raft_consensus.cc:515] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:30.462260  3411 leader_election.cc:290] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:30.481397  3410 leader_election.cc:290] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:30.477085  3427 leader_election.cc:290] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:30.585395  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:30.457016  3399 raft_consensus.cc:515] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:30.698254  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:30.706104  2334 raft_consensus.cc:2468] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 1.
I20260709 12:59:30.706511  3443 leader_election.cc:290] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:30.716692  3414 raft_consensus.cc:493] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 12:59:30.782991  3442 raft_consensus.cc:515] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:30.760874  1605 raft_consensus.cc:2468] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 1.
I20260709 12:59:30.716984  3415 raft_consensus.cc:493] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 12:59:30.761284  1601 raft_consensus.cc:2468] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 1.
I20260709 12:59:30.761672  2333 raft_consensus.cc:2468] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 2.
I20260709 12:59:30.753886  3439 leader_election.cc:290] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:30.762055  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:30.762413  2332 raft_consensus.cc:2468] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 2.
I20260709 12:59:30.762768  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:30.760908  1600 raft_consensus.cc:2468] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 2.
I20260709 12:59:30.763438  1602 raft_consensus.cc:2468] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 1.
I20260709 12:59:30.784554  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:30.481804  3444 raft_consensus.cc:493] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 12:59:30.752986  3356 raft_consensus.cc:493] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:59:30.764137  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 96439us
W20260709 12:59:30.764962  2299 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.582s	user 0.003s	sys 0.032s
I20260709 12:59:30.766610  3307 leader_election.cc:290] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:30.768630  3401 raft_consensus.cc:493] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 12:59:30.703012  3409 leader_election.cc:290] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:30.769621  3412 raft_consensus.cc:493] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
W20260709 12:59:30.771066  2301 outbound_call.cc:321] RPC callback for RPC call kudu.master.MasterService.TSHeartbeat -> {remote=127.0.107.254:33643, user_credentials={real_user=slave}} blocked reactor thread for 47525.7us
I20260709 12:59:30.773674  3403 leader_election.cc:290] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:30.773839  3297 leader_election.cc:290] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:30.769752  3314 raft_consensus.cc:515] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:30.775840  3413 raft_consensus.cc:493] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 12:59:30.778714  3402 leader_election.cc:290] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:30.771149  1569 leader_election.cc:304] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 12:59:30.677954  3425 leader_election.cc:290] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:30.786633  2331 raft_consensus.cc:2468] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 2.
I20260709 12:59:30.795921  3162 leader_election.cc:290] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 2 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:30.797223  3374 leader_election.cc:290] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 12:59:30.799921  2299 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.686s	user 0.003s	sys 0.032s
I20260709 12:59:30.800769  3422 leader_election.cc:290] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:30.802101  3450 raft_consensus.cc:2804] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:59:30.824956  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:30.823079  3450 raft_consensus.cc:493] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Starting leader election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 12:59:30.826306  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:30.824312  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 12:59:30.828401  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 58683.2us
I20260709 12:59:30.799031  3441 leader_election.cc:290] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:30.826068  3450 raft_consensus.cc:3060] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:30.820261  3401 raft_consensus.cc:515] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:30.831003  3398 raft_consensus.cc:2804] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:59:30.819182  3351 leader_election.cc:290] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:30.798386  3356 raft_consensus.cc:515] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:30.821341  3413 raft_consensus.cc:515] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:30.831533  3398 raft_consensus.cc:493] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Starting leader election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 12:59:30.809993  3434 leader_election.cc:290] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:30.840672  3398 raft_consensus.cc:3060] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:30.840932  3453 raft_consensus.cc:493] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 12:59:30.803555  3412 raft_consensus.cc:515] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:30.842952  1598 raft_consensus.cc:2468] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 2.
I20260709 12:59:30.844583  2298 leader_election.cc:304] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 12:59:30.845180  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:30.845930  2298 leader_election.cc:304] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 12:59:30.850418  3434 raft_consensus.cc:2804] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:59:30.853130  3399 leader_election.cc:290] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:30.853302  3434 raft_consensus.cc:493] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Starting leader election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 12:59:30.853453  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:30.853835  3434 raft_consensus.cc:3060] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:30.866739  3433 leader_election.cc:290] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:30.846647  3438 leader_election.cc:290] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:30.869165  3357 leader_election.cc:290] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:30.869802  1643 leader_election.cc:304] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 12:59:30.860316  1596 raft_consensus.cc:2468] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 2.
I20260709 12:59:30.873864  3439 raft_consensus.cc:2804] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:59:30.874378  3439 raft_consensus.cc:493] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Starting leader election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 12:59:30.874840  3439 raft_consensus.cc:3060] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:30.874946  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:30.841835  3453 raft_consensus.cc:515] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:30.875808  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:30.878281  1596 raft_consensus.cc:2468] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 1.
I20260709 12:59:30.871071  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:30.867599  3459 raft_consensus.cc:493] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 12:59:30.832811  3415 raft_consensus.cc:515] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:30.870033  3444 raft_consensus.cc:515] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:30.826577  3414 raft_consensus.cc:515] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:30.973155  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:30.905005  2298 leader_election.cc:304] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 12:59:30.911849  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:30.875813  1600 raft_consensus.cc:2468] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 1.
I20260709 12:59:30.875730  3307 raft_consensus.cc:2804] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:59:30.867749  3358 leader_election.cc:290] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:31.019789  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:31.025712  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:31.012296  3401 leader_election.cc:290] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:31.041841  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:31.043006  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:31.046519  1754 raft_consensus.cc:1275] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260709 12:59:31.048007  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:30.876098  3351 consensus_queue.cc:1048] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.162s
I20260709 12:59:31.049096  3295 raft_consensus.cc:515] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:31.050828  1599 raft_consensus.cc:2393] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 3: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 12:59:31.043300  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 12:59:30.983657  3397 consensus_meta.cc:287] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.537s	user 0.014s	sys 0.000s
I20260709 12:59:31.019778  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
W20260709 12:59:31.030826  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 126053us
I20260709 12:59:31.026475  1604 raft_consensus.cc:2468] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 2.
I20260709 12:59:30.949960  1605 raft_consensus.cc:1275] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260709 12:59:30.912652  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:31.038079  1678 raft_consensus.cc:2468] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 1.
I20260709 12:59:31.038257  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:31.049237  3409 raft_consensus.cc:2804] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:59:31.050027  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:31.056077  3409 raft_consensus.cc:493] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 12:59:31.056526  3409 raft_consensus.cc:3060] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:59:31.057425  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:30.995872  3329 leader_election.cc:290] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:30.904366  3314 leader_election.cc:290] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 12:59:31.011198  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 138551us
I20260709 12:59:30.994501  3458 raft_consensus.cc:493] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 12:59:31.029999  3307 raft_consensus.cc:493] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 12:59:31.063082  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:31.063725  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:31.064028  1679 raft_consensus.cc:3060] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:59:31.071229  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:31.065675  3404 leader_election.cc:290] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:31.075544  3412 leader_election.cc:290] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:31.078649  1671 raft_consensus.cc:2468] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 1.
I20260709 12:59:31.080116  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:31.081054  1673 raft_consensus.cc:2468] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 1.
I20260709 12:59:31.081884  2298 leader_election.cc:304] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 12:59:31.082077  3356 leader_election.cc:290] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:31.058305  3398 raft_consensus.cc:515] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:31.085521  3414 leader_election.cc:290] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:31.086138  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:31.087057  3411 raft_consensus.cc:2804] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:59:31.086942  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:31.087272  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:31.087571  3411 raft_consensus.cc:493] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Starting leader election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 12:59:31.087888  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:31.088644  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:31.088740  1750 raft_consensus.cc:2468] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 1.
I20260709 12:59:31.089344  3398 leader_election.cc:290] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 2 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:31.087994  3411 raft_consensus.cc:3060] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:31.066062  3307 raft_consensus.cc:3060] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:59:31.079977  3450 raft_consensus.cc:515] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:31.060487  3453 leader_election.cc:290] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:31.098747  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:31.102288  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:31.102413  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:31.102933  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:31.103190  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:31.099992  2299 leader_election.cc:304] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:31.106240  1679 raft_consensus.cc:2468] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 3.
I20260709 12:59:31.103904  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:31.109638  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:31.107914  1602 raft_consensus.cc:1240] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 1. Current term is 2. Ops: []
I20260709 12:59:31.109534  2333 raft_consensus.cc:2468] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 1.
I20260709 12:59:31.103840  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:31.111809  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:31.112146  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:31.113065  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:31.113741  2334 raft_consensus.cc:2468] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 1.
I20260709 12:59:31.059100  3459 raft_consensus.cc:515] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:31.115147  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:31.115474  2298 leader_election.cc:304] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 12:59:31.117900  1749 raft_consensus.cc:2468] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 1.
I20260709 12:59:31.119436  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:31.064348  3397 raft_consensus.cc:515] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:31.120559  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:31.115530  3458 raft_consensus.cc:515] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:31.121803  3444 leader_election.cc:290] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:31.119405  3442 leader_election.cc:290] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:31.127583  2298 leader_election.cc:304] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 12:59:31.126809  1747 raft_consensus.cc:2468] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 2.
I20260709 12:59:31.128243  1597 raft_consensus.cc:1240] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 2. Current term is 3. Ops: []
I20260709 12:59:31.128515  3410 raft_consensus.cc:2804] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:59:31.128997  3410 raft_consensus.cc:493] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 12:59:31.129379  3410 raft_consensus.cc:3060] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:59:31.130331  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:31.131193  1750 raft_consensus.cc:2468] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 2.
I20260709 12:59:31.131794  3397 leader_election.cc:290] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 3 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:31.132617  1643 leader_election.cc:304] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 12:59:31.135794  3351 raft_consensus.cc:2804] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:59:31.136233  3351 raft_consensus.cc:493] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:59:31.136636  3351 raft_consensus.cc:3060] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:59:31.126340  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:31.140719  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:31.141433  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:31.142650  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:31.143523  1603 raft_consensus.cc:1240] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 2. Current term is 3. Ops: []
I20260709 12:59:31.143980  1604 raft_consensus.cc:1240] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 1. Current term is 2. Ops: []
I20260709 12:59:31.144845  1599 raft_consensus.cc:1240] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 2. Current term is 3. Ops: []
I20260709 12:59:31.145057  3329 raft_consensus.cc:2804] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:59:31.146925  3427 raft_consensus.cc:2749] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:59:31.148392  1597 raft_consensus.cc:1275] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:59:31.148934  3415 leader_election.cc:290] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:31.149935  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:31.150786  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:31.156021  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 12:59:31.156561  3268 consensus_meta.cc:287] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.729s	user 0.011s	sys 0.005s
I20260709 12:59:31.157140  3268 raft_consensus.cc:515] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:31.159879  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:31.160627  1675 raft_consensus.cc:2468] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 2.
I20260709 12:59:31.158313  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:31.161527  3268 leader_election.cc:290] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 2 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:31.145514  3329 raft_consensus.cc:493] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Starting leader election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 12:59:31.124899  3413 leader_election.cc:290] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:31.163936  3459 leader_election.cc:290] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:31.167610  1602 raft_consensus.cc:1240] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 1. Current term is 2. Ops: []
I20260709 12:59:31.175681  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:31.173206  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:31.173979  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:31.174137  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:31.174485  1600 raft_consensus.cc:1240] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 1. Current term is 2. Ops: []
I20260709 12:59:31.174799  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:31.175006  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:31.169924  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:31.175602  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:31.176244  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:31.176880  3295 leader_election.cc:290] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 3 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:31.170568  3329 raft_consensus.cc:3060] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:31.177457  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:31.177968  2299 leader_election.cc:304] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 12:59:31.181576  3434 raft_consensus.cc:515] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:31.182102  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:31.177490  3458 leader_election.cc:290] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:31.183727  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:31.186302  1601 raft_consensus.cc:2468] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 4.
I20260709 12:59:31.193059  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:59:31.188234  1749 raft_consensus.cc:2468] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 2.
I20260709 12:59:31.195760  1677 raft_consensus.cc:3060] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:31.199883  3398 raft_consensus.cc:493] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 12:59:31.200523  3398 raft_consensus.cc:515] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:31.201483  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:31.161062  3409 raft_consensus.cc:515] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:31.202706  3427 raft_consensus.cc:2804] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:59:31.203243  2298 leader_election.cc:304] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 12:59:31.206076  2335 raft_consensus.cc:2393] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 2: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 12:59:31.210690  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:31.170174  3411 raft_consensus.cc:515] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:31.212687  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:31.212881  3431 raft_consensus.cc:2804] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:59:31.232259  3431 raft_consensus.cc:493] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 12:59:31.232481  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:31.232751  3431 raft_consensus.cc:3060] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:31.205137  3427 raft_consensus.cc:493] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Starting leader election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 12:59:31.213371  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:31.213775  3398 leader_election.cc:290] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:31.214147  1599 raft_consensus.cc:1275] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260709 12:59:31.214084  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:31.214890  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:31.221503  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:31.239094  3307 raft_consensus.cc:515] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:31.223990  3434 leader_election.cc:290] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 2 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:31.241590  3307 leader_election.cc:290] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 3 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:31.224507  1644 leader_election.cc:304] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 12:59:31.224633  3442 consensus_queue.cc:1059] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20260709 12:59:31.242405  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 53568.4us
I20260709 12:59:31.226809  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:31.227666  2299 leader_election.cc:304] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 12:59:31.213670  3410 raft_consensus.cc:515] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:31.229182  3459 consensus_queue.cc:1059] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:59:31.201828  3450 leader_election.cc:290] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 2 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:31.238056  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:31.243036  3442 raft_consensus.cc:3055] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:59:31.236207  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:31.238618  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:31.244616  3442 raft_consensus.cc:740] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 12:59:31.244879  1750 raft_consensus.cc:2468] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 1.
I20260709 12:59:31.245276  1747 raft_consensus.cc:2468] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 1.
I20260709 12:59:31.236904  3427 raft_consensus.cc:3060] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:31.243500  1644 leader_election.cc:304] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 12:59:31.247943  2298 leader_election.cc:304] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 12:59:31.245780  3442 consensus_queue.cc:260] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:31.248167  3307 raft_consensus.cc:2804] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:59:31.252614  3307 raft_consensus.cc:493] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Starting leader election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 12:59:31.253024  3307 raft_consensus.cc:3060] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:31.254125  3410 leader_election.cc:290] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 3 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:31.256476  2299 leader_election.cc:304] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 12:59:31.256117  3413 consensus_queue.cc:1059] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:59:31.249749  3442 raft_consensus.cc:3060] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:59:31.261133  3410 raft_consensus.cc:2804] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:59:31.268579  3420 raft_consensus.cc:2804] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:59:31.269069  3420 raft_consensus.cc:493] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Starting leader election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 12:59:31.268870  3459 consensus_queue.cc:1048] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:59:31.269214  3413 raft_consensus.cc:3055] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:59:31.269554  3420 raft_consensus.cc:3060] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:31.269740  3413 raft_consensus.cc:740] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 12:59:31.271303  2299 leader_election.cc:304] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:31.262602  1644 leader_election.cc:304] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:31.254974  3414 raft_consensus.cc:2804] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:59:31.255281  3434 raft_consensus.cc:2804] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:59:31.261761  3410 raft_consensus.cc:493] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Starting leader election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 12:59:31.285552  3434 raft_consensus.cc:493] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Starting leader election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 12:59:31.310830  3434 raft_consensus.cc:3060] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:31.285899  3414 raft_consensus.cc:493] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Starting leader election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 12:59:31.299542  3410 raft_consensus.cc:3060] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:59:31.333933  3458 raft_consensus.cc:3055] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:59:31.271066  3413 consensus_queue.cc:260] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:31.340271  3458 raft_consensus.cc:740] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 12:59:31.343906  3459 consensus_queue.cc:1059] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:59:31.344655  3453 consensus_queue.cc:1059] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:59:31.350171  1747 raft_consensus.cc:1275] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:59:31.353400  1677 raft_consensus.cc:2468] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 2.
I20260709 12:59:31.335506  3414 raft_consensus.cc:3060] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:31.341842  3456 consensus_queue.cc:1048] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.023s
I20260709 12:59:31.357684  1569 leader_election.cc:304] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 12:59:31.364177  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 12:59:31.367202  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 95975.1us
I20260709 12:59:31.368052  2299 leader_election.cc:304] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 12:59:31.370478  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 112608us
I20260709 12:59:31.374353  3416 raft_consensus.cc:2749] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:59:31.375303  3413 raft_consensus.cc:3060] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:59:31.379783  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 99573.7us
I20260709 12:59:31.379941  1750 raft_consensus.cc:2468] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 1.
I20260709 12:59:31.399497  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:59:31.395288  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:31.404372  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:31.379117  3409 leader_election.cc:290] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 3 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:31.404496  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:31.348555  3356 consensus_queue.cc:1059] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.034s
I20260709 12:59:31.419275  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
W20260709 12:59:31.399283  1571 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.107.195:40693 timed out after 1.596s (SENT)
I20260709 12:59:31.419435  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:59:31.420053  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:59:31.384164  3295 raft_consensus.cc:2804] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:59:31.427512  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:31.384047  3440 raft_consensus.cc:2749] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:59:31.391960  3439 raft_consensus.cc:515] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:31.433774  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:31.440564  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:31.441614  1749 raft_consensus.cc:2468] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 1.
I20260709 12:59:31.442667  1748 raft_consensus.cc:2468] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 4.
I20260709 12:59:31.442893  3440 raft_consensus.cc:2749] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:59:31.444092  2336 raft_consensus.cc:1240] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 1. Current term is 2. Ops: []
I20260709 12:59:31.447499  1717 leader_election.cc:304] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
W20260709 12:59:31.447796  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 115778us
W20260709 12:59:31.448407  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 63681.4us
I20260709 12:59:31.447841  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:31.455824  1569 leader_election.cc:304] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 12:59:31.457082  3268 raft_consensus.cc:2804] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:59:31.457516  3268 raft_consensus.cc:493] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 12:59:31.457928  3268 raft_consensus.cc:3060] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:59:31.461335  1674 raft_consensus.cc:1240] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 1. Current term is 2. Ops: []
I20260709 12:59:31.462733  1674 raft_consensus.cc:1240] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 2. Current term is 3. Ops: []
I20260709 12:59:31.462848  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:31.464265  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:31.464409  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:31.464958  1671 raft_consensus.cc:2468] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 1.
I20260709 12:59:31.465229  1676 raft_consensus.cc:2468] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 2.
I20260709 12:59:31.465978  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:31.449317  3440 consensus_queue.cc:1059] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.018s
I20260709 12:59:31.454257  1750 raft_consensus.cc:2468] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 1.
I20260709 12:59:31.431000  3295 raft_consensus.cc:493] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
W20260709 12:59:31.493665  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 44586.4us
W20260709 12:59:31.545872  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 90155.3us
I20260709 12:59:31.469069  3453 raft_consensus.cc:2804] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260709 12:59:31.554203  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:59:31.428072  3414 raft_consensus.cc:515] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:31.496496  3295 raft_consensus.cc:3060] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:59:31.572576  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:31.570716  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:31.527177  3440 raft_consensus.cc:3055] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:59:31.574033  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:31.431668  3458 consensus_queue.cc:260] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:31.576567  1569 leader_election.cc:304] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 12:59:31.575495  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:31.574167  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 12:59:31.551723  3453 raft_consensus.cc:493] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:59:31.591480  1679 raft_consensus.cc:1240] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 1. Current term is 2. Ops: []
I20260709 12:59:31.592082  1678 raft_consensus.cc:2468] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 4.
W20260709 12:59:31.585466  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 262480us
I20260709 12:59:31.598495  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:31.578207  3440 raft_consensus.cc:740] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 12:59:31.539705  3427 raft_consensus.cc:515] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:31.551183  3410 raft_consensus.cc:515] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:31.607659  1569 leader_election.cc:304] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 12:59:31.573186  3356 consensus_queue.cc:1048] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.027s
I20260709 12:59:31.594139  3453 raft_consensus.cc:3060] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:59:31.608891  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:31.596247  3450 raft_consensus.cc:2804] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260709 12:59:31.613873  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 245853us
I20260709 12:59:31.581475  3329 raft_consensus.cc:515] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:31.616763  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:31.600353  3307 raft_consensus.cc:515] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:31.618958  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:31.609771  2331 raft_consensus.cc:1240] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 2. Current term is 3. Ops: []
I20260709 12:59:31.612159  3450 raft_consensus.cc:493] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Starting leader election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 12:59:31.622263  2334 raft_consensus.cc:2393] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 2: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 12:59:31.624562  3433 consensus_queue.cc:1059] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:59:31.625089  3459 raft_consensus.cc:3055] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:59:31.625537  3459 raft_consensus.cc:740] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 12:59:31.625761  3433 raft_consensus.cc:3055] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:59:31.626142  3433 raft_consensus.cc:740] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 12:59:31.619418  3398 raft_consensus.cc:2804] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:59:31.630352  3398 raft_consensus.cc:493] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 12:59:31.630457  3329 leader_election.cc:290] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 2 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:31.628362  1747 raft_consensus.cc:2468] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 3.
I20260709 12:59:31.630904  3398 raft_consensus.cc:3060] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:59:31.633350  3412 raft_consensus.cc:3055] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:59:31.633790  3412 raft_consensus.cc:740] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 12:59:31.634181  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:59:31.634831  3412 consensus_queue.cc:260] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:31.635941  3412 raft_consensus.cc:3060] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:31.638242  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:59:31.638881  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 12:59:31.641392  1569 leader_election.cc:304] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 12:59:31.641882  3427 leader_election.cc:290] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 2 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:31.643582  3410 leader_election.cc:290] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 3 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:31.643918  3397 raft_consensus.cc:2804] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:59:31.644377  3397 raft_consensus.cc:493] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 12:59:31.644732  3397 raft_consensus.cc:3060] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:59:31.644886  3439 leader_election.cc:290] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 2 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:31.645718  3440 consensus_queue.cc:260] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:31.646981  3440 raft_consensus.cc:3060] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:59:31.647141  3458 raft_consensus.cc:3060] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:31.622294  3450 raft_consensus.cc:3060] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:31.628355  3431 raft_consensus.cc:515] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:31.650813  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:31.652675  3431 leader_election.cc:290] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 2 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:31.653766  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:31.654969  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:31.655295  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:31.656903  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:31.643472  1569 leader_election.cc:304] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 12:59:31.659385  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:31.659507  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:31.657622  3351 raft_consensus.cc:515] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:31.662082  1673 raft_consensus.cc:2393] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 3: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 12:59:31.663173  3351 leader_election.cc:290] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 3 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:31.667210  3415 raft_consensus.cc:2804] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:59:31.667646  3415 raft_consensus.cc:493] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Starting leader election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 12:59:31.675750  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:31.676375  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:31.676724  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:31.677429  1676 raft_consensus.cc:1275] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260709 12:59:31.680253  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:31.678730  1569 leader_election.cc:304] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 12:59:31.668056  3415 raft_consensus.cc:3060] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:31.683154  2331 raft_consensus.cc:1240] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 1. Current term is 2. Ops: []
I20260709 12:59:31.686622  1569 leader_election.cc:304] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 12:59:31.686767  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:31.689802  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:31.689970  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:31.690659  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:31.691752  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:31.697594  2333 raft_consensus.cc:1240] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 2. Current term is 3. Ops: []
I20260709 12:59:31.698654  2332 raft_consensus.cc:1240] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 1. Current term is 2. Ops: []
I20260709 12:59:31.698843  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:31.699656  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:31.700079  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:31.700891  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:31.701256  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:31.703740  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:59:31.704180  2329 raft_consensus.cc:1240] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 1. Current term is 2. Ops: []
I20260709 12:59:31.705569  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:31.702890  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:31.706866  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:31.708108  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:31.708297  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:31.709832  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:31.691552  3397 raft_consensus.cc:515] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:31.716049  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:31.626579  3459 consensus_queue.cc:260] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:31.722353  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 12:59:31.720542  3268 raft_consensus.cc:515] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:31.727157  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:31.727869  1677 raft_consensus.cc:2468] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 1.
I20260709 12:59:31.729086  3268 leader_election.cc:290] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 3 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:31.729765  3268 raft_consensus.cc:2804] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:59:31.692796  3398 raft_consensus.cc:515] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:31.692737  3433 consensus_queue.cc:260] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 12:59:31.733103  1717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:59:31.734305  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:59:31.731891  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:31.730109  3268 raft_consensus.cc:493] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Starting leader election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 12:59:31.743052  1643 leader_election.cc:304] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:31.743703  3268 raft_consensus.cc:3060] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:59:31.746384  3398 leader_election.cc:290] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 3 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:31.747713  3314 raft_consensus.cc:2749] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:59:31.740693  3466 raft_consensus.cc:2804] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260709 12:59:31.757896  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 192178us
W20260709 12:59:31.755350  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 68895.9us
I20260709 12:59:31.761727  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:31.758220  3459 raft_consensus.cc:3060] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:59:31.767485  1717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:59:31.749884  3444 raft_consensus.cc:493] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 12:59:31.750698  3397 leader_election.cc:290] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 3 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:31.782877  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:59:31.784360  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:59:31.785354  3453 raft_consensus.cc:515] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:31.787585  3444 raft_consensus.cc:515] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } }
I20260709 12:59:31.792893  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:59:31.758968  3466 raft_consensus.cc:493] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 12:59:31.804200  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:59:31.809144  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:31.813647  2298 leader_election.cc:304] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:31.814837  3329 raft_consensus.cc:2749] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:59:31.815508  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:59:31.819140  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:31.826628  1598 raft_consensus.cc:1240] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 2. Current term is 3. Ops: []
I20260709 12:59:31.828877  2298 leader_election.cc:304] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:31.768235  3314 consensus_queue.cc:1059] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.041s
W20260709 12:59:31.832372  3434 consensus_meta.cc:287] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.521s	user 0.009s	sys 0.013s
I20260709 12:59:31.822024  3450 raft_consensus.cc:515] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:31.947360  3453 leader_election.cc:290] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 12:59:31.924124  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 61041.3us
W20260709 12:59:31.970479  3420 consensus_meta.cc:287] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.700s	user 0.013s	sys 0.018s
I20260709 12:59:31.987676  1598 raft_consensus.cc:1240] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 1. Current term is 2. Ops: []
I20260709 12:59:31.987977  3420 raft_consensus.cc:515] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:31.964609  3415 raft_consensus.cc:515] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:31.989176  2335 raft_consensus.cc:1240] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 1. Current term is 2. Ops: []
I20260709 12:59:31.968385  3268 raft_consensus.cc:515] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:31.954226  3425 raft_consensus.cc:3055] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:59:31.991631  1605 raft_consensus.cc:1240] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 2. Current term is 3. Ops: []
I20260709 12:59:31.991623  3425 raft_consensus.cc:740] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 12:59:31.992264  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:59:31.992872  3268 leader_election.cc:290] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 3 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 12:59:31.991655  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:59:31.993076  3425 consensus_queue.cc:260] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:31.993851  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:31.994447  3425 raft_consensus.cc:3060] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:31.846836  3356 consensus_queue.cc:1059] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.148s
I20260709 12:59:31.996127  3453 raft_consensus.cc:3055] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:59:31.996479  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
W20260709 12:59:31.961472  3413 consensus_meta.cc:287] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.586s	user 0.007s	sys 0.006s
I20260709 12:59:31.843128  3411 leader_election.cc:290] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 2 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:31.936652  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:31.996619  3453 raft_consensus.cc:740] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 12:59:31.998140  1753 raft_consensus.cc:2468] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 3.
I20260709 12:59:32.001451  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:59:32.000943  3453 consensus_queue.cc:260] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:32.002688  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 12:59:31.936436  1718 consensus_peers.cc:597] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 -> Peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625): Couldn't send request to peer 3b879799899a499c8d409ee9f1277cd1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 983621a6fc5b4c9ea0159a648dfe7a36. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:59:31.996127  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:59:31.998937  3434 raft_consensus.cc:515] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:31.867025  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:59:31.999778  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:59:31.991832  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
W20260709 12:59:32.017962  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 86254.9us
W20260709 12:59:32.004932  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 195480us
I20260709 12:59:31.813078  3466 raft_consensus.cc:3060] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:59:32.021965  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 155412us
I20260709 12:59:32.006855  3420 leader_election.cc:290] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 2 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:32.010742  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:32.004482  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:32.011888  3414 leader_election.cc:290] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 2 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:32.026456  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:32.027683  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:59:32.027987  3307 leader_election.cc:290] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 2 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:32.028452  3356 consensus_queue.cc:1059] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:59:32.029289  3472 raft_consensus.cc:3055] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:59:32.029587  3356 raft_consensus.cc:3055] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:59:32.029696  3472 raft_consensus.cc:740] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 12:59:32.030015  3356 raft_consensus.cc:740] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 12:59:32.030829  3472 consensus_queue.cc:260] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:32.031080  3356 consensus_queue.cc:260] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:31.872645  2328 raft_consensus.cc:1240] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 1. Current term is 2. Ops: []
I20260709 12:59:32.033098  1599 raft_consensus.cc:1240] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 1. Current term is 2. Ops: []
I20260709 12:59:32.033385  1672 raft_consensus.cc:1240] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 2. Current term is 3. Ops: []
I20260709 12:59:32.024075  3453 raft_consensus.cc:3060] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:32.036507  3413 consensus_queue.cc:1059] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:59:32.036870  1598 raft_consensus.cc:1240] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 1. Current term is 2. Ops: []
I20260709 12:59:32.037402  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:59:32.037673  3413 consensus_queue.cc:1059] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:59:32.038615  3358 raft_consensus.cc:3055] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:59:32.039005  3358 raft_consensus.cc:740] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 12:59:32.039856  3444 leader_election.cc:290] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:32.039928  3358 consensus_queue.cc:260] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:32.041626  3358 raft_consensus.cc:3060] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:32.042074  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:31.832733  3461 raft_consensus.cc:2749] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:59:31.876812  3456 raft_consensus.cc:493] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 12:59:32.016299  3450 leader_election.cc:290] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 2 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:32.016423  2298 leader_election.cc:304] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:32.016793  3295 raft_consensus.cc:515] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:32.018473  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:31.860443  3433 raft_consensus.cc:3060] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:59:31.942138  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 12:59:32.077037  1646 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:59:32.043824  1569 leader_election.cc:304] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 12:59:32.044138  1604 raft_consensus.cc:1240] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 1. Current term is 2. Ops: []
I20260709 12:59:32.047811  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 12:59:32.039961  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:59:32.082607  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 12:59:32.082837  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 203871us
I20260709 12:59:32.084255  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:32.043221  1718 leader_election.cc:304] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 12:59:32.061357  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 45019.9us
I20260709 12:59:32.075448  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:59:32.003284  3415 leader_election.cc:290] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 2 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:32.078949  1679 raft_consensus.cc:1240] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 2. Current term is 3. Ops: []
I20260709 12:59:32.032348  3356 raft_consensus.cc:3060] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:59:32.083325  1753 raft_consensus.cc:2468] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 3.
I20260709 12:59:32.084964  1750 raft_consensus.cc:2468] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 4.
I20260709 12:59:32.088675  1605 raft_consensus.cc:1240] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 1. Current term is 2. Ops: []
I20260709 12:59:32.089651  3468 raft_consensus.cc:3055] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:59:32.090094  3468 raft_consensus.cc:740] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 12:59:32.090620  3462 raft_consensus.cc:2749] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:59:32.094766  1643 leader_election.cc:304] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:32.095000  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:32.098321  3468 consensus_queue.cc:260] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:32.099220  3307 raft_consensus.cc:2749] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:59:32.099704  3468 raft_consensus.cc:3060] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:32.100317  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 12:59:32.104339  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:59:32.105135  1673 raft_consensus.cc:2393] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 2: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
W20260709 12:59:32.108857  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 65304.9us
I20260709 12:59:32.120769  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:32.121522  1750 raft_consensus.cc:2468] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 2.
I20260709 12:59:32.115366  1569 leader_election.cc:304] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 12:59:32.124476  2335 raft_consensus.cc:2468] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 3.
I20260709 12:59:32.108006  3450 raft_consensus.cc:2749] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 12:59:32.125311  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 82166.5us
I20260709 12:59:32.126121  1718 leader_election.cc:304] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:32.109462  3462 consensus_queue.cc:1059] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.019s
I20260709 12:59:32.129212  1718 leader_election.cc:304] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:32.112186  3444 consensus_queue.cc:1059] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.015s
I20260709 12:59:32.137009  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:32.137640  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 12:59:32.137984  1569 leader_election.cc:304] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 12:59:32.140331  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:32.141247  1750 raft_consensus.cc:2468] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 2.
I20260709 12:59:32.122812  3434 leader_election.cc:290] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 2 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:32.120967  3459 raft_consensus.cc:2749] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:59:32.085933  3456 raft_consensus.cc:515] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:32.143889  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 12:59:32.144585  1718 leader_election.cc:304] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:32.151433  1603 raft_consensus.cc:1240] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 2. Current term is 3. Ops: []
I20260709 12:59:32.152217  1598 raft_consensus.cc:1240] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 2. Current term is 3. Ops: []
I20260709 12:59:32.136662  3412 raft_consensus.cc:2749] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 12:59:32.159822  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:59:32.143591  3415 raft_consensus.cc:2804] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:59:32.142184  3458 raft_consensus.cc:2749] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:59:32.169164  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:32.154964  3397 raft_consensus.cc:2749] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 12:59:32.181654  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 43744.8us
I20260709 12:59:32.182987  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 12:59:32.183573  1717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:59:32.183710  1750 raft_consensus.cc:2468] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 2.
I20260709 12:59:32.173349  3459 raft_consensus.cc:2749] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:59:32.189500  1679 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653"
dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
mode: GRACEFUL
new_leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
 from {username='slave'} at 127.0.0.1:43266
I20260709 12:59:32.192183  1679 raft_consensus.cc:606] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 LEADER]: Received request to transfer leadership to TS 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 12:59:32.192903  1717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:59:32.206194  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:32.206147  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:32.180439  3415 raft_consensus.cc:697] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 12:59:32.177690  3351 consensus_queue.cc:1059] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.024s
I20260709 12:59:32.178220  3444 consensus_queue.cc:1059] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.023s
I20260709 12:59:32.202227  3295 leader_election.cc:290] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 3 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:32.214701  1569 leader_election.cc:304] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
W20260709 12:59:32.214425  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 69942.9us
I20260709 12:59:32.218523  1718 leader_election.cc:304] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:32.219587  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:32.222617  1752 raft_consensus.cc:2393] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 5: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 12:59:32.233477  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:32.243322  3415 consensus_queue.cc:237] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:32.224467  3413 raft_consensus.cc:2749] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:59:32.244920  1748 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "b9bb5648969b4299a776843d72043448"
dest_uuid: "af520b67945d443b901bbeca37cc5701"
mode: GRACEFUL
new_leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
 from {username='slave'} at 127.0.0.1:51270
I20260709 12:59:32.245582  1748 raft_consensus.cc:606] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 2 LEADER]: Received request to transfer leadership to TS 816fadcea8fc4893882f5f9b9f22a03d
I20260709 12:59:32.248982   431 auto_leader_rebalancer.cc:452] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260709 12:59:32.224572  3295 raft_consensus.cc:2804] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:59:32.249909  3295 raft_consensus.cc:493] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Starting leader election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 12:59:32.250293  3295 raft_consensus.cc:3060] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:59:32.251436   431 auto_leader_rebalancer.cc:527] All tables' leader rebalancing finished this round
I20260709 12:59:32.252696  3424 raft_consensus.cc:3055] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:59:32.253114  3424 raft_consensus.cc:740] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 12:59:32.256608  3424 consensus_queue.cc:260] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:32.260342  3472 raft_consensus.cc:3060] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:32.261750  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:32.233083  3466 raft_consensus.cc:515] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:32.262961  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:32.264418  3466 leader_election.cc:290] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 2 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:32.264679  1598 raft_consensus.cc:1240] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 1. Current term is 2. Ops: []
I20260709 12:59:32.269511  3480 raft_consensus.cc:3055] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:59:32.270015  3480 raft_consensus.cc:740] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 12:59:32.270763  3486 consensus_queue.cc:1059] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:59:32.272064  3486 raft_consensus.cc:3055] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:59:32.272374  3486 raft_consensus.cc:740] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 12:59:32.277537  1750 raft_consensus.cc:2468] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 3.
I20260709 12:59:32.285665  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:32.288204  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 12:59:32.305888  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 75139.7us
I20260709 12:59:32.309345  1569 leader_election.cc:304] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 12:59:32.312587  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 12:59:32.313845  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:32.314132  2332 raft_consensus.cc:1240] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 1. Current term is 2. Ops: []
I20260709 12:59:32.315641  3444 raft_consensus.cc:993] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d: : Instructing follower 83595ad0fd5940c0a2bfc7c61acd969a to start an election
I20260709 12:59:32.316118  3444 raft_consensus.cc:1081] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 LEADER]: Signalling peer 83595ad0fd5940c0a2bfc7c61acd969a to start an election
I20260709 12:59:32.318687  1753 raft_consensus.cc:2468] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 2.
I20260709 12:59:32.320473  1747 raft_consensus.cc:2468] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 2.
I20260709 12:59:32.320719  1718 leader_election.cc:304] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:32.321456  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:32.322974  2299 leader_election.cc:304] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 12:59:32.283700  3473 log.cc:927] Time spent T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.052s	user 0.000s	sys 0.000s
I20260709 12:59:32.258395  3424 raft_consensus.cc:3060] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:59:32.321619  1752 raft_consensus.cc:2468] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 2.
I20260709 12:59:32.289815  3416 raft_consensus.cc:3055] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:59:32.328876  1676 raft_consensus.cc:1240] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 2. Current term is 3. Ops: []
I20260709 12:59:32.329885  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:32.328663  3416 raft_consensus.cc:740] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 12:59:32.331399  3466 raft_consensus.cc:2804] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:59:32.331878  3416 consensus_queue.cc:260] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:32.332772  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:32.333173  3416 raft_consensus.cc:3060] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:59:32.328661  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:32.338832  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:32.339078  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:32.339723  1745 raft_consensus.cc:2376] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 3. Re-sending same reply.
I20260709 12:59:32.340202  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:32.340592  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:32.334335  3425 raft_consensus.cc:2749] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:59:32.290828  3427 raft_consensus.cc:3055] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:59:32.341758  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:32.342010  3486 consensus_queue.cc:260] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:32.343011  3466 raft_consensus.cc:697] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 12:59:32.343011  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:32.336074  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:32.344362  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:32.344172  3466 consensus_queue.cc:237] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:32.345445  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:32.338014  3411 consensus_queue.cc:1059] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:59:32.347173  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:32.334128  3462 raft_consensus.cc:2749] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:59:32.346632  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:32.348996  1754 raft_consensus.cc:2468] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 3.
I20260709 12:59:32.271317  3480 consensus_queue.cc:260] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:32.355720  3480 raft_consensus.cc:3060] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:59:32.290372  3456 leader_election.cc:290] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:32.366641  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
W20260709 12:59:32.367419  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 46809.9us
I20260709 12:59:32.358090  1747 raft_consensus.cc:2468] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 2.
I20260709 12:59:32.354764  1745 raft_consensus.cc:2376] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 3. Re-sending same reply.
I20260709 12:59:32.364144  1753 raft_consensus.cc:2468] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 2.
I20260709 12:59:32.369422  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:32.372063  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:32.343546  3486 raft_consensus.cc:3060] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:32.380177  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:32.383292  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 12:59:32.380318  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 45227.6us
I20260709 12:59:32.388612  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:32.389424  2336 raft_consensus.cc:1240] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 2. Current term is 3. Ops: []
I20260709 12:59:32.342020  3427 raft_consensus.cc:740] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 12:59:32.396121  1718 leader_election.cc:304] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 12:59:32.401808  1604 raft_consensus.cc:1240] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 2. Current term is 3. Ops: []
I20260709 12:59:32.404027  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:59:32.410187  3413 raft_consensus.cc:2749] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 12:59:32.410584  1644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:59:32.411484  1644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:59:32.412607  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 12:59:32.413757  2336 raft_consensus.cc:2393] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 2: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 12:59:32.421068  1569 leader_election.cc:304] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 12:59:32.424120  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 12:59:32.424511  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:59:32.425303  2328 raft_consensus.cc:2393] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 3: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 12:59:32.427531  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 12:59:32.430482  1753 raft_consensus.cc:2468] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 2.
W20260709 12:59:32.461549  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 47041.3us
I20260709 12:59:32.462015  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 12:59:32.463040  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:32.464267  1569 leader_election.cc:304] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 12:59:32.466408  3397 raft_consensus.cc:2749] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:59:32.468528  1569 leader_election.cc:304] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 12:59:32.469566  3450 raft_consensus.cc:2804] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:59:32.470216  3450 raft_consensus.cc:697] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 12:59:32.471647  3450 consensus_queue.cc:237] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:32.474366  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:32.474825  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:32.475512  1751 raft_consensus.cc:2468] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 4.
I20260709 12:59:32.476465  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:32.477078  1752 raft_consensus.cc:2376] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 2. Re-sending same reply.
I20260709 12:59:32.478987  1569 leader_election.cc:304] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 12:59:32.482040  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:32.482789  1745 raft_consensus.cc:2376] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 2. Re-sending same reply.
I20260709 12:59:32.482893  1718 leader_election.cc:304] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:32.483824  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:32.484593  1751 raft_consensus.cc:2468] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 2.
I20260709 12:59:32.484673  1718 leader_election.cc:304] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:32.486917  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:32.488722  3413 raft_consensus.cc:2749] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:59:32.490106  3413 raft_consensus.cc:2804] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:59:32.490751  3413 raft_consensus.cc:493] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 12:59:32.491389  3413 raft_consensus.cc:3060] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:32.496731  1748 raft_consensus.cc:2468] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 3.
I20260709 12:59:32.497232  1750 raft_consensus.cc:2468] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 2.
I20260709 12:59:32.499907  1569 leader_election.cc:304] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
W20260709 12:59:32.578586  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 77150.8us
W20260709 12:59:32.578603  1719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:59:32.590734  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:32.592164  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 12:59:32.593294  2334 raft_consensus.cc:2393] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 2: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 12:59:32.595378  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:32.508097  3268 raft_consensus.cc:2749] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:59:32.589262  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:32.599835  1748 raft_consensus.cc:2393] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 2: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 12:59:32.594137  3481 consensus_queue.cc:1059] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.010s
I20260709 12:59:32.594806  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:32.590317  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:32.478904  3415 raft_consensus.cc:2749] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:59:32.604333  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:32.609618  1747 raft_consensus.cc:2393] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 3: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 12:59:32.610435  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 12:59:32.611709  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 127140us
I20260709 12:59:32.612461  1718 leader_election.cc:304] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 12:59:32.620203  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:32.616662  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:32.622283  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:32.624715  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:32.588068  3397 raft_consensus.cc:2749] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:59:32.626287  3356 raft_consensus.cc:2804] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:59:32.629467  1749 raft_consensus.cc:2468] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 2.
I20260709 12:59:32.639879  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:32.643078  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:32.643400  1747 raft_consensus.cc:2468] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 2.
I20260709 12:59:32.644411  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:32.653784  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:32.648996  2301 leader_election.cc:304] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 12:59:32.660564  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:32.681355  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:32.684273  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:32.684331  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:32.686653  1747 raft_consensus.cc:2376] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 2. Re-sending same reply.
I20260709 12:59:32.616497  3461 consensus_queue.cc:1048] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.051s
I20260709 12:59:32.642164  3356 raft_consensus.cc:493] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
W20260709 12:59:32.703635  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 54769us
I20260709 12:59:32.704761  2301 leader_election.cc:304] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 12:59:32.710781  3431 raft_consensus.cc:2804] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:59:32.711283  3431 raft_consensus.cc:493] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 12:59:32.711683  3431 raft_consensus.cc:3060] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:59:32.715111  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 12:59:32.724145  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 224541us
W20260709 12:59:32.730563  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 62904.3us
W20260709 12:59:32.732602  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 108149us
I20260709 12:59:32.611855  3427 consensus_queue.cc:260] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:32.682065  3481 raft_consensus.cc:493] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 12:59:32.713475  3356 raft_consensus.cc:3060] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:59:32.734231  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 135996us
I20260709 12:59:32.739439  1718 leader_election.cc:304] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:32.738687  2301 leader_election.cc:304] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 12:59:32.743216  2301 leader_election.cc:304] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 12:59:32.745375  2298 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:59:32.749295  1718 leader_election.cc:304] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:32.713040  3295 raft_consensus.cc:515] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:32.732990  3429 raft_consensus.cc:493] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 12:59:32.760421  3295 leader_election.cc:290] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 3 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:32.741276  3470 raft_consensus.cc:2804] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:59:32.762737  3470 raft_consensus.cc:493] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 12:59:32.763130  3470 raft_consensus.cc:3060] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:32.765434  3461 raft_consensus.cc:2804] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:59:32.765946  3461 raft_consensus.cc:493] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 12:59:32.766358  3461 raft_consensus.cc:3060] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:32.770259  3427 raft_consensus.cc:3060] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:59:32.747159  3486 raft_consensus.cc:2749] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:59:32.771845  1597 raft_consensus.cc:1240] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 2. Current term is 3. Ops: [2.3-2.3]
I20260709 12:59:32.775880  1746 raft_consensus.cc:2468] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 2.
I20260709 12:59:32.747546  3456 raft_consensus.cc:2749] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:59:32.778676  1677 raft_consensus.cc:2468] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 2.
I20260709 12:59:32.779948  3413 raft_consensus.cc:515] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:32.781878  3413 leader_election.cc:290] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 2 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:32.782917  3429 raft_consensus.cc:515] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:32.784876  3468 raft_consensus.cc:2804] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:59:32.785121  3429 leader_election.cc:290] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:32.785362  3468 raft_consensus.cc:493] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 12:59:32.755280  3444 consensus_queue.cc:1048] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.034s
I20260709 12:59:32.742225  3481 raft_consensus.cc:515] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:32.785790  3468 raft_consensus.cc:3060] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:59:32.801487  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 52961.9us
I20260709 12:59:32.802660  1718 leader_election.cc:304] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:32.757378  1745 raft_consensus.cc:2468] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 3.
W20260709 12:59:32.794451  1644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:59:32.805678  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 52052us
I20260709 12:59:32.808782  1718 leader_election.cc:304] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:32.826174  1646 leader_election.cc:304] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 12:59:32.840328  1718 leader_election.cc:304] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 12:59:32.822078  3413 raft_consensus.cc:2749] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:59:32.853968  1646 leader_election.cc:304] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 12:59:32.830646  3486 raft_consensus.cc:2749] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:59:32.806241  3431 raft_consensus.cc:515] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 12:59:32.860322  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 122296us
W20260709 12:59:32.854980  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 111051us
I20260709 12:59:32.858789  1718 leader_election.cc:304] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 12:59:32.809129  3470 raft_consensus.cc:515] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:32.871418  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
W20260709 12:59:32.874308  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 126546us
I20260709 12:59:32.874413  1752 raft_consensus.cc:2468] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 2.
W20260709 12:59:32.888763  1644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:59:32.860731  3416 consensus_meta.cc:287] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.527s	user 0.015s	sys 0.000s
W20260709 12:59:32.916683  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 62776.4us
I20260709 12:59:32.849817  3425 raft_consensus.cc:2749] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 12:59:32.919742  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 61048.6us
W20260709 12:59:32.918067  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 174866us
I20260709 12:59:32.920991  1646 leader_election.cc:304] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 12:59:32.859570  3480 raft_consensus.cc:2749] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:59:32.923125  1718 leader_election.cc:304] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:32.927557  1646 leader_election.cc:304] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 12:59:32.928324  2301 leader_election.cc:304] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:32.934146  1646 leader_election.cc:304] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 12:59:32.942539  2301 leader_election.cc:304] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 12:59:32.942940  1646 leader_election.cc:304] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 12:59:32.922055  3498 raft_consensus.cc:2749] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:59:32.919519  3358 raft_consensus.cc:2749] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:59:32.947423  3499 raft_consensus.cc:2804] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:59:32.947561  3486 raft_consensus.cc:2749] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:59:32.947953  3499 raft_consensus.cc:493] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 12:59:32.948422  3499 raft_consensus.cc:3060] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:32.949632  3444 raft_consensus.cc:493] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 12:59:32.941382  3475 raft_consensus.cc:2749] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:59:32.950228  3444 raft_consensus.cc:515] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:32.951695  3356 raft_consensus.cc:515] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:32.952240  1718 leader_election.cc:304] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:32.953624  1718 leader_election.cc:304] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:32.954340  3358 raft_consensus.cc:2749] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:59:32.954751  3486 raft_consensus.cc:2749] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:59:32.955907  3481 leader_election.cc:290] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:32.957723  3444 leader_election.cc:290] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:32.957780  3470 leader_election.cc:290] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 2 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:32.959952  3431 leader_election.cc:290] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 3 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:32.963655  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:59:32.967772  1676 raft_consensus.cc:2468] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 3.
I20260709 12:59:32.964406  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:59:32.933418  3472 raft_consensus.cc:2749] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:59:32.966984  1718 leader_election.cc:304] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:32.969710  2301 leader_election.cc:304] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 12:59:32.971180  3461 raft_consensus.cc:515] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:32.971465  3472 raft_consensus.cc:493] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 12:59:32.972250  3472 raft_consensus.cc:515] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:32.977809  1718 leader_election.cc:304] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:32.942090  3416 raft_consensus.cc:2749] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:59:32.984669  1646 leader_election.cc:304] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 12:59:32.994369  1718 leader_election.cc:304] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:32.996059  1718 leader_election.cc:304] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 12:59:32.997126  3500 raft_consensus.cc:2749] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:59:32.997226  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:59:32.998172  3358 raft_consensus.cc:2749] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:59:32.969681  3425 consensus_queue.cc:1048] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.033s
I20260709 12:59:33.004468  1571 leader_election.cc:304] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 12:59:33.005371  3496 raft_consensus.cc:3060] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:33.006494  1571 leader_election.cc:304] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 12:59:32.983505  3413 raft_consensus.cc:2749] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:59:33.007128  3431 raft_consensus.cc:2749] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:59:32.988234  3486 raft_consensus.cc:2749] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:59:33.007817  3471 raft_consensus.cc:3060] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:59:33.009876  3472 leader_election.cc:290] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:33.010528  1597 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40154 (request call id 1613) took 1024 ms. Trace:
I20260709 12:59:33.010986  1597 rpcz_store.cc:276] 0709 12:59:31.986418 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:32.220748 (+234330us) service_pool.cc:224] Handling call
0709 12:59:32.229322 (+  8574us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:59:32.772403 (+543081us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:33.010110 (+237707us) spinlock_profiling.cc:243] Waited 534 ms on lock 0x7b540028bc54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 12:59:33.010421 (+   311us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1229153536}
I20260709 12:59:33.013619  1571 leader_election.cc:304] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 12:59:33.014644  3496 raft_consensus.cc:2749] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:59:33.015771  3494 raft_consensus.cc:2749] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 12:59:33.021159  3424 consensus_meta.cc:287] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.665s	user 0.013s	sys 0.000s
I20260709 12:59:32.989594  3444 raft_consensus.cc:2749] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:59:33.022917  1571 leader_election.cc:304] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 12:59:33.026813  3471 raft_consensus.cc:2749] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:59:33.027969  3486 raft_consensus.cc:2749] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:59:33.028959  1718 leader_election.cc:304] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 12:59:33.032310  3486 raft_consensus.cc:2749] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:59:33.032981  3468 raft_consensus.cc:515] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:32.954074  3356 leader_election.cc:290] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:33.034830  1718 leader_election.cc:304] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 12:59:33.036234  3491 raft_consensus.cc:2749] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:59:33.037209  1571 leader_election.cc:304] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 12:59:33.039672  3491 raft_consensus.cc:2749] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:59:33.054904  1718 leader_election.cc:304] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:33.055783  3351 raft_consensus.cc:3055] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:59:33.056861  1571 leader_election.cc:304] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 12:59:33.057420  3356 raft_consensus.cc:2749] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 12:59:33.058724  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 77071.1us
I20260709 12:59:33.067541  1646 leader_election.cc:304] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 12:59:33.068871  1646 leader_election.cc:304] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 12:59:33.070528  3461 leader_election.cc:290] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 2 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:33.070652  3414 raft_consensus.cc:2749] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:59:33.071492  3414 raft_consensus.cc:2749] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:59:33.072320  3268 raft_consensus.cc:2749] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:59:33.070097  1646 leader_election.cc:304] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 12:59:33.073884  3444 raft_consensus.cc:2804] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:59:33.074296  3444 raft_consensus.cc:493] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 12:59:33.074729  3444 raft_consensus.cc:3060] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:59:33.075344  3468 leader_election.cc:290] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:33.076083  3486 raft_consensus.cc:2749] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:59:33.079203  1646 leader_election.cc:304] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 12:59:33.079756  3499 raft_consensus.cc:515] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:33.083451  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:59:33.085317  3414 raft_consensus.cc:2749] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:59:33.065500  3351 raft_consensus.cc:740] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 12:59:33.061863  3495 raft_consensus.cc:3060] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:59:33.090957  3351 consensus_queue.cc:260] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:33.092540  3351 raft_consensus.cc:3060] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:59:33.093937  3499 leader_election.cc:290] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 2 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 12:59:33.062673  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 94766.5us
I20260709 12:59:33.066646  1571 leader_election.cc:304] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 12:59:33.096649  3414 raft_consensus.cc:493] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 12:59:33.097371  3268 raft_consensus.cc:2804] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:59:33.097307  3414 raft_consensus.cc:515] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:33.097959  3268 raft_consensus.cc:697] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 12:59:33.098280  1571 leader_election.cc:304] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 12:59:33.099874  3268 consensus_queue.cc:237] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 12:59:33.060401  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 90786.4us
I20260709 12:59:33.100257  1571 leader_election.cc:304] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 12:59:33.101667  2301 leader_election.cc:304] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 12:59:33.101967  3415 raft_consensus.cc:2749] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:59:33.103434  1571 leader_election.cc:304] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 12:59:33.100332  3414 leader_election.cc:290] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:33.104295  3415 raft_consensus.cc:2749] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:59:33.104367  3499 raft_consensus.cc:2749] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:59:33.100905  3295 raft_consensus.cc:2804] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:59:33.105324  3295 raft_consensus.cc:697] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 12:59:33.106904  3295 consensus_queue.cc:237] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:33.111485  1571 leader_election.cc:304] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:59:33.113003  1571 leader_election.cc:304] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 12:59:33.113790  3495 raft_consensus.cc:2749] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:59:33.114274  3415 raft_consensus.cc:2804] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:59:33.114845  3415 raft_consensus.cc:697] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 12:59:33.115744  3415 consensus_queue.cc:237] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:33.116720  3491 raft_consensus.cc:2749] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260709 12:59:33.118269  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 74952.7us
I20260709 12:59:33.086869  1646 leader_election.cc:304] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 12:59:33.122802  1571 leader_election.cc:304] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 12:59:33.123458  1646 leader_election.cc:304] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 12:59:33.125310  1646 leader_election.cc:304] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 12:59:33.126354  3424 raft_consensus.cc:2749] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:59:33.127295  3424 raft_consensus.cc:2749] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:59:33.127928  3424 raft_consensus.cc:2749] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:59:33.131404  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:33.131690  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:33.132488  1745 raft_consensus.cc:2393] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 2: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 12:59:33.132553  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:33.133416  1754 raft_consensus.cc:2393] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 2: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 12:59:33.118209  3444 raft_consensus.cc:515] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:33.123600  3505 raft_consensus.cc:2804] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:59:33.135294  3505 raft_consensus.cc:493] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 12:59:33.135744  3505 raft_consensus.cc:3060] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:59:33.136865  3444 leader_election.cc:290] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 3 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:33.108201  1569 leader_election.cc:304] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
W20260709 12:59:33.132035  1646 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:59:33.138926  1718 leader_election.cc:304] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:33.138988  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:33.139982  1751 raft_consensus.cc:2393] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 2: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 12:59:33.140664  3468 raft_consensus.cc:2804] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:59:33.141165  3468 raft_consensus.cc:493] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 12:59:33.141590  3468 raft_consensus.cc:3060] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:33.103729  2301 leader_election.cc:304] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 12:59:33.142812  3499 raft_consensus.cc:2749] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260709 12:59:33.147866  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 44236.4us
I20260709 12:59:33.147949  3505 raft_consensus.cc:515] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:33.148828  2301 leader_election.cc:304] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 12:59:33.149667  3499 raft_consensus.cc:2749] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:59:33.134788  1571 leader_election.cc:304] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 12:59:33.150790  3491 raft_consensus.cc:2804] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:59:33.151746  1569 leader_election.cc:304] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 12:59:33.158861  3466 raft_consensus.cc:2804] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:59:33.159291  3466 raft_consensus.cc:493] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Starting leader election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 12:59:33.159642  3466 raft_consensus.cc:3060] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:33.160602  2301 leader_election.cc:304] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:33.161329  3491 raft_consensus.cc:697] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 12:59:33.162367  3491 consensus_queue.cc:237] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:33.166177  3424 raft_consensus.cc:493] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 12:59:33.166740  3424 raft_consensus.cc:515] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:33.168004  2301 leader_election.cc:304] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 12:59:33.169312  3461 raft_consensus.cc:2804] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:59:33.169549  3424 leader_election.cc:290] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:33.167063  3499 raft_consensus.cc:2804] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:59:33.169870  3461 raft_consensus.cc:697] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 2 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 12:59:33.170496  3499 raft_consensus.cc:697] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 3 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 12:59:33.171424  3461 consensus_queue.cc:237] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:33.171591  3499 consensus_queue.cc:237] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:33.173516  1677 raft_consensus.cc:1240] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 2. Current term is 3. Ops: []
I20260709 12:59:33.174808  3466 raft_consensus.cc:515] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 12:59:33.175959  1646 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:59:33.176424  3456 consensus_queue.cc:1059] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:59:33.177006  3424 raft_consensus.cc:493] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 12:59:33.177575  3456 raft_consensus.cc:3055] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:59:33.178004  3456 raft_consensus.cc:740] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 12:59:33.177785  3424 raft_consensus.cc:515] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:33.178926  3471 raft_consensus.cc:2804] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:59:33.179034  3456 consensus_queue.cc:260] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:33.180495  3456 raft_consensus.cc:3060] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:59:33.181633  1672 raft_consensus.cc:3060] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:59:33.182740  3495 raft_consensus.cc:493] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:33.183257  3495 raft_consensus.cc:515] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:33.184056  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:33.185663  3495 leader_election.cc:290] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:33.185792  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:33.187966  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:33.188663  1751 raft_consensus.cc:2393] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 2: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 12:59:33.189580  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:33.191551  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:33.192129  1753 raft_consensus.cc:2468] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 2.
I20260709 12:59:33.192446  1672 raft_consensus.cc:1275] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260709 12:59:33.177055  3466 leader_election.cc:290] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 2 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:33.179543  3471 raft_consensus.cc:697] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 12:59:33.150036  3468 raft_consensus.cc:515] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:33.196974  3471 consensus_queue.cc:237] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:33.199065  3468 leader_election.cc:290] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 2 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:33.199729  3468 raft_consensus.cc:493] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 12:59:33.219295  3468 raft_consensus.cc:515] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:33.203920  1569 leader_election.cc:304] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
W20260709 12:59:33.204661  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 69915.5us
I20260709 12:59:33.150251  3505 leader_election.cc:290] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 5 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:33.207311  1753 raft_consensus.cc:1240] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 4. Current term is 5. Ops: [4.7-4.7]
I20260709 12:59:33.210632  3424 leader_election.cc:290] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:33.227008  1571 leader_election.cc:304] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 12:59:33.228308  1571 leader_election.cc:304] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 12:59:33.229274  3466 raft_consensus.cc:2804] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:59:33.229835  3466 raft_consensus.cc:697] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 12:59:33.211166  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:33.211903  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:33.230772  1747 raft_consensus.cc:3060] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:59:33.230902  3466 consensus_queue.cc:237] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:33.214306  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
W20260709 12:59:33.217729  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 78905.6us
I20260709 12:59:33.234680  3295 raft_consensus.cc:2804] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:59:33.200996  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:33.235342  3295 raft_consensus.cc:697] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 12:59:33.236277  1746 raft_consensus.cc:2393] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 2: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 12:59:33.221472  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:33.236831  3295 consensus_queue.cc:237] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:33.237552  1750 raft_consensus.cc:3060] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:33.241410  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:33.223074  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:33.223862  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:33.225076  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:33.243001  1571 leader_election.cc:304] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 12:59:33.243525  1749 raft_consensus.cc:2468] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 3.
I20260709 12:59:33.244043  1747 raft_consensus.cc:2468] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 3.
I20260709 12:59:33.245270  3486 consensus_queue.cc:1059] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:59:33.245496  3398 consensus_queue.cc:1048] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:59:33.246474  3486 consensus_queue.cc:1059] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:59:33.247215  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:33.244094  3494 raft_consensus.cc:2749] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:59:33.247398  3413 raft_consensus.cc:3055] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:59:33.247571  3425 raft_consensus.cc:3055] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:59:33.248255  3413 raft_consensus.cc:740] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 12:59:33.248343  3425 raft_consensus.cc:740] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 12:59:33.249073  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 12:59:33.249671  3413 consensus_queue.cc:260] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:33.250633  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:33.251377  3413 raft_consensus.cc:3060] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:33.251571  2334 raft_consensus.cc:2393] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 3: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 12:59:33.252911   431 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:59:33.253283  2334 raft_consensus.cc:1240] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 1. Current term is 2. Ops: []
I20260709 12:59:33.249986  3425 consensus_queue.cc:260] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:33.254935  2334 raft_consensus.cc:1240] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 1. Current term is 2. Ops: []
I20260709 12:59:33.256093  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:59:33.256313  1676 raft_consensus.cc:1240] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 1. Current term is 2. Ops: []
I20260709 12:59:33.256470  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:59:33.257196  1571 leader_election.cc:304] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 12:59:33.257963  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 12:59:33.258764  2334 raft_consensus.cc:2393] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 2: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 12:59:33.260231  2334 raft_consensus.cc:1275] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260709 12:59:33.258852  3425 raft_consensus.cc:3060] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:59:33.262694  1644 leader_election.cc:304] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 12:59:33.263206  3415 raft_consensus.cc:2749] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:59:33.264038  3415 raft_consensus.cc:2804] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:59:33.264614  3415 raft_consensus.cc:697] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 12:59:33.265151  3497 raft_consensus.cc:3060] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:33.265651  3415 consensus_queue.cc:237] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:33.247385  2329 raft_consensus.cc:1240] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 1. Current term is 2. Ops: []
I20260709 12:59:33.270434  3505 consensus_queue.cc:1059] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20260709 12:59:33.271351  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:59:33.271577  3505 consensus_queue.cc:1059] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:59:33.271996  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 12:59:33.272764  3505 raft_consensus.cc:3055] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:59:33.273142  3505 raft_consensus.cc:740] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 12:59:33.273976  3505 consensus_queue.cc:260] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:33.275297  3505 raft_consensus.cc:3060] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:33.276515  1752 raft_consensus.cc:1240] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 2. Current term is 3. Ops: []
I20260709 12:59:33.276858  3398 raft_consensus.cc:3055] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:59:33.277258  3398 raft_consensus.cc:740] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 12:59:33.278321  3398 consensus_queue.cc:260] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:33.279821  3398 raft_consensus.cc:3060] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:33.284523  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:59:33.288267  3471 consensus_queue.cc:1048] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:59:33.294538  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:33.295595  1747 raft_consensus.cc:2393] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 2: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 12:59:33.296701  1718 leader_election.cc:304] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:33.297333  3356 consensus_queue.cc:1059] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:59:33.301604  3468 leader_election.cc:290] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 12:59:33.312194  2299 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:59:33.314805  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 57674us
I20260709 12:59:33.315032  2301 leader_election.cc:304] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 12:59:33.332777  1597 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653"
dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
 from {username='slave'} at 127.0.0.1:40166
I20260709 12:59:33.333561  1597 raft_consensus.cc:493] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:59:33.333184  2301 leader_election.cc:304] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 12:59:33.334043  1597 raft_consensus.cc:3060] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:33.333149  3518 ts_tablet_manager.cc:933] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1: Initiating tablet copy from peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:33.335667  2301 leader_election.cc:304] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 12:59:33.316371  2335 raft_consensus.cc:2468] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 3.
I20260709 12:59:33.336597  1604 raft_consensus.cc:1240] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 4. Current term is 5. Ops: []
I20260709 12:59:33.336925  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:33.317855  1747 raft_consensus.cc:1240] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 2. Current term is 3. Ops: []
I20260709 12:59:33.319676  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:59:33.333381  3456 raft_consensus.cc:2749] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:59:33.324680  1750 raft_consensus.cc:2468] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 2.
I20260709 12:59:33.328773  1646 leader_election.cc:304] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
W20260709 12:59:33.361066  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 64462.5us
I20260709 12:59:33.364367  2335 raft_consensus.cc:1240] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 1. Current term is 2. Ops: []
I20260709 12:59:33.365453  1718 leader_election.cc:304] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 12:59:33.368711  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:33.367316  3461 raft_consensus.cc:2749] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:59:33.367261  3358 raft_consensus.cc:2749] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:59:33.404796  3420 consensus_queue.cc:1059] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:59:33.406896  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:33.384891  2301 leader_election.cc:304] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 12:59:33.415726  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:33.359311  3464 raft_consensus.cc:2749] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:59:33.366957  3414 raft_consensus.cc:2804] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:59:33.431437  2301 leader_election.cc:304] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 12:59:33.432250  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:33.433004  2301 leader_election.cc:304] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 12:59:33.418269  3356 raft_consensus.cc:2749] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:59:33.369561  3495 consensus_queue.cc:1059] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.060s
I20260709 12:59:33.369503  3486 consensus_queue.cc:1059] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.060s
I20260709 12:59:33.422676  3461 raft_consensus.cc:2749] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:59:33.445856  3444 raft_consensus.cc:493] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:33.460922  3486 raft_consensus.cc:3055] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:59:33.461689  3486 raft_consensus.cc:740] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 12:59:33.460249  3444 raft_consensus.cc:515] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:33.441360  3462 consensus_queue.cc:1059] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.022s
I20260709 12:59:33.434289  3499 raft_consensus.cc:2749] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:59:33.425658  3414 raft_consensus.cc:493] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:59:33.464244  3462 raft_consensus.cc:3055] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:59:33.465113  3462 raft_consensus.cc:740] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 12:59:33.452944  1501 catalog_manager.cc:5697] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d reported cstate change: term changed from 2 to 3, leader changed from 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) to 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193). New cstate: current_term: 3 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:33.466826  3497 raft_consensus.cc:2749] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:59:33.466643  3462 consensus_queue.cc:260] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:33.464453  3414 raft_consensus.cc:3060] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:59:33.468807  3462 raft_consensus.cc:3060] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:59:33.437726  3431 raft_consensus.cc:2804] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Leader election won for term 3
W20260709 12:59:33.472326  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 106970us
I20260709 12:59:33.470975  2301 leader_election.cc:304] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:33.478998  3431 raft_consensus.cc:697] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 3 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 12:59:33.488966  1644 leader_election.cc:304] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: af520b67945d443b901bbeca37cc5701
I20260709 12:59:33.444540  3495 raft_consensus.cc:3055] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:59:33.497008  3518 tablet_copy_client.cc:323] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Beginning tablet copy session from remote peer at address 127.0.107.195:40693
I20260709 12:59:33.501021  1764 tablet_copy_service.cc:140] P af520b67945d443b901bbeca37cc5701: Received BeginTabletCopySession request for tablet 983621a6fc5b4c9ea0159a648dfe7a36 from peer 3b879799899a499c8d409ee9f1277cd1 ({username='slave'} at 127.0.0.1:43156)
I20260709 12:59:33.501960  1764 tablet_copy_service.cc:161] P af520b67945d443b901bbeca37cc5701: Beginning new tablet copy session on tablet 983621a6fc5b4c9ea0159a648dfe7a36 from peer 3b879799899a499c8d409ee9f1277cd1 at {username='slave'} at 127.0.0.1:43156: session id = 3b879799899a499c8d409ee9f1277cd1-983621a6fc5b4c9ea0159a648dfe7a36
I20260709 12:59:33.507686  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:33.508064  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:33.509634  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:33.511264  1679 raft_consensus.cc:3060] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:33.512882  1747 raft_consensus.cc:1275] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:59:33.513299  1764 tablet_copy_source_session.cc:215] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:59:33.513712  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:33.515108  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:33.494544  3424 consensus_queue.cc:1059] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.021s
I20260709 12:59:33.516060  1752 raft_consensus.cc:2376] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 2. Re-sending same reply.
I20260709 12:59:33.518405  1750 raft_consensus.cc:2393] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 5: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 12:59:33.518954  1679 raft_consensus.cc:1275] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:59:33.525529  1747 raft_consensus.cc:1240] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 1. Current term is 2. Ops: []
I20260709 12:59:33.515064  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:33.481189  3356 raft_consensus.cc:493] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 12:59:33.490087  3499 raft_consensus.cc:2804] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:59:33.539038  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:33.492774  3351 raft_consensus.cc:2804] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:59:33.496076  3495 raft_consensus.cc:740] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 12:59:33.471087  3486 consensus_queue.cc:260] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:33.548569  1748 raft_consensus.cc:1275] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:59:33.546694  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:33.575165  1753 raft_consensus.cc:2468] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 2.
I20260709 12:59:33.488950  3431 consensus_queue.cc:237] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:33.530334  3358 consensus_queue.cc:1059] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.033s
I20260709 12:59:33.555168  3499 raft_consensus.cc:697] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 2 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 12:59:33.578524  3416 raft_consensus.cc:493] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:33.545336  3356 raft_consensus.cc:515] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:33.554359  3495 consensus_queue.cc:260] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 12:59:33.617426  1717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:59:33.556319  3413 consensus_queue.cc:1059] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.039s
W20260709 12:59:33.627708  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 108605us
I20260709 12:59:33.633533  3518 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 983621a6fc5b4c9ea0159a648dfe7a36. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:59:33.628034  3411 raft_consensus.cc:3055] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:59:33.638895  3411 raft_consensus.cc:740] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 12:59:33.595073  1746 raft_consensus.cc:1240] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 1. Current term is 2. Ops: []
I20260709 12:59:33.640019  3411 consensus_queue.cc:260] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:33.641778  3411 raft_consensus.cc:3060] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:59:33.628273  3295 consensus_queue.cc:1048] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.015s
I20260709 12:59:33.628383  3416 raft_consensus.cc:515] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:33.625530  3468 raft_consensus.cc:3055] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:59:33.645057  3468 raft_consensus.cc:740] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 12:59:33.645726  3416 leader_election.cc:290] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:33.646205  3468 consensus_queue.cc:260] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:33.646461  1502 catalog_manager.cc:5697] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 reported cstate change: term changed from 1 to 2, leader changed from af520b67945d443b901bbeca37cc5701 (127.0.107.195) to 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196). New cstate: current_term: 2 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:59:33.652829  1677 raft_consensus.cc:1275] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:59:33.657142  1502 catalog_manager.cc:5697] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 reported cstate change: term changed from 2 to 3, leader changed from af520b67945d443b901bbeca37cc5701 (127.0.107.195) to 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196). New cstate: current_term: 3 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:59:33.654673  3499 consensus_queue.cc:237] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:33.688439  3444 leader_election.cc:290] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:33.679827  3351 raft_consensus.cc:697] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 12:59:33.692117  3351 consensus_queue.cc:237] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:33.664413  3461 consensus_queue.cc:1048] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.033s
I20260709 12:59:33.698800  3456 raft_consensus.cc:493] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 12:59:33.699676  3456 raft_consensus.cc:515] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:33.704314  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:33.704602  3356 leader_election.cc:290] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:33.680372  3495 raft_consensus.cc:3060] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:59:33.725278  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:33.726267  2334 raft_consensus.cc:2468] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 1.
W20260709 12:59:33.729125  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 193081us
I20260709 12:59:33.735813  2331 raft_consensus.cc:1275] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:59:33.747916  1718 leader_election.cc:304] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:59:33.757682  2329 raft_consensus.cc:1240] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 1. Current term is 2. Ops: []
I20260709 12:59:33.757218  3414 raft_consensus.cc:515] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:33.759122  1672 raft_consensus.cc:1240] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 3. Current term is 4. Ops: []
I20260709 12:59:33.748196  1597 raft_consensus.cc:515] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:33.761808  3414 leader_election.cc:290] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 4 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:33.765352  3486 raft_consensus.cc:3060] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:33.767938  1754 raft_consensus.cc:1275] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260709 12:59:33.772145  1600 raft_consensus.cc:1240] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 1. Current term is 2. Ops: []
I20260709 12:59:33.773672  1752 raft_consensus.cc:1275] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
W20260709 12:59:33.780897  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 148659us
I20260709 12:59:33.769063  1597 leader_election.cc:290] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 2 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:33.771450  3358 raft_consensus.cc:2804] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:59:33.784758  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:33.789124  1753 raft_consensus.cc:3060] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:59:33.796064  1643 leader_election.cc:304] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:33.806972  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:33.809734  2334 raft_consensus.cc:2468] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 3.
I20260709 12:59:33.779502  3414 raft_consensus.cc:493] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:33.811743  1753 raft_consensus.cc:2468] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 4.
I20260709 12:59:33.805685  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:33.813874  2335 raft_consensus.cc:2468] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 3.
I20260709 12:59:33.816712  1644 leader_election.cc:304] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 12:59:33.820992  2329 raft_consensus.cc:1240] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 3 LEADER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 2. Current term is 3. Ops: []
I20260709 12:59:33.822940  3525 raft_consensus.cc:2749] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:59:33.825273  3531 consensus_queue.cc:1048] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:59:33.826565  3456 leader_election.cc:290] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:33.796248  3358 raft_consensus.cc:493] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 12:59:33.804344  3527 raft_consensus.cc:493] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:33.835690  1644 leader_election.cc:304] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 12:59:33.838251  3358 raft_consensus.cc:3060] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:59:33.854833  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 57404.3us
I20260709 12:59:33.855862  1643 leader_election.cc:304] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 12:59:33.843595  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:59:33.856760  1672 raft_consensus.cc:3055] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:59:33.857152  1672 raft_consensus.cc:740] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 12:59:33.859714  1646 leader_election.cc:304] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:59:33.861709  1643 leader_election.cc:304] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 12:59:33.845893  3416 raft_consensus.cc:2804] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:59:33.815299  3351 consensus_queue.cc:1059] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.044s
I20260709 12:59:33.865437  1600 raft_consensus.cc:1240] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 2. Current term is 3. Ops: []
I20260709 12:59:33.840581  3425 raft_consensus.cc:3055] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:59:33.843416  3456 raft_consensus.cc:493] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 12:59:33.838928  3468 raft_consensus.cc:3060] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:33.858225  1672 consensus_queue.cc:260] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:33.882119  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:59:33.868207  1598 raft_consensus.cc:1240] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 1. Current term is 2. Ops: []
I20260709 12:59:33.870704  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:59:33.888806  3456 raft_consensus.cc:515] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:33.890761  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:59:33.861805  3444 raft_consensus.cc:2749] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:59:33.856967  3527 raft_consensus.cc:515] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:33.892177  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:33.893738  3456 leader_election.cc:290] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:33.890089  3544 consensus_queue.cc:1048] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.005s
I20260709 12:59:33.894986  3527 leader_election.cc:290] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:33.872601  3537 raft_consensus.cc:2804] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:59:33.876546  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:33.828321  3414 raft_consensus.cc:515] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:33.898458  1596 raft_consensus.cc:2468] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 2.
I20260709 12:59:33.859622  3424 raft_consensus.cc:2804] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:59:33.882850  1672 raft_consensus.cc:3060] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:33.884320  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:33.899823  1571 leader_election.cc:304] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 12:59:33.885021  1754 raft_consensus.cc:1275] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:59:33.863855  3416 raft_consensus.cc:493] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:59:33.879529  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 69787.7us
I20260709 12:59:33.900411  1752 raft_consensus.cc:3060] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:33.902097  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:33.878223  3461 consensus_queue.cc:1048] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.037s
I20260709 12:59:33.879083  3462 consensus_queue.cc:1048] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.048s
I20260709 12:59:33.874564  3425 raft_consensus.cc:740] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 12:59:33.941321  3295 raft_consensus.cc:3060] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:59:33.944507  3494 consensus_queue.cc:1059] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
I20260709 12:59:33.947994  3494 raft_consensus.cc:3055] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:59:33.947356  3425 consensus_queue.cc:260] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } }
I20260709 12:59:33.948510  3494 raft_consensus.cc:740] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 12:59:33.949505  3425 raft_consensus.cc:3060] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:59:33.914825  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 12:59:33.901589  3497 consensus_queue.cc:1175] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "af520b67945d443b901bbeca37cc5701" responder_term: 1 status { last_received { term: 1 index: 1 } last_committed_idx: 1 last_received_current_leader { term: 1 index: 1 } }
I20260709 12:59:33.950842  2331 raft_consensus.cc:2468] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 3.
I20260709 12:59:33.901292  3424 raft_consensus.cc:493] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:59:33.951946  3424 raft_consensus.cc:3060] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:59:33.910166  3414 leader_election.cc:290] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:33.915501  3527 raft_consensus.cc:3060] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:33.953600  1571 leader_election.cc:304] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 12:59:33.956422  1644 leader_election.cc:304] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 12:59:33.915190  1746 raft_consensus.cc:1240] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 4. Current term is 5. Ops: []
W20260709 12:59:33.962620  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 99489.4us
I20260709 12:59:33.968406  1752 raft_consensus.cc:2468] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 2.
W20260709 12:59:33.970608  1645 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:59:33.970760  1672 raft_consensus.cc:2468] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 2.
W20260709 12:59:33.986189  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 78566.7us
I20260709 12:59:33.987310  1571 leader_election.cc:304] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
W20260709 12:59:33.990823  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 129164us
W20260709 12:59:33.990859  1717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:59:33.992017  1643 leader_election.cc:304] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 12:59:33.910238  3416 raft_consensus.cc:3060] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:59:33.911003  3538 raft_consensus.cc:493] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:33.998682  1571 leader_election.cc:304] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 12:59:33.915223  3444 raft_consensus.cc:3055] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:59:34.006553  1643 leader_election.cc:304] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:34.008165  1643 leader_election.cc:304] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 12:59:33.929821  3537 raft_consensus.cc:697] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 12:59:34.009548  1569 leader_election.cc:304] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 12:59:33.965190  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:33.993377  3501 raft_consensus.cc:2749] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:59:34.007613  3351 raft_consensus.cc:2749] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:59:33.999738  3415 raft_consensus.cc:2749] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:59:34.023188  3537 consensus_queue.cc:237] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:34.016144  1673 raft_consensus.cc:1275] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:59:34.026292  1603 raft_consensus.cc:1275] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260709 12:59:33.949744  3494 consensus_queue.cc:260] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:34.028133  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:34.028787  3494 raft_consensus.cc:3060] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:34.031329  3547 raft_consensus.cc:2749] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:59:34.032548  3547 raft_consensus.cc:3060] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:59:34.033560  1719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:59:34.021900  3544 raft_consensus.cc:2804] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:59:34.005761  3495 raft_consensus.cc:2804] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:59:34.033437  3431 consensus_queue.cc:1048] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.004s
I20260709 12:59:34.005089  3444 raft_consensus.cc:740] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 12:59:33.995114  3471 consensus_queue.cc:1048] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.024s
I20260709 12:59:34.034740  3398 raft_consensus.cc:2804] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:59:34.064327  3398 raft_consensus.cc:697] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 12:59:34.060936  3466 consensus_queue.cc:1048] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:59:34.074502  1672 raft_consensus.cc:1275] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:59:34.062638  3472 raft_consensus.cc:3055] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:59:34.045291  3268 raft_consensus.cc:493] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 12:59:34.071239  1717 leader_election.cc:304] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:34.080895  3486 raft_consensus.cc:2749] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:59:34.013777  3496 consensus_queue.cc:1059] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.039s
I20260709 12:59:34.045231  3495 raft_consensus.cc:493] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:59:34.082785  3495 raft_consensus.cc:3060] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:59:34.019119  1746 raft_consensus.cc:2468] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 2.
I20260709 12:59:34.084510  1717 leader_election.cc:304] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:34.086026  1717 leader_election.cc:304] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:34.087489  1717 leader_election.cc:304] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:34.008111  3538 raft_consensus.cc:515] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:34.100653  3496 consensus_queue.cc:1059] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:59:34.102487  3486 raft_consensus.cc:2749] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:59:34.078191  3472 raft_consensus.cc:740] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 12:59:34.103358  3538 leader_election.cc:290] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:34.091722  3535 raft_consensus.cc:2749] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:59:34.104110  3472 consensus_queue.cc:260] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:34.065917  3398 consensus_queue.cc:237] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:34.094487  1754 raft_consensus.cc:1240] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260709 12:59:34.094074  3268 raft_consensus.cc:515] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:34.052769  3544 raft_consensus.cc:697] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 12:59:34.098680  3409 raft_consensus.cc:493] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 12:59:34.047973  3529 consensus_queue.cc:1059] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.079s
I20260709 12:59:34.092760  2329 raft_consensus.cc:3060] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:59:34.107367  1646 leader_election.cc:304] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:59:34.106415  3472 raft_consensus.cc:3060] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:34.135522  3495 raft_consensus.cc:515] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:34.159710  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 12:59:34.161401  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 103746us
W20260709 12:59:34.166766  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 59505.2us
I20260709 12:59:34.180267  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:34.134385  1749 raft_consensus.cc:1275] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:59:34.132046  3268 leader_election.cc:290] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:34.114977  3505 raft_consensus.cc:3055] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:59:34.106782  3356 raft_consensus.cc:2749] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:59:34.111006  3552 raft_consensus.cc:3055] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:59:34.188097  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:34.196501  2334 raft_consensus.cc:2468] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 3.
I20260709 12:59:34.176530  3295 raft_consensus.cc:2749] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:59:34.197916  1569 leader_election.cc:304] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 12:59:34.206990  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:34.173722  3420 raft_consensus.cc:493] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 12:59:34.209087  1677 raft_consensus.cc:2393] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 3: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 12:59:34.155318  3409 raft_consensus.cc:515] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:34.199216  3552 raft_consensus.cc:740] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 12:59:34.195950  3505 raft_consensus.cc:740] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 12:59:34.132242  3444 consensus_queue.cc:260] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:34.236780  1677 raft_consensus.cc:1275] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:59:34.206820  3521 raft_consensus.cc:493] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:34.234911  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:34.237960  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:59:34.239135  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:34.193933  3358 raft_consensus.cc:515] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:34.259523  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:34.213114  3553 raft_consensus.cc:493] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:34.229447  1679 raft_consensus.cc:2955] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } } }
I20260709 12:59:34.263399  1603 raft_consensus.cc:2468] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 3.
I20260709 12:59:34.232544  3499 raft_consensus.cc:493] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:34.268405  1680 raft_consensus.cc:2368] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 3. Current term is 4.
I20260709 12:59:34.258800  3544 consensus_queue.cc:237] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:34.271473  1747 raft_consensus.cc:1275] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:59:34.277091  3528 raft_consensus.cc:2804] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:59:34.233690  3531 raft_consensus.cc:2804] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:59:34.222950  3420 raft_consensus.cc:515] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:34.263957  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:34.282487  3528 raft_consensus.cc:493] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:59:34.307174  3528 raft_consensus.cc:3060] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:59:34.262706  3466 consensus_queue.cc:1059] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.026s
W20260709 12:59:34.319829  3557 log.cc:927] Time spent T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.051s	user 0.001s	sys 0.000s
I20260709 12:59:34.322611  1680 raft_consensus.cc:1275] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260709 12:59:34.282552  3499 raft_consensus.cc:515] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:34.326840  3495 leader_election.cc:290] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 4 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:34.286154  3356 raft_consensus.cc:493] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:34.330448  2335 raft_consensus.cc:1275] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:59:34.332090  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 12:59:34.285890  3531 raft_consensus.cc:493] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 12:59:34.339145  3462 consensus_queue.cc:1048] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.024s
W20260709 12:59:34.366935  1718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:59:34.367219  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 139764us
I20260709 12:59:34.266996  3521 raft_consensus.cc:515] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:34.328049  3358 leader_election.cc:290] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 2 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:34.358413  3531 raft_consensus.cc:3060] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:59:34.269331  3553 raft_consensus.cc:515] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:34.391486  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:34.392570  1680 raft_consensus.cc:2468] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 2.
I20260709 12:59:34.392971  1754 raft_consensus.cc:1275] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:59:34.371042  3499 leader_election.cc:290] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:34.395666  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:59:34.328835  3505 consensus_queue.cc:260] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 12:59:34.399748  3561 log.cc:927] Time spent T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.071s	user 0.000s	sys 0.002s
I20260709 12:59:34.357784  3356 raft_consensus.cc:515] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:34.405532  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:34.399103  3521 leader_election.cc:290] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:34.406666  1752 raft_consensus.cc:1275] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:59:34.407150  1747 raft_consensus.cc:1275] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:59:34.448654  1745 raft_consensus.cc:1275] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
W20260709 12:59:34.450989  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:59:34.456053  3559 log.cc:927] Time spent T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.084s	user 0.001s	sys 0.000s
I20260709 12:59:34.481464  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:34.481500  2299 leader_election.cc:304] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:34.482324  1673 raft_consensus.cc:2468] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 3.
I20260709 12:59:34.466564  3409 leader_election.cc:290] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:34.481120  3505 raft_consensus.cc:3060] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:59:34.487260  3424 consensus_meta.cc:287] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.534s	user 0.011s	sys 0.007s
I20260709 12:59:34.488147  3431 raft_consensus.cc:2749] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:59:34.478252  3521 raft_consensus.cc:493] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:34.489181  3521 raft_consensus.cc:515] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:34.491616  3521 leader_election.cc:290] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:34.487809  3424 raft_consensus.cc:515] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 12:59:34.497767  3527 consensus_meta.cc:287] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.504s	user 0.008s	sys 0.004s
I20260709 12:59:34.497913  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 12:59:34.498988  2332 raft_consensus.cc:3060] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:59:34.484583  2299 leader_election.cc:304] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:34.498255  3527 raft_consensus.cc:2749] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:59:34.510995  2336 raft_consensus.cc:1275] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260709 12:59:34.511422  3549 raft_consensus.cc:3055] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:59:34.512158  3420 leader_election.cc:290] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:34.518625  3518 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.0.0.1:39738 (request call id 525) took 1233 ms. Trace:
I20260709 12:59:34.527686  3518 rpcz_store.cc:276] 0709 12:59:33.271110 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:33.271774 (+   664us) service_pool.cc:224] Handling call
0709 12:59:33.336008 (+ 64234us) ts_tablet_manager.cc:934] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1: Initiating tablet copy from peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
0709 12:59:33.585414 (+249406us) tablet_copy_client.cc:368] Tablet copy session begun
0709 12:59:34.032037 (+446623us) tablet_metadata.cc:665] Metadata flushed
0709 12:59:34.032285 (+   248us) tablet_copy_client.cc:497] Wrote new tablet metadata
0709 12:59:34.493665 (+461380us) tablet_copy_client.cc:418] Wrote new consensus metadata
0709 12:59:34.494046 (+   381us) tablet_replica.cc:206] Creating consensus instance
0709 12:59:34.499680 (+  5634us) ts_tablet_manager.cc:989] Replica registered.
0709 12:59:34.504398 (+  4718us) inbound_call.cc:177] Queueing success response
Metrics: {"dns_us":69084,"tablet-copy.queue_time_us":60556,"thread_start_us":150798,"threads_started":2}
I20260709 12:59:34.529428  3518 tablet_copy_client.cc:806] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Starting download of 0 data blocks...
I20260709 12:59:34.498904  3494 consensus_queue.cc:1048] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.033s
I20260709 12:59:34.503808  3398 consensus_queue.cc:1048] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.030s
W20260709 12:59:34.518952  3416 consensus_meta.cc:287] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.514s	user 0.015s	sys 0.000s
I20260709 12:59:34.535172  3416 raft_consensus.cc:515] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:34.543159  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:34.544200  2328 raft_consensus.cc:2468] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 2.
I20260709 12:59:34.528570  3356 leader_election.cc:290] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:34.546983  3420 raft_consensus.cc:493] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:34.547880  3420 raft_consensus.cc:515] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:34.516012  3549 raft_consensus.cc:740] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
W20260709 12:59:34.552016  3567 log.cc:927] Time spent T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.110s	user 0.000s	sys 0.002s
I20260709 12:59:34.551757  3549 consensus_queue.cc:260] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:34.546093  1718 leader_election.cc:304] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:59:34.553663  3549 raft_consensus.cc:3060] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:34.561400  3356 raft_consensus.cc:2804] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:59:34.562397  1674 raft_consensus.cc:1275] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:59:34.562848  3356 raft_consensus.cc:493] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:59:34.563417  3356 raft_consensus.cc:3060] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:59:34.530350  3518 tablet_copy_client.cc:670] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Starting download of 1 WAL segments...
I20260709 12:59:34.578820  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:34.579633  1676 raft_consensus.cc:2468] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 3.
I20260709 12:59:34.581604  1718 leader_election.cc:304] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:59:34.586413  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:34.588006  2299 leader_election.cc:304] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 12:59:34.587179  1673 raft_consensus.cc:2468] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 3.
I20260709 12:59:34.591897  1569 leader_election.cc:400] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 3 pre-election: Vote denied by peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261) with higher term. Message: Invalid argument: T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 3. Current term is 4.
I20260709 12:59:34.592043  3529 raft_consensus.cc:493] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:34.592813  3529 raft_consensus.cc:515] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:34.594611  2299 leader_election.cc:304] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 12:59:34.596199  2299 leader_election.cc:304] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 12:59:34.598326  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:34.599155  1679 raft_consensus.cc:2468] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 2.
I20260709 12:59:34.600749  1718 leader_election.cc:304] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:59:34.601703  3358 raft_consensus.cc:2804] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:59:34.602195  3358 raft_consensus.cc:493] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:59:34.602663  3358 raft_consensus.cc:3060] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:59:34.593936  3496 raft_consensus.cc:493] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:34.610404  3496 raft_consensus.cc:515] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:34.611487  3529 leader_election.cc:290] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:34.612618  3496 leader_election.cc:290] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:34.613047  1680 raft_consensus.cc:1275] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260709 12:59:34.587669  3553 leader_election.cc:290] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:34.590926  3521 raft_consensus.cc:2804] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:59:34.630379  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:34.631446  1680 raft_consensus.cc:2468] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 3.
W20260709 12:59:34.636770  3547 consensus_meta.cc:287] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.604s	user 0.011s	sys 0.007s
I20260709 12:59:34.637499  3547 raft_consensus.cc:2749] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:59:34.609386  3431 raft_consensus.cc:2804] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:59:34.646291  1678 raft_consensus.cc:1275] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:59:34.598006  3499 consensus_queue.cc:1048] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.048s
I20260709 12:59:34.656476  3420 leader_election.cc:290] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:34.659883  3552 consensus_queue.cc:260] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:34.661430  3552 raft_consensus.cc:3060] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:59:34.625519  3521 raft_consensus.cc:493] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:59:34.663127  3521 raft_consensus.cc:3060] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:59:34.668061  1676 raft_consensus.cc:1240] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 LEADER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260709 12:59:34.668517  2332 raft_consensus.cc:2468] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 4.
I20260709 12:59:34.623540  3542 raft_consensus.cc:2804] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:59:34.624827  3504 raft_consensus.cc:2804] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:59:34.672688  3544 raft_consensus.cc:493] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:34.613840  3415 consensus_queue.cc:1048] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.051s
I20260709 12:59:34.649184  3431 raft_consensus.cc:493] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 12:59:34.623651  3558 consensus_queue.cc:1048] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.081s
I20260709 12:59:34.699342  3431 raft_consensus.cc:3060] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:59:34.704134  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:59:34.704815  1596 raft_consensus.cc:3060] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:59:34.704810  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:34.705900  1598 raft_consensus.cc:2468] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 3.
I20260709 12:59:34.676215  3542 raft_consensus.cc:493] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:59:34.675501  3504 raft_consensus.cc:493] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 12:59:34.686280  3495 raft_consensus.cc:493] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:34.717025  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:59:34.696911  2329 raft_consensus.cc:1275] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260709 12:59:34.718964  1598 raft_consensus.cc:3060] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Advancing to term 4
I20260709 12:59:34.718227  3504 raft_consensus.cc:3060] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:59:34.727376  1717 leader_election.cc:304] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:34.714875  3542 raft_consensus.cc:3060] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:59:34.685127  3544 raft_consensus.cc:515] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:34.728883  1717 leader_election.cc:304] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:34.729276  1643 leader_election.cc:304] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 12:59:34.715214  1569 leader_election.cc:304] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 12:59:34.714030  3425 raft_consensus.cc:493] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 12:59:34.726559  3466 consensus_queue.cc:1048] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.010s
W20260709 12:59:34.737002  3570 log.cc:927] Time spent T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.062s	user 0.000s	sys 0.001s
I20260709 12:59:34.725241  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:59:34.728092  3409 raft_consensus.cc:2749] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:59:34.751420  3531 raft_consensus.cc:515] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:34.755023  1604 raft_consensus.cc:2468] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 3.
I20260709 12:59:34.732497  3468 raft_consensus.cc:2749] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:59:34.718387  3358 raft_consensus.cc:515] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:34.769436  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:34.771021  3529 raft_consensus.cc:493] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 12:59:34.768483  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:34.772078  3558 raft_consensus.cc:2804] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:59:34.774362  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:34.776818  1598 raft_consensus.cc:2468] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 4.
I20260709 12:59:34.723409  3495 raft_consensus.cc:515] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:34.746430  3547 raft_consensus.cc:2804] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:59:34.786760  1753 raft_consensus.cc:1275] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:59:34.788825  1745 raft_consensus.cc:1275] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
W20260709 12:59:34.791427  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 62635.4us
I20260709 12:59:34.779409  3268 raft_consensus.cc:493] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:34.754918  3425 raft_consensus.cc:515] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:34.784808  3558 raft_consensus.cc:493] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:59:34.791467  3547 raft_consensus.cc:493] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:59:34.785971  3531 leader_election.cc:290] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 3 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:34.793025  3468 raft_consensus.cc:2749] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:59:34.819965  1502 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:43476 (request call id 190) took 1330 ms. Trace:
I20260709 12:59:34.499753   431 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:59:34.824747  1500 catalog_manager.cc:5697] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 1 to 2, leader changed from af520b67945d443b901bbeca37cc5701 (127.0.107.195) to 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193). New cstate: current_term: 2 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } }
I20260709 12:59:34.828838  1502 rpcz_store.cc:276] 0709 12:59:33.489305 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:33.489473 (+   168us) service_pool.cc:224] Handling call
0709 12:59:34.819828 (+1330355us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:59:33.896722 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:59:33.933373 (+ 36651us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:59:33.933439 (+    66us) write_op.cc:435] Acquired schema lock
0709 12:59:33.933456 (+    17us) tablet.cc:662] Decoding operations
0709 12:59:33.934261 (+   805us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:59:33.934337 (+    76us) write_op.cc:641] Partition lock acquired for write op
0709 12:59:33.934398 (+    61us) tablet.cc:685] Acquiring locks for 2 operations
0709 12:59:33.935118 (+   720us) tablet.cc:701] Row locks acquired
0709 12:59:33.935159 (+    41us) write_op.cc:260] PREPARE: finished
0709 12:59:33.935428 (+   269us) write_op.cc:270] Start()
0709 12:59:33.935618 (+   190us) write_op.cc:276] Timestamp: P: 1783601973935349 usec, L: 0
0709 12:59:33.935643 (+    25us) op_driver.cc:348] REPLICATION: starting
0709 12:59:33.937026 (+  1383us) log.cc:844] Serialized 919 byte log entry
0709 12:59:34.213819 (+276793us) op_driver.cc:464] REPLICATION: finished
0709 12:59:34.391347 (+177528us) write_op.cc:301] APPLY: starting
0709 12:59:34.391472 (+   125us) tablet.cc:1367] starting BulkCheckPresence
0709 12:59:34.391658 (+   186us) tablet.cc:1370] finished BulkCheckPresence
0709 12:59:34.391917 (+   259us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:59:34.394790 (+  2873us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:59:34.396527 (+  1737us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0709 12:59:34.396610 (+    83us) write_op.cc:312] APPLY: finished
0709 12:59:34.397046 (+   436us) log.cc:844] Serialized 29 byte log entry
0709 12:59:34.436459 (+ 39413us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:59:34.442139 (+  5680us) write_op.cc:454] Released schema lock
0709 12:59:34.449771 (+  7632us) write_op.cc:341] FINISH: Updating metrics
0709 12:59:34.463725 (+ 13954us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:59:34.463790 (+    65us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":2,"child_traces":[["op",{"apply.queue_time_us":173286,"apply.run_cpu_time_us":4276,"apply.run_wall_time_us":72292,"num_ops":2,"prepare.queue_time_us":197975,"prepare.run_cpu_time_us":5296,"prepare.run_wall_time_us":57921,"replication_time_us":265768,"thread_start_us":141016,"threads_started":1}]]}
I20260709 12:59:34.789426  3529 raft_consensus.cc:515] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:34.811825  3547 raft_consensus.cc:3060] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:59:34.811230  3558 raft_consensus.cc:3060] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:59:34.813746  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:34.842887  3575 raft_consensus.cc:493] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:34.864787  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
W20260709 12:59:34.867357  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 86073.7us
I20260709 12:59:34.809434  3268 raft_consensus.cc:515] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:34.892652  1675 raft_consensus.cc:3060] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:59:34.898864  3578 consensus_queue.cc:1048] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.018s
I20260709 12:59:34.908124  3529 leader_election.cc:290] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:34.866509  3424 leader_election.cc:290] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 4 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:34.907951  3584 raft_consensus.cc:493] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:34.893350  3580 raft_consensus.cc:493] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:34.856320  3468 raft_consensus.cc:493] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:34.843346  3358 leader_election.cc:290] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:34.813191  3356 raft_consensus.cc:515] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:34.910213  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:34.914036  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:34.844919  3531 consensus_queue.cc:1048] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.062s
I20260709 12:59:34.880673  3546 raft_consensus.cc:493] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:34.931496  3580 raft_consensus.cc:515] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:34.931885  3584 raft_consensus.cc:515] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:34.881882  3575 raft_consensus.cc:515] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:34.909708  3461 raft_consensus.cc:493] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:34.907497  3576 raft_consensus.cc:493] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:34.935087  1600 raft_consensus.cc:1275] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:59:34.952111  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:34.978407  2335 raft_consensus.cc:2393] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 3: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 12:59:34.953562  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:34.898245  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:34.953938  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 12:59:34.938302  3528 consensus_meta.cc:287] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.631s	user 0.016s	sys 0.000s
I20260709 12:59:34.963990  3544 leader_election.cc:290] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 12:59:34.972640  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 257518us
I20260709 12:59:34.988657  1718 leader_election.cc:304] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 12:59:34.975219  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:34.990589  1753 raft_consensus.cc:2468] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 2.
W20260709 12:59:34.982929  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 63973.8us
I20260709 12:59:34.960983  1596 raft_consensus.cc:2468] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 4.
I20260709 12:59:34.946494  3358 raft_consensus.cc:493] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 12:59:34.940119  3425 leader_election.cc:290] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:35.002233  1644 leader_election.cc:304] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:34.945851  3580 leader_election.cc:290] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:34.949003  3496 consensus_queue.cc:1048] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.050s
I20260709 12:59:35.003774  1597 raft_consensus.cc:2393] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 3: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 12:59:35.009347  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:59:34.934001  3468 raft_consensus.cc:515] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:35.004164  3358 raft_consensus.cc:515] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:35.017424  1677 raft_consensus.cc:1275] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:59:35.017099  3556 consensus_queue.cc:1048] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.005s
I20260709 12:59:34.979000  3420 consensus_queue.cc:1048] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.034s
I20260709 12:59:34.999756  3424 raft_consensus.cc:493] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:35.001564  3351 raft_consensus.cc:493] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 12:59:35.003949  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:34.995085  3578 consensus_queue.cc:1048] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.027s
I20260709 12:59:35.025988  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:35.002113  3495 leader_election.cc:290] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:34.991178  3461 raft_consensus.cc:515] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:35.027031  3424 raft_consensus.cc:515] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:35.031124  3582 raft_consensus.cc:493] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:35.014053  3580 raft_consensus.cc:493] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 12:59:34.941664  3546 raft_consensus.cc:515] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:35.009703  3425 raft_consensus.cc:493] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 12:59:34.995949  3528 raft_consensus.cc:515] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 12:59:35.042394  3577 log.cc:927] Time spent T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.107s	user 0.001s	sys 0.000s
I20260709 12:59:35.043543  3358 leader_election.cc:290] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:35.076936  1643 leader_election.cc:304] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 12:59:35.043985  3584 leader_election.cc:290] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:35.046667  3532 raft_consensus.cc:2804] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:59:35.048066  3356 leader_election.cc:290] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:35.056808  1673 raft_consensus.cc:2468] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 3.
I20260709 12:59:35.007618  3548 consensus_queue.cc:1048] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.051s
I20260709 12:59:35.075848  1671 raft_consensus.cc:1240] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260709 12:59:35.079792  3358 raft_consensus.cc:3060] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:59:34.991019  3576 raft_consensus.cc:515] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:35.081822  3444 raft_consensus.cc:3060] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:59:35.081948  3468 leader_election.cc:290] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:35.054282  3351 raft_consensus.cc:515] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:35.084734  3521 raft_consensus.cc:515] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:35.020881  3544 consensus_queue.cc:1048] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.047s
I20260709 12:59:35.075814  3582 raft_consensus.cc:515] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:35.093683  1643 leader_election.cc:304] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 12:59:35.075215  3528 leader_election.cc:290] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 3 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:35.095952  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:35.119617  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:59:35.097203  3525 raft_consensus.cc:2804] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:59:35.100309  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:35.110460  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 12:59:35.111846  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 123259us
I20260709 12:59:35.114017  1750 raft_consensus.cc:1240] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260709 12:59:35.125803  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:35.126087  3594 raft_consensus.cc:493] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:35.126824  3594 raft_consensus.cc:515] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:35.133024  3351 leader_election.cc:290] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:35.123291  3553 raft_consensus.cc:2804] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:59:35.116442  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:59:35.134902  3579 consensus_queue.cc:1048] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.066s
I20260709 12:59:35.136921  3420 raft_consensus.cc:493] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:35.137629  3521 leader_election.cc:290] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 4 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:35.137560  3420 raft_consensus.cc:515] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:35.071753  3580 raft_consensus.cc:515] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:35.143131  3582 leader_election.cc:290] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:35.118993  3468 raft_consensus.cc:493] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 12:59:35.145256  3580 leader_election.cc:290] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:35.144891  3468 raft_consensus.cc:515] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:35.096318  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:59:35.108165  3409 raft_consensus.cc:493] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:35.147344  1500 catalog_manager.cc:5697] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 2 to 3, leader changed from af520b67945d443b901bbeca37cc5701 (127.0.107.195) to 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193). New cstate: current_term: 3 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } }
I20260709 12:59:35.134625  3553 raft_consensus.cc:697] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 12:59:35.117718  1751 raft_consensus.cc:1240] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 2. Current term is 3. Ops: [2.5-2.5]
I20260709 12:59:35.096486  3425 raft_consensus.cc:515] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:35.148530  1748 raft_consensus.cc:1240] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260709 12:59:35.149259  1752 raft_consensus.cc:2468] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 3.
I20260709 12:59:35.273603  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:35.293617  1598 raft_consensus.cc:2468] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 2.
I20260709 12:59:35.237383  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:35.297056  1753 raft_consensus.cc:2468] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 2.
I20260709 12:59:35.149631  1605 raft_consensus.cc:2410] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 12:59:35.151302  1501 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35458 (request call id 303) took 1762 ms. Trace:
I20260709 12:59:35.151669  1750 raft_consensus.cc:2468] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 2.
I20260709 12:59:35.248781  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:35.148461  1599 raft_consensus.cc:1275] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 3. (term mismatch)
I20260709 12:59:35.302184  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:35.304345  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 12:59:35.304934  2334 raft_consensus.cc:3060] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:59:35.306780  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:35.307446  2335 raft_consensus.cc:2468] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 2.
I20260709 12:59:35.308676  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:35.309631  1599 pending_rounds.cc:85] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a: Aborting all ops after (but not including) 2
I20260709 12:59:35.310081  1599 pending_rounds.cc:107] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a: Aborting uncommitted NO_OP operation due to leader change: 3.3
I20260709 12:59:35.310457  1599 raft_consensus.cc:2889] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:59:35.111749  3588 raft_consensus.cc:493] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 12:59:35.115892  3532 raft_consensus.cc:697] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 12:59:35.311826  1604 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40166 (request call id 1668) took 1054 ms. Trace:
I20260709 12:59:35.313295  1718 leader_election.cc:304] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
W20260709 12:59:35.246423  3431 consensus_meta.cc:287] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.546s	user 0.013s	sys 0.002s
I20260709 12:59:35.325212  3416 leader_election.cc:290] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 4 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 12:59:35.243548  3504 consensus_meta.cc:287] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.513s	user 0.000s	sys 0.013s
I20260709 12:59:35.324677  3547 raft_consensus.cc:515] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:35.319713  1604 rpcz_store.cc:276] 0709 12:59:34.220749 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:34.220908 (+   159us) service_pool.cc:224] Handling call
0709 12:59:35.271046 (+1050138us) spinlock_profiling.cc:243] Waited 493 ms on lock 0x7b54002a1cd4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 12:59:35.275524 (+  4478us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1134680832}
W20260709 12:59:35.242081  3542 consensus_meta.cc:287] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.508s	user 0.012s	sys 0.001s
W20260709 12:59:35.366426  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 387714us
I20260709 12:59:35.175755  3268 leader_election.cc:290] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:35.230522  3420 leader_election.cc:290] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:35.163152  3411 raft_consensus.cc:493] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:35.165154  3525 raft_consensus.cc:697] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 4 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 12:59:35.374853  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:35.247931  3461 leader_election.cc:290] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 12:59:35.384087  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 70924.9us
I20260709 12:59:35.391987  1718 leader_election.cc:304] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:59:35.384939  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:35.385126  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:35.394646  1600 raft_consensus.cc:2468] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 3.
I20260709 12:59:35.395957  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:35.180532  3576 leader_election.cc:290] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 12:59:35.386263  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 56280.6us
I20260709 12:59:35.315096  3521 raft_consensus.cc:493] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:35.311535  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:35.396906  2336 raft_consensus.cc:2468] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 2.
I20260709 12:59:35.272193  3575 leader_election.cc:290] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:35.243968  3546 leader_election.cc:290] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:35.423318  3547 leader_election.cc:290] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 4 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:35.397434  1717 leader_election.cc:304] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:59:35.246409  3424 leader_election.cc:290] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:35.236416  3594 leader_election.cc:290] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:35.315843  1675 raft_consensus.cc:2468] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 3.
I20260709 12:59:35.364431  3468 leader_election.cc:290] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:35.326148  3471 consensus_queue.cc:1048] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.074s
I20260709 12:59:35.314460  3415 consensus_queue.cc:1048] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.089s
I20260709 12:59:35.314493  3552 consensus_queue.cc:1048] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.081s
W20260709 12:59:35.414036  3589 log.cc:927] Time spent T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.075s	user 0.000s	sys 0.000s
I20260709 12:59:35.415614  1597 raft_consensus.cc:2468] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 5.
I20260709 12:59:35.367213  3431 raft_consensus.cc:515] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:35.420377  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:35.419835  3532 consensus_queue.cc:237] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:35.421242  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:59:35.426661  1569 leader_election.cc:304] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 12:59:35.371411  3528 raft_consensus.cc:493] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 12:59:35.433092  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:35.385990  3356 raft_consensus.cc:2749] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:59:35.312821  3409 raft_consensus.cc:515] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:35.440043  1672 raft_consensus.cc:1240] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 2. Current term is 3. Ops: [2.2-2.2]
W20260709 12:59:35.407068  3566 log.cc:927] Time spent T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.068s	user 0.000s	sys 0.001s
I20260709 12:59:35.393154  3416 raft_consensus.cc:493] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:35.299052  1501 rpcz_store.cc:276] 0709 12:59:33.388567 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:33.388780 (+   213us) service_pool.cc:224] Handling call
0709 12:59:35.151212 (+1762432us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:59:33.728678 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:59:33.729656 (+   978us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:59:33.729759 (+   103us) write_op.cc:435] Acquired schema lock
0709 12:59:33.729822 (+    63us) tablet.cc:662] Decoding operations
0709 12:59:33.730812 (+   990us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:59:33.730946 (+   134us) write_op.cc:641] Partition lock acquired for write op
0709 12:59:33.731003 (+    57us) tablet.cc:685] Acquiring locks for 1 operations
0709 12:59:33.735528 (+  4525us) tablet.cc:701] Row locks acquired
0709 12:59:33.735627 (+    99us) write_op.cc:260] PREPARE: finished
0709 12:59:33.735944 (+   317us) write_op.cc:270] Start()
0709 12:59:33.736235 (+   291us) write_op.cc:276] Timestamp: P: 1783601973735858 usec, L: 0
0709 12:59:33.736322 (+    87us) op_driver.cc:348] REPLICATION: starting
0709 12:59:33.737551 (+  1229us) log.cc:844] Serialized 536 byte log entry
0709 12:59:34.093721 (+356170us) op_driver.cc:464] REPLICATION: finished
0709 12:59:34.358582 (+264861us) write_op.cc:301] APPLY: starting
0709 12:59:34.366918 (+  8336us) tablet.cc:1367] starting BulkCheckPresence
0709 12:59:34.387668 (+ 20750us) tablet.cc:1370] finished BulkCheckPresence
0709 12:59:34.402593 (+ 14925us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:59:34.403201 (+   608us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:59:34.403477 (+   276us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0709 12:59:34.403556 (+    79us) write_op.cc:312] APPLY: finished
0709 12:59:34.453121 (+ 49565us) log.cc:844] Serialized 23 byte log entry
0709 12:59:34.464984 (+ 11863us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:59:34.468855 (+  3871us) write_op.cc:454] Released schema lock
0709 12:59:34.473192 (+  4337us) write_op.cc:341] FINISH: Updating metrics
0709 12:59:34.483365 (+ 10173us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:59:34.483396 (+    31us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":346506,"reported_tablets":1,"spinlock_wait_cycles":119424,"child_traces":[["op",{"apply.queue_time_us":195918,"apply.run_cpu_time_us":4298,"apply.run_wall_time_us":170753,"num_ops":1,"prepare.queue_time_us":259109,"prepare.run_cpu_time_us":20177,"prepare.run_wall_time_us":129404,"raft.queue_time_us":236144,"raft.run_cpu_time_us":68553,"raft.run_wall_time_us":282522,"replication_time_us":357099,"thread_start_us":369227,"threads_started":4,"wal-append.queue_time_us":155816}]]}
I20260709 12:59:35.402173  3582 raft_consensus.cc:2804] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:59:35.391167  3580 raft_consensus.cc:493] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 12:59:35.409554  3461 raft_consensus.cc:493] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:35.395375  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:35.461090  2334 raft_consensus.cc:2468] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 3.
I20260709 12:59:35.398743  3505 consensus_queue.cc:1059] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.057s
I20260709 12:59:35.466599  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:35.399076  3411 raft_consensus.cc:515] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:35.369642  3588 raft_consensus.cc:515] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:35.442968  3553 consensus_queue.cc:237] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:35.373409  3504 raft_consensus.cc:515] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:35.387979  3542 raft_consensus.cc:515] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:35.433650  3521 raft_consensus.cc:515] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:35.467896  3425 leader_election.cc:290] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:35.711164  1603 raft_consensus.cc:2393] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 5: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
W20260709 12:59:35.648276  1646 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:59:35.592984  2333 raft_consensus.cc:1275] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:59:35.714977  2335 raft_consensus.cc:2468] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 3.
I20260709 12:59:35.710665  3491 raft_consensus.cc:493] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 12:59:35.704205  3592 raft_consensus.cc:493] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:35.716272  3491 raft_consensus.cc:515] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:35.716495  3592 raft_consensus.cc:515] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:35.534642  3414 raft_consensus.cc:493] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 12:59:35.718222  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:35.444212  3576 raft_consensus.cc:2804] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:59:35.719440  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 12:59:35.475181  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:35.720383  2336 raft_consensus.cc:3060] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:59:35.719614  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 292999us
I20260709 12:59:35.712316  2337 raft_consensus.cc:2468] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 2.
I20260709 12:59:35.739400  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:35.706925  3552 raft_consensus.cc:493] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:35.738324  3471 raft_consensus.cc:2804] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260709 12:59:35.711423  3444 consensus_meta.cc:287] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.596s	user 0.013s	sys 0.002s
I20260709 12:59:35.755963  3564 raft_consensus.cc:493] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:35.738288  3528 raft_consensus.cc:515] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:35.756937  3564 raft_consensus.cc:515] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:35.726117  3491 leader_election.cc:290] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:35.728403  3580 raft_consensus.cc:515] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:35.759837  3528 leader_election.cc:290] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 12:59:35.471005  3593 log.cc:927] Time spent T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.196s	user 0.000s	sys 0.002s
W20260709 12:59:35.590045  3558 consensus_meta.cc:287] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.710s	user 0.008s	sys 0.004s
I20260709 12:59:35.761070  3558 raft_consensus.cc:515] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:35.535669  3468 raft_consensus.cc:493] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:35.717905  3414 raft_consensus.cc:515] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:35.607327  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:35.529917  3525 consensus_queue.cc:237] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:35.749694  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:35.752964  3471 raft_consensus.cc:493] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:59:35.726843  2331 raft_consensus.cc:1240] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 2. Current term is 3. Ops: []
I20260709 12:59:35.728799  3576 raft_consensus.cc:493] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:59:35.570667  3358 raft_consensus.cc:2749] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:59:35.772491  2335 raft_consensus.cc:2468] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 3.
W20260709 12:59:35.776386  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 43811.5us
I20260709 12:59:35.705164  3595 raft_consensus.cc:493] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:35.768257  2332 raft_consensus.cc:2468] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 2.
I20260709 12:59:35.771914  2333 raft_consensus.cc:2468] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 3.
I20260709 12:59:35.764791  3558 leader_election.cc:290] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 4 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:35.587572  3594 raft_consensus.cc:493] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:35.796329  3592 leader_election.cc:290] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:35.797369  3594 raft_consensus.cc:515] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:35.798185  3531 consensus_queue.cc:1059] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:59:35.587863  3543 raft_consensus.cc:493] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 12:59:35.730504  3582 raft_consensus.cc:493] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:59:35.803196  3582 raft_consensus.cc:3060] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:59:35.802973  3543 raft_consensus.cc:515] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:35.724915  3424 raft_consensus.cc:493] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 12:59:35.782631  1599 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 1680) took 1050 ms. Trace:
I20260709 12:59:35.762871  3548 raft_consensus.cc:493] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:35.767066  1602 raft_consensus.cc:2368] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 for earlier term 3. Current term is 4.
I20260709 12:59:35.716867  3409 leader_election.cc:290] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:35.795961  1675 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f"
dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
mode: GRACEFUL
new_leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
 from {username='slave'} at 127.0.0.1:43266
I20260709 12:59:35.810825  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 12:59:35.810520  3424 raft_consensus.cc:515] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } }
I20260709 12:59:35.811233  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:35.731276  3461 raft_consensus.cc:515] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:35.764810  3411 leader_election.cc:290] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:35.762318  3542 leader_election.cc:290] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 3 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:35.763098  3552 raft_consensus.cc:515] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:35.779825  3591 raft_consensus.cc:493] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 12:59:35.765689  3431 leader_election.cc:290] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 3 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:35.780550  3471 raft_consensus.cc:3060] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:59:35.807930  1599 rpcz_store.cc:276] 0709 12:59:34.725841 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:34.725981 (+   140us) service_pool.cc:224] Handling call
0709 12:59:34.731871 (+  5890us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:59:35.311145 (+579274us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:35.776333 (+465188us) spinlock_profiling.cc:243] Waited 416 ms on lock 0x7b54002990d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 12:59:35.776577 (+   244us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":956832512}
I20260709 12:59:35.813997  1753 raft_consensus.cc:1275] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260709 12:59:35.811750  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:35.869127  1569 leader_election.cc:304] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 12:59:35.840621  3358 raft_consensus.cc:493] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 12:59:35.827370  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 12:59:35.841348  3491 raft_consensus.cc:493] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 12:59:35.782804  3576 raft_consensus.cc:3060] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:59:35.762416  3580 leader_election.cc:290] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:35.718360  3416 raft_consensus.cc:515] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:35.842379  3295 raft_consensus.cc:493] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:35.840054  3594 leader_election.cc:290] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:35.844368  3528 raft_consensus.cc:493] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 12:59:35.781461  3468 raft_consensus.cc:515] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:35.842595  3573 raft_consensus.cc:493] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:35.826712  3504 leader_election.cc:290] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 3 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:35.884156  3588 leader_election.cc:290] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:35.886237  3521 leader_election.cc:290] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:35.836218  3558 raft_consensus.cc:493] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:59:35.892163  2298 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.574s	user 0.024s	sys 0.038s
I20260709 12:59:35.824579  3564 leader_election.cc:290] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:35.876458  3592 raft_consensus.cc:493] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
W20260709 12:59:35.901547  2298 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.584s	user 0.025s	sys 0.039s
I20260709 12:59:35.901556  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 12:59:35.857954  2329 raft_consensus.cc:3060] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:59:35.911542  2328 raft_consensus.cc:2368] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 3. Current term is 4.
I20260709 12:59:35.890342  3295 raft_consensus.cc:515] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:35.889012  3268 raft_consensus.cc:2804] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:59:35.973631  1602 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36462 (request call id 270) took 1077 ms. Trace:
W20260709 12:59:35.969887  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 54735.7us
I20260709 12:59:35.977731  2298 leader_election.cc:304] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 12:59:35.878614  3358 raft_consensus.cc:515] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:35.974248  1602 rpcz_store.cc:276] 0709 12:59:34.896279 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:34.898581 (+  2302us) service_pool.cc:224] Handling call
0709 12:59:35.942884 (+1044303us) spinlock_profiling.cc:243] Waited 847 ms on lock 0x7b54000ac3d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 12:59:35.973518 (+ 30634us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1947346304}
I20260709 12:59:35.812656  3548 raft_consensus.cc:515] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:35.947737  2301 leader_election.cc:304] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:59:35.857721  2330 raft_consensus.cc:2468] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 2.
W20260709 12:59:35.985648  1645 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.614s	user 0.004s	sys 0.034s
W20260709 12:59:35.988095  1645 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.618s	user 0.005s	sys 0.034s
I20260709 12:59:35.887017  3491 raft_consensus.cc:515] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:35.972846  3268 raft_consensus.cc:493] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:59:36.009008  1569 leader_election.cc:304] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 12:59:35.905279  3528 raft_consensus.cc:515] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:35.912488  3558 raft_consensus.cc:515] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:35.998880  3518 tablet_copy_client.cc:538] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:59:36.018694  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:36.016676  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:35.842816  3595 raft_consensus.cc:515] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:36.001715  3575 raft_consensus.cc:2804] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:59:36.002225  1749 raft_consensus.cc:1275] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 12:59:35.982340  3504 consensus_queue.cc:1048] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.035s
I20260709 12:59:36.027041  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:36.012599  3268 raft_consensus.cc:3060] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:59:36.034188  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:59:36.058836  1673 raft_consensus.cc:3060] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:59:35.941234  3592 raft_consensus.cc:515] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:36.013872  3588 raft_consensus.cc:493] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 12:59:36.036899  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:36.062285  1747 raft_consensus.cc:2468] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 2.
I20260709 12:59:36.008741  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:36.024713  1748 raft_consensus.cc:2393] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 3: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 12:59:36.013520  3579 raft_consensus.cc:2804] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:59:35.905328  3591 raft_consensus.cc:515] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:36.060276  3424 leader_election.cc:290] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:36.037863  3575 raft_consensus.cc:493] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:59:35.901662  3573 raft_consensus.cc:515] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:36.016842  3531 consensus_queue.cc:1048] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.048s
I20260709 12:59:36.060334  3600 raft_consensus.cc:2804] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:59:36.077230  3592 leader_election.cc:290] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:36.077808  3491 leader_election.cc:290] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:36.079432  3579 raft_consensus.cc:493] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:59:36.081652  3599 log.cc:927] Time spent T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.177s	user 0.001s	sys 0.000s
I20260709 12:59:36.084290  3588 raft_consensus.cc:515] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:36.057422  3521 raft_consensus.cc:493] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 12:59:36.090663  3580 raft_consensus.cc:493] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:36.091346  3594 raft_consensus.cc:493] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:36.087500  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:36.077831  3558 leader_election.cc:290] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:36.070761  3416 leader_election.cc:290] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:36.096405  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:36.144980  1680 raft_consensus.cc:2468] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 5.
I20260709 12:59:36.072546  1646 leader_election.cc:304] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
W20260709 12:59:36.146446  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 74019.9us
I20260709 12:59:36.079537  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:36.129333  3600 raft_consensus.cc:493] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:59:36.129438  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:36.131492  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:36.095206  3575 raft_consensus.cc:3060] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:59:36.159657  1671 raft_consensus.cc:2468] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 1.
I20260709 12:59:36.094340  3425 raft_consensus.cc:493] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:36.133724  3358 leader_election.cc:290] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:36.167094  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:36.140256  3552 leader_election.cc:290] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:36.156302  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:36.138662  3579 raft_consensus.cc:3060] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:59:36.138290  3461 leader_election.cc:290] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 12:59:36.102627  3501 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.683s	user 0.005s	sys 0.024s
I20260709 12:59:36.167514  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:36.169701  1747 raft_consensus.cc:2468] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 2.
I20260709 12:59:36.189404  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:36.172358  1752 raft_consensus.cc:2468] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 3.
I20260709 12:59:36.191385  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:36.170300  1749 raft_consensus.cc:2468] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 5.
W20260709 12:59:36.175235  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 191972us
I20260709 12:59:36.159812  3600 raft_consensus.cc:3060] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:59:36.159026  3416 raft_consensus.cc:2804] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260709 12:59:36.205335  2299 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.837s	user 0.021s	sys 0.069s
W20260709 12:59:36.205757  2299 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.837s	user 0.021s	sys 0.069s
I20260709 12:59:36.148480  3591 leader_election.cc:290] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:36.163775  3468 leader_election.cc:290] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:36.188114  1680 raft_consensus.cc:2468] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 4.
W20260709 12:59:36.212493  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 881997us
I20260709 12:59:36.190163  1747 raft_consensus.cc:2468] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 2.
I20260709 12:59:36.192932  2298 leader_election.cc:304] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 12:59:36.218637  1718 leader_election.cc:304] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:36.218874  2298 leader_election.cc:400] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445) with higher term. Message: Invalid argument: T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 for earlier term 3. Current term is 4.
I20260709 12:59:36.222747  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:59:36.224566  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:59:36.225581  1602 raft_consensus.cc:2368] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 for earlier term 3. Current term is 4.
I20260709 12:59:36.226718  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:59:36.226945  2298 leader_election.cc:400] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 3 election: Vote denied by peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445) with higher term. Message: Invalid argument: T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 for earlier term 3. Current term is 4.
I20260709 12:59:36.226992  1718 leader_election.cc:304] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:36.227453  2298 leader_election.cc:403] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20260709 12:59:36.227802  1599 raft_consensus.cc:2393] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 3: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 12:59:36.228134  3504 raft_consensus.cc:3060] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:59:36.163169  3295 leader_election.cc:290] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:36.229997  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 12:59:36.187685  3501 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.768s	user 0.005s	sys 0.024s
W20260709 12:59:36.259297  1678 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.532s	user 0.010s	sys 0.042s
W20260709 12:59:36.259656  1678 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.533s	user 0.010s	sys 0.042s
I20260709 12:59:36.260556  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:36.261426  2333 raft_consensus.cc:2393] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 2: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 12:59:36.160317  3580 raft_consensus.cc:515] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:36.166826  3497 consensus_queue.cc:1048] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.047s
W20260709 12:59:36.234557  1751 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.774s	user 0.010s	sys 0.033s
W20260709 12:59:36.270213  1751 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.810s	user 0.010s	sys 0.040s
I20260709 12:59:36.159574  3521 raft_consensus.cc:515] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:36.204960  3416 raft_consensus.cc:493] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:59:36.230628  3598 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.732s	user 0.014s	sys 0.033s
I20260709 12:59:36.171116  3425 raft_consensus.cc:515] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:36.160143  3594 raft_consensus.cc:515] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } }
I20260709 12:59:36.320789  3594 leader_election.cc:290] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:36.321488  3594 raft_consensus.cc:493] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 12:59:36.324769  3425 leader_election.cc:290] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 12:59:36.286257  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 52950.3us
I20260709 12:59:36.242599  3358 raft_consensus.cc:2804] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Leader election won for term 3
W20260709 12:59:36.262665  3549 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.556s	user 0.013s	sys 0.029s
I20260709 12:59:36.326403  3414 leader_election.cc:290] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 12:59:36.287328  2336 consensus_meta.cc:287] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.549s	user 0.010s	sys 0.004s
I20260709 12:59:36.259076  3356 raft_consensus.cc:2749] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 12:59:36.328831  3549 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.622s	user 0.013s	sys 0.029s
I20260709 12:59:36.328621  3579 raft_consensus.cc:515] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:36.287634  3592 raft_consensus.cc:493] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 12:59:36.245052  3461 raft_consensus.cc:2804] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:59:36.322129  3594 raft_consensus.cc:515] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:36.331429  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:36.373759  1671 raft_consensus.cc:2468] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 2.
I20260709 12:59:36.262419  3431 raft_consensus.cc:493] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 12:59:36.335664  3528 leader_election.cc:290] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:36.338922  1675 raft_consensus.cc:606] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 4 LEADER]: Received request to transfer leadership to TS 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:36.269622  3444 raft_consensus.cc:493] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 12:59:36.355702  2336 raft_consensus.cc:2468] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 4.
W20260709 12:59:36.317231  3598 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.842s	user 0.015s	sys 0.034s
I20260709 12:59:36.329298  3358 raft_consensus.cc:697] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 3 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
W20260709 12:59:36.385742  1643 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.017s	user 0.030s	sys 0.044s
I20260709 12:59:36.314188  3416 raft_consensus.cc:3060] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:59:36.363265  1672 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52766 (request call id 1909) took 1354 ms. Trace:
I20260709 12:59:36.316624  3468 raft_consensus.cc:493] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 12:59:36.281715  3601 raft_consensus.cc:493] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 12:59:36.341297  1749 raft_consensus.cc:1240] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 1. Current term is 2. Ops: []
I20260709 12:59:36.336010  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 12:59:36.386190  1643 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.017s	user 0.030s	sys 0.044s
I20260709 12:59:36.364393  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:36.449707  1643 leader_election.cc:304] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 12:59:36.390550  2332 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39738 (request call id 537) took 1073 ms. Trace:
W20260709 12:59:36.430955  1754 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.894s	user 0.044s	sys 0.048s
I20260709 12:59:36.433985  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:36.370661  3544 raft_consensus.cc:493] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:36.372046  3461 raft_consensus.cc:493] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:59:36.431463  1672 rpcz_store.cc:276] 0709 12:59:35.008543 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:35.008673 (+   130us) service_pool.cc:224] Handling call
0709 12:59:35.439769 (+431096us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:59:35.711875 (+272106us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:36.362859 (+650984us) spinlock_profiling.cc:243] Waited 431 ms on lock 0x7b54001e87d0. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661618ad 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 12:59:36.363098 (+   239us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":990506368}
W20260709 12:59:36.453611  1754 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.917s	user 0.045s	sys 0.048s
I20260709 12:59:36.452858  2332 rpcz_store.cc:276] 0709 12:59:35.307869 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:35.308426 (+   557us) service_pool.cc:224] Handling call
0709 12:59:36.380627 (+1072201us) spinlock_profiling.cc:243] Waited 459 ms on lock 0x7b540077d654. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 12:59:36.380753 (+   126us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1055223552}
I20260709 12:59:36.472833  1718 leader_election.cc:304] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:59:36.383247  3466 raft_consensus.cc:493] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 12:59:36.487002  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
W20260709 12:59:36.499729  2335 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.588s	user 0.030s	sys 0.045s
W20260709 12:59:36.500182  2335 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.588s	user 0.030s	sys 0.045s
I20260709 12:59:36.434504  3578 raft_consensus.cc:3055] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:59:36.505052  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:36.375212  3592 raft_consensus.cc:515] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:36.387099  3431 raft_consensus.cc:515] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:36.388556  3444 raft_consensus.cc:515] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:36.505941  1672 raft_consensus.cc:2468] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 1.
I20260709 12:59:36.466035  3461 raft_consensus.cc:3060] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:59:36.506677  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:36.486442  3356 raft_consensus.cc:493] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 12:59:36.536674  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:59:36.494231  3595 leader_election.cc:290] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:36.537310  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:36.492965  3415 raft_consensus.cc:3055] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:59:36.556725  3595 raft_consensus.cc:493] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 12:59:36.556483  3356 raft_consensus.cc:515] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:36.556959  3415 raft_consensus.cc:740] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 12:59:36.557474  3595 raft_consensus.cc:515] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:36.558743  3415 consensus_queue.cc:260] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:36.560482  3415 raft_consensus.cc:3060] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:59:36.560783  1672 raft_consensus.cc:2368] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 for earlier term 3. Current term is 4.
I20260709 12:59:36.562036  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
W20260709 12:59:36.562656  1673 consensus_meta.cc:287] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.500s	user 0.013s	sys 0.000s
I20260709 12:59:36.442719  3468 raft_consensus.cc:515] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } }
I20260709 12:59:36.563321  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:36.489835  3497 raft_consensus.cc:2749] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:59:36.571705  3594 leader_election.cc:290] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:36.465400  3544 raft_consensus.cc:515] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:36.579501  3580 leader_election.cc:290] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:36.536886  3588 leader_election.cc:290] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:36.581070  1678 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52766 (request call id 1908) took 1574 ms. Trace:
I20260709 12:59:36.581588  1678 rpcz_store.cc:276] 0709 12:59:35.006910 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:35.007601 (+   691us) service_pool.cc:224] Handling call
0709 12:59:35.007984 (+   383us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:59:35.018301 (+ 10317us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:59:35.018335 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:59:35.027500 (+  9165us) log.cc:844] Serialized 26 byte log entry
0709 12:59:35.726858 (+699358us) spinlock_profiling.cc:243] Waited 5.81 ms on lock 0x7b180033f160. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f4263057e07 00007f426beb2ab9 00007f426307469b 00007f426307293f 00007f4263074c02 00007f4265cfc20b 00007f4265d01d23 00007f4265d020d1 00007f426611da0f
0709 12:59:36.260105 (+533247us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:59:36.260145 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:36.260249 (+   104us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:59:36.580714 (+320465us) raft_consensus.cc:1685] finished
0709 12:59:36.580738 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:59:36.580981 (+   243us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":6046,"raft.run_cpu_time_us":154,"raft.run_wall_time_us":5686,"spinlock_wait_cycles":13365504,"thread_start_us":532964,"threads_started":1,"wal-append.queue_time_us":1479668}
I20260709 12:59:36.540530  3528 raft_consensus.cc:493] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 12:59:36.587177  3528 raft_consensus.cc:515] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:36.522496  3425 raft_consensus.cc:2804] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260709 12:59:36.604388  1677 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.538s	user 0.023s	sys 0.047s
W20260709 12:59:36.604938  1677 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.539s	user 0.023s	sys 0.047s
I20260709 12:59:36.500741  2328 raft_consensus.cc:2468] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 3.
I20260709 12:59:36.538935  2331 raft_consensus.cc:2393] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 2: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 12:59:36.504559  3578 raft_consensus.cc:740] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 12:59:36.506733  3466 raft_consensus.cc:515] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 12:59:36.537930  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 51125.1us
I20260709 12:59:36.593405  1753 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55848 (request call id 1677) took 1240 ms. Trace:
I20260709 12:59:36.468808  3601 raft_consensus.cc:515] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:36.634793  1753 rpcz_store.cc:276] 0709 12:59:35.311520 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:35.325218 (+ 13698us) service_pool.cc:224] Handling call
0709 12:59:35.325558 (+   340us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:59:35.869535 (+543977us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:36.525730 (+656195us) spinlock_profiling.cc:243] Waited 488 ms on lock 0x7b5400019fd4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 12:59:36.551489 (+ 25759us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1122178304}
W20260709 12:59:36.610324  1679 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.593s	user 0.005s	sys 0.033s
I20260709 12:59:36.564059  1672 raft_consensus.cc:2468] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 2.
W20260709 12:59:36.647125  1679 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.630s	user 0.005s	sys 0.033s
I20260709 12:59:36.563098  1673 raft_consensus.cc:2468] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 4.
I20260709 12:59:36.491593  3532 raft_consensus.cc:493] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 12:59:36.604063  3425 raft_consensus.cc:493] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:59:36.593108  3528 leader_election.cc:290] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:36.659631  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 12:59:36.649027  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 318047us
I20260709 12:59:36.607654  3549 raft_consensus.cc:493] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
W20260709 12:59:36.631502  3582 consensus_meta.cc:287] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.827s	user 0.020s	sys 0.002s
I20260709 12:59:36.544396  3414 consensus_queue.cc:1048] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.089s
I20260709 12:59:36.660516  1571 leader_election.cc:304] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 12:59:36.662743  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:36.666783  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:36.544397  3358 consensus_queue.cc:237] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:36.646903  3588 raft_consensus.cc:493] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 12:59:36.704206  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:36.665990  3425 raft_consensus.cc:3060] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:59:36.659337  3594 raft_consensus.cc:493] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 12:59:36.653262  3580 raft_consensus.cc:493] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 12:59:36.669414  1749 raft_consensus.cc:1275] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:59:36.665489  3532 raft_consensus.cc:515] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:36.676813  3549 raft_consensus.cc:515] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 12:59:36.941249  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 242470us
W20260709 12:59:36.896096  1751 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.580s	user 0.029s	sys 0.021s
I20260709 12:59:36.962495  1753 raft_consensus.cc:2393] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 2: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
W20260709 12:59:36.703377  3576 consensus_meta.cc:287] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.813s	user 0.011s	sys 0.006s
W20260709 12:59:36.965823  1751 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.646s	user 0.033s	sys 0.025s
I20260709 12:59:36.967284  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 12:59:36.971051  3609 log.cc:927] Time spent T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.128s	user 0.000s	sys 0.001s
W20260709 12:59:36.970846  1747 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.596s	user 0.014s	sys 0.040s
W20260709 12:59:36.973124  1747 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.598s	user 0.014s	sys 0.040s
I20260709 12:59:36.699097  3578 consensus_queue.cc:260] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 12:59:36.932540  3605 log.cc:927] Time spent T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.388s	user 0.000s	sys 0.000s
W20260709 12:59:36.888355  3504 consensus_meta.cc:287] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.660s	user 0.003s	sys 0.010s
W20260709 12:59:36.898298  3268 consensus_meta.cc:287] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.819s	user 0.008s	sys 0.006s
I20260709 12:59:36.988063  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:36.860124  3558 raft_consensus.cc:2804] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:59:36.989797  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:37.005082  1680 raft_consensus.cc:2468] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 2.
I20260709 12:59:36.705729  3521 leader_election.cc:290] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:36.935932  3619 raft_consensus.cc:493] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:59:37.005342  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 163606us
I20260709 12:59:36.818993  3579 leader_election.cc:290] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 3 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:36.679976  3582 raft_consensus.cc:515] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:37.007041  3578 raft_consensus.cc:3060] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:59:37.023929  3619 raft_consensus.cc:515] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:37.060746  3431 leader_election.cc:290] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:37.026605  1677 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52766 (request call id 1925) took 1012 ms. Trace:
I20260709 12:59:36.987264  3576 raft_consensus.cc:515] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:37.027567  3601 leader_election.cc:290] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:37.030629  3518 tablet_bootstrap.cc:492] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1: Bootstrap starting.
I20260709 12:59:37.007076  3504 raft_consensus.cc:2749] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445) with higher term. Message: Invalid argument: T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 for earlier term 3. Current term is 4.
I20260709 12:59:36.958433  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:36.682163  3575 raft_consensus.cc:515] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 12:59:37.039012  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 57364.2us
I20260709 12:59:36.991058  1753 raft_consensus.cc:2468] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 1.
I20260709 12:59:36.897703  3468 leader_election.cc:290] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:36.907303  3572 consensus_queue.cc:1059] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.108s
I20260709 12:59:37.057961  3592 leader_election.cc:290] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:37.059276  3532 leader_election.cc:290] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:37.059477  3582 leader_election.cc:290] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:37.056177  1569 leader_election.cc:304] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 12:59:37.063371  1745 raft_consensus.cc:2393] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 3: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 12:59:37.061864  1677 rpcz_store.cc:276] 0709 12:59:36.014538 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:36.014684 (+   146us) service_pool.cc:224] Handling call
0709 12:59:36.015050 (+   366us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:59:36.017134 (+  2084us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:59:36.017161 (+    27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:59:36.017724 (+   563us) log.cc:844] Serialized 26 byte log entry
0709 12:59:36.605515 (+587791us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:59:36.605582 (+    67us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:36.605691 (+   109us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:59:37.026304 (+420613us) raft_consensus.cc:1685] finished
0709 12:59:37.026327 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:59:37.026521 (+   194us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1668,"raft.run_cpu_time_us":223,"raft.run_wall_time_us":10951,"thread_start_us":539194,"threads_started":1,"wal-append.queue_time_us":987711}
I20260709 12:59:37.087121  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:37.087764  1677 raft_consensus.cc:2468] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 5.
I20260709 12:59:37.088640  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:59:36.977734  3620 raft_consensus.cc:493] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
W20260709 12:59:37.089320  3415 consensus_meta.cc:287] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.528s	user 0.012s	sys 0.000s
I20260709 12:59:37.090345  1746 raft_consensus.cc:2393] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 4: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 12:59:37.057865  3544 leader_election.cc:290] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:36.971717  3594 raft_consensus.cc:515] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:37.074389  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:37.073169  1644 leader_election.cc:304] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: af520b67945d443b901bbeca37cc5701
I20260709 12:59:36.970746  3580 raft_consensus.cc:515] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:37.089177  1677 raft_consensus.cc:3060] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:59:37.089701  3620 raft_consensus.cc:515] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:37.061501  3619 leader_election.cc:290] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:37.032232  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:36.982123  3588 raft_consensus.cc:515] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:37.066784  3356 leader_election.cc:290] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:37.007184  3268 raft_consensus.cc:515] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 12:59:37.111555  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 55515.5us
I20260709 12:59:37.414467  1569 leader_election.cc:304] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 12:59:37.073205  3497 raft_consensus.cc:493] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
W20260709 12:59:37.073468  1676 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52766 (request call id 1910) took 2064 ms (client timeout 2224 ms). Trace:
I20260709 12:59:37.136128  1754 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43156 (request call id 301) took 2332 ms. Trace:
I20260709 12:59:37.445467  1569 leader_election.cc:304] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 12:59:37.401075  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:37.448735  1679 raft_consensus.cc:2468] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 2.
I20260709 12:59:37.449324  1672 raft_consensus.cc:2468] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 2.
W20260709 12:59:37.379336  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 286780us
I20260709 12:59:37.208698  1747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55840 (request call id 1944) took 1113 ms. Trace:
I20260709 12:59:37.451689  1571 leader_election.cc:304] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 12:59:37.462581  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:37.464705  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:37.466784  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:37.467679  1605 raft_consensus.cc:2468] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 1.
I20260709 12:59:37.468798  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:37.469616  1598 raft_consensus.cc:2468] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 3.
W20260709 12:59:37.445449  1676 rpcz_store.cc:269] 0709 12:59:35.008858 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:35.009105 (+   247us) service_pool.cc:224] Handling call
0709 12:59:37.073218 (+2064113us) spinlock_profiling.cc:243] Waited 1.42 s on lock 0x7b54000d7d54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 12:59:37.073369 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3274902912}
I20260709 12:59:37.451254  1747 rpcz_store.cc:276] 0709 12:59:36.095653 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:36.258384 (+162731us) service_pool.cc:224] Handling call
0709 12:59:36.259880 (+  1496us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:59:36.260570 (+   690us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:59:36.260604 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:59:36.304088 (+ 43484us) log.cc:844] Serialized 26 byte log entry
0709 12:59:36.973519 (+669431us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:59:36.973583 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:36.973682 (+    99us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:59:37.208393 (+234711us) raft_consensus.cc:1685] finished
0709 12:59:37.208424 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:59:37.208598 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":302,"raft.run_cpu_time_us":204,"raft.run_wall_time_us":26806,"thread_start_us":598507,"threads_started":1,"wal-append.queue_time_us":756582}
I20260709 12:59:37.529894  1569 leader_election.cc:304] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 12:59:37.303375  3633 raft_consensus.cc:493] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
W20260709 12:59:37.373586  3627 log.cc:927] Time spent T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.119s	user 0.000s	sys 0.000s
W20260709 12:59:37.362779  3471 consensus_meta.cc:287] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 1.480s	user 0.013s	sys 0.004s
I20260709 12:59:37.532651  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:37.540117  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:37.540606  1680 raft_consensus.cc:2468] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 1.
I20260709 12:59:37.541255  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:37.541652  1680 raft_consensus.cc:2468] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 2.
I20260709 12:59:37.543097  1680 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57092 (request call id 396) took 1278 ms. Trace:
I20260709 12:59:37.532333  3499 raft_consensus.cc:493] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
W20260709 12:59:37.097512  2329 consensus_meta.cc:287] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 1.116s	user 0.006s	sys 0.010s
I20260709 12:59:37.532996  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:37.543897  3499 raft_consensus.cc:515] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:37.534533  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:37.534824  3544 raft_consensus.cc:2804] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:59:37.545204  3544 raft_consensus.cc:493] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:59:37.545629  3544 raft_consensus.cc:3060] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:59:37.543538  1680 rpcz_store.cc:276] 0709 12:59:36.265110 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:37.542098 (+1276988us) service_pool.cc:224] Handling call
0709 12:59:37.542321 (+   223us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:59:37.542740 (+   419us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:59:37.542766 (+    26us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:59:37.542819 (+    53us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:59:37.542855 (+    36us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:37.542918 (+    63us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:59:37.543029 (+   111us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:59:37.535786  1569 leader_election.cc:304] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 12:59:37.415508  3531 raft_consensus.cc:2804] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:59:37.536790  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:37.547746  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:37.548623  1747 raft_consensus.cc:2468] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 5.
I20260709 12:59:37.549122  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:37.547791  3531 raft_consensus.cc:493] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:59:37.550086  3531 raft_consensus.cc:3060] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:59:37.371127  3461 consensus_meta.cc:287] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.819s	user 0.009s	sys 0.005s
I20260709 12:59:37.537501  1600 raft_consensus.cc:2368] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 2. Current term is 3.
I20260709 12:59:37.550904  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 12:59:37.538833  1717 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 1.736s (SENT) [suppressed 36 similar messages]
I20260709 12:59:37.551832  1603 raft_consensus.cc:2468] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 4.
I20260709 12:59:37.083709  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:37.545645  2329 raft_consensus.cc:2468] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 4.
W20260709 12:59:37.089321  3416 consensus_meta.cc:287] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.641s	user 0.007s	sys 0.003s
I20260709 12:59:37.550820  3461 raft_consensus.cc:515] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:37.415704  3532 raft_consensus.cc:493] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 12:59:37.560467  3532 raft_consensus.cc:515] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:37.561810  3461 leader_election.cc:290] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 6 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:37.531370  3598 raft_consensus.cc:493] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:37.562669  2332 raft_consensus.cc:2393] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 6: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 12:59:37.562764  3598 raft_consensus.cc:515] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:37.531607  3633 raft_consensus.cc:515] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:37.565229  3598 leader_election.cc:290] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:37.566038  3633 leader_election.cc:290] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:37.567076  3624 raft_consensus.cc:2804] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:59:37.567519  3532 leader_election.cc:290] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:37.567626  3624 raft_consensus.cc:697] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 12:59:37.567768  1597 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 280) took 1345 ms. Trace:
I20260709 12:59:37.113306  3622 consensus_queue.cc:1059] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.455s
I20260709 12:59:37.568236  1597 rpcz_store.cc:276] 0709 12:59:36.222012 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:36.222220 (+   208us) service_pool.cc:224] Handling call
0709 12:59:36.222775 (+   555us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:59:36.264190 (+ 41415us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:59:36.269756 (+  5566us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:59:36.405693 (+135937us) log.cc:844] Serialized 26 byte log entry
0709 12:59:36.840430 (+434737us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:59:36.840514 (+    84us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:36.840662 (+   148us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:59:37.567484 (+726822us) raft_consensus.cc:1685] finished
0709 12:59:37.567516 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:59:37.567688 (+   172us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":43677,"raft.queue_time_us":60629,"raft.run_cpu_time_us":226,"raft.run_wall_time_us":24190,"thread_start_us":307655,"threads_started":1,"wal-append.queue_time_us":621307}
I20260709 12:59:37.532667  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:37.554574  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 12:59:37.559891  3600 consensus_meta.cc:287] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 1.314s	user 0.021s	sys 0.006s
I20260709 12:59:37.573048  1747 raft_consensus.cc:2468] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 2.
I20260709 12:59:37.542793  3471 raft_consensus.cc:515] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:37.538519  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:37.073491  3608 raft_consensus.cc:493] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:37.132371  3295 raft_consensus.cc:2804] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:59:37.447162  1754 rpcz_store.cc:276] 0709 12:59:34.799469 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:34.799589 (+   120us) service_pool.cc:224] Handling call
0709 12:59:35.535147 (+735558us) spinlock_profiling.cc:243] Waited 1.76 ms on lock 0x7b58001ffe98. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f426303db71 00007f426303da22 00007f426bf56c09 00007f426b0b0ee0 00007f426b17a0ab 00007f426b17a131 00007f426b13bfc0 00007f426b126929 00007f426b1263d2 00007f426469dbd4
0709 12:59:35.535433 (+   286us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:59:35.535907 (+   474us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:59:35.535946 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:59:35.536655 (+   709us) log.cc:844] Serialized 26 byte log entry
0709 12:59:36.457224 (+920569us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:59:36.457287 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:36.457387 (+   100us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:59:37.111418 (+654031us) raft_consensus.cc:1685] finished
0709 12:59:37.111444 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:59:37.132243 (+ 20799us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":448,"raft.queue_time_us":1621,"raft.run_cpu_time_us":341,"raft.run_wall_time_us":22855,"spinlock_wait_cycles":4050816,"thread_start_us":919964,"threads_started":1,"wal-append.queue_time_us":1117400}
I20260709 12:59:37.141160  3466 leader_election.cc:290] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:37.211601  3521 raft_consensus.cc:493] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 12:59:37.379117  3549 leader_election.cc:290] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:37.308841  3575 leader_election.cc:290] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 3 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:37.531803  3572 raft_consensus.cc:2804] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260709 12:59:37.558601  3529 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.619s	user 0.013s	sys 0.022s
I20260709 12:59:37.532498  3543 leader_election.cc:290] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:37.573724  1672 raft_consensus.cc:2468] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 3.
I20260709 12:59:37.555804  1680 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57092 (request call id 398) took 1290 ms. Trace:
I20260709 12:59:37.528970  3631 raft_consensus.cc:2749] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:59:37.490841  3497 raft_consensus.cc:515] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:37.829427  2337 raft_consensus.cc:1240] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 2. Current term is 3. Ops: [2.5-2.5]
I20260709 12:59:37.675277  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 12:59:37.765544  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 229819us
I20260709 12:59:37.833693  1752 raft_consensus.cc:2468] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 3.
I20260709 12:59:37.811813  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 12:59:37.596311  1676 raft_consensus.cc:2468] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 4.
W20260709 12:59:37.785770  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 246927us
I20260709 12:59:37.597810  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:37.834651  1680 rpcz_store.cc:276] 0709 12:59:36.265702 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:37.555197 (+1289495us) service_pool.cc:224] Handling call
0709 12:59:37.555434 (+   237us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:59:37.555586 (+   152us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:59:37.555600 (+    14us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:59:37.555629 (+    29us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:59:37.555649 (+    20us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:37.555685 (+    36us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:59:37.555754 (+    69us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:59:37.834522  1752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55848 (request call id 1693) took 1179 ms. Trace:
I20260709 12:59:37.812181  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:37.740309  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:37.570724  1749 raft_consensus.cc:2468] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 3.
I20260709 12:59:37.863255  1672 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52778 (request call id 1563) took 1231 ms. Trace:
I20260709 12:59:37.868228  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 12:59:37.837009  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 763794us
I20260709 12:59:37.870553  1747 raft_consensus.cc:2393] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 5: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 12:59:37.871430  1747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55848 (request call id 1690) took 1217 ms. Trace:
I20260709 12:59:37.677444  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:37.841640  1717 leader_election.cc:304] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:59:37.836079  1677 raft_consensus.cc:2410] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for term 4 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
W20260709 12:59:37.780332  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 67593.4us
I20260709 12:59:37.867123  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:37.871942  1747 rpcz_store.cc:276] 0709 12:59:36.653831 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:37.597926 (+944095us) service_pool.cc:224] Handling call
0709 12:59:37.871353 (+273427us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:59:37.872171  1753 raft_consensus.cc:2468] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 2.
I20260709 12:59:37.873165  1753 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55840 (request call id 1955) took 1213 ms. Trace:
I20260709 12:59:37.873358  1717 leader_election.cc:304] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:59:37.873602  1753 rpcz_store.cc:276] 0709 12:59:36.659854 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:37.071510 (+411656us) service_pool.cc:224] Handling call
0709 12:59:37.677229 (+605719us) spinlock_profiling.cc:243] Waited 12 ms on lock 0x7b58001ffe98. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f426303db71 00007f426303da22 00007f426bf56c09 00007f426b0b0ee0 00007f426b17a0ab 00007f426b17a131 00007f426b1280a0 00007f426b127984 00007f42646a26d4 00007f42646a2662
0709 12:59:37.873077 (+195848us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":27554048}
I20260709 12:59:37.875677  1677 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52778 (request call id 1560) took 1244 ms. Trace:
I20260709 12:59:37.862912  1752 rpcz_store.cc:276] 0709 12:59:36.655121 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:37.674915 (+1019794us) service_pool.cc:224] Handling call
0709 12:59:37.834417 (+159502us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:59:37.865345  3461 raft_consensus.cc:493] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:37.868600  1672 rpcz_store.cc:276] 0709 12:59:36.632047 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:37.531052 (+899005us) service_pool.cc:224] Handling call
0709 12:59:37.863165 (+332113us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":11136}
I20260709 12:59:37.865613  3431 raft_consensus.cc:493] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 12:59:37.673538  3444 leader_election.cc:290] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 12:59:37.945262  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 72505.1us
I20260709 12:59:37.713307  3491 raft_consensus.cc:493] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:37.870019  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 12:59:37.572573  2329 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39726 (request call id 638) took 1838 ms (client timeout 1731 ms). Trace:
I20260709 12:59:37.876251  1677 rpcz_store.cc:276] 0709 12:59:36.631165 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:37.088395 (+457230us) service_pool.cc:224] Handling call
0709 12:59:37.875591 (+787196us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:59:37.667043  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:37.812340  2334 raft_consensus.cc:1240] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 2. Current term is 3. Ops: [2.5-2.5]
I20260709 12:59:37.857792  3598 raft_consensus.cc:493] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:37.874845  1644 leader_election.cc:304] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
W20260709 12:59:37.955334  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 80566us
W20260709 12:59:37.797910  3425 consensus_meta.cc:287] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.825s	user 0.013s	sys 0.000s
I20260709 12:59:37.834923  3295 raft_consensus.cc:493] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:59:37.838459  3529 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.900s	user 0.015s	sys 0.030s
I20260709 12:59:37.656301  3525 raft_consensus.cc:2804] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:59:37.944192  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:37.670313  3544 raft_consensus.cc:515] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:37.946391  1754 raft_consensus.cc:2468] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 2.
W20260709 12:59:37.668051  3610 log.cc:927] Time spent T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 1.123s	user 0.003s	sys 0.012s
I20260709 12:59:37.664108  3499 leader_election.cc:290] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:37.666004  3268 leader_election.cc:290] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 3 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:37.765667  3356 raft_consensus.cc:493] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:37.814980  3633 raft_consensus.cc:493] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 12:59:37.743103  3576 leader_election.cc:290] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 4 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:37.658936  3580 leader_election.cc:290] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:37.740676  3619 raft_consensus.cc:493] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 12:59:37.963301  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:37.656965  3532 raft_consensus.cc:493] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 12:59:37.863981  3572 raft_consensus.cc:493] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:59:37.866272  3594 leader_election.cc:290] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:37.826471  3624 consensus_queue.cc:237] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:38.029393  1604 raft_consensus.cc:2393] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 3: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
W20260709 12:59:37.949561  2329 rpcz_store.cc:269] 0709 12:59:35.733009 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:35.767945 (+ 34936us) service_pool.cc:224] Handling call
0709 12:59:37.571158 (+1803213us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:59:37.764533  3358 raft_consensus.cc:493] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:37.570997  3416 raft_consensus.cc:515] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:37.662833  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:37.946674  3575 raft_consensus.cc:493] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:59:37.721393  1718 connection.cc:256] RPC call timeout handler was delayed by -1.1494773189999705s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1912s for a response
W20260709 12:59:38.035353  1718 connection.cc:256] RPC call timeout handler was delayed by -1.0034571579999465s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15680000000000002s for a response
I20260709 12:59:37.595618  3600 raft_consensus.cc:515] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:37.974685  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:38.038053  1747 raft_consensus.cc:2468] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 2.
I20260709 12:59:38.038745  1747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55840 (request call id 1956) took 1370 ms. Trace:
I20260709 12:59:38.039124  1747 rpcz_store.cc:276] 0709 12:59:36.668290 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:37.954319 (+1286029us) service_pool.cc:224] Handling call
0709 12:59:38.038663 (+ 84344us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:59:38.040016  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 166722us
I20260709 12:59:38.040261  3648 raft_consensus.cc:493] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 12:59:38.040426  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:38.040808  3648 raft_consensus.cc:515] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:38.041476  1747 raft_consensus.cc:2393] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 4: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 12:59:37.845175  3622 consensus_queue.cc:1059] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.126s
I20260709 12:59:38.043519  3622 raft_consensus.cc:3055] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:59:38.043879  3622 raft_consensus.cc:740] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 12:59:38.044946  3622 consensus_queue.cc:260] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:38.045928  3649 raft_consensus.cc:493] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 12:59:38.046829  3580 raft_consensus.cc:493] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 12:59:38.046747  3649 raft_consensus.cc:515] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:38.047632  3580 raft_consensus.cc:515] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:38.049747  3649 leader_election.cc:290] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:37.997694  1754 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55848 (request call id 1694) took 1342 ms. Trace:
I20260709 12:59:37.965590  3468 raft_consensus.cc:2804] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:59:38.050623  1754 rpcz_store.cc:276] 0709 12:59:36.655394 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:37.869740 (+1214346us) service_pool.cc:224] Handling call
0709 12:59:37.997585 (+127845us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:59:37.962570  3444 raft_consensus.cc:493] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 12:59:38.052160  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:38.050760  3468 raft_consensus.cc:493] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:59:38.053082  1754 raft_consensus.cc:2468] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 2.
I20260709 12:59:38.053438  3468 raft_consensus.cc:3060] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:59:38.006067  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:38.054862  1605 raft_consensus.cc:2468] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 3.
I20260709 12:59:37.972165  3525 raft_consensus.cc:493] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 12:59:38.056169  3525 raft_consensus.cc:3060] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:59:37.955138  3564 consensus_queue.cc:1048] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.048s
I20260709 12:59:37.874243  3521 raft_consensus.cc:515] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:37.870600  3608 raft_consensus.cc:515] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:38.045047  3640 raft_consensus.cc:493] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 12:59:38.060534  3608 leader_election.cc:290] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:37.993993  3638 raft_consensus.cc:493] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:38.061537  3638 raft_consensus.cc:515] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 12:59:38.063764  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 104621us
I20260709 12:59:38.064683  3638 leader_election.cc:290] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:37.951941  3461 raft_consensus.cc:515] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:38.067503  3461 leader_election.cc:290] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:37.962756  3491 raft_consensus.cc:515] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:37.970038  3425 raft_consensus.cc:515] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:38.063066  3648 leader_election.cc:290] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:38.054381  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:38.069883  3491 leader_election.cc:290] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:37.968487  3598 raft_consensus.cc:515] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:37.954509  3431 raft_consensus.cc:515] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:38.071422  2333 raft_consensus.cc:2468] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 2.
W20260709 12:59:38.018564  3630 log.cc:927] Time spent T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.461s	user 0.000s	sys 0.001s
I20260709 12:59:38.050756  3594 raft_consensus.cc:493] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 12:59:38.051991  3444 raft_consensus.cc:515] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:37.970543  3295 raft_consensus.cc:3060] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:59:38.060261  3640 raft_consensus.cc:515] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:38.069989  3425 leader_election.cc:290] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:38.027825  3572 raft_consensus.cc:3060] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:59:37.879488  3634 raft_consensus.cc:2804] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260709 12:59:38.076392  3637 log.cc:927] Time spent T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.250s	user 0.001s	sys 0.000s
W20260709 12:59:38.031680  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 164177us
I20260709 12:59:38.019320  3633 raft_consensus.cc:515] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:38.068881  3648 raft_consensus.cc:493] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 12:59:38.078660  3649 raft_consensus.cc:2804] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:59:38.078747  1603 raft_consensus.cc:2410] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for term 6 because replica has last-logged OpId of term: 4 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20260709 12:59:38.070314  1754 raft_consensus.cc:2468] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 4.
W20260709 12:59:38.081662  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 133692us
W20260709 12:59:38.167564  1571 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:59:38.089725  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:38.019083  3356 raft_consensus.cc:515] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:38.168689  1717 leader_election.cc:304] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:59:38.025840  3619 raft_consensus.cc:515] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:38.172638  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
W20260709 12:59:38.168206  1571 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:59:38.084069  1671 raft_consensus.cc:2393] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 3: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 12:59:38.143924  3491 raft_consensus.cc:493] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:38.091192  1569 leader_election.cc:400] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 3 election: Vote denied by peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625) with higher term. Message: Invalid argument: T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 3. Current term is 4.
I20260709 12:59:38.175662  1569 leader_election.cc:403] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
W20260709 12:59:38.176190  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 85184.9us
I20260709 12:59:38.144660  2301 leader_election.cc:304] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:59:38.030138  3532 raft_consensus.cc:515] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:38.173292  1754 raft_consensus.cc:3060] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:59:38.173789  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 129895us
W20260709 12:59:38.173993  1571 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:59:38.177349  2301 leader_election.cc:304] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
W20260709 12:59:38.166339  3635 log.cc:927] Time spent T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.590s	user 0.002s	sys 0.000s
I20260709 12:59:38.090830  3588 leader_election.cc:290] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:38.164674  3622 raft_consensus.cc:3060] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:38.072021  3575 raft_consensus.cc:515] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:38.183768  1646 leader_election.cc:304] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:59:38.037223  3358 raft_consensus.cc:515] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:38.090852  3634 raft_consensus.cc:493] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 12:59:38.143260  3268 raft_consensus.cc:493] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 12:59:38.171181  3649 raft_consensus.cc:493] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:59:38.175127  3425 raft_consensus.cc:2804] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:59:38.086423  3594 raft_consensus.cc:515] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:38.205145  3634 raft_consensus.cc:3060] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:38.205322  3268 raft_consensus.cc:515] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 12:59:38.211679  3531 consensus_meta.cc:287] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.661s	user 0.011s	sys 0.001s
I20260709 12:59:38.185217  3499 raft_consensus.cc:2804] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:59:38.251202  3268 leader_election.cc:290] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:38.251319  3499 raft_consensus.cc:493] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:59:38.252049  3499 raft_consensus.cc:3060] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:59:38.253582  3633 leader_election.cc:290] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:38.261336  3654 raft_consensus.cc:493] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 12:59:38.262005  3654 raft_consensus.cc:515] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:38.182163  3639 raft_consensus.cc:493] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:38.174307  3648 raft_consensus.cc:515] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:38.181814  3645 raft_consensus.cc:3060] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:59:38.229044  3431 leader_election.cc:290] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:38.191373  3620 leader_election.cc:290] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:38.271175  1747 raft_consensus.cc:2468] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 2.
I20260709 12:59:38.184788  3464 raft_consensus.cc:2804] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:59:38.230180  3649 raft_consensus.cc:3060] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:59:38.227923  3471 leader_election.cc:290] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 4 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:38.229797  3425 raft_consensus.cc:493] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
W20260709 12:59:38.275835  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 94550.3us
I20260709 12:59:38.331919  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:38.181389  3491 raft_consensus.cc:515] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:38.244115  3608 raft_consensus.cc:2804] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:59:38.185557  3631 consensus_queue.cc:1059] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.084s
I20260709 12:59:38.247186  3580 leader_election.cc:290] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:38.188014  3641 consensus_queue.cc:1048] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.084s
I20260709 12:59:38.311570  3356 leader_election.cc:290] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:38.276701  1675 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:43266 (request call id 22) took 2493 ms. Trace:
I20260709 12:59:38.329587  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:38.337185  3608 raft_consensus.cc:493] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:59:38.342742  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 12:59:38.304993  3635 log.cc:927] Time spent T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.050s	user 0.000s	sys 0.001s
I20260709 12:59:38.262928  3531 raft_consensus.cc:515] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:38.277266  3639 raft_consensus.cc:515] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:38.285411  3598 leader_election.cc:290] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:38.423859  3416 leader_election.cc:290] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 3 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:38.338150  3425 raft_consensus.cc:3060] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:59:38.290408  3497 leader_election.cc:290] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:38.338675  3600 leader_election.cc:290] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 3 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:38.292539  3464 raft_consensus.cc:493] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:59:38.339092  3544 leader_election.cc:290] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 3 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:38.432449  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:38.532255  1677 raft_consensus.cc:2410] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for term 4 because replica has last-logged OpId of term: 2 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260709 12:59:38.540171  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 12:59:38.543195  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:38.543773  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:38.544039  2334 raft_consensus.cc:2468] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 3.
I20260709 12:59:38.545193  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:38.344167  3491 leader_election.cc:290] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:38.547564  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:38.548902  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:59:38.554299  1677 raft_consensus.cc:3060] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Advancing to term 3
I20260709 12:59:38.554728  1672 raft_consensus.cc:1240] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260709 12:59:38.555691  1599 raft_consensus.cc:2393] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 3: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 12:59:38.528584  3464 raft_consensus.cc:3060] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:59:38.519484  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 12:59:38.387287  3633 raft_consensus.cc:493] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:38.559105  3558 raft_consensus.cc:493] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 12:59:38.535060  3608 raft_consensus.cc:3060] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:59:38.498600  3573 leader_election.cc:290] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:38.448860  3505 consensus_queue.cc:1059] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.138s
I20260709 12:59:38.525241  3295 raft_consensus.cc:515] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:38.573364  1674 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52766 (request call id 1920) took 2572 ms. Trace:
I20260709 12:59:38.488941  3631 consensus_queue.cc:1059] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.113s
I20260709 12:59:38.581465  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:38.579052  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:38.583299  2334 raft_consensus.cc:2468] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 4.
I20260709 12:59:38.567772  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 12:59:38.590992  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:38.574536  1674 rpcz_store.cc:276] 0709 12:59:35.996891 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:35.997045 (+   154us) service_pool.cc:224] Handling call
0709 12:59:35.997456 (+   411us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:59:35.997902 (+   446us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:59:35.997953 (+    51us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:59:36.301677 (+303724us) spinlock_profiling.cc:243] Waited 1.46 ms on lock 0x7b180026f500. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f42630432a6 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4264a5ba11 00007f4264a6b372 00007f426619da01 00007f426616243a
0709 12:59:37.073767 (+772090us) spinlock_profiling.cc:243] Waited 2.4 ms on lock 0x7b2c00010828. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4264a5ba11 00007f4264a6b372 00007f426619da01 00007f426616243a 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2
0709 12:59:37.074365 (+   598us) log.cc:844] Serialized 26 byte log entry
0709 12:59:37.253958 (+179593us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:59:37.361622 (+107664us) log.cc:844] Serialized 14 byte log entry
0709 12:59:37.361946 (+   324us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:37.362061 (+   115us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:59:38.568664 (+1206603us) raft_consensus.cc:1685] finished
0709 12:59:38.568701 (+    37us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:59:38.568895 (+   194us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1805,"prepare.queue_time_us":107018,"prepare.run_cpu_time_us":66,"prepare.run_wall_time_us":67,"raft.queue_time_us":1988,"raft.run_cpu_time_us":339,"raft.run_wall_time_us":5997,"spinlock_wait_cycles":8874752,"thread_start_us":179092,"threads_started":1,"wal-append.queue_time_us":499400}
I20260709 12:59:38.592269  3532 leader_election.cc:290] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 12:59:38.592948  3655 log.cc:927] Time spent T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.086s	user 0.000s	sys 0.001s
I20260709 12:59:38.677243  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:38.580344  3573 raft_consensus.cc:3055] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:59:38.737025  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:38.677688  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:59:38.740267  3573 raft_consensus.cc:740] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
W20260709 12:59:38.741606  3572 consensus_meta.cc:287] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.576s	user 0.012s	sys 0.013s
I20260709 12:59:38.626564  2334 raft_consensus.cc:1275] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:59:38.737820  1677 raft_consensus.cc:2468] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 3.
I20260709 12:59:38.570789  3558 raft_consensus.cc:3060] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:59:38.676682  3466 raft_consensus.cc:493] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:38.664224  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:38.743992  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:38.726830  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
W20260709 12:59:38.727288  1754 consensus_meta.cc:287] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.550s	user 0.013s	sys 0.000s
I20260709 12:59:38.750083  1754 raft_consensus.cc:2468] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 6.
I20260709 12:59:38.746371  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:38.595434  3639 leader_election.cc:290] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:38.737452  3645 raft_consensus.cc:2749] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625) with higher term. Message: Invalid argument: T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 3. Current term is 4.
I20260709 12:59:38.747900  1748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43156 (request call id 307) took 2655 ms. Trace:
I20260709 12:59:38.751828  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:59:38.532495  1675 rpcz_store.cc:276] 0709 12:59:35.777576 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:35.780621 (+  3045us) service_pool.cc:224] Handling call
0709 12:59:38.248584 (+2467963us) spinlock_profiling.cc:243] Waited 480 ms on lock 0x7b54000591d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f426615c853 00007f426b12a60a 00007f42646b07d4 00007f42646b0762 00007f42646b0691 00007f42646b060c 00007f42646af8c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 12:59:38.270392 (+ 21808us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1103924096}
I20260709 12:59:38.753247  1674 raft_consensus.cc:3060] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:59:38.755182  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 12:59:38.593583  3468 consensus_meta.cc:287] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.539s	user 0.014s	sys 0.000s
I20260709 12:59:38.755825  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:38.761001  1596 raft_consensus.cc:2468] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 4.
I20260709 12:59:38.761320  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:38.757323  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 12:59:38.757797  1754 raft_consensus.cc:2468] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 2.
I20260709 12:59:38.758594  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:59:38.762316  2335 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39726 (request call id 642) took 2906 ms. Trace:
I20260709 12:59:38.762895  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:38.601608  3660 raft_consensus.cc:493] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 12:59:38.676573  3491 raft_consensus.cc:493] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 12:59:38.752637  1748 rpcz_store.cc:276] 0709 12:59:36.092995 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:36.093155 (+   160us) service_pool.cc:224] Handling call
0709 12:59:36.093599 (+   444us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:59:36.094084 (+   485us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:59:36.094131 (+    47us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:59:36.099629 (+  5498us) log.cc:844] Serialized 26 byte log entry
0709 12:59:38.163914 (+2064285us) spinlock_profiling.cc:243] Waited 4.82 ms on lock 0x7b18001827a0. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f4263057e07 00007f426beb2ab9 00007f426307469b 00007f426307293f 00007f4263074c02 00007f4265cfc20b 00007f4265d01d23 00007f4265d020d1 00007f426611da0f
0709 12:59:38.190708 (+ 26794us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:59:38.190765 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:38.191034 (+   269us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:59:38.743181 (+552147us) raft_consensus.cc:1685] finished
0709 12:59:38.743200 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:59:38.747811 (+  4611us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":299,"raft.run_cpu_time_us":237,"raft.run_wall_time_us":25038,"spinlock_wait_cycles":11076992,"thread_start_us":26570,"threads_started":1,"wal-append.queue_time_us":2176669}
I20260709 12:59:38.779634  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:38.764526  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 12:59:38.575500  3633 raft_consensus.cc:515] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:38.596064  3505 raft_consensus.cc:3055] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:59:38.758621  3594 leader_election.cc:290] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:38.773890  3649 raft_consensus.cc:515] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:38.766642  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:38.768296  3645 raft_consensus.cc:493] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:38.788481  3667 raft_consensus.cc:493] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:38.788367  3645 raft_consensus.cc:515] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:38.789127  3667 raft_consensus.cc:515] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:38.769315  3654 leader_election.cc:290] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:38.774741  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:38.775209  2329 raft_consensus.cc:2393] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 3: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 12:59:38.775468  1674 raft_consensus.cc:2468] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 4.
I20260709 12:59:38.775697  3415 raft_consensus.cc:3055] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:59:38.672173  3631 raft_consensus.cc:493] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 12:59:38.796433  3631 raft_consensus.cc:515] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:38.763329  2335 rpcz_store.cc:276] 0709 12:59:35.856366 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:35.856504 (+   138us) service_pool.cc:224] Handling call
0709 12:59:35.856899 (+   395us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:59:35.879561 (+ 22662us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:59:35.879608 (+    47us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:59:35.888333 (+  8725us) log.cc:844] Serialized 26 byte log entry
0709 12:59:36.500625 (+612292us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:59:36.500687 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:38.748614 (+2247927us) spinlock_profiling.cc:243] Waited 22.1 ms on lock 0x7b540065e0d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 12:59:38.748669 (+    55us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:59:38.748716 (+    47us) raft_consensus.cc:1685] finished
0709 12:59:38.748728 (+    12us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:59:38.762213 (+ 13485us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":641,"raft.run_cpu_time_us":213,"raft.run_wall_time_us":7396,"spinlock_wait_cycles":50835200,"thread_start_us":588390,"threads_started":1,"wal-append.queue_time_us":915066,"wal-append.run_cpu_time_us":2834,"wal-append.run_wall_time_us":1033797}
I20260709 12:59:38.796931  3645 leader_election.cc:290] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:38.799163  2330 raft_consensus.cc:3060] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:59:38.793845  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:59:38.803987  3531 leader_election.cc:290] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 3 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:38.794134  3648 leader_election.cc:290] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:38.808380  3631 leader_election.cc:290] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 12:59:38.809191  3499 consensus_meta.cc:287] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.557s	user 0.012s	sys 0.000s
I20260709 12:59:38.809697  3499 raft_consensus.cc:515] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:38.810544  3425 raft_consensus.cc:515] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 12:59:38.812232  3634 consensus_meta.cc:287] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.549s	user 0.007s	sys 0.002s
I20260709 12:59:38.812837  3425 leader_election.cc:290] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 4 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:38.812672  3634 raft_consensus.cc:515] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:38.741442  3573 consensus_queue.cc:260] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:38.745500  3575 leader_election.cc:290] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:38.780716  1748 raft_consensus.cc:2393] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 6: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 12:59:38.780980  2334 raft_consensus.cc:2376] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 4. Re-sending same reply.
I20260709 12:59:38.787251  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:38.678534  3619 leader_election.cc:290] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:38.749524  3572 raft_consensus.cc:515] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 12:59:38.782125  3525 consensus_meta.cc:287] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.725s	user 0.012s	sys 0.009s
W20260709 12:59:38.823674  2332 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39738 (request call id 541) took 2310 ms (client timeout 2823 ms). Trace:
I20260709 12:59:38.748008  3466 raft_consensus.cc:515] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:38.793223  1754 raft_consensus.cc:2468] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 3.
W20260709 12:59:38.824421  2332 rpcz_store.cc:269] 0709 12:59:36.506072 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:36.506292 (+   220us) service_pool.cc:224] Handling call
0709 12:59:38.798835 (+2292543us) spinlock_profiling.cc:243] Waited 1.01 s on lock 0x7b54005bca54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 12:59:38.816633 (+ 17798us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2328499072}
I20260709 12:59:38.827404  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:38.828295  1748 raft_consensus.cc:3060] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Advancing to term 3
I20260709 12:59:38.828428  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:38.829355  1754 raft_consensus.cc:2468] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 2.
I20260709 12:59:38.829967  1754 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55840 (request call id 1962) took 1165 ms. Trace:
I20260709 12:59:38.830376  1754 rpcz_store.cc:276] 0709 12:59:37.664388 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:38.828069 (+1163681us) service_pool.cc:224] Handling call
0709 12:59:38.829880 (+  1811us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:59:38.832276  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:38.787695  3505 raft_consensus.cc:740] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 12:59:38.833273  1597 raft_consensus.cc:2393] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 3: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 12:59:38.816931  3415 raft_consensus.cc:740] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 12:59:38.835594  3295 leader_election.cc:290] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 6 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:38.785928  3491 raft_consensus.cc:515] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:38.834792  3525 raft_consensus.cc:515] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:38.839857  3633 leader_election.cc:290] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:38.792320  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:38.840734  3521 leader_election.cc:290] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:38.843329  3649 leader_election.cc:290] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 3 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:38.764959  3468 raft_consensus.cc:515] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } }
I20260709 12:59:38.846964  1754 raft_consensus.cc:2393] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 4: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 12:59:38.851655  3667 leader_election.cc:290] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:38.850338  3505 consensus_queue.cc:260] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:38.852041  3548 leader_election.cc:290] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:38.855763  1603 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40154 (request call id 1648) took 1297 ms. Trace:
I20260709 12:59:38.890489  1603 rpcz_store.cc:276] 0709 12:59:37.558418 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:37.559569 (+  1151us) service_pool.cc:224] Handling call
0709 12:59:38.855491 (+1295922us) spinlock_profiling.cc:243] Waited 234 ms on lock 0x7b54002157d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 12:59:38.855684 (+   193us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":538385792}
I20260709 12:59:38.891245  3505 raft_consensus.cc:3060] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:59:38.857528  1750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55840 (request call id 1947) took 2761 ms. Trace:
W20260709 12:59:38.894013  1571 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:59:38.895711  1571 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:59:38.895820  3525 leader_election.cc:290] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 3 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 12:59:38.896366  1571 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:59:38.897565  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:38.898736  3415 consensus_queue.cc:260] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:38.900642  3578 raft_consensus.cc:3055] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:59:38.901023  1675 raft_consensus.cc:2393] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 4: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 12:59:38.901190  3578 raft_consensus.cc:740] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 12:59:38.904886  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:38.907413  3499 leader_election.cc:290] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 3 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:38.908490  3595 leader_election.cc:290] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 12:59:38.909858  1745 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55848 (request call id 1684) took 2883 ms (client timeout 2698 ms). Trace:
W20260709 12:59:38.910315  1745 rpcz_store.cc:269] 0709 12:59:36.026421 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:36.026711 (+   290us) service_pool.cc:224] Handling call
0709 12:59:38.909557 (+2882846us) spinlock_profiling.cc:243] Waited 1.01 s on lock 0x7b54001d5bd4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 12:59:38.909739 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2315939712}
I20260709 12:59:38.911890  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:38.912866  1745 raft_consensus.cc:2393] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 2: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 12:59:38.913642  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 12:59:38.860954  2299 connection.cc:256] RPC call timeout handler was delayed by -1.2820964549999871s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2585s for a response
W20260709 12:59:38.926692  2299 connection.cc:256] RPC call timeout handler was delayed by -1.1159564789999763s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1694s for a response
W20260709 12:59:38.927084  2299 connection.cc:256] RPC call timeout handler was delayed by -1.021456478999994s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1799s for a response
I20260709 12:59:38.928282  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:38.929060  3468 leader_election.cc:290] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 4 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:38.929550  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 12:59:38.930203  2334 raft_consensus.cc:3060] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:59:38.931931  1751 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55840 (request call id 1941) took 3809 ms. Trace:
I20260709 12:59:38.932312  1751 rpcz_store.cc:276] 0709 12:59:35.122498 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:35.315657 (+193159us) service_pool.cc:224] Handling call
0709 12:59:35.316097 (+   440us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:59:35.383977 (+ 67880us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:59:35.384009 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:59:35.393631 (+  9622us) log.cc:844] Serialized 26 byte log entry
0709 12:59:36.270767 (+877136us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:59:36.971185 (+700418us) log.cc:844] Serialized 14 byte log entry
0709 12:59:37.198103 (+226918us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:37.200167 (+  2064us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:59:38.896512 (+1696345us) spinlock_profiling.cc:243] Waited 27.5 ms on lock 0x7b240024b1c8. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f4262efb9c4 00007f4262ef9acb 00007f42630b4dc9 00007f42630b4a55 00007f42630b3fb0 00007f4265d2a488 00007f4265d2abe8 00007f4265d02398 00007f4265cfd76a 00007f4265cfcbb6
0709 12:59:38.931562 (+ 35050us) raft_consensus.cc:1685] finished
0709 12:59:38.931707 (+   145us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:59:38.931862 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":45246,"prepare.queue_time_us":867645,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":53,"raft.queue_time_us":186173,"raft.run_cpu_time_us":489,"raft.run_wall_time_us":47463,"spinlock_wait_cycles":63197952,"thread_start_us":1687351,"threads_started":3,"wal-append.queue_time_us":1038823}
I20260709 12:59:38.864320  2337 raft_consensus.cc:2393] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 3: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 12:59:38.867777  1598 raft_consensus.cc:2393] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 6: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 12:59:38.880767  1748 raft_consensus.cc:2468] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 3.
I20260709 12:59:38.935691  1748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55840 (request call id 1964) took 1216 ms. Trace:
I20260709 12:59:38.936079  1748 rpcz_store.cc:276] 0709 12:59:37.719115 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:38.827041 (+1107926us) service_pool.cc:224] Handling call
0709 12:59:38.935571 (+108530us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:59:38.871426  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:38.883497  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:38.937117  1753 raft_consensus.cc:2468] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 4.
I20260709 12:59:38.937706  1753 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55840 (request call id 1952) took 2281 ms. Trace:
I20260709 12:59:38.938097  1753 rpcz_store.cc:276] 0709 12:59:36.656391 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:38.883090 (+2226699us) service_pool.cc:224] Handling call
0709 12:59:38.937626 (+ 54536us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:59:38.780095  3660 raft_consensus.cc:515] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:38.885675  1754 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55840 (request call id 1961) took 1221 ms. Trace:
I20260709 12:59:38.869186  2329 raft_consensus.cc:1275] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:59:38.939033  1754 rpcz_store.cc:276] 0709 12:59:37.664071 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:38.831936 (+1167865us) service_pool.cc:224] Handling call
0709 12:59:38.885595 (+ 53659us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:59:38.940364  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 12:59:38.941465  3660 leader_election.cc:290] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:38.941941  3572 leader_election.cc:290] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 6 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:38.942975  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 12:59:38.944609  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 712899us
I20260709 12:59:38.944852  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:38.893760  1750 rpcz_store.cc:276] 0709 12:59:36.096539 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:36.510362 (+413823us) service_pool.cc:224] Handling call
0709 12:59:36.510836 (+   474us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:59:36.544387 (+ 33551us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:59:36.544423 (+    36us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:59:36.550873 (+  6450us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:59:36.550946 (+    73us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:38.857225 (+2306279us) spinlock_profiling.cc:243] Waited 33.2 ms on lock 0x7b54000bd954. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 12:59:38.857283 (+    58us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:59:38.857456 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":76450560}
I20260709 12:59:38.947841  3634 leader_election.cc:290] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 2 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:38.868150  3466 leader_election.cc:290] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:38.947947  3608 raft_consensus.cc:515] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:38.948734  1599 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40154 (request call id 1650) took 1310 ms. Trace:
I20260709 12:59:38.975615  1599 rpcz_store.cc:276] 0709 12:59:37.638046 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:37.639151 (+  1105us) service_pool.cc:224] Handling call
0709 12:59:38.948501 (+1309350us) spinlock_profiling.cc:243] Waited 605 ms on lock 0x7b54004222d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 12:59:38.948663 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1391544192}
I20260709 12:59:38.950052  1679 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57092 (request call id 397) took 2684 ms. Trace:
I20260709 12:59:38.924191  3573 raft_consensus.cc:3060] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:59:38.889350  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:38.950412  3491 leader_election.cc:290] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:38.954018  2299 leader_election.cc:400] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 3 election: Vote denied by peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261) with higher term. Message: Invalid argument: T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 for earlier term 3. Current term is 4.
I20260709 12:59:38.955518  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:38.906148  2328 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49082 (request call id 482) took 1095 ms. Trace:
I20260709 12:59:38.914265  1745 raft_consensus.cc:2468] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 2.
I20260709 12:59:38.958715  2334 raft_consensus.cc:2468] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 3.
I20260709 12:59:38.959323  3578 consensus_queue.cc:260] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 12:59:38.971666  3602 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 2.040s	user 0.047s	sys 0.040s
W20260709 12:59:38.994835  3602 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 2.064s	user 0.049s	sys 0.042s
I20260709 12:59:38.996420  3444 leader_election.cc:290] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:38.917781  3464 raft_consensus.cc:515] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 12:59:38.975100  3622 consensus_meta.cc:287] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.778s	user 0.022s	sys 0.007s
I20260709 12:59:38.984997  3558 raft_consensus.cc:515] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:39.008591  3608 leader_election.cc:290] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 3 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:39.009847  3558 leader_election.cc:290] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 3 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:38.975103  3634 raft_consensus.cc:493] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:39.011663  3634 raft_consensus.cc:515] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:38.986960  1751 raft_consensus.cc:2468] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 2.
I20260709 12:59:39.015502  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:39.016258  1751 raft_consensus.cc:3060] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:59:38.987402  2299 leader_election.cc:403] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
W20260709 12:59:39.017836  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 63964.6us
I20260709 12:59:39.018481  2299 leader_election.cc:304] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:39.019624  2299 leader_election.cc:304] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: af520b67945d443b901bbeca37cc5701
I20260709 12:59:38.974882  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:39.023056  2331 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49082 (request call id 481) took 1212 ms. Trace:
I20260709 12:59:39.023922  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:38.989494  1745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43156 (request call id 319) took 1153 ms. Trace:
I20260709 12:59:38.990218  2332 raft_consensus.cc:2468] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 4.
I20260709 12:59:39.025944  1745 rpcz_store.cc:276] 0709 12:59:37.836401 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:38.913393 (+1076992us) service_pool.cc:224] Handling call
0709 12:59:38.989422 (+ 76029us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:59:38.992501  1571 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:59:39.027927  1571 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:59:39.029670  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:39.030498  2332 raft_consensus.cc:2393] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 3: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 12:59:38.974303  1718 leader_election.cc:304] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
W20260709 12:59:39.031248  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 57065.4us
I20260709 12:59:38.993396  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:38.989224  2328 rpcz_store.cc:276] 0709 12:59:37.810821 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:37.814824 (+  4003us) service_pool.cc:224] Handling call
0709 12:59:38.904646 (+1089822us) spinlock_profiling.cc:243] Waited 3.84 ms on lock 0x7b5800178598. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f426303db71 00007f426303da22 00007f426bf56c09 00007f426b0b0ee0 00007f426b17a0ab 00007f426b17a131 00007f426b1280a0 00007f426b127984 00007f42646a26d4 00007f42646a2662
0709 12:59:38.906057 (+  1411us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":8829824}
I20260709 12:59:39.033864  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 12:59:38.999209  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 98713.3us
I20260709 12:59:39.000005  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:39.036303  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:39.036479  1748 raft_consensus.cc:2468] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 3.
W20260709 12:59:39.003434  1747 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55848 (request call id 1698) took 1635 ms (client timeout 1563 ms). Trace:
I20260709 12:59:39.037464  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 12:59:39.037559  1747 rpcz_store.cc:269] 0709 12:59:37.368466 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:38.042403 (+673937us) service_pool.cc:224] Handling call
0709 12:59:39.003151 (+960748us) spinlock_profiling.cc:243] Waited 103 ms on lock 0x7b54000e4054. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 12:59:39.003320 (+   169us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":236520832}
I20260709 12:59:39.038122  1745 raft_consensus.cc:2468] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 1.
I20260709 12:59:39.004132  3464 leader_election.cc:290] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 4 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:39.038300  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:39.010833  1605 raft_consensus.cc:1240] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 2. Current term is 3. Ops: []
I20260709 12:59:39.039662  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:59:38.976990  1679 rpcz_store.cc:276] 0709 12:59:36.265409 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:37.544464 (+1279055us) service_pool.cc:224] Handling call
0709 12:59:37.544770 (+   306us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:59:37.945455 (+400685us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:59:37.945505 (+    50us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:59:37.945572 (+    67us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:59:37.945620 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:38.949757 (+1004137us) spinlock_profiling.cc:243] Waited 400 ms on lock 0x7b54000d7854. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 12:59:38.949805 (+    48us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:59:38.949953 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":920619392}
I20260709 12:59:39.040567  1603 raft_consensus.cc:2393] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 4: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 12:59:39.022009  2336 raft_consensus.cc:2468] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 4.
I20260709 12:59:39.032156  1601 raft_consensus.cc:2468] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 2.
I20260709 12:59:39.032120  1718 leader_election.cc:304] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:39.039269  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:39.043687  1718 leader_election.cc:304] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:59:38.992771  3578 raft_consensus.cc:3060] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:59:39.041982  3468 raft_consensus.cc:2804] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:59:39.045183  3468 raft_consensus.cc:493] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:59:39.045527  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:39.045902  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:39.047371  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:39.047801  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:39.048559  1747 raft_consensus.cc:2393] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 2: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 12:59:39.048816  1569 leader_election.cc:304] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 12:59:39.049494  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:39.049757  1569 leader_election.cc:304] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 12:59:39.050582  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:39.050822  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:39.051134  3564 raft_consensus.cc:2804] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:59:39.031563  3504 raft_consensus.cc:2764] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260709 12:59:39.051740  3572 raft_consensus.cc:2749] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:59:39.026821  3461 raft_consensus.cc:2749] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:59:39.034243  3634 leader_election.cc:290] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:39.034245  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:39.054010  3521 raft_consensus.cc:2749] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:59:39.054150  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:39.054720  3595 raft_consensus.cc:2804] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:59:39.058746  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:39.059731  1569 leader_election.cc:304] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 12:59:39.045668  3468 raft_consensus.cc:3060] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:59:39.065304  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 12:59:39.113354  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 53698.6us
I20260709 12:59:39.113945  2329 raft_consensus.cc:2468] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 2.
I20260709 12:59:39.027242  3646 raft_consensus.cc:3060] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:59:39.115689  1569 leader_election.cc:304] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 12:59:39.117146  3640 leader_election.cc:290] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:39.023633  2331 rpcz_store.cc:276] 0709 12:59:37.810399 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:37.814756 (+  4357us) service_pool.cc:224] Handling call
0709 12:59:37.879806 (+ 65050us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:59:38.170275 (+290469us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:59:38.172016 (+  1741us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:59:38.214907 (+ 42891us) log.cc:844] Serialized 26 byte log entry
0709 12:59:38.290975 (+ 76068us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:59:38.308128 (+ 17153us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:39.015449 (+707321us) spinlock_profiling.cc:243] Waited 128 ms on lock 0x7b5400627ad4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 12:59:39.015529 (+    80us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:59:39.015639 (+   110us) raft_consensus.cc:1685] finished
0709 12:59:39.015655 (+    16us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:59:39.022917 (+  7262us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":416,"raft.queue_time_us":74834,"raft.run_cpu_time_us":216,"raft.run_wall_time_us":7172,"spinlock_wait_cycles":293953280,"thread_start_us":58943,"threads_started":1,"wal-append.queue_time_us":93569}
I20260709 12:59:39.120447  3595 raft_consensus.cc:697] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 3 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
W20260709 12:59:39.120679  1749 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55840 (request call id 1953) took 2464 ms (client timeout 1580 ms). Trace:
I20260709 12:59:39.071727  1598 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40154 (request call id 1647) took 1537 ms. Trace:
I20260709 12:59:39.071852  1717 leader_election.cc:304] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:39.073339  1751 raft_consensus.cc:2468] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 3.
I20260709 12:59:39.074506  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:39.113401  2332 raft_consensus.cc:2393] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 3: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 12:59:39.124142  1602 raft_consensus.cc:2468] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 4.
I20260709 12:59:39.094233  1747 raft_consensus.cc:2468] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 2.
I20260709 12:59:39.095075  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:39.097856  1678 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57092 (request call id 395) took 2833 ms. Trace:
I20260709 12:59:39.124002  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:39.125355  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:39.125329  1678 rpcz_store.cc:276] 0709 12:59:36.264797 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:36.583741 (+318944us) service_pool.cc:224] Handling call
0709 12:59:36.584195 (+   454us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:59:36.672380 (+ 88185us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:59:36.672432 (+    52us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:59:36.672491 (+    59us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:59:36.672550 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:39.076603 (+2404053us) spinlock_profiling.cc:243] Waited 87.9 ms on lock 0x7b5400058cd4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 12:59:39.097432 (+ 20829us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:59:39.097759 (+   327us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":202146048}
I20260709 12:59:39.126216  1747 raft_consensus.cc:3060] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:59:39.099414  3564 raft_consensus.cc:493] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:59:39.127300  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 55547.9us
I20260709 12:59:39.123028  1598 rpcz_store.cc:276] 0709 12:59:37.534069 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:37.534255 (+   186us) service_pool.cc:224] Handling call
0709 12:59:39.071452 (+1537197us) spinlock_profiling.cc:243] Waited 1.32 s on lock 0x7b5400301954. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 12:59:39.071632 (+   180us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3040745856}
I20260709 12:59:39.096664  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:39.130471  1745 raft_consensus.cc:3060] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:59:39.130548  1717 leader_election.cc:304] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:39.132607  1717 leader_election.cc:304] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:39.128288  1751 raft_consensus.cc:2393] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 3: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 12:59:39.096012  2330 raft_consensus.cc:2468] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 3.
I20260709 12:59:39.135344  3634 raft_consensus.cc:2749] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:59:39.129222  3642 raft_consensus.cc:2804] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:59:39.132861  3552 raft_consensus.cc:2804] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:59:39.158004  3552 raft_consensus.cc:493] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:59:39.145661  3521 raft_consensus.cc:2749] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:59:39.158702  3552 raft_consensus.cc:3060] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:59:39.154831  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 12:59:39.162360  3663 raft_consensus.cc:493] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 12:59:39.162990  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:59:39.163816  1599 raft_consensus.cc:3060] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:59:39.163290  3663 raft_consensus.cc:515] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:39.129077  3619 raft_consensus.cc:2749] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 12:59:39.140163  2333 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39726 (request call id 645) took 1273 ms (client timeout 1572 ms). Trace:
I20260709 12:59:39.142864  1717 leader_election.cc:304] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:39.141999  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 12:59:39.157075  1671 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57092 (request call id 401) took 2890 ms (client timeout 2205 ms). Trace:
I20260709 12:59:39.144793  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:39.174480  1751 raft_consensus.cc:2393] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 4: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 12:59:39.155809  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 12:59:39.121443  1749 rpcz_store.cc:269] 0709 12:59:36.656650 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:37.061969 (+405319us) service_pool.cc:224] Handling call
0709 12:59:39.120355 (+2058386us) spinlock_profiling.cc:243] Waited 5.19 ms on lock 0x7b54001d51d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 12:59:39.120519 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":11943296}
I20260709 12:59:39.167309  1569 leader_election.cc:304] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 12:59:39.131951  3564 raft_consensus.cc:3060] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:59:39.195663  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:59:39.196441  1677 raft_consensus.cc:3060] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Advancing to term 6
I20260709 12:59:39.122221  3595 consensus_queue.cc:237] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:39.178025  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:59:39.179136  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 12:59:39.172147  2333 rpcz_store.cc:269] 0709 12:59:37.866270 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:37.867971 (+  1701us) service_pool.cc:224] Handling call
0709 12:59:39.139924 (+1271953us) spinlock_profiling.cc:243] Waited 202 ms on lock 0x7b540065ed54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 12:59:39.140057 (+   133us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":465473920}
I20260709 12:59:39.181362  3634 raft_consensus.cc:493] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
W20260709 12:59:39.173887  1671 rpcz_store.cc:269] 0709 12:59:36.266583 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:36.508678 (+242095us) service_pool.cc:224] Handling call
0709 12:59:39.156786 (+2648108us) spinlock_profiling.cc:243] Waited 1.52 s on lock 0x7b54000d9654. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 12:59:39.156937 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3489811328}
I20260709 12:59:39.186759  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:59:39.204357  1679 raft_consensus.cc:3060] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:59:39.182386  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:39.189131  1569 leader_election.cc:304] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 12:59:39.205961  1673 raft_consensus.cc:2468] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 3.
I20260709 12:59:39.189229  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:39.190137  3663 leader_election.cc:290] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:39.191474  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:39.182093  3521 raft_consensus.cc:2749] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:59:39.192436  3471 raft_consensus.cc:2804] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:59:39.151034  3415 raft_consensus.cc:3060] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:59:39.165198  3642 raft_consensus.cc:493] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:59:39.146018  1746 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55848 (request call id 1681) took 3165 ms (client timeout 2757 ms). Trace:
I20260709 12:59:39.176396  1598 raft_consensus.cc:2468] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 2.
I20260709 12:59:39.186681  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:39.243258  1751 raft_consensus.cc:2468] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 3.
I20260709 12:59:39.243732  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:39.244381  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:39.245205  1751 raft_consensus.cc:2468] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 2.
I20260709 12:59:39.241845  3642 raft_consensus.cc:3060] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:59:39.244428  1598 raft_consensus.cc:2468] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 3.
I20260709 12:59:39.206295  1676 raft_consensus.cc:3060] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Advancing to term 3
I20260709 12:59:39.209223  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:39.210736  3573 raft_consensus.cc:2804] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:59:39.220163  1679 raft_consensus.cc:2468] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 6.
I20260709 12:59:39.218570  1680 raft_consensus.cc:2468] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 3.
W20260709 12:59:39.218237  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 123300us
I20260709 12:59:39.227835  1673 raft_consensus.cc:1275] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:59:39.225880  1677 raft_consensus.cc:2468] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 6.
I20260709 12:59:39.245774  1745 raft_consensus.cc:2468] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 3.
I20260709 12:59:39.239970  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:39.259454  1718 leader_election.cc:304] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:59:39.248229  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:59:39.248390  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:39.261500  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:39.261694  1598 raft_consensus.cc:3060] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:39.262125  1751 raft_consensus.cc:2468] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 2.
I20260709 12:59:39.262351  2332 raft_consensus.cc:2393] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 2: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 12:59:39.261000  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:39.263078  1747 raft_consensus.cc:2468] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 6.
I20260709 12:59:39.264034  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 12:59:39.249126  1569 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:59:39.274034  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:39.275646  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:39.275980  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:39.276688  1674 raft_consensus.cc:2393] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 3: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 12:59:39.277806  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:59:39.278664  1674 raft_consensus.cc:2393] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 3: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 12:59:39.278677  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:59:39.280068  1673 raft_consensus.cc:3060] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:59:39.280328  1598 raft_consensus.cc:2468] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 2.
I20260709 12:59:39.281625  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:59:39.282285  1598 raft_consensus.cc:3060] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:59:39.288658  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:39.289474  1751 raft_consensus.cc:3060] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:59:39.227051  1675 raft_consensus.cc:2468] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 3.
I20260709 12:59:39.293496  1676 raft_consensus.cc:2468] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 3.
I20260709 12:59:39.248217  3471 raft_consensus.cc:697] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 12:59:39.213126  3634 raft_consensus.cc:515] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 12:59:39.306360  1600 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40154 (request call id 1642) took 1851 ms (client timeout 1928 ms). Trace:
I20260709 12:59:39.305940  1747 raft_consensus.cc:2468] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 2.
I20260709 12:59:39.287990  3521 raft_consensus.cc:2804] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:59:39.320605  3521 raft_consensus.cc:493] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 12:59:39.321086  3521 raft_consensus.cc:3060] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:59:39.319118  1600 rpcz_store.cc:269] 0709 12:59:37.455370 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:37.462157 (+  6787us) service_pool.cc:224] Handling call
0709 12:59:39.306090 (+1843933us) spinlock_profiling.cc:243] Waited 73.9 ms on lock 0x7b54000f0d54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 12:59:39.306275 (+   185us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":169879424}
I20260709 12:59:39.324458  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:39.326761  1751 raft_consensus.cc:2468] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 3.
I20260709 12:59:39.328246  1751 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55840 (request call id 1959) took 1665 ms. Trace:
I20260709 12:59:39.328630  1751 rpcz_store.cc:276] 0709 12:59:37.663332 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:39.288367 (+1625035us) service_pool.cc:224] Handling call
0709 12:59:39.328185 (+ 39818us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:59:39.330107  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:39.331032  1751 raft_consensus.cc:2468] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 2.
I20260709 12:59:39.333452  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:39.334246  1751 raft_consensus.cc:3060] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Advancing to term 3
I20260709 12:59:39.293457  3573 raft_consensus.cc:697] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 12:59:39.337276  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:39.341997  1600 raft_consensus.cc:2410] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for term 3 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260709 12:59:39.306936  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:39.331410  2299 leader_election.cc:304] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 12:59:39.338837  1604 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40154 (request call id 1640) took 1885 ms (client timeout 2462 ms). Trace:
I20260709 12:59:39.339056  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
W20260709 12:59:39.342953  1604 rpcz_store.cc:269] 0709 12:59:37.453514 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:37.453685 (+   171us) service_pool.cc:224] Handling call
0709 12:59:39.338620 (+1884935us) spinlock_profiling.cc:243] Waited 473 ms on lock 0x7b54002797d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 12:59:39.338768 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1086967168}
I20260709 12:59:39.343700  1747 raft_consensus.cc:3060] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:59:39.242787  1746 rpcz_store.cc:269] 0709 12:59:35.975760 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:35.979821 (+  4061us) service_pool.cc:224] Handling call
0709 12:59:39.140979 (+3161158us) spinlock_profiling.cc:243] Waited 1.02 s on lock 0x7b540035ff54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 12:59:39.141206 (+   227us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2343647104}
I20260709 12:59:39.291359  1749 raft_consensus.cc:2468] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 3.
I20260709 12:59:39.351747  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:39.353744  1677 raft_consensus.cc:2393] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 3: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 12:59:39.355861  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:39.351125  1599 raft_consensus.cc:2468] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 4.
I20260709 12:59:39.356941  1600 raft_consensus.cc:2368] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 2. Current term is 3.
I20260709 12:59:39.360818  1598 raft_consensus.cc:2468] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 4.
I20260709 12:59:39.363507  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:39.339071  1673 raft_consensus.cc:2468] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 4.
I20260709 12:59:39.364794  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:39.372076  1569 leader_election.cc:304] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 12:59:39.365875  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:39.374852  2299 leader_election.cc:304] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:39.379233  1747 raft_consensus.cc:2468] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 3.
I20260709 12:59:39.380167  1747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43156 (request call id 320) took 1533 ms. Trace:
I20260709 12:59:39.380645  1747 rpcz_store.cc:276] 0709 12:59:37.846685 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:39.338691 (+1492006us) service_pool.cc:224] Handling call
0709 12:59:39.380067 (+ 41376us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:59:39.381822  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:39.382521  1747 raft_consensus.cc:2468] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 2.
I20260709 12:59:39.383595  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:39.384872  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:39.385782  1746 raft_consensus.cc:2393] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 4: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 12:59:39.385994  3633 raft_consensus.cc:2749] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:59:39.386677  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:39.387921  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:39.389011  3634 leader_election.cc:290] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:39.384253  1747 raft_consensus.cc:2468] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 2.
I20260709 12:59:39.357734  3461 raft_consensus.cc:2749] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:59:39.368396  1749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55848 (request call id 1703) took 1172 ms. Trace:
I20260709 12:59:39.389494  3468 raft_consensus.cc:515] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:39.390697  1569 leader_election.cc:304] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 12:59:39.391000  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:39.393018  2299 leader_election.cc:304] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 12:59:39.393081  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:39.398414  1569 leader_election.cc:304] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 12:59:39.393404  2332 raft_consensus.cc:1275] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 12:59:39.398836  1745 raft_consensus.cc:2368] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for earlier term 3. Current term is 4.
I20260709 12:59:39.399672  1569 leader_election.cc:304] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 12:59:39.381043  3653 raft_consensus.cc:2804] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:59:39.402933  3619 consensus_queue.cc:1048] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:59:39.403270  3653 raft_consensus.cc:493] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 12:59:39.401182  1569 leader_election.cc:304] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 12:59:39.400420  1747 raft_consensus.cc:2393] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 3: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 12:59:39.406037  1747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43156 (request call id 328) took 1018 ms. Trace:
I20260709 12:59:39.410465  3468 leader_election.cc:290] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 4 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:39.410933  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:39.400712  3268 raft_consensus.cc:2804] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:59:39.412029  3268 raft_consensus.cc:493] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:59:39.404376  3295 raft_consensus.cc:2804] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:59:39.406868  3624 consensus_queue.cc:1048] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.006s
I20260709 12:59:39.412510  3268 raft_consensus.cc:3060] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:59:39.412729  3295 raft_consensus.cc:697] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 12:59:39.406313  1747 rpcz_store.cc:276] 0709 12:59:38.387352 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:39.390729 (+1003377us) service_pool.cc:224] Handling call
0709 12:59:39.405981 (+ 15252us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:59:39.415477  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 12:59:39.406909  3584 raft_consensus.cc:2804] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:59:39.415988  2298 leader_election.cc:304] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:39.395071  1749 rpcz_store.cc:276] 0709 12:59:38.195779 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:39.202601 (+1006822us) service_pool.cc:224] Handling call
0709 12:59:39.359395 (+156794us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:59:39.418215  3584 raft_consensus.cc:697] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 12:59:39.431226  3584 consensus_queue.cc:237] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:39.405212  3573 consensus_queue.cc:237] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:39.407429  3653 raft_consensus.cc:3060] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:59:39.439087  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:39.415660  3295 consensus_queue.cc:237] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:39.441377  3646 raft_consensus.cc:2749] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261) with higher term. Message: Invalid argument: T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 for earlier term 3. Current term is 4.
I20260709 12:59:39.419696  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:39.421010  3518 tablet_bootstrap.cc:492] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20260709 12:59:39.443185  3518 tablet_bootstrap.cc:492] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1: Bootstrap complete.
I20260709 12:59:39.444142  3518 ts_tablet_manager.cc:1403] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1: Time spent bootstrapping tablet: real 2.414s	user 0.049s	sys 0.055s
I20260709 12:59:39.407694  3461 raft_consensus.cc:2804] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:59:39.445075  3461 raft_consensus.cc:697] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 4 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 12:59:39.446015  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:39.446859  1748 raft_consensus.cc:2376] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 3. Re-sending same reply.
I20260709 12:59:39.447268  1597 raft_consensus.cc:1275] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 5. (term mismatch)
I20260709 12:59:39.447702  1597 pending_rounds.cc:85] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a: Aborting all ops after (but not including) 4
I20260709 12:59:39.447917  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:39.448868  1748 raft_consensus.cc:2410] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 12:59:39.449751  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:39.447954  1597 pending_rounds.cc:107] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a: Aborting uncommitted NO_OP operation due to leader change: 2.5
I20260709 12:59:39.450760  1597 raft_consensus.cc:2889] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:59:39.446112  3461 consensus_queue.cc:237] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:39.452775  1748 raft_consensus.cc:3060] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:59:39.421262  3564 raft_consensus.cc:515] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:39.422356  3464 raft_consensus.cc:2749] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:59:39.422374  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:39.422627  3499 consensus_queue.cc:1059] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:59:39.455796  3653 raft_consensus.cc:515] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:39.456223  3564 leader_election.cc:290] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 4 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:39.456347  3499 raft_consensus.cc:3055] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:59:39.456821  3499 raft_consensus.cc:740] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 12:59:39.457095  1746 raft_consensus.cc:2376] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 6. Re-sending same reply.
I20260709 12:59:39.455844  2332 raft_consensus.cc:2410] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for term 4 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260709 12:59:39.423659  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:39.459832  1600 raft_consensus.cc:2393] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 3: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 12:59:39.457942  3499 consensus_queue.cc:260] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:39.462074  3499 raft_consensus.cc:3060] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:59:39.421096  3358 leader_election.cc:290] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:39.380473  3471 consensus_queue.cc:237] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:39.450165  3518 raft_consensus.cc:359] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 2 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:39.480716  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 12:59:39.488914  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 738965us
I20260709 12:59:39.490684  1747 raft_consensus.cc:2468] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 4.
I20260709 12:59:39.494001  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 12:59:39.711349  3665 raft_consensus.cc:493] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:39.458205  3552 raft_consensus.cc:515] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:39.717479  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:39.588943  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 12:59:39.718291  2334 raft_consensus.cc:2468] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 2.
I20260709 12:59:39.718847  2331 raft_consensus.cc:2393] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 3: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 12:59:39.719043  3552 leader_election.cc:290] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 5 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 12:59:39.496855  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 95726.9us
I20260709 12:59:39.714416  3665 raft_consensus.cc:515] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 12:59:39.610174  2300 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:59:39.721052  1646 leader_election.cc:304] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:59:39.567530  3358 raft_consensus.cc:493] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 12:59:39.721609  2298 leader_election.cc:304] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 12:59:39.611409  3653 leader_election.cc:290] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 4 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:39.458225  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:39.697513  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:39.703644  3660 raft_consensus.cc:493] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 12:59:39.614372  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:39.739369  1753 raft_consensus.cc:2468] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 3.
I20260709 12:59:39.616031  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:39.481564  3268 raft_consensus.cc:515] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:39.494716  2332 raft_consensus.cc:1275] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:59:39.720011  1569 leader_election.cc:304] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 12:59:39.616122  3642 raft_consensus.cc:515] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:39.696666  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 12:59:39.720681  2301 leader_election.cc:304] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:39.743263  2337 raft_consensus.cc:3060] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:59:39.531193  3444 consensus_queue.cc:1048] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.192s
I20260709 12:59:39.722236  3358 raft_consensus.cc:515] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:39.617523  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:39.551499  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:39.591331  1672 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52766 (request call id 1948) took 1204 ms. Trace:
I20260709 12:59:39.528416  1751 raft_consensus.cc:2468] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 3.
I20260709 12:59:39.724128  1646 leader_election.cc:304] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:59:39.724469  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 12:59:39.730355  2336 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49082 (request call id 483) took 1919 ms (client timeout 1571 ms). Trace:
I20260709 12:59:39.730991  1717 leader_election.cc:304] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:59:39.529541  1748 raft_consensus.cc:2468] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 3.
I20260709 12:59:39.753131  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:39.753969  1748 raft_consensus.cc:2468] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 3.
I20260709 12:59:39.754204  2301 leader_election.cc:304] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:39.552440  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:39.754698  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:39.755265  1754 raft_consensus.cc:2468] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 3.
I20260709 12:59:39.755857  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
W20260709 12:59:39.755937  1754 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55840 (request call id 1969) took 1213 ms (client timeout 1563 ms). Trace:
I20260709 12:59:39.756541  1751 raft_consensus.cc:3060] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Advancing to term 6
I20260709 12:59:39.757668  3543 consensus_queue.cc:1048] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
W20260709 12:59:39.756542  1754 rpcz_store.cc:269] 0709 12:59:38.542548 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:39.552077 (+1009529us) service_pool.cc:224] Handling call
0709 12:59:39.755848 (+203771us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:59:39.760447  1752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55840 (request call id 1949) took 3626 ms. Trace:
I20260709 12:59:39.760929  3687 raft_consensus.cc:2804] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260709 12:59:39.736377  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 253405us
W20260709 12:59:39.739871  1753 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55840 (request call id 1970) took 1192 ms (client timeout 1580 ms). Trace:
I20260709 12:59:39.760849  1752 rpcz_store.cc:276] 0709 12:59:36.134352 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:39.759444 (+3625092us) service_pool.cc:224] Handling call
0709 12:59:39.759865 (+   421us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:59:39.760133 (+   268us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:59:39.760159 (+    26us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:59:39.760202 (+    43us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:59:39.760238 (+    36us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:39.760290 (+    52us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:59:39.760384 (+    94us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:59:39.761590  1753 rpcz_store.cc:269] 0709 12:59:38.547780 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:39.613967 (+1066187us) service_pool.cc:224] Handling call
0709 12:59:39.739806 (+125839us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:59:39.740697  1750 raft_consensus.cc:2393] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 4: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 12:59:39.763587  1750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55840 (request call id 1974) took 1196 ms. Trace:
I20260709 12:59:39.753583  1646 leader_election.cc:304] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:59:39.763978  1750 rpcz_store.cc:276] 0709 12:59:38.567419 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:39.615682 (+1048263us) service_pool.cc:224] Handling call
0709 12:59:39.763522 (+147840us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:59:39.764636  1646 leader_election.cc:304] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:59:39.765725  1750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55848 (request call id 1700) took 2357 ms. Trace:
W20260709 12:59:39.752458  2336 rpcz_store.cc:269] 0709 12:59:37.811218 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:37.811378 (+   160us) service_pool.cc:224] Handling call
0709 12:59:39.730158 (+1918780us) spinlock_profiling.cc:243] Waited 1.16 s on lock 0x7b5400605254. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 12:59:39.730265 (+   107us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2672031104}
W20260709 12:59:39.733031  1643 connection.cc:256] RPC call timeout handler was delayed by -1.4156948340000213s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1513s for a response
W20260709 12:59:39.767066  1643 connection.cc:256] RPC call timeout handler was delayed by -1.4120948340000155s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1517s for a response
I20260709 12:59:39.767261  1672 rpcz_store.cc:276] 0709 12:59:38.387354 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:38.777517 (+390163us) service_pool.cc:224] Handling call
0709 12:59:38.777838 (+   321us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:59:38.778084 (+   246us) raft_consensus.cc:1534] Early marking committed up to index 2
0709 12:59:38.785425 (+  7341us) log.cc:844] Serialized 14 byte log entry
0709 12:59:39.152767 (+367342us) spinlock_profiling.cc:243] Waited 1.59 ms on lock 0x7b1800456040. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f4263057e07 00007f426beb2ab9 00007f426307469b 00007f426307293f 00007f4263074c02 00007f4265cfc20b 00007f4265d01d23 00007f4265d0223a 00007f426616ca2a
0709 12:59:39.590759 (+437992us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:59:39.590830 (+    71us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:59:39.590888 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:39.590984 (+    96us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:59:39.591225 (+   241us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":220619,"prepare.queue_time_us":7193,"prepare.run_cpu_time_us":66,"prepare.run_wall_time_us":67,"raft.queue_time_us":330894,"raft.run_cpu_time_us":135,"raft.run_wall_time_us":137,"spinlock_wait_cycles":3655040,"thread_start_us":217032,"threads_started":1,"wal-append.queue_time_us":578078}
W20260709 12:59:39.770366  1571 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:59:39.765812  1646 leader_election.cc:304] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1
W20260709 12:59:39.772482  1569 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:59:39.733608  3663 raft_consensus.cc:2804] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:59:39.774684  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:39.772495  1752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55840 (request call id 1951) took 3116 ms. Trace:
I20260709 12:59:39.778383  1748 raft_consensus.cc:2393] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 4: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 12:59:39.778308  1752 rpcz_store.cc:276] 0709 12:59:36.656128 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:39.761964 (+3105836us) service_pool.cc:224] Handling call
0709 12:59:39.762250 (+   286us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:59:39.770981 (+  8731us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:59:39.771101 (+   120us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:59:39.771217 (+   116us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:59:39.771301 (+    84us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:39.771957 (+   656us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:59:39.772366 (+   409us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:59:39.767584  1643 connection.cc:256] RPC call timeout handler was delayed by -1.3967948339999907s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1534s for a response
I20260709 12:59:39.780182  1748 raft_consensus.cc:1240] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260709 12:59:39.780977  1748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55840 (request call id 1958) took 2118 ms. Trace:
W20260709 12:59:39.780206  1643 connection.cc:256] RPC call timeout handler was delayed by -1.3850948340000286s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1547s for a response
I20260709 12:59:39.781332  1748 rpcz_store.cc:276] 0709 12:59:37.662876 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:39.779475 (+2116599us) service_pool.cc:224] Handling call
0709 12:59:39.779861 (+   386us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:59:39.780672 (+   811us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:39.780909 (+   237us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:59:39.776466  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 45534.4us
I20260709 12:59:39.531428  3564 raft_consensus.cc:2804] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:59:39.766360  1750 rpcz_store.cc:276] 0709 12:59:37.408445 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:39.764729 (+2356284us) service_pool.cc:224] Handling call
0709 12:59:39.765101 (+   372us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:59:39.765335 (+   234us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:59:39.765372 (+    37us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:59:39.765426 (+    54us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:59:39.765472 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:39.765540 (+    68us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:59:39.765657 (+   117us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:59:39.783584  1717 leader_election.cc:304] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:59:39.777508  3608 raft_consensus.cc:2804] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260709 12:59:39.788800  1571 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:59:39.789659  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 68135.1us
I20260709 12:59:39.790256  1745 raft_consensus.cc:2468] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 2.
W20260709 12:59:39.781705  1643 connection.cc:256] RPC call timeout handler was delayed by -1.3733948340000097s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15600000000000003s for a response
W20260709 12:59:39.791028  1643 connection.cc:256] RPC call timeout handler was delayed by -1.30949483400002s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16310000000000002s for a response
W20260709 12:59:39.791486  1643 connection.cc:256] RPC call timeout handler was delayed by -1.1573948340000015s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18000000000000002s for a response
W20260709 12:59:39.791812  1643 connection.cc:256] RPC call timeout handler was delayed by -1.1159948339999914s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18460000000000001s for a response
I20260709 12:59:39.734676  3518 raft_consensus.cc:740] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Initialized, Role: LEARNER
W20260709 12:59:39.792181  1643 connection.cc:256] RPC call timeout handler was delayed by -1.0952948340000148s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1869s for a response
I20260709 12:59:39.796084  1748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43156 (request call id 316) took 1964 ms. Trace:
W20260709 12:59:39.796537  1643 connection.cc:256] RPC call timeout handler was delayed by -1.0943948340000134s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18700000000000003s for a response
W20260709 12:59:39.797039  1643 connection.cc:256] RPC call timeout handler was delayed by -1.0430948339999873s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1927s for a response
I20260709 12:59:39.797019  1748 rpcz_store.cc:276] 0709 12:59:37.831449 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:39.794955 (+1963506us) service_pool.cc:224] Handling call
0709 12:59:39.795395 (+   440us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:59:39.795646 (+   251us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:59:39.795685 (+    39us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:59:39.795744 (+    59us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:59:39.795796 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:39.795868 (+    72us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:59:39.796004 (+   136us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:59:39.799150  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:39.799980  1676 raft_consensus.cc:2393] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 3: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 12:59:39.801023  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:39.798264  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:59:39.802193  1671 raft_consensus.cc:2393] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 4: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
W20260709 12:59:39.802090  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 47978.4us
I20260709 12:59:39.802987  2301 leader_election.cc:304] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 12:59:39.807036  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:39.807507  1751 raft_consensus.cc:2468] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 6.
I20260709 12:59:39.775981  3653 raft_consensus.cc:2804] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:59:39.810402  1751 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43156 (request call id 321) took 1940 ms. Trace:
I20260709 12:59:39.732893  3414 consensus_queue.cc:1048] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.054s
I20260709 12:59:39.776852  3543 raft_consensus.cc:2804] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:59:39.768517  3687 raft_consensus.cc:493] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 12:59:39.788475  3608 raft_consensus.cc:493] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 12:59:39.792822  1746 raft_consensus.cc:2393] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 3: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 12:59:39.777447  3646 raft_consensus.cc:2804] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:59:39.823163  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:39.825049  1748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43156 (request call id 318) took 1992 ms. Trace:
I20260709 12:59:39.813629  3564 raft_consensus.cc:697] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 12:59:39.887473  1748 rpcz_store.cc:276] 0709 12:59:37.833015 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:39.814276 (+1981261us) service_pool.cc:224] Handling call
0709 12:59:39.824488 (+ 10212us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:59:39.824736 (+   248us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:59:39.824761 (+    25us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:59:39.824802 (+    41us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:59:39.824832 (+    30us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:39.824884 (+    52us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:59:39.824986 (+   102us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:59:39.888083  1571 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:59:39.787876  2330 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49082 (request call id 490) took 1430 ms. Trace:
I20260709 12:59:39.822595  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:39.777711  3663 raft_consensus.cc:493] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 12:59:39.789386  2329 raft_consensus.cc:1275] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:59:39.791038  3504 raft_consensus.cc:2804] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:59:39.827998  2301 leader_election.cc:304] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 12:59:39.745251  3660 raft_consensus.cc:515] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:39.872063  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:59:39.909830  1677 raft_consensus.cc:3060] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:59:39.872910  1745 raft_consensus.cc:1240] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 1. Current term is 2. Ops: []
I20260709 12:59:39.884171  3653 raft_consensus.cc:493] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:59:39.917520  3689 raft_consensus.cc:2749] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:59:39.918295  3653 raft_consensus.cc:3060] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:59:39.886097  3518 consensus_queue.cc:260] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } }
W20260709 12:59:39.921846  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 93930.2us
I20260709 12:59:39.923327  3578 raft_consensus.cc:493] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:39.923923  3578 raft_consensus.cc:515] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:39.889786  3608 raft_consensus.cc:3060] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:59:39.880998  1751 rpcz_store.cc:276] 0709 12:59:37.870065 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:39.809349 (+1939284us) service_pool.cc:224] Handling call
0709 12:59:39.809743 (+   394us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:59:39.809974 (+   231us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:59:39.810012 (+    38us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:59:39.810072 (+    60us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:59:39.810120 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:39.810193 (+    73us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:59:39.810320 (+   127us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:59:39.932699  1751 raft_consensus.cc:1240] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 2. Current term is 3. Ops: [2.5-2.5]
I20260709 12:59:39.933295  1751 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55840 (request call id 1966) took 1758 ms. Trace:
I20260709 12:59:39.933614  1751 rpcz_store.cc:276] 0709 12:59:38.175110 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:39.932171 (+1757061us) service_pool.cc:224] Handling call
0709 12:59:39.932462 (+   291us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:59:39.933091 (+   629us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:39.933249 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:59:39.884064  1750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43156 (request call id 317) took 2051 ms. Trace:
I20260709 12:59:39.822108  3619 raft_consensus.cc:2804] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:59:39.889047  1571 leader_election.cc:304] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 12:59:39.896461  1748 raft_consensus.cc:1240] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260709 12:59:39.889958  3646 raft_consensus.cc:493] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 12:59:39.911561  1745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55848 (request call id 1701) took 2036 ms. Trace:
I20260709 12:59:39.938377  1745 rpcz_store.cc:276] 0709 12:59:37.875294 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:39.872412 (+1997118us) service_pool.cc:224] Handling call
0709 12:59:39.872700 (+   288us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:59:39.911209 (+ 38509us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:39.911483 (+   274us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:59:39.775794  3497 raft_consensus.cc:2804] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:59:39.913519  1596 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147"
dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
mode: GRACEFUL
new_leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
 from {username='slave'} at 127.0.0.1:47752
I20260709 12:59:39.915174  2336 raft_consensus.cc:1275] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 12:59:39.873520  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:59:39.927939  2301 leader_election.cc:304] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 12:59:39.827369  3521 raft_consensus.cc:515] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:39.883653  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:39.884343  3543 raft_consensus.cc:493] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 12:59:39.934686  1750 rpcz_store.cc:276] 0709 12:59:37.832133 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:39.828198 (+1996065us) service_pool.cc:224] Handling call
0709 12:59:39.828506 (+   308us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:59:39.880838 (+ 52332us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:59:39.880870 (+    32us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:59:39.880923 (+    53us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:59:39.880960 (+    37us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:39.883739 (+  2779us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:59:39.883953 (+   214us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:59:39.918062  3564 consensus_queue.cc:237] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 12:59:39.927636  1718 leader_election.cc:336] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Timed out: RequestConsensusVote RPC to 127.0.107.194:46261 timed out after 1.568s (SENT)
I20260709 12:59:39.887625  3687 raft_consensus.cc:3060] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:39.897799  3663 raft_consensus.cc:3060] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:59:39.937942  1748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55840 (request call id 1965) took 1766 ms. Trace:
I20260709 12:59:39.900786  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:39.940186  1745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43156 (request call id 326) took 1595 ms. Trace:
I20260709 12:59:39.963774  1745 rpcz_store.cc:276] 0709 12:59:38.344412 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:39.939463 (+1595051us) service_pool.cc:224] Handling call
0709 12:59:39.939732 (+   269us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:59:39.939906 (+   174us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:59:39.939929 (+    23us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:59:39.939967 (+    38us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:59:39.940000 (+    33us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:39.940046 (+    46us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:59:39.940133 (+    87us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:59:39.965333  1677 raft_consensus.cc:2468] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 4.
I20260709 12:59:39.966005  1745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55840 (request call id 1973) took 1405 ms. Trace:
I20260709 12:59:39.966228  1676 raft_consensus.cc:1240] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 3. Current term is 4. Ops: []
I20260709 12:59:39.943462  3358 leader_election.cc:290] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 12:59:39.947028  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 58093.1us
I20260709 12:59:39.970715  2337 raft_consensus.cc:2468] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 3.
I20260709 12:59:39.913058  1746 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55848 (request call id 1705) took 1375 ms. Trace:
I20260709 12:59:39.958146  1748 rpcz_store.cc:276] 0709 12:59:38.172060 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:39.895949 (+1723889us) service_pool.cc:224] Handling call
0709 12:59:39.896274 (+   325us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:59:39.937669 (+ 41395us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:39.937882 (+   213us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:59:39.941121  1596 raft_consensus.cc:606] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 LEADER]: Received request to transfer leadership to TS 816fadcea8fc4893882f5f9b9f22a03d
I20260709 12:59:39.974915  1748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55840 (request call id 1976) took 1219 ms. Trace:
I20260709 12:59:39.971505  1571 leader_election.cc:304] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 12:59:39.907227  3504 raft_consensus.cc:493] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 12:59:39.957645  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:59:39.928066  3665 leader_election.cc:290] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:39.968950  1745 rpcz_store.cc:276] 0709 12:59:38.560352 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:39.964980 (+1404628us) service_pool.cc:224] Handling call
0709 12:59:39.965346 (+   366us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:59:39.965602 (+   256us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:59:39.965639 (+    37us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:59:39.965695 (+    56us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:59:39.965739 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:39.965808 (+    69us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:59:39.965936 (+   128us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:59:39.975490  1748 rpcz_store.cc:276] 0709 12:59:38.755580 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:39.973992 (+1218412us) service_pool.cc:224] Handling call
0709 12:59:39.974260 (+   268us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:59:39.974501 (+   241us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:59:39.974529 (+    28us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:59:39.974571 (+    42us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:59:39.974609 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:39.974680 (+    71us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:59:39.974821 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:59:39.981660  1748 raft_consensus.cc:1240] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 2. Current term is 3. Ops: []
I20260709 12:59:39.982398  1748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43156 (request call id 329) took 1199 ms. Trace:
I20260709 12:59:39.982683  1745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55840 (request call id 1979) took 1141 ms. Trace:
W20260709 12:59:39.962774  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 197049us
I20260709 12:59:39.948318  3521 leader_election.cc:290] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:39.979732  1571 leader_election.cc:304] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:59:39.949478  3543 raft_consensus.cc:3060] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:59:39.949891  3686 log.cc:927] Time spent T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.081s	user 0.000s	sys 0.000s
I20260709 12:59:39.949260  1749 raft_consensus.cc:2393] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 3: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 12:59:39.991479  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:39.940392  3497 raft_consensus.cc:493] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
W20260709 12:59:40.029287  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 50051.7us
I20260709 12:59:40.044293  1571 leader_election.cc:304] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 12:59:39.983376  1745 rpcz_store.cc:276] 0709 12:59:38.841778 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:39.981608 (+1139830us) service_pool.cc:224] Handling call
0709 12:59:39.981965 (+   357us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:59:39.982279 (+   314us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:59:39.982316 (+    37us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:59:39.982369 (+    53us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:59:39.982416 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:39.982495 (+    79us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:59:39.982618 (+   123us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:59:39.996635  1678 raft_consensus.cc:2393] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 3: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
W20260709 12:59:40.047151  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 263637us
I20260709 12:59:39.948482  3660 leader_election.cc:290] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:39.971794  3689 raft_consensus.cc:2764] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
W20260709 12:59:39.951782  3681 log.cc:927] Time spent T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.070s	user 0.000s	sys 0.001s
I20260709 12:59:40.009389  1500 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35444 (request call id 258) took 6476 ms. Trace:
I20260709 12:59:40.040146  1646 leader_election.cc:304] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 12:59:39.982944  1748 rpcz_store.cc:276] 0709 12:59:38.783290 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:39.981019 (+1197729us) service_pool.cc:224] Handling call
0709 12:59:39.981381 (+   362us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:59:39.982117 (+   736us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:39.982329 (+   212us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:59:39.987802  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:59:40.011034  1502 catalog_manager.cc:5697] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 reported cstate change: term changed from 1 to 2, leader changed from af520b67945d443b901bbeca37cc5701 (127.0.107.195) to 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193). New cstate: current_term: 2 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } }
W20260709 12:59:40.012833  3499 consensus_meta.cc:287] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.550s	user 0.011s	sys 0.000s
I20260709 12:59:40.067678  1671 raft_consensus.cc:2393] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 3: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 12:59:40.047931  1717 leader_election.cc:304] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:59:39.987383  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:40.068334  1748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43156 (request call id 332) took 1225 ms. Trace:
I20260709 12:59:40.068696  1748 rpcz_store.cc:276] 0709 12:59:38.843119 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:40.067393 (+1224274us) service_pool.cc:224] Handling call
0709 12:59:40.067737 (+   344us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:59:40.067976 (+   239us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:59:40.068010 (+    34us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:59:40.068057 (+    47us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:59:40.068093 (+    36us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:40.068146 (+    53us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:59:40.068270 (+   124us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:59:40.069258  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:40.070343  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:40.070963  1571 leader_election.cc:304] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 12:59:40.071413  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:39.951195  3646 raft_consensus.cc:3060] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:59:40.073606  1748 raft_consensus.cc:1240] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 3. Current term is 4. Ops: []
I20260709 12:59:40.073746  1679 raft_consensus.cc:2393] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 3: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 12:59:40.049350  1745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43156 (request call id 331) took 1206 ms. Trace:
I20260709 12:59:40.074524  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:59:40.074674  1745 rpcz_store.cc:276] 0709 12:59:38.842672 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:40.048395 (+1205723us) service_pool.cc:224] Handling call
0709 12:59:40.048763 (+   368us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:59:40.049015 (+   252us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:59:40.049052 (+    37us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:59:40.049109 (+    57us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:59:40.049149 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:40.049198 (+    49us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:59:40.049294 (+    96us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:59:40.075564  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:59:40.007329  3518 ts_tablet_manager.cc:1434] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1: Time spent starting tablet: real 0.561s	user 0.007s	sys 0.014s
W20260709 12:59:40.076417  3518 ts_tablet_manager.cc:1467] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1: Tablet startup took 631ms
W20260709 12:59:40.076882  3518 ts_tablet_manager.cc:1469] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1: Trace:
0709 12:59:33.271110 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:33.271774 (+   664us) service_pool.cc:224] Handling call
0709 12:59:33.336008 (+ 64234us) ts_tablet_manager.cc:934] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1: Initiating tablet copy from peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
0709 12:59:33.585414 (+249406us) tablet_copy_client.cc:368] Tablet copy session begun
0709 12:59:34.032037 (+446623us) tablet_metadata.cc:665] Metadata flushed
0709 12:59:34.032285 (+   248us) tablet_copy_client.cc:497] Wrote new tablet metadata
0709 12:59:34.493665 (+461380us) tablet_copy_client.cc:418] Wrote new consensus metadata
0709 12:59:34.494046 (+   381us) tablet_replica.cc:206] Creating consensus instance
0709 12:59:34.499680 (+  5634us) ts_tablet_manager.cc:989] Replica registered.
0709 12:59:34.504398 (+  4718us) inbound_call.cc:177] Queueing success response
0709 12:59:34.823014 (+318616us) spinlock_profiling.cc:243] Waited 20.1 ms on lock 0x7b4c00332e40. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426442ea20 00007f426443fa29 00007f426444013f 00007f42644042ab 00007f4264431f3c 00007f4264432461 00007f4264432768 00007f426a1b866b 00007f426b0f515b 00007f426b0f4f98 00007f426b0e6322
0709 12:59:37.030159 (+2207145us) ts_tablet_manager.cc:1379] Bootstrapping tablet
0709 12:59:39.446639 (+2416480us) ts_tablet_manager.cc:1435] Starting tablet replica
0709 12:59:39.447353 (+   714us) tablet_replica.cc:261] Starting instrumentation
0709 12:59:39.448050 (+   697us) tablet_replica.cc:280] Starting consensus
Metrics: {"dns_us":69084,"mutex_wait_us":7868,"raft.queue_time_us":35803,"raft.run_cpu_time_us":207,"raft.run_wall_time_us":11550,"spinlock_wait_cycles":46147456,"tablet-copy.queue_time_us":60556,"tablet-download-pool.queue_time_us":222,"tablet-download-pool.run_cpu_time_us":6412,"tablet-download-pool.run_wall_time_us":1412355,"thread_start_us":150798,"threads_started":2}
I20260709 12:59:39.958122  1718 leader_election.cc:304] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 12:59:39.986055  3685 log.cc:927] Time spent T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.090s	user 0.000s	sys 0.000s
W20260709 12:59:39.986686  3682 log.cc:927] Time spent T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.103s	user 0.000s	sys 0.001s
I20260709 12:59:39.950737  3295 raft_consensus.cc:3060] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:59:40.083175  3646 raft_consensus.cc:515] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:40.086019  3646 leader_election.cc:290] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 3 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:40.072919  1717 leader_election.cc:304] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:59:40.087194  3358 raft_consensus.cc:2804] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:59:40.087395  1717 leader_election.cc:304] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:40.087738  3358 raft_consensus.cc:697] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [term 2 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 12:59:40.088672  1717 leader_election.cc:304] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:59:40.088709  3358 consensus_queue.cc:237] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:40.089718  1717 leader_election.cc:400] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445) with higher term. Message: Invalid argument: T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 2. Current term is 3.
I20260709 12:59:40.089991  1717 leader_election.cc:403] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:59:40.090739  3595 raft_consensus.cc:2749] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:59:40.076076  1679 raft_consensus.cc:3060] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Advancing to term 4
I20260709 12:59:40.090807  3468 raft_consensus.cc:2804] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:59:40.091627  3595 raft_consensus.cc:2749] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445) with higher term. Message: Invalid argument: T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 2. Current term is 3.
I20260709 12:59:40.092037  3468 raft_consensus.cc:697] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 4 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 12:59:40.072237  3521 raft_consensus.cc:2804] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:59:40.092592  3521 raft_consensus.cc:493] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 12:59:40.093000  3468 consensus_queue.cc:237] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 12:59:40.093501  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 165875us
W20260709 12:59:40.094204  1718 leader_election.cc:336] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Timed out: RequestConsensusVote RPC to 127.0.107.194:46261 timed out after 1.585s (SENT)
I20260709 12:59:40.093025  3521 raft_consensus.cc:3060] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:59:40.098892  3595 raft_consensus.cc:2749] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:59:40.105405  3521 raft_consensus.cc:515] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:40.066495  1500 rpcz_store.cc:276] 0709 12:59:33.532727 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:33.551860 (+ 19133us) service_pool.cc:224] Handling call
0709 12:59:40.009283 (+6457423us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:59:36.230692 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:59:36.447335 (+216643us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:59:36.455647 (+  8312us) write_op.cc:435] Acquired schema lock
0709 12:59:36.465521 (+  9874us) tablet.cc:662] Decoding operations
0709 12:59:36.590839 (+125318us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:59:36.590986 (+   147us) write_op.cc:641] Partition lock acquired for write op
0709 12:59:36.591037 (+    51us) tablet.cc:685] Acquiring locks for 2 operations
0709 12:59:36.678624 (+ 87587us) tablet.cc:701] Row locks acquired
0709 12:59:36.681258 (+  2634us) write_op.cc:260] PREPARE: finished
0709 12:59:36.841778 (+160520us) write_op.cc:270] Start()
0709 12:59:36.880773 (+ 38995us) write_op.cc:276] Timestamp: P: 1783601976841689 usec, L: 0
0709 12:59:36.880828 (+    55us) op_driver.cc:348] REPLICATION: starting
0709 12:59:36.930690 (+ 49862us) log.cc:844] Serialized 929 byte log entry
0709 12:59:38.845433 (+1914743us) spinlock_profiling.cc:243] Waited 17.3 ms on lock 0x7b4400022b4c. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f4262f9bc85 00007f4262f9c6a2 00007f4265d0e1db 00007f4265d09aaf 00007f42630580ac 00007f426beb2ab9 00007f426307469b 00007f426307293f 00007f4263074c02 00007f4265cfc20b
0709 12:59:39.139061 (+293628us) op_driver.cc:464] REPLICATION: finished
0709 12:59:39.269002 (+129941us) write_op.cc:301] APPLY: starting
0709 12:59:39.269139 (+   137us) tablet.cc:1367] starting BulkCheckPresence
0709 12:59:39.269362 (+   223us) tablet.cc:1370] finished BulkCheckPresence
0709 12:59:39.269416 (+    54us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:59:39.270329 (+   913us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:59:39.270681 (+   352us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0709 12:59:39.270795 (+   114us) write_op.cc:312] APPLY: finished
0709 12:59:39.271174 (+   379us) log.cc:844] Serialized 29 byte log entry
0709 12:59:39.271507 (+   333us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:59:39.271654 (+   147us) write_op.cc:454] Released schema lock
0709 12:59:39.271715 (+    61us) write_op.cc:341] FINISH: Updating metrics
0709 12:59:39.272351 (+   636us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:59:39.272400 (+    49us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":460369,"reported_tablets":7,"child_traces":[["op",{"apply.queue_time_us":129453,"apply.run_cpu_time_us":3205,"apply.run_wall_time_us":3272,"num_ops":2,"prepare.queue_time_us":786591,"prepare.run_cpu_time_us":107364,"prepare.run_wall_time_us":2812625,"raft.queue_time_us":108740,"raft.run_cpu_time_us":34894,"raft.run_wall_time_us":170188,"replication_time_us":2231260,"spinlock_wait_cycles":39785088,"thread_start_us":2721777,"threads_started":4,"wal-append.queue_time_us":2045299}]]}
I20260709 12:59:40.114775  3521 leader_election.cc:290] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:40.120764  1748 raft_consensus.cc:1240] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 3 LEADER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 2. Current term is 3. Ops: [2.5-2.5]
I20260709 12:59:39.955322  3619 raft_consensus.cc:493] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 12:59:39.985885  3668 raft_consensus.cc:2804] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:59:39.984853  3504 raft_consensus.cc:3060] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:59:40.065934  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:39.996491  3633 raft_consensus.cc:2749] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:59:40.127141  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:59:40.138073  1677 raft_consensus.cc:2393] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 4: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 12:59:39.995707  3642 leader_election.cc:290] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 5 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:40.142745  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:59:40.142930  3642 raft_consensus.cc:2804] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:59:40.143436  3642 raft_consensus.cc:493] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 12:59:40.143968  3642 raft_consensus.cc:3060] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:59:40.144018  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:40.145396  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:40.048995  3497 raft_consensus.cc:3060] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:59:39.995070  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:40.040903  3415 raft_consensus.cc:2804] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:59:39.889575  2330 rpcz_store.cc:276] 0709 12:59:38.347480 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:38.448346 (+100866us) service_pool.cc:224] Handling call
0709 12:59:39.749356 (+1301010us) spinlock_profiling.cc:243] Waited 232 ms on lock 0x7b54006ef854. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 12:59:39.777581 (+ 28225us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":532500224}
I20260709 12:59:40.130899  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:40.124574  3529 raft_consensus.cc:2749] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260709 12:59:40.118234  1718 leader_election.cc:336] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Timed out: RequestConsensusVote RPC to 127.0.107.194:46261 timed out after 1.736s (SENT)
I20260709 12:59:40.158022  1718 leader_election.cc:304] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:40.042618  3665 raft_consensus.cc:2804] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:59:40.163537  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:59:39.938995  3653 raft_consensus.cc:515] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:40.065872  3634 raft_consensus.cc:2804] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:59:40.148976  1679 raft_consensus.cc:2468] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 4.
I20260709 12:59:40.153905  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
W20260709 12:59:40.131350  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 91264.3us
I20260709 12:59:40.175071  1646 leader_election.cc:304] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1
W20260709 12:59:40.153085  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 196928us
I20260709 12:59:40.154225  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:40.128798  3619 raft_consensus.cc:3060] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:59:40.160136  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:40.129693  3668 raft_consensus.cc:493] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 12:59:39.973673  1746 rpcz_store.cc:276] 0709 12:59:38.534257 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:39.457892 (+923635us) service_pool.cc:224] Handling call
0709 12:59:39.909861 (+451969us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:59:40.164235  1677 raft_consensus.cc:2376] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 3. Re-sending same reply.
I20260709 12:59:40.148694  3660 raft_consensus.cc:2749] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20260709 12:59:40.175882  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 247952us
I20260709 12:59:40.178375  1680 raft_consensus.cc:2393] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 3: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 12:59:40.180480  1671 raft_consensus.cc:2393] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 4: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 12:59:40.176177  1678 raft_consensus.cc:2376] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 4. Re-sending same reply.
I20260709 12:59:40.182467  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:40.187428  2301 leader_election.cc:304] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:40.136806  3295 raft_consensus.cc:2749] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:59:40.184150  1646 leader_election.cc:304] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:59:40.195005  1646 leader_election.cc:304] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:59:40.189095  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:40.198179  1646 leader_election.cc:400] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 3 election: Vote denied by peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693) with higher term. Message: Invalid argument: T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for earlier term 3. Current term is 4.
I20260709 12:59:40.198551  1646 leader_election.cc:403] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20260709 12:59:40.163427  3415 raft_consensus.cc:697] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
W20260709 12:59:40.190207  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 100599us
W20260709 12:59:40.192214  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 74000.8us
I20260709 12:59:40.193125  1754 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55840 (request call id 1950) took 3537 ms. Trace:
I20260709 12:59:40.164036  3665 raft_consensus.cc:493] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 12:59:40.200708  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:40.200855  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:59:40.190330  2301 leader_election.cc:304] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 12:59:40.181322  3668 raft_consensus.cc:3060] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:59:40.206616  1671 raft_consensus.cc:2410] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for term 4 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260709 12:59:40.190202  3695 raft_consensus.cc:2804] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:59:40.221038  3695 raft_consensus.cc:493] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 12:59:40.221503  3695 raft_consensus.cc:3060] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:59:40.204180  3693 raft_consensus.cc:2804] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:59:40.222934  3693 raft_consensus.cc:493] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:59:40.223469  3693 raft_consensus.cc:3060] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:59:40.223089  3415 consensus_queue.cc:237] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:40.201920  1679 raft_consensus.cc:2393] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 3: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 12:59:40.211673  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:40.197532  3499 raft_consensus.cc:2804] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:59:40.200510  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:59:40.226085  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:40.214011  3414 raft_consensus.cc:2749] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693) with higher term. Message: Invalid argument: T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for earlier term 3. Current term is 4.
I20260709 12:59:40.229635  1678 raft_consensus.cc:2393] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 3: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 12:59:40.235334  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:40.237139  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:59:40.241045  1717 leader_election.cc:304] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:59:40.241510  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:40.242233  1718 leader_election.cc:304] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:40.243467  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:40.244983  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:40.250524  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:40.251735  1671 raft_consensus.cc:2368] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 2. Current term is 3.
I20260709 12:59:40.253664  1677 raft_consensus.cc:2393] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 6: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 12:59:40.209506  3665 raft_consensus.cc:3060] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:59:40.255160  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:40.255301  1717 leader_election.cc:304] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:40.209753  3660 raft_consensus.cc:2764] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20260709 12:59:40.262199  1718 leader_election.cc:304] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:40.262554  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:40.262797  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 12:59:40.268090  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 77818.7us
I20260709 12:59:40.268528  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:40.269199  2301 leader_election.cc:304] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:40.269779  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:59:40.270793  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:40.271584  1678 raft_consensus.cc:2393] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 4: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 12:59:40.267153  1750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55840 (request call id 1971) took 1713 ms. Trace:
I20260709 12:59:40.272585  1677 raft_consensus.cc:2393] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 3: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 12:59:40.272599  1750 rpcz_store.cc:276] 0709 12:59:38.553636 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:39.949885 (+1396249us) service_pool.cc:224] Handling call
0709 12:59:39.950160 (+   275us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:59:39.950364 (+   204us) raft_consensus.cc:1534] Early marking committed up to index 2
0709 12:59:40.155186 (+204822us) log.cc:844] Serialized 14 byte log entry
0709 12:59:40.266777 (+111591us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:59:40.266841 (+    64us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:59:40.266893 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:40.266967 (+    74us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:59:40.267087 (+   120us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":7531,"prepare.queue_time_us":211891,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":49,"thread_start_us":313177,"threads_started":2}
I20260709 12:59:40.271600  2301 leader_election.cc:304] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 12:59:40.277824  2301 leader_election.cc:304] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 12:59:40.289618  1671 raft_consensus.cc:2393] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 4: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 12:59:40.292906  1596 raft_consensus.cc:1240] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 3. Current term is 4. Ops: []
I20260709 12:59:40.295044  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:59:40.262740  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 12:59:40.230168  3521 consensus_queue.cc:1048] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.047s
I20260709 12:59:40.262590  3499 raft_consensus.cc:697] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 3 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 12:59:40.275913  3564 raft_consensus.cc:493] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:40.270195  3468 raft_consensus.cc:2749] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:59:40.272219  3646 raft_consensus.cc:2749] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20260709 12:59:40.316185  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 76052.5us
I20260709 12:59:40.262529  3358 raft_consensus.cc:2804] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:59:40.324024   431 auto_leader_rebalancer.cc:452] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260709 12:59:40.307536  3466 raft_consensus.cc:493] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:40.348438  3466 raft_consensus.cc:515] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:40.349463   431 auto_leader_rebalancer.cc:527] All tables' leader rebalancing finished this round
W20260709 12:59:40.326102  1569 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:59:40.276310  3595 raft_consensus.cc:2749] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:59:40.309448  3691 raft_consensus.cc:2804] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:59:40.350358  3691 raft_consensus.cc:493] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 12:59:40.350818  3691 raft_consensus.cc:3060] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:59:40.350648  3595 consensus_queue.cc:1048] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:59:40.268621  3694 raft_consensus.cc:2749] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:59:40.257256  3608 raft_consensus.cc:515] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 12:59:40.332139  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 136337us
I20260709 12:59:40.288563  3653 leader_election.cc:290] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 4 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:40.362085  1646 leader_election.cc:304] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:59:40.354748  1569 leader_election.cc:304] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 12:59:40.296482  3657 raft_consensus.cc:2749] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:59:40.366917  1646 leader_election.cc:304] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
W20260709 12:59:40.355930  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 84968us
I20260709 12:59:40.375257  1717 leader_election.cc:304] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 12:59:40.362910  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 71483.5us
I20260709 12:59:40.276510  3701 raft_consensus.cc:2804] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Leader election won for term 6
W20260709 12:59:40.312346  2300 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:59:40.389927  3444 raft_consensus.cc:2749] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:59:40.390933  3414 raft_consensus.cc:493] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 12:59:40.360611  3466 leader_election.cc:290] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:40.393131  3716 raft_consensus.cc:493] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 12:59:40.393751  3716 raft_consensus.cc:515] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:40.382218  3468 raft_consensus.cc:2749] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:59:40.391640  3414 raft_consensus.cc:515] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } }
I20260709 12:59:40.380980  1646 leader_election.cc:304] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 12:59:40.357564  3706 raft_consensus.cc:2749] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 12:59:40.397500  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 135364us
I20260709 12:59:40.436797  1718 leader_election.cc:304] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:40.439049  1718 leader_election.cc:304] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 12:59:40.374588  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 53455.8us
I20260709 12:59:40.445405  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:40.446574  1679 raft_consensus.cc:2393] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 3: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
W20260709 12:59:40.403014  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 48496.8us
I20260709 12:59:40.394615  3499 consensus_queue.cc:237] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:40.419287  1747 raft_consensus.cc:1240] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 2. Current term is 3. Ops: []
I20260709 12:59:40.445361  1718 leader_election.cc:304] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:40.203567  1754 rpcz_store.cc:276] 0709 12:59:36.655734 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:39.759877 (+3104143us) service_pool.cc:224] Handling call
0709 12:59:39.760258 (+   381us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:59:39.760562 (+   304us) raft_consensus.cc:1534] Early marking committed up to index 2
0709 12:59:39.882429 (+121867us) log.cc:844] Serialized 14 byte log entry
0709 12:59:40.192712 (+310283us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:59:40.192779 (+    67us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:59:40.192836 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:40.192911 (+    75us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:59:40.193051 (+   140us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":363646,"prepare.run_cpu_time_us":56,"prepare.run_wall_time_us":57,"raft.queue_time_us":13501,"raft.run_cpu_time_us":154,"raft.run_wall_time_us":5114,"thread_start_us":421603,"threads_started":2}
I20260709 12:59:40.433274  3634 raft_consensus.cc:493] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 12:59:40.356350  3358 raft_consensus.cc:697] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 4 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 12:59:40.411338  3504 raft_consensus.cc:515] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:40.467069  1764 tablet_copy_service.cc:342] P af520b67945d443b901bbeca37cc5701: Request end of tablet copy session 3b879799899a499c8d409ee9f1277cd1-983621a6fc5b4c9ea0159a648dfe7a36 received from {username='slave'} at 127.0.0.1:43156
I20260709 12:59:40.467648  1764 tablet_copy_service.cc:434] P af520b67945d443b901bbeca37cc5701: ending tablet copy session 3b879799899a499c8d409ee9f1277cd1-983621a6fc5b4c9ea0159a648dfe7a36 on tablet 983621a6fc5b4c9ea0159a648dfe7a36 with peer 3b879799899a499c8d409ee9f1277cd1
I20260709 12:59:40.440855  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:59:40.412880  3653 raft_consensus.cc:2804] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:59:40.452970  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:40.345419  3564 raft_consensus.cc:515] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:40.449529  3701 raft_consensus.cc:697] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 6 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 12:59:40.448470  3660 raft_consensus.cc:2804] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260709 12:59:40.457679  3707 log.cc:927] Time spent T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.079s	user 0.000s	sys 0.000s
I20260709 12:59:40.457794  3414 leader_election.cc:290] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 12:59:40.458531  3697 log.cc:927] Time spent T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.228s	user 0.000s	sys 0.001s
I20260709 12:59:40.459029  3634 raft_consensus.cc:3060] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:59:40.447542  3595 raft_consensus.cc:2749] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:59:40.462487  3721 raft_consensus.cc:493] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 12:59:40.463653  3444 consensus_queue.cc:1059] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:59:40.465324  3608 leader_election.cc:290] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 3 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:40.444355  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:40.398303  3713 raft_consensus.cc:2804] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Leader election won for term 3
W20260709 12:59:40.475140  3705 log.cc:927] Time spent T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.088s	user 0.000s	sys 0.001s
I20260709 12:59:40.475941  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:40.479633  3295 raft_consensus.cc:493] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:40.481014  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:40.524999  1680 raft_consensus.cc:2368] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 5. Current term is 6.
I20260709 12:59:40.497504  1675 raft_consensus.cc:1275] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 12:59:40.481948  2329 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49082 (request call id 506) took 1336 ms. Trace:
I20260709 12:59:40.482528  3468 raft_consensus.cc:2804] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:59:40.483031  3653 raft_consensus.cc:697] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
W20260709 12:59:40.456727  3709 log.cc:927] Time spent T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.074s	user 0.002s	sys 0.000s
I20260709 12:59:40.467298  3358 consensus_queue.cc:237] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } }
I20260709 12:59:40.534276  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 12:59:40.534886  2332 raft_consensus.cc:3060] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:59:40.517418  3715 raft_consensus.cc:2749] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:59:40.537088  3693 raft_consensus.cc:515] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 12:59:40.494626  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 44216.5us
W20260709 12:59:40.494978  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 49678.4us
I20260709 12:59:40.540927  1718 leader_election.cc:304] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:40.541816  3715 raft_consensus.cc:2749] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:59:40.542671  3701 consensus_queue.cc:237] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:40.542270  1718 leader_election.cc:304] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 12:59:40.546042  1718 leader_election.cc:304] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:40.523172  3646 consensus_queue.cc:1059] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.023s
I20260709 12:59:40.546870  3595 raft_consensus.cc:2749] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:59:40.547359  3646 raft_consensus.cc:3055] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:59:40.547521  3595 raft_consensus.cc:2749] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:59:40.495990  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:40.547847  3646 raft_consensus.cc:740] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 12:59:40.548255  1671 raft_consensus.cc:2468] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 4.
I20260709 12:59:40.548913  1718 leader_election.cc:304] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 12:59:40.549243  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:59:40.548826  3646 consensus_queue.cc:260] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:40.549814  1671 raft_consensus.cc:3060] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:59:40.507836  3414 consensus_queue.cc:1059] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.024s
I20260709 12:59:40.491734  3497 raft_consensus.cc:515] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:40.551208  3414 raft_consensus.cc:3055] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:59:40.551589  3414 raft_consensus.cc:740] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 12:59:40.551653  1674 raft_consensus.cc:1240] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 3. Current term is 4. Ops: []
I20260709 12:59:40.551848  3595 raft_consensus.cc:2749] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:59:40.412740  3663 raft_consensus.cc:515] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:40.551661  2301 leader_election.cc:304] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 12:59:40.552518  3414 consensus_queue.cc:260] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:40.553247  1678 raft_consensus.cc:1240] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 3. Current term is 4. Ops: []
I20260709 12:59:40.553413  3464 raft_consensus.cc:3060] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:59:40.405980  3687 raft_consensus.cc:515] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 12:59:40.508020  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 57155.8us
I20260709 12:59:40.490883  3543 raft_consensus.cc:515] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 12:59:40.497344  3639 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.685s	user 0.009s	sys 0.019s
I20260709 12:59:40.528908  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 12:59:40.550249  3646 raft_consensus.cc:3060] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:59:40.530004  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 12:59:40.579463  2333 raft_consensus.cc:3060] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:59:40.531210  2335 raft_consensus.cc:1240] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 2. Current term is 3. Ops: []
I20260709 12:59:40.531849  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:59:40.509093  3657 raft_consensus.cc:493] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
W20260709 12:59:40.532946  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 461980us
I20260709 12:59:40.584993  1571 leader_election.cc:304] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 12:59:40.587143  1571 leader_election.cc:304] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 12:59:40.537514  3716 leader_election.cc:290] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:40.554914  1676 raft_consensus.cc:2393] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 4: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 12:59:40.554991  1646 leader_election.cc:304] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 12:59:40.568982  3468 raft_consensus.cc:697] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 3 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
W20260709 12:59:40.588864  3723 log.cc:927] Time spent T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.070s	user 0.001s	sys 0.000s
I20260709 12:59:40.589237  3638 raft_consensus.cc:2804] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:59:40.589777  3638 raft_consensus.cc:697] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 12:59:40.589670  3468 consensus_queue.cc:237] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:40.525192  3713 raft_consensus.cc:697] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 3 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 12:59:40.590830  3638 consensus_queue.cc:237] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:40.591430  3713 consensus_queue.cc:237] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:40.588795  3466 raft_consensus.cc:2804] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:59:40.593130  3466 raft_consensus.cc:493] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 12:59:40.593509  3466 raft_consensus.cc:3060] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:59:40.595417  3687 leader_election.cc:290] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 2 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:40.597278  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:40.518227  3721 raft_consensus.cc:515] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:40.601120  3721 leader_election.cc:290] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 12:59:40.602109  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 47185.4us
I20260709 12:59:40.569680  3642 raft_consensus.cc:515] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:40.604682  2337 raft_consensus.cc:2468] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 4.
I20260709 12:59:40.604920  3642 leader_election.cc:290] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 3 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:40.605702  1601 raft_consensus.cc:1240] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 2. Current term is 3. Ops: []
I20260709 12:59:40.606410  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:40.608322  1571 leader_election.cc:304] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 12:59:40.494596  3660 raft_consensus.cc:493] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
W20260709 12:59:40.572997  3639 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.771s	user 0.014s	sys 0.022s
I20260709 12:59:40.612470  3660 raft_consensus.cc:3060] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:59:40.556412  1569 leader_election.cc:304] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 12:59:40.609193  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:40.489970  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:59:40.614459  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:40.614859  1673 raft_consensus.cc:1240] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 2. Current term is 3. Ops: []
I20260709 12:59:40.615173  1679 raft_consensus.cc:2468] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 1.
I20260709 12:59:40.521795  3504 leader_election.cc:290] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 3 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:40.576217  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:59:40.576651  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:40.620308  1750 raft_consensus.cc:1240] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [term 2 LEADER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 1. Current term is 2. Ops: []
I20260709 12:59:40.602267  3724 raft_consensus.cc:2749] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:59:40.608314  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:40.556192  1718 leader_election.cc:304] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:40.609442  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:40.610459  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:59:40.625908  2337 raft_consensus.cc:2468] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 4.
I20260709 12:59:40.611152  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:40.606402  3657 raft_consensus.cc:515] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:40.608765  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:40.630225  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:40.553958  3414 raft_consensus.cc:3060] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:59:40.632967  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:40.616236  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:59:40.564874  2332 raft_consensus.cc:2468] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 5.
I20260709 12:59:40.636866  2332 raft_consensus.cc:1240] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260709 12:59:40.638127  2299 leader_election.cc:304] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:40.576081  2301 leader_election.cc:304] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 12:59:40.616706  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
W20260709 12:59:40.620052  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 63717.4us
I20260709 12:59:40.529199  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 12:59:40.621413  1571 leader_election.cc:304] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 12:59:40.533409  3295 raft_consensus.cc:515] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 12:59:40.664268  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 108142us
I20260709 12:59:40.665369  1718 leader_election.cc:400] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261) with higher term. Message: Invalid argument: T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 2. Current term is 3.
I20260709 12:59:40.665714  1718 leader_election.cc:403] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
W20260709 12:59:40.657299  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 81293.3us
I20260709 12:59:40.667423  1680 raft_consensus.cc:2393] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 3: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 12:59:40.607848  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:40.669443  1644 leader_election.cc:304] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 12:59:40.654690  3584 raft_consensus.cc:2804] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:59:40.673602  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:40.676649  1750 raft_consensus.cc:2393] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 2: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 12:59:40.677052  1718 leader_election.cc:304] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 12:59:40.680157  2301 leader_election.cc:304] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 12:59:40.663451  3721 raft_consensus.cc:2749] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:59:40.602236  3691 raft_consensus.cc:515] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:40.529024  2329 rpcz_store.cc:276] 0709 12:59:39.145169 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:39.156725 (+ 11556us) service_pool.cc:224] Handling call
0709 12:59:39.734243 (+577518us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:59:39.897463 (+163220us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:40.481701 (+584238us) spinlock_profiling.cc:243] Waited 574 ms on lock 0x7b5400605250. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661618ad 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 12:59:40.481876 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1320520064}
I20260709 12:59:40.710433  1718 leader_election.cc:304] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:40.725728  1673 raft_consensus.cc:1275] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260709 12:59:40.672751  3688 raft_consensus.cc:2749] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:59:40.736172  1673 raft_consensus.cc:1275] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 12:59:40.675088  3584 raft_consensus.cc:493] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:59:40.738576  1673 raft_consensus.cc:1275] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 12:59:40.672551  3358 raft_consensus.cc:2749] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 12:59:40.733812  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 53748.3us
I20260709 12:59:40.679534  3639 raft_consensus.cc:2804] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:59:40.740576  1718 leader_election.cc:304] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:40.721544  3716 raft_consensus.cc:2749] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:59:40.731885  3578 leader_election.cc:290] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 12:59:40.809338  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 135109us
I20260709 12:59:40.743566  1673 raft_consensus.cc:1275] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:59:40.673755  3725 raft_consensus.cc:2804] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:59:40.682826  3658 raft_consensus.cc:2804] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:59:40.704195  3521 raft_consensus.cc:2749] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261) with higher term. Message: Invalid argument: T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 2. Current term is 3.
I20260709 12:59:40.812577  1673 raft_consensus.cc:1275] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 12:59:40.741035  3713 raft_consensus.cc:493] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 12:59:40.803480  3661 consensus_queue.cc:1059] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.064s
W20260709 12:59:40.805523  3695 consensus_meta.cc:287] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.584s	user 0.009s	sys 0.006s
I20260709 12:59:40.810401  1569 leader_election.cc:304] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 12:59:40.734755  3701 raft_consensus.cc:2749] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20260709 12:59:40.812547  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 72062.6us
I20260709 12:59:40.675376  3724 consensus_queue.cc:1059] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.070s
I20260709 12:59:40.743134  3584 raft_consensus.cc:3060] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:59:40.734970  3728 raft_consensus.cc:2749] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 12:59:40.866657  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 202195us
W20260709 12:59:40.855042  3665 consensus_meta.cc:287] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.565s	user 0.010s	sys 0.003s
I20260709 12:59:40.687685  3444 consensus_queue.cc:1059] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.122s
I20260709 12:59:40.755174  3564 leader_election.cc:290] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:40.747912  3639 raft_consensus.cc:493] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 12:59:40.856742  1718 leader_election.cc:304] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 12:59:40.812961  3358 raft_consensus.cc:2749] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:59:40.672230  3653 consensus_queue.cc:237] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:40.739173  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 12:59:40.824553  3658 raft_consensus.cc:493] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 12:59:40.890952  2329 raft_consensus.cc:1240] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 3. Current term is 4. Ops: []
I20260709 12:59:40.895000  3543 leader_election.cc:290] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 3 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 12:59:40.880554  3729 log.cc:927] Time spent T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.055s	user 0.000s	sys 0.000s
I20260709 12:59:40.907048  2333 raft_consensus.cc:2468] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 5.
I20260709 12:59:40.862747  3268 leader_election.cc:290] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 4 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:40.822935  3693 leader_election.cc:290] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 4 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 12:59:40.914358  3727 log.cc:927] Time spent T 00000000000000000000000000000000 P 919f6392d4f44fd68987e2c5e7ec37ca: Append to log took a long time: real 0.224s	user 0.001s	sys 0.001s
I20260709 12:59:40.832250  3725 raft_consensus.cc:697] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 12:59:40.914949  3661 raft_consensus.cc:3055] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:59:40.933379  2337 raft_consensus.cc:1240] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260709 12:59:40.910364  3665 raft_consensus.cc:515] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:40.919116  3639 raft_consensus.cc:3060] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:59:40.911790  3724 consensus_queue.cc:1243] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer af520b67945d443b901bbeca37cc5701 log is divergent from this leader: its last log entry 2.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260709 12:59:40.859423  3695 raft_consensus.cc:515] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:40.924002  3634 raft_consensus.cc:515] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:40.855571  3713 raft_consensus.cc:515] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:40.928786  3658 raft_consensus.cc:3060] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:59:40.938879  3661 raft_consensus.cc:740] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 12:59:40.940070  3736 raft_consensus.cc:493] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
W20260709 12:59:41.014146  3704 log.cc:927] Time spent T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.618s	user 0.011s	sys 0.000s
W20260709 12:59:41.009459  3668 consensus_meta.cc:287] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.778s	user 0.016s	sys 0.002s
W20260709 12:59:41.011241  3732 log.cc:927] Time spent T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.079s	user 0.000s	sys 0.009s
I20260709 12:59:41.017020  3736 raft_consensus.cc:515] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:41.020074  3695 leader_election.cc:290] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 3 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 12:59:41.014802  3731 log.cc:927] Time spent T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.152s	user 0.005s	sys 0.002s
W20260709 12:59:41.030330  3708 log.cc:927] Time spent T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.646s	user 0.000s	sys 0.012s
W20260709 12:59:41.072135  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 261548us
I20260709 12:59:41.074666  1569 leader_election.cc:304] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 12:59:40.867790  1571 leader_election.cc:304] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1
W20260709 12:59:40.859361  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 231442us
I20260709 12:59:40.886070  1671 raft_consensus.cc:2468] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 4.
I20260709 12:59:41.105940  1673 raft_consensus.cc:1240] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 2. Current term is 3. Ops: []
I20260709 12:59:41.106376  1679 raft_consensus.cc:1240] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 2. Current term is 3. Ops: []
I20260709 12:59:41.112627  1646 leader_election.cc:304] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
W20260709 12:59:41.124248  1717 leader_election.cc:336] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 1.764s (SENT)
W20260709 12:59:41.125190  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 268430us
I20260709 12:59:41.125379  1717 leader_election.cc:304] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:41.129209  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:41.132695  1753 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55848 (request call id 1695) took 3765 ms. Trace:
I20260709 12:59:41.132900  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:41.133028  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:41.134063  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:41.135200  1750 raft_consensus.cc:3060] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:59:41.135365  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:41.136150  1671 raft_consensus.cc:3055] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:59:41.136592  1671 raft_consensus.cc:740] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 12:59:41.124393  1745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55848 (request call id 1710) took 2011 ms. Trace:
I20260709 12:59:41.144213  1718 leader_election.cc:304] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 12:59:41.146318  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:41.146401  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 12:59:41.147084  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:41.147610  2330 raft_consensus.cc:2393] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 3: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 12:59:41.133237  1753 rpcz_store.cc:276] 0709 12:59:37.367442 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:39.762405 (+2394963us) service_pool.cc:224] Handling call
0709 12:59:39.762753 (+   348us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:59:39.786749 (+ 23996us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:59:39.786795 (+    46us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:59:39.795643 (+  8848us) log.cc:844] Serialized 26 byte log entry
0709 12:59:40.049178 (+253535us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:59:40.049225 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:40.049291 (+    66us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:59:40.955120 (+905829us) spinlock_profiling.cc:243] Waited 11.5 ms on lock 0x7b240024b1c8. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f4262efb9c4 00007f4262ef9acb 00007f42630b4dc9 00007f42630b4a55 00007f42630b3fb0 00007f4265d2a488 00007f4265d2abe8 00007f4265d02398 00007f4265cfd76a 00007f4265cfcbb6
0709 12:59:41.131975 (+176855us) raft_consensus.cc:1685] finished
0709 12:59:41.132023 (+    48us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:59:41.132417 (+   394us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":23494,"raft.queue_time_us":25634,"raft.run_cpu_time_us":178,"raft.run_wall_time_us":15682,"spinlock_wait_cycles":26541824,"thread_start_us":164191,"threads_started":1,"wal-append.queue_time_us":359102}
I20260709 12:59:41.138602  1745 rpcz_store.cc:276] 0709 12:59:39.112406 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:40.075645 (+963239us) service_pool.cc:224] Handling call
0709 12:59:40.075967 (+   322us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:59:40.108041 (+ 32074us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:59:40.108073 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:59:40.108693 (+   620us) log.cc:844] Serialized 26 byte log entry
0709 12:59:40.264341 (+155648us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:59:40.264397 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:40.264498 (+   101us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:59:40.941963 (+677465us) spinlock_profiling.cc:243] Waited 9.69 ms on lock 0x7b4400022b4c. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f4262f9bc85 00007f4262f9c6a2 00007f4265d0e1db 00007f4265d09aaf 00007f4265cfd86e 00007f4265cfcbb6 00007f4265cfc7d8 00007f4265d06221 00007f4265d061d9 00007f4265d06169
0709 12:59:41.123224 (+181261us) raft_consensus.cc:1685] finished
0709 12:59:41.123295 (+    71us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:59:41.123954 (+   659us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":10570,"raft.run_cpu_time_us":118,"raft.run_wall_time_us":119,"spinlock_wait_cycles":22278912,"thread_start_us":155184,"threads_started":1,"wal-append.queue_time_us":273739}
I20260709 12:59:41.138455  1671 consensus_queue.cc:260] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:41.166384  1671 raft_consensus.cc:3060] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:40.941781  3295 leader_election.cc:290] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:41.172001  3358 raft_consensus.cc:2749] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:59:41.173310  3358 raft_consensus.cc:2749] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:59:41.175271  3444 raft_consensus.cc:2749] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:59:41.177544  1745 raft_consensus.cc:1275] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
W20260709 12:59:41.189440  3466 consensus_meta.cc:287] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.595s	user 0.013s	sys 0.000s
I20260709 12:59:41.190569  3466 raft_consensus.cc:515] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:40.922541  3564 raft_consensus.cc:2804] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:59:41.192950  3521 raft_consensus.cc:2749] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:59:41.020959  3668 raft_consensus.cc:515] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:41.027791  3713 leader_election.cc:290] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:41.022902  3725 consensus_queue.cc:237] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 12:59:41.226801  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 152448us
I20260709 12:59:41.256918  3665 leader_election.cc:290] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 3 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:41.258576  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:41.036338  3736 leader_election.cc:290] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:41.234476  3268 raft_consensus.cc:2804] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:59:41.260967  3268 raft_consensus.cc:697] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 12:59:41.261597  3572 consensus_queue.cc:1048] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:59:41.233608  1750 raft_consensus.cc:2468] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 3.
I20260709 12:59:41.197638  3661 consensus_queue.cc:260] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:41.263427  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:41.264331  3661 raft_consensus.cc:3060] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:59:41.264734  2335 raft_consensus.cc:2410] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
W20260709 12:59:41.235934  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 367968us
I20260709 12:59:41.234007  3638 raft_consensus.cc:2804] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:59:41.236155  3564 raft_consensus.cc:697] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
W20260709 12:59:41.248684  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 85341.8us
I20260709 12:59:41.236311  3634 leader_election.cc:290] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:41.274530  2335 raft_consensus.cc:1275] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 12:59:41.268894  1571 leader_election.cc:304] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 12:59:41.269171  3268 consensus_queue.cc:237] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 12:59:41.282321  3712 log.cc:927] Time spent T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.889s	user 0.005s	sys 0.000s
I20260709 12:59:41.272065  3564 consensus_queue.cc:237] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:41.285758  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:59:41.266428  3584 raft_consensus.cc:515] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:41.028967  3691 leader_election.cc:290] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 4 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:41.270903  3638 raft_consensus.cc:493] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 12:59:41.288318  3638 raft_consensus.cc:3060] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:59:41.291482  3668 leader_election.cc:290] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 3 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:41.264520  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:41.292433  3668 raft_consensus.cc:2749] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:59:41.302973  1674 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52766 (request call id 1977) took 1414 ms. Trace:
W20260709 12:59:41.348126  3464 consensus_meta.cc:287] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.794s	user 0.010s	sys 0.018s
W20260709 12:59:41.356020  1717 leader_election.cc:336] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 2.134s (SENT)
I20260709 12:59:41.356957  1717 leader_election.cc:304] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 12:59:41.357409  1718 leader_election.cc:336] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Timed out: RequestConsensusVote RPC to 127.0.107.194:46261 timed out after 2.038s (SENT)
I20260709 12:59:41.357981  1718 leader_election.cc:304] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:41.358659  3634 raft_consensus.cc:493] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 12:59:41.360100  3728 raft_consensus.cc:2749] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260709 12:59:41.362565  1603 consensus_meta.cc:287] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.660s	user 0.001s	sys 0.019s
W20260709 12:59:40.578140  3702 consensus_queue.cc:1175] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" responder_term: 4 status { last_received { term: 3 index: 6 } last_committed_idx: 6 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 3. Current term is 4. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
W20260709 12:59:41.363739  1569 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:59:41.303521  1674 rpcz_store.cc:276] 0709 12:59:39.887968 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:39.888135 (+   167us) service_pool.cc:224] Handling call
0709 12:59:39.888473 (+   338us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:59:40.552132 (+663659us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:41.297616 (+745484us) spinlock_profiling.cc:243] Waited 663 ms on lock 0x7b5400092254. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 12:59:41.302696 (+  5080us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1524547968}
W20260709 12:59:41.825022  1671 consensus_meta.cc:287] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.658s	user 0.015s	sys 0.001s
W20260709 12:59:41.674090  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 318112us
I20260709 12:59:41.904712  1671 raft_consensus.cc:1275] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260709 12:59:41.802170  3661 consensus_meta.cc:287] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.537s	user 0.010s	sys 0.003s
I20260709 12:59:41.906620  1603 raft_consensus.cc:2468] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 2.
I20260709 12:59:41.912170  1603 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36462 (request call id 315) took 1302 ms. Trace:
I20260709 12:59:41.912242  2333 raft_consensus.cc:1240] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260709 12:59:41.912597  1603 rpcz_store.cc:276] 0709 12:59:40.609956 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:40.610135 (+   179us) service_pool.cc:224] Handling call
0709 12:59:41.912046 (+1301911us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:59:41.900624  3584 leader_election.cc:290] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 5 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:41.412922  3466 leader_election.cc:290] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 3 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:41.424289  1675 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52778 (request call id 1618) took 1232 ms. Trace:
I20260709 12:59:41.349435  2330 raft_consensus.cc:2468] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 3.
W20260709 12:59:41.915962  3660 consensus_meta.cc:287] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 1.297s	user 0.014s	sys 0.007s
W20260709 12:59:41.486508  3414 consensus_meta.cc:287] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.694s	user 0.007s	sys 0.015s
I20260709 12:59:41.916710  3660 raft_consensus.cc:515] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:41.917456  1673 raft_consensus.cc:1275] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:59:41.916005  1675 rpcz_store.cc:276] 0709 12:59:40.192151 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:40.298421 (+106270us) service_pool.cc:224] Handling call
0709 12:59:40.449667 (+151246us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:59:40.527915 (+ 78248us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:41.423691 (+895776us) spinlock_profiling.cc:243] Waited 92 ms on lock 0x7b54001f5c50. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661618ad 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 12:59:41.424032 (+   341us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":211610496}
I20260709 12:59:41.918123  1677 raft_consensus.cc:1240] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 2. Current term is 3. Ops: []
I20260709 12:59:41.491201  3464 raft_consensus.cc:2749] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:59:41.918756  1679 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57092 (request call id 425) took 1489 ms. Trace:
I20260709 12:59:41.919224  1679 rpcz_store.cc:276] 0709 12:59:40.430136 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:40.659345 (+229209us) service_pool.cc:224] Handling call
0709 12:59:40.662856 (+  3511us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:59:41.107209 (+444353us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:41.909591 (+802382us) spinlock_profiling.cc:243] Waited 442 ms on lock 0x7b5400347454. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 12:59:41.918685 (+  9094us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1017283328}
I20260709 12:59:41.920091  3466 raft_consensus.cc:493] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:41.920298  2332 raft_consensus.cc:1240] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 2. Current term is 3. Ops: []
I20260709 12:59:41.921079  3660 leader_election.cc:290] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 3 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:41.920848  3466 raft_consensus.cc:515] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:41.922287  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:41.923127  1750 raft_consensus.cc:3060] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:59:41.929085  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 660392us
W20260709 12:59:41.798417  3619 consensus_meta.cc:287] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 1.610s	user 0.022s	sys 0.019s
I20260709 12:59:41.730743  3715 raft_consensus.cc:493] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 12:59:41.933151  3728 raft_consensus.cc:2749] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:59:41.364365  3639 raft_consensus.cc:515] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:41.938503  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 12:59:41.940352  3639 leader_election.cc:290] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 5 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:41.936115  3619 raft_consensus.cc:515] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:41.947618  1751 raft_consensus.cc:1240] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 2. Current term is 3. Ops: []
W20260709 12:59:41.947904  3638 consensus_meta.cc:287] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.659s	user 0.009s	sys 0.003s
I20260709 12:59:41.942862  3634 raft_consensus.cc:515] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 12:59:41.943135  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 104921us
I20260709 12:59:41.949095  1750 raft_consensus.cc:2468] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 3.
I20260709 12:59:41.948634  3619 leader_election.cc:290] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 12:59:41.949590  3658 consensus_meta.cc:287] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.846s	user 0.008s	sys 0.008s
W20260709 12:59:41.950251  3653 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.550s	user 0.006s	sys 0.018s
W20260709 12:59:41.958520  3653 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.615s	user 0.007s	sys 0.018s
I20260709 12:59:41.959296  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:41.960090  1745 raft_consensus.cc:3060] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:59:41.961849  3497 leader_election.cc:290] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 4 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:41.962560  3634 leader_election.cc:290] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:41.948385  3638 raft_consensus.cc:515] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:41.953748  3715 raft_consensus.cc:515] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:41.954758  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 12:59:41.955448  3642 raft_consensus.cc:493] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 12:59:41.966979  3658 raft_consensus.cc:515] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:41.968313  3358 raft_consensus.cc:493] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:41.970676  3642 raft_consensus.cc:515] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:41.974900  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 12:59:41.975318  3358 raft_consensus.cc:515] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:41.976924  3642 leader_election.cc:290] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:41.971225  3466 leader_election.cc:290] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:41.972033  1745 raft_consensus.cc:2468] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 4.
I20260709 12:59:41.977679  3358 leader_election.cc:290] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:41.974069  3638 leader_election.cc:290] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 4 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:41.977969  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:41.988037  3715 leader_election.cc:290] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:41.989552  3657 leader_election.cc:290] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 12:59:41.992774  3698 log.cc:927] Time spent T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 1.685s	user 0.019s	sys 0.029s
I20260709 12:59:41.988040  3658 leader_election.cc:290] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 4 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:41.999787  3638 raft_consensus.cc:493] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:42.000662  3638 raft_consensus.cc:515] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 12:59:42.017552  1676 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52766 (request call id 1991) took 1585 ms (client timeout 1840 ms). Trace:
W20260709 12:59:42.018244  1676 rpcz_store.cc:269] 0709 12:59:40.432248 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:40.440473 (+  8225us) service_pool.cc:224] Handling call
0709 12:59:42.016937 (+1576464us) spinlock_profiling.cc:243] Waited 85.7 ms on lock 0x7b54001e87d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 12:59:42.017292 (+   355us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":197186304}
W20260709 12:59:42.025115  3711 log.cc:927] Time spent T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 1.633s	user 0.015s	sys 0.000s
I20260709 12:59:42.031391  1677 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57092 (request call id 424) took 1805 ms. Trace:
I20260709 12:59:42.031841  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:42.031920  1677 rpcz_store.cc:276] 0709 12:59:40.226417 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:41.287963 (+1061546us) service_pool.cc:224] Handling call
0709 12:59:41.288463 (+   500us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:59:41.918601 (+630138us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:42.031122 (+112521us) spinlock_profiling.cc:243] Waited 629 ms on lock 0x7b54000fe454. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 12:59:42.031316 (+   194us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1447131392}
I20260709 12:59:42.033903  3497 consensus_queue.cc:1059] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:59:42.034775  3638 leader_election.cc:290] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:42.035063  1569 leader_election.cc:304] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 12:59:42.035053  3543 raft_consensus.cc:3055] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 LEADER]: Stepping down as leader of term 4
W20260709 12:59:42.042599  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 46913.3us
I20260709 12:59:42.042898  3660 raft_consensus.cc:3055] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:59:42.042973  3657 raft_consensus.cc:493] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 12:59:42.036513  3639 consensus_queue.cc:1059] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.017s
I20260709 12:59:42.044610  3464 raft_consensus.cc:493] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:42.052536  3543 raft_consensus.cc:740] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 12:59:42.053998  3660 raft_consensus.cc:740] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
W20260709 12:59:42.063712  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 706218us
W20260709 12:59:42.059839  3564 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.716s	user 0.009s	sys 0.027s
W20260709 12:59:42.066795  3564 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.768s	user 0.009s	sys 0.028s
I20260709 12:59:42.063745  3640 raft_consensus.cc:493] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:42.076779  2334 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49082 (request call id 516) took 1544 ms. Trace:
I20260709 12:59:42.077075  3468 raft_consensus.cc:493] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 12:59:42.077250  2334 rpcz_store.cc:276] 0709 12:59:40.530311 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:40.530430 (+   119us) service_pool.cc:224] Handling call
0709 12:59:42.063530 (+1533100us) spinlock_profiling.cc:243] Waited 27.2 ms on lock 0x7b5800178598. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f426303db71 00007f426303da22 00007f426bf56c09 00007f426b0b0ee0 00007f426b17a0ab 00007f426b17a131 00007f426b13bfc0 00007f426b126929 00007f426b1263d2 00007f426469dbd4
0709 12:59:42.063904 (+   374us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:59:42.074033 (+ 10129us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:59:42.074061 (+    28us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:59:42.074110 (+    49us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:59:42.074166 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:42.074231 (+    65us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:59:42.074382 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":62637312}
I20260709 12:59:42.059931  3657 raft_consensus.cc:515] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:42.081841  3543 consensus_queue.cc:260] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 12:59:42.083042  3608 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 2 current term: 2 is higher.
I20260709 12:59:42.083369  3543 raft_consensus.cc:3060] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:59:42.084046  1643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:59:42.077750  3468 raft_consensus.cc:515] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:42.071309  3638 raft_consensus.cc:2804] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:59:42.086189  3468 leader_election.cc:290] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:42.074368  3640 raft_consensus.cc:515] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:42.087445  2336 raft_consensus.cc:2468] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 4.
I20260709 12:59:42.088579  1676 raft_consensus.cc:1240] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 3. Current term is 4. Ops: []
I20260709 12:59:42.088725  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:59:42.070271  3464 raft_consensus.cc:515] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:42.094480  3358 raft_consensus.cc:493] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 12:59:42.095223  3358 raft_consensus.cc:515] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:42.096150  3464 leader_election.cc:290] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:42.097620  3358 leader_election.cc:290] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:42.099220  3706 raft_consensus.cc:493] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:59:42.104729  1643 leader_election.cc:336] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 1.513s (SENT)
I20260709 12:59:42.105180  1643 leader_election.cc:304] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 12:59:42.106035  3608 raft_consensus.cc:2749] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 12:59:42.106626  1643 leader_election.cc:336] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 1.517s (SENT)
W20260709 12:59:42.108306  1643 leader_election.cc:336] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 1.534s (SENT)
I20260709 12:59:42.108789  1643 leader_election.cc:304] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:42.108981  1747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55848 (request call id 1697) took 4740 ms. Trace:
I20260709 12:59:42.109633  3608 raft_consensus.cc:2749] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:59:42.109398  1747 rpcz_store.cc:276] 0709 12:59:37.368160 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:39.763358 (+2395198us) service_pool.cc:224] Handling call
0709 12:59:40.419017 (+655659us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:59:40.451152 (+ 32135us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:42.108549 (+1657397us) spinlock_profiling.cc:243] Waited 655 ms on lock 0x7b54000e4550. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661618ad 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 12:59:42.108773 (+   224us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1506870016}
W20260709 12:59:42.110333  1643 leader_election.cc:336] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 1.547s (SENT)
I20260709 12:59:42.118480  1502 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35464 (request call id 283) took 6987 ms. Trace:
I20260709 12:59:42.119035  1643 leader_election.cc:304] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 12:59:42.120115  3497 raft_consensus.cc:2749] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:59:42.116487  1501 catalog_manager.cc:5697] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d reported cstate change: term changed from 1 to 2, leader changed from 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194) to 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193). New cstate: current_term: 2 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:42.115150  2337 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39726 (request call id 663) took 1564 ms. Trace:
I20260709 12:59:42.122959  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
W20260709 12:59:42.123204  1643 leader_election.cc:336] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 1.560s (SENT)
I20260709 12:59:42.123320  2337 rpcz_store.cc:276] 0709 12:59:40.550101 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:40.632334 (+ 82233us) service_pool.cc:224] Handling call
0709 12:59:40.932985 (+300651us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:59:40.939148 (+  6163us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:42.114703 (+1175555us) spinlock_profiling.cc:243] Waited 300 ms on lock 0x7b5400645350. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661618ad 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 12:59:42.114938 (+   235us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":690436864}
W20260709 12:59:42.124598  1643 leader_election.cc:336] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 1.631s (SENT)
I20260709 12:59:42.125092  1643 leader_election.cc:304] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:42.090380   431 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:59:42.126088  3497 raft_consensus.cc:2749] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:59:42.121862  1501 catalog_manager.cc:5697] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d reported cstate change: term changed from 2 to 4, leader changed from 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) to 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194). New cstate: current_term: 4 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20260709 12:59:42.127334  1680 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52778 (request call id 1630) took 1732 ms (client timeout 1958 ms). Trace:
W20260709 12:59:42.127815  1680 rpcz_store.cc:269] 0709 12:59:40.396805 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:40.525771 (+128966us) service_pool.cc:224] Handling call
0709 12:59:42.126801 (+1601030us) spinlock_profiling.cc:243] Waited 47.8 ms on lock 0x7b54000d9654. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 12:59:42.127107 (+   306us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":109873920}
I20260709 12:59:42.129046  1501 catalog_manager.cc:5697] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20260709 12:59:42.131354  2336 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49082 (request call id 511) took 1602 ms (client timeout 1723 ms). Trace:
W20260709 12:59:42.131857  2336 rpcz_store.cc:269] 0709 12:59:40.528800 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:40.528930 (+   130us) service_pool.cc:224] Handling call
0709 12:59:42.021752 (+1492822us) spinlock_profiling.cc:243] Waited 727 ms on lock 0x7b54005ba7a0. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42660d9dc0 00007f4266165e18 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6
0709 12:59:42.131088 (+109336us) spinlock_profiling.cc:243] Waited 522 ms on lock 0x7b54005b9fd4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 12:59:42.131232 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2870754944}
I20260709 12:59:42.132543  1501 catalog_manager.cc:5697] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:59:42.134567  3663 leader_election.cc:290] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 4 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:42.112749  3471 consensus_queue.cc:1048] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.023s
I20260709 12:59:42.138280  1678 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52766 (request call id 1983) took 1823 ms. Trace:
I20260709 12:59:42.138804  1678 rpcz_store.cc:276] 0709 12:59:40.314360 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:40.324468 (+ 10108us) service_pool.cc:224] Handling call
0709 12:59:40.324756 (+   288us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:59:40.553677 (+228921us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:42.137763 (+1584086us) spinlock_profiling.cc:243] Waited 228 ms on lock 0x7b5400091354. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 12:59:42.138062 (+   299us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":524967168}
I20260709 12:59:42.141645  2328 raft_consensus.cc:1240] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 2. Current term is 3. Ops: []
I20260709 12:59:42.119132  1502 rpcz_store.cc:276] 0709 12:59:35.130910 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:35.131094 (+   184us) service_pool.cc:224] Handling call
0709 12:59:42.118185 (+6987091us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:59:40.355362 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:59:40.459126 (+103764us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:59:40.459165 (+    39us) write_op.cc:435] Acquired schema lock
0709 12:59:40.459182 (+    17us) tablet.cc:662] Decoding operations
0709 12:59:40.459567 (+   385us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:59:40.459639 (+    72us) write_op.cc:641] Partition lock acquired for write op
0709 12:59:40.459665 (+    26us) tablet.cc:685] Acquiring locks for 1 operations
0709 12:59:40.459876 (+   211us) tablet.cc:701] Row locks acquired
0709 12:59:40.459893 (+    17us) write_op.cc:260] PREPARE: finished
0709 12:59:40.460205 (+   312us) write_op.cc:270] Start()
0709 12:59:40.460343 (+   138us) write_op.cc:276] Timestamp: P: 1783601980460126 usec, L: 0
0709 12:59:40.460368 (+    25us) op_driver.cc:348] REPLICATION: starting
0709 12:59:40.461143 (+   775us) log.cc:844] Serialized 541 byte log entry
0709 12:59:41.035891 (+574748us) op_driver.cc:464] REPLICATION: finished
0709 12:59:41.347139 (+311248us) write_op.cc:301] APPLY: starting
0709 12:59:41.347320 (+   181us) tablet.cc:1367] starting BulkCheckPresence
0709 12:59:41.347491 (+   171us) tablet.cc:1370] finished BulkCheckPresence
0709 12:59:41.347702 (+   211us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:59:41.348152 (+   450us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:59:41.348448 (+   296us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0709 12:59:41.348504 (+    56us) write_op.cc:312] APPLY: finished
0709 12:59:41.354757 (+  6253us) log.cc:844] Serialized 23 byte log entry
0709 12:59:41.355053 (+   296us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:59:41.355246 (+   193us) write_op.cc:454] Released schema lock
0709 12:59:41.355315 (+    69us) write_op.cc:341] FINISH: Updating metrics
0709 12:59:41.734044 (+378729us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:59:41.741939 (+  7895us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":475055,"reported_tablets":3,"child_traces":[["op",{"apply.queue_time_us":310766,"apply.run_cpu_time_us":3601,"apply.run_wall_time_us":338537,"num_ops":1,"prepare.queue_time_us":178364,"prepare.run_cpu_time_us":17285,"prepare.run_wall_time_us":252024,"raft.queue_time_us":6464,"raft.run_cpu_time_us":28037,"raft.run_wall_time_us":667715,"replication_time_us":575053,"thread_start_us":409065,"threads_started":4,"wal-append.queue_time_us":164070}]]}
I20260709 12:59:42.100561  3706 raft_consensus.cc:515] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:42.143733  1746 raft_consensus.cc:1275] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:59:42.112388  1569 leader_election.cc:304] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 12:59:42.144497  1746 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43156 (request call id 342) took 1701 ms. Trace:
I20260709 12:59:42.110250  3640 leader_election.cc:290] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:42.144951  1746 rpcz_store.cc:276] 0709 12:59:40.443429 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:40.447386 (+  3957us) service_pool.cc:224] Handling call
0709 12:59:42.142708 (+1695322us) spinlock_profiling.cc:243] Waited 2.19 ms on lock 0x7b58001ffe98. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f426303db71 00007f426303da22 00007f426bf56c09 00007f426b0b0ee0 00007f426b17a0ab 00007f426b17a131 00007f426b13bfc0 00007f426b126929 00007f426b1263d2 00007f426469dbd4
0709 12:59:42.143045 (+   337us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:59:42.144174 (+  1129us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:42.144431 (+   257us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5034880}
I20260709 12:59:42.146967  3660 consensus_queue.cc:260] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:42.148934  1752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55840 (request call id 1963) took 4484 ms. Trace:
I20260709 12:59:42.150979  3706 leader_election.cc:290] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:42.151129  3716 raft_consensus.cc:493] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 12:59:42.152094  3716 raft_consensus.cc:515] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:42.154914  3529 raft_consensus.cc:2804] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:59:42.155514  3529 raft_consensus.cc:493] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:59:42.155932  3529 raft_consensus.cc:3060] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:59:42.155828  1501 catalog_manager.cc:5697] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d reported cstate change: term changed from 2 to 3, leader changed from 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) to 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194). New cstate: current_term: 3 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20260709 12:59:42.157290  1717 leader_election.cc:336] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 2.462s (SENT)
I20260709 12:59:42.157835  1717 leader_election.cc:304] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 12:59:42.158759  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 46476.4us
I20260709 12:59:42.159289  3706 raft_consensus.cc:2749] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:59:42.158322  1501 catalog_manager.cc:5697] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d reported cstate change: term changed from 2 to 3, leader changed from 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194) to 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193). New cstate: current_term: 3 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:42.172163  2332 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39726 (request call id 668) took 1494 ms. Trace:
I20260709 12:59:42.172722  2332 rpcz_store.cc:276] 0709 12:59:40.677740 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:40.677905 (+   165us) service_pool.cc:224] Handling call
0709 12:59:40.678274 (+   369us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:59:41.920816 (+1242542us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:42.171757 (+250941us) spinlock_profiling.cc:243] Waited 1.24 s on lock 0x7b540072e554. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 12:59:42.172072 (+   315us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2855080832}
I20260709 12:59:42.175557  2328 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49082 (request call id 518) took 1641 ms. Trace:
I20260709 12:59:42.172433  1501 catalog_manager.cc:5697] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d reported cstate change: term changed from 1 to 2, leader changed from af520b67945d443b901bbeca37cc5701 (127.0.107.195) to 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193). New cstate: current_term: 2 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } }
I20260709 12:59:42.149281  1752 rpcz_store.cc:276] 0709 12:59:37.664696 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:39.780253 (+2115557us) service_pool.cc:224] Handling call
0709 12:59:39.780640 (+   387us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:59:39.780946 (+   306us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 12:59:39.968232 (+187286us) log.cc:844] Serialized 14 byte log entry
0709 12:59:42.147989 (+2179757us) spinlock_profiling.cc:243] Waited 5.53 ms on lock 0x7b18003f9160. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f4265d0a4bc 00007f426305848b 00007f426beb2ab9 00007f426307469b 00007f426307293f 00007f4263074c02 00007f4265cfc20b 00007f4265d01d23 00007f4265d0223a
0709 12:59:42.148346 (+   357us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:59:42.148419 (+    73us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:59:42.148485 (+    66us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:42.148603 (+   118us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:59:42.148751 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":365260,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":46,"spinlock_wait_cycles":12714368,"thread_start_us":364890,"threads_started":2,"wal-append.queue_time_us":243955,"wal-append.run_cpu_time_us":1588,"wal-append.run_wall_time_us":1002055}
I20260709 12:59:42.148769  3660 raft_consensus.cc:3060] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:59:42.176025  2328 rpcz_store.cc:276] 0709 12:59:40.549733 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:40.575544 (+ 25811us) service_pool.cc:224] Handling call
0709 12:59:42.140596 (+1565052us) spinlock_profiling.cc:243] Waited 18.3 ms on lock 0x7b5800178598. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f426303db71 00007f426303da22 00007f426bf56c09 00007f426b0b0ee0 00007f426b17a0ab 00007f426b17a131 00007f426b13bfc0 00007f426b126929 00007f426b1263d2 00007f426469dbd4
0709 12:59:42.141051 (+   455us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:59:42.142272 (+  1221us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:42.173319 (+ 31047us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":41975552}
I20260709 12:59:42.173403  3657 leader_election.cc:290] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:42.180902  3752 raft_consensus.cc:493] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 12:59:42.181531  3752 raft_consensus.cc:515] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:42.182359  3753 raft_consensus.cc:493] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 12:59:42.183112  3753 raft_consensus.cc:515] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:42.183904  3752 leader_election.cc:290] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:42.186254  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:42.186408  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:42.187714  3753 leader_election.cc:290] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 12:59:42.191365  1643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:59:42.193217  1601 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 311) took 1899 ms. Trace:
I20260709 12:59:42.201316  3716 leader_election.cc:290] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:42.204349  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:42.205163  1752 raft_consensus.cc:2468] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 6.
I20260709 12:59:42.207405  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:42.208277  1747 raft_consensus.cc:3060] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:59:42.099749  3638 raft_consensus.cc:697] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 12:59:42.180927  1751 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55848 (request call id 1704) took 3972 ms. Trace:
I20260709 12:59:42.211599  3638 consensus_queue.cc:237] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:42.211922  1751 rpcz_store.cc:276] 0709 12:59:38.208653 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:39.934195 (+1725542us) service_pool.cc:224] Handling call
0709 12:59:39.934399 (+   204us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:59:41.948078 (+2013679us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:42.180603 (+232525us) spinlock_profiling.cc:243] Waited 2.01 s on lock 0x7b54000e4f54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 12:59:42.180832 (+   229us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4628323712}
I20260709 12:59:42.186254  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:42.179409  3640 consensus_queue.cc:1059] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:59:42.204276  1646 leader_election.cc:304] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 12:59:42.215443  3663 raft_consensus.cc:3055] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:59:42.216104  3663 raft_consensus.cc:740] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 12:59:42.217048  1646 leader_election.cc:304] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 12:59:42.217434  3663 consensus_queue.cc:260] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:42.193742  1601 rpcz_store.cc:276] 0709 12:59:40.293173 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:40.294524 (+  1351us) service_pool.cc:224] Handling call
0709 12:59:40.294784 (+   260us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:59:40.606117 (+311333us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:42.192582 (+1586465us) spinlock_profiling.cc:243] Waited 311 ms on lock 0x7b5400216bd4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 12:59:42.192981 (+   399us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":714477440}
I20260709 12:59:42.219472  3663 raft_consensus.cc:3060] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:59:42.222661  3497 raft_consensus.cc:2749] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:59:42.223527  3497 raft_consensus.cc:2749] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:59:42.224072  1754 raft_consensus.cc:1275] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 12:59:42.225492  1745 raft_consensus.cc:1240] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 2. Current term is 3. Ops: []
W20260709 12:59:42.229660  1645 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:59:42.231263  1752 raft_consensus.cc:1275] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260709 12:59:42.234688  1718 leader_election.cc:304] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 12:59:42.235643  3715 raft_consensus.cc:2749] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 12:59:42.235967  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 62316.4us
I20260709 12:59:42.238507  1747 raft_consensus.cc:2468] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 4.
W20260709 12:59:42.238632  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 72453.8us
I20260709 12:59:42.241674  1718 leader_election.cc:304] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:42.242712  3715 raft_consensus.cc:2749] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:59:42.246872  3640 consensus_queue.cc:1048] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:59:42.246327  1646 leader_election.cc:304] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 12:59:42.250028  1646 leader_election.cc:304] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:59:42.250871  3552 raft_consensus.cc:493] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 12:59:42.251017  1646 leader_election.cc:304] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 12:59:42.251572  3552 raft_consensus.cc:515] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:42.251879  3639 raft_consensus.cc:2749] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:59:42.252322  1646 leader_election.cc:304] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:59:42.252981  3639 raft_consensus.cc:2804] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:59:42.253724  3640 raft_consensus.cc:2804] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:59:42.244840  1501 catalog_manager.cc:5697] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d reported cstate change: term changed from 3 to 4, leader changed from 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) to 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194). New cstate: current_term: 4 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } }
I20260709 12:59:42.253835  3552 leader_election.cc:290] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:42.256446  3608 raft_consensus.cc:2804] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader election won for term 4
W20260709 12:59:42.265928  1643 leader_election.cc:336] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 1.800s (SENT)
I20260709 12:59:42.276633  1643 leader_election.cc:304] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:42.274125  1501 catalog_manager.cc:5697] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d reported cstate change: term changed from 2 to 3, leader changed from 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194) to 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196). New cstate: current_term: 3 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:42.298527  1501 catalog_manager.cc:5697] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d reported cstate change: term changed from 2 to 3, leader changed from af520b67945d443b901bbeca37cc5701 (127.0.107.195) to 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193). New cstate: current_term: 3 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } }
I20260709 12:59:42.280884  3529 raft_consensus.cc:515] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:42.303429  1603 raft_consensus.cc:1275] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260709 12:59:42.307073  1601 raft_consensus.cc:1275] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 5. (term mismatch)
W20260709 12:59:42.302675  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 49461.2us
I20260709 12:59:42.309419  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:42.309854  1600 raft_consensus.cc:1240] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 3. Current term is 4. Ops: []
W20260709 12:59:42.305704  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 55157.6us
I20260709 12:59:42.312667  2336 raft_consensus.cc:3060] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 2 LEARNER]: Advancing to term 4
I20260709 12:59:42.309718  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:42.300752  1501 catalog_manager.cc:5697] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d reported cstate change: config changed from index -1 to 3, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New cstate: current_term: 2 leader_uuid: "af520b67945d443b901bbeca37cc5701" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } } }
I20260709 12:59:42.311391  1601 pending_rounds.cc:85] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a: Aborting all ops after (but not including) 4
I20260709 12:59:42.323271  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:42.303944  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:42.318121  1601 pending_rounds.cc:107] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a: Aborting uncommitted NO_OP operation due to leader change: 2.5
I20260709 12:59:42.349710  1601 raft_consensus.cc:2889] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:59:42.304230  3657 raft_consensus.cc:493] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 12:59:42.305509  3529 leader_election.cc:290] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 4 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:42.360807  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:42.360909  3764 raft_consensus.cc:2749] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:59:42.324193  3639 raft_consensus.cc:697] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 12:59:42.343143  3716 raft_consensus.cc:493] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:42.371521  1605 raft_consensus.cc:2468] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 4.
I20260709 12:59:42.319744  3715 raft_consensus.cc:493] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:42.372440  1718 leader_election.cc:304] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:42.383327  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:59:42.384441  1671 raft_consensus.cc:2468] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 2.
I20260709 12:59:42.384974  3716 raft_consensus.cc:515] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:42.386247  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:42.387603  1596 raft_consensus.cc:2468] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 3.
I20260709 12:59:42.387984  3716 leader_election.cc:290] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:42.388803  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:42.389295  3639 consensus_queue.cc:237] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:42.384053  3715 raft_consensus.cc:515] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:42.396920  3715 leader_election.cc:290] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:42.398741  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:42.403102  1677 raft_consensus.cc:1240] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 4. Current term is 5. Ops: []
W20260709 12:59:42.403533  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 45208.8us
W20260709 12:59:42.405594  3756 log.cc:927] Time spent T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.125s	user 0.001s	sys 0.000s
I20260709 12:59:42.405282  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:42.406581  1677 raft_consensus.cc:2468] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 2.
I20260709 12:59:42.410550  1677 raft_consensus.cc:1275] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260709 12:59:42.412925  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:59:42.414420  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
W20260709 12:59:42.418423  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 67280.4us
I20260709 12:59:42.418252  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:59:42.423022  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:42.430794  2336 raft_consensus.cc:1275] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 4 LEARNER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
W20260709 12:59:42.432327  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 110000us
W20260709 12:59:42.435017  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 101486us
I20260709 12:59:42.436554  1671 raft_consensus.cc:3060] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:59:42.374641  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:42.372903  3640 raft_consensus.cc:697] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 12:59:42.414388  3543 raft_consensus.cc:493] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:42.367851  3657 raft_consensus.cc:515] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:42.447530  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:42.458429  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:42.458958  1672 raft_consensus.cc:2468] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 2.
I20260709 12:59:42.449736  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:42.461730  3657 leader_election.cc:290] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:42.462598  3687 consensus_queue.cc:1048] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.023s
I20260709 12:59:42.463088  2330 raft_consensus.cc:2468] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 3.
I20260709 12:59:42.464187  3687 consensus_queue.cc:1059] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:59:42.418720  3715 raft_consensus.cc:2804] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:59:42.448824  1646 leader_election.cc:304] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:59:42.436566  2298 leader_election.cc:304] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 12:59:42.434288  3752 consensus_queue.cc:1059] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.039s
I20260709 12:59:42.488466  1718 leader_election.cc:304] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 12:59:42.483309  1646 leader_election.cc:304] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:59:42.489427  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:59:42.490229  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:59:42.532172  1676 raft_consensus.cc:3060] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Advancing to term 4
I20260709 12:59:42.498389  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:42.492326  2299 leader_election.cc:304] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:42.537112  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:59:42.489818  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:42.539458  1678 raft_consensus.cc:2468] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 4.
I20260709 12:59:42.541569  1673 raft_consensus.cc:2468] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 4.
W20260709 12:59:42.542204  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 59011.8us
I20260709 12:59:42.484426  1680 raft_consensus.cc:2468] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 3.
I20260709 12:59:42.449880  3608 raft_consensus.cc:697] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 12:59:42.483542  3764 raft_consensus.cc:2804] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:59:42.558969  1600 raft_consensus.cc:2368] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 3. Current term is 4.
I20260709 12:59:42.488937  3543 raft_consensus.cc:515] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 12:59:42.560555  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 294628us
I20260709 12:59:42.561357  3543 leader_election.cc:290] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:42.487332  3521 consensus_queue.cc:1048] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.075s
I20260709 12:59:42.562759  3753 raft_consensus.cc:2749] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:59:42.563309  3663 raft_consensus.cc:2804] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:59:42.481885  3640 consensus_queue.cc:237] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:42.566499  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
W20260709 12:59:42.567037  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 78682.7us
I20260709 12:59:42.567277  1603 raft_consensus.cc:3060] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:59:42.567587  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:42.568465  1598 raft_consensus.cc:2393] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 4: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 12:59:42.569475  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:42.458485  3716 consensus_queue.cc:1048] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.091s
I20260709 12:59:42.570066  1598 raft_consensus.cc:2468] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 3.
I20260709 12:59:42.570923  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:42.571540  1747 raft_consensus.cc:3060] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:59:42.571928  3716 consensus_queue.cc:1048] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:59:42.486964  3768 consensus_queue.cc:1059] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.087s
W20260709 12:59:42.561358  1643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:59:42.579200  1643 leader_election.cc:336] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 1.846s (SENT)
I20260709 12:59:42.559044  3608 consensus_queue.cc:237] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:42.579694  1643 leader_election.cc:304] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 12:59:42.580611  3716 raft_consensus.cc:493] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:42.581813  3716 raft_consensus.cc:515] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:42.582970  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:42.469131  3715 raft_consensus.cc:493] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 12:59:42.586966  1600 raft_consensus.cc:1275] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:59:42.487833  3728 consensus_queue.cc:1059] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.096s
I20260709 12:59:42.589845  1605 raft_consensus.cc:1275] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:59:42.590457  1671 raft_consensus.cc:2468] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 5.
I20260709 12:59:42.608570  1747 raft_consensus.cc:2468] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 3.
I20260709 12:59:42.610906  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:42.612242  1676 raft_consensus.cc:2468] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 4.
I20260709 12:59:42.614981  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:42.617373  1745 raft_consensus.cc:2468] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 4.
I20260709 12:59:42.615163  1751 raft_consensus.cc:1240] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 4. Current term is 5. Ops: []
I20260709 12:59:42.484747  3761 consensus_queue.cc:1048] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.209s
I20260709 12:59:42.583254  3764 raft_consensus.cc:493] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:59:42.609642  1718 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.107.196:46625 timed out after 2.002s (SENT) [suppressed 83 similar messages]
I20260709 12:59:42.491127  3765 consensus_queue.cc:1059] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.160s
I20260709 12:59:42.623410  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:42.586700  3715 raft_consensus.cc:3060] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:42.629469  1603 raft_consensus.cc:2468] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 5.
I20260709 12:59:42.619319  3663 raft_consensus.cc:697] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 12:59:42.589536  3753 raft_consensus.cc:493] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
W20260709 12:59:42.645797  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 293696us
W20260709 12:59:42.653264  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 43678.8us
I20260709 12:59:42.637732  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:42.657224  1600 raft_consensus.cc:1275] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:59:42.657264  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:42.619508  3764 raft_consensus.cc:3060] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:59:42.636600  3414 raft_consensus.cc:2749] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:59:42.670806  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:42.672430  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:42.672454  1605 raft_consensus.cc:2393] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 3: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 12:59:42.678867  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:42.652311  3706 raft_consensus.cc:493] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
W20260709 12:59:42.682279  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 103148us
I20260709 12:59:42.687052  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
W20260709 12:59:42.685863  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 221224us
I20260709 12:59:42.694077  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:42.702286  1603 raft_consensus.cc:1240] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 2. Current term is 3. Ops: []
I20260709 12:59:42.702545  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:42.703346  1601 raft_consensus.cc:2393] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 5: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 12:59:42.708778  1600 raft_consensus.cc:1240] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 3. Current term is 4. Ops: []
I20260709 12:59:42.711261  1604 raft_consensus.cc:1240] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 2. Current term is 3. Ops: []
I20260709 12:59:42.711736  3687 consensus_queue.cc:1059] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:59:42.712924  3687 consensus_queue.cc:1059] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:59:42.713310  1599 raft_consensus.cc:1240] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 3. Current term is 4. Ops: []
I20260709 12:59:42.717188  3773 raft_consensus.cc:3055] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 LEADER]: Stepping down as leader of term 2
W20260709 12:59:42.722141  3776 log.cc:927] Time spent T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.050s	user 0.000s	sys 0.001s
W20260709 12:59:42.729825  3769 log.cc:927] Time spent T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.074s	user 0.000s	sys 0.001s
I20260709 12:59:42.730085  3464 raft_consensus.cc:3055] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:59:42.731935  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:59:42.732070  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:42.732550  3702 consensus_queue.cc:1048] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.040s
I20260709 12:59:42.733644  1598 raft_consensus.cc:1240] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 1. Current term is 2. Ops: []
I20260709 12:59:42.733760  3760 raft_consensus.cc:2804] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:59:42.765956  3760 raft_consensus.cc:493] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 12:59:42.766382  3760 raft_consensus.cc:3060] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:59:42.750478  3414 consensus_queue.cc:1059] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.012s
I20260709 12:59:42.734185  3687 raft_consensus.cc:493] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:42.738737  3358 raft_consensus.cc:493] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 12:59:42.739094  3770 raft_consensus.cc:3055] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:59:42.738679  3663 consensus_queue.cc:237] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:42.741600  3716 leader_election.cc:290] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:42.722836  3521 raft_consensus.cc:3055] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:59:42.700098  3706 raft_consensus.cc:515] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:42.693264  3639 raft_consensus.cc:493] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 12:59:42.746848  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:42.748029  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:42.749126  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:42.749792  1603 raft_consensus.cc:1240] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 LEADER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 4. Current term is 5. Ops: []
I20260709 12:59:42.750788  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:42.741348  3780 raft_consensus.cc:2804] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:59:42.751629  1646 leader_election.cc:304] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
W20260709 12:59:42.798566  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 47032.8us
I20260709 12:59:42.770783  3687 raft_consensus.cc:515] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:42.758459  3464 raft_consensus.cc:740] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 12:59:42.681418  3753 raft_consensus.cc:515] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:42.799371  1646 leader_election.cc:304] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:59:42.804348  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:42.769039  1601 raft_consensus.cc:1240] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 4. Current term is 5. Ops: [4.3-4.3]
W20260709 12:59:42.771679  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 508357us
I20260709 12:59:42.768781  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 12:59:42.772174  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 75522.4us
I20260709 12:59:42.729254  3773 raft_consensus.cc:740] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 12:59:42.783833  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:42.780025  3770 raft_consensus.cc:740] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 12:59:42.775051  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 12:59:42.814134  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 322027us
W20260709 12:59:42.755889  1643 leader_election.cc:336] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 1.869s (SENT)
I20260709 12:59:42.802512  1599 raft_consensus.cc:2368] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for earlier term 3. Current term is 4.
I20260709 12:59:42.817391  1643 leader_election.cc:304] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:42.779673  3765 raft_consensus.cc:3055] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:59:42.810385  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:42.786316  3728 raft_consensus.cc:493] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:42.770741  3358 raft_consensus.cc:515] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:42.813295  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:42.826402  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:42.815822  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:42.801726  1597 raft_consensus.cc:2393] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 3: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 12:59:42.784103  3521 raft_consensus.cc:740] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 12:59:42.796835  1600 raft_consensus.cc:2368] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for earlier term 4. Current term is 5.
I20260709 12:59:42.801865  3414 raft_consensus.cc:2804] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:59:42.820832  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:42.818939  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:42.838742  1601 raft_consensus.cc:2393] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 4: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 12:59:42.824779  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:42.828231  3782 consensus_queue.cc:1048] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.044s
I20260709 12:59:42.820411  3780 raft_consensus.cc:697] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 2 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 12:59:42.830649  3358 leader_election.cc:290] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:42.831004  3771 raft_consensus.cc:2804] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:59:42.831382  2336 raft_consensus.cc:1275] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 12:59:42.811450  3552 consensus_queue.cc:1059] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.020s
I20260709 12:59:42.832472  1598 raft_consensus.cc:1275] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:59:42.843808  1602 raft_consensus.cc:2393] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 4: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 12:59:42.800434  3639 raft_consensus.cc:515] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:42.848448  3552 raft_consensus.cc:3055] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:59:42.848975  3552 raft_consensus.cc:740] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 12:59:42.821591  3765 raft_consensus.cc:740] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 12:59:42.835597  3789 raft_consensus.cc:3055] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:59:42.818584  3768 consensus_queue.cc:1059] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.027s
I20260709 12:59:42.903642  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:42.861471  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:42.857245  3789 raft_consensus.cc:740] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 12:59:42.922425  3552 consensus_queue.cc:260] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:42.923296  3789 consensus_queue.cc:260] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:42.924643  3552 raft_consensus.cc:3060] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:59:42.878688  3529 consensus_queue.cc:1048] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.047s
W20260709 12:59:42.927100  3766 log.cc:927] Time spent T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.270s	user 0.002s	sys 0.000s
W20260709 12:59:42.928172  3786 log.cc:927] Time spent T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.076s	user 0.000s	sys 0.000s
I20260709 12:59:42.927968  3773 consensus_queue.cc:260] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:42.850173  3608 raft_consensus.cc:2749] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:59:42.929585  3773 raft_consensus.cc:3060] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:59:42.930197  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:42.931872  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:42.883471  3753 leader_election.cc:290] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:42.838266  3728 raft_consensus.cc:515] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:42.934809  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:42.936446  3728 leader_election.cc:290] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 12:59:42.865046  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 136577us
W20260709 12:59:42.938493  1643 leader_election.cc:336] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 1.870s (SENT)
W20260709 12:59:42.865556  1645 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:59:42.865912  3787 consensus_queue.cc:1059] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:59:42.843458  3414 raft_consensus.cc:493] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:59:42.940769  3414 raft_consensus.cc:3060] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:59:42.866485  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:42.867266  1605 raft_consensus.cc:1240] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 3. Current term is 4. Ops: []
I20260709 12:59:42.857198  3464 consensus_queue.cc:260] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:42.868191  3780 consensus_queue.cc:237] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:42.868335  3521 consensus_queue.cc:260] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:42.869490  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:59:42.869505  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:42.862129  3783 raft_consensus.cc:3055] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:59:42.977330  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:42.870018  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:42.830713  3687 leader_election.cc:290] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:42.871062  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:42.873699  3771 raft_consensus.cc:697] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 12:59:42.885118  3706 leader_election.cc:290] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:42.885780  3638 consensus_queue.cc:1048] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:59:42.886880  3471 consensus_queue.cc:1048] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:59:42.887965  1569 leader_election.cc:304] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 12:59:42.889444  3466 consensus_queue.cc:1048] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:59:42.893491  3639 leader_election.cc:290] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:42.883019  3764 raft_consensus.cc:515] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:42.883997  3765 consensus_queue.cc:260] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:42.924813  3789 raft_consensus.cc:3060] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:59:42.863700  3715 raft_consensus.cc:515] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } }
W20260709 12:59:42.957569  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 140025us
I20260709 12:59:42.948742  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:42.971735  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:42.980156  1604 raft_consensus.cc:2368] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for earlier term 4. Current term is 5.
W20260709 12:59:42.983435  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 44993.2us
I20260709 12:59:43.022397  1747 raft_consensus.cc:1275] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
W20260709 12:59:43.037110  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 198822us
I20260709 12:59:43.037251  2299 leader_election.cc:304] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: af520b67945d443b901bbeca37cc5701
I20260709 12:59:43.039814  3788 raft_consensus.cc:493] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 12:59:43.040654  3788 raft_consensus.cc:515] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:43.042652  3657 raft_consensus.cc:493] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:43.043362  3657 raft_consensus.cc:515] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:43.046643  3761 raft_consensus.cc:493] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:43.047348  3761 raft_consensus.cc:515] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:43.050901  3781 raft_consensus.cc:2804] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:59:43.051479  3781 raft_consensus.cc:493] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Starting leader election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 12:59:43.051921  3781 raft_consensus.cc:3060] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:59:43.055943  3687 consensus_queue.cc:1048] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 12:59:43.056294  2332 raft_consensus.cc:1275] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 12:59:43.057526  3687 consensus_queue.cc:1059] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 12:59:43.057955  3790 raft_consensus.cc:3055] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:59:42.981092  3728 raft_consensus.cc:493] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 12:59:42.897943  3770 consensus_queue.cc:260] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:43.059159  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 12:59:43.061259  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 173420us
W20260709 12:59:43.069396  1643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:59:43.059769  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 12:59:43.059353  1644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:59:43.035228  1751 raft_consensus.cc:2468] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 2.
I20260709 12:59:43.059355  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 12:59:43.092615  1643 leader_election.cc:336] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 1.927s (SENT)
I20260709 12:59:43.074147  3702 consensus_queue.cc:1048] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.019s
I20260709 12:59:43.061372  3466 raft_consensus.cc:2804] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:59:43.093297  1643 leader_election.cc:304] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 12:59:43.094111  3466 raft_consensus.cc:697] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 12:59:43.095569  3466 consensus_queue.cc:237] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:43.097200  3622 consensus_queue.cc:1059] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:59:43.098030  3529 consensus_queue.cc:1059] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:59:43.058984  3728 raft_consensus.cc:515] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:43.098578  3471 consensus_queue.cc:1059] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:59:43.099298  3529 raft_consensus.cc:3055] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:59:43.099437  2299 leader_election.cc:304] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: af520b67945d443b901bbeca37cc5701
I20260709 12:59:43.099695  3471 raft_consensus.cc:3055] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:59:43.099740  3529 raft_consensus.cc:740] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 12:59:43.100162  3471 raft_consensus.cc:740] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 12:59:43.100778  3622 raft_consensus.cc:3055] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:59:43.101017  3529 consensus_queue.cc:260] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:43.101500  3622 raft_consensus.cc:740] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 12:59:43.103587  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 12:59:43.121034  2336 raft_consensus.cc:3060] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:59:43.112082  3764 leader_election.cc:290] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 7 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:43.112681  3640 consensus_queue.cc:1048] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:59:43.061058  3715 leader_election.cc:290] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 2 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:43.117309  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:43.050222  3761 leader_election.cc:290] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:43.139104  1752 raft_consensus.cc:2468] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 3.
I20260709 12:59:43.140688  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:43.046216  3657 leader_election.cc:290] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:43.144022  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:43.069600  3706 raft_consensus.cc:493] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:43.142098  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:42.977735  3783 raft_consensus.cc:740] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
W20260709 12:59:43.148113  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 55574.6us
I20260709 12:59:43.058411  3790 raft_consensus.cc:740] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 12:59:43.150755  1752 raft_consensus.cc:3060] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:59:43.155884  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:43.150841  1569 leader_election.cc:304] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 12:59:43.127454  3464 raft_consensus.cc:3060] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:59:43.160138  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 60797.4us
I20260709 12:59:43.166213  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:43.166771  1718 leader_election.cc:304] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:59:43.170558  3638 consensus_queue.cc:1048] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:59:43.171094  3783 consensus_queue.cc:260] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:43.171890  3788 leader_election.cc:290] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:43.172655  3783 raft_consensus.cc:3060] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:59:43.172958  3728 leader_election.cc:290] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:43.173337  3622 consensus_queue.cc:260] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:43.174707  3622 raft_consensus.cc:3060] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:59:43.174961  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:43.175062  3790 consensus_queue.cc:260] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:43.176242  3790 raft_consensus.cc:3060] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:59:43.176371  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:43.177237  1599 raft_consensus.cc:2368] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for earlier term 3. Current term is 4.
W20260709 12:59:43.177843  1643 leader_election.cc:336] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 1.982s (SENT)
I20260709 12:59:43.179129  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:43.156715  1753 raft_consensus.cc:2468] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 2.
I20260709 12:59:43.180730  1646 leader_election.cc:304] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:59:43.162627  2301 leader_election.cc:304] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:59:43.149577  3687 raft_consensus.cc:2804] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:59:43.147063  3639 raft_consensus.cc:2749] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260709 12:59:43.210640  2299 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:59:43.210500  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 47993.4us
I20260709 12:59:43.209555  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:43.208984  1751 raft_consensus.cc:1275] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:59:43.210466  1673 raft_consensus.cc:1240] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 2. Current term is 3. Ops: []
I20260709 12:59:43.220163  1671 raft_consensus.cc:2468] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 3.
I20260709 12:59:43.220592  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 12:59:43.221792  1643 leader_election.cc:336] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 2.024s (SENT)
I20260709 12:59:43.222280  1643 leader_election.cc:304] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 12:59:43.224578  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:43.225579  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:43.172652  3638 raft_consensus.cc:2804] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:59:43.171139  3793 raft_consensus.cc:3055] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:59:43.191205  3521 raft_consensus.cc:3060] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:59:43.190892  3640 raft_consensus.cc:2804] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:59:43.255380  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:43.256630  2332 raft_consensus.cc:2393] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 3: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 12:59:43.258700  1676 raft_consensus.cc:2468] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 3.
I20260709 12:59:43.262142  1754 raft_consensus.cc:1275] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
W20260709 12:59:43.263384  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 224282us
I20260709 12:59:43.263664  2301 leader_election.cc:304] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:59:43.266222  2336 raft_consensus.cc:2468] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 4.
I20260709 12:59:43.275969  2298 leader_election.cc:304] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 12:59:43.191850  3753 consensus_queue.cc:1048] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.084s
I20260709 12:59:43.227927  3728 raft_consensus.cc:2804] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:59:43.179554  3706 raft_consensus.cc:515] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:43.277201  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:43.280685  2298 leader_election.cc:304] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 12:59:43.226600  3797 raft_consensus.cc:2804] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:59:43.242699  3638 raft_consensus.cc:493] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:59:43.287798  3638 raft_consensus.cc:3060] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:59:43.261363  3764 raft_consensus.cc:2749] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:59:43.283169  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:43.283561  2335 raft_consensus.cc:2393] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 3: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 12:59:43.259366  3529 raft_consensus.cc:3060] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:59:43.069721  3771 consensus_queue.cc:237] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:43.261598  3793 raft_consensus.cc:740] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
W20260709 12:59:43.286621  3795 log.cc:927] Time spent T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.075s	user 0.001s	sys 0.000s
I20260709 12:59:43.273025  3687 raft_consensus.cc:697] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 4 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 12:59:43.159660  3471 consensus_queue.cc:260] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:43.277508  3640 raft_consensus.cc:493] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 12:59:43.280731  3780 raft_consensus.cc:2804] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:59:43.281508  3639 consensus_queue.cc:1048] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.005s
I20260709 12:59:43.281975  3728 raft_consensus.cc:493] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:59:43.306905  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 12:59:43.308215  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 86452.1us
W20260709 12:59:43.310752  1570 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:59:43.290333  3797 raft_consensus.cc:493] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:59:43.313397  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 12:59:43.302513  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 152448us
I20260709 12:59:43.329322  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 12:59:43.335206  1643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:59:43.352401  1599 raft_consensus.cc:2393] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 4: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 12:59:43.351079  1569 leader_election.cc:304] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 12:59:43.304627  3788 raft_consensus.cc:2804] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260709 12:59:43.367576  1643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:59:43.303488  3358 raft_consensus.cc:493] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 12:59:43.374004  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:43.308383  3640 raft_consensus.cc:3060] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:59:43.315063  3803 log.cc:927] Time spent T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.054s	user 0.000s	sys 0.001s
I20260709 12:59:43.378408  3725 raft_consensus.cc:2804] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:59:43.378779  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:43.357249  3797 raft_consensus.cc:3060] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:59:43.310047  3780 raft_consensus.cc:493] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:59:43.357040  3728 raft_consensus.cc:3060] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:59:43.425212  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:43.435122  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:43.431880  2299 leader_election.cc:304] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
W20260709 12:59:43.380671  3760 consensus_meta.cc:287] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.614s	user 0.014s	sys 0.001s
W20260709 12:59:43.433086  1643 leader_election.cc:336] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 2.042s (SENT)
I20260709 12:59:43.375479  3788 raft_consensus.cc:493] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 12:59:43.441927  1643 leader_election.cc:304] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 12:59:43.428617  3657 raft_consensus.cc:493] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:43.426528  3725 raft_consensus.cc:493] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:59:43.439015  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:59:43.445425  1671 raft_consensus.cc:2468] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 1.
I20260709 12:59:43.435182  3780 raft_consensus.cc:3060] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:59:43.454325  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:43.454097  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:43.379447  3358 raft_consensus.cc:515] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:43.454034  3800 raft_consensus.cc:2804] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:59:43.446946  3793 consensus_queue.cc:260] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:43.458091  2299 leader_election.cc:304] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 12:59:43.460903  1600 raft_consensus.cc:1240] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 LEADER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 4. Current term is 5. Ops: []
I20260709 12:59:43.440502  3471 raft_consensus.cc:3060] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:59:43.463467  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:59:43.440812  3788 raft_consensus.cc:3060] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:59:43.470713  1604 raft_consensus.cc:2393] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 4: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 12:59:43.433583  3767 consensus_queue.cc:1048] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.227s
I20260709 12:59:43.436040  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:43.455006  3725 raft_consensus.cc:3060] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:59:43.491930  1645 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:59:43.461424  3752 raft_consensus.cc:2804] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260709 12:59:43.496243  2300 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:59:43.467451  3543 raft_consensus.cc:2749] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 12:59:43.500022  3799 log.cc:927] Time spent T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.210s	user 0.000s	sys 0.001s
I20260709 12:59:43.464617  3800 raft_consensus.cc:493] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
W20260709 12:59:43.508152  3802 log.cc:927] Time spent T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.088s	user 0.001s	sys 0.000s
I20260709 12:59:43.510030  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:59:43.510926  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 12:59:43.551714  1645 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:59:43.498793  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:43.552402  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:59:43.552753  1600 raft_consensus.cc:2368] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for earlier term 3. Current term is 4.
I20260709 12:59:43.511996  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
W20260709 12:59:43.513386  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 162409us
I20260709 12:59:43.441052  3760 raft_consensus.cc:515] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:43.519361  1752 raft_consensus.cc:2468] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 7.
I20260709 12:59:43.472437  3753 raft_consensus.cc:493] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 12:59:43.526670  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:59:43.458187  3657 raft_consensus.cc:515] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:43.554241  1569 leader_election.cc:304] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 12:59:43.559489  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:43.560909  1569 leader_election.cc:304] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 12:59:43.555749  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
W20260709 12:59:43.570420  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 112408us
I20260709 12:59:43.486711  3663 consensus_queue.cc:1048] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.066s
W20260709 12:59:43.574340  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 150012us
I20260709 12:59:43.526593  3639 raft_consensus.cc:493] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:43.576812  3794 raft_consensus.cc:2749] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:59:43.577648  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:43.579195  1569 leader_election.cc:304] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 12:59:43.581305  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:43.522560  3800 raft_consensus.cc:3060] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:59:43.593549  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 124431us
I20260709 12:59:43.575465  3466 raft_consensus.cc:2749] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:59:43.473366  3687 consensus_queue.cc:237] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:43.611378  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:59:43.579896  3715 raft_consensus.cc:493] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:59:43.606560  1643 leader_election.cc:336] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 2.074s (SENT)
I20260709 12:59:43.614053  1643 leader_election.cc:304] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 12:59:43.607064  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 12:59:43.608306  1717 leader_election.cc:336] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 1.649s (SENT)
I20260709 12:59:43.610344  1646 leader_election.cc:304] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:59:43.626405  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:43.596828  3706 leader_election.cc:290] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:43.563957  3753 raft_consensus.cc:515] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:43.627770  1598 raft_consensus.cc:2468] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 3.
I20260709 12:59:43.633327  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:43.635272  1717 leader_election.cc:304] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:43.646857  3497 raft_consensus.cc:2749] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:59:43.590163  3639 raft_consensus.cc:515] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:43.626197  3543 raft_consensus.cc:2804] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:59:43.665032  3657 leader_election.cc:290] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:43.637943  3466 raft_consensus.cc:2804] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:59:43.667804  3466 raft_consensus.cc:697] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 12:59:43.668944  3466 consensus_queue.cc:237] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:43.670775  3706 raft_consensus.cc:2749] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 12:59:43.671324  3813 log.cc:927] Time spent T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.098s	user 0.001s	sys 0.000s
W20260709 12:59:43.671831  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 63556us
W20260709 12:59:43.672700  3414 consensus_meta.cc:287] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.731s	user 0.010s	sys 0.006s
W20260709 12:59:43.674644  3806 log.cc:927] Time spent T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.230s	user 0.000s	sys 0.001s
I20260709 12:59:43.674525  3657 raft_consensus.cc:493] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
W20260709 12:59:43.675904  3812 log.cc:927] Time spent T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.085s	user 0.000s	sys 0.000s
I20260709 12:59:43.638062  3794 raft_consensus.cc:493] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:59:43.680526  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 78971us
W20260709 12:59:43.655037  3820 log.cc:927] Time spent T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.100s	user 0.000s	sys 0.001s
I20260709 12:59:43.632383  3816 raft_consensus.cc:2804] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:59:43.692453  3414 raft_consensus.cc:515] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:43.632875  3715 raft_consensus.cc:515] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 12:59:43.719192  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 146934us
W20260709 12:59:43.707798  3552 consensus_meta.cc:287] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.774s	user 0.013s	sys 0.006s
W20260709 12:59:43.725025  1643 leader_election.cc:336] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 2.103s (SENT)
I20260709 12:59:43.724658  2299 leader_election.cc:304] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 12:59:43.727648  1680 raft_consensus.cc:1240] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260709 12:59:43.728571  1601 raft_consensus.cc:2468] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 3.
I20260709 12:59:43.733978  3414 leader_election.cc:290] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 5 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:43.730924  1718 leader_election.cc:304] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
W20260709 12:59:43.736105  3781 consensus_meta.cc:287] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.684s	user 0.014s	sys 0.001s
I20260709 12:59:43.736701  3810 raft_consensus.cc:493] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 12:59:43.736629  3781 raft_consensus.cc:515] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 12:59:43.737360  3808 log.cc:927] Time spent T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.200s	user 0.001s	sys 0.001s
I20260709 12:59:43.737972  2334 raft_consensus.cc:1240] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 3. Current term is 4. Ops: []
I20260709 12:59:43.738279  3800 raft_consensus.cc:515] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:43.738653  3638 raft_consensus.cc:515] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:43.740831  3638 leader_election.cc:290] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 3 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:43.737426  3810 raft_consensus.cc:515] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 12:59:43.737039  1719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:59:43.709007  3543 raft_consensus.cc:697] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 12:59:43.748445  1603 raft_consensus.cc:2468] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 4.
W20260709 12:59:43.736876  3818 log.cc:927] Time spent T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.057s	user 0.000s	sys 0.000s
I20260709 12:59:43.769562  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:43.709600  3657 raft_consensus.cc:515] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 12:59:43.772428  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 44092.7us
I20260709 12:59:43.756108  3782 raft_consensus.cc:2804] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:59:43.787001  3543 consensus_queue.cc:237] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:43.790509  1602 raft_consensus.cc:2468] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 4.
I20260709 12:59:43.783955  3706 raft_consensus.cc:2749] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:59:43.799628  1604 raft_consensus.cc:1275] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:59:43.728874  3471 consensus_queue.cc:1059] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.071s
I20260709 12:59:43.720635  3794 raft_consensus.cc:515] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:43.816434  1752 raft_consensus.cc:1275] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:59:43.859539  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:43.828270  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:43.872712  1745 raft_consensus.cc:2468] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 2.
W20260709 12:59:43.875078  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 147505us
W20260709 12:59:43.877260  2299 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:59:43.880110  3752 raft_consensus.cc:493] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Starting leader election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 12:59:43.880784  3752 raft_consensus.cc:3060] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:59:43.896538  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:43.897691  1745 raft_consensus.cc:2393] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 5: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 12:59:43.900470  1600 raft_consensus.cc:1240] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 3. Current term is 4. Ops: []
I20260709 12:59:43.912358  1598 raft_consensus.cc:1275] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:59:43.917414  1752 raft_consensus.cc:1275] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 12:59:43.918303  1745 raft_consensus.cc:1275] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 12:59:43.921674  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:43.921840  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:43.925482  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:43.926868  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:43.927772  1749 raft_consensus.cc:2468] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 4.
I20260709 12:59:43.929692  1748 raft_consensus.cc:1240] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 2. Current term is 3. Ops: []
I20260709 12:59:43.933112  1717 leader_election.cc:304] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:43.934144  3715 leader_election.cc:290] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 12:59:43.941259  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 63372.6us
W20260709 12:59:43.942736  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 47431.7us
I20260709 12:59:43.914235  3753 leader_election.cc:290] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:43.944661  1717 leader_election.cc:304] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1
W20260709 12:59:43.869843  3622 consensus_meta.cc:287] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.683s	user 0.008s	sys 0.004s
I20260709 12:59:43.800341  3358 leader_election.cc:290] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:43.801733  3782 raft_consensus.cc:493] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
W20260709 12:59:43.953346  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 76685.7us
W20260709 12:59:43.953594  1570 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:59:43.870771  3790 consensus_meta.cc:287] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.694s	user 0.014s	sys 0.002s
W20260709 12:59:43.972357  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 251301us
I20260709 12:59:43.973101  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:59:43.859707  3657 leader_election.cc:290] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:43.863889  3810 leader_election.cc:290] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:43.856019  3781 leader_election.cc:290] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 4 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:43.823973  3760 leader_election.cc:290] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 3 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 12:59:43.952152  3789 consensus_meta.cc:287] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.894s	user 0.013s	sys 0.013s
W20260709 12:59:43.952154  3806 log.cc:927] Time spent T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.135s	user 0.000s	sys 0.001s
I20260709 12:59:44.014782  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:43.852959  3829 consensus_queue.cc:1048] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.257s
I20260709 12:59:43.980094  3639 leader_election.cc:290] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:44.009313  3782 raft_consensus.cc:3060] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:59:44.025633  1643 leader_election.cc:336] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 2.124s (SENT)
I20260709 12:59:44.013423  2333 raft_consensus.cc:2468] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 3.
W20260709 12:59:44.028779  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 297897us
I20260709 12:59:44.026219  1643 leader_election.cc:304] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 12:59:43.954824  3715 raft_consensus.cc:3060] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:59:44.045802  1569 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:59:43.944736  3780 raft_consensus.cc:515] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:44.057089  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:44.040762  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:44.062484  2328 raft_consensus.cc:2468] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 3.
W20260709 12:59:44.063664  3521 consensus_meta.cc:287] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.787s	user 0.014s	sys 0.000s
I20260709 12:59:44.064373  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:44.065058  1748 raft_consensus.cc:3060] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:59:44.066684  3821 consensus_queue.cc:1059] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.036s
I20260709 12:59:44.069499  1752 raft_consensus.cc:2410] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260709 12:59:44.049741  3639 raft_consensus.cc:2749] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260709 12:59:44.063756  3825 log.cc:927] Time spent T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.205s	user 0.000s	sys 0.001s
W20260709 12:59:44.073603  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 48058.9us
W20260709 12:59:44.075896  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 132208us
W20260709 12:59:44.078305  3464 consensus_meta.cc:287] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.891s	user 0.011s	sys 0.002s
W20260709 12:59:44.079353  3797 consensus_meta.cc:287] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.640s	user 0.011s	sys 0.000s
I20260709 12:59:44.115530  2336 raft_consensus.cc:2468] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 3.
I20260709 12:59:44.079303  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:44.128043  1754 raft_consensus.cc:1275] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260709 12:59:44.132454  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:44.132715  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:44.140532  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
W20260709 12:59:44.140861  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 64152.2us
I20260709 12:59:44.079797  3782 raft_consensus.cc:515] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:44.144343  2301 leader_election.cc:304] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:44.145560  1754 raft_consensus.cc:2393] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 3: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 12:59:44.122246  3639 raft_consensus.cc:493] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 12:59:44.145833  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:44.157164  1676 raft_consensus.cc:2393] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 5: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 12:59:44.150805  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:44.132691  3543 raft_consensus.cc:493] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:44.172646  1672 raft_consensus.cc:1275] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:59:43.954705   431 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
W20260709 12:59:44.150288  3788 consensus_meta.cc:287] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.663s	user 0.007s	sys 0.005s
I20260709 12:59:44.179770  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:44.180317  3788 raft_consensus.cc:515] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:44.137135  3414 raft_consensus.cc:493] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 12:59:44.158427  1679 raft_consensus.cc:2468] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 1.
I20260709 12:59:44.160512  1599 raft_consensus.cc:2468] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 5.
W20260709 12:59:44.179180  3728 consensus_meta.cc:287] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.732s	user 0.012s	sys 0.000s
I20260709 12:59:44.179875  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:44.200114  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:44.176048  1748 raft_consensus.cc:2468] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 4.
I20260709 12:59:44.183184  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:44.126789  3752 raft_consensus.cc:515] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:44.187182  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:59:44.120426  3797 raft_consensus.cc:515] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:44.195372  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:44.195842  3780 leader_election.cc:290] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 4 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:44.196818  3782 leader_election.cc:290] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 2 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:44.158346  3794 leader_election.cc:290] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:44.198261  3788 leader_election.cc:290] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 5 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:44.144901  3816 raft_consensus.cc:697] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 12:59:44.198762  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 12:59:44.201673  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 59493.6us
I20260709 12:59:44.206736  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:44.219130  1680 raft_consensus.cc:2393] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 3: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
W20260709 12:59:44.186369  3773 consensus_meta.cc:287] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 1.256s	user 0.020s	sys 0.008s
I20260709 12:59:44.221321  1674 raft_consensus.cc:2393] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 4: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 12:59:44.223498  1672 raft_consensus.cc:2468] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 2.
I20260709 12:59:44.222275  1676 raft_consensus.cc:3060] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:44.145804  3464 consensus_queue.cc:1048] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.070s
I20260709 12:59:44.227632  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:44.225028  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:44.274402  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:59:44.229409  3800 leader_election.cc:290] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 4 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:44.229538  3529 raft_consensus.cc:493] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:44.216681  3687 raft_consensus.cc:2804] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:59:44.233771  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:44.236387  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:44.243057  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:44.245779  1571 leader_election.cc:304] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 12:59:44.176818  3639 raft_consensus.cc:515] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:44.266892  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:44.194711  3543 raft_consensus.cc:515] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:44.216238  3414 raft_consensus.cc:515] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:44.300446  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:44.286509  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:44.221716  3728 raft_consensus.cc:515] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:44.268211  3608 raft_consensus.cc:493] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:44.275135  3761 raft_consensus.cc:493] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:44.307224  1754 raft_consensus.cc:1240] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 1. Current term is 2. Ops: []
I20260709 12:59:44.285715  3529 raft_consensus.cc:515] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:44.307808  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 12:59:44.308358  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 62687.3us
W20260709 12:59:44.308837  3836 log.cc:927] Time spent T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.195s	user 0.001s	sys 0.000s
I20260709 12:59:44.300755  3622 raft_consensus.cc:2749] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:59:44.306121  3816 consensus_queue.cc:237] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:44.316090  3639 leader_election.cc:290] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:44.321401  3728 leader_election.cc:290] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 4 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:44.380882  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:59:44.381574  1605 raft_consensus.cc:1275] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:59:44.322297  1753 raft_consensus.cc:1240] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 3. Current term is 4. Ops: []
W20260709 12:59:44.322708  3640 consensus_meta.cc:287] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.887s	user 0.013s	sys 0.001s
I20260709 12:59:44.322844  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:44.324889  3414 leader_election.cc:290] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:44.301352  3780 raft_consensus.cc:493] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:44.335836  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:59:44.338169  3752 leader_election.cc:290] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 4 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:44.280551  3687 raft_consensus.cc:493] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:59:44.287541  3773 raft_consensus.cc:493] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:44.357723  1671 raft_consensus.cc:2468] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 4.
I20260709 12:59:44.341305  1676 raft_consensus.cc:2468] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 2.
W20260709 12:59:44.362406  1645 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:59:44.363957  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:44.377472  3765 raft_consensus.cc:3060] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:59:44.377669  1571 leader_election.cc:304] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 12:59:44.341051  3797 leader_election.cc:290] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 4 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 12:59:44.385250  1643 leader_election.cc:336] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 2.777s (SENT)
W20260709 12:59:44.385886  1646 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:59:44.393815  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:59:44.387550  1754 raft_consensus.cc:1275] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:59:44.361595  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:44.360693  3767 raft_consensus.cc:493] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:44.385138  1677 raft_consensus.cc:2468] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 3.
I20260709 12:59:44.400583  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:59:44.401327  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:44.402668  1601 raft_consensus.cc:2468] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 3.
I20260709 12:59:44.403578  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:44.394582  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:59:44.383553  3770 raft_consensus.cc:3060] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:59:44.404614  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:59:44.376770  3800 consensus_queue.cc:1059] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.020s
I20260709 12:59:44.405254  1672 raft_consensus.cc:3060] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:44.405344  1599 raft_consensus.cc:3060] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:59:44.402966  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:44.403782  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:44.407196  3529 leader_election.cc:290] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:44.407884  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 12:59:44.409322  3414 raft_consensus.cc:493] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:44.389513  3640 raft_consensus.cc:515] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:44.410183  1597 raft_consensus.cc:2393] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 4: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 12:59:44.379031  3829 consensus_queue.cc:1048] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.019s
I20260709 12:59:44.383325  3471 consensus_queue.cc:1048] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.019s
I20260709 12:59:44.390652  3687 raft_consensus.cc:3060] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:59:44.448901  3640 leader_election.cc:290] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 3 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:44.451272  2331 raft_consensus.cc:1240] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 1. Current term is 2. Ops: []
I20260709 12:59:44.410989  3543 leader_election.cc:290] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:44.376269  3761 raft_consensus.cc:515] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:44.455389  3761 leader_election.cc:290] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:44.458999  1675 raft_consensus.cc:2393] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 3: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 12:59:44.449952  3414 raft_consensus.cc:515] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:44.454349  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:59:44.399873  3466 raft_consensus.cc:2749] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:59:44.461772  1605 raft_consensus.cc:3060] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:59:44.416162  3783 consensus_meta.cc:287] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 1.243s	user 0.012s	sys 0.004s
I20260709 12:59:44.419255  3639 raft_consensus.cc:493] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:44.419885  2332 raft_consensus.cc:1240] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 4. Current term is 5. Ops: []
I20260709 12:59:44.420154  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:59:44.424062  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:44.428330  2299 leader_election.cc:304] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 12:59:44.355818  3608 raft_consensus.cc:515] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:44.473748  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:44.474274  3608 leader_election.cc:290] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:44.475194  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:44.476321  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:44.433976  3767 raft_consensus.cc:515] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 12:59:44.479084  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 534414us
W20260709 12:59:44.481551  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 53324.1us
I20260709 12:59:44.433245  1678 raft_consensus.cc:2393] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 5: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 12:59:44.433573  1754 raft_consensus.cc:1275] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:59:44.434918  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:59:44.435595  1677 raft_consensus.cc:2468] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 4.
I20260709 12:59:44.396982  3715 raft_consensus.cc:2749] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260709 12:59:44.439628  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 62052.5us
I20260709 12:59:44.486898  2299 leader_election.cc:304] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:44.488023  1676 raft_consensus.cc:3060] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:59:44.444037  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 58855us
I20260709 12:59:44.390805  3780 raft_consensus.cc:515] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:44.491694  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:59:44.396288  3773 raft_consensus.cc:515] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:44.470788  2330 raft_consensus.cc:2468] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 3.
I20260709 12:59:44.471421  2328 raft_consensus.cc:1275] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260709 12:59:44.471827  3414 leader_election.cc:290] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:44.472074  3464 consensus_queue.cc:1048] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20260709 12:59:44.490157  1643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:59:44.496775  1643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:59:44.498770  3543 consensus_queue.cc:1059] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:59:44.499327  3663 consensus_queue.cc:1059] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:59:44.499948  3543 raft_consensus.cc:3055] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:59:44.500391  3543 raft_consensus.cc:740] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 12:59:44.500860  3663 raft_consensus.cc:3055] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:59:44.501377  3543 consensus_queue.cc:260] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:44.502209  3640 consensus_queue.cc:1048] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:59:44.502959  3543 raft_consensus.cc:3060] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:59:44.501402  3663 raft_consensus.cc:740] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 12:59:44.504467  1643 leader_election.cc:304] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 12:59:44.496388  1672 raft_consensus.cc:2468] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 2.
I20260709 12:59:44.508258  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:59:44.505555  3663 consensus_queue.cc:260] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:44.509985  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:44.510191  3663 raft_consensus.cc:3060] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:59:44.510854  1672 raft_consensus.cc:2410] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 for term 6 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260709 12:59:44.511448  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 12:59:44.533739  1569 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:59:44.534101  1596 raft_consensus.cc:2468] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 2.
I20260709 12:59:44.540516  1599 raft_consensus.cc:2468] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 5.
I20260709 12:59:44.542380  2298 leader_election.cc:304] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 12:59:44.512441  1643 leader_election.cc:400] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 3 pre-election: Vote denied by peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445) with higher term. Message: Invalid argument: T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for earlier term 3. Current term is 4.
I20260709 12:59:44.546793  1643 leader_election.cc:403] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:59:44.492161  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:59:44.547755  3608 raft_consensus.cc:2749] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445) with higher term. Message: Invalid argument: T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for earlier term 3. Current term is 4.
I20260709 12:59:44.548218  1678 raft_consensus.cc:3060] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:59:44.514420  1677 raft_consensus.cc:2393] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 4: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
W20260709 12:59:44.517552  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 466309us
I20260709 12:59:44.490895  3358 raft_consensus.cc:2804] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:59:44.518829  1571 leader_election.cc:304] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 12:59:44.491292  3800 raft_consensus.cc:2749] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20260709 12:59:44.522773  1719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:59:44.480616  3767 leader_election.cc:290] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:44.482661  3639 raft_consensus.cc:515] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:44.528369  3835 raft_consensus.cc:493] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:44.530616  3773 leader_election.cc:290] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 12:59:44.531188  3725 consensus_meta.cc:287] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 1.019s	user 0.013s	sys 0.000s
I20260709 12:59:44.531773  3716 raft_consensus.cc:493] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:44.517939  2299 leader_election.cc:304] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:44.520462  3787 raft_consensus.cc:3060] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:59:44.551410  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:59:44.551367  1718 leader_election.cc:304] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 12:59:44.571516  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 53716.3us
I20260709 12:59:44.572279  2299 leader_election.cc:304] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:44.572563  1718 leader_election.cc:304] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:59:44.554255  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:59:44.561471  1643 leader_election.cc:304] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
W20260709 12:59:44.568701  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 49962us
I20260709 12:59:44.574652  1677 raft_consensus.cc:3060] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:59:44.574965  1571 leader_election.cc:304] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 12:59:44.490607  3810 consensus_queue.cc:1048] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.068s
I20260709 12:59:44.590651  1676 raft_consensus.cc:2468] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 3.
I20260709 12:59:44.597553  2299 leader_election.cc:304] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:44.558935  3358 raft_consensus.cc:493] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 12:59:44.679117  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:44.601042  2299 leader_election.cc:304] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:44.565447  3464 raft_consensus.cc:2804] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:59:44.693187  1754 raft_consensus.cc:1275] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260709 12:59:44.611691  1672 raft_consensus.cc:1275] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260709 12:59:44.598948  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:44.695338  1747 raft_consensus.cc:2410] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for term 5 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20260709 12:59:44.682590  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 81663.7us
W20260709 12:59:44.684206  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 109301us
I20260709 12:59:44.550930  3854 raft_consensus.cc:2749] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:59:44.588001  3657 raft_consensus.cc:2804] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:59:44.585289  3728 raft_consensus.cc:2749] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:59:44.591902  3471 raft_consensus.cc:493] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:44.706131  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
W20260709 12:59:44.705591  2299 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:59:44.707623  1676 raft_consensus.cc:1275] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
W20260709 12:59:44.707401  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 145983us
I20260709 12:59:44.680578  3358 raft_consensus.cc:3060] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:59:44.720991  1643 leader_election.cc:304] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:44.699985  3796 raft_consensus.cc:2804] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:59:44.597556  3767 raft_consensus.cc:2749] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:59:44.599103  3845 raft_consensus.cc:2749] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:59:44.567899  3608 consensus_queue.cc:1048] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.134s
I20260709 12:59:44.715420  1672 raft_consensus.cc:1275] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 6. (term mismatch)
I20260709 12:59:44.725690  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 12:59:44.572212  3725 raft_consensus.cc:515] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:44.584919  3773 consensus_queue.cc:1048] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.132s
I20260709 12:59:44.697995  3543 raft_consensus.cc:493] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:44.701917  3797 raft_consensus.cc:2749] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:59:44.571915  3835 raft_consensus.cc:515] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:44.716544  3657 raft_consensus.cc:493] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:59:44.709122  3464 raft_consensus.cc:697] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 5 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 12:59:44.594903  3761 raft_consensus.cc:2804] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:59:44.736543  1672 pending_rounds.cc:85] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d: Aborting all ops after (but not including) 5
I20260709 12:59:44.775866  1672 pending_rounds.cc:107] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d: Aborting uncommitted NO_OP operation due to leader change: 4.6
I20260709 12:59:44.776309  1672 raft_consensus.cc:2889] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:59:44.586673  3716 raft_consensus.cc:515] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:44.782553  3761 raft_consensus.cc:697] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 2 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 12:59:44.767014  3657 raft_consensus.cc:3060] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:59:44.766342  3725 leader_election.cc:290] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 5 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:44.747648  3414 raft_consensus.cc:2749] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:59:44.723253  3471 raft_consensus.cc:515] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:44.797421  3780 leader_election.cc:290] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:44.759400  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:44.762920  3728 raft_consensus.cc:2804] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:59:44.798681  3728 raft_consensus.cc:493] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:59:44.798775  1753 raft_consensus.cc:2468] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 2.
I20260709 12:59:44.799217  3728 raft_consensus.cc:3060] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:59:44.747711  3796 raft_consensus.cc:697] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 3 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 12:59:44.762840  3543 raft_consensus.cc:515] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:44.801054  3796 consensus_queue.cc:237] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:44.758141  1676 raft_consensus.cc:1275] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:59:44.781365  1672 raft_consensus.cc:1240] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 LEADER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 6. Current term is 7. Ops: [6.5-6.5]
I20260709 12:59:44.803156  3543 leader_election.cc:290] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:44.793363  3464 consensus_queue.cc:237] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:44.807772  3793 raft_consensus.cc:3060] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:59:44.734848  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:44.811126  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:44.816108  1753 raft_consensus.cc:2468] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 2.
W20260709 12:59:44.821442  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 100519us
I20260709 12:59:44.830574  1604 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 1747) took 1239 ms. Trace:
I20260709 12:59:44.743391  3800 consensus_queue.cc:1048] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.072s
I20260709 12:59:44.831035  1604 rpcz_store.cc:276] 0709 12:59:43.591131 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:43.591344 (+   213us) service_pool.cc:224] Handling call
0709 12:59:43.788270 (+196926us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:59:43.811956 (+ 23686us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:44.827874 (+1015918us) spinlock_profiling.cc:243] Waited 171 ms on lock 0x7b54002a1f50. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661618ad 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 12:59:44.830484 (+  2610us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":394069120}
I20260709 12:59:44.838243  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:44.838696  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:44.849787  1753 raft_consensus.cc:2468] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 3.
W20260709 12:59:44.851133  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 370572us
I20260709 12:59:44.854821  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:44.855629  1753 raft_consensus.cc:3060] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:59:44.856431  3823 raft_consensus.cc:3055] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:59:44.856884  3823 raft_consensus.cc:740] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 12:59:44.857905  3823 consensus_queue.cc:260] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:44.859230  3823 raft_consensus.cc:3060] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:59:44.859122  1717 leader_election.cc:304] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:44.837963  3857 raft_consensus.cc:493] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:44.862913  3857 raft_consensus.cc:515] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:44.865759  1753 raft_consensus.cc:2468] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 4.
I20260709 12:59:44.866520  1605 raft_consensus.cc:2468] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 4.
I20260709 12:59:44.867081  3835 leader_election.cc:290] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:44.867256  1753 raft_consensus.cc:1240] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 3. Current term is 4. Ops: []
I20260709 12:59:44.867976  3552 raft_consensus.cc:2749] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:59:44.808346  3471 leader_election.cc:290] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:44.889760  1678 raft_consensus.cc:2468] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 4.
W20260709 12:59:44.898758  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 197320us
I20260709 12:59:44.900182  1571 leader_election.cc:304] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 12:59:44.911054  3857 leader_election.cc:290] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:44.913429  3358 raft_consensus.cc:515] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:44.915099  1747 raft_consensus.cc:3060] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:59:44.948725  1570 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:59:44.899415  3543 consensus_queue.cc:1048] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.048s
I20260709 12:59:44.949250  1747 raft_consensus.cc:2468] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 4.
I20260709 12:59:44.938915  3857 raft_consensus.cc:493] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:59:44.952445  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 52424us
I20260709 12:59:44.918025  3829 consensus_queue.cc:1048] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.045s
W20260709 12:59:44.959894  3687 consensus_meta.cc:287] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.511s	user 0.011s	sys 0.002s
W20260709 12:59:44.961179  3770 consensus_meta.cc:287] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.527s	user 0.003s	sys 0.011s
I20260709 12:59:44.967608  1677 raft_consensus.cc:2468] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 4.
I20260709 12:59:44.970417  1679 raft_consensus.cc:2468] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 3.
I20260709 12:59:44.975550  1571 leader_election.cc:304] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 12:59:44.977494  1571 leader_election.cc:304] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 12:59:44.978636  3866 consensus_queue.cc:1048] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:59:44.979473  3846 raft_consensus.cc:2749] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:59:44.980300  3866 raft_consensus.cc:2749] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 12:59:44.981177  2336 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39738 (request call id 574) took 1698 ms (client timeout 2002 ms). Trace:
I20260709 12:59:44.979001  3471 raft_consensus.cc:2749] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20260709 12:59:44.981643  2336 rpcz_store.cc:269] 0709 12:59:43.282880 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:43.283014 (+   134us) service_pool.cc:224] Handling call
0709 12:59:44.980995 (+1697981us) spinlock_profiling.cc:243] Waited 777 ms on lock 0x7b54004880d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 12:59:44.981109 (+   114us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1786316160}
I20260709 12:59:44.970693  3687 raft_consensus.cc:515] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 12:59:44.987114  3858 log.cc:927] Time spent T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.159s	user 0.001s	sys 0.000s
I20260709 12:59:44.989277  1571 leader_election.cc:304] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 12:59:44.989701  2298 leader_election.cc:304] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 12:59:44.990653  1571 leader_election.cc:304] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 12:59:44.979395  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:44.953451  3862 raft_consensus.cc:493] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:44.955313  3529 raft_consensus.cc:2749] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:59:44.998261  1571 leader_election.cc:304] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 12:59:44.948662  3835 consensus_queue.cc:1048] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.026s
I20260709 12:59:44.918850  3761 consensus_queue.cc:237] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:45.023862  3529 raft_consensus.cc:2749] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:59:45.025370  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
W20260709 12:59:45.004689  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 145667us
I20260709 12:59:44.993860  3788 raft_consensus.cc:493] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 12:59:44.993333  3471 raft_consensus.cc:2749] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:59:44.993621  3817 raft_consensus.cc:493] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:45.030241  3788 raft_consensus.cc:515] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:45.030942  3817 raft_consensus.cc:515] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:44.968467  3857 raft_consensus.cc:515] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:45.012120  1675 raft_consensus.cc:1240] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 2. Current term is 3. Ops: []
I20260709 12:59:45.013530  1673 raft_consensus.cc:1240] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 1. Current term is 2. Ops: []
W20260709 12:59:45.034126  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 458287us
I20260709 12:59:45.034900  1718 leader_election.cc:304] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:59:45.006860  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:44.971022  3716 leader_election.cc:290] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:44.986447  3752 raft_consensus.cc:3055] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:59:45.007575  1746 raft_consensus.cc:3060] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:59:44.989007  3687 leader_election.cc:290] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 4 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:45.042088  1673 raft_consensus.cc:1275] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 12:59:45.003319  3846 raft_consensus.cc:2749] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:59:45.003444  3797 raft_consensus.cc:2804] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:59:45.033095  1717 leader_election.cc:304] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:45.020303  1677 raft_consensus.cc:1240] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 2. Current term is 3. Ops: []
I20260709 12:59:45.041603  3866 raft_consensus.cc:493] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 12:59:45.051154  3866 raft_consensus.cc:515] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:45.051625  3817 leader_election.cc:290] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:45.053263  3866 leader_election.cc:290] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:45.053689  3857 leader_election.cc:290] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:45.003123  3728 raft_consensus.cc:515] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:45.055817  1753 raft_consensus.cc:1240] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 6. Current term is 7. Ops: []
I20260709 12:59:45.054920  3797 raft_consensus.cc:697] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 4 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 12:59:45.054930  3788 leader_election.cc:290] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:45.057888  3829 consensus_queue.cc:1059] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.006s
I20260709 12:59:45.058614  1718 leader_election.cc:304] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:59:45.049556  3529 consensus_queue.cc:1059] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.006s
I20260709 12:59:45.039553  3752 raft_consensus.cc:740] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 12:59:45.075198  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:45.078402  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:45.079766  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:45.061818  3797 consensus_queue.cc:237] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:45.082664  3358 leader_election.cc:290] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 12:59:45.084250  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 51266.2us
I20260709 12:59:45.086694  3871 raft_consensus.cc:3060] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:59:45.090711  3716 consensus_queue.cc:1048] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:59:45.095808  3728 leader_election.cc:290] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 4 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:45.096277  1717 leader_election.cc:304] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:45.099552  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:45.100330  2330 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39738 (request call id 583) took 1069 ms. Trace:
I20260709 12:59:45.016888  3862 raft_consensus.cc:515] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:45.100759  2330 rpcz_store.cc:276] 0709 12:59:44.030826 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:44.031862 (+  1036us) service_pool.cc:224] Handling call
0709 12:59:45.100159 (+1068297us) spinlock_profiling.cc:243] Waited 341 ms on lock 0x7b54007d2654. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 12:59:45.100263 (+   104us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":785368320}
W20260709 12:59:45.103312  3663 consensus_meta.cc:287] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.566s	user 0.009s	sys 0.005s
I20260709 12:59:45.103389  3862 leader_election.cc:290] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:45.067519  3868 raft_consensus.cc:2804] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:59:45.104628  3868 raft_consensus.cc:493] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 12:59:45.105113  3868 raft_consensus.cc:3060] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:45.105131  3657 raft_consensus.cc:515] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:45.107638  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:59:45.108316  1600 raft_consensus.cc:3060] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:59:45.108584  3752 consensus_queue.cc:260] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:45.117192  1752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55840 (request call id 1992) took 1198 ms. Trace:
I20260709 12:59:45.123322  1752 rpcz_store.cc:276] 0709 12:59:43.918825 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:43.925138 (+  6313us) service_pool.cc:224] Handling call
0709 12:59:45.116862 (+1191724us) spinlock_profiling.cc:243] Waited 143 ms on lock 0x7b54001d42d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 12:59:45.117065 (+   203us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":328591744}
I20260709 12:59:45.130962  1717 leader_election.cc:304] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:45.115720  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 12:59:45.136025  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 77502.1us
I20260709 12:59:45.137404  1718 leader_election.cc:304] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 12:59:45.133101  1745 raft_consensus.cc:2468] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 3.
I20260709 12:59:45.144714  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:59:45.146615  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:45.151473  1717 leader_election.cc:304] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:45.110412  3752 raft_consensus.cc:3060] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:59:45.150768  1718 leader_election.cc:304] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:45.153002  3866 raft_consensus.cc:2749] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:59:45.144830  3358 raft_consensus.cc:2804] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:59:45.157160  1605 raft_consensus.cc:3060] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:59:45.108317  3657 leader_election.cc:290] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 5 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:45.152994  3753 raft_consensus.cc:3060] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:59:45.164793  3816 raft_consensus.cc:3055] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 LEADER]: Stepping down as leader of term 1
W20260709 12:59:45.168412  3843 log.cc:927] Time spent T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.938s	user 0.006s	sys 0.000s
I20260709 12:59:45.189145  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:45.194048  1745 raft_consensus.cc:2468] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 5.
W20260709 12:59:45.195978  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 45314.3us
I20260709 12:59:45.189363  1600 raft_consensus.cc:2468] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 4.
I20260709 12:59:45.197070  1718 leader_election.cc:304] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:59:45.211342  3805 raft_consensus.cc:3055] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:59:45.211912  3805 raft_consensus.cc:740] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 12:59:45.213109  3805 consensus_queue.cc:260] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:45.215484  3805 raft_consensus.cc:3060] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:59:45.216396  2335 raft_consensus.cc:2468] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 4.
W20260709 12:59:45.217212  2300 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:59:45.217706  2335 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39738 (request call id 579) took 1839 ms (client timeout 1808 ms). Trace:
W20260709 12:59:45.218027  3787 consensus_meta.cc:287] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.647s	user 0.013s	sys 0.006s
I20260709 12:59:45.218588  3787 raft_consensus.cc:2749] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:59:45.219089  3800 raft_consensus.cc:493] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:45.219633  3800 raft_consensus.cc:515] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:45.220062  3871 raft_consensus.cc:2749] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 12:59:45.218156  2335 rpcz_store.cc:269] 0709 12:59:43.378100 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:43.440896 (+ 62796us) service_pool.cc:224] Handling call
0709 12:59:45.217507 (+1776611us) spinlock_profiling.cc:243] Waited 1.76 s on lock 0x7b540045bfd4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 12:59:45.217623 (+   116us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4049457920}
I20260709 12:59:45.223776  2334 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39738 (request call id 581) took 1766 ms. Trace:
I20260709 12:59:45.224174  2334 rpcz_store.cc:276] 0709 12:59:43.457727 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:43.472984 (+ 15257us) service_pool.cc:224] Handling call
0709 12:59:43.473565 (+   581us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:59:43.738333 (+264768us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:45.223505 (+1485172us) spinlock_profiling.cc:243] Waited 264 ms on lock 0x7b540045b5d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 12:59:45.223706 (+   201us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":607456512}
W20260709 12:59:45.225360  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 73985.9us
I20260709 12:59:45.226122  1717 leader_election.cc:304] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 12:59:45.229132  2298 leader_election.cc:304] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 12:59:45.232685  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:45.234778  1754 raft_consensus.cc:1275] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 12:59:45.189868  3358 raft_consensus.cc:493] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 12:59:45.237699  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:45.241739  1646 leader_election.cc:304] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:59:45.191605  3816 raft_consensus.cc:740] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
W20260709 12:59:45.242842  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 320848us
I20260709 12:59:45.244331  1605 raft_consensus.cc:2468] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 4.
I20260709 12:59:45.244537  1571 leader_election.cc:304] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 12:59:45.250319  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:45.250355  1646 leader_election.cc:304] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:59:45.258240  1646 leader_election.cc:304] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:59:45.251161  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:45.205114  3866 raft_consensus.cc:2804] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:59:45.267637  1603 raft_consensus.cc:1275] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:59:45.269749  1604 raft_consensus.cc:3055] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:59:45.271438  1604 raft_consensus.cc:740] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 12:59:45.242192  3358 raft_consensus.cc:3060] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:59:45.246757  3871 raft_consensus.cc:2804] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260709 12:59:45.324312  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 79848.5us
I20260709 12:59:45.326119  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:45.328553  1600 raft_consensus.cc:2468] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 3.
I20260709 12:59:45.331660  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:45.248440  3787 raft_consensus.cc:2804] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:59:45.236357  3810 raft_consensus.cc:2749] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:59:45.277674  3728 raft_consensus.cc:2804] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:59:45.245534  3854 raft_consensus.cc:493] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:45.304404  1646 leader_election.cc:304] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:59:45.334626  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:45.250428  3706 raft_consensus.cc:493] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 12:59:45.323946  1604 consensus_queue.cc:260] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:45.266999  3857 raft_consensus.cc:2804] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:59:45.240590  3663 consensus_queue.cc:1048] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.086s
I20260709 12:59:45.325405  3608 raft_consensus.cc:2804] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:59:45.273424  3543 raft_consensus.cc:2804] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:59:45.325695  3866 raft_consensus.cc:697] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 2 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 12:59:45.248358  3471 consensus_queue.cc:1059] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.094s
W20260709 12:59:45.354701  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 128665us
I20260709 12:59:45.405526  1717 leader_election.cc:304] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 12:59:45.326109  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 57844.3us
I20260709 12:59:45.273910  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
W20260709 12:59:45.378804  2333 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39738 (request call id 572) took 2138 ms (client timeout 2186 ms). Trace:
I20260709 12:59:45.327420  3871 raft_consensus.cc:493] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 12:59:45.370419  3857 raft_consensus.cc:493] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:59:45.341012  3846 raft_consensus.cc:2749] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:59:45.341001  3728 raft_consensus.cc:493] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:59:45.337683  3859 consensus_queue.cc:1048] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.015s
I20260709 12:59:45.398027  3608 raft_consensus.cc:493] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 12:59:45.423058  3728 raft_consensus.cc:3060] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:59:45.423377  3608 raft_consensus.cc:3060] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:59:45.379673  3773 consensus_queue.cc:1048] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.064s
I20260709 12:59:45.407565  1643 leader_election.cc:304] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 12:59:45.402380  3765 raft_consensus.cc:2804] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:59:45.431354  3765 raft_consensus.cc:493] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:59:45.431797  3765 raft_consensus.cc:3060] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:59:45.405627  3716 raft_consensus.cc:493] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 12:59:45.432411  1643 leader_election.cc:304] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 12:59:45.432713  3716 raft_consensus.cc:515] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 12:59:45.411226  2333 rpcz_store.cc:269] 0709 12:59:43.214070 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:43.214195 (+   125us) service_pool.cc:224] Handling call
0709 12:59:45.352027 (+2137832us) spinlock_profiling.cc:243] Waited 771 ms on lock 0x7b54007602d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 12:59:45.352146 (+   119us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1774004608}
I20260709 12:59:45.434999  3716 leader_election.cc:290] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:45.436533  1605 raft_consensus.cc:1240] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260709 12:59:45.437489  1676 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52766 (request call id 2015) took 1021 ms. Trace:
I20260709 12:59:45.438521  1751 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55848 (request call id 1756) took 1692 ms. Trace:
I20260709 12:59:45.415848  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:45.438967  1751 rpcz_store.cc:276] 0709 12:59:43.745833 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:43.750655 (+  4822us) service_pool.cc:224] Handling call
0709 12:59:43.765377 (+ 14722us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:59:45.112273 (+1346896us) spinlock_profiling.cc:243] Waited 68.2 ms on lock 0x7b1800026ec0. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f4265d0a4bc 00007f426305848b 00007f426beb2ab9 00007f426307469b 00007f426307293f 00007f4263072455 00007f426615a060 00007f426615abfb 00007f426616564b
0709 12:59:45.112451 (+   178us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:59:45.112490 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:59:45.113164 (+   674us) log.cc:844] Serialized 26 byte log entry
0709 12:59:45.246923 (+133759us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:59:45.246978 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:45.247092 (+   114us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:59:45.417535 (+170443us) raft_consensus.cc:1685] finished
0709 12:59:45.417560 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:59:45.438446 (+ 20886us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":99256,"raft.run_cpu_time_us":201,"raft.run_wall_time_us":8105,"spinlock_wait_cycles":156907264,"thread_start_us":242634,"threads_started":2,"wal-append.queue_time_us":296123}
I20260709 12:59:45.399276  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:45.442099  3866 consensus_queue.cc:237] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } }
I20260709 12:59:45.353509  3787 raft_consensus.cc:697] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 4 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 12:59:45.383721  3543 raft_consensus.cc:493] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:59:45.353926  3706 raft_consensus.cc:515] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:45.443868  1600 raft_consensus.cc:3055] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:59:45.456290  1600 raft_consensus.cc:740] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 12:59:45.438055  1605 raft_consensus.cc:3055] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:59:45.456914  1605 raft_consensus.cc:740] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 12:59:45.448778  2298 leader_election.cc:304] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 12:59:45.437873  1676 rpcz_store.cc:276] 0709 12:59:44.416250 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:44.803352 (+387102us) service_pool.cc:224] Handling call
0709 12:59:44.803888 (+   536us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:59:44.812676 (+  8788us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:59:44.812728 (+    52us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:59:44.831577 (+ 18849us) log.cc:844] Serialized 26 byte log entry
0709 12:59:45.072628 (+241051us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:59:45.072670 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:45.072721 (+    51us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:59:45.437234 (+364513us) raft_consensus.cc:1685] finished
0709 12:59:45.437265 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:59:45.437422 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8215,"raft.queue_time_us":8985,"raft.run_cpu_time_us":230,"raft.run_wall_time_us":14227,"spinlock_wait_cycles":29440,"thread_start_us":176349,"threads_started":1,"wal-append.queue_time_us":405555}
I20260709 12:59:45.369980  3854 raft_consensus.cc:515] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:45.415012  1717 leader_election.cc:304] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 12:59:45.419114  3871 raft_consensus.cc:3060] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:59:45.444960  3787 consensus_queue.cc:237] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:45.459929  1605 consensus_queue.cc:260] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:45.455992  3543 raft_consensus.cc:3060] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:59:45.415902  3886 raft_consensus.cc:2749] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:59:45.457499  1746 raft_consensus.cc:2468] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 5.
I20260709 12:59:45.420672  3857 raft_consensus.cc:3060] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:59:45.466619  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 51685us
I20260709 12:59:45.468286  3862 raft_consensus.cc:2804] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:59:45.482226  1750 raft_consensus.cc:1240] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 3. Current term is 4. Ops: []
I20260709 12:59:45.465903  3793 raft_consensus.cc:2804] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:59:45.473701  1600 consensus_queue.cc:260] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 12:59:45.478299  1570 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:59:45.499543  1600 raft_consensus.cc:3060] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:59:45.499796  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 195445us
W20260709 12:59:45.478597  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 281580us
I20260709 12:59:45.479171  1605 raft_consensus.cc:3060] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:59:45.480959  1754 raft_consensus.cc:1275] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260709 12:59:45.501012  1646 leader_election.cc:304] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:59:45.519579  1600 raft_consensus.cc:1275] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
W20260709 12:59:45.501986  1717 connection.cc:256] RPC call timeout handler was delayed by -1.1565516219999949s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1553s for a response
I20260709 12:59:45.523538  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:45.524849  1600 raft_consensus.cc:2393] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 3: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 12:59:45.536691  1605 raft_consensus.cc:1275] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 5. (term mismatch)
I20260709 12:59:45.538890  2299 leader_election.cc:304] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:45.580250  2299 leader_election.cc:304] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 12:59:45.554594  3773 raft_consensus.cc:2749] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:59:45.482337  3862 raft_consensus.cc:493] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 12:59:45.571561  3888 raft_consensus.cc:493] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 12:59:45.582968  3862 raft_consensus.cc:3060] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:59:45.583415  3888 raft_consensus.cc:515] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:45.583882  3715 raft_consensus.cc:1027] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 4 LEADER]: attempt to promote peer 3b879799899a499c8d409ee9f1277cd1: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20260709 12:59:45.585919  3888 leader_election.cc:290] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:45.509546  3521 raft_consensus.cc:2749] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:59:45.586717  3811 raft_consensus.cc:2804] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:59:45.587208  3761 raft_consensus.cc:2804] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:59:45.587325  3811 raft_consensus.cc:697] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 4 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 12:59:45.587858  3761 raft_consensus.cc:697] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 4 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 12:59:45.497555  1746 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55840 (request call id 2004) took 1343 ms. Trace:
I20260709 12:59:45.588490  3811 consensus_queue.cc:237] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:45.588904  3761 consensus_queue.cc:237] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:45.589980  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:45.405050  3816 consensus_queue.cc:260] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } }
I20260709 12:59:45.539541  1718 leader_election.cc:304] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 12:59:45.593185  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 53760.5us
I20260709 12:59:45.594583  1718 leader_election.cc:304] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:45.542361  3823 consensus_queue.cc:1048] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.004s
I20260709 12:59:45.546217  3846 consensus_queue.cc:1048] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:59:45.546964  3471 consensus_queue.cc:1048] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:59:45.534008  3706 leader_election.cc:290] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:45.522248  3793 raft_consensus.cc:697] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 4 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
W20260709 12:59:45.566445  1570 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:59:45.543943  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:59:45.582353  3854 leader_election.cc:290] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:45.596889  1605 pending_rounds.cc:85] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a: Aborting all ops after (but not including) 4
I20260709 12:59:45.606541  1602 raft_consensus.cc:2468] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 4.
I20260709 12:59:45.607996  1605 pending_rounds.cc:107] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a: Aborting uncommitted NO_OP operation due to leader change: 6.5
I20260709 12:59:45.608557  1605 raft_consensus.cc:2889] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:59:45.614660  1571 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:59:45.637708  1717 leader_election.cc:336] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 1.553s (SENT)
I20260709 12:59:45.638551  1717 leader_election.cc:304] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:45.544265  3796 consensus_queue.cc:1048] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.034s
W20260709 12:59:45.677817  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 229817us
I20260709 12:59:45.684298  1643 leader_election.cc:304] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 12:59:45.606210  3835 raft_consensus.cc:3060] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:59:45.588449  1746 rpcz_store.cc:276] 0709 12:59:44.153839 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:44.181960 (+ 28121us) service_pool.cc:224] Handling call
0709 12:59:44.979216 (+797256us) spinlock_profiling.cc:243] Waited 3.26 ms on lock 0x7b58001ffe98. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f426303db71 00007f426303da22 00007f426bf56c09 00007f426b0b0ee0 00007f426b17a0ab 00007f426b17a131 00007f426b1280a0 00007f426b127984 00007f42646a26d4 00007f42646a2662
0709 12:59:45.483683 (+504467us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":7499008}
I20260709 12:59:45.671239  3888 raft_consensus.cc:2804] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:59:45.712944  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:45.713894  1602 raft_consensus.cc:2468] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 3.
I20260709 12:59:45.596289  3805 consensus_queue.cc:1048] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.101s
W20260709 12:59:45.705561  3753 consensus_meta.cc:287] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.507s	user 0.012s	sys 0.000s
I20260709 12:59:45.688097  3854 raft_consensus.cc:2749] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:59:45.706630  3888 raft_consensus.cc:493] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
W20260709 12:59:45.716215  3883 log.cc:927] Time spent T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.168s	user 0.001s	sys 0.000s
I20260709 12:59:45.674535  3859 consensus_queue.cc:1059] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.096s
I20260709 12:59:45.732333  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 12:59:45.733990  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 233051us
I20260709 12:59:45.710804  1597 raft_consensus.cc:2368] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 3. Current term is 4.
I20260709 12:59:45.738250  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:45.691329  3716 raft_consensus.cc:2749] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:59:45.740952  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:59:45.676816  3892 consensus_queue.cc:1048] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.113s
I20260709 12:59:45.750684  1680 raft_consensus.cc:3060] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:59:45.752627  1671 raft_consensus.cc:1275] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260709 12:59:45.737426  3753 raft_consensus.cc:2749] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:59:45.745579  3888 raft_consensus.cc:3060] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:59:45.765949  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:59:45.769970  1602 raft_consensus.cc:2410] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260709 12:59:45.779645  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 12:59:45.782554  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 220723us
W20260709 12:59:45.763895  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 177001us
I20260709 12:59:45.770426  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:45.730340  3728 raft_consensus.cc:515] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:45.793236  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:45.794121  2335 raft_consensus.cc:2468] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 5.
I20260709 12:59:45.795221  2337 raft_consensus.cc:1275] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260709 12:59:45.792122  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:45.802626  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 12:59:45.803181  3868 consensus_meta.cc:287] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.697s	user 0.015s	sys 0.000s
I20260709 12:59:45.803303  2336 raft_consensus.cc:2468] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 3.
I20260709 12:59:45.803709  3868 raft_consensus.cc:515] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:45.804337  3639 leader_election.cc:290] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 12:59:45.806581  3890 log.cc:927] Time spent T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.215s	user 0.001s	sys 0.000s
I20260709 12:59:45.781857  3796 raft_consensus.cc:493] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 12:59:45.811434  1750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55840 (request call id 2013) took 1613 ms. Trace:
I20260709 12:59:45.813869  1680 raft_consensus.cc:2468] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 5.
I20260709 12:59:45.790084  3854 raft_consensus.cc:2804] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:59:45.779399  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:45.819419  1750 rpcz_store.cc:276] 0709 12:59:44.197704 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:44.325623 (+127919us) service_pool.cc:224] Handling call
0709 12:59:45.417832 (+1092209us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:59:45.496712 (+ 78880us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:45.810065 (+313353us) spinlock_profiling.cc:243] Waited 64.1 ms on lock 0x7b540006fc54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 12:59:45.811214 (+  1149us) spinlock_profiling.cc:243] Waited 1.09 s on lock 0x7b540006fc50. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661618ad 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 12:59:45.811356 (+   142us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2658126336}
I20260709 12:59:45.752316  3898 consensus_queue.cc:1059] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.065s
I20260709 12:59:45.820363  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:45.841101  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:45.825045  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:45.821343  3834 consensus_queue.cc:1059] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:59:45.917244  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:45.916303  3466 consensus_queue.cc:1059] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:59:46.066983  1500 catalog_manager.cc:5697] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "af520b67945d443b901bbeca37cc5701" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:46.067798  1598 raft_consensus.cc:2376] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 5. Re-sending same reply.
I20260709 12:59:45.834932  3796 raft_consensus.cc:515] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:46.061946  3716 raft_consensus.cc:493] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 12:59:46.069324  3716 raft_consensus.cc:515] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:46.070482  2334 raft_consensus.cc:2468] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 3.
W20260709 12:59:46.070237  1598 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40154 (request call id 1701) took 1828 ms (client timeout 1553 ms). Trace:
W20260709 12:59:46.071252  1598 rpcz_store.cc:269] 0709 12:59:44.241372 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:44.297200 (+ 55828us) service_pool.cc:224] Handling call
0709 12:59:46.070018 (+1772818us) spinlock_profiling.cc:243] Waited 1.68 s on lock 0x7b54002990d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 12:59:46.070155 (+   137us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3863807872}
I20260709 12:59:46.071785  3716 leader_election.cc:290] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:46.072268  3796 leader_election.cc:290] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 12:59:46.004194  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 75744.1us
I20260709 12:59:45.832161  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:46.074360  1598 raft_consensus.cc:1240] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260709 12:59:46.074749  1597 raft_consensus.cc:1240] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 4. Current term is 5. Ops: [4.4-4.4]
I20260709 12:59:46.076237  1597 raft_consensus.cc:1240] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 3. Current term is 4. Ops: []
W20260709 12:59:45.878731  3885 log.cc:927] Time spent T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.167s	user 0.001s	sys 0.001s
I20260709 12:59:45.822247  3854 raft_consensus.cc:493] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:59:45.871244  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:46.091105  1750 raft_consensus.cc:3060] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:59:46.004242  1745 raft_consensus.cc:2468] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 2.
I20260709 12:59:46.095732  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:59:46.096553  1671 raft_consensus.cc:3060] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:59:46.097160  1677 raft_consensus.cc:1275] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
W20260709 12:59:46.006075  3884 log.cc:927] Time spent T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.441s	user 0.000s	sys 0.001s
I20260709 12:59:45.759930  3793 consensus_queue.cc:237] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:45.918438  3765 raft_consensus.cc:515] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:46.010965  3471 consensus_queue.cc:1048] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:59:45.810984  3899 raft_consensus.cc:493] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:45.821386  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 12:59:46.013695  3887 log.cc:927] Time spent T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.330s	user 0.000s	sys 0.001s
I20260709 12:59:45.894452  3639 raft_consensus.cc:493] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 12:59:46.012567  3529 raft_consensus.cc:493] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:45.823839  3896 raft_consensus.cc:2749] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20260709 12:59:46.015846  3867 log.cc:927] Time spent T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.937s	user 0.005s	sys 0.000s
I20260709 12:59:46.033092  3728 leader_election.cc:290] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 4 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 12:59:46.014088  3857 consensus_meta.cc:287] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.511s	user 0.004s	sys 0.009s
W20260709 12:59:46.006203  3903 log.cc:927] Time spent T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.188s	user 0.001s	sys 0.000s
W20260709 12:59:46.088608  3543 consensus_meta.cc:287] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.614s	user 0.013s	sys 0.000s
I20260709 12:59:46.111941  1675 raft_consensus.cc:1275] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 6. (term mismatch)
I20260709 12:59:46.111594  3901 consensus_queue.cc:1059] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.029s
I20260709 12:59:46.112089  1745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55848 (request call id 1776) took 1014 ms. Trace:
I20260709 12:59:46.169625  1745 rpcz_store.cc:276] 0709 12:59:45.098170 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:46.111029 (+1012859us) service_pool.cc:224] Handling call
0709 12:59:46.111430 (+   401us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:59:46.111685 (+   255us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:59:46.111716 (+    31us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:59:46.111765 (+    49us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:59:46.111802 (+    37us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:46.111862 (+    60us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:59:46.112017 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:59:46.112777  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:46.115067  1673 raft_consensus.cc:1240] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 1. Current term is 2. Ops: []
I20260709 12:59:46.121233  1677 raft_consensus.cc:1240] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 LEADER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 2. Current term is 3. Ops: []
I20260709 12:59:46.126143  1748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55848 (request call id 1773) took 1031 ms. Trace:
I20260709 12:59:46.127657  1680 raft_consensus.cc:1275] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:59:46.058596  3868 leader_election.cc:290] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 2 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:46.135522  1753 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55840 (request call id 2018) took 1072 ms. Trace:
I20260709 12:59:46.138902  1751 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55848 (request call id 1772) took 1044 ms. Trace:
W20260709 12:59:46.114815  3358 consensus_meta.cc:287] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.773s	user 0.013s	sys 0.001s
W20260709 12:59:46.114428  3845 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 7 current term: 7 is higher.
I20260709 12:59:46.114967  3854 raft_consensus.cc:3060] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:59:46.149072  1746 raft_consensus.cc:2393] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 4: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 12:59:46.172135  1599 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 346) took 1664 ms. Trace:
I20260709 12:59:46.185657  1671 raft_consensus.cc:2410] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for term 5 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260709 12:59:46.185263  1753 rpcz_store.cc:276] 0709 12:59:45.062966 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:45.080638 (+ 17672us) service_pool.cc:224] Handling call
0709 12:59:45.081023 (+   385us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:59:45.153255 (+ 72232us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:59:45.153313 (+    58us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:59:45.154007 (+   694us) log.cc:844] Serialized 26 byte log entry
0709 12:59:45.352246 (+198239us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:59:45.352298 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:45.808403 (+456105us) spinlock_profiling.cc:243] Waited 45.9 ms on lock 0x7b54001c7354. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 12:59:45.808458 (+    55us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:59:46.115403 (+306945us) raft_consensus.cc:1685] finished
0709 12:59:46.115436 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:59:46.135413 (+ 19977us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":52912,"raft.queue_time_us":26825,"raft.run_cpu_time_us":187,"raft.run_wall_time_us":186,"spinlock_wait_cycles":105477504,"thread_start_us":153013,"threads_started":1,"wal-append.queue_time_us":318449}
I20260709 12:59:46.183861  1748 rpcz_store.cc:276] 0709 12:59:45.094686 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:45.554254 (+459568us) service_pool.cc:224] Handling call
0709 12:59:45.568070 (+ 13816us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:59:45.568601 (+   531us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:59:45.568632 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:59:45.601390 (+ 32758us) log.cc:844] Serialized 26 byte log entry
0709 12:59:45.781390 (+180000us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:59:45.781633 (+   243us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:45.781740 (+   107us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:59:46.125868 (+344128us) raft_consensus.cc:1685] finished
0709 12:59:46.125899 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:59:46.126063 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":124,"raft.queue_time_us":129,"raft.run_cpu_time_us":166,"raft.run_wall_time_us":18568,"thread_start_us":90474,"threads_started":1,"wal-append.queue_time_us":420571}
I20260709 12:59:46.183187  1751 rpcz_store.cc:276] 0709 12:59:45.094390 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:45.440818 (+346428us) service_pool.cc:224] Handling call
0709 12:59:45.441153 (+   335us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:59:45.462977 (+ 21824us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:59:45.463014 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:59:45.476144 (+ 13130us) log.cc:844] Serialized 26 byte log entry
0709 12:59:45.760879 (+284735us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:59:45.760934 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:45.761043 (+   109us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:59:46.138597 (+377554us) raft_consensus.cc:1685] finished
0709 12:59:46.138628 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:59:46.138815 (+   187us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":15385,"raft.queue_time_us":21913,"raft.run_cpu_time_us":136,"raft.run_wall_time_us":2509,"thread_start_us":239210,"threads_started":1,"wal-append.queue_time_us":387689}
I20260709 12:59:46.181082  1675 pending_rounds.cc:85] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d: Aborting all ops after (but not including) 5
I20260709 12:59:46.206526  1675 pending_rounds.cc:107] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d: Aborting uncommitted NO_OP operation due to leader change: 4.6
I20260709 12:59:46.206723  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:46.206988  1675 raft_consensus.cc:2889] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:59:46.208334  1673 raft_consensus.cc:1240] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 3. Current term is 4. Ops: []
I20260709 12:59:46.152060  2330 raft_consensus.cc:2468] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 3.
I20260709 12:59:46.196583  3800 leader_election.cc:290] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:46.198240  1599 rpcz_store.cc:276] 0709 12:59:44.507792 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:44.541515 (+ 33723us) service_pool.cc:224] Handling call
0709 12:59:44.541997 (+   482us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:59:44.584680 (+ 42683us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:59:44.584717 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:59:44.684422 (+ 99705us) log.cc:844] Serialized 26 byte log entry
0709 12:59:44.954001 (+269579us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:59:44.954113 (+   112us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:44.954251 (+   138us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:59:46.014967 (+1060716us) spinlock_profiling.cc:243] Waited 28.1 ms on lock 0x7b4400022b4c. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f4262f9bc85 00007f4262f9c6a2 00007f4265d0e1db 00007f4265d09aaf 00007f4265cfd86e 00007f4265cfcbb6 00007f4265cfc7d8 00007f4265d06221 00007f4265d061d9 00007f4265d06169
0709 12:59:46.156922 (+141955us) raft_consensus.cc:1685] finished
0709 12:59:46.156963 (+    41us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:59:46.172035 (+ 15072us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":224112,"raft.queue_time_us":37251,"raft.run_cpu_time_us":207,"raft.run_wall_time_us":84313,"spinlock_wait_cycles":64739712,"thread_start_us":225900,"threads_started":1,"wal-append.queue_time_us":358708}
I20260709 12:59:46.167857  3892 raft_consensus.cc:493] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
W20260709 12:59:46.214947  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 465938us
I20260709 12:59:46.189893  3358 raft_consensus.cc:515] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:46.252722  3909 raft_consensus.cc:493] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
W20260709 12:59:46.253281  3862 consensus_meta.cc:287] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.670s	user 0.013s	sys 0.000s
I20260709 12:59:46.253475  3909 raft_consensus.cc:515] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:46.254125  3358 leader_election.cc:290] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:46.143357  3899 raft_consensus.cc:515] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } }
I20260709 12:59:46.156227  3639 raft_consensus.cc:515] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:46.219638  1750 raft_consensus.cc:2468] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 4.
I20260709 12:59:46.150786  3529 raft_consensus.cc:515] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 12:59:46.214005  3752 consensus_meta.cc:287] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 1.025s	user 0.015s	sys 0.001s
I20260709 12:59:46.246452  1671 raft_consensus.cc:1240] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 4. Current term is 5. Ops: [4.7-4.7]
I20260709 12:59:46.253813  3862 raft_consensus.cc:515] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:46.181221  3543 raft_consensus.cc:515] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:46.281392  3543 leader_election.cc:290] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 4 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:46.282848  1676 raft_consensus.cc:2393] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 4: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 12:59:46.284464  3529 leader_election.cc:290] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:46.254670  1747 raft_consensus.cc:1240] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 2. Current term is 3. Ops: []
W20260709 12:59:46.215628  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 402767us
I20260709 12:59:46.213161  3657 raft_consensus.cc:493] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:59:46.214087  3770 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.810s	user 0.015s	sys 0.033s
I20260709 12:59:46.256420  1753 raft_consensus.cc:1275] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 12:59:46.294538  1753 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55840 (request call id 2027) took 1055 ms. Trace:
I20260709 12:59:46.295019  1753 rpcz_store.cc:276] 0709 12:59:45.238634 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:46.245701 (+1007067us) service_pool.cc:224] Handling call
0709 12:59:46.256009 (+ 10308us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:59:46.294129 (+ 38120us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:46.294450 (+   321us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:59:46.296563  1753 raft_consensus.cc:1240] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 3. Current term is 4. Ops: []
I20260709 12:59:46.296314  3657 raft_consensus.cc:515] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:46.171836  3857 raft_consensus.cc:515] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:46.180961  3904 consensus_queue.cc:1048] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.078s
I20260709 12:59:46.265625  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:46.306336  1750 raft_consensus.cc:2468] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 5.
I20260709 12:59:46.273622  3639 leader_election.cc:290] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:46.274235  3909 leader_election.cc:290] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:46.276086  3899 leader_election.cc:290] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:46.278148  3862 leader_election.cc:290] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:46.278973  1679 raft_consensus.cc:2410] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for term 5 because replica has last-logged OpId of term: 2 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260709 12:59:46.220616  3892 raft_consensus.cc:515] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:46.301144  2301 leader_election.cc:304] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
W20260709 12:59:46.271450  3608 consensus_meta.cc:287] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.847s	user 0.008s	sys 0.007s
W20260709 12:59:46.294059  3770 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.892s	user 0.015s	sys 0.033s
I20260709 12:59:46.305747  3857 leader_election.cc:290] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 4 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:46.324424  2301 leader_election.cc:304] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:59:46.321431  3639 raft_consensus.cc:493] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
W20260709 12:59:46.334455  3871 consensus_meta.cc:287] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.851s	user 0.011s	sys 0.004s
I20260709 12:59:46.328317  3800 raft_consensus.cc:2749] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:59:46.341181  3871 raft_consensus.cc:515] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:46.337738  3639 raft_consensus.cc:515] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:46.338990  3765 leader_election.cc:290] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 4 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:46.333843  3857 raft_consensus.cc:493] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 12:59:46.341861  3796 raft_consensus.cc:2804] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:59:46.327498  3608 raft_consensus.cc:515] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 12:59:46.370962  3835 consensus_meta.cc:287] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.664s	user 0.007s	sys 0.007s
I20260709 12:59:46.371860  3835 raft_consensus.cc:2749] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 12:59:46.372212  3902 log.cc:927] Time spent T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.569s	user 0.007s	sys 0.007s
I20260709 12:59:46.373728  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:46.378463  3910 raft_consensus.cc:493] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:46.380635  3910 raft_consensus.cc:515] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:46.384699  3608 leader_election.cc:290] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 4 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:46.386113  3871 leader_election.cc:290] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 4 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:46.386703  1671 raft_consensus.cc:1240] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 3. Current term is 4. Ops: []
I20260709 12:59:46.360620  3657 leader_election.cc:290] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:46.362879  3857 raft_consensus.cc:515] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:46.361622  3796 raft_consensus.cc:493] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:59:46.401605  3910 leader_election.cc:290] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:46.401731  3892 leader_election.cc:290] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:46.403651  1672 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52766 (request call id 2016) took 1987 ms. Trace:
I20260709 12:59:46.403790  3796 raft_consensus.cc:3060] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:59:46.412988  1500 catalog_manager.cc:5697] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 reported cstate change: term changed from 1 to 3, leader changed from af520b67945d443b901bbeca37cc5701 (127.0.107.195) to 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196). New cstate: current_term: 3 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:59:46.409569  1501 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35458 (request call id 304) took 8204 ms. Trace:
I20260709 12:59:46.412616  3639 leader_election.cc:290] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 12:59:46.396481  3883 log.cc:927] Time spent T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.638s	user 0.010s	sys 0.000s
I20260709 12:59:46.416499  1672 rpcz_store.cc:276] 0709 12:59:44.416547 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:44.803371 (+386824us) service_pool.cc:224] Handling call
0709 12:59:45.099595 (+296224us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:59:45.100069 (+   474us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:59:45.100106 (+    37us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:59:45.100774 (+   668us) log.cc:844] Serialized 52 byte log entry
0709 12:59:45.253052 (+152278us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:59:45.485541 (+232489us) log.cc:844] Serialized 14 byte log entry
0709 12:59:45.548853 (+ 63312us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:45.549266 (+   413us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:59:45.582244 (+ 32978us) raft_consensus.cc:1685] finished
0709 12:59:45.582270 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:59:46.403377 (+821107us) spinlock_profiling.cc:243] Waited 296 ms on lock 0x7b5400163850. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661618ad 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 12:59:46.403557 (+   180us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":34334,"prepare.queue_time_us":318733,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":54,"raft.queue_time_us":268373,"raft.run_cpu_time_us":399,"raft.run_wall_time_us":22959,"spinlock_wait_cycles":680119296,"thread_start_us":372461,"threads_started":3,"wal-append.queue_time_us":345132}
I20260709 12:59:46.419656  1500 catalog_manager.cc:5697] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:59:46.431728  2337 raft_consensus.cc:1275] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:59:46.430617  1500 catalog_manager.cc:5697] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:59:46.421311  1501 rpcz_store.cc:276] 0709 12:59:38.205323 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:38.205527 (+   204us) service_pool.cc:224] Handling call
0709 12:59:39.558499 (+1352972us) spinlock_profiling.cc:243] Waited 775 ms on lock 0x7b540041cb54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cc090 00007f426616af3a 00007f426bf211cc 00007f426c01f37e 00007f426a4e02a4 00007f426a4e0232 00007f426a4e0161 00007f426a4e00dc 00007f426a4df392 00007f426449d48c 00007f426449c8c6
0709 12:59:46.409458 (+6850959us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:59:42.812206 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:59:42.833408 (+ 21202us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:59:42.833467 (+    59us) write_op.cc:435] Acquired schema lock
0709 12:59:42.833489 (+    22us) tablet.cc:662] Decoding operations
0709 12:59:42.834712 (+  1223us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:59:42.834801 (+    89us) write_op.cc:641] Partition lock acquired for write op
0709 12:59:42.834832 (+    31us) tablet.cc:685] Acquiring locks for 11 operations
0709 12:59:42.879644 (+ 44812us) tablet.cc:701] Row locks acquired
0709 12:59:42.879701 (+    57us) write_op.cc:260] PREPARE: finished
0709 12:59:42.879918 (+   217us) write_op.cc:270] Start()
0709 12:59:42.880086 (+   168us) write_op.cc:276] Timestamp: P: 1783601982879847 usec, L: 0
0709 12:59:42.880109 (+    23us) op_driver.cc:348] REPLICATION: starting
0709 12:59:42.881142 (+  1033us) log.cc:844] Serialized 4445 byte log entry
0709 12:59:43.510869 (+629727us) op_driver.cc:464] REPLICATION: finished
0709 12:59:43.662933 (+152064us) write_op.cc:301] APPLY: starting
0709 12:59:43.663058 (+   125us) tablet.cc:1367] starting BulkCheckPresence
0709 12:59:43.663366 (+   308us) tablet.cc:1370] finished BulkCheckPresence
0709 12:59:43.663501 (+   135us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:59:43.665055 (+  1554us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:59:43.665317 (+   262us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=11
0709 12:59:43.665362 (+    45us) write_op.cc:312] APPLY: finished
0709 12:59:43.665684 (+   322us) log.cc:844] Serialized 83 byte log entry
0709 12:59:43.665890 (+   206us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:59:43.666032 (+   142us) write_op.cc:454] Released schema lock
0709 12:59:43.666089 (+    57us) write_op.cc:341] FINISH: Updating metrics
0709 12:59:43.696481 (+ 30392us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:59:43.706819 (+ 10338us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":163100,"reported_tablets":14,"spinlock_wait_cycles":1781712640,"child_traces":[["op",{"apply.queue_time_us":147107,"apply.run_cpu_time_us":3614,"apply.run_wall_time_us":3615,"num_ops":11,"prepare.queue_time_us":294890,"prepare.run_cpu_time_us":43846,"prepare.run_wall_time_us":348689,"raft.queue_time_us":183641,"raft.run_cpu_time_us":25009,"raft.run_wall_time_us":133901,"replication_time_us":630533,"thread_start_us":568913,"threads_started":4,"wal-append.queue_time_us":342825,"wal-append.run_cpu_time_us":40007,"wal-append.run_wall_time_us":1443360}]]}
I20260709 12:59:46.432610  3857 leader_election.cc:290] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:46.472704  2335 raft_consensus.cc:1240] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 4. Current term is 5. Ops: []
I20260709 12:59:46.477797  3816 raft_consensus.cc:3060] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:59:46.480494  3888 consensus_meta.cc:287] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.689s	user 0.014s	sys 0.000s
I20260709 12:59:46.485957  2335 raft_consensus.cc:1240] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 3 LEADER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 2. Current term is 3. Ops: []
I20260709 12:59:46.513337  3783 raft_consensus.cc:493] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 12:59:46.513509  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 12:59:46.515482  3752 raft_consensus.cc:493] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 12:59:46.515873  2333 raft_consensus.cc:2410] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260709 12:59:46.516242  3752 raft_consensus.cc:515] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:46.518597  3521 raft_consensus.cc:493] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 12:59:46.519374  3521 raft_consensus.cc:515] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:46.522326  3521 leader_election.cc:290] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:46.529207  3752 leader_election.cc:290] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:46.530030  3752 raft_consensus.cc:493] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:46.514364  3783 raft_consensus.cc:515] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:46.534751  1604 raft_consensus.cc:3060] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:59:46.535842  3854 raft_consensus.cc:515] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:46.531996  3906 raft_consensus.cc:493] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:46.552106  3622 raft_consensus.cc:3055] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:59:46.491712  3888 raft_consensus.cc:515] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 12:59:46.548296  3787 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.653s	user 0.008s	sys 0.017s
I20260709 12:59:46.557545  3773 raft_consensus.cc:493] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 12:59:46.568966  3773 raft_consensus.cc:515] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:46.572407  3773 leader_election.cc:290] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:46.574563  3911 raft_consensus.cc:493] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 12:59:46.575531  3911 raft_consensus.cc:515] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:46.578531  3854 leader_election.cc:290] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 6 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:46.552753  3622 raft_consensus.cc:740] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 12:59:46.530941  3752 raft_consensus.cc:515] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 12:59:46.582329  3787 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.687s	user 0.009s	sys 0.018s
I20260709 12:59:46.607204  3622 consensus_queue.cc:260] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:46.643914  1604 raft_consensus.cc:1275] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
W20260709 12:59:46.581874  3895 log.cc:927] Time spent T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.815s	user 0.000s	sys 0.007s
W20260709 12:59:46.629048  3663 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.066s	user 0.005s	sys 0.059s
I20260709 12:59:46.643270  3783 leader_election.cc:290] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:46.549933  3906 raft_consensus.cc:515] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:46.662540  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 12:59:46.625262  1718 leader_election.cc:336] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625): Timed out: RequestConsensusVote RPC to 127.0.107.196:46625 timed out after 2.002s (SENT)
I20260709 12:59:46.664637  1600 raft_consensus.cc:2468] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 2.
I20260709 12:59:46.664783  1718 leader_election.cc:304] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 12:59:46.665167  1600 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40154 (request call id 1708) took 1180 ms. Trace:
I20260709 12:59:46.645170  1604 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 1754) took 1413 ms. Trace:
I20260709 12:59:46.651067  3911 leader_election.cc:290] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 12:59:46.652868  3663 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.093s	user 0.005s	sys 0.060s
I20260709 12:59:46.665570  1600 rpcz_store.cc:276] 0709 12:59:45.485169 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:45.529854 (+ 44685us) service_pool.cc:224] Handling call
0709 12:59:46.662232 (+1132378us) spinlock_profiling.cc:243] Waited 7.66 ms on lock 0x7b5800003698. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f426303db71 00007f426303da22 00007f426bf56c09 00007f426b0b0ee0 00007f426b17a0ab 00007f426b17a131 00007f426b1280a0 00007f426b127984 00007f42646a26d4 00007f42646a2662
0709 12:59:46.665097 (+  2865us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":17609088}
I20260709 12:59:46.689744  1596 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 1762) took 1359 ms. Trace:
I20260709 12:59:46.659451  3622 raft_consensus.cc:3060] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:59:46.692222  1596 rpcz_store.cc:276] 0709 12:59:45.330271 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:45.439519 (+109248us) service_pool.cc:224] Handling call
0709 12:59:45.439821 (+   302us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:59:45.440221 (+   400us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:59:45.440254 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:59:45.440815 (+   561us) log.cc:844] Serialized 26 byte log entry
0709 12:59:45.561906 (+121091us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:59:45.561988 (+    82us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:45.562167 (+   179us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:59:46.580750 (+1018583us) spinlock_profiling.cc:243] Waited 18.5 ms on lock 0x7b4400022b4c. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f4262f9bc85 00007f4262f9c6a2 00007f4265d0e1db 00007f4265d09aaf 00007f4265cfd86e 00007f4265cfcbb6 00007f4265cfc7d8 00007f4265d06221 00007f4265d061d9 00007f4265d06169
0709 12:59:46.664399 (+ 83649us) raft_consensus.cc:1685] finished
0709 12:59:46.664439 (+    40us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:59:46.689443 (+ 25004us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1281,"raft.run_cpu_time_us":131,"raft.run_wall_time_us":131,"spinlock_wait_cycles":42459904,"thread_start_us":120524,"threads_started":1,"wal-append.queue_time_us":288847}
I20260709 12:59:46.697849  1604 rpcz_store.cc:276] 0709 12:59:45.231820 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:45.232256 (+   436us) service_pool.cc:224] Handling call
0709 12:59:45.251996 (+ 19740us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:59:46.534331 (+1282335us) spinlock_profiling.cc:243] Waited 1.03 s on lock 0x7b580037b8d0. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42660be0aa 00007f4266140447 00007f4266158fe7 00007f426615a7de 00007f4266164b49 00007f4266165416 00007f4266161d8f 00007f426616183e 00007f426b126419 00007f426469dbd4
0709 12:59:46.644730 (+110399us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:46.645066 (+   336us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2366039936}
W20260709 12:59:46.658952  3761 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.739s	user 0.010s	sys 0.025s
I20260709 12:59:46.693948  3521 raft_consensus.cc:2749] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 12:59:46.703925  3793 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.548s	user 0.010s	sys 0.022s
W20260709 12:59:46.701566  3811 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.781s	user 0.020s	sys 0.052s
W20260709 12:59:46.734951  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 109753us
I20260709 12:59:46.658479  3796 raft_consensus.cc:515] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:46.741894  3796 leader_election.cc:290] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 4 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 12:59:46.704013  3846 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.808s	user 0.017s	sys 0.014s
I20260709 12:59:46.723460  3857 raft_consensus.cc:493] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 12:59:46.755149  3888 leader_election.cc:290] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:46.763019  3896 raft_consensus.cc:3055] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:59:46.766626  3906 leader_election.cc:290] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:46.770897  1752 raft_consensus.cc:2468] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 4.
W20260709 12:59:46.778749  3878 log.cc:927] Time spent T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 1.294s	user 0.007s	sys 0.010s
W20260709 12:59:46.735633  3793 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.580s	user 0.011s	sys 0.022s
W20260709 12:59:46.738332  3811 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.820s	user 0.020s	sys 0.053s
W20260709 12:59:46.723989  3761 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.804s	user 0.010s	sys 0.025s
W20260709 12:59:46.789677  3894 log.cc:927] Time spent T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 1.040s	user 0.005s	sys 0.000s
I20260709 12:59:46.792769  1676 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57092 (request call id 457) took 1061 ms. Trace:
I20260709 12:59:46.795507  1747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55840 (request call id 2020) took 1731 ms. Trace:
I20260709 12:59:46.929587  2334 raft_consensus.cc:1240] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 4 LEADER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 3. Current term is 4. Ops: []
W20260709 12:59:46.804399  1571 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:59:46.931217  1571 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:59:46.929203  1676 rpcz_store.cc:276] 0709 12:59:45.731796 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:45.731935 (+   139us) service_pool.cc:224] Handling call
0709 12:59:46.792493 (+1060558us) spinlock_profiling.cc:243] Waited 535 ms on lock 0x7b5400347454. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 12:59:46.792675 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1230739840}
I20260709 12:59:46.932648  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:59:46.933511  1676 raft_consensus.cc:3060] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:59:46.791363  1754 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55840 (request call id 2024) took 1726 ms. Trace:
W20260709 12:59:46.816934  1644 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.740s	user 0.014s	sys 0.028s
W20260709 12:59:46.935079  1644 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.858s	user 0.014s	sys 0.028s
I20260709 12:59:46.936244  1717 leader_election.cc:304] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:59:46.824564  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:46.937475  1751 raft_consensus.cc:2468] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 2.
I20260709 12:59:46.937592  1717 leader_election.cc:304] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:59:46.814669  3663 raft_consensus.cc:493] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:46.822889  3773 raft_consensus.cc:493] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:46.823112  3908 raft_consensus.cc:493] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:46.827529  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 12:59:46.828742  1569 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.693s	user 0.015s	sys 0.030s
W20260709 12:59:46.947330  1569 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.811s	user 0.015s	sys 0.030s
I20260709 12:59:46.947394  1748 raft_consensus.cc:2468] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 3.
W20260709 12:59:46.948088  1571 connection.cc:256] RPC call timeout handler was delayed by -1.1033444779999968s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2155s for a response
I20260709 12:59:46.948984  1717 leader_election.cc:400] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 election: Vote denied by peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445) with higher term. Message: Invalid argument: T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 3. Current term is 4.
W20260709 12:59:46.847995  1570 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.173s	user 0.019s	sys 0.031s
I20260709 12:59:46.848690  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:46.950050  1680 raft_consensus.cc:2468] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 4.
I20260709 12:59:46.850190  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 12:59:46.853332  2299 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.671s	user 0.013s	sys 0.036s
W20260709 12:59:46.865514  2298 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.022s	user 0.041s	sys 0.075s
W20260709 12:59:46.950950  2299 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.769s	user 0.013s	sys 0.036s
W20260709 12:59:46.951056  2298 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.108s	user 0.042s	sys 0.075s
I20260709 12:59:46.952584  2299 leader_election.cc:304] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:46.853260  3854 raft_consensus.cc:493] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:46.872701  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:46.964032  1679 raft_consensus.cc:2468] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 3.
I20260709 12:59:46.855198  3912 raft_consensus.cc:493] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:59:46.968708  1568 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.712s	user 0.000s	sys 0.044s
W20260709 12:59:46.969461  1568 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.834s	user 0.000s	sys 0.045s
W20260709 12:59:46.888495  1674 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.768s	user 0.028s	sys 0.028s
I20260709 12:59:46.878561  3752 leader_election.cc:290] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 12:59:46.909045  3846 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.012s	user 0.017s	sys 0.014s
W20260709 12:59:46.912559  1673 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.697s	user 0.005s	sys 0.094s
W20260709 12:59:46.975299  1673 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.763s	user 0.006s	sys 0.094s
W20260709 12:59:46.975879  3913 log.cc:927] Time spent T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.051s	user 0.000s	sys 0.000s
I20260709 12:59:46.909570  3857 raft_consensus.cc:515] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:46.976015  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:46.976665  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 12:59:46.977301  2335 raft_consensus.cc:3060] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:59:46.977711  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:46.978276  3857 leader_election.cc:290] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:46.978420  2328 raft_consensus.cc:1240] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 4 LEADER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 3. Current term is 4. Ops: []
I20260709 12:59:46.978780  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
W20260709 12:59:46.913313  1598 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.732s	user 0.003s	sys 0.064s
W20260709 12:59:46.925455  1753 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.594s	user 0.011s	sys 0.019s
W20260709 12:59:46.979565  1598 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.798s	user 0.004s	sys 0.064s
W20260709 12:59:46.979671  1753 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.649s	user 0.012s	sys 0.020s
I20260709 12:59:46.910420  3896 raft_consensus.cc:740] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 12:59:46.934729  1754 rpcz_store.cc:276] 0709 12:59:45.064706 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:45.273587 (+208881us) service_pool.cc:224] Handling call
0709 12:59:45.480470 (+206883us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:59:45.507342 (+ 26872us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:46.791028 (+1283686us) spinlock_profiling.cc:243] Waited 206 ms on lock 0x7b5400071550. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661618ad 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 12:59:46.791267 (+   239us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":474748800}
I20260709 12:59:46.992257  1678 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52766 (request call id 2018) took 2575 ms. Trace:
I20260709 12:59:46.945001  1751 raft_consensus.cc:1240] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 3. Current term is 4. Ops: []
I20260709 12:59:46.949455  1717 leader_election.cc:403] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
W20260709 12:59:46.943490  1749 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.605s	user 0.003s	sys 0.035s
I20260709 12:59:46.952075  1644 leader_election.cc:304] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 12:59:46.996898  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 44911us
I20260709 12:59:46.955955  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
W20260709 12:59:46.956585  1570 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.282s	user 0.019s	sys 0.031s
I20260709 12:59:46.956641  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:46.958115  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:59:46.959707  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:59:46.961573  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
W20260709 12:59:46.971475  1674 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.851s	user 0.029s	sys 0.029s
I20260709 12:59:46.975031  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:46.929904  1747 rpcz_store.cc:276] 0709 12:59:45.063600 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:45.217173 (+153573us) service_pool.cc:224] Handling call
0709 12:59:45.217540 (+   367us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:59:46.285303 (+1067763us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:46.795167 (+509864us) spinlock_profiling.cc:243] Waited 1.04 s on lock 0x7b54001c6954. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 12:59:46.795424 (+   257us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2384365440}
I20260709 12:59:46.980965  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:59:46.975092  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 12:59:46.992748  1678 rpcz_store.cc:276] 0709 12:59:44.417068 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:44.920686 (+503618us) service_pool.cc:224] Handling call
0709 12:59:44.921224 (+   538us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:59:44.941680 (+ 20456us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:59:44.941731 (+    51us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:59:44.949884 (+  8153us) log.cc:844] Serialized 26 byte log entry
0709 12:59:45.243971 (+294087us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:59:45.244026 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:45.244119 (+    93us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:59:46.777855 (+1533736us) spinlock_profiling.cc:243] Waited 5.41 ms on lock 0x7b4400022b4c. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f4262f9bc85 00007f4262f9c6a2 00007f4265d0e1db 00007f4265d09aaf 00007f4265cfd86e 00007f4265cfcbb6 00007f4265cfc7d8 00007f4265d06221 00007f4265d061d9 00007f4265d06169
0709 12:59:46.972799 (+194944us) raft_consensus.cc:1685] finished
0709 12:59:46.972822 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:59:46.992158 (+ 19336us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":19650,"raft.queue_time_us":19916,"raft.run_cpu_time_us":199,"raft.run_wall_time_us":5121,"spinlock_wait_cycles":12437248,"thread_start_us":237385,"threads_started":1,"wal-append.queue_time_us":428433}
I20260709 12:59:47.005858  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:59:47.007261  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:59:47.007941  1678 raft_consensus.cc:3060] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:59:46.923967  3543 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.638s	user 0.014s	sys 0.017s
I20260709 12:59:47.011662  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:46.947196  3639 consensus_queue.cc:1048] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.065s
I20260709 12:59:46.954586  3663 raft_consensus.cc:515] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:46.964972  3773 raft_consensus.cc:515] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } }
I20260709 12:59:46.965148  3908 raft_consensus.cc:515] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:47.215528  1747 raft_consensus.cc:1240] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 1. Current term is 2. Ops: []
I20260709 12:59:47.574738  1754 raft_consensus.cc:1240] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 2. Current term is 3. Ops: []
I20260709 12:59:47.602993  2334 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49082 (request call id 546) took 1253 ms. Trace:
I20260709 12:59:47.011745  1751 raft_consensus.cc:1240] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 2. Current term is 3. Ops: []
W20260709 12:59:47.214805  1745 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.995s	user 0.023s	sys 0.100s
I20260709 12:59:46.977763  3761 raft_consensus.cc:493] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:46.973090  3753 raft_consensus.cc:2804] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260709 12:59:47.661664  1745 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.449s	user 0.026s	sys 0.116s
W20260709 12:59:47.188581  1646 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.928s	user 0.015s	sys 0.060s
W20260709 12:59:47.003950  1749 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.666s	user 0.003s	sys 0.035s
I20260709 12:59:46.968071  3888 raft_consensus.cc:2804] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:59:47.394812  1748 raft_consensus.cc:1275] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260709 12:59:47.138237  1676 raft_consensus.cc:2468] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 3.
I20260709 12:59:47.654413  1644 leader_election.cc:304] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:47.699410  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 12:59:47.700078  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 45735.2us
I20260709 12:59:47.652958  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
W20260709 12:59:47.417752  3927 log.cc:927] Time spent T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.113s	user 0.000s	sys 0.001s
W20260709 12:59:47.698243  1717 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.702s	user 0.027s	sys 0.065s
I20260709 12:59:47.572643  3521 raft_consensus.cc:2749] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445) with higher term. Message: Invalid argument: T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 3. Current term is 4.
W20260709 12:59:47.381249  1500 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.896s	user 0.008s	sys 0.038s
W20260709 12:59:47.657423  1570 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:59:47.706427  1717 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.711s	user 0.027s	sys 0.067s
W20260709 12:59:47.706909  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 758002us
W20260709 12:59:47.707242  1500 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.222s	user 0.008s	sys 0.043s
I20260709 12:59:47.660265  2334 rpcz_store.cc:276] 0709 12:59:46.349335 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:46.423759 (+ 74424us) service_pool.cc:224] Handling call
0709 12:59:46.424305 (+   546us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:59:46.944691 (+520386us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:47.602637 (+657946us) spinlock_profiling.cc:243] Waited 501 ms on lock 0x7b5400627ad4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 12:59:47.602906 (+   269us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1153106176}
I20260709 12:59:47.669531  1598 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40154 (request call id 1716) took 2182 ms. Trace:
W20260709 12:59:47.596956  3916 log.cc:927] Time spent T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.683s	user 0.002s	sys 0.000s
W20260709 12:59:47.682137  2335 consensus_meta.cc:287] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.704s	user 0.012s	sys 0.001s
I20260709 12:59:47.719612  2335 raft_consensus.cc:2468] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 4.
W20260709 12:59:47.688973  1646 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.429s	user 0.028s	sys 0.092s
I20260709 12:59:46.963166  3816 raft_consensus.cc:493] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 12:59:47.688071  3753 raft_consensus.cc:493] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
W20260709 12:59:46.965454  3638 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.573s	user 0.003s	sys 0.013s
W20260709 12:59:47.393844  1752 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55848 (request call id 1774) took 2298 ms (client timeout 1634 ms). Trace:
W20260709 12:59:47.636310  3917 log.cc:927] Time spent T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.635s	user 0.003s	sys 0.000s
W20260709 12:59:47.026510  1675 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.772s	user 0.021s	sys 0.031s
W20260709 12:59:47.210228  2300 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.729s	user 0.022s	sys 0.015s
W20260709 12:59:47.455771  1677 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.063s	user 0.004s	sys 0.043s
I20260709 12:59:47.693435  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:59:47.598608  1746 raft_consensus.cc:1240] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 2. Current term is 3. Ops: []
W20260709 12:59:47.283800  3920 log.cc:927] Time spent T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.123s	user 0.001s	sys 0.000s
I20260709 12:59:46.997920  3823 raft_consensus.cc:493] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:46.981860  3793 raft_consensus.cc:2804] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Leader election won for term 5
W20260709 12:59:47.010941  3543 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.729s	user 0.014s	sys 0.017s
W20260709 12:59:47.426174  3918 log.cc:927] Time spent T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.210s	user 0.000s	sys 0.000s
I20260709 12:59:46.954550  3796 raft_consensus.cc:493] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
W20260709 12:59:47.740869  1675 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.486s	user 0.050s	sys 0.068s
I20260709 12:59:47.611795  2330 raft_consensus.cc:1240] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 4 LEADER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 2. Current term is 4. Ops: []
I20260709 12:59:46.973384  3854 raft_consensus.cc:515] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:47.746352  3783 raft_consensus.cc:493] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:47.700085  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:59:47.763114  1676 raft_consensus.cc:1240] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 4. Current term is 5. Ops: []
W20260709 12:59:47.304039  1750 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.924s	user 0.025s	sys 0.016s
W20260709 12:59:47.765553  1750 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.386s	user 0.025s	sys 0.025s
I20260709 12:59:47.191527  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 12:59:47.213058  2337 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.755s	user 0.012s	sys 0.026s
W20260709 12:59:47.772449  2337 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.315s	user 0.012s	sys 0.039s
W20260709 12:59:47.257218  1672 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.769s	user 0.019s	sys 0.022s
W20260709 12:59:47.773507  1672 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.286s	user 0.027s	sys 0.024s
I20260709 12:59:46.991779  3846 raft_consensus.cc:493] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:59:47.709203  1568 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.721s	user 0.001s	sys 0.016s
W20260709 12:59:47.705590  1678 consensus_meta.cc:287] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.697s	user 0.012s	sys 0.000s
W20260709 12:59:47.709647  1673 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.733s	user 0.005s	sys 0.013s
W20260709 12:59:47.705658  1718 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.966s	user 0.000s	sys 0.015s
I20260709 12:59:47.774720  3846 raft_consensus.cc:515] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:47.774979  1678 raft_consensus.cc:2468] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 2.
W20260709 12:59:47.775264  1718 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 1.036s	user 0.000s	sys 0.016s
I20260709 12:59:47.331360  3857 raft_consensus.cc:2804] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260709 12:59:47.724858  1643 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.771s	user 0.003s	sys 0.015s
I20260709 12:59:47.790149  3857 raft_consensus.cc:493] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:59:47.790267  1643 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.836s	user 0.004s	sys 0.015s
W20260709 12:59:47.790815  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 836635us
I20260709 12:59:47.790705  3922 consensus_queue.cc:1059] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:59:47.792307  3854 leader_election.cc:290] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:47.708384  1598 rpcz_store.cc:276] 0709 12:59:45.487579 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:46.075403 (+587824us) service_pool.cc:224] Handling call
0709 12:59:46.076017 (+   614us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:59:46.139551 (+ 63534us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:59:46.139590 (+    39us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:59:46.181303 (+ 41713us) log.cc:844] Serialized 532 byte log entry
0709 12:59:46.979995 (+798692us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:59:46.980039 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:46.980110 (+    71us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:59:47.412665 (+432555us) raft_consensus.cc:1685] finished
0709 12:59:47.412699 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:59:47.669430 (+256731us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":51175,"raft.queue_time_us":68445,"raft.run_cpu_time_us":247,"raft.run_wall_time_us":13447,"thread_start_us":798238,"threads_started":1,"wal-append.queue_time_us":1211844}
I20260709 12:59:47.660969  1751 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43156 (request call id 391) took 1173 ms. Trace:
I20260709 12:59:47.793035  2330 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49082 (request call id 549) took 1352 ms. Trace:
I20260709 12:59:47.793615  1751 rpcz_store.cc:276] 0709 12:59:46.487574 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:47.011211 (+523637us) service_pool.cc:224] Handling call
0709 12:59:47.011542 (+   331us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:59:47.660651 (+649109us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:47.660895 (+   244us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:59:47.793708  2330 rpcz_store.cc:276] 0709 12:59:46.441169 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:47.600309 (+1159140us) service_pool.cc:224] Handling call
0709 12:59:47.600774 (+   465us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:59:47.769602 (+168828us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:47.792942 (+ 23340us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1538432}
I20260709 12:59:47.731369  1748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43156 (request call id 387) took 1399 ms. Trace:
I20260709 12:59:47.790721  3857 raft_consensus.cc:3060] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:59:47.794844  1748 rpcz_store.cc:276] 0709 12:59:46.332016 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:46.963005 (+630989us) service_pool.cc:224] Handling call
0709 12:59:47.394195 (+431190us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:59:47.700731 (+306536us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:47.731083 (+ 30352us) spinlock_profiling.cc:243] Waited 431 ms on lock 0x7b5400019fd0. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661618ad 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 12:59:47.731275 (+   192us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":990613888}
I20260709 12:59:47.760174  3796 raft_consensus.cc:515] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:47.797724  3796 leader_election.cc:290] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:46.982393  3912 raft_consensus.cc:515] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:47.799546  1679 raft_consensus.cc:3060] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:59:47.732033  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:47.800530  1671 raft_consensus.cc:2468] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 3.
I20260709 12:59:47.661703  3898 consensus_queue.cc:1048] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.064s
W20260709 12:59:47.737197  2299 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.107.194:46261 timed out after 1.906s (SENT) [suppressed 6 similar messages]
W20260709 12:59:47.738724  1752 rpcz_store.cc:269] 0709 12:59:45.094949 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:45.146222 (+ 51273us) service_pool.cc:224] Handling call
0709 12:59:47.393449 (+2247227us) spinlock_profiling.cc:243] Waited 1.62 s on lock 0x7b5400019fd4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 12:59:47.393691 (+   242us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3733006720}
W20260709 12:59:47.743167  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 57477.9us
I20260709 12:59:47.744273  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:47.810910  1680 raft_consensus.cc:2393] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 4: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 12:59:47.811376  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:59:47.814577  1671 raft_consensus.cc:2393] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 4: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
W20260709 12:59:47.746381  1677 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.354s	user 0.007s	sys 0.045s
I20260709 12:59:47.725996  3888 raft_consensus.cc:493] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:59:47.730830  3521 raft_consensus.cc:493] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:59:47.749629  1569 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.793s	user 0.005s	sys 0.018s
W20260709 12:59:47.820686  1569 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.864s	user 0.005s	sys 0.018s
I20260709 12:59:47.822295  1569 leader_election.cc:304] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 12:59:47.305245  3896 consensus_queue.cc:260] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:47.735183  3753 raft_consensus.cc:3060] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:59:47.827752  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 12:59:47.738132  3638 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.356s	user 0.004s	sys 0.041s
I20260709 12:59:47.690757  3761 raft_consensus.cc:515] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:47.694171  3921 consensus_queue.cc:1048] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.053s
W20260709 12:59:47.745574  2300 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.265s	user 0.035s	sys 0.021s
I20260709 12:59:47.636796  3805 raft_consensus.cc:3055] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 LEADER]: Stepping down as leader of term 1
W20260709 12:59:47.759270  3622 consensus_meta.cc:287] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 1.061s	user 0.011s	sys 0.010s
W20260709 12:59:47.777805  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 1.03853e+06us
I20260709 12:59:47.755298  3823 raft_consensus.cc:515] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 12:59:47.774883  1568 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.787s	user 0.001s	sys 0.016s
I20260709 12:59:46.972391  3906 consensus_queue.cc:1048] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.724s
I20260709 12:59:47.792101  3922 consensus_queue.cc:1048] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:59:47.727342  1644 leader_election.cc:304] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:47.793033  3854 raft_consensus.cc:493] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:47.771351  3793 raft_consensus.cc:697] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 5 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 12:59:47.844050  3761 leader_election.cc:290] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:47.852852  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 12:59:47.775267  1673 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.799s	user 0.005s	sys 0.013s
I20260709 12:59:47.802994  1749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55840 (request call id 2025) took 2735 ms. Trace:
I20260709 12:59:47.862778  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:47.825384  3888 raft_consensus.cc:3060] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:59:47.878144  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 150873us
I20260709 12:59:47.881460  1644 leader_election.cc:304] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:47.850786  3805 raft_consensus.cc:740] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 12:59:47.800140  3783 raft_consensus.cc:515] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:47.855091  3944 raft_consensus.cc:493] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:59:47.904237  3924 log.cc:927] Time spent T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.336s	user 0.002s	sys 0.002s
I20260709 12:59:47.909381  1673 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52766 (request call id 2036) took 1805 ms. Trace:
I20260709 12:59:47.861738  3622 raft_consensus.cc:493] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:47.902768  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:47.925012  1671 raft_consensus.cc:2468] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 5.
I20260709 12:59:47.856777  3846 leader_election.cc:290] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 12:59:47.911063  2301 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.413s	user 0.039s	sys 0.074s
W20260709 12:59:47.929570  2301 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.431s	user 0.039s	sys 0.075s
I20260709 12:59:47.932456  1748 raft_consensus.cc:3055] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:59:47.932837  1748 raft_consensus.cc:740] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 12:59:47.933125  1644 leader_election.cc:304] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 12:59:47.933975  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:47.853031  3521 raft_consensus.cc:515] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:47.880163  3896 raft_consensus.cc:3060] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:59:47.935662  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:47.945737  1751 raft_consensus.cc:2468] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 3.
I20260709 12:59:47.749218  3816 raft_consensus.cc:515] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:47.914064  3905 raft_consensus.cc:2749] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:59:47.856158  3639 consensus_queue.cc:1048] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.064s
I20260709 12:59:47.949555  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:47.943251  1748 consensus_queue.cc:260] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:47.882426  3854 raft_consensus.cc:515] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:47.990576  2334 raft_consensus.cc:2468] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 5.
I20260709 12:59:47.959347  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
W20260709 12:59:47.947892  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 263301us
I20260709 12:59:47.987562  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 12:59:47.987947  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:47.939085  3908 leader_election.cc:290] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:48.007477  2333 raft_consensus.cc:2393] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 4: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
W20260709 12:59:47.950516  3930 log.cc:927] Time spent T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.244s	user 0.000s	sys 0.001s
W20260709 12:59:48.028601  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 303069us
I20260709 12:59:47.986303  1748 raft_consensus.cc:3060] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:59:48.074721  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 63564.4us
I20260709 12:59:47.937667  3932 raft_consensus.cc:2804] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:59:47.938838  3898 raft_consensus.cc:2749] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:59:47.949731  3921 raft_consensus.cc:2749] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:59:48.076519  1646 leader_election.cc:304] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:59:48.079736  3946 consensus_queue.cc:1048] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:59:48.079762  3947 consensus_queue.cc:1059] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260709 12:59:48.083274  3931 log.cc:927] Time spent T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.075s	user 0.000s	sys 0.000s
I20260709 12:59:48.083592  3823 leader_election.cc:290] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:48.083922  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:47.988902  3796 raft_consensus.cc:493] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:59:48.022274  1720 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.637s	user 0.023s	sys 0.016s
W20260709 12:59:48.180712  1720 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.025s	user 0.024s	sys 0.017s
I20260709 12:59:47.970268  3773 leader_election.cc:290] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:48.176118  1674 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57092 (request call id 456) took 2403 ms. Trace:
I20260709 12:59:47.986090  3948 raft_consensus.cc:493] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:47.989898  3543 raft_consensus.cc:2749] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:59:47.942348  3622 raft_consensus.cc:515] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:48.093175  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:48.093597  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:48.221457  1751 raft_consensus.cc:3060] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Advancing to term 4
I20260709 12:59:48.094494  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:59:48.222159  1603 raft_consensus.cc:3060] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:59:48.094825  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:48.094904  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:48.223562  1599 raft_consensus.cc:2393] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 3: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 12:59:48.096285  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:48.224155  1605 raft_consensus.cc:2468] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 2.
I20260709 12:59:48.097465  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 5 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:48.225621  1600 raft_consensus.cc:2410] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for term 5 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 3.
I20260709 12:59:48.098515  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:48.102075  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:48.098165  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:48.099368  2301 leader_election.cc:304] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
W20260709 12:59:48.231077  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 131801us
I20260709 12:59:48.008867  3663 leader_election.cc:290] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:48.135093  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:48.232764  2334 raft_consensus.cc:2393] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 4: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
W20260709 12:59:48.155853  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 418601us
I20260709 12:59:48.158759  1680 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52778 (request call id 1664) took 1302 ms. Trace:
I20260709 12:59:48.162175  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:59:48.162343  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:59:48.173736  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:47.949350  3846 raft_consensus.cc:3055] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:59:48.191128  1674 rpcz_store.cc:276] 0709 12:59:45.731236 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:45.731422 (+   186us) service_pool.cc:224] Handling call
0709 12:59:45.748324 (+ 16902us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:59:45.818474 (+ 70150us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:59:45.818508 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:59:45.963489 (+144981us) log.cc:844] Serialized 26 byte log entry
0709 12:59:47.000412 (+1036923us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:59:47.000452 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:47.000520 (+    68us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:59:47.837251 (+836731us) spinlock_profiling.cc:243] Waited 11.2 ms on lock 0x7b4400022b4c. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f4262f9bc85 00007f4262f9c6a2 00007f4265d0e1db 00007f4265d09aaf 00007f4265cfd86e 00007f4265cfcbb6 00007f4265cfc7d8 00007f4265d06221 00007f4265d061d9 00007f4265d06169
0709 12:59:48.118010 (+280759us) raft_consensus.cc:1685] finished
0709 12:59:48.118036 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:59:48.134832 (+ 16796us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":21014,"raft.queue_time_us":68992,"raft.run_cpu_time_us":101,"raft.run_wall_time_us":102,"spinlock_wait_cycles":25670016,"thread_start_us":940583,"threads_started":2,"wal-append.queue_time_us":1349985}
I20260709 12:59:48.099123  3932 raft_consensus.cc:493] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 12:59:47.920814  3944 raft_consensus.cc:515] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:47.961256  3805 consensus_queue.cc:260] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:48.236155  2335 raft_consensus.cc:2468] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 4.
I20260709 12:59:48.259958  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:59:48.235540  1680 rpcz_store.cc:276] 0709 12:59:46.856442 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:47.901602 (+1045160us) service_pool.cc:224] Handling call
0709 12:59:47.907327 (+  5725us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:59:47.907694 (+   367us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 12:59:47.941903 (+ 34209us) log.cc:844] Serialized 14 byte log entry
0709 12:59:48.134403 (+192500us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:59:48.158216 (+ 23813us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:59:48.158318 (+   102us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:48.158429 (+   111us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:59:48.158652 (+   223us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":13430,"prepare.queue_time_us":39984,"prepare.run_cpu_time_us":5122,"prepare.run_wall_time_us":40963,"thread_start_us":165135,"threads_started":1}
I20260709 12:59:48.235836  1598 raft_consensus.cc:2376] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 5. Re-sending same reply.
I20260709 12:59:48.260864  1605 raft_consensus.cc:3060] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:59:47.918903  1673 rpcz_store.cc:276] 0709 12:59:46.104050 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:46.209050 (+105000us) service_pool.cc:224] Handling call
0709 12:59:46.209373 (+   323us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:59:46.209751 (+   378us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:59:46.209777 (+    26us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:59:46.210265 (+   488us) log.cc:844] Serialized 26 byte log entry
0709 12:59:46.975815 (+765550us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:59:47.872777 (+896962us) log.cc:844] Serialized 14 byte log entry
0709 12:59:47.909011 (+ 36234us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:47.909097 (+    86us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:59:47.909148 (+    51us) raft_consensus.cc:1685] finished
0709 12:59:47.909160 (+    12us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:59:47.909307 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":35776,"prepare.queue_time_us":442317,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":41,"raft.queue_time_us":155173,"raft.run_cpu_time_us":246,"raft.run_wall_time_us":4776,"thread_start_us":1660407,"threads_started":2,"wal-append.queue_time_us":1352717}
I20260709 12:59:48.249919  1752 raft_consensus.cc:2468] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 4.
I20260709 12:59:48.002306  3793 consensus_queue.cc:237] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:48.254993  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:48.265107  1752 raft_consensus.cc:1275] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
W20260709 12:59:48.287509  1717 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.562s	user 0.000s	sys 0.023s
W20260709 12:59:48.288362  1717 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.562s	user 0.000s	sys 0.023s
W20260709 12:59:48.288817  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 562975us
W20260709 12:59:48.290230  2337 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.511s	user 0.009s	sys 0.013s
W20260709 12:59:48.290810  2337 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.518s	user 0.009s	sys 0.014s
I20260709 12:59:48.292012  2337 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49082 (request call id 548) took 1866 ms. Trace:
W20260709 12:59:48.293944  1643 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.502s	user 0.008s	sys 0.015s
W20260709 12:59:48.294417  1643 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.503s	user 0.008s	sys 0.015s
W20260709 12:59:48.294834  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 503152us
I20260709 12:59:48.328675  1748 raft_consensus.cc:1275] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:59:48.292609  2337 rpcz_store.cc:276] 0709 12:59:46.425995 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:46.444488 (+ 18493us) service_pool.cc:224] Handling call
0709 12:59:46.445017 (+   529us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:59:46.445548 (+   531us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:59:47.773015 (+1327467us) log.cc:844] Serialized 14 byte log entry
0709 12:59:48.291451 (+518436us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:59:48.291541 (+    90us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:59:48.291635 (+    94us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:48.291742 (+   107us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:59:48.291923 (+   181us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":1494949,"prepare.run_cpu_time_us":73,"prepare.run_wall_time_us":8743,"thread_start_us":1833179,"threads_started":2}
I20260709 12:59:48.189793  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:48.217679  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:48.439236  3952 raft_consensus.cc:2804] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260709 12:59:48.562000  3951 log.cc:927] Time spent T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.313s	user 0.000s	sys 0.000s
W20260709 12:59:48.562775  3938 log.cc:927] Time spent T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.471s	user 0.000s	sys 0.001s
I20260709 12:59:48.332051  3941 raft_consensus.cc:493] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 12:59:48.379308  2299 leader_election.cc:304] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 12:59:48.256078  3921 consensus_queue.cc:1059] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.392s
I20260709 12:59:48.247810  3846 raft_consensus.cc:740] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
W20260709 12:59:48.554414  3931 log.cc:927] Time spent T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.116s	user 0.001s	sys 0.000s
I20260709 12:59:48.565518  3854 leader_election.cc:290] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 12:59:48.493427  3857 consensus_meta.cc:287] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.698s	user 0.013s	sys 0.001s
I20260709 12:59:47.877103  1749 rpcz_store.cc:276] 0709 12:59:45.067336 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:45.428242 (+360906us) service_pool.cc:224] Handling call
0709 12:59:45.428728 (+   486us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:59:45.457729 (+ 29001us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:59:45.462501 (+  4772us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:59:45.467554 (+  5053us) log.cc:844] Serialized 26 byte log entry
0709 12:59:46.338117 (+870563us) spinlock_profiling.cc:243] Waited 10.7 ms on lock 0x7b180004af20. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f4263057e07 00007f426beb2ab9 00007f426307469b 00007f426307293f 00007f4263074c02 00007f4265cfc20b 00007f4265d01d23 00007f4265d020d1 00007f426611da0f
0709 12:59:47.702498 (+1364381us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:59:47.714789 (+ 12291us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:47.726537 (+ 11748us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:59:47.726610 (+    73us) raft_consensus.cc:1685] finished
0709 12:59:47.727168 (+   558us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:59:47.773662 (+ 46494us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":42204,"raft.queue_time_us":27065,"raft.run_cpu_time_us":203,"raft.run_wall_time_us":10907,"spinlock_wait_cycles":24617344,"thread_start_us":1352980,"threads_started":1,"wal-append.queue_time_us":2111985}
I20260709 12:59:48.458711  3912 leader_election.cc:290] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 12:59:48.564445  3934 log.cc:927] Time spent T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.309s	user 0.000s	sys 0.000s
I20260709 12:59:48.416122  3950 raft_consensus.cc:493] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:48.568776  3521 leader_election.cc:290] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 12:59:48.333676  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 257243us
I20260709 12:59:48.416277  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:48.344914  1603 raft_consensus.cc:2468] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 4.
I20260709 12:59:48.646122  3952 raft_consensus.cc:493] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:59:48.208701  3796 raft_consensus.cc:515] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 12:59:48.823706  1571 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:59:48.687573  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 12:59:48.642102  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:59:49.145359  1680 raft_consensus.cc:2468] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 5.
I20260709 12:59:49.146050  1680 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52766 (request call id 2042) took 1143 ms. Trace:
I20260709 12:59:48.877095  1753 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55840 (request call id 2029) took 3449 ms. Trace:
I20260709 12:59:49.107573  3953 raft_consensus.cc:493] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:49.146654  1680 rpcz_store.cc:276] 0709 12:59:48.002861 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:48.335042 (+332181us) service_pool.cc:224] Handling call
0709 12:59:49.145932 (+810890us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:59:48.975538  3977 raft_consensus.cc:493] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
W20260709 12:59:48.437502  1569 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.599s	user 0.000s	sys 0.021s
I20260709 12:59:48.824434  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 12:59:49.149149  1569 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.310s	user 0.018s	sys 0.039s
I20260709 12:59:48.711884  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:48.503360  1677 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52766 (request call id 2040) took 2398 ms. Trace:
W20260709 12:59:48.514671  1644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:59:48.646647  1605 raft_consensus.cc:2468] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 6.
W20260709 12:59:49.049537  1570 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.957s	user 0.000s	sys 0.021s
W20260709 12:59:48.602277  1643 connection.cc:256] RPC call timeout handler was delayed by -1.1775691990000041s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15880000000000002s for a response
I20260709 12:59:49.144210  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:48.771492  1678 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52778 (request call id 1661) took 1918 ms. Trace:
I20260709 12:59:49.154139  1605 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40166 (request call id 1775) took 1053 ms. Trace:
I20260709 12:59:49.176120  1646 leader_election.cc:304] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1
W20260709 12:59:49.161171  1571 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:59:49.149467  1752 raft_consensus.cc:2468] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 2.
I20260709 12:59:48.873162  1751 raft_consensus.cc:2468] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 4.
I20260709 12:59:49.179472  1754 raft_consensus.cc:2468] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 4.
I20260709 12:59:49.179893  1751 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43156 (request call id 395) took 1086 ms. Trace:
I20260709 12:59:49.153223  1677 rpcz_store.cc:276] 0709 12:59:46.105169 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:46.391393 (+286224us) service_pool.cc:224] Handling call
0709 12:59:46.391749 (+   356us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:59:46.391983 (+   234us) raft_consensus.cc:1534] Early marking committed up to index 2
0709 12:59:47.815916 (+1423933us) log.cc:844] Serialized 14 byte log entry
0709 12:59:48.502688 (+686772us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:59:48.502792 (+   104us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:59:48.502903 (+   111us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:48.503021 (+   118us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:59:48.503246 (+   225us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":1025873,"prepare.run_cpu_time_us":59,"prepare.run_wall_time_us":60,"raft.queue_time_us":277696,"raft.run_cpu_time_us":231,"raft.run_wall_time_us":74610,"thread_start_us":2109667,"threads_started":3}
W20260709 12:59:48.648345  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 269133us
W20260709 12:59:48.394687  1718 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.542s	user 0.006s	sys 0.013s
W20260709 12:59:49.181789  1718 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.330s	user 0.027s	sys 0.033s
W20260709 12:59:48.935340  1679 consensus_meta.cc:287] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 1.125s	user 0.018s	sys 0.000s
I20260709 12:59:49.183575  1679 raft_consensus.cc:2468] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 4.
W20260709 12:59:48.644976  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 412909us
I20260709 12:59:48.353008  2329 raft_consensus.cc:2393] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 4: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
W20260709 12:59:49.178068  1571 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:59:49.186481  1571 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:59:49.178802  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 1.33994e+06us
I20260709 12:59:48.495358  3663 raft_consensus.cc:493] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 12:59:48.509316  3947 raft_consensus.cc:493] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:49.177367  1605 rpcz_store.cc:276] 0709 12:59:48.100537 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:48.259568 (+159031us) service_pool.cc:224] Handling call
0709 12:59:49.154031 (+894463us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:59:49.182950  1603 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40166 (request call id 1768) took 1088 ms. Trace:
W20260709 12:59:49.226629  1679 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52778 (request call id 1657) took 2374 ms (client timeout 2726 ms). Trace:
W20260709 12:59:49.154990  1570 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.063s	user 0.000s	sys 0.025s
I20260709 12:59:48.417589  3805 raft_consensus.cc:3060] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:48.650835  1746 raft_consensus.cc:2410] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 for term 4 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
W20260709 12:59:49.187580  1717 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.897s	user 0.009s	sys 0.014s
W20260709 12:59:48.516548  3800 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.085s	user 0.007s	sys 0.014s
I20260709 12:59:48.719672  3944 leader_election.cc:290] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:48.361485  3915 raft_consensus.cc:493] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 12:59:48.255354  3911 raft_consensus.cc:3060] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:59:49.180480  1751 rpcz_store.cc:276] 0709 12:59:48.092980 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:48.093279 (+   299us) service_pool.cc:224] Handling call
0709 12:59:49.179740 (+1086461us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:59:48.650825  1675 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52766 (request call id 2037) took 2546 ms. Trace:
I20260709 12:59:49.345471  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:49.347110  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:49.348014  1751 raft_consensus.cc:2393] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 4: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 12:59:48.648707  1672 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52766 (request call id 2041) took 2543 ms. Trace:
I20260709 12:59:48.958235  3966 raft_consensus.cc:493] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 12:59:49.182382  1718 leader_election.cc:304] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:49.260609  1603 rpcz_store.cc:276] 0709 12:59:48.094073 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:48.094218 (+   145us) service_pool.cc:224] Handling call
0709 12:59:49.182860 (+1088642us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:59:49.327728  1680 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57092 (request call id 464) took 1546 ms. Trace:
I20260709 12:59:49.349370  1680 rpcz_store.cc:276] 0709 12:59:47.781401 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:49.175477 (+1394076us) service_pool.cc:224] Handling call
0709 12:59:49.327104 (+151627us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:59:49.327342 (+   238us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:59:49.327372 (+    30us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:59:49.327420 (+    48us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:59:49.327462 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:49.327524 (+    62us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:59:49.327663 (+   139us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:59:49.304690  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 113517us
I20260709 12:59:49.305765  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
W20260709 12:59:49.324347  1717 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.034s	user 0.009s	sys 0.034s
W20260709 12:59:49.352160  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 1.06211e+06us
I20260709 12:59:49.353515  1717 leader_election.cc:304] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 12:59:49.175800  1678 rpcz_store.cc:276] 0709 12:59:46.853207 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:47.776446 (+923239us) service_pool.cc:224] Handling call
0709 12:59:47.776881 (+   435us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:59:47.777182 (+   301us) raft_consensus.cc:1534] Early marking committed up to index 2
0709 12:59:47.826252 (+ 49070us) log.cc:844] Serialized 14 byte log entry
0709 12:59:48.770261 (+944009us) spinlock_profiling.cc:243] Waited 48.8 ms on lock 0x7b180039bce0. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f4265d0a4bc 00007f426305848b 00007f426beb2ab9 00007f426307469b 00007f426307293f 00007f4263072455 00007f426615a060 00007f426616c716 00007f426617d287
0709 12:59:48.770426 (+   165us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:59:48.771026 (+   600us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:59:48.771101 (+    75us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:48.771198 (+    97us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:59:48.771391 (+   193us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":24165,"prepare.queue_time_us":130575,"prepare.run_cpu_time_us":60,"prepare.run_wall_time_us":62,"raft.queue_time_us":251984,"raft.run_cpu_time_us":228,"raft.run_wall_time_us":35636,"spinlock_wait_cycles":112254208,"thread_start_us":672395,"threads_started":2,"wal-append.queue_time_us":1174133}
W20260709 12:59:49.182943  3888 consensus_meta.cc:287] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 1.269s	user 0.012s	sys 0.017s
I20260709 12:59:48.613353  3955 raft_consensus.cc:493] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 12:59:48.224815  3948 raft_consensus.cc:515] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 12:59:49.330873  3800 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.900s	user 0.014s	sys 0.036s
I20260709 12:59:49.331925  3922 raft_consensus.cc:493] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:48.353679  3753 raft_consensus.cc:515] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 12:59:49.332031  1746 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43156 (request call id 397) took 1236 ms (client timeout 1580 ms). Trace:
I20260709 12:59:49.331792  3921 raft_consensus.cc:493] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 12:59:49.332868  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:49.337800  3971 raft_consensus.cc:493] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 12:59:49.334065  1677 raft_consensus.cc:1275] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20260709 12:59:49.405268  1677 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57092 (request call id 470) took 1534 ms. Trace:
I20260709 12:59:49.405653  1677 rpcz_store.cc:276] 0709 12:59:47.871463 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:49.333369 (+1461906us) service_pool.cc:224] Handling call
0709 12:59:49.333696 (+   327us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:59:49.404914 (+ 71218us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:49.405200 (+   286us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:59:49.334565  3956 raft_consensus.cc:493] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:49.306989  3663 raft_consensus.cc:515] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:49.008818  3823 raft_consensus.cc:3055] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 LEADER]: Stepping down as leader of term 2
W20260709 12:59:49.324944  1679 rpcz_store.cc:269] 0709 12:59:46.851924 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:47.683960 (+832036us) service_pool.cc:224] Handling call
0709 12:59:49.226347 (+1542387us) spinlock_profiling.cc:243] Waited 53.4 ms on lock 0x7b54000230d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 12:59:49.226529 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":122916096}
I20260709 12:59:49.146798  1753 rpcz_store.cc:276] 0709 12:59:45.427414 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:46.297527 (+870113us) service_pool.cc:224] Handling call
0709 12:59:46.297863 (+   336us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:59:46.298290 (+   427us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:59:46.298320 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:59:46.316011 (+ 17691us) log.cc:844] Serialized 26 byte log entry
0709 12:59:46.980172 (+664161us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:59:47.985674 (+1005502us) log.cc:844] Serialized 14 byte log entry
0709 12:59:48.859154 (+873480us) spinlock_profiling.cc:243] Waited 139 ms on lock 0x7b540010e8a0. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42660dd733 00007f4266162dbf 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c
0709 12:59:48.859264 (+   110us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:48.859408 (+   144us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:59:48.859487 (+    79us) raft_consensus.cc:1685] finished
0709 12:59:48.876712 (+ 17225us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:59:48.876984 (+   272us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":1021603,"prepare.run_cpu_time_us":70,"prepare.run_wall_time_us":9468,"raft.queue_time_us":1016181,"raft.run_cpu_time_us":363,"raft.run_wall_time_us":14936,"spinlock_wait_cycles":320643840,"thread_start_us":1416197,"threads_started":3,"wal-append.queue_time_us":1244150}
I20260709 12:59:49.303088  2336 raft_consensus.cc:3060] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:59:49.307802  3976 consensus_queue.cc:1059] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.214s
I20260709 12:59:48.719023  3622 leader_election.cc:290] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:49.343377  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:48.648357  3906 raft_consensus.cc:2804] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:59:49.000118  3970 raft_consensus.cc:3055] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [term 2 LEADER]: Stepping down as leader of term 2
W20260709 12:59:49.360800  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 178484us
I20260709 12:59:48.255463  3932 raft_consensus.cc:3060] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:59:49.364300  1678 raft_consensus.cc:1275] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:59:49.348572  1672 rpcz_store.cc:276] 0709 12:59:46.105444 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:46.432888 (+327444us) service_pool.cc:224] Handling call
0709 12:59:46.433346 (+   458us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:59:46.433866 (+   520us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:59:46.433893 (+    27us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:59:46.443216 (+  9323us) log.cc:844] Serialized 52 byte log entry
0709 12:59:47.774070 (+1330854us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:59:47.795085 (+ 21015us) log.cc:844] Serialized 14 byte log entry
0709 12:59:47.795313 (+   228us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:47.795398 (+    85us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:59:48.648428 (+853030us) raft_consensus.cc:1685] finished
0709 12:59:48.648473 (+    45us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:59:48.648633 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":49450,"prepare.queue_time_us":52443,"prepare.run_cpu_time_us":64,"prepare.run_wall_time_us":64,"raft.queue_time_us":83137,"raft.run_cpu_time_us":471,"raft.run_wall_time_us":47277,"thread_start_us":1286211,"threads_started":1,"wal-append.queue_time_us":1494927}
I20260709 12:59:48.722641  3816 leader_election.cc:290] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:49.152338  3952 raft_consensus.cc:3060] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:59:48.775595  3638 raft_consensus.cc:493] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 12:59:49.335088  3639 raft_consensus.cc:493] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 12:59:49.341892  3947 raft_consensus.cc:515] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 12:59:49.336723  3972 log.cc:927] Time spent T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.161s	user 0.000s	sys 0.000s
I20260709 12:59:49.387802  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:49.383891  3888 raft_consensus.cc:515] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:49.260017  3975 raft_consensus.cc:2804] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:59:49.303743  3919 raft_consensus.cc:493] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:48.417248  3964 consensus_queue.cc:1059] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 1.010s
I20260709 12:59:48.651981  3857 raft_consensus.cc:515] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:49.149215  3978 consensus_queue.cc:1059] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.246s
I20260709 12:59:49.400936  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:49.566862  1752 raft_consensus.cc:2393] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 4: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 12:59:49.333953  3958 raft_consensus.cc:493] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
W20260709 12:59:49.345881  2329 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39726 (request call id 695) took 1353 ms (client timeout 1781 ms). Trace:
I20260709 12:59:48.647675  3941 raft_consensus.cc:515] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:49.367449  3979 raft_consensus.cc:2804] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:59:49.177016  3977 raft_consensus.cc:515] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 12:59:49.396848  3982 log.cc:927] Time spent T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.425s	user 0.001s	sys 0.000s
I20260709 12:59:49.377353  3521 raft_consensus.cc:2749] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:59:49.420509  3956 raft_consensus.cc:515] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:49.421859  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:49.420516  1747 raft_consensus.cc:1240] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 3. Current term is 4. Ops: []
I20260709 12:59:49.600358  1753 raft_consensus.cc:2393] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 2: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 12:59:48.611950  3783 leader_election.cc:290] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 12:59:49.420949  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 84047.8us
I20260709 12:59:49.335587  3974 consensus_queue.cc:1059] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.157s
I20260709 12:59:48.650107  3846 consensus_queue.cc:260] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:49.398049  3944 raft_consensus.cc:493] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:49.391311  3986 raft_consensus.cc:493] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:49.594475  3979 raft_consensus.cc:493] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 12:59:49.443549  1673 raft_consensus.cc:2468] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 5.
I20260709 12:59:49.360957  3961 consensus_queue.cc:1059] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.084s
I20260709 12:59:49.446883  1718 leader_election.cc:304] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
W20260709 12:59:49.588891  2329 rpcz_store.cc:269] 0709 12:59:47.987179 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:48.022379 (+ 35200us) service_pool.cc:224] Handling call
0709 12:59:49.340042 (+1317663us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:59:49.451381  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 264909us
I20260709 12:59:49.695631  2301 leader_election.cc:304] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:59:49.696651  2301 leader_election.cc:304] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:59:49.411226  3969 raft_consensus.cc:493] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 12:59:49.366336  3915 raft_consensus.cc:515] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:49.419346  3823 raft_consensus.cc:740] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
W20260709 12:59:49.397665  1746 rpcz_store.cc:269] 0709 12:59:48.095788 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:48.098395 (+  2607us) service_pool.cc:224] Handling call
0709 12:59:49.331938 (+1233543us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:59:49.465874  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:49.147303  3953 raft_consensus.cc:515] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:49.474951  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:49.477789  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:49.575372  3919 raft_consensus.cc:515] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:49.725319  1751 raft_consensus.cc:2468] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 3.
I20260709 12:59:49.224572  3854 raft_consensus.cc:493] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:49.491938  2336 raft_consensus.cc:2468] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 4.
I20260709 12:59:49.449781  3906 raft_consensus.cc:493] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:59:49.387374  3966 raft_consensus.cc:515] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:49.377975  1603 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc"
dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
mode: GRACEFUL
new_leader_uuid: "3b879799899a499c8d409ee9f1277cd1"
 from {username='slave'} at 127.0.0.1:47752
W20260709 12:59:49.503379  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 105021us
I20260709 12:59:49.449726  3622 raft_consensus.cc:493] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:49.385529  3955 raft_consensus.cc:515] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:49.449189  3970 raft_consensus.cc:740] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
W20260709 12:59:49.512861  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 105189us
I20260709 12:59:49.592705  3958 raft_consensus.cc:515] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:49.412837  3921 raft_consensus.cc:515] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:49.570834  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:49.564502  1571 leader_election.cc:304] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:59:49.416441  3971 raft_consensus.cc:515] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:49.473170  3816 consensus_queue.cc:1048] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.103s
I20260709 12:59:49.535223  3975 raft_consensus.cc:493] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 12:59:49.418896  1750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55848 (request call id 1777) took 3166 ms. Trace:
I20260709 12:59:49.607820  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:49.483740  3639 raft_consensus.cc:515] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 12:59:49.613091  1717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:59:49.648535  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:49.842888  1752 raft_consensus.cc:3060] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:59:49.654932  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 50251.6us
W20260709 12:59:49.844728  1643 leader_election.cc:336] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 1.942s (SENT)
I20260709 12:59:49.845242  1643 leader_election.cc:304] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:49.475353  3638 raft_consensus.cc:515] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } }
I20260709 12:59:49.566337  3946 consensus_queue.cc:1048] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.049s
I20260709 12:59:49.574961  3800 consensus_queue.cc:1048] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.080s
I20260709 12:59:49.181851  3950 raft_consensus.cc:515] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:49.665338  3521 raft_consensus.cc:493] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 12:59:49.614912  3796 leader_election.cc:290] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:49.449215  1678 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57092 (request call id 472) took 1499 ms. Trace:
I20260709 12:59:49.674490  3976 raft_consensus.cc:493] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 12:59:49.725329  3783 raft_consensus.cc:2804] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260709 12:59:49.723951  3954 log.cc:927] Time spent T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.337s	user 0.000s	sys 0.001s
I20260709 12:59:49.867810  3783 raft_consensus.cc:493] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:59:49.868230  3783 raft_consensus.cc:3060] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:59:49.725010  1599 raft_consensus.cc:2468] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 4.
I20260709 12:59:49.729542  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:49.743219  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:49.871593  1751 raft_consensus.cc:2468] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 6.
I20260709 12:59:49.846756  3958 leader_election.cc:290] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:49.872439  3958 raft_consensus.cc:2749] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:59:49.872558  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:49.873246  1751 raft_consensus.cc:2468] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 4.
I20260709 12:59:49.873340  3947 leader_election.cc:290] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:49.840253  3999 raft_consensus.cc:493] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:49.729220  1646 leader_election.cc:304] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:59:49.874138  3999 raft_consensus.cc:515] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 12:59:49.874861  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 145761us
W20260709 12:59:49.731348  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 284525us
I20260709 12:59:49.691852  3979 raft_consensus.cc:3060] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:59:49.850940  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 12:59:49.771986  1644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:59:49.879702  1746 raft_consensus.cc:2468] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 4.
I20260709 12:59:49.879914  3663 leader_election.cc:290] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:49.880537  3663 raft_consensus.cc:2804] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:59:49.880697  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:49.880951  3663 raft_consensus.cc:493] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 12:59:49.705403  1597 raft_consensus.cc:1275] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:59:49.881348  3663 raft_consensus.cc:3060] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:59:49.881529  1746 raft_consensus.cc:2393] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 3: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 12:59:49.398186  3922 raft_consensus.cc:515] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:49.786237  2299 leader_election.cc:304] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 12:59:49.787971  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:49.723861  3787 raft_consensus.cc:2804] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:59:49.793434  1754 raft_consensus.cc:3060] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:59:49.796628  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:49.742388  3906 raft_consensus.cc:3060] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:59:49.883931  3998 raft_consensus.cc:2804] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:59:49.797343  1596 raft_consensus.cc:2468] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 3.
W20260709 12:59:49.793440  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 229050us
W20260709 12:59:49.801113  3983 log.cc:927] Time spent T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.204s	user 0.001s	sys 0.000s
W20260709 12:59:49.801419  3991 log.cc:927] Time spent T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.126s	user 0.000s	sys 0.001s
I20260709 12:59:49.724962  3969 raft_consensus.cc:515] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 12:59:49.802068  3985 log.cc:927] Time spent T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.227s	user 0.000s	sys 0.000s
W20260709 12:59:49.802868  3997 log.cc:927] Time spent T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.122s	user 0.000s	sys 0.001s
I20260709 12:59:49.795401  3993 consensus_queue.cc:1059] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.008s
I20260709 12:59:49.803299  4011 raft_consensus.cc:3055] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:59:49.804028  3816 raft_consensus.cc:3055] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:59:49.804581  3823 consensus_queue.cc:260] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:49.804652  3970 consensus_queue.cc:260] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:49.737584  3978 raft_consensus.cc:2955] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 4 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } } }
I20260709 12:59:49.807654  4008 raft_consensus.cc:493] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:49.764717  3854 raft_consensus.cc:515] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:49.810223  3919 leader_election.cc:290] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:49.730047  3961 consensus_queue.cc:1243] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer 3b879799899a499c8d409ee9f1277cd1 log is divergent from this leader: its last log entry 4.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
W20260709 12:59:49.804066  3987 log.cc:927] Time spent T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.291s	user 0.000s	sys 0.001s
I20260709 12:59:49.785562  3622 raft_consensus.cc:515] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:49.812566  1750 rpcz_store.cc:276] 0709 12:59:46.252206 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:46.328531 (+ 76325us) service_pool.cc:224] Handling call
0709 12:59:46.340083 (+ 11552us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:59:46.340561 (+   478us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:59:46.340601 (+    40us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:59:46.349106 (+  8505us) log.cc:844] Serialized 26 byte log entry
0709 12:59:47.766112 (+1417006us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:59:47.766180 (+    68us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:47.766289 (+   109us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:59:48.563755 (+797466us) spinlock_profiling.cc:243] Waited 22.7 ms on lock 0x7b4400022b4c. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f4262f9bc85 00007f4262f9c6a2 00007f4265d0e1db 00007f4265d09aaf 00007f4265cfd86e 00007f4265cfcbb6 00007f4265cfc7d8 00007f4265d06221 00007f4265d061d9 00007f4265d06169
0709 12:59:49.400842 (+837087us) raft_consensus.cc:1685] finished
0709 12:59:49.400877 (+    35us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:59:49.418803 (+ 17926us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":617,"raft.run_cpu_time_us":149,"raft.run_wall_time_us":1253,"spinlock_wait_cycles":52160896,"thread_start_us":1386214,"threads_started":1,"wal-append.queue_time_us":1736310}
I20260709 12:59:49.829555  3975 raft_consensus.cc:3060] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:59:49.728210  3948 leader_election.cc:290] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:49.814750  1753 raft_consensus.cc:2393] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 4: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 12:59:49.686693  3944 raft_consensus.cc:515] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:49.692911  3986 raft_consensus.cc:515] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:49.846750  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:49.854287  1678 rpcz_store.cc:276] 0709 12:59:47.950118 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:49.355000 (+1404882us) service_pool.cc:224] Handling call
0709 12:59:49.355388 (+   388us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:59:49.448830 (+ 93442us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:49.449135 (+   305us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:59:49.852677  1643 leader_election.cc:304] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 12:59:49.860548  4006 raft_consensus.cc:2804] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:59:49.861250  3638 leader_election.cc:290] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:49.861464  3857 leader_election.cc:290] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 6 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:49.858230  1749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55840 (request call id 2034) took 1033 ms. Trace:
I20260709 12:59:49.866890  3976 raft_consensus.cc:515] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:49.870939  3956 leader_election.cc:290] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:49.359790  1675 rpcz_store.cc:276] 0709 12:59:46.104333 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:46.210222 (+105889us) service_pool.cc:224] Handling call
0709 12:59:46.244601 (+ 34379us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:59:46.252557 (+  7956us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:59:46.252607 (+    50us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:59:46.253444 (+   837us) log.cc:844] Serialized 26 byte log entry
0709 12:59:47.802887 (+1549443us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:59:47.909930 (+107043us) log.cc:844] Serialized 14 byte log entry
0709 12:59:47.920936 (+ 11006us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:47.937779 (+ 16843us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:59:48.650539 (+712760us) raft_consensus.cc:1685] finished
0709 12:59:48.650571 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:59:48.650739 (+   168us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2330,"prepare.queue_time_us":90895,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":42,"raft.queue_time_us":97334,"raft.run_cpu_time_us":344,"raft.run_wall_time_us":16467,"thread_start_us":1502963,"threads_started":1,"wal-append.queue_time_us":1667886}
I20260709 12:59:49.878043  1718 leader_election.cc:304] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:49.884377  1672 raft_consensus.cc:2393] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 4: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 12:59:49.885527  1673 raft_consensus.cc:2393] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 4: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 12:59:49.886644  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 12:59:49.883917  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 97813.5us
I20260709 12:59:49.897373  1646 leader_election.cc:304] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:59:49.874086  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:50.248593  1747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43156 (request call id 398) took 2152 ms. Trace:
W20260709 12:59:50.188257  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 482175us
I20260709 12:59:50.368618  1747 rpcz_store.cc:276] 0709 12:59:48.096170 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:48.096327 (+   157us) service_pool.cc:224] Handling call
0709 12:59:48.554252 (+457925us) spinlock_profiling.cc:243] Waited 5.81 ms on lock 0x7b58001ffe98. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f426303db71 00007f426303da22 00007f426bf56c09 00007f426b0b0ee0 00007f426b17a0ab 00007f426b17a131 00007f426b13bfc0 00007f426b126929 00007f426b1263d2 00007f426469dbd4
0709 12:59:48.554634 (+   382us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:59:49.600219 (+1045585us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:50.248209 (+647990us) spinlock_profiling.cc:243] Waited 865 ms on lock 0x7b5400019ad4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 12:59:50.248503 (+   294us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2003973120}
I20260709 12:59:49.902437  1752 raft_consensus.cc:2468] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 6.
W20260709 12:59:50.045727  1679 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.512s	user 0.034s	sys 0.072s
I20260709 12:59:50.367309  1600 raft_consensus.cc:2468] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 3.
W20260709 12:59:50.007143  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 277584us
I20260709 12:59:50.295068  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:50.372033  1751 raft_consensus.cc:2468] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 2.
I20260709 12:59:49.912631  1571 leader_election.cc:304] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
W20260709 12:59:50.327232  1717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:59:50.370676  1679 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.856s	user 0.050s	sys 0.102s
I20260709 12:59:50.247056  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 12:59:50.008283  1745 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.588s	user 0.003s	sys 0.006s
W20260709 12:59:50.378374  1745 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.958s	user 0.011s	sys 0.012s
W20260709 12:59:50.379793  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 467173us
W20260709 12:59:50.382164  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 504122us
I20260709 12:59:50.324710  1677 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57092 (request call id 473) took 2062 ms. Trace:
I20260709 12:59:50.385039  1677 rpcz_store.cc:276] 0709 12:59:48.262127 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:49.406784 (+1144657us) service_pool.cc:224] Handling call
0709 12:59:49.407284 (+   500us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:59:49.444127 (+ 36843us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:59:49.444186 (+    59us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:59:49.444246 (+    60us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:59:49.444305 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:50.324235 (+879930us) spinlock_profiling.cc:243] Waited 36.5 ms on lock 0x7b54000596d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 12:59:50.324327 (+    92us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:59:50.324595 (+   268us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":84029824}
I20260709 12:59:50.387653  1569 leader_election.cc:304] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 12:59:50.391302  2301 leader_election.cc:304] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
W20260709 12:59:49.903731  3984 log.cc:927] Time spent T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.140s	user 0.002s	sys 0.001s
I20260709 12:59:50.394661  1745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55840 (request call id 2026) took 5141 ms. Trace:
I20260709 12:59:50.398455  1718 leader_election.cc:304] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:50.381171  1571 leader_election.cc:304] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:59:50.403364  1601 raft_consensus.cc:1275] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260709 12:59:49.882428  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:49.908862  3955 leader_election.cc:290] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:50.400179  3996 consensus_queue.cc:1059] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.037s
I20260709 12:59:50.415181  3955 raft_consensus.cc:2804] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:59:50.415632  3955 raft_consensus.cc:493] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 12:59:50.415998  3955 raft_consensus.cc:3060] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:59:50.416546  3896 consensus_meta.cc:287] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 2.467s	user 0.017s	sys 0.002s
I20260709 12:59:50.417214  3657 consensus_queue.cc:1048] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:59:49.884581  3787 raft_consensus.cc:493] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:59:50.190414  3989 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.677s	user 0.017s	sys 0.011s
I20260709 12:59:49.903604  4000 raft_consensus.cc:3055] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:59:50.406239  1674 raft_consensus.cc:1275] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260709 12:59:50.401234  1745 rpcz_store.cc:276] 0709 12:59:45.238264 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:46.189770 (+951506us) service_pool.cc:224] Handling call
0709 12:59:46.193045 (+  3275us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:59:46.193334 (+   289us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 12:59:47.690656 (+1497322us) log.cc:844] Serialized 14 byte log entry
0709 12:59:48.641411 (+950755us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:59:48.655108 (+ 13697us) log.cc:844] Serialized 26 byte log entry
0709 12:59:49.147920 (+492812us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:59:49.403531 (+255611us) log.cc:844] Serialized 14 byte log entry
0709 12:59:50.378813 (+975282us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:50.378912 (+    99us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:59:50.378967 (+    55us) raft_consensus.cc:1685] finished
0709 12:59:50.378979 (+    12us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:59:50.379125 (+   146us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":2002422,"prepare.run_cpu_time_us":107,"prepare.run_wall_time_us":109,"raft.queue_time_us":1135178,"raft.run_cpu_time_us":425,"raft.run_wall_time_us":37627,"thread_start_us":3600759,"threads_started":5,"wal-append.queue_time_us":1477940}
I20260709 12:59:49.907951  3950 leader_election.cc:290] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:49.910049  3998 raft_consensus.cc:493] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:59:49.909798  3888 leader_election.cc:290] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:50.409710  1603 raft_consensus.cc:606] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Received request to transfer leadership to TS 3b879799899a499c8d409ee9f1277cd1
I20260709 12:59:50.447369  1603 raft_consensus.cc:614] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:59:50.411726  3638 raft_consensus.cc:2749] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:59:50.263940  3921 leader_election.cc:290] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:50.412252  4014 consensus_queue.cc:1059] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:59:49.907232  3977 leader_election.cc:290] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:49.904462  3966 leader_election.cc:290] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:49.885193  3521 raft_consensus.cc:515] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:50.369633  4011 raft_consensus.cc:740] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 12:59:50.402710  3970 raft_consensus.cc:3060] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:59:49.913028  3958 raft_consensus.cc:493] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
W20260709 12:59:50.468216  3911 consensus_meta.cc:287] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 1.098s	user 0.012s	sys 0.004s
I20260709 12:59:49.905197  3800 raft_consensus.cc:2804] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:59:50.498883  3800 raft_consensus.cc:493] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:59:50.419580  1671 raft_consensus.cc:1275] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260709 12:59:49.909540  3953 leader_election.cc:290] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:49.911310  3915 leader_election.cc:290] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:50.148826  3639 leader_election.cc:290] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:50.375823  3816 raft_consensus.cc:740] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 12:59:50.434113  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:50.366580  4006 raft_consensus.cc:493] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 12:59:50.399109  3961 raft_consensus.cc:2804] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:59:50.398916  3919 raft_consensus.cc:2804] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:59:50.377904  3971 leader_election.cc:290] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:50.399300  3956 raft_consensus.cc:2804] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:59:50.395124  3948 raft_consensus.cc:2804] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260709 12:59:50.415642  4013 log.cc:927] Time spent T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.103s	user 0.001s	sys 0.000s
W20260709 12:59:50.412742  3981 log.cc:927] Time spent T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.737s	user 0.005s	sys 0.010s
I20260709 12:59:50.458948  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:50.257244  3922 leader_election.cc:290] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 12:59:50.423321  3946 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.520s	user 0.029s	sys 0.016s
W20260709 12:59:50.434911  3989 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.925s	user 0.022s	sys 0.011s
I20260709 12:59:50.436103  3787 raft_consensus.cc:3060] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:59:50.469177  1602 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 1767) took 2368 ms. Trace:
W20260709 12:59:50.471207  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 93868.7us
I20260709 12:59:50.489082  3958 raft_consensus.cc:515] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:50.472030  1676 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52778 (request call id 1660) took 3619 ms. Trace:
I20260709 12:59:50.441967  3823 raft_consensus.cc:3060] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:59:50.444012  3657 raft_consensus.cc:2804] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260709 12:59:50.475015  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 83825.3us
I20260709 12:59:50.441402  4000 raft_consensus.cc:740] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
W20260709 12:59:50.464558  3805 consensus_meta.cc:287] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 1.123s	user 0.015s	sys 0.000s
I20260709 12:59:50.067420  4009 raft_consensus.cc:493] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
W20260709 12:59:50.481088  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 628392us
I20260709 12:59:50.551433  1604 raft_consensus.cc:1275] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:59:50.551293  4000 consensus_queue.cc:260] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:50.551982  1643 leader_election.cc:304] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 12:59:50.485596  1748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55840 (request call id 2036) took 1662 ms. Trace:
I20260709 12:59:50.461051  3999 leader_election.cc:290] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:50.437352  3993 consensus_queue.cc:1048] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.038s
I20260709 12:59:50.552676  4000 raft_consensus.cc:3060] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:59:50.446956  3998 raft_consensus.cc:3060] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:59:50.467819  4001 raft_consensus.cc:3055] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:59:50.554214  4001 raft_consensus.cc:740] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
W20260709 12:59:50.555147  4019 log.cc:927] Time spent T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.060s	user 0.000s	sys 0.000s
I20260709 12:59:50.555655  4001 consensus_queue.cc:260] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:50.557217  4001 raft_consensus.cc:3060] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:59:50.481564  3932 consensus_meta.cc:287] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 1.011s	user 0.012s	sys 0.000s
I20260709 12:59:50.558816  3932 raft_consensus.cc:515] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:50.561374  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:50.561805  3932 leader_election.cc:290] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 4 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 12:59:50.495848  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 114766us
I20260709 12:59:50.458361  4015 consensus_queue.cc:1059] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.028s
I20260709 12:59:50.563035  3944 leader_election.cc:290] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:50.515245  3948 raft_consensus.cc:493] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 12:59:50.563702  4015 raft_consensus.cc:3055] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:59:50.477468  3854 leader_election.cc:290] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:50.564350  4015 raft_consensus.cc:740] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 12:59:50.477766  3977 raft_consensus.cc:2749] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:59:50.379199  4008 raft_consensus.cc:515] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:50.499413  3800 raft_consensus.cc:3060] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:59:50.552699  1748 rpcz_store.cc:276] 0709 12:59:48.823577 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:48.823724 (+   147us) service_pool.cc:224] Handling call
0709 12:59:48.824185 (+   461us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:59:48.824744 (+   559us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:59:48.824788 (+    44us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:59:48.866077 (+ 41289us) log.cc:844] Serialized 26 byte log entry
0709 12:59:49.614560 (+748483us) spinlock_profiling.cc:243] Waited 48.4 ms on lock 0x7b18003f90a0. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f4265d0a4bc 00007f426305848b 00007f426beb2ab9 00007f426307469b 00007f426307293f 00007f4263074c02 00007f4265cfc20b 00007f4265d01d23 00007f4265d020d1
0709 12:59:49.615133 (+   573us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:59:49.619327 (+  4194us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:49.619409 (+    82us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:59:50.472786 (+853377us) raft_consensus.cc:1685] finished
0709 12:59:50.472820 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:59:50.485518 (+ 12698us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":324433,"raft.run_cpu_time_us":283,"raft.run_wall_time_us":88006,"spinlock_wait_cycles":111262464,"thread_start_us":268319,"threads_started":1,"wal-append.queue_time_us":627297}
I20260709 12:59:50.568642  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:50.569104  4008 leader_election.cc:290] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 12:59:50.506615  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 112056us
I20260709 12:59:50.513612  3753 leader_election.cc:290] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 4 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:50.494659  3911 raft_consensus.cc:2749] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:59:50.512511  2332 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39726 (request call id 700) took 2409 ms. Trace:
I20260709 12:59:50.570241  3969 leader_election.cc:290] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:50.500033  3908 raft_consensus.cc:493] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:50.525940  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 5 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:50.571048  3908 raft_consensus.cc:515] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:50.529002  1603 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:47752 (request call id 15) took 1167 ms. Trace:
W20260709 12:59:50.536474  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 164678us
I20260709 12:59:50.515537  3956 raft_consensus.cc:493] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:59:50.572618  3956 raft_consensus.cc:3060] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:59:50.572355  1603 rpcz_store.cc:276] 0709 12:59:49.377647 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:49.377822 (+   175us) service_pool.cc:224] Handling call
0709 12:59:50.528681 (+1150859us) spinlock_profiling.cc:243] Waited 681 ms on lock 0x7b540027a1d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f426615c853 00007f426b12a60a 00007f42646b07d4 00007f42646b0762 00007f42646b0691 00007f42646b060c 00007f42646af8c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 12:59:50.528857 (+   176us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1565179136}
I20260709 12:59:50.534265  1602 rpcz_store.cc:276] 0709 12:59:48.093582 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:48.093719 (+   137us) service_pool.cc:224] Handling call
0709 12:59:48.569041 (+475322us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:59:48.569565 (+   524us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:59:48.569607 (+    42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:59:48.643734 (+ 74127us) log.cc:844] Serialized 26 byte log entry
0709 12:59:49.066572 (+422838us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:59:49.066632 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:49.066730 (+    98us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:59:50.460935 (+1394205us) raft_consensus.cc:1685] finished
0709 12:59:50.460971 (+    36us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:59:50.461174 (+   203us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":10415,"raft.queue_time_us":42334,"raft.run_cpu_time_us":278,"raft.run_wall_time_us":25490,"thread_start_us":422300,"threads_started":1,"wal-append.queue_time_us":830163}
I20260709 12:59:50.574827  1602 raft_consensus.cc:1275] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:59:50.576308  1602 raft_consensus.cc:1275] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:59:50.576732  3986 leader_election.cc:290] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:50.577920  3976 leader_election.cc:290] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:50.535945  1676 rpcz_store.cc:276] 0709 12:59:46.852863 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:47.764302 (+911439us) service_pool.cc:224] Handling call
0709 12:59:47.764723 (+   421us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:59:48.099826 (+335103us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:59:48.099860 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:59:48.567237 (+467377us) log.cc:844] Serialized 26 byte log entry
0709 12:59:49.060519 (+493282us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:59:49.060574 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:49.060674 (+   100us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:59:50.471689 (+1411015us) raft_consensus.cc:1685] finished
0709 12:59:50.471726 (+    37us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:59:50.471935 (+   209us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":35648,"raft.run_cpu_time_us":200,"raft.run_wall_time_us":13836,"thread_start_us":690081,"threads_started":2,"wal-append.queue_time_us":727132}
I20260709 12:59:50.578296  1602 raft_consensus.cc:3055] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:59:50.579819  1602 raft_consensus.cc:740] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 12:59:50.580828  1602 consensus_queue.cc:260] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:50.516260  3919 raft_consensus.cc:493] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 12:59:50.583086  3919 raft_consensus.cc:3060] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:59:50.583693  1602 raft_consensus.cc:3060] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:59:50.586529  4001 raft_consensus.cc:493] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:59:50.533823  3946 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.630s	user 0.029s	sys 0.016s
I20260709 12:59:50.543931  2301 leader_election.cc:304] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 12:59:50.509382  3961 raft_consensus.cc:493] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:59:50.542981  3657 raft_consensus.cc:493] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 12:59:50.546612  4016 consensus_queue.cc:1048] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
W20260709 12:59:50.487118  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 589760us
I20260709 12:59:50.556529  1679 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57092 (request call id 476) took 2141 ms. Trace:
I20260709 12:59:50.595844  1569 leader_election.cc:304] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 12:59:50.596179  1679 rpcz_store.cc:276] 0709 12:59:48.415000 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:49.419765 (+1004765us) service_pool.cc:224] Handling call
0709 12:59:49.420302 (+   537us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:59:49.420777 (+   475us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:59:49.420817 (+    40us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:59:49.442095 (+ 21278us) log.cc:844] Serialized 26 byte log entry
0709 12:59:49.612602 (+170507us) spinlock_profiling.cc:243] Waited 19.9 ms on lock 0x7b18004571e0. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f4263075557 00007f42630777c1 00007f4263077779 00007f4263077709 00007f42630776d1 00007f42630769cd 000000000038f354 000000000038f199 00007f426305883a
0709 12:59:50.373469 (+760867us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:59:50.373515 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:50.373572 (+    57us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:59:50.556202 (+182630us) raft_consensus.cc:1685] finished
0709 12:59:50.556225 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:59:50.556452 (+   227us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":28803,"raft.run_cpu_time_us":239,"raft.run_wall_time_us":35962,"spinlock_wait_cycles":45770624,"thread_start_us":859025,"threads_started":1,"wal-append.queue_time_us":984026}
I20260709 12:59:50.563786  3948 raft_consensus.cc:3060] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:59:50.506964  1598 raft_consensus.cc:2468] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 5.
I20260709 12:59:50.569388  1748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55848 (request call id 1796) took 1109 ms. Trace:
I20260709 12:59:50.519483  1750 raft_consensus.cc:2468] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 3.
I20260709 12:59:50.565606  4015 consensus_queue.cc:260] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:50.571897  1752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55848 (request call id 1781) took 1376 ms. Trace:
I20260709 12:59:50.601334  1752 rpcz_store.cc:276] 0709 12:59:49.195013 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:50.525533 (+1330520us) service_pool.cc:224] Handling call
0709 12:59:50.571828 (+ 46295us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:59:50.531584  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:50.563060  1571 leader_election.cc:304] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:59:50.510689  4006 raft_consensus.cc:3060] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:59:50.589551  1754 consensus_meta.cc:287] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.705s	user 0.005s	sys 0.009s
W20260709 12:59:50.591411  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 47580us
I20260709 12:59:50.570628  2332 rpcz_store.cc:276] 0709 12:59:48.098781 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:48.100380 (+  1599us) service_pool.cc:224] Handling call
0709 12:59:48.100816 (+   436us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:59:48.101371 (+   555us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:59:48.101408 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:59:48.643734 (+542326us) log.cc:844] Serialized 26 byte log entry
0709 12:59:48.874533 (+230799us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:59:49.082566 (+208033us) log.cc:844] Serialized 14 byte log entry
0709 12:59:49.151519 (+ 68953us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:49.151630 (+   111us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:59:49.724917 (+573287us) spinlock_profiling.cc:243] Waited 6.53 ms on lock 0x7b1800465040. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f42630753f3 00007f42630777c1 00007f4263077779 00007f4263077709 00007f42630776d1 00007f42630769cd 000000000038f354 000000000038f199 00007f426305883a
0709 12:59:49.903088 (+178171us) spinlock_profiling.cc:243] Waited 25.3 ms on lock 0x7b4400022b4c. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f4262f9bc85 00007f4262f9c6a2 00007f4265d0e1db 00007f4265d09aaf 00007f4265cfd86e 00007f4265cfcbb6 00007f4265cfc7d8 00007f4265d06221 00007f4265d061d9 00007f4265d06169
0709 12:59:50.508418 (+605330us) raft_consensus.cc:1685] finished
0709 12:59:50.508462 (+    44us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:59:50.508608 (+   146us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":233921,"prepare.run_cpu_time_us":92,"prepare.run_wall_time_us":22611,"raft.queue_time_us":291,"raft.run_cpu_time_us":459,"raft.run_wall_time_us":70841,"spinlock_wait_cycles":73154048,"thread_start_us":230269,"threads_started":1,"wal-append.queue_time_us":970403}
I20260709 12:59:50.599959  1602 raft_consensus.cc:1275] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
W20260709 12:59:50.604384  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 52498.7us
I20260709 12:59:50.605840  1643 leader_election.cc:304] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 12:59:50.599375  3657 raft_consensus.cc:3060] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:59:50.610714  3854 raft_consensus.cc:2749] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:59:50.538988  4011 consensus_queue.cc:260] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:50.611793  4011 raft_consensus.cc:3060] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:59:50.614929  4015 raft_consensus.cc:3060] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:59:50.615762  3958 leader_election.cc:290] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:50.618467  1646 leader_election.cc:304] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:59:50.604439  1754 raft_consensus.cc:2468] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 4.
I20260709 12:59:50.602041  1748 rpcz_store.cc:276] 0709 12:59:49.459756 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:50.568329 (+1108573us) service_pool.cc:224] Handling call
0709 12:59:50.569324 (+   995us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:59:50.624740  3805 consensus_queue.cc:1059] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:59:50.625772  3805 raft_consensus.cc:2749] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:59:50.627601  3805 raft_consensus.cc:3055] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:59:50.627970  3805 raft_consensus.cc:740] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 12:59:50.628822  3805 consensus_queue.cc:260] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:50.630117  3805 raft_consensus.cc:3060] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:59:50.631106  1569 leader_election.cc:304] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 12:59:50.551296  4009 raft_consensus.cc:515] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:50.636600  4029 raft_consensus.cc:2804] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:59:50.637051  4029 raft_consensus.cc:493] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:59:50.637430  4029 raft_consensus.cc:3060] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:59:50.643069  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 80146.5us
I20260709 12:59:50.644387  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:50.598663  3961 raft_consensus.cc:3060] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:59:50.668622  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:50.674167  1750 raft_consensus.cc:2393] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 5: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
W20260709 12:59:50.670230  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 51905us
I20260709 12:59:50.598317  3993 raft_consensus.cc:2749] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:59:50.675081  1646 leader_election.cc:304] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:59:50.610664  3622 leader_election.cc:290] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:50.681663  1646 leader_election.cc:304] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
W20260709 12:59:50.684227  1571 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:59:50.680012  2301 leader_election.cc:304] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
W20260709 12:59:50.694710  1571 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:59:50.671689  3783 consensus_meta.cc:287] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.803s	user 0.013s	sys 0.000s
W20260709 12:59:50.698990  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:59:50.699182  1646 leader_election.cc:304] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 12:59:50.587082  4001 raft_consensus.cc:515] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 12:59:50.706998  1569 connection.cc:256] RPC call timeout handler was delayed by -1.9040351410000085s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1978s for a response
I20260709 12:59:50.708179  3638 raft_consensus.cc:493] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:50.708629  3622 raft_consensus.cc:493] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:50.708767  3638 raft_consensus.cc:515] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:50.710062  3971 raft_consensus.cc:493] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:50.710793  3971 raft_consensus.cc:515] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 12:59:50.713340  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:59:50.715145  1571 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:59:50.715734  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 317333us
I20260709 12:59:50.709112  3950 raft_consensus.cc:493] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:50.719106  4026 raft_consensus.cc:2804] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:59:50.719830  4023 raft_consensus.cc:2749] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:59:50.680505  3976 raft_consensus.cc:2804] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:59:50.720438  4017 raft_consensus.cc:493] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:50.721182  4018 raft_consensus.cc:493] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:50.718546  1718 leader_election.cc:304] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:59:50.748379  1718 leader_election.cc:304] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 12:59:50.725133  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:59:50.724339  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:59:50.753015  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:50.754055  1603 raft_consensus.cc:2393] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 3: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 12:59:50.669063  3932 raft_consensus.cc:2749] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:59:50.663802  1597 raft_consensus.cc:2468] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 4.
I20260709 12:59:50.714550  3922 raft_consensus.cc:493] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:50.767951  1679 raft_consensus.cc:1240] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 2. Current term is 3. Ops: []
I20260709 12:59:50.720132  3638 leader_election.cc:290] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:50.727401  3783 raft_consensus.cc:515] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:50.712796  3854 raft_consensus.cc:2804] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:59:50.733616  3971 leader_election.cc:290] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:50.734324  3521 leader_election.cc:290] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:50.735596  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:50.738615  3896 raft_consensus.cc:493] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:50.709160  3622 raft_consensus.cc:515] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } }
I20260709 12:59:50.587826  3816 consensus_queue.cc:260] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 12:59:50.746308  2299 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:59:50.713498  3952 consensus_meta.cc:287] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 1.230s	user 0.016s	sys 0.000s
I20260709 12:59:50.755182  4015 raft_consensus.cc:2804] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:59:50.747115  3993 raft_consensus.cc:2804] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:59:50.755839  4027 consensus_queue.cc:1048] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.087s
I20260709 12:59:50.756650  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 12:59:50.730248  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 50331.1us
I20260709 12:59:50.760805  1752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55848 (request call id 1786) took 1419 ms. Trace:
I20260709 12:59:50.741989  4009 leader_election.cc:290] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:50.696389  3793 consensus_queue.cc:1048] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.062s
I20260709 12:59:50.782027  1676 raft_consensus.cc:2410] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20260709 12:59:50.782838  1718 connection.cc:256] RPC call timeout handler was delayed by -3.7573301239999637s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1559s for a response
W20260709 12:59:50.815521  1718 connection.cc:256] RPC call timeout handler was delayed by -3.652030123999964s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1676s for a response
W20260709 12:59:50.815902  1718 connection.cc:256] RPC call timeout handler was delayed by -3.6340301239999349s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1696s for a response
I20260709 12:59:50.782753  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 12:59:50.816196  1718 connection.cc:256] RPC call timeout handler was delayed by -3.4864301239999804s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18600000000000003s for a response
I20260709 12:59:50.777911  4018 raft_consensus.cc:515] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 12:59:50.820794  1718 connection.cc:256] RPC call timeout handler was delayed by -3.452230123999982s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1898s for a response
W20260709 12:59:50.821141  1718 connection.cc:256] RPC call timeout handler was delayed by -3.2929301239999518s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20750000000000002s for a response
W20260709 12:59:50.821451  1718 connection.cc:256] RPC call timeout handler was delayed by -3.2002301239999724s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2178s for a response
W20260709 12:59:50.821734  1718 connection.cc:256] RPC call timeout handler was delayed by -3.199330123999971s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21789999999999998s for a response
W20260709 12:59:50.822113  1718 connection.cc:256] RPC call timeout handler was delayed by -2.7070301239999708s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2726s for a response
W20260709 12:59:50.822454  1718 connection.cc:256] RPC call timeout handler was delayed by -1.9456301239999334s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3572s for a response
I20260709 12:59:50.784421  3953 raft_consensus.cc:493] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 12:59:50.368273  1749 rpcz_store.cc:276] 0709 12:59:48.822638 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:48.856899 (+ 34261us) service_pool.cc:224] Handling call
0709 12:59:49.002957 (+146058us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:59:49.003332 (+   375us) raft_consensus.cc:1534] Early marking committed up to index 2
0709 12:59:49.188915 (+185583us) log.cc:844] Serialized 14 byte log entry
0709 12:59:49.760216 (+571301us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:59:49.789785 (+ 29569us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:59:49.793241 (+  3456us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:49.831707 (+ 38466us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:59:49.855976 (+ 24269us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":358194,"prepare.run_cpu_time_us":54,"prepare.run_wall_time_us":58,"raft.queue_time_us":253347,"raft.run_cpu_time_us":188,"raft.run_wall_time_us":35678,"thread_start_us":576733,"threads_started":3,"wal-append.queue_time_us":370414}
I20260709 12:59:50.791610  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:50.757516  4026 raft_consensus.cc:493] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 12:59:50.770017  3976 raft_consensus.cc:697] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 12:59:50.838375  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:50.804417  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 12:59:50.805140  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:50.805351  3800 raft_consensus.cc:515] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:50.806098  2301 leader_election.cc:304] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 12:59:50.804445  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:50.809118  3993 raft_consensus.cc:697] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 4 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 12:59:50.808959  3955 raft_consensus.cc:515] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:50.809791  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:50.809990  3816 raft_consensus.cc:3060] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:59:50.853305  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 47347.2us
W20260709 12:59:50.810389  3979 consensus_meta.cc:287] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.932s	user 0.002s	sys 0.012s
I20260709 12:59:50.810786  4015 raft_consensus.cc:697] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 4 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 12:59:50.781152  3854 raft_consensus.cc:493] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:59:50.811456  4020 raft_consensus.cc:493] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:50.732626  3950 raft_consensus.cc:515] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:50.878839  3979 raft_consensus.cc:515] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:50.879527  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:50.881083  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:50.881233  3979 leader_election.cc:290] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:50.881898  2329 raft_consensus.cc:2393] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 5: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 12:59:50.801167  1752 rpcz_store.cc:276] 0709 12:59:49.341134 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:50.619467 (+1278333us) service_pool.cc:224] Handling call
0709 12:59:50.619932 (+   465us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:59:50.620213 (+   281us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:59:50.707726 (+ 87513us) log.cc:844] Serialized 14 byte log entry
0709 12:59:50.760386 (+ 52660us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:59:50.760445 (+    59us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:59:50.760510 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:50.760586 (+    76us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:59:50.760738 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":18116,"prepare.queue_time_us":134817,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":50,"raft.queue_time_us":18246,"raft.run_wall_time_us":3152,"spinlock_wait_cycles":8205696,"thread_start_us":77981,"threads_started":2,"wal-append.queue_time_us":65718}
I20260709 12:59:50.882923  3800 leader_election.cc:290] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 6 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:50.883311  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:50.883978  1752 raft_consensus.cc:2468] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 4.
I20260709 12:59:50.884878  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:50.885432  1752 raft_consensus.cc:2468] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 4.
I20260709 12:59:50.823007  1674 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57092 (request call id 471) took 2908 ms. Trace:
I20260709 12:59:50.887053  3980 raft_consensus.cc:3055] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:59:50.886796  1674 rpcz_store.cc:276] 0709 12:59:47.914482 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:49.329261 (+1414779us) service_pool.cc:224] Handling call
0709 12:59:49.329582 (+   321us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:59:50.441527 (+1111945us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:50.818000 (+376473us) spinlock_profiling.cc:243] Waited 1.08 s on lock 0x7b54001e82d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 12:59:50.822903 (+  4903us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2474995584}
I20260709 12:59:50.887743  3980 raft_consensus.cc:740] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 12:59:50.823650  1718 leader_election.cc:304] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:50.783929  3977 raft_consensus.cc:493] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 12:59:50.738184  3948 raft_consensus.cc:515] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:50.787237  4031 raft_consensus.cc:2749] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:59:50.840571  2337 raft_consensus.cc:2468] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 3.
I20260709 12:59:50.838375  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:50.842072  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:59:50.897179  1603 raft_consensus.cc:3060] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:59:50.897243  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:59:50.843624  4024 raft_consensus.cc:2804] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:59:50.757678  4017 raft_consensus.cc:515] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:50.803475  3952 raft_consensus.cc:515] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:50.773988  3922 raft_consensus.cc:515] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:50.793256  3896 raft_consensus.cc:515] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:50.852396  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:50.863181  2334 raft_consensus.cc:1240] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 2 LEADER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 1. Current term is 2. Ops: []
I20260709 12:59:50.877617  3854 raft_consensus.cc:3060] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:59:50.888638  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:50.886205  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:50.859364  1748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55848 (request call id 1788) took 1500 ms. Trace:
W20260709 12:59:50.893798  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 70206.6us
W20260709 12:59:50.922812  1570 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:59:50.922959  1752 raft_consensus.cc:2393] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 4: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 12:59:50.923509  1718 leader_election.cc:304] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:50.842799  4026 raft_consensus.cc:3060] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:59:50.922896  1674 raft_consensus.cc:2410] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for term 4 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260709 12:59:50.898128  1677 raft_consensus.cc:3060] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:59:50.925505  1750 raft_consensus.cc:2410] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20260709 12:59:50.836920  3906 consensus_meta.cc:287] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.933s	user 0.011s	sys 0.002s
W20260709 12:59:50.845171  3663 consensus_meta.cc:287] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.963s	user 0.008s	sys 0.006s
I20260709 12:59:50.844731  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:50.922689  1748 rpcz_store.cc:276] 0709 12:59:49.359256 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:50.620867 (+1261611us) service_pool.cc:224] Handling call
0709 12:59:50.621247 (+   380us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:59:50.621532 (+   285us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 12:59:50.732527 (+110995us) log.cc:844] Serialized 14 byte log entry
0709 12:59:50.858888 (+126361us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:59:50.858965 (+    77us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:59:50.859042 (+    77us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:50.859134 (+    92us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:59:50.859286 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":43179,"prepare.queue_time_us":133786,"prepare.run_cpu_time_us":28,"prepare.run_wall_time_us":29,"raft.queue_time_us":112,"raft.run_cpu_time_us":163,"raft.run_wall_time_us":7968,"thread_start_us":164112,"threads_started":2,"wal-append.queue_time_us":111996}
I20260709 12:59:50.939539  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 12:59:50.933405  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:59:50.941843  1751 raft_consensus.cc:2468] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 4.
I20260709 12:59:50.857260  4027 raft_consensus.cc:2749] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:59:50.939539  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:50.939262  3906 raft_consensus.cc:515] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:50.949301  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:50.950031  3906 leader_election.cc:290] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 6 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:50.950523  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:50.951655  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:50.953285  1718 leader_election.cc:304] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:50.941376  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 12:59:50.923664  1570 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:59:50.955241  1570 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:59:50.942050  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:59:50.940239  4025 consensus_queue.cc:1059] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.052s
I20260709 12:59:50.940251  3663 raft_consensus.cc:515] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:50.947855  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:59:50.840814  3998 raft_consensus.cc:515] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 12:59:50.957538  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 126157us
I20260709 12:59:50.974298  3922 leader_election.cc:290] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:50.975385  3896 leader_election.cc:290] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 12:59:50.975507  4032 log.cc:927] Time spent T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.073s	user 0.000s	sys 0.000s
I20260709 12:59:50.955341  1718 leader_election.cc:304] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
W20260709 12:59:50.955861  1570 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:59:50.977114  1570 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:59:50.978552  1718 leader_election.cc:304] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:50.958158  1674 raft_consensus.cc:3060] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:59:50.979923  1718 leader_election.cc:304] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:50.896071  3979 raft_consensus.cc:2749] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:59:50.845619  3953 raft_consensus.cc:515] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:50.961108  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:50.962314  1677 raft_consensus.cc:2468] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 5.
I20260709 12:59:50.959417  1680 raft_consensus.cc:1240] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20260709 12:59:50.965785  3846 raft_consensus.cc:3060] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:59:50.970785  4025 raft_consensus.cc:3055] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:59:50.971392  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:50.960872  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:50.952329  3521 raft_consensus.cc:2804] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:59:50.953217  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:50.990265  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:50.971711  1603 raft_consensus.cc:2468] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 5.
I20260709 12:59:50.941329  4024 raft_consensus.cc:493] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:59:50.960050  3970 raft_consensus.cc:2749] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:59:50.993702  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:50.959466  3622 leader_election.cc:290] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:50.997920  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:50.990653  2331 raft_consensus.cc:2368] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 5. Current term is 6.
I20260709 12:59:51.011605  1749 raft_consensus.cc:1240] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 4. Current term is 5. Ops: []
I20260709 12:59:51.012290  1749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55840 (request call id 2038) took 1310 ms. Trace:
I20260709 12:59:50.968359  3986 raft_consensus.cc:493] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 12:59:50.987679  4031 raft_consensus.cc:2749] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:59:51.015101  3986 raft_consensus.cc:515] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:51.019804  3948 leader_election.cc:290] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 4 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:51.022090  3663 leader_election.cc:290] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 4 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:51.023038  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 12:59:51.024279  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:51.025002  2335 raft_consensus.cc:2393] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 4: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 12:59:51.025683  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:51.003738  3521 raft_consensus.cc:493] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 12:59:51.026778  3521 raft_consensus.cc:3060] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:59:51.027669  1751 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55840 (request call id 2041) took 1325 ms. Trace:
I20260709 12:59:51.028060  1751 rpcz_store.cc:276] 0709 12:59:49.702227 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:50.561049 (+858822us) service_pool.cc:224] Handling call
0709 12:59:51.027498 (+466449us) spinlock_profiling.cc:243] Waited 380 ms on lock 0x7b54001d51d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 12:59:51.027605 (+   107us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":873314048}
W20260709 12:59:51.033053  4035 log.cc:927] Time spent T 00000000000000000000000000000000 P 919f6392d4f44fd68987e2c5e7ec37ca: Append to log took a long time: real 0.365s	user 0.004s	sys 0.006s
I20260709 12:59:50.983558  1597 raft_consensus.cc:2376] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 4. Re-sending same reply.
I20260709 12:59:50.975553  3787 raft_consensus.cc:515] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:50.992165  3753 raft_consensus.cc:2804] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:59:50.977715  3956 raft_consensus.cc:515] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:50.958525  4006 raft_consensus.cc:515] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:50.926055  4020 raft_consensus.cc:515] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:50.901113  3977 raft_consensus.cc:515] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:51.012610  1749 rpcz_store.cc:276] 0709 12:59:49.701694 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:51.010986 (+1309292us) service_pool.cc:224] Handling call
0709 12:59:51.011389 (+   403us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:59:51.012018 (+   629us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:51.012219 (+   201us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:59:51.034721  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:51.008150  1748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55848 (request call id 1798) took 1509 ms. Trace:
I20260709 12:59:50.993161  4025 raft_consensus.cc:740] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 12:59:50.980326  3980 consensus_queue.cc:260] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:51.033594  3961 raft_consensus.cc:515] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:50.973578  3976 consensus_queue.cc:237] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 12:59:51.251299  2299 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:59:51.092760  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:51.124686  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:51.129194  2332 raft_consensus.cc:2368] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 3. Current term is 4.
I20260709 12:59:51.255064  1752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55848 (request call id 1799) took 1731 ms. Trace:
I20260709 12:59:51.103447  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 12:59:51.195253  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:59:51.255482  1752 rpcz_store.cc:276] 0709 12:59:49.523899 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:50.981300 (+1457401us) service_pool.cc:224] Handling call
0709 12:59:50.982772 (+  1472us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:59:50.983021 (+   249us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 12:59:50.992643 (+  9622us) log.cc:844] Serialized 14 byte log entry
0709 12:59:51.254641 (+261998us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:59:51.254714 (+    73us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:59:51.254792 (+    78us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:51.254872 (+    80us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:59:51.254999 (+   127us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":10517,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":55,"thread_start_us":221000,"threads_started":1}
W20260709 12:59:51.256948  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:59:51.257632  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:59:51.257895  1752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55848 (request call id 1801) took 1364 ms. Trace:
I20260709 12:59:51.254989  1748 rpcz_store.cc:276] 0709 12:59:49.510645 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:50.956226 (+1445581us) service_pool.cc:224] Handling call
0709 12:59:50.965734 (+  9508us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:59:50.965964 (+   230us) raft_consensus.cc:1534] Early marking committed up to index 3
0709 12:59:50.966266 (+   302us) log.cc:844] Serialized 14 byte log entry
0709 12:59:51.007313 (+ 41047us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:59:51.007360 (+    47us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:59:51.007404 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:51.007967 (+   563us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:59:51.008088 (+   121us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":925,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":47,"raft.queue_time_us":2412,"raft.run_cpu_time_us":157,"raft.run_wall_time_us":79173,"thread_start_us":39240,"threads_started":1,"wal-append.queue_time_us":210948}
W20260709 12:59:51.248908  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 269063us
I20260709 12:59:51.265192  1718 leader_election.cc:304] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:51.267514  1718 leader_election.cc:304] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:51.006798  4018 leader_election.cc:290] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:51.271569  4011 raft_consensus.cc:2749] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20260709 12:59:51.143539  3657 consensus_meta.cc:287] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.536s	user 0.014s	sys 0.000s
W20260709 12:59:51.116364  4000 consensus_meta.cc:287] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.560s	user 0.009s	sys 0.003s
W20260709 12:59:51.186980  3919 consensus_meta.cc:287] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.603s	user 0.007s	sys 0.005s
W20260709 12:59:51.179667  3978 consensus_meta.cc:287] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.685s	user 0.011s	sys 0.000s
I20260709 12:59:51.180816  3952 leader_election.cc:290] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 6 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:51.258355  1752 rpcz_store.cc:276] 0709 12:59:49.893785 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:51.257000 (+1363215us) service_pool.cc:224] Handling call
0709 12:59:51.257349 (+   349us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:59:51.257557 (+   208us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:59:51.257580 (+    23us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:59:51.257620 (+    40us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:59:51.257664 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:51.257718 (+    54us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:59:51.257837 (+   119us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:59:51.247653  1644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:59:51.291991  1752 raft_consensus.cc:1240] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 1. Current term is 2. Ops: []
I20260709 12:59:51.007618  4024 raft_consensus.cc:3060] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:59:51.013981  2330 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49082 (request call id 552) took 3065 ms. Trace:
I20260709 12:59:51.244782  3638 raft_consensus.cc:493] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 12:59:51.181532  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:51.127369  3986 leader_election.cc:290] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:51.114106  3783 leader_election.cc:290] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 5 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:51.179652  3953 leader_election.cc:290] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 12:59:51.362811  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:59:51.330883  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 63459us
I20260709 12:59:51.331998  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 12:59:51.180574  3823 consensus_meta.cc:287] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.593s	user 0.009s	sys 0.003s
I20260709 12:59:51.293758  1644 leader_election.cc:304] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 12:59:51.284900  1746 raft_consensus.cc:3060] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:59:51.270946  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:50.997582  3979 raft_consensus.cc:2749] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:59:51.275357  3980 raft_consensus.cc:3060] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:59:51.272569  3753 raft_consensus.cc:697] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 3 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 12:59:51.008399  3993 consensus_queue.cc:237] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:51.361205  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:51.364734  1600 raft_consensus.cc:2393] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 4: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 12:59:51.368063  1718 leader_election.cc:304] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1
W20260709 12:59:51.368759  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 75095.3us
I20260709 12:59:51.013913  4015 consensus_queue.cc:237] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:51.371801  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
W20260709 12:59:51.392658  1570 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:59:51.364709  3805 consensus_meta.cc:287] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.698s	user 0.011s	sys 0.003s
I20260709 12:59:51.354671  3948 raft_consensus.cc:2804] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:59:51.409490  1718 leader_election.cc:304] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 12:59:51.388680  1644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:59:51.390949  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:51.286535  3657 raft_consensus.cc:515] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:51.372198  3952 raft_consensus.cc:3060] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:59:51.287874  3919 raft_consensus.cc:515] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:51.354671  3638 raft_consensus.cc:515] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 12:59:51.396190  1754 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55848 (request call id 1784) took 2131 ms (client timeout 1605 ms). Trace:
I20260709 12:59:51.403666  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:51.384070  1749 raft_consensus.cc:1240] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 3. Current term is 4. Ops: []
W20260709 12:59:51.426410  1754 rpcz_store.cc:269] 0709 12:59:49.264571 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:49.305357 (+ 40786us) service_pool.cc:224] Handling call
0709 12:59:51.395971 (+2090614us) spinlock_profiling.cc:243] Waited 441 ms on lock 0x7b54000e4f54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 12:59:51.396097 (+   126us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1015220608}
W20260709 12:59:51.426875  1599 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36462 (request call id 374) took 1618 ms (client timeout 1577 ms). Trace:
I20260709 12:59:51.427165  1749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55848 (request call id 1802) took 1533 ms. Trace:
I20260709 12:59:51.414404  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:51.427624  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
W20260709 12:59:51.427225  1599 rpcz_store.cc:269] 0709 12:59:49.808377 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:49.870254 (+ 61877us) service_pool.cc:224] Handling call
0709 12:59:51.403439 (+1533185us) spinlock_profiling.cc:243] Waited 1.56 ms on lock 0x7b5800003698. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f426303db71 00007f426303da22 00007f426bf56c09 00007f426b0b0ee0 00007f426b17a0ab 00007f426b17a131 00007f426b1280a0 00007f426b127984 00007f42646a26d4 00007f42646a2662
0709 12:59:51.426794 (+ 23355us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3585792}
I20260709 12:59:51.428428  1754 raft_consensus.cc:3060] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:59:51.428308  1596 raft_consensus.cc:2468] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 5.
I20260709 12:59:51.428972  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:51.429557  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:59:51.414245  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:51.425577  1718 leader_election.cc:304] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:59:51.431892  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:59:51.429598  1599 raft_consensus.cc:2468] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 4.
I20260709 12:59:51.441830  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:51.441825  3753 consensus_queue.cc:237] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:51.443473  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:51.445024  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:51.445693  1599 raft_consensus.cc:2468] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 3.
I20260709 12:59:51.446053  3787 leader_election.cc:290] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 4 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:51.446841  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:51.447780  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 12:59:51.448168  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:51.449050  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 12:59:51.449532  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 12:59:51.447268  4048 log.cc:927] Time spent T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.057s	user 0.000s	sys 0.000s
I20260709 12:59:51.450210  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:51.461828  1754 raft_consensus.cc:2468] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 5.
I20260709 12:59:51.473616  1674 raft_consensus.cc:2468] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 6.
I20260709 12:59:51.473748  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:51.474756  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:51.475520  1674 raft_consensus.cc:2468] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 2.
I20260709 12:59:51.476594  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:51.421818  3979 raft_consensus.cc:2804] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:59:51.420003  4011 raft_consensus.cc:3060] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:59:51.446818  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:51.421639  3948 raft_consensus.cc:697] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 2 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 12:59:51.447671  3956 leader_election.cc:290] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 5 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 12:59:51.451346  3975 consensus_meta.cc:287] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 1.052s	user 0.007s	sys 0.007s
I20260709 12:59:51.450760  3854 raft_consensus.cc:515] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:51.487536  4006 leader_election.cc:290] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 12:59:51.490497  4029 consensus_meta.cc:287] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.853s	user 0.005s	sys 0.009s
I20260709 12:59:51.495258  4020 leader_election.cc:290] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:51.495788  3961 leader_election.cc:290] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 5 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 12:59:51.497928  4026 consensus_meta.cc:287] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.546s	user 0.010s	sys 0.000s
W20260709 12:59:51.500008  3816 consensus_meta.cc:287] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.611s	user 0.010s	sys 0.000s
I20260709 12:59:51.500552  3977 leader_election.cc:290] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 12:59:51.504479  3846 consensus_meta.cc:287] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.509s	user 0.009s	sys 0.000s
I20260709 12:59:51.572211  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:59:51.617028  3638 leader_election.cc:290] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:51.617259  1600 raft_consensus.cc:2376] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 6. Re-sending same reply.
I20260709 12:59:51.618765  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:51.620289  1680 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52766 (request call id 2043) took 3539 ms. Trace:
I20260709 12:59:51.620793  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:51.620748  1680 rpcz_store.cc:276] 0709 12:59:48.080931 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:49.399800 (+1318869us) service_pool.cc:224] Handling call
0709 12:59:49.421067 (+ 21267us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:59:50.990803 (+1569736us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:51.619588 (+628785us) spinlock_profiling.cc:243] Waited 1.54 s on lock 0x7b54000fda54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 12:59:51.620066 (+   478us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3536787328}
I20260709 12:59:51.624140  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:51.625276  1747 raft_consensus.cc:2468] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 3.
I20260709 12:59:51.628083  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 12:59:51.632607  2336 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39726 (request call id 694) took 3673 ms (client timeout 2514 ms). Trace:
W20260709 12:59:51.633302  2336 rpcz_store.cc:269] 0709 12:59:47.958783 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:47.958947 (+   164us) service_pool.cc:224] Handling call
0709 12:59:51.632059 (+3673112us) spinlock_profiling.cc:243] Waited 1.27 s on lock 0x7b54007d2654. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 12:59:51.632307 (+   248us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2918918528}
I20260709 12:59:51.637153  3978 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.BulkChangeConfig from 127.0.0.1:43150 (request call id 13) took 21683 ms. Trace:
I20260709 12:59:51.639930  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:51.640936  1753 raft_consensus.cc:2468] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 4.
I20260709 12:59:51.620842  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:51.644794  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 12:59:51.645828  2328 raft_consensus.cc:2368] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 3. Current term is 4.
I20260709 12:59:51.646828  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:51.648589  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:51.649752  1679 raft_consensus.cc:2393] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 4: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 12:59:51.529208  4029 raft_consensus.cc:515] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:51.652575  1596 raft_consensus.cc:2393] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 6: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
W20260709 12:59:51.573973  1570 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:59:51.450179  1599 raft_consensus.cc:2468] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 4.
I20260709 12:59:51.574219  1602 raft_consensus.cc:2468] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 2.
I20260709 12:59:51.577039  1677 raft_consensus.cc:2393] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 4: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 12:59:51.478534  1672 raft_consensus.cc:1240] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 2. Current term is 3. Ops: []
W20260709 12:59:51.577970  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 169622us
I20260709 12:59:51.579953  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:51.450606  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:51.657254  1603 raft_consensus.cc:2468] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 4.
I20260709 12:59:51.671641  1674 raft_consensus.cc:2393] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 5: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 12:59:51.441099  1749 rpcz_store.cc:276] 0709 12:59:49.894043 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:51.272834 (+1378791us) service_pool.cc:224] Handling call
0709 12:59:51.371082 (+ 98248us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:59:51.426778 (+ 55696us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:51.427081 (+   303us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:59:51.316005  2330 rpcz_store.cc:276] 0709 12:59:47.948358 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:47.950604 (+  2246us) service_pool.cc:224] Handling call
0709 12:59:48.079506 (+128902us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:59:48.099350 (+ 19844us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:59:48.117594 (+ 18244us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:59:48.649675 (+532081us) log.cc:844] Serialized 26 byte log entry
0709 12:59:49.030745 (+381070us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:59:49.047727 (+ 16982us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:49.062589 (+ 14862us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:59:50.855307 (+1792718us) spinlock_profiling.cc:243] Waited 17.7 ms on lock 0x7b240034ad68. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f4262efb9c4 00007f4262ef9acb 00007f42630b4dc9 00007f42630b4a55 00007f42630b3fb0 00007f4265d2a488 00007f4265d2abe8 00007f4265d02398 00007f4265cfd76a 00007f4265cfcbb6
0709 12:59:50.968299 (+112992us) raft_consensus.cc:1685] finished
0709 12:59:50.968330 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:59:51.009116 (+ 40786us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5510,"raft.queue_time_us":7911,"raft.run_cpu_time_us":200,"raft.run_wall_time_us":33268,"spinlock_wait_cycles":40626560,"thread_start_us":242043,"threads_started":1,"wal-append.queue_time_us":775002}
I20260709 12:59:51.457680  3969 raft_consensus.cc:2804] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:59:51.674458  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:51.674971  3969 raft_consensus.cc:493] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 12:59:51.675448  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 12:59:51.675705  3969 raft_consensus.cc:3060] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:59:51.676847  2330 raft_consensus.cc:2368] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 4. Current term is 5.
I20260709 12:59:51.677139  1718 leader_election.cc:400] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 election: Vote denied by peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625) with higher term. Message: Invalid argument: T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 3. Current term is 4.
I20260709 12:59:51.526921  3975 raft_consensus.cc:515] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:51.679672  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:51.680564  3975 leader_election.cc:290] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 5 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:51.680797  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:51.481609  3979 raft_consensus.cc:493] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 12:59:51.681397  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:51.483721  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:51.682832  1674 raft_consensus.cc:2468] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 3.
I20260709 12:59:51.587962  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 12:59:51.601649  4053 log.cc:927] Time spent T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.060s	user 0.000s	sys 0.001s
I20260709 12:59:51.604575  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 12:59:51.590261  3854 leader_election.cc:290] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 4 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:51.604908  1604 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 366) took 1799 ms. Trace:
I20260709 12:59:51.600987  3948 consensus_queue.cc:237] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:51.527861  3521 raft_consensus.cc:515] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:51.706416  3521 leader_election.cc:290] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:51.613544  3919 leader_election.cc:290] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 3 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:51.707933  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:51.629936  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:51.709662  1748 raft_consensus.cc:2468] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 1.
I20260709 12:59:51.711004  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:51.711803  1748 raft_consensus.cc:3060] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:59:51.642163  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:51.715235  1753 raft_consensus.cc:2393] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 5: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 12:59:51.716303  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:51.716948  1753 raft_consensus.cc:3060] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:59:51.655809  1570 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:59:51.719007  1570 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:59:51.656421  1750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55840 (request call id 2044) took 1005 ms. Trace:
I20260709 12:59:51.655263  4020 raft_consensus.cc:493] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 12:59:51.720973  1750 rpcz_store.cc:276] 0709 12:59:50.650725 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:50.726128 (+ 75403us) service_pool.cc:224] Handling call
0709 12:59:51.655916 (+929788us) spinlock_profiling.cc:243] Waited 122 ms on lock 0x7b54003705d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 12:59:51.656172 (+   256us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":280291072}
I20260709 12:59:51.721398  4020 raft_consensus.cc:515] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:51.722640  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:51.723469  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:51.723945  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:51.657266  4015 raft_consensus.cc:493] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:51.660379  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:59:51.724408  1747 raft_consensus.cc:2468] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 4.
I20260709 12:59:51.724815  4015 raft_consensus.cc:515] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:51.666497  3657 leader_election.cc:290] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 4 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:51.690292  3955 leader_election.cc:290] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 7 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:51.671792  2329 raft_consensus.cc:2393] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 4: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 12:59:51.675877  1597 raft_consensus.cc:2393] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 4: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 12:59:51.727286  1597 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40154 (request call id 1733) took 1009 ms. Trace:
I20260709 12:59:51.662575  3941 leader_election.cc:290] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:51.683342  1754 raft_consensus.cc:3060] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Advancing to term 6
I20260709 12:59:51.727749  1597 rpcz_store.cc:276] 0709 12:59:50.717642 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:51.673648 (+956006us) service_pool.cc:224] Handling call
0709 12:59:51.727188 (+ 53540us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:59:51.684273  1678 raft_consensus.cc:1240] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 4. Current term is 5. Ops: []
I20260709 12:59:51.384747  4025 consensus_queue.cc:260] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:51.681972  3979 raft_consensus.cc:3060] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:59:51.685055  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:51.532343  4026 raft_consensus.cc:515] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:51.737960  1601 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 359) took 2242 ms. Trace:
I20260709 12:59:51.738395  1598 raft_consensus.cc:1240] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260709 12:59:51.739343  4026 leader_election.cc:290] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 3 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:51.669925  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:59:51.740710  1680 raft_consensus.cc:3060] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:59:51.741029  1598 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 1778) took 1096 ms. Trace:
I20260709 12:59:51.741433  1598 rpcz_store.cc:276] 0709 12:59:50.644241 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:50.644490 (+   249us) service_pool.cc:224] Handling call
0709 12:59:51.737632 (+1093142us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:59:51.739166 (+  1534us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:51.740809 (+  1643us) spinlock_profiling.cc:243] Waited 1.05 s on lock 0x7b54002a1cd0. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661618ad 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 12:59:51.740963 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2424534784}
I20260709 12:59:51.687250  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:51.742841  2336 raft_consensus.cc:2393] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 4: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 12:59:51.743583  1605 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 355) took 2291 ms. Trace:
I20260709 12:59:51.744019  1605 rpcz_store.cc:276] 0709 12:59:49.452347 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:49.452618 (+   271us) service_pool.cc:224] Handling call
0709 12:59:49.453153 (+   535us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:59:50.458922 (+1005769us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:59:50.458970 (+    48us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:59:50.459651 (+   681us) log.cc:844] Serialized 26 byte log entry
0709 12:59:50.664244 (+204593us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:59:50.664304 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:51.742753 (+1078449us) spinlock_profiling.cc:243] Waited 995 ms on lock 0x7b540027a1d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 12:59:51.742845 (+    92us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:59:51.742929 (+    84us) raft_consensus.cc:1685] finished
0709 12:59:51.742952 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:59:51.743348 (+   396us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":17975,"raft.queue_time_us":10428,"raft.run_cpu_time_us":173,"raft.run_wall_time_us":11725,"spinlock_wait_cycles":2287850752,"thread_start_us":175033,"threads_started":1,"wal-append.queue_time_us":308642}
I20260709 12:59:51.687371  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 12:59:51.746083  4051 log.cc:927] Time spent T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.314s	user 0.004s	sys 0.000s
W20260709 12:59:51.692227  3974 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 4 current term: 4 is higher.
W20260709 12:59:51.693488  2299 leader_election.cc:336] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Timed out: RequestConsensusVote RPC to 127.0.107.194:46261 timed out after 1.906s (SENT)
I20260709 12:59:51.747656  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 12:59:51.748075  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 54916.2us
I20260709 12:59:51.693845  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:51.748406  1751 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55840 (request call id 2040) took 2046 ms. Trace:
I20260709 12:59:51.748855  1598 raft_consensus.cc:2410] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for term 6 because replica has last-logged OpId of term: 4 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20260709 12:59:51.749128  1675 raft_consensus.cc:2393] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 5: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 12:59:51.688252  4029 leader_election.cc:290] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 6 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:51.749534  1598 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40166 (request call id 1789) took 1101 ms. Trace:
I20260709 12:59:51.749943  1598 rpcz_store.cc:276] 0709 12:59:50.648404 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:51.747218 (+1098814us) service_pool.cc:224] Handling call
0709 12:59:51.749451 (+  2233us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:59:51.698415  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:59:51.754917  1599 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40166 (request call id 1790) took 1097 ms. Trace:
I20260709 12:59:51.700853  4017 leader_election.cc:290] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:51.755344  1599 rpcz_store.cc:276] 0709 12:59:50.648664 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:51.686887 (+1038223us) service_pool.cc:224] Handling call
0709 12:59:51.746373 (+ 59486us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:59:51.703610  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 12:59:51.714601  3969 raft_consensus.cc:515] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:51.724085  4020 leader_election.cc:290] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:51.654410  1717 leader_election.cc:304] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:59:51.725772  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:51.727948  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:51.735038  1749 raft_consensus.cc:2468] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 3.
I20260709 12:59:51.677552  1718 leader_election.cc:403] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
W20260709 12:59:51.761685  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 107680us
I20260709 12:59:51.761757  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 12:59:51.762007  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 85445.9us
I20260709 12:59:51.762720  3521 raft_consensus.cc:2804] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:59:51.763438  3521 raft_consensus.cc:697] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 5 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 12:59:51.763558  1745 raft_consensus.cc:1275] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:59:51.764047  1749 raft_consensus.cc:1275] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:59:51.764353  1718 leader_election.cc:400] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 4 election: Vote denied by peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625) with higher term. Message: Invalid argument: T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 4. Current term is 5.
I20260709 12:59:51.764729  1718 leader_election.cc:403] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
I20260709 12:59:51.728272  3846 raft_consensus.cc:493] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:51.765550  1599 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e"
dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
mode: GRACEFUL
new_leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
 from {username='slave'} at 127.0.0.1:47752
I20260709 12:59:51.765666  3846 raft_consensus.cc:515] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:51.764750  3521 consensus_queue.cc:237] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 12:59:51.769261  2298 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:59:51.678182  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:51.748924  1751 rpcz_store.cc:276] 0709 12:59:49.702059 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:51.028841 (+1326782us) service_pool.cc:224] Handling call
0709 12:59:51.029192 (+   351us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:59:51.284865 (+255673us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:59:51.284902 (+    37us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:59:51.285671 (+   769us) log.cc:844] Serialized 52 byte log entry
0709 12:59:51.434538 (+148867us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:59:51.434601 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:51.631397 (+196796us) spinlock_profiling.cc:243] Waited 255 ms on lock 0x7b5400071554. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 12:59:51.631534 (+   137us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:59:51.704125 (+ 72591us) spinlock_profiling.cc:243] Waited 6.21 ms on lock 0x7b4400022b4c. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f4262f9bc85 00007f4262f9c6a2 00007f4265d0e1db 00007f4265d09aaf 00007f4265cfd86e 00007f4265cfcbb6 00007f4265cfc7d8 00007f4265d06221 00007f4265d061d9 00007f4265d06169
0709 12:59:51.747871 (+ 43746us) raft_consensus.cc:1685] finished
0709 12:59:51.747905 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:59:51.748138 (+   233us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3325,"raft.queue_time_us":1244,"raft.run_cpu_time_us":242,"raft.run_wall_time_us":67781,"spinlock_wait_cycles":600638720,"thread_start_us":58383,"threads_started":1,"wal-append.queue_time_us":66983}
I20260709 12:59:51.731765  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:51.772141  1597 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40154 (request call id 1728) took 1056 ms. Trace:
I20260709 12:59:51.708860  1752 raft_consensus.cc:3060] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:59:51.725731  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:51.738511  1601 rpcz_store.cc:276] 0709 12:59:49.495699 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:49.511014 (+ 15315us) service_pool.cc:224] Handling call
0709 12:59:49.511493 (+   479us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:59:50.403792 (+892299us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:51.737216 (+1333424us) spinlock_profiling.cc:243] Waited 891 ms on lock 0x7b54002a1cd4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 12:59:51.737674 (+   458us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2049916800}
I20260709 12:59:51.744882  2336 raft_consensus.cc:2955] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 4 LEARNER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } } }
I20260709 12:59:51.685778  3908 leader_election.cc:290] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:51.750279  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 12:59:51.751156  1571 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:59:51.751081  4011 raft_consensus.cc:2749] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260709 12:59:51.751776  2299 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:59:51.695061  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:59:51.697904  1674 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52766 (request call id 2047) took 1030 ms. Trace:
I20260709 12:59:51.697640  1604 rpcz_store.cc:276] 0709 12:59:49.805814 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:49.812346 (+  6532us) service_pool.cc:224] Handling call
0709 12:59:49.812712 (+   366us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:59:50.551951 (+739239us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:51.604115 (+1052164us) spinlock_profiling.cc:243] Waited 738 ms on lock 0x7b54002995d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 12:59:51.604632 (+   517us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1697602304}
I20260709 12:59:51.754314  1673 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57092 (request call id 481) took 1374 ms. Trace:
I20260709 12:59:51.727550  4015 leader_election.cc:290] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:51.758041  2337 raft_consensus.cc:2376] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 4. Re-sending same reply.
I20260709 12:59:51.748447  1671 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57092 (request call id 462) took 4091 ms. Trace:
I20260709 12:59:51.766507  1599 raft_consensus.cc:606] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 LEADER]: Received request to transfer leadership to TS 816fadcea8fc4893882f5f9b9f22a03d
I20260709 12:59:51.637861  3978 rpcz_store.cc:276] 0709 12:59:29.920779 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:29.921033 (+   254us) service_pool.cc:224] Handling call
0709 12:59:29.962407 (+ 41374us) spinlock_profiling.cc:243] Waited 25.8 ms on lock 0x7b580051efd0. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42660be0aa 00007f4266140447 00007f426615d70a 00007f426615e76a 00007f426615d91b 00007f426616903c 00007f42661686b3 00007f426b128d29 00007f42646a71d4 00007f42646a7162
0709 12:59:30.255234 (+292827us) spinlock_profiling.cc:243] Waited 2.49 ms on lock 0x7b4c001811c0. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426442ea20 00007f426443fa29 00007f4264405644 00007f426442cf2d 00007f426442cb93 00007f42660bb1af 00007f42660baf3e 00007f4266140888 00007f426615d75c 00007f426615e76a 00007f426615d91b
0709 12:59:30.317961 (+ 62727us) log.cc:844] Serialized 506 byte log entry
0709 12:59:30.638153 (+320192us) spinlock_profiling.cc:243] Waited 9.81 ms on lock 0x7b18001b89e0. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f4262fefc0a 00007f426c72f219 00007f4263077a40 00007f42630724e4 00007f4263072455 00007f42660bbb0b 00007f4266140b63 00007f4266168752 00007f426b128d29
0709 12:59:30.966298 (+328145us) spinlock_profiling.cc:243] Waited 2.2 ms on lock 0x7b54002c90a0. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42660e2b26 00007f42660dc49a 00007f42660db714 00007f42660bbeaa 00007f42660c2d53 00007f42660c2cd9 00007f42660c2c69 00007f42660c2c31 00007f42660c1dfd 000000000038f354
0709 12:59:31.042115 (+ 75817us) spinlock_profiling.cc:243] Waited 20.2 ms on lock 0x7b54002c90a0. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42660db142 00007f42660bbeaa 00007f42660c2d53 00007f42660c2cd9 00007f42660c2c69 00007f42660c2c31 00007f42660c1dfd 000000000038f354 000000000038f199 00007f4263075475
0709 12:59:51.536408 (+20494293us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":271900,"raft.queue_time_us":477251,"raft.run_cpu_time_us":18856,"raft.run_wall_time_us":559805,"spinlock_wait_cycles":139663104,"thread_start_us":40284,"threads_started":1,"wal-append.queue_time_us":73775,"wal-append.run_cpu_time_us":3334,"wal-append.run_wall_time_us":1109982}
I20260709 12:59:51.775260  1674 rpcz_store.cc:276] 0709 12:59:50.665962 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:51.680919 (+1014957us) service_pool.cc:224] Handling call
0709 12:59:51.696147 (+ 15228us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:59:51.779500  1679 raft_consensus.cc:3060] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:59:51.779033  1677 raft_consensus.cc:2468] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 4.
I20260709 12:59:51.780478  1675 raft_consensus.cc:2410] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260709 12:59:51.782534  1675 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52778 (request call id 1671) took 1024 ms. Trace:
I20260709 12:59:51.782671  4006 raft_consensus.cc:2749] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625) with higher term. Message: Invalid argument: T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 4. Current term is 5.
I20260709 12:59:51.782932  1675 rpcz_store.cc:276] 0709 12:59:50.758252 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:51.749897 (+991645us) service_pool.cc:224] Handling call
0709 12:59:51.782427 (+ 32530us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:59:51.784861  1674 raft_consensus.cc:1240] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 1. Current term is 2. Ops: []
I20260709 12:59:51.785696  1674 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52766 (request call id 2046) took 1150 ms. Trace:
I20260709 12:59:51.772653  1597 rpcz_store.cc:276] 0709 12:59:50.716030 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:51.729654 (+1013624us) service_pool.cc:224] Handling call
0709 12:59:51.772050 (+ 42396us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:59:51.786170  3978 raft_consensus.cc:1064] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701: attempting to promote NON_VOTER 3b879799899a499c8d409ee9f1277cd1 to VOTER
I20260709 12:59:51.792557  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:51.789148  1599 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 1787) took 1141 ms. Trace:
I20260709 12:59:51.798869  1599 rpcz_store.cc:276] 0709 12:59:50.647840 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:51.787768 (+1139928us) service_pool.cc:224] Handling call
0709 12:59:51.788315 (+   547us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:59:51.788735 (+   420us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:59:51.788770 (+    35us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:59:51.788848 (+    78us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:59:51.788904 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:51.788963 (+    59us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:59:51.789059 (+    96us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:59:51.801141  1599 raft_consensus.cc:1240] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260709 12:59:51.797304  4028 consensus_queue.cc:1048] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.003s
I20260709 12:59:51.801838  1599 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40154 (request call id 1731) took 1084 ms. Trace:
I20260709 12:59:51.786073  1674 rpcz_store.cc:276] 0709 12:59:50.635496 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:51.784000 (+1148504us) service_pool.cc:224] Handling call
0709 12:59:51.784463 (+   463us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:59:51.785345 (+   882us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:51.785571 (+   226us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":30208}
I20260709 12:59:51.785696  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:51.802248  1599 rpcz_store.cc:276] 0709 12:59:50.716990 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:51.800206 (+1083216us) service_pool.cc:224] Handling call
0709 12:59:51.800782 (+   576us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:59:51.801562 (+   780us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:51.801782 (+   220us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:59:51.806856  1751 raft_consensus.cc:1240] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 2. Current term is 3. Ops: []
I20260709 12:59:51.793577  2299 leader_election.cc:304] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 12:59:51.777348  1671 rpcz_store.cc:276] 0709 12:59:47.656532 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:47.925976 (+269444us) service_pool.cc:224] Handling call
0709 12:59:47.926495 (+   519us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:59:50.499778 (+2573283us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:51.747907 (+1248129us) spinlock_profiling.cc:243] Waited 2.49 s on lock 0x7b54000d7d54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 12:59:51.748194 (+   287us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5731995904}
I20260709 12:59:51.816161  1680 raft_consensus.cc:2468] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 5.
I20260709 12:59:51.818293  1680 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52778 (request call id 1670) took 1060 ms. Trace:
I20260709 12:59:51.819516  1680 rpcz_store.cc:276] 0709 12:59:50.757945 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:51.622361 (+864416us) service_pool.cc:224] Handling call
0709 12:59:51.818208 (+195847us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:59:51.822207  1680 raft_consensus.cc:1240] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 5. Current term is 6. Ops: [5.6-5.6]
I20260709 12:59:51.825677  1680 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52766 (request call id 2050) took 1143 ms. Trace:
I20260709 12:59:51.826867  1680 rpcz_store.cc:276] 0709 12:59:50.682455 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:51.821228 (+1138773us) service_pool.cc:224] Handling call
0709 12:59:51.821724 (+   496us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:59:51.825318 (+  3594us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:51.825579 (+   261us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:59:51.775820  1673 rpcz_store.cc:276] 0709 12:59:50.379626 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:50.406822 (+ 27196us) service_pool.cc:224] Handling call
0709 12:59:50.421005 (+ 14183us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:59:50.612403 (+191398us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:59:50.612472 (+    69us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:59:50.613241 (+   769us) log.cc:844] Serialized 26 byte log entry
0709 12:59:50.714641 (+101400us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:59:50.714709 (+    68us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:51.753574 (+1038865us) spinlock_profiling.cc:243] Waited 149 ms on lock 0x7b54000d9654. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 12:59:51.753653 (+    79us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:59:51.753727 (+    74us) raft_consensus.cc:1685] finished
0709 12:59:51.753740 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:59:51.754080 (+   340us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":41298,"raft.queue_time_us":43023,"raft.run_cpu_time_us":153,"raft.run_wall_time_us":154,"spinlock_wait_cycles":343818112,"thread_start_us":100898,"threads_started":1,"wal-append.queue_time_us":173620}
I20260709 12:59:51.832847  1677 raft_consensus.cc:2393] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 5: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 12:59:51.833580  1673 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52766 (request call id 2060) took 1011 ms. Trace:
I20260709 12:59:51.834156  1677 raft_consensus.cc:1240] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 4. Current term is 5. Ops: []
I20260709 12:59:51.835227  1597 raft_consensus.cc:1240] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260709 12:59:51.834034  1673 rpcz_store.cc:276] 0709 12:59:50.822638 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:51.832428 (+1009790us) service_pool.cc:224] Handling call
0709 12:59:51.832816 (+   388us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:59:51.833251 (+   435us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:59:51.833276 (+    25us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:59:51.833339 (+    63us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:59:51.833377 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:51.833434 (+    57us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:59:51.833519 (+    85us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:59:51.784864  1675 raft_consensus.cc:1240] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 3. Current term is 4. Ops: []
I20260709 12:59:51.872303  1754 raft_consensus.cc:2468] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 6.
I20260709 12:59:51.865959  1598 raft_consensus.cc:2468] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 2.
W20260709 12:59:51.880227  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 86848.2us
W20260709 12:59:51.889411  1746 consensus_meta.cc:287] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.505s	user 0.009s	sys 0.014s
I20260709 12:59:51.887676  1598 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40166 (request call id 1781) took 1228 ms. Trace:
I20260709 12:59:51.891916  1598 rpcz_store.cc:276] 0709 12:59:50.645049 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:51.773668 (+1128619us) service_pool.cc:224] Handling call
0709 12:59:51.873781 (+100113us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:59:51.891070  1746 raft_consensus.cc:2468] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 4.
W20260709 12:59:51.901592  3980 consensus_meta.cc:287] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.514s	user 0.020s	sys 0.000s
I20260709 12:59:51.902729  4028 raft_consensus.cc:2804] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:59:51.903636  4028 raft_consensus.cc:697] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 6 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 12:59:51.903801  2299 leader_election.cc:304] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 12:59:51.905162  4028 consensus_queue.cc:237] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:51.906797  3787 raft_consensus.cc:2804] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:59:51.907230  3787 raft_consensus.cc:493] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:59:51.907567  3787 raft_consensus.cc:3060] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:59:51.907378  2301 leader_election.cc:304] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:59:51.908525  3657 raft_consensus.cc:2804] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:59:51.902770  1599 raft_consensus.cc:1240] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 6. Current term is 7. Ops: []
I20260709 12:59:51.914412  1675 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52766 (request call id 2045) took 1264 ms. Trace:
W20260709 12:59:51.916548  4024 consensus_meta.cc:287] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.553s	user 0.015s	sys 0.004s
W20260709 12:59:51.917738  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 80952.6us
I20260709 12:59:51.917203  4024 raft_consensus.cc:515] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:51.909358  2301 leader_election.cc:304] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
W20260709 12:59:51.940820  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:59:51.943356  1602 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40166 (request call id 1777) took 1299 ms. Trace:
I20260709 12:59:51.950985  1753 raft_consensus.cc:2468] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 4.
I20260709 12:59:51.944624  2332 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39738 (request call id 594) took 1153 ms. Trace:
I20260709 12:59:51.954661  1677 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52766 (request call id 2061) took 1121 ms. Trace:
I20260709 12:59:51.918642  1676 raft_consensus.cc:2468] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 4.
I20260709 12:59:51.955389  1677 rpcz_store.cc:276] 0709 12:59:50.833449 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:51.833547 (+1000098us) service_pool.cc:224] Handling call
0709 12:59:51.833963 (+   416us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:59:51.834703 (+   740us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:51.954582 (+119879us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:59:51.954090  2332 rpcz_store.cc:276] 0709 12:59:50.791184 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:50.791325 (+   141us) service_pool.cc:224] Handling call
0709 12:59:51.944150 (+1152825us) spinlock_profiling.cc:243] Waited 288 ms on lock 0x7b54007602d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 12:59:51.944402 (+   252us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":661836672}
I20260709 12:59:51.914312  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:51.951877  1602 rpcz_store.cc:276] 0709 12:59:50.643908 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:50.644065 (+   157us) service_pool.cc:224] Handling call
0709 12:59:51.943039 (+1298974us) spinlock_profiling.cc:243] Waited 928 ms on lock 0x7b54003edfd4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 12:59:51.943240 (+   201us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2135374080}
I20260709 12:59:51.989888  1752 raft_consensus.cc:2468] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 5.
I20260709 12:59:51.983011  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:59:51.986836  1679 raft_consensus.cc:2468] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 6.
I20260709 12:59:51.923590  3846 leader_election.cc:290] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:51.916832  1675 rpcz_store.cc:276] 0709 12:59:50.635231 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:51.783868 (+1148637us) service_pool.cc:224] Handling call
0709 12:59:51.784466 (+   598us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:59:51.874592 (+ 90126us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:51.897832 (+ 23240us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1280}
I20260709 12:59:51.930675  3998 leader_election.cc:290] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 4 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:52.009356  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:59:52.020884  1753 raft_consensus.cc:1240] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 2. Current term is 3. Ops: []
W20260709 12:59:52.021330  3952 consensus_meta.cc:287] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.564s	user 0.017s	sys 0.003s
I20260709 12:59:52.022207  3952 raft_consensus.cc:2749] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:59:52.022965  1673 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52766 (request call id 2065) took 1008 ms. Trace:
I20260709 12:59:52.023561  1673 rpcz_store.cc:276] 0709 12:59:51.014395 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:51.837078 (+822683us) service_pool.cc:224] Handling call
0709 12:59:51.956426 (+119348us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:59:52.022486 (+ 66060us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:59:52.022537 (+    51us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:59:52.022600 (+    63us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:59:52.022637 (+    37us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:52.022726 (+    89us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:59:52.022874 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:59:51.945835  3950 leader_election.cc:290] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:52.024910  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:52.029692  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:52.031005  1677 raft_consensus.cc:1275] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
W20260709 12:59:52.020690  4057 log.cc:927] Time spent T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.160s	user 0.002s	sys 0.000s
I20260709 12:59:52.036063  1599 raft_consensus.cc:3060] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:59:52.033126  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:52.043989  3783 consensus_queue.cc:1048] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:59:52.044021  1753 raft_consensus.cc:3060] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Advancing to term 4
I20260709 12:59:51.898128  3978 consensus_queue.cc:237] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 3, Current term: 4, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:52.055048  3793 raft_consensus.cc:2749] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260709 12:59:52.054216  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 144967us
I20260709 12:59:52.056171  3793 consensus_queue.cc:1048] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:59:52.058344  3979 raft_consensus.cc:515] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:52.064127  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:52.021121  3977 raft_consensus.cc:2749] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625) with higher term. Message: Invalid argument: T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 3. Current term is 4.
I20260709 12:59:52.039449  1749 raft_consensus.cc:1240] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 4. Current term is 5. Ops: [4.7-4.7]
I20260709 12:59:52.084273  1597 raft_consensus.cc:3060] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:59:52.085342  1750 raft_consensus.cc:2393] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 4: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
W20260709 12:59:52.061100  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 135405us
I20260709 12:59:52.106032  1604 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 1786) took 1444 ms. Trace:
I20260709 12:59:52.108255  1646 leader_election.cc:304] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:59:52.123816  1754 raft_consensus.cc:2468] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 3.
I20260709 12:59:52.086567  3657 raft_consensus.cc:697] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 5 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 12:59:52.127365  1753 raft_consensus.cc:2468] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 4.
I20260709 12:59:52.096657  3969 leader_election.cc:290] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 4 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:52.107542  1604 rpcz_store.cc:276] 0709 12:59:50.646369 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:51.775594 (+1129225us) service_pool.cc:224] Handling call
0709 12:59:51.787779 (+ 12185us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:59:51.788524 (+   745us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:59:51.788561 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:59:51.789197 (+   636us) log.cc:844] Serialized 26 byte log entry
0709 12:59:51.824003 (+ 34806us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:59:52.083932 (+259929us) log.cc:844] Serialized 14 byte log entry
0709 12:59:52.090618 (+  6686us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:52.090747 (+   129us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:59:52.090806 (+    59us) raft_consensus.cc:1685] finished
0709 12:59:52.090825 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:59:52.091037 (+   212us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":279021,"raft.queue_time_us":1669,"raft.run_cpu_time_us":343,"raft.run_wall_time_us":14652,"thread_start_us":293612,"threads_started":2,"wal-append.queue_time_us":34692}
I20260709 12:59:52.138706  3941 consensus_queue.cc:1048] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:59:52.139022  2301 leader_election.cc:304] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:59:52.140048  4071 consensus_queue.cc:1048] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:59:52.140678  3657 consensus_queue.cc:237] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 12:59:52.144483  1644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:59:52.151352  4024 leader_election.cc:290] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 5 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:52.159276  1752 raft_consensus.cc:1240] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 3. Current term is 4. Ops: []
W20260709 12:59:52.164945  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 103679us
W20260709 12:59:52.164865  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 164875us
I20260709 12:59:52.166129  1748 raft_consensus.cc:2468] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 5.
I20260709 12:59:52.173198  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:52.175611  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:52.178418  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:59:52.189158  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:52.190279  1752 raft_consensus.cc:2393] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 3: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 12:59:52.187305  3952 raft_consensus.cc:2804] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:59:52.196403  1671 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52766 (request call id 2058) took 1407 ms. Trace:
I20260709 12:59:52.197050  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:52.198246  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:52.199788  1604 raft_consensus.cc:3060] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:59:52.224498  1644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:59:52.196072  3800 raft_consensus.cc:2804] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:59:52.196949  1671 rpcz_store.cc:276] 0709 12:59:50.786532 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:51.829521 (+1042989us) service_pool.cc:224] Handling call
0709 12:59:51.835220 (+  5699us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:59:51.835616 (+   396us) raft_consensus.cc:1534] Early marking committed up to index 3
0709 12:59:51.999641 (+164025us) log.cc:844] Serialized 14 byte log entry
0709 12:59:52.189214 (+189573us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:59:52.189276 (+    62us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:59:52.193842 (+  4566us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:52.193936 (+    94us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:59:52.194061 (+   125us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":219644,"prepare.run_cpu_time_us":68,"prepare.run_wall_time_us":2102,"raft.queue_time_us":5315,"thread_start_us":295055,"threads_started":2}
I20260709 12:59:52.196115  3952 raft_consensus.cc:493] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
W20260709 12:59:52.234611  4063 log.cc:927] Time spent T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.091s	user 0.000s	sys 0.002s
I20260709 12:59:52.241128  4070 consensus_queue.cc:1048] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.055s
W20260709 12:59:52.242600  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:59:52.245048  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 8 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:52.247509  1677 raft_consensus.cc:1275] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:59:52.242298  1646 leader_election.cc:304] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
W20260709 12:59:52.252406  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 54086.4us
I20260709 12:59:52.260782  1754 raft_consensus.cc:2393] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 4: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 12:59:52.267522  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:52.264595  3800 raft_consensus.cc:697] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 6 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 12:59:52.274233  3800 consensus_queue.cc:237] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:52.256894  4074 consensus_queue.cc:1059] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.021s
I20260709 12:59:52.255059  3952 raft_consensus.cc:3060] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:59:52.280231  4028 raft_consensus.cc:3055] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:59:52.280380  1598 raft_consensus.cc:2410] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for term 4 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
W20260709 12:59:52.289479  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 62312.8us
I20260709 12:59:52.292972  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:52.293596  1754 raft_consensus.cc:3060] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:59:52.286664  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:52.297457  4026 raft_consensus.cc:2804] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:59:52.297901  4026 raft_consensus.cc:493] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 12:59:52.297998  4024 raft_consensus.cc:493] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:52.298343  4026 raft_consensus.cc:3060] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:52.298645  3998 raft_consensus.cc:493] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:52.299050  3964 raft_consensus.cc:493] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:52.299247  3998 raft_consensus.cc:515] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:52.299724  3964 raft_consensus.cc:515] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:52.301600  3998 leader_election.cc:290] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:52.302114  3998 raft_consensus.cc:493] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:52.283638  3979 leader_election.cc:290] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 4 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:52.322189  1597 raft_consensus.cc:2468] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 4.
I20260709 12:59:52.291041  4028 raft_consensus.cc:740] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
W20260709 12:59:52.333539  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 148944us
I20260709 12:59:52.332818  1601 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 1780) took 1682 ms. Trace:
I20260709 12:59:52.302951  3964 leader_election.cc:290] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:52.337042  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:52.344997  2301 leader_election.cc:304] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 12:59:52.345745  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:59:52.298688  4024 raft_consensus.cc:515] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:52.359536  1604 raft_consensus.cc:2468] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 5.
I20260709 12:59:52.336558  1601 rpcz_store.cc:276] 0709 12:59:50.644794 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:51.780767 (+1135973us) service_pool.cc:224] Handling call
0709 12:59:51.794138 (+ 13371us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:59:51.794608 (+   470us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 12:59:51.994683 (+200075us) log.cc:844] Serialized 14 byte log entry
0709 12:59:52.314885 (+320202us) spinlock_profiling.cc:243] Waited 3.65 ms on lock 0x7b18001ecc40. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f4265d0a4bc 00007f426305848b 00007f426beb2ab9 00007f426307469b 00007f426307293f 00007f4263074c02 00007f4265cfc20b 00007f4265d01d23 00007f4265d0223a
0709 12:59:52.327022 (+ 12137us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:59:52.327087 (+    65us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:59:52.327163 (+    76us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:52.327262 (+    99us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:59:52.327455 (+   193us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":975,"prepare.queue_time_us":236733,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":50,"raft.queue_time_us":11652,"spinlock_wait_cycles":46618112,"thread_start_us":271515,"threads_started":2,"wal-append.queue_time_us":198197}
I20260709 12:59:52.335958  3950 raft_consensus.cc:493] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 12:59:52.361006  1644 leader_election.cc:304] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:52.376626  1646 leader_election.cc:304] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:59:52.369120  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
W20260709 12:59:52.377619  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:59:52.378942  1748 raft_consensus.cc:2468] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 4.
I20260709 12:59:52.384984  1680 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52766 (request call id 2057) took 1603 ms. Trace:
I20260709 12:59:52.386480  1597 raft_consensus.cc:3060] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:59:52.387764  1680 rpcz_store.cc:276] 0709 12:59:50.782528 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:51.828818 (+1046290us) service_pool.cc:224] Handling call
0709 12:59:51.889095 (+ 60277us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:59:51.986476 (+ 97381us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 12:59:52.225562 (+239086us) log.cc:844] Serialized 14 byte log entry
0709 12:59:52.357351 (+131789us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:59:52.357491 (+   140us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:59:52.357645 (+   154us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:52.357843 (+   198us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:59:52.384482 (+ 26639us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":108484,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":52,"raft.queue_time_us":11239,"raft.run_cpu_time_us":207,"raft.run_wall_time_us":11593,"thread_start_us":309893,"threads_started":2}
I20260709 12:59:52.390064  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:59:52.392889  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 12:59:52.393812  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 206829us
I20260709 12:59:52.394639  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:52.396628  1571 leader_election.cc:304] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:59:52.397861  1571 leader_election.cc:304] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
W20260709 12:59:52.398967  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 62863.1us
I20260709 12:59:52.400421  1644 leader_election.cc:304] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 12:59:52.356704  3998 raft_consensus.cc:515] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:52.406157  1599 raft_consensus.cc:2468] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 5.
I20260709 12:59:52.407349  1754 raft_consensus.cc:2468] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 4.
I20260709 12:59:52.381824  3980 raft_consensus.cc:2749] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:59:52.409123  1646 leader_election.cc:304] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:59:52.391296  3854 raft_consensus.cc:2804] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:59:52.412321  1601 raft_consensus.cc:2468] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 5.
I20260709 12:59:52.416618  1597 raft_consensus.cc:2468] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 4.
I20260709 12:59:52.414533  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:52.419364  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:59:52.422258  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
W20260709 12:59:52.444417  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 53547.9us
I20260709 12:59:52.445986  1597 raft_consensus.cc:3060] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:59:52.455605  1597 raft_consensus.cc:2468] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 4.
W20260709 12:59:52.422299  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 74399us
I20260709 12:59:52.458981  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:52.459338  1604 raft_consensus.cc:3060] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:59:52.410562  3908 raft_consensus.cc:2749] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20260709 12:59:52.460804  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 58616.2us
I20260709 12:59:52.418217  3657 raft_consensus.cc:2749] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:59:52.416368  3854 raft_consensus.cc:493] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 12:59:52.463043  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:59:52.447539  3964 raft_consensus.cc:2804] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:59:52.469058  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:52.409165  4069 consensus_queue.cc:1048] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.049s
I20260709 12:59:52.478348  3976 raft_consensus.cc:2804] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:59:52.479017  3976 raft_consensus.cc:493] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:59:52.476006  1717 leader_election.cc:304] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1
W20260709 12:59:52.481292  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 83571.9us
I20260709 12:59:52.481446  1597 raft_consensus.cc:3060] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:59:52.483073  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:59:52.483882  1599 raft_consensus.cc:2368] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 4. Current term is 5.
I20260709 12:59:52.479480  3976 raft_consensus.cc:3060] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:59:52.477425  2298 leader_election.cc:304] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:52.413552  3950 raft_consensus.cc:515] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:52.486243  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
W20260709 12:59:52.486720  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 148672us
I20260709 12:59:52.489584  3846 raft_consensus.cc:2804] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:59:52.446378  4028 consensus_queue.cc:260] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:52.490141  3846 raft_consensus.cc:493] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:59:52.490633  3846 raft_consensus.cc:3060] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:59:52.491111  4079 log.cc:927] Time spent T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.078s	user 0.000s	sys 0.000s
I20260709 12:59:52.471235  3854 raft_consensus.cc:3060] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:59:52.470583  3964 raft_consensus.cc:493] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:59:52.490656  4028 raft_consensus.cc:3060] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:59:52.479717  3998 leader_election.cc:290] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:52.495355  1605 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 1779) took 1850 ms. Trace:
I20260709 12:59:52.499107  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:52.499881  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:52.500495  1601 raft_consensus.cc:2393] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 5: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 12:59:52.501224  1644 leader_election.cc:304] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 12:59:52.497709  1605 rpcz_store.cc:276] 0709 12:59:50.644536 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:51.777724 (+1133188us) service_pool.cc:224] Handling call
0709 12:59:51.831776 (+ 54052us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:59:51.832340 (+   564us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:59:51.832368 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:59:52.011856 (+179488us) log.cc:844] Serialized 26 byte log entry
0709 12:59:52.226066 (+214210us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:59:52.226133 (+    67us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:52.226241 (+   108us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:59:52.495113 (+268872us) raft_consensus.cc:1685] finished
0709 12:59:52.495131 (+    18us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:59:52.495274 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":870,"raft.run_cpu_time_us":138,"raft.run_wall_time_us":138,"thread_start_us":131753,"threads_started":1,"wal-append.queue_time_us":306688}
I20260709 12:59:52.503575  1644 leader_election.cc:304] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 12:59:52.503244  1605 raft_consensus.cc:2955] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } } }
I20260709 12:59:52.508785  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:52.510851  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:59:52.509711  1677 raft_consensus.cc:2393] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 6: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 12:59:52.513119  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:52.511780  1671 raft_consensus.cc:2368] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 3. Current term is 4.
I20260709 12:59:52.514652  3980 raft_consensus.cc:2749] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:59:52.514369  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:59:52.515584  3980 raft_consensus.cc:2749] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:59:52.518260  1644 leader_election.cc:304] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 12:59:52.497179  3979 raft_consensus.cc:2804] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:59:52.518755  1646 leader_election.cc:304] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
W20260709 12:59:52.521066  4076 log.cc:927] Time spent T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.288s	user 0.001s	sys 0.000s
I20260709 12:59:52.523334  1571 leader_election.cc:304] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 12:59:52.525265  1598 raft_consensus.cc:2368] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 3. Current term is 4.
I20260709 12:59:52.528647  4069 raft_consensus.cc:2749] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:59:52.529634  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 12:59:52.531186  1551 auto_rebalancer.cc:629] Failed to schedule move for tablet 5108457796a949b7b753ab22b575b7a0: Illegal state: Replica 816fadcea8fc4893882f5f9b9f22a03d is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 leader_uuid: "af520b67945d443b901bbeca37cc5701" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } }
I20260709 12:59:52.531698  4001 leader_election.cc:290] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:52.532342  1569 leader_election.cc:304] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 12:59:52.534770  3908 raft_consensus.cc:2804] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:59:52.535607  3908 raft_consensus.cc:697] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
W20260709 12:59:52.536163  3787 consensus_meta.cc:287] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.628s	user 0.015s	sys 0.000s
W20260709 12:59:52.536301  1746 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55848 (request call id 1783) took 3311 ms (client timeout 2514 ms). Trace:
I20260709 12:59:52.536674  3787 raft_consensus.cc:515] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 12:59:52.536876  1746 rpcz_store.cc:269] 0709 12:59:49.225059 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:49.882180 (+657121us) service_pool.cc:224] Handling call
0709 12:59:52.535989 (+2653809us) spinlock_profiling.cc:243] Waited 867 ms on lock 0x7b54003237d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 12:59:52.536137 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1993577344}
I20260709 12:59:52.539553  1646 leader_election.cc:304] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
W20260709 12:59:52.540553  1644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:59:52.540879  2301 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:59:52.542387  4069 raft_consensus.cc:493] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 12:59:52.539332  3980 raft_consensus.cc:2804] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:59:52.543113  4069 raft_consensus.cc:515] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:52.536891  3908 consensus_queue.cc:237] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:52.497400  3964 raft_consensus.cc:3060] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:59:52.545923  2301 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:59:52.547484  4024 leader_election.cc:290] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:52.551324  3950 leader_election.cc:290] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:52.525759  3979 raft_consensus.cc:493] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:59:52.552013  3979 raft_consensus.cc:3060] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:59:52.554467  1601 raft_consensus.cc:1275] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260709 12:59:52.538371  4077 log.cc:927] Time spent T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.291s	user 0.000s	sys 0.003s
I20260709 12:59:52.560781  3950 raft_consensus.cc:2804] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:59:52.564381  3950 raft_consensus.cc:493] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:59:52.563787  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 95245.5us
W20260709 12:59:52.569111  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 73875.4us
I20260709 12:59:52.546015  4026 raft_consensus.cc:515] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:52.564818  3950 raft_consensus.cc:3060] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:59:52.582103  1719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:59:52.583623  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 138019us
I20260709 12:59:52.586973  2298 leader_election.cc:304] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:52.587358  1599 raft_consensus.cc:1240] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 4. Current term is 5. Ops: [4.4-4.4]
I20260709 12:59:52.589289  1571 leader_election.cc:304] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:59:52.590617  3787 leader_election.cc:290] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 5 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:52.592576  4092 raft_consensus.cc:2764] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260709 12:59:52.592839  1643 leader_election.cc:304] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:52.593743  4092 consensus_queue.cc:1059] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:59:52.594841  4092 raft_consensus.cc:3055] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:59:52.595201  4092 raft_consensus.cc:740] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 12:59:52.596169  4092 consensus_queue.cc:260] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:52.597702  4092 raft_consensus.cc:3060] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:59:52.598803  1569 leader_election.cc:304] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 12:59:52.590018  4069 leader_election.cc:290] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:52.612958  1604 raft_consensus.cc:2468] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 3.
I20260709 12:59:52.625610  1597 raft_consensus.cc:2468] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 3.
W20260709 12:59:52.626603  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 87180.9us
I20260709 12:59:52.633625  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 12:59:52.633639  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:52.641149  4070 raft_consensus.cc:2749] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:59:52.647063  1671 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3"
dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
mode: GRACEFUL
new_leader_uuid: "af520b67945d443b901bbeca37cc5701"
 from {username='slave'} at 127.0.0.1:43266
I20260709 12:59:52.647812  1671 raft_consensus.cc:606] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Received request to transfer leadership to TS af520b67945d443b901bbeca37cc5701
I20260709 12:59:52.648231  1671 raft_consensus.cc:614] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:59:52.648762  1672 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57092 (request call id 482) took 2260 ms. Trace:
I20260709 12:59:52.649169  1672 rpcz_store.cc:276] 0709 12:59:50.388380 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:50.533454 (+145074us) service_pool.cc:224] Handling call
0709 12:59:50.533787 (+   333us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:59:51.668022 (+1134235us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:52.648288 (+980266us) spinlock_profiling.cc:243] Waited 937 ms on lock 0x7b54000d7854. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 12:59:52.648533 (+   245us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2156125440}
W20260709 12:59:52.658519  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 109662us
I20260709 12:59:52.660445  1646 leader_election.cc:304] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:59:52.641171  2334 raft_consensus.cc:2468] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 4.
W20260709 12:59:52.661960  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 63245.4us
I20260709 12:59:52.621862  3952 raft_consensus.cc:515] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:52.668407  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 12:59:52.669786  3999 raft_consensus.cc:493] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:52.670622  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:52.670549  3999 raft_consensus.cc:515] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:52.671875  4102 consensus_queue.cc:1059] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
W20260709 12:59:52.673529  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 91697.7us
I20260709 12:59:52.673918  1571 leader_election.cc:304] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:59:52.679962  4099 raft_consensus.cc:2749] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:59:52.682588  3952 leader_election.cc:290] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 5 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:52.683585  3952 raft_consensus.cc:2749] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:59:52.683835  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:52.684347  3952 raft_consensus.cc:2804] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:59:52.684757  2328 raft_consensus.cc:2468] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 3.
I20260709 12:59:52.684979  3952 raft_consensus.cc:697] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 5 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 12:59:52.661509  3980 raft_consensus.cc:697] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 4 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 12:59:52.692181  1643 leader_election.cc:304] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 12:59:52.693027  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 12:59:52.680397  3976 raft_consensus.cc:515] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:52.695030  3854 raft_consensus.cc:515] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:52.695843  3976 leader_election.cc:290] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 4 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:52.695816  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 12:59:52.696228  1677 raft_consensus.cc:2368] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 3. Current term is 4.
I20260709 12:59:52.696973  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 12:59:52.698587  3846 raft_consensus.cc:515] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:52.701843  3979 raft_consensus.cc:515] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:52.702322  3854 leader_election.cc:290] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 5 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:52.690711  3999 leader_election.cc:290] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:52.672783  3964 raft_consensus.cc:515] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:52.703655  1749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43156 (request call id 407) took 1295 ms. Trace:
I20260709 12:59:52.716653  4095 raft_consensus.cc:3060] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:59:52.716859  4026 leader_election.cc:290] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 2 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:52.716429  1749 rpcz_store.cc:276] 0709 12:59:51.408698 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:51.764966 (+356268us) service_pool.cc:224] Handling call
0709 12:59:51.998147 (+233181us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:59:52.099396 (+101249us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:52.703369 (+603973us) spinlock_profiling.cc:243] Waited 164 ms on lock 0x7b5400019fd0. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661618ad 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 12:59:52.703564 (+   195us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":376455424}
W20260709 12:59:52.720369  4081 log.cc:927] Time spent T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.274s	user 0.003s	sys 0.001s
I20260709 12:59:52.720536  3980 consensus_queue.cc:237] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:52.722293  1674 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52766 (request call id 2048) took 2052 ms. Trace:
W20260709 12:59:52.723413  4066 log.cc:927] Time spent T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.696s	user 0.002s	sys 0.000s
I20260709 12:59:52.722740  1674 rpcz_store.cc:276] 0709 12:59:50.669966 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:51.803179 (+1133213us) service_pool.cc:224] Handling call
0709 12:59:51.803735 (+   556us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:59:51.867919 (+ 64184us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:59:51.867972 (+    53us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:59:51.956353 (+ 88381us) log.cc:844] Serialized 26 byte log entry
0709 12:59:52.322170 (+365817us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:59:52.322205 (+    35us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:52.322263 (+    58us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:59:52.719491 (+397228us) spinlock_profiling.cc:243] Waited 5.12 ms on lock 0x7b4400022b4c. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f4262f9bc85 00007f4262f9c6a2 00007f4265d0e1db 00007f4265d09aaf 00007f4265cfd86e 00007f4265cfcbb6 00007f4265cfc7d8 00007f4265d06221 00007f4265d061d9 00007f4265d06169
0709 12:59:52.721937 (+  2446us) raft_consensus.cc:1685] finished
0709 12:59:52.721962 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:59:52.722216 (+   254us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":23500,"raft.queue_time_us":38952,"raft.run_cpu_time_us":273,"raft.run_wall_time_us":22053,"spinlock_wait_cycles":11780992,"thread_start_us":291394,"threads_started":1,"wal-append.queue_time_us":415719}
I20260709 12:59:52.724845  4092 raft_consensus.cc:2764] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
W20260709 12:59:52.729931  1603 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40166 (request call id 1796) took 1915 ms (client timeout 1649 ms). Trace:
I20260709 12:59:52.730552  4102 raft_consensus.cc:3055] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:59:52.731110  4102 raft_consensus.cc:740] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 12:59:52.677052  3908 raft_consensus.cc:2804] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:59:52.819309  3948 raft_consensus.cc:493] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:52.819725  3908 raft_consensus.cc:697] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 12:59:52.702948  3846 leader_election.cc:290] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 5 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 12:59:52.730548  1603 rpcz_store.cc:269] 0709 12:59:50.814807 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:51.289408 (+474601us) service_pool.cc:224] Handling call
0709 12:59:52.729639 (+1440231us) spinlock_profiling.cc:243] Waited 271 ms on lock 0x7b54004222d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 12:59:52.729823 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":624405376}
I20260709 12:59:52.759778  3952 consensus_queue.cc:237] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:52.753006  4102 consensus_queue.cc:260] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 12:59:52.772310  4094 log.cc:927] Time spent T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.068s	user 0.000s	sys 0.000s
I20260709 12:59:52.772164  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
W20260709 12:59:52.772985  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 145086us
W20260709 12:59:52.780665  1676 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52766 (request call id 2054) took 2029 ms (client timeout 1947 ms). Trace:
W20260709 12:59:52.828799  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:59:52.784328  1718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:59:52.831411  1718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:59:52.790535  1750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55848 (request call id 1812) took 1100 ms. Trace:
I20260709 12:59:52.790670  1745 raft_consensus.cc:1240] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 3. Current term is 4. Ops: []
I20260709 12:59:52.716885  1746 raft_consensus.cc:1240] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260709 12:59:52.733533  4024 raft_consensus.cc:2749] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:59:52.820251  3948 raft_consensus.cc:515] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:52.877652  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:52.878875  1604 raft_consensus.cc:1240] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 2. Current term is 3. Ops: []
I20260709 12:59:52.879648  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:52.881093  1602 raft_consensus.cc:1240] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 2. Current term is 3. Ops: []
I20260709 12:59:52.881563  1603 raft_consensus.cc:1240] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260709 12:59:52.881942  4102 raft_consensus.cc:3060] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:59:52.882367  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:52.887691  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:52.888077  3979 leader_election.cc:290] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 12:59:52.888751  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 211995us
I20260709 12:59:52.832031  1750 rpcz_store.cc:276] 0709 12:59:51.690090 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:51.723633 (+ 33543us) service_pool.cc:224] Handling call
0709 12:59:52.790158 (+1066525us) spinlock_profiling.cc:243] Waited 360 ms on lock 0x7b54000e4a54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 12:59:52.790408 (+   250us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":828838656}
W20260709 12:59:52.826959  1676 rpcz_store.cc:269] 0709 12:59:50.751062 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:50.951434 (+200372us) service_pool.cc:224] Handling call
0709 12:59:52.780346 (+1828912us) spinlock_profiling.cc:243] Waited 957 ms on lock 0x7b54001f3e54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 12:59:52.780548 (+   202us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2200480128}
W20260709 12:59:52.888942  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 157394us
W20260709 12:59:52.873550  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 46782.5us
W20260709 12:59:52.893507  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 219927us
I20260709 12:59:52.894624  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:52.882982  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:52.897058  1596 raft_consensus.cc:2393] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 6: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
W20260709 12:59:52.897920  1596 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40154 (request call id 1729) took 2181 ms (client timeout 2541 ms). Trace:
W20260709 12:59:52.905085  1644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:59:52.900908  1596 rpcz_store.cc:269] 0709 12:59:50.716392 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:52.773345 (+2056953us) service_pool.cc:224] Handling call
0709 12:59:52.897830 (+124485us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:59:52.900085  1747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43156 (request call id 410) took 1484 ms. Trace:
W20260709 12:59:52.914175  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:59:52.930151  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:52.930861  2301 leader_election.cc:304] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:59:52.931196  4027 consensus_queue.cc:1048] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:59:52.931905  2301 leader_election.cc:304] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:59:52.933475  4027 raft_consensus.cc:2804] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:59:52.918227  3846 consensus_queue.cc:1059] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:59:52.934881  3846 raft_consensus.cc:3055] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:59:52.935281  3846 raft_consensus.cc:740] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 12:59:52.935724  4091 consensus_queue.cc:1059] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:59:52.936311  3846 consensus_queue.cc:260] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:52.936839  3787 consensus_queue.cc:1048] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.002s
I20260709 12:59:52.938772  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:52.939281  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:52.940131  3787 raft_consensus.cc:3055] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:59:52.940541  3787 raft_consensus.cc:740] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 12:59:52.941455  3787 consensus_queue.cc:260] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:52.920619  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:52.942997  1671 raft_consensus.cc:2393] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 6: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 12:59:52.944172  1671 raft_consensus.cc:1240] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 3. Current term is 4. Ops: []
I20260709 12:59:52.944913  1678 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57092 (request call id 480) took 3156 ms. Trace:
I20260709 12:59:52.945631  1753 raft_consensus.cc:1275] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 12:59:52.945299  1678 rpcz_store.cc:276] 0709 12:59:49.788912 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:50.366773 (+577861us) service_pool.cc:224] Handling call
0709 12:59:50.367181 (+   408us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:59:51.729828 (+1362647us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:52.944635 (+1214807us) spinlock_profiling.cc:243] Waited 1.32 s on lock 0x7b5400347e54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 12:59:52.944846 (+   211us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3027437440}
I20260709 12:59:52.947522  3846 raft_consensus.cc:3060] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:59:52.923892  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:52.926402  3906 consensus_queue.cc:1048] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:59:52.926671  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:52.950820  1750 raft_consensus.cc:2376] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 4. Re-sending same reply.
I20260709 12:59:52.926877  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:59:52.952013  1672 raft_consensus.cc:3060] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:59:52.952132  2334 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49082 (request call id 567) took 1959 ms (client timeout 1870 ms). Trace:
W20260709 12:59:52.952665  2334 rpcz_store.cc:269] 0709 12:59:50.993015 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:50.993444 (+   429us) service_pool.cc:224] Handling call
0709 12:59:52.951844 (+1958400us) spinlock_profiling.cc:243] Waited 1.63 s on lock 0x7b54005843d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 12:59:52.952024 (+   180us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3748020992}
I20260709 12:59:52.953605  1569 leader_election.cc:304] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 12:59:52.927438  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:52.954469  3998 raft_consensus.cc:2749] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:59:52.954932  1749 raft_consensus.cc:2376] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 4. Re-sending same reply.
I20260709 12:59:52.928030  1680 raft_consensus.cc:1275] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 12:59:52.928315  3950 raft_consensus.cc:515] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } }
I20260709 12:59:52.885910  4095 raft_consensus.cc:2749] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:59:52.933682  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 12:59:52.933506  4070 raft_consensus.cc:2804] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:59:52.959937  1745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43156 (request call id 406) took 1551 ms. Trace:
I20260709 12:59:52.960363  1745 rpcz_store.cc:276] 0709 12:59:51.408359 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:51.764500 (+356141us) service_pool.cc:224] Handling call
0709 12:59:52.790368 (+1025868us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:59:52.868971 (+ 78603us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:52.959747 (+ 90776us) spinlock_profiling.cc:243] Waited 1.03 s on lock 0x7b54000e4a50. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661618ad 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 12:59:52.959873 (+   126us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2358100352}
I20260709 12:59:52.961712  4028 consensus_queue.cc:1059] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 12:59:52.958827  3950 leader_election.cc:290] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 3 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:52.962453  3800 consensus_queue.cc:1059] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:59:52.963016  3793 raft_consensus.cc:3055] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:59:52.963457  3793 raft_consensus.cc:740] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 12:59:52.964049  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 12:59:52.964393  3793 consensus_queue.cc:260] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:52.958799  4070 raft_consensus.cc:697] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 4 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 12:59:52.966650  3793 raft_consensus.cc:3060] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:59:52.966862  4070 consensus_queue.cc:237] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:52.914664  3948 leader_election.cc:290] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:52.918877  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:52.982263  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:52.947705  1644 leader_election.cc:304] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 12:59:52.948256  4027 raft_consensus.cc:493] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 12:59:52.921963  1646 leader_election.cc:304] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 12:59:52.930027  4025 raft_consensus.cc:3060] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:59:53.000262  1569 leader_election.cc:304] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
W20260709 12:59:53.001246  1551 auto_rebalancer.cc:629] Failed to schedule move for tablet 73dc0a9696be41f78e22f1f77a6eee72: Illegal state: Replica 83595ad0fd5940c0a2bfc7c61acd969a is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 4 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } }
I20260709 12:59:53.001576  4092 raft_consensus.cc:2749] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:59:53.003695  1569 leader_election.cc:304] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 12:59:53.004415  1753 raft_consensus.cc:1275] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 12:59:52.984478  4027 raft_consensus.cc:3060] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:59:53.006824  1569 leader_election.cc:304] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 12:59:52.979940  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:52.985057  1673 raft_consensus.cc:1240] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 4. Current term is 5. Ops: []
I20260709 12:59:52.908318  1747 rpcz_store.cc:276] 0709 12:59:51.409663 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:51.800441 (+390778us) service_pool.cc:224] Handling call
0709 12:59:51.800885 (+   444us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:59:51.801419 (+   534us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:59:51.801449 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:59:51.802006 (+   557us) log.cc:844] Serialized 26 byte log entry
0709 12:59:51.888215 (+ 86209us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:59:51.888282 (+    67us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:51.888385 (+   103us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:59:52.722534 (+834149us) spinlock_profiling.cc:243] Waited 4.29 ms on lock 0x7b4400022b4c. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f4262f9bc85 00007f4262f9c6a2 00007f4265d0e1db 00007f4265d09aaf 00007f4265cfd86e 00007f4265cfcbb6 00007f4265cfc7d8 00007f4265d06221 00007f4265d061d9 00007f4265d06169
0709 12:59:52.894095 (+171561us) raft_consensus.cc:1685] finished
0709 12:59:52.894112 (+    17us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:59:52.894266 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":2396,"raft.run_cpu_time_us":150,"raft.run_wall_time_us":150,"spinlock_wait_cycles":9866624,"thread_start_us":85704,"threads_started":1,"wal-append.queue_time_us":210325}
I20260709 12:59:52.988500  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 12:59:52.994810  1600 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40166 (request call id 1798) took 2121 ms (client timeout 2385 ms). Trace:
W20260709 12:59:52.995354  1717 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 2.541s (SENT) [suppressed 28 similar messages]
I20260709 12:59:52.996584  4074 consensus_queue.cc:1059] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 12:59:52.997354  4028 consensus_queue.cc:1059] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:59:52.910420  3964 leader_election.cc:290] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 5 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:53.007038  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:52.936707  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 12:59:52.979467  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 49050.1us
I20260709 12:59:53.105249  2298 leader_election.cc:304] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 12:59:53.008208  1748 raft_consensus.cc:1275] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260709 12:59:53.011449  1502 catalog_manager.cc:5697] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 reported cstate change: term changed from 1 to 2, leader changed from 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194) to af520b67945d443b901bbeca37cc5701 (127.0.107.195). New cstate: current_term: 2 leader_uuid: "af520b67945d443b901bbeca37cc5701" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: HEALTHY } } }
I20260709 12:59:53.014667  3906 consensus_queue.cc:1048] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 12:59:53.016793  1604 pending_rounds.cc:85] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a: Aborting all ops after (but not including) 6
W20260709 12:59:53.017894  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 70242.2us
W20260709 12:59:53.024816  1551 auto_rebalancer.cc:629] Failed to schedule move for tablet 65285571995c4ed5b593041598056a22: Illegal state: Replica 83595ad0fd5940c0a2bfc7c61acd969a is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 3 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } }
I20260709 12:59:53.040990  1754 raft_consensus.cc:2393] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 4: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
W20260709 12:59:53.020483  4073 log.cc:927] Time spent T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.780s	user 0.004s	sys 0.000s
I20260709 12:59:53.043460  1672 raft_consensus.cc:2468] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 5.
W20260709 12:59:53.044015  1600 rpcz_store.cc:269] 0709 12:59:50.873862 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:52.752982 (+1879120us) service_pool.cc:224] Handling call
0709 12:59:52.994729 (+241747us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:59:52.895531  3908 consensus_queue.cc:237] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.7, Last appended by leader: 7, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:53.023833  4107 raft_consensus.cc:2749] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20260709 12:59:53.145239  1644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:59:53.112027  1596 raft_consensus.cc:1275] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260709 12:59:53.146674  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:59:53.147506  1672 raft_consensus.cc:2393] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 6: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
W20260709 12:59:53.141891  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 481309us
I20260709 12:59:53.141450  3969 raft_consensus.cc:493] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 12:59:53.151365  3969 raft_consensus.cc:515] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:53.152671  1718 leader_election.cc:304] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 12:59:53.154103  3969 leader_election.cc:290] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 12:59:53.111931  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 105141us
I20260709 12:59:53.171295  3783 consensus_queue.cc:1048] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 12:59:53.170172  1671 raft_consensus.cc:1275] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 12:59:53.154645  3979 raft_consensus.cc:2749] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:59:53.154732  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:59:53.174906  1680 raft_consensus.cc:2393] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 4: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 12:59:53.175192  1671 raft_consensus.cc:1275] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:59:53.175750  4103 raft_consensus.cc:493] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:53.176343  4103 raft_consensus.cc:515] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:53.177186  1718 leader_election.cc:304] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:53.177448  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:53.178110  1680 raft_consensus.cc:2468] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 4.
I20260709 12:59:53.178232  3979 raft_consensus.cc:2749] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:59:53.155112  2334 raft_consensus.cc:1240] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 4. Current term is 5. Ops: []
I20260709 12:59:53.155238  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 8 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:53.155697  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 12:59:53.180374  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:53.180703  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:59:53.156337  4090 consensus_queue.cc:1059] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.172s
I20260709 12:59:53.181370  1673 raft_consensus.cc:3060] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:59:53.181597  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:53.181977  4090 raft_consensus.cc:3055] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:59:53.182399  4090 raft_consensus.cc:740] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 12:59:53.157486  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:53.183575  2336 raft_consensus.cc:2468] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 3.
I20260709 12:59:53.184497  3950 consensus_queue.cc:1059] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 12:59:53.185359  1718 leader_election.cc:304] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:53.186880  2299 leader_election.cc:304] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:53.158762  4096 raft_consensus.cc:2749] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:59:53.109212  3999 raft_consensus.cc:493] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:53.160473  4028 raft_consensus.cc:2749] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:59:53.108948  1604 pending_rounds.cc:107] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a: Aborting uncommitted NO_OP operation due to leader change: 4.7
I20260709 12:59:53.211006  1604 raft_consensus.cc:2889] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:59:53.164199  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:53.211771  1672 raft_consensus.cc:2468] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 4.
I20260709 12:59:53.167047  1679 raft_consensus.cc:1240] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 2. Current term is 3. Ops: []
I20260709 12:59:53.167115  3800 raft_consensus.cc:3055] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:59:53.043032  4095 raft_consensus.cc:493] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:53.217165  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 12:59:53.109673  3948 raft_consensus.cc:493] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:59:53.218289  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 233024us
I20260709 12:59:53.157835  4104 raft_consensus.cc:2764] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
W20260709 12:59:53.223266  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:59:53.188889  1719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:59:53.142828  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:59:53.196748  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 12:59:53.043038  3793 consensus_queue.cc:1059] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.125s
I20260709 12:59:53.221498  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 12:59:53.228917  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 307030us
W20260709 12:59:53.233961  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 48677us
I20260709 12:59:53.234261  1596 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2"
dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
mode: GRACEFUL
new_leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
 from {username='slave'} at 127.0.0.1:47752
I20260709 12:59:53.232729  1646 leader_election.cc:304] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 12:59:53.237917  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:53.195706  3941 raft_consensus.cc:3060] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:59:53.233757  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 12:59:53.243795  2334 raft_consensus.cc:3060] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:59:53.234151  4103 leader_election.cc:290] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:53.245832  1643 leader_election.cc:304] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 12:59:53.222636  3787 raft_consensus.cc:3060] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:59:53.266126  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:53.267046  1753 raft_consensus.cc:2468] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 4.
I20260709 12:59:53.217280  3800 raft_consensus.cc:740] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 12:59:53.216622  3979 raft_consensus.cc:2749] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260709 12:59:53.274942  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 56165us
W20260709 12:59:53.276103  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 233965us
I20260709 12:59:53.277084  2298 leader_election.cc:304] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:53.197520  4091 consensus_queue.cc:1059] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.057s
I20260709 12:59:53.271338  2299 leader_election.cc:304] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: af520b67945d443b901bbeca37cc5701
I20260709 12:59:53.274228  1747 raft_consensus.cc:1275] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:59:53.261835  1502 catalog_manager.cc:5697] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "af520b67945d443b901bbeca37cc5701" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: HEALTHY } } }
I20260709 12:59:53.243387  4100 raft_consensus.cc:2749] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:59:53.284399  1718 leader_election.cc:304] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:53.285549  3800 consensus_queue.cc:260] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:53.286690  1500 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:43476 (request call id 191) took 11099 ms. Trace:
I20260709 12:59:53.287170  3993 consensus_queue.cc:1059] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.059s
I20260709 12:59:53.287555  3783 raft_consensus.cc:3055] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:59:53.289760  1643 leader_election.cc:304] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 12:59:53.294304  1571 leader_election.cc:304] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
W20260709 12:59:53.301362  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 305980us
I20260709 12:59:53.273865  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 12:59:53.300590  1502 catalog_manager.cc:5697] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "af520b67945d443b901bbeca37cc5701" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:59:53.262519  3999 raft_consensus.cc:515] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:53.264737  4095 raft_consensus.cc:515] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:53.300541  3976 raft_consensus.cc:2804] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:59:53.260578  3948 raft_consensus.cc:515] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:53.279079  4028 consensus_queue.cc:1048] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.053s
W20260709 12:59:53.369083  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 73817.3us
I20260709 12:59:53.269950  4103 consensus_queue.cc:1048] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.064s
W20260709 12:59:53.342702  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 72400.7us
I20260709 12:59:53.291010  4074 consensus_queue.cc:1059] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 4.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.055s
I20260709 12:59:53.303383  3783 raft_consensus.cc:740] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 12:59:53.362478  1569 leader_election.cc:304] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 12:59:53.373382  2332 raft_consensus.cc:1240] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 3. Current term is 4. Ops: []
I20260709 12:59:53.303419  4069 raft_consensus.cc:2804] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Leader election won for term 5
W20260709 12:59:53.376334  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 86663.6us
W20260709 12:59:53.377846  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 133526us
I20260709 12:59:53.379007  1671 raft_consensus.cc:2468] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 3.
W20260709 12:59:53.383054  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 88857.2us
I20260709 12:59:53.383586  1718 leader_election.cc:400] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 pre-election: Vote denied by peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625) with higher term. Message: Invalid argument: T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 5. Current term is 6.
I20260709 12:59:53.384194  1718 leader_election.cc:403] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:59:53.340478  3521 raft_consensus.cc:493] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:53.359719  4107 raft_consensus.cc:2749] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:59:53.370820  3800 raft_consensus.cc:3060] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:59:53.396037  1571 leader_election.cc:304] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 12:59:53.370980  3977 raft_consensus.cc:2804] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader election won for term 5
W20260709 12:59:53.452236  2300 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:59:53.452409  1643 leader_election.cc:304] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:53.392292  3793 consensus_queue.cc:1048] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.061s
I20260709 12:59:53.340811  1502 catalog_manager.cc:5697] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 reported cstate change: config changed from index 3 to -1, term changed from 2 to 4, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) evicted, now has a pending config: VOTER 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193), VOTER 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194), VOTER af520b67945d443b901bbeca37cc5701 (127.0.107.195), NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196). New cstate: current_term: 4 leader_uuid: "af520b67945d443b901bbeca37cc5701" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: HEALTHY } } } pending_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } } }
I20260709 12:59:53.454761  4070 consensus_queue.cc:1059] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20260709 12:59:53.456590  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 114911us
I20260709 12:59:53.456089  1502 catalog_manager.cc:5697] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 reported cstate change: term changed from 3 to 4, leader changed from 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194) to af520b67945d443b901bbeca37cc5701 (127.0.107.195). New cstate: current_term: 4 leader_uuid: "af520b67945d443b901bbeca37cc5701" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: HEALTHY } } }
I20260709 12:59:53.457216  3793 raft_consensus.cc:3055] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:59:53.457456  1569 leader_election.cc:304] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
W20260709 12:59:53.458573  1644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:59:53.457803  3793 raft_consensus.cc:740] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 12:59:53.376152  4001 raft_consensus.cc:3055] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:59:53.457844  1502 catalog_manager.cc:5697] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 reported cstate change: term changed from 1 to 2, leader changed from af520b67945d443b901bbeca37cc5701 (127.0.107.195) to 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193). New cstate: current_term: 2 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:53.371562  3906 raft_consensus.cc:2749] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:59:53.395320  4074 raft_consensus.cc:3055] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:59:53.397826  4099 raft_consensus.cc:2804] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:59:53.381313  3976 raft_consensus.cc:697] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 12:59:53.376199  4024 raft_consensus.cc:2749] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260709 12:59:53.465380  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 81292.6us
I20260709 12:59:53.375928  3993 consensus_queue.cc:1048] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.090s
I20260709 12:59:53.502508  3783 consensus_queue.cc:260] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:53.504482  3783 raft_consensus.cc:3060] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:59:53.397614  4092 raft_consensus.cc:2749] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:59:53.395087  4069 raft_consensus.cc:697] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 5 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 12:59:53.506851  3948 leader_election.cc:290] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:53.506822  4069 consensus_queue.cc:237] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:53.507130  4027 raft_consensus.cc:515] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:53.507932  4102 raft_consensus.cc:2749] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:59:53.473280  4001 raft_consensus.cc:740] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 12:59:53.509454  4027 leader_election.cc:290] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 4 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:53.510505  4001 consensus_queue.cc:260] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:53.511914  4001 raft_consensus.cc:3060] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:59:53.513576  3911 raft_consensus.cc:3055] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:59:53.515329  4105 raft_consensus.cc:993] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a: : Instructing follower 816fadcea8fc4893882f5f9b9f22a03d to start an election
W20260709 12:59:53.494634  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 111384us
I20260709 12:59:53.514091  3911 raft_consensus.cc:740] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 12:59:53.483291  4099 raft_consensus.cc:493] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 12:59:53.499873  3908 raft_consensus.cc:2749] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:59:53.500946  4095 leader_election.cc:290] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:53.502847  1646 leader_election.cc:304] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
W20260709 12:59:53.499122  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 46817.9us
I20260709 12:59:53.459848  3978 raft_consensus.cc:2749] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: Vote denied by peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625) with higher term. Message: Invalid argument: T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 5. Current term is 6.
I20260709 12:59:53.486215  1569 leader_election.cc:304] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 12:59:53.528983  2334 raft_consensus.cc:2468] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 5.
I20260709 12:59:53.515898  4105 raft_consensus.cc:1081] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 LEADER]: Signalling peer 816fadcea8fc4893882f5f9b9f22a03d to start an election
I20260709 12:59:53.541198  4099 raft_consensus.cc:3060] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:59:53.566501  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 80393.4us
I20260709 12:59:53.500959  3976 consensus_queue.cc:237] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:53.568068  1569 leader_election.cc:304] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 12:59:53.568265  1643 leader_election.cc:304] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:53.580657  1718 leader_election.cc:400] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625) with higher term. Message: Invalid argument: T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 3. Current term is 4.
I20260709 12:59:53.581211  1718 leader_election.cc:403] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:59:53.481114  4074 raft_consensus.cc:740] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
W20260709 12:59:53.590821  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 194881us
I20260709 12:59:53.516062  1746 raft_consensus.cc:1275] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:59:53.541579  4106 raft_consensus.cc:2749] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:59:53.484949  3977 raft_consensus.cc:697] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 5 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
W20260709 12:59:53.604135  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 101362us
I20260709 12:59:53.538751  3846 raft_consensus.cc:3055] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:59:53.567333  3999 leader_election.cc:290] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:53.601938  1569 leader_election.cc:304] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 12:59:53.293615  1500 rpcz_store.cc:276] 0709 12:59:42.062982 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:42.073904 (+ 10922us) service_pool.cc:224] Handling call
0709 12:59:53.162278 (+11088374us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:59:47.912704 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:59:48.369024 (+456320us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:59:48.416915 (+ 47891us) write_op.cc:435] Acquired schema lock
0709 12:59:48.438831 (+ 21916us) tablet.cc:662] Decoding operations
0709 12:59:48.719842 (+281011us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:59:48.719939 (+    97us) write_op.cc:641] Partition lock acquired for write op
0709 12:59:48.719980 (+    41us) tablet.cc:685] Acquiring locks for 4 operations
0709 12:59:48.720527 (+   547us) tablet.cc:701] Row locks acquired
0709 12:59:48.720553 (+    26us) write_op.cc:260] PREPARE: finished
0709 12:59:48.720776 (+   223us) write_op.cc:270] Start()
0709 12:59:48.720933 (+   157us) write_op.cc:276] Timestamp: P: 1783601988720710 usec, L: 0
0709 12:59:48.720965 (+    32us) op_driver.cc:348] REPLICATION: starting
0709 12:59:48.721979 (+  1014us) log.cc:844] Serialized 1686 byte log entry
0709 12:59:49.885087 (+1163108us) op_driver.cc:464] REPLICATION: finished
0709 12:59:50.419803 (+534716us) write_op.cc:301] APPLY: starting
0709 12:59:50.436329 (+ 16526us) tablet.cc:1367] starting BulkCheckPresence
0709 12:59:50.468470 (+ 32141us) tablet.cc:1370] finished BulkCheckPresence
0709 12:59:50.501776 (+ 33306us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:59:50.514703 (+ 12927us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:59:50.515006 (+   303us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=4
0709 12:59:50.515050 (+    44us) write_op.cc:312] APPLY: finished
0709 12:59:50.593329 (+ 78279us) log.cc:844] Serialized 41 byte log entry
0709 12:59:50.712691 (+119362us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:59:50.729019 (+ 16328us) write_op.cc:454] Released schema lock
0709 12:59:50.744709 (+ 15690us) write_op.cc:341] FINISH: Updating metrics
0709 12:59:50.780513 (+ 35804us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:59:50.780552 (+    39us) write_op.cc:454] Released schema lock
0709 12:59:51.032417 (+251865us) spinlock_profiling.cc:243] Waited 1.75 ms on lock 0x7b4400022b4c. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f4262f9bc85 00007f4262f9c6a2 00007f4265d0e1db 00007f4265d09aaf 00007f4265cfd86e 00007f4265cfcbb6 00007f4265cfc7d8 00007f4265d06221 00007f4265d061d9 00007f4265d06169
Metrics: {"mutex_wait_us":1187635,"reported_tablets":14,"child_traces":[["op",{"apply.queue_time_us":485816,"apply.run_cpu_time_us":14747,"apply.run_wall_time_us":379268,"num_ops":4,"prepare.queue_time_us":1370204,"prepare.run_cpu_time_us":21405,"prepare.run_wall_time_us":1277597,"raft.queue_time_us":320069,"raft.run_cpu_time_us":16455,"raft.run_wall_time_us":395736,"replication_time_us":1109114,"spinlock_wait_cycles":4019968,"thread_start_us":2048863,"threads_started":5,"wal-append.queue_time_us":751338,"wal-append.run_cpu_time_us":36575,"wal-append.run_wall_time_us":2557683}]]}
I20260709 12:59:53.480548  3521 raft_consensus.cc:515] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:53.609894  1569 leader_election.cc:304] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 12:59:53.582767  1675 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57092 (request call id 489) took 1554 ms. Trace:
W20260709 12:59:53.609867  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 332832us
I20260709 12:59:53.598222  3964 raft_consensus.cc:2804] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:59:53.606511  1571 leader_election.cc:304] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
W20260709 12:59:53.611928  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 48616.1us
I20260709 12:59:53.612944  1679 raft_consensus.cc:1240] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 3. Current term is 4. Ops: []
I20260709 12:59:53.606995  3978 raft_consensus.cc:2749] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625) with higher term. Message: Invalid argument: T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 3. Current term is 4.
I20260709 12:59:53.598469  4105 raft_consensus.cc:2804] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:59:53.618046  1677 raft_consensus.cc:1240] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 4. Current term is 5. Ops: []
W20260709 12:59:53.611231  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 235891us
I20260709 12:59:53.603793  4024 raft_consensus.cc:493] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:59:53.624393  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:59:53.607627  3846 raft_consensus.cc:740] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 12:59:53.603530  1676 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e"
dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
 from {username='slave'} at 127.0.0.1:52766
I20260709 12:59:53.610560  4098 raft_consensus.cc:2804] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:59:53.606588  4126 raft_consensus.cc:2749] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:59:53.634954  1596 raft_consensus.cc:606] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Received request to transfer leadership to TS 816fadcea8fc4893882f5f9b9f22a03d
I20260709 12:59:53.619107  3964 raft_consensus.cc:493] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:59:53.646848  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 66438.9us
I20260709 12:59:53.647256  1605 raft_consensus.cc:1240] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260709 12:59:53.647761  1596 raft_consensus.cc:614] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:59:53.648303  1604 raft_consensus.cc:1240] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
W20260709 12:59:53.661317  1644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:59:53.633121  4105 raft_consensus.cc:493] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:59:53.637524  4106 raft_consensus.cc:2804] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:59:53.661023  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
W20260709 12:59:53.667945  1718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:59:53.669625  1718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:59:53.657049  3998 raft_consensus.cc:2749] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:59:53.619735  4124 consensus_queue.cc:1048] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.053s
I20260709 12:59:53.612037  3793 consensus_queue.cc:260] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:53.641481  4098 raft_consensus.cc:493] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:59:53.663030  4113 log.cc:927] Time spent T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.176s	user 0.000s	sys 0.001s
I20260709 12:59:53.637447  4028 consensus_queue.cc:1048] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.037s
I20260709 12:59:53.673921  3950 raft_consensus.cc:2749] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:59:53.664716  3964 raft_consensus.cc:3060] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:59:53.673583  4105 raft_consensus.cc:3060] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:59:53.693533  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 92340.5us
I20260709 12:59:53.696437  3941 raft_consensus.cc:2749] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:59:53.691435  1676 raft_consensus.cc:493] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
W20260709 12:59:53.713310  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 65997.2us
I20260709 12:59:53.714469  1676 raft_consensus.cc:3060] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:59:53.699839  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 90025.2us
W20260709 12:59:53.701885  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 60151.8us
I20260709 12:59:53.686450  4106 raft_consensus.cc:493] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:59:53.657790  4024 raft_consensus.cc:515] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:53.721036  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:53.722005  2330 raft_consensus.cc:2468] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 3.
W20260709 12:59:53.732324  1673 consensus_meta.cc:287] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.550s	user 0.007s	sys 0.009s
I20260709 12:59:53.732743  1673 raft_consensus.cc:2468] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 3.
I20260709 12:59:53.694952  4098 raft_consensus.cc:3060] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:59:53.728647  3846 consensus_queue.cc:260] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:53.729723  1571 leader_election.cc:304] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 12:59:53.631812  3911 consensus_queue.cc:260] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:53.694177  3978 raft_consensus.cc:493] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:53.641798  4074 consensus_queue.cc:260] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:53.614495  1675 rpcz_store.cc:276] 0709 12:59:52.028180 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:52.874536 (+846356us) service_pool.cc:224] Handling call
0709 12:59:52.966283 (+ 91747us) spinlock_profiling.cc:243] Waited 2.94 ms on lock 0x7b5800086a98. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f426303db71 00007f426303da22 00007f426bf56c09 00007f426b0b0ee0 00007f426b17a0ab 00007f426b17a131 00007f426b13bfc0 00007f426b126929 00007f426b1263d2 00007f426469dbd4
0709 12:59:52.966574 (+   291us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:59:52.966952 (+   378us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:59:52.967309 (+   357us) log.cc:844] Serialized 14 byte log entry
0709 12:59:53.474534 (+507225us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:59:53.477577 (+  3043us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:59:53.483026 (+  5449us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:53.518373 (+ 35347us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:59:53.568532 (+ 50159us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":13304,"prepare.queue_time_us":171,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":51,"raft.queue_time_us":33720,"raft.run_cpu_time_us":156,"raft.run_wall_time_us":7023,"spinlock_wait_cycles":6765952,"thread_start_us":453777,"threads_started":2,"wal-append.queue_time_us":396051}
W20260709 12:59:53.738981  4120 log.cc:927] Time spent T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.100s	user 0.000s	sys 0.001s
I20260709 12:59:53.741372  3793 raft_consensus.cc:3060] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:59:53.746163  1596 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147"
dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
mode: GRACEFUL
new_leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
 from {username='slave'} at 127.0.0.1:47752
I20260709 12:59:53.730845  4106 raft_consensus.cc:3060] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:59:53.750849  1643 leader_election.cc:304] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:53.752913  1643 leader_election.cc:304] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:53.761322  4138 raft_consensus.cc:2749] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:59:53.690584  3977 consensus_queue.cc:237] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:53.748014  3911 raft_consensus.cc:3060] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:59:53.756839  3846 raft_consensus.cc:3060] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:59:53.760109  4137 raft_consensus.cc:2749] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:59:53.753156  4107 raft_consensus.cc:2749] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:59:53.742882  4102 consensus_queue.cc:1059] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.020s
W20260709 12:59:53.776877  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 47246.7us
I20260709 12:59:53.743574  3978 raft_consensus.cc:515] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:53.764104  4074 raft_consensus.cc:3060] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:59:53.786870  1571 leader_election.cc:304] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 12:59:53.788430  1643 leader_election.cc:304] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:53.804082  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 12:59:53.807376  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 92496.5us
I20260709 12:59:53.814059  1718 leader_election.cc:304] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:59:53.785951  3948 raft_consensus.cc:493] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:53.810751  3950 raft_consensus.cc:2749] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:59:53.789881  3999 raft_consensus.cc:493] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:53.809650  3979 consensus_queue.cc:1059] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.010s
I20260709 12:59:53.825070  1571 leader_election.cc:304] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
W20260709 12:59:53.827937  2298 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:59:53.822309  2299 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:59:53.816789  4107 raft_consensus.cc:2749] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:59:53.829655  3999 raft_consensus.cc:515] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:53.837205  4139 raft_consensus.cc:3055] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:59:53.837692  4139 raft_consensus.cc:740] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 12:59:53.838724  4069 raft_consensus.cc:493] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 12:59:53.839296  4069 raft_consensus.cc:515] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:53.841605  4069 leader_election.cc:290] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:53.843919  1600 raft_consensus.cc:1275] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 12:59:53.846602  3999 leader_election.cc:290] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 12:59:53.847766  4108 log.cc:927] Time spent T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.371s	user 0.003s	sys 0.001s
I20260709 12:59:53.848532  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 12:59:53.849159  4025 consensus_meta.cc:287] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.849s	user 0.011s	sys 0.000s
I20260709 12:59:53.850932  4027 raft_consensus.cc:493] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:53.851580  4027 raft_consensus.cc:515] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:53.853358  2331 raft_consensus.cc:2468] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 4.
I20260709 12:59:53.853720  2328 raft_consensus.cc:2410] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260709 12:59:53.855536  3978 leader_election.cc:290] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:53.853336  1600 raft_consensus.cc:1275] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:59:53.860978  2330 raft_consensus.cc:1275] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 4 LEARNER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20260709 12:59:53.862542  1746 raft_consensus.cc:1275] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 12:59:53.863804  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:53.864159  1643 leader_election.cc:304] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 12:59:53.865425  3993 consensus_queue.cc:1048] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:59:53.866787  4136 raft_consensus.cc:2749] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:59:53.867002  3993 raft_consensus.cc:493] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:53.867741  3993 raft_consensus.cc:515] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:53.868829  1748 raft_consensus.cc:1240] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 3. Current term is 4. Ops: []
I20260709 12:59:53.866103  1643 leader_election.cc:304] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 12:59:53.880117  1674 raft_consensus.cc:1240] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 2. Current term is 3. Ops: []
I20260709 12:59:53.873548  1676 raft_consensus.cc:515] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:53.884305  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 12:59:53.887152  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 62146.1us
I20260709 12:59:53.828593  3521 leader_election.cc:290] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 12:59:53.876600  3787 consensus_meta.cc:287] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.595s	user 0.010s	sys 0.003s
I20260709 12:59:53.893749  1750 raft_consensus.cc:1275] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:59:53.893884  2299 leader_election.cc:304] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:53.895787  2336 raft_consensus.cc:1240] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 3. Current term is 4. Ops: []
I20260709 12:59:53.897061  1571 leader_election.cc:304] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 12:59:53.889412  3950 raft_consensus.cc:2749] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20260709 12:59:53.914474  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 48461.5us
I20260709 12:59:53.885216  3999 raft_consensus.cc:2804] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:59:53.902784  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:53.918386  1745 raft_consensus.cc:3060] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:59:53.881623  4024 leader_election.cc:290] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:53.830193  3948 raft_consensus.cc:515] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:53.890669  4096 raft_consensus.cc:2764] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260709 12:59:53.915949  1571 leader_election.cc:304] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 12:59:53.865463  4025 consensus_queue.cc:1048] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.045s
W20260709 12:59:53.925935  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 111999us
I20260709 12:59:53.929796  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:53.930655  2332 raft_consensus.cc:2468] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 4.
I20260709 12:59:53.931063  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:53.931977  2333 raft_consensus.cc:2410] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for term 6 because replica has last-logged OpId of term: 4 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20260709 12:59:53.920603  3999 raft_consensus.cc:493] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:59:53.910634  4137 consensus_queue.cc:1059] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.026s
I20260709 12:59:53.959686  4148 consensus_queue.cc:1048] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:59:53.960752  4090 consensus_queue.cc:260] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:53.961869  4090 raft_consensus.cc:3060] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:59:53.900400  4136 consensus_queue.cc:1059] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.044s
I20260709 12:59:53.951346  4024 raft_consensus.cc:2749] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:59:53.910372  4099 raft_consensus.cc:515] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:53.917594  4139 consensus_queue.cc:260] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:53.964435  4139 raft_consensus.cc:3060] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:59:53.963155  4136 raft_consensus.cc:3055] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 3 LEADER]: Stepping down as leader of term 3
W20260709 12:59:53.946210  1643 connection.cc:256] RPC call timeout handler was delayed by -1.7928726689999621s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16490000000000002s for a response
W20260709 12:59:53.976451  1643 connection.cc:256] RPC call timeout handler was delayed by -1.6896461619999741s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15660000000000002s for a response
W20260709 12:59:53.977041  1643 connection.cc:256] RPC call timeout handler was delayed by -1.6518461619999698s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16080000000000003s for a response
W20260709 12:59:53.977478  1643 connection.cc:256] RPC call timeout handler was delayed by -1.5861461619999773s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16810000000000003s for a response
W20260709 12:59:53.978266  1643 connection.cc:256] RPC call timeout handler was delayed by -1.5654461620000006s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1704s for a response
W20260709 12:59:53.978771  1643 connection.cc:256] RPC call timeout handler was delayed by -1.5159461619999774s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1759s for a response
W20260709 12:59:53.979202  1643 connection.cc:256] RPC call timeout handler was delayed by -1.1304726689999711s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2385s for a response
W20260709 12:59:53.979622  1643 connection.cc:256] RPC call timeout handler was delayed by -1.1136461619999523s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22060000000000002s for a response
W20260709 12:59:53.977349  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 92009.2us
I20260709 12:59:53.983703  1569 leader_election.cc:304] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 12:59:53.951537  3999 raft_consensus.cc:3060] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:59:53.989677  1569 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:59:53.994241  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:54.001160  1753 raft_consensus.cc:2468] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 4.
I20260709 12:59:53.998569  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:54.003819  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:54.013144  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:59:54.013933  1679 raft_consensus.cc:2393] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 4: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 12:59:54.012874  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:54.019531  1671 raft_consensus.cc:1275] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:59:54.020176  1717 leader_election.cc:304] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 12:59:54.022935  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:59:53.933974  4140 consensus_queue.cc:1048] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.107s
I20260709 12:59:53.969354  4136 raft_consensus.cc:740] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 12:59:54.027405  1680 raft_consensus.cc:2468] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 5.
I20260709 12:59:54.025380  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:54.029119  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:54.030830  1747 raft_consensus.cc:2410] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for term 7 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260709 12:59:54.032083  1672 raft_consensus.cc:1275] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260709 12:59:54.004575  4099 leader_election.cc:290] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 4 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:54.027117  3977 raft_consensus.cc:3060] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:59:54.059600  1643 leader_election.cc:304] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:54.062917  1718 leader_election.cc:304] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:54.065589  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:54.065878  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:54.066406  1679 raft_consensus.cc:2468] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 2.
I20260709 12:59:54.072453  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:54.061343  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:54.063375  4103 raft_consensus.cc:2749] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:59:54.088344  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:54.099498  1679 raft_consensus.cc:2468] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 4.
I20260709 12:59:54.083185  4136 consensus_queue.cc:260] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:54.093415  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:54.093674  1643 leader_election.cc:304] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 12:59:54.094300  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:54.069586  3521 consensus_queue.cc:1048] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.032s
I20260709 12:59:54.102711  1680 raft_consensus.cc:2468] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 3.
I20260709 12:59:54.101948  1678 raft_consensus.cc:2468] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 4.
I20260709 12:59:54.094841  1569 leader_election.cc:304] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 12:59:54.104547  1569 leader_election.cc:304] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 12:59:54.071224  4099 raft_consensus.cc:2804] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:59:54.119931  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:54.102730  4142 raft_consensus.cc:2749] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260709 12:59:54.122565  4153 log.cc:927] Time spent T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.098s	user 0.001s	sys 0.000s
I20260709 12:59:54.102977  4155 raft_consensus.cc:2804] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:59:54.124104  4164 raft_consensus.cc:2804] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:59:54.124181  4155 raft_consensus.cc:697] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 12:59:54.124751  4164 raft_consensus.cc:493] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 12:59:54.125181  4164 raft_consensus.cc:3060] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:59:54.125433  4155 consensus_queue.cc:237] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:54.126709  4157 raft_consensus.cc:493] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:54.127295  4157 raft_consensus.cc:515] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:54.128710  1643 leader_election.cc:304] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 12:59:54.129485  4103 raft_consensus.cc:2749] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:59:54.129698  4157 leader_election.cc:290] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:54.130298  4103 raft_consensus.cc:2804] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:59:54.130739  4103 raft_consensus.cc:493] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 12:59:54.131105  4103 raft_consensus.cc:3060] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:59:54.106123  1749 raft_consensus.cc:3060] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:59:54.149585  2331 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49082 (request call id 573) took 1583 ms (client timeout 1816 ms). Trace:
I20260709 12:59:54.150790  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:54.151490  1679 raft_consensus.cc:2468] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 6.
I20260709 12:59:54.151628  1678 raft_consensus.cc:1275] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20260709 12:59:54.152652  1679 raft_consensus.cc:1240] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 4. Current term is 5. Ops: []
I20260709 12:59:54.153208  1678 raft_consensus.cc:1275] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:59:54.153926  4027 leader_election.cc:290] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:54.154034  3952 raft_consensus.cc:493] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 12:59:54.154742  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:59:54.154637  3952 raft_consensus.cc:515] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } }
I20260709 12:59:54.157649  1596 raft_consensus.cc:606] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Received request to transfer leadership to TS 816fadcea8fc4893882f5f9b9f22a03d
I20260709 12:59:54.158142  1596 raft_consensus.cc:614] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:59:54.159281  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:54.161473  2333 raft_consensus.cc:1275] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 5 index: 8. (index mismatch)
I20260709 12:59:54.165858  1745 raft_consensus.cc:2468] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 4.
I20260709 12:59:54.176643  1750 raft_consensus.cc:1275] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 8. (index mismatch)
I20260709 12:59:54.172645  2301 leader_election.cc:304] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:54.119071  4102 raft_consensus.cc:2804] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader election won for term 5
W20260709 12:59:54.181454  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 103118us
I20260709 12:59:54.187885  1718 leader_election.cc:304] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:59:54.195719  1643 leader_election.cc:304] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 12:59:54.202939  1680 raft_consensus.cc:2468] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 4.
I20260709 12:59:54.206189  3977 raft_consensus.cc:2749] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:59:54.207535  4126 consensus_queue.cc:1048] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:59:54.210484  1643 leader_election.cc:304] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 12:59:54.210846  4106 raft_consensus.cc:515] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:54.211315  1746 raft_consensus.cc:1275] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260709 12:59:54.211926  1643 leader_election.cc:304] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:54.213207  4106 leader_election.cc:290] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 4 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:54.225895  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:54.227535  4150 consensus_queue.cc:1048] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.122s
I20260709 12:59:54.212953  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:54.231756  1599 raft_consensus.cc:2468] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 4.
I20260709 12:59:54.214282  3948 leader_election.cc:290] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 12:59:54.183516  4138 consensus_queue.cc:1175] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "af520b67945d443b901bbeca37cc5701" responder_term: 3 status { last_received { term: 3 index: 5 } last_committed_idx: 5 last_received_current_leader { term: 3 index: 5 } }
I20260709 12:59:54.215185  3952 leader_election.cc:290] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:54.191090  4027 raft_consensus.cc:2804] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:59:54.175488  3980 consensus_queue.cc:1048] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.041s
I20260709 12:59:54.163364  4099 raft_consensus.cc:697] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 12:59:54.221787  4098 raft_consensus.cc:515] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 12:59:54.150157  2331 rpcz_store.cc:269] 0709 12:59:52.566362 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:52.566561 (+   199us) service_pool.cc:224] Handling call
0709 12:59:52.936401 (+369840us) spinlock_profiling.cc:243] Waited 2.78 ms on lock 0x7b5800178598. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f426303de9c 00007f426303da5b 00007f426bf56c09 00007f426b0b0ee0 00007f426b17a0ab 00007f426b17a131 00007f426b1280a0 00007f426b127984 00007f42646a26d4 00007f42646a2662
0709 12:59:54.149366 (+1212965us) spinlock_profiling.cc:243] Waited 734 ms on lock 0x7b5400507b54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 12:59:54.149499 (+   133us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1695523968}
I20260709 12:59:54.223165  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
W20260709 12:59:54.239172  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 118763us
I20260709 12:59:54.180985  4136 raft_consensus.cc:3060] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:59:54.251854  2333 raft_consensus.cc:1275] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:59:54.257519  2298 leader_election.cc:304] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
W20260709 12:59:54.244640  3964 consensus_meta.cc:287] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.509s	user 0.009s	sys 0.003s
I20260709 12:59:54.260229  3964 raft_consensus.cc:515] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:54.265375  3964 leader_election.cc:290] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 5 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:54.266559  1753 raft_consensus.cc:2393] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 5: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 12:59:54.244021  4107 raft_consensus.cc:2749] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:59:54.243441  3976 raft_consensus.cc:2804] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:59:54.245733  4102 raft_consensus.cc:697] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 12:59:54.251492  3948 raft_consensus.cc:2749] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20260709 12:59:54.283046  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 95226.6us
I20260709 12:59:54.279276  3846 consensus_queue.cc:1059] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.032s
I20260709 12:59:54.254274  3854 raft_consensus.cc:2804] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:59:54.285243  3976 raft_consensus.cc:493] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 12:59:54.326838  1749 raft_consensus.cc:2468] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 5.
I20260709 12:59:54.327708  4171 raft_consensus.cc:2804] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:59:54.328255  4171 raft_consensus.cc:493] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:59:54.328403  1672 raft_consensus.cc:1240] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 1. Current term is 2. Ops: []
I20260709 12:59:54.328675  4171 raft_consensus.cc:3060] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:59:54.269621  3999 raft_consensus.cc:515] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:54.331652  1754 raft_consensus.cc:2368] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 3. Current term is 4.
I20260709 12:59:54.334352  3999 leader_election.cc:290] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 4 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:54.271795  4028 raft_consensus.cc:2804] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:59:54.247434  4126 consensus_queue.cc:1048] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.065s
W20260709 12:59:54.340087  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 104126us
I20260709 12:59:54.313378  4139 raft_consensus.cc:493] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:54.347040  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:54.348260  1672 raft_consensus.cc:2468] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 5.
I20260709 12:59:54.311545  4027 raft_consensus.cc:697] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 4 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 12:59:54.261883  3952 consensus_queue.cc:1048] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.072s
I20260709 12:59:54.271227  3979 consensus_queue.cc:1048] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.078s
I20260709 12:59:54.355218  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:59:54.355859  1671 raft_consensus.cc:3060] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:59:54.333657  3976 raft_consensus.cc:3060] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:59:54.343199  4028 raft_consensus.cc:493] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:59:54.344208  4070 raft_consensus.cc:493] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:59:54.349562  4105 consensus_meta.cc:287] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.614s	user 0.009s	sys 0.004s
I20260709 12:59:54.343499  3854 raft_consensus.cc:697] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 3 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 12:59:54.365999  4105 raft_consensus.cc:515] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:54.357610  4107 consensus_queue.cc:1048] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.013s
I20260709 12:59:54.367581  4070 raft_consensus.cc:515] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 12:59:54.371794  4161 log.cc:927] Time spent T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.051s	user 0.000s	sys 0.000s
I20260709 12:59:54.369307  3999 raft_consensus.cc:493] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 12:59:54.374521  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:59:54.387665  1672 raft_consensus.cc:2393] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 5: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 12:59:54.352240  4139 raft_consensus.cc:515] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:54.361265  4028 raft_consensus.cc:3060] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:59:54.375689  4074 consensus_meta.cc:287] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.568s	user 0.009s	sys 0.003s
I20260709 12:59:54.384449  3993 leader_election.cc:290] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:54.380575  3999 raft_consensus.cc:515] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:54.403313  4133 raft_consensus.cc:493] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 12:59:54.411317  4133 raft_consensus.cc:515] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 12:59:54.399592  4117 log.cc:927] Time spent T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.912s	user 0.002s	sys 0.002s
I20260709 12:59:54.418366  3999 leader_election.cc:290] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:54.425253  1677 raft_consensus.cc:1275] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260709 12:59:54.427196  3854 consensus_queue.cc:237] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:54.427816  4155 raft_consensus.cc:493] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:54.428467  4155 raft_consensus.cc:515] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:54.374665  4099 consensus_queue.cc:237] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } }
I20260709 12:59:54.437649  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:54.381572  4102 consensus_queue.cc:237] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:54.440169  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:54.443128  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:59:54.400625  4027 consensus_queue.cc:237] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:54.445636  4155 leader_election.cc:290] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:54.446067  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:54.446929  1675 raft_consensus.cc:2393] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 2: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 12:59:54.435127  4098 leader_election.cc:290] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 5 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:54.447952  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:54.448787  1672 raft_consensus.cc:2393] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 6: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 12:59:54.449599  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 12:59:54.437783  2300 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:59:54.451395  4139 leader_election.cc:290] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:54.451957  4139 raft_consensus.cc:493] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:54.437340  4164 raft_consensus.cc:515] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:54.473188  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:59:54.476677  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:54.477381  1677 raft_consensus.cc:2368] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 3. Current term is 4.
I20260709 12:59:54.480309  3999 raft_consensus.cc:493] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:54.504365  1671 raft_consensus.cc:2468] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 4.
I20260709 12:59:54.480978  3999 raft_consensus.cc:515] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:54.478763  3993 raft_consensus.cc:493] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:54.457645  4139 raft_consensus.cc:515] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:54.580214  3978 raft_consensus.cc:3055] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:59:54.581067  3978 raft_consensus.cc:740] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 12:59:54.587770  4133 leader_election.cc:290] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 12:59:54.596784  1646 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:59:54.549666  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:54.551848  1718 leader_election.cc:304] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
W20260709 12:59:54.611563  1717 leader_election.cc:336] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 1.720s (SENT)
W20260709 12:59:54.611958  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 60259.4us
I20260709 12:59:54.614964  3979 raft_consensus.cc:2749] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:59:54.567509  4105 leader_election.cc:290] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 5 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:54.559628  4164 leader_election.cc:290] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 3 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:54.625460  4150 consensus_queue.cc:1048] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.006s
W20260709 12:59:54.653380  4181 log.cc:927] Time spent T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.196s	user 0.000s	sys 0.001s
I20260709 12:59:54.659664  4177 raft_consensus.cc:493] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:54.660676  4177 raft_consensus.cc:515] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:54.664476  4096 raft_consensus.cc:493] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:54.665418  4096 raft_consensus.cc:515] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:54.670327  4096 leader_election.cc:290] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:54.609273  4133 raft_consensus.cc:493] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:54.565656  3993 raft_consensus.cc:515] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 12:59:54.716894  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 435590us
I20260709 12:59:54.664649  4139 leader_election.cc:290] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:54.613516  1718 leader_election.cc:304] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
W20260709 12:59:54.737663  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 124499us
I20260709 12:59:54.751070  1718 leader_election.cc:304] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 12:59:54.616796  3978 consensus_queue.cc:260] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:54.766374   431 auto_leader_rebalancer.cc:452] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 12:59:54.702540  4133 raft_consensus.cc:515] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:54.821115  4175 raft_consensus.cc:2804] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260709 12:59:54.771766  4136 consensus_meta.cc:287] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.500s	user 0.003s	sys 0.011s
I20260709 12:59:54.770957  3787 raft_consensus.cc:493] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:59:54.865969  4183 log.cc:927] Time spent T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.114s	user 0.002s	sys 0.000s
W20260709 12:59:54.867528  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 90947.9us
W20260709 12:59:54.875584  1747 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.591s	user 0.020s	sys 0.027s
W20260709 12:59:54.889029  1747 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.607s	user 0.020s	sys 0.027s
I20260709 12:59:54.870499  1678 raft_consensus.cc:2468] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 4.
I20260709 12:59:54.888542  1718 leader_election.cc:304] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:59:54.881176  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 12:59:54.893260  4176 raft_consensus.cc:2804] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:59:54.877537  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:54.894925  1676 leader_election.cc:290] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 5 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:54.895601  1718 leader_election.cc:304] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:59:54.848196  4175 raft_consensus.cc:493] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:59:54.916965  1751 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.579s	user 0.008s	sys 0.019s
I20260709 12:59:54.919313  1718 leader_election.cc:304] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 12:59:54.912910  1676 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RunLeaderElection from 127.0.0.1:52766 (request call id 2080) took 1300 ms. Trace:
I20260709 12:59:54.860162  4150 raft_consensus.cc:2749] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:59:54.914343  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: true dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 12:59:54.921054  2334 raft_consensus.cc:3060] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:59:54.755052  3979 consensus_queue.cc:1048] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.112s
I20260709 12:59:54.893967  4176 raft_consensus.cc:493] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:59:54.924012  1676 rpcz_store.cc:276] 0709 12:59:53.597400 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:53.603378 (+  5978us) service_pool.cc:224] Handling call
0709 12:59:54.878975 (+1275597us) spinlock_profiling.cc:243] Waited 51.7 ms on lock 0x7b54001e7dd4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266159a3d 00007f426b129cd0 00007f42646ae254 00007f42646ae1e2 00007f42646ae111 00007f42646ae08c 00007f42646ad342 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 12:59:54.897498 (+ 18523us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":118994176}
I20260709 12:59:54.925642  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:54.928536  1680 raft_consensus.cc:2468] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 6.
W20260709 12:59:54.895892  4140 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.506s	user 0.004s	sys 0.019s
I20260709 12:59:54.950973  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:54.908787  4155 raft_consensus.cc:493] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:54.952294   431 auto_leader_rebalancer.cc:527] All tables' leader rebalancing finished this round
W20260709 12:59:54.919023  1751 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.581s	user 0.008s	sys 0.019s
I20260709 12:59:54.866369  3787 raft_consensus.cc:515] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:54.929535  3999 leader_election.cc:290] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 12:59:54.990362  3854 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.545s	user 0.013s	sys 0.017s
I20260709 12:59:54.923822  4175 raft_consensus.cc:3060] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:59:54.989666  2331 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.605s	user 0.014s	sys 0.046s
I20260709 12:59:54.914016  4164 raft_consensus.cc:493] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 12:59:54.993266  2334 raft_consensus.cc:2468] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 5.
I20260709 12:59:54.917037  4105 raft_consensus.cc:493] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 12:59:54.912995  3978 raft_consensus.cc:3060] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:59:54.996191  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
W20260709 12:59:55.006672  2300 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:59:55.006402  2333 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.554s	user 0.023s	sys 0.010s
W20260709 12:59:55.008221  1719 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.778s	user 0.013s	sys 0.016s
I20260709 12:59:55.012080  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:54.930444  4176 raft_consensus.cc:3060] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:59:54.765911  3976 raft_consensus.cc:515] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 12:59:55.011868  4028 consensus_meta.cc:287] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.621s	user 0.017s	sys 0.008s
I20260709 12:59:54.997773  4150 raft_consensus.cc:2804] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260709 12:59:55.036545  1717 leader_election.cc:336] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 1.539s (SENT)
W20260709 12:59:55.035697  2333 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.583s	user 0.025s	sys 0.010s
I20260709 12:59:55.040467  1717 leader_election.cc:304] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 12:59:55.036037  1719 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.806s	user 0.014s	sys 0.016s
W20260709 12:59:55.003850  4140 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.628s	user 0.004s	sys 0.020s
I20260709 12:59:55.042559  1680 raft_consensus.cc:1240] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 4. Current term is 5. Ops: []
I20260709 12:59:55.042973  3993 leader_election.cc:290] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 12:59:55.027706  2331 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.643s	user 0.015s	sys 0.047s
I20260709 12:59:55.016903  3948 raft_consensus.cc:493] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 12:59:55.053488  2332 raft_consensus.cc:2468] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 3.
I20260709 12:59:55.056295  1676 raft_consensus.cc:1240] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 4. Current term is 5. Ops: []
I20260709 12:59:55.052241  3787 leader_election.cc:290] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:55.059265  1672 raft_consensus.cc:2410] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 12:59:55.059410  4132 raft_consensus.cc:493] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 12:59:55.060119  4132 raft_consensus.cc:515] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:55.045449  4150 raft_consensus.cc:493] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:59:55.062886  4150 raft_consensus.cc:3060] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:59:55.025429  3854 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.573s	user 0.013s	sys 0.018s
I20260709 12:59:55.045735  4177 leader_election.cc:290] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 12:59:55.069211  1719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:59:55.064428  4102 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.583s	user 0.004s	sys 0.023s
W20260709 12:59:55.059430  4169 log.cc:927] Time spent T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.685s	user 0.006s	sys 0.012s
I20260709 12:59:55.081257  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:55.034848  4164 raft_consensus.cc:515] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 12:59:55.076368  4171 consensus_meta.cc:287] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.747s	user 0.007s	sys 0.012s
W20260709 12:59:55.137264  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 53789.3us
W20260709 12:59:55.078702  4103 consensus_meta.cc:287] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.947s	user 0.011s	sys 0.005s
I20260709 12:59:55.042985  4105 raft_consensus.cc:515] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 12:59:55.080457  4102 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.599s	user 0.005s	sys 0.023s
I20260709 12:59:55.150193  3976 leader_election.cc:290] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 4 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:55.151435  1752 raft_consensus.cc:1275] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 12:59:55.152057  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:55.152949  3979 raft_consensus.cc:493] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 12:59:55.155540  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:55.025626  4155 raft_consensus.cc:515] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:55.139106  4177 raft_consensus.cc:2749] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:59:55.134774  4133 leader_election.cc:290] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:55.207427  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:55.157130  3999 raft_consensus.cc:2749] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:59:55.201352  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:55.068248  3948 raft_consensus.cc:515] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 12:59:55.181447  4173 thread.cc:641] apply [worker] (thread pool) Time spent creating pthread: real 0.679s	user 0.012s	sys 0.022s
I20260709 12:59:55.048837  4028 raft_consensus.cc:515] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 12:59:55.209748  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 52025.9us
I20260709 12:59:55.205421  3979 raft_consensus.cc:515] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:55.211978  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
W20260709 12:59:55.216724  1646 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:59:55.243291  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:55.204658  1749 raft_consensus.cc:3060] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:59:55.242707  1753 raft_consensus.cc:2468] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 4.
I20260709 12:59:55.204232  4132 leader_election.cc:290] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:55.250020  1752 raft_consensus.cc:2393] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 4: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 12:59:55.158073  4103 raft_consensus.cc:515] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:55.154850  4171 raft_consensus.cc:515] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:55.262859  1597 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 400) took 1424 ms. Trace:
W20260709 12:59:55.262753  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 343480us
I20260709 12:59:55.263510  1597 rpcz_store.cc:276] 0709 12:59:53.838803 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:53.838947 (+   144us) service_pool.cc:224] Handling call
0709 12:59:53.839332 (+   385us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:59:53.876836 (+ 37504us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:59:53.876876 (+    40us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:59:53.889552 (+ 12676us) log.cc:844] Serialized 26 byte log entry
0709 12:59:54.166233 (+276681us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:59:54.166294 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:54.166392 (+    98us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:59:54.898410 (+732018us) spinlock_profiling.cc:243] Waited 4.16 ms on lock 0x7b4400022b4c. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f4262f9bc85 00007f4262f9c6a2 00007f4265d0e1db 00007f4265d09aaf 00007f4265cfd86e 00007f4265cfcbb6 00007f4265cfc7d8 00007f4265d06221 00007f4265d061d9 00007f4265d06169
0709 12:59:55.262520 (+364110us) raft_consensus.cc:1685] finished
0709 12:59:55.262545 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:59:55.262787 (+   242us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":37058,"raft.queue_time_us":37375,"raft.run_cpu_time_us":139,"raft.run_wall_time_us":9833,"spinlock_wait_cycles":9571200,"thread_start_us":256485,"threads_started":1,"wal-append.queue_time_us":418145}
I20260709 12:59:55.263974  1718 leader_election.cc:304] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
W20260709 12:59:55.217617  4173 thread.cc:608] apply [worker] (thread pool) Time spent starting thread: real 0.823s	user 0.012s	sys 0.023s
I20260709 12:59:55.267989  1569 leader_election.cc:304] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 12:59:55.275198  1569 leader_election.cc:304] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 12:59:55.287029  2301 leader_election.cc:304] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 12:59:55.287559  1569 leader_election.cc:304] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 12:59:55.290524  1679 raft_consensus.cc:1240] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 5. Current term is 6. Ops: []
I20260709 12:59:55.291059  1718 leader_election.cc:304] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 12:59:55.261653  4190 log.cc:927] Time spent T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.115s	user 0.000s	sys 0.001s
I20260709 12:59:55.261657  4180 raft_consensus.cc:493] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:55.244858  3854 consensus_queue.cc:1048] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.047s
I20260709 12:59:55.298468  4155 leader_election.cc:290] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:55.300506  4171 leader_election.cc:290] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:55.276973  4090 raft_consensus.cc:493] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:55.308789  4103 leader_election.cc:290] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 6 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:55.309368  4090 raft_consensus.cc:515] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:55.277271  3521 raft_consensus.cc:2804] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:59:55.311125  4105 leader_election.cc:290] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:55.311920  4090 leader_election.cc:290] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:55.312526  3979 leader_election.cc:290] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:55.249176  4133 consensus_queue.cc:1048] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.042s
I20260709 12:59:55.304451  1676 raft_consensus.cc:1275] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 12:59:55.311173  3521 raft_consensus.cc:493] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 12:59:55.287537  4182 raft_consensus.cc:2804] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:59:55.306651  1569 leader_election.cc:304] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 12:59:55.277670  4102 raft_consensus.cc:493] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 12:59:55.288290  1746 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55848 (request call id 1847) took 1059 ms. Trace:
I20260709 12:59:55.293123  4104 raft_consensus.cc:2804] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:59:55.318477  3521 raft_consensus.cc:3060] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:59:55.328867  1748 raft_consensus.cc:1275] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 6. (term mismatch)
I20260709 12:59:55.329310  1748 pending_rounds.cc:85] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701: Aborting all ops after (but not including) 5
I20260709 12:59:55.329550  1748 pending_rounds.cc:107] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701: Aborting uncommitted NO_OP operation due to leader change: 4.6
I20260709 12:59:55.329988  1748 raft_consensus.cc:2889] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:59:55.331627  1718 leader_election.cc:400] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 election: Vote denied by peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261) with higher term. Message: Invalid argument: T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 3. Current term is 4.
I20260709 12:59:55.332049  1718 leader_election.cc:403] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20260709 12:59:55.317232  4137 raft_consensus.cc:2804] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:59:55.319016  3787 raft_consensus.cc:493] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:55.323429  3979 raft_consensus.cc:3060] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:59:55.323328  4104 raft_consensus.cc:493] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:59:55.306051  4177 consensus_queue.cc:1048] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.028s
I20260709 12:59:55.318169  4171 consensus_queue.cc:1048] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.030s
I20260709 12:59:55.325985  4182 raft_consensus.cc:697] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 12:59:55.363392  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:55.364169  1597 raft_consensus.cc:2468] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 7.
I20260709 12:59:55.360947  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:55.366290  1602 raft_consensus.cc:2468] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 3.
I20260709 12:59:55.325235  4105 consensus_queue.cc:1059] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.032s
I20260709 12:59:55.369136  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:55.318529  4180 raft_consensus.cc:515] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:55.370071  1602 raft_consensus.cc:2468] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 3.
I20260709 12:59:55.372390  1676 raft_consensus.cc:1240] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260709 12:59:55.376832  1676 raft_consensus.cc:1275] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260709 12:59:55.368526  4155 raft_consensus.cc:2804] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:59:55.378626  4164 leader_election.cc:290] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:55.378921  4155 raft_consensus.cc:493] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:59:55.379421  4155 raft_consensus.cc:3060] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:59:55.356592  3978 raft_consensus.cc:2749] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261) with higher term. Message: Invalid argument: T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 3. Current term is 4.
I20260709 12:59:55.323482  1746 rpcz_store.cc:276] 0709 12:59:54.219529 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:54.237942 (+ 18413us) service_pool.cc:224] Handling call
0709 12:59:54.283557 (+ 45615us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:59:54.334383 (+ 50826us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:59:54.747316 (+412933us) log.cc:844] Serialized 14 byte log entry
0709 12:59:55.145674 (+398358us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:59:55.150061 (+  4387us) log.cc:844] Serialized 52 byte log entry
0709 12:59:55.150369 (+   308us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:59:55.150469 (+   100us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:55.150616 (+   147us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:59:55.260056 (+109440us) raft_consensus.cc:1685] finished
0709 12:59:55.260080 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:59:55.279180 (+ 19100us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5687,"prepare.queue_time_us":802555,"prepare.run_cpu_time_us":2169,"prepare.run_wall_time_us":17347,"raft.queue_time_us":112909,"raft.run_cpu_time_us":309,"raft.run_wall_time_us":12535,"thread_start_us":735510,"threads_started":2,"wal-append.queue_time_us":487884}
I20260709 12:59:55.359153  4104 raft_consensus.cc:3060] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:59:55.392814  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:55.368412  1749 raft_consensus.cc:2468] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 5.
I20260709 12:59:55.395817  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:55.356774  3787 raft_consensus.cc:515] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:55.358894  4144 raft_consensus.cc:2749] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:59:55.330359  4102 raft_consensus.cc:515] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } }
W20260709 12:59:55.371382  4191 log.cc:927] Time spent T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.124s	user 0.000s	sys 0.001s
I20260709 12:59:55.410365  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:59:55.411130  1674 raft_consensus.cc:3060] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:59:55.401443  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 114471us
I20260709 12:59:55.391399  4154 raft_consensus.cc:3055] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:59:55.421721  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:55.423403  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:55.424472  2332 raft_consensus.cc:2368] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 4. Current term is 5.
W20260709 12:59:55.406286  4192 log.cc:927] Time spent T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.207s	user 0.000s	sys 0.001s
I20260709 12:59:55.431722  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:59:55.433707  1605 raft_consensus.cc:3060] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:59:55.438297  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:55.439100  1677 raft_consensus.cc:2393] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 3: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 12:59:55.433805  3978 raft_consensus.cc:493] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 12:59:55.441663  4182 consensus_queue.cc:237] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:55.423095  4171 raft_consensus.cc:493] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
W20260709 12:59:55.458112  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 215855us
I20260709 12:59:55.426448  4154 raft_consensus.cc:740] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 12:59:55.431406  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:55.458091  1602 raft_consensus.cc:3055] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:59:55.435330  4203 raft_consensus.cc:493] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 12:59:55.442423  3978 raft_consensus.cc:515] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:55.468446  2298 leader_election.cc:304] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 12:59:55.471489  2298 leader_election.cc:304] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 12:59:55.446627  4184 raft_consensus.cc:493] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 12:59:55.463147  1602 raft_consensus.cc:740] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 12:59:55.478564  1751 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55848 (request call id 1845) took 1313 ms. Trace:
I20260709 12:59:55.478974  1751 rpcz_store.cc:276] 0709 12:59:54.165323 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:54.176014 (+ 10691us) service_pool.cc:224] Handling call
0709 12:59:54.176423 (+   409us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:59:54.332077 (+155654us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:59:54.332123 (+    46us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:59:54.337451 (+  5328us) log.cc:844] Serialized 26 byte log entry
0709 12:59:55.015984 (+678533us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:59:55.016059 (+    75us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:55.016158 (+    99us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:59:55.478322 (+462164us) raft_consensus.cc:1685] finished
0709 12:59:55.478343 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:59:55.478494 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":150320,"raft.run_cpu_time_us":284,"raft.run_wall_time_us":287,"thread_start_us":817994,"threads_started":2,"wal-append.queue_time_us":805801}
I20260709 12:59:55.481262  1602 consensus_queue.cc:260] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:55.447810  3977 consensus_queue.cc:1048] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.029s
I20260709 12:59:55.471611  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:55.495025  2298 leader_election.cc:304] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 12:59:55.445465  4200 consensus_queue.cc:1059] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.043s
I20260709 12:59:55.480233  4144 raft_consensus.cc:2804] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:59:55.463572  4171 raft_consensus.cc:515] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:55.509099  1749 raft_consensus.cc:2468] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 4.
I20260709 12:59:55.502558  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:55.516456  1752 raft_consensus.cc:2468] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 6.
I20260709 12:59:55.518553  3993 raft_consensus.cc:493] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:55.519170  3800 raft_consensus.cc:493] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:55.519282  3993 raft_consensus.cc:515] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:55.520032  3800 raft_consensus.cc:515] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 12:59:55.500980  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 177148us
I20260709 12:59:55.475822  1746 raft_consensus.cc:2468] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 3.
I20260709 12:59:55.501833  2299 leader_election.cc:304] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 12:59:55.500721  4137 raft_consensus.cc:697] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 12:59:55.495066  1605 raft_consensus.cc:2468] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 5.
I20260709 12:59:55.501554  4147 raft_consensus.cc:493] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:55.502863  4133 raft_consensus.cc:2804] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:59:55.512202  4144 raft_consensus.cc:493] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:59:55.503651  4105 raft_consensus.cc:493] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:55.524709  1569 leader_election.cc:304] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 12:59:55.522702  4090 raft_consensus.cc:493] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 12:59:55.532594  4200 raft_consensus.cc:493] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 12:59:55.533136  4180 leader_election.cc:290] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:55.544438  4147 raft_consensus.cc:515] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:55.552309  4144 raft_consensus.cc:3060] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:59:55.553177  4171 leader_election.cc:290] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:55.532249  4209 raft_consensus.cc:2804] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260709 12:59:55.559957  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 100528us
I20260709 12:59:55.481663  4184 raft_consensus.cc:515] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:55.479020  4203 raft_consensus.cc:515] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:55.538919  4133 raft_consensus.cc:493] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:59:55.569571  4147 leader_election.cc:290] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:55.571959  3521 raft_consensus.cc:515] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:55.572888  4070 leader_election.cc:290] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:55.563584  4105 raft_consensus.cc:515] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:55.574241  3800 leader_election.cc:290] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:55.574923  4223 raft_consensus.cc:2804] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:59:55.559114  1674 raft_consensus.cc:2468] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 5.
I20260709 12:59:55.563459  1569 leader_election.cc:304] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 12:59:55.579473  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:55.581521  4218 raft_consensus.cc:493] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 12:59:55.532370  4206 raft_consensus.cc:3055] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:59:55.569432  4069 raft_consensus.cc:2749] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260709 12:59:55.582265  4213 log.cc:927] Time spent T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.076s	user 0.001s	sys 0.000s
I20260709 12:59:55.544808  3787 leader_election.cc:290] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:55.594287  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:55.608125  1674 raft_consensus.cc:2468] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 3.
I20260709 12:59:55.599515  4137 consensus_queue.cc:237] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 12:59:55.600338  4202 log.cc:927] Time spent T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.169s	user 0.001s	sys 0.000s
I20260709 12:59:55.607198  4206 raft_consensus.cc:740] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 12:59:55.614843  1678 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52778 (request call id 1703) took 1177 ms. Trace:
I20260709 12:59:55.615252  1678 rpcz_store.cc:276] 0709 12:59:54.437169 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:54.437310 (+   141us) service_pool.cc:224] Handling call
0709 12:59:55.614649 (+1177339us) spinlock_profiling.cc:243] Waited 432 ms on lock 0x7b540025b054. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 12:59:55.614778 (+   129us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":993379200}
I20260709 12:59:55.616528  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:55.615758  4206 consensus_queue.cc:260] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } }
I20260709 12:59:55.617429  1678 raft_consensus.cc:2410] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for term 6 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260709 12:59:55.585853  4102 leader_election.cc:290] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:55.553757  4176 raft_consensus.cc:515] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 12:59:55.609127  1717 leader_election.cc:336] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 1.864s (SENT)
I20260709 12:59:55.583082  4209 raft_consensus.cc:493] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 12:59:55.602509  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:55.630178  1717 leader_election.cc:304] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:55.513187  4154 consensus_queue.cc:260] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:55.630645  1673 raft_consensus.cc:2468] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 4.
I20260709 12:59:55.589910  4133 raft_consensus.cc:3060] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:59:55.564944  4090 raft_consensus.cc:515] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:55.604818  4070 raft_consensus.cc:493] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:59:55.632889  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 85026.4us
I20260709 12:59:55.607963  4182 raft_consensus.cc:2804] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:59:55.640792  2298 leader_election.cc:304] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 12:59:55.647469  4147 raft_consensus.cc:2804] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:59:55.648258  4147 raft_consensus.cc:697] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 5 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 12:59:55.649585  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:55.650482  1605 raft_consensus.cc:2468] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 5.
I20260709 12:59:55.649600  4147 consensus_queue.cc:237] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 12:59:55.634590  4150 consensus_meta.cc:287] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.571s	user 0.008s	sys 0.008s
I20260709 12:59:55.611711  4218 raft_consensus.cc:515] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:55.565487  4200 raft_consensus.cc:515] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:55.663316  4212 consensus_queue.cc:1059] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.025s
W20260709 12:59:55.932025  4225 log.cc:927] Time spent T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.060s	user 0.000s	sys 0.000s
W20260709 12:59:56.045993  1717 leader_election.cc:336] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 1.950s (SENT)
I20260709 12:59:55.814370  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:56.047349  1717 leader_election.cc:304] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:56.047788  4218 leader_election.cc:290] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:55.664942  4070 raft_consensus.cc:515] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 12:59:55.988713  1570 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:59:55.802788  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:55.871484  3977 raft_consensus.cc:493] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:55.788254  2298 leader_election.cc:304] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
W20260709 12:59:55.797690  4219 log.cc:927] Time spent T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.119s	user 0.000s	sys 0.001s
I20260709 12:59:55.782001  3521 leader_election.cc:290] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 7 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:55.978473  1646 leader_election.cc:304] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:59:56.047447  1596 raft_consensus.cc:2468] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 4.
W20260709 12:59:56.073975  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 95660.7us
I20260709 12:59:55.798355  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
W20260709 12:59:56.074862  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 286308us
I20260709 12:59:56.076642  2334 raft_consensus.cc:3060] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:59:55.679579  1747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55848 (request call id 1841) took 1543 ms. Trace:
I20260709 12:59:56.075018  1597 raft_consensus.cc:2468] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 5.
I20260709 12:59:55.763772  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:56.078315  1747 rpcz_store.cc:276] 0709 12:59:54.136293 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:54.141351 (+  5058us) service_pool.cc:224] Handling call
0709 12:59:54.141683 (+   332us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:59:54.255995 (+114312us) raft_consensus.cc:1534] Early marking committed up to index 3
0709 12:59:54.889477 (+633482us) log.cc:844] Serialized 14 byte log entry
0709 12:59:55.262507 (+373030us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:59:55.277171 (+ 14664us) log.cc:844] Serialized 26 byte log entry
0709 12:59:55.277359 (+   188us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:59:55.277393 (+    34us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:55.277823 (+   430us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:59:55.679256 (+401433us) raft_consensus.cc:1685] finished
0709 12:59:55.679290 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:59:55.679480 (+   190us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3788,"prepare.queue_time_us":802715,"prepare.run_cpu_time_us":87,"prepare.run_wall_time_us":4398,"raft.queue_time_us":175937,"raft.run_cpu_time_us":359,"raft.run_wall_time_us":36483,"thread_start_us":1054529,"threads_started":3,"wal-append.queue_time_us":451168}
I20260709 12:59:56.080826  1671 raft_consensus.cc:2468] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 5.
I20260709 12:59:56.045073  1569 leader_election.cc:400] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 4 pre-election: Vote denied by peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625) with higher term. Message: Invalid argument: T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 4. Current term is 5.
I20260709 12:59:56.084084  1569 leader_election.cc:403] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
W20260709 12:59:55.835029  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:59:55.799603  4207 log.cc:927] Time spent T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.235s	user 0.000s	sys 0.001s
I20260709 12:59:55.745862  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:55.681527  4171 raft_consensus.cc:2749] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:59:55.665326  4182 raft_consensus.cc:493] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:59:56.049600  4206 raft_consensus.cc:3060] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:59:55.815526  3979 raft_consensus.cc:2749] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260709 12:59:55.958503  4099 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.924s	user 0.006s	sys 0.009s
W20260709 12:59:55.811339  4222 log.cc:927] Time spent T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.136s	user 0.002s	sys 0.000s
I20260709 12:59:55.955627  4090 leader_election.cc:290] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:56.055841  4203 leader_election.cc:290] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 12:59:55.801769  4221 log.cc:927] Time spent T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.120s	user 0.001s	sys 0.000s
I20260709 12:59:55.632416  4209 raft_consensus.cc:3060] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:59:56.042472  4175 consensus_meta.cc:287] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.998s	user 0.012s	sys 0.012s
I20260709 12:59:55.920815  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:55.639305  4102 consensus_queue.cc:1048] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.481s
I20260709 12:59:56.077570  3906 raft_consensus.cc:2804] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:59:55.758617  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 12:59:56.082428  4138 raft_consensus.cc:2804] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:59:55.811021  4184 leader_election.cc:290] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:56.041256  4176 leader_election.cc:290] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 4 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:55.923949  4105 leader_election.cc:290] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:56.083228  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:55.756211  4223 raft_consensus.cc:697] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 12:59:56.093488  4182 raft_consensus.cc:3060] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:59:55.781384  4154 raft_consensus.cc:3060] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:59:56.121186  4099 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.094s	user 0.008s	sys 0.012s
I20260709 12:59:56.150949  2329 raft_consensus.cc:2468] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 5.
I20260709 12:59:56.078080  3977 raft_consensus.cc:515] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 12:59:56.151417  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 106510us
I20260709 12:59:56.151600  4212 raft_consensus.cc:2749] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625) with higher term. Message: Invalid argument: T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 4. Current term is 5.
I20260709 12:59:55.807075  4155 raft_consensus.cc:515] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:56.075057  4150 raft_consensus.cc:515] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:55.805449  4104 raft_consensus.cc:515] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 12:59:56.140791  4144 consensus_meta.cc:287] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.550s	user 0.013s	sys 0.001s
I20260709 12:59:56.153419  3979 raft_consensus.cc:2749] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:59:56.157150  3906 raft_consensus.cc:493] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:59:56.156517  4124 raft_consensus.cc:493] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
W20260709 12:59:56.179692  1571 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 2.125s	user 0.028s	sys 0.030s
W20260709 12:59:56.256963  1571 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 2.202s	user 0.028s	sys 0.030s
I20260709 12:59:56.160951  2331 raft_consensus.cc:3060] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:59:56.226579  1678 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.585s	user 0.018s	sys 0.082s
W20260709 12:59:56.259171  1678 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.617s	user 0.018s	sys 0.082s
I20260709 12:59:56.164906  4171 raft_consensus.cc:493] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:56.176012  4138 raft_consensus.cc:697] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 5 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 12:59:56.265702  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:56.234333  1754 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55848 (request call id 1850) took 1097 ms. Trace:
I20260709 12:59:56.145071  4175 raft_consensus.cc:515] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:56.271512  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:56.272194  1605 raft_consensus.cc:2376] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 5. Re-sending same reply.
I20260709 12:59:56.248009  2334 raft_consensus.cc:2468] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 5.
W20260709 12:59:55.757191  4215 consensus_queue.cc:1175] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "816fadcea8fc4893882f5f9b9f22a03d" responder_term: 5 status { last_received { term: 4 index: 3 } last_committed_idx: 3 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 4. Current term is 5. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260709 12:59:56.180115  2328 raft_consensus.cc:2468] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 3.
W20260709 12:59:56.254675  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 208715us
I20260709 12:59:56.268800  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:56.269577  1597 raft_consensus.cc:2368] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for earlier term 2. Current term is 3.
I20260709 12:59:56.266716  1754 rpcz_store.cc:276] 0709 12:59:55.136815 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:55.136949 (+   134us) service_pool.cc:224] Handling call
0709 12:59:55.144295 (+  7346us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:59:55.410664 (+266369us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:59:55.410695 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:59:55.435415 (+ 24720us) log.cc:844] Serialized 26 byte log entry
0709 12:59:55.490656 (+ 55241us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:59:55.490692 (+    36us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:55.490778 (+    86us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:59:55.800894 (+310116us) spinlock_profiling.cc:243] Waited 1.95 ms on lock 0x7b4400022b4c. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f4262f9bc85 00007f4262f9c6a2 00007f4265d0e1db 00007f4265d09aaf 00007f4265cfd86e 00007f4265cfcbb6 00007f4265cfc7d8 00007f4265d06221 00007f4265d061d9 00007f4265d06169
0709 12:59:56.231520 (+430626us) raft_consensus.cc:1685] finished
0709 12:59:56.231545 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:59:56.234249 (+  2704us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":106211,"raft.run_cpu_time_us":49,"raft.run_wall_time_us":54,"spinlock_wait_cycles":4474624,"thread_start_us":288036,"threads_started":2,"wal-append.queue_time_us":188963}
I20260709 12:59:56.462479  1600 raft_consensus.cc:1275] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 12:59:56.463246  1603 raft_consensus.cc:2468] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 5.
I20260709 12:59:56.256834  3906 raft_consensus.cc:3060] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:59:56.462584  1717 leader_election.cc:336] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 1.966s (SENT)
W20260709 12:59:56.416213  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 187476us
I20260709 12:59:56.464637  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:56.259446  4099 raft_consensus.cc:493] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:56.333379  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
W20260709 12:59:56.331094  4227 log.cc:927] Time spent T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.374s	user 0.001s	sys 0.001s
I20260709 12:59:56.468560  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:56.480440  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:59:56.468170  1717 leader_election.cc:304] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:56.469121  1598 raft_consensus.cc:1275] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 12:59:56.481436  1601 raft_consensus.cc:2393] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 5: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 12:59:56.358850  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:59:56.260488  4102 raft_consensus.cc:493] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
W20260709 12:59:56.322186  4198 log.cc:927] Time spent T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.067s	user 0.001s	sys 0.000s
I20260709 12:59:56.484191  1597 raft_consensus.cc:2468] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 4.
W20260709 12:59:56.273191  4235 log.cc:927] Time spent T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.102s	user 0.001s	sys 0.000s
I20260709 12:59:56.482810  4247 raft_consensus.cc:493] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:56.479730  4155 leader_election.cc:290] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 6 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:56.272589  4175 leader_election.cc:290] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 6 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 12:59:56.358043  4231 log.cc:927] Time spent T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.124s	user 0.000s	sys 0.001s
I20260709 12:59:56.497941  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:56.490847  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: true dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:59:56.499800  1600 raft_consensus.cc:3055] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:59:56.500229  1600 raft_consensus.cc:740] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 12:59:56.280714  2330 raft_consensus.cc:1240] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 4. Current term is 5. Ops: []
W20260709 12:59:56.487453  1717 leader_election.cc:336] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 1.985s (SENT)
I20260709 12:59:56.503512  1717 leader_election.cc:304] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:56.490116  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:59:56.503647  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:56.504506  1604 raft_consensus.cc:2468] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 4.
I20260709 12:59:56.504801  1605 raft_consensus.cc:2410] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for term 6 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 12:59:56.255055  4144 raft_consensus.cc:515] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:56.459398  4104 leader_election.cc:290] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 3 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:56.268302  4070 leader_election.cc:290] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:56.469942  3977 leader_election.cc:290] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:56.483736  4200 leader_election.cc:290] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:56.512291  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:56.276963  3948 leader_election.cc:290] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:56.509047  1600 consensus_queue.cc:260] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:56.487831  4242 raft_consensus.cc:2749] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:59:56.268522  4171 raft_consensus.cc:515] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:56.515233  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:59:56.512828  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:59:56.446352  4138 consensus_queue.cc:237] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 12:59:56.513970  1717 leader_election.cc:336] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 2.122s (SENT)
I20260709 12:59:56.521106  1600 raft_consensus.cc:3060] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:59:56.521862  2336 raft_consensus.cc:2368] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 7. Current term is 8.
I20260709 12:59:56.491144  4247 raft_consensus.cc:515] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } }
I20260709 12:59:56.517656  4175 raft_consensus.cc:3060] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:56.527246  4182 raft_consensus.cc:515] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:56.527794  4199 raft_consensus.cc:3055] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:59:56.264096  4124 raft_consensus.cc:515] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:56.518891  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:56.530774  1601 raft_consensus.cc:2393] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 5: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 12:59:56.530764  1605 raft_consensus.cc:2393] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 5: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 12:59:56.340819  4223 consensus_queue.cc:237] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:56.546608  1603 raft_consensus.cc:2468] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 4.
W20260709 12:59:56.534364  1717 leader_election.cc:336] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 2.541s (SENT)
I20260709 12:59:56.488478  4099 raft_consensus.cc:515] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:56.522505  2330 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49082 (request call id 590) took 1104 ms. Trace:
I20260709 12:59:56.535580  1598 raft_consensus.cc:1275] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
W20260709 12:59:56.530256  4230 log.cc:927] Time spent T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.249s	user 0.000s	sys 0.002s
I20260709 12:59:56.539117  4247 leader_election.cc:290] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:56.539631  4199 raft_consensus.cc:740] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 12:59:56.556529  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:56.559549  1605 raft_consensus.cc:1275] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260709 12:59:56.548156  1717 leader_election.cc:304] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:56.544878  1604 raft_consensus.cc:2468] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 5.
I20260709 12:59:56.550133  2330 rpcz_store.cc:276] 0709 12:59:55.417104 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:55.421821 (+  4717us) service_pool.cc:224] Handling call
0709 12:59:56.280243 (+858422us) spinlock_profiling.cc:243] Waited 6.55 ms on lock 0x7b5800178598. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f426303db71 00007f426303da22 00007f426bf56c09 00007f426b0b0ee0 00007f426b17a0ab 00007f426b17a131 00007f426b13bfc0 00007f426b126929 00007f426b1263d2 00007f426469dbd4
0709 12:59:56.280498 (+   255us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:59:56.507860 (+227362us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:56.521153 (+ 13293us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":15068416}
I20260709 12:59:56.562170  3978 leader_election.cc:290] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:56.499933  4102 raft_consensus.cc:515] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:56.557240  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:56.570102  1601 raft_consensus.cc:1240] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 4. Current term is 5. Ops: []
W20260709 12:59:56.572499  4133 consensus_meta.cc:287] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.926s	user 0.014s	sys 0.000s
I20260709 12:59:56.574909  4171 leader_election.cc:290] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:56.574939  4242 raft_consensus.cc:2749] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:59:56.576213  4209 raft_consensus.cc:515] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:56.576759  1600 raft_consensus.cc:2468] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 5.
I20260709 12:59:56.577616  4144 leader_election.cc:290] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 8 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 12:59:56.581619  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 47284.8us
I20260709 12:59:56.587949  4124 leader_election.cc:290] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:56.588521  4102 leader_election.cc:290] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 12:59:56.590097  4195 log.cc:927] Time spent T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 1.303s	user 0.002s	sys 0.000s
I20260709 12:59:56.567978  4199 consensus_queue.cc:260] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:56.591311  4199 raft_consensus.cc:3060] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:59:56.586172  4133 raft_consensus.cc:515] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:56.603387  1749 raft_consensus.cc:2410] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 12:59:56.605435  4133 leader_election.cc:290] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 4 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 12:59:56.610014  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:59:56.594336  2299 leader_election.cc:304] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 12:59:56.616153  2299 leader_election.cc:304] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:56.620231  2299 leader_election.cc:304] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 12:59:56.630306  3793 raft_consensus.cc:2804] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:59:56.630754  3793 raft_consensus.cc:493] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:59:56.630856  2333 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39726 (request call id 732) took 2215 ms. Trace:
I20260709 12:59:56.631155  3793 raft_consensus.cc:3060] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:59:56.631361  2333 rpcz_store.cc:276] 0709 12:59:54.415435 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:54.415625 (+   190us) service_pool.cc:224] Handling call
0709 12:59:54.416095 (+   470us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:59:54.431065 (+ 14970us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:59:54.431096 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:59:54.431614 (+   518us) log.cc:844] Serialized 26 byte log entry
0709 12:59:55.040951 (+609337us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:59:55.041004 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:55.041071 (+    67us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:59:56.589471 (+1548400us) spinlock_profiling.cc:243] Waited 3.66 ms on lock 0x7b4400022b4c. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f4262f9bc85 00007f4262f9c6a2 00007f4265d0e1db 00007f4265d09aaf 00007f4265cfd86e 00007f4265cfcbb6 00007f4265cfc7d8 00007f4265d06221 00007f4265d061d9 00007f4265d06169
0709 12:59:56.630542 (+ 41071us) raft_consensus.cc:1685] finished
0709 12:59:56.630568 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:59:56.630781 (+   213us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":14459,"raft.queue_time_us":15304,"raft.run_cpu_time_us":151,"raft.run_wall_time_us":3546,"spinlock_wait_cycles":8414592,"thread_start_us":588318,"threads_started":1,"wal-append.queue_time_us":771733}
I20260709 12:59:56.629076  4209 leader_election.cc:290] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 4 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:56.635385  1644 leader_election.cc:304] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 12:59:56.635737  2331 raft_consensus.cc:2468] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 4.
W20260709 12:59:56.636759  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:59:56.641867  1750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55848 (request call id 1855) took 1494 ms. Trace:
I20260709 12:59:56.643077  4184 raft_consensus.cc:2804] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:59:56.642426  1750 rpcz_store.cc:276] 0709 12:59:55.147361 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:55.154242 (+  6881us) service_pool.cc:224] Handling call
0709 12:59:55.154562 (+   320us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:59:55.245121 (+ 90559us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:59:55.245163 (+    42us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:59:55.245206 (+    43us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:59:55.245251 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:56.641563 (+1396312us) spinlock_profiling.cc:243] Waited 90.3 ms on lock 0x7b5400324bd4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 12:59:56.641631 (+    68us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:59:56.641790 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":207684352}
I20260709 12:59:56.643640  4184 raft_consensus.cc:697] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 5 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 12:59:56.624352  4144 raft_consensus.cc:2804] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:59:56.619805  4133 raft_consensus.cc:2749] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:59:56.639043  4180 raft_consensus.cc:493] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:56.652206  3793 raft_consensus.cc:515] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:56.648890  4144 raft_consensus.cc:493] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:59:56.672817  4182 leader_election.cc:290] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 7 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:56.673626  4144 raft_consensus.cc:3060] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:59:56.693181  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 48051.1us
I20260709 12:59:56.669961  4180 raft_consensus.cc:515] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 12:59:56.731241  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 95938.8us
I20260709 12:59:56.718307  4099 leader_election.cc:290] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:56.699100  4184 consensus_queue.cc:237] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:56.742946  4180 leader_election.cc:290] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 12:59:56.757570  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 170144us
I20260709 12:59:56.724828  3906 raft_consensus.cc:515] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:56.750568  4242 raft_consensus.cc:493] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
W20260709 12:59:56.763331  1717 leader_election.cc:336] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 2.145s (SENT)
I20260709 12:59:56.763938  1717 leader_election.cc:304] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:56.764506  3978 raft_consensus.cc:493] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:59:56.765229  1717 leader_election.cc:336] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 2.151s (SENT)
W20260709 12:59:56.766242  1718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:59:56.760056  4171 raft_consensus.cc:493] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:56.769711  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:56.768121  4206 raft_consensus.cc:2749] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:59:56.772070  2336 raft_consensus.cc:2468] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 4.
I20260709 12:59:56.773157  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:56.776161  2333 raft_consensus.cc:2468] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 3.
W20260709 12:59:56.776746  1569 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:59:56.777280  1569 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:59:56.780861  4206 raft_consensus.cc:493] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:56.781137  1718 leader_election.cc:304] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
W20260709 12:59:56.781651  4166 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 5 current term: 5 is higher.
I20260709 12:59:56.782140  1718 leader_election.cc:304] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:59:56.781418  4206 raft_consensus.cc:515] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:56.784695  1718 leader_election.cc:304] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:59:56.786777  4200 raft_consensus.cc:2804] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:59:56.787762  1718 leader_election.cc:400] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 7 pre-election: Vote denied by peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625) with higher term. Message: Invalid argument: T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 7. Current term is 8.
I20260709 12:59:56.789323  1718 leader_election.cc:403] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 7 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:59:56.785874  3999 raft_consensus.cc:493] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:56.780176  3793 leader_election.cc:290] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 5 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:56.789939  4200 raft_consensus.cc:493] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 12:59:56.791177  4244 raft_consensus.cc:3060] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Advancing to term 8
I20260709 12:59:56.788128  3977 raft_consensus.cc:2804] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:59:56.783993  3948 raft_consensus.cc:2804] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:59:56.767297  3978 raft_consensus.cc:515] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:56.790783  1718 leader_election.cc:304] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:59:56.766601  4242 raft_consensus.cc:515] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:56.784188  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:56.777272  4171 raft_consensus.cc:515] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:56.792029  4200 raft_consensus.cc:3060] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:59:56.793875  3948 raft_consensus.cc:493] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 12:59:56.814738  1679 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57092 (request call id 510) took 2655 ms. Trace:
I20260709 12:59:56.792829  3999 raft_consensus.cc:515] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } }
I20260709 12:59:56.815289  1679 rpcz_store.cc:276] 0709 12:59:54.159258 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:54.160203 (+   945us) service_pool.cc:224] Handling call
0709 12:59:54.190495 (+ 30292us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:59:55.298595 (+1108100us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:56.812069 (+1513474us) spinlock_profiling.cc:243] Waited 1.1 s on lock 0x7b54000d7d54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 12:59:56.814636 (+  2567us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2528933632}
I20260709 12:59:56.839057  4262 raft_consensus.cc:2804] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:59:56.840250  3977 raft_consensus.cc:697] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 5 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 12:59:56.801736  2331 raft_consensus.cc:2468] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 5.
I20260709 12:59:56.840711  4264 raft_consensus.cc:493] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
W20260709 12:59:56.844142  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 53447.1us
I20260709 12:59:56.845294  1718 leader_election.cc:304] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:59:56.849797  4099 raft_consensus.cc:493] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 12:59:56.850373  4099 raft_consensus.cc:515] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:56.851686  3999 leader_election.cc:290] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:56.852660  4099 leader_election.cc:290] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:56.852726  3999 raft_consensus.cc:2804] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:59:56.853417  3999 raft_consensus.cc:493] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 12:59:56.854003  3999 raft_consensus.cc:3060] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:59:56.810225  3948 raft_consensus.cc:3060] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:59:56.835477  4100 raft_consensus.cc:493] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:56.864643  3906 leader_election.cc:290] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 6 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:56.862875  4182 raft_consensus.cc:493] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 12:59:56.874842  4100 raft_consensus.cc:515] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:56.881639  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:56.883616  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:59:56.879597  2299 leader_election.cc:304] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 12:59:56.889150  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:56.885951  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:56.889879  1671 raft_consensus.cc:2468] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 3.
I20260709 12:59:56.896914  1678 raft_consensus.cc:2393] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 4: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 12:59:56.899480  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:56.900796  4028 leader_election.cc:290] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 5 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:56.902639  4175 raft_consensus.cc:2749] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:59:56.903623  1748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43156 (request call id 438) took 1706 ms. Trace:
I20260709 12:59:56.904162  1748 rpcz_store.cc:276] 0709 12:59:55.197555 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:55.269213 (+ 71658us) service_pool.cc:224] Handling call
0709 12:59:55.295561 (+ 26348us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:59:55.340757 (+ 45196us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:56.903204 (+1562447us) spinlock_profiling.cc:243] Waited 32.9 ms on lock 0x7b54000e4554. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 12:59:56.903529 (+   325us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":75597056}
I20260709 12:59:56.910784  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:56.910573  4182 raft_consensus.cc:515] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:56.911449  1680 raft_consensus.cc:2468] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 4.
I20260709 12:59:56.912258  4199 raft_consensus.cc:493] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 12:59:56.913072  4182 leader_election.cc:290] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:56.912878  4199 raft_consensus.cc:515] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:56.913625  3993 leader_election.cc:290] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:56.916193  4199 leader_election.cc:290] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:56.917680  4242 leader_election.cc:290] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:56.918834  1602 raft_consensus.cc:3060] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:59:56.131371   431 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:59:56.922772  4244 raft_consensus.cc:2749] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Leader pre-election lost for term 7. Reason: Vote denied by peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625) with higher term. Message: Invalid argument: T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 7. Current term is 8.
I20260709 12:59:56.858273  4264 raft_consensus.cc:515] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:56.925146  4264 leader_election.cc:290] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:56.927929  4200 raft_consensus.cc:515] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:56.929028  4070 consensus_queue.cc:1059] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:59:56.930457  4070 raft_consensus.cc:3055] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:59:56.930981  4070 raft_consensus.cc:740] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 12:59:56.932137  4070 consensus_queue.cc:260] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:56.933836  4070 raft_consensus.cc:3060] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:59:56.930251  4200 leader_election.cc:290] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 12:59:56.937595  4260 log.cc:927] Time spent T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.163s	user 0.001s	sys 0.000s
I20260709 12:59:56.939124  4100 leader_election.cc:290] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 12:59:56.939801  4256 log.cc:927] Time spent T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.174s	user 0.000s	sys 0.002s
I20260709 12:59:56.893273  4144 raft_consensus.cc:515] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:56.941962  4150 leader_election.cc:290] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:56.943094  3793 raft_consensus.cc:2804] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:59:56.954711  3793 raft_consensus.cc:493] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:59:56.955219  3793 raft_consensus.cc:3060] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:59:56.956149  4144 leader_election.cc:290] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 4 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:56.958509  4199 raft_consensus.cc:493] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:56.948637  3978 leader_election.cc:290] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:56.950167  4206 leader_election.cc:290] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:56.950845  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:59:56.959101  4199 raft_consensus.cc:515] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:56.960493  1676 raft_consensus.cc:3060] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:59:56.961671  4199 leader_election.cc:290] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:56.951174  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 12:59:56.962626  2336 raft_consensus.cc:2368] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 3. Current term is 4.
I20260709 12:59:56.863837  4099 raft_consensus.cc:493] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 12:59:56.963943  4099 raft_consensus.cc:515] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:56.951763  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:56.966987  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:56.951797  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:56.953495  4171 leader_election.cc:290] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:56.967911  1748 raft_consensus.cc:2468] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 3.
I20260709 12:59:56.968248  2328 raft_consensus.cc:2468] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 5.
I20260709 12:59:56.966198  4099 leader_election.cc:290] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:56.969576  1646 leader_election.cc:304] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:59:56.970986  4126 raft_consensus.cc:2804] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:59:56.971508  4126 raft_consensus.cc:493] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:59:56.971938  4126 raft_consensus.cc:3060] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:59:56.953637  3993 raft_consensus.cc:493] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 12:59:56.973786  3993 raft_consensus.cc:515] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:56.974977  3977 consensus_queue.cc:237] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:56.982601  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:56.982955  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 12:59:56.987069  1718 leader_election.cc:336] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Timed out: RequestConsensusVote RPC to 127.0.107.194:46261 timed out after 2.155s (SENT)
I20260709 12:59:56.987725  1718 leader_election.cc:304] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 12:59:56.986076  2298 leader_election.cc:304] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 12:59:56.993048  2298 leader_election.cc:304] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
W20260709 12:59:56.992995  1717 leader_election.cc:336] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 1.519s (SENT)
I20260709 12:59:56.977614  3993 leader_election.cc:290] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:57.013958  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:59:57.033215  1676 raft_consensus.cc:2468] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 5.
I20260709 12:59:56.993976  4142 raft_consensus.cc:2804] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:59:57.033872  4262 raft_consensus.cc:697] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 4 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 12:59:57.031081  4205 raft_consensus.cc:493] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:59:57.057305  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:59:57.111438  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 124375us
W20260709 12:59:57.330471  1718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:59:57.330992  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:57.331470  1718 leader_election.cc:304] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:59:57.219070  1672 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52778 (request call id 1706) took 2764 ms. Trace:
I20260709 12:59:57.332907  1672 rpcz_store.cc:276] 0709 12:59:54.441150 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:54.455513 (+ 14363us) service_pool.cc:224] Handling call
0709 12:59:57.205535 (+2750022us) spinlock_profiling.cc:243] Waited 581 ms on lock 0x7b54001f4d54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 12:59:57.205722 (+   187us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1336623872}
I20260709 12:59:57.333303  1718 leader_election.cc:304] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:59:57.207285  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:59:57.221431  1679 raft_consensus.cc:1275] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
W20260709 12:59:57.338317  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 350858us
I20260709 12:59:57.340149  1602 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 426) took 1892 ms. Trace:
I20260709 12:59:57.340485  1718 leader_election.cc:400] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 election: Vote denied by peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625) with higher term. Message: Invalid argument: T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 3. Current term is 4.
I20260709 12:59:57.341787  1718 leader_election.cc:403] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20260709 12:59:57.341511  1602 rpcz_store.cc:276] 0709 12:59:55.448078 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:55.451629 (+  3551us) service_pool.cc:224] Handling call
0709 12:59:55.451998 (+   369us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:59:56.918222 (+1466224us) spinlock_profiling.cc:243] Waited 731 ms on lock 0x7b58005aecd0. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42660be0aa 00007f4266140447 00007f4266158fe7 00007f426615a7de 00007f4266164b49 00007f4266165416 00007f4266161d8f 00007f426616183e 00007f426b126419 00007f426469dbd4
0709 12:59:57.339733 (+421511us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:59:57.339788 (+    55us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:59:57.339835 (+    47us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:59:57.339881 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:57.339967 (+    86us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:59:57.340085 (+   118us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":91681,"spinlock_wait_cycles":1681794304}
W20260709 12:59:57.331146  1717 leader_election.cc:336] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 1.810s (SENT)
I20260709 12:59:57.343111  1717 leader_election.cc:304] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 12:59:57.343828  2299 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:59:57.351423  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:57.357847  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:57.359110  1718 leader_election.cc:304] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
W20260709 12:59:57.293560  4138 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.532s	user 0.011s	sys 0.008s
I20260709 12:59:57.361286  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:57.352186  1751 raft_consensus.cc:2468] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 5.
W20260709 12:59:57.363111  1717 leader_election.cc:336] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 1.797s (SENT)
I20260709 12:59:57.031932  4150 raft_consensus.cc:3060] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:59:57.013267  4201 raft_consensus.cc:2804] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:59:57.369318  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:59:57.377494  1679 raft_consensus.cc:3060] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:59:57.035969  4171 raft_consensus.cc:493] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 12:59:57.302141  4200 raft_consensus.cc:493] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:57.407680  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:57.366725  1673 raft_consensus.cc:2468] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 4.
I20260709 12:59:57.369239  3521 raft_consensus.cc:2804] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:59:57.356947  4244 raft_consensus.cc:2804] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:59:57.365303  4175 raft_consensus.cc:3060] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:59:57.356741  4242 raft_consensus.cc:2804] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:59:57.366191  4144 raft_consensus.cc:493] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:57.386816  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:57.389185  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:57.370517  4217 raft_consensus.cc:493] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 12:59:57.375423  4264 raft_consensus.cc:2749] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:59:57.390884  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:57.358004  4205 raft_consensus.cc:515] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:57.391737  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:57.392568  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:57.393829  4184 raft_consensus.cc:493] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:57.067044  4206 raft_consensus.cc:493] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 12:59:57.330588  4142 raft_consensus.cc:493] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 12:59:57.204578  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
W20260709 12:59:57.369884  3948 consensus_meta.cc:287] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.515s	user 0.006s	sys 0.008s
W20260709 12:59:57.361703  4157 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 4 current term: 4 is higher.
I20260709 12:59:57.404330  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:57.405277  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:57.406065  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:57.406840  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:57.407439  2332 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39738 (request call id 617) took 1650 ms. Trace:
W20260709 12:59:57.407986  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 245311us
I20260709 12:59:57.142405  3978 raft_consensus.cc:493] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 12:59:57.414417  3521 raft_consensus.cc:493] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:59:57.413071  2299 leader_election.cc:304] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 12:59:57.415267  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 12:59:57.420979  1599 raft_consensus.cc:2468] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 5.
I20260709 12:59:57.607174  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:57.608088  1599 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40166 (request call id 1816) took 1287 ms. Trace:
I20260709 12:59:57.608521  1599 rpcz_store.cc:276] 0709 12:59:56.320784 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:56.321094 (+   310us) service_pool.cc:224] Handling call
0709 12:59:57.207092 (+885998us) spinlock_profiling.cc:243] Waited 1.77 ms on lock 0x7b5800003698. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f426303db71 00007f426303da22 00007f426bf56c09 00007f426b0b0ee0 00007f426b17a0ab 00007f426b17a131 00007f426b1280a0 00007f426b127984 00007f42646a26d4 00007f42646a2662
0709 12:59:57.607984 (+400892us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4063104}
I20260709 12:59:57.459082  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:57.609858  1747 raft_consensus.cc:3060] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:59:57.377115  4201 raft_consensus.cc:493] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:59:57.621301  1599 raft_consensus.cc:1240] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 2. Current term is 3. Ops: []
I20260709 12:59:57.497483  1676 raft_consensus.cc:2468] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 6.
I20260709 12:59:57.502880  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:57.623652  2334 raft_consensus.cc:2468] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 3.
I20260709 12:59:57.554680  1680 raft_consensus.cc:2393] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 6: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 12:59:57.542008  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:59:57.628688  1677 raft_consensus.cc:3060] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:59:57.531137  1717 leader_election.cc:336] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 1.994s (SENT)
I20260709 12:59:57.414615  4244 raft_consensus.cc:493] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:59:57.563304  1602 raft_consensus.cc:1275] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20260709 12:59:57.562081  1679 raft_consensus.cc:2468] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 7.
I20260709 12:59:57.567020  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:57.411716  4132 raft_consensus.cc:493] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
W20260709 12:59:57.585896  1569 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:59:57.585067  2332 rpcz_store.cc:276] 0709 12:59:55.756811 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:55.763462 (+  6651us) service_pool.cc:224] Handling call
0709 12:59:55.812940 (+ 49478us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:59:55.813363 (+   423us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:59:55.813387 (+    24us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:59:56.868031 (+1054644us) spinlock_profiling.cc:243] Waited 9.32 ms on lock 0x7b4c0012d000. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426442ea20 00007f426443fa29 00007f4264405644 00007f426442cf2d 00007f426442cb93 00007f426615d18b 00007f426615dcb5 00007f426615f004 00007f426615e729 00007f4266163d96 00007f4266158adb
0709 12:59:56.868856 (+   825us) log.cc:844] Serialized 564 byte log entry
0709 12:59:57.343746 (+474890us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:59:57.343783 (+    37us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:57.343855 (+    72us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:59:57.407114 (+ 63259us) raft_consensus.cc:1685] finished
0709 12:59:57.407139 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:59:57.407369 (+   230us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":21439744,"thread_start_us":474546,"threads_started":1,"wal-append.queue_time_us":496163}
W20260709 12:59:57.648240  1569 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:59:57.611678  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 198670us
W20260709 12:59:57.629912  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 98853.5us
I20260709 12:59:57.464527  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:57.653367  1745 raft_consensus.cc:3060] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:59:57.632815  1676 raft_consensus.cc:1275] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 12:59:57.499547  4242 raft_consensus.cc:493] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 12:59:57.639384  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:57.620059  3521 raft_consensus.cc:3060] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:59:57.649418  1569 leader_election.cc:304] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 12:59:57.608882  4166 raft_consensus.cc:493] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 12:59:57.587671  4206 raft_consensus.cc:515] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 12:59:57.623050  3793 consensus_meta.cc:287] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.667s	user 0.007s	sys 0.007s
I20260709 12:59:57.674721  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:57.675540  1602 raft_consensus.cc:2468] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 4.
I20260709 12:59:57.588047  4142 raft_consensus.cc:3060] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:59:57.501560  4217 raft_consensus.cc:515] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:57.665227  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:57.416034  4171 raft_consensus.cc:515] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:57.660517  1569 leader_election.cc:304] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 12:59:57.611532  3948 raft_consensus.cc:515] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:57.584478  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:57.622654  1675 raft_consensus.cc:3060] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:59:57.689097  2332 raft_consensus.cc:2468] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 3.
I20260709 12:59:57.697597  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:57.684221  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:57.690305  2334 raft_consensus.cc:1240] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 3. Current term is 4. Ops: []
I20260709 12:59:57.419386  4200 raft_consensus.cc:515] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:57.639452  4201 raft_consensus.cc:3060] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:59:57.707605  1752 raft_consensus.cc:2468] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 8.
I20260709 12:59:57.729486  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:57.639539  4140 raft_consensus.cc:2804] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:59:57.708889  1596 raft_consensus.cc:1240] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 LEADER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 3. Current term is 4. Ops: []
I20260709 12:59:57.662936  4244 raft_consensus.cc:3060] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:59:57.727946  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:57.753840  1601 raft_consensus.cc:2468] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 5.
I20260709 12:59:57.754846  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:57.665032  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:57.734515  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:59:57.735039  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:59:57.762910  1604 raft_consensus.cc:3060] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:59:57.567381  4184 raft_consensus.cc:515] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:57.739395  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:59:57.776903  1598 raft_consensus.cc:3060] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:59:57.740760  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:57.556211  4144 raft_consensus.cc:515] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:57.657666  4099 raft_consensus.cc:493] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:57.666347  4182 raft_consensus.cc:493] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 12:59:57.732999  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:59:57.757601  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:57.788478  1752 raft_consensus.cc:3060] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:59:57.667088  4242 raft_consensus.cc:3060] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:59:57.677910  4139 raft_consensus.cc:2804] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:59:57.764046  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:57.736986  1597 raft_consensus.cc:1275] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260709 12:59:57.639452  3978 raft_consensus.cc:515] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:57.663877  4132 raft_consensus.cc:515] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:57.697108  3793 raft_consensus.cc:515] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:57.673516  4166 raft_consensus.cc:515] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:57.624163  4262 consensus_queue.cc:237] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:57.803820  1752 raft_consensus.cc:2468] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 5.
I20260709 12:59:57.819161  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:57.809861  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:57.820420  1601 raft_consensus.cc:2393] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 6: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 12:59:57.770785  4140 raft_consensus.cc:697] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 5 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 12:59:57.821748  4140 consensus_queue.cc:237] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:57.813182  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:57.814082  4138 raft_consensus.cc:493] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:59:57.814843  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:59:57.785218  1745 raft_consensus.cc:2468] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 5.
I20260709 12:59:57.835453  4204 raft_consensus.cc:2804] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:59:57.836032  4204 raft_consensus.cc:493] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Starting leader election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 12:59:57.836431  4204 raft_consensus.cc:3060] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:59:57.780778  1749 raft_consensus.cc:2468] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 5.
I20260709 12:59:57.844517  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:57.859442  1745 raft_consensus.cc:2468] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 5.
I20260709 12:59:57.847851  1747 raft_consensus.cc:2468] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 4.
I20260709 12:59:57.801249  4139 raft_consensus.cc:493] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 12:59:57.862578  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:57.863457  1745 raft_consensus.cc:3060] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:59:57.866146  1604 raft_consensus.cc:2468] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 4.
I20260709 12:59:57.873636  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:57.881532  1601 raft_consensus.cc:2468] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 3.
I20260709 12:59:57.807236  4099 raft_consensus.cc:515] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:57.841806  4264 raft_consensus.cc:493] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:57.840468  3793 leader_election.cc:290] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 6 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 12:59:57.888114  4270 log.cc:927] Time spent T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.127s	user 0.000s	sys 0.000s
I20260709 12:59:57.838343  4132 leader_election.cc:290] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:57.868117  4139 raft_consensus.cc:3060] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:59:57.885788  2331 raft_consensus.cc:2410] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for term 7 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260709 12:59:57.942219  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:59:57.945081  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:57.805756  4182 raft_consensus.cc:515] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:57.903991  3948 leader_election.cc:290] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:57.904840  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:57.907441  4205 leader_election.cc:290] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:57.961783  1596 raft_consensus.cc:3060] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:59:57.911444  4264 raft_consensus.cc:515] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:57.921427  4184 leader_election.cc:290] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:57.931068  3978 leader_election.cc:290] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 12:59:57.879776  4070 consensus_meta.cc:287] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.945s	user 0.016s	sys 0.000s
I20260709 12:59:57.862468  4138 raft_consensus.cc:515] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:57.958734  1601 raft_consensus.cc:2368] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 3. Current term is 4.
I20260709 12:59:57.975376  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:57.976495  1604 raft_consensus.cc:2468] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 7.
I20260709 12:59:57.978399  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:57.979151  1597 raft_consensus.cc:2468] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 2.
W20260709 12:59:57.962577  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 311196us
W20260709 12:59:57.962684  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 285433us
W20260709 12:59:57.964052  1643 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.628s	user 0.006s	sys 0.055s
W20260709 12:59:57.981335  1643 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.645s	user 0.006s	sys 0.055s
W20260709 12:59:57.981667  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 651293us
W20260709 12:59:57.982861  1643 connection.cc:256] RPC call timeout handler was delayed by -1.6087558769999646s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18460000000000001s for a response
W20260709 12:59:57.983174  1643 connection.cc:256] RPC call timeout handler was delayed by -1.5756615959999749s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1559s for a response
W20260709 12:59:57.983446  1643 connection.cc:256] RPC call timeout handler was delayed by -1.488155876999997s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.198s for a response
W20260709 12:59:57.983755  1643 connection.cc:256] RPC call timeout handler was delayed by -1.4629558769999562s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2008s for a response
I20260709 12:59:57.967373  1745 raft_consensus.cc:2468] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 6.
W20260709 12:59:57.977540  4175 consensus_meta.cc:287] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.518s	user 0.008s	sys 0.004s
W20260709 12:59:57.993288  1643 connection.cc:256] RPC call timeout handler was delayed by -1.3612558769999623s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2121s for a response
W20260709 12:59:58.010739  1717 leader_election.cc:336] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 1.762s (SENT)
I20260709 12:59:58.003623  1676 raft_consensus.cc:1240] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260709 12:59:58.012480  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:59:58.004477  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:59:58.013396  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:59:58.014101  1678 raft_consensus.cc:3060] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:59:58.000829  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
W20260709 12:59:57.996431  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 335964us
I20260709 12:59:58.020306  1597 raft_consensus.cc:2368] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 4. Current term is 5.
I20260709 12:59:58.021773  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:59:58.006882  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 6 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:58.031418  2330 raft_consensus.cc:2410] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for term 6 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260709 12:59:58.031877  2330 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39738 (request call id 624) took 1258 ms. Trace:
I20260709 12:59:58.032248  2330 rpcz_store.cc:276] 0709 12:59:56.773080 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:56.775690 (+  2610us) service_pool.cc:224] Handling call
0709 12:59:58.006701 (+1231011us) spinlock_profiling.cc:243] Waited 1.49 ms on lock 0x7b5800178598. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f426303db71 00007f426303da22 00007f426bf56c09 00007f426b0b0ee0 00007f426b17a0ab 00007f426b17a131 00007f426b1280a0 00007f426b127984 00007f42646a26d4 00007f42646a2662
0709 12:59:58.031803 (+ 25102us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3417984}
I20260709 12:59:58.032987  4272 consensus_queue.cc:1048] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:59:58.014398  4175 raft_consensus.cc:2749] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625) with higher term. Message: Invalid argument: T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 3. Current term is 4.
I20260709 12:59:58.034338  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:58.034765  1754 raft_consensus.cc:2468] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 4.
I20260709 12:59:58.035071  1597 raft_consensus.cc:2468] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 5.
I20260709 12:59:58.035313  1569 leader_election.cc:304] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 12:59:58.036077  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:58.036729  4215 raft_consensus.cc:2804] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:59:58.037144  4215 raft_consensus.cc:493] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:59:58.037205  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:58.037573  4215 raft_consensus.cc:3060] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:59:58.037842  1597 raft_consensus.cc:2468] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 5.
I20260709 12:59:58.038139  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:58.038811  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:58.039324  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:58.039917  1603 raft_consensus.cc:2468] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 4.
I20260709 12:59:58.040007  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:58.040758  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:57.979970  3978 raft_consensus.cc:493] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 12:59:58.050743  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:58.057230  1680 raft_consensus.cc:2393] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 5: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 12:59:58.042083  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:58.044391  1569 leader_election.cc:304] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 12:59:58.046042  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:59:58.058470  1601 raft_consensus.cc:2468] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 3.
I20260709 12:59:58.059336  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:58.046134  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:58.060688  1678 raft_consensus.cc:2468] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 4.
I20260709 12:59:58.060617  1569 leader_election.cc:304] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 12:59:58.046944  1598 raft_consensus.cc:2468] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 6.
W20260709 12:59:58.048115  4126 consensus_meta.cc:287] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 1.076s	user 0.008s	sys 0.005s
I20260709 12:59:58.048524  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:58.049010  1596 raft_consensus.cc:2468] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 8.
I20260709 12:59:58.027621  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:58.034282  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:59:58.052541  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:59:58.041093  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:59:58.065526  1672 raft_consensus.cc:3060] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:59:58.053138  1643 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 1.559s (SENT) [suppressed 32 similar messages]
I20260709 12:59:58.000394  4184 raft_consensus.cc:493] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:58.066247  1604 raft_consensus.cc:2468] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 3.
I20260709 12:59:58.063743  2301 leader_election.cc:304] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:59:58.068697  4242 raft_consensus.cc:515] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:58.081209  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:58.068843  4199 consensus_queue.cc:1048] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:59:58.071525  1603 raft_consensus.cc:2393] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 5: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
W20260709 12:59:58.072144  1643 leader_election.cc:336] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 1.846s (SENT)
I20260709 12:59:58.099061  1643 leader_election.cc:304] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 12:59:58.079725  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:58.001259  3977 raft_consensus.cc:493] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:58.100595  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:58.009377  4206 leader_election.cc:290] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:58.101881  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:58.082356  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:58.096788  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:58.102907  1604 raft_consensus.cc:2393] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 5: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 12:59:58.103247  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:58.099180  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:59:58.104121  1605 raft_consensus.cc:3060] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:59:58.095857  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:59:58.096674  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 6 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 12:59:58.067642  4269 log.cc:927] Time spent T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.261s	user 0.004s	sys 0.000s
I20260709 12:59:58.107197  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:58.107867  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:59:58.108122  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:58.113235  1599 raft_consensus.cc:2468] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 4.
I20260709 12:59:58.108865  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:58.113688  1745 raft_consensus.cc:2368] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 4. Current term is 5.
I20260709 12:59:58.120641  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:59:58.060263  4171 leader_election.cc:290] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:58.121263  1603 raft_consensus.cc:3060] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:59:58.064247  4218 raft_consensus.cc:2804] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:59:58.073987  4144 leader_election.cc:290] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:58.124125  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:58.128542  1748 raft_consensus.cc:2468] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 5.
I20260709 12:59:58.128485  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:58.122225  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:58.130887  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:58.136276  1598 raft_consensus.cc:2468] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 4.
I20260709 12:59:58.131685  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:58.083113  1746 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55848 (request call id 1861) took 2032 ms. Trace:
I20260709 12:59:58.150278  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 12:59:58.151645  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 79528us
I20260709 12:59:58.163857  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:58.141567  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:58.165907  1601 raft_consensus.cc:2393] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 5: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 12:59:58.148609  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:58.166579  1604 raft_consensus.cc:2468] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 5.
I20260709 12:59:58.166725  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:59:58.152343  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:59:58.170168  1603 raft_consensus.cc:2468] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 6.
I20260709 12:59:58.082258  4126 raft_consensus.cc:515] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:58.101358  4217 leader_election.cc:290] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:58.137833  1678 raft_consensus.cc:2393] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 5: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 12:59:58.066994  3978 raft_consensus.cc:515] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:58.100907  4272 raft_consensus.cc:2804] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:59:58.231474  1672 raft_consensus.cc:2468] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 6.
W20260709 12:59:58.334585  2298 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.553s	user 0.029s	sys 0.138s
W20260709 12:59:58.335525  2298 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.554s	user 0.030s	sys 0.138s
I20260709 12:59:58.336835  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:58.337805  1598 raft_consensus.cc:2410] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 12:59:58.139344  2333 raft_consensus.cc:2468] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 6.
W20260709 12:59:58.339552  1718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:59:58.162822  1643 leader_election.cc:336] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 1.980s (SENT)
I20260709 12:59:58.121091  4274 raft_consensus.cc:2749] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:59:58.182010  1605 raft_consensus.cc:2468] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 4.
I20260709 12:59:58.082846  4204 raft_consensus.cc:515] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } }
I20260709 12:59:58.186818  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:58.188400  4126 leader_election.cc:290] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 5 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:58.188896  4171 consensus_queue.cc:1048] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:59:58.352357  4171 consensus_queue.cc:1243] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer 83595ad0fd5940c0a2bfc7c61acd969a log is divergent from this leader: its last log entry 2.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260709 12:59:58.097719  4184 raft_consensus.cc:515] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:58.191316  4138 leader_election.cc:290] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:58.191419  3521 raft_consensus.cc:515] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:58.356976  4184 leader_election.cc:290] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:58.357259  3521 leader_election.cc:290] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 6 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:58.357517  1676 raft_consensus.cc:3060] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:59:58.122807  4070 consensus_queue.cc:1048] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.056s
I20260709 12:59:58.358875  1750 raft_consensus.cc:2393] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 6: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 12:59:58.112067  3977 raft_consensus.cc:515] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:58.361259  3977 leader_election.cc:290] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:58.192865  4166 leader_election.cc:290] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:58.362182  4166 raft_consensus.cc:493] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:58.362705  4166 raft_consensus.cc:515] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:58.365764  1747 raft_consensus.cc:2468] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 6.
I20260709 12:59:58.128196  4218 raft_consensus.cc:493] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 12:59:58.366688  4218 raft_consensus.cc:3060] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:59:58.367645  1676 raft_consensus.cc:2468] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 5.
I20260709 12:59:58.193461  4099 leader_election.cc:290] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:58.193655  4201 raft_consensus.cc:515] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:58.196264  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:58.197834  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
W20260709 12:59:58.198599  3999 consensus_meta.cc:287] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 1.324s	user 0.015s	sys 0.001s
I20260709 12:59:58.192862  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:58.199476  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:58.200464  1599 raft_consensus.cc:1240] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 5. Current term is 6. Ops: []
I20260709 12:59:58.200717  4264 leader_election.cc:290] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 12:59:58.204795  1677 consensus_meta.cc:287] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.565s	user 0.010s	sys 0.001s
I20260709 12:59:58.385048  1677 raft_consensus.cc:2468] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 4.
I20260709 12:59:58.201903  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:58.232474  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:58.387004  1752 raft_consensus.cc:2468] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 4.
I20260709 12:59:58.340197  1643 leader_election.cc:304] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 12:59:58.335551  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:58.171785  1603 raft_consensus.cc:1240] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 LEADER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260709 12:59:58.388329  1748 raft_consensus.cc:2393] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 5: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
W20260709 12:59:58.244233  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 353812us
W20260709 12:59:58.336374  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 554725us
W20260709 12:59:58.328502  1779 heartbeater.cc:646] Failed to heartbeat to 127.0.107.254:33643 (0 consecutive failures): Timed out: Failed to send heartbeat to master: TSHeartbeat RPC to 127.0.107.254:33643 timed out after 15.000s (SENT)
W20260709 12:59:58.311431  4244 consensus_meta.cc:287] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.547s	user 0.013s	sys 0.000s
I20260709 12:59:58.394172  4244 raft_consensus.cc:515] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:58.396525  4244 leader_election.cc:290] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:58.397624  1753 raft_consensus.cc:2368] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 for earlier term 4. Current term is 5.
I20260709 12:59:58.375015  4139 raft_consensus.cc:515] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:58.400501  1602 raft_consensus.cc:1240] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 3. Current term is 4. Ops: []
W20260709 12:59:58.323539  4150 consensus_meta.cc:287] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.914s	user 0.012s	sys 0.001s
I20260709 12:59:58.402566  4139 leader_election.cc:290] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 4 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:58.402709  4150 raft_consensus.cc:2749] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:59:58.403672  1751 raft_consensus.cc:2468] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 4.
I20260709 12:59:58.406802  4242 leader_election.cc:290] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 12:59:58.407984  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 57542.1us
W20260709 12:59:58.340389  1671 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.592s	user 0.008s	sys 0.045s
I20260709 12:59:58.389631  2298 leader_election.cc:304] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 12:59:58.386402  2329 raft_consensus.cc:2468] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 4.
W20260709 12:59:58.391484  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 228644us
I20260709 12:59:58.389185  4171 raft_consensus.cc:493] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 12:59:58.392186  4184 raft_consensus.cc:2749] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:59:58.392943  4201 leader_election.cc:290] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 6 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:58.308815  3978 leader_election.cc:290] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:58.348452  4281 raft_consensus.cc:493] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
W20260709 12:59:58.413199  1671 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.665s	user 0.009s	sys 0.045s
W20260709 12:59:58.415102  1643 leader_election.cc:336] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 2.008s (SENT)
I20260709 12:59:58.312513  4132 raft_consensus.cc:2804] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:59:58.423028  2298 leader_election.cc:304] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 12:59:58.235252  1675 raft_consensus.cc:2468] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 6.
W20260709 12:59:58.431550  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 367716us
I20260709 12:59:58.430070  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
W20260709 12:59:58.432154  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:59:58.443554  1717 leader_election.cc:304] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:58.444142  2298 leader_election.cc:304] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 12:59:58.447221  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 448215us
W20260709 12:59:58.444702  1571 connection.cc:256] RPC call timeout handler was delayed by -2.7225603210000031s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15510000000000002s for a response
W20260709 12:59:58.448143  1571 connection.cc:256] RPC call timeout handler was delayed by -2.4561603209999703s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1847s for a response
W20260709 12:59:58.448560  1571 connection.cc:256] RPC call timeout handler was delayed by -2.394960320999985s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1915s for a response
W20260709 12:59:58.448861  1571 connection.cc:256] RPC call timeout handler was delayed by -2.3886603209999748s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1922s for a response
W20260709 12:59:58.449146  1571 connection.cc:256] RPC call timeout handler was delayed by -2.2671603210000058s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2057s for a response
W20260709 12:59:58.449434  1571 connection.cc:256] RPC call timeout handler was delayed by -2.228460321s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21000000000000002s for a response
I20260709 12:59:58.446419  1679 raft_consensus.cc:1240] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
W20260709 12:59:58.449774  1571 connection.cc:256] RPC call timeout handler was delayed by -2.1627603210000075s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21730000000000002s for a response
W20260709 12:59:58.450167  1571 connection.cc:256] RPC call timeout handler was delayed by -2.14566032099998s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21920000000000003s for a response
I20260709 12:59:58.446749  2301 leader_election.cc:304] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
W20260709 12:59:58.450604  1571 connection.cc:256] RPC call timeout handler was delayed by -2.06916032099997s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2277s for a response
I20260709 12:59:58.421890  4070 raft_consensus.cc:2804] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:59:58.454051  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:58.453243  1571 leader_election.cc:304] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 12:59:58.457340  2333 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39726 (request call id 737) took 1043 ms. Trace:
I20260709 12:59:58.457737  2333 rpcz_store.cc:276] 0709 12:59:57.413530 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:57.415067 (+  1537us) service_pool.cc:224] Handling call
0709 12:59:58.457143 (+1042076us) spinlock_profiling.cc:243] Waited 372 ms on lock 0x7b5400605ed4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 12:59:58.457273 (+   130us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":854458624}
I20260709 12:59:58.459213  2299 leader_election.cc:304] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 12:59:58.434930  4272 raft_consensus.cc:697] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 12:59:58.377200  4215 raft_consensus.cc:515] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:58.408156  3999 raft_consensus.cc:515] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:58.437358  4171 raft_consensus.cc:515] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:58.462812  1718 leader_election.cc:304] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:59:58.466748  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:58.453096  3978 raft_consensus.cc:2749] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:59:58.414508  4280 consensus_queue.cc:1059] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.043s
I20260709 12:59:58.444409  4139 raft_consensus.cc:493] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:58.453384  4281 raft_consensus.cc:515] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:58.476868  4244 raft_consensus.cc:2804] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:59:58.435834  4099 raft_consensus.cc:493] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 12:59:58.486311  1718 leader_election.cc:304] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 12:59:58.435603  4264 consensus_queue.cc:1059] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.035s
I20260709 12:59:58.435539  4242 consensus_queue.cc:1048] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.035s
I20260709 12:59:58.457497  4140 raft_consensus.cc:2804] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:59:58.490804  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:58.508256  1754 raft_consensus.cc:2468] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 4.
W20260709 12:59:58.454022  1675 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52766 (request call id 2109) took 1324 ms (client timeout 1759 ms). Trace:
I20260709 12:59:58.464804  4201 raft_consensus.cc:2749] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20260709 12:59:58.473096  4142 consensus_meta.cc:287] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.744s	user 0.011s	sys 0.001s
I20260709 12:59:58.456795  4070 raft_consensus.cc:493] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 12:59:58.514856  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:58.515933  1677 raft_consensus.cc:2393] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 5: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 12:59:58.495114  1718 leader_election.cc:304] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 12:59:58.496192  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 12:59:58.496922  1569 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:59:58.472002  4157 raft_consensus.cc:493] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:58.500411  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:58.524981  1672 raft_consensus.cc:2468] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 5.
I20260709 12:59:58.501729  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 12:59:58.504860  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 50384.7us
I20260709 12:59:58.466149  4132 raft_consensus.cc:697] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 8 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 12:59:58.499544  4280 raft_consensus.cc:2749] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:59:58.512651  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:58.510490  3906 raft_consensus.cc:2804] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:59:58.518154  1718 leader_election.cc:304] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:59:58.499464  4184 raft_consensus.cc:493] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
W20260709 12:59:58.510977  1675 rpcz_store.cc:269] 0709 12:59:57.128255 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:57.132710 (+  4455us) service_pool.cc:224] Handling call
0709 12:59:58.452718 (+1320008us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:59:58.507733  4140 raft_consensus.cc:493] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:59:58.521837  4301 raft_consensus.cc:2749] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:59:58.512744  4244 raft_consensus.cc:697] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [term 4 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 12:59:58.523764  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:58.523027  4215 leader_election.cc:290] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 5 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:58.522889  4283 raft_consensus.cc:2749] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:59:58.528853  3977 raft_consensus.cc:3055] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:59:58.540804  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:58.542063  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
W20260709 12:59:58.545038  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 98346us
I20260709 12:59:58.491689  4139 raft_consensus.cc:515] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:58.522539  4070 raft_consensus.cc:3060] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:59:58.552834  1674 raft_consensus.cc:2393] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 7: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 12:59:58.552700  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
W20260709 12:59:58.561589  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 108412us
I20260709 12:59:58.553674  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:58.562487  1571 leader_election.cc:304] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 12:59:58.553797  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:58.531611  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:58.564357  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:58.565335  1748 raft_consensus.cc:2393] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 8: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 12:59:58.512074  4302 raft_consensus.cc:493] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 12:59:58.555436  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 12:59:58.566265  2333 raft_consensus.cc:2368] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 5. Current term is 6.
I20260709 12:59:58.507616  4246 consensus_queue.cc:1048] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.040s
I20260709 12:59:58.557152  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:58.559803  1672 raft_consensus.cc:3060] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:59:58.504668  4099 raft_consensus.cc:515] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:58.546375  3993 raft_consensus.cc:2804] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:59:58.510803  4274 consensus_queue.cc:1048] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.041s
I20260709 12:59:58.589167  3993 raft_consensus.cc:697] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 4 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 12:59:58.590255  3993 consensus_queue.cc:237] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:58.529340  3854 consensus_queue.cc:1048] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.038s
I20260709 12:59:58.593753  1672 raft_consensus.cc:2468] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 6.
I20260709 12:59:58.567946  4297 raft_consensus.cc:493] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 12:59:58.587958  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:58.570849  1748 raft_consensus.cc:1275] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 12:59:58.560000  2301 leader_election.cc:304] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:59:58.523526  4150 consensus_queue.cc:1059] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.039s
I20260709 12:59:58.572327  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:58.572893  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:58.553243  4291 consensus_queue.cc:1059] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.025s
I20260709 12:59:58.575264  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:58.575943  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:58.577226  1718 leader_election.cc:304] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:59:58.525012  4142 raft_consensus.cc:515] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:58.577777  4281 leader_election.cc:290] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:58.578159  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:58.578267  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 12:59:58.578995  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 12:59:58.579142  3948 raft_consensus.cc:3055] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:59:58.573500  4301 raft_consensus.cc:2749] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:59:58.590193  1749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43156 (request call id 451) took 1186 ms. Trace:
I20260709 12:59:58.579258  4171 leader_election.cc:290] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:58.150607  1746 rpcz_store.cc:276] 0709 12:59:56.043979 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:56.044144 (+   165us) service_pool.cc:224] Handling call
0709 12:59:56.044533 (+   389us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:59:56.981492 (+936959us) spinlock_profiling.cc:243] Waited 3.15 ms on lock 0x7b1800460060. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f4265d0a4bc 00007f426305848b 00007f426beb2ab9 00007f426307469b 00007f426307293f 00007f4263072455 00007f426615a060 00007f426615abfb 00007f426616564b
0709 12:59:56.981670 (+   178us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:59:56.981701 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:59:56.982332 (+   631us) log.cc:844] Serialized 26 byte log entry
0709 12:59:57.376355 (+394023us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:59:57.390715 (+ 14360us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:57.390869 (+   154us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:59:58.008668 (+617799us) raft_consensus.cc:1685] finished
0709 12:59:58.008718 (+    50us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:59:58.066869 (+ 58151us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":29280,"raft.queue_time_us":99531,"raft.run_cpu_time_us":199,"raft.run_wall_time_us":8271,"spinlock_wait_cycles":7253888,"thread_start_us":447567,"threads_started":2,"wal-append.queue_time_us":585155}
I20260709 12:59:58.581272  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:58.581919  1751 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43156 (request call id 450) took 1178 ms. Trace:
I20260709 12:59:58.660902  1751 rpcz_store.cc:276] 0709 12:59:57.403278 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:57.407424 (+  4146us) service_pool.cc:224] Handling call
0709 12:59:58.581717 (+1174293us) spinlock_profiling.cc:243] Waited 992 ms on lock 0x7b54000195d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 12:59:58.581843 (+   126us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2280874368}
I20260709 12:59:58.662235  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:58.549623  3906 raft_consensus.cc:493] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 12:59:58.664022  3906 raft_consensus.cc:3060] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:59:58.665117  4099 leader_election.cc:290] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:58.555289  4184 raft_consensus.cc:515] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:58.555044  4140 raft_consensus.cc:3060] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:59:58.561076  4201 raft_consensus.cc:493] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 12:59:58.608718  4304 raft_consensus.cc:2804] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:59:58.687304  4304 raft_consensus.cc:697] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 4 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 12:59:58.688436  4304 consensus_queue.cc:237] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:58.587344  4289 raft_consensus.cc:2804] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader election won for term 6
W20260709 12:59:58.587945  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 140064us
I20260709 12:59:58.663354  1749 rpcz_store.cc:276] 0709 12:59:57.403771 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:57.403940 (+   169us) service_pool.cc:224] Handling call
0709 12:59:58.590015 (+1186075us) spinlock_profiling.cc:243] Waited 178 ms on lock 0x7b54001d56d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 12:59:58.590125 (+   110us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":408473984}
I20260709 12:59:58.691392  4133 consensus_queue.cc:1048] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:59:58.692095  1749 raft_consensus.cc:1240] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 3. Current term is 4. Ops: []
I20260709 12:59:58.588332  3999 leader_election.cc:290] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:58.588464  1745 raft_consensus.cc:1275] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 12:59:58.599117  2301 leader_election.cc:304] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
W20260709 12:59:58.695933  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 96889us
I20260709 12:59:58.697801  2301 leader_election.cc:304] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 12:59:58.599905  1671 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52766 (request call id 2106) took 1609 ms. Trace:
I20260709 12:59:58.604764  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:58.606186  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:58.536640  4157 raft_consensus.cc:515] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:58.577483  2328 raft_consensus.cc:1275] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260709 12:59:58.567374  3977 raft_consensus.cc:740] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 12:59:58.577510  4242 raft_consensus.cc:493] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:58.612286  4150 consensus_queue.cc:1243] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer 83595ad0fd5940c0a2bfc7c61acd969a log is divergent from this leader: its last log entry 4.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260709 12:59:58.652089  1753 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43156 (request call id 452) took 1248 ms. Trace:
W20260709 12:59:58.655164  1502 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35464 (request call id 284) took 15367 ms (client timeout 15000 ms). Trace:
I20260709 12:59:58.631587  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:58.656036  1500 catalog_manager.cc:5697] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 2 to 4, leader changed from af520b67945d443b901bbeca37cc5701 (127.0.107.195) to 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193). New cstate: current_term: 4 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: HEALTHY } } }
I20260709 12:59:58.597100  4138 raft_consensus.cc:493] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 12:59:58.659475  1718 leader_election.cc:304] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:59:58.653647  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:58.522605  4272 consensus_queue.cc:237] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:58.667138  4142 leader_election.cc:290] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 4 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:58.556631  4218 raft_consensus.cc:515] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:58.667472  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:58.667981  1746 raft_consensus.cc:1240] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 1. Current term is 2. Ops: []
I20260709 12:59:58.668344  1673 raft_consensus.cc:1240] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 4. Current term is 5. Ops: [4.4-4.4]
I20260709 12:59:58.668443  2337 raft_consensus.cc:2468] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 4.
I20260709 12:59:58.669060  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 12:59:58.651715  4139 leader_election.cc:290] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:58.671633  4182 leader_election.cc:290] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:58.647976  4132 consensus_queue.cc:237] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:58.603472  4126 raft_consensus.cc:493] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 12:59:58.635735  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:58.740254  4289 raft_consensus.cc:697] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 6 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
W20260709 12:59:58.675678  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 111530us
I20260709 12:59:58.783257  1571 leader_election.cc:304] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 12:59:58.785323  1571 leader_election.cc:400] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 3 pre-election: Vote denied by peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693) with higher term. Message: Invalid argument: T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 3. Current term is 4.
I20260709 12:59:58.785527  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:58.786150  1746 raft_consensus.cc:3060] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:59:58.679769  4265 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.885s	user 0.003s	sys 0.004s
I20260709 12:59:58.677523  2299 leader_election.cc:304] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 12:59:58.786620  4126 raft_consensus.cc:515] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 12:59:58.786654  4265 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.003s	user 0.003s	sys 0.004s
I20260709 12:59:58.596002  4297 raft_consensus.cc:515] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:58.785324  2330 raft_consensus.cc:2368] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 5. Current term is 6.
I20260709 12:59:58.788811  4182 raft_consensus.cc:2749] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:59:58.788986  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:58.789804  4144 raft_consensus.cc:2804] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:59:58.686088  4244 consensus_queue.cc:237] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:58.790413  4144 raft_consensus.cc:697] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 6 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 12:59:58.791430  4144 consensus_queue.cc:237] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:58.626998  4283 raft_consensus.cc:2804] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:59:58.700059  1748 raft_consensus.cc:3060] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:59:58.698585  1671 rpcz_store.cc:276] 0709 12:59:56.990367 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:56.990962 (+   595us) service_pool.cc:224] Handling call
0709 12:59:57.337805 (+346843us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:59:57.739768 (+401963us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:59:57.740047 (+   279us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:59:57.745763 (+  5716us) log.cc:844] Serialized 26 byte log entry
0709 12:59:58.444154 (+698391us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:59:58.444291 (+   137us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:58.444468 (+   177us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:59:58.599698 (+155230us) raft_consensus.cc:1685] finished
0709 12:59:58.599719 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:59:58.599839 (+   120us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":33371,"raft.queue_time_us":55434,"raft.run_cpu_time_us":86,"raft.run_wall_time_us":85,"thread_start_us":1062462,"threads_started":2,"wal-append.queue_time_us":776459}
I20260709 12:59:58.631042  3948 raft_consensus.cc:740] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 12:59:58.713075  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 12:59:58.722277  1754 raft_consensus.cc:1240] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 4. Current term is 5. Ops: []
I20260709 12:59:58.774731  4206 raft_consensus.cc:493] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:58.738014  1569 leader_election.cc:304] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
W20260709 12:59:58.799628  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 61690.8us
I20260709 12:59:58.756417  2327 tablet_service.cc:1558] Processing DeleteTablet for tablet 983621a6fc5b4c9ea0159a648dfe7a36 with delete_type TABLET_DATA_TOMBSTONED (TS 3b879799899a499c8d409ee9f1277cd1 not found in new config with opid_index -1) from {username='slave'} at 127.0.0.1:54234
I20260709 12:59:58.737763  1753 rpcz_store.cc:276] 0709 12:59:57.404157 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:57.404960 (+   803us) service_pool.cc:224] Handling call
0709 12:59:58.651859 (+1246899us) spinlock_profiling.cc:243] Waited 810 ms on lock 0x7b54001c8254. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 12:59:58.652024 (+   165us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1862553472}
I20260709 12:59:58.699635  1749 raft_consensus.cc:1240] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 4. Current term is 5. Ops: []
I20260709 12:59:58.762145  4138 raft_consensus.cc:515] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:58.764698  4070 raft_consensus.cc:515] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 12:59:58.738564  1502 rpcz_store.cc:269] 0709 12:59:43.287394 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:43.300729 (+ 13335us) service_pool.cc:224] Handling call
0709 12:59:58.655079 (+15354350us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:59:53.706291 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:59:53.718092 (+ 11801us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:59:53.718153 (+    61us) write_op.cc:435] Acquired schema lock
0709 12:59:53.718175 (+    22us) tablet.cc:662] Decoding operations
0709 12:59:53.751144 (+ 32969us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:59:53.751235 (+    91us) write_op.cc:641] Partition lock acquired for write op
0709 12:59:53.751267 (+    32us) tablet.cc:685] Acquiring locks for 6 operations
0709 12:59:53.751971 (+   704us) tablet.cc:701] Row locks acquired
0709 12:59:53.752021 (+    50us) write_op.cc:260] PREPARE: finished
0709 12:59:53.752263 (+   242us) write_op.cc:270] Start()
0709 12:59:53.752432 (+   169us) write_op.cc:276] Timestamp: P: 1783601993752160 usec, L: 0
0709 12:59:53.752458 (+    26us) op_driver.cc:348] REPLICATION: starting
0709 12:59:53.786235 (+ 33777us) log.cc:844] Serialized 2723 byte log entry
0709 12:59:54.394646 (+608411us) op_driver.cc:464] REPLICATION: finished
0709 12:59:55.301319 (+906673us) write_op.cc:301] APPLY: starting
0709 12:59:55.301471 (+   152us) tablet.cc:1367] starting BulkCheckPresence
0709 12:59:55.301769 (+   298us) tablet.cc:1370] finished BulkCheckPresence
0709 12:59:55.328497 (+ 26728us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:59:55.363905 (+ 35408us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:59:55.364293 (+   388us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=6
0709 12:59:55.364350 (+    57us) write_op.cc:312] APPLY: finished
0709 12:59:55.364755 (+   405us) log.cc:844] Serialized 53 byte log entry
0709 12:59:55.533774 (+169019us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:59:55.559399 (+ 25625us) write_op.cc:454] Released schema lock
0709 12:59:55.566759 (+  7360us) write_op.cc:341] FINISH: Updating metrics
0709 12:59:55.640399 (+ 73640us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:59:55.644415 (+  4016us) write_op.cc:454] Released schema lock
Metrics: {"dns_us":244,"mutex_wait_us":294501,"reported_tablets":17,"child_traces":[["op",{"apply.queue_time_us":906235,"apply.run_cpu_time_us":16293,"apply.run_wall_time_us":306703,"num_ops":6,"prepare.queue_time_us":108169,"prepare.run_cpu_time_us":23507,"prepare.run_wall_time_us":247321,"raft.queue_time_us":305218,"raft.run_cpu_time_us":39213,"raft.run_wall_time_us":919951,"replication_time_us":634341,"thread_start_us":1409054,"threads_started":5,"wal-append.queue_time_us":405139,"wal-append.run_cpu_time_us":27040,"wal-append.run_wall_time_us":2351332}]]}
I20260709 12:59:58.758975  4242 raft_consensus.cc:515] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:58.741343  1644 leader_election.cc:304] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 12:59:58.702862  4284 raft_consensus.cc:2804] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Leader election won for term 6
W20260709 12:59:58.771853  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 113441us
W20260709 12:59:58.825037  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 83773.4us
W20260709 12:59:58.826310  1718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:59:58.827183  1718 leader_election.cc:400] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 election: Vote denied by peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625) with higher term. Message: Invalid argument: T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 5. Current term is 6.
I20260709 12:59:58.827483  1718 leader_election.cc:403] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
I20260709 12:59:58.828203  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:58.828869  1753 raft_consensus.cc:3060] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:59:58.829241  4138 leader_election.cc:290] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:58.829774  4284 raft_consensus.cc:697] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 6 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 12:59:58.830791  4284 consensus_queue.cc:237] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:58.795318  4283 raft_consensus.cc:493] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:59:58.773869  3854 raft_consensus.cc:3055] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:59:58.717068  4290 raft_consensus.cc:2804] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:59:58.677145  1751 raft_consensus.cc:1240] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 4 LEADER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 3. Current term is 4. Ops: []
W20260709 12:59:58.787031  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 109588us
I20260709 12:59:58.586520  4302 raft_consensus.cc:515] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:58.790112  4297 leader_election.cc:290] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:58.794843  1571 leader_election.cc:400] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 4 election: Vote denied by peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693) with higher term. Message: Invalid argument: T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 4. Current term is 5.
I20260709 12:59:58.701335  4201 raft_consensus.cc:515] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:58.757650  1500 catalog_manager.cc:5697] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: HEALTHY } } }
W20260709 12:59:58.873073  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 78371.9us
I20260709 12:59:58.874337  1571 leader_election.cc:304] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 12:59:58.802882  4139 raft_consensus.cc:3060] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:59:58.790252  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:58.798688  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:58.884946  1754 raft_consensus.cc:2368] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 for earlier term 4. Current term is 5.
I20260709 12:59:58.765554  4301 consensus_queue.cc:1059] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:59:58.829085  1718 leader_election.cc:304] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 12:59:58.886487  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 57471us
I20260709 12:59:58.887571  1718 leader_election.cc:304] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 12:59:58.772665  4218 leader_election.cc:290] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 7 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:58.888528  3948 consensus_queue.cc:260] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:58.889704  3948 raft_consensus.cc:3060] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:59:58.804653  4100 consensus_queue.cc:1059] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.064s
I20260709 12:59:58.843044  4249 raft_consensus.cc:2804] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:59:58.849972  1748 raft_consensus.cc:2468] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 5.
I20260709 12:59:58.871627  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:58.843526  4171 raft_consensus.cc:2804] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:59:58.822886  2331 raft_consensus.cc:1275] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260709 12:59:58.874854  1500 catalog_manager.cc:5697] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:59:58.805049  4206 raft_consensus.cc:515] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:58.914831  2334 raft_consensus.cc:2468] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 5.
I20260709 12:59:58.850822  3854 raft_consensus.cc:740] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 12:59:58.773767  1675 raft_consensus.cc:2468] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 6.
I20260709 12:59:58.838919  2301 leader_election.cc:400] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 4 pre-election: Vote denied by peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693) with higher term. Message: Invalid argument: T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 for earlier term 4. Current term is 5.
I20260709 12:59:58.891173  1718 leader_election.cc:400] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 election: Vote denied by peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625) with higher term. Message: Invalid argument: T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 5. Current term is 6.
I20260709 12:59:58.918920  1718 leader_election.cc:403] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
W20260709 12:59:58.918879  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 94912.8us
I20260709 12:59:58.899533  1500 catalog_manager.cc:5697] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:59:58.823395  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 12:59:58.875563  1571 leader_election.cc:304] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 12:59:58.882422  1747 raft_consensus.cc:1275] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260709 12:59:58.851121  4290 raft_consensus.cc:493] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:59:58.908730  3906 raft_consensus.cc:515] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:58.887187  1753 raft_consensus.cc:2468] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 6.
I20260709 12:59:58.930583  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:58.851596  4283 raft_consensus.cc:3060] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:59:58.921197  2301 leader_election.cc:400] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 4 election: Vote denied by peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693) with higher term. Message: Invalid argument: T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 for earlier term 4. Current term is 5.
I20260709 12:59:58.878573  4310 raft_consensus.cc:2749] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20260709 12:59:58.924508  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 49008.6us
I20260709 12:59:58.948408  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:58.949225  1571 leader_election.cc:304] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 12:59:58.950559  1571 leader_election.cc:304] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 12:59:58.951828  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:59:58.952611  1676 raft_consensus.cc:2376] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 6. Re-sending same reply.
I20260709 12:59:58.920459  1500 catalog_manager.cc:5697] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } }
I20260709 12:59:58.898152  4249 raft_consensus.cc:493] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:59:58.893442  3999 raft_consensus.cc:2749] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:59:58.988122  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:58.946949  2301 leader_election.cc:403] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
I20260709 12:59:58.955555  1500 catalog_manager.cc:5697] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 2 to 4, leader changed from 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194) to 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193). New cstate: current_term: 4 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:59:58.921723  1749 raft_consensus.cc:1275] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
W20260709 12:59:59.000484  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 79419.1us
I20260709 12:59:58.968042  4249 raft_consensus.cc:3060] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:59:58.893492  4313 raft_consensus.cc:2749] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:59:58.970270  1571 leader_election.cc:304] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 12:59:58.924746  4171 raft_consensus.cc:697] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 5 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 12:59:58.974771  2299 leader_election.cc:304] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
W20260709 12:59:59.022565  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 47895.3us
I20260709 12:59:58.930678  4290 raft_consensus.cc:3060] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:59:58.868317  3977 consensus_queue.cc:260] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:58.972587  4201 leader_election.cc:290] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:58.928689  4301 raft_consensus.cc:2749] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:59:58.940762  2328 raft_consensus.cc:2376] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 6. Re-sending same reply.
I20260709 12:59:58.898092  4265 consensus_queue.cc:1048] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.072s
I20260709 12:59:58.983714  4204 leader_election.cc:290] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 5 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:58.985863  1746 raft_consensus.cc:2468] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 6.
I20260709 12:59:58.946198  3521 raft_consensus.cc:2749] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625) with higher term. Message: Invalid argument: T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 5. Current term is 6.
W20260709 12:59:58.987676  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:59:58.997630  1500 catalog_manager.cc:5697] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 3 to 4, leader changed from 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) to 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193). New cstate: current_term: 4 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:59:59.009594  4297 raft_consensus.cc:3060] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:59:59.010311  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:59.040421  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:58.969223  4138 raft_consensus.cc:493] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 12:59:59.008993  1569 leader_election.cc:304] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 12:59:58.970521  4273 raft_consensus.cc:493] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 12:59:58.974577  4199 raft_consensus.cc:2749] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:59:58.970813  4307 raft_consensus.cc:2804] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:59:59.048025  4157 leader_election.cc:290] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:59.048671  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:59.038394  4304 raft_consensus.cc:2804] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:59:59.034843  4310 raft_consensus.cc:2749] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:59:59.054301  4281 raft_consensus.cc:2749] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625) with higher term. Message: Invalid argument: T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 5. Current term is 6.
I20260709 12:59:59.034818  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:59.014068  1571 leader_election.cc:304] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 12:59:59.075179  4301 raft_consensus.cc:2749] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:59:59.098372  4157 raft_consensus.cc:2749] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:59:59.119369  3906 leader_election.cc:290] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 6 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:59.120440  4319 tablet_replica.cc:333] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1: stopping tablet replica
I20260709 12:59:59.121872  4302 leader_election.cc:290] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:59.082034  4307 raft_consensus.cc:493] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:59:59.124274  4307 raft_consensus.cc:3060] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:59:59.041112  4323 consensus_queue.cc:1048] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.059s
I20260709 12:59:59.082768  4138 raft_consensus.cc:515] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:59.066387  4184 leader_election.cc:290] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:58.958009  4289 consensus_queue.cc:237] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:59.056720  1500 catalog_manager.cc:5697] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } }
W20260709 12:59:59.072423  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 63522.3us
I20260709 12:59:59.055791  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:59.165153  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 12:59:59.052947  4070 leader_election.cc:290] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 4 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:59.166921  2331 raft_consensus.cc:3060] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:59:59.127547  4331 raft_consensus.cc:493] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 12:59:59.167887  4331 raft_consensus.cc:515] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:59.161814  1500 catalog_manager.cc:5697] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 3 to 4, leader changed from 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) to 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194). New cstate: current_term: 4 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } }
I20260709 12:59:59.091830  4304 raft_consensus.cc:493] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:59:59.166163  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:59.170457  1500 catalog_manager.cc:5697] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 2 to 3, leader changed from 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) to 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194). New cstate: current_term: 3 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } }
I20260709 12:59:59.176040  1500 catalog_manager.cc:5697] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 1 to 2, leader changed from 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) to 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196). New cstate: current_term: 2 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:59.177202  1500 catalog_manager.cc:5697] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 4 to 5, leader changed from 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194) to 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193). New cstate: current_term: 5 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:59:59.083230  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:59:59.171741  1569 leader_election.cc:304] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
W20260709 12:59:59.172380  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 158361us
I20260709 12:59:59.359939  1571 leader_election.cc:304] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 12:59:59.107345  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:59.045480  2301 leader_election.cc:304] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 12:59:59.172806  4171 consensus_queue.cc:237] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:59.364530  4299 raft_consensus.cc:2804] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:59:59.365091  4299 raft_consensus.cc:493] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 12:59:59.365484  4299 raft_consensus.cc:3060] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:59:59.266863  4284 raft_consensus.cc:493] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 12:59:59.366803  4284 raft_consensus.cc:515] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:59.368116  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:59.368827  2328 raft_consensus.cc:2468] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 5.
I20260709 12:59:59.369895  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 12:59:59.279042  4138 leader_election.cc:290] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:59.091538  4273 raft_consensus.cc:515] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:59.288601  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
W20260709 12:59:59.261835  4328 log.cc:927] Time spent T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.204s	user 0.000s	sys 0.001s
I20260709 12:59:59.267863  4200 leader_election.cc:290] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:59.267462  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:59:59.297581  4331 leader_election.cc:290] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:59.301811  4139 raft_consensus.cc:2749] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:59:59.309281  4290 raft_consensus.cc:515] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 12:59:59.313320  4140 consensus_meta.cc:287] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.627s	user 0.008s	sys 0.005s
I20260709 12:59:59.325361  2331 raft_consensus.cc:2468] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 5.
I20260709 12:59:59.399137  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:59.400791  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 12:59:59.401384  4218 raft_consensus.cc:3055] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:59:59.401993  4218 raft_consensus.cc:740] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 12:59:59.402621  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:59.403829  4307 raft_consensus.cc:515] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 12:59:59.229611  1718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:59:59.405609  1718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:59:59.362041  1571 leader_election.cc:304] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 12:59:59.406586  4126 leader_election.cc:290] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 12:59:59.357826  1750 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43156 (request call id 453) took 1953 ms (client timeout 2082 ms). Trace:
W20260709 12:59:59.407711  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 45757.8us
W20260709 12:59:59.407668  1750 rpcz_store.cc:269] 0709 12:59:57.404519 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:57.405799 (+  1280us) service_pool.cc:224] Handling call
0709 12:59:59.357472 (+1951673us) spinlock_profiling.cc:243] Waited 772 ms on lock 0x7b54003700d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 12:59:59.357723 (+   251us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1775526272}
I20260709 12:59:59.370517  4284 leader_election.cc:290] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:59.156030  4323 raft_consensus.cc:493] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 12:59:59.413386  2332 raft_consensus.cc:2393] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 5: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 12:59:59.378060  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:59.175786  4304 raft_consensus.cc:3060] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:59:59.173534  4242 leader_election.cc:290] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 12:59:59.388168  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 342694us
I20260709 12:59:59.382683  4182 raft_consensus.cc:2804] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:59:59.382015  4249 raft_consensus.cc:515] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:59.193377  1500 catalog_manager.cc:5697] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20260709 12:59:59.420374  1646 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.528s	user 0.006s	sys 0.020s
W20260709 12:59:59.420754  1646 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.529s	user 0.006s	sys 0.020s
W20260709 12:59:59.421068  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 554463us
I20260709 12:59:59.421406  1500 catalog_manager.cc:5697] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 4 to 5, leader changed from 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194) to 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193). New cstate: current_term: 5 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:59:59.404258  4273 leader_election.cc:290] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:59.403380  4218 consensus_queue.cc:260] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 12:59:59.277321  4300 log.cc:927] Time spent T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.756s	user 0.013s	sys 0.007s
I20260709 12:59:59.409379  1571 leader_election.cc:304] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 12:59:59.410027  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:59.411363  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:59.431535  2333 raft_consensus.cc:2468] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 5.
I20260709 12:59:59.432533  1571 leader_election.cc:304] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
W20260709 12:59:59.374164  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 202497us
I20260709 12:59:59.432579  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:59.394364  4070 raft_consensus.cc:2749] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:59:59.419878  2301 leader_election.cc:304] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:59:59.426397  1646 leader_election.cc:304] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:59:59.411679  4139 raft_consensus.cc:2749] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260709 12:59:59.438345  1569 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:59:59.442971  1644 leader_election.cc:304] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 12:59:59.461462  1671 raft_consensus.cc:2393] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 5: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 12:59:59.445245  1673 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52766 (request call id 2121) took 1320 ms. Trace:
I20260709 12:59:59.446225  1679 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52778 (request call id 1717) took 1442 ms. Trace:
I20260709 12:59:59.462726  1679 rpcz_store.cc:276] 0709 12:59:58.003526 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:58.003652 (+   126us) service_pool.cc:224] Handling call
0709 12:59:58.446155 (+442503us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:59:58.449915 (+  3760us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:59.445966 (+996051us) spinlock_profiling.cc:243] Waited 442 ms on lock 0x7b5400163850. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661618ad 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 12:59:59.446127 (+   161us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1016515456}
I20260709 12:59:59.443360  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:59:59.462275  1673 rpcz_store.cc:276] 0709 12:59:58.124690 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:58.184116 (+ 59426us) service_pool.cc:224] Handling call
0709 12:59:58.658989 (+474873us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:59:58.774767 (+115778us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:59.444947 (+670180us) spinlock_profiling.cc:243] Waited 474 ms on lock 0x7b54001f3e50. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661618ad 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 12:59:59.445143 (+   196us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1091066752}
W20260709 12:59:59.430923  3948 consensus_meta.cc:287] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.539s	user 0.007s	sys 0.007s
I20260709 12:59:59.473636  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:59:59.475185  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:59:59.476540  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:59.470046  1644 leader_election.cc:304] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: af520b67945d443b901bbeca37cc5701
I20260709 12:59:59.411893  4140 raft_consensus.cc:515] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:59.431207  4182 raft_consensus.cc:697] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 12:59:59.488435  1673 raft_consensus.cc:2468] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 6.
I20260709 12:59:59.469229  4215 raft_consensus.cc:2749] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:59:59.438488  4218 raft_consensus.cc:3060] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:59:59.421058  4323 raft_consensus.cc:515] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:59.460870  4272 raft_consensus.cc:2749] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:59:59.428905  4284 consensus_queue.cc:1048] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.051s
I20260709 12:59:59.466645  4144 raft_consensus.cc:2804] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Leader election won for term 6
W20260709 12:59:59.500006  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:59:59.508256  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:59.528041  2335 raft_consensus.cc:2393] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 6: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 12:59:59.512233  1569 leader_election.cc:304] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 12:59:59.531450  4182 consensus_queue.cc:237] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 12:59:59.534140  4335 log.cc:927] Time spent T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.172s	user 0.000s	sys 0.001s
I20260709 12:59:59.565946  2330 raft_consensus.cc:1240] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 5. Current term is 6. Ops: [5.6-5.6]
I20260709 12:59:59.567281  4329 raft_consensus.cc:2804] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:59:59.567883  4329 raft_consensus.cc:493] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Starting leader election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 12:59:59.568388  4329 raft_consensus.cc:3060] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:59:59.529693  4283 consensus_meta.cc:287] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.582s	user 0.011s	sys 0.009s
I20260709 12:59:59.524070  4310 consensus_queue.cc:1048] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.049s
I20260709 12:59:59.506188  4331 raft_consensus.cc:493] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 12:59:59.572340  4331 raft_consensus.cc:515] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:59.489763  4273 raft_consensus.cc:2804] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Leader election won for term 6
W20260709 12:59:59.488862  4325 log.cc:927] Time spent T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.057s	user 0.000s	sys 0.000s
I20260709 12:59:59.470866  4126 raft_consensus.cc:2804] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:59:59.492379  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:59.522691  4139 consensus_queue.cc:1048] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:59:59.525172  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:59.584071  1679 raft_consensus.cc:2410] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260709 12:59:59.584209  1605 raft_consensus.cc:2410] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for term 7 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260709 12:59:59.525497  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:59.526860  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:59.539654  4319 raft_consensus.cc:2243] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 12:59:59.594825  4319 pending_rounds.cc:70] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1: Trying to abort 1 pending ops.
I20260709 12:59:59.543176  4249 leader_election.cc:290] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 5 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:59.543638  1675 raft_consensus.cc:3060] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:59:59.595235  4319 pending_rounds.cc:77] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1: Aborting op as it isn't in flight: id { term: 4 index: 5 } timestamp: 7305633758408167424 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" old_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } } } new_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } } }
I20260709 12:59:59.596956  4319 raft_consensus.cc:2917] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Aborting config change with OpId 4.5: Aborted: Op aborted
I20260709 12:59:59.597559  4319 raft_consensus.cc:2889] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 4 LEARNER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20260709 12:59:59.598006  4319 raft_consensus.cc:2272] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 4 LEARNER]: Raft consensus is shut down!
I20260709 12:59:59.543952  4323 leader_election.cc:290] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:59.544251  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:59.546686  4263 raft_consensus.cc:2955] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 4 LEADER]: Committing config change with OpId 4.5: config changed from index 3 to 5, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:59.550016  4246 raft_consensus.cc:3055] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 LEADER]: Stepping down as leader of term 4
W20260709 12:59:59.609122  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 916808us
I20260709 12:59:59.551853  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:59.610086  1603 raft_consensus.cc:2393] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 5: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 12:59:59.555588  4307 leader_election.cc:290] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 5 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:59.555498  3854 consensus_queue.cc:260] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:59.556624  4301 raft_consensus.cc:2749] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:59:59.557411  4140 leader_election.cc:290] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 6 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 12:59:59.557839  1751 raft_consensus.cc:1240] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 4. Current term is 5. Ops: []
I20260709 12:59:59.558833  4144 raft_consensus.cc:697] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 6 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 12:59:59.560024  4285 consensus_queue.cc:1059] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.031s
I20260709 12:59:59.560429  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:59.622610  1748 raft_consensus.cc:2393] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 7: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 12:59:59.561308  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:59.563491  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 12:59:59.563840  4324 log.cc:927] Time spent T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.056s	user 0.000s	sys 0.000s
I20260709 12:59:59.569558  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:59.629446  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:59:59.513756  2334 raft_consensus.cc:1275] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:59:59.538131  3977 raft_consensus.cc:3060] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:59:59.512315  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:59.580466  1569 leader_election.cc:304] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 12:59:59.586863  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:59.631583  1754 raft_consensus.cc:3060] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:59:59.584688  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 114721us
I20260709 12:59:59.586802  4206 leader_election.cc:290] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 12:59:59.591017  4166 leader_election.cc:290] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:59.585335  4283 raft_consensus.cc:515] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 12:59:59.592935  4297 consensus_meta.cc:287] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.526s	user 0.008s	sys 0.005s
I20260709 12:59:59.644624  1673 raft_consensus.cc:1275] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:59:59.645062  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:59.645808  1604 raft_consensus.cc:2468] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 6.
I20260709 12:59:59.600052  1718 leader_election.cc:304] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 12:59:59.600499  4126 raft_consensus.cc:697] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 5 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 12:59:59.604198  1596 raft_consensus.cc:2393] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 4: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 12:59:59.652415  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:59.653525  1603 raft_consensus.cc:2393] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 5: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 12:59:59.655361  1602 raft_consensus.cc:2468] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 5.
W20260709 12:59:59.656906  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 56949us
I20260709 12:59:59.619432  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:59.634863  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:59:59.659359  1598 raft_consensus.cc:3060] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:59:59.632977  1644 leader_election.cc:304] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 12:59:59.638275  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:59.647537  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:59:59.662935  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:59.664005  2332 raft_consensus.cc:2393] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 4: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 12:59:59.601500  4273 raft_consensus.cc:697] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
W20260709 12:59:59.666667  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 135826us
I20260709 12:59:59.671147  1569 leader_election.cc:304] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 12:59:59.609597  4246 raft_consensus.cc:740] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 12:59:59.668162  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:59.675840  1569 leader_election.cc:304] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 12:59:59.677647  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:59.678526  1753 raft_consensus.cc:2393] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 6: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 12:59:59.674288  4138 raft_consensus.cc:2804] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:59:59.687544  4138 raft_consensus.cc:493] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 12:59:59.687934  4138 raft_consensus.cc:3060] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:59:59.687997  4139 raft_consensus.cc:2749] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:59:59.678725  1569 leader_election.cc:304] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 12:59:59.691000  4333 consensus_queue.cc:1059] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:59:59.618402  3906 consensus_queue.cc:1059] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.060s
I20260709 12:59:59.692567  1600 raft_consensus.cc:2468] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 4.
I20260709 12:59:59.693157  4307 raft_consensus.cc:2749] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:59:59.693550  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:59:59.693641  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:59:59.693866  4272 raft_consensus.cc:2749] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:59:59.694202  1673 raft_consensus.cc:3060] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:59:59.694595  4272 consensus_queue.cc:1059] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:59:59.696040  4199 raft_consensus.cc:3055] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:59:59.696509  4199 raft_consensus.cc:740] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 12:59:59.692893  1569 leader_election.cc:304] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 12:59:59.687067  4319 ts_tablet_manager.cc:1916] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:59:59.700395  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:59.701164  1596 raft_consensus.cc:2393] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 5: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 12:59:59.701944  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:59.704553  1673 raft_consensus.cc:2468] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 5.
I20260709 12:59:59.705199  4326 raft_consensus.cc:493] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 12:59:59.712224  4326 raft_consensus.cc:515] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:59.715204  4331 leader_election.cc:290] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:59.716348  1746 raft_consensus.cc:1240] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 5. Current term is 6. Ops: []
I20260709 12:59:59.719221  4273 consensus_queue.cc:237] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:59.722620  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 12:59:59.644850  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:59.724498  1749 raft_consensus.cc:2468] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 6.
I20260709 12:59:59.728595  4283 leader_election.cc:290] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 6 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 12:59:59.710103  2335 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39738 (request call id 637) took 1134 ms. Trace:
I20260709 12:59:59.705438  4199 consensus_queue.cc:260] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 4, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 12:59:59.736621  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 43776.7us
I20260709 12:59:59.735636  2335 rpcz_store.cc:276] 0709 12:59:58.575965 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:58.577994 (+  2029us) service_pool.cc:224] Handling call
0709 12:59:59.709900 (+1131906us) spinlock_profiling.cc:243] Waited 909 ms on lock 0x7b54005e0e54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 12:59:59.710038 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2090410368}
W20260709 12:59:59.737006  1718 consensus_peers.cc:597] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 -> Peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625): Couldn't send request to peer 3b879799899a499c8d409ee9f1277cd1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: STOPPING. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:59:59.720088  2329 raft_consensus.cc:1240] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20260709 12:59:59.734228  1752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55840 (request call id 2088) took 1564 ms. Trace:
W20260709 12:59:59.743574  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 85714.8us
I20260709 12:59:59.725956  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:59.746666   431 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:59:59.675668  4285 raft_consensus.cc:2804] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:59:59.671027  4262 raft_consensus.cc:2804] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:59:59.744021  1752 rpcz_store.cc:276] 0709 12:59:58.169589 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:58.569067 (+399478us) service_pool.cc:224] Handling call
0709 12:59:59.733972 (+1164905us) spinlock_profiling.cc:243] Waited 924 ms on lock 0x7b5400220954. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 12:59:59.734142 (+   170us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2125653376}
I20260709 12:59:59.752139  1600 raft_consensus.cc:2468] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 2.
I20260709 12:59:59.663069  4297 raft_consensus.cc:2749] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693) with higher term. Message: Invalid argument: T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 for earlier term 4. Current term is 5.
I20260709 12:59:59.759466  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:59:59.770542  2329 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39726 (request call id 744) took 1059 ms. Trace:
I20260709 12:59:59.751554  1569 leader_election.cc:304] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 12:59:59.771349  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:59.776690  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:59.778940  1604 raft_consensus.cc:2393] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 5: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 12:59:59.755342  4199 raft_consensus.cc:3060] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:59:59.781919  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:59.783609  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:59.741364  4139 raft_consensus.cc:2749] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:59:59.789527  1643 leader_election.cc:304] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 12:59:59.787213  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:59.788364  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:59.792392  1598 raft_consensus.cc:2468] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 6.
I20260709 12:59:59.793361  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:59:59.794029  1598 raft_consensus.cc:2368] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for earlier term 4. Current term is 5.
I20260709 12:59:59.795779  1754 raft_consensus.cc:2468] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 6.
W20260709 12:59:59.798229  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 134910us
I20260709 12:59:59.798969  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:59.799618  1753 raft_consensus.cc:2468] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 5.
I20260709 12:59:59.801077  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 12:59:59.801460  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 153499us
W20260709 12:59:59.802000  2337 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49082 (request call id 595) took 2228 ms (client timeout 2206 ms). Trace:
I20260709 12:59:59.801992  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:59.775306  1749 raft_consensus.cc:3060] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:59:59.768939  4262 raft_consensus.cc:493] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 12:59:59.804832  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 12:59:59.813066  1673 raft_consensus.cc:2468] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 2.
W20260709 12:59:59.806542  1718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:59:59.806718  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:59:59.818099  1671 raft_consensus.cc:3060] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:59:59.804394  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: true dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:59:59.819514  1596 raft_consensus.cc:2376] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 5. Re-sending same reply.
I20260709 12:59:59.820443  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
W20260709 12:59:59.802423  2337 rpcz_store.cc:269] 0709 12:59:57.573440 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:57.573935 (+   495us) service_pool.cc:224] Handling call
0709 12:59:57.763819 (+189884us) spinlock_profiling.cc:243] Waited 3.09 ms on lock 0x7b5800178598. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f426303db71 00007f426303da22 00007f426bf56c09 00007f426b0b0ee0 00007f426b17a0ab 00007f426b17a131 00007f426b1280a0 00007f426b127984 00007f42646a26d4 00007f42646a2662
0709 12:59:59.801801 (+2037982us) spinlock_profiling.cc:243] Waited 869 ms on lock 0x7b54006ef854. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 12:59:59.801912 (+   111us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2005781632}
I20260709 12:59:59.810793  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:59:59.781131  4272 raft_consensus.cc:493] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:59:59.824242  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:59:59.781018  4285 raft_consensus.cc:697] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 12:59:59.762841  4166 consensus_queue.cc:1059] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.041s
W20260709 12:59:59.829306  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 188775us
I20260709 12:59:59.790503  4140 raft_consensus.cc:3055] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:59:59.788556  4215 raft_consensus.cc:2804] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:59:59.811350  1674 raft_consensus.cc:2955] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Committing config change with OpId 4.5: config changed from index 3 to 5, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 12:59:59.831790  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 12:59:59.832880  1718 leader_election.cc:304] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 12:59:59.835177  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:59.794737  4249 raft_consensus.cc:2749] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:59:59.830394  4262 raft_consensus.cc:3060] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:59:59.838346  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:59.840255  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 12:59:59.780196  4283 consensus_queue.cc:1059] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.032s
I20260709 12:59:59.771020  2329 rpcz_store.cc:276] 0709 12:59:58.711224 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:58.715555 (+  4331us) service_pool.cc:224] Handling call
0709 12:59:59.719221 (+1003666us) spinlock_profiling.cc:243] Waited 5.6 ms on lock 0x7b5800178598. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f426303db71 00007f426303da22 00007f426bf56c09 00007f426b0b0ee0 00007f426b17a0ab 00007f426b17a131 00007f426b13bfc0 00007f426b126929 00007f426b1263d2 00007f426469dbd4
0709 12:59:59.719614 (+   393us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:59:59.760063 (+ 40449us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:59.770401 (+ 10338us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":12889344}
W20260709 12:59:59.873832  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 91772.7us
I20260709 12:59:59.836613  2301 leader_election.cc:304] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
W20260709 12:59:59.877245  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 44477.7us
I20260709 12:59:59.879197  1598 raft_consensus.cc:2468] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 6.
I20260709 12:59:59.893743  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:59.893957  1753 raft_consensus.cc:1275] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:59:59.893087  1569 leader_election.cc:304] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 12:59:59.895509  1598 raft_consensus.cc:2468] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 4.
I20260709 12:59:59.896783  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 12:59:59.897246  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 12:59:59.897637  1600 raft_consensus.cc:2376] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 5. Re-sending same reply.
I20260709 12:59:59.898137  1677 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52778 (request call id 1741) took 1266 ms. Trace:
I20260709 12:59:59.898623  1677 rpcz_store.cc:276] 0709 12:59:58.631497 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:59.880774 (+1249277us) service_pool.cc:224] Handling call
0709 12:59:59.898069 (+ 17295us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:59:59.900028  1718 leader_election.cc:304] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 12:59:59.877552  4215 raft_consensus.cc:493] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:59:59.839747  4140 raft_consensus.cc:740] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 12:59:59.902129  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:59.832782  4272 raft_consensus.cc:515] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 12:59:59.789949  4144 consensus_queue.cc:237] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:59.908432  1754 raft_consensus.cc:1275] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 12:59:59.888489  4206 raft_consensus.cc:2804] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:59:59.833113  4297 consensus_queue.cc:1048] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.064s
I20260709 12:59:59.885854  1671 raft_consensus.cc:2468] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 6.
I20260709 12:59:59.831558  4200 consensus_queue.cc:1048] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.068s
I20260709 12:59:59.805177  4126 consensus_queue.cc:237] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 12:59:59.912642  1754 raft_consensus.cc:1240] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 6. Current term is 7. Ops: []
I20260709 12:59:59.912721  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:59.915616  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 12:59:59.844643  4329 raft_consensus.cc:515] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 12:59:59.915949  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 79440.7us
W20260709 12:59:59.901511  4299 consensus_meta.cc:287] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.535s	user 0.010s	sys 0.006s
I20260709 12:59:59.915880  1718 leader_election.cc:304] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 12:59:59.895308  4150 raft_consensus.cc:3055] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:59:59.913424  4206 raft_consensus.cc:493] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:00.030405  4350 raft_consensus.cc:493] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:00:00.036767  4340 log.cc:927] Time spent T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.191s	user 0.001s	sys 0.000s
I20260709 13:00:00.036540  4350 raft_consensus.cc:515] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:00.038666  1676 raft_consensus.cc:1240] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 5. Current term is 6. Ops: []
I20260709 13:00:00.038843  1678 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52766 (request call id 2122) took 1099 ms. Trace:
I20260709 13:00:00.039372  1678 rpcz_store.cc:276] 0709 12:59:58.939543 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:58.941705 (+  2162us) service_pool.cc:224] Handling call
0709 12:59:58.942115 (+   410us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:59:59.413326 (+471211us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:59:59.413360 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:59:59.414001 (+   641us) log.cc:844] Serialized 26 byte log entry
0709 12:59:59.610726 (+196725us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:59:59.610767 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:59:59.610837 (+    70us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:00:00.038520 (+427683us) raft_consensus.cc:1685] finished
0709 13:00:00.038551 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:00.038758 (+   207us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":53140,"raft.run_cpu_time_us":221,"raft.run_wall_time_us":32929,"thread_start_us":648208,"threads_started":2,"wal-append.queue_time_us":350424}
I20260709 12:59:59.905424  4281 raft_consensus.cc:2749] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:59:59.916365  1598 raft_consensus.cc:2393] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 4: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:00:00.044510  4140 consensus_queue.cc:260] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:00.043449  4299 raft_consensus.cc:515] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:00.045324  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:00.052208  1597 raft_consensus.cc:1240] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 4. Current term is 5. Ops: []
W20260709 13:00:00.019599  1643 leader_election.cc:336] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 1.559s (SENT)
I20260709 13:00:00.057206  1643 leader_election.cc:304] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:00:00.019063  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:00.059262  1600 raft_consensus.cc:3060] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:00:00.053010  1598 raft_consensus.cc:2393] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 5: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
W20260709 13:00:00.061177  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 145389us
I20260709 12:59:59.944869  1752 raft_consensus.cc:1240] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 5. Current term is 6. Ops: [3.5-5.6]
W20260709 13:00:00.064334  1718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:00:00.066401  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 46880.5us
W20260709 13:00:00.034615  4347 log.cc:927] Time spent T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.128s	user 0.001s	sys 0.000s
I20260709 13:00:00.039274  4329 leader_election.cc:290] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 6 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:00.066903  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 12:59:59.908701  4215 raft_consensus.cc:3060] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Advancing to term 7
W20260709 13:00:00.058614  1570 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:00.113306  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:00.058131  4150 raft_consensus.cc:740] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:00:00.115314  4290 leader_election.cc:290] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 5 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:00.117197  1754 pending_rounds.cc:85] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701: Aborting all ops after (but not including) 1
I20260709 13:00:00.140602  1754 pending_rounds.cc:107] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20260709 13:00:00.141131  1754 raft_consensus.cc:2889] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:00:00.118252  1746 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43156 (request call id 470) took 1115 ms. Trace:
W20260709 13:00:00.120194  4354 log.cc:927] Time spent T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.064s	user 0.000s	sys 0.000s
I20260709 13:00:00.048882  4355 raft_consensus.cc:2804] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:00:00.035389  4206 raft_consensus.cc:3060] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:00:00.142683  1600 raft_consensus.cc:2468] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 6.
W20260709 13:00:00.072360  1643 leader_election.cc:336] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 1.610s (SENT)
I20260709 13:00:00.161900  1643 leader_election.cc:304] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
W20260709 13:00:00.164327  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 94608.4us
W20260709 13:00:00.165735  1643 leader_election.cc:336] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 1.631s (SENT)
W20260709 13:00:00.149103  1717 connection.cc:256] RPC call timeout handler was delayed by -1.5092517639999983s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2391s for a response
W20260709 13:00:00.130832  4341 log.cc:927] Time spent T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.303s	user 0.000s	sys 0.001s
I20260709 13:00:00.168591  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:00:00.169341  1717 connection.cc:256] RPC call timeout handler was delayed by -1.4909336140000278s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1849s for a response
W20260709 13:00:00.160152  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 267110us
I20260709 13:00:00.168627  1643 leader_election.cc:304] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:00:00.070360  4283 raft_consensus.cc:2804] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 13:00:00.169869  1598 raft_consensus.cc:2410] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for term 6 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 13:00:00.171468  4249 raft_consensus.cc:2749] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260709 13:00:00.172362  1643 leader_election.cc:336] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 1.655s (SENT)
W20260709 13:00:00.177809  1643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:00.183619  2298 leader_election.cc:304] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:00.191967  4133 raft_consensus.cc:2749] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:00:00.157181  4344 raft_consensus.cc:2749] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:00:00.195127  2298 leader_election.cc:304] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:00.201992  1598 raft_consensus.cc:1275] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
W20260709 13:00:00.165446  4348 log.cc:927] Time spent T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.122s	user 0.001s	sys 0.004s
I20260709 13:00:00.104138  4343 consensus_queue.cc:1059] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.089s
I20260709 13:00:00.175397  4277 raft_consensus.cc:2749] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 13:00:00.166779  4355 raft_consensus.cc:697] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:00:00.215591  2298 leader_election.cc:304] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
W20260709 13:00:00.264272  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 48763.9us
I20260709 13:00:00.284754  1598 raft_consensus.cc:1275] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260709 13:00:00.198072  4283 raft_consensus.cc:697] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 6 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 12:59:59.943785  4285 consensus_queue.cc:237] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:00.238351  4304 consensus_meta.cc:287] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.779s	user 0.008s	sys 0.005s
I20260709 13:00:00.206400  4200 raft_consensus.cc:493] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:00:00.302953  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 123037us
W20260709 13:00:00.305169  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 134714us
W20260709 13:00:00.308107  1717 connection.cc:256] RPC call timeout handler was delayed by -1.3193517640000323s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2602s for a response
W20260709 13:00:00.308950  1717 connection.cc:256] RPC call timeout handler was delayed by -1.2764479090000123s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.5156s for a response
I20260709 13:00:00.309935  1749 raft_consensus.cc:2468] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 5.
I20260709 13:00:00.281554  4133 raft_consensus.cc:2749] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:00:00.310921  1717 leader_election.cc:304] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:00.313289  1674 raft_consensus.cc:1275] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 13:00:00.314563  1749 raft_consensus.cc:1275] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
W20260709 13:00:00.320192  3977 consensus_meta.cc:287] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.671s	user 0.008s	sys 0.004s
I20260709 13:00:00.321048  1745 raft_consensus.cc:1275] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 3. (term mismatch)
I20260709 13:00:00.321389  1745 pending_rounds.cc:85] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701: Aborting all ops after (but not including) 2
I20260709 13:00:00.321636  1745 pending_rounds.cc:107] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701: Aborting uncommitted NO_OP operation due to leader change: 3.3
I20260709 13:00:00.321971  1745 raft_consensus.cc:2889] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:00:00.149784  1746 rpcz_store.cc:276] 0709 12:59:59.003254 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:59.092449 (+ 89195us) service_pool.cc:224] Handling call
0709 12:59:59.175661 (+ 83212us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:59:59.716925 (+541264us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:00.117902 (+400977us) spinlock_profiling.cc:243] Waited 487 ms on lock 0x7b54000e4a54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:00:00.118163 (+   261us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1119634304}
W20260709 13:00:00.325830  4356 log.cc:927] Time spent T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.256s	user 0.000s	sys 0.002s
I20260709 13:00:00.292116  3993 raft_consensus.cc:2804] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 13:00:00.331310  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:00.333232  1598 raft_consensus.cc:2468] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 6.
I20260709 13:00:00.336074  2298 leader_election.cc:304] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:00:00.344743  1680 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52778 (request call id 1733) took 1847 ms. Trace:
I20260709 13:00:00.307703  4277 raft_consensus.cc:493] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:00:00.346276  1643 leader_election.cc:336] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 1.796s (SENT)
I20260709 13:00:00.202859  4150 consensus_queue.cc:260] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:00.345495  1680 rpcz_store.cc:276] 0709 12:59:58.497575 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:58.497752 (+   177us) service_pool.cc:224] Handling call
0709 12:59:58.498167 (+   415us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:59:58.498490 (+   323us) raft_consensus.cc:1534] Early marking committed up to index 3
0709 12:59:58.509775 (+ 11285us) log.cc:844] Serialized 14 byte log entry
0709 13:00:00.193493 (+1683718us) spinlock_profiling.cc:243] Waited 2.28 ms on lock 0x7b18001bf880. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f4263057e07 00007f426beb2ab9 00007f426307469b 00007f426307293f 00007f4263074c02 00007f4265cfc20b 00007f4265d01d23 00007f4265d0223a 00007f426616ca2a
0709 13:00:00.344002 (+150509us) log.cc:844] Serialized 14 byte log entry
0709 13:00:00.344280 (+   278us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:00:00.344346 (+    66us) raft_consensus.cc:1645] Marking committed up to 3
0709 13:00:00.344415 (+    69us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:00.344510 (+    95us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:00.344662 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":11246,"prepare.run_cpu_time_us":65,"prepare.run_wall_time_us":63,"raft.queue_time_us":833,"spinlock_wait_cycles":5235584,"thread_start_us":149458,"threads_started":1}
I20260709 13:00:00.330773  4273 raft_consensus.cc:493] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
W20260709 13:00:00.358796  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:00.357280  1717 leader_election.cc:304] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 13:00:00.344218  4299 leader_election.cc:290] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 5 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:00.344084  4350 leader_election.cc:290] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:00.344614  4342 raft_consensus.cc:2804] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:00:00.324613  4200 raft_consensus.cc:515] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:00.376266  1717 leader_election.cc:304] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:00.307442  4304 raft_consensus.cc:515] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:00.347461  3993 raft_consensus.cc:697] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 6 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:00:00.379144  1569 leader_election.cc:304] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:00:00.356313  3977 raft_consensus.cc:2749] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260709 13:00:00.382483  1643 leader_election.cc:336] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 1.923s (SENT)
W20260709 13:00:00.376277  1675 consensus_meta.cc:287] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.764s	user 0.014s	sys 0.001s
I20260709 13:00:00.388175  1717 leader_election.cc:304] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:00.398313  1643 leader_election.cc:304] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
W20260709 13:00:00.400285  1643 leader_election.cc:336] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 1.928s (SENT)
W20260709 13:00:00.401201  1643 leader_election.cc:336] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 1.589s (SENT)
I20260709 13:00:00.401594  1643 leader_election.cc:304] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 13:00:00.380285  4199 consensus_meta.cc:287] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.592s	user 0.013s	sys 0.002s
I20260709 13:00:00.383083  3948 raft_consensus.cc:2804] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:00:00.383369  4150 raft_consensus.cc:3060] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:00:00.408401  1596 raft_consensus.cc:2410] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 for term 6 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260709 13:00:00.392045  4313 raft_consensus.cc:2749] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:00:00.409427  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:00:00.384935  4342 raft_consensus.cc:493] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:00.412788  2334 raft_consensus.cc:2393] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 5: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:00:00.414080  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:00.385550  4283 consensus_queue.cc:237] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:00.417547  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:00.421044  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:00.406039  4352 raft_consensus.cc:2749] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 13:00:00.379766  4273 raft_consensus.cc:515] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:00.398981  1675 raft_consensus.cc:2468] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 7.
I20260709 13:00:00.354079  4277 raft_consensus.cc:515] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:00.408088  3948 raft_consensus.cc:493] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:00.427525  4342 raft_consensus.cc:3060] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:00:00.414759  3977 raft_consensus.cc:3060] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:00:00.451938  2331 raft_consensus.cc:2393] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 7: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:00:00.458024  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:00.461287  1600 raft_consensus.cc:2393] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 6: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:00:00.439833  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:00:00.443652  4323 raft_consensus.cc:2749] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:00:00.457747  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
W20260709 13:00:00.469234  4138 consensus_meta.cc:287] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.781s	user 0.009s	sys 0.005s
I20260709 13:00:00.454814  3948 raft_consensus.cc:3060] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:00:00.470194  4304 leader_election.cc:290] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 7 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:00.470678  4299 raft_consensus.cc:3060] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:00:00.459370  3993 consensus_queue.cc:237] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:00.477795  4249 raft_consensus.cc:3055] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:00:00.478353  2328 raft_consensus.cc:3060] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:00:00.480666  2331 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49082 (request call id 616) took 1436 ms. Trace:
W20260709 13:00:00.484571  4363 log.cc:927] Time spent T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.079s	user 0.000s	sys 0.000s
I20260709 13:00:00.485013  4200 leader_election.cc:290] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:00.374209  4355 consensus_queue.cc:237] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:00.487480  4372 raft_consensus.cc:493] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:00.501814  4138 raft_consensus.cc:515] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:00.474267  3854 raft_consensus.cc:3060] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:00:00.540977  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
W20260709 13:00:00.558632  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 163124us
W20260709 13:00:00.560061  2298 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:00.495496  4206 raft_consensus.cc:515] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:00.582504  1674 raft_consensus.cc:1240] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 4. Current term is 5. Ops: [4.4-4.4]
W20260709 13:00:00.583608  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 182399us
I20260709 13:00:00.573506  4372 raft_consensus.cc:515] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:00.502099  4353 raft_consensus.cc:2749] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:00:00.584388  1677 raft_consensus.cc:1275] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260709 13:00:00.507445  4249 raft_consensus.cc:740] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:00:00.585697  1643 leader_election.cc:304] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:00:00.584448  4206 leader_election.cc:290] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:00.586404  4249 consensus_queue.cc:260] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:00.587139  4372 leader_election.cc:290] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:00.588119  4249 raft_consensus.cc:3060] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:00:00.596318  1752 raft_consensus.cc:1275] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 13:00:00.566495  1746 raft_consensus.cc:2393] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 5: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:00:00.598650  1643 leader_election.cc:304] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 13:00:00.598289  1752 raft_consensus.cc:1240] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 5 LEADER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 4. Current term is 5. Ops: [4.4-4.4]
I20260709 13:00:00.618201  1747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55848 (request call id 1883) took 2027 ms. Trace:
I20260709 13:00:00.618909  1747 rpcz_store.cc:276] 0709 12:59:58.590383 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:58.738085 (+147702us) service_pool.cc:224] Handling call
0709 12:59:58.738353 (+   268us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:59:58.921275 (+182922us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:00.617789 (+1696514us) spinlock_profiling.cc:243] Waited 144 ms on lock 0x7b54001c7354. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:00:00.618104 (+   315us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":330246528}
I20260709 13:00:00.625185  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:00.625535  1643 leader_election.cc:304] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:00:00.626039  1674 raft_consensus.cc:2468] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 5.
W20260709 13:00:00.625603  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 237460us
I20260709 13:00:00.626959  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:00.632772  1643 leader_election.cc:304] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:00.635866  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:00:00.527411  2331 rpcz_store.cc:276] 0709 12:59:59.044373 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:59.402118 (+357745us) service_pool.cc:224] Handling call
0709 13:00:00.480308 (+1078190us) spinlock_profiling.cc:243] Waited 1 s on lock 0x7b5400645354. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:00:00.480547 (+   239us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2308027264}
I20260709 13:00:00.637892  1717 leader_election.cc:304] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:00.599385  4304 raft_consensus.cc:493] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:00.643862  1717 leader_election.cc:304] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:00.646883  4206 raft_consensus.cc:2749] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:00:00.656073  4273 leader_election.cc:290] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:00.659466  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:00:00.660084  4262 consensus_meta.cc:287] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.768s	user 0.016s	sys 0.000s
I20260709 13:00:00.661602  4277 leader_election.cc:290] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:00.662350  4378 consensus_queue.cc:1048] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.087s
I20260709 13:00:00.721917  1599 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 448) took 1108 ms. Trace:
I20260709 13:00:00.684793  4070 raft_consensus.cc:493] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
W20260709 13:00:00.685495  1645 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:00.739364  4144 raft_consensus.cc:493] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:00.639387  4352 raft_consensus.cc:2804] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260709 13:00:00.727715  1570 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:00:00.747290  4263 consensus_meta.cc:287] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.953s	user 0.013s	sys 0.004s
I20260709 13:00:00.756218  1599 rpcz_store.cc:276] 0709 12:59:59.613181 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:59.618967 (+  5786us) service_pool.cc:224] Handling call
0709 12:59:59.664198 (+ 45231us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:59:59.664468 (+   270us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:59:59.664495 (+    27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:00:00.517385 (+852890us) spinlock_profiling.cc:243] Waited 1.49 ms on lock 0x7b2c00000658. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4264a5ba11 00007f4264a6b372 00007f426619da01 00007f426616243a 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2
0709 13:00:00.517903 (+   518us) log.cc:844] Serialized 26 byte log entry
0709 13:00:00.562341 (+ 44438us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:00:00.562409 (+    68us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:00.562560 (+   151us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:00:00.720297 (+157737us) raft_consensus.cc:1685] finished
0709 13:00:00.720344 (+    47us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:00.721316 (+   972us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3431296,"thread_start_us":43510,"threads_started":1,"wal-append.queue_time_us":156486}
W20260709 13:00:00.742802  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 99009.4us
I20260709 13:00:00.741894  3978 raft_consensus.cc:2749] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:00:00.767906  1717 leader_election.cc:304] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 13:00:00.773784  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 141107us
I20260709 13:00:00.752609  4273 raft_consensus.cc:2804] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:00:00.662895  4304 raft_consensus.cc:515] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:00.778241  1717 leader_election.cc:336] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 1.697s (SENT)
I20260709 13:00:00.778556  1643 leader_election.cc:304] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:00:00.758971  4277 raft_consensus.cc:2749] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260709 13:00:00.781054  1717 leader_election.cc:336] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 2.070s (SENT)
I20260709 13:00:00.757772  1746 raft_consensus.cc:2468] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 3.
W20260709 13:00:00.775120  4215 consensus_meta.cc:287] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.632s	user 0.015s	sys 0.003s
I20260709 13:00:00.762311  4138 leader_election.cc:290] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 6 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:00.793418  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:00.794283  1599 raft_consensus.cc:2468] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 3.
I20260709 13:00:00.795728  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:00.796459  1599 raft_consensus.cc:2468] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 4.
I20260709 13:00:00.768906  4352 raft_consensus.cc:493] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:00.755620  4262 raft_consensus.cc:515] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:00.774271  4206 raft_consensus.cc:2749] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 13:00:00.779153  4273 raft_consensus.cc:697] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:00:00.811306  1598 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 1857) took 1147 ms. Trace:
I20260709 13:00:00.803912  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:00.819959  1599 raft_consensus.cc:2393] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 5: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:00:00.812306  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:00.819418  1598 rpcz_store.cc:276] 0709 12:59:59.663776 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:00.334012 (+670236us) service_pool.cc:224] Handling call
0709 13:00:00.334356 (+   344us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:00:00.394207 (+ 59851us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:00:00.394253 (+    46us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:00:00.409542 (+ 15289us) log.cc:844] Serialized 26 byte log entry
0709 13:00:00.589366 (+179824us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:00:00.589417 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:00.589495 (+    78us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:00:00.811009 (+221514us) raft_consensus.cc:1685] finished
0709 13:00:00.811034 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:00.811228 (+   194us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":78406,"spinlock_wait_cycles":33170688,"thread_start_us":189896,"threads_started":2,"wal-append.queue_time_us":305240}
I20260709 13:00:00.764765  4070 raft_consensus.cc:515] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:00.824720  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:00.858043  1598 raft_consensus.cc:2468] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 4.
I20260709 13:00:00.860126  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:00.860769  1598 raft_consensus.cc:3060] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:00:00.860962  4273 consensus_queue.cc:237] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:00.827716  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:00.808307  4352 raft_consensus.cc:3060] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:00:00.833062  4304 leader_election.cc:290] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:00:00.833177  4361 log.cc:927] Time spent T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.477s	user 0.000s	sys 0.001s
I20260709 13:00:00.820995  4138 raft_consensus.cc:2749] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:00:00.870702  4262 leader_election.cc:290] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 6 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:00.837395  4395 raft_consensus.cc:493] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:00.878937  4184 raft_consensus.cc:2749] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:00:00.840794  1752 raft_consensus.cc:1275] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 4. (term mismatch)
I20260709 13:00:00.793409  4272 leader_election.cc:290] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:00:00.843027  1646 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:00.771095  4144 raft_consensus.cc:515] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:00.879459  4395 raft_consensus.cc:515] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:00.889820  4326 leader_election.cc:290] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:00.890386  4144 leader_election.cc:290] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:00.891429  1749 raft_consensus.cc:1275] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260709 13:00:00.891783  1680 raft_consensus.cc:1275] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20260709 13:00:00.845126  2332 raft_consensus.cc:1240] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 3. Current term is 4. Ops: [3.6-3.6]
W20260709 13:00:00.851652  1718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:00:00.854929  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:00.863019  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:00.866801  4310 raft_consensus.cc:3055] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 LEADER]: Stepping down as leader of term 4
W20260709 13:00:00.827903  4370 log.cc:927] Time spent T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.425s	user 0.004s	sys 0.000s
I20260709 13:00:00.871452  4342 raft_consensus.cc:515] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:00.879801  1752 pending_rounds.cc:85] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701: Aborting all ops after (but not including) 3
I20260709 13:00:00.880244  1679 raft_consensus.cc:1240] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 5. Current term is 6. Ops: []
I20260709 13:00:00.883560  1599 raft_consensus.cc:2393] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 5: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:00:00.883514  1598 raft_consensus.cc:2468] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 3.
I20260709 13:00:00.886036  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:00.886458  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:00.905510  1677 raft_consensus.cc:3060] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:00:00.893137  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:00.895673  2332 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49082 (request call id 618) took 1257 ms. Trace:
I20260709 13:00:00.806098  4215 raft_consensus.cc:515] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:00.908017  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:00.909178  1752 pending_rounds.cc:107] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701: Aborting uncommitted NO_OP operation due to leader change: 5.4
I20260709 13:00:00.910023  1752 raft_consensus.cc:2889] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:00:00.875988  4299 raft_consensus.cc:2749] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:00:00.906296  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:00.906322  2332 rpcz_store.cc:276] 0709 12:59:59.637903 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:59.677466 (+ 39563us) service_pool.cc:224] Handling call
0709 13:00:00.844470 (+1167004us) spinlock_profiling.cc:243] Waited 1.23 ms on lock 0x7b5800178598. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f426303db71 00007f426303da22 00007f426bf56c09 00007f426b0b0ee0 00007f426b17a0ab 00007f426b17a131 00007f426b13bfc0 00007f426b126929 00007f426b1263d2 00007f426469dbd4
0709 13:00:00.844737 (+   267us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:00:00.895407 (+ 50670us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:00.895613 (+   206us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2818944}
I20260709 13:00:00.915210  1598 raft_consensus.cc:2468] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 2.
I20260709 13:00:00.911187  1599 raft_consensus.cc:2393] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 7: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:00:00.900915  4126 raft_consensus.cc:493] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:00.901291  4310 raft_consensus.cc:740] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:00:00.916810  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:00.923871  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:00.958635  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:00:01.006827  2298 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:01.020439  1599 raft_consensus.cc:2376] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 4. Re-sending same reply.
I20260709 13:00:01.021303  1598 raft_consensus.cc:2368] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 5. Current term is 6.
I20260709 13:00:01.016830  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:01.021963  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:01.022800  1599 raft_consensus.cc:2393] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 7: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:00:01.024021  1754 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55840 (request call id 2113) took 1315 ms. Trace:
I20260709 13:00:01.022805  1680 raft_consensus.cc:2410] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 for term 5 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260709 13:00:01.024453  1754 rpcz_store.cc:276] 0709 12:59:59.708515 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:59.915248 (+206733us) service_pool.cc:224] Handling call
0709 13:00:00.116797 (+201549us) spinlock_profiling.cc:243] Waited 1.27 ms on lock 0x7b58001ffe98. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f426303db71 00007f426303deee 0000000000375939 00007f426b0b11e0 00007f426b17a0c4 00007f426b17a131 00007f426b13bfc0 00007f426b126929 00007f426b1263d2 00007f426469dbd4
0709 13:00:00.116977 (+   180us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:00:00.141999 (+ 25022us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:00:00.142053 (+    54us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:00:00.142720 (+   667us) log.cc:844] Serialized 26 byte log entry
0709 13:00:00.312657 (+169937us) raft_consensus.cc:1645] Marking committed up to 1
0709 13:00:00.312909 (+   252us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:00.313390 (+   481us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:00:00.826903 (+513513us) spinlock_profiling.cc:243] Waited 2.81 ms on lock 0x7b4400022b4c. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f4262f9bc85 00007f4262f9c6a2 00007f4265d0e1db 00007f4265d09aaf 00007f4265cfd86e 00007f4265cfcbb6 00007f4265cfc7d8 00007f4265d06221 00007f4265d061d9 00007f4265d06169
0709 13:00:01.023716 (+196813us) raft_consensus.cc:1685] finished
0709 13:00:01.023745 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:01.023939 (+   194us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":9415808,"thread_start_us":114207,"threads_started":1,"wal-append.queue_time_us":179057}
W20260709 13:00:01.026304  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 249297us
I20260709 13:00:01.027098  1677 raft_consensus.cc:2468] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 7.
I20260709 13:00:00.913573  1746 raft_consensus.cc:2393] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 3: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:00:00.938408  4126 raft_consensus.cc:515] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:01.028656  4395 leader_election.cc:290] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:01.028889  4070 leader_election.cc:290] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:00.983124  3948 raft_consensus.cc:515] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:01.032150  4396 log.cc:927] Time spent T 00000000000000000000000000000000 P 919f6392d4f44fd68987e2c5e7ec37ca: Append to log took a long time: real 0.139s	user 0.000s	sys 0.001s
I20260709 13:00:01.032910  3948 leader_election.cc:290] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:01.005622  4310 consensus_queue.cc:260] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:01.035660  4310 raft_consensus.cc:3060] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:00:01.027881  1643 leader_election.cc:304] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:00:01.024725  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 6 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:01.041710  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:01.042400  1599 raft_consensus.cc:2376] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 6. Re-sending same reply.
I20260709 13:00:01.043762  1643 leader_election.cc:304] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:00:01.044698  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:01.045281  1599 raft_consensus.cc:3060] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:00:01.046877  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:01.042956  4410 raft_consensus.cc:2749] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:00:00.987691  3977 raft_consensus.cc:2749] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:00:01.051287  4406 consensus_queue.cc:1048] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 13:00:00.982857  4215 leader_election.cc:290] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 7 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:00.984706  4319 ts_tablet_manager.cc:1929] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 4.5
I20260709 13:00:01.056450  1599 raft_consensus.cc:2468] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 5.
I20260709 13:00:01.056910  1747 raft_consensus.cc:2468] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 5.
I20260709 13:00:00.982197  2328 raft_consensus.cc:2468] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 6.
I20260709 13:00:01.059468  2337 raft_consensus.cc:1240] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 5. Current term is 6. Ops: [5.6-5.6]
I20260709 13:00:01.060248  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:01.060838  1680 raft_consensus.cc:3060] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:00:01.060892  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:01.061779  1671 raft_consensus.cc:2410] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260709 13:00:01.062850  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:01.057510  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:01.049746  4249 raft_consensus.cc:2749] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:00:01.057874  1677 raft_consensus.cc:1275] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260709 13:00:01.049559  3948 raft_consensus.cc:493] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:01.058683  4308 consensus_queue.cc:1048] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.019s
I20260709 13:00:01.070483  3948 raft_consensus.cc:515] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:01.071847  3977 consensus_queue.cc:1048] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 13:00:01.079131  3948 leader_election.cc:290] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:01.065732  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
W20260709 13:00:01.081533  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 457168us
I20260709 13:00:01.082988  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:01.058642  4410 raft_consensus.cc:493] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:01.085635  1601 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 1856) took 1434 ms. Trace:
I20260709 13:00:01.087167  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:00.957719  4383 consensus_queue.cc:1048] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.112s
I20260709 13:00:01.086206  1601 rpcz_store.cc:276] 0709 12:59:59.651534 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:00.326336 (+674802us) service_pool.cc:224] Handling call
0709 13:00:00.338130 (+ 11794us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:00:00.396129 (+ 57999us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:00:00.396164 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:00:00.404702 (+  8538us) log.cc:844] Serialized 26 byte log entry
0709 13:00:00.629097 (+224395us) raft_consensus.cc:1645] Marking committed up to 3
0709 13:00:00.648049 (+ 18952us) log.cc:844] Serialized 14 byte log entry
0709 13:00:00.900095 (+252046us) spinlock_profiling.cc:243] Waited 8.27 ms on lock 0x7b4800003cd8. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426b08baa1 00007f426b082b37 00007f426b0829ca 00007f426b17a26f 00007f426b179f37 00007f426b18e55c 00007f426b18e508 00007f426b18e488 00007f426b18e443 00007f426b18d6ec 00007f426a0b7a96
0709 13:00:01.085039 (+184944us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:01.085146 (+   107us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:00:01.085193 (+    47us) raft_consensus.cc:1685] finished
0709 13:00:01.085368 (+   175us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:01.085556 (+   188us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":443534,"prepare.queue_time_us":22354,"prepare.run_cpu_time_us":63,"prepare.run_wall_time_us":64,"raft.queue_time_us":538303,"raft.run_cpu_time_us":3504,"raft.run_wall_time_us":456741,"spinlock_wait_cycles":19030784,"thread_start_us":234173,"threads_started":2,"wal-append.queue_time_us":231300}
I20260709 13:00:01.086206  4410 raft_consensus.cc:515] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:01.083262  1599 raft_consensus.cc:2410] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for term 4 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260709 13:00:01.092490  1605 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 1855) took 1441 ms. Trace:
I20260709 13:00:01.095314  1643 leader_election.cc:304] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:00:01.098482  4378 raft_consensus.cc:493] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:01.098562  4272 raft_consensus.cc:493] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:01.105507  4378 raft_consensus.cc:515] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:01.108681  4378 leader_election.cc:290] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:01.105911  4272 raft_consensus.cc:515] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:01.109591  4378 consensus_queue.cc:1048] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:00:01.111828  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:01.112528  1600 raft_consensus.cc:2376] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 3. Re-sending same reply.
I20260709 13:00:01.112982  2301 leader_election.cc:304] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:00:01.113397  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:01.114066  4412 raft_consensus.cc:2804] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 13:00:01.114687  4249 raft_consensus.cc:2804] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:00:01.115337  4249 raft_consensus.cc:493] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:01.096477  4285 raft_consensus.cc:493] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:01.100611  4368 raft_consensus.cc:493] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:01.121493  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:01.104411  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:01.104640  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:00:01.108793  2333 raft_consensus.cc:1240] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20260709 13:00:01.096359  1605 rpcz_store.cc:276] 0709 12:59:59.651239 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:00.319962 (+668723us) service_pool.cc:224] Handling call
0709 13:00:00.320350 (+   388us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:00:00.380974 (+ 60624us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:00:00.381012 (+    38us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:00:00.381800 (+   788us) log.cc:844] Serialized 52 byte log entry
0709 13:00:00.536911 (+155111us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:00:00.537725 (+   814us) log.cc:844] Serialized 14 byte log entry
0709 13:00:00.614497 (+ 76772us) log.cc:844] Serialized 14 byte log entry
0709 13:00:01.080685 (+466188us) spinlock_profiling.cc:243] Waited 5.15 ms on lock 0x7b18004702c0. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f4262fefc0a 00007f426c72f219 00007f4263077a40 00007f42630724e4 00007f4263072455 00007f426615a060 00007f426616c716 00007f426617d287 00007f426617d208
0709 13:00:01.080963 (+   278us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:01.081106 (+   143us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:00:01.092250 (+ 11144us) raft_consensus.cc:1685] finished
0709 13:00:01.092277 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:01.092426 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":67592,"prepare.queue_time_us":6781,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":51,"raft.queue_time_us":640465,"raft.run_cpu_time_us":389,"raft.run_wall_time_us":27135,"spinlock_wait_cycles":37204736,"thread_start_us":205318,"threads_started":2,"wal-append.queue_time_us":231805}
I20260709 13:00:01.083216  4319 log.cc:1199] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1: Deleting WAL directory at /tmp/dist-test-taskakPCWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783601878574137-431-0/minicluster-data/ts-3-root/wals/983621a6fc5b4c9ea0159a648dfe7a36
I20260709 13:00:01.118508  1571 leader_election.cc:304] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:00:01.120018  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:01.120754  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:01.128878  1598 raft_consensus.cc:2468] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 5.
I20260709 13:00:01.115993  4249 raft_consensus.cc:3060] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:00:01.101321  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:01.122426  2332 raft_consensus.cc:1803] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 4 LEARNER]: voting while tombstoned based on last-logged opid 4.5
I20260709 13:00:01.133816  1601 raft_consensus.cc:3060] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:00:01.108681  1569 leader_election.cc:304] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:00:01.135833  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:01.114142  1600 raft_consensus.cc:2468] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 8.
I20260709 13:00:01.126662  1680 raft_consensus.cc:2468] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 3.
I20260709 13:00:01.136547  1605 raft_consensus.cc:2393] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 3: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:00:01.122373  1750 raft_consensus.cc:2468] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 5.
I20260709 13:00:01.133921  2332 raft_consensus.cc:2468] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 4 LEARNER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 4.
I20260709 13:00:01.139055  1718 leader_election.cc:304] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:00:01.130561  1571 leader_election.cc:304] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:00:01.123847  1644 leader_election.cc:304] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:01.140318  4376 raft_consensus.cc:2804] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:00:01.137135  4070 consensus_queue.cc:1048] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.004s
I20260709 13:00:01.140918  4376 raft_consensus.cc:493] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:01.121982  4201 consensus_queue.cc:1048] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.020s
W20260709 13:00:01.143700  1717 leader_election.cc:336] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 2.025s (SENT)
I20260709 13:00:01.144240  1717 leader_election.cc:304] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:01.145349  4206 raft_consensus.cc:2749] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 13:00:01.146384  4262 raft_consensus.cc:2804] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:00:01.146978  4262 raft_consensus.cc:697] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 3 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:00:01.148319  4201 consensus_queue.cc:1048] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 13:00:01.149675  4326 raft_consensus.cc:2749] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:00:01.147964  4262 consensus_queue.cc:237] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:01.119917  3948 raft_consensus.cc:493] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:00:01.155283  1645 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:01.141456  4376 raft_consensus.cc:3060] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Advancing to term 6
W20260709 13:00:01.145376  1717 leader_election.cc:336] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 2.138s (SENT)
I20260709 13:00:01.155974  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:00:01.157917  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:01.160178  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:01.164852  2332 raft_consensus.cc:2393] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 7: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:00:01.165112  1680 raft_consensus.cc:2393] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 3: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
W20260709 13:00:01.164893  1717 leader_election.cc:336] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 2.247s (SENT)
I20260709 13:00:01.147998  2328 raft_consensus.cc:1240] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 6. Current term is 7. Ops: []
I20260709 13:00:01.131321  4355 raft_consensus.cc:2749] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:00:01.165701  1717 leader_election.cc:304] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:01.168684  1571 leader_election.cc:304] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:00:01.157357  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:01.170014  1680 raft_consensus.cc:1275] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260709 13:00:01.129614  4412 raft_consensus.cc:697] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 6 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:00:01.175997  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:01.176930  1671 raft_consensus.cc:2393] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 7: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:00:01.175140  1605 raft_consensus.cc:2393] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 4: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:00:01.179432  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:01.129277  4383 consensus_queue.cc:1048] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.041s
I20260709 13:00:01.182369  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:01.183176  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:01.127764  4285 raft_consensus.cc:515] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:01.185109  1680 raft_consensus.cc:2410] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 13:00:01.158221  4388 raft_consensus.cc:2749] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:00:01.175161  1571 leader_election.cc:304] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 13:00:01.129452  4368 raft_consensus.cc:515] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:01.147997  4070 consensus_queue.cc:1059] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.033s
I20260709 13:00:01.192258  1605 raft_consensus.cc:2468] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 6.
I20260709 13:00:01.177482  4410 leader_election.cc:290] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:01.172724  4372 raft_consensus.cc:2749] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:00:01.196033  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:01.193779  1571 leader_election.cc:304] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:00:01.177680  3978 raft_consensus.cc:2749] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:00:01.207885  1571 leader_election.cc:304] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:00:01.166663  3948 raft_consensus.cc:515] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:01.218407  1599 raft_consensus.cc:2410] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 for term 5 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260709 13:00:01.191756  4215 raft_consensus.cc:2804] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:00:01.218308  1717 leader_election.cc:304] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
W20260709 13:00:01.231331  4409 log.cc:927] Time spent T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.109s	user 0.000s	sys 0.001s
I20260709 13:00:01.231002  4372 consensus_queue.cc:1059] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 13:00:01.231779  1597 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 446) took 1620 ms. Trace:
I20260709 13:00:01.232218  1597 rpcz_store.cc:276] 0709 12:59:59.611531 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:59.611701 (+   170us) service_pool.cc:224] Handling call
0709 12:59:59.611955 (+   254us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:00.057612 (+445657us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:01.231496 (+1173884us) spinlock_profiling.cc:243] Waited 437 ms on lock 0x7b5400216bd4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:00:01.231705 (+   209us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1004282112}
I20260709 13:00:01.233001  4372 raft_consensus.cc:3055] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:00:01.233632  4372 raft_consensus.cc:740] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:00:01.233870  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:01.234061  4319 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54234 (request call id 0) took 2478 ms. Trace:
W20260709 13:00:01.299929  1643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:01.221241  4215 raft_consensus.cc:493] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:01.224900  4246 consensus_queue.cc:260] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:01.219175  4355 raft_consensus.cc:2749] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:00:01.176523  4412 consensus_queue.cc:237] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:01.218420  4388 raft_consensus.cc:2749] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 13:00:01.300670  1597 raft_consensus.cc:2468] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 4.
W20260709 13:00:01.299753  1718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:00:01.305090  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 141278us
W20260709 13:00:01.299804  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 72674.3us
W20260709 13:00:01.314020  1569 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:00:01.309139  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 90911.1us
I20260709 13:00:01.314986  3978 raft_consensus.cc:2804] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:00:01.315737  3978 raft_consensus.cc:493] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:00:01.316300  3978 raft_consensus.cc:3060] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:00:01.314890  4390 consensus_queue.cc:1059] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.007s
I20260709 13:00:01.323038  1597 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5"
dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
mode: GRACEFUL
new_leader_uuid: "af520b67945d443b901bbeca37cc5701"
 from {username='slave'} at 127.0.0.1:47752
I20260709 13:00:01.326148  1597 raft_consensus.cc:606] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 LEADER]: Received request to transfer leadership to TS af520b67945d443b901bbeca37cc5701
I20260709 13:00:01.323688  1601 raft_consensus.cc:2468] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 7.
I20260709 13:00:01.323743  2298 leader_election.cc:304] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:00:01.326525  1717 leader_election.cc:304] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:00:01.307817  4215 raft_consensus.cc:3060] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:00:01.334089  2298 leader_election.cc:304] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:00:01.327998  1571 leader_election.cc:304] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:00:01.335541  1717 leader_election.cc:304] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:01.339069  1597 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 1859) took 1665 ms. Trace:
I20260709 13:00:01.339794  1597 rpcz_store.cc:276] 0709 12:59:59.673584 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:01.334355 (+1660771us) service_pool.cc:224] Handling call
0709 13:00:01.338280 (+  3925us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:01.338565 (+   285us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:00:01.338604 (+    39us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:00:01.338692 (+    88us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:00:01.338760 (+    68us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:01.338840 (+    80us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:01.338978 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:00:01.344597  1597 raft_consensus.cc:1240] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260709 13:00:01.346246  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:01.347379  1751 raft_consensus.cc:2410] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 13:00:01.350698  4395 raft_consensus.cc:3060] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:00:01.352973  1717 leader_election.cc:304] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:01.353607  1751 raft_consensus.cc:1275] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260709 13:00:01.318912  4378 raft_consensus.cc:493] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:00:01.362118  1601 raft_consensus.cc:1240] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20260709 13:00:01.362987  1601 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 1858) took 1695 ms. Trace:
I20260709 13:00:01.357451  1597 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 1864) took 1672 ms. Trace:
I20260709 13:00:01.363595  1717 leader_election.cc:304] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:00:01.363332  1601 rpcz_store.cc:276] 0709 12:59:59.667787 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:01.332559 (+1664772us) service_pool.cc:224] Handling call
0709 13:00:01.361835 (+ 29276us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:00:01.362619 (+   784us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:01.362921 (+   302us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:00:01.358892  1643 leader_election.cc:304] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:01.363761  1597 rpcz_store.cc:276] 0709 12:59:59.685271 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:01.343772 (+1658501us) service_pool.cc:224] Handling call
0709 13:00:01.344286 (+   514us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:00:01.356056 (+ 11770us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:01.356950 (+   894us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:00:01.366632  4285 leader_election.cc:290] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:01.368070  4307 raft_consensus.cc:2804] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:00:01.368813  4307 raft_consensus.cc:493] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:01.369364  4307 raft_consensus.cc:3060] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:00:01.369248  4378 raft_consensus.cc:515] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:01.373250  4378 leader_election.cc:290] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:01.340394  4383 raft_consensus.cc:2804] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:00:01.366696  4150 raft_consensus.cc:2804] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:00:01.335830  4262 raft_consensus.cc:2804] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:00:01.340953  4391 raft_consensus.cc:2804] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:00:01.320359  4352 raft_consensus.cc:515] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:01.377197  4333 raft_consensus.cc:3060] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:00:01.391269  4391 raft_consensus.cc:697] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 7 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:00:01.384083  4206 raft_consensus.cc:2749] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:00:01.361500  4263 raft_consensus.cc:493] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:01.385910  4262 raft_consensus.cc:493] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
W20260709 13:00:01.398870  1602 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40166 (request call id 1835) took 1873 ms (client timeout 1589 ms). Trace:
I20260709 13:00:01.414633  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:01.418299  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:01.418987  1750 raft_consensus.cc:2468] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 4.
W20260709 13:00:01.417948  1602 rpcz_store.cc:269] 0709 12:59:59.525079 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:59.525259 (+   180us) service_pool.cc:224] Handling call
0709 13:00:01.398559 (+1873300us) spinlock_profiling.cc:243] Waited 69.8 ms on lock 0x7b54002148d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:00:01.398743 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":160602496}
I20260709 13:00:01.365091  1717 leader_election.cc:304] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:00:01.384485  4383 raft_consensus.cc:493] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:01.384745  4150 raft_consensus.cc:493] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:00:01.421092  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 56106.7us
I20260709 13:00:01.423887  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:01.425418  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:01.404506  4265 raft_consensus.cc:493] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:00:01.426332  1602 raft_consensus.cc:2393] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 5: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:00:01.433460  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:01.433549  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 8 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:01.431329  1571 leader_election.cc:304] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:00:01.432389  4378 raft_consensus.cc:493] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:01.438090  4355 raft_consensus.cc:2804] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:00:01.438844  4355 raft_consensus.cc:493] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:01.438527  4378 raft_consensus.cc:515] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:01.439481  4355 raft_consensus.cc:3060] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:00:01.419648  4262 raft_consensus.cc:3060] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:00:01.449784  1602 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 1867) took 1763 ms. Trace:
I20260709 13:00:01.427750  4150 raft_consensus.cc:3060] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:00:01.449467  1569 leader_election.cc:304] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:00:01.426740  4383 raft_consensus.cc:3060] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:00:01.422977  4406 raft_consensus.cc:493] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:01.456698  1602 rpcz_store.cc:276] 0709 12:59:59.686232 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:01.448637 (+1762405us) service_pool.cc:224] Handling call
0709 13:00:01.449051 (+   414us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:01.449305 (+   254us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:00:01.449342 (+    37us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:00:01.449402 (+    60us) raft_consensus.cc:1645] Marking committed up to 3
0709 13:00:01.449475 (+    73us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:01.449553 (+    78us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:01.449707 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:00:01.464672  1602 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 455) took 1722 ms. Trace:
I20260709 13:00:01.465122  1602 rpcz_store.cc:276] 0709 12:59:59.742116 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:01.463617 (+1721501us) service_pool.cc:224] Handling call
0709 13:00:01.464028 (+   411us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:01.464285 (+   257us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:00:01.464310 (+    25us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:00:01.464356 (+    46us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:00:01.464397 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:01.464452 (+    55us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:01.464595 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:00:01.418762  4263 raft_consensus.cc:515] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:01.300290  4319 rpcz_store.cc:276] 0709 12:59:58.755883 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:58.756200 (+   317us) service_pool.cc:224] Handling call
0709 12:59:59.048472 (+292272us) ts_tablet_manager.cc:1151] Deleting tablet 983621a6fc5b4c9ea0159a648dfe7a36
0709 12:59:59.061686 (+ 13214us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0709 13:00:00.661298 (+1599612us) tablet_metadata.cc:665] Metadata flushed
0709 13:00:00.984423 (+323125us) tablet_metadata.cc:665] Metadata flushed
0709 13:00:01.233938 (+249515us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":393606,"tablet-delete.queue_time_us":210042,"thread_start_us":96763,"threads_started":1}
I20260709 13:00:01.447793  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:01.457017  4206 raft_consensus.cc:2804] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:00:01.442783  4285 consensus_queue.cc:1048] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.032s
W20260709 13:00:01.479979  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 48778.2us
I20260709 13:00:01.473661  4378 leader_election.cc:290] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:01.479625  1569 leader_election.cc:304] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
W20260709 13:00:01.530336  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 50793.7us
I20260709 13:00:01.443832  4265 raft_consensus.cc:515] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:01.528918  3948 leader_election.cc:290] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:00:01.500622  4366 log.cc:927] Time spent T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 1.110s	user 0.000s	sys 0.006s
I20260709 13:00:01.534277  1754 raft_consensus.cc:1240] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 6. Current term is 7. Ops: []
I20260709 13:00:01.475361  4301 raft_consensus.cc:2749] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:00:01.498734  4206 raft_consensus.cc:493] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:01.536965  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:00:01.535336  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 175082us
W20260709 13:00:01.545521  2298 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:01.502488  4263 leader_election.cc:290] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:01.541903  1676 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52778 (request call id 1748) took 1736 ms. Trace:
I20260709 13:00:01.543280  2332 raft_consensus.cc:1275] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260709 13:00:01.544173  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:01.551513  1676 rpcz_store.cc:276] 0709 12:59:59.805473 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:59.805610 (+   137us) service_pool.cc:224] Handling call
0709 12:59:59.805907 (+   297us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:00.050300 (+244393us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:01.541604 (+1491304us) spinlock_profiling.cc:243] Waited 232 ms on lock 0x7b54001f5c54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:00:01.541809 (+   205us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":534706944}
I20260709 13:00:01.499208  2328 raft_consensus.cc:1275] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260709 13:00:01.544411  1749 raft_consensus.cc:2468] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 6.
I20260709 13:00:01.554320  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:01.556326  1717 leader_election.cc:304] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:01.557329  2332 raft_consensus.cc:2393] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 7: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:00:01.527606  4166 consensus_queue.cc:1048] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.052s
I20260709 13:00:01.474052  4406 raft_consensus.cc:515] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:01.558111  1488 catalog_manager.cc:5028] TS 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625): tablet 983621a6fc5b4c9ea0159a648dfe7a36 (table test-workload [id=75d77b57b3f34786b9c215c4f11bd727]) successfully deleted
I20260709 13:00:01.560788  4406 leader_election.cc:290] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:01.564347  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:00:01.537292  4283 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.117s	user 0.007s	sys 0.014s
I20260709 13:00:01.537135  4378 raft_consensus.cc:2749] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260709 13:00:01.572384  4283 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.152s	user 0.008s	sys 0.015s
I20260709 13:00:01.572122  4422 raft_consensus.cc:493] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:01.554611  3948 raft_consensus.cc:493] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:01.548379  4206 raft_consensus.cc:3060] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:00:01.602348  3521 raft_consensus.cc:2749] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:00:01.605443  1644 leader_election.cc:304] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:01.606705  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:01.607040  4352 leader_election.cc:290] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 5 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:01.608035  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:01.609089  1605 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "65285571995c4ed5b593041598056a22"
dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
mode: GRACEFUL
new_leader_uuid: "af520b67945d443b901bbeca37cc5701"
 from {username='slave'} at 127.0.0.1:47752
I20260709 13:00:01.611081  1569 leader_election.cc:304] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:00:01.612037  4365 raft_consensus.cc:3055] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:00:01.612581  4365 raft_consensus.cc:740] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:00:01.613075  1569 leader_election.cc:304] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:00:01.614776  4301 raft_consensus.cc:2749] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:00:01.578783  4249 raft_consensus.cc:515] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:01.616551  4301 raft_consensus.cc:2749] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 13:00:01.617537  1598 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 457) took 1826 ms. Trace:
I20260709 13:00:01.618600  2336 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39726 (request call id 745) took 2907 ms. Trace:
I20260709 13:00:01.619498  1752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55840 (request call id 2119) took 1143 ms. Trace:
I20260709 13:00:01.619980  1752 rpcz_store.cc:276] 0709 13:00:00.475800 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:00.624029 (+148229us) service_pool.cc:224] Handling call
0709 13:00:00.624444 (+   415us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:00.910589 (+286145us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:01.619244 (+708655us) spinlock_profiling.cc:243] Waited 216 ms on lock 0x7b540035ff54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:00:01.619430 (+   186us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":496440576}
I20260709 13:00:01.619000  2336 rpcz_store.cc:276] 0709 12:59:58.711657 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:58.711788 (+   131us) service_pool.cc:224] Handling call
0709 12:59:59.440392 (+728604us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:59:59.440733 (+   341us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:59:59.627538 (+186805us) log.cc:844] Serialized 14 byte log entry
0709 13:00:00.125598 (+498060us) spinlock_profiling.cc:243] Waited 6.63 ms on lock 0x7b4400022b4c. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f4262f9bc85 00007f4262f9c6a2 00007f4265d0e1db 00007f4265d09aaf 00007f4265cfd86e 00007f4265cfcbb6 00007f4265cfc7d8 00007f4265d06221 00007f4265d061d9 00007f4265d06169
0709 13:00:00.492292 (+366694us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:00:00.492376 (+    84us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:00:00.492457 (+    81us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:00.492559 (+   102us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:01.618368 (+1125809us) spinlock_profiling.cc:243] Waited 673 ms on lock 0x7b540065e0d0. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661618ad 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:00:01.618531 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":90815,"prepare.queue_time_us":186524,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":41,"raft.queue_time_us":89397,"raft.run_cpu_time_us":210,"raft.run_wall_time_us":27606,"spinlock_wait_cycles":1563223296,"thread_start_us":887367,"threads_started":3,"wal-append.queue_time_us":415125,"wal-append.run_cpu_time_us":15059,"wal-append.run_wall_time_us":1260817}
I20260709 13:00:01.622721  1569 leader_election.cc:304] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:00:01.624533  4378 raft_consensus.cc:2749] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 13:00:01.628057  1597 raft_consensus.cc:1275] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260709 13:00:01.628656  1597 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 459) took 1792 ms. Trace:
I20260709 13:00:01.628943  1597 rpcz_store.cc:276] 0709 12:59:59.836111 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:01.627434 (+1791323us) service_pool.cc:224] Handling call
0709 13:00:01.627718 (+   284us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:01.628414 (+   696us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:01.628598 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:00:01.630352  4427 consensus_queue.cc:1059] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.034s
I20260709 13:00:01.631793  4272 leader_election.cc:290] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:01.632864  4428 raft_consensus.cc:3055] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:00:01.633574  4428 raft_consensus.cc:740] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:00:01.635063  4428 consensus_queue.cc:260] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:01.636585  4364 log.cc:927] Time spent T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 1.248s	user 0.000s	sys 0.003s
I20260709 13:00:01.617911  1598 rpcz_store.cc:276] 0709 12:59:59.791310 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:01.616484 (+1825174us) service_pool.cc:224] Handling call
0709 13:00:01.616867 (+   383us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:01.617168 (+   301us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:00:01.617203 (+    35us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:00:01.617258 (+    55us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:00:01.617299 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:01.617364 (+    65us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:01.617475 (+   111us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:00:01.630261  1597 raft_consensus.cc:1275] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 13:00:01.643031  1644 leader_election.cc:304] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:00:01.643939  1597 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 461) took 1751 ms. Trace:
I20260709 13:00:01.644460  1597 rpcz_store.cc:276] 0709 12:59:59.892255 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:01.629657 (+1737402us) service_pool.cc:224] Handling call
0709 13:00:01.629958 (+   301us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:01.643604 (+ 13646us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:01.643865 (+   261us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:00:01.489275  4391 consensus_queue.cc:237] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:01.640548  2298 leader_election.cc:304] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 13:00:01.593261  4310 consensus_meta.cc:287] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.557s	user 0.014s	sys 0.006s
I20260709 13:00:01.580245  4126 leader_election.cc:290] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:01.655534  1597 raft_consensus.cc:1240] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 6. Current term is 7. Ops: []
I20260709 13:00:01.656742  1600 raft_consensus.cc:2410] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for term 5 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260709 13:00:01.657517  1644 leader_election.cc:304] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:01.660130  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:00:01.660771  1597 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 464) took 1478 ms. Trace:
I20260709 13:00:01.661262  1753 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43156 (request call id 474) took 2021 ms. Trace:
I20260709 13:00:01.661964  1753 rpcz_store.cc:276] 0709 12:59:59.639748 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:59.896040 (+256292us) service_pool.cc:224] Handling call
0709 12:59:59.912768 (+ 16728us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:59:59.913209 (+   441us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:59:59.913238 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:59:59.943103 (+ 29865us) log.cc:844] Serialized 26 byte log entry
0709 13:00:00.208213 (+265110us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:00:00.208279 (+    66us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:00.208381 (+   102us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:00:01.635563 (+1427182us) spinlock_profiling.cc:243] Waited 14.2 ms on lock 0x7b4400022b4c. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f4262f9bc85 00007f4262f9c6a2 00007f4265d0e1db 00007f4265d09aaf 00007f4265cfd86e 00007f4265cfcbb6 00007f4265cfc7d8 00007f4265d06221 00007f4265d061d9 00007f4265d06169
0709 13:00:01.640514 (+  4951us) raft_consensus.cc:1685] finished
0709 13:00:01.640541 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:01.661173 (+ 20632us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":556,"raft.run_cpu_time_us":230,"raft.run_wall_time_us":69989,"spinlock_wait_cycles":32744320,"thread_start_us":146269,"threads_started":1,"wal-append.queue_time_us":306837}
I20260709 13:00:01.663738  1605 raft_consensus.cc:606] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Received request to transfer leadership to TS af520b67945d443b901bbeca37cc5701
I20260709 13:00:01.664191  1605 raft_consensus.cc:614] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 13:00:01.661259  1597 rpcz_store.cc:276] 0709 13:00:00.182561 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:01.652893 (+1470332us) service_pool.cc:224] Handling call
0709 13:00:01.653301 (+   408us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:01.660417 (+  7116us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:01.660698 (+   281us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:00:01.667600  1643 leader_election.cc:304] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:00:01.637308  4422 raft_consensus.cc:515] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:01.655614  4070 raft_consensus.cc:2749] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 13:00:01.668334  4133 consensus_queue.cc:1048] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.011s
I20260709 13:00:01.678649  4374 raft_consensus.cc:493] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:01.652974  4410 raft_consensus.cc:2749] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20260709 13:00:01.699035  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 145806us
I20260709 13:00:01.703133  1680 raft_consensus.cc:1275] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 13:00:01.703357  1674 raft_consensus.cc:1275] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260709 13:00:01.704317  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:00:01.706674  1597 pending_rounds.cc:85] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a: Aborting all ops after (but not including) 1
I20260709 13:00:01.707103  1597 pending_rounds.cc:107] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20260709 13:00:01.707415  1597 raft_consensus.cc:2889] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:00:01.665990  4184 raft_consensus.cc:2804] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Leader election won for term 6
I20260709 13:00:01.613936  4365 consensus_queue.cc:260] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:01.714084  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:01.715060  4352 consensus_queue.cc:1059] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 13:00:01.691740  4390 consensus_queue.cc:1059] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.035s
I20260709 13:00:01.672698  4326 raft_consensus.cc:2749] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 13:00:01.732524  4215 raft_consensus.cc:515] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:01.705602  4299 raft_consensus.cc:493] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:01.739974  4265 leader_election.cc:290] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:01.748745  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:00:01.671650  4412 consensus_queue.cc:1048] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.083s
W20260709 13:00:01.783344  1675 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52766 (request call id 2134) took 2747 ms (client timeout 1681 ms). Trace:
I20260709 13:00:01.784343  2328 raft_consensus.cc:3060] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Advancing to term 6
W20260709 13:00:01.784499  1675 rpcz_store.cc:269] 0709 12:59:59.036115 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:59.266887 (+230772us) service_pool.cc:224] Handling call
0709 13:00:01.783061 (+2516174us) spinlock_profiling.cc:243] Waited 154 ms on lock 0x7b5400163d54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:00:01.783245 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":353612672}
I20260709 13:00:01.642324  4376 raft_consensus.cc:515] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:01.787243  1678 raft_consensus.cc:1240] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 6. Current term is 7. Ops: []
I20260709 13:00:01.788177  4376 leader_election.cc:290] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 6 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:01.692896  4406 raft_consensus.cc:2804] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:00:01.656010  4301 consensus_queue.cc:1048] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.049s
I20260709 13:00:01.648907  3948 raft_consensus.cc:515] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:01.779484  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:01.783267  4244 consensus_queue.cc:1048] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:00:01.783476  4283 consensus_queue.cc:1048] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:00:01.679419  4374 raft_consensus.cc:515] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:01.719424  4249 leader_election.cc:290] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 3 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:01.810418  1602 raft_consensus.cc:1240] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 5. Current term is 6. Ops: []
W20260709 13:00:01.812188  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 126714us
W20260709 13:00:01.818629  1718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:01.812589  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:01.819859  1603 raft_consensus.cc:3060] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:00:01.824136  4307 raft_consensus.cc:515] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:01.789927  4406 raft_consensus.cc:493] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:01.806680  4326 raft_consensus.cc:3055] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:00:01.845901  1603 raft_consensus.cc:2468] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 5.
W20260709 13:00:01.812649  4359 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.428s	user 0.024s	sys 0.038s
I20260709 13:00:01.847580  1643 leader_election.cc:304] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 13:00:01.763955  3978 raft_consensus.cc:515] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:01.836431  4406 raft_consensus.cc:3060] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:00:01.857584  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:01.859609  1717 leader_election.cc:304] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:01.860174  3948 leader_election.cc:290] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:01.861038  3948 raft_consensus.cc:2749] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:00:01.839749  4265 raft_consensus.cc:493] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:00:01.795008  4299 raft_consensus.cc:515] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:00:01.848596  4359 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.473s	user 0.024s	sys 0.038s
I20260709 13:00:01.848308  4326 raft_consensus.cc:740] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:00:01.900702  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:01.862741  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:00:01.837842  4249 consensus_queue.cc:1048] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.039s
I20260709 13:00:01.907287  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:01.914381  1603 raft_consensus.cc:2468] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 5.
I20260709 13:00:01.915652  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:01.916450  1603 raft_consensus.cc:2468] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 6.
I20260709 13:00:01.917460  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:01.919597  1643 leader_election.cc:304] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:00:01.920989  1717 leader_election.cc:304] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:00:01.924521  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:01.871740  4283 raft_consensus.cc:493] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:00:01.838740  4355 raft_consensus.cc:515] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:01.900399  4263 raft_consensus.cc:493] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:01.937764  4436 consensus_queue.cc:1059] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 13:00:01.939087  4422 leader_election.cc:290] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:01.865409  4265 raft_consensus.cc:515] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:01.941946  4249 raft_consensus.cc:2804] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:00:01.942670  4299 leader_election.cc:290] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:01.943276  1745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55840 (request call id 2104) took 3571 ms. Trace:
I20260709 13:00:01.943663  1745 rpcz_store.cc:276] 0709 12:59:58.371551 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:58.694170 (+322619us) service_pool.cc:224] Handling call
0709 12:59:58.694518 (+   348us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:00.322593 (+1628075us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:01.942984 (+1620391us) spinlock_profiling.cc:243] Waited 1.63 s on lock 0x7b54001c6954. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:00:01.943179 (+   195us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3739317120}
I20260709 13:00:01.934018  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:01.945111  1603 raft_consensus.cc:2376] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 5. Re-sending same reply.
I20260709 13:00:01.946082  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:01.903278  3948 raft_consensus.cc:493] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:00:01.900086  4307 leader_election.cc:290] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 6 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:01.947305  1717 leader_election.cc:304] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 13:00:01.906173  4199 raft_consensus.cc:493] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:00:01.903855  4184 raft_consensus.cc:697] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:00:01.914999  4433 raft_consensus.cc:493] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:00:01.958537  1598 raft_consensus.cc:1240] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 4. Current term is 5. Ops: []
W20260709 13:00:01.955637  1646 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:01.956017  4199 raft_consensus.cc:515] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:01.927377  4244 raft_consensus.cc:2804] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260709 13:00:01.959287  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 126288us
I20260709 13:00:01.932720  4383 raft_consensus.cc:515] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:01.971807  2299 leader_election.cc:304] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 13:00:01.974269  2298 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:01.947973  4249 raft_consensus.cc:697] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:00:01.968287  3978 leader_election.cc:290] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 4 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:01.987483  4438 raft_consensus.cc:2804] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:00:01.987993  4438 raft_consensus.cc:493] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:01.960541  4200 raft_consensus.cc:2804] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:00:01.967870  4244 raft_consensus.cc:493] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:01.955217  1717 leader_election.cc:304] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:01.988343  4438 raft_consensus.cc:3060] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:00:01.948133  4283 raft_consensus.cc:515] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:01.979801  3793 raft_consensus.cc:3055] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:00:01.951906  4263 raft_consensus.cc:515] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:01.953514  3948 raft_consensus.cc:515] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:02.004375  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 49268us
I20260709 13:00:01.999924  4244 raft_consensus.cc:3060] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:00:01.997058  4215 leader_election.cc:290] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 6 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:02.010958  4199 leader_election.cc:290] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:02.003594  3793 raft_consensus.cc:740] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:00:01.969151  4433 raft_consensus.cc:515] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:02.006307  4200 raft_consensus.cc:697] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 5 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:00:02.013193  4298 raft_consensus.cc:2804] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:00:02.011592  3978 raft_consensus.cc:3060] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:00:02.013343  4374 leader_election.cc:290] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:02.035856  4433 leader_election.cc:290] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:00:02.037353  4357 log.cc:927] Time spent T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 1.990s	user 0.004s	sys 0.000s
I20260709 13:00:02.038259  4342 leader_election.cc:290] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 7 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:02.029690  4443 raft_consensus.cc:493] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:02.017061  4263 leader_election.cc:290] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:02.025933  3793 consensus_queue.cc:260] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:02.017861  4184 consensus_queue.cc:237] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:02.028473  4383 leader_election.cc:290] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 5 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:02.014333  4432 raft_consensus.cc:493] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:02.057646  4206 raft_consensus.cc:515] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:02.057287  4395 consensus_meta.cc:287] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.706s	user 0.006s	sys 0.010s
I20260709 13:00:02.058163  4432 raft_consensus.cc:515] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:00:02.060405  4358 log.cc:927] Time spent T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 1.752s	user 0.004s	sys 0.000s
I20260709 13:00:02.060408  4395 raft_consensus.cc:2749] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260709 13:00:02.065788  4150 consensus_meta.cc:287] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.594s	user 0.012s	sys 0.000s
W20260709 13:00:02.049934  4333 consensus_meta.cc:287] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.636s	user 0.008s	sys 0.006s
I20260709 13:00:02.070546  4333 raft_consensus.cc:2749] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:00:02.071794  4432 leader_election.cc:290] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:02.043474  4443 raft_consensus.cc:515] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:02.088181  4423 log.cc:927] Time spent T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.503s	user 0.002s	sys 0.002s
I20260709 13:00:02.092618  4428 raft_consensus.cc:3060] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:00:02.090328  4406 raft_consensus.cc:515] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:02.096591  4406 leader_election.cc:290] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 7 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:02.094099  4443 leader_election.cc:290] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:02.138691  1597 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40154 (request call id 1802) took 1685 ms. Trace:
I20260709 13:00:02.140020  1597 rpcz_store.cc:276] 0709 13:00:00.453455 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:01.669191 (+1215736us) service_pool.cc:224] Handling call
0709 13:00:01.700137 (+ 30946us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:00:01.770178 (+ 70041us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:00:01.770228 (+    50us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:00:01.775654 (+  5426us) log.cc:844] Serialized 52 byte log entry
0709 13:00:02.135940 (+360286us) raft_consensus.cc:1645] Marking committed up to 3
0709 13:00:02.136830 (+   890us) log.cc:844] Serialized 14 byte log entry
0709 13:00:02.137437 (+   607us) log.cc:844] Serialized 14 byte log entry
0709 13:00:02.137660 (+   223us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:02.137833 (+   173us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:00:02.137938 (+   105us) raft_consensus.cc:1685] finished
0709 13:00:02.137966 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:02.138302 (+   336us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":61764,"raft.run_cpu_time_us":175,"raft.run_wall_time_us":8864,"thread_start_us":278898,"threads_started":1,"wal-append.queue_time_us":179975}
I20260709 13:00:02.144129  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:01.958588  1603 raft_consensus.cc:1240] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 4. Current term is 5. Ops: [4.5-4.5]
W20260709 13:00:02.146503  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 226712us
I20260709 13:00:02.156682  1603 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40154 (request call id 1810) took 1695 ms. Trace:
I20260709 13:00:02.157214  1603 rpcz_store.cc:276] 0709 13:00:00.460807 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:01.948995 (+1488188us) service_pool.cc:224] Handling call
0709 13:00:01.949596 (+   601us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:00:02.154708 (+205112us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:02.156398 (+  1690us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:00:02.153185  1676 raft_consensus.cc:1240] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 6. Current term is 7. Ops: [6.5-6.5]
I20260709 13:00:02.158591  1644 leader_election.cc:304] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:02.159904  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:02.162616  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:02.163787  1603 raft_consensus.cc:3060] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:00:02.171545  1597 raft_consensus.cc:3060] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Advancing to term 5
W20260709 13:00:02.180617  1718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:00:02.185587  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 214534us
I20260709 13:00:02.196141  2330 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39726 (request call id 753) took 2560 ms. Trace:
I20260709 13:00:02.199193  2330 rpcz_store.cc:276] 0709 12:59:59.635976 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:59.636127 (+   151us) service_pool.cc:224] Handling call
0709 12:59:59.729549 (+ 93422us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:59:59.765964 (+ 36415us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:59:59.775117 (+  9153us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:59:59.846726 (+ 71609us) log.cc:844] Serialized 26 byte log entry
0709 13:00:00.145719 (+298993us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:00:00.145825 (+   106us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:00.159232 (+ 13407us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:00:01.995201 (+1835969us) spinlock_profiling.cc:243] Waited 94.6 ms on lock 0x7b4400022b4c. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f4262f9bc85 00007f4262f9c6a2 00007f4265d0e1db 00007f4265d09aaf 00007f4265cfd86e 00007f4265cfcbb6 00007f4265cfc7d8 00007f4265d06221 00007f4265d061d9 00007f4265d06169
0709 13:00:02.195277 (+200076us) raft_consensus.cc:1685] finished
0709 13:00:02.195317 (+    40us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:02.195815 (+   498us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":11577,"raft.queue_time_us":51082,"raft.run_cpu_time_us":145,"raft.run_wall_time_us":10365,"spinlock_wait_cycles":217625472,"thread_start_us":245084,"threads_started":1,"wal-append.queue_time_us":338324}
I20260709 13:00:02.203378  3793 raft_consensus.cc:3060] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:00:02.067425  4140 raft_consensus.cc:3060] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:00:02.208947  3948 leader_election.cc:290] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:02.066345  4150 raft_consensus.cc:515] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:02.221033  1718 leader_election.cc:304] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:00:02.226503  4200 consensus_queue.cc:237] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:02.223241  4171 consensus_queue.cc:1059] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 13:00:02.234787  4150 leader_election.cc:290] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 6 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:02.045964  4249 consensus_queue.cc:237] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:02.230160  3521 raft_consensus.cc:2749] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:00:02.081892  4355 leader_election.cc:290] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 5 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:02.270771  4395 raft_consensus.cc:3055] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 4 LEADER]: Stepping down as leader of term 4
W20260709 13:00:02.271102  4262 consensus_meta.cc:287] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.809s	user 0.014s	sys 0.003s
I20260709 13:00:02.273586  4206 leader_election.cc:290] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 9 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:02.274768  3521 raft_consensus.cc:493] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:02.277410  4446 raft_consensus.cc:2749] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 13:00:02.277607  4390 raft_consensus.cc:493] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:02.281368  4390 raft_consensus.cc:515] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:02.285171  4283 leader_election.cc:290] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:02.290968  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:02.291873  3521 raft_consensus.cc:515] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:02.293871  4262 raft_consensus.cc:515] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:02.278517  4395 raft_consensus.cc:740] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
W20260709 13:00:02.306563  2328 consensus_meta.cc:287] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.521s	user 0.010s	sys 0.014s
I20260709 13:00:02.314517  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:02.294505  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:02.329372  4450 raft_consensus.cc:493] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:00:02.336308  4395 consensus_queue.cc:260] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:02.340615  1603 raft_consensus.cc:2468] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 7.
I20260709 13:00:02.345060  2333 raft_consensus.cc:1240] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 7 LEADER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 6. Current term is 7. Ops: [6.8-6.8]
I20260709 13:00:02.345177  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:00:02.305096  4298 raft_consensus.cc:697] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 7 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:00:02.347600  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
W20260709 13:00:02.350586  1604 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36462 (request call id 454) took 2688 ms (client timeout 1745 ms). Trace:
I20260709 13:00:02.350768  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:02.350844  4298 consensus_queue.cc:237] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:02.351285  1604 rpcz_store.cc:269] 0709 12:59:59.662062 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:59.785532 (+123470us) service_pool.cc:224] Handling call
0709 13:00:02.350042 (+2564510us) spinlock_profiling.cc:243] Waited 926 ms on lock 0x7b54002170d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:00:02.350301 (+   259us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2130812672}
I20260709 13:00:02.352118  3521 leader_election.cc:290] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:02.352861  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:02.353989  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:02.355032  4262 leader_election.cc:290] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 13:00:02.358886  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 49383.5us
I20260709 13:00:02.375415  1604 raft_consensus.cc:2393] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 6: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:00:02.368664  2328 raft_consensus.cc:2468] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 6.
W20260709 13:00:02.377715  1672 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57092 (request call id 549) took 2752 ms (client timeout 2736 ms). Trace:
I20260709 13:00:02.375718  1643 leader_election.cc:304] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:00:02.378218  2332 raft_consensus.cc:1240] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 5. Current term is 6. Ops: []
W20260709 13:00:02.378328  1672 rpcz_store.cc:269] 0709 12:59:59.624864 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:59.629195 (+  4331us) service_pool.cc:224] Handling call
0709 13:00:02.377129 (+2747934us) spinlock_profiling.cc:243] Waited 870 ms on lock 0x7b540025b054. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:00:02.377432 (+   303us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2000974720}
W20260709 13:00:02.379587  1646 connection.cc:256] RPC call timeout handler was delayed by -1.0095947610000167s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1634s for a response
I20260709 13:00:02.380221  4446 raft_consensus.cc:2804] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:00:02.380972  4446 raft_consensus.cc:697] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
W20260709 13:00:02.381088  1747 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55840 (request call id 2120) took 1763 ms (client timeout 2128 ms). Trace:
I20260709 13:00:02.381856  4333 raft_consensus.cc:493] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:00:02.381578  1747 rpcz_store.cc:269] 0709 13:00:00.617073 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:00.626714 (+  9641us) service_pool.cc:224] Handling call
0709 13:00:02.380558 (+1753844us) spinlock_profiling.cc:243] Waited 429 ms on lock 0x7b540031f454. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:00:02.380752 (+   194us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":986948864}
I20260709 13:00:02.392588  2337 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39738 (request call id 645) took 1509 ms. Trace:
I20260709 13:00:02.393039  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:02.397619  1597 raft_consensus.cc:2468] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 5.
I20260709 13:00:02.382829  4333 raft_consensus.cc:515] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:02.400661  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:02.393232  2337 rpcz_store.cc:276] 0709 13:00:00.882759 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:00.882882 (+   123us) service_pool.cc:224] Handling call
0709 13:00:01.059169 (+176287us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:00:01.065190 (+  6021us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:02.392026 (+1326836us) spinlock_profiling.cc:243] Waited 156 ms on lock 0x7b540045b5d0. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661618ad 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:00:02.392284 (+   258us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":358949120}
I20260709 13:00:02.403365  4406 raft_consensus.cc:493] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:02.404520  4406 raft_consensus.cc:515] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:02.406910  4445 raft_consensus.cc:493] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:02.399055  1718 leader_election.cc:304] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:02.393270  4344 consensus_queue.cc:1059] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.018s
I20260709 13:00:02.345758  4438 raft_consensus.cc:515] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:02.414358  4344 consensus_queue.cc:1243] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer 3b879799899a499c8d409ee9f1277cd1 log is divergent from this leader: its last log entry 7.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20260709 13:00:02.416307  3521 raft_consensus.cc:2749] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:00:02.425283  4326 consensus_queue.cc:260] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:02.427646  1718 leader_election.cc:304] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:02.428795  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:00:02.440966  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 44779.5us
I20260709 13:00:02.382889  4446 consensus_queue.cc:237] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:02.430670  4350 raft_consensus.cc:3055] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:00:02.431890  1604 raft_consensus.cc:1240] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260709 13:00:02.433022  4368 leader_election.cc:290] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:02.435910  4372 consensus_queue.cc:260] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:02.465867  4368 raft_consensus.cc:493] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:02.466274  4372 raft_consensus.cc:3060] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Advancing to term 7
W20260709 13:00:02.466702  1570 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:02.439365  4326 raft_consensus.cc:3060] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Advancing to term 7
W20260709 13:00:02.470175  1596 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36462 (request call id 458) took 2661 ms (client timeout 1784 ms). Trace:
I20260709 13:00:02.471006  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:02.471969  4272 raft_consensus.cc:493] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:02.472244  4374 raft_consensus.cc:493] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:02.473371  1598 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 463) took 2483 ms. Trace:
I20260709 13:00:02.473023  4374 raft_consensus.cc:515] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:02.449750  4350 raft_consensus.cc:740] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:00:02.475164  4299 raft_consensus.cc:493] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:00:02.475953  4299 raft_consensus.cc:515] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:02.476042  4350 consensus_queue.cc:260] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:02.477110  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 86890.4us
I20260709 13:00:02.478046  4350 raft_consensus.cc:3060] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:00:02.478997  1603 raft_consensus.cc:2468] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 4.
I20260709 13:00:02.479516  1599 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36462 (request call id 468) took 1389 ms. Trace:
I20260709 13:00:02.387152  4262 raft_consensus.cc:493] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
W20260709 13:00:02.452129  4369 log.cc:927] Time spent T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 2.010s	user 0.001s	sys 0.004s
I20260709 13:00:02.453096  1718 leader_election.cc:304] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:00:02.453891  3521 raft_consensus.cc:2749] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:00:02.464893  1605 raft_consensus.cc:3060] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:00:02.472985  4272 raft_consensus.cc:515] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:02.363420  4450 raft_consensus.cc:515] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:02.480007  1599 rpcz_store.cc:276] 0709 13:00:01.089513 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:01.119759 (+ 30246us) service_pool.cc:224] Handling call
0709 13:00:02.479013 (+1359254us) spinlock_profiling.cc:243] Waited 87 ms on lock 0x7b54002a3354. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:00:02.479286 (+   273us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":199997696}
I20260709 13:00:02.490895  1604 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40154 (request call id 1813) took 1988 ms. Trace:
I20260709 13:00:02.473842  1598 rpcz_store.cc:276] 0709 12:59:59.989520 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:01.642134 (+1652614us) service_pool.cc:224] Handling call
0709 13:00:01.642558 (+   424us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:01.959169 (+316611us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:02.472703 (+513534us) spinlock_profiling.cc:243] Waited 316 ms on lock 0x7b54000f1754. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:00:02.473104 (+   401us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":726045440}
W20260709 13:00:02.470726  1596 rpcz_store.cc:269] 0709 12:59:59.808376 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:59.820203 (+ 11827us) service_pool.cc:224] Handling call
0709 13:00:02.469637 (+2649434us) spinlock_profiling.cc:243] Waited 587 ms on lock 0x7b54001009d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:00:02.469919 (+   282us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1349967232}
I20260709 13:00:02.499735  1604 rpcz_store.cc:276] 0709 13:00:00.461985 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:02.430554 (+1968569us) service_pool.cc:224] Handling call
0709 13:00:02.431372 (+   818us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:00:02.450110 (+ 18738us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:02.450504 (+   394us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:00:02.447752  4395 raft_consensus.cc:3060] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:00:02.408007  4445 raft_consensus.cc:515] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:02.502849  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:02.457823  4244 raft_consensus.cc:515] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:02.505937  4299 leader_election.cc:290] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:02.529294  2337 raft_consensus.cc:2410] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20260709 13:00:02.466746  4368 raft_consensus.cc:515] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:02.507645  4310 raft_consensus.cc:493] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:00:02.532631  4310 raft_consensus.cc:515] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:02.533212  4406 leader_election.cc:290] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:02.508251  4333 leader_election.cc:290] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:02.533885  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:02.534863  4442 raft_consensus.cc:493] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:02.535315  4262 raft_consensus.cc:515] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:02.535797  4442 raft_consensus.cc:515] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:02.536960  4374 leader_election.cc:290] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:02.522763  1646 leader_election.cc:304] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:00:02.534929  2333 raft_consensus.cc:2393] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 5: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
W20260709 13:00:02.544692  1644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:00:02.547327  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 94392.9us
I20260709 13:00:02.548072  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:02.548923  1675 raft_consensus.cc:3060] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:00:02.550132  4450 leader_election.cc:290] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:02.551872  4244 leader_election.cc:290] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 7 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:00:02.553539  4455 log.cc:927] Time spent T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.105s	user 0.000s	sys 0.000s
I20260709 13:00:02.554605  4200 raft_consensus.cc:2804] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 13:00:02.555375  4200 raft_consensus.cc:697] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 6 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:00:02.564075  1599 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 1870) took 2087 ms. Trace:
I20260709 13:00:02.564678  1599 rpcz_store.cc:276] 0709 13:00:00.476407 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:02.562600 (+2086193us) service_pool.cc:224] Handling call
0709 13:00:02.563199 (+   599us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:02.563552 (+   353us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:00:02.563608 (+    56us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:00:02.563707 (+    99us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:00:02.563771 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:02.563846 (+    75us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:02.563972 (+   126us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:00:02.566934  4262 leader_election.cc:290] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:02.579670  1675 raft_consensus.cc:2410] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 for term 5 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260709 13:00:02.557005  4200 consensus_queue.cc:237] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 5.8, Last appended by leader: 8, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:02.577795  4333 raft_consensus.cc:2749] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:00:02.527282  1598 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "a3accbb733bb4cafa69428319664f239"
dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
mode: GRACEFUL
new_leader_uuid: "af520b67945d443b901bbeca37cc5701"
 from {username='slave'} at 127.0.0.1:47752
I20260709 13:00:02.601558  2298 leader_election.cc:304] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
W20260709 13:00:02.602674  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 58866us
I20260709 13:00:02.609771  1605 raft_consensus.cc:2468] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 9.
I20260709 13:00:02.607870  1598 raft_consensus.cc:606] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Received request to transfer leadership to TS af520b67945d443b901bbeca37cc5701
I20260709 13:00:02.624789  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:00:02.626665  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 185350us
I20260709 13:00:02.627750  1646 leader_election.cc:304] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:00:02.631350  1598 raft_consensus.cc:614] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 13:00:02.611253  4272 leader_election.cc:290] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:02.620779  4436 raft_consensus.cc:2804] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:00:02.642627  4246 raft_consensus.cc:3060] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:00:02.671007  4445 leader_election.cc:290] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:02.651714  4368 leader_election.cc:290] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:00:02.643915  3978 consensus_meta.cc:287] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.604s	user 0.014s	sys 0.011s
I20260709 13:00:02.652364  1674 raft_consensus.cc:1240] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 LEADER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 4. Current term is 5. Ops: []
I20260709 13:00:02.652529  4406 raft_consensus.cc:3060] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:00:02.654758  1679 raft_consensus.cc:1240] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 2. Current term is 3. Ops: []
W20260709 13:00:02.655113  4358 log.cc:927] Time spent T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.087s	user 0.000s	sys 0.001s
I20260709 13:00:02.642863  4265 leader_election.cc:290] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:02.659807  1646 leader_election.cc:304] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1
W20260709 13:00:02.660590  1717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:02.654785  4436 raft_consensus.cc:697] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 5 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:00:02.662952  4333 raft_consensus.cc:493] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:00:02.649701  4353 consensus_queue.cc:1059] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.033s
I20260709 13:00:02.665793  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:02.635720  4442 leader_election.cc:290] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:02.639209  4310 leader_election.cc:290] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:02.652009  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:02.679929  1605 raft_consensus.cc:2468] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 5.
I20260709 13:00:02.707722  4410 raft_consensus.cc:493] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
W20260709 13:00:02.708611  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 49063.9us
I20260709 13:00:02.709352  4445 raft_consensus.cc:2804] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:00:02.710248  4265 raft_consensus.cc:493] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
W20260709 13:00:02.710853  1644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:02.711234  4390 leader_election.cc:290] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:02.712065  1597 raft_consensus.cc:1275] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 6. (term mismatch)
I20260709 13:00:02.740442  1597 pending_rounds.cc:85] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a: Aborting all ops after (but not including) 5
I20260709 13:00:02.714377  3978 raft_consensus.cc:2749] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:00:02.714833  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:02.716316  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:00:02.722923  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:00:02.741096  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 150042us
I20260709 13:00:02.744479  1599 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 467) took 2029 ms. Trace:
I20260709 13:00:02.746625  1676 raft_consensus.cc:2468] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 4.
I20260709 13:00:02.746991  4436 consensus_queue.cc:237] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:02.747889  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:02.749253  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:02.750581  4333 raft_consensus.cc:515] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:02.753043  4333 leader_election.cc:290] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:00:02.755244  2298 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:02.755802  1646 leader_election.cc:304] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:00:02.758867  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:02.740794  1597 pending_rounds.cc:107] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a: Aborting uncommitted NO_OP operation due to leader change: 3.6
I20260709 13:00:02.760099  1597 raft_consensus.cc:2889] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:00:02.741546  4445 raft_consensus.cc:493] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:02.765425  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:02.766752  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:02.767387  1674 raft_consensus.cc:2468] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 4.
I20260709 13:00:02.768297  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:02.769064  1674 raft_consensus.cc:2393] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 6: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:00:02.769815  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:02.771963  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:02.773831  1676 raft_consensus.cc:1275] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260709 13:00:02.774009  1678 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57092 (request call id 555) took 1920 ms. Trace:
I20260709 13:00:02.774596  1678 rpcz_store.cc:276] 0709 13:00:00.853308 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:00.856861 (+  3553us) service_pool.cc:224] Handling call
0709 13:00:01.787023 (+930162us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:01.787716 (+   693us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:02.773568 (+985852us) spinlock_profiling.cc:243] Waited 927 ms on lock 0x7b5400163d50. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661618ad 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:00:02.773807 (+   239us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2132555648}
I20260709 13:00:02.776430  1676 raft_consensus.cc:1240] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 5. Current term is 6. Ops: []
W20260709 13:00:02.778983  2298 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:02.782755  4365 raft_consensus.cc:3060] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:00:02.783911  1679 raft_consensus.cc:2376] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 7. Re-sending same reply.
I20260709 13:00:02.761370  4445 raft_consensus.cc:3060] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:00:02.800669  1597 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40154 (request call id 1815) took 2334 ms. Trace:
I20260709 13:00:02.809209  1597 rpcz_store.cc:276] 0709 13:00:00.463114 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:02.710947 (+2247833us) service_pool.cc:224] Handling call
0709 13:00:02.711491 (+   544us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:02.797120 (+ 85629us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:02.797384 (+   264us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:00:02.811012  4184 raft_consensus.cc:493] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:00:02.811965  4184 raft_consensus.cc:515] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } }
I20260709 13:00:02.799928  4265 raft_consensus.cc:515] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:02.816702  1600 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40166 (request call id 1877) took 1700 ms (client timeout 1795 ms). Trace:
I20260709 13:00:02.817242  4265 leader_election.cc:290] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:00:02.817181  1600 rpcz_store.cc:269] 0709 13:00:01.116593 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:01.136953 (+ 20360us) service_pool.cc:224] Handling call
0709 13:00:02.816148 (+1679195us) spinlock_profiling.cc:243] Waited 492 ms on lock 0x7b54002a1f54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:00:02.816481 (+   333us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1130573184}
I20260709 13:00:02.819629  1604 raft_consensus.cc:1275] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 4. (term mismatch)
I20260709 13:00:02.820060  1604 pending_rounds.cc:85] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a: Aborting all ops after (but not including) 3
I20260709 13:00:02.820322  1604 pending_rounds.cc:107] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a: Aborting uncommitted NO_OP operation due to leader change: 4.4
I20260709 13:00:02.820549  1604 raft_consensus.cc:2889] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 13:00:02.822819  2299 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:02.826356  4273 consensus_queue.cc:1048] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20260709 13:00:02.828157  4465 log.cc:927] Time spent T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.118s	user 0.002s	sys 0.000s
I20260709 13:00:02.829151  1646 leader_election.cc:304] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:02.830400  4273 raft_consensus.cc:2804] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:00:02.830802  1604 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40154 (request call id 1819) took 2366 ms. Trace:
I20260709 13:00:02.830983  4273 raft_consensus.cc:493] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Starting leader election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:00:02.831560  4273 raft_consensus.cc:3060] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:00:02.831316  1604 rpcz_store.cc:276] 0709 13:00:00.464678 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:02.500742 (+2036064us) service_pool.cc:224] Handling call
0709 13:00:02.819139 (+318397us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:02.821126 (+  1987us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:02.830577 (+  9451us) spinlock_profiling.cc:243] Waited 318 ms on lock 0x7b54002a1f50. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661618ad 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:00:02.830733 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":730894208}
I20260709 13:00:02.835033  4184 leader_election.cc:290] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:02.835973  4298 raft_consensus.cc:3055] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:00:02.836220  1600 raft_consensus.cc:1240] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20260709 13:00:02.836392  4298 raft_consensus.cc:740] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:00:02.837009  1600 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 1878) took 1720 ms. Trace:
I20260709 13:00:02.837414  1600 rpcz_store.cc:276] 0709 13:00:01.116820 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:02.818258 (+1701438us) service_pool.cc:224] Handling call
0709 13:00:02.835956 (+ 17698us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:00:02.836687 (+   731us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:02.836930 (+   243us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:00:02.839951  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:00:02.853261  1643 leader_election.cc:336] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 1.795s (SENT)
I20260709 13:00:02.854027  1643 leader_election.cc:304] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:00:02.855592  4184 raft_consensus.cc:2749] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:00:02.848181  1598 raft_consensus.cc:2468] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 4.
I20260709 13:00:02.786334  4442 raft_consensus.cc:3060] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:00:02.875700  1717 leader_election.cc:304] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:00:02.885492  2299 leader_election.cc:304] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:00:02.744966  1599 rpcz_store.cc:276] 0709 13:00:00.715549 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:02.565736 (+1850187us) service_pool.cc:224] Handling call
0709 13:00:02.600043 (+ 34307us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:02.672946 (+ 72903us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 13:00:02.708734 (+ 35788us) log.cc:844] Serialized 14 byte log entry
0709 13:00:02.743944 (+ 35210us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:00:02.744008 (+    64us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:00:02.744065 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:02.744176 (+   111us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:02.744404 (+   228us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":18772,"prepare.run_cpu_time_us":131,"prepare.run_wall_time_us":1300,"raft.queue_time_us":116,"raft.run_cpu_time_us":127,"raft.run_wall_time_us":126,"thread_start_us":34276,"threads_started":1,"wal-append.queue_time_us":34718}
W20260709 13:00:02.891745  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 48743.4us
I20260709 13:00:02.892947  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:00:02.897308  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 68314us
I20260709 13:00:02.900254  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:02.901070  1598 raft_consensus.cc:2393] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 5: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:00:02.899833  4436 raft_consensus.cc:2804] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:00:02.837528  4298 consensus_queue.cc:260] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:02.838043  4410 raft_consensus.cc:515] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:02.920904  1599 raft_consensus.cc:1240] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 6. Current term is 7. Ops: []
I20260709 13:00:02.921757  1599 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 1880) took 1804 ms. Trace:
I20260709 13:00:02.922101  1599 rpcz_store.cc:276] 0709 13:00:01.117409 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:02.886811 (+1769402us) service_pool.cc:224] Handling call
0709 13:00:02.910914 (+ 24103us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:02.921458 (+ 10544us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:02.921692 (+   234us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:00:02.907979  4436 raft_consensus.cc:493] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:00:02.928835  4070 consensus_queue.cc:1059] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 4.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.027s
I20260709 13:00:02.894522  4353 consensus_queue.cc:1059] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.024s
I20260709 13:00:02.933609  4395 raft_consensus.cc:493] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:02.919687  4265 raft_consensus.cc:2804] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:00:02.931087  4436 raft_consensus.cc:3060] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:00:02.942203  4310 raft_consensus.cc:493] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:00:02.948596  4310 raft_consensus.cc:515] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:02.949949  1717 leader_election.cc:304] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:00:02.931906  4200 raft_consensus.cc:493] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:00:02.953925  1599 raft_consensus.cc:1240] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 6. Current term is 7. Ops: []
I20260709 13:00:02.954787  1599 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40154 (request call id 1824) took 1599 ms. Trace:
I20260709 13:00:02.955124  1599 rpcz_store.cc:276] 0709 13:00:01.365250 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:02.922852 (+1557602us) service_pool.cc:224] Handling call
0709 13:00:02.951046 (+ 28194us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:02.954491 (+  3445us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:02.954719 (+   228us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:00:02.956048  4126 raft_consensus.cc:493] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:02.956642  4126 raft_consensus.cc:515] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:02.958055  4310 leader_election.cc:290] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:02.958847  1672 raft_consensus.cc:2468] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 5.
I20260709 13:00:02.961159  1602 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 456) took 3191 ms. Trace:
W20260709 13:00:02.970731  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 54215.5us
I20260709 13:00:02.963173  4200 raft_consensus.cc:515] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } }
I20260709 13:00:02.977003  4410 leader_election.cc:290] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:02.977849  4200 leader_election.cc:290] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:02.978947  4244 raft_consensus.cc:2804] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Leader election won for term 9
W20260709 13:00:02.980212  2298 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:02.974905  1598 raft_consensus.cc:1275] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 13:00:02.972875  1602 rpcz_store.cc:276] 0709 12:59:59.769393 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:01.466305 (+1696912us) service_pool.cc:224] Handling call
0709 13:00:01.477976 (+ 11671us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:01.819466 (+341490us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:02.960609 (+1141143us) spinlock_profiling.cc:243] Waited 332 ms on lock 0x7b5400215cd4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:00:02.960925 (+   316us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":763743104}
I20260709 13:00:02.981489  1598 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 472) took 1657 ms. Trace:
I20260709 13:00:02.962368  1596 raft_consensus.cc:1240] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 4. Current term is 5. Ops: []
I20260709 13:00:02.981928  1598 rpcz_store.cc:276] 0709 13:00:01.324385 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:02.904615 (+1580230us) service_pool.cc:224] Handling call
0709 13:00:02.939615 (+ 35000us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:02.981146 (+ 41531us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:02.981425 (+   279us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:00:02.945855  4265 raft_consensus.cc:493] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:00:02.949831  4070 consensus_queue.cc:1243] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer 816fadcea8fc4893882f5f9b9f22a03d log is divergent from this leader: its last log entry 5.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
W20260709 13:00:02.966143  4468 log.cc:927] Time spent T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.069s	user 0.001s	sys 0.000s
I20260709 13:00:02.988175  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:02.988485  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:02.964787  4298 raft_consensus.cc:3060] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:00:02.984184  1596 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 475) took 1386 ms. Trace:
I20260709 13:00:02.994093  1596 rpcz_store.cc:276] 0709 13:00:01.597637 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:02.961549 (+1363912us) service_pool.cc:224] Handling call
0709 13:00:02.962036 (+   487us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:02.982645 (+ 20609us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:02.984115 (+  1470us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:00:02.934386  4395 raft_consensus.cc:515] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:02.962908  4144 raft_consensus.cc:3055] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:00:02.965118  4438 leader_election.cc:290] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 6 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:02.998636  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:00:02.988786  4406 raft_consensus.cc:2749] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260709 13:00:02.995088  4326 consensus_meta.cc:287] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.527s	user 0.010s	sys 0.006s
I20260709 13:00:02.990815  4265 raft_consensus.cc:3060] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:00:03.012761  1671 raft_consensus.cc:2410] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 for term 7 because replica has last-logged OpId of term: 6 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20260709 13:00:02.988210  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:03.006209  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:03.019024  4150 raft_consensus.cc:493] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:03.019248  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:03.019708  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:03.019825  4150 raft_consensus.cc:515] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:03.020466  1596 raft_consensus.cc:1275] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 13:00:03.021162  1596 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 477) took 1273 ms. Trace:
I20260709 13:00:03.021478  1596 rpcz_store.cc:276] 0709 13:00:01.748212 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:02.999682 (+1251470us) service_pool.cc:224] Handling call
0709 13:00:03.020037 (+ 20355us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:03.020903 (+   866us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:03.021099 (+   196us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:00:03.022346  4150 leader_election.cc:290] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:03.022747  2329 raft_consensus.cc:3055] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:00:03.023161  2329 raft_consensus.cc:740] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:00:03.023615  4395 leader_election.cc:290] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:02.999614  4144 raft_consensus.cc:740] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:00:03.024457  1599 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 1885) took 1351 ms. Trace:
I20260709 13:00:03.024256  2329 consensus_queue.cc:260] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:03.009428  4262 consensus_queue.cc:1048] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.010s
I20260709 13:00:03.025146  4144 consensus_queue.cc:260] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:03.024864  1599 rpcz_store.cc:276] 0709 13:00:01.672801 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:02.984846 (+1312045us) service_pool.cc:224] Handling call
0709 13:00:03.006601 (+ 21755us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:03.024019 (+ 17418us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:00:03.024054 (+    35us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:00:03.024127 (+    73us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:00:03.024177 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:03.024258 (+    81us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:03.024385 (+   127us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:00:03.026641  4144 raft_consensus.cc:3060] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:00:03.027704  1599 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 479) took 1196 ms. Trace:
I20260709 13:00:03.028079  1599 rpcz_store.cc:276] 0709 13:00:01.831366 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:03.026652 (+1195286us) service_pool.cc:224] Handling call
0709 13:00:03.027058 (+   406us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:03.027349 (+   291us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:00:03.027375 (+    26us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:00:03.027443 (+    68us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:00:03.027484 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:03.027544 (+    60us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:03.027646 (+   102us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:00:03.029906  1599 raft_consensus.cc:1275] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 13:00:03.031216  2336 raft_consensus.cc:1275] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 13:00:03.020593  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:00:03.037617  1599 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40154 (request call id 1833) took 1093 ms. Trace:
I20260709 13:00:03.013597  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:03.039939  4395 raft_consensus.cc:493] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:03.040548  4395 raft_consensus.cc:515] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:03.017405  1673 raft_consensus.cc:2468] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 4.
I20260709 13:00:03.038054  1599 rpcz_store.cc:276] 0709 13:00:01.944688 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:03.029054 (+1084366us) service_pool.cc:224] Handling call
0709 13:00:03.029525 (+   471us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:03.030217 (+   692us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:03.037533 (+  7316us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:00:02.999229  4244 raft_consensus.cc:697] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 9 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:00:03.006776  4200 raft_consensus.cc:493] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:03.049870  2298 leader_election.cc:304] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
W20260709 13:00:03.053359  1718 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.107.196:46625 timed out after 1.977s (SENT) [suppressed 33 similar messages]
I20260709 13:00:03.052253  1598 raft_consensus.cc:1240] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 8. Current term is 9. Ops: [8.9-8.9]
W20260709 13:00:03.045214  4471 log.cc:927] Time spent T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.116s	user 0.002s	sys 0.000s
I20260709 13:00:03.025810  2329 raft_consensus.cc:3060] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:00:03.056989  4353 raft_consensus.cc:2804] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:00:03.065162  1602 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 1886) took 1392 ms. Trace:
I20260709 13:00:03.072014  1602 rpcz_store.cc:276] 0709 13:00:01.673121 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:02.992856 (+1319735us) service_pool.cc:224] Handling call
0709 13:00:03.064435 (+ 71579us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:03.064716 (+   281us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:00:03.064756 (+    40us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:00:03.064818 (+    62us) raft_consensus.cc:1645] Marking committed up to 3
0709 13:00:03.064902 (+    84us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:03.064986 (+    84us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:03.065096 (+   110us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:00:03.074880  1603 raft_consensus.cc:1275] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 13:00:03.075049  4200 raft_consensus.cc:515] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:03.075644  1603 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40154 (request call id 1817) took 2611 ms. Trace:
I20260709 13:00:03.076126  2331 pending_rounds.cc:85] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1: Aborting all ops after (but not including) 6
I20260709 13:00:03.076529  2331 pending_rounds.cc:107] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1: Aborting uncommitted NO_OP operation due to leader change: 5.7
I20260709 13:00:03.076802  2331 raft_consensus.cc:2889] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:00:03.078019  4200 leader_election.cc:290] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:03.003815  4410 consensus_queue.cc:1048] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.051s
I20260709 13:00:03.078773  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:03.079998  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:03.081816  1605 raft_consensus.cc:1240] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 3. Current term is 4. Ops: []
I20260709 13:00:03.082628  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:03.083112  4353 raft_consensus.cc:493] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:03.083568  1601 raft_consensus.cc:1275] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 6. (term mismatch)
I20260709 13:00:03.084527  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:03.089835  4365 consensus_queue.cc:1048] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 13:00:03.091053  4244 consensus_queue.cc:237] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:03.092000  3977 raft_consensus.cc:493] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
W20260709 13:00:03.094673  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 45070.7us
I20260709 13:00:03.140383  1680 raft_consensus.cc:2410] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for term 7 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260709 13:00:03.128945  1672 raft_consensus.cc:2468] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 3.
I20260709 13:00:03.095851  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:03.096628  2335 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49082 (request call id 630) took 1232 ms. Trace:
I20260709 13:00:03.139284  1569 leader_election.cc:304] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:00:03.128252  1601 pending_rounds.cc:85] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a: Aborting all ops after (but not including) 5
I20260709 13:00:03.143465  1601 pending_rounds.cc:107] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a: Aborting uncommitted NO_OP operation due to leader change: 5.6
I20260709 13:00:03.096798  1675 raft_consensus.cc:1275] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 7 index: 8. (term mismatch)
W20260709 13:00:03.104274  4469 log.cc:927] Time spent T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.208s	user 0.001s	sys 0.000s
W20260709 13:00:03.107909  4470 log.cc:927] Time spent T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.151s	user 0.000s	sys 0.001s
I20260709 13:00:03.109434  4395 leader_election.cc:290] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:03.114691  4446 consensus_queue.cc:1059] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.009s
I20260709 13:00:03.115494  4070 consensus_queue.cc:1059] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 13:00:03.186710  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:03.187670  4070 raft_consensus.cc:3055] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:00:03.188092  4070 raft_consensus.cc:740] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:00:03.188164  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:03.188721  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:03.189376  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:03.189189  4070 consensus_queue.cc:260] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:03.189987  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:03.190593  3977 raft_consensus.cc:515] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:03.191277  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:03.192538  1602 raft_consensus.cc:1275] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 13:00:03.076018  1603 rpcz_store.cc:276] 0709 13:00:00.463731 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:03.073988 (+2610257us) service_pool.cc:224] Handling call
0709 13:00:03.074430 (+   442us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:03.075348 (+   918us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:03.075555 (+   207us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:00:03.116210  4310 raft_consensus.cc:493] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
W20260709 13:00:03.116914  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 50648.2us
I20260709 13:00:03.061071  1598 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 476) took 1313 ms. Trace:
I20260709 13:00:03.143785  1601 raft_consensus.cc:2889] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:00:03.161337  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:03.106003  4445 raft_consensus.cc:515] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:03.183473  4474 log.cc:927] Time spent T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.065s	user 0.001s	sys 0.000s
I20260709 13:00:03.220366  1604 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 1879) took 2103 ms. Trace:
W20260709 13:00:03.221068  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 81869.2us
I20260709 13:00:03.160068  1675 pending_rounds.cc:85] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d: Aborting all ops after (but not including) 7
I20260709 13:00:03.194666  2299 leader_election.cc:304] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:00:03.188201  2298 leader_election.cc:304] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:00:03.197335  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:03.193145  3977 leader_election.cc:290] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:03.139858  4353 raft_consensus.cc:3060] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:00:03.202149  1598 rpcz_store.cc:276] 0709 13:00:01.747869 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:02.984803 (+1236934us) service_pool.cc:224] Handling call
0709 13:00:03.051869 (+ 67066us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:00:03.060697 (+  8828us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:03.061000 (+   303us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:00:03.142611  2335 rpcz_store.cc:276] 0709 13:00:01.864443 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:01.864660 (+   217us) service_pool.cc:224] Handling call
0709 13:00:01.877591 (+ 12931us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:00:02.855979 (+978388us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:00:02.856053 (+    74us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:00:02.856875 (+   822us) log.cc:844] Serialized 26 byte log entry
0709 13:00:02.946335 (+ 89460us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:00:02.946387 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:03.075307 (+128920us) spinlock_profiling.cc:243] Waited 894 ms on lock 0x7b54004f88d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:00:03.075365 (+    58us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:00:03.096139 (+ 20774us) raft_consensus.cc:1685] finished
0709 13:00:03.096163 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:03.096412 (+   249us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":82936,"raft.queue_time_us":83791,"raft.run_cpu_time_us":160,"raft.run_wall_time_us":160,"spinlock_wait_cycles":2057125760,"thread_start_us":88922,"threads_started":1,"wal-append.queue_time_us":237018}
I20260709 13:00:03.197872  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:03.179111  4372 raft_consensus.cc:2749] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260709 13:00:03.238210  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 63165.9us
I20260709 13:00:03.231491  1675 pending_rounds.cc:107] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d: Aborting uncommitted NO_OP operation due to leader change: 6.8
I20260709 13:00:03.238500  4365 raft_consensus.cc:2804] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260709 13:00:03.238689  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 50611.2us
I20260709 13:00:03.239926  1673 raft_consensus.cc:1275] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260709 13:00:03.240298  4297 raft_consensus.cc:2804] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:00:03.240850  4448 raft_consensus.cc:493] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:03.241428  4126 leader_election.cc:290] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:03.249823  4436 raft_consensus.cc:515] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:03.231415  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:03.237912  4070 raft_consensus.cc:3060] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:00:03.243575  4184 raft_consensus.cc:493] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:03.286602  2334 raft_consensus.cc:2468] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 5.
I20260709 13:00:03.245359  1674 raft_consensus.cc:1240] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 6. Current term is 7. Ops: []
I20260709 13:00:03.220894  1604 rpcz_store.cc:276] 0709 13:00:01.117147 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:02.832635 (+1715488us) service_pool.cc:224] Handling call
0709 13:00:02.833150 (+   515us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:00:02.833673 (+   523us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:00:02.833705 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:00:02.834277 (+   572us) log.cc:844] Serialized 26 byte log entry
0709 13:00:02.911081 (+ 76804us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:00:02.911235 (+   154us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:02.911426 (+   191us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:00:03.220071 (+308645us) raft_consensus.cc:1685] finished
0709 13:00:03.220096 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:03.220287 (+   191us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":122,"raft.run_cpu_time_us":143,"raft.run_wall_time_us":143,"thread_start_us":41843,"threads_started":1,"wal-append.queue_time_us":138179}
I20260709 13:00:03.186798  4157 raft_consensus.cc:493] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
W20260709 13:00:03.258831  2301 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:03.261495  2329 raft_consensus.cc:1275] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 5 index: 8. (index mismatch)
I20260709 13:00:03.138756  4326 consensus_queue.cc:1048] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.086s
W20260709 13:00:03.244037  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:03.278554  4297 raft_consensus.cc:493] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:03.308565  4157 raft_consensus.cc:515] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:03.264671  4273 raft_consensus.cc:515] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:03.264324  1675 raft_consensus.cc:2889] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:00:03.262765  4406 raft_consensus.cc:493] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:00:03.316469  4436 leader_election.cc:290] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 5 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 13:00:03.318118  1644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:03.229816  4298 consensus_queue.cc:1048] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.068s
W20260709 13:00:03.320267  2334 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39738 (request call id 648) took 1628 ms (client timeout 1977 ms). Trace:
I20260709 13:00:03.321597  4445 leader_election.cc:290] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 7 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:03.322932  2299 leader_election.cc:304] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:00:03.270152  4365 raft_consensus.cc:493] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:00:03.331032  4448 raft_consensus.cc:515] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:03.289510  1571 leader_election.cc:304] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 13:00:03.332917  4390 consensus_queue.cc:1048] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 13:00:03.351920  4406 raft_consensus.cc:515] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:03.372668  4273 leader_election.cc:290] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 3 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:03.345448  4310 raft_consensus.cc:515] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:03.382967  1601 raft_consensus.cc:1275] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
W20260709 13:00:03.387171  2334 rpcz_store.cc:269] 0709 13:00:01.692168 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:01.692338 (+   170us) service_pool.cc:224] Handling call
0709 13:00:03.319995 (+1627657us) spinlock_profiling.cc:243] Waited 1.38 s on lock 0x7b54007602d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:00:03.320158 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3184107776}
I20260709 13:00:03.286747  4184 raft_consensus.cc:515] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:03.349781  4297 raft_consensus.cc:3060] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:00:03.416132  4157 leader_election.cc:290] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:00:03.417083  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 184530us
I20260709 13:00:03.422602  4448 leader_election.cc:290] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:03.409410  4365 raft_consensus.cc:3060] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Advancing to term 5
W20260709 13:00:03.430932  4201 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 5 current term: 5 is higher.
I20260709 13:00:03.435071  1680 raft_consensus.cc:1275] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20260709 13:00:03.432235  4442 consensus_meta.cc:287] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.545s	user 0.012s	sys 0.000s
I20260709 13:00:03.439354  4157 raft_consensus.cc:493] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:03.424865  4265 raft_consensus.cc:515] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:00:03.461308  1676 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.599s	user 0.030s	sys 0.042s
I20260709 13:00:03.460645  4246 raft_consensus.cc:3055] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:00:03.463532  4157 raft_consensus.cc:515] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:03.469707  1676 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.607s	user 0.031s	sys 0.042s
W20260709 13:00:03.475417  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 488894us
W20260709 13:00:03.489203  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 435794us
I20260709 13:00:03.490658  4200 consensus_queue.cc:1059] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 13:00:03.459841  4442 raft_consensus.cc:2749] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:00:03.493988  1674 raft_consensus.cc:1240] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 4. Current term is 5. Ops: []
I20260709 13:00:03.490825  1718 leader_election.cc:304] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:00:03.499763  3521 raft_consensus.cc:2804] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:00:03.500263  3521 raft_consensus.cc:493] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:00:03.500685  3521 raft_consensus.cc:3060] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:00:03.501839  1596 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 478) took 1711 ms. Trace:
I20260709 13:00:03.506357  1718 leader_election.cc:304] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:03.469367  3977 raft_consensus.cc:493] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:03.506814  1596 rpcz_store.cc:276] 0709 13:00:01.790519 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:03.022202 (+1231683us) service_pool.cc:224] Handling call
0709 13:00:03.045347 (+ 23145us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:00:03.092605 (+ 47258us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:00:03.092642 (+    37us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:00:03.093417 (+   775us) log.cc:844] Serialized 52 byte log entry
0709 13:00:03.476126 (+382709us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:00:03.476155 (+    29us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:03.476210 (+    55us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:00:03.501493 (+ 25283us) raft_consensus.cc:1685] finished
0709 13:00:03.501527 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:03.501731 (+   204us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":34083,"raft.run_cpu_time_us":107,"raft.run_wall_time_us":108,"thread_start_us":382314,"threads_started":1,"wal-append.queue_time_us":390818}
I20260709 13:00:03.473724  4406 leader_election.cc:290] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:03.472049  4246 raft_consensus.cc:740] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:00:03.502321  4368 raft_consensus.cc:2804] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:00:03.477720  4395 raft_consensus.cc:493] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:00:03.530836  4438 raft_consensus.cc:493] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:03.538757  4298 raft_consensus.cc:2749] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 13:00:03.578249  1718 leader_election.cc:304] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:03.556623  4200 raft_consensus.cc:2749] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20260709 13:00:03.594695  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 306120us
W20260709 13:00:03.610206  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 287362us
I20260709 13:00:03.613037  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
W20260709 13:00:03.615145  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 49423.4us
I20260709 13:00:03.616309  1718 leader_election.cc:304] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:03.554200  3977 raft_consensus.cc:515] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:03.553309  4395 raft_consensus.cc:515] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:03.597249  4184 leader_election.cc:290] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:03.564881  4438 raft_consensus.cc:515] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:03.636195  1676 raft_consensus.cc:1275] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 6. (term mismatch)
I20260709 13:00:03.636620  1676 pending_rounds.cc:85] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d: Aborting all ops after (but not including) 5
I20260709 13:00:03.636909  1676 pending_rounds.cc:107] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d: Aborting uncommitted NO_OP operation due to leader change: 3.6
I20260709 13:00:03.637202  1676 raft_consensus.cc:2889] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 13:00:03.634516  2299 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:03.623641  4200 raft_consensus.cc:493] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:03.649459  1597 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 471) took 2542 ms. Trace:
I20260709 13:00:03.659211  1597 rpcz_store.cc:276] 0709 13:00:01.107103 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:02.844686 (+1737583us) service_pool.cc:224] Handling call
0709 13:00:02.884811 (+ 40125us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:02.919326 (+ 34515us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 13:00:02.954365 (+ 35039us) log.cc:844] Serialized 14 byte log entry
0709 13:00:03.649132 (+694767us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:00:03.649184 (+    52us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:00:03.649237 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:03.649302 (+    65us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:03.649392 (+    90us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":26466,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":49,"thread_start_us":627076,"threads_started":2,"wal-append.queue_time_us":538290}
W20260709 13:00:03.664492  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 442189us
W20260709 13:00:03.670495  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 225679us
I20260709 13:00:03.613019  4244 consensus_queue.cc:1048] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.048s
W20260709 13:00:03.673339  2301 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:03.661186  4310 leader_election.cc:290] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:03.668656  4246 consensus_queue.cc:260] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 5.8, Last appended by leader: 8, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:03.694065  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 8 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:03.694855  4246 raft_consensus.cc:3060] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:00:03.695144  1745 raft_consensus.cc:2410] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for term 8 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260709 13:00:03.696185  1750 raft_consensus.cc:1240] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 6 LEADER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 5. Current term is 6. Ops: []
I20260709 13:00:03.635485  4298 consensus_queue.cc:1059] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.049s
I20260709 13:00:03.667407  4144 consensus_queue.cc:1048] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.024s
I20260709 13:00:03.685772  4265 leader_election.cc:290] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:03.719425  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:03.731789  4244 consensus_queue.cc:1048] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:00:03.733161  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:03.734021  1754 raft_consensus.cc:2468] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 5.
I20260709 13:00:03.734617  4487 consensus_queue.cc:1048] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:00:03.699002  4449 consensus_queue.cc:1048] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.040s
W20260709 13:00:03.739898  4353 consensus_meta.cc:287] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.501s	user 0.009s	sys 0.005s
I20260709 13:00:03.740409  4353 raft_consensus.cc:515] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:03.743222  4298 consensus_queue.cc:1048] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 13:00:03.712973  4310 consensus_queue.cc:1048] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.035s
I20260709 13:00:03.745378  4310 raft_consensus.cc:493] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:03.709976  4478 raft_consensus.cc:2749] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 13:00:03.746915  4297 raft_consensus.cc:515] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:03.747994  4201 consensus_queue.cc:1048] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 13:00:03.748692  4395 leader_election.cc:290] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:03.749357  4297 leader_election.cc:290] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 6 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:00:03.750696  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 66644.9us
I20260709 13:00:03.752305  4500 raft_consensus.cc:493] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:03.752920  4500 raft_consensus.cc:515] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:03.753973  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:03.755831  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:03.757251  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:03.758582  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:03.759261  4297 raft_consensus.cc:3055] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:00:03.759725  4297 raft_consensus.cc:740] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:00:03.760677  4297 consensus_queue.cc:260] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:03.761385  3978 raft_consensus.cc:3055] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:00:03.761912  3978 raft_consensus.cc:740] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:00:03.762353  4297 raft_consensus.cc:3060] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:00:03.710798  4157 leader_election.cc:290] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:03.763935  3978 consensus_queue.cc:260] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:03.765710  3978 raft_consensus.cc:3060] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:00:03.770586  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:03.771422  1753 raft_consensus.cc:2468] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 6.
I20260709 13:00:03.692899  4200 raft_consensus.cc:515] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:03.732249  4481 log.cc:927] Time spent T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.111s	user 0.000s	sys 0.000s
I20260709 13:00:03.745987  4310 raft_consensus.cc:515] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:00:03.940588  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 280470us
I20260709 13:00:03.774106  1600 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 1882) took 2427 ms. Trace:
I20260709 13:00:03.825587  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:00:03.939886  1717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:00:04.090289  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 415984us
I20260709 13:00:03.863487  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:00:04.090680  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 404920us
I20260709 13:00:04.090901  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:03.976305  1752 raft_consensus.cc:1275] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
W20260709 13:00:03.828491  1644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:03.814050  1745 raft_consensus.cc:2468] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 4.
I20260709 13:00:03.997857  1753 raft_consensus.cc:1240] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20260709 13:00:03.805014  3521 raft_consensus.cc:515] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } }
I20260709 13:00:04.096619  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:04.073230  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:03.773339  4500 leader_election.cc:290] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:04.098361  1749 raft_consensus.cc:2393] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 4: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:00:04.098810  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:03.813038  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:04.099714  2337 raft_consensus.cc:2393] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 5: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
W20260709 13:00:04.087402  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 421773us
I20260709 13:00:04.086186  1600 rpcz_store.cc:276] 0709 13:00:01.346751 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:02.917504 (+1570753us) service_pool.cc:224] Handling call
0709 13:00:02.926978 (+  9474us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:02.942764 (+ 15786us) raft_consensus.cc:1534] Early marking committed up to index 3
0709 13:00:03.001943 (+ 59179us) log.cc:844] Serialized 14 byte log entry
0709 13:00:03.773706 (+771763us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:00:03.773781 (+    75us) raft_consensus.cc:1645] Marking committed up to 3
0709 13:00:03.773842 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:03.773915 (+    73us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:03.774036 (+   121us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":21328,"prepare.queue_time_us":46827,"prepare.run_cpu_time_us":86,"prepare.run_wall_time_us":2186,"raft.queue_time_us":252293,"raft.run_cpu_time_us":132,"raft.run_wall_time_us":131,"thread_start_us":712980,"threads_started":2,"wal-append.queue_time_us":445345}
W20260709 13:00:04.103994  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 455070us
W20260709 13:00:04.078819  4365 consensus_meta.cc:287] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.635s	user 0.014s	sys 0.002s
W20260709 13:00:04.111660  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 245616us
W20260709 13:00:04.112064  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:00:04.112534  1646 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:00:04.113103  1646 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:04.109184  3521 leader_election.cc:290] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 4 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:04.083595  4200 leader_election.cc:290] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:03.692276  4480 raft_consensus.cc:2749] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260709 13:00:04.092926  4070 consensus_meta.cc:287] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.804s	user 0.015s	sys 0.000s
I20260709 13:00:03.773355  4438 leader_election.cc:290] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:04.088328  3977 leader_election.cc:290] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:03.968685  4395 raft_consensus.cc:493] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:00:04.119068  3521 raft_consensus.cc:493] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:04.024221  4353 leader_election.cc:290] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 5 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:04.116148  4365 raft_consensus.cc:515] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:04.120103  4478 raft_consensus.cc:493] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:00:04.135483  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:04.133524  3521 raft_consensus.cc:515] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:04.146072  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:00:04.150104  2298 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:04.143975  1747 raft_consensus.cc:2468] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 6.
I20260709 13:00:04.157698  3521 leader_election.cc:290] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:04.133422  4496 raft_consensus.cc:493] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:04.134306  3977 raft_consensus.cc:493] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:04.145076  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:04.163050  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:04.167043  1601 raft_consensus.cc:1275] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260709 13:00:04.134164  4438 consensus_queue.cc:1048] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.033s
I20260709 13:00:04.156737  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:04.169291  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:04.172261  4496 raft_consensus.cc:515] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:04.175855  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:04.176154  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:04.176458  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:00:04.184036  2334 raft_consensus.cc:2376] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 5. Re-sending same reply.
I20260709 13:00:04.181497  1569 leader_election.cc:304] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:00:04.176883  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:04.185405  1603 raft_consensus.cc:3060] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:00:04.183933  1745 raft_consensus.cc:2468] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 5.
I20260709 13:00:04.179234  1751 raft_consensus.cc:3060] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:00:04.138080  4478 raft_consensus.cc:515] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:04.187749  3521 consensus_queue.cc:1048] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:00:04.168524  4505 raft_consensus.cc:493] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:04.183290  2333 raft_consensus.cc:2468] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 5.
I20260709 13:00:04.191577  4070 raft_consensus.cc:493] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:00:04.160845  4262 raft_consensus.cc:493] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:04.195127  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:04.157279  4395 raft_consensus.cc:515] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:04.198866  4262 raft_consensus.cc:515] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:04.200119  4489 raft_consensus.cc:493] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:04.200353  4395 leader_election.cc:290] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:04.192983  1717 leader_election.cc:304] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:04.197860  2337 raft_consensus.cc:2468] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 6.
I20260709 13:00:04.203619  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:04.179411  4512 raft_consensus.cc:493] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:00:04.205950  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:04.204844  1752 raft_consensus.cc:2468] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 4.
I20260709 13:00:04.204231  2333 raft_consensus.cc:2468] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 5.
I20260709 13:00:04.208542  1601 raft_consensus.cc:3060] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Advancing to term 5
W20260709 13:00:04.201154  4246 consensus_meta.cc:287] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.506s	user 0.011s	sys 0.003s
I20260709 13:00:04.179023  3977 raft_consensus.cc:515] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:04.211337  1745 raft_consensus.cc:1240] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 4. Current term is 5. Ops: []
I20260709 13:00:04.215736  1717 leader_election.cc:304] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:04.182837  4244 consensus_queue.cc:1048] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.025s
I20260709 13:00:04.222402  3521 raft_consensus.cc:2749] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:00:04.225183  1717 leader_election.cc:304] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:04.202683  4390 raft_consensus.cc:493] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:04.207224  4436 raft_consensus.cc:493] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:04.223351  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:04.200651  4489 raft_consensus.cc:515] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:04.228804  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 55445.5us
I20260709 13:00:04.202054  4502 raft_consensus.cc:2749] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:00:04.235631  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:04.239526  3521 raft_consensus.cc:2749] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:00:04.246228  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:04.246901  1753 raft_consensus.cc:3060] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:00:04.244761  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:04.231403  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:00:04.201161  4070 raft_consensus.cc:515] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:04.245407  1569 leader_election.cc:304] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:00:04.256244  4070 leader_election.cc:290] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:04.218417  4262 leader_election.cc:290] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:04.258925  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:04.259661  4500 consensus_queue.cc:1048] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:00:04.260149  3977 leader_election.cc:290] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:04.261868  4500 consensus_queue.cc:1059] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 13:00:04.222219  4395 raft_consensus.cc:2749] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 13:00:04.252477  4509 raft_consensus.cc:493] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:04.264694  1603 raft_consensus.cc:2468] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 6.
I20260709 13:00:04.264767  4509 raft_consensus.cc:515] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:04.268428  4509 leader_election.cc:290] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:04.270356  4500 raft_consensus.cc:3055] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:00:04.273274  1569 leader_election.cc:304] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 13:00:04.246543  4310 leader_election.cc:290] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:04.238698  4390 raft_consensus.cc:515] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:04.306633  4246 raft_consensus.cc:493] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:04.316690  4246 raft_consensus.cc:515] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:04.317444  4390 leader_election.cc:290] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:04.318140  4365 leader_election.cc:290] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 5 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:04.320592  1745 raft_consensus.cc:2468] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 3.
I20260709 13:00:04.324327  2334 raft_consensus.cc:1275] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 5 index: 8. Preceding OpId from leader: term: 6 index: 9. (index mismatch)
I20260709 13:00:04.206380  4505 raft_consensus.cc:515] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:04.328069  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 102944us
I20260709 13:00:04.283322  4500 raft_consensus.cc:740] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:00:04.237980  4436 raft_consensus.cc:515] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:04.211990  4512 raft_consensus.cc:515] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:04.313035  4438 raft_consensus.cc:493] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:00:04.351331  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 66727.3us
I20260709 13:00:04.248420  2328 raft_consensus.cc:2468] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 7.
I20260709 13:00:04.355767  2333 raft_consensus.cc:2468] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 5.
I20260709 13:00:04.367372  1749 raft_consensus.cc:1240] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 4. Current term is 5. Ops: []
I20260709 13:00:04.356859  4496 leader_election.cc:290] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:04.362339  4246 leader_election.cc:290] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:04.353950  4495 consensus_queue.cc:1048] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.015s
I20260709 13:00:04.384512  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:04.391789  1679 raft_consensus.cc:3060] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Advancing to term 5
W20260709 13:00:04.382854  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 124534us
I20260709 13:00:04.351061  4438 raft_consensus.cc:515] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:04.388943  1674 raft_consensus.cc:1240] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 6. Current term is 7. Ops: []
W20260709 13:00:04.394768  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 79016.4us
W20260709 13:00:04.401834  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 66487.1us
I20260709 13:00:04.402516  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:04.403313  1605 raft_consensus.cc:2468] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 7.
W20260709 13:00:04.408869  4511 log.cc:927] Time spent T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.197s	user 0.001s	sys 0.001s
I20260709 13:00:04.409951  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:04.411315  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:04.411687  4438 leader_election.cc:290] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:04.413332  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:04.390347  4310 raft_consensus.cc:493] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:04.370316  4500 consensus_queue.cc:260] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:04.449620  1604 raft_consensus.cc:1240] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 5. Current term is 6. Ops: []
I20260709 13:00:04.450165  1749 raft_consensus.cc:1240] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 3. Current term is 4. Ops: []
I20260709 13:00:04.418819  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:04.514379  1753 raft_consensus.cc:2468] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 7.
I20260709 13:00:04.575289  2336 raft_consensus.cc:1240] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 5. Current term is 6. Ops: []
W20260709 13:00:04.535115  2301 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:04.536302  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:04.546818  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:04.448222  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:04.377171  4502 raft_consensus.cc:2804] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:00:04.433326  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:04.434823  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:04.580614  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:04.487620  1751 raft_consensus.cc:2468] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 6.
W20260709 13:00:04.518193  1646 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:04.569227  1601 raft_consensus.cc:2468] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 5.
W20260709 13:00:04.520030  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:04.550207  3521 raft_consensus.cc:493] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:04.518633  1679 raft_consensus.cc:2468] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 5.
W20260709 13:00:04.582108  2301 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:00:04.590569  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 51275.9us
I20260709 13:00:04.550937  1718 leader_election.cc:304] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:00:04.582387  4512 leader_election.cc:290] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 13:00:04.572883  2298 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:00:04.446619  4499 log.cc:927] Time spent T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.582s	user 0.006s	sys 0.009s
I20260709 13:00:04.568557  1598 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 489) took 1293 ms. Trace:
I20260709 13:00:04.582279  1644 leader_election.cc:304] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
W20260709 13:00:04.600078  1646 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:04.588743  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:04.613041  1754 raft_consensus.cc:2468] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 5.
I20260709 13:00:04.601157  3521 raft_consensus.cc:515] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:04.580870  1674 raft_consensus.cc:2468] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 6.
I20260709 13:00:04.603722  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:04.604166  1643 leader_election.cc:304] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:04.611013  1598 rpcz_store.cc:276] 0709 13:00:03.274886 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:03.383238 (+108352us) service_pool.cc:224] Handling call
0709 13:00:03.412914 (+ 29676us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:00:03.645705 (+232791us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:00:03.645758 (+    53us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:00:03.672840 (+ 27082us) log.cc:844] Serialized 52 byte log entry
0709 13:00:04.089066 (+416226us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:00:04.089131 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:04.089240 (+   109us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:00:04.535821 (+446581us) raft_consensus.cc:1685] finished
0709 13:00:04.535844 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:04.568466 (+ 32622us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":263795,"raft.run_cpu_time_us":133,"raft.run_wall_time_us":135,"thread_start_us":575822,"threads_started":2,"wal-append.queue_time_us":467541}
I20260709 13:00:04.632426  1598 raft_consensus.cc:1275] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260709 13:00:04.629473  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
W20260709 13:00:04.581578  2299 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:04.602114  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:04.636417  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:04.636473  1601 raft_consensus.cc:2468] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 3.
W20260709 13:00:04.461442  3978 consensus_meta.cc:287] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.695s	user 0.014s	sys 0.000s
I20260709 13:00:04.661309  1605 raft_consensus.cc:3060] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:00:04.661269  1602 raft_consensus.cc:1240] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 5. Current term is 6. Ops: []
I20260709 13:00:04.665315  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:04.640264  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:00:04.662541  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 75191.6us
W20260709 13:00:04.667905  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 73037.6us
I20260709 13:00:04.456396  4505 leader_election.cc:290] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:04.671573  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:04.671713  4502 raft_consensus.cc:697] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:00:04.443149  4496 consensus_queue.cc:1059] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.229s
I20260709 13:00:04.662057  4526 raft_consensus.cc:493] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:00:04.433715  4478 leader_election.cc:290] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:04.444183  4368 raft_consensus.cc:493] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:04.673077  1750 raft_consensus.cc:2410] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for term 7 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 4.
I20260709 13:00:04.661681  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:04.661675  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:04.590806  1680 raft_consensus.cc:2468] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 7.
W20260709 13:00:04.658051  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 107209us
I20260709 13:00:04.733049  1718 leader_election.cc:304] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:00:04.638062  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:04.734181  1679 raft_consensus.cc:2393] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 6: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:00:04.665635  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:04.735605  1678 raft_consensus.cc:2393] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 7: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:00:04.660562  2299 leader_election.cc:304] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:00:04.657598  3521 leader_election.cc:290] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:04.519357  4246 raft_consensus.cc:493] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:00:04.683820  1677 raft_consensus.cc:2468] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 5.
I20260709 13:00:04.414290  4518 raft_consensus.cc:493] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:00:04.770393  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:04.772008  1718 leader_election.cc:304] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:04.722679  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:04.723857  1599 raft_consensus.cc:2393] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 3: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:00:04.679917  1717 leader_election.cc:304] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:00:04.727720  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:04.550042  4489 leader_election.cc:290] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:04.777587  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:04.556991  4395 raft_consensus.cc:493] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
W20260709 13:00:04.724783  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 48360.4us
I20260709 13:00:04.659078  4512 raft_consensus.cc:493] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:04.738034  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:04.773286  1674 raft_consensus.cc:2468] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 7.
I20260709 13:00:04.774572  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:04.664705  4508 raft_consensus.cc:2764] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
W20260709 13:00:04.776230  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 115759us
I20260709 13:00:04.788693  1718 leader_election.cc:304] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 13:00:04.776588  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 202525us
I20260709 13:00:04.738034  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:00:04.785650  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 216240us
I20260709 13:00:04.785703  2334 raft_consensus.cc:2468] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 3.
I20260709 13:00:04.728387  3978 raft_consensus.cc:493] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:04.789234  2299 leader_election.cc:304] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:00:04.738910  4368 raft_consensus.cc:3060] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Advancing to term 7
W20260709 13:00:04.793111  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 113291us
I20260709 13:00:04.783970  2298 leader_election.cc:304] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:04.788051  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:04.691268  4070 consensus_queue.cc:1048] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.092s
I20260709 13:00:04.769559  4262 raft_consensus.cc:2749] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:00:04.675309  4310 raft_consensus.cc:515] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:04.792327  4510 raft_consensus.cc:2749] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 13:00:04.775143  4372 raft_consensus.cc:493] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:04.823176  1750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55848 (request call id 1922) took 1122 ms. Trace:
I20260709 13:00:04.776217  4438 raft_consensus.cc:493] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:04.791136  4478 raft_consensus.cc:2749] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260709 13:00:04.824944  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 45726.7us
I20260709 13:00:04.867023  2298 leader_election.cc:304] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:00:04.779853  4436 leader_election.cc:290] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:04.826584  3521 raft_consensus.cc:2804] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:00:04.829677  4395 raft_consensus.cc:515] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:04.834874  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:04.835356  4502 consensus_queue.cc:237] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:04.835810  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:04.836551  4535 raft_consensus.cc:493] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:04.778401  4246 raft_consensus.cc:515] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:04.843506  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:04.844300  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:04.846834  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:04.847070  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 8 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:04.847584  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:00:04.848505  4482 log.cc:927] Time spent T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 1.227s	user 0.012s	sys 0.018s
I20260709 13:00:04.775372  4526 raft_consensus.cc:515] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:04.853576  4509 raft_consensus.cc:493] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:00:04.853926  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:04.787880  4500 raft_consensus.cc:3060] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:00:04.858985  1750 rpcz_store.cc:276] 0709 13:00:03.700927 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:03.706584 (+  5657us) service_pool.cc:224] Handling call
0709 13:00:04.244591 (+538007us) spinlock_profiling.cc:243] Waited 57.7 ms on lock 0x7b58001ffe98. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f426303db71 00007f426303da22 00007f426bf56c09 00007f426b0b0ee0 00007f426b17a0ab 00007f426b17a131 00007f426b1280a0 00007f426b127984 00007f42646a26d4 00007f42646a2662
0709 13:00:04.809946 (+565355us) spinlock_profiling.cc:243] Waited 425 ms on lock 0x7b5400220954. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:00:04.823088 (+ 13142us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1109892864}
I20260709 13:00:04.790293  4201 raft_consensus.cc:2804] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:00:04.863332  1754 raft_consensus.cc:1275] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 5. (term mismatch)
I20260709 13:00:04.872814  1605 raft_consensus.cc:2468] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 6.
I20260709 13:00:04.864198  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:04.874577  4372 raft_consensus.cc:515] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:00:04.875659  4523 log.cc:927] Time spent T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.058s	user 0.000s	sys 0.000s
I20260709 13:00:04.875370  4438 raft_consensus.cc:515] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:04.801657  4512 raft_consensus.cc:515] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:04.823876  4534 raft_consensus.cc:2804] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260709 13:00:04.864951  4525 log.cc:927] Time spent T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.081s	user 0.001s	sys 0.000s
I20260709 13:00:04.809360  3978 raft_consensus.cc:515] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:04.816632  4505 raft_consensus.cc:2804] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:00:04.881899  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:04.988881  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:04.793773  4530 consensus_queue.cc:1059] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.211s
I20260709 13:00:04.957707  1603 raft_consensus.cc:1275] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 5 index: 8. (index mismatch)
I20260709 13:00:05.032930  1753 raft_consensus.cc:2468] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 4.
W20260709 13:00:04.887311  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 98129.9us
I20260709 13:00:05.035758  1749 raft_consensus.cc:2468] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 6.
I20260709 13:00:04.793284  4518 raft_consensus.cc:515] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:05.032158  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 428003us
I20260709 13:00:05.038071  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:00:05.038728  1646 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:05.029829  1754 pending_rounds.cc:85] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701: Aborting all ops after (but not including) 4
I20260709 13:00:05.039315  1754 pending_rounds.cc:107] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701: Aborting uncommitted NO_OP operation due to leader change: 6.5
I20260709 13:00:05.039579  1754 raft_consensus.cc:2889] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:00:05.030325  1597 raft_consensus.cc:2468] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 2.
W20260709 13:00:04.942080  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:00:05.045459  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 263624us
W20260709 13:00:04.884143  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 153087us
I20260709 13:00:05.034147  1747 raft_consensus.cc:2376] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 6. Re-sending same reply.
I20260709 13:00:05.047037  1718 leader_election.cc:304] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:00:05.048532  3521 raft_consensus.cc:697] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 5 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:00:05.108590  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:05.050072  1748 raft_consensus.cc:2410] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for term 8 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260709 13:00:05.108608  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:05.109794  1753 raft_consensus.cc:2468] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 4.
I20260709 13:00:05.111845  1718 leader_election.cc:304] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:00:04.881800  4310 leader_election.cc:290] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:00:05.108565  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 65195.7us
I20260709 13:00:04.857918  4365 raft_consensus.cc:2749] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 13:00:05.123803  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:04.990734  4495 raft_consensus.cc:3055] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:00:05.120523  1747 raft_consensus.cc:2393] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 6: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:00:05.120167  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:05.109257  2299 leader_election.cc:304] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:00:04.957230  4297 raft_consensus.cc:2804] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 13:00:05.123903  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:05.037396  4534 raft_consensus.cc:493] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:05.132835  1753 raft_consensus.cc:2468] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 5.
I20260709 13:00:04.958205  4489 raft_consensus.cc:2804] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260709 13:00:05.126559  1646 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:05.120074  1605 raft_consensus.cc:2468] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 5.
I20260709 13:00:05.130762  4495 raft_consensus.cc:740] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:00:05.135183  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:05.135890  1605 raft_consensus.cc:2468] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 6.
I20260709 13:00:05.135375  4495 consensus_queue.cc:260] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:05.136919  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:05.137017  4495 raft_consensus.cc:3060] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:00:05.138010  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:05.127827  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:05.139516  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:05.127997  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:05.124559  1750 raft_consensus.cc:2376] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 6. Re-sending same reply.
I20260709 13:00:05.034499  4505 raft_consensus.cc:493] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:05.142231  4505 raft_consensus.cc:3060] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:00:05.143813  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:05.107376  4509 raft_consensus.cc:515] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:05.144414  1751 raft_consensus.cc:3060] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:00:05.145234  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:05.145790  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:05.146768  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:05.146692  4200 consensus_queue.cc:1059] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 13:00:05.147535  1745 raft_consensus.cc:2468] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 4.
I20260709 13:00:05.107776  4478 raft_consensus.cc:2804] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:00:05.170921  1746 raft_consensus.cc:1275] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 13:00:05.047672  4535 raft_consensus.cc:515] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:05.154731  1598 raft_consensus.cc:2468] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 4.
I20260709 13:00:05.157419  1605 raft_consensus.cc:2468] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 5.
I20260709 13:00:05.133039  3521 consensus_queue.cc:237] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:05.174880  1749 raft_consensus.cc:3060] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:00:05.173910  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:00:05.176554  4523 log.cc:927] Time spent T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.112s	user 0.000s	sys 0.001s
I20260709 13:00:05.185539  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:05.187377  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:05.190584  1599 raft_consensus.cc:2393] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 5: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:00:05.194310  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:05.195257  1753 raft_consensus.cc:3060] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:00:05.132004  4201 raft_consensus.cc:697] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 5 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:00:05.171087  4478 raft_consensus.cc:493] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:05.208501  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:05.217384  1603 raft_consensus.cc:2393] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 6: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:00:05.207813  1754 raft_consensus.cc:2393] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 6: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:00:05.214622  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:05.154026  4489 raft_consensus.cc:493] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:05.233319  4489 raft_consensus.cc:3060] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:00:05.234263  4246 leader_election.cc:290] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:05.235666  4246 raft_consensus.cc:493] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:00:05.238320  4512 leader_election.cc:290] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:05.239189  4518 leader_election.cc:290] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:05.240374  4395 leader_election.cc:290] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:05.187623  4297 raft_consensus.cc:697] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 6 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:00:05.241153  4201 consensus_queue.cc:237] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:05.241644  4297 consensus_queue.cc:237] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:05.243091  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:05.243862  1754 raft_consensus.cc:2468] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 6.
I20260709 13:00:05.222990  2331 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39738 (request call id 662) took 2153 ms. Trace:
I20260709 13:00:05.224056  3978 leader_election.cc:290] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:05.226115  4526 leader_election.cc:290] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:05.228317  4538 raft_consensus.cc:2749] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:00:05.154246  4534 raft_consensus.cc:3060] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:00:05.177846  3977 raft_consensus.cc:2804] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:00:05.231464  2337 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39726 (request call id 787) took 1026 ms. Trace:
I20260709 13:00:05.274837  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:05.273640  2337 rpcz_store.cc:276] 0709 13:00:04.204896 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:04.205014 (+   118us) service_pool.cc:224] Handling call
0709 13:00:04.209062 (+  4048us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:00:04.223243 (+ 14181us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 13:00:04.347986 (+124743us) log.cc:844] Serialized 14 byte log entry
0709 13:00:04.598622 (+250636us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:00:04.733540 (+134918us) log.cc:844] Serialized 52 byte log entry
0709 13:00:04.733760 (+   220us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:00:04.733817 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:04.733896 (+    79us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:00:05.231173 (+497277us) raft_consensus.cc:1685] finished
0709 13:00:05.231200 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:05.231361 (+   161us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":11952,"prepare.queue_time_us":151021,"prepare.run_cpu_time_us":56,"prepare.run_wall_time_us":57,"raft.queue_time_us":92297,"raft.run_cpu_time_us":273,"raft.run_wall_time_us":7130,"thread_start_us":296448,"threads_started":2,"wal-append.queue_time_us":314567}
I20260709 13:00:05.173617  4496 raft_consensus.cc:2804] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:00:05.276261  4496 raft_consensus.cc:493] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:05.276309  4535 leader_election.cc:290] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:05.276798  4496 raft_consensus.cc:3060] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:00:05.215773  1599 raft_consensus.cc:1275] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 13:00:05.248731  1717 leader_election.cc:304] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:00:05.280682  1749 raft_consensus.cc:2468] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 6.
I20260709 13:00:05.281176  4395 raft_consensus.cc:2804] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:00:05.281621  4395 raft_consensus.cc:493] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:00:05.282016  4395 raft_consensus.cc:3060] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:00:05.282421  3978 raft_consensus.cc:493] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:05.282968  3978 raft_consensus.cc:515] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:05.285281  4505 raft_consensus.cc:515] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:05.287006  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:05.287849  1752 raft_consensus.cc:2393] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 4: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:00:05.288647  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
W20260709 13:00:05.290920  1673 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.553s	user 0.014s	sys 0.041s
I20260709 13:00:05.291652  1717 leader_election.cc:304] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:05.292514  4510 raft_consensus.cc:2749] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260709 13:00:05.294729  1673 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.557s	user 0.014s	sys 0.041s
I20260709 13:00:05.266474  1751 raft_consensus.cc:2468] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 6.
I20260709 13:00:05.278030  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:05.296751  1746 raft_consensus.cc:2393] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 5: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:00:05.297621  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:05.298375  1746 raft_consensus.cc:2393] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 8: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:00:05.299206  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:05.299695  1717 leader_election.cc:304] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 13:00:05.300299  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:05.300518  4510 raft_consensus.cc:2804] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:00:05.300915  4510 raft_consensus.cc:493] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:05.300940  1717 leader_election.cc:304] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:00:05.301442  4510 raft_consensus.cc:3060] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:00:05.303347  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:05.301427  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:05.305323  1596 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 492) took 1145 ms. Trace:
W20260709 13:00:05.308250  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 309329us
W20260709 13:00:05.310755  2301 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:00:05.311256  2301 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:05.311069  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:05.305794  1596 rpcz_store.cc:276] 0709 13:00:04.159917 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:04.160072 (+   155us) service_pool.cc:224] Handling call
0709 13:00:04.172839 (+ 12767us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:00:04.187012 (+ 14173us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 13:00:04.327062 (+140050us) log.cc:844] Serialized 14 byte log entry
0709 13:00:04.491073 (+164011us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:00:04.535042 (+ 43969us) log.cc:844] Serialized 52 byte log entry
0709 13:00:04.535226 (+   184us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:00:04.535502 (+   276us) log.cc:844] Serialized 14 byte log entry
0709 13:00:04.535705 (+   203us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:04.535788 (+    83us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:00:05.132873 (+597085us) spinlock_profiling.cc:243] Waited 2.34 ms on lock 0x7b18003c0460. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f4263075522 00007f42630777c1 00007f4263077779 00007f4263077709 00007f42630776d1 00007f42630769cd 000000000038f354 000000000038f199 00007f426305883a
0709 13:00:05.305099 (+172226us) raft_consensus.cc:1685] finished
0709 13:00:05.305127 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:05.305262 (+   135us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1615,"prepare.queue_time_us":196770,"prepare.run_cpu_time_us":107,"prepare.run_wall_time_us":109,"raft.queue_time_us":13470,"raft.run_cpu_time_us":508,"raft.run_wall_time_us":25828,"spinlock_wait_cycles":5375744,"thread_start_us":250611,"threads_started":2,"wal-append.queue_time_us":337084}
I20260709 13:00:05.311004  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:05.313112  1747 raft_consensus.cc:2393] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 6: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:00:05.288954  4368 raft_consensus.cc:515] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:05.239439  4543 raft_consensus.cc:493] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:00:05.316257  1747 raft_consensus.cc:1240] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 3. Current term is 4. Ops: []
W20260709 13:00:05.329897  1644 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.512s	user 0.016s	sys 0.025s
W20260709 13:00:05.331516  1644 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.514s	user 0.016s	sys 0.025s
I20260709 13:00:05.322365  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:05.286293  3978 leader_election.cc:290] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:05.245985  4478 raft_consensus.cc:3060] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:00:05.331792  1754 raft_consensus.cc:1240] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 6. Current term is 7. Ops: []
W20260709 13:00:05.338244  2301 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:00:05.333600  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 541596us
I20260709 13:00:05.304549  4372 leader_election.cc:290] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:05.314801  4365 raft_consensus.cc:493] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:00:05.236810  4246 raft_consensus.cc:515] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:05.343690  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 8 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:00:05.344079  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 192391us
I20260709 13:00:05.357149  1604 raft_consensus.cc:2468] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 6.
I20260709 13:00:05.351640  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:05.351009  1600 raft_consensus.cc:1240] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 6. Current term is 7. Ops: []
I20260709 13:00:05.359661  1599 raft_consensus.cc:2468] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 7.
W20260709 13:00:05.368117  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 257288us
I20260709 13:00:05.370412  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:05.371308  1672 raft_consensus.cc:2468] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 5.
I20260709 13:00:05.383024  1596 raft_consensus.cc:2393] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 6: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:00:05.320097  1753 raft_consensus.cc:2468] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 5.
I20260709 13:00:05.327098  4505 leader_election.cc:290] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 8 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:05.392385  1644 leader_election.cc:304] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:00:05.395937  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:05.396871  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:05.397670  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:05.400717  2333 raft_consensus.cc:2468] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 5.
I20260709 13:00:05.397038  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:05.398628  1644 leader_election.cc:304] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
W20260709 13:00:05.394038  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 324133us
I20260709 13:00:05.351009  3978 raft_consensus.cc:2804] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:00:05.402401  2299 leader_election.cc:304] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:00:05.402894  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:05.404608  1644 leader_election.cc:304] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:00:05.368381  4438 leader_election.cc:290] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:05.406122  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:05.403553  1678 raft_consensus.cc:2468] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 3.
I20260709 13:00:05.410177  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:05.411439  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:05.428222  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:05.429162  1672 raft_consensus.cc:2468] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 7.
W20260709 13:00:05.429404  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 126829us
I20260709 13:00:05.477856  1599 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07"
dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
mode: GRACEFUL
new_leader_uuid: "af520b67945d443b901bbeca37cc5701"
 from {username='slave'} at 127.0.0.1:47752
I20260709 13:00:05.478673  1599 raft_consensus.cc:606] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Received request to transfer leadership to TS af520b67945d443b901bbeca37cc5701
I20260709 13:00:05.479194  1599 raft_consensus.cc:614] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 13:00:05.479547  1717 leader_election.cc:304] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:00:05.501396  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:05.502516  2337 raft_consensus.cc:1240] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 4. Current term is 5. Ops: []
I20260709 13:00:05.503232  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:00:05.504583  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 122860us
I20260709 13:00:05.506611  4507 raft_consensus.cc:493] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:00:05.507675  4540 log.cc:927] Time spent T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.250s	user 0.000s	sys 0.001s
I20260709 13:00:05.507380  4507 raft_consensus.cc:515] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:05.509673  4542 log.cc:927] Time spent T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.106s	user 0.000s	sys 0.000s
I20260709 13:00:05.428897  4509 leader_election.cc:290] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:05.400137  4530 raft_consensus.cc:2804] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:00:05.369346  4365 raft_consensus.cc:515] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:05.524091  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:05.525453  1677 raft_consensus.cc:2410] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20260709 13:00:05.525323  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 45863.8us
I20260709 13:00:05.527110  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:05.409605  4538 raft_consensus.cc:2804] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:00:05.409762  4512 raft_consensus.cc:2804] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:00:05.406793  3978 raft_consensus.cc:493] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:00:05.536892  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:05.505957  4395 raft_consensus.cc:515] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:05.521711  1569 leader_election.cc:304] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:00:05.530680  3521 consensus_queue.cc:1048] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 13:00:05.527732  2334 raft_consensus.cc:1803] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 4 LEARNER]: voting while tombstoned based on last-logged opid 4.5
I20260709 13:00:05.536707  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:05.432534  4526 raft_consensus.cc:2804] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:00:05.520673  4530 raft_consensus.cc:493] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:05.541610  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:05.579886  2337 raft_consensus.cc:2410] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260709 13:00:05.340446  4543 raft_consensus.cc:515] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:05.520982  4507 leader_election.cc:290] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:05.428323  4505 consensus_queue.cc:1048] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.131s
I20260709 13:00:05.575775  2334 raft_consensus.cc:2468] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 4 LEARNER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 4.
I20260709 13:00:05.578019  1569 leader_election.cc:304] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 13:00:05.581172  1747 raft_consensus.cc:1240] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260709 13:00:05.587522  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:05.609105  1599 raft_consensus.cc:3060] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:00:05.538146  4538 raft_consensus.cc:493] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:05.593231  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:05.544749  3978 raft_consensus.cc:3060] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Advancing to term 6
W20260709 13:00:05.591897  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 86651.4us
I20260709 13:00:05.610498  3521 raft_consensus.cc:2804] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:00:05.546568  4489 raft_consensus.cc:515] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:05.680339  2333 raft_consensus.cc:1275] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 9 index: 9. (term mismatch)
I20260709 13:00:06.042423  2333 pending_rounds.cc:85] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1: Aborting all ops after (but not including) 8
I20260709 13:00:06.043053  2333 pending_rounds.cc:107] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1: Aborting uncommitted NO_OP operation due to leader change: 8.9
I20260709 13:00:06.043395  2333 raft_consensus.cc:2889] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 13:00:05.642100  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 64184.6us
I20260709 13:00:05.931569  1599 raft_consensus.cc:2468] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 8.
I20260709 13:00:06.047567  3521 raft_consensus.cc:493] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:00:05.687263  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 181939us
I20260709 13:00:06.048259  3521 raft_consensus.cc:3060] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:00:05.575641  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:06.049592  2330 raft_consensus.cc:2468] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 5.
I20260709 13:00:05.258515  2331 rpcz_store.cc:276] 0709 13:00:03.069889 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:03.075488 (+  5599us) service_pool.cc:224] Handling call
0709 13:00:03.075879 (+   391us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:00:03.077853 (+  1974us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:00:03.077893 (+    40us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:00:03.084806 (+  6913us) log.cc:844] Serialized 26 byte log entry
0709 13:00:03.527486 (+442680us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:00:03.530460 (+  2974us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:03.537395 (+  6935us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:00:04.729265 (+1191870us) spinlock_profiling.cc:243] Waited 1.73 ms on lock 0x7b4400022b4c. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f4262f9bc85 00007f4262f9c6a2 00007f4265d0e1db 00007f4265d09aaf 00007f4265cfd86e 00007f4265cfcbb6 00007f4265cfc7d8 00007f4265d06221 00007f4265d061d9 00007f4265d06169
0709 13:00:05.187117 (+457852us) raft_consensus.cc:1685] finished
0709 13:00:05.190145 (+  3028us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:05.222877 (+ 32732us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3987200,"thread_start_us":413493,"threads_started":1,"wal-append.queue_time_us":461734}
I20260709 13:00:05.615394  2335 raft_consensus.cc:2468] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 6.
I20260709 13:00:06.081626  1600 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 499) took 1343 ms. Trace:
I20260709 13:00:06.046285  1569 leader_election.cc:304] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:00:06.086233  1600 rpcz_store.cc:276] 0709 13:00:04.738642 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:04.738794 (+   152us) service_pool.cc:224] Handling call
0709 13:00:04.784547 (+ 45753us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:05.365680 (+581133us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:06.081010 (+715330us) spinlock_profiling.cc:243] Waited 566 ms on lock 0x7b5400071054. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:00:06.081513 (+   503us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1301976960}
I20260709 13:00:06.087543  1569 leader_election.cc:304] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:00:05.571457  4526 raft_consensus.cc:493] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:00:06.083726  1751 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.746s	user 0.032s	sys 0.057s
I20260709 13:00:05.585247  4547 raft_consensus.cc:493] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:05.589264  4530 raft_consensus.cc:3060] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Advancing to term 8
W20260709 13:00:06.093871  1751 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.757s	user 0.032s	sys 0.057s
W20260709 13:00:05.688951  4552 log.cc:927] Time spent T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.118s	user 0.000s	sys 0.001s
W20260709 13:00:06.083014  1746 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.606s	user 0.004s	sys 0.009s
W20260709 13:00:06.125633  1746 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.694s	user 0.005s	sys 0.009s
W20260709 13:00:06.097604  2299 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:05.575212  4502 raft_consensus.cc:2804] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:00:05.679255  4500 raft_consensus.cc:2804] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:00:06.041555  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:06.118643  4548 raft_consensus.cc:493] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:00:06.040577  3977 raft_consensus.cc:493] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:05.642197  4538 raft_consensus.cc:3060] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:00:06.084914  4368 leader_election.cc:290] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 7 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:06.055824  4246 leader_election.cc:290] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:05.571928  4512 raft_consensus.cc:697] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 5 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:00:06.103833  4526 raft_consensus.cc:3060] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Advancing to term 4
W20260709 13:00:06.168111  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 119531us
I20260709 13:00:06.170933  2328 raft_consensus.cc:2393] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 5: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:00:06.163519  3977 raft_consensus.cc:3060] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:00:05.588547  4372 consensus_queue.cc:1059] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.533s
I20260709 13:00:06.131386  4502 raft_consensus.cc:493] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:06.165477  4543 leader_election.cc:290] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:00:06.250123  1752 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.530s	user 0.003s	sys 0.006s
W20260709 13:00:06.242324  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 154890us
W20260709 13:00:06.253706  1602 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.669s	user 0.014s	sys 0.024s
W20260709 13:00:06.282366  1602 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.697s	user 0.015s	sys 0.024s
W20260709 13:00:06.284423  1717 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.756s	user 0.029s	sys 0.021s
W20260709 13:00:06.284861  1717 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.757s	user 0.029s	sys 0.021s
I20260709 13:00:06.283061  1569 leader_election.cc:304] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
W20260709 13:00:06.286222  1752 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.599s	user 0.003s	sys 0.006s
I20260709 13:00:06.162485  4500 raft_consensus.cc:493] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:06.241391  4518 raft_consensus.cc:493] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
W20260709 13:00:06.316282  1750 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.744s	user 0.002s	sys 0.008s
W20260709 13:00:06.316933  1750 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.744s	user 0.002s	sys 0.008s
I20260709 13:00:06.317909  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:06.318928  1599 raft_consensus.cc:3060] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:00:06.269057  4502 raft_consensus.cc:3060] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Advancing to term 6
W20260709 13:00:06.340592  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 54018.2us
I20260709 13:00:06.341398  1605 raft_consensus.cc:1240] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 5. Current term is 6. Ops: [5.6-5.6]
I20260709 13:00:06.120909  4547 raft_consensus.cc:515] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:06.359975  3521 raft_consensus.cc:515] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:06.304045  4500 raft_consensus.cc:3060] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:00:06.368955  1643 leader_election.cc:304] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:06.307135  4395 leader_election.cc:290] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 7 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 13:00:06.473387  1720 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.516s	user 0.028s	sys 0.028s
W20260709 13:00:06.473966  1720 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.756s	user 0.029s	sys 0.028s
I20260709 13:00:06.189139  4548 raft_consensus.cc:515] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:06.403138  4495 consensus_meta.cc:287] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 1.266s	user 0.017s	sys 0.002s
I20260709 13:00:06.377410  4550 raft_consensus.cc:493] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:06.384019  4372 raft_consensus.cc:2804] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:00:06.401260  4558 raft_consensus.cc:493] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:00:06.342774  4518 raft_consensus.cc:515] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:06.516541  3521 leader_election.cc:290] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 8 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:06.520946  4571 raft_consensus.cc:2749] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260709 13:00:06.521600  1597 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.937s	user 0.007s	sys 0.003s
I20260709 13:00:06.525045  1750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55840 (request call id 2132) took 1664 ms. Trace:
W20260709 13:00:06.556613  1597 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.972s	user 0.007s	sys 0.003s
I20260709 13:00:06.570238  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 8 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:06.571281  1601 raft_consensus.cc:3060] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:00:06.492444  4505 raft_consensus.cc:493] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:06.554533  1750 rpcz_store.cc:276] 0709 13:00:04.845836 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:05.142386 (+296550us) service_pool.cc:224] Handling call
0709 13:00:05.142847 (+   461us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:00:05.143323 (+   476us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:00:05.143353 (+    30us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:00:05.144073 (+   720us) log.cc:844] Serialized 52 byte log entry
0709 13:00:05.345007 (+200934us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:00:05.570522 (+225515us) log.cc:844] Serialized 14 byte log entry
0709 13:00:06.317377 (+746855us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:06.317507 (+   130us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:00:06.317576 (+    69us) raft_consensus.cc:1685] finished
0709 13:00:06.317606 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:06.510197 (+192591us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":943319,"prepare.run_cpu_time_us":93,"prepare.run_wall_time_us":21109,"raft.queue_time_us":555592,"raft.run_cpu_time_us":313,"raft.run_wall_time_us":315,"thread_start_us":1120109,"threads_started":3,"wal-append.queue_time_us":360685}
W20260709 13:00:06.473922  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 190931us
I20260709 13:00:06.597290  4548 leader_election.cc:290] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:06.585598  1644 leader_election.cc:304] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:00:06.602087  3521 raft_consensus.cc:493] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:00:06.602805  3521 raft_consensus.cc:515] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:06.503028  4372 raft_consensus.cc:493] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Starting leader election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:00:06.509053  4509 raft_consensus.cc:2749] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260709 13:00:06.629177  4070 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.509s	user 0.012s	sys 0.004s
I20260709 13:00:06.574342  4495 raft_consensus.cc:493] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:06.636843  3521 leader_election.cc:290] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:00:06.643850  1570 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.508s	user 0.006s	sys 0.021s
I20260709 13:00:06.571136  4547 leader_election.cc:290] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:00:06.647826  1570 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.512s	user 0.006s	sys 0.021s
I20260709 13:00:06.654078  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:06.655586  1599 raft_consensus.cc:2468] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 7.
I20260709 13:00:06.565627  4583 raft_consensus.cc:3055] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:00:06.619705  4372 raft_consensus.cc:3060] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Advancing to term 4
W20260709 13:00:06.638796  4070 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.519s	user 0.012s	sys 0.005s
W20260709 13:00:06.680084  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 110232us
I20260709 13:00:06.563053  4572 raft_consensus.cc:493] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:00:06.695544  4438 raft_consensus.cc:2804] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:00:06.606789  4505 raft_consensus.cc:515] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:00:06.660212  4534 consensus_meta.cc:287] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 1.387s	user 0.014s	sys 0.004s
I20260709 13:00:06.701483  4438 raft_consensus.cc:493] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:00:06.511950  4512 consensus_queue.cc:237] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:06.668107  4583 raft_consensus.cc:740] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:00:06.711760  4438 raft_consensus.cc:3060] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:00:06.723534  4578 raft_consensus.cc:493] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:00:06.724261  4578 raft_consensus.cc:515] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:06.730701  4570 raft_consensus.cc:493] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:00:06.731477  4570 raft_consensus.cc:515] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:06.732474  4438 raft_consensus.cc:515] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:06.734932  4438 leader_election.cc:290] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 6 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:00:06.736943  4555 log.cc:927] Time spent T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.448s	user 0.003s	sys 0.000s
W20260709 13:00:06.739610  4565 log.cc:927] Time spent T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.228s	user 0.000s	sys 0.001s
W20260709 13:00:06.743916  4556 log.cc:927] Time spent T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.590s	user 0.002s	sys 0.000s
I20260709 13:00:06.653184  4495 raft_consensus.cc:515] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:06.512774  4550 raft_consensus.cc:515] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:06.521767  4558 raft_consensus.cc:515] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:06.692919  4547 raft_consensus.cc:2764] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Leader pre-election decision vote started in defunct term 7: won
I20260709 13:00:06.761531  4495 leader_election.cc:290] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:00:06.766593  4559 log.cc:927] Time spent T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.648s	user 0.002s	sys 0.001s
I20260709 13:00:06.719131  4534 raft_consensus.cc:515] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:06.778417  4534 leader_election.cc:290] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 6 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:06.787392  1501 catalog_manager.cc:5697] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d reported cstate change: term changed from 4 to 7. New cstate: current_term: 7 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: HEALTHY } } }
I20260709 13:00:06.789345  1501 catalog_manager.cc:5697] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } }
I20260709 13:00:06.779925  1717 leader_election.cc:304] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
W20260709 13:00:06.780494  1571 leader_election.cc:336] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Timed out: RequestConsensusVote RPC to 127.0.107.195:40693 timed out after 1.633s (SENT)
I20260709 13:00:06.794461  1571 leader_election.cc:304] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:00:06.783517  4505 leader_election.cc:290] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:06.803421  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:06.811133  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:06.806248  1717 leader_election.cc:304] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:00:06.811905  1750 raft_consensus.cc:2468] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 5.
W20260709 13:00:06.837631  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 279272us
W20260709 13:00:06.809326  4478 consensus_meta.cc:287] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 1.441s	user 0.006s	sys 0.012s
W20260709 13:00:06.809695  4510 consensus_meta.cc:287] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 1.508s	user 0.008s	sys 0.015s
I20260709 13:00:06.831111  1501 catalog_manager.cc:5697] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d reported cstate change: term changed from 3 to 4, leader changed from 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) to 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196). New cstate: current_term: 4 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
W20260709 13:00:06.863032  2301 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:00:06.926618  2301 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:00:06.927289  2301 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:06.979210  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:06.980579  1750 raft_consensus.cc:2468] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 3.
W20260709 13:00:06.982126  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 202645us
W20260709 13:00:06.978696  2301 leader_election.cc:336] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Timed out: RequestConsensusVote RPC to 127.0.107.195:40693 timed out after 2.006s (SENT)
I20260709 13:00:06.983815  2301 leader_election.cc:304] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
W20260709 13:00:06.985630  1643 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.582s	user 0.014s	sys 0.023s
W20260709 13:00:06.986011  1643 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.602s	user 0.014s	sys 0.023s
I20260709 13:00:06.907922  4505 raft_consensus.cc:2804] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:00:06.987130  1601 raft_consensus.cc:2468] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 8.
W20260709 13:00:06.987354  1568 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.688s	user 0.023s	sys 0.014s
W20260709 13:00:06.989782  1568 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.691s	user 0.023s	sys 0.014s
W20260709 13:00:06.988500  1571 leader_election.cc:336] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Timed out: RequestConsensusVote RPC to 127.0.107.195:40693 timed out after 1.718s (SENT)
W20260709 13:00:06.986685  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 617760us
I20260709 13:00:06.990360  1571 leader_election.cc:304] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:00:06.971190  1501 catalog_manager.cc:5697] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d reported cstate change: config changed from index -1 to 3, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added, no longer has a pending config: VOTER 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193), VOTER 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194), VOTER af520b67945d443b901bbeca37cc5701 (127.0.107.195), NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196). New cstate: current_term: 4 leader_uuid: "af520b67945d443b901bbeca37cc5701" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } } }
I20260709 13:00:06.990087  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:06.975422  4570 leader_election.cc:290] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:06.707160  4572 raft_consensus.cc:515] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:00:06.985855  4538 consensus_meta.cc:287] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.741s	user 0.020s	sys 0.001s
I20260709 13:00:06.998234  4569 raft_consensus.cc:493] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:06.973229  4507 raft_consensus.cc:493] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:00:06.998878  1569 leader_election.cc:304] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
W20260709 13:00:06.986105  4496 consensus_meta.cc:287] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 1.709s	user 0.014s	sys 0.015s
I20260709 13:00:06.993396  1643 leader_election.cc:304] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:00:07.001976  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:06.983074  4547 raft_consensus.cc:2749] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 13:00:06.981217  4578 leader_election.cc:290] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:00:07.003228  4500 consensus_meta.cc:287] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.635s	user 0.014s	sys 0.000s
I20260709 13:00:06.943528  4478 raft_consensus.cc:515] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:07.006592  1501 catalog_manager.cc:5697] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 13:00:07.014729  1643 leader_election.cc:304] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:00:07.000305  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:00:06.962065  4510 raft_consensus.cc:515] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:06.999010  4534 consensus_queue.cc:1048] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.037s
I20260709 13:00:07.054322  4262 raft_consensus.cc:3055] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:00:07.054885  4262 raft_consensus.cc:740] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:00:07.055699  4510 leader_election.cc:290] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:07.055838  4580 raft_consensus.cc:2804] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Leader election won for term 6
I20260709 13:00:07.056569  4580 raft_consensus.cc:697] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:00:07.057899  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:07.006755  4518 leader_election.cc:290] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:07.066282  1501 catalog_manager.cc:5697] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d reported cstate change: term changed from 2 to 6, leader changed from 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194) to 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196). New cstate: current_term: 6 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 13:00:07.067796  1753 raft_consensus.cc:2368] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 for earlier term 5. Current term is 6.
I20260709 13:00:07.068962  1601 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23"
dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
mode: GRACEFUL
new_leader_uuid: "af520b67945d443b901bbeca37cc5701"
 from {username='slave'} at 127.0.0.1:47752
I20260709 13:00:07.069662  1601 raft_consensus.cc:606] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Received request to transfer leadership to TS af520b67945d443b901bbeca37cc5701
I20260709 13:00:07.070031  1601 raft_consensus.cc:614] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260709 13:00:07.076423  1500 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35444 (request call id 259) took 12923 ms (client timeout 15000 ms). Trace:
I20260709 13:00:07.076676  4587 raft_consensus.cc:493] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:00:07.077343  4587 raft_consensus.cc:515] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:07.078768  4574 raft_consensus.cc:2804] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 13:00:07.079421  4574 raft_consensus.cc:697] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 8 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:00:07.065362  4495 raft_consensus.cc:2749] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:00:07.080391  4574 consensus_queue.cc:237] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:07.019622  4246 consensus_queue.cc:1059] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.049s
I20260709 13:00:07.087343  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:07.019915  4496 raft_consensus.cc:515] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:07.020118  4507 raft_consensus.cc:515] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:07.075207  4526 consensus_meta.cc:287] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.791s	user 0.016s	sys 0.001s
I20260709 13:00:07.012147  4538 raft_consensus.cc:515] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:07.035791  4500 raft_consensus.cc:515] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:07.095988  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:07.087038  4200 raft_consensus.cc:493] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:00:07.098344   431 auto_leader_rebalancer.cc:452] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 13:00:07.100548  4538 leader_election.cc:290] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 7 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:00:07.076978  1500 rpcz_store.cc:269] 0709 12:59:54.088399 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:54.089067 (+   668us) service_pool.cc:224] Handling call
0709 13:00:07.003675 (+12914608us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:59:59.845047 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 13:00:00.038796 (+193749us) write_op.cc:432] Acquiring schema lock in shared mode
0709 13:00:00.038855 (+    59us) write_op.cc:435] Acquired schema lock
0709 13:00:00.038887 (+    32us) tablet.cc:662] Decoding operations
0709 13:00:00.041163 (+  2276us) write_op.cc:620] Acquiring the partition lock for write op
0709 13:00:00.041241 (+    78us) write_op.cc:641] Partition lock acquired for write op
0709 13:00:00.041284 (+    43us) tablet.cc:685] Acquiring locks for 14 operations
0709 13:00:00.206365 (+165081us) tablet.cc:701] Row locks acquired
0709 13:00:00.216514 (+ 10149us) write_op.cc:260] PREPARE: finished
0709 13:00:00.292444 (+ 75930us) write_op.cc:270] Start()
0709 13:00:00.306055 (+ 13611us) write_op.cc:276] Timestamp: P: 1783602000283589 usec, L: 0
0709 13:00:00.308270 (+  2215us) op_driver.cc:348] REPLICATION: starting
0709 13:00:00.381533 (+ 73263us) log.cc:844] Serialized 5546 byte log entry
0709 13:00:01.211520 (+829987us) op_driver.cc:464] REPLICATION: finished
0709 13:00:01.502264 (+290744us) write_op.cc:301] APPLY: starting
0709 13:00:01.529519 (+ 27255us) tablet.cc:1367] starting BulkCheckPresence
0709 13:00:01.590616 (+ 61097us) tablet.cc:1370] finished BulkCheckPresence
0709 13:00:01.590764 (+   148us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 13:00:01.677298 (+ 86534us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 13:00:01.677779 (+   481us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=14
0709 13:00:01.677852 (+    73us) write_op.cc:312] APPLY: finished
0709 13:00:01.678388 (+   536us) log.cc:844] Serialized 101 byte log entry
0709 13:00:01.700326 (+ 21938us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:00:01.730616 (+ 30290us) write_op.cc:454] Released schema lock
0709 13:00:01.792506 (+ 61890us) write_op.cc:341] FINISH: Updating metrics
0709 13:00:01.832931 (+ 40425us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:00:01.834659 (+  1728us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":55563,"reported_tablets":31,"child_traces":[["op",{"apply.queue_time_us":255373,"apply.run_cpu_time_us":6513,"apply.run_wall_time_us":321748,"num_ops":14,"prepare.queue_time_us":272031,"prepare.run_cpu_time_us":18287,"prepare.run_wall_time_us":726939,"raft.queue_time_us":137487,"raft.run_cpu_time_us":25685,"raft.run_wall_time_us":190949,"replication_time_us":885991,"thread_start_us":560877,"threads_started":4,"wal-append.queue_time_us":417542,"wal-append.run_cpu_time_us":40015,"wal-append.run_wall_time_us":1948345}]]}
I20260709 13:00:07.101357  4601 raft_consensus.cc:2749] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:00:07.101639  4598 raft_consensus.cc:493] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:07.102985  4551 consensus_queue.cc:1048] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.039s
I20260709 13:00:07.103888  4507 leader_election.cc:290] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:07.098062  1501 catalog_manager.cc:5697] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 13:00:07.106083  4500 leader_election.cc:290] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 6 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:07.105521  1501 catalog_manager.cc:5697] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d reported cstate change: term changed from 2 to 4, leader changed from 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) to 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194). New cstate: current_term: 4 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } }
I20260709 13:00:07.106554  4564 raft_consensus.cc:2804] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:00:07.106833  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:07.107087  4564 raft_consensus.cc:493] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:07.099011  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:07.074925  4591 raft_consensus.cc:2749] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 13:00:07.016502  4569 raft_consensus.cc:515] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:07.099359  4573 log.cc:927] Time spent T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.119s	user 0.000s	sys 0.000s
I20260709 13:00:07.109004  4478 leader_election.cc:290] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 6 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:07.087430  4561 consensus_queue.cc:1059] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.280s
W20260709 13:00:07.109275  3978 consensus_meta.cc:287] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 1.000s	user 0.010s	sys 0.006s
I20260709 13:00:07.100658  4589 raft_consensus.cc:493] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:07.109939  1752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43156 (request call id 486) took 1677 ms. Trace:
I20260709 13:00:07.099462  4200 raft_consensus.cc:515] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:07.111310  4496 leader_election.cc:290] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 8 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 13:00:07.112876  4530 consensus_meta.cc:287] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.949s	user 0.007s	sys 0.007s
I20260709 13:00:07.107466  4564 raft_consensus.cc:3060] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:00:07.112918  4580 consensus_queue.cc:237] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:07.113488  4530 raft_consensus.cc:515] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:07.002765  4583 consensus_queue.cc:260] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:07.114859  1751 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55840 (request call id 2151) took 1836 ms. Trace:
I20260709 13:00:07.110410  1752 rpcz_store.cc:276] 0709 13:00:05.432207 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:05.432531 (+   324us) service_pool.cc:224] Handling call
0709 13:00:05.502615 (+ 70084us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:00:05.527657 (+ 25042us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:00:05.527695 (+    38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:00:05.540391 (+ 12696us) log.cc:844] Serialized 26 byte log entry
0709 13:00:06.286852 (+746461us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:00:06.286909 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:06.330554 (+ 43645us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:00:07.109710 (+779156us) raft_consensus.cc:1685] finished
0709 13:00:07.109731 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:07.109869 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":239492,"raft.queue_time_us":103608,"raft.run_cpu_time_us":296,"raft.run_wall_time_us":148237,"thread_start_us":599161,"threads_started":1,"wal-append.queue_time_us":1055089}
I20260709 13:00:07.116182  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:07.117600  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:07.098793  4503 raft_consensus.cc:493] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
W20260709 13:00:07.118425  1753 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43156 (request call id 485) took 1723 ms (client timeout 2241 ms). Trace:
I20260709 13:00:07.118716  4503 raft_consensus.cc:515] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:07.103534   431 auto_leader_rebalancer.cc:527] All tables' leader rebalancing finished this round
I20260709 13:00:07.119813  4526 raft_consensus.cc:515] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:07.118295  1752 raft_consensus.cc:2468] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 5.
I20260709 13:00:07.121346  1676 raft_consensus.cc:2393] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 4: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:00:07.117656  4558 leader_election.cc:290] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 13:00:07.124344  4554 log.cc:927] Time spent T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 1.106s	user 0.006s	sys 0.001s
W20260709 13:00:07.116449  3977 consensus_meta.cc:287] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.819s	user 0.013s	sys 0.000s
I20260709 13:00:07.107450  1501 catalog_manager.cc:5697] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: HEALTHY } } }
W20260709 13:00:07.118888  1753 rpcz_store.cc:269] 0709 13:00:05.395515 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:05.395659 (+   144us) service_pool.cc:224] Handling call
0709 13:00:07.118193 (+1722534us) spinlock_profiling.cc:243] Waited 1.66 s on lock 0x7b54003237d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:00:07.118362 (+   169us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3821686528}
I20260709 13:00:07.169092  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:07.116225  4246 raft_consensus.cc:3055] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:00:07.124497  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:07.131529  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 8 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:00:07.175024  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 196330us
I20260709 13:00:07.110394  4589 raft_consensus.cc:515] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:07.156791  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:07.115262  1751 rpcz_store.cc:276] 0709 13:00:05.278583 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:05.315367 (+ 36784us) service_pool.cc:224] Handling call
0709 13:00:05.315770 (+   403us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:00:06.129643 (+813873us) raft_consensus.cc:1534] Early marking committed up to index 3
0709 13:00:06.320284 (+190641us) log.cc:844] Serialized 14 byte log entry
0709 13:00:06.630161 (+309877us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:00:06.630902 (+   741us) log.cc:844] Serialized 26 byte log entry
0709 13:00:06.631076 (+   174us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:00:06.631474 (+   398us) log.cc:844] Serialized 14 byte log entry
0709 13:00:06.631736 (+   262us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:06.631820 (+    84us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:00:07.114618 (+482798us) raft_consensus.cc:1685] finished
0709 13:00:07.114644 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:07.114793 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":36416,"prepare.queue_time_us":381736,"prepare.run_cpu_time_us":101,"prepare.run_wall_time_us":110,"raft.queue_time_us":107626,"raft.run_cpu_time_us":331,"raft.run_wall_time_us":52204,"spinlock_wait_cycles":14502656,"thread_start_us":1148793,"threads_started":3,"wal-append.queue_time_us":678804}
I20260709 13:00:07.156791  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:07.176170  1750 raft_consensus.cc:2376] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 3. Re-sending same reply.
I20260709 13:00:07.176501  1675 raft_consensus.cc:2468] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 5.
I20260709 13:00:07.170264  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:07.176764  1598 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40154 (request call id 1855) took 2308 ms. Trace:
I20260709 13:00:07.177456  1679 raft_consensus.cc:2468] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 4.
I20260709 13:00:07.175310  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:07.119558  4200 leader_election.cc:290] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:07.099736  4563 raft_consensus.cc:493] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:00:07.188069  1646 connection.cc:256] RPC call timeout handler was delayed by -1.76912672200001s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2068s for a response
I20260709 13:00:07.174466  1752 raft_consensus.cc:2376] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 7. Re-sending same reply.
I20260709 13:00:07.190557  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:07.191378  1675 raft_consensus.cc:3060] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:00:07.183494  1674 raft_consensus.cc:2410] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for term 8 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
W20260709 13:00:07.184630  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 196128us
W20260709 13:00:07.192689  1571 leader_election.cc:336] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Timed out: RequestConsensusVote RPC to 127.0.107.195:40693 timed out after 1.722s (SENT)
I20260709 13:00:07.193120  1571 leader_election.cc:304] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:00:07.193225  1501 catalog_manager.cc:5697] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d reported cstate change: term changed from 2 to 3, leader changed from 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) to 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194). New cstate: current_term: 3 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:00:07.166642  4526 leader_election.cc:290] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 4 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:00:07.184878  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 2.06418e+06us
I20260709 13:00:07.102367  4598 raft_consensus.cc:515] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:07.200567  2298 leader_election.cc:304] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:07.201913  2298 leader_election.cc:304] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:00:07.177709  1598 rpcz_store.cc:276] 0709 13:00:04.868245 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:05.214662 (+346417us) service_pool.cc:224] Handling call
0709 13:00:05.257174 (+ 42512us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:00:05.257735 (+   561us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:00:05.257769 (+    34us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:00:05.272227 (+ 14458us) log.cc:844] Serialized 52 byte log entry
0709 13:00:06.638698 (+1366471us) spinlock_profiling.cc:243] Waited 18.3 ms on lock 0x7b18003be1e0. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f4263057e07 00007f426beb2ab9 00007f426307469b 00007f426307293f 00007f4263074c02 00007f4265cfc20b 00007f4265d01d23 00007f4265d020d1 00007f426611da0f
0709 13:00:06.982540 (+343842us) raft_consensus.cc:1645] Marking committed up to 3
0709 13:00:07.000642 (+ 18102us) log.cc:844] Serialized 14 byte log entry
0709 13:00:07.165904 (+165262us) log.cc:844] Serialized 14 byte log entry
0709 13:00:07.166225 (+   321us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:07.166772 (+   547us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:00:07.176457 (+  9685us) raft_consensus.cc:1685] finished
0709 13:00:07.176486 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:07.176680 (+   194us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":12003,"prepare.queue_time_us":18031,"prepare.run_cpu_time_us":7715,"prepare.run_wall_time_us":14196,"raft.queue_time_us":76253,"raft.run_cpu_time_us":496,"raft.run_wall_time_us":40250,"spinlock_wait_cycles":59466368,"thread_start_us":492519,"threads_started":2,"wal-append.queue_time_us":1799077}
I20260709 13:00:07.203824  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:07.118893  4372 raft_consensus.cc:515] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:07.174062  4246 raft_consensus.cc:740] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:00:07.156243  3978 raft_consensus.cc:515] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:07.172789  4505 raft_consensus.cc:697] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 7 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:00:07.206578  1501 catalog_manager.cc:5697] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d reported cstate change: term changed from 3 to 4, leader changed from 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) to 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194). New cstate: current_term: 4 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: HEALTHY } } }
W20260709 13:00:07.200075  4577 log.cc:927] Time spent T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.139s	user 0.001s	sys 0.000s
W20260709 13:00:07.197528  4502 consensus_meta.cc:287] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.733s	user 0.009s	sys 0.004s
I20260709 13:00:07.187368  4589 leader_election.cc:290] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:07.172853  3977 raft_consensus.cc:515] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:07.225731  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:07.225251  4502 raft_consensus.cc:515] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:07.227823  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:07.241380  1601 raft_consensus.cc:3060] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:00:07.244194  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:07.245157  1599 raft_consensus.cc:2468] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 7.
I20260709 13:00:07.209594  4496 raft_consensus.cc:2749] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:00:07.231014  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:07.232234  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:07.250257  1751 raft_consensus.cc:2368] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 for earlier term 5. Current term is 6.
I20260709 13:00:07.247771  1598 raft_consensus.cc:2468] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 4.
I20260709 13:00:07.241250  2301 leader_election.cc:304] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:00:07.241868  4591 raft_consensus.cc:493] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:07.234742  4505 consensus_queue.cc:237] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:07.256109  4591 raft_consensus.cc:515] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:07.257129  4500 consensus_queue.cc:1059] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:00:07.255913  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:07.258849  1600 raft_consensus.cc:2468] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 4.
I20260709 13:00:07.230394  4598 leader_election.cc:290] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:07.258306  4500 raft_consensus.cc:3055] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:00:07.262248  1601 raft_consensus.cc:2468] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 5.
I20260709 13:00:07.263628  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:00:07.272977  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 83989.2us
I20260709 13:00:07.277824  1675 raft_consensus.cc:2468] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 8.
I20260709 13:00:07.279000  1603 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40154 (request call id 1859) took 2322 ms. Trace:
W20260709 13:00:07.279464  1646 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:07.280299  1603 rpcz_store.cc:276] 0709 13:00:04.956616 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:05.224918 (+268302us) service_pool.cc:224] Handling call
0709 13:00:05.225756 (+   838us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:00:05.226642 (+   886us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:00:05.226702 (+    60us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:00:05.227691 (+   989us) log.cc:844] Serialized 26 byte log entry
0709 13:00:05.410091 (+182400us) raft_consensus.cc:1645] Marking committed up to 1
0709 13:00:05.410134 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:05.410204 (+    70us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:00:07.123413 (+1713209us) spinlock_profiling.cc:243] Waited 643 ms on lock 0x7b4400022b4c. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f4262f9bc85 00007f4262f9c6a2 00007f4265d0e1db 00007f4265d09aaf 00007f4265cfd86e 00007f4265cfcbb6 00007f4265cfc7d8 00007f4265d06221 00007f4265d061d9 00007f4265d06169
0709 13:00:07.262772 (+139359us) raft_consensus.cc:1685] finished
0709 13:00:07.262798 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:07.278905 (+ 16107us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":4398,"raft.run_cpu_time_us":108,"raft.run_wall_time_us":109,"spinlock_wait_cycles":1479847680,"thread_start_us":180969,"threads_started":1,"wal-append.queue_time_us":462000}
I20260709 13:00:07.282476  1717 leader_election.cc:304] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:00:07.283691  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:07.228761  4507 raft_consensus.cc:2804] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:00:07.285652  2301 leader_election.cc:304] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:00:07.292088  1717 leader_election.cc:304] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:00:07.230449  3978 leader_election.cc:290] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 6 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 13:00:07.294129  1646 connection.cc:256] RPC call timeout handler was delayed by -1.596978871000033s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2223s for a response
W20260709 13:00:07.294556  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 76456.6us
I20260709 13:00:07.294659  1747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43156 (request call id 489) took 1652 ms. Trace:
I20260709 13:00:07.233016  4365 leader_election.cc:290] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:07.231570  4502 leader_election.cc:290] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 6 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:07.226531  4603 raft_consensus.cc:2749] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 13:00:07.239614  4587 leader_election.cc:290] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:07.239456  4262 consensus_queue.cc:260] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:07.300808  1747 rpcz_store.cc:276] 0709 13:00:05.642181 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:05.679406 (+ 37225us) service_pool.cc:224] Handling call
0709 13:00:05.679756 (+   350us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:00:05.680226 (+   470us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:00:05.680254 (+    28us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:00:06.041217 (+360963us) log.cc:844] Serialized 52 byte log entry
0709 13:00:06.250746 (+209529us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:00:06.250795 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:06.250884 (+    89us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:00:07.294407 (+1043523us) raft_consensus.cc:1685] finished
0709 13:00:07.294429 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:07.294586 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":438974,"raft.run_cpu_time_us":86,"raft.run_wall_time_us":86,"thread_start_us":209031,"threads_started":1,"wal-append.queue_time_us":947780}
I20260709 13:00:07.297258  1717 leader_election.cc:304] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:00:07.305981  1717 leader_election.cc:304] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:00:07.307138  1717 leader_election.cc:304] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
W20260709 13:00:07.301244  1646 connection.cc:256] RPC call timeout handler was delayed by -1.2030983230000629s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1898s for a response
W20260709 13:00:07.308306  1646 connection.cc:256] RPC call timeout handler was delayed by -1.1592929170000161s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1503s for a response
I20260709 13:00:07.198282  4563 raft_consensus.cc:515] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:07.303835  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:07.265304  4500 raft_consensus.cc:740] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
W20260709 13:00:07.308900  1646 connection.cc:256] RPC call timeout handler was delayed by -1.0638929170000324s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16090000000000002s for a response
I20260709 13:00:07.340159  2301 leader_election.cc:400] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 5 pre-election: Vote denied by peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693) with higher term. Message: Invalid argument: T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 for earlier term 5. Current term is 6.
I20260709 13:00:07.299721  4598 raft_consensus.cc:2804] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 13:00:07.303661  4478 raft_consensus.cc:2804] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:00:07.310735  3978 raft_consensus.cc:2804] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:00:07.349774  1747 raft_consensus.cc:2468] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 7.
I20260709 13:00:07.295872  4605 raft_consensus.cc:2804] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:00:07.310921  4578 raft_consensus.cc:2804] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260709 13:00:07.361584  1604 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36462 (request call id 501) took 2553 ms (client timeout 1920 ms). Trace:
W20260709 13:00:07.362076  1604 rpcz_store.cc:269] 0709 13:00:04.808605 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:04.823386 (+ 14781us) service_pool.cc:224] Handling call
0709 13:00:07.361347 (+2537961us) spinlock_profiling.cc:243] Waited 322 ms on lock 0x7b540029a9d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:00:07.361491 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":741310336}
W20260709 13:00:07.365998  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:07.296221  4507 raft_consensus.cc:493] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:07.348152  4512 raft_consensus.cc:2804] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:00:07.379314  1501 catalog_manager.cc:5697] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d reported cstate change: term changed from 4 to 5, leader changed from 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) to 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196). New cstate: current_term: 5 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 13:00:07.315902  4510 raft_consensus.cc:2804] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:00:07.359323  4605 raft_consensus.cc:493] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:07.417927  4605 raft_consensus.cc:3060] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:00:07.419477  3977 leader_election.cc:290] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 7 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:00:07.412143  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 110152us
I20260709 13:00:07.360477  3978 raft_consensus.cc:493] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:07.347005  4572 leader_election.cc:290] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:07.350605  4478 raft_consensus.cc:493] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:00:07.409826  4512 raft_consensus.cc:493] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:00:07.304008  4246 consensus_queue.cc:260] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:07.417186  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:00:07.425611  1501 catalog_manager.cc:5697] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d reported cstate change: term changed from 1 to 2, leader changed from 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) to 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196). New cstate: current_term: 2 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 13:00:07.373914  4578 raft_consensus.cc:493] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
W20260709 13:00:07.429283  2301 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:00:07.437986  2301 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:07.360603  4598 raft_consensus.cc:697] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 8 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:00:07.381687  4530 leader_election.cc:290] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 8 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:00:07.434257  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 74106.7us
I20260709 13:00:07.440634  1569 leader_election.cc:304] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:00:07.409593  4507 raft_consensus.cc:3060] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:00:07.361202  4365 consensus_queue.cc:1059] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 5, Time since last communication: 0.069s
I20260709 13:00:07.456859  1501 catalog_manager.cc:5697] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d reported cstate change: term changed from 2 to 3, leader changed from 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) to 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196). New cstate: current_term: 3 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
W20260709 13:00:07.459231  2298 connection.cc:256] RPC call timeout handler was delayed by -1.2341090869999789s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.192s for a response
I20260709 13:00:07.461591  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:00:07.462558  4589 raft_consensus.cc:2749] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 13:00:07.465217  4563 leader_election.cc:290] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:00:07.466571  4608 log.cc:927] Time spent T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.119s	user 0.000s	sys 0.001s
I20260709 13:00:07.436952  4478 raft_consensus.cc:3060] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:00:07.430878  3978 raft_consensus.cc:3060] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:00:07.410379  4534 consensus_queue.cc:1059] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.043s
I20260709 13:00:07.445235  4512 raft_consensus.cc:3060] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:00:07.436976  4510 raft_consensus.cc:697] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 5 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:00:07.447098  4578 raft_consensus.cc:3060] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:00:07.471161  1501 catalog_manager.cc:5697] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d reported cstate change: term changed from 2 to 4, leader changed from 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) to 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194). New cstate: current_term: 4 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 13:00:07.416280  4551 consensus_queue.cc:1059] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.040s
I20260709 13:00:07.453781  4496 raft_consensus.cc:493] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:07.480377  1646 leader_election.cc:304] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 13:00:07.424543  4500 consensus_queue.cc:260] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } }
W20260709 13:00:07.485697  4568 log.cc:927] Time spent T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.847s	user 0.006s	sys 0.004s
I20260709 13:00:07.486864  4598 consensus_queue.cc:237] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:07.406533  4593 consensus_queue.cc:1059] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.094s
I20260709 13:00:07.445370  4518 consensus_queue.cc:1048] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.034s
I20260709 13:00:07.491573  3521 raft_consensus.cc:493] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:07.468392  4298 raft_consensus.cc:493] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:00:07.537315  4298 raft_consensus.cc:515] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:07.499541  1501 catalog_manager.cc:5697] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } }
I20260709 13:00:07.536068  3521 raft_consensus.cc:515] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:07.541626  1749 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55840 (request call id 2137) took 2495 ms (client timeout 1718 ms). Trace:
W20260709 13:00:07.541231  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 60726.7us
W20260709 13:00:07.542138  1749 rpcz_store.cc:269] 0709 13:00:05.045505 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:05.108140 (+ 62635us) service_pool.cc:224] Handling call
0709 13:00:07.540437 (+2432297us) spinlock_profiling.cc:243] Waited 28.4 ms on lock 0x7b54001d51d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:00:07.540582 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":65371392}
I20260709 13:00:07.547109  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:07.547851  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:07.551510  2332 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39738 (request call id 673) took 2028 ms. Trace:
I20260709 13:00:07.479362  4571 consensus_queue.cc:1059] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.058s
I20260709 13:00:07.552556  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:07.551854  2332 rpcz_store.cc:276] 0709 13:00:05.523012 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:05.575736 (+ 52724us) service_pool.cc:224] Handling call
0709 13:00:07.547668 (+1971932us) spinlock_profiling.cc:243] Waited 3.07 ms on lock 0x7b5800178598. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f426303db71 00007f426303da22 00007f426bf56c09 00007f426b0b0ee0 00007f426b17a0ab 00007f426b17a131 00007f426b1280a0 00007f426b127984 00007f42646a26d4 00007f42646a2662
0709 13:00:07.551436 (+  3768us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":7053184}
I20260709 13:00:07.560513  2337 raft_consensus.cc:2468] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 6.
I20260709 13:00:07.553059  1749 raft_consensus.cc:2468] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 5.
I20260709 13:00:07.554641  1501 catalog_manager.cc:5697] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 13:00:07.563469  1646 leader_election.cc:304] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 13:00:07.542408  4551 raft_consensus.cc:2804] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:00:07.597903  3521 leader_election.cc:290] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:07.574916  1501 catalog_manager.cc:5697] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } }
I20260709 13:00:07.577026  4530 raft_consensus.cc:2804] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:00:07.550077  4564 raft_consensus.cc:515] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:07.582903  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:00:07.533692  4510 consensus_queue.cc:237] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:07.603359  4530 raft_consensus.cc:493] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:07.605726  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:00:07.606956  4619 log.cc:927] Time spent T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.064s	user 0.000s	sys 0.000s
I20260709 13:00:07.607249  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:07.587124  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:07.584028  4298 leader_election.cc:290] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 13:00:07.592904  1676 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57092 (request call id 603) took 2114 ms (client timeout 1607 ms). Trace:
I20260709 13:00:07.594640  4507 raft_consensus.cc:515] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:07.600126  1646 leader_election.cc:304] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 13:00:07.605967  4530 raft_consensus.cc:3060] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:00:07.607923  2337 raft_consensus.cc:2468] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 5.
I20260709 13:00:07.498090  4496 raft_consensus.cc:515] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:07.608091  1678 raft_consensus.cc:1240] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260709 13:00:07.610529  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:07.614118  1749 raft_consensus.cc:2393] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 4: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:00:07.614800  2335 raft_consensus.cc:2393] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 4: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:00:07.616818  1754 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43156 (request call id 487) took 2076 ms. Trace:
I20260709 13:00:07.610785  4564 leader_election.cc:290] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 4 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:07.611105  4591 leader_election.cc:290] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:00:07.608495  1676 rpcz_store.cc:269] 0709 13:00:05.478398 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:05.501040 (+ 22642us) service_pool.cc:224] Handling call
0709 13:00:07.592481 (+2091441us) spinlock_profiling.cc:243] Waited 1.62 s on lock 0x7b540025add4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:00:07.592760 (+   279us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3722698112}
I20260709 13:00:07.603631  4551 raft_consensus.cc:493] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:07.627506  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:07.628227  2337 raft_consensus.cc:2468] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 4.
I20260709 13:00:07.629199  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:00:07.620440  4571 raft_consensus.cc:2804] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:00:07.631278  2332 raft_consensus.cc:3060] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:00:07.624118  1646 leader_election.cc:304] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:00:07.632112  4551 raft_consensus.cc:3060] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:00:07.633761  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:07.630276  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:07.595890  4605 raft_consensus.cc:515] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:07.643664  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:07.644526  1749 raft_consensus.cc:2410] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for term 6 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260709 13:00:07.645223  4605 leader_election.cc:290] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 6 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:07.645391  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:07.617348  1754 rpcz_store.cc:276] 0709 13:00:05.540073 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:05.540241 (+   168us) service_pool.cc:224] Handling call
0709 13:00:05.540605 (+   364us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:00:05.572967 (+ 32362us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:00:05.573005 (+    38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:00:05.645317 (+ 72312us) log.cc:844] Serialized 26 byte log entry
0709 13:00:07.484742 (+1839425us) spinlock_profiling.cc:243] Waited 65.8 ms on lock 0x7b4400022b4c. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f4262f9bc85 00007f4262f9c6a2 00007f4265d0e1db 00007f4265d09aaf 00007f4265cfd86e 00007f4265cfcbb6 00007f4265cfc7d8 00007f4265d06221 00007f4265d061d9 00007f4265d06169
0709 13:00:07.616128 (+131386us) spinlock_profiling.cc:243] Waited 27.9 ms on lock 0x7b18001418e0. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f4265d0a4bc 00007f426305848b 00007f426beb2ab9 00007f426307469b 00007f426307293f 00007f4263074c02 00007f4265cfc20b 00007f4265d01d23 00007f4265d020d1
0709 13:00:07.616361 (+   233us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:00:07.616402 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:07.616489 (+    87us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:00:07.616555 (+    66us) raft_consensus.cc:1685] finished
0709 13:00:07.616574 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:07.616731 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":544536,"raft.run_cpu_time_us":271,"raft.run_wall_time_us":14322,"spinlock_wait_cycles":215532160,"thread_start_us":163504,"threads_started":1,"wal-append.queue_time_us":666647}
I20260709 13:00:07.646312  2337 raft_consensus.cc:2393] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 7: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:00:07.646953  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:07.648108  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:07.648816  1754 raft_consensus.cc:2376] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 6. Re-sending same reply.
I20260709 13:00:07.653620  1646 leader_election.cc:304] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 13:00:07.655074  1646 leader_election.cc:304] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:07.666483  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:07.631098  4496 leader_election.cc:290] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:07.631733  4571 raft_consensus.cc:493] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:07.680639  1646 leader_election.cc:304] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:00:07.683800  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:00:07.685878  2337 raft_consensus.cc:2393] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 8: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:00:07.687083  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:07.688577  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:07.689193  1749 raft_consensus.cc:2468] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 6.
I20260709 13:00:07.690459  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:00:07.690939  4507 leader_election.cc:290] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 4 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:07.692299  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:07.681358  1754 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43156 (request call id 507) took 1027 ms. Trace:
I20260709 13:00:07.694344  1754 rpcz_store.cc:276] 0709 13:00:06.653603 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:07.666046 (+1012443us) service_pool.cc:224] Handling call
0709 13:00:07.681280 (+ 15234us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:00:07.655419  4564 raft_consensus.cc:2749] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:00:07.696918  1749 raft_consensus.cc:2393] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 6: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:00:07.695598  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:07.691833  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:00:07.625530  4538 consensus_queue.cc:1048] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.055s
I20260709 13:00:07.657334  4593 raft_consensus.cc:2804] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:00:07.630425  4512 raft_consensus.cc:515] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:07.716094  4538 raft_consensus.cc:2749] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:00:07.717059  4489 leader_election.cc:290] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 7 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:07.657871  4534 consensus_queue.cc:1048] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.053s
I20260709 13:00:07.696169  4571 raft_consensus.cc:3060] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:00:07.727022  2333 raft_consensus.cc:1240] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 5. Current term is 6. Ops: []
I20260709 13:00:07.714203  2330 raft_consensus.cc:3060] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Advancing to term 7
I20260709 13:00:07.730792  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:00:07.726462  1646 leader_election.cc:336] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Timed out: RequestConsensusVote RPC to 127.0.107.195:40693 timed out after 2.419s (SENT)
I20260709 13:00:07.733421  1646 leader_election.cc:304] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:00:07.721239  4593 raft_consensus.cc:697] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 7 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:00:07.742733  2332 raft_consensus.cc:2468] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 6.
W20260709 13:00:07.744473  1748 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55848 (request call id 1948) took 2899 ms (client timeout 2419 ms). Trace:
I20260709 13:00:07.720364  4564 raft_consensus.cc:2804] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Leader election won for term 3
W20260709 13:00:07.744964  1748 rpcz_store.cc:269] 0709 13:00:04.844677 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:04.846602 (+  1925us) service_pool.cc:224] Handling call
0709 13:00:07.742602 (+2896000us) spinlock_profiling.cc:243] Waited 12 ms on lock 0x7b5400071554. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:00:07.744381 (+  1779us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":27496192}
I20260709 13:00:07.746407  1745 raft_consensus.cc:1275] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260709 13:00:07.754424  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:07.758750  1748 raft_consensus.cc:2393] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 6: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:00:07.740268  3978 raft_consensus.cc:515] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:07.752635  4539 raft_consensus.cc:2749] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:00:07.753981  4564 raft_consensus.cc:697] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 3 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:00:07.783322  4564 consensus_queue.cc:237] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:07.770540  4593 consensus_queue.cc:237] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:07.772305  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 10 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:07.756932  4530 raft_consensus.cc:515] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:07.796502  4530 leader_election.cc:290] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 7 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:07.797314  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:07.798466  1748 raft_consensus.cc:2468] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 3.
I20260709 13:00:07.800066  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:07.803242  1748 raft_consensus.cc:2468] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 4.
I20260709 13:00:07.804497  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:07.773649  4580 raft_consensus.cc:3055] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:00:07.767994  4601 raft_consensus.cc:493] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:07.807226  1749 raft_consensus.cc:2368] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 5. Current term is 6.
I20260709 13:00:07.808630  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:07.809348  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:07.810985  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:07.827016  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:07.820086  1748 raft_consensus.cc:2468] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 5.
I20260709 13:00:07.829624  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:07.830291  1747 raft_consensus.cc:2468] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 6.
I20260709 13:00:07.833992  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:07.835309  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:07.838145  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:07.806236  4578 raft_consensus.cc:515] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:07.820542  4580 raft_consensus.cc:740] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:00:07.852414  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:07.857019  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:07.858920  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:07.863848  1748 raft_consensus.cc:2368] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 for earlier term 5. Current term is 6.
I20260709 13:00:07.833529  4539 raft_consensus.cc:493] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:07.869423  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
W20260709 13:00:07.866353  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:07.834095  4512 leader_election.cc:290] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 6 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:07.871623  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:07.867153  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:07.873754  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:07.876009  1675 raft_consensus.cc:3060] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:00:07.877068  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:07.874224  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:07.878643  1750 raft_consensus.cc:2393] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 5: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:00:07.879083  1752 raft_consensus.cc:2393] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 6: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:00:07.876381  1674 raft_consensus.cc:2468] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 7.
W20260709 13:00:07.891026  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:07.896101  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:07.896463  1569 leader_election.cc:304] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:00:07.896973  1748 raft_consensus.cc:2393] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 4: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:00:07.897727  4537 raft_consensus.cc:2749] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:00:07.897994  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:07.899516  1748 raft_consensus.cc:1240] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 9 LEADER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 8. Current term is 9. Ops: []
I20260709 13:00:07.900141  1748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43156 (request call id 490) took 2050 ms. Trace:
I20260709 13:00:07.900517  1748 rpcz_store.cc:276] 0709 13:00:05.849800 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:07.899022 (+2049222us) service_pool.cc:224] Handling call
0709 13:00:07.899339 (+   317us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:07.899925 (+   586us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:07.900089 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:00:07.902547  1569 leader_election.cc:304] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:00:07.903481  1748 raft_consensus.cc:1275] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 13:00:07.831557  4601 raft_consensus.cc:515] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } }
I20260709 13:00:07.906023  1748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43156 (request call id 491) took 1935 ms. Trace:
I20260709 13:00:07.906504  1748 rpcz_store.cc:276] 0709 13:00:05.970454 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:07.901364 (+1930910us) service_pool.cc:224] Handling call
0709 13:00:07.901657 (+   293us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:07.905658 (+  4001us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:07.905924 (+   266us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:00:07.907492  4601 leader_election.cc:290] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:07.908226  4310 raft_consensus.cc:2804] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:00:07.908335  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:07.908706  4310 raft_consensus.cc:493] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:07.909106  4310 raft_consensus.cc:3060] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:00:07.909622  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:00:07.877269  2329 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39726 (request call id 799) took 2363 ms (client timeout 1925 ms). Trace:
W20260709 13:00:07.912104  2329 rpcz_store.cc:269] 0709 13:00:05.502102 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:05.527127 (+ 25025us) service_pool.cc:224] Handling call
0709 13:00:07.808471 (+2281344us) spinlock_profiling.cc:243] Waited 38.7 ms on lock 0x7b5800178598. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f426303db71 00007f426303deee 0000000000375939 00007f426b0b11e0 00007f426b17a0c4 00007f426b17a131 00007f426b1280a0 00007f426b127984 00007f42646a26d4 00007f42646a2662
0709 13:00:07.861412 (+ 52941us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":89088384}
I20260709 13:00:07.910383  1680 raft_consensus.cc:2368] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 3. Current term is 4.
I20260709 13:00:07.914393  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:07.915194  1747 raft_consensus.cc:2393] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 6: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:00:07.915700  1748 raft_consensus.cc:1240] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 6 LEADER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20260709 13:00:07.916747  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:07.922166  1569 leader_election.cc:400] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 3 pre-election: Vote denied by peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261) with higher term. Message: Invalid argument: T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 3. Current term is 4.
I20260709 13:00:07.922730  1569 leader_election.cc:403] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:00:07.923439  1748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43156 (request call id 492) took 1889 ms. Trace:
I20260709 13:00:07.923952  1752 raft_consensus.cc:2410] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 13:00:07.924643  4589 raft_consensus.cc:3060] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 2 FOLLOWER]: Advancing to term 4
I20260709 13:00:07.924656  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:07.925354  1752 raft_consensus.cc:1240] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 5 LEADER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 4. Current term is 5. Ops: []
I20260709 13:00:07.925838  1680 raft_consensus.cc:2410] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for term 6 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 4.
I20260709 13:00:07.926009  1752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43156 (request call id 494) took 1424 ms. Trace:
I20260709 13:00:07.926467  1752 rpcz_store.cc:276] 0709 13:00:06.502228 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:07.924792 (+1422564us) service_pool.cc:224] Handling call
0709 13:00:07.925145 (+   353us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:07.925763 (+   618us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:07.925954 (+   191us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:00:07.927016  1680 raft_consensus.cc:1240] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 7. Current term is 8. Ops: []
I20260709 13:00:07.927765  1752 raft_consensus.cc:1240] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 5. Current term is 6. Ops: []
I20260709 13:00:07.928465  1752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43156 (request call id 495) took 1424 ms. Trace:
I20260709 13:00:07.929087  4572 consensus_queue.cc:1059] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 13:00:07.929250  4310 raft_consensus.cc:515] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:07.930204  4591 raft_consensus.cc:3055] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:00:07.930627  4591 raft_consensus.cc:740] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:00:07.928804  1752 rpcz_store.cc:276] 0709 13:00:06.504398 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:07.927282 (+1422884us) service_pool.cc:224] Handling call
0709 13:00:07.927595 (+   313us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:07.928223 (+   628us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:07.928413 (+   190us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:00:07.933040  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:07.934139  1680 raft_consensus.cc:3060] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:00:07.924021  1748 rpcz_store.cc:276] 0709 13:00:06.031993 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:07.907529 (+1875536us) service_pool.cc:224] Handling call
0709 13:00:07.908010 (+   481us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:00:07.920393 (+ 12383us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:07.920669 (+   276us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:00:07.936626  1646 leader_election.cc:336] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Timed out: RequestConsensusVote RPC to 127.0.107.195:40693 timed out after 2.154s (SENT)
I20260709 13:00:07.937373  1646 leader_election.cc:304] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:00:07.940238  1752 raft_consensus.cc:1240] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 6. Current term is 7. Ops: []
I20260709 13:00:07.941248  1752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43156 (request call id 496) took 1436 ms. Trace:
I20260709 13:00:07.941563  1752 rpcz_store.cc:276] 0709 13:00:06.504848 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:07.933805 (+1428957us) service_pool.cc:224] Handling call
0709 13:00:07.940051 (+  6246us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:07.941020 (+   969us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:07.941194 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:00:07.943485  4310 leader_election.cc:290] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 8 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:07.940207  4628 raft_consensus.cc:2749] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:00:07.946794  3978 leader_election.cc:290] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 6 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 13:00:07.948436  4553 log.cc:927] Time spent T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 1.937s	user 0.003s	sys 0.002s
I20260709 13:00:07.948740  4578 leader_election.cc:290] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:07.950098  2330 raft_consensus.cc:2468] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 7.
I20260709 13:00:07.951377  1748 raft_consensus.cc:1275] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20260709 13:00:07.952181  1748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43156 (request call id 497) took 1447 ms. Trace:
I20260709 13:00:07.952502  1748 rpcz_store.cc:276] 0709 13:00:06.505202 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:07.936234 (+1431032us) service_pool.cc:224] Handling call
0709 13:00:07.950969 (+ 14735us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:07.951871 (+   902us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:07.952124 (+   253us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:00:07.878284  4539 raft_consensus.cc:515] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:07.933240  4580 consensus_queue.cc:260] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:07.977701  4580 raft_consensus.cc:3060] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:00:07.962939  1747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43156 (request call id 493) took 1561 ms. Trace:
I20260709 13:00:07.983870  1680 raft_consensus.cc:2468] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 8.
I20260709 13:00:07.982084  1747 rpcz_store.cc:276] 0709 13:00:06.401575 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:07.924130 (+1522555us) service_pool.cc:224] Handling call
0709 13:00:07.944944 (+ 20814us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:07.962401 (+ 17457us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:00:07.962477 (+    76us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:00:07.962546 (+    69us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:00:07.962617 (+    71us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:07.962705 (+    88us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:07.962855 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:00:07.996701  1675 raft_consensus.cc:2468] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 6.
I20260709 13:00:07.919868  4551 raft_consensus.cc:515] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:07.959617  4550 leader_election.cc:290] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:08.041299  1569 leader_election.cc:304] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:00:07.951846  4591 consensus_queue.cc:260] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:08.046998  4478 consensus_meta.cc:287] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.566s	user 0.013s	sys 0.001s
I20260709 13:00:08.059862  4372 leader_election.cc:290] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 4 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:08.048871  4576 raft_consensus.cc:2804] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Leader election won for term 8
I20260709 13:00:08.064033  1569 leader_election.cc:304] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:00:08.069561  4372 raft_consensus.cc:2804] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader election won for term 6
I20260709 13:00:08.061672  4591 raft_consensus.cc:3060] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:00:08.083484  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:08.084398  1675 raft_consensus.cc:2393] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 4: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:00:08.061023  4478 raft_consensus.cc:515] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:08.064954  4576 raft_consensus.cc:697] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:00:08.070190  4372 raft_consensus.cc:697] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:00:08.094985  4372 consensus_queue.cc:237] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:08.095985  4551 leader_election.cc:290] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 5 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:00:08.104985  4560 log.cc:927] Time spent T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 1.856s	user 0.000s	sys 0.006s
I20260709 13:00:08.109985  4539 leader_election.cc:290] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:08.120805  1597 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40154 (request call id 1852) took 3255 ms. Trace:
I20260709 13:00:08.121387  1597 rpcz_store.cc:276] 0709 13:00:04.864190 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:05.150971 (+286781us) service_pool.cc:224] Handling call
0709 13:00:05.195358 (+ 44387us) spinlock_profiling.cc:243] Waited 33.4 ms on lock 0x7b5800003698. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f426303db71 00007f426303da22 00007f426bf56c09 00007f426b0b0ee0 00007f426b17a0ab 00007f426b17a131 00007f426b13bfc0 00007f426b126929 00007f426b1263d2 00007f426469dbd4
0709 13:00:05.195642 (+   284us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:00:05.196651 (+  1009us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:00:05.196714 (+    63us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:00:05.197415 (+   701us) log.cc:844] Serialized 26 byte log entry
0709 13:00:05.520430 (+323015us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:00:06.557312 (+1036882us) log.cc:844] Serialized 14 byte log entry
0709 13:00:06.557638 (+   326us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:06.557756 (+   118us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:00:08.104297 (+1546541us) spinlock_profiling.cc:243] Waited 454 ms on lock 0x7b4400022b4c. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f4262f9bc85 00007f4262f9c6a2 00007f4265d0e1db 00007f4265d09aaf 00007f4265cfd86e 00007f4265cfcbb6 00007f4265cfc7d8 00007f4265d06221 00007f4265d061d9 00007f4265d06169
0709 13:00:08.118877 (+ 14580us) raft_consensus.cc:1685] finished
0709 13:00:08.118903 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:08.119055 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":17853,"prepare.queue_time_us":1000635,"prepare.run_cpu_time_us":57,"prepare.run_wall_time_us":58,"raft.queue_time_us":202295,"raft.run_cpu_time_us":376,"raft.run_wall_time_us":63571,"spinlock_wait_cycles":1120890112,"thread_start_us":1203108,"threads_started":2,"wal-append.queue_time_us":847027}
I20260709 13:00:08.130797   431 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:00:08.132628  4576 consensus_queue.cc:237] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:08.133553  4566 log.cc:927] Time spent T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 1.773s	user 0.005s	sys 0.000s
I20260709 13:00:08.137432  1671 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52766 (request call id 2181) took 2742 ms. Trace:
I20260709 13:00:08.137874  1671 rpcz_store.cc:276] 0709 13:00:05.395236 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:05.395781 (+   545us) service_pool.cc:224] Handling call
0709 13:00:05.405649 (+  9868us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:00:05.518510 (+112861us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:00:05.518560 (+    50us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:00:05.550104 (+ 31544us) log.cc:844] Serialized 26 byte log entry
0709 13:00:05.957117 (+407013us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:00:05.957262 (+   145us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:05.957623 (+   361us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:00:08.132799 (+2175176us) spinlock_profiling.cc:243] Waited 319 ms on lock 0x7b4400022b4c. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f4262f9bc85 00007f4262f9c6a2 00007f4265d0e1db 00007f4265d09aaf 00007f4265cfd86e 00007f4265cfcbb6 00007f4265cfc7d8 00007f4265d06221 00007f4265d061d9 00007f4265d06169
0709 13:00:08.134733 (+  1934us) raft_consensus.cc:1685] finished
0709 13:00:08.134756 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:08.137305 (+  2549us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":163360,"raft.queue_time_us":680394,"raft.run_cpu_time_us":172,"raft.run_wall_time_us":32909,"spinlock_wait_cycles":732915456,"thread_start_us":276159,"threads_started":1,"wal-append.queue_time_us":634365}
W20260709 13:00:08.162810  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:08.164446  4637 raft_consensus.cc:493] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:00:08.169085  1752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43156 (request call id 500) took 1662 ms. Trace:
I20260709 13:00:08.169335  4571 raft_consensus.cc:515] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:08.195051  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:08.171123  1752 rpcz_store.cc:276] 0709 13:00:06.506249 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:07.942296 (+1436047us) service_pool.cc:224] Handling call
0709 13:00:07.960148 (+ 17852us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:07.964893 (+  4745us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 13:00:08.114279 (+149386us) log.cc:844] Serialized 14 byte log entry
0709 13:00:08.168647 (+ 54368us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:00:08.168713 (+    66us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:00:08.168774 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:08.168864 (+    90us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:08.169013 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":104643,"prepare.run_cpu_time_us":56,"prepare.run_wall_time_us":57,"raft.queue_time_us":100,"raft.run_cpu_time_us":152,"raft.run_wall_time_us":768,"thread_start_us":121838,"threads_started":2}
I20260709 13:00:08.205448  4541 raft_consensus.cc:3055] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:00:08.209168  4541 raft_consensus.cc:740] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:00:08.212174  4571 leader_election.cc:290] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 5 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:08.169030  4637 raft_consensus.cc:515] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:08.216296  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 48879.8us
I20260709 13:00:08.222056  4571 raft_consensus.cc:493] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:08.222774  4571 raft_consensus.cc:515] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:08.225116  4571 leader_election.cc:290] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:08.226826  4478 leader_election.cc:290] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 8 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 13:00:08.239235  4640 log.cc:927] Time spent T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.073s	user 0.000s	sys 0.000s
I20260709 13:00:08.244079  4637 leader_election.cc:290] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:00:08.245787  4641 log.cc:927] Time spent T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.072s	user 0.000s	sys 0.001s
I20260709 13:00:08.251650  4607 raft_consensus.cc:3055] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:00:08.252162  4607 raft_consensus.cc:740] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:00:08.253115  4607 consensus_queue.cc:260] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:08.216163  4589 raft_consensus.cc:2749] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261) with higher term. Message: Invalid argument: T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 3. Current term is 4.
I20260709 13:00:08.210217  4541 consensus_queue.cc:260] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:08.278028  1680 raft_consensus.cc:1240] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 7. Current term is 8. Ops: []
I20260709 13:00:08.282460  1675 raft_consensus.cc:1240] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 4. Current term is 5. Ops: []
I20260709 13:00:08.286733  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:00:08.288236  1571 leader_election.cc:336] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Timed out: RequestConsensusVote RPC to 127.0.107.195:40693 timed out after 1.766s (SENT)
I20260709 13:00:08.288937  1571 leader_election.cc:304] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:00:08.290915  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:08.260020  4541 raft_consensus.cc:3060] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:00:08.292829  1672 raft_consensus.cc:2393] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 6: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:00:08.296295  1680 raft_consensus.cc:1275] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260709 13:00:08.262558  4607 raft_consensus.cc:3060] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:00:08.296082  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:08.301514  1675 raft_consensus.cc:3060] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:00:08.311298  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:08.313975  1671 raft_consensus.cc:2393] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 4: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:00:08.313236  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:08.315383  1672 raft_consensus.cc:2468] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 6.
I20260709 13:00:08.321378  1749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55840 (request call id 2171) took 1234 ms. Trace:
I20260709 13:00:08.321831  1749 rpcz_store.cc:276] 0709 13:00:07.086640 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:07.721038 (+634398us) service_pool.cc:224] Handling call
0709 13:00:08.321131 (+600093us) spinlock_profiling.cc:243] Waited 75.5 ms on lock 0x7b540006fc54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:00:08.321300 (+   169us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":173561600}
I20260709 13:00:08.323457  1749 raft_consensus.cc:1240] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20260709 13:00:08.324126  4503 leader_election.cc:290] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:08.306995  4547 raft_consensus.cc:2749] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:00:08.327678  1675 raft_consensus.cc:2468] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 7.
I20260709 13:00:08.328704  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:08.329488  1601 raft_consensus.cc:2468] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 5.
I20260709 13:00:08.329926  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:08.332166  1671 raft_consensus.cc:2393] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 8: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:00:08.332685  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:08.330293  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:08.335181  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:08.335935  1605 raft_consensus.cc:2468] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 4.
I20260709 13:00:08.342582  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:08.343688  1675 raft_consensus.cc:2393] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 4: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:00:08.345444  1605 raft_consensus.cc:1240] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20260709 13:00:08.346336  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:08.346738  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:08.339411  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:08.349916  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:08.350932  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:08.352407  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:08.353304  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:08.353214  1597 raft_consensus.cc:2468] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 6.
I20260709 13:00:08.354188  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:08.354809  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:08.355504  1601 raft_consensus.cc:3060] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:00:08.317800  4591 raft_consensus.cc:493] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:08.358862  1752 raft_consensus.cc:1240] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20260709 13:00:08.359589  1672 raft_consensus.cc:1240] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 6. Current term is 7. Ops: []
I20260709 13:00:08.364109  1599 raft_consensus.cc:1240] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 6. Current term is 7. Ops: []
I20260709 13:00:08.364600  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:08.369657  1596 raft_consensus.cc:3060] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:00:08.365705  1603 raft_consensus.cc:1240] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 7. Current term is 8. Ops: [4.4-7.5]
I20260709 13:00:08.367537  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:08.367105  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:08.371796  1604 raft_consensus.cc:2468] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 6.
I20260709 13:00:08.373900  1601 raft_consensus.cc:2468] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 7.
I20260709 13:00:08.365643  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:08.376394  1749 raft_consensus.cc:2393] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 5: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
W20260709 13:00:08.382648  1571 leader_election.cc:336] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Timed out: RequestConsensusVote RPC to 127.0.107.195:40693 timed out after 1.893s (SENT)
I20260709 13:00:08.383339  1571 leader_election.cc:304] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:00:08.380219  1643 leader_election.cc:304] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 13:00:08.379213  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 88131.2us
I20260709 13:00:08.339516  4572 raft_consensus.cc:493] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:08.386400  1643 leader_election.cc:304] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:00:08.387936  2299 leader_election.cc:304] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:00:08.389884  1671 raft_consensus.cc:1240] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 5. Current term is 6. Ops: []
I20260709 13:00:08.389839  1605 raft_consensus.cc:2468] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 4.
I20260709 13:00:08.396518  1672 raft_consensus.cc:1240] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 4. Current term is 5. Ops: []
W20260709 13:00:08.398453  1571 leader_election.cc:336] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Timed out: RequestConsensusVote RPC to 127.0.107.195:40693 timed out after 1.954s (SENT)
I20260709 13:00:08.399094  1571 leader_election.cc:304] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:00:08.401083  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:08.401378  1745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55840 (request call id 2141) took 3254 ms. Trace:
I20260709 13:00:08.401461  1643 leader_election.cc:304] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:00:08.402052  1752 raft_consensus.cc:2410] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for term 6 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 4.
I20260709 13:00:08.404070  1643 leader_election.cc:304] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:00:08.401798  1745 rpcz_store.cc:276] 0709 13:00:05.143366 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:05.149274 (+  5908us) service_pool.cc:224] Handling call
0709 13:00:07.743325 (+2594051us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:07.747801 (+  4476us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:08.397398 (+649597us) spinlock_profiling.cc:243] Waited 2.59 s on lock 0x7b5400071550. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661618ad 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:00:08.397508 (+   110us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5964356480}
I20260709 13:00:08.389660  4547 raft_consensus.cc:2749] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:00:08.395797  4593 raft_consensus.cc:2749] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20260709 13:00:08.413307  2300 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:08.416055  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:08.417131  1745 raft_consensus.cc:2393] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 7: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:00:08.402968  4539 raft_consensus.cc:2804] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:00:08.411285  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:08.390215  4512 consensus_queue.cc:1059] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 5, Time since last communication: 0.022s
I20260709 13:00:08.408429  4438 raft_consensus.cc:2749] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:00:08.425171  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:08.394274  4390 consensus_queue.cc:1048] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.021s
I20260709 13:00:08.394275  4353 consensus_queue.cc:1059] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.019s
I20260709 13:00:08.408704  4551 raft_consensus.cc:2749] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:00:08.381795  4591 raft_consensus.cc:515] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:08.397413  4572 raft_consensus.cc:515] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:08.429572  1571 leader_election.cc:304] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:00:08.431464  2299 leader_election.cc:304] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:00:08.436098  4390 raft_consensus.cc:2804] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:00:08.436523  4390 raft_consensus.cc:493] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Starting leader election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:00:08.436939  4390 raft_consensus.cc:3060] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:00:08.440580  1750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55840 (request call id 2165) took 1373 ms. Trace:
I20260709 13:00:08.441213  1750 rpcz_store.cc:276] 0709 13:00:07.067223 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:07.826692 (+759469us) service_pool.cc:224] Handling call
0709 13:00:08.440217 (+613525us) spinlock_profiling.cc:243] Waited 43.7 ms on lock 0x7b5400322dd4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:00:08.440468 (+   251us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":100456704}
I20260709 13:00:08.424422  4539 raft_consensus.cc:493] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Starting leader election (detected failure of leader af520b67945d443b901bbeca37cc5701)
W20260709 13:00:08.464165  1569 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:00:08.472462  2300 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:00:08.477553  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 48074.9us
I20260709 13:00:08.479708  1571 leader_election.cc:304] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 13:00:08.481230  1571 leader_election.cc:304] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:00:08.429296  4649 raft_consensus.cc:2749] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20260709 13:00:08.482810  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 50901.9us
I20260709 13:00:08.487946  1643 leader_election.cc:304] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:00:08.489393  1643 leader_election.cc:304] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
W20260709 13:00:08.491032  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:00:08.497892  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:00:08.499987  4567 log.cc:927] Time spent T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 2.139s	user 0.003s	sys 0.003s
I20260709 13:00:08.501466  1746 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55840 (request call id 2154) took 3222 ms. Trace:
I20260709 13:00:08.501883  4390 raft_consensus.cc:515] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:08.502642  4545 log.cc:927] Time spent T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 2.645s	user 0.005s	sys 0.005s
I20260709 13:00:08.501878  1746 rpcz_store.cc:276] 0709 13:00:05.279461 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:05.315259 (+ 35798us) service_pool.cc:224] Handling call
0709 13:00:05.336962 (+ 21703us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:00:05.358529 (+ 21567us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:00:05.358581 (+    52us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:00:05.370772 (+ 12191us) log.cc:844] Serialized 26 byte log entry
0709 13:00:06.126381 (+755609us) raft_consensus.cc:1645] Marking committed up to 3
0709 13:00:06.492080 (+365699us) log.cc:844] Serialized 14 byte log entry
0709 13:00:06.602952 (+110872us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:06.603088 (+   136us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:00:08.499307 (+1896219us) spinlock_profiling.cc:243] Waited 358 ms on lock 0x7b4400022b4c. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f4262f9bc85 00007f4262f9c6a2 00007f4265d0e1db 00007f4265d09aaf 00007f4265cfd86e 00007f4265cfcbb6 00007f4265cfc7d8 00007f4265d06221 00007f4265d061d9 00007f4265d06169
0709 13:00:08.501093 (+  1786us) raft_consensus.cc:1685] finished
0709 13:00:08.501206 (+   113us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:08.501385 (+   179us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":110387,"prepare.queue_time_us":338177,"prepare.run_cpu_time_us":59,"prepare.run_wall_time_us":61,"raft.queue_time_us":250821,"raft.run_cpu_time_us":371,"raft.run_wall_time_us":37237,"spinlock_wait_cycles":824514432,"thread_start_us":1014478,"threads_started":2,"wal-append.queue_time_us":870884}
I20260709 13:00:08.504586  1673 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57092 (request call id 592) took 4135 ms. Trace:
W20260709 13:00:08.506537  1645 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:08.505081  1673 rpcz_store.cc:276] 0709 13:00:04.369615 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:04.369767 (+   152us) service_pool.cc:224] Handling call
0709 13:00:04.384551 (+ 14784us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:00:04.411337 (+ 26786us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 13:00:04.737107 (+325770us) log.cc:844] Serialized 14 byte log entry
0709 13:00:05.297261 (+560154us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:00:05.297859 (+   598us) log.cc:844] Serialized 26 byte log entry
0709 13:00:05.298017 (+   158us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:00:05.298051 (+    34us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:05.298133 (+    82us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:00:08.501942 (+3203809us) spinlock_profiling.cc:243] Waited 728 ms on lock 0x7b4400022b4c. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f4262f9bc85 00007f4262f9c6a2 00007f4265d0e1db 00007f4265d09aaf 00007f4265cfd86e 00007f4265cfcbb6 00007f4265cfc7d8 00007f4265d06221 00007f4265d061d9 00007f4265d06169
0709 13:00:08.504216 (+  2274us) raft_consensus.cc:1685] finished
0709 13:00:08.504329 (+   113us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:08.504483 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":16888,"prepare.queue_time_us":423200,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":45,"raft.queue_time_us":21383,"raft.run_cpu_time_us":183,"raft.run_wall_time_us":184,"spinlock_wait_cycles":1675061248,"thread_start_us":883007,"threads_started":2,"wal-append.queue_time_us":802895}
I20260709 13:00:08.507233  4246 raft_consensus.cc:3060] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:00:08.508262  4390 leader_election.cc:290] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 7 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:08.510787  4654 raft_consensus.cc:3055] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:00:08.511008  4512 raft_consensus.cc:2804] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:00:08.511325  4654 raft_consensus.cc:740] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:00:08.511722  4512 raft_consensus.cc:697] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 7 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:00:08.512184  4390 consensus_queue.cc:1059] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 6.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 13:00:08.512493  4654 consensus_queue.cc:260] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:08.513360  4390 raft_consensus.cc:493] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:08.513924  4654 raft_consensus.cc:3060] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:00:08.513943  4390 raft_consensus.cc:515] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:08.516129  4591 leader_election.cc:290] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:08.512910  4512 consensus_queue.cc:237] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 5.8, Last appended by leader: 8, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:08.518065  4572 leader_election.cc:290] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:08.519704  4353 raft_consensus.cc:3055] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:00:08.520155  4353 raft_consensus.cc:740] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:00:08.523375  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:08.473601  4539 raft_consensus.cc:3060] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:00:08.528544  4438 raft_consensus.cc:2804] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:00:08.529201  4438 raft_consensus.cc:697] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
W20260709 13:00:08.529706  1645 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:08.533094  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:08.533828  1673 raft_consensus.cc:3060] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:00:08.534380  4569 leader_election.cc:290] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:08.534931  4656 raft_consensus.cc:3055] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:00:08.535378  4656 raft_consensus.cc:740] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:00:08.535770  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:08.536515  2328 raft_consensus.cc:2468] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 5.
I20260709 13:00:08.536441  4656 consensus_queue.cc:260] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:08.537206  2328 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49082 (request call id 645) took 1609 ms (client timeout 1847 ms). Trace:
I20260709 13:00:08.537963  4656 raft_consensus.cc:3060] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:00:08.488188  4648 raft_consensus.cc:2804] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:00:08.524399  4534 consensus_queue.cc:1059] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.015s
I20260709 13:00:08.539465  4648 raft_consensus.cc:493] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:08.539918  4648 raft_consensus.cc:3060] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:00:08.540056  4070 consensus_queue.cc:1059] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 13:00:08.541190  4070 consensus_queue.cc:1059] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 13:00:08.543196  4353 consensus_queue.cc:260] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:08.544579  4618 raft_consensus.cc:3055] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:00:08.488358  4547 raft_consensus.cc:3060] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:00:08.545137  4518 raft_consensus.cc:2804] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:00:08.546335  4518 raft_consensus.cc:697] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:00:08.546579  4583 raft_consensus.cc:3060] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:00:08.547472  4518 consensus_queue.cc:237] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:08.526634  4551 consensus_queue.cc:1059] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.022s
I20260709 13:00:08.545133  4618 raft_consensus.cc:740] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:00:08.549439  4551 raft_consensus.cc:3055] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:00:08.549863  4551 raft_consensus.cc:740] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:00:08.550203  4618 consensus_queue.cc:260] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:08.550758  4551 consensus_queue.cc:260] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:08.551762  4618 raft_consensus.cc:3060] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:00:08.553488  4653 raft_consensus.cc:3055] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:00:08.553923  4653 raft_consensus.cc:740] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:00:08.544816  4353 raft_consensus.cc:3060] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:00:08.554963  4653 consensus_queue.cc:260] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:08.556283  4653 raft_consensus.cc:3060] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Advancing to term 6
W20260709 13:00:08.557588  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:08.559727  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:00:08.560237  4557 log.cc:927] Time spent T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 2.454s	user 0.005s	sys 0.001s
I20260709 13:00:08.530287  4438 consensus_queue.cc:237] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:08.566418  1602 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40154 (request call id 1850) took 3709 ms. Trace:
W20260709 13:00:08.537657  2328 rpcz_store.cc:269] 0709 13:00:06.928429 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:06.977893 (+ 49464us) service_pool.cc:224] Handling call
0709 13:00:08.535536 (+1557643us) spinlock_profiling.cc:243] Waited 2.15 ms on lock 0x7b5800178598. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f426303db71 00007f426303da22 00007f426bf56c09 00007f426b0b0ee0 00007f426b17a0ab 00007f426b17a131 00007f426b1280a0 00007f426b127984 00007f42646a26d4 00007f42646a2662
0709 13:00:08.537134 (+  1598us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4954496}
I20260709 13:00:08.569908  1602 rpcz_store.cc:276] 0709 13:00:04.857109 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:05.141720 (+284611us) service_pool.cc:224] Handling call
0709 13:00:05.185186 (+ 43466us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:00:05.190270 (+  5084us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:00:05.190309 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:00:05.214517 (+ 24208us) log.cc:844] Serialized 26 byte log entry
0709 13:00:05.552833 (+338316us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:00:06.282853 (+730020us) log.cc:844] Serialized 14 byte log entry
0709 13:00:06.565347 (+282494us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:06.565449 (+   102us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:00:06.727850 (+162401us) spinlock_profiling.cc:243] Waited 5.58 ms on lock 0x7b4800003cd8. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426b08baa1 00007f426b082b37 00007f426b0829ca 00007f426b17a26f 00007f426b179f37 00007f426b18e55c 00007f426b18e508 00007f426b18e488 00007f426b18e443 00007f426b18d6ec 00007f426a0b7a96
0709 13:00:07.959089 (+1231239us) spinlock_profiling.cc:243] Waited 14.8 ms on lock 0x7b1800428aa0. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f42630753f3 00007f42630777c1 00007f4263077779 00007f4263077709 00007f42630776d1 00007f42630769cd 000000000038f354 000000000038f199 00007f426305883a
0709 13:00:08.559418 (+600329us) spinlock_profiling.cc:243] Waited 578 ms on lock 0x7b4400022b4c. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f4262f9bc85 00007f4262f9c6a2 00007f4265d0e1db 00007f4265d09aaf 00007f4265cfd86e 00007f4265cfcbb6 00007f4265cfc7d8 00007f4265d06221 00007f4265d061d9 00007f4265d06169
0709 13:00:08.566171 (+  6753us) raft_consensus.cc:1685] finished
0709 13:00:08.566201 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:08.566340 (+   139us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":117042,"prepare.queue_time_us":968342,"prepare.run_cpu_time_us":55,"prepare.run_wall_time_us":56,"raft.queue_time_us":385083,"raft.run_cpu_time_us":34381,"raft.run_wall_time_us":1522818,"spinlock_wait_cycles":1375375104,"thread_start_us":1258293,"threads_started":3,"wal-append.queue_time_us":726539}
I20260709 13:00:08.573273  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:08.574002  1602 raft_consensus.cc:2468] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 5.
I20260709 13:00:08.575213  2333 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39726 (request call id 801) took 1200 ms. Trace:
I20260709 13:00:08.575755  2333 rpcz_store.cc:276] 0709 13:00:07.374975 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:07.376262 (+  1287us) service_pool.cc:224] Handling call
0709 13:00:07.376765 (+   503us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:07.727534 (+350769us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:08.574867 (+847333us) spinlock_profiling.cc:243] Waited 350 ms on lock 0x7b540065ed54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:00:08.575121 (+   254us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":804658432}
I20260709 13:00:08.552096  4551 raft_consensus.cc:3060] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:00:08.578076  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:08.520459  4390 leader_election.cc:290] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:08.594539  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:00:08.602911  1643 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 1.554s (SENT) [suppressed 45 similar messages]
I20260709 13:00:08.598325  1596 raft_consensus.cc:2468] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 6.
W20260709 13:00:08.567436  4586 log.cc:927] Time spent T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 1.594s	user 0.000s	sys 0.004s
I20260709 13:00:08.630281  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:08.632361  2298 leader_election.cc:304] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:00:08.647327  1678 raft_consensus.cc:1240] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 5. Current term is 6. Ops: []
I20260709 13:00:08.540192  4534 raft_consensus.cc:3055] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:00:08.637735  1746 raft_consensus.cc:2468] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 6.
I20260709 13:00:08.666021  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:08.667488  1602 raft_consensus.cc:2393] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 4: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
W20260709 13:00:08.669628  1571 leader_election.cc:336] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Timed out: RequestConsensusVote RPC to 127.0.107.195:40693 timed out after 1.846s (SENT)
I20260709 13:00:08.670346  1571 leader_election.cc:304] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:00:08.629623  4642 raft_consensus.cc:2749] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:00:08.662830  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:08.680415  1599 raft_consensus.cc:2468] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 6.
I20260709 13:00:08.672785  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:08.683023  1596 raft_consensus.cc:2468] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 5.
I20260709 13:00:08.685153  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:08.669737  4650 raft_consensus.cc:3055] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:00:08.656518  4534 raft_consensus.cc:740] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
W20260709 13:00:08.687809  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 151881us
I20260709 13:00:08.687615  1599 raft_consensus.cc:2393] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 6: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:00:08.671186  4390 raft_consensus.cc:2804] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:00:08.675308  4587 raft_consensus.cc:2749] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:00:08.705540  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:08.710927  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:08.712533  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 10 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:08.721860  1602 raft_consensus.cc:2368] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 for earlier term 6. Current term is 7.
I20260709 13:00:08.723304  1677 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57092 (request call id 602) took 3290 ms. Trace:
I20260709 13:00:08.727227  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:08.728293  1599 raft_consensus.cc:3060] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:00:08.731516  1597 raft_consensus.cc:1240] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 4. Current term is 5. Ops: []
I20260709 13:00:08.723932  1677 rpcz_store.cc:276] 0709 13:00:05.433037 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:05.595100 (+162063us) service_pool.cc:224] Handling call
0709 13:00:05.679491 (+ 84391us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:00:06.187059 (+507568us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:00:06.187097 (+    38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:00:06.187784 (+   687us) log.cc:844] Serialized 26 byte log entry
0709 13:00:06.628255 (+440471us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:00:06.628310 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:06.628394 (+    84us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:00:08.566667 (+1938273us) spinlock_profiling.cc:243] Waited 2.1 ms on lock 0x7b4400022b4c. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f4262f9bc85 00007f4262f9c6a2 00007f4265d0e1db 00007f4265d09aaf 00007f4265cfd86e 00007f4265cfcbb6 00007f4265cfc7d8 00007f4265d06221 00007f4265d061d9 00007f4265d06169
0709 13:00:08.709115 (+142448us) raft_consensus.cc:1685] finished
0709 13:00:08.709150 (+    35us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:08.723189 (+ 14039us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":507021,"raft.queue_time_us":507983,"raft.run_cpu_time_us":160,"raft.run_wall_time_us":161,"spinlock_wait_cycles":4819200,"thread_start_us":439938,"threads_started":1,"wal-append.queue_time_us":508206}
I20260709 13:00:08.732332  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:08.703465  4650 raft_consensus.cc:740] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
W20260709 13:00:08.738642  4623 log.cc:927] Time spent T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 1.193s	user 0.016s	sys 0.002s
I20260709 13:00:08.709017  4390 raft_consensus.cc:493] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:00:08.745214  1596 raft_consensus.cc:2410] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 for term 8 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 7.
W20260709 13:00:08.756259  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 117102us
W20260709 13:00:08.758889  1718 connection.cc:256] RPC call timeout handler was delayed by -2.8240729639999813s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17830000000000001s for a response
W20260709 13:00:08.759348  1718 connection.cc:256] RPC call timeout handler was delayed by -2.7898729639999829s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1821s for a response
W20260709 13:00:08.759707  1718 connection.cc:256] RPC call timeout handler was delayed by -2.7205729639999845s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1898s for a response
W20260709 13:00:08.760886  1718 connection.cc:256] RPC call timeout handler was delayed by -2.6125729639999804s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20179999999999998s for a response
W20260709 13:00:08.761466  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 158557us
W20260709 13:00:08.762475  1718 connection.cc:256] RPC call timeout handler was delayed by -2.5531729639999412s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20840000000000003s for a response
W20260709 13:00:08.766151  1718 connection.cc:256] RPC call timeout handler was delayed by -2.4694729639999764s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2177s for a response
W20260709 13:00:08.766461  1718 connection.cc:256] RPC call timeout handler was delayed by -2.4442729639999357s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22050000000000003s for a response
I20260709 13:00:08.750494  4390 raft_consensus.cc:3060] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:00:08.774464  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:08.774588  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:00:08.807376  4607 consensus_meta.cc:287] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.502s	user 0.009s	sys 0.008s
I20260709 13:00:08.776758  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:08.828723  4650 consensus_queue.cc:260] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:08.830374  4650 raft_consensus.cc:3060] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:00:08.826370  2298 leader_election.cc:304] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:08.834022  2298 leader_election.cc:304] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:00:08.779017  1599 raft_consensus.cc:2468] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 6.
I20260709 13:00:08.839129  1644 leader_election.cc:304] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:00:08.845302  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
W20260709 13:00:08.847290  1753 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55840 (request call id 2159) took 1784 ms (client timeout 1766 ms). Trace:
W20260709 13:00:08.847951  1753 rpcz_store.cc:269] 0709 13:00:07.062662 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:07.168682 (+106020us) service_pool.cc:224] Handling call
0709 13:00:08.846989 (+1678307us) spinlock_profiling.cc:243] Waited 197 ms on lock 0x7b5400384d54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:00:08.847185 (+   196us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":453497728}
I20260709 13:00:08.801954  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 8 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:08.822950  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:08.851383  1601 raft_consensus.cc:2393] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 4: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:00:08.852653  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:08.854496  1599 raft_consensus.cc:2468] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 5.
I20260709 13:00:08.854645  2298 leader_election.cc:304] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:08.806217  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:00:08.826960  1571 leader_election.cc:336] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Timed out: RequestConsensusVote RPC to 127.0.107.195:40693 timed out after 2.043s (SENT)
I20260709 13:00:08.858330  1571 leader_election.cc:304] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
W20260709 13:00:08.859479  1718 connection.cc:256] RPC call timeout handler was delayed by -2.2858729639999638s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.23809999999999998s for a response
W20260709 13:00:08.860071  1718 connection.cc:256] RPC call timeout handler was delayed by -1.701488774999973s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1912s for a response
W20260709 13:00:08.860466  1718 connection.cc:256] RPC call timeout handler was delayed by -1.701488774999973s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1912s for a response
I20260709 13:00:08.864765  1673 raft_consensus.cc:2468] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 7.
W20260709 13:00:08.863982  1718 connection.cc:256] RPC call timeout handler was delayed by -1.5070887749999429s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21280000000000002s for a response
W20260709 13:00:08.865767  1718 connection.cc:256] RPC call timeout handler was delayed by -1.4160199509999529s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2049s for a response
W20260709 13:00:08.863106  2301 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:00:08.867158  2301 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:00:08.868717  2301 connection.cc:256] RPC call timeout handler was delayed by -1.177811738999992s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2398s for a response
W20260709 13:00:08.869179  2301 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:00:08.869694  2301 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:08.871002  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:08.871044  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:08.871995  1599 raft_consensus.cc:2410] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for term 8 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 7.
I20260709 13:00:08.871364  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:08.872975  1750 raft_consensus.cc:3060] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Advancing to term 6
W20260709 13:00:08.875919  2299 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:00:08.876652  2299 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:08.788213  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
W20260709 13:00:08.866277  1718 connection.cc:256] RPC call timeout handler was delayed by -1.3837887749999709s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22650000000000003s for a response
W20260709 13:00:08.894790  1718 connection.cc:256] RPC call timeout handler was delayed by -1.3684887749999461s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22820000000000001s for a response
W20260709 13:00:08.896294  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 69389.6us
W20260709 13:00:08.897997  2300 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:00:08.898733  2300 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:08.899091  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:08.872010  1749 raft_consensus.cc:2468] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 7.
I20260709 13:00:08.903126  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
W20260709 13:00:08.808619  4658 log.cc:927] Time spent T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.062s	user 0.001s	sys 0.000s
I20260709 13:00:08.904414  1599 raft_consensus.cc:1275] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
W20260709 13:00:08.895319  1718 connection.cc:256] RPC call timeout handler was delayed by -1.309205737999946s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.223s for a response
I20260709 13:00:08.908322  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:08.837558  4070 raft_consensus.cc:2749] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 13:00:08.909533  1745 raft_consensus.cc:1240] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 5. Current term is 6. Ops: []
I20260709 13:00:08.815976  4478 raft_consensus.cc:493] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:08.780712  4570 raft_consensus.cc:493] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:08.781025  4574 raft_consensus.cc:493] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:08.871253  3977 raft_consensus.cc:493] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:08.870309  4607 raft_consensus.cc:2804] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:00:08.929111  1675 raft_consensus.cc:1240] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 6. Current term is 7. Ops: []
I20260709 13:00:08.905085  4587 raft_consensus.cc:2749] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:00:08.755637  4534 consensus_queue.cc:260] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:08.870261  4550 raft_consensus.cc:493] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:08.895704  4580 raft_consensus.cc:2764] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
I20260709 13:00:08.937217  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:08.894236  1674 raft_consensus.cc:2393] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 6: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
W20260709 13:00:08.943743  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 47400.2us
I20260709 13:00:08.955297  2298 leader_election.cc:400] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 6 pre-election: Vote denied by peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445) with higher term. Message: Invalid argument: T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 for earlier term 6. Current term is 7.
I20260709 13:00:08.955669  2298 leader_election.cc:403] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 6 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:00:08.955435  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:00:08.948994  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:00:08.956729  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:08.913872  4512 raft_consensus.cc:3060] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:00:08.959817  1747 raft_consensus.cc:1240] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 8 LEADER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 7. Current term is 8. Ops: [7.5-7.5]
W20260709 13:00:08.936620  4543 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 2.535s	user 0.012s	sys 0.035s
W20260709 13:00:08.963362  1645 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:08.970369  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:00:08.972627  1571 leader_election.cc:336] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Timed out: RequestConsensusVote RPC to 127.0.107.195:40693 timed out after 2.206s (SENT)
I20260709 13:00:08.973073  1571 leader_election.cc:304] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:00:08.944486  4607 raft_consensus.cc:493] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:00:08.980749  1645 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:00:08.967619  4543 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 2.592s	user 0.012s	sys 0.036s
I20260709 13:00:08.986624  1605 raft_consensus.cc:1240] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 LEADER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 7. Current term is 8. Ops: []
W20260709 13:00:08.987663  1571 leader_election.cc:336] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Timed out: RequestConsensusVote RPC to 127.0.107.195:40693 timed out after 2.220s (SENT)
I20260709 13:00:08.988356  1605 raft_consensus.cc:1240] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 LEADER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 5. Current term is 6. Ops: []
I20260709 13:00:08.989549  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:08.988250  1571 leader_election.cc:304] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:00:08.990121  1605 raft_consensus.cc:1275] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20260709 13:00:08.932090  4570 raft_consensus.cc:515] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:08.935793  4478 raft_consensus.cc:515] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:08.969715  4070 raft_consensus.cc:493] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:08.977236  4534 raft_consensus.cc:3060] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:00:08.945407  3977 raft_consensus.cc:515] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:08.982510  4607 raft_consensus.cc:3060] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:00:08.941426  4574 raft_consensus.cc:515] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:08.961045  4628 consensus_queue.cc:1059] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.035s
I20260709 13:00:08.973658  4550 raft_consensus.cc:515] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:09.039347  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 51733.2us
I20260709 13:00:08.975106  4563 raft_consensus.cc:3055] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:00:08.996443  4502 raft_consensus.cc:2749] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:00:08.985257  4368 raft_consensus.cc:2749] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:00:09.051754  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
W20260709 13:00:09.054836  4539 consensus_meta.cc:287] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.508s	user 0.019s	sys 0.000s
W20260709 13:00:09.059012  4246 consensus_meta.cc:287] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.551s	user 0.015s	sys 0.000s
W20260709 13:00:09.059963  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 136173us
I20260709 13:00:09.065038  2333 raft_consensus.cc:1240] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 7 LEADER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 6. Current term is 7. Ops: [6.9-6.9]
I20260709 13:00:09.073362  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
W20260709 13:00:09.074010  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:09.049947  4438 raft_consensus.cc:493] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:09.077304  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:08.996629  4667 consensus_queue.cc:1059] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 4.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.075s
I20260709 13:00:09.080082  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:00:09.052503  1602 raft_consensus.cc:1240] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 6. Current term is 7. Ops: []
I20260709 13:00:09.079236  1718 leader_election.cc:304] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:09.084815  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:00:09.087792  1718 leader_election.cc:304] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
W20260709 13:00:09.089447  4551 consensus_meta.cc:287] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.512s	user 0.014s	sys 0.000s
I20260709 13:00:09.089847  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:09.090807  2334 raft_consensus.cc:2393] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 6: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:00:09.090934  1679 raft_consensus.cc:2410] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for term 8 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 4.
I20260709 13:00:09.091547  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:09.092188  2334 raft_consensus.cc:1240] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 7. Current term is 8. Ops: []
I20260709 13:00:09.089975  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:00:09.093379  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:00:09.093798  1602 raft_consensus.cc:1275] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 13:00:09.094655  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:00:09.095834  1602 raft_consensus.cc:1240] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 6. Current term is 7. Ops: []
I20260709 13:00:09.098508  4438 raft_consensus.cc:515] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:09.101159  4438 leader_election.cc:290] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:09.103677  4570 leader_election.cc:290] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:09.104051  4478 leader_election.cc:290] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:09.104415  4570 raft_consensus.cc:2749] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 13:00:09.105247  4570 raft_consensus.cc:2749] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:00:09.105794  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:09.107537  2332 raft_consensus.cc:2368] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 3. Current term is 4.
I20260709 13:00:09.108086  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:09.072127  4539 raft_consensus.cc:515] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:09.111754  4539 leader_election.cc:290] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 5 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:09.122516  4662 raft_consensus.cc:493] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:00:09.122957  4667 raft_consensus.cc:2749] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: Vote denied by peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445) with higher term. Message: Invalid argument: T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 for earlier term 6. Current term is 7.
I20260709 13:00:09.123358  4662 raft_consensus.cc:515] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:09.123834  4667 raft_consensus.cc:3055] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:00:09.124264  4667 raft_consensus.cc:740] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:00:09.125236  4667 consensus_queue.cc:260] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:09.059017  4563 raft_consensus.cc:740] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:00:09.129468  4662 leader_election.cc:290] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:09.137723  4550 leader_election.cc:290] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:00:09.144071  1644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:00:09.141709  4353 consensus_meta.cc:287] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.586s	user 0.009s	sys 0.013s
W20260709 13:00:09.147732  4547 consensus_meta.cc:287] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.602s	user 0.017s	sys 0.000s
I20260709 13:00:09.148267  4547 raft_consensus.cc:2749] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:00:09.149432  1605 raft_consensus.cc:1240] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 3. Current term is 4. Ops: []
I20260709 13:00:09.155786  3977 leader_election.cc:290] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 13:00:09.085530  4618 consensus_meta.cc:287] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.533s	user 0.012s	sys 0.012s
I20260709 13:00:09.157095  1672 raft_consensus.cc:2468] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 5.
I20260709 13:00:09.159531  4549 raft_consensus.cc:493] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:09.160171  4549 raft_consensus.cc:515] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:00:09.166004  1569 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:09.166509  1675 raft_consensus.cc:1275] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260709 13:00:09.167354  1680 raft_consensus.cc:1275] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260709 13:00:09.167676  2333 raft_consensus.cc:2468] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 8.
I20260709 13:00:09.177935  1596 raft_consensus.cc:1240] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 5. Current term is 6. Ops: []
I20260709 13:00:09.072342  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:09.129468  4563 consensus_queue.cc:260] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:09.192888  4563 raft_consensus.cc:3060] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:00:09.193583  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:09.170804  4574 leader_election.cc:290] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:09.197424  1750 raft_consensus.cc:2468] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 6.
I20260709 13:00:09.042798  4070 raft_consensus.cc:515] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:09.201480  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 246285us
W20260709 13:00:09.202039  1644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:09.203521  1717 leader_election.cc:304] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:00:09.206597  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:09.214212  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:09.200851  4667 raft_consensus.cc:3060] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:00:09.243337  2298 leader_election.cc:304] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 13:00:09.209728  4653 consensus_meta.cc:287] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.653s	user 0.007s	sys 0.010s
I20260709 13:00:09.208629  4569 raft_consensus.cc:493] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:09.248430  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:09.274274  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
W20260709 13:00:09.248309  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 44910.7us
I20260709 13:00:09.249723  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:00:09.279224  2332 raft_consensus.cc:3060] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:00:09.278327  2336 raft_consensus.cc:2410] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for term 6 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260709 13:00:09.279950  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:09.282354  4070 leader_election.cc:290] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:09.303892  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:09.306809  1680 raft_consensus.cc:1240] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 5. Current term is 6. Ops: []
I20260709 13:00:09.309257  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:09.309964  1680 raft_consensus.cc:2468] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 4.
I20260709 13:00:09.294904  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:09.313856  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:09.314181  1676 raft_consensus.cc:2468] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 4.
I20260709 13:00:09.313221  1674 raft_consensus.cc:1275] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260709 13:00:09.294963  1646 leader_election.cc:304] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:09.314806  2335 raft_consensus.cc:2468] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 9.
W20260709 13:00:09.320338  1643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:09.296733  4580 raft_consensus.cc:3055] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:00:09.324247  4580 raft_consensus.cc:740] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:00:09.325282  4580 consensus_queue.cc:260] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:09.249751  4668 log.cc:927] Time spent T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.166s	user 0.000s	sys 0.000s
I20260709 13:00:09.326881  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:09.327375  1747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55848 (request call id 1967) took 1601 ms. Trace:
I20260709 13:00:09.327615  2335 raft_consensus.cc:1240] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 6 LEADER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 5. Current term is 6. Ops: []
I20260709 13:00:09.297165  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:09.328516  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:09.329375  1596 raft_consensus.cc:2393] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 7: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:00:09.329257  4580 raft_consensus.cc:3060] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:00:09.333364  4500 raft_consensus.cc:3060] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:00:09.336999  4618 raft_consensus.cc:2804] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:00:09.337489  4618 raft_consensus.cc:493] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:00:09.337953  4618 raft_consensus.cc:3060] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:00:09.338090  1646 leader_election.cc:304] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:00:09.327811  1747 rpcz_store.cc:276] 0709 13:00:07.725700 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:08.071116 (+345416us) service_pool.cc:224] Handling call
0709 13:00:08.087407 (+ 16291us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:00:08.960349 (+872942us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:09.327023 (+366674us) spinlock_profiling.cc:243] Waited 872 ms on lock 0x7b5400220954. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:00:09.327296 (+   273us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2005150976}
I20260709 13:00:09.340965  1646 leader_election.cc:304] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:00:09.342187  4530 raft_consensus.cc:2749] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:00:09.342943  4635 raft_consensus.cc:3060] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:00:09.288342  4390 raft_consensus.cc:515] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:09.330294  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:09.348807  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:09.349251  4390 leader_election.cc:290] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 6 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:09.349408  1596 raft_consensus.cc:2468] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 3.
I20260709 13:00:09.349789  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:09.350137  4518 raft_consensus.cc:493] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:00:09.351310  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:09.353369  1601 raft_consensus.cc:1275] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260709 13:00:09.288828  4679 raft_consensus.cc:2804] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:00:09.356546  2329 raft_consensus.cc:2468] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 5.
I20260709 13:00:09.360023  4629 consensus_queue.cc:1059] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 13:00:09.363898  1603 raft_consensus.cc:2468] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 6.
I20260709 13:00:09.391865  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:09.388043  1750 raft_consensus.cc:1240] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 4. Current term is 6. Ops: []
I20260709 13:00:09.394675  1750 raft_consensus.cc:1240] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 7 LEADER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 6. Current term is 7. Ops: []
W20260709 13:00:09.387703  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 106895us
I20260709 13:00:09.396523  1717 leader_election.cc:304] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
W20260709 13:00:09.398499  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 155201us
I20260709 13:00:09.399171  1753 raft_consensus.cc:1240] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 7. Current term is 8. Ops: []
W20260709 13:00:09.248212  4648 consensus_meta.cc:287] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.708s	user 0.012s	sys 0.006s
W20260709 13:00:09.400888  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 91919.3us
I20260709 13:00:09.401489  2298 leader_election.cc:304] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:00:09.405045  2299 leader_election.cc:304] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:00:09.405507  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:09.411237  1754 raft_consensus.cc:2468] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 4.
I20260709 13:00:09.412433  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:09.413084  1752 raft_consensus.cc:1240] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 5. Current term is 6. Ops: []
I20260709 13:00:09.410130  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:09.413602  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:09.407981  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:09.414243  1753 raft_consensus.cc:3060] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:00:09.412261  2332 raft_consensus.cc:2410] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for term 7 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
I20260709 13:00:09.405269  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:09.416498  1747 raft_consensus.cc:2468] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 5.
I20260709 13:00:09.419116  1749 raft_consensus.cc:2393] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 6: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:00:09.421227  1754 raft_consensus.cc:1240] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 4. Current term is 5. Ops: []
W20260709 13:00:09.421773  2300 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:09.384357  4679 raft_consensus.cc:493] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:00:09.419081  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 331377us
I20260709 13:00:09.422695  1749 raft_consensus.cc:1275] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:00:09.425969  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:09.427356  1718 leader_election.cc:304] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 13:00:09.416795  4583 consensus_meta.cc:287] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.870s	user 0.021s	sys 0.000s
W20260709 13:00:09.430364  1644 leader_election.cc:336] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625): Timed out: RequestConsensusVote RPC to 127.0.107.196:46625 timed out after 1.925s (SENT)
I20260709 13:00:09.435585  4676 raft_consensus.cc:2804] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:00:09.435727  1644 leader_election.cc:304] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:09.436477  4574 raft_consensus.cc:2804] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 13:00:09.437031  4574 raft_consensus.cc:697] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 6 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:00:09.437255  4246 raft_consensus.cc:2749] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:00:09.437462  2299 leader_election.cc:304] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:00:09.438473  4070 consensus_queue.cc:1059] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20260709 13:00:09.420893  4654 consensus_meta.cc:287] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.903s	user 0.010s	sys 0.013s
I20260709 13:00:09.439640  4070 raft_consensus.cc:2804] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:00:09.440026  4070 raft_consensus.cc:493] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:09.440369  4070 raft_consensus.cc:3060] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:00:09.440850  4512 raft_consensus.cc:2749] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:00:09.405230  4653 raft_consensus.cc:2749] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 13:00:09.392908  4518 raft_consensus.cc:515] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:09.443614  4518 leader_election.cc:290] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:09.441670  2334 raft_consensus.cc:1240] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 5. Current term is 6. Ops: []
I20260709 13:00:09.446573  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:09.448138  4676 raft_consensus.cc:697] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 7 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:00:09.430677  1747 raft_consensus.cc:1240] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 4. Current term is 5. Ops: []
I20260709 13:00:09.449829  2337 raft_consensus.cc:2468] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 5.
I20260709 13:00:09.454725  4534 raft_consensus.cc:3055] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:00:09.455145  4534 raft_consensus.cc:740] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:00:09.456027  4534 consensus_queue.cc:260] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:09.412199  4648 raft_consensus.cc:515] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:09.456854  4070 raft_consensus.cc:515] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:09.416000  4629 consensus_queue.cc:1059] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.043s
I20260709 13:00:09.459515  4648 leader_election.cc:290] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 7 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:09.380867  4569 raft_consensus.cc:515] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:09.460379  1643 leader_election.cc:304] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:00:09.461505  4070 leader_election.cc:290] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 6 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:09.461598  1597 raft_consensus.cc:1240] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 6. Current term is 7. Ops: []
I20260709 13:00:09.471335  1571 leader_election.cc:304] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:00:09.472229  4648 raft_consensus.cc:2749] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:00:09.472764  1571 leader_election.cc:304] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:00:09.438621  1598 raft_consensus.cc:2468] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 5.
I20260709 13:00:09.473667  4310 raft_consensus.cc:493] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:09.474185  4310 raft_consensus.cc:515] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:09.449246  4676 consensus_queue.cc:237] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:09.475425  4648 raft_consensus.cc:2749] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:00:09.422457  4262 raft_consensus.cc:3060] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:00:09.482647  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
W20260709 13:00:09.490584  1645 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:09.491122  4635 raft_consensus.cc:2749] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:00:09.441666  4512 raft_consensus.cc:3055] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:00:09.495277  4512 raft_consensus.cc:740] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:00:09.465152  4671 raft_consensus.cc:3055] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 LEADER]: Stepping down as leader of term 7
W20260709 13:00:09.496272  2300 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:09.496526  4671 raft_consensus.cc:740] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:00:09.497571  4671 consensus_queue.cc:260] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:00:09.498980  4682 log.cc:927] Time spent T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.141s	user 0.000s	sys 0.003s
W20260709 13:00:09.499202  2300 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:09.465795  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:09.500010  1676 raft_consensus.cc:3060] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Advancing to term 6
I20260709 13:00:09.496274  4310 leader_election.cc:290] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:09.467701  4593 raft_consensus.cc:2804] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:00:09.503346  4593 raft_consensus.cc:493] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:09.503808  4593 raft_consensus.cc:3060] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:00:09.472903  1597 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 507) took 1068 ms. Trace:
I20260709 13:00:09.461760  4569 leader_election.cc:290] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:09.438026  4574 consensus_queue.cc:237] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:09.506649  1597 rpcz_store.cc:276] 0709 13:00:08.404657 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:08.735270 (+330613us) service_pool.cc:224] Handling call
0709 13:00:08.737987 (+  2717us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:09.471623 (+733636us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:09.472614 (+   991us) spinlock_profiling.cc:243] Waited 723 ms on lock 0x7b54001009d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:00:09.472820 (+   206us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1663413504}
I20260709 13:00:09.474505  1598 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40166 (request call id 1912) took 1145 ms. Trace:
I20260709 13:00:09.510834  1598 rpcz_store.cc:276] 0709 13:00:08.328911 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:08.330394 (+  1483us) service_pool.cc:224] Handling call
0709 13:00:09.474247 (+1143853us) spinlock_profiling.cc:243] Waited 1.09 s on lock 0x7b54001e0354. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:00:09.474391 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2501795712}
I20260709 13:00:09.474272  1571 leader_election.cc:304] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:00:09.463267  1643 leader_election.cc:304] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:09.512974  1571 leader_election.cc:304] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:00:09.514074  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:09.514932  1571 leader_election.cc:304] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
W20260709 13:00:09.516696  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 115381us
W20260709 13:00:09.518311  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 55115.4us
I20260709 13:00:09.520483  1571 leader_election.cc:304] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:00:09.496305  4512 consensus_queue.cc:260] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:09.521636  4583 raft_consensus.cc:2749] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:00:09.521696  4684 raft_consensus.cc:2749] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 13:00:09.522826  4512 raft_consensus.cc:3060] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:00:09.522245  4569 raft_consensus.cc:2749] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:00:09.521629  1643 leader_election.cc:304] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:00:09.434979  4679 raft_consensus.cc:3060] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:00:09.498973  4671 raft_consensus.cc:3060] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:00:09.526484  4487 raft_consensus.cc:2749] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 13:00:09.464339  4549 leader_election.cc:290] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:09.528848  1571 leader_election.cc:304] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:00:09.519780  4310 raft_consensus.cc:3060] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:00:09.519940  4654 raft_consensus.cc:2804] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 13:00:09.524190  4635 raft_consensus.cc:2749] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 13:00:09.435135  4607 raft_consensus.cc:515] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:09.556810  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:00:09.538348  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 327995us
W20260709 13:00:09.549777  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:09.537343  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:09.558295  1597 raft_consensus.cc:2468] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 5.
I20260709 13:00:09.559909  1569 leader_election.cc:304] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:00:09.563799  1598 raft_consensus.cc:1275] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:00:09.564740  1753 raft_consensus.cc:2468] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 5.
I20260709 13:00:09.520331  4648 consensus_queue.cc:1048] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.017s
I20260709 13:00:09.527889  4530 raft_consensus.cc:2749] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 13:00:09.573386  1569 leader_election.cc:304] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:00:09.576138  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:09.582363  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:09.583160  1598 raft_consensus.cc:2468] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 4.
I20260709 13:00:09.586477  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:09.590725  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:09.586169  4654 raft_consensus.cc:697] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 6 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:00:09.574167  4628 consensus_queue.cc:1048] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.020s
I20260709 13:00:09.579871  4549 consensus_queue.cc:1059] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.016s
I20260709 13:00:09.589668  1597 raft_consensus.cc:1275] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260709 13:00:09.572553  4500 raft_consensus.cc:2804] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:00:09.611625  1597 raft_consensus.cc:1275] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 13:00:09.562094  4667 consensus_queue.cc:1048] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.035s
I20260709 13:00:09.603577  4583 raft_consensus.cc:2804] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:00:09.621251  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:09.629201  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:09.623814  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:09.629532  2336 raft_consensus.cc:1275] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260709 13:00:09.628046  1672 raft_consensus.cc:2468] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 8.
I20260709 13:00:09.630069  1598 raft_consensus.cc:2468] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 6.
I20260709 13:00:09.630024  4603 raft_consensus.cc:493] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:09.637418  1643 leader_election.cc:304] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1
W20260709 13:00:09.617606  2330 consensus_meta.cc:287] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.522s	user 0.014s	sys 0.000s
I20260709 13:00:09.651803  1646 leader_election.cc:304] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:00:09.661463  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:09.662366  1601 raft_consensus.cc:3060] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:00:09.663734  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:09.665438  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:00:09.672202  2299 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:00:09.673225  2299 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:00:09.673902  2299 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:09.617869  4500 raft_consensus.cc:493] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:00:09.686650  4642 raft_consensus.cc:493] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:09.687023  4666 consensus_queue.cc:1048] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:00:09.687480  4642 raft_consensus.cc:515] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } }
W20260709 13:00:09.679544  1644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:09.680716  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:09.686803  4537 raft_consensus.cc:493] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:09.608534  4487 consensus_queue.cc:1048] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.062s
I20260709 13:00:09.627664  4583 raft_consensus.cc:493] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Starting leader election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:00:09.696918  1646 leader_election.cc:304] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 13:00:09.698709  2301 leader_election.cc:304] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:00:09.608703  4547 consensus_queue.cc:1048] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.067s
I20260709 13:00:09.686182  4310 raft_consensus.cc:2749] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:00:09.702212  2301 leader_election.cc:304] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:00:09.667061  2330 raft_consensus.cc:2468] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 7.
I20260709 13:00:09.722512  1646 leader_election.cc:304] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:00:09.699883  4537 raft_consensus.cc:515] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:09.665750  4653 raft_consensus.cc:493] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:09.706131  4541 consensus_queue.cc:1059] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 13:00:09.692652  4563 consensus_queue.cc:1048] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.015s
W20260709 13:00:09.707511  1719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:09.712112  4583 raft_consensus.cc:3060] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:00:09.657315  4656 consensus_queue.cc:1048] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.060s
I20260709 13:00:09.714900  1601 raft_consensus.cc:2468] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 6.
I20260709 13:00:09.717980  2301 leader_election.cc:400] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 5 election: Vote denied by peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693) with higher term. Message: Invalid argument: T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 for earlier term 5. Current term is 6.
I20260709 13:00:09.711623  4680 raft_consensus.cc:2749] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:00:09.691123  4518 raft_consensus.cc:2804] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:00:09.686715  4500 raft_consensus.cc:3060] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:00:09.721151  1676 raft_consensus.cc:2468] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 6.
I20260709 13:00:09.695644  3977 raft_consensus.cc:493] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:09.738174  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:09.607143  4569 consensus_queue.cc:1059] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.058s
I20260709 13:00:09.746717  2299 leader_election.cc:304] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:00:09.746281  1646 leader_election.cc:304] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:00:09.654672  4603 raft_consensus.cc:515] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:09.735890  4530 raft_consensus.cc:2804] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:00:09.778133  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:09.786589  1745 raft_consensus.cc:2468] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 6.
I20260709 13:00:09.781037  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:00:09.758309  1645 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:09.754769  1746 raft_consensus.cc:1275] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 13:00:09.755198  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:09.692217  4672 consensus_queue.cc:1059] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 6.6, Next index: 7, Last known committed idx: 4, Time since last communication: 0.071s
I20260709 13:00:09.724530  4686 raft_consensus.cc:2804] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260709 13:00:09.791320  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 45142.1us
I20260709 13:00:09.792193  1747 raft_consensus.cc:3060] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:00:09.749261  4635 raft_consensus.cc:2764] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260709 13:00:09.777137  4353 raft_consensus.cc:2804] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 13:00:09.808363  1646 leader_election.cc:304] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:00:09.749357  4549 raft_consensus.cc:2749] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20260709 13:00:10.154726  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:09.749501  4518 raft_consensus.cc:493] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:00:09.886462  2300 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:00:09.959928  1717 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.549s	user 0.018s	sys 0.016s
W20260709 13:00:10.155812  1717 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.745s	user 0.024s	sys 0.020s
I20260709 13:00:09.753875  4246 consensus_queue.cc:1048] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.072s
I20260709 13:00:09.803369  4580 raft_consensus.cc:2804] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:00:09.722729  4654 consensus_queue.cc:237] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:09.739506  4653 raft_consensus.cc:515] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:09.754060  4666 consensus_queue.cc:1059] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.072s
I20260709 13:00:09.800146  4686 raft_consensus.cc:493] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
W20260709 13:00:09.959102  1599 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.532s	user 0.006s	sys 0.033s
W20260709 13:00:10.127594  1718 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.699s	user 0.006s	sys 0.006s
W20260709 13:00:10.168586  1718 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.740s	user 0.007s	sys 0.006s
W20260709 13:00:10.169556  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 742132us
I20260709 13:00:10.138242  2298 leader_election.cc:304] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:10.139973  4648 raft_consensus.cc:493] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:10.142210  4642 leader_election.cc:290] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:09.779333  3977 raft_consensus.cc:515] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:09.802035  4618 raft_consensus.cc:515] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:09.793579  4607 leader_election.cc:290] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 7 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:09.792584  4530 raft_consensus.cc:493] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:00:10.096839  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 343741us
I20260709 13:00:09.811730  4070 raft_consensus.cc:3055] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:00:09.801241  4680 consensus_queue.cc:1048] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.067s
W20260709 13:00:10.156160  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 745799us
I20260709 13:00:10.143841  4583 raft_consensus.cc:515] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:10.168520  1599 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.742s	user 0.015s	sys 0.034s
I20260709 13:00:10.136772  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:10.205554  1601 raft_consensus.cc:3060] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:00:10.144250  4603 leader_election.cc:290] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:10.183252  1718 leader_election.cc:304] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:00:10.143884  4534 raft_consensus.cc:3060] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Advancing to term 7
W20260709 13:00:10.192690  1645 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:10.144872  4353 raft_consensus.cc:697] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [term 6 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:00:10.153746  4628 raft_consensus.cc:493] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:10.161223  4518 raft_consensus.cc:3060] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:00:10.195184  4530 raft_consensus.cc:3060] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:00:10.211702  4541 raft_consensus.cc:2749] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:00:10.212158  4618 leader_election.cc:290] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 7 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:10.176402  4686 raft_consensus.cc:3060] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:00:10.181541  4656 raft_consensus.cc:493] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:10.182188  4549 consensus_queue.cc:1048] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.033s
I20260709 13:00:10.214761  4656 raft_consensus.cc:515] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:10.217239  4656 leader_election.cc:290] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:10.218036  4656 raft_consensus.cc:2749] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:00:10.219246  4353 consensus_queue.cc:237] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:10.219915  4583 leader_election.cc:290] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 5 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:10.181739  4648 raft_consensus.cc:515] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:10.220410  4628 raft_consensus.cc:515] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:10.222617  4628 leader_election.cc:290] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:10.222843  4686 raft_consensus.cc:515] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:10.225092  1718 leader_election.cc:304] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:10.225677  4603 raft_consensus.cc:493] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:00:10.226276  4603 raft_consensus.cc:515] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:10.226754  1718 leader_election.cc:304] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:10.227893  1751 raft_consensus.cc:2410] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 for term 6 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260709 13:00:10.228251  4550 raft_consensus.cc:2749] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:00:10.228523  4603 leader_election.cc:290] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:10.228994  2301 leader_election.cc:304] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:00:10.222193  4648 leader_election.cc:290] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:10.229013  4550 raft_consensus.cc:2749] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:00:10.231195  1718 leader_election.cc:304] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:00:10.231390  4537 leader_election.cc:290] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:10.232254  1718 leader_election.cc:304] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:10.232460  4653 leader_election.cc:290] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:10.232832  4672 consensus_queue.cc:1059] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 13:00:10.233078  4653 consensus_queue.cc:1048] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 13:00:10.233418  1718 leader_election.cc:304] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:10.234246  4574 raft_consensus.cc:2804] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:00:10.234467  4690 raft_consensus.cc:2804] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:00:10.234707  4574 raft_consensus.cc:493] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:10.235041  4690 raft_consensus.cc:493] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:00:10.243440  1747 raft_consensus.cc:2468] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 7.
I20260709 13:00:10.247349  1601 raft_consensus.cc:2468] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 7.
I20260709 13:00:10.249577  1718 leader_election.cc:304] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:10.238173  2301 leader_election.cc:304] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:00:10.257965  2301 leader_election.cc:304] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
W20260709 13:00:10.263780  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 455425us
I20260709 13:00:10.265455  1717 leader_election.cc:304] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:00:10.225378  4478 raft_consensus.cc:2749] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:00:10.303351  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:10.303531  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:10.304764  1680 raft_consensus.cc:1275] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 13:00:10.244565  4550 raft_consensus.cc:2749] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:00:10.310798  1717 leader_election.cc:304] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:00:10.235265  4574 raft_consensus.cc:3060] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:00:10.234858  4653 raft_consensus.cc:3055] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:00:10.200804  4070 raft_consensus.cc:740] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:00:10.314471  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:10.234258  4672 raft_consensus.cc:2749] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 13:00:10.323619  4478 consensus_queue.cc:1048] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 13:00:10.326169  4550 raft_consensus.cc:2804] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:00:10.326818  2329 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39726 (request call id 814) took 1252 ms. Trace:
I20260709 13:00:10.338140  4518 raft_consensus.cc:515] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:00:10.340395  4512 consensus_meta.cc:287] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.817s	user 0.026s	sys 0.000s
I20260709 13:00:10.341130  4530 raft_consensus.cc:515] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:10.247445  4690 raft_consensus.cc:3060] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Advancing to term 5
W20260709 13:00:10.353287  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 463186us
I20260709 13:00:10.364306  2298 leader_election.cc:304] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:10.225147  4686 leader_election.cc:290] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 4 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:00:10.352859  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 45551.1us
I20260709 13:00:10.345768  1674 raft_consensus.cc:2468] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 6.
I20260709 13:00:10.368439  2336 raft_consensus.cc:1275] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260709 13:00:10.320590  4703 raft_consensus.cc:493] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:10.320605  4262 raft_consensus.cc:2804] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260709 13:00:10.352341  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 102882us
I20260709 13:00:10.370370  1599 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 517) took 1882 ms. Trace:
W20260709 13:00:10.370463  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 222580us
I20260709 13:00:10.343420  4653 raft_consensus.cc:740] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:00:10.359808  4570 raft_consensus.cc:2749] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:00:10.375319  1599 rpcz_store.cc:276] 0709 13:00:08.488140 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:08.962866 (+474726us) service_pool.cc:224] Handling call
0709 13:00:08.981522 (+ 18656us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:08.981770 (+   248us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 13:00:08.982067 (+   297us) log.cc:844] Serialized 14 byte log entry
0709 13:00:09.426806 (+444739us) spinlock_profiling.cc:243] Waited 4.25 ms on lock 0x7b1800416680. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f4263057e07 00007f426beb2ab9 00007f426307469b 00007f426307293f 00007f4263074c02 00007f4265cfc20b 00007f4265d01d23 00007f4265d0223a 00007f426616ca2a
0709 13:00:10.369879 (+943073us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:00:10.369948 (+    69us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:00:10.370013 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:10.370166 (+   153us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:10.370297 (+   131us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":126,"prepare.run_cpu_time_us":70,"prepare.run_wall_time_us":2552,"raft.queue_time_us":24750,"raft.run_cpu_time_us":63,"raft.run_wall_time_us":63,"spinlock_wait_cycles":9773312,"thread_start_us":942768,"threads_started":2}
I20260709 13:00:10.369768  2299 leader_election.cc:304] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:00:10.375770  1643 leader_election.cc:304] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:00:10.349668  4550 raft_consensus.cc:493] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:10.363272  4656 raft_consensus.cc:2804] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:00:10.368199  4681 raft_consensus.cc:2749] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:00:10.395247  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:10.315634  4698 consensus_queue.cc:1059] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.080s
I20260709 13:00:10.365324  4580 raft_consensus.cc:697] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 5 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:00:10.372692  4582 raft_consensus.cc:2804] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:00:10.377622  4262 raft_consensus.cc:493] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:10.407536  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:10.411866  4262 raft_consensus.cc:3060] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:00:10.409508  1600 raft_consensus.cc:2468] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 6.
I20260709 13:00:10.416278  4070 consensus_queue.cc:260] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:10.416913  4700 log.cc:927] Time spent T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.074s	user 0.000s	sys 0.001s
I20260709 13:00:10.417771  4535 raft_consensus.cc:493] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:10.391170  4550 raft_consensus.cc:3060] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:00:10.418372  4535 raft_consensus.cc:515] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:10.349401  2329 rpcz_store.cc:276] 0709 13:00:09.073958 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:09.075338 (+  1380us) service_pool.cc:224] Handling call
0709 13:00:10.309177 (+1233839us) spinlock_profiling.cc:243] Waited 195 ms on lock 0x7b54004e02d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:00:10.326723 (+ 17546us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":448100736}
I20260709 13:00:10.423534  4653 consensus_queue.cc:260] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:10.391405  4656 raft_consensus.cc:493] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:10.425004  4656 raft_consensus.cc:3060] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:00:10.426409  1599 raft_consensus.cc:1240] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 5. Current term is 6. Ops: []
I20260709 13:00:10.426664  4698 raft_consensus.cc:3055] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:00:10.427122  4698 raft_consensus.cc:740] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:00:10.428138  4716 raft_consensus.cc:493] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:00:10.428047  4698 consensus_queue.cc:260] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:10.428737  4716 raft_consensus.cc:515] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:10.429383  4698 raft_consensus.cc:3060] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:00:10.432938  2298 leader_election.cc:304] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:00:10.434134  1718 leader_election.cc:304] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:10.435262  1718 leader_election.cc:304] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:10.436208  1718 leader_election.cc:304] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:10.431955  1569 leader_election.cc:304] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
W20260709 13:00:10.415750  4679 consensus_meta.cc:287] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.880s	user 0.020s	sys 0.000s
I20260709 13:00:10.399523  4709 raft_consensus.cc:2804] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:00:10.402251  4711 raft_consensus.cc:2749] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 13:00:10.388242  4680 consensus_queue.cc:1048] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.045s
I20260709 13:00:10.449182  1643 leader_election.cc:304] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:00:10.451086  1674 raft_consensus.cc:1275] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 5 index: 8. Preceding OpId from leader: term: 7 index: 9. (index mismatch)
I20260709 13:00:10.430900  4686 raft_consensus.cc:2804] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:00:10.442162  4478 raft_consensus.cc:2749] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 13:00:10.415220  4570 consensus_queue.cc:1059] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.038s
I20260709 13:00:10.441895  4618 raft_consensus.cc:493] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:10.445982  1718 leader_election.cc:304] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:10.468314  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:00:10.472934  2333 raft_consensus.cc:2376] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 5. Re-sending same reply.
W20260709 13:00:10.459475  4671 consensus_meta.cc:287] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.924s	user 0.020s	sys 0.007s
I20260709 13:00:10.385058  4703 raft_consensus.cc:515] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:10.476768  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:10.460659  4070 raft_consensus.cc:3060] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:00:10.474429  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
W20260709 13:00:10.476007  2300 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:10.452860  4709 raft_consensus.cc:493] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:10.478497  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:10.480894  2337 raft_consensus.cc:1240] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 5. Current term is 6. Ops: []
W20260709 13:00:10.453946  4593 consensus_meta.cc:287] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.950s	user 0.019s	sys 0.011s
I20260709 13:00:10.482112  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:10.460526  4710 raft_consensus.cc:2804] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:00:10.464538  4686 raft_consensus.cc:493] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:10.481026  1674 raft_consensus.cc:2468] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 4.
I20260709 13:00:10.459137  4679 raft_consensus.cc:515] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:10.469264  4628 consensus_queue.cc:1059] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.056s
I20260709 13:00:10.474205  4715 consensus_queue.cc:1048] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.019s
I20260709 13:00:10.469064  4704 raft_consensus.cc:493] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:10.469398  4549 raft_consensus.cc:2804] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:00:10.489764  4716 leader_election.cc:290] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:10.491305  4518 leader_election.cc:290] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 5 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:10.494019  1675 raft_consensus.cc:1240] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 5. Current term is 6. Ops: []
I20260709 13:00:10.494390  4574 raft_consensus.cc:515] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:10.495818  4719 raft_consensus.cc:493] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:10.496239  4671 raft_consensus.cc:493] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:10.496140  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:10.496620  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:10.467473  4697 raft_consensus.cc:2749] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 13:00:10.531250  1752 raft_consensus.cc:1240] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 6. Current term is 7. Ops: []
I20260709 13:00:10.531414  4671 raft_consensus.cc:515] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:10.532145  4549 raft_consensus.cc:493] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:10.532308  4679 leader_election.cc:290] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 7 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:10.532672  4549 raft_consensus.cc:3060] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:00:10.533277  4679 raft_consensus.cc:2749] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 13:00:10.533723  4671 leader_election.cc:290] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:10.534011  4679 consensus_queue.cc:1048] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 13:00:10.534381  1602 raft_consensus.cc:3060] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:00:10.479744  4618 raft_consensus.cc:515] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:10.538828  4618 leader_election.cc:290] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:10.533388  4703 leader_election.cc:290] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:10.539479  4541 raft_consensus.cc:3055] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:00:10.539901  4541 raft_consensus.cc:740] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:00:10.540421  4710 raft_consensus.cc:697] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 7 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:00:10.540192  4704 raft_consensus.cc:515] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:10.540901  4541 consensus_queue.cc:260] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:10.541442  4710 consensus_queue.cc:237] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:10.526297  4709 raft_consensus.cc:3060] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:00:10.539029  4571 raft_consensus.cc:3055] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:00:10.542414  4541 raft_consensus.cc:3060] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Advancing to term 5
W20260709 13:00:10.547189  4701 log.cc:927] Time spent T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.145s	user 0.003s	sys 0.000s
I20260709 13:00:10.519692  2331 raft_consensus.cc:1275] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 13:00:10.544092  4571 raft_consensus.cc:740] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:00:10.548419  4571 consensus_queue.cc:260] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:10.548787  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:00:10.526659  1672 raft_consensus.cc:2468] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 7.
I20260709 13:00:10.508966  1678 raft_consensus.cc:1240] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 6. Current term is 7. Ops: []
I20260709 13:00:10.550384  1680 raft_consensus.cc:2468] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 8.
I20260709 13:00:10.567085  1679 raft_consensus.cc:2468] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 5.
I20260709 13:00:10.570891  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:10.572362  1601 raft_consensus.cc:2468] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 7.
I20260709 13:00:10.573776  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:00:10.578758  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 268069us
I20260709 13:00:10.579681  1601 raft_consensus.cc:2468] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 8.
I20260709 13:00:10.581524  1601 raft_consensus.cc:1240] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 3. Current term is 4. Ops: []
I20260709 13:00:10.580256  1643 leader_election.cc:304] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:00:10.584321  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:10.588330  1601 raft_consensus.cc:3060] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:00:10.583292  1643 leader_election.cc:304] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:00:10.600391  1602 raft_consensus.cc:2468] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 5.
W20260709 13:00:10.523768  4500 consensus_meta.cc:287] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.746s	user 0.009s	sys 0.010s
I20260709 13:00:10.604918  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:10.559085  4686 raft_consensus.cc:3060] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:00:10.486305  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:00:10.474748  4718 raft_consensus.cc:2804] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:00:10.554057  4715 raft_consensus.cc:493] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:10.561113  4603 raft_consensus.cc:493] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
W20260709 13:00:10.623493  1645 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:10.571384  4704 leader_election.cc:290] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:10.624083  1601 raft_consensus.cc:2468] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 6.
I20260709 13:00:10.598852  4618 raft_consensus.cc:2804] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:00:10.605542  4628 raft_consensus.cc:2804] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:00:10.617660  1602 raft_consensus.cc:1240] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 4. Current term is 5. Ops: []
I20260709 13:00:10.486183  4478 raft_consensus.cc:2804] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:00:10.603308  1643 leader_election.cc:304] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:00:10.621533  4718 raft_consensus.cc:493] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:00:10.638006  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 156072us
I20260709 13:00:10.570819  4593 raft_consensus.cc:515] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:10.653520  4570 consensus_queue.cc:1059] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 13:00:10.517866  4690 raft_consensus.cc:515] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:10.656731  1602 raft_consensus.cc:1275] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260709 13:00:10.656746  1718 leader_election.cc:304] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:10.622673  2329 raft_consensus.cc:2376] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 7. Re-sending same reply.
I20260709 13:00:10.611042  4262 raft_consensus.cc:515] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:10.631196  4618 raft_consensus.cc:493] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:10.571756  4580 consensus_queue.cc:237] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:10.612288  4500 raft_consensus.cc:515] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:10.545295  4719 raft_consensus.cc:515] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:10.633958  4628 raft_consensus.cc:493] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:10.669855  1677 raft_consensus.cc:1240] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 5. Current term is 6. Ops: []
I20260709 13:00:10.636484  4478 raft_consensus.cc:493] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:10.674225  1718 leader_election.cc:304] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:10.649230  4518 raft_consensus.cc:2804] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Leader election won for term 5
W20260709 13:00:10.764966  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 90875.2us
I20260709 13:00:10.766698  1718 leader_election.cc:304] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:10.747077  4635 raft_consensus.cc:493] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:10.772943  4716 raft_consensus.cc:493] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:10.767954  4478 raft_consensus.cc:3060] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:00:10.773504  4716 raft_consensus.cc:515] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:10.634531  4603 raft_consensus.cc:515] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:10.779915  1605 raft_consensus.cc:1240] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 5. Current term is 6. Ops: []
I20260709 13:00:10.780699  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:10.781399  1602 raft_consensus.cc:2468] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 5.
I20260709 13:00:10.784614  1643 leader_election.cc:304] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:00:10.786793  4534 raft_consensus.cc:2804] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:00:10.787343  4534 raft_consensus.cc:493] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:10.787801  4534 raft_consensus.cc:3060] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:00:10.796115  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:10.823662  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:10.825740  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:10.662144  4718 raft_consensus.cc:3060] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:00:10.760794  4618 raft_consensus.cc:3060] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:00:10.746574  4680 raft_consensus.cc:493] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:10.746011  4570 raft_consensus.cc:2749] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:00:10.762243  4650 raft_consensus.cc:493] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:10.768567  4628 raft_consensus.cc:3060] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:00:10.628472  4715 raft_consensus.cc:515] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:10.867851  1571 leader_election.cc:304] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
W20260709 13:00:10.870429  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 103828us
I20260709 13:00:10.828120  4716 leader_election.cc:290] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:10.804775  4574 leader_election.cc:290] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 6 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:10.842155  4676 raft_consensus.cc:493] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:00:10.769932  4656 raft_consensus.cc:515] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:10.859391  4730 raft_consensus.cc:2764] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20260709 13:00:10.898250  2335 raft_consensus.cc:2468] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 5.
I20260709 13:00:10.772989  4635 raft_consensus.cc:515] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:10.768831  4723 consensus_queue.cc:1048] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.134s
I20260709 13:00:10.885883  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:00:10.891801  4671 raft_consensus.cc:493] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:10.941468  4728 raft_consensus.cc:2749] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:00:10.894251  4731 consensus_queue.cc:1048] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.066s
I20260709 13:00:10.945416  4593 leader_election.cc:290] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 6 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:00:10.948257  4729 log.cc:927] Time spent T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.062s	user 0.000s	sys 0.001s
I20260709 13:00:10.948922  4574 consensus_queue.cc:1059] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 13:00:10.950834  2334 raft_consensus.cc:1275] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260709 13:00:10.957909  4719 leader_election.cc:290] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:00:10.942751  4735 log.cc:927] Time spent T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.055s	user 0.000s	sys 0.002s
I20260709 13:00:10.874670  4550 raft_consensus.cc:515] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:10.874111  4650 raft_consensus.cc:515] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:10.961484  4671 raft_consensus.cc:515] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:10.935881  4716 consensus_queue.cc:1048] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.044s
I20260709 13:00:10.898602  4676 raft_consensus.cc:515] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:10.973834  4698 consensus_meta.cc:287] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.544s	user 0.004s	sys 0.011s
I20260709 13:00:10.993505  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:10.964327  2329 raft_consensus.cc:2376] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 6. Re-sending same reply.
I20260709 13:00:10.980443  4574 consensus_queue.cc:1243] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer 3b879799899a499c8d409ee9f1277cd1 log is divergent from this leader: its last log entry 7.9 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
I20260709 13:00:10.981209  4719 raft_consensus.cc:493] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:11.004882  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:11.005841  2334 raft_consensus.cc:2468] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 8.
I20260709 13:00:11.015759  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:00:11.017032  2334 raft_consensus.cc:3060] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:00:10.866129  4680 raft_consensus.cc:515] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:11.024003  4550 leader_election.cc:290] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 7 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 13:00:11.028129  4733 log.cc:927] Time spent T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.055s	user 0.000s	sys 0.000s
W20260709 13:00:11.031026  4725 log.cc:927] Time spent T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.176s	user 0.000s	sys 0.001s
W20260709 13:00:11.032291  2300 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:11.032999  4671 leader_election.cc:290] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:11.038039  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:11.039173  2329 raft_consensus.cc:2368] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 3. Current term is 4.
I20260709 13:00:11.040333  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:00:11.053241  1645 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:11.057596  4518 raft_consensus.cc:697] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
W20260709 13:00:11.060186  1719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:11.041026  2329 raft_consensus.cc:2468] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 7.
I20260709 13:00:11.068537  4635 leader_election.cc:290] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:00:11.069339  4549 consensus_meta.cc:287] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.536s	user 0.007s	sys 0.007s
I20260709 13:00:11.069662  4656 leader_election.cc:290] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 8 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:11.069787  4549 raft_consensus.cc:515] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:11.079080  1747 raft_consensus.cc:3055] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:00:11.079540  1747 raft_consensus.cc:740] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:00:11.080391  1747 consensus_queue.cc:260] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.9, Last appended by leader: 9, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:11.081161  4709 consensus_meta.cc:287] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.538s	user 0.005s	sys 0.004s
I20260709 13:00:11.082288  4549 leader_election.cc:290] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 7 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:11.083005  4478 raft_consensus.cc:515] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:11.084800  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:11.086573  1753 raft_consensus.cc:3060] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:00:11.090052  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:11.085175  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:11.090792  1750 raft_consensus.cc:3060] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:00:11.090996  1745 raft_consensus.cc:2468] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 6.
I20260709 13:00:11.092922  4715 leader_election.cc:290] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:11.059095  4518 consensus_queue.cc:237] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:11.085829  2334 raft_consensus.cc:2468] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 5.
I20260709 13:00:11.098129  1749 raft_consensus.cc:1240] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 6. Current term is 7. Ops: [3.5-6.6]
I20260709 13:00:11.098616  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:11.099367  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:11.100522  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:11.101325  1749 raft_consensus.cc:2393] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 7: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:00:11.061628  4534 raft_consensus.cc:515] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:11.102972  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:11.103724  1745 raft_consensus.cc:2468] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 7.
I20260709 13:00:11.104148  1750 raft_consensus.cc:2468] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 4.
I20260709 13:00:11.104628  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:11.105224  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:11.105252  1745 raft_consensus.cc:3060] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:00:11.102097  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:11.110980  1749 raft_consensus.cc:2468] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 6.
I20260709 13:00:11.105829  1750 raft_consensus.cc:3060] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:00:11.117885  4534 leader_election.cc:290] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 6 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:11.019294  4719 raft_consensus.cc:515] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:11.125510  2301 leader_election.cc:304] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:00:11.125813  4582 raft_consensus.cc:493] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:11.126348  4582 raft_consensus.cc:3060] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:00:11.126653  2301 leader_election.cc:304] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:00:11.126427  4618 raft_consensus.cc:515] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:11.128190  4710 raft_consensus.cc:2804] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:00:11.128612  4710 raft_consensus.cc:493] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:11.128959  4710 raft_consensus.cc:3060] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:00:11.131397  4739 raft_consensus.cc:2804] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:00:11.131961  4739 raft_consensus.cc:697] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 4 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:00:11.132936  4739 consensus_queue.cc:237] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:11.134882  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:11.135162  4310 raft_consensus.cc:3055] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:00:11.135612  4310 raft_consensus.cc:740] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:00:11.139912  4535 leader_election.cc:290] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:11.135579  1749 raft_consensus.cc:2468] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 4.
I20260709 13:00:11.143445  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:11.144239  2301 leader_election.cc:304] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
W20260709 13:00:11.147114  4734 log.cc:927] Time spent T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.097s	user 0.001s	sys 0.000s
I20260709 13:00:11.147706  4310 consensus_queue.cc:260] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:11.149098  4607 raft_consensus.cc:2804] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:00:11.149561  4607 raft_consensus.cc:493] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:11.149974  4607 raft_consensus.cc:3060] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:00:11.151001  4721 raft_consensus.cc:493] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:11.151618  1605 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 1937) took 1594 ms. Trace:
I20260709 13:00:11.151578  4721 raft_consensus.cc:515] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:11.152451  4714 raft_consensus.cc:493] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:00:11.153009  4714 raft_consensus.cc:515] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:11.154021  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:11.154619  1602 raft_consensus.cc:1275] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 13:00:11.154201  4721 leader_election.cc:290] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:11.154744  1599 raft_consensus.cc:2468] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 7.
I20260709 13:00:11.155488  4583 raft_consensus.cc:493] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:00:11.155994  1597 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 531) took 1234 ms. Trace:
W20260709 13:00:11.156157  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 566388us
I20260709 13:00:11.156840  4569 raft_consensus.cc:493] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:11.157075  4368 raft_consensus.cc:493] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:00:11.157478  4569 raft_consensus.cc:515] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:11.157639  4368 raft_consensus.cc:515] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:11.149415  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:00:11.159670  2333 raft_consensus.cc:2376] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 5. Re-sending same reply.
I20260709 13:00:11.157090  1569 leader_election.cc:304] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:00:11.156437  1597 rpcz_store.cc:276] 0709 13:00:09.921775 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:10.134796 (+213021us) service_pool.cc:224] Handling call
0709 13:00:10.135452 (+   656us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:10.135732 (+   280us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 13:00:10.366280 (+230548us) log.cc:844] Serialized 14 byte log entry
0709 13:00:11.155615 (+789335us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:00:11.155683 (+    68us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:00:11.155760 (+    77us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:11.155825 (+    65us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:11.155930 (+   105us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":570706,"prepare.queue_time_us":317807,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":47,"thread_start_us":407569,"threads_started":2,"wal-append.queue_time_us":376704}
I20260709 13:00:11.152076  1605 rpcz_store.cc:276] 0709 13:00:09.556854 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:09.557132 (+   278us) service_pool.cc:224] Handling call
0709 13:00:09.573002 (+ 15870us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:10.780381 (+1207379us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:11.151372 (+370991us) spinlock_profiling.cc:243] Waited 1.21 s on lock 0x7b540028a854. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:00:11.151552 (+   180us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2774780800}
I20260709 13:00:11.149566  1603 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 533) took 1012 ms. Trace:
I20260709 13:00:11.183763  1603 rpcz_store.cc:276] 0709 13:00:10.137629 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:10.137759 (+   130us) service_pool.cc:224] Handling call
0709 13:00:10.140862 (+  3103us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:00:10.141305 (+   443us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:00:10.141343 (+    38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:00:10.141899 (+   556us) log.cc:844] Serialized 26 byte log entry
0709 13:00:10.311540 (+169641us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:00:10.466925 (+155385us) log.cc:844] Serialized 14 byte log entry
0709 13:00:11.148919 (+681994us) spinlock_profiling.cc:243] Waited 14.7 ms on lock 0x7b1800416380. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f4262fefc0a 00007f426c72f219 00007f4263077a40 00007f42630724e4 00007f4263072455 00007f426615a060 00007f426616c716 00007f426617d287 00007f426617d208
0709 13:00:11.149139 (+   220us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:11.149226 (+    87us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:00:11.149270 (+    44us) raft_consensus.cc:1685] finished
0709 13:00:11.149365 (+    95us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:11.149498 (+   133us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":19569,"prepare.queue_time_us":174111,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":54,"raft.queue_time_us":683538,"raft.run_cpu_time_us":147,"raft.run_wall_time_us":150,"spinlock_wait_cycles":33905920,"thread_start_us":267349,"threads_started":2,"wal-append.queue_time_us":278664}
I20260709 13:00:11.189852  4353 raft_consensus.cc:493] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:11.190676  4353 raft_consensus.cc:515] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:11.146448  4070 consensus_meta.cc:287] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.659s	user 0.005s	sys 0.008s
I20260709 13:00:11.081681  4709 raft_consensus.cc:515] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:11.198280  1750 raft_consensus.cc:2468] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 7.
I20260709 13:00:11.199227  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:11.200301  1676 raft_consensus.cc:2468] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 5.
I20260709 13:00:11.200893  1646 leader_election.cc:304] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:11.201637  2299 leader_election.cc:304] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:00:11.202641  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:11.145787  4676 leader_election.cc:290] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:00:11.210038  4737 log.cc:927] Time spent T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.058s	user 0.000s	sys 0.001s
I20260709 13:00:11.211100  4602 raft_consensus.cc:493] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:11.211966  4602 raft_consensus.cc:515] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:11.212641  4680 leader_election.cc:290] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:11.212957  4718 raft_consensus.cc:515] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:11.214581  4650 leader_election.cc:290] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:11.214890  4070 raft_consensus.cc:2804] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:00:11.215346  4070 raft_consensus.cc:493] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:11.215502  4714 leader_election.cc:290] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:11.220072  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:11.220412  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:11.221325  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:11.223780  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:11.224171  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:00:11.235447  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 52262.3us
I20260709 13:00:11.236821  1569 leader_election.cc:304] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:00:11.237726  4487 raft_consensus.cc:2804] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:00:11.156191  4583 raft_consensus.cc:515] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:11.255332  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:11.238147  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:11.239502  1602 raft_consensus.cc:2393] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 5: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:00:11.256922  1599 raft_consensus.cc:2468] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 5.
I20260709 13:00:11.178812  4535 raft_consensus.cc:493] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:00:11.240029  1569 leader_election.cc:304] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:00:11.240999  1745 raft_consensus.cc:2468] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 8.
I20260709 13:00:11.259644  2301 leader_election.cc:304] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:00:11.144246  1749 raft_consensus.cc:2468] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 5.
I20260709 13:00:11.217412  4709 leader_election.cc:290] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 7 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:11.252187  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:11.254408  1598 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 1945) took 1097 ms. Trace:
I20260709 13:00:11.274806  1598 rpcz_store.cc:276] 0709 13:00:10.156683 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:10.158120 (+  1437us) service_pool.cc:224] Handling call
0709 13:00:10.177204 (+ 19084us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:00:10.411685 (+234481us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:00:10.411755 (+    70us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:00:10.436812 (+ 25057us) log.cc:844] Serialized 52 byte log entry
0709 13:00:10.660272 (+223460us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:00:10.832590 (+172318us) log.cc:844] Serialized 14 byte log entry
0709 13:00:11.253878 (+421288us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:11.254008 (+   130us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:00:11.254074 (+    66us) raft_consensus.cc:1685] finished
0709 13:00:11.254095 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:11.254312 (+   217us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":332886,"prepare.queue_time_us":179023,"prepare.run_cpu_time_us":69,"prepare.run_wall_time_us":70,"raft.queue_time_us":398125,"raft.run_cpu_time_us":263,"raft.run_wall_time_us":9884,"thread_start_us":508307,"threads_started":3,"wal-append.queue_time_us":491495}
I20260709 13:00:11.277077  4518 raft_consensus.cc:493] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:11.238811  4070 raft_consensus.cc:3060] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:00:11.269398  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:11.284741  1750 raft_consensus.cc:2468] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 4.
I20260709 13:00:11.266841  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:00:11.241319  4686 consensus_meta.cc:287] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.633s	user 0.008s	sys 0.007s
I20260709 13:00:11.290091  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:11.291255  1745 raft_consensus.cc:2393] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 6: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:00:11.252811  4618 leader_election.cc:290] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 8 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:11.219102  4534 raft_consensus.cc:2804] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:00:11.278954  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
W20260709 13:00:11.257036  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 382648us
I20260709 13:00:11.264478  1646 leader_election.cc:304] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:00:11.255219  4478 leader_election.cc:290] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:11.286087  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:11.247038  1753 raft_consensus.cc:2468] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 7.
W20260709 13:00:11.293438  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 53475.1us
I20260709 13:00:11.297689  1718 leader_election.cc:304] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:00:11.305444  1569 leader_election.cc:304] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
W20260709 13:00:11.306124  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 438322us
I20260709 13:00:11.307418  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:11.255677  4676 consensus_queue.cc:1048] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.039s
I20260709 13:00:11.322223  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:11.306418  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:11.333547  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:11.307333  1718 leader_election.cc:304] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:00:11.284595  1604 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 1946) took 1127 ms. Trace:
I20260709 13:00:11.336640  4368 leader_election.cc:290] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:11.296249  4602 leader_election.cc:290] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:11.337620  4478 raft_consensus.cc:2749] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:00:11.337986  4719 leader_election.cc:290] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:11.338622  1646 leader_election.cc:304] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:11.308983  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:11.342008  1750 raft_consensus.cc:2393] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 7: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:00:11.289110  4714 raft_consensus.cc:2749] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:00:11.341876  1718 leader_election.cc:304] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:11.343379  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:11.344394  1569 leader_election.cc:400] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 3 pre-election: Vote denied by peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625) with higher term. Message: Invalid argument: T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 3. Current term is 4.
I20260709 13:00:11.341871  1745 raft_consensus.cc:2393] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 5: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:00:11.346099  1753 raft_consensus.cc:2468] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 5.
I20260709 13:00:11.283643  4518 raft_consensus.cc:515] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:11.289588  4353 leader_election.cc:290] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:11.402187  1753 raft_consensus.cc:1240] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 3. Current term is 4. Ops: []
I20260709 13:00:11.403723  1745 raft_consensus.cc:1240] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260709 13:00:11.403879  1753 raft_consensus.cc:1240] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 7. Current term is 8. Ops: [7.8-7.8]
I20260709 13:00:11.309810  4537 raft_consensus.cc:2804] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:00:11.271314  4535 raft_consensus.cc:515] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:00:11.392246  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 48021.6us
I20260709 13:00:11.410565  1569 leader_election.cc:304] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:00:11.382946  1749 raft_consensus.cc:3060] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Advancing to term 7
W20260709 13:00:11.348627  1571 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:11.397281  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:11.314759  4702 raft_consensus.cc:2749] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260709 13:00:11.417680  4628 consensus_meta.cc:287] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.530s	user 0.013s	sys 0.000s
I20260709 13:00:11.417459  4754 consensus_queue.cc:1048] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.139s
I20260709 13:00:11.418224  4628 raft_consensus.cc:515] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:11.419282  4753 consensus_queue.cc:1048] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.130s
I20260709 13:00:11.419714  4717 consensus_queue.cc:1059] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.232s
I20260709 13:00:11.393498  4549 consensus_queue.cc:1059] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.029s
I20260709 13:00:11.422727  4549 raft_consensus.cc:3055] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:00:11.423080  4549 raft_consensus.cc:740] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:00:11.424007  4549 consensus_queue.cc:260] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:11.425325  4549 raft_consensus.cc:3060] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:00:11.434986  1569 leader_election.cc:304] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:00:11.335104  1604 rpcz_store.cc:276] 0709 13:00:10.157063 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:10.157212 (+   149us) service_pool.cc:224] Handling call
0709 13:00:10.179450 (+ 22238us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:00:10.196881 (+ 17431us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:00:10.196928 (+    47us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:00:10.648318 (+451390us) spinlock_profiling.cc:243] Waited 11.5 ms on lock 0x7b2c00000658. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4264a5ba11 00007f4264a6b372 00007f426619da01 00007f426616243a 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2
0709 13:00:10.652037 (+  3719us) log.cc:844] Serialized 52 byte log entry
0709 13:00:10.771846 (+119809us) raft_consensus.cc:1645] Marking committed up to 3
0709 13:00:10.863203 (+ 91357us) log.cc:844] Serialized 14 byte log entry
0709 13:00:11.284083 (+420880us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:11.284208 (+   125us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:00:11.284255 (+    47us) raft_consensus.cc:1685] finished
0709 13:00:11.284349 (+    94us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:11.284526 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":430767,"prepare.queue_time_us":99028,"prepare.run_cpu_time_us":62,"prepare.run_wall_time_us":63,"raft.queue_time_us":392034,"raft.run_cpu_time_us":205,"raft.run_wall_time_us":207,"spinlock_wait_cycles":26348416,"thread_start_us":243802,"threads_started":2,"wal-append.queue_time_us":233168}
I20260709 13:00:11.310366  4709 raft_consensus.cc:2804] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:00:11.302983  4487 raft_consensus.cc:697] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
W20260709 13:00:11.400584  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 58776.2us
I20260709 13:00:11.310869  4569 leader_election.cc:290] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:11.408427  4736 raft_consensus.cc:2749] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:00:11.316009  4534 raft_consensus.cc:697] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:00:11.412151  1750 raft_consensus.cc:2468] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 6.
I20260709 13:00:11.362336  4719 raft_consensus.cc:2804] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:00:11.361882  4368 raft_consensus.cc:2804] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader election won for term 6
I20260709 13:00:11.415272  4759 raft_consensus.cc:493] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:11.459237  4718 leader_election.cc:290] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 9 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:11.295591  4686 raft_consensus.cc:515] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:11.438912  4702 raft_consensus.cc:2804] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:00:11.460295  1749 raft_consensus.cc:2468] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 7.
I20260709 13:00:11.464731  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:11.464831  1718 leader_election.cc:304] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:00:11.470700  4755 raft_consensus.cc:2804] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:00:11.518975  4755 raft_consensus.cc:493] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:11.519225  4723 raft_consensus.cc:3055] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:00:11.519420  4755 raft_consensus.cc:3060] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:00:11.519734  4723 raft_consensus.cc:740] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:00:11.520853  4723 consensus_queue.cc:260] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:11.522636  4635 raft_consensus.cc:493] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:00:11.523310  4635 raft_consensus.cc:515] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:00:11.524165  1751 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43156 (request call id 498) took 5018 ms (client timeout 2398 ms). Trace:
I20260709 13:00:11.524298  4757 consensus_queue.cc:1059] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.028s
W20260709 13:00:11.524641  1751 rpcz_store.cc:269] 0709 13:00:06.505543 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:07.283343 (+777800us) service_pool.cc:224] Handling call
0709 13:00:11.523859 (+4240516us) spinlock_profiling.cc:243] Waited 2.94 s on lock 0x7b54001c8254. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:00:11.524012 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":6768060800}
I20260709 13:00:11.525884  4635 leader_election.cc:290] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:11.527369  4628 leader_election.cc:290] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 9 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:11.528602  1748 raft_consensus.cc:1275] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 6. (term mismatch)
I20260709 13:00:11.528846  4390 raft_consensus.cc:493] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:11.407315  4537 raft_consensus.cc:493] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:11.476763  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:11.540800  1602 raft_consensus.cc:3060] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:00:11.496502  4719 raft_consensus.cc:493] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:00:11.486066  1569 leader_election.cc:304] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:00:11.493835  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:11.494347  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:11.464354  4709 raft_consensus.cc:493] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:00:11.440820  1645 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:00:11.551849  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 65876.8us
I20260709 13:00:11.506242  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:11.552676  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:11.553287  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:11.529078  1748 pending_rounds.cc:85] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701: Aborting all ops after (but not including) 5
I20260709 13:00:11.472973  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:11.530725  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
W20260709 13:00:11.536207  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 71468.2us
I20260709 13:00:11.482024  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:11.554768  1605 raft_consensus.cc:3060] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:00:11.554994  1604 raft_consensus.cc:2468] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 4.
I20260709 13:00:11.555473  1718 leader_election.cc:304] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:00:11.548127  2336 raft_consensus.cc:2368] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for earlier term 6. Current term is 7.
I20260709 13:00:11.548674  1603 raft_consensus.cc:3060] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Advancing to term 7
W20260709 13:00:11.550345  4750 log.cc:927] Time spent T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.134s	user 0.001s	sys 0.000s
I20260709 13:00:11.554127  1748 pending_rounds.cc:107] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701: Aborting uncommitted NO_OP operation due to leader change: 5.6
I20260709 13:00:11.556656  1748 raft_consensus.cc:2889] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:00:11.553432  1569 leader_election.cc:304] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:00:11.565723  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:11.566876  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:11.567140  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:11.573912  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:11.575979  1675 raft_consensus.cc:1275] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260709 13:00:11.578150  1680 raft_consensus.cc:2393] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 5: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:00:11.581254  1671 raft_consensus.cc:2393] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 7: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:00:11.522392  4723 raft_consensus.cc:3060] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:00:11.529819  4368 raft_consensus.cc:697] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:00:11.590993  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:11.592885  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:11.593909  1679 raft_consensus.cc:2468] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 5.
I20260709 13:00:11.593892  1597 raft_consensus.cc:2468] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 7.
I20260709 13:00:11.545176  4537 raft_consensus.cc:3060] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:00:11.545770  4598 raft_consensus.cc:2749] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 13:00:11.604044  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:11.554380  4719 raft_consensus.cc:3060] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Advancing to term 5
W20260709 13:00:11.598856  1717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:11.501292  4582 raft_consensus.cc:515] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:11.606205  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:11.613582  1679 raft_consensus.cc:3060] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:00:11.613781  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:11.544916  4702 raft_consensus.cc:697] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:00:11.643343  4702 consensus_queue.cc:237] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:11.645951  1679 raft_consensus.cc:2468] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 9.
I20260709 13:00:11.647486  4747 raft_consensus.cc:2804] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 13:00:11.537055  4759 raft_consensus.cc:515] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:11.544992  4390 raft_consensus.cc:515] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:11.577843  4753 raft_consensus.cc:2804] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:00:11.625593  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:11.656070  2334 raft_consensus.cc:2468] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 6.
I20260709 13:00:11.626041  1602 raft_consensus.cc:2468] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 6.
I20260709 13:00:11.578346  4478 raft_consensus.cc:2749] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 13:00:11.666862  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:11.668556  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:11.668710  1569 leader_election.cc:304] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:00:11.630283  1718 leader_election.cc:304] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:00:11.610984  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:11.631714  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:11.634979  4368 consensus_queue.cc:237] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:11.697340  1596 raft_consensus.cc:1240] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 6. Current term is 7. Ops: []
I20260709 13:00:11.585912  4717 raft_consensus.cc:2804] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:00:11.627996  4487 consensus_queue.cc:237] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:11.638891  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:00:11.584956  4718 consensus_queue.cc:1059] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.061s
I20260709 13:00:11.639402  4676 raft_consensus.cc:493] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:00:11.642283  2300 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:11.648166  4747 raft_consensus.cc:697] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 8 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:00:11.562173  4709 raft_consensus.cc:3060] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:00:11.678531  1569 leader_election.cc:304] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:00:11.681653  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:11.684104  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:11.716514  1597 raft_consensus.cc:3060] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:00:11.685787  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:11.686939  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:11.719966  1598 raft_consensus.cc:2468] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 5.
I20260709 13:00:11.695294  4598 raft_consensus.cc:3060] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:00:11.658775  4390 leader_election.cc:290] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:11.701367  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:11.649742  4583 leader_election.cc:290] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:11.660586  4753 raft_consensus.cc:493] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:11.707345  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:00:11.708978  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 186020us
I20260709 13:00:11.709924  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:00:11.744657  1644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:00:11.710165  4726 log.cc:927] Time spent T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.053s	user 0.001s	sys 0.000s
I20260709 13:00:11.654481  4754 raft_consensus.cc:493] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:11.712278  1605 raft_consensus.cc:2468] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 7.
I20260709 13:00:11.613090  4749 raft_consensus.cc:3060] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:00:11.663944  4759 leader_election.cc:290] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 13:00:11.715518  1645 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:11.709682  4763 raft_consensus.cc:493] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:00:11.690732  4710 consensus_meta.cc:287] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.561s	user 0.010s	sys 0.005s
I20260709 13:00:11.677038  2331 raft_consensus.cc:2468] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 7.
I20260709 13:00:11.707360  4603 leader_election.cc:290] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:11.719131  4676 raft_consensus.cc:515] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:11.744820  1671 raft_consensus.cc:2468] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 6.
I20260709 13:00:11.623685  4534 consensus_queue.cc:237] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:11.748241  4765 raft_consensus.cc:493] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:11.700686  4717 raft_consensus.cc:493] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Starting leader election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:00:11.766840  1603 raft_consensus.cc:2468] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 7.
I20260709 13:00:11.749562  4753 raft_consensus.cc:3060] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:00:11.768710  4518 leader_election.cc:290] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:00:11.745278  1644 connection.cc:256] RPC call timeout handler was delayed by -1.0081723689999649s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1535s for a response
I20260709 13:00:11.769188  4535 leader_election.cc:290] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:11.795650  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:11.802351  4603 raft_consensus.cc:2749] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:00:11.804464  4717 raft_consensus.cc:3060] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:00:11.835588  1644 leader_election.cc:304] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:00:11.768608  4754 raft_consensus.cc:515] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:11.837224  1644 leader_election.cc:304] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:00:11.846392  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:11.847476  1673 raft_consensus.cc:2468] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 8.
I20260709 13:00:11.794039  4710 raft_consensus.cc:515] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:11.818869  4686 leader_election.cc:290] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 7 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:11.861851  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:11.863057  1603 raft_consensus.cc:2393] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 7: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:00:11.826366  4628 raft_consensus.cc:493] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:00:11.864086  1752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55848 (request call id 1979) took 2466 ms. Trace:
I20260709 13:00:11.867852  1752 rpcz_store.cc:276] 0709 13:00:09.397831 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:09.420081 (+ 22250us) service_pool.cc:224] Handling call
0709 13:00:09.420390 (+   309us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:10.531703 (+1111313us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:11.863787 (+1332084us) spinlock_profiling.cc:243] Waited 1.11 s on lock 0x7b54000e4f54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:00:11.863988 (+   201us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2553947904}
I20260709 13:00:11.846809  4535 raft_consensus.cc:2804] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Leader election won for term 7
W20260709 13:00:11.896831  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 59700.6us
I20260709 13:00:11.913058  4676 leader_election.cc:290] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:11.865502  4603 raft_consensus.cc:493] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:00:11.799355  4763 raft_consensus.cc:515] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:11.931479  1597 raft_consensus.cc:2468] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 9.
I20260709 13:00:11.933743  1602 raft_consensus.cc:1240] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 8. Current term is 9. Ops: [4.4-8.5]
I20260709 13:00:11.899571  4702 raft_consensus.cc:493] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:11.800318  4747 consensus_queue.cc:237] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:11.859606  4580 raft_consensus.cc:493] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:11.925611  1605 raft_consensus.cc:1275] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 13:00:11.826884  4765 raft_consensus.cc:515] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:11.927431  1645 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:11.940176  1644 leader_election.cc:304] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:00:11.944918  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:11.895107  4582 leader_election.cc:290] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 7 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:11.948951  1599 raft_consensus.cc:1240] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 LEADER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 4. Current term is 5. Ops: []
I20260709 13:00:11.949380  1604 raft_consensus.cc:2393] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 5: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:00:11.951988  2329 raft_consensus.cc:2468] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 9.
W20260709 13:00:11.952947  4752 log.cc:927] Time spent T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.279s	user 0.002s	sys 0.000s
I20260709 13:00:11.938673  4773 raft_consensus.cc:2804] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:00:11.955216  4773 raft_consensus.cc:493] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:00:11.955878  4773 raft_consensus.cc:3060] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:00:11.923589  4563 raft_consensus.cc:2749] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:00:11.923521  4535 raft_consensus.cc:697] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:00:11.959139  1601 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40154 (request call id 1898) took 1377 ms. Trace:
W20260709 13:00:11.967296  4771 log.cc:927] Time spent T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.068s	user 0.000s	sys 0.001s
I20260709 13:00:11.968272  4707 consensus_queue.cc:1048] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.128s
I20260709 13:00:11.975891  2331 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49082 (request call id 665) took 1477 ms. Trace:
I20260709 13:00:11.976586  2331 rpcz_store.cc:276] 0709 13:00:10.498223 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:10.548420 (+ 50197us) service_pool.cc:224] Handling call
0709 13:00:11.975547 (+1427127us) spinlock_profiling.cc:243] Waited 656 ms on lock 0x7b54005843d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:00:11.975767 (+   220us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1508076928}
I20260709 13:00:11.988318  1673 raft_consensus.cc:1240] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 6. Current term is 7. Ops: []
I20260709 13:00:11.988587  1671 raft_consensus.cc:1240] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 8. Current term is 9. Ops: []
I20260709 13:00:11.887207  4628 raft_consensus.cc:515] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:11.989555  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:11.999284  1680 raft_consensus.cc:2393] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 6: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:00:11.965294  1601 rpcz_store.cc:276] 0709 13:00:10.582186 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:10.626604 (+ 44418us) service_pool.cc:224] Handling call
0709 13:00:10.626985 (+   381us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:00:11.157331 (+530346us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:00:11.157366 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:00:11.157948 (+   582us) log.cc:844] Serialized 26 byte log entry
0709 13:00:11.344715 (+186767us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:00:11.344780 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:11.344884 (+   104us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:00:11.958839 (+613955us) raft_consensus.cc:1685] finished
0709 13:00:11.958875 (+    36us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:11.959049 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":529889,"raft.queue_time_us":551954,"spinlock_wait_cycles":19225984,"thread_start_us":186293,"threads_started":1,"wal-append.queue_time_us":332934}
I20260709 13:00:11.946354  4702 raft_consensus.cc:515] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:12.002794  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:12.003592  4707 consensus_queue.cc:1048] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 13:00:12.009079  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:12.007894  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:11.992413  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
W20260709 13:00:11.952569  4762 log.cc:927] Time spent T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.166s	user 0.000s	sys 0.001s
I20260709 13:00:11.958065  4754 leader_election.cc:290] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:12.019517  1680 raft_consensus.cc:2393] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 7: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:00:12.017812  4782 raft_consensus.cc:2804] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:00:12.022681  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:12.022969  4782 raft_consensus.cc:697] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 5 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:00:12.005911  4582 raft_consensus.cc:493] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:12.023945  4582 raft_consensus.cc:515] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:12.024829  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:00:12.025652  2331 raft_consensus.cc:1803] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 4 LEARNER]: voting while tombstoned based on last-logged opid 4.5
I20260709 13:00:12.027093  4537 raft_consensus.cc:515] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:12.028108  4070 consensus_meta.cc:287] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.732s	user 0.012s	sys 0.002s
I20260709 13:00:12.032018  4070 raft_consensus.cc:515] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:12.033643  3977 leader_election.cc:290] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:12.028652  4765 leader_election.cc:290] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:12.028837  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:12.044843  1604 raft_consensus.cc:2410] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 13:00:12.030791  2337 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39726 (request call id 821) took 1563 ms. Trace:
W20260709 13:00:12.045805  1645 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:12.045255  2337 rpcz_store.cc:276] 0709 13:00:10.467545 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:10.467885 (+   340us) service_pool.cc:224] Handling call
0709 13:00:10.468275 (+   390us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:10.550024 (+ 81749us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:12.030435 (+1480411us) spinlock_profiling.cc:243] Waited 12.3 ms on lock 0x7b54005e01d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:00:12.030690 (+   255us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":28376320}
I20260709 13:00:12.046761  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:12.048079  1680 raft_consensus.cc:3060] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:00:11.937671  4603 raft_consensus.cc:515] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:12.050033  1676 raft_consensus.cc:1240] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 7. Current term is 8. Ops: [7.5-7.5]
I20260709 13:00:12.026221  2331 raft_consensus.cc:3060] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 4 LEARNER]: Advancing to term 6
I20260709 13:00:11.996811  4731 raft_consensus.cc:493] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:12.047380  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:12.070850  2337 raft_consensus.cc:2468] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 6.
I20260709 13:00:11.997037  4686 raft_consensus.cc:493] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:00:12.055778  4702 leader_election.cc:290] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:12.036109  4563 raft_consensus.cc:493] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:11.966058  4580 raft_consensus.cc:515] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:12.080256  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:12.091938  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:12.062355  4535 consensus_queue.cc:237] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:12.095954  1680 raft_consensus.cc:2468] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 5.
I20260709 13:00:12.081954  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:12.084115  4719 raft_consensus.cc:515] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:12.097095  1643 leader_election.cc:304] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:00:12.112475  1643 leader_election.cc:304] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:00:12.099113  1604 raft_consensus.cc:1275] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
W20260709 13:00:12.112475  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 439321us
I20260709 13:00:12.099689  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:00:12.115236  1718 leader_election.cc:304] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:00:12.115860  2337 raft_consensus.cc:2376] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 8. Re-sending same reply.
I20260709 13:00:12.117691  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:12.119657  2337 raft_consensus.cc:2393] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 5: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
W20260709 13:00:12.123144  2300 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:12.121490  1643 leader_election.cc:304] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:00:12.143107  1673 raft_consensus.cc:1240] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 4. Current term is 5. Ops: []
W20260709 13:00:12.187791  1679 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.534s	user 0.007s	sys 0.040s
W20260709 13:00:12.164352  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 485811us
W20260709 13:00:12.164669  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 431463us
W20260709 13:00:12.200277  1679 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.547s	user 0.007s	sys 0.042s
I20260709 13:00:12.203711  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:12.024552  4782 consensus_queue.cc:237] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:12.204922  1680 raft_consensus.cc:3060] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Advancing to term 7
W20260709 13:00:12.207013  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 262730us
I20260709 13:00:12.203946  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:00:12.208956  1645 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:12.210026  1644 leader_election.cc:304] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:12.210320  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:00:12.170154  4607 consensus_meta.cc:287] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 1.020s	user 0.014s	sys 0.004s
I20260709 13:00:12.201761  1569 leader_election.cc:304] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:00:12.139967  4683 raft_consensus.cc:2749] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:00:12.095428  4518 raft_consensus.cc:493] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:12.217376  4518 raft_consensus.cc:515] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:12.218994  1569 leader_election.cc:304] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:00:12.220346  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:12.089295  4686 raft_consensus.cc:515] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:12.222137  1676 raft_consensus.cc:2410] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 for term 7 because replica has last-logged OpId of term: 6 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20260709 13:00:12.124209  4629 raft_consensus.cc:2804] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:00:12.167454  4537 leader_election.cc:290] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 9 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:12.144866  4582 leader_election.cc:290] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:12.164690  4070 leader_election.cc:290] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 6 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:12.219545  2299 leader_election.cc:304] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:00:12.211333  4749 raft_consensus.cc:2749] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:00:12.124291  3977 consensus_queue.cc:1048] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.101s
I20260709 13:00:12.243395  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:12.121928  4731 raft_consensus.cc:515] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:12.148391  4783 raft_consensus.cc:2749] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 13:00:12.095654  4563 raft_consensus.cc:515] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:00:12.243222  4709 consensus_meta.cc:287] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.508s	user 0.016s	sys 0.000s
I20260709 13:00:12.227169  4768 raft_consensus.cc:493] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:12.223263  4368 raft_consensus.cc:2804] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Leader election won for term 9
I20260709 13:00:12.256639  1569 leader_election.cc:304] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:00:12.226809  4754 raft_consensus.cc:2804] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:00:12.224742  4710 leader_election.cc:290] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 8 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:12.227972  4629 raft_consensus.cc:493] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:00:12.238492  4549 consensus_meta.cc:287] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.813s	user 0.015s	sys 0.006s
I20260709 13:00:12.259851  4792 raft_consensus.cc:493] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:12.269589  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:12.244225  4628 leader_election.cc:290] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:12.220808  4607 raft_consensus.cc:515] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:12.301486  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 185402us
I20260709 13:00:12.258622  4070 raft_consensus.cc:2804] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:00:12.283550  1680 raft_consensus.cc:2468] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 7.
I20260709 13:00:12.221844  4790 consensus_queue.cc:1059] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.071s
W20260709 13:00:12.291226  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 169749us
I20260709 13:00:12.263497  4707 raft_consensus.cc:2804] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:00:12.258170  4582 raft_consensus.cc:493] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
W20260709 13:00:12.306217  4755 consensus_meta.cc:287] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.786s	user 0.014s	sys 0.003s
I20260709 13:00:12.275058  4629 raft_consensus.cc:3060] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:00:12.269160  4754 raft_consensus.cc:493] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:00:12.295017  4487 raft_consensus.cc:2749] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:00:12.294013  4368 raft_consensus.cc:697] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:00:12.329649  4368 consensus_queue.cc:237] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:12.331218  2331 raft_consensus.cc:2468] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 6 LEARNER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 6.
I20260709 13:00:12.315492  4070 raft_consensus.cc:493] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:12.332520  4070 raft_consensus.cc:3060] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:00:12.303603  4549 raft_consensus.cc:2749] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20260709 13:00:12.322190  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 112228us
I20260709 13:00:12.269243  4709 raft_consensus.cc:515] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:12.322885  4518 leader_election.cc:290] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:00:12.323165  4781 log.cc:927] Time spent T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.123s	user 0.000s	sys 0.001s
I20260709 13:00:12.320250  4754 raft_consensus.cc:3060] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Advancing to term 8
W20260709 13:00:12.327713  1571 leader_election.cc:336] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Timed out: RequestConsensusVote RPC to 127.0.107.195:40693 timed out after 1.991s (SENT)
I20260709 13:00:12.320286  2334 raft_consensus.cc:2368] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 7. Current term is 8.
I20260709 13:00:12.344079  1678 raft_consensus.cc:2468] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 6.
I20260709 13:00:12.317059  4707 raft_consensus.cc:493] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:12.306668  4686 leader_election.cc:290] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:12.346307  1644 leader_election.cc:304] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 13:00:12.351359  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 47381.2us
I20260709 13:00:12.351653  1643 leader_election.cc:304] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
W20260709 13:00:12.357185  1571 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:12.271431  4768 raft_consensus.cc:515] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:12.388031  4603 leader_election.cc:290] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:12.388777  4563 leader_election.cc:290] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:12.389683  4563 raft_consensus.cc:2749] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:00:12.389801  4768 leader_election.cc:290] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:12.390219  4709 leader_election.cc:290] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 7 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:12.389895  1644 leader_election.cc:304] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:00:12.391072  4709 raft_consensus.cc:2804] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Leader election won for term 6
I20260709 13:00:12.345364  4763 leader_election.cc:290] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:12.355518  4707 raft_consensus.cc:3060] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:00:12.299079  4792 raft_consensus.cc:515] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } }
W20260709 13:00:12.396186  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 269759us
I20260709 13:00:12.391911  1643 leader_election.cc:304] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:00:12.398522  1718 leader_election.cc:304] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 13:00:12.401435  1644 leader_election.cc:336] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625): Timed out: RequestConsensusVote RPC to 127.0.107.196:46625 timed out after 1.966s (SENT)
I20260709 13:00:12.402117  1644 leader_election.cc:304] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:00:12.319820  4755 raft_consensus.cc:515] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:12.405040  1718 leader_election.cc:304] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:12.335649  4582 raft_consensus.cc:515] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:12.384582  4580 leader_election.cc:290] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:12.411199  4582 leader_election.cc:290] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:12.399998  4709 raft_consensus.cc:697] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:00:12.396144  4731 leader_election.cc:290] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:12.407763  3977 raft_consensus.cc:2749] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 13:00:12.438867  2337 raft_consensus.cc:1240] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 5. Current term is 6. Ops: []
I20260709 13:00:12.403822  4790 consensus_queue.cc:1059] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.026s
I20260709 13:00:12.442883  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:12.419670  4563 raft_consensus.cc:2804] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:00:12.416494  4722 raft_consensus.cc:493] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:12.447229  4792 leader_election.cc:290] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:12.418829  4802 raft_consensus.cc:2749] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 13:00:12.444458  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:00:12.448453  2337 raft_consensus.cc:3060] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:00:12.451205  4607 leader_election.cc:290] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 8 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:00:12.451654  4717 consensus_meta.cc:287] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.612s	user 0.009s	sys 0.008s
I20260709 13:00:12.448782  4790 consensus_queue.cc:1243] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer 816fadcea8fc4893882f5f9b9f22a03d log is divergent from this leader: its last log entry 7.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20260709 13:00:12.452251  4717 raft_consensus.cc:515] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:12.444814  4731 consensus_queue.cc:1059] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.013s
I20260709 13:00:12.435509  4797 raft_consensus.cc:2804] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:00:12.447611  4722 raft_consensus.cc:515] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:12.458815  4773 consensus_meta.cc:287] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.502s	user 0.013s	sys 0.002s
W20260709 13:00:12.465758  4786 log.cc:927] Time spent T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.129s	user 0.001s	sys 0.000s
I20260709 13:00:12.465885  4790 raft_consensus.cc:3055] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:00:12.466595  4790 raft_consensus.cc:740] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:00:12.468225  4790 consensus_queue.cc:260] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:12.462401  4797 raft_consensus.cc:493] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:12.470872  4797 raft_consensus.cc:3060] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:00:12.517599  1603 raft_consensus.cc:1240] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 5. Current term is 6. Ops: []
W20260709 13:00:12.519506  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 117902us
I20260709 13:00:12.414076  1643 leader_election.cc:304] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
W20260709 13:00:12.524094  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 110005us
I20260709 13:00:12.525655  1643 leader_election.cc:304] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:00:12.528306  1643 leader_election.cc:304] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
W20260709 13:00:12.540683  2300 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:12.551666  4518 raft_consensus.cc:2749] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 13:00:12.444587  4563 raft_consensus.cc:493] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Starting leader election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:00:12.555225  4549 raft_consensus.cc:2804] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:00:12.556507  4549 raft_consensus.cc:697] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:00:12.557464  4690 leader_election.cc:290] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:12.469635  4773 raft_consensus.cc:515] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:12.468922  4754 raft_consensus.cc:515] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:12.575232  4549 consensus_queue.cc:237] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:12.579674  4717 leader_election.cc:290] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 6 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:12.583920  4500 leader_election.cc:290] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 6 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:12.589087  4708 raft_consensus.cc:2804] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader election won for term 9
I20260709 13:00:12.589332  4783 raft_consensus.cc:493] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:12.596995  4518 raft_consensus.cc:2804] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:00:12.597393  4783 raft_consensus.cc:515] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:12.598137  4518 raft_consensus.cc:697] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 7 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
W20260709 13:00:12.587363  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 57031.8us
I20260709 13:00:12.601568  4722 leader_election.cc:290] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 13:00:12.585536  4753 consensus_meta.cc:287] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.760s	user 0.004s	sys 0.012s
I20260709 13:00:12.605547  2332 raft_consensus.cc:1240] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 6. Current term is 8. Ops: []
I20260709 13:00:12.555254  4563 raft_consensus.cc:3060] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:00:12.616832  4754 leader_election.cc:290] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 8 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:12.619009  4755 leader_election.cc:290] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 8 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:12.461880  4709 consensus_queue.cc:237] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:12.624135  4310 raft_consensus.cc:3060] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:00:12.626755  4708 raft_consensus.cc:697] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
W20260709 13:00:12.602008  2298 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.359s	user 0.010s	sys 0.014s
W20260709 13:00:12.606920  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 194492us
I20260709 13:00:12.627460  4798 consensus_queue.cc:1048] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.002s
W20260709 13:00:12.631295  2298 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.388s	user 0.011s	sys 0.015s
I20260709 13:00:12.637667  2332 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39726 (request call id 828) took 2140 ms. Trace:
I20260709 13:00:12.638307  2332 rpcz_store.cc:276] 0709 13:00:10.497616 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:10.497898 (+   282us) service_pool.cc:224] Handling call
0709 13:00:12.604173 (+2106275us) spinlock_profiling.cc:243] Waited 4.09 ms on lock 0x7b5800178598. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f426303db71 00007f426303da22 00007f426bf56c09 00007f426b0b0ee0 00007f426b17a0ab 00007f426b17a131 00007f426b13bfc0 00007f426b126929 00007f426b1263d2 00007f426469dbd4
0709 13:00:12.604737 (+   564us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:12.629594 (+ 24857us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:12.637497 (+  7903us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":9411968}
I20260709 13:00:12.640398  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
W20260709 13:00:12.643855  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 1.40085e+06us
I20260709 13:00:12.649359  4547 raft_consensus.cc:3055] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:00:12.649786  4535 raft_consensus.cc:493] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:12.651208  4535 raft_consensus.cc:515] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:12.656510  4571 raft_consensus.cc:3060] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Advancing to term 6
W20260709 13:00:12.657006  1645 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:00:12.657740  1645 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:12.658802  4783 leader_election.cc:290] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:00:12.664052  4723 consensus_meta.cc:287] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 1.054s	user 0.010s	sys 0.010s
I20260709 13:00:12.666157  4773 leader_election.cc:290] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 5 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:12.667510  4535 leader_election.cc:290] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:12.642216  4753 raft_consensus.cc:515] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } }
I20260709 13:00:12.678628  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:12.682018  4518 consensus_queue.cc:237] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:12.650101  4547 raft_consensus.cc:740] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:00:12.707172  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:12.714749  4753 leader_election.cc:290] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 5 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:12.717062  2299 leader_election.cc:304] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:00:12.721084  4607 raft_consensus.cc:2804] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:00:12.721760  4607 raft_consensus.cc:697] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 7 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:00:12.722401  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:12.723275  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:12.723515  1599 raft_consensus.cc:2468] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 4.
I20260709 13:00:12.724421  1605 raft_consensus.cc:2393] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 5: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:00:12.729897  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:12.732587  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:00:12.630406  4708 consensus_queue.cc:237] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:12.705794  4070 raft_consensus.cc:515] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:12.725881  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:00:12.745962  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:12.717736  4797 raft_consensus.cc:515] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:12.758934  4607 consensus_queue.cc:237] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:12.760095  1677 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57092 (request call id 633) took 3273 ms. Trace:
I20260709 13:00:12.760571  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:12.770594  2332 raft_consensus.cc:3060] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:00:12.772490  1596 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 538) took 1619 ms. Trace:
W20260709 13:00:12.773514  2300 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:12.774688  1597 raft_consensus.cc:2468] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 6.
I20260709 13:00:12.775496  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:12.776397  1673 raft_consensus.cc:2468] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 4.
I20260709 13:00:12.776800  1679 raft_consensus.cc:2468] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 5.
I20260709 13:00:12.778246  1679 raft_consensus.cc:1240] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 4. Current term is 5. Ops: []
I20260709 13:00:12.781574  1643 leader_election.cc:304] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:00:12.782234  4719 leader_election.cc:290] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 5 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:12.778193  1673 raft_consensus.cc:1240] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 4. Current term is 5. Ops: []
I20260709 13:00:12.794148  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:12.760541  1677 rpcz_store.cc:276] 0709 13:00:09.486273 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:09.486481 (+   208us) service_pool.cc:224] Handling call
0709 13:00:09.486864 (+   383us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:10.764155 (+1277291us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:12.759480 (+1995325us) spinlock_profiling.cc:243] Waited 1.18 s on lock 0x7b5400091d54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:00:12.759887 (+   407us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2719780608}
I20260709 13:00:12.795167  1671 raft_consensus.cc:2468] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 4.
I20260709 13:00:12.766080  2298 leader_election.cc:304] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:12.772990  1596 rpcz_store.cc:276] 0709 13:00:11.153340 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:11.155528 (+  2188us) service_pool.cc:224] Handling call
0709 13:00:11.155850 (+   322us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:11.697782 (+541932us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:12.771848 (+1074066us) spinlock_profiling.cc:243] Waited 541 ms on lock 0x7b54002792d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:00:12.772234 (+   386us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1244675840}
W20260709 13:00:12.807992  1644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:12.816277  4703 raft_consensus.cc:493] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:12.795135  4653 raft_consensus.cc:3060] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:00:12.828061  4703 raft_consensus.cc:515] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:12.833918  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:12.843227  4759 raft_consensus.cc:2749] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:00:12.843426  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:12.837381  4549 raft_consensus.cc:2804] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:00:12.889854  1674 raft_consensus.cc:2468] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 6.
W20260709 13:00:12.890062  2335 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49082 (request call id 671) took 2144 ms (client timeout 1639 ms). Trace:
I20260709 13:00:12.884151  1747 raft_consensus.cc:3060] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Advancing to term 7
W20260709 13:00:12.890591  2335 rpcz_store.cc:269] 0709 13:00:10.745672 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:10.825381 (+ 79709us) service_pool.cc:224] Handling call
0709 13:00:12.889610 (+2064229us) spinlock_profiling.cc:243] Waited 71.1 ms on lock 0x7b54005bac54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:00:12.889800 (+   190us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":163621120}
I20260709 13:00:12.891264  4534 raft_consensus.cc:493] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:12.891494  1680 raft_consensus.cc:1240] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 8. Current term is 9. Ops: [8.6-8.6]
I20260709 13:00:12.891965  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:12.892196  1674 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52766 (request call id 2229) took 1327 ms. Trace:
I20260709 13:00:12.852617  4547 consensus_queue.cc:260] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:00:12.854797  1719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:12.855502  1643 leader_election.cc:304] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:00:12.892634  1674 rpcz_store.cc:276] 0709 13:00:11.565081 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:11.565298 (+   217us) service_pool.cc:224] Handling call
0709 13:00:12.891960 (+1326662us) spinlock_profiling.cc:243] Waited 1.26 s on lock 0x7b54000d8754. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:00:12.892103 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2901135104}
I20260709 13:00:12.890380  4549 raft_consensus.cc:493] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:12.893594  4547 raft_consensus.cc:3060] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:00:12.893929  4549 raft_consensus.cc:3060] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:00:12.896306  2328 raft_consensus.cc:1240] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 8 LEADER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 7. Current term is 8. Ops: []
I20260709 13:00:12.897127  2328 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39726 (request call id 827) took 2399 ms. Trace:
I20260709 13:00:12.897508  2328 rpcz_store.cc:276] 0709 13:00:10.497294 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:10.497434 (+   140us) service_pool.cc:224] Handling call
0709 13:00:12.895456 (+2398022us) spinlock_profiling.cc:243] Waited 4.54 ms on lock 0x7b5800178598. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f426303db71 00007f426303da22 00007f426bf56c09 00007f426b0b0ee0 00007f426b17a0ab 00007f426b17a131 00007f426b13bfc0 00007f426b126929 00007f426b1263d2 00007f426469dbd4
0709 13:00:12.895792 (+   336us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:12.896814 (+  1022us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:12.897066 (+   252us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":10453248}
I20260709 13:00:12.856698  1671 raft_consensus.cc:1240] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 LEADER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 6. Current term is 7. Ops: []
W20260709 13:00:12.899065  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 172897us
I20260709 13:00:12.899590  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:12.900607  2298 leader_election.cc:304] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:00:12.861186  4719 consensus_queue.cc:1059] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 13:00:12.894282  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:12.862517  4518 consensus_queue.cc:1059] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 13:00:12.857307  1718 leader_election.cc:304] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:00:12.862814  4635 consensus_queue.cc:1059] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:00:12.867967  1600 raft_consensus.cc:2468] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 6.
W20260709 13:00:12.868903  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 54846.7us
I20260709 13:00:12.879714  4802 raft_consensus.cc:3055] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:00:12.879911  4707 raft_consensus.cc:515] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:12.884404  1569 leader_election.cc:304] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:00:12.885097  4070 leader_election.cc:290] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 7 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:12.885941  2329 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49082 (request call id 676) took 1257 ms. Trace:
W20260709 13:00:12.887198  2334 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39738 (request call id 688) took 1225 ms (client timeout 1629 ms). Trace:
I20260709 13:00:12.891861  4534 raft_consensus.cc:515] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:12.892824  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:12.899356  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:12.851509  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:12.867296  4797 leader_election.cc:290] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 7 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:12.858754  4709 raft_consensus.cc:493] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
W20260709 13:00:12.908634  1719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:12.902398  2298 leader_election.cc:304] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:12.881783  4563 raft_consensus.cc:515] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:12.911098  2332 raft_consensus.cc:2468] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 9.
I20260709 13:00:12.886790  4782 consensus_queue.cc:1059] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 13:00:12.917076  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:12.921393  4580 raft_consensus.cc:2804] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:00:12.929394  4580 raft_consensus.cc:493] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:12.929891  4580 raft_consensus.cc:3060] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:00:12.916893  4690 raft_consensus.cc:493] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:00:12.933176  4690 raft_consensus.cc:515] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:12.902510  4728 raft_consensus.cc:493] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:00:12.902377  4530 leader_election.cc:290] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 6 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:12.923858  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:12.936296  1596 raft_consensus.cc:2393] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 6: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
W20260709 13:00:12.915715  2334 rpcz_store.cc:269] 0709 13:00:11.661189 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:11.668274 (+  7085us) service_pool.cc:224] Handling call
0709 13:00:12.886760 (+1218486us) spinlock_profiling.cc:243] Waited 651 ms on lock 0x7b54005802d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:00:12.886986 (+   226us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1496239360}
I20260709 13:00:12.925509  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:12.936982  4690 leader_election.cc:290] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:12.926667  1673 raft_consensus.cc:3060] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:00:12.937912  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:00:12.929909  4746 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.508s	user 0.005s	sys 0.005s
I20260709 13:00:12.938810  2334 raft_consensus.cc:2393] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 8: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
W20260709 13:00:12.938946  4746 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.517s	user 0.005s	sys 0.005s
I20260709 13:00:12.935891  4728 raft_consensus.cc:515] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:12.909092  4802 raft_consensus.cc:740] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:00:12.943818  4709 raft_consensus.cc:515] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:12.945322  4728 leader_election.cc:290] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:12.945438  4563 leader_election.cc:290] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 6 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:00:12.945667  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 61578.7us
I20260709 13:00:12.915477  2329 rpcz_store.cc:276] 0709 13:00:11.628575 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:11.767826 (+139251us) service_pool.cc:224] Handling call
0709 13:00:12.885441 (+1117615us) spinlock_profiling.cc:243] Waited 117 ms on lock 0x7b54005bca54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:00:12.885681 (+   240us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":268055296}
I20260709 13:00:12.946722  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:12.924140  4512 raft_consensus.cc:2804] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:00:12.897147  4790 raft_consensus.cc:3060] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:00:12.942430  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:12.952994  4721 raft_consensus.cc:493] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:00:12.953838  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 130060us
I20260709 13:00:12.958603  4709 leader_election.cc:290] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:12.961110  4478 raft_consensus.cc:2749] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:00:12.961601  1675 raft_consensus.cc:1275] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 13:00:12.956842  4512 raft_consensus.cc:493] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:12.962390  4512 raft_consensus.cc:3060] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Advancing to term 6
W20260709 13:00:12.962188  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 60015.9us
I20260709 13:00:12.973261  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:12.974366  1680 raft_consensus.cc:2393] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 7: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:00:12.974380  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:12.976562  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:12.980445  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:12.983192  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
W20260709 13:00:12.985749  4629 consensus_meta.cc:287] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.650s	user 0.005s	sys 0.012s
I20260709 13:00:12.986284  4629 raft_consensus.cc:515] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:12.986786  1605 raft_consensus.cc:3060] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:00:12.980438  2335 raft_consensus.cc:2468] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 7.
I20260709 13:00:12.958844  4707 leader_election.cc:290] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 9 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:12.987849  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:12.989813  1677 raft_consensus.cc:2368] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 5. Current term is 6.
I20260709 13:00:12.994005  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:12.990711  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:12.959550  4070 consensus_queue.cc:1059] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.032s
I20260709 13:00:12.996837  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
W20260709 13:00:12.997741  4598 consensus_meta.cc:287] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 1.211s	user 0.011s	sys 0.014s
I20260709 13:00:12.987845  4530 consensus_queue.cc:1059] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.011s
I20260709 13:00:12.998412  4650 raft_consensus.cc:2749] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260709 13:00:13.001724  4807 log.cc:927] Time spent T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.186s	user 0.000s	sys 0.002s
I20260709 13:00:13.007225  1748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43156 (request call id 511) took 5583 ms. Trace:
W20260709 13:00:13.009640  2336 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39726 (request call id 818) took 2543 ms (client timeout 1966 ms). Trace:
I20260709 13:00:13.011587  4262 leader_election.cc:290] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 4 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:13.014890  4722 consensus_queue.cc:1059] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 13:00:13.017606  1602 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40154 (request call id 1905) took 1168 ms. Trace:
I20260709 13:00:13.018134  2332 raft_consensus.cc:1275] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260709 13:00:13.024866  1680 raft_consensus.cc:2368] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 8. Current term is 9.
I20260709 13:00:13.028683  1602 rpcz_store.cc:276] 0709 13:00:11.848907 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:11.849241 (+   334us) service_pool.cc:224] Handling call
0709 13:00:11.933431 (+ 84190us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:00:11.934189 (+   758us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:13.017121 (+1082932us) spinlock_profiling.cc:243] Waited 83.6 ms on lock 0x7b54002175d0. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661618ad 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:00:13.017362 (+   241us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":192313728}
I20260709 13:00:13.025694  1748 rpcz_store.cc:276] 0709 13:00:07.423614 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:07.953209 (+529595us) service_pool.cc:224] Handling call
0709 13:00:11.528152 (+3574943us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:11.557305 (+ 29153us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:13.006727 (+1449422us) spinlock_profiling.cc:243] Waited 3.57 s on lock 0x7b54001c8250. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661618ad 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:00:13.006974 (+   247us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":8209964928}
I20260709 13:00:13.031194  1597 raft_consensus.cc:3060] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:00:13.033133  2332 raft_consensus.cc:1275] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260709 13:00:12.995813  4703 leader_election.cc:290] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:12.945451  4802 consensus_queue.cc:260] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:00:13.027045  2336 rpcz_store.cc:269] 0709 13:00:10.465721 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:10.465859 (+   138us) service_pool.cc:224] Handling call
0709 13:00:13.009148 (+2543289us) spinlock_profiling.cc:243] Waited 1.06 s on lock 0x7b54007d2654. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:00:13.009393 (+   245us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2443467136}
I20260709 13:00:13.040200  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:12.978052  4721 raft_consensus.cc:515] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:13.046963  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:13.034741  1718 leader_election.cc:304] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:13.053663  1569 leader_election.cc:304] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:00:13.053004  1671 raft_consensus.cc:3060] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:00:13.073051  1679 raft_consensus.cc:1275] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260709 13:00:13.075875  1678 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52766 (request call id 2228) took 1503 ms. Trace:
I20260709 13:00:13.044975  4530 consensus_queue.cc:1243] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer 3b879799899a499c8d409ee9f1277cd1 log is divergent from this leader: its last log entry 8.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20260709 13:00:13.076342  1678 rpcz_store.cc:276] 0709 13:00:11.552531 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:11.553019 (+   488us) service_pool.cc:224] Handling call
0709 13:00:13.044893 (+1491874us) spinlock_profiling.cc:243] Waited 741 ms on lock 0x7b54001e7dd4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:00:13.056085 (+ 11192us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1705143040}
I20260709 13:00:13.079811  4802 raft_consensus.cc:3060] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Advancing to term 9
W20260709 13:00:13.082602  2337 consensus_meta.cc:287] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.520s	user 0.014s	sys 0.003s
I20260709 13:00:13.083158  2337 raft_consensus.cc:2468] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 7.
I20260709 13:00:13.084743  2337 raft_consensus.cc:1240] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 4 LEADER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 3. Current term is 4. Ops: []
I20260709 13:00:13.087595  2330 raft_consensus.cc:1240] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 5. Current term is 7. Ops: []
I20260709 13:00:13.088673  4721 leader_election.cc:290] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:13.050203  4598 raft_consensus.cc:2749] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:00:13.033857  4534 leader_election.cc:290] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:13.103585  1602 raft_consensus.cc:2468] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 6.
W20260709 13:00:13.106674  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 66871.5us
W20260709 13:00:13.111811  2300 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:13.084764  4759 raft_consensus.cc:3055] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:00:13.122015  4721 raft_consensus.cc:2804] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:00:13.122786  4704 raft_consensus.cc:493] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:13.122982  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:00:13.123878  1673 raft_consensus.cc:2468] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 9.
I20260709 13:00:13.128489  1605 raft_consensus.cc:2468] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 6.
I20260709 13:00:13.152757  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:13.155844  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:00:13.156787  2337 raft_consensus.cc:2393] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 7: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:00:13.117777  2330 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39726 (request call id 820) took 2649 ms. Trace:
I20260709 13:00:13.127632  4537 raft_consensus.cc:2804] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:00:13.150141  2336 raft_consensus.cc:1803] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 6 LEARNER]: voting while tombstoned based on last-logged opid 4.5
I20260709 13:00:13.158205  1679 raft_consensus.cc:3055] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 7 LEADER]: Stepping down as leader of term 7
W20260709 13:00:13.168087  4758 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.488s	user 0.003s	sys 0.013s
W20260709 13:00:13.168542  4758 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.521s	user 0.003s	sys 0.013s
I20260709 13:00:13.161556  4704 raft_consensus.cc:515] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:13.162185  2330 rpcz_store.cc:276] 0709 13:00:10.467100 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:10.474516 (+  7416us) service_pool.cc:224] Handling call
0709 13:00:12.602915 (+2128399us) spinlock_profiling.cc:243] Waited 14.8 ms on lock 0x7b5800178598. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f426303db71 00007f426303da22 00007f426bf56c09 00007f426b0b0ee0 00007f426b17a0ab 00007f426b17a131 00007f426b13bfc0 00007f426b126929 00007f426b1263d2 00007f426469dbd4
0709 13:00:13.087179 (+484264us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:13.088039 (+   860us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:13.101169 (+ 13130us) spinlock_profiling.cc:243] Waited 484 ms on lock 0x7b540065ed50. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661618ad 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:00:13.115457 (+ 14288us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1146241664}
I20260709 13:00:13.172264  4704 leader_election.cc:290] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:13.172545  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:13.126215  4759 raft_consensus.cc:740] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:00:13.166720  1679 raft_consensus.cc:740] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:00:13.127699  4721 raft_consensus.cc:493] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:13.206183  1605 raft_consensus.cc:1275] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 13:00:13.207774  2336 raft_consensus.cc:2393] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 6 LEARNER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 6: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
W20260709 13:00:13.203033  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 52301.8us
I20260709 13:00:13.212625  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:13.181223  2328 raft_consensus.cc:1240] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 7 LEADER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 6. Current term is 7. Ops: []
I20260709 13:00:13.166210  4537 raft_consensus.cc:493] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:13.223127  1602 raft_consensus.cc:3060] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:00:13.217994  4703 consensus_queue.cc:1059] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.016s
I20260709 13:00:13.234148  2330 raft_consensus.cc:1240] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 5. Current term is 6. Ops: []
I20260709 13:00:13.203999  4721 raft_consensus.cc:3060] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:00:13.245606  4537 raft_consensus.cc:3060] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:00:13.296589  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:13.305178  1747 raft_consensus.cc:1275] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 6 index: 9. Preceding OpId from leader: term: 7 index: 9. (term mismatch)
I20260709 13:00:13.305823  1747 pending_rounds.cc:85] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701: Aborting all ops after (but not including) 8
I20260709 13:00:13.285874  4629 leader_election.cc:290] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 6 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:13.293023  1679 consensus_queue.cc:260] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:13.320183  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 10 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:13.312536  1747 pending_rounds.cc:107] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701: Aborting uncommitted NO_OP operation due to leader change: 6.9
W20260709 13:00:13.311204  1672 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57092 (request call id 636) took 2985 ms (client timeout 2232 ms). Trace:
I20260709 13:00:13.325102  1747 raft_consensus.cc:2889] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 13:00:13.325176  1672 rpcz_store.cc:269] 0709 13:00:10.303150 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:10.303280 (+   130us) service_pool.cc:224] Handling call
0709 13:00:13.271304 (+2968024us) spinlock_profiling.cc:243] Waited 222 ms on lock 0x7b54001f5c54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:00:13.288870 (+ 17566us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":509743488}
I20260709 13:00:13.330845  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:13.294823  4580 raft_consensus.cc:515] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:13.335047  4746 raft_consensus.cc:3055] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:00:13.335528  4746 raft_consensus.cc:740] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:00:13.336501  4746 consensus_queue.cc:260] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:13.337811  4746 raft_consensus.cc:3060] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:00:13.339078  4650 raft_consensus.cc:3055] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:00:13.339493  4650 raft_consensus.cc:740] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:00:13.340502  1644 leader_election.cc:304] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:00:13.340415  4650 consensus_queue.cc:260] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:13.342572  4709 consensus_queue.cc:1048] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 13:00:13.342891  4593 raft_consensus.cc:2804] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:00:13.343469  4593 raft_consensus.cc:697] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 7 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:00:13.344040  4709 consensus_queue.cc:1059] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 13:00:13.344806  4709 consensus_queue.cc:1243] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer 3b879799899a499c8d409ee9f1277cd1 log is divergent from this leader: its last log entry 4.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260709 13:00:13.344457  4593 consensus_queue.cc:237] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:13.345495  4549 raft_consensus.cc:515] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:00:13.351580  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:00:13.354647  1600 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40166 (request call id 1966) took 1849 ms (client timeout 2042 ms). Trace:
I20260709 13:00:13.354761  4601 raft_consensus.cc:493] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
W20260709 13:00:13.355090  1600 rpcz_store.cc:269] 0709 13:00:11.505529 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:11.505852 (+   323us) service_pool.cc:224] Handling call
0709 13:00:13.354337 (+1848485us) spinlock_profiling.cc:243] Waited 1.31 s on lock 0x7b54002170d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:00:13.354550 (+   213us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3023170304}
I20260709 13:00:13.355340  4601 raft_consensus.cc:515] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:00:13.356382  4653 consensus_meta.cc:287] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.517s	user 0.008s	sys 0.005s
I20260709 13:00:13.357259  2333 raft_consensus.cc:2468] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 7.
I20260709 13:00:13.357929  4779 raft_consensus.cc:493] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:00:13.358719  4817 log.cc:927] Time spent T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.290s	user 0.001s	sys 0.000s
I20260709 13:00:13.358911  1747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43156 (request call id 526) took 2284 ms. Trace:
I20260709 13:00:13.359030  4749 raft_consensus.cc:3055] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:00:13.359519  4749 raft_consensus.cc:740] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
W20260709 13:00:13.360710  4697 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 7 current term: 7 is higher.
I20260709 13:00:13.358539  4779 raft_consensus.cc:515] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:13.359665  4601 leader_election.cc:290] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:13.362880  1674 raft_consensus.cc:2368] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 5. Current term is 6.
I20260709 13:00:13.359369  1747 rpcz_store.cc:276] 0709 13:00:11.074683 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:11.078551 (+  3868us) service_pool.cc:224] Handling call
0709 13:00:11.078920 (+   369us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:12.882215 (+1803295us) spinlock_profiling.cc:243] Waited 176 ms on lock 0x7b58006f7ad0. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42660be0aa 00007f4266140447 00007f4266158fe7 00007f426615a7de 00007f4266164b49 00007f4266165416 00007f4266161d8f 00007f426616183e 00007f426b126419 00007f426469dbd4
0709 13:00:13.358529 (+476314us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:13.358809 (+   280us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":403753728}
I20260709 13:00:13.365795  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:00:13.368562  4820 log.cc:927] Time spent T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.068s	user 0.001s	sys 0.000s
I20260709 13:00:13.371219  4650 raft_consensus.cc:3060] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:00:13.325716  4823 consensus_queue.cc:1059] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.071s
I20260709 13:00:13.383198  4823 raft_consensus.cc:3055] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:00:13.383601  4823 raft_consensus.cc:740] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:00:13.384563  4823 consensus_queue.cc:260] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:13.360844  4749 consensus_queue.cc:260] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:13.358237  4512 raft_consensus.cc:515] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:13.412055  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:00:13.419075  1644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:13.434633  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:13.433439  2337 raft_consensus.cc:1275] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 13:00:13.407130  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:13.476773  1601 raft_consensus.cc:3060] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Advancing to term 7
W20260709 13:00:13.402737  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 79480.6us
I20260709 13:00:13.386001  1603 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40154 (request call id 1907) took 1517 ms. Trace:
I20260709 13:00:13.420225  4653 raft_consensus.cc:493] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:13.481046  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:00:13.481355  4653 raft_consensus.cc:515] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:13.482074  2337 raft_consensus.cc:2393] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 6: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:00:13.475276  1597 raft_consensus.cc:2468] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 9.
I20260709 13:00:13.427441  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:13.484187  1671 raft_consensus.cc:2468] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 5.
I20260709 13:00:13.412393  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
W20260709 13:00:13.380009  4816 log.cc:927] Time spent T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.272s	user 0.002s	sys 0.000s
I20260709 13:00:13.386132  4823 raft_consensus.cc:3060] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:00:13.479022  1603 rpcz_store.cc:276] 0709 13:00:11.868680 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:11.868835 (+   155us) service_pool.cc:224] Handling call
0709 13:00:11.869199 (+   364us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:12.519482 (+650283us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:13.385521 (+866039us) spinlock_profiling.cc:243] Waited 647 ms on lock 0x7b54001e05d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:00:13.385881 (+   360us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1488912128}
W20260709 13:00:13.392637  4822 log.cc:927] Time spent T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.081s	user 0.000s	sys 0.000s
W20260709 13:00:13.399113  4821 log.cc:927] Time spent T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.183s	user 0.000s	sys 0.001s
I20260709 13:00:13.482956  2298 leader_election.cc:304] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:13.507413  2337 raft_consensus.cc:1275] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260709 13:00:13.514191  2332 raft_consensus.cc:1240] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 5. Current term is 6. Ops: []
I20260709 13:00:13.492093  1600 raft_consensus.cc:2393] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 8: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:00:13.484656  1604 raft_consensus.cc:2468] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 6.
I20260709 13:00:13.423111  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:00:13.408847  4580 leader_election.cc:290] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 7 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:13.532476  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:00:13.531653  1644 leader_election.cc:304] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:13.547224  4749 raft_consensus.cc:3060] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:00:13.548002  1675 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52778 (request call id 1828) took 1559 ms. Trace:
I20260709 13:00:13.548480  1675 rpcz_store.cc:276] 0709 13:00:11.988316 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:11.989094 (+   778us) service_pool.cc:224] Handling call
0709 13:00:12.898824 (+909730us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:12.962042 (+ 63218us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:13.547092 (+585050us) spinlock_profiling.cc:243] Waited 49.7 ms on lock 0x7b54000d8754. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:00:13.547709 (+   617us) spinlock_profiling.cc:243] Waited 909 ms on lock 0x7b54000d8750. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661618ad 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:00:13.547876 (+   167us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2205218304}
I20260709 13:00:13.528681  2330 raft_consensus.cc:2376] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 7. Re-sending same reply.
I20260709 13:00:13.551472  1599 raft_consensus.cc:1275] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260709 13:00:13.551579  4549 leader_election.cc:290] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 6 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:00:13.554728  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 153910us
I20260709 13:00:13.558254  4686 consensus_queue.cc:1048] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.066s
I20260709 13:00:13.567060  1569 leader_election.cc:304] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:00:13.573566  2336 raft_consensus.cc:2393] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 8: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
W20260709 13:00:13.574498  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 51770.6us
W20260709 13:00:13.579643  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 96797.5us
I20260709 13:00:13.582223  4653 leader_election.cc:290] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:00:13.585556  1717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:13.579325  4601 raft_consensus.cc:2804] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:00:13.391366  4759 consensus_queue.cc:260] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:13.632620  4759 raft_consensus.cc:3060] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:00:13.543708  2329 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39738 (request call id 690) took 1870 ms. Trace:
I20260709 13:00:13.634830  2329 rpcz_store.cc:276] 0709 13:00:11.670100 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:12.946913 (+1276813us) service_pool.cc:224] Handling call
0709 13:00:12.947448 (+   535us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:00:12.980825 (+ 33377us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:00:12.982023 (+  1198us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:00:13.002914 (+ 20891us) log.cc:844] Serialized 26 byte log entry
0709 13:00:13.134135 (+131221us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:00:13.300275 (+166140us) log.cc:844] Serialized 14 byte log entry
0709 13:00:13.320106 (+ 19831us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:13.327154 (+  7048us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:00:13.518689 (+191535us) raft_consensus.cc:1685] finished
0709 13:00:13.523529 (+  4840us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:13.529825 (+  6296us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6185,"prepare.queue_time_us":27563,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":47,"raft.queue_time_us":31904,"raft.run_cpu_time_us":305,"raft.run_wall_time_us":9191,"thread_start_us":241697,"threads_started":2,"wal-append.queue_time_us":212224}
W20260709 13:00:13.586403  1717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:13.586997  4601 raft_consensus.cc:493] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:13.589552  4537 raft_consensus.cc:515] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:00:13.586531  1570 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:13.586573  4656 raft_consensus.cc:2749] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260709 13:00:13.593492  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 61943.3us
I20260709 13:00:13.644201  2335 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39726 (request call id 836) took 1210 ms. Trace:
I20260709 13:00:13.647114  1644 leader_election.cc:304] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:13.611621  4512 leader_election.cc:290] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 6 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:13.645552  1602 raft_consensus.cc:2468] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 6.
I20260709 13:00:13.643664  4601 raft_consensus.cc:3060] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Advancing to term 7
W20260709 13:00:13.642319  1717 connection.cc:256] RPC call timeout handler was delayed by -1.220065371999965s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1534s for a response
I20260709 13:00:13.660110  2298 leader_election.cc:304] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:13.643864  4549 raft_consensus.cc:2749] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
W20260709 13:00:13.666949  2300 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:13.646929  2335 rpcz_store.cc:276] 0709 13:00:12.434032 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:13.024282 (+590250us) service_pool.cc:224] Handling call
0709 13:00:13.046298 (+ 22016us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:00:13.105188 (+ 58890us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:00:13.105243 (+    55us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:00:13.127565 (+ 22322us) log.cc:844] Serialized 26 byte log entry
0709 13:00:13.327324 (+199759us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:00:13.378319 (+ 50995us) log.cc:844] Serialized 14 byte log entry
0709 13:00:13.487426 (+109107us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:13.487551 (+   125us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:00:13.638490 (+150939us) raft_consensus.cc:1685] finished
0709 13:00:13.638535 (+    45us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:13.644119 (+  5584us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":26583,"prepare.queue_time_us":53531,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":48,"raft.queue_time_us":63134,"raft.run_cpu_time_us":327,"raft.run_wall_time_us":16567,"spinlock_wait_cycles":1029888,"thread_start_us":119000,"threads_started":1,"wal-append.queue_time_us":337889}
W20260709 13:00:13.667718  1571 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.107.195:40693 timed out after 2.132s (SENT) [suppressed 27 similar messages]
W20260709 13:00:13.693164  4807 log.cc:927] Time spent T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.376s	user 0.002s	sys 0.000s
I20260709 13:00:13.694475  1602 raft_consensus.cc:1240] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 LEADER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 8. Current term is 9. Ops: []
I20260709 13:00:13.694765  4757 raft_consensus.cc:493] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:00:13.694826  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:13.698016  1605 raft_consensus.cc:1240] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 LEADER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 5. Current term is 6. Ops: []
I20260709 13:00:13.699847  2328 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39738 (request call id 691) took 2004 ms. Trace:
I20260709 13:00:13.700284  2328 rpcz_store.cc:276] 0709 13:00:11.695430 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:12.947880 (+1252450us) service_pool.cc:224] Handling call
0709 13:00:12.954328 (+  6448us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:13.272840 (+318512us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:13.699554 (+426714us) spinlock_profiling.cc:243] Waited 169 ms on lock 0x7b5400567f54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:00:13.699765 (+   211us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":389246208}
I20260709 13:00:13.706169  1753 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55840 (request call id 2189) took 2594 ms. Trace:
W20260709 13:00:13.708019  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 189271us
I20260709 13:00:13.713070  1601 raft_consensus.cc:2468] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 7.
I20260709 13:00:13.663648  4580 raft_consensus.cc:2749] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 13:00:13.706584  1753 rpcz_store.cc:276] 0709 13:00:11.111926 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:11.404686 (+292760us) service_pool.cc:224] Handling call
0709 13:00:11.405056 (+   370us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:00:11.405691 (+   635us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 13:00:12.406508 (+1000817us) spinlock_profiling.cc:243] Waited 18.4 ms on lock 0x7b1800447280. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f4263057e07 00007f426beb2ab9 00007f426307469b 00007f426307293f 00007f4263072455 00007f4269b5a3c7 00007f426616c9a7 00007f426617c0bf 00007f426617c038
0709 13:00:12.522311 (+115803us) log.cc:844] Serialized 14 byte log entry
0709 13:00:12.707147 (+184836us) log.cc:844] Serialized 14 byte log entry
0709 13:00:12.727013 (+ 19866us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:00:12.778623 (+ 51610us) log.cc:844] Serialized 26 byte log entry
0709 13:00:12.778808 (+   185us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:00:12.779072 (+   264us) log.cc:844] Serialized 14 byte log entry
0709 13:00:12.779320 (+   248us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:12.779637 (+   317us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:00:13.000883 (+221246us) spinlock_profiling.cc:243] Waited 8.11 ms on lock 0x7b4400022b4c. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f4262f9bc85 00007f4262f9c6a2 00007f4265d0e1db 00007f4265d09aaf 00007f4265cfd86e 00007f4265cfcbb6 00007f4265cfc7d8 00007f4265d06221 00007f4265d061d9 00007f4265d06169
0709 13:00:13.692861 (+691978us) spinlock_profiling.cc:243] Waited 3.82 ms on lock 0x7b240024b2e8. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f4262efb9c4 00007f4262ef9acb 00007f42630b4dc9 00007f42630b4a55 00007f42630b409d 00007f4265ce610e 00007f4265cfd88c 00007f4265cfcbb6 00007f4265cfc7d8 00007f4265d06221
0709 13:00:13.705815 (+ 12954us) raft_consensus.cc:1685] finished
0709 13:00:13.705843 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:13.705990 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":1073639,"prepare.run_cpu_time_us":196,"prepare.run_wall_time_us":200,"raft.queue_time_us":25647,"raft.run_cpu_time_us":805,"raft.run_wall_time_us":17328,"spinlock_wait_cycles":69779968,"thread_start_us":187832,"threads_started":2,"wal-append.queue_time_us":218020}
I20260709 13:00:13.719205  1599 raft_consensus.cc:1240] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 5. Current term is 6. Ops: []
I20260709 13:00:13.658987  4707 consensus_queue.cc:1059] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.048s
I20260709 13:00:13.722229  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:13.723160  1605 raft_consensus.cc:2393] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 5: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
W20260709 13:00:13.724608  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 64615us
I20260709 13:00:13.725536  2298 leader_election.cc:304] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:13.686636  4656 raft_consensus.cc:2749] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:00:13.687023  4593 raft_consensus.cc:493] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:00:13.732923  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:13.735435  1569 leader_election.cc:400] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 8 election: Vote denied by peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261) with higher term. Message: Invalid argument: T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 8. Current term is 9.
I20260709 13:00:13.735965  1569 leader_election.cc:403] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 8 election: Cancelling election due to peer responding with higher term
I20260709 13:00:13.737779  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:13.741647  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:00:13.750418  2331 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49082 (request call id 680) took 2117 ms. Trace:
I20260709 13:00:13.751307  2331 rpcz_store.cc:276] 0709 13:00:11.638679 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:12.347249 (+708570us) service_pool.cc:224] Handling call
0709 13:00:12.347784 (+   535us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:00:12.348318 (+   534us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:00:12.348360 (+    42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:00:13.386939 (+1038579us) spinlock_profiling.cc:243] Waited 1.29 ms on lock 0x7b2c00021c88. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4264a5ba11 00007f4264a6b372 00007f426619da01 00007f426616243a 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2
0709 13:00:13.387567 (+   628us) log.cc:844] Serialized 26 byte log entry
0709 13:00:13.659676 (+272109us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:00:13.665744 (+  6068us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:13.681533 (+ 15789us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:00:13.749752 (+ 68219us) raft_consensus.cc:1685] finished
0709 13:00:13.749786 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:13.750171 (+   385us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":110,"raft.run_cpu_time_us":142,"raft.run_wall_time_us":3664,"spinlock_wait_cycles":2969856,"thread_start_us":271636,"threads_started":1,"wal-append.queue_time_us":360016}
W20260709 13:00:13.754809  4826 log.cc:927] Time spent T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.404s	user 0.003s	sys 0.000s
I20260709 13:00:13.765390  4648 raft_consensus.cc:2749] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Leader election lost for term 8. Reason: Vote denied by peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261) with higher term. Message: Invalid argument: T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 8. Current term is 9.
I20260709 13:00:13.770707  1679 raft_consensus.cc:3060] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:00:13.771811  1602 raft_consensus.cc:1275] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 5 index: 8. (index mismatch)
I20260709 13:00:13.695427  4757 raft_consensus.cc:515] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:13.777984  4676 raft_consensus.cc:2749] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:00:13.751876  4628 raft_consensus.cc:3055] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 LEADER]: Stepping down as leader of term 3
W20260709 13:00:13.781445  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 56019.4us
I20260709 13:00:13.783838  1569 leader_election.cc:304] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:00:13.781142  4628 raft_consensus.cc:740] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:00:13.796245  2298 leader_election.cc:304] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:00:13.790632  4593 raft_consensus.cc:515] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:13.799397  4593 leader_election.cc:290] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:13.799012  4628 consensus_queue.cc:260] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:13.795296  4703 raft_consensus.cc:2804] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader election won for term 6
I20260709 13:00:13.800741  4628 raft_consensus.cc:3060] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:00:13.802242  4757 leader_election.cc:290] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:13.802896  4757 consensus_queue.cc:1048] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20260709 13:00:13.811172  2333 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49082 (request call id 675) took 2182 ms (client timeout 1717 ms). Trace:
W20260709 13:00:13.811658  2333 rpcz_store.cc:269] 0709 13:00:11.628307 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:11.707047 (+ 78740us) service_pool.cc:224] Handling call
0709 13:00:13.810922 (+2103875us) spinlock_profiling.cc:243] Waited 1.61 s on lock 0x7b54005f3a54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:00:13.811077 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3708348672}
W20260709 13:00:13.819588  2300 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:00:13.828329  1645 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:13.831207  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:13.831218  1679 raft_consensus.cc:1275] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 8. (term mismatch)
I20260709 13:00:13.832767  1679 pending_rounds.cc:85] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d: Aborting all ops after (but not including) 7
I20260709 13:00:13.833027  1679 pending_rounds.cc:107] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d: Aborting uncommitted NO_OP operation due to leader change: 7.8
I20260709 13:00:13.833825  1679 raft_consensus.cc:2889] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:00:13.837553  2298 leader_election.cc:304] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:00:13.837222  4703 raft_consensus.cc:697] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:00:13.841077  4703 consensus_queue.cc:237] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:13.843122  4823 raft_consensus.cc:493] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:13.844724  4512 raft_consensus.cc:2804] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:00:13.845289  4512 raft_consensus.cc:697] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 7 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:00:13.846282  4512 consensus_queue.cc:237] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:13.847078  4779 leader_election.cc:290] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:13.843938  4823 raft_consensus.cc:515] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:13.819165  4070 raft_consensus.cc:2804] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:00:13.862717  4070 raft_consensus.cc:493] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:13.863140  4070 raft_consensus.cc:3060] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:00:13.855358  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:13.855981  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:13.860101  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:00:13.862094  1719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:00:13.875622  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 883925us
I20260709 13:00:13.879257  1599 raft_consensus.cc:3060] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:00:13.865190  4598 raft_consensus.cc:2749] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:00:13.863145  4549 raft_consensus.cc:3055] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 5 LEADER]: Stepping down as leader of term 5
W20260709 13:00:13.879245  4721 consensus_meta.cc:287] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.577s	user 0.009s	sys 0.009s
I20260709 13:00:13.896575  1718 leader_election.cc:304] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
W20260709 13:00:13.887087  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 75355.7us
I20260709 13:00:13.901804  4534 raft_consensus.cc:493] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:13.902025  4478 raft_consensus.cc:493] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:13.888592  4549 raft_consensus.cc:740] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
W20260709 13:00:13.904280  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 113167us
I20260709 13:00:13.888082  4802 raft_consensus.cc:493] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:00:13.906702  4746 consensus_meta.cc:287] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.568s	user 0.010s	sys 0.006s
I20260709 13:00:13.917212  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:00:13.920404  2301 leader_election.cc:336] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Timed out: RequestConsensusVote RPC to 127.0.107.195:40693 timed out after 1.634s (SENT)
I20260709 13:00:13.921183  1679 raft_consensus.cc:2393] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 7: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
W20260709 13:00:13.921953  1569 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:00:13.947373  1569 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:14.029808  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:13.902678  4478 raft_consensus.cc:515] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:14.033326  4601 raft_consensus.cc:515] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:14.034648  4478 leader_election.cc:290] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:14.035786  4601 leader_election.cc:290] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 7 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:14.042623  4683 raft_consensus.cc:493] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:14.043532  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:14.044185  4736 raft_consensus.cc:493] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:14.044773  4731 raft_consensus.cc:493] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:13.897598  4721 raft_consensus.cc:515] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:14.048249  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:14.119158  1747 raft_consensus.cc:2468] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 5.
I20260709 13:00:14.044836  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:13.910063  4598 raft_consensus.cc:493] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:13.916532  4790 raft_consensus.cc:493] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:00:14.125447  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 90132.2us
I20260709 13:00:14.051841  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:14.147573  1677 raft_consensus.cc:2393] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 5: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:00:14.147336  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:14.153373  1674 raft_consensus.cc:2393] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 8: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:00:14.159368  1603 raft_consensus.cc:1240] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 5. Current term is 6. Ops: []
I20260709 13:00:14.131297  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:14.162196  1748 raft_consensus.cc:2468] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 5.
I20260709 13:00:14.124462  4746 raft_consensus.cc:493] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:14.158231  1749 raft_consensus.cc:2376] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 7. Re-sending same reply.
I20260709 13:00:14.029255  4262 raft_consensus.cc:493] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:00:14.048768  4534 raft_consensus.cc:515] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:14.181521  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 266132us
I20260709 13:00:14.180059  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:14.176406  1753 raft_consensus.cc:2468] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 5.
I20260709 13:00:14.189656  1569 leader_election.cc:304] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
W20260709 13:00:14.194063  1643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:14.196141  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:14.201694  1752 raft_consensus.cc:3060] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:00:14.204030  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:14.205478  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:14.217947  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:14.133723  4790 raft_consensus.cc:515] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:14.156994  4802 raft_consensus.cc:515] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:14.207006  1717 leader_election.cc:304] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:14.214581  4262 raft_consensus.cc:515] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:14.229091  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:14.229874  1750 raft_consensus.cc:3060] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:00:14.230739  4262 leader_election.cc:290] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:14.231379  4574 raft_consensus.cc:493] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:14.231925  4574 raft_consensus.cc:515] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:14.232520  4679 raft_consensus.cc:2804] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 13:00:14.233125  4679 raft_consensus.cc:697] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 6 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:00:14.234583  1745 raft_consensus.cc:3060] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:00:14.234354  4679 consensus_queue.cc:237] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 6, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:14.240939  4534 leader_election.cc:290] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:14.241737  4534 raft_consensus.cc:493] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:14.200855  4549 consensus_queue.cc:260] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:14.121990  4736 raft_consensus.cc:515] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:14.244546  1746 raft_consensus.cc:1275] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260709 13:00:14.245434  4736 leader_election.cc:290] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:14.180645  4598 raft_consensus.cc:515] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:14.245788  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:14.247161  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:14.247682  4598 leader_election.cc:290] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:14.225533  4601 raft_consensus.cc:493] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:14.249162  4601 raft_consensus.cc:515] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:14.251222  4601 leader_election.cc:290] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:14.252122  4601 raft_consensus.cc:2804] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Leader election won for term 9
I20260709 13:00:14.252621  4601 raft_consensus.cc:697] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:00:14.253618  4601 consensus_queue.cc:237] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:14.218969  1753 raft_consensus.cc:2468] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 5.
I20260709 13:00:12.302202   431 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
W20260709 13:00:14.254786  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 45063.7us
I20260709 13:00:14.256078  4790 leader_election.cc:290] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:00:14.257973  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 362580us
I20260709 13:00:14.260641  1749 raft_consensus.cc:2468] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 7.
I20260709 13:00:14.260728  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:14.234223  4574 leader_election.cc:290] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:14.261111  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:14.157491  4731 raft_consensus.cc:515] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:14.276823  1676 raft_consensus.cc:3060] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:00:14.270388  1747 raft_consensus.cc:2468] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 6.
I20260709 13:00:14.214109  1643 leader_election.cc:304] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:14.277650  1601 raft_consensus.cc:1275] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260709 13:00:14.274600  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:14.280886  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:14.281556  1600 raft_consensus.cc:3060] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Advancing to term 6
I20260709 13:00:14.177580  4683 raft_consensus.cc:515] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:14.189216  4746 raft_consensus.cc:515] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:00:14.291188  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 77216.4us
I20260709 13:00:14.292662  1643 leader_election.cc:304] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:00:14.296379  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:14.286051  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:14.302745  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:14.303458  1748 raft_consensus.cc:3060] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Advancing to term 8
I20260709 13:00:14.307808  2337 raft_consensus.cc:1275] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260709 13:00:14.308164  1754 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43156 (request call id 532) took 3210 ms. Trace:
I20260709 13:00:14.166738  4535 raft_consensus.cc:3055] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:00:14.326645  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:14.327318  1746 raft_consensus.cc:2468] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 5.
I20260709 13:00:14.327781  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 10 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:14.316851  1643 leader_election.cc:304] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:00:14.318349  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:14.262817  4823 leader_election.cc:290] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:14.334450  1746 raft_consensus.cc:1275] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 13:00:14.336457  1718 leader_election.cc:400] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 7 pre-election: Vote denied by peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625) with higher term. Message: Invalid argument: T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 7. Current term is 8.
I20260709 13:00:14.336793  1718 leader_election.cc:403] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 7 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:00:14.336447  1598 raft_consensus.cc:1275] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260709 13:00:14.339881  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:14.343029  1680 raft_consensus.cc:1275] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260709 13:00:14.342985  1750 raft_consensus.cc:2468] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 6.
I20260709 13:00:14.371336  1677 raft_consensus.cc:1240] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 6. Current term is 7. Ops: []
I20260709 13:00:14.323444  4629 raft_consensus.cc:2804] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260709 13:00:14.373389  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 56647.7us
I20260709 13:00:14.242383  4534 raft_consensus.cc:515] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:14.377192  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 188746us
I20260709 13:00:14.324942  4598 raft_consensus.cc:2749] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 13:00:14.380801  1643 leader_election.cc:304] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:00:14.308550  1754 rpcz_store.cc:276] 0709 13:00:11.097642 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:11.097774 (+   132us) service_pool.cc:224] Handling call
0709 13:00:11.098088 (+   314us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:00:12.818068 (+1719980us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:00:12.819444 (+  1376us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:00:12.827914 (+  8470us) log.cc:844] Serialized 26 byte log entry
0709 13:00:12.909962 (+ 82048us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:00:12.939905 (+ 29943us) log.cc:844] Serialized 14 byte log entry
0709 13:00:13.861074 (+921169us) spinlock_profiling.cc:243] Waited 5.93 ms on lock 0x7b1800141520. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f4262fefc0a 00007f426c72f219 00007f4263077a40 00007f42630724e4 00007f4263072455 00007f426615a060 00007f426616c716 00007f426617d287 00007f426617d208
0709 13:00:13.861287 (+   213us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:14.307725 (+446438us) spinlock_profiling.cc:243] Waited 1.7 s on lock 0x7b54003700d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:00:14.307780 (+    55us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:00:14.307826 (+    46us) raft_consensus.cc:1685] finished
0709 13:00:14.307839 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:14.307961 (+   122us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":17876,"prepare.queue_time_us":30140,"prepare.run_cpu_time_us":70,"prepare.run_wall_time_us":71,"raft.queue_time_us":945120,"raft.run_cpu_time_us":325,"raft.run_wall_time_us":13803,"spinlock_wait_cycles":3912479872,"thread_start_us":81501,"threads_started":1,"wal-append.queue_time_us":153856}
I20260709 13:00:14.388362  1746 raft_consensus.cc:2468] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 7.
I20260709 13:00:14.409719  1745 raft_consensus.cc:2468] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 8.
I20260709 13:00:14.322381  4535 raft_consensus.cc:740] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:00:14.418238  1643 leader_election.cc:304] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
W20260709 13:00:14.419261  1644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:14.402908  1569 leader_election.cc:304] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:00:14.422215  1753 raft_consensus.cc:1275] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 13:00:14.383639  4628 raft_consensus.cc:2804] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Leader election won for term 6
I20260709 13:00:14.326401  4656 consensus_queue.cc:1048] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.064s
I20260709 13:00:14.376140  4629 raft_consensus.cc:493] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:00:14.426689  2300 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:14.429368  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:14.442209  1600 raft_consensus.cc:2468] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 6.
I20260709 13:00:14.370275  4262 consensus_queue.cc:1059] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.052s
I20260709 13:00:14.466027  1673 raft_consensus.cc:1275] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
W20260709 13:00:14.471540  1571 leader_election.cc:336] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Timed out: RequestConsensusVote RPC to 127.0.107.195:40693 timed out after 1.845s (SENT)
I20260709 13:00:14.474557  1754 raft_consensus.cc:1275] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260709 13:00:14.421937  4721 leader_election.cc:290] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 6 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:14.431697  4846 raft_consensus.cc:2804] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:00:14.470979  4782 raft_consensus.cc:2804] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader election won for term 9
I20260709 13:00:14.465147  4628 raft_consensus.cc:697] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:00:14.464453  4629 raft_consensus.cc:3060] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Advancing to term 7
W20260709 13:00:14.523665  4839 log.cc:927] Time spent T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.106s	user 0.000s	sys 0.000s
W20260709 13:00:14.524680  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 54036us
W20260709 13:00:14.517145  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 98940.9us
I20260709 13:00:14.560648  1748 raft_consensus.cc:2468] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 8.
I20260709 13:00:14.634778  4690 raft_consensus.cc:2749] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Leader pre-election lost for term 7. Reason: Vote denied by peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625) with higher term. Message: Invalid argument: T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 7. Current term is 8.
I20260709 13:00:14.550648  4846 raft_consensus.cc:493] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Starting leader election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:00:14.662607  4547 raft_consensus.cc:3055] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:00:14.663244  4547 raft_consensus.cc:740] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
W20260709 13:00:14.664495  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 332210us
W20260709 13:00:14.672206  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 326439us
I20260709 13:00:14.632273  4574 raft_consensus.cc:493] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:14.616786  4593 raft_consensus.cc:493] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:14.567041  4790 consensus_queue.cc:1048] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.098s
I20260709 13:00:14.679905  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:00:14.672626  1599 consensus_meta.cc:287] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.784s	user 0.015s	sys 0.010s
I20260709 13:00:14.685065  1599 raft_consensus.cc:2468] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 6.
W20260709 13:00:14.686961  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 178873us
I20260709 13:00:14.659346  4846 raft_consensus.cc:3060] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:00:14.681478  1753 raft_consensus.cc:2468] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 8.
I20260709 13:00:14.570284  4849 consensus_queue.cc:1048] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.122s
W20260709 13:00:14.680621  4759 consensus_meta.cc:287] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 1.047s	user 0.012s	sys 0.004s
W20260709 13:00:14.693984  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 222451us
I20260709 13:00:14.673197  4802 leader_election.cc:290] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:14.680274  4746 leader_election.cc:290] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:14.707094  1569 leader_election.cc:304] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:00:14.741427  1569 leader_election.cc:304] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
W20260709 13:00:14.707962  4841 log.cc:927] Time spent T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.336s	user 0.001s	sys 0.000s
W20260709 13:00:14.742619  4825 log.cc:927] Time spent T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 1.395s	user 0.007s	sys 0.000s
I20260709 13:00:14.743822  4721 raft_consensus.cc:2804] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:00:14.744300  4721 raft_consensus.cc:493] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:00:14.744669  4721 raft_consensus.cc:3060] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Advancing to term 8
W20260709 13:00:14.708840  4843 log.cc:927] Time spent T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.202s	user 0.001s	sys 0.000s
I20260709 13:00:14.746915  2334 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39726 (request call id 835) took 2313 ms. Trace:
I20260709 13:00:14.747320  2334 rpcz_store.cc:276] 0709 13:00:12.433721 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:12.970116 (+536395us) service_pool.cc:224] Handling call
0709 13:00:12.970573 (+   457us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:00:12.992804 (+ 22231us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:00:12.992915 (+   111us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:00:13.034582 (+ 41667us) log.cc:844] Serialized 26 byte log entry
0709 13:00:13.298630 (+264048us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:00:13.319976 (+ 21346us) log.cc:844] Serialized 14 byte log entry
0709 13:00:13.323028 (+  3052us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:13.323114 (+    86us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:00:14.741789 (+1418675us) spinlock_profiling.cc:243] Waited 15.9 ms on lock 0x7b4400022b4c. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f4262f9bc85 00007f4262f9c6a2 00007f4265d0e1db 00007f4265d09aaf 00007f4265cfd86e 00007f4265cfcbb6 00007f4265cfc7d8 00007f4265d06221 00007f4265d061d9 00007f4265d06169
0709 13:00:14.746513 (+  4724us) raft_consensus.cc:1685] finished
0709 13:00:14.746666 (+   153us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:14.746822 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":66012,"prepare.queue_time_us":20961,"prepare.run_cpu_time_us":55,"prepare.run_wall_time_us":55,"raft.queue_time_us":18837,"raft.run_cpu_time_us":314,"raft.run_wall_time_us":29349,"spinlock_wait_cycles":36631296,"thread_start_us":234424,"threads_started":1,"wal-append.queue_time_us":292053}
I20260709 13:00:14.750424  4823 consensus_queue.cc:1048] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.004s
I20260709 13:00:14.753356  4549 raft_consensus.cc:3060] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:00:14.753563  4847 consensus_queue.cc:1048] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.006s
I20260709 13:00:14.753734  4845 raft_consensus.cc:2749] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:00:14.699501  4731 leader_election.cc:290] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:14.692035  4574 raft_consensus.cc:515] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:14.683410  4535 consensus_queue.cc:260] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:14.713209  1718 leader_election.cc:304] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:14.718703  4547 consensus_queue.cc:260] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:14.726797  4534 leader_election.cc:290] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:00:14.727442  4840 log.cc:927] Time spent T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.359s	user 0.000s	sys 0.001s
I20260709 13:00:14.728260  4848 raft_consensus.cc:2804] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Leader election won for term 6
W20260709 13:00:14.687077  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 55615.3us
W20260709 13:00:14.730357  1501 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35458 (request call id 305) took 18459 ms (client timeout 15000 ms). Trace:
I20260709 13:00:14.736847  1676 raft_consensus.cc:2468] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 5.
I20260709 13:00:14.708428  4628 consensus_queue.cc:237] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:00:14.706580  4749 consensus_meta.cc:287] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 1.159s	user 0.017s	sys 0.000s
I20260709 13:00:14.827432  1751 raft_consensus.cc:1275] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
W20260709 13:00:14.841005  1752 consensus_meta.cc:287] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.612s	user 0.007s	sys 0.006s
I20260709 13:00:14.841683  1752 raft_consensus.cc:2468] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 7.
I20260709 13:00:14.843502  4848 raft_consensus.cc:697] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:00:14.845003  4848 consensus_queue.cc:237] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:14.763921  1603 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 562) took 1352 ms. Trace:
I20260709 13:00:14.846599  1603 rpcz_store.cc:276] 0709 13:00:13.428020 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:13.505215 (+ 77195us) service_pool.cc:224] Handling call
0709 13:00:13.505551 (+   336us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:14.166648 (+661097us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:14.763652 (+597004us) spinlock_profiling.cc:243] Waited 653 ms on lock 0x7b540006e354. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:00:14.763836 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1502868736}
W20260709 13:00:14.771572  1571 leader_election.cc:336] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Timed out: RequestConsensusVote RPC to 127.0.107.195:40693 timed out after 1.847s (SENT)
W20260709 13:00:14.707974  4851 log.cc:927] Time spent T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.059s	user 0.000s	sys 0.000s
I20260709 13:00:14.853942  4759 consensus_queue.cc:1048] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 13:00:14.854636  4740 consensus_queue.cc:1059] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 13:00:14.730278  4849 consensus_queue.cc:1048] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.006s
I20260709 13:00:14.856158  4740 consensus_queue.cc:1048] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 13:00:14.857666  4740 raft_consensus.cc:3055] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:00:14.704306  4852 raft_consensus.cc:493] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:14.858033  4740 raft_consensus.cc:740] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:00:14.858403  4852 raft_consensus.cc:515] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:14.860347  4598 raft_consensus.cc:3055] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:00:14.860461  1717 leader_election.cc:304] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:14.860769  4852 leader_election.cc:290] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:14.860852  4598 raft_consensus.cc:740] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:00:14.862603  4683 leader_election.cc:290] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:14.809854  4547 raft_consensus.cc:3060] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Advancing to term 5
W20260709 13:00:14.867312  1646 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:00:14.811371  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 98289.9us
I20260709 13:00:14.762435  4537 leader_election.cc:290] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 5 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:14.814566  4731 raft_consensus.cc:2749] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 13:00:14.815771  4845 consensus_queue.cc:1048] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
W20260709 13:00:14.806969  1501 rpcz_store.cc:269] 0709 12:59:56.270847 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:59:56.272022 (+  1175us) service_pool.cc:224] Handling call
0709 13:00:14.730048 (+18458026us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 13:00:07.945168 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 13:00:08.055312 (+110144us) write_op.cc:432] Acquiring schema lock in shared mode
0709 13:00:08.055389 (+    77us) write_op.cc:435] Acquired schema lock
0709 13:00:08.055419 (+    30us) tablet.cc:662] Decoding operations
0709 13:00:08.166040 (+110621us) write_op.cc:620] Acquiring the partition lock for write op
0709 13:00:08.168174 (+  2134us) write_op.cc:641] Partition lock acquired for write op
0709 13:00:08.168216 (+    42us) tablet.cc:685] Acquiring locks for 18 operations
0709 13:00:08.230759 (+ 62543us) tablet.cc:701] Row locks acquired
0709 13:00:08.230825 (+    66us) write_op.cc:260] PREPARE: finished
0709 13:00:08.231045 (+   220us) write_op.cc:270] Start()
0709 13:00:08.231222 (+   177us) write_op.cc:276] Timestamp: P: 1783602008230970 usec, L: 0
0709 13:00:08.231248 (+    26us) op_driver.cc:348] REPLICATION: starting
0709 13:00:08.232180 (+   932us) log.cc:844] Serialized 7146 byte log entry
0709 13:00:08.827871 (+595691us) op_driver.cc:464] REPLICATION: finished
0709 13:00:09.171482 (+343611us) write_op.cc:301] APPLY: starting
0709 13:00:09.171592 (+   110us) tablet.cc:1367] starting BulkCheckPresence
0709 13:00:09.194817 (+ 23225us) tablet.cc:1370] finished BulkCheckPresence
0709 13:00:09.194974 (+   157us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 13:00:09.445248 (+250274us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 13:00:09.445717 (+   469us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=18
0709 13:00:09.445770 (+    53us) write_op.cc:312] APPLY: finished
0709 13:00:09.446306 (+   536us) log.cc:844] Serialized 125 byte log entry
0709 13:00:09.446577 (+   271us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:00:09.446789 (+   212us) write_op.cc:454] Released schema lock
0709 13:00:09.446874 (+    85us) write_op.cc:341] FINISH: Updating metrics
0709 13:00:09.447430 (+   556us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:00:09.447463 (+    33us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":2168811,"reported_tablets":35,"child_traces":[["op",{"apply.queue_time_us":343285,"apply.run_cpu_time_us":10125,"apply.run_wall_time_us":275912,"num_ops":18,"prepare.queue_time_us":165235,"prepare.run_cpu_time_us":18888,"prepare.run_wall_time_us":404804,"raft.queue_time_us":285524,"raft.run_cpu_time_us":10485,"raft.run_wall_time_us":153168,"replication_time_us":596385,"thread_start_us":473781,"threads_started":4,"wal-append.queue_time_us":126496,"wal-append.run_cpu_time_us":59833,"wal-append.run_wall_time_us":2096395}]]}
W20260709 13:00:14.824786  4070 consensus_meta.cc:287] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.961s	user 0.008s	sys 0.007s
I20260709 13:00:14.700837  4593 raft_consensus.cc:515] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:14.775031  2328 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39726 (request call id 848) took 1046 ms. Trace:
I20260709 13:00:14.826073  4535 raft_consensus.cc:3060] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Advancing to term 7
W20260709 13:00:14.848963  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 77446.1us
I20260709 13:00:14.869091  2332 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39738 (request call id 694) took 2125 ms. Trace:
I20260709 13:00:14.880528  1718 leader_election.cc:304] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:00:14.898347  2332 rpcz_store.cc:276] 0709 13:00:12.743307 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:13.042820 (+299513us) service_pool.cc:224] Handling call
0709 13:00:13.043223 (+   403us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:13.514825 (+471602us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:14.868768 (+1353943us) spinlock_profiling.cc:243] Waited 460 ms on lock 0x7b54005bac54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:00:14.869005 (+   237us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1058550016}
I20260709 13:00:14.884268  1603 raft_consensus.cc:1275] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 13:00:14.824898  4574 leader_election.cc:290] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:14.891342  4683 raft_consensus.cc:2749] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:00:14.801770  4782 raft_consensus.cc:697] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
W20260709 13:00:14.903714  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 215171us
I20260709 13:00:14.906571  1644 leader_election.cc:304] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:14.890892  4070 raft_consensus.cc:515] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:14.911320  4731 raft_consensus.cc:493] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:14.895740  2328 rpcz_store.cc:276] 0709 13:00:13.729069 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:13.732705 (+  3636us) service_pool.cc:224] Handling call
0709 13:00:13.747112 (+ 14407us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:00:13.784367 (+ 37255us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:00:13.785430 (+  1063us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:00:13.837962 (+ 52532us) log.cc:844] Serialized 26 byte log entry
0709 13:00:14.112214 (+274252us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:00:14.119745 (+  7531us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:14.135411 (+ 15666us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:00:14.764021 (+628610us) raft_consensus.cc:1685] finished
0709 13:00:14.772413 (+  8392us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:14.772670 (+   257us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":10786,"raft.queue_time_us":31055,"raft.run_cpu_time_us":114,"raft.run_wall_time_us":110,"thread_start_us":79052,"threads_started":1,"wal-append.queue_time_us":445958}
I20260709 13:00:14.888912  4690 consensus_queue.cc:1048] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.023s
I20260709 13:00:14.912019  4731 raft_consensus.cc:515] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:14.913174  4690 raft_consensus.cc:2749] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:00:14.862131  4598 consensus_queue.cc:260] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:14.915560  4749 consensus_queue.cc:1059] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 13:00:14.916324  4749 consensus_queue.cc:1243] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Peer 83595ad0fd5940c0a2bfc7c61acd969a log is divergent from this leader: its last log entry 6.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260709 13:00:14.887693  4852 raft_consensus.cc:493] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:14.917640  4852 raft_consensus.cc:515] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:14.919876  4598 raft_consensus.cc:3060] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:00:14.898697  4262 consensus_queue.cc:1059] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.024s
I20260709 13:00:14.923115  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:14.887841  4802 consensus_queue.cc:1059] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.036s
I20260709 13:00:14.925028  4723 raft_consensus.cc:3055] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:00:14.925390  4723 raft_consensus.cc:740] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:00:14.925963  1569 leader_election.cc:304] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:00:14.926576  4723 consensus_queue.cc:260] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:14.927205  4845 raft_consensus.cc:2804] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:00:14.927654  4845 raft_consensus.cc:493] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:14.927810  4723 raft_consensus.cc:3060] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:00:14.928051  4845 raft_consensus.cc:3060] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:00:14.886238  4790 consensus_queue.cc:1059] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.044s
I20260709 13:00:14.923269  4262 consensus_queue.cc:1243] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer 83595ad0fd5940c0a2bfc7c61acd969a log is divergent from this leader: its last log entry 9.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260709 13:00:14.932246  4574 raft_consensus.cc:3055] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:00:14.932713  4574 raft_consensus.cc:740] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:00:14.933802  4574 consensus_queue.cc:260] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:14.938127  4574 raft_consensus.cc:3060] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:00:14.944156  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:14.944824  1673 raft_consensus.cc:3060] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Advancing to term 7
W20260709 13:00:14.944835  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 43536.4us
I20260709 13:00:14.946992  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:14.948154  1754 raft_consensus.cc:2410] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for term 9 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 5.
I20260709 13:00:14.947985  1643 leader_election.cc:304] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:00:14.949122  4849 consensus_queue.cc:1048] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:00:14.950801  4849 raft_consensus.cc:2804] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Leader election won for term 6
I20260709 13:00:14.951315  4849 raft_consensus.cc:697] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 6 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:00:14.910233  4070 leader_election.cc:290] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 7 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:14.952296  4849 consensus_queue.cc:237] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:14.955317  4629 raft_consensus.cc:515] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:00:14.951306  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 45758.6us
I20260709 13:00:14.922652  4740 consensus_queue.cc:260] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:14.957934  4740 raft_consensus.cc:3060] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:00:14.955681  4858 raft_consensus.cc:493] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:00:14.960139  4858 raft_consensus.cc:515] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:00:14.962602  2300 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:14.938689  4731 leader_election.cc:290] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:00:14.968048  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 61974us
W20260709 13:00:14.977556  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 74082.5us
I20260709 13:00:14.979180  2298 leader_election.cc:304] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:00:14.984114  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:14.985411  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:14.986569  1599 raft_consensus.cc:2393] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 9: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:00:14.975466  4648 raft_consensus.cc:493] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:00:14.988166  4648 raft_consensus.cc:515] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:14.988919  4721 raft_consensus.cc:515] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:14.979792  1644 leader_election.cc:304] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:14.996842  1673 raft_consensus.cc:2468] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 7.
I20260709 13:00:14.998867  2299 leader_election.cc:304] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:00:14.962783  4858 leader_election.cc:290] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:15.010857  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:15.015161  1601 raft_consensus.cc:2468] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 5.
I20260709 13:00:15.016829  1678 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52766 (request call id 2253) took 1104 ms. Trace:
I20260709 13:00:15.036959  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 8 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:15.038672  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:15.040035  1754 raft_consensus.cc:1275] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260709 13:00:14.978678  4518 raft_consensus.cc:2749] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 13:00:15.010166  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:15.042199  1603 raft_consensus.cc:2468] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 7.
I20260709 13:00:15.047298  1569 leader_election.cc:400] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 5 election: Vote denied by peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261) with higher term. Message: Invalid argument: T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 5. Current term is 6.
I20260709 13:00:15.047825  1569 leader_election.cc:403] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
I20260709 13:00:14.984972  4847 raft_consensus.cc:493] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:15.057227  1677 raft_consensus.cc:1240] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20260709 13:00:15.021667  1678 rpcz_store.cc:276] 0709 13:00:13.912282 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:13.912429 (+   147us) service_pool.cc:224] Handling call
0709 13:00:13.915072 (+  2643us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:13.919882 (+  4810us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 13:00:14.366656 (+446774us) log.cc:844] Serialized 14 byte log entry
0709 13:00:15.016344 (+649688us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:00:15.016411 (+    67us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:00:15.016483 (+    72us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:15.016583 (+   100us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:15.016740 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":24268,"prepare.queue_time_us":602117,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":57,"raft.queue_time_us":61233,"raft.run_cpu_time_us":135,"raft.run_wall_time_us":135,"thread_start_us":890707,"threads_started":3,"wal-append.queue_time_us":352548}
I20260709 13:00:14.990672  4759 raft_consensus.cc:2804] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 13:00:15.056141  2332 raft_consensus.cc:1275] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260709 13:00:15.058859  1569 leader_election.cc:304] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
W20260709 13:00:15.062958  4783 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 8 current term: 8 is higher.
I20260709 13:00:15.062815  4672 raft_consensus.cc:2804] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:00:15.059965  4537 raft_consensus.cc:2749] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261) with higher term. Message: Invalid argument: T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 5. Current term is 6.
I20260709 13:00:15.067394  4537 raft_consensus.cc:2804] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:00:15.068040  4537 raft_consensus.cc:697] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:00:15.069414  4537 consensus_queue.cc:237] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } }
I20260709 13:00:15.070087  1747 raft_consensus.cc:1275] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260709 13:00:15.071105  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:15.072046  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:15.073226  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:15.082805  1746 raft_consensus.cc:3060] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:00:15.073606  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:15.036242  4708 raft_consensus.cc:2749] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:00:15.075985  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:15.012120  4648 leader_election.cc:290] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:14.981155  4707 consensus_queue.cc:1048] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.072s
I20260709 13:00:15.014750  4580 raft_consensus.cc:2749] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:00:15.078533  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:15.112288  1745 raft_consensus.cc:2468] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 5.
I20260709 13:00:15.083657  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:15.113718  1601 raft_consensus.cc:2393] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 6: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:00:15.091225  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:15.083726  1749 raft_consensus.cc:2468] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 7.
I20260709 13:00:15.082192  1753 raft_consensus.cc:2468] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 7.
I20260709 13:00:15.094702  2337 raft_consensus.cc:2468] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 6.
I20260709 13:00:15.114952  1752 raft_consensus.cc:2468] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 5.
I20260709 13:00:15.116951  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:15.118127  1596 raft_consensus.cc:2468] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 6.
I20260709 13:00:15.037774  4512 consensus_queue.cc:1059] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.069s
I20260709 13:00:15.125164  1644 leader_election.cc:304] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:00:15.093920  4672 raft_consensus.cc:697] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 7 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:00:15.126727  1601 raft_consensus.cc:1240] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20260709 13:00:15.076699  4857 consensus_queue.cc:1048] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.047s
I20260709 13:00:15.135555  1751 raft_consensus.cc:1240] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 6 LEADER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 5. Current term is 6. Ops: []
I20260709 13:00:15.136822  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:15.061439  4847 raft_consensus.cc:515] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:15.131775  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:15.124313  4629 leader_election.cc:290] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 7 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:15.121781  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:15.099998  4846 raft_consensus.cc:515] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:15.142021  1596 raft_consensus.cc:1275] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20260709 13:00:15.142609  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:15.143270  1754 raft_consensus.cc:2410] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 13:00:15.143234  1752 raft_consensus.cc:3060] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:00:15.127669  4721 leader_election.cc:290] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 8 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:00:15.145979  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 249718us
I20260709 13:00:15.151710  1601 raft_consensus.cc:1240] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 5. Current term is 6. Ops: [5.8-5.8]
W20260709 13:00:15.159121  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 278656us
I20260709 13:00:15.153669  1605 raft_consensus.cc:2468] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 9.
I20260709 13:00:15.169322  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:15.170042  1596 raft_consensus.cc:2468] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 5.
I20260709 13:00:15.169323  1718 leader_election.cc:304] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 13:00:15.179332  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 120594us
I20260709 13:00:15.183754  1569 leader_election.cc:400] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 5 pre-election: Vote denied by peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261) with higher term. Message: Invalid argument: T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 5. Current term is 6.
I20260709 13:00:15.184268  1569 leader_election.cc:403] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 5 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:00:15.192863  1605 raft_consensus.cc:1275] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260709 13:00:15.173494  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:15.197096  4731 raft_consensus.cc:2804] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:00:15.197983  4731 raft_consensus.cc:697] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [term 5 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:00:15.197870  4749 consensus_queue.cc:1059] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 13:00:15.199515  4262 raft_consensus.cc:3055] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:00:15.199468  4731 consensus_queue.cc:237] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:15.200078  4262 raft_consensus.cc:740] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:00:15.200091  1718 leader_election.cc:304] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:15.201421  4749 raft_consensus.cc:2804] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:00:15.201210  4262 consensus_queue.cc:260] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:15.201913  4749 raft_consensus.cc:493] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:15.202301  4749 raft_consensus.cc:3060] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:00:15.155923  4823 raft_consensus.cc:493] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:00:15.202247  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:15.218775  1569 leader_election.cc:304] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:00:15.231271  1752 raft_consensus.cc:2468] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 7.
I20260709 13:00:15.166970  4547 raft_consensus.cc:493] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:00:15.240738  1751 raft_consensus.cc:2468] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 4.
I20260709 13:00:15.241946  1754 raft_consensus.cc:2393] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 8: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:00:15.242463  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:15.127808  4782 consensus_queue.cc:237] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:00:15.246884  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 46918us
I20260709 13:00:15.250326  1753 raft_consensus.cc:2393] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 5: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
W20260709 13:00:15.228729  4679 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.564s	user 0.005s	sys 0.014s
I20260709 13:00:15.251587  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:00:15.257805  1717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:15.239001  4601 raft_consensus.cc:2749] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: Vote denied by peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261) with higher term. Message: Invalid argument: T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 5. Current term is 6.
I20260709 13:00:15.235023  4262 raft_consensus.cc:3060] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Advancing to term 5
W20260709 13:00:15.254935  4679 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.652s	user 0.005s	sys 0.021s
I20260709 13:00:15.252817  4721 raft_consensus.cc:493] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:15.291695  1753 raft_consensus.cc:1240] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 5. Current term is 6. Ops: []
I20260709 13:00:15.251729  4759 raft_consensus.cc:697] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 6 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:00:15.294592  1754 raft_consensus.cc:1275] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260709 13:00:15.254355  4872 raft_consensus.cc:2749] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260709 13:00:15.277801  4861 log.cc:927] Time spent T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.055s	user 0.000s	sys 0.001s
I20260709 13:00:15.303539  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:15.249339  4823 raft_consensus.cc:515] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:15.247534  4672 consensus_queue.cc:237] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:15.309996  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 8 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:15.258841  4547 raft_consensus.cc:515] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:15.304755  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 293624us
I20260709 13:00:15.343998  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:15.312693  1596 raft_consensus.cc:2393] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 7: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:00:15.346292  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
W20260709 13:00:15.329685  1570 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:15.353912  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:15.352754  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:15.385154  2334 raft_consensus.cc:2410] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for term 8 because replica has last-logged OpId of term: 6 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260709 13:00:15.346292  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:15.352317  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:15.387632  1751 raft_consensus.cc:3060] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:00:15.353480  1752 raft_consensus.cc:3060] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:00:15.388021  2337 raft_consensus.cc:2393] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 7: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:00:15.355424  1745 raft_consensus.cc:1240] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20260709 13:00:15.356251  1748 raft_consensus.cc:3060] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Advancing to term 6
W20260709 13:00:15.479326  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 94875.7us
W20260709 13:00:15.799715  1645 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:00:15.461917  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 336811us
W20260709 13:00:15.399088  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 150777us
I20260709 13:00:15.356781  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:15.803320  1569 leader_election.cc:304] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:00:15.415437  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:15.405063  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:15.567425  1746 raft_consensus.cc:2468] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 5.
I20260709 13:00:15.805760  2335 raft_consensus.cc:2393] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 7: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
W20260709 13:00:15.388731  2301 leader_election.cc:336] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Timed out: RequestConsensusVote RPC to 127.0.107.195:40693 timed out after 1.561s (SENT)
I20260709 13:00:15.806879  2301 leader_election.cc:304] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:00:15.302482  4512 raft_consensus.cc:493] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:00:15.356896  4863 log.cc:927] Time spent T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.112s	user 0.000s	sys 0.000s
I20260709 13:00:15.459681  1672 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57092 (request call id 673) took 1138 ms. Trace:
I20260709 13:00:15.812134  1747 raft_consensus.cc:2393] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 6: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
W20260709 13:00:15.351392  4865 log.cc:927] Time spent T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.059s	user 0.000s	sys 0.000s
I20260709 13:00:15.357285  4683 raft_consensus.cc:493] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:00:15.443519  4864 log.cc:927] Time spent T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.100s	user 0.000s	sys 0.000s
W20260709 13:00:15.764247  4740 consensus_meta.cc:287] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.806s	user 0.014s	sys 0.000s
W20260709 13:00:15.638250  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 89549.8us
I20260709 13:00:15.564363  1599 raft_consensus.cc:1275] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260709 13:00:15.422489  4877 raft_consensus.cc:3055] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:00:15.363915  4679 consensus_queue.cc:1048] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.475s
I20260709 13:00:15.342990  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 8 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:15.523756  4790 raft_consensus.cc:493] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:15.415977  1751 raft_consensus.cc:2468] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 9.
W20260709 13:00:15.849306  1643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:15.551469  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:15.854779  1749 raft_consensus.cc:3060] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Advancing to term 6
W20260709 13:00:15.828791  1644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:15.359999  4852 leader_election.cc:290] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:00:15.817442  4845 consensus_meta.cc:287] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.889s	user 0.007s	sys 0.010s
I20260709 13:00:15.640508  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:00:15.844704  2300 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:15.788962  4876 raft_consensus.cc:493] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:00:15.815227  1672 rpcz_store.cc:276] 0709 13:00:14.321119 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:14.321263 (+   144us) service_pool.cc:224] Handling call
0709 13:00:14.345603 (+ 24340us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:14.426542 (+ 80939us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 13:00:14.685229 (+258687us) log.cc:844] Serialized 14 byte log entry
0709 13:00:15.106246 (+421017us) log.cc:844] Serialized 14 byte log entry
0709 13:00:15.458932 (+352686us) spinlock_profiling.cc:243] Waited 2.19 ms on lock 0x7b18000fbc80. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f4262fefc0a 00007f426c72f219 00007f4263077a40 00007f42630724e4 00007f4263072455 00007f426615a060 00007f426616c716 00007f426617d287 00007f426617d208
0709 13:00:15.459175 (+   243us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:00:15.459255 (+    80us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:00:15.459323 (+    68us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:15.459436 (+   113us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:15.459597 (+   161us) inbound_call.cc:177] Queueing success response
0709 13:00:15.694404 (+234807us) spinlock_profiling.cc:243] Waited 1.13 ms on lock 0x7b18000fbc80. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f4263075522 00007f42630777c1 00007f4263077779 00007f4263077709 00007f42630776d1 00007f42630769cd 000000000038f354 000000000038f199 00007f426305883a
Metrics: {"mutex_wait_us":15642,"prepare.queue_time_us":61911,"prepare.run_cpu_time_us":70,"prepare.run_wall_time_us":72,"raft.queue_time_us":379166,"raft.run_cpu_time_us":287,"raft.run_wall_time_us":1925,"spinlock_wait_cycles":7636480,"thread_start_us":555021,"threads_started":2,"wal-append.queue_time_us":458208}
I20260709 13:00:15.456405  4593 leader_election.cc:290] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:15.852838  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:00:15.561600  4598 consensus_meta.cc:287] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.641s	user 0.009s	sys 0.005s
I20260709 13:00:15.854063  1752 raft_consensus.cc:1275] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260709 13:00:15.563390  4802 raft_consensus.cc:3055] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:00:15.796589  4547 leader_election.cc:290] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:00:15.828011  1646 leader_election.cc:336] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Timed out: RequestConsensusVote RPC to 127.0.107.195:40693 timed out after 2.069s (SENT)
W20260709 13:00:15.792135  4723 consensus_meta.cc:287] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.864s	user 0.016s	sys 0.000s
I20260709 13:00:15.442376  4869 consensus_queue.cc:1059] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.442s
W20260709 13:00:15.637583  4535 consensus_meta.cc:287] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.735s	user 0.016s	sys 0.000s
W20260709 13:00:15.827924  4656 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.531s	user 0.000s	sys 0.011s
I20260709 13:00:15.814879  4872 raft_consensus.cc:2804] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:00:15.462236  4580 raft_consensus.cc:2804] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:00:15.868573  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:15.868804  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:15.887609  1598 raft_consensus.cc:2410] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for term 9 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20260709 13:00:15.818934  4512 raft_consensus.cc:515] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:15.829829  4847 leader_election.cc:290] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:15.567857  4873 consensus_queue.cc:1048] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.416s
I20260709 13:00:15.459945  4823 leader_election.cc:290] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:15.830411  4877 raft_consensus.cc:740] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:00:15.307634  4721 raft_consensus.cc:515] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:15.891990  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:16.063534  1671 raft_consensus.cc:2468] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 8.
I20260709 13:00:16.063675  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:00:15.902498  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 513715us
I20260709 13:00:16.064292  1747 raft_consensus.cc:2376] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 9. Re-sending same reply.
W20260709 13:00:16.065092  2301 leader_election.cc:336] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Timed out: RequestConsensusVote RPC to 127.0.107.195:40693 timed out after 1.573s (SENT)
I20260709 13:00:16.010474  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:16.011962  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:15.963220  1748 raft_consensus.cc:1275] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260709 13:00:16.048446  1676 raft_consensus.cc:2468] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 7.
W20260709 13:00:16.046007  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 182639us
I20260709 13:00:15.929147  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:16.070611  1678 raft_consensus.cc:2393] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 5: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:00:16.047170  1751 raft_consensus.cc:1240] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 7. Current term is 8. Ops: [7.8-7.8]
I20260709 13:00:16.054375  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:16.048858  1602 raft_consensus.cc:2393] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 6: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:00:15.892437  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:15.845742  4848 raft_consensus.cc:493] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:16.073879  1675 raft_consensus.cc:2468] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 5.
I20260709 13:00:15.947244  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 8 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:16.061952  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:16.075146  1752 raft_consensus.cc:2410] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for term 9 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 5.
I20260709 13:00:16.075230  1674 raft_consensus.cc:2410] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for term 8 because replica has last-logged OpId of term: 7 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 8.
I20260709 13:00:16.047863  1599 raft_consensus.cc:2393] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 7: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:00:16.075620  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:00:15.932291  4863 log.cc:927] Time spent T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.073s	user 0.000s	sys 0.000s
W20260709 13:00:16.049129  4656 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.758s	user 0.000s	sys 0.013s
I20260709 13:00:15.892146  4070 raft_consensus.cc:2749] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:00:15.524874  4759 consensus_queue.cc:237] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:16.085572  1746 raft_consensus.cc:1275] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260709 13:00:15.962858  4875 raft_consensus.cc:493] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:00:16.076431  4846 leader_election.cc:290] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 10 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:16.048755  4802 raft_consensus.cc:740] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:00:16.000670  4537 raft_consensus.cc:493] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:16.143682  1745 raft_consensus.cc:1275] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260709 13:00:16.147166  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:00:16.119927  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 105675us
I20260709 13:00:16.062824  4547 raft_consensus.cc:493] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:00:16.077306  4731 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.731s	user 0.008s	sys 0.047s
I20260709 13:00:16.003808  4882 raft_consensus.cc:493] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
W20260709 13:00:16.129280  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 301246us
I20260709 13:00:16.077689  4872 raft_consensus.cc:697] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:00:16.134994  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:16.153748  1752 raft_consensus.cc:2468] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 7.
I20260709 13:00:15.861793  4790 raft_consensus.cc:515] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:16.136412  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:15.852378  4683 raft_consensus.cc:515] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:16.076977  4847 raft_consensus.cc:493] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
W20260709 13:00:16.121418  4574 consensus_meta.cc:287] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 1.180s	user 0.016s	sys 0.000s
I20260709 13:00:16.054566  4580 raft_consensus.cc:493] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:15.919338  4845 raft_consensus.cc:515] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:16.150414  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:16.073930  4848 raft_consensus.cc:515] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:16.161352  1752 raft_consensus.cc:1275] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260709 13:00:16.161340  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:16.161667  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:16.162367  1753 raft_consensus.cc:1275] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260709 13:00:16.164602  4547 raft_consensus.cc:515] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } }
W20260709 13:00:16.150710  4731 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.807s	user 0.008s	sys 0.047s
W20260709 13:00:16.167004  4262 consensus_meta.cc:287] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.893s	user 0.009s	sys 0.004s
I20260709 13:00:16.168279  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:16.169008  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:16.257037  1745 raft_consensus.cc:2376] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 5. Re-sending same reply.
W20260709 13:00:16.170691  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 105622us
I20260709 13:00:16.171232  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:00:16.172618  1646 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:16.169431  1677 raft_consensus.cc:1240] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 LEADER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 5. Current term is 6. Ops: []
I20260709 13:00:16.174046  4782 raft_consensus.cc:493] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:16.174765  4708 raft_consensus.cc:493] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:16.175088  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:16.179950  4740 raft_consensus.cc:493] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:00:16.122759  4887 raft_consensus.cc:493] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:16.189116  1751 raft_consensus.cc:2468] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 7.
W20260709 13:00:16.224442  1569 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:00:16.270975  1569 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:00:16.271463  1569 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:16.141481  4628 raft_consensus.cc:493] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:16.165493  4882 raft_consensus.cc:515] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:16.151495  4537 raft_consensus.cc:515] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:16.239936  4580 raft_consensus.cc:3060] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:00:16.233743  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:16.234086  4721 leader_election.cc:290] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:16.297091  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:16.254112  1747 raft_consensus.cc:2410] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for term 9 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
W20260709 13:00:16.253978  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 184092us
I20260709 13:00:16.173017  4629 raft_consensus.cc:493] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:16.260932  1748 raft_consensus.cc:2410] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for term 10 because replica has last-logged OpId of term: 8 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 5.
I20260709 13:00:16.267567  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:16.182111  4846 raft_consensus.cc:493] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:16.273835  4896 raft_consensus.cc:493] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:00:16.273921  4601 raft_consensus.cc:493] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:00:16.274693  1570 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:16.276786  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:16.232231  4690 raft_consensus.cc:3055] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:00:16.280653  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:00:16.296391  2300 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:16.297827  1745 raft_consensus.cc:2468] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 7.
I20260709 13:00:16.301257  1751 raft_consensus.cc:1240] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [term 5 LEADER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 4. Current term is 5. Ops: []
I20260709 13:00:16.003189  4876 raft_consensus.cc:515] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:16.144567  4875 raft_consensus.cc:515] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:16.313241  4598 raft_consensus.cc:493] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:16.316223  4882 leader_election.cc:290] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:16.278509  4740 raft_consensus.cc:515] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:16.278738  4708 raft_consensus.cc:515] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:16.323483  4892 log.cc:927] Time spent T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.054s	user 0.000s	sys 0.000s
I20260709 13:00:16.237619  4847 raft_consensus.cc:515] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:16.453480  1643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:16.424438  1718 leader_election.cc:304] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:00:16.430780  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:16.324692  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:16.430518  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:16.234753  4877 consensus_queue.cc:260] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:16.518546  1677 raft_consensus.cc:2393] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 7: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:00:16.519837  1673 raft_consensus.cc:3060] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Advancing to term 6
W20260709 13:00:16.518184  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 216321us
W20260709 13:00:16.431592  1646 leader_election.cc:336] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Timed out: RequestConsensusVote RPC to 127.0.107.195:40693 timed out after 1.592s (SENT)
W20260709 13:00:16.521065  1749 consensus_meta.cc:287] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.662s	user 0.011s	sys 0.002s
I20260709 13:00:16.521874  1749 raft_consensus.cc:2468] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 6.
I20260709 13:00:16.300755  4679 raft_consensus.cc:493] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:16.521490  1646 leader_election.cc:304] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:00:16.533051  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:16.534472  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:16.535301  1603 raft_consensus.cc:2393] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 8: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
W20260709 13:00:16.524840  1644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:16.536041  1749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55848 (request call id 2035) took 1106 ms. Trace:
I20260709 13:00:16.467891  1599 raft_consensus.cc:3055] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:00:16.536883  1749 rpcz_store.cc:276] 0709 13:00:15.421044 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:15.465915 (+ 44871us) service_pool.cc:224] Handling call
0709 13:00:16.522410 (+1056495us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:00:16.548250  1752 raft_consensus.cc:1240] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 4. Current term is 5. Ops: []
I20260709 13:00:16.307839  4723 raft_consensus.cc:493] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:16.426673  4849 raft_consensus.cc:493] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:16.561285  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:00:16.565467  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:00:16.566155  2329 raft_consensus.cc:3060] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:00:16.567392  4790 leader_election.cc:290] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 13:00:16.527868  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 103592us
I20260709 13:00:16.529222  4882 raft_consensus.cc:493] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:16.530094  4547 leader_election.cc:290] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:16.326159  4628 raft_consensus.cc:515] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:16.530524  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 8 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:16.531114  4574 raft_consensus.cc:2804] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:00:16.463723  4690 raft_consensus.cc:740] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:00:16.326373  4846 raft_consensus.cc:515] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:16.485381  4598 raft_consensus.cc:515] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:16.295285  4887 raft_consensus.cc:515] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:16.425166  4601 raft_consensus.cc:515] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:16.567852  4847 leader_election.cc:290] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:00:16.616895  2301 leader_election.cc:336] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Timed out: RequestConsensusVote RPC to 127.0.107.195:40693 timed out after 1.619s (SENT)
I20260709 13:00:16.563580  1673 raft_consensus.cc:2468] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 6.
I20260709 13:00:16.763207  2301 leader_election.cc:304] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:00:16.763392  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:00:16.764536  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 147771us
W20260709 13:00:16.765458  2301 leader_election.cc:336] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Timed out: RequestConsensusVote RPC to 127.0.107.195:40693 timed out after 1.805s (SENT)
I20260709 13:00:16.765606  1597 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 2011) took 1340 ms. Trace:
I20260709 13:00:16.766034  2301 leader_election.cc:304] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
W20260709 13:00:16.705605  1569 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:16.304064  4782 raft_consensus.cc:515] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:16.326267  4629 raft_consensus.cc:515] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:16.554646  1599 raft_consensus.cc:740] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
W20260709 13:00:16.781278  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 299426us
I20260709 13:00:16.658156  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:16.589123  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:16.790477  1749 raft_consensus.cc:2368] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 5. Current term is 6.
I20260709 13:00:16.614354  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:00:16.568269  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 136718us
W20260709 13:00:16.798563  1646 leader_election.cc:336] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Timed out: RequestConsensusVote RPC to 127.0.107.195:40693 timed out after 1.614s (SENT)
I20260709 13:00:16.781348  1718 leader_election.cc:304] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:00:16.766397  1597 rpcz_store.cc:276] 0709 13:00:15.424830 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:15.425018 (+   188us) service_pool.cc:224] Handling call
0709 13:00:15.433923 (+  8905us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:00:15.507043 (+ 73120us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:00:15.507093 (+    50us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:00:15.859114 (+352021us) log.cc:844] Serialized 26 byte log entry
0709 13:00:16.295837 (+436723us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:00:16.295884 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:16.295956 (+    72us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:00:16.685116 (+389160us) raft_consensus.cc:1685] finished
0709 13:00:16.685139 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:16.765503 (+ 80364us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":64393,"raft.queue_time_us":112381,"raft.run_cpu_time_us":150,"raft.run_wall_time_us":151,"thread_start_us":241326,"threads_started":1,"wal-append.queue_time_us":609628}
I20260709 13:00:16.542482  4897 raft_consensus.cc:493] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:00:16.800597  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:16.821751  1718 leader_election.cc:304] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:16.824033  1718 leader_election.cc:304] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:16.694238  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:16.323853  4896 raft_consensus.cc:515] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:16.302634  4731 raft_consensus.cc:493] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:16.556105  4862 raft_consensus.cc:2749] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260709 13:00:16.784463  1569 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:00:16.622777  4749 consensus_meta.cc:287] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 1.382s	user 0.016s	sys 0.000s
I20260709 13:00:16.544276  4679 raft_consensus.cc:515] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:00:16.771865  4895 log.cc:927] Time spent T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.463s	user 0.001s	sys 0.000s
I20260709 13:00:16.323966  4746 raft_consensus.cc:493] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:00:16.829639  1754 consensus_meta.cc:287] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.769s	user 0.008s	sys 0.004s
I20260709 13:00:16.851586  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
W20260709 13:00:16.862804  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 64279.8us
W20260709 13:00:16.865427  1646 leader_election.cc:336] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Timed out: RequestConsensusVote RPC to 127.0.107.195:40693 timed out after 1.619s (SENT)
I20260709 13:00:16.865900  1646 leader_election.cc:304] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:00:16.863642  1754 raft_consensus.cc:2468] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 6.
I20260709 13:00:16.424940  4802 consensus_queue.cc:260] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:16.832870  1678 raft_consensus.cc:1240] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 4. Current term is 5. Ops: []
I20260709 13:00:16.762650  4574 raft_consensus.cc:493] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:16.837888  1671 raft_consensus.cc:1275] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 13:00:16.614334  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:00:16.843451  1569 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:16.844988  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:16.796154  1599 consensus_queue.cc:260] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:16.881716  1599 raft_consensus.cc:3060] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:00:16.834688  4537 leader_election.cc:290] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:16.771291  4740 leader_election.cc:290] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:16.692132  4708 leader_election.cc:290] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:16.793272  4547 raft_consensus.cc:493] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:16.793840  4512 leader_election.cc:290] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:16.900417  1677 raft_consensus.cc:1240] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20260709 13:00:16.869779  1678 raft_consensus.cc:1240] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 5. Current term is 6. Ops: []
W20260709 13:00:16.879046  1570 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:16.800478  4790 raft_consensus.cc:493] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:16.523178  4872 consensus_queue.cc:237] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:16.800434  4893 raft_consensus.cc:2749] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:00:16.599400  4723 raft_consensus.cc:515] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:16.890051  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 124617us
I20260709 13:00:16.848235  4887 leader_election.cc:290] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:16.842083  4779 raft_consensus.cc:2749] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260709 13:00:16.881419  4884 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.551s	user 0.022s	sys 0.018s
I20260709 13:00:16.849468  4904 raft_consensus.cc:2804] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:00:16.880621  4574 raft_consensus.cc:3060] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:00:16.792773  4882 raft_consensus.cc:515] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:16.582746  4849 raft_consensus.cc:515] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:16.916116  1749 raft_consensus.cc:1240] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 6. Current term is 7. Ops: []
I20260709 13:00:16.842424  4897 raft_consensus.cc:515] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:16.894286  4862 raft_consensus.cc:493] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:00:16.923780  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:16.905253  4802 raft_consensus.cc:3060] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:00:16.894186  4845 leader_election.cc:290] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 7 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:16.905978  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:00:16.902120  4683 leader_election.cc:290] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:16.929342  4904 raft_consensus.cc:493] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
W20260709 13:00:16.916627  4884 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.591s	user 0.022s	sys 0.018s
I20260709 13:00:16.891983  2330 raft_consensus.cc:2468] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 4.
I20260709 13:00:16.849632  4731 raft_consensus.cc:515] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:16.886950  4746 raft_consensus.cc:515] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:16.947359  4862 raft_consensus.cc:515] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:16.942247  1752 raft_consensus.cc:2368] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 8. Current term is 9.
I20260709 13:00:17.028843  4862 leader_election.cc:290] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:00:16.945189  2300 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:00:16.947103  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 123120us
I20260709 13:00:17.030121  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:16.930015  4848 leader_election.cc:290] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:16.873278  4749 raft_consensus.cc:515] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:00:16.978701  4903 log.cc:927] Time spent T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.405s	user 0.001s	sys 0.000s
I20260709 13:00:17.038045  4749 leader_election.cc:290] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:16.963131  2336 raft_consensus.cc:2468] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 5.
I20260709 13:00:17.040733  4601 leader_election.cc:290] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:00:17.041860  2298 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:17.046080  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:17.047149  1679 raft_consensus.cc:1275] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 13:00:16.941145  4893 raft_consensus.cc:2804] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:00:16.911711  4547 raft_consensus.cc:515] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:16.910430  4790 raft_consensus.cc:515] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:17.066974  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:17.067687  1596 raft_consensus.cc:2468] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 5.
I20260709 13:00:17.068555  4547 leader_election.cc:290] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:17.068890  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:17.069564  1598 raft_consensus.cc:2468] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 7.
I20260709 13:00:17.070987  1596 raft_consensus.cc:1240] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 5. Current term is 6. Ops: []
W20260709 13:00:16.980116  2301 leader_election.cc:336] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Timed out: RequestConsensusVote RPC to 127.0.107.195:40693 timed out after 1.976s (SENT)
I20260709 13:00:17.071893  2301 leader_election.cc:304] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:00:17.072320  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:17.068794  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:00:17.072971  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 92908.8us
W20260709 13:00:17.073685  2329 consensus_meta.cc:287] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.507s	user 0.004s	sys 0.008s
I20260709 13:00:17.073921  4887 raft_consensus.cc:2749] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 13:00:17.074039  2298 leader_election.cc:304] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:00:16.969324  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:16.984038  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:00:16.986501  4908 log.cc:927] Time spent T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.081s	user 0.000s	sys 0.001s
W20260709 13:00:16.991602  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:16.995867  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:16.996295  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:17.077972  1745 raft_consensus.cc:2468] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 6.
I20260709 13:00:17.078365  1751 raft_consensus.cc:2393] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 5: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:00:17.079304  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:16.997066  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:16.999400  4846 leader_election.cc:290] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:17.004623  4876 leader_election.cc:290] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:17.005071  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:17.006670  1747 raft_consensus.cc:2468] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 5.
I20260709 13:00:17.010308  4723 leader_election.cc:290] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:17.010931  4683 raft_consensus.cc:493] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:17.015697  4679 leader_election.cc:290] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:17.020006  4897 leader_election.cc:290] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:16.959329  4875 leader_election.cc:290] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:16.906184  4690 consensus_queue.cc:260] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.8, Last appended by leader: 8, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:16.947042  4847 raft_consensus.cc:493] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:17.031139  1752 raft_consensus.cc:2368] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 5. Current term is 6.
I20260709 13:00:17.031999  1718 leader_election.cc:304] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:00:17.082398  4779 raft_consensus.cc:2804] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:00:17.120455  4690 raft_consensus.cc:3060] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:00:17.120994  2328 raft_consensus.cc:1275] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 13:00:17.121070  4896 leader_election.cc:290] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:17.033881  4598 leader_election.cc:290] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:17.046872  1605 raft_consensus.cc:2468] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 4.
W20260709 13:00:17.056910  1645 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:17.120477  4779 raft_consensus.cc:493] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:00:17.127737  4779 raft_consensus.cc:3060] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:00:17.127921  4882 leader_election.cc:290] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:17.059501  1746 raft_consensus.cc:1275] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260709 13:00:17.128679  4859 raft_consensus.cc:493] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:17.129252  4859 raft_consensus.cc:515] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:17.129745  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:16.979787  4628 leader_election.cc:290] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:17.066220  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:17.046271  4601 raft_consensus.cc:493] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:17.075191  1749 raft_consensus.cc:3060] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:00:16.944700  1502 catalog_manager.cc:5697] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 reported cstate change: term changed from 2 to 4, leader changed from 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) to af520b67945d443b901bbeca37cc5701 (127.0.107.195). New cstate: current_term: 4 leader_uuid: "af520b67945d443b901bbeca37cc5701" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 13:00:17.068590  1569 leader_election.cc:304] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
W20260709 13:00:17.143258  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 74759.3us
W20260709 13:00:17.073712  2301 leader_election.cc:336] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Timed out: RequestConsensusVote RPC to 127.0.107.195:40693 timed out after 2.015s (SENT)
I20260709 13:00:17.145864  2301 leader_election.cc:304] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:00:17.080814  1717 leader_election.cc:304] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 13:00:17.147553  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 66853.5us
I20260709 13:00:17.148583  1717 leader_election.cc:304] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:17.082705  1753 raft_consensus.cc:2393] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 7: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:00:16.976301  4904 raft_consensus.cc:3060] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:00:17.086061  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:17.150110  1717 leader_election.cc:304] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
W20260709 13:00:17.084659  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 219249us
W20260709 13:00:17.153187  1646 leader_election.cc:336] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Timed out: RequestConsensusVote RPC to 127.0.107.195:40693 timed out after 1.687s (SENT)
I20260709 13:00:17.153589  1646 leader_election.cc:304] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:00:17.075218  2298 leader_election.cc:304] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:00:17.100402  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:16.986555  4849 leader_election.cc:290] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:17.060889  4518 raft_consensus.cc:2749] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 13:00:17.050859  4722 raft_consensus.cc:2749] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:00:17.028900  4746 leader_election.cc:290] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:00:17.171602  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 96462.5us
I20260709 13:00:17.051030  4893 raft_consensus.cc:493] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:00:17.068555  4790 leader_election.cc:290] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:17.032092  4782 leader_election.cc:290] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:17.068696  4759 raft_consensus.cc:493] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:17.109405  4847 raft_consensus.cc:515] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:17.110522  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:17.074090  2329 raft_consensus.cc:2468] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 8.
I20260709 13:00:17.116842  4731 leader_election.cc:290] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:00:17.117784  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 85833.7us
I20260709 13:00:17.118341  4920 raft_consensus.cc:2749] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:00:17.107329  4683 raft_consensus.cc:515] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:00:17.167183  1569 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:17.162497  1502 catalog_manager.cc:5697] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 reported cstate change: term changed from 1 to 5. New cstate: current_term: 5 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
W20260709 13:00:17.212504  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 138816us
I20260709 13:00:17.168181  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:17.168569  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:17.213822  1751 raft_consensus.cc:2368] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 8. Current term is 9.
I20260709 13:00:17.172798  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:17.214989  1672 raft_consensus.cc:2410] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for term 7 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 4.
I20260709 13:00:17.173688  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:17.174492  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:17.217036  1680 raft_consensus.cc:2393] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 7: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:00:17.177831  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:00:17.171406  1644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:17.217875  1674 raft_consensus.cc:2468] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 7.
I20260709 13:00:17.188738  1749 raft_consensus.cc:2468] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 7.
I20260709 13:00:17.195395  4740 raft_consensus.cc:2804] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:00:17.175539  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:17.200407  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:17.220162  1747 raft_consensus.cc:2468] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 7.
W20260709 13:00:17.221154  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 67986.6us
I20260709 13:00:17.221441  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
W20260709 13:00:17.222075  1646 leader_election.cc:336] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 9 election: RPC error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Timed out: RequestConsensusVote RPC to 127.0.107.195:40693 timed out after 1.911s (SENT)
I20260709 13:00:17.222739  1747 raft_consensus.cc:2368] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 8. Current term is 9.
W20260709 13:00:17.223333  1646 leader_election.cc:336] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Timed out: RequestConsensusVote RPC to 127.0.107.195:40693 timed out after 1.978s (SENT)
I20260709 13:00:17.224218  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:17.225090  1747 raft_consensus.cc:2468] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 7.
I20260709 13:00:17.202128  1752 raft_consensus.cc:2393] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 8: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:00:17.169915  4679 raft_consensus.cc:2749] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:00:17.164127  4598 raft_consensus.cc:2749] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:00:17.164474  4846 raft_consensus.cc:2804] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader election won for term 6
I20260709 13:00:17.167945  4723 raft_consensus.cc:2804] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:00:17.196116  4893 raft_consensus.cc:3060] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Advancing to term 6
W20260709 13:00:17.213187  2301 leader_election.cc:336] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Timed out: RequestConsensusVote RPC to 127.0.107.195:40693 timed out after 2.149s (SENT)
I20260709 13:00:17.143131  4896 raft_consensus.cc:493] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:17.201323  1671 raft_consensus.cc:1240] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 6. Current term is 7. Ops: []
I20260709 13:00:17.217785  2298 leader_election.cc:304] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:00:17.226979  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:17.256974  1747 raft_consensus.cc:3060] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Advancing to term 5
W20260709 13:00:17.207271  4580 consensus_meta.cc:287] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.888s	user 0.014s	sys 0.000s
I20260709 13:00:17.239840  1745 raft_consensus.cc:2468] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 5.
I20260709 13:00:17.240754  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:17.241398  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:17.260325  1751 raft_consensus.cc:2468] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 5.
I20260709 13:00:17.260684  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:17.261302  1745 raft_consensus.cc:2468] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 7.
I20260709 13:00:17.210505  1718 leader_election.cc:304] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:00:17.242190  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:00:17.228811  4672 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.643s	user 0.011s	sys 0.036s
I20260709 13:00:17.251775  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:17.171038  4749 consensus_queue.cc:1059] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.059s
W20260709 13:00:17.254365  1643 leader_election.cc:336] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 2.042s (SENT)
I20260709 13:00:17.171880  4648 consensus_queue.cc:1048] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.057s
I20260709 13:00:17.166584  4601 raft_consensus.cc:515] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:17.267822  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 50102.1us
I20260709 13:00:17.225819  4746 raft_consensus.cc:2749] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:00:17.273224  1502 catalog_manager.cc:5697] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 reported cstate change: term changed from 3 to 6, leader changed from 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) to af520b67945d443b901bbeca37cc5701 (127.0.107.195). New cstate: current_term: 6 leader_uuid: "af520b67945d443b901bbeca37cc5701" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 13:00:17.262394  1675 raft_consensus.cc:2468] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 8.
I20260709 13:00:17.200524  4887 raft_consensus.cc:2749] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 13:00:17.277737  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:17.280304  1749 raft_consensus.cc:2393] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 5: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:00:17.252867  2330 raft_consensus.cc:1240] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 7. Current term is 8. Ops: []
I20260709 13:00:17.211313  4759 raft_consensus.cc:515] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:17.269754  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:17.270948  1671 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52766 (request call id 2272) took 1043 ms. Trace:
I20260709 13:00:17.281137  1502 catalog_manager.cc:5697] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 reported cstate change: term changed from 4 to 7, leader changed from 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194) to 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196). New cstate: current_term: 7 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:00:17.225970  4740 raft_consensus.cc:493] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:00:17.271073  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 60709.5us
I20260709 13:00:17.271965  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:17.251367  4723 raft_consensus.cc:493] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:00:17.263389  4672 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.696s	user 0.011s	sys 0.036s
I20260709 13:00:17.302081  1671 rpcz_store.cc:276] 0709 13:00:16.227513 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:16.875579 (+648066us) service_pool.cc:224] Handling call
0709 13:00:17.198292 (+322713us) spinlock_profiling.cc:243] Waited 12.1 ms on lock 0x7b54001e7c28. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426b08baa1 00007f426b106ebc 00007f426b13c08c 00007f426b12693b 00007f426b1263d2 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c
0709 13:00:17.201101 (+  2809us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:17.270560 (+ 69459us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:17.270862 (+   302us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":27789056}
W20260709 13:00:17.266110  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 112488us
W20260709 13:00:17.282415  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 69250.9us
I20260709 13:00:17.211155  4790 consensus_queue.cc:1048] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.056s
I20260709 13:00:17.333564  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:17.261225  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:17.334407  1749 raft_consensus.cc:2468] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 5.
I20260709 13:00:17.273656  4920 raft_consensus.cc:2749] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:00:17.325465  1752 raft_consensus.cc:2368] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 8. Current term is 9.
I20260709 13:00:17.336006  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:17.337599  2334 raft_consensus.cc:1240] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 7 LEADER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 6. Current term is 7. Ops: []
I20260709 13:00:17.337782  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:17.337970  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:17.306311  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:17.328963  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:17.328619  1502 catalog_manager.cc:5697] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 reported cstate change: term changed from 1 to 6, leader changed from af520b67945d443b901bbeca37cc5701 (127.0.107.195) to 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196). New cstate: current_term: 6 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:00:17.281471  4849 raft_consensus.cc:493] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:00:17.332530  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 78201us
I20260709 13:00:17.336689  1751 raft_consensus.cc:2393] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 7: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:00:17.340299  1745 raft_consensus.cc:2468] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 6.
W20260709 13:00:17.343190  1643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:17.338685  1671 raft_consensus.cc:2468] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 6.
I20260709 13:00:17.338410  1672 raft_consensus.cc:2468] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 4.
I20260709 13:00:17.346076  1747 raft_consensus.cc:2468] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 5.
I20260709 13:00:17.269284  4896 raft_consensus.cc:515] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:17.324478  4740 raft_consensus.cc:3060] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:00:17.326835  4723 raft_consensus.cc:3060] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:00:17.282116  4875 raft_consensus.cc:2804] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:00:17.301546  4731 raft_consensus.cc:3055] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:00:17.365082  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:17.264438  4846 raft_consensus.cc:697] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
W20260709 13:00:17.391782  1569 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:17.269133  4580 raft_consensus.cc:515] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:17.402240  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
W20260709 13:00:17.410483  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 185856us
W20260709 13:00:17.482900  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:17.333397  4887 raft_consensus.cc:493] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:17.424371  1750 raft_consensus.cc:2468] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 6.
I20260709 13:00:17.301453  4574 raft_consensus.cc:515] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:00:17.437350  1569 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:17.400693  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
W20260709 13:00:17.488816  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:17.484082  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:00:17.484006  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 437280us
W20260709 13:00:17.504459  1571 leader_election.cc:336] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Timed out: RequestConsensusVote RPC to 127.0.107.195:40693 timed out after 2.349s (SENT)
I20260709 13:00:17.505395  1502 catalog_manager.cc:5697] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 reported cstate change: term changed from 2 to 5. New cstate: current_term: 5 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 13:00:17.507308  2300 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:17.438844  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:00:17.489099  1646 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:17.508834  1675 raft_consensus.cc:2468] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 9.
W20260709 13:00:17.523175  4925 log.cc:927] Time spent T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.198s	user 0.001s	sys 0.000s
I20260709 13:00:17.527272  4846 consensus_queue.cc:237] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:17.401702  4849 raft_consensus.cc:515] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:17.400601  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:17.570889  1753 raft_consensus.cc:2368] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 8. Current term is 9.
W20260709 13:00:17.619851  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 287517us
W20260709 13:00:17.617373  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 273480us
I20260709 13:00:17.617908  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:17.537045  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:17.629305  1676 raft_consensus.cc:2368] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 6. Current term is 7.
W20260709 13:00:17.610270  1718 consensus_peers.cc:597] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 -> Peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625): Couldn't send request to peer 3b879799899a499c8d409ee9f1277cd1. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:00:17.630319  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 309206us
W20260709 13:00:17.630487  1717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:17.617879  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:00:17.620479  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 8 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:17.436161  4731 raft_consensus.cc:740] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:00:17.635577  1751 raft_consensus.cc:2468] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 7.
W20260709 13:00:17.501675  4921 log.cc:927] Time spent T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.234s	user 0.001s	sys 0.000s
I20260709 13:00:17.650520  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:00:17.529820  4929 log.cc:927] Time spent T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.158s	user 0.000s	sys 0.000s
W20260709 13:00:17.650050  2301 leader_election.cc:336] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Timed out: RequestConsensusVote RPC to 127.0.107.195:40693 timed out after 2.111s (SENT)
W20260709 13:00:17.648212  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 143773us
I20260709 13:00:17.658864  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:17.635988  1502 catalog_manager.cc:5697] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 reported cstate change: term changed from 2 to 5, leader changed from 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194) to 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196). New cstate: current_term: 5 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:00:17.660781  2301 leader_election.cc:304] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
W20260709 13:00:17.529095  4923 log.cc:927] Time spent T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.158s	user 0.000s	sys 0.000s
I20260709 13:00:17.656335  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:17.672909  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:17.491658  4574 leader_election.cc:290] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 9 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:17.673506  1675 raft_consensus.cc:2468] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 5.
I20260709 13:00:17.673777  1602 raft_consensus.cc:2410] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260709 13:00:17.490906  4580 leader_election.cc:290] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 7 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:17.677265  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:17.677603  1671 raft_consensus.cc:1275] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260709 13:00:17.678072  1749 raft_consensus.cc:2393] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 5: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:00:17.606655  4875 raft_consensus.cc:493] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
W20260709 13:00:17.576846  4902 log.cc:927] Time spent T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 1.135s	user 0.003s	sys 0.000s
W20260709 13:00:17.535498  4860 log.cc:927] Time spent T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 2.127s	user 0.020s	sys 0.021s
I20260709 13:00:17.484524  4859 leader_election.cc:290] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:17.682619  4707 consensus_queue.cc:1048] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
W20260709 13:00:17.576859  4917 log.cc:927] Time spent T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.454s	user 0.001s	sys 0.000s
I20260709 13:00:17.662447  4931 raft_consensus.cc:493] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:17.692210  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:17.693301  1601 raft_consensus.cc:2410] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for term 8 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20260709 13:00:17.656342  4679 raft_consensus.cc:493] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:17.665638  4740 raft_consensus.cc:515] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:17.666487  4905 consensus_queue.cc:1048] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.014s
I20260709 13:00:17.668706  1599 raft_consensus.cc:1240] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 9. Current term is 10. Ops: []
I20260709 13:00:17.669126  1605 raft_consensus.cc:1240] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 5. Current term is 6. Ops: []
W20260709 13:00:17.672211  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 44482.7us
W20260709 13:00:17.608000  4928 log.cc:927] Time spent T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.086s	user 0.000s	sys 0.001s
I20260709 13:00:17.678790  1747 raft_consensus.cc:2468] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 6.
I20260709 13:00:17.635140  4759 leader_election.cc:290] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:17.576524  4683 leader_election.cc:290] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:17.681313  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:17.489456  4887 raft_consensus.cc:515] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:17.528066  4601 leader_election.cc:290] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:17.687573  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:17.714021  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:17.650772  1569 leader_election.cc:304] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:00:17.690055  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:17.728045  1598 raft_consensus.cc:2468] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 9.
I20260709 13:00:17.687500  4897 raft_consensus.cc:2749] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20260709 13:00:17.698618  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 48710.5us
I20260709 13:00:17.707875  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:17.733815  1599 raft_consensus.cc:2468] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 4.
I20260709 13:00:17.708024  4731 consensus_queue.cc:260] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:17.708379  1643 leader_election.cc:304] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:17.708725  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:17.711099  4723 raft_consensus.cc:515] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:17.743316  1745 raft_consensus.cc:2368] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 5. Current term is 7.
I20260709 13:00:17.744120  4859 raft_consensus.cc:2749] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:00:17.744221  4723 leader_election.cc:290] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 7 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:17.745498  4859 consensus_queue.cc:1048] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:00:17.581771  4904 raft_consensus.cc:515] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:17.540673  4629 leader_election.cc:290] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:17.719193  1751 raft_consensus.cc:2393] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 8: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:00:17.703222  4931 raft_consensus.cc:515] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:17.748400  4904 leader_election.cc:290] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 8 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:17.748591  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:17.685812  4875 raft_consensus.cc:3060] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:00:17.722659  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:17.719113  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:00:17.768535  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:17.768960  1596 raft_consensus.cc:2393] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 9: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:00:17.771984  2336 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49082 (request call id 706) took 1205 ms. Trace:
I20260709 13:00:17.772408  2336 rpcz_store.cc:276] 0709 13:00:16.568690 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:16.770220 (+201530us) service_pool.cc:224] Handling call
0709 13:00:17.771787 (+1001567us) spinlock_profiling.cc:243] Waited 141 ms on lock 0x7b5400627ad4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:00:17.771912 (+   125us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":325042560}
I20260709 13:00:17.765755  4723 consensus_queue.cc:1059] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.008s
I20260709 13:00:17.774405  4904 raft_consensus.cc:3055] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:00:17.725025  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:17.686173  1502 catalog_manager.cc:5697] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 reported cstate change: term changed from 3 to 4, leader changed from 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) to 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196). New cstate: current_term: 4 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:00:17.782055  1502 catalog_manager.cc:5697] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 reported cstate change: term changed from 6 to 8. New cstate: current_term: 8 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 13:00:17.730381  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 79694us
I20260709 13:00:17.732007  1603 raft_consensus.cc:2410] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for term 8 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260709 13:00:17.735780  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:17.784375  1749 raft_consensus.cc:2393] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 7: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
W20260709 13:00:17.740206  1570 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:00:17.785355  1569 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:17.741070  4601 raft_consensus.cc:2804] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Leader election won for term 8
I20260709 13:00:17.741495  4731 raft_consensus.cc:3060] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:00:17.750108  4931 leader_election.cc:290] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 13:00:17.764093  1571 connection.cc:256] RPC call timeout handler was delayed by -2.0546427050000489s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21320000000000003s for a response
I20260709 13:00:17.750233  1751 raft_consensus.cc:1275] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 7. (term mismatch)
I20260709 13:00:17.720620  4920 raft_consensus.cc:493] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:17.713483  4749 raft_consensus.cc:493] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:00:17.790019  1646 leader_election.cc:336] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Timed out: RequestConsensusVote RPC to 127.0.107.195:40693 timed out after 1.521s (SENT)
I20260709 13:00:17.817333  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:17.813701  1751 pending_rounds.cc:85] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701: Aborting all ops after (but not including) 6
I20260709 13:00:17.818055  1751 pending_rounds.cc:107] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701: Aborting uncommitted NO_OP operation due to leader change: 5.7
I20260709 13:00:17.815284  2332 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49082 (request call id 699) took 1251 ms. Trace:
I20260709 13:00:17.816025  1673 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52766 (request call id 2262) took 1675 ms. Trace:
I20260709 13:00:17.820994  1751 raft_consensus.cc:2889] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 13:00:17.795029  4893 consensus_meta.cc:287] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.517s	user 0.012s	sys 0.003s
W20260709 13:00:17.814251  1571 leader_election.cc:336] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 9 election: RPC error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Timed out: RequestConsensusVote RPC to 127.0.107.195:40693 timed out after 1.512s (SENT)
I20260709 13:00:17.818130  1597 raft_consensus.cc:3060] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Advancing to term 9
I20260709 13:00:17.766875  4896 leader_election.cc:290] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:17.818572  1673 rpcz_store.cc:276] 0709 13:00:16.140815 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:16.786914 (+646099us) service_pool.cc:224] Handling call
0709 13:00:16.787366 (+   452us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:00:16.965427 (+178061us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:00:16.965478 (+    51us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:00:16.969166 (+  3688us) log.cc:844] Serialized 26 byte log entry
0709 13:00:17.178787 (+209621us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:00:17.178839 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:17.178910 (+    71us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:00:17.815783 (+636873us) raft_consensus.cc:1685] finished
0709 13:00:17.815805 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:17.815957 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":111323,"raft.run_cpu_time_us":156,"raft.run_wall_time_us":3191,"thread_start_us":357616,"threads_started":2,"wal-append.queue_time_us":296236}
I20260709 13:00:17.819623  2332 rpcz_store.cc:276] 0709 13:00:16.563932 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:16.566558 (+  2626us) service_pool.cc:224] Handling call
0709 13:00:16.566896 (+   338us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:00:16.656716 (+ 89820us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:00:16.656784 (+    68us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:00:16.771568 (+114784us) log.cc:844] Serialized 26 byte log entry
0709 13:00:17.049936 (+278368us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:00:17.049979 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:17.050050 (+    71us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:00:17.815057 (+765007us) raft_consensus.cc:1685] finished
0709 13:00:17.815079 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:17.815209 (+   130us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":42166,"raft.queue_time_us":90007,"raft.run_cpu_time_us":115,"raft.run_wall_time_us":21806,"thread_start_us":200068,"threads_started":1,"wal-append.queue_time_us":334828}
I20260709 13:00:17.833132  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:17.827788  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:17.834910  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:17.775030  4904 raft_consensus.cc:740] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:00:17.841668  4783 consensus_queue.cc:1048] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 13:00:17.843473  4708 consensus_queue.cc:1048] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 13:00:17.780239  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:17.724052  4679 raft_consensus.cc:515] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:17.879130  4936 log.cc:927] Time spent T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.087s	user 0.000s	sys 0.000s
I20260709 13:00:17.880167  4679 leader_election.cc:290] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:17.847203  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:17.881752  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:17.882746  2337 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39726 (request call id 857) took 1268 ms. Trace:
I20260709 13:00:17.847798  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:17.884063  1680 raft_consensus.cc:2410] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260709 13:00:17.849478  1502 catalog_manager.cc:5697] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 reported cstate change: term changed from 2 to 6, leader changed from 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) to 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196). New cstate: current_term: 6 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 13:00:17.885921  1569 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:17.885627  1502 catalog_manager.cc:5697] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:00:17.886660  1569 leader_election.cc:304] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 13:00:17.888469  1569 leader_election.cc:304] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:00:17.858680  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:17.889304  1671 raft_consensus.cc:2393] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 5: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:00:17.890661  4537 raft_consensus.cc:2804] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:00:17.891099  4537 raft_consensus.cc:493] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:17.891444  4537 raft_consensus.cc:3060] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:00:17.860394  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:17.861894  1749 raft_consensus.cc:1275] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260709 13:00:17.831769  4749 raft_consensus.cc:515] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:17.863973  4849 leader_election.cc:290] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:17.865489  4887 leader_election.cc:290] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:17.867561  4740 leader_election.cc:290] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 8 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:17.800218  4512 consensus_queue.cc:1048] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.069s
I20260709 13:00:17.936105  1679 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57092 (request call id 685) took 1769 ms. Trace:
I20260709 13:00:17.870009  4877 raft_consensus.cc:3060] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Advancing to term 6
W20260709 13:00:17.870589  4690 consensus_meta.cc:287] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.750s	user 0.008s	sys 0.006s
I20260709 13:00:17.871600  1600 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 2013) took 2283 ms. Trace:
I20260709 13:00:17.871716  4872 raft_consensus.cc:493] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:17.872391  2334 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49082 (request call id 697) took 1329 ms. Trace:
I20260709 13:00:17.941524  2334 rpcz_store.cc:276] 0709 13:00:16.542881 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:16.558526 (+ 15645us) service_pool.cc:224] Handling call
0709 13:00:16.618804 (+ 60278us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:17.338046 (+719242us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:17.872161 (+534115us) spinlock_profiling.cc:243] Waited 718 ms on lock 0x7b54005e01d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:00:17.872326 (+   165us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1652387200}
I20260709 13:00:17.942003  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:00:17.872782  4779 consensus_meta.cc:287] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.744s	user 0.010s	sys 0.003s
I20260709 13:00:17.872969  1746 raft_consensus.cc:2368] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 5. Current term is 6.
I20260709 13:00:17.875613  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:00:17.875869  1644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:17.883180  2337 rpcz_store.cc:276] 0709 13:00:16.614820 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:17.097544 (+482724us) service_pool.cc:224] Handling call
0709 13:00:17.109487 (+ 11943us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:00:17.122996 (+ 13509us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:00:17.123028 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:00:17.172410 (+ 49382us) log.cc:844] Serialized 26 byte log entry
0709 13:00:17.408334 (+235924us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:00:17.408395 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:17.408506 (+   111us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:00:17.882498 (+473992us) raft_consensus.cc:1685] finished
0709 13:00:17.882523 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:17.882669 (+   146us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":38479,"raft.queue_time_us":13562,"raft.run_cpu_time_us":161,"raft.run_wall_time_us":16379,"spinlock_wait_cycles":1820544,"thread_start_us":177309,"threads_started":1,"wal-append.queue_time_us":495609}
I20260709 13:00:17.887794  1502 catalog_manager.cc:5697] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 reported cstate change: term changed from 3 to 6, leader changed from af520b67945d443b901bbeca37cc5701 (127.0.107.195) to 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196). New cstate: current_term: 6 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:00:17.856974  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:17.896595  1604 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 2015) took 2219 ms. Trace:
I20260709 13:00:17.897008  1677 raft_consensus.cc:1275] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20260709 13:00:17.898770  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:17.949283  1671 raft_consensus.cc:2393] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 7: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:00:17.954730  1672 raft_consensus.cc:2410] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for term 8 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260709 13:00:17.970515  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:17.896618  1569 leader_election.cc:400] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 6 pre-election: Vote denied by peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261) with higher term. Message: Invalid argument: T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 6. Current term is 7.
I20260709 13:00:17.928555  2299 leader_election.cc:304] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:00:17.929328  1748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55848 (request call id 2046) took 1768 ms. Trace:
I20260709 13:00:17.984198  1569 leader_election.cc:403] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 6 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:00:17.984401  1748 rpcz_store.cc:276] 0709 13:00:16.160890 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:16.161019 (+   129us) service_pool.cc:224] Handling call
0709 13:00:17.924969 (+1763950us) spinlock_profiling.cc:243] Waited 7.84 ms on lock 0x7b54002f49d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:00:17.929262 (+  4293us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":18027264}
W20260709 13:00:17.984804  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 88318.9us
I20260709 13:00:17.929661  1751 raft_consensus.cc:1275] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 9 index: 6. (term mismatch)
I20260709 13:00:17.894238  4859 consensus_queue.cc:1048] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 13:00:17.932580  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:00:17.986634  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 58189us
I20260709 13:00:17.937952  1600 rpcz_store.cc:276] 0709 13:00:15.569723 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:15.569909 (+   186us) service_pool.cc:224] Handling call
0709 13:00:15.570386 (+   477us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:00:15.611218 (+ 40832us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:00:15.611271 (+    53us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:00:16.574660 (+963389us) spinlock_profiling.cc:243] Waited 216 ms on lock 0x7b2c00000658. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4264a5ba11 00007f4264a6b372 00007f426619da01 00007f426616243a 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2
0709 13:00:16.602987 (+ 28327us) log.cc:844] Serialized 26 byte log entry
0709 13:00:16.913610 (+310623us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:00:16.913671 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:16.913787 (+   116us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:00:17.828813 (+915026us) raft_consensus.cc:1685] finished
0709 13:00:17.837027 (+  8214us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:17.852551 (+ 15524us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":156423,"raft.run_cpu_time_us":145,"raft.run_wall_time_us":145,"spinlock_wait_cycles":495821056,"thread_start_us":126705,"threads_started":1,"wal-append.queue_time_us":280715}
I20260709 13:00:17.900456  4549 consensus_queue.cc:1048] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.030s
I20260709 13:00:17.936596  1679 rpcz_store.cc:276] 0709 13:00:16.166984 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:16.841104 (+674120us) service_pool.cc:224] Handling call
0709 13:00:16.841434 (+   330us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:17.047582 (+206148us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:17.935761 (+888179us) spinlock_profiling.cc:243] Waited 205 ms on lock 0x7b5400164754. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:00:17.936020 (+   259us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":472196864}
I20260709 13:00:17.942680  1675 raft_consensus.cc:2468] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 7.
I20260709 13:00:17.944468  1747 raft_consensus.cc:2468] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 7.
W20260709 13:00:17.946221  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 131994us
I20260709 13:00:17.994298  2299 leader_election.cc:304] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 13:00:17.994724  1571 leader_election.cc:336] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Timed out: RequestConsensusVote RPC to 127.0.107.195:40693 timed out after 1.550s (SENT)
I20260709 13:00:17.995306  1571 leader_election.cc:304] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
W20260709 13:00:17.974664  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 230364us
I20260709 13:00:17.985764  1569 leader_election.cc:304] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:00:17.985594  1751 pending_rounds.cc:85] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701: Aborting all ops after (but not including) 5
I20260709 13:00:17.998979  1751 pending_rounds.cc:107] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701: Aborting uncommitted NO_OP operation due to leader change: 8.6
W20260709 13:00:17.999284  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 64633.7us
I20260709 13:00:17.965987  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:17.851675  4884 consensus_queue.cc:1048] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.092s
I20260709 13:00:17.992422  1603 raft_consensus.cc:2468] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 5.
I20260709 13:00:17.948385  1604 rpcz_store.cc:276] 0709 13:00:15.639484 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:15.640735 (+  1251us) service_pool.cc:224] Handling call
0709 13:00:15.641246 (+   511us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:00:16.130960 (+489714us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:00:16.131014 (+    54us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:00:16.143540 (+ 12526us) log.cc:844] Serialized 52 byte log entry
0709 13:00:16.233509 (+ 89969us) raft_consensus.cc:1645] Marking committed up to 1
0709 13:00:16.233561 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:17.070164 (+836603us) spinlock_profiling.cc:243] Waited 479 ms on lock 0x7b54002797d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:00:17.070223 (+    59us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:00:17.575999 (+505776us) spinlock_profiling.cc:243] Waited 52.8 ms on lock 0x7b4400022b4c. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f4262f9bc85 00007f4262f9c6a2 00007f4265d0e1db 00007f4265d09aaf 00007f4265cfd86e 00007f4265cfcbb6 00007f4265cfc7d8 00007f4265d06221 00007f4265d061d9 00007f4265d06169
0709 13:00:17.858979 (+282980us) raft_consensus.cc:1685] finished
0709 13:00:17.859013 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:17.859220 (+   207us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":7658,"raft.run_cpu_time_us":117,"raft.run_wall_time_us":3741,"spinlock_wait_cycles":1223051264,"thread_start_us":68549,"threads_started":1,"wal-append.queue_time_us":133511}
I20260709 13:00:17.933684  2333 raft_consensus.cc:2468] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 7.
I20260709 13:00:17.999490  1751 raft_consensus.cc:2889] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:00:17.903568  2329 raft_consensus.cc:2468] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 7.
I20260709 13:00:17.939481  4905 raft_consensus.cc:2804] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:00:17.897913  4722 consensus_queue.cc:1048] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.068s
I20260709 13:00:18.000321  1646 leader_election.cc:304] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:00:17.987138  1597 raft_consensus.cc:2468] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 9.
I20260709 13:00:18.001978  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:18.016860  1674 raft_consensus.cc:3060] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:00:18.002806  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:18.004360  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:18.005242  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:18.019265  2334 raft_consensus.cc:2468] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 6.
I20260709 13:00:17.940930  4709 consensus_queue.cc:1048] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.081s
I20260709 13:00:18.024309  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:18.026885  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:18.026937  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:18.029788  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:18.031586  2333 raft_consensus.cc:2468] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 9.
I20260709 13:00:18.033504  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:18.034889  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:18.035637  1672 raft_consensus.cc:2468] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 5.
I20260709 13:00:17.965689  4690 raft_consensus.cc:493] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:00:18.044422  4874 log.cc:927] Time spent T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 2.657s	user 0.012s	sys 0.024s
I20260709 13:00:18.044808  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:00:18.045504  4940 log.cc:927] Time spent T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.102s	user 0.000s	sys 0.000s
W20260709 13:00:18.047036  4933 log.cc:927] Time spent T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.186s	user 0.000s	sys 0.001s
I20260709 13:00:18.046892  4722 raft_consensus.cc:2804] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:00:18.050516  1502 catalog_manager.cc:5697] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 reported cstate change: term changed from 2 to 6, leader changed from 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) to 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196). New cstate: current_term: 6 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:00:18.047605  4722 raft_consensus.cc:493] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
W20260709 13:00:18.040525  1571 leader_election.cc:336] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Timed out: RequestConsensusVote RPC to 127.0.107.195:40693 timed out after 2.050s (SENT)
I20260709 13:00:18.055984  1571 leader_election.cc:304] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:00:18.056293  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:17.943321  4849 consensus_queue.cc:1048] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.079s
I20260709 13:00:17.946009  4746 consensus_queue.cc:1048] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.094s
I20260709 13:00:17.952574  4759 consensus_queue.cc:1048] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.088s
I20260709 13:00:18.064651  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:17.846390  4893 raft_consensus.cc:515] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:17.947481  4875 raft_consensus.cc:515] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:17.990895  4740 consensus_queue.cc:1048] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.069s
I20260709 13:00:18.025017  4905 raft_consensus.cc:493] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:00:18.073067  4887 raft_consensus.cc:2749] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:00:18.088562  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:18.089360  1749 raft_consensus.cc:2376] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 6. Re-sending same reply.
I20260709 13:00:18.090881  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:18.091583  1749 raft_consensus.cc:2468] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 7.
I20260709 13:00:18.092690  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:18.093401  1749 raft_consensus.cc:2468] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 7.
I20260709 13:00:18.094805  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:18.095634  1748 raft_consensus.cc:2468] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 6.
I20260709 13:00:18.096765  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:18.097506  1748 raft_consensus.cc:2468] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 7.
I20260709 13:00:18.070940  4690 raft_consensus.cc:515] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:17.952776  4872 raft_consensus.cc:515] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:18.101477  4690 leader_election.cc:290] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:18.023082  4593 consensus_queue.cc:1048] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.062s
I20260709 13:00:18.102336  4593 consensus_queue.cc:1243] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer af520b67945d443b901bbeca37cc5701 log is divergent from this leader: its last log entry 8.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260709 13:00:18.106606  1502 catalog_manager.cc:5697] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 reported cstate change: term changed from 4 to 5, leader changed from 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) to 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194). New cstate: current_term: 5 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 13:00:18.108489  1502 catalog_manager.cc:5697] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 reported cstate change: term changed from 5 to 7, leader changed from 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) to 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196). New cstate: current_term: 7 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:00:18.063768  4598 raft_consensus.cc:493] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:17.991566  4904 consensus_queue.cc:260] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:18.110976  4598 raft_consensus.cc:515] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:18.111846  4904 raft_consensus.cc:3060] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:00:18.113862  4598 leader_election.cc:290] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:18.075785  4896 raft_consensus.cc:3060] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Advancing to term 7
I20260709 13:00:17.968266  4779 raft_consensus.cc:515] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:18.117681  4779 leader_election.cc:290] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 6 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:18.118597  2328 raft_consensus.cc:1240] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 5. Current term is 6. Ops: []
I20260709 13:00:18.118723  4857 raft_consensus.cc:493] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:18.119288  4857 raft_consensus.cc:515] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:18.122390  4857 leader_election.cc:290] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:17.858600  4920 raft_consensus.cc:515] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:18.126076  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:18.048270  4683 raft_consensus.cc:493] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:18.134176  1748 raft_consensus.cc:2468] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 7.
I20260709 13:00:18.130805  1502 catalog_manager.cc:5697] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 reported cstate change: term changed from 1 to 6, leader changed from af520b67945d443b901bbeca37cc5701 (127.0.107.195) to 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196). New cstate: current_term: 6 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:00:18.126827  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:18.138888  1749 raft_consensus.cc:2410] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 for term 6 because replica has last-logged OpId of term: 5 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 7.
I20260709 13:00:18.063406  4722 raft_consensus.cc:3060] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:00:18.132179  1674 raft_consensus.cc:2468] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 9.
I20260709 13:00:18.141302  2331 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39726 (request call id 858) took 1525 ms. Trace:
I20260709 13:00:18.147739  2331 rpcz_store.cc:276] 0709 13:00:16.615398 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:17.119243 (+503845us) service_pool.cc:224] Handling call
0709 13:00:17.119612 (+   369us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:00:17.123562 (+  3950us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:00:17.123592 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:00:17.157338 (+ 33746us) log.cc:844] Serialized 26 byte log entry
0709 13:00:17.361547 (+204209us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:00:17.362827 (+  1280us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:17.362905 (+    78us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:00:18.140970 (+778065us) raft_consensus.cc:1685] finished
0709 13:00:18.141014 (+    44us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:18.141214 (+   200us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3538,"raft.queue_time_us":3966,"raft.run_cpu_time_us":169,"raft.run_wall_time_us":10542,"thread_start_us":127181,"threads_started":1,"wal-append.queue_time_us":547242}
I20260709 13:00:18.124323  4875 leader_election.cc:290] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 5 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:18.156893  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
W20260709 13:00:18.158624  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 219633us
I20260709 13:00:18.086584  4601 raft_consensus.cc:697] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:00:18.159564  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:18.126307  4905 raft_consensus.cc:3060] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:00:18.124598  4574 raft_consensus.cc:493] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:18.102223  4872 leader_election.cc:290] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:18.161402  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:18.170907  1748 raft_consensus.cc:2468] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 7.
I20260709 13:00:18.158160  1749 raft_consensus.cc:2376] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 8. Re-sending same reply.
W20260709 13:00:18.164381  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:18.163513  1718 leader_election.cc:304] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 13:00:18.166580  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 180857us
I20260709 13:00:18.150327  4707 raft_consensus.cc:2749] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:00:18.175067  1718 leader_election.cc:304] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:18.186921  1502 catalog_manager.cc:5697] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 13:00:18.192931  1569 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:00:18.225023  1569 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:18.225236  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:18.225939  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:18.226675  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:00:18.227728  1717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:18.226715  1601 raft_consensus.cc:2468] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 6.
I20260709 13:00:18.227618  1600 raft_consensus.cc:1275] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260709 13:00:18.158125  4683 raft_consensus.cc:515] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:18.227363  1749 raft_consensus.cc:2468] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 6.
I20260709 13:00:18.165372  4598 raft_consensus.cc:493] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:00:18.231031  1646 leader_election.cc:336] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Timed out: RequestConsensusVote RPC to 127.0.107.195:40693 timed out after 1.843s (SENT)
I20260709 13:00:18.163965  4894 raft_consensus.cc:2749] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 13:00:18.232024  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:18.235040  1502 catalog_manager.cc:5697] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:00:18.180718  4547 raft_consensus.cc:2804] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:00:18.240236  1644 leader_election.cc:304] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
W20260709 13:00:18.242599  2301 leader_election.cc:336] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Timed out: RequestConsensusVote RPC to 127.0.107.195:40693 timed out after 1.944s (SENT)
I20260709 13:00:18.245878  1674 raft_consensus.cc:2468] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 7.
I20260709 13:00:18.249269  2301 leader_election.cc:304] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:00:18.246299  1569 leader_election.cc:304] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
W20260709 13:00:18.246807  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 56869.4us
I20260709 13:00:18.252352  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:18.252982  1602 raft_consensus.cc:3060] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:00:18.253908  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:18.254185  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:18.254822  2337 raft_consensus.cc:2468] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 6.
I20260709 13:00:18.255193  2333 raft_consensus.cc:2410] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for term 8 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260709 13:00:18.256294  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:17.929708  4922 consensus_queue.cc:1048] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.080s
I20260709 13:00:18.256811  2333 raft_consensus.cc:1803] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 6 LEARNER]: voting while tombstoned based on last-logged opid 4.5
I20260709 13:00:18.256750  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:18.192725  4690 raft_consensus.cc:2749] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:00:18.192533  4574 raft_consensus.cc:515] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:18.192406  4946 raft_consensus.cc:2749] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:00:18.259593  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:18.259980  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
W20260709 13:00:18.273607  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 233071us
I20260709 13:00:18.274537  2332 raft_consensus.cc:3060] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Advancing to term 7
W20260709 13:00:18.274655  1571 connection.cc:256] RPC call timeout handler was delayed by -1.8120138399999632s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1814s for a response
I20260709 13:00:18.260385  4601 consensus_queue.cc:237] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:18.232750  4779 raft_consensus.cc:493] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:00:18.262961  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:18.276527  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:18.226013  4872 raft_consensus.cc:2749] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:00:18.265846  2331 raft_consensus.cc:1240] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 4. Current term is 5. Ops: []
I20260709 13:00:18.259320  4549 raft_consensus.cc:2804] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:00:18.270875  1600 raft_consensus.cc:3060] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:00:18.247815  4547 raft_consensus.cc:493] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:17.925110  4900 consensus_queue.cc:1048] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.097s
I20260709 13:00:18.272868  2333 raft_consensus.cc:2468] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 6 LEARNER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 6.
I20260709 13:00:18.275324  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:00:18.275512  1571 connection.cc:256] RPC call timeout handler was delayed by -1.7805138399999692s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1849s for a response
I20260709 13:00:18.276916  2301 leader_election.cc:304] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:00:18.276554  1601 raft_consensus.cc:3060] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:00:18.229174  4931 consensus_queue.cc:1048] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.041s
I20260709 13:00:18.302775  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:00:18.247664  4884 consensus_queue.cc:1048] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.042s
I20260709 13:00:18.290305  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:18.291224  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:18.309357  1600 raft_consensus.cc:2468] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 8.
I20260709 13:00:18.310165  2334 raft_consensus.cc:2393] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 5: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:00:18.291410  1602 raft_consensus.cc:2468] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 5.
I20260709 13:00:18.265152  4759 raft_consensus.cc:2749] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260709 13:00:18.296334  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 56929.3us
I20260709 13:00:18.296609  2301 leader_election.cc:304] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:18.312976  2301 leader_election.cc:304] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:18.283226  2298 leader_election.cc:304] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:00:18.244098  4598 raft_consensus.cc:515] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:18.286621  4549 raft_consensus.cc:493] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:00:18.312299  1569 leader_election.cc:304] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:00:18.293699  4547 raft_consensus.cc:3060] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:00:18.335549  4959 raft_consensus.cc:2804] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:00:18.337073  4598 leader_election.cc:290] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:00:18.338037  1571 connection.cc:256] RPC call timeout handler was delayed by -1.6644138400000088s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1978s for a response
I20260709 13:00:18.286834  4948 consensus_queue.cc:1048] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.047s
I20260709 13:00:18.348963  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:00:18.377192  1571 connection.cc:256] RPC call timeout handler was delayed by -1.3271125790000156s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.164s for a response
W20260709 13:00:18.377637  1571 connection.cc:256] RPC call timeout handler was delayed by -1.3082125790000418s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16610000000000003s for a response
I20260709 13:00:18.377674  1748 raft_consensus.cc:2468] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 7.
W20260709 13:00:18.377952  1571 connection.cc:256] RPC call timeout handler was delayed by -1.25961257900002s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1715s for a response
W20260709 13:00:18.315124  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 320841us
I20260709 13:00:18.293457  4749 leader_election.cc:290] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:18.353139  2332 raft_consensus.cc:2468] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 7.
W20260709 13:00:18.365171  1753 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55840 (request call id 2223) took 1244 ms (client timeout 1597 ms). Trace:
W20260709 13:00:18.383991  1753 rpcz_store.cc:269] 0709 13:00:17.120328 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:17.151101 (+ 30773us) service_pool.cc:224] Handling call
0709 13:00:18.364861 (+1213760us) spinlock_profiling.cc:243] Waited 273 ms on lock 0x7b5400384d54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:00:18.365081 (+   220us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":628624128}
I20260709 13:00:18.307751  4512 raft_consensus.cc:2804] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:00:18.385350  4512 raft_consensus.cc:493] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:00:18.385938  4512 raft_consensus.cc:3060] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:00:18.394300  2299 leader_election.cc:304] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:18.282083  4779 raft_consensus.cc:515] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:18.400326  1754 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55848 (request call id 2028) took 3014 ms (client timeout 1521 ms). Trace:
W20260709 13:00:18.407578  1754 rpcz_store.cc:269] 0709 13:00:15.351809 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:15.351977 (+   168us) service_pool.cc:224] Handling call
0709 13:00:18.365783 (+3013806us) spinlock_profiling.cc:243] Waited 673 ms on lock 0x7b54000bd954. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:00:18.365930 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1547489536}
W20260709 13:00:18.411257  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 98376.5us
I20260709 13:00:18.413319  1644 leader_election.cc:304] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:00:18.351341  4549 raft_consensus.cc:3060] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:00:18.449388  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:18.450117  1602 raft_consensus.cc:1275] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 13:00:18.417799  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:18.451164  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:18.451552  1753 raft_consensus.cc:2468] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 5.
I20260709 13:00:18.451809  1602 raft_consensus.cc:2468] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 6.
I20260709 13:00:18.420061  1601 raft_consensus.cc:2468] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 6.
I20260709 13:00:18.430099  1747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55840 (request call id 2234) took 1232 ms. Trace:
I20260709 13:00:18.453136  1747 rpcz_store.cc:276] 0709 13:00:17.198126 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:17.722281 (+524155us) service_pool.cc:224] Handling call
0709 13:00:18.429906 (+707625us) spinlock_profiling.cc:243] Waited 176 ms on lock 0x7b540031f454. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:00:18.430030 (+   124us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":404742912}
I20260709 13:00:18.431502  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:18.432545  1751 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55840 (request call id 2232) took 1236 ms. Trace:
W20260709 13:00:18.435804  1750 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55848 (request call id 2049) took 2272 ms (client timeout 2020 ms). Trace:
W20260709 13:00:18.456442  1750 rpcz_store.cc:269] 0709 13:00:16.188293 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:16.188437 (+   144us) service_pool.cc:224] Handling call
0709 13:00:18.435601 (+2247164us) spinlock_profiling.cc:243] Waited 1.13 s on lock 0x7b54001d42d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:00:18.435719 (+   118us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2597899136}
W20260709 13:00:18.444406  1646 leader_election.cc:336] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Timed out: RequestConsensusVote RPC to 127.0.107.195:40693 timed out after 1.961s (SENT)
I20260709 13:00:18.447011  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:18.447196  4955 raft_consensus.cc:493] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:00:18.447711  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 100276us
I20260709 13:00:18.447992  4722 raft_consensus.cc:515] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:18.448768  1600 raft_consensus.cc:1240] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 4. Current term is 5. Ops: []
I20260709 13:00:18.418243  1644 leader_election.cc:304] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
W20260709 13:00:18.461130  1646 leader_election.cc:336] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Timed out: RequestConsensusVote RPC to 127.0.107.195:40693 timed out after 2.007s (SENT)
I20260709 13:00:18.486765  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:18.479765  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:18.488152  1748 raft_consensus.cc:2468] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 7.
I20260709 13:00:18.488351  4857 raft_consensus.cc:2804] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:00:18.488894  4857 raft_consensus.cc:697] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 7 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:00:18.489619  4779 leader_election.cc:290] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:18.489882  4857 consensus_queue.cc:237] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:18.491046  4683 leader_election.cc:290] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:00:18.483414  4944 log.cc:927] Time spent T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.167s	user 0.001s	sys 0.000s
I20260709 13:00:18.498600  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
W20260709 13:00:18.499166  1646 leader_election.cc:336] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Timed out: RequestConsensusVote RPC to 127.0.107.195:40693 timed out after 2.008s (SENT)
I20260709 13:00:18.499897  1646 leader_election.cc:304] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:00:18.406693  4956 raft_consensus.cc:2804] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:00:18.482905  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:18.452692  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:18.483654  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:18.484082  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:18.484158  1569 leader_election.cc:304] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
W20260709 13:00:18.485062  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 66890us
I20260709 13:00:18.461210  1599 raft_consensus.cc:1275] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260709 13:00:18.529116  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:18.529852  1599 raft_consensus.cc:2468] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 6.
I20260709 13:00:18.455700  1751 rpcz_store.cc:276] 0709 13:00:17.195925 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:17.867025 (+671100us) service_pool.cc:224] Handling call
0709 13:00:17.929258 (+ 62233us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:18.024332 (+ 95074us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:18.432336 (+408004us) spinlock_profiling.cc:243] Waited 61.9 ms on lock 0x7b54002f49d0. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661618ad 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:00:18.432474 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":142348672}
I20260709 13:00:18.397737  4951 raft_consensus.cc:2804] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:00:18.397509  4884 raft_consensus.cc:2804] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260709 13:00:18.503849  1646 leader_election.cc:336] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Timed out: RequestConsensusVote RPC to 127.0.107.195:40693 timed out after 2.130s (SENT)
I20260709 13:00:18.439855  4897 raft_consensus.cc:2804] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:00:18.414249  4580 raft_consensus.cc:2804] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:00:18.454361  4629 consensus_queue.cc:1059] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.079s
W20260709 13:00:18.500084  4537 consensus_meta.cc:287] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.605s	user 0.009s	sys 0.014s
I20260709 13:00:18.537397  1646 leader_election.cc:304] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
W20260709 13:00:18.537685  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 254612us
W20260709 13:00:18.537418  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 53334.6us
I20260709 13:00:18.540750  1717 leader_election.cc:304] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:00:18.489275  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:18.449856  4893 leader_election.cc:290] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 6 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:18.487668  4723 raft_consensus.cc:493] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:18.478259  4959 raft_consensus.cc:697] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:00:18.550164  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:18.536898  4951 raft_consensus.cc:493] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:18.524931  4922 raft_consensus.cc:3060] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:00:18.558315  1601 raft_consensus.cc:1275] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
W20260709 13:00:18.574499  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 153316us
I20260709 13:00:18.648591  1674 raft_consensus.cc:1275] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260709 13:00:18.648931  2299 leader_election.cc:304] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:18.542837  4884 raft_consensus.cc:493] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:18.650259  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:18.567533  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:18.574582  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:18.652675  1569 leader_election.cc:304] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:00:18.654812  1748 raft_consensus.cc:2468] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 4.
W20260709 13:00:18.655978  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 115323us
I20260709 13:00:18.657173  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:18.657954  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:18.661851  1753 raft_consensus.cc:3060] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:00:18.662707  1674 raft_consensus.cc:1275] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260709 13:00:18.663244  1717 leader_election.cc:304] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:00:18.665652  1717 leader_election.cc:304] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 13:00:18.669189  1569 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:18.528466  4956 raft_consensus.cc:493] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:18.709770  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:18.554728  4580 raft_consensus.cc:493] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:18.543470  4897 raft_consensus.cc:493] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
W20260709 13:00:18.718385  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 216883us
I20260709 13:00:18.495628  4955 raft_consensus.cc:515] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:00:18.693758  1571 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.107.195:40693 timed out after 1.528s (SENT) [suppressed 64 similar messages]
W20260709 13:00:18.697031  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 45704.7us
I20260709 13:00:18.725518  2298 leader_election.cc:304] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:00:18.714937  4956 raft_consensus.cc:3060] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Advancing to term 7
W20260709 13:00:18.706117  1569 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:00:18.727607  1569 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:18.728299  1569 leader_election.cc:304] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 13:00:18.701810  4598 raft_consensus.cc:2804] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:00:18.730645  4954 consensus_queue.cc:1048] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:00:18.730986  4598 raft_consensus.cc:493] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:18.731577  4598 raft_consensus.cc:3060] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:00:18.701836  4690 raft_consensus.cc:2749] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260709 13:00:18.733537  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 82139.3us
I20260709 13:00:18.710718  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:18.670456  4628 raft_consensus.cc:3055] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:00:18.558784  4948 raft_consensus.cc:2749] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:00:18.673451  4884 raft_consensus.cc:3060] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:00:18.720175  1646 leader_election.cc:304] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
W20260709 13:00:18.748663  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 54933.3us
I20260709 13:00:18.709928  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:18.697482  1717 leader_election.cc:304] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 13:00:18.756183  1571 leader_election.cc:336] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Timed out: RequestConsensusVote RPC to 127.0.107.195:40693 timed out after 1.640s (SENT)
W20260709 13:00:18.757040  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 59701.5us
I20260709 13:00:18.757798  1717 leader_election.cc:304] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 13:00:18.758651  1571 leader_election.cc:336] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Timed out: RequestConsensusVote RPC to 127.0.107.195:40693 timed out after 1.648s (SENT)
I20260709 13:00:18.758980  1717 leader_election.cc:304] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:18.759251  1571 leader_election.cc:304] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:00:18.756215  1754 raft_consensus.cc:2376] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 8. Re-sending same reply.
W20260709 13:00:18.760975  1571 leader_election.cc:336] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Timed out: RequestConsensusVote RPC to 127.0.107.195:40693 timed out after 1.661s (SENT)
I20260709 13:00:18.761615  1717 leader_election.cc:304] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:18.699206  4683 raft_consensus.cc:2804] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260709 13:00:18.720723  4731 consensus_meta.cc:287] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.894s	user 0.006s	sys 0.007s
I20260709 13:00:18.653745  4951 raft_consensus.cc:3060] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:00:18.715271  4580 raft_consensus.cc:3060] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:00:18.720886  4897 raft_consensus.cc:3060] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Advancing to term 7
W20260709 13:00:18.810889  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 48697.3us
W20260709 13:00:19.001791  1571 leader_election.cc:336] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Timed out: RequestConsensusVote RPC to 127.0.107.195:40693 timed out after 1.715s (SENT)
I20260709 13:00:18.770026  4683 raft_consensus.cc:493] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:18.670454  4723 raft_consensus.cc:515] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:18.657320  4537 raft_consensus.cc:515] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:18.755064  4628 raft_consensus.cc:740] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:00:18.789106  4574 leader_election.cc:290] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:18.917488  4950 raft_consensus.cc:2749] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:00:18.811326  4931 raft_consensus.cc:2749] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:00:18.720881  4849 raft_consensus.cc:2749] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:00:18.748207  4759 raft_consensus.cc:2804] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:00:18.699141  4953 consensus_queue.cc:1048] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.140s
W20260709 13:00:19.033104  4962 log.cc:927] Time spent T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.270s	user 0.000s	sys 0.001s
I20260709 13:00:18.709542  4779 raft_consensus.cc:2804] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:00:18.946601  4959 consensus_queue.cc:237] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:18.733173  4872 raft_consensus.cc:2804] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:00:18.994752  1597 raft_consensus.cc:1275] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260709 13:00:18.984194  4893 raft_consensus.cc:2749] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 13:00:18.856383  4857 consensus_queue.cc:1048] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.096s
I20260709 13:00:19.054220  4574 raft_consensus.cc:493] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:19.030464  1748 raft_consensus.cc:2410] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for term 7 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
W20260709 13:00:18.986478  4957 log.cc:927] Time spent T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.427s	user 0.001s	sys 0.009s
W20260709 13:00:19.058580  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 56833.6us
W20260709 13:00:19.004266  4904 consensus_meta.cc:287] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.892s	user 0.014s	sys 0.007s
I20260709 13:00:18.995105  4945 raft_consensus.cc:2749] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:00:19.061606  1603 raft_consensus.cc:1240] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 7. Current term is 8. Ops: []
I20260709 13:00:18.985761  1717 leader_election.cc:304] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:00:19.059247  4722 leader_election.cc:290] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 9 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:19.010138  4690 raft_consensus.cc:2749] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260709 13:00:18.992628  2335 consensus_meta.cc:287] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.648s	user 0.010s	sys 0.007s
W20260709 13:00:19.069640  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 349424us
W20260709 13:00:19.072595  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 334951us
I20260709 13:00:19.048946  4683 raft_consensus.cc:3060] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:00:19.068959  4779 raft_consensus.cc:493] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
W20260709 13:00:19.062168  4923 log.cc:927] Time spent T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.059s	user 0.000s	sys 0.000s
I20260709 13:00:19.058326  4672 raft_consensus.cc:2804] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:00:19.027402  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:00:19.084276  1571 leader_election.cc:336] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Timed out: RequestConsensusVote RPC to 127.0.107.195:40693 timed out after 1.814s (SENT)
W20260709 13:00:19.085306  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 99630us
I20260709 13:00:18.984046  4949 raft_consensus.cc:2749] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:00:19.087638  2335 raft_consensus.cc:2468] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 6.
I20260709 13:00:19.088074  1646 leader_election.cc:304] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1
W20260709 13:00:19.075659  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 352244us
I20260709 13:00:19.076071  4872 raft_consensus.cc:493] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
W20260709 13:00:19.103772  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 378269us
W20260709 13:00:19.147908  1569 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:19.177567  1717 leader_election.cc:304] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:00:19.178113  2298 leader_election.cc:304] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:00:19.176384  1754 raft_consensus.cc:2393] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 8: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:00:19.117043  1643 leader_election.cc:304] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:19.179513  2298 leader_election.cc:304] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:00:19.181119  1754 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55848 (request call id 2057) took 1029 ms. Trace:
W20260709 13:00:19.181771  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 93765.7us
I20260709 13:00:19.183383  1646 leader_election.cc:304] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 13:00:19.099023  4849 raft_consensus.cc:2804] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:00:19.181627  1754 rpcz_store.cc:276] 0709 13:00:18.151395 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:18.792139 (+640744us) service_pool.cc:224] Handling call
0709 13:00:19.181022 (+388883us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:00:19.088915  4931 raft_consensus.cc:2764] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260709 13:00:19.100692  4779 raft_consensus.cc:3060] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:00:19.190300  1600 raft_consensus.cc:1240] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 LEADER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 7. Current term is 8. Ops: []
I20260709 13:00:19.100574  4904 raft_consensus.cc:493] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:19.097869  4893 raft_consensus.cc:493] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:19.189880  4849 raft_consensus.cc:493] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:19.176085  4875 raft_consensus.cc:493] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:19.179323  4872 raft_consensus.cc:3060] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:00:19.183995  4722 raft_consensus.cc:2804] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 13:00:19.189695  4690 raft_consensus.cc:2804] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 13:00:19.086616  4574 raft_consensus.cc:515] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:00:19.203620  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 119362us
W20260709 13:00:19.221601  1571 leader_election.cc:336] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Timed out: RequestConsensusVote RPC to 127.0.107.195:40693 timed out after 1.965s (SENT)
I20260709 13:00:19.222134  1571 leader_election.cc:304] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:00:19.208595  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:00:19.223497  1569 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:19.197532  4967 raft_consensus.cc:2804] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 13:00:19.196321  4672 raft_consensus.cc:697] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 5 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:00:19.229784  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
W20260709 13:00:19.230754  1571 leader_election.cc:336] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Timed out: RequestConsensusVote RPC to 127.0.107.195:40693 timed out after 2.132s (SENT)
I20260709 13:00:19.239259  2328 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39726 (request call id 861) took 2622 ms. Trace:
I20260709 13:00:19.239367  4987 consensus_queue.cc:1059] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 13:00:19.239846  2328 rpcz_store.cc:276] 0709 13:00:16.616309 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:17.121924 (+505615us) service_pool.cc:224] Handling call
0709 13:00:17.791995 (+670071us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:18.119190 (+327195us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:19.237324 (+1118134us) spinlock_profiling.cc:243] Waited 326 ms on lock 0x7b5400627ad4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:00:19.238991 (+  1667us) spinlock_profiling.cc:243] Waited 645 ms on lock 0x7b5400627ad0. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661618ad 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:00:19.239158 (+   167us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2233221632}
I20260709 13:00:19.240428  4987 raft_consensus.cc:2804] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:00:19.240857  4987 raft_consensus.cc:493] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:19.241221  4987 raft_consensus.cc:3060] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:00:19.242005  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:19.247805  1751 raft_consensus.cc:2393] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 6: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:00:19.248312  1751 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55848 (request call id 2054) took 1107 ms. Trace:
I20260709 13:00:19.248643  1751 rpcz_store.cc:276] 0709 13:00:18.140573 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:19.241716 (+1101143us) service_pool.cc:224] Handling call
0709 13:00:19.248214 (+  6498us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:00:19.249678  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:19.242679  4857 raft_consensus.cc:2804] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:00:19.250471  4857 raft_consensus.cc:493] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:19.250898  4857 raft_consensus.cc:3060] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Advancing to term 8
W20260709 13:00:19.243050  4975 log.cc:927] Time spent T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.168s	user 0.000s	sys 0.000s
I20260709 13:00:19.243348  1603 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 594) took 1064 ms. Trace:
I20260709 13:00:19.201814  4982 raft_consensus.cc:493] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:19.245704  1748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55848 (request call id 2056) took 1094 ms. Trace:
I20260709 13:00:19.271466  1748 rpcz_store.cc:276] 0709 13:00:18.151076 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:18.660954 (+509878us) service_pool.cc:224] Handling call
0709 13:00:19.245483 (+584529us) spinlock_profiling.cc:243] Waited 319 ms on lock 0x7b54001711d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:00:19.245637 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":733892864}
I20260709 13:00:19.272466  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:19.272745  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:00:19.243403  4969 log.cc:927] Time spent T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.235s	user 0.000s	sys 0.001s
W20260709 13:00:19.234824  1753 consensus_meta.cc:287] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.572s	user 0.009s	sys 0.004s
I20260709 13:00:19.203294  4849 raft_consensus.cc:3060] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:00:19.279265  1753 raft_consensus.cc:2468] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 8.
I20260709 13:00:19.280423  1753 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43156 (request call id 575) took 1045 ms. Trace:
I20260709 13:00:19.280946  1753 rpcz_store.cc:276] 0709 13:00:18.234895 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:18.542992 (+308097us) service_pool.cc:224] Handling call
0709 13:00:19.280326 (+737334us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:00:19.260573  1754 raft_consensus.cc:2468] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 6.
I20260709 13:00:19.262931  1603 rpcz_store.cc:276] 0709 13:00:18.179324 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:18.179471 (+   147us) service_pool.cc:224] Handling call
0709 13:00:18.232325 (+ 52854us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:19.084410 (+852085us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:19.243073 (+158663us) spinlock_profiling.cc:243] Waited 829 ms on lock 0x7b54002a1cd4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:00:19.243277 (+   204us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1906372480}
I20260709 13:00:19.283555  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:19.285187  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:19.244411  4981 raft_consensus.cc:2749] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:00:19.287770  1748 raft_consensus.cc:2393] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 4: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:00:19.290241  1748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55840 (request call id 2251) took 1098 ms. Trace:
I20260709 13:00:19.216279  4966 raft_consensus.cc:493] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:00:19.290711  1748 rpcz_store.cc:276] 0709 13:00:18.191754 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:19.272492 (+1080738us) service_pool.cc:224] Handling call
0709 13:00:19.290162 (+ 17670us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:00:19.296128  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:19.296851  1748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55848 (request call id 2074) took 1124 ms. Trace:
I20260709 13:00:19.297191  1748 rpcz_store.cc:276] 0709 13:00:18.172343 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:19.295878 (+1123535us) service_pool.cc:224] Handling call
0709 13:00:19.296797 (+   919us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:00:19.298915  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:19.232945  4648 raft_consensus.cc:2749] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:00:19.298170  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:19.299923  1753 raft_consensus.cc:2468] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 9.
I20260709 13:00:19.253803  4690 raft_consensus.cc:697] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 9 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
W20260709 13:00:19.291509  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 60752.4us
W20260709 13:00:19.311273  1571 leader_election.cc:336] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Timed out: RequestConsensusVote RPC to 127.0.107.195:40693 timed out after 2.189s (SENT)
I20260709 13:00:19.202917  4904 raft_consensus.cc:515] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:19.271971  4547 consensus_meta.cc:287] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.874s	user 0.009s	sys 0.005s
I20260709 13:00:19.306398  1748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55848 (request call id 2075) took 1133 ms. Trace:
I20260709 13:00:19.312510  1748 rpcz_store.cc:276] 0709 13:00:18.172587 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:19.297906 (+1125319us) service_pool.cc:224] Handling call
0709 13:00:19.306341 (+  8435us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:00:19.313520  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:19.314036  1748 raft_consensus.cc:2376] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 9. Re-sending same reply.
I20260709 13:00:19.314509  1748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55848 (request call id 2063) took 1161 ms. Trace:
I20260709 13:00:19.293659  1603 pending_rounds.cc:85] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a: Aborting all ops after (but not including) 7
I20260709 13:00:19.314821  1748 rpcz_store.cc:276] 0709 13:00:18.153245 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:19.313278 (+1160033us) service_pool.cc:224] Handling call
0709 13:00:19.314430 (+  1152us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:00:19.315171  1603 pending_rounds.cc:107] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a: Aborting uncommitted NO_OP operation due to leader change: 6.8
I20260709 13:00:19.315459  1603 raft_consensus.cc:2889] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:00:19.306770  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:19.306372  1753 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55840 (request call id 2249) took 1115 ms. Trace:
I20260709 13:00:19.307143  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:19.316249  1753 rpcz_store.cc:276] 0709 13:00:18.191116 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:19.282124 (+1091008us) service_pool.cc:224] Handling call
0709 13:00:19.306308 (+ 24184us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:00:19.203511  4893 raft_consensus.cc:515] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:19.207736  4875 raft_consensus.cc:515] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:19.261732  4967 raft_consensus.cc:697] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 6 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
W20260709 13:00:19.292541  4896 consensus_meta.cc:287] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 1.178s	user 0.011s	sys 0.006s
W20260709 13:00:19.300841  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 117763us
I20260709 13:00:19.206734  4952 consensus_queue.cc:1048] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.086s
W20260709 13:00:19.322610  1569 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:19.324240  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:19.290462  4980 raft_consensus.cc:493] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:19.327883  1646 leader_election.cc:304] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:00:19.308459  4537 leader_election.cc:290] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 8 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:00:19.335071  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 218133us
I20260709 13:00:19.327345  4723 leader_election.cc:290] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:19.331506  4896 raft_consensus.cc:2749] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: Vote denied by peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261) with higher term. Message: Invalid argument: T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 6. Current term is 7.
I20260709 13:00:19.366953  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:19.307641  4966 raft_consensus.cc:515] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:19.409271  4920 leader_election.cc:290] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:19.411175  4955 leader_election.cc:290] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:00:19.416599  4979 log.cc:927] Time spent T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.180s	user 0.001s	sys 0.000s
W20260709 13:00:19.419142  4977 log.cc:927] Time spent T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.148s	user 0.001s	sys 0.000s
W20260709 13:00:19.422236  4992 log.cc:927] Time spent T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.178s	user 0.000s	sys 0.001s
I20260709 13:00:19.367627  4580 raft_consensus.cc:515] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:19.334906  4722 raft_consensus.cc:697] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 8 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:00:19.367878  4967 consensus_queue.cc:237] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 5.8, Last appended by leader: 8, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:19.368717  4947 log.cc:927] Time spent T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 1.135s	user 0.001s	sys 0.005s
I20260709 13:00:19.369267  4790 raft_consensus.cc:493] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:19.369433  4893 leader_election.cc:290] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:19.371869  1752 raft_consensus.cc:2468] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 5.
W20260709 13:00:19.454881  1752 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55840 (request call id 2242) took 2208 ms (client timeout 2217 ms). Trace:
I20260709 13:00:19.379294  4997 raft_consensus.cc:2804] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260709 13:00:19.455319  1752 rpcz_store.cc:269] 0709 13:00:17.247046 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:19.366582 (+2119536us) service_pool.cc:224] Handling call
0709 13:00:19.454805 (+ 88223us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:00:19.463971  4997 raft_consensus.cc:493] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:19.464452  4997 raft_consensus.cc:3060] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:00:19.464564  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:19.337949  2330 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39726 (request call id 860) took 2718 ms. Trace:
I20260709 13:00:19.383693  4759 raft_consensus.cc:493] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:00:19.349541  4672 consensus_queue.cc:237] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:19.361074  4980 raft_consensus.cc:515] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:19.467453  1752 raft_consensus.cc:2410] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for term 7 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20260709 13:00:19.467932  1752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55848 (request call id 2086) took 1008 ms. Trace:
I20260709 13:00:19.468057  2337 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49082 (request call id 716) took 1209 ms. Trace:
I20260709 13:00:19.468292  1752 rpcz_store.cc:276] 0709 13:00:18.460024 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:19.464264 (+1004240us) service_pool.cc:224] Handling call
0709 13:00:19.467875 (+  3611us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:00:19.469437  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:19.468441  2337 rpcz_store.cc:276] 0709 13:00:18.258728 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:18.306694 (+ 47966us) service_pool.cc:224] Handling call
0709 13:00:18.307035 (+   341us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:18.479349 (+172314us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 13:00:18.565445 (+ 86096us) log.cc:844] Serialized 14 byte log entry
0709 13:00:18.573180 (+  7735us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:00:18.573263 (+    83us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:00:18.573336 (+    73us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:19.467807 (+894471us) spinlock_profiling.cc:243] Waited 172 ms on lock 0x7b540084fb54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:00:19.467862 (+    55us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:19.467993 (+   131us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8057,"prepare.queue_time_us":671545,"prepare.run_cpu_time_us":61,"prepare.run_wall_time_us":63,"raft.queue_time_us":117,"raft.run_cpu_time_us":151,"raft.run_wall_time_us":152,"spinlock_wait_cycles":395615488,"thread_start_us":46000,"threads_started":2,"wal-append.queue_time_us":445685}
I20260709 13:00:19.471791  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:19.472420  1747 raft_consensus.cc:2468] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 5.
I20260709 13:00:19.472579  2330 rpcz_store.cc:276] 0709 13:00:16.616016 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:17.119811 (+503795us) service_pool.cc:224] Handling call
0709 13:00:17.210608 (+ 90797us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:17.329469 (+118861us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:19.327578 (+1998109us) spinlock_profiling.cc:243] Waited 87.6 ms on lock 0x7b540083f750. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661618ad 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:00:19.334194 (+  6616us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":201407872}
I20260709 13:00:19.473415  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:19.474148  1747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55848 (request call id 2064) took 1320 ms. Trace:
I20260709 13:00:19.391178  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:19.395682  2331 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49082 (request call id 721) took 1122 ms. Trace:
W20260709 13:00:19.396042  4976 log.cc:927] Time spent T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.207s	user 0.000s	sys 0.001s
I20260709 13:00:19.396684  4993 consensus_queue.cc:1048] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.027s
I20260709 13:00:19.290241  4982 raft_consensus.cc:515] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:19.399413  1604 raft_consensus.cc:2468] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 7.
W20260709 13:00:19.400110  4971 log.cc:927] Time spent T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.199s	user 0.001s	sys 0.000s
I20260709 13:00:19.397867  4690 consensus_queue.cc:237] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:19.445966  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:19.509938  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:19.454545  4790 raft_consensus.cc:515] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:19.328395  4547 raft_consensus.cc:515] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:19.474807  4759 raft_consensus.cc:3060] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:00:19.387293  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:19.465543  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:19.477401  1747 rpcz_store.cc:276] 0709 13:00:18.153531 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:19.473173 (+1319642us) service_pool.cc:224] Handling call
0709 13:00:19.474087 (+   914us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:00:19.478044  1597 raft_consensus.cc:3060] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:00:19.478258  2331 rpcz_store.cc:276] 0709 13:00:18.273250 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:18.316467 (+ 43217us) service_pool.cc:224] Handling call
0709 13:00:18.348860 (+ 32393us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:00:18.376358 (+ 27498us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:00:18.376391 (+    33us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:00:18.402365 (+ 25974us) log.cc:844] Serialized 52 byte log entry
0709 13:00:18.542305 (+139940us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:00:18.542350 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:18.542423 (+    73us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:00:19.395333 (+852910us) raft_consensus.cc:1685] finished
0709 13:00:19.395367 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:19.395576 (+   209us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":26987,"raft.queue_time_us":59481,"raft.run_cpu_time_us":134,"raft.run_wall_time_us":30238,"thread_start_us":63727,"threads_started":1,"wal-append.queue_time_us":247582}
W20260709 13:00:19.497699  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 153985us
I20260709 13:00:19.523164  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:19.523320  1646 leader_election.cc:304] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:00:19.524061  1747 raft_consensus.cc:2468] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 7.
I20260709 13:00:19.524750  1747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55848 (request call id 2088) took 1064 ms. Trace:
I20260709 13:00:19.525166  1747 rpcz_store.cc:276] 0709 13:00:18.460641 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:19.518337 (+1057696us) service_pool.cc:224] Handling call
0709 13:00:19.524670 (+  6333us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:00:19.527050  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:19.528137  1747 raft_consensus.cc:2410] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for term 7 because replica has last-logged OpId of term: 5 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 7.
I20260709 13:00:19.528630  1747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55840 (request call id 2250) took 1337 ms. Trace:
I20260709 13:00:19.529026  1747 rpcz_store.cc:276] 0709 13:00:18.191376 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:19.526724 (+1335348us) service_pool.cc:224] Handling call
0709 13:00:19.528556 (+  1832us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:00:19.449945  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:19.511247  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:19.413995  4628 consensus_queue.cc:260] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:19.511459  1603 raft_consensus.cc:2468] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 6.
I20260709 13:00:19.531214  1599 raft_consensus.cc:2410] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260709 13:00:19.531898  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:19.533951  1752 raft_consensus.cc:2393] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 5: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:00:19.511236  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:19.537175  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:19.541256  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:19.565385  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:19.566118  1752 raft_consensus.cc:2468] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 6.
I20260709 13:00:19.566615  1752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55840 (request call id 2255) took 1024 ms. Trace:
I20260709 13:00:19.566895  1752 rpcz_store.cc:276] 0709 13:00:18.542372 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:19.565132 (+1022760us) service_pool.cc:224] Handling call
0709 13:00:19.566555 (+  1423us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:00:19.567799  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:19.512549  4847 leader_election.cc:290] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:19.576546  1752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55840 (request call id 2252) took 1035 ms. Trace:
I20260709 13:00:19.577085  1752 rpcz_store.cc:276] 0709 13:00:18.541584 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:19.567543 (+1025959us) service_pool.cc:224] Handling call
0709 13:00:19.576480 (+  8937us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:00:19.578004  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 64095.2us
I20260709 13:00:19.565385  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:00:19.578455  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 55219.5us
I20260709 13:00:19.580241  1646 leader_election.cc:304] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 13:00:19.563946  1747 raft_consensus.cc:2468] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 9.
I20260709 13:00:19.582897  1747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55848 (request call id 2087) took 1122 ms. Trace:
I20260709 13:00:19.583406  1747 rpcz_store.cc:276] 0709 13:00:18.460397 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:19.532421 (+1072024us) service_pool.cc:224] Handling call
0709 13:00:19.582808 (+ 50387us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:00:19.584753  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:00:19.599871  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 85912.8us
W20260709 13:00:19.603560  1746 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55840 (request call id 2237) took 2404 ms (client timeout 1715 ms). Trace:
I20260709 13:00:19.603780  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 10 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:19.604828  1752 raft_consensus.cc:2410] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 for term 10 because replica has last-logged OpId of term: 9 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 8.
I20260709 13:00:19.603885  1597 raft_consensus.cc:2468] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 7.
I20260709 13:00:19.603005  1747 raft_consensus.cc:2393] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 6: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:00:19.607961  1747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55840 (request call id 2256) took 1053 ms. Trace:
W20260709 13:00:19.604058  1746 rpcz_store.cc:269] 0709 13:00:17.199160 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:17.226840 (+ 27680us) service_pool.cc:224] Handling call
0709 13:00:19.582682 (+2355842us) spinlock_profiling.cc:243] Waited 609 ms on lock 0x7b5400322dd4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:00:19.603464 (+ 20782us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1399904128}
I20260709 13:00:19.610186  1752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43156 (request call id 570) took 1530 ms. Trace:
I20260709 13:00:19.610613  1752 rpcz_store.cc:276] 0709 13:00:18.080203 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:19.603468 (+1523265us) service_pool.cc:224] Handling call
0709 13:00:19.610120 (+  6652us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:00:19.608400  1747 rpcz_store.cc:276] 0709 13:00:18.554203 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:19.584374 (+1030171us) service_pool.cc:224] Handling call
0709 13:00:19.607871 (+ 23497us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:00:19.564647  4746 raft_consensus.cc:2749] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 13:00:19.617842  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:19.618477  1750 raft_consensus.cc:2393] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 8: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:00:19.623787  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:19.625139  1600 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 2033) took 1130 ms. Trace:
I20260709 13:00:19.625504  1600 rpcz_store.cc:276] 0709 13:00:18.495123 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:18.554347 (+ 59224us) service_pool.cc:224] Handling call
0709 13:00:18.554680 (+   333us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:19.193479 (+638799us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:19.624912 (+431433us) spinlock_profiling.cc:243] Waited 635 ms on lock 0x7b540029a9d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:00:19.625075 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1461003648}
I20260709 13:00:19.626472  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:19.626784  4993 raft_consensus.cc:2804] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:00:19.627177  4993 raft_consensus.cc:493] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:19.627599  4993 raft_consensus.cc:3060] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:00:19.627748  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:19.628392  1746 raft_consensus.cc:2376] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 6. Re-sending same reply.
I20260709 13:00:19.628880  1746 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55848 (request call id 2061) took 1476 ms. Trace:
I20260709 13:00:19.629194  1746 rpcz_store.cc:276] 0709 13:00:18.152606 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:19.620040 (+1467434us) service_pool.cc:224] Handling call
0709 13:00:19.628818 (+  8778us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:00:19.605844  4722 consensus_queue.cc:237] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:19.627293  1750 raft_consensus.cc:2393] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 5: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:00:19.634509  1750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55848 (request call id 2066) took 1480 ms. Trace:
I20260709 13:00:19.634596  1746 raft_consensus.cc:1275] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 13:00:19.634866  1750 rpcz_store.cc:276] 0709 13:00:18.154175 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:19.626171 (+1471996us) service_pool.cc:224] Handling call
0709 13:00:19.634416 (+  8245us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:00:19.644731  1746 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43156 (request call id 566) took 1567 ms. Trace:
I20260709 13:00:19.645251  1746 rpcz_store.cc:276] 0709 13:00:18.077668 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:19.629820 (+1552152us) service_pool.cc:224] Handling call
0709 13:00:19.630145 (+   325us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:19.635000 (+  4855us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:19.644632 (+  9632us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:00:19.647598  4995 raft_consensus.cc:493] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:19.648068  4988 raft_consensus.cc:493] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:19.648458  4995 raft_consensus.cc:515] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:19.651549  1750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43156 (request call id 569) took 1573 ms. Trace:
I20260709 13:00:19.651952  1750 rpcz_store.cc:276] 0709 13:00:18.079851 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:19.635947 (+1556096us) service_pool.cc:224] Handling call
0709 13:00:19.650821 (+ 14874us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:19.651189 (+   368us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 13:00:19.651220 (+    31us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:00:19.651266 (+    46us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:00:19.651306 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:19.651370 (+    64us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:19.651488 (+   118us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:00:19.653556  1750 raft_consensus.cc:1275] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 13:00:19.654134  1750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55848 (request call id 2059) took 1502 ms. Trace:
I20260709 13:00:19.654664  4998 consensus_queue.cc:1059] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.042s
I20260709 13:00:19.654947  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:19.655874  4998 raft_consensus.cc:3055] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:00:19.656226  2328 raft_consensus.cc:2410] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for term 9 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
I20260709 13:00:19.656292  4998 raft_consensus.cc:740] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:00:19.648741  4988 raft_consensus.cc:515] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:19.611071  4904 leader_election.cc:290] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:19.651342  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:19.662382  2299 leader_election.cc:304] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:19.666178  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:00:19.669322  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 89167us
I20260709 13:00:19.668198  1750 rpcz_store.cc:276] 0709 13:00:18.152006 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:19.652902 (+1500896us) service_pool.cc:224] Handling call
0709 13:00:19.653221 (+   319us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:19.653889 (+   668us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:19.654053 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:00:19.672070  1745 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55840 (request call id 2220) took 2552 ms (client timeout 2087 ms). Trace:
W20260709 13:00:19.676540  1745 rpcz_store.cc:269] 0709 13:00:17.119417 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:17.343528 (+224111us) service_pool.cc:224] Handling call
0709 13:00:19.671750 (+2328222us) spinlock_profiling.cc:243] Waited 259 ms on lock 0x7b540006fc54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:00:19.671955 (+   205us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":596737280}
I20260709 13:00:19.684206  1746 raft_consensus.cc:1275] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260709 13:00:19.685156  1746 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55848 (request call id 2058) took 1533 ms. Trace:
I20260709 13:00:19.685566  1746 rpcz_store.cc:276] 0709 13:00:18.151711 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:19.646394 (+1494683us) service_pool.cc:224] Handling call
0709 13:00:19.683732 (+ 37338us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:19.684806 (+  1074us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:19.685078 (+   272us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:00:19.691704  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 47033.1us
I20260709 13:00:19.711854  1750 raft_consensus.cc:1240] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 6. Current term is 7. Ops: []
I20260709 13:00:19.712831  1750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55848 (request call id 2070) took 1541 ms. Trace:
I20260709 13:00:19.713263  1750 rpcz_store.cc:276] 0709 13:00:18.171370 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:19.676232 (+1504862us) service_pool.cc:224] Handling call
0709 13:00:19.711553 (+ 35321us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:19.712489 (+   936us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:19.712747 (+   258us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:00:19.717371  1646 leader_election.cc:336] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Timed out: RequestConsensusVote RPC to 127.0.107.195:40693 timed out after 2.425s (SENT)
I20260709 13:00:19.718070  1646 leader_election.cc:304] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:00:19.724164  1745 raft_consensus.cc:1275] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260709 13:00:19.724455  1646 leader_election.cc:304] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:19.725100  1745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55848 (request call id 2071) took 1553 ms. Trace:
I20260709 13:00:19.725596  1745 rpcz_store.cc:276] 0709 13:00:18.171616 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:19.684494 (+1512878us) service_pool.cc:224] Handling call
0709 13:00:19.719555 (+ 35061us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:19.724739 (+  5184us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:19.725017 (+   278us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:00:19.696997  4598 consensus_meta.cc:287] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.965s	user 0.005s	sys 0.018s
W20260709 13:00:19.726924  1644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:19.733975  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:19.688333  4966 leader_election.cc:290] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:19.688846  4900 raft_consensus.cc:2749] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 13:00:19.740550  2298 leader_election.cc:304] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:19.738107  4981 raft_consensus.cc:493] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:19.764667  4981 raft_consensus.cc:515] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:19.767910  4942 log.cc:927] Time spent T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 1.504s	user 0.001s	sys 0.002s
I20260709 13:00:19.771399  4995 leader_election.cc:290] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:19.772078  4995 raft_consensus.cc:2804] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:00:19.772423  4995 raft_consensus.cc:493] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:00:19.772779  4995 raft_consensus.cc:3060] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:00:19.775696  4988 leader_election.cc:290] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:00:19.776268  4512 consensus_meta.cc:287] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 1.390s	user 0.015s	sys 0.008s
I20260709 13:00:19.657507  4998 consensus_queue.cc:260] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:19.784657  1678 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52766 (request call id 2283) took 2256 ms. Trace:
I20260709 13:00:19.739009  5010 raft_consensus.cc:493] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:19.776862  4512 raft_consensus.cc:515] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:19.785264  1678 rpcz_store.cc:276] 0709 13:00:17.528725 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:17.651781 (+123056us) service_pool.cc:224] Handling call
0709 13:00:17.652165 (+   384us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:00:17.672332 (+ 20167us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:00:17.672363 (+    31us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:00:17.682651 (+ 10288us) log.cc:844] Serialized 52 byte log entry
0709 13:00:17.953520 (+270869us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:00:17.953560 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:17.953622 (+    62us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:00:19.690022 (+1736400us) spinlock_profiling.cc:243] Waited 9.5 ms on lock 0x7b4400022b4c. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f4262f9bc85 00007f4262f9c6a2 00007f4265d0e1db 00007f4265d09aaf 00007f4265cfd86e 00007f4265cfcbb6 00007f4265cfc7d8 00007f4265d06221 00007f4265d061d9 00007f4265d06169
0709 13:00:19.779108 (+ 89086us) raft_consensus.cc:1685] finished
0709 13:00:19.779138 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:19.784555 (+  5417us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":19641,"raft.queue_time_us":24472,"raft.run_cpu_time_us":103,"raft.run_wall_time_us":3663,"spinlock_wait_cycles":21853568,"thread_start_us":270441,"threads_started":1,"wal-append.queue_time_us":483910}
W20260709 13:00:19.775708  4905 consensus_meta.cc:287] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 1.542s	user 0.013s	sys 0.007s
I20260709 13:00:19.758422  4999 raft_consensus.cc:2749] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260709 13:00:19.803195  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 78839.7us
W20260709 13:00:19.785694  4549 consensus_meta.cc:287] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 1.305s	user 0.016s	sys 0.006s
I20260709 13:00:19.742815  4598 raft_consensus.cc:515] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:19.787138  4580 leader_election.cc:290] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 7 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:19.788991  4920 raft_consensus.cc:493] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:19.759271  5011 raft_consensus.cc:2749] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260709 13:00:19.794442  4951 consensus_meta.cc:287] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.801s	user 0.015s	sys 0.004s
W20260709 13:00:19.791368  4956 consensus_meta.cc:287] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 1.065s	user 0.021s	sys 0.003s
I20260709 13:00:19.803040  4985 raft_consensus.cc:493] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:19.790124  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:19.841692  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:19.851159  2332 raft_consensus.cc:1240] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 7. Current term is 8. Ops: []
W20260709 13:00:19.836043  4872 consensus_meta.cc:287] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.587s	user 0.011s	sys 0.010s
I20260709 13:00:19.754617  4601 consensus_queue.cc:1048] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.050s
I20260709 13:00:19.866842  4998 raft_consensus.cc:3060] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:00:19.806223  4905 raft_consensus.cc:515] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:00:19.873318  4922 consensus_meta.cc:287] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 1.204s	user 0.010s	sys 0.014s
W20260709 13:00:19.882673  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 99122.7us
I20260709 13:00:19.874652  4790 leader_election.cc:290] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:19.892753  1596 raft_consensus.cc:2468] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 4.
I20260709 13:00:19.838702  4549 raft_consensus.cc:515] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:19.880609  5005 log.cc:927] Time spent T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.123s	user 0.001s	sys 0.000s
I20260709 13:00:19.866669  4920 raft_consensus.cc:515] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } }
I20260709 13:00:19.806914  5010 raft_consensus.cc:515] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:19.870940  4956 raft_consensus.cc:515] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:19.911726  4897 consensus_meta.cc:287] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.876s	user 0.011s	sys 0.012s
W20260709 13:00:19.908047  4987 consensus_meta.cc:287] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.666s	user 0.011s	sys 0.004s
W20260709 13:00:19.913355  1604 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36462 (request call id 591) took 1759 ms (client timeout 1874 ms). Trace:
I20260709 13:00:19.918805  5007 ts_tablet_manager.cc:933] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1: Initiating tablet copy from peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:19.942423  4981 leader_election.cc:290] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:00:19.919776  5006 log.cc:927] Time spent T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.141s	user 0.001s	sys 0.000s
I20260709 13:00:19.924952  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:19.927699  4905 leader_election.cc:290] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 4 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:19.932262  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:19.935081  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:19.895301  4922 raft_consensus.cc:2749] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260709 13:00:19.952172  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 56158.8us
I20260709 13:00:19.956152  1753 raft_consensus.cc:2393] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 5: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
W20260709 13:00:19.960587  5014 log.cc:927] Time spent T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.053s	user 0.000s	sys 0.000s
I20260709 13:00:19.961535  5010 leader_election.cc:290] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:19.880650  4872 raft_consensus.cc:515] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:19.966384  1754 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55848 (request call id 2085) took 1783 ms (client timeout 2020 ms). Trace:
W20260709 13:00:19.967881  1646 leader_election.cc:336] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Timed out: RequestConsensusVote RPC to 127.0.107.195:40693 timed out after 2.905s (SENT)
I20260709 13:00:19.942688  4790 raft_consensus.cc:493] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:19.865569  4951 raft_consensus.cc:515] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:19.919756  2332 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39726 (request call id 874) took 1500 ms. Trace:
I20260709 13:00:19.903620  4959 consensus_queue.cc:1048] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.053s
I20260709 13:00:19.878232  4985 raft_consensus.cc:515] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:20.059250  1602 raft_consensus.cc:2410] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for term 8 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
W20260709 13:00:20.047168  5013 log.cc:927] Time spent T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.094s	user 0.000s	sys 0.000s
I20260709 13:00:20.060299  4722 raft_consensus.cc:493] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:00:20.061270  4722 raft_consensus.cc:515] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:20.062264  4872 leader_election.cc:290] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 8 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:00:20.061710  1754 rpcz_store.cc:269] 0709 13:00:18.182712 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:19.193082 (+1010370us) service_pool.cc:224] Handling call
0709 13:00:19.966179 (+773097us) spinlock_profiling.cc:243] Waited 36.9 ms on lock 0x7b54001d42d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:00:19.966292 (+   113us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":84822784}
I20260709 13:00:20.061710  1597 raft_consensus.cc:1240] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 7. Current term is 8. Ops: []
I20260709 13:00:20.064126  4722 leader_election.cc:290] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:20.065214  2332 rpcz_store.cc:276] 0709 13:00:18.419707 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:18.420433 (+   726us) service_pool.cc:224] Handling call
0709 13:00:18.420827 (+   394us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:19.872024 (+1451197us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:19.919472 (+ 47448us) spinlock_profiling.cc:243] Waited 1.43 s on lock 0x7b540065ed54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:00:19.919679 (+   207us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3288555392}
W20260709 13:00:20.070096  1596 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40166 (request call id 2023) took 1625 ms (client timeout 1876 ms). Trace:
W20260709 13:00:20.070509  1596 rpcz_store.cc:269] 0709 13:00:18.444682 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:18.449115 (+  4433us) service_pool.cc:224] Handling call
0709 13:00:20.069912 (+1620797us) spinlock_profiling.cc:243] Waited 1.44 s on lock 0x7b54003edfd4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:00:20.070030 (+   118us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3317290880}
I20260709 13:00:20.071424  1678 raft_consensus.cc:3060] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:00:20.072603  1679 raft_consensus.cc:3060] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:00:20.074609  4897 raft_consensus.cc:515] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:20.060281  4956 leader_election.cc:290] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 7 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:00:20.064723  4994 log.cc:927] Time spent T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.820s	user 0.004s	sys 0.000s
I20260709 13:00:19.991515  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:20.065843  1646 leader_election.cc:304] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:00:20.066994  1752 raft_consensus.cc:2468] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 8.
I20260709 13:00:20.083611  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:20.084540  1754 raft_consensus.cc:3060] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:00:20.084414  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:00:20.058085  4779 consensus_meta.cc:287] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.857s	user 0.008s	sys 0.012s
I20260709 13:00:20.085939  1752 raft_consensus.cc:2468] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 7.
W20260709 13:00:20.066846  1604 rpcz_store.cc:269] 0709 13:00:18.153636 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:18.154266 (+   630us) service_pool.cc:224] Handling call
0709 13:00:19.913158 (+1758892us) spinlock_profiling.cc:243] Waited 1.23 s on lock 0x7b540027a1d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:00:19.913285 (+   127us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2826972416}
I20260709 13:00:20.052487  1745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55848 (request call id 2078) took 1878 ms. Trace:
I20260709 13:00:20.031172  4598 leader_election.cc:290] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 10 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 13:00:20.095156  2335 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49082 (request call id 696) took 3555 ms (client timeout 1985 ms). Trace:
I20260709 13:00:20.101102  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:00:20.092986  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 125180us
W20260709 13:00:20.112622  1643 leader_election.cc:336] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 1.567s (SENT)
I20260709 13:00:20.104944  1571 leader_election.cc:304] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
W20260709 13:00:20.156615  1646 leader_election.cc:336] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Timed out: RequestConsensusVote RPC to 127.0.107.195:40693 timed out after 3.345s (SENT)
W20260709 13:00:20.142860  2335 rpcz_store.cc:269] 0709 13:00:16.539542 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:16.540415 (+   873us) service_pool.cc:224] Handling call
0709 13:00:20.094948 (+3554533us) spinlock_profiling.cc:243] Waited 1.78 s on lock 0x7b54005e3b54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:00:20.095066 (+   118us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4099053312}
W20260709 13:00:20.157541  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 52724.1us
I20260709 13:00:20.143430  1747 raft_consensus.cc:2468] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 5.
I20260709 13:00:20.101910  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:00:19.986289  4849 consensus_meta.cc:287] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.687s	user 0.015s	sys 0.006s
I20260709 13:00:20.159122  1571 leader_election.cc:304] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
W20260709 13:00:20.157048  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 44877.9us
I20260709 13:00:20.161523  1571 leader_election.cc:304] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:00:20.159221  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:20.157548  1643 leader_election.cc:304] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:00:20.087270  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:20.141209  1745 rpcz_store.cc:276] 0709 13:00:18.173591 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:19.730480 (+1556889us) service_pool.cc:224] Handling call
0709 13:00:19.730981 (+   501us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:00:19.746391 (+ 15410us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:00:19.746451 (+    60us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:00:19.755320 (+  8869us) log.cc:844] Serialized 26 byte log entry
0709 13:00:19.852007 (+ 96687us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:00:19.852079 (+    72us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:19.852186 (+   107us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:00:20.052232 (+200046us) raft_consensus.cc:1685] finished
0709 13:00:20.052254 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:20.052410 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":16251,"raft.queue_time_us":50785,"raft.run_cpu_time_us":167,"raft.run_wall_time_us":7907,"thread_start_us":90294,"threads_started":1,"wal-append.queue_time_us":228723}
I20260709 13:00:20.163244  1749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55848 (request call id 2065) took 2009 ms. Trace:
W20260709 13:00:20.165789  1646 leader_election.cc:336] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Timed out: RequestConsensusVote RPC to 127.0.107.195:40693 timed out after 3.498s (SENT)
I20260709 13:00:20.165758  1749 rpcz_store.cc:276] 0709 13:00:18.153854 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:19.534045 (+1380191us) service_pool.cc:224] Handling call
0709 13:00:20.101714 (+567669us) spinlock_profiling.cc:243] Waited 2 ms on lock 0x7b58001ffe98. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f426303db71 00007f426303da22 00007f426bf56c09 00007f426b0b0ee0 00007f426b17a0ab 00007f426b17a131 00007f426b1280a0 00007f426b127984 00007f42646a26d4 00007f42646a2662
0709 13:00:20.163175 (+ 61461us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4589952}
I20260709 13:00:20.163710  1752 raft_consensus.cc:2468] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 4.
I20260709 13:00:20.166539  1646 leader_election.cc:304] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:00:20.101133  1680 raft_consensus.cc:2410] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for term 9 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
W20260709 13:00:20.087244  4683 consensus_meta.cc:287] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.985s	user 0.016s	sys 0.007s
W20260709 13:00:20.169426  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 56864.8us
I20260709 13:00:19.969048  4601 consensus_queue.cc:1048] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.190s
W20260709 13:00:20.098778  5016 log.cc:927] Time spent T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.176s	user 0.002s	sys 0.000s
W20260709 13:00:20.099934  5021 log.cc:927] Time spent T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.071s	user 0.000s	sys 0.000s
I20260709 13:00:19.965451  4993 raft_consensus.cc:515] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:20.160413  4847 raft_consensus.cc:2749] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260709 13:00:20.174696  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 1.62715e+06us
I20260709 13:00:20.103017  4920 leader_election.cc:290] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:20.175844  1571 leader_election.cc:304] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 13:00:20.176903  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:20.175716  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:20.192123  1745 raft_consensus.cc:2468] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 8.
I20260709 13:00:20.192543  1752 raft_consensus.cc:2468] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 6.
I20260709 13:00:20.093289  5010 raft_consensus.cc:2749] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:00:19.994920  4759 raft_consensus.cc:515] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:20.165351  4872 raft_consensus.cc:2804] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:00:20.214752  5017 consensus_queue.cc:1059] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.006s
I20260709 13:00:20.220232  4872 raft_consensus.cc:493] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:20.220672  4872 raft_consensus.cc:3060] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:00:20.221834  4995 raft_consensus.cc:515] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:20.222865  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:20.223040  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:20.223800  2328 raft_consensus.cc:2410] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for term 6 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260709 13:00:20.192706  1571 leader_election.cc:304] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:00:20.224261  4995 leader_election.cc:290] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:20.224659  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:20.224931  4995 raft_consensus.cc:3055] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:00:20.225360  4995 raft_consensus.cc:740] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:00:20.225596  1571 leader_election.cc:304] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:00:20.225925  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:20.226643  4959 raft_consensus.cc:2749] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:00:20.226763  1674 raft_consensus.cc:1240] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 9. Current term is 10. Ops: []
I20260709 13:00:20.227387  4959 raft_consensus.cc:2749] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:00:20.227465  1754 raft_consensus.cc:2468] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 7.
I20260709 13:00:20.229206  4951 leader_election.cc:290] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 8 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:20.094017  4980 leader_election.cc:290] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:00:20.197085  4857 consensus_meta.cc:287] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.946s	user 0.015s	sys 0.008s
I20260709 13:00:20.236339  4857 raft_consensus.cc:515] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:20.169142  5011 consensus_queue.cc:1059] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.028s
I20260709 13:00:20.202814  4537 raft_consensus.cc:2804] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:00:20.238849  4537 raft_consensus.cc:493] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:20.239291  4537 raft_consensus.cc:3060] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:00:20.203347  1675 raft_consensus.cc:1240] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 7. Current term is 8. Ops: []
I20260709 13:00:20.174008  4580 consensus_queue.cc:1059] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.030s
I20260709 13:00:20.185097  4988 raft_consensus.cc:2749] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:00:20.097005  4875 leader_election.cc:290] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:20.215147  1678 raft_consensus.cc:2468] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 8.
I20260709 13:00:20.073414  4790 raft_consensus.cc:515] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:20.260471  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:20.209291  4922 raft_consensus.cc:493] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
W20260709 13:00:20.272759  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 107330us
I20260709 13:00:20.204551  4993 leader_election.cc:290] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 8 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:00:20.219288  1718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:20.222048  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:20.168889  4849 raft_consensus.cc:515] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:20.079183  4987 raft_consensus.cc:515] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:20.193689  1745 raft_consensus.cc:1240] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 9. Current term is 10. Ops: [7.5-9.6]
I20260709 13:00:20.235131  1571 leader_election.cc:304] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:00:20.078217  4897 leader_election.cc:290] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 7 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:20.103425  4905 consensus_queue.cc:1048] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.084s
I20260709 13:00:20.175262  4967 raft_consensus.cc:493] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:00:20.243026  1753 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55848 (request call id 2060) took 2090 ms (client timeout 2425 ms). Trace:
I20260709 13:00:20.287756  1745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55848 (request call id 2081) took 2106 ms. Trace:
I20260709 13:00:20.159634  4779 raft_consensus.cc:515] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:20.239317  5011 raft_consensus.cc:3055] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:00:20.245456  1599 raft_consensus.cc:1275] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260709 13:00:20.185819  4956 raft_consensus.cc:2749] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20260709 13:00:20.229139  4997 consensus_meta.cc:287] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.764s	user 0.026s	sys 0.007s
I20260709 13:00:20.248762  2301 leader_election.cc:304] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 13:00:20.249624  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 264205us
I20260709 13:00:20.241500  4802 consensus_queue.cc:1048] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.013s
I20260709 13:00:20.254948  4857 leader_election.cc:290] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 8 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:20.255452  4746 raft_consensus.cc:3055] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:00:20.185513  4683 raft_consensus.cc:515] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:20.278945  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:20.299129  4981 raft_consensus.cc:2749] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:00:20.303136  4601 raft_consensus.cc:2749] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 13:00:20.316431  4872 raft_consensus.cc:515] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:20.282910  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:20.288185  1745 rpcz_store.cc:276] 0709 13:00:18.181713 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:20.192950 (+2011237us) service_pool.cc:224] Handling call
0709 13:00:20.193354 (+   404us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:00:20.287440 (+ 94086us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:20.287692 (+   252us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:00:20.318513  4887 raft_consensus.cc:2804] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:00:20.318794  4790 leader_election.cc:290] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:20.319023  4887 raft_consensus.cc:493] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:00:20.319455  4887 raft_consensus.cc:3060] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:00:20.319732  1598 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 2035) took 1790 ms. Trace:
I20260709 13:00:20.320521  4875 raft_consensus.cc:493] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
W20260709 13:00:20.301458  4884 consensus_meta.cc:287] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 1.475s	user 0.013s	sys 0.012s
I20260709 13:00:20.321130  4875 raft_consensus.cc:515] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:20.321233  4884 raft_consensus.cc:515] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:20.322029  4849 leader_election.cc:290] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 8 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:20.322628  2331 raft_consensus.cc:2393] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 8: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:00:20.305265  1671 raft_consensus.cc:1240] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 9. Current term is 10. Ops: []
W20260709 13:00:20.303704  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 55038.6us
I20260709 13:00:20.309835  4897 consensus_queue.cc:1048] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.014s
I20260709 13:00:20.324148  2301 leader_election.cc:304] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:00:20.324740  4683 leader_election.cc:290] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 6 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:20.325088  1748 raft_consensus.cc:2368] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 5. Current term is 6.
I20260709 13:00:20.325212  4884 leader_election.cc:290] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 8 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:20.325735  4951 raft_consensus.cc:3060] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:00:20.327399  4759 leader_election.cc:290] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 8 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:00:20.327895  5012 log.cc:927] Time spent T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.406s	user 0.002s	sys 0.000s
I20260709 13:00:20.312456  4746 raft_consensus.cc:740] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:00:20.329030  4746 consensus_queue.cc:260] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:20.306190  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 71140.6us
I20260709 13:00:20.284930  4920 raft_consensus.cc:493] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:20.306687  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:20.332795  1676 raft_consensus.cc:3060] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:00:20.226227  4995 consensus_queue.cc:260] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:20.309741  4997 raft_consensus.cc:515] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:20.303444  4967 raft_consensus.cc:515] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:20.299707  5011 raft_consensus.cc:740] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:00:20.322355  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:20.347308  1678 raft_consensus.cc:2393] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 7: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:00:20.329718  4985 leader_election.cc:290] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:20.329738  1750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55848 (request call id 2077) took 2156 ms. Trace:
I20260709 13:00:20.335848  2301 leader_election.cc:304] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:00:20.330402  4746 raft_consensus.cc:3060] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:00:20.331872  4987 leader_election.cc:290] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 6 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:00:20.280109  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 114334us
W20260709 13:00:20.339433  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 150902us
W20260709 13:00:20.287693  1753 rpcz_store.cc:269] 0709 13:00:18.152295 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:19.316892 (+1164597us) service_pool.cc:224] Handling call
0709 13:00:19.841424 (+524532us) spinlock_profiling.cc:243] Waited 7.15 ms on lock 0x7b58001ffe98. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f426303db71 00007f426303da22 00007f426bf56c09 00007f426b0b0ee0 00007f426b17a0ab 00007f426b17a131 00007f426b1280a0 00007f426b127984 00007f42646a26d4 00007f42646a2662
0709 13:00:20.235938 (+394514us) spinlock_profiling.cc:243] Waited 84.1 ms on lock 0x7b54000190d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:00:20.242935 (+  6997us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":209795200}
I20260709 13:00:20.320163  1598 rpcz_store.cc:276] 0709 13:00:18.529630 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:18.787245 (+257615us) service_pool.cc:224] Handling call
0709 13:00:18.981397 (+194152us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 13:00:19.011714 (+ 30317us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:00:19.011770 (+    56us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 13:00:19.027513 (+ 15743us) log.cc:844] Serialized 78 byte log entry
0709 13:00:19.208239 (+180726us) raft_consensus.cc:1645] Marking committed up to 3
0709 13:00:19.208283 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:19.208352 (+    69us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:00:20.064089 (+855737us) spinlock_profiling.cc:243] Waited 11.9 ms on lock 0x7b4400022b4c. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f4262f9bc85 00007f4262f9c6a2 00007f4265d0e1db 00007f4265d09aaf 00007f4265cfd86e 00007f4265cfcbb6 00007f4265cfc7d8 00007f4265d06221 00007f4265d061d9 00007f4265d06169
0709 13:00:20.300314 (+236225us) raft_consensus.cc:1685] finished
0709 13:00:20.300348 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:20.319655 (+ 19307us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":12855,"raft.queue_time_us":67508,"raft.run_cpu_time_us":518,"raft.run_wall_time_us":32293,"spinlock_wait_cycles":27423488,"thread_start_us":132772,"threads_started":1,"wal-append.queue_time_us":183274}
I20260709 13:00:20.358915  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:20.327903  4875 leader_election.cc:290] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:20.361485  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:20.362146  2335 raft_consensus.cc:2468] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 8.
I20260709 13:00:20.351119  2301 leader_election.cc:304] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:20.366024  2332 raft_consensus.cc:1240] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 5 LEADER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 4. Current term is 5. Ops: []
I20260709 13:00:20.331094  4920 raft_consensus.cc:515] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:20.352154  4884 raft_consensus.cc:2804] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:00:20.350785  4995 raft_consensus.cc:3060] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Advancing to term 8
W20260709 13:00:20.280611  1718 connection.cc:256] RPC call timeout handler was delayed by -1.0844865860000255s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1663s for a response
I20260709 13:00:20.280048  4922 raft_consensus.cc:515] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:20.385597  5011 consensus_queue.cc:260] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 5.8, Last appended by leader: 8, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:20.387346  5011 raft_consensus.cc:3060] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:00:20.388264  4922 leader_election.cc:290] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:20.357667  1673 raft_consensus.cc:2368] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 4. Current term is 5.
I20260709 13:00:20.391109  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:00:20.391486  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 79651.8us
I20260709 13:00:20.367971  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:00:20.369474  2333 raft_consensus.cc:1240] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 8. Current term is 9. Ops: []
I20260709 13:00:20.325742  4952 consensus_queue.cc:1048] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.031s
I20260709 13:00:20.370677  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:20.371371  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:20.348033  1750 rpcz_store.cc:276] 0709 13:00:18.173349 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:19.720880 (+1547531us) service_pool.cc:224] Handling call
0709 13:00:19.721679 (+   799us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:00:19.739962 (+ 18283us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:00:19.739993 (+    31us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:00:19.757984 (+ 17991us) log.cc:844] Serialized 52 byte log entry
0709 13:00:19.886744 (+128760us) raft_consensus.cc:1645] Marking committed up to 3
0709 13:00:19.886808 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:19.886917 (+   109us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:00:20.075550 (+188633us) spinlock_profiling.cc:243] Waited 7.34 ms on lock 0x7b4800121ed8. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426b08baa1 00007f426b082b37 00007f426b0829ca 00007f426b17a26f 00007f426b179f37 00007f426b18e55c 00007f426b18e508 00007f426b18e488 00007f426b18e443 00007f426b18d6ec 00007f426a0b7a96
0709 13:00:20.329358 (+253808us) raft_consensus.cc:1685] finished
0709 13:00:20.329385 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:20.329664 (+   279us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":17822,"raft.queue_time_us":18459,"raft.run_cpu_time_us":2503,"raft.run_wall_time_us":332566,"spinlock_wait_cycles":16887680,"thread_start_us":119769,"threads_started":1,"wal-append.queue_time_us":201107}
I20260709 13:00:20.396754  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:20.397570  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:00:20.377295  1718 leader_election.cc:304] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:20.402377  1718 leader_election.cc:304] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:20.403311  4802 raft_consensus.cc:2749] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:00:20.404062  4802 raft_consensus.cc:2749] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:00:20.398056  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:20.405470  1750 raft_consensus.cc:2393] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 4: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:00:20.405884  1753 raft_consensus.cc:2468] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 10.
I20260709 13:00:20.407758  1718 leader_election.cc:304] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:20.408753  4690 raft_consensus.cc:2749] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:00:20.410408  1718 leader_election.cc:304] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:00:20.379676  1601 raft_consensus.cc:1240] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 5. Current term is 6. Ops: []
I20260709 13:00:20.382828  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:20.383364  1676 raft_consensus.cc:2468] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 8.
I20260709 13:00:20.384116  1679 raft_consensus.cc:2468] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 6.
I20260709 13:00:20.384317  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:20.416172  1718 leader_election.cc:304] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:00:20.368992  4884 raft_consensus.cc:493] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:20.406404  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:20.364620  4574 leader_election.cc:290] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:20.419409  1750 raft_consensus.cc:3060] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:00:20.419623  1718 leader_election.cc:304] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:00:20.409498  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:20.379360  1717 leader_election.cc:304] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:00:20.413419  1646 leader_election.cc:304] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:20.385282  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:00:20.432739  1753 raft_consensus.cc:2468] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 6.
I20260709 13:00:20.433350  1646 leader_election.cc:304] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:00:20.434607  1646 leader_election.cc:304] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:00:20.436000  1646 leader_election.cc:304] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
W20260709 13:00:20.436718  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 57450.8us
I20260709 13:00:20.437888  1717 leader_election.cc:304] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:20.374625  4875 raft_consensus.cc:2804] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:00:20.438969  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:20.439289  1717 leader_election.cc:304] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:00:20.444743  1717 leader_election.cc:304] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:20.454205  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 8 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:20.458077  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:20.459347  1753 raft_consensus.cc:2410] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for term 7 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20260709 13:00:20.422029  4690 raft_consensus.cc:2749] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:00:20.421774  4884 raft_consensus.cc:3060] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:00:20.460628  1646 leader_election.cc:304] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:00:20.422961  4802 raft_consensus.cc:2749] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:00:20.447511  4875 raft_consensus.cc:493] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:00:20.445020  2328 raft_consensus.cc:3060] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:00:20.440958  4598 raft_consensus.cc:2749] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:00:20.485910  1646 leader_election.cc:304] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:00:20.445783  4683 raft_consensus.cc:2804] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:00:20.444504  4849 raft_consensus.cc:2804] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:00:20.412782  4987 consensus_queue.cc:1048] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.048s
I20260709 13:00:20.449563  4922 raft_consensus.cc:2804] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:00:20.456533  4574 raft_consensus.cc:2804] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:00:20.420073  4988 consensus_queue.cc:1048] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.063s
I20260709 13:00:20.420348  4978 consensus_queue.cc:1059] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.064s
I20260709 13:00:20.495326  2329 raft_consensus.cc:1240] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 6. Current term is 7. Ops: []
I20260709 13:00:20.555424  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
W20260709 13:00:20.556594  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 157138us
I20260709 13:00:20.456387  4749 raft_consensus.cc:2749] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:00:20.418200  5007 tablet_copy_client.cc:323] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Beginning tablet copy session from remote peer at address 127.0.107.195:40693
W20260709 13:00:20.544212  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 58421.9us
I20260709 13:00:20.543671  4875 raft_consensus.cc:3060] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:00:20.543560  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:20.456065  5033 raft_consensus.cc:2804] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:00:20.545897  5028 raft_consensus.cc:493] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:20.494472  5010 raft_consensus.cc:2804] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:00:20.551631  1750 raft_consensus.cc:2468] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 8.
I20260709 13:00:20.482657  4993 raft_consensus.cc:2764] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20260709 13:00:20.486195  4790 raft_consensus.cc:2749] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:00:20.559260  1569 leader_election.cc:304] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:00:20.575520  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:20.576278  1601 raft_consensus.cc:2468] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 8.
I20260709 13:00:20.576900  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:20.549669  4990 raft_consensus.cc:2804] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:00:20.545749  4683 raft_consensus.cc:493] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:20.435520  4857 consensus_queue.cc:1048] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.057s
I20260709 13:00:20.566100  1753 raft_consensus.cc:2393] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 7: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
W20260709 13:00:20.565979  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 84386.4us
I20260709 13:00:20.548826  4849 raft_consensus.cc:493] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Starting leader election (detected failure of leader af520b67945d443b901bbeca37cc5701)
W20260709 13:00:20.572978  1751 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55840 (request call id 2258) took 1886 ms (client timeout 1597 ms). Trace:
I20260709 13:00:20.576937  1569 leader_election.cc:304] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:00:20.577821  1750 raft_consensus.cc:2468] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 6.
W20260709 13:00:20.586826  1751 rpcz_store.cc:269] 0709 13:00:18.686843 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:19.249389 (+562546us) service_pool.cc:224] Handling call
0709 13:00:20.572749 (+1323360us) spinlock_profiling.cc:243] Waited 849 ms on lock 0x7b5400384d54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:00:20.572895 (+   146us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1951619968}
I20260709 13:00:20.554272  4922 raft_consensus.cc:493] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:20.591182  1569 leader_election.cc:304] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:00:20.587760  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:20.592617  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:00:20.559679  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 115012us
I20260709 13:00:20.588133  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:20.588768  1643 leader_election.cc:304] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:20.589717  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:20.543601  4537 raft_consensus.cc:515] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:20.564869  5033 raft_consensus.cc:493] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:20.594056  1751 raft_consensus.cc:2468] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 5.
I20260709 13:00:20.594864  1753 raft_consensus.cc:3060] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:00:20.576608  4872 leader_election.cc:290] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 10 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:20.595391  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:20.562351  4959 consensus_queue.cc:1059] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.027s
I20260709 13:00:20.582386  4779 leader_election.cc:290] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 10 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:20.566696  4574 raft_consensus.cc:697] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 7 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:00:20.571316  5010 raft_consensus.cc:493] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:20.614029  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:20.595039  4922 raft_consensus.cc:3060] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:00:20.610456  1752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55848 (request call id 2082) took 2428 ms. Trace:
I20260709 13:00:20.610680  1604 raft_consensus.cc:1240] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 6. Current term is 7. Ops: []
I20260709 13:00:20.617379  2336 raft_consensus.cc:2468] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 8.
I20260709 13:00:20.580289  4847 raft_consensus.cc:2804] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Leader election won for term 8
I20260709 13:00:20.613343  4601 raft_consensus.cc:2804] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Leader election won for term 8
I20260709 13:00:20.589267  4849 raft_consensus.cc:3060] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Advancing to term 6
W20260709 13:00:20.607530  5030 log.cc:927] Time spent T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.167s	user 0.001s	sys 0.000s
W20260709 13:00:20.615005  1645 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:20.669072  2301 leader_election.cc:304] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:20.640550  4978 raft_consensus.cc:2749] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:00:20.658046  1752 rpcz_store.cc:276] 0709 13:00:18.181956 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:20.193572 (+2011616us) service_pool.cc:224] Handling call
0709 13:00:20.193964 (+   392us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:00:20.212746 (+ 18782us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:00:20.212798 (+    52us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:00:20.213653 (+   855us) log.cc:844] Serialized 52 byte log entry
0709 13:00:20.365093 (+151440us) raft_consensus.cc:1645] Marking committed up to 1
0709 13:00:20.365140 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:20.365211 (+    71us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:00:20.610148 (+244937us) raft_consensus.cc:1685] finished
0709 13:00:20.610176 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:20.610345 (+   169us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":17953,"raft.queue_time_us":18253,"raft.run_cpu_time_us":154,"raft.run_wall_time_us":28998,"thread_start_us":111092,"threads_started":1,"wal-append.queue_time_us":228124}
I20260709 13:00:20.655902  5010 raft_consensus.cc:3060] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:00:20.587303  4683 raft_consensus.cc:3060] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:00:20.609539  4537 leader_election.cc:290] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 7 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:20.611903  4990 raft_consensus.cc:697] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 7 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:00:20.609737  5033 raft_consensus.cc:3060] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:00:20.600778  4997 leader_election.cc:290] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 5 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:20.574120  5028 raft_consensus.cc:515] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:20.667151  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:00:20.614409  4956 raft_consensus.cc:2804] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader election won for term 6
I20260709 13:00:20.656651  4857 consensus_queue.cc:1048] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.026s
I20260709 13:00:20.672843  1745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43156 (request call id 576) took 2431 ms. Trace:
W20260709 13:00:20.674301  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 108405us
I20260709 13:00:20.594070  5045 consensus_queue.cc:1059] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 7, Time since last communication: 0.088s
I20260709 13:00:20.656879  4967 leader_election.cc:290] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:20.670683  4847 raft_consensus.cc:697] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:00:20.712001  5045 raft_consensus.cc:3055] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:00:20.712435  5045 raft_consensus.cc:740] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:00:20.713342  5045 consensus_queue.cc:260] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:20.715229  5038 log.cc:927] Time spent T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.161s	user 0.001s	sys 0.000s
I20260709 13:00:20.717119  1747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55848 (request call id 2080) took 2543 ms. Trace:
I20260709 13:00:20.717543  1747 rpcz_store.cc:276] 0709 13:00:18.174064 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:20.183193 (+2009129us) service_pool.cc:224] Handling call
0709 13:00:20.183623 (+   430us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:00:20.213652 (+ 30029us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:00:20.213703 (+    51us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:00:20.214304 (+   601us) log.cc:844] Serialized 26 byte log entry
0709 13:00:20.343173 (+128869us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:00:20.357607 (+ 14434us) log.cc:844] Serialized 14 byte log entry
0709 13:00:20.357859 (+   252us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:20.357939 (+    80us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:00:20.716865 (+358926us) raft_consensus.cc:1685] finished
0709 13:00:20.716890 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:20.717041 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":40206,"prepare.queue_time_us":17413,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":50,"raft.queue_time_us":19810,"raft.run_cpu_time_us":101,"raft.run_wall_time_us":101,"spinlock_wait_cycles":8380672,"thread_start_us":84358,"threads_started":1,"wal-append.queue_time_us":187108}
I20260709 13:00:20.719887  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:20.720513  1747 raft_consensus.cc:3060] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:00:20.724352  1599 raft_consensus.cc:1275] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 13:00:20.728329  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:00:20.729149  2334 raft_consensus.cc:2368] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 7. Current term is 8.
I20260709 13:00:20.679481  4779 raft_consensus.cc:2804] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Leader election won for term 8
W20260709 13:00:20.691202  5037 log.cc:927] Time spent T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.134s	user 0.001s	sys 0.000s
I20260709 13:00:20.707669  1753 raft_consensus.cc:2468] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 8.
W20260709 13:00:20.708106  1646 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:20.710394  5032 raft_consensus.cc:3055] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:00:20.713011  4847 consensus_queue.cc:237] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:20.600085  5036 consensus_queue.cc:1059] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.088s
W20260709 13:00:20.744872  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 131948us
I20260709 13:00:20.705489  1745 rpcz_store.cc:276] 0709 13:00:18.241152 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:20.318844 (+2077692us) service_pool.cc:224] Handling call
0709 13:00:20.319222 (+   378us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:00:20.319659 (+   437us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:00:20.319688 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:00:20.320293 (+   605us) log.cc:844] Serialized 26 byte log entry
0709 13:00:20.397573 (+ 77280us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:00:20.671817 (+274244us) log.cc:844] Serialized 14 byte log entry
0709 13:00:20.672366 (+   549us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:20.672483 (+   117us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:00:20.672541 (+    58us) raft_consensus.cc:1685] finished
0709 13:00:20.672561 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:20.672751 (+   190us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":35439,"prepare.queue_time_us":167140,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":47,"raft.queue_time_us":725,"raft.run_cpu_time_us":217,"raft.run_wall_time_us":5603,"thread_start_us":288330,"threads_started":2,"wal-append.queue_time_us":229007}
I20260709 13:00:20.747507  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:20.748268  1745 raft_consensus.cc:3060] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Advancing to term 6
W20260709 13:00:20.736531  1571 leader_election.cc:336] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Timed out: RequestConsensusVote RPC to 127.0.107.195:40693 timed out after 1.597s (SENT)
I20260709 13:00:20.739990  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:20.752007  1753 raft_consensus.cc:3060] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:00:20.753604  1569 leader_election.cc:400] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 4 election: Vote denied by peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261) with higher term. Message: Invalid argument: T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 4. Current term is 5.
I20260709 13:00:20.753942  1569 leader_election.cc:403] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
I20260709 13:00:20.743072  1597 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 606) took 1228 ms. Trace:
I20260709 13:00:20.754985  1597 rpcz_store.cc:276] 0709 13:00:19.515163 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:19.607746 (+ 92583us) service_pool.cc:224] Handling call
0709 13:00:19.608190 (+   444us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:20.067253 (+459063us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:20.742824 (+675571us) spinlock_profiling.cc:243] Waited 453 ms on lock 0x7b54002170d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:00:20.743002 (+   178us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1042266880}
I20260709 13:00:20.704525  4956 raft_consensus.cc:697] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:00:20.809307  1672 raft_consensus.cc:1240] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 7. Current term is 8. Ops: [7.8-7.8]
I20260709 13:00:20.809417  4956 consensus_queue.cc:237] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:20.749219  4779 raft_consensus.cc:697] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 8 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:00:20.810683  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:00:20.811575  5015 log.cc:927] Time spent T 00000000000000000000000000000000 P 919f6392d4f44fd68987e2c5e7ec37ca: Append to log took a long time: real 0.889s	user 0.003s	sys 0.001s
I20260709 13:00:20.811383  5033 raft_consensus.cc:515] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:20.811478  4779 consensus_queue.cc:237] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:20.812158  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:20.816216  1672 raft_consensus.cc:3060] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:00:20.822751  1753 raft_consensus.cc:2468] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 5.
I20260709 13:00:20.757036  1747 raft_consensus.cc:2468] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 8.
W20260709 13:00:20.758966  5043 log.cc:927] Time spent T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.159s	user 0.000s	sys 0.001s
I20260709 13:00:20.705794  4574 consensus_queue.cc:237] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:20.768189  4982 leader_election.cc:290] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:20.745926  4951 raft_consensus.cc:2749] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260709 13:00:20.773784  5040 log.cc:927] Time spent T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.224s	user 0.001s	sys 0.000s
I20260709 13:00:20.774859  4547 leader_election.cc:290] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 6 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:20.776247  5036 consensus_queue.cc:1243] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer af520b67945d443b901bbeca37cc5701 log is divergent from this leader: its last log entry 8.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260709 13:00:20.778972  2331 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39738 (request call id 708) took 1242 ms. Trace:
I20260709 13:00:20.879220  2331 rpcz_store.cc:276] 0709 13:00:19.536765 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:19.536906 (+   141us) service_pool.cc:224] Handling call
0709 13:00:20.778704 (+1241798us) spinlock_profiling.cc:243] Waited 784 ms on lock 0x7b54005f3a54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:00:20.778890 (+   186us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1804193024}
I20260709 13:00:20.779695  2336 raft_consensus.cc:1240] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 7. Current term is 8. Ops: []
W20260709 13:00:20.780668  1748 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55840 (request call id 2259) took 2093 ms (client timeout 1805 ms). Trace:
W20260709 13:00:20.880710  1748 rpcz_store.cc:269] 0709 13:00:18.687187 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:19.315579 (+628392us) service_pool.cc:224] Handling call
0709 13:00:20.780413 (+1464834us) spinlock_profiling.cc:243] Waited 982 ms on lock 0x7b54003705d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:00:20.780576 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2257624832}
I20260709 13:00:20.882177  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:20.882957  1748 raft_consensus.cc:3060] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:00:20.781807  4920 leader_election.cc:290] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:20.783068  4648 raft_consensus.cc:3060] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:00:20.785763  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:20.788025  2298 leader_election.cc:304] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
W20260709 13:00:20.889243  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 311419us
W20260709 13:00:20.878357  4746 consensus_meta.cc:287] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.522s	user 0.011s	sys 0.003s
I20260709 13:00:20.745631  5045 raft_consensus.cc:3060] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:00:20.741919  5032 raft_consensus.cc:740] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:00:20.800112  1746 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55848 (request call id 2076) took 2627 ms. Trace:
W20260709 13:00:20.806646  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 61028.6us
I20260709 13:00:20.807317  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
W20260709 13:00:20.808292  5049 log.cc:927] Time spent T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.131s	user 0.000s	sys 0.000s
I20260709 13:00:20.905536  1746 rpcz_store.cc:276] 0709 13:00:18.173100 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:19.692518 (+1519418us) service_pool.cc:224] Handling call
0709 13:00:19.720668 (+ 28150us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:00:19.743091 (+ 22423us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 13:00:19.948105 (+205014us) log.cc:844] Serialized 14 byte log entry
0709 13:00:20.290257 (+342152us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:00:20.336562 (+ 46305us) log.cc:844] Serialized 26 byte log entry
0709 13:00:20.336816 (+   254us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:00:20.341895 (+  5079us) log.cc:844] Serialized 14 byte log entry
0709 13:00:20.342639 (+   744us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:20.342726 (+    87us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:00:20.799819 (+457093us) raft_consensus.cc:1685] finished
0709 13:00:20.799846 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:20.800002 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":38972,"prepare.queue_time_us":288184,"prepare.run_cpu_time_us":102,"prepare.run_wall_time_us":104,"raft.queue_time_us":35613,"raft.run_cpu_time_us":442,"raft.run_wall_time_us":18599,"thread_start_us":431798,"threads_started":2,"wal-append.queue_time_us":377424}
I20260709 13:00:20.808135  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:20.756304  1571 leader_election.cc:304] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
W20260709 13:00:20.909960  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 173427us
W20260709 13:00:20.910789  1571 leader_election.cc:336] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Timed out: RequestConsensusVote RPC to 127.0.107.195:40693 timed out after 1.805s (SENT)
I20260709 13:00:20.909988  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:20.854854  1646 leader_election.cc:304] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:00:20.858845  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:20.873363  4628 raft_consensus.cc:3060] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 4 FOLLOWER]: Advancing to term 5
W20260709 13:00:20.868799  5037 log.cc:927] Time spent T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.119s	user 0.000s	sys 0.000s
I20260709 13:00:20.790836  1751 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43156 (request call id 577) took 2549 ms. Trace:
I20260709 13:00:20.899312  1754 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55848 (request call id 2083) took 2717 ms. Trace:
I20260709 13:00:20.860121  4990 consensus_queue.cc:237] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:20.879812  5033 leader_election.cc:290] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 8 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:20.878990  5010 raft_consensus.cc:515] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:20.902101  5046 log.cc:927] Time spent T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.221s	user 0.000s	sys 0.001s
I20260709 13:00:20.870815  4872 raft_consensus.cc:3055] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:00:20.910885  1677 raft_consensus.cc:1275] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260709 13:00:20.911366  1571 leader_election.cc:304] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:00:20.853528  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:20.963691  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:20.964586  1747 raft_consensus.cc:3060] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Advancing to term 8
W20260709 13:00:20.966766  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 56001.1us
W20260709 13:00:20.869194  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 46324.7us
I20260709 13:00:20.968911  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:20.969513  2299 leader_election.cc:304] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:20.969664  1596 raft_consensus.cc:2468] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 6.
I20260709 13:00:20.970810  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:20.971123  2299 leader_election.cc:304] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:00:20.972901  2299 leader_election.cc:304] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:00:20.900309  1749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55848 (request call id 2079) took 2726 ms. Trace:
I20260709 13:00:20.973987  1749 rpcz_store.cc:276] 0709 13:00:18.173830 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:20.177440 (+2003610us) service_pool.cc:224] Handling call
0709 13:00:20.177875 (+   435us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:00:20.178395 (+   520us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:00:20.178449 (+    54us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:00:20.193767 (+ 15318us) log.cc:844] Serialized 26 byte log entry
0709 13:00:20.357620 (+163853us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:00:20.451728 (+ 94108us) log.cc:844] Serialized 14 byte log entry
0709 13:00:20.452106 (+   378us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:20.452210 (+   104us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:00:20.771567 (+319357us) spinlock_profiling.cc:243] Waited 3.53 ms on lock 0x7b18002dbce0. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f42630432a6 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f4262f9bc85 00007f4262f9c6a2 00007f4265d0e1db
0709 13:00:20.772988 (+  1421us) spinlock_profiling.cc:243] Waited 7.06 ms on lock 0x7b4400022b4c. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f4262f9bc85 00007f4262f9c6a2 00007f4265d0e1db 00007f4265d09aaf 00007f4265cfd86e 00007f4265cfcbb6 00007f4265cfc7d8 00007f4265d06221 00007f4265d061d9 00007f4265d06169
0709 13:00:20.900011 (+127023us) raft_consensus.cc:1685] finished
0709 13:00:20.900044 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:20.900217 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":20479,"prepare.queue_time_us":124749,"prepare.run_cpu_time_us":6936,"prepare.run_wall_time_us":48672,"raft.queue_time_us":199,"raft.run_cpu_time_us":281,"raft.run_wall_time_us":22951,"spinlock_wait_cycles":24369280,"thread_start_us":111423,"threads_started":2,"wal-append.queue_time_us":173728}
I20260709 13:00:20.977303  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:20.978062  1749 raft_consensus.cc:3060] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Advancing to term 10
W20260709 13:00:20.917480  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 62744.2us
I20260709 13:00:20.912683  1751 rpcz_store.cc:276] 0709 13:00:18.241433 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:20.611228 (+2369795us) service_pool.cc:224] Handling call
0709 13:00:20.611632 (+   404us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:00:20.612033 (+   401us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:00:20.612067 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:00:20.612612 (+   545us) log.cc:844] Serialized 26 byte log entry
0709 13:00:20.678682 (+ 66070us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:00:20.691179 (+ 12497us) log.cc:844] Serialized 14 byte log entry
0709 13:00:20.741921 (+ 50742us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:20.742001 (+    80us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:00:20.790521 (+ 48520us) raft_consensus.cc:1685] finished
0709 13:00:20.790550 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:20.790758 (+   208us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":50387,"prepare.queue_time_us":12103,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":52,"raft.queue_time_us":57977,"raft.run_cpu_time_us":319,"raft.run_wall_time_us":93004,"thread_start_us":65612,"threads_started":1,"wal-append.queue_time_us":174342}
I20260709 13:00:20.914239  1754 rpcz_store.cc:276] 0709 13:00:18.182192 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:20.244145 (+2061953us) service_pool.cc:224] Handling call
0709 13:00:20.262758 (+ 18613us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:00:20.263148 (+   390us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:00:20.263177 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:00:20.277851 (+ 14674us) log.cc:844] Serialized 26 byte log entry
0709 13:00:20.413942 (+136091us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:00:20.416845 (+  2903us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:20.416956 (+   111us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:00:20.899062 (+482106us) raft_consensus.cc:1685] finished
0709 13:00:20.899087 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:20.899238 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4311,"raft.queue_time_us":2595,"raft.run_cpu_time_us":120,"raft.run_wall_time_us":3460,"thread_start_us":68808,"threads_started":1,"wal-append.queue_time_us":245853}
I20260709 13:00:20.922017  1753 raft_consensus.cc:3060] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:00:20.923059  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:20.923465  1745 raft_consensus.cc:2468] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 6.
I20260709 13:00:20.981989  1748 raft_consensus.cc:2468] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 7.
I20260709 13:00:20.954139  1569 leader_election.cc:400] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 7 election: Vote denied by peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625) with higher term. Message: Invalid argument: T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 7. Current term is 8.
I20260709 13:00:20.983902  1569 leader_election.cc:403] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 7 election: Cancelling election due to peer responding with higher term
I20260709 13:00:20.956355  2301 leader_election.cc:304] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:20.904124  4964 raft_consensus.cc:2804] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:00:20.957890  2334 raft_consensus.cc:1240] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 8. Current term is 9. Ops: []
I20260709 13:00:20.878819  4998 consensus_queue.cc:1059] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.041s
I20260709 13:00:20.983528  1748 raft_consensus.cc:1275] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260709 13:00:20.986783  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:20.986918  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:20.993013  1746 raft_consensus.cc:3060] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:00:20.994285  1748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55840 (request call id 2257) took 2342 ms. Trace:
I20260709 13:00:21.000380  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:21.000701  1748 rpcz_store.cc:276] 0709 13:00:18.651816 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:20.982726 (+2330910us) service_pool.cc:224] Handling call
0709 13:00:20.983192 (+   466us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:20.993998 (+ 10806us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:20.994212 (+   214us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:00:21.000890  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:21.002173  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:20.996182  1671 raft_consensus.cc:2393] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 10: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:00:21.004482  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:20.979805  1646 leader_election.cc:304] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:00:21.007467  4857 raft_consensus.cc:2804] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:00:21.007735  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:20.996547  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:21.008183  4857 raft_consensus.cc:697] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 5 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:00:21.008430  1748 raft_consensus.cc:2468] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 8.
I20260709 13:00:21.008724  1597 raft_consensus.cc:2468] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 8.
I20260709 13:00:21.009238  4857 consensus_queue.cc:237] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:21.009820  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:21.010001  1748 raft_consensus.cc:1275] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260709 13:00:20.996999  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:20.997768  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:21.010728  1748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55840 (request call id 2262) took 2303 ms. Trace:
I20260709 13:00:21.010846  1599 raft_consensus.cc:2468] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 10.
I20260709 13:00:21.011118  1748 rpcz_store.cc:276] 0709 13:00:18.706957 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:21.009342 (+2302385us) service_pool.cc:224] Handling call
0709 13:00:21.009659 (+   317us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:21.010419 (+   760us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:21.010671 (+   252us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:00:21.011752  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:20.999367  1676 raft_consensus.cc:1240] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 4. Current term is 5. Ops: []
I20260709 13:00:21.011826  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:21.005928  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:21.019954  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:21.020890  1597 raft_consensus.cc:2468] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 7.
I20260709 13:00:21.021639  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:21.023910  1671 raft_consensus.cc:1240] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 5. Current term is 6. Ops: []
W20260709 13:00:21.041673  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 68902.2us
I20260709 13:00:21.043056  2299 leader_election.cc:304] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 13:00:21.045269  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 91291us
I20260709 13:00:21.050359  1646 leader_election.cc:304] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:00:20.957929  4746 raft_consensus.cc:2804] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Leader election won for term 8
I20260709 13:00:20.952250  4601 raft_consensus.cc:697] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:00:20.959332  4872 raft_consensus.cc:740] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:00:20.915668  5001 consensus_queue.cc:1059] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.084s
W20260709 13:00:21.000891  4995 consensus_meta.cc:287] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.610s	user 0.014s	sys 0.004s
I20260709 13:00:21.060163  2331 raft_consensus.cc:1240] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 8 LEADER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 7. Current term is 8. Ops: []
I20260709 13:00:21.053939  2299 leader_election.cc:304] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:21.055785  1571 leader_election.cc:304] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:00:21.063045  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:21.077344  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:20.988798  4964 raft_consensus.cc:493] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:21.001567  4967 raft_consensus.cc:2749] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:00:20.923127  4978 consensus_queue.cc:1048] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.130s
W20260709 13:00:21.046779  5037 log.cc:927] Time spent T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.094s	user 0.001s	sys 0.000s
I20260709 13:00:21.042933  4512 leader_election.cc:290] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 8 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:21.055588  4954 raft_consensus.cc:3060] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:00:21.217067  2329 raft_consensus.cc:1275] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
W20260709 13:00:21.073669  4887 consensus_meta.cc:287] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.754s	user 0.017s	sys 0.004s
I20260709 13:00:21.098419  1747 raft_consensus.cc:2468] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 8.
I20260709 13:00:21.255002  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:21.255409  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:21.256220  1673 raft_consensus.cc:2468] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 6.
I20260709 13:00:21.256560  1747 raft_consensus.cc:1240] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 5. Current term is 6. Ops: []
I20260709 13:00:21.257289  1747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43156 (request call id 583) took 1139 ms. Trace:
I20260709 13:00:21.187172  4547 raft_consensus.cc:493] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:21.210062  1753 raft_consensus.cc:2468] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 8.
I20260709 13:00:21.112058  1749 raft_consensus.cc:2468] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 10.
I20260709 13:00:21.259438  1646 leader_election.cc:304] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:00:21.270006  1746 raft_consensus.cc:2468] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 10.
I20260709 13:00:21.271044  1749 raft_consensus.cc:1240] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 6. Current term is 7. Ops: [4.6-6.7]
I20260709 13:00:21.271718  1746 raft_consensus.cc:1240] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 7. Current term is 8. Ops: []
I20260709 13:00:21.040628  4722 raft_consensus.cc:2749] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260709 13:00:21.251447  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 195739us
W20260709 13:00:21.278970  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 234585us
I20260709 13:00:21.279867  1571 leader_election.cc:304] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 13:00:20.997872  4959 consensus_queue.cc:1059] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.192s
I20260709 13:00:21.259438  1569 leader_election.cc:304] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:00:21.257304  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:21.289566  1673 raft_consensus.cc:3060] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:00:21.257702  1747 rpcz_store.cc:276] 0709 13:00:20.117885 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:21.256115 (+1138230us) service_pool.cc:224] Handling call
0709 13:00:21.256388 (+   273us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:21.256987 (+   599us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:21.257213 (+   226us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:00:21.064626  4905 raft_consensus.cc:3060] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:00:21.271785  1749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43156 (request call id 585) took 1115 ms. Trace:
I20260709 13:00:21.219803  4951 raft_consensus.cc:3055] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:00:21.213804  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:21.273589  1678 raft_consensus.cc:2468] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 8.
I20260709 13:00:21.056833  4980 raft_consensus.cc:2804] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Leader election won for term 8
W20260709 13:00:21.084885  2328 consensus_meta.cc:287] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.594s	user 0.010s	sys 0.006s
I20260709 13:00:21.189755  4964 raft_consensus.cc:3060] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:00:21.065456  4900 raft_consensus.cc:2749] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:00:21.288753  1753 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43156 (request call id 584) took 1151 ms. Trace:
I20260709 13:00:21.093166  4746 raft_consensus.cc:697] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:00:21.320398  1571 leader_election.cc:304] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:00:21.072194  4672 raft_consensus.cc:2749] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
W20260709 13:00:21.259326  4683 consensus_meta.cc:287] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.569s	user 0.006s	sys 0.007s
I20260709 13:00:21.301935  1749 rpcz_store.cc:276] 0709 13:00:20.156177 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:21.270412 (+1114235us) service_pool.cc:224] Handling call
0709 13:00:21.270770 (+   358us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:00:21.271489 (+   719us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:21.271715 (+   226us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:00:21.288460  5053 raft_consensus.cc:2804] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Leader election won for term 6
I20260709 13:00:21.089399  4999 consensus_queue.cc:1059] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.190s
I20260709 13:00:21.301635  1646 leader_election.cc:304] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:00:21.312701  1718 leader_election.cc:304] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:21.313205  1753 rpcz_store.cc:276] 0709 13:00:20.118184 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:21.268451 (+1150267us) service_pool.cc:224] Handling call
0709 13:00:21.268806 (+   355us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:21.269036 (+   230us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:00:21.269065 (+    29us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:00:21.269109 (+    44us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:00:21.269159 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:21.269218 (+    59us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:21.269372 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:00:21.344683  1753 raft_consensus.cc:1240] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 7. Current term is 8. Ops: [3.5-7.7]
I20260709 13:00:21.219195  5028 leader_election.cc:290] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:00:21.321954  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 62641.6us
I20260709 13:00:21.272547  1754 raft_consensus.cc:1240] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 7. Current term is 8. Ops: [7.8-7.8]
I20260709 13:00:21.068385  4849 raft_consensus.cc:515] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:21.347568  1753 raft_consensus.cc:1240] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 9. Current term is 10. Ops: [9.6-9.6]
I20260709 13:00:21.308657  4951 raft_consensus.cc:740] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:00:21.301607  4967 raft_consensus.cc:2749] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
W20260709 13:00:21.356617  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 55083.2us
I20260709 13:00:21.349686  1749 raft_consensus.cc:1240] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 5. Current term is 6. Ops: []
I20260709 13:00:21.313551  4987 raft_consensus.cc:2804] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Leader election won for term 8
I20260709 13:00:21.317754  4920 raft_consensus.cc:493] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:21.270596  4887 raft_consensus.cc:515] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:21.095058  4955 consensus_queue.cc:1059] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.239s
I20260709 13:00:21.317926  2328 raft_consensus.cc:2468] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 7.
I20260709 13:00:21.381359  1753 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55840 (request call id 2281) took 1024 ms. Trace:
I20260709 13:00:21.402866  1753 rpcz_store.cc:276] 0709 13:00:20.357364 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:21.345783 (+988419us) service_pool.cc:224] Handling call
0709 13:00:21.346157 (+   374us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:00:21.361444 (+ 15287us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:21.381263 (+ 19819us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:00:21.399474  1749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55840 (request call id 2279) took 1044 ms. Trace:
I20260709 13:00:21.293421  4547 raft_consensus.cc:515] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:21.339530  4948 raft_consensus.cc:2804] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:00:21.398720  1646 leader_election.cc:304] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:00:21.407342  1749 rpcz_store.cc:276] 0709 13:00:20.355415 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:21.330409 (+974994us) service_pool.cc:224] Handling call
0709 13:00:21.349413 (+ 19004us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:21.396961 (+ 47548us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:21.399362 (+  2401us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:00:21.349447  1673 raft_consensus.cc:2468] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 6.
I20260709 13:00:21.341939  5000 raft_consensus.cc:493] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:21.357635  5044 raft_consensus.cc:2749] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:00:21.329126  5075 raft_consensus.cc:2804] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader election won for term 10
I20260709 13:00:21.321868  4897 consensus_queue.cc:1059] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.066s
W20260709 13:00:21.395591  4884 consensus_meta.cc:287] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.852s	user 0.013s	sys 0.001s
I20260709 13:00:21.419094  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:21.367652  4981 raft_consensus.cc:2804] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Leader election won for term 8
I20260709 13:00:21.368860  5053 raft_consensus.cc:697] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 6 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:00:21.413362  4948 raft_consensus.cc:493] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:21.440728  2334 raft_consensus.cc:1240] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 5. Current term is 6. Ops: []
I20260709 13:00:21.441046  5000 raft_consensus.cc:515] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:21.446125  5060 raft_consensus.cc:493] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:00:21.446885  5010 leader_election.cc:290] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 5 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:21.446623  1764 tablet_copy_service.cc:140] P af520b67945d443b901bbeca37cc5701: Received BeginTabletCopySession request for tablet 983621a6fc5b4c9ea0159a648dfe7a36 from peer 3b879799899a499c8d409ee9f1277cd1 ({username='slave'} at 127.0.0.1:43156)
I20260709 13:00:21.447873  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:21.382730  4802 raft_consensus.cc:2749] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:00:21.342844  4683 raft_consensus.cc:515] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:21.455749  1673 raft_consensus.cc:2468] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 7.
I20260709 13:00:21.416676  4959 raft_consensus.cc:2804] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:00:21.410327  4987 raft_consensus.cc:697] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:00:21.523461  4987 consensus_queue.cc:237] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:21.369683  4779 consensus_queue.cc:1059] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.091s
I20260709 13:00:21.419755  2328 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39726 (request call id 879) took 1054 ms. Trace:
I20260709 13:00:21.528129  4547 leader_election.cc:290] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:21.484723  5075 raft_consensus.cc:697] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:00:21.490762  1601 raft_consensus.cc:1275] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260709 13:00:21.493623  1598 raft_consensus.cc:3055] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:00:21.454761  5036 raft_consensus.cc:2804] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Leader election won for term 10
I20260709 13:00:21.325903  4601 consensus_queue.cc:237] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:21.500662  1605 raft_consensus.cc:1275] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260709 13:00:21.412850  4920 raft_consensus.cc:515] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:21.508145  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 156870us
I20260709 13:00:21.483439  4948 raft_consensus.cc:3060] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:00:21.515704  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:21.535130  1598 raft_consensus.cc:740] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:00:21.551640  1674 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52766 (request call id 2301) took 1620 ms. Trace:
I20260709 13:00:21.571484  1674 rpcz_store.cc:276] 0709 13:00:19.931431 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:19.934158 (+  2727us) service_pool.cc:224] Handling call
0709 13:00:19.934517 (+   359us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:20.227245 (+292728us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:21.551322 (+1324077us) spinlock_profiling.cc:243] Waited 292 ms on lock 0x7b54000fe954. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:00:21.551551 (+   229us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":671404416}
I20260709 13:00:21.573460  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:21.574205  1752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43156 (request call id 578) took 3332 ms. Trace:
I20260709 13:00:21.465847  4884 raft_consensus.cc:515] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:21.570226  2336 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49082 (request call id 727) took 1200 ms. Trace:
I20260709 13:00:21.582685  2336 rpcz_store.cc:276] 0709 13:00:20.370020 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:20.695780 (+325760us) service_pool.cc:224] Handling call
0709 13:00:20.779391 (+ 83611us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:20.880336 (+100945us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:21.569938 (+689602us) spinlock_profiling.cc:243] Waited 83.2 ms on lock 0x7b54005f3a50. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661618ad 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:00:21.570133 (+   195us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":191255296}
I20260709 13:00:21.574832  1752 rpcz_store.cc:276] 0709 13:00:18.241693 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:20.672306 (+2430613us) service_pool.cc:224] Handling call
0709 13:00:20.672669 (+   363us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:00:20.673212 (+   543us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:00:20.673255 (+    43us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:00:20.673866 (+   611us) log.cc:844] Serialized 26 byte log entry
0709 13:00:20.913907 (+240041us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:00:20.956880 (+ 42973us) log.cc:844] Serialized 14 byte log entry
0709 13:00:20.966828 (+  9948us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:20.966930 (+   102us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:00:21.573871 (+606941us) raft_consensus.cc:1685] finished
0709 13:00:21.573906 (+    35us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:21.574107 (+   201us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8596,"prepare.queue_time_us":47294,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":43,"raft.queue_time_us":12039,"raft.run_cpu_time_us":190,"raft.run_wall_time_us":52009,"spinlock_wait_cycles":21165312,"thread_start_us":217689,"threads_started":1,"wal-append.queue_time_us":875168}
I20260709 13:00:21.484846  4981 raft_consensus.cc:697] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
W20260709 13:00:21.502324  4922 consensus_meta.cc:287] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.835s	user 0.012s	sys 0.005s
W20260709 13:00:21.509549  5064 log.cc:927] Time spent T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.321s	user 0.000s	sys 0.001s
I20260709 13:00:21.574381  1674 raft_consensus.cc:2468] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 6.
W20260709 13:00:21.541568  1672 consensus_meta.cc:287] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.724s	user 0.014s	sys 0.000s
I20260709 13:00:21.568619  1673 raft_consensus.cc:2410] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for term 9 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
W20260709 13:00:21.568171  1680 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52778 (request call id 1868) took 3019 ms (client timeout 1730 ms). Trace:
I20260709 13:00:21.552014  4985 raft_consensus.cc:493] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:21.596933  4985 raft_consensus.cc:515] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:21.598861  5069 log.cc:927] Time spent T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.102s	user 0.001s	sys 0.000s
I20260709 13:00:21.599265  4920 leader_election.cc:290] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:21.588253  1675 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57092 (request call id 712) took 1921 ms. Trace:
I20260709 13:00:21.462479  4746 consensus_queue.cc:237] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:21.593662  1672 raft_consensus.cc:2468] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 8.
I20260709 13:00:21.569078  1598 consensus_queue.cc:260] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:21.632688  1598 raft_consensus.cc:3060] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:00:21.601521  1675 rpcz_store.cc:276] 0709 13:00:19.666576 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:19.666743 (+   167us) service_pool.cc:224] Handling call
0709 13:00:19.693109 (+ 26366us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:20.240116 (+547007us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:21.587963 (+1347847us) spinlock_profiling.cc:243] Waited 510 ms on lock 0x7b5400090e54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:00:21.588177 (+   214us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1172754816}
W20260709 13:00:21.593631  1680 rpcz_store.cc:269] 0709 13:00:18.548737 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:18.549100 (+   363us) service_pool.cc:224] Handling call
0709 13:00:21.567885 (+3018785us) spinlock_profiling.cc:243] Waited 1.44 s on lock 0x7b54001f5c54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:00:21.568064 (+   179us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3309731072}
I20260709 13:00:21.602727  1752 pending_rounds.cc:85] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701: Aborting all ops after (but not including) 7
I20260709 13:00:21.634166  1752 pending_rounds.cc:107] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701: Aborting uncommitted NO_OP operation due to leader change: 5.8
I20260709 13:00:21.634490  1752 raft_consensus.cc:2889] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:00:21.603470  1753 raft_consensus.cc:1275] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260709 13:00:21.592924  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:21.635360  1745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55848 (request call id 2089) took 3174 ms. Trace:
I20260709 13:00:21.635902  1745 rpcz_store.cc:276] 0709 13:00:18.460877 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:20.982636 (+2521759us) service_pool.cc:224] Handling call
0709 13:00:20.982985 (+   349us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:00:20.983364 (+   379us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:00:20.983394 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:00:21.003462 (+ 20068us) log.cc:844] Serialized 26 byte log entry
0709 13:00:21.370154 (+366692us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:00:21.370215 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:21.370306 (+    91us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:00:21.635055 (+264749us) raft_consensus.cc:1685] finished
0709 13:00:21.635089 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:21.635274 (+   185us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":4059,"spinlock_wait_cycles":95557504,"thread_start_us":182442,"threads_started":1,"wal-append.queue_time_us":501534}
I20260709 13:00:21.509771  4722 consensus_queue.cc:1059] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.084s
I20260709 13:00:21.619593  4981 consensus_queue.cc:237] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:21.566883  4802 raft_consensus.cc:493] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:21.621057  4985 leader_election.cc:290] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:00:21.623564  5072 log.cc:927] Time spent T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.079s	user 0.001s	sys 0.000s
W20260709 13:00:21.624154  5073 log.cc:927] Time spent T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.057s	user 0.001s	sys 0.000s
I20260709 13:00:21.552093  5075 consensus_queue.cc:237] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:21.627144  1678 raft_consensus.cc:1240] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 7. Current term is 8. Ops: []
I20260709 13:00:21.551390  4952 consensus_queue.cc:1048] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.077s
W20260709 13:00:21.555529  5065 log.cc:927] Time spent T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.273s	user 0.000s	sys 0.001s
W20260709 13:00:21.563126  5061 log.cc:927] Time spent T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.434s	user 0.001s	sys 0.001s
I20260709 13:00:21.535753  4959 raft_consensus.cc:493] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
W20260709 13:00:21.548545  4875 consensus_meta.cc:287] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.975s	user 0.014s	sys 0.001s
W20260709 13:00:21.642135  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 321781us
I20260709 13:00:21.692224  1571 leader_election.cc:304] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:00:21.642038  2299 leader_election.cc:304] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:00:21.645488  1745 raft_consensus.cc:1240] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 9. Current term is 10. Ops: [9.6-9.6]
I20260709 13:00:21.646739  4980 raft_consensus.cc:697] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 8 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:00:21.695013  1745 raft_consensus.cc:1240] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 6. Current term is 7. Ops: [6.5-6.5]
I20260709 13:00:21.527637  2328 rpcz_store.cc:276] 0709 13:00:20.365481 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:20.368666 (+  3185us) service_pool.cc:224] Handling call
0709 13:00:21.416464 (+1047798us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1408}
I20260709 13:00:21.543915  4683 leader_election.cc:290] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 7 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:21.629732  4887 leader_election.cc:290] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 6 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:21.619817  4549 leader_election.cc:290] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:21.600091  4920 raft_consensus.cc:493] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:00:21.578581  5036 raft_consensus.cc:697] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:00:21.548182  1764 tablet_copy_service.cc:161] P af520b67945d443b901bbeca37cc5701: Beginning new tablet copy session on tablet 983621a6fc5b4c9ea0159a648dfe7a36 from peer 3b879799899a499c8d409ee9f1277cd1 at {username='slave'} at 127.0.0.1:43156: session id = 3b879799899a499c8d409ee9f1277cd1-983621a6fc5b4c9ea0159a648dfe7a36
I20260709 13:00:21.505622  5060 raft_consensus.cc:515] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } }
W20260709 13:00:21.680414  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:21.680497  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
W20260709 13:00:21.716104  1570 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:00:21.716347  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 74473us
I20260709 13:00:21.601151  4922 raft_consensus.cc:515] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:21.677547  4985 raft_consensus.cc:493] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:21.717733  2299 leader_election.cc:304] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 13:00:21.705192  5049 log.cc:927] Time spent T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.054s	user 0.000s	sys 0.001s
I20260709 13:00:21.707211  1571 leader_election.cc:304] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:00:21.721338  4779 raft_consensus.cc:2804] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Leader election won for term 8
W20260709 13:00:21.722592  5067 log.cc:927] Time spent T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.130s	user 0.000s	sys 0.000s
I20260709 13:00:21.722309  4887 consensus_queue.cc:1059] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 13:00:21.705204  4959 raft_consensus.cc:3060] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Advancing to term 7
W20260709 13:00:21.725963  5070 log.cc:927] Time spent T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.055s	user 0.000s	sys 0.000s
I20260709 13:00:21.709102  4537 raft_consensus.cc:2749] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:00:21.726898  4849 leader_election.cc:290] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 6 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:21.768395  4779 raft_consensus.cc:697] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 8 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:00:21.769441  4537 raft_consensus.cc:493] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:21.771148  1671 raft_consensus.cc:1240] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 8. Current term is 9. Ops: []
I20260709 13:00:21.728219  5036 consensus_queue.cc:237] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:21.730891  5060 leader_election.cc:290] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:21.739223  4985 raft_consensus.cc:515] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:21.716521  1678 raft_consensus.cc:3060] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:00:21.745647  1571 leader_election.cc:304] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:00:21.672063  4802 raft_consensus.cc:515] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:21.750766  4846 raft_consensus.cc:2749] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:00:21.767268  4887 raft_consensus.cc:3055] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:00:21.713279  4884 leader_election.cc:290] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 9 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:21.774843  4537 raft_consensus.cc:515] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:21.777806  5077 log.cc:927] Time spent T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.063s	user 0.001s	sys 0.000s
I20260709 13:00:21.712743  4920 raft_consensus.cc:515] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:21.778117  4922 leader_election.cc:290] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 9 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:21.675662  5053 consensus_queue.cc:237] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:21.715996  5048 raft_consensus.cc:493] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:00:21.744935  4952 raft_consensus.cc:2804] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Leader election won for term 8
W20260709 13:00:21.790904  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 45360.1us
I20260709 13:00:21.836092  1571 leader_election.cc:304] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1
W20260709 13:00:21.771113  5065 log.cc:927] Time spent T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.061s	user 0.000s	sys 0.000s
I20260709 13:00:21.711321  4875 raft_consensus.cc:515] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:21.820948  1748 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.520s	user 0.006s	sys 0.009s
W20260709 13:00:21.840361  1748 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.540s	user 0.006s	sys 0.010s
I20260709 13:00:21.802522  4887 raft_consensus.cc:740] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:00:21.801775  4547 raft_consensus.cc:2749] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
W20260709 13:00:21.825268  1599 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.605s	user 0.000s	sys 0.016s
W20260709 13:00:21.841502  1599 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.639s	user 0.000s	sys 0.016s
I20260709 13:00:21.844615  1748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43156 (request call id 582) took 1727 ms. Trace:
I20260709 13:00:21.845188  1748 rpcz_store.cc:276] 0709 13:00:20.117501 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:21.011970 (+894469us) service_pool.cc:224] Handling call
0709 13:00:21.012283 (+   313us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:00:21.012714 (+   431us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:00:21.012763 (+    49us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:00:21.026519 (+ 13756us) log.cc:844] Serialized 26 byte log entry
0709 13:00:21.214575 (+188056us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:00:21.843570 (+628995us) log.cc:844] Serialized 14 byte log entry
0709 13:00:21.844125 (+   555us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:21.844238 (+   113us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:00:21.844297 (+    59us) raft_consensus.cc:1685] finished
0709 13:00:21.844316 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:21.844528 (+   212us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":108,"raft.run_cpu_time_us":94,"raft.run_wall_time_us":94,"thread_start_us":730390,"threads_started":2,"wal-append.queue_time_us":576025}
I20260709 13:00:21.853750  1571 leader_election.cc:304] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:00:21.822147  4683 raft_consensus.cc:493] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:00:21.820648  5060 raft_consensus.cc:493] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:00:21.871282  1571 leader_election.cc:304] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:00:21.880319  4537 leader_election.cc:290] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:21.809540  5081 consensus_queue.cc:1059] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 7.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.099s
I20260709 13:00:21.882659  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:21.883900  2330 raft_consensus.cc:2393] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 8: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
W20260709 13:00:21.884500  2330 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39726 (request call id 880) took 1518 ms (client timeout 1856 ms). Trace:
I20260709 13:00:21.884989  4875 leader_election.cc:290] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 7 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:21.860740  4846 raft_consensus.cc:2804] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:00:21.886786  1604 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 2043) took 1165 ms. Trace:
I20260709 13:00:21.886950  4846 raft_consensus.cc:493] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:21.887678  4846 raft_consensus.cc:3060] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:00:21.887383  1604 rpcz_store.cc:276] 0709 13:00:20.721071 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:20.735185 (+ 14114us) service_pool.cc:224] Handling call
0709 13:00:20.750519 (+ 15334us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:00:20.750914 (+   395us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:00:20.750937 (+    23us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:00:20.816455 (+ 65518us) log.cc:844] Serialized 26 byte log entry
0709 13:00:21.250323 (+433868us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:00:21.250395 (+    72us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:21.250538 (+   143us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:00:21.886466 (+635928us) raft_consensus.cc:1685] finished
0709 13:00:21.886500 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:21.886697 (+   197us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":105,"raft.run_cpu_time_us":131,"raft.run_wall_time_us":54445,"thread_start_us":375179,"threads_started":1,"wal-append.queue_time_us":626560}
I20260709 13:00:21.857550  4952 raft_consensus.cc:697] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [term 8 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:00:21.890743  5079 raft_consensus.cc:493] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:00:21.891644  5079 raft_consensus.cc:515] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:21.892014  4952 consensus_queue.cc:237] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:21.769876  4779 consensus_queue.cc:237] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:21.894593  4547 raft_consensus.cc:2749] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:00:21.839269  5048 raft_consensus.cc:515] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:21.900279  5048 leader_election.cc:290] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:21.904872  5000 leader_election.cc:290] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:21.905507  4512 raft_consensus.cc:3055] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:00:21.905946  4512 raft_consensus.cc:740] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:00:21.907428  4920 leader_election.cc:290] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:21.908566  4512 consensus_queue.cc:260] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:21.909660  1680 raft_consensus.cc:1275] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 5 index: 8. Preceding OpId from leader: term: 6 index: 9. (index mismatch)
W20260709 13:00:21.885030  2330 rpcz_store.cc:269] 0709 13:00:20.365874 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:20.368783 (+  2909us) service_pool.cc:224] Handling call
0709 13:00:21.882508 (+1513725us) spinlock_profiling.cc:243] Waited 8.16 ms on lock 0x7b5800178598. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f426303df8c 00007f426303def6 0000000000375939 00007f426b0b11e0 00007f426b17a0c4 00007f426b17a131 00007f426b1280a0 00007f426b127984 00007f42646a26d4 00007f42646a2662
0709 13:00:21.884409 (+  1901us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":18812288}
W20260709 13:00:21.915038  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 43836us
I20260709 13:00:21.882143  5081 raft_consensus.cc:3055] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:00:21.870994  2337 raft_consensus.cc:1240] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 6. Current term is 7. Ops: []
I20260709 13:00:21.926716  1571 leader_election.cc:304] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 13:00:21.915779  1598 raft_consensus.cc:1275] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260709 13:00:21.917935  1748 raft_consensus.cc:1240] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 7. Current term is 8. Ops: [5.7-7.8]
I20260709 13:00:21.868147  1678 raft_consensus.cc:2468] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 9.
I20260709 13:00:21.930016  1748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43156 (request call id 594) took 1211 ms. Trace:
I20260709 13:00:21.944794  1748 rpcz_store.cc:276] 0709 13:00:20.718392 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:21.858244 (+1139852us) service_pool.cc:224] Handling call
0709 13:00:21.917346 (+ 59102us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:00:21.929608 (+ 12262us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:21.929931 (+   323us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:00:21.933437  4951 consensus_queue.cc:260] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:21.925879  5045 consensus_meta.cc:287] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.974s	user 0.012s	sys 0.006s
I20260709 13:00:21.967083  1571 leader_election.cc:304] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
W20260709 13:00:21.954538  4628 consensus_meta.cc:287] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 1.035s	user 0.019s	sys 0.000s
W20260709 13:00:21.956161  4648 consensus_meta.cc:287] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 1.058s	user 0.013s	sys 0.005s
I20260709 13:00:21.926038  5048 raft_consensus.cc:2749] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260709 13:00:21.994093  1602 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40166 (request call id 2029) took 3523 ms (client timeout 1699 ms). Trace:
I20260709 13:00:21.925151  5081 raft_consensus.cc:740] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:00:21.954538  4547 raft_consensus.cc:2804] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:00:21.822744  4980 consensus_queue.cc:237] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:22.003618  1602 rpcz_store.cc:269] 0709 13:00:18.446397 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:18.452490 (+  6093us) service_pool.cc:224] Handling call
0709 13:00:21.969520 (+3517030us) spinlock_profiling.cc:243] Waited 1.53 s on lock 0x7b54001e0354. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:00:21.969710 (+   190us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3524677376}
I20260709 13:00:21.913178  4683 raft_consensus.cc:515] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:21.917793  5060 raft_consensus.cc:515] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:22.018823  5000 raft_consensus.cc:493] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:22.026876  2331 raft_consensus.cc:1275] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
W20260709 13:00:22.001772  4905 consensus_meta.cc:287] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.690s	user 0.013s	sys 0.002s
I20260709 13:00:21.993070  4998 raft_consensus.cc:493] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:22.000506  5045 raft_consensus.cc:2804] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader election won for term 10
W20260709 13:00:22.029950  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 453966us
I20260709 13:00:21.971141  4955 raft_consensus.cc:493] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:00:22.039356  4905 raft_consensus.cc:2749] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Leader election lost for term 7. Reason: Vote denied by peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625) with higher term. Message: Invalid argument: T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 7. Current term is 8.
I20260709 13:00:21.968815  4887 consensus_queue.cc:260] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:22.002897  4648 raft_consensus.cc:2749] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261) with higher term. Message: Invalid argument: T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 4. Current term is 5.
I20260709 13:00:22.043012  2329 raft_consensus.cc:1275] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260709 13:00:21.994431  4537 consensus_queue.cc:1048] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.060s
I20260709 13:00:22.025470  4580 raft_consensus.cc:493] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:00:22.063731  1751 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.616s	user 0.015s	sys 0.035s
W20260709 13:00:22.065783  1751 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.618s	user 0.015s	sys 0.035s
W20260709 13:00:22.049901  1596 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.740s	user 0.022s	sys 0.013s
W20260709 13:00:22.066642  1596 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.777s	user 0.024s	sys 0.013s
I20260709 13:00:22.128796  1751 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43156 (request call id 579) took 3886 ms. Trace:
I20260709 13:00:22.129531  1751 rpcz_store.cc:276] 0709 13:00:18.241921 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:20.998224 (+2756303us) service_pool.cc:224] Handling call
0709 13:00:20.998580 (+   356us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:00:21.047955 (+ 49375us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:00:21.048006 (+    51us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:00:21.066448 (+ 18442us) log.cc:844] Serialized 26 byte log entry
0709 13:00:21.399346 (+332898us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:00:22.071599 (+672253us) log.cc:844] Serialized 14 byte log entry
0709 13:00:22.128287 (+ 56688us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:22.128413 (+   126us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:00:22.128479 (+    66us) raft_consensus.cc:1685] finished
0709 13:00:22.128497 (+    18us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:22.128692 (+   195us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":56251,"raft.queue_time_us":688,"raft.run_cpu_time_us":186,"raft.run_wall_time_us":16417,"thread_start_us":746374,"threads_started":2,"wal-append.queue_time_us":447097}
W20260709 13:00:22.110757  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:21.996738  5084 consensus_queue.cc:1048] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.109s
I20260709 13:00:22.134497  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:22.163350  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:00:22.168138  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:00:22.167347  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:22.201578  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:00:22.202574  2329 raft_consensus.cc:2393] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 10: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:00:22.201872  2336 raft_consensus.cc:2393] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 7: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:00:22.067926  5045 raft_consensus.cc:697] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
W20260709 13:00:22.160866  5074 log.cc:927] Time spent T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.100s	user 0.000s	sys 0.001s
I20260709 13:00:22.207603  1596 raft_consensus.cc:2955] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Committing config change with OpId 4.5: config changed from index 3 to 5, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 13:00:22.060119  4998 raft_consensus.cc:515] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:22.131853  4887 raft_consensus.cc:3060] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Advancing to term 8
W20260709 13:00:22.208562  1746 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.642s	user 0.005s	sys 0.026s
W20260709 13:00:22.209839  2335 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39738 (request call id 713) took 1989 ms (client timeout 1501 ms). Trace:
W20260709 13:00:22.216426  1746 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.669s	user 0.005s	sys 0.027s
I20260709 13:00:22.042115  5000 raft_consensus.cc:515] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:22.216470  2335 rpcz_store.cc:269] 0709 13:00:20.220568 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:20.225639 (+  5071us) service_pool.cc:224] Handling call
0709 13:00:22.209592 (+1983953us) spinlock_profiling.cc:243] Waited 135 ms on lock 0x7b54005802d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:00:22.209741 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":311509760}
I20260709 13:00:22.104239  4580 raft_consensus.cc:515] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:22.208134  5077 log.cc:927] Time spent T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.076s	user 0.000s	sys 0.000s
I20260709 13:00:22.098767  4959 raft_consensus.cc:515] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:22.067144  4955 raft_consensus.cc:515] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:22.112264  5010 raft_consensus.cc:493] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
W20260709 13:00:22.225696  4964 consensus_meta.cc:287] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.883s	user 0.011s	sys 0.008s
I20260709 13:00:22.214325  4601 raft_consensus.cc:493] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:00:22.253602  1747 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.623s	user 0.007s	sys 0.014s
W20260709 13:00:22.254283  1644 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.609s	user 0.012s	sys 0.011s
W20260709 13:00:22.256517  1752 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.567s	user 0.043s	sys 0.051s
W20260709 13:00:22.258198  1747 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.627s	user 0.008s	sys 0.015s
I20260709 13:00:22.259130  1672 raft_consensus.cc:1275] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260709 13:00:22.259598  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:00:22.221561  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
W20260709 13:00:22.258066  1644 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.613s	user 0.012s	sys 0.011s
I20260709 13:00:22.257274  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
W20260709 13:00:22.262493  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 617911us
W20260709 13:00:22.235183  5011 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.682s	user 0.035s	sys 0.081s
I20260709 13:00:22.228111  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:00:22.267617  1752 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.579s	user 0.043s	sys 0.051s
I20260709 13:00:22.272634  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:00:22.251977  5079 leader_election.cc:290] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:00:22.269614  5102 log.cc:927] Time spent T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.249s	user 0.000s	sys 0.001s
W20260709 13:00:22.272946  5011 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.717s	user 0.035s	sys 0.081s
W20260709 13:00:22.282945  4990 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.574s	user 0.004s	sys 0.022s
I20260709 13:00:22.254338  5010 raft_consensus.cc:515] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:22.262086  4964 raft_consensus.cc:515] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:22.265652  4601 raft_consensus.cc:515] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:22.302449  4990 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.611s	user 0.005s	sys 0.023s
W20260709 13:00:22.312705  5082 log.cc:927] Time spent T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.368s	user 0.002s	sys 0.000s
I20260709 13:00:22.300688  5079 raft_consensus.cc:493] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:00:22.408118  1748 raft_consensus.cc:1240] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 7. Current term is 8. Ops: []
I20260709 13:00:22.303193  4547 raft_consensus.cc:697] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
W20260709 13:00:22.475848  1717 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.527s	user 0.016s	sys 0.012s
W20260709 13:00:22.476634  1717 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.528s	user 0.016s	sys 0.012s
I20260709 13:00:22.294338  5045 consensus_queue.cc:237] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:00:22.515340  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 567280us
I20260709 13:00:22.514489  1748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55840 (request call id 2282) took 1778 ms. Trace:
W20260709 13:00:22.513957  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 547062us
I20260709 13:00:22.538494  1748 rpcz_store.cc:276] 0709 13:00:20.736165 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:21.946003 (+1209838us) service_pool.cc:224] Handling call
0709 13:00:22.407858 (+461855us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:22.409890 (+  2032us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:22.514386 (+104496us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:00:22.547868  1717 leader_election.cc:304] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
W20260709 13:00:22.523770  2301 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:22.486357  2329 raft_consensus.cc:1275] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 13:00:22.586748  1754 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43156 (request call id 580) took 4159 ms. Trace:
I20260709 13:00:22.587455  1754 rpcz_store.cc:276] 0709 13:00:18.427623 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:20.981061 (+2553438us) service_pool.cc:224] Handling call
0709 13:00:21.272306 (+291245us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:00:21.353097 (+ 80791us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:22.586373 (+1233276us) spinlock_profiling.cc:243] Waited 291 ms on lock 0x7b54000e4a50. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661618ad 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:00:22.586619 (+   246us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":668665216}
W20260709 13:00:22.590351  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 48929.4us
I20260709 13:00:22.590587  1750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55840 (request call id 2273) took 2252 ms. Trace:
I20260709 13:00:22.592283  1750 rpcz_store.cc:276] 0709 13:00:20.338142 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:20.595127 (+256985us) service_pool.cc:224] Handling call
0709 13:00:22.590318 (+1995191us) spinlock_profiling.cc:243] Waited 1.44 s on lock 0x7b540006fc54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:00:22.590514 (+   196us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3313026432}
I20260709 13:00:22.602392  2337 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49082 (request call id 723) took 2236 ms. Trace:
I20260709 13:00:22.603188  2337 rpcz_store.cc:276] 0709 13:00:20.366229 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:20.366355 (+   126us) service_pool.cc:224] Handling call
0709 13:00:20.366718 (+   363us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:21.925437 (+1558719us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:22.602082 (+676645us) spinlock_profiling.cc:243] Waited 1.5 s on lock 0x7b54007602d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:00:22.602298 (+   216us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3458622720}
I20260709 13:00:22.555159  1644 leader_election.cc:304] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
W20260709 13:00:22.609467  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 61746.4us
I20260709 13:00:22.610404  1717 leader_election.cc:304] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:22.613685  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:22.614940  1717 leader_election.cc:304] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 13:00:22.615398  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:22.616425  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:22.617727  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:22.613152  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:22.621670  1598 raft_consensus.cc:2410] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for term 8 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
W20260709 13:00:22.514384  5087 log.cc:927] Time spent T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.112s	user 0.000s	sys 0.000s
W20260709 13:00:22.509264  5096 log.cc:927] Time spent T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.150s	user 0.000s	sys 0.000s
I20260709 13:00:22.649444  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:22.627018  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:22.651794  1599 raft_consensus.cc:3060] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Advancing to term 8
W20260709 13:00:22.656461  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 101436us
I20260709 13:00:22.662096  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:22.663813  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:22.670835  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:22.672350  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:22.671799  1605 raft_consensus.cc:2468] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 8.
I20260709 13:00:22.471318  5060 leader_election.cc:290] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:22.590965  5053 raft_consensus.cc:493] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:22.674131  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:00:22.627156  5090 log.cc:927] Time spent T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.410s	user 0.000s	sys 0.001s
I20260709 13:00:22.608609  5011 raft_consensus.cc:493] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:00:22.712739  5099 log.cc:927] Time spent T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.066s	user 0.000s	sys 0.000s
I20260709 13:00:22.717597  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:22.475965  5079 raft_consensus.cc:515] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:22.728973  1747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55840 (request call id 2278) took 2374 ms. Trace:
I20260709 13:00:22.735740  1747 rpcz_store.cc:276] 0709 13:00:20.354997 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:21.291017 (+936020us) service_pool.cc:224] Handling call
0709 13:00:21.291533 (+   516us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:00:21.292089 (+   556us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:00:21.292134 (+    45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:00:21.305959 (+ 13825us) log.cc:844] Serialized 26 byte log entry
0709 13:00:21.587113 (+281154us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:00:22.263223 (+676110us) log.cc:844] Serialized 14 byte log entry
0709 13:00:22.265210 (+  1987us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:22.265314 (+   104us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:00:22.728623 (+463309us) raft_consensus.cc:1685] finished
0709 13:00:22.728659 (+    36us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:22.728879 (+   220us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":682828,"raft.queue_time_us":10541,"raft.run_cpu_time_us":298,"raft.run_wall_time_us":12556,"spinlock_wait_cycles":7849216,"thread_start_us":854384,"threads_started":2,"wal-append.queue_time_us":522775}
I20260709 13:00:22.729264  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:22.736061  1643 leader_election.cc:304] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:22.738166  1604 raft_consensus.cc:2468] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 7.
W20260709 13:00:22.730963  1596 consensus_meta.cc:287] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.515s	user 0.015s	sys 0.000s
I20260709 13:00:22.732816  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:22.644387  4574 raft_consensus.cc:2804] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:00:22.647859  5084 raft_consensus.cc:493] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:00:22.608978  5048 consensus_queue.cc:1059] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.116s
I20260709 13:00:22.674801  4985 leader_election.cc:290] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:22.647030  5086 raft_consensus.cc:2749] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260709 13:00:22.747128  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 132224us
I20260709 13:00:22.624657  4629 consensus_queue.cc:1048] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.103s
I20260709 13:00:22.754851  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:22.773257  1601 raft_consensus.cc:2393] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 6: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:00:22.777992  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:22.724642  5060 raft_consensus.cc:2749] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 13:00:22.734409  4998 leader_election.cc:290] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:22.680830  5081 consensus_queue.cc:260] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 7.9, Last appended by leader: 9, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:22.722010  5053 raft_consensus.cc:515] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:00:22.819850  5071 log.cc:927] Time spent T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 1.459s	user 0.007s	sys 0.010s
I20260709 13:00:22.746008  4984 raft_consensus.cc:2749] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:00:22.842586  1679 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52766 (request call id 2312) took 2245 ms. Trace:
I20260709 13:00:22.843192  1679 rpcz_store.cc:276] 0709 13:00:20.597224 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:20.597399 (+   175us) service_pool.cc:224] Handling call
0709 13:00:20.597833 (+   434us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:00:20.598163 (+   330us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:00:20.598202 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:00:20.621432 (+ 23230us) log.cc:844] Serialized 26 byte log entry
0709 13:00:20.988104 (+366672us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:00:20.988179 (+    75us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:20.988263 (+    84us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:00:22.818850 (+1830587us) spinlock_profiling.cc:243] Waited 32 ms on lock 0x7b4400022b4c. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f4262f9bc85 00007f4262f9c6a2 00007f4265d0e1db 00007f4265d09aaf 00007f4265cfd86e 00007f4265cfcbb6 00007f4265cfc7d8 00007f4265d06221 00007f4265d061d9 00007f4265d06169
0709 13:00:22.842223 (+ 23373us) raft_consensus.cc:1685] finished
0709 13:00:22.842252 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:22.842506 (+   254us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":73645440,"thread_start_us":320619,"threads_started":1,"wal-append.queue_time_us":598019}
I20260709 13:00:22.843916  1596 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40154 (request call id 1959) took 1854 ms. Trace:
I20260709 13:00:22.846180  1596 rpcz_store.cc:276] 0709 13:00:20.989974 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:20.993372 (+  3398us) service_pool.cc:224] Handling call
0709 13:00:20.994174 (+   802us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:00:21.042112 (+ 47938us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:00:21.042149 (+    37us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:00:21.190878 (+148729us) log.cc:844] Serialized 590 byte log entry
0709 13:00:22.067170 (+876292us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:00:22.743889 (+676719us) log.cc:844] Serialized 14 byte log entry
0709 13:00:22.744207 (+   318us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:22.744318 (+   111us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:00:22.843463 (+ 99145us) raft_consensus.cc:1685] finished
0709 13:00:22.843518 (+    55us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:22.843791 (+   273us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":139263,"raft.queue_time_us":142042,"raft.run_cpu_time_us":296,"raft.run_wall_time_us":17112,"thread_start_us":777599,"threads_started":1,"wal-append.queue_time_us":1039762}
I20260709 13:00:22.734059  5108 raft_consensus.cc:493] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:22.745723  5000 leader_election.cc:290] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:22.852105  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:22.855486  1604 raft_consensus.cc:2410] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for term 7 because replica has last-logged OpId of term: 6 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 5.
W20260709 13:00:22.827833  5104 log.cc:927] Time spent T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.426s	user 0.000s	sys 0.001s
W20260709 13:00:22.825897  5097 log.cc:927] Time spent T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.324s	user 0.000s	sys 0.001s
W20260709 13:00:22.831229  5098 log.cc:927] Time spent T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.051s	user 0.000s	sys 0.000s
I20260709 13:00:22.833091  4959 leader_election.cc:290] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 7 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:22.824983  4964 leader_election.cc:290] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 9 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:00:22.829221  4954 consensus_meta.cc:287] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 1.569s	user 0.015s	sys 0.002s
I20260709 13:00:22.862393  1679 raft_consensus.cc:1275] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260709 13:00:22.817392  1764 tablet_copy_source_session.cc:215] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 13:00:22.834807  4997 raft_consensus.cc:3055] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:00:22.824139  4955 leader_election.cc:290] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:22.825239  4987 raft_consensus.cc:2749] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:00:22.737102  5011 raft_consensus.cc:515] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:22.824333  4601 leader_election.cc:290] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:22.823314  5010 leader_election.cc:290] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:00:22.895290  4846 consensus_meta.cc:287] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 1.007s	user 0.011s	sys 0.007s
W20260709 13:00:22.884303  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 272353us
I20260709 13:00:22.858347  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:22.854081  5108 raft_consensus.cc:515] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:22.864796  4948 consensus_meta.cc:287] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 1.263s	user 0.010s	sys 0.007s
W20260709 13:00:22.893994  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 157982us
I20260709 13:00:22.851380  4985 raft_consensus.cc:2804] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:00:22.781199  4980 consensus_queue.cc:1048] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.112s
W20260709 13:00:22.880676  5036 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.612s	user 0.007s	sys 0.078s
W20260709 13:00:22.939762  5036 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.674s	user 0.007s	sys 0.078s
I20260709 13:00:22.940660  5108 leader_election.cc:290] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:22.943572  1599 raft_consensus.cc:2468] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 8.
I20260709 13:00:22.883540  4997 raft_consensus.cc:740] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:00:22.890182  4954 raft_consensus.cc:2749] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:00:22.774924  1717 leader_election.cc:304] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:22.750622  4574 raft_consensus.cc:493] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:22.917484  1643 leader_election.cc:304] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:22.920565  1749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55848 (request call id 2105) took 2136 ms. Trace:
I20260709 13:00:22.928992  1597 raft_consensus.cc:2468] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 10.
W20260709 13:00:22.773041  5106 log.cc:927] Time spent T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.060s	user 0.001s	sys 0.001s
I20260709 13:00:22.938166  1764 rpcz_store.cc:275] Call kudu.tserver.TabletCopyService.BeginTabletCopySession from 127.0.0.1:43156 (request call id 595) took 2015 ms. Trace:
I20260709 13:00:22.935256  4948 raft_consensus.cc:515] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:22.933528  4985 raft_consensus.cc:493] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:22.895407  5086 raft_consensus.cc:493] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:22.880558  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:22.914217  4846 raft_consensus.cc:515] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:22.955821  1749 rpcz_store.cc:276] 0709 13:00:20.784098 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:22.269753 (+1485655us) service_pool.cc:224] Handling call
0709 13:00:22.482565 (+212812us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:22.482906 (+   341us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 13:00:22.558708 (+ 75802us) log.cc:844] Serialized 14 byte log entry
0709 13:00:22.853720 (+295012us) log.cc:844] Serialized 14 byte log entry
0709 13:00:22.919965 (+ 66245us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:00:22.920057 (+    92us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:00:22.920134 (+    77us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:22.920238 (+   104us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:22.920438 (+   200us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":19015,"prepare.queue_time_us":110732,"prepare.run_cpu_time_us":166,"prepare.run_wall_time_us":12548,"raft.queue_time_us":118208,"raft.run_cpu_time_us":286,"raft.run_wall_time_us":29196,"spinlock_wait_cycles":11643648,"thread_start_us":355783,"threads_started":2,"wal-append.queue_time_us":382587}
W20260709 13:00:22.956050  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 181351us
I20260709 13:00:22.995683  1717 leader_election.cc:304] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:22.959440  1643 leader_election.cc:304] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:22.953275  1751 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55848 (request call id 2104) took 2195 ms. Trace:
I20260709 13:00:22.954804  4537 raft_consensus.cc:493] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:22.773658  5084 raft_consensus.cc:515] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:22.951211  1746 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55840 (request call id 2276) took 2605 ms. Trace:
W20260709 13:00:22.991127  4897 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.461s	user 0.023s	sys 0.045s
I20260709 13:00:22.997790  1751 rpcz_store.cc:276] 0709 13:00:20.757651 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:22.131691 (+1374040us) service_pool.cc:224] Handling call
0709 13:00:22.132165 (+   474us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:22.132472 (+   307us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 13:00:22.495658 (+363186us) log.cc:844] Serialized 14 byte log entry
0709 13:00:22.952835 (+457177us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:00:22.952902 (+    67us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:00:22.952967 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:22.953069 (+   102us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:22.953201 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":168582,"prepare.run_cpu_time_us":110,"prepare.run_wall_time_us":47799,"raft.queue_time_us":89,"thread_start_us":784958,"threads_started":2}
I20260709 13:00:22.992386  1605 raft_consensus.cc:2410] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for term 7 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260709 13:00:22.974496  4574 raft_consensus.cc:3060] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:00:23.008442  5011 leader_election.cc:290] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:23.007388  1746 rpcz_store.cc:276] 0709 13:00:20.345973 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:21.272901 (+926928us) service_pool.cc:224] Handling call
0709 13:00:21.306586 (+ 33685us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:00:21.483669 (+177083us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:00:21.483724 (+    55us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:00:21.495415 (+ 11691us) log.cc:844] Serialized 26 byte log entry
0709 13:00:22.217309 (+721894us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:00:22.583942 (+366633us) log.cc:844] Serialized 14 byte log entry
0709 13:00:22.888801 (+304859us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:22.888910 (+   109us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:00:22.950968 (+ 62058us) raft_consensus.cc:1685] finished
0709 13:00:22.950996 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:22.951135 (+   139us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":100454,"prepare.queue_time_us":141207,"prepare.run_cpu_time_us":52,"prepare.run_wall_time_us":5264,"raft.queue_time_us":295690,"raft.run_cpu_time_us":463,"raft.run_wall_time_us":49567,"thread_start_us":1366294,"threads_started":4,"wal-append.queue_time_us":794836}
I20260709 13:00:22.986884  5108 raft_consensus.cc:493] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:22.781200  4547 consensus_queue.cc:237] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:22.981860  1764 rpcz_store.cc:276] 0709 13:00:20.964993 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:21.089064 (+124071us) service_pool.cc:224] Handling call
0709 13:00:22.938035 (+1848971us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:00:22.973829  5123 raft_consensus.cc:2749] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:00:22.992956  4985 raft_consensus.cc:3060] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:00:22.975049  5128 raft_consensus.cc:493] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:00:22.985841  5120 raft_consensus.cc:2749] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260709 13:00:23.011334  4897 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.487s	user 0.023s	sys 0.045s
I20260709 13:00:23.005098  5010 raft_consensus.cc:493] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:23.011744  5060 raft_consensus.cc:3060] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:00:23.010169  4537 raft_consensus.cc:515] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:23.013253  5032 consensus_queue.cc:260] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:23.028218  5053 leader_election.cc:290] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:22.997483  5086 raft_consensus.cc:515] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:23.036077  5123 raft_consensus.cc:493] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:00:23.043900  4846 leader_election.cc:290] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 7 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:23.046816  5032 raft_consensus.cc:3060] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:00:23.023641  5108 raft_consensus.cc:515] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:23.043563  5123 raft_consensus.cc:515] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:23.056304  4948 leader_election.cc:290] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 7 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:23.060055  5137 raft_consensus.cc:2749] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
W20260709 13:00:23.062103  4887 consensus_meta.cc:287] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.839s	user 0.013s	sys 0.004s
I20260709 13:00:23.036012  5010 raft_consensus.cc:515] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:23.069592  5079 leader_election.cc:290] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:23.069800  4802 leader_election.cc:290] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:23.032259  5128 raft_consensus.cc:515] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:23.021077  4997 consensus_queue.cc:260] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:23.092623  5084 leader_election.cc:290] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:22.992143  1603 raft_consensus.cc:3060] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:00:23.006022  1600 raft_consensus.cc:1240] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 6. Current term is 7. Ops: []
I20260709 13:00:22.999588  1601 raft_consensus.cc:2468] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 7.
I20260709 13:00:23.207515  4956 raft_consensus.cc:493] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:23.210827  4846 raft_consensus.cc:493] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:23.212026  4846 raft_consensus.cc:515] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:23.095098  5086 leader_election.cc:290] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:23.088172  5010 leader_election.cc:290] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:23.216919  4948 raft_consensus.cc:493] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:23.218052  4948 raft_consensus.cc:515] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:23.220141  4952 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.179s	user 0.003s	sys 0.018s
W20260709 13:00:23.220834  4952 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.192s	user 0.003s	sys 0.018s
I20260709 13:00:23.209491  4956 raft_consensus.cc:515] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:23.225828  4956 leader_election.cc:290] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:23.227774  4846 leader_election.cc:290] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:00:23.235834  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 240108us
I20260709 13:00:23.237670  1717 leader_election.cc:304] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:00:23.237604  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:23.233970  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:23.229666  4997 raft_consensus.cc:3060] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:00:23.100076  5128 leader_election.cc:290] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:23.244977  1596 raft_consensus.cc:2468] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 7.
I20260709 13:00:23.245137  1643 leader_election.cc:304] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:00:23.100016  5123 leader_election.cc:290] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:23.248736  1671 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57092 (request call id 722) took 2444 ms. Trace:
I20260709 13:00:23.099697  5108 leader_election.cc:290] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:23.222026  4948 leader_election.cc:290] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:23.249352  1671 rpcz_store.cc:276] 0709 13:00:20.804033 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:21.025563 (+221530us) service_pool.cc:224] Handling call
0709 13:00:21.026084 (+   521us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:21.783880 (+757796us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:23.247927 (+1464047us) spinlock_profiling.cc:243] Waited 745 ms on lock 0x7b5400090954. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:00:23.248446 (+   519us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1712778624}
W20260709 13:00:23.228685  1570 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:23.171751  1680 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52778 (request call id 1880) took 2117 ms. Trace:
I20260709 13:00:23.254426  1672 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57092 (request call id 726) took 2345 ms. Trace:
I20260709 13:00:23.162357  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:23.264288  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:23.265612  5128 raft_consensus.cc:2749] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:00:23.229626  4872 consensus_queue.cc:260] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:23.267997  1677 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52766 (request call id 2310) took 2958 ms. Trace:
I20260709 13:00:23.256690  1672 rpcz_store.cc:276] 0709 13:00:20.908861 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:21.907469 (+998608us) service_pool.cc:224] Handling call
0709 13:00:21.907829 (+   360us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:22.261080 (+353251us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:23.253650 (+992570us) spinlock_profiling.cc:243] Waited 351 ms on lock 0x7b540025add4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:00:23.254174 (+   524us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":806801664}
I20260709 13:00:23.268586  1677 rpcz_store.cc:276] 0709 13:00:20.309268 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:20.309441 (+   173us) service_pool.cc:224] Handling call
0709 13:00:20.314717 (+  5276us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:20.961147 (+646430us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:23.267252 (+2306105us) spinlock_profiling.cc:243] Waited 596 ms on lock 0x7b5400164254. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:00:23.267715 (+   463us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1370005760}
I20260709 13:00:23.274156  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:23.271936  1679 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52778 (request call id 1886) took 1876 ms. Trace:
I20260709 13:00:23.276499  1717 leader_election.cc:304] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:23.276389  1745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55848 (request call id 2103) took 2564 ms. Trace:
I20260709 13:00:23.280964  1679 rpcz_store.cc:276] 0709 13:00:21.394989 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:22.844907 (+1449918us) service_pool.cc:224] Handling call
0709 13:00:22.845256 (+   349us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:22.882517 (+ 37261us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:23.271224 (+388707us) spinlock_profiling.cc:243] Waited 14.8 ms on lock 0x7b5400162e54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:00:23.271631 (+   407us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":34148096}
I20260709 13:00:23.270510  1672 raft_consensus.cc:1240] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 7. Current term is 8. Ops: [7.8-7.8]
I20260709 13:00:23.282951  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:23.281652  1745 rpcz_store.cc:276] 0709 13:00:20.711662 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:21.696103 (+984441us) service_pool.cc:224] Handling call
0709 13:00:21.714875 (+ 18772us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:00:21.842712 (+127837us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:00:21.842772 (+    60us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:00:21.857045 (+ 14273us) log.cc:844] Serialized 26 byte log entry
0709 13:00:22.295878 (+438833us) raft_consensus.cc:1645] Marking committed up to 3
0709 13:00:22.295940 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:23.275332 (+979392us) spinlock_profiling.cc:243] Waited 108 ms on lock 0x7b5400322dd4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:00:23.275432 (+   100us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:00:23.275540 (+   108us) raft_consensus.cc:1685] finished
0709 13:00:23.275568 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:23.276053 (+   485us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":8842,"raft.run_cpu_time_us":236,"raft.run_wall_time_us":20582,"spinlock_wait_cycles":248315776,"thread_start_us":405524,"threads_started":1,"wal-append.queue_time_us":619720}
I20260709 13:00:23.283618  1672 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57092 (request call id 729) took 1653 ms. Trace:
I20260709 13:00:23.271071  1676 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57092 (request call id 714) took 2469 ms. Trace:
I20260709 13:00:23.284488  1676 rpcz_store.cc:276] 0709 13:00:20.801857 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:20.806396 (+  4539us) service_pool.cc:224] Handling call
0709 13:00:20.806840 (+   444us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:21.013203 (+206363us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:23.262912 (+2249709us) spinlock_profiling.cc:243] Waited 192 ms on lock 0x7b54000fe454. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:00:23.270827 (+  7915us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":442226432}
I20260709 13:00:23.284049  1672 rpcz_store.cc:276] 0709 13:00:21.639895 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:23.269490 (+1629595us) service_pool.cc:224] Handling call
0709 13:00:23.270016 (+   526us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:00:23.283242 (+ 13226us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:23.283521 (+   279us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:00:23.271265  1674 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52766 (request call id 2317) took 2333 ms. Trace:
I20260709 13:00:23.271773  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:23.289825  1674 rpcz_store.cc:276] 0709 13:00:20.925067 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:21.911217 (+986150us) service_pool.cc:224] Handling call
0709 13:00:21.911561 (+   344us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:22.079444 (+167883us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:00:22.079497 (+    53us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:00:22.079548 (+    51us) raft_consensus.cc:1645] Marking committed up to 3
0709 13:00:22.079581 (+    33us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:23.258053 (+1178472us) spinlock_profiling.cc:243] Waited 168 ms on lock 0x7b54001f4d54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:00:23.258129 (+    76us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:23.258393 (+   264us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":385343872}
I20260709 13:00:23.256549  1680 rpcz_store.cc:276] 0709 13:00:21.054219 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:21.931356 (+877137us) service_pool.cc:224] Handling call
0709 13:00:21.931928 (+   572us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:00:21.932496 (+   568us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:00:21.932540 (+    44us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:00:21.933655 (+  1115us) log.cc:844] Serialized 26 byte log entry
0709 13:00:22.286080 (+352425us) raft_consensus.cc:1645] Marking committed up to 3
0709 13:00:22.286140 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:22.286227 (+    87us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:00:23.170876 (+884649us) raft_consensus.cc:1685] finished
0709 13:00:23.170972 (+    96us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:23.171449 (+   477us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":142,"raft.run_cpu_time_us":123,"raft.run_wall_time_us":124,"thread_start_us":350955,"threads_started":1,"wal-append.queue_time_us":711809}
I20260709 13:00:23.298596  5123 raft_consensus.cc:2749] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20260709 13:00:23.305737  1646 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:00:23.307367  1646 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:23.308825  1600 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 633) took 1810 ms. Trace:
I20260709 13:00:23.301940  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
W20260709 13:00:23.316252  5139 log.cc:927] Time spent T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.054s	user 0.000s	sys 0.002s
W20260709 13:00:23.322389  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 77826.2us
I20260709 13:00:23.324118  1680 raft_consensus.cc:1275] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260709 13:00:23.312726  1600 rpcz_store.cc:276] 0709 13:00:21.497875 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:21.498061 (+   186us) service_pool.cc:224] Handling call
0709 13:00:21.498472 (+   411us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:23.289567 (+1791095us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:23.308404 (+ 18837us) spinlock_profiling.cc:243] Waited 1.51 s on lock 0x7b5400214dd4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:00:23.308615 (+   211us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3465957248}
I20260709 13:00:23.327903  1678 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52766 (request call id 2319) took 1886 ms. Trace:
I20260709 13:00:23.328424  1678 rpcz_store.cc:276] 0709 13:00:21.404325 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:23.269192 (+1864867us) service_pool.cc:224] Handling call
0709 13:00:23.269890 (+   698us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:23.270409 (+   519us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:00:23.279358 (+  8949us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:00:23.279422 (+    64us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:00:23.279479 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:23.288320 (+  8841us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:23.290122 (+  1802us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:00:23.329839  5028 raft_consensus.cc:3055] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:00:23.330387  5028 raft_consensus.cc:740] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:00:23.330897  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:23.324910  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:23.334412  1604 raft_consensus.cc:2410] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for term 10 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20260709 13:00:23.335423  4995 raft_consensus.cc:493] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:23.336445  4995 raft_consensus.cc:515] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:23.330251  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:23.360383  1643 leader_election.cc:304] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 13:00:23.363363  1671 raft_consensus.cc:2468] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 10.
I20260709 13:00:23.332167  5028 consensus_queue.cc:260] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:23.366058  5143 raft_consensus.cc:2804] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260709 13:00:23.375523  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 99288us
I20260709 13:00:23.379120  5017 raft_consensus.cc:2804] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Leader election won for term 10
I20260709 13:00:23.380129  5017 raft_consensus.cc:697] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:00:23.382970  2334 raft_consensus.cc:2468] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 8.
W20260709 13:00:23.388094  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 46391us
I20260709 13:00:23.388443  4648 consensus_queue.cc:1059] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 13:00:23.389516  1643 leader_election.cc:304] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:00:23.390358  4948 raft_consensus.cc:3055] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:00:23.381604  5143 raft_consensus.cc:493] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:23.393285  4948 raft_consensus.cc:740] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:00:23.393718  4746 raft_consensus.cc:2804] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:00:23.398370  4746 raft_consensus.cc:493] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:23.399915  4746 raft_consensus.cc:3060] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Advancing to term 8
W20260709 13:00:23.403287  5134 log.cc:927] Time spent T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.373s	user 0.003s	sys 0.001s
I20260709 13:00:23.404285  5028 raft_consensus.cc:3060] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Advancing to term 7
W20260709 13:00:23.409710  1718 connection.cc:256] RPC call timeout handler was delayed by -1.7099097339999503s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1501s for a response
I20260709 13:00:23.410575  5017 consensus_queue.cc:237] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.9, Last appended by leader: 9, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:23.413100  5123 raft_consensus.cc:493] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:00:23.414103  5123 raft_consensus.cc:515] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:23.417057  5123 leader_election.cc:290] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:23.418592  4759 raft_consensus.cc:3055] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:00:23.419164  4759 raft_consensus.cc:740] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:00:23.391923  1646 leader_election.cc:304] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:00:23.421283  5127 raft_consensus.cc:2749] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20260709 13:00:23.423238  1645 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:23.425079  5095 raft_consensus.cc:493] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:23.426023  5095 raft_consensus.cc:515] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:23.428073  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 76016.6us
I20260709 13:00:23.427917  5086 consensus_queue.cc:1048] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 13:00:23.430074  1571 leader_election.cc:304] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:00:23.420711  4759 consensus_queue.cc:260] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:23.432041  4648 consensus_queue.cc:1059] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 13:00:23.433291  1717 leader_election.cc:304] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:00:23.433768  5108 raft_consensus.cc:3055] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:00:23.434325  5108 raft_consensus.cc:740] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:00:23.434839  4875 raft_consensus.cc:493] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:23.435565  4875 raft_consensus.cc:515] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:23.435813  5108 consensus_queue.cc:260] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:23.437943  5108 raft_consensus.cc:3060] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:00:23.432762  4759 raft_consensus.cc:3060] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:00:23.440544  5095 leader_election.cc:290] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:23.431753  5145 consensus_queue.cc:1059] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.017s
I20260709 13:00:23.395756  5143 raft_consensus.cc:3060] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:00:23.453002  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:00:23.457888  1645 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:23.450920  1598 raft_consensus.cc:2468] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 5.
I20260709 13:00:23.437981  4512 raft_consensus.cc:3060] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:00:23.463238  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:23.463392  4875 leader_election.cc:290] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:23.464569  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:23.465415  1674 raft_consensus.cc:3060] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Advancing to term 7
W20260709 13:00:23.470537  5144 log.cc:927] Time spent T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.076s	user 0.000s	sys 0.000s
I20260709 13:00:23.470954  4802 raft_consensus.cc:2804] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:00:23.471447  4802 raft_consensus.cc:493] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:23.471828  4802 raft_consensus.cc:3060] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Advancing to term 9
W20260709 13:00:23.455084  5048 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.536s	user 0.011s	sys 0.006s
W20260709 13:00:23.472385  5048 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.645s	user 0.011s	sys 0.006s
W20260709 13:00:23.473116  1718 connection.cc:256] RPC call timeout handler was delayed by -1.4021097339999642s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18430000000000002s for a response
W20260709 13:00:23.473575  1718 connection.cc:256] RPC call timeout handler was delayed by -1.2968097339999645s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.196s for a response
W20260709 13:00:23.473901  1718 connection.cc:256] RPC call timeout handler was delayed by -1.2914097339999557s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1966s for a response
I20260709 13:00:23.474750  5123 raft_consensus.cc:493] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
W20260709 13:00:23.475102  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:00:23.474228  1718 connection.cc:256] RPC call timeout handler was delayed by -1.0302713639999865s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2016s for a response
I20260709 13:00:23.409725  4683 leader_election.cc:290] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:23.475615  5123 raft_consensus.cc:515] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:23.478583  1717 leader_election.cc:304] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:00:23.428073  4948 consensus_queue.cc:260] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:23.479954  5079 raft_consensus.cc:2804] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:00:23.480404  5079 raft_consensus.cc:493] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:00:23.480784  5079 raft_consensus.cc:3060] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:00:23.480878  5123 leader_election.cc:290] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:23.483413  4985 raft_consensus.cc:515] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:23.484614  4628 consensus_queue.cc:1059] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 13:00:23.485862  4628 raft_consensus.cc:3055] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:00:23.487154  4580 leader_election.cc:290] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:23.490413  1675 raft_consensus.cc:2468] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 10.
I20260709 13:00:23.486073  4985 leader_election.cc:290] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:23.492486  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:23.497298  1680 raft_consensus.cc:2393] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 8: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:00:23.498514  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:23.486320  4628 raft_consensus.cc:740] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:00:23.499691  1680 raft_consensus.cc:2410] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for term 6 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260709 13:00:23.500551  5048 raft_consensus.cc:2749] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:00:23.500909  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:23.501835  1680 raft_consensus.cc:2468] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 5.
I20260709 13:00:23.502808  4984 raft_consensus.cc:3055] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:00:23.503289  1748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55848 (request call id 2107) took 2453 ms. Trace:
I20260709 13:00:23.503419  4984 raft_consensus.cc:740] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:00:23.505385  4984 consensus_queue.cc:260] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:23.511246  4683 raft_consensus.cc:493] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:23.512266  4683 raft_consensus.cc:515] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:00:23.487188  5109 log.cc:927] Time spent T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.657s	user 0.002s	sys 0.000s
I20260709 13:00:23.519424  4984 raft_consensus.cc:3060] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:00:23.520328  4648 raft_consensus.cc:493] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:23.521098  4648 raft_consensus.cc:515] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:23.523918  4648 leader_election.cc:290] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:23.524686  4951 raft_consensus.cc:3060] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:00:23.524588  5011 consensus_queue.cc:1048] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 13:00:23.523921  5036 raft_consensus.cc:493] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:00:23.527100  1718 leader_election.cc:304] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:23.527240  5036 raft_consensus.cc:515] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:23.528630  5145 consensus_queue.cc:1059] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 13:00:23.529116  5048 consensus_queue.cc:1059] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 13:00:23.529771  5145 consensus_queue.cc:1048] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 13:00:23.527498  4846 raft_consensus.cc:493] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:23.531224  4846 raft_consensus.cc:515] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:23.531745  4648 consensus_queue.cc:1048] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 13:00:23.514259  4537 leader_election.cc:290] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:23.548879  1674 raft_consensus.cc:2468] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 7.
I20260709 13:00:23.503855  1748 rpcz_store.cc:276] 0709 13:00:21.049350 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:22.547174 (+1497824us) service_pool.cc:224] Handling call
0709 13:00:22.586362 (+ 39188us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:22.586762 (+   400us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 13:00:22.654721 (+ 67959us) log.cc:844] Serialized 14 byte log entry
0709 13:00:22.975362 (+320641us) log.cc:844] Serialized 14 byte log entry
0709 13:00:23.479812 (+504450us) spinlock_profiling.cc:243] Waited 1.58 ms on lock 0x7b1800090fa0. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f4263075522 00007f42630777c1 00007f4263077779 00007f4263077709 00007f42630776d1 00007f42630769cd 000000000038f354 000000000038f199 00007f426305883a
0709 13:00:23.502148 (+ 22336us) spinlock_profiling.cc:243] Waited 3.51 ms on lock 0x7b1800090fa0. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f4265d0a4bc 00007f426305848b 00007f426beb2ab9 00007f426307469b 00007f426307293f 00007f4263072455 00007f426615a060 00007f426616c716 00007f426617d287
0709 13:00:23.502324 (+   176us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:00:23.502464 (+   140us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:00:23.502656 (+   192us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:23.502818 (+   162us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:23.503026 (+   208us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":104821,"prepare.queue_time_us":89925,"prepare.run_cpu_time_us":100,"prepare.run_wall_time_us":102,"raft.queue_time_us":72119,"spinlock_wait_cycles":21932032,"thread_start_us":324880,"threads_started":2,"wal-append.queue_time_us":343725}
I20260709 13:00:23.535673  4846 leader_election.cc:290] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:23.552444  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:23.476771  5127 consensus_queue.cc:1059] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.067s
I20260709 13:00:23.536103  5036 leader_election.cc:290] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:23.533617  4648 consensus_queue.cc:1048] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.076s
I20260709 13:00:23.612304  5123 raft_consensus.cc:2749] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 13:00:23.554718  4948 raft_consensus.cc:3060] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Advancing to term 10
W20260709 13:00:23.644193  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 214511us
W20260709 13:00:23.636765  4981 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.372s	user 0.014s	sys 0.010s
I20260709 13:00:23.649128  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:23.652231  1603 raft_consensus.cc:2468] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 6.
I20260709 13:00:23.652321  5123 raft_consensus.cc:493] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:23.653378  5123 raft_consensus.cc:515] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:23.656235  5123 leader_election.cc:290] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:23.658742  5060 raft_consensus.cc:2749] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260709 13:00:23.658414  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 140484us
I20260709 13:00:23.661291  1718 leader_election.cc:304] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:00:23.663858  1677 raft_consensus.cc:1240] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 6. Current term is 7. Ops: []
I20260709 13:00:23.612521  4683 leader_election.cc:290] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:23.553941  4956 consensus_queue.cc:1059] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.123s
W20260709 13:00:23.666505  4981 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.535s	user 0.014s	sys 0.011s
I20260709 13:00:23.662469  5123 raft_consensus.cc:2749] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:00:23.674757  4978 raft_consensus.cc:493] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:00:23.685868  1571 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:23.696861  4995 leader_election.cc:290] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:23.697654  4995 raft_consensus.cc:493] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:00:23.698419  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 51750.5us
I20260709 13:00:23.699821  1673 raft_consensus.cc:1240] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 LEADER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 7. Current term is 8. Ops: []
W20260709 13:00:23.702148  5032 consensus_meta.cc:287] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.649s	user 0.012s	sys 0.013s
I20260709 13:00:23.704463  5084 raft_consensus.cc:493] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
W20260709 13:00:23.714419  2301 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:23.714951  1753 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43156 (request call id 589) took 3341 ms. Trace:
I20260709 13:00:23.716796  4985 raft_consensus.cc:493] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:23.609375  4628 consensus_queue.cc:260] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:23.698213  4995 raft_consensus.cc:515] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:23.678157  4802 raft_consensus.cc:515] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:23.726753  2301 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:23.715926  4978 raft_consensus.cc:515] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:23.727061  1753 rpcz_store.cc:276] 0709 13:00:20.373032 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:21.518069 (+1145037us) service_pool.cc:224] Handling call
0709 13:00:21.568110 (+ 50041us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:21.635025 (+ 66915us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:23.676014 (+2040989us) spinlock_profiling.cc:243] Waited 34.8 ms on lock 0x7b54003700d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:00:23.714703 (+ 38689us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":80137600}
W20260709 13:00:23.738700  1646 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:00:23.744079  2301 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:23.756218  5150 raft_consensus.cc:493] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
W20260709 13:00:23.756541  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 69188.1us
I20260709 13:00:23.757207  5150 raft_consensus.cc:515] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:23.760926  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 99820.2us
I20260709 13:00:23.749563  5123 raft_consensus.cc:493] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:23.760399  4990 raft_consensus.cc:493] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:00:23.774596  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 136183us
W20260709 13:00:23.778882  2301 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:23.778206  5084 raft_consensus.cc:515] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:23.783152  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:23.783394  5149 raft_consensus.cc:493] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:23.784057  1748 raft_consensus.cc:2393] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 9: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:00:23.783322  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:23.786363  1746 raft_consensus.cc:2393] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 10: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:00:23.786276  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:23.786715  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:23.787177  1745 raft_consensus.cc:2468] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 5.
I20260709 13:00:23.787451  1754 raft_consensus.cc:3060] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:00:23.782382  5084 leader_election.cc:290] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:23.791126  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:23.791517  4628 raft_consensus.cc:3060] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:00:23.791952  1750 raft_consensus.cc:3060] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:00:23.803092  4990 raft_consensus.cc:515] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:23.805402  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
W20260709 13:00:23.805718  2334 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39726 (request call id 884) took 1645 ms (client timeout 2051 ms). Trace:
W20260709 13:00:23.806185  2334 rpcz_store.cc:269] 0709 13:00:22.162698 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:22.162939 (+   241us) service_pool.cc:224] Handling call
0709 13:00:23.805433 (+1642494us) spinlock_profiling.cc:243] Waited 1.04 s on lock 0x7b54007d2654. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:00:23.805593 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2387783552}
I20260709 13:00:23.793795  4802 leader_election.cc:290] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 9 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:00:23.807570  1646 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:00:23.807996  1645 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:23.810274  1750 raft_consensus.cc:2468] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 7.
I20260709 13:00:23.816128  1718 leader_election.cc:304] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:00:23.823829  5123 raft_consensus.cc:515] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:23.824838  4512 consensus_queue.cc:1059] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 13:00:23.827631  4995 leader_election.cc:290] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:23.831470  5150 leader_election.cc:290] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:23.765379  4985 raft_consensus.cc:515] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:23.849500  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:23.854508  1675 raft_consensus.cc:2468] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 8.
I20260709 13:00:23.854638  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:00:23.855808  4574 consensus_meta.cc:287] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.714s	user 0.015s	sys 0.006s
I20260709 13:00:23.856303  4574 raft_consensus.cc:515] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:00:23.859937  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:23.861667  2332 raft_consensus.cc:2468] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 6.
I20260709 13:00:23.867799  4978 leader_election.cc:290] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:23.784238  5149 raft_consensus.cc:515] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:23.868266  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:23.874049  1675 raft_consensus.cc:2468] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 8.
W20260709 13:00:23.874981  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 100456us
I20260709 13:00:23.876855  1718 leader_election.cc:304] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:23.878746  1753 raft_consensus.cc:3060] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:00:23.879961  4949 consensus_queue.cc:1048] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 13:00:23.881839  4759 raft_consensus.cc:3055] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:00:23.882459  4759 raft_consensus.cc:740] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:00:23.883315  4802 raft_consensus.cc:2749] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:00:23.884382  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:23.883908  4759 consensus_queue.cc:260] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.9, Last appended by leader: 9, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:23.885682  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:23.888602  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:23.889325  1747 raft_consensus.cc:2468] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 6.
I20260709 13:00:23.890493  4537 raft_consensus.cc:3055] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:00:23.891049  4537 raft_consensus.cc:740] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:00:23.892897  4537 consensus_queue.cc:260] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:23.894891  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 43738.6us
I20260709 13:00:23.895602  1600 raft_consensus.cc:1240] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 6. Current term is 7. Ops: [6.5-6.5]
I20260709 13:00:23.896332  1600 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 2061) took 1262 ms. Trace:
I20260709 13:00:23.896939  5079 raft_consensus.cc:515] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:00:23.897915  1603 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40154 (request call id 1965) took 2898 ms (client timeout 2126 ms). Trace:
W20260709 13:00:23.898336  1603 rpcz_store.cc:269] 0709 13:00:20.999966 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:21.000107 (+   141us) service_pool.cc:224] Handling call
0709 13:00:23.897697 (+2897590us) spinlock_profiling.cc:243] Waited 1.99 s on lock 0x7b540006dbd4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:00:23.897823 (+   126us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4578344192}
I20260709 13:00:23.899729  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:23.901077  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:23.901629  1603 raft_consensus.cc:2468] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 8.
I20260709 13:00:23.891093  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:23.772517  4746 raft_consensus.cc:515] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:23.886332  1750 raft_consensus.cc:3060] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:00:23.878127  1718 leader_election.cc:304] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:00:23.904592  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:23.902408  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:23.896656  1600 rpcz_store.cc:276] 0709 13:00:22.633463 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:23.894868 (+1261405us) service_pool.cc:224] Handling call
0709 13:00:23.895312 (+   444us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:00:23.896030 (+   718us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:23.896252 (+   222us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:00:23.918978  1680 raft_consensus.cc:2393] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 8: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:00:23.922071  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:23.921023  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:00:23.921080  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 57281.8us
I20260709 13:00:23.899740  4512 consensus_queue.cc:1059] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.016s
I20260709 13:00:23.924317  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:00:23.921195  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 56496.4us
I20260709 13:00:23.909940  4759 raft_consensus.cc:3060] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:00:23.930292  1747 raft_consensus.cc:3060] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:00:23.932328  5129 raft_consensus.cc:2804] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 13:00:23.933945  5083 raft_consensus.cc:493] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:23.930143  4512 raft_consensus.cc:3055] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 7 LEADER]: Stepping down as leader of term 7
W20260709 13:00:23.934505  5146 log.cc:927] Time spent T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.258s	user 0.001s	sys 0.001s
I20260709 13:00:23.934667  5083 raft_consensus.cc:515] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:23.936935  1674 raft_consensus.cc:1240] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 6 LEADER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 5. Current term is 6. Ops: []
I20260709 13:00:23.933959  5007 ts_tablet_manager.cc:1916] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1: Deleting tablet data with delete state TABLET_DATA_COPYING
I20260709 13:00:23.917506  5123 leader_election.cc:290] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:23.946878  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:23.927235  4574 leader_election.cc:290] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 7 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:23.972080  1748 raft_consensus.cc:2468] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 8.
I20260709 13:00:23.979175  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:23.934721  4512 raft_consensus.cc:740] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:00:23.984401  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:23.939519  4990 leader_election.cc:290] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:00:23.989259  5085 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.158s	user 0.006s	sys 0.013s
I20260709 13:00:23.989184  4900 raft_consensus.cc:493] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:23.990228  1753 raft_consensus.cc:2468] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 6.
I20260709 13:00:23.946357  4746 leader_election.cc:290] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 8 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:23.994046  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:23.995764  1600 raft_consensus.cc:3060] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Advancing to term 9
W20260709 13:00:23.996292  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 67819.8us
I20260709 13:00:23.999984  1754 raft_consensus.cc:2468] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 7.
I20260709 13:00:23.971199  5129 raft_consensus.cc:697] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [term 6 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
W20260709 13:00:23.997161  5085 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.168s	user 0.007s	sys 0.013s
I20260709 13:00:24.004652  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:00:24.008311  1645 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:24.001890  1717 leader_election.cc:304] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:00:24.012573  1676 raft_consensus.cc:1275] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 13:00:24.016320  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:24.016908  1754 raft_consensus.cc:3060] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:00:24.019990  1748 raft_consensus.cc:2468] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 7.
I20260709 13:00:24.021685  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:24.022477  1718 leader_election.cc:304] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:00:23.997309  4900 raft_consensus.cc:515] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:24.024255  2337 raft_consensus.cc:1240] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 8. Current term is 9. Ops: []
I20260709 13:00:24.011273  4951 raft_consensus.cc:493] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:24.032554  2334 raft_consensus.cc:1240] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 6. Current term is 7. Ops: []
I20260709 13:00:24.013414  4985 leader_election.cc:290] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:24.034710  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:24.037067  5094 raft_consensus.cc:493] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:00:24.037447  2337 raft_consensus.cc:1275] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260709 13:00:24.037806  5094 raft_consensus.cc:515] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:24.039911  1643 leader_election.cc:336] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 2.607s (SENT)
I20260709 13:00:24.046030  1750 raft_consensus.cc:2468] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 6.
I20260709 13:00:24.046873  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:24.047585  1569 leader_election.cc:304] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:00:24.048727  1746 raft_consensus.cc:2468] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 8.
I20260709 13:00:24.049803  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:24.052726  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:24.018690  4847 consensus_queue.cc:1048] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.035s
I20260709 13:00:24.042268  4779 raft_consensus.cc:493] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:00:24.055249  5145 raft_consensus.cc:3055] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:00:24.057808  5145 raft_consensus.cc:740] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:00:24.059327  1746 raft_consensus.cc:1240] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 6. Current term is 7. Ops: [4.6-6.7]
I20260709 13:00:24.058940  5145 consensus_queue.cc:260] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:24.060523  5145 raft_consensus.cc:3060] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:00:24.061707  1746 raft_consensus.cc:1240] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 6. Current term is 7. Ops: []
I20260709 13:00:24.058367  2330 raft_consensus.cc:1275] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
W20260709 13:00:24.066220  2332 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39726 (request call id 887) took 1900 ms (client timeout 1720 ms). Trace:
W20260709 13:00:24.066720  2332 rpcz_store.cc:269] 0709 13:00:22.165873 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:22.167837 (+  1964us) service_pool.cc:224] Handling call
0709 13:00:24.065989 (+1898152us) spinlock_profiling.cc:243] Waited 1.55 s on lock 0x7b540072e554. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:00:24.066138 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3558277888}
I20260709 13:00:24.047266  4574 raft_consensus.cc:2804] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:00:24.069216  1748 raft_consensus.cc:1275] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260709 13:00:24.046015  5149 leader_election.cc:290] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:24.035665  4512 consensus_queue.cc:260] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:24.075045  5129 consensus_queue.cc:237] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:24.075907  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:24.078809  5154 raft_consensus.cc:2804] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader pre-election won for term 11
W20260709 13:00:24.079035  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 56678.7us
I20260709 13:00:24.080153  1746 raft_consensus.cc:2393] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 11: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:00:24.081171  1569 leader_election.cc:304] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:00:24.082301  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:24.082999  2332 raft_consensus.cc:2468] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 10.
I20260709 13:00:24.068404  5094 leader_election.cc:290] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:24.069864  4574 raft_consensus.cc:493] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:24.081163  5154 raft_consensus.cc:493] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:24.090535  1600 raft_consensus.cc:2468] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 9.
I20260709 13:00:24.090999  1718 leader_election.cc:304] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:24.092696  5154 raft_consensus.cc:3060] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:00:24.083935  4779 raft_consensus.cc:515] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:24.072365  5028 raft_consensus.cc:493] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:24.055833  4951 raft_consensus.cc:515] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:24.100517  1569 leader_election.cc:304] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:00:24.095121  4847 raft_consensus.cc:2804] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:00:24.102672  4847 raft_consensus.cc:493] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:24.103098  4847 raft_consensus.cc:3060] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:00:24.095491  4574 raft_consensus.cc:3060] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:00:24.110239  4978 raft_consensus.cc:493] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:24.110685  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:24.112493  1751 raft_consensus.cc:1275] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260709 13:00:24.112313  5083 leader_election.cc:290] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:24.095707  5094 raft_consensus.cc:493] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:00:24.101316  5108 consensus_meta.cc:287] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.652s	user 0.013s	sys 0.002s
I20260709 13:00:24.102027  5149 raft_consensus.cc:2749] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:00:24.122205  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:24.114620  5044 raft_consensus.cc:2804] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:00:24.132220  5044 raft_consensus.cc:697] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:00:24.133738  5044 consensus_queue.cc:237] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 5.8, Last appended by leader: 8, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:24.125240  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:24.118978  4779 leader_election.cc:290] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:24.129504  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:24.129766  4512 raft_consensus.cc:3060] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:00:24.105938  5028 raft_consensus.cc:515] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:24.158932  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:24.159745  1748 raft_consensus.cc:2368] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for earlier term 6. Current term is 7.
I20260709 13:00:24.160386  4900 leader_election.cc:290] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:24.160522  5028 leader_election.cc:290] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:24.160660  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:24.130425  4978 raft_consensus.cc:515] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:24.161736  1673 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57092 (request call id 725) took 3287 ms. Trace:
I20260709 13:00:24.161948  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:24.162139  1673 rpcz_store.cc:276] 0709 13:00:20.873918 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:21.636570 (+762652us) service_pool.cc:224] Handling call
0709 13:00:21.636959 (+   389us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:23.710997 (+2074038us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:24.161450 (+450453us) spinlock_profiling.cc:243] Waited 2.06 s on lock 0x7b54001f5254. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:00:24.161668 (+   218us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4742448000}
I20260709 13:00:24.163089  4978 leader_election.cc:290] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:24.129557  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
W20260709 13:00:24.150995  1598 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40166 (request call id 2053) took 1531 ms (client timeout 1565 ms). Trace:
W20260709 13:00:24.164777  1598 rpcz_store.cc:269] 0709 13:00:22.619854 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:22.663446 (+ 43592us) service_pool.cc:224] Handling call
0709 13:00:24.150746 (+1487300us) spinlock_profiling.cc:243] Waited 638 ms on lock 0x7b54003edfd4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:00:24.150929 (+   183us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1467210496}
I20260709 13:00:24.166023  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:24.135567  4847 raft_consensus.cc:515] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:24.137233  1569 leader_election.cc:304] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:00:24.167273  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:24.143185  1600 raft_consensus.cc:2376] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 8. Re-sending same reply.
W20260709 13:00:24.167969  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:24.144951  1751 raft_consensus.cc:1240] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 7 LEADER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 6. Current term is 7. Ops: [6.5-6.5]
I20260709 13:00:24.145404  4872 raft_consensus.cc:3060] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:00:24.150179  1601 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40166 (request call id 2056) took 1520 ms. Trace:
W20260709 13:00:24.135761  1571 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.107.195:40693 timed out after 1.534s (SENT) [suppressed 26 similar messages]
I20260709 13:00:24.151590  2332 raft_consensus.cc:2468] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 7.
I20260709 13:00:24.151831  1604 raft_consensus.cc:1275] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260709 13:00:24.154241  4951 leader_election.cc:290] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:24.169669  1598 raft_consensus.cc:3060] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:00:24.170459  1748 raft_consensus.cc:3060] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Advancing to term 9
W20260709 13:00:24.172663  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:24.174937  1754 raft_consensus.cc:2468] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 7.
W20260709 13:00:24.183842  1570 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:00:24.183692  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 68023.1us
W20260709 13:00:24.184403  1570 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:00:24.184900  1570 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:24.190980  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:24.174862  1601 rpcz_store.cc:276] 0709 13:00:22.631619 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:22.858036 (+226417us) service_pool.cc:224] Handling call
0709 13:00:24.149757 (+1291721us) spinlock_profiling.cc:243] Waited 33.7 ms on lock 0x7b540027a1d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:00:24.149941 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":77593856}
I20260709 13:00:24.193567  1754 raft_consensus.cc:1240] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 7. Current term is 8. Ops: []
W20260709 13:00:24.195992  1645 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:24.199417  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:24.203747  1597 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 2058) took 1571 ms. Trace:
I20260709 13:00:24.205617  1602 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 2044) took 1595 ms. Trace:
I20260709 13:00:24.206599  1602 rpcz_store.cc:276] 0709 13:00:22.610580 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:22.610738 (+   158us) service_pool.cc:224] Handling call
0709 13:00:22.611108 (+   370us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:22.611360 (+   252us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 13:00:22.924256 (+312896us) log.cc:844] Serialized 14 byte log entry
0709 13:00:23.390755 (+466499us) spinlock_profiling.cc:243] Waited 1.84 ms on lock 0x7b4400022b4c. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f4262f9bc85 00007f4262f9c6a2 00007f4265d0e1db 00007f4265d09aaf 00007f4265cfd86e 00007f4265cfcbb6 00007f4265cfc7d8 00007f4265d06221 00007f4265d061d9 00007f4265d06169
0709 13:00:24.164483 (+773728us) spinlock_profiling.cc:243] Waited 6.02 ms on lock 0x7b180043b280. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f4262fefc0a 00007f426c72f219 00007f4263077a40 00007f42630724e4 00007f4263072455 00007f4269b5a3c7 00007f426616c9a7 00007f426617c0bf 00007f426617c038
0709 13:00:24.201274 (+ 36791us) log.cc:844] Serialized 14 byte log entry
0709 13:00:24.204674 (+  3400us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:00:24.204754 (+    80us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:00:24.204876 (+   122us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:24.205251 (+   375us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:24.205408 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":17964,"prepare.queue_time_us":1502748,"prepare.run_cpu_time_us":103,"prepare.run_wall_time_us":106,"raft.queue_time_us":178,"raft.run_cpu_time_us":290,"raft.run_wall_time_us":9322,"spinlock_wait_cycles":18065664,"thread_start_us":340540,"threads_started":3,"wal-append.queue_time_us":85680}
I20260709 13:00:24.210783  1745 raft_consensus.cc:1240] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 9. Current term is 10. Ops: [7.5-9.6]
I20260709 13:00:24.213310  1753 raft_consensus.cc:1275] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 13:00:24.215197  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:24.215386  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:24.217952  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:00:24.121680  5094 raft_consensus.cc:515] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:24.215214  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:24.224354  1676 raft_consensus.cc:2468] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 7.
I20260709 13:00:24.224924  5129 consensus_queue.cc:1048] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.004s
I20260709 13:00:24.225569  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:24.231009  2330 raft_consensus.cc:2468] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 7.
I20260709 13:00:24.231194  4847 leader_election.cc:290] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 6 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:24.232106  1677 raft_consensus.cc:1275] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 13:00:24.233111  1754 raft_consensus.cc:1275] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 13:00:24.228612  1752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43156 (request call id 593) took 3850 ms. Trace:
W20260709 13:00:24.227716  2300 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:24.227813  1675 raft_consensus.cc:1240] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 7. Current term is 8. Ops: []
W20260709 13:00:24.234294  2300 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:24.234289  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:24.237524  1673 raft_consensus.cc:1275] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260709 13:00:24.238919  1746 raft_consensus.cc:2393] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 6: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:00:24.207399  4648 raft_consensus.cc:493] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:00:24.242548  1602 raft_consensus.cc:2468] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 7.
I20260709 13:00:24.241901  1751 raft_consensus.cc:1275] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260709 13:00:24.245235  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:24.241968  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:24.243948  1718 leader_election.cc:304] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:24.246688  4847 raft_consensus.cc:493] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:00:24.249707  4847 raft_consensus.cc:515] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:00:24.250172  5045 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.369s	user 0.011s	sys 0.012s
W20260709 13:00:24.250649  5045 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.428s	user 0.012s	sys 0.012s
I20260709 13:00:24.247243  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:24.206073  1597 rpcz_store.cc:276] 0709 13:00:22.632382 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:22.959598 (+327216us) service_pool.cc:224] Handling call
0709 13:00:22.959934 (+   336us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:22.960245 (+   311us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 13:00:22.986372 (+ 26127us) log.cc:844] Serialized 14 byte log entry
0709 13:00:24.165234 (+1178862us) log.cc:844] Serialized 14 byte log entry
0709 13:00:24.202921 (+ 37687us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:00:24.203003 (+    82us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:00:24.203116 (+   113us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:24.203211 (+    95us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:24.203446 (+   235us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2305843,"prepare.queue_time_us":1037124,"prepare.run_cpu_time_us":82,"prepare.run_wall_time_us":83,"raft.queue_time_us":215378,"raft.run_cpu_time_us":434,"raft.run_wall_time_us":434,"thread_start_us":55083,"threads_started":2,"wal-append.queue_time_us":106174,"wal-append.run_cpu_time_us":2442,"wal-append.run_wall_time_us":1023791}
I20260709 13:00:24.259879  1598 raft_consensus.cc:2468] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 8.
I20260709 13:00:24.267210  5094 leader_election.cc:290] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:24.268951  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:24.270174  1598 raft_consensus.cc:2393] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 6: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:00:24.270380  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:00:24.272161  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 136413us
W20260709 13:00:24.239928  5143 consensus_meta.cc:287] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.744s	user 0.009s	sys 0.007s
I20260709 13:00:24.273312  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:24.275154  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:24.275543  1601 raft_consensus.cc:2468] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 5.
I20260709 13:00:24.276336  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:24.278620  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:24.282277  2333 raft_consensus.cc:1240] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 5. Current term is 6. Ops: []
I20260709 13:00:24.282969  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:24.286674  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:24.287505  1597 raft_consensus.cc:3060] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:00:24.287978  4978 raft_consensus.cc:493] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:00:24.288656  4978 raft_consensus.cc:515] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:24.291092  4978 leader_election.cc:290] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:24.260763  4900 raft_consensus.cc:493] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:24.295002  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:24.233594  1752 rpcz_store.cc:276] 0709 13:00:20.378225 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:21.595582 (+1217357us) service_pool.cc:224] Handling call
0709 13:00:21.596080 (+   498us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:00:21.635201 (+ 39121us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:00:21.635245 (+    44us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:00:21.635709 (+   464us) log.cc:844] Serialized 26 byte log entry
0709 13:00:22.271165 (+635456us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:00:22.301644 (+ 30479us) log.cc:844] Serialized 14 byte log entry
0709 13:00:22.302012 (+   368us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:24.227953 (+1925941us) spinlock_profiling.cc:243] Waited 6.37 ms on lock 0x7b54000e3b54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:00:24.228032 (+    79us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:00:24.228112 (+    80us) raft_consensus.cc:1685] finished
0709 13:00:24.228237 (+   125us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:24.228389 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":43601,"prepare.queue_time_us":137484,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":48,"raft.queue_time_us":1976,"raft.run_cpu_time_us":226,"raft.run_wall_time_us":51204,"spinlock_wait_cycles":14643072,"thread_start_us":581946,"threads_started":1,"wal-append.queue_time_us":1007668,"wal-append.run_cpu_time_us":4623,"wal-append.run_wall_time_us":1316099}
I20260709 13:00:24.296150  1598 raft_consensus.cc:3060] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:00:24.294809  4900 raft_consensus.cc:515] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:24.298017  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 54177.8us
I20260709 13:00:24.308837  1597 raft_consensus.cc:2468] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 9.
I20260709 13:00:24.301151  1602 raft_consensus.cc:2376] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 6. Re-sending same reply.
I20260709 13:00:24.271683  5149 raft_consensus.cc:2749] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 13:00:24.251263  4648 raft_consensus.cc:515] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:24.318823  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:24.315685  1747 raft_consensus.cc:2468] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 9.
I20260709 13:00:24.309244  1679 raft_consensus.cc:3055] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:00:24.317469  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:00:24.322963  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:24.325042  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:24.319653  1679 raft_consensus.cc:740] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:00:24.329530  2330 raft_consensus.cc:2393] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 9: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:00:24.330806  2332 raft_consensus.cc:2410] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for term 7 because replica has last-logged OpId of term: 6 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 8.
I20260709 13:00:24.331462  1747 raft_consensus.cc:1240] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 6. Current term is 7. Ops: []
I20260709 13:00:24.333626  1753 raft_consensus.cc:2393] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 8: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:00:24.335283  1718 leader_election.cc:304] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:24.336361  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:24.337898  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:24.338737  2334 raft_consensus.cc:2393] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 6: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:00:24.339982  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:00:24.340605  1646 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:24.342176  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:24.321825  5123 raft_consensus.cc:2749] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:00:24.322608  5110 raft_consensus.cc:3055] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:00:24.298588  5143 raft_consensus.cc:515] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:24.359050  1598 raft_consensus.cc:2468] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 7.
I20260709 13:00:24.300420  5154 raft_consensus.cc:515] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:24.356621  1677 raft_consensus.cc:2468] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 6.
I20260709 13:00:24.357636  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:00:24.382542  1645 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:24.387636  1596 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 2059) took 1726 ms. Trace:
I20260709 13:00:24.341055  5149 raft_consensus.cc:2749] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:00:24.377903  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:00:24.390249  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 44861.3us
I20260709 13:00:24.391083  4896 raft_consensus.cc:2804] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:00:24.392997  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:24.408180  1751 raft_consensus.cc:2468] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 6.
I20260709 13:00:24.408465  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:24.394225  1752 raft_consensus.cc:1275] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20260709 13:00:24.394697  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:24.409245  1601 raft_consensus.cc:2468] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 5.
I20260709 13:00:24.394948  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:00:24.395566  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:24.397353  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:24.410279  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:24.398480  4900 leader_election.cc:290] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:24.395673  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:24.394225  1753 raft_consensus.cc:1275] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260709 13:00:24.417657  2330 raft_consensus.cc:2393] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 9: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:00:24.400928  5154 leader_election.cc:290] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:24.398252  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
W20260709 13:00:24.414386  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 79190.8us
I20260709 13:00:24.437707  1718 leader_election.cc:304] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:24.356290  5110 raft_consensus.cc:740] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:00:24.408213  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:24.420441  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 9 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:24.420596  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:24.421625  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:24.422113  5081 raft_consensus.cc:3060] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:00:24.424242  4648 leader_election.cc:290] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:24.418609  1754 raft_consensus.cc:2468] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 7.
I20260709 13:00:24.425673  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:24.453040  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:24.430347  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:24.431002  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:24.454672  1751 raft_consensus.cc:2393] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 8: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:00:24.455822  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:24.457952  1602 raft_consensus.cc:2393] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 11: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:00:24.432863  1753 raft_consensus.cc:1275] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 13:00:24.418782  2298 leader_election.cc:304] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
W20260709 13:00:24.435377  1645 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:24.410122  2337 raft_consensus.cc:2376] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 7. Re-sending same reply.
I20260709 13:00:24.439600  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:00:24.434276  2333 raft_consensus.cc:1275] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 13:00:24.460867  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:24.462136  1605 raft_consensus.cc:2468] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 9.
W20260709 13:00:24.462658  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 98623.2us
I20260709 13:00:24.444236  1748 raft_consensus.cc:2468] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 9.
I20260709 13:00:24.445019  1680 raft_consensus.cc:3060] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Advancing to term 6
W20260709 13:00:24.464227  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 102911us
I20260709 13:00:24.476307  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:24.477202  1597 raft_consensus.cc:2410] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for term 11 because replica has last-logged OpId of term: 9 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 5.
I20260709 13:00:24.444193  5086 raft_consensus.cc:2749] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:00:24.481669  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:00:24.482293  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 44664us
I20260709 13:00:24.478883  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:24.457087  1679 consensus_queue.cc:260] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:24.479545  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:24.490171  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:24.496703  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:24.498610  1599 raft_consensus.cc:1240] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 LEADER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 9. Current term is 10. Ops: [9.6-9.6]
I20260709 13:00:24.499359  1751 raft_consensus.cc:1275] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 13:00:24.499527  4547 raft_consensus.cc:493] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:24.500036  4547 raft_consensus.cc:515] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:24.500964  1717 leader_election.cc:304] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:00:24.502115  1717 leader_election.cc:304] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:00:24.503422  5129 raft_consensus.cc:2804] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:00:24.503849  5129 raft_consensus.cc:493] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:24.504139  1717 leader_election.cc:304] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:00:24.504233  5129 raft_consensus.cc:3060] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:00:24.505019  5058 raft_consensus.cc:2804] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 13:00:24.505618  5058 raft_consensus.cc:697] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 9 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:00:24.506105  1745 raft_consensus.cc:2393] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 7: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:00:24.507033  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:24.506780  5058 consensus_queue.cc:237] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:24.507927  1753 raft_consensus.cc:2393] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 7: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:00:24.508826  4985 raft_consensus.cc:2804] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:00:24.505702  1754 raft_consensus.cc:1240] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 7. Current term is 8. Ops: [6.7-7.8]
I20260709 13:00:24.510591  4990 consensus_queue.cc:1048] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:00:24.510768  5110 consensus_queue.cc:260] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:24.512081  5110 raft_consensus.cc:3060] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:00:24.513355  5143 leader_election.cc:290] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 8 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:24.512125  1643 leader_election.cc:304] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:00:24.492482  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:24.520649  1717 leader_election.cc:304] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:00:24.486596  2299 leader_election.cc:304] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:00:24.530977  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 9 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:24.531996  2334 raft_consensus.cc:2410] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for term 9 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20260709 13:00:24.481034  4951 raft_consensus.cc:2804] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Leader election won for term 9
W20260709 13:00:24.536837  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 89033us
I20260709 13:00:24.537039  1754 raft_consensus.cc:1275] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 13:00:24.540341  1601 raft_consensus.cc:2468] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 7.
I20260709 13:00:24.547392  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:24.547871  1679 raft_consensus.cc:3060] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:00:24.548512  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:24.549324  2333 raft_consensus.cc:1275] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260709 13:00:24.551810  1753 raft_consensus.cc:1240] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 10. Current term is 11. Ops: []
I20260709 13:00:24.554492  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:24.556350  1745 raft_consensus.cc:1275] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260709 13:00:24.562523  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:24.566527  2329 raft_consensus.cc:1275] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 13:00:24.544302  2337 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb"
dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
mode: GRACEFUL
new_leader_uuid: "af520b67945d443b901bbeca37cc5701"
 from {username='slave'} at 127.0.0.1:35790
I20260709 13:00:24.569717  2337 raft_consensus.cc:606] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Received request to transfer leadership to TS af520b67945d443b901bbeca37cc5701
I20260709 13:00:24.570098  2337 raft_consensus.cc:614] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 13:00:24.571094  4629 raft_consensus.cc:2804] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:00:24.571544  4629 raft_consensus.cc:493] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:24.571717  2337 raft_consensus.cc:1275] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 13:00:24.572038  4629 raft_consensus.cc:3060] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:00:24.578738  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:24.579541  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:24.580437  1749 raft_consensus.cc:2393] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 10: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:00:24.581357  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:24.582712  1599 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 2057) took 1950 ms. Trace:
I20260709 13:00:24.583176  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:24.583110  1599 rpcz_store.cc:276] 0709 13:00:22.632050 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:22.954081 (+322031us) service_pool.cc:224] Handling call
0709 13:00:22.954641 (+   560us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:00:24.499126 (+1544485us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:24.582388 (+ 83262us) spinlock_profiling.cc:243] Waited 1.38 s on lock 0x7b540006d6d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:00:24.582641 (+   253us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3182929152}
I20260709 13:00:24.583874  1754 raft_consensus.cc:2376] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 6. Re-sending same reply.
I20260709 13:00:24.585041  1599 raft_consensus.cc:1240] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 6. Current term is 7. Ops: []
I20260709 13:00:24.585641  1599 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 2066) took 1584 ms. Trace:
I20260709 13:00:24.586684  1598 raft_consensus.cc:2468] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 5.
I20260709 13:00:24.589641  4547 leader_election.cc:290] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:24.590181  1602 raft_consensus.cc:2468] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 7.
I20260709 13:00:24.554060  4759 raft_consensus.cc:2804] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 13:00:24.585963  1599 rpcz_store.cc:276] 0709 13:00:23.000874 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:24.584380 (+1583506us) service_pool.cc:224] Handling call
0709 13:00:24.584753 (+   373us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:24.585414 (+   661us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:24.585587 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:00:24.592931  2300 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:24.593959  1597 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 2064) took 1927 ms. Trace:
I20260709 13:00:24.579556  1747 raft_consensus.cc:2468] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 7.
I20260709 13:00:24.594408  1597 rpcz_store.cc:276] 0709 13:00:22.637583 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:24.490983 (+1853400us) service_pool.cc:224] Handling call
0709 13:00:24.546530 (+ 55547us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:24.561183 (+ 14653us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:00:24.561213 (+    30us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:00:24.561267 (+    54us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:00:24.561302 (+    35us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:24.561363 (+    61us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:24.561473 (+   110us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:00:24.589717  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 69145.7us
I20260709 13:00:24.610522  1751 raft_consensus.cc:2468] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 7.
I20260709 13:00:24.605393  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:24.612828  1749 raft_consensus.cc:2468] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 7.
I20260709 13:00:24.553799  5086 raft_consensus.cc:2804] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:00:24.625828  1597 raft_consensus.cc:1275] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260709 13:00:24.626600  1597 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 2073) took 1460 ms. Trace:
I20260709 13:00:24.626945  1597 rpcz_store.cc:276] 0709 13:00:23.166489 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:24.612322 (+1445833us) service_pool.cc:224] Handling call
0709 13:00:24.625416 (+ 13094us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:24.626281 (+   865us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:24.626518 (+   237us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:00:24.627519  1601 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40154 (request call id 1977) took 1959 ms. Trace:
I20260709 13:00:24.627903  1601 rpcz_store.cc:276] 0709 13:00:22.668016 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:24.545894 (+1877878us) service_pool.cc:224] Handling call
0709 13:00:24.626797 (+ 80903us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:24.627177 (+   380us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:00:24.627210 (+    33us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:00:24.627258 (+    48us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:00:24.627291 (+    33us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:24.627354 (+    63us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:24.627460 (+   106us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":117,"raft.run_cpu_time_us":91,"raft.run_wall_time_us":91}
W20260709 13:00:24.630823  2300 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:24.629995  1601 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 638) took 1076 ms. Trace:
I20260709 13:00:24.631711  1601 rpcz_store.cc:276] 0709 13:00:23.553574 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:24.629129 (+1075555us) service_pool.cc:224] Handling call
0709 13:00:24.629463 (+   334us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:24.629658 (+   195us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:00:24.629686 (+    28us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:00:24.629732 (+    46us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:00:24.629767 (+    35us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:24.629821 (+    54us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:24.629916 (+    95us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:00:24.633946  1601 raft_consensus.cc:1240] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 8. Current term is 9. Ops: [8.8-8.8]
I20260709 13:00:24.567471  5150 raft_consensus.cc:2749] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:00:24.628394  1597 raft_consensus.cc:1275] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20260709 13:00:24.596192  4951 raft_consensus.cc:697] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 9 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
W20260709 13:00:24.646330  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 145709us
I20260709 13:00:24.611464  4759 raft_consensus.cc:697] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 9 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:00:24.616449  5152 raft_consensus.cc:493] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:24.560969  4900 consensus_queue.cc:1048] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.077s
W20260709 13:00:24.664505  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:00:24.669169  1645 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:24.635092  5086 raft_consensus.cc:493] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:00:24.701401  1597 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 2074) took 1474 ms. Trace:
I20260709 13:00:24.701990  1597 rpcz_store.cc:276] 0709 13:00:23.167048 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:24.627733 (+1460685us) service_pool.cc:224] Handling call
0709 13:00:24.628054 (+   321us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:24.641276 (+ 13222us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:24.641552 (+   276us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:00:24.616662  4779 consensus_queue.cc:1048] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.041s
I20260709 13:00:24.645730  4985 raft_consensus.cc:697] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 7 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:00:24.711793  1646 leader_election.cc:400] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 6 election: Vote denied by peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693) with higher term. Message: Invalid argument: T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for earlier term 6. Current term is 7.
I20260709 13:00:24.713404  1646 leader_election.cc:403] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 6 election: Cancelling election due to peer responding with higher term
I20260709 13:00:24.714394  2299 leader_election.cc:304] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
W20260709 13:00:24.721154  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:24.701184  5086 raft_consensus.cc:3060] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Advancing to term 6
W20260709 13:00:24.723075  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 54420.7us
I20260709 13:00:24.638538  5094 consensus_queue.cc:1059] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.068s
W20260709 13:00:24.734323  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:00:24.736562  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 224494us
I20260709 13:00:24.723629  2299 leader_election.cc:304] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:24.736136  5129 raft_consensus.cc:515] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:24.741918  5129 leader_election.cc:290] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 6 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:00:24.750195  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 155173us
I20260709 13:00:24.753170  1597 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 642) took 1050 ms. Trace:
I20260709 13:00:24.753603  1597 rpcz_store.cc:276] 0709 13:00:23.699071 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:24.708478 (+1009407us) service_pool.cc:224] Handling call
0709 13:00:24.737842 (+ 29364us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:24.738076 (+   234us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:00:24.738102 (+    26us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:00:24.748721 (+ 10619us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:00:24.748806 (+    85us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:24.748901 (+    95us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:24.749059 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:00:24.703433  5152 raft_consensus.cc:515] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:24.755656  1643 leader_election.cc:304] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:00:24.772367  4896 raft_consensus.cc:493] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:00:24.772888  4896 raft_consensus.cc:3060] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:00:24.770916  4900 raft_consensus.cc:2804] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:00:24.774200  4900 raft_consensus.cc:493] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:24.774623  4900 raft_consensus.cc:3060] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:00:24.776388  4952 consensus_queue.cc:1059] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 4.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 13:00:24.777184  1601 raft_consensus.cc:1240] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 6. Current term is 7. Ops: []
I20260709 13:00:24.777916  1601 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 643) took 1078 ms. Trace:
I20260709 13:00:24.778290  5158 raft_consensus.cc:2804] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:00:24.778236  1601 rpcz_store.cc:276] 0709 13:00:23.699497 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:24.734290 (+1034793us) service_pool.cc:224] Handling call
0709 13:00:24.751313 (+ 17023us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:24.777668 (+ 26355us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:24.777855 (+   187us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:00:24.779026  5158 raft_consensus.cc:493] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:00:24.779488  5158 raft_consensus.cc:3060] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:00:24.779765  1601 raft_consensus.cc:1240] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 LEADER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 6. Current term is 7. Ops: []
I20260709 13:00:24.781077  1601 raft_consensus.cc:1240] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 7. Current term is 8. Ops: []
W20260709 13:00:24.785049  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 130659us
I20260709 13:00:24.770848  1598 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 2068) took 1769 ms. Trace:
W20260709 13:00:24.815613  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 60102.5us
I20260709 13:00:24.830498  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:24.831636  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:24.833274  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:24.835574  1646 leader_election.cc:304] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:00:24.843803  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:24.828393  5173 raft_consensus.cc:3060] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:00:24.835080  4537 raft_consensus.cc:3060] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:00:24.850138  1602 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 2069) took 1848 ms. Trace:
W20260709 13:00:24.862042  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 141094us
W20260709 13:00:24.855427  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 78430.7us
I20260709 13:00:24.860710  1602 rpcz_store.cc:276] 0709 13:00:23.001927 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:24.591022 (+1589095us) service_pool.cc:224] Handling call
0709 13:00:24.591454 (+   432us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:24.591711 (+   257us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 13:00:24.592043 (+   332us) log.cc:844] Serialized 14 byte log entry
0709 13:00:24.849518 (+257475us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:00:24.849591 (+    73us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:00:24.849650 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:24.849728 (+    78us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:24.849992 (+   264us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":6413,"prepare.run_cpu_time_us":71,"prepare.run_wall_time_us":4215,"raft.queue_time_us":4240,"thread_start_us":254369,"threads_started":1}
I20260709 13:00:24.842087  5058 raft_consensus.cc:493] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:24.867789  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:24.765348  4759 consensus_queue.cc:237] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:24.867239  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:24.867009  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:24.843528  5129 raft_consensus.cc:493] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:00:24.873257  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:24.877163  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:24.872635  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:24.873368  2301 leader_election.cc:304] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:00:24.851872  4547 consensus_queue.cc:1048] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.018s
I20260709 13:00:24.826865  4951 consensus_queue.cc:237] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:24.887274  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:24.872467  5058 raft_consensus.cc:515] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:24.891309  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
W20260709 13:00:24.912933  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 95016.9us
I20260709 13:00:24.891034  4629 raft_consensus.cc:515] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:24.891997  1676 raft_consensus.cc:2393] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 9: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:00:24.892305  4922 raft_consensus.cc:2749] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:00:24.895198  5129 raft_consensus.cc:515] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:24.896857  1569 leader_election.cc:304] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:00:24.878296  4872 consensus_queue.cc:1048] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.023s
I20260709 13:00:24.899070  5182 consensus_queue.cc:1048] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 13:00:24.900077  4847 leader_election.cc:290] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:24.792773  1598 rpcz_store.cc:276] 0709 13:00:23.001618 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:24.587482 (+1585864us) service_pool.cc:224] Handling call
0709 13:00:24.587891 (+   409us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:24.588135 (+   244us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 13:00:24.588516 (+   381us) log.cc:844] Serialized 14 byte log entry
0709 13:00:24.753447 (+164931us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:00:24.753515 (+    68us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:00:24.770516 (+ 17001us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:24.770613 (+    97us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:24.770752 (+   139us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":961,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":45,"raft.queue_time_us":9578,"raft.run_cpu_time_us":158,"raft.run_wall_time_us":8578,"thread_start_us":157989,"threads_started":1}
I20260709 13:00:24.902566  5172 raft_consensus.cc:2804] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Leader election won for term 8
I20260709 13:00:24.853540  4779 consensus_queue.cc:1059] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 6.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.022s
I20260709 13:00:24.939512  5182 consensus_queue.cc:1048] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 13:00:24.939620  5086 raft_consensus.cc:515] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:24.907522  5045 raft_consensus.cc:493] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:00:24.907012  5188 raft_consensus.cc:3055] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 4 LEADER]: Stepping down as leader of term 4
W20260709 13:00:24.910037  1570 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:24.921109  1672 raft_consensus.cc:2393] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 6: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
W20260709 13:00:24.924574  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 56021.5us
I20260709 13:00:24.930347  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:24.933491  4629 leader_election.cc:290] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 8 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:24.934067  4512 raft_consensus.cc:2804] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:00:24.937115  5058 leader_election.cc:290] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:24.937865  5129 leader_election.cc:290] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:24.940341  4779 raft_consensus.cc:3055] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:00:24.905772  1673 raft_consensus.cc:2393] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 8: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
W20260709 13:00:24.941138  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 44355.9us
I20260709 13:00:24.941295  1598 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40154 (request call id 1988) took 1079 ms. Trace:
I20260709 13:00:24.991053  1598 rpcz_store.cc:276] 0709 13:00:23.861687 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:24.938976 (+1077289us) service_pool.cc:224] Handling call
0709 13:00:24.939377 (+   401us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:24.940919 (+  1542us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:00:24.940950 (+    31us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:00:24.941000 (+    50us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:00:24.941051 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:24.941111 (+    60us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:24.941228 (+   117us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:00:24.941928  5036 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 10 current term: 10 is higher.
I20260709 13:00:24.941813  4847 raft_consensus.cc:2804] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:00:24.943748  5172 raft_consensus.cc:697] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 8 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:00:24.944916  5175 consensus_queue.cc:1059] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 13:00:24.925782  5044 raft_consensus.cc:493] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:00:24.946635  5174 log.cc:927] Time spent T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.063s	user 0.001s	sys 0.000s
I20260709 13:00:24.992937  4779 raft_consensus.cc:740] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:00:24.946611  4746 consensus_queue.cc:1048] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 13:00:25.003991  4779 consensus_queue.cc:260] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:24.999096  4847 raft_consensus.cc:493] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:00:24.943329  4900 raft_consensus.cc:515] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:25.005478  4779 raft_consensus.cc:3060] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:00:24.996574  4512 raft_consensus.cc:697] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 7 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:00:24.946146  1646 leader_election.cc:304] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:00:24.946277  5189 consensus_queue.cc:1048] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.053s
W20260709 13:00:24.976348  1570 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:24.856143  4985 consensus_queue.cc:237] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:24.983459  2298 leader_election.cc:304] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:00:24.930927  5151 raft_consensus.cc:493] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:00:24.997169  1569 leader_election.cc:304] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:00:24.975960  5188 raft_consensus.cc:740] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:00:25.002396  1598 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 2077) took 1135 ms. Trace:
I20260709 13:00:25.007673  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:25.017310  1673 raft_consensus.cc:3060] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:00:25.018596  4900 leader_election.cc:290] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 10 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:25.018935  5079 leader_election.cc:290] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 8 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:25.020089  4922 raft_consensus.cc:3060] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Advancing to term 6
W20260709 13:00:25.020166  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 74177.1us
I20260709 13:00:25.021548  1646 leader_election.cc:304] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:00:25.022547  5044 raft_consensus.cc:515] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:25.023809  4547 raft_consensus.cc:493] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:25.024154  4746 consensus_queue.cc:1048] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 13:00:25.024446  4547 raft_consensus.cc:515] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:25.025079  5044 leader_election.cc:290] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:25.005563  4847 raft_consensus.cc:3060] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:00:25.027943  4512 consensus_queue.cc:237] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:25.017009  1598 rpcz_store.cc:276] 0709 13:00:23.863075 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:24.991904 (+1128829us) service_pool.cc:224] Handling call
0709 13:00:24.992179 (+   275us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:24.992404 (+   225us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:00:24.992432 (+    28us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:00:24.997767 (+  5335us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:00:24.997825 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:24.997900 (+    75us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:24.998026 (+   126us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:00:24.957686  1571 leader_election.cc:304] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:00:25.001636  4872 consensus_queue.cc:1048] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.010s
I20260709 13:00:25.012122  5175 raft_consensus.cc:3055] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:00:25.040426  1675 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57092 (request call id 735) took 1118 ms. Trace:
W20260709 13:00:25.036099  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 52758.4us
I20260709 13:00:25.034082  1718 leader_election.cc:304] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:24.933493  4896 raft_consensus.cc:515] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:00:25.041947  1605 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40166 (request call id 2067) took 2040 ms (client timeout 1585 ms). Trace:
I20260709 13:00:25.048980  1646 leader_election.cc:304] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:00:25.037439  2301 leader_election.cc:304] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:25.051856  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:25.048581  1599 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 2070) took 2028 ms. Trace:
I20260709 13:00:25.054613  1598 raft_consensus.cc:2468] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 7.
I20260709 13:00:25.040866  1675 rpcz_store.cc:276] 0709 13:00:23.922307 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:23.926615 (+  4308us) service_pool.cc:224] Handling call
0709 13:00:24.224688 (+298073us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:24.238984 (+ 14296us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:25.040181 (+801197us) spinlock_profiling.cc:243] Waited 293 ms on lock 0x7b54001e7dd0. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661618ad 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:00:25.040353 (+   172us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":675002112}
W20260709 13:00:25.049873  1605 rpcz_store.cc:269] 0709 13:00:23.001267 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:23.161323 (+160056us) service_pool.cc:224] Handling call
0709 13:00:25.041747 (+1880424us) spinlock_profiling.cc:243] Waited 1.17 s on lock 0x7b540028ad54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:00:25.041865 (+   118us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2695581440}
W20260709 13:00:25.067214  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:00:25.015987  5145 consensus_meta.cc:287] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.955s	user 0.017s	sys 0.003s
I20260709 13:00:25.069453  1598 raft_consensus.cc:1240] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 10. Current term is 11. Ops: []
I20260709 13:00:25.070248  1598 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 2082) took 1052 ms. Trace:
I20260709 13:00:25.070747  1598 rpcz_store.cc:276] 0709 13:00:24.018006 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:25.055664 (+1037658us) service_pool.cc:224] Handling call
0709 13:00:25.056023 (+   359us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:25.069960 (+ 13937us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:25.070172 (+   212us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:00:25.072680  2301 leader_election.cc:304] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:24.388899  1596 rpcz_store.cc:276] 0709 13:00:22.632674 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:23.351167 (+718493us) service_pool.cc:224] Handling call
0709 13:00:23.395061 (+ 43894us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:23.466827 (+ 71766us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 13:00:24.166483 (+699656us) log.cc:844] Serialized 14 byte log entry
0709 13:00:24.352508 (+186025us) spinlock_profiling.cc:243] Waited 2.24 ms on lock 0x7b18003ea9a0. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f4265d0a4bc 00007f426305848b 00007f426beb2ab9 00007f426307469b 00007f426307293f 00007f4263074c02 00007f4265cfc20b 00007f4265d01d23 00007f4265d0223a
0709 13:00:24.352979 (+   471us) log.cc:844] Serialized 14 byte log entry
0709 13:00:24.353379 (+   400us) log.cc:844] Serialized 14 byte log entry
0709 13:00:24.353545 (+   166us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:00:24.353591 (+    46us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:00:24.353659 (+    68us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:24.353726 (+    67us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:24.358871 (+  5145us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":699225,"prepare.queue_time_us":708645,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":18982,"raft.queue_time_us":8195,"raft.run_cpu_time_us":155,"raft.run_wall_time_us":25291,"spinlock_wait_cycles":54581376,"thread_start_us":42217,"threads_started":1,"wal-append.queue_time_us":42726}
I20260709 13:00:25.052975  1599 rpcz_store.cc:276] 0709 13:00:23.002242 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:24.593361 (+1591119us) service_pool.cc:224] Handling call
0709 13:00:24.634653 (+ 41292us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:00:24.669176 (+ 34523us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:00:24.669229 (+    53us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:00:24.701045 (+ 31816us) log.cc:844] Serialized 26 byte log entry
0709 13:00:24.807035 (+105990us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:00:24.807100 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:24.807206 (+   106us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:00:25.030079 (+222873us) raft_consensus.cc:1685] finished
0709 13:00:25.030112 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:25.030296 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":21229,"raft.run_cpu_time_us":218,"raft.run_wall_time_us":10383,"thread_start_us":68330,"threads_started":1,"wal-append.queue_time_us":303897}
I20260709 13:00:25.090497  1677 raft_consensus.cc:1240] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 7. Current term is 8. Ops: []
W20260709 13:00:25.090695  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 58614.8us
W20260709 13:00:25.091480  1680 consensus_meta.cc:287] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.576s	user 0.012s	sys 0.000s
I20260709 13:00:25.091877  1680 raft_consensus.cc:2468] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 6.
I20260709 13:00:24.980484  5045 raft_consensus.cc:515] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:25.093863  1678 raft_consensus.cc:1275] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 13:00:25.045365  5044 raft_consensus.cc:493] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:25.112438  4629 consensus_queue.cc:1059] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 13:00:25.095934  1600 raft_consensus.cc:2468] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 6.
I20260709 13:00:25.049077  5186 raft_consensus.cc:3060] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:00:25.117545  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:25.101626  2298 leader_election.cc:304] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 13:00:25.102465  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 68447.2us
I20260709 13:00:25.120384  1718 leader_election.cc:304] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:00:25.048403  5175 raft_consensus.cc:740] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:00:25.107375  4547 leader_election.cc:290] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:25.107726  4537 raft_consensus.cc:493] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:25.110715  5188 consensus_queue.cc:260] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:25.048381  4900 consensus_queue.cc:1048] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.064s
I20260709 13:00:25.077241  4580 raft_consensus.cc:3060] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:00:25.080822  4759 raft_consensus.cc:2804] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 13:00:25.114181  1603 raft_consensus.cc:2468] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 7.
I20260709 13:00:25.099949  5145 raft_consensus.cc:2749] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:00:25.100925  1605 raft_consensus.cc:1240] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 5. Current term is 6. Ops: []
I20260709 13:00:25.122112  1604 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 2063) took 2488 ms. Trace:
I20260709 13:00:25.126940  1604 rpcz_store.cc:276] 0709 13:00:22.634131 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:23.804338 (+1170207us) service_pool.cc:224] Handling call
0709 13:00:24.151375 (+347037us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:24.175896 (+ 24521us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:25.116279 (+940383us) spinlock_profiling.cc:243] Waited 346 ms on lock 0x7b54003edfd0. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661618ad 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:00:25.122023 (+  5744us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":796866944}
I20260709 13:00:25.112969  5044 raft_consensus.cc:515] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:25.132992  5044 leader_election.cc:290] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:25.122056  4872 raft_consensus.cc:2804] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:00:25.133975  4872 raft_consensus.cc:697] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:00:25.113642  5149 raft_consensus.cc:2749] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:00:25.077734  5172 consensus_queue.cc:237] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:25.046041  5151 raft_consensus.cc:515] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:00:25.128163  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 131094us
I20260709 13:00:25.118250  5182 raft_consensus.cc:2804] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:00:25.167205  1718 leader_election.cc:304] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:00:25.116390  5152 leader_election.cc:290] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:25.175747  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:25.176321  1569 leader_election.cc:304] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:00:25.178193  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:25.178458  1569 leader_election.cc:304] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:00:25.179159  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:25.179553  5145 raft_consensus.cc:2804] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:00:25.179997  5145 raft_consensus.cc:493] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:00:25.180349  5145 raft_consensus.cc:3060] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:00:25.182683  4547 consensus_queue.cc:1048] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
W20260709 13:00:25.183111  5185 log.cc:927] Time spent T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.232s	user 0.004s	sys 0.000s
I20260709 13:00:25.130298  4537 raft_consensus.cc:515] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:25.184391  4547 consensus_queue.cc:1048] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 13:00:25.186532  4537 leader_election.cc:290] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:25.186635  5044 raft_consensus.cc:2764] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
I20260709 13:00:25.190754  5152 raft_consensus.cc:3060] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:00:25.176131  4512 raft_consensus.cc:3060] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:00:25.194836  1601 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40154 (request call id 1984) took 1380 ms. Trace:
I20260709 13:00:25.195616  5145 raft_consensus.cc:515] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:25.195259  1601 rpcz_store.cc:276] 0709 13:00:23.814572 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:24.781733 (+967161us) service_pool.cc:224] Handling call
0709 13:00:24.782079 (+   346us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:00:24.795920 (+ 13841us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 13:00:24.795967 (+    47us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:00:24.840748 (+ 44781us) log.cc:844] Serialized 26 byte log entry
0709 13:00:24.927779 (+ 87031us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:00:24.927844 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:24.927934 (+    90us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:00:25.194523 (+266589us) raft_consensus.cc:1685] finished
0709 13:00:25.194555 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:25.194752 (+   197us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":13337,"raft.run_cpu_time_us":218,"raft.run_wall_time_us":25200,"thread_start_us":59294,"threads_started":1,"wal-append.queue_time_us":286758}
I20260709 13:00:25.198158  5145 leader_election.cc:290] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 7 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:25.214831  5188 raft_consensus.cc:3060] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 4 FOLLOWER]: Advancing to term 5
W20260709 13:00:25.218559  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 120572us
W20260709 13:00:25.219485  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 81887.9us
W20260709 13:00:25.221293  2300 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:25.224224  2298 leader_election.cc:304] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:00:25.140044  4985 raft_consensus.cc:2804] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:00:25.222071  1646 leader_election.cc:304] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:25.225374  1597 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 648) took 1433 ms. Trace:
I20260709 13:00:25.135056  4872 consensus_queue.cc:237] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:00:25.202239  4574 consensus_meta.cc:287] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 1.080s	user 0.006s	sys 0.006s
I20260709 13:00:25.241052  1602 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40154 (request call id 1987) took 1382 ms. Trace:
I20260709 13:00:25.236697  1673 raft_consensus.cc:2468] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 8.
I20260709 13:00:25.232407  1597 rpcz_store.cc:276] 0709 13:00:23.791581 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:24.773917 (+982336us) service_pool.cc:224] Handling call
0709 13:00:24.774282 (+   365us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:00:24.774904 (+   622us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:00:24.774934 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:00:24.775593 (+   659us) log.cc:844] Serialized 26 byte log entry
0709 13:00:24.849786 (+ 74193us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:00:24.849877 (+    91us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:24.849996 (+   119us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:00:25.030830 (+180834us) spinlock_profiling.cc:243] Waited 26.2 ms on lock 0x7b4400022b4c. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f4262f9bc85 00007f4262f9c6a2 00007f4265d0e1db 00007f4265d09aaf 00007f4265cfd86e 00007f4265cfcbb6 00007f4265cfc7d8 00007f4265d06221 00007f4265d061d9 00007f4265d06169
0709 13:00:25.217995 (+187165us) raft_consensus.cc:1685] finished
0709 13:00:25.218030 (+    35us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:25.225265 (+  7235us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":3228,"raft.run_cpu_time_us":117,"raft.run_wall_time_us":117,"spinlock_wait_cycles":60253952,"thread_start_us":73605,"threads_started":1,"wal-append.queue_time_us":174007}
I20260709 13:00:25.173137  5182 raft_consensus.cc:493] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:25.232182  4985 raft_consensus.cc:493] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:00:25.237427  5183 raft_consensus.cc:2804] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:00:25.261299  1718 leader_election.cc:304] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
W20260709 13:00:25.263485  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 190876us
W20260709 13:00:25.249516  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 318063us
I20260709 13:00:25.322136  1571 leader_election.cc:304] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:00:25.254638  1646 leader_election.cc:304] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:00:25.252705  5182 raft_consensus.cc:3060] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:00:25.289366  5123 raft_consensus.cc:2804] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Leader election won for term 8
W20260709 13:00:25.297957  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 73834.6us
I20260709 13:00:25.245834  5189 raft_consensus.cc:2804] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:00:25.198261  5158 raft_consensus.cc:515] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:25.250943  4746 raft_consensus.cc:2749] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:00:25.239219  5149 consensus_queue.cc:1059] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.023s
I20260709 13:00:25.238103  5079 consensus_queue.cc:1048] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.024s
I20260709 13:00:25.323625  2301 leader_election.cc:304] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
W20260709 13:00:25.333282  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 154889us
W20260709 13:00:25.334201  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 153721us
I20260709 13:00:25.263919  5183 raft_consensus.cc:493] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
W20260709 13:00:25.347687  1644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:25.348121  5183 raft_consensus.cc:3060] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:00:25.299090  5086 leader_election.cc:290] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 6 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:25.349334  2301 leader_election.cc:304] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:00:25.264294  4985 raft_consensus.cc:3060] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:00:25.248140  4574 raft_consensus.cc:515] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:25.340081  5189 raft_consensus.cc:493] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:00:25.344722  5110 consensus_meta.cc:287] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.832s	user 0.013s	sys 0.006s
I20260709 13:00:25.343837  4512 raft_consensus.cc:2749] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260709 13:00:25.358680  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 104144us
I20260709 13:00:25.362272  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:25.364612  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:25.365237  1646 leader_election.cc:304] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:00:25.347579  5123 raft_consensus.cc:697] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 8 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:00:25.367820  1750 raft_consensus.cc:2393] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 9: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:00:25.353698  5149 consensus_queue.cc:1243] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer 816fadcea8fc4893882f5f9b9f22a03d log is divergent from this leader: its last log entry 6.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260709 13:00:25.365252  1752 raft_consensus.cc:2376] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 9. Re-sending same reply.
I20260709 13:00:25.354382  4900 raft_consensus.cc:2804] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:00:25.354226  4951 raft_consensus.cc:2804] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:00:25.365325  1571 leader_election.cc:304] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:00:25.358592  5189 raft_consensus.cc:3060] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Advancing to term 7
W20260709 13:00:25.373299  1645 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:25.352592  4629 consensus_queue.cc:1048] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.019s
I20260709 13:00:25.263581  5175 consensus_queue.cc:260] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:25.362613  4896 leader_election.cc:290] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 10 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:00:25.421797  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 45289.4us
I20260709 13:00:25.461850  1749 raft_consensus.cc:1240] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 9 LEADER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 8. Current term is 9. Ops: [8.8-8.8]
W20260709 13:00:25.468082  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 102823us
I20260709 13:00:25.469300  1571 leader_election.cc:304] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
W20260709 13:00:25.475450  1570 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:25.372891  4759 raft_consensus.cc:697] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 6 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
W20260709 13:00:25.490794  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 118520us
I20260709 13:00:25.457406  4900 raft_consensus.cc:493] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:00:25.389891  5172 consensus_queue.cc:1048] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.097s
I20260709 13:00:25.464035  4951 raft_consensus.cc:493] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:25.370669  5199 raft_consensus.cc:2749] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:00:25.498193  1571 leader_election.cc:304] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 13:00:25.473718  5045 leader_election.cc:290] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:25.466030  5205 raft_consensus.cc:493] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:25.432047  4512 consensus_queue.cc:1059] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.067s
I20260709 13:00:25.500335  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:00:25.474260  5207 raft_consensus.cc:2749] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:00:25.507203  4900 raft_consensus.cc:3060] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:00:25.521598  1571 leader_election.cc:304] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 13:00:25.519793  2330 raft_consensus.cc:1240] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 8 LEADER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 7. Current term is 8. Ops: [7.8-7.8]
I20260709 13:00:25.490547  4580 raft_consensus.cc:2749] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:00:25.524868  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:25.525081  5212 raft_consensus.cc:493] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:25.525599  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:25.525759  5212 raft_consensus.cc:515] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:25.526467  2329 raft_consensus.cc:2393] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 10: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:00:25.527010  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:00:25.507341  5110 raft_consensus.cc:2804] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:00:25.527730  5110 raft_consensus.cc:493] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:25.528278  5110 raft_consensus.cc:3060] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:00:25.528545  5212 leader_election.cc:290] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:25.529230  5212 raft_consensus.cc:2804] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:00:25.508164  5175 raft_consensus.cc:3060] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:00:25.527685  2328 raft_consensus.cc:2376] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 8. Re-sending same reply.
W20260709 13:00:25.535668  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 50601.3us
I20260709 13:00:25.536255  1571 leader_election.cc:304] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 13:00:25.519394  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:00:25.538616  2335 raft_consensus.cc:2376] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 6. Re-sending same reply.
I20260709 13:00:25.536535  1644 leader_election.cc:304] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:25.555115  1677 raft_consensus.cc:2468] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 8.
W20260709 13:00:25.574321  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 209142us
I20260709 13:00:25.504312  4951 raft_consensus.cc:3060] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:00:25.581748  1646 leader_election.cc:304] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1
W20260709 13:00:25.582333  1645 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:25.544523  4537 raft_consensus.cc:2804] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260709 13:00:25.586906  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 50441.6us
I20260709 13:00:25.590389  1644 leader_election.cc:304] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:25.512377  5208 raft_consensus.cc:2749] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:00:25.535377  5145 raft_consensus.cc:2804] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Leader election won for term 7
W20260709 13:00:25.615760  5195 log.cc:927] Time spent T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.157s	user 0.000s	sys 0.000s
I20260709 13:00:25.529610  5212 raft_consensus.cc:493] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:25.571085  5079 raft_consensus.cc:493] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:25.623227  1644 leader_election.cc:304] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 13:00:25.637557  1672 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.561s	user 0.006s	sys 0.018s
W20260709 13:00:25.638031  1672 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.563s	user 0.006s	sys 0.018s
W20260709 13:00:25.622573  5187 log.cc:927] Time spent T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.593s	user 0.003s	sys 0.000s
W20260709 13:00:25.619846  5200 log.cc:927] Time spent T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.156s	user 0.001s	sys 0.000s
I20260709 13:00:25.588605  4537 raft_consensus.cc:493] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:25.618278  5176 raft_consensus.cc:493] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:25.513526  5205 raft_consensus.cc:515] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:25.658303  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 115823us
I20260709 13:00:25.620183  5137 raft_consensus.cc:493] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:25.621459  5214 raft_consensus.cc:3055] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:00:25.627118  5212 raft_consensus.cc:3060] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Advancing to term 9
W20260709 13:00:25.659825  1643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:00:25.640128  5202 log.cc:927] Time spent T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.293s	user 0.001s	sys 0.000s
I20260709 13:00:25.627177  5207 raft_consensus.cc:2804] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260709 13:00:25.660226  5204 log.cc:927] Time spent T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.293s	user 0.001s	sys 0.000s
I20260709 13:00:25.669553  1569 leader_election.cc:304] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 13:00:25.636431  5158 leader_election.cc:290] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 9 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 13:00:25.680598  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 108607us
I20260709 13:00:25.654790  5223 raft_consensus.cc:493] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:25.541008  5123 consensus_queue.cc:237] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:00:25.682754  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 59596.3us
I20260709 13:00:25.621410  4629 consensus_queue.cc:1048] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.036s
I20260709 13:00:25.614602  5007 ts_tablet_manager.cc:1929] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 4.5
I20260709 13:00:25.650319  5172 raft_consensus.cc:2804] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader election won for term 9
I20260709 13:00:25.624953  5145 raft_consensus.cc:697] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:00:25.646214  5218 raft_consensus.cc:2749] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:00:25.649626  5208 raft_consensus.cc:493] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:25.660367  4537 raft_consensus.cc:3060] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:00:25.657119  4574 leader_election.cc:290] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 9 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:00:25.693171  5173 consensus_meta.cc:287] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.818s	user 0.017s	sys 0.008s
I20260709 13:00:25.635340  5079 raft_consensus.cc:515] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:25.680316  5214 raft_consensus.cc:740] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:00:25.666078  5213 raft_consensus.cc:2749] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:00:25.719375  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:25.819921  1679 raft_consensus.cc:2410] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 for term 7 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
W20260709 13:00:25.705365  5081 consensus_meta.cc:287] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 1.247s	user 0.013s	sys 0.004s
I20260709 13:00:25.746088  1749 raft_consensus.cc:1240] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 8. Current term is 9. Ops: []
I20260709 13:00:25.624550  4759 consensus_queue.cc:237] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:25.706604  5152 consensus_meta.cc:287] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.515s	user 0.014s	sys 0.000s
I20260709 13:00:25.751291  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:25.865298  5152 raft_consensus.cc:2749] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:00:25.694689  5207 raft_consensus.cc:493] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:25.751590  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:00:25.866781  5207 raft_consensus.cc:3060] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:00:25.867023  1748 raft_consensus.cc:1275] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260709 13:00:25.867749  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:00:25.868170  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:25.869165  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:25.869354  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:00:25.752076  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:25.870419  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:25.871383  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:25.753328  1644 leader_election.cc:304] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 13:00:25.757480  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 221297us
I20260709 13:00:25.760972  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:00:25.873351  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 120159us
I20260709 13:00:25.873790  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:25.874429  5219 raft_consensus.cc:2749] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:00:25.875131  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:25.875255  5219 consensus_queue.cc:1048] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 13:00:25.875777  1680 raft_consensus.cc:3060] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Advancing to term 10
I20260709 13:00:25.876495  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:25.758760  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:25.877580  5219 consensus_queue.cc:1048] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 13:00:25.767753  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:25.879072  1675 raft_consensus.cc:2410] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 for term 9 because replica has last-logged OpId of term: 8 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 8.
I20260709 13:00:25.879666  5219 consensus_queue.cc:1048] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 13:00:25.707746  4580 raft_consensus.cc:2749] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260709 13:00:25.770061  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:25.757397  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:00:25.774922  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 217019us
I20260709 13:00:25.882768  2329 raft_consensus.cc:2468] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 6.
I20260709 13:00:25.882973  2298 leader_election.cc:304] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:00:25.695045  5137 raft_consensus.cc:515] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:25.886070  5137 leader_election.cc:290] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:25.760385  5172 raft_consensus.cc:697] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 9 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:00:25.889170  5172 consensus_queue.cc:237] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:25.809214  5173 raft_consensus.cc:2749] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Leader election lost for term 6. Reason: Vote denied by peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693) with higher term. Message: Invalid argument: T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for earlier term 6. Current term is 7.
I20260709 13:00:25.708089  5215 consensus_queue.cc:1059] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.115s
I20260709 13:00:25.841917  2337 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39726 (request call id 904) took 1438 ms. Trace:
I20260709 13:00:25.890857  2337 rpcz_store.cc:276] 0709 13:00:24.397709 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:24.572582 (+174873us) service_pool.cc:224] Handling call
0709 13:00:24.605210 (+ 32628us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:00:24.845427 (+240217us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:00:24.845542 (+   115us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:00:24.873382 (+ 27840us) log.cc:844] Serialized 26 byte log entry
0709 13:00:25.160156 (+286774us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:00:25.160208 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:25.161065 (+   857us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:00:25.836093 (+675028us) raft_consensus.cc:1685] finished
0709 13:00:25.836131 (+    38us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:25.836343 (+   212us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":21400,"raft.run_cpu_time_us":153,"raft.run_wall_time_us":152,"thread_start_us":435276,"threads_started":2,"wal-append.queue_time_us":328834}
I20260709 13:00:25.841461  1717 leader_election.cc:304] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:25.703974  5083 consensus_queue.cc:1059] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.067s
I20260709 13:00:25.698323  5176 raft_consensus.cc:515] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:25.894968  5176 leader_election.cc:290] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:25.887421  1673 raft_consensus.cc:1240] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20260709 13:00:25.899641  5176 raft_consensus.cc:2749] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260709 13:00:25.900700  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 59345.3us
I20260709 13:00:25.901502  1671 raft_consensus.cc:2468] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 7.
W20260709 13:00:25.872521  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 170349us
I20260709 13:00:25.905359  2328 raft_consensus.cc:2468] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 8.
I20260709 13:00:25.920106  1674 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52778 (request call id 1903) took 1005 ms. Trace:
I20260709 13:00:25.835284  5094 raft_consensus.cc:3055] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:00:25.720959  5223 raft_consensus.cc:515] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:25.824743  5218 raft_consensus.cc:493] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:00:25.927348  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:00:25.954665  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 71781.9us
I20260709 13:00:25.954936  1679 raft_consensus.cc:2468] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 7.
I20260709 13:00:25.771471  5208 raft_consensus.cc:515] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:25.922953  5176 raft_consensus.cc:493] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:00:25.937952  4922 consensus_meta.cc:287] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.917s	user 0.013s	sys 0.002s
I20260709 13:00:25.960847  1753 raft_consensus.cc:1275] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
W20260709 13:00:25.919503  5209 log.cc:927] Time spent T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.364s	user 0.001s	sys 0.001s
I20260709 13:00:25.946365  1674 rpcz_store.cc:276] 0709 13:00:24.915070 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:24.929832 (+ 14762us) service_pool.cc:224] Handling call
0709 13:00:24.981998 (+ 52166us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:00:25.002148 (+ 20150us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:00:25.002188 (+    40us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:00:25.016684 (+ 14496us) log.cc:844] Serialized 52 byte log entry
0709 13:00:25.250038 (+233354us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:00:25.250103 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:25.250201 (+    98us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:00:25.919715 (+669514us) raft_consensus.cc:1685] finished
0709 13:00:25.919750 (+    35us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:25.920008 (+   258us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":14568,"raft.run_cpu_time_us":106,"raft.run_wall_time_us":105,"thread_start_us":148651,"threads_started":1,"wal-append.queue_time_us":284072}
I20260709 13:00:25.931036  5215 consensus_queue.cc:1243] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer af520b67945d443b901bbeca37cc5701 log is divergent from this leader: its last log entry 9.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20260709 13:00:25.934418  5137 raft_consensus.cc:493] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
W20260709 13:00:25.965312  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 89616us
W20260709 13:00:25.958884  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 85671.3us
I20260709 13:00:25.968720  1571 leader_election.cc:304] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:00:25.951609  4779 raft_consensus.cc:493] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:25.951960  5094 raft_consensus.cc:740] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:00:25.953850  5007 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 983621a6fc5b4c9ea0159a648dfe7a36. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:25.957623  5210 raft_consensus.cc:493] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
W20260709 13:00:25.973661  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 391935us
W20260709 13:00:25.986897  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 318972us
I20260709 13:00:25.967967  4922 raft_consensus.cc:2749] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:00:25.974210  1680 raft_consensus.cc:2468] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 10.
I20260709 13:00:26.012893  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:25.940799  5145 consensus_queue.cc:237] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:25.986956  5210 raft_consensus.cc:515] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:26.049727  5225 consensus_queue.cc:1059] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.082s
I20260709 13:00:26.053309  5232 raft_consensus.cc:2804] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader election won for term 6
I20260709 13:00:25.978120  4779 raft_consensus.cc:515] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:26.138267  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 47106.8us
I20260709 13:00:26.054771  5221 raft_consensus.cc:2804] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:00:26.055191  5079 leader_election.cc:290] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:00:26.057988  4847 consensus_meta.cc:287] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 1.032s	user 0.005s	sys 0.010s
W20260709 13:00:26.046689  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 75976.3us
I20260709 13:00:25.975929  5215 consensus_queue.cc:1059] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.042s
I20260709 13:00:26.064844  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:25.971628  5137 raft_consensus.cc:515] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:26.076385  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 64089.9us
I20260709 13:00:26.257647  1597 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e"
dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
mode: GRACEFUL
new_leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
 from {username='slave'} at 127.0.0.1:47752
I20260709 13:00:25.977581  5176 raft_consensus.cc:515] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:26.259619  1569 leader_election.cc:304] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:00:26.260895  5176 leader_election.cc:290] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:25.960106  5218 raft_consensus.cc:515] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:26.210089  5207 raft_consensus.cc:515] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:26.047490  5205 leader_election.cc:290] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:25.246547  1602 rpcz_store.cc:276] 0709 13:00:23.858682 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:24.865266 (+1006584us) service_pool.cc:224] Handling call
0709 13:00:24.910021 (+ 44755us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:24.984996 (+ 74975us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 13:00:25.008520 (+ 23524us) log.cc:844] Serialized 14 byte log entry
0709 13:00:25.240513 (+231993us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:00:25.240600 (+    87us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:00:25.240684 (+    84us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:25.240781 (+    97us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:25.240967 (+   186us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":23545,"prepare.run_cpu_time_us":77,"prepare.run_wall_time_us":4068,"raft.queue_time_us":15722,"thread_start_us":157498,"threads_started":1}
I20260709 13:00:26.266330  5129 raft_consensus.cc:3055] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:00:26.266120  4847 raft_consensus.cc:515] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:26.267022  5129 raft_consensus.cc:740] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:00:26.268038  5129 consensus_queue.cc:260] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:26.014410  5229 consensus_queue.cc:1048] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.088s
I20260709 13:00:26.269238  5129 raft_consensus.cc:3060] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:00:26.270054  5229 raft_consensus.cc:493] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:26.079830  4872 raft_consensus.cc:2749] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260709 13:00:26.112993  5188 consensus_meta.cc:287] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.897s	user 0.015s	sys 0.000s
I20260709 13:00:26.292866  5215 raft_consensus.cc:3055] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:00:26.293540  5215 raft_consensus.cc:740] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
W20260709 13:00:26.294067  5186 consensus_meta.cc:287] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 1.164s	user 0.014s	sys 0.001s
I20260709 13:00:26.294579  5186 raft_consensus.cc:2749] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:00:26.295392  5205 raft_consensus.cc:493] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:26.115191  5223 leader_election.cc:290] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:26.116533  5208 leader_election.cc:290] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:26.122730  5244 raft_consensus.cc:493] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:26.055269  5216 consensus_queue.cc:1059] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.090s
I20260709 13:00:26.187522  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:00:26.196455  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:00:26.136198  5110 consensus_meta.cc:287] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.607s	user 0.010s	sys 0.004s
W20260709 13:00:26.165956  4900 consensus_meta.cc:287] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.646s	user 0.014s	sys 0.000s
W20260709 13:00:26.199337  4951 consensus_meta.cc:287] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.612s	user 0.009s	sys 0.003s
W20260709 13:00:26.187445  4985 consensus_meta.cc:287] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.822s	user 0.008s	sys 0.006s
I20260709 13:00:26.188321  4952 raft_consensus.cc:493] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:26.206743  5242 consensus_queue.cc:1048] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
W20260709 13:00:26.211146  5239 log.cc:927] Time spent T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.133s	user 0.000s	sys 0.000s
I20260709 13:00:26.257863  5137 leader_election.cc:290] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:26.258458  1597 raft_consensus.cc:606] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Received request to transfer leadership to TS 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:00:26.266755  5241 raft_consensus.cc:493] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:26.255541  1749 raft_consensus.cc:2468] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 9.
W20260709 13:00:26.095014  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 241994us
I20260709 13:00:26.377260  1597 raft_consensus.cc:614] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 13:00:26.377496  2329 raft_consensus.cc:2468] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 5.
W20260709 13:00:26.219775  5183 consensus_meta.cc:287] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.862s	user 0.012s	sys 0.004s
W20260709 13:00:26.105108  2331 consensus_meta.cc:287] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.592s	user 0.011s	sys 0.003s
I20260709 13:00:26.213840  5158 raft_consensus.cc:493] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:26.379415  2331 raft_consensus.cc:2468] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 11.
I20260709 13:00:26.296355  2334 raft_consensus.cc:1240] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 10. Current term is 11. Ops: [10.10-10.10]
I20260709 13:00:26.270596  5229 raft_consensus.cc:515] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:26.380366  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 192195us
W20260709 13:00:26.376274  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 119569us
I20260709 13:00:26.382498  5207 leader_election.cc:290] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 9 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:00:26.382586  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 135851us
I20260709 13:00:26.252887  5221 raft_consensus.cc:493] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:26.411792  1646 leader_election.cc:304] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:00:26.296247  5205 raft_consensus.cc:515] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:26.253968  5232 raft_consensus.cc:697] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:00:26.371912  4872 raft_consensus.cc:3060] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:00:26.390000  1718 leader_election.cc:304] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 13:00:26.373107  5175 consensus_meta.cc:287] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.816s	user 0.009s	sys 0.006s
W20260709 13:00:26.399850  1643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:00:26.405745  1676 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.637s	user 0.050s	sys 0.081s
W20260709 13:00:26.440634  1676 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.672s	user 0.050s	sys 0.081s
I20260709 13:00:26.388475  4985 raft_consensus.cc:515] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:26.388111  5110 raft_consensus.cc:515] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:26.390161  5244 raft_consensus.cc:515] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:26.453662  5205 leader_election.cc:290] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:26.442983  5172 raft_consensus.cc:493] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:00:26.454490  5218 leader_election.cc:290] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:26.454372  5172 raft_consensus.cc:515] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:26.413988  5221 raft_consensus.cc:3060] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:00:26.455046  5244 leader_election.cc:290] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:26.409540  5207 consensus_queue.cc:1048] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 13:00:26.457322  5172 leader_election.cc:290] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:26.458922  5259 raft_consensus.cc:493] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:00:26.459751  1745 raft_consensus.cc:1240] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 7. Current term is 8. Ops: []
I20260709 13:00:26.459671  5259 raft_consensus.cc:515] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:26.461999  5259 leader_election.cc:290] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:26.383797  4847 leader_election.cc:290] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 8 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:00:26.416467  1644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:00:26.368897  5189 consensus_meta.cc:287] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.889s	user 0.012s	sys 0.001s
I20260709 13:00:26.372236  5176 raft_consensus.cc:493] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:26.436343  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:26.445358  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:26.468248  1680 raft_consensus.cc:2468] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 6.
I20260709 13:00:26.468551  2329 raft_consensus.cc:2410] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for term 9 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260709 13:00:26.387133  4951 raft_consensus.cc:515] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:26.386819  4952 raft_consensus.cc:515] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:26.357446  5215 consensus_queue.cc:260] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:26.447636  5232 consensus_queue.cc:237] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:26.449172  5242 consensus_queue.cc:1059] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 13:00:26.450471  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:26.450567  4985 leader_election.cc:290] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 8 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:26.387636  5241 raft_consensus.cc:515] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:26.454828  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:26.390640  4900 raft_consensus.cc:515] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:26.462915  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:26.389927  5223 raft_consensus.cc:493] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
W20260709 13:00:26.412007  4537 consensus_meta.cc:287] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.605s	user 0.014s	sys 0.000s
I20260709 13:00:26.454535  2336 raft_consensus.cc:1240] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 10. Current term is 11. Ops: []
W20260709 13:00:26.443708  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 55022.4us
I20260709 13:00:26.470147  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:00:26.472555  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:26.532035  1678 raft_consensus.cc:2468] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 8.
I20260709 13:00:26.437734  4978 raft_consensus.cc:2804] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:00:26.389395  5183 raft_consensus.cc:515] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:26.501585  1674 raft_consensus.cc:2410] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for term 11 because replica has last-logged OpId of term: 9 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 5.
I20260709 13:00:26.459028  1676 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52778 (request call id 1904) took 1509 ms. Trace:
I20260709 13:00:26.446015  5086 raft_consensus.cc:493] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:26.397020  5158 raft_consensus.cc:515] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:26.508074  2335 raft_consensus.cc:2393] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 8: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:00:26.509500  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:00:26.513430  5081 raft_consensus.cc:3055] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:00:26.560954  5081 raft_consensus.cc:740] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:00:26.515834  5007 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.0.0.1:39738 (request call id 705) took 7155 ms. Trace:
I20260709 13:00:26.516328  4900 leader_election.cc:290] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 8 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:26.517191  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:26.517701  4952 leader_election.cc:290] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:26.518141  4951 leader_election.cc:290] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 8 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:00:26.473057  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 77350.1us
I20260709 13:00:26.541765  1679 raft_consensus.cc:2393] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 5: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:00:26.537513  1676 rpcz_store.cc:276] 0709 13:00:24.949188 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:24.949340 (+   152us) service_pool.cc:224] Handling call
0709 13:00:24.950927 (+  1587us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:24.951311 (+   384us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 13:00:25.057589 (+106278us) log.cc:844] Serialized 14 byte log entry
0709 13:00:25.768969 (+711380us) spinlock_profiling.cc:243] Waited 32.5 ms on lock 0x7b18004c6780. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f4263057e07 00007f426beb2ab9 00007f426307469b 00007f426307293f 00007f4263074c02 00007f4265cfc20b 00007f4265d01d23 00007f4265d0223a 00007f426616ca2a
0709 13:00:26.458615 (+689646us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:00:26.458684 (+    69us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:00:26.458741 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:26.458821 (+    80us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:26.458955 (+   134us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":17307,"prepare.queue_time_us":106398,"prepare.run_cpu_time_us":86,"prepare.run_wall_time_us":1773,"raft.queue_time_us":20445,"raft.run_cpu_time_us":95,"raft.run_wall_time_us":96,"spinlock_wait_cycles":74861440,"thread_start_us":777786,"threads_started":2,"wal-append.queue_time_us":1441080}
I20260709 13:00:26.542155  4978 raft_consensus.cc:493] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:26.542376  5223 raft_consensus.cc:515] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:26.543164  4537 raft_consensus.cc:515] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:26.573815  4537 leader_election.cc:290] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 8 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:26.558344  5241 leader_election.cc:290] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:26.557961  5086 raft_consensus.cc:515] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:26.576891  5086 leader_election.cc:290] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:26.578310  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:26.580287  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:26.580894  1748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43156 (request call id 618) took 1206 ms. Trace:
I20260709 13:00:26.581688  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:26.560456  5158 leader_election.cc:290] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:26.583150  5172 raft_consensus.cc:493] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:26.583765  5172 raft_consensus.cc:515] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:26.584245  5151 leader_election.cc:290] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:26.585175  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:26.586387  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:26.587106  5110 leader_election.cc:290] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 9 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:26.590845  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:26.591508  1680 raft_consensus.cc:2468] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 6.
I20260709 13:00:26.476678  5255 raft_consensus.cc:2749] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 13:00:26.570478  4978 raft_consensus.cc:3060] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:00:26.597645  5183 leader_election.cc:290] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 8 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:26.472048  5189 raft_consensus.cc:515] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:26.612552  1747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43156 (request call id 620) took 1221 ms. Trace:
I20260709 13:00:26.581359  1748 rpcz_store.cc:276] 0709 13:00:25.374381 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:25.456814 (+ 82433us) service_pool.cc:224] Handling call
0709 13:00:25.458135 (+  1321us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:25.867643 (+409508us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:26.580619 (+712976us) spinlock_profiling.cc:243] Waited 408 ms on lock 0x7b54000195d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:00:26.580823 (+   204us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":938914688}
I20260709 13:00:26.586133  5172 leader_election.cc:290] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:26.475289  5176 raft_consensus.cc:515] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:00:26.605077  2301 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:26.613101  1747 rpcz_store.cc:276] 0709 13:00:25.391287 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:25.429313 (+ 38026us) service_pool.cc:224] Handling call
0709 13:00:25.471632 (+ 42319us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:00:25.511839 (+ 40207us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 13:00:25.687747 (+175908us) log.cc:844] Serialized 14 byte log entry
0709 13:00:25.976206 (+288459us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:00:25.981998 (+  5792us) log.cc:844] Serialized 52 byte log entry
0709 13:00:25.982676 (+   678us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:00:26.611649 (+628973us) log.cc:844] Serialized 14 byte log entry
0709 13:00:26.612095 (+   446us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:26.612211 (+   116us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:00:26.612274 (+    63us) raft_consensus.cc:1685] finished
0709 13:00:26.612293 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:26.612465 (+   172us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":640634,"prepare.queue_time_us":295570,"prepare.run_cpu_time_us":3774,"prepare.run_wall_time_us":17849,"raft.queue_time_us":21382,"raft.run_cpu_time_us":395,"raft.run_wall_time_us":28035,"thread_start_us":343541,"threads_started":2,"wal-append.queue_time_us":377528}
I20260709 13:00:26.622788  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:26.630149  1749 raft_consensus.cc:2468] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 10.
I20260709 13:00:26.562039  5081 consensus_queue.cc:260] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:26.562204  4872 raft_consensus.cc:2749] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:00:26.564327  5215 raft_consensus.cc:3060] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:00:26.638130  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:26.638962  1748 raft_consensus.cc:2468] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 10.
I20260709 13:00:26.657212  1680 raft_consensus.cc:1240] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 6. Current term is 7. Ops: [6.4-6.4]
W20260709 13:00:26.663300  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 91923.2us
W20260709 13:00:26.661275  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 252613us
I20260709 13:00:26.670181  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:26.670334  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:26.671060  1747 raft_consensus.cc:2468] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 7.
I20260709 13:00:26.665542  1718 leader_election.cc:304] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:00:26.671741  1674 raft_consensus.cc:2468] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 7.
I20260709 13:00:26.672636  1748 raft_consensus.cc:1240] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 5. Current term is 6. Ops: []
I20260709 13:00:26.675714  1597 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c"
dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
mode: GRACEFUL
new_leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
 from {username='slave'} at 127.0.0.1:47752
I20260709 13:00:26.676538  1597 raft_consensus.cc:606] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Received request to transfer leadership to TS 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:00:26.676949  1597 raft_consensus.cc:614] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 13:00:26.678707  1571 leader_election.cc:304] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:00:26.680521  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:26.686583  1717 leader_election.cc:304] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:26.678764  1718 leader_election.cc:304] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:00:26.680507  5137 raft_consensus.cc:2749] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:00:26.709563  1718 leader_election.cc:304] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:00:26.729264  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:26.742725  1718 leader_election.cc:304] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:00:26.756820  1672 raft_consensus.cc:2468] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 10.
I20260709 13:00:26.703755  4872 raft_consensus.cc:2749] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:00:26.706009  5129 raft_consensus.cc:2749] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:00:26.561684  5007 rpcz_store.cc:276] 0709 13:00:19.443863 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:19.444642 (+   779us) service_pool.cc:224] Handling call
0709 13:00:20.083039 (+638397us) ts_tablet_manager.cc:934] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1: Initiating tablet copy from peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
0709 13:00:23.932567 (+3849528us) tablet_copy_client.cc:368] Tablet copy session begun
0709 13:00:24.765213 (+832646us) tablet_metadata.cc:665] Metadata flushed
0709 13:00:25.587474 (+822261us) tablet_metadata.cc:665] Metadata flushed
0709 13:00:25.809930 (+222456us) tablet_copy_client.cc:463] Replaced tombstoned tablet metadata.
0709 13:00:26.358729 (+548799us) tablet_copy_client.cc:418] Wrote new consensus metadata
0709 13:00:26.464080 (+105351us) tablet_replica.cc:206] Creating consensus instance
0709 13:00:26.515566 (+ 51486us) ts_tablet_manager.cc:989] Replica registered.
0709 13:00:26.515705 (+   139us) inbound_call.cc:177] Queueing success response
Metrics: {"dns_us":61776,"tablet-copy.queue_time_us":248185,"thread_start_us":298341,"threads_started":2}
I20260709 13:00:26.706616  5229 leader_election.cc:290] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:26.766592  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 8 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:26.724979  5123 raft_consensus.cc:2749] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:00:26.795711  1602 raft_consensus.cc:1275] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
W20260709 13:00:26.796726  1600 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40154 (request call id 1982) took 3018 ms (client timeout 2021 ms). Trace:
I20260709 13:00:26.797362  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
W20260709 13:00:26.798399  1603 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40154 (request call id 1983) took 2984 ms (client timeout 1892 ms). Trace:
W20260709 13:00:26.798862  1603 rpcz_store.cc:269] 0709 13:00:23.814054 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:24.031400 (+217346us) service_pool.cc:224] Handling call
0709 13:00:26.798214 (+2766814us) spinlock_profiling.cc:243] Waited 1.06 s on lock 0x7b5400216bd4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:00:26.798319 (+   105us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2438766336}
I20260709 13:00:26.799499  1605 raft_consensus.cc:1275] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 8. (term mismatch)
I20260709 13:00:26.799961  1605 pending_rounds.cc:85] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a: Aborting all ops after (but not including) 7
I20260709 13:00:26.800212  1605 pending_rounds.cc:107] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a: Aborting uncommitted NO_OP operation due to leader change: 7.8
I20260709 13:00:26.800510  1605 raft_consensus.cc:2889] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:00:26.801904  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:26.803357  1676 raft_consensus.cc:3055] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:00:26.803727  1676 raft_consensus.cc:740] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:00:26.776026  2329 raft_consensus.cc:2410] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for term 8 because replica has last-logged OpId of term: 6 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 8.
W20260709 13:00:26.797250  1600 rpcz_store.cc:269] 0709 13:00:23.777814 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:24.181835 (+404021us) service_pool.cc:224] Handling call
0709 13:00:26.796481 (+2614646us) spinlock_profiling.cc:243] Waited 904 ms on lock 0x7b54002a3354. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:00:26.796613 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2079241088}
I20260709 13:00:26.804347  1604 raft_consensus.cc:1275] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 13:00:26.809684  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:26.767215  1672 pending_rounds.cc:85] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d: Aborting all ops after (but not including) 5
I20260709 13:00:26.812889  1672 pending_rounds.cc:107] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d: Aborting uncommitted NO_OP operation due to leader change: 9.6
I20260709 13:00:26.813165  1672 raft_consensus.cc:2889] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 13:00:26.764240  5212 consensus_meta.cc:287] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 1.061s	user 0.011s	sys 0.004s
I20260709 13:00:26.826799  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:26.804590  1676 consensus_queue.cc:260] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:00:26.790845  5222 log.cc:927] Time spent T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.886s	user 0.000s	sys 0.004s
W20260709 13:00:26.831048  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 88635.6us
I20260709 13:00:26.765326  5086 raft_consensus.cc:2804] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:00:26.829311  2337 raft_consensus.cc:2468] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 5.
W20260709 13:00:26.865787  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:26.762931  5252 raft_consensus.cc:2804] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:00:26.919090  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:26.789569  5007 tablet_copy_client.cc:806] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Starting download of 0 data blocks...
I20260709 13:00:26.929001  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:26.810566  5145 raft_consensus.cc:493] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:00:26.926075  1644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:26.934736  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:26.871498  5086 raft_consensus.cc:493] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
W20260709 13:00:26.927026  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 103015us
I20260709 13:00:26.938488  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:26.939287  1600 raft_consensus.cc:3060] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:00:26.839184  5172 consensus_queue.cc:1059] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.097s
I20260709 13:00:26.943310  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:26.839375  4746 consensus_queue.cc:1048] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.103s
I20260709 13:00:26.839241  4580 consensus_queue.cc:1048] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.100s
I20260709 13:00:26.925554  5252 raft_consensus.cc:493] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
W20260709 13:00:26.935827  5182 consensus_meta.cc:287] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 1.592s	user 0.013s	sys 0.004s
W20260709 13:00:26.938657  5264 log.cc:927] Time spent T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.181s	user 0.000s	sys 0.001s
I20260709 13:00:26.986397  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:26.865407  5212 raft_consensus.cc:515] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:26.994874  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:26.988451  1599 raft_consensus.cc:1275] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 13:00:27.031373  5176 leader_election.cc:290] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:26.995664  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:26.998860  1600 raft_consensus.cc:2468] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 8.
I20260709 13:00:26.927521  5196 raft_consensus.cc:493] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:00:27.004168  1597 raft_consensus.cc:2468] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 10.
I20260709 13:00:27.005276  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:27.006242  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:26.827471  5262 raft_consensus.cc:493] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:27.006836  1596 raft_consensus.cc:1240] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 7. Current term is 8. Ops: []
I20260709 13:00:26.935674  5007 tablet_copy_client.cc:670] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Starting download of 1 WAL segments...
W20260709 13:00:26.954779  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 87496us
I20260709 13:00:26.940586  5086 raft_consensus.cc:3060] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:00:26.923069  4978 raft_consensus.cc:515] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:27.075088  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 8 candidate_status { last_received { term: 6 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:27.030421  1602 raft_consensus.cc:1240] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 8. Current term is 9. Ops: []
I20260709 13:00:27.034698  1745 raft_consensus.cc:1240] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 6. Current term is 7. Ops: []
I20260709 13:00:27.072897  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:27.031179  5252 raft_consensus.cc:3060] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:00:27.096463  1599 raft_consensus.cc:2393] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 8: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:00:27.042012  2334 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49082 (request call id 763) took 1532 ms. Trace:
I20260709 13:00:27.097059  2334 rpcz_store.cc:276] 0709 13:00:25.509097 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:25.519223 (+ 10126us) service_pool.cc:224] Handling call
0709 13:00:25.519691 (+   468us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:00:26.388269 (+868578us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:27.041621 (+653352us) spinlock_profiling.cc:243] Waited 776 ms on lock 0x7b54005bca54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:00:27.041908 (+   287us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1785185152}
W20260709 13:00:27.044921  1678 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52778 (request call id 1911) took 1912 ms (client timeout 1896 ms). Trace:
I20260709 13:00:27.071720  1604 raft_consensus.cc:3060] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:00:27.073529  1601 raft_consensus.cc:2468] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 8.
W20260709 13:00:27.098613  1678 rpcz_store.cc:269] 0709 13:00:25.132108 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:25.151920 (+ 19812us) service_pool.cc:224] Handling call
0709 13:00:27.044658 (+1892738us) spinlock_profiling.cc:243] Waited 1.27 s on lock 0x7b5400163854. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:00:27.044793 (+   135us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2911237888}
I20260709 13:00:27.073185  1675 raft_consensus.cc:1240] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 8. Current term is 9. Ops: [8.8-8.8]
W20260709 13:00:26.960425  5198 log.cc:927] Time spent T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.166s	user 0.000s	sys 0.001s
I20260709 13:00:27.074414  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:27.101981  1600 raft_consensus.cc:3060] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:00:27.089927  1747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55848 (request call id 2144) took 1047 ms. Trace:
I20260709 13:00:27.102423  1747 rpcz_store.cc:276] 0709 13:00:26.042507 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:26.671799 (+629292us) service_pool.cc:224] Handling call
0709 13:00:26.686108 (+ 14309us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:00:26.686633 (+   525us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:00:26.686675 (+    42us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:00:26.710628 (+ 23953us) log.cc:844] Serialized 52 byte log entry
0709 13:00:26.837886 (+127258us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:00:26.838307 (+   421us) log.cc:844] Serialized 14 byte log entry
0709 13:00:26.875466 (+ 37159us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:26.875657 (+   191us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:00:27.089720 (+214063us) raft_consensus.cc:1685] finished
0709 13:00:27.089742 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:27.089868 (+   126us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":36198,"prepare.queue_time_us":191,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":41,"raft.queue_time_us":83684,"raft.run_cpu_time_us":430,"raft.run_wall_time_us":21701,"thread_start_us":13734,"threads_started":1,"wal-append.queue_time_us":149064}
W20260709 13:00:27.095058  5221 consensus_meta.cc:287] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.630s	user 0.013s	sys 0.000s
I20260709 13:00:27.074196  5196 raft_consensus.cc:515] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:27.034402  5129 raft_consensus.cc:493] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:27.112365  5129 raft_consensus.cc:515] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:27.113373  4978 leader_election.cc:290] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:27.113745  5196 leader_election.cc:290] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:00:27.065359  5274 log.cc:927] Time spent T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.070s	user 0.000s	sys 0.000s
I20260709 13:00:27.095456  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:27.121161  1596 raft_consensus.cc:2393] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 9: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:00:26.938939  5145 raft_consensus.cc:515] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:27.075294  1603 raft_consensus.cc:2468] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 7.
I20260709 13:00:27.140666  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:27.073206  5212 leader_election.cc:290] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 9 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:27.106859  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:27.106426  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:27.141705  1601 raft_consensus.cc:2468] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 6.
I20260709 13:00:27.095321  1718 leader_election.cc:304] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:00:27.043553  4847 raft_consensus.cc:3055] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:00:27.037423  5223 leader_election.cc:290] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:27.146091  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:27.039038  5277 raft_consensus.cc:493] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
W20260709 13:00:27.152952  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 57742.3us
I20260709 13:00:27.158869  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:27.159168  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:27.159634  1596 raft_consensus.cc:2368] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for earlier term 9. Current term is 10.
I20260709 13:00:27.153097  2334 raft_consensus.cc:1240] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 7. Current term is 8. Ops: []
I20260709 13:00:27.005824  5189 leader_election.cc:290] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 7 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:27.154058  1678 raft_consensus.cc:1240] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 9 LEADER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 8. Current term is 9. Ops: []
I20260709 13:00:27.145840  5129 leader_election.cc:290] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:27.162919  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:27.162775  5277 raft_consensus.cc:515] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:27.164673  5271 log.cc:927] Time spent T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.208s	user 0.000s	sys 0.001s
I20260709 13:00:27.165200  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:27.166019  1605 raft_consensus.cc:2410] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for term 6 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260709 13:00:27.167115  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:27.035347  4746 consensus_queue.cc:1048] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.118s
I20260709 13:00:27.081794  5182 raft_consensus.cc:515] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:27.168262  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:27.119856  5232 raft_consensus.cc:493] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:27.096581  5262 raft_consensus.cc:515] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:27.155076  4847 raft_consensus.cc:740] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
W20260709 13:00:27.178783  5279 log.cc:927] Time spent T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.063s	user 0.000s	sys 0.001s
I20260709 13:00:27.121526  5221 raft_consensus.cc:515] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:27.182330  1604 raft_consensus.cc:2468] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 8.
I20260709 13:00:27.184679  5210 leader_election.cc:290] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:27.119585  5228 consensus_queue.cc:1048] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.066s
I20260709 13:00:27.116333  5158 consensus_queue.cc:1048] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.069s
I20260709 13:00:27.188169  1676 raft_consensus.cc:3060] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:00:27.190594  1603 raft_consensus.cc:3060] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:00:27.167708  5145 leader_election.cc:290] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:27.194911  1599 raft_consensus.cc:2468] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 10.
I20260709 13:00:27.220609  2332 raft_consensus.cc:1240] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 7. Current term is 8. Ops: [7.8-7.8]
I20260709 13:00:27.033124  5205 consensus_queue.cc:1059] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.140s
W20260709 13:00:27.197703  1643 leader_election.cc:336] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 1.585s (SENT)
I20260709 13:00:27.197911  1600 raft_consensus.cc:2468] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 6.
I20260709 13:00:27.199872  1718 leader_election.cc:304] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:00:27.232904  1643 leader_election.cc:304] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:27.201813  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 8 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:27.233572  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:27.203261  1601 raft_consensus.cc:2468] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 6.
I20260709 13:00:27.234582  1600 raft_consensus.cc:2410] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for term 11 because replica has last-logged OpId of term: 9 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 5.
I20260709 13:00:27.194280  5123 raft_consensus.cc:2804] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:00:27.217675  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:27.237790  1604 raft_consensus.cc:2393] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 8: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:00:27.222947  1678 raft_consensus.cc:1240] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 9. Current term is 10. Ops: []
I20260709 13:00:27.236508  1596 raft_consensus.cc:1275] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260709 13:00:27.210969  5232 raft_consensus.cc:515] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:27.243348  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:27.244270  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:27.247820  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:27.247985  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:27.248685  1600 raft_consensus.cc:2468] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 5.
I20260709 13:00:27.250453  1601 raft_consensus.cc:2393] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 9: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
W20260709 13:00:27.256340  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 58664.4us
I20260709 13:00:27.237186  5123 raft_consensus.cc:493] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:27.267027  1598 raft_consensus.cc:2468] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 7.
I20260709 13:00:27.264195  1600 raft_consensus.cc:1275] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260709 13:00:27.265553  1643 leader_election.cc:304] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:27.268654  1717 leader_election.cc:304] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:27.274205  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:27.275020  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:27.275662  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:27.273144  1643 leader_election.cc:304] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:00:27.273020  1717 leader_election.cc:304] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:00:27.285784  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:27.256436  4580 raft_consensus.cc:2749] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:00:27.255997  5223 raft_consensus.cc:493] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:27.286460  5277 leader_election.cc:290] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:27.291239  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:27.330024  5094 consensus_queue.cc:260] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:27.331393  5094 raft_consensus.cc:3060] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:00:27.333549  5232 leader_election.cc:290] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:27.330598  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:27.318964  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:27.331334  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:27.329015  5290 raft_consensus.cc:2749] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 13:00:27.285073  1603 raft_consensus.cc:2468] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 8.
W20260709 13:00:27.292476  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 184263us
I20260709 13:00:27.290870  1749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43156 (request call id 625) took 1018 ms. Trace:
I20260709 13:00:27.271534  5123 raft_consensus.cc:3060] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:00:27.342669  2298 leader_election.cc:304] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:00:27.339408  1749 rpcz_store.cc:276] 0709 13:00:26.272709 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:26.795029 (+522320us) service_pool.cc:224] Handling call
0709 13:00:26.795347 (+   318us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:00:26.795838 (+   491us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:00:26.795876 (+    38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:00:26.796509 (+   633us) log.cc:844] Serialized 26 byte log entry
0709 13:00:26.932654 (+136145us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:00:26.932720 (+    66us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:26.932822 (+   102us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:00:27.290635 (+357813us) raft_consensus.cc:1685] finished
0709 13:00:27.290659 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:27.290793 (+   134us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1436,"raft.run_cpu_time_us":132,"raft.run_wall_time_us":129,"thread_start_us":135727,"threads_started":1,"wal-append.queue_time_us":290797}
I20260709 13:00:27.278702  5205 raft_consensus.cc:2749] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:00:27.356952  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:00:27.338721  5215 consensus_meta.cc:287] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.676s	user 0.015s	sys 0.000s
I20260709 13:00:27.278335  5285 raft_consensus.cc:493] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:27.286293  5182 leader_election.cc:290] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 8 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:27.250499  4847 consensus_queue.cc:260] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.10, Last appended by leader: 10, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:27.361534  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 8 candidate_status { last_received { term: 6 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:27.287696  4746 raft_consensus.cc:2804] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:00:27.372479  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:27.375037  1678 raft_consensus.cc:2468] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 7.
I20260709 13:00:27.355124  5228 raft_consensus.cc:2804] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 13:00:27.383669  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:27.384160  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:27.363778  5290 raft_consensus.cc:2804] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:00:27.382247  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:27.372242  5149 raft_consensus.cc:493] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:00:27.387877  5149 raft_consensus.cc:515] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } }
I20260709 13:00:27.345788  5277 consensus_queue.cc:1048] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.031s
I20260709 13:00:27.354537  4759 consensus_queue.cc:1048] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.026s
I20260709 13:00:27.349395  5210 consensus_queue.cc:1059] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.031s
W20260709 13:00:27.394215  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 184692us
I20260709 13:00:27.340106  5223 raft_consensus.cc:515] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:27.376483  4746 raft_consensus.cc:493] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:27.364166  5086 raft_consensus.cc:515] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:27.400691  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 127746us
I20260709 13:00:27.392110  5290 raft_consensus.cc:493] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:27.402832  1717 leader_election.cc:304] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:00:27.391410  5228 raft_consensus.cc:697] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 10 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:00:27.405246  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:27.403680  2299 leader_election.cc:304] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 13:00:27.407298  1671 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52778 (request call id 1900) took 2534 ms (client timeout 2089 ms). Trace:
I20260709 13:00:27.407296  1603 raft_consensus.cc:3060] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:00:27.409355  1673 raft_consensus.cc:1275] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260709 13:00:27.408128  1717 leader_election.cc:304] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
W20260709 13:00:27.409819  1671 rpcz_store.cc:269] 0709 13:00:24.869395 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:24.875124 (+  5729us) service_pool.cc:224] Handling call
0709 13:00:27.403725 (+2528601us) spinlock_profiling.cc:243] Waited 977 ms on lock 0x7b54001e87d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:00:27.403859 (+   134us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2247572224}
I20260709 13:00:27.402889  4746 raft_consensus.cc:3060] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:00:27.412060  5079 raft_consensus.cc:2804] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:00:27.412454  5079 raft_consensus.cc:493] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:00:27.412632  5291 raft_consensus.cc:2804] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 13:00:27.412842  5079 raft_consensus.cc:3060] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:00:27.413275  5291 raft_consensus.cc:697] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 8 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:00:27.418735  5086 leader_election.cc:290] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:27.400208  5262 leader_election.cc:290] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:27.419029  5228 consensus_queue.cc:237] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:27.403038  4847 raft_consensus.cc:3060] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:00:27.423382  5215 raft_consensus.cc:3055] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:00:27.423794  5215 raft_consensus.cc:740] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:00:27.425097  5149 leader_election.cc:290] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:27.425681  5221 leader_election.cc:290] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 8 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:27.426890  5223 leader_election.cc:290] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:27.418781  5214 consensus_queue.cc:260] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:27.427618  5189 raft_consensus.cc:3055] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:00:27.428056  5189 raft_consensus.cc:740] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:00:27.428951  5189 consensus_queue.cc:260] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.10, Last appended by leader: 10, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:27.384167  5285 raft_consensus.cc:515] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:27.405381  5290 raft_consensus.cc:3060] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:00:27.437784  1675 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57092 (request call id 746) took 1689 ms. Trace:
I20260709 13:00:27.438213  1675 rpcz_store.cc:276] 0709 13:00:25.748059 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:25.879726 (+131667us) service_pool.cc:224] Handling call
0709 13:00:25.880118 (+   392us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:00:27.106875 (+1226757us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:27.437447 (+330572us) spinlock_profiling.cc:243] Waited 1.19 s on lock 0x7b540025b054. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:00:27.437704 (+   257us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2742841728}
I20260709 13:00:27.440366  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:00:27.441283  2337 raft_consensus.cc:2393] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 8: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:00:27.443182  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:27.443838  2329 raft_consensus.cc:2468] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 5.
I20260709 13:00:27.449405  1602 raft_consensus.cc:1240] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 7. Current term is 8. Ops: []
I20260709 13:00:27.424739  5215 consensus_queue.cc:260] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:27.430214  5189 raft_consensus.cc:3060] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:00:27.453014  1674 raft_consensus.cc:1275] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 13:00:27.453531  1676 raft_consensus.cc:1275] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260709 13:00:27.442282  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:27.414340  5291 consensus_queue.cc:237] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:00:27.464684  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 61080.9us
I20260709 13:00:27.467682  5158 consensus_queue.cc:1059] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 13:00:27.465696  5210 raft_consensus.cc:2749] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 13:00:27.475404  5285 leader_election.cc:290] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:27.479216  2299 leader_election.cc:304] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:27.459290  5083 consensus_queue.cc:1059] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.024s
I20260709 13:00:27.481568  5215 raft_consensus.cc:3060] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:00:27.488725  4746 raft_consensus.cc:515] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:27.482924  5210 raft_consensus.cc:2749] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 13:00:27.491446  5123 raft_consensus.cc:515] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:00:27.510074  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:00:27.513548  4549 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 7 current term: 7 is higher.
I20260709 13:00:27.514845  1748 raft_consensus.cc:1275] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
W20260709 13:00:27.513629  1750 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43156 (request call id 606) took 3469 ms (client timeout 1821 ms). Trace:
W20260709 13:00:27.516497  1750 rpcz_store.cc:269] 0709 13:00:24.043822 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:24.048079 (+  4257us) service_pool.cc:224] Handling call
0709 13:00:27.513300 (+3465221us) spinlock_profiling.cc:243] Waited 1.31 s on lock 0x7b5400385254. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:00:27.513507 (+   207us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3021350144}
I20260709 13:00:27.517709  1603 raft_consensus.cc:2468] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 8.
I20260709 13:00:27.520364  5123 leader_election.cc:290] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 7 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:27.524573  5123 raft_consensus.cc:493] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:00:27.515743  4746 leader_election.cc:290] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 11 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:27.523072  2332 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49082 (request call id 766) took 2007 ms. Trace:
I20260709 13:00:27.526281  2332 rpcz_store.cc:276] 0709 13:00:25.515486 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:25.526156 (+ 10670us) service_pool.cc:224] Handling call
0709 13:00:25.526545 (+   389us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:00:27.221201 (+1694656us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:27.522763 (+301562us) spinlock_profiling.cc:243] Waited 1.69 s on lock 0x7b54005046d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:00:27.522985 (+   222us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3894748928}
I20260709 13:00:27.527696  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:00:27.528306  2332 raft_consensus.cc:2376] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 11. Re-sending same reply.
I20260709 13:00:27.529613  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:00:27.525204  5123 raft_consensus.cc:515] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:27.525167  5252 raft_consensus.cc:515] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:27.532208  5123 leader_election.cc:290] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 13:00:27.532768  5267 log.cc:927] Time spent T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.665s	user 0.000s	sys 0.003s
I20260709 13:00:27.533483  5252 leader_election.cc:290] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 7 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:27.534534  5210 raft_consensus.cc:3055] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:00:27.534951  5210 raft_consensus.cc:740] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:00:27.535912  5210 consensus_queue.cc:260] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:27.537230  5210 raft_consensus.cc:3060] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Advancing to term 9
W20260709 13:00:27.537281  2335 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39738 (request call id 731) took 1786 ms (client timeout 2173 ms). Trace:
I20260709 13:00:27.538028  5290 raft_consensus.cc:515] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:27.538563  2331 raft_consensus.cc:1240] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 7. Current term is 8. Ops: [5.7-7.8]
W20260709 13:00:27.541513  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 44528.5us
I20260709 13:00:27.543620  2298 leader_election.cc:304] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
W20260709 13:00:27.537807  2335 rpcz_store.cc:269] 0709 13:00:25.751256 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:25.870178 (+118922us) service_pool.cc:224] Handling call
0709 13:00:27.537031 (+1666853us) spinlock_profiling.cc:243] Waited 637 ms on lock 0x7b54004e02d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:00:27.537199 (+   168us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1464265472}
I20260709 13:00:27.482661  5094 consensus_queue.cc:1059] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.049s
I20260709 13:00:27.547739  2301 leader_election.cc:304] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
W20260709 13:00:27.553789  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 172674us
I20260709 13:00:27.557973  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:27.571084  1753 raft_consensus.cc:2393] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 6: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:00:27.565178  1569 leader_election.cc:304] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:00:27.560058  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:27.574929  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:27.575369  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:27.575884  1676 raft_consensus.cc:2468] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 5.
I20260709 13:00:27.572315  1718 leader_election.cc:304] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:00:27.577199  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:27.578286  1750 raft_consensus.cc:2393] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 8: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:00:27.580129  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:27.581703  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:27.576090  1749 raft_consensus.cc:2468] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 7.
I20260709 13:00:27.585223  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:27.589088  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:27.549504  5221 raft_consensus.cc:2804] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 13:00:27.589951  1745 raft_consensus.cc:2393] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 6: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:00:27.590241  1672 raft_consensus.cc:2393] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 5: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:00:27.590121  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:00:27.597002  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 188912us
I20260709 13:00:27.595317  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:27.598301  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:27.598770  1753 raft_consensus.cc:2393] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 7: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:00:27.596450  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:27.571414  4847 raft_consensus.cc:493] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:00:27.597851  1717 leader_election.cc:304] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:00:27.600273  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:27.597195  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:27.608862  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:27.609321  1750 raft_consensus.cc:2393] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 9: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:00:27.601307  1745 raft_consensus.cc:2393] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 10: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:00:27.601593  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:27.610672  1675 raft_consensus.cc:3060] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:00:27.610875  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:27.601765  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:27.589344  5123 raft_consensus.cc:2804] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:00:27.606927  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:27.576638  5247 raft_consensus.cc:2749] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:00:27.602671  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:27.572793  5296 raft_consensus.cc:2749] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:00:27.612145  1676 raft_consensus.cc:2410] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for term 8 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
I20260709 13:00:27.612825  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:27.618741  1747 raft_consensus.cc:2393] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 7: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:00:27.609876  5252 raft_consensus.cc:493] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:27.620050  5252 raft_consensus.cc:515] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:27.627444  5137 raft_consensus.cc:2804] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 13:00:27.628711  5214 raft_consensus.cc:3060] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:00:27.579699  5260 consensus_queue.cc:1048] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.033s
I20260709 13:00:27.614984  1717 leader_election.cc:304] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 13:00:27.614385  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:27.637475  1718 leader_election.cc:304] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:27.639753  2329 raft_consensus.cc:1275] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 6 index: 9. Preceding OpId from leader: term: 7 index: 9. (term mismatch)
I20260709 13:00:27.640161  2329 pending_rounds.cc:85] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1: Aborting all ops after (but not including) 8
I20260709 13:00:27.640376  2329 pending_rounds.cc:107] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1: Aborting uncommitted NO_OP operation due to leader change: 6.9
I20260709 13:00:27.640623  2329 raft_consensus.cc:2889] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:00:27.642581  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:27.615700  5123 raft_consensus.cc:493] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:00:27.644341  5123 raft_consensus.cc:3060] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:00:27.643347  1749 raft_consensus.cc:2468] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 5.
I20260709 13:00:27.646636  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:27.648319  4779 leader_election.cc:290] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:27.649132  5079 raft_consensus.cc:515] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:27.641892  5137 raft_consensus.cc:697] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 6 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:00:27.651079  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 8 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:27.652001  1753 raft_consensus.cc:2410] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for term 8 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 4.
I20260709 13:00:27.652123  5137 consensus_queue.cc:237] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:27.618511  4847 raft_consensus.cc:515] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:27.657356  4847 leader_election.cc:290] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:00:27.660148  1645 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:27.660247  1597 raft_consensus.cc:1240] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 8. Current term is 9. Ops: [8.8-8.8]
I20260709 13:00:27.652832  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:27.669035  1675 raft_consensus.cc:2468] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 7.
I20260709 13:00:27.670172  1753 raft_consensus.cc:2410] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for term 9 because replica has last-logged OpId of term: 8 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 8.
I20260709 13:00:27.672791  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:27.673388  1749 raft_consensus.cc:3060] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:00:27.674103  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:27.674350  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:27.675498  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:27.676764  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:27.675849  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:27.680586  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:27.680753  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:27.680847  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:27.681525  1675 raft_consensus.cc:3060] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:00:27.681866  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:27.681965  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:27.682170  5303 raft_consensus.cc:2764] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
W20260709 13:00:27.683465  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 46129us
I20260709 13:00:27.684370  1718 leader_election.cc:304] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:27.685146  5223 raft_consensus.cc:2749] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:00:27.685886  1718 leader_election.cc:304] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:00:27.686429  5223 raft_consensus.cc:3060] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:00:27.669094  5036 raft_consensus.cc:493] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:00:27.687327  5036 raft_consensus.cc:515] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:27.690104  1673 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52778 (request call id 1915) took 1252 ms. Trace:
I20260709 13:00:27.690534  1673 rpcz_store.cc:276] 0709 13:00:26.438054 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:26.441210 (+  3156us) service_pool.cc:224] Handling call
0709 13:00:27.403991 (+962781us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:27.410031 (+  6040us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:27.689896 (+279865us) spinlock_profiling.cc:243] Waited 952 ms on lock 0x7b54001e87d0. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661618ad 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:00:27.690035 (+   139us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2190545792}
I20260709 13:00:27.692096  2331 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39726 (request call id 914) took 1033 ms. Trace:
I20260709 13:00:27.692493  2331 rpcz_store.cc:276] 0709 13:00:26.658622 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:26.658792 (+   170us) service_pool.cc:224] Handling call
0709 13:00:27.538169 (+879377us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:00:27.539028 (+   859us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:27.691867 (+152839us) spinlock_profiling.cc:243] Waited 879 ms on lock 0x7b54004e02d0. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661618ad 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:00:27.692026 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2020956032}
I20260709 13:00:27.683537  1678 raft_consensus.cc:1275] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 5 index: 8. Preceding OpId from leader: term: 7 index: 9. (index mismatch)
I20260709 13:00:27.695005  2336 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39726 (request call id 911) took 2197 ms. Trace:
I20260709 13:00:27.698264  2332 raft_consensus.cc:2468] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 8.
I20260709 13:00:27.695405  2336 rpcz_store.cc:276] 0709 13:00:25.497916 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:25.498037 (+   121us) service_pool.cc:224] Handling call
0709 13:00:26.387845 (+889808us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:26.530141 (+142296us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:27.694789 (+1164648us) spinlock_profiling.cc:243] Waited 872 ms on lock 0x7b5400605250. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661618ad 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:00:27.694919 (+   130us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2006358784}
I20260709 13:00:27.696868  1718 leader_election.cc:304] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
W20260709 13:00:27.702147  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 87242.7us
I20260709 13:00:27.711210  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:27.714140  1675 raft_consensus.cc:2468] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 8.
I20260709 13:00:27.714242  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:27.673311  5221 raft_consensus.cc:697] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 8 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:00:27.690270  5036 leader_election.cc:290] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:27.726845  1718 leader_election.cc:304] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:00:27.731016  5149 consensus_queue.cc:1048] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 13:00:27.702553  5176 raft_consensus.cc:2749] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 13:00:27.738993  1569 leader_election.cc:304] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:00:27.702713  5252 leader_election.cc:290] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:27.748360  1569 leader_election.cc:304] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:00:27.734067  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:27.721263  5223 raft_consensus.cc:2749] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:00:27.750810  1675 raft_consensus.cc:2393] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 9: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:00:27.734880  5277 raft_consensus.cc:2804] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Leader election won for term 7
W20260709 13:00:27.755748  2298 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:27.726095  4985 raft_consensus.cc:493] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:00:27.751035  4847 raft_consensus.cc:2804] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:00:27.728905  5152 raft_consensus.cc:2804] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260709 13:00:27.759806  2298 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:00:27.762744  2298 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:00:27.763247  2298 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:27.760393  2330 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49082 (request call id 767) took 1367 ms. Trace:
I20260709 13:00:27.750844  1569 leader_election.cc:304] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
W20260709 13:00:27.761703  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 177164us
I20260709 13:00:27.713233  5303 consensus_queue.cc:1048] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.043s
I20260709 13:00:27.753578  4547 raft_consensus.cc:2804] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:00:27.766906  2301 leader_election.cc:304] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:00:27.768008  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:27.765156  2330 rpcz_store.cc:276] 0709 13:00:26.393045 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:26.393274 (+   229us) service_pool.cc:224] Handling call
0709 13:00:26.421761 (+ 28487us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:26.422042 (+   281us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 13:00:26.708890 (+286848us) log.cc:844] Serialized 14 byte log entry
0709 13:00:27.685595 (+976705us) spinlock_profiling.cc:243] Waited 17.2 ms on lock 0x7b1800342d60. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f4263057e07 00007f426beb2ab9 00007f426307469b 00007f426307293f 00007f4263074c02 00007f4265cfc20b 00007f4265d01d23 00007f4265d0223a 00007f426616ca2a
0709 13:00:27.759189 (+ 73594us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:00:27.759256 (+    67us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:00:27.760122 (+   866us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:27.760197 (+    75us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:27.760328 (+   131us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":286356,"prepare.run_cpu_time_us":140,"prepare.run_wall_time_us":15690,"raft.queue_time_us":2500,"spinlock_wait_cycles":39509760,"thread_start_us":317182,"threads_started":2}
I20260709 13:00:27.775211  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:27.775725  2301 leader_election.cc:304] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:00:27.776263  1671 raft_consensus.cc:2376] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 8. Re-sending same reply.
I20260709 13:00:27.776773  5296 raft_consensus.cc:2749] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 13:00:27.779116  2301 leader_election.cc:304] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:27.763938  4847 raft_consensus.cc:493] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:27.764598  5152 raft_consensus.cc:493] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:00:27.770004  5197 raft_consensus.cc:2804] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader election won for term 10
W20260709 13:00:27.777995  5238 log.cc:927] Time spent T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 1.702s	user 0.006s	sys 0.000s
I20260709 13:00:27.773425  5215 raft_consensus.cc:2749] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 13:00:27.770349  5277 raft_consensus.cc:697] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 7 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:00:27.802042  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:27.797180  5290 leader_election.cc:290] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 9 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:27.805349  5317 raft_consensus.cc:493] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:27.780931  4547 raft_consensus.cc:697] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:00:27.806737  4547 consensus_queue.cc:237] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:27.806044  5317 raft_consensus.cc:515] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:27.809881  5317 leader_election.cc:290] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:27.784148  5145 consensus_queue.cc:1059] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.028s
I20260709 13:00:27.812659  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:27.812762  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:27.814855  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:00:27.815271  2330 raft_consensus.cc:1275] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 13:00:27.816087  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:00:27.765842  4985 raft_consensus.cc:515] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:27.795431  5309 raft_consensus.cc:2804] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:00:27.794922  4847 raft_consensus.cc:3060] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:00:27.794018  5152 raft_consensus.cc:3060] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Advancing to term 6
W20260709 13:00:27.834944  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 54706.2us
I20260709 13:00:27.818841  5197 raft_consensus.cc:697] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:00:27.847898  5262 consensus_queue.cc:1048] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 13:00:27.847970  5277 consensus_queue.cc:237] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:27.836184  5309 raft_consensus.cc:493] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:27.846911  1569 leader_election.cc:304] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
W20260709 13:00:27.857493  1677 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52778 (request call id 1912) took 2725 ms (client timeout 2714 ms). Trace:
I20260709 13:00:27.804406  5247 consensus_queue.cc:1048] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.038s
I20260709 13:00:27.839610  4896 raft_consensus.cc:3055] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 LEADER]: Stepping down as leader of term 6
W20260709 13:00:27.858135  1677 rpcz_store.cc:269] 0709 13:00:25.132453 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:25.152071 (+ 19618us) service_pool.cc:224] Handling call
0709 13:00:27.857219 (+2705148us) spinlock_profiling.cc:243] Waited 375 ms on lock 0x7b54000fda54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:00:27.857387 (+   168us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":862704000}
I20260709 13:00:27.851019  5309 raft_consensus.cc:3060] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:00:27.870218  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:27.871027  1676 raft_consensus.cc:2468] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 8.
I20260709 13:00:27.871188  1569 leader_election.cc:304] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:00:27.799019  5221 consensus_queue.cc:237] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:27.877269  2333 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49082 (request call id 764) took 2364 ms. Trace:
I20260709 13:00:27.878309  2333 rpcz_store.cc:276] 0709 13:00:25.513370 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:25.513519 (+   149us) service_pool.cc:224] Handling call
0709 13:00:25.525014 (+ 11495us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:00:25.724268 (+199254us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:00:25.724304 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:00:25.725024 (+   720us) log.cc:844] Serialized 26 byte log entry
0709 13:00:25.957396 (+232372us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:00:25.957449 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:25.957536 (+    87us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:00:27.686355 (+1728819us) spinlock_profiling.cc:243] Waited 2.7 ms on lock 0x7b4400022b4c. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f4262f9bc85 00007f4262f9c6a2 00007f4265d0e1db 00007f4265d09aaf 00007f4265cfd86e 00007f4265cfcbb6 00007f4265cfc7d8 00007f4265d06221 00007f4265d061d9 00007f4265d06169
0709 13:00:27.873961 (+187606us) raft_consensus.cc:1685] finished
0709 13:00:27.873987 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:27.877181 (+  3194us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":67294,"raft.queue_time_us":97110,"raft.run_cpu_time_us":180,"raft.run_wall_time_us":3320,"spinlock_wait_cycles":6205184,"thread_start_us":391139,"threads_started":2,"wal-append.queue_time_us":221249}
I20260709 13:00:27.863184  4896 raft_consensus.cc:740] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:00:27.882449  4896 consensus_queue.cc:260] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:27.883483  5233 consensus_queue.cc:1059] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 13:00:27.884541  5233 raft_consensus.cc:3055] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:00:27.885010  5233 raft_consensus.cc:740] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:00:27.886127  5233 consensus_queue.cc:260] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:27.863890  5262 raft_consensus.cc:2749] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 13:00:27.892278  5233 raft_consensus.cc:3060] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:00:27.899688  5007 tablet_copy_client.cc:538] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20260709 13:00:27.903638  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:27.924031  1749 raft_consensus.cc:2468] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 8.
I20260709 13:00:27.921030  4896 raft_consensus.cc:3060] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:00:27.938118  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:27.939062  1749 raft_consensus.cc:2468] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 9.
I20260709 13:00:27.929442  4547 raft_consensus.cc:2804] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Leader election won for term 11
I20260709 13:00:27.949564  2301 leader_election.cc:304] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:00:27.957599  1598 raft_consensus.cc:1275] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260709 13:00:27.930932  5262 consensus_queue.cc:1048] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.025s
I20260709 13:00:27.968077  5123 raft_consensus.cc:515] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:27.956164  5197 consensus_queue.cc:237] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:27.971710  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:27.974505  5260 raft_consensus.cc:2804] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:00:27.975033  5260 raft_consensus.cc:493] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:00:27.975205  5305 log.cc:927] Time spent T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.365s	user 0.001s	sys 0.002s
I20260709 13:00:27.975478  5260 raft_consensus.cc:3060] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:00:27.981691  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:27.982342  1753 raft_consensus.cc:3060] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:00:27.994206  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:27.995059  1749 raft_consensus.cc:2468] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 7.
I20260709 13:00:27.996143  5123 leader_election.cc:290] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 6 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:27.960320  4547 raft_consensus.cc:697] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:00:27.986068  5317 consensus_queue.cc:1048] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.017s
I20260709 13:00:27.997645  4547 consensus_queue.cc:237] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:28.008579  1746 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43156 (request call id 617) took 2634 ms. Trace:
I20260709 13:00:28.004175  4985 leader_election.cc:290] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:28.009006  1746 rpcz_store.cc:276] 0709 13:00:25.373990 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:25.455739 (+ 81749us) service_pool.cc:224] Handling call
0709 13:00:25.456235 (+   496us) raft_consensus.cc:1471] Updating replica for 4 ops
0709 13:00:25.490173 (+ 33938us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:00:25.490260 (+    87us) raft_consensus.cc:1539] Triggering prepare for 4 ops
0709 13:00:25.511863 (+ 21603us) log.cc:844] Serialized 104 byte log entry
0709 13:00:25.676079 (+164216us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:00:25.943546 (+267467us) log.cc:844] Serialized 14 byte log entry
0709 13:00:25.948316 (+  4770us) log.cc:844] Serialized 14 byte log entry
0709 13:00:26.600034 (+651718us) spinlock_profiling.cc:243] Waited 2.57 ms on lock 0x7b18000f7180. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f4262fefc0a 00007f426c72f219 00007f4263077a40 00007f42630724e4 00007f4263072455 00007f4269b5a3c7 00007f426616c9a7 00007f426617c0bf 00007f426617c038
0709 13:00:26.600274 (+   240us) log.cc:844] Serialized 14 byte log entry
0709 13:00:26.600552 (+   278us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:26.600770 (+   218us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:00:27.053652 (+452882us) spinlock_profiling.cc:243] Waited 15.1 ms on lock 0x7b18000f7180. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f42630753f3 00007f42630777c1 00007f4263077779 00007f4263077709 00007f42630776d1 00007f42630769cd 000000000038f354 000000000038f199 00007f426305883a
0709 13:00:27.624636 (+570984us) spinlock_profiling.cc:243] Waited 8.28 ms on lock 0x7b18000f7180. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f42630753f3 00007f42630777c1 00007f4263077779 00007f4263077709 00007f42630776d1 00007f42630769cd 000000000038f354 000000000038f199 00007f426305883a
0709 13:00:27.753160 (+128524us) spinlock_profiling.cc:243] Waited 5.26 ms on lock 0x7b18000f7180. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f4263075522 00007f42630777c1 00007f4263077779 00007f4263077709 00007f42630776d1 00007f42630769cd 000000000038f354 000000000038f199 00007f426305883a
0709 13:00:27.997524 (+244364us) spinlock_profiling.cc:243] Waited 13.8 ms on lock 0x7b18000f7180. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f42630753f3 00007f42630777c1 00007f4263077779 00007f4263077709 00007f42630776d1 00007f42630769cd 000000000038f354 000000000038f199 00007f426305883a
0709 13:00:28.008299 (+ 10775us) raft_consensus.cc:1685] finished
0709 13:00:28.008321 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:28.008505 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":658262,"prepare.queue_time_us":918029,"prepare.run_cpu_time_us":134,"prepare.run_wall_time_us":135,"raft.queue_time_us":29338,"raft.run_cpu_time_us":686,"raft.run_wall_time_us":21702,"spinlock_wait_cycles":103435520,"thread_start_us":311058,"threads_started":2,"wal-append.queue_time_us":406761}
I20260709 13:00:28.013588  5081 raft_consensus.cc:3060] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:00:28.013762  1644 leader_election.cc:304] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:28.017351  2337 raft_consensus.cc:1275] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 9 index: 8. (term mismatch)
I20260709 13:00:28.017838  2337 pending_rounds.cc:85] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1: Aborting all ops after (but not including) 7
I20260709 13:00:28.018077  2337 pending_rounds.cc:107] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1: Aborting uncommitted NO_OP operation due to leader change: 8.8
W20260709 13:00:28.018106  5254 log.cc:927] Time spent T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 1.431s	user 0.001s	sys 0.007s
I20260709 13:00:28.018427  2337 raft_consensus.cc:2889] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:00:28.018631  5172 raft_consensus.cc:2749] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:00:28.020543  1751 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55848 (request call id 2138) took 2036 ms. Trace:
I20260709 13:00:28.016336  1745 raft_consensus.cc:1275] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260709 13:00:28.022576  1644 leader_election.cc:304] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:00:28.024093  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:28.025192  2331 raft_consensus.cc:1800] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: voting while copying based on last-logged opid 4.5
I20260709 13:00:28.025712  2331 raft_consensus.cc:2468] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 6.
I20260709 13:00:28.034224  1748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43156 (request call id 623) took 1932 ms. Trace:
I20260709 13:00:28.034816  1748 rpcz_store.cc:276] 0709 13:00:26.101354 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:26.770669 (+669315us) service_pool.cc:224] Handling call
0709 13:00:27.514358 (+743689us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:27.515332 (+   974us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:28.033992 (+518660us) spinlock_profiling.cc:243] Waited 693 ms on lock 0x7b5400385250. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661618ad 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:00:28.034152 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1594984320}
I20260709 13:00:28.038590  1745 raft_consensus.cc:1240] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 7. Current term is 8. Ops: []
I20260709 13:00:28.041186  1748 raft_consensus.cc:1240] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 8. Current term is 9. Ops: []
I20260709 13:00:28.022668  5158 raft_consensus.cc:493] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:28.045212  2333 raft_consensus.cc:1240] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 7. Current term is 8. Ops: []
W20260709 13:00:28.047509  1643 leader_election.cc:336] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 1.668s (SENT)
I20260709 13:00:28.048053  1643 leader_election.cc:304] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:00:28.062922  1745 raft_consensus.cc:1240] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 9 LEADER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 8. Current term is 9. Ops: [8.8-8.8]
I20260709 13:00:28.084149  1748 raft_consensus.cc:1275] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260709 13:00:28.020920  1751 rpcz_store.cc:276] 0709 13:00:25.984322 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:25.995214 (+ 10892us) service_pool.cc:224] Handling call
0709 13:00:25.995575 (+   361us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:00:25.996029 (+   454us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:00:25.996062 (+    33us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:00:26.005363 (+  9301us) log.cc:844] Serialized 52 byte log entry
0709 13:00:26.390783 (+385420us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:00:26.675210 (+284427us) log.cc:844] Serialized 14 byte log entry
0709 13:00:26.678007 (+  2797us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:26.678086 (+    79us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:00:28.017433 (+1339347us) spinlock_profiling.cc:243] Waited 14.2 ms on lock 0x7b4400022b4c. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f4262f9bc85 00007f4262f9c6a2 00007f4265d0e1db 00007f4265d09aaf 00007f4265cfd86e 00007f4265cfcbb6 00007f4265cfc7d8 00007f4265d06221 00007f4265d061d9 00007f4265d06169
0709 13:00:28.020227 (+  2794us) raft_consensus.cc:1685] finished
0709 13:00:28.020327 (+   100us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:28.020476 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":242394,"prepare.queue_time_us":242838,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":45,"raft.queue_time_us":11770,"raft.run_cpu_time_us":353,"raft.run_wall_time_us":40222,"spinlock_wait_cycles":32631680,"thread_start_us":369681,"threads_started":2,"wal-append.queue_time_us":511648}
I20260709 13:00:28.092235  1753 raft_consensus.cc:2468] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 8.
I20260709 13:00:28.093568  1749 raft_consensus.cc:1240] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 7. Current term is 8. Ops: [4.6-7.8]
W20260709 13:00:28.085438  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 153775us
I20260709 13:00:28.102630  2331 raft_consensus.cc:1240] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 8. Current term is 9. Ops: []
I20260709 13:00:28.037122  5172 raft_consensus.cc:2804] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:00:28.117842  1596 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "a3accbb733bb4cafa69428319664f239"
dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
mode: GRACEFUL
new_leader_uuid: "af520b67945d443b901bbeca37cc5701"
 from {username='slave'} at 127.0.0.1:47752
I20260709 13:00:28.118536  1596 raft_consensus.cc:606] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 LEADER]: Received request to transfer leadership to TS af520b67945d443b901bbeca37cc5701
I20260709 13:00:28.118702  5214 raft_consensus.cc:493] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:28.044013  5158 raft_consensus.cc:515] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:28.144382  2298 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:28.145967  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:28.146616  2333 raft_consensus.cc:1800] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: voting while copying based on last-logged opid 4.5
I20260709 13:00:28.023970  4847 raft_consensus.cc:515] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:28.146976  2333 raft_consensus.cc:2468] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 6.
I20260709 13:00:28.119999  5172 raft_consensus.cc:493] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:28.119342  5214 raft_consensus.cc:515] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:28.040827  5036 consensus_queue.cc:1059] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 6, Time since last communication: 0.098s
W20260709 13:00:28.174443  2298 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:28.159744  5303 raft_consensus.cc:493] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:28.161062  5172 raft_consensus.cc:3060] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:00:28.148900  5262 consensus_queue.cc:1059] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.041s
W20260709 13:00:28.189505  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 147270us
W20260709 13:00:28.196442  2298 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:28.162288  5322 raft_consensus.cc:493] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:00:28.211301  2328 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39738 (request call id 729) took 2484 ms (client timeout 2131 ms). Trace:
W20260709 13:00:28.211954  2328 rpcz_store.cc:269] 0709 13:00:25.720734 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:25.870092 (+149358us) service_pool.cc:224] Handling call
0709 13:00:28.204903 (+2334811us) spinlock_profiling.cc:243] Waited 28.1 ms on lock 0x7b540065e0d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:00:28.205023 (+   120us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":64612352}
W20260709 13:00:28.214665  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 167181us
I20260709 13:00:28.164678  5309 raft_consensus.cc:515] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:28.199950  5123 raft_consensus.cc:493] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:00:28.242518  2330 raft_consensus.cc:1240] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 7. Current term is 8. Ops: []
I20260709 13:00:28.201571  5303 raft_consensus.cc:515] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:28.225528  5129 raft_consensus.cc:2749] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:00:28.207706  5322 raft_consensus.cc:515] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:28.258242  5323 log.cc:927] Time spent T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.121s	user 0.000s	sys 0.001s
I20260709 13:00:28.260480  5228 raft_consensus.cc:3055] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:00:28.264307  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:28.265578  1676 raft_consensus.cc:1275] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260709 13:00:28.270270  2299 leader_election.cc:304] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:28.281217  1752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55848 (request call id 2139) took 2296 ms. Trace:
I20260709 13:00:28.281764  1752 rpcz_store.cc:276] 0709 13:00:25.984629 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:25.993846 (+  9217us) service_pool.cc:224] Handling call
0709 13:00:25.994293 (+   447us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 13:00:26.015759 (+ 21466us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:00:26.015793 (+    34us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 13:00:26.059749 (+ 43956us) log.cc:844] Serialized 78 byte log entry
0709 13:00:27.942627 (+1882878us) spinlock_profiling.cc:243] Waited 2.42 ms on lock 0x7b1800352000. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f4263057e07 00007f426beb2ab9 00007f426307469b 00007f426307293f 00007f4263074c02 00007f4265cfc20b 00007f4265d01d23 00007f4265d020d1 00007f426611da0f
0709 13:00:28.097320 (+154693us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:00:28.215224 (+117904us) log.cc:844] Serialized 14 byte log entry
0709 13:00:28.220287 (+  5063us) log.cc:844] Serialized 14 byte log entry
0709 13:00:28.222283 (+  1996us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:28.222395 (+   112us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:00:28.280952 (+ 58557us) raft_consensus.cc:1685] finished
0709 13:00:28.280983 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:28.281142 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":36830,"raft.queue_time_us":62684,"raft.run_cpu_time_us":556,"raft.run_wall_time_us":44028,"spinlock_wait_cycles":13386240,"thread_start_us":217353,"threads_started":2,"wal-append.queue_time_us":2109434}
I20260709 13:00:28.286569  4779 raft_consensus.cc:2804] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:00:28.287056  4779 raft_consensus.cc:493] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:00:28.287493  4779 raft_consensus.cc:3060] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:00:28.288565  5007 tablet_bootstrap.cc:492] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1: Bootstrap starting.
I20260709 13:00:28.268999  5228 raft_consensus.cc:740] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:00:28.291917  2337 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39738 (request call id 740) took 1092 ms. Trace:
I20260709 13:00:28.292256  5228 consensus_queue.cc:260] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.8, Last appended by leader: 8, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:28.292328  2337 rpcz_store.cc:276] 0709 13:00:27.199422 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:27.478136 (+278714us) service_pool.cc:224] Handling call
0709 13:00:27.544072 (+ 65936us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:28.018986 (+474914us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:28.291641 (+272655us) spinlock_profiling.cc:243] Waited 473 ms on lock 0x7b54005e0e54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:00:28.291848 (+   207us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1087340416}
I20260709 13:00:28.293525  5228 raft_consensus.cc:3060] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:00:28.291143  5336 raft_consensus.cc:2749] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260709 13:00:28.299819  2298 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:28.232779  5123 raft_consensus.cc:515] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:28.297503  2299 leader_election.cc:304] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:28.320641  1596 raft_consensus.cc:1275] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
W20260709 13:00:28.324908  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 130238us
I20260709 13:00:28.327486  2299 leader_election.cc:304] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:00:28.341138  5188 consensus_queue.cc:1059] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.002s
W20260709 13:00:28.344343  1644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:28.329147  5229 raft_consensus.cc:2749] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260709 13:00:28.351547  2298 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:28.354259  2299 leader_election.cc:304] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:00:28.357661  2299 leader_election.cc:304] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:28.362739  5345 consensus_queue.cc:1059] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 13:00:28.363371  5309 leader_election.cc:290] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 8 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:28.364125  5309 raft_consensus.cc:2749] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:00:28.365173  5123 leader_election.cc:290] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:28.366957  5322 leader_election.cc:290] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:28.367807  5342 consensus_queue.cc:1048] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 13:00:28.334595  5217 raft_consensus.cc:2749] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260709 13:00:28.371915  5331 log.cc:927] Time spent T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.051s	user 0.000s	sys 0.000s
I20260709 13:00:28.362021  5260 raft_consensus.cc:515] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:28.373783  1605 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40154 (request call id 1997) took 1434 ms (client timeout 1715 ms). Trace:
W20260709 13:00:28.349538  5330 log.cc:927] Time spent T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.072s	user 0.001s	sys 0.000s
I20260709 13:00:28.355846  5083 raft_consensus.cc:3055] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:00:28.339545  4847 leader_election.cc:290] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 7 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:28.390261  4900 raft_consensus.cc:2749] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:00:28.390378  1677 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653"
dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
mode: GRACEFUL
new_leader_uuid: "af520b67945d443b901bbeca37cc5701"
 from {username='slave'} at 127.0.0.1:43266
W20260709 13:00:28.381459  1643 leader_election.cc:336] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 1.652s (SENT)
I20260709 13:00:28.437413  1643 leader_election.cc:304] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
W20260709 13:00:28.419013  1605 rpcz_store.cc:269] 0709 13:00:26.939659 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:26.941532 (+  1873us) service_pool.cc:224] Handling call
0709 13:00:28.373513 (+1431981us) spinlock_profiling.cc:243] Waited 94.5 ms on lock 0x7b54003edfd4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:00:28.373707 (+   194us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":217361792}
I20260709 13:00:28.392414  2299 leader_election.cc:304] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:28.437438  1677 raft_consensus.cc:606] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 LEADER]: Received request to transfer leadership to TS af520b67945d443b901bbeca37cc5701
W20260709 13:00:28.438901  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 46625.7us
W20260709 13:00:28.415732  5344 log.cc:927] Time spent T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.066s	user 0.001s	sys 0.000s
W20260709 13:00:28.474040  1569 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:00:28.476037  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:28.477524  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:00:28.477813  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:28.477595  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:00:28.476212  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:28.484476  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:28.485647  2335 raft_consensus.cc:2468] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 8.
W20260709 13:00:28.492005  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 110582us
I20260709 13:00:28.436653  5083 raft_consensus.cc:740] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:00:28.472553  5188 raft_consensus.cc:2749] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:00:28.396909  5342 consensus_queue.cc:1059] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.103s
I20260709 13:00:28.504655  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:28.520107  2333 raft_consensus.cc:2393] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 8: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
W20260709 13:00:28.521275  2299 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:00:28.522707  1717 leader_election.cc:336] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 1.892s (SENT)
I20260709 13:00:28.523224  1717 leader_election.cc:304] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:28.494727  5348 raft_consensus.cc:493] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:28.502209  5243 raft_consensus.cc:2749] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:00:28.531977  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:00:28.536883  5303 leader_election.cc:290] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:28.546347   431 auto_leader_rebalancer.cc:452] table: test-workload, leader rebalance finish, leader transfer count: 5
W20260709 13:00:28.562361  5081 consensus_meta.cc:287] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.548s	user 0.009s	sys 0.005s
I20260709 13:00:28.563112  5260 leader_election.cc:290] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 10 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:28.564381  2334 raft_consensus.cc:1275] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260709 13:00:28.567399  5215 raft_consensus.cc:493] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:28.566143  2334 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39726 (request call id 921) took 1858 ms. Trace:
I20260709 13:00:28.570230  2334 rpcz_store.cc:276] 0709 13:00:26.707442 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:27.197352 (+489910us) service_pool.cc:224] Handling call
0709 13:00:27.197795 (+   443us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:28.564911 (+1367116us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:28.565776 (+   865us) spinlock_profiling.cc:243] Waited 1.37 s on lock 0x7b54005e01d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:00:28.566052 (+   276us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3141291264}
I20260709 13:00:28.559522  5224 raft_consensus.cc:3055] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:00:28.575543  5224 raft_consensus.cc:740] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:00:28.576822  5224 consensus_queue.cc:260] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:28.533077  5348 raft_consensus.cc:515] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:28.543475  5083 consensus_queue.cc:260] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 6, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:28.568048  5215 raft_consensus.cc:515] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:28.603019  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 80347.1us
I20260709 13:00:28.570639  5350 raft_consensus.cc:493] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:28.608523  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:28.610031  1752 raft_consensus.cc:3060] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:00:28.588282  5346 raft_consensus.cc:493] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:00:28.612237  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:28.614837  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:28.614962  1718 leader_election.cc:304] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:00:28.611549  5086 raft_consensus.cc:2749] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:00:28.622098  5290 consensus_queue.cc:1048] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 13:00:28.624218  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:28.624969  1749 raft_consensus.cc:3060] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:00:28.622017  5172 raft_consensus.cc:515] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:28.627831  1718 leader_election.cc:304] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:00:28.633140  1718 leader_election.cc:304] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:00:28.639137  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:28.641305  1718 leader_election.cc:304] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:00:28.614078  4746 raft_consensus.cc:493] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:28.630046  5083 raft_consensus.cc:3060] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:00:28.665921  5086 raft_consensus.cc:3060] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:00:28.667788  5277 raft_consensus.cc:493] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:00:28.671270  5351 log.cc:927] Time spent T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.101s	user 0.001s	sys 0.000s
I20260709 13:00:28.641976  5255 raft_consensus.cc:2804] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:00:28.671572  5277 raft_consensus.cc:515] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:28.672297  5255 raft_consensus.cc:493] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:28.672807  5255 raft_consensus.cc:3060] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:00:28.675812  4779 raft_consensus.cc:515] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:28.640765  5290 raft_consensus.cc:3060] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:00:28.641302  4985 raft_consensus.cc:2749] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 13:00:28.683982  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:28.674139  5277 leader_election.cc:290] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:28.678467  4779 leader_election.cc:290] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 8 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:28.702103  1749 raft_consensus.cc:2468] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 10.
I20260709 13:00:28.616567  5350 raft_consensus.cc:515] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:28.619118  5346 raft_consensus.cc:515] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:28.710059  1677 raft_consensus.cc:3055] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:00:28.678988  4746 raft_consensus.cc:515] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:28.715760  1678 raft_consensus.cc:1275] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260709 13:00:28.713968  1677 raft_consensus.cc:740] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:00:28.722525  2337 raft_consensus.cc:2468] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 5.
I20260709 13:00:28.723724  5337 raft_consensus.cc:493] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:00:28.724398  5337 raft_consensus.cc:515] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:28.732640  1646 leader_election.cc:304] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:00:28.734153  5354 raft_consensus.cc:493] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:00:28.729053  1677 consensus_queue.cc:260] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:28.741104  1752 raft_consensus.cc:2468] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 11.
I20260709 13:00:28.741240  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:28.738597  5214 leader_election.cc:290] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:28.742933  1677 raft_consensus.cc:3060] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:00:28.743996  5348 leader_election.cc:290] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:00:28.749203  5333 log.cc:927] Time spent T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.420s	user 0.000s	sys 0.003s
I20260709 13:00:28.750845  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:00:28.751395  5324 log.cc:927] Time spent T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.582s	user 0.001s	sys 0.000s
I20260709 13:00:28.751585  2337 raft_consensus.cc:2468] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 11.
I20260709 13:00:28.752547  5346 leader_election.cc:290] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:28.753974  5346 raft_consensus.cc:2804] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:00:28.754369  5346 raft_consensus.cc:493] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:00:28.754598  5350 leader_election.cc:290] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:28.754829  5346 raft_consensus.cc:3060] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:00:28.755223  1718 leader_election.cc:304] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:00:28.756340  5223 raft_consensus.cc:2804] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:00:28.756866  5223 raft_consensus.cc:697] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 5 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:00:28.759191  5158 leader_election.cc:290] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:00:28.760855  5340 log.cc:927] Time spent T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.478s	user 0.000s	sys 0.002s
I20260709 13:00:28.762856  1646 leader_election.cc:304] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 13:00:28.764422  1602 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 677) took 1318 ms. Trace:
I20260709 13:00:28.764971  5350 raft_consensus.cc:2804] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:00:28.765497  5350 raft_consensus.cc:493] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:28.765902  5350 raft_consensus.cc:3060] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Advancing to term 9
W20260709 13:00:28.769269  5335 log.cc:927] Time spent T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.487s	user 0.000s	sys 0.002s
I20260709 13:00:28.770531  5354 raft_consensus.cc:515] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:28.764868  1602 rpcz_store.cc:276] 0709 13:00:27.445971 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:27.457831 (+ 11860us) service_pool.cc:224] Handling call
0709 13:00:27.874132 (+416301us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:00:28.069036 (+194904us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:00:28.069071 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:00:28.083156 (+ 14085us) log.cc:844] Serialized 26 byte log entry
0709 13:00:28.244593 (+161437us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:00:28.244665 (+    72us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:28.244763 (+    98us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:00:28.374352 (+129589us) raft_consensus.cc:1685] finished
0709 13:00:28.374375 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:28.764122 (+389747us) spinlock_profiling.cc:243] Waited 408 ms on lock 0x7b5400216bd0. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661618ad 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:00:28.764346 (+   224us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":30033,"raft.queue_time_us":70093,"raft.run_cpu_time_us":169,"raft.run_wall_time_us":9822,"spinlock_wait_cycles":937654528,"thread_start_us":234619,"threads_started":2,"wal-append.queue_time_us":213441}
I20260709 13:00:28.772239  5215 leader_election.cc:290] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:28.780254  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:28.780926  1673 raft_consensus.cc:3060] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:00:28.781462  5346 raft_consensus.cc:515] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:00:28.783010  5152 consensus_meta.cc:287] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.939s	user 0.010s	sys 0.006s
I20260709 13:00:28.783751  5346 leader_election.cc:290] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 11 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:28.785194  5350 raft_consensus.cc:515] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:28.788642  1718 leader_election.cc:304] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:00:28.789597  5300 raft_consensus.cc:2804] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:00:28.790038  5300 raft_consensus.cc:493] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:28.790390  5300 raft_consensus.cc:3060] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:00:28.800298  1751 raft_consensus.cc:1240] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 5. Current term is 6. Ops: []
I20260709 13:00:28.783485  5152 raft_consensus.cc:515] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } }
I20260709 13:00:28.806015  1673 raft_consensus.cc:2468] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 9.
I20260709 13:00:28.807704  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 8 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:28.809340  2301 leader_election.cc:304] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:00:28.811251  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:28.805492   431 auto_leader_rebalancer.cc:527] All tables' leader rebalancing finished this round
I20260709 13:00:28.811570  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:28.742018  1678 raft_consensus.cc:2468] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 8.
I20260709 13:00:28.820822  1751 raft_consensus.cc:3060] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:00:28.820194  1752 raft_consensus.cc:3060] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:00:28.823195  2301 leader_election.cc:304] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:28.827087  1680 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52766 (request call id 2369) took 1150 ms. Trace:
I20260709 13:00:28.827517  1673 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52778 (request call id 1941) took 1030 ms. Trace:
I20260709 13:00:28.827620  1680 rpcz_store.cc:276] 0709 13:00:27.676992 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:27.681502 (+  4510us) service_pool.cc:224] Handling call
0709 13:00:28.782493 (+1100991us) spinlock_profiling.cc:243] Waited 2.92 ms on lock 0x7b5800086a98. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f426303db71 00007f426303deee 0000000000375939 00007f426b0b11e0 00007f426b17a0c4 00007f426b17a131 00007f426b1280a0 00007f426b127984 00007f42646a26d4 00007f42646a2662
0709 13:00:28.827005 (+ 44512us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":6730880}
I20260709 13:00:28.827950  1673 rpcz_store.cc:276] 0709 13:00:27.797131 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:27.797350 (+   219us) service_pool.cc:224] Handling call
0709 13:00:28.780078 (+982728us) spinlock_profiling.cc:243] Waited 1.89 ms on lock 0x7b5800086a98. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f426303db71 00007f426303deee 0000000000375939 00007f426b0b11e0 00007f426b17a0c4 00007f426b17a131 00007f426b1280a0 00007f426b127984 00007f42646a26d4 00007f42646a2662
0709 13:00:28.827440 (+ 47362us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4376448}
I20260709 13:00:28.829612  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:28.834028  1752 raft_consensus.cc:2468] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 10.
I20260709 13:00:28.757953  5223 consensus_queue.cc:237] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:28.839021  1717 leader_election.cc:304] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:28.841739  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:28.824380  5183 raft_consensus.cc:2804] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:00:28.856369  1717 leader_election.cc:304] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:28.819486  5337 leader_election.cc:290] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:28.863534  5343 raft_consensus.cc:493] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:28.864408  5343 raft_consensus.cc:515] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:28.863148  5214 raft_consensus.cc:2804] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 13:00:28.867337  5303 raft_consensus.cc:493] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:28.862596  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:28.868134  5303 raft_consensus.cc:515] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:28.867476  5214 raft_consensus.cc:697] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 8 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:00:28.870174  5214 consensus_queue.cc:237] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:28.870824  5303 leader_election.cc:290] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:28.872784  5343 leader_election.cc:290] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:28.872918  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:28.874886  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:00:28.877089  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 53964.7us
I20260709 13:00:28.826539  5215 consensus_queue.cc:1048] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.052s
I20260709 13:00:28.886703  5300 raft_consensus.cc:515] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:28.889446  5300 leader_election.cc:290] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 12 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:28.890158  5215 consensus_queue.cc:1059] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 13:00:28.890795  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:00:28.891471  2337 raft_consensus.cc:3060] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:00:28.891608  5210 raft_consensus.cc:3055] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:00:28.892114  5210 raft_consensus.cc:740] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:00:28.893054  5210 consensus_queue.cc:260] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:28.894595  5210 raft_consensus.cc:3060] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Advancing to term 6
W20260709 13:00:28.896204  2299 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:00:28.896955  2299 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:28.901881  5290 raft_consensus.cc:2749] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:00:28.928037  2301 leader_election.cc:304] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:00:28.905483  2337 raft_consensus.cc:2468] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 12.
I20260709 13:00:28.909580  5045 raft_consensus.cc:493] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:00:28.912657  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:00:28.913719  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 57489.9us
I20260709 13:00:28.939967  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:28.918180  1751 raft_consensus.cc:2468] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 8.
W20260709 13:00:28.925324  1571 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:28.936343  1718 leader_election.cc:304] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:00:28.912328  5183 raft_consensus.cc:493] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:28.861316  4985 raft_consensus.cc:2749] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:00:28.926666  5300 raft_consensus.cc:2749] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 13:00:28.943482  5123 raft_consensus.cc:2804] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:00:28.911439  5277 consensus_queue.cc:1048] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.045s
I20260709 13:00:28.966868  1752 raft_consensus.cc:2468] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 9.
I20260709 13:00:28.956138  4847 raft_consensus.cc:493] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:28.954936  5183 raft_consensus.cc:3060] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:00:28.965274  5210 raft_consensus.cc:2804] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 13:00:28.945065  5045 raft_consensus.cc:515] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:28.994592  4847 raft_consensus.cc:515] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:28.995581  5045 leader_election.cc:290] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:28.996836  5210 raft_consensus.cc:697] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 10 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:00:28.997221  4847 leader_election.cc:290] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:28.997893  5086 raft_consensus.cc:2749] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:00:28.998410  5210 consensus_queue.cc:237] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:28.999692  5152 leader_election.cc:290] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 6 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:28.944303  5303 consensus_queue.cc:1048] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.024s
I20260709 13:00:28.980932  5123 raft_consensus.cc:697] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 12 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:00:29.013005  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:00:28.980573  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 240228us
I20260709 13:00:29.033617  1751 raft_consensus.cc:1275] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260709 13:00:29.036360  1718 leader_election.cc:304] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:00:29.024600  2301 leader_election.cc:304] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
W20260709 13:00:29.019981  5228 consensus_meta.cc:287] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.726s	user 0.013s	sys 0.004s
I20260709 13:00:29.026000  5083 raft_consensus.cc:493] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:29.044183  5354 leader_election.cc:290] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:29.045451  5036 raft_consensus.cc:3055] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:00:29.051208  5083 raft_consensus.cc:515] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:29.051633  5123 consensus_queue.cc:237] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:29.055599  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 319015us
I20260709 13:00:29.052194  5129 raft_consensus.cc:2749] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20260709 13:00:29.061661  1644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:29.063207  2329 raft_consensus.cc:1275] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260709 13:00:29.064091  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:29.065598  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:29.065896  1677 raft_consensus.cc:2393] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 9: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
W20260709 13:00:29.067968  1569 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:29.068580  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:29.069429  1673 raft_consensus.cc:2468] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 6.
W20260709 13:00:29.070861  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 292556us
I20260709 13:00:29.072289  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:29.073112  1678 raft_consensus.cc:3060] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:00:29.085819  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:29.086910  1679 raft_consensus.cc:2410] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 for term 12 because replica has last-logged OpId of term: 10 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 9.
I20260709 13:00:29.092319  1678 raft_consensus.cc:2468] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 8.
I20260709 13:00:29.069787  1569 leader_election.cc:304] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:00:29.079363  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:29.094151  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:29.096094  1677 raft_consensus.cc:1275] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260709 13:00:29.097443  1672 raft_consensus.cc:1240] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 9. Current term is 10. Ops: [9.8-9.8]
W20260709 13:00:29.135581  2299 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:00:29.156051  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:29.164271  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:00:29.055441  5036 raft_consensus.cc:740] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:00:29.164271  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:29.172303  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:29.166531  1673 raft_consensus.cc:1275] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260709 13:00:29.099720  5182 raft_consensus.cc:2804] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 13:00:29.176196  5182 raft_consensus.cc:697] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 8 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:00:29.177450  5182 consensus_queue.cc:237] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:29.190470  2332 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49082 (request call id 773) took 1665 ms (client timeout 2125 ms). Trace:
I20260709 13:00:29.191329  5319 raft_consensus.cc:2749] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260709 13:00:29.191200  2332 rpcz_store.cc:269] 0709 13:00:27.525353 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:27.529374 (+  4021us) service_pool.cc:224] Handling call
0709 13:00:29.190166 (+1660792us) spinlock_profiling.cc:243] Waited 159 ms on lock 0x7b540045bfd4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:00:29.190350 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":366158592}
I20260709 13:00:29.193876  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:29.194906  1675 raft_consensus.cc:2468] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 6.
I20260709 13:00:29.197932  1675 raft_consensus.cc:1240] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 7. Current term is 8. Ops: [7.9-7.9]
W20260709 13:00:29.196702  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 126946us
I20260709 13:00:29.194887  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:00:29.205929  2332 raft_consensus.cc:3060] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:00:29.204581  1675 raft_consensus.cc:1240] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 10. Current term is 11. Ops: []
I20260709 13:00:29.230968  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:29.079077  5336 raft_consensus.cc:3055] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 7 LEADER]: Stepping down as leader of term 7
W20260709 13:00:29.270220  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 169566us
W20260709 13:00:29.271063  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 186596us
I20260709 13:00:29.275441  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:29.276381  1675 raft_consensus.cc:3060] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:00:29.082393  4847 raft_consensus.cc:493] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:00:29.287506  2299 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:29.282675  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
W20260709 13:00:29.166913  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 105189us
I20260709 13:00:29.056375  5086 consensus_queue.cc:1048] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.116s
I20260709 13:00:29.094461  5345 raft_consensus.cc:493] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:00:29.295994  1672 raft_consensus.cc:2393] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 7: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:00:29.293268  1679 raft_consensus.cc:1240] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 10. Current term is 11. Ops: []
I20260709 13:00:29.056077  5255 raft_consensus.cc:515] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:29.294144  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:29.297561  2328 raft_consensus.cc:2410] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for term 7 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260709 13:00:29.165741  5045 raft_consensus.cc:493] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:29.279277  5214 raft_consensus.cc:493] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:29.308939  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:29.316110  1678 raft_consensus.cc:2410] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for term 8 because replica has last-logged OpId of term: 7 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20260709 13:00:29.312422  1596 raft_consensus.cc:1275] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260709 13:00:29.167341  5223 consensus_queue.cc:1048] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.134s
I20260709 13:00:29.172130  5337 consensus_queue.cc:1048] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.133s
I20260709 13:00:29.272557  5336 raft_consensus.cc:740] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:00:29.337984  5336 consensus_queue.cc:260] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:29.339504  5336 raft_consensus.cc:3060] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:00:29.279374  5251 consensus_queue.cc:1059] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.039s
I20260709 13:00:29.302132  5303 raft_consensus.cc:493] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:00:29.360154  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:29.361057  1597 raft_consensus.cc:2376] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 8. Re-sending same reply.
I20260709 13:00:29.288143  4759 consensus_queue.cc:1048] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.039s
I20260709 13:00:29.333328  1605 raft_consensus.cc:1240] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 5. Current term is 6. Ops: []
I20260709 13:00:29.333353  1604 raft_consensus.cc:1240] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 8. Current term is 9. Ops: [8.5-8.5]
I20260709 13:00:29.335731  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:29.335789  5358 consensus_queue.cc:1059] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 13:00:29.369423  1602 raft_consensus.cc:2410] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for term 7 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260709 13:00:29.336328  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:29.173135  5081 consensus_queue.cc:1048] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.130s
I20260709 13:00:29.298841  5137 raft_consensus.cc:493] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:00:29.346618  2332 raft_consensus.cc:2468] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 6.
I20260709 13:00:29.291221  5363 raft_consensus.cc:493] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:00:29.325356  5228 raft_consensus.cc:493] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:29.354811  2299 leader_election.cc:304] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:00:29.385839  2299 leader_election.cc:304] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:00:29.314198  1679 raft_consensus.cc:1275] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260709 13:00:29.331946  1598 raft_consensus.cc:1240] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 7. Current term is 8. Ops: []
I20260709 13:00:29.335004  5224 raft_consensus.cc:3060] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:00:29.347728  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:29.362355  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:29.371835  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:00:29.405082  1643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:29.405567  1603 raft_consensus.cc:2468] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 6.
I20260709 13:00:29.375415  1571 leader_election.cc:304] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:00:29.333585  5036 consensus_queue.cc:260] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:29.412663  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:29.413801  1571 leader_election.cc:304] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:00:29.295472  4847 raft_consensus.cc:515] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:29.418113  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:29.391741  5081 raft_consensus.cc:2804] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:00:29.392359  4951 raft_consensus.cc:2804] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 13:00:29.392206  5137 raft_consensus.cc:515] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:29.393127  5363 raft_consensus.cc:515] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:29.438277  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:29.394270  5319 consensus_queue.cc:1048] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 13:00:29.400806  4746 leader_election.cc:290] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:29.407084  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:29.454707  1598 raft_consensus.cc:2468] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 9.
I20260709 13:00:29.410820  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:29.421119  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:29.456290  1597 raft_consensus.cc:2468] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 8.
I20260709 13:00:29.419893  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:00:29.424985  1643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:29.457687  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:29.430023  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:29.304350  5345 raft_consensus.cc:515] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:29.315198  5045 raft_consensus.cc:515] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:29.325871  5214 raft_consensus.cc:515] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:29.461498  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:29.430668  5081 raft_consensus.cc:493] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:29.403946  5322 raft_consensus.cc:493] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:00:29.475091  1600 raft_consensus.cc:1275] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
W20260709 13:00:29.488711  1643 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 1.583s (SENT) [suppressed 31 similar messages]
I20260709 13:00:29.452899  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:29.428884  5346 raft_consensus.cc:493] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:00:29.375486  5303 raft_consensus.cc:515] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:29.461524  4951 raft_consensus.cc:697] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 8 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:00:29.486236  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:29.416741  5079 leader_election.cc:290] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 8 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:29.493744  5081 raft_consensus.cc:3060] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:00:29.499950  4896 raft_consensus.cc:493] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
W20260709 13:00:29.500147  5366 log.cc:927] Time spent T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.071s	user 0.000s	sys 0.000s
I20260709 13:00:29.455204  5036 raft_consensus.cc:3060] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:00:29.484400  1675 raft_consensus.cc:2468] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 6.
I20260709 13:00:29.550076  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:00:29.578390  1643 leader_election.cc:336] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 1.946s (SENT)
I20260709 13:00:29.686156  1643 leader_election.cc:304] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
W20260709 13:00:29.687791  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 109497us
I20260709 13:00:29.421523  5228 raft_consensus.cc:515] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:29.686327  1602 raft_consensus.cc:2468] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 9.
W20260709 13:00:29.656131  2337 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.620s	user 0.026s	sys 0.049s
W20260709 13:00:29.514514  5317 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.528s	user 0.024s	sys 0.055s
W20260709 13:00:29.691208  1718 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.596s	user 0.022s	sys 0.024s
W20260709 13:00:29.709403  1718 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.615s	user 0.022s	sys 0.024s
W20260709 13:00:29.709882  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 649036us
I20260709 13:00:29.718154  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:29.719233  1679 raft_consensus.cc:2368] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 8. Current term is 9.
W20260709 13:00:29.697005  2337 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.661s	user 0.026s	sys 0.049s
I20260709 13:00:29.655519  5079 raft_consensus.cc:493] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:29.652694  5243 raft_consensus.cc:3055] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:00:29.743861  1718 leader_election.cc:304] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
W20260709 13:00:29.705561  5317 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.715s	user 0.025s	sys 0.056s
I20260709 13:00:29.500677  5322 raft_consensus.cc:515] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:00:29.706758  5183 consensus_meta.cc:287] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.718s	user 0.013s	sys 0.000s
I20260709 13:00:29.508054  5346 raft_consensus.cc:515] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:29.706211  5359 raft_consensus.cc:493] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:29.774701  4847 leader_election.cc:290] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:29.775840  5300 raft_consensus.cc:493] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:29.777153  5303 leader_election.cc:290] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:29.778640  5086 raft_consensus.cc:2804] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:00:29.746134  5243 raft_consensus.cc:740] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:00:29.784257  5322 leader_election.cc:290] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:29.787034  5356 raft_consensus.cc:493] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:00:29.795179  5348 raft_consensus.cc:2749] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 13:00:29.794652  4951 consensus_queue.cc:237] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.9, Last appended by leader: 9, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:29.797494  5363 leader_election.cc:290] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:29.799765  5228 leader_election.cc:290] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:29.743132  5290 raft_consensus.cc:493] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:29.898368  5290 raft_consensus.cc:515] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:29.680534  4896 raft_consensus.cc:515] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:29.858485  5045 leader_election.cc:290] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:29.881654  4847 raft_consensus.cc:2749] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:00:29.851256  5086 raft_consensus.cc:493] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:29.893916  5083 leader_election.cc:290] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:29.895358  5350 leader_election.cc:290] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 9 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:29.896270  4872 raft_consensus.cc:2749] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:00:29.757860  5079 raft_consensus.cc:515] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:00:29.906777  2301 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.599s	user 0.024s	sys 0.025s
I20260709 13:00:29.859370  5137 leader_election.cc:290] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:29.851984  1601 raft_consensus.cc:1275] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 8. (term mismatch)
I20260709 13:00:29.967335  1601 pending_rounds.cc:85] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a: Aborting all ops after (but not including) 7
I20260709 13:00:29.854627  5300 raft_consensus.cc:515] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:29.885699  5325 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.673s	user 0.012s	sys 0.033s
I20260709 13:00:29.848286  5255 leader_election.cc:290] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 8 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:29.881575  5356 raft_consensus.cc:515] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:29.954510  5290 leader_election.cc:290] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:29.847725  5359 raft_consensus.cc:515] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:29.967670  1601 pending_rounds.cc:107] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a: Aborting uncommitted NO_OP operation due to leader change: 7.8
I20260709 13:00:30.012786  1601 raft_consensus.cc:2889] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:00:29.810019  5183 raft_consensus.cc:515] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:29.968046  2301 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.660s	user 0.024s	sys 0.025s
W20260709 13:00:29.969823  1717 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.845s	user 0.008s	sys 0.016s
W20260709 13:00:29.949760  2334 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.649s	user 0.006s	sys 0.020s
W20260709 13:00:30.015098  2334 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.714s	user 0.007s	sys 0.020s
I20260709 13:00:29.950228  5086 raft_consensus.cc:3060] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:00:30.016245  2334 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39726 (request call id 930) took 1084 ms. Trace:
W20260709 13:00:30.016752  1717 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.892s	user 0.008s	sys 0.017s
W20260709 13:00:29.994199  5325 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.782s	user 0.012s	sys 0.033s
W20260709 13:00:30.341349  1645 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.643s	user 0.013s	sys 0.021s
W20260709 13:00:30.130406  1599 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.792s	user 0.017s	sys 0.026s
W20260709 13:00:30.515033  1645 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.817s	user 0.019s	sys 0.024s
W20260709 13:00:30.515331  1599 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.177s	user 0.023s	sys 0.031s
I20260709 13:00:30.328673  5346 leader_election.cc:290] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:00:30.286334  5247 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.858s	user 0.015s	sys 0.024s
W20260709 13:00:30.045550  1644 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.745s	user 0.003s	sys 0.026s
W20260709 13:00:30.517127  1644 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.216s	user 0.007s	sys 0.042s
I20260709 13:00:30.517617  2334 rpcz_store.cc:276] 0709 13:00:28.931648 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:28.931866 (+   218us) service_pool.cc:224] Handling call
0709 13:00:28.943973 (+ 12107us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:28.944239 (+   266us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 13:00:29.300761 (+356522us) log.cc:844] Serialized 14 byte log entry
0709 13:00:30.015744 (+714983us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:00:30.015823 (+    79us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:00:30.015921 (+    98us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:30.016017 (+    96us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:30.016171 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":341917,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":44,"raft.queue_time_us":124,"raft.run_cpu_time_us":112,"raft.run_wall_time_us":112,"thread_start_us":1056662,"threads_started":2,"wal-append.queue_time_us":1193139}
W20260709 13:00:30.215273  2336 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.788s	user 0.015s	sys 0.023s
W20260709 13:00:30.520936  2336 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.094s	user 0.021s	sys 0.026s
I20260709 13:00:30.522118  2336 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39726 (request call id 933) took 1541 ms. Trace:
W20260709 13:00:30.340276  5342 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.837s	user 0.005s	sys 0.019s
W20260709 13:00:30.503098  1749 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.721s	user 0.009s	sys 0.011s
W20260709 13:00:30.098682  1568 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.786s	user 0.020s	sys 0.013s
W20260709 13:00:30.236541  1674 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.909s	user 0.009s	sys 0.027s
W20260709 13:00:30.524154  1749 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.742s	user 0.010s	sys 0.011s
W20260709 13:00:30.237104  1646 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.936s	user 0.012s	sys 0.013s
W20260709 13:00:30.528728  1646 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.228s	user 0.015s	sys 0.019s
W20260709 13:00:30.525163  1568 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.212s	user 0.024s	sys 0.018s
I20260709 13:00:30.098609  5359 leader_election.cc:290] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:00:30.530779  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:00:30.398922  2335 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.986s	user 0.012s	sys 0.014s
W20260709 13:00:30.542629  2335 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.130s	user 0.016s	sys 0.014s
W20260709 13:00:30.209813  2300 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.851s	user 0.003s	sys 0.020s
I20260709 13:00:30.204447  5079 leader_election.cc:290] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:30.003281  5345 leader_election.cc:290] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:00:30.515363  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 1.39064e+06us
W20260709 13:00:30.537654  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 1.25292e+06us
W20260709 13:00:30.552088  1717 connection.cc:256] RPC call timeout handler was delayed by -1.2202056849999963s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1781s for a response
I20260709 13:00:30.526770  2333 raft_consensus.cc:2393] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 9: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
W20260709 13:00:30.523864  5342 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.032s	user 0.007s	sys 0.022s
I20260709 13:00:30.527535  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:30.554193  1751 raft_consensus.cc:2468] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 8.
W20260709 13:00:30.553468  1717 connection.cc:256] RPC call timeout handler was delayed by -1.118626699999993s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1715s for a response
W20260709 13:00:30.054096  4985 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.764s	user 0.012s	sys 0.017s
W20260709 13:00:30.047058  2299 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.660s	user 0.007s	sys 0.019s
W20260709 13:00:30.558252  2299 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.171s	user 0.015s	sys 0.037s
W20260709 13:00:30.013039  5007 log.cc:927] Time spent T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.062s	user 0.000s	sys 0.000s
I20260709 13:00:30.556648  1644 leader_election.cc:304] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
W20260709 13:00:30.532065  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:00:30.562918  1717 connection.cc:256] RPC call timeout handler was delayed by -1.0477156740000169s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1998s for a response
W20260709 13:00:30.524462  1674 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.208s	user 0.012s	sys 0.030s
W20260709 13:00:30.534776  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 1.26143e+06us
W20260709 13:00:30.034561  5182 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.851s	user 0.026s	sys 0.028s
I20260709 13:00:30.522699  2336 rpcz_store.cc:276] 0709 13:00:28.980260 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:28.980409 (+   149us) service_pool.cc:224] Handling call
0709 13:00:29.003218 (+ 22809us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:29.003584 (+   366us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 13:00:29.426713 (+423129us) log.cc:844] Serialized 14 byte log entry
0709 13:00:30.521571 (+1094858us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:00:30.521647 (+    76us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:00:30.521728 (+    81us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:30.521828 (+   100us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:30.522035 (+   207us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":327433,"prepare.run_cpu_time_us":58,"prepare.run_wall_time_us":59,"raft.queue_time_us":5907,"raft.run_cpu_time_us":128,"raft.run_wall_time_us":132,"thread_start_us":1483117,"threads_started":2}
W20260709 13:00:30.251387  1569 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.972s	user 0.024s	sys 0.017s
W20260709 13:00:30.516438  5247 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.122s	user 0.027s	sys 0.027s
W20260709 13:00:30.515883  1645 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:00:30.569947  1718 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.806s	user 0.006s	sys 0.017s
W20260709 13:00:30.570408  1718 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.806s	user 0.006s	sys 0.017s
W20260709 13:00:30.570783  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 826864us
W20260709 13:00:30.216636  1600 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.717s	user 0.006s	sys 0.016s
I20260709 13:00:30.571660  1718 leader_election.cc:304] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
W20260709 13:00:30.571880  1600 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.073s	user 0.006s	sys 0.022s
W20260709 13:00:30.543915  2300 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.186s	user 0.009s	sys 0.021s
I20260709 13:00:30.030870  5296 raft_consensus.cc:493] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:00:30.526643  5370 log.cc:927] Time spent T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.824s	user 0.005s	sys 0.015s
W20260709 13:00:30.438563  5336 consensus_meta.cc:287] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 1.099s	user 0.019s	sys 0.000s
I20260709 13:00:30.584636  2330 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39738 (request call id 752) took 1797 ms. Trace:
I20260709 13:00:30.584879  1605 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 686) took 1186 ms. Trace:
W20260709 13:00:30.558727  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 1.17279e+06us
I20260709 13:00:30.585081  2330 rpcz_store.cc:276] 0709 13:00:28.787570 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:28.802067 (+ 14497us) service_pool.cc:224] Handling call
0709 13:00:28.802604 (+   537us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:00:28.803118 (+   514us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:00:28.803161 (+    43us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:00:28.803804 (+   643us) log.cc:844] Serialized 26 byte log entry
0709 13:00:29.176283 (+372479us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:00:29.176337 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:29.176424 (+    87us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:00:30.525734 (+1349310us) spinlock_profiling.cc:243] Waited 3.18 ms on lock 0x7b4400022b4c. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f4262f9bc85 00007f4262f9c6a2 00007f4265d0e1db 00007f4265d09aaf 00007f4265cfd86e 00007f4265cfcbb6 00007f4265cfc7d8 00007f4265d06221 00007f4265d061d9 00007f4265d06169
0709 13:00:30.584202 (+ 58468us) raft_consensus.cc:1685] finished
0709 13:00:30.584229 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:30.584553 (+   324us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1061,"raft.run_cpu_time_us":185,"raft.run_wall_time_us":5797,"spinlock_wait_cycles":7323904,"thread_start_us":351796,"threads_started":1,"wal-append.queue_time_us":650020}
W20260709 13:00:30.566983  1569 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.288s	user 0.029s	sys 0.026s
W20260709 13:00:30.589332  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 1.31046e+06us
I20260709 13:00:30.585353  1605 rpcz_store.cc:276] 0709 13:00:29.398653 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:29.487840 (+ 89187us) service_pool.cc:224] Handling call
0709 13:00:29.499053 (+ 11213us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:29.519488 (+ 20435us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:00:29.519539 (+    51us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:00:29.519604 (+    65us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:00:29.519670 (+    66us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:30.584625 (+1064955us) spinlock_profiling.cc:243] Waited 20.1 ms on lock 0x7b54001e0354. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:00:30.584682 (+    57us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:30.584816 (+   134us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":46129920}
I20260709 13:00:30.556629  2301 leader_election.cc:304] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
W20260709 13:00:30.413421  5129 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.079s	user 0.009s	sys 0.044s
I20260709 13:00:30.063566  4896 leader_election.cc:290] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:30.502588  5383 raft_consensus.cc:493] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:00:30.566583  5182 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.383s	user 0.031s	sys 0.049s
I20260709 13:00:30.021001  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:30.602069  1754 raft_consensus.cc:2468] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 8.
I20260709 13:00:30.602463  1718 leader_election.cc:304] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
W20260709 13:00:30.603005  1603 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.862s	user 0.005s	sys 0.014s
W20260709 13:00:30.603328  1603 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.862s	user 0.005s	sys 0.014s
I20260709 13:00:30.569520  5346 raft_consensus.cc:2804] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Leader election won for term 6
W20260709 13:00:30.560768  4985 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.273s	user 0.012s	sys 0.042s
W20260709 13:00:30.567612  1596 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.765s	user 0.000s	sys 0.018s
I20260709 13:00:30.547648  5214 leader_election.cc:290] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:30.547027  5300 leader_election.cc:290] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:30.557215  5228 raft_consensus.cc:493] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:00:30.598031  1672 raft_consensus.cc:1275] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
W20260709 13:00:30.660346  1569 connection.cc:256] RPC call timeout handler was delayed by -1.0194636209999999s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1637s for a response
W20260709 13:00:30.598311  1671 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.720s	user 0.009s	sys 0.027s
W20260709 13:00:30.661103  1671 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.783s	user 0.010s	sys 0.027s
I20260709 13:00:30.599170  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:30.560730  5356 leader_election.cc:290] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:30.561547  5079 raft_consensus.cc:493] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
W20260709 13:00:30.667157  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 97064us
W20260709 13:00:30.654995  1596 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.854s	user 0.002s	sys 0.025s
W20260709 13:00:30.605144  1598 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.707s	user 0.018s	sys 0.010s
W20260709 13:00:30.669209  1598 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.771s	user 0.019s	sys 0.010s
W20260709 13:00:30.657574  2337 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.936s	user 0.007s	sys 0.020s
I20260709 13:00:30.662055  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:00:30.596092  5129 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.262s	user 0.013s	sys 0.044s
I20260709 13:00:30.658366  1602 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 696) took 1256 ms. Trace:
I20260709 13:00:30.673624  1602 rpcz_store.cc:276] 0709 13:00:29.401626 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:29.707533 (+305907us) service_pool.cc:224] Handling call
0709 13:00:29.754314 (+ 46781us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:29.946123 (+191809us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:00:29.946172 (+    49us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:00:29.946237 (+    65us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:00:29.946283 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:30.658036 (+711753us) spinlock_profiling.cc:243] Waited 191 ms on lock 0x7b54000acb54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:00:30.658097 (+    61us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:30.658284 (+   187us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":440302976}
W20260709 13:00:30.661721  4746 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.764s	user 0.012s	sys 0.012s
W20260709 13:00:30.676378  4746 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.778s	user 0.012s	sys 0.012s
W20260709 13:00:30.623297  1571 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.208s	user 0.017s	sys 0.015s
W20260709 13:00:30.671839  2337 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.951s	user 0.008s	sys 0.020s
I20260709 13:00:30.679241  2337 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39726 (request call id 932) took 1706 ms. Trace:
I20260709 13:00:30.672117  1675 raft_consensus.cc:2468] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 8.
I20260709 13:00:30.669767  1569 leader_election.cc:304] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:00:30.679656  2337 rpcz_store.cc:276] 0709 13:00:28.973022 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:28.981600 (+  8578us) service_pool.cc:224] Handling call
0709 13:00:29.026590 (+ 44990us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:29.026908 (+   318us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 13:00:29.720975 (+694067us) log.cc:844] Serialized 14 byte log entry
0709 13:00:30.678771 (+957796us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:00:30.678857 (+    86us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:00:30.678934 (+    77us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:30.679010 (+    76us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:30.679164 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5702,"prepare.queue_time_us":388099,"prepare.run_cpu_time_us":60,"prepare.run_wall_time_us":61,"thread_start_us":1642480,"threads_started":2}
I20260709 13:00:30.681728  2333 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39726 (request call id 937) took 1585 ms. Trace:
I20260709 13:00:30.598134  5215 raft_consensus.cc:493] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
W20260709 13:00:30.659765  5224 consensus_meta.cc:287] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 1.234s	user 0.018s	sys 0.000s
I20260709 13:00:30.604822  5152 raft_consensus.cc:2804] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Leader election won for term 9
W20260709 13:00:30.656534  5007 log.cc:927] Time spent T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.067s	user 0.002s	sys 0.001s
I20260709 13:00:30.682124  2333 rpcz_store.cc:276] 0709 13:00:29.096329 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:29.099167 (+  2838us) service_pool.cc:224] Handling call
0709 13:00:30.681515 (+1582348us) spinlock_profiling.cc:243] Waited 1.36 s on lock 0x7b540065e0d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:00:30.681660 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3130198272}
I20260709 13:00:30.588798  5296 raft_consensus.cc:515] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:30.726598  1673 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.826s	user 0.015s	sys 0.011s
W20260709 13:00:30.827267  1673 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.927s	user 0.015s	sys 0.014s
I20260709 13:00:30.306610   431 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:00:30.686297  1646 leader_election.cc:304] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:00:30.623643  5389 raft_consensus.cc:493] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:30.605610  5247 raft_consensus.cc:493] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:30.624442  4872 raft_consensus.cc:493] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:30.659838  5358 raft_consensus.cc:493] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:00:30.681354  1571 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.266s	user 0.026s	sys 0.017s
I20260709 13:00:30.660238  5172 leader_election.cc:290] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 6 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:30.666218  5336 raft_consensus.cc:493] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:00:30.776019  1717 leader_election.cc:336] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 1.715s (SENT)
I20260709 13:00:30.666507  5214 raft_consensus.cc:493] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:00:30.685397  5356 raft_consensus.cc:493] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:00:30.670938  5123 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.657s	user 0.018s	sys 0.017s
I20260709 13:00:30.808245  5398 consensus_queue.cc:1048] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 13:00:30.811187  5390 raft_consensus.cc:2804] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:00:30.672580  5228 raft_consensus.cc:515] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:30.679416  5083 raft_consensus.cc:493] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:00:30.683360  5346 raft_consensus.cc:697] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:00:30.720512  5342 raft_consensus.cc:493] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:00:30.623862  5383 raft_consensus.cc:515] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:30.751320  5303 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.806s	user 0.009s	sys 0.030s
W20260709 13:00:30.854782  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 1.44082e+06us
I20260709 13:00:30.858134  1717 leader_election.cc:304] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:30.920459  1571 leader_election.cc:304] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
W20260709 13:00:30.920871  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 144880us
I20260709 13:00:30.683631  5079 raft_consensus.cc:515] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:30.826964  5300 raft_consensus.cc:2804] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader election won for term 6
W20260709 13:00:30.916206  5081 consensus_meta.cc:287] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 1.219s	user 0.018s	sys 0.000s
I20260709 13:00:30.744599  5215 raft_consensus.cc:515] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:30.826120  5369 raft_consensus.cc:493] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:30.856877  4985 raft_consensus.cc:493] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:00:30.876432  5036 consensus_meta.cc:287] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 1.179s	user 0.013s	sys 0.005s
W20260709 13:00:30.912510  5123 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.877s	user 0.022s	sys 0.019s
I20260709 13:00:30.864322  5152 raft_consensus.cc:697] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 9 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:00:30.857071  4872 raft_consensus.cc:515] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:30.804216  5251 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.315s	user 0.015s	sys 0.020s
W20260709 13:00:30.921751  5303 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.014s	user 0.010s	sys 0.031s
I20260709 13:00:30.914914  5390 raft_consensus.cc:493] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Starting leader election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:00:30.848553  5247 raft_consensus.cc:515] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:00:30.950542  1604 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.329s	user 0.031s	sys 0.058s
I20260709 13:00:30.863399  5224 consensus_queue.cc:1059] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 5.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.077s
I20260709 13:00:30.923791  5183 leader_election.cc:290] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 9 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:30.921914  5403 raft_consensus.cc:493] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:30.925036  5182 raft_consensus.cc:2804] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:00:30.929114  1571 leader_election.cc:304] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:00:30.872049  5336 raft_consensus.cc:515] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:30.842592  5389 raft_consensus.cc:515] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:30.922606  5083 raft_consensus.cc:515] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:30.865826  5358 raft_consensus.cc:515] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:30.874372  5214 raft_consensus.cc:515] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:30.949051  5081 raft_consensus.cc:515] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:30.973204  5336 leader_election.cc:290] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:30.948756  5380 raft_consensus.cc:2749] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
W20260709 13:00:30.962644  1604 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.360s	user 0.031s	sys 0.058s
I20260709 13:00:30.913461  5356 raft_consensus.cc:515] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:30.998055  5358 leader_election.cc:290] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:30.965557  4896 consensus_queue.cc:1059] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 13:00:30.953630  5369 raft_consensus.cc:515] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:30.954015  4985 raft_consensus.cc:515] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:30.971369  5247 leader_election.cc:290] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:30.966877  5182 raft_consensus.cc:493] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:30.972018  5403 raft_consensus.cc:515] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:30.963671  5390 raft_consensus.cc:3060] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Advancing to term 7
W20260709 13:00:31.027263  5394 log.cc:927] Time spent T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.153s	user 0.000s	sys 0.001s
I20260709 13:00:31.029927  5081 leader_election.cc:290] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 7 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:30.926347  5342 raft_consensus.cc:515] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:00:31.030836  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:31.032590  5342 leader_election.cc:290] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:00:30.964476  5210 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.155s	user 0.012s	sys 0.034s
W20260709 13:00:31.033993  5210 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.225s	user 0.012s	sys 0.034s
W20260709 13:00:30.957727  5251 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.555s	user 0.015s	sys 0.021s
I20260709 13:00:31.037590  5403 leader_election.cc:290] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:30.989710  5224 raft_consensus.cc:2749] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:00:30.990290  4872 leader_election.cc:290] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:30.991330  5296 leader_election.cc:290] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:30.993059  5389 leader_election.cc:290] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:30.994911  5152 consensus_queue.cc:237] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:30.967120  5391 raft_consensus.cc:493] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:00:30.998803  5356 leader_election.cc:290] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:30.920037  5392 consensus_queue.cc:1059] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 10.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.046s
I20260709 13:00:30.999437  5228 leader_election.cc:290] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:00:31.003638  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 74601.7us
I20260709 13:00:31.052877  1571 leader_election.cc:304] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:00:30.986903  5388 raft_consensus.cc:3060] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:00:31.010295  5182 raft_consensus.cc:3060] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:00:30.988951  5223 raft_consensus.cc:2749] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:00:31.026175  5393 raft_consensus.cc:493] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:31.042280  2331 raft_consensus.cc:1240] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 10 LEADER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 9. Current term is 10. Ops: [9.9-9.9]
W20260709 13:00:31.026134  5406 log.cc:927] Time spent T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.064s	user 0.000s	sys 0.000s
W20260709 13:00:31.047734  2335 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.504s	user 0.004s	sys 0.009s
I20260709 13:00:31.038352  5403 raft_consensus.cc:493] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:31.083582  2335 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49082 (request call id 782) took 2071 ms. Trace:
I20260709 13:00:31.055516  1571 leader_election.cc:304] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:00:31.085352  1571 leader_election.cc:304] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:00:31.086301  4896 raft_consensus.cc:2749] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 13:00:31.086663  5251 raft_consensus.cc:3060] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:00:31.092320  1571 leader_election.cc:304] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:00:31.093930  4896 raft_consensus.cc:2749] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:00:31.094640  1571 leader_election.cc:304] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:00:31.095819  4896 raft_consensus.cc:2749] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 13:00:31.089406  2335 rpcz_store.cc:276] 0709 13:00:29.012086 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:29.012223 (+   137us) service_pool.cc:224] Handling call
0709 13:00:29.037657 (+ 25434us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:29.037898 (+   241us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 13:00:29.412319 (+374421us) log.cc:844] Serialized 14 byte log entry
0709 13:00:31.083055 (+1670736us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:00:31.083141 (+    86us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:00:31.083211 (+    70us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:31.083304 (+    93us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:31.083492 (+   188us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":411476,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":3687,"raft.queue_time_us":129486,"raft.run_cpu_time_us":200,"raft.run_wall_time_us":42749,"thread_start_us":2042108,"threads_started":3,"wal-append.queue_time_us":1405038}
I20260709 13:00:31.096513  1571 leader_election.cc:304] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:00:31.103402  4896 raft_consensus.cc:2749] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:00:31.103935  1571 leader_election.cc:304] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:00:31.104904  4896 raft_consensus.cc:2749] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:00:31.105520  1571 leader_election.cc:304] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
W20260709 13:00:31.067739  1749 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.542s	user 0.012s	sys 0.016s
W20260709 13:00:31.073315  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 486254us
I20260709 13:00:31.106389  4896 raft_consensus.cc:2749] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260709 13:00:31.106825  1749 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.581s	user 0.012s	sys 0.016s
W20260709 13:00:31.078827  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:31.108918  4896 raft_consensus.cc:493] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:31.083344  5403 raft_consensus.cc:515] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:31.056625  1600 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 688) took 1657 ms. Trace:
W20260709 13:00:31.057246  5325 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.524s	user 0.003s	sys 0.010s
I20260709 13:00:31.109673  4896 raft_consensus.cc:515] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:31.123623  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:31.129352  1672 raft_consensus.cc:2393] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 9: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:00:31.123024  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:00:31.132457  2299 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:31.132507  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:31.133772  1678 raft_consensus.cc:3060] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:00:30.987576  5346 consensus_queue.cc:237] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:31.131158  1679 raft_consensus.cc:2468] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 8.
W20260709 13:00:31.116444  5325 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.583s	user 0.003s	sys 0.010s
I20260709 13:00:31.053522  1674 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52766 (request call id 2378) took 1774 ms. Trace:
I20260709 13:00:31.161176  1674 rpcz_store.cc:276] 0709 13:00:29.279186 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:29.290642 (+ 11456us) service_pool.cc:224] Handling call
0709 13:00:29.316471 (+ 25829us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:29.316689 (+   218us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 13:00:30.566716 (+1250027us) log.cc:844] Serialized 14 byte log entry
0709 13:00:31.053109 (+486393us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:00:31.053169 (+    60us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:00:31.053221 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:31.053303 (+    82us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:31.053450 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":1643102,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":47,"raft.queue_time_us":99,"raft.run_cpu_time_us":211,"raft.run_wall_time_us":20718,"thread_start_us":1732797,"threads_started":2}
I20260709 13:00:31.115820  1603 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 694) took 1714 ms. Trace:
I20260709 13:00:31.075094  5224 raft_consensus.cc:2749] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260709 13:00:31.177093  1645 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:31.114720  1600 rpcz_store.cc:276] 0709 13:00:29.399253 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:29.488249 (+ 88996us) service_pool.cc:224] Handling call
0709 13:00:29.488590 (+   341us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:29.488872 (+   282us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 13:00:30.572570 (+1083698us) log.cc:844] Serialized 14 byte log entry
0709 13:00:31.056185 (+483615us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:00:31.056248 (+    63us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:00:31.056317 (+    69us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:31.056399 (+    82us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:31.056547 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8859,"prepare.queue_time_us":714900,"prepare.run_cpu_time_us":54,"prepare.run_wall_time_us":56,"raft.queue_time_us":24837,"raft.run_cpu_time_us":194,"raft.run_wall_time_us":29951,"thread_start_us":1547474,"threads_started":2}
I20260709 13:00:31.132905  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:31.114933  5210 raft_consensus.cc:493] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:31.163749  1603 rpcz_store.cc:276] 0709 13:00:29.401023 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:29.622120 (+221097us) service_pool.cc:224] Handling call
0709 13:00:29.700454 (+ 78334us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:29.700779 (+   325us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 13:00:30.609291 (+908512us) log.cc:844] Serialized 14 byte log entry
0709 13:00:31.082466 (+473175us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:00:31.082555 (+    89us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:00:31.082639 (+    84us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:31.082736 (+    97us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:31.115710 (+ 32974us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":30715,"prepare.queue_time_us":503559,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":41,"raft.queue_time_us":30886,"spinlock_wait_cycles":65294720,"thread_start_us":1304105,"threads_started":2}
I20260709 13:00:31.143437  1571 leader_election.cc:304] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:00:31.165448  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:31.195915  1672 raft_consensus.cc:2468] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 8.
W20260709 13:00:31.202772  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 59459.4us
I20260709 13:00:31.197356  5129 raft_consensus.cc:493] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:00:31.198292  1643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:31.203871  1571 leader_election.cc:304] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:00:31.052244  5391 raft_consensus.cc:515] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:31.204756  1596 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 2104) took 1872 ms. Trace:
I20260709 13:00:31.198660  2333 raft_consensus.cc:1240] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 5. Current term is 6. Ops: []
I20260709 13:00:31.205178  1596 rpcz_store.cc:276] 0709 13:00:29.335224 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:29.345515 (+ 10291us) service_pool.cc:224] Handling call
0709 13:00:29.345880 (+   365us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:00:29.754997 (+409117us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:00:29.755048 (+    51us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:00:29.769180 (+ 14132us) log.cc:844] Serialized 26 byte log entry
0709 13:00:30.668699 (+899519us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:00:30.668768 (+    69us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:30.668871 (+   103us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:00:31.204516 (+535645us) raft_consensus.cc:1685] finished
0709 13:00:31.204539 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:31.204683 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":9736,"raft.queue_time_us":495684,"raft.run_cpu_time_us":186,"raft.run_wall_time_us":13935,"thread_start_us":1249619,"threads_started":2,"wal-append.queue_time_us":1119283}
W20260709 13:00:31.194641  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 406755us
I20260709 13:00:31.196360  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:31.157375  1678 raft_consensus.cc:2468] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 7.
I20260709 13:00:31.134112  5183 raft_consensus.cc:493] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:00:31.211577  2299 leader_election.cc:304] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
W20260709 13:00:31.212316  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 526022us
I20260709 13:00:31.213555  1646 leader_election.cc:304] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 13:00:31.213963  2332 raft_consensus.cc:1240] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 7. Current term is 8. Ops: []
I20260709 13:00:31.212174  1747 raft_consensus.cc:1275] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260709 13:00:31.215317  1646 leader_election.cc:304] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:00:31.215823  1747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43156 (request call id 652) took 1197 ms. Trace:
I20260709 13:00:31.216193  1747 rpcz_store.cc:276] 0709 13:00:30.018876 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:30.020585 (+  1709us) service_pool.cc:224] Handling call
0709 13:00:31.211581 (+1190996us) spinlock_profiling.cc:243] Waited 27 ms on lock 0x7b58001ffe98. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f426303db71 00007f426303da22 00007f426bf56c09 00007f426b0b0ee0 00007f426b17a0ab 00007f426b17a131 00007f426b13bfc0 00007f426b126929 00007f426b1263d2 00007f426469dbd4
0709 13:00:31.211825 (+   244us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:31.215560 (+  3735us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:31.215755 (+   195us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":62046976}
I20260709 13:00:31.160751  5300 raft_consensus.cc:697] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 6 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
W20260709 13:00:31.202834  5401 log.cc:927] Time spent T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.244s	user 0.000s	sys 0.001s
I20260709 13:00:31.110946  5393 raft_consensus.cc:515] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:31.220502  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:31.231200  2332 raft_consensus.cc:2468] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 6.
I20260709 13:00:31.131151  5081 consensus_queue.cc:1048] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.074s
I20260709 13:00:31.199155  5079 leader_election.cc:290] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:31.203218  5383 leader_election.cc:290] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:31.238442  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:31.202386  5417 raft_consensus.cc:493] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:31.233901  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:31.251930  2336 raft_consensus.cc:2393] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 12: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:00:31.201712  5223 raft_consensus.cc:493] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:00:31.239804  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:00:31.253760  2337 raft_consensus.cc:2376] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 8. Re-sending same reply.
I20260709 13:00:31.204533  5215 leader_election.cc:290] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:31.225322  5356 raft_consensus.cc:2804] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Leader election won for term 10
I20260709 13:00:31.233227  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:00:31.212152  4746 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.535s	user 0.004s	sys 0.016s
I20260709 13:00:31.243669  5425 raft_consensus.cc:493] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:00:31.127981  5243 consensus_queue.cc:260] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:31.236843  5296 raft_consensus.cc:2804] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Leader pre-election won for term 10
W20260709 13:00:31.246613  5407 log.cc:927] Time spent T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.238s	user 0.001s	sys 0.000s
I20260709 13:00:31.247043  5362 raft_consensus.cc:493] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:31.246970  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:31.218977  5247 raft_consensus.cc:2749] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:00:31.211391  5405 raft_consensus.cc:2749] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:00:31.241693  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:31.227178  5420 raft_consensus.cc:493] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:31.225417  5342 raft_consensus.cc:2804] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Leader election won for term 11
W20260709 13:00:31.261955  1718 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.658s	user 0.015s	sys 0.007s
W20260709 13:00:31.300984  1718 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.697s	user 0.015s	sys 0.007s
W20260709 13:00:31.301390  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 697810us
I20260709 13:00:31.302215  1718 leader_election.cc:400] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 8 election: Vote denied by peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261) with higher term. Message: Invalid argument: T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 8. Current term is 9.
I20260709 13:00:31.301955  2301 leader_election.cc:304] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:00:31.302629  1718 leader_election.cc:403] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 8 election: Cancelling election due to peer responding with higher term
W20260709 13:00:31.265224  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 208139us
I20260709 13:00:31.269655  1604 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 693) took 1868 ms. Trace:
I20260709 13:00:31.303320  1604 rpcz_store.cc:276] 0709 13:00:29.400738 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:29.499788 (+ 99050us) service_pool.cc:224] Handling call
0709 13:00:29.500175 (+   387us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:29.500487 (+   312us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 13:00:30.999559 (+1499072us) log.cc:844] Serialized 14 byte log entry
0709 13:00:31.269157 (+269598us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:00:31.269242 (+    85us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:00:31.269319 (+    77us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:31.269396 (+    77us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:31.269565 (+   169us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":703635,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":43,"thread_start_us":1665894,"threads_started":2}
W20260709 13:00:31.280315  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 68815.8us
W20260709 13:00:31.263792  4746 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.587s	user 0.004s	sys 0.017s
W20260709 13:00:31.284335  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 77037.6us
I20260709 13:00:31.268409  5214 leader_election.cc:290] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:31.234922  5224 consensus_queue.cc:1048] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.036s
I20260709 13:00:31.225255  5129 raft_consensus.cc:515] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:31.227429  5183 raft_consensus.cc:515] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:31.206110  5210 raft_consensus.cc:515] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:31.262884  5079 raft_consensus.cc:493] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:31.285301  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:31.356951  2330 raft_consensus.cc:2468] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 8.
I20260709 13:00:31.304739  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:31.358705  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:31.359402  2332 raft_consensus.cc:2468] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 8.
I20260709 13:00:31.278930  5296 raft_consensus.cc:493] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:00:31.345369  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 135824us
I20260709 13:00:31.338572  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:31.369755  1679 raft_consensus.cc:2393] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 11: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:00:31.259181  5417 raft_consensus.cc:515] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:31.306284  5367 log.cc:927] Time spent T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 1.651s	user 0.004s	sys 0.015s
W20260709 13:00:31.383785  5408 log.cc:927] Time spent T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.165s	user 0.000s	sys 0.001s
I20260709 13:00:31.384830  4896 leader_election.cc:290] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:00:31.385617  5419 log.cc:927] Time spent T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.094s	user 0.000s	sys 0.000s
I20260709 13:00:31.388012  2299 leader_election.cc:304] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:00:31.341846  5215 raft_consensus.cc:2804] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:00:31.390869  5215 raft_consensus.cc:493] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:31.391340  5215 raft_consensus.cc:3060] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:00:31.394994  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:31.272202  5425 raft_consensus.cc:515] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:31.397306  5081 raft_consensus.cc:2804] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:00:31.397831  5081 raft_consensus.cc:493] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:31.398206  5081 raft_consensus.cc:3060] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:00:31.398363  5354 raft_consensus.cc:493] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:00:31.314339  5243 raft_consensus.cc:3060] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Advancing to term 8
W20260709 13:00:31.356973  5418 log.cc:927] Time spent T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.116s	user 0.000s	sys 0.001s
I20260709 13:00:31.379181  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:31.259487  5223 raft_consensus.cc:515] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:31.370673  1571 leader_election.cc:304] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:00:31.381181  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:31.372285  5296 raft_consensus.cc:3060] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:00:31.416847  1674 raft_consensus.cc:2393] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 7: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:00:31.346766  5342 raft_consensus.cc:697] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:00:31.375319  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:31.271833  5356 raft_consensus.cc:697] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:00:31.371886  5228 raft_consensus.cc:493] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:31.395000  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:31.292737  5362 raft_consensus.cc:515] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:31.389801  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
W20260709 13:00:31.422714  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 112286us
I20260709 13:00:31.424137  2337 raft_consensus.cc:2368] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 8. Current term is 9.
I20260709 13:00:31.340886  5303 raft_consensus.cc:3060] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:00:31.404398  1671 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52778 (request call id 1956) took 1687 ms. Trace:
I20260709 13:00:31.380337  4985 leader_election.cc:290] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:31.428580  1671 rpcz_store.cc:276] 0709 13:00:29.717253 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:29.717389 (+   136us) service_pool.cc:224] Handling call
0709 13:00:29.752990 (+ 35601us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:00:29.753468 (+   478us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:00:29.753493 (+    25us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:00:29.769680 (+ 16187us) log.cc:844] Serialized 26 byte log entry
0709 13:00:30.661651 (+891971us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:00:30.661702 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:30.661771 (+    69us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:00:31.404055 (+742284us) raft_consensus.cc:1685] finished
0709 13:00:31.404098 (+    43us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:31.404305 (+   207us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":129942,"raft.run_cpu_time_us":113,"raft.run_wall_time_us":113,"thread_start_us":783326,"threads_started":1,"wal-append.queue_time_us":1134209}
I20260709 13:00:31.375495  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:00:31.405349  1643 leader_election.cc:336] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 11 election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 1.583s (SENT)
I20260709 13:00:31.407974  1599 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 2105) took 2072 ms. Trace:
W20260709 13:00:31.413058  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 59904.4us
I20260709 13:00:31.410208  2334 raft_consensus.cc:2468] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 8.
I20260709 13:00:31.430998  1599 rpcz_store.cc:276] 0709 13:00:29.335577 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:29.336907 (+  1330us) service_pool.cc:224] Handling call
0709 13:00:29.337383 (+   476us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:00:30.516001 (+1178618us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:00:30.516040 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:00:30.552274 (+ 36234us) log.cc:844] Serialized 26 byte log entry
0709 13:00:31.048695 (+496421us) raft_consensus.cc:1645] Marking committed up to 3
0709 13:00:31.048774 (+    79us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:31.048870 (+    96us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:00:31.407643 (+358773us) raft_consensus.cc:1685] finished
0709 13:00:31.407672 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:31.407884 (+   212us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":399499,"raft.run_cpu_time_us":216,"raft.run_wall_time_us":28149,"thread_start_us":1631334,"threads_started":2,"wal-append.queue_time_us":674646}
I20260709 13:00:31.421413  1598 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 698) took 2019 ms. Trace:
I20260709 13:00:31.402801  5431 raft_consensus.cc:493] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:31.431733  1569 leader_election.cc:304] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:00:31.432964  1598 rpcz_store.cc:276] 0709 13:00:29.402244 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:29.896513 (+494269us) service_pool.cc:224] Handling call
0709 13:00:29.896932 (+   419us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:00:29.897437 (+   505us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 13:00:30.674574 (+777137us) log.cc:844] Serialized 14 byte log entry
0709 13:00:31.209734 (+535160us) log.cc:844] Serialized 14 byte log entry
0709 13:00:31.276973 (+ 67239us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:00:31.285694 (+  8721us) log.cc:844] Serialized 26 byte log entry
0709 13:00:31.285912 (+   218us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:00:31.285950 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:31.286038 (+    88us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:00:31.421133 (+135095us) raft_consensus.cc:1685] finished
0709 13:00:31.421160 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:31.421322 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":7045,"prepare.queue_time_us":630640,"prepare.run_cpu_time_us":109,"prepare.run_wall_time_us":109,"raft.queue_time_us":102089,"raft.run_cpu_time_us":435,"raft.run_wall_time_us":48063,"spinlock_wait_cycles":20036864,"thread_start_us":1218615,"threads_started":3,"wal-append.queue_time_us":637146}
I20260709 13:00:31.426079  2299 leader_election.cc:304] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:00:31.437326  1644 leader_election.cc:304] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:00:31.355983  5300 consensus_queue.cc:237] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } }
I20260709 13:00:31.420590  5224 consensus_queue.cc:1048] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.021s
I20260709 13:00:31.400027  5079 raft_consensus.cc:515] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:31.432999  5247 raft_consensus.cc:3055] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:00:31.407516  5354 raft_consensus.cc:515] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:00:31.475000  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 104455us
I20260709 13:00:31.455513  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:31.475909  1677 raft_consensus.cc:2468] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 8.
I20260709 13:00:31.463402  1749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43156 (request call id 649) took 2439 ms. Trace:
I20260709 13:00:31.473253  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:31.474287  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:31.470621  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:31.478318  1753 raft_consensus.cc:2468] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 7.
I20260709 13:00:31.463347  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:31.477874  1749 rpcz_store.cc:276] 0709 13:00:29.024022 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:29.024174 (+   152us) service_pool.cc:224] Handling call
0709 13:00:29.780913 (+756739us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:00:29.781479 (+   566us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 13:00:30.525041 (+743562us) log.cc:844] Serialized 14 byte log entry
0709 13:00:31.108001 (+582960us) log.cc:844] Serialized 14 byte log entry
0709 13:00:31.108190 (+   189us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:00:31.108820 (+   630us) log.cc:844] Serialized 26 byte log entry
0709 13:00:31.110294 (+  1474us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:00:31.110335 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:31.110420 (+    85us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:00:31.396888 (+286468us) raft_consensus.cc:1685] finished
0709 13:00:31.396912 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:31.463202 (+ 66290us) spinlock_profiling.cc:243] Waited 756 ms on lock 0x7b5400385250. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661618ad 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:00:31.463331 (+   129us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":1025803,"prepare.run_cpu_time_us":92,"prepare.run_wall_time_us":95,"raft.queue_time_us":139299,"raft.run_cpu_time_us":484,"raft.run_wall_time_us":52914,"spinlock_wait_cycles":1739294976,"thread_start_us":1324888,"threads_started":2,"wal-append.queue_time_us":745463}
I20260709 13:00:31.462946  1750 raft_consensus.cc:1275] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20260709 13:00:31.470960  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:31.485906  1751 raft_consensus.cc:2468] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 5.
I20260709 13:00:31.467473  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:31.466250  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:31.487077  1754 raft_consensus.cc:2468] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 8.
I20260709 13:00:31.486959  1644 leader_election.cc:304] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:00:31.487522  1752 raft_consensus.cc:2468] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 8.
W20260709 13:00:31.454370  1502 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:43476 (request call id 192) took 26850 ms (client timeout 15000 ms). Trace:
I20260709 13:00:31.490880  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:31.491593  1752 raft_consensus.cc:3060] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:00:31.499058  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:31.479921  1748 raft_consensus.cc:2468] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 7.
I20260709 13:00:31.325460  5420 raft_consensus.cc:515] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:31.512238  1745 raft_consensus.cc:1240] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 9. Current term is 10. Ops: []
I20260709 13:00:31.449538  5007 tablet_bootstrap.cc:492] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=5 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
W20260709 13:00:31.463080  5423 log.cc:927] Time spent T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.108s	user 0.000s	sys 0.000s
W20260709 13:00:31.517688  1569 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:31.522879  5426 raft_consensus.cc:493] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:31.549679  5426 raft_consensus.cc:515] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:31.456598  5210 leader_election.cc:290] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:31.551069  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:31.552543  5426 leader_election.cc:290] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:31.554319  5420 leader_election.cc:290] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:00:31.555788  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 79545us
I20260709 13:00:31.557963  1571 leader_election.cc:304] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:00:31.524643  4746 raft_consensus.cc:2804] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:00:31.558756  4746 raft_consensus.cc:493] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:31.559194  4746 raft_consensus.cc:3060] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:00:31.559597  5223 leader_election.cc:290] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:31.472173  5425 leader_election.cc:290] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:31.560597  5342 consensus_queue.cc:237] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 9.6, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:00:31.530237  1643 leader_election.cc:336] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 2.194s (SENT)
I20260709 13:00:31.532480  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:31.562884  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:31.461797  5255 raft_consensus.cc:493] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:31.533565  1750 raft_consensus.cc:1240] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 6. Current term is 7. Ops: []
I20260709 13:00:31.534158  1749 raft_consensus.cc:1240] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 7. Current term is 8. Ops: []
I20260709 13:00:31.565409  2330 raft_consensus.cc:2468] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 9.
I20260709 13:00:31.534220  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:00:31.511593  5411 log.cc:927] Time spent T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.476s	user 0.003s	sys 0.003s
I20260709 13:00:31.447914  5233 raft_consensus.cc:3055] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:00:31.510007  4896 raft_consensus.cc:2804] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Leader election won for term 8
I20260709 13:00:31.461452  5247 raft_consensus.cc:740] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:00:31.512930  5182 raft_consensus.cc:515] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:31.461057  5183 leader_election.cc:290] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 13:00:31.543110  5086 consensus_meta.cc:287] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.983s	user 0.017s	sys 0.000s
W20260709 13:00:31.548036  5390 consensus_meta.cc:287] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.521s	user 0.010s	sys 0.003s
I20260709 13:00:31.459656  5403 leader_election.cc:290] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:31.536630  5214 raft_consensus.cc:2804] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:00:31.559311  5440 consensus_queue.cc:1048] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 13:00:31.474403  5428 raft_consensus.cc:2804] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:00:31.474108  5251 raft_consensus.cc:2749] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:00:31.692262  5247 consensus_queue.cc:260] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 7.9, Last appended by leader: 9, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:31.523871  1748 raft_consensus.cc:1240] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 8. Current term is 9. Ops: []
I20260709 13:00:31.697453  5247 raft_consensus.cc:3060] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:00:31.530936  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:31.529685  1754 raft_consensus.cc:1240] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 7. Current term is 8. Ops: []
W20260709 13:00:31.561178  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 48998us
I20260709 13:00:31.703239  5007 tablet_bootstrap.cc:492] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1: Bootstrap complete.
I20260709 13:00:31.704586  5007 ts_tablet_manager.cc:1403] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1: Time spent bootstrapping tablet: real 3.416s	user 0.057s	sys 0.047s
I20260709 13:00:31.573382  5233 raft_consensus.cc:740] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:00:31.457731  5079 leader_election.cc:290] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:31.459052  5380 raft_consensus.cc:2804] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:00:31.570359  1571 leader_election.cc:304] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
W20260709 13:00:31.716213  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 145917us
I20260709 13:00:31.564944  5255 raft_consensus.cc:515] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:31.625105  1752 raft_consensus.cc:2468] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 11.
I20260709 13:00:31.433012  5228 raft_consensus.cc:515] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:31.664307  2333 raft_consensus.cc:1240] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 8 LEADER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 7. Current term is 8. Ops: []
I20260709 13:00:31.665998  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:31.684211  1646 leader_election.cc:304] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:00:31.725056  1646 leader_election.cc:304] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:00:31.568764  5420 raft_consensus.cc:493] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:31.445856  5431 raft_consensus.cc:515] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } }
I20260709 13:00:31.696676  5428 raft_consensus.cc:493] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:31.458904  5129 leader_election.cc:290] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:31.665570  5325 raft_consensus.cc:493] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:31.462453  5356 consensus_queue.cc:237] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:31.714629  2299 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:00:31.695438  5388 consensus_meta.cc:287] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.583s	user 0.012s	sys 0.003s
I20260709 13:00:31.717159  1571 leader_election.cc:304] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 13:00:31.469604  5417 leader_election.cc:290] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:31.741494  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:00:31.489174  1502 rpcz_store.cc:269] 0709 13:00:04.604006 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:04.604372 (+   366us) service_pool.cc:224] Handling call
0709 13:00:31.454068 (+26849696us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 13:00:18.737993 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 13:00:19.203711 (+465718us) write_op.cc:432] Acquiring schema lock in shared mode
0709 13:00:19.216215 (+ 12504us) write_op.cc:435] Acquired schema lock
0709 13:00:19.216236 (+    21us) tablet.cc:662] Decoding operations
0709 13:00:19.331231 (+114995us) write_op.cc:620] Acquiring the partition lock for write op
0709 13:00:19.334554 (+  3323us) write_op.cc:641] Partition lock acquired for write op
0709 13:00:19.337833 (+  3279us) tablet.cc:685] Acquiring locks for 18 operations
0709 13:00:19.365924 (+ 28091us) tablet.cc:701] Row locks acquired
0709 13:00:19.365981 (+    57us) write_op.cc:260] PREPARE: finished
0709 13:00:19.366216 (+   235us) write_op.cc:270] Start()
0709 13:00:19.366407 (+   191us) write_op.cc:276] Timestamp: P: 1783602019366126 usec, L: 0
0709 13:00:19.366454 (+    47us) op_driver.cc:348] REPLICATION: starting
0709 13:00:19.618178 (+251724us) log.cc:844] Serialized 7048 byte log entry
0709 13:00:20.809829 (+1191651us) spinlock_profiling.cc:243] Waited 20.3 ms on lock 0x7b4400022b4c. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f4262f9bc85 00007f4262f9c6a2 00007f4265d0e1db 00007f4265d09aaf 00007f4265cfd86e 00007f4265cfcbb6 00007f4265cfc7d8 00007f4265d06221 00007f4265d061d9 00007f4265d06169
0709 13:00:21.419725 (+609896us) op_driver.cc:464] REPLICATION: finished
0709 13:00:22.040547 (+620822us) write_op.cc:301] APPLY: starting
0709 13:00:22.067516 (+ 26969us) tablet.cc:1367] starting BulkCheckPresence
0709 13:00:22.270055 (+202539us) tablet.cc:1370] finished BulkCheckPresence
0709 13:00:22.281359 (+ 11304us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 13:00:22.493155 (+211796us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 13:00:22.494007 (+   852us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=18
0709 13:00:22.494105 (+    98us) write_op.cc:312] APPLY: finished
0709 13:00:22.829222 (+335117us) log.cc:844] Serialized 125 byte log entry
0709 13:00:23.346149 (+516927us) spinlock_profiling.cc:243] Waited 21 ms on lock 0x7b540041cb54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f426615f32a 00007f42660e4f14 00007f42660e4ec3 00007f42660e4e38 00007f42660e4df3 00007f42660e40dc 00007f426610de16 00007f42660f5cbb 00007f42660e039d 00007f42660e3d75
0709 13:00:23.932934 (+586785us) spinlock_profiling.cc:243] Waited 12.2 ms on lock 0x7b4400022b4c. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f4262f9bc85 00007f4262f9c6a2 00007f4265d0e1db 00007f4265d09aaf 00007f42630580ac 00007f426beb2ab9 00007f426307469b 00007f426307293f 00007f4263074c02 00007f4265cfc20b
0709 13:00:24.023807 (+ 90873us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:00:24.024214 (+   407us) write_op.cc:454] Released schema lock
0709 13:00:24.024340 (+   126us) write_op.cc:341] FINISH: Updating metrics
0709 13:00:24.025018 (+   678us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:00:24.025046 (+    28us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":73102,"reported_tablets":35,"child_traces":[["op",{"apply.queue_time_us":551419,"apply.run_cpu_time_us":30190,"apply.run_wall_time_us":2017518,"num_ops":18,"prepare.queue_time_us":182991,"prepare.run_cpu_time_us":21826,"prepare.run_wall_time_us":1173025,"raft.queue_time_us":504700,"raft.run_cpu_time_us":40333,"raft.run_wall_time_us":2028505,"replication_time_us":2001794,"spinlock_wait_cycles":123060096,"thread_start_us":1949836,"threads_started":5,"wal-append.queue_time_us":1444146,"wal-append.run_cpu_time_us":66092,"wal-append.run_wall_time_us":3331340}]]}
I20260709 13:00:31.677197  4896 raft_consensus.cc:697] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:00:31.658026  4872 raft_consensus.cc:2749] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 13:00:31.710211  5214 raft_consensus.cc:697] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 7 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:00:31.734303  1673 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52766 (request call id 2376) took 2528 ms. Trace:
I20260709 13:00:31.737531  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:31.738379  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:31.739413  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:31.715523  5380 raft_consensus.cc:493] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:00:31.690366  5215 raft_consensus.cc:515] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:31.688074  5086 raft_consensus.cc:515] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:31.717779  5251 raft_consensus.cc:2749] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:00:31.742203  1749 raft_consensus.cc:2468] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 6.
I20260709 13:00:31.744328  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:31.788478  1600 raft_consensus.cc:2468] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 8.
I20260709 13:00:31.840313  1754 raft_consensus.cc:2468] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 10.
I20260709 13:00:31.745012  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:31.786724  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:31.833828  1596 raft_consensus.cc:2393] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 9: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:00:31.731155  5359 raft_consensus.cc:2804] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:00:31.790455  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:31.752120  2301 leader_election.cc:304] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:00:31.792048  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:31.730214  5342 raft_consensus.cc:2804] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:00:31.841663  1745 raft_consensus.cc:2468] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 9.
I20260709 13:00:31.750276  5233 consensus_queue.cc:260] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:31.750327  1673 rpcz_store.cc:276] 0709 13:00:29.206623 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:29.207084 (+   461us) service_pool.cc:224] Handling call
0709 13:00:29.207538 (+   454us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:00:29.757508 (+549970us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:00:29.757540 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:00:29.773148 (+ 15608us) log.cc:844] Serialized 26 byte log entry
0709 13:00:30.827765 (+1054617us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:00:30.827809 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:30.827911 (+   102us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:00:31.465858 (+637947us) spinlock_profiling.cc:243] Waited 4.93 ms on lock 0x7b4400022b4c. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f4262f9bc85 00007f4262f9c6a2 00007f4265d0e1db 00007f4265d09aaf 00007f4265cfd86e 00007f4265cfcbb6 00007f4265cfc7d8 00007f4265d06221 00007f4265d061d9 00007f4265d06169
0709 13:00:31.733833 (+267975us) raft_consensus.cc:1685] finished
0709 13:00:31.733868 (+    35us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:31.734216 (+   348us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":63699,"raft.queue_time_us":127427,"raft.run_cpu_time_us":103,"raft.run_wall_time_us":103,"spinlock_wait_cycles":11338112,"thread_start_us":1397755,"threads_started":2,"wal-append.queue_time_us":1174656}
I20260709 13:00:31.841137  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:31.789098  1598 raft_consensus.cc:2468] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 7.
W20260709 13:00:31.855015  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 102971us
I20260709 13:00:31.855520  1600 raft_consensus.cc:3060] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:00:31.856009  2301 leader_election.cc:304] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
W20260709 13:00:31.793900  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 68939.1us
I20260709 13:00:31.857146  1646 leader_election.cc:304] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:00:31.693696  5390 raft_consensus.cc:515] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:31.731369  5428 raft_consensus.cc:3060] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:00:31.858946  1646 leader_election.cc:304] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:00:31.749223  5224 raft_consensus.cc:2804] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:00:31.864928  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:31.790958  5380 raft_consensus.cc:3060] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:00:31.900059  1605 raft_consensus.cc:1275] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 13:00:31.896836  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:31.730836  5420 raft_consensus.cc:515] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:31.904371  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:31.751616  5388 raft_consensus.cc:2749] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:00:31.912921  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:31.913182  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:31.908188  1599 raft_consensus.cc:2393] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 7: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:00:31.739976  5446 consensus_queue.cc:1048] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.127s
I20260709 13:00:31.853820  5342 raft_consensus.cc:493] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:00:31.854043  5359 raft_consensus.cc:493] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:00:31.924103  1603 raft_consensus.cc:1240] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 6. Current term is 7. Ops: []
I20260709 13:00:31.791364  5417 consensus_queue.cc:1048] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.114s
I20260709 13:00:31.865798  4951 raft_consensus.cc:2804] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:00:31.925007  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:31.925431  1600 raft_consensus.cc:2468] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 8.
I20260709 13:00:31.900023  5233 raft_consensus.cc:3060] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Advancing to term 8
W20260709 13:00:31.928330  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 69475.1us
I20260709 13:00:31.934329  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:31.897492  5224 raft_consensus.cc:493] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:31.936441  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:31.936764  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:31.937978  2331 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39726 (request call id 931) took 3006 ms. Trace:
W20260709 13:00:31.938264  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 82380.8us
I20260709 13:00:31.938470  1601 raft_consensus.cc:1275] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260709 13:00:31.938498  2331 rpcz_store.cc:276] 0709 13:00:28.932065 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:28.972710 (+ 40645us) service_pool.cc:224] Handling call
0709 13:00:28.973119 (+   409us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:00:31.080749 (+2107630us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:31.935078 (+854329us) spinlock_profiling.cc:243] Waited 2.07 s on lock 0x7b54006bfb54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:00:31.937907 (+  2829us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4757140736}
I20260709 13:00:31.939550  2301 leader_election.cc:304] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:00:31.913710  5183 raft_consensus.cc:2804] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:00:31.938508  1599 raft_consensus.cc:2468] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 6.
W20260709 13:00:31.970784  5422 log.cc:927] Time spent T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.058s	user 0.001s	sys 0.000s
I20260709 13:00:31.940614  2301 leader_election.cc:304] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:00:31.924957  5359 raft_consensus.cc:3060] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:00:31.973507  1600 raft_consensus.cc:2410] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for term 12 because replica has last-logged OpId of term: 10 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 9.
I20260709 13:00:31.974265  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:31.979219  1673 raft_consensus.cc:2468] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 9.
I20260709 13:00:31.735384  5325 raft_consensus.cc:515] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:31.935141  4951 raft_consensus.cc:493] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:31.927731  5342 raft_consensus.cc:3060] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:00:31.933920  5403 raft_consensus.cc:493] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:31.943442  5224 raft_consensus.cc:3060] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:00:31.985881  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:31.927413  5454 raft_consensus.cc:2749] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 13:00:31.942694  5397 raft_consensus.cc:2804] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:00:31.972916  5183 raft_consensus.cc:493] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:00:32.005776  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:32.010106  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:32.010900  1599 raft_consensus.cc:2468] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 6.
I20260709 13:00:32.012524  5255 leader_election.cc:290] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:32.012919  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:32.014331  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:31.998003  1601 raft_consensus.cc:2468] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 8.
I20260709 13:00:31.988014  4951 raft_consensus.cc:3060] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:00:31.993077  5391 leader_election.cc:290] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:31.980049  5172 consensus_queue.cc:1059] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.033s
I20260709 13:00:32.001960  5397 raft_consensus.cc:493] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:32.027693  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:32.028345  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:32.036509  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:32.043870  1600 raft_consensus.cc:1275] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260709 13:00:32.010879  5183 raft_consensus.cc:3060] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:00:31.932226  4896 consensus_queue.cc:237] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:32.015504  5429 log.cc:927] Time spent T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.293s	user 0.000s	sys 0.001s
I20260709 13:00:32.046345  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:32.054908  5172 raft_consensus.cc:3055] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:00:31.943498  5214 consensus_queue.cc:237] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:31.992151  5403 raft_consensus.cc:515] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:32.095880  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:32.086184  1602 raft_consensus.cc:2468] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 9.
W20260709 13:00:32.042881  5449 log.cc:927] Time spent T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.185s	user 0.001s	sys 0.000s
W20260709 13:00:32.086860  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 146388us
W20260709 13:00:32.037727  5448 log.cc:927] Time spent T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.130s	user 0.000s	sys 0.000s
I20260709 13:00:32.038132  5215 leader_election.cc:290] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 9 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:32.102936  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:32.112825  2301 leader_election.cc:304] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:00:32.104295  1601 raft_consensus.cc:3060] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Advancing to term 12
W20260709 13:00:32.093291  4746 consensus_meta.cc:287] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.534s	user 0.013s	sys 0.001s
W20260709 13:00:32.104830  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 168776us
I20260709 13:00:32.102490  1604 raft_consensus.cc:2468] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 7.
I20260709 13:00:32.056769  5391 raft_consensus.cc:2804] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:00:32.045174  5325 leader_election.cc:290] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:32.094947  5172 raft_consensus.cc:740] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:00:32.128592  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:32.123767  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:32.051122  5397 raft_consensus.cc:3060] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Advancing to term 8
W20260709 13:00:32.159911  1753 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.526s	user 0.012s	sys 0.023s
I20260709 13:00:32.223732  5452 raft_consensus.cc:2804] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:00:32.235940  5452 raft_consensus.cc:493] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:00:32.236304  5247 consensus_meta.cc:287] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.538s	user 0.014s	sys 0.000s
I20260709 13:00:32.236497  5452 raft_consensus.cc:3060] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Advancing to term 11
W20260709 13:00:32.183917  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 48548.5us
I20260709 13:00:32.225050  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:32.238106  2331 raft_consensus.cc:1240] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 8 LEADER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 7. Current term is 8. Ops: [7.7-7.7]
I20260709 13:00:32.238525  1646 leader_election.cc:304] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:00:32.139605  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:32.231317  1602 raft_consensus.cc:2468] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 9.
W20260709 13:00:32.231565  1753 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.597s	user 0.012s	sys 0.029s
I20260709 13:00:32.038715  5431 leader_election.cc:290] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:32.053871  5182 leader_election.cc:290] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 10 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:32.241820  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:32.231359  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:00:32.217391  5081 consensus_meta.cc:287] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.819s	user 0.012s	sys 0.002s
I20260709 13:00:32.247454  1599 raft_consensus.cc:2468] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 6.
W20260709 13:00:32.156687  5455 log.cc:927] Time spent T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.051s	user 0.001s	sys 0.000s
I20260709 13:00:32.121014  5417 raft_consensus.cc:2804] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:00:32.123425  5228 leader_election.cc:290] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:32.211805  5420 leader_election.cc:290] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:32.257280  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:32.259333  1602 raft_consensus.cc:2393] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 9: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:00:32.260371  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:32.261276  1599 raft_consensus.cc:2468] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 8.
I20260709 13:00:32.261513  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:32.236008  5083 leader_election.cc:290] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:32.238605  5391 raft_consensus.cc:493] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:32.266278  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:32.267323  1602 raft_consensus.cc:2468] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 7.
W20260709 13:00:32.256109  5450 log.cc:927] Time spent T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.291s	user 0.000s	sys 0.001s
W20260709 13:00:32.256330  5451 log.cc:927] Time spent T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.290s	user 0.001s	sys 0.000s
I20260709 13:00:32.147892  5454 consensus_queue.cc:1059] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.120s
I20260709 13:00:32.274231  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:32.275399  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:32.255218  5417 raft_consensus.cc:493] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:32.124974  4746 raft_consensus.cc:515] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:32.259338  5466 raft_consensus.cc:2804] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Leader election won for term 11
I20260709 13:00:32.217377  5007 raft_consensus.cc:359] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:32.269982  5391 raft_consensus.cc:3060] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Advancing to term 7
W20260709 13:00:32.299590  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 46856.7us
I20260709 13:00:32.300099  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:32.303206  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:32.304139  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:32.304839  1602 raft_consensus.cc:2468] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 6.
I20260709 13:00:32.290141  1599 raft_consensus.cc:2376] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 8. Re-sending same reply.
I20260709 13:00:32.306249  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:32.306944  1599 raft_consensus.cc:2468] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 6.
I20260709 13:00:32.307866  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:32.308454  1599 raft_consensus.cc:2468] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 6.
I20260709 13:00:32.309653  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:32.310422  1602 raft_consensus.cc:2468] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 8.
I20260709 13:00:32.311506  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:32.312232  1602 raft_consensus.cc:2468] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 7.
I20260709 13:00:32.312834  5454 raft_consensus.cc:3055] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:00:32.313344  5454 raft_consensus.cc:740] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:00:32.313153  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:32.314832  1602 raft_consensus.cc:2393] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 7: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:00:32.301636  1751 raft_consensus.cc:1240] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 7. Current term is 8. Ops: []
I20260709 13:00:32.315237  1750 raft_consensus.cc:1240] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 6 LEADER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 5. Current term is 6. Ops: []
I20260709 13:00:32.276679  5172 consensus_queue.cc:260] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:32.321517  1601 raft_consensus.cc:2410] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for term 12 because replica has last-logged OpId of term: 10 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 9.
I20260709 13:00:32.323565  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:32.256708  5458 consensus_queue.cc:1059] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.047s
I20260709 13:00:32.253827  5081 raft_consensus.cc:515] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:32.325572  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:32.326077  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:32.333582  1601 raft_consensus.cc:2376] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 9. Re-sending same reply.
I20260709 13:00:32.328190  1677 raft_consensus.cc:2393] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 9: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:00:32.329757  4746 leader_election.cc:290] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 9 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:32.330215  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:32.330758  1745 raft_consensus.cc:3055] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:00:32.331528  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:32.334909  1745 raft_consensus.cc:740] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:00:32.331976  1750 raft_consensus.cc:1240] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 5. Current term is 6. Ops: []
I20260709 13:00:32.337960  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:32.338285  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:32.339641  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:32.256871  5460 consensus_queue.cc:1059] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.075s
I20260709 13:00:32.298804  5417 raft_consensus.cc:3060] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:00:32.343894  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:32.345656  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:00:32.322316  5243 consensus_meta.cc:287] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.910s	user 0.007s	sys 0.007s
I20260709 13:00:32.323263  5172 raft_consensus.cc:3060] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Advancing to term 10
W20260709 13:00:32.350560  1571 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:32.350680  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:32.333841  5413 raft_consensus.cc:3055] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:00:32.336504  1745 consensus_queue.cc:260] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.8, Last appended by leader: 8, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:32.352010  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:32.321189  5390 leader_election.cc:290] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 7 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:32.356982  1745 raft_consensus.cc:3060] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:00:32.351338  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:32.359148  1492 maintenance_manager.cc:419] P 919f6392d4f44fd68987e2c5e7ec37ca: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033362
I20260709 13:00:32.354651  1680 raft_consensus.cc:2468] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 8.
I20260709 13:00:32.324848  5466 raft_consensus.cc:697] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:00:32.355819  1750 raft_consensus.cc:1240] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 10. Current term is 11. Ops: [10.6-10.6]
I20260709 13:00:32.355811  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:32.354267  2334 raft_consensus.cc:2393] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 9: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:00:32.363837  1748 raft_consensus.cc:2468] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 8.
I20260709 13:00:32.346413  5007 raft_consensus.cc:740] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Initialized, Role: FOLLOWER
W20260709 13:00:32.357061  5296 consensus_meta.cc:287] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.937s	user 0.016s	sys 0.000s
I20260709 13:00:32.353971  5403 leader_election.cc:290] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:32.424561  5403 raft_consensus.cc:493] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:32.425185  5403 raft_consensus.cc:515] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:32.378207  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:32.388563  1676 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52778 (request call id 1966) took 1046 ms. Trace:
W20260709 13:00:32.398207  1569 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:32.396576  5081 leader_election.cc:290] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 9 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:32.405136  5393 leader_election.cc:290] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:32.408157  2333 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49082 (request call id 787) took 1197 ms. Trace:
I20260709 13:00:32.459914  2333 rpcz_store.cc:276] 0709 13:00:31.210951 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:31.224973 (+ 14022us) service_pool.cc:224] Handling call
0709 13:00:31.232813 (+  7840us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:31.726632 (+493819us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:32.407847 (+681215us) spinlock_profiling.cc:243] Waited 431 ms on lock 0x7b5400627ad4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:00:32.408075 (+   228us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":991612288}
I20260709 13:00:32.360441  5243 raft_consensus.cc:493] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:00:32.378880  5470 consensus_queue.cc:1048] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.053s
I20260709 13:00:32.362519  5425 raft_consensus.cc:493] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:32.356577  5413 raft_consensus.cc:740] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:00:32.418799  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:32.420078  1718 leader_election.cc:304] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:00:32.422955  5466 consensus_queue.cc:237] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.6, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:32.405414  5391 raft_consensus.cc:515] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:32.434715  1676 rpcz_store.cc:276] 0709 13:00:31.341800 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:32.057502 (+715702us) service_pool.cc:224] Handling call
0709 13:00:32.058018 (+   516us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:00:32.058574 (+   556us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:00:32.058616 (+    42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:00:32.098598 (+ 39982us) log.cc:844] Serialized 26 byte log entry
0709 13:00:32.184404 (+ 85806us) raft_consensus.cc:1645] Marking committed up to 3
0709 13:00:32.184468 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:32.184580 (+   112us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:00:32.388204 (+203624us) raft_consensus.cc:1685] finished
0709 13:00:32.388238 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:32.388461 (+   223us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":135,"raft.run_cpu_time_us":191,"raft.run_wall_time_us":11204,"thread_start_us":50075,"threads_started":1,"wal-append.queue_time_us":186512}
W20260709 13:00:32.464985  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 141089us
I20260709 13:00:32.314335  5454 consensus_queue.cc:260] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:32.485550  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 65535.1us
I20260709 13:00:32.486781  1718 leader_election.cc:400] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 8 election: Vote denied by peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625) with higher term. Message: Invalid argument: T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 8. Current term is 9.
I20260709 13:00:32.487815  1718 leader_election.cc:304] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
W20260709 13:00:32.501192  5303 consensus_meta.cc:287] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 1.063s	user 0.008s	sys 0.014s
I20260709 13:00:32.501761  5303 raft_consensus.cc:2749] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Leader election lost for term 8. Reason: Vote denied by peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261) with higher term. Message: Invalid argument: T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 8. Current term is 9.
I20260709 13:00:32.453490  5224 raft_consensus.cc:515] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:32.505956  2298 leader_election.cc:304] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:32.384934  5296 raft_consensus.cc:515] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:32.426354  5359 raft_consensus.cc:515] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:32.517952  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
W20260709 13:00:32.518960  5233 consensus_meta.cc:287] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.573s	user 0.013s	sys 0.001s
I20260709 13:00:32.520078  1599 raft_consensus.cc:2410] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for term 9 because replica has last-logged OpId of term: 7 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 6.
I20260709 13:00:32.485565  5007 consensus_queue.cc:260] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 6.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:32.521085  5460 raft_consensus.cc:493] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:32.472379  4951 raft_consensus.cc:515] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:32.521670  2333 raft_consensus.cc:3060] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:00:32.521754  5083 consensus_queue.cc:1048] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 13:00:32.509560  5152 raft_consensus.cc:2804] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:00:32.526741  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:32.528247  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:00:32.535126  1718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:32.536722  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:32.537422  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:32.537539  1604 raft_consensus.cc:2468] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 9.
I20260709 13:00:32.521667  5460 raft_consensus.cc:515] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:32.540158  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:32.503737  5434 raft_consensus.cc:2804] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:00:32.541746  5434 raft_consensus.cc:493] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:00:32.515151  5214 consensus_queue.cc:1048] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.027s
I20260709 13:00:32.542256  5434 raft_consensus.cc:3060] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:00:32.470057  5243 raft_consensus.cc:515] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:32.545877  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:32.549765  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:32.551170  2329 raft_consensus.cc:2368] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for earlier term 9. Current term is 10.
I20260709 13:00:32.553800  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:32.516964  5398 raft_consensus.cc:2749] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:00:32.473033  5425 raft_consensus.cc:515] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:32.563062  2333 raft_consensus.cc:2468] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 7.
I20260709 13:00:32.529793  5152 raft_consensus.cc:493] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:32.535554  5454 raft_consensus.cc:3060] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Advancing to term 7
W20260709 13:00:32.534873  5428 consensus_meta.cc:287] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.633s	user 0.015s	sys 0.000s
I20260709 13:00:32.570890  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:32.529295  5255 consensus_queue.cc:1048] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.038s
I20260709 13:00:32.594201  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:00:32.607122  5472 log.cc:927] Time spent T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.104s	user 0.000s	sys 0.000s
I20260709 13:00:32.611717  5428 raft_consensus.cc:515] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:32.609421  5296 leader_election.cc:290] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 10 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:32.614792  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:32.616009  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:32.616128  1672 raft_consensus.cc:2468] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 8.
I20260709 13:00:32.616677  1748 raft_consensus.cc:3060] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Advancing to term 10
I20260709 13:00:32.617277  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:32.618144  5403 leader_election.cc:290] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:32.533793  5474 consensus_queue.cc:1059] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.088s
I20260709 13:00:32.622383  4896 raft_consensus.cc:493] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:32.623031  4896 raft_consensus.cc:515] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:32.576547  5152 raft_consensus.cc:3060] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:00:32.625291  4896 leader_election.cc:290] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:32.626181  1601 raft_consensus.cc:2393] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 9: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:00:32.531433  5413 consensus_queue.cc:260] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.7, Last appended by leader: 7, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:32.616129  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:32.631115  1750 raft_consensus.cc:3060] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:00:32.627748  5007 ts_tablet_manager.cc:1434] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1: Time spent starting tablet: real 0.923s	user 0.016s	sys 0.000s
I20260709 13:00:32.631424  5417 raft_consensus.cc:515] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:32.632001  5007 ts_tablet_manager.cc:1467] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1: Tablet startup took 927ms
W20260709 13:00:32.575479  5342 consensus_meta.cc:287] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.570s	user 0.013s	sys 0.000s
I20260709 13:00:32.634776  5342 raft_consensus.cc:515] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:32.636998  5342 leader_election.cc:290] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 8 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:32.638065  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:32.639106  1752 raft_consensus.cc:2393] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 7: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:00:32.628059  1677 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52766 (request call id 2390) took 1188 ms. Trace:
I20260709 13:00:32.543742  5303 consensus_queue.cc:1048] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.034s
I20260709 13:00:32.640110  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:32.643213  1752 raft_consensus.cc:2393] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 6: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:00:32.643990  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:32.645411  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 9 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:32.640028  1677 rpcz_store.cc:276] 0709 13:00:31.439999 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:31.516352 (+ 76353us) service_pool.cc:224] Handling call
0709 13:00:32.627834 (+1111482us) spinlock_profiling.cc:243] Waited 776 ms on lock 0x7b5400091d54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:00:32.627995 (+   161us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1784878464}
I20260709 13:00:32.612658  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:32.571928  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:32.656790  1750 raft_consensus.cc:2468] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 9.
I20260709 13:00:32.663252  1746 raft_consensus.cc:2468] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 9.
I20260709 13:00:32.618646  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 12 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:32.670207  1749 raft_consensus.cc:2410] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 for term 9 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260709 13:00:32.670212  1598 raft_consensus.cc:1240] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 7. Current term is 8. Ops: []
I20260709 13:00:32.673663  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:32.680764  2337 raft_consensus.cc:1275] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260709 13:00:32.649024  1643 leader_election.cc:304] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:00:32.680307  2301 leader_election.cc:304] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
W20260709 13:00:32.682399  1643 leader_election.cc:336] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 1.933s (SENT)
I20260709 13:00:32.682489  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:32.683161  1643 leader_election.cc:304] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:00:32.683460  1598 raft_consensus.cc:2468] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 8.
W20260709 13:00:32.683948  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 167444us
I20260709 13:00:32.684679  1676 raft_consensus.cc:1275] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 10 index: 9. (term mismatch)
I20260709 13:00:32.685257  1676 pending_rounds.cc:85] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d: Aborting all ops after (but not including) 8
I20260709 13:00:32.685611  1676 pending_rounds.cc:107] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d: Aborting uncommitted NO_OP operation due to leader change: 9.9
I20260709 13:00:32.686944  1717 leader_election.cc:304] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:00:32.688040  1717 leader_election.cc:304] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:00:32.688977  1717 leader_election.cc:304] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 13:00:32.698505  1676 raft_consensus.cc:2889] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:00:32.703382  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
W20260709 13:00:32.706903  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 56461.3us
I20260709 13:00:32.713289  1678 raft_consensus.cc:1240] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 LEADER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 5. Current term is 6. Ops: []
I20260709 13:00:32.713901  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:32.716302  1598 raft_consensus.cc:3060] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:00:32.717355  1604 raft_consensus.cc:3060] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:00:32.718688  1752 raft_consensus.cc:1275] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260709 13:00:32.721095  1643 leader_election.cc:304] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:00:32.736300  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:32.739847  1748 raft_consensus.cc:2468] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 10.
I20260709 13:00:32.640275  5243 leader_election.cc:290] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:32.758020  1571 leader_election.cc:304] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:00:32.759447  1571 leader_election.cc:304] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
W20260709 13:00:32.707056  5380 consensus_meta.cc:287] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.782s	user 0.010s	sys 0.005s
I20260709 13:00:32.741482  1718 leader_election.cc:304] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:32.743633  2301 leader_election.cc:304] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
W20260709 13:00:32.748567  1643 leader_election.cc:336] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 1.587s (SENT)
I20260709 13:00:32.763778  1643 leader_election.cc:304] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:32.764603  1571 leader_election.cc:304] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:00:32.765072  5342 raft_consensus.cc:2749] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:00:32.765937  1571 leader_election.cc:304] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:00:32.766402  4872 raft_consensus.cc:2749] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:00:32.766834  5388 raft_consensus.cc:2749] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:00:32.712884  5233 raft_consensus.cc:493] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:32.769595  1571 leader_election.cc:304] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:00:32.737267  5397 raft_consensus.cc:515] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:32.863567  5405 raft_consensus.cc:2749] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260709 13:00:32.864935  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 95448.8us
I20260709 13:00:32.865379  5397 leader_election.cc:290] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 8 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:32.862166  5474 raft_consensus.cc:493] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:32.811350  5391 leader_election.cc:290] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 7 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:32.825325  5475 raft_consensus.cc:2749] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:00:32.866357  5405 consensus_queue.cc:1059] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 13:00:32.779819  5182 raft_consensus.cc:2804] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Leader election won for term 10
W20260709 13:00:32.867597  5480 log.cc:927] Time spent T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.054s	user 0.000s	sys 0.000s
I20260709 13:00:32.827323  5490 raft_consensus.cc:2749] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:00:32.865020  1680 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57092 (request call id 783) took 1733 ms. Trace:
I20260709 13:00:32.779330  5247 raft_consensus.cc:2749] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20260709 13:00:32.893944  2330 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39738 (request call id 760) took 1550 ms (client timeout 2005 ms). Trace:
W20260709 13:00:32.894593  2330 rpcz_store.cc:269] 0709 13:00:31.343413 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:31.357730 (+ 14317us) service_pool.cc:224] Handling call
0709 13:00:32.893695 (+1535965us) spinlock_profiling.cc:243] Waited 143 ms on lock 0x7b540065ed54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:00:32.893863 (+   168us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":328973952}
I20260709 13:00:32.869764  1680 rpcz_store.cc:276] 0709 13:00:31.131962 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:31.132099 (+   137us) service_pool.cc:224] Handling call
0709 13:00:32.864825 (+1732726us) spinlock_profiling.cc:243] Waited 1.15 s on lock 0x7b54000d9654. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:00:32.864950 (+   125us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2638143872}
I20260709 13:00:32.896019  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:32.848069  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:32.735415  5336 raft_consensus.cc:2749] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260709 13:00:32.632369  5007 ts_tablet_manager.cc:1469] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1: Trace:
0709 13:00:19.443863 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:19.444642 (+   779us) service_pool.cc:224] Handling call
0709 13:00:20.083039 (+638397us) ts_tablet_manager.cc:934] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1: Initiating tablet copy from peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
0709 13:00:23.932567 (+3849528us) tablet_copy_client.cc:368] Tablet copy session begun
0709 13:00:24.765213 (+832646us) tablet_metadata.cc:665] Metadata flushed
0709 13:00:25.587474 (+822261us) tablet_metadata.cc:665] Metadata flushed
0709 13:00:25.809930 (+222456us) tablet_copy_client.cc:463] Replaced tombstoned tablet metadata.
0709 13:00:26.358729 (+548799us) tablet_copy_client.cc:418] Wrote new consensus metadata
0709 13:00:26.464080 (+105351us) tablet_replica.cc:206] Creating consensus instance
0709 13:00:26.515566 (+ 51486us) ts_tablet_manager.cc:989] Replica registered.
0709 13:00:26.515705 (+   139us) inbound_call.cc:177] Queueing success response
0709 13:00:28.288061 (+1772356us) ts_tablet_manager.cc:1379] Bootstrapping tablet
0709 13:00:31.705007 (+3416946us) ts_tablet_manager.cc:1435] Starting tablet replica
0709 13:00:32.034705 (+329698us) tablet_replica.cc:261] Starting instrumentation
0709 13:00:32.035519 (+   814us) tablet_replica.cc:280] Starting consensus
Metrics: {"dns_us":61776,"mutex_wait_us":18614,"raft.queue_time_us":12213,"raft.run_cpu_time_us":115,"raft.run_wall_time_us":116,"tablet-copy.queue_time_us":248185,"tablet-download-pool.queue_time_us":77437,"tablet-download-pool.run_cpu_time_us":1433,"tablet-download-pool.run_wall_time_us":736017,"thread_start_us":298341,"threads_started":2}
W20260709 13:00:32.769960  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 142922us
I20260709 13:00:32.792379  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:32.873263  1678 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52778 (request call id 1959) took 1560 ms. Trace:
I20260709 13:00:32.915410  1678 rpcz_store.cc:276] 0709 13:00:31.312963 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:31.314398 (+  1435us) service_pool.cc:224] Handling call
0709 13:00:31.337592 (+ 23194us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:32.715423 (+1377831us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:32.872915 (+157492us) spinlock_profiling.cc:243] Waited 1.38 s on lock 0x7b54000fe454. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:00:32.873177 (+   262us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3162823552}
I20260709 13:00:32.917071  1678 raft_consensus.cc:1240] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 5. Current term is 6. Ops: []
I20260709 13:00:32.917765  1678 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52766 (request call id 2392) took 1477 ms. Trace:
I20260709 13:00:32.901861  1675 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52778 (request call id 1967) took 1552 ms. Trace:
I20260709 13:00:32.918663  1675 rpcz_store.cc:276] 0709 13:00:31.349611 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:32.326619 (+977008us) service_pool.cc:224] Handling call
0709 13:00:32.350555 (+ 23936us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:32.350773 (+   218us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 13:00:32.651649 (+300876us) log.cc:844] Serialized 14 byte log entry
0709 13:00:32.901442 (+249793us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:00:32.901509 (+    67us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:00:32.901559 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:32.901642 (+    83us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:32.901788 (+   146us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":421158,"prepare.run_cpu_time_us":55,"prepare.run_wall_time_us":57,"raft.queue_time_us":103,"thread_start_us":540684,"threads_started":2,"wal-append.queue_time_us":218447}
I20260709 13:00:32.869374  1571 leader_election.cc:304] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 13:00:32.902778  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:32.920807  1753 raft_consensus.cc:3060] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:00:32.801359  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
W20260709 13:00:32.905551  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 216608us
I20260709 13:00:32.923627  1717 leader_election.cc:304] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:32.907267  1754 raft_consensus.cc:2393] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 9: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:00:32.843096  1604 raft_consensus.cc:2468] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 10.
I20260709 13:00:32.918150  1678 rpcz_store.cc:276] 0709 13:00:31.440641 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:32.916503 (+1475862us) service_pool.cc:224] Handling call
0709 13:00:32.916874 (+   371us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:32.917521 (+   647us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:32.917695 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:00:32.799592  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:32.720677  5172 raft_consensus.cc:2804] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260709 13:00:32.827005  5471 log.cc:927] Time spent T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.215s	user 0.000s	sys 0.001s
I20260709 13:00:32.760476  5300 raft_consensus.cc:2804] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260709 13:00:32.764093  5356 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.728s	user 0.007s	sys 0.011s
I20260709 13:00:32.786309  1598 raft_consensus.cc:2468] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 10.
W20260709 13:00:32.927886  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 58580.3us
I20260709 13:00:32.931876  1717 leader_election.cc:304] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
W20260709 13:00:32.856719  5183 consensus_meta.cc:287] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.758s	user 0.013s	sys 0.002s
I20260709 13:00:32.721071  5462 raft_consensus.cc:2804] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 13:00:32.768062  5385 consensus_queue.cc:1059] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.169s
I20260709 13:00:32.815640  5255 raft_consensus.cc:2804] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:00:32.809770  5224 leader_election.cc:290] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 10 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:32.946794  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:00:32.768206  5243 raft_consensus.cc:493] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:00:32.950868  1754 raft_consensus.cc:1240] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 8. Current term is 9. Ops: []
I20260709 13:00:32.769966  5389 raft_consensus.cc:3055] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:00:32.959807  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:00:32.960307  1717 leader_election.cc:304] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:32.800444  5428 leader_election.cc:290] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 9 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:32.962227  1717 leader_election.cc:304] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:32.964104  1717 leader_election.cc:304] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
W20260709 13:00:32.944923  5356 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.909s	user 0.009s	sys 0.016s
I20260709 13:00:32.926290  5172 raft_consensus.cc:493] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:00:32.956583  1675 raft_consensus.cc:1240] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 7. Current term is 8. Ops: [7.9-7.9]
I20260709 13:00:33.002560  1675 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52766 (request call id 2393) took 1561 ms. Trace:
I20260709 13:00:32.866496  5474 raft_consensus.cc:515] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:32.758328  5403 consensus_queue.cc:1048] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.181s
I20260709 13:00:32.955632  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
W20260709 13:00:32.950042  5443 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.923s	user 0.015s	sys 0.009s
I20260709 13:00:32.911234  5346 raft_consensus.cc:2749] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:00:32.788941  5393 consensus_queue.cc:1059] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.166s
W20260709 13:00:32.968673  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 227233us
I20260709 13:00:33.010332  5172 raft_consensus.cc:3060] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:00:33.003050  1675 rpcz_store.cc:276] 0709 13:00:31.440974 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:32.920345 (+1479371us) service_pool.cc:224] Handling call
0709 13:00:32.956196 (+ 35851us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:00:33.002110 (+ 45914us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:33.002484 (+   374us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:00:32.906579  5182 raft_consensus.cc:697] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 10 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:00:32.944475  5300 raft_consensus.cc:493] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:00:33.083884  5300 raft_consensus.cc:3060] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Advancing to term 7
W20260709 13:00:33.012215  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 73110.7us
I20260709 13:00:32.959182  5490 raft_consensus.cc:2749] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:00:33.014034  5369 leader_election.cc:290] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:00:33.014575  5485 log.cc:927] Time spent T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.160s	user 0.000s	sys 0.000s
I20260709 13:00:32.966734  5336 consensus_queue.cc:1059] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.083s
W20260709 13:00:33.016598  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 52618.4us
I20260709 13:00:32.773159  5380 raft_consensus.cc:515] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:32.938844  5388 raft_consensus.cc:2804] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:00:33.036813  1753 raft_consensus.cc:2468] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 8.
I20260709 13:00:32.955935  5255 raft_consensus.cc:493] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:33.127301  5255 raft_consensus.cc:3060] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:00:32.957993  5183 raft_consensus.cc:515] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:32.913645  5233 raft_consensus.cc:515] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:32.970773  5484 raft_consensus.cc:2804] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:00:33.130797  5484 raft_consensus.cc:493] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:00:33.131358  5484 raft_consensus.cc:3060] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:00:33.131793  5425 leader_election.cc:290] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:33.132447  5233 leader_election.cc:290] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:33.133088  5490 raft_consensus.cc:2749] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:00:33.133865  5490 raft_consensus.cc:2804] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 13:00:33.134330  5490 raft_consensus.cc:697] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 8 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:00:33.126117  5388 raft_consensus.cc:493] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:33.135138  5388 raft_consensus.cc:3060] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:00:33.135298  5490 consensus_queue.cc:237] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:33.136451  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:32.959407  5389 raft_consensus.cc:740] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:00:32.862469  5482 raft_consensus.cc:2804] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:00:33.002081  5428 raft_consensus.cc:493] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:00:33.034668  5443 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.012s	user 0.016s	sys 0.010s
W20260709 13:00:33.080668  1571 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:33.171123  1673 raft_consensus.cc:2393] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 7: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:00:33.016426  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:32.952008  5405 consensus_queue.cc:1048] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.099s
I20260709 13:00:32.969534  5243 raft_consensus.cc:515] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:33.120500  1717 leader_election.cc:304] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:00:33.122083  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:33.124678  1675 raft_consensus.cc:1240] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 6. Current term is 7. Ops: [5.7-6.8]
I20260709 13:00:33.141649  5183 leader_election.cc:290] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 9 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:33.165551  1679 raft_consensus.cc:1275] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 13:00:33.177557  1675 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52766 (request call id 2395) took 1736 ms. Trace:
I20260709 13:00:33.177945  1675 rpcz_store.cc:276] 0709 13:00:31.446112 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:33.081822 (+1635710us) service_pool.cc:224] Handling call
0709 13:00:33.124286 (+ 42464us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:00:33.177207 (+ 52921us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:33.177487 (+   280us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:00:33.132540  5489 log.cc:927] Time spent T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.176s	user 0.000s	sys 0.000s
I20260709 13:00:33.181918  1675 raft_consensus.cc:1275] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260709 13:00:33.173316  5428 raft_consensus.cc:515] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:33.170630  5385 consensus_queue.cc:1243] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer af520b67945d443b901bbeca37cc5701 log is divergent from this leader: its last log entry 6.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20260709 13:00:33.166827  5482 raft_consensus.cc:493] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:00:33.183648  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 63671.7us
I20260709 13:00:33.199563  1717 leader_election.cc:304] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 13:00:33.098985  5152 raft_consensus.cc:515] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:33.209509  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:33.184545  5390 consensus_queue.cc:1059] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.032s
I20260709 13:00:33.216437  5496 raft_consensus.cc:2749] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260709 13:00:33.217263  1643 leader_election.cc:336] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 2.052s (SENT)
I20260709 13:00:33.217823  1643 leader_election.cc:304] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:00:33.204222  5389 consensus_queue.cc:260] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:33.220037  5389 raft_consensus.cc:3060] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:00:33.221302  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:33.219121  5336 raft_consensus.cc:2749] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 13:00:33.223466  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:33.224689  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:33.225886  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:33.228248  5380 leader_election.cc:290] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 7 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:00:33.230845  5492 log.cc:927] Time spent T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.063s	user 0.000s	sys 0.001s
I20260709 13:00:33.236099  5086 leader_election.cc:290] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 9 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:33.236462  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:33.237262  5417 leader_election.cc:290] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 10 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:33.238574  5243 leader_election.cc:290] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:33.207679  5182 consensus_queue.cc:237] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:33.240795  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:33.241508  1601 raft_consensus.cc:2376] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 10. Re-sending same reply.
I20260709 13:00:33.241977  1604 raft_consensus.cc:2393] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 10: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:00:33.245548  5460 leader_election.cc:290] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:33.247622  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:33.248306  1675 raft_consensus.cc:3060] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:00:33.249083  1597 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40154 (request call id 2019) took 1503 ms. Trace:
I20260709 13:00:33.253180  5474 leader_election.cc:290] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:33.253765  5428 leader_election.cc:290] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:33.254551  5362 leader_election.cc:290] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:33.255631  1718 leader_election.cc:304] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:00:33.249466  1597 rpcz_store.cc:276] 0709 13:00:31.745300 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:31.745432 (+   132us) service_pool.cc:224] Handling call
0709 13:00:31.745782 (+   350us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:00:32.764667 (+1018885us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:00:32.764715 (+    48us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:00:32.765346 (+   631us) log.cc:844] Serialized 26 byte log entry
0709 13:00:32.927914 (+162568us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:00:33.103744 (+175830us) log.cc:844] Serialized 14 byte log entry
0709 13:00:33.120135 (+ 16391us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:33.248642 (+128507us) spinlock_profiling.cc:243] Waited 1.02 s on lock 0x7b540006dbd4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:00:33.248695 (+    53us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:00:33.248759 (+    64us) raft_consensus.cc:1685] finished
0709 13:00:33.248859 (+   100us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:33.249014 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":321970,"raft.queue_time_us":2462,"raft.run_cpu_time_us":273,"raft.run_wall_time_us":3161,"spinlock_wait_cycles":2341799296,"thread_start_us":316240,"threads_started":2,"wal-append.queue_time_us":271148}
W20260709 13:00:33.259068  5495 log.cc:927] Time spent T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.173s	user 0.000s	sys 0.002s
W20260709 13:00:33.260347  5452 consensus_meta.cc:287] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 1.013s	user 0.010s	sys 0.005s
I20260709 13:00:33.260496  1677 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52778 (request call id 1971) took 1857 ms. Trace:
I20260709 13:00:33.260869  5452 raft_consensus.cc:515] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:33.261684  5497 log.cc:927] Time spent T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.170s	user 0.000s	sys 0.000s
W20260709 13:00:33.263134  5487 log.cc:927] Time spent T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.289s	user 0.001s	sys 0.000s
I20260709 13:00:33.204860  5482 raft_consensus.cc:3060] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:00:33.268606  5152 leader_election.cc:290] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 9 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:33.210521  1602 raft_consensus.cc:2468] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 6.
I20260709 13:00:33.260994  1677 rpcz_store.cc:276] 0709 13:00:31.403560 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:32.681549 (+1277989us) service_pool.cc:224] Handling call
0709 13:00:32.725451 (+ 43902us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:00:32.725686 (+   235us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:00:32.725719 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:00:32.726321 (+   602us) log.cc:844] Serialized 26 byte log entry
0709 13:00:32.880708 (+154387us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:00:32.880763 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:32.880878 (+   115us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:00:33.258232 (+377354us) spinlock_profiling.cc:243] Waited 31.6 ms on lock 0x7b4400022b4c. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f4262f9bc85 00007f4262f9c6a2 00007f4265d0e1db 00007f4265d09aaf 00007f4265cfd86e 00007f4265cfcbb6 00007f4265cfc7d8 00007f4265d06221 00007f4265d061d9 00007f4265d06169
0709 13:00:33.260111 (+  1879us) raft_consensus.cc:1685] finished
0709 13:00:33.260134 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:33.260419 (+   285us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":72643840,"thread_start_us":67758,"threads_started":1,"wal-append.queue_time_us":180099}
I20260709 13:00:33.237223  1598 raft_consensus.cc:2468] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 8.
I20260709 13:00:33.277369  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:33.290781  1672 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57092 (request call id 787) took 1852 ms. Trace:
I20260709 13:00:33.291530  1672 rpcz_store.cc:276] 0709 13:00:31.438566 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:32.684738 (+1246172us) service_pool.cc:224] Handling call
0709 13:00:32.685058 (+   320us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:00:32.720013 (+ 34955us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:00:32.720045 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:00:32.771725 (+ 51680us) log.cc:844] Serialized 26 byte log entry
0709 13:00:32.856460 (+ 84735us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:00:32.856506 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:32.856596 (+    90us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:00:33.290344 (+433748us) raft_consensus.cc:1685] finished
0709 13:00:33.290377 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:33.290675 (+   298us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":20885,"raft.queue_time_us":40190,"raft.run_cpu_time_us":169,"raft.run_wall_time_us":21484,"thread_start_us":44889,"threads_started":1,"wal-append.queue_time_us":161452}
I20260709 13:00:33.299345  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:33.303848  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:33.304759  2331 raft_consensus.cc:2468] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 6.
I20260709 13:00:33.306592  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:33.309453  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:33.310333  2331 raft_consensus.cc:2468] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 8.
I20260709 13:00:33.309368  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:33.311480  1602 raft_consensus.cc:3060] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Advancing to term 9
W20260709 13:00:33.314904  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 160629us
I20260709 13:00:33.315371  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:33.322427  1602 raft_consensus.cc:2468] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 9.
I20260709 13:00:33.325822  1675 raft_consensus.cc:2468] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 9.
I20260709 13:00:33.222028  5484 raft_consensus.cc:515] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:33.328855  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:33.332958  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:33.334925  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:33.336237  1602 raft_consensus.cc:2410] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for term 11 because replica has last-logged OpId of term: 10 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 8.
I20260709 13:00:33.305685  5385 raft_consensus.cc:493] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:00:33.341943  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:33.353314  2330 raft_consensus.cc:2393] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 9: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:00:33.315747  5428 raft_consensus.cc:3055] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:00:33.343545  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:33.347157  1643 leader_election.cc:304] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:33.360582  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:33.361546  1602 raft_consensus.cc:2468] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 8.
I20260709 13:00:33.362715  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:33.363776  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:33.365763  2330 raft_consensus.cc:2468] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 5.
I20260709 13:00:33.372376  2330 raft_consensus.cc:1275] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 12 index: 10. (index mismatch)
I20260709 13:00:33.355777  5428 raft_consensus.cc:740] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:00:33.358610  5405 raft_consensus.cc:493] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:00:33.380723  2330 raft_consensus.cc:1240] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 7 LEADER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 6. Current term is 7. Ops: [6.8-6.8]
I20260709 13:00:33.383225  1643 leader_election.cc:304] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:00:33.379211  5362 raft_consensus.cc:2749] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:00:33.393788  5319 raft_consensus.cc:493] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:00:33.400313  5319 raft_consensus.cc:515] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:33.402173  5503 raft_consensus.cc:2804] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:00:33.402864  5503 raft_consensus.cc:493] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:00:33.403261  5503 raft_consensus.cc:3060] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:00:33.403491  5431 raft_consensus.cc:3055] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:00:33.403983  5431 raft_consensus.cc:740] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:00:33.404881  5319 leader_election.cc:290] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:33.405211  5431 consensus_queue.cc:260] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:00:33.406634  1570 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:33.395109  5460 raft_consensus.cc:2804] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:00:33.407250  5460 raft_consensus.cc:493] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Starting leader election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:00:33.407737  5460 raft_consensus.cc:3060] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Advancing to term 9
W20260709 13:00:33.410899  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 155343us
I20260709 13:00:33.412672  5086 raft_consensus.cc:2804] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:00:33.413198  5086 raft_consensus.cc:493] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:33.413897  5086 raft_consensus.cc:3060] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:00:33.395605  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:33.413237  5483 raft_consensus.cc:2804] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:00:33.420099  1646 leader_election.cc:304] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:33.436573  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:33.426148  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:33.424659  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:33.429073  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:00:33.424371  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 224924us
I20260709 13:00:33.438776  1745 raft_consensus.cc:2393] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 9: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:00:33.442775  1675 raft_consensus.cc:3060] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:00:33.439044  1717 leader_election.cc:304] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:00:33.476403  1747 pending_rounds.cc:85] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701: Aborting all ops after (but not including) 7
I20260709 13:00:33.484409  1747 pending_rounds.cc:107] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701: Aborting uncommitted NO_OP operation due to leader change: 5.8
I20260709 13:00:33.484882  1747 raft_consensus.cc:2889] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:00:33.452343  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:33.484818  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:00:33.487005  2330 raft_consensus.cc:2393] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 9: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:00:33.452121  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:33.451534  1750 raft_consensus.cc:1240] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 7. Current term is 8. Ops: [6.5-7.6]
I20260709 13:00:33.358193  5385 raft_consensus.cc:515] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:33.479363  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
W20260709 13:00:33.491155  5434 consensus_meta.cc:287] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.931s	user 0.013s	sys 0.002s
I20260709 13:00:33.491667  5434 raft_consensus.cc:515] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:33.494159  5434 leader_election.cc:290] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 9 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:33.435693  5483 raft_consensus.cc:493] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:33.439429  5431 raft_consensus.cc:3060] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:00:33.453693  1675 raft_consensus.cc:2468] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 9.
I20260709 13:00:33.446085  1717 leader_election.cc:304] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 13:00:33.398126  5484 leader_election.cc:290] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 8 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 13:00:33.498665  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 73049.5us
W20260709 13:00:33.429656  5454 consensus_meta.cc:287] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.828s	user 0.015s	sys 0.000s
I20260709 13:00:33.494655  1751 raft_consensus.cc:1240] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 8. Current term is 9. Ops: []
I20260709 13:00:33.445482  5346 raft_consensus.cc:2804] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Leader election won for term 8
I20260709 13:00:33.447937  5243 raft_consensus.cc:493] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:00:33.534413  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 88441.6us
I20260709 13:00:33.488292  5152 raft_consensus.cc:2804] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:00:33.505779  5483 raft_consensus.cc:3060] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:00:33.453394  5086 raft_consensus.cc:515] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:33.541042  1717 leader_election.cc:304] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:00:33.505560  5452 leader_election.cc:290] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 11 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:33.539412  1718 leader_election.cc:304] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:00:33.422966  5405 raft_consensus.cc:515] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:33.533442  5354 leader_election.cc:290] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:33.544344  5346 raft_consensus.cc:697] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
W20260709 13:00:33.548894  5509 log.cc:927] Time spent T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.171s	user 0.001s	sys 0.000s
I20260709 13:00:33.536046  5403 raft_consensus.cc:2804] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:00:33.478834  5388 raft_consensus.cc:515] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:33.542985  5152 raft_consensus.cc:493] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:33.476339  5428 consensus_queue.cc:260] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:33.563149  5403 raft_consensus.cc:493] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:33.554643  5460 raft_consensus.cc:515] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:33.580060  5460 leader_election.cc:290] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 9 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:33.580617  5482 raft_consensus.cc:515] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:33.585090  5482 leader_election.cc:290] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 10 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:33.586360  5503 raft_consensus.cc:515] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:33.590698  5503 leader_election.cc:290] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 7 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 13:00:33.630810  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 89492.5us
I20260709 13:00:33.632179  1671 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57092 (request call id 789) took 2183 ms. Trace:
W20260709 13:00:33.635402  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 95859.2us
I20260709 13:00:33.636863  5086 leader_election.cc:290] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 10 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:33.638765  5086 raft_consensus.cc:2804] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:00:33.644351  5405 leader_election.cc:290] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:33.641865  5503 raft_consensus.cc:2804] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:00:33.664431  5405 raft_consensus.cc:493] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:33.575817  5403 raft_consensus.cc:3060] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:00:33.666450  1602 raft_consensus.cc:2468] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 5.
I20260709 13:00:33.655943  5503 raft_consensus.cc:493] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:33.645984  5086 raft_consensus.cc:493] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:33.567463  5152 raft_consensus.cc:3060] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:00:33.674968  4872 raft_consensus.cc:493] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:00:33.676050  4872 raft_consensus.cc:515] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:33.674069  5389 raft_consensus.cc:493] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:00:33.679466  5389 raft_consensus.cc:515] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:33.550982  5243 raft_consensus.cc:515] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:33.689705  5388 leader_election.cc:290] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 9 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:33.691812  1603 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40154 (request call id 2014) took 1948 ms. Trace:
W20260709 13:00:33.692662  5255 consensus_meta.cc:287] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.564s	user 0.015s	sys 0.000s
I20260709 13:00:33.690284  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:33.693419  5255 raft_consensus.cc:515] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:33.694904  1604 raft_consensus.cc:2410] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for term 7 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260709 13:00:33.692533  1603 rpcz_store.cc:276] 0709 13:00:31.737440 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:31.746100 (+  8660us) service_pool.cc:224] Handling call
0709 13:00:31.843524 (+ 97424us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:31.924587 (+ 81063us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:33.685420 (+1760833us) spinlock_profiling.cc:243] Waited 80.3 ms on lock 0x7b540028bc54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:00:33.686048 (+   628us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":184709888}
W20260709 13:00:33.697353  1746 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43156 (request call id 659) took 2238 ms (client timeout 2122 ms). Trace:
I20260709 13:00:33.665832  5405 raft_consensus.cc:515] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:33.697983  1746 rpcz_store.cc:269] 0709 13:00:31.458217 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:31.463049 (+  4832us) service_pool.cc:224] Handling call
0709 13:00:33.696752 (+2233703us) spinlock_profiling.cc:243] Waited 1.02 s on lock 0x7b5400019fd4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:00:33.697018 (+   266us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2348662016}
I20260709 13:00:33.666121  5385 leader_election.cc:290] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:33.699685  5086 raft_consensus.cc:3060] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:00:33.670077  5503 raft_consensus.cc:3060] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:00:33.698625  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:00:33.708146  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 69748.2us
I20260709 13:00:33.633155  1671 rpcz_store.cc:276] 0709 13:00:31.448608 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:33.137453 (+1688845us) service_pool.cc:224] Handling call
0709 13:00:33.137841 (+   388us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:00:33.138265 (+   424us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:00:33.138299 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:00:33.168642 (+ 30343us) log.cc:844] Serialized 26 byte log entry
0709 13:00:33.352468 (+183826us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:00:33.374133 (+ 21665us) log.cc:844] Serialized 14 byte log entry
0709 13:00:33.377186 (+  3053us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:33.377325 (+   139us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:00:33.630904 (+253579us) raft_consensus.cc:1685] finished
0709 13:00:33.631014 (+   110us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:33.631742 (+   728us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6370,"prepare.queue_time_us":19447,"prepare.run_cpu_time_us":85,"prepare.run_wall_time_us":2602,"raft.queue_time_us":6997,"raft.run_cpu_time_us":101,"raft.run_wall_time_us":2564,"spinlock_wait_cycles":101411584,"thread_start_us":147520,"threads_started":1,"wal-append.queue_time_us":187564}
I20260709 13:00:33.707175  1604 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40166 (request call id 2126) took 1156 ms. Trace:
I20260709 13:00:33.708263  1603 raft_consensus.cc:2468] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 9.
I20260709 13:00:33.714318  5427 raft_consensus.cc:493] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:33.715267  1754 raft_consensus.cc:1240] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 5. Current term is 6. Ops: []
I20260709 13:00:33.715497  5427 raft_consensus.cc:515] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:33.716686  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:00:33.719807  5427 leader_election.cc:290] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:33.720042  1750 raft_consensus.cc:1240] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 8. Current term is 10. Ops: []
I20260709 13:00:33.721915  4951 leader_election.cc:290] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 9 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:33.714941  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:33.726619  4872 leader_election.cc:290] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:33.727240  5243 leader_election.cc:290] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:33.709558  1604 rpcz_store.cc:276] 0709 13:00:32.546214 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:33.257718 (+711504us) service_pool.cc:224] Handling call
0709 13:00:33.689494 (+431776us) spinlock_profiling.cc:243] Waited 37.2 ms on lock 0x7b5800003698. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f426303db71 00007f426303da22 00007f426bf56c09 00007f426b0b0ee0 00007f426b17a0ab 00007f426b17a131 00007f426b1280a0 00007f426b127984 00007f42646a26d4 00007f42646a2662
0709 13:00:33.702933 (+ 13439us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":85593472}
W20260709 13:00:33.737063  5172 consensus_meta.cc:287] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.640s	user 0.016s	sys 0.001s
I20260709 13:00:33.737982  5172 raft_consensus.cc:515] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:33.742141  5405 leader_election.cc:290] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:33.746675  5389 leader_election.cc:290] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:33.750372  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:33.779206  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:33.780030  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:33.780651  1754 raft_consensus.cc:2393] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 10: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:00:33.782488  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:33.783960  1746 raft_consensus.cc:2468] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 7.
I20260709 13:00:33.792569  1748 raft_consensus.cc:1240] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 6. Current term is 7. Ops: []
I20260709 13:00:33.759483  5172 leader_election.cc:290] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 9 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:33.798900  5354 consensus_queue.cc:1059] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 13:00:33.800132  5336 consensus_queue.cc:1059] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
W20260709 13:00:33.801236  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 167839us
W20260709 13:00:33.805217  1644 connection.cc:256] RPC call timeout handler was delayed by -1.1357541859999856s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2214s for a response
I20260709 13:00:33.811163  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:33.787760  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
W20260709 13:00:33.819326  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 51061.1us
I20260709 13:00:33.826805  5482 consensus_queue.cc:1048] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 13:00:33.842885  5346 consensus_queue.cc:237] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:33.852232  5502 raft_consensus.cc:493] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:33.853658  5502 raft_consensus.cc:515] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:33.865039  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:33.869179  1751 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55848 (request call id 2187) took 1078 ms. Trace:
I20260709 13:00:33.869827  1751 rpcz_store.cc:276] 0709 13:00:32.847154 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:32.847345 (+   191us) service_pool.cc:224] Handling call
0709 13:00:32.850417 (+  3072us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:33.518458 (+668041us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:33.868504 (+350046us) spinlock_profiling.cc:243] Waited 644 ms on lock 0x7b54000e4a54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:00:33.868909 (+   405us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1480854784}
I20260709 13:00:33.872918  1680 raft_consensus.cc:1240] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 6. Current term is 7. Ops: []
I20260709 13:00:33.869297  1603 raft_consensus.cc:2393] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 9: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:00:33.876770  5475 raft_consensus.cc:493] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:00:33.877907  5475 raft_consensus.cc:515] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:33.880602  5475 leader_election.cc:290] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:00:33.876778  1673 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57092 (request call id 784) took 2479 ms (client timeout 2116 ms). Trace:
W20260709 13:00:33.882879  1673 rpcz_store.cc:269] 0709 13:00:31.396897 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:32.016343 (+619446us) service_pool.cc:224] Handling call
0709 13:00:33.871627 (+1855284us) spinlock_profiling.cc:243] Waited 1.12 s on lock 0x7b54000d7854. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:00:33.876462 (+  4835us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2579461504}
W20260709 13:00:33.883798  2329 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39726 (request call id 943) took 2652 ms (client timeout 2214 ms). Trace:
W20260709 13:00:33.884539  2329 rpcz_store.cc:269] 0709 13:00:31.231422 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:31.232068 (+   646us) service_pool.cc:224] Handling call
0709 13:00:33.883143 (+2651075us) spinlock_profiling.cc:243] Waited 1.27 s on lock 0x7b540083f754. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:00:33.883481 (+   338us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2912779136}
W20260709 13:00:33.887038  2333 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49082 (request call id 798) took 1370 ms (client timeout 1824 ms). Trace:
I20260709 13:00:33.887706  5502 leader_election.cc:290] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:33.888605  5502 raft_consensus.cc:493] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:00:33.887724  2333 rpcz_store.cc:269] 0709 13:00:32.515929 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:32.516089 (+   160us) service_pool.cc:224] Handling call
0709 13:00:33.886405 (+1370316us) spinlock_profiling.cc:243] Waited 2.85 ms on lock 0x7b540073afd4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:00:33.886709 (+   304us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":6543616}
I20260709 13:00:33.889449  5502 raft_consensus.cc:515] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:33.893121  5255 leader_election.cc:290] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 7 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:33.918483  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:33.921353  2335 raft_consensus.cc:1217] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Deduplicated request from leader. Original: 4.5->[6.6-6.6]   Dedup: 6.6->[]
I20260709 13:00:33.922158  2335 raft_consensus.cc:1240] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 6. Current term is 7. Ops: [6.6-6.6]
I20260709 13:00:33.931640  1569 leader_election.cc:304] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:00:33.961694  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:33.936812  1764 tablet_copy_service.cc:342] P af520b67945d443b901bbeca37cc5701: Request end of tablet copy session 3b879799899a499c8d409ee9f1277cd1-983621a6fc5b4c9ea0159a648dfe7a36 received from {username='slave'} at 127.0.0.1:43156
W20260709 13:00:33.939085  5520 log.cc:927] Time spent T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.130s	user 0.000s	sys 0.002s
I20260709 13:00:33.942762  1643 leader_election.cc:304] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:00:33.949680  1646 leader_election.cc:304] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
W20260709 13:00:33.951098  5462 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.808s	user 0.002s	sys 0.006s
I20260709 13:00:33.954021  1680 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52766 (request call id 2391) took 2513 ms. Trace:
I20260709 13:00:33.955536  1605 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40154 (request call id 2029) took 2041 ms. Trace:
W20260709 13:00:33.957012  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 87076.1us
I20260709 13:00:33.957257  1750 raft_consensus.cc:2393] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 9: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:00:33.985489  1605 rpcz_store.cc:276] 0709 13:00:31.914211 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:31.939931 (+ 25720us) service_pool.cc:224] Handling call
0709 13:00:31.979370 (+ 39439us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:32.010959 (+ 31589us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:00:32.011002 (+    43us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:00:32.011056 (+    54us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:00:32.011097 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:33.954925 (+1943828us) spinlock_profiling.cc:243] Waited 31.2 ms on lock 0x7b540029aed4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:00:33.955022 (+    97us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:33.955300 (+   278us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":2387,"raft.run_cpu_time_us":118,"raft.run_wall_time_us":118,"spinlock_wait_cycles":71650176}
I20260709 13:00:33.957893  1604 raft_consensus.cc:2468] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 6.
I20260709 13:00:33.935637  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:33.963655  1676 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57092 (request call id 785) took 2566 ms. Trace:
I20260709 13:00:33.965022  1599 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40154 (request call id 2054) took 1705 ms. Trace:
I20260709 13:00:33.964228  5502 leader_election.cc:290] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:33.964910  5475 raft_consensus.cc:2749] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 13:00:33.980829  1597 raft_consensus.cc:2468] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 8.
I20260709 13:00:33.982195  1764 tablet_copy_service.cc:434] P af520b67945d443b901bbeca37cc5701: ending tablet copy session 3b879799899a499c8d409ee9f1277cd1-983621a6fc5b4c9ea0159a648dfe7a36 on tablet 983621a6fc5b4c9ea0159a648dfe7a36 with peer 3b879799899a499c8d409ee9f1277cd1
I20260709 13:00:33.998579  1597 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36462 (request call id 714) took 1322 ms. Trace:
I20260709 13:00:33.984983  1680 rpcz_store.cc:276] 0709 13:00:31.440340 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:32.914691 (+1474351us) service_pool.cc:224] Handling call
0709 13:00:33.872062 (+957371us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:33.876142 (+  4080us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:33.953641 (+ 77499us) spinlock_profiling.cc:243] Waited 927 ms on lock 0x7b54000d7850. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661618ad 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:00:33.953876 (+   235us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2132937600}
I20260709 13:00:33.988531  1599 rpcz_store.cc:276] 0709 13:00:32.259728 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:32.343676 (+ 83948us) service_pool.cc:224] Handling call
0709 13:00:33.964419 (+1620743us) spinlock_profiling.cc:243] Waited 171 ms on lock 0x7b54000f0d54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:00:33.964703 (+   284us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":393083648}
I20260709 13:00:33.997977  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:33.999089  1597 rpcz_store.cc:276] 0709 13:00:32.676680 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:33.256645 (+579965us) service_pool.cc:224] Handling call
0709 13:00:33.864409 (+607764us) spinlock_profiling.cc:243] Waited 21.4 ms on lock 0x7b5800003698. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f426303db71 00007f426303da22 00007f426bf56c09 00007f426b0b0ee0 00007f426b17a0ab 00007f426b17a131 00007f426b1280a0 00007f426b127984 00007f42646a26d4 00007f42646a2662
0709 13:00:33.998497 (+134088us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":49123712}
W20260709 13:00:34.001076  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 89708.2us
I20260709 13:00:33.988402  1676 rpcz_store.cc:276] 0709 13:00:31.397269 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:32.483852 (+1086583us) service_pool.cc:224] Handling call
0709 13:00:32.485215 (+  1363us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:32.699387 (+214172us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:33.962907 (+1263520us) spinlock_profiling.cc:243] Waited 166 ms on lock 0x7b5400090954. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:00:33.963370 (+   463us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":381367168}
I20260709 13:00:33.968426  5152 raft_consensus.cc:515] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:33.987699  5462 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.938s	user 0.003s	sys 0.007s
I20260709 13:00:34.003166  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:34.003249  5466 raft_consensus.cc:2804] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260709 13:00:34.009775  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 85060.8us
I20260709 13:00:34.015173  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:34.017396  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:34.018898  5359 leader_election.cc:290] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 6 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:34.019202  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:34.020740  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:34.022027  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:00:34.022821  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 80496.1us
I20260709 13:00:34.025097  5434 consensus_queue.cc:1048] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 13:00:34.024890  1500 catalog_manager.cc:5697] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 reported cstate change: term changed from 4 to 5, leader changed from 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) to af520b67945d443b901bbeca37cc5701 (127.0.107.195). New cstate: current_term: 5 leader_uuid: "af520b67945d443b901bbeca37cc5701" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 13:00:34.027245  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:34.031157  1751 raft_consensus.cc:3060] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:00:34.032055  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:34.034307  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:34.035116  1603 raft_consensus.cc:2376] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 10. Re-sending same reply.
I20260709 13:00:34.035629  1603 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36462 (request call id 702) took 1413 ms. Trace:
I20260709 13:00:34.036008  1603 rpcz_store.cc:276] 0709 13:00:32.621699 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:34.033664 (+1411965us) service_pool.cc:224] Handling call
0709 13:00:34.035560 (+  1896us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:00:34.036401  5462 raft_consensus.cc:697] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 9 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:00:34.038228  5462 consensus_queue.cc:237] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:34.004712  1718 leader_election.cc:304] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:00:34.043332  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:34.044332  1596 raft_consensus.cc:3060] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:00:34.010955  5466 raft_consensus.cc:493] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:34.051478  1602 raft_consensus.cc:2393] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 9: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:00:33.982678  5392 consensus_queue.cc:1059] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.069s
I20260709 13:00:34.067659  1717 leader_election.cc:304] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:34.076361  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:00:34.075542  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 72065.3us
I20260709 13:00:34.077791  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:34.081902  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:34.083612  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:00:34.084455  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 43567.9us
I20260709 13:00:34.085215  1718 leader_election.cc:304] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:00:34.067346  5466 raft_consensus.cc:3060] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:00:34.093140  1603 raft_consensus.cc:2468] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 8.
W20260709 13:00:34.088406  5490 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.717s	user 0.007s	sys 0.007s
I20260709 13:00:34.102684  5514 raft_consensus.cc:2804] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 13:00:34.103402  1718 leader_election.cc:304] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
W20260709 13:00:34.104377  1571 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:34.104476  5458 raft_consensus.cc:3055] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:00:34.105019  5458 raft_consensus.cc:740] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:00:34.105191  1753 raft_consensus.cc:1240] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 8. Current term is 9. Ops: [8.9-8.9]
I20260709 13:00:34.103605  5514 raft_consensus.cc:697] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 9 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:00:34.107255  5431 raft_consensus.cc:2804] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:00:34.107774  5431 raft_consensus.cc:493] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Starting leader election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:00:34.107378  5514 consensus_queue.cc:237] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:34.108239  5431 raft_consensus.cc:3060] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:00:34.104561  5079 raft_consensus.cc:2804] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 13:00:34.110778  5079 raft_consensus.cc:697] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 9 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:00:34.111218  4896 raft_consensus.cc:493] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:34.111518  1604 raft_consensus.cc:2468] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 8.
I20260709 13:00:34.111946  4896 raft_consensus.cc:515] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:34.112170  5079 consensus_queue.cc:237] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:34.112210  1604 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40166 (request call id 2134) took 1561 ms. Trace:
I20260709 13:00:34.115281  1604 rpcz_store.cc:276] 0709 13:00:32.550707 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:34.008257 (+1457550us) service_pool.cc:224] Handling call
0709 13:00:34.112135 (+103878us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:00:34.115970  5483 consensus_meta.cc:287] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.554s	user 0.012s	sys 0.008s
I20260709 13:00:34.116681  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:34.116572  5483 raft_consensus.cc:515] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:34.117590  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:34.117954  1745 raft_consensus.cc:2393] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 7: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:00:34.117385  1603 raft_consensus.cc:2376] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 9. Re-sending same reply.
I20260709 13:00:34.127050  1603 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36462 (request call id 704) took 1501 ms. Trace:
I20260709 13:00:34.127761  1603 rpcz_store.cc:276] 0709 13:00:32.626930 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:34.116265 (+1489335us) service_pool.cc:224] Handling call
0709 13:00:34.123270 (+  7005us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:00:34.121711  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:34.130064  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:00:34.134975  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 121754us
I20260709 13:00:34.135020  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:34.118410  1605 raft_consensus.cc:2468] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 9.
I20260709 13:00:34.139155  2329 raft_consensus.cc:1240] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 10. Current term is 11. Ops: []
I20260709 13:00:34.120488  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:34.145866  1752 raft_consensus.cc:2393] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 7: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:00:34.110191  5428 raft_consensus.cc:3060] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:00:34.148162  5403 raft_consensus.cc:515] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:34.150697  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 104711us
I20260709 13:00:34.151883  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:00:34.153316  5392 consensus_queue.cc:1243] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer 3b879799899a499c8d409ee9f1277cd1 log is divergent from this leader: its last log entry 7.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20260709 13:00:34.154376  5506 consensus_queue.cc:1048] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 13:00:34.155692  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:00:34.158879  5300 consensus_meta.cc:287] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 1.074s	user 0.022s	sys 0.005s
I20260709 13:00:34.159525  5300 raft_consensus.cc:515] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:34.166973  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:34.167770  1597 raft_consensus.cc:2393] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 9: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
W20260709 13:00:34.101763  5490 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.751s	user 0.008s	sys 0.008s
W20260709 13:00:34.167770  1569 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:34.173041  1603 raft_consensus.cc:1275] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260709 13:00:34.175033  5483 leader_election.cc:290] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 7 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:34.174027  1603 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 700) took 1552 ms. Trace:
I20260709 13:00:34.088145  5454 consensus_queue.cc:1059] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 8.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.057s
I20260709 13:00:34.119536  5224 raft_consensus.cc:3055] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:00:34.179131  1603 rpcz_store.cc:276] 0709 13:00:32.621160 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:34.171813 (+1550653us) service_pool.cc:224] Handling call
0709 13:00:34.172587 (+   774us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:34.173630 (+  1043us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:34.173916 (+   286us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:00:34.122815  1500 catalog_manager.cc:5697] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 reported cstate change: term changed from 8 to 9, leader changed from 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) to af520b67945d443b901bbeca37cc5701 (127.0.107.195). New cstate: current_term: 9 leader_uuid: "af520b67945d443b901bbeca37cc5701" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 13:00:34.191359  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:00:34.134680  5400 raft_consensus.cc:3055] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:00:34.194923  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:34.142791  5434 raft_consensus.cc:2749] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:00:34.176664  5255 raft_consensus.cc:2804] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:00:34.166165  5502 raft_consensus.cc:3055] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:00:34.199338  1751 raft_consensus.cc:2468] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 9.
I20260709 13:00:34.208500  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:34.213706  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 12 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:34.193275  5224 raft_consensus.cc:740] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:00:34.216519  1717 leader_election.cc:304] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:34.211172  1605 raft_consensus.cc:3060] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:00:34.218020  1597 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40154 (request call id 2068) took 1498 ms. Trace:
I20260709 13:00:34.218581  1597 rpcz_store.cc:276] 0709 13:00:32.719983 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:34.168419 (+1448436us) service_pool.cc:224] Handling call
0709 13:00:34.217919 (+ 49500us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:00:34.220508  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:34.196858  5400 raft_consensus.cc:740] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:00:34.207399  5255 raft_consensus.cc:493] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:34.106273  5458 consensus_queue.cc:260] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:00:34.230865  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 127664us
I20260709 13:00:34.230540  1603 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 719) took 1548 ms. Trace:
I20260709 13:00:34.232086  1603 rpcz_store.cc:276] 0709 13:00:32.678448 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:34.201951 (+1523503us) service_pool.cc:224] Handling call
0709 13:00:34.222485 (+ 20534us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:34.222859 (+   374us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:00:34.222893 (+    34us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:00:34.226694 (+  3801us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:00:34.226743 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:34.226835 (+    92us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:34.226972 (+   137us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":361}
I20260709 13:00:34.233093  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:34.234009  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:34.234335  5470 raft_consensus.cc:3055] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 LEADER]: Stepping down as leader of term 8
W20260709 13:00:34.234557  2334 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49082 (request call id 793) took 2111 ms (client timeout 1698 ms). Trace:
I20260709 13:00:34.234853  5470 raft_consensus.cc:740] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
W20260709 13:00:34.235164  2334 rpcz_store.cc:269] 0709 13:00:32.122468 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:32.122607 (+   139us) service_pool.cc:224] Handling call
0709 13:00:34.229281 (+2106674us) spinlock_profiling.cc:243] Waited 114 ms on lock 0x7b5400443754. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:00:34.234273 (+  4992us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":263179648}
I20260709 13:00:34.236176  5470 consensus_queue.cc:260] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:34.213419  4896 leader_election.cc:290] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:34.206722  5502 raft_consensus.cc:740] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:00:34.238135  5470 raft_consensus.cc:3060] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:00:34.240909  5502 consensus_queue.cc:260] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:34.243886  5224 consensus_queue.cc:260] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } }
I20260709 13:00:34.245129  5224 raft_consensus.cc:3060] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:00:34.245725  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:34.245906  5359 raft_consensus.cc:493] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:34.246742  5359 raft_consensus.cc:515] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:34.247246  1602 raft_consensus.cc:1240] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 8. Current term is 9. Ops: [8.8-8.8]
I20260709 13:00:34.247859  1680 raft_consensus.cc:1240] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 LEADER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 7. Current term is 8. Ops: [7.6-7.6]
I20260709 13:00:34.247984  1602 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 720) took 1569 ms. Trace:
I20260709 13:00:34.248432  1602 rpcz_store.cc:276] 0709 13:00:32.678755 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:34.212116 (+1533361us) service_pool.cc:224] Handling call
0709 13:00:34.246911 (+ 34795us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:00:34.247663 (+   752us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:34.247898 (+   235us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:00:34.234877  2330 raft_consensus.cc:2468] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 6.
I20260709 13:00:34.215610  5475 raft_consensus.cc:493] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:34.254909  1673 raft_consensus.cc:1240] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 LEADER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 10. Current term is 11. Ops: [10.7-10.7]
I20260709 13:00:34.227384  5255 raft_consensus.cc:3060] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:00:34.261521  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:34.262571  1671 raft_consensus.cc:2410] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for term 7 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
I20260709 13:00:34.258893  2331 raft_consensus.cc:1275] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260709 13:00:34.259054  1680 raft_consensus.cc:1240] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 5. Current term is 6. Ops: []
I20260709 13:00:34.268231  1569 leader_election.cc:304] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
W20260709 13:00:34.270164  1570 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:34.270375  5325 raft_consensus.cc:493] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:34.271098  5325 raft_consensus.cc:515] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:34.272086  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:34.272836  1748 raft_consensus.cc:2376] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 9. Re-sending same reply.
I20260709 13:00:34.273586  5325 leader_election.cc:290] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:34.255295  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:34.237601  5400 consensus_queue.cc:260] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:34.282749  5393 raft_consensus.cc:493] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:34.283469  5393 raft_consensus.cc:515] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:34.251572  5502 raft_consensus.cc:3060] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:00:34.286870  1601 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 703) took 1664 ms. Trace:
I20260709 13:00:34.287101  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:34.287436  5393 leader_election.cc:290] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:34.274197  5325 raft_consensus.cc:493] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:00:34.287361  1601 rpcz_store.cc:276] 0709 13:00:32.621948 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:34.183490 (+1561542us) service_pool.cc:224] Handling call
0709 13:00:34.239366 (+ 55876us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:34.278733 (+ 39367us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 13:00:34.278767 (+    34us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:00:34.286533 (+  7766us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:00:34.286593 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:34.286661 (+    68us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:34.286773 (+   112us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":9232,"raft.queue_time_us":9343}
I20260709 13:00:34.290709  1597 raft_consensus.cc:1275] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260709 13:00:34.303103  1597 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40154 (request call id 2064) took 1621 ms. Trace:
I20260709 13:00:34.291759  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:34.291862  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:00:34.292675  1746 raft_consensus.cc:1275] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 10 index: 8. (index mismatch)
I20260709 13:00:34.292817  1500 catalog_manager.cc:5697] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } }
I20260709 13:00:34.294210  1602 raft_consensus.cc:1275] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 12 index: 10. (term mismatch)
I20260709 13:00:34.295575  5403 leader_election.cc:290] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 9 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:00:34.296036  5086 consensus_meta.cc:287] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.568s	user 0.018s	sys 0.001s
I20260709 13:00:34.322248  1750 raft_consensus.cc:1240] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 7. Current term is 8. Ops: []
I20260709 13:00:34.322781  5086 raft_consensus.cc:515] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:34.323935  5300 leader_election.cc:290] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 7 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:34.324146  1754 raft_consensus.cc:1240] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 7. Current term is 8. Ops: []
I20260709 13:00:34.324937  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:34.325697  1671 raft_consensus.cc:1275] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 11 index: 10. (term mismatch)
I20260709 13:00:34.326078  1671 pending_rounds.cc:85] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d: Aborting all ops after (but not including) 9
I20260709 13:00:34.326171  1750 raft_consensus.cc:2393] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 10: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:00:34.326445  1671 pending_rounds.cc:107] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d: Aborting uncommitted NO_OP operation due to leader change: 10.10
I20260709 13:00:34.326831  1671 raft_consensus.cc:2889] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:00:34.328840  1750 raft_consensus.cc:1240] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 5. Current term is 6. Ops: []
I20260709 13:00:34.330150  1752 raft_consensus.cc:1275] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20260709 13:00:34.289872  5359 leader_election.cc:290] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:34.279301  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 9 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:34.290831  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:34.335841  1751 raft_consensus.cc:3060] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:00:34.307548  1597 rpcz_store.cc:276] 0709 13:00:32.681257 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:34.219521 (+1538264us) service_pool.cc:224] Handling call
0709 13:00:34.269321 (+ 49800us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:34.302701 (+ 33380us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:34.303013 (+   312us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:00:34.308907  2330 raft_consensus.cc:2368] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for earlier term 9. Current term is 10.
I20260709 13:00:34.306326  1602 pending_rounds.cc:85] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a: Aborting all ops after (but not including) 9
I20260709 13:00:34.337493  1602 pending_rounds.cc:107] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a: Aborting uncommitted NO_OP operation due to leader change: 10.10
I20260709 13:00:34.337793  1602 raft_consensus.cc:2889] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:00:34.315234  1673 raft_consensus.cc:1240] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 LEADER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 10. Current term is 11. Ops: []
I20260709 13:00:34.316373  5475 raft_consensus.cc:515] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:00:34.318008  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 52718.9us
I20260709 13:00:34.319331  1748 raft_consensus.cc:1240] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 8. Current term is 9. Ops: []
I20260709 13:00:34.292088  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:34.320665  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:34.332868  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:34.342746  5369 raft_consensus.cc:2749] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:00:34.344017  1602 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40154 (request call id 2069) took 1619 ms. Trace:
I20260709 13:00:34.344529  1602 rpcz_store.cc:276] 0709 13:00:32.724699 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:34.249316 (+1524617us) service_pool.cc:224] Handling call
0709 13:00:34.249804 (+   488us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:34.343669 (+ 93865us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:34.343940 (+   271us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:00:34.345636  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:00:34.352478  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 169029us
I20260709 13:00:34.346419  1671 raft_consensus.cc:2468] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 9.
I20260709 13:00:34.299258  5325 raft_consensus.cc:515] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:34.360391  5475 leader_election.cc:290] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:34.363234  1748 raft_consensus.cc:1240] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 6. Current term is 7. Ops: []
I20260709 13:00:34.362901  5369 consensus_queue.cc:1059] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 13:00:34.363840  5251 consensus_queue.cc:1059] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 13:00:34.364666  5454 consensus_queue.cc:1048] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 13:00:34.365974  1598 raft_consensus.cc:2468] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 7.
I20260709 13:00:34.375514  1596 raft_consensus.cc:2468] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 9.
W20260709 13:00:34.390899  1596 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40154 (request call id 2067) took 1679 ms (client timeout 1640 ms). Trace:
W20260709 13:00:34.378153  5503 consensus_meta.cc:287] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.624s	user 0.008s	sys 0.008s
W20260709 13:00:34.395069  1596 rpcz_store.cc:269] 0709 13:00:32.711586 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:32.760757 (+ 49171us) service_pool.cc:224] Handling call
0709 13:00:34.039786 (+1279029us) spinlock_profiling.cc:243] Waited 1.27 ms on lock 0x7b5800003698. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f426303db71 00007f426303da22 00007f426bf56c09 00007f426b0b0ee0 00007f426b17a0ab 00007f426b17a131 00007f426b1280a0 00007f426b127984 00007f42646a26d4 00007f42646a2662
0709 13:00:34.390759 (+350973us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2924800}
I20260709 13:00:34.397636  1680 raft_consensus.cc:3060] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:00:34.398855  1605 raft_consensus.cc:2468] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 9.
I20260709 13:00:34.399477  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:34.400372  1671 raft_consensus.cc:2410] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for term 9 because replica has last-logged OpId of term: 8 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 8.
I20260709 13:00:34.401635  2337 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49082 (request call id 799) took 1858 ms. Trace:
I20260709 13:00:34.379698  5359 consensus_queue.cc:1059] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.022s
I20260709 13:00:34.401288  5358 consensus_queue.cc:1048] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.018s
I20260709 13:00:34.402848  2333 raft_consensus.cc:1240] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 8. Current term is 9. Ops: []
I20260709 13:00:34.407020  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:34.405328  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:00:34.407626  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 155990us
I20260709 13:00:34.404971  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:34.402065  2337 rpcz_store.cc:276] 0709 13:00:32.543317 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:32.543488 (+   171us) service_pool.cc:224] Handling call
0709 13:00:32.543899 (+   411us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:32.681292 (+137393us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:34.391930 (+1710638us) spinlock_profiling.cc:243] Waited 125 ms on lock 0x7b5400605254. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:00:34.401403 (+  9473us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":286627584}
I20260709 13:00:34.422720  2331 raft_consensus.cc:2468] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 8.
I20260709 13:00:34.423979  2330 raft_consensus.cc:2393] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 10: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
W20260709 13:00:34.407956  2298 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.419s	user 0.041s	sys 0.053s
W20260709 13:00:34.425036  2298 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.477s	user 0.041s	sys 0.054s
I20260709 13:00:34.407323  5036 raft_consensus.cc:493] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:34.431295  5369 raft_consensus.cc:493] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:34.431862  5036 raft_consensus.cc:515] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:34.432338  5369 raft_consensus.cc:515] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:34.435704  5036 leader_election.cc:290] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:34.435922  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:34.437177  1673 raft_consensus.cc:2393] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 9: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:00:34.438644  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:34.439172  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:34.439769  1671 raft_consensus.cc:2393] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 9: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:00:34.439983  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:34.440585  5473 consensus_queue.cc:1059] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.012s
I20260709 13:00:34.439886  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:34.441437  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:34.443081  1644 leader_election.cc:400] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 9 election: Vote denied by peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625) with higher term. Message: Invalid argument: T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for earlier term 9. Current term is 10.
I20260709 13:00:34.443478  1644 leader_election.cc:403] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 9 election: Cancelling election due to peer responding with higher term
I20260709 13:00:34.444392  5346 consensus_queue.cc:1048] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 13:00:34.445447  5369 leader_election.cc:290] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:34.447608  5086 leader_election.cc:290] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 8 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:34.446295  5346 raft_consensus.cc:3060] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:00:34.461272  5346 raft_consensus.cc:2749] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Leader election lost for term 9. Reason: Vote denied by peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625) with higher term. Message: Invalid argument: T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for earlier term 9. Current term is 10.
I20260709 13:00:34.464115  1500 catalog_manager.cc:5697] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "af520b67945d443b901bbeca37cc5701" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: HEALTHY } } }
I20260709 13:00:34.467901  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:34.468670  5325 leader_election.cc:290] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:34.469179  1672 raft_consensus.cc:2468] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 8.
I20260709 13:00:34.474586  5400 raft_consensus.cc:3060] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:00:34.441524  5473 consensus_queue.cc:1243] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer 3b879799899a499c8d409ee9f1277cd1 log is divergent from this leader: its last log entry 9.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20260709 13:00:34.476966  5228 raft_consensus.cc:493] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:34.407444  5496 raft_consensus.cc:2749] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260709 13:00:34.482288  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 1.53362e+06us
I20260709 13:00:34.485409  5413 raft_consensus.cc:3060] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:00:34.487151  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:34.447010  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:34.491602  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:34.493257  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:34.495121  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:34.496744  1678 raft_consensus.cc:1240] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 7. Current term is 8. Ops: []
W20260709 13:00:34.503705  5536 log.cc:927] Time spent T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.073s	user 0.000s	sys 0.000s
I20260709 13:00:34.429976  5255 raft_consensus.cc:515] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:34.504742  5457 raft_consensus.cc:493] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:00:34.507383  1717 leader_election.cc:304] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:34.510033  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:34.519311  1674 raft_consensus.cc:2410] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for term 9 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20260709 13:00:34.511535  5454 raft_consensus.cc:3055] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:00:34.406549  5382 consensus_queue.cc:1059] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 8.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.115s
I20260709 13:00:34.533514  5228 raft_consensus.cc:515] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:34.536530  2334 raft_consensus.cc:2368] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for earlier term 9. Current term is 10.
I20260709 13:00:34.527897  1672 raft_consensus.cc:2468] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 8.
I20260709 13:00:34.538561  1604 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40166 (request call id 2151) took 1217 ms. Trace:
I20260709 13:00:34.547709  1604 rpcz_store.cc:276] 0709 13:00:33.321129 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:34.486703 (+1165574us) service_pool.cc:224] Handling call
0709 13:00:34.538412 (+ 51709us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:00:34.540232  1676 raft_consensus.cc:2468] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 7.
I20260709 13:00:34.418387  5503 raft_consensus.cc:515] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:34.556403  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 9 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:34.557014  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:34.562960  1671 raft_consensus.cc:2410] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for term 8 because replica has last-logged OpId of term: 7 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20260709 13:00:34.546167  5454 raft_consensus.cc:740] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:00:34.570586  1601 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 2146) took 1274 ms. Trace:
W20260709 13:00:34.577836  1646 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:34.565215  5303 raft_consensus.cc:2749] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:00:34.568672  1599 raft_consensus.cc:2468] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 11.
I20260709 13:00:34.582428  1751 raft_consensus.cc:2468] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 11.
I20260709 13:00:34.578704  1601 rpcz_store.cc:276] 0709 13:00:33.288175 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:34.343052 (+1054877us) service_pool.cc:224] Handling call
0709 13:00:34.343600 (+   548us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:34.343976 (+   376us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 13:00:34.510928 (+166952us) log.cc:844] Serialized 14 byte log entry
0709 13:00:34.562018 (+ 51090us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:00:34.562106 (+    88us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:00:34.562196 (+    90us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:34.562297 (+   101us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:34.562499 (+   202us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":10683,"thread_start_us":185790,"threads_started":2}
I20260709 13:00:34.586199  1599 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40166 (request call id 2139) took 1312 ms. Trace:
I20260709 13:00:34.586656  1599 rpcz_store.cc:276] 0709 13:00:33.273676 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:34.116238 (+842562us) service_pool.cc:224] Handling call
0709 13:00:34.286851 (+170613us) spinlock_profiling.cc:243] Waited 1.99 ms on lock 0x7b5800003698. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f426303db71 00007f426303da22 00007f426bf56c09 00007f426b0b0ee0 00007f426b17a0ab 00007f426b17a131 00007f426b1280a0 00007f426b127984 00007f42646a26d4 00007f42646a2662
0709 13:00:34.586125 (+299274us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4580224}
W20260709 13:00:34.588362  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 219942us
I20260709 13:00:34.588872  1746 raft_consensus.cc:2393] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 9: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:00:34.596897  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:34.594741  5228 leader_election.cc:290] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:34.600919  5458 raft_consensus.cc:3060] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:00:34.606333  1596 raft_consensus.cc:1240] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 8. Current term is 9. Ops: [8.5-8.5]
I20260709 13:00:34.606596  5390 raft_consensus.cc:493] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:34.607225  5390 raft_consensus.cc:515] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:34.609766  5390 leader_election.cc:290] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:34.612692  5503 leader_election.cc:290] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 9 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:34.614326  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:34.614952  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:34.615844  1673 raft_consensus.cc:2393] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 9: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:00:34.615744  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:34.616307  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 9 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:34.616860  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:34.617205  1674 raft_consensus.cc:2410] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 for term 9 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260709 13:00:34.617859  1673 raft_consensus.cc:2393] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 9: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:00:34.592459  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:34.619998  1599 raft_consensus.cc:2368] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for earlier term 9. Current term is 10.
W20260709 13:00:34.619207  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 110445us
I20260709 13:00:34.620522  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:34.621199  1751 raft_consensus.cc:2468] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 9.
I20260709 13:00:34.622568  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:34.623260  1671 raft_consensus.cc:2468] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 9.
I20260709 13:00:34.626042  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:34.592492  1500 catalog_manager.cc:5697] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 13:00:34.625869  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:34.553210  5457 raft_consensus.cc:515] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:34.635414  1745 raft_consensus.cc:1240] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 10. Current term is 11. Ops: [6.7-10.8]
I20260709 13:00:34.636063  1679 raft_consensus.cc:1275] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 5 index: 8. Preceding OpId from leader: term: 8 index: 10. (index mismatch)
I20260709 13:00:34.636507  1751 raft_consensus.cc:1275] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260709 13:00:34.637151  2330 raft_consensus.cc:1240] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 9 LEADER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 8. Current term is 9. Ops: []
I20260709 13:00:34.637458  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:34.643904  1752 raft_consensus.cc:1275] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260709 13:00:34.637145  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:34.638378  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:00:34.647435  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 225371us
I20260709 13:00:34.648430  1754 raft_consensus.cc:1240] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 6. Current term is 7. Ops: []
I20260709 13:00:34.650575  2298 leader_election.cc:304] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d
W20260709 13:00:34.652206  5538 log.cc:927] Time spent T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.067s	user 0.001s	sys 0.000s
I20260709 13:00:34.652277  1598 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40166 (request call id 2129) took 2104 ms. Trace:
I20260709 13:00:34.652864  1598 rpcz_store.cc:276] 0709 13:00:32.548903 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:34.177614 (+1628711us) service_pool.cc:224] Handling call
0709 13:00:34.652037 (+474423us) spinlock_profiling.cc:243] Waited 145 ms on lock 0x7b54004222d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:00:34.652188 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":332609792}
I20260709 13:00:34.655799  5392 raft_consensus.cc:3055] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:00:34.626911  5390 raft_consensus.cc:493] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:00:34.659763  5542 log.cc:927] Time spent T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.104s	user 0.003s	sys 0.000s
I20260709 13:00:34.667630  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:34.667915  1679 raft_consensus.cc:1275] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 13:00:34.668001  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:34.685153  1680 raft_consensus.cc:2468] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 9.
W20260709 13:00:34.628439  1569 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.107.196:46625 timed out after 1.824s (SENT) [suppressed 54 similar messages]
I20260709 13:00:34.671355  1500 catalog_manager.cc:5697] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "af520b67945d443b901bbeca37cc5701" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: HEALTHY } } }
I20260709 13:00:34.683602  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:34.688066  1671 raft_consensus.cc:2468] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 9.
I20260709 13:00:34.692967  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:34.676259  5392 raft_consensus.cc:740] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:00:34.695626  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:34.696869  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:34.697592  1673 raft_consensus.cc:2393] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 7: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:00:34.698994  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:34.698781  5392 consensus_queue.cc:260] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:34.700306  5392 raft_consensus.cc:3060] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:00:34.700376  1678 raft_consensus.cc:1240] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 8. Current term is 9. Ops: []
I20260709 13:00:34.702339  5390 raft_consensus.cc:515] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:34.705303  5390 leader_election.cc:290] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:34.707727  2298 leader_election.cc:304] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:00:34.712845  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:34.712947  5475 raft_consensus.cc:3055] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:00:34.713759  5475 raft_consensus.cc:740] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:00:34.694003  5303 consensus_queue.cc:1059] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true }, Status: INVALID_TERM, Last received: 6.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.020s
I20260709 13:00:34.715011  5303 raft_consensus.cc:3055] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:00:34.715523  5303 raft_consensus.cc:740] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:00:34.716822  5346 raft_consensus.cc:3055] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:00:34.717592  1718 leader_election.cc:304] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:00:34.589417  5431 raft_consensus.cc:515] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:34.725221  1677 raft_consensus.cc:2393] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 10: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:00:34.726403  5431 leader_election.cc:290] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:34.702962  5152 leader_election.cc:290] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 7 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:34.717372  5346 raft_consensus.cc:740] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:00:34.723994  1600 raft_consensus.cc:2410] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for term 11 because replica has last-logged OpId of term: 10 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 6.
I20260709 13:00:34.726318  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:34.728798  1674 raft_consensus.cc:3060] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:00:34.728873  5346 consensus_queue.cc:260] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:34.713176  5228 raft_consensus.cc:2804] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 13:00:34.730832  5228 raft_consensus.cc:697] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 10 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:00:34.731849  5228 consensus_queue.cc:237] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:34.703037  5255 leader_election.cc:290] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 8 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:34.736901  1646 leader_election.cc:304] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:00:34.737659  5362 consensus_queue.cc:1059] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 13:00:34.738682  5358 raft_consensus.cc:2804] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:00:34.739166  5358 raft_consensus.cc:493] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:34.739004  5362 consensus_queue.cc:1048] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 13:00:34.739586  5358 raft_consensus.cc:3060] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Advancing to term 10
W20260709 13:00:34.740456  1600 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40166 (request call id 2118) took 2214 ms (client timeout 1583 ms). Trace:
I20260709 13:00:34.740649  5362 raft_consensus.cc:3055] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:00:34.741140  5362 raft_consensus.cc:740] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
W20260709 13:00:34.740900  1600 rpcz_store.cc:269] 0709 13:00:32.526053 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:32.526283 (+   230us) service_pool.cc:224] Handling call
0709 13:00:34.740156 (+2213873us) spinlock_profiling.cc:243] Waited 2.19 s on lock 0x7b54002175d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:00:34.740342 (+   186us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5039508352}
I20260709 13:00:34.742211  5036 raft_consensus.cc:3055] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:00:34.742718  5036 raft_consensus.cc:740] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:00:34.745939  2329 raft_consensus.cc:2393] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 9: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:00:34.747576  1605 raft_consensus.cc:1275] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 10 index: 7. Preceding OpId from leader: term: 11 index: 7. (term mismatch)
I20260709 13:00:34.748044  1605 pending_rounds.cc:85] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a: Aborting all ops after (but not including) 6
I20260709 13:00:34.748343  1605 pending_rounds.cc:107] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a: Aborting uncommitted NO_OP operation due to leader change: 10.7
I20260709 13:00:34.748644  1605 raft_consensus.cc:2889] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:00:34.684028  5462 raft_consensus.cc:2804] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:00:34.750911  5462 raft_consensus.cc:493] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:00:34.751405  5462 raft_consensus.cc:3060] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:00:34.751686  5457 leader_election.cc:290] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:34.765836  1500 catalog_manager.cc:5697] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 reported cstate change: term changed from 3 to 5, leader changed from 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194) to af520b67945d443b901bbeca37cc5701 (127.0.107.195). New cstate: current_term: 5 leader_uuid: "af520b67945d443b901bbeca37cc5701" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: HEALTHY } } }
W20260709 13:00:34.768049  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 57927.2us
W20260709 13:00:34.776925  1644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:34.730309  5346 raft_consensus.cc:3060] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:00:34.782591  1718 leader_election.cc:304] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:34.715068  5475 consensus_queue.cc:260] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:34.716676  5303 consensus_queue.cc:260] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:34.742314  5362 consensus_queue.cc:260] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:34.789702  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:34.790182  5362 raft_consensus.cc:3060] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:00:34.790712  5413 raft_consensus.cc:493] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
W20260709 13:00:34.790796  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 86799.5us
I20260709 13:00:34.791404  5413 raft_consensus.cc:515] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:34.793691  5413 leader_election.cc:290] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:34.790477  1678 raft_consensus.cc:2468] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 11.
I20260709 13:00:34.769764  5496 raft_consensus.cc:2749] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:00:34.781694  1672 raft_consensus.cc:2468] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 10.
I20260709 13:00:34.744036  5036 consensus_queue.cc:260] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.8, Last appended by leader: 8, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:34.787922  5224 consensus_meta.cc:287] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.542s	user 0.016s	sys 0.004s
I20260709 13:00:34.804997  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
W20260709 13:00:34.784005  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 155578us
W20260709 13:00:34.787662  5470 consensus_meta.cc:287] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.547s	user 0.013s	sys 0.003s
I20260709 13:00:34.813916  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
W20260709 13:00:34.815800  1570 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:34.819254  1603 raft_consensus.cc:1275] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 13:00:34.798143  5255 raft_consensus.cc:3060] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:00:34.804563  1674 raft_consensus.cc:2468] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 10.
I20260709 13:00:34.724080  5454 consensus_queue.cc:260] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:34.797353  5502 raft_consensus.cc:493] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
W20260709 13:00:34.832857  5428 consensus_meta.cc:287] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.686s	user 0.011s	sys 0.001s
I20260709 13:00:34.834724  5502 raft_consensus.cc:515] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:34.836122  1599 raft_consensus.cc:1275] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20260709 13:00:34.825923  2301 leader_election.cc:304] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:00:34.821504  5475 raft_consensus.cc:3060] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:00:34.852571  1718 leader_election.cc:304] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
W20260709 13:00:34.855890  1569 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:00:34.856603  1569 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:00:34.857095  1569 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:34.813370  5506 consensus_queue.cc:1059] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.042s
I20260709 13:00:34.859136  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:34.863737  1672 raft_consensus.cc:2468] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 9.
I20260709 13:00:34.860366  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:34.862962  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:34.868466  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:34.869378  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:34.870029  1599 raft_consensus.cc:2468] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 8.
I20260709 13:00:34.872048  1678 raft_consensus.cc:1240] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 6. Current term is 7. Ops: []
I20260709 13:00:34.824474  5431 consensus_queue.cc:1059] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.039s
I20260709 13:00:34.878134  1680 raft_consensus.cc:2468] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 7.
W20260709 13:00:34.881659  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 61910.4us
I20260709 13:00:34.885864  1679 raft_consensus.cc:2468] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 6.
I20260709 13:00:34.890374  2299 leader_election.cc:304] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:00:34.899052  5491 consensus_queue.cc:1048] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
W20260709 13:00:34.902343  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:34.904932  1569 leader_election.cc:304] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:00:34.868237  5482 raft_consensus.cc:2804] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:00:34.871095  5303 raft_consensus.cc:3060] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:00:34.875934  5454 raft_consensus.cc:3060] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:00:34.907985  2301 leader_election.cc:304] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:00:34.915190  1674 raft_consensus.cc:2468] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 6.
I20260709 13:00:34.893934  5457 raft_consensus.cc:2804] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 13:00:34.917198  1500 catalog_manager.cc:5697] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } }
W20260709 13:00:34.925491  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 83409.6us
I20260709 13:00:34.924422  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:34.910140  5482 raft_consensus.cc:493] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:34.927774  1600 raft_consensus.cc:2468] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 7.
I20260709 13:00:34.936407  1674 raft_consensus.cc:1240] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 9. Current term is 10. Ops: []
I20260709 13:00:34.939855  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:34.934350  5482 raft_consensus.cc:3060] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:00:34.934448  5083 raft_consensus.cc:2749] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:00:34.943658  1500 catalog_manager.cc:5697] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 reported cstate change: term changed from 5 to 6, leader changed from 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194) to af520b67945d443b901bbeca37cc5701 (127.0.107.195). New cstate: current_term: 6 leader_uuid: "af520b67945d443b901bbeca37cc5701" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 13:00:34.924108  5413 raft_consensus.cc:2749] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260709 13:00:34.950295  5466 consensus_meta.cc:287] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.813s	user 0.005s	sys 0.018s
I20260709 13:00:34.949366  1500 catalog_manager.cc:5697] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 reported cstate change: config changed from index 3 to 5, term changed from 4 to 5, leader changed from af520b67945d443b901bbeca37cc5701 (127.0.107.195) to <none>, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New cstate: current_term: 5 committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 13:00:34.952957  5466 raft_consensus.cc:515] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:34.930646  5506 raft_consensus.cc:2804] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 13:00:34.961606  1675 raft_consensus.cc:2368] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 for earlier term 7. Current term is 8.
I20260709 13:00:34.958127  1500 catalog_manager.cc:5697] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 reported cstate change: term changed from 2 to 4, leader changed from 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194) to 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193). New cstate: current_term: 4 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } }
I20260709 13:00:34.969373  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:34.970083  1600 raft_consensus.cc:3060] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:00:34.970968  1597 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 2145) took 1671 ms. Trace:
I20260709 13:00:34.934708  5457 raft_consensus.cc:697] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 9 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
W20260709 13:00:34.974150  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 66279.1us
I20260709 13:00:34.975294  1674 raft_consensus.cc:1240] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 8. Current term is 9. Ops: []
I20260709 13:00:34.991542  1597 rpcz_store.cc:276] 0709 13:00:33.287846 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:34.366958 (+1079112us) service_pool.cc:224] Handling call
0709 13:00:34.402356 (+ 35398us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:34.402781 (+   425us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 13:00:34.588345 (+185564us) log.cc:844] Serialized 14 byte log entry
0709 13:00:34.958946 (+370601us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:00:34.959029 (+    83us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:00:34.959110 (+    81us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:34.959202 (+    92us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:34.959371 (+   169us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":251897,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":54,"raft.queue_time_us":30035,"thread_start_us":436837,"threads_started":2}
I20260709 13:00:34.961901  5466 leader_election.cc:290] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 8 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:00:35.002856  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 52110.3us
I20260709 13:00:35.004731  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:35.005652  1678 raft_consensus.cc:2393] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 7: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
W20260709 13:00:35.007472  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 132043us
I20260709 13:00:34.928330  5224 consensus_queue.cc:1059] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.068s
W20260709 13:00:35.013350  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 138662us
I20260709 13:00:34.980643  5506 raft_consensus.cc:697] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 9 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:00:34.991305  5325 raft_consensus.cc:493] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:00:35.024580  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 62313.7us
W20260709 13:00:35.043481  1644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:00:35.081712  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 132639us
W20260709 13:00:35.085071  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 58179.2us
I20260709 13:00:35.087299  2301 leader_election.cc:304] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
W20260709 13:00:35.092746  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 68576.1us
W20260709 13:00:35.095460  5550 log.cc:927] Time spent T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.095s	user 0.001s	sys 0.000s
I20260709 13:00:35.096686   431 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:00:35.101583  5255 raft_consensus.cc:2749] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:00:35.103022  5506 consensus_queue.cc:237] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 7.9, Last appended by leader: 9, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:35.107436  5413 consensus_queue.cc:1048] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 13:00:35.108013  1569 leader_election.cc:304] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 13:00:35.109135  5413 raft_consensus.cc:2764] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader pre-election decision vote started in defunct term 9: won
I20260709 13:00:35.109545  1569 leader_election.cc:304] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
W20260709 13:00:35.111414  5544 log.cc:927] Time spent T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.141s	user 0.000s	sys 0.000s
I20260709 13:00:35.114522  5502 leader_election.cc:290] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:35.115131  5502 raft_consensus.cc:2749] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:00:35.118687  1603 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40154 (request call id 2066) took 2436 ms. Trace:
I20260709 13:00:35.119107  1603 rpcz_store.cc:276] 0709 13:00:32.681874 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:34.233245 (+1551371us) service_pool.cc:224] Handling call
0709 13:00:34.233745 (+   500us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:34.819650 (+585905us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:35.117125 (+297475us) spinlock_profiling.cc:243] Waited 3.82 ms on lock 0x7b5400420ca0. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42660d9dc0 00007f4266165a02 00007f4266161dd6 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2
0709 13:00:35.118379 (+  1254us) spinlock_profiling.cc:243] Waited 582 ms on lock 0x7b54003edad4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:00:35.118614 (+   235us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1346289792}
I20260709 13:00:35.120468  1602 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 2149) took 1831 ms. Trace:
I20260709 13:00:35.023425  5083 consensus_queue.cc:1059] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.099s
I20260709 13:00:35.123571  5083 raft_consensus.cc:3055] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:00:35.123944  5083 raft_consensus.cc:740] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:00:35.124938  5083 consensus_queue.cc:260] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:35.082494  5325 raft_consensus.cc:515] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:35.128372  1646 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:35.131256  5036 raft_consensus.cc:3060] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Advancing to term 11
W20260709 13:00:35.132472  5547 log.cc:927] Time spent T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.161s	user 0.000s	sys 0.000s
I20260709 13:00:35.135749  5545 raft_consensus.cc:3055] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:00:35.136191  5545 raft_consensus.cc:740] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
W20260709 13:00:35.137238  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:35.137140  5545 consensus_queue.cc:260] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:35.139405  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:35.140058  1748 raft_consensus.cc:1275] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20260709 13:00:35.140269  1746 raft_consensus.cc:2393] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 8: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:00:35.140839  2298 leader_election.cc:304] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:00:35.141459  1500 catalog_manager.cc:5697] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 reported cstate change: term changed from 1 to 3, leader changed from 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) to af520b67945d443b901bbeca37cc5701 (127.0.107.195). New cstate: current_term: 3 leader_uuid: "af520b67945d443b901bbeca37cc5701" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: HEALTHY } } }
I20260709 13:00:35.142782  5214 raft_consensus.cc:2749] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:00:35.143904  2299 leader_election.cc:304] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:00:35.143764  5358 raft_consensus.cc:515] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:35.145613  2299 leader_election.cc:304] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:00:35.147845  5522 raft_consensus.cc:2749] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 13:00:35.147966  5398 raft_consensus.cc:2749] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:00:35.148820  5522 consensus_queue.cc:1048] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 13:00:35.148998  5398 consensus_queue.cc:1048] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 13:00:35.120872  1602 rpcz_store.cc:276] 0709 13:00:33.289124 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:34.345359 (+1056235us) service_pool.cc:224] Handling call
0709 13:00:34.345895 (+   536us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:00:34.346355 (+   460us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:00:34.346385 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:00:34.375170 (+ 28785us) log.cc:844] Serialized 26 byte log entry
0709 13:00:34.797658 (+422488us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:00:34.824938 (+ 27280us) log.cc:844] Serialized 14 byte log entry
0709 13:00:34.825175 (+   237us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:34.825260 (+    85us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:00:35.120214 (+294954us) raft_consensus.cc:1685] finished
0709 13:00:35.120240 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:35.120400 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":22224,"prepare.run_cpu_time_us":69,"prepare.run_wall_time_us":4785,"raft.queue_time_us":2217,"raft.run_cpu_time_us":365,"raft.run_wall_time_us":32517,"thread_start_us":292632,"threads_started":1,"wal-append.queue_time_us":487094}
I20260709 13:00:35.152839  5473 consensus_queue.cc:1048] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 13:00:35.153609  5473 consensus_queue.cc:1243] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer af520b67945d443b901bbeca37cc5701 log is divergent from this leader: its last log entry 8.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260709 13:00:35.130332  5325 leader_election.cc:290] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:35.155685  5358 leader_election.cc:290] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:35.142366  2298 leader_election.cc:304] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:00:35.171458  1600 raft_consensus.cc:2468] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 10.
I20260709 13:00:35.171655  1605 raft_consensus.cc:2468] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 6.
W20260709 13:00:35.172282  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 91050.8us
W20260709 13:00:35.173731  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 80256.1us
W20260709 13:00:35.174175  1643 leader_election.cc:336] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 1.740s (SENT)
W20260709 13:00:35.182719  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 73256.1us
I20260709 13:00:35.174958  1643 leader_election.cc:304] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:00:35.138311  5545 raft_consensus.cc:3060] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:00:35.126343  5083 raft_consensus.cc:3060] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:00:35.179414  5382 raft_consensus.cc:493] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:00:35.197628  2299 leader_election.cc:304] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:35.198942  1569 leader_election.cc:304] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:00:35.196298  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:35.203655  1748 raft_consensus.cc:3060] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:00:35.093838  5224 consensus_queue.cc:1243] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer 816fadcea8fc4893882f5f9b9f22a03d log is divergent from this leader: its last log entry 8.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260709 13:00:35.154897  5383 raft_consensus.cc:2804] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 13:00:35.215715  1754 pending_rounds.cc:85] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701: Aborting all ops after (but not including) 4
I20260709 13:00:35.216179  1754 pending_rounds.cc:107] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701: Aborting uncommitted NO_OP operation due to leader change: 8.5
I20260709 13:00:35.216411  1599 raft_consensus.cc:3055] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:00:35.216883  1599 raft_consensus.cc:740] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:00:35.199867  5457 consensus_queue.cc:237] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:35.216473  1754 raft_consensus.cc:2889] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:00:35.223165  1599 consensus_queue.cc:260] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:35.224809  1599 raft_consensus.cc:3060] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Advancing to term 8
W20260709 13:00:35.226533  1644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:35.187186  5228 consensus_queue.cc:1048] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.046s
I20260709 13:00:35.239220  5362 raft_consensus.cc:2749] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260709 13:00:35.243076  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 46125.7us
I20260709 13:00:35.251425  1746 raft_consensus.cc:1240] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 8. Current term is 9. Ops: []
I20260709 13:00:35.254091  1672 raft_consensus.cc:2468] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 9.
I20260709 13:00:35.251514  1644 leader_election.cc:304] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: af520b67945d443b901bbeca37cc5701
W20260709 13:00:35.256805  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 114522us
I20260709 13:00:35.257318  1644 leader_election.cc:304] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
W20260709 13:00:35.259506  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 61965.1us
I20260709 13:00:35.235684  5224 raft_consensus.cc:2749] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:00:35.269879  2299 leader_election.cc:304] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:35.271236  1644 leader_election.cc:400] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 9 pre-election: Vote denied by peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625) with higher term. Message: Invalid argument: T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for earlier term 9. Current term is 10.
I20260709 13:00:35.273259  1746 raft_consensus.cc:1240] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 9. Current term is 10. Ops: []
I20260709 13:00:35.275344  1752 raft_consensus.cc:1275] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 13:00:35.285810  1752 raft_consensus.cc:1240] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 6. Current term is 7. Ops: []
I20260709 13:00:35.253070  5398 raft_consensus.cc:2804] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 13:00:35.286070  1571 leader_election.cc:304] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:00:35.264710  5228 raft_consensus.cc:2749] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260709 13:00:35.291604  5462 consensus_meta.cc:287] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.532s	user 0.014s	sys 0.002s
I20260709 13:00:35.301349  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:00:35.302264  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:35.304680  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:35.304800  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:00:35.301146  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:00:35.298990  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 100143us
I20260709 13:00:35.268716  5358 raft_consensus.cc:2804] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:00:35.299294  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:00:35.278092  5473 raft_consensus.cc:2749] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:00:35.307361  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:35.310055  2330 raft_consensus.cc:3060] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Advancing to term 8
I20260709 13:00:35.256592  5214 consensus_queue.cc:1059] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.042s
I20260709 13:00:35.308925  2333 raft_consensus.cc:2376] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 7. Re-sending same reply.
I20260709 13:00:35.312016  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:35.314991  1569 leader_election.cc:304] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:00:35.315999  1751 raft_consensus.cc:2393] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 9: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:00:35.311311  5358 raft_consensus.cc:493] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:35.318007  5358 raft_consensus.cc:3060] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Advancing to term 9
W20260709 13:00:35.321847  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 227668us
I20260709 13:00:35.297827  5522 raft_consensus.cc:2749] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:00:35.323666  5522 consensus_queue.cc:1059] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20260709 13:00:35.324540  1643 leader_election.cc:336] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 1.793s (SENT)
I20260709 13:00:35.325346  5522 consensus_queue.cc:1059] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 9.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 13:00:35.315140  2332 raft_consensus.cc:2468] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 12.
I20260709 13:00:35.327206  1644 leader_election.cc:304] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:00:35.313282  2299 leader_election.cc:304] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:35.349031  1599 raft_consensus.cc:1275] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 8 index: 6. (term mismatch)
W20260709 13:00:35.351435  1643 leader_election.cc:336] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 1.849s (SENT)
I20260709 13:00:35.354817  1643 leader_election.cc:304] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:35.264954  5382 raft_consensus.cc:515] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:35.308525  5224 raft_consensus.cc:2749] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 13:00:35.356669  1569 leader_election.cc:304] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:00:35.339774  5362 raft_consensus.cc:2749] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:00:35.356406  1599 pending_rounds.cc:85] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a: Aborting all ops after (but not including) 5
I20260709 13:00:35.368275  1599 pending_rounds.cc:107] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a: Aborting uncommitted NO_OP operation due to leader change: 7.6
I20260709 13:00:35.368680  1599 raft_consensus.cc:2889] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:00:35.375473  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:00:35.377161  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 64011.5us
I20260709 13:00:35.361543  5083 raft_consensus.cc:3055] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:00:35.307474  5482 raft_consensus.cc:515] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:35.360639  5325 raft_consensus.cc:2749] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:00:35.316169  5462 raft_consensus.cc:515] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:35.395731  1569 leader_election.cc:304] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
W20260709 13:00:35.384675  5458 consensus_meta.cc:287] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.783s	user 0.013s	sys 0.005s
I20260709 13:00:35.375018  5392 raft_consensus.cc:2749] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260709 13:00:35.401151  5390 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 8 current term: 8 is higher.
I20260709 13:00:35.394977  5470 raft_consensus.cc:2749] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:00:35.427572  2336 raft_consensus.cc:1240] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 8. Current term is 9. Ops: []
W20260709 13:00:35.420058  5551 log.cc:927] Time spent T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.209s	user 0.000s	sys 0.003s
I20260709 13:00:35.394820  5083 raft_consensus.cc:740] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
W20260709 13:00:35.432617  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 51053.7us
I20260709 13:00:35.438267  1569 leader_election.cc:304] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:00:35.440093  1569 leader_election.cc:304] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
W20260709 13:00:35.423687  5454 consensus_meta.cc:287] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.509s	user 0.013s	sys 0.000s
I20260709 13:00:35.394767  5503 consensus_queue.cc:1059] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 9.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.037s
W20260709 13:00:35.448474  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 145829us
I20260709 13:00:35.424360  5224 raft_consensus.cc:2749] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260709 13:00:35.453217  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 101808us
I20260709 13:00:35.446796  1569 leader_election.cc:304] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:00:35.454913  5392 raft_consensus.cc:3060] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Advancing to term 7
W20260709 13:00:35.461987  1644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:35.463665  1569 leader_election.cc:304] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
W20260709 13:00:35.452438  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 123913us
I20260709 13:00:35.427078  5383 raft_consensus.cc:697] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 11 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:00:35.441906  5214 raft_consensus.cc:3055] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:00:35.448390  5325 raft_consensus.cc:2804] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:00:35.477031  1569 leader_election.cc:304] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:00:35.446827  5475 raft_consensus.cc:2749] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:00:35.437076  5255 consensus_queue.cc:1059] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 8.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.038s
W20260709 13:00:35.515146  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 82818.1us
I20260709 13:00:35.480638  5214 raft_consensus.cc:740] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:00:35.451610  5428 consensus_queue.cc:1059] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 7.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.042s
I20260709 13:00:35.442133  5457 consensus_queue.cc:1048] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.035s
I20260709 13:00:35.515686  5413 raft_consensus.cc:2804] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:00:35.526109  1600 raft_consensus.cc:1275] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 7 index: 8. (term mismatch)
I20260709 13:00:35.529155  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:35.541072  1600 pending_rounds.cc:85] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a: Aborting all ops after (but not including) 7
I20260709 13:00:35.541417  1605 raft_consensus.cc:3060] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:00:35.527535  1597 raft_consensus.cc:1240] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 8. Current term is 9. Ops: []
I20260709 13:00:35.532611  2334 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "a40e423ed583480280352b261913276c"
dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
mode: GRACEFUL
new_leader_uuid: "af520b67945d443b901bbeca37cc5701"
 from {username='slave'} at 127.0.0.1:35790
I20260709 13:00:35.543758  2334 raft_consensus.cc:606] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Received request to transfer leadership to TS af520b67945d443b901bbeca37cc5701
I20260709 13:00:35.544134  2334 raft_consensus.cc:614] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 13:00:35.526755  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:35.545846  5428 raft_consensus.cc:3055] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:00:35.546295  1750 raft_consensus.cc:1240] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 10. Current term is 11. Ops: []
I20260709 13:00:35.546185  1602 raft_consensus.cc:2376] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 10. Re-sending same reply.
I20260709 13:00:35.530279  2335 raft_consensus.cc:2468] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 7.
I20260709 13:00:35.514813  5083 consensus_queue.cc:260] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:35.541457  1600 pending_rounds.cc:107] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a: Aborting uncommitted NO_OP operation due to leader change: 6.8
I20260709 13:00:35.548017  1600 raft_consensus.cc:2889] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:00:35.548462  5083 raft_consensus.cc:3060] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Advancing to term 9
W20260709 13:00:35.480896  5556 log.cc:927] Time spent T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.190s	user 0.000s	sys 0.001s
I20260709 13:00:35.476456  5502 raft_consensus.cc:2804] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:00:35.532753  5503 raft_consensus.cc:3055] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:00:35.550745  5502 raft_consensus.cc:493] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:35.551106  5503 raft_consensus.cc:740] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:00:35.551318  5502 raft_consensus.cc:3060] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Advancing to term 9
W20260709 13:00:35.489938  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 84277.7us
I20260709 13:00:35.552712  5228 raft_consensus.cc:3055] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:00:35.552325  5503 consensus_queue.cc:260] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:35.553210  5228 raft_consensus.cc:740] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:00:35.523334  5383 consensus_queue.cc:237] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.8, Last appended by leader: 8, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:35.556084  5413 raft_consensus.cc:493] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:35.554498  1718 leader_election.cc:304] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:00:35.556591  5413 raft_consensus.cc:3060] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:00:35.561301  2299 leader_election.cc:304] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:00:35.550289  1754 raft_consensus.cc:1240] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 6 LEADER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 5. Current term is 6. Ops: []
I20260709 13:00:35.555088  5522 consensus_queue.cc:1048] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.008s
I20260709 13:00:35.535863  5462 leader_election.cc:290] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 9 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:35.566426  1598 raft_consensus.cc:1240] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 9. Current term is 10. Ops: []
I20260709 13:00:35.554311  5228 consensus_queue.cc:260] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:35.567234  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:35.562423  5457 consensus_queue.cc:1059] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.006s
I20260709 13:00:35.571594  5152 raft_consensus.cc:2804] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 13:00:35.571780  5457 raft_consensus.cc:3055] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:00:35.572257  5152 raft_consensus.cc:697] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 9 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:00:35.572324  5457 raft_consensus.cc:740] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:00:35.573478  5457 consensus_queue.cc:260] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:35.570194  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:35.587247  5563 raft_consensus.cc:493] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:00:35.587818  2300 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:35.588444  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:35.589722  1745 raft_consensus.cc:2393] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 8: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:00:35.590680  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:35.591529  1745 raft_consensus.cc:2393] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 7: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:00:35.516196  5564 consensus_queue.cc:1059] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.055s
I20260709 13:00:35.592862  5564 raft_consensus.cc:3055] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:00:35.593298  5564 raft_consensus.cc:740] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:00:35.570552  5228 raft_consensus.cc:3060] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:00:35.569278  5431 consensus_queue.cc:1059] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 13:00:35.575042  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:35.604523  1751 raft_consensus.cc:2393] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 7: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:00:35.575115  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:35.605510  1754 raft_consensus.cc:2468] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 7.
I20260709 13:00:35.546321  5428 raft_consensus.cc:740] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:00:35.576315  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:35.577466  2299 leader_election.cc:304] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:00:35.578148  5462 raft_consensus.cc:2804] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Leader pre-election won for term 12
W20260709 13:00:35.554536  1646 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:35.578910  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:35.579530  5382 leader_election.cc:290] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:35.579854  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:35.624305  1753 raft_consensus.cc:2393] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 7: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:00:35.581319  5482 leader_election.cc:290] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 7 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:35.581431  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:35.530299  5545 raft_consensus.cc:3060] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:00:35.583099  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:00:35.584003  5303 consensus_meta.cc:287] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.669s	user 0.011s	sys 0.003s
I20260709 13:00:35.584414  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:35.537680  5496 raft_consensus.cc:493] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:35.588042  5563 raft_consensus.cc:515] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:35.573444  5152 consensus_queue.cc:237] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.9, Last appended by leader: 9, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:35.492051  1643 leader_election.cc:336] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 1.891s (SENT)
I20260709 13:00:35.597784  1748 raft_consensus.cc:2468] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 10.
I20260709 13:00:35.514048  1717 leader_election.cc:304] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:35.486806  5325 raft_consensus.cc:493] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:35.578621  5503 raft_consensus.cc:3060] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:00:35.607492  2330 raft_consensus.cc:2468] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 8.
I20260709 13:00:35.651280  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:35.609642  1750 raft_consensus.cc:2393] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 9: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:00:35.616816  1605 raft_consensus.cc:2468] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 7.
W20260709 13:00:35.620486  2301 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:35.634961  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:35.607810  5462 raft_consensus.cc:493] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:35.665104  1601 raft_consensus.cc:3060] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:00:35.638967  1604 pending_rounds.cc:85] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a: Aborting all ops after (but not including) 6
I20260709 13:00:35.639351  5214 consensus_queue.cc:260] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:35.640635  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 63280.1us
I20260709 13:00:35.622761  5390 raft_consensus.cc:493] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:35.640924  5564 consensus_queue.cc:260] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:35.642575  1597 raft_consensus.cc:1240] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 8. Current term is 9. Ops: [8.6-8.6]
I20260709 13:00:35.651165  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:35.638198  5398 raft_consensus.cc:697] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 9 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:00:35.656493  1643 leader_election.cc:304] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:35.665798  1604 pending_rounds.cc:107] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a: Aborting uncommitted NO_OP operation due to leader change: 7.7
I20260709 13:00:35.686935  1604 raft_consensus.cc:2889] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:00:35.687295  2336 raft_consensus.cc:2393] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 7: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:00:35.642792  5358 raft_consensus.cc:515] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:35.636796  5255 raft_consensus.cc:3055] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:00:35.669462  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:35.728632  1605 raft_consensus.cc:2393] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 7: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:00:35.723493  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:35.631673  5562 consensus_queue.cc:1059] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.040s
I20260709 13:00:35.653853  5325 raft_consensus.cc:3060] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:00:35.671325  5462 raft_consensus.cc:3060] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Advancing to term 12
W20260709 13:00:35.739696  1644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:35.740814  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:35.727792  5255 raft_consensus.cc:740] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
W20260709 13:00:35.743547  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 262063us
I20260709 13:00:35.743418  5255 consensus_queue.cc:260] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:00:35.744447  1643 leader_election.cc:336] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 1.945s (SENT)
I20260709 13:00:35.744907  5255 raft_consensus.cc:3060] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:00:35.745340  5382 raft_consensus.cc:2749] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:00:35.747812  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:35.746331  1643 leader_election.cc:304] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:35.746182  5382 raft_consensus.cc:493] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:35.643559  5560 consensus_queue.cc:1048] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.086s
I20260709 13:00:35.757629  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
W20260709 13:00:35.760025  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 121385us
I20260709 13:00:35.683812  5572 raft_consensus.cc:2804] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 13:00:35.650897  5496 raft_consensus.cc:515] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:35.765476  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
W20260709 13:00:35.770471  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 293497us
I20260709 13:00:35.775707  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:35.777310  1569 leader_election.cc:304] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:00:35.705348  5390 raft_consensus.cc:515] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:35.802889  2298 leader_election.cc:304] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:00:35.805588  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:35.806564  1596 raft_consensus.cc:2376] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 9. Re-sending same reply.
W20260709 13:00:35.814291  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 68035.1us
I20260709 13:00:35.815711  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:35.723624  5398 consensus_queue.cc:237] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:35.820577  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:35.827140  1596 raft_consensus.cc:2393] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 9: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:00:35.821874  5573 raft_consensus.cc:493] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:00:35.802903  5036 consensus_meta.cc:287] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.671s	user 0.016s	sys 0.000s
I20260709 13:00:35.785326  5431 raft_consensus.cc:2749] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:00:35.770346  5382 raft_consensus.cc:515] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:35.834954  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:35.835237  5496 leader_election.cc:290] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:35.842648  1569 leader_election.cc:304] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1
W20260709 13:00:35.845824  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:35.849663  1643 leader_election.cc:304] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:35.757578  5428 consensus_queue.cc:260] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:35.831943  5564 raft_consensus.cc:3060] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:00:35.855525  1597 raft_consensus.cc:1275] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
W20260709 13:00:35.854555  2301 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:35.864615  5575 raft_consensus.cc:2804] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:00:35.871644  5390 leader_election.cc:290] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:35.872241  5454 raft_consensus.cc:3055] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:00:35.874800  5470 raft_consensus.cc:2749] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 13:00:35.875790  5563 leader_election.cc:290] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:00:35.924634  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 80042.7us
W20260709 13:00:35.856987  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 130202us
I20260709 13:00:35.925091  1644 leader_election.cc:304] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:00:35.925876  2299 leader_election.cc:304] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: af520b67945d443b901bbeca37cc5701
W20260709 13:00:35.918262  5570 log.cc:927] Time spent T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.065s	user 0.000s	sys 0.001s
I20260709 13:00:35.849442  5560 consensus_queue.cc:1059] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 9.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.071s
I20260709 13:00:35.932513  2330 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd"
dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
mode: GRACEFUL
new_leader_uuid: "af520b67945d443b901bbeca37cc5701"
 from {username='slave'} at 127.0.0.1:35790
I20260709 13:00:35.935169  2330 raft_consensus.cc:606] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Received request to transfer leadership to TS af520b67945d443b901bbeca37cc5701
I20260709 13:00:35.935750  2330 raft_consensus.cc:614] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 13:00:35.839977  5573 raft_consensus.cc:515] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:35.917161  5036 raft_consensus.cc:493] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:35.937582  1643 leader_election.cc:304] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1
W20260709 13:00:35.947182  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 144399us
I20260709 13:00:35.964102  2336 raft_consensus.cc:1240] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 8. Current term is 9. Ops: []
I20260709 13:00:35.924482  5454 raft_consensus.cc:740] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:00:35.930740  5390 raft_consensus.cc:493] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:35.930661  5473 raft_consensus.cc:2804] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Leader election won for term 8
W20260709 13:00:35.964167  5568 log.cc:927] Time spent T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.086s	user 0.000s	sys 0.002s
I20260709 13:00:35.985277  5596 raft_consensus.cc:2804] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Leader election won for term 9
W20260709 13:00:35.985417  1571 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:00:36.005911  1571 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:00:36.006453  1571 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:35.988066  5591 raft_consensus.cc:493] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:35.967749  5358 leader_election.cc:290] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 9 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:35.998577  5473 raft_consensus.cc:697] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 8 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:00:35.974874  5390 raft_consensus.cc:515] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:35.953591  5575 raft_consensus.cc:697] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 7 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:00:35.954030  5362 raft_consensus.cc:2804] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260709 13:00:35.968623  5392 consensus_meta.cc:287] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.513s	user 0.015s	sys 0.000s
I20260709 13:00:35.929767  5413 raft_consensus.cc:515] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:00:36.005755  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 64450.6us
I20260709 13:00:36.010344  1601 raft_consensus.cc:2468] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 10.
I20260709 13:00:36.023481  1571 leader_election.cc:304] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:00:35.972450  5572 raft_consensus.cc:697] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 10 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:00:35.970955  5563 raft_consensus.cc:2804] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:00:36.008355  5573 leader_election.cc:290] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:35.951206  5036 raft_consensus.cc:515] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:36.018630  1643 leader_election.cc:304] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
W20260709 13:00:35.987754  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 155393us
I20260709 13:00:36.064975  1569 leader_election.cc:304] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:00:36.025723  2299 leader_election.cc:304] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:00:36.068979  1569 leader_election.cc:304] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:00:36.014863  5545 raft_consensus.cc:2749] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260709 13:00:36.048679  5574 log.cc:927] Time spent T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.083s	user 0.000s	sys 0.000s
I20260709 13:00:36.048942  5390 leader_election.cc:290] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:36.018611  5362 raft_consensus.cc:493] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:36.054468  1602 raft_consensus.cc:2368] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 7. Current term is 8.
I20260709 13:00:36.055688  5413 leader_election.cc:290] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 8 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:36.056551  5590 raft_consensus.cc:493] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:36.022933  5596 raft_consensus.cc:697] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:00:36.139027  5590 raft_consensus.cc:515] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:36.140232  5358 raft_consensus.cc:2749] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260709 13:00:36.141036  5502 consensus_meta.cc:287] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.589s	user 0.013s	sys 0.001s
I20260709 13:00:36.044939  5325 raft_consensus.cc:515] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:36.141660  5590 leader_election.cc:290] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:36.141486  5572 consensus_queue.cc:237] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:36.141629  5502 raft_consensus.cc:515] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:36.072335  5382 leader_election.cc:290] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:36.075163  5473 consensus_queue.cc:237] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:36.046663  5563 raft_consensus.cc:493] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
W20260709 13:00:36.081750  1644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:36.086620  5573 raft_consensus.cc:2749] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260709 13:00:36.087750  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 62091.5us
I20260709 13:00:36.078604  5428 raft_consensus.cc:3060] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:00:36.050604  5214 raft_consensus.cc:3060] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:00:36.026342  5392 raft_consensus.cc:2749] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260709 13:00:36.114790  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 91362.9us
W20260709 13:00:36.116184  2298 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:00:36.117605  5598 log.cc:927] Time spent T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.067s	user 0.000s	sys 0.000s
W20260709 13:00:36.117697  5228 consensus_meta.cc:287] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.513s	user 0.012s	sys 0.000s
W20260709 13:00:36.126545  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 612532us
I20260709 13:00:36.126747  5224 raft_consensus.cc:2749] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:00:36.127017  5597 raft_consensus.cc:2749] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:00:36.127285  5569 raft_consensus.cc:2749] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20260709 13:00:36.128201  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 148316us
I20260709 13:00:36.168824  1643 leader_election.cc:304] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
W20260709 13:00:36.133399  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 64493.1us
I20260709 13:00:36.173662  1569 leader_election.cc:304] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:00:36.174710  1569 leader_election.cc:304] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:00:36.133848  5362 raft_consensus.cc:3060] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:00:36.134799  5581 raft_consensus.cc:2804] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260709 13:00:36.137255  1718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:36.137766  5303 raft_consensus.cc:493] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:36.139732  5596 consensus_queue.cc:237] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:36.062872  2330 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78"
dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
mode: GRACEFUL
new_leader_uuid: "af520b67945d443b901bbeca37cc5701"
 from {username='slave'} at 127.0.0.1:35790
I20260709 13:00:36.186602  2330 raft_consensus.cc:606] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Received request to transfer leadership to TS af520b67945d443b901bbeca37cc5701
I20260709 13:00:36.187134  2330 raft_consensus.cc:614] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 13:00:36.143594  1599 raft_consensus.cc:1240] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 8. Current term is 9. Ops: []
I20260709 13:00:36.144053  1605 raft_consensus.cc:1240] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 8. Current term is 9. Ops: [8.10-8.10]
I20260709 13:00:36.187718  1569 leader_election.cc:304] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:00:36.143589  5325 leader_election.cc:290] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 9 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:36.146552  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:00:36.016472  5591 raft_consensus.cc:515] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:36.157126  5586 log.cc:927] Time spent T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.150s	user 0.001s	sys 0.000s
I20260709 13:00:36.072880  5036 leader_election.cc:290] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:36.171336  1643 leader_election.cc:304] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:36.200358  1643 leader_election.cc:304] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:00:36.201318  5431 raft_consensus.cc:2749] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:00:36.201814  1643 leader_election.cc:304] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:00:36.202075  5431 raft_consensus.cc:2749] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:00:36.179965  5346 raft_consensus.cc:2804] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:00:36.180574  5581 raft_consensus.cc:493] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:00:36.148928  5358 raft_consensus.cc:493] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:36.148106  5563 raft_consensus.cc:3060] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:00:36.184131  5303 raft_consensus.cc:515] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:36.183024  5573 raft_consensus.cc:2804] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:00:36.160357  5462 raft_consensus.cc:515] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:36.192800  2335 raft_consensus.cc:3060] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:00:36.151832  2299 leader_election.cc:400] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 7 pre-election: Vote denied by peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261) with higher term. Message: Invalid argument: T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 for earlier term 7. Current term is 8.
I20260709 13:00:36.244206  2299 leader_election.cc:403] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 7 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:00:36.185644  5575 consensus_queue.cc:237] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:36.164606  5382 consensus_queue.cc:1059] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.034s
W20260709 13:00:36.229887  1646 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:36.203604  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:36.209960  1717 leader_election.cc:304] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:36.191066  1569 leader_election.cc:304] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:00:36.178395  5597 raft_consensus.cc:2804] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260709 13:00:36.267900  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 76916.2us
W20260709 13:00:36.202363  5083 consensus_meta.cc:287] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.653s	user 0.009s	sys 0.003s
I20260709 13:00:36.239743  5581 raft_consensus.cc:3060] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:00:36.204142  5224 raft_consensus.cc:493] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:36.273777  1643 leader_election.cc:304] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:00:36.275650  1569 leader_election.cc:304] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:00:36.266507  5036 raft_consensus.cc:2804] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader election won for term 9
I20260709 13:00:36.266920  5597 raft_consensus.cc:493] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Starting leader election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:00:36.278164  5454 consensus_queue.cc:260] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:36.279549  5601 raft_consensus.cc:2749] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260709 13:00:36.280489  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 128819us
I20260709 13:00:36.207319  5591 leader_election.cc:290] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:00:36.283006  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 73149us
I20260709 13:00:36.283567  5152 raft_consensus.cc:2749] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 13:00:36.206195  5325 raft_consensus.cc:493] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:36.288621  1569 leader_election.cc:304] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:00:36.237305  5346 raft_consensus.cc:493] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:36.292749  1596 raft_consensus.cc:2468] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 7.
I20260709 13:00:36.272480  5506 raft_consensus.cc:2749] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Leader pre-election lost for term 7. Reason: Vote denied by peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261) with higher term. Message: Invalid argument: T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 for earlier term 7. Current term is 8.
I20260709 13:00:36.223745  5578 raft_consensus.cc:2749] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:00:36.267627  5428 raft_consensus.cc:493] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:36.195158  5590 consensus_queue.cc:1048] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 7, Time since last communication: 0.091s
I20260709 13:00:36.267323  5491 raft_consensus.cc:493] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:00:36.210292  5496 consensus_queue.cc:1048] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.080s
I20260709 13:00:36.308488  1597 raft_consensus.cc:1275] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260709 13:00:36.321996  5036 raft_consensus.cc:697] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:00:36.311707  5303 leader_election.cc:290] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:36.312023  1717 leader_election.cc:304] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:36.312897  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:36.314242  1679 raft_consensus.cc:1275] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 9 index: 10. (index mismatch)
I20260709 13:00:36.244068  5573 raft_consensus.cc:493] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Starting leader election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:00:36.340216  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:36.340554  5573 raft_consensus.cc:3060] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:00:36.340698  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:00:36.339375  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:00:36.209143  5482 consensus_queue.cc:1048] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.086s
I20260709 13:00:36.343061  5036 consensus_queue.cc:237] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.6, Last appended by leader: 6, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:36.320190  1598 raft_consensus.cc:1275] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 9 index: 9. (term mismatch)
I20260709 13:00:36.245992  5358 raft_consensus.cc:515] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:36.322228  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:36.345652  1677 raft_consensus.cc:2468] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 6.
I20260709 13:00:36.332201  5454 raft_consensus.cc:3060] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:00:36.210851  5569 consensus_queue.cc:1048] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.084s
I20260709 13:00:36.293426  5392 raft_consensus.cc:2804] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:00:36.346987  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:36.347294  5392 raft_consensus.cc:493] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:36.347793  5392 raft_consensus.cc:3060] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:00:36.294206  5224 raft_consensus.cc:515] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } }
I20260709 13:00:36.348790  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:00:36.324029  1673 raft_consensus.cc:1240] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 9. Current term is 10. Ops: []
I20260709 13:00:36.344372  5482 raft_consensus.cc:2749] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:00:36.353224  5569 consensus_queue.cc:1048] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 13:00:36.354923  1717 leader_election.cc:304] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:00:36.294535  5597 raft_consensus.cc:3060] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:00:36.311345  5362 raft_consensus.cc:515] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:36.358672  1717 leader_election.cc:400] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 7 election: Vote denied by peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445) with higher term. Message: Invalid argument: T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 7. Current term is 8.
I20260709 13:00:36.359026  1717 leader_election.cc:403] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 7 election: Cancelling election due to peer responding with higher term
I20260709 13:00:36.359643  5462 leader_election.cc:290] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 12 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:36.360164  5362 leader_election.cc:290] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 9 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:36.307235  5431 raft_consensus.cc:2749] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 13:00:36.339318  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:36.339766  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:00:36.339936  5608 raft_consensus.cc:493] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:00:36.365929  5608 raft_consensus.cc:515] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:36.366882  5502 leader_election.cc:290] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 9 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:36.344920  1598 pending_rounds.cc:85] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a: Aborting all ops after (but not including) 8
I20260709 13:00:36.368496  1598 pending_rounds.cc:107] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a: Aborting uncommitted NO_OP operation due to leader change: 8.9
I20260709 13:00:36.368841  1598 raft_consensus.cc:2889] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:00:36.369923  5431 raft_consensus.cc:493] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:36.370251  1596 raft_consensus.cc:1240] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 9. Current term is 10. Ops: []
I20260709 13:00:36.370398  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:36.370735  5431 raft_consensus.cc:515] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:36.371348  1598 raft_consensus.cc:2393] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 9: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:00:36.319411  5346 raft_consensus.cc:3060] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:00:36.356997  5224 leader_election.cc:290] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 13:00:36.362735  1644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:36.362731  5482 raft_consensus.cc:2804] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 13:00:36.364058  5152 raft_consensus.cc:3060] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Advancing to term 8
W20260709 13:00:36.326692  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 57281.9us
I20260709 13:00:36.374028  5358 leader_election.cc:290] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:36.373852  5431 leader_election.cc:290] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:36.324838  5325 raft_consensus.cc:515] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:36.337405  5491 raft_consensus.cc:515] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:36.333349  5428 raft_consensus.cc:515] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:36.379444  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:00:36.397809  5503 consensus_meta.cc:287] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.734s	user 0.014s	sys 0.003s
I20260709 13:00:36.324625  5591 raft_consensus.cc:2804] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:00:36.412410  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:36.430552  1676 raft_consensus.cc:3060] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:00:36.412607  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:36.412547  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:00:36.414227  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:36.416527  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:36.418394  1643 leader_election.cc:304] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:36.421974  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:36.438055  1673 raft_consensus.cc:3060] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:00:36.422911  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:00:36.425896  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:00:36.425913  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:36.439934  1599 raft_consensus.cc:2468] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 8.
I20260709 13:00:36.408092  5362 raft_consensus.cc:493] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:36.427270  5491 leader_election.cc:290] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 13:00:36.427655  5582 log.cc:927] Time spent T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.364s	user 0.002s	sys 0.001s
I20260709 13:00:36.456941  2329 raft_consensus.cc:2468] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 7.
I20260709 13:00:36.465652  5482 raft_consensus.cc:697] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 10 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:00:36.467491  5482 consensus_queue.cc:237] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:36.429512  5325 leader_election.cc:290] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:36.475069  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:00:36.480540  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:36.490835  1599 raft_consensus.cc:2393] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 6: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:00:36.444417  5591 raft_consensus.cc:493] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:36.461865  5470 raft_consensus.cc:2749] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:00:36.500835  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:36.508450  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:00:36.462227  5362 raft_consensus.cc:515] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:36.499032  2335 raft_consensus.cc:2468] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 9.
I20260709 13:00:36.545161  1643 leader_election.cc:400] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 9 election: Vote denied by peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445) with higher term. Message: Invalid argument: T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for earlier term 9. Current term is 10.
I20260709 13:00:36.545363  2329 raft_consensus.cc:2468] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 7.
I20260709 13:00:36.501809  5591 raft_consensus.cc:3060] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:00:36.550177  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:00:36.555476  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:00:36.561163  1644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:36.562517  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:36.560869  1745 raft_consensus.cc:2468] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 7.
I20260709 13:00:36.563747  1752 raft_consensus.cc:2393] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 9: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:00:36.566569  2337 raft_consensus.cc:2393] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 8: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:00:36.566599  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:36.568744  1674 raft_consensus.cc:3060] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Advancing to term 9
W20260709 13:00:36.570827  5610 log.cc:927] Time spent T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.070s	user 0.001s	sys 0.000s
I20260709 13:00:36.506837  5346 raft_consensus.cc:515] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:36.592610  1678 raft_consensus.cc:2393] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 10: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:00:36.594101  5346 leader_election.cc:290] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:36.503049  5612 consensus_queue.cc:1059] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.070s
I20260709 13:00:36.621688  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:36.634402  1674 raft_consensus.cc:2468] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 9.
I20260709 13:00:36.657929  2337 raft_consensus.cc:1275] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
W20260709 13:00:36.632246  5586 log.cc:927] Time spent T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.065s	user 0.001s	sys 0.000s
I20260709 13:00:36.672276  2332 raft_consensus.cc:2468] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 9.
I20260709 13:00:36.626803  5496 raft_consensus.cc:3055] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 8 LEADER]: Stepping down as leader of term 8
W20260709 13:00:36.663544  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 183354us
I20260709 13:00:36.696962  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:00:36.705127  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:36.706064  1752 raft_consensus.cc:2468] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 7.
I20260709 13:00:36.714886  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:36.679940  5496 raft_consensus.cc:740] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:00:36.731969  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:36.680101  5563 raft_consensus.cc:515] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:36.754583  1718 leader_election.cc:304] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 13:00:36.755203  5615 log.cc:927] Time spent T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.123s	user 0.001s	sys 0.000s
W20260709 13:00:36.756279  5614 log.cc:927] Time spent T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.080s	user 0.001s	sys 0.000s
I20260709 13:00:36.756933  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:00:36.758001  5599 log.cc:927] Time spent T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.601s	user 0.003s	sys 0.001s
W20260709 13:00:36.759143  5619 log.cc:927] Time spent T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.052s	user 0.001s	sys 0.000s
I20260709 13:00:36.714143  5428 leader_election.cc:290] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:36.760877  5457 raft_consensus.cc:3060] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:00:36.762864  5496 consensus_queue.cc:260] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:36.744011  5617 raft_consensus.cc:493] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:36.793779  2336 raft_consensus.cc:2393] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 9: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:00:36.774670  2331 raft_consensus.cc:2468] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 8.
I20260709 13:00:36.776078  1745 raft_consensus.cc:2468] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 8.
I20260709 13:00:36.718753  5622 raft_consensus.cc:493] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:36.785801  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:36.785878  1718 leader_election.cc:304] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:36.795534  2335 raft_consensus.cc:2468] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 9.
I20260709 13:00:36.799516  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:00:36.800158  2329 raft_consensus.cc:2376] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 9. Re-sending same reply.
I20260709 13:00:36.806409  5622 raft_consensus.cc:515] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:36.810941  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:00:36.790568  5255 raft_consensus.cc:2804] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:00:36.776180  5597 raft_consensus.cc:515] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:36.816784  5362 leader_election.cc:290] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:36.791339  5462 consensus_queue.cc:1048] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.027s
I20260709 13:00:36.791531  5083 consensus_queue.cc:1059] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.027s
I20260709 13:00:36.846346  5597 leader_election.cc:290] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 7 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:36.851959  5083 raft_consensus.cc:3055] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 9 LEADER]: Stepping down as leader of term 9
W20260709 13:00:36.852728  5621 log.cc:927] Time spent T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.108s	user 0.001s	sys 0.000s
W20260709 13:00:36.852073  5628 log.cc:927] Time spent T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.053s	user 0.000s	sys 0.000s
I20260709 13:00:36.852780  5083 raft_consensus.cc:740] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:00:36.857851  5622 leader_election.cc:290] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:36.871158  5152 raft_consensus.cc:2749] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Leader election lost for term 7. Reason: Vote denied by peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445) with higher term. Message: Invalid argument: T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 7. Current term is 8.
I20260709 13:00:36.875998  5152 raft_consensus.cc:493] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:00:36.877805  5152 raft_consensus.cc:515] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:36.882874  5428 raft_consensus.cc:493] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:36.883658  5563 leader_election.cc:290] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 9 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:36.883544  5428 raft_consensus.cc:515] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:36.884097  5152 leader_election.cc:290] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:36.864917  1673 raft_consensus.cc:2468] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 7.
I20260709 13:00:36.885845  5428 leader_election.cc:290] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:36.886379  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
W20260709 13:00:36.886652  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 319335us
I20260709 13:00:36.888525  1643 leader_election.cc:304] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:00:36.888883  1605 raft_consensus.cc:1275] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 13:00:36.867276  1676 raft_consensus.cc:2468] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 8.
I20260709 13:00:36.800601  5617 raft_consensus.cc:515] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:36.890674  5473 consensus_queue.cc:1048] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 13:00:36.892844  5617 leader_election.cc:290] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:36.892624  5473 consensus_queue.cc:1048] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 13:00:36.897315  1717 leader_election.cc:304] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:00:36.902112  5617 raft_consensus.cc:2804] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:00:36.902777  5617 raft_consensus.cc:493] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:00:36.903353  5617 raft_consensus.cc:3060] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:00:36.873529  5482 raft_consensus.cc:2804] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:00:36.904700  5482 raft_consensus.cc:697] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 7 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:00:36.906267  5482 consensus_queue.cc:237] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:36.907838  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:36.908238  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:36.890203  1643 leader_election.cc:304] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:00:36.910869  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:36.911296  1571 leader_election.cc:304] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:00:36.913482  1604 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 735) took 1279 ms. Trace:
I20260709 13:00:36.915153  1748 raft_consensus.cc:1275] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 9 index: 10. (index mismatch)
I20260709 13:00:36.912084  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:36.913074  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:36.919945  2298 leader_election.cc:304] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
W20260709 13:00:36.878265  5454 consensus_meta.cc:287] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.532s	user 0.010s	sys 0.002s
W20260709 13:00:36.890290  5573 consensus_meta.cc:287] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.549s	user 0.010s	sys 0.002s
I20260709 13:00:36.934855  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:36.941179  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:36.942840  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:36.943782  1748 raft_consensus.cc:2468] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 7.
I20260709 13:00:36.945129  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:36.948423  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:36.952967  1571 leader_election.cc:304] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:00:36.814888  5392 raft_consensus.cc:515] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:36.918308  5413 raft_consensus.cc:2749] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:00:36.961658  5597 raft_consensus.cc:3060] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:00:36.917750  5428 raft_consensus.cc:493] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:00:36.961876  5325 raft_consensus.cc:493] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:36.966779  5428 raft_consensus.cc:515] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:36.967118  5325 raft_consensus.cc:515] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:36.969249  5428 leader_election.cc:290] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:36.969885  5325 leader_election.cc:290] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:36.961143  5633 raft_consensus.cc:3060] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Advancing to term 7
W20260709 13:00:36.973644  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 83565.6us
I20260709 13:00:36.975397  1643 leader_election.cc:304] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:00:36.976583  1643 leader_election.cc:304] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:00:36.985070  5591 raft_consensus.cc:515] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:36.990280  1643 leader_election.cc:304] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:36.963097  5454 raft_consensus.cc:2804] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:00:36.959995  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:36.910600  5473 raft_consensus.cc:2804] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:00:36.994356  1596 raft_consensus.cc:1240] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 12. Current term is 13. Ops: [12.10-12.10]
I20260709 13:00:36.997064  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:37.001576  1746 raft_consensus.cc:2468] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 9.
I20260709 13:00:36.998246  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:37.002161  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:37.002918  1675 raft_consensus.cc:2468] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 9.
I20260709 13:00:36.913985  1604 rpcz_store.cc:276] 0709 13:00:35.634117 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:35.634299 (+   182us) service_pool.cc:224] Handling call
0709 13:00:35.638726 (+  4427us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:00:35.723983 (+ 85257us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:00:35.724019 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:00:35.724568 (+   549us) log.cc:844] Serialized 26 byte log entry
0709 13:00:35.918950 (+194382us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:00:35.965503 (+ 46553us) log.cc:844] Serialized 14 byte log entry
0709 13:00:36.426757 (+461254us) spinlock_profiling.cc:243] Waited 7.49 ms on lock 0x7b4400022b4c. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f4262f9bc85 00007f4262f9c6a2 00007f4265d0e1db 00007f4265d09aaf 00007f4265cfd86e 00007f4265cfcbb6 00007f4265cfc7d8 00007f4265d06221 00007f4265d061d9 00007f4265d06169
0709 13:00:36.782890 (+356133us) spinlock_profiling.cc:243] Waited 4.83 ms on lock 0x7b1800423ac0. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f4263057e07 00007f426beb2ab9 00007f426307469b 00007f426307293f 00007f4263072455 00007f426615a060 00007f426616c716 00007f426617d287 00007f426617d208
0709 13:00:36.913018 (+130128us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:36.913197 (+   179us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:00:36.913249 (+    52us) raft_consensus.cc:1685] finished
0709 13:00:36.913261 (+    12us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:36.913401 (+   140us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":7976,"prepare.queue_time_us":299907,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":44,"raft.queue_time_us":23395,"raft.run_cpu_time_us":120,"raft.run_wall_time_us":120,"spinlock_wait_cycles":28340224,"thread_start_us":301621,"threads_started":2,"wal-append.queue_time_us":206423}
I20260709 13:00:36.998618  1569 leader_election.cc:304] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:00:37.000721  1753 raft_consensus.cc:2468] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 10.
I20260709 13:00:37.002653  1751 raft_consensus.cc:2468] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 7.
W20260709 13:00:36.999416  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 213626us
I20260709 13:00:37.006193  1643 leader_election.cc:304] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:00:36.993759  5362 raft_consensus.cc:2804] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Leader election won for term 10
I20260709 13:00:36.955377  5573 raft_consensus.cc:515] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:36.994194  5563 raft_consensus.cc:2804] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:00:37.000311  5470 raft_consensus.cc:2804] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Leader election won for term 6
W20260709 13:00:37.052243  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 132385us
I20260709 13:00:37.046675  5591 leader_election.cc:290] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 7 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:37.002086  5473 raft_consensus.cc:493] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Starting leader election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:00:37.003001  5036 raft_consensus.cc:2749] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 13:00:37.000010  5454 raft_consensus.cc:493] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Starting leader election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:00:36.994131  5634 raft_consensus.cc:493] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:36.993111  5431 consensus_queue.cc:1048] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.035s
I20260709 13:00:37.005220  5358 raft_consensus.cc:493] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:00:37.088256  1747 raft_consensus.cc:2468] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 8.
I20260709 13:00:37.049934  5563 raft_consensus.cc:493] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:00:37.098940  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 90438.2us
I20260709 13:00:37.074290  5473 raft_consensus.cc:3060] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:00:37.100500  1718 leader_election.cc:304] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:37.073701  5255 raft_consensus.cc:697] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 7 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:00:37.085355  5454 raft_consensus.cc:3060] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:00:37.085199  5470 raft_consensus.cc:697] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
W20260709 13:00:37.141714  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 136114us
I20260709 13:00:37.145896  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:37.146765  1677 raft_consensus.cc:3060] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:00:37.090569  5591 raft_consensus.cc:2804] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Leader election won for term 9
I20260709 13:00:37.152947  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:37.154699  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:37.093817  5428 raft_consensus.cc:493] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:00:37.159502  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 59108.7us
I20260709 13:00:37.160400  1718 leader_election.cc:304] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:37.160660  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:37.161444  1676 raft_consensus.cc:2393] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 10: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:00:37.100589  5563 raft_consensus.cc:3060] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:00:37.095027  5475 raft_consensus.cc:493] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:00:37.151300  5609 log.cc:927] Time spent T 00000000000000000000000000000000 P 919f6392d4f44fd68987e2c5e7ec37ca: Append to log took a long time: real 0.055s	user 0.000s	sys 0.001s
I20260709 13:00:37.139645  5036 raft_consensus.cc:2749] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260709 13:00:37.151818  5581 consensus_meta.cc:287] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.875s	user 0.010s	sys 0.007s
I20260709 13:00:37.172412  1673 raft_consensus.cc:2468] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 7.
I20260709 13:00:37.094501  5457 consensus_queue.cc:1048] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.067s
I20260709 13:00:37.166043  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:37.175123  1678 raft_consensus.cc:2393] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 10: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:00:37.091220  5634 raft_consensus.cc:515] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:37.094924  5358 raft_consensus.cc:515] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:37.180611  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:37.184697  1677 raft_consensus.cc:2468] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 10.
I20260709 13:00:37.187258  1676 raft_consensus.cc:1240] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 9. Current term is 10. Ops: [9.6-9.6]
I20260709 13:00:37.171273  5475 raft_consensus.cc:515] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:37.192878  5255 consensus_queue.cc:237] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:37.170122  5591 raft_consensus.cc:697] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:00:37.163101  5635 consensus_queue.cc:1048] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.044s
I20260709 13:00:37.177497  5362 raft_consensus.cc:697] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:00:37.233184  5647 raft_consensus.cc:493] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:37.189738  5325 raft_consensus.cc:493] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:00:37.166924  5428 raft_consensus.cc:515] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:37.237583  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 77282us
I20260709 13:00:37.190871  5643 raft_consensus.cc:493] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:37.248548  5647 raft_consensus.cc:515] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:37.251652  5643 raft_consensus.cc:515] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:37.253340  5428 leader_election.cc:290] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:37.171140  5640 consensus_queue.cc:1048] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.072s
I20260709 13:00:37.237622  5646 raft_consensus.cc:493] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:00:37.255700  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:37.256088  5646 raft_consensus.cc:515] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:37.256656  5358 leader_election.cc:290] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:37.247444  5482 raft_consensus.cc:493] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:37.255335  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:37.226099  5641 raft_consensus.cc:2804] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 13:00:37.179914  5581 raft_consensus.cc:515] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:37.240954  5392 leader_election.cc:290] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:37.270850  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:37.273577  1718 leader_election.cc:304] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:37.256522  1678 raft_consensus.cc:3060] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:00:37.327222  1673 raft_consensus.cc:2368] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 7. Current term is 8.
I20260709 13:00:37.253575  5647 leader_election.cc:290] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:37.333995  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:37.255194  5650 raft_consensus.cc:2804] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:00:37.253938  5428 raft_consensus.cc:493] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:37.248967  5325 raft_consensus.cc:515] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:37.254163  5643 leader_election.cc:290] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:37.344491  2332 raft_consensus.cc:3060] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Advancing to term 7
W20260709 13:00:37.347852  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 74444us
I20260709 13:00:37.351248  1718 leader_election.cc:304] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:37.214356  5470 consensus_queue.cc:237] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:37.355562  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:37.331904  5358 raft_consensus.cc:493] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:37.274734  5482 raft_consensus.cc:515] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:37.373330  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:37.374207  1673 raft_consensus.cc:2376] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 7. Re-sending same reply.
I20260709 13:00:37.335028  5641 raft_consensus.cc:697] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 8 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:00:37.396299  5646 leader_election.cc:290] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:37.395926  5641 consensus_queue.cc:237] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:37.340561  5475 leader_election.cc:290] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:37.343902  5457 raft_consensus.cc:493] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:37.341073  5650 raft_consensus.cc:493] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:37.365447  5635 raft_consensus.cc:2804] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 13:00:37.429908  2332 raft_consensus.cc:2468] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 7.
I20260709 13:00:37.423449  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:37.432402  5634 leader_election.cc:290] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:37.383137  5362 consensus_queue.cc:237] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:37.438095  5642 raft_consensus.cc:493] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:00:37.434386  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 430233us
I20260709 13:00:37.441190  5651 raft_consensus.cc:493] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:37.441498  5642 raft_consensus.cc:515] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:37.442240  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:37.442121  5651 raft_consensus.cc:515] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:37.443245  1673 raft_consensus.cc:2393] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 9: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:00:37.443143  5325 leader_election.cc:290] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:00:37.447268  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 96561.7us
I20260709 13:00:37.350697  5428 raft_consensus.cc:515] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:37.444140  1646 leader_election.cc:304] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 13:00:37.458316  1673 raft_consensus.cc:1275] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260709 13:00:37.423229  5650 raft_consensus.cc:3060] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:00:37.461632  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:37.461126  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:37.463622  1673 raft_consensus.cc:2468] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 9.
I20260709 13:00:37.463792  2329 raft_consensus.cc:2468] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 10.
I20260709 13:00:37.376382  5358 raft_consensus.cc:515] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:37.471361  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:37.436502  5036 raft_consensus.cc:493] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:37.479499  1678 raft_consensus.cc:2468] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 7.
I20260709 13:00:37.447958  5431 raft_consensus.cc:493] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:37.449048  5346 raft_consensus.cc:493] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:37.450906  5634 raft_consensus.cc:2749] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 13:00:37.371256  5591 consensus_queue.cc:237] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:37.434953  5457 raft_consensus.cc:515] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:37.458945  5654 raft_consensus.cc:493] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:00:37.490422  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:37.467274  5635 raft_consensus.cc:697] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 9 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:00:37.484871  5648 raft_consensus.cc:493] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:37.473258  5578 raft_consensus.cc:493] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
W20260709 13:00:37.493530  5617 consensus_meta.cc:287] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.589s	user 0.013s	sys 0.002s
I20260709 13:00:37.516575  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:37.507476  1601 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 737) took 1312 ms. Trace:
I20260709 13:00:37.509909  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:37.487324  5325 raft_consensus.cc:493] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:00:37.517552  1601 rpcz_store.cc:276] 0709 13:00:36.195028 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:36.195193 (+   165us) service_pool.cc:224] Handling call
0709 13:00:36.195606 (+   413us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:00:36.211431 (+ 15825us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:00:36.211479 (+    48us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:00:36.212090 (+   611us) log.cc:844] Serialized 26 byte log entry
0709 13:00:36.251701 (+ 39611us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:00:36.251935 (+   234us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:37.506971 (+1255036us) spinlock_profiling.cc:243] Waited 15.3 ms on lock 0x7b54002995d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:00:37.507029 (+    58us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:00:37.507106 (+    77us) raft_consensus.cc:1685] finished
0709 13:00:37.507121 (+    15us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:37.507398 (+   277us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":11844,"raft.run_cpu_time_us":99,"raft.run_wall_time_us":99,"spinlock_wait_cycles":35186048,"thread_start_us":38819,"threads_started":1,"wal-append.queue_time_us":39401,"wal-append.run_cpu_time_us":2468,"wal-append.run_wall_time_us":1073878}
W20260709 13:00:37.503435  1718 leader_election.cc:336] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Timed out: RequestConsensusVote RPC to 127.0.107.194:46261 timed out after 2.005s (SENT)
I20260709 13:00:37.526891  1678 raft_consensus.cc:2393] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 8: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:00:37.499226  5475 raft_consensus.cc:493] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:37.528864  1673 raft_consensus.cc:1275] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
W20260709 13:00:37.531144  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 58954.8us
W20260709 13:00:37.524848  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 469027us
I20260709 13:00:37.487689  5036 raft_consensus.cc:515] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:37.528452  1678 raft_consensus.cc:1240] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 8. Current term is 9. Ops: []
I20260709 13:00:37.493052  5346 raft_consensus.cc:515] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:37.490084  5431 raft_consensus.cc:515] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:37.524032  5617 raft_consensus.cc:515] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:37.526280  5325 raft_consensus.cc:515] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:37.546378  5661 raft_consensus.cc:2804] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Leader election won for term 10
I20260709 13:00:37.517473  5648 raft_consensus.cc:515] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:37.628810  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 96707us
W20260709 13:00:37.636010  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 100574us
W20260709 13:00:37.621963  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 118537us
I20260709 13:00:37.602784  5581 leader_election.cc:290] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 6 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:00:37.641896  1718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:37.642804  1718 leader_election.cc:304] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:00:37.544469  5475 raft_consensus.cc:515] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:37.508795  5654 raft_consensus.cc:515] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:37.624403  5627 log.cc:927] Time spent T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.180s	user 0.001s	sys 0.000s
I20260709 13:00:37.525656  5578 raft_consensus.cc:515] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:37.656075  5646 raft_consensus.cc:2749] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 13:00:37.661312  5578 leader_election.cc:290] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:37.662707  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:37.668593  1679 raft_consensus.cc:2468] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 8.
W20260709 13:00:37.669265  1679 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52766 (request call id 2428) took 1267 ms (client timeout 1642 ms). Trace:
I20260709 13:00:37.654887  1673 raft_consensus.cc:1275] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 9 index: 10. (index mismatch)
W20260709 13:00:37.669718  1679 rpcz_store.cc:269] 0709 13:00:36.401537 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:36.401737 (+   200us) service_pool.cc:224] Handling call
0709 13:00:37.662506 (+1260769us) spinlock_profiling.cc:243] Waited 2.06 ms on lock 0x7b5800086a98. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f426303db71 00007f426303da22 00007f426bf56c09 00007f426b0b0ee0 00007f426b17a0ab 00007f426b17a131 00007f426b1280a0 00007f426b127984 00007f42646a26d4 00007f42646a2662
0709 13:00:37.669174 (+  6668us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4740352}
I20260709 13:00:37.677969  5617 leader_election.cc:290] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 9 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 13:00:37.678776  5665 log.cc:927] Time spent T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.069s	user 0.000s	sys 0.000s
I20260709 13:00:37.672857  1718 leader_election.cc:304] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:37.677763  1601 raft_consensus.cc:1275] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
W20260709 13:00:37.690707  5659 log.cc:927] Time spent T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.081s	user 0.001s	sys 0.000s
I20260709 13:00:37.690851  5325 leader_election.cc:290] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:00:37.691215  5454 consensus_meta.cc:287] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.526s	user 0.014s	sys 0.000s
I20260709 13:00:37.691907  5454 raft_consensus.cc:515] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:37.692601  5428 leader_election.cc:290] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:00:37.695127  5656 log.cc:927] Time spent T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.202s	user 0.000s	sys 0.001s
I20260709 13:00:37.670773  5255 consensus_queue.cc:1059] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.029s
I20260709 13:00:37.700152  5255 raft_consensus.cc:3055] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:00:37.700376  5358 leader_election.cc:290] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:37.700737  5255 raft_consensus.cc:740] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:00:37.713470  1671 raft_consensus.cc:1240] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 7. Current term is 8. Ops: []
I20260709 13:00:37.704934  1597 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40154 (request call id 2090) took 1344 ms. Trace:
W20260709 13:00:37.718353  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 45612.2us
I20260709 13:00:37.719671  1718 leader_election.cc:304] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:37.674942  5651 leader_election.cc:290] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:00:37.717217  5473 consensus_meta.cc:287] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.573s	user 0.017s	sys 0.000s
I20260709 13:00:37.683171  5482 leader_election.cc:290] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:37.736429  1718 leader_election.cc:304] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:00:37.741010  1675 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57092 (request call id 818) took 1440 ms. Trace:
I20260709 13:00:37.741549  5654 leader_election.cc:290] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:37.741432  1675 rpcz_store.cc:276] 0709 13:00:36.300167 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:36.312606 (+ 12439us) service_pool.cc:224] Handling call
0709 13:00:37.740844 (+1428238us) spinlock_profiling.cc:243] Waited 610 ms on lock 0x7b54001e7dd4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:00:37.740950 (+   106us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1402214144}
I20260709 13:00:37.734702  5346 leader_election.cc:290] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:37.737301  5634 raft_consensus.cc:2804] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:00:37.747210  5634 raft_consensus.cc:697] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 7 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:00:37.748370  5634 consensus_queue.cc:237] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:37.731251  5431 leader_election.cc:290] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:37.751438  2299 leader_election.cc:304] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:37.752769  5663 consensus_queue.cc:1048] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 6, Time since last communication: 0.118s
W20260709 13:00:37.753334  5597 consensus_meta.cc:287] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.789s	user 0.014s	sys 0.001s
I20260709 13:00:37.702207  5255 consensus_queue.cc:260] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:37.753937  5597 raft_consensus.cc:2749] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:00:37.639892  5635 consensus_queue.cc:237] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:37.754696  5560 raft_consensus.cc:2804] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 13:00:37.752776  5475 leader_election.cc:290] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:37.755460  5560 raft_consensus.cc:697] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 9 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:00:37.757386  5658 raft_consensus.cc:3060] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:00:37.757324  5560 consensus_queue.cc:237] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:37.738322  5566 consensus_queue.cc:1059] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 7.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.214s
I20260709 13:00:37.731020  5457 leader_election.cc:290] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:37.754747  5255 raft_consensus.cc:3060] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:00:37.758971  5457 raft_consensus.cc:3055] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:00:37.759361  5457 raft_consensus.cc:740] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:00:37.751682  5454 leader_election.cc:290] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 8 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:37.760804  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:37.761577  1680 raft_consensus.cc:2468] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 7.
I20260709 13:00:37.761389  5457 consensus_queue.cc:260] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 8.10, Last appended by leader: 10, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:37.762887  5457 raft_consensus.cc:3060] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:00:37.763660  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:37.737836  5473 raft_consensus.cc:515] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:37.767514  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:37.767833  1677 raft_consensus.cc:1240] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 8. Current term is 9. Ops: []
I20260709 13:00:37.768748  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:37.769366  1679 raft_consensus.cc:2468] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 8.
I20260709 13:00:37.769891  5473 leader_election.cc:290] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 9 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:37.771282  1680 raft_consensus.cc:1275] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
W20260709 13:00:37.771951  5563 consensus_meta.cc:287] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.595s	user 0.008s	sys 0.007s
I20260709 13:00:37.772383  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:37.770197  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:37.773393  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:37.774165  1673 raft_consensus.cc:2393] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 9: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:00:37.774015  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:37.774637  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:37.775074  1718 leader_election.cc:304] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:00:37.775075  1675 raft_consensus.cc:2468] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 9.
I20260709 13:00:37.776463  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:37.777871  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:37.789870  1675 raft_consensus.cc:2393] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 9: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:00:37.719261  5642 leader_election.cc:290] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:37.796952  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:37.797430  5617 raft_consensus.cc:2749] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 13:00:37.772606  5563 raft_consensus.cc:515] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:37.798300  5641 consensus_queue.cc:1048] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 13:00:37.800235  1676 raft_consensus.cc:1275] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 13:00:37.801663  5563 leader_election.cc:290] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 8 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:37.791836  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:37.802851  1680 raft_consensus.cc:2468] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 10.
I20260709 13:00:37.803519  1676 raft_consensus.cc:1240] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 8. Current term is 9. Ops: []
I20260709 13:00:37.804550  1680 raft_consensus.cc:1275] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 10 index: 8. Preceding OpId from leader: term: 11 index: 9. (index mismatch)
I20260709 13:00:37.807690  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:37.797508  1718 leader_election.cc:304] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:00:37.808743  1673 raft_consensus.cc:2393] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 9: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:00:37.811278  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:37.812696  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:37.791646  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:37.817425  1679 raft_consensus.cc:2368] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 for earlier term 6. Current term is 7.
I20260709 13:00:37.790035  5482 raft_consensus.cc:2749] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:00:37.810000  1718 leader_election.cc:304] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:00:37.821615  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:37.822428  1675 raft_consensus.cc:2468] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 9.
I20260709 13:00:37.836169  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:37.837801  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:37.839529  1679 raft_consensus.cc:1275] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260709 13:00:37.815479  5641 raft_consensus.cc:2804] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Leader pre-election won for term 10
W20260709 13:00:37.853924  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 214980us
I20260709 13:00:37.858232  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:37.856099  5641 raft_consensus.cc:493] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:37.862372  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:37.854934  1718 leader_election.cc:304] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:00:37.859176  1680 raft_consensus.cc:2468] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 9.
I20260709 13:00:37.867802  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:37.868839  1675 raft_consensus.cc:2368] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 7. Current term is 8.
I20260709 13:00:37.856984  1646 leader_election.cc:304] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:00:37.870703  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:37.871789  1680 raft_consensus.cc:2376] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 10. Re-sending same reply.
I20260709 13:00:37.869896  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:37.851404  5635 raft_consensus.cc:2804] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:00:37.831844  5661 raft_consensus.cc:697] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
W20260709 13:00:37.868440  5633 consensus_meta.cc:287] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.898s	user 0.013s	sys 0.005s
I20260709 13:00:37.793270  5650 raft_consensus.cc:515] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:37.863127  5641 raft_consensus.cc:3060] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:00:37.887046  1646 leader_election.cc:304] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
W20260709 13:00:37.902104  5680 log.cc:927] Time spent T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.053s	user 0.000s	sys 0.000s
I20260709 13:00:37.907985  5608 leader_election.cc:290] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:37.913973  5658 raft_consensus.cc:2749] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:00:37.851871  5617 consensus_queue.cc:1048] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.031s
I20260709 13:00:37.925299  5617 consensus_queue.cc:1243] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer 816fadcea8fc4893882f5f9b9f22a03d log is divergent from this leader: its last log entry 6.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260709 13:00:37.927084  5496 raft_consensus.cc:3060] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:00:37.929096  5563 raft_consensus.cc:493] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:00:37.929975  5563 raft_consensus.cc:515] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:37.931676  5661 consensus_queue.cc:237] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 7.9, Last appended by leader: 9, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:37.873458  5482 raft_consensus.cc:2804] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:00:37.935014  5482 raft_consensus.cc:493] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:37.935650  5482 raft_consensus.cc:3060] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:00:37.919574  5362 raft_consensus.cc:2804] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:00:37.938791  5362 raft_consensus.cc:493] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:37.939459  5362 raft_consensus.cc:3060] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:00:37.939718  5563 leader_election.cc:290] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:37.944497  5650 leader_election.cc:290] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 9 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:37.886142  5470 raft_consensus.cc:2804] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:00:37.946007  5470 raft_consensus.cc:493] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:37.946475  5470 raft_consensus.cc:3060] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:00:37.892586  5633 raft_consensus.cc:2749] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:00:37.950676  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:37.952092  1680 raft_consensus.cc:3060] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:00:37.953492  5573 leader_election.cc:290] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 8 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:37.890000  5635 raft_consensus.cc:697] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 7 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:00:37.955560  5635 consensus_queue.cc:237] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:37.958922  2299 leader_election.cc:304] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:37.959458  1646 leader_election.cc:304] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:00:37.973870  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:00:37.988533  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 11 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:37.990139  1751 raft_consensus.cc:2410] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for term 11 because replica has last-logged OpId of term: 8 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 7.
W20260709 13:00:37.994819  1718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:00:38.075858  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 105771us
I20260709 13:00:38.077867  2299 leader_election.cc:400] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 6 election: Vote denied by peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261) with higher term. Message: Invalid argument: T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 for earlier term 6. Current term is 7.
I20260709 13:00:38.078564  2299 leader_election.cc:403] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 6 election: Cancelling election due to peer responding with higher term
I20260709 13:00:38.115435  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:00:38.123739  1753 raft_consensus.cc:1275] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260709 13:00:38.123776  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:00:38.146184  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 68620.2us
I20260709 13:00:38.147431  2299 leader_election.cc:304] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
W20260709 13:00:38.210249  1718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:38.212821  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:38.213305  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:38.215181  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:38.218906  1752 raft_consensus.cc:1275] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20260709 13:00:37.971704  5580 raft_consensus.cc:2804] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:00:38.145191  1750 raft_consensus.cc:2468] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 11.
I20260709 13:00:37.972041  5642 raft_consensus.cc:2749] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 13:00:38.225274  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:38.119518  5563 consensus_queue.cc:1048] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.125s
I20260709 13:00:37.971602  5566 consensus_queue.cc:1048] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.156s
I20260709 13:00:37.975597  5622 raft_consensus.cc:493] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:38.112119  5650 raft_consensus.cc:493] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:38.126010  5560 raft_consensus.cc:493] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:38.117547  5654 consensus_queue.cc:1059] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 8.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.130s
W20260709 13:00:38.260274  1719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:38.235723  1680 raft_consensus.cc:2468] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 8.
I20260709 13:00:38.240818  5580 raft_consensus.cc:493] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:38.328372  5580 raft_consensus.cc:3060] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:00:38.330627  5650 raft_consensus.cc:515] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:38.240774  5454 raft_consensus.cc:493] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:00:38.371867  2334 raft_consensus.cc:2468] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 9.
I20260709 13:00:38.250531  5563 consensus_queue.cc:1243] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer 816fadcea8fc4893882f5f9b9f22a03d log is divergent from this leader: its last log entry 10.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20260709 13:00:38.382197  5641 raft_consensus.cc:515] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:38.384078  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:38.390344  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:00:38.260475  5658 raft_consensus.cc:493] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:00:38.254910  5642 raft_consensus.cc:2749] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Leader election lost for term 6. Reason: Vote denied by peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261) with higher term. Message: Invalid argument: T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 for earlier term 6. Current term is 7.
I20260709 13:00:38.259155  5566 raft_consensus.cc:493] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:00:38.335114  5663 raft_consensus.cc:2804] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:00:38.386540  1646 leader_election.cc:304] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:00:38.369220  5670 raft_consensus.cc:3055] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:00:38.403436  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:38.413182  1746 raft_consensus.cc:1275] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 11 index: 9. (index mismatch)
I20260709 13:00:38.419178  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:38.278970  5622 raft_consensus.cc:515] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:38.423924  1679 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.528s	user 0.055s	sys 0.163s
W20260709 13:00:38.424501  1679 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.528s	user 0.056s	sys 0.163s
I20260709 13:00:38.425020  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:38.432477  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:38.431332  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:38.433502  1752 raft_consensus.cc:2468] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 6.
I20260709 13:00:38.434625  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:38.437922  1745 raft_consensus.cc:1240] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 9. Current term is 10. Ops: []
I20260709 13:00:38.368656  5675 consensus_queue.cc:1048] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.051s
I20260709 13:00:38.440665  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:38.441612  1748 raft_consensus.cc:3060] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:00:38.360610  5560 raft_consensus.cc:515] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:38.405422  5663 raft_consensus.cc:493] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Starting leader election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:00:38.408156  5670 raft_consensus.cc:740] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:00:38.447388  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:38.451818  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:38.452582  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:38.393050  5454 raft_consensus.cc:515] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:38.457353  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:38.390808  5676 consensus_queue.cc:1048] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.039s
I20260709 13:00:38.463249  2331 raft_consensus.cc:1240] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 11. Current term is 12. Ops: [11.10-11.10]
I20260709 13:00:38.459918  2333 raft_consensus.cc:2468] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 10.
W20260709 13:00:38.468216  2329 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.555s	user 0.039s	sys 0.067s
I20260709 13:00:38.436787  5681 raft_consensus.cc:2804] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:00:38.429529  5642 raft_consensus.cc:493] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:38.404757  5658 raft_consensus.cc:515] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:38.449543  5663 raft_consensus.cc:3060] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Advancing to term 10
W20260709 13:00:38.469668  2329 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.556s	user 0.039s	sys 0.067s
I20260709 13:00:38.408366  5566 raft_consensus.cc:515] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:38.482247  5454 leader_election.cc:290] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:00:38.482679  1718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:38.446805  5654 raft_consensus.cc:493] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:00:38.483487  1676 pending_rounds.cc:85] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d: Aborting all ops after (but not including) 4
I20260709 13:00:38.497220  1676 pending_rounds.cc:107] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d: Aborting uncommitted NO_OP operation due to leader change: 6.5
I20260709 13:00:38.497617  1676 raft_consensus.cc:2889] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:00:38.475203  5681 raft_consensus.cc:493] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:38.536257  5670 consensus_queue.cc:260] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:38.544425  5622 leader_election.cc:290] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:38.498230  5675 raft_consensus.cc:493] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:00:38.546823  5560 leader_election.cc:290] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:00:38.547302  5685 log.cc:927] Time spent T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.111s	user 0.000s	sys 0.000s
I20260709 13:00:38.556926  5681 raft_consensus.cc:3060] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:00:38.482739  5642 raft_consensus.cc:515] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:38.569370  5650 leader_election.cc:290] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:00:38.574397  5457 consensus_meta.cc:287] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.809s	user 0.018s	sys 0.000s
I20260709 13:00:38.505383  5654 raft_consensus.cc:515] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:38.580436  5470 consensus_meta.cc:287] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.633s	user 0.016s	sys 0.000s
I20260709 13:00:38.576633  5670 raft_consensus.cc:3060] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Advancing to term 9
W20260709 13:00:38.580857  5496 consensus_meta.cc:287] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.653s	user 0.011s	sys 0.005s
I20260709 13:00:38.595940  1754 raft_consensus.cc:2410] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for term 9 because replica has last-logged OpId of term: 8 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 5.
I20260709 13:00:38.594941  5470 raft_consensus.cc:515] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:38.601612  2330 raft_consensus.cc:2468] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 7.
I20260709 13:00:38.568576  5675 raft_consensus.cc:515] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:38.604411  5641 leader_election.cc:290] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 10 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:38.628774  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:38.641497  1748 raft_consensus.cc:2468] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 8.
I20260709 13:00:38.644400  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:00:38.645071  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 497680us
I20260709 13:00:38.646723  2299 leader_election.cc:304] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:00:38.628957  5470 leader_election.cc:290] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 8 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:00:38.643572  5482 consensus_meta.cc:287] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.707s	user 0.009s	sys 0.009s
I20260709 13:00:38.638268  5654 leader_election.cc:290] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:38.641074  5658 leader_election.cc:290] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:38.657265  5482 raft_consensus.cc:515] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:38.677224  5482 leader_election.cc:290] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:38.645777  5566 leader_election.cc:290] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:38.662668  5457 raft_consensus.cc:2804] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:00:38.678098  5457 raft_consensus.cc:493] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:38.678570  5457 raft_consensus.cc:3060] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:00:38.663453  5036 leader_election.cc:290] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:38.665194  5642 leader_election.cc:290] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:38.667032  5675 leader_election.cc:290] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:00:38.679227  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 52044.4us
I20260709 13:00:38.682582  1718 leader_election.cc:304] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
W20260709 13:00:38.684119  5362 consensus_meta.cc:287] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.744s	user 0.010s	sys 0.008s
I20260709 13:00:38.685784  2329 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39738 (request call id 794) took 1215 ms. Trace:
I20260709 13:00:38.693197  5482 raft_consensus.cc:2804] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 13:00:38.699623  5482 raft_consensus.cc:697] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 9 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:00:38.700721  5482 consensus_queue.cc:237] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:38.704547  5690 consensus_queue.cc:1048] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 13:00:38.705080  2337 raft_consensus.cc:1275] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 9 index: 10. (index mismatch)
I20260709 13:00:38.704577  2336 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39738 (request call id 774) took 2376 ms. Trace:
I20260709 13:00:38.708262  2301 leader_election.cc:304] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
W20260709 13:00:38.711875  1718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:38.708191  2336 rpcz_store.cc:276] 0709 13:00:36.328245 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:36.328449 (+   204us) service_pool.cc:224] Handling call
0709 13:00:38.704334 (+2375885us) spinlock_profiling.cc:243] Waited 444 ms on lock 0x7b5400567f54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:00:38.704493 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1020903168}
I20260709 13:00:38.716104  1751 raft_consensus.cc:1240] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 8. Current term is 9. Ops: []
I20260709 13:00:38.691695  5648 leader_election.cc:290] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:38.726609  1673 raft_consensus.cc:1240] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 LEADER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 9. Current term is 10. Ops: [9.10-9.10]
I20260709 13:00:38.709964  5362 raft_consensus.cc:515] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:38.734339  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:38.686010  5580 raft_consensus.cc:515] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:38.736589  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:38.714704  5692 consensus_queue.cc:1048] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 6, Time since last communication: 0.013s
I20260709 13:00:38.744804  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:38.722090  5642 raft_consensus.cc:2804] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:00:38.747555  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:38.751633  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:38.749352  5642 raft_consensus.cc:493] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:00:38.768970  1753 raft_consensus.cc:1240] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 9. Current term is 10. Ops: [9.10-9.10]
I20260709 13:00:38.772601  5580 leader_election.cc:290] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:38.773542  5362 leader_election.cc:290] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 9 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:38.775380  5642 raft_consensus.cc:3060] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:00:38.777906  2336 raft_consensus.cc:2468] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 9.
I20260709 13:00:38.772329  5681 raft_consensus.cc:515] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:38.781569  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:38.782404  5681 leader_election.cc:290] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 12 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:38.783478  1677 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52778 (request call id 1999) took 1532 ms. Trace:
I20260709 13:00:38.795313  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:38.783917  1677 rpcz_store.cc:276] 0709 13:00:37.251446 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:37.257218 (+  5772us) service_pool.cc:224] Handling call
0709 13:00:37.257547 (+   329us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:37.768311 (+510764us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:38.783203 (+1014892us) spinlock_profiling.cc:243] Waited 510 ms on lock 0x7b54000d8754. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:00:38.783406 (+   203us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1172916608}
I20260709 13:00:38.796612  1675 raft_consensus.cc:2468] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 8.
I20260709 13:00:38.800642  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:38.808923  1676 raft_consensus.cc:1275] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260709 13:00:38.810501  1676 raft_consensus.cc:1240] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 9. Current term is 10. Ops: []
I20260709 13:00:38.819669  1677 raft_consensus.cc:1240] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 11. Current term is 12. Ops: [9.8-11.9]
I20260709 13:00:38.815363  5457 raft_consensus.cc:515] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:38.830406  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:38.831926  1675 raft_consensus.cc:2468] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 7.
I20260709 13:00:38.827906  5675 raft_consensus.cc:493] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:38.801120  5560 raft_consensus.cc:493] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:38.834580  5675 raft_consensus.cc:515] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:38.835008  5560 raft_consensus.cc:515] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:38.836699  5663 raft_consensus.cc:515] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:38.837424  5675 leader_election.cc:290] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:38.801370  5670 raft_consensus.cc:493] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:38.875474  5563 consensus_queue.cc:1048] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 13:00:38.877270  2301 leader_election.cc:304] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:00:38.880868  5673 raft_consensus.cc:2749] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 13:00:38.845978  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:38.848297  5457 leader_election.cc:290] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 10 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:38.849069  5658 raft_consensus.cc:493] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:38.850224  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:38.894460  1675 raft_consensus.cc:3060] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:00:38.851620  5663 leader_election.cc:290] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 10 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:38.852716  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:38.853842  5642 raft_consensus.cc:515] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:38.816751  5690 raft_consensus.cc:493] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:00:38.857385  2332 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39738 (request call id 786) took 2172 ms (client timeout 1691 ms). Trace:
I20260709 13:00:38.864993  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:38.862125  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:38.887862  1752 raft_consensus.cc:2393] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 10: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:00:38.889278  5362 consensus_queue.cc:1059] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 8.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 13:00:38.896795  5641 raft_consensus.cc:493] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:38.897535  1672 raft_consensus.cc:1275] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
W20260709 13:00:38.899725  1719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:38.901770  1749 raft_consensus.cc:3060] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:00:38.838397  5560 leader_election.cc:290] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:00:38.900159  2332 rpcz_store.cc:269] 0709 13:00:36.685121 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:36.696442 (+ 11321us) service_pool.cc:224] Handling call
0709 13:00:38.857065 (+2160623us) spinlock_profiling.cc:243] Waited 646 ms on lock 0x7b5400551754. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:00:38.857261 (+   196us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1486642560}
I20260709 13:00:38.946805  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:38.909402  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:38.908622  5663 raft_consensus.cc:493] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:00:38.962394  2298 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:38.947916  5641 raft_consensus.cc:515] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:38.976507  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 99273.1us
I20260709 13:00:38.893514  5670 raft_consensus.cc:515] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:38.979111  5560 consensus_queue.cc:1059] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 13:00:38.980199  2301 leader_election.cc:304] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:00:38.909799  5658 raft_consensus.cc:515] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:38.908432  5690 raft_consensus.cc:515] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:38.995119  5560 consensus_queue.cc:1059] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 8.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 13:00:38.997295  5673 raft_consensus.cc:3055] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:00:38.997426  5690 leader_election.cc:290] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:38.997766  5673 raft_consensus.cc:740] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:00:38.998955  5673 consensus_queue.cc:260] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 9.10, Last appended by leader: 10, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:39.000254  5673 raft_consensus.cc:3060] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:00:39.000468  5641 leader_election.cc:290] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:39.001930  1750 raft_consensus.cc:1275] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 7 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20260709 13:00:38.993853  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:39.002736  1675 raft_consensus.cc:2468] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 10.
I20260709 13:00:39.008396  5712 raft_consensus.cc:493] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:39.009179  5658 leader_election.cc:290] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:38.993181  5563 raft_consensus.cc:2804] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 13:00:39.009303  5712 raft_consensus.cc:515] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:39.010267  5563 raft_consensus.cc:697] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 8 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:00:39.011509  5563 consensus_queue.cc:237] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.6, Last appended by leader: 6, Current term: 8, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:39.021209  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 76967.3us
I20260709 13:00:39.023217  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:39.024076  1603 raft_consensus.cc:2468] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 7.
I20260709 13:00:39.024005  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:39.026732  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:39.026846  5642 leader_election.cc:290] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 7 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:39.028071  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:39.029768  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:39.031273  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:39.032486  1603 raft_consensus.cc:2468] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 9.
I20260709 13:00:39.032850  1675 raft_consensus.cc:1275] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 13:00:39.033555  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:00:39.038774  5638 log.cc:927] Time spent T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 1.903s	user 0.004s	sys 0.003s
I20260709 13:00:39.034210  1603 raft_consensus.cc:2468] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 7.
I20260709 13:00:39.044615  5641 raft_consensus.cc:493] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:39.045315  5641 raft_consensus.cc:515] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:39.047801  5641 leader_election.cc:290] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:37.714156  1597 rpcz_store.cc:276] 0709 13:00:36.360999 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:36.378978 (+ 17979us) service_pool.cc:224] Handling call
0709 13:00:36.405848 (+ 26870us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:00:36.406227 (+   379us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:00:36.406270 (+    43us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:00:36.411705 (+  5435us) log.cc:844] Serialized 26 byte log entry
0709 13:00:36.648272 (+236567us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:00:36.702842 (+ 54570us) log.cc:844] Serialized 14 byte log entry
0709 13:00:36.703110 (+   268us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:36.703215 (+   105us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:00:36.974022 (+270807us) spinlock_profiling.cc:243] Waited 28.9 ms on lock 0x7b180033aae0. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f4263075522 00007f42630777c1 00007f4263077779 00007f4263077709 00007f42630776d1 00007f42630769cd 000000000038f354 000000000038f199 00007f426305883a
0709 13:00:37.381349 (+407327us) spinlock_profiling.cc:243] Waited 3.45 ms on lock 0x7b180033aae0. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f42630753f3 00007f42630777c1 00007f4263077779 00007f4263077709 00007f42630776d1 00007f42630769cd 000000000038f354 000000000038f199 00007f426305883a
0709 13:00:37.704630 (+323281us) raft_consensus.cc:1685] finished
0709 13:00:37.704662 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:37.704852 (+   190us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":44937,"prepare.run_cpu_time_us":79,"prepare.run_wall_time_us":1702,"raft.queue_time_us":1655,"raft.run_cpu_time_us":147,"raft.run_wall_time_us":4121,"spinlock_wait_cycles":74287360,"thread_start_us":218494,"threads_started":1,"wal-append.queue_time_us":273363}
I20260709 13:00:39.055351  5670 leader_election.cc:290] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:39.070849  1749 raft_consensus.cc:2468] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 9.
I20260709 13:00:38.959532  5663 raft_consensus.cc:515] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:39.072662  5083 consensus_queue.cc:260] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.10, Last appended by leader: 10, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:39.074918  2332 raft_consensus.cc:2393] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 8: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:00:39.097986  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:39.098989  1597 raft_consensus.cc:2468] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 7.
I20260709 13:00:39.100205  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:39.100916  1597 raft_consensus.cc:2468] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 9.
I20260709 13:00:39.101955  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:39.104527  1597 raft_consensus.cc:2468] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 7.
I20260709 13:00:39.047636  5712 leader_election.cc:290] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:39.129849  5681 raft_consensus.cc:493] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:39.130733  5681 raft_consensus.cc:515] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:39.133960  5681 leader_election.cc:290] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:39.137897  2337 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39738 (request call id 795) took 1285 ms. Trace:
I20260709 13:00:39.138413  2337 rpcz_store.cc:276] 0709 13:00:37.852970 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:37.853135 (+   165us) service_pool.cc:224] Handling call
0709 13:00:38.704667 (+851532us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:38.705471 (+   804us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:39.137634 (+432163us) spinlock_profiling.cc:243] Waited 843 ms on lock 0x7b5400567f50. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661618ad 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:00:39.137815 (+   181us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1937842432}
I20260709 13:00:39.144326  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:39.146099  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:00:39.146924  2337 raft_consensus.cc:3060] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:00:39.158967  2337 raft_consensus.cc:2468] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 11.
I20260709 13:00:39.160291  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:39.162562  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:00:39.163848  2336 raft_consensus.cc:2410] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for term 10 because replica has last-logged OpId of term: 9 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 9.
W20260709 13:00:39.164976  2298 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:39.164988  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:00:39.166121  2336 raft_consensus.cc:2393] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 9: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:00:39.167570  2336 raft_consensus.cc:1275] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 7 index: 9. (index mismatch)
I20260709 13:00:39.170307  2336 raft_consensus.cc:1275] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 7 index: 9. (index mismatch)
I20260709 13:00:39.177378  2298 leader_election.cc:304] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:00:39.178779  5671 raft_consensus.cc:2804] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:00:39.179304  5671 raft_consensus.cc:493] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:39.179710  5671 raft_consensus.cc:3060] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:00:39.187391  2298 leader_election.cc:304] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:00:39.189007  5673 raft_consensus.cc:2804] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:00:39.189553  5673 raft_consensus.cc:493] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Starting leader election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:00:39.190050  5673 raft_consensus.cc:3060] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:00:39.195631  2298 leader_election.cc:304] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:00:39.204535  5670 raft_consensus.cc:2804] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:00:39.205123  5670 raft_consensus.cc:493] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:39.205543  5670 raft_consensus.cc:3060] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:00:39.214967  2298 leader_election.cc:304] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:00:39.221138  2298 leader_election.cc:304] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:00:39.222474  2298 leader_election.cc:304] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:00:39.223886  5711 raft_consensus.cc:2804] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:00:39.224392  5711 raft_consensus.cc:493] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:39.224732  5711 raft_consensus.cc:3060] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:00:39.235076  5671 raft_consensus.cc:515] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:39.238062  5671 leader_election.cc:290] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 8 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:39.240549  5709 raft_consensus.cc:2804] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:00:39.267004  5709 raft_consensus.cc:493] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:39.267167  5671 raft_consensus.cc:2804] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:00:39.267534  5709 raft_consensus.cc:3060] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:00:39.267836  5671 raft_consensus.cc:493] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:39.268309  5671 raft_consensus.cc:3060] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:00:39.243965  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:39.270176  1603 raft_consensus.cc:3060] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:00:39.229555  5636 raft_consensus.cc:493] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:00:39.182135  5664 log.cc:927] Time spent T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 1.572s	user 0.006s	sys 0.000s
W20260709 13:00:39.255460  1644 connection.cc:256] RPC call timeout handler was delayed by -1.4718969579999452s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1582s for a response
I20260709 13:00:39.272361  5670 raft_consensus.cc:515] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:39.272185  5636 raft_consensus.cc:515] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:00:39.286223  1644 connection.cc:256] RPC call timeout handler was delayed by -1.2293341059999534s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21610000000000001s for a response
I20260709 13:00:39.290522  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:39.291285  1602 raft_consensus.cc:3060] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:00:39.301213  5473 raft_consensus.cc:493] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:00:39.289286  5636 leader_election.cc:290] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:39.303769  1644 leader_election.cc:304] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:00:39.306564  5681 raft_consensus.cc:2804] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader election won for term 9
I20260709 13:00:39.307156  5681 raft_consensus.cc:697] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 9 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:00:39.307745  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:00:39.302343  5473 raft_consensus.cc:515] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:39.308723  2333 raft_consensus.cc:2368] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 7. Current term is 8.
I20260709 13:00:39.309890  1644 leader_election.cc:304] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:00:39.311625  1603 raft_consensus.cc:2468] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 8.
I20260709 13:00:39.313576  5675 raft_consensus.cc:493] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:39.313957  5580 raft_consensus.cc:2749] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:00:39.314481  5675 raft_consensus.cc:515] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:39.315224  5709 raft_consensus.cc:515] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:39.316749  5675 leader_election.cc:290] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:39.317875  5709 leader_election.cc:290] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 10 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:39.321918  1602 raft_consensus.cc:2468] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 8.
I20260709 13:00:39.323431  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:39.324147  1597 raft_consensus.cc:3060] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:00:39.308133  5670 leader_election.cc:290] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 8 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 13:00:39.411716  2298 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:39.271696  5673 raft_consensus.cc:515] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:39.468369  5473 leader_election.cc:290] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:39.308224  5681 consensus_queue.cc:237] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:39.432659  5636 raft_consensus.cc:493] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:00:39.746163  1644 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.107.196:46625 timed out after 2.040s (SENT) [suppressed 26 similar messages]
W20260709 13:00:39.664844  5721 log.cc:927] Time spent T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.109s	user 0.000s	sys 0.001s
I20260709 13:00:39.540241  5636 raft_consensus.cc:515] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:40.186638  2298 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.772s	user 0.011s	sys 0.007s
W20260709 13:00:40.187479  2298 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.773s	user 0.012s	sys 0.008s
W20260709 13:00:40.187952  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 773596us
W20260709 13:00:40.189548  2298 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:00:40.134982  5711 consensus_meta.cc:287] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.910s	user 0.016s	sys 0.000s
I20260709 13:00:40.366588  5709 raft_consensus.cc:493] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
W20260709 13:00:40.369374  1626 debug-util.cc:398] Leaking SignalData structure 0x7b08002a1520 after lost signal to thread 431
W20260709 13:00:40.370024  1626 debug-util.cc:398] Leaking SignalData structure 0x7b0800352ca0 after lost signal to thread 434
W20260709 13:00:40.519140  1597 consensus_meta.cc:287] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 1.194s	user 0.016s	sys 0.000s
I20260709 13:00:40.519934  1597 raft_consensus.cc:2468] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 10.
I20260709 13:00:40.520809  1597 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36462 (request call id 767) took 1201 ms. Trace:
I20260709 13:00:40.521255  1597 rpcz_store.cc:276] 0709 13:00:39.319125 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:39.319288 (+   163us) service_pool.cc:224] Handling call
0709 13:00:40.520698 (+1201410us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:00:40.546479  1754 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55840 (request call id 2384) took 3634 ms (client timeout 2003 ms). Trace:
W20260709 13:00:40.547299  1754 rpcz_store.cc:269] 0709 13:00:36.912132 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:36.912600 (+   468us) service_pool.cc:224] Handling call
0709 13:00:40.546056 (+3633456us) spinlock_profiling.cc:243] Waited 1.65 s on lock 0x7b540031f454. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:00:40.546331 (+   275us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3793321728}
I20260709 13:00:40.367796  5709 raft_consensus.cc:515] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:40.355340  5711 raft_consensus.cc:515] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:40.552783  5721 log.cc:927] Time spent T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.231s	user 0.000s	sys 0.001s
W20260709 13:00:40.595474  1571 connection.cc:256] RPC call timeout handler was delayed by -2.9980408850000231s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1587s for a response
W20260709 13:00:40.587342  5560 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.257s	user 0.076s	sys 0.148s
W20260709 13:00:40.603031  5560 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.277s	user 0.076s	sys 0.149s
I20260709 13:00:40.606941  5709 leader_election.cc:290] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:40.609550  5711 leader_election.cc:290] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 8 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:40.599807  5636 leader_election.cc:290] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:00:40.601711  1601 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 2.898s	user 0.018s	sys 0.018s
I20260709 13:00:40.619714  5732 raft_consensus.cc:493] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:40.620533  5732 raft_consensus.cc:515] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:40.623833  5732 leader_election.cc:290] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:40.628273  5727 raft_consensus.cc:493] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:40.634223  5727 raft_consensus.cc:515] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:40.637143  5727 leader_election.cc:290] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 13:00:40.598402  2298 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:40.617046  5636 raft_consensus.cc:493] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
W20260709 13:00:40.650935  1601 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 2.947s	user 0.020s	sys 0.019s
I20260709 13:00:40.651360  1598 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 749) took 1830 ms. Trace:
I20260709 13:00:40.659544  5731 raft_consensus.cc:493] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:00:40.652719  1598 rpcz_store.cc:276] 0709 13:00:38.821294 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:40.230991 (+1409697us) service_pool.cc:224] Handling call
0709 13:00:40.371028 (+140037us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:40.371363 (+   335us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 13:00:40.371844 (+   481us) log.cc:844] Serialized 14 byte log entry
0709 13:00:40.650780 (+278936us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:00:40.650864 (+    84us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:00:40.650941 (+    77us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:40.651056 (+   115us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:40.651259 (+   203us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":6554,"prepare.run_cpu_time_us":71,"prepare.run_wall_time_us":77,"thread_start_us":278259,"threads_started":2,"wal-append.queue_time_us":212760}
I20260709 13:00:40.650214  5727 raft_consensus.cc:493] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:40.649950  5732 raft_consensus.cc:493] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:40.669142  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:40.679250  1602 raft_consensus.cc:3060] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Advancing to term 10
W20260709 13:00:40.686498  1571 connection.cc:256] RPC call timeout handler was delayed by -2.5246408849999966s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21130000000000002s for a response
I20260709 13:00:40.686874  1754 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55848 (request call id 2265) took 1792 ms. Trace:
W20260709 13:00:40.687279  1571 connection.cc:256] RPC call timeout handler was delayed by -2.4163904449999905s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20030000000000003s for a response
W20260709 13:00:40.688158  1571 connection.cc:256] RPC call timeout handler was delayed by -2.2505401820000088s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15360000000000001s for a response
I20260709 13:00:40.687479  1754 rpcz_store.cc:276] 0709 13:00:38.894418 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:40.548611 (+1654193us) service_pool.cc:224] Handling call
0709 13:00:40.549105 (+   494us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:40.549386 (+   281us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 13:00:40.549705 (+   319us) log.cc:844] Serialized 14 byte log entry
0709 13:00:40.686375 (+136670us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:00:40.686470 (+    95us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:00:40.686544 (+    74us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:40.686626 (+    82us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:40.686797 (+   171us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":136199,"threads_started":2}
I20260709 13:00:40.660215  5731 raft_consensus.cc:515] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:40.650262  5636 raft_consensus.cc:515] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:00:40.688691  1571 connection.cc:256] RPC call timeout handler was delayed by -2.2442709470000182s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22120000000000004s for a response
W20260709 13:00:40.731555  1571 connection.cc:256] RPC call timeout handler was delayed by -2.052728338999998s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1844s for a response
W20260709 13:00:40.732079  1571 connection.cc:256] RPC call timeout handler was delayed by -2.0331819749999909s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17350000000000002s for a response
W20260709 13:00:40.732359  1571 connection.cc:256] RPC call timeout handler was delayed by -1.9946473979999837s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16060000000000002s for a response
W20260709 13:00:40.732631  1571 connection.cc:256] RPC call timeout handler was delayed by -1.4478530940000383s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21000000000000002s for a response
W20260709 13:00:40.732971  1571 connection.cc:256] RPC call timeout handler was delayed by -1.199113404000002s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2459s for a response
I20260709 13:00:40.715461  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:40.733775  1597 raft_consensus.cc:3060] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:00:40.684808  5733 raft_consensus.cc:493] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:40.736528  5733 raft_consensus.cc:515] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:40.737713  5671 consensus_meta.cc:287] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 1.469s	user 0.001s	sys 0.013s
I20260709 13:00:40.667193  5734 raft_consensus.cc:493] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:00:40.743041  1597 raft_consensus.cc:2468] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 8.
I20260709 13:00:40.742901  5734 raft_consensus.cc:515] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:40.744374  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:40.744979  5636 leader_election.cc:290] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:40.745873  5734 leader_election.cc:290] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:40.745671  5636 raft_consensus.cc:493] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:00:40.745056  1597 raft_consensus.cc:2468] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 8.
I20260709 13:00:40.746577  5636 raft_consensus.cc:515] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:40.748034  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 10 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:40.719527  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:40.755664  1597 raft_consensus.cc:2468] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 9.
I20260709 13:00:40.755978  1598 raft_consensus.cc:3060] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Advancing to term 8
W20260709 13:00:40.756731  1599 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36462 (request call id 754) took 1892 ms (client timeout 1641 ms). Trace:
I20260709 13:00:40.756889  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:00:40.757222  1599 rpcz_store.cc:269] 0709 13:00:38.864462 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:40.756480 (+1892018us) service_pool.cc:211] Skipping call since client already timed out
0709 13:00:40.756624 (+   144us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260709 13:00:40.757654  1597 raft_consensus.cc:2468] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 10.
I20260709 13:00:40.758802  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:40.758266  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:40.759471  1597 raft_consensus.cc:2468] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 5.
I20260709 13:00:40.759922  1599 raft_consensus.cc:2468] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 8.
I20260709 13:00:40.749246  5636 leader_election.cc:290] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:40.761237  1599 raft_consensus.cc:1240] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 7. Current term is 8. Ops: []
I20260709 13:00:40.690946  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:40.764156  1750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55848 (request call id 2264) took 1878 ms. Trace:
I20260709 13:00:40.764629  1750 rpcz_store.cc:276] 0709 13:00:38.866780 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:38.965105 (+ 98325us) service_pool.cc:224] Handling call
0709 13:00:38.965553 (+   448us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:39.002495 (+ 36942us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:40.745323 (+1742828us) spinlock_profiling.cc:243] Waited 35.8 ms on lock 0x7b5400019fd4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:00:40.745537 (+   214us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":82376448}
I20260709 13:00:40.766693  1599 raft_consensus.cc:1240] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 6. Current term is 7. Ops: []
W20260709 13:00:40.768533  5735 log.cc:927] Time spent T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.109s	user 0.001s	sys 0.000s
I20260709 13:00:40.689045  5732 raft_consensus.cc:515] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:40.738198  5671 raft_consensus.cc:515] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:40.771946  5671 leader_election.cc:290] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 9 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:40.766166  5731 leader_election.cc:290] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:40.684476  5727 raft_consensus.cc:515] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:40.775910  1747 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55840 (request call id 2391) took 3521 ms (client timeout 2100 ms). Trace:
W20260709 13:00:40.776417  1747 rpcz_store.cc:269] 0709 13:00:37.254803 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:37.255001 (+   198us) service_pool.cc:224] Handling call
0709 13:00:40.775667 (+3520666us) spinlock_profiling.cc:243] Waited 532 ms on lock 0x7b54003246d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:00:40.775823 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1223253760}
I20260709 13:00:40.777351  5727 leader_election.cc:290] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:40.778486  5732 leader_election.cc:290] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:40.754854  5733 leader_election.cc:290] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:40.684087  5681 raft_consensus.cc:493] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:00:40.779810  5732 raft_consensus.cc:493] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:40.780145  5681 raft_consensus.cc:515] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:40.780501  5732 raft_consensus.cc:515] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:40.781234  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:00:40.781912  2331 raft_consensus.cc:3060] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:00:40.782922  5732 leader_election.cc:290] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:40.785620  1596 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40154 (request call id 2093) took 4423 ms. Trace:
I20260709 13:00:40.786012  1596 rpcz_store.cc:276] 0709 13:00:36.362063 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:36.371408 (+  9345us) service_pool.cc:224] Handling call
0709 13:00:36.371796 (+   388us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:00:37.002285 (+630489us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:40.785259 (+3782974us) spinlock_profiling.cc:243] Waited 622 ms on lock 0x7b54002157d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:00:40.785506 (+   247us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1430859520}
I20260709 13:00:40.787370  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:40.788214  1601 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 746) took 3085 ms. Trace:
I20260709 13:00:40.788568  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:40.789376  1596 raft_consensus.cc:3060] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Advancing to term 9
I20260709 13:00:40.788635  1601 rpcz_store.cc:276] 0709 13:00:37.702408 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:37.702586 (+   178us) service_pool.cc:224] Handling call
0709 13:00:37.703056 (+   470us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:00:37.703550 (+   494us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 13:00:40.651593 (+2948043us) log.cc:844] Serialized 14 byte log entry
0709 13:00:40.752193 (+100600us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:00:40.752865 (+   672us) log.cc:844] Serialized 26 byte log entry
0709 13:00:40.753057 (+   192us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:00:40.753099 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:40.753182 (+    83us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:00:40.787921 (+ 34739us) raft_consensus.cc:1685] finished
0709 13:00:40.787952 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:40.788112 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":519988,"prepare.run_cpu_time_us":126,"prepare.run_wall_time_us":3519,"raft.queue_time_us":10018,"raft.run_cpu_time_us":327,"raft.run_wall_time_us":4517,"thread_start_us":2979303,"threads_started":2,"wal-append.queue_time_us":91607}
I20260709 13:00:40.791327  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:40.791949  1601 raft_consensus.cc:2468] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 9.
I20260709 13:00:40.792824  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:40.793529  1601 raft_consensus.cc:2393] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 9: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:00:40.796788  5681 leader_election.cc:290] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:40.797433  5681 raft_consensus.cc:493] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:40.797974  5681 raft_consensus.cc:515] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:40.799254  1605 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36462 (request call id 752) took 1935 ms (client timeout 1879 ms). Trace:
W20260709 13:00:40.799737  1605 rpcz_store.cc:269] 0709 13:00:38.863632 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:40.799010 (+1935378us) service_pool.cc:211] Skipping call since client already timed out
0709 13:00:40.799149 (+   139us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260709 13:00:40.800679  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:40.801833  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:40.807669  1598 raft_consensus.cc:2468] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 8.
I20260709 13:00:40.807996  2331 raft_consensus.cc:2410] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for term 13 because replica has last-logged OpId of term: 12 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 9.
I20260709 13:00:40.808600  1597 raft_consensus.cc:1275] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
W20260709 13:00:40.811220  2331 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49082 (request call id 837) took 2309 ms (client timeout 2459 ms). Trace:
I20260709 13:00:40.811764  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:40.813004  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:00:40.811676  2331 rpcz_store.cc:269] 0709 13:00:38.501823 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:40.780886 (+2279063us) service_pool.cc:224] Handling call
0709 13:00:40.811152 (+ 30266us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:00:40.798589  1571 leader_election.cc:304] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:00:40.813284  1596 raft_consensus.cc:2468] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 9.
I20260709 13:00:40.814414  5734 raft_consensus.cc:2749] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:00:40.815210  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:40.815850  1596 raft_consensus.cc:2468] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 10.
I20260709 13:00:40.816427  1571 leader_election.cc:304] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:00:40.816722  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:40.817834  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:40.817917  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:40.819054  1604 raft_consensus.cc:2468] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 10.
W20260709 13:00:40.798601  2330 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49082 (request call id 831) took 2371 ms (client timeout 2046 ms). Trace:
W20260709 13:00:40.820022  2330 rpcz_store.cc:269] 0709 13:00:38.427479 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:38.427629 (+   150us) service_pool.cc:224] Handling call
0709 13:00:40.798354 (+2370725us) spinlock_profiling.cc:243] Waited 148 ms on lock 0x7b54005e3b54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:00:40.798495 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":340318592}
I20260709 13:00:40.821316  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:40.822146  1603 raft_consensus.cc:2468] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 11.
I20260709 13:00:40.822309  1571 leader_election.cc:304] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:00:40.823269  5731 raft_consensus.cc:2804] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:00:40.823370  1571 leader_election.cc:304] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:00:40.823771  5731 raft_consensus.cc:493] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:40.824255  5731 raft_consensus.cc:3060] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:00:40.824447  1596 raft_consensus.cc:1275] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260709 13:00:40.826351  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:40.826582  1598 raft_consensus.cc:2468] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 10.
I20260709 13:00:40.827333  5734 raft_consensus.cc:2804] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:00:40.827502  1596 raft_consensus.cc:2393] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 10: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:00:40.829207  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:40.829491  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:40.831075  1596 raft_consensus.cc:2468] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 7.
I20260709 13:00:40.832581  5740 raft_consensus.cc:493] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:00:40.834621  5714 log.cc:927] Time spent T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 1.869s	user 0.013s	sys 0.026s
I20260709 13:00:40.835086  5740 raft_consensus.cc:515] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:40.826654  5739 raft_consensus.cc:2804] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Leader election won for term 8
I20260709 13:00:40.836807  5731 raft_consensus.cc:515] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:40.837574  5740 leader_election.cc:290] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:40.838167  5740 raft_consensus.cc:493] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:00:40.839011  5740 raft_consensus.cc:515] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:40.802399  1605 raft_consensus.cc:2468] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 8.
I20260709 13:00:40.841831  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:40.842476  1605 raft_consensus.cc:2468] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 7.
I20260709 13:00:40.833709  5458 raft_consensus.cc:493] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:40.843360  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:40.844115  1605 raft_consensus.cc:2368] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for earlier term 7. Current term is 8.
I20260709 13:00:40.843607  5458 raft_consensus.cc:515] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:40.844906  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:40.846361  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:40.848008  1596 raft_consensus.cc:1240] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 LEADER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 8. Current term is 9. Ops: [8.6-8.6]
I20260709 13:00:40.849304  1596 raft_consensus.cc:1240] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 7. Current term is 8. Ops: []
I20260709 13:00:40.866739  5458 leader_election.cc:290] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:39.074187  5083 raft_consensus.cc:3060] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:00:40.943470  2328 raft_consensus.cc:1240] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 9. Current term is 10. Ops: []
I20260709 13:00:40.937258  5036 raft_consensus.cc:493] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:40.944286  2328 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49082 (request call id 835) took 2443 ms. Trace:
I20260709 13:00:40.944645  2328 rpcz_store.cc:276] 0709 13:00:38.501027 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:40.942814 (+2441787us) service_pool.cc:224] Handling call
0709 13:00:40.943236 (+   422us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:40.943969 (+   733us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:40.944214 (+   245us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:00:40.944399  5036 raft_consensus.cc:515] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:40.938300  1600 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36462 (request call id 751) took 2075 ms (client timeout 1600 ms). Trace:
W20260709 13:00:40.946704  1600 rpcz_store.cc:269] 0709 13:00:38.863101 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:40.937945 (+2074844us) service_pool.cc:211] Skipping call since client already timed out
0709 13:00:40.938171 (+   226us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260709 13:00:40.948279  5228 raft_consensus.cc:493] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:40.948478  2298 leader_election.cc:304] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:00:40.949193  5228 raft_consensus.cc:515] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:40.949766  2298 leader_election.cc:304] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:00:40.950670  5692 raft_consensus.cc:2804] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 13:00:40.951224  5727 raft_consensus.cc:2804] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 13:00:40.951354  5692 raft_consensus.cc:697] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 8 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:00:40.952011  5727 raft_consensus.cc:697] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 8 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:00:40.953145  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:40.952733  5692 consensus_queue.cc:237] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:40.954069  1601 raft_consensus.cc:3060] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:00:40.953186  5727 consensus_queue.cc:237] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:40.954306  1569 connection.cc:256] RPC call timeout handler was delayed by -3.1550116149999781s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1642s for a response
I20260709 13:00:40.959125  1596 raft_consensus.cc:1240] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 7. Current term is 8. Ops: []
I20260709 13:00:40.956969  2298 leader_election.cc:304] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:00:40.833936  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:40.961699  1603 raft_consensus.cc:3060] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:00:40.793452  1602 raft_consensus.cc:2468] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 10.
I20260709 13:00:40.971374  1599 raft_consensus.cc:1240] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 9. Current term is 10. Ops: []
I20260709 13:00:40.973475  1602 raft_consensus.cc:3060] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:00:40.986893  1598 raft_consensus.cc:2393] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 7: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:00:40.989820  5712 raft_consensus.cc:493] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:40.993491  5712 raft_consensus.cc:515] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:41.001638  1605 raft_consensus.cc:1240] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 8. Current term is 9. Ops: []
I20260709 13:00:41.017048  5748 raft_consensus.cc:493] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:41.017652  5228 leader_election.cc:290] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:00:41.018376  1569 connection.cc:256] RPC call timeout handler was delayed by -3.113611614999968s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1688s for a response
W20260709 13:00:41.018968  1717 connection.cc:256] RPC call timeout handler was delayed by -1.7488710549999951s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1505s for a response
W20260709 13:00:41.019479  1717 connection.cc:256] RPC call timeout handler was delayed by -1.64447105499994s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16210000000000002s for a response
W20260709 13:00:41.019876  1717 connection.cc:256] RPC call timeout handler was delayed by -1.6363710549999837s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.163s for a response
W20260709 13:00:41.020262  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 63400.2us
W20260709 13:00:41.019172  1569 connection.cc:256] RPC call timeout handler was delayed by -3.0308116150000046s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17800000000000002s for a response
W20260709 13:00:41.020996  1717 connection.cc:256] RPC call timeout handler was delayed by -1.4744761369999537s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1777s for a response
I20260709 13:00:41.021382  5640 raft_consensus.cc:2804] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Leader election won for term 10
W20260709 13:00:41.021600  1717 connection.cc:256] RPC call timeout handler was delayed by -1.4608710549999842s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1825s for a response
W20260709 13:00:41.020920  1569 connection.cc:256] RPC call timeout handler was delayed by -2.9912116149999974s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1824s for a response
W20260709 13:00:41.022763  1569 connection.cc:256] RPC call timeout handler was delayed by -2.8391116149999789s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19930000000000003s for a response
I20260709 13:00:41.023449  5739 raft_consensus.cc:697] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:00:41.023770  1749 raft_consensus.cc:1275] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
W20260709 13:00:41.023983  1569 connection.cc:256] RPC call timeout handler was delayed by -2.8033942269999557s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15360000000000001s for a response
W20260709 13:00:41.024417  1569 connection.cc:256] RPC call timeout handler was delayed by -2.7311116149999748s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21130000000000002s for a response
W20260709 13:00:41.024737  1569 connection.cc:256] RPC call timeout handler was delayed by -2.7016942269999618s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16490000000000002s for a response
I20260709 13:00:41.017964  5748 raft_consensus.cc:515] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:41.026722  1717 connection.cc:256] RPC call timeout handler was delayed by -1.4590710549999812s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1827s for a response
W20260709 13:00:41.029034  1717 connection.cc:256] RPC call timeout handler was delayed by -1.1971710549999557s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2118s for a response
W20260709 13:00:41.029361  1717 connection.cc:256] RPC call timeout handler was delayed by -1.0106793329999846s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1526s for a response
I20260709 13:00:41.024808  5739 consensus_queue.cc:237] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:00:41.025080  1569 connection.cc:256] RPC call timeout handler was delayed by -2.6242942269999503s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17350000000000002s for a response
W20260709 13:00:41.031544  1569 connection.cc:256] RPC call timeout handler was delayed by -2.5261942269999622s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1844s for a response
I20260709 13:00:41.032917  1605 raft_consensus.cc:1275] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20260709 13:00:41.046298  1603 raft_consensus.cc:2468] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 9.
I20260709 13:00:41.042829  2298 leader_election.cc:304] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
W20260709 13:00:41.037557  1569 connection.cc:256] RPC call timeout handler was delayed by -2.4573448939999594s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16060000000000002s for a response
W20260709 13:00:41.058493  1569 connection.cc:256] RPC call timeout handler was delayed by -2.4224116149999873s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2456s for a response
W20260709 13:00:41.058799  1569 connection.cc:256] RPC call timeout handler was delayed by -2.3443942269999525s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2046s for a response
W20260709 13:00:41.059075  1569 connection.cc:256] RPC call timeout handler was delayed by -2.3443942269999525s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2046s for a response
W20260709 13:00:41.059365  1569 connection.cc:256] RPC call timeout handler was delayed by -2.2613116149999541s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2635s for a response
W20260709 13:00:41.059638  1569 connection.cc:256] RPC call timeout handler was delayed by -2.1949942269999951s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22120000000000004s for a response
W20260709 13:00:41.059918  1569 connection.cc:256] RPC call timeout handler was delayed by -2.16709422699995s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2243s for a response
W20260709 13:00:41.060213  1569 connection.cc:256] RPC call timeout handler was delayed by -2.16709422699995s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2243s for a response
W20260709 13:00:41.060483  1569 connection.cc:256] RPC call timeout handler was delayed by -2.1506152260000135s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17170000000000002s for a response
W20260709 13:00:41.060761  1569 connection.cc:256] RPC call timeout handler was delayed by -2.1506152260000135s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17170000000000002s for a response
I20260709 13:00:41.057540  2298 leader_election.cc:304] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:00:41.062211  2298 leader_election.cc:304] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:00:41.065456  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:41.066179  1605 raft_consensus.cc:2468] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 6.
I20260709 13:00:41.064627  2298 leader_election.cc:304] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
W20260709 13:00:41.061034  1569 connection.cc:256] RPC call timeout handler was delayed by -2.1109319679999885s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1794s for a response
W20260709 13:00:41.067307  1569 connection.cc:256] RPC call timeout handler was delayed by -2.0917888199999766s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1577s for a response
W20260709 13:00:41.067615  1569 connection.cc:256] RPC call timeout handler was delayed by -2.0768152260000079s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1799s for a response
W20260709 13:00:41.067889  1569 connection.cc:256] RPC call timeout handler was delayed by -2.0768152260000079s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1799s for a response
W20260709 13:00:41.068168  1569 connection.cc:256] RPC call timeout handler was delayed by -1.8365116150000063s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.31070000000000003s for a response
W20260709 13:00:41.068434  1569 connection.cc:256] RPC call timeout handler was delayed by -1.8059152260000246s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21000000000000002s for a response
W20260709 13:00:41.068697  1569 connection.cc:256] RPC call timeout handler was delayed by -1.5470795809999913s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19490000000000002s for a response
W20260709 13:00:41.068969  1569 connection.cc:256] RPC call timeout handler was delayed by -1.5124319679999871s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2459s for a response
W20260709 13:00:41.069240  1569 connection.cc:256] RPC call timeout handler was delayed by -1.5062485569999922s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19490000000000002s for a response
W20260709 13:00:41.069507  1569 connection.cc:256] RPC call timeout handler was delayed by -1.2805390159999774s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2202s for a response
W20260709 13:00:41.069769  1569 connection.cc:256] RPC call timeout handler was delayed by -1.0074770859999944s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3058s for a response
I20260709 13:00:41.072971  1601 raft_consensus.cc:2468] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 9.
I20260709 13:00:41.073766  1603 raft_consensus.cc:1240] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 8. Current term is 9. Ops: [8.6-8.6]
I20260709 13:00:41.076745  1569 leader_election.cc:304] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:00:41.080461  1569 leader_election.cc:304] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:00:41.088805  1569 leader_election.cc:304] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:00:41.031611  5491 raft_consensus.cc:493] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:41.093343  2298 leader_election.cc:304] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:00:41.082553  5572 raft_consensus.cc:2804] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 13:00:41.115952  2298 leader_election.cc:304] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:00:41.087752  5587 raft_consensus.cc:2804] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:00:41.095065  5036 leader_election.cc:290] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:41.095403  1680 raft_consensus.cc:1275] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260709 13:00:41.090750  5750 raft_consensus.cc:493] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:41.033425  5738 raft_consensus.cc:493] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:41.033592  5636 raft_consensus.cc:493] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:00:41.042197  5640 raft_consensus.cc:697] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 10 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:00:41.042317  5747 raft_consensus.cc:493] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:41.109055  1569 leader_election.cc:304] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:00:41.110839  5754 raft_consensus.cc:2749] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260709 13:00:41.118851  2298 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:41.121107  5587 raft_consensus.cc:493] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:41.088850  5681 leader_election.cc:290] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:41.093829  5228 raft_consensus.cc:2804] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:00:41.144834  5572 raft_consensus.cc:697] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 8 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:00:41.087949  5749 raft_consensus.cc:2749] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:00:41.104291  5661 raft_consensus.cc:493] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:00:41.121197  5491 raft_consensus.cc:515] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:41.148211  1604 raft_consensus.cc:1275] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260709 13:00:41.115744  5745 raft_consensus.cc:2749] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:00:41.133778  5575 raft_consensus.cc:2804] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Leader pre-election won for term 12
W20260709 13:00:41.162696  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 53940.7us
W20260709 13:00:41.164842  2298 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:41.094672  1602 raft_consensus.cc:1275] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 9 index: 10. (index mismatch)
I20260709 13:00:41.134099  5711 raft_consensus.cc:2804] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Leader election won for term 10
W20260709 13:00:41.176122  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 97294.5us
I20260709 13:00:41.096166  5732 raft_consensus.cc:2804] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 13:00:41.185405  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:41.185673  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:41.186313  1605 raft_consensus.cc:2468] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 10.
I20260709 13:00:41.187528  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:41.176798  5587 raft_consensus.cc:3060] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:00:41.176093  5746 raft_consensus.cc:2749] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 13:00:41.156189  5750 raft_consensus.cc:515] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:41.160089  5738 raft_consensus.cc:515] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:41.180615  5575 raft_consensus.cc:493] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:41.177913  5740 leader_election.cc:290] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:41.161058  5636 raft_consensus.cc:515] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:41.166267  5747 raft_consensus.cc:515] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:41.187011  5661 raft_consensus.cc:515] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:41.199076  5732 raft_consensus.cc:697] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 9 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:00:41.199594  5572 consensus_queue.cc:237] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:41.190456  5711 raft_consensus.cc:697] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 10 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:00:41.178390  5228 raft_consensus.cc:493] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:00:41.232097  2299 connection.cc:256] RPC call timeout handler was delayed by -1.5711405900000273s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1879s for a response
I20260709 13:00:41.198873  5748 leader_election.cc:290] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:41.241853  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:41.298035  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:00:41.300091  2299 connection.cc:256] RPC call timeout handler was delayed by -1.066846465000026s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16410000000000002s for a response
I20260709 13:00:41.301193  2299 leader_election.cc:304] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:41.238878  5575 raft_consensus.cc:3060] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:00:41.259298  5640 consensus_queue.cc:237] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:41.306136  5711 consensus_queue.cc:237] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:41.238130  5727 raft_consensus.cc:493] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:41.275422  5491 leader_election.cc:290] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:41.279867  5636 leader_election.cc:290] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:41.280223  5750 leader_election.cc:290] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:41.236742  5691 consensus_queue.cc:1048] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.048s
I20260709 13:00:41.306082  5587 raft_consensus.cc:515] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:41.288599  5747 leader_election.cc:290] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:41.242275  1601 raft_consensus.cc:1275] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
W20260709 13:00:41.359920  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 200425us
I20260709 13:00:49.331911  1673 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57092 (request call id 831) took 11553 ms. Trace:
I20260709 13:00:49.332656  1673 rpcz_store.cc:276] 0709 13:00:37.778192 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:37.816338 (+ 38146us) service_pool.cc:224] Handling call
0709 13:00:37.823789 (+  7451us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:00:38.728560 (+904771us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:49.331100 (+10602540us) spinlock_profiling.cc:243] Waited 902 ms on lock 0x7b5400347e54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:00:49.331640 (+   540us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2075307264}
I20260709 13:00:49.333493  1717 leader_election.cc:304] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:00:41.491156  2299 leader_election.cc:304] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 13:00:49.337816  1673 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57092 (request call id 854) took 8131 ms (client timeout 1936 ms). Trace:
W20260709 13:00:49.338470  1673 rpcz_store.cc:269] 0709 13:00:41.206774 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:49.337496 (+8130722us) service_pool.cc:211] Skipping call since client already timed out
0709 13:00:49.337722 (+   226us) inbound_call.cc:177] Queueing failure response
Metrics: {}
W20260709 13:00:49.339812  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 7.84709e+06us
W20260709 13:00:49.340215  1673 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57092 (request call id 856) took 8132 ms (client timeout 2020 ms). Trace:
W20260709 13:00:49.345167  1673 rpcz_store.cc:269] 0709 13:00:41.207419 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:49.339982 (+8132563us) service_pool.cc:211] Skipping call since client already timed out
0709 13:00:49.340138 (+   156us) inbound_call.cc:177] Queueing failure response
Metrics: {}
W20260709 13:00:49.346120  1673 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57092 (request call id 853) took 8139 ms (client timeout 2344 ms). Trace:
W20260709 13:00:49.346493  1673 rpcz_store.cc:269] 0709 13:00:41.206410 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:49.345927 (+8139517us) service_pool.cc:211] Skipping call since client already timed out
0709 13:00:49.346046 (+   119us) inbound_call.cc:177] Queueing failure response
Metrics: {}
W20260709 13:00:49.347294  1673 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57092 (request call id 860) took 8138 ms (client timeout 2612 ms). Trace:
W20260709 13:00:49.347635  1673 rpcz_store.cc:269] 0709 13:00:41.208513 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:49.347116 (+8138603us) service_pool.cc:211] Skipping call since client already timed out
0709 13:00:49.347237 (+   121us) inbound_call.cc:177] Queueing failure response
Metrics: {}
W20260709 13:00:49.348523  1673 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57092 (request call id 857) took 8140 ms (client timeout 2716 ms). Trace:
W20260709 13:00:49.348948  1673 rpcz_store.cc:269] 0709 13:00:41.207688 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:49.348323 (+8140635us) service_pool.cc:211] Skipping call since client already timed out
0709 13:00:49.348463 (+   140us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260709 13:00:48.731997  1749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43156 (request call id 720) took 9845 ms. Trace:
I20260709 13:00:49.350654  1749 rpcz_store.cc:276] 0709 13:00:38.886024 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:41.007247 (+2121223us) service_pool.cc:224] Handling call
0709 13:00:41.007758 (+   511us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:41.024187 (+ 16429us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:48.731246 (+7707059us) spinlock_profiling.cc:243] Waited 14.5 ms on lock 0x7b5400019ad4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:00:48.731723 (+   477us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":33448704}
W20260709 13:00:48.954246  2301 connection.cc:256] RPC call timeout handler was delayed by -8.6651956410000253s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1559s for a response
W20260709 13:00:49.354113  2301 connection.cc:256] RPC call timeout handler was delayed by -8.2826956410000321s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19840000000000002s for a response
W20260709 13:00:49.354576  2301 connection.cc:256] RPC call timeout handler was delayed by -8.2001416310000081s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18660000000000002s for a response
W20260709 13:00:49.354943  2301 connection.cc:256] RPC call timeout handler was delayed by -8.0870411660000059s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20240000000000002s for a response
W20260709 13:00:49.355270  2301 connection.cc:256] RPC call timeout handler was delayed by -7.9673761610000042s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2191s for a response
I20260709 13:00:49.358577  1673 raft_consensus.cc:1275] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20260709 13:00:49.359455  1673 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57092 (request call id 855) took 8152 ms. Trace:
I20260709 13:00:49.359826  1673 rpcz_store.cc:276] 0709 13:00:41.207100 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:49.349705 (+8142605us) service_pool.cc:224] Handling call
0709 13:00:49.350345 (+   640us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:49.359040 (+  8695us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:49.359366 (+   326us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:00:48.986061  2298 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:49.361688  1673 raft_consensus.cc:1275] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 9 index: 10. (index mismatch)
W20260709 13:00:49.362337  2298 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:00:49.363065  2298 connection.cc:256] RPC call timeout handler was delayed by -5.8734388500000136s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1945s for a response
I20260709 13:00:38.697011  2329 rpcz_store.cc:276] 0709 13:00:37.470034 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:37.470182 (+   148us) service_pool.cc:224] Handling call
0709 13:00:37.912568 (+442386us) spinlock_profiling.cc:243] Waited 1.7 ms on lock 0x7b5800178598. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f426303db71 00007f426303da22 00007f426bf56c09 00007f426b0b0ee0 00007f426b17a0ab 00007f426b17a131 00007f426b13bfc0 00007f426b126929 00007f426b1263d2 00007f426469dbd4
0709 13:00:37.912844 (+   276us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:37.913102 (+   258us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 13:00:38.486935 (+573833us) log.cc:844] Serialized 14 byte log entry
0709 13:00:38.685219 (+198284us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:00:38.685307 (+    88us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:00:38.685384 (+    77us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:38.685494 (+   110us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:38.685688 (+   194us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":570001,"prepare.run_cpu_time_us":67,"prepare.run_wall_time_us":68,"raft.queue_time_us":2198,"raft.run_cpu_time_us":225,"raft.run_wall_time_us":4339,"spinlock_wait_cycles":3909504,"thread_start_us":770749,"threads_started":2}
I20260709 13:00:49.047407  1491 maintenance_manager.cc:643] P 919f6392d4f44fd68987e2c5e7ec37ca: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 16.656s	user 0.235s	sys 0.187s Metrics: {"bytes_written":30123,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":361915,"dirs.run_cpu_time_us":802,"dirs.run_wall_time_us":80184,"drs_written":1,"lbm_read_time_us":495,"lbm_reads_lt_1ms":4,"lbm_write_time_us":27483,"lbm_writes_1-10_ms":2,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":31,"peak_mem_usage":0,"reinsert_count":0,"rows_written":63,"thread_start_us":170960,"threads_started":2,"update_count":12,"wal-append.queue_time_us":22200,"wal-append.run_cpu_time_us":502,"wal-append.run_wall_time_us":1000507}
I20260709 13:00:49.088371  1753 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43156 (request call id 717) took 10312 ms. Trace:
I20260709 13:00:49.368234  1753 rpcz_store.cc:276] 0709 13:00:38.775948 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:38.817032 (+ 41084us) service_pool.cc:224] Handling call
0709 13:00:38.817425 (+   393us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 13:00:38.824092 (+  6667us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:00:38.824147 (+    55us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 13:00:38.826338 (+  2191us) log.cc:844] Serialized 78 byte log entry
0709 13:00:39.015940 (+189602us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:00:39.034736 (+ 18796us) log.cc:844] Serialized 14 byte log entry
0709 13:00:39.035026 (+   290us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:39.035153 (+   127us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:00:49.087714 (+10052561us) raft_consensus.cc:1685] finished
0709 13:00:49.087767 (+    53us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:49.088138 (+   371us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":12569,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":50,"raft.queue_time_us":6419,"raft.run_cpu_time_us":367,"raft.run_wall_time_us":6752,"thread_start_us":75051,"threads_started":1,"wal-append.queue_time_us":137379,"wal-append.run_cpu_time_us":4400,"wal-append.run_wall_time_us":1074019}
W20260709 13:00:41.789666  5083 consensus_meta.cc:287] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.845s	user 0.009s	sys 0.012s
I20260709 13:00:47.904412  5712 leader_election.cc:290] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:00:48.720191  5639 log.cc:927] Time spent T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 11.609s	user 0.018s	sys 0.112s
I20260709 13:00:41.354243  5727 raft_consensus.cc:515] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:49.431802  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:49.432888  1602 raft_consensus.cc:3060] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:00:49.436866  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:48.728273  5673 leader_election.cc:290] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 10 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:00:48.692369  1569 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 7.507s	user 0.089s	sys 0.113s
I20260709 13:00:41.237062  1604 raft_consensus.cc:1240] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 9. Current term is 10. Ops: []
I20260709 13:00:49.921905  1604 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40154 (request call id 2130) took 8732 ms. Trace:
I20260709 13:00:48.952437  1603 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 2185) took 7885 ms. Trace:
I20260709 13:00:49.924026  1603 rpcz_store.cc:276] 0709 13:00:41.056704 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:41.056865 (+   161us) service_pool.cc:224] Handling call
0709 13:00:41.073504 (+ 16639us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:00:41.074227 (+   723us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:48.941846 (+7867619us) spinlock_profiling.cc:243] Waited 16.3 ms on lock 0x7b54002a1f50. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661618ad 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:00:48.942265 (+   419us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":37454080}
I20260709 13:00:49.925823  5770 raft_consensus.cc:493] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:49.926777  5770 raft_consensus.cc:515] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:49.112924  1597 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 774) took 8392 ms. Trace:
I20260709 13:00:49.931107  1597 rpcz_store.cc:276] 0709 13:00:40.720511 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:40.760156 (+ 39645us) service_pool.cc:224] Handling call
0709 13:00:40.808221 (+ 48065us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:40.811039 (+  2818us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:49.112302 (+8301263us) spinlock_profiling.cc:243] Waited 47.7 ms on lock 0x7b540028bc50. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661618ad 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:00:49.112643 (+   341us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":109663104}
W20260709 13:00:49.077344  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 92387.6us
W20260709 13:00:49.935026  1643 leader_election.cc:336] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 1.795s (SENT)
I20260709 13:00:49.935547  1643 leader_election.cc:304] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
W20260709 13:00:49.076651  1644 connection.cc:256] RPC call timeout handler was delayed by -7.7311811330000069s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15280000000000002s for a response
W20260709 13:00:49.938405  1644 connection.cc:256] RPC call timeout handler was delayed by -7.3549507349999885s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1567s for a response
W20260709 13:00:49.938823  1644 connection.cc:256] RPC call timeout handler was delayed by -6.5643998909999937s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1716s for a response
W20260709 13:00:49.939113  1644 connection.cc:256] RPC call timeout handler was delayed by -6.5508998909999718s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17310000000000003s for a response
W20260709 13:00:49.939394  1644 connection.cc:256] RPC call timeout handler was delayed by -6.5112998909999646s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1775s for a response
W20260709 13:00:49.939677  1644 connection.cc:256] RPC call timeout handler was delayed by -6.4283097440000461s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16310000000000002s for a response
W20260709 13:00:49.939989  1644 connection.cc:256] RPC call timeout handler was delayed by -6.3195915740000146s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1577s for a response
W20260709 13:00:49.940263  1644 connection.cc:256] RPC call timeout handler was delayed by -6.2601915740000322s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1643s for a response
W20260709 13:00:49.940532  1644 connection.cc:256] RPC call timeout handler was delayed by -6.1197915740000326s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1799s for a response
W20260709 13:00:49.940807  1644 connection.cc:256] RPC call timeout handler was delayed by -6.10179157400006s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1819s for a response
W20260709 13:00:49.941071  1644 connection.cc:256] RPC call timeout handler was delayed by -6.0846915740000327s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18380000000000002s for a response
W20260709 13:00:49.941346  1644 connection.cc:256] RPC call timeout handler was delayed by -5.902891574000023s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20400000000000001s for a response
W20260709 13:00:49.941612  1644 connection.cc:256] RPC call timeout handler was delayed by -5.8389915740000333s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21110000000000004s for a response
W20260709 13:00:49.941895  1644 connection.cc:256] RPC call timeout handler was delayed by -5.6049915740000529s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2371s for a response
W20260709 13:00:49.942173  1644 connection.cc:256] RPC call timeout handler was delayed by -4.8408915740000111s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.32200000000000006s for a response
I20260709 13:00:49.100207  1671 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52778 (request call id 1989) took 11956 ms. Trace:
I20260709 13:00:49.943121  1671 rpcz_store.cc:276] 0709 13:00:37.144068 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:37.144773 (+   705us) service_pool.cc:224] Handling call
0709 13:00:37.145254 (+   481us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:37.714084 (+568830us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:49.099553 (+11385469us) spinlock_profiling.cc:243] Waited 559 ms on lock 0x7b54001f4d54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:00:49.099984 (+   431us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1285923712}
W20260709 13:00:48.967536  1672 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52766 (request call id 2454) took 7893 ms (client timeout 1648 ms). Trace:
W20260709 13:00:49.945776  1672 rpcz_store.cc:269] 0709 13:00:41.073812 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:48.967000 (+7893188us) service_pool.cc:211] Skipping call since client already timed out
0709 13:00:48.967279 (+   279us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260709 13:00:48.952950  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:00:49.949029  1675 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52778 (request call id 2028) took 10908 ms (client timeout 2817 ms). Trace:
W20260709 13:00:49.949399  1675 rpcz_store.cc:269] 0709 13:00:39.040226 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:39.040410 (+   184us) service_pool.cc:224] Handling call
0709 13:00:49.948902 (+10908492us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:00:46.869513  1676 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52766 (request call id 2456) took 5793 ms (client timeout 1749 ms). Trace:
I20260709 13:00:41.298951  1680 raft_consensus.cc:2468] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 10.
W20260709 13:00:49.952937  2298 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:00:49.953261  1680 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57092 (request call id 858) took 8745 ms (client timeout 1567 ms). Trace:
W20260709 13:00:49.953604  2298 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:00:49.953760  1680 rpcz_store.cc:269] 0709 13:00:41.207952 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:41.297670 (+ 89718us) service_pool.cc:224] Handling call
0709 13:00:49.953003 (+8655333us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:00:49.015784  1674 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52778 (request call id 1995) took 11846 ms. Trace:
I20260709 13:00:49.967291  1674 rpcz_store.cc:276] 0709 13:00:37.169500 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:37.169743 (+   243us) service_pool.cc:224] Handling call
0709 13:00:37.185345 (+ 15602us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:37.215448 (+ 30103us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:00:37.215498 (+    50us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:00:37.215559 (+    61us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:00:37.215623 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:49.015143 (+11799520us) spinlock_profiling.cc:243] Waited 12.6 ms on lock 0x7b54001f6654. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:00:49.015221 (+    78us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:49.015534 (+   313us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":29059328}
W20260709 13:00:48.951879  1677 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52766 (request call id 2455) took 7876 ms (client timeout 1783 ms). Trace:
W20260709 13:00:49.969883  1677 rpcz_store.cc:269] 0709 13:00:41.075101 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:48.951400 (+7876299us) service_pool.cc:211] Skipping call since client already timed out
0709 13:00:48.951601 (+   201us) inbound_call.cc:177] Queueing failure response
Metrics: {}
W20260709 13:00:49.257844  1626 debug-util.cc:398] Leaking SignalData structure 0x7b08000475e0 after lost signal to thread 1537
W20260709 13:00:49.974689  1626 debug-util.cc:398] Leaking SignalData structure 0x7b080033c1e0 after lost signal to thread 1629
W20260709 13:00:49.975013  1626 debug-util.cc:398] Leaking SignalData structure 0x7b08003caf00 after lost signal to thread 1630
W20260709 13:00:49.976737  1626 debug-util.cc:398] Leaking SignalData structure 0x7b08003b31a0 after lost signal to thread 1704
W20260709 13:00:49.266480  1571 connection.cc:256] RPC call timeout handler was delayed by -7.2417999629999485s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1577s for a response
I20260709 13:00:49.270123  5774 raft_consensus.cc:493] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:49.272547  1598 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 785) took 7997 ms. Trace:
I20260709 13:00:49.255210  5777 raft_consensus.cc:493] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:49.979847  5774 raft_consensus.cc:515] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:49.980669  5777 raft_consensus.cc:515] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:49.313948  1599 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40154 (request call id 2104) took 8561 ms. Trace:
I20260709 13:00:49.982283  1599 rpcz_store.cc:276] 0709 13:00:40.752525 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:40.769963 (+ 17438us) service_pool.cc:224] Handling call
0709 13:00:40.971099 (+201136us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:40.971861 (+   762us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:49.313458 (+8341597us) spinlock_profiling.cc:243] Waited 199 ms on lock 0x7b54002792d0. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661618ad 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:00:49.313714 (+   256us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":458384128}
I20260709 13:00:49.336720  1601 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40154 (request call id 2131) took 8129 ms. Trace:
I20260709 13:00:49.983793  1601 rpcz_store.cc:276] 0709 13:00:41.189771 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:41.204395 (+ 14624us) service_pool.cc:224] Handling call
0709 13:00:41.241877 (+ 37482us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:49.318414 (+8076537us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:49.318951 (+   537us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:00:49.374172  1492 maintenance_manager.cc:419] P 919f6392d4f44fd68987e2c5e7ec37ca: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 9877 bytes on disk
I20260709 13:00:49.362365  1673 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57092 (request call id 859) took 8154 ms. Trace:
I20260709 13:00:49.406816  5778 raft_consensus.cc:493] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:49.979717  1598 rpcz_store.cc:276] 0709 13:00:41.273012 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:41.293597 (+ 20585us) service_pool.cc:224] Handling call
0709 13:00:49.153691 (+7860094us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:49.154317 (+   626us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 13:00:49.164847 (+ 10530us) log.cc:844] Serialized 14 byte log entry
0709 13:00:49.269389 (+104542us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:00:49.269449 (+    60us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:00:49.269721 (+   272us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:49.269865 (+   144us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:49.270055 (+   190us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":9668,"prepare.run_cpu_time_us":63,"prepare.run_wall_time_us":65,"raft.queue_time_us":1425,"raft.run_cpu_time_us":352,"raft.run_wall_time_us":355,"thread_start_us":113717,"threads_started":3,"wal-append.queue_time_us":108698}
I20260709 13:00:49.987072  5778 raft_consensus.cc:515] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:49.988948  1569 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 8.804s	user 0.089s	sys 0.117s
I20260709 13:00:49.457410  1602 raft_consensus.cc:2468] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 10.
W20260709 13:00:49.131536  1706 maintenance_manager.cc:445] Time spent finding best maintenance operation: real 12.122s	user 0.064s	sys 0.093s
W20260709 13:00:48.704298  1718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:49.343860  1717 leader_election.cc:304] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
W20260709 13:00:49.341921  1719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:00:50.023573  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 679821us
I20260709 13:00:49.341395  2299 leader_election.cc:304] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:00:49.395818  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:00:50.036892  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 101940us
W20260709 13:00:50.037520  1643 connection.cc:256] RPC call timeout handler was delayed by -9.7045964349999849s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21110000000000004s for a response
W20260709 13:00:50.037864  1643 connection.cc:256] RPC call timeout handler was delayed by -9.6055964349999954s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22210000000000002s for a response
I20260709 13:00:50.037773  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
W20260709 13:00:50.038290  1643 connection.cc:256] RPC call timeout handler was delayed by -8.7100964349999686s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.32160000000000005s for a response
I20260709 13:00:50.040176  5790 consensus_queue.cc:1059] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 13:00:49.408969  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:50.044780  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:49.397593  2330 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "dd3f391a609743de886c525de483fc29"
dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
mode: GRACEFUL
new_leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
 from {username='slave'} at 127.0.0.1:35790
I20260709 13:00:50.048607  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:49.399722  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:50.050989  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:00:50.051252  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:50.052178  1753 raft_consensus.cc:2468] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 10.
I20260709 13:00:50.052489  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:00:50.053335  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:50.053763  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:00:49.407950  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:50.055568  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:50.056088  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:50.056389  1746 raft_consensus.cc:2468] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 8.
I20260709 13:00:50.056954  2333 raft_consensus.cc:2468] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 10.
I20260709 13:00:50.058020  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:49.402266  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:49.374797  2337 raft_consensus.cc:2410] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for term 11 because replica has last-logged OpId of term: 9 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 9.
I20260709 13:00:50.067320  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 9 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:50.067881  5654 raft_consensus.cc:493] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:50.068751  5778 leader_election.cc:290] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:50.068598  5654 raft_consensus.cc:515] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:50.069492  2337 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39738 (request call id 806) took 11067 ms (client timeout 2817 ms). Trace:
I20260709 13:00:49.379365  5083 raft_consensus.cc:2804] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:00:50.072770  5654 leader_election.cc:290] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:41.303517  5228 raft_consensus.cc:3060] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:00:50.087684  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:48.511950  5470 raft_consensus.cc:493] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:50.092047  5470 raft_consensus.cc:515] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:42.067418  5491 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.707s	user 0.047s	sys 0.103s
I20260709 13:00:48.953539  5566 raft_consensus.cc:493] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:00:43.749022  5572 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.213s	user 0.008s	sys 0.002s
W20260709 13:00:42.694286  5575 consensus_meta.cc:287] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 1.390s	user 0.017s	sys 0.000s
I20260709 13:00:50.146051  5575 raft_consensus.cc:515] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:41.793613  5587 leader_election.cc:290] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 10 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:49.334987  5634 raft_consensus.cc:2804] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:00:50.149104  5575 leader_election.cc:290] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 12 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:48.699635  5648 raft_consensus.cc:3055] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:00:50.153082  5648 raft_consensus.cc:740] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:00:50.154486  5648 consensus_queue.cc:260] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:50.155691  5648 raft_consensus.cc:3060] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:00:41.258821  5661 leader_election.cc:290] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:48.515287  5663 leader_election.cc:290] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:49.423975  5670 consensus_queue.cc:1048] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 13:00:41.314888  5671 raft_consensus.cc:2749] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260709 13:00:43.974282  5687 log.cc:927] Time spent T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 5.345s	user 0.009s	sys 0.055s
W20260709 13:00:47.855283  5708 log.cc:927] Time spent T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 8.890s	user 0.015s	sys 0.076s
W20260709 13:00:43.716153  5711 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.179s	user 0.004s	sys 0.005s
I20260709 13:00:49.441568  5727 leader_election.cc:290] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:41.257794  5732 consensus_queue.cc:237] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:49.196746  5731 leader_election.cc:290] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 9 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:00:50.179093  5711 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 7.642s	user 0.006s	sys 0.006s
W20260709 13:00:50.179258  5491 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 8.815s	user 0.051s	sys 0.104s
I20260709 13:00:41.273859  5738 leader_election.cc:290] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:48.738190  5734 raft_consensus.cc:493] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:50.181653  5734 raft_consensus.cc:3060] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Advancing to term 11
W20260709 13:00:47.228722  1646 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.107.195:40693 timed out after 2.240s (SENT) [suppressed 9 similar messages]
W20260709 13:00:50.199669  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 2.97068e+06us
I20260709 13:00:49.831732  1751 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55848 (request call id 2255) took 11113 ms. Trace:
I20260709 13:00:49.851223  2335 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39738 (request call id 793) took 12390 ms. Trace:
W20260709 13:00:49.880784  2332 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49082 (request call id 840) took 11299 ms (client timeout 2100 ms). Trace:
I20260709 13:00:50.200699  1751 rpcz_store.cc:276] 0709 13:00:38.718348 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:38.718774 (+   426us) service_pool.cc:224] Handling call
0709 13:00:38.719189 (+   415us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:00:38.724231 (+  5042us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 13:00:49.823745 (+11099514us) spinlock_profiling.cc:243] Waited 4.91 ms on lock 0x7b18003f34c0. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f4263057e07 00007f426beb2ab9 00007f426307469b 00007f426307293f 00007f4263072455 00007f4269b5a3c7 00007f426616c9a7 00007f426617c0bf 00007f426617c038
0709 13:00:49.826786 (+  3041us) log.cc:844] Serialized 14 byte log entry
0709 13:00:49.828084 (+  1298us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:00:49.828947 (+   863us) log.cc:844] Serialized 26 byte log entry
0709 13:00:49.829127 (+   180us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:00:49.829169 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:49.829303 (+   134us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:00:49.831137 (+  1834us) raft_consensus.cc:1685] finished
0709 13:00:49.831281 (+   144us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:49.831488 (+   207us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":766,"prepare.queue_time_us":85800,"prepare.run_cpu_time_us":57,"prepare.run_wall_time_us":58,"raft.queue_time_us":34581,"raft.run_cpu_time_us":271,"raft.run_wall_time_us":2193,"spinlock_wait_cycles":11287808,"thread_start_us":3302,"threads_started":2,"wal-append.queue_time_us":1221}
W20260709 13:00:50.202653  2332 rpcz_store.cc:269] 0709 13:00:38.581527 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:39.030962 (+449435us) service_pool.cc:224] Handling call
0709 13:00:49.880134 (+10849172us) spinlock_profiling.cc:243] Waited 42.8 ms on lock 0x7b54007602d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:00:49.880473 (+   339us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":98460032}
I20260709 13:00:50.203804  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:50.204243  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:00:50.204885  1646 connection.cc:256] RPC call timeout handler was delayed by -4.8265361229999826s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15200000000000002s for a response
W20260709 13:00:50.205396  1646 connection.cc:256] RPC call timeout handler was delayed by -4.784236122999971s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1567s for a response
W20260709 13:00:50.205802  1646 connection.cc:256] RPC call timeout handler was delayed by -4.7213546869999732s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.189s for a response
W20260709 13:00:50.206239  1646 connection.cc:256] RPC call timeout handler was delayed by -4.6021572260000312s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1819s for a response
I20260709 13:00:49.988727  1491 maintenance_manager.cc:643] P 919f6392d4f44fd68987e2c5e7ec37ca: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.003s	user 0.001s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":689,"lbm_reads_lt_1ms":4}
I20260709 13:00:49.414491  5781 raft_consensus.cc:493] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:50.208683  5781 raft_consensus.cc:515] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:50.211459  5781 leader_election.cc:290] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:49.986636  1673 rpcz_store.cc:276] 0709 13:00:41.208256 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:49.360655 (+8152399us) service_pool.cc:224] Handling call
0709 13:00:49.361167 (+   512us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:49.362076 (+   909us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:49.362272 (+   196us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:00:49.992856  5754 raft_consensus.cc:493] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:00:50.212761  5754 raft_consensus.cc:515] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:50.215689  5754 leader_election.cc:290] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:49.922358  1604 rpcz_store.cc:276] 0709 13:00:41.189570 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:41.190258 (+   688us) service_pool.cc:224] Handling call
0709 13:00:41.236853 (+ 46595us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:49.921342 (+8684489us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:49.921674 (+   332us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:00:50.002108  1718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:00:49.996737  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 8.81035e+06us
I20260709 13:00:50.022616  5673 consensus_queue.cc:1048] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 13:00:50.027706  5496 raft_consensus.cc:493] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:50.027798  5482 raft_consensus.cc:2804] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260709 13:00:50.029500  1626 debug-util.cc:398] Leaking SignalData structure 0x7b08003752a0 after lost signal to thread 1778
I20260709 13:00:50.219901  5482 raft_consensus.cc:493] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:00:50.220396  5482 raft_consensus.cc:3060] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Advancing to term 8
W20260709 13:00:50.221668  1626 debug-util.cc:398] Leaking SignalData structure 0x7b0800352f80 after lost signal to thread 2361
W20260709 13:00:50.222088  1626 debug-util.cc:398] Leaking SignalData structure 0x7b0800324be0 after lost signal to thread 2362
I20260709 13:00:50.222772  5496 raft_consensus.cc:515] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:00:50.033191  1644 leader_election.cc:336] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 election: RPC error from VoteRequest() call to peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625): Timed out: RequestConsensusVote RPC to 127.0.107.196:46625 timed out after 1.528s (SENT)
I20260709 13:00:50.224063  1751 raft_consensus.cc:2468] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 6.
I20260709 13:00:50.224408  1644 leader_election.cc:304] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:00:50.034636  5770 leader_election.cc:290] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:00:50.225934  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 206922us
I20260709 13:00:50.042330  5787 raft_consensus.cc:2749] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260709 13:00:50.226748  1644 leader_election.cc:336] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 12 election: RPC error from VoteRequest() call to peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625): Timed out: RequestConsensusVote RPC to 127.0.107.196:46625 timed out after 1.567s (SENT)
I20260709 13:00:50.227030  5787 raft_consensus.cc:2749] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
W20260709 13:00:50.032603  1571 connection.cc:256] RPC call timeout handler was delayed by -7.0464999629999738s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1794s for a response
I20260709 13:00:50.227555  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
W20260709 13:00:50.227957  1571 connection.cc:256] RPC call timeout handler was delayed by -6.7062826439999981s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1648s for a response
W20260709 13:00:50.045217  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 704000us
I20260709 13:00:50.044783  1499 catalog_manager.cc:5697] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 4 to 7, leader changed from 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) to 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194). New cstate: current_term: 7 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } }
W20260709 13:00:50.228363  1644 leader_election.cc:336] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625): Timed out: RequestConsensusVote RPC to 127.0.107.196:46625 timed out after 1.643s (SENT)
W20260709 13:00:50.229454  1644 leader_election.cc:336] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625): Timed out: RequestConsensusVote RPC to 127.0.107.196:46625 timed out after 1.716s (SENT)
W20260709 13:00:50.230223  1644 leader_election.cc:336] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625): Timed out: RequestConsensusVote RPC to 127.0.107.196:46625 timed out after 1.731s (SENT)
W20260709 13:00:50.223906  1626 debug-util.cc:398] Leaking SignalData structure 0x7b080034ffe0 after lost signal to thread 5622
I20260709 13:00:50.229555  1499 catalog_manager.cc:5697] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 4 to 5, leader changed from 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) to af520b67945d443b901bbeca37cc5701 (127.0.107.195). New cstate: current_term: 5 leader_uuid: "af520b67945d443b901bbeca37cc5701" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } }
I20260709 13:00:50.232415  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:50.046993  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:50.047322  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:50.047493  1677 raft_consensus.cc:2468] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 9.
I20260709 13:00:49.403868  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 11 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:50.242233  1752 raft_consensus.cc:2410] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for term 10 because replica has last-logged OpId of term: 9 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 5.
I20260709 13:00:50.242714  5670 consensus_queue.cc:1048] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 13:00:50.243326  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:50.244724  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 9 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:50.245750  1752 raft_consensus.cc:2410] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for term 9 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260709 13:00:50.058862  1746 raft_consensus.cc:1275] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 9 index: 6. (term mismatch)
I20260709 13:00:50.247509  1746 pending_rounds.cc:85] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701: Aborting all ops after (but not including) 5
I20260709 13:00:50.247795  1746 pending_rounds.cc:107] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701: Aborting uncommitted NO_OP operation due to leader change: 8.6
I20260709 13:00:50.248095  1746 raft_consensus.cc:2889] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:00:50.248889  5670 consensus_queue.cc:1059] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 13:00:50.249816  5670 consensus_queue.cc:1243] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer 816fadcea8fc4893882f5f9b9f22a03d log is divergent from this leader: its last log entry 10.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20260709 13:00:50.250129  1746 raft_consensus.cc:1240] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 8. Current term is 9. Ops: []
I20260709 13:00:50.061009  5777 leader_election.cc:290] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:50.061426  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:50.258057  1748 raft_consensus.cc:1275] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 13:00:50.062999  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:50.259590  5673 consensus_queue.cc:1048] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 13:00:50.261234  1747 raft_consensus.cc:1275] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20260709 13:00:50.063520  5774 leader_election.cc:290] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:50.066267  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 9 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:50.070263  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:00:50.069965  2337 rpcz_store.cc:269] 0709 13:00:39.002568 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:39.159941 (+157373us) service_pool.cc:224] Handling call
0709 13:00:50.069230 (+10909289us) spinlock_profiling.cc:243] Waited 10.2 s on lock 0x7b540065e0d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:00:50.069410 (+   180us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":23485090560}
I20260709 13:00:50.072767  5083 raft_consensus.cc:493] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:00:50.083173  5792 consensus_queue.cc:1059] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
W20260709 13:00:50.086977  1500 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35464 (request call id 285) took 39201 ms (client timeout 15000 ms). Trace:
I20260709 13:00:50.091138  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:50.103914  2328 raft_consensus.cc:2468] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 10.
I20260709 13:00:50.111984  1717 leader_election.cc:304] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:00:50.111938  5790 consensus_queue.cc:1059] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 13:00:50.115154  5470 leader_election.cc:290] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:00:50.122489  2358 debug-util.cc:398] Leaking SignalData structure 0x7b08002f5160 after lost signal to thread 431
I20260709 13:00:50.140890  5793 raft_consensus.cc:3055] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:00:50.142277  5566 raft_consensus.cc:515] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:50.150383  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:49.343307  5612 raft_consensus.cc:493] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:50.279227  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:50.279487  5612 raft_consensus.cc:515] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:50.164078  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 83969us
I20260709 13:00:50.170459  1679 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52778 (request call id 2021) took 12316 ms. Trace:
I20260709 13:00:50.280668  1679 rpcz_store.cc:276] 0709 13:00:37.853875 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:37.870360 (+ 16485us) service_pool.cc:224] Handling call
0709 13:00:37.870930 (+   570us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:00:37.871362 (+   432us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:00:37.871394 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:00:37.872014 (+   620us) log.cc:844] Serialized 26 byte log entry
0709 13:00:38.425034 (+553020us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:00:38.425090 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:38.425182 (+    92us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:00:43.973312 (+5548130us) spinlock_profiling.cc:243] Waited 4.32 ms on lock 0x7b4400022b4c. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f4262f9bc85 00007f4262f9c6a2 00007f4265d0e1db 00007f4265d09aaf 00007f4265cfd86e 00007f4265cfcbb6 00007f4265cfc7d8 00007f4265d06221 00007f4265d061d9 00007f4265d06169
0709 13:00:50.169761 (+6196449us) raft_consensus.cc:1685] finished
0709 13:00:50.169806 (+    45us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:50.170193 (+   387us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":3022,"raft.run_cpu_time_us":200,"raft.run_wall_time_us":6443,"spinlock_wait_cycles":9935744,"thread_start_us":528801,"threads_started":1,"wal-append.queue_time_us":730389}
I20260709 13:00:50.151124  5634 raft_consensus.cc:493] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:00:50.205638  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:00:50.178753  5572 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 7.640s	user 0.011s	sys 0.002s
W20260709 13:00:49.951324  1676 rpcz_store.cc:269] 0709 13:00:41.075997 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:46.868623 (+5792626us) service_pool.cc:211] Skipping call since client already timed out
0709 13:00:46.869084 (+   461us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260709 13:00:50.282600  5634 raft_consensus.cc:3060] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:00:50.200789  2335 rpcz_store.cc:276] 0709 13:00:37.460580 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:37.461234 (+   654us) service_pool.cc:224] Handling call
0709 13:00:37.461634 (+   400us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:00:37.520431 (+ 58797us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:00:37.520481 (+    50us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:00:37.528117 (+  7636us) log.cc:844] Serialized 26 byte log entry
0709 13:00:37.709638 (+181521us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:00:37.709705 (+    67us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:49.850352 (+12140647us) spinlock_profiling.cc:243] Waited 58.1 ms on lock 0x7b540077d654. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:00:49.850458 (+   106us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:00:49.850553 (+    95us) raft_consensus.cc:1685] finished
0709 13:00:49.850566 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:49.850969 (+   403us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":2669,"raft.run_cpu_time_us":149,"raft.run_wall_time_us":3311,"spinlock_wait_cycles":133600000,"thread_start_us":107682,"threads_started":1,"wal-append.queue_time_us":257298,"wal-append.run_cpu_time_us":1953,"wal-append.run_wall_time_us":1002029}
I20260709 13:00:50.283838  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:50.284024  5470 consensus_queue.cc:1048] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 13:00:50.285219  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:50.286095  2335 raft_consensus.cc:2393] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 12: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
W20260709 13:00:50.210228  1646 connection.cc:256] RPC call timeout handler was delayed by -3.7953208349999841s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.4765s for a response
W20260709 13:00:50.217361  1718 connection.cc:256] RPC call timeout handler was delayed by -9.2713750219999724s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1775s for a response
I20260709 13:00:50.284138  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:00:50.288348  1718 connection.cc:256] RPC call timeout handler was delayed by -9.2290750220000177s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18220000000000003s for a response
I20260709 13:00:50.288645  1601 raft_consensus.cc:2468] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 9.
W20260709 13:00:50.288950  1718 connection.cc:256] RPC call timeout handler was delayed by -9.2263750220000134s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1825s for a response
W20260709 13:00:50.289362  1718 connection.cc:256] RPC call timeout handler was delayed by -9.047275022000008s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20240000000000002s for a response
W20260709 13:00:50.289744  1718 connection.cc:256] RPC call timeout handler was delayed by -8.9626750219999849s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2118s for a response
I20260709 13:00:50.289870  5661 raft_consensus.cc:493] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:00:50.032341  5791 raft_consensus.cc:493] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:50.226056  5496 leader_election.cc:290] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:50.232319  1499 catalog_manager.cc:5697] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 4 to 7, leader changed from 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) to 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194). New cstate: current_term: 7 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } }
I20260709 13:00:50.045845  5581 raft_consensus.cc:2749] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260709 13:00:50.228330  1571 connection.cc:256] RPC call timeout handler was delayed by -6.6792999629999485s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2202s for a response
I20260709 13:00:50.266566  5083 raft_consensus.cc:3060] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:00:50.060278  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:50.241139  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:50.259848  2299 leader_election.cc:304] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:50.065172  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:50.250448  5670 raft_consensus.cc:3055] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 13:00:50.300135  5670 raft_consensus.cc:740] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
W20260709 13:00:50.273965  2358 debug-util.cc:398] Leaking SignalData structure 0x7b08002f4bc0 after lost signal to thread 434
I20260709 13:00:50.268875  1752 raft_consensus.cc:1275] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260709 13:00:50.277282  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:50.273991  5793 raft_consensus.cc:740] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:00:50.278517  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:50.280012  2331 raft_consensus.cc:2468] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 9.
I20260709 13:00:50.275812  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:50.282066  2329 raft_consensus.cc:2410] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for term 10 because replica has last-logged OpId of term: 8 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 7.
I20260709 13:00:50.169677  5228 raft_consensus.cc:515] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:49.150171  5773 raft_consensus.cc:493] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:50.306318  5670 consensus_queue.cc:260] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.9, Last appended by leader: 9, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:50.306803  5773 raft_consensus.cc:515] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:50.308151  5670 raft_consensus.cc:3060] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:00:50.309705  5773 leader_election.cc:290] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:50.310514  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:50.311439  1750 raft_consensus.cc:2393] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 9: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:00:50.311360  2337 raft_consensus.cc:2468] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 8.
I20260709 13:00:50.312264  2336 raft_consensus.cc:2368] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for earlier term 9. Current term is 10.
I20260709 13:00:50.218942  1569 leader_election.cc:304] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:00:50.304342  5787 raft_consensus.cc:493] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:00:50.276553  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:50.312458  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:50.290969  5791 raft_consensus.cc:515] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:50.290122  1718 connection.cc:256] RPC call timeout handler was delayed by -8.8392668739999749s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2118s for a response
W20260709 13:00:50.296360  1571 connection.cc:256] RPC call timeout handler was delayed by -6.6153826440000216s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17490000000000003s for a response
W20260709 13:00:50.234912  1644 leader_election.cc:336] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625): Timed out: RequestConsensusVote RPC to 127.0.107.196:46625 timed out after 1.775s (SENT)
W20260709 13:00:50.287647  1646 leader_election.cc:336] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 9 election: RPC error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Timed out: RequestConsensusVote RPC to 127.0.107.195:40693 timed out after 1.520s (SENT)
I20260709 13:00:50.294087  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:50.306111  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
W20260709 13:00:50.266937  1500 rpcz_store.cc:269] 0709 13:00:10.885806 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:10.896272 (+ 10466us) service_pool.cc:224] Handling call
0709 13:00:50.086646 (+39190374us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 13:00:35.829501 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 13:00:36.006374 (+176873us) write_op.cc:432] Acquiring schema lock in shared mode
0709 13:00:36.016363 (+  9989us) write_op.cc:435] Acquired schema lock
0709 13:00:36.024427 (+  8064us) tablet.cc:662] Decoding operations
0709 13:00:36.131417 (+106990us) write_op.cc:620] Acquiring the partition lock for write op
0709 13:00:36.131568 (+   151us) write_op.cc:641] Partition lock acquired for write op
0709 13:00:36.131616 (+    48us) tablet.cc:685] Acquiring locks for 12 operations
0709 13:00:36.133306 (+  1690us) tablet.cc:701] Row locks acquired
0709 13:00:36.133376 (+    70us) write_op.cc:260] PREPARE: finished
0709 13:00:36.133698 (+   322us) write_op.cc:270] Start()
0709 13:00:36.133913 (+   215us) write_op.cc:276] Timestamp: P: 1783602036133580 usec, L: 0
0709 13:00:36.133946 (+    33us) op_driver.cc:348] REPLICATION: starting
0709 13:00:36.283243 (+149297us) log.cc:844] Serialized 4806 byte log entry
0709 13:00:36.759728 (+476485us) op_driver.cc:464] REPLICATION: finished
0709 13:00:36.904700 (+144972us) write_op.cc:301] APPLY: starting
0709 13:00:36.904837 (+   137us) tablet.cc:1367] starting BulkCheckPresence
0709 13:00:36.905985 (+  1148us) tablet.cc:1370] finished BulkCheckPresence
0709 13:00:36.906135 (+   150us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 13:00:37.046556 (+140421us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 13:00:37.046971 (+   415us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=24
0709 13:00:37.047031 (+    60us) write_op.cc:312] APPLY: finished
0709 13:00:37.047525 (+   494us) log.cc:844] Serialized 89 byte log entry
0709 13:00:37.047765 (+   240us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:00:37.047966 (+   201us) write_op.cc:454] Released schema lock
0709 13:00:37.048057 (+    91us) write_op.cc:341] FINISH: Updating metrics
0709 13:00:37.048650 (+   593us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:00:37.048701 (+    51us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":1997332,"reported_tablets":42,"spinlock_wait_cycles":1280,"child_traces":[["op",{"apply.queue_time_us":144533,"apply.run_cpu_time_us":6179,"apply.run_wall_time_us":143926,"num_ops":12,"prepare.queue_time_us":333053,"prepare.run_cpu_time_us":40644,"prepare.run_wall_time_us":522230,"raft.queue_time_us":246986,"raft.run_cpu_time_us":80148,"raft.run_wall_time_us":257235,"replication_time_us":569636,"spinlock_wait_cycles":45952,"thread_start_us":573667,"threads_started":4,"wal-append.queue_time_us":115117,"wal-append.run_cpu_time_us":11909,"wal-append.run_wall_time_us":1816090}]]}
I20260709 13:00:50.303129  2298 leader_election.cc:304] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:00:50.311673  5793 consensus_queue.cc:260] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:50.313252  5581 raft_consensus.cc:2804] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Leader election won for term 10
W20260709 13:00:50.302059  2358 debug-util.cc:398] Leaking SignalData structure 0x7b08002f5600 after lost signal to thread 1537
I20260709 13:00:50.314400  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 11 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:00:50.318742  1646 leader_election.cc:336] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 12 election: RPC error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Timed out: RequestConsensusVote RPC to 127.0.107.195:40693 timed out after 1.567s (SENT)
I20260709 13:00:50.297055  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:50.316017  2328 raft_consensus.cc:1275] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260709 13:00:50.312469  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
W20260709 13:00:50.316632  1718 connection.cc:256] RPC call timeout handler was delayed by -8.7853689109999777s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1526s for a response
W20260709 13:00:50.316798  1571 connection.cc:256] RPC call timeout handler was delayed by -6.584782644000029s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17830000000000001s for a response
I20260709 13:00:50.319386  1748 raft_consensus.cc:2468] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 10.
W20260709 13:00:50.323779  2358 debug-util.cc:398] Leaking SignalData structure 0x7b08002f59c0 after lost signal to thread 1629
I20260709 13:00:50.317986  2333 raft_consensus.cc:1275] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 13:00:50.313472  5661 raft_consensus.cc:515] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } }
I20260709 13:00:50.322413  5793 raft_consensus.cc:3060] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Advancing to term 8
W20260709 13:00:50.325922  1571 connection.cc:256] RPC call timeout handler was delayed by -6.5476707789999864s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16670000000000001s for a response
W20260709 13:00:50.317546  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 82691us
W20260709 13:00:50.326457  2358 debug-util.cc:398] Leaking SignalData structure 0x7b08002f59a0 after lost signal to thread 1630
I20260709 13:00:50.327078  2337 raft_consensus.cc:1275] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 9 index: 10. Preceding OpId from leader: term: 10 index: 10. (term mismatch)
W20260709 13:00:50.350963  1571 connection.cc:256] RPC call timeout handler was delayed by -6.33049129099993s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1932s for a response
W20260709 13:00:50.349768  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 131091us
W20260709 13:00:50.351423  1571 connection.cc:256] RPC call timeout handler was delayed by -6.2452707789999522s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20030000000000003s for a response
I20260709 13:00:50.352075  2337 pending_rounds.cc:85] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1: Aborting all ops after (but not including) 9
W20260709 13:00:50.350610  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 90955us
W20260709 13:00:50.352543  1571 connection.cc:256] RPC call timeout handler was delayed by -6.2344707789999916s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2015s for a response
I20260709 13:00:50.352651  2337 pending_rounds.cc:107] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1: Aborting uncommitted NO_OP operation due to leader change: 9.10
W20260709 13:00:50.352936  1571 connection.cc:256] RPC call timeout handler was delayed by -6.1579707789999816s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21000000000000002s for a response
I20260709 13:00:50.352993  2337 raft_consensus.cc:2889] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 13:00:50.353312  1571 connection.cc:256] RPC call timeout handler was delayed by -6.0483999629999516s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2903s for a response
W20260709 13:00:50.353700  1571 connection.cc:256] RPC call timeout handler was delayed by -5.9088999629999535s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3058s for a response
W20260709 13:00:50.354261  1571 connection.cc:256] RPC call timeout handler was delayed by -5.8008999629999494s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3178s for a response
W20260709 13:00:50.354591  1571 connection.cc:256] RPC call timeout handler was delayed by -5.6451999629999818s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3351s for a response
W20260709 13:00:50.354911  1571 connection.cc:256] RPC call timeout handler was delayed by -5.433470778999947s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2905s for a response
I20260709 13:00:50.319511  1753 raft_consensus.cc:2376] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 9. Re-sending same reply.
I20260709 13:00:50.358781  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:50.359192  1598 raft_consensus.cc:1275] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260709 13:00:50.379209  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 10 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:50.380739  2335 raft_consensus.cc:2393] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 12: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:00:50.331785  1646 leader_election.cc:304] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
W20260709 13:00:50.383322  1718 connection.cc:256] RPC call timeout handler was delayed by -8.45050155499996s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17930000000000001s for a response
W20260709 13:00:50.383873  1718 connection.cc:256] RPC call timeout handler was delayed by -8.28990442899999s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20520000000000002s for a response
W20260709 13:00:50.384218  1718 connection.cc:256] RPC call timeout handler was delayed by -8.2391910669999788s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1615s for a response
W20260709 13:00:50.384531  1718 connection.cc:256] RPC call timeout handler was delayed by -7.7347855799999934s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2134s for a response
W20260709 13:00:50.384812  1718 connection.cc:256] RPC call timeout handler was delayed by -7.7272088609999514s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.25520000000000004s for a response
W20260709 13:00:50.385094  1718 connection.cc:256] RPC call timeout handler was delayed by -7.3635838679999779s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.33190000000000003s for a response
W20260709 13:00:50.385383  1718 connection.cc:256] RPC call timeout handler was delayed by -7.3265499720000093s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.31150000000000005s for a response
W20260709 13:00:50.385651  1718 connection.cc:256] RPC call timeout handler was delayed by -7.297705140000005s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.31150000000000005s for a response
W20260709 13:00:50.385922  1718 connection.cc:256] RPC call timeout handler was delayed by -7.1665300839999873s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2817s for a response
W20260709 13:00:50.386191  1718 connection.cc:256] RPC call timeout handler was delayed by -7.1665300839999873s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2817s for a response
W20260709 13:00:50.380434  1644 leader_election.cc:336] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625): Timed out: RequestConsensusVote RPC to 127.0.107.196:46625 timed out after 1.799s (SENT)
I20260709 13:00:50.324262  5575 raft_consensus.cc:2804] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:00:50.395283  5575 raft_consensus.cc:493] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:50.395784  5575 raft_consensus.cc:3060] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:00:50.318252  5738 raft_consensus.cc:3055] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:00:50.400475  5738 raft_consensus.cc:740] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:00:50.389156  5792 raft_consensus.cc:493] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:50.403112  5792 raft_consensus.cc:515] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:00:50.390487  2358 debug-util.cc:398] Leaking SignalData structure 0x7b08003dade0 after lost signal to thread 1704
W20260709 13:00:50.404209  2358 debug-util.cc:398] Leaking SignalData structure 0x7b080037bb00 after lost signal to thread 1705
I20260709 13:00:50.405450  5648 raft_consensus.cc:2749] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 13:00:50.356209  5773 raft_consensus.cc:2749] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:00:50.392226  1750 raft_consensus.cc:3060] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:00:50.393074  5791 leader_election.cc:290] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:50.393225  1499 catalog_manager.cc:5697] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 1 to 7, leader changed from 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) to 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194). New cstate: current_term: 7 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } }
W20260709 13:00:50.394009  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 75297.8us
I20260709 13:00:50.321919  5566 leader_election.cc:290] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:50.389401  1644 leader_election.cc:304] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:00:50.319751  5491 raft_consensus.cc:2804] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:00:50.420825  5491 raft_consensus.cc:493] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:00:50.421380  5491 raft_consensus.cc:3060] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:00:50.321991  5581 raft_consensus.cc:697] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [term 10 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:00:50.422691  5228 leader_election.cc:290] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 10 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:50.423022  5612 leader_election.cc:290] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:50.423595  5581 consensus_queue.cc:237] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:50.313273  5787 raft_consensus.cc:515] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:50.403529  1601 raft_consensus.cc:3060] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:00:50.316438  5496 raft_consensus.cc:3055] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [term 6 LEADER]: Stepping down as leader of term 6
W20260709 13:00:50.447624  2358 debug-util.cc:398] Leaking SignalData structure 0x7b08003c5560 after lost signal to thread 1778
W20260709 13:00:50.449544  2358 debug-util.cc:398] Leaking SignalData structure 0x7b0800323cc0 after lost signal to thread 2361
W20260709 13:00:50.450945  2358 debug-util.cc:398] Leaking SignalData structure 0x7b080037af60 after lost signal to thread 2362
I20260709 13:00:50.416132  1499 catalog_manager.cc:5697] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } }
W20260709 13:00:50.456434  2298 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:50.457073  1748 raft_consensus.cc:1275] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 13:00:50.401763  5738 consensus_queue.cc:260] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.10, Last appended by leader: 10, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:50.471544  1753 raft_consensus.cc:1275] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260709 13:00:50.477052  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:50.442708  5496 raft_consensus.cc:740] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
W20260709 13:00:50.480628  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 49498.2us
I20260709 13:00:50.476889  5734 raft_consensus.cc:515] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:50.486616  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:00:50.487403  2363 maintenance_manager.cc:445] Time spent finding best maintenance operation: real 13.701s	user 0.071s	sys 0.077s
I20260709 13:00:50.460004  5228 consensus_queue.cc:1048] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.032s
I20260709 13:00:50.428251  5482 raft_consensus.cc:515] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:00:50.495296  1701 debug-util.cc:398] Leaking SignalData structure 0x7b0800176520 after lost signal to thread 431
I20260709 13:00:50.477635  5470 raft_consensus.cc:2749] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260709 13:00:50.498250  1701 debug-util.cc:398] Leaking SignalData structure 0x7b08001acb60 after lost signal to thread 434
W20260709 13:00:50.509567  1701 debug-util.cc:398] Leaking SignalData structure 0x7b08001ebb00 after lost signal to thread 436
I20260709 13:00:50.465863  5777 consensus_queue.cc:1059] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 7, Time since last communication: 0.034s
I20260709 13:00:50.509867  1602 raft_consensus.cc:3060] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Advancing to term 12
W20260709 13:00:50.510150  1701 debug-util.cc:398] Leaking SignalData structure 0x7b080009ffa0 after lost signal to thread 1496
I20260709 13:00:50.510461  5777 raft_consensus.cc:3055] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 LEADER]: Stepping down as leader of term 11
W20260709 13:00:50.502617  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 390795us
I20260709 13:00:50.510900  5777 raft_consensus.cc:740] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:00:50.512060  5777 consensus_queue.cc:260] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.10, Last appended by leader: 10, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:50.513425  1717 leader_election.cc:304] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:00:50.513798  5777 raft_consensus.cc:3060] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:00:50.505762  5738 raft_consensus.cc:3060] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:00:50.508926  5734 leader_election.cc:290] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:00:50.510591  1701 debug-util.cc:398] Leaking SignalData structure 0x7b08001ebba0 after lost signal to thread 1498
I20260709 13:00:50.515650  1499 catalog_manager.cc:5697] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 3 to 7, leader changed from 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) to 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196). New cstate: current_term: 7 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 13:00:50.499922  5496 consensus_queue.cc:260] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:00:50.517338  1701 debug-util.cc:398] Leaking SignalData structure 0x7b08000a1fe0 after lost signal to thread 1537
W20260709 13:00:50.517935  1701 debug-util.cc:398] Leaking SignalData structure 0x7b08000a7da0 after lost signal to thread 1557
I20260709 13:00:50.517817  5496 raft_consensus.cc:3060] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Advancing to term 7
W20260709 13:00:50.519054  1701 debug-util.cc:398] Leaking SignalData structure 0x7b08001acc00 after lost signal to thread 1604
W20260709 13:00:50.519958  1701 debug-util.cc:398] Leaking SignalData structure 0x7b08000842a0 after lost signal to thread 1629
W20260709 13:00:50.520254  1701 debug-util.cc:398] Leaking SignalData structure 0x7b08001c0740 after lost signal to thread 1630
I20260709 13:00:50.520618  1569 leader_election.cc:304] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:00:50.521073  5612 raft_consensus.cc:2804] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:00:50.521548  5612 raft_consensus.cc:493] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:00:50.523542  1701 debug-util.cc:398] Leaking SignalData structure 0x7b08003ac3a0 after lost signal to thread 1680
W20260709 13:00:50.524267  1701 debug-util.cc:398] Leaking SignalData structure 0x7b08003346e0 after lost signal to thread 1704
W20260709 13:00:50.524551  1701 debug-util.cc:398] Leaking SignalData structure 0x7b080033b380 after lost signal to thread 1705
W20260709 13:00:50.525918  1701 debug-util.cc:398] Leaking SignalData structure 0x7b08003ddea0 after lost signal to thread 1778
W20260709 13:00:50.526187  1701 debug-util.cc:398] Leaking SignalData structure 0x7b080002f700 after lost signal to thread 1779
W20260709 13:00:50.530850  1701 debug-util.cc:398] Leaking SignalData structure 0x7b08003bc1c0 after lost signal to thread 2361
W20260709 13:00:50.531159  1701 debug-util.cc:398] Leaking SignalData structure 0x7b080039dd80 after lost signal to thread 2362
W20260709 13:00:50.531440  1701 debug-util.cc:398] Leaking SignalData structure 0x7b080039dea0 after lost signal to thread 5228
W20260709 13:00:50.531695  1701 debug-util.cc:398] Leaking SignalData structure 0x7b0800369ce0 after lost signal to thread 5470
I20260709 13:00:50.532032  1601 raft_consensus.cc:2468] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 10.
I20260709 13:00:50.517655  1499 catalog_manager.cc:5697] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
W20260709 13:00:50.534003   436 debug-util.cc:398] Leaking SignalData structure 0x7b080000fd40 after lost signal to thread 1646
W20260709 13:00:50.535051  1701 debug-util.cc:398] Leaking SignalData structure 0x7b08003c8e20 after lost signal to thread 5491
W20260709 13:00:50.535919  1701 debug-util.cc:398] Leaking SignalData structure 0x7b08000de1a0 after lost signal to thread 5566
W20260709 13:00:50.536453  1701 debug-util.cc:398] Leaking SignalData structure 0x7b0800312da0 after lost signal to thread 5572
W20260709 13:00:50.536963  1701 debug-util.cc:398] Leaking SignalData structure 0x7b0800361220 after lost signal to thread 5575
I20260709 13:00:50.521992  5612 raft_consensus.cc:3060] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:00:50.537957  5470 consensus_queue.cc:1048] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 13:00:50.541529  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:50.518855  5083 raft_consensus.cc:515] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:50.545081  5734 raft_consensus.cc:2764] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader pre-election decision vote started in defunct term 8: won
W20260709 13:00:50.545737  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 188928us
I20260709 13:00:50.546805  5787 leader_election.cc:290] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:50.547551  1717 leader_election.cc:304] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:00:50.535286  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
W20260709 13:00:50.551374  1701 debug-util.cc:398] Leaking SignalData structure 0x7b08000a8040 after lost signal to thread 5587
I20260709 13:00:50.559108  1754 raft_consensus.cc:3060] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Advancing to term 9
W20260709 13:00:50.562420  1701 debug-util.cc:398] Leaking SignalData structure 0x7b08002332e0 after lost signal to thread 5648
W20260709 13:00:50.562891  1701 debug-util.cc:398] Leaking SignalData structure 0x7b08003190e0 after lost signal to thread 5661
W20260709 13:00:50.563143  1701 debug-util.cc:398] Leaking SignalData structure 0x7b08002b8240 after lost signal to thread 5663
I20260709 13:00:50.565855  1602 raft_consensus.cc:2468] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 12.
W20260709 13:00:50.567564  1701 debug-util.cc:398] Leaking SignalData structure 0x7b08001ffb60 after lost signal to thread 5671
I20260709 13:00:50.562582  1499 catalog_manager.cc:5697] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 5 to 7, leader changed from 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194) to 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196). New cstate: current_term: 7 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
W20260709 13:00:50.576715  1701 debug-util.cc:398] Leaking SignalData structure 0x7b0800359360 after lost signal to thread 5687
I20260709 13:00:50.574690  2298 leader_election.cc:304] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
W20260709 13:00:50.577323  1701 debug-util.cc:398] Leaking SignalData structure 0x7b08003d16e0 after lost signal to thread 5708
W20260709 13:00:50.577677  1701 debug-util.cc:398] Leaking SignalData structure 0x7b08003c3360 after lost signal to thread 5711
W20260709 13:00:50.577998  1701 debug-util.cc:398] Leaking SignalData structure 0x7b080034d7a0 after lost signal to thread 5732
W20260709 13:00:50.578226  1701 debug-util.cc:398] Leaking SignalData structure 0x7b080029ba60 after lost signal to thread 5734
W20260709 13:00:50.578490  1701 debug-util.cc:398] Leaking SignalData structure 0x7b080038ad40 after lost signal to thread 5738
W20260709 13:00:50.576695  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 54483.8us
I20260709 13:00:50.583760  2332 raft_consensus.cc:2393] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 9: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:00:50.584399  2330 raft_consensus.cc:606] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Received request to transfer leadership to TS 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:50.584824  2330 raft_consensus.cc:614] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 13:00:50.594817  1569 leader_election.cc:400] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 7 election: Vote denied by peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625) with higher term. Message: Invalid argument: T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 7. Current term is 8.
I20260709 13:00:50.595167  5804 raft_consensus.cc:2804] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:00:50.595607  2332 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39726 (request call id 981) took 1187 ms. Trace:
I20260709 13:00:50.595829  5804 raft_consensus.cc:493] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:50.596406  5804 raft_consensus.cc:3060] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:00:50.597322  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:50.598634  1676 raft_consensus.cc:2468] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 10.
I20260709 13:00:50.598690  5792 leader_election.cc:290] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:50.596086  2332 rpcz_store.cc:276] 0709 13:00:49.408326 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:50.205343 (+797017us) service_pool.cc:224] Handling call
0709 13:00:50.595268 (+389925us) spinlock_profiling.cc:243] Waited 301 ms on lock 0x7b5400605ed4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:00:50.595493 (+   225us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":691607424}
I20260709 13:00:50.595378  1569 leader_election.cc:403] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 7 election: Cancelling election due to peer responding with higher term
W20260709 13:00:50.575899  1571 leader_election.cc:336] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 9 election: RPC error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Timed out: RequestConsensusVote RPC to 127.0.107.195:40693 timed out after 2.003s (SENT)
I20260709 13:00:50.601363  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:00:50.582412  1644 leader_election.cc:336] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625): Timed out: RequestConsensusVote RPC to 127.0.107.196:46625 timed out after 1.838s (SENT)
I20260709 13:00:50.604489  2330 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:35790 (request call id 4) took 1203 ms. Trace:
W20260709 13:00:50.610123  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 151080us
I20260709 13:00:50.604923  2330 rpcz_store.cc:276] 0709 13:00:49.396941 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:49.397173 (+   232us) service_pool.cc:224] Handling call
0709 13:00:50.600115 (+1202942us) spinlock_profiling.cc:243] Waited 535 ms on lock 0x7b5400605ed4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f426615c853 00007f426b12a60a 00007f42646b07d4 00007f42646b0762 00007f42646b0691 00007f42646b060c 00007f42646af8c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:00:50.600238 (+   123us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1230876928}
I20260709 13:00:50.615603  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:50.606649  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:00:50.616257  1644 leader_election.cc:336] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625): Timed out: RequestConsensusVote RPC to 127.0.107.196:46625 timed out after 2.040s (SENT)
I20260709 13:00:50.617448  1677 raft_consensus.cc:2468] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 10.
W20260709 13:00:50.618739  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 96450.8us
I20260709 13:00:50.618937  2298 leader_election.cc:304] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:00:50.622035  1679 raft_consensus.cc:1240] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 7. Current term is 8. Ops: []
I20260709 13:00:50.622853  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:50.625953  1675 raft_consensus.cc:2393] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 12: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:00:50.624354  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:50.629915  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:50.622539  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:00:50.625031  1644 leader_election.cc:336] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625): Timed out: RequestConsensusVote RPC to 127.0.107.196:46625 timed out after 2.371s (SENT)
I20260709 13:00:50.622720  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:00:50.635802  1644 leader_election.cc:336] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625): Timed out: RequestConsensusVote RPC to 127.0.107.196:46625 timed out after 3.220s (SENT)
I20260709 13:00:50.636499  1644 leader_election.cc:304] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:00:50.633821  1499 catalog_manager.cc:5697] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 3 to 6, leader changed from 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) to 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196). New cstate: current_term: 6 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 13:00:50.620990  5661 leader_election.cc:290] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:50.643537  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:00:50.647612  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 71770.3us
I20260709 13:00:50.648887  5807 raft_consensus.cc:2749] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:00:50.648936  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:50.649842  1674 raft_consensus.cc:2468] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 8.
I20260709 13:00:50.658857  5792 consensus_queue.cc:1048] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 13:00:50.661118  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:50.662606  5671 raft_consensus.cc:2804] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:00:50.663349  5671 raft_consensus.cc:697] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 12 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:00:50.664598  5671 consensus_queue.cc:237] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:50.673928  1676 raft_consensus.cc:2468] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 8.
W20260709 13:00:50.674624  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 113364us
I20260709 13:00:50.675753  1569 leader_election.cc:304] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:00:50.677412  1569 leader_election.cc:304] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:00:50.679224  5692 raft_consensus.cc:2804] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 13:00:50.679952  5692 raft_consensus.cc:697] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 10 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:00:50.681246  5692 consensus_queue.cc:237] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 8.10, Last appended by leader: 10, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:50.683939  5083 leader_election.cc:290] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 9 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 13:00:50.684810  5801 log.cc:927] Time spent T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.055s	user 0.000s	sys 0.000s
I20260709 13:00:50.685662  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:50.686538  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:50.688364  5482 leader_election.cc:290] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 8 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 13:00:50.681162  1646 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:50.689563  5634 raft_consensus.cc:515] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:50.692030  1749 raft_consensus.cc:2393] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 6: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:00:50.646694  1673 raft_consensus.cc:2468] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 10.
I20260709 13:00:50.699018  5754 raft_consensus.cc:2749] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 13:00:50.698803  2330 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a"
dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
mode: GRACEFUL
new_leader_uuid: "af520b67945d443b901bbeca37cc5701"
 from {username='slave'} at 127.0.0.1:35790
I20260709 13:00:50.699957  2330 raft_consensus.cc:606] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 10 LEADER]: Received request to transfer leadership to TS af520b67945d443b901bbeca37cc5701
I20260709 13:00:50.700122  5754 raft_consensus.cc:2764] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Leader pre-election decision vote started in defunct term 7: won
I20260709 13:00:50.702852  1754 raft_consensus.cc:2468] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 9.
W20260709 13:00:50.702805  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 54600.5us
W20260709 13:00:50.703868  1571 leader_election.cc:336] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Timed out: RequestConsensusVote RPC to 127.0.107.195:40693 timed out after 2.100s (SENT)
W20260709 13:00:50.705943  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 62805.3us
I20260709 13:00:50.712284  1745 raft_consensus.cc:1240] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 5. Current term is 6. Ops: []
W20260709 13:00:50.723759  1718 leader_election.cc:336] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Timed out: RequestConsensusVote RPC to 127.0.107.194:46261 timed out after 1.775s (SENT)
I20260709 13:00:50.724514  1718 leader_election.cc:304] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:00:50.727366  1717 leader_election.cc:304] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:00:50.692075  5634 leader_election.cc:290] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 6 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 13:00:50.731402  1718 leader_election.cc:336] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Timed out: RequestConsensusVote RPC to 127.0.107.194:46261 timed out after 1.822s (SENT)
I20260709 13:00:50.734114  1750 raft_consensus.cc:2410] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for term 12 because replica has last-logged OpId of term: 11 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 8.
I20260709 13:00:50.735562  1751 raft_consensus.cc:2468] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 8.
I20260709 13:00:50.737644  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:50.729504  5778 raft_consensus.cc:2749] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Leader election lost for term 7. Reason: Vote denied by peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625) with higher term. Message: Invalid argument: T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 7. Current term is 8.
I20260709 13:00:50.734906  5792 raft_consensus.cc:2804] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:00:50.742839  5792 raft_consensus.cc:493] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:00:50.743221  5792 raft_consensus.cc:3060] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:00:50.750710  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:50.751406  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:50.752125  1752 raft_consensus.cc:2468] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 11.
I20260709 13:00:50.752847  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:50.753419  5807 raft_consensus.cc:493] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:50.753618  1748 raft_consensus.cc:2468] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 10.
I20260709 13:00:50.754033  5807 raft_consensus.cc:515] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:50.754725  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:50.756623  5807 leader_election.cc:290] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:50.736852  1499 catalog_manager.cc:5697] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } }
I20260709 13:00:50.729807  5482 raft_consensus.cc:2749] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 13:00:50.763335  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:50.773427  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:50.775271  1569 leader_election.cc:304] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:00:50.777300   431 auto_leader_rebalancer.cc:452] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 13:00:50.781220   431 auto_leader_rebalancer.cc:527] All tables' leader rebalancing finished this round
I20260709 13:00:50.763211  5663 consensus_queue.cc:1048] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.019s
I20260709 13:00:50.785838  5778 consensus_queue.cc:1059] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 13:00:50.787109  5778 raft_consensus.cc:2749] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:00:50.787631  5777 raft_consensus.cc:3055] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:00:50.788080  5777 raft_consensus.cc:740] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:00:50.789104  5777 consensus_queue.cc:260] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:50.790586  5777 raft_consensus.cc:3060] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:00:50.789983  1499 catalog_manager.cc:5697] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 13:00:50.791337  5575 raft_consensus.cc:515] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:50.791601  1499 catalog_manager.cc:5697] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } }
I20260709 13:00:50.792969  1499 catalog_manager.cc:5697] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 4 to 6, leader changed from 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) to 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196). New cstate: current_term: 6 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 13:00:50.795542  1569 leader_election.cc:304] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:00:50.797216  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:50.797920  1596 raft_consensus.cc:3060] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:00:50.800407  1749 raft_consensus.cc:2468] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 13.
I20260709 13:00:50.801131  5754 raft_consensus.cc:2764] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Leader election decision vote started in defunct term 7: won
I20260709 13:00:50.763983  1717 leader_election.cc:304] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
W20260709 13:00:50.804807  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 69318.9us
I20260709 13:00:50.806566  5482 raft_consensus.cc:2804] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:00:50.807019  5482 raft_consensus.cc:493] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:50.807449  5482 raft_consensus.cc:3060] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:00:50.808555  1569 leader_election.cc:304] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:00:50.808339  5804 raft_consensus.cc:515] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:50.764940  2299 leader_election.cc:304] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:00:50.811267  5496 consensus_queue.cc:1059] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 13:00:50.766319  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:00:50.812028  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 51123.8us
W20260709 13:00:50.767871  1718 leader_election.cc:336] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 10 election: RPC error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Timed out: RequestConsensusVote RPC to 127.0.107.194:46261 timed out after 2.024s (SENT)
W20260709 13:00:50.813439  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 45621.5us
I20260709 13:00:50.813874  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:50.813498  5670 raft_consensus.cc:2804] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:00:50.815001  2337 raft_consensus.cc:1240] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 9 LEADER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 8. Current term is 9. Ops: []
I20260709 13:00:50.814811  5711 consensus_queue.cc:1048] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 13:00:50.815289  5670 raft_consensus.cc:493] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:50.815837  5670 raft_consensus.cc:3060] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:00:50.770309  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:50.816927  5793 consensus_queue.cc:1059] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 13:00:50.817732  5496 consensus_queue.cc:1059] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 13:00:50.818013  5793 raft_consensus.cc:3055] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:00:50.818492  5793 raft_consensus.cc:740] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:00:50.818933  5794 raft_consensus.cc:3055] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:00:50.819440  5794 raft_consensus.cc:740] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:00:50.819571  5793 consensus_queue.cc:260] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.9, Last appended by leader: 9, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:50.820686  5794 consensus_queue.cc:260] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.9, Last appended by leader: 9, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:50.821192  5793 raft_consensus.cc:3060] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:00:50.822078  5794 raft_consensus.cc:3060] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:00:50.822646  5791 raft_consensus.cc:3055] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:00:50.823066  5791 raft_consensus.cc:740] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:00:50.824208  5791 consensus_queue.cc:260] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:50.824687  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:50.826831  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:50.827757  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 10 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:50.828672  1747 raft_consensus.cc:2410] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 for term 10 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260709 13:00:50.828605  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:50.827876  1751 raft_consensus.cc:1240] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 10. Current term is 11. Ops: []
I20260709 13:00:50.814895  2332 raft_consensus.cc:1275] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 13:00:50.830711  1717 leader_election.cc:304] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:00:50.826107  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:50.826910  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:50.835139  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:50.838513  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:50.838979  1750 raft_consensus.cc:3055] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:00:50.839395  1750 raft_consensus.cc:740] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:00:50.840462  1750 consensus_queue.cc:260] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:50.841425  5738 raft_consensus.cc:2804] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 13:00:50.842224  5738 raft_consensus.cc:697] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 9 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:00:50.843073  5754 raft_consensus.cc:3060] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:00:50.843320  5738 consensus_queue.cc:237] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:00:50.844337  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 208551us
I20260709 13:00:50.846367  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:50.849846  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:50.852165  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:00:50.856168  1718 leader_election.cc:336] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 9 election: RPC error from VoteRequest() call to peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625): Timed out: RequestConsensusVote RPC to 127.0.107.196:46625 timed out after 3.319s (SENT)
I20260709 13:00:50.857317  5491 raft_consensus.cc:515] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:50.857968  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:50.857990  2330 raft_consensus.cc:1240] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 6. Current term is 7. Ops: []
I20260709 13:00:50.859082  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:50.859829  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:50.892050  2337 raft_consensus.cc:2468] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 8.
I20260709 13:00:50.841611  1750 raft_consensus.cc:3060] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:00:50.869582  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:50.869910  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:50.870698  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:50.870810  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:50.894300  2334 raft_consensus.cc:2468] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 12.
I20260709 13:00:50.871912  2329 raft_consensus.cc:1240] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 10 LEADER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 9. Current term is 10. Ops: []
I20260709 13:00:50.895680  5496 consensus_queue.cc:1048] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 13:00:50.896044  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:50.883221  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:50.890959  2335 raft_consensus.cc:2468] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 7.
I20260709 13:00:50.899480  1499 catalog_manager.cc:5697] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 6 to 9. New cstate: current_term: 9 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } }
W20260709 13:00:50.879030  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 47311.9us
I20260709 13:00:50.885313  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:50.885690  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:50.902696  1600 raft_consensus.cc:3060] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:00:50.887598  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:50.889598  5791 raft_consensus.cc:3060] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:00:50.906801  1598 raft_consensus.cc:3060] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:00:50.907553  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:50.908509  1754 raft_consensus.cc:3060] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:00:50.912945  1749 raft_consensus.cc:2468] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 10.
I20260709 13:00:50.913482  2335 raft_consensus.cc:1240] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 9. Current term is 10. Ops: []
I20260709 13:00:50.907660  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:50.914529  1599 raft_consensus.cc:3060] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:00:50.915023  1751 raft_consensus.cc:1275] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20260709 13:00:50.920413  5804 leader_election.cc:290] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:50.835878  5575 leader_election.cc:290] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 10 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:50.936316  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:50.937999  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:50.938683  1600 raft_consensus.cc:2468] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 9.
I20260709 13:00:50.938916  1677 raft_consensus.cc:3060] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:00:50.933223  5491 leader_election.cc:290] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 10 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:50.943692  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:50.957612  1600 raft_consensus.cc:2468] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 12.
I20260709 13:00:50.950651  1745 raft_consensus.cc:2410] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 for term 8 because replica has last-logged OpId of term: 7 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 8.
I20260709 13:00:50.954069  1717 leader_election.cc:304] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
W20260709 13:00:50.946550  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:50.952217  1750 raft_consensus.cc:1275] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 10 index: 9. (term mismatch)
I20260709 13:00:50.942394  1754 raft_consensus.cc:2468] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 12.
I20260709 13:00:50.960372  1750 pending_rounds.cc:85] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701: Aborting all ops after (but not including) 8
I20260709 13:00:50.960742  1750 pending_rounds.cc:107] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701: Aborting uncommitted NO_OP operation due to leader change: 9.9
I20260709 13:00:50.961028  1750 raft_consensus.cc:2889] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:00:50.943028  1499 catalog_manager.cc:5697] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 2 to 4, leader changed from 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) to 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194). New cstate: current_term: 4 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 13:00:50.954396  1599 raft_consensus.cc:2468] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 8.
I20260709 13:00:50.958055  1596 raft_consensus.cc:2468] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 10.
I20260709 13:00:50.962915  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:50.945641  1598 raft_consensus.cc:2468] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 8.
I20260709 13:00:50.963984  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:50.964781  2332 raft_consensus.cc:2468] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 9.
I20260709 13:00:50.971056  2298 leader_election.cc:304] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:00:50.975219  1604 raft_consensus.cc:1275] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20260709 13:00:50.977368  5692 raft_consensus.cc:2804] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 13:00:50.978062  5692 raft_consensus.cc:697] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 10 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:00:50.979213  5692 consensus_queue.cc:237] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:50.981007  2301 leader_election.cc:336] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Timed out: RequestConsensusVote RPC to 127.0.107.195:40693 timed out after 1.559s (SENT)
I20260709 13:00:50.984200  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:50.984989  1754 raft_consensus.cc:2468] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 10.
I20260709 13:00:50.986348  5228 consensus_queue.cc:1048] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 13:00:50.986774  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:50.984071  1604 raft_consensus.cc:1275] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 10 index: 9. (term mismatch)
I20260709 13:00:50.909937  5792 raft_consensus.cc:515] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:50.967844  5807 consensus_queue.cc:1048] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 13:00:50.997041  1644 leader_election.cc:304] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:00:51.007753  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:51.008868  1677 raft_consensus.cc:2468] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 10.
I20260709 13:00:50.923529  5612 raft_consensus.cc:515] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:00:51.013995  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:51.014901  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 10 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:50.960055  5648 raft_consensus.cc:493] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:51.017246  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:50.988983  1604 pending_rounds.cc:85] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a: Aborting all ops after (but not including) 8
I20260709 13:00:51.017923  1596 raft_consensus.cc:3060] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Advancing to term 9
I20260709 13:00:51.017984  1604 pending_rounds.cc:107] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a: Aborting uncommitted NO_OP operation due to leader change: 9.9
I20260709 13:00:51.018415  1604 raft_consensus.cc:2889] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:00:51.021745  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:51.022246  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:00:51.022879  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 214474us
I20260709 13:00:50.998458  1499 catalog_manager.cc:5697] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: UNKNOWN } } }
I20260709 13:00:51.024300  1749 raft_consensus.cc:1275] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 7 index: 9. Preceding OpId from leader: term: 8 index: 9. (term mismatch)
I20260709 13:00:51.023080  1754 raft_consensus.cc:2393] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 11: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:00:51.024709  1749 pending_rounds.cc:85] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701: Aborting all ops after (but not including) 8
I20260709 13:00:51.025022  1749 pending_rounds.cc:107] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701: Aborting uncommitted NO_OP operation due to leader change: 7.9
I20260709 13:00:51.025283  1749 raft_consensus.cc:2889] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:00:51.026165  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:51.027326  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:51.000728  5790 raft_consensus.cc:2804] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:00:50.969175  5470 consensus_queue.cc:1048] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 6, Time since last communication: 0.059s
I20260709 13:00:51.030570  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:51.030805  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 10 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:51.031257  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:51.035403  5648 raft_consensus.cc:515] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:51.036711  1676 raft_consensus.cc:3060] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:00:51.039391  5673 consensus_queue.cc:1048] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 13:00:51.040328  1569 leader_election.cc:304] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:00:50.969594  5787 consensus_queue.cc:1048] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.057s
I20260709 13:00:51.044422  5807 raft_consensus.cc:2804] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:00:51.008020  5661 consensus_queue.cc:1059] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.037s
I20260709 13:00:51.045212  5754 raft_consensus.cc:2749] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:00:51.047116  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:51.048543  5612 leader_election.cc:290] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:51.081980  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:51.025923  5228 consensus_queue.cc:1048] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 6, Time since last communication: 0.028s
I20260709 13:00:51.053172  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:51.056372  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 10 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:51.055779  1499 catalog_manager.cc:5697] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 6 to 7, leader changed from 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) to 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193). New cstate: current_term: 7 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 13:00:51.057744  5792 leader_election.cc:290] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 12 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:51.070904  5790 raft_consensus.cc:493] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:00:51.105633  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:51.105721  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:51.091796  1753 raft_consensus.cc:2468] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 8.
W20260709 13:00:51.066975  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 182497us
I20260709 13:00:51.072301  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:51.109958  2299 leader_election.cc:304] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:51.110339  1677 raft_consensus.cc:2376] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 10. Re-sending same reply.
I20260709 13:00:51.075295  1746 raft_consensus.cc:2410] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 for term 8 because replica has last-logged OpId of term: 7 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 8.
W20260709 13:00:51.111029  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:51.079388  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:51.111775  1600 raft_consensus.cc:2468] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 6.
I20260709 13:00:51.086205  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:51.080814  5807 raft_consensus.cc:493] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:51.090548  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:51.100770  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 9 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:51.081152  1745 raft_consensus.cc:2468] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 10.
I20260709 13:00:51.090893  1672 raft_consensus.cc:2468] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 8.
I20260709 13:00:51.094398  1596 raft_consensus.cc:2468] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 9.
I20260709 13:00:51.059796  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:51.103202  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:51.060143  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:51.149187  1598 raft_consensus.cc:2376] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 8. Re-sending same reply.
I20260709 13:00:51.149452  1749 raft_consensus.cc:2393] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 11: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:00:51.078076  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:51.089072  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 9 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:51.113114  1678 raft_consensus.cc:3060] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:00:51.116691  5791 consensus_queue.cc:1059] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 13:00:51.123900  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:51.126605  1753 raft_consensus.cc:1240] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 9. Current term is 10. Ops: []
I20260709 13:00:51.128057  5804 consensus_queue.cc:1048] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 13:00:51.146559  1499 catalog_manager.cc:5697] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 4 to 5, leader changed from af520b67945d443b901bbeca37cc5701 (127.0.107.195) to 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193). New cstate: current_term: 5 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 13:00:51.151080  1751 raft_consensus.cc:2410] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for term 9 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260709 13:00:51.152638  1599 raft_consensus.cc:1275] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 13:00:51.154728  1604 raft_consensus.cc:2368] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for earlier term 9. Current term is 10.
I20260709 13:00:51.152818  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:00:51.142373  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 102161us
W20260709 13:00:51.151908  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 154959us
I20260709 13:00:51.214908  1644 leader_election.cc:304] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:00:51.108978  5790 raft_consensus.cc:3060] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:00:51.156423  1601 raft_consensus.cc:2393] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 10: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
W20260709 13:00:51.213420  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 103526us
I20260709 13:00:51.144112  5612 consensus_queue.cc:1048] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.072s
I20260709 13:00:51.078814  5482 raft_consensus.cc:515] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:51.228453  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 74196.1us
W20260709 13:00:51.223979  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 132564us
I20260709 13:00:51.229578  1718 leader_election.cc:304] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:51.090996  5820 consensus_queue.cc:1059] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 9.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.142s
I20260709 13:00:51.151813  5777 raft_consensus.cc:2764] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Leader pre-election decision vote started in defunct term 7: won
I20260709 13:00:51.239727  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:51.240104  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:51.239542  1499 catalog_manager.cc:5697] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 6 to 9, leader changed from 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) to 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193). New cstate: current_term: 9 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 13:00:51.155840  5738 consensus_queue.cc:1059] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.089s
I20260709 13:00:51.143237  5793 consensus_queue.cc:1048] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.097s
I20260709 13:00:51.265805  1604 raft_consensus.cc:2393] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 8: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
W20260709 13:00:51.274756  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 45296.3us
I20260709 13:00:51.148357  5807 raft_consensus.cc:3060] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:00:51.266606  1499 catalog_manager.cc:5697] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 5 to 6, leader changed from 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) to 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193). New cstate: current_term: 6 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 13:00:51.283584  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 53563.7us
W20260709 13:00:51.285336  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 46913.7us
I20260709 13:00:51.301360  1718 leader_election.cc:304] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:00:51.233440  5670 raft_consensus.cc:515] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:51.303567  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 9 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:51.306737  1601 raft_consensus.cc:2368] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for earlier term 9. Current term is 10.
I20260709 13:00:51.308233  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:51.308971  5482 leader_election.cc:290] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 9 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:51.309103  1604 raft_consensus.cc:2468] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 9.
I20260709 13:00:51.309311  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:51.311436  1676 raft_consensus.cc:1275] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260709 13:00:51.310187  1680 raft_consensus.cc:2468] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 9.
I20260709 13:00:51.319506  1499 catalog_manager.cc:5697] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 2 to 6, leader changed from 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) to 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193). New cstate: current_term: 6 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 13:00:51.320815  1569 leader_election.cc:304] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
W20260709 13:00:51.315268  5815 log.cc:927] Time spent T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.202s	user 0.000s	sys 0.001s
I20260709 13:00:51.311890  5792 raft_consensus.cc:2804] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Leader pre-election won for term 11
W20260709 13:00:51.331488  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 116644us
I20260709 13:00:51.316443  5794 raft_consensus.cc:3055] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:00:51.347388  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:00:51.347427  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 115706us
I20260709 13:00:51.314652  5823 raft_consensus.cc:493] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:51.319024  5827 raft_consensus.cc:3055] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:00:51.354686  1604 raft_consensus.cc:2393] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 8: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
W20260709 13:00:51.358302  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:51.354924  5754 raft_consensus.cc:2749] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:00:51.347744  5792 raft_consensus.cc:493] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:00:51.363029  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 47405us
W20260709 13:00:51.363977  5824 log.cc:927] Time spent T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.083s	user 0.000s	sys 0.000s
I20260709 13:00:51.364359  1569 leader_election.cc:304] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:00:51.366647  2301 leader_election.cc:304] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:51.371471  5831 raft_consensus.cc:2764] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Leader pre-election decision vote started in defunct term 12: won
I20260709 13:00:51.375022  1499 catalog_manager.cc:5697] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 6 to 7, leader changed from 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194) to af520b67945d443b901bbeca37cc5701 (127.0.107.195). New cstate: current_term: 7 leader_uuid: "af520b67945d443b901bbeca37cc5701" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } }
I20260709 13:00:51.362206  5792 raft_consensus.cc:3060] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:00:51.378937  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:51.380599  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:51.381279  1599 raft_consensus.cc:2376] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 9. Re-sending same reply.
I20260709 13:00:51.382385  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:51.348045  5794 raft_consensus.cc:740] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:00:51.382931  5754 raft_consensus.cc:2749] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 13:00:51.383769  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:51.383940  5794 consensus_queue.cc:260] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:51.385084  5794 raft_consensus.cc:3060] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Advancing to term 8
W20260709 13:00:51.363816  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 78979.9us
I20260709 13:00:51.377676  1499 catalog_manager.cc:5697] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 7 to 8, leader changed from 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) to 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193). New cstate: current_term: 8 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:00:51.335723  5612 raft_consensus.cc:2804] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:00:51.357151  5825 raft_consensus.cc:2804] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:00:51.399118  5612 raft_consensus.cc:493] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:51.399435  5825 raft_consensus.cc:493] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:00:51.399641  5612 raft_consensus.cc:3060] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:00:51.399983  5825 raft_consensus.cc:3060] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:00:51.401845  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:51.406104  1675 raft_consensus.cc:1275] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260709 13:00:51.406234  5670 leader_election.cc:290] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 11 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:51.416139  5491 raft_consensus.cc:3055] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:00:51.416689  5491 raft_consensus.cc:740] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:00:51.417681  5491 consensus_queue.cc:260] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:51.355065  5587 raft_consensus.cc:2804] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 13:00:51.419272  5587 raft_consensus.cc:697] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 10 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:00:51.420312  5587 consensus_queue.cc:237] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:51.355695  5827 raft_consensus.cc:740] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:00:51.423504  5819 raft_consensus.cc:3055] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:00:51.424026  5819 raft_consensus.cc:740] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:00:51.406941  5823 raft_consensus.cc:515] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:51.406833  5648 leader_election.cc:290] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:51.386510  1569 leader_election.cc:304] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
W20260709 13:00:51.431373  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 61357.9us
I20260709 13:00:51.443673  2301 leader_election.cc:304] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:00:51.444517  1678 raft_consensus.cc:2468] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 9.
I20260709 13:00:51.444839  1499 catalog_manager.cc:5697] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } }
I20260709 13:00:51.435302  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:00:51.448800  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 172893us
W20260709 13:00:51.483136  1717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:51.447906  1604 raft_consensus.cc:2468] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 10.
I20260709 13:00:51.488978  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
W20260709 13:00:51.497905  1717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:51.498925  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:51.500787  1602 raft_consensus.cc:2410] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for term 10 because replica has last-logged OpId of term: 9 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 5.
W20260709 13:00:51.501257  1717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:00:51.503223  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 59631.7us
I20260709 13:00:51.504326  2301 leader_election.cc:304] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
W20260709 13:00:51.509294  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:51.516810  1675 raft_consensus.cc:3060] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:00:51.448711  5777 raft_consensus.cc:2749] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:00:51.523365  2301 leader_election.cc:304] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:00:51.529381  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:00:51.532218  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:51.534773  1718 leader_election.cc:304] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:00:51.539422  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:51.540143  1604 raft_consensus.cc:2468] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 5.
W20260709 13:00:51.544337  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 158040us
W20260709 13:00:51.544337  1717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:51.546517  1602 raft_consensus.cc:2468] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 6.
I20260709 13:00:51.524101  5228 raft_consensus.cc:2749] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 13:00:51.424038  5827 consensus_queue.cc:260] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:51.534732  5673 raft_consensus.cc:2749] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:00:51.535022  5823 leader_election.cc:290] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:00:51.549219  1646 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.534s	user 0.003s	sys 0.010s
W20260709 13:00:51.561120  1646 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.546s	user 0.003s	sys 0.010s
W20260709 13:00:51.561502  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 546816us
I20260709 13:00:51.542999  5575 raft_consensus.cc:2749] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 13:00:51.569657  1598 raft_consensus.cc:1275] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260709 13:00:51.557958  2301 leader_election.cc:304] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
W20260709 13:00:51.563490  1646 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:51.524214  5804 consensus_queue.cc:1059] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.023s
W20260709 13:00:51.564826  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 60218.3us
I20260709 13:00:51.565645  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:51.566636  1569 leader_election.cc:304] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 13:00:51.576830  1604 raft_consensus.cc:1275] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
W20260709 13:00:51.580356  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:51.586714  1747 raft_consensus.cc:2393] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 9: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:00:51.523988  5819 consensus_queue.cc:260] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:51.587730  1718 leader_election.cc:304] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:51.610989  1603 raft_consensus.cc:1275] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 13:00:51.582085  5830 raft_consensus.cc:493] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:00:51.613910  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 47356.8us
I20260709 13:00:51.575340  5833 raft_consensus.cc:2804] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:00:51.603150  5777 raft_consensus.cc:2804] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:00:51.601099  5827 raft_consensus.cc:3060] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:00:51.626734  5754 consensus_queue.cc:1048] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.013s
I20260709 13:00:51.639802  5732 consensus_queue.cc:1059] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 10.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.111s
I20260709 13:00:51.640947  5692 raft_consensus.cc:993] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1: : Instructing follower af520b67945d443b901bbeca37cc5701 to start an election
I20260709 13:00:51.641606  5754 consensus_queue.cc:1048] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 13:00:51.625861  5833 raft_consensus.cc:493] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:51.643105  5083 raft_consensus.cc:3055] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:00:51.655504  1673 raft_consensus.cc:3060] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Advancing to term 10
I20260709 13:00:51.625438  5835 raft_consensus.cc:2804] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:00:51.629161  5777 raft_consensus.cc:493] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
W20260709 13:00:51.656292  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 68642.1us
I20260709 13:00:51.626786  5228 consensus_queue.cc:1048] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.031s
I20260709 13:00:51.649729  5581 raft_consensus.cc:1081] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 10 LEADER]: Signalling peer af520b67945d443b901bbeca37cc5701 to start an election
W20260709 13:00:51.676872  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 56370us
I20260709 13:00:51.679315  1569 leader_election.cc:304] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:00:51.686379  5692 consensus_queue.cc:1048] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 13:00:51.637785  5670 consensus_queue.cc:1048] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.039s
I20260709 13:00:51.660779  5833 raft_consensus.cc:3060] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:00:51.660745  5083 raft_consensus.cc:740] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:00:51.629521  5830 raft_consensus.cc:515] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:51.677811  5835 raft_consensus.cc:493] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:51.680219  5777 raft_consensus.cc:3060] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:00:51.710868  1603 raft_consensus.cc:1275] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260709 13:00:51.708432  1718 leader_election.cc:304] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:00:51.713513  1571 leader_election.cc:304] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:00:51.716533  5835 raft_consensus.cc:3060] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:00:51.713168  5845 raft_consensus.cc:2749] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:00:51.773600  5804 raft_consensus.cc:2804] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:00:51.794588  5804 raft_consensus.cc:493] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
W20260709 13:00:51.827533  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 115572us
I20260709 13:00:51.827651  1675 raft_consensus.cc:2468] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 11.
W20260709 13:00:51.823101  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 109668us
W20260709 13:00:51.832279  5832 log.cc:927] Time spent T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.225s	user 0.000s	sys 0.001s
I20260709 13:00:51.829540  5804 raft_consensus.cc:3060] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:00:51.839133  5661 raft_consensus.cc:493] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:00:51.839804  5661 raft_consensus.cc:515] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:51.842303  5661 leader_election.cc:290] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:51.842676  5841 consensus_queue.cc:1048] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 13:00:51.843389  5661 consensus_queue.cc:1048] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 13:00:51.844933  5754 raft_consensus.cc:2804] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:00:51.845432  5754 raft_consensus.cc:493] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:51.845772  5754 raft_consensus.cc:3060] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:00:51.853006  5830 leader_election.cc:290] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:00:51.855298  5838 log.cc:927] Time spent T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.187s	user 0.000s	sys 0.001s
I20260709 13:00:51.842335  5648 consensus_queue.cc:1048] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.014s
I20260709 13:00:51.831401  5845 consensus_queue.cc:1048] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.027s
I20260709 13:00:51.808960  5849 raft_consensus.cc:2749] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:00:51.864915  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:51.865320  1569 leader_election.cc:304] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 13:00:51.865830  1754 raft_consensus.cc:2368] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 11. Current term is 12.
W20260709 13:00:51.867225  1719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:51.872570  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:51.876030  5834 consensus_queue.cc:1048] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.003s
I20260709 13:00:51.882802  5849 raft_consensus.cc:2804] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:00:51.883311  5849 raft_consensus.cc:493] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:51.883354  5491 raft_consensus.cc:3060] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:00:51.883765  5849 raft_consensus.cc:3060] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:00:51.873925  5612 raft_consensus.cc:515] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:51.888700  5612 leader_election.cc:290] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:00:51.892439  5807 consensus_meta.cc:287] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.610s	user 0.011s	sys 0.002s
I20260709 13:00:51.893093  5807 raft_consensus.cc:515] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:51.895151  1679 raft_consensus.cc:2393] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 8: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:00:51.895944  1718 leader_election.cc:304] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:00:51.896283  5817 raft_consensus.cc:3055] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:00:51.896782  5817 raft_consensus.cc:740] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:00:51.897433  1718 leader_election.cc:304] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:00:51.900542  5791 raft_consensus.cc:2749] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260709 13:00:51.900934  5825 consensus_meta.cc:287] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.500s	user 0.006s	sys 0.009s
I20260709 13:00:51.901332  5791 raft_consensus.cc:2764] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Leader pre-election decision vote started in defunct term 11: won
I20260709 13:00:51.901461  5825 raft_consensus.cc:515] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:51.875150  5792 raft_consensus.cc:515] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:51.905678  5792 leader_election.cc:290] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:51.910728  5692 consensus_queue.cc:1048] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 13:00:51.912123  1718 leader_election.cc:304] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:00:51.914494  5671 consensus_queue.cc:1048] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 13:00:51.915098  5825 leader_election.cc:290] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 9 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:51.915488  5581 consensus_queue.cc:1048] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 13:00:51.923755  5804 raft_consensus.cc:515] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:51.927547  5790 consensus_meta.cc:287] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.688s	user 0.010s	sys 0.004s
I20260709 13:00:51.928099  5790 raft_consensus.cc:515] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:51.921422  5754 raft_consensus.cc:515] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:51.938897  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 73682.6us
I20260709 13:00:51.804030  5083 consensus_queue.cc:260] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:51.944415  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:51.938251  5834 consensus_queue.cc:1059] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.009s
I20260709 13:00:51.947855  5482 raft_consensus.cc:3055] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:00:51.944303  5612 raft_consensus.cc:2804] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:00:51.948523  5482 raft_consensus.cc:740] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:00:51.948664  5612 raft_consensus.cc:493] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:51.949291  5612 raft_consensus.cc:3060] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:00:51.938577  5807 leader_election.cc:290] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 8 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:51.956871   431 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:00:51.958761  1673 raft_consensus.cc:1275] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 8 index: 10. Preceding OpId from leader: term: 10 index: 11. (index mismatch)
I20260709 13:00:51.979545  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:00:51.983912  1719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:51.990000  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:51.991070  1747 raft_consensus.cc:2468] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 6.
I20260709 13:00:51.993386  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:00:52.020679  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:52.036800  1601 raft_consensus.cc:1240] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 10. Current term is 11. Ops: [10.9-10.9]
W20260709 13:00:52.040884  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 99506.4us
I20260709 13:00:52.041016  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:52.044189  1603 raft_consensus.cc:1275] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260709 13:00:51.914113  5833 raft_consensus.cc:515] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:51.897833  5817 consensus_queue.cc:260] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:52.063908  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:52.073784  2299 leader_election.cc:304] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:00:52.074193  1753 raft_consensus.cc:2376] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 9. Re-sending same reply.
I20260709 13:00:52.063627  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:52.064697  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:00:52.072309  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 230951us
I20260709 13:00:51.966408  5804 leader_election.cc:290] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:52.078794  1751 raft_consensus.cc:2393] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 6: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
W20260709 13:00:52.083374  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 44008.5us
I20260709 13:00:52.085997  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:52.086961  1754 raft_consensus.cc:2468] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 8.
W20260709 13:00:52.085902  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 382138us
I20260709 13:00:51.949996  5482 consensus_queue.cc:260] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:51.940780  5835 raft_consensus.cc:515] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:52.115244  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:52.096388  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:52.096388  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:52.137254  1752 raft_consensus.cc:2393] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 10: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
W20260709 13:00:52.138803  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 65096.5us
W20260709 13:00:52.093117  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 493254us
I20260709 13:00:52.142407  1646 leader_election.cc:304] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
W20260709 13:00:52.142910  1748 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.509s	user 0.000s	sys 0.015s
I20260709 13:00:52.144013  1749 raft_consensus.cc:2393] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 11: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
W20260709 13:00:52.146476  1643 leader_election.cc:336] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 9 election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 1.582s (SENT)
I20260709 13:00:52.147626  1643 leader_election.cc:304] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:52.148708  5807 raft_consensus.cc:2749] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
W20260709 13:00:52.149263  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 59457.6us
W20260709 13:00:52.152576  1643 leader_election.cc:336] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 1.590s (SENT)
I20260709 13:00:52.153155  2299 leader_election.cc:304] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:52.136720  5790 leader_election.cc:290] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:52.157245  1752 raft_consensus.cc:1240] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 8. Current term is 9. Ops: [8.5-8.5]
W20260709 13:00:52.159950  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 66981.4us
I20260709 13:00:52.171847  2299 leader_election.cc:304] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:00:52.177587  2299 leader_election.cc:304] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:00:52.183692  1646 leader_election.cc:304] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:00:52.186532  5581 raft_consensus.cc:2804] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 13:00:52.186767  5825 consensus_queue.cc:1059] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 9.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 13:00:52.187888  5825 raft_consensus.cc:2749] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:00:52.188887  5849 raft_consensus.cc:515] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:52.178383  5807 raft_consensus.cc:2804] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:00:52.189620  5648 raft_consensus.cc:3055] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:00:52.199399  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:52.199800  5648 raft_consensus.cc:740] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:00:52.200994  5648 consensus_queue.cc:260] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.7, Last appended by leader: 7, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:52.201659  1748 raft_consensus.cc:1240] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 10. Current term is 11. Ops: []
I20260709 13:00:52.202118  5849 leader_election.cc:290] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:52.207032  5849 consensus_queue.cc:1048] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 13:00:52.213999  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:52.215073  1746 raft_consensus.cc:2393] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 9: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:00:52.216595  1746 raft_consensus.cc:1240] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 10. Current term is 11. Ops: [10.7-10.7]
I20260709 13:00:52.219630  5083 raft_consensus.cc:3060] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:00:52.192936  5860 consensus_queue.cc:1059] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.053s
W20260709 13:00:52.193710  2300 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:52.194453  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:52.144835  5804 raft_consensus.cc:493] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:00:52.202646  5836 log.cc:927] Time spent T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.078s	user 0.001s	sys 0.000s
I20260709 13:00:52.199765  5807 raft_consensus.cc:493] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Starting leader election (detected failure of leader af520b67945d443b901bbeca37cc5701)
W20260709 13:00:52.191807  5827 consensus_meta.cc:287] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.557s	user 0.002s	sys 0.011s
I20260709 13:00:52.200984  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:52.203680  5648 raft_consensus.cc:3060] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:00:52.192950  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:52.087628  5663 consensus_queue.cc:1048] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.086s
I20260709 13:00:52.166867  5673 raft_consensus.cc:2804] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 13:00:52.165845  5831 raft_consensus.cc:2804] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:00:52.239704  1747 raft_consensus.cc:2468] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 8.
I20260709 13:00:52.181330  5819 raft_consensus.cc:3060] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:00:52.182551  5482 raft_consensus.cc:3060] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Advancing to term 10
W20260709 13:00:52.255398  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 78144.5us
W20260709 13:00:52.256559  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 116234us
I20260709 13:00:52.258570  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:52.269371  5727 consensus_queue.cc:1048] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 13:00:52.271504  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:52.273301  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:52.274027  1598 raft_consensus.cc:3060] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:00:52.263104  1603 raft_consensus.cc:1275] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 7 index: 9. Preceding OpId from leader: term: 10 index: 11. (index mismatch)
I20260709 13:00:52.269502  5228 consensus_queue.cc:1048] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.007s
W20260709 13:00:52.279033  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 128867us
W20260709 13:00:52.279837  1643 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 1.602s (SENT) [suppressed 53 similar messages]
I20260709 13:00:52.166167  5587 consensus_queue.cc:1048] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.077s
W20260709 13:00:52.286597  1644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:52.237082  5807 raft_consensus.cc:3060] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:00:52.242055  5831 raft_consensus.cc:493] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:52.238941  5581 raft_consensus.cc:697] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 11 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:00:52.259766  5670 raft_consensus.cc:3060] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:00:52.299500  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:00:52.302790  1643 leader_election.cc:336] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 1.613s (SENT)
I20260709 13:00:52.306547  1643 leader_election.cc:304] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:00:52.308945  1674 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57092 (request call id 867) took 1288 ms. Trace:
I20260709 13:00:52.309608  1674 rpcz_store.cc:276] 0709 13:00:51.011134 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:51.015628 (+  4494us) service_pool.cc:224] Handling call
0709 13:00:51.058347 (+ 42719us) spinlock_profiling.cc:243] Waited 22.2 ms on lock 0x7b5800086a98. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f426303db71 00007f426303deee 0000000000375939 00007f426b0b11e0 00007f426b17a0c4 00007f426b17a131 00007f426b13bfc0 00007f426b126929 00007f426b1263d2 00007f426469dbd4
0709 13:00:51.058605 (+   258us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:00:51.059145 (+   540us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:00:51.059179 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:00:51.059779 (+   600us) log.cc:844] Serialized 26 byte log entry
0709 13:00:51.537437 (+477658us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:00:51.555848 (+ 18411us) log.cc:844] Serialized 14 byte log entry
0709 13:00:51.837986 (+282138us) spinlock_profiling.cc:243] Waited 8.67 ms on lock 0x7b1800417a60. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f4262fefc0a 00007f426c72f219 00007f4263077a40 00007f42630724e4 00007f4263072455 00007f426615a060 00007f426616c716 00007f426617d287 00007f426617d208
0709 13:00:51.838209 (+   223us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:51.838290 (+    81us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:00:51.876429 (+ 38139us) spinlock_profiling.cc:243] Waited 10.2 ms on lock 0x7b1800417a60. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f42630753f3 00007f42630777c1 00007f4263077779 00007f4263077709 00007f42630776d1 00007f42630769cd 000000000038f354 000000000038f199 00007f426305883a
0709 13:00:52.295654 (+419225us) raft_consensus.cc:1685] finished
0709 13:00:52.295694 (+    40us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:52.299440 (+  3746us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":50736,"prepare.queue_time_us":18189,"prepare.run_cpu_time_us":64,"prepare.run_wall_time_us":6496,"raft.queue_time_us":489308,"raft.run_cpu_time_us":503,"raft.run_wall_time_us":35862,"spinlock_wait_cycles":94401664,"thread_start_us":477203,"threads_started":1,"wal-append.queue_time_us":526551}
I20260709 13:00:52.322355  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:52.322044  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:52.242125  5804 raft_consensus.cc:515] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:52.292609  5673 raft_consensus.cc:697] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 9 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:00:52.264937  5661 consensus_queue.cc:1059] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 10.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.050s
I20260709 13:00:52.304901  5831 raft_consensus.cc:3060] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:00:52.334908  1571 leader_election.cc:400] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 election: Vote denied by peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693) with higher term. Message: Invalid argument: T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 11. Current term is 12.
I20260709 13:00:52.278720  5787 consensus_queue.cc:1059] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 9.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.040s
I20260709 13:00:52.282823  5692 consensus_queue.cc:1059] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.036s
W20260709 13:00:52.357180  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 89765.7us
I20260709 13:00:52.358392  1718 leader_election.cc:304] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:52.360862  1717 leader_election.cc:304] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:52.361146  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:52.362387  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:52.326707  5754 leader_election.cc:290] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:52.374977  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:52.379221  1752 raft_consensus.cc:3060] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Advancing to term 11
W20260709 13:00:52.380517  1719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:52.380939  1598 raft_consensus.cc:2468] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 11.
I20260709 13:00:52.346685  5827 raft_consensus.cc:2749] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260709 13:00:52.348798  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 165341us
I20260709 13:00:52.296976  5671 consensus_queue.cc:1048] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.039s
W20260709 13:00:52.415096  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 150638us
I20260709 13:00:52.346855  1571 leader_election.cc:403] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 election: Cancelling election due to peer responding with higher term
I20260709 13:00:52.355927  5793 raft_consensus.cc:493] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:52.393792  1602 raft_consensus.cc:2468] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 9.
I20260709 13:00:52.394913  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:52.365037  5792 raft_consensus.cc:2749] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:00:52.371256  5806 raft_consensus.cc:493] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
W20260709 13:00:52.392985  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 90211.2us
I20260709 13:00:52.359334  5738 raft_consensus.cc:493] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:00:52.434938  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 76639us
I20260709 13:00:52.436623  1598 raft_consensus.cc:2468] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 9.
W20260709 13:00:52.437103  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 79585.6us
W20260709 13:00:52.451704  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 120921us
I20260709 13:00:52.459692  1571 leader_election.cc:304] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:00:52.452643  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:52.420724  5834 raft_consensus.cc:493] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:52.461407  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:52.395088  5496 raft_consensus.cc:2749] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260709 13:00:52.431354  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 70553.6us
W20260709 13:00:52.466066  1719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:52.466506  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:52.443620  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:52.476737  1603 raft_consensus.cc:3060] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:00:52.467371  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:52.470297  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:00:52.477661  1717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:52.468624  1602 raft_consensus.cc:2468] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 7.
I20260709 13:00:52.474542  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:52.477413  1601 raft_consensus.cc:3060] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:00:52.450165  5842 raft_consensus.cc:493] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:52.479756  5842 raft_consensus.cc:515] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:52.474491  5575 consensus_queue.cc:1048] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 8, Time since last communication: 0.008s
I20260709 13:00:52.483320  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:52.484164  1674 raft_consensus.cc:2393] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 8: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
W20260709 13:00:52.485548  5859 log.cc:927] Time spent T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.070s	user 0.000s	sys 0.000s
I20260709 13:00:52.482242  5842 leader_election.cc:290] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 13:00:52.487149  5862 log.cc:927] Time spent T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.201s	user 0.001s	sys 0.000s
I20260709 13:00:52.487298  1571 leader_election.cc:304] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:00:52.489179  1571 leader_election.cc:304] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:00:52.489773  1600 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 2214) took 1111 ms. Trace:
I20260709 13:00:52.490006  5849 raft_consensus.cc:2804] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:00:52.490538  5849 raft_consensus.cc:493] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:52.491114  5849 raft_consensus.cc:3060] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:00:52.467070  5581 consensus_queue.cc:237] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 9.10, Last appended by leader: 10, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:52.491488  5865 log.cc:927] Time spent T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.292s	user 0.000s	sys 0.003s
W20260709 13:00:52.492426  5777 consensus_meta.cc:287] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.765s	user 0.010s	sys 0.004s
I20260709 13:00:52.492892  5777 raft_consensus.cc:515] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:52.493522  5863 log.cc:927] Time spent T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.276s	user 0.003s	sys 0.000s
W20260709 13:00:52.495107  5491 consensus_meta.cc:287] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.611s	user 0.006s	sys 0.007s
I20260709 13:00:52.495229  1745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43156 (request call id 755) took 1283 ms. Trace:
I20260709 13:00:52.467147  5855 raft_consensus.cc:2749] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader election lost for term 11. Reason: Vote denied by peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693) with higher term. Message: Invalid argument: T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 11. Current term is 12.
I20260709 13:00:52.495707  1745 rpcz_store.cc:276] 0709 13:00:51.212340 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:51.256763 (+ 44423us) service_pool.cc:224] Handling call
0709 13:00:51.280494 (+ 23731us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:00:51.355745 (+ 75251us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:00:51.355798 (+    53us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:00:51.358378 (+  2580us) log.cc:844] Serialized 26 byte log entry
0709 13:00:51.907641 (+549263us) spinlock_profiling.cc:243] Waited 30 ms on lock 0x7b18004ab000. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f4263057e07 00007f426beb2ab9 00007f426307469b 00007f426307293f 00007f4263074c02 00007f4265cfc20b 00007f4265d01d23 00007f4265d020d1 00007f426611da0f
0709 13:00:52.082210 (+174569us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:00:52.134416 (+ 52206us) log.cc:844] Serialized 14 byte log entry
0709 13:00:52.144977 (+ 10561us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:52.145071 (+    94us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:00:52.430820 (+285749us) spinlock_profiling.cc:243] Waited 10.9 ms on lock 0x7b4400022b4c. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f4262f9bc85 00007f4262f9c6a2 00007f4265d0e1db 00007f4265d09aaf 00007f4265cfd86e 00007f4265cfcbb6 00007f4265cfc7d8 00007f4265d06221 00007f4265d061d9 00007f4265d06169
0709 13:00:52.494900 (+ 64080us) raft_consensus.cc:1685] finished
0709 13:00:52.494925 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:52.495123 (+   198us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":29764,"prepare.queue_time_us":54362,"prepare.run_cpu_time_us":99,"prepare.run_wall_time_us":8940,"raft.queue_time_us":45449,"raft.run_cpu_time_us":352,"raft.run_wall_time_us":13924,"spinlock_wait_cycles":94032128,"thread_start_us":147104,"threads_started":1,"wal-append.queue_time_us":821903}
W20260709 13:00:52.499162  5612 consensus_meta.cc:287] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.549s	user 0.010s	sys 0.003s
I20260709 13:00:52.499631  5612 raft_consensus.cc:515] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:52.501832  5754 raft_consensus.cc:2749] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:00:52.502080  1754 raft_consensus.cc:1240] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 9. Current term is 10. Ops: [9.9-9.9]
I20260709 13:00:52.486979  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:00:52.504112  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:52.504936  1604 raft_consensus.cc:2393] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 9: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:00:52.504716  5849 raft_consensus.cc:515] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:52.505872  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:52.506721  1597 raft_consensus.cc:1240] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 9. Current term is 10. Ops: []
I20260709 13:00:52.507270  5849 leader_election.cc:290] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 7 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:52.508251  5841 raft_consensus.cc:3055] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:00:52.508659  5841 raft_consensus.cc:740] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:00:52.509575  5841 consensus_queue.cc:260] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.7, Last appended by leader: 7, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:00:52.511508  5864 log.cc:927] Time spent T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.073s	user 0.001s	sys 0.000s
I20260709 13:00:52.512933  1676 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57092 (request call id 877) took 1492 ms. Trace:
I20260709 13:00:52.513296  1676 rpcz_store.cc:276] 0709 13:00:51.020447 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:51.312711 (+292264us) service_pool.cc:224] Handling call
0709 13:00:51.313251 (+   540us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:00:51.321764 (+  8513us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:00:51.321825 (+    61us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:00:51.332363 (+ 10538us) log.cc:844] Serialized 26 byte log entry
0709 13:00:51.932992 (+600629us) spinlock_profiling.cc:243] Waited 28.7 ms on lock 0x7b1800462ca0. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f4263057e07 00007f426beb2ab9 00007f426307469b 00007f426307293f 00007f4263074c02 00007f4265cfc20b 00007f4265d01d23 00007f4265d020d1 00007f426611da0f
0709 13:00:52.235650 (+302658us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:00:52.235696 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:52.235772 (+    76us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:00:52.512551 (+276779us) raft_consensus.cc:1685] finished
0709 13:00:52.512574 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:52.512860 (+   286us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4396,"raft.queue_time_us":33892,"raft.run_cpu_time_us":157,"raft.run_wall_time_us":6347,"spinlock_wait_cycles":65971456,"thread_start_us":302409,"threads_started":1,"wal-append.queue_time_us":960912}
W20260709 13:00:52.513303  5861 log.cc:927] Time spent T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.297s	user 0.000s	sys 0.002s
I20260709 13:00:52.516151  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:00:52.517004  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:00:52.518128  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:00:52.519277  2328 raft_consensus.cc:1240] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 9. Current term is 10. Ops: []
I20260709 13:00:52.520126  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:52.520768  1602 raft_consensus.cc:3060] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Advancing to term 11
W20260709 13:00:52.523362  5871 log.cc:927] Time spent T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.067s	user 0.000s	sys 0.000s
I20260709 13:00:52.528113  1604 raft_consensus.cc:2393] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 7: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:00:52.528358  5860 raft_consensus.cc:3055] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:00:52.528863  5860 raft_consensus.cc:740] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:00:52.529313  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 13 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:52.529803  1596 raft_consensus.cc:1240] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 6. Current term is 7. Ops: []
I20260709 13:00:52.530323  1604 raft_consensus.cc:2410] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 for term 13 because replica has last-logged OpId of term: 11 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 9.
I20260709 13:00:52.531183  5825 raft_consensus.cc:3055] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:00:52.531652  5825 raft_consensus.cc:740] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:00:52.532864  1604 raft_consensus.cc:1240] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 7. Current term is 8. Ops: [7.5-7.5]
I20260709 13:00:52.532699  5825 consensus_queue.cc:260] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:52.555176  1718 leader_election.cc:304] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:52.490425  1571 leader_election.cc:304] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:00:52.451089  5833 leader_election.cc:290] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 9 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:52.554543  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:52.557961  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 11 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:52.558737  1752 raft_consensus.cc:2468] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 11.
I20260709 13:00:52.490216  1600 rpcz_store.cc:276] 0709 13:00:51.378001 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:51.384867 (+  6866us) service_pool.cc:224] Handling call
0709 13:00:51.385272 (+   405us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:00:51.385792 (+   520us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:00:51.385829 (+    37us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:00:51.931112 (+545283us) spinlock_profiling.cc:243] Waited 4.51 ms on lock 0x7b2c00000658. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4264a5ba11 00007f4264a6b372 00007f426619da01 00007f426616243a 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2
0709 13:00:51.932048 (+   936us) log.cc:844] Serialized 52 byte log entry
0709 13:00:52.061145 (+129097us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:00:52.237510 (+176365us) log.cc:844] Serialized 14 byte log entry
0709 13:00:52.244127 (+  6617us) log.cc:844] Serialized 14 byte log entry
0709 13:00:52.244603 (+   476us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:52.244722 (+   119us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:00:52.489492 (+244770us) raft_consensus.cc:1685] finished
0709 13:00:52.489517 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:52.489677 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":11157,"prepare.queue_time_us":515714,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":47,"raft.queue_time_us":366,"raft.run_cpu_time_us":2113,"raft.run_wall_time_us":287528,"spinlock_wait_cycles":32815744,"thread_start_us":219138,"threads_started":2,"wal-append.queue_time_us":233215}
I20260709 13:00:52.560799  1603 raft_consensus.cc:2468] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 6.
I20260709 13:00:52.543485  5663 raft_consensus.cc:3055] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 10 LEADER]: Stepping down as leader of term 10
W20260709 13:00:52.566291  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:52.567514  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:52.570611  1598 raft_consensus.cc:2368] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 8. Current term is 9.
I20260709 13:00:52.573355  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:52.565191  1601 raft_consensus.cc:2468] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 11.
I20260709 13:00:52.580528  1601 raft_consensus.cc:1240] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 8. Current term is 9. Ops: []
I20260709 13:00:52.582275  2332 raft_consensus.cc:2393] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 11: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
W20260709 13:00:52.583130  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 45582.5us
W20260709 13:00:52.588523  1643 leader_election.cc:336] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 1.650s (SENT)
I20260709 13:00:52.589121  1643 leader_election.cc:304] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
W20260709 13:00:52.584858  1646 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:52.608832  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:00:52.609061  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:00:52.610251  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 13 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:00:52.613474  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 123102us
I20260709 13:00:52.449790  5793 raft_consensus.cc:515] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:00:52.614001  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 58972.7us
I20260709 13:00:52.545987  5835 leader_election.cc:290] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 11 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:00:52.590808  5852 log.cc:927] Time spent T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.356s	user 0.000s	sys 0.002s
W20260709 13:00:52.616037  1646 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:52.615171  1674 raft_consensus.cc:2410] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 for term 13 because replica has last-logged OpId of term: 11 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 9.
I20260709 13:00:52.472958  5738 raft_consensus.cc:515] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:52.621112  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:52.621388  1571 leader_election.cc:304] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:00:52.622646  1718 leader_election.cc:304] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:52.625623  5793 leader_election.cc:290] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:52.572870  5855 raft_consensus.cc:2749] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:00:52.529927  5860 consensus_queue.cc:260] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.7, Last appended by leader: 7, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:52.626937  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:52.619210  5820 raft_consensus.cc:2749] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 13:00:52.627451  5866 raft_consensus.cc:2804] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Leader election won for term 9
I20260709 13:00:52.615427  1598 raft_consensus.cc:1275] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20260709 13:00:52.475476  5806 raft_consensus.cc:515] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:52.611024  5841 raft_consensus.cc:3060] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:00:52.638546  1602 raft_consensus.cc:2468] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 11.
W20260709 13:00:52.638902  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:52.649513  1672 raft_consensus.cc:3060] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:00:52.570951  5827 raft_consensus.cc:493] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:52.651896  1571 leader_election.cc:304] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:00:52.565897  5663 raft_consensus.cc:740] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:00:52.469211  5673 consensus_queue.cc:237] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:52.701489  1644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:52.535463  5834 raft_consensus.cc:515] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:52.630561  5825 raft_consensus.cc:3060] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:00:52.612432  5661 consensus_queue.cc:1059] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 6, Time since last communication: 0.029s
I20260709 13:00:52.653381  5648 raft_consensus.cc:2749] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:00:52.649097  5833 raft_consensus.cc:2749] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 13:00:52.712545  1748 raft_consensus.cc:1240] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 11. Current term is 12. Ops: []
I20260709 13:00:52.657673  5854 raft_consensus.cc:2764] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Leader pre-election decision vote started in defunct term 7: won
W20260709 13:00:52.655766  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 387195us
W20260709 13:00:52.723028  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 71221.5us
I20260709 13:00:52.726217  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:52.725344  2337 raft_consensus.cc:2468] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 11.
W20260709 13:00:52.708940  5830 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.511s	user 0.006s	sys 0.011s
I20260709 13:00:52.732362  1680 raft_consensus.cc:1275] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260709 13:00:52.707960  5878 raft_consensus.cc:493] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:52.750787  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:52.739450  1571 leader_election.cc:304] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:00:52.679482  5866 raft_consensus.cc:697] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:00:52.740864  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:52.700352  5496 raft_consensus.cc:493] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:52.742558  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:52.754333  1678 raft_consensus.cc:3060] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:00:52.667342  5793 consensus_queue.cc:1059] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 9.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.074s
I20260709 13:00:52.757109  1674 raft_consensus.cc:2410] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for term 11 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20260709 13:00:52.731531  5860 raft_consensus.cc:3060] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:00:52.706738  5827 raft_consensus.cc:515] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:52.742031  5830 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.551s	user 0.007s	sys 0.012s
I20260709 13:00:52.759872  5869 raft_consensus.cc:2749] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:00:52.760998  1748 raft_consensus.cc:1275] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 7 index: 9. Preceding OpId from leader: term: 8 index: 9. (term mismatch)
I20260709 13:00:52.775599  1748 pending_rounds.cc:85] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701: Aborting all ops after (but not including) 8
I20260709 13:00:52.775883  1748 pending_rounds.cc:107] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701: Aborting uncommitted NO_OP operation due to leader change: 7.9
I20260709 13:00:52.776203  1748 raft_consensus.cc:2889] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:00:52.731805  5777 leader_election.cc:290] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 10 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:52.771344  1754 raft_consensus.cc:1240] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 10. Current term is 11. Ops: [10.9-10.9]
W20260709 13:00:52.772744  1719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:52.780583  1680 raft_consensus.cc:1240] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 8. Current term is 9. Ops: []
I20260709 13:00:52.783044  1571 leader_election.cc:304] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:00:52.785743  1749 raft_consensus.cc:1275] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 9 index: 9. (term mismatch)
I20260709 13:00:52.755796  5661 raft_consensus.cc:3055] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:00:52.800901  1754 raft_consensus.cc:1240] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 10. Current term is 11. Ops: []
I20260709 13:00:52.786237  1749 pending_rounds.cc:85] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701: Aborting all ops after (but not including) 8
W20260709 13:00:52.778209  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 189605us
I20260709 13:00:52.764164  5849 raft_consensus.cc:3060] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:00:52.798727  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:52.803504  1752 raft_consensus.cc:3060] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:00:52.801833  1749 pending_rounds.cc:107] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701: Aborting uncommitted NO_OP operation due to leader change: 8.9
I20260709 13:00:52.804246  1749 raft_consensus.cc:2889] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:00:52.818351  5663 consensus_queue.cc:260] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.9, Last appended by leader: 9, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:52.819622  5881 log.cc:927] Time spent T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.064s	user 0.000s	sys 0.000s
I20260709 13:00:52.820749  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:52.756537  5878 raft_consensus.cc:515] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:52.769018  5804 leader_election.cc:290] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:52.801378  5661 raft_consensus.cc:740] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
W20260709 13:00:52.827908  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 52199.9us
W20260709 13:00:52.833361  1644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:52.874648  1674 raft_consensus.cc:2468] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 6.
I20260709 13:00:52.875697  1747 raft_consensus.cc:1240] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 10. Current term is 11. Ops: []
W20260709 13:00:52.877863  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 95786.1us
I20260709 13:00:52.763302  5496 raft_consensus.cc:515] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:52.764434  5648 consensus_queue.cc:1059] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 11.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.105s
I20260709 13:00:52.884050  1746 raft_consensus.cc:1240] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 10. Current term is 11. Ops: [10.9-10.9]
I20260709 13:00:52.887799  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 11 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:52.878681  1569 leader_election.cc:304] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:00:52.892699  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:52.893869  1745 raft_consensus.cc:1240] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 8. Current term is 9. Ops: []
W20260709 13:00:52.885481  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 79245.2us
I20260709 13:00:52.868069  1754 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a"
dest_uuid: "af520b67945d443b901bbeca37cc5701"
 from {username='slave'} at 127.0.0.1:43156
I20260709 13:00:52.896818  1754 raft_consensus.cc:493] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Starting forced leader election (received explicit request)
W20260709 13:00:52.896786  1643 leader_election.cc:336] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 1.747s (SENT)
I20260709 13:00:52.897541  1754 raft_consensus.cc:3060] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Advancing to term 12
W20260709 13:00:52.899171  5890 log.cc:927] Time spent T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.104s	user 0.001s	sys 0.004s
I20260709 13:00:52.900089  1671 raft_consensus.cc:2468] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 9.
I20260709 13:00:52.901126  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:52.902323  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:52.905382  1674 raft_consensus.cc:2410] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for term 9 because replica has last-logged OpId of term: 8 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 5.
I20260709 13:00:52.907205  1678 raft_consensus.cc:2468] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 11.
I20260709 13:00:52.907209  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:52.917366  1671 raft_consensus.cc:3060] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:00:52.917943  1672 raft_consensus.cc:2468] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 9.
W20260709 13:00:52.908584  5851 log.cc:927] Time spent T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.056s	user 0.000s	sys 0.001s
I20260709 13:00:52.914060  1752 raft_consensus.cc:2468] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 11.
I20260709 13:00:52.923977  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:52.923990  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:52.924724  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:52.927765  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:52.928262  1600 raft_consensus.cc:1217] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Deduplicated request from leader. Original: 6.7->[8.8-9.9]   Dedup: 8.8->[9.9-9.9]
I20260709 13:00:52.928896  1600 raft_consensus.cc:1240] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 9. Current term is 10. Ops: [8.8-9.9]
I20260709 13:00:52.930958  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:52.902493  5663 raft_consensus.cc:3060] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:00:52.902266  5738 leader_election.cc:290] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:00:52.943280  1643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:00:52.947357  5886 log.cc:927] Time spent T 00000000000000000000000000000000 P 919f6392d4f44fd68987e2c5e7ec37ca: Append to log took a long time: real 0.064s	user 0.002s	sys 0.000s
I20260709 13:00:52.947827  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
W20260709 13:00:52.949849  1643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:52.950979  1643 leader_election.cc:304] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:52.953447  1674 raft_consensus.cc:1240] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 8. Current term is 9. Ops: []
I20260709 13:00:52.958153  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:52.970168  1754 raft_consensus.cc:515] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:52.941664  5806 leader_election.cc:290] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:52.984382  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:00:52.924252  5661 consensus_queue.cc:260] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:52.991313  1754 leader_election.cc:290] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 12 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:52.992481  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:00:52.976810  5827 leader_election.cc:290] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:52.992481  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
W20260709 13:00:53.029244  1719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:52.980720  5834 leader_election.cc:290] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:00:53.034133  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 251134us
I20260709 13:00:53.035302  1571 leader_election.cc:304] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
W20260709 13:00:53.034793  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 45523.4us
I20260709 13:00:53.039862  1643 leader_election.cc:304] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 13:00:52.989019  5806 raft_consensus.cc:2749] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:00:52.929203  5866 consensus_queue.cc:237] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:00:53.043403  5882 log.cc:927] Time spent T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.124s	user 0.002s	sys 0.007s
I20260709 13:00:53.054253  2330 raft_consensus.cc:1240] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 10. Current term is 11. Ops: [10.10-10.10]
W20260709 13:00:53.047868  5482 consensus_meta.cc:287] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.761s	user 0.022s	sys 0.030s
W20260709 13:00:53.054409  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:53.062264  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:53.063104  1643 leader_election.cc:304] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:00:53.064895  1751 raft_consensus.cc:2410] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for term 10 because replica has last-logged OpId of term: 9 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 7.
I20260709 13:00:53.051285  5830 raft_consensus.cc:2804] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:00:53.056537  5661 raft_consensus.cc:3060] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:00:53.080677  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:00:53.083233  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:53.084028  2330 raft_consensus.cc:2393] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 10: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
W20260709 13:00:53.072326  5670 consensus_meta.cc:287] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.763s	user 0.014s	sys 0.003s
I20260709 13:00:53.064443  5496 leader_election.cc:290] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:53.087615  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:53.088941  2328 raft_consensus.cc:2410] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for term 13 because replica has last-logged OpId of term: 11 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 8.
I20260709 13:00:53.095018  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:00:53.072587  5830 raft_consensus.cc:493] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:00:53.099336  1643 leader_election.cc:304] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:00:53.099717  5830 raft_consensus.cc:3060] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:00:53.087070  5648 raft_consensus.cc:2749] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:00:53.085793  5878 leader_election.cc:290] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:53.090106  2301 leader_election.cc:304] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
W20260709 13:00:53.102669  5888 log.cc:927] Time spent T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.090s	user 0.000s	sys 0.001s
I20260709 13:00:53.120083  5842 consensus_queue.cc:1048] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 13:00:53.080497  5587 consensus_queue.cc:1059] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 6, Time since last communication: 0.029s
W20260709 13:00:53.123112  5807 consensus_meta.cc:287] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.804s	user 0.015s	sys 0.000s
I20260709 13:00:53.123049  5587 raft_consensus.cc:3055] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:00:53.131683  5587 raft_consensus.cc:740] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:00:53.131695  5807 raft_consensus.cc:515] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:53.132750  5587 consensus_queue.cc:260] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:53.134121  5587 raft_consensus.cc:3060] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:00:53.134546  5807 leader_election.cc:290] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 8 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:53.134641  5844 consensus_queue.cc:1048] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.046s
W20260709 13:00:53.136157  5893 log.cc:927] Time spent T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.144s	user 0.001s	sys 0.000s
I20260709 13:00:53.095036  5083 consensus_queue.cc:1059] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 10.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.044s
I20260709 13:00:53.098809  5670 raft_consensus.cc:2749] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:00:53.139192  5894 consensus_queue.cc:1059] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.053s
W20260709 13:00:53.140398  5892 log.cc:927] Time spent T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.096s	user 0.000s	sys 0.000s
W20260709 13:00:53.141772  5897 log.cc:927] Time spent T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.054s	user 0.000s	sys 0.000s
W20260709 13:00:53.144984  5819 consensus_meta.cc:287] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.862s	user 0.012s	sys 0.000s
I20260709 13:00:53.146940  5842 consensus_queue.cc:1059] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 8.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
W20260709 13:00:53.149578  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:53.150125  1644 leader_election.cc:304] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:00:53.157418  1677 raft_consensus.cc:2410] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for term 11 because replica has last-logged OpId of term: 10 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 6.
I20260709 13:00:53.159124  5830 raft_consensus.cc:515] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:53.160497  2332 raft_consensus.cc:2410] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for term 11 because replica has last-logged OpId of term: 10 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 8.
I20260709 13:00:53.172235  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:53.236519  2329 raft_consensus.cc:2468] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 7.
I20260709 13:00:53.236508  2336 raft_consensus.cc:1240] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 7. Current term is 8. Ops: [7.8-7.8]
W20260709 13:00:53.229074  5885 log.cc:927] Time spent T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.236s	user 0.001s	sys 0.000s
I20260709 13:00:53.112697  5673 raft_consensus.cc:2749] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 13:00:53.123811  5898 raft_consensus.cc:2804] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:00:53.265980  5898 raft_consensus.cc:493] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Starting leader election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:00:53.266652  5898 raft_consensus.cc:3060] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Advancing to term 10
W20260709 13:00:53.179394  5882 log.cc:927] Time spent T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.069s	user 0.000s	sys 0.000s
I20260709 13:00:53.273370  5819 raft_consensus.cc:493] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:00:53.274296  5819 raft_consensus.cc:515] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:53.277114  5819 leader_election.cc:290] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:00:53.281452  5825 consensus_meta.cc:287] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.559s	user 0.008s	sys 0.004s
I20260709 13:00:53.088092  5806 consensus_queue.cc:1048] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.044s
W20260709 13:00:53.224476  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 125224us
I20260709 13:00:53.235409  1747 raft_consensus.cc:1275] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
W20260709 13:00:53.238003  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 87952.2us
I20260709 13:00:53.293860  1644 leader_election.cc:304] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
W20260709 13:00:53.241084  5841 consensus_meta.cc:287] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.540s	user 0.011s	sys 0.001s
I20260709 13:00:53.241900  5830 leader_election.cc:290] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 12 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:53.177824  5849 raft_consensus.cc:2749] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:00:53.256831  5612 leader_election.cc:290] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 10 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:53.258711  5894 raft_consensus.cc:2749] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 13:00:53.282467  5854 consensus_queue.cc:1059] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 13:00:53.283366  5817 raft_consensus.cc:3060] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:00:53.283479  5777 raft_consensus.cc:3060] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:00:53.284579  1751 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55840 (request call id 2431) took 1202 ms. Trace:
I20260709 13:00:53.284809  5869 raft_consensus.cc:2749] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20260709 13:00:53.285816  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 250578us
I20260709 13:00:53.286283  5896 raft_consensus.cc:2804] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Leader election won for term 8
I20260709 13:00:53.286984  1748 raft_consensus.cc:1275] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 10 index: 9. (term mismatch)
I20260709 13:00:53.327400  1748 pending_rounds.cc:85] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701: Aborting all ops after (but not including) 8
I20260709 13:00:53.327802  1748 pending_rounds.cc:107] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701: Aborting uncommitted NO_OP operation due to leader change: 9.9
I20260709 13:00:53.328159  1748 raft_consensus.cc:2889] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:00:53.287508  5754 raft_consensus.cc:493] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:00:53.284617  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 454700us
I20260709 13:00:53.289718  5663 raft_consensus.cc:3055] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:00:53.223229  1671 raft_consensus.cc:2468] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 10.
I20260709 13:00:53.239802  5842 raft_consensus.cc:3055] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:00:53.296303  1643 leader_election.cc:304] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:53.317554  5860 raft_consensus.cc:3060] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:00:53.303896  1644 leader_election.cc:304] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:00:53.308578  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:00:53.341825  1569 leader_election.cc:304] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:00:53.309794  1603 raft_consensus.cc:1275] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 12 index: 9. (index mismatch)
I20260709 13:00:53.314941  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:53.316247  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:53.334874  1571 leader_election.cc:304] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
W20260709 13:00:53.316602  1719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:53.317037  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:53.345897  1752 raft_consensus.cc:2410] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for term 13 because replica has last-logged OpId of term: 11 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 8.
I20260709 13:00:53.320329  1751 rpcz_store.cc:276] 0709 13:00:52.081753 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:52.094127 (+ 12374us) service_pool.cc:224] Handling call
0709 13:00:53.284344 (+1190217us) spinlock_profiling.cc:243] Waited 926 ms on lock 0x7b54001c7854. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:00:53.284484 (+   140us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2128989184}
I20260709 13:00:53.325466  1600 raft_consensus.cc:1240] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 10. Current term is 11. Ops: []
I20260709 13:00:53.336869  1643 leader_election.cc:304] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:53.285677  5899 raft_consensus.cc:493] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:53.323503  5901 raft_consensus.cc:493] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:53.343384  1746 raft_consensus.cc:2468] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 6.
I20260709 13:00:53.349336  1646 leader_election.cc:304] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:00:53.348271  2335 raft_consensus.cc:2393] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 12: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:00:53.337756  5830 raft_consensus.cc:2749] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:00:53.343565  1749 raft_consensus.cc:2468] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 7.
W20260709 13:00:53.330053  5831 consensus_meta.cc:287] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.968s	user 0.016s	sys 0.001s
I20260709 13:00:53.359531  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:53.360788  5915 raft_consensus.cc:2804] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Leader election won for term 11
I20260709 13:00:53.361408  5901 raft_consensus.cc:515] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:53.365232  1746 raft_consensus.cc:1240] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 11. Current term is 12. Ops: []
I20260709 13:00:53.355188  5894 raft_consensus.cc:2804] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:00:53.335390  5663 raft_consensus.cc:740] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:00:53.356391  5849 raft_consensus.cc:2749] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:00:53.354795  5899 raft_consensus.cc:515] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:53.382025  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:53.384199  1605 raft_consensus.cc:1275] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20260709 13:00:53.385059  5904 raft_consensus.cc:2749] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:00:53.379576  1643 leader_election.cc:304] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:00:53.387987  1752 raft_consensus.cc:2393] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 8: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:00:53.394415  1672 raft_consensus.cc:1240] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 9. Current term is 10. Ops: []
I20260709 13:00:53.397006  5915 raft_consensus.cc:697] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:00:53.339787  5842 raft_consensus.cc:740] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
W20260709 13:00:53.402114  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 151354us
I20260709 13:00:53.327292  5873 consensus_queue.cc:1059] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.065s
I20260709 13:00:53.362668  5841 raft_consensus.cc:2749] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260709 13:00:53.408152  1717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:53.340723  5754 raft_consensus.cc:515] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:53.394308  5894 raft_consensus.cc:493] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:53.366220  5612 consensus_queue.cc:1059] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.048s
I20260709 13:00:53.367108  5869 consensus_queue.cc:1059] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 8.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.052s
I20260709 13:00:53.418581  1717 leader_election.cc:304] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:53.407575  5482 raft_consensus.cc:493] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:53.355326  5855 consensus_queue.cc:1048] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.055s
I20260709 13:00:53.422690  1717 leader_election.cc:304] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:00:53.409492  5834 raft_consensus.cc:493] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:53.415895  5878 raft_consensus.cc:2749] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:00:53.417973  5894 raft_consensus.cc:3060] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:00:53.435951  1717 leader_election.cc:304] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:53.426062  5612 raft_consensus.cc:2749] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 13:00:53.395870  5831 raft_consensus.cc:515] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:53.425999  5819 raft_consensus.cc:3055] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 7 LEADER]: Stepping down as leader of term 7
W20260709 13:00:53.461441  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 81955.3us
I20260709 13:00:53.462756  1643 leader_election.cc:304] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:00:53.444093  5804 raft_consensus.cc:2804] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:00:53.484787  2334 raft_consensus.cc:1240] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 9. Current term is 10. Ops: [9.10-9.10]
W20260709 13:00:53.486372  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 80276.2us
I20260709 13:00:53.459805  5889 raft_consensus.cc:493] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:53.432327  5482 raft_consensus.cc:515] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:00:53.493194  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 150048us
I20260709 13:00:53.505131  1569 leader_election.cc:304] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
W20260709 13:00:53.498229  1644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:53.430574  5663 consensus_queue.cc:260] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:53.503525  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:53.459287  5819 raft_consensus.cc:740] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:00:53.461108  5738 raft_consensus.cc:2749] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:00:53.513586  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:53.467471  5854 raft_consensus.cc:3055] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:00:53.509922  2299 leader_election.cc:304] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:53.518039  1569 leader_election.cc:304] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:00:53.519006  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:53.512151  1750 raft_consensus.cc:1275] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 12 index: 9. (index mismatch)
I20260709 13:00:53.476397  5804 raft_consensus.cc:493] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:53.450512  5834 raft_consensus.cc:515] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:53.513232  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:00:53.513449  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:53.511003  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:53.491114  5727 raft_consensus.cc:493] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:00:53.527710  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 65087us
I20260709 13:00:53.530362  1643 leader_election.cc:304] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:00:53.533450  2331 raft_consensus.cc:2468] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 13.
I20260709 13:00:53.466642  5915 consensus_queue.cc:237] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:53.513629  5661 raft_consensus.cc:2749] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 13:00:53.539713  1604 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 854) took 1020 ms. Trace:
I20260709 13:00:53.456101  5842 consensus_queue.cc:260] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.7, Last appended by leader: 7, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:53.534719  5878 raft_consensus.cc:2804] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:00:53.507889  5883 raft_consensus.cc:493] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:53.516796  5854 raft_consensus.cc:740] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:00:53.527351  5804 raft_consensus.cc:3060] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:00:53.523640  5663 raft_consensus.cc:3060] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:00:53.542846  1604 rpcz_store.cc:276] 0709 13:00:52.519579 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:52.617508 (+ 97929us) service_pool.cc:224] Handling call
0709 13:00:52.635018 (+ 17510us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:00:52.655082 (+ 20064us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:00:52.655144 (+    62us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:00:52.660203 (+  5059us) log.cc:844] Serialized 26 byte log entry
0709 13:00:53.187832 (+527629us) spinlock_profiling.cc:243] Waited 1.19 ms on lock 0x7b1800439cc0. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f4263057e07 00007f426beb2ab9 00007f426307469b 00007f426307293f 00007f4263074c02 00007f4265cfc20b 00007f4265d01d23 00007f4265d020d1 00007f426611da0f
0709 13:00:53.332224 (+144392us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:00:53.332269 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:53.332351 (+    82us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:00:53.539343 (+206992us) raft_consensus.cc:1685] finished
0709 13:00:53.539369 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:53.539611 (+   242us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4185,"raft.queue_time_us":5799,"raft.run_cpu_time_us":115,"raft.run_wall_time_us":116,"spinlock_wait_cycles":2736512,"thread_start_us":144155,"threads_started":1,"wal-append.queue_time_us":775182}
I20260709 13:00:53.550371  5754 leader_election.cc:290] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:53.534996  5727 raft_consensus.cc:515] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:53.551677  5903 log.cc:927] Time spent T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.074s	user 0.000s	sys 0.001s
I20260709 13:00:53.548120  5711 raft_consensus.cc:2749] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:00:53.559250  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: true dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:00:53.510480  5889 raft_consensus.cc:515] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:53.534905  5896 raft_consensus.cc:697] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:00:53.542414  5855 raft_consensus.cc:2804] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260709 13:00:53.591073  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 58753.6us
I20260709 13:00:53.591956  1643 leader_election.cc:304] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
W20260709 13:00:53.590422  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 72474.5us
I20260709 13:00:53.548050  5878 raft_consensus.cc:493] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Starting leader election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:00:53.593552  1569 leader_election.cc:304] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:00:53.595714  1643 leader_election.cc:304] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:00:53.601188  2332 raft_consensus.cc:1240] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 9. Current term is 10. Ops: []
I20260709 13:00:53.584743  5842 raft_consensus.cc:3060] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:00:53.556870  5914 raft_consensus.cc:2804] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:00:53.540766  5898 raft_consensus.cc:515] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:53.610888  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:53.611279  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:53.608892  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:53.617942  1597 raft_consensus.cc:2468] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 7.
I20260709 13:00:53.627930  1569 leader_election.cc:304] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:00:53.598191  5878 raft_consensus.cc:3060] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:00:53.595487  5855 raft_consensus.cc:493] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Starting leader election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:00:53.628746  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:53.554163  5919 consensus_queue.cc:1048] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.065s
I20260709 13:00:53.598943  5860 raft_consensus.cc:2749] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260709 13:00:53.653865  1719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:00:53.648660  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 53037.5us
I20260709 13:00:53.660174  1643 leader_election.cc:304] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 13:00:53.660475  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:00:53.634729  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 87111.8us
W20260709 13:00:53.627897  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 192045us
I20260709 13:00:53.649847  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:53.607684  5831 leader_election.cc:290] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 10 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:53.658821  1571 leader_election.cc:304] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
W20260709 13:00:53.663205  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:53.647606  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:53.663857  1600 raft_consensus.cc:2468] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 13.
I20260709 13:00:53.619920  5914 raft_consensus.cc:493] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Starting leader election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:00:53.664573  1749 raft_consensus.cc:1240] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 10. Current term is 12. Ops: []
I20260709 13:00:53.654919  1646 leader_election.cc:304] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
W20260709 13:00:53.666842  2301 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:53.599857  5581 consensus_queue.cc:1048] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.050s
I20260709 13:00:53.664839  1717 leader_election.cc:304] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:53.671895  1643 leader_election.cc:304] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:53.629593  5849 raft_consensus.cc:2749] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:00:53.565675  5819 consensus_queue.cc:260] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:53.648451  5923 raft_consensus.cc:493] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:53.630524  5855 raft_consensus.cc:3060] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:00:53.596851  5920 consensus_queue.cc:1059] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 10.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.068s
I20260709 13:00:53.626915  5899 leader_election.cc:290] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:53.658116  5924 raft_consensus.cc:2804] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:00:53.703581  5924 raft_consensus.cc:493] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Starting leader election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:00:53.703706  5899 consensus_queue.cc:1059] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 8.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 13:00:53.704079  5924 raft_consensus.cc:3060] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Advancing to term 7
W20260709 13:00:53.702494  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 47676.9us
I20260709 13:00:53.557474  5883 raft_consensus.cc:515] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:53.648464  5901 leader_election.cc:290] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:53.715482  1717 leader_election.cc:304] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
W20260709 13:00:53.720155  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 48395.5us
I20260709 13:00:53.728425  1643 leader_election.cc:304] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
W20260709 13:00:53.729061  1718 leader_election.cc:336] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Timed out: RequestConsensusVote RPC to 127.0.107.194:46261 timed out after 1.793s (SENT)
I20260709 13:00:53.739833  5920 raft_consensus.cc:3055] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:00:53.740444  5920 raft_consensus.cc:740] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:00:53.699147  5904 raft_consensus.cc:2804] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:00:53.616044  5854 consensus_queue.cc:260] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:53.713600  5923 raft_consensus.cc:515] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:53.701490  5834 leader_election.cc:290] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:53.746318  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:53.765493  1671 raft_consensus.cc:3060] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Advancing to term 10
W20260709 13:00:53.751327  1644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:53.701524  5738 raft_consensus.cc:3060] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:00:53.692009  5914 raft_consensus.cc:3060] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:00:53.749810  1717 leader_election.cc:400] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 8 election: Vote denied by peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445) with higher term. Message: Invalid argument: T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 8. Current term is 9.
I20260709 13:00:53.779647  1717 leader_election.cc:403] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 8 election: Cancelling election due to peer responding with higher term
I20260709 13:00:53.757373  5806 raft_consensus.cc:2749] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:00:53.754048  1718 leader_election.cc:304] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
W20260709 13:00:53.786162  1718 leader_election.cc:336] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 9 election: RPC error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Timed out: RequestConsensusVote RPC to 127.0.107.194:46261 timed out after 1.825s (SENT)
I20260709 13:00:53.699332  5926 raft_consensus.cc:2804] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Leader election won for term 11
I20260709 13:00:53.713742  5927 raft_consensus.cc:2804] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:00:53.761502  5849 raft_consensus.cc:2749] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:00:53.754729  5878 raft_consensus.cc:515] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:00:53.748528  5922 log.cc:927] Time spent T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.074s	user 0.000s	sys 0.000s
I20260709 13:00:53.752439  5661 consensus_queue.cc:1059] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 11.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.025s
I20260709 13:00:53.750002  5612 raft_consensus.cc:2804] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader election won for term 6
W20260709 13:00:53.799252  5925 log.cc:927] Time spent T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.088s	user 0.000s	sys 0.000s
I20260709 13:00:53.799731  5612 raft_consensus.cc:697] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 6 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:00:53.691644  5896 consensus_queue.cc:237] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:53.801024  5612 consensus_queue.cc:237] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:53.802541  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:53.803364  1673 raft_consensus.cc:2468] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 10.
W20260709 13:00:53.804317  1719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:53.746992  1676 raft_consensus.cc:1240] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 10. Current term is 11. Ops: []
I20260709 13:00:53.741520  5920 consensus_queue.cc:260] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:53.794555  5927 raft_consensus.cc:493] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:53.805830  5927 raft_consensus.cc:3060] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:00:53.805886  5920 raft_consensus.cc:3060] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Advancing to term 11
W20260709 13:00:53.747786  5916 log.cc:927] Time spent T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.191s	user 0.000s	sys 0.001s
I20260709 13:00:53.808117  5930 raft_consensus.cc:2749] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Leader election lost for term 8. Reason: Vote denied by peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445) with higher term. Message: Invalid argument: T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 8. Current term is 9.
I20260709 13:00:53.780805  1717 leader_election.cc:304] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:00:53.812749  1676 raft_consensus.cc:1275] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 9 index: 10. Preceding OpId from leader: term: 11 index: 11. (index mismatch)
I20260709 13:00:53.725327  5860 raft_consensus.cc:2804] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:00:53.821744  5860 raft_consensus.cc:493] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Starting leader election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:00:53.796413  5482 leader_election.cc:290] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:53.753643  5855 raft_consensus.cc:515] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:53.783325  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:53.740208  5933 raft_consensus.cc:2749] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
W20260709 13:00:53.741180  5918 log.cc:927] Time spent T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.092s	user 0.000s	sys 0.000s
I20260709 13:00:53.749117  5807 raft_consensus.cc:3055] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:00:53.843369  5807 raft_consensus.cc:740] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:00:53.835669  5496 raft_consensus.cc:2804] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 13:00:53.844875  5873 consensus_queue.cc:1048] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 13:00:53.762554  5901 raft_consensus.cc:493] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:53.847709  5901 raft_consensus.cc:515] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:53.849859  5727 leader_election.cc:290] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:53.822310  5860 raft_consensus.cc:3060] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:00:53.850570  5727 consensus_queue.cc:1048] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 13:00:53.741549  5904 raft_consensus.cc:493] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:53.748848  5866 raft_consensus.cc:3055] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:00:53.852116  5866 raft_consensus.cc:740] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:00:53.852504  5889 leader_election.cc:290] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:53.853298  5866 consensus_queue.cc:260] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:53.822335  5898 leader_election.cc:290] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:53.794734  5819 raft_consensus.cc:3060] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:00:53.745576  5923 leader_election.cc:290] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:53.743237  5581 consensus_queue.cc:1059] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.047s
I20260709 13:00:53.878502  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:53.790390  5787 raft_consensus.cc:3055] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 LEADER]: Stepping down as leader of term 11
I20260709 13:00:53.798357  5926 raft_consensus.cc:697] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:00:53.798954  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:53.851583  5904 raft_consensus.cc:3060] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:00:53.815548  1752 raft_consensus.cc:1240] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 8. Current term is 9. Ops: []
I20260709 13:00:53.940439  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:53.819094  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
W20260709 13:00:53.792397  5910 log.cc:927] Time spent T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.278s	user 0.002s	sys 0.000s
I20260709 13:00:53.827749  2299 leader_election.cc:304] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
W20260709 13:00:53.781543  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 115720us
I20260709 13:00:53.941313  1753 raft_consensus.cc:2468] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 10.
I20260709 13:00:53.829902  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:53.833680  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:53.947592  2331 raft_consensus.cc:2393] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 10: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:00:53.790648  5834 raft_consensus.cc:2804] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:00:53.842756  2334 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39738 (request call id 827) took 1319 ms. Trace:
I20260709 13:00:53.875190  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:53.950521  1745 raft_consensus.cc:2468] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 11.
W20260709 13:00:53.951749  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 124090us
W20260709 13:00:53.953145  2300 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:53.799261  5869 raft_consensus.cc:3055] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:00:53.850128  5901 leader_election.cc:290] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:53.899960  1671 raft_consensus.cc:2468] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 10.
I20260709 13:00:53.929774  5806 raft_consensus.cc:493] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:53.804266  5878 leader_election.cc:290] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 6 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:53.934885  1672 raft_consensus.cc:2468] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 12.
I20260709 13:00:53.817396  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:53.956513  1644 leader_election.cc:304] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:00:53.950066  2334 rpcz_store.cc:276] 0709 13:00:52.516383 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:52.517737 (+  1354us) service_pool.cc:224] Handling call
0709 13:00:52.518186 (+   449us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:00:53.485483 (+967297us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:53.835257 (+349774us) spinlock_profiling.cc:243] Waited 966 ms on lock 0x7b5400567f54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:00:53.835525 (+   268us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2221811328}
I20260709 13:00:53.945755  1569 leader_election.cc:304] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
W20260709 13:00:53.960413  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 232028us
I20260709 13:00:53.875880  5496 raft_consensus.cc:697] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 11 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:00:53.899376  5787 raft_consensus.cc:740] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:00:53.969101  1569 leader_election.cc:304] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
W20260709 13:00:53.969792  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 155974us
I20260709 13:00:53.970470  2299 leader_election.cc:304] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:00:53.900180  5866 raft_consensus.cc:3060] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Advancing to term 9
W20260709 13:00:53.966625  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 307929us
I20260709 13:00:53.968585  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:53.971884  1676 raft_consensus.cc:2368] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 8. Current term is 9.
I20260709 13:00:53.971850  1571 leader_election.cc:304] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:00:53.970803  1717 leader_election.cc:304] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:00:53.973815  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:53.958613  5834 raft_consensus.cc:493] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:53.876241  5482 consensus_queue.cc:1048] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.084s
I20260709 13:00:53.975529  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:53.973402  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:53.976317  2299 leader_election.cc:304] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:00:53.949092  5833 raft_consensus.cc:3055] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:00:53.957566  5820 raft_consensus.cc:493] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:54.000326  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:53.963027  5228 raft_consensus.cc:2804] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:00:54.004913  5228 raft_consensus.cc:493] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:54.005352  5228 raft_consensus.cc:3060] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:00:53.959093  5869 raft_consensus.cc:740] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:00:54.001837  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:54.015244  1677 raft_consensus.cc:2410] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for term 11 because replica has last-logged OpId of term: 10 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 6.
I20260709 13:00:53.989293  1671 raft_consensus.cc:2468] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 9.
I20260709 13:00:53.981514  5834 raft_consensus.cc:3060] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:00:53.981277  5930 raft_consensus.cc:2804] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Leader election won for term 11
W20260709 13:00:53.989318  1717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:54.008065  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 14 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:53.979897  5901 raft_consensus.cc:2749] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:00:54.008617  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:54.010878  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:54.010759  5926 consensus_queue.cc:237] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:54.011400  5938 raft_consensus.cc:2749] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:00:53.967188  5806 raft_consensus.cc:515] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:54.012984  5849 raft_consensus.cc:493] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:00:54.013442  1569 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:54.013173  5496 consensus_queue.cc:237] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:54.013784  5923 raft_consensus.cc:493] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:53.981279  5692 raft_consensus.cc:2804] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 13:00:53.963243  5670 consensus_queue.cc:1048] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.043s
I20260709 13:00:53.978742  5896 raft_consensus.cc:2804] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader election won for term 10
I20260709 13:00:54.049998  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:54.025141  5915 raft_consensus.cc:2749] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
W20260709 13:00:53.965972  5894 consensus_meta.cc:287] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.513s	user 0.009s	sys 0.006s
I20260709 13:00:54.025449  2334 raft_consensus.cc:1240] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 9 LEADER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 8. Current term is 9. Ops: []
I20260709 13:00:54.024260  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:54.052029  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:54.022181  5817 raft_consensus.cc:493] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:54.054966  2331 raft_consensus.cc:2468] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 12.
I20260709 13:00:54.055485  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:54.087988  1677 raft_consensus.cc:2468] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 12.
I20260709 13:00:54.087990  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:54.001034  5833 raft_consensus.cc:740] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:00:54.011967  5787 consensus_queue.cc:260] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:00:54.069926  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 98134.3us
I20260709 13:00:54.072098  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:54.105966  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:54.109825  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:54.015254  5873 raft_consensus.cc:2804] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:00:54.051326  2333 raft_consensus.cc:2410] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for term 14 because replica has last-logged OpId of term: 12 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 9.
I20260709 13:00:54.093971  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
W20260709 13:00:54.081713  5885 log.cc:927] Time spent T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.052s	user 0.001s	sys 0.000s
I20260709 13:00:54.132051  1674 raft_consensus.cc:2368] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 10. Current term is 11.
I20260709 13:00:54.134723  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:54.017590  5820 raft_consensus.cc:515] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:54.121302  5777 consensus_meta.cc:287] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.796s	user 0.019s	sys 0.001s
I20260709 13:00:54.135488  1604 raft_consensus.cc:2376] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 11. Re-sending same reply.
I20260709 13:00:54.090742  5938 raft_consensus.cc:493] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:00:54.093016  5692 raft_consensus.cc:697] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 10 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:00:54.136296  5943 raft_consensus.cc:493] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:00:54.149925  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:54.168823  1677 raft_consensus.cc:2410] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for term 11 because replica has last-logged OpId of term: 10 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 6.
W20260709 13:00:54.151297  1570 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:54.084743  5923 raft_consensus.cc:515] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:54.132474  5896 raft_consensus.cc:697] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:00:54.156162  5820 leader_election.cc:290] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:54.138522  5855 leader_election.cc:290] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 7 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:54.145470  5777 raft_consensus.cc:2749] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:00:54.078088  5849 raft_consensus.cc:515] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:54.166106  1674 raft_consensus.cc:1240] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 9. Current term is 10. Ops: [9.10-9.10]
I20260709 13:00:54.177057  1671 raft_consensus.cc:2468] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 10.
I20260709 13:00:54.199290  1717 leader_election.cc:304] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:54.186671  1602 raft_consensus.cc:1240] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 8. Current term is 9. Ops: []
W20260709 13:00:54.145895  5663 consensus_meta.cc:287] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.579s	user 0.011s	sys 0.005s
I20260709 13:00:54.125288  5873 raft_consensus.cc:493] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:00:54.188230  2301 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:54.094489  5817 raft_consensus.cc:515] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:54.207687   431 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:00:54.206063  1717 leader_election.cc:304] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:00:54.230242  2333 raft_consensus.cc:1240] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 10. Current term is 11. Ops: [10.10-10.10]
I20260709 13:00:54.231769  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:54.173552  5482 raft_consensus.cc:493] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:00:54.183868  5912 log.cc:927] Time spent T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.657s	user 0.007s	sys 0.001s
I20260709 13:00:54.131783  5924 raft_consensus.cc:515] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } }
I20260709 13:00:54.235733  1600 raft_consensus.cc:2468] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 9.
I20260709 13:00:54.238263  1748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43156 (request call id 761) took 1439 ms. Trace:
I20260709 13:00:54.188316  5854 raft_consensus.cc:3060] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:00:54.190502  5738 raft_consensus.cc:2749] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:00:54.189456  5858 raft_consensus.cc:3055] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:00:54.236279  2329 raft_consensus.cc:3060] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:00:54.174659  5915 raft_consensus.cc:493] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:54.095368  5894 raft_consensus.cc:515] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:54.203982  5825 raft_consensus.cc:493] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:54.283648  1674 raft_consensus.cc:1240] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 6. Current term is 7. Ops: []
I20260709 13:00:54.205449  5787 raft_consensus.cc:3060] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:00:54.176415  1675 raft_consensus.cc:2468] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 6.
I20260709 13:00:54.288362  1675 raft_consensus.cc:1240] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 7. Current term is 8. Ops: [7.8-7.8]
I20260709 13:00:54.187680  5849 leader_election.cc:290] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:54.174965  5943 raft_consensus.cc:515] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:54.259220  1748 rpcz_store.cc:276] 0709 13:00:52.799235 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:52.799484 (+   249us) service_pool.cc:224] Handling call
0709 13:00:53.286466 (+486982us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:53.332074 (+ 45608us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:54.234675 (+902601us) spinlock_profiling.cc:243] Waited 440 ms on lock 0x7b54001c7850. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661618ad 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:00:54.238169 (+  3494us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1012222848}
I20260709 13:00:54.226263  5873 raft_consensus.cc:3060] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:00:54.291191  1675 raft_consensus.cc:1240] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 5. Current term is 6. Ops: []
I20260709 13:00:54.291612  5943 leader_election.cc:290] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:54.292769  1675 raft_consensus.cc:1240] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 8. Current term is 9. Ops: []
I20260709 13:00:54.294283  1675 raft_consensus.cc:1240] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 9. Current term is 10. Ops: []
I20260709 13:00:54.234560  5833 consensus_queue.cc:260] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:54.298920  1754 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43156 (request call id 774) took 1012 ms. Trace:
I20260709 13:00:54.169658  5938 raft_consensus.cc:515] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:54.300519  5820 consensus_queue.cc:1059] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 9.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 13:00:54.263352  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:00:54.302570  5949 log.cc:927] Time spent T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.128s	user 0.001s	sys 0.000s
I20260709 13:00:54.302600  5825 raft_consensus.cc:515] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:54.092005  5869 consensus_queue.cc:260] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:54.306843  5869 raft_consensus.cc:3060] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:00:54.282706  5915 raft_consensus.cc:515] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:54.309607  5692 consensus_queue.cc:237] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:54.225265  5855 raft_consensus.cc:493] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:54.312925  5855 raft_consensus.cc:515] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:54.313805  5924 leader_election.cc:290] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 7 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:54.299360  1754 rpcz_store.cc:276] 0709 13:00:53.286252 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:53.286795 (+   543us) service_pool.cc:224] Handling call
0709 13:00:53.287220 (+   425us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:00:53.287606 (+   386us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:00:53.287635 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:00:53.288220 (+   585us) log.cc:844] Serialized 26 byte log entry
0709 13:00:53.425819 (+137599us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:00:53.444314 (+ 18495us) log.cc:844] Serialized 14 byte log entry
0709 13:00:53.444615 (+   301us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:53.444742 (+   127us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:00:54.183133 (+738391us) spinlock_profiling.cc:243] Waited 10.9 ms on lock 0x7b4400022b4c. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f4262f9bc85 00007f4262f9c6a2 00007f4265d0e1db 00007f4265d09aaf 00007f4265cfd86e 00007f4265cfcbb6 00007f4265cfc7d8 00007f4265d06221 00007f4265d061d9 00007f4265d06169
0709 13:00:54.298538 (+115405us) raft_consensus.cc:1685] finished
0709 13:00:54.298706 (+   168us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:54.298843 (+   137us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":295,"prepare.queue_time_us":24778,"prepare.run_cpu_time_us":57,"prepare.run_wall_time_us":58,"raft.queue_time_us":3924,"raft.run_cpu_time_us":292,"raft.run_wall_time_us":8982,"spinlock_wait_cycles":24984832,"thread_start_us":137128,"threads_started":1,"wal-append.queue_time_us":175743}
I20260709 13:00:54.323567  5806 leader_election.cc:290] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:54.325156  5951 raft_consensus.cc:493] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:00:54.326131  5951 raft_consensus.cc:515] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } }
I20260709 13:00:54.334165  5938 leader_election.cc:290] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:00:54.345378  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 57926.1us
I20260709 13:00:54.349319  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:54.350206  2333 raft_consensus.cc:2468] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 11.
I20260709 13:00:54.354995  5663 raft_consensus.cc:493] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:54.355450  5738 raft_consensus.cc:2749] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:00:54.269487  5858 raft_consensus.cc:740] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:00:54.358110  1644 leader_election.cc:304] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:00:54.253434  5823 consensus_queue.cc:1059] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.047s
W20260709 13:00:54.371889  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 165905us
I20260709 13:00:54.259227  5482 raft_consensus.cc:515] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:00:54.374065  1570 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:00:54.376525  1570 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:54.375023  1717 leader_election.cc:304] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:54.379995  1717 leader_election.cc:304] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:54.356104  5663 raft_consensus.cc:515] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:54.334404  5915 leader_election.cc:290] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:54.357064  5738 raft_consensus.cc:2804] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:00:54.388176  1717 leader_election.cc:304] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 13:00:54.359468  5804 consensus_meta.cc:287] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.765s	user 0.005s	sys 0.013s
I20260709 13:00:54.358162  5817 leader_election.cc:290] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:54.375765  5806 raft_consensus.cc:2804] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:00:54.376394  5896 consensus_queue.cc:237] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:54.417892  1717 leader_election.cc:304] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:00:54.406347  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:54.456921  1672 raft_consensus.cc:2410] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for term 11 because replica has last-logged OpId of term: 10 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 6.
I20260709 13:00:54.450822  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:00:54.460177  2333 raft_consensus.cc:3060] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:00:54.436671  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:54.464039  1598 raft_consensus.cc:2376] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 9. Re-sending same reply.
W20260709 13:00:54.426261  5927 consensus_meta.cc:287] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.620s	user 0.012s	sys 0.004s
I20260709 13:00:54.408703  5738 raft_consensus.cc:493] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:54.419938  5804 raft_consensus.cc:515] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:54.471503  5738 raft_consensus.cc:3060] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:00:54.471436  5927 raft_consensus.cc:515] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:54.475728  1673 raft_consensus.cc:1240] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 7. Current term is 8. Ops: []
W20260709 13:00:54.430902  5914 consensus_meta.cc:287] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.636s	user 0.012s	sys 0.003s
I20260709 13:00:54.399482  5825 leader_election.cc:290] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:54.516335  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:54.428434  5663 leader_election.cc:290] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:54.474201  5804 leader_election.cc:290] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 10 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:54.396266  5855 leader_election.cc:290] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:54.534986  1643 leader_election.cc:304] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
W20260709 13:00:54.419623  5842 consensus_meta.cc:287] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.794s	user 0.011s	sys 0.005s
I20260709 13:00:54.418345  5806 raft_consensus.cc:493] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:54.542208  5806 raft_consensus.cc:3060] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:00:54.542881  5942 raft_consensus.cc:493] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:54.543677  5942 raft_consensus.cc:515] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:54.546970  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:54.450526  5923 leader_election.cc:290] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:54.447609  5951 leader_election.cc:290] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:54.550906  5663 raft_consensus.cc:493] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:00:54.551625  5663 raft_consensus.cc:515] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:54.555482  5663 leader_election.cc:290] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:54.556408  5948 raft_consensus.cc:493] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:54.562031  2329 raft_consensus.cc:2468] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 12.
I20260709 13:00:54.440198  5817 raft_consensus.cc:2749] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:00:54.444363  5894 leader_election.cc:290] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 8 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:54.440099  5953 raft_consensus.cc:2749] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:00:54.557255  5948 raft_consensus.cc:515] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:54.394397  5930 raft_consensus.cc:697] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 11 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:00:54.443475  5962 raft_consensus.cc:2749] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:00:54.537122  5777 raft_consensus.cc:493] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:54.558190  5806 raft_consensus.cc:515] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:54.611727  5777 raft_consensus.cc:515] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:54.537938  5915 raft_consensus.cc:493] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:54.540390  5943 raft_consensus.cc:493] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:54.614825  5915 raft_consensus.cc:515] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:54.615247  5943 raft_consensus.cc:515] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:54.617872  5915 leader_election.cc:290] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:54.537251  5827 raft_consensus.cc:2804] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader election won for term 9
I20260709 13:00:54.537542  1569 leader_election.cc:304] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:00:54.397290  5482 leader_election.cc:290] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:54.600270  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:54.620683  1672 raft_consensus.cc:2376] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 10. Re-sending same reply.
I20260709 13:00:54.621829  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: true dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:54.622390  1672 raft_consensus.cc:3060] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Advancing to term 12
I20260709 13:00:54.594568  5952 raft_consensus.cc:2804] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Leader election won for term 11
W20260709 13:00:54.624542  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 87104.8us
I20260709 13:00:54.626286  1569 leader_election.cc:304] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
W20260709 13:00:54.645339  1680 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.568s	user 0.001s	sys 0.018s
W20260709 13:00:54.645941  1680 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.569s	user 0.001s	sys 0.018s
I20260709 13:00:54.431488  5858 consensus_queue.cc:260] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:54.522059  5914 raft_consensus.cc:515] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:00:54.673214  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 47020.5us
W20260709 13:00:54.673936  1719 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.787s	user 0.009s	sys 0.010s
W20260709 13:00:54.674520  1719 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.788s	user 0.009s	sys 0.010s
W20260709 13:00:54.636273  5961 log.cc:927] Time spent T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.077s	user 0.000s	sys 0.001s
W20260709 13:00:54.670305  1718 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.616s	user 0.013s	sys 0.010s
I20260709 13:00:54.676460  1646 leader_election.cc:304] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:00:54.682157  1646 leader_election.cc:304] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
W20260709 13:00:54.683595  1718 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.629s	user 0.013s	sys 0.010s
W20260709 13:00:54.683974  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 629753us
I20260709 13:00:54.683609  1672 raft_consensus.cc:2468] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 12.
I20260709 13:00:54.674336  1569 leader_election.cc:304] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:00:54.624163  5943 leader_election.cc:290] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:54.623391  5777 leader_election.cc:290] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:54.624302  5806 leader_election.cc:290] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 11 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:54.706202  5943 raft_consensus.cc:3060] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:00:54.636920  5923 raft_consensus.cc:2749] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260709 13:00:54.674569  5866 consensus_meta.cc:287] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.694s	user 0.016s	sys 0.004s
I20260709 13:00:54.715658  1569 leader_election.cc:304] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:00:54.623819  5827 raft_consensus.cc:697] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 9 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:00:54.657061  5915 raft_consensus.cc:2749] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:00:54.715978  1598 raft_consensus.cc:2393] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 10: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:00:54.657378  5952 raft_consensus.cc:697] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 11 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:00:54.720459  1569 leader_election.cc:304] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
W20260709 13:00:54.698625  5228 consensus_meta.cc:287] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.693s	user 0.014s	sys 0.005s
W20260709 13:00:54.703199  5819 consensus_meta.cc:287] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.847s	user 0.010s	sys 0.009s
W20260709 13:00:54.724228  1718 leader_election.cc:336] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Timed out: RequestConsensusVote RPC to 127.0.107.194:46261 timed out after 2.118s (SENT)
I20260709 13:00:54.730082  5952 consensus_queue.cc:237] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:54.734421  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:54.734841  5228 raft_consensus.cc:515] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:54.736927  2329 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39738 (request call id 836) took 1236 ms. Trace:
I20260709 13:00:54.737421  2329 rpcz_store.cc:276] 0709 13:00:53.500559 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:53.501852 (+  1293us) service_pool.cc:224] Handling call
0709 13:00:54.736635 (+1234783us) spinlock_profiling.cc:243] Waited 648 ms on lock 0x7b54006bfb54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:00:54.736783 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1490043648}
I20260709 13:00:54.737975  5228 leader_election.cc:290] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 11 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:54.718750  5817 raft_consensus.cc:493] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:54.750885  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
W20260709 13:00:54.741210  5904 consensus_meta.cc:287] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.783s	user 0.007s	sys 0.009s
W20260709 13:00:54.741704  1717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:54.752866  1717 leader_election.cc:304] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
W20260709 13:00:54.753516  1718 leader_election.cc:336] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Timed out: RequestConsensusVote RPC to 127.0.107.194:46261 timed out after 2.118s (SENT)
I20260709 13:00:54.742594  5827 consensus_queue.cc:237] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:54.743875  1679 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57092 (request call id 892) took 1393 ms. Trace:
I20260709 13:00:54.745918  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:54.703508  5825 raft_consensus.cc:2804] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:00:54.703749  5924 raft_consensus.cc:493] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:54.715287  5962 raft_consensus.cc:493] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:54.704001  5894 raft_consensus.cc:2804] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:00:54.751609  1600 raft_consensus.cc:2376] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 9. Re-sending same reply.
I20260709 13:00:54.755715  1679 rpcz_store.cc:276] 0709 13:00:53.350133 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:53.363843 (+ 13710us) service_pool.cc:224] Handling call
0709 13:00:53.399010 (+ 35167us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:53.759866 (+360856us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 13:00:53.760262 (+   396us) log.cc:844] Serialized 14 byte log entry
0709 13:00:53.900789 (+140527us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:00:53.900880 (+    91us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:00:53.900946 (+    66us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:54.703886 (+802940us) spinlock_profiling.cc:243] Waited 33 ms on lock 0x7b180025ebe0. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f42630432a6 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419
0709 13:00:54.743574 (+ 39688us) spinlock_profiling.cc:243] Waited 360 ms on lock 0x7b540025a154. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:00:54.743645 (+    71us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:54.743782 (+   137us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":258,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":40,"raft.queue_time_us":492,"raft.run_cpu_time_us":205,"raft.run_wall_time_us":10697,"spinlock_wait_cycles":904868352,"thread_start_us":139713,"threads_started":1,"wal-append.queue_time_us":260478}
I20260709 13:00:54.757886  1597 raft_consensus.cc:2376] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 8. Re-sending same reply.
I20260709 13:00:54.773753  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:54.776533  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:54.777182  1600 raft_consensus.cc:2376] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 8. Re-sending same reply.
I20260709 13:00:54.778075  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 9 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:54.781255  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:00:54.782150  1570 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:54.756563  5942 leader_election.cc:290] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:00:54.772696  5854 consensus_meta.cc:287] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.508s	user 0.008s	sys 0.008s
W20260709 13:00:54.677891  1719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:00:54.755312  5860 consensus_meta.cc:287] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.904s	user 0.014s	sys 0.003s
I20260709 13:00:54.727808  5965 consensus_queue.cc:1059] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.060s
I20260709 13:00:54.728744  5923 raft_consensus.cc:2749] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:00:54.749480  5927 leader_election.cc:290] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 8 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:54.762656  5953 raft_consensus.cc:493] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:54.803881  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:54.805474  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:54.788256  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:54.769603  5819 raft_consensus.cc:2804] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:00:54.808807  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 14 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:54.762894  5817 raft_consensus.cc:515] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:54.814042  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:00:54.800810  1570 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:54.815771  1748 raft_consensus.cc:2393] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 10: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:00:54.764585  5866 raft_consensus.cc:2749] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:00:54.771113  5894 raft_consensus.cc:493] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:54.818890  5894 raft_consensus.cc:3060] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:00:54.821643  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:54.762269  5825 raft_consensus.cc:493] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:54.822810  5825 raft_consensus.cc:3060] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:00:54.772199  5924 raft_consensus.cc:515] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:54.827458  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:54.762849  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:54.807564  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:00:54.808106  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 384702us
I20260709 13:00:54.834923  1751 raft_consensus.cc:2468] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 11.
I20260709 13:00:54.757468  5904 raft_consensus.cc:515] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:54.765066  5930 consensus_queue.cc:237] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.7, Last appended by leader: 7, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:54.857688  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:54.859846  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:54.861578  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:54.813513  5953 raft_consensus.cc:515] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:54.809105  5819 raft_consensus.cc:493] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:54.810933  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:54.883376  1597 raft_consensus.cc:2468] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 10.
I20260709 13:00:54.811564  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:54.812887  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:54.817000  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:54.800472  1569 leader_election.cc:304] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:00:54.885725  1745 raft_consensus.cc:2410] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for term 9 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
W20260709 13:00:54.886004  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 85641.2us
I20260709 13:00:54.842402  5948 leader_election.cc:290] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:54.845172  5914 leader_election.cc:290] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 10 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:54.848984  5904 leader_election.cc:290] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:54.852124  1676 raft_consensus.cc:2368] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 9. Current term is 10.
W20260709 13:00:54.855206  2301 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:54.780336  5962 raft_consensus.cc:515] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:00:54.865675  5787 consensus_meta.cc:287] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.521s	user 0.016s	sys 0.001s
I20260709 13:00:54.862247  1674 raft_consensus.cc:3060] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:00:54.870872  1598 raft_consensus.cc:3060] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:00:54.870941  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:54.876204  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:54.881338  2333 raft_consensus.cc:2468] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 8.
I20260709 13:00:54.929394  1569 leader_election.cc:304] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:00:54.837271  5482 raft_consensus.cc:3055] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:00:54.902913  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:54.906775  1601 raft_consensus.cc:2468] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 12.
I20260709 13:00:54.807658  1600 raft_consensus.cc:2468] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 9.
I20260709 13:00:54.939397  1597 raft_consensus.cc:2468] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 8.
I20260709 13:00:54.940058  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:54.940341  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:54.896416  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:54.890498  5819 raft_consensus.cc:3060] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:00:54.948575  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:54.922458  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:54.924139  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:54.924886  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:54.925451  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:00:54.897336  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 143849us
W20260709 13:00:54.950630  1718 leader_election.cc:336] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Timed out: RequestConsensusVote RPC to 127.0.107.194:46261 timed out after 2.134s (SENT)
I20260709 13:00:54.951182  1718 leader_election.cc:304] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 13:00:54.937224  2301 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:54.943524  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:54.953311  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:54.953504  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:54.805970  5860 raft_consensus.cc:515] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:54.954190  1600 raft_consensus.cc:2468] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 11.
I20260709 13:00:54.947337  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:54.955447  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:54.956175  1600 raft_consensus.cc:3060] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:00:54.959651  1752 raft_consensus.cc:2393] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 11: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:00:54.959816  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:54.949280  1748 raft_consensus.cc:3060] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Advancing to term 8
I20260709 13:00:54.950050  1679 raft_consensus.cc:2410] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for term 10 because replica has last-logged OpId of term: 9 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 8.
I20260709 13:00:54.942353  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 12 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:54.963642  1751 raft_consensus.cc:2468] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 11.
I20260709 13:00:54.968586  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:54.960695  1605 raft_consensus.cc:2468] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 8.
I20260709 13:00:54.970036  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:54.970721  1605 raft_consensus.cc:3060] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:00:54.970870  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:54.970211  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:54.978361  1718 leader_election.cc:400] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 8 election: Vote denied by peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261) with higher term. Message: Invalid argument: T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 8. Current term is 9.
I20260709 13:00:54.983047  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:54.984000  1752 raft_consensus.cc:2468] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 8.
I20260709 13:00:54.985108  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:54.985900  1751 raft_consensus.cc:2393] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 11: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:00:54.992004  1718 leader_election.cc:400] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 10 election: Vote denied by peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261) with higher term. Message: Invalid argument: T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 10. Current term is 11.
I20260709 13:00:54.994128  1718 leader_election.cc:403] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 10 election: Cancelling election due to peer responding with higher term
I20260709 13:00:55.007189  2332 raft_consensus.cc:1275] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 11 index: 9. (term mismatch)
I20260709 13:00:54.937026  5844 raft_consensus.cc:493] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:00:55.007885  2332 pending_rounds.cc:85] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1: Aborting all ops after (but not including) 8
I20260709 13:00:55.054217  1680 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52766 (request call id 2474) took 1046 ms. Trace:
I20260709 13:00:55.056454  2332 pending_rounds.cc:107] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1: Aborting uncommitted NO_OP operation due to leader change: 10.9
I20260709 13:00:55.009879  1646 leader_election.cc:304] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:00:54.947371  5854 raft_consensus.cc:2804] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:00:54.937862  5482 raft_consensus.cc:740] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:00:55.027173  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:55.026906  2337 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc"
dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
mode: GRACEFUL
new_leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
 from {username='slave'} at 127.0.0.1:35790
I20260709 13:00:55.028228  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:55.063635  1601 raft_consensus.cc:2468] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 11.
I20260709 13:00:55.056499  1680 rpcz_store.cc:276] 0709 13:00:54.008085 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:54.008256 (+   171us) service_pool.cc:224] Handling call
0709 13:00:54.031117 (+ 22861us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:00:54.031669 (+   552us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:00:54.031708 (+    39us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:00:54.070228 (+ 38520us) log.cc:844] Serialized 52 byte log entry
0709 13:00:54.646504 (+576276us) raft_consensus.cc:1645] Marking committed up to 3
0709 13:00:54.646571 (+    67us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:54.646677 (+   106us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:00:55.053949 (+407272us) raft_consensus.cc:1685] finished
0709 13:00:55.053974 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:55.054140 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":705,"raft.run_cpu_time_us":151,"raft.run_wall_time_us":6321,"thread_start_us":572100,"threads_started":1,"wal-append.queue_time_us":828297}
I20260709 13:00:55.008679  5914 raft_consensus.cc:493] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
W20260709 13:00:55.071396  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 91488.2us
I20260709 13:00:55.031359  1672 raft_consensus.cc:2468] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 7.
I20260709 13:00:55.036072  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:54.958550  5777 raft_consensus.cc:2749] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:00:55.040385  5962 leader_election.cc:290] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 13:00:55.041843  5873 consensus_meta.cc:287] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.752s	user 0.012s	sys 0.007s
I20260709 13:00:55.042918  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
W20260709 13:00:55.043293  5968 log.cc:927] Time spent T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.096s	user 0.000s	sys 0.001s
I20260709 13:00:55.061492  5927 raft_consensus.cc:2804] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:00:54.961012  5787 raft_consensus.cc:493] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:00:54.947989  5904 raft_consensus.cc:493] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:00:55.055716  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 68156.7us
I20260709 13:00:54.982156  5587 raft_consensus.cc:493] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:55.033119  1571 leader_election.cc:304] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:00:55.057040  2332 raft_consensus.cc:2889] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:00:55.112695  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:55.113865  1680 raft_consensus.cc:2468] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 9.
I20260709 13:00:55.002068  5948 raft_consensus.cc:493] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:54.983664  5738 raft_consensus.cc:515] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:00:55.044735  5869 consensus_meta.cc:287] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.737s	user 0.013s	sys 0.007s
I20260709 13:00:55.099758  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:55.066128  5820 raft_consensus.cc:2749] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Leader election lost for term 10. Reason: Vote denied by peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261) with higher term. Message: Invalid argument: T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 10. Current term is 11.
I20260709 13:00:55.101653  5927 raft_consensus.cc:493] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:55.065776  1677 raft_consensus.cc:2468] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 12.
I20260709 13:00:55.109228  5962 consensus_queue.cc:1059] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 9.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 13:00:55.109704  1598 raft_consensus.cc:2468] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 7.
I20260709 13:00:55.191679  1598 raft_consensus.cc:1240] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 9. Current term is 10. Ops: []
W20260709 13:00:55.057533  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 47805.8us
I20260709 13:00:55.111235  1718 leader_election.cc:304] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:00:55.087302  5965 consensus_queue.cc:1059] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 8.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.066s
W20260709 13:00:55.130995  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 97924.1us
I20260709 13:00:55.139365  1605 raft_consensus.cc:2468] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 10.
I20260709 13:00:55.200047  1605 raft_consensus.cc:1240] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 6. Current term is 7. Ops: []
I20260709 13:00:55.062670  5228 consensus_queue.cc:1048] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.040s
I20260709 13:00:55.155030  1604 raft_consensus.cc:1240] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20260709 13:00:55.093315  5952 consensus_queue.cc:1059] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.022s
I20260709 13:00:55.157536  1601 raft_consensus.cc:2393] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 7: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:00:55.111604  5924 leader_election.cc:290] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:00:55.087837  5834 consensus_meta.cc:287] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 1.021s	user 0.013s	sys 0.005s
I20260709 13:00:55.193089  5927 raft_consensus.cc:3060] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:00:55.123528  1600 raft_consensus.cc:2468] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 6.
W20260709 13:00:55.193874  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 82759.7us
I20260709 13:00:55.216261  1718 leader_election.cc:304] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:55.216357  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:55.217583  1718 leader_election.cc:304] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:00:55.218710  1604 raft_consensus.cc:1240] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 5. Current term is 6. Ops: []
I20260709 13:00:55.088541  5914 raft_consensus.cc:515] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:55.153913  5855 raft_consensus.cc:493] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:55.194406  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:55.153519  5777 raft_consensus.cc:2804] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260709 13:00:55.206902  1570 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:55.101382  5873 raft_consensus.cc:515] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:55.258019  1680 raft_consensus.cc:2468] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 8.
W20260709 13:00:55.258659  1719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:55.209352  1598 raft_consensus.cc:1240] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 6. Current term is 7. Ops: []
W20260709 13:00:55.197338  5964 log.cc:927] Time spent T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.392s	user 0.000s	sys 0.002s
I20260709 13:00:55.061214  5844 raft_consensus.cc:515] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:55.111732  5787 raft_consensus.cc:515] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:55.213675  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:55.219583  1718 leader_election.cc:304] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:55.111701  5587 raft_consensus.cc:515] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:55.140678  5904 raft_consensus.cc:515] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:55.295383  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 76962.9us
I20260709 13:00:55.203176  5930 raft_consensus.cc:2749] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:00:55.239245  5820 raft_consensus.cc:2804] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:00:55.159509  5948 raft_consensus.cc:515] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:55.267688  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 338467us
I20260709 13:00:55.221568  5817 leader_election.cc:290] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:55.270102  1644 leader_election.cc:304] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:00:55.240865  5777 raft_consensus.cc:493] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:55.240134  5962 raft_consensus.cc:2804] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Leader election won for term 10
W20260709 13:00:55.354910  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 84928.2us
I20260709 13:00:55.299759  1596 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40154 (request call id 2166) took 1058 ms. Trace:
I20260709 13:00:55.308643  1718 leader_election.cc:304] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
W20260709 13:00:55.359328  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 50756.3us
I20260709 13:00:55.360571  1718 leader_election.cc:304] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:00:55.319123  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:55.305555  1674 raft_consensus.cc:2468] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 11.
I20260709 13:00:55.245138  1748 raft_consensus.cc:2468] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 8.
I20260709 13:00:55.340229  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:55.220098  5834 raft_consensus.cc:515] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:55.298260  5957 raft_consensus.cc:2749] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:00:55.344298  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:55.286692  5924 raft_consensus.cc:493] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:55.297096  5952 raft_consensus.cc:2749] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 13:00:55.384435  1680 raft_consensus.cc:3060] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:00:55.354317  5777 raft_consensus.cc:3060] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:00:55.362528  1674 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52778 (request call id 2066) took 1027 ms. Trace:
I20260709 13:00:55.430612  1674 rpcz_store.cc:276] 0709 13:00:54.335104 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:54.861257 (+526153us) service_pool.cc:224] Handling call
0709 13:00:55.362397 (+501140us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:00:55.306785  5860 leader_election.cc:290] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 7 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:55.369164  1747 raft_consensus.cc:1240] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 6. Current term is 7. Ops: []
I20260709 13:00:55.257690  5855 raft_consensus.cc:515] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:00:55.410622  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 226632us
I20260709 13:00:55.386813  1671 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52778 (request call id 2058) took 1236 ms. Trace:
I20260709 13:00:55.441067  2301 leader_election.cc:304] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:00:55.441496  1671 rpcz_store.cc:276] 0709 13:00:54.150121 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:54.234235 (+ 84114us) service_pool.cc:224] Handling call
0709 13:00:54.236418 (+  2183us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:00:54.237018 (+   600us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:00:54.237049 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:00:54.237606 (+   557us) log.cc:844] Serialized 26 byte log entry
0709 13:00:54.628105 (+390499us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:00:54.628152 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:54.628227 (+    75us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:00:55.386407 (+758180us) raft_consensus.cc:1685] finished
0709 13:00:55.386474 (+    67us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:55.386708 (+   234us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":164,"raft.run_cpu_time_us":173,"raft.run_wall_time_us":20347,"thread_start_us":348860,"threads_started":1,"wal-append.queue_time_us":489893}
I20260709 13:00:55.444255  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:55.445142  1671 raft_consensus.cc:3060] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:00:55.384905  1749 raft_consensus.cc:2393] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 14: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:00:55.432323  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:55.363909  1600 raft_consensus.cc:2393] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 7: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:00:55.356372  1596 rpcz_store.cc:276] 0709 13:00:54.241787 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:54.243439 (+  1652us) service_pool.cc:224] Handling call
0709 13:00:54.243866 (+   427us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:00:54.244421 (+   555us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:00:54.244460 (+    39us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:00:54.259434 (+ 14974us) log.cc:844] Serialized 52 byte log entry
0709 13:00:54.670235 (+410801us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:00:55.000386 (+330151us) log.cc:844] Serialized 14 byte log entry
0709 13:00:55.002777 (+  2391us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:55.002894 (+   117us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:00:55.299495 (+296601us) raft_consensus.cc:1685] finished
0709 13:00:55.299522 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:55.299691 (+   169us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1925,"prepare.queue_time_us":74544,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":34,"raft.queue_time_us":36704,"raft.run_cpu_time_us":342,"raft.run_wall_time_us":30972,"thread_start_us":539513,"threads_started":2,"wal-append.queue_time_us":557810}
I20260709 13:00:55.452431  1674 raft_consensus.cc:2468] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 8.
W20260709 13:00:55.385075  5963 log.cc:927] Time spent T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.083s	user 0.000s	sys 0.001s
I20260709 13:00:55.453565  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:55.384012  5948 leader_election.cc:290] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:55.398689  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:55.455608  1598 raft_consensus.cc:2393] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 7: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:00:55.410918  5496 raft_consensus.cc:2749] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
W20260709 13:00:55.378366  5979 log.cc:927] Time spent T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.080s	user 0.000s	sys 0.000s
W20260709 13:00:55.413026  5943 consensus_meta.cc:287] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.706s	user 0.014s	sys 0.004s
W20260709 13:00:55.368726  5986 log.cc:927] Time spent T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.061s	user 0.000s	sys 0.000s
I20260709 13:00:55.358253  5820 raft_consensus.cc:697] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 12 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
W20260709 13:00:55.377472  5972 log.cc:927] Time spent T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.190s	user 0.000s	sys 0.001s
W20260709 13:00:55.379107  5977 log.cc:927] Time spent T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.226s	user 0.000s	sys 0.001s
I20260709 13:00:55.454358  1674 raft_consensus.cc:2468] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 6.
I20260709 13:00:55.456674  1569 leader_election.cc:304] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:00:55.479408  1569 leader_election.cc:304] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:00:55.397322  5787 leader_election.cc:290] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:55.399457  5844 leader_election.cc:290] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:55.388444  5904 leader_election.cc:290] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:55.381760  5834 leader_election.cc:290] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 14 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:55.489465  1569 leader_election.cc:304] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:00:55.399600  5738 leader_election.cc:290] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 8 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:55.410372  5962 raft_consensus.cc:697] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 10 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:00:55.496469  5962 consensus_queue.cc:237] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.7, Last appended by leader: 7, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:55.471531  5943 raft_consensus.cc:2749] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:00:55.500021  1597 raft_consensus.cc:2468] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 11.
I20260709 13:00:55.500810  5833 raft_consensus.cc:3060] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:00:55.358774  5960 consensus_queue.cc:1059] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.122s
I20260709 13:00:55.482883  1748 raft_consensus.cc:1275] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 11 index: 11. (index mismatch)
I20260709 13:00:55.484704  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:55.486315  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:55.506924  2335 raft_consensus.cc:2468] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 11.
I20260709 13:00:55.487131  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 12 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:55.465355  5996 raft_consensus.cc:493] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:55.489537  2331 raft_consensus.cc:1240] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 10. Current term is 11. Ops: [10.10-10.10]
I20260709 13:00:55.360615  5983 raft_consensus.cc:2804] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Leader election won for term 8
I20260709 13:00:55.491519  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:00:55.490206  5915 raft_consensus.cc:493] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:55.492043  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:00:55.486887  5990 raft_consensus.cc:2804] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260709 13:00:55.493614  5982 log.cc:927] Time spent T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.136s	user 0.000s	sys 0.000s
I20260709 13:00:55.502678  5904 raft_consensus.cc:493] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:00:55.385457  5924 raft_consensus.cc:515] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:55.459880  5860 consensus_queue.cc:1059] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.028s
W20260709 13:00:55.491865  5988 log.cc:927] Time spent T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.062s	user 0.000s	sys 0.000s
I20260709 13:00:55.484745  5869 raft_consensus.cc:2804] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260709 13:00:55.513571  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 153120us
I20260709 13:00:55.513962  1597 raft_consensus.cc:1240] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 9. Current term is 10. Ops: []
I20260709 13:00:55.514768  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:55.534902  1674 raft_consensus.cc:3060] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:00:55.488145  5873 leader_election.cc:290] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 13 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:55.526484  6002 raft_consensus.cc:2804] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:00:55.537802  6002 raft_consensus.cc:493] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:55.520701  5990 raft_consensus.cc:493] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Starting leader election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:00:55.540201  5990 raft_consensus.cc:3060] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:00:55.518226  1569 leader_election.cc:304] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:00:55.544214  5923 raft_consensus.cc:2749] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:00:55.494768  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 9 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:55.545320  5923 consensus_queue.cc:1048] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 13:00:55.545972  2328 raft_consensus.cc:2468] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 8.
I20260709 13:00:55.546223  5820 consensus_queue.cc:237] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:55.526683  5983 raft_consensus.cc:697] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:00:55.518404  5996 raft_consensus.cc:515] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:55.552192  5983 consensus_queue.cc:237] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:55.526332  5915 raft_consensus.cc:515] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:55.557722  5915 leader_election.cc:290] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:55.533185  5904 raft_consensus.cc:515] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:55.562208  5587 leader_election.cc:290] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:55.538285  6002 raft_consensus.cc:3060] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:00:55.554090  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:55.564677  1752 raft_consensus.cc:3060] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:00:55.571918  1672 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52778 (request call id 2072) took 1233 ms. Trace:
I20260709 13:00:55.572448  1672 rpcz_store.cc:276] 0709 13:00:54.338270 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:54.762372 (+424102us) service_pool.cc:224] Handling call
0709 13:00:55.571667 (+809295us) spinlock_profiling.cc:243] Waited 177 ms on lock 0x7b54000d8254. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:00:55.571810 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":407511296}
I20260709 13:00:55.573963  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:55.502786  5994 raft_consensus.cc:493] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:00:55.512210  5943 raft_consensus.cc:2749] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:00:55.574584  5855 leader_election.cc:290] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:55.579059  5883 leader_election.cc:290] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:55.529979  5869 raft_consensus.cc:493] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:55.587400  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:55.586128  1571 leader_election.cc:304] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:00:55.517861  1671 raft_consensus.cc:2468] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 11.
I20260709 13:00:55.591295  1718 leader_election.cc:304] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:00:55.592597  1672 raft_consensus.cc:2376] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 6. Re-sending same reply.
I20260709 13:00:55.595163  1674 raft_consensus.cc:2410] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for term 9 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20260709 13:00:55.552909  2332 raft_consensus.cc:2468] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 11.
I20260709 13:00:55.600612  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:00:55.645787  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 54583.1us
I20260709 13:00:55.582231  5914 leader_election.cc:290] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:55.628024  1752 raft_consensus.cc:2468] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 13.
I20260709 13:00:55.566980  5904 leader_election.cc:290] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:00:55.648095  2301 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:55.650333  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:00:55.652743  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 66746.2us
I20260709 13:00:55.654526  1597 raft_consensus.cc:2468] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 9.
I20260709 13:00:55.655323  1674 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52778 (request call id 2070) took 1318 ms. Trace:
I20260709 13:00:55.636703  5915 raft_consensus.cc:493] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:55.607471  5996 leader_election.cc:290] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:55.656473  5960 raft_consensus.cc:2804] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:00:55.589296  5787 raft_consensus.cc:493] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:55.659029  2299 leader_election.cc:304] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:00:55.661031  1674 rpcz_store.cc:276] 0709 13:00:54.337233 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:55.514424 (+1177191us) service_pool.cc:224] Handling call
0709 13:00:55.655215 (+140791us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:00:55.661276  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:55.686448  2331 raft_consensus.cc:2468] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 8.
I20260709 13:00:55.662774  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:55.687508  1672 raft_consensus.cc:2376] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 11. Re-sending same reply.
I20260709 13:00:55.688685  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:55.689590  1672 raft_consensus.cc:2393] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 7: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:00:55.663837  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:55.608497  1680 raft_consensus.cc:2468] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 7.
I20260709 13:00:55.646719  5869 raft_consensus.cc:3060] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:00:55.657572  5860 raft_consensus.cc:493] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:55.661679  5855 raft_consensus.cc:3060] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:00:55.678954  2334 raft_consensus.cc:1275] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
W20260709 13:00:55.659029  5825 consensus_meta.cc:287] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.835s	user 0.009s	sys 0.009s
I20260709 13:00:55.680823  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:55.684881  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 12 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:55.686836  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:55.713346  1674 raft_consensus.cc:2468] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 9.
I20260709 13:00:55.714576  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:55.715566  1674 raft_consensus.cc:2393] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 10: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:00:55.716531  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:55.717932  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:55.678547  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:55.692189  1569 leader_election.cc:304] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:00:55.697957  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:55.699334  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
W20260709 13:00:55.699786  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 83905.9us
I20260709 13:00:55.700413  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:55.657519  5989 raft_consensus.cc:493] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:55.709476  1671 raft_consensus.cc:2468] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 8.
I20260709 13:00:55.681643  5904 raft_consensus.cc:493] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:00:55.685461  5894 consensus_meta.cc:287] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.866s	user 0.018s	sys 0.001s
I20260709 13:00:55.601606  5994 raft_consensus.cc:515] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:55.682140  5960 raft_consensus.cc:493] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:00:55.691664  5612 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.844s	user 0.016s	sys 0.040s
I20260709 13:00:55.773666  1643 leader_election.cc:304] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:55.767923  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:55.719146  5860 raft_consensus.cc:515] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:00:55.761016  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 90719.4us
I20260709 13:00:55.768201  1752 raft_consensus.cc:1275] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 11 index: 9. (index mismatch)
I20260709 13:00:55.788021  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:55.796505  2335 raft_consensus.cc:2393] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 13: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:00:55.724748  1674 raft_consensus.cc:2393] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 9: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:00:55.798830  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:55.855571  1749 raft_consensus.cc:2393] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 12: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:00:55.734145  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:00:55.818028  1676 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52778 (request call id 2049) took 1833 ms (client timeout 2118 ms). Trace:
W20260709 13:00:55.857156  1676 rpcz_store.cc:269] 0709 13:00:53.974158 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:53.975327 (+  1169us) service_pool.cc:224] Handling call
0709 13:00:55.807663 (+1832336us) spinlock_profiling.cc:243] Waited 851 ms on lock 0x7b54001f3e54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:00:55.807854 (+   191us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1958284544}
I20260709 13:00:55.859112  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:55.799110  2331 raft_consensus.cc:3060] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:00:55.805994  1599 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 872) took 1572 ms. Trace:
I20260709 13:00:55.808322  1675 raft_consensus.cc:1240] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 9. Current term is 10. Ops: []
I20260709 13:00:55.811825  1752 raft_consensus.cc:1275] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260709 13:00:55.799374  1671 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52778 (request call id 2084) took 1076 ms. Trace:
I20260709 13:00:55.823083  6009 raft_consensus.cc:2749] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:00:55.695701  5787 raft_consensus.cc:515] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:55.831892  5994 leader_election.cc:290] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:55.774120  5991 raft_consensus.cc:2804] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 13:00:55.790357  5914 raft_consensus.cc:493] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:55.812217  1747 raft_consensus.cc:2468] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 9.
W20260709 13:00:55.805188  1718 leader_election.cc:336] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Timed out: RequestConsensusVote RPC to 127.0.107.194:46261 timed out after 2.817s (SENT)
I20260709 13:00:55.879254  1718 leader_election.cc:304] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
W20260709 13:00:55.796824  5985 log.cc:927] Time spent T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.294s	user 0.003s	sys 0.000s
I20260709 13:00:55.689958  5915 raft_consensus.cc:515] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:55.845412  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 147881us
I20260709 13:00:55.884109  5991 raft_consensus.cc:697] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 11 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:00:55.884739  2301 leader_election.cc:304] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:00:55.887681  5994 raft_consensus.cc:2804] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 13:00:55.888609  5994 raft_consensus.cc:697] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 13 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:00:55.837244  5960 raft_consensus.cc:3060] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:00:55.891366  1644 leader_election.cc:304] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:00:55.893348  1600 raft_consensus.cc:2468] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 8.
W20260709 13:00:55.801801  5998 log.cc:927] Time spent T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.100s	user 0.000s	sys 0.000s
I20260709 13:00:55.851661  1672 raft_consensus.cc:2393] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 6: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:00:55.895630  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:55.895001  5807 consensus_queue.cc:260] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:55.864038  1671 rpcz_store.cc:276] 0709 13:00:54.722897 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:55.663496 (+940599us) service_pool.cc:224] Handling call
0709 13:00:55.799278 (+135782us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:00:55.896878  5807 raft_consensus.cc:3060] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:00:55.897158  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:55.897922  1671 raft_consensus.cc:2468] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 9.
I20260709 13:00:55.867138  1749 raft_consensus.cc:1275] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
W20260709 13:00:55.834277  5927 consensus_meta.cc:287] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.594s	user 0.013s	sys 0.003s
W20260709 13:00:55.899245  5819 consensus_meta.cc:287] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.917s	user 0.009s	sys 0.008s
I20260709 13:00:55.898943  5927 raft_consensus.cc:515] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:55.858028  5943 raft_consensus.cc:2749] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:00:55.899731  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:55.900519  1671 raft_consensus.cc:2468] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 9.
I20260709 13:00:55.900677  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:55.901342  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:55.874271  5951 consensus_queue.cc:1048] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 13:00:55.902160  5927 leader_election.cc:290] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 12 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:55.763031  5924 leader_election.cc:290] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:55.860729  1599 rpcz_store.cc:276] 0709 13:00:54.233590 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:54.236608 (+  3018us) service_pool.cc:224] Handling call
0709 13:00:54.237010 (+   402us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:00:54.349889 (+112879us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 13:00:54.642804 (+292915us) log.cc:844] Serialized 14 byte log entry
0709 13:00:55.104492 (+461688us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:00:55.154583 (+ 50091us) log.cc:844] Serialized 26 byte log entry
0709 13:00:55.154762 (+   179us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:00:55.154796 (+    34us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:55.804534 (+649738us) spinlock_profiling.cc:243] Waited 25.6 ms on lock 0x7b18003f5740. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f42630432a6 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419
0709 13:00:55.805622 (+  1088us) spinlock_profiling.cc:243] Waited 106 ms on lock 0x7b5400071054. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:00:55.805678 (+    56us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:00:55.805752 (+    74us) raft_consensus.cc:1685] finished
0709 13:00:55.805765 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:55.805921 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":24009,"prepare.queue_time_us":394558,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":35,"raft.queue_time_us":36475,"raft.run_cpu_time_us":470,"raft.run_wall_time_us":22461,"spinlock_wait_cycles":303368320,"thread_start_us":646308,"threads_started":2,"wal-append.queue_time_us":586946}
I20260709 13:00:55.881891  1571 leader_election.cc:304] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
W20260709 13:00:55.882179  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 77018.8us
I20260709 13:00:55.762384  5883 consensus_queue.cc:1048] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.110s
I20260709 13:00:55.796648  5894 raft_consensus.cc:515] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:55.890972  5914 raft_consensus.cc:515] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:55.853737  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 161667us
I20260709 13:00:55.932132  5894 leader_election.cc:290] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 12 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:55.935957  1679 raft_consensus.cc:2368] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 11. Current term is 12.
W20260709 13:00:55.796428  5612 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.949s	user 0.016s	sys 0.041s
I20260709 13:00:55.901433  1680 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52778 (request call id 2085) took 1177 ms. Trace:
I20260709 13:00:55.903159  1671 raft_consensus.cc:3060] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:00:55.906520  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:55.908183  1718 leader_election.cc:304] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 13:00:55.874152  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 100588us
I20260709 13:00:55.909205  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:55.755546  5825 raft_consensus.cc:515] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:55.904088  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 9 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:55.911348  6016 raft_consensus.cc:2804] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:00:55.940285  1747 raft_consensus.cc:2376] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 13. Re-sending same reply.
I20260709 13:00:55.912469  5860 leader_election.cc:290] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:55.914299  5754 consensus_queue.cc:1048] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 13:00:55.915299  5943 raft_consensus.cc:2804] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:00:55.916828  1672 raft_consensus.cc:2468] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 10.
I20260709 13:00:55.918628  5787 leader_election.cc:290] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:00:55.920133  2301 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:55.921193  1673 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57092 (request call id 905) took 1951 ms. Trace:
I20260709 13:00:56.153205  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:56.153956  1673 rpcz_store.cc:276] 0709 13:00:53.969869 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:53.970014 (+   145us) service_pool.cc:224] Handling call
0709 13:00:54.002736 (+ 32722us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:54.511645 (+508909us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:55.920820 (+1409175us) spinlock_profiling.cc:243] Waited 473 ms on lock 0x7b540025add4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:00:55.921091 (+   271us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1087141632}
I20260709 13:00:55.905432  5969 raft_consensus.cc:3060] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:00:55.941075  1752 raft_consensus.cc:2468] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 9.
I20260709 13:00:55.939507  1680 rpcz_store.cc:276] 0709 13:00:54.723672 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:55.882703 (+1159031us) service_pool.cc:224] Handling call
0709 13:00:55.901364 (+ 18661us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:00:55.944258  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:55.941095  1643 leader_election.cc:304] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:00:56.160712  2331 raft_consensus.cc:2468] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 14.
I20260709 13:00:56.163704  1747 raft_consensus.cc:2468] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 8.
I20260709 13:00:55.899792  5819 raft_consensus.cc:515] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:55.939209  5951 raft_consensus.cc:493] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:55.943127  5930 raft_consensus.cc:2749] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 12 LEADER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 13:00:55.947345  5899 raft_consensus.cc:493] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:55.868954  5923 consensus_queue.cc:1048] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.067s
I20260709 13:00:55.961380  1671 raft_consensus.cc:2468] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 13.
I20260709 13:00:56.229852  1750 raft_consensus.cc:2393] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 10: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:00:56.160717  5957 raft_consensus.cc:493] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
W20260709 13:00:55.962963  5946 log.cc:927] Time spent T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 1.913s	user 0.008s	sys 0.000s
I20260709 13:00:55.946372  5612 raft_consensus.cc:493] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:55.950315  1749 raft_consensus.cc:2410] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 for term 9 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
W20260709 13:00:55.969278  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 61202.9us
I20260709 13:00:55.936749  5960 raft_consensus.cc:515] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:55.973412  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:00:55.968971  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 77703.3us
I20260709 13:00:55.987747  1605 raft_consensus.cc:1275] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20260709 13:00:55.974287  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:56.155463  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:56.133749  5943 raft_consensus.cc:493] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:55.932205  5914 leader_election.cc:290] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:56.166455  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:55.885859  5991 consensus_queue.cc:237] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.8, Last appended by leader: 8, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:55.950079  5978 log.cc:927] Time spent T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.754s	user 0.000s	sys 0.001s
I20260709 13:00:56.173462  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:56.165784  2301 leader_election.cc:304] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:00:55.869992  5989 raft_consensus.cc:515] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:56.179473  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 9 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:56.183584  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:55.946489  6016 raft_consensus.cc:493] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:55.952462  5825 leader_election.cc:290] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 11 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:56.197146  5999 raft_consensus.cc:493] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:56.199215  5827 raft_consensus.cc:3055] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:00:55.945786  6002 raft_consensus.cc:515] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:56.205847  5990 consensus_meta.cc:287] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.665s	user 0.018s	sys 0.000s
I20260709 13:00:56.206962  5926 raft_consensus.cc:493] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:56.211038  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:00:56.311215  6005 log.cc:927] Time spent T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.060s	user 0.001s	sys 0.000s
I20260709 13:00:56.216189  5965 raft_consensus.cc:3055] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:00:56.218930  1676 raft_consensus.cc:2468] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 10.
I20260709 13:00:55.950062  5953 leader_election.cc:290] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:56.227635  1569 leader_election.cc:304] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
W20260709 13:00:56.349659  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 122143us
I20260709 13:00:55.890147  5994 consensus_queue.cc:237] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.9, Last appended by leader: 9, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:56.230156  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:55.849133  5904 raft_consensus.cc:515] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:56.234534  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 293529us
I20260709 13:00:56.356555  1643 leader_election.cc:304] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
W20260709 13:00:56.194232  5833 consensus_meta.cc:287] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.693s	user 0.014s	sys 0.005s
I20260709 13:00:55.974184  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 12 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:56.192373  5787 raft_consensus.cc:2804] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader election won for term 9
I20260709 13:00:56.352578  5965 raft_consensus.cc:740] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:00:56.260622  1644 leader_election.cc:304] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:00:56.253938  1599 raft_consensus.cc:2468] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 11.
I20260709 13:00:56.226708  5820 raft_consensus.cc:2749] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:00:56.305881  1601 raft_consensus.cc:3060] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:00:56.361451  6019 raft_consensus.cc:493] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:56.372680  6019 raft_consensus.cc:515] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:56.311345  1673 raft_consensus.cc:2468] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 8.
I20260709 13:00:56.365336  5787 raft_consensus.cc:697] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 9 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:00:56.375742  5930 raft_consensus.cc:493] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:56.375947  5787 consensus_queue.cc:237] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:56.376679  5930 raft_consensus.cc:515] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:56.340627  5883 raft_consensus.cc:2804] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:00:56.379056  5883 raft_consensus.cc:493] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:56.379621  5883 raft_consensus.cc:3060] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:00:56.380180  5754 raft_consensus.cc:2804] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:00:56.380760  5754 raft_consensus.cc:493] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:56.381016  6002 leader_election.cc:290] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 12 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:00:56.368037  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 107520us
I20260709 13:00:56.382380  1644 leader_election.cc:304] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:00:56.383335  5738 raft_consensus.cc:3055] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:00:56.383774  5738 raft_consensus.cc:740] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:00:56.384059  6018 raft_consensus.cc:2804] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Leader election won for term 11
I20260709 13:00:56.365242  5953 raft_consensus.cc:493] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:56.385170  6018 raft_consensus.cc:697] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 11 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:00:56.256917  5612 raft_consensus.cc:515] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:56.347644  5990 raft_consensus.cc:515] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:00:56.387287  1679 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52778 (request call id 2068) took 2051 ms (client timeout 2086 ms). Trace:
I20260709 13:00:56.190104  5951 raft_consensus.cc:515] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:56.385213  5953 raft_consensus.cc:515] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:56.363350  2332 raft_consensus.cc:2393] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 12: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:00:56.389825  5990 leader_election.cc:290] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 7 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:56.248888  5957 raft_consensus.cc:515] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:56.391207  5825 raft_consensus.cc:2804] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:00:56.391709  5938 consensus_queue.cc:1048] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 13:00:56.263625  5943 raft_consensus.cc:3060] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:00:56.372013  5820 raft_consensus.cc:493] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:56.381053  5965 consensus_queue.cc:260] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:56.394003  5820 raft_consensus.cc:515] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:56.397339  5960 leader_election.cc:290] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 9 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:56.375794  1673 raft_consensus.cc:1240] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 9. Current term is 10. Ops: []
I20260709 13:00:56.311772  5914 raft_consensus.cc:493] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:56.398563  5965 raft_consensus.cc:3060] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:00:56.344398  1680 raft_consensus.cc:2468] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 8.
I20260709 13:00:56.211233  2298 leader_election.cc:304] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
W20260709 13:00:56.419723  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 208537us
I20260709 13:00:56.338956  6016 raft_consensus.cc:3060] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:00:56.358889  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:56.422941  1602 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 861) took 3128 ms. Trace:
I20260709 13:00:56.392453  5825 raft_consensus.cc:493] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:56.397533  5820 leader_election.cc:290] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:56.341964  5827 raft_consensus.cc:740] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:00:56.399664  1673 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52778 (request call id 2064) took 2096 ms. Trace:
I20260709 13:00:56.439824  1673 rpcz_store.cc:276] 0709 13:00:54.303501 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:56.375117 (+2071616us) service_pool.cc:224] Handling call
0709 13:00:56.375588 (+   471us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:56.399279 (+ 23691us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:56.399570 (+   291us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:00:56.402889  5953 leader_election.cc:290] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:56.222375  5899 raft_consensus.cc:515] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:56.404891  1674 raft_consensus.cc:1240] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 6. Current term is 7. Ops: []
I20260709 13:00:56.353307  5926 raft_consensus.cc:515] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:56.387763  1679 rpcz_store.cc:269] 0709 13:00:54.336067 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:54.999869 (+663802us) service_pool.cc:224] Handling call
0709 13:00:56.386989 (+1387120us) spinlock_profiling.cc:243] Waited 872 ms on lock 0x7b54000fe954. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:00:56.387195 (+   206us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2005878528}
I20260709 13:00:56.415091  1754 raft_consensus.cc:3060] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Advancing to term 11
W20260709 13:00:56.325728  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 160026us
I20260709 13:00:56.312471  5915 leader_election.cc:290] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:56.400868  5612 leader_election.cc:290] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:56.381186  5754 raft_consensus.cc:3060] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:00:56.384909  5738 consensus_queue.cc:260] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } }
I20260709 13:00:56.432531  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:56.426930  1602 rpcz_store.cc:276] 0709 13:00:53.288917 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:53.290527 (+  1610us) service_pool.cc:224] Handling call
0709 13:00:53.293309 (+  2782us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:54.200178 (+906869us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:56.416743 (+2216565us) spinlock_profiling.cc:243] Waited 893 ms on lock 0x7b54000f0d54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:00:56.417062 (+   319us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2053794176}
I20260709 13:00:56.449044  1671 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52778 (request call id 2065) took 2145 ms. Trace:
I20260709 13:00:56.442499  1674 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52778 (request call id 2069) took 2106 ms. Trace:
I20260709 13:00:56.449925  1671 rpcz_store.cc:276] 0709 13:00:54.304018 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:56.358489 (+2054471us) service_pool.cc:224] Handling call
0709 13:00:56.448955 (+ 90466us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:00:56.450102  1674 rpcz_store.cc:276] 0709 13:00:54.336504 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:56.403959 (+2067455us) service_pool.cc:224] Handling call
0709 13:00:56.404612 (+   653us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:56.442061 (+ 37449us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:56.442383 (+   322us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:00:56.433727  1601 raft_consensus.cc:2468] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 8.
I20260709 13:00:56.451704  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:56.452131  1671 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57092 (request call id 909) took 1982 ms. Trace:
I20260709 13:00:56.452605  1746 raft_consensus.cc:2468] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 8.
I20260709 13:00:56.452731  1671 rpcz_store.cc:276] 0709 13:00:54.470121 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:56.450950 (+1980829us) service_pool.cc:224] Handling call
0709 13:00:56.451359 (+   409us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:56.451658 (+   299us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:00:56.451698 (+    40us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:00:56.451764 (+    66us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:00:56.451815 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:56.451904 (+    89us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:56.452050 (+   146us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:00:56.399977  5819 leader_election.cc:290] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 10 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:56.404429  5904 leader_election.cc:290] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:56.445223  2301 leader_election.cc:304] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:00:56.371675  5989 leader_election.cc:290] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:56.393225  5957 leader_election.cc:290] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:56.467406  1673 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57092 (request call id 907) took 2048 ms. Trace:
I20260709 13:00:56.475109  1673 rpcz_store.cc:276] 0709 13:00:54.419048 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:56.441226 (+2022178us) service_pool.cc:224] Handling call
0709 13:00:56.466601 (+ 25375us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:56.466904 (+   303us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:00:56.466937 (+    33us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:00:56.467010 (+    73us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:00:56.467063 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:56.467148 (+    85us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:56.467305 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:00:56.470808  1598 raft_consensus.cc:1240] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 LEADER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 9. Current term is 10. Ops: []
I20260709 13:00:56.472441  1679 raft_consensus.cc:1240] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 11. Current term is 12. Ops: [11.11-11.11]
I20260709 13:00:56.477561  1679 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57092 (request call id 908) took 2008 ms. Trace:
I20260709 13:00:56.478022  1679 rpcz_store.cc:276] 0709 13:00:54.469336 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:56.444402 (+1975066us) service_pool.cc:224] Handling call
0709 13:00:56.472075 (+ 27673us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:00:56.477179 (+  5104us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:56.477480 (+   301us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:00:56.435758  5951 leader_election.cc:290] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:56.435235  5825 raft_consensus.cc:3060] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:00:56.347709  5999 raft_consensus.cc:515] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:56.439759  5990 raft_consensus.cc:493] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:56.400128  5914 raft_consensus.cc:515] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:00:56.448959  5869 consensus_meta.cc:287] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.737s	user 0.009s	sys 0.009s
I20260709 13:00:56.450378  5923 raft_consensus.cc:493] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:00:56.492990  1679 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57092 (request call id 913) took 1340 ms. Trace:
I20260709 13:00:56.450999  5894 raft_consensus.cc:2804] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Leader election won for term 8
I20260709 13:00:56.509631  1673 raft_consensus.cc:1240] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 9. Current term is 10. Ops: []
I20260709 13:00:56.510618  1673 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57092 (request call id 911) took 1415 ms. Trace:
I20260709 13:00:56.511122  1673 rpcz_store.cc:276] 0709 13:00:55.095627 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:56.489827 (+1394200us) service_pool.cc:224] Handling call
0709 13:00:56.509348 (+ 19521us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:56.510263 (+   915us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:56.510548 (+   285us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:00:56.505519  1680 raft_consensus.cc:1275] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260709 13:00:56.515024  1680 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52766 (request call id 2482) took 1508 ms. Trace:
I20260709 13:00:56.515666  1680 rpcz_store.cc:276] 0709 13:00:55.006740 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:56.475538 (+1468798us) service_pool.cc:224] Handling call
0709 13:00:56.504985 (+ 29447us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:56.514588 (+  9603us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:56.514922 (+   334us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:00:56.505070  1679 rpcz_store.cc:276] 0709 13:00:55.152545 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:56.490143 (+1337598us) service_pool.cc:224] Handling call
0709 13:00:56.492137 (+  1994us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:56.492433 (+   296us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:00:56.492477 (+    44us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:00:56.492549 (+    72us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:00:56.492607 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:56.492694 (+    87us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:56.492892 (+   198us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:00:56.507560  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:56.517762  1601 raft_consensus.cc:3060] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:00:56.472076  6009 raft_consensus.cc:493] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:00:56.520915  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 164483us
I20260709 13:00:56.486778  5738 raft_consensus.cc:3060] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:00:56.533202  1643 leader_election.cc:304] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:00:56.465483  5915 raft_consensus.cc:493] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:00:56.538121  1754 raft_consensus.cc:2468] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 11.
I20260709 13:00:56.549975  1680 raft_consensus.cc:1275] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20260709 13:00:56.552340  1680 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52766 (request call id 2489) took 1177 ms. Trace:
I20260709 13:00:56.553937  1680 rpcz_store.cc:276] 0709 13:00:55.375181 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:56.517002 (+1141821us) service_pool.cc:224] Handling call
0709 13:00:56.548934 (+ 31932us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:56.551931 (+  2997us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:56.552258 (+   327us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:00:56.520598  5869 raft_consensus.cc:515] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:00:56.571208  6015 log.cc:927] Time spent T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.229s	user 0.001s	sys 0.000s
I20260709 13:00:56.523178  5827 consensus_queue.cc:260] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:56.575953  5827 raft_consensus.cc:3060] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Advancing to term 7
W20260709 13:00:56.555668  5777 consensus_meta.cc:287] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 1.109s	user 0.010s	sys 0.009s
I20260709 13:00:56.580659  1680 raft_consensus.cc:1275] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 9 index: 10. (index mismatch)
I20260709 13:00:56.548853  5948 raft_consensus.cc:493] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:56.498232  5990 raft_consensus.cc:515] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:56.561764  5958 raft_consensus.cc:2804] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:00:56.587070  1601 raft_consensus.cc:2468] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 12.
I20260709 13:00:56.563115  5894 raft_consensus.cc:697] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 8 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:00:56.564211  5989 raft_consensus.cc:2804] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Leader election won for term 8
W20260709 13:00:56.678064  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 144946us
I20260709 13:00:56.681113  1643 leader_election.cc:304] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:56.582913  5777 raft_consensus.cc:515] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:56.524132  5923 raft_consensus.cc:515] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:56.587484  5843 raft_consensus.cc:2804] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:00:56.679629  5948 raft_consensus.cc:515] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:56.707230  5843 raft_consensus.cc:493] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:00:56.707830  5843 raft_consensus.cc:3060] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:00:56.711087  5894 consensus_queue.cc:237] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:56.592761  5958 raft_consensus.cc:493] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:00:56.714368  5926 leader_election.cc:290] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:56.714602  5958 raft_consensus.cc:3060] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Advancing to term 9
W20260709 13:00:56.716691  6025 log.cc:927] Time spent T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.136s	user 0.000s	sys 0.000s
I20260709 13:00:56.719082  5904 raft_consensus.cc:493] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:00:56.720422  1600 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40166 (request call id 2253) took 1040 ms. Trace:
I20260709 13:00:56.720978  1600 rpcz_store.cc:276] 0709 13:00:55.680005 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:55.680487 (+   482us) service_pool.cc:224] Handling call
0709 13:00:56.720094 (+1039607us) spinlock_profiling.cc:243] Waited 180 ms on lock 0x7b540028bc54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:00:56.720330 (+   236us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":414676352}
I20260709 13:00:56.730628  5914 leader_election.cc:290] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:56.733043  5899 leader_election.cc:290] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:56.719964  5904 raft_consensus.cc:515] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:00:56.583408  6020 log.cc:927] Time spent T 00000000000000000000000000000000 P 919f6392d4f44fd68987e2c5e7ec37ca: Append to log took a long time: real 0.058s	user 0.000s	sys 0.002s
I20260709 13:00:56.589197  6019 leader_election.cc:290] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:56.737824  5904 leader_election.cc:290] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:56.705406  5482 consensus_queue.cc:260] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.10, Last appended by leader: 10, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:56.739372  5482 raft_consensus.cc:3060] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:00:56.563500  5915 raft_consensus.cc:515] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:00:56.744624  5855 consensus_meta.cc:287] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 1.011s	user 0.016s	sys 0.001s
I20260709 13:00:56.745360  5855 raft_consensus.cc:2749] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:00:56.487763  6018 consensus_queue.cc:237] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:56.756894  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 75899.9us
I20260709 13:00:56.757918  1643 leader_election.cc:304] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:56.542320  6009 raft_consensus.cc:515] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:56.761502  1596 raft_consensus.cc:2468] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 14.
W20260709 13:00:56.762583  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 411807us
I20260709 13:00:56.765380  1569 leader_election.cc:304] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:00:56.760540  5904 raft_consensus.cc:2749] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:00:56.781244  1569 leader_election.cc:304] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
W20260709 13:00:56.793322  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:00:56.794921  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:00:56.795430  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:00:56.791103  5807 consensus_meta.cc:287] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.894s	user 0.016s	sys 0.001s
I20260709 13:00:56.797662  1678 raft_consensus.cc:2410] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for term 11 because replica has last-logged OpId of term: 10 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 9.
I20260709 13:00:56.797730  5807 raft_consensus.cc:2749] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:00:56.798216  5806 raft_consensus.cc:493] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:56.798812  5806 raft_consensus.cc:515] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:56.803434  5858 raft_consensus.cc:3060] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:00:56.807369  5777 leader_election.cc:290] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 9 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:56.811396  1604 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40154 (request call id 2175) took 2001 ms. Trace:
I20260709 13:00:56.811880  1604 rpcz_store.cc:276] 0709 13:00:54.810491 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:55.204360 (+393869us) service_pool.cc:224] Handling call
0709 13:00:55.218517 (+ 14157us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:55.219150 (+   633us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:56.811150 (+1592000us) spinlock_profiling.cc:243] Waited 13.8 ms on lock 0x7b540029b3d0. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661618ad 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:00:56.811317 (+   167us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":31854464}
W20260709 13:00:56.814493  1677 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52778 (request call id 2057) took 2664 ms (client timeout 1797 ms). Trace:
W20260709 13:00:56.814978  1677 rpcz_store.cc:269] 0709 13:00:54.149644 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:54.172177 (+ 22533us) service_pool.cc:224] Handling call
0709 13:00:56.814227 (+2642050us) spinlock_profiling.cc:243] Waited 711 ms on lock 0x7b54000ff354. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:00:56.814358 (+   131us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1635033344}
I20260709 13:00:56.816005  6009 leader_election.cc:290] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:56.818915  2298 leader_election.cc:304] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:00:56.823649  5930 leader_election.cc:290] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:56.823771  5869 leader_election.cc:290] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 9 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:00:56.825034  1719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:00:56.855609  1717 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.689s	user 0.028s	sys 0.071s
I20260709 13:00:56.784770  5855 raft_consensus.cc:2804] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:00:56.828545  1598 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 888) took 1241 ms. Trace:
I20260709 13:00:56.781391  5915 leader_election.cc:290] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:56.844662  5923 leader_election.cc:290] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:00:56.881074  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 62272.8us
W20260709 13:00:56.850349  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 405163us
I20260709 13:00:56.825240  5990 leader_election.cc:290] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:56.857326  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
W20260709 13:00:56.856053  1717 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.690s	user 0.028s	sys 0.071s
W20260709 13:00:56.859254  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:56.874202  1598 rpcz_store.cc:276] 0709 13:00:55.587341 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:55.587506 (+   165us) service_pool.cc:224] Handling call
0709 13:00:55.587927 (+   421us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:56.476602 (+888675us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:56.827937 (+351335us) spinlock_profiling.cc:243] Waited 882 ms on lock 0x7b540027a1d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:00:56.828453 (+   516us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2029574912}
I20260709 13:00:56.874783  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:56.906003  1746 raft_consensus.cc:2368] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for earlier term 7. Current term is 8.
W20260709 13:00:56.879536  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 98437.5us
I20260709 13:00:56.907889  1569 leader_election.cc:304] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
W20260709 13:00:56.903981  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 745980us
I20260709 13:00:56.910538  1717 leader_election.cc:304] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:00:56.921146  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:56.923063  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:00:56.876400  5969 consensus_meta.cc:287] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.688s	user 0.015s	sys 0.000s
I20260709 13:00:56.927289  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:56.931152  1569 leader_election.cc:304] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:00:56.944538  1600 raft_consensus.cc:1275] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 9 index: 10. (index mismatch)
I20260709 13:00:56.938910  1717 leader_election.cc:304] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:00:56.876827  6033 raft_consensus.cc:493] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:00:56.953445  1644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:56.937824  1746 raft_consensus.cc:2468] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 8.
I20260709 13:00:56.954883  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 9 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:56.959751  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 12 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:56.961812  1746 raft_consensus.cc:3060] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:00:56.962881  1753 raft_consensus.cc:2410] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for term 9 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20260709 13:00:56.967244  1753 raft_consensus.cc:1240] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 6. Current term is 7. Ops: []
I20260709 13:00:56.903913  5938 raft_consensus.cc:2804] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:00:56.926698  5993 raft_consensus.cc:2804] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:00:56.921108  5855 raft_consensus.cc:697] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:00:56.905251  5915 raft_consensus.cc:493] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
W20260709 13:00:56.985863  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 54810.9us
I20260709 13:00:56.995975  1569 leader_election.cc:304] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:00:56.873579  5883 raft_consensus.cc:515] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:56.951264  5969 raft_consensus.cc:2749] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:00:56.989509  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:57.003729  1745 raft_consensus.cc:2468] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 9.
W20260709 13:00:56.998759  1646 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:56.999223  1569 leader_election.cc:304] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
W20260709 13:00:56.970719  1676 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52766 (request call id 2481) took 1987 ms (client timeout 2082 ms). Trace:
W20260709 13:00:57.005394  1676 rpcz_store.cc:269] 0709 13:00:54.983631 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:55.858639 (+875008us) service_pool.cc:224] Handling call
0709 13:00:56.970393 (+1111754us) spinlock_profiling.cc:243] Waited 359 ms on lock 0x7b5400346a54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:00:56.970621 (+   228us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":825056000}
W20260709 13:00:57.005407  1754 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43156 (request call id 789) took 2013 ms (client timeout 1505 ms). Trace:
W20260709 13:00:57.007215  1754 rpcz_store.cc:269] 0709 13:00:54.991533 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:55.027600 (+ 36067us) service_pool.cc:224] Handling call
0709 13:00:57.005131 (+1977531us) spinlock_profiling.cc:243] Waited 1.32 s on lock 0x7b5400220454. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:00:57.005296 (+   165us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3036224896}
I20260709 13:00:56.932180  6028 raft_consensus.cc:2804] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:00:56.936518  6026 raft_consensus.cc:2749] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 13:00:56.883020  6016 raft_consensus.cc:515] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:56.968075  5930 raft_consensus.cc:2804] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 13:00:57.016111  5855 consensus_queue.cc:237] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:57.017174  5806 leader_election.cc:290] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:56.986129  5938 raft_consensus.cc:493] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:00:56.996205  5943 consensus_meta.cc:287] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.603s	user 0.011s	sys 0.005s
I20260709 13:00:57.018348  1675 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52778 (request call id 2063) took 2715 ms. Trace:
I20260709 13:00:56.994503  5869 raft_consensus.cc:2749] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20260709 13:00:57.019186  1750 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55840 (request call id 2444) took 2131 ms (client timeout 1763 ms). Trace:
I20260709 13:00:57.020714  5854 raft_consensus.cc:493] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:57.073633  1673 raft_consensus.cc:1240] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 11. Current term is 12. Ops: []
I20260709 13:00:57.021523  5999 leader_election.cc:290] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:57.021814  1674 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57092 (request call id 910) took 2505 ms. Trace:
I20260709 13:00:57.025578  5948 leader_election.cc:290] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:57.032469  1717 leader_election.cc:304] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:00:57.039810  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:57.040915  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
W20260709 13:00:57.070873  1750 rpcz_store.cc:269] 0709 13:00:54.875264 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:54.875476 (+   212us) service_pool.cc:224] Handling call
0709 13:00:57.006004 (+2130528us) spinlock_profiling.cc:243] Waited 1.28 s on lock 0x7b54001711d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:00:57.006190 (+   186us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2938936192}
W20260709 13:00:57.043581  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 44466.8us
I20260709 13:00:57.044137  1752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43156 (request call id 799) took 1136 ms. Trace:
I20260709 13:00:57.087181  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:57.047225  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:57.051882  1751 raft_consensus.cc:1240] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 10. Current term is 11. Ops: []
I20260709 13:00:57.088265  1753 raft_consensus.cc:2468] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 10.
I20260709 13:00:56.983907  6033 raft_consensus.cc:515] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:57.046156  5990 raft_consensus.cc:3060] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Advancing to term 12
W20260709 13:00:57.059490  1678 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52778 (request call id 2051) took 3044 ms (client timeout 2817 ms). Trace:
I20260709 13:00:57.085916  1569 leader_election.cc:304] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:00:57.086376  1752 rpcz_store.cc:276] 0709 13:00:55.907608 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:56.170076 (+262468us) service_pool.cc:224] Handling call
0709 13:00:56.170524 (+   448us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:56.393690 (+223166us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 13:00:56.394184 (+   494us) log.cc:844] Serialized 14 byte log entry
0709 13:00:56.792143 (+397959us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:00:56.792211 (+    68us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:00:56.792269 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:57.043836 (+251567us) spinlock_profiling.cc:243] Waited 223 ms on lock 0x7b5400385254. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:00:57.043900 (+    64us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:57.044059 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":400,"prepare.queue_time_us":72418,"prepare.run_cpu_time_us":101,"prepare.run_wall_time_us":7005,"raft.queue_time_us":849,"raft.run_cpu_time_us":200,"raft.run_wall_time_us":4302,"spinlock_wait_cycles":512334592,"thread_start_us":395709,"threads_started":1,"wal-append.queue_time_us":601306}
W20260709 13:00:57.059803  1747 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.500s	user 0.005s	sys 0.040s
W20260709 13:00:57.095661  1747 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.536s	user 0.006s	sys 0.040s
I20260709 13:00:57.059847  1676 raft_consensus.cc:1275] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
W20260709 13:00:57.060428  1672 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57092 (request call id 912) took 1957 ms (client timeout 2183 ms). Trace:
W20260709 13:00:57.090850  1678 rpcz_store.cc:269] 0709 13:00:54.016361 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:54.051951 (+ 35590us) service_pool.cc:224] Handling call
0709 13:00:57.059235 (+3007284us) spinlock_profiling.cc:243] Waited 2.7 s on lock 0x7b5400347e54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:00:57.059404 (+   169us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":6209941888}
W20260709 13:00:57.097296  1672 rpcz_store.cc:269] 0709 13:00:55.102620 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:55.895219 (+792599us) service_pool.cc:224] Handling call
0709 13:00:57.060178 (+1164959us) spinlock_profiling.cc:243] Waited 20.4 ms on lock 0x7b5400347454. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:00:57.060342 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":46832000}
I20260709 13:00:57.068043  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:57.069964  1675 rpcz_store.cc:276] 0709 13:00:54.303037 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:54.303184 (+   147us) service_pool.cc:224] Handling call
0709 13:00:55.808021 (+1504837us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:55.863078 (+ 55057us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:57.018080 (+1155002us) spinlock_profiling.cc:243] Waited 1.5 s on lock 0x7b54001f3e50. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661618ad 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:00:57.018242 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3459332864}
I20260709 13:00:57.030717  6028 raft_consensus.cc:493] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:57.013831  5915 raft_consensus.cc:515] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:57.003463  5957 consensus_queue.cc:1059] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.061s
I20260709 13:00:56.968243  6021 consensus_queue.cc:1059] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 9.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.105s
I20260709 13:00:57.074627  5843 raft_consensus.cc:515] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:57.078107  6000 consensus_queue.cc:1048] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.045s
I20260709 13:00:57.123008  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:57.123310  6021 raft_consensus.cc:3055] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:00:57.123804  6021 raft_consensus.cc:740] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:00:57.101428  5999 raft_consensus.cc:2804] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:00:57.124961  5999 raft_consensus.cc:493] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:57.124845  6021 consensus_queue.cc:260] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:57.101965  1569 leader_election.cc:304] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1
W20260709 13:00:57.126469  5754 consensus_meta.cc:287] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.646s	user 0.007s	sys 0.007s
I20260709 13:00:57.126976  5754 raft_consensus.cc:515] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:57.082103  1674 rpcz_store.cc:276] 0709 13:00:54.516732 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:56.466018 (+1949286us) service_pool.cc:224] Handling call
0709 13:00:56.970629 (+504611us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:56.970962 (+   333us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:00:56.971003 (+    41us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:00:56.971073 (+    70us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:00:56.971125 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:56.971207 (+    82us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:57.021595 (+ 50388us) spinlock_profiling.cc:243] Waited 504 ms on lock 0x7b5400346a50. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661618ad 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:00:57.021732 (+   137us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1159280512}
I20260709 13:00:57.072124  5938 raft_consensus.cc:3060] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:00:57.129786  5923 raft_consensus.cc:2804] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:00:57.012925  6029 raft_consensus.cc:2749] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:00:57.130261  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:57.131193  1676 raft_consensus.cc:2468] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 10.
I20260709 13:00:57.129325  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:57.134027  1646 leader_election.cc:400] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 7 election: Vote denied by peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693) with higher term. Message: Invalid argument: T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for earlier term 7. Current term is 8.
I20260709 13:00:57.134516  1646 leader_election.cc:403] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 7 election: Cancelling election due to peer responding with higher term
I20260709 13:00:57.045389  6027 consensus_queue.cc:1048] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.054s
I20260709 13:00:57.074168  5962 raft_consensus.cc:493] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:57.075124  5482 raft_consensus.cc:493] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:57.151944  1680 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52766 (request call id 2494) took 1212 ms. Trace:
I20260709 13:00:57.112246  5957 raft_consensus.cc:3055] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:00:57.118528  5869 consensus_queue.cc:1048] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 13:00:57.118839  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:00:57.119812  6030 log.cc:927] Time spent T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.062s	user 0.000s	sys 0.000s
I20260709 13:00:57.121218  5915 leader_election.cc:290] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:57.104562  6028 raft_consensus.cc:3060] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:00:57.121654  6016 leader_election.cc:290] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 12 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:57.165094  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:57.124404  5843 leader_election.cc:290] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 9 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:57.125474  5999 raft_consensus.cc:3060] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:00:57.078977  5854 raft_consensus.cc:3060] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:00:57.130291  5923 raft_consensus.cc:493] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:57.068918  5969 raft_consensus.cc:493] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:57.106828  1750 raft_consensus.cc:1240] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 9. Current term is 10. Ops: []
I20260709 13:00:57.141726  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:57.204919  1674 raft_consensus.cc:2393] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 10: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:00:57.142604  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:57.154956  5957 raft_consensus.cc:740] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:00:57.209731  2299 leader_election.cc:304] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:00:57.213944  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:57.216480  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:57.217396  1672 raft_consensus.cc:2468] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 10.
I20260709 13:00:57.142802  1678 raft_consensus.cc:1275] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 11 index: 9. (index mismatch)
I20260709 13:00:57.145452  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:57.093004  5904 raft_consensus.cc:493] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:57.226065  1569 leader_election.cc:304] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:00:57.110857  6026 consensus_queue.cc:1048] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 13:00:57.152580  1680 rpcz_store.cc:276] 0709 13:00:55.940028 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:56.582152 (+642124us) service_pool.cc:224] Handling call
0709 13:00:56.756074 (+173922us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:57.151461 (+395387us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:00:57.151499 (+    38us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:00:57.151619 (+   120us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:00:57.151664 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:57.151738 (+    74us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:57.151878 (+   140us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":40326,"raft.run_cpu_time_us":99,"raft.run_wall_time_us":99,"thread_start_us":394115,"threads_started":1}
W20260709 13:00:57.155506  1679 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.575s	user 0.009s	sys 0.029s
W20260709 13:00:57.158046  1571 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.557s	user 0.027s	sys 0.028s
I20260709 13:00:57.075508  5943 raft_consensus.cc:515] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:57.162300  1673 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52766 (request call id 2488) took 1812 ms. Trace:
I20260709 13:00:57.164099  5970 raft_consensus.cc:493] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:57.230667  1673 rpcz_store.cc:276] 0709 13:00:55.350324 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:56.512199 (+1161875us) service_pool.cc:224] Handling call
0709 13:00:56.512623 (+   424us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:57.074126 (+561503us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:57.161941 (+ 87815us) spinlock_profiling.cc:243] Waited 561 ms on lock 0x7b5400164254. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:00:57.162209 (+   268us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1289455360}
I20260709 13:00:57.163761  5873 raft_consensus.cc:493] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:57.179490  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:57.180299  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:57.232571  1569 leader_election.cc:304] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
W20260709 13:00:57.185355  1719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:57.150233  1646 leader_election.cc:304] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
W20260709 13:00:57.192636  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 434720us
I20260709 13:00:57.155179  6009 raft_consensus.cc:3060] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:00:57.223819  1598 raft_consensus.cc:2368] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 for earlier term 9. Current term is 10.
W20260709 13:00:57.194880  5738 consensus_meta.cc:287] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.635s	user 0.009s	sys 0.007s
W20260709 13:00:57.228894  1679 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.651s	user 0.009s	sys 0.029s
W20260709 13:00:57.228993  1571 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.628s	user 0.027s	sys 0.028s
W20260709 13:00:57.256141  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 655423us
I20260709 13:00:57.211623  5670 consensus_queue.cc:1059] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.018s
W20260709 13:00:57.252934  1718 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.542s	user 0.005s	sys 0.043s
W20260709 13:00:57.260473  1718 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.549s	user 0.005s	sys 0.044s
I20260709 13:00:57.229979  5899 raft_consensus.cc:493] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:00:57.254485  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 104318us
W20260709 13:00:57.261127  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 550309us
I20260709 13:00:57.209403  6021 raft_consensus.cc:3060] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:00:57.159271  5482 raft_consensus.cc:515] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:57.234733  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:57.188647  5923 raft_consensus.cc:3060] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:00:57.212092  6019 raft_consensus.cc:493] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:57.263679  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:57.158598  5962 raft_consensus.cc:515] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:57.234123  5989 raft_consensus.cc:697] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:00:57.234370  1675 raft_consensus.cc:2468] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 11.
I20260709 13:00:57.238727  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:57.275738  1671 raft_consensus.cc:3060] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:00:57.278894  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:57.279179  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:57.278923  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:57.272222  5806 raft_consensus.cc:2804] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:00:57.268787  1646 leader_election.cc:304] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
W20260709 13:00:57.286496  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:57.234035  6029 raft_consensus.cc:2749] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:00:57.254611  1643 leader_election.cc:304] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:57.233657  5904 raft_consensus.cc:515] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:57.225577  5915 raft_consensus.cc:493] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:57.260762  5970 raft_consensus.cc:515] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:57.221103  5969 raft_consensus.cc:515] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:57.303565  1718 leader_election.cc:304] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:57.259550  5873 raft_consensus.cc:515] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:57.346517  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:00:57.278120  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 68492.2us
I20260709 13:00:57.443377  1750 raft_consensus.cc:2468] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 9.
W20260709 13:00:57.353089  1644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:00:57.448349  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 179658us
I20260709 13:00:57.326114  1745 raft_consensus.cc:2410] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for term 8 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20260709 13:00:57.351001  1754 raft_consensus.cc:2410] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for term 11 because replica has last-logged OpId of term: 9 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 9.
W20260709 13:00:57.439566  2301 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:00:57.457222  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:00:57.442680  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 139235us
I20260709 13:00:57.280983  5883 leader_election.cc:290] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 10 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:00:57.322359  5958 consensus_meta.cc:287] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.607s	user 0.014s	sys 0.003s
I20260709 13:00:57.276772  5738 raft_consensus.cc:493] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:57.380323  5957 consensus_queue.cc:260] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:00:57.316433  6018 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.552s	user 0.018s	sys 0.027s
I20260709 13:00:57.470621  1671 raft_consensus.cc:2468] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 10.
W20260709 13:00:57.471295  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:57.459609  1718 leader_election.cc:304] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
W20260709 13:00:57.521267  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 61773.1us
W20260709 13:00:57.523137  1718 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.107.194:46261 timed out after 1.641s (SENT) [suppressed 22 similar messages]
W20260709 13:00:57.472697  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 218200us
W20260709 13:00:57.523592  1746 consensus_meta.cc:287] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.541s	user 0.012s	sys 0.003s
I20260709 13:00:57.524197  1643 leader_election.cc:304] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:00:57.524318  1746 raft_consensus.cc:2468] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 12.
W20260709 13:00:57.388917  6031 log.cc:927] Time spent T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.316s	user 0.000s	sys 0.001s
W20260709 13:00:57.320535  5787 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.602s	user 0.038s	sys 0.019s
I20260709 13:00:57.488693  1747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55840 (request call id 2454) took 1573 ms. Trace:
I20260709 13:00:57.404855  5869 raft_consensus.cc:2804] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:00:57.493435  5969 leader_election.cc:290] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:57.496306  5970 leader_election.cc:290] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:57.447782  5806 raft_consensus.cc:493] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:57.317106  6019 raft_consensus.cc:515] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:57.498035  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
W20260709 13:00:57.532965  5787 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.814s	user 0.038s	sys 0.022s
I20260709 13:00:57.499482  5919 raft_consensus.cc:3055] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:00:57.500216  5904 leader_election.cc:290] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:57.271265  5899 raft_consensus.cc:515] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:57.473512  6029 raft_consensus.cc:493] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:57.322160  5754 leader_election.cc:290] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 10 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:57.555099  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:57.554956  6029 raft_consensus.cc:515] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:57.556341  1750 raft_consensus.cc:2393] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 9: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:00:57.557785  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:57.502182  6016 raft_consensus.cc:2749] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:00:57.455547  5915 raft_consensus.cc:515] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:57.561662  5915 leader_election.cc:290] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:57.504074  1677 raft_consensus.cc:3060] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:00:57.505323  1752 raft_consensus.cc:2468] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 9.
I20260709 13:00:57.322371  5671 raft_consensus.cc:2804] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:00:57.562911  5671 raft_consensus.cc:493] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:57.562988  5899 leader_election.cc:290] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:57.563429  5671 raft_consensus.cc:3060] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Advancing to term 9
W20260709 13:00:57.468319  6035 log.cc:927] Time spent T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.200s	user 0.000s	sys 0.001s
I20260709 13:00:57.464217  6027 raft_consensus.cc:2749] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:00:57.525394  1748 raft_consensus.cc:1240] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 11. Current term is 12. Ops: [9.10-11.11]
I20260709 13:00:57.508255  5958 raft_consensus.cc:515] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:57.515516  5858 consensus_meta.cc:287] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.711s	user 0.009s	sys 0.008s
W20260709 13:00:57.519930  6018 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.770s	user 0.018s	sys 0.027s
I20260709 13:00:57.541213  1643 leader_election.cc:304] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
W20260709 13:00:57.535920  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 69994.3us
I20260709 13:00:57.464576  5991 consensus_queue.cc:1059] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 9.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.058s
I20260709 13:00:57.603950  2337 raft_consensus.cc:606] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Received request to transfer leadership to TS 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:00:57.604429  2337 raft_consensus.cc:614] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 13:00:57.533185  1747 rpcz_store.cc:276] 0709 13:00:55.915231 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:56.194790 (+279559us) service_pool.cc:224] Handling call
0709 13:00:56.195439 (+   649us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:00:56.474060 (+278621us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:00:56.474107 (+    47us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:00:56.476041 (+  1934us) log.cc:844] Serialized 26 byte log entry
0709 13:00:57.096112 (+620071us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:00:57.096161 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:57.096239 (+    78us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:00:57.488345 (+392106us) raft_consensus.cc:1685] finished
0709 13:00:57.488376 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:57.488604 (+   228us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":25632,"raft.run_cpu_time_us":144,"raft.run_wall_time_us":144,"thread_start_us":813554,"threads_started":2,"wal-append.queue_time_us":849125}
W20260709 13:00:57.553699  5833 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.533s	user 0.006s	sys 0.013s
W20260709 13:00:57.605342  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 372809us
I20260709 13:00:57.527818  6033 leader_election.cc:290] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:57.531107  5965 raft_consensus.cc:2804] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:00:57.348455  5692 raft_consensus.cc:493] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
W20260709 13:00:57.568230  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 535762us
I20260709 13:00:57.526743  6022 raft_consensus.cc:493] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:57.529117  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:57.571949  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:57.532069  6044 raft_consensus.cc:2804] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:00:57.572839  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:57.574190  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:57.621618  1569 leader_election.cc:304] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:00:57.574771  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:00:57.623162  2333 raft_consensus.cc:3060] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:00:57.539930  5869 raft_consensus.cc:493] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:57.552194  5806 raft_consensus.cc:3060] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Advancing to term 10
W20260709 13:00:57.603217  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 62117.3us
I20260709 13:00:57.593680  6029 leader_election.cc:290] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:57.565745  5957 raft_consensus.cc:3060] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:00:57.608067  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 12 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:00:57.643944  1569 leader_election.cc:304] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:00:57.644325  1643 leader_election.cc:304] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 13:00:57.610837  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:00:57.569381  5993 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.511s	user 0.007s	sys 0.009s
I20260709 13:00:57.565707  5919 raft_consensus.cc:740] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:00:57.661041  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:57.614414  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:57.616683  2299 leader_election.cc:304] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:00:57.617153  1677 raft_consensus.cc:2468] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 7.
I20260709 13:00:57.662523  2334 raft_consensus.cc:2468] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 11.
I20260709 13:00:57.620358  1717 leader_election.cc:304] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:00:57.571727  5777 raft_consensus.cc:2804] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:00:57.590982  5930 raft_consensus.cc:697] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 10 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:00:57.615736  5943 leader_election.cc:290] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 10 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:57.551379  5883 consensus_queue.cc:1048] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.056s
I20260709 13:00:57.569697  6046 raft_consensus.cc:2749] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 13:00:57.628062  5899 raft_consensus.cc:2804] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:00:57.692170  5834 raft_consensus.cc:2804] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 13:00:57.566203  5738 raft_consensus.cc:515] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:57.603581  5873 leader_election.cc:290] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:00:57.651528  1719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:57.619479  2328 raft_consensus.cc:2468] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 9.
I20260709 13:00:57.667961  1605 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 890) took 1456 ms. Trace:
W20260709 13:00:57.613449  5833 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.593s	user 0.006s	sys 0.013s
I20260709 13:00:57.620870  5955 raft_consensus.cc:493] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:57.621948  6044 raft_consensus.cc:493] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:00:57.680768  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 64210.1us
W20260709 13:00:57.682659  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 62455.4us
I20260709 13:00:57.734071  2333 raft_consensus.cc:2468] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 12.
I20260709 13:00:57.545184  5989 consensus_queue.cc:237] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:57.686206  2298 leader_election.cc:400] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 9 election: Vote denied by peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445) with higher term. Message: Invalid argument: T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 for earlier term 9. Current term is 10.
I20260709 13:00:57.736063  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:57.636732  5869 raft_consensus.cc:3060] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:00:57.692692  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:57.701689  5899 raft_consensus.cc:493] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting leader election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:00:57.693467  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:57.695956  1749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55840 (request call id 2457) took 1269 ms. Trace:
W20260709 13:00:57.648603  5993 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.593s	user 0.008s	sys 0.009s
I20260709 13:00:57.684859  2337 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:35790 (request call id 6) took 2688 ms. Trace:
I20260709 13:00:57.608577  5787 raft_consensus.cc:2749] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 13:00:57.678577  5994 raft_consensus.cc:3055] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 13:00:57.706194  1752 raft_consensus.cc:1275] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 9 index: 10. Preceding OpId from leader: term: 10 index: 10. (term mismatch)
I20260709 13:00:57.759529  1752 pending_rounds.cc:85] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701: Aborting all ops after (but not including) 9
I20260709 13:00:57.759871  1752 pending_rounds.cc:107] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701: Aborting uncommitted NO_OP operation due to leader change: 9.10
I20260709 13:00:57.760192  1752 raft_consensus.cc:2889] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 13:00:57.718968  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 254713us
I20260709 13:00:57.751544  1749 rpcz_store.cc:276] 0709 13:00:56.426148 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:56.426359 (+   211us) service_pool.cc:224] Handling call
0709 13:00:56.426956 (+   597us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:00:56.458032 (+ 31076us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:00:56.458076 (+    44us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:00:56.458815 (+   739us) log.cc:844] Serialized 26 byte log entry
0709 13:00:56.951341 (+492526us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:00:56.951412 (+    71us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:56.951519 (+   107us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:00:57.695557 (+744038us) raft_consensus.cc:1685] finished
0709 13:00:57.695615 (+    58us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:57.695856 (+   241us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":16997,"raft.queue_time_us":41655,"raft.run_cpu_time_us":170,"raft.run_wall_time_us":1734,"thread_start_us":442619,"threads_started":1,"wal-append.queue_time_us":689442}
I20260709 13:00:57.635540  5858 consensus_queue.cc:1048] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.063s
I20260709 13:00:57.683454  6018 raft_consensus.cc:2764] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader pre-election decision vote started in defunct term 8: won
I20260709 13:00:57.634514  5754 consensus_queue.cc:1059] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.064s
I20260709 13:00:57.704730  1605 rpcz_store.cc:276] 0709 13:00:56.211183 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:56.255295 (+ 44112us) service_pool.cc:224] Handling call
0709 13:00:56.722493 (+467198us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:56.722830 (+   337us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 13:00:57.146010 (+423180us) log.cc:844] Serialized 14 byte log entry
0709 13:00:57.567018 (+421008us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:00:57.567110 (+    92us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:00:57.567195 (+    85us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:57.567354 (+   159us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:57.667713 (+100359us) spinlock_profiling.cc:243] Waited 467 ms on lock 0x7b540028bc50. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661618ad 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:00:57.667869 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":717659,"prepare.run_cpu_time_us":89,"prepare.run_wall_time_us":8279,"raft.queue_time_us":130,"raft.run_cpu_time_us":205,"raft.run_wall_time_us":8299,"spinlock_wait_cycles":1073069312,"thread_start_us":842930,"threads_started":2,"wal-append.queue_time_us":556258}
I20260709 13:00:57.623955  5482 leader_election.cc:290] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:57.629073  5692 raft_consensus.cc:515] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:57.736407  2298 leader_election.cc:403] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 9 election: Cancelling election due to peer responding with higher term
I20260709 13:00:57.687153  1643 leader_election.cc:304] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 13:00:57.738425  1746 raft_consensus.cc:2468] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 9.
I20260709 13:00:57.629351  5965 raft_consensus.cc:697] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 14 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
W20260709 13:00:57.791152  6041 log.cc:927] Time spent T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.142s	user 0.001s	sys 0.000s
I20260709 13:00:57.629312  6022 raft_consensus.cc:515] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:57.747813  1679 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52766 (request call id 2493) took 2164 ms. Trace:
I20260709 13:00:57.793087  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:57.792181  1679 rpcz_store.cc:276] 0709 13:00:55.583761 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:56.519806 (+936045us) service_pool.cc:224] Handling call
0709 13:00:56.545817 (+ 26011us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:56.546185 (+   368us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 13:00:57.256188 (+710003us) log.cc:844] Serialized 14 byte log entry
0709 13:00:57.747365 (+491177us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:00:57.747454 (+    89us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:00:57.747518 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:57.747608 (+    90us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:57.747732 (+   124us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5019,"prepare.queue_time_us":803380,"prepare.run_cpu_time_us":117,"prepare.run_wall_time_us":9574,"raft.queue_time_us":11448,"raft.run_cpu_time_us":189,"raft.run_wall_time_us":8535,"thread_start_us":1158877,"threads_started":2}
I20260709 13:00:57.705127  5943 raft_consensus.cc:2804] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:00:57.753091  2337 rpcz_store.cc:276] 0709 13:00:54.969924 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:54.970629 (+   705us) service_pool.cc:224] Handling call
0709 13:00:57.658506 (+2687877us) spinlock_profiling.cc:243] Waited 2.54 s on lock 0x7b5400443754. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f426615c853 00007f426b12a60a 00007f42646b07d4 00007f42646b0762 00007f42646b0691 00007f42646b060c 00007f42646af8c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:00:57.658680 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5841654656}
W20260709 13:00:57.732008  5825 consensus_meta.cc:287] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 1.228s	user 0.012s	sys 0.005s
I20260709 13:00:57.803121  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:57.720150  5915 raft_consensus.cc:493] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:57.719573  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:57.826598  1747 raft_consensus.cc:3060] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:00:57.722482  5991 raft_consensus.cc:2804] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:00:57.783465  6033 consensus_queue.cc:1048] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 13:00:57.769937  5926 raft_consensus.cc:493] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:57.775954  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:57.734966  6044 raft_consensus.cc:3060] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:00:57.758965  5994 raft_consensus.cc:740] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:00:57.783804  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:57.737375  5834 raft_consensus.cc:697] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [term 8 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:00:57.788918  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:57.789237  2336 raft_consensus.cc:1275] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 12 index: 10. (index mismatch)
I20260709 13:00:57.755590  5993 raft_consensus.cc:697] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 12 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
W20260709 13:00:57.790021  6048 log.cc:927] Time spent T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.127s	user 0.000s	sys 0.001s
I20260709 13:00:57.769142  5873 raft_consensus.cc:493] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:57.790387  5692 leader_election.cc:290] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:57.794507  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:57.757210  5899 raft_consensus.cc:3060] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:00:57.799804  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 12 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:00:57.824997  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 138938us
I20260709 13:00:57.890342  1749 raft_consensus.cc:2410] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 for term 12 because replica has last-logged OpId of term: 10 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 9.
I20260709 13:00:57.768348  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:00:57.856144  5827 consensus_meta.cc:287] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 1.280s	user 0.015s	sys 0.003s
I20260709 13:00:57.778450  5904 consensus_queue.cc:1059] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.088s
I20260709 13:00:57.838310  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:57.838610  5883 raft_consensus.cc:3060] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Advancing to term 10
W20260709 13:00:57.921928  5990 consensus_meta.cc:287] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.821s	user 0.016s	sys 0.000s
I20260709 13:00:57.922544  5990 raft_consensus.cc:2749] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:00:57.841542  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 12 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:57.927219  1672 raft_consensus.cc:2410] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 for term 12 because replica has last-logged OpId of term: 10 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 9.
I20260709 13:00:57.928357  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:57.842474  6019 leader_election.cc:290] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:57.843533  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:57.933038  1679 raft_consensus.cc:1240] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 7. Current term is 8. Ops: []
I20260709 13:00:57.848133  2335 raft_consensus.cc:2393] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 10: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:00:57.849078  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:57.936558  1678 raft_consensus.cc:2468] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 12.
I20260709 13:00:57.850642  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:57.850019  1601 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 897) took 1042 ms. Trace:
I20260709 13:00:57.937781  1601 rpcz_store.cc:276] 0709 13:00:56.807627 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:56.808039 (+   412us) service_pool.cc:224] Handling call
0709 13:00:56.808431 (+   392us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:56.808667 (+   236us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 13:00:57.278259 (+469592us) log.cc:844] Serialized 14 byte log entry
0709 13:00:57.849574 (+571315us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:00:57.849640 (+    66us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:00:57.849708 (+    68us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:57.849791 (+    83us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:57.849940 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":23498,"prepare.queue_time_us":704766,"prepare.run_cpu_time_us":60,"prepare.run_wall_time_us":63,"raft.queue_time_us":156432,"raft.run_cpu_time_us":156,"raft.run_wall_time_us":3576,"thread_start_us":1018562,"threads_started":3}
I20260709 13:00:57.939994  5754 raft_consensus.cc:3055] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 LEADER]: Stepping down as leader of term 11
I20260709 13:00:57.940523  5754 raft_consensus.cc:740] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:00:57.941884  6022 leader_election.cc:290] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:00:57.854792  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 276329us
I20260709 13:00:58.020354  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:58.021232  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:57.857139  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:58.023056  1680 raft_consensus.cc:2368] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 for earlier term 9. Current term is 10.
I20260709 13:00:58.023248  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:00:57.802770  5943 raft_consensus.cc:493] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:58.024080  5943 raft_consensus.cc:3060] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:00:57.861791  5833 raft_consensus.cc:476] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 12 LEADER]: Not starting pre-election -- already a leader
I20260709 13:00:57.876910  5834 consensus_queue.cc:237] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:00:57.828109  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 141071us
W20260709 13:00:57.898630  1718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:58.027658  1643 leader_election.cc:304] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:58.029968  1643 leader_election.cc:304] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:57.837273  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:57.859578  6016 raft_consensus.cc:2804] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:00:57.944191  5883 raft_consensus.cc:2749] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Leader election lost for term 9. Reason: Vote denied by peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445) with higher term. Message: Invalid argument: T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 for earlier term 9. Current term is 10.
I20260709 13:00:57.859110  5930 consensus_queue.cc:237] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:58.036144  6022 raft_consensus.cc:493] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:57.956501  5738 leader_election.cc:290] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:57.955058  5958 leader_election.cc:290] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 9 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:57.962953  5993 consensus_queue.cc:237] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:58.037611  6022 raft_consensus.cc:515] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:58.032863  6069 raft_consensus.cc:2749] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 13:00:57.733897  5955 raft_consensus.cc:515] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:57.965982  1676 raft_consensus.cc:2410] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for term 8 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20260709 13:00:57.941555  5754 consensus_queue.cc:260] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.10, Last appended by leader: 10, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:57.886270  6060 raft_consensus.cc:493] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:58.044492  6060 raft_consensus.cc:515] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:57.917889  5965 consensus_queue.cc:237] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.10, Last appended by leader: 10, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:57.864709  5926 raft_consensus.cc:515] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } }
I20260709 13:00:57.980921  5994 consensus_queue.cc:260] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.11, Last appended by leader: 11, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:57.829782  5825 raft_consensus.cc:515] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:57.853579  5915 raft_consensus.cc:515] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:57.989820  5827 raft_consensus.cc:493] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:57.990582  5787 raft_consensus.cc:493] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:00:57.991765  5938 consensus_meta.cc:287] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.862s	user 0.012s	sys 0.002s
I20260709 13:00:57.986176  5904 raft_consensus.cc:3055] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 9 LEADER]: Stepping down as leader of term 9
W20260709 13:00:57.993889  6028 consensus_meta.cc:287] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.817s	user 0.011s	sys 0.002s
W20260709 13:00:58.003814  5854 consensus_meta.cc:287] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.797s	user 0.012s	sys 0.002s
I20260709 13:00:58.007418  5806 raft_consensus.cc:515] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:57.744063  5919 consensus_queue.cc:260] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:58.022186  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:00:58.106168  1717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:58.034318  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:58.034508  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:57.917639  5873 raft_consensus.cc:515] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:58.032342  5833 raft_consensus.cc:493] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:00:57.962306  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:00:58.027590  6055 log.cc:927] Time spent T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.277s	user 0.001s	sys 0.003s
I20260709 13:00:58.046000  2331 raft_consensus.cc:2468] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 10.
I20260709 13:00:58.121647  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:58.122489  1750 raft_consensus.cc:2468] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 8.
I20260709 13:00:58.075421  6069 raft_consensus.cc:2749] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:00:57.992233  1752 raft_consensus.cc:1240] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 8. Current term is 9. Ops: []
I20260709 13:00:58.133867  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:58.135766  1752 raft_consensus.cc:1275] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 11 index: 9. (index mismatch)
I20260709 13:00:58.084386  5827 raft_consensus.cc:515] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:58.085101  5787 raft_consensus.cc:515] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:58.085322  5938 raft_consensus.cc:515] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:58.045672  6022 leader_election.cc:290] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:58.088541  5955 leader_election.cc:290] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:58.089185  5754 raft_consensus.cc:3060] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:00:58.161967  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:58.162782  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:58.164158  5777 raft_consensus.cc:493] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:58.164357  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:58.164733  5777 raft_consensus.cc:3060] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Advancing to term 9
W20260709 13:00:58.042856  6009 consensus_meta.cc:287] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.771s	user 0.010s	sys 0.006s
I20260709 13:00:58.165505  5938 leader_election.cc:290] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 10 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:00:58.117673  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 396556us
I20260709 13:00:58.165745  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:58.162822  1748 raft_consensus.cc:2376] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 11. Re-sending same reply.
I20260709 13:00:58.117815  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:00:58.120678  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:58.054069  1643 leader_election.cc:304] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:00:58.073055  6016 raft_consensus.cc:697] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:00:58.083101  5904 raft_consensus.cc:740] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:00:58.138226  1675 raft_consensus.cc:2393] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 10: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:00:58.139212  1752 raft_consensus.cc:1275] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 11 index: 9. Preceding OpId from leader: term: 13 index: 10. (index mismatch)
I20260709 13:00:58.175042  2301 leader_election.cc:304] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:00:58.145834  1747 raft_consensus.cc:2468] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 10.
I20260709 13:00:58.145838  2332 raft_consensus.cc:2468] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 10.
I20260709 13:00:58.154474  1646 leader_election.cc:304] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:00:58.122790  5994 raft_consensus.cc:3060] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:00:58.116953  2337 raft_consensus.cc:2393] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 10: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:00:58.168569  1571 leader_election.cc:304] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:00:58.169219  1753 raft_consensus.cc:2468] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 11.
I20260709 13:00:58.139490  5919 raft_consensus.cc:3060] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:00:58.186707  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:58.188079  1680 raft_consensus.cc:2410] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for term 12 because replica has last-logged OpId of term: 10 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 6.
W20260709 13:00:58.157876  5999 consensus_meta.cc:287] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.951s	user 0.006s	sys 0.010s
I20260709 13:00:58.177590  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:58.179474  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:58.078122  5869 raft_consensus.cc:515] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:58.100505  6028 raft_consensus.cc:515] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:00:58.181313  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 127405us
I20260709 13:00:58.116079  5854 raft_consensus.cc:515] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:58.194645  1749 raft_consensus.cc:1240] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 10 LEADER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 9. Current term is 10. Ops: []
I20260709 13:00:58.203833  1745 raft_consensus.cc:2393] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 9: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:00:58.195248  5955 raft_consensus.cc:2804] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:00:58.132231  5833 raft_consensus.cc:515] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:58.166208  5844 consensus_queue.cc:1048] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.027s
I20260709 13:00:58.224105  5827 leader_election.cc:290] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:58.227803  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:00:58.229265  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:58.207502  6000 consensus_queue.cc:1048] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.023s
I20260709 13:00:58.176815  6009 raft_consensus.cc:2749] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Leader election lost for term 7. Reason: Vote denied by peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693) with higher term. Message: Invalid argument: T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for earlier term 7. Current term is 8.
I20260709 13:00:58.204833  1571 leader_election.cc:304] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:00:58.199214  1643 leader_election.cc:304] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:00:58.198125  5869 leader_election.cc:290] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:58.211544  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
W20260709 13:00:58.192863  6021 consensus_meta.cc:287] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.919s	user 0.017s	sys 0.000s
I20260709 13:00:58.207602  5955 raft_consensus.cc:493] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:58.198153  6028 leader_election.cc:290] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 9 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:58.237432  1643 leader_election.cc:304] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
W20260709 13:00:58.294222  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 56906.2us
I20260709 13:00:58.234725  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:00:58.251086  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 46359.2us
I20260709 13:00:58.296051  1643 leader_election.cc:304] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:00:58.296411  1571 leader_election.cc:304] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:00:58.200299  6018 raft_consensus.cc:2804] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:00:58.200639  6046 raft_consensus.cc:2804] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:00:58.324868  2299 leader_election.cc:304] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:00:58.315312  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:58.320890  1745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55840 (request call id 2466) took 1055 ms. Trace:
I20260709 13:00:58.276103  5833 leader_election.cc:290] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:58.327122  1745 rpcz_store.cc:276] 0709 13:00:57.265144 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:57.470785 (+205641us) service_pool.cc:224] Handling call
0709 13:00:58.320523 (+849738us) spinlock_profiling.cc:243] Waited 646 ms on lock 0x7b54001d4cd4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:00:58.320785 (+   262us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1486587136}
I20260709 13:00:58.328424  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:58.250552  6069 raft_consensus.cc:2804] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:00:58.335749  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:00:58.277908  5923 consensus_meta.cc:287] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.953s	user 0.016s	sys 0.000s
I20260709 13:00:58.252431  5915 leader_election.cc:290] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:58.256258  6060 leader_election.cc:290] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:58.322463  6018 raft_consensus.cc:493] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:00:58.284126  6058 log.cc:927] Time spent T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.407s	user 0.000s	sys 0.002s
I20260709 13:00:58.377002  6013 raft_consensus.cc:2749] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:00:58.279527  5854 leader_election.cc:290] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:00:58.384341  6050 log.cc:927] Time spent T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.657s	user 0.001s	sys 0.001s
I20260709 13:00:58.280164  5873 leader_election.cc:290] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:58.211774  5926 leader_election.cc:290] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 13:00:58.235841  5671 consensus_meta.cc:287] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.665s	user 0.013s	sys 0.003s
I20260709 13:00:58.211166  6067 raft_consensus.cc:2749] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:00:58.250252  5938 raft_consensus.cc:3060] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:00:58.329257  5955 raft_consensus.cc:3060] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Advancing to term 9
W20260709 13:00:58.283515  5957 consensus_meta.cc:287] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.622s	user 0.012s	sys 0.001s
I20260709 13:00:58.206642  5999 raft_consensus.cc:515] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:58.210538  6062 consensus_queue.cc:1059] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.193s
I20260709 13:00:58.429817  6062 consensus_queue.cc:1243] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer af520b67945d443b901bbeca37cc5701 log is divergent from this leader: its last log entry 7.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260709 13:00:58.349926  5989 raft_consensus.cc:2804] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Leader election won for term 6
I20260709 13:00:58.381249  5991 raft_consensus.cc:493] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
W20260709 13:00:58.414505  6044 consensus_meta.cc:287] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.526s	user 0.013s	sys 0.002s
I20260709 13:00:58.402017  5923 raft_consensus.cc:515] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:58.401885  6069 raft_consensus.cc:697] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:00:58.420956  5962 leader_election.cc:290] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:58.400722  6018 raft_consensus.cc:3060] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:00:58.274596  5943 raft_consensus.cc:515] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:58.393896  6046 raft_consensus.cc:697] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
W20260709 13:00:58.517242  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:58.434904  6062 raft_consensus.cc:3055] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:00:58.560626  6062 raft_consensus.cc:740] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
W20260709 13:00:58.561159  6029 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.781s	user 0.001s	sys 0.010s
W20260709 13:00:58.334026  1718 leader_election.cc:336] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 election: RPC error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Timed out: RequestConsensusVote RPC to 127.0.107.194:46261 timed out after 2.086s (SENT)
W20260709 13:00:58.536839  1717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:00:58.539278  1719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:58.543802  1751 raft_consensus.cc:2393] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 12: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:00:58.545425  2301 leader_election.cc:304] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:00:58.418510  5777 raft_consensus.cc:515] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:58.251369  5806 leader_election.cc:290] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:58.398586  2337 raft_consensus.cc:1240] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 11. Current term is 12. Ops: [11.10-11.10]
I20260709 13:00:58.419487  5671 raft_consensus.cc:515] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:58.379948  5825 leader_election.cc:290] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 9 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:58.597859  5777 leader_election.cc:290] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 9 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:58.414868  5787 leader_election.cc:290] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:00:58.607164  5899 consensus_meta.cc:287] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.680s	user 0.012s	sys 0.006s
I20260709 13:00:58.607303  5962 raft_consensus.cc:493] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:58.607949  5899 raft_consensus.cc:515] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:58.608353  5962 raft_consensus.cc:515] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } }
I20260709 13:00:58.609642  5943 leader_election.cc:290] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 12 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:58.268512  5904 consensus_queue.cc:260] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:58.441186  5989 raft_consensus.cc:697] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:00:58.597910  5671 leader_election.cc:290] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 9 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:58.440042  5991 raft_consensus.cc:3060] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Advancing to term 10
W20260709 13:00:58.432273  6051 log.cc:927] Time spent T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.541s	user 0.001s	sys 0.000s
W20260709 13:00:58.518666  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 222112us
I20260709 13:00:58.467669  6069 consensus_queue.cc:237] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } }
I20260709 13:00:58.274880  6016 consensus_queue.cc:237] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.10, Last appended by leader: 10, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:58.442473  6044 raft_consensus.cc:515] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:58.627242  6044 leader_election.cc:290] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 11 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:58.627516  1676 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52766 (request call id 2499) took 1486 ms. Trace:
I20260709 13:00:58.628099  1676 rpcz_store.cc:276] 0709 13:00:57.140925 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:57.142103 (+  1178us) service_pool.cc:224] Handling call
0709 13:00:58.626955 (+1484852us) spinlock_profiling.cc:243] Waited 799 ms on lock 0x7b54000d9154. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:00:58.627249 (+   294us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1838497664}
I20260709 13:00:58.470331  5999 leader_election.cc:290] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 12 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:00:58.564646  6029 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.789s	user 0.002s	sys 0.010s
I20260709 13:00:58.635203  2335 raft_consensus.cc:1275] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20260709 13:00:58.566196  5869 raft_consensus.cc:2804] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:00:58.566401  1604 raft_consensus.cc:1240] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 10. Current term is 11. Ops: []
W20260709 13:00:58.573194  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 238997us
I20260709 13:00:58.635959  5869 raft_consensus.cc:493] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:58.636631  5869 raft_consensus.cc:3060] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:00:58.387953  2299 leader_election.cc:304] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
W20260709 13:00:58.638361  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 93452.7us
I20260709 13:00:58.640245  2301 leader_election.cc:304] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
W20260709 13:00:58.641074  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 301368us
I20260709 13:00:58.566797  6062 consensus_queue.cc:260] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:58.643054  6002 raft_consensus.cc:2749] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 13:00:58.643584  5919 consensus_queue.cc:1048] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 13:00:58.644392  6062 raft_consensus.cc:3060] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:00:58.645726  5923 leader_election.cc:290] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:00:58.594465  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:58.383040  5827 raft_consensus.cc:2749] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 13:00:58.644444  6002 raft_consensus.cc:2804] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 13:00:58.648790  6046 consensus_queue.cc:237] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.10, Last appended by leader: 10, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:58.650887  1718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:58.652243  5999 raft_consensus.cc:493] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:58.648818  6002 raft_consensus.cc:697] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 10 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:00:58.653043  5999 raft_consensus.cc:515] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:58.653283  5827 consensus_queue.cc:1059] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 13:00:58.654649  5827 consensus_queue.cc:1243] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer af520b67945d443b901bbeca37cc5701 log is divergent from this leader: its last log entry 10.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260709 13:00:58.655875  5738 raft_consensus.cc:493] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:00:58.656500  5827 raft_consensus.cc:3055] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:00:58.657024  5827 raft_consensus.cc:740] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:00:58.656869  5738 raft_consensus.cc:515] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:58.661068  5738 leader_election.cc:290] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:58.662940  1571 leader_election.cc:304] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
W20260709 13:00:58.664757  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 45734.6us
I20260709 13:00:58.666724  6067 raft_consensus.cc:2749] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
W20260709 13:00:58.670099  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:00:58.601007  1677 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52766 (request call id 2496) took 1472 ms (client timeout 1556 ms). Trace:
I20260709 13:00:58.601454  2333 raft_consensus.cc:2393] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 9: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
W20260709 13:00:58.603008  1719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:58.633566  2331 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39726 (request call id 1038) took 1061 ms. Trace:
I20260709 13:00:58.672282  2331 rpcz_store.cc:276] 0709 13:00:57.572410 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:57.572559 (+   149us) service_pool.cc:224] Handling call
0709 13:00:58.633011 (+1060452us) spinlock_profiling.cc:243] Waited 424 ms on lock 0x7b5400443754. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:00:58.633286 (+   275us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":974071680}
W20260709 13:00:58.670958  1677 rpcz_store.cc:269] 0709 13:00:57.127981 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:57.128911 (+   930us) service_pool.cc:224] Handling call
0709 13:00:58.600437 (+1471526us) spinlock_profiling.cc:243] Waited 371 ms on lock 0x7b5400346f54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:00:58.600722 (+   285us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":854211584}
I20260709 13:00:58.690797  5827 consensus_queue.cc:260] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 9.5, Last appended by leader: 5, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:58.613651  5989 consensus_queue.cc:237] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:58.697260  1571 leader_election.cc:304] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:00:58.654690  6002 consensus_queue.cc:237] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.7, Last appended by leader: 7, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:58.701619  2332 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f"
dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
mode: GRACEFUL
new_leader_uuid: "af520b67945d443b901bbeca37cc5701"
 from {username='slave'} at 127.0.0.1:35790
I20260709 13:00:58.702521  2328 raft_consensus.cc:1240] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 10 LEADER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 9. Current term is 10. Ops: [9.10-9.10]
I20260709 13:00:58.703581  1746 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55840 (request call id 2471) took 1436 ms. Trace:
I20260709 13:00:58.703455  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:58.704161  1746 rpcz_store.cc:276] 0709 13:00:57.267064 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:57.528711 (+261647us) service_pool.cc:224] Handling call
0709 13:00:58.703101 (+1174390us) spinlock_profiling.cc:243] Waited 119 ms on lock 0x7b54001d51d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:00:58.703301 (+   200us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":274222336}
I20260709 13:00:58.704989  5899 leader_election.cc:290] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:58.709386  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 9 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:58.709911  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:58.710289  1752 raft_consensus.cc:3060] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:00:58.710882  1749 raft_consensus.cc:1240] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 8. Current term is 9. Ops: []
I20260709 13:00:58.697520  5962 leader_election.cc:290] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:58.682651  2299 leader_election.cc:304] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:00:58.717146  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:58.718039  1747 raft_consensus.cc:3060] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:00:58.730103  1752 raft_consensus.cc:2410] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for term 9 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20260709 13:00:58.730784  6067 raft_consensus.cc:2804] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:00:58.731351  6067 raft_consensus.cc:493] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:58.731746  6067 raft_consensus.cc:3060] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:00:58.710955  1745 raft_consensus.cc:3060] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:00:58.738965  5827 raft_consensus.cc:3060] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:00:58.740939  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:58.742887  1604 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 906) took 1588 ms. Trace:
I20260709 13:00:58.744141  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 9 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:00:58.744418  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:58.745637  1673 raft_consensus.cc:2393] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 12: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
W20260709 13:00:58.745850  1751 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43156 (request call id 804) took 1566 ms (client timeout 1925 ms). Trace:
W20260709 13:00:58.746368  1751 rpcz_store.cc:269] 0709 13:00:57.179877 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:57.180008 (+   131us) service_pool.cc:224] Handling call
0709 13:00:58.745586 (+1565578us) spinlock_profiling.cc:243] Waited 1.29 s on lock 0x7b540019e6d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:00:58.745770 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2960402816}
I20260709 13:00:58.747121  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:58.743469  1604 rpcz_store.cc:276] 0709 13:00:57.170024 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:57.187081 (+ 17057us) service_pool.cc:224] Handling call
0709 13:00:57.225165 (+ 38084us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:58.636139 (+1410974us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:58.742402 (+106263us) spinlock_profiling.cc:243] Waited 1.33 s on lock 0x7b54001e0354. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:00:58.742708 (+   306us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3049071872}
I20260709 13:00:58.756186  5999 leader_election.cc:290] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:58.756682  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:58.769506  6080 consensus_queue.cc:1059] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 13:00:58.771070  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:58.771976  1676 raft_consensus.cc:3060] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:00:58.748143  1746 raft_consensus.cc:2468] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 11.
I20260709 13:00:58.779132  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:58.780105  1677 raft_consensus.cc:3060] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:00:58.782042  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:58.782649  6029 raft_consensus.cc:493] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:58.782989  1672 raft_consensus.cc:3060] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:00:58.783489  6029 raft_consensus.cc:515] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:58.789325  5904 raft_consensus.cc:3060] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:00:58.789659  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:58.790529  1675 raft_consensus.cc:3060] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:00:58.792420  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:58.794982  6080 raft_consensus.cc:3055] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:00:58.795454  5228 raft_consensus.cc:2804] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:00:58.795585  6080 raft_consensus.cc:740] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:00:58.796048  5228 raft_consensus.cc:493] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:58.796496  5228 raft_consensus.cc:3060] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:00:58.797009  6080 consensus_queue.cc:260] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:58.819016  2333 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39738 (request call id 846) took 1052 ms. Trace:
I20260709 13:00:58.825253  2333 rpcz_store.cc:276] 0709 13:00:57.766890 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:57.767945 (+  1055us) service_pool.cc:224] Handling call
0709 13:00:58.809959 (+1042014us) spinlock_profiling.cc:243] Waited 709 ms on lock 0x7b5400551754. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:00:58.818922 (+  8963us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1631147264}
I20260709 13:00:58.827628  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 9 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:58.829031  1748 raft_consensus.cc:2410] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 for term 9 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
W20260709 13:00:58.829594  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 164510us
I20260709 13:00:58.830988  2299 leader_election.cc:400] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 9 election: Vote denied by peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261) with higher term. Message: Invalid argument: T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 for earlier term 9. Current term is 10.
I20260709 13:00:58.738111  5938 raft_consensus.cc:2749] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:00:58.834650  1754 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43156 (request call id 811) took 1072 ms. Trace:
I20260709 13:00:58.835146  1754 rpcz_store.cc:276] 0709 13:00:57.734881 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:57.737179 (+  2298us) service_pool.cc:224] Handling call
0709 13:00:58.741139 (+1003960us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:00:58.741594 (+   455us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:00:58.741628 (+    34us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:00:58.741691 (+    63us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:00:58.741764 (+    73us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:00:58.741853 (+    89us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:00:58.807265 (+ 65412us) spinlock_profiling.cc:243] Waited 945 ms on lock 0x7b54001d51d0. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661618ad 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:00:58.807416 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2173303680}
I20260709 13:00:58.836958  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:00:58.839529  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 543326us
I20260709 13:00:58.842703  6018 raft_consensus.cc:515] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:58.842162  1643 leader_election.cc:304] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:00:58.786306  6029 leader_election.cc:290] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:58.856870  5962 raft_consensus.cc:493] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:58.857616  5962 raft_consensus.cc:515] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:58.859082  5738 raft_consensus.cc:493] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:58.860209  5962 leader_election.cc:290] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:58.859891  5738 raft_consensus.cc:515] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:58.861616  5962 consensus_queue.cc:1048] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 13:00:58.863047  5738 leader_election.cc:290] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:58.869895  1747 raft_consensus.cc:2468] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 11.
I20260709 13:00:58.872083  5899 raft_consensus.cc:493] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:58.872164  1569 leader_election.cc:304] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
W20260709 13:00:58.877696  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 193477us
I20260709 13:00:58.879989  1646 leader_election.cc:304] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:00:58.885192  6026 raft_consensus.cc:493] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:58.886009  6026 raft_consensus.cc:515] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:58.888800  6026 leader_election.cc:290] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:58.873021  5923 raft_consensus.cc:493] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:58.891117  6018 leader_election.cc:290] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 12 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:58.893325  6069 raft_consensus.cc:2749] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260709 13:00:58.894327  1644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:00:58.894886  1644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:58.896032  6029 raft_consensus.cc:2804] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader election won for term 10
I20260709 13:00:58.868254  1753 raft_consensus.cc:2468] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 12.
I20260709 13:00:58.855583  6080 raft_consensus.cc:3060] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Advancing to term 9
W20260709 13:00:58.903281  2301 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:58.910822  1745 raft_consensus.cc:2468] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 10.
I20260709 13:00:58.889891  5923 raft_consensus.cc:515] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:58.937253  1677 raft_consensus.cc:2468] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 9.
I20260709 13:00:58.865273  5938 raft_consensus.cc:493] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:58.948335  1672 raft_consensus.cc:2468] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 11.
I20260709 13:00:58.869968  5955 raft_consensus.cc:515] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:58.877538  5899 raft_consensus.cc:515] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:58.866211  5869 raft_consensus.cc:515] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:58.959451  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:58.960512  1754 raft_consensus.cc:1275] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 12 index: 10. (index mismatch)
I20260709 13:00:58.944947  6029 raft_consensus.cc:697] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:00:58.972914  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:58.965224  1644 leader_election.cc:304] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:00:58.959828  6018 raft_consensus.cc:2804] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Leader pre-election won for term 12
W20260709 13:00:58.976715  1717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:58.977861  2335 raft_consensus.cc:1240] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 8. Current term is 9. Ops: []
I20260709 13:00:58.982257  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:00:58.982080  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:00:58.985443  2333 raft_consensus.cc:3060] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:00:58.986687  1753 raft_consensus.cc:2393] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 10: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
W20260709 13:00:58.985065  1718 leader_election.cc:336] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 9 election: RPC error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Timed out: RequestConsensusVote RPC to 127.0.107.194:46261 timed out after 2.395s (SENT)
W20260709 13:00:58.981602  5994 consensus_meta.cc:287] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.771s	user 0.018s	sys 0.009s
I20260709 13:00:58.979089  6018 raft_consensus.cc:493] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:58.987767  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
W20260709 13:00:58.994987  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 153031us
I20260709 13:00:58.998454  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:58.994905  1718 leader_election.cc:304] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:59.002979  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:59.003703  1749 raft_consensus.cc:2468] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 7.
I20260709 13:00:59.004587  1746 raft_consensus.cc:2393] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 10: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:00:59.005285  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:59.006210  1601 raft_consensus.cc:2393] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 10: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:00:59.008997  6046 raft_consensus.cc:2749] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:00:59.012044  6029 consensus_queue.cc:237] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:59.013141  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 66705.9us
I20260709 13:00:59.016654  1605 raft_consensus.cc:1240] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 10. Current term is 11. Ops: []
I20260709 13:00:59.018594  5833 raft_consensus.cc:493] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:59.019268  5833 raft_consensus.cc:515] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:59.021737  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:59.033444  1675 raft_consensus.cc:2468] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 11.
I20260709 13:00:59.039644  1676 raft_consensus.cc:2468] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 12.
I20260709 13:00:58.974224  5938 raft_consensus.cc:515] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:59.042207  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:58.995188  6018 raft_consensus.cc:3060] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:00:59.044795  1745 raft_consensus.cc:2393] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 10: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:00:59.046633  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:59.048293  2333 raft_consensus.cc:2468] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 13.
I20260709 13:00:59.050952  1604 raft_consensus.cc:1275] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 10 index: 8. (index mismatch)
W20260709 13:00:59.053367  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 73982.6us
I20260709 13:00:59.044446  5738 raft_consensus.cc:2749] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 13:00:59.059705  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:59.063395  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:59.058060  5970 raft_consensus.cc:2804] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:00:59.066677  2332 raft_consensus.cc:606] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Received request to transfer leadership to TS af520b67945d443b901bbeca37cc5701
I20260709 13:00:59.067083  2332 raft_consensus.cc:614] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 13:00:59.067534  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:59.070837  5926 raft_consensus.cc:493] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:59.071715  5926 raft_consensus.cc:515] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:59.074719  5938 leader_election.cc:290] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:59.081818  1643 leader_election.cc:304] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:00:59.093037  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:59.094514  1604 raft_consensus.cc:2468] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 7.
I20260709 13:00:59.095140  1569 leader_election.cc:304] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
W20260709 13:00:59.097190  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 115289us
I20260709 13:00:59.098590  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:59.099371  1598 raft_consensus.cc:2468] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 12.
I20260709 13:00:59.100761  5869 leader_election.cc:290] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 10 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:59.101032  5899 leader_election.cc:290] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:59.105784  5970 raft_consensus.cc:493] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:00:59.097342  5923 leader_election.cc:290] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:59.112253  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:00:59.112967  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 49804.7us
I20260709 13:00:59.115082  1600 raft_consensus.cc:1275] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 12 index: 10. (index mismatch)
I20260709 13:00:59.119725  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:59.117488  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:59.115495  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:59.111852  1605 raft_consensus.cc:1275] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
W20260709 13:00:59.133426  1719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:59.106413  5970 raft_consensus.cc:3060] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Advancing to term 12
W20260709 13:00:59.136307  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 68637.3us
I20260709 13:00:59.114250  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:59.142491  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:59.146412  6013 consensus_queue.cc:1048] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.011s
I20260709 13:00:59.148895  5228 raft_consensus.cc:515] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:59.149452  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
W20260709 13:00:59.151252  6085 log.cc:927] Time spent T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.099s	user 0.002s	sys 0.000s
I20260709 13:00:59.152788  5926 leader_election.cc:290] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:59.112471  5991 raft_consensus.cc:515] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:59.155069  1718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:59.158687  1749 raft_consensus.cc:2468] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 7.
I20260709 13:00:59.139297  5938 raft_consensus.cc:2749] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:00:59.167809  1679 raft_consensus.cc:1240] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 8. Current term is 9. Ops: []
I20260709 13:00:59.169440  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 9 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:59.170423  1673 raft_consensus.cc:2393] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 9: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:00:59.171828  5955 leader_election.cc:290] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 9 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:59.172623  5955 raft_consensus.cc:493] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:00:59.173126  5955 raft_consensus.cc:515] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:59.173460  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:59.174230  1749 raft_consensus.cc:3060] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:00:59.175526  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:59.176494  1605 raft_consensus.cc:2393] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 9: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
W20260709 13:00:59.177445  6094 log.cc:927] Time spent T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.059s	user 0.000s	sys 0.001s
I20260709 13:00:59.177663  5833 leader_election.cc:290] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:59.181043  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:59.181917  1602 raft_consensus.cc:2393] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 9: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:00:59.177683  2332 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "a40e423ed583480280352b261913276c"
dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
mode: GRACEFUL
new_leader_uuid: "af520b67945d443b901bbeca37cc5701"
 from {username='slave'} at 127.0.0.1:35790
I20260709 13:00:59.183135  2332 raft_consensus.cc:606] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Received request to transfer leadership to TS af520b67945d443b901bbeca37cc5701
I20260709 13:00:59.183259  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:59.183558  2332 raft_consensus.cc:614] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 13:00:59.185361  1676 raft_consensus.cc:1275] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 10 index: 8. Preceding OpId from leader: term: 13 index: 10. (index mismatch)
I20260709 13:00:59.187337  1601 raft_consensus.cc:1240] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 9. Current term is 10. Ops: []
W20260709 13:00:59.187558  6088 log.cc:927] Time spent T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.146s	user 0.001s	sys 0.000s
I20260709 13:00:59.187855  1646 leader_election.cc:304] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:00:59.187558  2332 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49"
dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
mode: GRACEFUL
new_leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
 from {username='slave'} at 127.0.0.1:35790
I20260709 13:00:59.189010  1646 leader_election.cc:304] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:00:59.189379  1599 raft_consensus.cc:1240] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 6. Current term is 7. Ops: [6.5-6.5]
I20260709 13:00:59.189564  2332 raft_consensus.cc:606] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Received request to transfer leadership to TS 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:00:59.190042  2332 raft_consensus.cc:614] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 13:00:59.190820  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:00:59.192297  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:59.192770  1749 raft_consensus.cc:2468] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 9.
I20260709 13:00:59.193886  2301 leader_election.cc:304] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:00:59.190094  1646 leader_election.cc:304] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:00:59.181795  6067 raft_consensus.cc:515] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:59.200713  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:59.113186  6046 raft_consensus.cc:2749] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:00:59.210908  5919 raft_consensus.cc:2804] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 13:00:59.211199  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:59.211673  5919 raft_consensus.cc:697] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 9 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:00:59.212440  2336 raft_consensus.cc:2393] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 9: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:00:59.212968  5919 consensus_queue.cc:237] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:59.213824  6062 raft_consensus.cc:2804] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:00:59.214423  6062 raft_consensus.cc:493] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:59.214855  6062 raft_consensus.cc:3060] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:00:59.171376  1501 catalog_manager.cc:5697] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d reported cstate change: term changed from 7 to 9. New cstate: current_term: 9 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } }
W20260709 13:00:59.207018  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 111928us
I20260709 13:00:59.205489  1598 raft_consensus.cc:2468] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 14.
W20260709 13:00:59.219174  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:00:59.219732  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:59.220878  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:00:59.226418  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:59.220528  1644 leader_election.cc:304] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:00:59.229192  1599 raft_consensus.cc:1240] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 11. Current term is 12. Ops: [11.7-11.7]
I20260709 13:00:59.232587  1602 raft_consensus.cc:2393] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 10: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
W20260709 13:00:59.237439  1719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:59.236025  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:59.241312  1751 raft_consensus.cc:1275] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 10 index: 8. (index mismatch)
I20260709 13:00:59.242717  6069 raft_consensus.cc:2804] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Leader election won for term 10
I20260709 13:00:59.243825  5991 leader_election.cc:290] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 10 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:59.245568  6069 raft_consensus.cc:697] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 10 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:00:59.246788  6069 consensus_queue.cc:237] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 8.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:59.247165  1569 leader_election.cc:304] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:00:59.248068  6067 leader_election.cc:290] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 10 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:59.249814  5926 raft_consensus.cc:2804] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:00:59.251861  5926 raft_consensus.cc:697] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:00:59.251574  1501 catalog_manager.cc:5697] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d reported cstate change: term changed from 5 to 6, leader changed from <none> to af520b67945d443b901bbeca37cc5701 (127.0.107.195). New cstate: current_term: 6 leader_uuid: "af520b67945d443b901bbeca37cc5701" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 13:00:59.254621  6016 raft_consensus.cc:2804] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Leader election won for term 11
I20260709 13:00:59.250865  1604 raft_consensus.cc:1240] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 9. Current term is 10. Ops: []
I20260709 13:00:59.255344  6016 raft_consensus.cc:697] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 11 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:00:59.256498  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:59.256654  6016 consensus_queue.cc:237] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:59.259610  1646 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:59.238426  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:59.262409  5989 raft_consensus.cc:493] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:00:59.262840  6013 consensus_queue.cc:1048] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 13:00:59.263334  5989 raft_consensus.cc:515] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:59.250065  1569 leader_election.cc:304] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:00:59.268108  1569 leader_election.cc:304] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:00:59.271479  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:00:59.278013  1646 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:59.275295  2336 raft_consensus.cc:2468] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 11.
I20260709 13:00:59.281610  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:59.291196  1643 leader_election.cc:304] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 13:00:59.294407  2332 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23"
dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
mode: GRACEFUL
new_leader_uuid: "af520b67945d443b901bbeca37cc5701"
 from {username='slave'} at 127.0.0.1:35790
W20260709 13:00:59.274535  6094 log.cc:927] Time spent T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.051s	user 0.000s	sys 0.001s
I20260709 13:00:59.298697  1751 raft_consensus.cc:2368] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 for earlier term 10. Current term is 11.
I20260709 13:00:59.266741  1501 catalog_manager.cc:5697] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d reported cstate change: term changed from 4 to 9, leader changed from af520b67945d443b901bbeca37cc5701 (127.0.107.195) to 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194). New cstate: current_term: 9 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } }
I20260709 13:00:59.301928  1569 leader_election.cc:304] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:00:59.303020  2335 raft_consensus.cc:1240] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 12. Current term is 13. Ops: [12.10-12.10]
I20260709 13:00:59.303139  1753 raft_consensus.cc:1275] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20260709 13:00:59.253607  5926 consensus_queue.cc:237] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:59.275825  5938 raft_consensus.cc:2804] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Leader election won for term 11
I20260709 13:00:59.295135  2332 raft_consensus.cc:606] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 12 LEADER]: Received request to transfer leadership to TS af520b67945d443b901bbeca37cc5701
I20260709 13:00:59.306368  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:59.313246  1600 raft_consensus.cc:2393] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 10: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:00:59.294162  6080 raft_consensus.cc:2804] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:00:59.326079  6080 raft_consensus.cc:493] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:59.326540  6080 raft_consensus.cc:3060] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:00:59.309302  5827 raft_consensus.cc:2764] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader pre-election decision vote started in defunct term 8: won
I20260709 13:00:59.340571  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:59.319399  5228 leader_election.cc:290] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 13 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:59.343041  1603 raft_consensus.cc:2393] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 12: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:00:59.334582  2333 raft_consensus.cc:1275] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
I20260709 13:00:59.337131  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:59.337994  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:59.345781  1599 raft_consensus.cc:2393] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 9: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:00:59.345655  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:59.339288  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:59.346601  1601 raft_consensus.cc:2468] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 12.
W20260709 13:00:59.347543  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 142515us
I20260709 13:00:59.349717  1644 leader_election.cc:304] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:00:59.339423  1569 leader_election.cc:304] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
W20260709 13:00:59.339883  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 74626.8us
I20260709 13:00:59.340526  5989 leader_election.cc:290] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:00:59.341600  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 269675us
I20260709 13:00:59.323817  5962 raft_consensus.cc:493] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:59.322719  1501 catalog_manager.cc:5697] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d reported cstate change: term changed from 6 to 7, leader changed from af520b67945d443b901bbeca37cc5701 (127.0.107.195) to 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196). New cstate: current_term: 7 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 13:00:59.316258  6067 raft_consensus.cc:2804] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:00:59.326706  6082 raft_consensus.cc:2804] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Leader election won for term 11
I20260709 13:00:59.374940  5894 raft_consensus.cc:2804] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:00:59.375651  5894 raft_consensus.cc:697] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:00:59.379503  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:59.330124  5938 raft_consensus.cc:697] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:00:59.377142  5228 raft_consensus.cc:493] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:59.381894  6029 raft_consensus.cc:2749] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 13:00:59.413331  5663 consensus_queue.cc:1059] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 10.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
W20260709 13:00:59.411688  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 57744.9us
I20260709 13:00:59.386109  6067 raft_consensus.cc:697] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:00:59.395560  6082 raft_consensus.cc:697] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:00:59.382256  5833 raft_consensus.cc:493] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:00:59.422721  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:59.423810  1604 raft_consensus.cc:1275] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 10 index: 7. Preceding OpId from leader: term: 11 index: 8. (index mismatch)
W20260709 13:00:59.422626  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 60568.7us
I20260709 13:00:59.427784  1644 leader_election.cc:304] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:00:59.403055  5962 raft_consensus.cc:515] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:59.429113  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:59.423363  1501 catalog_manager.cc:5697] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } }
I20260709 13:00:59.430337  1672 raft_consensus.cc:2410] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for term 9 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 4.
I20260709 13:00:59.434634  5844 raft_consensus.cc:3055] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:00:59.435129  5844 raft_consensus.cc:740] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:00:59.436239  5844 consensus_queue.cc:260] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:59.437463  5844 raft_consensus.cc:3060] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:00:59.438508  1597 raft_consensus.cc:1275] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260709 13:00:59.429162  1674 raft_consensus.cc:1240] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 6. Current term is 7. Ops: []
I20260709 13:00:59.421126  5955 leader_election.cc:290] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:00:59.439962  1499 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35444 (request call id 260) took 32528 ms (client timeout 15000 ms). Trace:
I20260709 13:00:59.443259  6082 consensus_queue.cc:237] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:59.441373  5833 raft_consensus.cc:515] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:59.419831  5228 raft_consensus.cc:515] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:59.384779  5894 consensus_queue.cc:237] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:59.451186  1674 raft_consensus.cc:1240] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 11. Current term is 12. Ops: []
I20260709 13:00:59.452725  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:59.453933  1672 raft_consensus.cc:2393] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 13: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:00:59.454385   431 auto_leader_rebalancer.cc:452] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 13:00:59.455919  5955 consensus_queue.cc:1059] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 10.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 13:00:59.457387  1605 raft_consensus.cc:1275] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 10 index: 8. (index mismatch)
I20260709 13:00:59.457628  5671 raft_consensus.cc:3055] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:00:59.458166  5671 raft_consensus.cc:740] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:00:59.460664  5671 consensus_queue.cc:260] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 10.11, Last appended by leader: 11, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:59.474125  6106 log.cc:927] Time spent T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.059s	user 0.000s	sys 0.001s
I20260709 13:00:59.462149  5671 raft_consensus.cc:3060] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:00:59.492218  1602 raft_consensus.cc:2468] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 7.
I20260709 13:00:59.472885  6069 raft_consensus.cc:2749] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
W20260709 13:00:59.440614  1499 rpcz_store.cc:269] 0709 13:00:26.868326 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:26.941221 (+ 72895us) service_pool.cc:224] Handling call
0709 13:00:59.385935 (+32444714us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 13:00:52.265317 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 13:00:52.375795 (+110478us) write_op.cc:432] Acquiring schema lock in shared mode
0709 13:00:52.375871 (+    76us) write_op.cc:435] Acquired schema lock
0709 13:00:52.375895 (+    24us) tablet.cc:662] Decoding operations
0709 13:00:52.480277 (+104382us) write_op.cc:620] Acquiring the partition lock for write op
0709 13:00:52.480383 (+   106us) write_op.cc:641] Partition lock acquired for write op
0709 13:00:52.480415 (+    32us) tablet.cc:685] Acquiring locks for 24 operations
0709 13:00:52.565790 (+ 85375us) tablet.cc:701] Row locks acquired
0709 13:00:52.566317 (+   527us) write_op.cc:260] PREPARE: finished
0709 13:00:52.566587 (+   270us) write_op.cc:270] Start()
0709 13:00:52.566773 (+   186us) write_op.cc:276] Timestamp: P: 1783602052566497 usec, L: 0
0709 13:00:52.566815 (+    42us) op_driver.cc:348] REPLICATION: starting
0709 13:00:52.622153 (+ 55338us) log.cc:844] Serialized 9393 byte log entry
0709 13:00:53.515150 (+892997us) op_driver.cc:464] REPLICATION: finished
0709 13:00:54.004646 (+489496us) write_op.cc:301] APPLY: starting
0709 13:00:54.004821 (+   175us) tablet.cc:1367] starting BulkCheckPresence
0709 13:00:55.401671 (+1396850us) tablet.cc:1370] finished BulkCheckPresence
0709 13:00:55.401887 (+   216us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 13:00:55.933376 (+531489us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 13:00:55.934132 (+   756us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=48,key_file_lookups=48,delta_file_lookups=48,mrs_lookups=0
0709 13:00:55.934241 (+   109us) write_op.cc:312] APPLY: finished
0709 13:00:55.935300 (+  1059us) log.cc:844] Serialized 212 byte log entry
0709 13:00:56.363831 (+428531us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:00:56.440563 (+ 76732us) write_op.cc:454] Released schema lock
0709 13:00:56.466058 (+ 25495us) write_op.cc:341] FINISH: Updating metrics
0709 13:00:56.560249 (+ 94191us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:00:56.564623 (+  4374us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":1143802,"reported_tablets":46,"child_traces":[["op",{"apply.queue_time_us":433044,"apply.run_cpu_time_us":273839,"apply.run_wall_time_us":2530095,"cfile_cache_hit":94,"cfile_cache_hit_bytes":98418,"cfile_cache_miss":4,"cfile_cache_miss_bytes":6221,"cfile_init":2,"lbm_read_time_us":18772,"lbm_reads_10-100_ms":1,"lbm_reads_lt_1ms":11,"num_ops":24,"prepare.queue_time_us":438650,"prepare.run_cpu_time_us":21406,"prepare.run_wall_time_us":545036,"raft.queue_time_us":367006,"raft.run_cpu_time_us":19056,"raft.run_wall_time_us":243446,"replication_time_us":908391,"thread_start_us":931008,"threads_started":5,"wal-append.queue_time_us":757109,"wal-append.run_cpu_time_us":23517,"wal-append.run_wall_time_us":2756354}]]}
W20260709 13:00:59.504159  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 164096us
I20260709 13:00:59.432204  1501 catalog_manager.cc:5697] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:00:59.509557  6067 consensus_queue.cc:237] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:59.521641  2301 leader_election.cc:400] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 10 election: Vote denied by peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693) with higher term. Message: Invalid argument: T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 for earlier term 10. Current term is 11.
I20260709 13:00:59.522325  2301 leader_election.cc:403] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 10 election: Cancelling election due to peer responding with higher term
I20260709 13:00:59.508620  1501 catalog_manager.cc:5697] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d reported cstate change: term changed from 3 to 4, leader changed from 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194) to 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196). New cstate: current_term: 4 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 13:00:59.531395  1671 raft_consensus.cc:1240] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 11. Current term is 12. Ops: []
I20260709 13:00:59.535017  1718 leader_election.cc:304] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:00:59.542218  1569 leader_election.cc:304] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
W20260709 13:00:59.542330  2301 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:00:59.548046  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 120384us
I20260709 13:00:59.535087  5989 consensus_queue.cc:1048] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.015s
I20260709 13:00:59.550011  5738 raft_consensus.cc:2804] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:00:59.550774  5738 raft_consensus.cc:493] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:59.551446  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:59.552217  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:59.553099  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:00:59.550802  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:59.555078  5999 consensus_queue.cc:1048] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 13:00:59.550086  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:59.550705  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:00:59.551364  5738 raft_consensus.cc:3060] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:00:59.554265  6029 raft_consensus.cc:493] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:59.561167  6029 raft_consensus.cc:515] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:59.562777  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:59.563540  1679 raft_consensus.cc:3060] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:00:59.556442  5777 raft_consensus.cc:2749] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:00:59.567138  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:59.575809  6029 leader_election.cc:290] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:59.550735  5938 consensus_queue.cc:237] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:59.583238  1671 raft_consensus.cc:1240] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 10. Current term is 11. Ops: []
I20260709 13:00:59.583381  1604 raft_consensus.cc:1275] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
I20260709 13:00:59.547501  5994 raft_consensus.cc:3060] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:00:59.586220  5962 leader_election.cc:290] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:59.588284  2298 leader_election.cc:304] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:00:59.590657  6111 consensus_queue.cc:1059] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 10.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.031s
I20260709 13:00:59.591862  6111 raft_consensus.cc:3055] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:00:59.592320  6111 raft_consensus.cc:740] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:00:59.592865  5955 consensus_queue.cc:1048] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 13:00:59.593425  6111 consensus_queue.cc:260] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.10, Last appended by leader: 10, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:59.599470  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
W20260709 13:00:59.601348  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 44536.8us
I20260709 13:00:59.609452  1679 raft_consensus.cc:2468] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 13.
I20260709 13:00:59.576421  6080 raft_consensus.cc:515] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:59.616936  1680 raft_consensus.cc:2468] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 8.
W20260709 13:00:59.622288  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:00:59.628260  1717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:59.641490  1752 raft_consensus.cc:2468] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 11.
I20260709 13:00:59.606539  1501 catalog_manager.cc:5697] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } }
I20260709 13:00:59.664294  1602 raft_consensus.cc:1275] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 10 index: 5. (index mismatch)
I20260709 13:00:59.664760  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:59.665534  1604 raft_consensus.cc:2468] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 6.
W20260709 13:00:59.671718  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 129595us
I20260709 13:00:59.672214  1679 raft_consensus.cc:1240] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 LEADER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 5. Current term is 6. Ops: []
W20260709 13:00:59.675263  1718 leader_election.cc:336] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Timed out: RequestConsensusVote RPC to 127.0.107.194:46261 timed out after 1.641s (SENT)
I20260709 13:00:59.652014  5991 raft_consensus.cc:2804] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:00:59.656651  6111 raft_consensus.cc:3060] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Advancing to term 13
W20260709 13:00:59.678246  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 121115us
I20260709 13:00:59.662684  1501 catalog_manager.cc:5697] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } }
I20260709 13:00:59.685447  1501 catalog_manager.cc:5697] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d reported cstate change: term changed from 5 to 6, leader changed from af520b67945d443b901bbeca37cc5701 (127.0.107.195) to 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194). New cstate: current_term: 6 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } }
I20260709 13:00:59.692757  1673 raft_consensus.cc:1240] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 10. Current term is 11. Ops: []
I20260709 13:00:59.703150  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:00:59.703747  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 107760us
I20260709 13:00:59.704190  1604 raft_consensus.cc:2468] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 10.
I20260709 13:00:59.682109  5991 raft_consensus.cc:493] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:59.652496  6002 consensus_queue.cc:1059] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.039s
I20260709 13:00:59.710338  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:59.711768  1501 catalog_manager.cc:5697] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d reported cstate change: term changed from 1 to 5. New cstate: current_term: 5 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } }
W20260709 13:00:59.706408  6018 consensus_meta.cc:287] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.649s	user 0.013s	sys 0.009s
I20260709 13:00:59.649936  6069 consensus_queue.cc:1059] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.055s
I20260709 13:00:59.718530  1673 raft_consensus.cc:2368] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 11. Current term is 12.
I20260709 13:00:59.718791  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:00:59.724987  1604 raft_consensus.cc:2393] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 10: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:00:59.724004  6080 leader_election.cc:290] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 12 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:59.732352  1569 leader_election.cc:304] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d
W20260709 13:00:59.738334  2298 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:59.717264  5991 raft_consensus.cc:3060] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:00:59.741259  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:00:59.733443  6062 consensus_meta.cc:287] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.510s	user 0.011s	sys 0.008s
I20260709 13:00:59.746206  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:00:59.759083  1603 raft_consensus.cc:1240] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 8. Current term is 9. Ops: []
W20260709 13:00:59.760704  1719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:00:59.761288  1719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:59.761920  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:00:59.762717  1646 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:59.752440  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:59.721556  5228 leader_election.cc:290] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 13:00:59.754743  5970 consensus_meta.cc:287] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.536s	user 0.016s	sys 0.007s
I20260709 13:00:59.710875  5919 consensus_queue.cc:1048] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.047s
W20260709 13:00:59.785801  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:59.790570  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
W20260709 13:00:59.791325  6116 log.cc:927] Time spent T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.211s	user 0.001s	sys 0.001s
W20260709 13:00:59.791550  2300 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:59.792598  6029 raft_consensus.cc:2804] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Leader election won for term 13
I20260709 13:00:59.793268  6029 raft_consensus.cc:697] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:00:59.794359  6029 consensus_queue.cc:237] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.10, Last appended by leader: 10, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:59.796986  6100 log.cc:927] Time spent T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.274s	user 0.005s	sys 0.000s
I20260709 13:00:59.799171  5833 leader_election.cc:290] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 13:00:59.801357  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:59.802587  1571 leader_election.cc:304] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:00:59.804268  1571 leader_election.cc:304] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1
W20260709 13:00:59.805243  1646 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:00:59.791589  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 116374us
I20260709 13:00:59.807487  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:59.814667  1597 raft_consensus.cc:2410] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 for term 13 because replica has last-logged OpId of term: 11 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 9.
I20260709 13:00:59.733479  5777 consensus_queue.cc:1048] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.051s
I20260709 13:00:59.737787  6018 raft_consensus.cc:515] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:00:59.818246  1718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:59.804801  6122 raft_consensus.cc:493] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:59.821163  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:59.832792  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:59.839121  1676 raft_consensus.cc:2468] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 8.
I20260709 13:00:59.839705  1677 raft_consensus.cc:1240] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 LEADER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 6. Current term is 7. Ops: []
W20260709 13:00:59.840505  6113 log.cc:927] Time spent T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.057s	user 0.000s	sys 0.000s
W20260709 13:00:59.842486  6115 log.cc:927] Time spent T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.105s	user 0.000s	sys 0.001s
I20260709 13:00:59.760069  6062 raft_consensus.cc:515] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:00:59.799165  1501 catalog_manager.cc:5697] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d reported cstate change: term changed from 6 to 7, leader changed from 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) to 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194). New cstate: current_term: 7 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:00:59.837713  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:00:59.836697  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:59.820307  6067 raft_consensus.cc:2804] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:00:59.833132  1679 raft_consensus.cc:1240] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 10. Current term is 11. Ops: []
I20260709 13:00:59.843988  5938 raft_consensus.cc:2749] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 13:00:59.788753  5970 raft_consensus.cc:515] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:59.859107  1675 raft_consensus.cc:1240] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 11. Current term is 12. Ops: [11.9-11.9]
I20260709 13:00:59.854626  1680 raft_consensus.cc:2468] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 10.
I20260709 13:00:59.856015  6013 raft_consensus.cc:493] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:59.862835  6013 raft_consensus.cc:515] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:59.863327  1677 raft_consensus.cc:1217] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Deduplicated request from leader. Original: 8.9->[9.10-9.10]   Dedup: 9.10->[]
I20260709 13:00:59.863937  1677 raft_consensus.cc:1240] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 9. Current term is 10. Ops: [9.10-9.10]
W20260709 13:00:59.865613  1718 leader_election.cc:336] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Timed out: RequestConsensusVote RPC to 127.0.107.194:46261 timed out after 1.673s (SENT)
I20260709 13:00:59.832896  5894 raft_consensus.cc:493] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:59.866323  1718 leader_election.cc:304] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:00:59.871788  5894 raft_consensus.cc:515] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:59.877974  5663 raft_consensus.cc:493] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:59.878530  5663 raft_consensus.cc:515] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:59.880421  5894 leader_election.cc:290] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:59.874091  5833 raft_consensus.cc:2749] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:00:59.882578  6018 leader_election.cc:290] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 12 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:00:59.851934  6067 raft_consensus.cc:493] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:00:59.884789  2299 leader_election.cc:304] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:00:59.880868  5663 leader_election.cc:290] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:00:59.888190  5738 raft_consensus.cc:515] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:59.832388  5923 raft_consensus.cc:493] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:59.872896  1501 catalog_manager.cc:5697] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d reported cstate change: term changed from 4 to 5, leader changed from 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) to 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194). New cstate: current_term: 5 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:00:59.914860  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:00:59.916059  1680 raft_consensus.cc:2393] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 9: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:00:59.847254  6122 raft_consensus.cc:515] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:59.924007  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:00:59.925511  5991 raft_consensus.cc:515] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:00:59.954650  6121 raft_consensus.cc:2749] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 13:00:59.955982  6080 raft_consensus.cc:3055] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:00:59.958906  6013 leader_election.cc:290] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 13:00:59.959476  6114 log.cc:927] Time spent T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.085s	user 0.000s	sys 0.000s
I20260709 13:00:59.882783  5873 consensus_queue.cc:1059] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 8.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.069s
I20260709 13:00:59.885847  6067 raft_consensus.cc:3060] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Advancing to term 10
W20260709 13:00:59.972378  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 45368us
I20260709 13:00:59.960052   431 auto_leader_rebalancer.cc:527] All tables' leader rebalancing finished this round
I20260709 13:00:59.974627  1750 raft_consensus.cc:2393] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 12: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:00:59.977499  2333 raft_consensus.cc:1240] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 12. Current term is 13. Ops: []
I20260709 13:00:59.978071  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:00:59.995481  2299 leader_election.cc:304] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:00:59.978295  5991 leader_election.cc:290] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 15 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:00:59.981027  5738 leader_election.cc:290] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 12 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:00:59.981472  5844 consensus_meta.cc:287] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.544s	user 0.019s	sys 0.002s
W20260709 13:00:59.982429  1644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:59.986693  5994 raft_consensus.cc:2749] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Leader election lost for term 10. Reason: Vote denied by peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693) with higher term. Message: Invalid argument: T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 for earlier term 10. Current term is 11.
I20260709 13:00:59.979503  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:00.003988  2331 raft_consensus.cc:3060] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:00:59.987521  2336 raft_consensus.cc:1275] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 11 index: 11. Preceding OpId from leader: term: 12 index: 11. (term mismatch)
W20260709 13:00:59.976500  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 50936.8us
I20260709 13:00:59.995585  2335 raft_consensus.cc:2468] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 10.
I20260709 13:00:59.979501  1679 raft_consensus.cc:1240] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 12. Current term is 13. Ops: []
W20260709 13:00:59.996621  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 136959us
I20260709 13:00:59.998957  1569 leader_election.cc:304] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:01:00.011145  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:00.004487  2336 pending_rounds.cc:85] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1: Aborting all ops after (but not including) 10
I20260709 13:01:00.013823  2328 raft_consensus.cc:2468] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 8.
I20260709 13:01:00.013990  1600 raft_consensus.cc:1240] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 LEADER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 10. Current term is 11. Ops: []
I20260709 13:01:00.015275  1569 leader_election.cc:304] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:01:00.015553  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:00.018850  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:00:59.989188  6080 raft_consensus.cc:740] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:01:00.030659  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:00.031675  1679 raft_consensus.cc:2410] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for term 9 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 5.
I20260709 13:01:00.013819  2336 pending_rounds.cc:107] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1: Aborting uncommitted NO_OP operation due to leader change: 11.11
I20260709 13:01:00.033417  2336 raft_consensus.cc:2889] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:01:00.029793  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:00.036127  1501 catalog_manager.cc:5697] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d reported cstate change: term changed from 6 to 8. New cstate: current_term: 8 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
W20260709 13:01:00.036548  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 232334us
W20260709 13:01:00.039638  1718 leader_election.cc:336] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Timed out: RequestConsensusVote RPC to 127.0.107.194:46261 timed out after 1.743s (SENT)
I20260709 13:00:59.975731  5923 raft_consensus.cc:515] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:00.047255  1571 leader_election.cc:304] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:01:00.004773  6013 consensus_queue.cc:1048] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 7, Time since last communication: 0.038s
I20260709 13:01:00.021611  5777 raft_consensus.cc:493] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:00.053143  1718 leader_election.cc:304] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:00.027555  6062 leader_election.cc:290] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 13 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:00.032806  6082 raft_consensus.cc:2804] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:01:00.055433  1501 catalog_manager.cc:5697] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d reported cstate change: term changed from 4 to 7, leader changed from 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194) to 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196). New cstate: current_term: 7 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
W20260709 13:01:00.078610  6130 log.cc:927] Time spent T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.057s	user 0.000s	sys 0.001s
I20260709 13:01:00.052702  2299 leader_election.cc:304] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: af520b67945d443b901bbeca37cc5701
W20260709 13:01:00.112738  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 60150us
I20260709 13:01:00.063241  5926 raft_consensus.cc:2749] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:01:00.081792  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:00.064149  6080 consensus_queue.cc:260] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:00.119827  1672 raft_consensus.cc:2410] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for term 10 because replica has last-logged OpId of term: 9 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 5.
I20260709 13:01:00.058724  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:00.050247  5663 consensus_queue.cc:1059] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true }, Status: INVALID_TERM, Last received: 8.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.043s
I20260709 13:01:00.039207  6018 consensus_queue.cc:1059] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 7, Time since last communication: 0.055s
I20260709 13:01:00.082062  6013 consensus_queue.cc:1243] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer 816fadcea8fc4893882f5f9b9f22a03d log is divergent from this leader: its last log entry 10.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20260709 13:01:00.092029  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:00.095017  5970 leader_election.cc:290] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 12 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:00.096742  5777 raft_consensus.cc:515] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:00.098817  1571 leader_election.cc:304] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:00.032706  6131 raft_consensus.cc:2749] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:01:00.037564  6029 raft_consensus.cc:2804] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:01:00.119480  6082 raft_consensus.cc:493] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:00.179105  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:00.176783  1745 raft_consensus.cc:1275] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260709 13:01:00.177296  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:00.153456  6029 raft_consensus.cc:493] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:00.181643  6029 raft_consensus.cc:3060] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:01:00.181813  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:00.182865  1680 raft_consensus.cc:2410] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for term 9 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 4.
I20260709 13:01:00.184047  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:00.184793  5991 consensus_queue.cc:1059] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 9.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 13:01:00.188125  5777 leader_election.cc:290] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:00.189735  5671 raft_consensus.cc:3055] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:01:00.190191  5671 raft_consensus.cc:740] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:01:00.191221  5671 consensus_queue.cc:260] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:00.184625  5926 raft_consensus.cc:2804] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:01:00.193226  5926 raft_consensus.cc:493] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:00.193630  5926 raft_consensus.cc:3060] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:01:00.194602  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:00.194993  2301 leader_election.cc:304] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:00.195449  1678 raft_consensus.cc:2393] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 9: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:01:00.197991  6125 raft_consensus.cc:2804] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:01:00.154647  5955 raft_consensus.cc:2804] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Leader election won for term 13
W20260709 13:01:00.206943  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 173975us
W20260709 13:01:00.211633  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 196415us
I20260709 13:01:00.184689  1680 raft_consensus.cc:2468] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 8.
I20260709 13:01:00.210464  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:00.218056  2332 raft_consensus.cc:2468] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 11.
I20260709 13:01:00.209303  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:00.232195  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:00.233556  1673 raft_consensus.cc:2376] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 8. Re-sending same reply.
I20260709 13:01:00.233952  1749 raft_consensus.cc:3060] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:01:00.222030  1679 raft_consensus.cc:2410] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for term 9 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 4.
W20260709 13:01:00.219072  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 146646us
W20260709 13:01:00.228302  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 48311.4us
I20260709 13:01:00.204460  5991 raft_consensus.cc:3055] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 13:01:00.137969  5738 raft_consensus.cc:2749] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:01:00.208549  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:00.232460  2299 leader_election.cc:304] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:01:00.198413  6125 raft_consensus.cc:493] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:00.245814  1674 raft_consensus.cc:2393] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 9: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:01:00.241567  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:00.209393  6080 raft_consensus.cc:3060] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:01:00.246858  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:00.248112  1680 raft_consensus.cc:2393] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 9: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:01:00.207581  5955 raft_consensus.cc:697] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 13 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:01:00.200838  5970 raft_consensus.cc:993] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1: : Instructing follower af520b67945d443b901bbeca37cc5701 to start an election
I20260709 13:01:00.211831  6082 raft_consensus.cc:3060] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:01:00.250344  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:00.261636  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:00.220148  2331 raft_consensus.cc:2468] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 12.
I20260709 13:01:00.262540  1596 raft_consensus.cc:3060] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:01:00.253328  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:00.253589  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:00.257596  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:00.252429  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:00.264497  1673 raft_consensus.cc:2393] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 13: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:01:00.261044  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:00.266479  2330 raft_consensus.cc:2393] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 12: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:01:00.267805  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:00.270799  1674 raft_consensus.cc:2410] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for term 10 because replica has last-logged OpId of term: 9 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 4.
I20260709 13:01:00.209668  6029 raft_consensus.cc:515] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:01:00.272730  1718 leader_election.cc:336] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Timed out: RequestConsensusVote RPC to 127.0.107.194:46261 timed out after 2.076s (SENT)
W20260709 13:01:00.266288  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 68933.8us
I20260709 13:01:00.275115  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:00.276057  1679 raft_consensus.cc:2410] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for term 9 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 5.
I20260709 13:01:00.277637  2329 raft_consensus.cc:1240] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 14. Current term is 15. Ops: [14.11-14.11]
W20260709 13:01:00.277886  6112 log.cc:927] Time spent T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.602s	user 0.003s	sys 0.000s
I20260709 13:01:00.278807  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:00.244001  5991 raft_consensus.cc:740] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:01:00.250869  6125 raft_consensus.cc:3060] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:01:00.281407  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:01:00.280133  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:00.271839  5919 raft_consensus.cc:2804] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:01:00.306787  5923 leader_election.cc:290] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:00.307106  5919 raft_consensus.cc:493] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:00.307572  5919 raft_consensus.cc:3060] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:01:00.308043  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:00.308395  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:00.309401  1679 raft_consensus.cc:2393] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 13: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:01:00.269706  6126 raft_consensus.cc:1081] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 12 LEADER]: Signalling peer af520b67945d443b901bbeca37cc5701 to start an election
I20260709 13:01:00.298527  2299 leader_election.cc:304] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:00.298925  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:00.310352  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:00.319346  1679 raft_consensus.cc:2410] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for term 10 because replica has last-logged OpId of term: 9 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 5.
I20260709 13:01:00.280551  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:00.330870  1673 raft_consensus.cc:2468] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 10.
I20260709 13:01:00.334844  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:00.341536  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:00.342375  1672 raft_consensus.cc:1240] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 11. Current term is 12. Ops: []
I20260709 13:01:00.308068  5844 raft_consensus.cc:3055] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 8 LEADER]: Stepping down as leader of term 8
W20260709 13:01:00.344992  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 246269us
W20260709 13:01:00.348011  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 81568.4us
W20260709 13:01:00.348258  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 49895.2us
I20260709 13:01:00.346007  1749 raft_consensus.cc:2468] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 13.
I20260709 13:01:00.346808  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:00.350621  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
W20260709 13:01:00.352147  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:00.354564  1644 leader_election.cc:304] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:01:00.356595  1569 leader_election.cc:304] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:00.359462  1571 leader_election.cc:304] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:01:00.362912  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:00.366377  1644 leader_election.cc:304] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:01:00.363160  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:00.371708  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:00.372115  2328 raft_consensus.cc:1275] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260709 13:01:00.369968  1644 leader_election.cc:304] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:01:00.347801  5844 raft_consensus.cc:740] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:01:00.376791  1672 raft_consensus.cc:1275] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260709 13:01:00.353453  5873 raft_consensus.cc:2749] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:01:00.377233  6069 raft_consensus.cc:2804] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:01:00.399256  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:00.405300  6140 raft_consensus.cc:2749] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:01:00.406908  2330 raft_consensus.cc:2468] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 12.
I20260709 13:01:00.408114  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:01:00.409740  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 59517.9us
I20260709 13:01:00.373263  6046 raft_consensus.cc:2804] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:01:00.373517  6145 raft_consensus.cc:2804] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:01:00.415977  1679 raft_consensus.cc:1275] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260709 13:01:00.365494  5833 consensus_queue.cc:1059] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 12.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.022s
I20260709 13:01:00.386816  5923 raft_consensus.cc:2749] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
W20260709 13:01:00.422181  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 62776.4us
I20260709 13:01:00.361629  5955 consensus_queue.cc:237] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:00.360509  5991 consensus_queue.cc:260] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 9, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:00.458222  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:00.416402  6069 raft_consensus.cc:697] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:01:00.420070  6145 raft_consensus.cc:493] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:00.419294  6046 raft_consensus.cc:493] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:01:00.440990  6140 consensus_queue.cc:1048] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.064s
I20260709 13:01:00.442391  1571 leader_election.cc:304] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:01:00.581918  1679 raft_consensus.cc:1275] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
I20260709 13:01:00.583782  1596 raft_consensus.cc:2468] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 15.
W20260709 13:01:00.589016  2299 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:00.504520  1753 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23"
dest_uuid: "af520b67945d443b901bbeca37cc5701"
 from {username='slave'} at 127.0.0.1:43156
I20260709 13:01:00.590692  1753 raft_consensus.cc:493] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:01:00.591203  1753 raft_consensus.cc:3060] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:01:00.594483  1751 raft_consensus.cc:1275] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
W20260709 13:01:00.601781  1718 leader_election.cc:336] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Timed out: RequestConsensusVote RPC to 127.0.107.194:46261 timed out after 2.111s (SENT)
W20260709 13:01:00.602846  1644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:00.613248  1718 leader_election.cc:304] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:00.584438  6046 raft_consensus.cc:3060] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:01:00.620972  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:00.583220  6145 raft_consensus.cc:3060] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:01:00.466251  5844 consensus_queue.cc:260] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.7, Last appended by leader: 7, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:00.636718  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 194399us
I20260709 13:01:00.641693  1603 raft_consensus.cc:1240] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 8. Current term is 9. Ops: []
I20260709 13:01:00.632305  5777 raft_consensus.cc:493] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:00.645869  5777 raft_consensus.cc:515] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:00.649374  5777 leader_election.cc:290] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:00.645130  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:00.656630  1678 raft_consensus.cc:1275] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 10 index: 5. (term mismatch)
I20260709 13:01:00.658179  1569 leader_election.cc:304] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:01:00.622731  6140 raft_consensus.cc:2749] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 13:01:00.657429  1678 pending_rounds.cc:85] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d: Aborting all ops after (but not including) 4
I20260709 13:01:00.661531  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:01:00.662994  2300 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:00.663493  1571 leader_election.cc:304] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:01:00.661505  1678 pending_rounds.cc:107] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d: Aborting uncommitted NO_OP operation due to leader change: 9.5
I20260709 13:01:00.667038  1678 raft_consensus.cc:2889] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:01:00.658506  6122 leader_election.cc:290] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:00.671576  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:00.664332  5777 raft_consensus.cc:2804] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:01:00.662479  6161 raft_consensus.cc:493] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:00.639603  6125 raft_consensus.cc:515] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:00.689865  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:00.674054  6143 raft_consensus.cc:2749] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:01:00.700155  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:00.679272  5844 raft_consensus.cc:3060] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:01:00.698522  1679 raft_consensus.cc:2468] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 10.
W20260709 13:01:00.702878  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 64107.7us
I20260709 13:01:00.682816  5777 raft_consensus.cc:493] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:00.714659  5777 raft_consensus.cc:3060] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Advancing to term 13
W20260709 13:01:00.716940  1646 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:00.717723  6016 raft_consensus.cc:493] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:00.718413  6016 raft_consensus.cc:515] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:00.719053  2335 raft_consensus.cc:1275] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260709 13:01:00.719923  6160 raft_consensus.cc:2749] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:01:00.720956  6016 leader_election.cc:290] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:00.705309  1678 raft_consensus.cc:2468] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 7.
W20260709 13:01:00.717764  1646 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:00.725865  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:01:00.726045  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 140689us
W20260709 13:01:00.703600  6153 log.cc:927] Time spent T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.120s	user 0.001s	sys 0.000s
I20260709 13:01:00.724118  1571 leader_election.cc:304] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:01:00.729543  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:00.727197  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:00.731070  1674 raft_consensus.cc:2393] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 12: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:01:00.701112  6140 raft_consensus.cc:493] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:00.732492  6140 raft_consensus.cc:515] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:01:00.735077  5926 consensus_meta.cc:287] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.541s	user 0.014s	sys 0.007s
I20260709 13:01:00.735612  5926 raft_consensus.cc:515] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } }
I20260709 13:01:00.688861  6161 raft_consensus.cc:515] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:00.744055  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:01:00.746992  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 132817us
I20260709 13:01:00.676298  6069 consensus_queue.cc:237] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:00.751622  6140 leader_election.cc:290] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:00.752381  6143 raft_consensus.cc:3060] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:01:00.755442  1748 raft_consensus.cc:1275] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
W20260709 13:01:00.746778  6067 consensus_meta.cc:287] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.725s	user 0.017s	sys 0.001s
I20260709 13:01:00.757892  6067 raft_consensus.cc:515] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:00.760264  6067 leader_election.cc:290] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 10 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:00.761063  1596 raft_consensus.cc:1240] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 9. Current term is 10. Ops: []
W20260709 13:01:00.761296  6133 log.cc:927] Time spent T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.606s	user 0.000s	sys 0.010s
I20260709 13:01:00.765208  5671 raft_consensus.cc:3060] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:01:00.766616  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:00.767578  1752 raft_consensus.cc:1275] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260709 13:01:00.770061  2298 leader_election.cc:304] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:01:00.773851  1748 raft_consensus.cc:2468] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 9.
I20260709 13:01:00.774840  5919 raft_consensus.cc:515] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:00.780378  5919 leader_election.cc:290] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 9 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:00.780458  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
W20260709 13:01:00.784308  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:00.772868  6149 raft_consensus.cc:2804] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 13:01:00.784936  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:00.785957  1677 raft_consensus.cc:2468] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 7.
I20260709 13:01:00.792042  6125 leader_election.cc:290] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 12 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:00.738340  5926 leader_election.cc:290] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 8 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:00.743227  6016 raft_consensus.cc:493] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:01:00.803714  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 99954.1us
I20260709 13:01:00.794677  1753 raft_consensus.cc:515] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:00.807147  1751 raft_consensus.cc:1240] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 12. Current term is 13. Ops: []
W20260709 13:01:00.775195  6080 consensus_meta.cc:287] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.518s	user 0.007s	sys 0.007s
W20260709 13:01:00.811349  1718 leader_election.cc:336] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 10 election: RPC error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Timed out: RequestConsensusVote RPC to 127.0.107.194:46261 timed out after 2.360s (SENT)
I20260709 13:01:00.812706  1571 leader_election.cc:304] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:01:00.775456  5989 raft_consensus.cc:493] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:00.818866  1673 raft_consensus.cc:1275] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 9 index: 9. (term mismatch)
I20260709 13:01:00.820797  1673 pending_rounds.cc:85] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d: Aborting all ops after (but not including) 8
I20260709 13:01:00.821211  1673 pending_rounds.cc:107] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d: Aborting uncommitted NO_OP operation due to leader change: 8.9
I20260709 13:01:00.821512  1673 raft_consensus.cc:2889] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 13:01:00.823817  2298 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:00.826753  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:00.827389  1680 raft_consensus.cc:3060] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:01:00.827806  1753 leader_election.cc:290] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 13 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:00.825977  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:00.829344  1599 raft_consensus.cc:3060] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Advancing to term 9
I20260709 13:01:00.773114  6082 raft_consensus.cc:515] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:00.847434  5999 raft_consensus.cc:493] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:00.848255  5999 raft_consensus.cc:515] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:00.849081  6082 leader_election.cc:290] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 9 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:00.812093  6149 raft_consensus.cc:697] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 15 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:01:00.845980  6161 leader_election.cc:290] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 13:01:00.840960  1717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:00.835062  5989 raft_consensus.cc:515] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:00.826545  6002 raft_consensus.cc:493] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:00.870597  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:01:00.874025  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 67750.2us
I20260709 13:01:00.809265  6016 raft_consensus.cc:515] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:00.878405  1676 raft_consensus.cc:2468] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 10.
I20260709 13:01:00.879583  1571 leader_election.cc:304] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
W20260709 13:01:00.882843  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 77738us
W20260709 13:01:00.884493  1718 leader_election.cc:336] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Timed out: RequestConsensusVote RPC to 127.0.107.194:46261 timed out after 1.899s (SENT)
I20260709 13:01:00.884928  1718 leader_election.cc:304] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:00.851495  5926 raft_consensus.cc:2749] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:01:00.865192  6143 raft_consensus.cc:2749] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:01:00.890833  1675 raft_consensus.cc:3055] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:01:00.866710  6162 raft_consensus.cc:493] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:01:00.891435  1718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:00.894176  1675 raft_consensus.cc:740] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
W20260709 13:01:00.895298  1718 leader_election.cc:336] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 9 election: RPC error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Timed out: RequestConsensusVote RPC to 127.0.107.194:46261 timed out after 1.961s (SENT)
I20260709 13:01:00.895249  1675 consensus_queue.cc:260] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.11, Last appended by leader: 11, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:00.908079  1675 raft_consensus.cc:3060] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:01:00.914314  1680 raft_consensus.cc:2468] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 11.
I20260709 13:01:00.893530  6080 raft_consensus.cc:493] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:00.881428  6018 raft_consensus.cc:493] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:00.898677  6162 raft_consensus.cc:515] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:00.918447  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:00.921644  1571 leader_election.cc:304] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 13:01:00.898828  6082 raft_consensus.cc:2749] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:01:00.892794  6161 raft_consensus.cc:2749] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:01:00.881021  6002 raft_consensus.cc:515] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:00.926318  1571 leader_election.cc:304] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:01:00.927189  1753 raft_consensus.cc:1275] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260709 13:01:00.929518  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 10 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:00.930845  1676 raft_consensus.cc:2410] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for term 10 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20260709 13:01:00.927515  6143 raft_consensus.cc:2804] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:01:00.931453  6143 raft_consensus.cc:493] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:00.931885  6143 raft_consensus.cc:3060] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:01:00.931969  5989 leader_election.cc:290] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:00.922547  5999 leader_election.cc:290] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:00.942054  6029 leader_election.cc:290] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:00.946985  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:00.947847  1671 raft_consensus.cc:2393] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 12: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:01:00.950364  6162 leader_election.cc:290] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:00.955108  1646 leader_election.cc:304] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:01:00.955613  6016 leader_election.cc:290] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:00.956985  6016 raft_consensus.cc:2749] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 13:01:00.956918  6171 consensus_queue.cc:1048] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 13:01:00.939309  5926 raft_consensus.cc:2749] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 13:01:00.958797  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:00.959017  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 12 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:00.922925  6080 raft_consensus.cc:515] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:00.959726  1602 raft_consensus.cc:2468] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 11.
I20260709 13:01:00.961567  6080 leader_election.cc:290] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:00.966089  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:00.966662  5738 raft_consensus.cc:3055] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:01:00.966831  1751 raft_consensus.cc:2468] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 9.
I20260709 13:01:00.967144  5738 raft_consensus.cc:740] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:01:00.967837  5938 consensus_queue.cc:1048] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 13:01:00.968132  6147 consensus_queue.cc:1048] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 13:01:00.971064  2298 leader_election.cc:304] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:01:00.973315  2299 leader_election.cc:304] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:00.974391  5873 raft_consensus.cc:2804] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:01:00.974820  5873 raft_consensus.cc:493] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:00.975112  6002 leader_election.cc:290] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:00.975204  5873 raft_consensus.cc:3060] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:01:00.975775  6002 raft_consensus.cc:2804] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:01:00.976191  6002 raft_consensus.cc:493] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:00.976547  6002 raft_consensus.cc:3060] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:01:00.976421  1646 leader_election.cc:304] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
W20260709 13:01:00.978654  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 50955.4us
I20260709 13:01:00.968176  5738 consensus_queue.cc:260] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.11, Last appended by leader: 11, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:00.979449  6046 raft_consensus.cc:515] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:00.980137  5738 raft_consensus.cc:3060] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:01:00.979946  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:00.981837  6046 leader_election.cc:290] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 11 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:00.984393  6162 consensus_queue.cc:1059] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 13:01:00.985589  6162 raft_consensus.cc:3055] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:01:00.986011  6162 raft_consensus.cc:740] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:01:00.986491  6171 raft_consensus.cc:493] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:00.987064  6171 raft_consensus.cc:515] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:00.989805  6080 raft_consensus.cc:493] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:00.938731  6149 consensus_queue.cc:237] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.10, Last appended by leader: 10, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:00.990577  6080 raft_consensus.cc:515] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:00.995673  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 14 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:00.998346  6069 raft_consensus.cc:2804] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Leader election won for term 13
W20260709 13:01:00.987331  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 76088.5us
I20260709 13:01:00.981125  1676 raft_consensus.cc:3060] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:01:01.009016  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:01.012876  1599 raft_consensus.cc:2468] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 9.
W20260709 13:01:01.014972  1719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:01.016985  1672 pending_rounds.cc:85] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d: Aborting all ops after (but not including) 7
I20260709 13:01:01.017372  1672 pending_rounds.cc:107] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d: Aborting uncommitted NO_OP operation due to leader change: 10.8
I20260709 13:01:01.017701  1672 raft_consensus.cc:2889] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 13:01:01.018761  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 45545us
I20260709 13:01:01.019291  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 12 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:01.020102  1679 raft_consensus.cc:2468] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 10.
I20260709 13:01:01.029978  1675 raft_consensus.cc:1275] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 13 index: 11. (term mismatch)
I20260709 13:01:01.030355  1675 pending_rounds.cc:85] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d: Aborting all ops after (but not including) 10
I20260709 13:01:01.031344  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 14 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:01.032747  1646 leader_election.cc:304] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:01.031385  1675 pending_rounds.cc:107] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d: Aborting uncommitted NO_OP operation due to leader change: 12.11
I20260709 13:01:01.033744  1675 raft_consensus.cc:2889] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:01:01.034257  6046 consensus_queue.cc:1048] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 13:01:01.035506  6175 consensus_queue.cc:1048] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 13:01:01.035946  6046 raft_consensus.cc:2804] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:01:01.036320  6033 raft_consensus.cc:493] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:01.032159  2298 leader_election.cc:304] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:01:01.043377  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:01.044569  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:01.037082  6175 consensus_queue.cc:1048] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 13:01:01.049923  5991 raft_consensus.cc:3060] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:01:01.051247  6171 leader_election.cc:290] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:00.987077  6162 consensus_queue.cc:260] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:01.053220  6162 raft_consensus.cc:3060] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:01:01.054971  6135 raft_consensus.cc:2804] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 13:01:01.055562  6135 raft_consensus.cc:697] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 9 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:01:01.056550  6135 consensus_queue.cc:237] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:01.058768  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:01.059569  1748 raft_consensus.cc:3060] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:01:01.058820  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: true dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:01.061885  2336 raft_consensus.cc:3055] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:01:01.064492  1673 raft_consensus.cc:1240] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 8. Current term is 9. Ops: []
W20260709 13:01:01.061866  5894 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 13 current term: 13 is higher.
I20260709 13:01:01.062683  2336 raft_consensus.cc:740] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:01:01.043497  5663 consensus_queue.cc:1048] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.043s
I20260709 13:01:00.999022  6069 raft_consensus.cc:697] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:01:00.928856  6018 raft_consensus.cc:515] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:00.958153  6145 raft_consensus.cc:515] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:01.036341  6046 raft_consensus.cc:493] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:01:01.086371  1717 leader_election.cc:336] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 1.571s (SENT)
I20260709 13:01:01.069149  2301 leader_election.cc:304] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:01:01.103796   431 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:01:01.106958  2336 consensus_queue.cc:260] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.9, Last appended by leader: 9, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:01.107628  5926 raft_consensus.cc:493] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:01.108292  2336 raft_consensus.cc:3060] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:01:01.108235  5926 raft_consensus.cc:515] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:01:01.111469  6169 log.cc:927] Time spent T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.176s	user 0.000s	sys 0.000s
I20260709 13:01:01.112783  6174 raft_consensus.cc:493] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:01.113389  6174 raft_consensus.cc:515] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:01.115860  6174 leader_election.cc:290] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:01.087638  6016 consensus_queue.cc:1048] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.038s
I20260709 13:01:01.123648  6160 consensus_queue.cc:1059] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 13:01:01.124724  6160 consensus_queue.cc:1059] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 13:01:01.109786  5873 raft_consensus.cc:515] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:01.131304  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 62247.1us
I20260709 13:01:01.037036  6033 raft_consensus.cc:515] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:01.129949  1676 raft_consensus.cc:2468] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 9.
I20260709 13:01:01.134141  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:01.134405  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:01.135427  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:01.136518  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:01.134951  1598 raft_consensus.cc:3060] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:01:01.133481  2301 leader_election.cc:304] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:01:01.140367  1600 raft_consensus.cc:1240] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 8. Current term is 9. Ops: []
W20260709 13:01:01.162930  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 195510us
I20260709 13:01:01.133453  1717 leader_election.cc:304] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:01.186370  1569 leader_election.cc:304] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:01:01.187892  1569 leader_election.cc:304] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:01:01.192878  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:01:01.194778  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 61373.5us
I20260709 13:01:01.201759  1680 raft_consensus.cc:2393] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 9: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:01:01.218856  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:01.219035  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:01:01.227463  1644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:01.231750  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:01.245215  2331 raft_consensus.cc:2468] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 10.
I20260709 13:01:01.126971  6046 raft_consensus.cc:3060] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Advancing to term 10
W20260709 13:01:01.254401  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 59933.8us
I20260709 13:01:01.260053  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:01.260906  1599 raft_consensus.cc:2393] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 10: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:01:01.253666  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:01.126354  6080 leader_election.cc:290] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:01.085148  6171 consensus_queue.cc:1048] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 6, Time since last communication: 0.106s
I20260709 13:01:01.277097  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:01.253539  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:01.270394  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:01.191149  5663 raft_consensus.cc:2749] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:01:01.282203  1605 raft_consensus.cc:2393] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 10: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:01:01.282842  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:01.203725  5926 leader_election.cc:290] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:01.257243  5919 raft_consensus.cc:2749] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 13:01:01.263906  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:01.291416  1602 raft_consensus.cc:3060] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:01:01.273201  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
W20260709 13:01:01.273667  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 77994.6us
I20260709 13:01:01.263967  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:01.279752  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:01.193689  6082 raft_consensus.cc:493] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:01.285595  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:01.267876  1569 leader_election.cc:304] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:01:01.288146  1748 raft_consensus.cc:2468] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 12.
I20260709 13:01:01.278779  1671 raft_consensus.cc:2468] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 13.
I20260709 13:01:01.159410  6111 consensus_queue.cc:1059] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.122s
I20260709 13:01:01.259303  5833 raft_consensus.cc:3055] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:01:01.266208  5894 raft_consensus.cc:493] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:01:01.303817  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 217444us
I20260709 13:01:01.304627  6069 consensus_queue.cc:237] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:01.277951  5738 raft_consensus.cc:2749] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:01:01.265748  5999 consensus_queue.cc:1059] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.044s
I20260709 13:01:01.264935  6161 consensus_queue.cc:1059] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 10.6, Next index: 7, Last known committed idx: 4, Time since last communication: 0.041s
I20260709 13:01:01.311146  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:01.306661  6082 raft_consensus.cc:515] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:01.316982  1599 raft_consensus.cc:1240] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 9. Current term is 10. Ops: []
I20260709 13:01:01.318836  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:01.319317  1679 raft_consensus.cc:1240] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 10. Current term is 11. Ops: []
I20260709 13:01:01.320330  5873 leader_election.cc:290] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 12 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:01.272984  5938 raft_consensus.cc:2804] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Leader election won for term 11
I20260709 13:01:01.189492  6160 consensus_queue.cc:1059] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.094s
I20260709 13:01:01.333633  5926 consensus_queue.cc:1048] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.019s
I20260709 13:01:01.301517  1600 raft_consensus.cc:2468] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 9.
I20260709 13:01:01.325960  1602 raft_consensus.cc:2468] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 12.
I20260709 13:01:01.330552  5833 raft_consensus.cc:740] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:01:01.366317  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 11 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:01.269547  6140 raft_consensus.cc:2804] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:01:01.273391  6163 raft_consensus.cc:3055] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:01:01.309602  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:01.336979  2336 raft_consensus.cc:2468] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 13.
W20260709 13:01:01.292677  5777 consensus_meta.cc:287] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.577s	user 0.010s	sys 0.007s
W20260709 13:01:01.344728  2299 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:01.345252  2298 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:01.348192  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 314413us
I20260709 13:01:01.291987  5955 consensus_queue.cc:1048] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.058s
I20260709 13:01:01.361495  1674 raft_consensus.cc:2393] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 11: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:01:01.331043  5923 raft_consensus.cc:2804] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Leader election won for term 9
I20260709 13:01:01.366324  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:01.366609  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:01.367343  1600 raft_consensus.cc:2393] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 11: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:01:01.368815  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:01.370190  1605 raft_consensus.cc:1240] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 8. Current term is 9. Ops: []
W20260709 13:01:01.372762  6167 log.cc:927] Time spent T 00000000000000000000000000000000 P 919f6392d4f44fd68987e2c5e7ec37ca: Append to log took a long time: real 0.471s	user 0.003s	sys 0.000s
I20260709 13:01:01.373950  6082 leader_election.cc:290] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:01.374465  1597 raft_consensus.cc:2468] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 12.
I20260709 13:01:01.374927  5938 raft_consensus.cc:697] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:01:01.409723  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:01.376183  6033 leader_election.cc:290] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:01:01.393167  2298 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:01.393903  1603 raft_consensus.cc:2393] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 9: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:01:01.392396  1646 leader_election.cc:304] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:01.418567  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:01.419705  1600 raft_consensus.cc:2393] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 10: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:01:01.397804  1596 raft_consensus.cc:2468] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 12.
I20260709 13:01:01.385495  6163 raft_consensus.cc:740] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:01:01.368582  6140 raft_consensus.cc:493] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:01.389988  5738 raft_consensus.cc:493] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:01.429232  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:01.435551  1600 raft_consensus.cc:2393] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 10: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:01:01.436388  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:01.401525  5777 raft_consensus.cc:515] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:01.404078  1601 raft_consensus.cc:2468] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 9.
I20260709 13:01:01.352785  5894 raft_consensus.cc:515] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:01.422914  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:01.447817  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:01.426625  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:01.447956  1599 raft_consensus.cc:2393] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 9: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:01:01.426723  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 11 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:01.463279  1596 raft_consensus.cc:2468] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 10.
I20260709 13:01:01.484059  1604 raft_consensus.cc:2368] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for earlier term 12. Current term is 13.
I20260709 13:01:01.412753  6018 leader_election.cc:290] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:01.408797  5923 raft_consensus.cc:697] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:01:01.506930  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:01.508371  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:01.486456  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:01.510133  1603 raft_consensus.cc:2393] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 8: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:01:01.487721  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: true dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:01.510747  1600 raft_consensus.cc:3060] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Advancing to term 13
W20260709 13:01:01.489919  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 89117us
I20260709 13:01:01.433843  6140 raft_consensus.cc:3060] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Advancing to term 11
W20260709 13:01:01.493135  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 80781.1us
W20260709 13:01:01.494927  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 138218us
I20260709 13:01:01.497987  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:01.502104  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:01.516390  1602 raft_consensus.cc:2393] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 10: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:01:01.522982  1748 raft_consensus.cc:2393] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 13: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:01:01.516724  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:01.523877  1601 raft_consensus.cc:2468] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 7.
I20260709 13:01:01.521729  2299 leader_election.cc:304] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:01.521180  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:01.527603  1599 raft_consensus.cc:2393] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 9: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:01:01.530465  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:01.522943  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:01.531992  1601 raft_consensus.cc:1240] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 7. Current term is 8. Ops: []
W20260709 13:01:01.534272  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 241567us
I20260709 13:01:01.537206  1598 raft_consensus.cc:2468] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 11.
W20260709 13:01:01.489106  1646 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:01.541864  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:01.541208  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 12 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
W20260709 13:01:01.527825  6143 consensus_meta.cc:287] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.595s	user 0.012s	sys 0.004s
I20260709 13:01:01.521925  6145 leader_election.cc:290] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 9 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:01.485031  5738 raft_consensus.cc:515] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:01.553380  5833 consensus_queue.cc:260] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:01.536278  6018 raft_consensus.cc:2804] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:01:01.548094  1601 raft_consensus.cc:1275] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260709 13:01:01.581120  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:01.541975  6121 raft_consensus.cc:2749] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:01:01.587366  6190 consensus_queue.cc:1059] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 13:01:01.556242  5777 leader_election.cc:290] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:01.553429  6143 raft_consensus.cc:515] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:01.569453  6192 raft_consensus.cc:3055] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 13:01:01.588827  1602 raft_consensus.cc:1240] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 7. Current term is 8. Ops: []
I20260709 13:01:01.589099  6192 raft_consensus.cc:740] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:01:01.558696  5738 leader_election.cc:290] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:01.559917  5923 consensus_queue.cc:237] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:01.574544  6189 consensus_queue.cc:1048] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.016s
I20260709 13:01:01.575423  6018 raft_consensus.cc:493] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:01.590349  6192 consensus_queue.cc:260] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.7, Last appended by leader: 7, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:01.591356  6018 raft_consensus.cc:3060] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:01:01.592000  6192 raft_consensus.cc:3060] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:01:01.561425  5833 raft_consensus.cc:3060] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:01:01.555804  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
W20260709 13:01:01.573966  6002 consensus_meta.cc:287] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.597s	user 0.013s	sys 0.002s
I20260709 13:01:01.563959  5894 leader_election.cc:290] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:01.594918  6002 raft_consensus.cc:515] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:01.564801  5938 consensus_queue.cc:237] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:01.578877  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:01.551522  1599 raft_consensus.cc:1275] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260709 13:01:01.590373  5738 consensus_queue.cc:1048] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.005s
I20260709 13:01:01.554220  6163 consensus_queue.cc:260] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.7, Last appended by leader: 7, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:01.635169  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:01.544212  1718 leader_election.cc:304] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
W20260709 13:01:01.636423  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 92371us
I20260709 13:01:01.605822  1600 raft_consensus.cc:2468] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 13.
I20260709 13:01:01.554716  1598 raft_consensus.cc:3060] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Advancing to term 12
W20260709 13:01:01.612843  2331 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49082 (request call id 889) took 1288 ms (client timeout 1704 ms). Trace:
I20260709 13:01:01.601560  1601 raft_consensus.cc:2468] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 6.
I20260709 13:01:01.624850  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:01.655922  1643 leader_election.cc:304] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:01.655690  1718 leader_election.cc:304] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:01.660522  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:01:01.653898  2331 rpcz_store.cc:269] 0709 13:01:00.324706 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:00.339012 (+ 14306us) service_pool.cc:224] Handling call
0709 13:01:01.612635 (+1273623us) spinlock_profiling.cc:243] Waited 775 ms on lock 0x7b54007602d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:01:01.612771 (+   136us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1781643648}
I20260709 13:01:01.661044  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:01.662146  1749 raft_consensus.cc:2468] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 6.
I20260709 13:01:01.663353  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 10 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:01.664405  1749 raft_consensus.cc:2410] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for term 10 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20260709 13:01:01.661164  1747 raft_consensus.cc:2468] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 10.
I20260709 13:01:01.665423  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:01.667336  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:01.665731  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:01.669559  1747 raft_consensus.cc:3060] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Advancing to term 8
W20260709 13:01:01.672022  6162 consensus_meta.cc:287] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.618s	user 0.016s	sys 0.000s
I20260709 13:01:01.660275  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:01.679414  1750 raft_consensus.cc:2468] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 11.
I20260709 13:01:01.679858  1753 raft_consensus.cc:1275] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260709 13:01:01.683485  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:01.684156  1745 raft_consensus.cc:3060] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:01:01.688407  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:01.688701  1571 leader_election.cc:304] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:01:01.552644  6160 raft_consensus.cc:3055] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:01:01.689818  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:01:01.690580  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 155043us
W20260709 13:01:01.683195  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 149070us
I20260709 13:01:01.634370  6143 leader_election.cc:290] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 8 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:01.699324  1598 raft_consensus.cc:2468] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 12.
I20260709 13:01:01.699788  6111 consensus_queue.cc:1243] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer 816fadcea8fc4893882f5f9b9f22a03d log is divergent from this leader: its last log entry 10.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260709 13:01:01.752770  6002 leader_election.cc:290] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 11 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:01.756640  6046 raft_consensus.cc:515] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:01.693358  6160 raft_consensus.cc:740] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
W20260709 13:01:01.743263  5991 consensus_meta.cc:287] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.693s	user 0.010s	sys 0.005s
I20260709 13:01:01.722754  6186 consensus_queue.cc:1048] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.029s
W20260709 13:01:01.760582  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 104997us
W20260709 13:01:01.774485  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 80300.2us
W20260709 13:01:01.765890  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 69606us
I20260709 13:01:01.777789  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:01.779821  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:01.783205  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:01.775303  1718 leader_election.cc:304] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:01.784184  1750 raft_consensus.cc:2393] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 13: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:01:01.785248  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:01.789453  1671 raft_consensus.cc:1275] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20260709 13:01:01.792302  6190 raft_consensus.cc:2804] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:01:01.792922  6190 raft_consensus.cc:493] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:01.793372  6190 raft_consensus.cc:3060] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:01:01.804698  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:01.809165  1678 raft_consensus.cc:2368] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 9. Current term is 10.
W20260709 13:01:01.802944  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 108312us
W20260709 13:01:01.814343  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 125799us
I20260709 13:01:01.815308  1571 leader_election.cc:304] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:01:01.815951  2301 leader_election.cc:304] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:01.818620  1718 leader_election.cc:304] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:01.820319  1571 leader_election.cc:304] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:01:01.821398  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:01.821642  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:01.822016  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:01.821643  2331 raft_consensus.cc:1240] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 11. Current term is 12. Ops: []
I20260709 13:01:01.822507  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 11 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:01:01.823323  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 272976us
I20260709 13:01:01.824937  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:01.825006  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:01.826400  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:01.826538  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:01.827831  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:01.828143  2330 raft_consensus.cc:1240] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 11. Current term is 12. Ops: []
I20260709 13:01:01.823319  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 12 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:01.823818  2328 raft_consensus.cc:2368] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for earlier term 11. Current term is 12.
I20260709 13:01:01.828795  1679 raft_consensus.cc:3060] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Advancing to term 12
I20260709 13:01:01.829578  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:01.830212  2337 raft_consensus.cc:3060] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:01:01.831503  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:01.832134  1680 raft_consensus.cc:3060] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:01:01.831549  1644 leader_election.cc:304] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:01:01.761919  6189 consensus_queue.cc:1048] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.052s
I20260709 13:01:01.842499  1571 leader_election.cc:304] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:01.840834  1745 raft_consensus.cc:2468] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 11.
I20260709 13:01:01.765269  5738 consensus_queue.cc:1059] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 9.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.053s
I20260709 13:01:01.855275  6160 consensus_queue.cc:260] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.8, Last appended by leader: 8, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:01:01.854648  6182 log.cc:927] Time spent T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.060s	user 0.000s	sys 0.000s
I20260709 13:01:01.843446  5919 raft_consensus.cc:2804] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:01:01.815815  6002 raft_consensus.cc:493] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:02.030164  6002 raft_consensus.cc:515] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:02.026638  6140 raft_consensus.cc:515] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:02.042549  6002 leader_election.cc:290] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:02.043396  1601 raft_consensus.cc:2393] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 11: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:01:02.043921  6140 leader_election.cc:290] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:02.046587  5919 raft_consensus.cc:697] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 12 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:01:02.047868  5919 consensus_queue.cc:237] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.10, Last appended by leader: 10, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:02.019716  6192 raft_consensus.cc:2764] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Leader pre-election decision vote started in defunct term 10: won
I20260709 13:01:01.856464  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:01.830279  5923 raft_consensus.cc:2749] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:01:02.007947  1671 raft_consensus.cc:1275] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 13 index: 10. (index mismatch)
I20260709 13:01:02.013660  1679 raft_consensus.cc:2468] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 12.
I20260709 13:01:02.008284  6188 raft_consensus.cc:2749] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
W20260709 13:01:02.016824  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 198278us
I20260709 13:01:02.057090  1718 leader_election.cc:304] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:02.004289  6183 raft_consensus.cc:2749] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:01:02.003679  6143 raft_consensus.cc:2749] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 13:01:02.017236  2329 raft_consensus.cc:2468] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 10.
I20260709 13:01:02.018457  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 11 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:02.019518  1680 raft_consensus.cc:2468] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 11.
I20260709 13:01:02.058940  2299 leader_election.cc:304] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:01:02.023243  2337 raft_consensus.cc:2468] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 13.
I20260709 13:01:02.061681  2331 raft_consensus.cc:1240] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 12. Current term is 13. Ops: []
I20260709 13:01:02.015779  6186 raft_consensus.cc:493] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:01:02.022073  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 179686us
I20260709 13:01:02.012949  1747 raft_consensus.cc:2468] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 8.
I20260709 13:01:02.028743  6018 raft_consensus.cc:515] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:02.066388  1748 raft_consensus.cc:1240] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 7. Current term is 8. Ops: [7.5-7.5]
I20260709 13:01:02.025276  6176 raft_consensus.cc:2804] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:01:02.069384  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:02.072021  1675 raft_consensus.cc:2468] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 12.
I20260709 13:01:02.060423  2299 leader_election.cc:304] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:01:02.017957  6161 raft_consensus.cc:2749] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:01:02.073896  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:02.091936  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:02.065804  1571 leader_election.cc:304] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:01:02.072248  6176 raft_consensus.cc:493] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:02.104187  1718 leader_election.cc:304] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:02.068375  6190 raft_consensus.cc:515] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:02.107527  6190 leader_election.cc:290] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 8 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:02.105341  6176 raft_consensus.cc:3060] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:01:02.107527  1747 raft_consensus.cc:1275] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 13 index: 10. (term mismatch)
I20260709 13:01:02.113102  1747 pending_rounds.cc:85] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701: Aborting all ops after (but not including) 9
I20260709 13:01:02.113386  1747 pending_rounds.cc:107] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701: Aborting uncommitted NO_OP operation due to leader change: 12.10
I20260709 13:01:02.113616  1747 raft_consensus.cc:2889] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:01:02.066586  5833 raft_consensus.cc:3060] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:01:02.067237  5663 raft_consensus.cc:2804] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:01:02.112526  1598 raft_consensus.cc:1275] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 13:01:02.115767  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:02.119189  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:02.116518  1717 leader_election.cc:304] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:02.076551  5991 raft_consensus.cc:493] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:01:02.099812  6185 raft_consensus.cc:2804] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 13:01:02.118083  5663 raft_consensus.cc:697] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 12 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
W20260709 13:01:02.125281  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 59558.5us
I20260709 13:01:02.072913  6186 raft_consensus.cc:515] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:02.128747  1571 leader_election.cc:304] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:02.124593  6190 raft_consensus.cc:2749] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 13:01:02.124559  6161 raft_consensus.cc:2749] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 13:01:02.102679  6135 consensus_queue.cc:1048] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.025s
I20260709 13:01:02.140676  1717 leader_election.cc:304] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:02.126564  5991 raft_consensus.cc:515] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:02.148710  5663 consensus_queue.cc:237] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 10.11, Last appended by leader: 11, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:02.150454  6161 raft_consensus.cc:2764] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Leader pre-election decision vote started in defunct term 12: won
I20260709 13:01:02.138031  6143 consensus_queue.cc:1048] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.014s
I20260709 13:01:02.153438  6161 consensus_queue.cc:1048] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 13:01:02.127851  5923 raft_consensus.cc:2749] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:01:02.127287  6046 leader_election.cc:290] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:02.155776  5991 leader_election.cc:290] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:02.136215  6183 raft_consensus.cc:2804] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:01:02.171307  1717 leader_election.cc:304] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 13:01:02.176843  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 78124.1us
I20260709 13:01:02.179219  1718 leader_election.cc:304] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:02.182572  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:02.184089  2329 raft_consensus.cc:1240] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 15 LEADER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 14. Current term is 15. Ops: [14.11-14.11]
I20260709 13:01:02.191815  1718 leader_election.cc:304] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
W20260709 13:01:02.196480  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 49759.1us
I20260709 13:01:02.197786  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 10 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:02.164309  6183 raft_consensus.cc:493] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:02.174098  6212 raft_consensus.cc:493] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
W20260709 13:01:02.211242  6209 log.cc:927] Time spent T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.072s	user 0.000s	sys 0.001s
I20260709 13:01:02.214533  6186 leader_election.cc:290] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:01:02.215022  6207 log.cc:927] Time spent T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.054s	user 0.000s	sys 0.000s
I20260709 13:01:02.192823  6201 raft_consensus.cc:2804] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:01:02.193086  6198 raft_consensus.cc:493] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:02.176093  6190 consensus_queue.cc:1048] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.033s
I20260709 13:01:02.205212  5738 raft_consensus.cc:2804] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 13:01:02.217757  6212 raft_consensus.cc:515] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:02.207758  6183 raft_consensus.cc:3060] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:01:02.193063  6206 consensus_queue.cc:1048] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.045s
W20260709 13:01:02.235451  6203 log.cc:927] Time spent T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.103s	user 0.000s	sys 0.001s
I20260709 13:01:02.217562  6201 raft_consensus.cc:493] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:01:02.240852  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 69680.9us
I20260709 13:01:02.242069  1717 leader_election.cc:304] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:02.238430  6183 raft_consensus.cc:515] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:02.228849  6018 leader_election.cc:290] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:02.244650  6190 raft_consensus.cc:2749] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:01:02.245129  6176 raft_consensus.cc:515] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:02.246621  6212 leader_election.cc:290] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:02.243780  1717 leader_election.cc:304] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:02.246672  6183 leader_election.cc:290] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 12 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:02.248476  6135 consensus_queue.cc:1048] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 13:01:02.249919  6186 raft_consensus.cc:2749] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260709 13:01:02.251777  1646 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:02.251937  6194 raft_consensus.cc:493] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:02.252446  6194 raft_consensus.cc:515] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:02.255677  6199 consensus_queue.cc:1059] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 10.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 13:01:02.256510  6199 consensus_queue.cc:1243] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer 83595ad0fd5940c0a2bfc7c61acd969a log is divergent from this leader: its last log entry 11.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
W20260709 13:01:02.255986  6208 log.cc:927] Time spent T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.056s	user 0.000s	sys 0.000s
I20260709 13:01:02.239707  5738 raft_consensus.cc:697] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 13 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:01:02.251214  6176 leader_election.cc:290] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 7 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:02.258955  5738 consensus_queue.cc:237] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.9, Last appended by leader: 9, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:02.260040  1717 leader_election.cc:304] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:02.260533  1678 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52778 (request call id 2136) took 1067 ms. Trace:
I20260709 13:01:02.260934  1678 rpcz_store.cc:276] 0709 13:01:01.192780 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:01.253226 (+ 60446us) service_pool.cc:224] Handling call
0709 13:01:02.260324 (+1007098us) spinlock_profiling.cc:243] Waited 510 ms on lock 0x7b54001f4d54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:01:02.260466 (+   142us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1173743872}
I20260709 13:01:02.261514  1717 leader_election.cc:304] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:02.264662  6186 raft_consensus.cc:2749] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:01:02.265107  1754 raft_consensus.cc:1275] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 12 index: 7. (term mismatch)
I20260709 13:01:02.265384  6186 raft_consensus.cc:2749] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 13:01:02.266084  1746 raft_consensus.cc:3055] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:01:02.266502  1746 raft_consensus.cc:740] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:01:02.266937  1717 leader_election.cc:304] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:02.267356  1746 consensus_queue.cc:260] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:02.268296  1717 leader_election.cc:304] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:02.268980  1746 raft_consensus.cc:3060] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:01:02.227769  6198 raft_consensus.cc:515] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:02.272408  6199 raft_consensus.cc:2749] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 13:01:02.272977  6198 leader_election.cc:290] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:02.273164  6199 raft_consensus.cc:3060] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:01:02.265508  1754 pending_rounds.cc:85] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701: Aborting all ops after (but not including) 6
I20260709 13:01:02.278183  1754 pending_rounds.cc:107] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701: Aborting uncommitted NO_OP operation due to leader change: 11.7
I20260709 13:01:02.278998  1754 raft_consensus.cc:2889] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:01:02.247881  6201 raft_consensus.cc:3060] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Advancing to term 14
W20260709 13:01:02.280828  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 78559.5us
W20260709 13:01:02.283569  6210 log.cc:927] Time spent T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.055s	user 0.000s	sys 0.000s
W20260709 13:01:02.284559  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 647868us
I20260709 13:01:02.285181  5989 raft_consensus.cc:2749] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:01:02.286480  1672 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57092 (request call id 957) took 1297 ms. Trace:
I20260709 13:01:02.286639  6196 raft_consensus.cc:493] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:02.279297  6013 consensus_queue.cc:1059] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.008s
W20260709 13:01:02.289474  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 457946us
I20260709 13:01:02.290026  6069 raft_consensus.cc:2749] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:01:02.291407  6194 leader_election.cc:290] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:02.292912  1717 leader_election.cc:304] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:02.308038  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:02.296123  1643 leader_election.cc:304] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:01:02.295774  1672 rpcz_store.cc:276] 0709 13:01:00.989543 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:00.997689 (+  8146us) service_pool.cc:224] Handling call
0709 13:01:01.016733 (+ 19044us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 13:01:01.084974 (+ 68241us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:01.085016 (+    42us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 13:01:01.093443 (+  8427us) log.cc:844] Serialized 78 byte log entry
0709 13:01:01.382757 (+289314us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:01:01.383197 (+   440us) log.cc:844] Serialized 14 byte log entry
0709 13:01:01.763073 (+379876us) log.cc:844] Serialized 14 byte log entry
0709 13:01:02.251298 (+488225us) spinlock_profiling.cc:243] Waited 2.94 ms on lock 0x7b18003f9fa0. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f4262fefc0a 00007f426c72f219 00007f426a03658f 00007f4263075571 00007f42630777c1 00007f4263077779 00007f4263077709 00007f42630776d1 00007f42630769cd
0709 13:01:02.285626 (+ 34328us) log.cc:844] Serialized 14 byte log entry
0709 13:01:02.285871 (+   245us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:02.286074 (+   203us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:02.286132 (+    58us) raft_consensus.cc:1685] finished
0709 13:01:02.286146 (+    14us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:02.286327 (+   181us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":543723,"prepare.queue_time_us":1013,"prepare.run_cpu_time_us":137,"prepare.run_wall_time_us":142,"raft.queue_time_us":632437,"raft.run_cpu_time_us":453,"raft.run_wall_time_us":18373,"spinlock_wait_cycles":6792832,"thread_start_us":642471,"threads_started":2,"wal-append.queue_time_us":412479}
I20260709 13:01:02.300312  2336 raft_consensus.cc:1275] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 7 index: 9. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20260709 13:01:02.311870  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:02.334311  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:02.335464  2335 raft_consensus.cc:2468] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 9.
I20260709 13:01:02.338474  1644 leader_election.cc:400] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 11 pre-election: Vote denied by peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625) with higher term. Message: Invalid argument: T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for earlier term 11. Current term is 12.
I20260709 13:01:02.338956  1644 leader_election.cc:403] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 11 pre-election: Cancelling election due to peer responding with higher term
W20260709 13:01:02.341367  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 66343.3us
I20260709 13:01:02.344389  1569 leader_election.cc:304] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:01:02.343470  6194 raft_consensus.cc:2749] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260709 13:01:02.347937  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 52879.9us
I20260709 13:01:02.348279  6215 raft_consensus.cc:2749] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:01:02.349608  1643 leader_election.cc:304] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:01:02.350499  6194 raft_consensus.cc:2749] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:01:02.344868  6046 raft_consensus.cc:2749] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Leader pre-election lost for term 11. Reason: Vote denied by peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625) with higher term. Message: Invalid argument: T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for earlier term 11. Current term is 12.
I20260709 13:01:02.347723  6183 consensus_queue.cc:1059] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 10.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
W20260709 13:01:02.352941  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 60118.3us
I20260709 13:01:02.353609  1717 leader_election.cc:304] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:02.352360  1569 leader_election.cc:304] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:01:02.346833  6206 raft_consensus.cc:2749] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:01:02.364204  1643 leader_election.cc:304] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:01:02.307755  6196 raft_consensus.cc:515] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:02.368106  6196 leader_election.cc:290] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:02.371078  6184 consensus_queue.cc:1048] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.012s
I20260709 13:01:02.359118  6198 raft_consensus.cc:2749] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:01:02.361229  6183 raft_consensus.cc:3060] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:01:02.360818  6140 consensus_queue.cc:1048] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.008s
I20260709 13:01:02.342545  5989 consensus_queue.cc:1059] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 11.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.019s
I20260709 13:01:02.379011  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:02.387288  5999 raft_consensus.cc:3055] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 13:01:02.387730  5999 raft_consensus.cc:740] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:01:02.388726  5999 consensus_queue.cc:260] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.7, Last appended by leader: 7, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:02.389735  5999 raft_consensus.cc:3060] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:01:02.393779  5989 consensus_queue.cc:1059] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 13:01:02.394834  5989 raft_consensus.cc:3055] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:01:02.395169  5989 raft_consensus.cc:740] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:01:02.396044  5989 consensus_queue.cc:260] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.10, Last appended by leader: 10, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:02.397298  5989 raft_consensus.cc:3060] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:01:02.400624  6111 raft_consensus.cc:3055] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 13:01:02.401270  6111 raft_consensus.cc:740] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:01:02.380683  6188 raft_consensus.cc:2749] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 13 LEADER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:01:02.403087  1643 leader_election.cc:304] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:02.404182  5894 raft_consensus.cc:2749] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 13:01:02.406401  1643 leader_election.cc:304] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:01:02.410276  1599 raft_consensus.cc:1275] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 15 index: 11. (index mismatch)
I20260709 13:01:02.413525  1643 leader_election.cc:304] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 13:01:02.416188  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 62624.5us
I20260709 13:01:02.416080  1643 leader_election.cc:304] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:01:02.380347  5833 raft_consensus.cc:2749] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:01:02.382192  6196 raft_consensus.cc:2749] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:01:02.402581  6111 consensus_queue.cc:260] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.8, Last appended by leader: 8, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:02.421347  5894 raft_consensus.cc:2804] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:01:02.431613  6206 raft_consensus.cc:493] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:02.443596  6111 raft_consensus.cc:3060] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:01:02.443961  6206 raft_consensus.cc:515] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:02.428772  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:02.446123  1600 raft_consensus.cc:2393] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 9: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:01:02.446230  1717 leader_election.cc:304] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:02.419292  6194 raft_consensus.cc:2749] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260709 13:01:02.471216  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 55210us
W20260709 13:01:02.438320  6204 log.cc:927] Time spent T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.397s	user 0.000s	sys 0.001s
I20260709 13:01:02.477517  1643 leader_election.cc:304] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
W20260709 13:01:02.427728  6195 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 11 current term: 11 is higher.
I20260709 13:01:02.479187  1643 leader_election.cc:304] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:02.480036  6196 raft_consensus.cc:2804] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:01:02.480801  6196 raft_consensus.cc:493] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:02.456724  5894 raft_consensus.cc:697] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:01:02.481194  6201 raft_consensus.cc:515] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:02.482607  5894 consensus_queue.cc:237] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.7, Last appended by leader: 7, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:02.481261  6196 raft_consensus.cc:3060] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:01:02.484066  6046 raft_consensus.cc:2749] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:01:02.484431  6201 leader_election.cc:290] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 14 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:02.488591  1717 leader_election.cc:304] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:02.458737  5833 raft_consensus.cc:2749] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:01:02.459038  6184 raft_consensus.cc:2804] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:01:02.494634  6206 leader_election.cc:290] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:02.510146  1605 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 2265) took 1349 ms. Trace:
I20260709 13:01:02.505059  5999 raft_consensus.cc:2749] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:01:02.497313  1717 leader_election.cc:304] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:02.497293  1643 leader_election.cc:304] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:01:02.516191  1605 rpcz_store.cc:276] 0709 13:01:01.160864 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:01.304275 (+143411us) service_pool.cc:224] Handling call
0709 13:01:01.366455 (+ 62180us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:01.402111 (+ 35656us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:02.509900 (+1107789us) spinlock_profiling.cc:243] Waited 61.7 ms on lock 0x7b54004222d0. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661618ad 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:01:02.510068 (+   168us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":141923584}
I20260709 13:01:02.491051  6185 raft_consensus.cc:697] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 11 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:01:02.513252  6199 raft_consensus.cc:2749] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260709 13:01:02.519941  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 156962us
I20260709 13:01:02.533838  5938 raft_consensus.cc:493] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:02.534464  5938 raft_consensus.cc:515] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:02.540086  6069 raft_consensus.cc:3055] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 11 LEADER]: Stepping down as leader of term 11
I20260709 13:01:02.540730  6069 raft_consensus.cc:740] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:01:02.521726  6206 raft_consensus.cc:3060] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:01:02.523072  6016 raft_consensus.cc:2804] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Leader election won for term 11
I20260709 13:01:02.546779  6016 raft_consensus.cc:697] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:01:02.547915  6016 consensus_queue.cc:237] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.7, Last appended by leader: 7, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:02.522866  6184 raft_consensus.cc:697] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:01:02.550323  6069 consensus_queue.cc:260] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:02.551462  6184 consensus_queue.cc:237] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.7, Last appended by leader: 7, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:02.542693  6046 raft_consensus.cc:493] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:02.554687  6046 raft_consensus.cc:515] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:02.557075  6046 leader_election.cc:290] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:02.562580  1717 leader_election.cc:304] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:02.564035  1717 leader_election.cc:304] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:02.567592  5999 raft_consensus.cc:2749] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:01:02.568607  6199 raft_consensus.cc:2749] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:01:02.546603  6163 raft_consensus.cc:3060] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:01:02.552430  6069 raft_consensus.cc:3060] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Advancing to term 12
W20260709 13:01:02.569015  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 83921.9us
I20260709 13:01:02.572853  1569 leader_election.cc:304] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:01:02.573992  6232 consensus_queue.cc:1059] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 11.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 13:01:02.568935  1717 leader_election.cc:304] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:02.584677  1569 leader_election.cc:304] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:02.587543  5999 raft_consensus.cc:2804] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:01:02.586036  5989 consensus_queue.cc:1059] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.004s
I20260709 13:01:02.589994  5999 raft_consensus.cc:493] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:02.590615  5989 raft_consensus.cc:3055] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:01:02.590929  6147 raft_consensus.cc:2804] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Leader election won for term 13
I20260709 13:01:02.591087  5989 raft_consensus.cc:740] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:01:02.591629  6147 raft_consensus.cc:697] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:01:02.592185  5989 consensus_queue.cc:260] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } }
I20260709 13:01:02.592634  6147 consensus_queue.cc:237] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.10, Last appended by leader: 10, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:02.593592  5989 raft_consensus.cc:3060] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:01:02.594559  6194 consensus_queue.cc:1048] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 13:01:02.595274  6232 raft_consensus.cc:3055] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 LEADER]: Stepping down as leader of term 11
I20260709 13:01:02.595661  6232 raft_consensus.cc:740] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:01:02.596388  6194 raft_consensus.cc:493] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:01:02.596962  6194 raft_consensus.cc:515] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:02.599538  6194 leader_election.cc:290] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:02.590615  1717 leader_election.cc:304] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:02.609719  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:02.610730  1598 raft_consensus.cc:3060] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:01:02.613263  1601 raft_consensus.cc:1240] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 10. Current term is 11. Ops: [10.8-10.8]
I20260709 13:01:02.613310  1600 raft_consensus.cc:1240] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 LEADER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 8. Current term is 9. Ops: [8.5-8.5]
I20260709 13:01:02.614369  1597 raft_consensus.cc:1240] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 11. Current term is 12. Ops: [11.8-11.8]
I20260709 13:01:02.619280  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:02.620137  1601 raft_consensus.cc:2468] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 15.
I20260709 13:01:02.622334  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:02.623958  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:01:02.628094  1646 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:02.634029  6200 thread.cc:641] apply [worker] (thread pool) Time spent creating pthread: real 0.584s	user 0.011s	sys 0.016s
W20260709 13:01:02.634528  6200 thread.cc:608] apply [worker] (thread pool) Time spent starting thread: real 0.585s	user 0.012s	sys 0.016s
I20260709 13:01:02.647157  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 14 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
W20260709 13:01:02.648900  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 200071us
I20260709 13:01:02.650215  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:02.653600  1643 leader_election.cc:400] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 12 election: Vote denied by peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445) with higher term. Message: Invalid argument: T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for earlier term 12. Current term is 13.
I20260709 13:01:02.656632  1598 raft_consensus.cc:2468] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 8.
I20260709 13:01:02.587939  6143 raft_consensus.cc:2804] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader election won for term 10
I20260709 13:01:02.651211  2328 raft_consensus.cc:2468] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 9.
I20260709 13:01:02.600627  5999 raft_consensus.cc:3060] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:01:02.606725  6192 raft_consensus.cc:2749] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:01:02.661461  1643 leader_election.cc:304] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:02.639066  6183 raft_consensus.cc:2749] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:01:02.668048  1643 leader_election.cc:304] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:02.674890  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
W20260709 13:01:02.675228  6231 log.cc:927] Time spent T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.068s	user 0.000s	sys 0.000s
I20260709 13:01:02.675596  1601 raft_consensus.cc:3060] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:01:02.676707  6184 raft_consensus.cc:2749] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:01:02.677498  6184 raft_consensus.cc:2749] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
W20260709 13:01:02.678843  1717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:02.679749  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:02.680361  1598 raft_consensus.cc:3060] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:01:02.606585  6046 consensus_queue.cc:1048] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.058s
I20260709 13:01:02.683137  6206 raft_consensus.cc:2749] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:01:02.683192  1571 leader_election.cc:304] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:02.686781  1718 leader_election.cc:400] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 9 election: Vote denied by peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261) with higher term. Message: Invalid argument: T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 9. Current term is 10.
I20260709 13:01:02.687158  1718 leader_election.cc:403] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 9 election: Cancelling election due to peer responding with higher term
W20260709 13:01:02.688454  1643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:02.688648  1571 leader_election.cc:304] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
W20260709 13:01:02.689215  1643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:02.692010  1598 raft_consensus.cc:2468] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 10.
I20260709 13:01:02.654549  6185 consensus_queue.cc:237] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:02.683475  6140 raft_consensus.cc:493] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:02.701172  6140 raft_consensus.cc:515] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:02.704586  6140 leader_election.cc:290] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:02.706809  6082 raft_consensus.cc:2804] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Leader election won for term 8
I20260709 13:01:02.707443  6082 raft_consensus.cc:697] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:01:02.708032  6080 raft_consensus.cc:493] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:02.711905  6080 raft_consensus.cc:515] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:02.714569  6160 raft_consensus.cc:3060] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:01:02.660146  6199 raft_consensus.cc:493] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:02.717562  5228 consensus_queue.cc:1048] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.000s
W20260709 13:01:02.717922  1719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:02.719658  2299 leader_election.cc:304] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:01:02.726639  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:02.727329  1746 raft_consensus.cc:3060] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:01:02.724591  1751 raft_consensus.cc:1240] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 11. Current term is 12. Ops: [10.9-11.10]
I20260709 13:01:02.734412  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:02.735167  1604 raft_consensus.cc:2468] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 9.
I20260709 13:01:02.735551  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:02.735812  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:02.769253  1676 raft_consensus.cc:1275] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
W20260709 13:01:02.769665  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 52279.9us
W20260709 13:01:02.768299  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 48730.1us
I20260709 13:01:02.770584  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:02.749891  1680 raft_consensus.cc:1240] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 LEADER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 11. Current term is 12. Ops: [11.7-11.7]
I20260709 13:01:02.760496  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:02.768465  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:02.767460  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:02.775110  1752 raft_consensus.cc:2468] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 8.
I20260709 13:01:02.737828  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:02.738803  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:02.825404  1746 raft_consensus.cc:2468] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 10.
I20260709 13:01:02.774119  1750 raft_consensus.cc:2468] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 10.
I20260709 13:01:02.741691  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:02.835439  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:02.835793  1600 raft_consensus.cc:1240] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 LEADER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 12. Current term is 13. Ops: []
I20260709 13:01:02.685801  6143 raft_consensus.cc:697] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
W20260709 13:01:02.855026  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 80438.6us
I20260709 13:01:02.837165  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:02.857332  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:02.858057  1672 raft_consensus.cc:3060] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Advancing to term 11
W20260709 13:01:02.837283  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 66628.6us
I20260709 13:01:02.854910  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:02.858896  1680 raft_consensus.cc:2468] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 9.
I20260709 13:01:02.860276  1599 raft_consensus.cc:1240] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 6. Current term is 7. Ops: [6.5-6.5]
I20260709 13:01:02.863762  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:02.865103  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:02.865502  1675 raft_consensus.cc:1240] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 8. Current term is 9. Ops: [8.5-8.5]
I20260709 13:01:02.711253  6046 raft_consensus.cc:493] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:02.866003  1752 raft_consensus.cc:2410] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for term 10 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20260709 13:01:02.866415  1680 raft_consensus.cc:1275] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260709 13:01:02.868072  1643 leader_election.cc:304] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:01:02.869310  1746 raft_consensus.cc:1275] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 5. (term mismatch)
I20260709 13:01:02.869710  1746 pending_rounds.cc:85] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701: Aborting all ops after (but not including) 4
I20260709 13:01:02.869946  1746 pending_rounds.cc:107] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701: Aborting uncommitted NO_OP operation due to leader change: 6.5
I20260709 13:01:02.865999  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 10 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:02.870380  1746 raft_consensus.cc:2889] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:01:02.854648  6145 raft_consensus.cc:493] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:02.873592  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:02.876274  1753 raft_consensus.cc:1275] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
W20260709 13:01:02.876585  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 106144us
I20260709 13:01:02.736867  5833 raft_consensus.cc:3060] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:01:02.867718  1718 leader_election.cc:304] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:02.882084  1718 leader_election.cc:304] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
W20260709 13:01:02.873215  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 135828us
I20260709 13:01:02.732445  5777 raft_consensus.cc:2749] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 13:01:02.836400  5894 raft_consensus.cc:493] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:02.885039  1717 leader_election.cc:304] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 13:01:02.885612  1680 raft_consensus.cc:1240] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 10. Current term is 11. Ops: [10.10-10.10]
I20260709 13:01:02.892154  1717 leader_election.cc:304] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:02.886384  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 14 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:02.726001  6080 leader_election.cc:290] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:02.894469  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:02.904654  1674 raft_consensus.cc:3060] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:01:02.856922  5991 raft_consensus.cc:2804] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:01:02.911352  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:02.732877  6029 consensus_queue.cc:1048] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.164s
I20260709 13:01:02.883602  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:02.912107  1746 raft_consensus.cc:2468] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 10.
I20260709 13:01:02.913983  1752 raft_consensus.cc:1275] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20260709 13:01:02.923362  1673 raft_consensus.cc:1275] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20260709 13:01:02.737123  6199 raft_consensus.cc:515] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:02.890260  6192 raft_consensus.cc:3060] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:01:02.908344  6206 raft_consensus.cc:2804] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:01:02.879655  6189 consensus_queue.cc:1059] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.043s
I20260709 13:01:02.872682  6145 raft_consensus.cc:515] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:02.908342  6195 raft_consensus.cc:2804] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:01:02.909897  5991 raft_consensus.cc:493] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:02.944962  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:02.918870  6201 raft_consensus.cc:2804] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 13:01:02.955456  1602 raft_consensus.cc:1275] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 12 index: 9. Preceding OpId from leader: term: 13 index: 10. (index mismatch)
I20260709 13:01:02.880683  6196 raft_consensus.cc:515] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:02.880437  6046 raft_consensus.cc:515] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:02.708573  6082 consensus_queue.cc:237] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } }
I20260709 13:01:02.972667  6189 consensus_queue.cc:1243] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer 83595ad0fd5940c0a2bfc7c61acd969a log is divergent from this leader: its last log entry 9.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260709 13:01:02.945223  6206 raft_consensus.cc:493] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:02.979311  1677 raft_consensus.cc:2410] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for term 10 because replica has last-logged OpId of term: 9 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 8.
I20260709 13:01:02.908646  5894 raft_consensus.cc:515] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:02.985492  1752 raft_consensus.cc:3055] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:01:03.016166  1752 raft_consensus.cc:740] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:01:02.985770  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:02.835202  6232 consensus_queue.cc:260] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 9, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:03.001067  1671 raft_consensus.cc:1240] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 10. Current term is 11. Ops: [10.5-10.5]
I20260709 13:01:03.003165  6215 raft_consensus.cc:3055] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:01:03.008100  6218 raft_consensus.cc:3055] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:01:03.008586  6197 raft_consensus.cc:493] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:01:03.009233  6145 leader_election.cc:290] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:03.009928  6189 consensus_queue.cc:1048] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 13:01:03.010123  5938 leader_election.cc:290] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:03.014169  6196 leader_election.cc:290] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 11 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:03.014571  6046 leader_election.cc:290] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:03.015218  1679 raft_consensus.cc:3060] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:01:02.956750  6195 raft_consensus.cc:493] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:03.017418  1673 raft_consensus.cc:2393] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 11: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:01:02.968022  5991 raft_consensus.cc:3060] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:01:03.020516  1672 raft_consensus.cc:2468] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 11.
I20260709 13:01:03.022900  1601 raft_consensus.cc:2468] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 10.
W20260709 13:01:03.017488  6238 log.cc:927] Time spent T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.370s	user 0.003s	sys 0.011s
I20260709 13:01:03.037380  1680 raft_consensus.cc:2410] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for term 11 because replica has last-logged OpId of term: 9 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 5.
I20260709 13:01:03.037515  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:03.016602  6206 raft_consensus.cc:3060] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:01:03.025528  1752 consensus_queue.cc:260] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:03.023898  6144 raft_consensus.cc:493] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:03.049173  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:03.042711  1674 raft_consensus.cc:2468] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 14.
I20260709 13:01:03.019021  6215 raft_consensus.cc:740] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:01:03.021642  6218 raft_consensus.cc:740] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:01:02.956220  6143 consensus_queue.cc:237] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:03.076364  1675 raft_consensus.cc:1240] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 13. Current term is 14. Ops: [13.10-13.10]
I20260709 13:01:03.063560  1679 raft_consensus.cc:2468] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 8.
I20260709 13:01:03.019893  5833 raft_consensus.cc:2749] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Leader election lost for term 9. Reason: Vote denied by peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261) with higher term. Message: Invalid argument: T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 9. Current term is 10.
I20260709 13:01:03.075763  1673 raft_consensus.cc:2468] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 11.
I20260709 13:01:03.048010  6195 raft_consensus.cc:3060] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:01:03.112020  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:03.115948  1746 raft_consensus.cc:2468] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 12.
I20260709 13:01:03.050364  6080 raft_consensus.cc:2749] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260709 13:01:03.129531  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 261540us
I20260709 13:01:03.053925  6140 raft_consensus.cc:493] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:03.127501  1752 raft_consensus.cc:3060] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:01:03.135752  1746 raft_consensus.cc:1275] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
I20260709 13:01:03.133575  1643 leader_election.cc:304] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
W20260709 13:01:03.131238  6069 consensus_meta.cc:287] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.545s	user 0.013s	sys 0.000s
I20260709 13:01:03.136575  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:03.128405  1571 leader_election.cc:304] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:03.115541  6232 raft_consensus.cc:3060] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Advancing to term 12
W20260709 13:01:03.111441  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 143606us
I20260709 13:01:03.109691  6199 leader_election.cc:290] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:03.148814  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 11 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:03.035226  6197 raft_consensus.cc:515] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:03.151681  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:03.171947  2299 leader_election.cc:304] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:03.177228  5228 raft_consensus.cc:3060] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:01:03.178162  2299 leader_election.cc:304] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:03.190583  1745 raft_consensus.cc:2468] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 10.
W20260709 13:01:03.144688  6111 consensus_meta.cc:287] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.701s	user 0.013s	sys 0.002s
I20260709 13:01:03.147991  5833 raft_consensus.cc:493] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:03.106899  6144 raft_consensus.cc:515] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:03.137780  6183 consensus_queue.cc:1048] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.064s
I20260709 13:01:03.206046  6201 raft_consensus.cc:697] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 8 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:01:03.175618  6082 raft_consensus.cc:2804] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:01:03.221159  1752 raft_consensus.cc:1275] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 5. (term mismatch)
I20260709 13:01:03.225481  1752 pending_rounds.cc:85] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701: Aborting all ops after (but not including) 4
I20260709 13:01:03.184078  6199 raft_consensus.cc:493] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:03.198770  5663 raft_consensus.cc:2804] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Leader pre-election won for term 12
W20260709 13:01:03.229465  6239 log.cc:927] Time spent T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.186s	user 0.001s	sys 0.000s
I20260709 13:01:03.230582  6144 leader_election.cc:290] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:03.144517  6140 raft_consensus.cc:515] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:03.208485  5833 raft_consensus.cc:515] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:01:03.235642  5999 consensus_meta.cc:287] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.549s	user 0.006s	sys 0.006s
I20260709 13:01:03.176354  5938 consensus_queue.cc:1048] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.047s
I20260709 13:01:03.239104  5663 raft_consensus.cc:493] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:01:03.247731  6197 leader_election.cc:290] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:03.220722  6082 raft_consensus.cc:493] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:03.207829  6215 consensus_queue.cc:260] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:03.258981  6201 consensus_queue.cc:237] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:03.217895  6046 raft_consensus.cc:2804] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Leader election won for term 10
I20260709 13:01:03.225752  1752 pending_rounds.cc:107] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701: Aborting uncommitted NO_OP operation due to leader change: 8.5
W20260709 13:01:03.268576  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 136580us
I20260709 13:01:03.287160  1752 raft_consensus.cc:2889] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:01:03.287591  1643 leader_election.cc:304] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
W20260709 13:01:03.283236  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 151440us
I20260709 13:01:03.288875  1643 leader_election.cc:304] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:03.289258  1646 leader_election.cc:304] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:01:03.277683  5228 raft_consensus.cc:2749] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:01:03.309863  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:03.310680  2331 raft_consensus.cc:2376] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 10. Re-sending same reply.
I20260709 13:01:03.310845  2337 raft_consensus.cc:1240] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 8. Current term is 9. Ops: []
I20260709 13:01:03.313369  2335 raft_consensus.cc:1275] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 13 index: 11. (index mismatch)
I20260709 13:01:03.293308  1643 leader_election.cc:304] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:01:03.242170  6199 raft_consensus.cc:515] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:03.328051  1646 leader_election.cc:304] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:03.328387  2332 raft_consensus.cc:1240] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 10. Current term is 11. Ops: []
I20260709 13:01:03.269564  5999 raft_consensus.cc:515] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:03.334225  1676 raft_consensus.cc:1275] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 13:01:03.330305  2337 raft_consensus.cc:1275] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 12 index: 9. Preceding OpId from leader: term: 13 index: 10. (index mismatch)
I20260709 13:01:03.330845  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:03.342751  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:03.283260  5663 raft_consensus.cc:3060] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:01:03.344506  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:03.349243  2336 raft_consensus.cc:2468] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 9.
W20260709 13:01:03.363675  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 212044us
I20260709 13:01:03.351167  6135 consensus_queue.cc:1048] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 13:01:03.331624  5894 leader_election.cc:290] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:03.333387  6199 leader_election.cc:290] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:03.331526  6184 raft_consensus.cc:2749] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:01:03.342136  6145 consensus_queue.cc:1059] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.012s
I20260709 13:01:03.334553  6046 raft_consensus.cc:697] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:01:03.329697  6069 raft_consensus.cc:2804] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader pre-election won for term 10
W20260709 13:01:03.332305  6163 consensus_meta.cc:287] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.749s	user 0.011s	sys 0.003s
I20260709 13:01:03.355973  1754 raft_consensus.cc:1240] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 6. Current term is 7. Ops: []
I20260709 13:01:03.357156  6244 raft_consensus.cc:2804] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:01:03.357944  5999 leader_election.cc:290] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 7 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:03.362244  6171 raft_consensus.cc:2804] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Leader election won for term 10
I20260709 13:01:03.359274  6197 consensus_queue.cc:1048] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20260709 13:01:03.360575  1643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:03.290972  6082 raft_consensus.cc:3060] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:01:03.326272  6251 raft_consensus.cc:2749] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:01:03.368948  5228 raft_consensus.cc:493] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:03.464444  5228 raft_consensus.cc:515] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:03.371405  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:03.466387  1601 raft_consensus.cc:2393] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 8: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:01:03.372948  1597 raft_consensus.cc:1275] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 11 index: 8. (index mismatch)
I20260709 13:01:03.373869  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:01:03.363039  6246 log.cc:927] Time spent T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.076s	user 0.001s	sys 0.000s
W20260709 13:01:03.354343  6233 log.cc:927] Time spent T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.721s	user 0.009s	sys 0.000s
I20260709 13:01:03.377300  1604 raft_consensus.cc:1275] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 12 index: 8. (index mismatch)
I20260709 13:01:03.333756  5833 leader_election.cc:290] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:03.404965  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 11 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:03.358819  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:03.437328  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:03.438623  1749 raft_consensus.cc:2468] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 10.
I20260709 13:01:03.440688  1750 raft_consensus.cc:1275] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
W20260709 13:01:03.441239  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 59757.2us
I20260709 13:01:03.454092  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:03.472092  1748 raft_consensus.cc:2468] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 10.
I20260709 13:01:03.455826  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:03.377847  6069 raft_consensus.cc:493] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:03.407840  6244 raft_consensus.cc:493] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:03.372049  6241 consensus_queue.cc:1048] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.091s
I20260709 13:01:03.482267  1569 leader_election.cc:304] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:01:03.490489  1569 leader_election.cc:304] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
W20260709 13:01:03.492224  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:03.477164  2301 leader_election.cc:304] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
W20260709 13:01:03.488559  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 77271.1us
I20260709 13:01:03.484087  5228 leader_election.cc:290] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:03.498667  5228 raft_consensus.cc:2749] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260709 13:01:03.504841  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 332813us
I20260709 13:01:03.484967  6069 raft_consensus.cc:3060] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:01:03.490816  6244 raft_consensus.cc:3060] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Advancing to term 11
W20260709 13:01:03.495060  5989 consensus_meta.cc:287] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.901s	user 0.014s	sys 0.005s
I20260709 13:01:03.506754  1644 leader_election.cc:304] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
W20260709 13:01:03.511335  6192 consensus_meta.cc:287] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.554s	user 0.014s	sys 0.001s
I20260709 13:01:03.511879  6192 raft_consensus.cc:2749] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:01:03.512869  6140 leader_election.cc:290] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:03.493378  6262 raft_consensus.cc:3055] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:01:03.535483  1674 raft_consensus.cc:1275] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
W20260709 13:01:03.506767  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 151493us
W20260709 13:01:03.550174  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 227622us
I20260709 13:01:03.555199  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 13 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:03.556434  1672 raft_consensus.cc:2410] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for term 13 because replica has last-logged OpId of term: 12 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 6.
I20260709 13:01:03.502979  6171 raft_consensus.cc:697] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 10 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
W20260709 13:01:03.534420  5923 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.557s	user 0.014s	sys 0.018s
I20260709 13:01:03.511615  6251 consensus_queue.cc:1059] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 13:01:03.541254  6262 raft_consensus.cc:740] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:01:03.570907  6262 consensus_queue.cc:260] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:03.572338  6262 raft_consensus.cc:3060] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:01:03.574826  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:03.575953  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:03.576709  6195 raft_consensus.cc:515] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:03.580279  1753 raft_consensus.cc:1240] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 15. Current term is 16. Ops: []
I20260709 13:01:03.583607  1718 leader_election.cc:304] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:03.585181  1718 leader_election.cc:304] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1
W20260709 13:01:03.590013  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 50663.4us
I20260709 13:01:03.591363  1646 leader_election.cc:304] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:01:03.546624  6194 raft_consensus.cc:2804] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:01:03.559988  5938 raft_consensus.cc:2749] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 13:01:03.561563  6135 raft_consensus.cc:493] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:03.603564  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 13 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:03.604401  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:03.604961  1605 raft_consensus.cc:1275] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 10 index: 7. Preceding OpId from leader: term: 12 index: 8. (index mismatch)
W20260709 13:01:03.605579  1571 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:03.629477  1597 raft_consensus.cc:2393] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 7: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:01:03.503690  6046 consensus_queue.cc:237] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 8.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:03.631318  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:03.635144  1600 raft_consensus.cc:2393] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 8: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:01:03.584842  5989 raft_consensus.cc:493] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:03.596804  6192 raft_consensus.cc:2749] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:01:03.599185  6194 raft_consensus.cc:493] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:01:03.639268  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 148905us
W20260709 13:01:03.637846  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 52777.8us
I20260709 13:01:03.565016  6268 consensus_queue.cc:1048] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.072s
I20260709 13:01:03.624501  6199 raft_consensus.cc:2804] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:01:03.561818  6140 raft_consensus.cc:2749] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
W20260709 13:01:03.693763  1571 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:03.698716  5894 raft_consensus.cc:2749] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:01:03.641723  6194 raft_consensus.cc:3060] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:01:03.585721  6171 consensus_queue.cc:237] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:01:03.718927  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 69338.1us
I20260709 13:01:03.645499  6111 raft_consensus.cc:493] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:03.720587  1569 leader_election.cc:304] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:01:03.651437  6145 consensus_queue.cc:1048] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.088s
W20260709 13:01:03.742718  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 159041us
I20260709 13:01:03.637071  6259 consensus_queue.cc:1048] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.108s
I20260709 13:01:03.642939  5989 raft_consensus.cc:515] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:03.734980  6257 log.cc:927] Time spent T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.247s	user 0.000s	sys 0.001s
I20260709 13:01:03.716396  6195 leader_election.cc:290] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:03.643049  6135 raft_consensus.cc:515] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:03.746388  6254 log.cc:927] Time spent T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.207s	user 0.000s	sys 0.001s
W20260709 13:01:03.791451  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 43839.2us
I20260709 13:01:03.653864  6241 consensus_queue.cc:1048] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.129s
I20260709 13:01:03.787497  5926 raft_consensus.cc:2749] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:01:03.796191  6163 raft_consensus.cc:493] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:03.790653  6199 raft_consensus.cc:697] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 14 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:01:03.798372  1643 leader_election.cc:304] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
W20260709 13:01:03.783936  6231 log.cc:927] Time spent T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.074s	user 0.000s	sys 0.001s
I20260709 13:01:03.743680  5938 consensus_queue.cc:1048] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.059s
I20260709 13:01:03.744277  5777 consensus_queue.cc:1059] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 8.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.061s
W20260709 13:01:03.805526  6160 consensus_meta.cc:287] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 1.073s	user 0.014s	sys 0.002s
I20260709 13:01:03.734637  6270 consensus_queue.cc:1048] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.117s
I20260709 13:01:03.747375  6111 raft_consensus.cc:515] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:03.823678  6269 log.cc:927] Time spent T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.196s	user 0.000s	sys 0.001s
I20260709 13:01:03.824007  6111 leader_election.cc:290] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:03.826709  1598 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 2292) took 1056 ms. Trace:
W20260709 13:01:03.832147  1643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:03.827145  1598 rpcz_store.cc:276] 0709 13:01:02.770749 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:02.770953 (+   204us) service_pool.cc:224] Handling call
0709 13:01:02.771300 (+   347us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:02.943081 (+171781us) raft_consensus.cc:1534] Early marking committed up to index 3
0709 13:01:02.943120 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:01:02.960287 (+ 17167us) log.cc:844] Serialized 26 byte log entry
0709 13:01:03.196010 (+235723us) raft_consensus.cc:1645] Marking committed up to 3
0709 13:01:03.196054 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:03.196143 (+    89us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:03.826446 (+630303us) raft_consensus.cc:1685] finished
0709 13:01:03.826476 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:03.826634 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8995,"raft.queue_time_us":90640,"raft.run_cpu_time_us":151,"raft.run_wall_time_us":12194,"thread_start_us":347729,"threads_started":2,"wal-append.queue_time_us":358365}
W20260709 13:01:03.842643  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 197908us
I20260709 13:01:03.843827  1718 leader_election.cc:304] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:03.845567  1601 raft_consensus.cc:1240] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 7. Current term is 8. Ops: []
I20260709 13:01:03.847079  1596 raft_consensus.cc:1275] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
I20260709 13:01:03.852079  2328 raft_consensus.cc:1240] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 11 LEADER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 10. Current term is 11. Ops: []
I20260709 13:01:03.798848  6184 consensus_queue.cc:1048] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.020s
W20260709 13:01:03.856456  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 265131us
I20260709 13:01:03.855887  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:03.810847  6163 raft_consensus.cc:515] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:03.856444  5991 consensus_meta.cc:287] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.782s	user 0.013s	sys 0.000s
I20260709 13:01:03.848791  6111 raft_consensus.cc:493] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:01:03.908263  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 431176us
I20260709 13:01:03.829344  5894 raft_consensus.cc:2749] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 13:01:03.899364  6163 leader_election.cc:290] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:01:03.900451  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 56752.4us
I20260709 13:01:03.917888  1718 leader_election.cc:304] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:03.904728  6160 raft_consensus.cc:2749] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:01:03.935199  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:03.937197  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:03.937743  1680 raft_consensus.cc:2468] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 13.
I20260709 13:01:03.935199  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:01:03.914199  6206 consensus_meta.cc:287] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.807s	user 0.016s	sys 0.000s
W20260709 13:01:03.910740  6201 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.549s	user 0.003s	sys 0.040s
I20260709 13:01:03.939555  1718 leader_election.cc:304] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:03.907756  6213 raft_consensus.cc:493] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:03.939677  6163 raft_consensus.cc:3060] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:01:03.942852  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:03.942999  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:01:03.944115  6273 log.cc:927] Time spent T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.151s	user 0.001s	sys 0.000s
W20260709 13:01:03.944255  6284 log.cc:927] Time spent T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.154s	user 0.001s	sys 0.000s
I20260709 13:01:03.975049  6244 raft_consensus.cc:515] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:03.916754  6185 consensus_queue.cc:1048] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.028s
I20260709 13:01:03.947227  6135 leader_election.cc:290] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:03.909379  5991 raft_consensus.cc:515] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:03.909651  6111 raft_consensus.cc:515] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:03.943022  6293 raft_consensus.cc:493] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:03.949353  6069 raft_consensus.cc:515] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:03.873488  6280 consensus_queue.cc:1048] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.094s
W20260709 13:01:03.951884  5663 consensus_meta.cc:287] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.581s	user 0.013s	sys 0.000s
I20260709 13:01:03.953236  6218 consensus_queue.cc:260] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:03.902877  6199 consensus_queue.cc:237] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 10, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:03.968919  1718 leader_election.cc:304] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:03.972424  6192 raft_consensus.cc:2749] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:01:03.930648  6285 raft_consensus.cc:493] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:03.970114  6195 raft_consensus.cc:493] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:01:03.970275  6259 raft_consensus.cc:493] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:01:03.970295  6201 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.608s	user 0.004s	sys 0.041s
I20260709 13:01:03.993255  5663 raft_consensus.cc:515] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:03.967319  6206 raft_consensus.cc:515] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:03.996917  5991 leader_election.cc:290] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 13 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:03.997064  6111 leader_election.cc:290] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:04.002619  1677 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52778 (request call id 2140) took 1258 ms. Trace:
I20260709 13:01:04.018828  1677 rpcz_store.cc:276] 0709 13:01:02.744687 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:02.745427 (+   740us) service_pool.cc:224] Handling call
0709 13:01:04.002381 (+1256954us) spinlock_profiling.cc:243] Waited 108 ms on lock 0x7b54000fe454. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:01:04.002553 (+   172us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":248526592}
I20260709 13:01:03.976496  6082 raft_consensus.cc:515] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:04.002302  2301 leader_election.cc:304] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:03.995131  6218 raft_consensus.cc:3060] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:01:03.972853  6213 raft_consensus.cc:515] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:04.017236  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:03.994274  5989 leader_election.cc:290] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:04.005196  5994 consensus_queue.cc:1048] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 6, Time since last communication: 0.009s
I20260709 13:01:04.022990  5228 raft_consensus.cc:2804] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Leader pre-election won for term 11
W20260709 13:01:04.047075  6292 log.cc:927] Time spent T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.074s	user 0.001s	sys 0.000s
I20260709 13:01:04.047153  5228 raft_consensus.cc:493] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:04.047722  5228 raft_consensus.cc:3060] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:01:03.999367  6293 raft_consensus.cc:515] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:04.050172  6293 leader_election.cc:290] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:04.057809  6293 raft_consensus.cc:493] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:04.059141  6293 raft_consensus.cc:515] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:04.014667  6259 raft_consensus.cc:515] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:04.008718  6195 raft_consensus.cc:515] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:03.994279  6069 leader_election.cc:290] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:04.024718  5663 leader_election.cc:290] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 12 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:04.025229  2329 raft_consensus.cc:1240] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 11. Current term is 12. Ops: []
I20260709 13:01:04.077456  6293 leader_election.cc:290] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:04.006817  6285 raft_consensus.cc:515] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:01:04.080044  1749 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55840 (request call id 2501) took 2449 ms (client timeout 2912 ms). Trace:
W20260709 13:01:04.080591  1749 rpcz_store.cc:269] 0709 13:01:01.630972 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:01.667049 (+ 36077us) service_pool.cc:224] Handling call
0709 13:01:04.079766 (+2412717us) spinlock_profiling.cc:243] Waited 1.77 s on lock 0x7b54000bd954. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:01:04.079935 (+   169us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4068854528}
I20260709 13:01:04.081787  6285 leader_election.cc:290] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:04.083927  6206 leader_election.cc:290] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 10 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:04.084353  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:04.085043  1753 raft_consensus.cc:3060] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:01:04.034690  1718 leader_election.cc:304] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:04.035342  2335 raft_consensus.cc:1275] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 13 index: 11. (index mismatch)
I20260709 13:01:04.038108  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:04.038318  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:04.089620  2337 raft_consensus.cc:3060] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:01:04.090310  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:04.091061  1748 raft_consensus.cc:2468] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 12.
I20260709 13:01:04.072129  6189 raft_consensus.cc:493] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:04.092190  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:04.092810  5989 raft_consensus.cc:493] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:04.093215  1602 raft_consensus.cc:2393] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 11: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:01:04.032959  2336 raft_consensus.cc:2410] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for term 13 because replica has last-logged OpId of term: 12 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 10.
I20260709 13:01:04.093515  5989 raft_consensus.cc:515] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:04.005903  5999 raft_consensus.cc:2804] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:01:04.095963  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:04.096719  1677 raft_consensus.cc:3060] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:01:04.097414  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:04.098323  1678 raft_consensus.cc:1275] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 10 index: 11. Preceding OpId from leader: term: 12 index: 12. (index mismatch)
I20260709 13:01:04.098749  6206 raft_consensus.cc:493] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:04.098882  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:04.099239  1676 raft_consensus.cc:1240] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 10. Current term is 11. Ops: []
I20260709 13:01:04.099538  1672 raft_consensus.cc:3060] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:01:04.090502  6259 leader_election.cc:290] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:04.099376  6206 raft_consensus.cc:515] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:04.092975  6069 raft_consensus.cc:493] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:04.102380  6206 leader_election.cc:290] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:04.082127  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:04.102778  6069 raft_consensus.cc:515] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } }
I20260709 13:01:04.103426  1749 raft_consensus.cc:2468] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 10.
I20260709 13:01:04.096076  5999 raft_consensus.cc:493] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:04.068742  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:04.091912  6189 raft_consensus.cc:515] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:04.054176  6215 raft_consensus.cc:3060] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:01:04.110306  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:04.108464  1748 raft_consensus.cc:2468] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 12.
I20260709 13:01:04.116724  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 13 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:01:04.118238  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 83667.2us
I20260709 13:01:04.021787  6046 raft_consensus.cc:493] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:04.125638  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:04.127388  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:04.120595  5999 raft_consensus.cc:3060] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:01:04.130158  6144 consensus_queue.cc:1048] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 13:01:04.032421  6213 leader_election.cc:290] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:04.075811  6218 consensus_queue.cc:1048] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.056s
I20260709 13:01:04.131850  6244 leader_election.cc:290] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 11 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:01:04.165141  6263 log.cc:927] Time spent T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.604s	user 0.003s	sys 0.000s
I20260709 13:01:04.197113  2301 leader_election.cc:304] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:04.131892  5663 consensus_queue.cc:1059] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 10.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 13:01:04.198134  6195 leader_election.cc:290] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:04.132881  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:04.135424  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:04.186578  1646 leader_election.cc:304] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 13:01:04.194841  6188 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 9 is higher.
I20260709 13:01:04.199481  1745 raft_consensus.cc:1275] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 12 index: 8. (index mismatch)
I20260709 13:01:04.200096  5663 consensus_queue.cc:1059] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 13:01:04.200820  5663 consensus_queue.cc:1243] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer af520b67945d443b901bbeca37cc5701 log is divergent from this leader: its last log entry 14.11 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 10
I20260709 13:01:04.201388  5663 raft_consensus.cc:2804] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:01:04.201720  5663 raft_consensus.cc:493] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
W20260709 13:01:04.136125  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 352578us
I20260709 13:01:04.201798  1646 leader_election.cc:304] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:04.202621  6162 raft_consensus.cc:3055] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 13:01:04.202855  6259 raft_consensus.cc:2804] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:01:04.203045  6162 raft_consensus.cc:740] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:01:04.203382  6259 raft_consensus.cc:493] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:04.203850  6259 raft_consensus.cc:3060] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:01:04.137626  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:04.204501  2332 raft_consensus.cc:3060] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:01:04.138060  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:04.141314  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:04.145151  6189 leader_election.cc:290] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:04.137759  5777 raft_consensus.cc:3055] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:01:04.206869  1602 raft_consensus.cc:2468] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 10.
I20260709 13:01:04.207008  5777 raft_consensus.cc:740] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:01:04.208112  5777 consensus_queue.cc:260] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.10, Last appended by leader: 10, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:04.209231  5777 raft_consensus.cc:3060] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:01:04.144972  5938 raft_consensus.cc:493] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:04.194707  6295 raft_consensus.cc:493] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:01:04.145670  1717 leader_election.cc:304] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:04.211697  6295 raft_consensus.cc:515] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:04.146327  1751 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55848 (request call id 2393) took 1010 ms. Trace:
I20260709 13:01:04.212950  1751 rpcz_store.cc:276] 0709 13:01:03.136193 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:03.136323 (+   130us) service_pool.cc:224] Handling call
0709 13:01:04.146099 (+1009776us) spinlock_profiling.cc:243] Waited 914 ms on lock 0x7b540004fe54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:01:04.146252 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2103098240}
W20260709 13:01:04.159479  1571 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.107.195:40693 timed out after 2.912s (SENT) [suppressed 39 similar messages]
I20260709 13:01:04.163335  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:04.214459  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:04.166111  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:04.169335  2337 raft_consensus.cc:2468] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 10.
I20260709 13:01:04.215324  1751 raft_consensus.cc:3060] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Advancing to term 10
I20260709 13:01:04.170136  5228 raft_consensus.cc:515] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:04.215557  1599 raft_consensus.cc:2468] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 12.
I20260709 13:01:04.178700  1752 raft_consensus.cc:1240] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 14 LEADER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 13. Current term is 14. Ops: [13.10-13.10]
I20260709 13:01:04.096117  5989 leader_election.cc:290] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:04.114347  6069 leader_election.cc:290] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:04.182361  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:04.091058  6082 leader_election.cc:290] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:04.225325  1646 leader_election.cc:304] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:01:04.200276  6244 raft_consensus.cc:2804] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:01:04.202147  5663 raft_consensus.cc:3060] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:01:04.144972  1754 raft_consensus.cc:1275] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 10 index: 8. Preceding OpId from leader: term: 11 index: 8. (term mismatch)
I20260709 13:01:04.227146  1754 pending_rounds.cc:85] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701: Aborting all ops after (but not including) 7
I20260709 13:01:04.227548  1754 pending_rounds.cc:107] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701: Aborting uncommitted NO_OP operation due to leader change: 10.8
I20260709 13:01:04.223557  1677 raft_consensus.cc:2468] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 12.
I20260709 13:01:04.227985  1754 raft_consensus.cc:2889] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:01:04.210764  5938 raft_consensus.cc:515] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:04.161212  6140 raft_consensus.cc:493] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:04.200840  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:04.242836  1745 raft_consensus.cc:2468] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 8.
W20260709 13:01:04.148862  6262 consensus_meta.cc:287] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.576s	user 0.013s	sys 0.002s
I20260709 13:01:04.202587  1569 leader_election.cc:304] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:01:04.158597  1672 raft_consensus.cc:2468] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 13.
I20260709 13:01:04.177876  1753 raft_consensus.cc:2468] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 13.
W20260709 13:01:04.226655  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 43807.7us
I20260709 13:01:04.248136  1675 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57092 (request call id 978) took 1261 ms. Trace:
W20260709 13:01:04.249125  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 103537us
I20260709 13:01:04.249545  1671 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52778 (request call id 2142) took 1502 ms. Trace:
I20260709 13:01:04.229398  2299 leader_election.cc:304] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:01:04.255703  2332 raft_consensus.cc:2468] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 10.
I20260709 13:01:04.162920  6046 raft_consensus.cc:515] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:04.256737  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:04.248560  1675 rpcz_store.cc:276] 0709 13:01:03.016561 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:03.016985 (+   424us) service_pool.cc:224] Handling call
0709 13:01:03.075994 (+ 59009us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:03.106457 (+ 30463us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:04.247923 (+1141466us) spinlock_profiling.cc:243] Waited 45.9 ms on lock 0x7b5400090950. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661618ad 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:01:04.248071 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":105494784}
I20260709 13:01:04.201804  6156 raft_consensus.cc:3055] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:01:04.259127  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:04.249996  1671 rpcz_store.cc:276] 0709 13:01:02.746879 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:02.747610 (+   731us) service_pool.cc:224] Handling call
0709 13:01:02.748058 (+   448us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:03.017338 (+269280us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:04.249272 (+1231934us) spinlock_profiling.cc:243] Waited 253 ms on lock 0x7b54001f4854. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:01:04.249481 (+   209us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":581127040}
I20260709 13:01:04.265087  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:04.258317  1674 raft_consensus.cc:2410] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for term 7 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
I20260709 13:01:04.265121  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:04.268589  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:04.269601  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:04.270256  1672 raft_consensus.cc:2468] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 11.
I20260709 13:01:04.228695  6244 raft_consensus.cc:493] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:01:04.276551  5228 leader_election.cc:290] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 11 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:04.238507  6069 raft_consensus.cc:2749] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:01:04.278209  1569 leader_election.cc:304] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:01:04.279637  1675 raft_consensus.cc:1275] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 5. (term mismatch)
I20260709 13:01:04.257802  6176 raft_consensus.cc:2804] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Leader election won for term 11
I20260709 13:01:04.281675  1675 pending_rounds.cc:85] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d: Aborting all ops after (but not including) 4
I20260709 13:01:04.280848  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:04.283233  1718 leader_election.cc:304] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:04.226105  6194 raft_consensus.cc:515] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:04.278607  6163 raft_consensus.cc:2749] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:01:04.306046  1599 raft_consensus.cc:1275] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 10 index: 11. Preceding OpId from leader: term: 12 index: 12. (index mismatch)
I20260709 13:01:04.273402  6199 consensus_queue.cc:1059] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.035s
I20260709 13:01:04.308444  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:04.306900  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:04.289800  6242 raft_consensus.cc:493] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:04.298305  6261 consensus_queue.cc:1059] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 11.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.009s
I20260709 13:01:04.309666  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:04.310379  6261 raft_consensus.cc:3055] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 LEADER]: Stepping down as leader of term 11
I20260709 13:01:04.295497  6082 consensus_queue.cc:1048] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.012s
I20260709 13:01:04.298285  5938 leader_election.cc:290] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:04.310288  1745 raft_consensus.cc:3060] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:01:04.312350  6298 consensus_queue.cc:1048] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.135s
I20260709 13:01:04.313724  1677 raft_consensus.cc:1275] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 13 index: 11. (index mismatch)
I20260709 13:01:04.261130  6195 raft_consensus.cc:493] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:04.310081  6242 raft_consensus.cc:515] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:04.303807  6176 raft_consensus.cc:697] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:01:04.317665  6242 leader_election.cc:290] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:04.267346  6156 raft_consensus.cc:740] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:01:04.263254  6144 raft_consensus.cc:2804] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Leader election won for term 12
W20260709 13:01:04.318823  6297 log.cc:927] Time spent T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.078s	user 0.001s	sys 0.000s
I20260709 13:01:04.203986  6162 consensus_queue.cc:260] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.11, Last appended by leader: 11, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:04.277400  6244 raft_consensus.cc:3060] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Advancing to term 13
W20260709 13:01:04.322427  6266 log.cc:927] Time spent T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.670s	user 0.004s	sys 0.000s
W20260709 13:01:04.323800  6275 log.cc:927] Time spent T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.604s	user 0.003s	sys 0.000s
I20260709 13:01:04.324751  1752 raft_consensus.cc:1275] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 11 index: 8. Preceding OpId from leader: term: 12 index: 8. (term mismatch)
I20260709 13:01:04.344085  1752 pending_rounds.cc:85] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701: Aborting all ops after (but not including) 7
W20260709 13:01:04.343868  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 184396us
I20260709 13:01:04.344439  1752 pending_rounds.cc:107] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701: Aborting uncommitted NO_OP operation due to leader change: 11.8
I20260709 13:01:04.344759  1752 raft_consensus.cc:2889] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:01:04.317708  6176 consensus_queue.cc:237] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.7, Last appended by leader: 7, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:04.263471  6189 raft_consensus.cc:2749] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 13:01:04.310818  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:04.349933  2334 raft_consensus.cc:2393] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 10: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:01:04.319268  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:04.314857  6195 raft_consensus.cc:515] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:04.306680  1750 raft_consensus.cc:1240] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 7. Current term is 8. Ops: []
I20260709 13:01:04.331353  1678 raft_consensus.cc:2393] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 10: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:01:04.333465  1675 pending_rounds.cc:107] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d: Aborting uncommitted NO_OP operation due to leader change: 7.5
W20260709 13:01:04.333091  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 49954.3us
I20260709 13:01:04.358174  1675 raft_consensus.cc:2889] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:01:04.338884  1672 raft_consensus.cc:1240] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 11. Current term is 12. Ops: [11.10-11.10]
I20260709 13:01:04.257710  6140 raft_consensus.cc:515] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:04.280377  6206 raft_consensus.cc:493] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:04.369911  1745 raft_consensus.cc:2468] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 11.
I20260709 13:01:04.328122  6067 raft_consensus.cc:2804] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader election won for term 10
I20260709 13:01:04.310855  6261 raft_consensus.cc:740] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:01:04.308470  6194 leader_election.cc:290] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:04.338418  6199 raft_consensus.cc:2804] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:01:04.382808  1745 raft_consensus.cc:1240] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 8. Current term is 9. Ops: []
I20260709 13:01:04.385681  1751 raft_consensus.cc:2468] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 10.
I20260709 13:01:04.341919  6285 raft_consensus.cc:2749] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:01:04.331296  6029 consensus_queue.cc:1048] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 7, Time since last communication: 0.046s
W20260709 13:01:04.405432  6249 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.702s	user 0.004s	sys 0.017s
W20260709 13:01:04.405977  6249 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.072s	user 0.005s	sys 0.017s
I20260709 13:01:04.375886  6192 raft_consensus.cc:493] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:01:04.407163  6192 raft_consensus.cc:515] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:04.407881  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:04.408633  1601 raft_consensus.cc:2468] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 12.
I20260709 13:01:04.409556  6192 leader_election.cc:290] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:04.409657  6194 raft_consensus.cc:493] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:01:04.379842  6206 raft_consensus.cc:515] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:04.410514  6194 raft_consensus.cc:515] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:01:04.412537  2301 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:04.391878  6067 raft_consensus.cc:697] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:01:04.392036  6232 consensus_queue.cc:1059] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 10.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.003s
I20260709 13:01:04.343294  6144 raft_consensus.cc:697] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [term 12 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:01:04.393483  6111 raft_consensus.cc:493] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:01:04.320749  6156 consensus_queue.cc:260] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:04.402258  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:04.397997  2298 leader_election.cc:304] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:01:04.401677  1602 raft_consensus.cc:1275] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 13:01:04.337841  6258 consensus_queue.cc:1048] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.164s
I20260709 13:01:04.429960  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:04.427803  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:04.441640  1598 raft_consensus.cc:2468] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 12.
I20260709 13:01:04.432590  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:04.433578  2298 leader_election.cc:304] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:01:04.433810  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:04.444449  1600 raft_consensus.cc:2468] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 11.
I20260709 13:01:04.435544  6229 raft_consensus.cc:2804] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:01:04.434177  5999 raft_consensus.cc:515] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:04.436211  6277 consensus_queue.cc:1048] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.041s
I20260709 13:01:04.387382  6199 raft_consensus.cc:493] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:01:04.376164  6259 raft_consensus.cc:515] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:01:04.438422  6300 log.cc:927] Time spent T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.095s	user 0.000s	sys 0.000s
I20260709 13:01:04.488057  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:04.479750  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:04.483510  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:04.494706  1604 raft_consensus.cc:2393] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 12: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:01:04.427462  6171 raft_consensus.cc:493] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:01:04.496073  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:04.444514  6258 consensus_queue.cc:1243] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer af520b67945d443b901bbeca37cc5701 log is divergent from this leader: its last log entry 11.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260709 13:01:04.510401  1679 raft_consensus.cc:2393] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 11: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:01:04.486272  6199 raft_consensus.cc:3060] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:01:04.463608  6229 raft_consensus.cc:493] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:04.499217  6192 raft_consensus.cc:476] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 13 LEADER]: Not starting pre-election -- already a leader
I20260709 13:01:04.500995  6206 leader_election.cc:290] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:04.433060  6111 raft_consensus.cc:515] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:04.511071  6156 raft_consensus.cc:3060] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Advancing to term 11
W20260709 13:01:04.528774  1779 heartbeater.cc:646] Failed to heartbeat to 127.0.107.254:33643 (2 consecutive failures): Timed out: Failed to send heartbeat to master: TSHeartbeat RPC to 127.0.107.254:33643 timed out after 15.000s (SENT)
I20260709 13:01:04.496893  6126 consensus_queue.cc:1048] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.046s
I20260709 13:01:04.534354  6229 raft_consensus.cc:3060] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:01:04.560315  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:04.532172  6293 raft_consensus.cc:493] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:01:04.564164  1603 raft_consensus.cc:3060] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:01:04.535092  6069 raft_consensus.cc:493] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:01:04.567268  6195 leader_election.cc:290] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:04.569464  6290 raft_consensus.cc:493] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:04.509858  6171 raft_consensus.cc:515] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:04.606958  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 13 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:04.578775  6194 leader_election.cc:290] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:04.535179  6144 consensus_queue.cc:237] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.10, Last appended by leader: 10, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:04.529594  5989 raft_consensus.cc:493] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:04.635156  1747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55848 (request call id 2384) took 1908 ms. Trace:
I20260709 13:01:04.600539  6293 raft_consensus.cc:515] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:04.547343  6067 consensus_queue.cc:237] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:04.604568  6277 raft_consensus.cc:493] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:04.653738  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:04.703382  1747 rpcz_store.cc:276] 0709 13:01:02.727194 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:02.727358 (+   164us) service_pool.cc:224] Handling call
0709 13:01:02.727651 (+   293us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:02.967284 (+239633us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 13:01:03.171550 (+204266us) log.cc:844] Serialized 14 byte log entry
0709 13:01:03.592385 (+420835us) log.cc:844] Serialized 14 byte log entry
0709 13:01:03.592984 (+   599us) log.cc:844] Serialized 14 byte log entry
0709 13:01:03.596231 (+  3247us) log.cc:844] Serialized 14 byte log entry
0709 13:01:03.599772 (+  3541us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:01:03.599834 (+    62us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:01:03.599892 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:03.996975 (+397083us) spinlock_profiling.cc:243] Waited 26.1 ms on lock 0x7b1800427900. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f4263075522 00007f42630777c1 00007f4263077779 00007f4263077709 00007f42630776d1 00007f42630769cd 000000000038f354 000000000038f199 00007f426305883a
0709 13:01:04.023249 (+ 26274us) spinlock_profiling.cc:243] Waited 16.6 ms on lock 0x7b1800427900. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f42630432a6 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426b08baa1 00007f426b082b37 00007f426b0829ca 00007f426b17a26f
0709 13:01:04.055252 (+ 32003us) spinlock_profiling.cc:243] Waited 26.6 ms on lock 0x7b1800427900. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f42630753f3 00007f42630777c1 00007f4263077779 00007f4263077709 00007f42630776d1 00007f42630769cd 000000000038f354 000000000038f199 00007f426305883a
0709 13:01:04.147492 (+ 92240us) spinlock_profiling.cc:243] Waited 20.6 ms on lock 0x7b4400022b4c. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f4262f9bc85 00007f4262f9c6a2 00007f4265d0e1db 00007f4265d09aaf 00007f4265cfd86e 00007f4265cfcbb6 00007f4265cfc7d8 00007f4265d06221 00007f4265d061d9 00007f4265d06169
0709 13:01:04.198379 (+ 50887us) spinlock_profiling.cc:243] Waited 9.85 ms on lock 0x7b4800121ed8. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426b08baa1 00007f426b082b37 00007f426b0829ca 00007f426b17a26f 00007f426b179f37 00007f426b18e55c 00007f426b18e508 00007f426b18e488 00007f426b18e443 00007f426b18d6ec 00007f426a0b7a96
0709 13:01:04.634844 (+436465us) spinlock_profiling.cc:243] Waited 239 ms on lock 0x7b54000e4054. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:01:04.634919 (+    75us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:04.635066 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":90189,"prepare.queue_time_us":415445,"prepare.run_cpu_time_us":238,"prepare.run_wall_time_us":11873,"raft.queue_time_us":33483,"raft.run_cpu_time_us":6021,"raft.run_wall_time_us":618540,"spinlock_wait_cycles":779634048,"thread_start_us":513251,"threads_started":3,"wal-append.queue_time_us":331919}
I20260709 13:01:04.649243  6245 raft_consensus.cc:493] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:04.701749  6251 raft_consensus.cc:493] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:04.728840  1717 leader_election.cc:304] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:04.732689  1643 leader_election.cc:304] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:04.735416  1717 leader_election.cc:304] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:04.705734  6206 raft_consensus.cc:493] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:04.743880  1717 leader_election.cc:304] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
W20260709 13:01:04.726922  6305 log.cc:927] Time spent T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.241s	user 0.001s	sys 0.000s
I20260709 13:01:04.747915  6183 raft_consensus.cc:493] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:04.745132  6189 raft_consensus.cc:2749] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:01:04.748996  6201 consensus_queue.cc:1048] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:01:04.660257  6069 raft_consensus.cc:515] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:04.702500  6290 raft_consensus.cc:515] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:04.745355  6188 raft_consensus.cc:2804] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:01:04.741284  5999 leader_election.cc:290] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 14 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:04.744937  6194 raft_consensus.cc:2749] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:01:04.755659  6259 leader_election.cc:290] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 11 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:04.776959  1675 raft_consensus.cc:2468] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 7.
I20260709 13:01:04.712858  5989 raft_consensus.cc:515] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:04.761204  5991 raft_consensus.cc:2749] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:01:04.723635  6277 raft_consensus.cc:515] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:04.766530  6195 raft_consensus.cc:2804] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Leader pre-election won for term 13
W20260709 13:01:04.773161  5663 consensus_meta.cc:287] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.515s	user 0.008s	sys 0.005s
I20260709 13:01:04.734870  6245 raft_consensus.cc:515] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:04.772701  6188 raft_consensus.cc:493] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:04.755980  6206 raft_consensus.cc:515] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:04.746385  6251 raft_consensus.cc:515] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:04.765352  6183 raft_consensus.cc:515] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:04.816408  6308 log.cc:927] Time spent T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.225s	user 0.001s	sys 0.000s
I20260709 13:01:04.820184  5989 leader_election.cc:290] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:04.823256  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:04.823587  6293 leader_election.cc:290] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:04.824707  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:04.824887  1596 raft_consensus.cc:2468] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 12.
W20260709 13:01:04.824903  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 81113.1us
I20260709 13:01:04.804466  6195 raft_consensus.cc:493] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:04.829417  1717 leader_election.cc:304] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:04.811131  6188 raft_consensus.cc:3060] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:01:04.809729  6111 leader_election.cc:290] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:04.829053  6195 raft_consensus.cc:3060] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:01:04.843063  6271 raft_consensus.cc:3055] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:01:04.843537  6271 raft_consensus.cc:740] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:01:04.844439  6271 consensus_queue.cc:260] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:04.845237  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:04.820216  6290 leader_election.cc:290] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:04.809429  5663 raft_consensus.cc:515] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:04.845544  6271 raft_consensus.cc:3060] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:01:04.849805  1717 leader_election.cc:304] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:04.833453  6201 raft_consensus.cc:2804] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:01:04.853191  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:04.854664  6293 raft_consensus.cc:2749] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:01:04.858587  1603 raft_consensus.cc:2468] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 10.
I20260709 13:01:04.861503  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:04.862469  1601 raft_consensus.cc:2468] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 12.
W20260709 13:01:04.864183  6317 log.cc:927] Time spent T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.084s	user 0.001s	sys 0.000s
I20260709 13:01:04.865633  6206 leader_election.cc:290] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:04.867090  6251 leader_election.cc:290] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:01:04.871296  6319 log.cc:927] Time spent T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.065s	user 0.000s	sys 0.000s
I20260709 13:01:04.872685  6069 leader_election.cc:290] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:04.859977  6201 raft_consensus.cc:493] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:04.875384  6201 raft_consensus.cc:3060] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:01:04.876070  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 12 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:04.878238  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:04.879889  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:04.884569  2328 raft_consensus.cc:1240] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 12. Current term is 13. Ops: []
I20260709 13:01:04.888996  1672 raft_consensus.cc:1240] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 10. Current term is 11. Ops: [10.10-10.10]
I20260709 13:01:04.891700  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:04.892287  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:04.894424  1598 raft_consensus.cc:1240] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 11. Current term is 12. Ops: []
I20260709 13:01:04.891839  2336 raft_consensus.cc:1240] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 8. Current term is 9. Ops: []
I20260709 13:01:04.895810  1598 raft_consensus.cc:1275] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260709 13:01:04.883586  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:04.896808  2334 raft_consensus.cc:2468] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 11.
I20260709 13:01:04.899544  6183 leader_election.cc:290] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:04.906810  1717 leader_election.cc:304] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
W20260709 13:01:04.883908  6315 log.cc:927] Time spent T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.178s	user 0.000s	sys 0.001s
W20260709 13:01:04.885398  6311 log.cc:927] Time spent T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.187s	user 0.000s	sys 0.001s
I20260709 13:01:04.913692  5833 raft_consensus.cc:493] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:01:04.895262  6069 raft_consensus.cc:493] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:01:04.909215  6171 leader_election.cc:290] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:04.905082  5663 leader_election.cc:290] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 13 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:01:04.920976  6309 log.cc:927] Time spent T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.158s	user 0.001s	sys 0.001s
I20260709 13:01:04.918704  6259 consensus_queue.cc:1048] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.007s
I20260709 13:01:04.925992  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:04.933643  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:04.934404  1605 raft_consensus.cc:2393] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 11: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:01:04.917661  6249 raft_consensus.cc:3055] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:01:04.917119  6206 raft_consensus.cc:2804] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:01:04.941138  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:04.930668  6069 raft_consensus.cc:515] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:04.953619  6069 leader_election.cc:290] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:04.954605  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:04.954864  1677 raft_consensus.cc:1240] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 10. Current term is 11. Ops: [10.5-10.5]
I20260709 13:01:04.958401  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:04.962762  2301 leader_election.cc:304] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:04.964392  6218 raft_consensus.cc:2804] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 13:01:04.964959  6218 raft_consensus.cc:697] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 13 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:01:04.965909  6140 leader_election.cc:290] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:04.966872  5994 consensus_queue.cc:1059] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 10.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 13:01:04.967643  5994 consensus_queue.cc:1243] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer af520b67945d443b901bbeca37cc5701 log is divergent from this leader: its last log entry 14.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 9
I20260709 13:01:04.968684  5994 raft_consensus.cc:3055] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 13:01:04.969026  5994 raft_consensus.cc:740] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:01:04.940304  6242 raft_consensus.cc:493] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:04.970855  6242 raft_consensus.cc:515] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:04.972993  6310 log.cc:927] Time spent T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.444s	user 0.000s	sys 0.001s
I20260709 13:01:04.973264  6242 leader_election.cc:290] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:04.917492  6290 consensus_queue.cc:1059] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 11.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.023s
I20260709 13:01:04.965939  6218 consensus_queue.cc:237] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 9, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:04.978315  6312 log.cc:927] Time spent T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.333s	user 0.000s	sys 0.001s
I20260709 13:01:04.979172  6183 consensus_queue.cc:1048] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 13:01:04.979892  6183 consensus_queue.cc:1243] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer 816fadcea8fc4893882f5f9b9f22a03d log is divergent from this leader: its last log entry 12.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 9
W20260709 13:01:04.984486  6244 consensus_meta.cc:287] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.621s	user 0.005s	sys 0.007s
I20260709 13:01:04.984974  6244 raft_consensus.cc:515] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:04.986121  1672 raft_consensus.cc:1240] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 12. Current term is 13. Ops: []
I20260709 13:01:04.928906  5833 raft_consensus.cc:515] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:04.940558  6206 raft_consensus.cc:493] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:01:04.992127  6206 raft_consensus.cc:3060] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:01:04.993643  5833 leader_election.cc:290] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:04.939057  6249 raft_consensus.cc:740] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:01:04.969898  5994 consensus_queue.cc:260] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 13.10, Last appended by leader: 10, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:04.996371  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:05.027384  5994 raft_consensus.cc:3060] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:01:05.029069  1672 pending_rounds.cc:85] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d: Aborting all ops after (but not including) 9
I20260709 13:01:05.029397  1672 pending_rounds.cc:107] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d: Aborting uncommitted NO_OP operation due to leader change: 12.10
I20260709 13:01:05.029649  1672 raft_consensus.cc:2889] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:01:04.998178  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:01:05.016367  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 54153.5us
I20260709 13:01:05.018015  6298 consensus_queue.cc:1048] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 13:01:05.030552  1677 raft_consensus.cc:2468] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 11.
I20260709 13:01:05.031287  1646 leader_election.cc:304] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:05.019712  6290 raft_consensus.cc:3055] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 LEADER]: Stepping down as leader of term 11
I20260709 13:01:05.032163  1718 leader_election.cc:304] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:05.032382  6290 raft_consensus.cc:740] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:01:05.033353  6290 consensus_queue.cc:260] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:05.034700  6290 raft_consensus.cc:3060] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:01:05.020568  6046 leader_election.cc:290] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:05.022174  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:05.025789  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:01:05.039394  1674 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52778 (request call id 2150) took 1485 ms (client timeout 1874 ms). Trace:
W20260709 13:01:05.039834  1674 rpcz_store.cc:269] 0709 13:01:03.554244 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:03.560687 (+  6443us) service_pool.cc:224] Handling call
0709 13:01:05.039186 (+1478499us) spinlock_profiling.cc:243] Waited 682 ms on lock 0x7b54001f6654. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:01:05.039329 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1569207040}
I20260709 13:01:05.040791  6249 consensus_queue.cc:260] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.5, Last appended by leader: 5, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:05.042366  6249 raft_consensus.cc:3060] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Advancing to term 11
W20260709 13:01:05.049280  6320 log.cc:927] Time spent T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.269s	user 0.001s	sys 0.000s
W20260709 13:01:05.053637  6275 log.cc:927] Time spent T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.683s	user 0.003s	sys 0.000s
I20260709 13:01:05.056882  6160 raft_consensus.cc:3060] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:01:05.058936  1673 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57092 (request call id 984) took 1749 ms. Trace:
I20260709 13:01:05.059391  1673 rpcz_store.cc:276] 0709 13:01:03.309710 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:03.309894 (+   184us) service_pool.cc:224] Handling call
0709 13:01:04.003028 (+693134us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:04.003304 (+   276us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 13:01:04.015683 (+ 12379us) log.cc:844] Serialized 14 byte log entry
0709 13:01:04.276798 (+261115us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:01:04.276859 (+    61us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:01:04.276915 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:04.276991 (+    76us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:05.058708 (+781717us) spinlock_profiling.cc:243] Waited 693 ms on lock 0x7b54000fe450. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661618ad 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:01:05.058859 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":15109,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":48,"raft.queue_time_us":558,"spinlock_wait_cycles":1596921600,"thread_start_us":257515,"threads_started":1,"wal-append.queue_time_us":355990}
I20260709 13:01:05.064900  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:05.068197  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:05.070411  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:05.071273  1671 raft_consensus.cc:2468] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 7.
I20260709 13:01:05.034127  1718 leader_election.cc:304] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:04.959367  1605 raft_consensus.cc:2468] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 11.
W20260709 13:01:05.079787  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 48610.3us
I20260709 13:01:05.080420  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:05.080813  1646 leader_election.cc:304] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:01:05.082343  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:01:05.085282  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 50374us
I20260709 13:01:05.085579  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:05.086107  1569 leader_election.cc:304] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:01:05.090077  1673 raft_consensus.cc:2393] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 11: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:01:05.089917  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:05.090713  2332 raft_consensus.cc:2468] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 8.
I20260709 13:01:05.101977  1646 leader_election.cc:304] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:01:05.105257  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:05.113776  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:05.114665  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:05.115418  2336 raft_consensus.cc:2468] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 11.
W20260709 13:01:05.115767  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 81715.9us
I20260709 13:01:04.949551  6245 leader_election.cc:290] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:05.117161  1718 leader_election.cc:304] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:05.122217  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:05.145927  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 12 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:05.146927  1602 raft_consensus.cc:2410] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for term 12 because replica has last-logged OpId of term: 10 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 9.
I20260709 13:01:05.023841  6259 raft_consensus.cc:493] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:05.144171  6232 raft_consensus.cc:493] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:05.177268  6232 raft_consensus.cc:515] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:05.177791  2328 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39726 (request call id 1077) took 1140 ms. Trace:
I20260709 13:01:05.179643  6232 leader_election.cc:290] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:05.158731  6325 raft_consensus.cc:493] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:01:05.180065  1597 raft_consensus.cc:1275] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260709 13:01:05.180516  6325 raft_consensus.cc:515] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:05.182817  6325 leader_election.cc:290] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:05.155817  6245 consensus_queue.cc:1048] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.012s
I20260709 13:01:05.183596  6261 consensus_queue.cc:260] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:05.160619  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:05.112238  6171 raft_consensus.cc:2749] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 13:01:05.187433  6261 raft_consensus.cc:3060] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:01:05.186678  6259 raft_consensus.cc:515] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:05.189193  6295 leader_election.cc:290] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:01:05.195416  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 48168.5us
W20260709 13:01:05.198705  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 81619.6us
I20260709 13:01:05.199822  1718 leader_election.cc:304] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:05.119621  5938 raft_consensus.cc:2804] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:01:05.105695  6069 raft_consensus.cc:493] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:05.178223  2328 rpcz_store.cc:276] 0709 13:01:04.037295 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:04.037494 (+   199us) service_pool.cc:224] Handling call
0709 13:01:04.269506 (+232012us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:04.885009 (+615503us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:05.176687 (+291678us) spinlock_profiling.cc:243] Waited 615 ms on lock 0x7b540065e0d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:01:05.177523 (+   836us) spinlock_profiling.cc:243] Waited 187 ms on lock 0x7b540065e0d0. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661618ad 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:01:05.177693 (+   170us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1843650560}
I20260709 13:01:05.123122  6163 raft_consensus.cc:2804] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:01:05.275887  1748 raft_consensus.cc:1275] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20260709 13:01:05.277242  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:05.277976  1751 raft_consensus.cc:2468] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 10.
W20260709 13:01:05.278414  1643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:05.279068  1643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:05.279989  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:05.280802  1749 raft_consensus.cc:2468] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 8.
I20260709 13:01:05.285899  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:05.286726  1751 raft_consensus.cc:2393] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 10: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:01:05.289496  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 15 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:05.156414  6285 raft_consensus.cc:493] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
W20260709 13:01:05.301002  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 73147.3us
I20260709 13:01:05.294487  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
W20260709 13:01:05.295817  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 96111.3us
I20260709 13:01:05.297250  1746 raft_consensus.cc:1275] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260709 13:01:05.193678  6259 leader_election.cc:290] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:05.309012  1718 leader_election.cc:304] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:05.206735  5833 raft_consensus.cc:2749] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 13:01:05.336333  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:05.207250  6325 raft_consensus.cc:493] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:01:05.297442  5999 raft_consensus.cc:493] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:05.335718  1749 raft_consensus.cc:2393] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 11: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:01:05.108952  6242 consensus_queue.cc:1059] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 12.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.188s
I20260709 13:01:05.266774  5938 raft_consensus.cc:493] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:05.351414  1746 raft_consensus.cc:1275] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260709 13:01:05.346239  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:05.297305  6189 raft_consensus.cc:493] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:05.355538  2328 raft_consensus.cc:1275] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20260709 13:01:05.273195  6163 raft_consensus.cc:493] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:01:05.283730  6176 raft_consensus.cc:493] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:05.310511  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:05.360450  6189 raft_consensus.cc:515] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:05.376403  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:05.377290  1751 raft_consensus.cc:2468] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 9.
W20260709 13:01:05.363847  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 244506us
I20260709 13:01:05.395252  1646 leader_election.cc:304] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:05.359478  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:05.288900  6293 raft_consensus.cc:493] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:05.287982  6069 raft_consensus.cc:515] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:05.335008  6247 raft_consensus.cc:2749] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:01:05.345808  6111 raft_consensus.cc:493] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:05.267082  6188 raft_consensus.cc:515] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:05.273720  6195 raft_consensus.cc:515] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:05.357440  5989 raft_consensus.cc:2749] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:01:05.354825  5938 raft_consensus.cc:3060] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:01:05.395809  6163 raft_consensus.cc:3060] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:01:05.399978  6215 raft_consensus.cc:3055] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:01:05.434326  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:05.435346  1751 raft_consensus.cc:3060] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:01:05.410806  6259 raft_consensus.cc:2804] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Leader election won for term 10
I20260709 13:01:05.412600  5833 raft_consensus.cc:2804] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:01:05.351840  6285 raft_consensus.cc:515] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:05.426290  6244 leader_election.cc:290] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 13 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:05.355440  6325 raft_consensus.cc:515] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:05.445199  1746 raft_consensus.cc:1240] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 12. Current term is 13. Ops: [10.7-12.8]
I20260709 13:01:05.368491  5999 raft_consensus.cc:515] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:05.455307  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 335096us
W20260709 13:01:05.464593  1570 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:05.467196  6328 log.cc:927] Time spent T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.101s	user 0.001s	sys 0.000s
I20260709 13:01:05.424408  6111 raft_consensus.cc:515] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:05.408752  6176 raft_consensus.cc:515] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:05.473111  1571 leader_election.cc:304] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:05.439291  6215 raft_consensus.cc:740] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:01:05.447669  5833 raft_consensus.cc:493] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:01:05.488162  5833 raft_consensus.cc:3060] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:01:05.491662  6277 leader_election.cc:290] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:05.493366  6147 raft_consensus.cc:493] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:05.493942  6147 raft_consensus.cc:515] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:05.483470  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 174556us
I20260709 13:01:05.484484  1571 leader_election.cc:304] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:05.486433  6232 raft_consensus.cc:2804] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Leader election won for term 11
I20260709 13:01:05.501645  1677 raft_consensus.cc:1275] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 14 index: 10. (index mismatch)
I20260709 13:01:05.487449  6067 consensus_queue.cc:1048] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 13:01:05.424899  6293 raft_consensus.cc:515] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:05.501230  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:05.486819  6069 leader_election.cc:290] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:01:05.479691  6199 consensus_meta.cc:287] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.936s	user 0.013s	sys 0.005s
I20260709 13:01:05.513511  1751 raft_consensus.cc:2468] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 11.
I20260709 13:01:05.520190  2301 leader_election.cc:304] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 13:01:05.527827  1676 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.504s	user 0.000s	sys 0.020s
W20260709 13:01:05.529219  1676 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.506s	user 0.000s	sys 0.021s
I20260709 13:01:05.532269  1754 raft_consensus.cc:2468] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 10.
I20260709 13:01:05.535310  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:05.558728  1569 leader_election.cc:304] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
W20260709 13:01:05.564409  6313 log.cc:927] Time spent T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.819s	user 0.009s	sys 0.004s
I20260709 13:01:05.574616  6183 raft_consensus.cc:2749] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:01:05.576927  6069 raft_consensus.cc:493] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:05.577525  6069 raft_consensus.cc:515] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:05.580437  6069 leader_election.cc:290] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:05.465863   431 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:01:05.605618  1752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55840 (request call id 2532) took 1418 ms. Trace:
I20260709 13:01:05.606231  1752 rpcz_store.cc:276] 0709 13:01:04.187656 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:04.351049 (+163393us) service_pool.cc:224] Handling call
0709 13:01:04.351535 (+   486us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:04.373869 (+ 22334us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:04.373954 (+    85us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:01:04.374968 (+  1014us) log.cc:844] Serialized 26 byte log entry
0709 13:01:04.572089 (+197121us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:01:04.599709 (+ 27620us) log.cc:844] Serialized 14 byte log entry
0709 13:01:04.603584 (+  3875us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:04.603674 (+    90us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:05.367063 (+763389us) spinlock_profiling.cc:243] Waited 3.75 ms on lock 0x7b1800301720. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f4263075522 00007f42630777c1 00007f4263077779 00007f4263077709 00007f42630776d1 00007f42630769cd 000000000038f354 000000000038f199 00007f426305883a
0709 13:01:05.563645 (+196582us) spinlock_profiling.cc:243] Waited 2.61 ms on lock 0x7b4400022b4c. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f4262f9bc85 00007f4262f9c6a2 00007f4265d0e1db 00007f4265d09aaf 00007f4265cfd86e 00007f4265cfcbb6 00007f4265cfc7d8 00007f4265d06221 00007f4265d061d9 00007f4265d06169
0709 13:01:05.605313 (+ 41668us) raft_consensus.cc:1685] finished
0709 13:01:05.605346 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:05.605515 (+   169us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":20965,"prepare.queue_time_us":27706,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":51,"raft.queue_time_us":24339,"raft.run_cpu_time_us":309,"raft.run_wall_time_us":20839,"spinlock_wait_cycles":14633088,"thread_start_us":165148,"threads_started":1,"wal-append.queue_time_us":254039}
I20260709 13:01:05.609686  6270 raft_consensus.cc:493] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:05.610461  6270 raft_consensus.cc:515] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:05.621565  6331 log.cc:927] Time spent T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.254s	user 0.001s	sys 0.000s
I20260709 13:01:05.626361  1679 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52766 (request call id 2580) took 1361 ms. Trace:
I20260709 13:01:05.626996  1679 rpcz_store.cc:276] 0709 13:01:04.265312 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:04.268313 (+  3001us) service_pool.cc:224] Handling call
0709 13:01:05.626062 (+1357749us) spinlock_profiling.cc:243] Waited 241 ms on lock 0x7b54000d8754. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:01:05.626271 (+   209us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":554088704}
I20260709 13:01:05.642513  6189 leader_election.cc:290] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:05.648973  6267 raft_consensus.cc:493] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:05.640573  6195 leader_election.cc:290] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 13 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:05.655294  6195 raft_consensus.cc:493] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:05.657124  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:05.659528  6192 raft_consensus.cc:493] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:05.660283  6192 raft_consensus.cc:515] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:05.661375  6162 raft_consensus.cc:3060] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:01:05.657490  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:05.663623  6195 raft_consensus.cc:515] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:05.700984  1644 leader_election.cc:304] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:01:05.665714  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:05.672960  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:05.674680  1751 raft_consensus.cc:1275] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260709 13:01:05.676681  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:05.677951  6192 leader_election.cc:290] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:05.681977  6270 leader_election.cc:290] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:05.683689  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:01:05.684758  1501 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35458 (request call id 306) took 42380 ms (client timeout 15000 ms). Trace:
I20260709 13:01:05.687100  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 15 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:05.727169  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:05.727945  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:05.688597  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:05.649871  6267 raft_consensus.cc:515] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:05.689039  1502 catalog_manager.cc:5697] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 reported cstate change: term changed from 7 to 9. New cstate: current_term: 9 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:01:05.699417  1646 leader_election.cc:304] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:01:05.531760  6333 raft_consensus.cc:2804] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:01:05.528383  6232 raft_consensus.cc:697] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:01:05.535457  6218 raft_consensus.cc:2804] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 13:01:05.779729  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:05.781138  1754 raft_consensus.cc:2468] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 7.
W20260709 13:01:05.730933  6254 log.cc:927] Time spent T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.169s	user 0.000s	sys 0.000s
I20260709 13:01:05.517469  6247 consensus_queue.cc:1048] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.181s
I20260709 13:01:05.790527  5989 consensus_queue.cc:1059] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 12.11, Next index: 12, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 13:01:05.792030  6325 leader_election.cc:290] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:05.517985  6199 raft_consensus.cc:515] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:05.747764  1502 catalog_manager.cc:5697] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 reported cstate change: term changed from 5 to 6, leader changed from 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) to 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196). New cstate: current_term: 6 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:01:05.741513  6333 raft_consensus.cc:493] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:05.738984  1747 raft_consensus.cc:1240] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 12. Current term is 13. Ops: [12.8-12.8]
I20260709 13:01:05.822584  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:05.739770  6258 raft_consensus.cc:2804] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:01:05.717861  6339 raft_consensus.cc:493] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:01:05.771085  6327 log.cc:927] Time spent T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.212s	user 0.001s	sys 0.000s
W20260709 13:01:05.794893  6323 log.cc:927] Time spent T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.091s	user 0.001s	sys 0.000s
I20260709 13:01:05.827569  6244 raft_consensus.cc:2749] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
W20260709 13:01:05.725549  1501 rpcz_store.cc:269] 0709 13:00:23.306479 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:23.306918 (+   439us) service_pool.cc:224] Handling call
0709 13:01:05.684620 (+42377702us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 13:01:00.462643 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 13:01:00.655073 (+192430us) write_op.cc:432] Acquiring schema lock in shared mode
0709 13:01:00.655126 (+    53us) write_op.cc:435] Acquired schema lock
0709 13:01:00.655152 (+    26us) tablet.cc:662] Decoding operations
0709 13:01:00.701746 (+ 46594us) write_op.cc:620] Acquiring the partition lock for write op
0709 13:01:00.730556 (+ 28810us) write_op.cc:641] Partition lock acquired for write op
0709 13:01:00.732717 (+  2161us) tablet.cc:685] Acquiring locks for 15 operations
0709 13:01:00.734241 (+  1524us) tablet.cc:701] Row locks acquired
0709 13:01:00.734301 (+    60us) write_op.cc:260] PREPARE: finished
0709 13:01:00.734660 (+   359us) write_op.cc:270] Start()
0709 13:01:00.734849 (+   189us) write_op.cc:276] Timestamp: P: 1783602060734553 usec, L: 0
0709 13:01:00.734893 (+    44us) op_driver.cc:348] REPLICATION: starting
0709 13:01:00.735864 (+   971us) log.cc:844] Serialized 5997 byte log entry
0709 13:01:02.037749 (+1301885us) op_driver.cc:464] REPLICATION: finished
0709 13:01:02.570670 (+532921us) spinlock_profiling.cc:243] Waited 5.5 ms on lock 0x7b4400022b4c. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f4262f9bc85 00007f4262f9c6a2 00007f4265d0e1db 00007f4265d09aaf 00007f42630580ac 00007f426beb2ab9 00007f426307469b 00007f426307293f 00007f4263074c02 00007f4269b98ef0
0709 13:01:02.634346 (+ 63676us) write_op.cc:301] APPLY: starting
0709 13:01:02.634488 (+   142us) tablet.cc:1367] starting BulkCheckPresence
0709 13:01:03.235791 (+601303us) tablet.cc:1370] finished BulkCheckPresence
0709 13:01:03.256805 (+ 21014us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 13:01:04.030885 (+774080us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 13:01:04.031353 (+   468us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=30,key_file_lookups=30,delta_file_lookups=30,mrs_lookups=0
0709 13:01:04.031440 (+    87us) write_op.cc:312] APPLY: finished
0709 13:01:04.032010 (+   570us) log.cc:844] Serialized 139 byte log entry
0709 13:01:04.272697 (+240687us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:01:04.297738 (+ 25041us) write_op.cc:454] Released schema lock
0709 13:01:04.298579 (+   841us) write_op.cc:341] FINISH: Updating metrics
0709 13:01:04.376680 (+ 78101us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:01:04.379303 (+  2623us) write_op.cc:454] Released schema lock
0709 13:01:04.892812 (+513509us) spinlock_profiling.cc:243] Waited 11.4 ms on lock 0x7b1800255880. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f42630753f3 00007f42630777c1 00007f4263077779 00007f4263077709 00007f42630776d1 00007f42630769cd 000000000038f354 000000000038f199 00007f426305883a
Metrics: {"mutex_wait_us":66886,"reported_tablets":46,"child_traces":[["op",{"apply.queue_time_us":585163,"apply.run_cpu_time_us":54883,"apply.run_wall_time_us":1703026,"cfile_cache_hit":62,"cfile_cache_hit_bytes":66947,"num_ops":15,"prepare.queue_time_us":114659,"prepare.run_cpu_time_us":39574,"prepare.run_wall_time_us":342616,"raft.queue_time_us":306715,"raft.run_cpu_time_us":36347,"raft.run_wall_time_us":780500,"replication_time_us":1275126,"spinlock_wait_cycles":38766592,"thread_start_us":1129788,"threads_started":5,"wal-append.queue_time_us":476875,"wal-append.run_cpu_time_us":48842,"wal-append.run_wall_time_us":1967818}]]}
W20260709 13:01:05.819072  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 119713us
I20260709 13:01:05.914855  1646 leader_election.cc:304] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:05.860605  6325 raft_consensus.cc:493] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:01:05.860988  1502 catalog_manager.cc:5697] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 reported cstate change: term changed from 7 to 8, leader changed from 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) to 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194). New cstate: current_term: 8 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 13:01:05.827241  6145 raft_consensus.cc:493] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:05.555425  6215 consensus_queue.cc:260] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.7, Last appended by leader: 7, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:05.810879  6242 raft_consensus.cc:493] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:05.841625  6046 raft_consensus.cc:2804] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:01:05.938316  1502 catalog_manager.cc:5697] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 reported cstate change: term changed from 7 to 8, leader changed from 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) to 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193). New cstate: current_term: 8 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 13:01:05.857033  6332 raft_consensus.cc:3055] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:01:05.757054  6245 raft_consensus.cc:493] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:05.924269  1745 raft_consensus.cc:1240] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 11. Current term is 12. Ops: []
I20260709 13:01:05.847002  6218 raft_consensus.cc:697] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 11 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:01:05.857860  6188 leader_election.cc:290] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 13 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:05.914132  6192 raft_consensus.cc:493] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:05.862386  6333 raft_consensus.cc:3060] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:01:05.902645  6258 raft_consensus.cc:493] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:05.945333  1646 leader_election.cc:304] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:05.926859  6189 raft_consensus.cc:493] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:05.960438  1502 catalog_manager.cc:5697] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 reported cstate change: term changed from 7 to 10, leader changed from 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) to 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196). New cstate: current_term: 10 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 13:01:05.965010  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 365562us
W20260709 13:01:05.947331  6317 log.cc:927] Time spent T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.085s	user 0.001s	sys 0.000s
I20260709 13:01:05.970609  1718 leader_election.cc:304] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:05.973827  1679 raft_consensus.cc:1275] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
W20260709 13:01:05.977273  1646 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:05.945312  6332 raft_consensus.cc:740] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:01:05.938242  6259 raft_consensus.cc:697] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
W20260709 13:01:05.983799  1570 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:05.989850  1502 catalog_manager.cc:5697] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 reported cstate change: term changed from 4 to 8. New cstate: current_term: 8 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:01:05.955485  6293 leader_election.cc:290] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:05.963635  6258 raft_consensus.cc:3060] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:01:05.924456  5989 consensus_queue.cc:1048] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.045s
I20260709 13:01:05.924189  6339 raft_consensus.cc:515] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:05.954128  6343 raft_consensus.cc:2804] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader pre-election won for term 10
W20260709 13:01:06.054329  1570 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:05.975831  5994 consensus_meta.cc:287] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.948s	user 0.013s	sys 0.008s
I20260709 13:01:05.994020  5999 leader_election.cc:290] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:05.966915  6188 raft_consensus.cc:493] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:01:06.073860  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 229564us
I20260709 13:01:05.969616  1602 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 2324) took 1022 ms. Trace:
I20260709 13:01:05.945119  6145 raft_consensus.cc:515] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:06.099519  2336 raft_consensus.cc:2410] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for term 15 because replica has last-logged OpId of term: 13 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 9.
I20260709 13:01:05.953599  6262 consensus_queue.cc:1059] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.045s
I20260709 13:01:05.940250  6325 raft_consensus.cc:515] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:01:06.104918  2332 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.822s	user 0.012s	sys 0.020s
I20260709 13:01:05.953398  6281 consensus_queue.cc:1048] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.054s
W20260709 13:01:06.004788  6144 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.847s	user 0.010s	sys 0.025s
I20260709 13:01:05.947088  6245 raft_consensus.cc:515] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:06.044035  1600 raft_consensus.cc:1275] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 11 index: 10. (term mismatch)
I20260709 13:01:06.039572  6067 raft_consensus.cc:493] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:05.995728  6244 raft_consensus.cc:2804] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:01:05.994899  6337 raft_consensus.cc:493] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
W20260709 13:01:06.065618  1569 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.506s	user 0.042s	sys 0.027s
I20260709 13:01:06.046933  6293 raft_consensus.cc:2749] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:01:05.966257  6147 leader_election.cc:290] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:01:06.125363  1569 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.566s	user 0.044s	sys 0.037s
I20260709 13:01:05.961314  6192 raft_consensus.cc:515] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:06.149927  6192 leader_election.cc:290] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:06.086378  1602 rpcz_store.cc:276] 0709 13:01:04.947599 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:05.185260 (+237661us) service_pool.cc:224] Handling call
0709 13:01:05.185583 (+   323us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:05.185800 (+   217us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 13:01:05.534546 (+348746us) log.cc:844] Serialized 14 byte log entry
0709 13:01:05.965871 (+431325us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:01:05.966162 (+   291us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:01:05.966227 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:05.966313 (+    86us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:05.969456 (+  3143us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":303950,"prepare.queue_time_us":304976,"prepare.run_cpu_time_us":29,"prepare.run_wall_time_us":29,"raft.queue_time_us":39489,"raft.run_cpu_time_us":110,"raft.run_wall_time_us":111,"thread_start_us":474165,"threads_started":2}
W20260709 13:01:06.106802  2332 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.825s	user 0.013s	sys 0.020s
I20260709 13:01:06.048825  6176 leader_election.cc:290] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:05.964993  6232 consensus_queue.cc:237] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:05.955269  6242 raft_consensus.cc:515] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:06.118459  1600 pending_rounds.cc:85] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a: Aborting all ops after (but not including) 9
W20260709 13:01:06.099798  2301 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:06.126400  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 567672us
I20260709 13:01:06.118366  6281 consensus_queue.cc:1243] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer af520b67945d443b901bbeca37cc5701 log is divergent from this leader: its last log entry 7.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260709 13:01:06.064121  6343 raft_consensus.cc:493] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
W20260709 13:01:06.134146  6340 log.cc:927] Time spent T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.181s	user 0.000s	sys 0.001s
I20260709 13:01:06.136090  6046 raft_consensus.cc:493] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
W20260709 13:01:06.137171  6336 log.cc:927] Time spent T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.219s	user 0.000s	sys 0.001s
I20260709 13:01:06.117703  6332 consensus_queue.cc:260] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:06.147450  6267 leader_election.cc:290] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:01:06.126003  6341 log.cc:927] Time spent T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.382s	user 0.000s	sys 0.002s
W20260709 13:01:06.240366  6261 consensus_meta.cc:287] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 1.052s	user 0.013s	sys 0.001s
I20260709 13:01:06.101675  1502 catalog_manager.cc:5697] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 reported cstate change: term changed from 5 to 6, leader changed from 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) to 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193). New cstate: current_term: 6 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
W20260709 13:01:06.161006  1672 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.568s	user 0.010s	sys 0.064s
W20260709 13:01:06.115365  6144 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.970s	user 0.010s	sys 0.026s
I20260709 13:01:06.253369  1502 catalog_manager.cc:5697] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 reported cstate change: term changed from 8 to 10. New cstate: current_term: 10 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:01:06.255291  1502 catalog_manager.cc:5697] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 reported cstate change: term changed from 5 to 6, leader changed from 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) to 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194). New cstate: current_term: 6 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
W20260709 13:01:06.164568  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 325431us
I20260709 13:01:06.170967  1644 leader_election.cc:304] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:01:06.063817  6295 consensus_queue.cc:1059] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 10.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.063s
I20260709 13:01:05.985329  6189 raft_consensus.cc:515] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:06.169309  1600 pending_rounds.cc:107] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a: Aborting uncommitted NO_OP operation due to leader change: 10.10
I20260709 13:01:06.172693  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:06.258598  1600 raft_consensus.cc:2889] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:01:06.259253  1745 raft_consensus.cc:2468] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 10.
I20260709 13:01:06.173869  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:06.177762  6242 leader_election.cc:290] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:06.178041  6285 leader_election.cc:290] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:06.179107  6195 leader_election.cc:290] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:06.183933  6245 leader_election.cc:290] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:06.130292  6244 raft_consensus.cc:493] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:01:06.185123  6346 log.cc:927] Time spent T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.064s	user 0.000s	sys 0.000s
I20260709 13:01:06.186715  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:06.187096  6111 leader_election.cc:290] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:06.187393  6145 leader_election.cc:290] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:01:06.190048  6162 consensus_meta.cc:287] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.528s	user 0.005s	sys 0.010s
I20260709 13:01:06.198614  6259 consensus_queue.cc:237] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.7, Last appended by leader: 7, Current term: 10, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:06.201267  6325 leader_election.cc:290] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:06.319090  2328 raft_consensus.cc:2468] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 16.
I20260709 13:01:06.201920  6199 leader_election.cc:290] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 12 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:06.156775  6067 raft_consensus.cc:515] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:06.325067  6189 leader_election.cc:290] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:06.325666  6125 raft_consensus.cc:493] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:06.326222  6125 raft_consensus.cc:515] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:06.326933  6067 leader_election.cc:290] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:01:06.168540  6271 consensus_meta.cc:287] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 1.286s	user 0.009s	sys 0.008s
W20260709 13:01:06.209601  5833 consensus_meta.cc:287] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.721s	user 0.017s	sys 0.021s
W20260709 13:01:06.213099  6342 log.cc:927] Time spent T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.147s	user 0.000s	sys 0.000s
I20260709 13:01:06.327930  5833 raft_consensus.cc:515] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:01:06.214324  6349 log.cc:927] Time spent T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.097s	user 0.003s	sys 0.000s
I20260709 13:01:06.194214  6215 raft_consensus.cc:3060] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:01:06.329144  1601 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 2325) took 1381 ms. Trace:
I20260709 13:01:06.330789  1746 raft_consensus.cc:2393] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 8: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:01:06.305948  6069 raft_consensus.cc:2749] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:01:06.329531  1601 rpcz_store.cc:276] 0709 13:01:04.947965 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:05.186142 (+238177us) service_pool.cc:224] Handling call
0709 13:01:05.186535 (+   393us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:01:05.298235 (+111700us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:05.298283 (+    48us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:01:05.344228 (+ 45945us) log.cc:844] Serialized 52 byte log entry
0709 13:01:05.694479 (+350251us) raft_consensus.cc:1645] Marking committed up to 3
0709 13:01:05.694525 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:05.694629 (+   104us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:06.212290 (+517661us) spinlock_profiling.cc:243] Waited 17.4 ms on lock 0x7b4400022b4c. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f4262f9bc85 00007f4262f9c6a2 00007f4265d0e1db 00007f4265d09aaf 00007f4265cfd86e 00007f4265cfcbb6 00007f4265cfc7d8 00007f4265d06221 00007f4265d061d9 00007f4265d06169
0709 13:01:06.328743 (+116453us) raft_consensus.cc:1685] finished
0709 13:01:06.328768 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:06.329045 (+   277us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":154390,"raft.queue_time_us":113913,"raft.run_cpu_time_us":169,"raft.run_wall_time_us":16714,"spinlock_wait_cycles":40057600,"thread_start_us":279812,"threads_started":1,"wal-append.queue_time_us":595348}
W20260709 13:01:06.224606  6290 consensus_meta.cc:287] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 1.189s	user 0.014s	sys 0.000s
I20260709 13:01:06.158483  6293 raft_consensus.cc:493] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:06.332147  6293 raft_consensus.cc:515] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:06.219964  6339 leader_election.cc:290] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:06.169292  6337 raft_consensus.cc:515] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:06.233780  1569 leader_election.cc:304] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:06.239675  6270 raft_consensus.cc:493] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:06.066483  6251 raft_consensus.cc:493] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:06.103829  6188 raft_consensus.cc:515] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:06.251752  1672 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.659s	user 0.010s	sys 0.064s
W20260709 13:01:06.252794  6206 consensus_meta.cc:287] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 1.260s	user 0.014s	sys 0.002s
I20260709 13:01:06.234382  6176 raft_consensus.cc:493] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:01:06.260733  2301 leader_election.cc:304] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
W20260709 13:01:06.339972  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 79341.5us
W20260709 13:01:06.341320  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 107622us
W20260709 13:01:06.246250  6201 consensus_meta.cc:287] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 1.351s	user 0.013s	sys 0.002s
I20260709 13:01:06.067327  6218 consensus_queue.cc:237] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:06.258102  6029 raft_consensus.cc:493] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:06.245460  6046 raft_consensus.cc:3060] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:01:06.243265  6343 raft_consensus.cc:3060] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Advancing to term 10
W20260709 13:01:06.283747  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 112868us
I20260709 13:01:06.301815  1605 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 2314) took 1392 ms. Trace:
I20260709 13:01:06.292078  1502 catalog_manager.cc:5697] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 reported cstate change: term changed from 5 to 6, leader changed from af520b67945d443b901bbeca37cc5701 (127.0.107.195) to 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196). New cstate: current_term: 6 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 13:01:06.308661  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 338129us
W20260709 13:01:06.340092  5938 consensus_meta.cc:287] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.899s	user 0.007s	sys 0.010s
I20260709 13:01:06.320027  6245 raft_consensus.cc:493] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:01:06.258149  6249 consensus_meta.cc:287] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 1.215s	user 0.010s	sys 0.004s
I20260709 13:01:06.260644  6140 raft_consensus.cc:493] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:06.313220  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:06.313760  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:06.313760  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:06.408257  1745 raft_consensus.cc:2393] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 14: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:01:06.315893  1676 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52778 (request call id 2159) took 1955 ms. Trace:
I20260709 13:01:06.316497  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:06.320604  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:06.324486  2331 raft_consensus.cc:1275] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 14 index: 10. (term mismatch)
I20260709 13:01:06.410609  2331 pending_rounds.cc:85] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1: Aborting all ops after (but not including) 9
I20260709 13:01:06.406654  2329 raft_consensus.cc:2468] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 11.
I20260709 13:01:06.261953  5989 raft_consensus.cc:493] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:06.413420  6140 raft_consensus.cc:515] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:06.413901  5989 raft_consensus.cc:515] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:06.366302  6162 raft_consensus.cc:2804] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:01:06.415568  6162 raft_consensus.cc:493] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:06.416078  6162 raft_consensus.cc:3060] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:01:06.344080  6176 raft_consensus.cc:515] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:06.409539  1676 rpcz_store.cc:276] 0709 13:01:04.360774 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:04.930651 (+569877us) service_pool.cc:224] Handling call
0709 13:01:04.930981 (+   330us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:04.931233 (+   252us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:04.931250 (+    17us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:01:04.931622 (+   372us) log.cc:844] Serialized 26 byte log entry
0709 13:01:05.530765 (+599143us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:01:05.530831 (+    66us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:05.530945 (+   114us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:06.315578 (+784633us) raft_consensus.cc:1685] finished
0709 13:01:06.315614 (+    36us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:06.315803 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":507114,"threads_started":1,"wal-append.queue_time_us":777352}
I20260709 13:01:06.389894  6354 consensus_queue.cc:1048] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.030s
I20260709 13:01:06.366115  6201 raft_consensus.cc:515] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:06.413980  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:06.231957  6258 raft_consensus.cc:515] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:06.342137  1569 leader_election.cc:304] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:01:06.342592  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:06.345039  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:01:06.431478  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 89424.8us
I20260709 13:01:06.431541  2330 raft_consensus.cc:2468] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 11.
I20260709 13:01:06.307500  6244 raft_consensus.cc:3060] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:01:06.345193  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:06.293656  6332 raft_consensus.cc:3060] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:01:06.373481  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 12 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:06.401508  1718 leader_election.cc:304] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:06.440145  1718 leader_election.cc:304] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:06.330356  5833 leader_election.cc:290] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 8 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:06.337677  6125 leader_election.cc:290] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:06.410936  2331 pending_rounds.cc:107] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1: Aborting uncommitted NO_OP operation due to leader change: 13.10
I20260709 13:01:06.443718  2331 raft_consensus.cc:2889] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:01:06.232970  6147 raft_consensus.cc:493] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:06.343626  6067 raft_consensus.cc:2749] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:01:06.390417  1605 rpcz_store.cc:276] 0709 13:01:04.909563 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:05.200790 (+291227us) service_pool.cc:224] Handling call
0709 13:01:05.201246 (+   456us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:05.201703 (+   457us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:05.201743 (+    40us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:01:05.273652 (+ 71909us) log.cc:844] Serialized 26 byte log entry
0709 13:01:05.478501 (+204849us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:01:05.478573 (+    72us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:05.478671 (+    98us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:06.301454 (+822783us) raft_consensus.cc:1685] finished
0709 13:01:06.301492 (+    38us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:06.301715 (+   223us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":16367,"raft.run_cpu_time_us":209,"raft.run_wall_time_us":23617,"thread_start_us":77526,"threads_started":1,"wal-append.queue_time_us":442189}
I20260709 13:01:06.341795  6293 leader_election.cc:290] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 13:01:06.397745  6163 consensus_meta.cc:287] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.953s	user 0.009s	sys 0.017s
I20260709 13:01:06.454840  6163 raft_consensus.cc:515] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:06.455567  2334 raft_consensus.cc:1240] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [term 12 LEADER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 11. Current term is 12. Ops: []
I20260709 13:01:06.456123  1750 raft_consensus.cc:1240] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 7. Current term is 8. Ops: []
I20260709 13:01:06.445963  6325 raft_consensus.cc:493] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:06.342626  6206 raft_consensus.cc:515] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:06.458312  6163 leader_election.cc:290] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 8 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:06.458236  6325 raft_consensus.cc:515] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:06.292131  6229 consensus_meta.cc:287] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 1.703s	user 0.013s	sys 0.004s
I20260709 13:01:06.424067  1753 raft_consensus.cc:1240] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 11. Current term is 12. Ops: [11.8-11.8]
I20260709 13:01:06.458405  2334 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49082 (request call id 908) took 1478 ms. Trace:
I20260709 13:01:06.460104  1750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55840 (request call id 2536) took 1184 ms. Trace:
I20260709 13:01:06.426610  1502 catalog_manager.cc:5697] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 reported cstate change: term changed from 6 to 8. New cstate: current_term: 8 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:01:06.431608  1747 raft_consensus.cc:2468] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 16.
I20260709 13:01:06.458850  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:01:06.367590  1570 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:06.433580  1569 leader_election.cc:304] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:06.462747  1750 rpcz_store.cc:276] 0709 13:01:05.275514 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:05.277278 (+  1764us) service_pool.cc:224] Handling call
0709 13:01:05.277643 (+   365us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:06.456636 (+1178993us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:06.459830 (+  3194us) spinlock_profiling.cc:243] Waited 1.18 s on lock 0x7b5400324bd4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:01:06.460011 (+   181us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2709412096}
I20260709 13:01:06.465384  1569 leader_election.cc:304] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:01:06.434921  6350 raft_consensus.cc:2804] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:01:06.435523  6357 consensus_queue.cc:1048] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 13:01:06.460120  6229 raft_consensus.cc:515] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:06.468341  2335 raft_consensus.cc:2368] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 12. Current term is 13.
I20260709 13:01:06.416706  5989 leader_election.cc:290] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:06.469110  6229 leader_election.cc:290] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 13 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:06.460681  6249 raft_consensus.cc:2804] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 13:01:06.460326  6206 leader_election.cc:290] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 12 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:06.470184  6249 raft_consensus.cc:697] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 10 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:01:06.471202  6249 consensus_queue.cc:237] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:06.471810  6140 leader_election.cc:290] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:06.437160  6188 leader_election.cc:290] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:06.342942  6270 raft_consensus.cc:515] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:06.470180  6350 raft_consensus.cc:493] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:06.475042  6350 raft_consensus.cc:3060] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:01:06.460772  6199 raft_consensus.cc:2749] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:01:06.442042  6290 raft_consensus.cc:493] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:06.476959  6290 raft_consensus.cc:515] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:06.442560  1748 raft_consensus.cc:2410] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for term 12 because replica has last-logged OpId of term: 10 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 4.
I20260709 13:01:06.479166  6290 leader_election.cc:290] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:06.461947  2334 rpcz_store.cc:276] 0709 13:01:04.980520 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:04.980995 (+   475us) service_pool.cc:224] Handling call
0709 13:01:04.981362 (+   367us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:06.456147 (+1474785us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:06.458135 (+  1988us) spinlock_profiling.cc:243] Waited 1.47 s on lock 0x7b54006b6cd4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:01:06.458338 (+   203us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3389450496}
I20260709 13:01:06.420367  6337 leader_election.cc:290] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:06.407267  6339 raft_consensus.cc:493] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:06.442868  1680 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52766 (request call id 2585) took 1860 ms. Trace:
I20260709 13:01:06.404804  6189 raft_consensus.cc:493] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:06.450749  6147 raft_consensus.cc:515] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:06.475376  6270 leader_election.cc:290] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:06.344342  6251 raft_consensus.cc:515] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:06.421938  6201 leader_election.cc:290] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 12 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:06.408885  5938 raft_consensus.cc:515] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:06.486997  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:01:06.400282  6160 consensus_meta.cc:287] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 1.343s	user 0.006s	sys 0.018s
W20260709 13:01:06.475417  6356 log.cc:927] Time spent T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.080s	user 0.001s	sys 0.000s
I20260709 13:01:06.404104  6029 raft_consensus.cc:515] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:06.488492  1680 rpcz_store.cc:276] 0709 13:01:04.582203 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:04.933215 (+351012us) service_pool.cc:224] Handling call
0709 13:01:04.933552 (+   337us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:04.933968 (+   416us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 13:01:05.404672 (+470704us) log.cc:844] Serialized 14 byte log entry
0709 13:01:05.821371 (+416699us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:01:05.821953 (+   582us) log.cc:844] Serialized 26 byte log entry
0709 13:01:05.822122 (+   169us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:01:05.822164 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:05.822255 (+    91us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:06.442616 (+620361us) raft_consensus.cc:1685] finished
0709 13:01:06.442643 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:06.442796 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":60428,"prepare.queue_time_us":306962,"prepare.run_cpu_time_us":108,"prepare.run_wall_time_us":7114,"raft.queue_time_us":10855,"raft.run_cpu_time_us":300,"raft.run_wall_time_us":15110,"thread_start_us":883351,"threads_started":3,"wal-append.queue_time_us":131461}
I20260709 13:01:06.418531  6176 leader_election.cc:290] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:01:06.503022  1571 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.984s	user 0.004s	sys 0.010s
I20260709 13:01:06.409646  6245 raft_consensus.cc:515] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:06.483667  6067 raft_consensus.cc:2749] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:01:06.487772  6293 raft_consensus.cc:493] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:01:06.512073  6013 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.512s	user 0.006s	sys 0.008s
W20260709 13:01:06.506027  6333 consensus_meta.cc:287] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.511s	user 0.012s	sys 0.000s
W20260709 13:01:06.515409  1571 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.996s	user 0.004s	sys 0.010s
I20260709 13:01:06.514566  6160 raft_consensus.cc:2749] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20260709 13:01:06.582672  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 1.09806e+06us
W20260709 13:01:06.515717  1598 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.514s	user 0.002s	sys 0.021s
I20260709 13:01:06.497929  6325 leader_election.cc:290] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:06.504443  6290 raft_consensus.cc:493] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:01:06.530818  1502 catalog_manager.cc:5697] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 reported cstate change: term changed from 6 to 8. New cstate: current_term: 8 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:01:06.486663  6215 raft_consensus.cc:493] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:01:06.545645  1748 pending_rounds.cc:85] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701: Aborting all ops after (but not including) 4
I20260709 13:01:06.546736  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:06.593662  1748 pending_rounds.cc:107] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701: Aborting uncommitted NO_OP operation due to leader change: 7.5
I20260709 13:01:06.546321  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:06.594527  1675 raft_consensus.cc:2468] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 8.
I20260709 13:01:06.594993  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:06.553072  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:06.595477  6258 leader_election.cc:290] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 12 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:06.595719  1680 raft_consensus.cc:2468] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 11.
I20260709 13:01:06.554853  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:06.555248  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:06.597050  1604 raft_consensus.cc:3060] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:01:06.598572  6215 raft_consensus.cc:515] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:01:06.599269  6352 log.cc:927] Time spent T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.060s	user 0.000s	sys 0.000s
I20260709 13:01:06.601054  6215 leader_election.cc:290] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:06.601274  1672 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52766 (request call id 2592) took 1599 ms. Trace:
I20260709 13:01:06.581353  6156 consensus_queue.cc:1048] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.021s
I20260709 13:01:06.601718  1672 rpcz_store.cc:276] 0709 13:01:05.001378 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:05.028491 (+ 27113us) service_pool.cc:224] Handling call
0709 13:01:05.028813 (+   322us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:01:05.078922 (+ 50109us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:05.078973 (+    51us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:01:05.585295 (+506322us) spinlock_profiling.cc:243] Waited 5.07 ms on lock 0x7b180047c140. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f42630432a6 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4264a5ba11 00007f4264a6b372 00007f426619da01 00007f426616243a
0709 13:01:05.585839 (+   544us) spinlock_profiling.cc:243] Waited 39.3 ms on lock 0x7b2c00010828. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4264a5ba11 00007f4264a6b372 00007f426619da01 00007f426616243a 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2
0709 13:01:05.592191 (+  6352us) log.cc:844] Serialized 52 byte log entry
0709 13:01:06.336684 (+744493us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:01:06.336750 (+    66us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:06.336865 (+   115us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:06.601018 (+264153us) raft_consensus.cc:1685] finished
0709 13:01:06.601044 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:06.601205 (+   161us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":53918,"raft.queue_time_us":49160,"raft.run_cpu_time_us":180,"raft.run_wall_time_us":25427,"spinlock_wait_cycles":101972992,"thread_start_us":744030,"threads_started":1,"wal-append.queue_time_us":908445}
I20260709 13:01:06.604139  6337 consensus_queue.cc:1048] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 13:01:06.606685  6345 raft_consensus.cc:3055] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:01:06.607200  6345 raft_consensus.cc:740] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
W20260709 13:01:06.607770  1570 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:06.608174  6345 consensus_queue.cc:260] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.8, Last appended by leader: 8, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:06.609438  6345 raft_consensus.cc:3060] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:01:06.609570  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:06.611919  6290 raft_consensus.cc:515] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:06.614236  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 13 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:06.615446  1677 raft_consensus.cc:2393] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 11: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:01:06.618790  1604 raft_consensus.cc:2468] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 13.
I20260709 13:01:06.590801  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:06.502734  6189 raft_consensus.cc:515] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:06.501099  6339 raft_consensus.cc:515] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } }
I20260709 13:01:06.594085  1748 raft_consensus.cc:2889] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 13:01:06.584693  1598 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.583s	user 0.002s	sys 0.022s
I20260709 13:01:06.621580  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:06.538815  6262 consensus_queue.cc:1243] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer af520b67945d443b901bbeca37cc5701 log is divergent from this leader: its last log entry 11.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260709 13:01:06.594034  1502 catalog_manager.cc:5697] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 reported cstate change: term changed from 6 to 9, leader changed from 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) to af520b67945d443b901bbeca37cc5701 (127.0.107.195). New cstate: current_term: 9 leader_uuid: "af520b67945d443b901bbeca37cc5701" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 13:01:06.627247  1749 raft_consensus.cc:1240] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 11. Current term is 12. Ops: []
I20260709 13:01:06.578202  6271 raft_consensus.cc:493] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:06.639034  1745 raft_consensus.cc:1240] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 12. Current term is 13. Ops: [12.8-12.8]
I20260709 13:01:06.639328  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:06.646090  1601 raft_consensus.cc:1275] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
W20260709 13:01:06.648337  1751 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.825s	user 0.020s	sys 0.000s
W20260709 13:01:06.648774  1751 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.826s	user 0.020s	sys 0.000s
I20260709 13:01:06.651340  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
W20260709 13:01:06.631282  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 44548.4us
I20260709 13:01:06.635030  6189 leader_election.cc:290] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:06.632241  1502 catalog_manager.cc:5697] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 reported cstate change: term changed from 6 to 8. New cstate: current_term: 8 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:01:06.653249  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:06.657516  1571 leader_election.cc:304] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 13:01:06.575834  6293 raft_consensus.cc:515] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:06.673316  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:06.676318  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:06.678849  1753 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55848 (request call id 2418) took 1005 ms. Trace:
I20260709 13:01:06.683597  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 12 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:06.588460  6333 raft_consensus.cc:515] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:06.679399  1753 rpcz_store.cc:276] 0709 13:01:05.673101 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:05.674634 (+  1533us) service_pool.cc:224] Handling call
0709 13:01:05.675030 (+   396us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:06.461226 (+786196us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:06.678456 (+217230us) spinlock_profiling.cc:243] Waited 749 ms on lock 0x7b540035ff54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:01:06.678746 (+   290us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1721702784}
W20260709 13:01:06.688560  1646 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:06.656747  6140 raft_consensus.cc:2804] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:01:06.697621  1502 catalog_manager.cc:5697] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 reported cstate change: term changed from 4 to 7, leader changed from 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194) to 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196). New cstate: current_term: 7 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:01:06.702899  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:06.705482  1718 leader_election.cc:304] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:06.714021  1718 leader_election.cc:304] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:06.719298  1502 catalog_manager.cc:5697] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 reported cstate change: term changed from 6 to 9. New cstate: current_term: 9 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:01:06.672616  6337 raft_consensus.cc:2804] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:01:06.724615  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:06.730484  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:06.731240  1603 raft_consensus.cc:2376] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 10. Re-sending same reply.
I20260709 13:01:06.724208  1502 catalog_manager.cc:5697] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 reported cstate change: term changed from 7 to 8, leader changed from 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) to 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193). New cstate: current_term: 8 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 13:01:06.725401  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 12 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:06.733012  1605 raft_consensus.cc:2410] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for term 12 because replica has last-logged OpId of term: 10 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 4.
I20260709 13:01:06.725605  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:06.725714  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:06.728498  1596 raft_consensus.cc:1240] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 8. Current term is 9. Ops: []
I20260709 13:01:06.726519  2298 leader_election.cc:304] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:01:06.734903  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:06.735761  1680 raft_consensus.cc:2468] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 13.
I20260709 13:01:06.736743  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:06.647624  6046 raft_consensus.cc:515] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } }
I20260709 13:01:06.740238  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:06.742588  1643 leader_election.cc:304] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:01:06.643728  6271 raft_consensus.cc:515] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:06.753019  1502 catalog_manager.cc:5697] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 reported cstate change: term changed from 7 to 8, leader changed from 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) to 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194). New cstate: current_term: 8 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 13:01:06.753475  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:06.750809  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:01:06.814253  2332 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.557s	user 0.021s	sys 0.043s
I20260709 13:01:06.753451  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:01:06.814723  2332 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.557s	user 0.021s	sys 0.043s
I20260709 13:01:06.746621  1603 raft_consensus.cc:2393] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 12: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:01:06.752735  1745 raft_consensus.cc:3060] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Advancing to term 11
W20260709 13:01:06.806263  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 55430.6us
I20260709 13:01:06.729374  6337 raft_consensus.cc:493] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:06.817929  1747 raft_consensus.cc:2393] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 16: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:01:06.690405  6357 consensus_queue.cc:1059] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 11.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.044s
I20260709 13:01:06.809739  2336 raft_consensus.cc:1240] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 11 LEADER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 10. Current term is 11. Ops: []
I20260709 13:01:06.738358  6160 raft_consensus.cc:2804] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260709 13:01:06.721843  6232 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.543s	user 0.013s	sys 0.014s
I20260709 13:01:06.750164  1643 leader_election.cc:304] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:01:06.814649  1600 raft_consensus.cc:2468] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 7.
I20260709 13:01:06.808045  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:06.737567  6242 consensus_queue.cc:1059] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 7.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.081s
I20260709 13:01:06.822398  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:06.845875  1747 raft_consensus.cc:2468] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 7.
I20260709 13:01:06.847152  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:06.848759  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:06.748930  6360 raft_consensus.cc:2804] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 13:01:06.826735  1677 raft_consensus.cc:2468] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 9.
I20260709 13:01:06.828083  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:06.743697  6370 raft_consensus.cc:493] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:06.753561  6171 raft_consensus.cc:2749] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:01:06.849478  1747 raft_consensus.cc:2468] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 11.
W20260709 13:01:06.835284  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 430518us
I20260709 13:01:06.805023  1604 raft_consensus.cc:2468] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 13.
I20260709 13:01:06.734539  5989 consensus_queue.cc:1059] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 9.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.084s
I20260709 13:01:06.842948  1753 raft_consensus.cc:2468] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 8.
I20260709 13:01:06.827165  6337 raft_consensus.cc:3060] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Advancing to term 11
W20260709 13:01:06.844359  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 94275.7us
I20260709 13:01:06.852649  1745 raft_consensus.cc:1275] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 11 index: 8. (index mismatch)
I20260709 13:01:06.855758  1718 leader_election.cc:304] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:06.860813  1502 catalog_manager.cc:5697] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 reported cstate change: term changed from 8 to 10, leader changed from 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) to 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196). New cstate: current_term: 10 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:01:06.745203  6189 raft_consensus.cc:2804] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:01:06.863301  1643 leader_election.cc:304] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 13:01:06.864907  1645 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.503s	user 0.007s	sys 0.012s
W20260709 13:01:06.865944  1645 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.723s	user 0.008s	sys 0.012s
W20260709 13:01:06.841715  6232 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.666s	user 0.014s	sys 0.015s
W20260709 13:01:06.868494  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 418849us
I20260709 13:01:06.868736  1718 leader_election.cc:304] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:06.872423  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:06.842451  6160 raft_consensus.cc:493] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:01:06.874977  1718 leader_election.cc:304] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:06.876999  6290 leader_election.cc:290] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:06.885178  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:06.885772  6365 raft_consensus.cc:493] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:01:06.886621  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:06.887997  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:06.889596  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:06.890377  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:06.890942  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:06.891880  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:06.842165  6251 leader_election.cc:290] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:06.905006  2333 raft_consensus.cc:2468] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 10.
I20260709 13:01:06.904506  2335 raft_consensus.cc:2410] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for term 17 because replica has last-logged OpId of term: 15 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 11.
I20260709 13:01:06.911489  1718 leader_election.cc:304] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:06.911849  1751 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43156 (request call id 890) took 1170 ms. Trace:
I20260709 13:01:06.912524  1751 rpcz_store.cc:276] 0709 13:01:05.741599 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:05.759422 (+ 17823us) service_pool.cc:224] Handling call
0709 13:01:05.759744 (+   322us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:05.760003 (+   259us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 13:01:05.822717 (+ 62714us) log.cc:844] Serialized 14 byte log entry
0709 13:01:06.902087 (+1079370us) log.cc:844] Serialized 14 byte log entry
0709 13:01:06.911381 (+  9294us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:01:06.911461 (+    80us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:01:06.911555 (+    94us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:06.911655 (+   100us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:06.911781 (+   126us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":71071,"prepare.queue_time_us":63038,"prepare.run_cpu_time_us":80,"prepare.run_wall_time_us":7803,"raft.queue_time_us":1138,"raft.run_cpu_time_us":178,"raft.run_wall_time_us":3576,"thread_start_us":1076027,"threads_started":2}
I20260709 13:01:06.864548  6354 raft_consensus.cc:493] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:01:06.851668  6372 raft_consensus.cc:2749] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
W20260709 13:01:06.916065  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 52865us
I20260709 13:01:06.927503  1643 leader_election.cc:304] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:06.871454  6360 raft_consensus.cc:697] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 13 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:01:06.893261  6201 raft_consensus.cc:2749] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 13:01:06.864800  6125 raft_consensus.cc:493] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:06.871363  6189 raft_consensus.cc:493] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:01:06.928243  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:06.918087  1718 leader_election.cc:400] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 12 pre-election: Vote denied by peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625) with higher term. Message: Invalid argument: T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 12. Current term is 13.
W20260709 13:01:06.968786  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 50838.6us
I20260709 13:01:06.920136  6163 consensus_queue.cc:1048] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.004s
I20260709 13:01:06.929749  1644 leader_election.cc:304] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:01:06.930341  6258 raft_consensus.cc:493] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:01:06.931515  6366 log.cc:927] Time spent T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.090s	user 0.000s	sys 0.000s
W20260709 13:01:06.932331  6367 log.cc:927] Time spent T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.117s	user 0.000s	sys 0.000s
W20260709 13:01:06.986136  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 56493.3us
I20260709 13:01:06.988648  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:06.989315  1753 raft_consensus.cc:1240] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 13. Current term is 14. Ops: []
I20260709 13:01:06.967960  1643 leader_election.cc:304] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:06.861835  6370 raft_consensus.cc:515] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:06.943545  6259 raft_consensus.cc:493] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:06.910530  6364 raft_consensus.cc:2804] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:01:06.900959  6160 raft_consensus.cc:3060] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:01:06.965770  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:06.925859  6171 raft_consensus.cc:2749] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:01:06.991838  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:06.967990  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:06.921736  5833 raft_consensus.cc:2749] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:01:06.917840  1502 catalog_manager.cc:5697] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 reported cstate change: term changed from 6 to 7, leader changed from 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) to 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196). New cstate: current_term: 7 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:01:06.994477  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:06.926795  6140 raft_consensus.cc:493] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:06.986282  6290 raft_consensus.cc:2749] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:01:06.904354  6249 raft_consensus.cc:2804] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:01:07.041028  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:06.974025  6189 raft_consensus.cc:3060] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:01:06.922829  6365 raft_consensus.cc:515] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:07.091388  1751 raft_consensus.cc:2468] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 9.
I20260709 13:01:07.103713  1673 raft_consensus.cc:2393] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 11: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:01:07.100941  1502 catalog_manager.cc:5697] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 reported cstate change: term changed from 6 to 8, leader changed from 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) to 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196). New cstate: current_term: 8 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:01:07.004277  6242 raft_consensus.cc:493] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:01:07.092057  6162 consensus_meta.cc:287] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.675s	user 0.014s	sys 0.000s
I20260709 13:01:07.090826  6251 raft_consensus.cc:3060] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:01:07.154279  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:06.992381  6258 raft_consensus.cc:515] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:07.046182  6364 raft_consensus.cc:493] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:06.945518  6354 raft_consensus.cc:515] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:07.142944  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:07.165315  1751 raft_consensus.cc:2468] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 9.
I20260709 13:01:07.166525  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:07.167294  1751 raft_consensus.cc:3060] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:01:07.143715  1502 catalog_manager.cc:5697] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 reported cstate change: term changed from 7 to 8, leader changed from 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) to 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194). New cstate: current_term: 8 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 13:01:07.046694  6259 raft_consensus.cc:515] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:07.145792  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 12 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:07.177085  1680 raft_consensus.cc:2468] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 11.
I20260709 13:01:07.147953  6290 raft_consensus.cc:2749] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:01:06.972950  6125 raft_consensus.cc:515] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:07.097922  6333 leader_election.cc:290] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 9 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:07.034224  6339 leader_election.cc:290] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:07.102675  6140 raft_consensus.cc:3060] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:01:07.138371  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:07.159219  1597 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 1011) took 1173 ms. Trace:
W20260709 13:01:07.194423  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 226534us
I20260709 13:01:07.155042  2336 raft_consensus.cc:2468] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 8.
I20260709 13:01:07.196091  1643 leader_election.cc:304] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:01:07.108846  6249 raft_consensus.cc:493] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:07.131191  6171 raft_consensus.cc:493] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:07.134744  6270 raft_consensus.cc:493] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
W20260709 13:01:07.145538  2301 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:07.102720  6244 consensus_meta.cc:287] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.641s	user 0.014s	sys 0.005s
I20260709 13:01:07.174506  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:07.155455  6293 leader_election.cc:290] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:07.179718  1502 catalog_manager.cc:5697] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 reported cstate change: term changed from 4 to 8. New cstate: current_term: 8 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 13:01:07.171530  1571 connection.cc:256] RPC call timeout handler was delayed by -1.3772688990000006s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 1.5s for a response
I20260709 13:01:07.197427  1643 leader_election.cc:304] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 13:01:07.287479  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 90101.4us
I20260709 13:01:07.287695  2330 raft_consensus.cc:2468] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 9.
I20260709 13:01:07.288813  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:07.288902  1643 leader_election.cc:304] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:01:07.290014  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 13 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:07.290287  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:07.161860  6046 leader_election.cc:290] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 9 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:07.194775  1597 rpcz_store.cc:276] 0709 13:01:05.985578 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:05.987130 (+  1552us) service_pool.cc:224] Handling call
0709 13:01:05.987550 (+   420us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:05.988045 (+   495us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:05.988085 (+    40us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:01:05.988750 (+   665us) log.cc:844] Serialized 26 byte log entry
0709 13:01:06.480094 (+491344us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:01:06.480141 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:06.480245 (+   104us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:07.158930 (+678685us) raft_consensus.cc:1685] finished
0709 13:01:07.158955 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:07.159135 (+   180us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":7234,"raft.run_cpu_time_us":153,"raft.run_wall_time_us":41737,"thread_start_us":411841,"threads_started":1,"wal-append.queue_time_us":651379}
I20260709 13:01:07.200835  6363 raft_consensus.cc:2749] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260709 13:01:07.177827  6369 log.cc:927] Time spent T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.349s	user 0.002s	sys 0.000s
I20260709 13:01:07.268931  1751 raft_consensus.cc:2468] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 13.
I20260709 13:01:07.200846  6249 raft_consensus.cc:3060] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:01:07.158231  6242 raft_consensus.cc:515] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:07.158903  6229 raft_consensus.cc:493] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:01:07.290755  1602 raft_consensus.cc:3060] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:01:07.291352  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:07.292589  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:07.292546  1643 leader_election.cc:304] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 13:01:07.336827  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 12 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
W20260709 13:01:07.337028  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 44551.3us
I20260709 13:01:07.337528  1604 raft_consensus.cc:3060] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:01:07.337852  1749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55848 (request call id 2412) took 2051 ms. Trace:
I20260709 13:01:07.293193  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:07.338233  1749 rpcz_store.cc:276] 0709 13:01:05.286669 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:05.355116 (+ 68447us) service_pool.cc:224] Handling call
0709 13:01:05.355483 (+   367us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:06.632639 (+1277156us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:07.337568 (+704929us) spinlock_profiling.cc:243] Waited 1.27 s on lock 0x7b54001c7d54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:01:07.337776 (+   208us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2923837824}
I20260709 13:01:07.338888  1598 raft_consensus.cc:2468] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 10.
I20260709 13:01:07.340073  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:07.169170  6364 raft_consensus.cc:3060] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:01:07.295253  1671 raft_consensus.cc:1275] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 13 index: 10. (index mismatch)
I20260709 13:01:07.298310  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:07.309996  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:07.159544  6162 raft_consensus.cc:515] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:07.318998  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:07.350551  1643 leader_election.cc:304] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:07.351461  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:07.352905  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:07.269656  1678 raft_consensus.cc:1275] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260709 13:01:07.161450  6271 leader_election.cc:290] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:07.332347  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:07.148073  6360 consensus_queue.cc:237] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 9, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:07.347543  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:07.408582  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:01:07.409538  1569 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.544s	user 0.002s	sys 0.002s
W20260709 13:01:07.409997  1569 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.545s	user 0.003s	sys 0.002s
I20260709 13:01:07.401533  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:07.392511  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:07.417990  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:01:07.418969  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 68577.1us
I20260709 13:01:07.419596  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:07.420997  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:07.421195  1643 leader_election.cc:304] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:01:07.421684  1605 raft_consensus.cc:2468] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 10.
I20260709 13:01:07.423434  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:07.423861  1643 leader_election.cc:304] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:07.301908  6333 raft_consensus.cc:493] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:07.315323  6371 raft_consensus.cc:2749] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:01:07.322141  6290 raft_consensus.cc:2749] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:01:07.444104  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:07.323958  6390 raft_consensus.cc:493] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:07.448709  1749 raft_consensus.cc:1275] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 11 index: 9. Preceding OpId from leader: term: 13 index: 10. (index mismatch)
I20260709 13:01:07.456609  1605 raft_consensus.cc:2468] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 10.
I20260709 13:01:07.464355  2337 raft_consensus.cc:2393] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 11: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:01:07.461987  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:07.467095  1600 raft_consensus.cc:2368] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 12. Current term is 13.
I20260709 13:01:07.286546  6171 raft_consensus.cc:515] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:07.318120  6298 consensus_queue.cc:1048] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.122s
I20260709 13:01:07.299719  6270 raft_consensus.cc:515] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:07.299229  6244 raft_consensus.cc:515] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:07.389370  6293 raft_consensus.cc:493] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:07.471705  6147 leader_election.cc:290] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:01:07.480209  6379 log.cc:927] Time spent T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.340s	user 0.000s	sys 0.001s
W20260709 13:01:07.480801  6383 log.cc:927] Time spent T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.133s	user 0.000s	sys 0.000s
W20260709 13:01:07.480983  6375 log.cc:927] Time spent T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.334s	user 0.000s	sys 0.001s
W20260709 13:01:07.481734  6384 log.cc:927] Time spent T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.167s	user 0.001s	sys 0.000s
W20260709 13:01:07.482301  6382 log.cc:927] Time spent T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.288s	user 0.000s	sys 0.000s
W20260709 13:01:07.489792  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 66020us
I20260709 13:01:07.488076  1749 raft_consensus.cc:1275] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 10 index: 8. (index mismatch)
W20260709 13:01:07.474071  6374 log.cc:927] Time spent T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.494s	user 0.000s	sys 0.002s
W20260709 13:01:07.481734  6385 log.cc:927] Time spent T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.320s	user 0.000s	sys 0.001s
I20260709 13:01:07.543799  1679 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52778 (request call id 2175) took 1079 ms. Trace:
I20260709 13:01:07.548182  1679 rpcz_store.cc:276] 0709 13:01:06.464376 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:06.464561 (+   185us) service_pool.cc:224] Handling call
0709 13:01:06.611570 (+147009us) spinlock_profiling.cc:243] Waited 55.9 ms on lock 0x7b5800086a98. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f426303db71 00007f426303da22 00007f426bf56c09 00007f426b0b0ee0 00007f426b17a0ab 00007f426b17a131 00007f426b13bfc0 00007f426b126929 00007f426b1263d2 00007f426469dbd4
0709 13:01:06.611812 (+   242us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:06.612267 (+   455us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:06.612301 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:01:06.612931 (+   630us) log.cc:844] Serialized 26 byte log entry
0709 13:01:07.043875 (+430944us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:01:07.043919 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:07.044012 (+    93us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:07.543360 (+499348us) raft_consensus.cc:1685] finished
0709 13:01:07.543401 (+    41us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:07.543708 (+   307us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":112,"raft.run_cpu_time_us":137,"raft.run_wall_time_us":138,"spinlock_wait_cycles":128576896,"thread_start_us":430459,"threads_started":1,"wal-append.queue_time_us":552305}
I20260709 13:01:07.406226  6229 raft_consensus.cc:515] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:07.553560  6258 leader_election.cc:290] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:07.559269  6365 leader_election.cc:290] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:07.563081  6397 raft_consensus.cc:2749] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:01:07.462280  6390 raft_consensus.cc:515] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:07.568861  6390 leader_election.cc:290] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:07.565634  6397 raft_consensus.cc:493] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:01:07.569769  6281 raft_consensus.cc:493] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:01:07.570291  6397 raft_consensus.cc:515] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:07.570474  6281 raft_consensus.cc:515] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:07.498597  6290 raft_consensus.cc:2749] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:01:07.580850  1603 raft_consensus.cc:2468] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 10.
I20260709 13:01:07.537737  6363 raft_consensus.cc:2749] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:01:07.583693  1672 raft_consensus.cc:2393] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 8: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
W20260709 13:01:07.507583  6368 log.cc:927] Time spent T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.407s	user 0.001s	sys 0.001s
I20260709 13:01:07.451031  6333 raft_consensus.cc:515] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:01:07.537945  6376 log.cc:927] Time spent T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.446s	user 0.001s	sys 0.001s
W20260709 13:01:07.623682  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 87870.6us
I20260709 13:01:07.588027  1602 raft_consensus.cc:2410] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for term 13 because replica has last-logged OpId of term: 12 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 7.
I20260709 13:01:07.596452  6281 leader_election.cc:290] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:07.603057  6344 raft_consensus.cc:3055] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:01:07.608723  5991 raft_consensus.cc:493] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:07.615566  6029 leader_election.cc:290] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:01:07.617297  6392 log.cc:927] Time spent T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.158s	user 0.000s	sys 0.000s
I20260709 13:01:07.587224  6365 raft_consensus.cc:493] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:07.510022  6293 raft_consensus.cc:515] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:07.645104  1601 raft_consensus.cc:3060] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:01:07.620792  6370 leader_election.cc:290] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:07.643841  1604 raft_consensus.cc:2410] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for term 12 because replica has last-logged OpId of term: 11 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 7.
W20260709 13:01:07.634044  6379 log.cc:927] Time spent T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.094s	user 0.000s	sys 0.001s
I20260709 13:01:07.631937  6357 raft_consensus.cc:493] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:07.663043  1754 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55848 (request call id 2430) took 1277 ms. Trace:
I20260709 13:01:07.633848  6344 raft_consensus.cc:740] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
W20260709 13:01:07.647641  6343 consensus_meta.cc:287] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 1.236s	user 0.015s	sys 0.000s
W20260709 13:01:07.672755  6381 log.cc:927] Time spent T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.565s	user 0.003s	sys 0.001s
I20260709 13:01:07.663570  1754 rpcz_store.cc:276] 0709 13:01:06.385207 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:06.385410 (+   203us) service_pool.cc:224] Handling call
0709 13:01:06.385953 (+   543us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:01:06.386601 (+   648us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:06.386650 (+    49us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:01:06.433304 (+ 46654us) log.cc:844] Serialized 52 byte log entry
0709 13:01:06.912500 (+479196us) raft_consensus.cc:1645] Marking committed up to 3
0709 13:01:06.912616 (+   116us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:06.912763 (+   147us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:07.662749 (+749986us) raft_consensus.cc:1685] finished
0709 13:01:07.662778 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:07.662951 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1767,"raft.run_cpu_time_us":172,"raft.run_wall_time_us":7586,"thread_start_us":397696,"threads_started":1,"wal-append.queue_time_us":687294}
I20260709 13:01:07.640843  5991 raft_consensus.cc:515] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:07.697321  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 71992.6us
I20260709 13:01:07.663599  6354 leader_election.cc:290] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:07.699539  1569 leader_election.cc:304] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:07.652958  6365 raft_consensus.cc:515] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:07.722676  1569 leader_election.cc:304] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:01:07.714288  1601 raft_consensus.cc:2410] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for term 13 because replica has last-logged OpId of term: 12 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 7.
I20260709 13:01:07.668042  6357 raft_consensus.cc:515] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:07.693851  6259 leader_election.cc:290] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:07.757153  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:07.767664  2301 leader_election.cc:304] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:07.672153  6343 raft_consensus.cc:515] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:07.776708  2331 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31"
dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
mode: GRACEFUL
new_leader_uuid: "af520b67945d443b901bbeca37cc5701"
 from {username='slave'} at 127.0.0.1:35790
W20260709 13:01:07.778033  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 55441.9us
I20260709 13:01:07.781582  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:07.786630  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:07.781976  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:07.785012  2335 raft_consensus.cc:3060] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:01:07.780436  2331 raft_consensus.cc:606] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 10 LEADER]: Received request to transfer leadership to TS af520b67945d443b901bbeca37cc5701
I20260709 13:01:07.716442  6390 raft_consensus.cc:2749] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:01:07.719578  6125 leader_election.cc:290] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:07.723630  6162 leader_election.cc:290] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 11 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:01:07.794364  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 198692us
I20260709 13:01:07.727210  6344 consensus_queue.cc:260] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.8, Last appended by leader: 8, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:07.821446  6344 raft_consensus.cc:3060] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:01:07.827548  6259 raft_consensus.cc:493] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:07.828318  6259 raft_consensus.cc:515] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:07.830648  6298 raft_consensus.cc:493] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:07.831007  6261 consensus_queue.cc:1059] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 13:01:07.831211  6298 raft_consensus.cc:515] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:07.795419  6407 raft_consensus.cc:2804] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:01:07.792137  6277 raft_consensus.cc:2804] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:01:07.803411  6171 leader_election.cc:290] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:07.804327  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:07.806650  6408 consensus_queue.cc:1048] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 13:01:07.809207  5991 leader_election.cc:290] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:01:07.809396  6345 consensus_meta.cc:287] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 1.199s	user 0.015s	sys 0.001s
I20260709 13:01:07.812075  6365 leader_election.cc:290] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:07.812959  6397 leader_election.cc:290] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:07.813621  6147 raft_consensus.cc:493] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:07.814735  6244 leader_election.cc:290] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 8 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:01:07.819124  6387 log.cc:927] Time spent T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.120s	user 0.000s	sys 0.000s
W20260709 13:01:07.819207  6350 consensus_meta.cc:287] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 1.344s	user 0.017s	sys 0.000s
I20260709 13:01:07.831815  6261 consensus_queue.cc:1243] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer 3b879799899a499c8d409ee9f1277cd1 log is divergent from this leader: its last log entry 11.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 9
I20260709 13:01:07.834400  6298 leader_election.cc:290] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:07.850361  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:01:07.853163  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 54869.8us
I20260709 13:01:07.852166  6343 leader_election.cc:290] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:07.810083  6270 leader_election.cc:290] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 13:01:07.863247  1644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:07.865414  1745 raft_consensus.cc:1275] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260709 13:01:07.868655  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:07.899714  1678 raft_consensus.cc:2393] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 12: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
W20260709 13:01:07.845258  6332 consensus_meta.cc:287] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 1.402s	user 0.010s	sys 0.004s
W20260709 13:01:07.843336  6337 consensus_meta.cc:287] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.971s	user 0.014s	sys 0.000s
I20260709 13:01:07.871384  1676 raft_consensus.cc:2410] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for term 13 because replica has last-logged OpId of term: 12 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 7.
W20260709 13:01:07.845328  6140 consensus_meta.cc:287] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.568s	user 0.006s	sys 0.003s
I20260709 13:01:07.922039  6140 raft_consensus.cc:515] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:07.922761  6414 consensus_queue.cc:1048] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 7, Time since last communication: 0.004s
I20260709 13:01:07.924320  6333 leader_election.cc:290] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:07.924541  6140 leader_election.cc:290] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 16 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:07.851349  6407 raft_consensus.cc:493] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:07.925379  6407 raft_consensus.cc:3060] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:01:07.925235  6365 consensus_queue.cc:1048] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 13:01:07.801127  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 12 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:07.928854  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:07.930047  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:07.878170  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:07.934331  6293 leader_election.cc:290] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:07.935555  6259 leader_election.cc:290] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:07.879798  5989 raft_consensus.cc:2804] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:01:07.884364  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:07.852813  6277 raft_consensus.cc:493] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:07.781070  6410 raft_consensus.cc:493] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:01:07.943754  6410 raft_consensus.cc:515] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:07.882483  6390 raft_consensus.cc:493] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:07.945793  6390 raft_consensus.cc:515] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:07.946559  6277 raft_consensus.cc:3060] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Advancing to term 12
W20260709 13:01:07.888589  6160 consensus_meta.cc:287] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.789s	user 0.013s	sys 0.000s
I20260709 13:01:07.948184  6390 leader_election.cc:290] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:07.948475  6160 raft_consensus.cc:515] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:07.897756  1675 raft_consensus.cc:2393] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 9: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:01:07.883561  6350 raft_consensus.cc:515] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:07.951884  6176 raft_consensus.cc:3055] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:01:07.951936  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 15 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:07.952509  6176 raft_consensus.cc:740] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:01:07.880656  6147 raft_consensus.cc:515] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:07.904698  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:07.953440  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:07.873039  5991 raft_consensus.cc:493] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:07.920068  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:07.851329  2332 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49082 (request call id 912) took 2705 ms. Trace:
I20260709 13:01:07.967733  2330 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39738 (request call id 900) took 1080 ms. Trace:
I20260709 13:01:07.942066  1750 raft_consensus.cc:2468] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 9.
I20260709 13:01:07.969378  1677 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52778 (request call id 2181) took 1348 ms. Trace:
I20260709 13:01:07.969849  1677 rpcz_store.cc:276] 0709 13:01:06.621119 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:06.621289 (+   170us) service_pool.cc:224] Handling call
0709 13:01:07.969159 (+1347870us) spinlock_profiling.cc:243] Waited 196 ms on lock 0x7b54000230d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:01:07.969294 (+   135us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":449819008}
I20260709 13:01:07.971371  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:07.854091  6249 raft_consensus.cc:515] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:07.942683  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:07.886834  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:07.955291  1598 raft_consensus.cc:2393] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 9: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:01:07.954898  1748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55848 (request call id 2431) took 1410 ms. Trace:
I20260709 13:01:08.084854  1674 raft_consensus.cc:2468] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 12.
I20260709 13:01:07.956684  1646 leader_election.cc:304] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
W20260709 13:01:08.087591  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 130980us
I20260709 13:01:08.088678  1646 leader_election.cc:304] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
W20260709 13:01:08.088766  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 125589us
I20260709 13:01:07.955857  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 15 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:07.960649  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:07.959847  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:01:08.077877  1644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:07.968236  2330 rpcz_store.cc:276] 0709 13:01:06.887434 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:06.887618 (+   184us) service_pool.cc:224] Handling call
0709 13:01:07.967507 (+1079889us) spinlock_profiling.cc:243] Waited 385 ms on lock 0x7b540072e554. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:01:07.967645 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":885186816}
I20260709 13:01:08.090513  1678 raft_consensus.cc:2468] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 13.
I20260709 13:01:07.966578  2332 rpcz_store.cc:276] 0709 13:01:05.146093 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:05.147438 (+  1345us) service_pool.cc:224] Handling call
0709 13:01:05.147858 (+   420us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:01:05.203355 (+ 55497us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 13:01:06.155909 (+952554us) log.cc:844] Serialized 14 byte log entry
0709 13:01:07.141973 (+986064us) log.cc:844] Serialized 14 byte log entry
0709 13:01:07.147617 (+  5644us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:01:07.148380 (+   763us) log.cc:844] Serialized 52 byte log entry
0709 13:01:07.148610 (+   230us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:01:07.148841 (+   231us) log.cc:844] Serialized 14 byte log entry
0709 13:01:07.149573 (+   732us) log.cc:844] Serialized 14 byte log entry
0709 13:01:07.157188 (+  7615us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:07.157333 (+   145us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:07.612095 (+454762us) spinlock_profiling.cc:243] Waited 6.54 ms on lock 0x7b18002b9900. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f42630753f3 00007f42630777c1 00007f4263077779 00007f4263077709 00007f42630776d1 00007f42630769cd 000000000038f354 000000000038f199 00007f426305883a
0709 13:01:07.851032 (+238937us) raft_consensus.cc:1685] finished
0709 13:01:07.851061 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:07.851228 (+   167us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":55782,"prepare.queue_time_us":1894705,"prepare.run_cpu_time_us":255,"prepare.run_wall_time_us":47127,"raft.queue_time_us":83768,"raft.run_cpu_time_us":618,"raft.run_wall_time_us":64749,"spinlock_wait_cycles":25978624,"thread_start_us":1728946,"threads_started":3,"wal-append.queue_time_us":733605}
I20260709 13:01:08.082526  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:08.115722  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:08.116108  1677 raft_consensus.cc:2393] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 12: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:01:08.116771  1674 raft_consensus.cc:2468] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 12.
W20260709 13:01:08.117038  6364 consensus_meta.cc:287] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.672s	user 0.012s	sys 0.000s
I20260709 13:01:08.117568  6364 raft_consensus.cc:515] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:08.117866  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:08.117710  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:08.120282  1674 raft_consensus.cc:2468] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 9.
I20260709 13:01:08.119053  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:08.121697  1674 raft_consensus.cc:1275] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 11 index: 6. (term mismatch)
I20260709 13:01:08.122785  1674 pending_rounds.cc:85] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d: Aborting all ops after (but not including) 5
I20260709 13:01:08.123076  1674 pending_rounds.cc:107] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d: Aborting uncommitted NO_OP operation due to leader change: 9.6
I20260709 13:01:08.123318  1674 raft_consensus.cc:2889] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:01:08.123770  6160 leader_election.cc:290] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 9 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:08.123664  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:08.124647  2332 raft_consensus.cc:2468] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 11.
I20260709 13:01:08.124990  1674 raft_consensus.cc:1240] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 9. Current term is 10. Ops: []
I20260709 13:01:07.953543  6350 leader_election.cc:290] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 9 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:08.126111  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:08.126407  5991 raft_consensus.cc:515] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:08.126920  2332 raft_consensus.cc:2468] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 10.
I20260709 13:01:08.130586  1599 raft_consensus.cc:3055] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:01:08.121618  1677 raft_consensus.cc:1275] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 11 index: 8. (index mismatch)
I20260709 13:01:08.132493  1599 raft_consensus.cc:740] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:01:08.132566  1569 leader_election.cc:304] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:01:08.133649  1569 leader_election.cc:304] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:01:08.133507  1599 consensus_queue.cc:260] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.10, Last appended by leader: 10, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:08.134722  6140 raft_consensus.cc:2804] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:01:08.135360  6140 raft_consensus.cc:493] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:08.135865  6140 raft_consensus.cc:3060] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:01:08.136020  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:08.128085  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 11 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:08.137877  2332 raft_consensus.cc:2468] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 10.
I20260709 13:01:08.138620  6125 raft_consensus.cc:493] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:08.139343  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:08.139184  6125 raft_consensus.cc:515] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:08.134825  1599 raft_consensus.cc:3060] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Advancing to term 11
W20260709 13:01:08.142270  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 53712.9us
I20260709 13:01:08.143203  1646 leader_election.cc:304] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 13:01:08.085165  1748 rpcz_store.cc:276] 0709 13:01:06.544409 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:06.544601 (+   192us) service_pool.cc:224] Handling call
0709 13:01:06.545293 (+   692us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:01:06.640687 (+ 95394us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:06.640720 (+    33us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:01:06.641205 (+   485us) log.cc:844] Serialized 52 byte log entry
0709 13:01:07.092788 (+451583us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:01:07.092836 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:07.092924 (+    88us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:07.954628 (+861704us) raft_consensus.cc:1685] finished
0709 13:01:07.954656 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:07.954822 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":4883,"raft.run_cpu_time_us":156,"raft.run_wall_time_us":7495,"thread_start_us":402886,"threads_started":1,"wal-append.queue_time_us":974701}
I20260709 13:01:08.119208  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:01:08.148356  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 58619us
I20260709 13:01:08.161670  1674 raft_consensus.cc:1240] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 8. Current term is 9. Ops: []
I20260709 13:01:08.142185  6171 raft_consensus.cc:2749] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:01:08.189862  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
W20260709 13:01:08.194061  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 227711us
I20260709 13:01:08.200146  1596 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 1013) took 2186 ms. Trace:
I20260709 13:01:07.953639  5989 raft_consensus.cc:493] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:07.946322  6410 leader_election.cc:290] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:08.084559  6343 raft_consensus.cc:493] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:08.200857  1596 rpcz_store.cc:276] 0709 13:01:05.986207 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:05.986338 (+   131us) service_pool.cc:224] Handling call
0709 13:01:05.993922 (+  7584us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:06.745643 (+751721us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:08.172702 (+1427059us) spinlock_profiling.cc:243] Waited 734 ms on lock 0x7b54000acb54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:01:08.172970 (+   268us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1688535424}
I20260709 13:01:08.083873  6293 raft_consensus.cc:493] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:01:07.939065  6337 raft_consensus.cc:515] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:08.274958  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:08.128990  5991 leader_election.cc:290] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:01:08.235064  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 72578.4us
I20260709 13:01:08.238766  1751 raft_consensus.cc:1240] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 11. Current term is 12. Ops: []
I20260709 13:01:08.235606  6147 leader_election.cc:290] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:01:08.240545  1718 connection.cc:256] RPC call timeout handler was delayed by -1.0501984890000244s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2868s for a response
I20260709 13:01:08.279031  1718 leader_election.cc:304] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:08.250842  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:08.162781  6350 raft_consensus.cc:2804] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:01:08.161031  6363 raft_consensus.cc:2804] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:01:08.273466  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 11 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:08.173056  6371 raft_consensus.cc:2804] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:01:08.287052  1596 raft_consensus.cc:2393] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 12: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:01:08.290385  1674 raft_consensus.cc:2468] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 10.
I20260709 13:01:08.155706  6414 consensus_queue.cc:1048] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.101s
I20260709 13:01:08.293056  6371 raft_consensus.cc:493] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:08.291826  6281 consensus_queue.cc:1048] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.006s
I20260709 13:01:08.320588  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:08.320803  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:08.321143  6147 consensus_queue.cc:1059] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
W20260709 13:01:08.328262  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 58430.9us
I20260709 13:01:08.236277  5989 raft_consensus.cc:3060] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:01:08.348695  1746 raft_consensus.cc:2468] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 10.
I20260709 13:01:08.349143  1674 raft_consensus.cc:2393] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 13: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:01:08.334911  2301 leader_election.cc:304] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:07.953577  6176 consensus_queue.cc:260] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:08.371313  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:08.372287  2336 raft_consensus.cc:3060] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:01:08.233855  6332 raft_consensus.cc:493] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
W20260709 13:01:08.364655  1746 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55840 (request call id 2552) took 1540 ms (client timeout 1989 ms). Trace:
W20260709 13:01:08.384099  1746 rpcz_store.cc:269] 0709 13:01:06.823763 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:06.859380 (+ 35617us) service_pool.cc:224] Handling call
0709 13:01:08.320374 (+1460994us) spinlock_profiling.cc:243] Waited 5.31 ms on lock 0x7b58001ffe98. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f426303de9c 00007f426303da5b 00007f426bf56c09 00007f426b0b0ee0 00007f426b17a0ab 00007f426b17a131 00007f426b1280a0 00007f426b127984 00007f42646a26d4 00007f42646a2662
0709 13:01:08.364551 (+ 44177us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":12226560}
I20260709 13:01:08.368372  1599 raft_consensus.cc:1275] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 11 index: 10. (term mismatch)
I20260709 13:01:08.385818  1599 pending_rounds.cc:85] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a: Aborting all ops after (but not including) 9
I20260709 13:01:08.386161  1599 pending_rounds.cc:107] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a: Aborting uncommitted NO_OP operation due to leader change: 10.10
I20260709 13:01:08.386508  1599 raft_consensus.cc:2889] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:01:08.334365  1602 raft_consensus.cc:1240] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 10. Current term is 11. Ops: []
I20260709 13:01:08.295985  6245 leader_election.cc:290] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:08.293545  6270 raft_consensus.cc:493] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:08.389928  6270 raft_consensus.cc:515] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:08.392584  6344 consensus_meta.cc:287] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.571s	user 0.012s	sys 0.000s
I20260709 13:01:08.393007  6270 leader_election.cc:290] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:08.393694  6270 raft_consensus.cc:2804] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:01:08.394049  6270 raft_consensus.cc:493] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:08.394393  6270 raft_consensus.cc:3060] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:01:08.290114  6363 raft_consensus.cc:493] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:01:08.378295  1672 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52766 (request call id 2605) took 1390 ms. Trace:
I20260709 13:01:08.408349  1672 rpcz_store.cc:276] 0709 13:01:06.988025 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:06.988331 (+   306us) service_pool.cc:224] Handling call
0709 13:01:08.377936 (+1389605us) spinlock_profiling.cc:243] Waited 590 ms on lock 0x7b54001f6654. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:01:08.378207 (+   271us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1357214464}
I20260709 13:01:08.409756  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:08.381150  1748 raft_consensus.cc:1240] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 11. Current term is 12. Ops: [11.8-11.8]
I20260709 13:01:08.389103  2329 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39738 (request call id 903) took 1499 ms. Trace:
W20260709 13:01:08.389354  6029 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 11 current term: 11 is higher.
I20260709 13:01:08.412138  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:08.412854  1748 raft_consensus.cc:3060] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:01:08.395299  1747 raft_consensus.cc:1240] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 12. Current term is 13. Ops: [10.7-12.8]
I20260709 13:01:08.396377  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:08.396791  1675 raft_consensus.cc:2468] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 13.
I20260709 13:01:08.414312  1754 raft_consensus.cc:2468] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 12.
I20260709 13:01:08.411247  2329 rpcz_store.cc:276] 0709 13:01:06.889877 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:06.890112 (+   235us) service_pool.cc:224] Handling call
0709 13:01:08.378978 (+1488866us) spinlock_profiling.cc:243] Waited 1.27 s on lock 0x7b54005f3a54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:01:08.389017 (+ 10039us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2919848192}
I20260709 13:01:08.398422  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:08.415866  1679 raft_consensus.cc:2468] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 11.
I20260709 13:01:08.246080  6333 consensus_queue.cc:1059] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 9.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.109s
I20260709 13:01:08.415489  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:01:08.356110  6412 log.cc:927] Time spent T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.082s	user 0.001s	sys 0.000s
I20260709 13:01:08.401249  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:08.422480  1746 raft_consensus.cc:3060] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:01:08.290343  6350 raft_consensus.cc:493] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:08.414844  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:08.397013  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:01:08.365203  6189 consensus_meta.cc:287] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 1.219s	user 0.016s	sys 0.000s
I20260709 13:01:08.344566  6371 raft_consensus.cc:3060] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:01:08.272982  6394 consensus_queue.cc:1048] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.107s
I20260709 13:01:08.281056  6343 raft_consensus.cc:515] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:01:08.420265  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 85444.8us
W20260709 13:01:08.421896  2335 consensus_meta.cc:287] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.625s	user 0.012s	sys 0.000s
I20260709 13:01:08.426187  1749 raft_consensus.cc:2393] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 12: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:01:08.435369  2301 leader_election.cc:304] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:01:08.435598  2335 raft_consensus.cc:2468] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 12.
I20260709 13:01:08.427057  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:08.436331  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 12 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:08.436581  1754 raft_consensus.cc:2468] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 11.
I20260709 13:01:08.437783  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 11 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:08.443081  1749 raft_consensus.cc:2410] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for term 12 because replica has last-logged OpId of term: 9 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 6.
I20260709 13:01:08.443653  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 12 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:08.281852  6293 raft_consensus.cc:515] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:08.379635  6408 consensus_queue.cc:1048] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.039s
I20260709 13:01:08.444577  1674 raft_consensus.cc:2468] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 11.
W20260709 13:01:08.464578  1644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:08.274076  6418 consensus_queue.cc:1048] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.192s
I20260709 13:01:08.413062  6363 raft_consensus.cc:3060] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:01:08.465843  2336 raft_consensus.cc:2468] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 9.
I20260709 13:01:08.488062  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:08.441793  6125 leader_election.cc:290] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:08.490020  1747 raft_consensus.cc:2468] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 9.
I20260709 13:01:08.429491  6350 raft_consensus.cc:3060] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:01:08.442484  6162 raft_consensus.cc:2804] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:01:08.506968  6162 raft_consensus.cc:493] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:08.507179  1680 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57092 (request call id 1004) took 1328 ms. Trace:
I20260709 13:01:08.507444  6162 raft_consensus.cc:3060] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:01:08.507627  1680 rpcz_store.cc:276] 0709 13:01:07.178916 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:07.179062 (+   146us) service_pool.cc:224] Handling call
0709 13:01:07.262265 (+ 83203us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:07.274205 (+ 11940us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:07.274268 (+    63us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:01:07.274335 (+    67us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:01:07.274402 (+    67us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:08.506879 (+1232477us) spinlock_profiling.cc:243] Waited 11.6 ms on lock 0x7b54000d7d54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:01:08.506938 (+    59us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:08.507102 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":26698624}
I20260709 13:01:08.510382  1671 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57092 (request call id 1010) took 1195 ms. Trace:
I20260709 13:01:08.511005  1753 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43156 (request call id 897) took 1641 ms. Trace:
I20260709 13:01:08.511461  1753 rpcz_store.cc:276] 0709 13:01:06.870105 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:06.873772 (+  3667us) service_pool.cc:224] Handling call
0709 13:01:06.916359 (+ 42587us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:06.990078 (+ 73719us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:08.510709 (+1520631us) spinlock_profiling.cc:243] Waited 72.7 ms on lock 0x7b54001d5bd4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:01:08.510930 (+   221us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":167237376}
W20260709 13:01:08.502045  2337 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39738 (request call id 890) took 2190 ms (client timeout 2868 ms). Trace:
I20260709 13:01:08.467283  6154 raft_consensus.cc:2749] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:01:08.503964  6082 raft_consensus.cc:3055] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 LEADER]: Stepping down as leader of term 11
I20260709 13:01:08.504220  2330 raft_consensus.cc:1240] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 10. Current term is 11. Ops: [10.10-10.10]
W20260709 13:01:08.519822  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 388841us
I20260709 13:01:08.505961  6373 raft_consensus.cc:3055] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 LEADER]: Stepping down as leader of term 11
I20260709 13:01:08.518631  1680 raft_consensus.cc:1275] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260709 13:01:08.505234  2334 raft_consensus.cc:1275] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 10 index: 8. (index mismatch)
W20260709 13:01:08.492204  6251 consensus_meta.cc:287] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 1.328s	user 0.014s	sys 0.003s
I20260709 13:01:08.533113  1644 leader_election.cc:304] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
W20260709 13:01:08.513034  2337 rpcz_store.cc:269] 0709 13:01:06.311270 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:06.316216 (+  4946us) service_pool.cc:224] Handling call
0709 13:01:08.501640 (+2185424us) spinlock_profiling.cc:243] Waited 1.05 s on lock 0x7b540045bfd4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:01:08.501831 (+   191us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2409210112}
I20260709 13:01:08.524608  2299 leader_election.cc:304] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:01:08.510915  1671 rpcz_store.cc:276] 0709 13:01:07.315603 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:07.394608 (+ 79005us) service_pool.cc:224] Handling call
0709 13:01:07.395018 (+   410us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:07.500357 (+105339us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:07.500393 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:01:07.511652 (+ 11259us) log.cc:844] Serialized 26 byte log entry
0709 13:01:07.856243 (+344591us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:01:07.857480 (+  1237us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:08.509879 (+652399us) spinlock_profiling.cc:243] Waited 81.3 ms on lock 0x7b5400164754. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:01:08.509944 (+    65us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:08.510001 (+    57us) raft_consensus.cc:1685] finished
0709 13:01:08.510020 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:08.510309 (+   289us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":23228,"raft.queue_time_us":87057,"raft.run_cpu_time_us":114,"raft.run_wall_time_us":114,"spinlock_wait_cycles":186957056,"thread_start_us":310025,"threads_started":1,"wal-append.queue_time_us":623279}
I20260709 13:01:08.537201  2299 leader_election.cc:304] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:01:08.524174  1746 raft_consensus.cc:2468] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 8.
I20260709 13:01:08.526079  1748 raft_consensus.cc:2468] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 10.
W20260709 13:01:08.529080  1673 consensus_meta.cc:287] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.608s	user 0.012s	sys 0.000s
I20260709 13:01:08.540714  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:01:08.505993  6421 log.cc:927] Time spent T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.135s	user 0.000s	sys 0.001s
I20260709 13:01:08.544288  6293 leader_election.cc:290] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:08.545027  6337 leader_election.cc:290] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:08.425215  6332 raft_consensus.cc:515] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:08.554728  6364 leader_election.cc:290] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 12 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:08.522682  6373 raft_consensus.cc:740] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:01:08.556689  6357 leader_election.cc:290] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:08.556620  6373 consensus_queue.cc:260] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 11.8, Last appended by leader: 8, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:08.558075  6373 raft_consensus.cc:3060] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Advancing to term 12
W20260709 13:01:08.559381  6422 log.cc:927] Time spent T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.069s	user 0.000s	sys 0.000s
I20260709 13:01:08.433568  6189 raft_consensus.cc:515] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:08.556859  6251 raft_consensus.cc:2749] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:01:08.542244  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 11 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:08.539945  1673 raft_consensus.cc:2468] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 13.
I20260709 13:01:08.575184  1604 raft_consensus.cc:1275] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 11 index: 9. Preceding OpId from leader: term: 13 index: 10. (index mismatch)
I20260709 13:01:08.585106  2328 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39738 (request call id 904) took 1694 ms. Trace:
W20260709 13:01:08.586746  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 307790us
I20260709 13:01:08.591835  1718 leader_election.cc:304] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:08.601082  1717 leader_election.cc:304] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:08.556725  6332 leader_election.cc:290] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:08.604302  6140 raft_consensus.cc:515] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:08.605764  1600 raft_consensus.cc:1240] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 12. Current term is 13. Ops: [12.11-12.11]
I20260709 13:01:08.607956  2298 leader_election.cc:304] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:01:08.609843  6201 raft_consensus.cc:2804] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:01:08.610332  6201 raft_consensus.cc:493] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:08.610730  6201 raft_consensus.cc:3060] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:01:08.614370  5994 raft_consensus.cc:2749] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:01:08.615007  6046 raft_consensus.cc:493] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:08.615753  6046 raft_consensus.cc:515] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:08.618230  5938 leader_election.cc:290] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 12 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:08.522667  6082 raft_consensus.cc:740] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:01:08.623597  1752 raft_consensus.cc:1240] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 11. Current term is 12. Ops: []
I20260709 13:01:08.624981  1679 raft_consensus.cc:1240] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 13. Current term is 14. Ops: []
W20260709 13:01:08.626420  1601 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40154 (request call id 2267) took 1300 ms (client timeout 1689 ms). Trace:
W20260709 13:01:08.627079  1601 rpcz_store.cc:269] 0709 13:01:07.288996 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:07.290120 (+  1124us) service_pool.cc:224] Handling call
0709 13:01:08.589008 (+1298888us) spinlock_profiling.cc:243] Waited 253 ms on lock 0x7b54002990d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:01:08.589250 (+   242us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":582371712}
I20260709 13:01:08.575250  6365 raft_consensus.cc:2804] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:01:08.624603  1718 leader_election.cc:304] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
W20260709 13:01:08.614259  6277 consensus_meta.cc:287] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.667s	user 0.009s	sys 0.001s
I20260709 13:01:08.585891  2328 rpcz_store.cc:276] 0709 13:01:06.890240 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:06.890374 (+   134us) service_pool.cc:224] Handling call
0709 13:01:06.890750 (+   376us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:07.198860 (+308110us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 13:01:07.448091 (+249231us) log.cc:844] Serialized 14 byte log entry
0709 13:01:07.630408 (+182317us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:01:07.630516 (+   108us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:01:07.630615 (+    99us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:08.584707 (+954092us) spinlock_profiling.cc:243] Waited 308 ms on lock 0x7b54005802d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:01:08.584791 (+    84us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:08.585002 (+   211us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":28625,"prepare.queue_time_us":232842,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":55,"raft.queue_time_us":2720,"raft.run_cpu_time_us":161,"raft.run_wall_time_us":7978,"spinlock_wait_cycles":707827456,"thread_start_us":367571,"threads_started":2,"wal-append.queue_time_us":295942}
W20260709 13:01:08.635403  1603 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40154 (request call id 2268) took 1346 ms (client timeout 1614 ms). Trace:
I20260709 13:01:08.578702  6249 leader_election.cc:290] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:08.637509  6360 raft_consensus.cc:993] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1: : Instructing follower af520b67945d443b901bbeca37cc5701 to start an election
I20260709 13:01:08.636436  6365 raft_consensus.cc:493] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:08.571355  6363 raft_consensus.cc:515] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:01:08.639549  1603 rpcz_store.cc:269] 0709 13:01:07.289293 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:07.291086 (+  1793us) service_pool.cc:224] Handling call
0709 13:01:08.635182 (+1344096us) spinlock_profiling.cc:243] Waited 245 ms on lock 0x7b54001027d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:01:08.635328 (+   146us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":563463040}
I20260709 13:01:08.636859  1718 leader_election.cc:304] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:08.663666  6293 raft_consensus.cc:2804] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:01:08.664223  6293 raft_consensus.cc:493] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:08.664677  6293 raft_consensus.cc:3060] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:01:08.666014  1718 leader_election.cc:304] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:08.669885  1718 leader_election.cc:304] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:08.654917  6249 raft_consensus.cc:2804] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:01:08.656190  6365 raft_consensus.cc:3060] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:01:08.656061  6408 raft_consensus.cc:1081] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 10 LEADER]: Signalling peer af520b67945d443b901bbeca37cc5701 to start an election
I20260709 13:01:08.640671  6277 raft_consensus.cc:515] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:08.661868  6140 leader_election.cc:290] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:08.710813  6249 raft_consensus.cc:493] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:08.707612  6082 consensus_queue.cc:260] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:01:08.714210  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 430303us
I20260709 13:01:08.723678  1596 raft_consensus.cc:1240] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 11. Current term is 12. Ops: []
I20260709 13:01:08.859519  1571 leader_election.cc:304] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
W20260709 13:01:08.868054  1569 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:08.871449  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 374089us
I20260709 13:01:08.868726  1571 leader_election.cc:304] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
W20260709 13:01:08.882298  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 212352us
I20260709 13:01:08.733955  1747 raft_consensus.cc:1275] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 11 index: 9. Preceding OpId from leader: term: 13 index: 10. (index mismatch)
I20260709 13:01:08.891898  1718 leader_election.cc:304] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:08.742903  6270 raft_consensus.cc:515] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:08.765419  6189 leader_election.cc:290] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 12 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:08.789341  6350 raft_consensus.cc:515] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:08.897508  6189 raft_consensus.cc:2804] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:01:08.898077  6189 raft_consensus.cc:493] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:08.786135  6082 raft_consensus.cc:3060] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Advancing to term 12
W20260709 13:01:08.898891  6271 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.320s	user 0.008s	sys 0.028s
W20260709 13:01:08.899767  6271 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.360s	user 0.008s	sys 0.028s
I20260709 13:01:08.736413  6249 raft_consensus.cc:3060] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Advancing to term 12
W20260709 13:01:08.782717  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 173314us
I20260709 13:01:08.790257  6277 leader_election.cc:290] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 12 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:08.908756  1753 raft_consensus.cc:1240] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 13. Current term is 14. Ops: [13.10-13.10]
I20260709 13:01:08.743633  6162 raft_consensus.cc:515] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:08.911450  6232 raft_consensus.cc:2804] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:01:08.912353  6232 raft_consensus.cc:493] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:08.912958  6232 raft_consensus.cc:3060] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:01:08.756474  6343 leader_election.cc:290] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:01:08.764585  6290 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.184s	user 0.011s	sys 0.029s
I20260709 13:01:08.898267  5989 raft_consensus.cc:515] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:08.917033  6290 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.344s	user 0.011s	sys 0.030s
I20260709 13:01:08.916941  6357 raft_consensus.cc:2749] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:01:08.917433  6189 raft_consensus.cc:3060] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:01:08.923732  6270 leader_election.cc:290] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 14 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:08.882229  1569 leader_election.cc:304] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:08.885519  1571 leader_election.cc:304] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:01:08.938403  2298 leader_election.cc:304] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:01:08.940109  1675 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52766 (request call id 2625) took 1006 ms. Trace:
I20260709 13:01:08.940737  1675 rpcz_store.cc:276] 0709 13:01:07.933234 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:07.960404 (+ 27170us) service_pool.cc:224] Handling call
0709 13:01:08.939505 (+979101us) spinlock_profiling.cc:243] Waited 307 ms on lock 0x7b5400162e54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:01:08.939794 (+   289us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":705266048}
I20260709 13:01:08.941813  6353 raft_consensus.cc:2749] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 13:01:08.943853  6363 leader_election.cc:290] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:08.947564  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:08.947464  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:08.948246  1679 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57092 (request call id 1017) took 1016 ms. Trace:
W20260709 13:01:08.954315  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 48224.5us
W20260709 13:01:08.933969  1673 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57092 (request call id 1003) took 1770 ms (client timeout 1924 ms). Trace:
W20260709 13:01:08.958313  1673 rpcz_store.cc:269] 0709 13:01:07.161772 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:07.174159 (+ 12387us) service_pool.cc:224] Handling call
0709 13:01:08.925465 (+1751306us) spinlock_profiling.cc:243] Waited 644 ms on lock 0x7b5400347e54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:01:08.931777 (+  6312us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1481856896}
W20260709 13:01:08.959862  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 556125us
I20260709 13:01:08.950222  6350 leader_election.cc:290] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:08.951531  6162 leader_election.cc:290] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 10 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:08.956230  6140 raft_consensus.cc:2749] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260709 13:01:08.806311  6407 consensus_meta.cc:287] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.880s	user 0.013s	sys 0.000s
I20260709 13:01:08.957741  1679 rpcz_store.cc:276] 0709 13:01:07.931780 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:08.436346 (+504566us) service_pool.cc:224] Handling call
0709 13:01:08.436664 (+   318us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:08.625543 (+188879us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:08.947296 (+321753us) spinlock_profiling.cc:243] Waited 188 ms on lock 0x7b54001f5c54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:01:08.947924 (+   628us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":432365440}
I20260709 13:01:08.946012  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:08.942018  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 12 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:01:08.980294  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 95142.1us
I20260709 13:01:08.801117  6442 raft_consensus.cc:2749] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20260709 13:01:08.988515  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 455459us
I20260709 13:01:08.740320  6428 raft_consensus.cc:2749] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:01:08.992110  1677 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52766 (request call id 2620) took 1122 ms. Trace:
I20260709 13:01:08.993208  2298 leader_election.cc:304] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:08.993505  1752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55840 (request call id 2549) took 2176 ms. Trace:
I20260709 13:01:08.996162  6438 consensus_queue.cc:1059] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 8.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 13:01:08.997480  6140 raft_consensus.cc:2749] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260709 13:01:09.015626  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 133716us
I20260709 13:01:09.021526  1751 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55848 (request call id 2438) took 2044 ms. Trace:
I20260709 13:01:09.022398  1751 rpcz_store.cc:276] 0709 13:01:06.971502 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:07.409071 (+437569us) service_pool.cc:224] Handling call
0709 13:01:07.409388 (+   317us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:08.277445 (+868057us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:09.003613 (+726168us) spinlock_profiling.cc:243] Waited 794 ms on lock 0x7b54000e4054. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:01:09.016025 (+ 12412us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1825643904}
I20260709 13:01:09.027854  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:08.993984  1752 rpcz_store.cc:276] 0709 13:01:06.816802 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:06.816971 (+   169us) service_pool.cc:224] Handling call
0709 13:01:06.817326 (+   355us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:08.624259 (+1806933us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:08.992865 (+368606us) spinlock_profiling.cc:243] Waited 1.8 s on lock 0x7b54000e4f54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:01:08.993253 (+   388us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4148104064}
W20260709 13:01:09.042409  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 150808us
I20260709 13:01:08.998329  6438 raft_consensus.cc:2749] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:01:08.999486  6354 consensus_queue.cc:1059] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 13.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.053s
I20260709 13:01:09.048475  6354 raft_consensus.cc:3055] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 13:01:09.049003  6354 raft_consensus.cc:740] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:01:08.998999  6408 raft_consensus.cc:3055] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:01:09.050727  6408 raft_consensus.cc:740] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:01:08.992699  1677 rpcz_store.cc:276] 0709 13:01:07.869460 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:08.133419 (+263959us) service_pool.cc:224] Handling call
0709 13:01:08.133780 (+   361us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:08.134183 (+   403us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 13:01:08.198561 (+ 64378us) log.cc:844] Serialized 14 byte log entry
0709 13:01:08.364113 (+165552us) log.cc:844] Serialized 14 byte log entry
0709 13:01:08.949392 (+585279us) spinlock_profiling.cc:243] Waited 11.8 ms on lock 0x7b1800394b40. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f4262fefc0a 00007f426c72f219 00007f4263077a40 00007f42630724e4 00007f4263072455 00007f426615a060 00007f426616c716 00007f426617d287 00007f426617d208
0709 13:01:08.974527 (+ 25135us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:01:08.991306 (+ 16779us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:01:08.991433 (+   127us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:08.991605 (+   172us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:08.991845 (+   240us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":18733,"prepare.queue_time_us":1089353,"prepare.run_cpu_time_us":75,"prepare.run_wall_time_us":76,"raft.queue_time_us":4710,"raft.run_cpu_time_us":211,"raft.run_wall_time_us":27463,"spinlock_wait_cycles":27077504,"thread_start_us":164681,"threads_started":1,"wal-append.queue_time_us":347143}
I20260709 13:01:09.053169  2330 raft_consensus.cc:2393] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 12: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:01:09.052383  6408 consensus_queue.cc:260] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:09.057353  6408 raft_consensus.cc:3060] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:01:09.051010  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:09.059471  6069 raft_consensus.cc:2749] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:01:09.062868  5989 leader_election.cc:290] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 13 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:09.064782  6171 raft_consensus.cc:493] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:09.065665  6171 raft_consensus.cc:515] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:09.066663  6442 raft_consensus.cc:493] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:09.067538  6442 raft_consensus.cc:515] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:09.001005  6407 raft_consensus.cc:515] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:09.071201  1718 leader_election.cc:304] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:09.074429  1646 leader_election.cc:304] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:09.075065  2333 raft_consensus.cc:2368] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 16. Current term is 17.
I20260709 13:01:09.073618  6407 leader_election.cc:290] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 17 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:08.765652  6420 raft_consensus.cc:2804] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:01:09.051167  1569 leader_election.cc:304] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:01:09.093024  6428 raft_consensus.cc:493] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:09.093474  6029 raft_consensus.cc:2804] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Leader election won for term 9
I20260709 13:01:09.094350  6147 raft_consensus.cc:3055] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:01:09.094200  6428 raft_consensus.cc:515] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:09.094990  6147 raft_consensus.cc:740] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:01:09.095357  1571 leader_election.cc:304] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:09.096419  6147 consensus_queue.cc:260] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.7, Last appended by leader: 7, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:09.085857  6442 leader_election.cc:290] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:09.089556  6270 raft_consensus.cc:3060] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:01:09.099808  6029 raft_consensus.cc:697] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:01:09.076464  6293 raft_consensus.cc:515] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:01:09.119890  1569 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:09.105492  6029 consensus_queue.cc:237] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:09.123310  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 79107.8us
I20260709 13:01:09.125386  6140 raft_consensus.cc:493] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:09.126579  1598 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40154 (request call id 2284) took 1802 ms. Trace:
I20260709 13:01:09.126753  6298 raft_consensus.cc:2804] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:01:09.127310  6298 raft_consensus.cc:493] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:09.127948  6298 raft_consensus.cc:3060] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:01:09.118067  2335 raft_consensus.cc:2468] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 14.
I20260709 13:01:09.126394  6140 raft_consensus.cc:515] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:09.143568  1750 raft_consensus.cc:1240] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 8. Current term is 9. Ops: []
W20260709 13:01:09.143877  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 72931.6us
I20260709 13:01:09.139495  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:09.145812  1751 raft_consensus.cc:3060] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:01:09.139904  1679 raft_consensus.cc:1240] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 9. Current term is 10. Ops: []
I20260709 13:01:09.127182  1598 rpcz_store.cc:276] 0709 13:01:07.324234 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:07.423138 (+ 98904us) service_pool.cc:224] Handling call
0709 13:01:09.125833 (+1702695us) spinlock_profiling.cc:243] Waited 531 ms on lock 0x7b54000f19d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:01:09.126187 (+   354us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1220572928}
I20260709 13:01:09.140848  1745 raft_consensus.cc:1240] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 7. Current term is 8. Ops: []
I20260709 13:01:09.148933  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:09.149065  6407 consensus_queue.cc:1048] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 13:01:09.150180  6293 leader_election.cc:290] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 12 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:09.142860  1752 raft_consensus.cc:1240] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 11. Current term is 12. Ops: []
I20260709 13:01:09.152302  6189 raft_consensus.cc:515] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:09.142553  6201 consensus_meta.cc:287] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.531s	user 0.015s	sys 0.011s
I20260709 13:01:09.050393  6354 consensus_queue.cc:260] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.10, Last appended by leader: 10, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:09.169230  1644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:09.175987  6344 raft_consensus.cc:493] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
W20260709 13:01:09.176640  6447 log.cc:927] Time spent T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.146s	user 0.000s	sys 0.001s
I20260709 13:01:09.177290  6344 raft_consensus.cc:515] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:09.179953  1644 leader_election.cc:304] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:01:09.174302  6350 raft_consensus.cc:2804] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:01:09.201313  1644 leader_election.cc:304] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:01:09.204509  6154 consensus_queue.cc:1048] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 13:01:09.207967  1745 raft_consensus.cc:1240] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 10. Current term is 11. Ops: []
I20260709 13:01:09.209880  6160 raft_consensus.cc:2749] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260709 13:01:09.211012  1718 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.107.194:46261 timed out after 1.897s (SENT) [suppressed 26 similar messages]
I20260709 13:01:09.214578  6232 raft_consensus.cc:515] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:09.216920  6270 raft_consensus.cc:2749] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:01:09.217274  6232 leader_election.cc:290] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 8 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:09.218572  1750 raft_consensus.cc:1240] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 11. Current term is 12. Ops: []
I20260709 13:01:09.183635  6171 leader_election.cc:290] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:09.185837  5989 raft_consensus.cc:493] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
W20260709 13:01:09.225986  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 101362us
W20260709 13:01:09.229086  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 99177.5us
I20260709 13:01:09.200747  6350 raft_consensus.cc:493] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting leader election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:01:09.218657  6332 raft_consensus.cc:2804] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:01:09.232687  6013 consensus_queue.cc:1059] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 11.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.002s
I20260709 13:01:09.229197  6407 consensus_queue.cc:1059] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 10.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.010s
I20260709 13:01:09.218029  6232 consensus_queue.cc:1048] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.011s
I20260709 13:01:09.198258  6428 leader_election.cc:290] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:01:09.248406  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 66600us
I20260709 13:01:09.258096  1751 raft_consensus.cc:2468] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 9.
I20260709 13:01:09.260553  6344 leader_election.cc:290] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:09.261391  6171 raft_consensus.cc:2804] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:01:09.261909  6171 raft_consensus.cc:493] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:09.264225  6407 raft_consensus.cc:3055] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 13:01:09.264693  6407 raft_consensus.cc:740] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:01:09.266752  6162 raft_consensus.cc:3055] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:01:09.267169  6162 raft_consensus.cc:740] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:01:09.267653  6298 raft_consensus.cc:515] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:09.268376  6162 consensus_queue.cc:260] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.8, Last appended by leader: 8, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:09.269182  1679 raft_consensus.cc:2393] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 13: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:01:09.269874  6162 raft_consensus.cc:3060] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:01:09.245117  6332 raft_consensus.cc:493] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:09.270955  6332 raft_consensus.cc:3060] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:01:09.274004  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:09.244808  6350 raft_consensus.cc:3060] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:01:09.275180  6354 raft_consensus.cc:3060] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:01:09.278159  5989 raft_consensus.cc:515] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:09.279678  6189 leader_election.cc:290] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 9 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:09.280952  5989 leader_election.cc:290] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:09.281625  1679 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72"
dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
mode: GRACEFUL
new_leader_uuid: "af520b67945d443b901bbeca37cc5701"
 from {username='slave'} at 127.0.0.1:43266
I20260709 13:01:09.262315  6171 raft_consensus.cc:3060] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:01:09.265888  6407 consensus_queue.cc:260] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:09.293627  6407 raft_consensus.cc:3060] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:01:09.294656  6344 raft_consensus.cc:493] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:09.295710  1679 raft_consensus.cc:606] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Received request to transfer leadership to TS af520b67945d443b901bbeca37cc5701
I20260709 13:01:09.286954  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:09.231519  6438 consensus_queue.cc:1059] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 11.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.056s
I20260709 13:01:09.255041  6140 leader_election.cc:290] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:09.306691  1679 raft_consensus.cc:614] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 13:01:09.295665  6344 raft_consensus.cc:515] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:09.299894  1676 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52778 (request call id 2174) took 2855 ms (client timeout 1897 ms). Trace:
W20260709 13:01:09.302629  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 46684.9us
I20260709 13:01:09.302815  6290 raft_consensus.cc:493] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:01:09.286485  1674 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57092 (request call id 1021) took 1297 ms. Trace:
I20260709 13:01:09.191702  6201 raft_consensus.cc:515] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:09.311967  2298 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:09.313568  1571 leader_election.cc:304] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
W20260709 13:01:09.310400  1676 rpcz_store.cc:269] 0709 13:01:06.463922 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:06.464192 (+   270us) service_pool.cc:224] Handling call
0709 13:01:06.614061 (+149869us) spinlock_profiling.cc:243] Waited 56.4 ms on lock 0x7b5800086a98. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f426303db71 00007f426303da22 00007f426bf56c09 00007f426b0b0ee0 00007f426b17a0ab 00007f426b17a131 00007f426b1280a0 00007f426b127984 00007f42646a26d4 00007f42646a2662
0709 13:01:09.299346 (+2685285us) spinlock_profiling.cc:243] Waited 1.26 s on lock 0x7b54000fe954. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:01:09.299616 (+   270us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3018325504}
W20260709 13:01:09.316205  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 105242us
I20260709 13:01:09.324410  1598 raft_consensus.cc:3060] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:01:09.326192  6232 raft_consensus.cc:2749] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
W20260709 13:01:09.327060  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 174539us
I20260709 13:01:09.328271  6290 raft_consensus.cc:515] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:09.329488  6229 leader_election.cc:290] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:09.332264  6242 leader_election.cc:290] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:09.333271  6298 leader_election.cc:290] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 13 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:09.335168  6046 leader_election.cc:290] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:09.322834  1569 leader_election.cc:304] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:01:09.336652  6201 leader_election.cc:290] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 10 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:09.311336  1674 rpcz_store.cc:276] 0709 13:01:07.932965 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:08.489733 (+556768us) service_pool.cc:224] Handling call
0709 13:01:08.523193 (+ 33460us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:01:09.001786 (+478593us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:09.001855 (+    69us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:01:09.003199 (+  1344us) log.cc:844] Serialized 52 byte log entry
0709 13:01:09.080568 (+ 77369us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:01:09.080694 (+   126us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:09.087331 (+  6637us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:09.222878 (+135547us) raft_consensus.cc:1685] finished
0709 13:01:09.222915 (+    37us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:09.230176 (+  7261us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":441050,"raft.queue_time_us":540606,"raft.run_cpu_time_us":149,"raft.run_wall_time_us":149,"thread_start_us":74270,"threads_started":1,"wal-append.queue_time_us":114544}
I20260709 13:01:09.361044  6290 leader_election.cc:290] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:09.366281  6242 raft_consensus.cc:2764] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Leader pre-election decision vote started in defunct term 12: won
I20260709 13:01:09.368062  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
W20260709 13:01:09.362249  6365 consensus_meta.cc:287] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.638s	user 0.020s	sys 0.004s
I20260709 13:01:09.372768  6365 raft_consensus.cc:515] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:09.377357  6082 consensus_queue.cc:1059] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 9.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 13:01:09.386399  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:09.387359  6365 leader_election.cc:290] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 11 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:09.379786  6171 raft_consensus.cc:515] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:09.391213  6360 consensus_queue.cc:1048] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.019s
I20260709 13:01:09.372905  6046 consensus_queue.cc:1048] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.025s
I20260709 13:01:09.399663  6171 leader_election.cc:290] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 11 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:09.406831  1571 leader_election.cc:304] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:09.413273  6242 raft_consensus.cc:2804] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:01:09.413887  6242 raft_consensus.cc:493] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:09.419014  1672 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57092 (request call id 1020) took 1378 ms. Trace:
W20260709 13:01:09.413322  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 787358us
I20260709 13:01:09.414358  6242 raft_consensus.cc:3060] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Advancing to term 11
W20260709 13:01:09.439853  2299 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:09.443962  6364 consensus_queue.cc:1048] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.066s
I20260709 13:01:09.446571  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:09.447891  1676 raft_consensus.cc:2393] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 8: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:01:09.449759  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:09.450080  6420 raft_consensus.cc:493] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:01:09.450898  6420 raft_consensus.cc:3060] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:01:09.450678  6232 consensus_queue.cc:1059] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 13:01:09.455103  1676 raft_consensus.cc:2468] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 13.
I20260709 13:01:09.456691  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:01:09.457022  6371 consensus_meta.cc:287] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 1.008s	user 0.017s	sys 0.007s
I20260709 13:01:09.458760  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:09.459849  1679 raft_consensus.cc:2393] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 8: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:01:09.457857  6371 raft_consensus.cc:515] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:09.462962  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:09.463285  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:09.463990  1675 raft_consensus.cc:2468] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 13.
I20260709 13:01:09.464823  6371 leader_election.cc:290] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:09.466727  6147 raft_consensus.cc:3060] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Advancing to term 10
W20260709 13:01:09.467643  6455 log.cc:927] Time spent T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.105s	user 0.005s	sys 0.000s
W20260709 13:01:09.467769  6249 consensus_meta.cc:287] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.500s	user 0.020s	sys 0.005s
W20260709 13:01:09.468029  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 254411us
I20260709 13:01:09.468312  6249 raft_consensus.cc:515] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:09.469429  1646 leader_election.cc:304] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:09.476158  6454 raft_consensus.cc:3055] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:01:09.476676  6046 raft_consensus.cc:2804] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Leader election won for term 10
I20260709 13:01:09.477316  6046 raft_consensus.cc:697] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:01:09.479398  1646 leader_election.cc:304] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:09.481247  6171 raft_consensus.cc:2804] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Leader election won for term 8
I20260709 13:01:09.481832  6171 raft_consensus.cc:697] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:01:09.379413  6290 raft_consensus.cc:493] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:01:09.486863  1600 raft_consensus.cc:1240] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 12. Current term is 13. Ops: [12.11-12.11]
I20260709 13:01:09.483649  1646 leader_election.cc:304] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:09.488255  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:09.431497  6344 leader_election.cc:290] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:09.508154  1671 raft_consensus.cc:3060] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:01:09.510767  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:01:09.511497  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 61490.9us
I20260709 13:01:09.530260  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
W20260709 13:01:09.516137  1571 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:09.516556  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:09.482499  6350 raft_consensus.cc:515] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:09.483048  6171 consensus_queue.cc:237] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:09.518549  6370 raft_consensus.cc:493] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:09.518862  1717 leader_election.cc:304] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:09.510114  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:09.522698  6290 raft_consensus.cc:515] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:01:09.524245  6446 log.cc:927] Time spent T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.448s	user 0.003s	sys 0.000s
I20260709 13:01:09.528103  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:09.508867  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:09.541386  2330 raft_consensus.cc:2376] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 12. Re-sending same reply.
I20260709 13:01:09.543658  6456 raft_consensus.cc:3055] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:01:09.544348  6456 raft_consensus.cc:740] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:01:09.544454  6350 leader_election.cc:290] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:09.544780  1717 leader_election.cc:400] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 12 election: Vote denied by peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445) with higher term. Message: Invalid argument: T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 12. Current term is 13.
I20260709 13:01:09.545151  1717 leader_election.cc:403] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 12 election: Cancelling election due to peer responding with higher term
I20260709 13:01:09.483700  6046 consensus_queue.cc:237] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.7, Last appended by leader: 7, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:09.546689  6364 raft_consensus.cc:3060] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:01:09.545758  6456 consensus_queue.cc:260] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:09.547664  6456 raft_consensus.cc:3060] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:01:09.550138  6201 raft_consensus.cc:2749] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Leader election lost for term 12. Reason: Vote denied by peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445) with higher term. Message: Invalid argument: T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 12. Current term is 13.
W20260709 13:01:09.555397  2333 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49082 (request call id 926) took 1613 ms (client timeout 1952 ms). Trace:
W20260709 13:01:09.555940  2333 rpcz_store.cc:269] 0709 13:01:07.941751 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:07.942404 (+   653us) service_pool.cc:224] Handling call
0709 13:01:09.555098 (+1612694us) spinlock_profiling.cc:243] Waited 990 ms on lock 0x7b54005bca54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:01:09.555303 (+   205us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2276368128}
I20260709 13:01:09.557152  1598 raft_consensus.cc:2468] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 17.
I20260709 13:01:09.558148  2337 raft_consensus.cc:1240] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 10. Current term is 11. Ops: []
I20260709 13:01:09.559720  6370 raft_consensus.cc:515] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:09.476722  6454 raft_consensus.cc:740] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:01:09.530303  2332 raft_consensus.cc:2468] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 13.
I20260709 13:01:09.420966  1672 rpcz_store.cc:276] 0709 13:01:07.932674 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:08.437659 (+504985us) service_pool.cc:224] Handling call
0709 13:01:08.438053 (+   394us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:01:08.960208 (+522155us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 13:01:08.960918 (+   710us) log.cc:844] Serialized 14 byte log entry
0709 13:01:09.084175 (+123257us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:01:09.143079 (+ 58904us) log.cc:844] Serialized 52 byte log entry
0709 13:01:09.143328 (+   249us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:01:09.305468 (+162140us) log.cc:844] Serialized 14 byte log entry
0709 13:01:09.305916 (+   448us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:09.310551 (+  4635us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:09.310611 (+    60us) raft_consensus.cc:1685] finished
0709 13:01:09.310623 (+    12us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:09.310790 (+   167us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":535444,"prepare.queue_time_us":18458,"prepare.run_cpu_time_us":109,"prepare.run_wall_time_us":1611,"raft.queue_time_us":636051,"raft.run_cpu_time_us":461,"raft.run_wall_time_us":11694,"spinlock_wait_cycles":13408512,"thread_start_us":268097,"threads_started":2,"wal-append.queue_time_us":207467}
I20260709 13:01:09.532567  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
W20260709 13:01:09.535802  1569 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:09.536504  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:09.572188  6371 raft_consensus.cc:2804] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:01:09.575429  6371 raft_consensus.cc:493] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:01:09.575922  6371 raft_consensus.cc:3060] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:01:09.580497  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:09.581311  6242 raft_consensus.cc:515] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:09.581800  1717 leader_election.cc:304] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:09.582751  6277 raft_consensus.cc:493] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:09.583698  6277 raft_consensus.cc:515] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:09.585500  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:09.586640  2328 raft_consensus.cc:2468] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 13.
I20260709 13:01:09.586925  6277 leader_election.cc:290] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:09.587229  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:09.587641  6277 raft_consensus.cc:493] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:09.588143  1604 raft_consensus.cc:2393] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 13: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:01:09.588308  6277 raft_consensus.cc:515] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:09.589843  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 12 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:09.591364  6277 leader_election.cc:290] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:09.592056  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:09.595801  6160 raft_consensus.cc:2804] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:01:09.596352  6160 raft_consensus.cc:493] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:09.596791  6160 raft_consensus.cc:3060] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:01:09.597083  6290 leader_election.cc:290] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:09.594208  6408 raft_consensus.cc:3055] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:01:09.598769  6408 raft_consensus.cc:740] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:01:09.599941  6408 consensus_queue.cc:260] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.11, Last appended by leader: 11, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:09.601141  6444 log.cc:927] Time spent T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.549s	user 0.001s	sys 0.000s
I20260709 13:01:09.601184  6408 raft_consensus.cc:3060] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:01:09.607291  1569 leader_election.cc:304] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:01:09.608124  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:09.608341  6249 leader_election.cc:290] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 12 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:09.608732  1569 leader_election.cc:304] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:01:09.609022  1605 raft_consensus.cc:2468] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 10.
I20260709 13:01:09.611137  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:09.611384  6454 consensus_queue.cc:260] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:09.612269  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:09.612869  6454 raft_consensus.cc:3060] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:01:09.613612  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:09.615142  6067 raft_consensus.cc:3055] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:01:09.615736  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:09.617321  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:09.617738  1676 raft_consensus.cc:1240] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 8. Current term is 9. Ops: []
I20260709 13:01:09.610134  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:09.619066  6029 raft_consensus.cc:2804] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:01:09.619146  6357 raft_consensus.cc:2804] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:01:09.627101  6357 raft_consensus.cc:493] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:09.627562  6357 raft_consensus.cc:3060] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:01:09.627444  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:09.618287  1603 raft_consensus.cc:2410] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for term 13 because replica has last-logged OpId of term: 12 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 7.
I20260709 13:01:09.615134  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:09.629262  6176 raft_consensus.cc:3060] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:01:09.637411  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:09.639719  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:09.642876  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:09.645421  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:09.642943  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 13 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:09.650319  1671 raft_consensus.cc:2468] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 12.
I20260709 13:01:09.639271  1569 leader_election.cc:304] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:09.615754  6067 raft_consensus.cc:740] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:01:09.653074  1644 leader_election.cc:304] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:01:09.652343  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:09.624145  6029 raft_consensus.cc:493] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:09.629993  6290 raft_consensus.cc:493] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:09.677682  1672 raft_consensus.cc:2393] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 13: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:01:09.684146  2336 raft_consensus.cc:2393] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 10: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:01:09.686714  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 12 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:09.689056  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:09.691601  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:09.696061  1605 raft_consensus.cc:2393] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 11: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:01:09.672549  6029 raft_consensus.cc:3060] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Advancing to term 14
W20260709 13:01:09.702714  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 63633.3us
I20260709 13:01:09.703758  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:09.709280  1602 raft_consensus.cc:2468] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 10.
I20260709 13:01:09.710561  1644 leader_election.cc:304] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:01:09.711026  1603 raft_consensus.cc:2393] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 11: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:01:09.711841  1644 leader_election.cc:304] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:01:09.711961  1569 leader_election.cc:304] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:09.713424  1644 leader_election.cc:304] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:01:09.714252  6215 raft_consensus.cc:2749] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
W20260709 13:01:09.715708  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:09.719184  6364 raft_consensus.cc:2749] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:01:09.719373  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 11 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:09.720192  6242 leader_election.cc:290] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:09.720655  2331 raft_consensus.cc:2393] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 11: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:01:09.713622  1569 leader_election.cc:400] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 16 election: Vote denied by peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625) with higher term. Message: Invalid argument: T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 16. Current term is 17.
I20260709 13:01:09.722914  1569 leader_election.cc:403] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 16 election: Cancelling election due to peer responding with higher term
I20260709 13:01:09.723259  6363 raft_consensus.cc:2804] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:01:09.723868  6363 raft_consensus.cc:697] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 12 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:01:09.724929  6363 consensus_queue.cc:237] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:09.727110  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 12 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:09.727459  6345 raft_consensus.cc:2804] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:01:09.727938  6345 raft_consensus.cc:493] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:01:09.728201  1596 raft_consensus.cc:2410] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for term 12 because replica has last-logged OpId of term: 11 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 7.
I20260709 13:01:09.728312  6345 raft_consensus.cc:3060] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:01:09.729269  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:09.729648  6453 raft_consensus.cc:2804] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:01:09.730216  6453 raft_consensus.cc:493] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:09.730619  6453 raft_consensus.cc:3060] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:01:09.731190  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:09.731907  1754 raft_consensus.cc:2468] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 12.
I20260709 13:01:09.733330  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:09.734331  1748 raft_consensus.cc:2410] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for term 11 because replica has last-logged OpId of term: 10 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20260709 13:01:09.734704  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:09.734838  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:09.735189  1672 raft_consensus.cc:1240] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 10. Current term is 11. Ops: [7.8-10.9]
I20260709 13:01:09.735579  1745 raft_consensus.cc:2393] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 16: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:01:09.735507  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:09.736218  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:09.736557  1752 raft_consensus.cc:2393] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 12: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:01:09.736894  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:09.737775  1749 raft_consensus.cc:2393] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 13: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:01:09.737713  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:09.739807  6261 raft_consensus.cc:2749] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Leader election lost for term 16. Reason: Vote denied by peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625) with higher term. Message: Invalid argument: T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 16. Current term is 17.
I20260709 13:01:09.742483  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:09.743194  1680 raft_consensus.cc:3060] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:01:09.737748  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 13 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:09.744567  1676 raft_consensus.cc:2410] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for term 13 because replica has last-logged OpId of term: 12 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 7.
I20260709 13:01:09.729301  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:09.745551  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:09.748831  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:09.734331  2333 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec"
dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
mode: GRACEFUL
new_leader_uuid: "af520b67945d443b901bbeca37cc5701"
 from {username='slave'} at 127.0.0.1:35790
I20260709 13:01:09.727121  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:09.752053  2337 raft_consensus.cc:2468] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 12.
I20260709 13:01:09.750329  2333 raft_consensus.cc:606] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Received request to transfer leadership to TS af520b67945d443b901bbeca37cc5701
I20260709 13:01:09.756165  2333 raft_consensus.cc:614] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260709 13:01:09.756875  1644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:09.760381  2301 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:09.765360  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:09.766355  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:09.770119  1602 raft_consensus.cc:2468] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 10.
I20260709 13:01:09.771118  6332 raft_consensus.cc:515] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:09.771970  1753 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43156 (request call id 914) took 1254 ms. Trace:
I20260709 13:01:09.798002  1601 raft_consensus.cc:2468] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 12.
I20260709 13:01:09.748847  6370 leader_election.cc:290] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:09.778735  6067 consensus_queue.cc:260] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } }
I20260709 13:01:09.784463  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:09.784838  6350 raft_consensus.cc:493] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:09.730510  6390 raft_consensus.cc:2749] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 13:01:09.807854  1747 raft_consensus.cc:2393] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 9: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:01:09.807986  1569 leader_election.cc:304] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 13:01:09.787875  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:09.694198  6249 raft_consensus.cc:493] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:09.792507  1671 raft_consensus.cc:2393] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 10: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:01:09.790105  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:09.798966  1676 raft_consensus.cc:1240] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 9. Current term is 10. Ops: []
I20260709 13:01:09.794284  2298 leader_election.cc:304] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:01:09.801816  2329 raft_consensus.cc:2393] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 13: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:01:09.803144  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:09.797904  1753 rpcz_store.cc:276] 0709 13:01:08.517390 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:08.577277 (+ 59887us) service_pool.cc:224] Handling call
0709 13:01:08.577612 (+   335us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:08.909660 (+332048us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:09.771559 (+861899us) spinlock_profiling.cc:243] Waited 330 ms on lock 0x7b5400220454. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:01:09.771883 (+   324us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":759149440}
I20260709 13:01:09.803123  1644 leader_election.cc:304] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:01:09.818789  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:09.837087  1751 raft_consensus.cc:2468] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 8.
I20260709 13:01:09.840428  1674 raft_consensus.cc:2368] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 11. Current term is 12.
I20260709 13:01:09.840777  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:09.849311  2328 raft_consensus.cc:2468] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 11.
I20260709 13:01:09.850158  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:09.759856  6290 raft_consensus.cc:515] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:09.852017  2330 raft_consensus.cc:1240] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 10. Current term is 11. Ops: [10.10-10.10]
I20260709 13:01:09.858994  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:09.867508  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:09.821548  6046 raft_consensus.cc:2764] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Leader pre-election decision vote started in defunct term 11: won
I20260709 13:01:09.808768  6456 consensus_queue.cc:1059] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.056s
I20260709 13:01:09.847290  6277 raft_consensus.cc:2804] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:01:09.874192  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:09.877908  2329 raft_consensus.cc:2393] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 10: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:01:09.797538  6420 raft_consensus.cc:515] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:09.881868  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 12 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:09.859706  6013 raft_consensus.cc:2804] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 13:01:09.892058  6013 raft_consensus.cc:697] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 17 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:01:09.899268  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:01:09.911154  1570 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:09.886178  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 78288.6us
I20260709 13:01:09.913659  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:01:09.914532  1569 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:09.915856  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:09.887542  2336 raft_consensus.cc:2393] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 9: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:01:09.865341  6350 raft_consensus.cc:515] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:09.893450  6013 consensus_queue.cc:237] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.11, Last appended by leader: 11, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:09.925889  6464 log.cc:927] Time spent T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.051s	user 0.000s	sys 0.000s
I20260709 13:01:09.935246  1602 raft_consensus.cc:2468] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 12.
I20260709 13:01:09.937660  1569 leader_election.cc:304] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:09.912350  1747 raft_consensus.cc:3060] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:01:09.929371  2332 raft_consensus.cc:1240] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 13. Current term is 14. Ops: []
I20260709 13:01:09.940323  2337 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c"
dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
mode: GRACEFUL
new_leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
 from {username='slave'} at 127.0.0.1:35790
I20260709 13:01:09.933286  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
W20260709 13:01:09.946832  2298 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:09.918699  6277 raft_consensus.cc:697] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:01:09.955673  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:01:09.934264  6354 consensus_meta.cc:287] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.659s	user 0.012s	sys 0.005s
I20260709 13:01:09.958057  2329 raft_consensus.cc:1275] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260709 13:01:09.881070  6249 raft_consensus.cc:515] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:01:09.962173  1570 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:09.965384  1751 raft_consensus.cc:2468] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 13.
I20260709 13:01:09.893712  6029 raft_consensus.cc:515] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:09.972280  5938 consensus_queue.cc:1059] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 13.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 13:01:09.974272  6140 raft_consensus.cc:3055] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 LEADER]: Stepping down as leader of term 13
I20260709 13:01:09.974815  6140 raft_consensus.cc:740] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:01:09.975915  6140 consensus_queue.cc:260] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.11, Last appended by leader: 11, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:09.977656  6140 raft_consensus.cc:3060] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Advancing to term 14
W20260709 13:01:09.978925  2300 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:09.952404  6261 raft_consensus.cc:2749] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:01:09.983986  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:09.984457  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:09.985388  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:09.987190  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:09.987824  1753 raft_consensus.cc:3060] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:01:09.992738  6350 leader_election.cc:290] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:09.993836  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:09.994746  1746 raft_consensus.cc:2393] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 8: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:01:09.995656  6298 consensus_queue.cc:1059] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 11.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 13:01:09.996013  1752 raft_consensus.cc:1240] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 10. Current term is 11. Ops: []
I20260709 13:01:09.996779  1603 raft_consensus.cc:2468] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 8.
I20260709 13:01:09.997464  6082 raft_consensus.cc:493] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:09.985567  1676 raft_consensus.cc:2393] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 10: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:01:09.998085  6082 raft_consensus.cc:515] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:10.001892  1674 raft_consensus.cc:1240] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 12. Current term is 13. Ops: []
I20260709 13:01:09.997236  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:10.004236  1596 raft_consensus.cc:3060] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:01:09.953797  6067 raft_consensus.cc:3060] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Advancing to term 9
W20260709 13:01:10.014228  1717 leader_election.cc:336] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 1.524s (SENT)
W20260709 13:01:10.015038  1570 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:10.018616  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:09.997426  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:10.029528  1749 raft_consensus.cc:2468] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 12.
I20260709 13:01:10.036345  1603 raft_consensus.cc:1240] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 12. Current term is 13. Ops: []
I20260709 13:01:10.037647  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:10.038540  1603 raft_consensus.cc:2393] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 13: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:01:10.040865  1717 leader_election.cc:304] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:10.049072  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:10.007099  1680 raft_consensus.cc:2468] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 12.
W20260709 13:01:10.057708  1679 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.595s	user 0.010s	sys 0.020s
W20260709 13:01:10.058336  1679 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.596s	user 0.010s	sys 0.021s
I20260709 13:01:10.062371  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:10.063309  1601 raft_consensus.cc:2393] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 12: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:01:10.065565  1674 raft_consensus.cc:2468] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 9.
W20260709 13:01:10.066651  1569 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:10.067222  1747 raft_consensus.cc:2468] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 11.
I20260709 13:01:10.068295  1569 leader_election.cc:304] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:01:10.071555  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 12 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:10.069649  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:10.075606  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:10.076790  1598 raft_consensus.cc:2393] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 13: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:01:10.078945  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:10.087383  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:10.088367  1601 raft_consensus.cc:2393] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 8: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:01:10.090976  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:10.092509  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 11 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:10.093339  1676 raft_consensus.cc:2393] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 11: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:01:10.065969  6344 raft_consensus.cc:2804] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:01:10.042183  6261 consensus_queue.cc:1048] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.058s
I20260709 13:01:10.103538  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:10.062876  6364 raft_consensus.cc:2749] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:01:10.104249  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:10.109877  1598 raft_consensus.cc:2393] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 8: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:01:10.093890  1753 raft_consensus.cc:2468] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 13.
I20260709 13:01:10.082963  6350 raft_consensus.cc:2804] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:01:10.107837  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:10.117333  1603 raft_consensus.cc:2368] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for earlier term 13. Current term is 14.
I20260709 13:01:10.109438  6407 consensus_queue.cc:1059] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 11.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.022s
W20260709 13:01:10.127480  1599 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.504s	user 0.033s	sys 0.069s
W20260709 13:01:10.127954  1599 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.586s	user 0.033s	sys 0.069s
I20260709 13:01:10.128271  6365 raft_consensus.cc:3055] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 13:01:10.128717  6365 raft_consensus.cc:740] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
W20260709 13:01:10.128830  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 121140us
I20260709 13:01:10.129913  6365 consensus_queue.cc:260] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 13.10, Last appended by leader: 10, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:10.116664  6350 raft_consensus.cc:493] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:10.132784  6365 raft_consensus.cc:3060] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:01:10.133225  6350 raft_consensus.cc:3060] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Advancing to term 10
W20260709 13:01:10.138489  6445 thread.cc:641] apply [worker] (thread pool) Time spent creating pthread: real 0.878s	user 0.026s	sys 0.042s
W20260709 13:01:10.139017  6445 thread.cc:608] apply [worker] (thread pool) Time spent starting thread: real 1.083s	user 0.026s	sys 0.042s
W20260709 13:01:10.140020  2335 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.667s	user 0.048s	sys 0.061s
W20260709 13:01:10.140641  2335 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.668s	user 0.048s	sys 0.061s
I20260709 13:01:10.095139  6332 leader_election.cc:290] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 12 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:01:10.164376  2301 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:10.164716  6454 consensus_meta.cc:287] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.551s	user 0.016s	sys 0.000s
I20260709 13:01:10.107895  6201 raft_consensus.cc:493] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:10.171011  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:10.157361  6364 raft_consensus.cc:493] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:10.194518  1597 raft_consensus.cc:2468] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 10.
I20260709 13:01:10.215831  1601 raft_consensus.cc:2468] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 10.
I20260709 13:01:10.107431  6357 raft_consensus.cc:515] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:10.091323  6277 consensus_queue.cc:237] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.10, Last appended by leader: 10, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:10.177659  6420 leader_election.cc:290] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 14 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:10.240932  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:10.244614  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:01:10.242220  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 48537us
I20260709 13:01:10.220871  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:10.256042  1596 raft_consensus.cc:2468] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 12.
I20260709 13:01:10.267145  1604 raft_consensus.cc:2368] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 13. Current term is 14.
W20260709 13:01:10.248175  6371 consensus_meta.cc:287] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.669s	user 0.015s	sys 0.000s
I20260709 13:01:10.225526  6201 raft_consensus.cc:515] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:10.274039  1599 raft_consensus.cc:1240] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 10. Current term is 11. Ops: []
W20260709 13:01:10.274536  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 485027us
I20260709 13:01:10.306926  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:10.251443  6290 leader_election.cc:290] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:01:10.269273  6160 consensus_meta.cc:287] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.672s	user 0.017s	sys 0.000s
I20260709 13:01:10.316444  2299 leader_election.cc:304] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:01:10.325337  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:10.329010  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:10.329903  2330 raft_consensus.cc:3060] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:01:10.331032  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
W20260709 13:01:10.333809  2301 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:10.339003  1644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:10.260419  6364 raft_consensus.cc:515] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:10.343948  1599 raft_consensus.cc:2393] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 12: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:01:10.346102  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:10.346581  6201 leader_election.cc:290] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:10.324900  6160 raft_consensus.cc:515] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:10.274570  6371 raft_consensus.cc:515] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:10.373021  6249 leader_election.cc:290] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 13:01:10.373943  6472 log.cc:927] Time spent T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.206s	user 0.000s	sys 0.001s
I20260709 13:01:10.334656  6290 raft_consensus.cc:493] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:10.335620  6438 raft_consensus.cc:2749] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:01:10.326642  6171 raft_consensus.cc:493] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:10.396534  2330 raft_consensus.cc:2468] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 12.
I20260709 13:01:10.403156  1675 raft_consensus.cc:1240] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 9. Current term is 10. Ops: []
I20260709 13:01:10.405282  6160 leader_election.cc:290] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:10.406091  1596 raft_consensus.cc:1275] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 10 index: 8. (index mismatch)
I20260709 13:01:10.435079  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:01:10.431623  1643 leader_election.cc:336] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 1.566s (SENT)
I20260709 13:01:10.406587  6371 leader_election.cc:290] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:10.407006  6201 consensus_queue.cc:1059] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 13:01:10.408950  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:10.419001  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:10.435819  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:10.439260  1674 raft_consensus.cc:3060] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:01:10.439751  1599 raft_consensus.cc:2393] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 8: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:01:10.443472  1753 raft_consensus.cc:1240] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 8. Current term is 9. Ops: []
I20260709 13:01:10.418062  1602 raft_consensus.cc:2468] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 9.
W20260709 13:01:10.426177  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 296321us
W20260709 13:01:10.431183  1570 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:10.446738  1643 leader_election.cc:304] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
W20260709 13:01:10.447935  1570 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:10.432078  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 315591us
I20260709 13:01:10.436724  1644 leader_election.cc:304] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:01:10.452361  1718 leader_election.cc:304] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
W20260709 13:01:10.453748  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 51282.9us
I20260709 13:01:10.437306  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:10.458086  1596 raft_consensus.cc:2393] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 13: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:01:10.452585  6171 raft_consensus.cc:515] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:10.463210  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:10.467092  1752 raft_consensus.cc:2393] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 10: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:01:10.480513  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:10.481285  1746 raft_consensus.cc:3060] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:01:10.481130  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:10.445356  6249 raft_consensus.cc:493] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:10.484131  1718 leader_election.cc:304] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:10.479871  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 11 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
W20260709 13:01:10.485245  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 51806.3us
I20260709 13:01:10.487643  1749 raft_consensus.cc:1240] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 10. Current term is 11. Ops: []
I20260709 13:01:10.489060  2299 leader_election.cc:304] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:10.480566  1753 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31"
dest_uuid: "af520b67945d443b901bbeca37cc5701"
 from {username='slave'} at 127.0.0.1:43156
I20260709 13:01:10.491217  1753 raft_consensus.cc:493] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:01:10.491595  1753 raft_consensus.cc:3060] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:01:10.493914  6480 raft_consensus.cc:493] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:10.494617  6480 raft_consensus.cc:515] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:10.498725  1718 leader_election.cc:304] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:10.500559  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:10.433476  6290 raft_consensus.cc:515] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:10.499750  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
W20260709 13:01:10.500371  1569 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:10.478096  6160 raft_consensus.cc:3055] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 10 LEADER]: Stepping down as leader of term 10
W20260709 13:01:10.511207  1569 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:10.512813  1570 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:10.514103  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:10.491770  6298 raft_consensus.cc:2804] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Leader election won for term 12
W20260709 13:01:10.502974  6476 log.cc:927] Time spent T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.093s	user 0.000s	sys 0.001s
I20260709 13:01:10.496963  6357 leader_election.cc:290] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 14 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:10.521911  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:10.488533  6481 raft_consensus.cc:3060] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:01:10.503453  6201 raft_consensus.cc:2749] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 13:01:10.520399  1753 raft_consensus.cc:515] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:10.531962  2330 raft_consensus.cc:2468] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 14.
I20260709 13:01:10.533345  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:01:10.524231  2299 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:10.509826  6438 raft_consensus.cc:2749] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:01:10.537551  1754 raft_consensus.cc:1240] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 10. Current term is 11. Ops: []
I20260709 13:01:10.506156  6160 raft_consensus.cc:740] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:01:10.497056  6480 leader_election.cc:290] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:10.542152  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:10.492583  6249 raft_consensus.cc:515] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:10.554677  1753 leader_election.cc:290] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:10.552862  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:10.558701  1745 raft_consensus.cc:2468] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 13.
I20260709 13:01:10.561023  6344 raft_consensus.cc:493] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Starting leader election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:01:10.561553  6344 raft_consensus.cc:3060] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:01:10.558538  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:10.569242  1644 leader_election.cc:304] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:01:10.538313  6298 raft_consensus.cc:697] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:01:10.585904  1603 raft_consensus.cc:1275] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260709 13:01:10.538033  6485 raft_consensus.cc:2749] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
W20260709 13:01:10.589049  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 163272us
W20260709 13:01:10.589178  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 90532.8us
I20260709 13:01:10.574424  6420 raft_consensus.cc:493] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:10.618737  6420 raft_consensus.cc:515] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:10.621196  6420 leader_election.cc:290] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:10.623087  1718 leader_election.cc:304] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:10.624086  6046 raft_consensus.cc:3055] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 11 LEADER]: Stepping down as leader of term 11
I20260709 13:01:10.624542  6046 raft_consensus.cc:740] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:01:10.625548  6046 consensus_queue.cc:260] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:10.627569  6290 leader_election.cc:290] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:10.591852  1601 raft_consensus.cc:1275] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 10 index: 8. (index mismatch)
I20260709 13:01:10.579926  6480 raft_consensus.cc:2804] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Leader pre-election won for term 14
W20260709 13:01:10.588842  6453 consensus_meta.cc:287] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.858s	user 0.016s	sys 0.000s
W20260709 13:01:10.635301  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 141786us
W20260709 13:01:10.622287  6345 consensus_meta.cc:287] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.893s	user 0.005s	sys 0.009s
W20260709 13:01:10.644692  1717 leader_election.cc:336] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 1.512s (SENT)
W20260709 13:01:10.630867  6478 log.cc:927] Time spent T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.133s	user 0.000s	sys 0.001s
W20260709 13:01:10.628785  6408 consensus_meta.cc:287] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 1.022s	user 0.013s	sys 0.005s
I20260709 13:01:10.575563  5994 consensus_queue.cc:1059] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 12.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.117s
I20260709 13:01:10.659029  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:10.667164  1601 raft_consensus.cc:2468] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 11.
I20260709 13:01:10.660477  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:10.660923  1596 raft_consensus.cc:1275] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 15 index: 11. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
I20260709 13:01:10.668373  1752 raft_consensus.cc:2393] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 13: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
W20260709 13:01:10.643867  6350 consensus_meta.cc:287] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.510s	user 0.016s	sys 0.001s
I20260709 13:01:10.666231  2337 raft_consensus.cc:606] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Received request to transfer leadership to TS 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:10.669222  2337 raft_consensus.cc:614] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 13:01:10.642268  6480 raft_consensus.cc:493] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:10.651775  6364 leader_election.cc:290] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:10.682926  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:10.683660  1753 raft_consensus.cc:2468] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 12.
I20260709 13:01:10.684898  6357 consensus_queue.cc:1059] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 11.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 13:01:10.686045  6357 consensus_queue.cc:1059] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 13:01:10.687428  6261 raft_consensus.cc:3055] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 LEADER]: Stepping down as leader of term 11
I20260709 13:01:10.687896  6261 raft_consensus.cc:740] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:01:10.688905  6261 consensus_queue.cc:260] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:10.659547  6438 raft_consensus.cc:493] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:01:10.658003  6365 raft_consensus.cc:493] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:10.669493  6160 consensus_queue.cc:260] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.9, Last appended by leader: 9, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:10.674722  6046 raft_consensus.cc:3060] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:01:10.699469  6160 raft_consensus.cc:3060] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:01:10.675657  6350 raft_consensus.cc:515] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:10.702665  6365 raft_consensus.cc:515] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:10.703898  6350 leader_election.cc:290] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 10 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:10.704576  6350 consensus_queue.cc:1059] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 13:01:10.714778  1605 raft_consensus.cc:1240] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 9. Current term is 10. Ops: []
W20260709 13:01:10.720468  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 76770.3us
W20260709 13:01:10.724140  1644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:10.704944  6365 leader_election.cc:290] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:10.652667  6453 raft_consensus.cc:515] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:10.745535  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:10.683594  6029 leader_election.cc:290] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 14 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:10.693130  6370 raft_consensus.cc:3055] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:01:10.693240  6140 consensus_queue.cc:1059] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.042s
W20260709 13:01:10.750082  6482 log.cc:927] Time spent T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.228s	user 0.000s	sys 0.001s
I20260709 13:01:10.657732  6345 raft_consensus.cc:515] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:10.719745  6277 raft_consensus.cc:3055] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 LEADER]: Stepping down as leader of term 11
W20260709 13:01:10.750262  6492 log.cc:927] Time spent T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.118s	user 0.006s	sys 0.000s
I20260709 13:01:10.763296  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:10.782927  1749 raft_consensus.cc:2393] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 8: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:01:10.676759  6480 raft_consensus.cc:3060] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Advancing to term 14
W20260709 13:01:10.761482  1643 leader_election.cc:336] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 13 election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 1.674s (SENT)
I20260709 13:01:10.783703  1643 leader_election.cc:304] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:01:10.753840  6485 raft_consensus.cc:2804] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 13:01:10.786011  1671 raft_consensus.cc:2368] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 for earlier term 13. Current term is 14.
I20260709 13:01:10.750913  6496 raft_consensus.cc:3060] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:01:10.777155  1674 raft_consensus.cc:2468] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 14.
I20260709 13:01:10.787338  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 11 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:10.763811  6370 raft_consensus.cc:740] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:01:10.793576  1673 raft_consensus.cc:1240] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 13. Current term is 14. Ops: []
I20260709 13:01:10.751108  5989 raft_consensus.cc:493] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:01:10.790710  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 125545us
I20260709 13:01:10.776634  6171 leader_election.cc:290] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:10.778563  6277 raft_consensus.cc:740] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:01:10.802242  2298 leader_election.cc:304] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:01:10.753391  6438 raft_consensus.cc:515] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:10.816846  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:10.808375  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:10.822649  1749 raft_consensus.cc:2393] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 10: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:01:10.823616  6013 raft_consensus.cc:3055] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:01:10.824169  6013 raft_consensus.cc:740] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:01:10.825448  6013 consensus_queue.cc:260] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 12.12, Last appended by leader: 12, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:10.880219  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 222896us
I20260709 13:01:10.890640  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:01:10.885273  1597 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40154 (request call id 2293) took 1278 ms (client timeout 1524 ms). Trace:
W20260709 13:01:10.879848  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 97787.9us
W20260709 13:01:10.887197  1717 leader_election.cc:336] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 1.579s (SENT)
I20260709 13:01:10.886988  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:10.887979  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:10.899029  1717 leader_election.cc:304] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:10.764016  6298 consensus_queue.cc:237] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:10.879491  5994 raft_consensus.cc:2804] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Leader pre-election won for term 12
W20260709 13:01:10.916630  2301 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:10.902691  1753 raft_consensus.cc:2393] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 9: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:01:10.820218  6332 raft_consensus.cc:2749] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
W20260709 13:01:10.898780  1597 rpcz_store.cc:269] 0709 13:01:09.607137 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:09.610753 (+  3616us) service_pool.cc:224] Handling call
0709 13:01:10.885031 (+1274278us) spinlock_profiling.cc:243] Waited 582 ms on lock 0x7b540006dbd4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:01:10.885198 (+   167us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1339559296}
I20260709 13:01:10.901793  1718 leader_election.cc:304] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:10.810376  6485 raft_consensus.cc:697] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 11 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
W20260709 13:01:10.910638  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 88674.3us
I20260709 13:01:10.911185  6497 raft_consensus.cc:2749] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:01:10.913300  1601 raft_consensus.cc:2468] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 9.
I20260709 13:01:10.915109  1749 raft_consensus.cc:1240] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 10. Current term is 11. Ops: []
W20260709 13:01:10.915663  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 154250us
I20260709 13:01:10.923956  1571 leader_election.cc:304] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
W20260709 13:01:10.903825  6491 log.cc:927] Time spent T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.212s	user 0.000s	sys 0.001s
I20260709 13:01:10.940717  1599 raft_consensus.cc:2468] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 14.
I20260709 13:01:10.942775  1746 raft_consensus.cc:2468] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 17.
W20260709 13:01:10.951650  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 49986.4us
I20260709 13:01:10.906014  6249 leader_election.cc:290] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:10.919245  5994 raft_consensus.cc:493] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:10.983033  1718 leader_election.cc:400] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 election: Vote denied by peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261) with higher term. Message: Invalid argument: T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 11. Current term is 12.
W20260709 13:01:10.980230  1717 leader_election.cc:336] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 1.614s (SENT)
I20260709 13:01:10.984634  1717 leader_election.cc:304] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:10.989012  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:10.893247  5989 raft_consensus.cc:515] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:10.987829  6013 raft_consensus.cc:3060] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:01:10.997462  1746 raft_consensus.cc:1275] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 10 index: 8. (index mismatch)
I20260709 13:01:10.995705  1718 leader_election.cc:304] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:10.955989  6495 raft_consensus.cc:2749] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 13:01:10.987632  5994 raft_consensus.cc:3060] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Advancing to term 12
W20260709 13:01:10.991406  6499 log.cc:927] Time spent T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.100s	user 0.000s	sys 0.001s
I20260709 13:01:11.010880  1751 raft_consensus.cc:1275] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
W20260709 13:01:11.011725  1643 leader_election.cc:336] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 1.714s (SENT)
W20260709 13:01:10.999128  6242 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.658s	user 0.031s	sys 0.039s
I20260709 13:01:11.015439  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: true dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
W20260709 13:01:11.014370  6490 log.cc:927] Time spent T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.308s	user 0.001s	sys 0.000s
I20260709 13:01:10.957597  6370 consensus_queue.cc:260] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:11.017442  5938 raft_consensus.cc:2749] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:01:11.022874  6029 raft_consensus.cc:493] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:11.019032  6502 raft_consensus.cc:493] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:01:11.010231  6484 log.cc:927] Time spent T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.336s	user 0.000s	sys 0.001s
I20260709 13:01:11.007990  6249 raft_consensus.cc:2749] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:01:11.049775  6513 raft_consensus.cc:493] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:01:11.010320  6497 raft_consensus.cc:2804] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:01:11.060032  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:11.062659  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:11.070453  1746 raft_consensus.cc:2393] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 14: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
W20260709 13:01:11.050118  6242 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.722s	user 0.031s	sys 0.040s
I20260709 13:01:11.063274  2334 raft_consensus.cc:3055] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 10 LEADER]: Stepping down as leader of term 10
W20260709 13:01:11.051460  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 90660.7us
I20260709 13:01:11.067855  1753 raft_consensus.cc:1240] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 11. Current term is 12. Ops: []
I20260709 13:01:11.007464  6344 raft_consensus.cc:515] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:11.062525  6494 log.cc:927] Time spent T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.277s	user 0.001s	sys 0.000s
I20260709 13:01:11.073280  2334 raft_consensus.cc:740] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:01:11.067365  6029 raft_consensus.cc:515] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:11.076589  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 71475.7us
I20260709 13:01:10.994342  6277 consensus_queue.cc:260] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.6, Last appended by leader: 6, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:11.091181  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:11.091941  1746 raft_consensus.cc:2468] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 12.
I20260709 13:01:11.086829  2334 consensus_queue.cc:260] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:11.095114  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:11.070672  6360 raft_consensus.cc:493] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:11.105681  5989 leader_election.cc:290] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:11.106362  6029 leader_election.cc:290] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:11.071768  6453 leader_election.cc:290] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 14 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:11.103497  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:11.071749  6513 raft_consensus.cc:515] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:11.072742  6502 raft_consensus.cc:515] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:11.122932  1754 raft_consensus.cc:2410] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for term 11 because replica has last-logged OpId of term: 10 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20260709 13:01:11.124218  6513 leader_election.cc:290] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:11.124631  6502 leader_election.cc:290] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:11.127214  1753 raft_consensus.cc:2468] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 13.
I20260709 13:01:11.068454  6485 consensus_queue.cc:237] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:11.130782  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:11.131592  6360 raft_consensus.cc:515] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:11.132833  2329 raft_consensus.cc:2468] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 14.
I20260709 13:01:11.134064  6360 leader_election.cc:290] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:11.136421  2334 raft_consensus.cc:3060] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:01:11.145717  6013 raft_consensus.cc:493] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:11.088294  6497 raft_consensus.cc:697] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 12 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:01:11.119715  6438 leader_election.cc:290] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:11.096803  6345 leader_election.cc:290] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 14 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:11.108887  6277 raft_consensus.cc:3060] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:01:11.163447  6497 consensus_queue.cc:237] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.5, Last appended by leader: 5, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:11.164255  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:01:11.147487  1717 leader_election.cc:336] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 1.647s (SENT)
I20260709 13:01:11.166463  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:11.166894  1717 leader_election.cc:304] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:11.107434  6515 raft_consensus.cc:493] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:01:11.149200  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:11.167953  1601 raft_consensus.cc:2468] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 13.
I20260709 13:01:11.151937  1754 raft_consensus.cc:1240] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 7. Current term is 8. Ops: []
I20260709 13:01:11.152326  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:11.152599  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:11.153355  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:11.102908  5938 raft_consensus.cc:493] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:11.154881  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:11.156494  6344 leader_election.cc:290] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 15 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:11.157109  1753 raft_consensus.cc:1240] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 8. Current term is 9. Ops: []
I20260709 13:01:11.158512  1745 raft_consensus.cc:1275] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260709 13:01:11.159340  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:11.161911  2334 raft_consensus.cc:2468] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 11.
W20260709 13:01:11.173364  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 70799us
I20260709 13:01:11.150326  1646 leader_election.cc:304] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:01:11.170082  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:11.177450  1754 raft_consensus.cc:2468] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 11.
W20260709 13:01:11.179165  2299 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:11.181051  6515 raft_consensus.cc:515] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:11.183542  6515 leader_election.cc:290] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:01:11.183979  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 260091us
I20260709 13:01:11.184334  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:11.187369  1571 leader_election.cc:304] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
W20260709 13:01:11.214354  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:11.194011  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:11.219447  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:11.219885  1680 raft_consensus.cc:2468] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 14.
I20260709 13:01:11.220448  1749 raft_consensus.cc:2468] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 9.
I20260709 13:01:11.194597  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:11.221338  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:11.221995  1753 raft_consensus.cc:2468] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 14.
I20260709 13:01:11.146317  6013 raft_consensus.cc:515] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:11.217679  6515 raft_consensus.cc:2749] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 13:01:11.194178  1571 leader_election.cc:304] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:11.237597  1752 raft_consensus.cc:1275] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
W20260709 13:01:11.227469  1717 leader_election.cc:336] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 1.648s (SENT)
W20260709 13:01:11.240962  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 46854.8us
I20260709 13:01:11.241113  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:11.185446  6160 raft_consensus.cc:2749] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 13:01:11.247668  1717 leader_election.cc:304] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:11.249511  1675 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52766 (request call id 2640) took 1706 ms. Trace:
I20260709 13:01:11.249933  1675 rpcz_store.cc:276] 0709 13:01:09.535229 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:09.551457 (+ 16228us) service_pool.cc:224] Handling call
0709 13:01:09.551870 (+   413us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:10.434846 (+882976us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:11.241721 (+806875us) spinlock_profiling.cc:243] Waited 851 ms on lock 0x7b54000226d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:01:11.241917 (+   196us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1956978048}
I20260709 13:01:11.217921  6345 raft_consensus.cc:2749] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 13:01:11.252884  1571 leader_election.cc:304] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:01:11.248487  6515 raft_consensus.cc:2804] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Leader election won for term 13
I20260709 13:01:11.256373  6485 raft_consensus.cc:2749] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:01:11.249245  6046 consensus_queue.cc:1059] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 10.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.012s
I20260709 13:01:11.228173  5938 raft_consensus.cc:515] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:11.253376  6509 log.cc:927] Time spent T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.060s	user 0.001s	sys 0.000s
W20260709 13:01:11.269599  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 51238.1us
I20260709 13:01:11.269903  5938 leader_election.cc:290] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:01:11.270462  1717 leader_election.cc:336] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 13 election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 1.689s (SENT)
I20260709 13:01:11.270944  1717 leader_election.cc:304] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:11.254825  6344 consensus_queue.cc:1048] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.017s
I20260709 13:01:11.281945  1571 leader_election.cc:304] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:01:11.280428  1717 leader_election.cc:304] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:11.273699  1646 leader_election.cc:304] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
W20260709 13:01:11.295696  2299 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:11.295177  2301 leader_election.cc:304] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:01:11.303694  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:11.304526  1571 leader_election.cc:304] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:01:11.305094  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:11.305163  6521 consensus_queue.cc:1048] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 13:01:11.305972  1571 leader_election.cc:304] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:01:11.306947  1646 leader_election.cc:304] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:01:11.308142  6046 raft_consensus.cc:2804] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:01:11.308516  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:11.309039  6345 raft_consensus.cc:2749] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 13:01:11.309162  1679 raft_consensus.cc:3060] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:01:11.311359  1717 leader_election.cc:304] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 13:01:11.304482  1754 raft_consensus.cc:2468] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 14.
I20260709 13:01:11.307993  6363 consensus_queue.cc:1059] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 10.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 13:01:11.312903  1717 leader_election.cc:304] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:11.313546  6363 raft_consensus.cc:3055] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:01:11.313786  6487 raft_consensus.cc:2749] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:01:11.314000  6363 raft_consensus.cc:740] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:01:11.302567  6485 raft_consensus.cc:2749] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:01:11.308594  6046 raft_consensus.cc:493] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:11.314388  1717 leader_election.cc:304] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:11.312916  6201 raft_consensus.cc:2804] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:01:11.253005  5989 consensus_queue.cc:1048] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.041s
I20260709 13:01:11.318689  6516 consensus_queue.cc:1048] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.019s
I20260709 13:01:11.292328  6160 raft_consensus.cc:2749] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 13:01:11.301528  2298 leader_election.cc:304] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
W20260709 13:01:11.341394  1569 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:11.332653  1643 leader_election.cc:304] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:11.345980  1717 leader_election.cc:304] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:11.346647  6485 raft_consensus.cc:2749] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
W20260709 13:01:11.360407  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 65473.3us
I20260709 13:01:11.293413  6344 raft_consensus.cc:493] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:11.346994  6201 raft_consensus.cc:493] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:11.334702  6013 leader_election.cc:290] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:01:11.326327  6481 consensus_meta.cc:287] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.769s	user 0.004s	sys 0.010s
I20260709 13:01:11.366149  1569 leader_election.cc:304] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:01:11.322376  6350 raft_consensus.cc:2749] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260709 13:01:11.368453  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 67055.2us
I20260709 13:01:11.373502  2301 leader_election.cc:304] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:01:11.306974  6521 raft_consensus.cc:2749] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 13:01:11.373903  2298 leader_election.cc:304] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:01:11.331535  6496 raft_consensus.cc:2749] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:01:11.324831  5938 raft_consensus.cc:2749] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:01:11.339035  6242 raft_consensus.cc:2749] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 13:01:11.376282  1717 leader_election.cc:304] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:11.377875  1643 leader_election.cc:304] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:01:11.397058  1643 leader_election.cc:304] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:11.368372  6201 raft_consensus.cc:3060] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Advancing to term 11
W20260709 13:01:11.448276  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 54445us
I20260709 13:01:11.411552  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:11.400729  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:11.352489  6046 raft_consensus.cc:3060] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Advancing to term 13
W20260709 13:01:11.397927  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 91061us
I20260709 13:01:11.457778  1678 raft_consensus.cc:2393] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 15: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:01:11.457693  1746 raft_consensus.cc:3060] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Advancing to term 14
W20260709 13:01:11.459811  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 83687.8us
I20260709 13:01:11.456863  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:11.460947  1677 raft_consensus.cc:3060] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:01:11.463224  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:11.461341  1675 raft_consensus.cc:2468] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 14.
I20260709 13:01:11.463804  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:11.464129  1680 raft_consensus.cc:2468] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 12.
I20260709 13:01:11.464493  1672 raft_consensus.cc:2468] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 7.
W20260709 13:01:11.467671  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 161750us
I20260709 13:01:11.471932  1643 leader_election.cc:304] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:01:11.461673  1646 leader_election.cc:304] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:11.368178  6487 raft_consensus.cc:2749] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:01:11.487061  1679 raft_consensus.cc:2468] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 14.
I20260709 13:01:11.494974  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:11.376641  6481 raft_consensus.cc:2749] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:01:11.376001  5989 raft_consensus.cc:2749] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 13:01:11.496830  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:11.497813  1749 raft_consensus.cc:2468] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 13.
I20260709 13:01:11.365724  6407 raft_consensus.cc:2749] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 13:01:11.448041  6485 raft_consensus.cc:2749] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 13:01:11.438079  6345 raft_consensus.cc:3060] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:01:11.507686  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:11.508291  1753 raft_consensus.cc:3060] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:01:11.368808  6344 raft_consensus.cc:515] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:11.503015  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:11.398747  6013 raft_consensus.cc:2749] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 13:01:11.399258  6510 raft_consensus.cc:2804] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:01:11.338840  6363 consensus_queue.cc:260] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:11.525940  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:11.448279  6524 raft_consensus.cc:2749] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:01:11.511549  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:11.529886  1747 raft_consensus.cc:2393] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 9: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:01:11.511689  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:11.512185  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:11.531013  1750 raft_consensus.cc:3060] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Advancing to term 14
I20260709 13:01:11.377167  6512 raft_consensus.cc:2804] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Leader pre-election won for term 10
W20260709 13:01:11.522481  6480 consensus_meta.cc:287] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.731s	user 0.008s	sys 0.005s
I20260709 13:01:11.545858  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:01:11.546109  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 74258.3us
I20260709 13:01:11.527829  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:11.499501  6069 raft_consensus.cc:2749] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 13:01:11.405395  6332 consensus_queue.cc:1048] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.105s
I20260709 13:01:11.526703  1751 raft_consensus.cc:2376] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 9. Re-sending same reply.
I20260709 13:01:11.505406  6516 raft_consensus.cc:493] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:11.524338  5989 raft_consensus.cc:2804] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:01:11.513479  6242 consensus_queue.cc:1059] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 9.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.035s
I20260709 13:01:11.544365  6344 leader_election.cc:290] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:01:11.545106  5994 consensus_meta.cc:287] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.528s	user 0.010s	sys 0.001s
I20260709 13:01:11.535925  6512 raft_consensus.cc:493] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:11.538856  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:11.548029  1746 raft_consensus.cc:2468] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 14.
I20260709 13:01:11.551514  6519 raft_consensus.cc:2804] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:01:11.576109  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:11.546543  1679 raft_consensus.cc:2468] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 8.
I20260709 13:01:11.561758  1643 leader_election.cc:304] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:11.528913  5938 raft_consensus.cc:493] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:11.565171  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:11.583978  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:11.569157  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:11.567188  1604 raft_consensus.cc:1275] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20260709 13:01:11.547642  1749 raft_consensus.cc:2393] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 12: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:01:11.639685  1745 raft_consensus.cc:2368] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 16. Current term is 17.
I20260709 13:01:11.567561  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:11.547976  6510 raft_consensus.cc:697] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 14 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:01:11.624871  1571 leader_election.cc:304] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:01:11.577684  1673 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52766 (request call id 2653) took 1528 ms. Trace:
I20260709 13:01:11.641610  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:11.644111  1673 rpcz_store.cc:276] 0709 13:01:10.049688 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:10.049844 (+   156us) service_pool.cc:224] Handling call
0709 13:01:10.069649 (+ 19805us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:10.794099 (+724450us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:11.577410 (+783311us) spinlock_profiling.cc:243] Waited 724 ms on lock 0x7b5400163d54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:01:11.577611 (+   201us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1664109824}
W20260709 13:01:11.633505  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 71822.8us
I20260709 13:01:11.583603  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:11.568444  1753 raft_consensus.cc:2468] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 14.
I20260709 13:01:11.634685  6524 raft_consensus.cc:2749] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260709 13:01:11.565249  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 104140us
I20260709 13:01:11.652556  1646 leader_election.cc:304] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:11.655987  6529 consensus_queue.cc:1048] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 13:01:11.656625  6046 raft_consensus.cc:515] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:11.635599  6519 raft_consensus.cc:697] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:01:11.548079  1569 leader_election.cc:304] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:01:11.636828  6512 raft_consensus.cc:3060] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Advancing to term 10
W20260709 13:01:11.667382  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 119391us
I20260709 13:01:11.668910  1569 leader_election.cc:304] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:11.596225  1718 leader_election.cc:304] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
W20260709 13:01:11.670939  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 74810.8us
I20260709 13:01:11.671964  1718 leader_election.cc:304] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:11.672849  6485 raft_consensus.cc:2804] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:01:11.673338  6485 raft_consensus.cc:493] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:11.673749  6485 raft_consensus.cc:3060] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:01:11.583060  5994 raft_consensus.cc:515] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:11.675920  6029 raft_consensus.cc:2749] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:01:11.676704  6029 raft_consensus.cc:2804] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:01:11.677124  6029 raft_consensus.cc:493] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:11.677481  6029 raft_consensus.cc:3060] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:01:11.592748  1717 leader_election.cc:304] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:11.559320  5989 raft_consensus.cc:493] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:11.681008  5989 raft_consensus.cc:3060] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:01:11.683003  6510 consensus_queue.cc:237] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.11, Last appended by leader: 11, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:11.681033  6067 raft_consensus.cc:2749] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 13:01:11.688990  1718 leader_election.cc:304] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:11.560822  6516 raft_consensus.cc:515] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:11.692368  6516 leader_election.cc:290] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:01:11.693007  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 68238.3us
I20260709 13:01:11.693842  1571 leader_election.cc:304] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:01:11.694676  6067 raft_consensus.cc:2749] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:01:11.676596  5994 leader_election.cc:290] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 12 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:11.695845  1571 leader_election.cc:304] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 13:01:11.670115  1569 leader_election.cc:304] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:01:11.696676  6067 raft_consensus.cc:2804] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Leader election won for term 9
I20260709 13:01:11.697235  6067 raft_consensus.cc:697] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:01:11.697232  1571 leader_election.cc:304] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:01:11.698406  6067 consensus_queue.cc:237] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:11.568789  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:11.700525  1596 raft_consensus.cc:2468] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 12.
I20260709 13:01:11.577692  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:11.632107  1754 raft_consensus.cc:2468] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 11.
I20260709 13:01:11.566247  6363 raft_consensus.cc:3060] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:01:11.647890  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:11.647891  1677 raft_consensus.cc:2468] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 14.
I20260709 13:01:11.649240  1604 raft_consensus.cc:3060] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:01:11.650990  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:11.661314  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:11.704285  1747 raft_consensus.cc:2376] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 11. Re-sending same reply.
I20260709 13:01:11.701086  1718 leader_election.cc:304] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:11.552050  6480 raft_consensus.cc:515] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:11.578639  6523 raft_consensus.cc:493] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:11.663525  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:11.674650  6501 raft_consensus.cc:2749] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 13:01:11.583134  6527 raft_consensus.cc:493] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:01:11.707316  1571 leader_election.cc:400] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 16 election: Vote denied by peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693) with higher term. Message: Invalid argument: T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 16. Current term is 17.
I20260709 13:01:11.641880  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: true dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:11.662297  6345 raft_consensus.cc:2749] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:01:11.709663  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:11.712036  6046 leader_election.cc:290] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 13 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:11.643088  5938 raft_consensus.cc:515] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:11.713212  6480 leader_election.cc:290] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 14 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:11.702580  6344 raft_consensus.cc:2804] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Leader pre-election won for term 15
W20260709 13:01:11.714244  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 302689us
I20260709 13:01:11.717208  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:11.700078  6529 consensus_queue.cc:1048] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.053s
I20260709 13:01:11.757958  1749 raft_consensus.cc:2393] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 8: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:01:11.731981  1750 raft_consensus.cc:2468] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 14.
I20260709 13:01:11.759848  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
W20260709 13:01:11.733938  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 169184us
I20260709 13:01:11.744328  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:11.760528  1596 raft_consensus.cc:2376] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 10. Re-sending same reply.
I20260709 13:01:11.701954  6496 raft_consensus.cc:493] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:01:11.745697  1602 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40166 (request call id 2337) took 1710 ms (client timeout 1812 ms). Trace:
I20260709 13:01:11.761416  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:11.708333  6497 raft_consensus.cc:2749] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20260709 13:01:11.756358  1644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:11.731946  6140 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.931s	user 0.017s	sys 0.000s
I20260709 13:01:11.766395  1672 raft_consensus.cc:1240] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 8. Current term is 9. Ops: []
I20260709 13:01:11.719692  1603 raft_consensus.cc:3060] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:01:11.757321  2298 leader_election.cc:304] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:01:11.764560  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:11.765542  1754 raft_consensus.cc:2393] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 14: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
W20260709 13:01:11.761274  1602 rpcz_store.cc:269] 0709 13:01:10.035212 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:10.050530 (+ 15318us) service_pool.cc:224] Handling call
0709 13:01:11.745466 (+1694936us) spinlock_profiling.cc:243] Waited 333 ms on lock 0x7b54003edad4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:01:11.745609 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":765677440}
I20260709 13:01:11.762158  1597 raft_consensus.cc:2376] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 9. Re-sending same reply.
W20260709 13:01:11.755322  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 298515us
I20260709 13:01:11.766538  2299 leader_election.cc:304] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:01:11.777066  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:11.775924  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:11.780416  1749 raft_consensus.cc:3060] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:01:11.780148  6408 raft_consensus.cc:2804] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:01:11.787456  6408 raft_consensus.cc:493] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:11.787842  6408 raft_consensus.cc:3060] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:01:11.781189  6496 raft_consensus.cc:515] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:11.791067  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:11.791916  2337 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:35790 (request call id 13) took 1882 ms. Trace:
I20260709 13:01:11.792366  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:11.792311  2337 rpcz_store.cc:276] 0709 13:01:09.909579 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:09.922970 (+ 13391us) service_pool.cc:224] Handling call
0709 13:01:11.791691 (+1868721us) spinlock_profiling.cc:243] Waited 725 ms on lock 0x7b54006b6cd4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f426615c853 00007f426b12a60a 00007f42646b07d4 00007f42646b0762 00007f42646b0691 00007f42646b060c 00007f42646af8c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:01:11.791811 (+   120us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1667084032}
I20260709 13:01:11.793540  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:11.793756  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:11.794323  1754 raft_consensus.cc:3060] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:01:11.766505  6505 raft_consensus.cc:2749] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:01:11.781904  5994 raft_consensus.cc:2804] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:01:11.795717  5994 raft_consensus.cc:493] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:11.796094  5994 raft_consensus.cc:3060] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:01:11.796484  6519 consensus_queue.cc:237] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.7, Last appended by leader: 7, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:11.742790  6531 raft_consensus.cc:2804] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:01:11.799434  6531 raft_consensus.cc:493] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Starting leader election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:01:11.799829  6531 raft_consensus.cc:3060] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:01:11.765179  6501 raft_consensus.cc:2804] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:01:11.800602  6501 raft_consensus.cc:697] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 12 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:01:11.801667  6487 raft_consensus.cc:2804] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 13:01:11.801592  6501 consensus_queue.cc:237] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.7, Last appended by leader: 7, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:11.756735  6344 raft_consensus.cc:493] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:01:11.769286  6140 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.984s	user 0.017s	sys 0.000s
W20260709 13:01:11.863080  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 44486.5us
I20260709 13:01:11.874183  1643 leader_election.cc:304] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:01:11.883708  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:11.885332  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:11.886932  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:11.887743  1750 raft_consensus.cc:2393] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 14: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:01:11.778048  6242 raft_consensus.cc:493] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:01:11.891904  2299 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:11.893335  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 67647.2us
I20260709 13:01:11.796089  6526 raft_consensus.cc:3055] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:01:11.870719  6261 raft_consensus.cc:3060] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:01:11.870014  6344 raft_consensus.cc:3060] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:01:11.950600  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 18 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:11.870170  6046 raft_consensus.cc:493] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:11.719189  6523 raft_consensus.cc:515] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:11.923878  6013 raft_consensus.cc:493] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:11.743342  6527 raft_consensus.cc:515] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:11.928417  6505 raft_consensus.cc:493] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:11.927861  6526 raft_consensus.cc:740] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:01:11.950187  6497 raft_consensus.cc:493] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:11.958040  1750 raft_consensus.cc:2468] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 17.
I20260709 13:01:11.966154  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:11.997601  1599 raft_consensus.cc:3060] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:01:11.929302  5938 leader_election.cc:290] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:11.969877  1603 raft_consensus.cc:2468] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 11.
W20260709 13:01:11.970644  6515 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.714s	user 0.010s	sys 0.021s
W20260709 13:01:11.970907  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 269857us
I20260709 13:01:11.929077  6171 raft_consensus.cc:493] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:11.980676  1571 leader_election.cc:304] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:11.981587  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:11.983362  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:11.984663  1646 leader_election.cc:304] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:01:11.989802  6067 raft_consensus.cc:493] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:01:11.996025  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 121933us
I20260709 13:01:11.999105  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:12.013988  1601 raft_consensus.cc:2468] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 13.
I20260709 13:01:11.943157  6242 raft_consensus.cc:515] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:12.025656  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 45089.2us
I20260709 13:01:12.030648  1646 leader_election.cc:304] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:01:12.030965  1571 leader_election.cc:304] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:12.015728  1754 raft_consensus.cc:2468] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 10.
W20260709 13:01:12.026960  1569 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:12.027473  1749 raft_consensus.cc:2468] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 13.
W20260709 13:01:12.013047  6515 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.746s	user 0.010s	sys 0.022s
I20260709 13:01:11.996187  6529 raft_consensus.cc:2749] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 13:01:12.038100  1571 leader_election.cc:304] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 13:01:12.046627  1599 raft_consensus.cc:2468] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 13.
I20260709 13:01:12.054186  1646 leader_election.cc:304] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:12.060177  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:12.061286  1599 raft_consensus.cc:2393] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 13: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:01:12.047809  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:12.063803  1600 raft_consensus.cc:3060] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:01:12.065210  1643 leader_election.cc:304] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:11.967787  6046 raft_consensus.cc:515] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:12.033703  5938 raft_consensus.cc:2804] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:01:11.991665  6013 raft_consensus.cc:515] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:12.034264  6277 raft_consensus.cc:493] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:01:12.070957  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 478286us
I20260709 13:01:12.076493  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:12.077900  1673 raft_consensus.cc:1240] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 9. Current term is 10. Ops: []
I20260709 13:01:12.036782  6515 raft_consensus.cc:697] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:01:12.011488  6497 raft_consensus.cc:515] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:12.081462  1643 leader_election.cc:304] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:12.017472  6508 consensus_queue.cc:1048] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.052s
I20260709 13:01:12.086650  6546 raft_consensus.cc:3060] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 7 FOLLOWER]: Advancing to term 8
W20260709 13:01:12.087107  6537 log.cc:927] Time spent T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.074s	user 0.000s	sys 0.001s
I20260709 13:01:12.089628  6013 leader_election.cc:290] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:12.031386  6067 raft_consensus.cc:515] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:12.090543  6277 raft_consensus.cc:515] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:12.091677  6527 leader_election.cc:290] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:12.092235  6067 leader_election.cc:290] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:12.092756  6277 leader_election.cc:290] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:12.092870  6067 consensus_queue.cc:1059] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 13.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 13:01:12.094707  1717 leader_election.cc:304] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:12.104703  1597 raft_consensus.cc:2393] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 12: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:01:12.026202  6171 raft_consensus.cc:515] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:12.123492  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:12.037724  5989 raft_consensus.cc:515] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:11.998898  6505 raft_consensus.cc:515] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:12.131973  6542 raft_consensus.cc:493] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
W20260709 13:01:12.133036  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 191577us
W20260709 13:01:12.133738  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 79667.5us
W20260709 13:01:12.144209  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 62846.4us
I20260709 13:01:12.165946  1677 raft_consensus.cc:2468] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 12.
I20260709 13:01:12.144649  6538 raft_consensus.cc:2804] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:01:12.116264  6529 raft_consensus.cc:2749] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:01:12.095800  6534 raft_consensus.cc:493] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:12.153172  1752 raft_consensus.cc:1240] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 10. Current term is 11. Ops: []
I20260709 13:01:12.144826  6533 raft_consensus.cc:493] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:12.144995  6524 raft_consensus.cc:2749] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:01:12.146564  6532 raft_consensus.cc:493] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:12.148279  6364 raft_consensus.cc:493] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
W20260709 13:01:12.157415  1671 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57092 (request call id 1028) took 2176 ms (client timeout 1924 ms). Trace:
I20260709 13:01:12.158237  6527 raft_consensus.cc:493] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:12.158370  6261 raft_consensus.cc:3055] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 LEADER]: Stepping down as leader of term 13
W20260709 13:01:12.158679  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 78127.1us
I20260709 13:01:12.195542  1717 leader_election.cc:304] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:12.163748  1643 leader_election.cc:304] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
W20260709 13:01:12.175918  6201 consensus_meta.cc:287] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.673s	user 0.013s	sys 0.000s
I20260709 13:01:12.104918  5938 raft_consensus.cc:697] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:01:12.105469  6508 consensus_queue.cc:1243] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer af520b67945d443b901bbeca37cc5701 log is divergent from this leader: its last log entry 14.11 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 10
I20260709 13:01:12.161684  2301 leader_election.cc:304] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
W20260709 13:01:12.174233  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 136189us
I20260709 13:01:12.095950  6528 consensus_queue.cc:1048] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.286s
W20260709 13:01:12.190788  1671 rpcz_store.cc:269] 0709 13:01:09.981428 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:09.985111 (+  3683us) service_pool.cc:224] Handling call
0709 13:01:12.157164 (+2172053us) spinlock_profiling.cc:243] Waited 800 ms on lock 0x7b5400347e54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:01:12.157321 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1839136640}
I20260709 13:01:12.160211  6548 raft_consensus.cc:493] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:01:12.152081  6046 leader_election.cc:290] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:12.168668  6538 raft_consensus.cc:493] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:12.210898  6201 raft_consensus.cc:515] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:12.227793  6538 raft_consensus.cc:3060] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:01:12.229866  6201 leader_election.cc:290] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:12.232126  6082 leader_election.cc:290] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:12.163479  6554 raft_consensus.cc:2749] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:01:12.234797  6201 raft_consensus.cc:2749] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 13:01:12.208662  5994 raft_consensus.cc:515] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:12.210870  1604 consensus_meta.cc:287] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.502s	user 0.012s	sys 0.003s
I20260709 13:01:12.236204  1604 raft_consensus.cc:2468] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 11.
I20260709 13:01:12.237284  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 12 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:12.237417  6496 leader_election.cc:290] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:12.238418  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:12.239035  1604 raft_consensus.cc:3060] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:01:12.214175  1571 leader_election.cc:304] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:12.160853  6512 raft_consensus.cc:515] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:12.227216  6289 raft_consensus.cc:493] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:01:12.232403  6543 log.cc:927] Time spent T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.081s	user 0.000s	sys 0.001s
I20260709 13:01:12.232787  6536 raft_consensus.cc:2804] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:01:12.159271  6515 consensus_queue.cc:237] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:12.172818  6485 raft_consensus.cc:515] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:12.257032  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:01:12.198891  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 150234us
I20260709 13:01:12.197154  6261 raft_consensus.cc:740] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
W20260709 13:01:12.257915  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 96334us
I20260709 13:01:12.261667  2301 leader_election.cc:304] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
W20260709 13:01:12.219914  6029 consensus_meta.cc:287] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.542s	user 0.009s	sys 0.004s
W20260709 13:01:12.252029  2299 leader_election.cc:336] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 13 election: RPC error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Timed out: RequestConsensusVote RPC to 127.0.107.194:46261 timed out after 1.924s (SENT)
I20260709 13:01:12.165261  6542 raft_consensus.cc:515] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:12.224381  1646 leader_election.cc:304] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:12.267329  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:01:12.269512  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 45306.7us
I20260709 13:01:12.297530  1646 leader_election.cc:304] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:12.269550  1752 raft_consensus.cc:2468] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 8.
I20260709 13:01:12.300148  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:12.300930  1752 raft_consensus.cc:3060] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:01:12.205040  6364 raft_consensus.cc:515] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:12.272168  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:12.273366  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:12.274451  2332 raft_consensus.cc:1275] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20260709 13:01:12.310868  2334 raft_consensus.cc:3060] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:01:12.278748  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:12.294920  2299 leader_election.cc:304] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:01:12.270268  1604 raft_consensus.cc:2468] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 14.
I20260709 13:01:12.315599  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:12.197671  6532 raft_consensus.cc:515] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:12.279639  2333 raft_consensus.cc:1275] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20260709 13:01:12.321802  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:12.322060  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:12.322538  1671 raft_consensus.cc:2468] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 11.
I20260709 13:01:12.323506  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:12.324136  2333 raft_consensus.cc:2468] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 11.
I20260709 13:01:12.277874  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:12.308470  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:12.326133  6530 raft_consensus.cc:2749] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 13:01:12.351506  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:12.352439  1750 raft_consensus.cc:2468] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 10.
I20260709 13:01:12.353773  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:12.355356  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:12.356417  1753 raft_consensus.cc:2410] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 for term 12 because replica has last-logged OpId of term: 10 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 9.
I20260709 13:01:12.357486  1753 raft_consensus.cc:1240] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 7. Current term is 8. Ops: []
W20260709 13:01:12.352470  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 55078.5us
I20260709 13:01:12.365046  1675 raft_consensus.cc:1275] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20260709 13:01:12.369077  6344 raft_consensus.cc:515] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:12.304332  6046 raft_consensus.cc:2804] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Leader election won for term 13
I20260709 13:01:12.371315  6046 raft_consensus.cc:697] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:01:12.372373  6046 consensus_queue.cc:237] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 11.8, Last appended by leader: 8, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:12.267936  6541 raft_consensus.cc:2764] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Leader pre-election decision vote started in defunct term 9: won
I20260709 13:01:12.377453  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:12.309371  5994 leader_election.cc:290] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 14 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:12.266628  6550 raft_consensus.cc:2749] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:01:12.267345  6536 raft_consensus.cc:697] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:01:12.384509  6536 consensus_queue.cc:237] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:12.268165  6029 raft_consensus.cc:515] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:12.389183  5989 leader_election.cc:290] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 14 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:12.393141  6497 leader_election.cc:290] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 13:01:12.394160  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 230474us
I20260709 13:01:12.396972  1643 leader_election.cc:304] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:12.397619  1750 raft_consensus.cc:2393] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 12: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:01:12.403663  2331 raft_consensus.cc:1275] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20260709 13:01:12.404961  6532 leader_election.cc:290] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:12.316293  1604 raft_consensus.cc:2468] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 12.
I20260709 13:01:12.294958  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:12.202231  6527 raft_consensus.cc:515] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:12.272895  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:12.258977  6082 raft_consensus.cc:2804] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:01:12.266937  6487 raft_consensus.cc:697] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 11 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:01:12.198041  6533 raft_consensus.cc:515] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:12.177486  6534 raft_consensus.cc:515] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:12.444029  1750 raft_consensus.cc:3055] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 13:01:12.444666  1750 raft_consensus.cc:740] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:01:12.259653  6524 raft_consensus.cc:2804] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader election won for term 9
I20260709 13:01:12.326928  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:12.327348  6505 leader_election.cc:290] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:12.452847  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:12.328150  6261 consensus_queue.cc:260] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.11, Last appended by leader: 11, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:12.328655  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:12.456405  2335 raft_consensus.cc:3060] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:01:12.259256  6289 raft_consensus.cc:515] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:12.316649  6523 leader_election.cc:290] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:12.238783  6548 raft_consensus.cc:515] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:12.329481  1569 leader_election.cc:304] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:01:12.330659  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:12.332304  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:12.334897  5938 consensus_queue.cc:237] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.8, Last appended by leader: 8, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:12.318985  6554 raft_consensus.cc:2804] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Leader election won for term 10
W20260709 13:01:12.336889  6408 consensus_meta.cc:287] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.549s	user 0.014s	sys 0.000s
I20260709 13:01:12.337272  6512 leader_election.cc:290] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 10 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:12.339881  6546 raft_consensus.cc:2749] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:01:12.342118  6364 leader_election.cc:290] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:01:12.343912  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 91882.8us
I20260709 13:01:12.345938  6171 leader_election.cc:290] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:12.411019  1643 leader_election.cc:304] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:01:12.435010  2334 raft_consensus.cc:2468] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 14.
I20260709 13:01:12.438390  1600 raft_consensus.cc:2468] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 12.
I20260709 13:01:12.441076  2328 raft_consensus.cc:2468] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 14.
I20260709 13:01:12.443192  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:12.498208  1751 raft_consensus.cc:2468] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 12.
I20260709 13:01:12.498634  6554 raft_consensus.cc:697] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
W20260709 13:01:12.499401  6517 log.cc:927] Time spent T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 1.286s	user 0.009s	sys 0.013s
I20260709 13:01:12.499625  1751 pending_rounds.cc:85] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701: Aborting all ops after (but not including) 10
I20260709 13:01:12.500123  1751 pending_rounds.cc:107] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701: Aborting uncommitted NO_OP operation due to leader change: 14.11
I20260709 13:01:12.499770  6554 consensus_queue.cc:237] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:12.501127  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:12.326520  1679 raft_consensus.cc:2393] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 14: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:01:12.421484  6082 raft_consensus.cc:493] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:12.509063  1605 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 1070) took 2052 ms. Trace:
I20260709 13:01:12.509250  6082 raft_consensus.cc:3060] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:01:12.444613  1752 raft_consensus.cc:2468] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 12.
I20260709 13:01:12.509469  1605 rpcz_store.cc:276] 0709 13:01:10.456824 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:10.568654 (+111830us) service_pool.cc:224] Handling call
0709 13:01:10.585745 (+ 17091us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:10.737792 (+152047us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:12.508789 (+1770997us) spinlock_profiling.cc:243] Waited 121 ms on lock 0x7b540028a854. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:01:12.508967 (+   178us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":277990656}
I20260709 13:01:12.450515  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 18 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:12.351959  6485 leader_election.cc:290] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 15 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:01:12.513000  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 102106us
I20260709 13:01:12.327598  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:01:12.514487  6518 log.cc:927] Time spent T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 1.249s	user 0.008s	sys 0.012s
I20260709 13:01:12.515667  6453 raft_consensus.cc:2749] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:01:12.456694  1745 raft_consensus.cc:2393] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 13: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:01:12.427922  5989 raft_consensus.cc:3060] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:01:12.518697  6277 raft_consensus.cc:2764] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader election decision vote started in defunct term 11: won
I20260709 13:01:12.530217  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:12.466034  6524 raft_consensus.cc:697] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:01:12.459542  6556 raft_consensus.cc:3060] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Advancing to term 15
W20260709 13:01:12.468814  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 139399us
I20260709 13:01:12.467921  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:12.544319  1753 raft_consensus.cc:2393] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 15: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:01:12.473887  1746 raft_consensus.cc:1275] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 9 index: 10. (index mismatch)
I20260709 13:01:12.546957  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:12.549420  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:12.500456  1751 raft_consensus.cc:2889] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:01:12.502527  2332 raft_consensus.cc:1240] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 13. Current term is 14. Ops: []
I20260709 13:01:12.504879  6481 raft_consensus.cc:3055] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:01:12.490964  6408 raft_consensus.cc:515] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:12.559188  1676 raft_consensus.cc:1240] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 14. Current term is 15. Ops: []
I20260709 13:01:12.507580  6542 leader_election.cc:290] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:12.507787  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:12.508244  1748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43156 (request call id 925) took 2023 ms. Trace:
I20260709 13:01:12.508494  2328 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23"
dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
mode: GRACEFUL
new_leader_uuid: "af520b67945d443b901bbeca37cc5701"
 from {username='slave'} at 127.0.0.1:35790
I20260709 13:01:12.560251  2334 raft_consensus.cc:2468] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 11.
I20260709 13:01:12.560992  2328 raft_consensus.cc:606] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Received request to transfer leadership to TS af520b67945d443b901bbeca37cc5701
I20260709 13:01:12.561478  2328 raft_consensus.cc:614] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 13:01:12.539252  2335 raft_consensus.cc:2468] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 14.
I20260709 13:01:12.471338  6261 raft_consensus.cc:3060] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Advancing to term 14
W20260709 13:01:12.450549  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 255065us
I20260709 13:01:12.563795  1717 leader_election.cc:304] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 13:01:12.475412  6523 raft_consensus.cc:2804] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:01:12.542733  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:12.476366  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:12.550752  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:12.567358  1679 raft_consensus.cc:2393] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 15: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:01:12.555069  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:12.552096  1644 leader_election.cc:304] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:12.446247  1750 consensus_queue.cc:260] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.11, Last appended by leader: 11, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:12.568420  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:12.568703  1750 raft_consensus.cc:3060] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:01:12.571143  1643 leader_election.cc:304] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:01:12.541491  6512 raft_consensus.cc:493] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:01:12.557675  6531 consensus_meta.cc:287] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.757s	user 0.017s	sys 0.000s
I20260709 13:01:12.578576  1747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55840 (request call id 2601) took 1027 ms. Trace:
I20260709 13:01:12.582062  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:12.582648  1717 leader_election.cc:304] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:12.583016  2330 raft_consensus.cc:2468] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 11.
I20260709 13:01:12.586124  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:12.586429  6160 raft_consensus.cc:2804] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:01:12.587002  1676 raft_consensus.cc:2468] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 13.
I20260709 13:01:12.534686  6298 consensus_queue.cc:1048] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.044s
I20260709 13:01:12.572512  6523 raft_consensus.cc:493] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:12.588454  6523 raft_consensus.cc:3060] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:01:12.589268  6298 raft_consensus.cc:3060] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:01:12.587006  6160 raft_consensus.cc:493] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:12.590354  6160 raft_consensus.cc:3060] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:01:12.564589  6487 consensus_queue.cc:237] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:12.592803  6370 raft_consensus.cc:3060] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:01:12.560104  1748 rpcz_store.cc:276] 0709 13:01:10.485496 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:10.492095 (+  6599us) service_pool.cc:224] Handling call
0709 13:01:10.497356 (+  5261us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:10.631064 (+133708us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:10.631103 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:01:10.647944 (+ 16841us) log.cc:844] Serialized 26 byte log entry
0709 13:01:12.201515 (+1553571us) spinlock_profiling.cc:243] Waited 1.55 ms on lock 0x7b18002bac80. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f4263057e07 00007f426beb2ab9 00007f426307469b 00007f426307293f 00007f4263074c02 00007f4265cfc20b 00007f4265d01d23 00007f4265d020d1 00007f426611da0f
0709 13:01:12.441603 (+240088us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:01:12.441646 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:12.441711 (+    65us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:12.507886 (+ 66175us) raft_consensus.cc:1685] finished
0709 13:01:12.507911 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:12.508172 (+   261us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":37665,"raft.queue_time_us":162349,"raft.run_cpu_time_us":197,"raft.run_wall_time_us":15338,"spinlock_wait_cycles":3558272,"thread_start_us":349037,"threads_started":2,"wal-append.queue_time_us":1833779}
I20260709 13:01:12.547276  6345 raft_consensus.cc:493] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:01:12.595283  6345 raft_consensus.cc:515] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:12.596333  6524 consensus_queue.cc:237] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } }
I20260709 13:01:12.558853  6481 raft_consensus.cc:740] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:01:12.598242  6481 consensus_queue.cc:260] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.8, Last appended by leader: 8, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:12.598719  6531 raft_consensus.cc:515] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:12.601070  6531 leader_election.cc:290] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 9 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:12.601572  6408 leader_election.cc:290] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 15 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:01:12.604020  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 52034.4us
I20260709 13:01:12.605130  1644 leader_election.cc:304] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:01:12.605548  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:12.606829  1596 raft_consensus.cc:2393] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 9: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:01:12.608273  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:12.612722  1644 leader_election.cc:304] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:12.614585  1717 leader_election.cc:304] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:12.559557  6344 leader_election.cc:290] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 15 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:12.627563  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:12.628314  2329 raft_consensus.cc:2393] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 15: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
W20260709 13:01:12.627225  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 57037.5us
I20260709 13:01:12.579018  1747 rpcz_store.cc:276] 0709 13:01:11.551112 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:11.710484 (+159372us) service_pool.cc:224] Handling call
0709 13:01:11.710896 (+   412us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:11.870753 (+159857us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:11.870803 (+    50us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:01:11.891122 (+ 20319us) log.cc:844] Serialized 26 byte log entry
0709 13:01:12.211353 (+320231us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:01:12.263280 (+ 51927us) log.cc:844] Serialized 14 byte log entry
0709 13:01:12.263746 (+   466us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:12.263848 (+   102us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:12.500709 (+236861us) spinlock_profiling.cc:243] Waited 1.7 ms on lock 0x7b18003768a0. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f42630753f3 00007f42630777c1 00007f4263077779 00007f4263077709 00007f42630776d1 00007f42630769cd 000000000038f354 000000000038f199 00007f426305883a
0709 13:01:12.566264 (+ 65555us) raft_consensus.cc:1685] finished
0709 13:01:12.578316 (+ 12052us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:12.578495 (+   179us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8990,"prepare.queue_time_us":39330,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":50,"raft.queue_time_us":164687,"raft.run_cpu_time_us":197,"raft.run_wall_time_us":43686,"spinlock_wait_cycles":32868608,"thread_start_us":338648,"threads_started":2,"wal-append.queue_time_us":362771}
I20260709 13:01:12.632498  1748 raft_consensus.cc:2393] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 15: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:01:12.632977  1644 leader_election.cc:304] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:01:12.563154  6364 raft_consensus.cc:493] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:12.631282  1718 leader_election.cc:304] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:12.642087  1644 leader_election.cc:304] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:01:12.559281  6171 consensus_queue.cc:1048] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.081s
I20260709 13:01:12.630586  6497 raft_consensus.cc:2749] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 13:01:12.669220  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:12.670408  1718 leader_election.cc:304] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:12.634908  6546 raft_consensus.cc:2749] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 13:01:12.638201  6528 raft_consensus.cc:2804] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:01:12.630796  6568 raft_consensus.cc:2749] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:01:12.641289  6029 leader_election.cc:290] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 9 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:12.681397  1750 raft_consensus.cc:1275] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 11 index: 11. Preceding OpId from leader: term: 12 index: 11. (term mismatch)
I20260709 13:01:12.681792  1750 pending_rounds.cc:85] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701: Aborting all ops after (but not including) 10
I20260709 13:01:12.682065  1750 pending_rounds.cc:107] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701: Aborting uncommitted NO_OP operation due to leader change: 11.11
I20260709 13:01:12.663944  6536 raft_consensus.cc:2804] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:01:12.682395  1750 raft_consensus.cc:2889] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:01:12.679062  1718 leader_election.cc:304] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:12.628199  6512 raft_consensus.cc:515] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:12.684650  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:12.687350  1718 leader_election.cc:304] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:12.671705  6344 raft_consensus.cc:2804] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:01:12.674782  6485 raft_consensus.cc:2804] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:01:12.698356  1749 raft_consensus.cc:2393] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 12: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
W20260709 13:01:12.706869  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 92405.7us
I20260709 13:01:12.679880  6497 raft_consensus.cc:2749] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:01:12.682126  6345 leader_election.cc:290] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:12.648350  6364 raft_consensus.cc:515] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:12.686275  6363 consensus_meta.cc:287] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.970s	user 0.010s	sys 0.005s
W20260709 13:01:12.715577  1644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:12.696695  6201 raft_consensus.cc:2804] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:01:12.690752  6544 raft_consensus.cc:2804] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 13:01:12.687434  6528 raft_consensus.cc:697] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:01:12.731129  1717 leader_election.cc:400] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 13 election: Vote denied by peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445) with higher term. Message: Invalid argument: T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 13. Current term is 14.
I20260709 13:01:12.687602  6536 raft_consensus.cc:493] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:12.708817  6485 raft_consensus.cc:493] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:12.762388  6029 raft_consensus.cc:493] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:12.771786  6029 raft_consensus.cc:515] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:12.772300  6528 consensus_queue.cc:237] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.11, Last appended by leader: 11, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:12.774362  6029 leader_election.cc:290] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:12.777216  6544 raft_consensus.cc:697] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 8 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:01:12.747232  6534 leader_election.cc:290] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:12.778354  6544 consensus_queue.cc:237] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:12.792626  6526 consensus_queue.cc:260] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:12.793946  6526 raft_consensus.cc:3060] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:01:12.794275  6160 raft_consensus.cc:515] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:12.796268  6242 leader_election.cc:290] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:12.796810  6160 leader_election.cc:290] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 13 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:12.800547  6298 raft_consensus.cc:2749] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:01:12.801731  6082 raft_consensus.cc:515] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:12.803017  1598 raft_consensus.cc:1240] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 17. Current term is 18. Ops: [17.12-17.12]
I20260709 13:01:12.807425  6082 leader_election.cc:290] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 18 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:12.816691  1603 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40166 (request call id 2384) took 1063 ms. Trace:
I20260709 13:01:12.817384  1603 rpcz_store.cc:276] 0709 13:01:11.753340 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:11.998744 (+245404us) service_pool.cc:224] Handling call
0709 13:01:12.816329 (+817585us) spinlock_profiling.cc:243] Waited 459 ms on lock 0x7b540006d6d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:01:12.816577 (+   248us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1055711488}
I20260709 13:01:12.726718  6561 raft_consensus.cc:2749] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 13:01:12.826385  1596 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40166 (request call id 2382) took 1075 ms. Trace:
I20260709 13:01:12.827069  1596 rpcz_store.cc:276] 0709 13:01:11.750726 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:11.802737 (+ 52011us) service_pool.cc:224] Handling call
0709 13:01:12.826105 (+1023368us) spinlock_profiling.cc:243] Waited 722 ms on lock 0x7b54002166d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:01:12.826277 (+   172us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1660171008}
I20260709 13:01:12.713346  6564 consensus_queue.cc:1059] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 14.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.167s
I20260709 13:01:12.832101  6485 raft_consensus.cc:3060] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:01:12.861447  6527 leader_election.cc:290] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:01:12.850461  6538 consensus_meta.cc:287] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.622s	user 0.006s	sys 0.009s
I20260709 13:01:12.867290  6538 raft_consensus.cc:515] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:12.869040  1599 raft_consensus.cc:2393] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 12: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:01:12.873888  6507 raft_consensus.cc:3055] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:01:12.874511  6507 raft_consensus.cc:740] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:01:12.874749  6364 leader_election.cc:290] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:12.875488  6507 consensus_queue.cc:260] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:12.876695  6507 raft_consensus.cc:3060] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:01:12.878048  1569 leader_election.cc:304] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:01:12.879916  6277 raft_consensus.cc:2749] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20260709 13:01:12.847477  6543 log.cc:927] Time spent T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.085s	user 0.000s	sys 0.000s
I20260709 13:01:12.903362  6512 leader_election.cc:290] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:12.904297  6542 raft_consensus.cc:3055] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:01:12.904840  6542 raft_consensus.cc:740] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:01:12.915521  1601 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40166 (request call id 2373) took 1351 ms. Trace:
I20260709 13:01:12.916013  1601 rpcz_store.cc:276] 0709 13:01:11.565033 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:11.568875 (+  3842us) service_pool.cc:224] Handling call
0709 13:01:12.915313 (+1346438us) spinlock_profiling.cc:243] Waited 375 ms on lock 0x7b54002990d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:01:12.915432 (+   119us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":861545216}
I20260709 13:01:12.917441  6529 raft_consensus.cc:493] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:12.923480  1646 leader_election.cc:304] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:01:12.924185  1745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55840 (request call id 2598) took 1374 ms. Trace:
I20260709 13:01:12.924768  1745 rpcz_store.cc:276] 0709 13:01:11.550039 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:11.650681 (+100642us) service_pool.cc:224] Handling call
0709 13:01:12.923874 (+1273193us) spinlock_profiling.cc:243] Waited 752 ms on lock 0x7b5400384d54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:01:12.924063 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1730029952}
I20260709 13:01:12.905907  6542 consensus_queue.cc:260] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.10, Last appended by leader: 10, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:12.926903  1570 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:12.930261  2301 leader_election.cc:304] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:12.931226  6360 raft_consensus.cc:2804] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:01:12.931834  6360 raft_consensus.cc:697] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 12 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:01:12.932842  6360 consensus_queue.cc:237] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.8, Last appended by leader: 8, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:12.933197  6527 raft_consensus.cc:493] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:12.934849  6527 raft_consensus.cc:515] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } }
I20260709 13:01:12.935220  6515 raft_consensus.cc:493] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:12.936813  6515 raft_consensus.cc:515] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:12.937160  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:01:12.938462  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 47013.6us
I20260709 13:01:12.939340  1569 leader_election.cc:304] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:12.934149  6555 consensus_queue.cc:1048] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.002s
I20260709 13:01:12.940429  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 18 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:12.900985  1672 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52766 (request call id 2662) took 1435 ms. Trace:
I20260709 13:01:12.941257  1672 rpcz_store.cc:276] 0709 13:01:11.465403 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:11.496924 (+ 31521us) service_pool.cc:224] Handling call
0709 13:01:11.519321 (+ 22397us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:11.785535 (+266214us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:12.900644 (+1115109us) spinlock_profiling.cc:243] Waited 244 ms on lock 0x7b54001e87d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:01:12.900879 (+   235us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":562218880}
I20260709 13:01:12.918094  6529 raft_consensus.cc:515] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:12.832638  6289 leader_election.cc:290] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:12.947289  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:12.948264  2330 raft_consensus.cc:2468] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 13.
I20260709 13:01:12.946182  2298 leader_election.cc:304] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:01:12.950166  2298 leader_election.cc:304] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:12.950827  6289 raft_consensus.cc:2804] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:01:12.956046  6289 raft_consensus.cc:697] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [term 14 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:01:12.956416  6542 raft_consensus.cc:3060] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:01:12.957329  6289 consensus_queue.cc:237] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.11, Last appended by leader: 11, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:12.972119  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 284799us
I20260709 13:01:13.019321  6515 leader_election.cc:290] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:13.022485  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:13.024803  6527 leader_election.cc:290] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:13.024869  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:13.026680  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:13.027354  1599 raft_consensus.cc:2468] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 12.
I20260709 13:01:13.028213  6556 raft_consensus.cc:2749] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:01:13.028748  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:13.029562  1599 raft_consensus.cc:2393] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 11: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:01:12.951174  6536 raft_consensus.cc:3060] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:01:13.030795  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:01:13.032037  1571 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:13.032533  1571 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:13.032709  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:13.035331  1599 raft_consensus.cc:1275] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 14 index: 12. (index mismatch)
W20260709 13:01:13.037135  5989 consensus_meta.cc:287] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.518s	user 0.007s	sys 0.002s
I20260709 13:01:13.037648  5989 raft_consensus.cc:2749] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:01:13.019122  6481 raft_consensus.cc:3060] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:01:13.038419  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:13.040180  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:13.042716  2330 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147"
dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
mode: GRACEFUL
new_leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
 from {username='slave'} at 127.0.0.1:35790
W20260709 13:01:13.047206  1570 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:13.047629  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 15 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:13.106515  2331 raft_consensus.cc:2410] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for term 15 because replica has last-logged OpId of term: 13 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 9.
I20260709 13:01:13.105182  1676 raft_consensus.cc:2468] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 12.
I20260709 13:01:13.055742  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
W20260709 13:01:13.063608  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 140202us
I20260709 13:01:13.049229  2336 raft_consensus.cc:1240] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 14. Current term is 15. Ops: []
I20260709 13:01:13.082913  1571 leader_election.cc:304] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:01:13.057883  6542 raft_consensus.cc:493] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:13.065917  6160 raft_consensus.cc:493] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:13.118281  6505 raft_consensus.cc:493] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:01:13.032222  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 82147.9us
I20260709 13:01:13.089234  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:13.120672  1677 raft_consensus.cc:2468] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 13.
I20260709 13:01:13.067828  6298 raft_consensus.cc:2749] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 13:01:13.067000  6548 leader_election.cc:290] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:13.094116  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:13.074687  6277 raft_consensus.cc:2749] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:01:13.092731  6171 raft_consensus.cc:493] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:13.101639  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:13.103207  1717 leader_election.cc:403] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 13 election: Cancelling election due to peer responding with higher term
I20260709 13:01:13.101222  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:13.108804  2330 raft_consensus.cc:606] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Received request to transfer leadership to TS 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:13.133963  2330 raft_consensus.cc:614] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260709 13:01:13.072976  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 133741us
I20260709 13:01:13.093084  6332 raft_consensus.cc:493] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:13.140897  6332 raft_consensus.cc:515] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:13.143024  2333 raft_consensus.cc:1240] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 9. Current term is 10. Ops: []
I20260709 13:01:13.089143  6546 raft_consensus.cc:493] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:01:13.113344  1571 leader_election.cc:304] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:13.144771  6546 raft_consensus.cc:515] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:13.143061  6357 raft_consensus.cc:493] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:13.147070  6555 raft_consensus.cc:2804] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:01:13.113993  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:12.747627  6201 raft_consensus.cc:493] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:13.122934  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:13.153090  1749 raft_consensus.cc:2468] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 12.
I20260709 13:01:13.123692  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 18 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
W20260709 13:01:13.069855  6453 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.523s	user 0.004s	sys 0.002s
I20260709 13:01:13.130095  1679 raft_consensus.cc:3060] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:01:13.139410  6578 raft_consensus.cc:2749] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Leader election lost for term 13. Reason: Vote denied by peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445) with higher term. Message: Invalid argument: T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 13. Current term is 14.
I20260709 13:01:13.127761  6160 raft_consensus.cc:515] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:13.115815  6507 raft_consensus.cc:2749] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:01:13.139923  6171 raft_consensus.cc:515] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:13.147011  6357 raft_consensus.cc:515] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:13.148653  1571 leader_election.cc:304] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:01:13.161463  1677 raft_consensus.cc:1275] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 14 index: 12. (index mismatch)
I20260709 13:01:13.151870  1672 raft_consensus.cc:2393] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 15: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:01:13.147164  6546 leader_election.cc:290] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:01:13.159915  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 428843us
I20260709 13:01:12.757442  6344 raft_consensus.cc:697] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:01:13.164750  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:13.189239  1717 leader_election.cc:304] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:13.191627  1569 leader_election.cc:304] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
W20260709 13:01:13.157526  6453 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.624s	user 0.005s	sys 0.002s
I20260709 13:01:13.116891  6046 consensus_queue.cc:1048] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.047s
I20260709 13:01:13.147716  6555 raft_consensus.cc:493] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:01:13.204048  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 57398.8us
W20260709 13:01:13.209659  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:13.211371  1717 leader_election.cc:304] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:13.118883  6505 raft_consensus.cc:515] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:13.119251  6542 raft_consensus.cc:515] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:13.197258  1569 leader_election.cc:304] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:01:13.158936  6201 raft_consensus.cc:3060] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:01:13.148906  6529 leader_election.cc:290] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:13.219017  2299 leader_election.cc:304] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:01:13.163955  6538 leader_election.cc:290] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 12 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:13.194110  6350 raft_consensus.cc:3060] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:01:13.232970  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 15 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:01:13.216732  6523 consensus_meta.cc:287] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.628s	user 0.010s	sys 0.004s
I20260709 13:01:13.240913  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:13.212373  6277 raft_consensus.cc:2749] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:01:13.212384  6487 raft_consensus.cc:2749] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:01:13.292996  1749 raft_consensus.cc:1240] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 10. Current term is 11. Ops: []
I20260709 13:01:13.227873  6555 raft_consensus.cc:3060] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Advancing to term 11
W20260709 13:01:13.262702  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 43870.3us
I20260709 13:01:13.226964  6546 raft_consensus.cc:493] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:13.317512  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:13.338176  1747 raft_consensus.cc:2468] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 12.
I20260709 13:01:13.300547  1677 raft_consensus.cc:1275] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 12 index: 6. (index mismatch)
I20260709 13:01:13.316918  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:13.340059  2329 raft_consensus.cc:3060] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Advancing to term 12
W20260709 13:01:13.322424  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 111191us
I20260709 13:01:13.323099  1599 raft_consensus.cc:1240] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 10. Current term is 11. Ops: [10.8-10.8]
I20260709 13:01:13.233397  1643 leader_election.cc:304] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:13.324806  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:13.329336  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:13.331135  1749 raft_consensus.cc:1240] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 12. Current term is 13. Ops: []
I20260709 13:01:13.336026  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:13.337420  1679 raft_consensus.cc:2468] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 14.
I20260709 13:01:13.321952  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:01:13.335291  6519 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.298s	user 0.011s	sys 0.037s
I20260709 13:01:13.231696  6507 consensus_queue.cc:1048] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.105s
W20260709 13:01:13.391764  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 158475us
I20260709 13:01:13.392354  2333 raft_consensus.cc:1275] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 14 index: 12. (index mismatch)
I20260709 13:01:13.393705  6529 raft_consensus.cc:2749] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:01:13.233701  6578 raft_consensus.cc:2749] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:01:13.322507  6513 raft_consensus.cc:2804] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:01:13.424615  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:13.439085  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:13.402709  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:13.429960  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:13.444764  1677 raft_consensus.cc:3060] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:01:13.445191  1674 raft_consensus.cc:2468] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 11.
I20260709 13:01:13.422081  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:13.395051  1745 raft_consensus.cc:2468] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 12.
I20260709 13:01:13.446336  1678 raft_consensus.cc:2468] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 11.
I20260709 13:01:13.447538  1643 leader_election.cc:304] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:01:13.397553  1680 raft_consensus.cc:1240] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 7. Current term is 8. Ops: []
I20260709 13:01:13.399070  1600 raft_consensus.cc:2468] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 12.
I20260709 13:01:13.398614  1753 raft_consensus.cc:2468] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 9.
I20260709 13:01:13.444052  1675 raft_consensus.cc:2468] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 9.
I20260709 13:01:13.453770  1569 leader_election.cc:304] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:01:13.336680  6480 raft_consensus.cc:493] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:13.460980  1643 leader_election.cc:304] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:01:13.483423  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:13.256623  6082 raft_consensus.cc:2804] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:01:13.480404  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:13.489946  1678 raft_consensus.cc:2468] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 12.
I20260709 13:01:13.410804  6332 leader_election.cc:290] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:01:13.395208  6519 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.355s	user 0.013s	sys 0.039s
I20260709 13:01:13.415256  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:13.546816  1673 raft_consensus.cc:2393] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 13: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:01:13.549538  1673 raft_consensus.cc:1275] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
I20260709 13:01:13.549911  6332 raft_consensus.cc:493] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:13.551072  1673 raft_consensus.cc:1275] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
I20260709 13:01:13.550709  6332 raft_consensus.cc:515] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:13.553993  6332 leader_election.cc:290] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:01:13.541293  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 80402.1us
W20260709 13:01:13.541285  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 194523us
I20260709 13:01:13.542429  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:13.560352  2336 raft_consensus.cc:2468] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 10.
I20260709 13:01:13.564513  2329 raft_consensus.cc:2468] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 12.
I20260709 13:01:13.402256  6535 consensus_queue.cc:1048] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.402s
I20260709 13:01:13.411636  6160 leader_election.cc:290] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:13.407752  6171 leader_election.cc:290] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:01:13.488137  6501 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.606s	user 0.022s	sys 0.029s
W20260709 13:01:13.571045  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:13.572122  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:13.438839  6542 leader_election.cc:290] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:13.570334  2299 leader_election.cc:304] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:01:13.424867  6357 leader_election.cc:290] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:13.479120  6524 raft_consensus.cc:2749] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 13:01:13.423621  6505 leader_election.cc:290] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:13.592664  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:13.593173  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:13.396618  6013 consensus_queue.cc:1059] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 9.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.178s
I20260709 13:01:13.593464  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 12 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:13.594218  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:01:13.594616  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 140958us
I20260709 13:01:13.626451  1601 raft_consensus.cc:2410] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for term 12 because replica has last-logged OpId of term: 11 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 5.
I20260709 13:01:13.437652  1500 catalog_manager.cc:5697] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 5 to 7, leader changed from 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) to af520b67945d443b901bbeca37cc5701 (127.0.107.195). New cstate: current_term: 7 leader_uuid: "af520b67945d443b901bbeca37cc5701" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } }
I20260709 13:01:13.595798  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:13.459581  6513 raft_consensus.cc:493] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:13.607939  1571 leader_election.cc:304] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:13.314181  6523 raft_consensus.cc:515] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:13.470371  6544 raft_consensus.cc:493] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:01:13.370180  6546 raft_consensus.cc:515] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:13.371635  6360 consensus_queue.cc:1059] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 10.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.171s
W20260709 13:01:13.620122  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 261868us
I20260709 13:01:13.707496  2298 leader_election.cc:304] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:01:13.709334  2298 leader_election.cc:304] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:01:13.614931  2299 leader_election.cc:304] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:01:13.491178  6082 raft_consensus.cc:493] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:13.580222  6487 raft_consensus.cc:493] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:13.723598  6487 raft_consensus.cc:515] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:13.621244  6516 raft_consensus.cc:2804] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:01:13.631297  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:13.621523  6515 raft_consensus.cc:2804] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:01:13.631655  6513 raft_consensus.cc:3060] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:01:13.621706  6067 raft_consensus.cc:493] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:13.632299  6535 raft_consensus.cc:493] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:13.632655  6533 leader_election.cc:290] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:13.635311   431 auto_leader_rebalancer.cc:452] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260709 13:01:13.589378  6496 raft_consensus.cc:493] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:01:13.638638  1645 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:13.640208  1598 raft_consensus.cc:2393] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 12: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:01:13.646762  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:13.644860  1603 raft_consensus.cc:2468] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 9.
I20260709 13:01:13.656383  6013 raft_consensus.cc:3055] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:01:13.492630  6480 raft_consensus.cc:515] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:13.651300  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:13.422290  6344 consensus_queue.cc:237] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 13.10, Last appended by leader: 10, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:13.732038  6524 consensus_queue.cc:1048] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.102s
I20260709 13:01:13.669006  6201 raft_consensus.cc:515] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:13.817768  1748 raft_consensus.cc:1240] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 13. Current term is 14. Ops: []
I20260709 13:01:13.639154  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:13.819370  1599 raft_consensus.cc:2410] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for term 12 because replica has last-logged OpId of term: 11 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 9.
W20260709 13:01:13.597978  6501 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.716s	user 0.022s	sys 0.030s
I20260709 13:01:13.674772  6357 raft_consensus.cc:2804] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader pre-election won for term 13
W20260709 13:01:13.674091  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 66213.2us
I20260709 13:01:13.821120  6357 raft_consensus.cc:493] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:13.821557  1571 leader_election.cc:304] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:01:13.753234  6544 raft_consensus.cc:515] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:13.824961  6544 leader_election.cc:290] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:13.825430  6201 leader_election.cc:290] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 14 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 13:01:13.677102  6595 log.cc:927] Time spent T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.310s	user 0.000s	sys 0.000s
I20260709 13:01:13.678839  1644 leader_election.cc:304] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
W20260709 13:01:13.698406  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 140831us
W20260709 13:01:13.828956  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 150203us
I20260709 13:01:13.830703  1643 leader_election.cc:304] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:01:13.831514  6507 raft_consensus.cc:2749] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 13:01:13.561556  6046 raft_consensus.cc:3060] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:01:13.613372  1677 raft_consensus.cc:2468] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 9.
I20260709 13:01:13.714617  6289 consensus_queue.cc:1059] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 15.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 13:01:13.835152  6592 raft_consensus.cc:2804] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:01:13.835795  6592 raft_consensus.cc:493] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:13.836053  6289 raft_consensus.cc:3055] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 13:01:13.836273  6592 raft_consensus.cc:3060] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:01:13.836598  6289 raft_consensus.cc:740] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:01:13.715104  6365 raft_consensus.cc:2749] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:01:13.716287  6548 raft_consensus.cc:2804] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:01:13.838660  6548 raft_consensus.cc:493] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:13.839088  6548 raft_consensus.cc:3060] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:01:13.718580  6546 leader_election.cc:290] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:01:13.714285  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 99441us
I20260709 13:01:13.722410  6556 raft_consensus.cc:2749] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
W20260709 13:01:13.791096  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 163231us
I20260709 13:01:13.791494  6082 raft_consensus.cc:3060] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:01:13.791079  6516 raft_consensus.cc:493] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:13.807087  6013 raft_consensus.cc:740] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:01:13.803215  6067 raft_consensus.cc:515] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:13.811302  6496 raft_consensus.cc:515] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } }
I20260709 13:01:13.790455  6487 leader_election.cc:290] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:13.807425  6535 raft_consensus.cc:515] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:13.812961  6533 consensus_queue.cc:1048] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 13:01:13.795069  6515 raft_consensus.cc:493] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:13.821659  6357 raft_consensus.cc:3060] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:01:13.717962  1674 raft_consensus.cc:2468] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 14.
I20260709 13:01:13.837673  6289 consensus_queue.cc:260] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.12, Last appended by leader: 12, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:13.913800  6289 raft_consensus.cc:3060] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:01:13.846180  6201 consensus_queue.cc:1048] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 13:01:13.918977  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:13.878401  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:13.920326  1746 raft_consensus.cc:1275] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 13 index: 9. (index mismatch)
I20260709 13:01:13.881072  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 13 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:13.793696  6350 raft_consensus.cc:2749] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:01:13.884545  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:13.921411  1603 raft_consensus.cc:2410] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for term 13 because replica has last-logged OpId of term: 10 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 4.
I20260709 13:01:13.921680  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 13 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:13.887130  2299 leader_election.cc:304] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:01:13.923979  6496 leader_election.cc:290] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:13.877445  1569 leader_election.cc:304] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:01:13.893509  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:13.926111  6067 leader_election.cc:290] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:13.888993  6546 raft_consensus.cc:493] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:13.926862  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:13.926988  6546 raft_consensus.cc:515] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:13.928056  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:13.928642  1753 raft_consensus.cc:2468] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 9.
I20260709 13:01:13.896560  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:13.898833  1747 raft_consensus.cc:1275] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 14 index: 12. (index mismatch)
I20260709 13:01:13.899883  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:13.901222  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:13.902864  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:13.906585  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:13.906905  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:01:13.853866  6453 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.598s	user 0.106s	sys 0.162s
I20260709 13:01:13.908028  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:13.912254  6515 raft_consensus.cc:3060] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:01:13.932127  1752 raft_consensus.cc:2468] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 10.
W20260709 13:01:13.903983  6526 consensus_meta.cc:287] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 1.103s	user 0.016s	sys 0.000s
I20260709 13:01:13.932497  1602 raft_consensus.cc:2468] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 8.
I20260709 13:01:13.931821  1754 raft_consensus.cc:2393] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 12: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:01:13.944870  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:13.951031  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:01:13.938083  6453 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.699s	user 0.109s	sys 0.162s
I20260709 13:01:13.883435  6524 raft_consensus.cc:2804] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:01:13.933597  1680 raft_consensus.cc:3060] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:01:13.934670  1605 raft_consensus.cc:2468] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 14.
I20260709 13:01:13.904665  6516 raft_consensus.cc:3060] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Advancing to term 13
W20260709 13:01:13.935340  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 59911.8us
I20260709 13:01:13.930503  1604 raft_consensus.cc:1275] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 14 index: 12. (index mismatch)
I20260709 13:01:13.896481  6568 raft_consensus.cc:493] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:13.935776  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:13.896817  5989 raft_consensus.cc:493] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:13.975157  2301 leader_election.cc:304] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:13.940119  1718 leader_election.cc:304] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:13.952358  2330 raft_consensus.cc:1275] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260709 13:01:13.945349  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:13.950989  2335 raft_consensus.cc:2393] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 10: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:01:13.933213  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:13.945459  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:13.956209  1754 raft_consensus.cc:1275] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260709 13:01:13.957202  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:13.958277  1571 leader_election.cc:304] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:13.940541  6350 raft_consensus.cc:2804] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:01:13.929608  6546 leader_election.cc:290] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:13.929031  6289 raft_consensus.cc:2749] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
W20260709 13:01:13.973914  6587 log.cc:927] Time spent T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.070s	user 0.001s	sys 0.000s
I20260709 13:01:13.993858  1746 raft_consensus.cc:2468] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 9.
I20260709 13:01:13.993858  2329 raft_consensus.cc:2468] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 10.
W20260709 13:01:13.985036  2337 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.262s	user 0.004s	sys 0.020s
I20260709 13:01:13.995950  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:01:14.000825  2337 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.278s	user 0.005s	sys 0.021s
I20260709 13:01:14.001220  1754 raft_consensus.cc:2393] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 11: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:01:14.005991  1604 raft_consensus.cc:1240] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 8. Current term is 9. Ops: []
I20260709 13:01:14.007340  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:13.992410  1569 leader_election.cc:304] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:01:14.015034  1680 raft_consensus.cc:2468] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 14.
I20260709 13:01:14.048398  1746 raft_consensus.cc:1275] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 10 index: 7. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
I20260709 13:01:14.048722  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:01:14.051702  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 59408.3us
I20260709 13:01:14.052896  1569 leader_election.cc:304] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:01:13.935664  6556 consensus_queue.cc:1048] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.086s
I20260709 13:01:14.071264  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:01:14.073431  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 74495.6us
W20260709 13:01:13.970604  6481 consensus_meta.cc:287] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.863s	user 0.013s	sys 0.000s
I20260709 13:01:13.951159  6507 raft_consensus.cc:493] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:13.955230  6067 raft_consensus.cc:493] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:01:13.978871  6536 consensus_meta.cc:287] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.948s	user 0.010s	sys 0.003s
I20260709 13:01:14.068794  1569 leader_election.cc:304] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:01:13.966781  6524 raft_consensus.cc:493] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:01:13.996325  6496 raft_consensus.cc:2749] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:01:13.999058  6526 raft_consensus.cc:2804] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:01:13.995647  6350 raft_consensus.cc:493] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:14.077646  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:13.966725  6201 raft_consensus.cc:493] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:14.072273  1605 raft_consensus.cc:2468] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 13.
I20260709 13:01:14.074837  1718 leader_election.cc:304] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:13.998586  6487 raft_consensus.cc:493] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:14.084270  1671 raft_consensus.cc:2468] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 11.
I20260709 13:01:14.075953  1599 raft_consensus.cc:1240] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 8. Current term is 9. Ops: []
I20260709 13:01:14.046895  6532 raft_consensus.cc:2804] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:01:14.064152  6538 raft_consensus.cc:493] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:13.994663  6568 raft_consensus.cc:515] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:14.102089  1717 leader_election.cc:304] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:14.079373  6519 raft_consensus.cc:2749] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 13:01:13.995940  5989 raft_consensus.cc:515] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:14.104539  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:14.073606  6531 raft_consensus.cc:2804] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:01:14.090633  6524 raft_consensus.cc:3060] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:01:14.096873  6350 raft_consensus.cc:3060] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:01:14.097270  6332 raft_consensus.cc:493] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:14.120901  1599 raft_consensus.cc:1240] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 12. Current term is 13. Ops: []
W20260709 13:01:14.121560  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 46839.7us
I20260709 13:01:14.096935  6526 raft_consensus.cc:493] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:14.087637  6505 consensus_queue.cc:1048] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.031s
I20260709 13:01:14.091542  6527 consensus_queue.cc:1048] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.033s
W20260709 13:01:14.140113  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 165027us
I20260709 13:01:14.141263  2301 leader_election.cc:304] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:14.087904  6067 raft_consensus.cc:515] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:14.113639  6531 raft_consensus.cc:493] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:01:14.136201  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 414277us
W20260709 13:01:14.125463  6579 log.cc:927] Time spent T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.506s	user 0.000s	sys 0.001s
I20260709 13:01:14.062923  6013 consensus_queue.cc:260] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 9.10, Last appended by leader: 10, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:14.096203  6507 raft_consensus.cc:515] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:14.092770  6536 raft_consensus.cc:515] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:14.098642  6201 raft_consensus.cc:515] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:14.132833  6526 raft_consensus.cc:3060] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:01:14.134653  6542 raft_consensus.cc:2804] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:01:14.130759  6519 raft_consensus.cc:493] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
W20260709 13:01:14.198755  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 130044us
I20260709 13:01:14.098904  6578 consensus_queue.cc:1048] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.048s
I20260709 13:01:14.138352  6523 leader_election.cc:290] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:14.147465  6277 raft_consensus.cc:2804] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Leader election won for term 9
I20260709 13:01:14.104496  6487 raft_consensus.cc:515] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:14.119104  6538 raft_consensus.cc:515] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:01:14.215679  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 112181us
I20260709 13:01:14.192843  6531 raft_consensus.cc:3060] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:01:14.221308  1717 leader_election.cc:304] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:14.229705  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:14.229979  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:14.230645  1673 raft_consensus.cc:3060] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:01:14.230543  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:14.231693  1680 raft_consensus.cc:2468] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 11.
I20260709 13:01:14.230671  1569 leader_election.cc:304] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:01:14.195148  6289 raft_consensus.cc:2804] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Leader pre-election won for term 13
W20260709 13:01:14.227613  6580 log.cc:927] Time spent T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.664s	user 0.001s	sys 0.001s
I20260709 13:01:14.136710  6332 raft_consensus.cc:515] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:14.270146  1746 raft_consensus.cc:1275] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 10 index: 8. Preceding OpId from leader: term: 12 index: 9. (index mismatch)
I20260709 13:01:14.156756  6360 consensus_queue.cc:1059] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 12.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.081s
I20260709 13:01:14.219321  6542 raft_consensus.cc:697] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 14 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:01:14.240834  6615 raft_consensus.cc:493] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:14.287922  1717 leader_election.cc:304] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:14.241317  6505 raft_consensus.cc:493] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
W20260709 13:01:14.319785  2301 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:14.256229  6578 raft_consensus.cc:2749] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:01:14.262368  6289 raft_consensus.cc:493] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:14.261667  6013 raft_consensus.cc:3060] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:01:14.314975  6624 raft_consensus.cc:493] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:14.316468  6365 raft_consensus.cc:493] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:01:14.345602  6624 raft_consensus.cc:515] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:14.345824  6365 raft_consensus.cc:515] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:14.348709  6365 leader_election.cc:290] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:14.316999  5989 leader_election.cc:290] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:14.257905  6606 raft_consensus.cc:2804] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:01:14.349891  6606 raft_consensus.cc:493] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:01:14.350466  6606 raft_consensus.cc:3060] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:01:14.253660  6277 raft_consensus.cc:697] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:01:14.354988  6277 consensus_queue.cc:237] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:14.214392  6513 raft_consensus.cc:515] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:14.358824  6513 leader_election.cc:290] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 12 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:01:14.342657  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 54870us
I20260709 13:01:14.363205  6360 raft_consensus.cc:3055] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:01:14.363644  6360 raft_consensus.cc:740] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:01:14.364625  6360 consensus_queue.cc:260] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.10, Last appended by leader: 10, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:14.366044  6360 raft_consensus.cc:3060] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Advancing to term 13
W20260709 13:01:14.366976  6485 consensus_meta.cc:287] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 1.508s	user 0.017s	sys 0.000s
I20260709 13:01:14.367479  6485 raft_consensus.cc:515] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:14.369668  6485 leader_election.cc:290] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 12 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:14.370579  1673 raft_consensus.cc:2468] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 13.
W20260709 13:01:14.371676  6593 log.cc:927] Time spent T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.276s	user 0.000s	sys 0.001s
I20260709 13:01:14.370647  6601 raft_consensus.cc:2749] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
W20260709 13:01:14.373895  6597 log.cc:927] Time spent T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.180s	user 0.000s	sys 0.001s
I20260709 13:01:14.373982  2337 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39726 (request call id 1138) took 1729 ms. Trace:
I20260709 13:01:14.375533  6507 leader_election.cc:290] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:14.374537  2337 rpcz_store.cc:276] 0709 13:01:12.645068 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:12.645214 (+   146us) service_pool.cc:224] Handling call
0709 13:01:12.645599 (+   385us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:12.680542 (+ 34943us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:12.680583 (+    41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:01:12.683184 (+  2601us) log.cc:844] Serialized 26 byte log entry
0709 13:01:14.001334 (+1318150us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:01:14.001379 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:14.002175 (+   796us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:14.373708 (+371533us) raft_consensus.cc:1685] finished
0709 13:01:14.373733 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:14.373888 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":10235,"raft.queue_time_us":10980,"raft.run_cpu_time_us":134,"raft.run_wall_time_us":6640,"thread_start_us":1278562,"threads_started":1,"wal-append.queue_time_us":1511075}
I20260709 13:01:14.376147  6507 raft_consensus.cc:2804] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:01:14.376751  6507 raft_consensus.cc:697] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:01:14.377653  6507 consensus_queue.cc:237] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:14.378351  6350 raft_consensus.cc:515] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:14.379009  6568 leader_election.cc:290] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:14.379458  6201 leader_election.cc:290] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:14.334637  6615 raft_consensus.cc:515] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:14.381036  6350 leader_election.cc:290] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 10 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:14.382095  6615 leader_election.cc:290] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:01:14.383386  6598 log.cc:927] Time spent T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.250s	user 0.000s	sys 0.001s
I20260709 13:01:14.384080  1753 raft_consensus.cc:2393] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 12: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:01:14.263892  6523 consensus_queue.cc:1059] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.081s
I20260709 13:01:14.387528  6536 leader_election.cc:290] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 13 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:14.342111  6505 raft_consensus.cc:515] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:14.388559  6555 consensus_meta.cc:287] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 1.018s	user 0.011s	sys 0.000s
I20260709 13:01:14.371613  6487 leader_election.cc:290] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:14.411393  1597 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 2391) took 1887 ms. Trace:
I20260709 13:01:14.411840  1597 rpcz_store.cc:276] 0709 13:01:12.524168 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:12.524317 (+   149us) service_pool.cc:224] Handling call
0709 13:01:12.524751 (+   434us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:12.525149 (+   398us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:12.525178 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:01:12.554098 (+ 28920us) log.cc:844] Serialized 26 byte log entry
0709 13:01:12.902537 (+348439us) spinlock_profiling.cc:243] Waited 6.57 ms on lock 0x7b4800003cd8. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426b08baa1 00007f426b082b37 00007f426b0829ca 00007f426b17a26f 00007f426b179f37 00007f426b18e55c 00007f426b18e508 00007f426b18e488 00007f426b18e443 00007f426b18d6ec 00007f426a0b7a96
0709 13:01:13.059559 (+157022us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:01:13.059604 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:13.059725 (+   121us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:14.411051 (+1351326us) raft_consensus.cc:1685] finished
0709 13:01:14.411077 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:14.411314 (+   237us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":32250,"raft.queue_time_us":600,"raft.run_cpu_time_us":2776,"raft.run_wall_time_us":373228,"spinlock_wait_cycles":15111552,"thread_start_us":432077,"threads_started":1,"wal-append.queue_time_us":644088}
W20260709 13:01:14.413322  2299 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:14.389708  6289 raft_consensus.cc:3060] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:01:14.416424  1671 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57092 (request call id 1063) took 1214 ms. Trace:
I20260709 13:01:14.416823  1671 rpcz_store.cc:276] 0709 13:01:13.202527 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:13.218025 (+ 15498us) service_pool.cc:224] Handling call
0709 13:01:14.416222 (+1198197us) spinlock_profiling.cc:243] Waited 792 ms on lock 0x7b540025a154. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:01:14.416353 (+   131us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1821064064}
I20260709 13:01:14.336298  6527 raft_consensus.cc:493] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:14.391091  2332 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39726 (request call id 1139) took 1348 ms. Trace:
I20260709 13:01:14.361845  1717 leader_election.cc:304] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:14.228746  6519 raft_consensus.cc:515] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:14.406678  6523 raft_consensus.cc:3055] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:01:14.426086  6523 raft_consensus.cc:740] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:01:14.427453  6480 leader_election.cc:290] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:14.427179  6523 consensus_queue.cc:260] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.10, Last appended by leader: 10, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:14.428498  6523 raft_consensus.cc:3060] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:01:14.389017  6542 consensus_queue.cc:237] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:14.430717  5989 raft_consensus.cc:493] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:14.431284  5989 raft_consensus.cc:515] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:14.433707  5989 leader_election.cc:290] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:14.436354  6505 leader_election.cc:290] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:01:14.439678  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 77927.6us
I20260709 13:01:14.439810  6538 leader_election.cc:290] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:14.420207  2332 rpcz_store.cc:276] 0709 13:01:13.043018 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:13.057097 (+ 14079us) service_pool.cc:224] Handling call
0709 13:01:13.057501 (+   404us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:13.067343 (+  9842us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:13.067382 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:01:13.081099 (+ 13717us) log.cc:844] Serialized 26 byte log entry
0709 13:01:13.532730 (+451631us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:01:13.532804 (+    74us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:13.532918 (+   114us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:14.390822 (+857904us) raft_consensus.cc:1685] finished
0709 13:01:14.390851 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:14.391015 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3543,"raft.queue_time_us":7899,"raft.run_cpu_time_us":146,"raft.run_wall_time_us":12416,"thread_start_us":433367,"threads_started":1,"wal-append.queue_time_us":993245}
I20260709 13:01:14.440701  6578 raft_consensus.cc:2804] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 13:01:14.425833  6519 leader_election.cc:290] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:14.441155  1717 leader_election.cc:304] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:14.441537  6578 raft_consensus.cc:697] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 9 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:01:14.358970  6624 leader_election.cc:290] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:01:14.400283  6592 consensus_meta.cc:287] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.563s	user 0.009s	sys 0.006s
I20260709 13:01:14.402052  6357 raft_consensus.cc:515] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:14.424149  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:14.425822  6527 raft_consensus.cc:515] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:14.443270  1717 leader_election.cc:304] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:14.465416  1671 raft_consensus.cc:1275] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 14 index: 12. (index mismatch)
I20260709 13:01:14.465790  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:01:14.444741  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 214128us
I20260709 13:01:14.466641  2335 raft_consensus.cc:2468] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 10.
I20260709 13:01:14.447268  1718 leader_election.cc:304] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
W20260709 13:01:14.410478  6581 log.cc:927] Time spent T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 1.042s	user 0.001s	sys 0.003s
I20260709 13:01:14.457042  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:01:14.469815  2336 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.628s	user 0.005s	sys 0.003s
W20260709 13:01:14.472100  2336 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.630s	user 0.005s	sys 0.003s
W20260709 13:01:14.471112  1749 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.539s	user 0.007s	sys 0.000s
I20260709 13:01:14.471458  1673 raft_consensus.cc:1240] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 12. Current term is 13. Ops: []
W20260709 13:01:14.473111  1749 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.541s	user 0.008s	sys 0.000s
I20260709 13:01:14.473060  1675 raft_consensus.cc:2368] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 for earlier term 12. Current term is 13.
I20260709 13:01:14.479071  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:14.479851  2332 raft_consensus.cc:3060] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:01:14.485991  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:14.492537  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
W20260709 13:01:14.497458  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 73706.6us
I20260709 13:01:14.470803  6619 raft_consensus.cc:2749] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 13:01:14.495462  1569 leader_election.cc:304] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:01:14.493539  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:14.512845  1673 raft_consensus.cc:3060] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:01:14.474278  6487 raft_consensus.cc:2804] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:01:14.518855  6487 raft_consensus.cc:493] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:14.519310  6487 raft_consensus.cc:3060] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:01:14.508201  1750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55840 (request call id 2618) took 1216 ms. Trace:
I20260709 13:01:14.522310  1750 rpcz_store.cc:276] 0709 13:01:13.291620 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:13.291780 (+   160us) service_pool.cc:224] Handling call
0709 13:01:13.292194 (+   414us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:13.671143 (+378949us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:13.671182 (+    39us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:01:13.671231 (+    49us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:01:13.671281 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:14.507871 (+836590us) spinlock_profiling.cc:243] Waited 336 ms on lock 0x7b5400071554. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:01:14.507959 (+    88us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:14.508120 (+   161us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":42280,"raft.queue_time_us":42821,"raft.run_cpu_time_us":98,"raft.run_wall_time_us":99,"spinlock_wait_cycles":773097216}
I20260709 13:01:14.537829  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:14.530381  1673 raft_consensus.cc:2468] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 10.
I20260709 13:01:14.544384  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:14.479377  6485 raft_consensus.cc:2749] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:01:14.549048  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:14.560776  6568 raft_consensus.cc:493] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:14.561538  2330 raft_consensus.cc:2393] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 12: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:01:14.561606  6568 raft_consensus.cc:515] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:14.564929  6542 raft_consensus.cc:493] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:14.565552  6542 raft_consensus.cc:515] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:14.566902  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:14.567790  1602 raft_consensus.cc:2468] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 10.
I20260709 13:01:14.489599  6201 raft_consensus.cc:493] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:14.420464  6555 raft_consensus.cc:515] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:14.456557  6526 raft_consensus.cc:515] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } }
I20260709 13:01:14.485589  6480 raft_consensus.cc:493] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:14.550078  2337 raft_consensus.cc:2468] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 11.
I20260709 13:01:14.600056  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:14.503957  6544 raft_consensus.cc:493] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:14.582598  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:14.583086  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:14.582603  2299 leader_election.cc:400] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 12 election: Vote denied by peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261) with higher term. Message: Invalid argument: T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 for earlier term 12. Current term is 13.
I20260709 13:01:14.609880  2299 leader_election.cc:403] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 12 election: Cancelling election due to peer responding with higher term
I20260709 13:01:14.585711  6624 consensus_queue.cc:1048] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 9, Time since last communication: 0.001s
W20260709 13:01:14.594211  2300 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:14.500036  6160 raft_consensus.cc:493] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:01:14.565625  6568 leader_election.cc:290] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:14.598579  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:14.503033  6519 raft_consensus.cc:493] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
W20260709 13:01:14.610615  6046 consensus_meta.cc:287] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.778s	user 0.010s	sys 0.004s
W20260709 13:01:14.581270  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 85786.9us
I20260709 13:01:14.606356  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:14.610177  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:14.466197  6592 raft_consensus.cc:515] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:14.508857  6538 raft_consensus.cc:493] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:14.624962  1598 raft_consensus.cc:1240] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 10. Current term is 11. Ops: []
I20260709 13:01:14.625089  1604 raft_consensus.cc:2393] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 10: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:01:14.634296  1569 leader_election.cc:304] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:01:14.636498  1569 leader_election.cc:304] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:01:14.641301  1569 leader_election.cc:304] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:14.642696  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:14.643707  1673 raft_consensus.cc:2468] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 9.
I20260709 13:01:14.595395  6201 raft_consensus.cc:515] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:01:14.640734  6548 consensus_meta.cc:287] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.801s	user 0.012s	sys 0.003s
I20260709 13:01:14.703258  1677 raft_consensus.cc:2393] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 11: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:01:14.640492  6046 raft_consensus.cc:2749] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:01:14.616930  6629 raft_consensus.cc:2804] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:01:14.707914  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:14.715170  1675 raft_consensus.cc:2468] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 8.
W20260709 13:01:14.647083  6515 consensus_meta.cc:287] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.700s	user 0.008s	sys 0.005s
I20260709 13:01:14.635672  6568 raft_consensus.cc:493] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:14.649178  6505 raft_consensus.cc:2804] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader election won for term 10
I20260709 13:01:14.614895  6544 raft_consensus.cc:515] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:14.503134  6578 consensus_queue.cc:237] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:14.612340  6480 raft_consensus.cc:515] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:14.730911  1679 raft_consensus.cc:1240] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 8. Current term is 9. Ops: []
I20260709 13:01:14.705227  6615 raft_consensus.cc:493] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:14.622826  6160 raft_consensus.cc:515] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:01:14.720504  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 147235us
I20260709 13:01:14.712316  1673 raft_consensus.cc:1240] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 13. Current term is 14. Ops: [13.8-13.8]
I20260709 13:01:14.646559  6538 raft_consensus.cc:515] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:14.749073  1671 raft_consensus.cc:2468] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 11.
I20260709 13:01:14.746523  1718 leader_election.cc:304] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:14.712951  6614 raft_consensus.cc:2804] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:01:14.637850  6519 raft_consensus.cc:515] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:14.702595  6365 raft_consensus.cc:493] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:14.701407  6626 raft_consensus.cc:2804] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Leader election won for term 13
I20260709 13:01:14.714042  6629 raft_consensus.cc:493] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:14.752810  6201 leader_election.cc:290] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 13:01:14.768946  2301 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:14.723698  6568 raft_consensus.cc:515] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:14.834133  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:01:14.751768  6633 log.cc:927] Time spent T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.055s	user 0.001s	sys 0.000s
I20260709 13:01:14.714257  6548 raft_consensus.cc:515] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:14.787983  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:14.842185  1602 raft_consensus.cc:2468] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 8.
I20260709 13:01:14.820521  6643 raft_consensus.cc:493] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:14.790913  6527 leader_election.cc:290] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:01:14.794147  6639 log.cc:927] Time spent T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.083s	user 0.000s	sys 0.000s
I20260709 13:01:14.799119  6542 leader_election.cc:290] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:14.787983  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:01:14.803284  6634 log.cc:927] Time spent T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.086s	user 0.000s	sys 0.000s
I20260709 13:01:14.863432  1717 leader_election.cc:304] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:14.804311  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:14.857800  1643 leader_election.cc:304] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:01:14.805528  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:14.865125  1643 leader_election.cc:304] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 13:01:14.829571  6629 raft_consensus.cc:3060] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:01:14.811064  6289 raft_consensus.cc:515] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:14.812203  6632 log.cc:927] Time spent T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.069s	user 0.000s	sys 0.000s
I20260709 13:01:14.817880  6600 raft_consensus.cc:3055] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:01:14.719400  6515 raft_consensus.cc:515] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:14.733872  6612 raft_consensus.cc:493] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:14.734387  6505 raft_consensus.cc:697] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
W20260709 13:01:14.742517  6082 consensus_meta.cc:287] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.840s	user 0.009s	sys 0.005s
I20260709 13:01:14.751976  6637 raft_consensus.cc:2804] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:01:14.866055  6526 leader_election.cc:290] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 10 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:14.822105  6614 raft_consensus.cc:493] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:14.840227  2332 raft_consensus.cc:2468] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 13.
I20260709 13:01:14.869532  6542 raft_consensus.cc:2749] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:01:14.790503  6532 raft_consensus.cc:493] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:14.749043  6531 raft_consensus.cc:515] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:14.848594  1748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55840 (request call id 2616) took 1725 ms. Trace:
I20260709 13:01:14.749481  6357 leader_election.cc:290] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:14.923556  1748 rpcz_store.cc:276] 0709 13:01:13.123211 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:13.126168 (+  2957us) service_pool.cc:224] Handling call
0709 13:01:13.126584 (+   416us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:13.818305 (+691721us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:14.848330 (+1030025us) spinlock_profiling.cc:243] Waited 691 ms on lock 0x7b54001c7d54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:01:14.848522 (+   192us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1588785408}
I20260709 13:01:14.757830  6615 raft_consensus.cc:515] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:14.852701  6516 consensus_meta.cc:287] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.858s	user 0.012s	sys 0.002s
I20260709 13:01:14.803977  6496 raft_consensus.cc:2749] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:01:14.863282  1672 raft_consensus.cc:2468] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 8.
W20260709 13:01:14.810981  6582 log.cc:927] Time spent T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 1.496s	user 0.004s	sys 0.007s
I20260709 13:01:14.836205  6626 raft_consensus.cc:697] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:01:14.915241  6637 raft_consensus.cc:493] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:14.867331  6501 consensus_queue.cc:1059] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.002s
I20260709 13:01:14.869132  6536 consensus_queue.cc:1048] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.002s
I20260709 13:01:14.868709  1717 leader_election.cc:304] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 13:01:14.876672  6612 raft_consensus.cc:515] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:14.829708  6365 raft_consensus.cc:515] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:14.878216  6160 leader_election.cc:290] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:14.884262  6515 leader_election.cc:290] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:14.884946  1596 raft_consensus.cc:2368] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for earlier term 12. Current term is 13.
I20260709 13:01:14.900738  6067 leader_election.cc:290] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:14.900511  6527 raft_consensus.cc:493] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:14.999548  6527 raft_consensus.cc:515] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:15.001569  6365 leader_election.cc:290] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:15.001830  6527 leader_election.cc:290] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:01:14.902642  1674 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52766 (request call id 2681) took 1467 ms (client timeout 1755 ms). Trace:
I20260709 13:01:14.905469  6535 leader_election.cc:290] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:01:14.909457  1569 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.107.194:46261 timed out after 1.755s (SENT) [suppressed 29 similar messages]
W20260709 13:01:14.838358  2299 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:14.872408  6606 consensus_meta.cc:287] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.521s	user 0.007s	sys 0.006s
W20260709 13:01:14.846621  6524 consensus_meta.cc:287] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.712s	user 0.011s	sys 0.002s
I20260709 13:01:14.842425  1675 raft_consensus.cc:1240] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 8. Current term is 9. Ops: []
I20260709 13:01:14.871542  6546 raft_consensus.cc:2804] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:01:14.871321  6600 raft_consensus.cc:740] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:01:14.874131  6555 leader_election.cc:290] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:14.839826  6201 raft_consensus.cc:493] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
W20260709 13:01:14.920943  6639 log.cc:927] Time spent T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.051s	user 0.000s	sys 0.000s
I20260709 13:01:14.966367  6615 leader_election.cc:290] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:14.966768  1597 raft_consensus.cc:1240] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 12. Current term is 13. Ops: []
I20260709 13:01:14.923015  6513 raft_consensus.cc:2749] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Leader election lost for term 12. Reason: Vote denied by peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261) with higher term. Message: Invalid argument: T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 for earlier term 12. Current term is 13.
I20260709 13:01:14.967309  6626 consensus_queue.cc:237] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:14.929786  6528 raft_consensus.cc:2804] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:01:14.929311  6516 raft_consensus.cc:515] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:14.969908  6584 log.cc:927] Time spent T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 1.649s	user 0.002s	sys 0.003s
I20260709 13:01:14.976765  6602 raft_consensus.cc:493] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:14.977267  6485 raft_consensus.cc:493] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:14.977698  6619 raft_consensus.cc:493] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:14.985282  6289 leader_election.cc:290] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 13 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:14.856011  6643 raft_consensus.cc:515] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:14.925124  6532 raft_consensus.cc:3060] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:01:14.966071  6531 leader_election.cc:290] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:01:15.002630  1674 rpcz_store.cc:269] 0709 13:01:13.435488 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:13.481769 (+ 46281us) service_pool.cc:224] Handling call
0709 13:01:14.902349 (+1420580us) spinlock_profiling.cc:243] Waited 234 ms on lock 0x7b5400164254. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:01:14.902531 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":537501568}
I20260709 13:01:15.002942  6480 leader_election.cc:290] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:14.986411  6637 raft_consensus.cc:3060] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:01:15.082139  2331 raft_consensus.cc:1240] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 12. Current term is 13. Ops: [12.11-12.11]
I20260709 13:01:14.925362  6592 leader_election.cc:290] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 11 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:01:15.064004  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 195354us
I20260709 13:01:15.024225  6546 raft_consensus.cc:493] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:14.922701  6614 raft_consensus.cc:3060] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:01:15.062367  6544 leader_election.cc:290] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 13:01:15.071138  6645 log.cc:927] Time spent T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.240s	user 0.001s	sys 0.000s
W20260709 13:01:15.085515  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 213475us
I20260709 13:01:15.064692  6332 leader_election.cc:290] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:15.026615  6524 raft_consensus.cc:515] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:15.027019  6606 raft_consensus.cc:515] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:15.097412  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 187991us
W20260709 13:01:15.104022  2300 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:15.045567  6201 raft_consensus.cc:515] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:15.110639  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:15.125185  1604 raft_consensus.cc:2468] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 10.
I20260709 13:01:15.090557  6519 leader_election.cc:290] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:15.111510  6590 raft_consensus.cc:493] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:15.112039  6332 consensus_queue.cc:1059] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 13:01:15.113019  2329 raft_consensus.cc:1240] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 9. Current term is 10. Ops: []
I20260709 13:01:15.112888  6612 leader_election.cc:290] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:15.113657  6531 raft_consensus.cc:493] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:15.114338  6606 leader_election.cc:290] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 10 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:15.109530  6289 raft_consensus.cc:493] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:15.074491  6485 raft_consensus.cc:515] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:14.918149  6082 raft_consensus.cc:515] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:15.119673  6646 log.cc:927] Time spent T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.186s	user 0.000s	sys 0.001s
I20260709 13:01:15.121227  6652 raft_consensus.cc:2804] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:01:15.124123  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:15.088034  6546 raft_consensus.cc:3060] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:01:15.028784  6505 consensus_queue.cc:237] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:15.072261  6602 raft_consensus.cc:515] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:15.132341  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:15.134008  6501 consensus_queue.cc:1059] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.003s
I20260709 13:01:15.224300  6501 raft_consensus.cc:3055] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:01:15.224718  6501 raft_consensus.cc:740] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
W20260709 13:01:15.226082  6647 log.cc:927] Time spent T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.130s	user 0.000s	sys 0.000s
I20260709 13:01:15.225754  6501 consensus_queue.cc:260] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.7, Last appended by leader: 7, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:15.227103  6501 raft_consensus.cc:3060] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:01:15.227290  1643 leader_election.cc:304] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
W20260709 13:01:15.131531  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 45146us
I20260709 13:01:15.137889  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:15.110087  6544 raft_consensus.cc:493] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:15.143636  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:15.134413  6590 raft_consensus.cc:515] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:15.123343  6568 leader_election.cc:290] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:15.120723  6524 leader_election.cc:290] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 12 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:15.121667  2336 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39726 (request call id 1145) took 1282 ms. Trace:
I20260709 13:01:15.148675  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:15.153206  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:15.319280  1599 raft_consensus.cc:2468] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 10.
I20260709 13:01:15.139168  6548 leader_election.cc:290] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 14 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:15.187953  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:01:15.140849  6487 consensus_meta.cc:287] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.621s	user 0.015s	sys 0.000s
I20260709 13:01:15.146715  6289 raft_consensus.cc:515] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:15.194697  6651 raft_consensus.cc:493] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:15.195602  6629 raft_consensus.cc:515] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:15.196421  6082 leader_election.cc:290] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:15.074554  6619 raft_consensus.cc:515] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:15.200047  1751 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43156 (request call id 945) took 2702 ms. Trace:
I20260709 13:01:15.201064  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:15.230159  6332 raft_consensus.cc:2804] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:01:15.230281  1605 raft_consensus.cc:1240] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 9. Current term is 10. Ops: []
I20260709 13:01:15.220846  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:15.235867  1717 leader_election.cc:304] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 13:01:15.404596  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 166690us
I20260709 13:01:15.149423  5989 raft_consensus.cc:3055] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:01:15.222903  6652 raft_consensus.cc:493] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:15.347687  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:15.093814  6600 consensus_queue.cc:260] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:15.318177  2336 rpcz_store.cc:276] 0709 13:01:13.839173 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:13.840438 (+  1265us) service_pool.cc:224] Handling call
0709 13:01:13.840919 (+   481us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:13.841375 (+   456us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 13:01:14.472590 (+631215us) log.cc:844] Serialized 14 byte log entry
0709 13:01:14.559607 (+ 87017us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:01:14.594369 (+ 34762us) log.cc:844] Serialized 26 byte log entry
0709 13:01:14.594598 (+   229us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:01:14.594633 (+    35us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:14.594733 (+   100us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:15.121384 (+526651us) raft_consensus.cc:1685] finished
0709 13:01:15.121418 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:15.121588 (+   170us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":673046,"prepare.run_cpu_time_us":65,"prepare.run_wall_time_us":66,"raft.queue_time_us":815,"raft.run_cpu_time_us":116,"raft.run_wall_time_us":115,"spinlock_wait_cycles":33916160,"thread_start_us":654265,"threads_started":2,"wal-append.queue_time_us":120901}
I20260709 13:01:15.144716  6531 raft_consensus.cc:515] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:15.395388  1751 rpcz_store.cc:276] 0709 13:01:12.497948 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:12.498979 (+  1031us) service_pool.cc:224] Handling call
0709 13:01:12.499378 (+   399us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:01:12.573289 (+ 73911us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:12.573328 (+    39us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:01:12.573959 (+   631us) log.cc:844] Serialized 52 byte log entry
0709 13:01:13.060706 (+486747us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:01:13.333739 (+273033us) log.cc:844] Serialized 14 byte log entry
0709 13:01:13.481471 (+147732us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:13.481603 (+   132us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:14.418061 (+936458us) spinlock_profiling.cc:243] Waited 12.8 ms on lock 0x7b18004452a0. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f42630753f3 00007f42630777c1 00007f4263077779 00007f4263077709 00007f42630776d1 00007f42630769cd 000000000038f354 000000000038f199 00007f426305883a
0709 13:01:14.969001 (+550940us) spinlock_profiling.cc:243] Waited 13.9 ms on lock 0x7b4400022b4c. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f4262f9bc85 00007f4262f9c6a2 00007f4265d0e1db 00007f4265d09aaf 00007f4265cfd86e 00007f4265cfcbb6 00007f4265cfc7d8 00007f4265d06221 00007f4265d061d9 00007f4265d06169
0709 13:01:15.199682 (+230681us) raft_consensus.cc:1685] finished
0709 13:01:15.199715 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:15.199953 (+   238us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":175575,"prepare.queue_time_us":432171,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":50,"raft.queue_time_us":165263,"raft.run_cpu_time_us":365,"raft.run_wall_time_us":88957,"spinlock_wait_cycles":61411072,"thread_start_us":641120,"threads_started":2,"wal-append.queue_time_us":551038}
I20260709 13:01:15.424402  1604 raft_consensus.cc:2393] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 12: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:01:15.349092  1602 raft_consensus.cc:2468] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 9.
I20260709 13:01:15.412303  2333 raft_consensus.cc:2368] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 13. Current term is 14.
I20260709 13:01:15.414685  1601 raft_consensus.cc:2368] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for earlier term 12. Current term is 13.
W20260709 13:01:15.363401  6523 consensus_meta.cc:287] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.934s	user 0.007s	sys 0.010s
W20260709 13:01:15.411461  6584 log.cc:927] Time spent T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.064s	user 0.001s	sys 0.000s
I20260709 13:01:15.318611  6544 raft_consensus.cc:515] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:15.363788  6485 leader_election.cc:290] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:15.400861  6332 raft_consensus.cc:493] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:15.413879  5989 raft_consensus.cc:740] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:01:15.451695  1598 raft_consensus.cc:2468] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 11.
I20260709 13:01:15.411688  6615 raft_consensus.cc:493] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:15.369382  6487 raft_consensus.cc:515] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:15.421657  6652 raft_consensus.cc:3060] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:01:15.396062  6651 raft_consensus.cc:515] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:15.451103  6600 raft_consensus.cc:3060] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:01:15.420742  6532 raft_consensus.cc:515] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:15.459087  6332 raft_consensus.cc:3060] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:01:15.459713  6643 leader_election.cc:290] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 13:01:15.491261  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 58625us
I20260709 13:01:15.453568  6578 consensus_queue.cc:1059] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.033s
I20260709 13:01:15.467691  6615 raft_consensus.cc:515] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:15.502836  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 94123.2us
I20260709 13:01:15.504115  1717 leader_election.cc:304] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 13:01:15.505643  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 167816us
I20260709 13:01:15.520332  6615 leader_election.cc:290] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:15.500675  6201 leader_election.cc:290] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:15.525066  1717 leader_election.cc:304] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:15.527045  6201 raft_consensus.cc:2749] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 13:01:15.541712  1717 leader_election.cc:304] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:15.495329  5989 consensus_queue.cc:260] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:15.538298  6590 leader_election.cc:290] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:15.536338  6659 raft_consensus.cc:2804] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Leader pre-election won for term 11
W20260709 13:01:15.567705  6360 consensus_meta.cc:287] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 1.175s	user 0.011s	sys 0.004s
I20260709 13:01:15.573947  1678 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52766 (request call id 2679) took 2139 ms. Trace:
I20260709 13:01:15.575729  6602 leader_election.cc:290] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:15.574942  1678 rpcz_store.cc:276] 0709 13:01:13.434832 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:13.542298 (+107466us) service_pool.cc:224] Handling call
0709 13:01:13.542740 (+   442us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:13.816151 (+273411us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 13:01:14.092330 (+276179us) log.cc:844] Serialized 14 byte log entry
0709 13:01:14.274616 (+182286us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:01:14.274704 (+    88us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:01:14.274793 (+    89us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:15.573580 (+1298787us) spinlock_profiling.cc:243] Waited 273 ms on lock 0x7b54000fda54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:01:15.573654 (+    74us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:15.573856 (+   202us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":458788,"prepare.run_cpu_time_us":69,"prepare.run_wall_time_us":71,"raft.queue_time_us":6798,"raft.run_cpu_time_us":105,"raft.run_wall_time_us":102,"spinlock_wait_cycles":652049024,"thread_start_us":440492,"threads_started":2,"wal-append.queue_time_us":314968,"wal-append.run_cpu_time_us":1274,"wal-append.run_wall_time_us":1006947}
W20260709 13:01:15.583374  1596 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40166 (request call id 2403) took 1705 ms (client timeout 1584 ms). Trace:
W20260709 13:01:15.584059  1596 rpcz_store.cc:269] 0709 13:01:13.877912 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:13.878056 (+   144us) service_pool.cc:224] Handling call
0709 13:01:15.583030 (+1704974us) spinlock_profiling.cc:243] Waited 964 ms on lock 0x7b540029a4d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:01:15.583242 (+   212us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2217885568}
I20260709 13:01:15.558753  5989 raft_consensus.cc:3060] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:01:15.555768  6659 raft_consensus.cc:493] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:15.585812  2328 raft_consensus.cc:2468] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 13.
I20260709 13:01:15.561151  6619 leader_election.cc:290] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:15.563445  6289 leader_election.cc:290] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:15.562283  6201 raft_consensus.cc:2804] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260709 13:01:15.602859  1571 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:15.588351  6546 raft_consensus.cc:515] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:15.634735  6651 leader_election.cc:290] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:15.583663  6487 leader_election.cc:290] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:15.662420  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:15.660012  1643 leader_election.cc:304] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 13:01:15.665329  1643 leader_election.cc:304] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 13:01:15.671177  1643 leader_election.cc:400] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 12 election: Vote denied by peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445) with higher term. Message: Invalid argument: T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for earlier term 12. Current term is 13.
I20260709 13:01:15.600839  6659 raft_consensus.cc:3060] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:01:15.645311  6201 raft_consensus.cc:493] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:15.644743  6532 leader_election.cc:290] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 13 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:15.659045  6615 raft_consensus.cc:493] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:01:15.679775  1671 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52778 (request call id 2229) took 1230 ms. Trace:
I20260709 13:01:15.691334  1671 rpcz_store.cc:276] 0709 13:01:14.449488 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:14.466762 (+ 17274us) service_pool.cc:224] Handling call
0709 13:01:15.679530 (+1212768us) spinlock_profiling.cc:243] Waited 262 ms on lock 0x7b54000596d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:01:15.679691 (+   161us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":603018496}
I20260709 13:01:15.679411  1643 leader_election.cc:403] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 12 election: Cancelling election due to peer responding with higher term
I20260709 13:01:15.672401  6529 raft_consensus.cc:2804] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:01:15.677631  6524 raft_consensus.cc:2804] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:01:15.679989  6531 leader_election.cc:290] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:15.698366  6536 raft_consensus.cc:3060] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:01:15.700846  6524 raft_consensus.cc:493] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:15.701931  6524 raft_consensus.cc:3060] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:01:15.698446  6650 raft_consensus.cc:3060] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Advancing to term 10
W20260709 13:01:15.712320  2300 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:15.696197  6529 raft_consensus.cc:493] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:15.713464  6529 raft_consensus.cc:3060] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:01:15.718672  6531 raft_consensus.cc:493] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:15.719223  6531 raft_consensus.cc:515] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:15.721622  6531 leader_election.cc:290] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:15.722136  6531 raft_consensus.cc:493] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:15.723621  6523 raft_consensus.cc:493] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:15.724124  6519 raft_consensus.cc:493] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:15.726032  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:15.728116  1644 leader_election.cc:304] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
W20260709 13:01:15.729245  1645 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:15.736799  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:15.730553  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:15.737632  1596 raft_consensus.cc:3060] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:01:15.688202  6201 raft_consensus.cc:3060] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:01:15.672202  6649 consensus_queue.cc:1059] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.025s
I20260709 13:01:15.738996  6519 raft_consensus.cc:515] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:15.743395  6648 raft_consensus.cc:2804] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Leader election won for term 13
I20260709 13:01:15.739118  6523 raft_consensus.cc:515] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:15.744025  6614 consensus_meta.cc:287] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.647s	user 0.015s	sys 0.000s
I20260709 13:01:15.745050  6481 raft_consensus.cc:493] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:15.697320  6615 raft_consensus.cc:515] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:15.743395  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:15.766611  1754 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55840 (request call id 2626) took 1815 ms. Trace:
I20260709 13:01:15.763898  2336 raft_consensus.cc:1240] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 12. Current term is 13. Ops: []
I20260709 13:01:15.773957  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:15.775960  2332 raft_consensus.cc:3060] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:01:15.773710  1754 rpcz_store.cc:276] 0709 13:01:13.951605 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:14.019241 (+ 67636us) service_pool.cc:224] Handling call
0709 13:01:14.019526 (+   285us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:14.110919 (+ 91393us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:14.110972 (+    53us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:01:15.586663 (+1475691us) spinlock_profiling.cc:243] Waited 11.7 ms on lock 0x7b2c000bc788. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4264a5ba11 00007f4264a6b372 00007f426619da01 00007f426616243a 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2
0709 13:01:15.587365 (+   702us) log.cc:844] Serialized 26 byte log entry
0709 13:01:15.697280 (+109915us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:01:15.697338 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:15.698661 (+  1323us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:15.766243 (+ 67582us) raft_consensus.cc:1685] finished
0709 13:01:15.766268 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:15.766531 (+   263us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":92629,"raft.queue_time_us":77386,"raft.run_cpu_time_us":153,"raft.run_wall_time_us":154,"spinlock_wait_cycles":26973568,"thread_start_us":109382,"threads_started":1,"wal-append.queue_time_us":174473}
I20260709 13:01:15.785548  1597 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40154 (request call id 2339) took 1293 ms. Trace:
I20260709 13:01:15.723660  6650 raft_consensus.cc:2749] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:01:15.778826  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 12 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:15.797940  2336 raft_consensus.cc:2468] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 11.
I20260709 13:01:15.773569  1601 raft_consensus.cc:2368] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for earlier term 12. Current term is 13.
I20260709 13:01:15.792765  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:15.789083  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:15.791286  1597 rpcz_store.cc:276] 0709 13:01:14.491807 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:14.491970 (+   163us) service_pool.cc:224] Handling call
0709 13:01:14.492449 (+   479us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:15.061038 (+568589us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:15.785229 (+724191us) spinlock_profiling.cc:243] Waited 474 ms on lock 0x7b54002797d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:01:15.785463 (+   234us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1090002176}
I20260709 13:01:15.799371  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:01:15.793202  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 251535us
I20260709 13:01:15.805397  2336 raft_consensus.cc:1240] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 12. Current term is 13. Ops: []
I20260709 13:01:15.806367  1717 leader_election.cc:304] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:15.780172  1596 raft_consensus.cc:2468] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 14.
I20260709 13:01:15.803640  2333 raft_consensus.cc:2468] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 10.
W20260709 13:01:15.759184  6637 consensus_meta.cc:287] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.666s	user 0.014s	sys 0.000s
I20260709 13:01:15.758883  6046 raft_consensus.cc:3055] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:01:15.761163  6648 raft_consensus.cc:697] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 13 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:01:15.811853  1602 raft_consensus.cc:2468] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 13.
I20260709 13:01:15.761631  6600 raft_consensus.cc:493] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:15.817224  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:15.773582  6536 raft_consensus.cc:2749] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Leader election lost for term 12. Reason: Vote denied by peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445) with higher term. Message: Invalid argument: T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for earlier term 12. Current term is 13.
I20260709 13:01:15.821805  2336 raft_consensus.cc:2468] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 15.
I20260709 13:01:15.822333  2333 raft_consensus.cc:1275] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
I20260709 13:01:15.822384  6487 raft_consensus.cc:3060] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:01:15.827503  2332 raft_consensus.cc:2468] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 11.
I20260709 13:01:15.831072  6546 leader_election.cc:290] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 15 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:15.833101  6523 leader_election.cc:290] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:15.754601  6524 raft_consensus.cc:515] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:15.738588  6531 raft_consensus.cc:515] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:15.794286  6481 raft_consensus.cc:515] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:15.839875  2331 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49082 (request call id 957) took 2211 ms. Trace:
I20260709 13:01:15.840268  2331 rpcz_store.cc:276] 0709 13:01:13.628298 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:13.628434 (+   136us) service_pool.cc:224] Handling call
0709 13:01:13.655530 (+ 27096us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:15.082770 (+1427240us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:15.839546 (+756776us) spinlock_profiling.cc:243] Waited 1.42 s on lock 0x7b54004f88d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:01:15.839779 (+   233us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3268045696}
I20260709 13:01:15.822579  6519 leader_election.cc:290] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:15.853668  6648 consensus_queue.cc:237] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:15.769377  6652 raft_consensus.cc:515] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:15.819941  6345 raft_consensus.cc:493] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:15.820477  6046 raft_consensus.cc:740] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
W20260709 13:01:15.863431  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 135419us
I20260709 13:01:15.778316  6614 raft_consensus.cc:515] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:15.843430  6578 raft_consensus.cc:493] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:15.865275  2331 raft_consensus.cc:1275] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 14 index: 12. (index mismatch)
I20260709 13:01:15.868870  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:15.869169  1678 raft_consensus.cc:1240] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 14. Current term is 15. Ops: [14.12-14.12]
I20260709 13:01:15.870985  1644 leader_election.cc:304] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:01:15.875419  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:15.876250  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:15.859452  6601 raft_consensus.cc:493] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:15.863234  6602 raft_consensus.cc:493] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:15.866245  6345 raft_consensus.cc:515] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:15.823313  6637 raft_consensus.cc:515] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:15.884007  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:15.852321  6600 raft_consensus.cc:515] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:15.863804  6666 raft_consensus.cc:493] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:15.887786  1644 leader_election.cc:304] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:15.871784  6501 raft_consensus.cc:493] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:15.893289  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:15.892797   431 auto_leader_rebalancer.cc:527] All tables' leader rebalancing finished this round
I20260709 13:01:15.898855  1500 catalog_manager.cc:5697] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 7 to 9, leader changed from 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194) to 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193). New cstate: current_term: 9 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } }
I20260709 13:01:15.907020  6614 leader_election.cc:290] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:15.877494  6659 raft_consensus.cc:515] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:15.909610  6659 leader_election.cc:290] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:15.883915  6298 consensus_queue.cc:1059] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 12.11, Next index: 12, Last known committed idx: 10, Time since last communication: 0.027s
I20260709 13:01:15.912143  6481 leader_election.cc:290] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:15.913023  6481 consensus_queue.cc:1059] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 12.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 13:01:15.913888  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:15.914515  1673 raft_consensus.cc:3060] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Advancing to term 13
I20260709 13:01:15.906872  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:15.891647  6531 leader_election.cc:290] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:15.919530  6652 leader_election.cc:290] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 14 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:15.920140  6652 raft_consensus.cc:493] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:15.920580  6652 raft_consensus.cc:515] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:15.930377  6201 raft_consensus.cc:515] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:15.932777  6201 leader_election.cc:290] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 9 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:15.936146  6629 leader_election.cc:290] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 12 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:15.936996  6649 raft_consensus.cc:3055] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:01:15.937419  6649 raft_consensus.cc:740] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:01:15.938390  6649 consensus_queue.cc:260] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:15.939834  6637 leader_election.cc:290] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 12 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:15.940776  6345 leader_election.cc:290] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:15.940900  1679 raft_consensus.cc:2468] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 13.
I20260709 13:01:15.941083  6529 raft_consensus.cc:515] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:15.943732  6529 leader_election.cc:290] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 11 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:15.886111  6619 raft_consensus.cc:493] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:01:15.942317  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:15.956383  1679 raft_consensus.cc:2393] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 13: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:01:15.948460  1747 raft_consensus.cc:2393] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 9: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:01:15.949244  1680 raft_consensus.cc:1240] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 10. Current term is 11. Ops: []
I20260709 13:01:15.943809  6615 leader_election.cc:290] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:15.910204  6659 raft_consensus.cc:493] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:15.949781  1678 raft_consensus.cc:2410] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for term 11 because replica has last-logged OpId of term: 10 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 5.
I20260709 13:01:15.955607  1673 raft_consensus.cc:2468] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 13.
I20260709 13:01:15.876940  6578 raft_consensus.cc:515] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:15.962518  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:15.892551  6671 raft_consensus.cc:2804] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Leader pre-election won for term 12
W20260709 13:01:15.964787  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 73584us
I20260709 13:01:15.966640  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:15.971561  1597 raft_consensus.cc:3060] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Advancing to term 13
W20260709 13:01:15.946784  6582 log.cc:927] Time spent T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.060s	user 0.000s	sys 0.000s
I20260709 13:01:15.972026  1644 leader_election.cc:304] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:01:15.979182  1644 leader_election.cc:304] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:01:15.888644  6602 raft_consensus.cc:515] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:15.885898  6601 raft_consensus.cc:515] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:15.946749  6672 raft_consensus.cc:2749] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:01:15.971743  6659 raft_consensus.cc:515] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:15.894243  6666 raft_consensus.cc:515] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:15.980065  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:15.985766  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:15.972997  6671 raft_consensus.cc:493] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:16.000725  6602 leader_election.cc:290] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:15.999222  6659 leader_election.cc:290] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:15.970649  6524 leader_election.cc:290] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:15.901623  6501 raft_consensus.cc:515] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:15.989634  6578 leader_election.cc:290] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:15.995914  6600 leader_election.cc:290] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:15.983085  6652 leader_election.cc:290] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:15.987090  6529 raft_consensus.cc:2804] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:01:16.082167  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:16.082975  1752 raft_consensus.cc:2468] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 14.
I20260709 13:01:16.081408  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:16.001210  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:15.961647  6619 raft_consensus.cc:515] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:15.994156  6487 raft_consensus.cc:2749] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:01:16.016040  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:15.984474  6298 raft_consensus.cc:2804] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Leader election won for term 11
I20260709 13:01:15.985250  6676 raft_consensus.cc:493] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:16.089090  1754 raft_consensus.cc:1240] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 9. Current term is 10. Ops: []
I20260709 13:01:16.080969  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:16.081756  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:16.084949  1749 raft_consensus.cc:2468] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 9.
I20260709 13:01:16.085273  1673 raft_consensus.cc:2393] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 15: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:01:15.897100  6046 consensus_queue.cc:260] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } }
I20260709 13:01:15.980580  6536 consensus_queue.cc:1048] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.102s
I20260709 13:01:16.087077  1751 raft_consensus.cc:1240] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 10. Current term is 11. Ops: []
I20260709 13:01:15.978144  6332 raft_consensus.cc:515] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:16.088060  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 12 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:16.090420  1750 raft_consensus.cc:2468] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 10.
I20260709 13:01:16.090971  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:16.003392  6201 raft_consensus.cc:493] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:16.104256  6201 raft_consensus.cc:515] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:16.105409  1748 raft_consensus.cc:2468] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 14.
I20260709 13:01:16.093863  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:16.107523  1746 raft_consensus.cc:2393] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 11: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:01:16.108443  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:16.108989  1746 raft_consensus.cc:3060] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:01:16.014331  6671 raft_consensus.cc:3060] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:01:16.096776  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:16.112030  1752 raft_consensus.cc:3060] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:01:16.098600  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:16.099627  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:16.100071  1745 raft_consensus.cc:2393] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 12: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:01:16.099896  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:16.096553  6676 raft_consensus.cc:515] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:16.103519  6046 raft_consensus.cc:3060] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:01:16.115033  1674 raft_consensus.cc:2368] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 10. Current term is 11.
I20260709 13:01:16.115203  1750 raft_consensus.cc:2393] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 13: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:01:16.115804  1749 raft_consensus.cc:2468] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 8.
I20260709 13:01:16.087671  6529 raft_consensus.cc:493] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:16.089295  6629 raft_consensus.cc:493] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:16.134872  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:16.135596  1745 raft_consensus.cc:3060] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Advancing to term 11
W20260709 13:01:16.131055  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 152455us
I20260709 13:01:16.136205  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:16.137377  1751 raft_consensus.cc:2393] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 11: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:01:16.093195  6069 raft_consensus.cc:493] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:16.138398  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:16.143136  1749 raft_consensus.cc:2468] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 8.
I20260709 13:01:16.143988  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:16.144217  1752 raft_consensus.cc:2468] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 13.
I20260709 13:01:16.140707  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:16.145637  1750 raft_consensus.cc:2468] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 10.
I20260709 13:01:16.097374  6298 raft_consensus.cc:697] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 11 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:01:16.144922  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:16.113034  6619 leader_election.cc:290] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:16.146556  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:16.153750  1750 raft_consensus.cc:3060] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:01:16.139014  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:16.157017  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:16.157749  1752 raft_consensus.cc:2468] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 12.
I20260709 13:01:16.138973  6529 raft_consensus.cc:3060] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:01:16.151403  1746 raft_consensus.cc:2468] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 13.
I20260709 13:01:16.170197  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:16.168558  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:16.171236  1679 raft_consensus.cc:3060] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:01:16.149102  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:16.175508  1749 raft_consensus.cc:2468] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 9.
I20260709 13:01:16.176662  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 12 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:16.177732  1749 raft_consensus.cc:2410] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for term 12 because replica has last-logged OpId of term: 10 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 7.
I20260709 13:01:16.179541  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:16.184926  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:16.186264  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:01:16.192876  1717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:16.197525  1750 raft_consensus.cc:2468] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 13.
I20260709 13:01:16.199661  1749 raft_consensus.cc:1275] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 9 index: 7. (term mismatch)
I20260709 13:01:16.200105  1749 pending_rounds.cc:85] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701: Aborting all ops after (but not including) 6
I20260709 13:01:16.200320  1749 pending_rounds.cc:107] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701: Aborting uncommitted NO_OP operation due to leader change: 8.7
I20260709 13:01:16.200554  1749 raft_consensus.cc:2889] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:01:16.201812  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:16.202831  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:16.204146  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:16.204775  1752 raft_consensus.cc:2468] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 11.
I20260709 13:01:16.169054  6507 raft_consensus.cc:493] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:16.207329  1679 raft_consensus.cc:2410] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for term 13 because replica has last-logged OpId of term: 12 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 5.
I20260709 13:01:16.169773  6201 leader_election.cc:290] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:16.145160  6629 raft_consensus.cc:515] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:16.212157  1750 raft_consensus.cc:2368] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 for earlier term 12. Current term is 13.
I20260709 13:01:16.147179  6069 raft_consensus.cc:515] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:01:16.217530  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 74922.1us
I20260709 13:01:16.218865  1746 raft_consensus.cc:1240] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 10. Current term is 11. Ops: []
I20260709 13:01:16.219113  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:16.219713  1750 raft_consensus.cc:3060] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:01:16.210099  6298 consensus_queue.cc:237] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:16.226176  6666 leader_election.cc:290] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:16.230163  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:16.230645  6332 leader_election.cc:290] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 11 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:16.230875  1746 raft_consensus.cc:2468] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 12.
I20260709 13:01:16.220268  6501 leader_election.cc:290] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:16.210994  6507 raft_consensus.cc:515] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:16.222078  6601 leader_election.cc:290] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:16.270493  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:16.271143  1746 raft_consensus.cc:3060] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:01:16.248536  6648 raft_consensus.cc:493] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:16.248690  6672 raft_consensus.cc:493] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:16.248849  1646 leader_election.cc:304] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:01:16.249815  6679 consensus_queue.cc:1048] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 13:01:16.232474  6536 raft_consensus.cc:493] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:16.255831  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:16.298915  1752 raft_consensus.cc:2468] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 14.
I20260709 13:01:16.239674  6629 leader_election.cc:290] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:01:16.241047  1753 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55840 (request call id 2620) took 2329 ms (client timeout 1906 ms). Trace:
I20260709 13:01:16.245039  6160 raft_consensus.cc:3055] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:01:16.263569  1597 raft_consensus.cc:2468] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 13.
I20260709 13:01:16.244550  6676 leader_election.cc:290] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:16.265262  1679 raft_consensus.cc:1275] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20260709 13:01:16.269403  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:16.303467  1749 raft_consensus.cc:2468] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 15.
I20260709 13:01:16.274209  1745 raft_consensus.cc:2468] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 11.
W20260709 13:01:16.276991  2299 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:16.279920  1750 raft_consensus.cc:2468] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 13.
I20260709 13:01:16.283572  6507 leader_election.cc:290] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:16.266011  6069 leader_election.cc:290] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:16.285120  6554 raft_consensus.cc:2749] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:01:16.285293  1646 leader_election.cc:304] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:01:16.288434  6568 consensus_queue.cc:1048] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 13:01:16.290521  6648 raft_consensus.cc:515] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:16.291461  6672 raft_consensus.cc:515] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:16.294092  1599 raft_consensus.cc:1275] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 14 index: 12. (index mismatch)
I20260709 13:01:16.300976  1746 raft_consensus.cc:2468] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 10.
W20260709 13:01:16.301488  1753 rpcz_store.cc:269] 0709 13:01:13.911242 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:13.929259 (+ 18017us) service_pool.cc:224] Handling call
0709 13:01:16.240668 (+2311409us) spinlock_profiling.cc:243] Waited 389 ms on lock 0x7b54001d51d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:01:16.240893 (+   225us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":895761280}
I20260709 13:01:16.311929  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:16.327170  1752 raft_consensus.cc:2393] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 11: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:01:16.302783  6160 raft_consensus.cc:740] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
W20260709 13:01:16.314608  2299 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:16.330888  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:16.331984  2336 raft_consensus.cc:2393] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 11: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:01:16.331923  1646 leader_election.cc:304] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:16.324646  6344 raft_consensus.cc:2749] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:01:16.334272  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:16.336398  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:16.343108  2332 raft_consensus.cc:2468] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 15.
I20260709 13:01:16.347287  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:16.350070  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:16.352519  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:16.353255  2333 raft_consensus.cc:3060] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:01:16.347822  1646 leader_election.cc:304] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:16.354331  6592 raft_consensus.cc:2804] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:01:16.360173  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:16.360916  1753 raft_consensus.cc:2368] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for earlier term 12. Current term is 13.
I20260709 13:01:16.362097  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:16.362695  1753 raft_consensus.cc:2468] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 15.
I20260709 13:01:16.363557  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:16.364156  1752 raft_consensus.cc:2468] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 11.
I20260709 13:01:16.365840  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:16.354770  6592 raft_consensus.cc:493] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:16.366375  1753 raft_consensus.cc:3060] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Advancing to term 11
I20260709 13:01:16.366767  6592 raft_consensus.cc:3060] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:01:16.367784  6345 consensus_queue.cc:1059] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 10.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 13:01:16.369249  6672 leader_election.cc:290] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:16.350677  6554 raft_consensus.cc:2804] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:01:16.370023  6554 raft_consensus.cc:493] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:01:16.370422  6554 raft_consensus.cc:3060] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:01:16.375607  6648 leader_election.cc:290] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:16.376454  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:16.377107  1746 raft_consensus.cc:2468] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 9.
I20260709 13:01:16.379011  1753 raft_consensus.cc:2468] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 11.
I20260709 13:01:16.379522  6160 consensus_queue.cc:260] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:16.380796  6160 raft_consensus.cc:3060] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:01:16.382162  1672 raft_consensus.cc:2410] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for term 10 because replica has last-logged OpId of term: 9 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 5.
I20260709 13:01:16.339043  6685 raft_consensus.cc:493] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:16.382848  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:16.383064  6685 raft_consensus.cc:515] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:16.385655  6685 leader_election.cc:290] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:16.386677  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:16.387377  1599 raft_consensus.cc:2468] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 18.
I20260709 13:01:16.335731  6360 raft_consensus.cc:493] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:16.388485  6360 raft_consensus.cc:515] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:16.367846  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:16.392263  1752 raft_consensus.cc:2468] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 13.
I20260709 13:01:16.395421  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:16.396355  1599 raft_consensus.cc:2410] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 for term 11 because replica has last-logged OpId of term: 9 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 9.
I20260709 13:01:16.326081  6536 raft_consensus.cc:515] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:16.399729  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:16.402746  1646 leader_election.cc:400] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 12 election: Vote denied by peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693) with higher term. Message: Invalid argument: T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for earlier term 12. Current term is 13.
I20260709 13:01:16.402348  2336 raft_consensus.cc:2468] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 15.
I20260709 13:01:16.405638  1646 leader_election.cc:304] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:16.338642  6671 raft_consensus.cc:515] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:16.413992  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:16.414870  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:16.422644  6676 raft_consensus.cc:493] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:16.425729  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:16.416505  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 15 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:16.428529  1677 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52778 (request call id 2222) took 2523 ms. Trace:
I20260709 13:01:16.430272  2332 raft_consensus.cc:2410] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for term 15 because replica has last-logged OpId of term: 13 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 9.
I20260709 13:01:16.429953  1677 rpcz_store.cc:276] 0709 13:01:13.904963 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:13.905918 (+   955us) service_pool.cc:224] Handling call
0709 13:01:16.428280 (+2522362us) spinlock_profiling.cc:243] Waited 783 ms on lock 0x7b54000230d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:01:16.428435 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1801407360}
I20260709 13:01:16.431298  2336 raft_consensus.cc:2393] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 13: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:01:16.432756  1718 leader_election.cc:304] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:16.430882  1646 leader_election.cc:304] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:16.434163  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:16.434912  6501 raft_consensus.cc:2749] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:01:16.435352  6345 raft_consensus.cc:2804] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:01:16.442399  6345 raft_consensus.cc:493] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:16.442734  6648 raft_consensus.cc:2804] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:01:16.442909  6345 raft_consensus.cc:3060] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:01:16.443178  6648 raft_consensus.cc:493] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:16.443585  6648 raft_consensus.cc:3060] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:01:16.445006  6685 consensus_queue.cc:1059] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 13.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 13:01:16.436249  6601 consensus_queue.cc:1059] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 13:01:16.447563  6592 raft_consensus.cc:515] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:01:16.448935  6675 log.cc:927] Time spent T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.458s	user 0.004s	sys 0.000s
I20260709 13:01:16.436971  6676 raft_consensus.cc:515] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:16.438079  1677 raft_consensus.cc:2468] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 12.
I20260709 13:01:16.438767  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:16.439646  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:16.446182  6681 raft_consensus.cc:3055] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:01:16.451261  1678 raft_consensus.cc:2468] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 9.
I20260709 13:01:16.453080  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:16.453765  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:16.440773  6360 leader_election.cc:290] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:16.456226  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:16.453830  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:16.457226  1677 raft_consensus.cc:2368] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 11. Current term is 12.
I20260709 13:01:16.454754  2332 raft_consensus.cc:2468] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 10.
I20260709 13:01:16.438970  1646 leader_election.cc:304] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:16.459012  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:16.451674  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:16.462399  1674 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52778 (request call id 2234) took 1319 ms. Trace:
I20260709 13:01:16.451375  1752 raft_consensus.cc:2468] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 9.
I20260709 13:01:16.462952  2336 raft_consensus.cc:2393] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 13: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:01:16.462846  1674 rpcz_store.cc:276] 0709 13:01:15.143171 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:15.143338 (+   167us) service_pool.cc:224] Handling call
0709 13:01:16.462166 (+1318828us) spinlock_profiling.cc:243] Waited 876 ms on lock 0x7b54000226d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:01:16.462327 (+   161us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2013880192}
I20260709 13:01:16.452389  6681 raft_consensus.cc:740] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:01:16.464151  1646 leader_election.cc:304] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:16.459798  1677 raft_consensus.cc:2393] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 14: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:01:16.449896  6592 leader_election.cc:290] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 12 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:16.465775  1646 leader_election.cc:304] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:16.467087  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:16.464377  6679 raft_consensus.cc:2804] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Leader election won for term 11
I20260709 13:01:16.468266  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:16.468910  6679 raft_consensus.cc:697] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:01:16.468938  1674 raft_consensus.cc:3060] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:01:16.469977  6360 raft_consensus.cc:493] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:16.470094  6679 consensus_queue.cc:237] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:16.470695  6360 raft_consensus.cc:515] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:16.474305  6681 consensus_queue.cc:260] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.12, Last appended by leader: 12, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:16.473829  6360 leader_election.cc:290] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:16.474361  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:16.477768  1677 raft_consensus.cc:2393] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 12: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:01:16.479223  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 15 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:01:16.479488  1645 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:16.482638  6546 raft_consensus.cc:493] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:16.481868  6542 consensus_queue.cc:1059] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 13:01:16.484616  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:16.484930  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:16.485551  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:16.493809  1750 raft_consensus.cc:3060] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Advancing to term 12
I20260709 13:01:16.475730  6298 raft_consensus.cc:2804] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:01:16.496704  6344 raft_consensus.cc:493] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:01:16.497619  6344 raft_consensus.cc:515] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:16.499821  6344 leader_election.cc:290] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:16.485708  1677 raft_consensus.cc:2410] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for term 15 because replica has last-logged OpId of term: 14 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 7.
I20260709 13:01:16.478044  1674 raft_consensus.cc:2468] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 9.
I20260709 13:01:16.488268  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:16.485679  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:16.487910  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:16.491277  6507 raft_consensus.cc:493] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:01:16.491756  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:16.503091  1746 raft_consensus.cc:2393] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 11: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
W20260709 13:01:16.499912  2301 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:16.491408  6160 raft_consensus.cc:493] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:16.501963  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:16.506860  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:16.507973  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:16.508560  1752 raft_consensus.cc:2468] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 15.
I20260709 13:01:16.502646  1745 raft_consensus.cc:2468] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 10.
I20260709 13:01:16.501777  1749 raft_consensus.cc:2468] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 9.
I20260709 13:01:16.511507  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:16.511699  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:16.512364  1749 raft_consensus.cc:2468] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 11.
I20260709 13:01:16.513252  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:16.514004  1749 raft_consensus.cc:2410] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 for term 11 because replica has last-logged OpId of term: 9 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 9.
I20260709 13:01:16.505651  1646 leader_election.cc:304] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:16.516019  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:16.518635  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:16.518934  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:16.519276  1675 raft_consensus.cc:3060] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:01:16.519577  1748 raft_consensus.cc:2468] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 11.
I20260709 13:01:16.483242  6546 raft_consensus.cc:515] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:16.529960  1571 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:16.475616  6681 raft_consensus.cc:3060] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:01:16.527076  1646 leader_election.cc:304] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:16.465071  6676 leader_election.cc:290] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:16.532409  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 12 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:16.529538  6592 raft_consensus.cc:2804] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:01:16.533558  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:16.535311  1748 raft_consensus.cc:2393] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 12: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:01:16.533617  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:16.536479  1599 raft_consensus.cc:2468] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 10.
I20260709 13:01:16.500645  6298 raft_consensus.cc:493] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:16.515724  6344 raft_consensus.cc:2804] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:01:16.539471  1748 raft_consensus.cc:1240] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 12. Current term is 13. Ops: []
I20260709 13:01:16.544430  1674 raft_consensus.cc:2468] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 15.
I20260709 13:01:16.538003  6592 raft_consensus.cc:493] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:16.548545  1748 raft_consensus.cc:1275] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 11 index: 9. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
I20260709 13:01:16.544472  6298 raft_consensus.cc:3060] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:01:16.557611  1750 raft_consensus.cc:2468] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 12.
I20260709 13:01:16.541428  6332 raft_consensus.cc:2804] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:01:16.520056  6046 consensus_queue.cc:1048] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.029s
I20260709 13:01:16.522176  6160 raft_consensus.cc:515] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:16.521731  6507 raft_consensus.cc:515] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:16.547936  6592 raft_consensus.cc:3060] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Advancing to term 10
W20260709 13:01:16.578619  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 51635.6us
I20260709 13:01:16.578938  1746 raft_consensus.cc:2468] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 17.
I20260709 13:01:16.546624  6344 raft_consensus.cc:493] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
W20260709 13:01:16.580397  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 48551.3us
I20260709 13:01:16.565788  6332 raft_consensus.cc:493] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
W20260709 13:01:16.594370  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:16.583925  6344 raft_consensus.cc:3060] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:01:16.612561  2333 raft_consensus.cc:2468] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 12.
I20260709 13:01:16.597522  6691 consensus_queue.cc:1048] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.022s
I20260709 13:01:16.616395  6554 raft_consensus.cc:515] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:16.619582  6160 leader_election.cc:290] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:16.624307  1676 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52766 (request call id 2680) took 3189 ms. Trace:
I20260709 13:01:16.624738  1676 rpcz_store.cc:276] 0709 13:01:13.435188 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:13.543901 (+108713us) service_pool.cc:224] Handling call
0709 13:01:13.591097 (+ 47196us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:15.529149 (+1938052us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 13:01:15.529198 (+    49us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:01:15.530779 (+  1581us) log.cc:844] Serialized 26 byte log entry
0709 13:01:15.795000 (+264221us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:01:15.795071 (+    71us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:16.623822 (+828751us) spinlock_profiling.cc:243] Waited 1.94 s on lock 0x7b5400164754. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:01:16.623885 (+    63us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:16.623937 (+    52us) raft_consensus.cc:1685] finished
0709 13:01:16.623951 (+    14us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:16.624208 (+   257us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1288,"raft.queue_time_us":1422,"raft.run_cpu_time_us":167,"raft.run_wall_time_us":4679,"spinlock_wait_cycles":4452352256,"thread_start_us":261210,"threads_started":1,"wal-append.queue_time_us":347836}
I20260709 13:01:16.628597  6546 leader_election.cc:290] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:16.629456  6649 raft_consensus.cc:3060] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:01:16.630946  5989 raft_consensus.cc:3055] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:01:16.631369  5989 raft_consensus.cc:740] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:01:16.632246  5989 consensus_queue.cc:260] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:16.641589  1500 catalog_manager.cc:5697] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 6 to 8. New cstate: current_term: 8 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } }
I20260709 13:01:16.643890  1500 catalog_manager.cc:5697] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 1 to 5, leader changed from 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) to 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194). New cstate: current_term: 5 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } }
I20260709 13:01:16.645265  1500 catalog_manager.cc:5697] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 9 to 11. New cstate: current_term: 11 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } }
I20260709 13:01:16.642606  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:16.647678  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:01:16.647900  1747 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55848 (request call id 2515) took 2737 ms (client timeout 2276 ms). Trace:
I20260709 13:01:16.648602  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:16.648962  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:16.649796  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:16.646251  1604 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40166 (request call id 2415) took 1516 ms. Trace:
I20260709 13:01:16.595805  6332 raft_consensus.cc:3060] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:01:16.598873  6536 leader_election.cc:290] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:01:16.648331  1747 rpcz_store.cc:269] 0709 13:01:13.910280 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:13.930941 (+ 20661us) service_pool.cc:224] Handling call
0709 13:01:15.662123 (+1731182us) spinlock_profiling.cc:243] Waited 3.63 ms on lock 0x7b58001ffe98. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f426303db71 00007f426303da22 00007f426bf56c09 00007f426b0b0ee0 00007f426b17a0ab 00007f426b17a131 00007f426b1280a0 00007f426b127984 00007f42646a26d4 00007f42646a2662
0709 13:01:16.647674 (+985551us) spinlock_profiling.cc:243] Waited 285 ms on lock 0x7b5400116cd4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:01:16.647809 (+   135us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":663156352}
I20260709 13:01:16.655967  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 15 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:16.669116  1679 raft_consensus.cc:1240] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 9. Current term is 10. Ops: []
I20260709 13:01:16.652905  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:16.650391  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:16.650755  1604 rpcz_store.cc:276] 0709 13:01:15.130257 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:15.137615 (+  7358us) service_pool.cc:224] Handling call
0709 13:01:16.646030 (+1508415us) spinlock_profiling.cc:243] Waited 189 ms on lock 0x7b54000b1154. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:01:16.646180 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":434310016}
I20260709 13:01:16.722153  6538 leader_election.cc:290] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:16.722693  2329 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49082 (request call id 958) took 2264 ms. Trace:
I20260709 13:01:16.723135  2329 rpcz_store.cc:276] 0709 13:01:14.458082 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:14.458913 (+   831us) service_pool.cc:224] Handling call
0709 13:01:14.487125 (+ 28212us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:15.128902 (+641777us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:16.722406 (+1593504us) spinlock_profiling.cc:243] Waited 625 ms on lock 0x7b5400507b54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:01:16.722619 (+   213us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1438613760}
I20260709 13:01:16.654508  6536 raft_consensus.cc:493] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:01:16.730543  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 213450us
W20260709 13:01:16.735288  2301 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:16.739439  2328 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49082 (request call id 962) took 2152 ms (client timeout 1627 ms). Trace:
I20260709 13:01:16.739846  6516 leader_election.cc:290] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 13 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:16.681839  6408 consensus_queue.cc:1059] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 10.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.059s
W20260709 13:01:16.739974  2328 rpcz_store.cc:269] 0709 13:01:14.587585 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:14.587738 (+   153us) service_pool.cc:224] Handling call
0709 13:01:16.739189 (+2151451us) spinlock_profiling.cc:243] Waited 952 ms on lock 0x7b54005843d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:01:16.739327 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2188765568}
I20260709 13:01:16.682771  6691 raft_consensus.cc:493] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:16.723757  6513 consensus_queue.cc:1059] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 9.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.025s
I20260709 13:01:16.741707  6507 leader_election.cc:290] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:16.652941  6688 consensus_queue.cc:1059] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.089s
I20260709 13:01:16.742216  6536 raft_consensus.cc:515] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:16.752527  6536 leader_election.cc:290] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:16.753142  6507 raft_consensus.cc:3055] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 LEADER]: Stepping down as leader of term 13
I20260709 13:01:16.753559  6507 raft_consensus.cc:740] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:01:16.754477  6507 consensus_queue.cc:260] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.8, Last appended by leader: 8, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:16.742426  6344 raft_consensus.cc:515] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:16.757262  6485 raft_consensus.cc:3055] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:01:16.757670  6485 raft_consensus.cc:740] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:01:16.758059  6344 leader_election.cc:290] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 15 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:16.758625  6485 consensus_queue.cc:260] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 12.8, Last appended by leader: 8, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:16.759179  6554 leader_election.cc:290] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 15 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:16.760083  6485 raft_consensus.cc:3060] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:01:16.762745  6528 raft_consensus.cc:493] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:16.763276  6528 raft_consensus.cc:3060] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Advancing to term 10
W20260709 13:01:16.763430  6674 log.cc:927] Time spent T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.765s	user 0.000s	sys 0.002s
I20260709 13:01:16.769335  6692 consensus_queue.cc:1059] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 12.11, Next index: 12, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 13:01:16.770632  6672 raft_consensus.cc:3055] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:01:16.771041  6672 raft_consensus.cc:740] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:01:16.772007  6672 consensus_queue.cc:260] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:16.773046  6672 raft_consensus.cc:3060] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:01:16.758682  6544 leader_election.cc:290] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:16.780035  6691 raft_consensus.cc:515] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:16.784905  6345 raft_consensus.cc:515] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:16.787653  1680 raft_consensus.cc:2393] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 12: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:01:16.805269  1677 raft_consensus.cc:2468] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 13.
W20260709 13:01:16.745455  6529 consensus_meta.cc:287] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.561s	user 0.010s	sys 0.005s
I20260709 13:01:16.825603  6648 raft_consensus.cc:515] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:16.828563  6345 leader_election.cc:290] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 12 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:16.841159  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:16.841557  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 12 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:16.842092  1672 raft_consensus.cc:2393] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 12: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:01:16.844846  1672 raft_consensus.cc:1275] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260709 13:01:16.849583  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:16.850446  1672 raft_consensus.cc:2468] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 9.
I20260709 13:01:16.852294  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:16.853041  1676 raft_consensus.cc:3060] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Advancing to term 11
I20260709 13:01:16.854334  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:16.855906  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:16.859206  1672 raft_consensus.cc:1275] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
I20260709 13:01:16.860723  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 15 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:16.866052  1718 leader_election.cc:304] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:16.877214  1675 raft_consensus.cc:2468] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 14.
I20260709 13:01:16.877017  1718 leader_election.cc:304] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:16.818535  6529 raft_consensus.cc:515] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:16.873703  1569 leader_election.cc:304] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 13:01:16.881579  2298 leader_election.cc:304] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
W20260709 13:01:16.888003  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 301611us
I20260709 13:01:16.887822  1718 leader_election.cc:304] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
W20260709 13:01:16.903533  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 323255us
I20260709 13:01:16.886353  6629 raft_consensus.cc:493] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:16.909053  1678 raft_consensus.cc:2393] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 16: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:01:16.912976  2298 leader_election.cc:304] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:16.907315  1571 leader_election.cc:304] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 13:01:16.894346  6676 raft_consensus.cc:2804] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:01:16.913875  1745 raft_consensus.cc:1275] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 13 index: 11. (index mismatch)
W20260709 13:01:16.891845  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 157854us
I20260709 13:01:16.916992  1643 leader_election.cc:400] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 12 pre-election: Vote denied by peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445) with higher term. Message: Invalid argument: T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for earlier term 12. Current term is 13.
I20260709 13:01:16.918401  1643 leader_election.cc:403] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 12 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:01:16.920905  1571 leader_election.cc:304] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:01:16.925098  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:01:16.922111  2299 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:16.905905  1718 leader_election.cc:304] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1
W20260709 13:01:16.927044  2299 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:16.905797  1569 leader_election.cc:304] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 13:01:16.892616  6516 raft_consensus.cc:493] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:16.885502  6650 raft_consensus.cc:2804] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:01:16.925220  2298 leader_election.cc:304] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
W20260709 13:01:17.023360  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 100158us
I20260709 13:01:17.023479  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:17.024258  1747 raft_consensus.cc:3060] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:01:17.024549  2298 leader_election.cc:304] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:01:16.908159  6408 raft_consensus.cc:2804] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:01:16.923956  6438 raft_consensus.cc:2749] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:01:16.924644  6629 raft_consensus.cc:515] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:16.951237  6538 raft_consensus.cc:2804] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:01:16.955883  6529 leader_election.cc:290] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 16 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:16.957755  1676 raft_consensus.cc:2468] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 11.
I20260709 13:01:16.942915  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:16.929996  1596 raft_consensus.cc:1275] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 10 index: 8. Preceding OpId from leader: term: 12 index: 9. (index mismatch)
I20260709 13:01:16.959746  1571 leader_election.cc:304] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:16.960484  2299 leader_election.cc:304] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:01:16.949801  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:01:16.967002  6693 log.cc:927] Time spent T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.067s	user 0.001s	sys 0.000s
W20260709 13:01:17.048219  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 87819.2us
I20260709 13:01:17.023064  6650 raft_consensus.cc:493] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:01:17.052685  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 93037.3us
I20260709 13:01:17.047205   431 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:01:16.915231  6344 consensus_queue.cc:1048] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.060s
I20260709 13:01:16.979382  6671 leader_election.cc:290] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 12 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:16.983392  6622 raft_consensus.cc:3055] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 10 LEADER]: Stepping down as leader of term 10
W20260709 13:01:16.938606  1644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:16.989805  1502 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:43476 (request call id 193) took 41563 ms (client timeout 15000 ms). Trace:
I20260709 13:01:17.061189  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:17.062108  2333 raft_consensus.cc:2410] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for term 15 because replica has last-logged OpId of term: 14 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 11.
I20260709 13:01:17.032789  6360 raft_consensus.cc:2804] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:01:17.064128  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:16.949151  6298 raft_consensus.cc:515] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:17.024853  6516 raft_consensus.cc:515] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:17.083932  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:17.084913  1751 raft_consensus.cc:3060] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:01:16.925426  6676 raft_consensus.cc:493] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:16.991521  6046 raft_consensus.cc:2749] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Leader pre-election lost for term 12. Reason: Vote denied by peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445) with higher term. Message: Invalid argument: T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for earlier term 12. Current term is 13.
I20260709 13:01:16.992599  6507 raft_consensus.cc:3060] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Advancing to term 14
W20260709 13:01:16.992478  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 75630.3us
I20260709 13:01:16.993667  1500 catalog_manager.cc:5697] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 8 to 9, leader changed from 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194) to 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196). New cstate: current_term: 9 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 13:01:16.995688  1752 raft_consensus.cc:1240] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 10. Current term is 11. Ops: []
I20260709 13:01:16.997174  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:17.089241  1753 raft_consensus.cc:2468] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 9.
I20260709 13:01:17.002424  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:17.011739  5989 raft_consensus.cc:3060] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:01:17.015019  6688 raft_consensus.cc:2804] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:01:17.016096  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:17.183230  2334 raft_consensus.cc:1275] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 13 index: 11. (index mismatch)
I20260709 13:01:17.019459  6648 leader_election.cc:290] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:17.021701  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:17.184954  1746 raft_consensus.cc:3060] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:01:17.185264  6680 consensus_queue.cc:1048] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20260709 13:01:17.187472  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 95907.5us
W20260709 13:01:17.022157  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 116311us
I20260709 13:01:17.191021  1718 leader_election.cc:304] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 13:01:17.191864  6516 leader_election.cc:290] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:17.192422  1718 leader_election.cc:304] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:17.193674  1718 leader_election.cc:400] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 pre-election: Vote denied by peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261) with higher term. Message: Invalid argument: T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 11. Current term is 12.
I20260709 13:01:17.193925  1718 leader_election.cc:403] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:01:17.041791  2328 raft_consensus.cc:2468] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 11.
I20260709 13:01:17.195047  6544 raft_consensus.cc:2804] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:01:17.195425  6544 raft_consensus.cc:493] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:17.195770  6544 raft_consensus.cc:3060] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:01:17.196506  6708 raft_consensus.cc:2804] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:01:17.196975  6708 raft_consensus.cc:493] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:17.197317  6708 raft_consensus.cc:3060] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:01:17.197652  1646 leader_election.cc:304] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:16.908535  6496 raft_consensus.cc:2804] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:01:17.026290  6512 raft_consensus.cc:2804] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 13:01:17.199194  6512 raft_consensus.cc:697] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 13 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:01:17.199561  6680 raft_consensus.cc:2804] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:01:17.200004  6680 raft_consensus.cc:493] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:17.200358  6680 raft_consensus.cc:3060] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:01:17.200184  6512 consensus_queue.cc:237] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:17.054162  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:17.026317  6638 raft_consensus.cc:2804] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:01:17.054723  1571 leader_election.cc:304] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 13:01:17.203437  6638 raft_consensus.cc:493] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:17.203930  6638 raft_consensus.cc:3060] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:01:17.206754  6706 raft_consensus.cc:493] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:17.207370  6706 raft_consensus.cc:515] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:17.059659  1502 rpcz_store.cc:269] 0709 13:00:35.475788 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:35.515683 (+ 39895us) service_pool.cc:224] Handling call
0709 13:01:16.989537 (+41473854us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 13:01:07.751538 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 13:01:07.858745 (+107207us) write_op.cc:432] Acquiring schema lock in shared mode
0709 13:01:07.863102 (+  4357us) write_op.cc:435] Acquired schema lock
0709 13:01:07.864859 (+  1757us) tablet.cc:662] Decoding operations
0709 13:01:08.119273 (+254414us) write_op.cc:620] Acquiring the partition lock for write op
0709 13:01:08.119417 (+   144us) write_op.cc:641] Partition lock acquired for write op
0709 13:01:08.119468 (+    51us) tablet.cc:685] Acquiring locks for 23 operations
0709 13:01:08.121379 (+  1911us) tablet.cc:701] Row locks acquired
0709 13:01:08.121401 (+    22us) write_op.cc:260] PREPARE: finished
0709 13:01:08.121595 (+   194us) write_op.cc:270] Start()
0709 13:01:08.121746 (+   151us) write_op.cc:276] Timestamp: P: 1783602068121535 usec, L: 0
0709 13:01:08.121772 (+    26us) op_driver.cc:348] REPLICATION: starting
0709 13:01:08.298158 (+176386us) log.cc:844] Serialized 8963 byte log entry
0709 13:01:09.055362 (+757204us) op_driver.cc:464] REPLICATION: finished
0709 13:01:09.478765 (+423403us) spinlock_profiling.cc:243] Waited 21.2 ms on lock 0x7b4400022b4c. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f4262f9bc85 00007f4262f9c6a2 00007f4265d0e1db 00007f4265d09aaf 00007f42630580ac 00007f426beb2ab9 00007f426307469b 00007f426307293f 00007f4263074c02 00007f4269b98ef0
0709 13:01:10.193813 (+715048us) write_op.cc:301] APPLY: starting
0709 13:01:10.246584 (+ 52771us) tablet.cc:1367] starting BulkCheckPresence
0709 13:01:11.126840 (+880256us) tablet.cc:1370] finished BulkCheckPresence
0709 13:01:11.126889 (+    49us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 13:01:11.710361 (+583472us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 13:01:11.710896 (+   535us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=46,key_file_lookups=46,delta_file_lookups=46,mrs_lookups=0
0709 13:01:11.710950 (+    54us) write_op.cc:312] APPLY: finished
0709 13:01:11.711577 (+   627us) log.cc:844] Serialized 204 byte log entry
0709 13:01:12.001790 (+290213us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:01:12.150403 (+148613us) write_op.cc:454] Released schema lock
0709 13:01:12.150517 (+   114us) write_op.cc:341] FINISH: Updating metrics
0709 13:01:12.151238 (+   721us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:01:12.151270 (+    32us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":82779,"reported_tablets":36,"child_traces":[["op",{"apply.queue_time_us":1051425,"apply.run_cpu_time_us":94392,"apply.run_wall_time_us":1983040,"cfile_cache_hit":94,"cfile_cache_hit_bytes":100451,"num_ops":23,"prepare.queue_time_us":75248,"prepare.run_cpu_time_us":44108,"prepare.run_wall_time_us":957938,"raft.queue_time_us":251888,"raft.run_cpu_time_us":69455,"raft.run_wall_time_us":1084862,"replication_time_us":933209,"spinlock_wait_cycles":48808832,"thread_start_us":1760566,"threads_started":5,"wal-append.queue_time_us":860806,"wal-append.run_cpu_time_us":8417,"wal-append.run_wall_time_us":2169289}]]}
I20260709 13:01:17.209734  6706 leader_election.cc:290] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:17.213567  6629 leader_election.cc:290] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:17.214210  6629 raft_consensus.cc:2804] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:01:17.214605  6629 raft_consensus.cc:493] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:17.214990  6629 raft_consensus.cc:3060] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:01:17.217378  6680 raft_consensus.cc:515] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:17.066107  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:17.151902  6676 raft_consensus.cc:3060] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:01:16.928711  6691 leader_election.cc:290] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:16.989342  6528 raft_consensus.cc:515] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:17.072299  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:17.029512  6365 consensus_queue.cc:1048] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.030s
I20260709 13:01:17.033721  6408 raft_consensus.cc:697] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 14 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
W20260709 13:01:17.149698  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 80598.7us
I20260709 13:01:17.039031  6538 raft_consensus.cc:493] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:17.156483  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
W20260709 13:01:17.156855  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 251116us
I20260709 13:01:17.244181  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:17.244329  1751 raft_consensus.cc:2468] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 10.
I20260709 13:01:17.061098  6699 raft_consensus.cc:493] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:01:17.157429  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 88970.6us
I20260709 13:01:17.247360  2298 leader_election.cc:304] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:01:17.248656  2298 leader_election.cc:304] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:01:17.061345  6643 raft_consensus.cc:2804] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:01:17.055662  6650 raft_consensus.cc:3060] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:01:17.060396  6672 raft_consensus.cc:493] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:01:17.068023  6360 raft_consensus.cc:493] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:17.060971  6622 raft_consensus.cc:740] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:01:17.060707  6700 raft_consensus.cc:2804] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:01:17.165043  6298 leader_election.cc:290] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 15 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:17.166960  6599 raft_consensus.cc:3055] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:01:17.167248  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:17.089293  6485 raft_consensus.cc:493] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:17.293570  1677 raft_consensus.cc:1275] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 14 index: 8. (index mismatch)
I20260709 13:01:17.066031  6696 raft_consensus.cc:3060] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Advancing to term 11
W20260709 13:01:17.094528  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:17.242489  6529 raft_consensus.cc:493] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:17.175237  1500 catalog_manager.cc:5697] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 6 to 8. New cstate: current_term: 8 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 13:01:17.179309  1754 raft_consensus.cc:3060] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:01:17.182334  6688 raft_consensus.cc:493] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:17.062573  1747 raft_consensus.cc:2468] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 12.
I20260709 13:01:17.247020  6538 raft_consensus.cc:3060] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:01:17.085860  6592 raft_consensus.cc:515] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:17.241706  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:17.254506  1746 raft_consensus.cc:2468] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 15.
I20260709 13:01:17.269021  1678 raft_consensus.cc:1240] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 9. Current term is 10. Ops: []
I20260709 13:01:17.272297  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:17.275141  1675 raft_consensus.cc:1240] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 12. Current term is 13. Ops: []
I20260709 13:01:17.275211  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 14 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:17.231843  6160 raft_consensus.cc:2749] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 13:01:17.286417  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:01:17.230805  1719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:17.266716  6643 raft_consensus.cc:493] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:17.313467  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:17.314152  1751 raft_consensus.cc:3060] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Advancing to term 15
I20260709 13:01:17.318599  1500 catalog_manager.cc:5697] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 7 to 9, leader changed from 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) to af520b67945d443b901bbeca37cc5701 (127.0.107.195). New cstate: current_term: 9 leader_uuid: "af520b67945d443b901bbeca37cc5701" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } }
I20260709 13:01:17.322469  1747 raft_consensus.cc:1275] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20260709 13:01:17.326723  6605 raft_consensus.cc:2804] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:01:17.327324  6605 raft_consensus.cc:493] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Starting leader election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:01:17.327792  6605 raft_consensus.cc:3060] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:01:17.302234  6485 raft_consensus.cc:515] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:17.331094  6685 consensus_queue.cc:1048] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 13:01:17.332948  6485 leader_election.cc:290] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:17.337885  6622 consensus_queue.cc:260] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:17.341252  1676 raft_consensus.cc:1240] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 10. Current term is 11. Ops: []
I20260709 13:01:17.342526  6604 raft_consensus.cc:2804] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:01:17.343003  6604 raft_consensus.cc:493] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:17.271656  6699 raft_consensus.cc:515] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:17.343365  6604 raft_consensus.cc:3060] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:01:17.345744  6699 leader_election.cc:290] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:17.346575  2335 raft_consensus.cc:2468] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 10.
I20260709 13:01:17.347426  1671 raft_consensus.cc:1240] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 9. Current term is 10. Ops: []
I20260709 13:01:17.348155  6592 leader_election.cc:290] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:17.291813  6700 raft_consensus.cc:493] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:17.348806  6700 raft_consensus.cc:3060] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Advancing to term 9
W20260709 13:01:17.350090  6703 log.cc:927] Time spent T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.065s	user 0.000s	sys 0.000s
I20260709 13:01:17.291030  6672 raft_consensus.cc:515] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:17.355707  6672 leader_election.cc:290] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:17.356503  1672 raft_consensus.cc:3060] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:01:17.291471  6360 raft_consensus.cc:3060] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Advancing to term 19
W20260709 13:01:17.303864  6701 log.cc:927] Time spent T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.056s	user 0.001s	sys 0.000s
I20260709 13:01:17.311815  1646 leader_election.cc:304] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:17.361016  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:01:17.370399  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 58668.6us
I20260709 13:01:17.367720  6496 raft_consensus.cc:697] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 12 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:01:17.374552  6496 consensus_queue.cc:237] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.9, Last appended by leader: 9, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:17.303939  6688 raft_consensus.cc:3060] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:01:17.359650  1746 raft_consensus.cc:1275] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 11 index: 9. (index mismatch)
I20260709 13:01:17.385221  2301 leader_election.cc:304] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:01:17.378804  1500 catalog_manager.cc:5697] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 5 to 6, leader changed from 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194) to af520b67945d443b901bbeca37cc5701 (127.0.107.195). New cstate: current_term: 6 leader_uuid: "af520b67945d443b901bbeca37cc5701" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } }
I20260709 13:01:17.363250  1751 raft_consensus.cc:2468] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 15.
I20260709 13:01:17.295691  6599 raft_consensus.cc:740] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:01:17.371385  1646 leader_election.cc:304] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:17.378173  1679 raft_consensus.cc:2393] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 11: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:01:17.380136  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:17.391664  2328 raft_consensus.cc:3060] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Advancing to term 12
I20260709 13:01:17.381798  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 12 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:17.392861  2334 raft_consensus.cc:2393] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 12: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:01:17.334283  6485 raft_consensus.cc:3055] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 12 LEADER]: Stepping down as leader of term 12
W20260709 13:01:17.384382  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 136752us
I20260709 13:01:17.393872  1569 leader_election.cc:304] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
W20260709 13:01:17.394780  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 340074us
I20260709 13:01:17.395251  1569 leader_election.cc:304] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:01:17.399135  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:17.395661  1571 leader_election.cc:304] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:17.400116  2334 raft_consensus.cc:2393] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 13: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
W20260709 13:01:17.400578  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 206993us
I20260709 13:01:17.401366  1754 raft_consensus.cc:2468] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 15.
I20260709 13:01:17.401643  1718 leader_election.cc:400] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 10 pre-election: Vote denied by peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261) with higher term. Message: Invalid argument: T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 10. Current term is 11.
I20260709 13:01:17.402006  1718 leader_election.cc:403] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 10 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:01:17.358319  6643 raft_consensus.cc:3060] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:01:17.415385  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:17.413228  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:17.416479  2332 raft_consensus.cc:3060] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Advancing to term 16
W20260709 13:01:17.367206  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 63588.1us
I20260709 13:01:17.417775  1500 catalog_manager.cc:5697] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 8 to 9, leader changed from 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) to 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193). New cstate: current_term: 9 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 13:01:17.425750  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:17.426829  1596 raft_consensus.cc:2393] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 11: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:01:17.308831  6529 raft_consensus.cc:515] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:17.428005  1718 leader_election.cc:304] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:17.429442  2334 raft_consensus.cc:1240] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 12. Current term is 13. Ops: []
I20260709 13:01:17.419934  2301 leader_election.cc:304] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:17.388274  6592 raft_consensus.cc:2804] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Leader election won for term 10
I20260709 13:01:17.419071  1646 leader_election.cc:304] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:17.437893  1646 leader_election.cc:304] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:17.400986  6485 raft_consensus.cc:740] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:01:17.414033  6695 raft_consensus.cc:2804] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Leader election won for term 15
I20260709 13:01:17.447904  2328 raft_consensus.cc:2468] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 12.
I20260709 13:01:17.448089  2301 leader_election.cc:304] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:17.309432  6601 raft_consensus.cc:2749] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Leader pre-election lost for term 11. Reason: Vote denied by peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261) with higher term. Message: Invalid argument: T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 11. Current term is 12.
I20260709 13:01:17.445930  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:17.413332  6298 raft_consensus.cc:493] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:17.449957  1718 leader_election.cc:304] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:17.434274  6702 raft_consensus.cc:2749] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:01:17.461333  2332 raft_consensus.cc:2468] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 16.
W20260709 13:01:17.445914  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 150638us
I20260709 13:01:17.453543  1500 catalog_manager.cc:5697] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 9 to 10, leader changed from 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) to 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194). New cstate: current_term: 10 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } }
I20260709 13:01:17.417045  6672 consensus_queue.cc:1048] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.034s
I20260709 13:01:17.466152  6592 raft_consensus.cc:697] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:01:17.444877  6345 raft_consensus.cc:2804] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Leader election won for term 15
W20260709 13:01:17.476331  1644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:17.452147  6046 raft_consensus.cc:2804] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Leader election won for term 15
I20260709 13:01:17.451190  6699 raft_consensus.cc:493] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:17.444357  6714 raft_consensus.cc:2749] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Leader pre-election lost for term 10. Reason: Vote denied by peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261) with higher term. Message: Invalid argument: T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 10. Current term is 11.
I20260709 13:01:17.417197  6684 consensus_queue.cc:1048] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.049s
I20260709 13:01:17.484325  1500 catalog_manager.cc:5697] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 5 to 6, leader changed from 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) to af520b67945d443b901bbeca37cc5701 (127.0.107.195). New cstate: current_term: 6 leader_uuid: "af520b67945d443b901bbeca37cc5701" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } }
I20260709 13:01:17.403523  6408 consensus_queue.cc:237] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 12.12, Last appended by leader: 12, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:17.464056  6160 raft_consensus.cc:2804] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Leader election won for term 9
W20260709 13:01:17.489329  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 163536us
I20260709 13:01:17.454699  6516 raft_consensus.cc:2804] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Leader pre-election won for term 13
W20260709 13:01:17.492955  1645 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:17.471772  6695 raft_consensus.cc:697] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:01:17.489058  6671 raft_consensus.cc:493] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:17.463824  6298 raft_consensus.cc:515] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } }
I20260709 13:01:17.489756  6171 raft_consensus.cc:493] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:17.507541  1500 catalog_manager.cc:5697] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 5 to 10, leader changed from 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) to 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196). New cstate: current_term: 10 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 13:01:17.497390  6649 raft_consensus.cc:493] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
W20260709 13:01:17.573050  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 177456us
W20260709 13:01:17.574684  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 179505us
I20260709 13:01:17.504143  6516 raft_consensus.cc:493] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:17.508900  6345 raft_consensus.cc:697] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:01:17.607616  1500 catalog_manager.cc:5697] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 6 to 8. New cstate: current_term: 8 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 13:01:17.507041  6717 raft_consensus.cc:2749] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:01:17.515118  6046 raft_consensus.cc:697] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:01:17.475028  6599 consensus_queue.cc:260] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 9.10, Last appended by leader: 10, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:17.569048  6671 raft_consensus.cc:515] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:17.619995  6298 leader_election.cc:290] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:17.499753  6592 consensus_queue.cc:237] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:17.568892  6528 leader_election.cc:290] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:17.583870  6516 raft_consensus.cc:3060] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:01:17.503973  6485 consensus_queue.cc:260] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 12.6, Last appended by leader: 6, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:01:17.629783  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 179886us
I20260709 13:01:17.652035  1718 leader_election.cc:304] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:17.629163  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:17.609938  1500 catalog_manager.cc:5697] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 6 to 9, leader changed from 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194) to af520b67945d443b901bbeca37cc5701 (127.0.107.195). New cstate: current_term: 9 leader_uuid: "af520b67945d443b901bbeca37cc5701" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } }
I20260709 13:01:17.637023  1672 raft_consensus.cc:2410] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 for term 13 because replica has last-logged OpId of term: 12 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 10.
I20260709 13:01:17.653316  1753 raft_consensus.cc:2468] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 10.
I20260709 13:01:17.571208  6681 raft_consensus.cc:2804] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Leader pre-election won for term 15
W20260709 13:01:17.648185  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 200159us
I20260709 13:01:17.645007  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:17.500393  6699 raft_consensus.cc:515] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:17.643080  1569 leader_election.cc:304] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:17.643719  1571 leader_election.cc:304] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:17.654768  2301 leader_election.cc:304] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:17.660360  2298 leader_election.cc:304] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:17.625015  6672 raft_consensus.cc:493] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:17.669180  2301 leader_election.cc:304] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:17.670583  2301 leader_election.cc:304] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:17.572468  6171 raft_consensus.cc:515] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:17.671731  2301 leader_election.cc:304] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:01:17.581048  6649 raft_consensus.cc:515] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:17.678745  1500 catalog_manager.cc:5697] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 6 to 7, leader changed from 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194) to af520b67945d443b901bbeca37cc5701 (127.0.107.195). New cstate: current_term: 7 leader_uuid: "af520b67945d443b901bbeca37cc5701" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } }
W20260709 13:01:17.639020  6332 consensus_meta.cc:287] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.988s	user 0.018s	sys 0.000s
I20260709 13:01:17.690518  2301 leader_election.cc:304] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
W20260709 13:01:17.666533  6712 log.cc:927] Time spent T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.253s	user 0.000s	sys 0.001s
I20260709 13:01:17.649793  6714 raft_consensus.cc:2749] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20260709 13:01:17.698149  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 46173.3us
I20260709 13:01:17.679374  6702 raft_consensus.cc:2764] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Leader pre-election decision vote started in defunct term 15: won
I20260709 13:01:17.705956  1718 leader_election.cc:304] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:17.706847  6726 raft_consensus.cc:2804] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 13:01:17.706696  6528 consensus_queue.cc:1048] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 13:01:17.707592  6691 raft_consensus.cc:3060] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:01:17.708421  1500 catalog_manager.cc:5697] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 5 to 8, leader changed from 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194) to af520b67945d443b901bbeca37cc5701 (127.0.107.195). New cstate: current_term: 8 leader_uuid: "af520b67945d443b901bbeca37cc5701" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } }
I20260709 13:01:17.709621  6649 leader_election.cc:290] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:17.695261  6332 raft_consensus.cc:515] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:17.683566  6512 raft_consensus.cc:2804] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 13:01:17.706279  1644 leader_election.cc:304] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
W20260709 13:01:17.751377  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 45162.3us
I20260709 13:01:17.657619  6724 raft_consensus.cc:2804] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Leader election won for term 13
W20260709 13:01:17.713912  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 139551us
I20260709 13:01:17.680176  6684 raft_consensus.cc:3060] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:01:17.679054  6717 raft_consensus.cc:2804] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Leader election won for term 13
W20260709 13:01:17.691213  6713 log.cc:927] Time spent T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.270s	user 0.000s	sys 0.001s
I20260709 13:01:17.693235  6485 raft_consensus.cc:3060] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:01:17.755929  1571 leader_election.cc:304] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:17.730654  1718 leader_election.cc:304] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:17.647405  6638 raft_consensus.cc:515] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:17.757658  1644 leader_election.cc:304] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:17.751044  1500 catalog_manager.cc:5697] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 6 to 8, leader changed from af520b67945d443b901bbeca37cc5701 (127.0.107.195) to 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196). New cstate: current_term: 8 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 13:01:17.763214  1644 leader_election.cc:304] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
W20260709 13:01:17.765385  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 122385us
I20260709 13:01:17.691573  6672 raft_consensus.cc:515] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:17.766692  1569 leader_election.cc:304] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:17.712253  6716 raft_consensus.cc:2749] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:01:17.749018  6714 raft_consensus.cc:2749] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:01:17.639854  6695 consensus_queue.cc:237] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.7, Last appended by leader: 7, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:17.774281  1571 leader_election.cc:304] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:17.754093  6702 raft_consensus.cc:2749] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:01:17.750969  6529 leader_election.cc:290] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:17.754930  6649 raft_consensus.cc:2749] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 13:01:17.756908  6711 raft_consensus.cc:493] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:01:17.782276  1569 leader_election.cc:304] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:01:17.773141  6694 raft_consensus.cc:2804] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:01:17.772930  6496 raft_consensus.cc:2804] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:01:17.760795  6730 raft_consensus.cc:2804] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:01:17.796273  1500 catalog_manager.cc:5697] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 9 to 11. New cstate: current_term: 11 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } }
I20260709 13:01:17.797401  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:17.797680  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 12 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:17.803061  2328 raft_consensus.cc:2393] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 12: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:01:17.798570  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:17.772848  6724 raft_consensus.cc:697] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:01:17.749018  6345 consensus_queue.cc:237] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 12.8, Last appended by leader: 8, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:01:17.812692  6721 log.cc:927] Time spent T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.052s	user 0.001s	sys 0.000s
I20260709 13:01:17.813367  6671 leader_election.cc:290] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:17.776468  6728 raft_consensus.cc:2749] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:01:17.777840  6732 raft_consensus.cc:2804] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Leader election won for term 16
I20260709 13:01:17.815304  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 15 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:01:17.822461  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:17.848901  2298 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:17.850394  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:17.801627  6496 raft_consensus.cc:493] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:17.861183  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:17.762574  6046 consensus_queue.cc:237] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.9, Last appended by leader: 9, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:17.804682  6730 raft_consensus.cc:493] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:17.814718  6694 raft_consensus.cc:697] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:01:17.847231  6726 raft_consensus.cc:697] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 10 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:01:17.871822  2337 raft_consensus.cc:2468] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 10.
I20260709 13:01:17.870667  1500 catalog_manager.cc:5697] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
W20260709 13:01:17.872628  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 90425.5us
I20260709 13:01:17.876297  1569 leader_election.cc:304] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:01:17.867095  6496 raft_consensus.cc:3060] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:01:17.882411  6332 leader_election.cc:290] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 12 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:01:17.886910  6731 log.cc:927] Time spent T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.106s	user 0.000s	sys 0.001s
I20260709 13:01:17.885375  6694 consensus_queue.cc:237] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.8, Last appended by leader: 8, Current term: 12, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:17.887032  6726 consensus_queue.cc:237] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:17.885783  6708 consensus_meta.cc:287] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.688s	user 0.005s	sys 0.007s
I20260709 13:01:17.887571  6724 consensus_queue.cc:237] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:01:17.892692  5989 consensus_meta.cc:287] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.694s	user 0.010s	sys 0.003s
W20260709 13:01:17.893378  6544 consensus_meta.cc:287] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.697s	user 0.012s	sys 0.001s
I20260709 13:01:17.893561  6688 raft_consensus.cc:515] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:17.897816  6714 raft_consensus.cc:493] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:17.898965  6516 raft_consensus.cc:515] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:17.902827  6729 raft_consensus.cc:2749] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:01:17.904918  6735 raft_consensus.cc:2804] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Leader election won for term 11
I20260709 13:01:17.873148  1500 catalog_manager.cc:5697] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 5 to 9. New cstate: current_term: 9 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 13:01:17.905160  6699 leader_election.cc:290] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:17.869597  6730 raft_consensus.cc:3060] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:01:17.866488  6732 raft_consensus.cc:697] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 16 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:01:17.906651  6672 leader_election.cc:290] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:01:17.909420  6604 consensus_meta.cc:287] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.566s	user 0.008s	sys 0.004s
I20260709 13:01:17.984444  6604 raft_consensus.cc:515] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:17.870194  6736 consensus_queue.cc:1048] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.044s
I20260709 13:01:17.915454  1569 leader_election.cc:304] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:01:17.986698  6715 raft_consensus.cc:2749] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
W20260709 13:01:17.987321  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 71948.2us
I20260709 13:01:17.987830  6715 raft_consensus.cc:2749] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:01:17.988548  1569 leader_election.cc:304] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:01:17.988610  6715 consensus_queue.cc:1048] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 13:01:17.918656  6717 raft_consensus.cc:697] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:01:17.990564  6622 raft_consensus.cc:3060] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:01:17.991294  6717 consensus_queue.cc:237] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.6, Last appended by leader: 6, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:17.994822  6735 raft_consensus.cc:697] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:01:17.995081  6732 consensus_queue.cc:237] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.10, Last appended by leader: 10, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:17.995949  6735 consensus_queue.cc:237] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:17.877753  6737 raft_consensus.cc:2804] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:01:17.998783  6737 raft_consensus.cc:493] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:17.999213  6737 raft_consensus.cc:3060] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:01:17.796036  6711 raft_consensus.cc:515] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:17.932056  6700 consensus_meta.cc:287] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.583s	user 0.009s	sys 0.005s
I20260709 13:01:17.916636  6723 raft_consensus.cc:2749] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 13:01:17.916364  6512 raft_consensus.cc:697] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 13 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
W20260709 13:01:17.954238  6727 log.cc:927] Time spent T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.182s	user 0.001s	sys 0.001s
W20260709 13:01:17.981272  6360 consensus_meta.cc:287] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.580s	user 0.016s	sys 0.005s
I20260709 13:01:17.996511  6680 leader_election.cc:290] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:17.919165  6160 raft_consensus.cc:697] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 9 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:01:18.000203  1569 leader_election.cc:304] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:01:17.957537  6171 leader_election.cc:290] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:18.017990  1569 leader_election.cc:400] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 election: Vote denied by peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625) with higher term. Message: Invalid argument: T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 13. Current term is 14.
W20260709 13:01:17.964298  6676 consensus_meta.cc:287] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.720s	user 0.016s	sys 0.002s
W20260709 13:01:18.016585  1644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:18.018383  1569 leader_election.cc:403] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 election: Cancelling election due to peer responding with higher term
I20260709 13:01:18.017050  6360 raft_consensus.cc:515] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:18.017581  6740 raft_consensus.cc:493] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:18.027455  6740 raft_consensus.cc:515] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:18.028395  6360 leader_election.cc:290] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 19 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:18.029211  2329 raft_consensus.cc:2368] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 18. Current term is 19.
I20260709 13:01:18.029719  6740 leader_election.cc:290] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:18.029810  6711 leader_election.cc:290] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:18.031345  6688 leader_election.cc:290] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 9 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:18.031024  6512 consensus_queue.cc:237] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.11, Last appended by leader: 11, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:18.032151  6688 raft_consensus.cc:2804] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Leader election won for term 11
I20260709 13:01:17.975715  6714 raft_consensus.cc:515] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:18.040565  1596 raft_consensus.cc:1275] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 13 index: 12. (index mismatch)
I20260709 13:01:17.998807  6681 raft_consensus.cc:493] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:17.961658  6544 raft_consensus.cc:515] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:17.960291  6708 raft_consensus.cc:515] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:18.044039  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:18.003734  6736 raft_consensus.cc:2804] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:01:18.048225  1596 raft_consensus.cc:3060] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Advancing to term 12
I20260709 13:01:18.049561  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:18.056495  1597 raft_consensus.cc:3060] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:01:18.061475  6604 leader_election.cc:290] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 12 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:01:18.061717  6744 log.cc:927] Time spent T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.053s	user 0.000s	sys 0.000s
I20260709 13:01:18.010499  6700 raft_consensus.cc:515] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:18.063884  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:18.064520  1599 raft_consensus.cc:2468] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 12.
I20260709 13:01:18.034142  6699 raft_consensus.cc:493] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:18.018707  6638 leader_election.cc:290] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 13 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:18.032646  6688 raft_consensus.cc:697] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:01:18.009017  6715 consensus_queue.cc:1048] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.046s
I20260709 13:01:18.047374  6681 raft_consensus.cc:3060] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:01:18.052223  6736 raft_consensus.cc:493] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:01:18.069314  6605 consensus_meta.cc:287] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.741s	user 0.014s	sys 0.002s
I20260709 13:01:18.001760  6729 consensus_queue.cc:1048] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.064s
I20260709 13:01:18.087796  1754 raft_consensus.cc:2393] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 10: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:01:18.098137  1596 raft_consensus.cc:2468] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 12.
I20260709 13:01:18.098093  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:01:18.099447  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 79181.7us
I20260709 13:01:18.041772  6676 raft_consensus.cc:515] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:18.113741  1569 leader_election.cc:304] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:01:18.100128  6736 raft_consensus.cc:3060] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:01:18.090677  6516 leader_election.cc:290] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 13 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:18.086277  6743 raft_consensus.cc:2749] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader election lost for term 13. Reason: Vote denied by peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625) with higher term. Message: Invalid argument: T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 13. Current term is 14.
W20260709 13:01:18.099285  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 325036us
I20260709 13:01:18.109812  6723 consensus_queue.cc:1059] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 11.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.013s
I20260709 13:01:18.140424  6748 consensus_queue.cc:1048] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 13:01:18.118714  1597 raft_consensus.cc:2468] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 19.
I20260709 13:01:18.158694  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:18.159575  2328 raft_consensus.cc:2368] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 11. Current term is 12.
I20260709 13:01:18.159370  6748 consensus_queue.cc:1059] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 13:01:18.160279  6748 raft_consensus.cc:3055] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:01:18.160643  6748 raft_consensus.cc:740] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
W20260709 13:01:18.128032  6643 consensus_meta.cc:287] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.699s	user 0.014s	sys 0.004s
I20260709 13:01:18.161302  6643 raft_consensus.cc:515] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:18.161633  6748 consensus_queue.cc:260] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:18.162936  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:18.163578  6643 leader_election.cc:290] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 16 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:18.127699  6753 consensus_queue.cc:1059] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.037s
I20260709 13:01:18.129262  6700 leader_election.cc:290] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 9 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:18.164223  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:18.162963  6748 raft_consensus.cc:3060] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:01:18.164990  2336 raft_consensus.cc:2393] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 16: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:01:18.165657  6754 raft_consensus.cc:2804] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:01:18.117568  6742 consensus_queue.cc:1048] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.010s
I20260709 13:01:18.105613  6752 consensus_queue.cc:1059] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 9.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.027s
W20260709 13:01:18.115291  6696 consensus_meta.cc:287] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.797s	user 0.007s	sys 0.012s
I20260709 13:01:18.167987  6696 raft_consensus.cc:2749] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:01:18.137513  1745 raft_consensus.cc:2393] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 16: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:01:18.119544  6160 consensus_queue.cc:237] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:18.177425  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:18.165601  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:18.113701  2298 leader_election.cc:304] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:01:18.160640  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:18.180316  2328 raft_consensus.cc:2468] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 11.
I20260709 13:01:18.181072  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:01:18.124871  6507 consensus_meta.cc:287] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.977s	user 0.013s	sys 0.001s
I20260709 13:01:18.091004  6699 raft_consensus.cc:515] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:18.127241  1571 leader_election.cc:304] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:18.184808  2299 leader_election.cc:304] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:18.186141  2331 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39726 (request call id 1164) took 1116 ms. Trace:
W20260709 13:01:18.187064  1717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:18.187610  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:18.183876  6528 raft_consensus.cc:493] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:18.191339  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:18.191500  6528 raft_consensus.cc:515] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:18.125945  6680 raft_consensus.cc:493] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:18.126304  1569 leader_election.cc:304] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
W20260709 13:01:18.184938  6650 consensus_meta.cc:287] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.894s	user 0.009s	sys 0.005s
W20260709 13:01:18.197048  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 83482.6us
I20260709 13:01:18.186599  2331 rpcz_store.cc:276] 0709 13:01:17.069614 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:17.069742 (+   128us) service_pool.cc:224] Handling call
0709 13:01:17.157270 (+ 87528us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:17.162523 (+  5253us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 13:01:17.162841 (+   318us) log.cc:844] Serialized 14 byte log entry
0709 13:01:17.247352 (+ 84511us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:01:17.247413 (+    61us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:01:17.247477 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:17.636143 (+388666us) spinlock_profiling.cc:243] Waited 19.7 ms on lock 0x7b180044d6a0. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f42630432a6 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419
0709 13:01:18.185834 (+549691us) spinlock_profiling.cc:243] Waited 4.96 ms on lock 0x7b540065e0d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:01:18.185918 (+    84us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:18.186073 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":40655,"prepare.queue_time_us":3833,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":36,"raft.queue_time_us":156,"raft.run_cpu_time_us":162,"raft.run_wall_time_us":18294,"spinlock_wait_cycles":56619648,"thread_start_us":84000,"threads_started":1,"wal-append.queue_time_us":154948}
I20260709 13:01:18.200736  2298 leader_election.cc:304] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:01:18.174633  6754 raft_consensus.cc:493] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
W20260709 13:01:18.202294  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 75167.6us
I20260709 13:01:18.105980  6605 raft_consensus.cc:515] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:18.195065  6753 raft_consensus.cc:3055] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:01:18.226261  1675 raft_consensus.cc:1275] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 12 index: 10. (index mismatch)
W20260709 13:01:18.226394  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 100158us
I20260709 13:01:18.204293  2298 leader_election.cc:304] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:01:18.229247  1569 leader_election.cc:304] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:18.205597  6438 raft_consensus.cc:2749] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 13:01:18.206753  6672 raft_consensus.cc:493] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:18.235165  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:18.214903  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:18.235898  2332 raft_consensus.cc:2468] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 12.
I20260709 13:01:18.236452  1671 raft_consensus.cc:2468] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 15.
I20260709 13:01:18.216712  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:18.221263  1571 leader_election.cc:304] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:01:18.223979  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:18.205030  6742 raft_consensus.cc:2804] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:01:18.237917  1679 raft_consensus.cc:3060] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:01:18.232157  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:18.239552  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:18.244904  1602 raft_consensus.cc:1240] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 8. Current term is 9. Ops: []
I20260709 13:01:18.245615  1597 raft_consensus.cc:1240] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 8. Current term is 9. Ops: []
I20260709 13:01:18.255770  1571 leader_election.cc:304] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:18.258366  2328 raft_consensus.cc:2393] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 14: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:01:18.271909  6496 raft_consensus.cc:515] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:18.275221  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 46072.4us
I20260709 13:01:18.283020  6496 leader_election.cc:290] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 12 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:18.272534  1679 raft_consensus.cc:2468] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 16.
I20260709 13:01:18.202100  6643 raft_consensus.cc:2804] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:01:18.237059  6592 raft_consensus.cc:493] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:18.226117  6753 raft_consensus.cc:740] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:01:18.227349  6754 raft_consensus.cc:3060] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:01:18.201933  6680 raft_consensus.cc:515] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:18.231683  6507 raft_consensus.cc:2804] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:01:18.254248  6516 raft_consensus.cc:2804] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 13:01:18.235623  6700 raft_consensus.cc:2749] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20260709 13:01:18.326660  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 70971.1us
I20260709 13:01:18.265717  6735 raft_consensus.cc:2749] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:01:18.220387  6650 raft_consensus.cc:515] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:18.236927  6638 consensus_queue.cc:1048] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 7, Time since last communication: 0.083s
I20260709 13:01:18.265292  6742 raft_consensus.cc:697] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 12 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:01:18.344437  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:18.344585  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:18.420351  1598 raft_consensus.cc:2468] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 10.
I20260709 13:01:18.329699  6592 raft_consensus.cc:515] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:18.421809  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:18.196079  6688 consensus_queue.cc:237] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 9.10, Last appended by leader: 10, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:18.423203  6592 leader_election.cc:290] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:18.297243  6643 raft_consensus.cc:493] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:18.343353  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:18.300704  6715 raft_consensus.cc:2749] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:01:18.355335  6528 leader_election.cc:290] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:18.254840  6672 raft_consensus.cc:515] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:18.329056  6438 consensus_queue.cc:1059] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.028s
I20260709 13:01:18.359474  1750 raft_consensus.cc:2393] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 16: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:01:18.360054  6680 leader_election.cc:290] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:18.360781  6650 leader_election.cc:290] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:18.332280  6507 raft_consensus.cc:493] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:18.365875  6676 leader_election.cc:290] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:18.372881  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:01:18.376878  6691 consensus_meta.cc:287] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.625s	user 0.011s	sys 0.001s
I20260709 13:01:18.338981  6717 raft_consensus.cc:493] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:18.377707  1746 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55848 (request call id 2554) took 1065 ms. Trace:
I20260709 13:01:18.378459  2333 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39726 (request call id 1161) took 1315 ms. Trace:
W20260709 13:01:18.379689  6684 consensus_meta.cc:287] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.620s	user 0.012s	sys 0.000s
I20260709 13:01:18.384094  2337 raft_consensus.cc:1275] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 11 index: 9. (index mismatch)
W20260709 13:01:18.385155  6757 log.cc:927] Time spent T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.287s	user 0.000s	sys 0.001s
I20260709 13:01:18.388309  2335 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39726 (request call id 1166) took 1316 ms. Trace:
I20260709 13:01:18.388960  6736 raft_consensus.cc:515] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:18.389950  1646 leader_election.cc:304] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
W20260709 13:01:18.495867  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 59530.3us
W20260709 13:01:18.496138  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 106292us
I20260709 13:01:18.391158  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:18.392081  6638 raft_consensus.cc:493] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:18.394932  6753 consensus_queue.cc:260] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:18.395782  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 15 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:18.396448  6516 raft_consensus.cc:697] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 19 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:01:18.396695  6716 raft_consensus.cc:493] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:18.396633  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
W20260709 13:01:18.397535  6761 log.cc:927] Time spent T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.067s	user 0.000s	sys 0.000s
I20260709 13:01:18.464277  6507 raft_consensus.cc:3060] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:01:18.399320  2336 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39726 (request call id 1171) took 1155 ms. Trace:
I20260709 13:01:18.539420  2336 rpcz_store.cc:276] 0709 13:01:17.243639 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:17.243857 (+   218us) service_pool.cc:224] Handling call
0709 13:01:18.399082 (+1155225us) spinlock_profiling.cc:243] Waited 854 ms on lock 0x7b540045b5d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:01:18.399240 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1965247232}
I20260709 13:01:18.399791  6711 raft_consensus.cc:493] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:01:18.400914  1745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55848 (request call id 2543) took 1476 ms. Trace:
W20260709 13:01:18.348140  6629 consensus_meta.cc:287] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 1.133s	user 0.010s	sys 0.005s
I20260709 13:01:18.406999  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:18.408125  6730 raft_consensus.cc:515] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:18.329525  6752 raft_consensus.cc:2804] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:01:18.412161  6605 leader_election.cc:290] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 14 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:18.425101  1600 raft_consensus.cc:2410] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for term 14 because replica has last-logged OpId of term: 12 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 7.
I20260709 13:01:18.426085  1598 raft_consensus.cc:2468] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 13.
W20260709 13:01:18.356976  6538 consensus_meta.cc:287] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.998s	user 0.016s	sys 0.006s
I20260709 13:01:18.332653  6726 consensus_queue.cc:1059] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 8.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.105s
I20260709 13:01:18.433972  6643 raft_consensus.cc:3060] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:01:18.453894  1718 leader_election.cc:304] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
W20260709 13:01:18.575222  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 121484us
I20260709 13:01:18.575357  6160 raft_consensus.cc:2804] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:01:18.575973  6160 raft_consensus.cc:493] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:01:18.576447  6160 raft_consensus.cc:3060] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:01:18.576269  1718 leader_election.cc:304] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:18.577356  6650 raft_consensus.cc:2749] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 13:01:18.438511  6700 raft_consensus.cc:493] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:01:18.461042  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:18.464900  2299 leader_election.cc:304] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
W20260709 13:01:18.578727  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:18.578325  6700 raft_consensus.cc:515] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:18.582039  6700 leader_election.cc:290] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:01:18.496830  1644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:18.497380  1751 raft_consensus.cc:2468] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 11.
I20260709 13:01:18.459316  6699 leader_election.cc:290] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:18.495440  2335 rpcz_store.cc:276] 0709 13:01:17.071564 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:17.072058 (+   494us) service_pool.cc:224] Handling call
0709 13:01:18.388125 (+1316067us) spinlock_profiling.cc:243] Waited 114 ms on lock 0x7b54005e01d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:01:18.388239 (+   114us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":263124736}
I20260709 13:01:18.398170  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:18.532727  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:18.494339  2333 rpcz_store.cc:276] 0709 13:01:17.063402 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:17.063887 (+   485us) service_pool.cc:224] Handling call
0709 13:01:18.378212 (+1314325us) spinlock_profiling.cc:243] Waited 859 ms on lock 0x7b5400627ad4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:01:18.378364 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1976591104}
I20260709 13:01:18.586403  1753 raft_consensus.cc:2410] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for term 14 because replica has last-logged OpId of term: 12 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 7.
I20260709 13:01:18.494760  1746 rpcz_store.cc:276] 0709 13:01:17.312320 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:17.312505 (+   185us) service_pool.cc:224] Handling call
0709 13:01:17.312833 (+   328us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:17.387103 (+ 74270us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:18.377390 (+990287us) spinlock_profiling.cc:243] Waited 46.4 ms on lock 0x7b54000e4554. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:01:18.377631 (+   241us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":106806144}
W20260709 13:01:18.495802  6485 consensus_meta.cc:287] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.733s	user 0.013s	sys 0.000s
I20260709 13:01:18.544571  1745 rpcz_store.cc:276] 0709 13:01:16.924565 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:16.924735 (+   170us) service_pool.cc:224] Handling call
0709 13:01:18.400713 (+1475978us) spinlock_profiling.cc:243] Waited 1.21 s on lock 0x7b54001d5bd4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:01:18.400848 (+   135us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2784514944}
I20260709 13:01:18.494619  6691 raft_consensus.cc:2749] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:01:18.498948  6684 raft_consensus.cc:2749] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:01:18.554880  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:18.457437  6737 raft_consensus.cc:515] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:18.602591  2335 raft_consensus.cc:1275] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 12 index: 10. (index mismatch)
I20260709 13:01:18.557366  2336 raft_consensus.cc:1240] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 13. Current term is 14. Ops: []
I20260709 13:01:18.603797  1748 raft_consensus.cc:2410] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for term 11 because replica has last-logged OpId of term: 9 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 5.
I20260709 13:01:18.604513  1646 leader_election.cc:304] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:18.543113  6695 raft_consensus.cc:2804] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:01:18.583600  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:18.584252  1602 raft_consensus.cc:2368] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 10. Current term is 11.
I20260709 13:01:18.588598  1598 raft_consensus.cc:2468] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 10.
W20260709 13:01:18.591084  6766 log.cc:927] Time spent T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.091s	user 0.000s	sys 0.001s
I20260709 13:01:18.611371  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:18.591369  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:01:18.612860  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 148062us
I20260709 13:01:18.593434  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:18.531065  6717 raft_consensus.cc:515] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:18.615556  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:18.617949  1751 raft_consensus.cc:3060] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Advancing to term 19
I20260709 13:01:18.545401  6638 raft_consensus.cc:515] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:18.619268  1599 raft_consensus.cc:2468] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 15.
I20260709 13:01:18.622354  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:01:18.624431  1719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:18.626864  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:18.627478  1598 raft_consensus.cc:3060] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:01:18.587723  6753 raft_consensus.cc:3060] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:01:18.551839  6629 raft_consensus.cc:515] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:18.640231  1717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:18.544955  6716 raft_consensus.cc:515] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:18.584815  6752 raft_consensus.cc:697] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:01:18.593501  6730 leader_election.cc:290] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:18.561508  6538 raft_consensus.cc:515] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:18.654982  1751 raft_consensus.cc:2468] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 19.
I20260709 13:01:18.550837  6711 raft_consensus.cc:515] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:18.643795  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:18.648037  1600 raft_consensus.cc:2468] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 10.
I20260709 13:01:18.621584  6604 raft_consensus.cc:2804] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Leader pre-election won for term 16
W20260709 13:01:18.631167  6763 log.cc:927] Time spent T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.091s	user 0.001s	sys 0.000s
I20260709 13:01:18.658191  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:18.614607  6695 raft_consensus.cc:493] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:18.672015  1603 raft_consensus.cc:1240] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 9. Current term is 10. Ops: []
I20260709 13:01:18.661938  1598 raft_consensus.cc:2468] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 14.
I20260709 13:01:18.541456  6742 consensus_queue.cc:237] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:18.676363  1749 raft_consensus.cc:3060] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:01:18.649772  6672 leader_election.cc:290] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:18.663393  1747 raft_consensus.cc:2468] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 13.
I20260709 13:01:18.598358  6681 raft_consensus.cc:515] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:18.622169  6748 raft_consensus.cc:493] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:18.667629  1604 raft_consensus.cc:1240] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 12. Current term is 13. Ops: [12.9-12.9]
I20260709 13:01:18.668109  1751 raft_consensus.cc:2393] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 12: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:01:18.665014  6604 raft_consensus.cc:493] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:01:18.699759  2298 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:18.652869  6046 raft_consensus.cc:2804] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:01:18.618320  6485 consensus_queue.cc:1048] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 7, Time since last communication: 0.046s
I20260709 13:01:18.627197  6516 consensus_queue.cc:237] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.12, Last appended by leader: 12, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:18.667341  6746 raft_consensus.cc:3055] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:01:18.718858  6717 leader_election.cc:290] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:18.724315  6743 raft_consensus.cc:3055] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:01:18.725020  6743 raft_consensus.cc:740] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:01:18.725898  6737 leader_election.cc:290] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 12 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:18.731146  6736 leader_election.cc:290] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 10 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:18.713766  6748 raft_consensus.cc:515] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:18.718233  6746 raft_consensus.cc:740] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:01:18.737639  6746 consensus_queue.cc:260] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:18.739264  6746 raft_consensus.cc:3060] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:01:18.738911  6743 consensus_queue.cc:260] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:18.741324  6743 raft_consensus.cc:3060] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:01:18.744323  6638 leader_election.cc:290] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:18.746946  6716 leader_election.cc:290] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:18.693780  1749 raft_consensus.cc:2468] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 12.
I20260709 13:01:18.674278  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:18.749550  1598 raft_consensus.cc:2468] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 13.
I20260709 13:01:18.750586  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:18.751719  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:18.752388  1598 raft_consensus.cc:2393] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 12: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:01:18.703480  6604 raft_consensus.cc:3060] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:01:18.708559  6681 leader_election.cc:290] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 15 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:01:18.709514  6767 log.cc:927] Time spent T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.103s	user 0.001s	sys 0.000s
I20260709 13:01:18.656298  6650 consensus_queue.cc:1059] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 13.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.032s
I20260709 13:01:18.710788  2328 raft_consensus.cc:1240] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 14. Current term is 15. Ops: []
I20260709 13:01:18.713595  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:18.757256  1747 raft_consensus.cc:3060] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:01:18.674927  6695 raft_consensus.cc:3060] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:01:18.762884  6708 leader_election.cc:290] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 9 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:18.754170  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:18.764045  1598 raft_consensus.cc:2410] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for term 14 because replica has last-logged OpId of term: 12 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 7.
I20260709 13:01:18.765051  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:18.767172  1752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55848 (request call id 2564) took 1143 ms. Trace:
I20260709 13:01:18.767674  1752 rpcz_store.cc:276] 0709 13:01:17.624025 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:17.624277 (+   252us) service_pool.cc:224] Handling call
0709 13:01:18.766931 (+1142654us) spinlock_profiling.cc:243] Waited 890 ms on lock 0x7b5400116cd4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:01:18.767089 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2046429568}
I20260709 13:01:18.762075  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:18.769198  6538 leader_election.cc:290] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 10 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:18.769680  1599 raft_consensus.cc:2376] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 14. Re-sending same reply.
I20260709 13:01:18.771970  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:18.738059  6748 leader_election.cc:290] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:18.748023  6752 consensus_queue.cc:237] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:18.777356  1747 raft_consensus.cc:2468] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 10.
I20260709 13:01:18.781026  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:18.782404  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:18.784302  1596 raft_consensus.cc:2410] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for term 15 because replica has last-logged OpId of term: 14 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 11.
I20260709 13:01:18.717028  6046 raft_consensus.cc:493] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:18.794278  1750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55848 (request call id 2550) took 1764 ms. Trace:
I20260709 13:01:18.811004  1750 rpcz_store.cc:276] 0709 13:01:17.030327 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:17.031145 (+   818us) service_pool.cc:224] Handling call
0709 13:01:18.794018 (+1762873us) spinlock_profiling.cc:243] Waited 1.16 s on lock 0x7b5400360954. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:01:18.794194 (+   176us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2660539136}
W20260709 13:01:18.795310  2329 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39738 (request call id 956) took 1866 ms (client timeout 1905 ms). Trace:
W20260709 13:01:18.812623  2329 rpcz_store.cc:269] 0709 13:01:16.928825 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:16.942551 (+ 13726us) service_pool.cc:224] Handling call
0709 13:01:18.795037 (+1852486us) spinlock_profiling.cc:243] Waited 989 ms on lock 0x7b54005bca54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:01:18.795228 (+   191us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2274197248}
I20260709 13:01:18.796478  1754 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55848 (request call id 2561) took 1353 ms. Trace:
I20260709 13:01:18.814074  1754 rpcz_store.cc:276] 0709 13:01:17.443330 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:17.445652 (+  2322us) service_pool.cc:224] Handling call
0709 13:01:18.796208 (+1350556us) spinlock_profiling.cc:243] Waited 638 ms on lock 0x7b540019bed4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:01:18.796394 (+   186us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1467756928}
I20260709 13:01:18.815400  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:18.816432  1754 raft_consensus.cc:2393] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 16: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:01:18.798408  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:18.818017  1597 raft_consensus.cc:2393] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 12: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:01:18.819979  1751 raft_consensus.cc:1240] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 9. Current term is 10. Ops: []
I20260709 13:01:18.827834  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:18.831475  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:18.833010  1597 raft_consensus.cc:2393] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 14: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
W20260709 13:01:18.835314  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:18.836730  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:18.837568  1602 raft_consensus.cc:2368] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 18. Current term is 19.
I20260709 13:01:18.838620  2299 leader_election.cc:304] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:01:18.847105  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:18.847966  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:18.848412  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:18.848945  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:18.844204  1754 raft_consensus.cc:1240] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 12. Current term is 13. Ops: []
I20260709 13:01:18.849771  2335 raft_consensus.cc:2368] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for earlier term 10. Current term is 11.
I20260709 13:01:18.850874  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:18.851217  1672 raft_consensus.cc:1275] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 13 index: 12. (index mismatch)
I20260709 13:01:18.852629  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:18.854626  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:18.855355  1597 raft_consensus.cc:3060] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:01:18.782632  6711 leader_election.cc:290] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:18.858861  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:18.830381  6046 raft_consensus.cc:3060] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:01:18.888998  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:18.889595  1747 raft_consensus.cc:1240] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 13. Current term is 14. Ops: []
I20260709 13:01:18.893002  1676 raft_consensus.cc:1275] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 12 index: 12. Preceding OpId from leader: term: 14 index: 13. (index mismatch)
I20260709 13:01:18.894366  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 13 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:18.895555  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:18.892565  6512 raft_consensus.cc:493] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:18.897832  6512 raft_consensus.cc:515] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:18.900190  6512 leader_election.cc:290] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:18.902026  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:18.902535  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:18.907204  6711 raft_consensus.cc:493] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:18.907603  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:01:18.908870  6770 log.cc:927] Time spent T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.193s	user 0.000s	sys 0.000s
I20260709 13:01:18.907955  6711 raft_consensus.cc:515] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:18.852147  6344 raft_consensus.cc:493] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:18.901876  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:18.765046  6507 raft_consensus.cc:515] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:18.877080  6705 raft_consensus.cc:2804] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 13:01:18.923372  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:18.923813  1672 raft_consensus.cc:1240] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 LEADER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 14. Current term is 15. Ops: [14.8-14.8]
I20260709 13:01:18.924005  1754 raft_consensus.cc:3060] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:01:18.891192  6648 raft_consensus.cc:493] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:18.929399  1672 raft_consensus.cc:1240] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 LEADER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 14. Current term is 15. Ops: []
W20260709 13:01:18.926378  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 100605us
I20260709 13:01:18.885227  1596 raft_consensus.cc:2468] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 11.
I20260709 13:01:18.945948  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:18.946082  1597 raft_consensus.cc:2468] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 11.
I20260709 13:01:18.947221  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:18.947926  1597 raft_consensus.cc:2376] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 11. Re-sending same reply.
I20260709 13:01:18.948812  1672 raft_consensus.cc:1240] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 12. Current term is 13. Ops: []
I20260709 13:01:18.961033  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:18.963168  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:18.972842  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:18.975209  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:18.941601  6705 raft_consensus.cc:697] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 16 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:01:18.919972  6344 raft_consensus.cc:515] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:18.983599  1747 raft_consensus.cc:1275] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260709 13:01:19.006532  1569 leader_election.cc:304] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:18.982551  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:19.007586  2301 leader_election.cc:304] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:01:19.009016  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:19.011816  2301 leader_election.cc:304] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:01:19.020800  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:01:18.999255  6777 log.cc:927] Time spent T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.059s	user 0.000s	sys 0.000s
I20260709 13:01:18.945351  6648 raft_consensus.cc:515] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:18.996546  6507 leader_election.cc:290] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:19.022256  6740 raft_consensus.cc:2749] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:01:18.980151  6711 leader_election.cc:290] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:18.995255  6629 leader_election.cc:290] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:19.025990  6705 consensus_queue.cc:237] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.10, Last appended by leader: 10, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:19.026988  6714 leader_election.cc:290] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:18.984090  6538 consensus_queue.cc:1059] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 12.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.056s
I20260709 13:01:18.948824  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
W20260709 13:01:19.013283  6754 consensus_meta.cc:287] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.663s	user 0.014s	sys 0.001s
I20260709 13:01:19.034201  1672 raft_consensus.cc:1275] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 13 index: 12. (index mismatch)
I20260709 13:01:18.949100  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:19.009420  6676 raft_consensus.cc:493] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:19.023850  6684 raft_consensus.cc:2749] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:01:19.009684  6160 raft_consensus.cc:515] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:19.016386  6729 raft_consensus.cc:2749] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 13:01:19.042279  1754 raft_consensus.cc:2468] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 15.
I20260709 13:01:19.029012  6706 consensus_queue.cc:1048] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.029s
I20260709 13:01:19.017792  6695 raft_consensus.cc:515] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } }
I20260709 13:01:19.042845  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:19.056465  1597 raft_consensus.cc:3060] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 8 FOLLOWER]: Advancing to term 10
I20260709 13:01:18.983918  6699 consensus_queue.cc:1048] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.055s
I20260709 13:01:19.059756  1596 raft_consensus.cc:2368] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 10. Current term is 11.
I20260709 13:01:19.014256  6688 consensus_queue.cc:1059] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 14.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.033s
I20260709 13:01:19.071830  2337 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39726 (request call id 1182) took 1239 ms. Trace:
I20260709 13:01:19.037951  6332 consensus_queue.cc:1048] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.027s
I20260709 13:01:19.074064  6160 leader_election.cc:290] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 13 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:19.074994  1750 raft_consensus.cc:2393] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 13: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:01:19.091728  1602 raft_consensus.cc:2393] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 9: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:01:19.065117  6538 consensus_queue.cc:1243] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer 816fadcea8fc4893882f5f9b9f22a03d log is divergent from this leader: its last log entry 15.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20260709 13:01:19.097018  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:19.112200  1596 raft_consensus.cc:2468] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 10.
I20260709 13:01:19.112991  1644 leader_election.cc:304] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:01:19.113545  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:19.114077  1597 raft_consensus.cc:2468] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 10.
I20260709 13:01:19.099910  1646 leader_election.cc:304] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:19.096521  2337 rpcz_store.cc:276] 0709 13:01:17.832534 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:18.528022 (+695488us) service_pool.cc:224] Handling call
0709 13:01:18.551126 (+ 23104us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:18.551621 (+   495us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:18.551663 (+    42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:01:18.564761 (+ 13098us) log.cc:844] Serialized 26 byte log entry
0709 13:01:18.754850 (+190089us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:01:18.754915 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:18.755025 (+   110us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:19.071478 (+316453us) raft_consensus.cc:1685] finished
0709 13:01:19.071507 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:19.071751 (+   244us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":28587,"raft.queue_time_us":2034,"raft.run_cpu_time_us":151,"raft.run_wall_time_us":1894,"thread_start_us":123215,"threads_started":1,"wal-append.queue_time_us":252085}
I20260709 13:01:19.086200  6695 leader_election.cc:290] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 11 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:19.115291  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:19.116243  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:19.102880  2301 leader_election.cc:304] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:01:19.119243  1644 leader_election.cc:400] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 pre-election: Vote denied by peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625) with higher term. Message: Invalid argument: T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for earlier term 10. Current term is 11.
I20260709 13:01:19.119608  1644 leader_election.cc:403] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:01:19.120570  1646 leader_election.cc:304] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:01:19.122596  2301 leader_election.cc:304] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:19.060992  6754 raft_consensus.cc:515] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:19.127105  2336 raft_consensus.cc:1275] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 14 index: 10. Preceding OpId from leader: term: 16 index: 11. (index mismatch)
I20260709 13:01:19.128216  6516 raft_consensus.cc:2804] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 13:01:19.128495  6592 raft_consensus.cc:2749] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
W20260709 13:01:19.129304  1717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:19.129828  6754 leader_election.cc:290] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:19.067785  6676 raft_consensus.cc:515] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:19.117322  6714 raft_consensus.cc:493] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:19.067138  6740 consensus_queue.cc:1048] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.055s
I20260709 13:01:19.120052  6688 consensus_queue.cc:1243] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer 816fadcea8fc4893882f5f9b9f22a03d log is divergent from this leader: its last log entry 15.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
I20260709 13:01:19.120648  6739 raft_consensus.cc:2749] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 13:01:19.125367  6783 raft_consensus.cc:2749] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Leader pre-election lost for term 10. Reason: Vote denied by peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625) with higher term. Message: Invalid argument: T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for earlier term 10. Current term is 11.
I20260709 13:01:19.120770  6785 raft_consensus.cc:2749] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:01:19.126477  6695 raft_consensus.cc:2804] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Leader election won for term 10
W20260709 13:01:19.159420  6771 log.cc:927] Time spent T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.440s	user 0.002s	sys 0.002s
I20260709 13:01:19.122044  6600 consensus_queue.cc:1059] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 12.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.027s
I20260709 13:01:19.125720  6512 consensus_queue.cc:1048] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.025s
W20260709 13:01:19.162144  6643 consensus_meta.cc:287] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.563s	user 0.012s	sys 0.000s
I20260709 13:01:19.126166  6742 consensus_queue.cc:1059] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 13.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.025s
I20260709 13:01:19.147989  6714 raft_consensus.cc:515] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:19.162945  6604 raft_consensus.cc:515] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:19.164333  6789 log.cc:927] Time spent T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.112s	user 0.001s	sys 0.000s
I20260709 13:01:19.199720  6685 raft_consensus.cc:3055] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:01:19.200403  6685 raft_consensus.cc:740] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:01:19.164988  6676 leader_election.cc:290] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:19.152364  2336 raft_consensus.cc:1240] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 12. Current term is 13. Ops: []
I20260709 13:01:19.143092  6345 consensus_queue.cc:1048] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.025s
I20260709 13:01:19.204177  6643 raft_consensus.cc:515] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:19.206070  2331 raft_consensus.cc:2393] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 14: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:01:19.180521  6795 raft_consensus.cc:3055] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:01:19.181670  6485 raft_consensus.cc:3055] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 13:01:19.144403  6516 raft_consensus.cc:697] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 15 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:01:19.188755  6599 raft_consensus.cc:3060] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:01:19.190235  6544 leader_election.cc:290] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 10 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:01:19.192229  6753 consensus_meta.cc:287] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.544s	user 0.008s	sys 0.005s
I20260709 13:01:19.192396  6648 leader_election.cc:290] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:19.188786  6604 leader_election.cc:290] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 16 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:19.193885  6344 leader_election.cc:290] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:19.578169  6069 consensus_queue.cc:1048] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.002s
I20260709 13:01:19.216063  6485 raft_consensus.cc:740] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:01:19.415649   431 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:01:19.584045  6485 consensus_queue.cc:260] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.10, Last appended by leader: 10, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:19.425860  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:19.435611  1597 raft_consensus.cc:2468] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 13.
I20260709 13:01:19.466146  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:19.123214  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:19.590350  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:19.590927  6485 raft_consensus.cc:3060] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:01:19.591568  1596 raft_consensus.cc:2368] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 9. Current term is 10.
I20260709 13:01:19.592375  1596 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40154 (request call id 2354) took 1247 ms. Trace:
I20260709 13:01:19.592911  1596 rpcz_store.cc:276] 0709 13:01:18.345263 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:19.122990 (+777727us) service_pool.cc:224] Handling call
0709 13:01:19.592249 (+469259us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:01:19.483635  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 347671us
I20260709 13:01:19.489585  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:19.599159  1671 raft_consensus.cc:3060] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Advancing to term 16
W20260709 13:01:19.514020  1717 leader_election.cc:336] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 1.508s (SENT)
I20260709 13:01:19.600562  1717 leader_election.cc:304] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 13:01:19.515930  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:19.197595  2328 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39726 (request call id 1184) took 1284 ms. Trace:
W20260709 13:01:19.603020  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:19.603076  2328 rpcz_store.cc:276] 0709 13:01:17.913560 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:18.543568 (+630008us) service_pool.cc:224] Handling call
0709 13:01:18.543947 (+   379us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:18.756782 (+212835us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:19.197187 (+440405us) spinlock_profiling.cc:243] Waited 167 ms on lock 0x7b540077d654. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:01:19.197479 (+   292us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":383091968}
I20260709 13:01:19.604285  6544 raft_consensus.cc:493] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:19.317865  6516 consensus_queue.cc:237] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.11, Last appended by leader: 11, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:19.604964  6171 raft_consensus.cc:493] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:19.566017  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:19.605475  6544 raft_consensus.cc:515] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:19.606547  2337 raft_consensus.cc:2393] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 16: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
W20260709 13:01:19.607052  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:19.514091  1719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:19.610697  6160 raft_consensus.cc:2749] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:01:19.611893  6160 raft_consensus.cc:493] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:19.612651  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:19.612843  6160 raft_consensus.cc:515] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:19.613735  1754 raft_consensus.cc:2468] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 13.
I20260709 13:01:19.615557  1747 raft_consensus.cc:1275] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 10 index: 7. Preceding OpId from leader: term: 15 index: 10. (index mismatch)
I20260709 13:01:19.606047  6171 raft_consensus.cc:515] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:19.617447  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:19.618422  1745 raft_consensus.cc:2468] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 13.
I20260709 13:01:19.572153  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:19.641582  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:19.644428  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:01:19.646687  1644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:19.652060  1745 raft_consensus.cc:1275] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20260709 13:01:19.652242  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:19.653838  1754 raft_consensus.cc:2468] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 11.
I20260709 13:01:19.201948  6685 consensus_queue.cc:260] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:19.173622  6695 raft_consensus.cc:697] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:01:19.345844  6714 leader_election.cc:290] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 13:01:19.287823  6743 consensus_meta.cc:287] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.546s	user 0.010s	sys 0.006s
I20260709 13:01:19.258013  6795 raft_consensus.cc:740] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:01:19.765283  6795 consensus_queue.cc:260] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:19.682230  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:19.767110  1596 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40166 (request call id 2439) took 1008 ms. Trace:
I20260709 13:01:19.767529  1596 rpcz_store.cc:276] 0709 13:01:18.758823 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:19.646216 (+887393us) service_pool.cc:224] Handling call
0709 13:01:19.766988 (+120772us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:01:19.768891  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:19.769763  1596 raft_consensus.cc:3060] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:01:19.703091  2330 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39738 (request call id 957) took 1568 ms. Trace:
I20260709 13:01:19.705535  6592 consensus_queue.cc:1059] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 14.11, Next index: 12, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 13:01:19.715221  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:19.716305  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:19.718989  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:19.720104  6171 leader_election.cc:290] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:19.723982  6638 raft_consensus.cc:493] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:19.726025  6160 leader_election.cc:290] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:19.732604  1747 raft_consensus.cc:1275] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 15 index: 8. (index mismatch)
W20260709 13:01:19.707481  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 193782us
I20260709 13:01:19.729228  1754 raft_consensus.cc:1275] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 15 index: 9. (index mismatch)
I20260709 13:01:19.772177  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:19.805698  6544 leader_election.cc:290] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:19.816702  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:19.818449  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:19.820380  1671 raft_consensus.cc:2468] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 16.
I20260709 13:01:19.822510  2333 raft_consensus.cc:2468] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 10.
W20260709 13:01:19.826372  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 67330.1us
I20260709 13:01:19.830835  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:19.831815  2335 raft_consensus.cc:3060] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Advancing to term 13
W20260709 13:01:19.835578  1644 leader_election.cc:336] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 election: RPC error from VoteRequest() call to peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625): Timed out: RequestConsensusVote RPC to 127.0.107.196:46625 timed out after 1.684s (SENT)
I20260709 13:01:19.835880  1598 raft_consensus.cc:2468] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 10.
I20260709 13:01:19.837911  1718 leader_election.cc:304] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
W20260709 13:01:19.841400  6777 log.cc:927] Time spent T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.135s	user 0.000s	sys 0.002s
I20260709 13:01:19.844123  6695 consensus_queue.cc:237] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:19.844874  1753 raft_consensus.cc:1275] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 14 index: 10. Preceding OpId from leader: term: 16 index: 11. (index mismatch)
I20260709 13:01:19.845992  6171 raft_consensus.cc:493] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:19.846228  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:01:19.847047  2298 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:19.847170  1753 raft_consensus.cc:2468] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 13.
I20260709 13:01:19.846889  6171 raft_consensus.cc:515] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:19.848168  2298 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:19.848775  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:19.859323  1599 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40154 (request call id 2359) took 1433 ms. Trace:
I20260709 13:01:19.859975  1599 rpcz_store.cc:276] 0709 13:01:18.425956 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:19.848073 (+1422117us) service_pool.cc:224] Handling call
0709 13:01:19.859208 (+ 11135us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:01:19.862332  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:19.849833  6516 raft_consensus.cc:493] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:19.850375  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:01:19.851382  1749 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55848 (request call id 2574) took 1455 ms (client timeout 1888 ms). Trace:
I20260709 13:01:19.852437  6785 raft_consensus.cc:3055] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:01:19.854091  1605 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 1152) took 1590 ms. Trace:
W20260709 13:01:19.854637  1600 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40154 (request call id 2360) took 1423 ms (client timeout 1519 ms). Trace:
I20260709 13:01:19.855268  2331 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39738 (request call id 958) took 1733 ms. Trace:
I20260709 13:01:19.856113  6643 leader_election.cc:290] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 14 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:01:19.849413  2298 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:19.863653  6160 raft_consensus.cc:2804] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:01:19.865470  6512 raft_consensus.cc:3055] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 13:01:19.822091  1752 raft_consensus.cc:1240] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 14. Current term is 15. Ops: []
I20260709 13:01:19.770638  2330 rpcz_store.cc:276] 0709 13:01:18.121056 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:18.544169 (+423113us) service_pool.cc:224] Handling call
0709 13:01:18.574155 (+ 29986us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:18.574662 (+   507us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 13:01:18.672297 (+ 97635us) log.cc:844] Serialized 14 byte log entry
0709 13:01:19.081149 (+408852us) spinlock_profiling.cc:243] Waited 5.26 ms on lock 0x7b18004aaca0. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f4263075557 00007f42630777c1 00007f4263077779 00007f4263077709 00007f42630776d1 00007f42630769cd 000000000038f354 000000000038f199 00007f426305883a
0709 13:01:19.569881 (+488732us) spinlock_profiling.cc:243] Waited 8.73 ms on lock 0x7b18004aaca0. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f4265d0a4bc 00007f426305848b 00007f426beb2ab9 00007f426307469b 00007f426307293f 00007f4263074c02 00007f4265cfc20b 00007f4265d01d23 00007f4265d0223a
0709 13:01:19.602208 (+ 32327us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:01:19.602342 (+   134us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:01:19.602532 (+   190us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:19.635307 (+ 32775us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:19.689468 (+ 54161us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":29531,"prepare.queue_time_us":126522,"prepare.run_cpu_time_us":62,"prepare.run_wall_time_us":23288,"raft.queue_time_us":128166,"raft.run_cpu_time_us":322,"raft.run_wall_time_us":323,"spinlock_wait_cycles":32168064,"thread_start_us":215943,"threads_started":2,"wal-append.queue_time_us":247849}
I20260709 13:01:19.887562  2331 rpcz_store.cc:276] 0709 13:01:18.121421 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:18.220982 (+ 99561us) service_pool.cc:224] Handling call
0709 13:01:19.854761 (+1633779us) spinlock_profiling.cc:243] Waited 965 ms on lock 0x7b54005843d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:01:19.855040 (+   279us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2220321152}
I20260709 13:01:19.869786  6729 raft_consensus.cc:3055] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:01:19.874652  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:19.875392  6544 raft_consensus.cc:493] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:19.844344  6638 raft_consensus.cc:515] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:19.890708  2335 raft_consensus.cc:2468] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 13.
W20260709 13:01:19.884352  1749 rpcz_store.cc:269] 0709 13:01:18.395349 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:18.396371 (+  1022us) service_pool.cc:224] Handling call
0709 13:01:19.850919 (+1454548us) spinlock_profiling.cc:243] Waited 148 ms on lock 0x7b54000bd954. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:01:19.851162 (+   243us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":341183232}
I20260709 13:01:19.874758  6171 leader_election.cc:290] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:19.908970  1748 raft_consensus.cc:1275] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 19 index: 13. (index mismatch)
I20260709 13:01:19.909149  6516 raft_consensus.cc:515] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:19.912811  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:01:19.886941  1600 rpcz_store.cc:269] 0709 13:01:18.431521 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:18.583218 (+151697us) service_pool.cc:224] Handling call
0709 13:01:19.854187 (+1270969us) spinlock_profiling.cc:243] Waited 37.7 ms on lock 0x7b54000ac8d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:01:19.854385 (+   198us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":86769024}
I20260709 13:01:19.922175  1754 raft_consensus.cc:2468] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 13.
I20260709 13:01:19.885721  6785 raft_consensus.cc:740] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:01:19.926211  6785 consensus_queue.cc:260] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:19.927120  6046 consensus_meta.cc:287] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 1.013s	user 0.008s	sys 0.022s
I20260709 13:01:19.927665  6046 raft_consensus.cc:515] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:19.928089  6785 raft_consensus.cc:3060] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:01:19.902130  6160 raft_consensus.cc:493] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:19.930119  6046 leader_election.cc:290] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 12 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:19.942518  6513 raft_consensus.cc:3055] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:01:19.943449  1604 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 1140) took 2518 ms. Trace:
I20260709 13:01:19.902395  6512 raft_consensus.cc:740] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:01:19.905357  6729 raft_consensus.cc:740] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:01:19.954792  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:19.944557  6160 raft_consensus.cc:3060] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:01:19.959771  1599 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40166 (request call id 2445) took 1197 ms. Trace:
I20260709 13:01:19.963112  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:19.956821  6544 raft_consensus.cc:515] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:20.030290  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:20.039472  1749 raft_consensus.cc:3060] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:01:19.965693  6516 leader_election.cc:290] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 13:01:20.040891  1570 service_pool.cc:136] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:36462 dropped due to backpressure: service queue is full with 50 items
I20260709 13:01:20.041124  6743 raft_consensus.cc:493] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
W20260709 13:01:19.966199  1643 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 1.717s (SENT) [suppressed 46 similar messages]
I20260709 13:01:19.968575  6638 leader_election.cc:290] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:20.037472  1600 raft_consensus.cc:2393] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 10: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:01:19.972152  1598 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40154 (request call id 2378) took 1383 ms. Trace:
I20260709 13:01:20.036651  1599 rpcz_store.cc:276] 0709 13:01:18.760935 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:19.952064 (+1191129us) service_pool.cc:224] Handling call
0709 13:01:19.958099 (+  6035us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:01:19.947600  1604 rpcz_store.cc:276] 0709 13:01:17.425211 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:17.425356 (+   145us) service_pool.cc:224] Handling call
0709 13:01:17.425798 (+   442us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:18.683755 (+1257957us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:19.942714 (+1258959us) spinlock_profiling.cc:243] Waited 1.24 s on lock 0x7b54000f0d54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:01:19.943161 (+   447us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2854967168}
I20260709 13:01:19.951236  6513 raft_consensus.cc:740] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:01:19.974666  1747 raft_consensus.cc:3060] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:01:19.978631  1597 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40166 (request call id 2451) took 1103 ms. Trace:
I20260709 13:01:19.978801  2299 leader_election.cc:304] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:01:19.980181  1754 raft_consensus.cc:1275] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 12 index: 10. (index mismatch)
W20260709 13:01:19.986608  6786 log.cc:927] Time spent T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.904s	user 0.025s	sys 0.032s
I20260709 13:01:19.986965  2334 raft_consensus.cc:2468] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 10.
I20260709 13:01:19.886545  1605 rpcz_store.cc:276] 0709 13:01:18.263771 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:18.263946 (+   175us) service_pool.cc:224] Handling call
0709 13:01:18.323326 (+ 59380us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:18.419304 (+ 95978us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:18.419345 (+    41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:01:18.425198 (+  5853us) log.cc:844] Serialized 26 byte log entry
0709 13:01:18.670591 (+245393us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:01:18.670652 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:19.853267 (+1182615us) spinlock_profiling.cc:243] Waited 95.5 ms on lock 0x7b540027a1d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:01:19.853344 (+    77us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:19.853462 (+   118us) raft_consensus.cc:1685] finished
0709 13:01:19.853476 (+    14us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:19.853845 (+   369us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":278,"raft.run_cpu_time_us":107,"raft.run_wall_time_us":109,"spinlock_wait_cycles":219564800,"thread_start_us":245050,"threads_started":1,"wal-append.queue_time_us":278602,"wal-append.run_cpu_time_us":2441,"wal-append.run_wall_time_us":1002396}
I20260709 13:01:20.006819  2328 raft_consensus.cc:2468] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 10.
I20260709 13:01:20.049845  1596 raft_consensus.cc:2468] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 12.
I20260709 13:01:20.050696  1596 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40166 (request call id 2446) took 1289 ms. Trace:
I20260709 13:01:20.051604  1596 rpcz_store.cc:276] 0709 13:01:18.761282 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:19.768348 (+1007066us) service_pool.cc:224] Handling call
0709 13:01:20.050609 (+282261us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:01:20.043080  1598 rpcz_store.cc:276] 0709 13:01:18.589178 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:18.874338 (+285160us) service_pool.cc:224] Handling call
0709 13:01:19.971903 (+1097565us) spinlock_profiling.cc:243] Waited 864 ms on lock 0x7b54000f19d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:01:19.972080 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1986517888}
I20260709 13:01:19.998425  1751 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43156 (request call id 975) took 1598 ms. Trace:
I20260709 13:01:20.036440  6680 consensus_queue.cc:1059] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 12.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.006s
I20260709 13:01:20.055365  1600 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40166 (request call id 2444) took 1294 ms. Trace:
I20260709 13:01:20.055820  1600 rpcz_store.cc:276] 0709 13:01:18.760558 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:19.956643 (+1196085us) service_pool.cc:224] Handling call
0709 13:01:20.055285 (+ 98642us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:01:20.044350  1597 rpcz_store.cc:276] 0709 13:01:18.875076 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:19.115033 (+239957us) service_pool.cc:224] Handling call
0709 13:01:19.978313 (+863280us) spinlock_profiling.cc:243] Waited 319 ms on lock 0x7b54002170d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:01:19.978520 (+   207us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":733176704}
I20260709 13:01:20.056031  6680 raft_consensus.cc:3055] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:01:20.064491  6680 raft_consensus.cc:740] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:01:20.066524  6516 raft_consensus.cc:493] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:20.054173  1751 rpcz_store.cc:276] 0709 13:01:18.400145 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:18.693054 (+292909us) service_pool.cc:224] Handling call
0709 13:01:18.819694 (+126640us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:18.820688 (+   994us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:19.997890 (+1177202us) spinlock_profiling.cc:243] Waited 126 ms on lock 0x7b540019bed0. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661618ad 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:01:19.998163 (+   273us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":290126592}
W20260709 13:01:20.076045  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 292902us
W20260709 13:01:20.076671  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 98088.9us
W20260709 13:01:20.077965  1644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:20.078760  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:20.041894  6743 raft_consensus.cc:515] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:20.082461  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 225173us
W20260709 13:01:20.084017  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 136939us
W20260709 13:01:20.085886  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:20.074024  6680 consensus_queue.cc:260] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:20.069200  6513 consensus_queue.cc:260] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:20.095346  2298 leader_election.cc:304] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:01:20.067333  6516 raft_consensus.cc:515] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:20.098402  6544 leader_election.cc:290] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:20.023361  6512 consensus_queue.cc:260] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.10, Last appended by leader: 10, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:20.121647  1646 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:20.121536  6408 raft_consensus.cc:493] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
W20260709 13:01:20.122083  1645 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:20.108013  6680 raft_consensus.cc:3060] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:01:20.028421  6729 consensus_queue.cc:260] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:01:20.134042  1717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:20.136485  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:20.133942  1754 raft_consensus.cc:1240] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 13. Current term is 14. Ops: []
I20260709 13:01:20.135944  6408 raft_consensus.cc:515] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:20.141026  1646 leader_election.cc:304] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 13:01:20.142323  6332 raft_consensus.cc:2804] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:01:20.142824  6332 raft_consensus.cc:493] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:20.143227  1646 leader_election.cc:304] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:20.143415  1718 leader_election.cc:304] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:20.144672  1718 leader_election.cc:304] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:20.144904  1646 leader_election.cc:304] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:20.145375  6528 raft_consensus.cc:2804] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Leader pre-election won for term 14
W20260709 13:01:20.145742  1717 leader_election.cc:336] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 1.519s (SENT)
I20260709 13:01:20.145877  6528 raft_consensus.cc:493] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:20.143124  6408 leader_election.cc:290] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:01:20.151173  2299 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:20.146399  1717 leader_election.cc:304] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 13:01:20.149602  1719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:20.146682  6729 raft_consensus.cc:3060] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:01:20.146404  6528 raft_consensus.cc:3060] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:01:20.143229  6332 raft_consensus.cc:3060] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:01:20.144941  6344 consensus_queue.cc:1048] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 7, Time since last communication: 0.020s
I20260709 13:01:20.164455  6544 raft_consensus.cc:2804] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 13:01:20.164294  6438 raft_consensus.cc:2804] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 13:01:20.149415  6513 raft_consensus.cc:3060] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:01:20.169297  6485 raft_consensus.cc:2804] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 13:01:20.170161  1676 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52766 (request call id 2724) took 1229 ms. Trace:
I20260709 13:01:20.169938  6345 consensus_queue.cc:1048] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 13:01:20.173578  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:20.180692  6599 raft_consensus.cc:2749] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 13:01:20.165401  6512 raft_consensus.cc:3060] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:01:20.165318  6171 raft_consensus.cc:2804] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:01:20.194355  6438 raft_consensus.cc:697] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 16 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:01:20.197506  6438 consensus_queue.cc:237] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.12, Last appended by leader: 12, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:20.201287  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:20.202374  1602 raft_consensus.cc:2410] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for term 15 because replica has last-logged OpId of term: 14 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 11.
I20260709 13:01:20.149871  6554 consensus_queue.cc:1048] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.039s
I20260709 13:01:20.168248  6046 consensus_queue.cc:1048] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 6, Time since last communication: 0.041s
W20260709 13:01:20.210381  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:20.218094  1602 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40166 (request call id 2463) took 1105 ms. Trace:
I20260709 13:01:20.218508  1602 rpcz_store.cc:276] 0709 13:01:19.112379 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:19.592005 (+479626us) service_pool.cc:224] Handling call
0709 13:01:20.201007 (+609002us) spinlock_profiling.cc:243] Waited 1.38 ms on lock 0x7b5800003698. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f426303de9c 00007f426303da5b 00007f426bf56c09 00007f426b0b0ee0 00007f426b17a0ab 00007f426b17a131 00007f426b1280a0 00007f426b127984 00007f42646a26d4 00007f42646a2662
0709 13:01:20.218020 (+ 17013us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3179392}
I20260709 13:01:20.183907  1676 rpcz_store.cc:276] 0709 13:01:18.940954 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:18.941871 (+   917us) service_pool.cc:224] Handling call
0709 13:01:18.957113 (+ 15242us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:18.957684 (+   571us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:18.957719 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:01:18.958388 (+   669us) log.cc:844] Serialized 26 byte log entry
0709 13:01:19.052586 (+ 94198us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:01:19.052646 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:19.052736 (+    90us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:19.857937 (+805201us) spinlock_profiling.cc:243] Waited 8.15 ms on lock 0x7b4400022b4c. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f4262f9bc85 00007f4262f9c6a2 00007f4265d0e1db 00007f4265d09aaf 00007f4265cfd86e 00007f4265cfcbb6 00007f4265cfc7d8 00007f4265d06221 00007f4265d061d9 00007f4265d06169
0709 13:01:20.169532 (+311595us) raft_consensus.cc:1685] finished
0709 13:01:20.169562 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:20.169924 (+   362us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":22848,"raft.queue_time_us":3726,"raft.run_cpu_time_us":219,"raft.run_wall_time_us":34494,"spinlock_wait_cycles":18738432,"thread_start_us":69440,"threads_started":1,"wal-append.queue_time_us":73423}
I20260709 13:01:20.203747  6171 raft_consensus.cc:493] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:20.041317  1570 service_pool.cc:137] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:36462 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40166 (request call id 2465)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40154 (request call id 2366)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40166 (request call id 2456)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40154 (request call id 2382)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40166 (request call id 2457)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40166 (request call id 2449)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40154 (request call id 2358)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40166 (request call id 2459)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40166 (request call id 2466)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40166 (request call id 2467)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40166 (request call id 2464)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40166 (request call id 2482)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40166 (request call id 2483)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40154 (request call id 2367)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40154 (request call id 2362)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40166 (request call id 2468)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40166 (request call id 2469)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40166 (request call id 2473)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40166 (request call id 2481)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40154 (request call id 2379)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40166 (request call id 2472)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40166 (request call id 2474)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40166 (request call id 2434)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40166 (request call id 2475)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40166 (request call id 2476)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40166 (request call id 2479)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40166 (request call id 2462)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40166 (request call id 2453)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36462 (request call id 1155)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40154 (request call id 2349)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40154 (request call id 2350)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40154 (request call id 2356)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40154 (request call id 2357)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 1153)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 1154)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40154 (request call id 2361)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40154 (request call id 2365)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 2428)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 2430)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 2433)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 2438)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 2447)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 2450)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 2454)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 2470)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 2471)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 2477)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 2478)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 2480)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 1157)
I20260709 13:01:20.194984  6544 raft_consensus.cc:697] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 13 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
W20260709 13:01:20.228602  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 152093us
I20260709 13:01:20.208878  6485 raft_consensus.cc:697] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 10 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:01:20.235193  6516 leader_election.cc:290] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:20.227658  6171 raft_consensus.cc:3060] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:01:20.240470  6685 raft_consensus.cc:3060] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:01:20.243592  6743 leader_election.cc:290] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:20.227264  6676 raft_consensus.cc:2749] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:01:20.245117  6676 raft_consensus.cc:493] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:20.245770  6676 raft_consensus.cc:515] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:20.246775  2301 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:20.247241  6807 log.cc:927] Time spent T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.088s	user 0.001s	sys 0.000s
I20260709 13:01:20.247627  6592 consensus_queue.cc:1048] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 13:01:20.182286  6643 consensus_queue.cc:1048] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.040s
I20260709 13:01:20.230299  1675 raft_consensus.cc:2468] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 11.
I20260709 13:01:20.255213  6171 raft_consensus.cc:515] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:20.257951  6171 leader_election.cc:290] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 12 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:20.249963  6706 consensus_queue.cc:1048] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 13:01:20.221902  6046 consensus_queue.cc:1243] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer af520b67945d443b901bbeca37cc5701 log is divergent from this leader: its last log entry 11.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260709 13:01:20.243963  1570 service_pool.cc:137] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:36462 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40166 (request call id 2466)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40166 (request call id 2467)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40166 (request call id 2464)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40166 (request call id 2482)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40166 (request call id 2483)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40154 (request call id 2367)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40154 (request call id 2362)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40166 (request call id 2468)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40166 (request call id 2469)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40166 (request call id 2473)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40166 (request call id 2481)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40154 (request call id 2379)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40166 (request call id 2472)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40166 (request call id 2484)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40166 (request call id 2474)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40166 (request call id 2434)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40166 (request call id 2475)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40166 (request call id 2476)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40166 (request call id 2479)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36462 (request call id 1164)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40166 (request call id 2462)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40166 (request call id 2453)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36462 (request call id 1155)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36462 (request call id 1161)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40154 (request call id 2349)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40154 (request call id 2350)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40154 (request call id 2356)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40154 (request call id 2357)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 1153)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 1154)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40154 (request call id 2361)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40154 (request call id 2365)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 2428)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 2430)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 2433)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 2438)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 2447)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 2450)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 2454)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 2470)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 2471)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 2477)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 2478)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 2480)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 1157)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 1159)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 1160)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 1162)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 1163)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 1165)
I20260709 13:01:20.273870  1646 leader_election.cc:304] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
W20260709 13:01:20.280378  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 56629us
I20260709 13:01:20.291756  6544 consensus_queue.cc:237] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:20.294691  6069 raft_consensus.cc:2804] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:01:20.295233  6069 raft_consensus.cc:493] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:20.295600  6069 raft_consensus.cc:3060] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:01:20.301319  1646 leader_election.cc:304] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:20.322515  6485 consensus_queue.cc:237] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:20.325340  2299 leader_election.cc:304] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:01:20.326915  6643 raft_consensus.cc:2804] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:01:20.327344  6643 raft_consensus.cc:493] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:20.327726  6643 raft_consensus.cc:3060] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Advancing to term 12
W20260709 13:01:20.333510  1601 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40154 (request call id 2345) took 1990 ms (client timeout 1780 ms). Trace:
I20260709 13:01:20.345273  2332 raft_consensus.cc:1275] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 10 index: 8. Preceding OpId from leader: term: 12 index: 9. (index mismatch)
I20260709 13:01:20.349624  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
W20260709 13:01:20.355201  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 209541us
W20260709 13:01:20.356187  1717 leader_election.cc:336] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 1.780s (SENT)
W20260709 13:01:20.358175  2298 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:20.358706  2298 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:20.358606  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:20.359760  1679 raft_consensus.cc:2393] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 10: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
W20260709 13:01:20.357379  1717 leader_election.cc:336] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 14 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 1.855s (SENT)
I20260709 13:01:20.360558  1717 leader_election.cc:304] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:20.361140  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:01:20.362377  6816 log.cc:927] Time spent T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.064s	user 0.001s	sys 0.000s
I20260709 13:01:20.362908  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:20.363487  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:20.363802  1672 raft_consensus.cc:2393] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 16: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:01:20.366583  6714 raft_consensus.cc:2749] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:01:20.366757  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:01:20.333994  1601 rpcz_store.cc:269] 0709 13:01:18.342734 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:18.342915 (+   181us) service_pool.cc:224] Handling call
0709 13:01:20.333063 (+1990148us) spinlock_profiling.cc:243] Waited 1.02 s on lock 0x7b54000b1154. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:01:20.333253 (+   190us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2355464448}
W20260709 13:01:20.387277  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 103256us
I20260709 13:01:20.387962  1747 raft_consensus.cc:2468] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 11.
I20260709 13:01:20.327739  6676 leader_election.cc:290] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:20.428722  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:20.427435  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:20.442207  1749 raft_consensus.cc:2468] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 14.
I20260709 13:01:20.444532  1570 service_pool.cc:137] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:36462 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40166 (request call id 2467)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40166 (request call id 2464)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40166 (request call id 2482)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40166 (request call id 2483)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40154 (request call id 2367)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40154 (request call id 2362)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40166 (request call id 2468)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40166 (request call id 2469)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40166 (request call id 2473)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40166 (request call id 2481)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40154 (request call id 2379)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36462 (request call id 1167)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40166 (request call id 2472)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40166 (request call id 2484)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40166 (request call id 2474)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40166 (request call id 2434)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40166 (request call id 2475)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40166 (request call id 2476)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40166 (request call id 2479)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36462 (request call id 1164)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40166 (request call id 2462)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40166 (request call id 2453)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36462 (request call id 1155)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36462 (request call id 1161)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40154 (request call id 2349)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40154 (request call id 2350)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40154 (request call id 2356)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40154 (request call id 2357)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 1153)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 1154)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40154 (request call id 2361)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40154 (request call id 2365)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 2428)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 2430)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 2433)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 2438)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 2447)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 2450)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 2454)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 2470)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 2471)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 2477)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 2478)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 2480)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 1157)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 1159)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 1160)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 1162)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 1163)
W20260709 13:01:20.453181  6785 consensus_meta.cc:287] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.525s	user 0.016s	sys 0.000s
I20260709 13:01:20.467082  1747 raft_consensus.cc:2393] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 13: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
W20260709 13:01:20.467458  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:20.444478  6345 raft_consensus.cc:2804] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:01:20.399101  6528 raft_consensus.cc:515] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:20.469403  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:20.451155  6438 raft_consensus.cc:493] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:20.423508  6680 consensus_queue.cc:1048] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.041s
I20260709 13:01:20.470259  6345 raft_consensus.cc:493] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:20.472364  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:20.461812  6742 raft_consensus.cc:493] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:20.489754  6742 raft_consensus.cc:515] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:20.495098  6742 leader_election.cc:290] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 13:01:20.498814  1717 leader_election.cc:336] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 14 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 1.739s (SENT)
I20260709 13:01:20.500898  1753 raft_consensus.cc:1275] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 10 index: 8. Preceding OpId from leader: term: 12 index: 9. (index mismatch)
I20260709 13:01:20.481621  6345 raft_consensus.cc:3060] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:01:20.507408  1717 leader_election.cc:304] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:20.509737  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:20.510578  1747 raft_consensus.cc:3060] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Advancing to term 12
W20260709 13:01:20.512202  2298 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:20.512352  1746 raft_consensus.cc:3060] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Advancing to term 12
W20260709 13:01:20.517815  1643 leader_election.cc:336] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 11 election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 1.556s (SENT)
I20260709 13:01:20.511907  1718 leader_election.cc:304] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:20.530948  2301 leader_election.cc:304] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:20.511384  1601 raft_consensus.cc:2376] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 11. Re-sending same reply.
I20260709 13:01:20.488911  6438 raft_consensus.cc:515] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:20.550678  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:20.554932  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:20.559333  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:01:20.560907  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 61032.2us
I20260709 13:01:20.514088  6332 raft_consensus.cc:515] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:20.582777  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:20.583925  1745 pending_rounds.cc:85] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701: Aborting all ops after (but not including) 6
I20260709 13:01:20.584316  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:20.556738  6676 raft_consensus.cc:2749] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
W20260709 13:01:20.576050  6160 consensus_meta.cc:287] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.552s	user 0.014s	sys 0.002s
W20260709 13:01:20.587263  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 79489.7us
I20260709 13:01:20.583133  1570 service_pool.cc:137] RequestConsensusVote request on kudu.consensus.ConsensusService from 127.0.0.1:36462 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40166 (request call id 2482)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40166 (request call id 2483)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40154 (request call id 2367)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40154 (request call id 2362)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40166 (request call id 2468)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40166 (request call id 2469)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40166 (request call id 2473)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40166 (request call id 2481)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40154 (request call id 2379)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36462 (request call id 1167)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40166 (request call id 2472)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40166 (request call id 2484)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40166 (request call id 2474)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40166 (request call id 2434)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40166 (request call id 2475)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40166 (request call id 2476)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40166 (request call id 2479)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36462 (request call id 1164)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40166 (request call id 2462)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40166 (request call id 2453)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36462 (request call id 1169)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36462 (request call id 1155)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36462 (request call id 1161)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36462 (request call id 1171)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40154 (request call id 2349)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40154 (request call id 2350)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40154 (request call id 2356)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40154 (request call id 2357)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 1153)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 1154)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40154 (request call id 2361)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40154 (request call id 2365)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 2428)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 2430)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 2433)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 2438)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 2447)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 2450)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 2454)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 2470)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 2471)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 2477)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 2478)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 2480)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 1157)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 1159)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 1160)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 1162)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 1163)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 1168)
I20260709 13:01:20.584333  1745 pending_rounds.cc:107] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701: Aborting uncommitted NO_OP operation due to leader change: 11.7
I20260709 13:01:20.587858  1601 raft_consensus.cc:2468] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 14.
I20260709 13:01:20.588392  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:20.593190  1601 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40166 (request call id 2464) took 1159 ms. Trace:
I20260709 13:01:20.593727  1601 rpcz_store.cc:276] 0709 13:01:19.434876 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:20.540607 (+1105731us) service_pool.cc:224] Handling call
0709 13:01:20.593096 (+ 52489us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:01:20.594765  1745 raft_consensus.cc:2889] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:01:20.579305  6485 raft_consensus.cc:2749] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
W20260709 13:01:20.596743  1643 leader_election.cc:336] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 15 election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 1.734s (SENT)
W20260709 13:01:20.606292  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 118515us
I20260709 13:01:20.607057  1643 leader_election.cc:304] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 13:01:20.619071  2301 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:20.619933  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:20.624392  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:20.626339  1718 leader_election.cc:304] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:20.595124  6160 raft_consensus.cc:515] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:20.627758  1750 raft_consensus.cc:2376] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 12. Re-sending same reply.
W20260709 13:01:20.629498  1717 leader_election.cc:336] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 14 election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 1.770s (SENT)
I20260709 13:01:20.636463  1717 leader_election.cc:304] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:20.627926  6592 raft_consensus.cc:2749] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 13:01:20.642661  6726 raft_consensus.cc:2804] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:01:20.643390  6726 raft_consensus.cc:697] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 14 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:01:20.644582  6726 consensus_queue.cc:237] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.10, Last appended by leader: 10, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:20.646792  6752 raft_consensus.cc:493] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:20.647424  6752 raft_consensus.cc:515] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:20.649991  6752 leader_election.cc:290] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:20.651818  6332 leader_election.cc:290] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 14 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:20.652005  6160 leader_election.cc:290] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:20.640727  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:01:20.652868  6818 log.cc:927] Time spent T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.095s	user 0.001s	sys 0.000s
I20260709 13:01:20.653779  6528 leader_election.cc:290] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 14 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:20.654860  1601 raft_consensus.cc:2393] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 9: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:01:20.655005  1603 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 1136) took 3692 ms. Trace:
I20260709 13:01:20.655520  1603 rpcz_store.cc:276] 0709 13:01:16.962596 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:16.962713 (+   117us) service_pool.cc:224] Handling call
0709 13:01:16.963030 (+   317us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:18.672586 (+1709556us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:20.654403 (+1981817us) spinlock_profiling.cc:243] Waited 1.71 s on lock 0x7b54002a1cd4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:01:20.654782 (+   379us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3929212800}
I20260709 13:01:20.657097  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:20.657907  1603 raft_consensus.cc:2468] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 13.
I20260709 13:01:20.656028  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:01:20.659404  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 68303.1us
I20260709 13:01:20.667539  1601 raft_consensus.cc:2468] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 10.
W20260709 13:01:20.671789  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 45065.1us
I20260709 13:01:20.658422  1603 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40154 (request call id 2367) took 2222 ms. Trace:
I20260709 13:01:20.687084  1603 rpcz_store.cc:276] 0709 13:01:18.435971 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:20.656737 (+2220766us) service_pool.cc:224] Handling call
0709 13:01:20.658369 (+  1632us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:01:20.698932  1746 raft_consensus.cc:2468] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 12.
I20260709 13:01:20.703339  1747 raft_consensus.cc:2468] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 12.
I20260709 13:01:20.706627  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:20.672043  6676 raft_consensus.cc:2749] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 13:01:20.667096  6160 raft_consensus.cc:493] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:20.709261  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:20.658713  6600 raft_consensus.cc:493] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:01:20.713903  6815 log.cc:927] Time spent T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.171s	user 0.001s	sys 0.001s
I20260709 13:01:20.717303  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:01:20.708833  1717 leader_election.cc:336] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 1.788s (SENT)
I20260709 13:01:20.658946  6821 raft_consensus.cc:2749] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 13:01:20.710250  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:20.720610  1601 raft_consensus.cc:2393] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 12: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:01:20.721060  1601 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40154 (request call id 2362) took 2288 ms. Trace:
I20260709 13:01:20.721371  1601 rpcz_store.cc:276] 0709 13:01:18.432166 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:20.686128 (+2253962us) service_pool.cc:224] Handling call
0709 13:01:20.720995 (+ 34867us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:01:20.724323  2329 raft_consensus.cc:1240] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 12. Current term is 13. Ops: [12.11-12.11]
I20260709 13:01:20.727393  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:20.730309  6438 leader_election.cc:290] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:20.730531  6600 raft_consensus.cc:515] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:20.732553  1717 leader_election.cc:336] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 1.915s (SENT)
I20260709 13:01:20.733065  1717 leader_election.cc:304] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:20.735620  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:20.736403  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:20.737270  2331 raft_consensus.cc:2393] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 14: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:01:20.738863  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
W20260709 13:01:20.739367  1643 leader_election.cc:336] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 2.054s (SENT)
I20260709 13:01:20.742772  6821 raft_consensus.cc:2749] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:01:20.742831  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:20.743620  2331 raft_consensus.cc:3060] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:01:20.745999  2333 raft_consensus.cc:2468] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 11.
I20260709 13:01:20.748034  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:20.749401  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:20.749614  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:01:20.750726  1717 leader_election.cc:336] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 2.028s (SENT)
W20260709 13:01:20.752053  1717 leader_election.cc:336] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 2.057s (SENT)
I20260709 13:01:20.752434  1717 leader_election.cc:304] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:20.752761  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:20.754601  1679 raft_consensus.cc:2393] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 12: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:01:20.756707  1750 raft_consensus.cc:1275] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 15 index: 12. (index mismatch)
I20260709 13:01:20.756774  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:01:20.758652  1719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:20.772622  1717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:20.774322  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:20.775233  1746 raft_consensus.cc:2468] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 12.
I20260709 13:01:20.778580  1603 raft_consensus.cc:2393] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 14: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:01:20.779181  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:01:20.786046  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 175906us
I20260709 13:01:20.786902  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:20.794257  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:20.760303  6160 raft_consensus.cc:515] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:20.799966  2334 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39726 (request call id 1181) took 2992 ms (client timeout 1684 ms). Trace:
W20260709 13:01:20.800428  2334 rpcz_store.cc:269] 0709 13:01:17.807766 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:18.177155 (+369389us) service_pool.cc:224] Handling call
0709 13:01:20.799699 (+2622544us) spinlock_profiling.cc:243] Waited 1.59 s on lock 0x7b5400605ed4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:01:20.799850 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3660509440}
I20260709 13:01:20.771286  6069 raft_consensus.cc:515] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:20.808775  1671 raft_consensus.cc:2393] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 14: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:01:20.809934  6069 leader_election.cc:290] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 14 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:20.777143  6729 raft_consensus.cc:493] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:20.814823  1569 leader_election.cc:304] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:01:20.819975  6729 raft_consensus.cc:515] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:20.820652  6752 raft_consensus.cc:2749] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:01:20.821985  1569 leader_election.cc:304] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:01:20.772329  6821 raft_consensus.cc:2749] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 13:01:20.843848  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
W20260709 13:01:20.844704  2298 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:20.845669  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:20.810294  6643 raft_consensus.cc:515] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:20.825191  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:20.827549  6629 raft_consensus.cc:3055] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:01:20.851112  6629 raft_consensus.cc:740] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:01:20.852298  6629 consensus_queue.cc:260] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 14.8, Last appended by leader: 8, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:20.853380  6360 consensus_queue.cc:1059] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 10.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 13:01:20.854048  6629 raft_consensus.cc:3060] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:01:20.832051  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 15 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:20.857612  1569 leader_election.cc:304] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:20.858546  1596 raft_consensus.cc:2410] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for term 15 because replica has last-logged OpId of term: 14 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 7.
I20260709 13:01:20.835210  1718 leader_election.cc:304] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:20.836020  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 14 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:20.837096  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:20.798573  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 15 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:20.839118  6160 leader_election.cc:290] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:20.850659  1672 raft_consensus.cc:1275] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 15 index: 12. (index mismatch)
W20260709 13:01:20.865324  1596 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40154 (request call id 2358) took 2445 ms (client timeout 2793 ms). Trace:
I20260709 13:01:20.855461  6729 leader_election.cc:290] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:20.855620  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:20.857045  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 14 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:20.857122  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:20.831215  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:20.865195  1677 raft_consensus.cc:2393] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 14: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
W20260709 13:01:20.865936  1596 rpcz_store.cc:269] 0709 13:01:18.419682 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:20.831728 (+2412046us) service_pool.cc:224] Handling call
0709 13:01:20.865245 (+ 33517us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:01:20.873034  1600 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40166 (request call id 2457) took 1879 ms (client timeout 2054 ms). Trace:
I20260709 13:01:20.875300  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:01:20.875375  1600 rpcz_store.cc:269] 0709 13:01:18.993407 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:20.836784 (+1843377us) service_pool.cc:224] Handling call
0709 13:01:20.872964 (+ 36180us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:01:20.883003  1603 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40166 (request call id 2473) took 1044 ms. Trace:
I20260709 13:01:20.883581  1603 rpcz_store.cc:276] 0709 13:01:19.838395 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:20.847707 (+1009312us) service_pool.cc:224] Handling call
0709 13:01:20.882924 (+ 35217us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:01:20.888132  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:20.891542  1596 raft_consensus.cc:3060] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:01:20.889469  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:20.893460  1680 raft_consensus.cc:2393] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 10: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:01:20.890793  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:20.894425  1600 raft_consensus.cc:2468] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 13.
I20260709 13:01:20.895025  1600 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40166 (request call id 2472) took 1057 ms. Trace:
I20260709 13:01:20.895323  1600 rpcz_store.cc:276] 0709 13:01:19.838062 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:20.877506 (+1039444us) service_pool.cc:224] Handling call
0709 13:01:20.894957 (+ 17451us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:01:20.896334  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:20.897482  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:20.898217  1600 raft_consensus.cc:2468] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 11.
I20260709 13:01:20.899379  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
W20260709 13:01:20.899912  1643 leader_election.cc:336] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 1.500s (SENT)
I20260709 13:01:20.900316  1598 raft_consensus.cc:2410] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for term 14 because replica has last-logged OpId of term: 13 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 7.
I20260709 13:01:20.901873  6795 raft_consensus.cc:3060] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Advancing to term 13
W20260709 13:01:20.902635  1597 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40166 (request call id 2456) took 1909 ms (client timeout 2039 ms). Trace:
W20260709 13:01:20.906852  1597 rpcz_store.cc:269] 0709 13:01:18.993010 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:20.830891 (+1837881us) service_pool.cc:224] Handling call
0709 13:01:20.902554 (+ 71663us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:01:20.903546  1672 raft_consensus.cc:1240] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 LEADER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 9. Current term is 10. Ops: [9.7-9.7]
I20260709 13:01:20.904243  1601 raft_consensus.cc:2393] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 12: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:01:20.909242  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:20.908557  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:20.904733  1673 raft_consensus.cc:2468] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 12.
I20260709 13:01:20.910017  1601 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40154 (request call id 2379) took 2301 ms. Trace:
I20260709 13:01:20.912245  1597 raft_consensus.cc:2468] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 10.
I20260709 13:01:20.911091  1601 rpcz_store.cc:276] 0709 13:01:18.608952 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:20.856689 (+2247737us) service_pool.cc:224] Handling call
0709 13:01:20.909951 (+ 53262us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":26752}
I20260709 13:01:20.916049  6729 raft_consensus.cc:2749] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:01:20.903606  6345 raft_consensus.cc:515] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:01:20.927242  1598 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40154 (request call id 2366) took 2491 ms (client timeout 2571 ms). Trace:
W20260709 13:01:20.927685  1598 rpcz_store.cc:269] 0709 13:01:18.435682 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:20.856652 (+2420970us) service_pool.cc:224] Handling call
0709 13:01:20.927170 (+ 70518us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:01:20.928717  1602 raft_consensus.cc:2410] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for term 15 because replica has last-logged OpId of term: 13 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 7.
I20260709 13:01:20.928926  1679 raft_consensus.cc:1240] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 13. Current term is 14. Ops: []
W20260709 13:01:20.929219  1602 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40166 (request call id 2459) took 1935 ms (client timeout 2355 ms). Trace:
W20260709 13:01:20.929580  1602 rpcz_store.cc:269] 0709 13:01:18.994153 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:20.798201 (+1804048us) service_pool.cc:224] Handling call
0709 13:01:20.929156 (+130955us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:01:20.882939  6821 raft_consensus.cc:2749] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:01:20.935339  1603 raft_consensus.cc:2468] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 14.
I20260709 13:01:20.935894  1603 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40166 (request call id 2434) took 2179 ms. Trace:
I20260709 13:01:20.936041  1643 leader_election.cc:304] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:20.936229  1603 rpcz_store.cc:276] 0709 13:01:18.756930 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:20.899193 (+2142263us) service_pool.cc:224] Handling call
0709 13:01:20.935822 (+ 36629us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:01:20.891249  6743 raft_consensus.cc:3060] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:01:20.928685  6345 leader_election.cc:290] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 14 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:01:20.945681  1717 leader_election.cc:336] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 18 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 1.905s (SENT)
I20260709 13:01:20.947232  1717 leader_election.cc:304] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:20.943704  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:20.950874  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 14 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:20.952817  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:20.967062  2331 raft_consensus.cc:2468] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 11.
W20260709 13:01:20.970052  1644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:20.970062  1674 raft_consensus.cc:2393] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 14: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:01:20.978026  1596 raft_consensus.cc:2468] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 14.
I20260709 13:01:20.948840  6069 raft_consensus.cc:2749] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260709 13:01:21.073923  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 191354us
W20260709 13:01:21.074402  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 216857us
W20260709 13:01:21.074874  1643 leader_election.cc:336] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 1.510s (SENT)
W20260709 13:01:21.076201  1643 leader_election.cc:336] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 1.684s (SENT)
I20260709 13:01:21.076593  1643 leader_election.cc:304] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 13:01:21.076961  1569 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:21.077441  1569 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:21.077585  6046 raft_consensus.cc:2749] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
W20260709 13:01:21.077930  1643 leader_election.cc:336] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 11 election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 1.744s (SENT)
I20260709 13:01:21.079282  6046 raft_consensus.cc:493] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:21.080101  6046 raft_consensus.cc:515] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:21.083102  1674 raft_consensus.cc:1275] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 9 index: 10. Preceding OpId from leader: term: 11 index: 11. (index mismatch)
I20260709 13:01:21.085732  6046 leader_election.cc:290] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:21.093644  1643 leader_election.cc:304] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:21.098541  6046 raft_consensus.cc:2749] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 13:01:21.108165  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
W20260709 13:01:21.111799  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 166127us
W20260709 13:01:21.117158  1643 leader_election.cc:336] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 14 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 1.790s (SENT)
I20260709 13:01:21.120428  1600 raft_consensus.cc:3060] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:01:21.125566  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:20.976521  6160 raft_consensus.cc:2749] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:01:21.131057  1643 leader_election.cc:304] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 13:01:21.132093  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:21.140491  6069 raft_consensus.cc:2749] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:01:21.150374  1569 leader_election.cc:304] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
W20260709 13:01:21.151494  1717 leader_election.cc:336] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 2.150s (SENT)
I20260709 13:01:21.143662  6643 leader_election.cc:290] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 12 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:21.155550  1717 leader_election.cc:304] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 13:01:21.159170  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 55055.8us
I20260709 13:01:21.139470  6638 raft_consensus.cc:493] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:01:21.193491  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 44500.7us
I20260709 13:01:21.196352  6638 raft_consensus.cc:515] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:21.199452  6638 leader_election.cc:290] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:21.201229  6544 raft_consensus.cc:2749] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20260709 13:01:21.204084  1717 leader_election.cc:336] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 9 election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 2.414s (SENT)
I20260709 13:01:21.207311  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:21.208501  1596 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40166 (request call id 2462) took 2147 ms. Trace:
I20260709 13:01:21.208943  1596 rpcz_store.cc:276] 0709 13:01:19.060960 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:20.980983 (+1920023us) service_pool.cc:224] Handling call
0709 13:01:21.208411 (+227428us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:01:21.210049  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:21.211392  1596 raft_consensus.cc:2393] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 10: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:01:21.212414  1596 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40166 (request call id 2453) took 2259 ms. Trace:
I20260709 13:01:21.213280  1596 rpcz_store.cc:276] 0709 13:01:18.953491 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:21.209711 (+2256220us) service_pool.cc:224] Handling call
0709 13:01:21.212355 (+  2644us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:01:21.215481  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:21.217131  1596 raft_consensus.cc:2468] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 11.
W20260709 13:01:21.220680  1643 leader_election.cc:336] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 1.811s (SENT)
I20260709 13:01:21.223184  6729 raft_consensus.cc:2749] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
W20260709 13:01:21.223861  6847 log.cc:927] Time spent T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.063s	user 0.000s	sys 0.000s
W20260709 13:01:21.230266  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 120628us
W20260709 13:01:21.233662  1599 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40166 (request call id 2465) took 1798 ms (client timeout 1556 ms). Trace:
W20260709 13:01:21.234648  1599 rpcz_store.cc:269] 0709 13:01:19.435717 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:21.232884 (+1797167us) service_pool.cc:211] Skipping call since client already timed out
0709 13:01:21.233554 (+   670us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260709 13:01:21.234948  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 13 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:21.241123  6629 consensus_queue.cc:1059] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 7.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 13:01:21.242237  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:01:21.238296  6828 log.cc:927] Time spent T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.620s	user 0.002s	sys 0.006s
I20260709 13:01:21.240110  6821 raft_consensus.cc:493] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
W20260709 13:01:21.244431  1643 leader_election.cc:336] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 1.873s (SENT)
I20260709 13:01:21.245462  1596 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36462 (request call id 1155) took 2824 ms. Trace:
I20260709 13:01:21.245872  1596 rpcz_store.cc:276] 0709 13:01:18.421525 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:21.225117 (+2803592us) service_pool.cc:224] Handling call
0709 13:01:21.245388 (+ 20271us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:01:21.246568  6821 raft_consensus.cc:515] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:21.252516  1718 leader_election.cc:304] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:21.253014  6821 leader_election.cc:290] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:21.253839  6821 raft_consensus.cc:2804] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:01:21.254222  6821 raft_consensus.cc:493] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:21.254606  6821 raft_consensus.cc:3060] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:01:21.258673  1599 raft_consensus.cc:2468] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 14.
I20260709 13:01:21.259286  1599 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36462 (request call id 1161) took 1017 ms. Trace:
I20260709 13:01:21.259583  1599 rpcz_store.cc:276] 0709 13:01:20.241848 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:21.235376 (+993528us) service_pool.cc:224] Handling call
0709 13:01:21.259221 (+ 23845us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:01:21.261188  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:21.261930  1599 raft_consensus.cc:2468] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 11.
I20260709 13:01:21.246402  6629 consensus_queue.cc:1243] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer 816fadcea8fc4893882f5f9b9f22a03d log is divergent from this leader: its last log entry 10.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260709 13:01:21.263923  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 12 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:21.265211  2337 raft_consensus.cc:2468] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 13.
I20260709 13:01:21.276273  1600 raft_consensus.cc:2468] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 16.
I20260709 13:01:21.276475  6743 raft_consensus.cc:2749] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:01:21.276919  1600 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36462 (request call id 1164) took 1034 ms. Trace:
I20260709 13:01:21.280884  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:21.283483  1676 raft_consensus.cc:1240] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 9. Current term is 10. Ops: []
I20260709 13:01:21.277307  1600 rpcz_store.cc:276] 0709 13:01:20.242851 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:21.107868 (+865017us) service_pool.cc:224] Handling call
0709 13:01:21.276852 (+168984us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:01:21.285312  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:21.287254  1599 raft_consensus.cc:1240] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 8. Current term is 10. Ops: []
I20260709 13:01:21.293749  1603 raft_consensus.cc:2468] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 12.
I20260709 13:01:21.296298  1596 raft_consensus.cc:2410] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 for term 14 because replica has last-logged OpId of term: 12 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 10.
I20260709 13:01:21.297233  1599 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40154 (request call id 2349) took 2950 ms. Trace:
I20260709 13:01:21.297847  1599 rpcz_store.cc:276] 0709 13:01:18.343974 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:21.284420 (+2940446us) service_pool.cc:224] Handling call
0709 13:01:21.287059 (+  2639us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:21.293919 (+  6860us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:21.294134 (+   215us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:01:21.287317  6826 log.cc:927] Time spent T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.666s	user 0.005s	sys 0.005s
W20260709 13:01:21.304374  1643 leader_election.cc:336] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 15 election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 1.702s (SENT)
I20260709 13:01:21.288822  6600 leader_election.cc:290] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:21.286847  6743 consensus_queue.cc:1048] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.019s
I20260709 13:01:21.308267  6684 raft_consensus.cc:3055] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:01:21.308708  6684 raft_consensus.cc:740] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:01:21.309731  1603 raft_consensus.cc:1240] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 LEADER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 9. Current term is 10. Ops: []
I20260709 13:01:21.310515  1603 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40154 (request call id 2356) took 2963 ms. Trace:
I20260709 13:01:21.310964  1603 rpcz_store.cc:276] 0709 13:01:18.347024 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:21.306279 (+2959255us) service_pool.cc:224] Handling call
0709 13:01:21.309501 (+  3222us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:21.310190 (+   689us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:21.310415 (+   225us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:01:21.308794  1643 leader_election.cc:336] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 1.900s (SENT)
W20260709 13:01:21.312666  1604 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40166 (request call id 2449) took 2550 ms (client timeout 2440 ms). Trace:
W20260709 13:01:21.313107  1604 rpcz_store.cc:269] 0709 13:01:18.762389 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:21.310641 (+2548252us) service_pool.cc:211] Skipping call since client already timed out
0709 13:01:21.312577 (+  1936us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260709 13:01:21.316638  1604 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 1154) took 2895 ms. Trace:
I20260709 13:01:21.317305  1604 rpcz_store.cc:276] 0709 13:01:18.421232 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:21.315451 (+2894219us) service_pool.cc:224] Handling call
0709 13:01:21.315901 (+   450us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:21.316193 (+   292us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:21.316217 (+    24us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:01:21.316290 (+    73us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:01:21.316363 (+    73us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:21.316428 (+    65us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:21.316544 (+   116us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:01:21.319839  1719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:21.309700  6684 consensus_queue.cc:260] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.9, Last appended by leader: 9, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:21.331615  6684 raft_consensus.cc:3060] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:01:21.334112  6605 raft_consensus.cc:493] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:21.334764  6605 raft_consensus.cc:515] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:21.335163  1604 raft_consensus.cc:1275] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 14 index: 8. (term mismatch)
I20260709 13:01:21.335669  1604 pending_rounds.cc:85] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a: Aborting all ops after (but not including) 7
I20260709 13:01:21.337802  1604 pending_rounds.cc:107] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a: Aborting uncommitted NO_OP operation due to leader change: 13.8
I20260709 13:01:21.339197  1604 raft_consensus.cc:2889] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:01:21.347097  1604 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40154 (request call id 2361) took 2915 ms. Trace:
I20260709 13:01:21.351058  1604 rpcz_store.cc:276] 0709 13:01:18.431870 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:21.329243 (+2897373us) service_pool.cc:224] Handling call
0709 13:01:21.334762 (+  5519us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:21.346785 (+ 12023us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:21.347015 (+   230us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:01:21.347500  6821 raft_consensus.cc:515] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:21.348245  1598 raft_consensus.cc:1240] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 12. Current term is 13. Ops: [12.6-12.6]
I20260709 13:01:21.337247  1599 raft_consensus.cc:1217] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 LEADER]: Deduplicated request from leader. Original: 11.7->[12.8-12.8]   Dedup: 12.8->[]
I20260709 13:01:21.353286  1599 raft_consensus.cc:1240] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 LEADER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 12. Current term is 13. Ops: [12.8-12.8]
I20260709 13:01:21.354692  6345 raft_consensus.cc:493] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:21.355852  2337 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39726 (request call id 1196) took 1456 ms. Trace:
I20260709 13:01:21.356324  1598 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40154 (request call id 2365) took 2921 ms. Trace:
I20260709 13:01:21.356719  1598 rpcz_store.cc:276] 0709 13:01:18.435372 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:21.347470 (+2912098us) service_pool.cc:224] Handling call
0709 13:01:21.347929 (+   459us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:21.356038 (+  8109us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:21.356259 (+   221us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:01:21.355331  6345 raft_consensus.cc:515] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:21.356248  2337 rpcz_store.cc:276] 0709 13:01:19.899360 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:19.912319 (+ 12959us) service_pool.cc:224] Handling call
0709 13:01:21.355658 (+1443339us) spinlock_profiling.cc:243] Waited 1.35 s on lock 0x7b540065ed54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:01:21.355782 (+   124us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3094338816}
I20260709 13:01:21.359516  1749 raft_consensus.cc:1275] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260709 13:01:21.358366  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:21.360121  1752 raft_consensus.cc:3060] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:01:21.360782  1599 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40154 (request call id 2357) took 3010 ms. Trace:
I20260709 13:01:21.361179  1599 rpcz_store.cc:276] 0709 13:01:18.347256 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:21.313332 (+2966076us) service_pool.cc:224] Handling call
0709 13:01:21.337088 (+ 23756us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:21.357349 (+ 20261us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:21.357567 (+   218us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:01:21.361968  1603 raft_consensus.cc:1275] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 12 index: 12. Preceding OpId from leader: term: 14 index: 13. (index mismatch)
I20260709 13:01:21.365701  1603 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 1153) took 2944 ms. Trace:
I20260709 13:01:21.367682  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:21.362017  1569 leader_election.cc:304] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:21.376950  1747 raft_consensus.cc:1275] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 13 index: 12. (index mismatch)
I20260709 13:01:21.381572  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:21.383674  1746 raft_consensus.cc:3060] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:01:21.381163  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:21.386613  1751 raft_consensus.cc:3060] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:01:21.387215  1646 leader_election.cc:304] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:21.366153  1603 rpcz_store.cc:276] 0709 13:01:18.420897 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:21.313879 (+2892982us) service_pool.cc:224] Handling call
0709 13:01:21.352741 (+ 38862us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:21.362332 (+  9591us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:21.365614 (+  3282us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:01:21.391315  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 82542us
I20260709 13:01:21.367311  6592 raft_consensus.cc:493] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:21.381219  6714 raft_consensus.cc:493] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:21.409430  1604 raft_consensus.cc:1240] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 10. Current term is 11. Ops: []
I20260709 13:01:21.390005  6743 raft_consensus.cc:2749] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:01:21.413753  1599 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 2430) took 2766 ms. Trace:
I20260709 13:01:21.414788  1604 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 2428) took 2787 ms. Trace:
I20260709 13:01:21.416273  1604 rpcz_store.cc:276] 0709 13:01:18.627275 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:21.370657 (+2743382us) service_pool.cc:224] Handling call
0709 13:01:21.404780 (+ 34123us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:21.414499 (+  9719us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:21.414720 (+   221us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:01:21.415921  1599 rpcz_store.cc:276] 0709 13:01:18.647050 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:21.370602 (+2723552us) service_pool.cc:224] Handling call
0709 13:01:21.375527 (+  4925us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:21.408333 (+ 32806us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:21.408377 (+    44us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:01:21.413410 (+  5033us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:01:21.413485 (+    75us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:21.413548 (+    63us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:21.413684 (+   136us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:01:21.417609  1643 leader_election.cc:336] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 12 election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 2.065s (SENT)
I20260709 13:01:21.418465  6069 raft_consensus.cc:2804] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Leader election won for term 11
I20260709 13:01:21.423215  1643 leader_election.cc:304] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:01:21.426626  1746 raft_consensus.cc:2468] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 14.
I20260709 13:01:21.415748  6528 consensus_queue.cc:1048] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.013s
I20260709 13:01:21.433420  6605 leader_election.cc:290] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:21.437563  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:21.443243  1752 raft_consensus.cc:2468] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 14.
I20260709 13:01:21.446821  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:21.419111  6069 raft_consensus.cc:697] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:01:21.449891  1603 raft_consensus.cc:3060] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:01:21.456897  1569 leader_election.cc:304] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:01:21.414259  6592 raft_consensus.cc:515] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:21.465469  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 56135.9us
I20260709 13:01:21.427278  6714 raft_consensus.cc:515] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:21.458150  6528 raft_consensus.cc:2749] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 13:01:21.475768  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:01:21.471315  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 59045.1us
I20260709 13:01:21.479254  1599 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 2438) took 2720 ms. Trace:
I20260709 13:01:21.481034  1599 rpcz_store.cc:276] 0709 13:01:18.758485 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:21.439514 (+2681029us) service_pool.cc:224] Handling call
0709 13:01:21.468024 (+ 28510us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:21.478880 (+ 10856us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:21.478911 (+    31us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:01:21.478963 (+    52us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:01:21.479005 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:21.479068 (+    63us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:21.479189 (+   121us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:01:21.481842  1646 leader_election.cc:304] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:21.485380  6345 leader_election.cc:290] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:21.485280  6069 consensus_queue.cc:237] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 9.6, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } }
I20260709 13:01:21.492974  6729 raft_consensus.cc:3055] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:01:21.493439  6729 raft_consensus.cc:740] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:01:21.477774  6743 raft_consensus.cc:2804] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Leader election won for term 11
I20260709 13:01:21.494839  6743 raft_consensus.cc:697] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:01:21.496225  6743 consensus_queue.cc:237] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.10, Last appended by leader: 10, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:21.500248  2301 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:21.500933  2301 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:21.505383  6821 leader_election.cc:290] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 13 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:21.507308  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:21.508612  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:21.509284  1598 raft_consensus.cc:2468] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 9.
I20260709 13:01:21.510661  6345 raft_consensus.cc:2804] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:01:21.511329  6345 raft_consensus.cc:697] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:01:21.512432  6345 consensus_queue.cc:237] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 11.8, Last appended by leader: 8, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:21.516163  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:21.516835  1596 raft_consensus.cc:3060] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Advancing to term 12
W20260709 13:01:21.520255  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:21.520852  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:21.522092  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:21.524204  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:21.524940  1597 raft_consensus.cc:3060] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Advancing to term 12
W20260709 13:01:21.526974  1605 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40154 (request call id 2382) took 2903 ms (client timeout 2414 ms). Trace:
W20260709 13:01:21.527402  1605 rpcz_store.cc:269] 0709 13:01:18.623515 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:21.526716 (+2903201us) service_pool.cc:211] Skipping call since client already timed out
0709 13:01:21.526873 (+   157us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260709 13:01:21.528357  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:21.528924  1605 raft_consensus.cc:2376] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 12. Re-sending same reply.
I20260709 13:01:21.531998  6592 leader_election.cc:290] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:21.532748  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:21.533649  1677 raft_consensus.cc:3060] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:01:21.533519  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 12 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:21.532721  1674 raft_consensus.cc:1240] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 13. Current term is 14. Ops: []
I20260709 13:01:21.535490  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:21.536183  1680 raft_consensus.cc:2468] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 8.
I20260709 13:01:21.536516  6821 raft_consensus.cc:493] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:21.537034  6821 raft_consensus.cc:515] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:21.505235  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:21.543185  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:21.494478  6729 consensus_queue.cc:260] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.7, Last appended by leader: 7, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:01:21.544004  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 54261.1us
I20260709 13:01:21.529848  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:21.541177  1601 raft_consensus.cc:2376] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 10. Re-sending same reply.
I20260709 13:01:21.552320  1677 raft_consensus.cc:2468] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 12.
I20260709 13:01:21.549901  1646 leader_election.cc:304] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
W20260709 13:01:21.553920  2298 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:21.555078  1751 raft_consensus.cc:2468] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 14.
W20260709 13:01:21.555717  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:21.556879  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:21.557504  1596 raft_consensus.cc:2468] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 12.
W20260709 13:01:21.560341  2299 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:21.561021  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:21.566790  1599 raft_consensus.cc:2410] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for term 14 because replica has last-logged OpId of term: 13 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 11.
I20260709 13:01:21.568259  2298 leader_election.cc:304] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d
W20260709 13:01:21.570181  1717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:21.571255  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:21.565382  6821 leader_election.cc:290] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:21.578766  6699 raft_consensus.cc:493] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:21.579947  6600 raft_consensus.cc:493] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:21.578339  6513 raft_consensus.cc:493] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:21.580632  6600 raft_consensus.cc:515] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:21.581453  6513 raft_consensus.cc:515] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:21.584299  6513 leader_election.cc:290] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:21.584700  6714 leader_election.cc:290] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:21.580650  6605 raft_consensus.cc:493] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:21.591320  1646 leader_election.cc:304] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:21.591254  6605 raft_consensus.cc:515] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:21.591353  1644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:21.591548  6046 raft_consensus.cc:2804] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:01:21.579560  6699 raft_consensus.cc:515] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:21.600631  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:21.610057  6046 raft_consensus.cc:697] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:01:21.617017  1596 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 2454) took 2625 ms. Trace:
W20260709 13:01:21.623962  1569 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:21.625227  1501 catalog_manager.cc:5697] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d reported cstate change: term changed from 8 to 9, leader changed from 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) to 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196). New cstate: current_term: 9 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 13:01:21.629215  1676 raft_consensus.cc:2410] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 for term 14 because replica has last-logged OpId of term: 12 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 10.
I20260709 13:01:21.636082  1605 raft_consensus.cc:1275] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20260709 13:01:21.641330  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:21.617622  1596 rpcz_store.cc:276] 0709 13:01:18.992107 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:21.558688 (+2566581us) service_pool.cc:224] Handling call
0709 13:01:21.616085 (+ 57397us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:21.616495 (+   410us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:21.616536 (+    41us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:01:21.616619 (+    83us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:01:21.616683 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:21.616757 (+    74us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:21.616922 (+   165us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:01:21.650200  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:21.651710  1644 leader_election.cc:304] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:01:21.668067  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:21.670833  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:21.671615  1598 raft_consensus.cc:1240] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 8. Current term is 9. Ops: []
I20260709 13:01:21.671772  1599 raft_consensus.cc:2468] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 13.
I20260709 13:01:21.661705  6046 consensus_queue.cc:237] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.6, Last appended by leader: 6, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:21.661643  1605 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 2450) took 2878 ms. Trace:
I20260709 13:01:21.673354  1605 rpcz_store.cc:276] 0709 13:01:18.766938 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:21.556785 (+2789847us) service_pool.cc:224] Handling call
0709 13:01:21.596544 (+ 39759us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:21.636637 (+ 40093us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:21.644811 (+  8174us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:01:21.627494  1501 catalog_manager.cc:5697] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d reported cstate change: term changed from 10 to 13, leader changed from 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) to 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193). New cstate: current_term: 13 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 13:01:21.627841  6853 raft_consensus.cc:2804] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Leader election won for term 12
W20260709 13:01:21.677748  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 93773.8us
I20260709 13:01:21.680524  1598 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 2470) took 1843 ms. Trace:
I20260709 13:01:21.680922  1598 rpcz_store.cc:276] 0709 13:01:19.837293 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:21.563787 (+1726494us) service_pool.cc:224] Handling call
0709 13:01:21.635805 (+ 72018us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:21.680235 (+ 44430us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:21.680458 (+   223us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:01:21.695715  1603 raft_consensus.cc:2468] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 11.
W20260709 13:01:21.697724  1719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:21.593312  6795 consensus_queue.cc:1048] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 9, Time since last communication: 0.065s
I20260709 13:01:21.704794  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:21.710078  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:21.678997  2299 leader_election.cc:304] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
W20260709 13:01:21.722669  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 311534us
I20260709 13:01:21.723613  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 12 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:21.724779  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:21.725445  2333 raft_consensus.cc:2468] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 13.
I20260709 13:01:21.728207  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:21.729000  1603 raft_consensus.cc:2393] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 10: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:01:21.722273  1605 raft_consensus.cc:2468] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 13.
I20260709 13:01:21.595924  6752 consensus_queue.cc:1059] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.067s
I20260709 13:01:21.656384  6360 raft_consensus.cc:2804] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:01:21.677498  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:21.738312  1751 raft_consensus.cc:1275] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
I20260709 13:01:21.724299  2329 raft_consensus.cc:2468] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 11.
I20260709 13:01:21.721657  2330 raft_consensus.cc:2468] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 10.
I20260709 13:01:21.750242  1501 catalog_manager.cc:5697] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d reported cstate change: term changed from 7 to 12, leader changed from af520b67945d443b901bbeca37cc5701 (127.0.107.195) to 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194). New cstate: current_term: 12 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } }
I20260709 13:01:21.744719  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:21.746395  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:21.759599  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:21.760273  1603 raft_consensus.cc:3060] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:01:21.657387  6714 raft_consensus.cc:493] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:21.755191  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 15 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:21.755486  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:01:21.760840  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 192631us
I20260709 13:01:21.628125  6643 consensus_queue.cc:1059] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 13.12, Next index: 13, Last known committed idx: 10, Time since last communication: 0.070s
W20260709 13:01:21.761575  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 82703.3us
I20260709 13:01:21.824448  1605 raft_consensus.cc:2468] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 14.
W20260709 13:01:21.831576  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 201244us
I20260709 13:01:21.825522  2298 leader_election.cc:304] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
W20260709 13:01:21.829145  1717 leader_election.cc:336] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 15 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 2.793s (SENT)
I20260709 13:01:21.837805  1717 leader_election.cc:304] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:21.772998  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:21.757213  6854 raft_consensus.cc:2804] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:01:21.756412  6360 raft_consensus.cc:493] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:21.845661  1751 raft_consensus.cc:1275] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 10 index: 6. (term mismatch)
W20260709 13:01:21.839716  1643 leader_election.cc:336] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 2.147s (SENT)
I20260709 13:01:21.846547  1643 leader_election.cc:304] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:21.745783  6853 raft_consensus.cc:697] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 12 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:01:21.761052  6729 raft_consensus.cc:3060] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:01:21.842687  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:01:21.843672  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 192029us
I20260709 13:01:21.848565  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:21.864995  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:21.867671  1596 raft_consensus.cc:3060] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:01:21.845088  1501 catalog_manager.cc:5697] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d reported cstate change: term changed from 8 to 12. New cstate: current_term: 12 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 13:01:21.846138  1751 pending_rounds.cc:85] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701: Aborting all ops after (but not including) 5
I20260709 13:01:21.869180  1751 pending_rounds.cc:107] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701: Aborting uncommitted NO_OP operation due to leader change: 9.6
I20260709 13:01:21.869545  1751 raft_consensus.cc:2889] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:01:21.852909  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 12 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:21.872967  1600 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40154 (request call id 2350) took 3502 ms. Trace:
I20260709 13:01:21.873631  1600 rpcz_store.cc:276] 0709 13:01:18.344205 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:21.297744 (+2953539us) service_pool.cc:224] Handling call
0709 13:01:21.311721 (+ 13977us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:21.321578 (+  9857us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 13:01:21.414635 (+ 93057us) log.cc:844] Serialized 14 byte log entry
0709 13:01:21.663228 (+248593us) spinlock_profiling.cc:243] Waited 3.21 ms on lock 0x7b1800089ce0. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f4265d0a4bc 00007f426305848b 00007f426beb2ab9 00007f426307469b 00007f426307293f 00007f4263074c02 00007f4265cfc20b 00007f4265d01d23 00007f4265d0223a
0709 13:01:21.844524 (+181296us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:01:21.844603 (+    79us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:01:21.844668 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:21.844754 (+    86us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:21.846316 (+  1562us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":136538,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":44,"spinlock_wait_cycles":7376256,"thread_start_us":297582,"threads_started":3,"wal-append.queue_time_us":131581}
I20260709 13:01:21.887547  1603 raft_consensus.cc:2468] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 15.
I20260709 13:01:21.842953  6714 raft_consensus.cc:515] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:21.897112  6622 consensus_queue.cc:1048] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 13:01:21.771382  6408 consensus_queue.cc:1048] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.085s
I20260709 13:01:21.902427  1603 raft_consensus.cc:1275] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 12 index: 9. (index mismatch)
I20260709 13:01:21.903652  1603 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 2480) took 1886 ms. Trace:
I20260709 13:01:21.904065  1603 rpcz_store.cc:276] 0709 13:01:20.017146 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:21.888476 (+1871330us) service_pool.cc:224] Handling call
0709 13:01:21.888768 (+   292us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:21.903302 (+ 14534us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:21.903569 (+   267us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:01:21.842171  6854 raft_consensus.cc:493] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:21.851778  6360 raft_consensus.cc:3060] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:01:21.844892  6638 raft_consensus.cc:2749] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:01:21.912325  1599 raft_consensus.cc:1275] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 15 index: 9. (index mismatch)
I20260709 13:01:21.919981  1603 raft_consensus.cc:1275] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260709 13:01:21.920890  1603 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 1157) took 1881 ms. Trace:
I20260709 13:01:21.921311  1603 rpcz_store.cc:276] 0709 13:01:20.039515 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:21.905075 (+1865560us) service_pool.cc:224] Handling call
0709 13:01:21.912785 (+  7710us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:21.920537 (+  7752us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:21.920810 (+   273us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:01:21.923099  1599 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 2477) took 1907 ms. Trace:
I20260709 13:01:21.876837  6860 raft_consensus.cc:2804] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:01:21.928267  1599 rpcz_store.cc:276] 0709 13:01:20.016047 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:21.878900 (+1862853us) service_pool.cc:224] Handling call
0709 13:01:21.906592 (+ 27692us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:21.919326 (+ 12734us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:21.923027 (+  3701us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:01:21.932688  1602 raft_consensus.cc:1275] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 15 index: 8. (term mismatch)
I20260709 13:01:21.933314  1602 pending_rounds.cc:85] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a: Aborting all ops after (but not including) 7
I20260709 13:01:21.933670  1602 pending_rounds.cc:107] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a: Aborting uncommitted NO_OP operation due to leader change: 14.8
I20260709 13:01:21.934155  1602 raft_consensus.cc:2889] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:01:21.938679  1596 raft_consensus.cc:2468] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 14.
I20260709 13:01:21.940732  1596 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 1163) took 1698 ms. Trace:
I20260709 13:01:21.941208  1596 rpcz_store.cc:276] 0709 13:01:20.242541 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:21.939664 (+1697123us) service_pool.cc:224] Handling call
0709 13:01:21.940065 (+   401us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:21.940317 (+   252us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:21.940357 (+    40us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:01:21.940413 (+    56us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:01:21.940461 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:21.940531 (+    70us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:21.940661 (+   130us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:01:21.942795  1569 leader_election.cc:304] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:21.944505  1603 raft_consensus.cc:1240] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 8. Current term is 9. Ops: []
I20260709 13:01:21.945335  1603 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 1160) took 1703 ms. Trace:
I20260709 13:01:21.945730  1603 rpcz_store.cc:276] 0709 13:01:20.241470 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:21.922309 (+1680839us) service_pool.cc:224] Handling call
0709 13:01:21.944257 (+ 21948us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:21.945020 (+   763us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:21.945264 (+   244us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:01:21.949128  1605 raft_consensus.cc:1240] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 13. Current term is 14. Ops: []
I20260709 13:01:21.951653  1605 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 1159) took 1710 ms. Trace:
I20260709 13:01:21.952111  1605 rpcz_store.cc:276] 0709 13:01:20.241095 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:21.917781 (+1676686us) service_pool.cc:224] Handling call
0709 13:01:21.933041 (+ 15260us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:21.951295 (+ 18254us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:21.951578 (+   283us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:01:21.917171  6854 raft_consensus.cc:3060] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:01:21.933822  6860 raft_consensus.cc:493] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:01:21.980232  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 162621us
W20260709 13:01:21.973220  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 140166us
I20260709 13:01:21.947852  6605 leader_election.cc:290] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:21.987819  1596 raft_consensus.cc:1240] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 11. Current term is 12. Ops: []
I20260709 13:01:21.986048  2299 leader_election.cc:304] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
W20260709 13:01:21.989588  1717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:21.986276  1605 raft_consensus.cc:1275] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 15 index: 11. Preceding OpId from leader: term: 19 index: 13. (index mismatch)
I20260709 13:01:21.989558  1602 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 2478) took 1973 ms. Trace:
I20260709 13:01:21.993520  1602 rpcz_store.cc:276] 0709 13:01:20.016401 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:21.879992 (+1863591us) service_pool.cc:224] Handling call
0709 13:01:21.921611 (+ 41619us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:21.989238 (+ 67627us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:21.989495 (+   257us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:01:21.989406  1596 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 1168) took 1512 ms. Trace:
I20260709 13:01:21.995877  1596 rpcz_store.cc:276] 0709 13:01:20.477120 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:21.950097 (+1472977us) service_pool.cc:224] Handling call
0709 13:01:21.987598 (+ 37501us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:21.989126 (+  1528us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:21.989325 (+   199us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:01:21.993460  1601 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 2447) took 3223 ms. Trace:
I20260709 13:01:21.992662  1599 raft_consensus.cc:1240] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 8. Current term is 9. Ops: []
I20260709 13:01:21.963042  6705 raft_consensus.cc:493] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:22.002943  6046 consensus_queue.cc:1048] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.004s
I20260709 13:01:21.982590  6860 raft_consensus.cc:3060] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:01:21.991045  6863 raft_consensus.cc:493] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:01:22.075320  6858 log.cc:927] Time spent T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.074s	user 0.000s	sys 0.000s
I20260709 13:01:22.001536  6706 consensus_queue.cc:1048] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.041s
I20260709 13:01:22.019472  6714 leader_election.cc:290] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:22.003194  6699 leader_election.cc:290] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:01:22.063375  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:22.040820  6852 log.cc:927] Time spent T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.378s	user 0.001s	sys 0.000s
I20260709 13:01:22.004868  6853 consensus_queue.cc:237] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:22.073390  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:22.074476  6870 raft_consensus.cc:2749] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:01:22.132400  1604 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 2433) took 3378 ms. Trace:
I20260709 13:01:22.075253  1750 raft_consensus.cc:1240] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 14. Current term is 15. Ops: [14.8-14.8]
I20260709 13:01:22.081712  6821 raft_consensus.cc:493] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
W20260709 13:01:22.076037  6856 log.cc:927] Time spent T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.198s	user 0.000s	sys 0.000s
I20260709 13:01:21.990370  6862 raft_consensus.cc:493] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:22.077734  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:22.019932  1602 raft_consensus.cc:1275] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 12 index: 10. (index mismatch)
W20260709 13:01:22.078272  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 56101.3us
W20260709 13:01:22.139755  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 269842us
I20260709 13:01:22.145102  1752 raft_consensus.cc:2468] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 13.
I20260709 13:01:22.145630  1644 leader_election.cc:304] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:01:22.018698  6643 raft_consensus.cc:2804] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Leader pre-election won for term 12
W20260709 13:01:22.083899  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:22.148779  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:22.149297  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:22.084798  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 142125us
I20260709 13:01:22.086723  1598 raft_consensus.cc:2468] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 15.
W20260709 13:01:22.101290  1597 consensus_meta.cc:287] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.576s	user 0.016s	sys 0.000s
I20260709 13:01:22.150621  1597 raft_consensus.cc:2468] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 12.
I20260709 13:01:22.148548  1602 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 1177) took 1081 ms. Trace:
I20260709 13:01:22.079274  1601 rpcz_store.cc:276] 0709 13:01:18.761649 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:21.552598 (+2790949us) service_pool.cc:224] Handling call
0709 13:01:21.564551 (+ 11953us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:21.564831 (+   280us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 13:01:21.565192 (+   361us) log.cc:844] Serialized 14 byte log entry
0709 13:01:21.984564 (+419372us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:01:21.984645 (+    81us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:01:21.984737 (+    92us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:21.984832 (+    95us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:21.984992 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":14005,"prepare.queue_time_us":1000,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":44,"raft.queue_time_us":279608,"raft.run_cpu_time_us":275,"raft.run_wall_time_us":25868,"thread_start_us":381292,"threads_started":2,"wal-append.queue_time_us":358316}
I20260709 13:01:22.112118  1599 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 1162) took 1860 ms. Trace:
I20260709 13:01:22.153095  1599 rpcz_store.cc:276] 0709 13:01:20.242181 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:21.929376 (+1687195us) service_pool.cc:224] Handling call
0709 13:01:21.964604 (+ 35228us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:22.102061 (+137457us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:22.102421 (+   360us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:01:22.121177  1596 raft_consensus.cc:1275] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260709 13:01:22.124596  1746 raft_consensus.cc:1240] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 11. Current term is 12. Ops: []
W20260709 13:01:22.041615  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 216202us
W20260709 13:01:22.063375  6861 log.cc:927] Time spent T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.223s	user 0.000s	sys 0.001s
I20260709 13:01:22.076045  1753 raft_consensus.cc:1240] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 9. Current term is 10. Ops: []
I20260709 13:01:22.155479  2298 leader_election.cc:304] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:22.136984  1604 rpcz_store.cc:276] 0709 13:01:18.753800 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:21.417449 (+2663649us) service_pool.cc:224] Handling call
0709 13:01:21.463873 (+ 46424us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:21.464148 (+   275us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 13:01:21.604281 (+140133us) log.cc:844] Serialized 14 byte log entry
0709 13:01:22.131436 (+527155us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:01:22.131516 (+    80us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:01:22.131579 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:22.131676 (+    97us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:22.132321 (+   645us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":12531,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":50,"raft.queue_time_us":159156,"thread_start_us":575580,"threads_started":3}
I20260709 13:01:22.165828  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:22.152343  1602 rpcz_store.cc:276] 0709 13:01:21.067285 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:21.994840 (+927555us) service_pool.cc:224] Handling call
0709 13:01:22.006158 (+ 11318us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:22.145923 (+139765us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:22.148446 (+  2523us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:01:22.168331  1605 raft_consensus.cc:1275] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 15 index: 10. (index mismatch)
I20260709 13:01:22.166771  2330 raft_consensus.cc:2468] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 13.
I20260709 13:01:22.131618  6867 raft_consensus.cc:2749] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260709 13:01:22.143155  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 378495us
W20260709 13:01:22.172187  1569 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:22.169634  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:22.170229  1644 leader_election.cc:304] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:01:22.173749  2329 raft_consensus.cc:1275] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
W20260709 13:01:22.174911  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 344692us
I20260709 13:01:22.176719  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:22.180320  2332 raft_consensus.cc:2393] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 11: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:01:22.111387  6705 raft_consensus.cc:515] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:22.146870  6699 raft_consensus.cc:2804] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:01:22.178565  2337 raft_consensus.cc:2468] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 13.
W20260709 13:01:22.138885  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 253313us
I20260709 13:01:22.192245  1603 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 1174) took 1126 ms. Trace:
I20260709 13:01:22.201126  1603 rpcz_store.cc:276] 0709 13:01:21.066309 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:21.946638 (+880329us) service_pool.cc:224] Handling call
0709 13:01:21.947153 (+   515us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:21.954175 (+  7022us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 13:01:21.985814 (+ 31639us) log.cc:844] Serialized 14 byte log entry
0709 13:01:22.191759 (+205945us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:01:22.191836 (+    77us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:01:22.191917 (+    81us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:22.192005 (+    88us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:22.192160 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":22311,"prepare.queue_time_us":32898,"prepare.run_cpu_time_us":55,"prepare.run_wall_time_us":57,"raft.queue_time_us":23622,"thread_start_us":164348,"threads_started":1}
W20260709 13:01:22.196223  2301 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:22.198084  2298 leader_election.cc:304] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:01:22.148197  6643 raft_consensus.cc:493] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Starting leader election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:01:22.234040  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:22.171342  6866 raft_consensus.cc:493] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:22.237309  2301 leader_election.cc:304] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:01:22.175772  6512 raft_consensus.cc:2749] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:01:22.245306  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:22.246695  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:22.167409  6862 raft_consensus.cc:515] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:22.249119  6862 leader_election.cc:290] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:22.249523  6360 raft_consensus.cc:515] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:22.250636  2330 raft_consensus.cc:2468] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 14.
I20260709 13:01:22.255931  1677 raft_consensus.cc:1275] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 16 index: 13. (index mismatch)
I20260709 13:01:22.258100  2334 raft_consensus.cc:1240] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 14. Current term is 15. Ops: []
I20260709 13:01:22.148116  6821 raft_consensus.cc:515] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:22.259622  1717 leader_election.cc:336] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 14 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 1.503s (SENT)
I20260709 13:01:22.261057  1717 leader_election.cc:304] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 13:01:22.265492  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 95340.2us
I20260709 13:01:22.239137  6643 raft_consensus.cc:3060] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Advancing to term 12
W20260709 13:01:22.287479  1644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:22.141543  6863 raft_consensus.cc:515] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:22.289268  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 91342.8us
I20260709 13:01:22.291363  1644 leader_election.cc:304] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:01:22.295575  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:22.240839  6699 raft_consensus.cc:697] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 12 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:01:22.307034  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:22.281383  6528 raft_consensus.cc:2804] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Leader pre-election won for term 14
W20260709 13:01:22.316059  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 56473.9us
I20260709 13:01:22.314070  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:22.301156  6862 raft_consensus.cc:2804] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:01:22.310570  6714 raft_consensus.cc:2749] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:01:22.330811  6875 raft_consensus.cc:493] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:01:22.331447  6863 leader_election.cc:290] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:22.332384  6705 leader_election.cc:290] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:22.314867  6512 raft_consensus.cc:2749] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:01:22.320119  6528 raft_consensus.cc:493] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:22.253700  6866 raft_consensus.cc:515] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:01:22.340374  6877 log.cc:927] Time spent T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.089s	user 0.001s	sys 0.000s
I20260709 13:01:22.342062  6360 leader_election.cc:290] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 15 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:22.343536  2298 leader_election.cc:304] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:22.351790  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:22.380357  2330 raft_consensus.cc:2468] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 13.
I20260709 13:01:22.363257  6876 raft_consensus.cc:2749] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:01:22.364696  6875 raft_consensus.cc:515] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:22.286289  2332 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec"
dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
mode: GRACEFUL
new_leader_uuid: "af520b67945d443b901bbeca37cc5701"
 from {username='slave'} at 127.0.0.1:35790
I20260709 13:01:22.356491  6862 raft_consensus.cc:493] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:01:22.367791  6885 raft_consensus.cc:493] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:22.369179  6821 leader_election.cc:290] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:01:22.376317  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 171625us
W20260709 13:01:22.389181  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 45805.8us
I20260709 13:01:22.381713  1569 leader_election.cc:304] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:01:22.394057  2332 raft_consensus.cc:606] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Received request to transfer leadership to TS af520b67945d443b901bbeca37cc5701
I20260709 13:01:22.406762  2332 raft_consensus.cc:614] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 13:01:22.366614  6699 consensus_queue.cc:237] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:22.375744  6528 raft_consensus.cc:3060] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:01:22.408802  2298 leader_election.cc:304] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:22.417567  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:22.404560  6862 raft_consensus.cc:3060] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:01:22.401005  6512 raft_consensus.cc:2804] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:01:22.421715  1677 raft_consensus.cc:2468] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 13.
I20260709 13:01:22.422510  6512 raft_consensus.cc:493] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:22.424563  6360 raft_consensus.cc:2749] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:01:22.426806  6512 raft_consensus.cc:3060] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:01:22.408142  6896 raft_consensus.cc:2804] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:01:22.429900  6854 raft_consensus.cc:515] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:22.409456  6884 raft_consensus.cc:2804] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:01:22.440820  1677 raft_consensus.cc:1240] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 13. Current term is 14. Ops: [13.7-13.7]
W20260709 13:01:22.474110  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 267685us
I20260709 13:01:22.432595  6896 raft_consensus.cc:493] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:01:22.474782  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 165176us
W20260709 13:01:22.449084  6729 consensus_meta.cc:287] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.554s	user 0.014s	sys 0.000s
I20260709 13:01:22.437294  6884 raft_consensus.cc:493] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:22.495244  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
W20260709 13:01:22.475363  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 188335us
I20260709 13:01:22.480688  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:22.433665  6705 consensus_queue.cc:1048] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.046s
I20260709 13:01:22.482964  2332 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "a3accbb733bb4cafa69428319664f239"
dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
mode: GRACEFUL
new_leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
 from {username='slave'} at 127.0.0.1:35790
I20260709 13:01:22.521130  2299 leader_election.cc:304] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:01:22.521482  2332 raft_consensus.cc:606] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Received request to transfer leadership to TS 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:22.521889  2332 raft_consensus.cc:614] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 13:01:22.480014  6896 raft_consensus.cc:3060] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:01:22.473866  6863 consensus_queue.cc:1059] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 11.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.013s
I20260709 13:01:22.488796  6854 leader_election.cc:290] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 13 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:22.496946  6881 consensus_queue.cc:1048] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.064s
I20260709 13:01:22.497148  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:22.505475  6884 raft_consensus.cc:3060] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:01:22.503548  6875 leader_election.cc:290] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:22.409646  6885 raft_consensus.cc:515] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:22.519973  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:22.520704  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:22.523262  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:22.525393  6904 consensus_queue.cc:1059] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 11.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.017s
I20260709 13:01:22.534420  6866 leader_election.cc:290] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:22.515183  6876 consensus_queue.cc:1048] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.019s
I20260709 13:01:22.559646  1675 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "b9bb5648969b4299a776843d72043448"
dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
mode: GRACEFUL
new_leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
 from {username='slave'} at 127.0.0.1:43266
I20260709 13:01:22.560326  1675 raft_consensus.cc:606] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 LEADER]: Received request to transfer leadership to TS 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:22.536929  6888 raft_consensus.cc:2749] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:01:22.529261  6605 consensus_queue.cc:1048] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.033s
W20260709 13:01:22.541678  1719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:22.565290  6887 log.cc:927] Time spent T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.138s	user 0.000s	sys 0.001s
W20260709 13:01:22.565712  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:22.543192  1718 leader_election.cc:304] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:22.567142  2332 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca"
dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
mode: GRACEFUL
new_leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
 from {username='slave'} at 127.0.0.1:35790
I20260709 13:01:22.544094  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:22.567871  2332 raft_consensus.cc:606] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 14 LEADER]: Received request to transfer leadership to TS 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:22.569154  6513 raft_consensus.cc:3055] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 13:01:22.569566  6513 raft_consensus.cc:740] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:01:22.570492  6513 consensus_queue.cc:260] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 13.12, Last appended by leader: 12, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:22.571271  2332 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a"
dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
mode: GRACEFUL
new_leader_uuid: "af520b67945d443b901bbeca37cc5701"
 from {username='slave'} at 127.0.0.1:35790
I20260709 13:01:22.571592  6513 raft_consensus.cc:3060] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:01:22.572007  2332 raft_consensus.cc:606] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Received request to transfer leadership to TS af520b67945d443b901bbeca37cc5701
I20260709 13:01:22.572465  2332 raft_consensus.cc:614] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 13:01:22.546741  6821 raft_consensus.cc:493] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:22.574462  6821 raft_consensus.cc:515] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:22.548102  6714 consensus_queue.cc:1059] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.028s
I20260709 13:01:22.552130  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:22.577903  1601 raft_consensus.cc:2468] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 14.
I20260709 13:01:22.555140  1672 raft_consensus.cc:2468] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 15.
I20260709 13:01:22.554242  1604 raft_consensus.cc:1275] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 16 index: 13. (index mismatch)
I20260709 13:01:22.555450  1671 raft_consensus.cc:3060] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:01:22.564873  6860 raft_consensus.cc:515] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:22.551144  1643 leader_election.cc:336] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 2.153s (SENT)
W20260709 13:01:22.584165  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 177933us
I20260709 13:01:22.591557  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:22.592442  1646 leader_election.cc:304] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:01:22.597028  6046 raft_consensus.cc:2749] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 13:01:22.597329  2298 leader_election.cc:304] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:01:22.557788  6408 raft_consensus.cc:3055] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 11 LEADER]: Stepping down as leader of term 11
W20260709 13:01:22.597304  1717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:22.552917  6865 raft_consensus.cc:3055] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 LEADER]: Stepping down as leader of term 13
I20260709 13:01:22.555837  6878 raft_consensus.cc:493] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
W20260709 13:01:22.592743  6857 log.cc:927] Time spent T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.751s	user 0.003s	sys 0.000s
I20260709 13:01:22.602319  6904 raft_consensus.cc:2804] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:01:22.598394  1646 leader_election.cc:304] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:22.610780  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
W20260709 13:01:22.615237  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 64153.8us
I20260709 13:01:22.616097  1674 raft_consensus.cc:1240] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 13. Current term is 14. Ops: []
I20260709 13:01:22.518981  6512 raft_consensus.cc:515] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:22.619104  1646 leader_election.cc:304] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:22.618952  6046 raft_consensus.cc:2804] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:01:22.627255  1646 leader_election.cc:304] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:22.602798  6408 raft_consensus.cc:740] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:01:22.602087  6879 consensus_queue.cc:1048] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.013s
I20260709 13:01:22.634802  6046 raft_consensus.cc:697] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 14 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:01:22.604981  6865 raft_consensus.cc:740] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:01:22.637703  2334 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23"
dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
mode: GRACEFUL
new_leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
 from {username='slave'} at 127.0.0.1:35790
I20260709 13:01:22.637980  6864 raft_consensus.cc:2804] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:01:22.638685  6864 raft_consensus.cc:697] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 14 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:01:22.611438  6904 raft_consensus.cc:493] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:22.641112  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:22.641577  2299 leader_election.cc:304] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:22.638370  2334 raft_consensus.cc:606] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Received request to transfer leadership to TS 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:22.642431  2334 raft_consensus.cc:614] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 13:01:22.631546  6898 raft_consensus.cc:2804] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:01:22.653962  1671 raft_consensus.cc:2468] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 13.
W20260709 13:01:22.657138  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:22.664659  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:01:22.671046  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 127908us
I20260709 13:01:22.667097  6046 consensus_queue.cc:237] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.6, Last appended by leader: 6, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:22.665463  1750 raft_consensus.cc:2468] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 11.
I20260709 13:01:22.683933  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:22.684681  1750 raft_consensus.cc:2468] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 14.
I20260709 13:01:22.684477  6699 raft_consensus.cc:2749] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
W20260709 13:01:22.695291  1569 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:22.608103  6592 consensus_queue.cc:1059] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 14.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.035s
W20260709 13:01:22.708808  1643 leader_election.cc:336] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 1.717s (SENT)
I20260709 13:01:22.709429  1643 leader_election.cc:304] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:01:22.710218  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:22.710893  1753 raft_consensus.cc:2468] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 12.
I20260709 13:01:22.711629  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:22.712200  1752 raft_consensus.cc:2468] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 8.
I20260709 13:01:22.643328  6904 raft_consensus.cc:3060] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:01:22.716044  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:22.708213   431 auto_leader_rebalancer.cc:452] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260709 13:01:22.639389  6643 raft_consensus.cc:515] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:22.656349  6876 consensus_queue.cc:1048] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.075s
I20260709 13:01:22.681809  6898 raft_consensus.cc:697] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:01:22.737927  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:01:22.760916  1717 leader_election.cc:336] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 1.565s (SENT)
I20260709 13:01:22.788121  6600 leader_election.cc:290] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:22.788254  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:22.767620  6605 raft_consensus.cc:493] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:22.789229  1747 raft_consensus.cc:2393] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 15: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:01:22.789109  6605 raft_consensus.cc:515] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:22.790627  6898 consensus_queue.cc:237] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:22.791687  6605 leader_election.cc:290] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:22.725914  6714 raft_consensus.cc:493] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:22.719017  6885 leader_election.cc:290] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:22.623057  6878 raft_consensus.cc:515] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:22.830678  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:22.827832  6916 raft_consensus.cc:3055] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:01:22.835877  1752 raft_consensus.cc:1240] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 9. Current term is 10. Ops: [9.7-9.7]
I20260709 13:01:22.801335  6821 leader_election.cc:290] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 13:01:22.840379  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 167325us
I20260709 13:01:22.801180  6911 raft_consensus.cc:2804] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:01:22.844177  1718 leader_election.cc:304] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:22.851791  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:22.852308  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:22.852494  1752 raft_consensus.cc:3060] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:01:22.852931  1604 raft_consensus.cc:3060] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:01:22.851922  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:22.852807  1718 leader_election.cc:304] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:22.835294  6885 raft_consensus.cc:493] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:22.837574  6909 raft_consensus.cc:493] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:01:22.858908  1717 leader_election.cc:336] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 1.660s (SENT)
I20260709 13:01:22.837301  6916 raft_consensus.cc:740] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:01:22.859803  1754 raft_consensus.cc:1240] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 13 LEADER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 12. Current term is 13. Ops: [12.11-12.11]
I20260709 13:01:22.817600  6714 raft_consensus.cc:515] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:22.860780  1717 leader_election.cc:304] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:22.831037  6864 consensus_queue.cc:237] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 12.6, Last appended by leader: 6, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:22.801496  6408 consensus_queue.cc:260] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:22.859289  6860 leader_election.cc:290] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 10 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:22.851238  6911 raft_consensus.cc:493] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:22.927953  6911 raft_consensus.cc:3060] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:01:22.866153  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:22.873447  6914 raft_consensus.cc:2804] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:01:22.871827  6885 raft_consensus.cc:515] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:22.860431  6528 raft_consensus.cc:515] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:22.943346  6528 leader_election.cc:290] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 14 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:22.935401  6914 raft_consensus.cc:493] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:22.944965  6914 raft_consensus.cc:3060] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:01:22.948180  6885 leader_election.cc:290] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:22.870059  6643 leader_election.cc:290] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 12 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:22.952632  6890 raft_consensus.cc:3055] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:01:22.953166  6890 raft_consensus.cc:740] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:01:22.954190  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:22.954087  6890 consensus_queue.cc:260] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.12, Last appended by leader: 12, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:22.954936  1748 raft_consensus.cc:2468] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 10.
I20260709 13:01:22.872726  1754 raft_consensus.cc:1275] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 13 index: 8. (term mismatch)
W20260709 13:01:22.874255  2300 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:22.836580  6862 raft_consensus.cc:515] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:22.778841  6865 consensus_queue.cc:260] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 13.7, Last appended by leader: 7, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:01:22.871470  6895 log.cc:927] Time spent T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.443s	user 0.002s	sys 0.000s
I20260709 13:01:22.871186  6916 consensus_queue.cc:260] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:22.872785  6908 consensus_queue.cc:1048] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.113s
I20260709 13:01:22.838420  6896 raft_consensus.cc:515] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:22.872951  6878 leader_election.cc:290] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:22.867298  6821 raft_consensus.cc:2804] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:01:22.952801  1679 raft_consensus.cc:2393] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 14: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:01:22.955714  1754 pending_rounds.cc:85] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701: Aborting all ops after (but not including) 7
I20260709 13:01:22.911720  2330 raft_consensus.cc:2468] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 13.
I20260709 13:01:22.955853  1751 raft_consensus.cc:1217] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Deduplicated request from leader. Original: 10.7->[12.8-14.9]   Dedup: 14.9->[]
W20260709 13:01:22.956384  2300 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:22.932520  6879 raft_consensus.cc:493] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:01:22.965646  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 106758us
W20260709 13:01:22.969403  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:22.932011  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:22.983240  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:22.865947  6909 raft_consensus.cc:515] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:22.978044  1673 raft_consensus.cc:1240] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 11. Current term is 12. Ops: []
I20260709 13:01:22.984201  1751 raft_consensus.cc:1240] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 14. Current term is 15. Ops: [12.8-14.9]
I20260709 13:01:22.983764  1754 pending_rounds.cc:107] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701: Aborting uncommitted NO_OP operation due to leader change: 12.8
I20260709 13:01:22.989068  1754 raft_consensus.cc:2889] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 13:01:22.990504  6917 log.cc:927] Time spent T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.069s	user 0.000s	sys 0.001s
I20260709 13:01:22.992525  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:22.993381  1750 raft_consensus.cc:2393] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 10: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:01:22.993590  6909 leader_election.cc:290] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:22.998701  6408 raft_consensus.cc:3060] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:01:23.002889  6862 leader_election.cc:290] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 12 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:23.003865  1678 raft_consensus.cc:1240] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 10. Current term is 11. Ops: []
I20260709 13:01:23.005180  6896 leader_election.cc:290] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 11 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:23.006716  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:23.007346  1599 raft_consensus.cc:3060] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:01:22.980697   431 auto_leader_rebalancer.cc:527] All tables' leader rebalancing finished this round
I20260709 13:01:23.012873  6879 raft_consensus.cc:515] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:23.015146  6879 leader_election.cc:290] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:23.019052  6512 leader_election.cc:290] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 13 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:22.963536  6890 raft_consensus.cc:3060] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:01:22.980779  6916 raft_consensus.cc:3060] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:01:23.024938  6878 raft_consensus.cc:493] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:23.025617  6878 raft_consensus.cc:515] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:23.028245  6878 leader_election.cc:290] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:22.983593  6926 raft_consensus.cc:2749] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:01:23.029815  1748 raft_consensus.cc:3060] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Advancing to term 12
W20260709 13:01:23.037583  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 184802us
I20260709 13:01:22.985164  1747 raft_consensus.cc:2393] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 11: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:01:22.974502  1752 raft_consensus.cc:2468] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 9.
W20260709 13:01:22.986001  1717 leader_election.cc:336] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 14 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 2.571s (SENT)
I20260709 13:01:23.043159  1717 leader_election.cc:304] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:23.046802  2334 raft_consensus.cc:1240] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 11. Current term is 12. Ops: []
I20260709 13:01:23.050155  2332 raft_consensus.cc:1275] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 13 index: 11. (index mismatch)
I20260709 13:01:23.051287  1718 leader_election.cc:304] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 13:01:23.053820  1604 raft_consensus.cc:2468] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 10.
I20260709 13:01:23.055977  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:23.057648  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:23.060729  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:23.061697  1752 raft_consensus.cc:2393] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 11: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:01:23.055260  2298 leader_election.cc:304] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:01:23.060846  1749 raft_consensus.cc:1240] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 13. Current term is 14. Ops: []
I20260709 13:01:23.063047  2329 raft_consensus.cc:2393] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 13: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:01:23.065047  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:23.066560  1754 raft_consensus.cc:1240] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 11. Current term is 12. Ops: []
I20260709 13:01:23.066788  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:23.067436  1747 raft_consensus.cc:3060] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:01:23.067427  1718 leader_election.cc:304] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:23.070933  2301 leader_election.cc:304] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:23.073274  2301 leader_election.cc:304] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:22.988257  6821 raft_consensus.cc:493] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:23.084206  1604 raft_consensus.cc:2468] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 10.
I20260709 13:01:23.079216  2301 leader_election.cc:304] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:23.067118  6513 raft_consensus.cc:2804] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 13:01:23.064812  6908 raft_consensus.cc:2749] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:01:23.086968  2301 leader_election.cc:304] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:01:23.087507  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:23.090251  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:23.096206  1749 raft_consensus.cc:2368] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 13. Current term is 14.
I20260709 13:01:23.075358  6926 raft_consensus.cc:2804] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:01:23.099606  2298 leader_election.cc:304] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:01:23.100557  1599 raft_consensus.cc:2468] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 12.
W20260709 13:01:23.101632  1570 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:23.101750  2298 leader_election.cc:304] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
W20260709 13:01:23.102094  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 116134us
I20260709 13:01:23.080405  6879 raft_consensus.cc:2804] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:01:23.060665  6911 raft_consensus.cc:515] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:23.080977  6821 raft_consensus.cc:3060] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:01:23.085899  6485 raft_consensus.cc:2804] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 13:01:23.079680  6512 raft_consensus.cc:2804] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:01:23.083189  6909 raft_consensus.cc:493] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:23.094336  6643 raft_consensus.cc:2804] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:01:23.143126  6643 raft_consensus.cc:493] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:23.143601  6643 raft_consensus.cc:3060] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:01:23.120585  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:23.100982  6926 raft_consensus.cc:493] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:23.121645  6513 raft_consensus.cc:697] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 10 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:01:23.123050  6699 raft_consensus.cc:2804] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:01:23.111469  6879 raft_consensus.cc:493] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:23.125742  2331 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39726 (request call id 1200) took 1524 ms. Trace:
I20260709 13:01:23.126327  6894 raft_consensus.cc:2749] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 13:01:23.129541  6897 raft_consensus.cc:493] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:01:23.130699  1644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:23.135926  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:23.136677  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:23.169066  1677 raft_consensus.cc:3060] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Advancing to term 10
I20260709 13:01:23.137526  1501 catalog_manager.cc:5697] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d reported cstate change: term changed from 5 to 7, leader changed from 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194) to 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193). New cstate: current_term: 7 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } }
I20260709 13:01:23.138998  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:23.170807  1671 raft_consensus.cc:2468] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 10.
W20260709 13:01:23.140133  6924 log.cc:927] Time spent T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.212s	user 0.003s	sys 0.000s
W20260709 13:01:23.142203  6921 log.cc:927] Time spent T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.224s	user 0.000s	sys 0.001s
I20260709 13:01:23.111967  6706 raft_consensus.cc:2804] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:01:23.119048  6512 raft_consensus.cc:493] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:23.149019  1747 raft_consensus.cc:2410] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for term 13 because replica has last-logged OpId of term: 12 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 8.
I20260709 13:01:23.150573  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:23.150861  1604 raft_consensus.cc:2468] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 13.
W20260709 13:01:23.183465  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:23.124204  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 415355us
I20260709 13:01:23.183693  1599 raft_consensus.cc:3060] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Advancing to term 13
W20260709 13:01:23.184849  1643 leader_election.cc:336] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 1.753s (SENT)
I20260709 13:01:23.185451  1643 leader_election.cc:304] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:01:23.112362  6605 raft_consensus.cc:2749] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:01:23.192167  1751 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55840 (request call id 2654) took 1121 ms. Trace:
I20260709 13:01:23.152221  2331 rpcz_store.cc:276] 0709 13:01:21.600890 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:21.628105 (+ 27215us) service_pool.cc:224] Handling call
0709 13:01:21.628451 (+   346us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:21.648234 (+ 19783us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 13:01:21.704551 (+ 56317us) log.cc:844] Serialized 14 byte log entry
0709 13:01:22.162635 (+458084us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:01:22.162714 (+    79us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:01:22.162791 (+    77us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:23.125458 (+962667us) spinlock_profiling.cc:243] Waited 19.5 ms on lock 0x7b540072e554. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:01:23.125518 (+    60us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:23.125666 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":115260,"prepare.run_cpu_time_us":54,"prepare.run_wall_time_us":57,"raft.queue_time_us":201012,"raft.run_cpu_time_us":145,"raft.run_wall_time_us":5658,"spinlock_wait_cycles":44798848,"thread_start_us":457514,"threads_started":2,"wal-append.queue_time_us":315046,"wal-append.run_cpu_time_us":3953,"wal-append.run_wall_time_us":1046057}
I20260709 13:01:23.193179  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
W20260709 13:01:23.148100  6861 log.cc:927] Time spent T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.051s	user 0.000s	sys 0.000s
I20260709 13:01:23.165824  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:23.118736  6915 raft_consensus.cc:493] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:23.188405  1671 raft_consensus.cc:1275] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20260709 13:01:23.149094  6926 raft_consensus.cc:3060] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:01:23.188306  1644 leader_election.cc:304] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:01:23.193806  2329 raft_consensus.cc:3060] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:01:23.192626  1751 rpcz_store.cc:276] 0709 13:01:22.071034 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:22.071304 (+   270us) service_pool.cc:224] Handling call
0709 13:01:22.071804 (+   500us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:01:22.987023 (+915219us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:23.191895 (+204872us) spinlock_profiling.cc:243] Waited 884 ms on lock 0x7b54002f49d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:01:23.192082 (+   187us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2032438528}
I20260709 13:01:23.163601  6879 raft_consensus.cc:3060] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Advancing to term 12
W20260709 13:01:23.166991  6884 consensus_meta.cc:287] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.619s	user 0.011s	sys 0.002s
I20260709 13:01:23.205149  2337 raft_consensus.cc:1240] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 12. Current term is 13. Ops: []
I20260709 13:01:23.205399  2298 leader_election.cc:304] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:01:23.151239  6909 raft_consensus.cc:515] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:23.166642  6699 raft_consensus.cc:697] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 12 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:01:23.186457  6512 raft_consensus.cc:3060] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:01:23.212826  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:23.187511  6706 raft_consensus.cc:493] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:23.192066  6714 leader_election.cc:290] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:23.217479  6513 consensus_queue.cc:237] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:23.259372  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:23.259634  6894 consensus_queue.cc:1059] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 12.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 13:01:23.260905  6859 raft_consensus.cc:493] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:23.261518  6699 consensus_queue.cc:237] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:23.206125  6854 raft_consensus.cc:493] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:23.290335  2337 raft_consensus.cc:1275] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 11 index: 11. (index mismatch)
I20260709 13:01:23.278029  1677 raft_consensus.cc:2468] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 10.
W20260709 13:01:23.292032  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 107217us
I20260709 13:01:23.279422  1599 raft_consensus.cc:2468] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 13.
W20260709 13:01:23.279825  1569 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:23.284925  1748 raft_consensus.cc:2468] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 12.
I20260709 13:01:23.256568  6706 raft_consensus.cc:3060] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:01:23.206445  6528 raft_consensus.cc:2749] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:01:23.210860  6882 raft_consensus.cc:2804] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Leader election won for term 13
I20260709 13:01:23.309438  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:23.310977  1604 raft_consensus.cc:1275] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
W20260709 13:01:23.305127  1643 leader_election.cc:336] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 1.821s (SENT)
I20260709 13:01:23.312987  1643 leader_election.cc:304] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:23.187211  6897 raft_consensus.cc:515] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:23.256206  6605 raft_consensus.cc:2804] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:01:23.312539  6863 consensus_queue.cc:1048] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.008s
I20260709 13:01:23.285874  6909 leader_election.cc:290] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:23.327207  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:23.327374  6885 raft_consensus.cc:493] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:23.328069  1599 raft_consensus.cc:2393] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 12: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:01:23.327927  6885 raft_consensus.cc:515] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:23.329136  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:23.330044  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:23.331188  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:23.332254  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:23.333287  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 14 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:23.333957  1601 raft_consensus.cc:2393] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 14: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:01:23.330286  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:23.334813  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:23.335268  1599 raft_consensus.cc:2468] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 15.
I20260709 13:01:23.217628  6914 raft_consensus.cc:515] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:23.336122  6894 raft_consensus.cc:2749] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20260709 13:01:23.337565  6935 log.cc:927] Time spent T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.118s	user 0.000s	sys 0.001s
I20260709 13:01:23.339970  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:23.344375  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:23.341472  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:23.354354  1604 raft_consensus.cc:2393] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 9: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:01:23.355389  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:23.356539  2332 raft_consensus.cc:1240] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 12. Current term is 13. Ops: []
I20260709 13:01:23.360522  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:23.312467  6905 raft_consensus.cc:3055] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 LEADER]: Stepping down as leader of term 12
W20260709 13:01:23.365932  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 60847.6us
I20260709 13:01:23.368907  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:23.369378  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:23.369760  1596 raft_consensus.cc:2368] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 12. Current term is 13.
I20260709 13:01:23.370072  1598 raft_consensus.cc:2468] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 11.
I20260709 13:01:23.370978  1596 raft_consensus.cc:1240] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 9. Current term is 10. Ops: []
W20260709 13:01:23.369158  6931 log.cc:927] Time spent T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.221s	user 0.000s	sys 0.001s
W20260709 13:01:23.375236  6936 log.cc:927] Time spent T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.098s	user 0.000s	sys 0.000s
W20260709 13:01:23.376560  6912 log.cc:927] Time spent T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.075s	user 0.001s	sys 0.000s
I20260709 13:01:23.215051  6884 raft_consensus.cc:515] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:23.318373  6605 raft_consensus.cc:493] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:23.382273  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 14 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:23.403789  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:23.366825  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:23.404758  1603 raft_consensus.cc:2468] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 9.
I20260709 13:01:23.405707  1601 raft_consensus.cc:2410] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for term 14 because replica has last-logged OpId of term: 13 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 7.
I20260709 13:01:23.406461  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 17 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:23.409642  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:23.410163  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:23.315729  6854 raft_consensus.cc:515] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:23.386381  6911 leader_election.cc:290] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 14 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:23.386945  6878 consensus_queue.cc:1048] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 13:01:23.387948  6485 raft_consensus.cc:697] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 13 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:01:23.342628  6859 raft_consensus.cc:515] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:23.388460  1602 raft_consensus.cc:1275] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
I20260709 13:01:23.365206  6905 raft_consensus.cc:740] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
W20260709 13:01:23.389086  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 70320.7us
I20260709 13:01:23.389650  6909 consensus_queue.cc:1048] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 13:01:23.389128  6932 raft_consensus.cc:493] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:23.393334  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:23.394264  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:23.425568  2332 raft_consensus.cc:3060] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:01:23.351486  6882 raft_consensus.cc:697] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 13 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:01:23.396104  6914 leader_election.cc:290] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 15 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:23.396342  6897 leader_election.cc:290] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:23.398926  1679 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57092 (request call id 1137) took 1135 ms. Trace:
I20260709 13:01:23.441972  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:23.399076  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
W20260709 13:01:23.400103  6904 consensus_meta.cc:287] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.673s	user 0.012s	sys 0.000s
I20260709 13:01:23.216123  6915 raft_consensus.cc:515] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:23.403684  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:23.410475  1598 raft_consensus.cc:2393] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 12: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:01:23.380687  6863 consensus_queue.cc:1059] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.056s
I20260709 13:01:23.411865  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:23.384143  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 15 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:23.462841  1604 raft_consensus.cc:2393] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 15: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:01:23.422988  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:23.423803  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:23.430384  1605 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 2489) took 1944 ms. Trace:
W20260709 13:01:23.433173  6929 log.cc:927] Time spent T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.314s	user 0.000s	sys 0.001s
I20260709 13:01:23.463783  1605 rpcz_store.cc:276] 0709 13:01:21.486394 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:22.002390 (+515996us) service_pool.cc:224] Handling call
0709 13:01:22.151685 (+149295us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:22.168763 (+ 17078us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:23.430077 (+1261314us) spinlock_profiling.cc:243] Waited 149 ms on lock 0x7b5400301950. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661618ad 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:01:23.430296 (+   219us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":342372096}
W20260709 13:01:23.433211  2300 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:23.434892  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:23.440732  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:23.442744  1747 raft_consensus.cc:2468] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 12.
I20260709 13:01:23.466176  1599 raft_consensus.cc:2468] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 10.
I20260709 13:01:23.466852  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:23.467583  1747 raft_consensus.cc:3060] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Advancing to term 14
I20260709 13:01:23.404834  6605 raft_consensus.cc:3060] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Advancing to term 14
W20260709 13:01:23.443288  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:23.446516  2335 raft_consensus.cc:3060] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:01:23.442390  1679 rpcz_store.cc:276] 0709 13:01:22.263197 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:22.298577 (+ 35380us) service_pool.cc:224] Handling call
0709 13:01:23.398689 (+1100112us) spinlock_profiling.cc:243] Waited 633 ms on lock 0x7b5400090e54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:01:23.398833 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1454840704}
W20260709 13:01:23.418279  6935 log.cc:927] Time spent T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.054s	user 0.000s	sys 0.001s
I20260709 13:01:23.362944  6643 raft_consensus.cc:515] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:23.456928  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:23.385020  6160 consensus_queue.cc:1059] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 11.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
W20260709 13:01:23.470352  6880 log.cc:927] Time spent T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 1.206s	user 0.004s	sys 0.000s
I20260709 13:01:23.470927  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:23.480911  1749 raft_consensus.cc:2368] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for earlier term 13. Current term is 14.
I20260709 13:01:23.482322  1602 raft_consensus.cc:2393] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 14: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:01:23.471343  6854 leader_election.cc:290] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:23.488554  1747 raft_consensus.cc:2468] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 14.
I20260709 13:01:23.489380  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:23.491225  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:23.498268  1747 raft_consensus.cc:1275] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 11 index: 8. Preceding OpId from leader: term: 12 index: 9. (index mismatch)
I20260709 13:01:23.492202  1596 raft_consensus.cc:2468] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 15.
I20260709 13:01:23.510244  2335 raft_consensus.cc:2468] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 15.
I20260709 13:01:23.514935  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:23.447134  6905 consensus_queue.cc:260] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.9, Last appended by leader: 9, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:23.532055  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:01:23.547452  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 61356.8us
I20260709 13:01:23.548738  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:23.458830  6904 raft_consensus.cc:515] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:23.562068  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:23.562899  1602 raft_consensus.cc:2468] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 12.
I20260709 13:01:23.586287  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:23.587420  1602 raft_consensus.cc:2468] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 15.
I20260709 13:01:23.588640  2332 raft_consensus.cc:2468] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 14.
I20260709 13:01:23.461961  6932 raft_consensus.cc:515] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:23.593665  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 14 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:23.600740  1596 raft_consensus.cc:2393] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 14: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:01:23.499565  6939 consensus_queue.cc:1048] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.097s
I20260709 13:01:23.528262  6875 consensus_queue.cc:1048] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.048s
I20260709 13:01:23.606072  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:23.610891  1602 raft_consensus.cc:2468] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 13.
W20260709 13:01:23.614372  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:23.623005  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:23.596784  6905 raft_consensus.cc:3060] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:01:23.625483  6863 raft_consensus.cc:3055] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:01:23.629189  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:23.528236  6485 consensus_queue.cc:237] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:23.633972  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:23.634635  1679 raft_consensus.cc:3060] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:01:23.636435  6863 raft_consensus.cc:740] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:01:23.637573  6863 consensus_queue.cc:260] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.11, Last appended by leader: 11, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:23.638657  6944 log.cc:927] Time spent T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.067s	user 0.001s	sys 0.000s
I20260709 13:01:23.639060  6863 raft_consensus.cc:3060] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:01:23.642863  6643 leader_election.cc:290] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 10 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:23.645924  1676 raft_consensus.cc:1240] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 LEADER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 11. Current term is 12. Ops: []
I20260709 13:01:23.647316  1675 raft_consensus.cc:1240] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 LEADER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 13. Current term is 14. Ops: []
I20260709 13:01:23.647389  6859 leader_election.cc:290] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:23.651271  6884 leader_election.cc:290] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 14 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:23.651947  6904 leader_election.cc:290] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 15 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:23.655923  6932 leader_election.cc:290] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:01:23.657339  1644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:23.569434  6882 consensus_queue.cc:237] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:23.666554  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:23.667241  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:23.667430  1596 raft_consensus.cc:2468] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 11.
I20260709 13:01:23.668438  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:23.669085  1753 raft_consensus.cc:2468] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 13.
I20260709 13:01:23.670050  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:23.668382  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
W20260709 13:01:23.632030  6890 consensus_meta.cc:287] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.607s	user 0.011s	sys 0.003s
I20260709 13:01:23.673628  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:23.679652  2329 raft_consensus.cc:2468] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 12.
I20260709 13:01:23.680094  2332 raft_consensus.cc:2393] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 15: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:01:23.678419  1596 raft_consensus.cc:3060] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:01:23.680413  2333 raft_consensus.cc:1240] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 11. Current term is 12. Ops: []
I20260709 13:01:23.681224  1644 leader_election.cc:304] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:01:23.682181  1673 raft_consensus.cc:2468] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 15.
I20260709 13:01:23.679458  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:23.685635  1679 raft_consensus.cc:2468] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 14.
I20260709 13:01:23.688727  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:23.689318  2336 raft_consensus.cc:3060] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Advancing to term 14
W20260709 13:01:23.707767  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 94057.3us
I20260709 13:01:23.711760  1643 leader_election.cc:304] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:23.714885  1646 leader_election.cc:400] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 13 election: Vote denied by peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693) with higher term. Message: Invalid argument: T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for earlier term 13. Current term is 14.
I20260709 13:01:23.716182  1646 leader_election.cc:304] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
W20260709 13:01:23.719396  1570 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:23.688378  6867 raft_consensus.cc:2804] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:01:23.720990  1646 leader_election.cc:304] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:01:23.721993  6046 raft_consensus.cc:2749] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 13:01:23.725148  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:01:23.726155  1719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:23.729830  1569 leader_election.cc:304] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:23.730580  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:23.731606  1602 raft_consensus.cc:3060] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:01:23.730949  2299 leader_election.cc:304] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
W20260709 13:01:23.934091  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 203202us
I20260709 13:01:23.737223  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:23.738525  1596 raft_consensus.cc:2468] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 15.
I20260709 13:01:23.935727  1748 raft_consensus.cc:3060] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Advancing to term 15
W20260709 13:01:23.732362  6916 consensus_meta.cc:287] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.708s	user 0.003s	sys 0.011s
I20260709 13:01:23.723196  6592 raft_consensus.cc:2749] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:01:23.752270  1569 leader_election.cc:304] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
W20260709 13:01:23.940039  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 187843us
I20260709 13:01:23.773303  6692 consensus_queue.cc:1048] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.048s
I20260709 13:01:23.792107  1600 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 2471) took 3954 ms. Trace:
W20260709 13:01:23.790059  6926 consensus_meta.cc:287] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.572s	user 0.012s	sys 0.000s
I20260709 13:01:23.751628  6752 raft_consensus.cc:2749] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:01:23.764897  6954 consensus_queue.cc:1059] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 11.11, Next index: 12, Last known committed idx: 10, Time since last communication: 0.035s
I20260709 13:01:23.802238  1745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43156 (request call id 1021) took 1104 ms. Trace:
I20260709 13:01:23.801584  1501 catalog_manager.cc:5697] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d reported cstate change: term changed from 6 to 12, leader changed from af520b67945d443b901bbeca37cc5701 (127.0.107.195) to 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194). New cstate: current_term: 12 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 13:01:23.803349  6512 consensus_meta.cc:287] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.514s	user 0.013s	sys 0.000s
I20260709 13:01:23.804646  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:01:23.806677  6945 log.cc:927] Time spent T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.309s	user 0.000s	sys 0.002s
I20260709 13:01:23.810616  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:23.813943  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:23.956972  1598 raft_consensus.cc:2468] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 14.
I20260709 13:01:23.817181  6605 raft_consensus.cc:515] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:23.818538  6706 raft_consensus.cc:515] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:23.953781  1745 rpcz_store.cc:276] 0709 13:01:22.697472 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:22.714083 (+ 16611us) service_pool.cc:224] Handling call
0709 13:01:22.727136 (+ 13053us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:22.934783 (+207647us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:22.934836 (+    53us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:01:22.934903 (+    67us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:01:22.934963 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:23.801867 (+866904us) spinlock_profiling.cc:243] Waited 207 ms on lock 0x7b5400019ad4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:01:23.801943 (+    76us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:23.802150 (+   207us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":476741888}
I20260709 13:01:23.819487  6160 raft_consensus.cc:3055] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 13:01:23.820423  6714 raft_consensus.cc:493] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:01:23.820876  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 251853us
I20260709 13:01:23.821959  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
W20260709 13:01:23.823513  6949 log.cc:927] Time spent T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.203s	user 0.000s	sys 0.002s
I20260709 13:01:23.824733  2336 raft_consensus.cc:2468] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 14.
W20260709 13:01:23.826282  6948 log.cc:927] Time spent T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.172s	user 0.000s	sys 0.001s
I20260709 13:01:23.826558  1752 raft_consensus.cc:1275] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
W20260709 13:01:23.827968  6947 log.cc:927] Time spent T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.201s	user 0.001s	sys 0.000s
I20260709 13:01:23.829320  6865 raft_consensus.cc:3060] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:01:23.832878  6859 consensus_queue.cc:1059] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 13:01:23.833657  6956 consensus_queue.cc:1059] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.064s
I20260709 13:01:23.812371  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:23.714334  6879 raft_consensus.cc:515] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:23.854766  6896 consensus_queue.cc:1048] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 13:01:23.998917  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:23.732894  6867 raft_consensus.cc:697] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 12 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:01:23.848762  6622 raft_consensus.cc:493] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:01:23.866307  6950 log.cc:927] Time spent T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.369s	user 0.003s	sys 0.000s
W20260709 13:01:23.906108  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:23.981969  6954 raft_consensus.cc:3055] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 LEADER]: Stepping down as leader of term 11
I20260709 13:01:23.920425  1749 raft_consensus.cc:1275] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 14 index: 8. (index mismatch)
I20260709 13:01:23.898123  6898 consensus_queue.cc:1059] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 8.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.024s
W20260709 13:01:23.923840  6922 log.cc:927] Time spent T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.993s	user 0.004s	sys 0.009s
I20260709 13:01:23.937184  2298 leader_election.cc:304] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
W20260709 13:01:24.023056  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 86011.5us
I20260709 13:01:23.937743  1643 leader_election.cc:304] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
W20260709 13:01:23.941277  1569 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:24.025580  1569 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:23.727371  6854 raft_consensus.cc:2804] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:01:23.941972  1600 rpcz_store.cc:276] 0709 13:01:19.837685 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:21.876046 (+2038361us) service_pool.cc:224] Handling call
0709 13:01:21.876505 (+   459us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:22.491071 (+614566us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 13:01:22.491111 (+    40us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:01:22.491161 (+    50us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:01:22.491216 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:23.791662 (+1300446us) spinlock_profiling.cc:243] Waited 614 ms on lock 0x7b540029aed4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:01:23.791762 (+   100us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:23.792005 (+   243us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1833,"raft.run_cpu_time_us":100,"raft.run_wall_time_us":101,"spinlock_wait_cycles":1412365056}
I20260709 13:01:23.939517  6939 raft_consensus.cc:2749] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
W20260709 13:01:23.948110  6957 log.cc:927] Time spent T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.057s	user 0.000s	sys 0.001s
I20260709 13:01:23.959846  1718 leader_election.cc:304] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:23.960202  1748 raft_consensus.cc:2468] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 15.
W20260709 13:01:23.960675  1751 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.551s	user 0.063s	sys 0.097s
W20260709 13:01:24.200150  1751 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.790s	user 0.063s	sys 0.097s
I20260709 13:01:23.962479  1747 raft_consensus.cc:3060] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:01:23.963797  1717 leader_election.cc:400] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 12 pre-election: Vote denied by peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445) with higher term. Message: Invalid argument: T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 12. Current term is 13.
W20260709 13:01:23.934768  6959 log.cc:927] Time spent T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.061s	user 0.000s	sys 0.000s
I20260709 13:01:23.967095  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:24.201748  1717 leader_election.cc:403] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 12 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:01:24.202278  2335 raft_consensus.cc:2393] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 14: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
W20260709 13:01:23.968003  1570 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:23.949921  6884 raft_consensus.cc:2804] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:01:23.993862  2331 raft_consensus.cc:2393] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 11: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:01:23.996169  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:23.963181  6160 raft_consensus.cc:740] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:01:23.997169  6879 leader_election.cc:290] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 12 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:23.945029  6875 consensus_queue.cc:1048] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.039s
I20260709 13:01:23.752200  6915 leader_election.cc:290] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:24.014614  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:24.006958  6592 raft_consensus.cc:493] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:01:23.953279  6926 raft_consensus.cc:515] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:23.962455  6512 raft_consensus.cc:515] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:23.966467  6714 raft_consensus.cc:515] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:01:24.012848  6821 consensus_meta.cc:287] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.864s	user 0.016s	sys 0.000s
I20260709 13:01:24.016537  6954 raft_consensus.cc:740] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:01:24.066253  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:24.016009  6859 consensus_queue.cc:1243] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer 816fadcea8fc4893882f5f9b9f22a03d log is divergent from this leader: its last log entry 14.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
W20260709 13:01:24.153083  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 126778us
I20260709 13:01:24.250579  6592 raft_consensus.cc:515] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:24.063639  6643 raft_consensus.cc:2804] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Leader election won for term 15
W20260709 13:01:24.101390  1645 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:24.116729  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 105681us
I20260709 13:01:24.266883  6643 raft_consensus.cc:697] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 15 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:01:24.015306  6345 consensus_queue.cc:1048] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.053s
I20260709 13:01:24.267997  6643 consensus_queue.cc:237] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.10, Last appended by leader: 10, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:24.269418  6960 log.cc:927] Time spent T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.097s	user 0.000s	sys 0.000s
I20260709 13:01:24.048743  6752 raft_consensus.cc:493] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:24.270102  6752 raft_consensus.cc:515] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:24.272244  6963 raft_consensus.cc:3055] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 LEADER]: Stepping down as leader of term 13
I20260709 13:01:24.272430  6752 leader_election.cc:290] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:24.272725  6963 raft_consensus.cc:740] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:01:24.273900  6915 raft_consensus.cc:493] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:24.274330  1751 raft_consensus.cc:1240] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 15. Current term is 16. Ops: []
I20260709 13:01:24.274523  6915 raft_consensus.cc:515] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } }
I20260709 13:01:24.275842  1751 raft_consensus.cc:1275] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 9 index: 10. Preceding OpId from leader: term: 11 index: 11. (index mismatch)
I20260709 13:01:24.276604  6915 leader_election.cc:290] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 13:01:24.100841  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:24.239197  6908 raft_consensus.cc:493] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:24.277190  1751 raft_consensus.cc:1275] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 14 index: 7. (index mismatch)
I20260709 13:01:24.277431  6908 raft_consensus.cc:515] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:24.279590  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 317093us
W20260709 13:01:24.280380  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 342677us
I20260709 13:01:24.239425  6909 raft_consensus.cc:2804] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 13:01:24.153801  6956 consensus_queue.cc:1243] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer 816fadcea8fc4893882f5f9b9f22a03d log is divergent from this leader: its last log entry 12.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260709 13:01:24.209336  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 14 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:24.291813  1748 raft_consensus.cc:2393] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 14: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:01:24.085361  6692 raft_consensus.cc:2804] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Leader pre-election won for term 15
W20260709 13:01:24.227998  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 268204us
I20260709 13:01:24.292765  1718 leader_election.cc:304] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:24.229929  1680 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57092 (request call id 1145) took 1096 ms. Trace:
I20260709 13:01:24.243878  1602 raft_consensus.cc:2468] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 10.
I20260709 13:01:24.207093  6914 raft_consensus.cc:493] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:01:24.222183  6958 log.cc:927] Time spent T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.269s	user 0.000s	sys 0.001s
I20260709 13:01:24.257613  1747 raft_consensus.cc:2468] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 10.
I20260709 13:01:24.259052  6638 raft_consensus.cc:3060] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:01:24.259217  6160 consensus_queue.cc:260] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:24.261121  6714 leader_election.cc:290] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 13:01:24.261646  6946 log.cc:927] Time spent T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.604s	user 0.005s	sys 0.000s
I20260709 13:01:24.261559  6821 raft_consensus.cc:515] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:24.263176  6968 log.cc:927] Time spent T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.199s	user 0.000s	sys 0.000s
I20260709 13:01:24.262590  6954 consensus_queue.cc:260] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 11.11, Last appended by leader: 11, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:24.273813  6963 consensus_queue.cc:260] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.8, Last appended by leader: 8, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:01:24.331278  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 48995.9us
I20260709 13:01:24.306577  6932 raft_consensus.cc:2804] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:01:24.067145  6854 raft_consensus.cc:697] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:01:24.017580  6622 raft_consensus.cc:515] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:24.328168  6926 leader_election.cc:290] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:24.348791  6622 leader_election.cc:290] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:01:24.128441  1644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:24.348943  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:24.349733  1748 raft_consensus.cc:3060] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Advancing to term 14
W20260709 13:01:24.285477  1717 leader_election.cc:336] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 2.117s (SENT)
I20260709 13:01:24.351141  6963 raft_consensus.cc:3060] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Advancing to term 14
W20260709 13:01:24.351907  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 66458.2us
W20260709 13:01:24.203161  6964 log.cc:927] Time spent T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.116s	user 0.000s	sys 0.001s
W20260709 13:01:24.286542  1571 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:24.221611  6940 raft_consensus.cc:493] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:24.231595  6927 raft_consensus.cc:493] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:01:24.301879  1570 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:24.302714  2330 raft_consensus.cc:2393] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 14: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:01:24.310304  1569 leader_election.cc:304] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 13:01:24.314682  2334 raft_consensus.cc:1240] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 12 LEADER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 11. Current term is 12. Ops: []
I20260709 13:01:24.268002  6592 leader_election.cc:290] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:24.294695  1680 rpcz_store.cc:276] 0709 13:01:23.133742 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:23.140242 (+  6500us) service_pool.cc:224] Handling call
0709 13:01:23.140614 (+   372us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:01:23.378186 (+237572us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:23.378222 (+    36us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:01:23.400631 (+ 22409us) log.cc:844] Serialized 52 byte log entry
0709 13:01:23.459975 (+ 59344us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:01:23.460037 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:23.832741 (+372704us) spinlock_profiling.cc:243] Waited 218 ms on lock 0x7b5400164254. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:01:23.832799 (+    58us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:23.865305 (+ 32506us) spinlock_profiling.cc:243] Waited 8.96 ms on lock 0x7b4400022b4c. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f4262f9bc85 00007f4262f9c6a2 00007f4265d0e1db 00007f4265d09aaf 00007f4265cfd86e 00007f4265cfcbb6 00007f4265cfc7d8 00007f4265d06221 00007f4265d061d9 00007f4265d06169
0709 13:01:24.229626 (+364321us) raft_consensus.cc:1685] finished
0709 13:01:24.229656 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:24.229848 (+   192us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4289,"raft.queue_time_us":4501,"raft.run_cpu_time_us":168,"raft.run_wall_time_us":12297,"spinlock_wait_cycles":522031872,"thread_start_us":58905,"threads_started":1,"wal-append.queue_time_us":88200}
I20260709 13:01:24.330971  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:24.279713  6908 leader_election.cc:290] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:01:24.403821  1571 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:24.301462  6692 raft_consensus.cc:493] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Starting leader election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:01:24.311940  6909 raft_consensus.cc:697] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 15 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:01:24.339524  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:24.339833  6854 consensus_queue.cc:237] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:24.344818  6706 leader_election.cc:290] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 11 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 13:01:24.475903  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 165707us
I20260709 13:01:24.388105  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:24.391338  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:24.376472  1754 raft_consensus.cc:1240] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 13. Current term is 14. Ops: []
I20260709 13:01:24.397557  1748 raft_consensus.cc:2468] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 14.
I20260709 13:01:24.384713  1671 raft_consensus.cc:1240] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 LEADER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 11. Current term is 12. Ops: []
I20260709 13:01:24.222635  6867 consensus_queue.cc:237] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.11, Last appended by leader: 11, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:24.309598  6914 raft_consensus.cc:515] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:24.332652  6605 leader_election.cc:290] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 14 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:24.444957  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:24.502718  1601 raft_consensus.cc:3060] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:01:24.445835  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:24.454134  1600 raft_consensus.cc:1275] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20260709 13:01:24.503726  1602 raft_consensus.cc:3060] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Advancing to term 11
W20260709 13:01:24.483194  6969 log.cc:927] Time spent T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.245s	user 0.000s	sys 0.001s
W20260709 13:01:24.486707  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 108538us
W20260709 13:01:24.489214  6965 log.cc:927] Time spent T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.084s	user 0.000s	sys 0.000s
I20260709 13:01:24.490873  6980 raft_consensus.cc:2804] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:01:24.453642  6932 raft_consensus.cc:697] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 14 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:01:24.491770  6638 raft_consensus.cc:2749] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Leader pre-election lost for term 12. Reason: Vote denied by peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445) with higher term. Message: Invalid argument: T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 12. Current term is 13.
I20260709 13:01:24.071969   431 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:01:24.509064  6046 consensus_queue.cc:1048] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 13:01:24.450241  6692 raft_consensus.cc:3060] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:01:24.510062  1501 catalog_manager.cc:5697] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d reported cstate change: term changed from 9 to 13. New cstate: current_term: 13 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 13:01:24.514483  1602 raft_consensus.cc:2468] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 11.
I20260709 13:01:24.521955  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:24.523651  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:24.524317  1602 raft_consensus.cc:2376] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 11. Re-sending same reply.
I20260709 13:01:24.403378  6884 raft_consensus.cc:697] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:01:24.529215  6046 consensus_queue.cc:1048] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 13:01:24.534576  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:24.535226  1597 raft_consensus.cc:1275] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20260709 13:01:24.535992  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:24.536521  2331 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39726 (request call id 1220) took 1063 ms. Trace:
I20260709 13:01:24.536801  1598 raft_consensus.cc:2393] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 10: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:01:24.536953  2331 rpcz_store.cc:276] 0709 13:01:23.472637 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:23.472799 (+   162us) service_pool.cc:224] Handling call
0709 13:01:24.536300 (+1063501us) spinlock_profiling.cc:243] Waited 491 ms on lock 0x7b5400605ed4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:01:24.536447 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1130395904}
I20260709 13:01:24.539517  6905 consensus_queue.cc:1048] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 13:01:24.542876  6984 raft_consensus.cc:493] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:24.542953  6975 consensus_queue.cc:1048] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 13:01:24.543699  6984 raft_consensus.cc:515] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:24.548913  2301 leader_election.cc:304] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:01:24.551002  6882 raft_consensus.cc:3055] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:01:24.551448  6882 raft_consensus.cc:740] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
W20260709 13:01:24.553601  6865 consensus_meta.cc:287] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.562s	user 0.010s	sys 0.001s
I20260709 13:01:24.516639  6885 leader_election.cc:290] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:01:24.334941  6859 consensus_queue.cc:1175] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "3b879799899a499c8d409ee9f1277cd1" responder_term: 12 status { last_received { term: 9 index: 10 } last_committed_idx: 10 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 11. Current term is 12. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260709 13:01:24.475417  6908 raft_consensus.cc:493] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:24.570137  1601 raft_consensus.cc:2468] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 14.
I20260709 13:01:24.536401  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:24.561215  1571 leader_election.cc:304] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:01:24.569177  6980 raft_consensus.cc:493] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
W20260709 13:01:24.577047  6961 log.cc:927] Time spent T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.651s	user 0.003s	sys 0.000s
I20260709 13:01:24.578382  6977 raft_consensus.cc:493] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:24.578202  6962 raft_consensus.cc:3055] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 LEADER]: Stepping down as leader of term 11
I20260709 13:01:24.580339  6859 raft_consensus.cc:2749] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:01:24.581192  6932 consensus_queue.cc:237] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:24.582299  6914 leader_election.cc:290] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:24.582573  6512 leader_election.cc:290] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 14 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:24.626636  1604 raft_consensus.cc:1275] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260709 13:01:24.628703  2330 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39726 (request call id 1212) took 1463 ms. Trace:
I20260709 13:01:24.629113  2330 rpcz_store.cc:276] 0709 13:01:23.165398 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:23.165553 (+   155us) service_pool.cc:224] Handling call
0709 13:01:24.628478 (+1462925us) spinlock_profiling.cc:243] Waited 1.11 s on lock 0x7b54007d2654. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:01:24.628625 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2545699200}
W20260709 13:01:24.583472  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:24.449934  6927 raft_consensus.cc:515] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:24.584179  1754 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55848 (request call id 2629) took 1162 ms. Trace:
I20260709 13:01:24.576005  6983 consensus_queue.cc:1048] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.008s
I20260709 13:01:24.584940  1571 leader_election.cc:304] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
W20260709 13:01:24.595793  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 46970.1us
I20260709 13:01:24.595326  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:24.607192  2298 leader_election.cc:304] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:01:24.613600  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:24.435866  6940 raft_consensus.cc:515] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:24.605448  6980 raft_consensus.cc:3060] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:01:24.605324  6605 raft_consensus.cc:2749] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 13:01:24.636363  1754 rpcz_store.cc:276] 0709 13:01:23.426883 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:23.487117 (+ 60234us) service_pool.cc:224] Handling call
0709 13:01:23.529662 (+ 42545us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:24.488194 (+958532us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:24.583890 (+ 95696us) spinlock_profiling.cc:243] Waited 846 ms on lock 0x7b54000e4054. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:01:24.584105 (+   215us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1946563456}
W20260709 13:01:24.591114  6971 log.cc:927] Time spent T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.383s	user 0.004s	sys 0.000s
W20260709 13:01:24.632858  6863 consensus_meta.cc:287] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.979s	user 0.010s	sys 0.007s
I20260709 13:01:24.585196  6905 consensus_queue.cc:1048] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.055s
I20260709 13:01:24.616671  6962 raft_consensus.cc:740] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
W20260709 13:01:24.657256  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 304667us
I20260709 13:01:24.653735  2301 leader_election.cc:304] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:24.589586  6908 raft_consensus.cc:515] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:24.674382  1717 leader_election.cc:336] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 1.500s (SENT)
I20260709 13:01:24.675820  1717 leader_election.cc:304] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:24.755045  1605 raft_consensus.cc:1275] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 13 index: 11. (term mismatch)
I20260709 13:01:24.757191  1605 pending_rounds.cc:85] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a: Aborting all ops after (but not including) 10
I20260709 13:01:24.758075  1605 pending_rounds.cc:107] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a: Aborting uncommitted NO_OP operation due to leader change: 12.11
I20260709 13:01:24.758874  1605 raft_consensus.cc:2889] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 13:01:24.756570  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 82251.9us
I20260709 13:01:24.648221  6859 raft_consensus.cc:2804] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Leader pre-election won for term 14
W20260709 13:01:24.769851  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 116186us
W20260709 13:01:24.770840  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 104941us
W20260709 13:01:24.772877  1718 leader_election.cc:336] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 14 pre-election: RPC error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Timed out: RequestConsensusVote RPC to 127.0.107.194:46261 timed out after 1.650s (SENT)
W20260709 13:01:24.777149  1719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:24.652949  6699 raft_consensus.cc:2804] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 13:01:24.591809  6884 consensus_queue.cc:237] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:24.790027  1750 raft_consensus.cc:2468] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 13.
I20260709 13:01:24.644121  6916 consensus_queue.cc:1059] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.032s
W20260709 13:01:24.782584  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 197710us
I20260709 13:01:24.783715  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:24.795061  1671 raft_consensus.cc:3060] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:01:24.779767  2298 leader_election.cc:304] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:01:24.657390  6984 leader_election.cc:290] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:24.798005  1680 raft_consensus.cc:1240] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 12. Current term is 13. Ops: []
W20260709 13:01:24.801728  1717 leader_election.cc:336] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 1.512s (SENT)
I20260709 13:01:24.802244  1717 leader_election.cc:304] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:24.801605  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:24.651285  6512 consensus_queue.cc:1059] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 15.12, Next index: 13, Last known committed idx: 9, Time since last communication: 0.130s
I20260709 13:01:24.804046  1672 raft_consensus.cc:2468] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 14.
I20260709 13:01:24.812618  1673 raft_consensus.cc:1275] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260709 13:01:24.806883  1571 leader_election.cc:304] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:01:24.621131  6909 consensus_queue.cc:237] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.10, Last appended by leader: 10, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:24.782205  6859 raft_consensus.cc:493] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:24.779134  6485 raft_consensus.cc:2749] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260709 13:01:24.816943  1717 leader_election.cc:336] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 2.206s (SENT)
I20260709 13:01:24.625588  6977 raft_consensus.cc:515] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:01:24.804373  6961 log.cc:927] Time spent T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.067s	user 0.001s	sys 0.000s
I20260709 13:01:24.835085  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:24.836660  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:24.822278  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:24.783893  6408 raft_consensus.cc:2804] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:01:24.814270  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:01:24.913445  6986 log.cc:927] Time spent T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.335s	user 0.003s	sys 0.000s
I20260709 13:01:24.913923  6908 leader_election.cc:290] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:01:24.831800  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 52111.6us
W20260709 13:01:24.831705  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 59577.6us
I20260709 13:01:24.833912  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:24.853799  1678 raft_consensus.cc:1240] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 9. Current term is 10. Ops: [9.6-9.6]
I20260709 13:01:24.838042  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:24.811731  6956 consensus_queue.cc:1059] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 11.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.051s
I20260709 13:01:24.815315  6821 leader_election.cc:290] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 14 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:01:24.855084  1717 leader_election.cc:336] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 13 election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 2.238s (SENT)
I20260709 13:01:24.871230  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:24.859136  6984 raft_consensus.cc:2749] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:01:24.861531  6983 raft_consensus.cc:493] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:24.935503  1746 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43156 (request call id 1026) took 1333 ms. Trace:
W20260709 13:01:24.936381  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 81334.7us
I20260709 13:01:24.875456  6954 raft_consensus.cc:3060] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:01:24.827787  6916 raft_consensus.cc:2749] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:01:24.879221  6940 leader_election.cc:290] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:01:24.879546  6967 log.cc:927] Time spent T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.404s	user 0.004s	sys 0.014s
I20260709 13:01:24.879845  6962 consensus_queue.cc:260] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 11.6, Last appended by leader: 6, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:01:24.893751  6982 log.cc:927] Time spent T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.303s	user 0.000s	sys 0.000s
I20260709 13:01:24.814637  6605 raft_consensus.cc:493] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:24.897665  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:24.836272  6859 raft_consensus.cc:3060] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:01:24.904261  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
W20260709 13:01:24.980063  1718 leader_election.cc:336] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 14 pre-election: RPC error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Timed out: RequestConsensusVote RPC to 127.0.107.194:46261 timed out after 1.931s (SENT)
W20260709 13:01:24.905544  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 98788.6us
I20260709 13:01:24.940148  6908 raft_consensus.cc:493] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:24.912195  6927 leader_election.cc:290] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:24.861892  6600 raft_consensus.cc:2804] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Leader election won for term 11
W20260709 13:01:24.937609  1717 leader_election.cc:336] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 2.245s (SENT)
I20260709 13:01:25.001699  1717 leader_election.cc:304] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:24.932838  1671 raft_consensus.cc:2468] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 11.
I20260709 13:01:24.858990  6699 raft_consensus.cc:697] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 10 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:01:24.936096  1746 rpcz_store.cc:276] 0709 13:01:23.601973 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:24.244883 (+642910us) service_pool.cc:224] Handling call
0709 13:01:24.296839 (+ 51956us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:24.326951 (+ 30112us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:24.326995 (+    44us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:01:24.347920 (+ 20925us) log.cc:844] Serialized 26 byte log entry
0709 13:01:24.501714 (+153794us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:01:24.501766 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:24.501845 (+    79us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:24.913034 (+411189us) spinlock_profiling.cc:243] Waited 5.62 ms on lock 0x7b240024b1c8. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f4262efb9c4 00007f4262ef9acb 00007f42630b4dc9 00007f42630b4a55 00007f42630b409d 00007f4265ce610e 00007f4265cfd88c 00007f4265cfcbb6 00007f4265cfc7d8 00007f4265d06221
0709 13:01:24.929049 (+ 16015us) raft_consensus.cc:1685] finished
0709 13:01:24.929083 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:24.935396 (+  6313us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":65435,"raft.queue_time_us":13163,"raft.run_cpu_time_us":131,"raft.run_wall_time_us":2568,"spinlock_wait_cycles":12933888,"thread_start_us":153333,"threads_started":1,"wal-append.queue_time_us":229612}
I20260709 13:01:24.979132  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:24.978693  1501 catalog_manager.cc:5697] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d reported cstate change: term changed from 11 to 13. New cstate: current_term: 13 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } }
I20260709 13:01:24.811017  6875 consensus_queue.cc:1048] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.052s
W20260709 13:01:24.982784  1570 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:25.009548  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:25.010603  6962 raft_consensus.cc:3060] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Advancing to term 12
W20260709 13:01:25.010851  6994 log.cc:927] Time spent T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.075s	user 0.001s	sys 0.000s
W20260709 13:01:24.985589  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 68615.9us
W20260709 13:01:24.952437  6963 consensus_meta.cc:287] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.601s	user 0.012s	sys 0.002s
I20260709 13:01:25.003095  1571 leader_election.cc:304] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:01:25.003806  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:24.909953  6980 raft_consensus.cc:515] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:25.028729  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 91158.7us
W20260709 13:01:25.039351  1717 leader_election.cc:336] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 2.407s (SENT)
I20260709 13:01:25.039907  1717 leader_election.cc:304] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 13:01:25.041706  1717 leader_election.cc:336] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 14 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 3.101s (SENT)
I20260709 13:01:25.042994  1717 leader_election.cc:304] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:25.044332  6485 raft_consensus.cc:2749] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
W20260709 13:01:25.042361  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 124034us
I20260709 13:01:25.044409  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:25.045455  6699 consensus_queue.cc:237] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:25.047006  1596 raft_consensus.cc:1275] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 15 index: 11. (index mismatch)
I20260709 13:01:25.055507  1753 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55848 (request call id 2634) took 1623 ms. Trace:
I20260709 13:01:25.055931  1753 rpcz_store.cc:276] 0709 13:01:23.432068 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:23.675629 (+243561us) service_pool.cc:224] Handling call
0709 13:01:23.676032 (+   403us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:23.823980 (+147948us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:23.824015 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:01:23.824572 (+   557us) log.cc:844] Serialized 26 byte log entry
0709 13:01:24.020414 (+195842us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:01:24.020469 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:24.020551 (+    82us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:24.878824 (+858273us) spinlock_profiling.cc:243] Waited 24.7 ms on lock 0x7b4400022b4c. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f4262f9bc85 00007f4262f9c6a2 00007f4265d0e1db 00007f4265d09aaf 00007f4265cfd86e 00007f4265cfcbb6 00007f4265cfc7d8 00007f4265d06221 00007f4265d061d9 00007f4265d06169
0709 13:01:25.055166 (+176342us) raft_consensus.cc:1685] finished
0709 13:01:25.055193 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:25.055436 (+   243us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":143506,"raft.queue_time_us":143741,"raft.run_cpu_time_us":75,"raft.run_wall_time_us":76,"spinlock_wait_cycles":56917376,"thread_start_us":195399,"threads_started":1,"wal-append.queue_time_us":490106}
I20260709 13:01:25.027835  6600 raft_consensus.cc:697] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 11 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
W20260709 13:01:25.058100  6966 log.cc:927] Time spent T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.737s	user 0.002s	sys 0.000s
I20260709 13:01:25.058687  6600 consensus_queue.cc:237] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 9.10, Last appended by leader: 10, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:24.987659  6605 raft_consensus.cc:515] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:25.059666  6970 log.cc:927] Time spent T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.568s	user 0.001s	sys 0.000s
I20260709 13:01:25.061498  6605 leader_election.cc:290] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:25.042802  6927 raft_consensus.cc:2749] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:01:25.062979  1603 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 1206) took 1399 ms. Trace:
I20260709 13:01:25.063359  1603 rpcz_store.cc:276] 0709 13:01:23.663310 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:23.867027 (+203717us) service_pool.cc:224] Handling call
0709 13:01:23.867438 (+   411us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:23.867909 (+   471us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:23.867948 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:01:23.868650 (+   702us) log.cc:844] Serialized 26 byte log entry
0709 13:01:24.151443 (+282793us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:01:24.151495 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:24.151612 (+   117us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:25.059011 (+907399us) spinlock_profiling.cc:243] Waited 4.8 ms on lock 0x7b4400022b4c. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f4262f9bc85 00007f4262f9c6a2 00007f4265d0e1db 00007f4265d09aaf 00007f4265cfd86e 00007f4265cfcbb6 00007f4265cfc7d8 00007f4265d06221 00007f4265d061d9 00007f4265d06169
0709 13:01:25.062250 (+  3239us) raft_consensus.cc:1685] finished
0709 13:01:25.062275 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:25.062902 (+   627us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":204,"raft.queue_time_us":15861,"raft.run_cpu_time_us":130,"raft.run_wall_time_us":904,"spinlock_wait_cycles":11046144,"thread_start_us":204944,"threads_started":1,"wal-append.queue_time_us":452637}
I20260709 13:01:25.040895  6821 raft_consensus.cc:2749] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:01:25.045773  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:25.066457  1601 raft_consensus.cc:2468] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 12.
I20260709 13:01:25.068051  6980 leader_election.cc:290] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 15 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:01:25.068065  6692 consensus_meta.cc:287] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.530s	user 0.007s	sys 0.004s
I20260709 13:01:25.068626  6863 raft_consensus.cc:3055] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 LEADER]: Stepping down as leader of term 11
W20260709 13:01:25.069052  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 86404.8us
I20260709 13:01:25.069181  2299 leader_election.cc:304] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
W20260709 13:01:25.069886  1717 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 3.370s (SENT) [suppressed 45 similar messages]
I20260709 13:01:25.070399  1571 leader_election.cc:304] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:01:24.960188  6940 consensus_queue.cc:1048] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.062s
W20260709 13:01:25.071715  1717 leader_election.cc:336] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 14 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 1.931s (SENT)
I20260709 13:01:25.073937  1717 leader_election.cc:304] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 13:01:25.075560  1717 leader_election.cc:336] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 14 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 2.150s (SENT)
I20260709 13:01:25.075992  1717 leader_election.cc:304] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:25.004303  6908 raft_consensus.cc:515] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:24.947598  6983 raft_consensus.cc:515] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:25.091702  1717 leader_election.cc:336] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 2.249s (SENT)
I20260709 13:01:25.092298  1717 leader_election.cc:304] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 13:01:25.093750  1717 leader_election.cc:336] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 2.303s (SENT)
W20260709 13:01:25.099056  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:25.104450  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:25.109654  1644 leader_election.cc:304] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
W20260709 13:01:25.110724  1717 leader_election.cc:336] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 12 election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 3.134s (SENT)
I20260709 13:01:25.111320  1717 leader_election.cc:304] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:25.069077  6863 raft_consensus.cc:740] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:01:25.080561  6821 raft_consensus.cc:2749] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:01:25.084249  6956 raft_consensus.cc:2749] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 13:01:25.126300  6909 raft_consensus.cc:2749] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 13:01:25.095535  6878 raft_consensus.cc:2749] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:01:25.087647  6512 raft_consensus.cc:493] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:01:25.094923  6485 raft_consensus.cc:2749] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:01:25.095672  6605 raft_consensus.cc:493] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:25.069083  6692 raft_consensus.cc:515] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:25.152072  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:25.152634  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 13 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:25.158093  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:25.159046  2337 raft_consensus.cc:2468] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 14.
I20260709 13:01:25.161073  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
W20260709 13:01:25.172464  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 97103.4us
W20260709 13:01:25.174500  6991 log.cc:927] Time spent T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.079s	user 0.000s	sys 0.000s
I20260709 13:01:25.184258  1605 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40154 (request call id 2412) took 1755 ms. Trace:
I20260709 13:01:25.184715  1605 rpcz_store.cc:276] 0709 13:01:23.429278 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:23.815489 (+386211us) service_pool.cc:224] Handling call
0709 13:01:23.815965 (+   476us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:24.759978 (+944013us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:25.179332 (+419354us) spinlock_profiling.cc:243] Waited 862 ms on lock 0x7b5400215cd4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:01:25.184187 (+  4855us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1982593920}
W20260709 13:01:25.181805  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 111488us
I20260709 13:01:25.186481  1571 leader_election.cc:304] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:25.129508  6997 raft_consensus.cc:2749] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20260709 13:01:25.189920  6995 log.cc:927] Time spent T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.119s	user 0.001s	sys 0.000s
I20260709 13:01:25.190102  6863 consensus_queue.cc:260] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 11.8, Last appended by leader: 8, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:25.191648  6863 raft_consensus.cc:3060] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:01:25.192715  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:25.193871  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:25.194640  1675 raft_consensus.cc:2393] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 14: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:01:25.195380  1674 raft_consensus.cc:1275] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 13 index: 7. (index mismatch)
I20260709 13:01:25.196781  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:25.197592  6977 leader_election.cc:290] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:25.198814  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:25.200891  1678 raft_consensus.cc:1240] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 10. Current term is 11. Ops: [10.10-10.10]
I20260709 13:01:25.108814  6345 consensus_queue.cc:1059] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 13.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.045s
I20260709 13:01:25.203938  6345 raft_consensus.cc:3055] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 13 LEADER]: Stepping down as leader of term 13
I20260709 13:01:25.204329  6345 raft_consensus.cc:740] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:01:25.200945  6692 leader_election.cc:290] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 15 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:25.195742  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:25.214207  1677 raft_consensus.cc:2468] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 10.
I20260709 13:01:25.195431  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:25.180006  2298 leader_election.cc:304] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:01:25.198173  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:25.218114  1674 raft_consensus.cc:2410] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for term 14 because replica has last-logged OpId of term: 13 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 7.
I20260709 13:01:25.207967  1673 raft_consensus.cc:1240] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 14. Current term is 15. Ops: []
W20260709 13:01:25.219174  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 109585us
W20260709 13:01:25.213483  1643 leader_election.cc:336] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 11 election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 1.940s (SENT)
I20260709 13:01:25.223367  1672 raft_consensus.cc:2468] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 15.
W20260709 13:01:25.227627  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 47780.5us
I20260709 13:01:25.117826  7000 consensus_queue.cc:1048] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.105s
I20260709 13:01:25.233939  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 12 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:25.202770  7007 consensus_queue.cc:1048] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.025s
I20260709 13:01:25.149801  6512 raft_consensus.cc:515] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:25.243685  6512 leader_election.cc:290] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:25.244287  6512 raft_consensus.cc:2804] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:01:25.244673  6512 raft_consensus.cc:493] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:25.245018  6512 raft_consensus.cc:3060] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Advancing to term 13
W20260709 13:01:25.252401  1719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:25.259753  6927 raft_consensus.cc:493] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:25.260365  6927 raft_consensus.cc:515] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:25.260464  6859 raft_consensus.cc:515] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:25.205226  6345 consensus_queue.cc:260] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.11, Last appended by leader: 11, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:25.263473  6345 raft_consensus.cc:3060] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:01:25.265975  6513 raft_consensus.cc:3055] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 13:01:25.266207  7008 raft_consensus.cc:2804] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Leader pre-election won for term 15
W20260709 13:01:25.268486  7003 log.cc:927] Time spent T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.053s	user 0.000s	sys 0.000s
W20260709 13:01:25.268780  6867 consensus_queue.cc:1175] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "3b879799899a499c8d409ee9f1277cd1" responder_term: 13 status { last_received { term: 13 index: 11 } last_committed_idx: 11 last_received_current_leader { term: 13 index: 11 } }
W20260709 13:01:25.270085  7006 log.cc:927] Time spent T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.088s	user 0.000s	sys 0.000s
W20260709 13:01:25.274576  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 61117.6us
I20260709 13:01:25.275061  6160 raft_consensus.cc:3060] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:01:25.282980  6859 leader_election.cc:290] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 14 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:25.312044  1750 raft_consensus.cc:1275] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20260709 13:01:25.313778  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:25.317981  1752 raft_consensus.cc:2393] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 14: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:01:25.162904  6605 raft_consensus.cc:515] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:25.316461  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:25.329478  1750 raft_consensus.cc:1275] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 15 index: 11. (index mismatch)
I20260709 13:01:25.338712  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:25.288054  6513 raft_consensus.cc:740] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:01:25.340601  1753 raft_consensus.cc:1275] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
I20260709 13:01:25.326995  6821 raft_consensus.cc:493] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:25.344383  6821 raft_consensus.cc:515] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:25.314716  6638 raft_consensus.cc:493] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:25.316681  7008 raft_consensus.cc:493] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:25.359333  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:25.362390  2301 leader_election.cc:304] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:01:25.390777  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:25.392066  1751 raft_consensus.cc:1275] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20260709 13:01:25.401414  1753 raft_consensus.cc:1240] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 15. Current term is 16. Ops: []
I20260709 13:01:25.352142  6927 leader_election.cc:290] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:25.358047  7009 raft_consensus.cc:493] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:25.367687  7008 raft_consensus.cc:3060] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:01:25.367180  6905 consensus_queue.cc:1048] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.054s
W20260709 13:01:25.423974  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 61656.1us
W20260709 13:01:25.427799  7001 log.cc:927] Time spent T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.075s	user 0.000s	sys 0.000s
I20260709 13:01:25.430912  1747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55848 (request call id 2646) took 1161 ms. Trace:
I20260709 13:01:25.431015  7015 raft_consensus.cc:2749] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 13:01:25.431330  1747 rpcz_store.cc:276] 0709 13:01:24.269692 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:24.308059 (+ 38367us) service_pool.cc:224] Handling call
0709 13:01:24.322703 (+ 14644us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:24.541581 (+218878us) spinlock_profiling.cc:243] Waited 19.3 ms on lock 0x7b4800121ed8. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426b08baa1 00007f426b082b37 00007f426b0829ca 00007f426b17a26f 00007f426b179f37 00007f426b18e55c 00007f426b18e508 00007f426b18e488 00007f426b18e443 00007f426b18d6ec 00007f426a0b7a96
0709 13:01:24.776469 (+234888us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:24.776521 (+    52us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:01:24.808786 (+ 32265us) log.cc:844] Serialized 26 byte log entry
0709 13:01:25.037846 (+229060us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:01:25.037899 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:25.038917 (+  1018us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:25.430509 (+391592us) raft_consensus.cc:1685] finished
0709 13:01:25.430537 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:25.430832 (+   295us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":71523,"raft.queue_time_us":112505,"raft.run_cpu_time_us":2251,"raft.run_wall_time_us":97424,"spinlock_wait_cycles":44310400,"thread_start_us":536526,"threads_started":2,"wal-append.queue_time_us":371729}
W20260709 13:01:25.420392  6876 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.513s	user 0.006s	sys 0.012s
W20260709 13:01:25.433606  6876 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.527s	user 0.006s	sys 0.012s
I20260709 13:01:25.440678  6592 raft_consensus.cc:493] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:25.441303  6592 raft_consensus.cc:515] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:25.443966  6605 leader_election.cc:290] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:25.446969  6983 leader_election.cc:290] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:25.365968  6638 raft_consensus.cc:515] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:25.451217  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 97353us
I20260709 13:01:25.414175  7014 raft_consensus.cc:493] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:01:25.459333  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 142621us
I20260709 13:01:25.461567  1753 raft_consensus.cc:1240] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 14 LEADER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 13. Current term is 14. Ops: []
I20260709 13:01:25.409050  6408 raft_consensus.cc:697] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 14 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:01:25.461766  1749 raft_consensus.cc:2468] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 12.
W20260709 13:01:25.460814  1643 leader_election.cc:336] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 15 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 2.074s (SENT)
I20260709 13:01:25.464206  1643 leader_election.cc:304] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 13:01:25.464332  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:25.473253  6046 raft_consensus.cc:2749] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:01:25.476334  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:25.477080  1752 raft_consensus.cc:2468] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 13.
I20260709 13:01:25.480556  6985 consensus_queue.cc:1048] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 13:01:25.418414  6513 consensus_queue.cc:260] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:25.487208  6821 leader_election.cc:290] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:25.488082  6513 raft_consensus.cc:3060] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:01:25.488461  6972 consensus_queue.cc:1048] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 13:01:25.460976  6908 leader_election.cc:290] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:25.490624  6972 consensus_queue.cc:1059] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 13:01:25.491356  6972 consensus_queue.cc:1243] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer af520b67945d443b901bbeca37cc5701 log is divergent from this leader: its last log entry 14.12 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 11
I20260709 13:01:25.491995  6983 raft_consensus.cc:493] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:25.492604  6983 raft_consensus.cc:515] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:25.416152  7009 raft_consensus.cc:515] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:25.496484  7009 leader_election.cc:290] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:25.498329  6638 leader_election.cc:290] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:25.499220  6408 consensus_queue.cc:237] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.9, Last appended by leader: 9, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:25.460299  6605 raft_consensus.cc:493] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:01:25.524004  6914 raft_consensus.cc:3055] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:01:25.524669  6914 raft_consensus.cc:740] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:01:25.528645  1751 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43156 (request call id 1033) took 1577 ms. Trace:
I20260709 13:01:25.463124  7014 raft_consensus.cc:515] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:25.529122  1751 rpcz_store.cc:276] 0709 13:01:23.951480 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:24.278956 (+327476us) service_pool.cc:224] Handling call
0709 13:01:24.279271 (+   315us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:25.392636 (+1113365us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:25.528295 (+135659us) spinlock_profiling.cc:243] Waited 1.11 s on lock 0x7b54001c7854. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:01:25.528554 (+   259us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2557969792}
I20260709 13:01:25.531024  6512 raft_consensus.cc:515] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:25.532573  6592 leader_election.cc:290] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:25.533470  6512 leader_election.cc:290] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 13 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:25.525882  6914 consensus_queue.cc:260] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:25.537847  6914 raft_consensus.cc:3060] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:01:25.533125  6605 raft_consensus.cc:515] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:25.536917  1751 raft_consensus.cc:1275] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 14 index: 12. (index mismatch)
I20260709 13:01:25.541049  6605 leader_election.cc:290] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:25.543015  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:25.543087  6345 consensus_queue.cc:1048] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
W20260709 13:01:25.512538  6985 consensus_queue.cc:1175] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "af520b67945d443b901bbeca37cc5701" responder_term: 16 status { last_received { term: 15 index: 12 } last_committed_idx: 9 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 15. Current term is 16. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260709 13:01:25.491827  6972 raft_consensus.cc:3055] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 13 LEADER]: Stepping down as leader of term 13
I20260709 13:01:25.550829  6972 raft_consensus.cc:740] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:01:25.553046  2332 raft_consensus.cc:1240] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 11. Current term is 12. Ops: []
W20260709 13:01:25.535830  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 75040.8us
I20260709 13:01:25.531386  7014 leader_election.cc:290] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:25.560546  2299 leader_election.cc:304] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:01:25.565330  2331 raft_consensus.cc:1240] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 13. Current term is 14. Ops: []
I20260709 13:01:25.569345  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:25.574178  2332 raft_consensus.cc:1275] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 14 index: 12. (index mismatch)
I20260709 13:01:25.570273  2337 raft_consensus.cc:2468] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 13.
W20260709 13:01:25.555650  6981 log.cc:927] Time spent T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.943s	user 0.005s	sys 0.003s
I20260709 13:01:25.602859  2337 raft_consensus.cc:1275] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 9 index: 10. Preceding OpId from leader: term: 12 index: 12. (index mismatch)
I20260709 13:01:25.563665  6979 raft_consensus.cc:493] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:01:25.551918  6972 consensus_queue.cc:260] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.12, Last appended by leader: 12, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:25.620065  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:25.605225  6605 raft_consensus.cc:2749] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:01:25.623878  2337 raft_consensus.cc:2410] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for term 12 because replica has last-logged OpId of term: 10 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 9.
W20260709 13:01:25.630373  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:25.604817  6979 raft_consensus.cc:515] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:25.535835  6882 consensus_queue.cc:260] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:25.650799  1750 raft_consensus.cc:2410] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for term 12 because replica has last-logged OpId of term: 10 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 9.
W20260709 13:01:25.691049  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 504596us
I20260709 13:01:25.691911  1571 leader_election.cc:304] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:01:25.692889  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:25.693779  1750 raft_consensus.cc:2468] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 11.
W20260709 13:01:25.695024  1717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:25.651072  6983 leader_election.cc:290] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:25.656718  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:25.656939  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:25.657672  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:25.658407  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:25.659085  6979 leader_election.cc:290] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:25.666647  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:25.666862  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:25.703660  2329 raft_consensus.cc:2393] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 13: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:01:25.668345  6974 consensus_queue.cc:1059] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 11.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 13:01:25.671154  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:25.673933  2335 raft_consensus.cc:1275] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 14 index: 12. (index mismatch)
I20260709 13:01:25.674605  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:25.677342  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:25.635465  6972 raft_consensus.cc:3060] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:01:25.679909  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:25.682853  6752 raft_consensus.cc:2749] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
W20260709 13:01:25.638368  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:25.688956  6884 raft_consensus.cc:493] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:25.719822  6884 raft_consensus.cc:515] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } }
I20260709 13:01:25.722340  6884 leader_election.cc:290] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:25.689618  1644 leader_election.cc:304] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
W20260709 13:01:25.689991  1570 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:25.689738  6898 consensus_queue.cc:1048] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 13:01:25.725056  6898 raft_consensus.cc:2804] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:01:25.725450  6898 raft_consensus.cc:493] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:25.725781  6898 raft_consensus.cc:3060] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:01:25.726629  6890 consensus_queue.cc:1048] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 13:01:25.727552  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:25.728284  1597 raft_consensus.cc:2468] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 15.
I20260709 13:01:25.728539  1644 leader_election.cc:304] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:01:25.690212  6865 raft_consensus.cc:493] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:25.703572  1749 raft_consensus.cc:3060] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:01:25.729468  6865 raft_consensus.cc:515] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:25.668706  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:25.731173  6896 raft_consensus.cc:2749] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:01:25.731873  6865 leader_election.cc:290] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:25.716250  2332 raft_consensus.cc:2393] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 15: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:01:25.734411  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:25.736958  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:25.737789  1597 raft_consensus.cc:3060] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Advancing to term 13
W20260709 13:01:25.745730  1718 leader_election.cc:336] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Timed out: RequestConsensusVote RPC to 127.0.107.194:46261 timed out after 2.206s (SENT)
W20260709 13:01:25.746954  1645 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:25.749388  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:25.750155  1674 raft_consensus.cc:2393] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 11: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
W20260709 13:01:25.759990  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 69545.1us
I20260709 13:01:25.759696  1501 catalog_manager.cc:5697] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d reported cstate change: term changed from 4 to 11, leader changed from 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) to 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194). New cstate: current_term: 11 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:01:25.760711  1569 leader_election.cc:304] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:01:25.762157  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:25.762756  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:01:25.762986  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 64217.2us
I20260709 13:01:25.763434  1674 raft_consensus.cc:2468] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 15.
I20260709 13:01:25.761662  1501 catalog_manager.cc:5697] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d reported cstate change: term changed from 7 to 10, leader changed from af520b67945d443b901bbeca37cc5701 (127.0.107.195) to 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194). New cstate: current_term: 10 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } }
I20260709 13:01:25.768255  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:25.769172  2332 raft_consensus.cc:2468] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 14.
I20260709 13:01:25.746337  1718 leader_election.cc:304] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:25.775055  1644 leader_election.cc:304] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:01:25.777104  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:25.779457  1747 raft_consensus.cc:1275] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20260709 13:01:25.710592  6863 raft_consensus.cc:2804] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:01:25.783982  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:25.785816  1749 raft_consensus.cc:2468] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 13.
I20260709 13:01:25.787691  2301 leader_election.cc:304] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:25.797099  1754 raft_consensus.cc:1275] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 12 index: 6. Preceding OpId from leader: term: 14 index: 7. (index mismatch)
I20260709 13:01:25.778852  6926 raft_consensus.cc:2749] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:01:25.802968  2299 leader_election.cc:304] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:01:25.804679  2299 leader_election.cc:304] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:01:25.806764  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:01:25.820417  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 46421.6us
I20260709 13:01:25.743121  6952 raft_consensus.cc:493] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
W20260709 13:01:25.822175  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 130347us
I20260709 13:01:25.710639  6512 consensus_queue.cc:1048] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.078s
I20260709 13:01:25.829911  1597 raft_consensus.cc:2468] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 13.
W20260709 13:01:25.930974  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 170318us
W20260709 13:01:25.931854  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 109468us
I20260709 13:01:25.930065  1718 leader_election.cc:304] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:25.814664  6896 raft_consensus.cc:2804] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Leader pre-election won for term 15
W20260709 13:01:25.938406  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 113338us
I20260709 13:01:25.807075  6914 raft_consensus.cc:493] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:25.789409  6863 raft_consensus.cc:493] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:25.778596  6980 consensus_queue.cc:1059] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.058s
W20260709 13:01:25.830884  7018 log.cc:927] Time spent T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.050s	user 0.000s	sys 0.001s
I20260709 13:01:25.827337  7000 raft_consensus.cc:2804] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:01:25.944830  1571 leader_election.cc:304] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1
W20260709 13:01:25.931070  7032 log.cc:927] Time spent T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.144s	user 0.001s	sys 0.000s
I20260709 13:01:25.827586  6977 raft_consensus.cc:2749] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 13:01:25.823761  7009 raft_consensus.cc:3060] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:01:25.938694  6896 raft_consensus.cc:493] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:25.958377  6160 raft_consensus.cc:2749] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:01:25.960598  1718 leader_election.cc:304] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
W20260709 13:01:25.954756  1643 leader_election.cc:336] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 14 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 3.236s (SENT)
I20260709 13:01:25.930156  6926 consensus_queue.cc:1048] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.026s
I20260709 13:01:25.941708  6954 raft_consensus.cc:493] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:01:25.973969  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 152598us
I20260709 13:01:25.945166  6752 raft_consensus.cc:2804] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader election won for term 15
I20260709 13:01:25.976855  1643 leader_election.cc:304] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
W20260709 13:01:25.979171  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 191556us
I20260709 13:01:25.944128  6863 raft_consensus.cc:3060] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:01:25.989236  1718 leader_election.cc:304] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:25.831048  6952 raft_consensus.cc:515] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:25.958237  7000 raft_consensus.cc:493] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:01:26.008477  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 69182.2us
I20260709 13:01:25.944108  6859 consensus_queue.cc:1048] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.048s
I20260709 13:01:26.003854  6908 raft_consensus.cc:2804] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Leader pre-election won for term 11
W20260709 13:01:26.018410  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 210570us
I20260709 13:01:26.021504  2299 leader_election.cc:304] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:01:25.981845  6896 raft_consensus.cc:3060] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:01:26.015426  1718 leader_election.cc:304] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:25.948776  6914 raft_consensus.cc:515] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:26.029561  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:26.013610  7003 log.cc:927] Time spent T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.051s	user 0.001s	sys 0.000s
I20260709 13:01:26.053242  1571 leader_election.cc:304] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
W20260709 13:01:26.062328  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 63497.8us
W20260709 13:01:26.054248  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 99540.9us
I20260709 13:01:26.001241  6932 raft_consensus.cc:2749] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 13:01:26.013332  6997 raft_consensus.cc:2749] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:01:26.052762  1718 leader_election.cc:304] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:26.010802  6977 raft_consensus.cc:2764] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader election decision vote started in defunct term 9: won
I20260709 13:01:26.016347  7000 raft_consensus.cc:3060] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:01:26.011096  6752 raft_consensus.cc:697] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
W20260709 13:01:26.061666  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 73905.8us
I20260709 13:01:26.021020  6908 raft_consensus.cc:493] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:01:26.066672  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 45269.9us
W20260709 13:01:26.066618  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 133637us
I20260709 13:01:26.054934  6160 raft_consensus.cc:2804] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:01:26.000731  6954 raft_consensus.cc:515] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:26.093290  2301 leader_election.cc:304] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
W20260709 13:01:26.096529  1643 leader_election.cc:336] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 3.364s (SENT)
I20260709 13:01:26.108453  6160 raft_consensus.cc:493] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:26.112900  6160 raft_consensus.cc:3060] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:01:26.109297  1718 leader_election.cc:304] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:26.113646  2301 leader_election.cc:304] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
W20260709 13:01:26.108767  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 95915.3us
W20260709 13:01:26.114423  1646 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:26.091259  6884 raft_consensus.cc:2749] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 13:01:26.092089  6408 raft_consensus.cc:2749] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:01:26.097820  6932 raft_consensus.cc:2749] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 13:01:26.125608  1597 raft_consensus.cc:1275] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20260709 13:01:26.110231  6908 raft_consensus.cc:3060] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:01:26.133767  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:26.134613  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:26.143904  1571 leader_election.cc:400] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 election: Vote denied by peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693) with higher term. Message: Invalid argument: T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 13. Current term is 14.
I20260709 13:01:26.148725  1571 leader_election.cc:403] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 election: Cancelling election due to peer responding with higher term
I20260709 13:01:26.154137  1571 leader_election.cc:304] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:01:26.124254  6911 raft_consensus.cc:2749] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:01:26.099521  6978 consensus_queue.cc:1048] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.031s
I20260709 13:01:26.128611  6643 raft_consensus.cc:2804] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 13:01:26.159046  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:26.161091  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:01:26.152426  7008 consensus_meta.cc:287] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.726s	user 0.013s	sys 0.001s
W20260709 13:01:26.166167  7038 log.cc:927] Time spent T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.228s	user 0.001s	sys 0.000s
I20260709 13:01:26.158484  6980 raft_consensus.cc:2749] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 13:01:26.150604  6622 raft_consensus.cc:2749] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:01:26.143712  6939 raft_consensus.cc:493] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:26.175201  6952 leader_election.cc:290] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:26.114267  6898 raft_consensus.cc:515] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:26.188311  1644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:26.171828  7042 log.cc:927] Time spent T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.147s	user 0.000s	sys 0.001s
W20260709 13:01:26.177994  7039 log.cc:927] Time spent T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.111s	user 0.001s	sys 0.000s
I20260709 13:01:26.185873  6939 raft_consensus.cc:515] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:26.200888  6160 raft_consensus.cc:515] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:26.217577  7041 log.cc:927] Time spent T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.107s	user 0.000s	sys 0.001s
I20260709 13:01:26.182133  1571 leader_election.cc:304] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 13:01:26.180621  6752 consensus_queue.cc:237] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.12, Last appended by leader: 12, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:26.156872  6932 consensus_queue.cc:1059] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 14.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.024s
I20260709 13:01:26.206422  7048 raft_consensus.cc:493] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:01:26.181784  7008 raft_consensus.cc:515] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:26.224104  6914 leader_election.cc:290] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:26.244699  6980 raft_consensus.cc:2764] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Leader election decision vote started in defunct term 12: won
W20260709 13:01:26.259451  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 80016.2us
I20260709 13:01:26.275097  1571 leader_election.cc:304] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:01:26.267765  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 14 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:26.246865  6978 raft_consensus.cc:493] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
W20260709 13:01:26.270404  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 152864us
I20260709 13:01:26.287287  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:26.287287  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:26.289988  2335 raft_consensus.cc:1275] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 14 index: 12. (index mismatch)
I20260709 13:01:26.290370  1571 leader_election.cc:304] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:01:26.292243  1718 leader_election.cc:304] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:26.296563  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:26.304589  1601 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 1224) took 1228 ms. Trace:
I20260709 13:01:26.306427  1718 leader_election.cc:304] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:26.258838  6643 raft_consensus.cc:697] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 13 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:01:26.304774  1571 leader_election.cc:304] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:01:26.295603  1671 raft_consensus.cc:2393] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 14: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:01:26.314306  1601 rpcz_store.cc:276] 0709 13:01:25.075999 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:25.076139 (+   140us) service_pool.cc:224] Handling call
0709 13:01:25.076504 (+   365us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:25.682700 (+606196us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:25.682734 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:01:25.683299 (+   565us) log.cc:844] Serialized 26 byte log entry
0709 13:01:25.827900 (+144601us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:01:25.827986 (+    86us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:25.828094 (+   108us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:26.297515 (+469421us) raft_consensus.cc:1685] finished
0709 13:01:26.297547 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:26.304489 (+  6942us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":605736,"raft.queue_time_us":606696,"raft.run_cpu_time_us":60,"raft.run_wall_time_us":60,"spinlock_wait_cycles":1792,"thread_start_us":143518,"threads_started":1,"wal-append.queue_time_us":297293}
W20260709 13:01:26.350730  1643 leader_election.cc:336] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 12 election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 1.714s (SENT)
W20260709 13:01:26.352478  1643 leader_election.cc:336] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 14 election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 1.719s (SENT)
I20260709 13:01:26.295066  6859 raft_consensus.cc:493] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:26.294358  6952 raft_consensus.cc:2749] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:01:26.310576  7061 raft_consensus.cc:493] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:26.310319  6911 raft_consensus.cc:2749] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:01:26.307135  7051 raft_consensus.cc:2749] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:01:26.324393  6980 raft_consensus.cc:2749] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:01:26.374876  6896 raft_consensus.cc:515] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:26.320870  6954 leader_election.cc:290] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:26.356923  6345 raft_consensus.cc:493] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:26.307459  6962 consensus_queue.cc:1048] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.062s
I20260709 13:01:26.260236  7048 raft_consensus.cc:515] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:26.359002  6854 raft_consensus.cc:493] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:26.356232  6692 raft_consensus.cc:493] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:26.393653  2329 raft_consensus.cc:1240] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 11. Current term is 12. Ops: []
I20260709 13:01:26.295524  6978 raft_consensus.cc:515] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:26.368674  7054 raft_consensus.cc:2749] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:01:26.316072  6914 consensus_queue.cc:1048] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.058s
I20260709 13:01:26.364362  6884 raft_consensus.cc:2749] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader election lost for term 13. Reason: Vote denied by peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693) with higher term. Message: Invalid argument: T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 13. Current term is 14.
I20260709 13:01:26.403076  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:26.367422  6859 raft_consensus.cc:515] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:26.412567  2335 raft_consensus.cc:3060] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:01:26.398414  6898 leader_election.cc:290] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 14 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:26.423743  2333 raft_consensus.cc:2410] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for term 14 because replica has last-logged OpId of term: 13 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 10.
I20260709 13:01:26.376080  7061 raft_consensus.cc:515] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:26.432266  2335 raft_consensus.cc:2468] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 14.
I20260709 13:01:26.441354  6896 leader_election.cc:290] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 15 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:26.405334  6954 consensus_queue.cc:1048] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.025s
I20260709 13:01:26.403462  6345 raft_consensus.cc:515] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:26.461585  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
W20260709 13:01:26.462250  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 53662.5us
I20260709 13:01:26.462590  1750 raft_consensus.cc:3060] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:01:26.412619  6692 raft_consensus.cc:515] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:26.436645  7054 raft_consensus.cc:493] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:26.409734  6854 raft_consensus.cc:515] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:26.482944  7070 raft_consensus.cc:493] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:26.401646  6643 consensus_queue.cc:237] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:26.462930  7008 leader_election.cc:290] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 15 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:26.437619  6898 consensus_queue.cc:1059] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 11.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.047s
I20260709 13:01:26.485631  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:26.497320  2332 raft_consensus.cc:3060] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:01:26.492028  6962 raft_consensus.cc:493] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:26.502650  6962 raft_consensus.cc:515] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:26.504961  6962 leader_election.cc:290] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:26.509589  6854 leader_election.cc:290] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:26.511286  6345 leader_election.cc:290] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:26.512930  1750 raft_consensus.cc:2468] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 15.
W20260709 13:01:26.513819  6972 consensus_meta.cc:287] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.755s	user 0.009s	sys 0.000s
I20260709 13:01:26.514765  1676 raft_consensus.cc:1275] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 13 index: 12. Preceding OpId from leader: term: 14 index: 12. (term mismatch)
I20260709 13:01:26.515138  1676 pending_rounds.cc:85] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d: Aborting all ops after (but not including) 11
I20260709 13:01:26.515378  1676 pending_rounds.cc:107] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d: Aborting uncommitted NO_OP operation due to leader change: 13.12
I20260709 13:01:26.515717  1676 raft_consensus.cc:2889] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:01:26.522545  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:26.524683  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:01:26.515337  7009 consensus_meta.cc:287] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.534s	user 0.013s	sys 0.000s
I20260709 13:01:26.529238  6692 leader_election.cc:290] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:26.531035  6859 leader_election.cc:290] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:26.532608  6882 raft_consensus.cc:3060] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:01:26.533072  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:26.533834  2334 raft_consensus.cc:2468] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 16.
I20260709 13:01:26.534698  7067 raft_consensus.cc:493] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:26.535167  1644 leader_election.cc:304] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:01:26.535295  7067 raft_consensus.cc:515] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:26.535724  7061 leader_election.cc:290] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:26.536126  6692 raft_consensus.cc:2804] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:01:26.536607  6692 raft_consensus.cc:493] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:26.536840  6978 leader_election.cc:290] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:26.537002  6692 raft_consensus.cc:3060] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:01:26.533075  7048 leader_election.cc:290] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:26.537739  7067 leader_election.cc:290] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:26.492168  7070 raft_consensus.cc:515] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:26.541098  7070 leader_election.cc:290] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:26.485123  7054 raft_consensus.cc:515] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:26.546083  7054 leader_election.cc:290] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:26.553316  6939 leader_election.cc:290] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:26.554754  7009 raft_consensus.cc:2749] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:01:26.556629  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:26.556541  6908 raft_consensus.cc:515] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:26.557504  2332 raft_consensus.cc:2468] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 15.
I20260709 13:01:26.559181  2331 raft_consensus.cc:1240] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 14. Current term is 15. Ops: [12.11-14.12]
I20260709 13:01:26.564404  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:26.565954  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:26.565071  1677 raft_consensus.cc:2468] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 10.
I20260709 13:01:26.569270  1644 leader_election.cc:304] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:01:26.569512  7048 raft_consensus.cc:493] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:26.570307  7012 raft_consensus.cc:2804] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Leader election won for term 15
I20260709 13:01:26.570053  7048 raft_consensus.cc:515] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:26.568244  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:26.572538  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:26.573173  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 12 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:26.575873  1753 raft_consensus.cc:2393] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 14: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:01:26.580102  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:26.580233  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:26.570876  7012 raft_consensus.cc:697] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:01:26.599705  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:26.601714  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 16 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:26.616075  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:26.618325  1569 leader_election.cc:304] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:01:26.620635  6859 raft_consensus.cc:2804] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:01:26.621071  6859 raft_consensus.cc:493] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:26.621425  6859 raft_consensus.cc:3060] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Advancing to term 11
W20260709 13:01:26.622599  6863 consensus_meta.cc:287] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.603s	user 0.009s	sys 0.006s
I20260709 13:01:26.622831  7012 consensus_queue.cc:237] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.11, Last appended by leader: 11, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:26.625473  6692 raft_consensus.cc:515] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:26.627753  6908 leader_election.cc:290] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:26.581708  7000 raft_consensus.cc:515] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:26.634701  7048 leader_election.cc:290] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:26.634873  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:26.635521  1680 raft_consensus.cc:3060] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:01:26.623086  6863 raft_consensus.cc:515] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:26.638857  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:26.639810  2332 raft_consensus.cc:2393] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 15: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:01:26.646754  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:26.647047  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:26.646893  2337 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49082 (request call id 1003) took 1012 ms. Trace:
I20260709 13:01:26.656132  6983 raft_consensus.cc:3055] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 11 LEADER]: Stepping down as leader of term 11
I20260709 13:01:26.656384  7000 leader_election.cc:290] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 16 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:26.656690  6983 raft_consensus.cc:740] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:01:26.648176  2337 rpcz_store.cc:276] 0709 13:01:25.634382 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:25.634581 (+   199us) service_pool.cc:224] Handling call
0709 13:01:25.635003 (+   422us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:25.635307 (+   304us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 13:01:25.636295 (+   988us) log.cc:844] Serialized 14 byte log entry
0709 13:01:26.435942 (+799647us) spinlock_profiling.cc:243] Waited 1.54 ms on lock 0x7b180048f6a0. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f4263057e07 00007f426beb2ab9 00007f426307469b 00007f426307293f 00007f4263074c02 00007f4265cfc20b 00007f4265d01d23 00007f4265d0223a 00007f426616ca2a
0709 13:01:26.646155 (+210213us) log.cc:844] Serialized 14 byte log entry
0709 13:01:26.646501 (+   346us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:01:26.646560 (+    59us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:01:26.646617 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:26.646705 (+    88us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:26.646831 (+   126us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":7096,"prepare.queue_time_us":62415,"prepare.run_cpu_time_us":84,"prepare.run_wall_time_us":85,"raft.queue_time_us":169,"raft.run_cpu_time_us":103,"raft.run_wall_time_us":105,"spinlock_wait_cycles":3543808,"thread_start_us":198285,"threads_started":2,"wal-append.queue_time_us":974516}
I20260709 13:01:26.661569  1680 raft_consensus.cc:2468] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 11.
I20260709 13:01:26.645642  6692 leader_election.cc:290] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 16 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:26.673941  6863 leader_election.cc:290] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:26.657724  6983 consensus_queue.cc:260] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 9, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:26.676678  6859 raft_consensus.cc:515] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:26.648094  6605 raft_consensus.cc:2804] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:01:26.677465  6987 raft_consensus.cc:493] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:01:26.680615  7030 log.cc:927] Time spent T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.751s	user 0.011s	sys 0.004s
I20260709 13:01:26.667021  6160 leader_election.cc:290] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 14 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:26.683820  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
W20260709 13:01:26.703960  1645 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:26.708002  6605 raft_consensus.cc:493] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:01:26.708544  6605 raft_consensus.cc:3060] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:01:26.709570  2332 raft_consensus.cc:2468] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 16.
I20260709 13:01:26.711061  1644 leader_election.cc:304] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:01:26.701663  6987 raft_consensus.cc:515] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:26.712018  6859 leader_election.cc:290] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:26.711894  6867 raft_consensus.cc:2804] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Leader election won for term 16
I20260709 13:01:26.712854  6867 raft_consensus.cc:697] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:01:26.713861  6867 consensus_queue.cc:237] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:26.715416  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:26.716145  6160 consensus_queue.cc:1048] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 13:01:26.718642  1751 raft_consensus.cc:1240] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 13. Current term is 14. Ops: []
I20260709 13:01:26.723804  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:26.724707  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:26.725399  1601 raft_consensus.cc:3060] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:01:26.725564  1602 raft_consensus.cc:2393] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 9: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:01:26.726537  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:26.699395  6692 consensus_queue.cc:1048] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.025s
I20260709 13:01:26.727094  1501 catalog_manager.cc:5697] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d reported cstate change: term changed from 6 to 7, leader changed from af520b67945d443b901bbeca37cc5701 (127.0.107.195) to 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193). New cstate: current_term: 7 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } }
I20260709 13:01:26.728911  1501 catalog_manager.cc:5697] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d reported cstate change: term changed from 7 to 12, leader changed from 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194) to 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196). New cstate: current_term: 12 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 13:01:26.730566  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:26.730365  1501 catalog_manager.cc:5697] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d reported cstate change: term changed from 4 to 9. New cstate: current_term: 9 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 13:01:26.731053  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:26.731793  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 12 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:26.732249  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:26.731693  1501 catalog_manager.cc:5697] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d reported cstate change: term changed from 6 to 9, leader changed from 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194) to 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193). New cstate: current_term: 9 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } }
I20260709 13:01:26.733047  1750 raft_consensus.cc:2393] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 11: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:01:26.733867  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 14 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:26.734534  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:26.735595  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:26.733199  1501 catalog_manager.cc:5697] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d reported cstate change: term changed from 9 to 11, leader changed from af520b67945d443b901bbeca37cc5701 (127.0.107.195) to 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193). New cstate: current_term: 11 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } }
I20260709 13:01:26.741948  1748 raft_consensus.cc:1240] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 13 LEADER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 12. Current term is 13. Ops: []
I20260709 13:01:26.743557  1601 raft_consensus.cc:2468] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 15.
I20260709 13:01:26.738608  1501 catalog_manager.cc:5697] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d reported cstate change: term changed from 3 to 13, leader changed from af520b67945d443b901bbeca37cc5701 (127.0.107.195) to 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196). New cstate: current_term: 13 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 13:01:26.756742  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:26.757572  1602 raft_consensus.cc:2393] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 14: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:01:26.759527  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:26.760262  1605 raft_consensus.cc:2468] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 10.
I20260709 13:01:26.762598  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 13 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:26.764070  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:26.765292  1605 raft_consensus.cc:2410] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for term 13 because replica has last-logged OpId of term: 11 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 5.
I20260709 13:01:26.766309  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 12 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:26.769657  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 16 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:26.723505  6752 raft_consensus.cc:493] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:26.770143  1602 raft_consensus.cc:2468] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 11.
I20260709 13:01:26.771258  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:26.779618  1754 raft_consensus.cc:1275] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 12 index: 9. Preceding OpId from leader: term: 14 index: 10. (index mismatch)
I20260709 13:01:26.780133  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:26.781165  1602 raft_consensus.cc:2393] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 11: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
W20260709 13:01:26.766162  7043 log.cc:927] Time spent T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.662s	user 0.002s	sys 0.000s
I20260709 13:01:26.783906  1605 raft_consensus.cc:3055] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:01:26.784314  1605 raft_consensus.cc:740] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:01:26.785120  1605 consensus_queue.cc:260] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 12.7, Last appended by leader: 7, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:26.786401  1605 raft_consensus.cc:3060] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:01:26.718811  6882 consensus_queue.cc:1048] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.049s
I20260709 13:01:26.788379  2301 leader_election.cc:304] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:26.789952  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:26.792680  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:26.792770  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:26.795094  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:26.798370  1602 raft_consensus.cc:2468] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 10.
I20260709 13:01:26.807418  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:26.807785  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:26.808079  1602 raft_consensus.cc:2468] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 15.
I20260709 13:01:26.809633  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:26.810356  1604 raft_consensus.cc:2376] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 15. Re-sending same reply.
I20260709 13:01:26.812747  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:26.815814  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:26.816133  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:26.816558  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:26.816639  1602 raft_consensus.cc:2468] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 12.
I20260709 13:01:26.816826  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:26.817417  1598 raft_consensus.cc:2468] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 13.
I20260709 13:01:26.817795  1599 raft_consensus.cc:2393] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 14: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:01:26.820070  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:26.820456  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:26.821290  1602 raft_consensus.cc:2393] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 9: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:01:26.822774  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:01:26.791185  7079 log.cc:927] Time spent T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.127s	user 0.001s	sys 0.000s
I20260709 13:01:26.816793  1596 raft_consensus.cc:2468] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 15.
I20260709 13:01:26.825166  1599 raft_consensus.cc:2393] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 13: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:01:26.826014  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:26.826720  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:26.824338  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:26.826870  1604 raft_consensus.cc:2410] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for term 12 because replica has last-logged OpId of term: 11 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 6.
I20260709 13:01:26.767424  6859 consensus_queue.cc:1059] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 11.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.029s
I20260709 13:01:26.836571  1599 raft_consensus.cc:2393] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 11: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:01:26.834183  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:26.842697  1598 raft_consensus.cc:1275] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 15 index: 11. (index mismatch)
I20260709 13:01:26.814133  6876 raft_consensus.cc:2804] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 13:01:26.838768  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:26.847643  1599 raft_consensus.cc:2410] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for term 14 because replica has last-logged OpId of term: 13 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 7.
I20260709 13:01:26.844369  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:26.845055  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:26.789275  6752 raft_consensus.cc:515] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:26.848301  1596 raft_consensus.cc:2468] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 10.
I20260709 13:01:26.849689  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:26.850351  1602 raft_consensus.cc:2393] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 11: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:01:26.852067  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 17 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:26.852632  1599 raft_consensus.cc:2376] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 15. Re-sending same reply.
I20260709 13:01:26.853436  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 13 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:26.854086  1599 raft_consensus.cc:2410] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for term 13 because replica has last-logged OpId of term: 11 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 5.
I20260709 13:01:26.856112  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:26.855279  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:26.856426  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:26.858672  1602 raft_consensus.cc:2468] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 13.
I20260709 13:01:26.864744  1602 raft_consensus.cc:1275] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
I20260709 13:01:26.845558  6714 raft_consensus.cc:3055] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:01:26.867009  1602 raft_consensus.cc:1275] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 14 index: 8. (term mismatch)
I20260709 13:01:26.867476  1602 pending_rounds.cc:85] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a: Aborting all ops after (but not including) 7
I20260709 13:01:26.867329  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:26.867785  1602 pending_rounds.cc:107] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a: Aborting uncommitted NO_OP operation due to leader change: 13.8
I20260709 13:01:26.868002  1604 raft_consensus.cc:2468] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 12.
I20260709 13:01:26.868135  1602 raft_consensus.cc:2889] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:01:26.870139  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:26.869251  6714 raft_consensus.cc:740] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
W20260709 13:01:26.887431  7084 log.cc:927] Time spent T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.074s	user 0.000s	sys 0.000s
I20260709 13:01:26.888630  6714 consensus_queue.cc:260] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.8, Last appended by leader: 8, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:26.852396  6876 raft_consensus.cc:697] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 15 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:01:26.899814  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:26.907313  1605 raft_consensus.cc:1275] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 13 index: 7. (term mismatch)
I20260709 13:01:26.907824  1605 pending_rounds.cc:85] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a: Aborting all ops after (but not including) 6
I20260709 13:01:26.908058  1605 pending_rounds.cc:107] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a: Aborting uncommitted NO_OP operation due to leader change: 12.7
I20260709 13:01:26.908298  1605 raft_consensus.cc:2889] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:01:26.891149  6714 raft_consensus.cc:3060] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:01:26.915876  6605 raft_consensus.cc:515] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:26.911164  6876 consensus_queue.cc:237] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 13.12, Last appended by leader: 12, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:26.922703  7089 log.cc:927] Time spent T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.066s	user 0.000s	sys 0.001s
I20260709 13:01:26.927559  1600 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 1230) took 1201 ms. Trace:
I20260709 13:01:26.928881  6752 leader_election.cc:290] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:26.928179  1600 rpcz_store.cc:276] 0709 13:01:25.725636 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:25.726537 (+   901us) service_pool.cc:224] Handling call
0709 13:01:25.726920 (+   383us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:25.929614 (+202694us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:25.930031 (+   417us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:01:26.673496 (+743465us) spinlock_profiling.cc:243] Waited 14.6 ms on lock 0x7b2c00000658. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4264a5ba11 00007f4264a6b372 00007f426619da01 00007f426616243a 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2
0709 13:01:26.673985 (+   489us) log.cc:844] Serialized 26 byte log entry
0709 13:01:26.788572 (+114587us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:01:26.788634 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:26.788741 (+   107us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:26.926780 (+138039us) raft_consensus.cc:1685] finished
0709 13:01:26.926834 (+    54us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:26.927383 (+   549us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":95415,"raft.queue_time_us":203561,"spinlock_wait_cycles":42255488,"thread_start_us":113057,"threads_started":1,"wal-append.queue_time_us":174608}
I20260709 13:01:26.932307  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:26.933662  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:26.934392  1749 raft_consensus.cc:2393] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 11: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:01:26.936496  1604 raft_consensus.cc:1275] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 11 index: 6. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
I20260709 13:01:26.940578  1717 leader_election.cc:304] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:26.941788  1717 leader_election.cc:304] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:26.943280  1643 leader_election.cc:304] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:01:26.955015  1596 raft_consensus.cc:1275] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 13 index: 7. Preceding OpId from leader: term: 14 index: 7. (term mismatch)
I20260709 13:01:26.955498  1596 pending_rounds.cc:85] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a: Aborting all ops after (but not including) 6
I20260709 13:01:26.955785  1596 pending_rounds.cc:107] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a: Aborting uncommitted NO_OP operation due to leader change: 13.7
I20260709 13:01:26.956015  1596 raft_consensus.cc:2889] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:01:26.956537  1717 leader_election.cc:304] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:26.961314  1643 leader_election.cc:304] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:26.962680  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:01:26.964334  1571 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:26.947045  6638 raft_consensus.cc:2804] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:01:26.966198  6638 raft_consensus.cc:493] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:26.966768  6638 raft_consensus.cc:3060] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:01:26.975752  6605 leader_election.cc:290] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 12 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:26.978673  1596 raft_consensus.cc:1240] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 8. Current term is 9. Ops: []
I20260709 13:01:26.979241  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:26.980265  1596 raft_consensus.cc:1240] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 13. Current term is 14. Ops: []
I20260709 13:01:26.980496  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:26.981427  1680 raft_consensus.cc:2393] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 11: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:01:26.982069  1596 raft_consensus.cc:1275] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 11 index: 8. Preceding OpId from leader: term: 12 index: 9. (index mismatch)
I20260709 13:01:26.983577  6997 consensus_queue.cc:1048] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 13:01:26.984642  6976 raft_consensus.cc:2749] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260709 13:01:26.987113  1719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:26.954813  6867 raft_consensus.cc:2749] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 13:01:26.965276  1717 leader_election.cc:304] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:26.989655  1596 raft_consensus.cc:1275] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 10 index: 5. Preceding OpId from leader: term: 14 index: 7. (index mismatch)
I20260709 13:01:27.011112  1673 raft_consensus.cc:1275] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 15 index: 13. (index mismatch)
I20260709 13:01:27.012140  1596 raft_consensus.cc:1275] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 11 index: 11. Preceding OpId from leader: term: 12 index: 12. (index mismatch)
I20260709 13:01:26.989707  2331 raft_consensus.cc:1240] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 15. Current term is 16. Ops: []
I20260709 13:01:26.990724  1672 raft_consensus.cc:1240] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 LEADER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 14. Current term is 15. Ops: []
I20260709 13:01:27.013515  2333 raft_consensus.cc:1240] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 14. Current term is 15. Ops: [12.11-14.12]
W20260709 13:01:26.965413  1571 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:27.007630  2332 raft_consensus.cc:1240] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 11. Current term is 12. Ops: []
I20260709 13:01:27.009693  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:01:27.018043  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 52890.5us
I20260709 13:01:27.018913  1717 leader_election.cc:304] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:27.020804  1717 leader_election.cc:304] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:26.987789  6884 raft_consensus.cc:3055] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:01:27.069053  1571 leader_election.cc:304] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
W20260709 13:01:27.067571  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 48671.4us
W20260709 13:01:27.079073  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 58287.1us
I20260709 13:01:27.025494  6884 raft_consensus.cc:740] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:01:27.080296  6752 raft_consensus.cc:493] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:27.025681  6821 raft_consensus.cc:2804] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:01:27.080925  6752 raft_consensus.cc:515] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:27.081221  6863 consensus_queue.cc:1059] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 13:01:27.082587  6962 raft_consensus.cc:493] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:27.087447  1717 leader_election.cc:304] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:27.025651  6908 raft_consensus.cc:2804] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:01:27.081336  6821 raft_consensus.cc:493] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
W20260709 13:01:27.122381  7090 log.cc:927] Time spent T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.103s	user 0.000s	sys 0.001s
I20260709 13:01:27.025020  6859 consensus_queue.cc:1059] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 11.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.095s
I20260709 13:01:27.070938  7050 raft_consensus.cc:493] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:27.131075  6821 raft_consensus.cc:3060] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Advancing to term 11
W20260709 13:01:27.145156  1718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:27.093921  6926 raft_consensus.cc:2804] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:01:27.094588  6867 raft_consensus.cc:493] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:27.130421  7071 raft_consensus.cc:3055] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 LEADER]: Stepping down as leader of term 15
I20260709 13:01:27.139276  6908 raft_consensus.cc:493] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:27.084612  7067 consensus_queue.cc:1059] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 11.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.077s
I20260709 13:01:27.083578  6962 raft_consensus.cc:515] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:27.164219  6859 raft_consensus.cc:2749] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:01:27.159749  6926 raft_consensus.cc:493] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:27.186758  6962 leader_election.cc:290] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:27.170405  7071 raft_consensus.cc:740] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:01:27.188679  6926 raft_consensus.cc:3060] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:01:27.177593  6908 raft_consensus.cc:3060] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:01:27.159346  7050 raft_consensus.cc:515] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:27.164973  6867 raft_consensus.cc:515] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:27.197234  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 123878us
I20260709 13:01:27.217716  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:27.220968  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:27.241752  1680 raft_consensus.cc:2468] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 9.
I20260709 13:01:27.203644  7067 raft_consensus.cc:3055] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 LEADER]: Stepping down as leader of term 11
W20260709 13:01:27.243969  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 175017us
I20260709 13:01:27.246608  1571 leader_election.cc:304] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
W20260709 13:01:27.261422  7093 log.cc:927] Time spent T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.095s	user 0.000s	sys 0.001s
I20260709 13:01:27.230453  6962 raft_consensus.cc:493] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:27.269134  1571 leader_election.cc:304] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:01:27.242908  6859 raft_consensus.cc:493] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:27.244891  7067 raft_consensus.cc:740] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:01:27.275100  1571 leader_election.cc:304] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:01:27.245754  6865 raft_consensus.cc:493] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:27.266577  6345 raft_consensus.cc:493] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:27.278483  7102 raft_consensus.cc:493] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:27.312923  6345 raft_consensus.cc:515] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:27.298162  6882 raft_consensus.cc:493] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
W20260709 13:01:27.310535  7091 log.cc:927] Time spent T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.178s	user 0.001s	sys 0.000s
I20260709 13:01:27.301453  7099 raft_consensus.cc:2749] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:01:27.275638  6962 raft_consensus.cc:515] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:27.347559  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:27.248596  7071 consensus_queue.cc:260] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.13, Last appended by leader: 13, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:27.277704  6859 raft_consensus.cc:515] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:27.352041  1645 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:27.358613  6882 raft_consensus.cc:515] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:27.302238  6865 raft_consensus.cc:515] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:27.327632  7101 raft_consensus.cc:2749] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 13:01:27.405185  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:27.389102  6882 leader_election.cc:290] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:01:27.387415  7098 log.cc:927] Time spent T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.081s	user 0.000s	sys 0.000s
W20260709 13:01:27.391345  7097 log.cc:927] Time spent T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.146s	user 0.000s	sys 0.001s
I20260709 13:01:27.334220  7102 raft_consensus.cc:515] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:27.380002  7099 raft_consensus.cc:2749] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:01:27.419327  2331 raft_consensus.cc:1275] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 16 index: 13. (index mismatch)
W20260709 13:01:27.403641  7095 log.cc:927] Time spent T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.208s	user 0.000s	sys 0.001s
I20260709 13:01:27.380355  7071 raft_consensus.cc:3060] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:01:27.376433  7104 raft_consensus.cc:2804] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:01:27.392237  6987 leader_election.cc:290] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:27.308080  6752 leader_election.cc:290] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:01:27.433766  7063 log.cc:927] Time spent T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 1.006s	user 0.004s	sys 0.000s
I20260709 13:01:27.447853  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:27.451699  6865 leader_election.cc:290] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:01:27.454281  7040 log.cc:927] Time spent T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 1.362s	user 0.003s	sys 0.000s
I20260709 13:01:27.455747  1603 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 1242) took 1142 ms. Trace:
I20260709 13:01:27.471339  1603 rpcz_store.cc:276] 0709 13:01:26.313661 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:26.313819 (+   158us) service_pool.cc:224] Handling call
0709 13:01:26.355548 (+ 41729us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:26.400482 (+ 44934us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:26.400516 (+    34us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:01:26.400562 (+    46us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:01:26.400618 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:27.455497 (+1054879us) spinlock_profiling.cc:243] Waited 44.7 ms on lock 0x7b54000f19d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:01:27.455550 (+    53us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:27.455679 (+   129us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":102730624}
I20260709 13:01:27.370218  7067 consensus_queue.cc:260] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 11.11, Last appended by leader: 11, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:27.445950  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:27.478966  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:27.431661  7104 raft_consensus.cc:493] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
W20260709 13:01:27.462011  6714 consensus_meta.cc:287] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.548s	user 0.008s	sys 0.007s
I20260709 13:01:27.481398  1749 raft_consensus.cc:2393] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 16: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:01:27.479149  1751 raft_consensus.cc:3060] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:01:27.466524  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:27.480348  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:01:27.470891  6638 consensus_meta.cc:287] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.504s	user 0.011s	sys 0.004s
I20260709 13:01:27.495471  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:27.484177  1680 raft_consensus.cc:2468] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 10.
I20260709 13:01:27.498728  7067 raft_consensus.cc:3060] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:01:27.499888  6926 raft_consensus.cc:515] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:27.502250  6345 leader_election.cc:290] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:27.506772  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:27.497004  2298 leader_election.cc:304] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:01:27.462878  7050 leader_election.cc:290] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:27.484144  1604 raft_consensus.cc:2468] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 15.
W20260709 13:01:27.517370  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 242787us
I20260709 13:01:27.518875  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:27.526953  1571 leader_election.cc:304] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:01:27.527177  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:27.527969  1753 raft_consensus.cc:2393] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 16: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:01:27.522285  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:01:27.533354  1570 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:27.504422  6714 raft_consensus.cc:493] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:27.483050  7104 raft_consensus.cc:3060] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:01:27.521373  6987 raft_consensus.cc:493] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:27.551081  2298 leader_election.cc:304] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:01:27.541632  1604 raft_consensus.cc:1275] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 10 index: 8. Preceding OpId from leader: term: 14 index: 10. (index mismatch)
W20260709 13:01:27.549854  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 462436us
I20260709 13:01:27.478068  6940 consensus_queue.cc:1048] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.055s
I20260709 13:01:27.538889  1571 leader_election.cc:304] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:01:27.532446  7011 raft_consensus.cc:2749] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:01:27.556563  6821 raft_consensus.cc:515] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:27.528674  6345 raft_consensus.cc:493] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:27.556159  1717 leader_election.cc:304] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:27.564198  1717 leader_election.cc:304] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:27.516500  6638 raft_consensus.cc:515] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } }
I20260709 13:01:27.537563  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:27.568544  1749 raft_consensus.cc:2468] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 8.
I20260709 13:01:27.574828  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:27.546641  7103 raft_consensus.cc:2749] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:01:27.580641  1571 leader_election.cc:304] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:01:27.576460  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:27.576879  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:27.581938  1571 leader_election.cc:304] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:01:27.582113  1749 raft_consensus.cc:2468] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 14.
I20260709 13:01:27.561501  6926 leader_election.cc:290] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 13 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:27.586783  1571 leader_election.cc:304] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:01:27.550474  6714 raft_consensus.cc:515] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:27.579012  1751 raft_consensus.cc:2468] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 14.
I20260709 13:01:27.570322  6643 raft_consensus.cc:2804] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:01:27.579479  6997 consensus_queue.cc:1048] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.005s
I20260709 13:01:27.595358  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:27.600049  1749 raft_consensus.cc:2393] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 15: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:01:27.576388  6962 leader_election.cc:290] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:27.558573  6987 raft_consensus.cc:515] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:27.585645  7099 raft_consensus.cc:2749] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:01:27.587373  6752 raft_consensus.cc:2749] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:01:27.605418  1717 leader_election.cc:304] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:27.572626  6345 raft_consensus.cc:515] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:27.636941  6987 leader_election.cc:290] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:27.637662  6345 leader_election.cc:290] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:27.593715  6643 raft_consensus.cc:493] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:27.598881  6926 raft_consensus.cc:3060] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:01:27.590086  7106 raft_consensus.cc:2749] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:01:27.622243  1749 raft_consensus.cc:1240] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 12. Current term is 13. Ops: []
I20260709 13:01:27.623808  1601 raft_consensus.cc:2410] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 for term 16 because replica has last-logged OpId of term: 15 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 12.
I20260709 13:01:27.628063  6638 leader_election.cc:290] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 12 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:27.630155  6714 leader_election.cc:290] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:27.630944  7105 raft_consensus.cc:3060] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:01:27.593901  7011 consensus_queue.cc:1048] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.041s
I20260709 13:01:27.608072  7103 raft_consensus.cc:2749] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:01:27.635632  7102 leader_election.cc:290] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:27.639422  6821 leader_election.cc:290] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:27.659951  7102 raft_consensus.cc:2749] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:01:27.660534  6821 raft_consensus.cc:493] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:27.661090  6821 raft_consensus.cc:515] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:01:27.661868  6956 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 13 current term: 13 is higher.
I20260709 13:01:27.654791  6643 raft_consensus.cc:3060] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:01:27.664371  1749 raft_consensus.cc:1275] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 16 index: 13. (index mismatch)
I20260709 13:01:27.640141  6908 raft_consensus.cc:515] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:27.644455  2335 raft_consensus.cc:1240] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 14. Current term is 15. Ops: [12.11-14.12]
I20260709 13:01:27.612026  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 15 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:27.646986  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
W20260709 13:01:27.646485  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 59735us
I20260709 13:01:27.650923  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:27.653898  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:27.671429  1571 leader_election.cc:304] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:01:27.671723  1673 raft_consensus.cc:2468] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 11.
I20260709 13:01:27.674510  1749 raft_consensus.cc:1275] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 13 index: 12. Preceding OpId from leader: term: 15 index: 13. (index mismatch)
W20260709 13:01:27.676575  1718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:27.677386  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:27.677580  1678 raft_consensus.cc:3060] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:01:27.678061  1679 raft_consensus.cc:3060] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:01:27.679330  1676 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52778 (request call id 2314) took 1998 ms. Trace:
I20260709 13:01:27.679733  1676 rpcz_store.cc:276] 0709 13:01:25.680898 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:25.681087 (+   189us) service_pool.cc:224] Handling call
0709 13:01:25.681425 (+   338us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:26.516213 (+834788us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:27.679063 (+1162850us) spinlock_profiling.cc:243] Waited 833 ms on lock 0x7b540025b054. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:01:27.679263 (+   200us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1915223424}
I20260709 13:01:27.681083  6867 leader_election.cc:290] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:27.681283  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:27.681977  1673 raft_consensus.cc:2468] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 14.
I20260709 13:01:27.676153  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:27.687299  1569 leader_election.cc:304] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:01:27.687592  1571 leader_election.cc:304] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:01:27.693755  2299 leader_election.cc:304] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:01:27.698823  1569 leader_election.cc:304] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 13:01:27.699281  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:27.699805  6160 raft_consensus.cc:2749] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 13:01:27.705895  1679 raft_consensus.cc:2468] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 16.
I20260709 13:01:27.690290  6914 raft_consensus.cc:2749] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:01:27.709362  6821 leader_election.cc:290] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:27.715534  6940 raft_consensus.cc:2749] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:01:27.717133  7071 raft_consensus.cc:2804] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:01:27.717571  7071 raft_consensus.cc:493] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:27.717952  7071 raft_consensus.cc:3060] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:01:27.696943  7101 raft_consensus.cc:2749] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:01:27.722122  1751 raft_consensus.cc:1240] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 15. Current term is 16. Ops: []
I20260709 13:01:27.668166  7109 raft_consensus.cc:493] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
W20260709 13:01:27.734854  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 146570us
I20260709 13:01:27.737839  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:27.701120  6863 raft_consensus.cc:493] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:27.739569  6863 raft_consensus.cc:515] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:27.741950  6863 leader_election.cc:290] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:27.721169  6752 raft_consensus.cc:2804] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:01:27.742916  6752 raft_consensus.cc:493] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:27.743301  6752 raft_consensus.cc:3060] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:01:27.727711  6867 raft_consensus.cc:493] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:01:27.747159  6867 raft_consensus.cc:515] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:27.756784  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:27.765461  1718 leader_election.cc:304] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:27.766882  1718 leader_election.cc:304] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
W20260709 13:01:27.771456  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 166103us
I20260709 13:01:27.772666  1598 raft_consensus.cc:1240] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 10. Current term is 11. Ops: []
I20260709 13:01:27.683771  7106 consensus_queue.cc:1059] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 9, Time since last communication: 0.070s
I20260709 13:01:27.727396  6821 raft_consensus.cc:493] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:27.770722  1646 leader_election.cc:304] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:01:27.780236  1646 leader_election.cc:304] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:27.781136  1646 leader_election.cc:304] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:27.749635  6867 leader_election.cc:290] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:01:27.787546  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 100021us
I20260709 13:01:27.788962  7101 raft_consensus.cc:2749] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:01:27.791150  6932 consensus_queue.cc:1048] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 13:01:27.778012  7102 raft_consensus.cc:2804] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 13:01:27.793684  7102 raft_consensus.cc:697] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 11 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:01:27.782684  6592 raft_consensus.cc:2749] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:01:27.794778  7102 consensus_queue.cc:237] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.6, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:27.795288  1717 leader_election.cc:304] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:27.795508  7106 consensus_queue.cc:1243] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer 816fadcea8fc4893882f5f9b9f22a03d log is divergent from this leader: its last log entry 15.12 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 9
I20260709 13:01:27.796661  1717 leader_election.cc:304] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:27.793206  1571 leader_election.cc:304] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:01:27.784638  6821 raft_consensus.cc:515] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:27.800103  1571 leader_election.cc:304] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:01:27.801061  6821 leader_election.cc:290] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:27.801374  1571 leader_election.cc:304] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:01:27.799096  7101 raft_consensus.cc:2749] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 13:01:27.779656  6714 raft_consensus.cc:2804] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:01:27.809168  1603 raft_consensus.cc:2468] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 12.
I20260709 13:01:27.847213  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:27.841511  7106 raft_consensus.cc:2749] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 13:01:27.752403  7109 raft_consensus.cc:515] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:27.810403  6867 raft_consensus.cc:2804] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:01:27.838773  6932 raft_consensus.cc:2804] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:01:27.839080  6977 raft_consensus.cc:2749] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
W20260709 13:01:27.857360  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 156523us
I20260709 13:01:27.842236  6714 raft_consensus.cc:493] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:27.810207  6976 raft_consensus.cc:2804] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:01:27.810034  6956 consensus_queue.cc:1059] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 11.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.056s
I20260709 13:01:27.850833  6821 raft_consensus.cc:2749] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260709 13:01:27.866222  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 85142.8us
W20260709 13:01:27.868912  1569 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:27.869078  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 72505.2us
I20260709 13:01:27.873639  1717 leader_election.cc:304] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:27.858404  6867 raft_consensus.cc:493] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:27.860195  6932 raft_consensus.cc:493] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:27.878149  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:27.862687  6926 raft_consensus.cc:2749] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:01:27.882725  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:27.875368  1646 leader_election.cc:304] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:27.867653  6714 raft_consensus.cc:3060] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:01:27.895004  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:27.888890  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:27.896770  1745 raft_consensus.cc:2410] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for term 13 because replica has last-logged OpId of term: 12 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 8.
I20260709 13:01:27.891755  1597 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 1239) took 1770 ms. Trace:
I20260709 13:01:27.897552  1597 rpcz_store.cc:276] 0709 13:01:26.120976 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:26.121129 (+   153us) service_pool.cc:224] Handling call
0709 13:01:26.121468 (+   339us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:26.129188 (+  7720us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:27.891485 (+1762297us) spinlock_profiling.cc:243] Waited 3.74 ms on lock 0x7b54002a3354. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:01:27.891679 (+   194us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":8597760}
I20260709 13:01:27.860637  6875 consensus_queue.cc:1048] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.016s
I20260709 13:01:27.899402  1678 raft_consensus.cc:1275] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 13 index: 12. Preceding OpId from leader: term: 15 index: 13. (index mismatch)
I20260709 13:01:27.869896  6976 raft_consensus.cc:493] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:27.907511  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:27.909873  1717 leader_election.cc:304] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:27.883226  6867 raft_consensus.cc:3060] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:01:27.884318  6932 raft_consensus.cc:3060] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:01:27.916479  1717 leader_election.cc:304] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:27.918826  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 17 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:27.919010  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:27.926051  1598 raft_consensus.cc:3060] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:01:27.926918  1717 leader_election.cc:304] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:27.919291  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:27.925064  1571 leader_election.cc:304] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:01:27.930784  1597 raft_consensus.cc:2376] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 12. Re-sending same reply.
I20260709 13:01:27.931898  1571 leader_election.cc:304] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:01:27.920980  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:27.919705  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:27.931968  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:27.933185  1571 leader_election.cc:304] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:01:27.934494  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:27.907092  6976 raft_consensus.cc:3060] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:01:27.940708  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:27.941449  1596 raft_consensus.cc:2468] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 13.
I20260709 13:01:27.936873  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:27.934664  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:27.942550  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:27.943562  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:27.944075  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:27.945812  1598 raft_consensus.cc:2468] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 14.
I20260709 13:01:27.904654  6643 raft_consensus.cc:515] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:27.904033  6345 consensus_queue.cc:1048] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.037s
I20260709 13:01:27.909423  7011 raft_consensus.cc:493] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:01:27.913703  6908 leader_election.cc:290] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:27.931636  7113 raft_consensus.cc:2804] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:01:27.913163  7120 raft_consensus.cc:2749] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:01:27.907800  7024 consensus_queue.cc:1048] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.038s
I20260709 13:01:27.947800  1601 pending_rounds.cc:85] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a: Aborting all ops after (but not including) 6
I20260709 13:01:27.941848  6977 raft_consensus.cc:2749] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:01:27.941615  7103 raft_consensus.cc:2749] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:01:27.946005  7106 raft_consensus.cc:2749] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:01:27.914647  6926 consensus_queue.cc:1048] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.042s
I20260709 13:01:27.978294  1601 pending_rounds.cc:107] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a: Aborting uncommitted NO_OP operation due to leader change: 9.7
I20260709 13:01:27.953100  7104 raft_consensus.cc:515] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:27.922634  6997 consensus_queue.cc:1059] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 13.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.044s
I20260709 13:01:27.954586  6821 raft_consensus.cc:2749] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:01:27.983423  1601 raft_consensus.cc:2889] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:01:27.964996  6345 consensus_queue.cc:1243] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer 83595ad0fd5940c0a2bfc7c61acd969a log is divergent from this leader: its last log entry 11.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260709 13:01:27.936499  7119 consensus_queue.cc:1048] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.056s
I20260709 13:01:27.978679  7113 raft_consensus.cc:493] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:27.941227  7110 consensus_queue.cc:1048] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.050s
W20260709 13:01:27.986805  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 59979.1us
I20260709 13:01:27.977653  6638 raft_consensus.cc:2749] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:01:27.990829  1717 leader_election.cc:304] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:27.994001  1747 raft_consensus.cc:1240] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 13. Current term is 14. Ops: []
I20260709 13:01:27.967734  7050 consensus_queue.cc:1048] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.020s
I20260709 13:01:27.986860  7120 raft_consensus.cc:493] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:27.976871  7011 raft_consensus.cc:515] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:27.996075  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 120893us
I20260709 13:01:28.043907  1748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43156 (request call id 1068) took 1152 ms. Trace:
I20260709 13:01:27.988921  7113 raft_consensus.cc:3060] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:01:28.042062  2299 leader_election.cc:304] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: af520b67945d443b901bbeca37cc5701
W20260709 13:01:28.007226  7122 log.cc:927] Time spent T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.071s	user 0.001s	sys 0.000s
W20260709 13:01:28.054231  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 58661.6us
I20260709 13:01:27.996994  7024 raft_consensus.cc:2804] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Leader election won for term 14
W20260709 13:01:28.046258  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 170944us
W20260709 13:01:28.045985  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 131087us
I20260709 13:01:28.065876  1646 leader_election.cc:304] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:01:28.041008  7120 raft_consensus.cc:515] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:28.011615  6908 raft_consensus.cc:2749] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:01:28.003621  6643 leader_election.cc:290] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 16 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:28.063530  1717 leader_election.cc:304] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:28.055603  1748 rpcz_store.cc:276] 0709 13:01:26.891973 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:26.892210 (+   237us) service_pool.cc:224] Handling call
0709 13:01:26.907486 (+ 15276us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:26.907989 (+   503us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 13:01:27.856288 (+948299us) log.cc:844] Serialized 14 byte log entry
0709 13:01:28.043479 (+187191us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:01:28.043546 (+    67us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:01:28.043601 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:28.043675 (+    74us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:28.043825 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":798397,"prepare.queue_time_us":874118,"prepare.run_cpu_time_us":52,"prepare.run_wall_time_us":54,"raft.queue_time_us":17585,"raft.run_cpu_time_us":140,"raft.run_wall_time_us":9178,"thread_start_us":301442,"threads_started":3,"wal-append.queue_time_us":110708}
I20260709 13:01:28.070037  7126 consensus_queue.cc:1048] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.009s
I20260709 13:01:28.073103  6987 consensus_queue.cc:1048] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 13:01:28.080123  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:28.080403  2337 raft_consensus.cc:1240] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 15. Current term is 16. Ops: []
I20260709 13:01:28.080893  2332 raft_consensus.cc:2468] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 11.
I20260709 13:01:28.080991  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:28.061400  7103 raft_consensus.cc:493] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:28.064677  6875 raft_consensus.cc:2804] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:01:28.090066  1605 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 2525) took 1248 ms. Trace:
I20260709 13:01:28.077193  6638 raft_consensus.cc:2749] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:01:28.115020  1602 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 2531) took 1262 ms. Trace:
I20260709 13:01:28.120502  1602 rpcz_store.cc:276] 0709 13:01:26.852180 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:26.940719 (+ 88539us) service_pool.cc:224] Handling call
0709 13:01:26.941079 (+   360us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:26.943798 (+  2719us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:26.943844 (+    46us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:01:26.977075 (+ 33231us) log.cc:844] Serialized 26 byte log entry
0709 13:01:27.893378 (+916303us) spinlock_profiling.cc:243] Waited 11 ms on lock 0x7b18003936a0. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f4263057e07 00007f426beb2ab9 00007f426307469b 00007f426307293f 00007f4263074c02 00007f4265cfc20b 00007f4265d01d23 00007f4265d020d1 00007f426611da0f
0709 13:01:28.085573 (+192195us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:01:28.085638 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:28.085734 (+    96us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:28.085811 (+    77us) raft_consensus.cc:1685] finished
0709 13:01:28.085829 (+    18us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:28.114918 (+ 29089us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2238,"raft.queue_time_us":5628,"raft.run_cpu_time_us":134,"raft.run_wall_time_us":5082,"spinlock_wait_cycles":25295616,"thread_start_us":191981,"threads_started":1,"wal-append.queue_time_us":1101396}
I20260709 13:01:28.123291  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:28.053252  7071 raft_consensus.cc:515] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:28.081702  6643 raft_consensus.cc:493] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:28.090529  1605 rpcz_store.cc:276] 0709 13:01:26.842083 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:27.799883 (+957800us) service_pool.cc:224] Handling call
0709 13:01:27.800266 (+   383us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:27.800688 (+   422us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:27.800722 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:01:27.801258 (+   536us) log.cc:844] Serialized 26 byte log entry
0709 13:01:28.023726 (+222468us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:01:28.067309 (+ 43583us) log.cc:844] Serialized 14 byte log entry
0709 13:01:28.067568 (+   259us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:28.067651 (+    83us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:28.089808 (+ 22157us) raft_consensus.cc:1685] finished
0709 13:01:28.089832 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:28.089991 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":27001,"prepare.queue_time_us":55197,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":49,"raft.queue_time_us":1897,"raft.run_cpu_time_us":284,"raft.run_wall_time_us":27897,"thread_start_us":236818,"threads_started":2,"wal-append.queue_time_us":276976}
I20260709 13:01:28.072228  7024 raft_consensus.cc:697] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:01:28.132920  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:28.117103  6908 raft_consensus.cc:493] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:28.130069  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:28.072532  6345 raft_consensus.cc:2749] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:01:28.130265  1717 leader_election.cc:304] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 13:01:28.088043  7078 log.cc:927] Time spent T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 1.433s	user 0.000s	sys 0.008s
I20260709 13:01:28.103843  7126 raft_consensus.cc:493] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:28.131220  6821 raft_consensus.cc:493] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:28.144876  1602 raft_consensus.cc:3060] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Advancing to term 13
W20260709 13:01:28.147877  7096 log.cc:927] Time spent T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 1.054s	user 0.007s	sys 0.000s
I20260709 13:01:28.148648  1600 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 2527) took 1299 ms. Trace:
I20260709 13:01:28.158844  1600 rpcz_store.cc:276] 0709 13:01:26.850792 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:27.970801 (+1120009us) service_pool.cc:224] Handling call
0709 13:01:27.971260 (+   459us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:27.971785 (+   525us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 13:01:27.980900 (+  9115us) log.cc:844] Serialized 14 byte log entry
0709 13:01:28.074473 (+ 93573us) log.cc:844] Serialized 14 byte log entry
0709 13:01:28.074697 (+   224us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:01:28.083851 (+  9154us) log.cc:844] Serialized 26 byte log entry
0709 13:01:28.084059 (+   208us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:01:28.084313 (+   254us) log.cc:844] Serialized 14 byte log entry
0709 13:01:28.084522 (+   209us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:28.084611 (+    89us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:28.148371 (+ 63760us) raft_consensus.cc:1685] finished
0709 13:01:28.148400 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:28.148572 (+   172us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":29090,"prepare.queue_time_us":9084,"raft.queue_time_us":6093,"raft.run_cpu_time_us":523,"raft.run_wall_time_us":28335,"spinlock_wait_cycles":9164672,"thread_start_us":50293,"threads_started":1,"wal-append.queue_time_us":99617}
I20260709 13:01:28.161300  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:28.090978  6875 raft_consensus.cc:493] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:28.153788  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:28.164283  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:28.165067  1605 raft_consensus.cc:2468] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 14.
I20260709 13:01:28.165930  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 15 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:28.153049  1717 leader_election.cc:304] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:28.167388  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:28.140050  6643 raft_consensus.cc:515] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:28.170998  1717 leader_election.cc:304] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:28.162678  6875 raft_consensus.cc:3060] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:01:28.093495  7103 raft_consensus.cc:515] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:28.127996  6926 raft_consensus.cc:493] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:28.177681  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
W20260709 13:01:28.178261  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 89850.3us
I20260709 13:01:28.153565  6638 raft_consensus.cc:2749] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:01:28.181906  1643 leader_election.cc:304] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:28.179919  1605 raft_consensus.cc:2468] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 14.
I20260709 13:01:28.190585  6643 leader_election.cc:290] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:28.191040  7011 leader_election.cc:290] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:28.194774  6638 raft_consensus.cc:2749] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 13:01:28.163018  6821 raft_consensus.cc:515] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:28.197536  6821 leader_election.cc:290] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:28.198836  6859 leader_election.cc:290] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:28.204988  1643 leader_election.cc:304] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:28.118880  6752 raft_consensus.cc:515] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:28.152249  6997 consensus_queue.cc:1059] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 12.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.036s
I20260709 13:01:28.211485  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:28.151708  6908 raft_consensus.cc:515] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:28.231074  1602 raft_consensus.cc:2468] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 13.
I20260709 13:01:28.178752  7135 raft_consensus.cc:2749] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
W20260709 13:01:28.206732  7105 consensus_meta.cc:287] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.545s	user 0.014s	sys 0.000s
I20260709 13:01:28.231990  1605 raft_consensus.cc:1240] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 12. Current term is 13. Ops: [12.7-12.7]
I20260709 13:01:28.209515  1643 leader_election.cc:304] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:01:28.151105  7137 consensus_queue.cc:1059] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 10.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.066s
I20260709 13:01:28.208058  7110 raft_consensus.cc:2749] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:01:28.238286  1597 raft_consensus.cc:2393] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 10: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:01:28.160064  7126 raft_consensus.cc:515] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:28.210579  7104 leader_election.cc:290] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 14 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:28.213718  6345 raft_consensus.cc:2749] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:01:28.243064  1643 leader_election.cc:304] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 13:01:28.232870  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 143369us
W20260709 13:01:28.261554  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 90626.6us
I20260709 13:01:28.202942  6926 raft_consensus.cc:515] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:28.263028  1717 leader_election.cc:304] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:28.239702  7105 raft_consensus.cc:2749] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:01:28.228827  6714 raft_consensus.cc:515] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:28.268419  7125 log.cc:927] Time spent T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.197s	user 0.000s	sys 0.001s
I20260709 13:01:28.268482  1717 leader_election.cc:304] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:28.299878  1750 raft_consensus.cc:2468] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 9.
W20260709 13:01:28.300334  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 57341.5us
I20260709 13:01:28.302613  1643 leader_election.cc:304] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
W20260709 13:01:28.279973  7131 log.cc:927] Time spent T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.161s	user 0.001s	sys 0.001s
I20260709 13:01:28.268179  7146 raft_consensus.cc:3055] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:01:28.272202  7128 raft_consensus.cc:2804] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:01:28.269073  7104 raft_consensus.cc:2804] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:01:28.305549  6987 raft_consensus.cc:2749] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:01:28.315184  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
W20260709 13:01:28.319612  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 51290.2us
I20260709 13:01:28.272468  6875 raft_consensus.cc:515] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:28.307304  7120 leader_election.cc:290] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:28.307741  7128 raft_consensus.cc:493] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Starting leader election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:01:28.308444  7146 raft_consensus.cc:740] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
W20260709 13:01:28.314992  7138 log.cc:927] Time spent T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.079s	user 0.001s	sys 0.000s
I20260709 13:01:28.320689  1717 leader_election.cc:304] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:28.331954  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:28.307621  6345 raft_consensus.cc:2749] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:01:28.308326  7105 raft_consensus.cc:2749] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:01:28.317917  6905 consensus_queue.cc:1059] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 15.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.015s
I20260709 13:01:28.336699  6932 raft_consensus.cc:515] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:28.311710  7104 raft_consensus.cc:493] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:28.354367  1717 leader_election.cc:304] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:28.377242  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:28.351794  7071 leader_election.cc:290] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:28.315728  7137 consensus_queue.cc:1048] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 9, Time since last communication: 0.038s
W20260709 13:01:28.374787  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 70349.8us
I20260709 13:01:28.348419  7128 raft_consensus.cc:3060] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:01:28.382510  1751 raft_consensus.cc:1240] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 15. Current term is 16. Ops: []
I20260709 13:01:28.352705  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:28.266688  7024 consensus_queue.cc:237] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.9, Last appended by leader: 9, Current term: 14, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:28.374490  7104 raft_consensus.cc:3060] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:01:28.382148  1752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43156 (request call id 1057) took 2102 ms. Trace:
W20260709 13:01:28.386399  1643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:28.359777  6714 leader_election.cc:290] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 12 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:28.378142  1601 raft_consensus.cc:2468] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 14.
I20260709 13:01:28.390306  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:28.391050  1745 raft_consensus.cc:2468] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 13.
I20260709 13:01:28.392308  1571 leader_election.cc:304] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:28.385253  1752 rpcz_store.cc:276] 0709 13:01:26.277602 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:26.278549 (+   947us) service_pool.cc:224] Handling call
0709 13:01:26.290931 (+ 12382us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:26.291403 (+   472us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:26.291459 (+    56us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:01:26.292132 (+   673us) log.cc:844] Serialized 26 byte log entry
0709 13:01:28.040613 (+1748481us) spinlock_profiling.cc:243] Waited 4.18 ms on lock 0x7b1800488620. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f4263057e07 00007f426beb2ab9 00007f426307469b 00007f426307293f 00007f4263074c02 00007f4265cfc20b 00007f4265d01d23 00007f4265d020d1 00007f426611da0f
0709 13:01:28.127454 (+ 86841us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:01:28.127515 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:28.127591 (+    76us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:28.375029 (+247438us) raft_consensus.cc:1685] finished
0709 13:01:28.375055 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:28.379510 (+  4455us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":3987,"raft.run_cpu_time_us":210,"raft.run_wall_time_us":40448,"spinlock_wait_cycles":9611008,"thread_start_us":86025,"threads_started":1,"wal-append.queue_time_us":1859318}
I20260709 13:01:28.380277  7148 raft_consensus.cc:2804] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 13:01:28.398964  7120 raft_consensus.cc:2804] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:01:28.423269  7101 raft_consensus.cc:2804] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:01:28.423913  7101 raft_consensus.cc:493] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:28.424294  7101 raft_consensus.cc:3060] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:01:28.397408  7146 consensus_queue.cc:260] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.7, Last appended by leader: 7, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:28.402206  7103 leader_election.cc:290] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:28.432051  6875 leader_election.cc:290] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 14 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:28.441949  1600 pending_rounds.cc:85] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a: Aborting all ops after (but not including) 5
I20260709 13:01:28.450889  1600 pending_rounds.cc:107] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a: Aborting uncommitted NO_OP operation due to leader change: 11.6
I20260709 13:01:28.451395  1600 raft_consensus.cc:2889] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 13:01:28.443244  7144 log.cc:927] Time spent T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.063s	user 0.000s	sys 0.000s
I20260709 13:01:28.404740  6908 leader_election.cc:290] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:28.441812  6932 leader_election.cc:290] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 13:01:28.503912  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 111689us
I20260709 13:01:28.441543  7126 leader_election.cc:290] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:28.425114  7148 raft_consensus.cc:697] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 13 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
W20260709 13:01:28.506279  6867 consensus_meta.cc:287] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.570s	user 0.016s	sys 0.000s
I20260709 13:01:28.501217  7146 raft_consensus.cc:3060] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:01:28.507356  7071 raft_consensus.cc:493] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:28.524104  7071 raft_consensus.cc:515] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:28.507243  7151 raft_consensus.cc:493] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:28.527597  7151 raft_consensus.cc:515] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:28.529954  7151 leader_election.cc:290] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:28.533448  7071 leader_election.cc:290] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:28.536020  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:28.534639  6867 raft_consensus.cc:515] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:28.544709  6867 leader_election.cc:290] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 15 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:28.548280  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:28.550349  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
W20260709 13:01:28.558787  1646 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:28.537465  6976 consensus_meta.cc:287] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.583s	user 0.016s	sys 0.000s
I20260709 13:01:28.445207  6884 consensus_queue.cc:260] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.12, Last appended by leader: 12, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:28.562500  1745 raft_consensus.cc:2393] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 14: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:01:28.566936  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:28.571316  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:28.571964  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:28.572252  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:28.572696  1750 raft_consensus.cc:3060] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:01:28.572049  7105 raft_consensus.cc:493] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:01:28.625011  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 44502.4us
I20260709 13:01:28.575188  6932 raft_consensus.cc:493] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:28.578888  6884 raft_consensus.cc:3060] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:01:28.561885  7148 consensus_queue.cc:237] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.7, Last appended by leader: 7, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:28.569546  6976 raft_consensus.cc:515] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:28.636859  1750 raft_consensus.cc:2468] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 15.
I20260709 13:01:28.640890  1646 leader_election.cc:304] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:28.632957  7105 raft_consensus.cc:515] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:28.663312  7164 raft_consensus.cc:2804] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Leader election won for term 15
I20260709 13:01:28.677629  7109 leader_election.cc:290] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:28.680884  6908 raft_consensus.cc:493] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:28.681919  6976 leader_election.cc:290] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:28.638554  6932 raft_consensus.cc:515] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:28.682307  7158 log.cc:927] Time spent T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.117s	user 0.000s	sys 0.001s
I20260709 13:01:28.681963  1602 raft_consensus.cc:1275] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20260709 13:01:28.699326  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:28.700035  1602 raft_consensus.cc:2468] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 14.
I20260709 13:01:28.693874  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:28.701931  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:28.702510  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:28.696739  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:28.704332  1754 raft_consensus.cc:3060] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:01:28.694746  1717 leader_election.cc:304] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:28.707012  7105 leader_election.cc:290] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 13:01:28.710165  1646 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:28.712443  2298 leader_election.cc:304] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:01:28.714582  7109 raft_consensus.cc:2804] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:01:28.715121  7109 raft_consensus.cc:493] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:28.717792  7135 consensus_queue.cc:1048] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 13:01:28.717131  7109 raft_consensus.cc:3060] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:01:28.720665  6979 consensus_queue.cc:1048] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 13:01:28.704581  7164 raft_consensus.cc:697] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 15 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:01:28.723265  6345 raft_consensus.cc:3055] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 LEADER]: Stepping down as leader of term 15
I20260709 13:01:28.723707  6345 raft_consensus.cc:740] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:01:28.723809  7164 consensus_queue.cc:237] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.11, Last appended by leader: 11, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:28.724732  6345 consensus_queue.cc:260] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.9, Last appended by leader: 9, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:28.728025  6932 leader_election.cc:290] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:28.729972  7007 raft_consensus.cc:2749] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 13:01:28.697829  7014 raft_consensus.cc:493] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:28.731513  7014 raft_consensus.cc:515] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:28.746788  6345 raft_consensus.cc:3060] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:01:28.703790  6908 raft_consensus.cc:515] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:28.757378  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:28.758368  1751 raft_consensus.cc:2393] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 11: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
W20260709 13:01:28.784837  7124 log.cc:927] Time spent T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.792s	user 0.002s	sys 0.000s
I20260709 13:01:28.787532  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:28.802719  1605 raft_consensus.cc:1240] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 10. Current term is 11. Ops: []
W20260709 13:01:28.808158  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 65963.5us
W20260709 13:01:28.845055  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 128644us
I20260709 13:01:28.846288  1717 leader_election.cc:304] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:28.850667  7102 raft_consensus.cc:2804] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Leader pre-election won for term 16
W20260709 13:01:28.851257  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 139042us
I20260709 13:01:28.852566  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:28.868120  1596 raft_consensus.cc:1240] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 13. Current term is 14. Ops: []
W20260709 13:01:28.869566  1571 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:28.869515  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:28.867180  7102 raft_consensus.cc:493] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:28.890051  1754 raft_consensus.cc:2468] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 10.
I20260709 13:01:28.870095  6867 raft_consensus.cc:493] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:01:28.851600  6592 consensus_queue.cc:1059] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 15.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.035s
I20260709 13:01:28.905975  6939 consensus_queue.cc:1059] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 10.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 13:01:28.907408  6939 raft_consensus.cc:3055] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:01:28.907868  6939 raft_consensus.cc:740] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:01:28.914487  6939 consensus_queue.cc:260] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:28.915504  7174 log.cc:927] Time spent T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.069s	user 0.000s	sys 0.005s
I20260709 13:01:28.915949  1747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55848 (request call id 2706) took 1451 ms. Trace:
I20260709 13:01:28.916436  1747 rpcz_store.cc:276] 0709 13:01:27.464845 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:27.657326 (+192481us) service_pool.cc:224] Handling call
0709 13:01:27.657649 (+   323us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:27.994592 (+336943us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:28.915630 (+921038us) spinlock_profiling.cc:243] Waited 334 ms on lock 0x7b5400220954. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:01:28.915869 (+   239us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":769276672}
I20260709 13:01:28.918911  6926 leader_election.cc:290] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:28.882196  7104 raft_consensus.cc:515] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:28.920455  6592 raft_consensus.cc:3055] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 LEADER]: Stepping down as leader of term 15
I20260709 13:01:28.920768  1604 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 1260) took 1260 ms. Trace:
I20260709 13:01:28.921538  1752 raft_consensus.cc:1240] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 14. Current term is 15. Ops: []
W20260709 13:01:28.923591  7113 consensus_meta.cc:287] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.852s	user 0.023s	sys 0.002s
I20260709 13:01:28.924083  7113 raft_consensus.cc:515] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:01:28.925304  7128 consensus_meta.cc:287] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.530s	user 0.014s	sys 0.000s
I20260709 13:01:28.927161  6908 leader_election.cc:290] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:28.875152  6898 consensus_queue.cc:1059] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 13.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.053s
I20260709 13:01:28.935662  1751 raft_consensus.cc:1240] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 13. Current term is 14. Ops: []
I20260709 13:01:28.936148  7101 raft_consensus.cc:515] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:28.895835  7102 raft_consensus.cc:3060] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:01:28.922605  6867 raft_consensus.cc:515] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:28.945088  6867 leader_election.cc:290] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:28.920917  6592 raft_consensus.cc:740] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:01:28.955233  6592 consensus_queue.cc:260] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.10, Last appended by leader: 10, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:28.956859  6592 raft_consensus.cc:3060] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:01:28.922233  7104 leader_election.cc:290] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 15 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:28.960775  1602 raft_consensus.cc:1240] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 13. Current term is 14. Ops: []
I20260709 13:01:28.962201  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:28.962986  1679 raft_consensus.cc:2468] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 12.
I20260709 13:01:28.964884  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:28.875082  7120 raft_consensus.cc:493] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:28.921453  1604 rpcz_store.cc:276] 0709 13:01:27.659977 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:27.695429 (+ 35452us) service_pool.cc:224] Handling call
0709 13:01:27.695744 (+   315us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:01:27.699638 (+  3894us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:27.699670 (+    32us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:01:27.721786 (+ 22116us) log.cc:844] Serialized 52 byte log entry
0709 13:01:28.494154 (+772368us) spinlock_profiling.cc:243] Waited 1.24 ms on lock 0x7b180033c7c0. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f4263057e07 00007f426beb2ab9 00007f426307469b 00007f426307293f 00007f4263074c02 00007f4265cfc20b 00007f4265d01d23 00007f4265d020d1 00007f426611da0f
0709 13:01:28.574584 (+ 80430us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:01:28.664657 (+ 90073us) log.cc:844] Serialized 14 byte log entry
0709 13:01:28.664916 (+   259us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:28.665001 (+    85us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:28.916836 (+251835us) spinlock_profiling.cc:243] Waited 3.35 ms on lock 0x7b180033c7c0. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f42630753f3 00007f42630777c1 00007f4263077779 00007f4263077709 00007f42630776d1 00007f42630769cd 000000000038f354 000000000038f199 00007f426305883a
0709 13:01:28.920477 (+  3641us) raft_consensus.cc:1685] finished
0709 13:01:28.920511 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:28.920689 (+   178us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":103679,"prepare.queue_time_us":91582,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":34,"raft.queue_time_us":7253,"raft.run_cpu_time_us":314,"raft.run_wall_time_us":18003,"spinlock_wait_cycles":10556160,"thread_start_us":80160,"threads_started":1,"wal-append.queue_time_us":917664}
I20260709 13:01:28.925806  7128 raft_consensus.cc:515] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:28.982640  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:28.984738  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:28.989147  1646 leader_election.cc:304] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:28.972602  7120 raft_consensus.cc:3060] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:01:29.004359  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:28.973043  6979 consensus_queue.cc:1059] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.033s
I20260709 13:01:28.992794  6867 consensus_queue.cc:1059] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 13.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.019s
I20260709 13:01:28.978569  7101 leader_election.cc:290] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 14 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:28.996312  7146 raft_consensus.cc:2804] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Leader election won for term 10
W20260709 13:01:29.014550  1643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:28.992949  7113 leader_election.cc:290] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:28.993160  6939 raft_consensus.cc:3060] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:01:28.972707  7050 consensus_queue.cc:1059] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 13.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.045s
I20260709 13:01:29.018714  7143 raft_consensus.cc:493] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:29.019632  7136 raft_consensus.cc:493] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:29.070153  1599 raft_consensus.cc:2410] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for term 15 because replica has last-logged OpId of term: 14 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 11.
I20260709 13:01:29.037889  7146 raft_consensus.cc:697] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
W20260709 13:01:29.101472  2299 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:29.111106  2299 leader_election.cc:304] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:29.114429  1747 raft_consensus.cc:1275] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 12 index: 9. Preceding OpId from leader: term: 14 index: 10. (index mismatch)
W20260709 13:01:29.112519  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 102730us
I20260709 13:01:29.117654  6867 consensus_queue.cc:1048] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 13:01:29.031666  7109 raft_consensus.cc:515] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:29.086808  7179 raft_consensus.cc:3055] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 13:01:29.079614  7143 raft_consensus.cc:515] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:29.167209  6979 raft_consensus.cc:493] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:29.078608  7136 raft_consensus.cc:515] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:29.189932  7179 raft_consensus.cc:740] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:01:29.198824  7180 raft_consensus.cc:2804] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Leader election won for term 16
W20260709 13:01:29.229447  1646 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:29.214648  7184 consensus_queue.cc:1048] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 10, Time since last communication: 0.032s
I20260709 13:01:29.254739  7143 leader_election.cc:290] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:29.205127  6979 raft_consensus.cc:515] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:29.231858  7128 leader_election.cc:290] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 14 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:29.240132  7181 raft_consensus.cc:493] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:29.277931  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:29.278151  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:29.263356  7180 raft_consensus.cc:697] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 16 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
W20260709 13:01:29.304262  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 156455us
I20260709 13:01:29.305183  2299 leader_election.cc:304] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: af520b67945d443b901bbeca37cc5701
W20260709 13:01:29.293648  7173 log.cc:927] Time spent T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.250s	user 0.001s	sys 0.000s
I20260709 13:01:29.228314  7146 consensus_queue.cc:237] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:29.273622  7102 raft_consensus.cc:515] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:29.314785  7189 raft_consensus.cc:493] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:29.267963  7179 consensus_queue.cc:260] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.11, Last appended by leader: 11, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:29.345307  1753 raft_consensus.cc:2393] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 16: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:01:29.350751  7055 raft_consensus.cc:2804] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:01:29.351423  7055 raft_consensus.cc:493] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Starting leader election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:01:29.351883  7055 raft_consensus.cc:3060] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:01:29.352311  7179 raft_consensus.cc:3060] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:01:29.356178  6979 leader_election.cc:290] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:29.294400  7181 raft_consensus.cc:515] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:29.359324  7181 leader_election.cc:290] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:29.361430  7180 consensus_queue.cc:237] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.12, Last appended by leader: 12, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:29.363706  7115 log.cc:927] Time spent T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 1.381s	user 0.000s	sys 0.002s
I20260709 13:01:29.365170  1749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43156 (request call id 1075) took 1754 ms. Trace:
I20260709 13:01:29.365957  1644 leader_election.cc:304] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:29.336617  7109 leader_election.cc:290] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 15 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:29.365574  1749 rpcz_store.cc:276] 0709 13:01:27.610413 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:27.675566 (+ 65153us) service_pool.cc:224] Handling call
0709 13:01:27.675866 (+   300us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:27.676285 (+   419us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:27.676316 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:01:27.676869 (+   553us) log.cc:844] Serialized 26 byte log entry
0709 13:01:27.870924 (+194055us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:01:27.870991 (+    67us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:27.871087 (+    96us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:29.363068 (+1491981us) spinlock_profiling.cc:243] Waited 55.9 ms on lock 0x7b4400022b4c. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f4262f9bc85 00007f4262f9c6a2 00007f4265d0e1db 00007f4265d09aaf 00007f4265cfd86e 00007f4265cfcbb6 00007f4265cfc7d8 00007f4265d06221 00007f4265d061d9 00007f4265d06169
0709 13:01:29.364825 (+  1757us) raft_consensus.cc:1685] finished
0709 13:01:29.364847 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:29.365088 (+   241us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":98,"raft.run_cpu_time_us":171,"raft.run_wall_time_us":3294,"spinlock_wait_cycles":128645376,"thread_start_us":173632,"threads_started":1,"wal-append.queue_time_us":268749}
W20260709 13:01:29.371196  7187 log.cc:927] Time spent T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.145s	user 0.001s	sys 0.000s
I20260709 13:01:29.382254  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 15 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:29.383029  1604 raft_consensus.cc:2468] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 14.
I20260709 13:01:29.339265  7189 raft_consensus.cc:515] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:29.385012  1748 raft_consensus.cc:1240] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 13. Current term is 14. Ops: []
I20260709 13:01:29.400266  7189 leader_election.cc:290] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:29.384850  7194 raft_consensus.cc:2749] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:01:29.408463  6983 raft_consensus.cc:3060] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:01:29.401059  6867 consensus_queue.cc:1048] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.023s
W20260709 13:01:29.432179  7176 log.cc:927] Time spent T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.363s	user 0.000s	sys 0.001s
W20260709 13:01:29.436287  1599 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40166 (request call id 2544) took 1517 ms (client timeout 1762 ms). Trace:
W20260709 13:01:29.436924  1599 rpcz_store.cc:269] 0709 13:01:27.919266 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:27.919441 (+   175us) service_pool.cc:224] Handling call
0709 13:01:29.435956 (+1516515us) spinlock_profiling.cc:243] Waited 1.14 s on lock 0x7b54002170d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:01:29.436190 (+   234us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2614264192}
I20260709 13:01:29.445184  1598 raft_consensus.cc:1275] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 15 index: 12. (term mismatch)
I20260709 13:01:29.445693  1598 pending_rounds.cc:85] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a: Aborting all ops after (but not including) 11
I20260709 13:01:29.445983  1598 pending_rounds.cc:107] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a: Aborting uncommitted NO_OP operation due to leader change: 14.12
I20260709 13:01:29.446238  1598 raft_consensus.cc:2889] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:01:29.454121  7102 leader_election.cc:290] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:29.472365  7106 raft_consensus.cc:493] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:29.473052  7106 raft_consensus.cc:515] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:29.449229  6714 raft_consensus.cc:493] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:29.449021  7153 raft_consensus.cc:493] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:01:29.486742  6714 raft_consensus.cc:515] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:29.465931  7055 raft_consensus.cc:515] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:29.516119  6714 leader_election.cc:290] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:29.519043  1674 raft_consensus.cc:2468] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 12.
I20260709 13:01:29.491910  7153 raft_consensus.cc:515] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:01:29.522511  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 205394us
I20260709 13:01:29.519940  6905 raft_consensus.cc:493] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:01:29.525274  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 48396.3us
W20260709 13:01:29.521432  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 216310us
I20260709 13:01:29.526808  2299 leader_election.cc:304] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:01:29.527225  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:29.527979  2331 raft_consensus.cc:2468] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 14.
I20260709 13:01:29.528074  7208 raft_consensus.cc:493] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:29.528637  7202 raft_consensus.cc:2749] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 13:01:29.529337  2299 leader_election.cc:304] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:29.529642  6905 raft_consensus.cc:515] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:29.531522  7202 raft_consensus.cc:2749] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:01:29.532271  7153 leader_election.cc:290] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:29.532927  7106 leader_election.cc:290] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:01:29.535292  7120 consensus_meta.cc:287] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.514s	user 0.013s	sys 0.002s
I20260709 13:01:29.536278  2330 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39738 (request call id 1007) took 1477 ms. Trace:
I20260709 13:01:29.539784  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:29.540321  2330 rpcz_store.cc:276] 0709 13:01:28.058507 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:28.058678 (+   171us) service_pool.cc:224] Handling call
0709 13:01:28.059060 (+   382us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:28.063925 (+  4865us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:28.063956 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:01:28.064535 (+   579us) log.cc:844] Serialized 26 byte log entry
0709 13:01:29.387332 (+1322797us) spinlock_profiling.cc:243] Waited 23 ms on lock 0x7b18004875a0. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f4263057e07 00007f426beb2ab9 00007f426307469b 00007f426307293f 00007f4263074c02 00007f4265cfc20b 00007f4265d01d23 00007f4265d020d1 00007f426611da0f
0709 13:01:29.485200 (+ 97868us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:01:29.485265 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:29.485372 (+   107us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:29.535754 (+ 50382us) raft_consensus.cc:1685] finished
0709 13:01:29.535779 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:29.536196 (+   417us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4434,"raft.queue_time_us":16821,"raft.run_cpu_time_us":236,"raft.run_wall_time_us":13364,"spinlock_wait_cycles":52997376,"thread_start_us":97566,"threads_started":1,"wal-append.queue_time_us":1444865}
I20260709 13:01:29.543191  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:29.543128  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:29.545639  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:29.546015  7106 raft_consensus.cc:493] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:29.546262  2333 raft_consensus.cc:3060] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:01:29.546852  7106 raft_consensus.cc:515] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:29.550922  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:01:29.552852  2299 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:29.555414  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:29.556948  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:29.563882  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:29.577543  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:29.578935  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 15 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:29.584908  2298 leader_election.cc:304] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:01:29.585975  2330 raft_consensus.cc:1275] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 15 index: 12. (index mismatch)
I20260709 13:01:29.556121  6699 raft_consensus.cc:493] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:29.551379  7120 raft_consensus.cc:515] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:29.595794  2333 raft_consensus.cc:2468] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 15.
I20260709 13:01:29.590978  6699 raft_consensus.cc:515] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:29.585680  7106 leader_election.cc:290] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:29.599116  1747 raft_consensus.cc:1240] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 14. Current term is 15. Ops: []
I20260709 13:01:29.599934  6699 leader_election.cc:290] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:29.600584  6699 raft_consensus.cc:493] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:01:29.601011  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:29.601243  6699 raft_consensus.cc:515] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:29.616911  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:29.617828  2330 raft_consensus.cc:3060] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:01:29.618418  1644 leader_election.cc:304] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:01:29.620951  6983 raft_consensus.cc:2749] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:01:29.622350  1644 leader_election.cc:304] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:29.622488  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:29.623383  2332 raft_consensus.cc:2393] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 12: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:01:29.624073  7153 raft_consensus.cc:2749] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:01:29.624570  6997 raft_consensus.cc:3055] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:01:29.624970  6997 raft_consensus.cc:740] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:01:29.625931  1644 leader_election.cc:304] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 13:01:29.626356  7178 log.cc:927] Time spent T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.557s	user 0.001s	sys 0.001s
I20260709 13:01:29.625794  6997 consensus_queue.cc:260] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.8, Last appended by leader: 8, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:29.589977  7180 raft_consensus.cc:2749] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 13:01:29.628087  1674 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57092 (request call id 1191) took 1953 ms. Trace:
I20260709 13:01:29.628571  2330 raft_consensus.cc:2468] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 12.
I20260709 13:01:29.630970  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:29.632155  1644 leader_election.cc:304] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:29.632841  6983 raft_consensus.cc:2749] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:01:29.636301  7146 raft_consensus.cc:2749] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:01:29.628511  1674 rpcz_store.cc:276] 0709 13:01:27.674780 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:27.675910 (+  1130us) service_pool.cc:224] Handling call
0709 13:01:29.627867 (+1951957us) spinlock_profiling.cc:243] Waited 1.83 s on lock 0x7b54000230d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:01:29.628000 (+   133us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4220422016}
I20260709 13:01:29.629966  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:29.640785  7055 leader_election.cc:290] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 13 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:29.641405  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:29.642074  1598 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 2566) took 1007 ms. Trace:
I20260709 13:01:29.642673  1674 raft_consensus.cc:3060] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:01:29.642896  7067 raft_consensus.cc:493] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:29.642123  7106 raft_consensus.cc:493] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:29.643807  7067 raft_consensus.cc:515] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:29.644330  7106 raft_consensus.cc:515] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:29.644831  7165 log.cc:927] Time spent T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.966s	user 0.002s	sys 0.000s
I20260709 13:01:29.642683  1598 rpcz_store.cc:276] 0709 13:01:28.635059 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:28.736337 (+101278us) service_pool.cc:224] Handling call
0709 13:01:29.442568 (+706231us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:29.446830 (+  4262us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:29.641878 (+195048us) spinlock_profiling.cc:243] Waited 706 ms on lock 0x7b54002170d0. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661618ad 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:01:29.642006 (+   128us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1623047040}
I20260709 13:01:29.647431  7067 leader_election.cc:290] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:29.648516  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:29.648808  7136 leader_election.cc:290] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:29.648731  6905 leader_election.cc:290] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:29.654117  7106 leader_election.cc:290] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:27.738101   431 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:01:29.664945  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:29.665860  1748 raft_consensus.cc:2393] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 16: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:01:29.666885  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:29.668428  1748 raft_consensus.cc:1275] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 13 index: 11. (index mismatch)
I20260709 13:01:29.668918  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:29.671407  2299 leader_election.cc:304] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:01:29.673064  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:29.673241  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:29.673646  1750 raft_consensus.cc:3060] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:01:29.674091  2332 raft_consensus.cc:2468] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 12.
I20260709 13:01:29.658116  1501 catalog_manager.cc:5697] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d reported cstate change: term changed from 4 to 9, leader changed from af520b67945d443b901bbeca37cc5701 (127.0.107.195) to 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193). New cstate: current_term: 9 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } }
I20260709 13:01:29.676469  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:29.678843  1644 leader_election.cc:304] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:01:29.680856  1674 raft_consensus.cc:2468] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 13.
I20260709 13:01:29.682238  2299 leader_election.cc:304] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:01:29.682816  1644 leader_election.cc:304] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:01:29.687866  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:29.694154  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:29.528651  7208 raft_consensus.cc:515] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:29.702915  1571 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:29.651881  1718 connection.cc:256] RPC call timeout handler was delayed by -1.5286342629999581s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15780000000000002s for a response
I20260709 13:01:29.723873  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
W20260709 13:01:29.727133  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 44966.2us
I20260709 13:01:29.730545  2301 leader_election.cc:304] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:01:29.731797  1596 raft_consensus.cc:1240] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 13. Current term is 14. Ops: []
I20260709 13:01:29.733062  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:29.734203  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:29.734935  1604 raft_consensus.cc:2468] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 14.
I20260709 13:01:29.707577  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 15 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:29.741402  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:29.743789  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:29.749521  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:29.749490  2301 leader_election.cc:304] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:01:29.751358  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:29.703801  7036 raft_consensus.cc:2804] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:01:29.692354  7136 raft_consensus.cc:2749] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:01:29.763597  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:29.765022  2301 leader_election.cc:304] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
W20260709 13:01:29.749430  7130 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.067s	user 0.000s	sys 0.012s
I20260709 13:01:29.705657  6905 raft_consensus.cc:2804] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:01:29.801419  6905 raft_consensus.cc:493] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:29.802032  6905 raft_consensus.cc:3060] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:01:29.806447  6699 leader_election.cc:290] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:29.701764  7191 consensus_queue.cc:1048] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.066s
I20260709 13:01:29.762650  7009 raft_consensus.cc:2749] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:01:29.728410  7067 raft_consensus.cc:493] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:29.798278  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:29.706496  7159 raft_consensus.cc:493] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:29.808656  7067 raft_consensus.cc:515] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:29.809692  1596 raft_consensus.cc:2468] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 14.
I20260709 13:01:29.810745  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:29.761993  7036 raft_consensus.cc:493] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:29.705662  6914 raft_consensus.cc:493] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:29.790378  6160 consensus_queue.cc:1059] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.005s
I20260709 13:01:29.727757  7111 raft_consensus.cc:493] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:01:29.768813  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
W20260709 13:01:29.808207  7130 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.125s	user 0.000s	sys 0.014s
I20260709 13:01:29.808755  7159 raft_consensus.cc:515] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:29.799880  1718 leader_election.cc:304] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:29.706763  7106 raft_consensus.cc:493] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:29.729895  6714 raft_consensus.cc:493] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:29.749589  7180 raft_consensus.cc:2804] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Leader election won for term 13
W20260709 13:01:29.849079  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 215227us
I20260709 13:01:29.796635  2301 leader_election.cc:304] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:01:29.806352  1750 raft_consensus.cc:2468] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 14.
I20260709 13:01:29.819967  1599 raft_consensus.cc:3060] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Advancing to term 16
W20260709 13:01:29.852191  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 55673.8us
I20260709 13:01:29.792369  6875 raft_consensus.cc:2749] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:01:29.798911  7179 raft_consensus.cc:2749] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20260709 13:01:29.819380  7207 log.cc:927] Time spent T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.070s	user 0.000s	sys 0.000s
W20260709 13:01:29.832696  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 73759.8us
I20260709 13:01:29.816951  7036 raft_consensus.cc:3060] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:01:29.816115  7067 leader_election.cc:290] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:29.825529  7109 raft_consensus.cc:3055] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 13:01:29.826169  7111 raft_consensus.cc:515] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:29.861613  6592 raft_consensus.cc:493] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:29.821101  6914 raft_consensus.cc:515] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:29.867818  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:29.887705  2301 leader_election.cc:304] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:01:29.862730  7186 raft_consensus.cc:2749] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:01:29.861810  7146 raft_consensus.cc:493] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:29.868992  7180 raft_consensus.cc:697] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 13 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:01:29.872493  7109 raft_consensus.cc:740] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:01:29.854357  7106 raft_consensus.cc:515] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } }
I20260709 13:01:29.864022  6714 raft_consensus.cc:515] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:29.893144  7055 raft_consensus.cc:2749] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:01:29.892930  7199 raft_consensus.cc:493] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:29.895617  7191 consensus_queue.cc:1048] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.060s
I20260709 13:01:29.939030  7146 raft_consensus.cc:515] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:29.964403  7159 leader_election.cc:290] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:29.959416  7119 raft_consensus.cc:3055] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 LEADER]: Stepping down as leader of term 13
I20260709 13:01:29.964084  7180 consensus_queue.cc:237] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.10, Last appended by leader: 10, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:29.964598  7199 raft_consensus.cc:515] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:29.969535  7208 leader_election.cc:290] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:29.979166  7222 raft_consensus.cc:493] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
W20260709 13:01:29.979631  7215 log.cc:927] Time spent T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.116s	user 0.001s	sys 0.000s
I20260709 13:01:29.979931  7222 raft_consensus.cc:515] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:29.984619  7222 leader_election.cc:290] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:29.836875  1597 raft_consensus.cc:1275] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
I20260709 13:01:29.867027  1571 leader_election.cc:304] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:30.041797  1598 raft_consensus.cc:3060] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Advancing to term 14
W20260709 13:01:30.043164  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 175996us
I20260709 13:01:30.046141  1597 raft_consensus.cc:1275] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 15 index: 13. Preceding OpId from leader: term: 16 index: 13. (term mismatch)
I20260709 13:01:30.047156  1597 pending_rounds.cc:85] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a: Aborting all ops after (but not including) 12
I20260709 13:01:30.048454  1597 pending_rounds.cc:107] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a: Aborting uncommitted NO_OP operation due to leader change: 15.13
I20260709 13:01:30.049590  1597 raft_consensus.cc:2889] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:01:30.088171  1643 leader_election.cc:304] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
W20260709 13:01:30.111581  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 282837us
I20260709 13:01:30.114037  1718 leader_election.cc:304] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:30.118501  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:30.122148  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:30.123652  1750 raft_consensus.cc:2393] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 16: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:01:30.140270  1717 leader_election.cc:304] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:30.143067  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:30.144245  1748 raft_consensus.cc:2468] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 11.
I20260709 13:01:30.147006  1571 leader_election.cc:304] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:30.161607  1598 raft_consensus.cc:2468] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 14.
W20260709 13:01:30.243685  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 103797us
W20260709 13:01:30.249393  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 135682us
W20260709 13:01:30.248733  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 158971us
W20260709 13:01:30.257035  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 110189us
I20260709 13:01:29.963845  7109 consensus_queue.cc:260] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.11, Last appended by leader: 11, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:29.970250  7119 raft_consensus.cc:740] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:01:30.273240  7119 consensus_queue.cc:260] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 13.9, Last appended by leader: 9, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:30.286660  7123 consensus_queue.cc:1048] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 13:01:30.307292  7067 raft_consensus.cc:2804] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Leader pre-election won for term 12
W20260709 13:01:30.308023  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 146789us
I20260709 13:01:30.305612  7219 raft_consensus.cc:2804] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:01:30.280531  7218 raft_consensus.cc:493] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:30.374534  1718 leader_election.cc:304] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:29.892748  6592 raft_consensus.cc:515] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:30.387110  1571 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:30.394882  7219 raft_consensus.cc:697] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
W20260709 13:01:30.405704  1747 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.597s	user 0.004s	sys 0.012s
W20260709 13:01:30.406474  1747 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.738s	user 0.004s	sys 0.012s
I20260709 13:01:30.406788  7218 raft_consensus.cc:515] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:30.411526  7188 raft_consensus.cc:2804] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:01:30.412271  7188 raft_consensus.cc:493] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:30.412806  7188 raft_consensus.cc:3060] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:01:30.417133  7109 raft_consensus.cc:3060] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:01:30.421538  7222 raft_consensus.cc:2804] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:01:30.422382  7159 raft_consensus.cc:493] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:30.423714  7159 raft_consensus.cc:515] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:01:30.426461  1603 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36462 (request call id 1255) took 2954 ms (client timeout 2014 ms). Trace:
W20260709 13:01:30.427048  1603 rpcz_store.cc:269] 0709 13:01:27.471887 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:27.472760 (+   873us) service_pool.cc:224] Handling call
0709 13:01:30.425890 (+2953130us) spinlock_profiling.cc:243] Waited 257 ms on lock 0x7b54000acb54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:01:30.426158 (+   268us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":590792448}
I20260709 13:01:30.430102  1752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55848 (request call id 2722) took 2085 ms. Trace:
I20260709 13:01:30.432945  1675 raft_consensus.cc:1217] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 LEADER]: Deduplicated request from leader. Original: 8.9->[11.10-14.12]   Dedup: 12.11->[14.12-14.12]
I20260709 13:01:30.433665  1675 raft_consensus.cc:1240] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 LEADER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 14. Current term is 15. Ops: [11.10-14.12]
I20260709 13:01:30.434818  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:30.435356  7126 raft_consensus.cc:2804] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:01:30.435958  7126 raft_consensus.cc:493] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:30.436332  7126 raft_consensus.cc:3060] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Advancing to term 15
W20260709 13:01:30.439497  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 126076us
I20260709 13:01:30.440913  1718 leader_election.cc:304] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:30.430691  1752 rpcz_store.cc:276] 0709 13:01:28.344531 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:28.422961 (+ 78430us) service_pool.cc:224] Handling call
0709 13:01:28.428987 (+  6026us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:28.922057 (+493070us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:30.429462 (+1507405us) spinlock_profiling.cc:243] Waited 492 ms on lock 0x7b54000e4f54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:01:30.429858 (+   396us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1132135168}
I20260709 13:01:30.444520  1601 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 2559) took 2058 ms. Trace:
I20260709 13:01:30.381371  7067 raft_consensus.cc:493] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:30.447651  7067 raft_consensus.cc:3060] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:01:30.448033  7222 raft_consensus.cc:697] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 12 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:01:30.449855  7120 leader_election.cc:290] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 15 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:30.447809  7219 consensus_queue.cc:237] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:30.457042  7189 raft_consensus.cc:493] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:01:30.470038  7111 leader_election.cc:290] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:30.449743  7222 consensus_queue.cc:237] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:30.465369  6592 leader_election.cc:290] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:30.466389  7227 raft_consensus.cc:493] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
W20260709 13:01:30.472437  7213 log.cc:927] Time spent T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.524s	user 0.000s	sys 0.006s
W20260709 13:01:30.460639  7163 log.cc:927] Time spent T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 1.790s	user 0.004s	sys 0.000s
I20260709 13:01:30.473213  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:30.537933  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:30.565518  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:30.566968  6752 leader_election.cc:290] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 10 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:30.568213  6752 raft_consensus.cc:493] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:30.569046  6752 raft_consensus.cc:515] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:30.575639  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:30.577220  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:30.444972  1601 rpcz_store.cc:276] 0709 13:01:28.385683 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:28.389685 (+  4002us) service_pool.cc:224] Handling call
0709 13:01:28.390040 (+   355us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:28.390464 (+   424us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:28.390498 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:01:28.391016 (+   518us) log.cc:844] Serialized 26 byte log entry
0709 13:01:29.619759 (+1228743us) spinlock_profiling.cc:243] Waited 2.6 ms on lock 0x7b1800280780. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f4263057e07 00007f426beb2ab9 00007f426307469b 00007f426307293f 00007f4263074c02 00007f4265cfc20b 00007f4265d01d23 00007f4265d020d1 00007f426611da0f
0709 13:01:29.746008 (+126249us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:01:30.051536 (+305528us) log.cc:844] Serialized 14 byte log entry
0709 13:01:30.369779 (+318243us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:30.395787 (+ 26008us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:30.443964 (+ 48177us) raft_consensus.cc:1685] finished
0709 13:01:30.444106 (+   142us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:30.444295 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":15,"prepare.queue_time_us":70825,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":48,"raft.queue_time_us":218327,"raft.run_cpu_time_us":659,"raft.run_wall_time_us":59203,"spinlock_wait_cycles":5980544,"thread_start_us":572007,"threads_started":3,"wal-append.queue_time_us":1410491}
I20260709 13:01:30.583595  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:30.586711  1676 raft_consensus.cc:2393] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 14: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:01:30.528668  7218 leader_election.cc:290] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:30.577308  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:01:30.590505  7163 log.cc:927] Time spent T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.055s	user 0.001s	sys 0.000s
I20260709 13:01:30.608673  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 16 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:30.624936  1677 raft_consensus.cc:2468] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 16.
I20260709 13:01:30.605122  7119 raft_consensus.cc:3060] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Advancing to term 14
W20260709 13:01:30.642858  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 373692us
I20260709 13:01:30.591480  7111 raft_consensus.cc:493] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:30.640208  1674 raft_consensus.cc:2468] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 16.
I20260709 13:01:30.597529  7120 raft_consensus.cc:2749] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:01:30.604233  7233 raft_consensus.cc:493] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:30.672511  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:01:30.644662  7162 log.cc:927] Time spent T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 1.813s	user 0.006s	sys 0.000s
I20260709 13:01:30.674206  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:30.675644  7199 leader_election.cc:290] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:30.676038  7014 leader_election.cc:290] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:30.681030  7234 raft_consensus.cc:493] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:01:30.686594  7050 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.537s	user 0.020s	sys 0.055s
I20260709 13:01:30.689041  1596 raft_consensus.cc:1240] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 13. Current term is 14. Ops: []
I20260709 13:01:30.693708  7236 raft_consensus.cc:493] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:30.695335  2334 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39726 (request call id 1264) took 1146 ms. Trace:
I20260709 13:01:30.695623  1678 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57092 (request call id 1211) took 1142 ms. Trace:
I20260709 13:01:30.597864  7189 raft_consensus.cc:515] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:30.626614  7188 raft_consensus.cc:515] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:30.717859  1501 catalog_manager.cc:5697] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d reported cstate change: term changed from 10 to 12. New cstate: current_term: 12 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } }
I20260709 13:01:30.727607  7232 raft_consensus.cc:2749] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260709 13:01:30.728114  6905 consensus_meta.cc:287] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.925s	user 0.023s	sys 0.008s
I20260709 13:01:30.729645  6714 leader_election.cc:290] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:30.730453  6914 leader_election.cc:290] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:30.730698  7146 leader_election.cc:290] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:30.670162  7218 raft_consensus.cc:493] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:30.767143  7218 raft_consensus.cc:515] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:30.768868  7014 raft_consensus.cc:493] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:30.769651  7014 raft_consensus.cc:515] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:30.730919  7236 raft_consensus.cc:515] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:30.738622  7111 raft_consensus.cc:515] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:30.720098  2334 rpcz_store.cc:276] 0709 13:01:29.548980 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:29.549118 (+   138us) service_pool.cc:224] Handling call
0709 13:01:29.646149 (+ 97031us) spinlock_profiling.cc:243] Waited 1.31 ms on lock 0x7b5800178598. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f426303db71 00007f426303da22 00007f426bf56c09 00007f426b0b0ee0 00007f426b17a0ab 00007f426b17a131 00007f426b13bfc0 00007f426b126929 00007f426b1263d2 00007f426469dbd4
0709 13:01:29.646371 (+   222us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:29.795162 (+148791us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:29.795212 (+    50us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:01:29.795280 (+    68us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:01:29.795333 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:30.694700 (+899367us) spinlock_profiling.cc:243] Waited 148 ms on lock 0x7b540065e0d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:01:30.694790 (+    90us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:30.695075 (+   285us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":344518272}
I20260709 13:01:30.619163  7227 raft_consensus.cc:515] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:01:30.719173  7050 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.645s	user 0.020s	sys 0.055s
I20260709 13:01:30.738655  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:30.771682  2333 raft_consensus.cc:1275] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 12 index: 9. Preceding OpId from leader: term: 14 index: 10. (index mismatch)
I20260709 13:01:30.775372  1746 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55848 (request call id 2724) took 2266 ms. Trace:
I20260709 13:01:30.770074  7218 leader_election.cc:290] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:30.773984  1600 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 2560) took 2387 ms. Trace:
I20260709 13:01:30.720177  1678 rpcz_store.cc:276] 0709 13:01:29.553483 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:29.553698 (+   215us) service_pool.cc:224] Handling call
0709 13:01:29.554149 (+   451us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:29.554486 (+   337us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:29.554523 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:01:29.585076 (+ 30553us) log.cc:844] Serialized 26 byte log entry
0709 13:01:29.769613 (+184537us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:01:29.769661 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:29.769754 (+    93us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:30.471288 (+701534us) spinlock_profiling.cc:243] Waited 3.06 ms on lock 0x7b4400022b4c. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f4262f9bc85 00007f4262f9c6a2 00007f4265d0e1db 00007f4265d09aaf 00007f4265cfd86e 00007f4265cfcbb6 00007f4265cfc7d8 00007f4265d06221 00007f4265d061d9 00007f4265d06169
0709 13:01:30.695018 (+223730us) raft_consensus.cc:1685] finished
0709 13:01:30.695050 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:30.695384 (+   334us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":7031168,"thread_start_us":174648,"threads_started":1,"wal-append.queue_time_us":279051}
I20260709 13:01:30.772773  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:30.774811  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
W20260709 13:01:30.806452  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 89911.7us
I20260709 13:01:30.802274  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:30.772104  7106 leader_election.cc:290] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:30.790086  1746 rpcz_store.cc:276] 0709 13:01:28.509045 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:28.509212 (+   167us) service_pool.cc:224] Handling call
0709 13:01:28.509551 (+   339us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:28.509972 (+   421us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 13:01:28.520878 (+ 10906us) log.cc:844] Serialized 14 byte log entry
0709 13:01:28.575745 (+ 54867us) log.cc:844] Serialized 14 byte log entry
0709 13:01:28.576422 (+   677us) log.cc:844] Serialized 14 byte log entry
0709 13:01:28.577076 (+   654us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:01:28.631124 (+ 54048us) log.cc:844] Serialized 26 byte log entry
0709 13:01:28.631715 (+   591us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:01:28.631859 (+   144us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:28.632251 (+   392us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:28.911239 (+278988us) spinlock_profiling.cc:243] Waited 1.99 ms on lock 0x7b1800457060. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f42630753f3 00007f42630777c1 00007f4263077779 00007f4263077709 00007f42630776d1 00007f42630769cd 000000000038f354 000000000038f199 00007f426305883a
0709 13:01:29.421992 (+510753us) spinlock_profiling.cc:243] Waited 5.25 ms on lock 0x7b1800457060. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f4263075522 00007f42630777c1 00007f4263077779 00007f4263077709 00007f42630776d1 00007f42630769cd 000000000038f354 000000000038f199 00007f426305883a
0709 13:01:30.379121 (+957129us) spinlock_profiling.cc:243] Waited 2.37 ms on lock 0x7b1800457060. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f42630753f3 00007f42630777c1 00007f4263077779 00007f4263077709 00007f42630776d1 00007f42630769cd 000000000038f354 000000000038f199 00007f426305883a
0709 13:01:30.459340 (+ 80219us) spinlock_profiling.cc:243] Waited 20.4 ms on lock 0x7b4400022b4c. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f4262f9bc85 00007f4262f9c6a2 00007f4265d0e1db 00007f4265d09aaf 00007f4265cfd86e 00007f4265cfcbb6 00007f4265cfc7d8 00007f4265d06221 00007f4265d061d9 00007f4265d06169
0709 13:01:30.718603 (+259263us) raft_consensus.cc:1685] finished
0709 13:01:30.718628 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:30.775094 (+ 56466us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":28844,"prepare.queue_time_us":347015,"prepare.run_cpu_time_us":113,"prepare.run_wall_time_us":116,"raft.queue_time_us":2535,"raft.run_cpu_time_us":882,"raft.run_wall_time_us":42454,"spinlock_wait_cycles":68988672,"thread_start_us":53245,"threads_started":1,"wal-append.queue_time_us":142464}
I20260709 13:01:30.813324  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:30.814196  6914 consensus_queue.cc:1048] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 13:01:30.770627  6905 raft_consensus.cc:515] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:30.817191  1679 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57092 (request call id 1207) took 1276 ms. Trace:
I20260709 13:01:30.817430  6905 leader_election.cc:290] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 13 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:30.817658  1679 rpcz_store.cc:276] 0709 13:01:29.540881 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:29.541439 (+   558us) service_pool.cc:224] Handling call
0709 13:01:29.541843 (+   404us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:29.542290 (+   447us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 13:01:29.688924 (+146634us) log.cc:844] Serialized 14 byte log entry
0709 13:01:30.542234 (+853310us) spinlock_profiling.cc:243] Waited 1.7 ms on lock 0x7b1800370420. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f4265d0a4bc 00007f426305848b 00007f426beb2ab9 00007f426307469b 00007f426307293f 00007f4263074c02 00007f4265cfc20b 00007f4265d01d23 00007f4265d0223a
0709 13:01:30.542725 (+   491us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:01:30.543733 (+  1008us) log.cc:844] Serialized 26 byte log entry
0709 13:01:30.556926 (+ 13193us) raft_consensus.cc:1645] Marking committed up to 13
0709 13:01:30.762833 (+205907us) log.cc:844] Serialized 14 byte log entry
0709 13:01:30.807898 (+ 45065us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:30.816654 (+  8756us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:30.816797 (+   143us) raft_consensus.cc:1685] finished
0709 13:01:30.816812 (+    15us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:30.816971 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":44053,"prepare.queue_time_us":294625,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":52,"raft.queue_time_us":4191,"raft.run_cpu_time_us":455,"raft.run_wall_time_us":457,"spinlock_wait_cycles":3921152,"thread_start_us":474918,"threads_started":3,"wal-append.queue_time_us":281641}
I20260709 13:01:30.803280  2335 raft_consensus.cc:2468] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 11.
W20260709 13:01:30.821184  1753 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55848 (request call id 2740) took 1544 ms (client timeout 1689 ms). Trace:
I20260709 13:01:30.808912  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:30.823443  6752 leader_election.cc:290] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:30.823958  1752 raft_consensus.cc:2468] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 10.
W20260709 13:01:30.824203  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 102935us
I20260709 13:01:30.813774  2328 raft_consensus.cc:2410] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for term 14 because replica has last-logged OpId of term: 13 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 10.
I20260709 13:01:30.761932  7120 raft_consensus.cc:493] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:30.807931  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:30.774881  7220 raft_consensus.cc:493] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
W20260709 13:01:30.821652  1753 rpcz_store.cc:269] 0709 13:01:29.276951 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:29.277758 (+   807us) service_pool.cc:224] Handling call
0709 13:01:30.820677 (+1542919us) spinlock_profiling.cc:243] Waited 55.5 ms on lock 0x7b54001c7d54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:01:30.820930 (+   253us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":127562624}
I20260709 13:01:30.775215  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:30.826296  1672 raft_consensus.cc:2468] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 16.
I20260709 13:01:30.839102  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:30.842551  7231 raft_consensus.cc:493] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:30.843451  7231 raft_consensus.cc:515] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:30.850350  7220 raft_consensus.cc:515] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:30.853689  7220 leader_election.cc:290] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:30.856981  2331 raft_consensus.cc:2468] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 11.
I20260709 13:01:30.864324  6345 raft_consensus.cc:3055] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 LEADER]: Stepping down as leader of term 13
I20260709 13:01:30.865022  6345 raft_consensus.cc:740] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:01:30.868388  7236 leader_election.cc:290] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:01:30.869585  1646 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:30.878727  7236 raft_consensus.cc:493] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:30.879827  7236 raft_consensus.cc:515] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:30.882858  7236 leader_election.cc:290] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:30.884822  1604 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40154 (request call id 2482) took 1133 ms. Trace:
I20260709 13:01:30.886741  7238 raft_consensus.cc:493] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:30.887588  7238 raft_consensus.cc:515] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:30.894547  7241 raft_consensus.cc:493] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:30.895696  7241 raft_consensus.cc:515] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:30.897642  7159 leader_election.cc:290] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:01:30.901643  1599 consensus_meta.cc:287] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.936s	user 0.013s	sys 0.013s
I20260709 13:01:30.901937  1596 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40154 (request call id 2483) took 1148 ms. Trace:
I20260709 13:01:30.902256  1599 raft_consensus.cc:2468] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 16.
I20260709 13:01:30.903182  1599 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40154 (request call id 2487) took 1147 ms. Trace:
I20260709 13:01:30.903568  1599 rpcz_store.cc:276] 0709 13:01:29.755594 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:29.768495 (+ 12901us) service_pool.cc:224] Handling call
0709 13:01:30.903054 (+1134559us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:01:30.911149  1646 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:30.914245  6914 raft_consensus.cc:493] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:30.915199  6914 raft_consensus.cc:515] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:30.916343  7200 consensus_queue.cc:1059] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 14.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 13:01:30.917733  7199 raft_consensus.cc:3055] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:01:30.916265  6714 raft_consensus.cc:493] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:30.910362  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:30.909701  1596 rpcz_store.cc:276] 0709 13:01:29.754323 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:29.816232 (+ 61909us) service_pool.cc:224] Handling call
0709 13:01:30.688574 (+872342us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:30.730197 (+ 41623us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:30.901675 (+171478us) spinlock_profiling.cc:243] Waited 872 ms on lock 0x7b54000b1150. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661618ad 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:01:30.901857 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2004711808}
I20260709 13:01:30.918422  7199 raft_consensus.cc:740] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:01:30.926771  1501 catalog_manager.cc:5697] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d reported cstate change: term changed from 7 to 9. New cstate: current_term: 9 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 13:01:30.927309  7199 consensus_queue.cc:260] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 14.7, Last appended by leader: 7, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:30.927629  6714 raft_consensus.cc:515] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:30.928714  7199 raft_consensus.cc:3060] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:01:30.928946  1501 catalog_manager.cc:5697] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d reported cstate change: term changed from 7 to 11. New cstate: current_term: 11 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } }
I20260709 13:01:30.932312  7227 leader_election.cc:290] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:30.933911  7188 leader_election.cc:290] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 16 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:30.936231  7188 consensus_queue.cc:1059] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 14.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 13:01:30.936618  7189 leader_election.cc:290] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:30.940315  7242 raft_consensus.cc:493] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:30.941205  6876 raft_consensus.cc:493] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:01:30.946943  1501 catalog_manager.cc:5697] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d reported cstate change: term changed from 5 to 6, leader changed from af520b67945d443b901bbeca37cc5701 (127.0.107.195) to 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194). New cstate: current_term: 6 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } }
I20260709 13:01:30.948688  1752 raft_consensus.cc:1240] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 14. Current term is 15. Ops: []
I20260709 13:01:30.946359  7189 raft_consensus.cc:3055] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:01:30.951251  7189 raft_consensus.cc:740] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:01:30.952821  7189 consensus_queue.cc:260] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.8, Last appended by leader: 8, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:30.953841  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:30.954367  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:30.955633  6997 raft_consensus.cc:3060] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:01:30.956200  1675 raft_consensus.cc:1240] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 13. Current term is 14. Ops: []
I20260709 13:01:30.958014  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:30.958875  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:30.890635  7234 raft_consensus.cc:515] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:30.961462  7234 leader_election.cc:290] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:30.957995  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:30.790916  1600 rpcz_store.cc:276] 0709 13:01:28.386017 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:28.441303 (+ 55286us) service_pool.cc:224] Handling call
0709 13:01:28.441697 (+   394us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:01:28.498028 (+ 56331us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:28.498085 (+    57us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:01:28.498971 (+   886us) log.cc:844] Serialized 52 byte log entry
0709 13:01:28.567038 (+ 68067us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:01:28.579215 (+ 12177us) log.cc:844] Serialized 14 byte log entry
0709 13:01:28.665975 (+ 86760us) log.cc:844] Serialized 14 byte log entry
0709 13:01:28.666352 (+   377us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:28.666475 (+   123us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:28.954741 (+288266us) spinlock_profiling.cc:243] Waited 1.41 ms on lock 0x7b4800003cd8. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426b08baa1 00007f426b082b37 00007f426b0829ca 00007f426b17a26f 00007f426b179f37 00007f426b18e55c 00007f426b18e508 00007f426b18e488 00007f426b18e443 00007f426b18d6ec 00007f426a0b7a96
0709 13:01:29.358477 (+403736us) spinlock_profiling.cc:243] Waited 1.8 ms on lock 0x7b1800149d40. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f42630753f3 00007f42630777c1 00007f4263077779 00007f4263077709 00007f42630776d1 00007f42630769cd 000000000038f354 000000000038f199 00007f426305883a
0709 13:01:29.585663 (+227186us) spinlock_profiling.cc:243] Waited 10.1 ms on lock 0x7b1800149d40. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f4263075522 00007f42630777c1 00007f4263077779 00007f4263077709 00007f42630776d1 00007f42630769cd 000000000038f354 000000000038f199 00007f426305883a
0709 13:01:30.457250 (+871587us) spinlock_profiling.cc:243] Waited 14.6 ms on lock 0x7b4400022b4c. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f4262f9bc85 00007f4262f9c6a2 00007f4265d0e1db 00007f4265d09aaf 00007f4265cfd86e 00007f4265cfcbb6 00007f4265cfc7d8 00007f4265d06221 00007f4265d061d9 00007f4265d06169
0709 13:01:30.745790 (+288540us) raft_consensus.cc:1685] finished
0709 13:01:30.745826 (+    36us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:30.773742 (+ 27916us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":47827,"prepare.queue_time_us":801455,"prepare.run_cpu_time_us":56,"prepare.run_wall_time_us":57,"raft.queue_time_us":13523,"raft.run_cpu_time_us":2581,"raft.run_wall_time_us":299107,"spinlock_wait_cycles":64177664,"thread_start_us":55019,"threads_started":1,"wal-append.queue_time_us":136485}
I20260709 13:01:30.963974  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:30.964689  1675 raft_consensus.cc:3060] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:01:30.966343  7189 raft_consensus.cc:3060] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:01:30.941109  7242 raft_consensus.cc:515] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:30.966773  1754 raft_consensus.cc:2468] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 14.
I20260709 13:01:30.968608  6938 raft_consensus.cc:493] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:30.968755  7242 leader_election.cc:290] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:30.969426  6938 raft_consensus.cc:515] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:30.944290  6876 raft_consensus.cc:515] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:30.975072  6876 leader_election.cc:290] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:30.975379  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:30.977883  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:30.930207  6714 leader_election.cc:290] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:30.979307  1746 raft_consensus.cc:2468] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 13.
I20260709 13:01:30.842096  7111 leader_election.cc:290] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:30.980451  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 16 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:30.981026  7120 raft_consensus.cc:515] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:30.994094  7036 consensus_meta.cc:287] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 1.116s	user 0.014s	sys 0.006s
I20260709 13:01:31.000496  7036 raft_consensus.cc:515] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:30.885428  1604 rpcz_store.cc:276] 0709 13:01:29.751268 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:29.752389 (+  1121us) service_pool.cc:224] Handling call
0709 13:01:29.752803 (+   414us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:01:29.869665 (+116862us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:29.870079 (+   414us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:01:29.870876 (+   797us) log.cc:844] Serialized 52 byte log entry
0709 13:01:30.520117 (+649241us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:01:30.532743 (+ 12626us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:30.578371 (+ 45628us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:30.883403 (+305032us) spinlock_profiling.cc:243] Waited 59.2 ms on lock 0x7b54002e43a0. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42660d7dbe 00007f42660e2930 00007f42660e28d8 00007f42660e2858 00007f42660e2813 00007f42660e1a9c 00007f426bfcd626 00007f426bfc7dd8 00007f426611e7cb 00007f42661212c0
0709 13:01:30.884126 (+   723us) raft_consensus.cc:1685] finished
0709 13:01:30.884154 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:30.884544 (+   390us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":127206,"raft.run_cpu_time_us":135,"raft.run_wall_time_us":9230,"spinlock_wait_cycles":136192768,"thread_start_us":299021,"threads_started":2,"wal-append.queue_time_us":355340}
W20260709 13:01:31.006211  1500 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35444 (request call id 261) took 38868 ms (client timeout 15000 ms). Trace:
I20260709 13:01:30.801014  7233 raft_consensus.cc:515] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } }
I20260709 13:01:30.846293  7231 leader_election.cc:290] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:01:31.013034  7109 consensus_meta.cc:287] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.595s	user 0.018s	sys 0.002s
W20260709 13:01:31.018775  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 208202us
I20260709 13:01:31.019604  7241 leader_election.cc:290] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:31.023923  6914 leader_election.cc:290] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:30.925532  6345 consensus_queue.cc:260] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.8, Last appended by leader: 8, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:31.027594  7238 leader_election.cc:290] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:31.027830  6938 leader_election.cc:290] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:31.029247  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:31.101715  7146 consensus_queue.cc:1059] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 14.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 13:01:31.030862  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:31.111192  7199 raft_consensus.cc:3055] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:01:31.111676  7199 raft_consensus.cc:740] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:01:31.032275  1678 raft_consensus.cc:1275] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20260709 13:01:31.037144  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 17 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:31.113338  1596 raft_consensus.cc:2410] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 for term 17 because replica has last-logged OpId of term: 15 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 10.
I20260709 13:01:31.041210  1748 raft_consensus.cc:1240] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 10. Current term is 11. Ops: []
I20260709 13:01:31.042024  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:30.873862  7067 raft_consensus.cc:515] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:31.044160  1752 raft_consensus.cc:1240] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 10. Current term is 11. Ops: []
I20260709 13:01:31.044808  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 12 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:30.983630  7120 leader_election.cc:290] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:31.046840  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:31.048830  1673 raft_consensus.cc:2468] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 11.
I20260709 13:01:31.049279  7014 leader_election.cc:290] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 13:01:31.060031  1569 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.511s	user 0.032s	sys 0.054s
I20260709 13:01:30.949424  1501 catalog_manager.cc:5697] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d reported cstate change: term changed from 7 to 10, leader changed from af520b67945d443b901bbeca37cc5701 (127.0.107.195) to 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194). New cstate: current_term: 10 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } }
W20260709 13:01:31.120734  1569 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.572s	user 0.032s	sys 0.054s
I20260709 13:01:31.121012  6345 raft_consensus.cc:3060] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:01:31.121759  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:31.046875  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:31.066176  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:31.069590  1751 raft_consensus.cc:1275] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20260709 13:01:31.071134  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:31.124097  1745 raft_consensus.cc:2468] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 11.
I20260709 13:01:31.124832  1749 raft_consensus.cc:2393] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 16: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:01:31.124514  1501 catalog_manager.cc:5697] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d reported cstate change: term changed from 8 to 12. New cstate: current_term: 12 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 13:01:31.089295  1671 raft_consensus.cc:2468] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 16.
I20260709 13:01:31.095920  6592 raft_consensus.cc:493] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:31.097429  1750 raft_consensus.cc:1240] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 11. Current term is 12. Ops: []
I20260709 13:01:31.098698  1646 leader_election.cc:304] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:31.126119  1569 leader_election.cc:304] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:01:31.099043  7233 leader_election.cc:290] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:31.102825  7243 raft_consensus.cc:493] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:31.131817  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:31.132953  2330 raft_consensus.cc:2393] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 13: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:01:31.030156  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:31.134363  1675 raft_consensus.cc:2468] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 12.
W20260709 13:01:31.095805  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 655085us
I20260709 13:01:31.108029  1644 leader_election.cc:304] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
W20260709 13:01:31.079967  1500 rpcz_store.cc:269] 0709 13:00:52.137256 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:52.150308 (+ 13052us) service_pool.cc:224] Handling call
0709 13:01:31.005807 (+38855499us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 13:01:18.266909 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 13:01:18.267799 (+   890us) write_op.cc:432] Acquiring schema lock in shared mode
0709 13:01:18.267890 (+    91us) write_op.cc:435] Acquired schema lock
0709 13:01:18.267918 (+    28us) tablet.cc:662] Decoding operations
0709 13:01:18.270239 (+  2321us) write_op.cc:620] Acquiring the partition lock for write op
0709 13:01:18.270342 (+   103us) write_op.cc:641] Partition lock acquired for write op
0709 13:01:18.270385 (+    43us) tablet.cc:685] Acquiring locks for 21 operations
0709 13:01:18.290961 (+ 20576us) tablet.cc:701] Row locks acquired
0709 13:01:18.291234 (+   273us) write_op.cc:260] PREPARE: finished
0709 13:01:18.292113 (+   879us) write_op.cc:270] Start()
0709 13:01:18.985378 (+693265us) spinlock_profiling.cc:243] Waited 3.01 ms on lock 0x7b540041c9a8. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426c51e9e1 00007f426c51d64c 00007f4269bad624 00007f4269b98286 00007f4269b97d58 00007f4269b9c9f1 00007f4269b9c9a9 00007f4269b9c939 00007f4269b9c901 00007f4269b9bb2d 000000000038f354
0709 13:01:18.985743 (+   365us) write_op.cc:276] Timestamp: P: 1783602078291892 usec, L: 0
0709 13:01:18.985774 (+    31us) op_driver.cc:348] REPLICATION: starting
0709 13:01:19.084506 (+ 98732us) log.cc:844] Serialized 8321 byte log entry
0709 13:01:19.679651 (+595145us) op_driver.cc:464] REPLICATION: finished
0709 13:01:19.859183 (+179532us) write_op.cc:301] APPLY: starting
0709 13:01:19.859340 (+   157us) tablet.cc:1367] starting BulkCheckPresence
0709 13:01:20.476827 (+617487us) tablet.cc:1370] finished BulkCheckPresence
0709 13:01:20.486366 (+  9539us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 13:01:20.902930 (+416564us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 13:01:20.903385 (+   455us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=42,key_file_lookups=42,delta_file_lookups=42,mrs_lookups=0
0709 13:01:20.903438 (+    53us) write_op.cc:312] APPLY: finished
0709 13:01:20.913477 (+ 10039us) log.cc:844] Serialized 188 byte log entry
0709 13:01:21.228493 (+315016us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:01:21.231343 (+  2850us) write_op.cc:454] Released schema lock
0709 13:01:21.231453 (+   110us) write_op.cc:341] FINISH: Updating metrics
0709 13:01:21.232170 (+   717us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:01:21.232203 (+    33us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":3333643,"reported_tablets":48,"child_traces":[["op",{"apply.queue_time_us":179003,"apply.run_cpu_time_us":227737,"apply.run_wall_time_us":1372992,"cfile_cache_hit":86,"cfile_cache_hit_bytes":92075,"num_ops":21,"prepare.queue_time_us":125496,"prepare.run_cpu_time_us":22960,"prepare.run_wall_time_us":993100,"raft.queue_time_us":317707,"raft.run_cpu_time_us":18729,"raft.run_wall_time_us":127737,"replication_time_us":693502,"spinlock_wait_cycles":6942464,"thread_start_us":821742,"threads_started":5,"wal-append.queue_time_us":581145,"wal-append.run_cpu_time_us":21634,"wal-append.run_wall_time_us":2352871}]]}
I20260709 13:01:31.128191  1676 raft_consensus.cc:2368] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 12. Current term is 13.
I20260709 13:01:31.103397  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:31.154028  1677 raft_consensus.cc:2393] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 16: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:01:31.157292  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:31.159969  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
W20260709 13:01:31.161557  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 62961.8us
I20260709 13:01:31.164327  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:31.171890  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:31.173167  1674 raft_consensus.cc:2468] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 15.
I20260709 13:01:31.181730  1501 catalog_manager.cc:5697] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d reported cstate change: term changed from 9 to 12, leader changed from af520b67945d443b901bbeca37cc5701 (127.0.107.195) to 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193). New cstate: current_term: 12 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } }
W20260709 13:01:31.188747  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 80996.4us
I20260709 13:01:31.185249  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:31.212212  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:31.219242  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 17 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:31.173171  7146 raft_consensus.cc:2804] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:01:31.219411  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:01:31.223282  1571 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.510s	user 0.008s	sys 0.017s
W20260709 13:01:31.224287  1571 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.511s	user 0.008s	sys 0.017s
I20260709 13:01:31.220816  1600 raft_consensus.cc:2468] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 11.
I20260709 13:01:31.225836  1754 raft_consensus.cc:2410] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 for term 17 because replica has last-logged OpId of term: 15 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 10.
I20260709 13:01:31.226377  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:31.227458  1672 raft_consensus.cc:2468] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 10.
I20260709 13:01:31.226468  1501 catalog_manager.cc:5697] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d reported cstate change: term changed from 8 to 10. New cstate: current_term: 10 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 13:01:31.234513  1676 raft_consensus.cc:2468] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 14.
W20260709 13:01:31.238571  1720 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.506s	user 0.006s	sys 0.065s
W20260709 13:01:31.238976  1720 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.663s	user 0.006s	sys 0.065s
I20260709 13:01:31.228266  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:31.242632  1718 leader_election.cc:304] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:31.242940  7234 raft_consensus.cc:493] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:31.243680  7234 raft_consensus.cc:515] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:31.245990  7234 leader_election.cc:290] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:31.246963  1746 raft_consensus.cc:2468] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 10.
I20260709 13:01:31.247547  1501 catalog_manager.cc:5697] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d reported cstate change: term changed from 6 to 8. New cstate: current_term: 8 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:01:31.251708  7036 leader_election.cc:290] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 13 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:31.249643  1501 catalog_manager.cc:5697] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d reported cstate change: term changed from 6 to 8, leader changed from 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) to 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196). New cstate: current_term: 8 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 13:01:31.231199  7189 consensus_queue.cc:1048] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.029s
I20260709 13:01:31.259963  1501 catalog_manager.cc:5697] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d reported cstate change: term changed from 6 to 11, leader changed from 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194) to 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193). New cstate: current_term: 11 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } }
I20260709 13:01:31.261865  1501 catalog_manager.cc:5697] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d reported cstate change: term changed from 8 to 10, leader changed from 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) to 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194). New cstate: current_term: 10 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } }
I20260709 13:01:31.263487  7244 raft_consensus.cc:493] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:31.264127  7244 raft_consensus.cc:515] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:31.231026  6997 raft_consensus.cc:2804] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:01:31.268826  6997 raft_consensus.cc:493] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:31.269387  6997 raft_consensus.cc:3060] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:01:31.273387  1501 catalog_manager.cc:5697] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d reported cstate change: term changed from 5 to 8, leader changed from af520b67945d443b901bbeca37cc5701 (127.0.107.195) to 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193). New cstate: current_term: 8 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } }
I20260709 13:01:31.258561  7243 raft_consensus.cc:515] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:31.161635  6592 raft_consensus.cc:515] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:31.286072  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 54369.8us
I20260709 13:01:31.294663  1680 raft_consensus.cc:2468] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 16.
I20260709 13:01:31.294503  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:31.298662  1674 raft_consensus.cc:2393] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 13: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
W20260709 13:01:31.300323  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 109329us
W20260709 13:01:31.306708  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 180681us
I20260709 13:01:31.307672  1569 leader_election.cc:304] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:01:31.309311  1569 leader_election.cc:304] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:01:31.192869  7111 raft_consensus.cc:3060] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:01:31.224759  7146 raft_consensus.cc:493] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
W20260709 13:01:31.300431  7126 consensus_meta.cc:287] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.864s	user 0.012s	sys 0.007s
I20260709 13:01:31.319396  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:31.335639  1750 raft_consensus.cc:2393] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 12: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:01:31.342854  2299 leader_election.cc:304] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:01:31.334725  7146 raft_consensus.cc:3060] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:01:31.362828  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:01:31.358605  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 134281us
I20260709 13:01:31.368158  2301 leader_election.cc:304] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:31.315788  7109 raft_consensus.cc:2749] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20260709 13:01:31.383149  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 95560.1us
I20260709 13:01:31.229254  7199 consensus_queue.cc:260] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 14.7, Last appended by leader: 7, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:31.343572  7227 raft_consensus.cc:2749] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20260709 13:01:31.356496  1718 connection.cc:256] RPC call timeout handler was delayed by -1.6665508629999977s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1915s for a response
I20260709 13:01:31.390493  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:31.348337  7106 raft_consensus.cc:2749] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
W20260709 13:01:31.395522  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 86260.2us
W20260709 13:01:31.393888  1718 connection.cc:256] RPC call timeout handler was delayed by -1.6017508630000066s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19870000000000002s for a response
W20260709 13:01:31.397792  1718 connection.cc:256] RPC call timeout handler was delayed by -1.5243508629999951s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2073s for a response
W20260709 13:01:31.398231  1718 connection.cc:256] RPC call timeout handler was delayed by -1.5180508629999849s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20800000000000002s for a response
I20260709 13:01:31.399801  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:31.401463  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:31.404479  1748 raft_consensus.cc:2468] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 16.
I20260709 13:01:31.370582  7036 raft_consensus.cc:2804] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:01:31.407076  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:31.382522  7233 raft_consensus.cc:493] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:31.411298  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:31.345999  7126 raft_consensus.cc:515] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:01:31.406219  7119 consensus_meta.cc:287] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.747s	user 0.011s	sys 0.004s
I20260709 13:01:31.417989  1745 raft_consensus.cc:2393] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 15: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:01:31.398589  7067 leader_election.cc:290] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 12 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:31.415966  2329 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39726 (request call id 1263) took 1766 ms. Trace:
I20260709 13:01:31.415277  1754 raft_consensus.cc:2393] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 15: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
W20260709 13:01:31.409026  1718 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.107.194:46261 timed out after 1.549s (SENT) [suppressed 24 similar messages]
I20260709 13:01:31.407815  7036 raft_consensus.cc:493] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:31.407394  7244 leader_election.cc:290] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:31.438987  1677 raft_consensus.cc:2468] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 14.
I20260709 13:01:31.443771  1569 leader_election.cc:304] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:01:31.446568  1717 leader_election.cc:304] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:31.436079  7199 raft_consensus.cc:3060] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:01:31.482054  1569 leader_election.cc:304] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:01:31.440857  7036 raft_consensus.cc:3060] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:01:31.421257  2329 rpcz_store.cc:276] 0709 13:01:29.548616 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:29.548798 (+   182us) service_pool.cc:224] Handling call
0709 13:01:30.697613 (+1148815us) spinlock_profiling.cc:243] Waited 1.56 ms on lock 0x7b5800178598. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f426303db71 00007f426303da22 00007f426bf56c09 00007f426b0b0ee0 00007f426b17a0ab 00007f426b17a131 00007f426b13bfc0 00007f426b126929 00007f426b1263d2 00007f426469dbd4
0709 13:01:30.698042 (+   429us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:30.742317 (+ 44275us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:30.742362 (+    45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:01:30.743067 (+   705us) log.cc:844] Serialized 26 byte log entry
0709 13:01:31.121850 (+378783us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:01:31.121923 (+    73us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:31.122022 (+    99us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:31.275848 (+153826us) raft_consensus.cc:1685] finished
0709 13:01:31.275873 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:31.315222 (+ 39349us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":25594,"raft.queue_time_us":25778,"raft.run_cpu_time_us":193,"raft.run_wall_time_us":194,"spinlock_wait_cycles":3584768,"thread_start_us":369042,"threads_started":1,"wal-append.queue_time_us":527444}
W20260709 13:01:31.446483  7219 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.542s	user 0.029s	sys 0.054s
I20260709 13:01:31.485724  7170 raft_consensus.cc:2804] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Leader pre-election won for term 12
W20260709 13:01:31.502532  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 90317us
W20260709 13:01:31.494915  7219 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.594s	user 0.030s	sys 0.055s
I20260709 13:01:31.504869  1747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43156 (request call id 1087) took 1839 ms. Trace:
I20260709 13:01:31.513119  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:31.440357  7233 raft_consensus.cc:515] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:31.509263  1747 rpcz_store.cc:276] 0709 13:01:29.665383 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:29.667709 (+  2326us) service_pool.cc:224] Handling call
0709 13:01:29.667994 (+   285us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:29.668236 (+   242us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 13:01:30.945006 (+1276770us) spinlock_profiling.cc:243] Waited 70.8 ms on lock 0x7b18003f4c60. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f4265d0a4bc 00007f426305848b 00007f426beb2ab9 00007f426307469b 00007f426307293f 00007f4263072455 00007f4269b5a3c7 00007f426616c9a7 00007f426617c0bf
0709 13:01:30.945294 (+   288us) log.cc:844] Serialized 14 byte log entry
0709 13:01:31.419785 (+474491us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:01:31.432080 (+ 12295us) raft_consensus.cc:1645] Marking committed up to 13
0709 13:01:31.438512 (+  6432us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:31.446726 (+  8214us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:31.504765 (+ 58039us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":630485,"prepare.run_cpu_time_us":2673,"prepare.run_wall_time_us":73001,"raft.queue_time_us":4577,"raft.run_cpu_time_us":259,"raft.run_wall_time_us":16013,"spinlock_wait_cycles":162867456,"thread_start_us":1200589,"threads_started":2,"wal-append.queue_time_us":549840}
I20260709 13:01:31.502161  7170 raft_consensus.cc:493] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:31.529688  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:31.513651  7244 raft_consensus.cc:2804] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 13:01:31.533239  1748 raft_consensus.cc:2468] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 11.
I20260709 13:01:31.524438  7234 raft_consensus.cc:2804] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:01:31.566681  7234 raft_consensus.cc:493] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:31.567221  7234 raft_consensus.cc:3060] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Advancing to term 16
W20260709 13:01:31.569952  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 121795us
I20260709 13:01:31.544307  7135 raft_consensus.cc:2749] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 13:01:31.546603  1674 raft_consensus.cc:1275] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 16 index: 13. (index mismatch)
I20260709 13:01:31.550680  7243 leader_election.cc:290] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:31.555090  6592 leader_election.cc:290] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:31.562384  7126 leader_election.cc:290] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 15 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:31.572582  1750 raft_consensus.cc:2468] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 14.
I20260709 13:01:31.532552  7170 raft_consensus.cc:3060] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Advancing to term 12
W20260709 13:01:31.585098  1644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:31.586144  1644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:31.577556  1746 raft_consensus.cc:2393] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 14: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
W20260709 13:01:31.578032  1643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:31.578477  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 239190us
I20260709 13:01:31.578229  7244 raft_consensus.cc:697] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 16 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:01:31.598347  1675 raft_consensus.cc:1275] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 13 index: 11. (index mismatch)
W20260709 13:01:31.600579  6160 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.734s	user 0.006s	sys 0.020s
I20260709 13:01:31.600294  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:01:31.602378  6160 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.894s	user 0.007s	sys 0.020s
I20260709 13:01:31.608738  1569 leader_election.cc:304] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:01:31.605732  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 12 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:31.612900  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:31.618556  1679 raft_consensus.cc:2468] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 14.
I20260709 13:01:31.576869  7222 consensus_queue.cc:1059] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 13.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.030s
I20260709 13:01:31.623051  1596 raft_consensus.cc:2410] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for term 12 because replica has last-logged OpId of term: 11 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 5.
I20260709 13:01:31.589277  6997 raft_consensus.cc:515] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:31.631183  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 251380us
I20260709 13:01:31.636222  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:31.632552  1646 leader_election.cc:304] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:01:31.633709  7067 raft_consensus.cc:2804] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:01:31.629640  7243 raft_consensus.cc:2749] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:01:31.657305  1599 raft_consensus.cc:1275] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
W20260709 13:01:31.662806  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 72570.8us
I20260709 13:01:31.644439  7126 raft_consensus.cc:3055] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 13:01:31.627187  7119 consensus_queue.cc:1059] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 10.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.030s
I20260709 13:01:31.670560  1569 leader_election.cc:304] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
W20260709 13:01:31.670457  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 284160us
I20260709 13:01:31.673178  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:31.674060  2301 leader_election.cc:304] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:01:31.671080  1604 raft_consensus.cc:1275] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 14 index: 10. (index mismatch)
I20260709 13:01:31.636392  7188 consensus_queue.cc:1059] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 10.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.026s
I20260709 13:01:31.639479  7189 consensus_queue.cc:1048] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.028s
I20260709 13:01:31.685051  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:31.695041  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:31.674520  7126 raft_consensus.cc:740] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:01:31.705633  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:31.706804  7219 raft_consensus.cc:2804] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:01:31.707295  7219 raft_consensus.cc:493] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:31.707695  7219 raft_consensus.cc:3060] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:01:31.708221  6882 raft_consensus.cc:2749] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:01:31.708778  6605 raft_consensus.cc:2749] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:01:31.709964  1745 raft_consensus.cc:3055] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:01:31.710358  1745 raft_consensus.cc:740] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:01:31.711129  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:31.711211  1745 consensus_queue.cc:260] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.10, Last appended by leader: 10, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:31.712409  1745 raft_consensus.cc:3060] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:01:31.713423  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:31.714495  1599 raft_consensus.cc:2393] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 11: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:01:31.715407  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:31.726073  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:31.727074  1605 raft_consensus.cc:2393] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 16: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:01:31.729002  1599 raft_consensus.cc:1240] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 10. Current term is 11. Ops: []
I20260709 13:01:31.729923  2299 leader_election.cc:304] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:01:31.729969  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:31.731376  1646 leader_election.cc:304] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:31.725761  1600 raft_consensus.cc:2468] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 15.
I20260709 13:01:31.642812  7244 consensus_queue.cc:237] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.8, Last appended by leader: 8, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:31.732547  7188 raft_consensus.cc:3055] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 LEADER]: Stepping down as leader of term 10
W20260709 13:01:31.828510  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 83478.9us
I20260709 13:01:31.701254  1644 leader_election.cc:304] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:01:31.733675  6997 leader_election.cc:290] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 11 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:31.734095  2330 raft_consensus.cc:1240] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 13 LEADER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 12. Current term is 13. Ops: []
I20260709 13:01:31.682977  7233 leader_election.cc:290] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:01:31.742211  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 333155us
I20260709 13:01:31.738934  2333 raft_consensus.cc:2468] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 14.
I20260709 13:01:31.841763  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:31.842516  1753 raft_consensus.cc:3060] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Advancing to term 13
I20260709 13:01:31.731233  7143 consensus_queue.cc:1059] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 11.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
W20260709 13:01:31.767021  1646 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:31.818531  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 148058us
I20260709 13:01:31.821478  6882 raft_consensus.cc:2804] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:01:31.856251  6882 raft_consensus.cc:493] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:01:31.856702  6882 raft_consensus.cc:3060] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:01:31.824247  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:31.858796  1749 raft_consensus.cc:2468] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 15.
I20260709 13:01:31.824496  7126 consensus_queue.cc:260] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.7, Last appended by leader: 7, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:31.827214  7263 consensus_queue.cc:1048] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 13:01:31.829020  7188 raft_consensus.cc:740] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
W20260709 13:01:31.863298  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 162130us
I20260709 13:01:31.863713  7188 consensus_queue.cc:260] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:31.864887  6987 raft_consensus.cc:2749] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 13:01:31.865589  7188 raft_consensus.cc:3060] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:01:31.866866  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:31.867753  1604 raft_consensus.cc:2468] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 13.
I20260709 13:01:31.868997  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:31.869732  1604 raft_consensus.cc:2468] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 15.
I20260709 13:01:31.871857  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:31.873064  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:31.873679  1599 raft_consensus.cc:2468] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 14.
I20260709 13:01:31.876075  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:31.878685  1602 raft_consensus.cc:1240] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 11. Current term is 12. Ops: []
I20260709 13:01:31.879109  6882 raft_consensus.cc:515] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:31.880046  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:31.881371  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:31.882836  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:31.889595  1678 raft_consensus.cc:2468] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 15.
I20260709 13:01:31.902678  7181 consensus_queue.cc:1048] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.056s
W20260709 13:01:31.908291  7146 consensus_meta.cc:287] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.547s	user 0.012s	sys 0.001s
I20260709 13:01:31.908831  7146 raft_consensus.cc:515] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:31.922281  1601 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 1297) took 1185 ms. Trace:
I20260709 13:01:31.922937  1601 rpcz_store.cc:276] 0709 13:01:30.737345 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:30.739168 (+  1823us) service_pool.cc:224] Handling call
0709 13:01:30.739584 (+   416us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:31.128294 (+388710us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:31.128329 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:01:31.156971 (+ 28642us) log.cc:844] Serialized 26 byte log entry
0709 13:01:31.626801 (+469830us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:01:31.626869 (+    68us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:31.626971 (+   102us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:31.921944 (+294973us) raft_consensus.cc:1685] finished
0709 13:01:31.921985 (+    41us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:31.922186 (+   201us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":63796,"raft.run_cpu_time_us":179,"raft.run_wall_time_us":11056,"thread_start_us":799895,"threads_started":2,"wal-append.queue_time_us":736642}
W20260709 13:01:31.926007  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 100800us
I20260709 13:01:31.848317  1569 leader_election.cc:304] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
W20260709 13:01:31.935298  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 87070.3us
I20260709 13:01:31.877421  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:31.945603  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:31.946593  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:31.951604  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:01:31.966114  1718 connection.cc:256] RPC call timeout handler was delayed by -1.5036508630000185s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2096s for a response
W20260709 13:01:31.982355  1644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:32.012321  1571 leader_election.cc:304] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
W20260709 13:01:32.015164  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 48179us
I20260709 13:01:32.017097  1569 leader_election.cc:304] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
W20260709 13:01:32.019711  1717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:32.028651  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 414557us
I20260709 13:01:32.019629  1599 raft_consensus.cc:3060] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:01:31.947538  7106 raft_consensus.cc:2804] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:01:31.961814  7233 raft_consensus.cc:493] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:32.054831  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:32.055336  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:32.056017  1673 raft_consensus.cc:1275] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
I20260709 13:01:32.065550  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:32.016235  7143 raft_consensus.cc:493] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:32.072568  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:01:32.086031  1644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:32.090607  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:32.119400  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:32.124742  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:32.128778  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:32.092289  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:32.124730  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:32.049958  7106 raft_consensus.cc:493] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:01:32.126223  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 198105us
I20260709 13:01:32.132411  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:32.078266  6882 leader_election.cc:290] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 11 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:32.135600  1679 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52778 (request call id 2336) took 1171 ms. Trace:
W20260709 13:01:32.132179  2331 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.520s	user 0.054s	sys 0.042s
W20260709 13:01:32.141251  1718 connection.cc:256] RPC call timeout handler was delayed by -1.4145508629999881s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2195s for a response
W20260709 13:01:32.149096  2331 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.541s	user 0.055s	sys 0.045s
W20260709 13:01:32.149151  1718 connection.cc:256] RPC call timeout handler was delayed by -1.3983508630000188s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22130000000000002s for a response
I20260709 13:01:32.148931  1679 rpcz_store.cc:276] 0709 13:01:30.955026 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:30.957597 (+  2571us) service_pool.cc:224] Handling call
0709 13:01:32.126642 (+1169045us) spinlock_profiling.cc:243] Waited 385 ms on lock 0x7b54000ff354. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:01:32.126797 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":884465536}
I20260709 13:01:32.131225  1745 raft_consensus.cc:1275] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 13 index: 11. (index mismatch)
W20260709 13:01:32.122990  7036 consensus_meta.cc:287] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.604s	user 0.015s	sys 0.000s
I20260709 13:01:32.157187  1677 raft_consensus.cc:1275] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
W20260709 13:01:32.137809  7234 consensus_meta.cc:287] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.570s	user 0.011s	sys 0.004s
I20260709 13:01:32.164408  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:32.142211  7106 raft_consensus.cc:3060] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:01:32.175598  2331 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39726 (request call id 1277) took 1062 ms. Trace:
I20260709 13:01:32.084165  7233 raft_consensus.cc:515] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:32.182834  6997 raft_consensus.cc:3055] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 LEADER]: Stepping down as leader of term 11
I20260709 13:01:32.184990  6976 raft_consensus.cc:3055] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:01:32.171195  7234 raft_consensus.cc:515] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:32.189441  7227 raft_consensus.cc:2749] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:01:32.190315  7192 raft_consensus.cc:493] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:32.062119  7219 raft_consensus.cc:515] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:32.192231  1678 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52778 (request call id 2346) took 1160 ms. Trace:
I20260709 13:01:32.182570  6985 raft_consensus.cc:2804] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:01:32.194253  2331 rpcz_store.cc:276] 0709 13:01:31.112902 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:31.121047 (+  8145us) service_pool.cc:224] Handling call
0709 13:01:31.121397 (+   350us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:31.161143 (+ 39746us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 13:01:31.607767 (+446624us) log.cc:844] Serialized 14 byte log entry
0709 13:01:32.175049 (+567282us) log.cc:844] Serialized 14 byte log entry
0709 13:01:32.175240 (+   191us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:01:32.175291 (+    51us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:01:32.175352 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:32.175417 (+    65us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:32.175525 (+   108us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":24926,"prepare.queue_time_us":519093,"prepare.run_cpu_time_us":85,"prepare.run_wall_time_us":86,"raft.queue_time_us":57574,"raft.run_cpu_time_us":174,"raft.run_wall_time_us":175,"thread_start_us":916200,"threads_started":2}
I20260709 13:01:32.164023  7146 leader_election.cc:290] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 14 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:32.125771  7143 raft_consensus.cc:515] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:32.210857  1674 raft_consensus.cc:2468] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 15.
I20260709 13:01:32.198001  6997 raft_consensus.cc:740] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
W20260709 13:01:32.208998  1501 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.507s	user 0.018s	sys 0.016s
I20260709 13:01:32.225076  1678 rpcz_store.cc:276] 0709 13:01:31.031666 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:31.132040 (+100374us) service_pool.cc:224] Handling call
0709 13:01:32.192003 (+1059963us) spinlock_profiling.cc:243] Waited 715 ms on lock 0x7b54000d8754. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:01:32.192153 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1645626624}
W20260709 13:01:32.227293  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 215069us
W20260709 13:01:32.233640  1501 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.532s	user 0.019s	sys 0.017s
W20260709 13:01:32.211338  1603 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.536s	user 0.020s	sys 0.018s
I20260709 13:01:32.164850  7036 raft_consensus.cc:515] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:32.230180  1718 leader_election.cc:304] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
W20260709 13:01:32.234357  1603 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.559s	user 0.021s	sys 0.018s
W20260709 13:01:32.234694  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:32.200899  6976 raft_consensus.cc:740] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:01:32.246168  1599 raft_consensus.cc:2468] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 15.
I20260709 13:01:32.258364  1675 raft_consensus.cc:1240] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 12. Current term is 13. Ops: []
I20260709 13:01:32.236472  7227 raft_consensus.cc:2749] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:01:32.262940  2337 raft_consensus.cc:2393] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 15: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:01:32.267958  6997 consensus_queue.cc:260] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } }
W20260709 13:01:32.275655  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 258620us
I20260709 13:01:32.276291  7233 leader_election.cc:290] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:32.280872  7126 raft_consensus.cc:3060] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Advancing to term 14
W20260709 13:01:32.257907  7111 consensus_meta.cc:287] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.921s	user 0.007s	sys 0.007s
I20260709 13:01:32.299762  6997 raft_consensus.cc:3060] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:01:32.299470  6976 consensus_queue.cc:260] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:32.302065  7067 raft_consensus.cc:493] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:32.303460  1569 leader_election.cc:304] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:32.227969  7192 raft_consensus.cc:515] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:32.286376  7170 consensus_meta.cc:287] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.686s	user 0.008s	sys 0.001s
I20260709 13:01:32.305230  7143 leader_election.cc:290] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:32.330055  1753 raft_consensus.cc:2468] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 13.
I20260709 13:01:32.290854  7244 raft_consensus.cc:2749] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:01:32.302136  7036 leader_election.cc:290] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 15 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:01:32.348009  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 44687us
I20260709 13:01:32.357391  1569 leader_election.cc:304] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:32.347301  7170 raft_consensus.cc:515] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:32.327026  7111 raft_consensus.cc:2749] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:01:32.336719  7067 raft_consensus.cc:3060] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:01:32.336170  7234 leader_election.cc:290] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 16 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:32.341096  7227 raft_consensus.cc:493] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:32.400104  1597 raft_consensus.cc:1240] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 10. Current term is 11. Ops: []
I20260709 13:01:32.355635  6976 raft_consensus.cc:3060] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Advancing to term 11
W20260709 13:01:32.406940  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 158900us
I20260709 13:01:32.412446  1571 leader_election.cc:304] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:32.411141  7111 raft_consensus.cc:2804] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Leader election won for term 11
I20260709 13:01:32.442051  1571 leader_election.cc:304] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
W20260709 13:01:32.445530  2301 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.515s	user 0.000s	sys 0.020s
W20260709 13:01:32.445770  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 234633us
W20260709 13:01:32.446024  2301 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.516s	user 0.000s	sys 0.020s
I20260709 13:01:32.453752  1718 leader_election.cc:304] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:32.455760  7170 leader_election.cc:290] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 12 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:32.458127  1718 leader_election.cc:400] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 12 pre-election: Vote denied by peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261) with higher term. Message: Invalid argument: T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 12. Current term is 13.
I20260709 13:01:32.458550  1718 leader_election.cc:403] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 12 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:01:32.459229  7233 raft_consensus.cc:2749] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Leader pre-election lost for term 12. Reason: Vote denied by peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261) with higher term. Message: Invalid argument: T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 12. Current term is 13.
I20260709 13:01:32.459056  7135 consensus_queue.cc:1048] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 13:01:32.461120  1718 leader_election.cc:304] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:32.455888  7244 raft_consensus.cc:2749] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:01:32.462473  7244 raft_consensus.cc:2749] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 13:01:32.463620  1718 leader_election.cc:304] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:32.464579  7244 raft_consensus.cc:2749] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 13:01:32.467727  1718 leader_election.cc:304] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:32.475446  7244 raft_consensus.cc:2749] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:01:32.475519  7135 consensus_queue.cc:1048] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 13:01:32.478830  7135 consensus_queue.cc:1048] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 13:01:32.479621  6985 raft_consensus.cc:493] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Starting leader election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:01:32.480144  6985 raft_consensus.cc:3060] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:01:32.482849  7234 raft_consensus.cc:2804] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:01:32.483299  7234 raft_consensus.cc:493] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Starting leader election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:01:32.483695  7234 raft_consensus.cc:3060] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:01:32.478994  1718 leader_election.cc:304] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:32.408835  7227 raft_consensus.cc:515] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:01:32.508044  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 269989us
I20260709 13:01:32.493467  7219 leader_election.cc:290] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 16 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:01:32.555711  1644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:32.563383  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 61213.1us
W20260709 13:01:32.565253  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 207964us
I20260709 13:01:32.575425  1718 leader_election.cc:304] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:32.502976  7135 raft_consensus.cc:2804] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:01:32.497534  7143 raft_consensus.cc:493] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:32.499938  7243 raft_consensus.cc:2804] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader pre-election won for term 13
W20260709 13:01:32.568089  1643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:32.584617  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:32.596312  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:01:32.596897  1643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:32.589066  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:32.592804  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:32.596979  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:32.599787  1671 raft_consensus.cc:2393] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 14: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:01:32.599675  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:32.600210  1680 raft_consensus.cc:2393] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 15: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:01:32.600080  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:32.603919  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:32.596273  7243 raft_consensus.cc:493] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:32.601982  1677 raft_consensus.cc:2393] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 15: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:01:32.605978  2328 raft_consensus.cc:2368] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for earlier term 11. Current term is 12.
W20260709 13:01:32.607097  1644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:32.619191  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:32.619940  1569 leader_election.cc:304] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:01:32.620793  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 14 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:32.620878  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:32.629077  1678 raft_consensus.cc:2393] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 16: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:01:32.629168  1673 raft_consensus.cc:2410] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for term 14 because replica has last-logged OpId of term: 12 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 6.
I20260709 13:01:32.624183  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:32.629678  1569 leader_election.cc:304] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:01:32.630273  1677 raft_consensus.cc:2393] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 16: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
W20260709 13:01:32.625684  1646 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:32.631101  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 189107us
W20260709 13:01:32.631448  1646 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:32.631857  1571 leader_election.cc:304] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
W20260709 13:01:32.642376  1717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:32.608930  7243 raft_consensus.cc:3060] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:01:32.605844  7192 leader_election.cc:290] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:01:32.736635  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 91078.1us
I20260709 13:01:32.738973  1571 leader_election.cc:304] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:01:32.600486  7290 raft_consensus.cc:2804] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 13:01:32.606871  7135 raft_consensus.cc:697] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 14 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:01:32.653151  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:01:32.659307  1644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:32.606814  7283 raft_consensus.cc:2749] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:01:32.733366  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:01:32.733886  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 118755us
I20260709 13:01:32.601375  7143 raft_consensus.cc:515] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:32.745833  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:32.615950  7111 raft_consensus.cc:697] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:01:32.752512  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:32.763229  1676 raft_consensus.cc:3060] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:01:32.644649  7219 raft_consensus.cc:3060] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:01:32.644526  7297 raft_consensus.cc:3060] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:01:32.823065  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:32.824045  1748 raft_consensus.cc:2468] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 13.
I20260709 13:01:32.734566  7289 raft_consensus.cc:3060] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Advancing to term 14
W20260709 13:01:32.827740  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 75051.1us
W20260709 13:01:32.867666  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 127269us
I20260709 13:01:32.878046  1571 leader_election.cc:304] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:32.879338  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:32.885238  1748 raft_consensus.cc:1240] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 10. Current term is 11. Ops: []
I20260709 13:01:32.890229  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:32.892000  6160 raft_consensus.cc:493] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
W20260709 13:01:32.887439  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 253994us
I20260709 13:01:32.892020  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:32.890506  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:32.895957  2328 raft_consensus.cc:2393] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 16: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:01:32.759810  7290 raft_consensus.cc:697] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 15 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:01:32.902871  7135 consensus_queue.cc:237] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.8, Last appended by leader: 8, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:32.825093  7036 raft_consensus.cc:493] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:32.930941  7036 raft_consensus.cc:515] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:32.933873  7036 leader_election.cc:290] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:01:32.937568  7106 consensus_meta.cc:287] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.726s	user 0.009s	sys 0.003s
I20260709 13:01:32.902403  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:01:32.904956  7294 log.cc:927] Time spent T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.315s	user 0.000s	sys 0.000s
I20260709 13:01:32.910784  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
W20260709 13:01:32.911862  7291 log.cc:927] Time spent T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.164s	user 0.000s	sys 0.000s
I20260709 13:01:32.913156  6160 raft_consensus.cc:515] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:32.897174  7247 raft_consensus.cc:493] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:01:32.917100  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:01:32.943727  7126 consensus_meta.cc:287] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.639s	user 0.015s	sys 0.000s
W20260709 13:01:32.944984  7188 consensus_meta.cc:287] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 1.079s	user 0.012s	sys 0.012s
I20260709 13:01:32.953903  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:01:32.954607  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 64730.2us
I20260709 13:01:32.874866  7278 raft_consensus.cc:493] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:32.900540  7283 raft_consensus.cc:2749] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:01:32.879844  7287 raft_consensus.cc:493] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:32.900408  7244 raft_consensus.cc:493] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:32.973002  1676 raft_consensus.cc:2468] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 15.
I20260709 13:01:32.894961  1646 leader_election.cc:304] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 13:01:32.965658  1717 leader_election.cc:304] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:32.983397  2301 leader_election.cc:304] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:32.902948  7305 raft_consensus.cc:2804] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader pre-election won for term 15
W20260709 13:01:32.991672  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 96786.8us
I20260709 13:01:32.993176  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:32.993880  1676 raft_consensus.cc:2376] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 15. Re-sending same reply.
I20260709 13:01:33.002650  2330 raft_consensus.cc:1240] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 14. Current term is 15. Ops: []
I20260709 13:01:33.003046  1597 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 2589) took 1311 ms. Trace:
I20260709 13:01:33.019487  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:33.020507  1680 raft_consensus.cc:2393] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 16: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
W20260709 13:01:33.021638  2337 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39738 (request call id 1020) took 1336 ms (client timeout 1735 ms). Trace:
I20260709 13:01:33.003623  1597 rpcz_store.cc:276] 0709 13:01:31.691944 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:31.695318 (+  3374us) service_pool.cc:224] Handling call
0709 13:01:31.728917 (+ 33599us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:32.409240 (+680323us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:33.002711 (+593471us) spinlock_profiling.cc:243] Waited 671 ms on lock 0x7b54002166d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:01:33.002955 (+   244us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1542769408}
I20260709 13:01:32.950196  7234 raft_consensus.cc:515] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:33.026567  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
W20260709 13:01:33.028188  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 62619.4us
I20260709 13:01:33.032008  1717 leader_election.cc:304] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 13:01:33.022075  2337 rpcz_store.cc:269] 0709 13:01:31.666344 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:31.666588 (+   244us) service_pool.cc:224] Handling call
0709 13:01:33.002097 (+1335509us) spinlock_profiling.cc:243] Waited 573 ms on lock 0x7b540065e0d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:01:33.002272 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1317770112}
I20260709 13:01:33.033879  1644 leader_election.cc:400] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 11 election: Vote denied by peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625) with higher term. Message: Invalid argument: T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for earlier term 11. Current term is 12.
I20260709 13:01:33.034302  1644 leader_election.cc:403] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 11 election: Cancelling election due to peer responding with higher term
I20260709 13:01:33.010756  7188 raft_consensus.cc:2804] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:01:32.995456  7227 leader_election.cc:290] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:32.995136  7305 raft_consensus.cc:493] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:33.031006  1673 raft_consensus.cc:2376] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 16. Re-sending same reply.
I20260709 13:01:33.051328  6160 leader_election.cc:290] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:33.055974  6160 raft_consensus.cc:2749] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:01:32.967262  7106 raft_consensus.cc:515] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:01:33.060850  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 49743.8us
I20260709 13:01:33.034614  7126 raft_consensus.cc:2749] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:01:32.994509  7014 consensus_queue.cc:1048] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.049s
I20260709 13:01:32.969473  7247 raft_consensus.cc:515] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:33.040146  7263 raft_consensus.cc:2804] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 13:01:33.040024  7200 raft_consensus.cc:3060] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:01:32.998224  7244 raft_consensus.cc:515] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:33.076090  1747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55848 (request call id 2775) took 1057 ms. Trace:
I20260709 13:01:32.998911  7278 raft_consensus.cc:515] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:33.083842  2336 raft_consensus.cc:2468] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 12.
I20260709 13:01:33.084638  7278 leader_election.cc:290] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:01:33.085752  7296 log.cc:927] Time spent T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.186s	user 0.000s	sys 0.001s
I20260709 13:01:33.050770  7188 raft_consensus.cc:493] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:33.048138  7305 raft_consensus.cc:3060] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:01:33.079843  1747 rpcz_store.cc:276] 0709 13:01:32.018530 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:32.018843 (+   313us) service_pool.cc:224] Handling call
0709 13:01:32.047454 (+ 28611us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:32.234697 (+187243us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:32.234741 (+    44us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:01:32.257348 (+ 22607us) log.cc:844] Serialized 26 byte log entry
0709 13:01:32.447478 (+190130us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:01:32.447526 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:33.004867 (+557341us) spinlock_profiling.cc:243] Waited 167 ms on lock 0x7b54001d60d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:01:33.004952 (+    85us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:33.068470 (+ 63518us) raft_consensus.cc:1685] finished
0709 13:01:33.068506 (+    36us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:33.076005 (+  7499us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5511,"raft.queue_time_us":184,"raft.run_cpu_time_us":237,"raft.run_wall_time_us":25348,"spinlock_wait_cycles":385064832,"thread_start_us":115615,"threads_started":1,"wal-append.queue_time_us":211638}
I20260709 13:01:33.103492  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:33.104386  1718 leader_election.cc:304] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:32.986155  7287 raft_consensus.cc:515] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:33.106963  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:33.118096  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:33.115659  6905 raft_consensus.cc:493] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:33.114180  7126 raft_consensus.cc:2749] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:01:33.120532  6905 raft_consensus.cc:515] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:33.100817  7290 consensus_queue.cc:237] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 14.7, Last appended by leader: 7, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:33.103878  7227 raft_consensus.cc:493] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:33.128136  7227 raft_consensus.cc:515] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:33.130857  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:33.132402  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:33.133031  1605 raft_consensus.cc:2376] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 14. Re-sending same reply.
I20260709 13:01:33.136090  7289 raft_consensus.cc:2749] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:01:33.127509  7287 leader_election.cc:290] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:33.139781  2331 raft_consensus.cc:1275] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
W20260709 13:01:33.140434  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 69116.9us
W20260709 13:01:33.169395  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 130530us
I20260709 13:01:33.113442  1718 leader_election.cc:304] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:33.144080  1600 raft_consensus.cc:1275] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 16 index: 9. (index mismatch)
I20260709 13:01:33.170547  1717 leader_election.cc:304] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:33.147807  7234 leader_election.cc:290] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 16 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:33.153906  7106 leader_election.cc:290] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 12 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:33.154747  7143 leader_election.cc:290] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:33.155157  7233 raft_consensus.cc:493] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:01:33.155618  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:33.156476  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:33.047308  7111 consensus_queue.cc:237] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:33.159780  7247 leader_election.cc:290] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:33.147408  7243 raft_consensus.cc:515] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:33.175122  1717 leader_election.cc:304] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:33.174808  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:33.178828  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:33.179132  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:33.109172  7188 raft_consensus.cc:3060] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:01:33.181638  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:01:33.182235  1644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:33.172592  6997 consensus_meta.cc:287] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.842s	user 0.012s	sys 0.005s
W20260709 13:01:33.184396  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 71074.1us
I20260709 13:01:33.189203  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:33.190129  1598 raft_consensus.cc:2468] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 10.
I20260709 13:01:33.186038  1603 raft_consensus.cc:2468] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 10.
W20260709 13:01:33.187233  1644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:33.173028  6976 consensus_meta.cc:287] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.761s	user 0.012s	sys 0.002s
I20260709 13:01:33.185019  7307 raft_consensus.cc:3060] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Advancing to term 15
W20260709 13:01:33.195611  7067 consensus_meta.cc:287] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.788s	user 0.008s	sys 0.006s
I20260709 13:01:33.181344  7126 raft_consensus.cc:2749] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:01:33.197328  7263 raft_consensus.cc:697] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 13 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:01:33.175714  7189 raft_consensus.cc:493] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:01:33.198374  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 92955.3us
I20260709 13:01:33.199443  1675 raft_consensus.cc:2468] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 12.
W20260709 13:01:33.198618  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 92011.1us
I20260709 13:01:33.189231  1717 leader_election.cc:304] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
W20260709 13:01:33.206283  1644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:33.209348  1597 raft_consensus.cc:2393] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 13: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:01:33.209928  7243 leader_election.cc:290] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:33.210619  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:33.214128  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
W20260709 13:01:33.195628  6985 consensus_meta.cc:287] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.715s	user 0.013s	sys 0.002s
I20260709 13:01:33.221032  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:33.221125  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:33.221650  7189 raft_consensus.cc:515] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:33.205771  7247 raft_consensus.cc:2749] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:01:33.230502  1718 leader_election.cc:304] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:33.207547  6997 raft_consensus.cc:493] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:33.183514  7233 raft_consensus.cc:515] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:33.239323  1718 leader_election.cc:304] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:33.249933  7244 leader_election.cc:290] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:33.250581  7244 raft_consensus.cc:2749] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:01:33.236025  6985 raft_consensus.cc:515] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:33.254671  6985 leader_election.cc:290] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 15 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:33.243489  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:33.215698  7067 raft_consensus.cc:515] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:33.258348  7297 raft_consensus.cc:2749] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:01:33.271287  7311 raft_consensus.cc:2804] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 13:01:33.272190  7311 raft_consensus.cc:697] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 15 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:01:33.273218  7311 consensus_queue.cc:237] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 13.9, Last appended by leader: 9, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:33.273833  7067 leader_election.cc:290] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 17 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:33.276137  7305 raft_consensus.cc:515] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:33.277256  1599 raft_consensus.cc:1240] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 14. Current term is 15. Ops: []
I20260709 13:01:33.267297  7263 consensus_queue.cc:237] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 9, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:33.279067  1601 raft_consensus.cc:1240] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 16. Current term is 17. Ops: []
I20260709 13:01:33.279182  7305 leader_election.cc:290] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 15 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:33.284723  6997 raft_consensus.cc:515] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:33.285768  1679 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57092 (request call id 1236) took 1054 ms. Trace:
I20260709 13:01:33.287410  6997 leader_election.cc:290] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:33.289217  7233 leader_election.cc:290] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:33.292238  7143 raft_consensus.cc:493] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:33.293277  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:33.295270  7189 leader_election.cc:290] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:33.262905  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:33.298790  6160 raft_consensus.cc:2749] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:01:33.303426  7143 raft_consensus.cc:515] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:01:33.299513  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 61360.4us
I20260709 13:01:33.300071  2301 leader_election.cc:304] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:01:33.305545  7180 raft_consensus.cc:2749] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 13:01:33.306986  2301 leader_election.cc:304] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:01:33.303210  7247 raft_consensus.cc:493] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:33.308547  7247 raft_consensus.cc:515] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:33.308848  7143 leader_election.cc:290] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:33.310722  7247 leader_election.cc:290] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:33.311388  7287 raft_consensus.cc:2749] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:01:33.313516  7247 consensus_queue.cc:1048] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 13:01:33.302142  7314 consensus_queue.cc:1059] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 14.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.016s
I20260709 13:01:33.318678  7314 raft_consensus.cc:3055] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:01:33.319041  7314 raft_consensus.cc:740] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:01:33.319846  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:33.319977  7314 consensus_queue.cc:260] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.12, Last appended by leader: 12, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:33.321528  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:33.321709  7314 raft_consensus.cc:3060] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:01:33.323344  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:33.324077  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 15 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:33.324962  2337 raft_consensus.cc:2410] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for term 15 because replica has last-logged OpId of term: 13 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 10.
I20260709 13:01:33.326784  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:33.286345  1679 rpcz_store.cc:276] 0709 13:01:32.231008 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:32.231134 (+   126us) service_pool.cc:224] Handling call
0709 13:01:32.476172 (+245038us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:32.476648 (+   476us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:32.476682 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:01:32.477310 (+   628us) log.cc:844] Serialized 26 byte log entry
0709 13:01:32.587752 (+110442us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:01:32.587817 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:32.588005 (+   188us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:32.929204 (+341199us) raft_consensus.cc:1685] finished
0709 13:01:32.929228 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:33.285377 (+356149us) spinlock_profiling.cc:243] Waited 245 ms on lock 0x7b54000596d0. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661618ad 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:01:33.285672 (+   295us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1240,"raft.run_cpu_time_us":81,"raft.run_wall_time_us":81,"spinlock_wait_cycles":562429184,"thread_start_us":109928,"threads_started":1,"wal-append.queue_time_us":173017}
W20260709 13:01:33.329520  1646 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:33.330874  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 112729us
I20260709 13:01:33.343490  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:33.344298  1671 raft_consensus.cc:3060] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:01:33.345052  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:01:33.330245  1646 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:33.345822  1748 raft_consensus.cc:2468] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 11.
I20260709 13:01:33.348281  7319 raft_consensus.cc:493] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:33.348815  2299 leader_election.cc:304] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:33.348902  7319 raft_consensus.cc:515] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:33.351207  7181 raft_consensus.cc:2749] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:01:33.351897  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:33.352504  2333 raft_consensus.cc:3060] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:01:33.344419  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:33.348320  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:33.346230  1646 leader_election.cc:304] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 13:01:33.352679  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:01:33.350207  7219 consensus_meta.cc:287] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.527s	user 0.010s	sys 0.002s
I20260709 13:01:33.354727  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:33.362344  1717 leader_election.cc:304] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:33.362263  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:33.366110  1754 raft_consensus.cc:2468] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 16.
I20260709 13:01:33.367197  1745 raft_consensus.cc:2468] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 15.
I20260709 13:01:33.369983  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:33.370965  6160 raft_consensus.cc:2804] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:01:33.371450  6160 raft_consensus.cc:493] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:33.371791  6160 raft_consensus.cc:3060] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:01:33.371588  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:33.373049  1717 leader_election.cc:304] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:33.374691  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:33.267742  6905 leader_election.cc:290] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:33.330837  7200 raft_consensus.cc:2749] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Leader election lost for term 11. Reason: Vote denied by peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625) with higher term. Message: Invalid argument: T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for earlier term 11. Current term is 12.
I20260709 13:01:33.378310  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:33.382323  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:33.370891  1752 raft_consensus.cc:2468] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 16.
I20260709 13:01:33.379130  1676 raft_consensus.cc:1240] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 13. Current term is 14. Ops: []
I20260709 13:01:33.382665  1717 leader_election.cc:304] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:33.385239  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:33.386201  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:33.390472  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:33.392531  1569 leader_election.cc:304] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:01:33.397274  2335 raft_consensus.cc:2393] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 16: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:01:33.409071  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:33.373523  7219 raft_consensus.cc:2749] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:01:33.421962  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:33.411970  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:33.411559  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:33.416889  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:33.408401  1671 raft_consensus.cc:2468] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 12.
W20260709 13:01:33.426083  1643 leader_election.cc:336] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 15 election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 1.762s (SENT)
I20260709 13:01:33.429036  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:33.424643  1600 raft_consensus.cc:2468] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 14.
I20260709 13:01:33.429852  1598 raft_consensus.cc:2468] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 16.
I20260709 13:01:33.422964  1603 raft_consensus.cc:2468] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 16.
I20260709 13:01:33.428330  7067 raft_consensus.cc:2804] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:01:33.434221  6643 consensus_queue.cc:1059] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 16.13, Next index: 14, Last known committed idx: 12, Time since last communication: 0.057s
I20260709 13:01:33.434551  7067 raft_consensus.cc:493] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
W20260709 13:01:33.434808  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 98328.2us
I20260709 13:01:33.435097  7067 raft_consensus.cc:3060] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:01:33.435266  6643 raft_consensus.cc:3055] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 13:01:33.435734  6643 raft_consensus.cc:740] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:01:33.436641  1718 leader_election.cc:304] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:33.436875  6643 consensus_queue.cc:260] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 16.13, Last appended by leader: 13, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:33.438732  7164 raft_consensus.cc:2804] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:01:33.439184  7164 raft_consensus.cc:493] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:33.439549  7164 raft_consensus.cc:3060] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:01:33.441610  7126 raft_consensus.cc:2804] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 13:01:33.443282  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:33.444923  7067 raft_consensus.cc:515] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } }
I20260709 13:01:33.447132  7067 leader_election.cc:290] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 12 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:33.378382  7143 raft_consensus.cc:493] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:33.450359  1569 leader_election.cc:304] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:01:33.390564  7244 raft_consensus.cc:2749] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:01:33.438203  6643 raft_consensus.cc:3060] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:01:33.456271  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:33.442198  7126 raft_consensus.cc:697] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 16 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:01:33.370185  7247 consensus_queue.cc:1048] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.048s
I20260709 13:01:33.460939  7067 raft_consensus.cc:493] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:33.461895  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:33.404323  7280 raft_consensus.cc:493] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:33.403606  7229 raft_consensus.cc:493] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:33.465826  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 14 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:33.469864  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 15 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:33.411744  7233 raft_consensus.cc:2804] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:01:33.470713  7143 raft_consensus.cc:515] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:33.413770  7305 raft_consensus.cc:493] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:33.474418  7227 leader_election.cc:290] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:33.475616  6976 raft_consensus.cc:493] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:33.431521  6997 raft_consensus.cc:493] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:33.477095  7319 leader_election.cc:290] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 13:01:33.488061  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 96136us
I20260709 13:01:33.461913  1596 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40154 (request call id 2492) took 1603 ms. Trace:
W20260709 13:01:33.494602  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 44361.2us
I20260709 13:01:33.503901  2333 raft_consensus.cc:2468] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 13.
I20260709 13:01:33.498334  1673 raft_consensus.cc:2468] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 12.
I20260709 13:01:33.544260  1569 leader_election.cc:304] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:33.452502  7290 raft_consensus.cc:493] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:33.537004  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
W20260709 13:01:33.542837  1643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:33.506084  7301 raft_consensus.cc:2749] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:01:33.542297  1596 rpcz_store.cc:276] 0709 13:01:31.858498 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:31.883754 (+ 25256us) service_pool.cc:224] Handling call
0709 13:01:31.884127 (+   373us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:01:31.884534 (+   407us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:31.884565 (+    31us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:01:31.885339 (+   774us) log.cc:844] Serialized 52 byte log entry
0709 13:01:32.134384 (+249045us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:01:32.188962 (+ 54578us) spinlock_profiling.cc:243] Waited 44.6 ms on lock 0x7b4800003cd8. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426b08baa1 00007f426b082b37 00007f426b0829ca 00007f426b17a26f 00007f426b179f37 00007f426b18e55c 00007f426b18e508 00007f426b18e488 00007f426b18e443 00007f426b18d6ec 00007f426a0b7a96
0709 13:01:33.294883 (+1105921us) spinlock_profiling.cc:243] Waited 23.6 ms on lock 0x7b18000fd2a0. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f4265d0a4bc 00007f426305848b 00007f426beb2ab9 00007f426307469b 00007f426307293f 00007f4263072455 00007f4269b5a3c7 00007f426616c9a7 00007f426617c0bf
0709 13:01:33.295128 (+   245us) log.cc:844] Serialized 14 byte log entry
0709 13:01:33.461562 (+166434us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:33.461654 (+    92us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:33.461700 (+    46us) raft_consensus.cc:1685] finished
0709 13:01:33.461713 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:33.461847 (+   134us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4257,"prepare.queue_time_us":245647,"prepare.run_cpu_time_us":87,"prepare.run_wall_time_us":2237,"raft.queue_time_us":12881,"raft.run_cpu_time_us":3872,"raft.run_wall_time_us":302972,"spinlock_wait_cycles":156864512,"thread_start_us":648029,"threads_started":3,"wal-append.queue_time_us":585231,"wal-append.run_cpu_time_us":2216,"wal-append.run_wall_time_us":1027883}
I20260709 13:01:33.547312  1569 leader_election.cc:304] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
W20260709 13:01:33.548889  1643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:33.552615  6876 consensus_queue.cc:1059] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 14.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 13:01:33.546257  6997 raft_consensus.cc:515] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:33.572978  6876 raft_consensus.cc:3055] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:01:33.573360  6876 raft_consensus.cc:740] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:01:33.573469  7301 raft_consensus.cc:2749] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:01:33.574339  6876 consensus_queue.cc:260] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.10, Last appended by leader: 10, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:33.577082  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:33.577873  1749 raft_consensus.cc:3060] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Advancing to term 12
I20260709 13:01:33.524111  7229 raft_consensus.cc:515] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:33.546290  7200 raft_consensus.cc:493] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:33.542935  7305 raft_consensus.cc:515] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:33.553831  1746 raft_consensus.cc:2393] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 16: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:01:33.554705  7143 leader_election.cc:290] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:33.598049  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:33.599049  1745 raft_consensus.cc:2468] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 12.
I20260709 13:01:33.599747  6160 raft_consensus.cc:515] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:33.600266  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:33.601123  1751 raft_consensus.cc:3060] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Advancing to term 13
I20260709 13:01:33.601840  1643 leader_election.cc:304] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:33.557060  7290 raft_consensus.cc:515] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:33.512156  7233 raft_consensus.cc:493] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:33.512665  7280 raft_consensus.cc:515] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:33.560190  7188 raft_consensus.cc:515] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:33.621722  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:33.584097  7146 raft_consensus.cc:493] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:01:33.566637  6938 raft_consensus.cc:493] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:01:33.546103  6976 raft_consensus.cc:515] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:33.634891  1717 leader_election.cc:304] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:33.577138  1569 leader_election.cc:304] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:01:33.582762  2299 leader_election.cc:304] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:33.578186  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
W20260709 13:01:33.609758  1644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:33.611300  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:33.578696  7227 raft_consensus.cc:2749] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:01:33.639038  1604 raft_consensus.cc:2468] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 14.
I20260709 13:01:33.618635  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:33.511719  7307 raft_consensus.cc:2749] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:01:33.641901  1717 leader_election.cc:304] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:33.626580  6160 leader_election.cc:290] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:33.627249  2301 leader_election.cc:304] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:01:33.644548  2301 leader_election.cc:304] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:33.541149  7067 raft_consensus.cc:515] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:33.630867  1571 leader_election.cc:304] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:01:33.648182  1571 leader_election.cc:304] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:01:33.632055  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:33.632411  1603 raft_consensus.cc:2368] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 11. Current term is 12.
I20260709 13:01:33.547405  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:33.564253  7164 raft_consensus.cc:515] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:33.655464  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 72821.6us
I20260709 13:01:33.646166  2301 leader_election.cc:304] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
W20260709 13:01:33.637900  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 60831.4us
I20260709 13:01:33.620460  7233 raft_consensus.cc:3060] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:01:33.649030  1672 raft_consensus.cc:1240] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 15. Current term is 16. Ops: []
I20260709 13:01:33.632850  6882 raft_consensus.cc:2749] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:01:33.667086  1753 raft_consensus.cc:1240] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 14. Current term is 15. Ops: []
I20260709 13:01:33.667336  1602 raft_consensus.cc:2468] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 15.
I20260709 13:01:33.623066  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:33.610066  7298 raft_consensus.cc:493] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:33.653678  1747 raft_consensus.cc:2393] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 12: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
W20260709 13:01:33.653960  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 217385us
I20260709 13:01:33.603631  7200 raft_consensus.cc:515] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:33.626796  7319 raft_consensus.cc:493] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:33.632166  7311 raft_consensus.cc:493] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:33.687371  7227 raft_consensus.cc:493] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:01:33.649663  1571 leader_election.cc:304] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:01:33.676934  1678 raft_consensus.cc:1240] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 13. Current term is 14. Ops: []
I20260709 13:01:33.647324  7111 raft_consensus.cc:2804] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Leader election won for term 13
W20260709 13:01:33.822928  1644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:33.651084  6876 raft_consensus.cc:3060] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:01:33.679376  1569 leader_election.cc:304] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:33.687098  1718 leader_election.cc:304] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:33.655158  7297 raft_consensus.cc:2749] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 13:01:33.657786  7219 raft_consensus.cc:2749] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:01:33.658789  7014 raft_consensus.cc:2749] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:01:33.635013  7146 raft_consensus.cc:515] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:33.947069  7146 leader_election.cc:290] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:33.663166  7310 raft_consensus.cc:2804] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:01:33.948010  7310 raft_consensus.cc:697] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 12 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:01:33.949214  7310 consensus_queue.cc:237] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:33.834607  7319 raft_consensus.cc:515] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:33.666757  7126 consensus_queue.cc:237] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.10, Last appended by leader: 10, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:33.953070  7319 leader_election.cc:290] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:33.953812  7319 raft_consensus.cc:2749] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:01:33.954499  7301 raft_consensus.cc:2749] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 13:01:33.655481  7279 raft_consensus.cc:2764] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Leader pre-election decision vote started in defunct term 12: won
I20260709 13:01:33.654090  6938 raft_consensus.cc:515] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:33.680971  7244 raft_consensus.cc:2749] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:01:33.968145  6938 leader_election.cc:290] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:33.689706  6985 raft_consensus.cc:2749] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
W20260709 13:01:33.723621  7321 log.cc:927] Time spent T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.084s	user 0.000s	sys 0.001s
I20260709 13:01:33.970690  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:01:33.669775  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 90969.3us
I20260709 13:01:33.973204  1643 leader_election.cc:304] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:01:33.653409  7135 raft_consensus.cc:2804] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:01:33.974704  7135 raft_consensus.cc:493] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:33.975216  7135 raft_consensus.cc:3060] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:01:33.975533  1750 raft_consensus.cc:1240] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 10. Current term is 11. Ops: []
I20260709 13:01:33.976294  6997 leader_election.cc:290] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:33.977473  6997 raft_consensus.cc:2749] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20260709 13:01:33.823215  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 173615us
I20260709 13:01:33.979001  1571 leader_election.cc:304] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:33.979141  2331 raft_consensus.cc:1240] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 14. Current term is 15. Ops: []
I20260709 13:01:33.980195  1571 leader_election.cc:304] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
W20260709 13:01:33.866514  1644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:33.863041  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 230158us
I20260709 13:01:33.663250  1752 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5"
dest_uuid: "af520b67945d443b901bbeca37cc5701"
mode: GRACEFUL
new_leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
 from {username='slave'} at 127.0.0.1:39850
I20260709 13:01:33.982501  1752 raft_consensus.cc:606] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Received request to transfer leadership to TS 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:33.982883  1752 raft_consensus.cc:614] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 13:01:33.931282  1749 raft_consensus.cc:2468] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 12.
W20260709 13:01:33.932699  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 253403us
W20260709 13:01:33.941840  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 254838us
W20260709 13:01:33.942548  2300 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:33.871582  7289 raft_consensus.cc:2749] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 13:01:33.987056  1751 raft_consensus.cc:2468] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 13.
I20260709 13:01:33.990356  1748 raft_consensus.cc:2468] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 16.
I20260709 13:01:33.877521  7111 raft_consensus.cc:697] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:01:33.981766  1646 leader_election.cc:304] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:33.997673  1646 leader_election.cc:304] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:33.996423  1569 leader_election.cc:304] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
W20260709 13:01:33.984123  7318 log.cc:927] Time spent T 00000000000000000000000000000000 P 919f6392d4f44fd68987e2c5e7ec37ca: Append to log took a long time: real 0.363s	user 0.007s	sys 0.000s
I20260709 13:01:34.020244  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:34.022979  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:34.030851  1673 raft_consensus.cc:2468] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 15.
W20260709 13:01:34.047693  1644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:34.058377  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 60801.6us
I20260709 13:01:33.864354  7311 raft_consensus.cc:515] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:34.066066  1675 raft_consensus.cc:1240] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 10. Current term is 11. Ops: []
I20260709 13:01:34.059543  1646 leader_election.cc:304] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:33.878119  7227 raft_consensus.cc:515] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:33.990137  7301 raft_consensus.cc:2804] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:01:33.783051  7298 raft_consensus.cc:515] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:34.078588  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:34.095685  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:34.095364  2330 raft_consensus.cc:2468] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 16.
I20260709 13:01:34.005652  7219 raft_consensus.cc:2749] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
W20260709 13:01:34.093987  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 112539us
I20260709 13:01:34.106078  1671 raft_consensus.cc:2468] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 12.
I20260709 13:01:34.106900  1643 leader_election.cc:304] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:01:34.108019  1672 raft_consensus.cc:1275] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 11 index: 9. Preceding OpId from leader: term: 13 index: 10. (index mismatch)
W20260709 13:01:34.093266  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 113118us
I20260709 13:01:34.109674  1571 leader_election.cc:304] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:01:34.068395  6882 raft_consensus.cc:2804] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:01:34.070641  7289 raft_consensus.cc:493] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:34.114535  1745 raft_consensus.cc:2468] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 10.
I20260709 13:01:34.114810  1596 raft_consensus.cc:2468] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 13.
I20260709 13:01:33.999553  7146 consensus_queue.cc:1048] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 7, Time since last communication: 0.109s
I20260709 13:01:34.005012  7297 consensus_queue.cc:1059] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 13.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.101s
I20260709 13:01:34.091991  7301 raft_consensus.cc:493] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:34.101034  7305 leader_election.cc:290] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:34.116937  6882 raft_consensus.cc:493] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:34.100847  7315 consensus_queue.cc:1059] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 15.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.029s
I20260709 13:01:34.122058  7289 raft_consensus.cc:515] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:34.142777  7281 raft_consensus.cc:2804] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:01:34.126258  7290 leader_election.cc:290] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:34.130941  7219 raft_consensus.cc:2749] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 13:01:34.132810  7301 raft_consensus.cc:3060] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:01:34.170691  7326 raft_consensus.cc:493] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:34.198482  6997 raft_consensus.cc:2804] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:01:34.198947  6997 raft_consensus.cc:493] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:34.199344  6997 raft_consensus.cc:3060] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:01:34.201602  7146 raft_consensus.cc:2749] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 13:01:34.152041  6882 raft_consensus.cc:3060] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:01:34.163218  7111 consensus_queue.cc:237] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 11.11, Last appended by leader: 11, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:34.149518  7229 leader_election.cc:290] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:34.198487  7326 raft_consensus.cc:515] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:34.187000  1717 leader_election.cc:336] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 15 election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 1.501s (SENT)
W20260709 13:01:34.172248  7339 log.cc:927] Time spent T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.051s	user 0.000s	sys 0.000s
I20260709 13:01:34.174463  7337 consensus_queue.cc:1048] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.179s
W20260709 13:01:34.177695  7328 log.cc:927] Time spent T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.249s	user 0.001s	sys 0.000s
I20260709 13:01:34.183724  7311 leader_election.cc:290] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:34.168663  7281 raft_consensus.cc:697] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:01:34.184592  7297 raft_consensus.cc:3055] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 LEADER]: Stepping down as leader of term 13
I20260709 13:01:34.223089  7297 raft_consensus.cc:740] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:01:34.223592  7281 consensus_queue.cc:237] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.8, Last appended by leader: 8, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:34.224159  7297 consensus_queue.cc:260] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.8, Last appended by leader: 8, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:34.137158  7188 leader_election.cc:290] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 16 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:01:34.187618  7331 log.cc:927] Time spent T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.366s	user 0.001s	sys 0.001s
I20260709 13:01:34.187816  7219 raft_consensus.cc:493] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:34.227264  7219 raft_consensus.cc:515] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:34.189783  7227 leader_election.cc:290] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:34.159077  7338 raft_consensus.cc:2804] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:01:34.241093  7338 raft_consensus.cc:493] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:34.241637  7338 raft_consensus.cc:3060] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:01:34.244705  7219 leader_election.cc:290] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:34.252411  6976 leader_election.cc:290] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:01:34.193370  6643 consensus_meta.cc:287] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.690s	user 0.010s	sys 0.000s
I20260709 13:01:34.193572  7315 raft_consensus.cc:3055] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 13:01:34.261720  7315 raft_consensus.cc:740] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:01:34.262830  7315 consensus_queue.cc:260] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 15.13, Last appended by leader: 13, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:34.264282  7315 raft_consensus.cc:3060] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:01:34.265358  1602 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40154 (request call id 2505) took 1122 ms. Trace:
I20260709 13:01:34.265803  1602 rpcz_store.cc:276] 0709 13:01:33.142594 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:33.142739 (+   145us) service_pool.cc:224] Handling call
0709 13:01:34.265173 (+1122434us) spinlock_profiling.cc:243] Waited 275 ms on lock 0x7b54002797d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:01:34.265293 (+   120us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":633213312}
I20260709 13:01:34.269837  7280 leader_election.cc:290] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:34.194991  7298 leader_election.cc:290] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:34.158855  7067 leader_election.cc:290] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:34.197860  7200 leader_election.cc:290] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:01:34.285630  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 98656.9us
I20260709 13:01:34.264292  7164 leader_election.cc:290] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 14 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:34.358847  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:34.359655  2330 raft_consensus.cc:1275] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 14 index: 11. (term mismatch)
I20260709 13:01:34.360379  2330 pending_rounds.cc:85] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1: Aborting all ops after (but not including) 10
I20260709 13:01:34.360770  2330 pending_rounds.cc:107] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1: Aborting uncommitted NO_OP operation due to leader change: 13.11
I20260709 13:01:34.361068  2330 raft_consensus.cc:2889] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:01:34.363559  7298 raft_consensus.cc:493] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
W20260709 13:01:34.363924  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 249385us
I20260709 13:01:34.364279  7298 raft_consensus.cc:515] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:34.365926  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:34.353706  2298 leader_election.cc:304] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:01:34.326718  7311 raft_consensus.cc:493] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:34.330621  7290 raft_consensus.cc:493] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:34.374413  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:34.378854  7326 leader_election.cc:290] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:34.382344  2328 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31"
dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
mode: GRACEFUL
new_leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
 from {username='slave'} at 127.0.0.1:35790
W20260709 13:01:34.392566  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 281246us
W20260709 13:01:34.396663  1569 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:34.398072  7311 raft_consensus.cc:515] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:34.396196  2328 raft_consensus.cc:606] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 12 LEADER]: Received request to transfer leadership to TS 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:34.399493  1571 leader_election.cc:304] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1
W20260709 13:01:34.407068  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 347606us
W20260709 13:01:34.417259  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 67293.9us
I20260709 13:01:34.399765  1569 leader_election.cc:304] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:01:34.425511  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:34.426538  1750 raft_consensus.cc:2468] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 12.
I20260709 13:01:34.395722  7289 leader_election.cc:290] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:34.428939  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:34.429875  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:34.432005  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:34.432667  1750 raft_consensus.cc:2468] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 13.
I20260709 13:01:34.434490  7326 raft_consensus.cc:493] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:34.434666  1718 leader_election.cc:304] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:34.435279  7326 raft_consensus.cc:515] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:34.437501  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:34.438627  7244 raft_consensus.cc:493] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:34.439308  7244 raft_consensus.cc:515] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:34.441684  7244 leader_election.cc:290] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:34.442023  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:34.443078  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:34.443647  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:34.444020  2333 raft_consensus.cc:2393] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 15: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:01:34.443965  1672 raft_consensus.cc:2468] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 14.
I20260709 13:01:34.444406  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:34.445225  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:34.442397  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:34.446200  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:34.447438  7244 raft_consensus.cc:2804] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:01:34.447947  7244 raft_consensus.cc:493] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:34.448544  7244 raft_consensus.cc:3060] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:01:34.448802  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:34.449595  1672 raft_consensus.cc:2468] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 11.
I20260709 13:01:34.450055  7360 consensus_queue.cc:1059] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 14.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 13:01:34.438199  7326 leader_election.cc:290] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:34.458424  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:34.412425  7264 raft_consensus.cc:2804] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:01:34.460028  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:34.413223  7126 raft_consensus.cc:493] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
W20260709 13:01:34.462404  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 64519.2us
I20260709 13:01:34.463289  1678 raft_consensus.cc:2468] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 11.
W20260709 13:01:34.462204  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 45473.6us
I20260709 13:01:34.403991  7290 raft_consensus.cc:515] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:34.428364  7305 raft_consensus.cc:2804] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Leader election won for term 16
I20260709 13:01:34.466545  1571 leader_election.cc:304] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
W20260709 13:01:34.565956  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 99492.1us
I20260709 13:01:34.513929  1569 leader_election.cc:304] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:01:34.574349  1717 leader_election.cc:400] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 pre-election: Vote denied by peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445) with higher term. Message: Invalid argument: T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 11. Current term is 12.
I20260709 13:01:34.574908  1717 leader_election.cc:403] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:01:34.418370  7342 consensus_queue.cc:1048] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.165s
W20260709 13:01:34.573701  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 107178us
W20260709 13:01:34.592645  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 78874.4us
I20260709 13:01:34.463172  7264 raft_consensus.cc:493] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Starting leader election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:01:34.559551  7143 raft_consensus.cc:493] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:34.595938  1569 leader_election.cc:304] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
W20260709 13:01:34.598908  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 149682us
W20260709 13:01:34.602854  1718 leader_election.cc:336] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 15 election: RPC error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Timed out: RequestConsensusVote RPC to 127.0.107.194:46261 timed out after 1.843s (SENT)
I20260709 13:01:34.603459  1718 leader_election.cc:304] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:34.607952  1569 leader_election.cc:304] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:01:34.576660  7067 raft_consensus.cc:2804] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:01:34.611846  2336 raft_consensus.cc:1240] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 12 LEADER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 11. Current term is 12. Ops: []
I20260709 13:01:34.611920  1569 leader_election.cc:304] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:34.562788  7347 raft_consensus.cc:2804] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:01:34.619006  1643 leader_election.cc:304] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:01:34.597631  7219 raft_consensus.cc:2804] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:01:34.470029  7126 raft_consensus.cc:515] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:34.592074  7305 raft_consensus.cc:697] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:01:34.558969  7316 consensus_queue.cc:1059] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 11.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.144s
W20260709 13:01:34.617072  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 195986us
I20260709 13:01:34.609733  7337 raft_consensus.cc:493] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:34.576004  7350 consensus_queue.cc:1059] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 10.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.049s
W20260709 13:01:34.612386  7135 consensus_meta.cc:287] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.633s	user 0.008s	sys 0.008s
I20260709 13:01:34.598878  7264 raft_consensus.cc:3060] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:01:34.609229  7229 raft_consensus.cc:493] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:34.655435  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:34.655774  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:34.656066  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:34.657392  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:34.663450  1751 raft_consensus.cc:2468] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 14.
I20260709 13:01:34.685415  1752 raft_consensus.cc:2393] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 11: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
W20260709 13:01:34.655819  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 85883.5us
I20260709 13:01:34.614821  7111 raft_consensus.cc:2804] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader pre-election won for term 15
W20260709 13:01:34.612288  7233 consensus_meta.cc:287] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.930s	user 0.006s	sys 0.010s
I20260709 13:01:34.616434  7067 raft_consensus.cc:493] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:34.616004  7280 consensus_queue.cc:1048] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.081s
I20260709 13:01:34.775990  7281 raft_consensus.cc:2749] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:01:34.621383  7347 raft_consensus.cc:493] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:34.610240  7143 raft_consensus.cc:515] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:01:34.890726  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 278824us
I20260709 13:01:34.735544  7067 raft_consensus.cc:3060] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:01:34.894102  7352 raft_consensus.cc:493] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:34.894639  7345 raft_consensus.cc:2804] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:01:34.711984  7111 raft_consensus.cc:493] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:01:34.922214  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 303281us
I20260709 13:01:34.929179  1643 leader_election.cc:304] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:01:34.735102  7233 raft_consensus.cc:515] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:34.930751  1643 leader_election.cc:304] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:01:34.685024  7229 raft_consensus.cc:515] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:34.876330  7314 raft_consensus.cc:493] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:34.660486  7337 raft_consensus.cc:515] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:34.672107  7135 raft_consensus.cc:515] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:34.956418  1643 leader_election.cc:304] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:34.814286  7347 raft_consensus.cc:3060] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:01:34.923898  7345 raft_consensus.cc:493] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:34.970860  1747 raft_consensus.cc:1240] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 10. Current term is 11. Ops: []
I20260709 13:01:34.975090  1643 leader_election.cc:304] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:01:34.927598  7298 leader_election.cc:290] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:01:34.966475  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 304532us
I20260709 13:01:34.938320  7111 raft_consensus.cc:3060] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:01:35.008874  1643 leader_election.cc:304] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:34.945842  7348 raft_consensus.cc:2749] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 13:01:35.016101  1753 raft_consensus.cc:1240] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 14. Current term is 15. Ops: []
I20260709 13:01:34.897756  7288 consensus_queue.cc:1059] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 13.11, Next index: 12, Last known committed idx: 10, Time since last communication: 0.516s
W20260709 13:01:35.020016  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 417110us
I20260709 13:01:34.953475  7360 raft_consensus.cc:493] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
W20260709 13:01:35.025810  1718 leader_election.cc:336] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 election: RPC error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Timed out: RequestConsensusVote RPC to 127.0.107.194:46261 timed out after 1.892s (SENT)
I20260709 13:01:34.965978  7146 raft_consensus.cc:2749] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:01:34.964290  6905 raft_consensus.cc:2749] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 13:01:35.032373  7188 raft_consensus.cc:2749] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
W20260709 13:01:35.034242  7356 log.cc:927] Time spent T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.089s	user 0.000s	sys 0.001s
I20260709 13:01:35.034570  7363 raft_consensus.cc:2749] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
W20260709 13:01:35.035606  7349 log.cc:927] Time spent T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.230s	user 0.001s	sys 0.000s
I20260709 13:01:35.035842  7363 raft_consensus.cc:493] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:35.036418  7363 raft_consensus.cc:515] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:35.036896  7135 leader_election.cc:290] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 15 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:35.034770  7364 raft_consensus.cc:2749] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Leader pre-election lost for term 11. Reason: Vote denied by peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445) with higher term. Message: Invalid argument: T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 11. Current term is 12.
I20260709 13:01:35.031407  7350 consensus_queue.cc:1059] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.007s
I20260709 13:01:35.040936  7363 leader_election.cc:290] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:35.042732  7290 leader_election.cc:290] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:35.044061  7229 leader_election.cc:290] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:35.044624  7229 consensus_queue.cc:1059] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 10.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 13:01:35.045962  6905 raft_consensus.cc:3055] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:01:35.046339  6905 raft_consensus.cc:740] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:01:34.928918  7305 consensus_queue.cc:237] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.11, Last appended by leader: 11, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:35.047284  6905 consensus_queue.cc:260] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.7, Last appended by leader: 7, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:34.937387  7352 raft_consensus.cc:515] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:35.051460  7297 raft_consensus.cc:3060] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:01:35.036830  7348 raft_consensus.cc:2749] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:01:35.055168  7360 raft_consensus.cc:515] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:35.057695  7360 leader_election.cc:290] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:35.050396  7352 leader_election.cc:290] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:01:35.066638  7301 consensus_meta.cc:287] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.868s	user 0.015s	sys 0.003s
W20260709 13:01:35.067642  6882 consensus_meta.cc:287] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.865s	user 0.016s	sys 0.004s
I20260709 13:01:35.067397  7301 raft_consensus.cc:515] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:35.068279  6882 raft_consensus.cc:515] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:35.070103  7301 leader_election.cc:290] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:01:35.070895  7338 consensus_meta.cc:287] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.829s	user 0.019s	sys 0.000s
I20260709 13:01:35.071650  7233 leader_election.cc:290] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:35.072064  7346 raft_consensus.cc:2804] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:01:35.079676  7346 raft_consensus.cc:697] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 14 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:01:35.079963  7337 leader_election.cc:290] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:35.080662  6882 leader_election.cc:290] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 17 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:35.080788  7346 consensus_queue.cc:237] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 13.7, Last appended by leader: 7, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:35.075762  7347 raft_consensus.cc:515] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:35.003422  7345 raft_consensus.cc:3060] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:01:35.092222  1718 leader_election.cc:304] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:34.963724  7314 raft_consensus.cc:515] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:35.096585  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:35.104879  1752 raft_consensus.cc:2468] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 11.
W20260709 13:01:35.111042  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 102248us
I20260709 13:01:35.113579  1643 leader_election.cc:304] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:35.116047  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:35.120108  1754 raft_consensus.cc:2393] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 15: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:01:35.090487  6882 raft_consensus.cc:493] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:35.095300  7280 raft_consensus.cc:493] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:35.141381  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:35.142371  1673 raft_consensus.cc:2393] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 11: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:01:35.091156  7338 raft_consensus.cc:515] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:35.146639  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:35.150763  1602 raft_consensus.cc:1275] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 13 index: 9. Preceding OpId from leader: term: 15 index: 10. (index mismatch)
I20260709 13:01:35.147131  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:35.121060  7301 raft_consensus.cc:493] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:35.149130  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:35.151913  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:35.152107  1751 raft_consensus.cc:1275] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 11 index: 11. Preceding OpId from leader: term: 13 index: 12. (index mismatch)
I20260709 13:01:35.149042  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:35.127046  7135 raft_consensus.cc:2749] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 13:01:35.159190  1678 raft_consensus.cc:2376] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 15. Re-sending same reply.
I20260709 13:01:35.152954  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:35.153198  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:35.119692  7350 consensus_queue.cc:1243] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer 3b879799899a499c8d409ee9f1277cd1 log is divergent from this leader: its last log entry 12.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 9
I20260709 13:01:35.123559  7126 leader_election.cc:290] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:35.152997  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:35.135572  7146 raft_consensus.cc:2749] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:01:35.167505  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:35.167538  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:35.107548  7111 raft_consensus.cc:515] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:35.168874  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:35.186017  1598 raft_consensus.cc:3060] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:01:35.123044  7219 raft_consensus.cc:697] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
W20260709 13:01:35.179862  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 154072us
I20260709 13:01:35.182926  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:35.183573  1678 raft_consensus.cc:2393] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 14: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:01:35.184204  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:35.191744  1679 raft_consensus.cc:1275] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 16 index: 9. (index mismatch)
I20260709 13:01:35.149264  7311 leader_election.cc:290] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:35.157219  7298 raft_consensus.cc:493] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:35.198212  1677 raft_consensus.cc:2368] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 11. Current term is 12.
I20260709 13:01:35.214557  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
W20260709 13:01:35.191814  7244 consensus_meta.cc:287] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.743s	user 0.013s	sys 0.000s
I20260709 13:01:35.138525  6882 raft_consensus.cc:515] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:35.222597  1601 raft_consensus.cc:2393] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 15: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:01:35.169627  7366 raft_consensus.cc:493] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:35.143877  7280 raft_consensus.cc:515] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:35.231550  1677 raft_consensus.cc:1275] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
W20260709 13:01:35.237795  7264 consensus_meta.cc:287] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.503s	user 0.008s	sys 0.003s
I20260709 13:01:35.238060  7386 raft_consensus.cc:493] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:35.238363  7264 raft_consensus.cc:515] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:35.238798  7386 raft_consensus.cc:515] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:35.240820  7264 leader_election.cc:290] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 14 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:35.166318  7301 raft_consensus.cc:515] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:35.242082  7314 leader_election.cc:290] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:35.242654  1600 raft_consensus.cc:1275] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
I20260709 13:01:35.243818  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:35.245371  1678 raft_consensus.cc:1240] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 16. Current term is 17. Ops: []
I20260709 13:01:35.242687  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:35.247531  7219 consensus_queue.cc:237] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } }
I20260709 13:01:35.249639  7388 raft_consensus.cc:493] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:35.250286  7388 raft_consensus.cc:515] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:35.253337  7338 leader_election.cc:290] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 16 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:35.254503  7143 leader_election.cc:290] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:35.256889  6882 leader_election.cc:290] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:35.233778  7366 raft_consensus.cc:515] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:35.260528  1718 leader_election.cc:304] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:35.263696  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:35.264840  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:35.264905  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:35.265877  1674 raft_consensus.cc:2468] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 13.
I20260709 13:01:35.216723  7126 raft_consensus.cc:493] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:35.275116  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:35.276207  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:35.268226  1679 raft_consensus.cc:1275] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 14 index: 7. Preceding OpId from leader: term: 15 index: 8. (index mismatch)
I20260709 13:01:35.269761  1677 raft_consensus.cc:1275] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 13 index: 9. Preceding OpId from leader: term: 15 index: 10. (index mismatch)
I20260709 13:01:35.279721  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:01:35.278383  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 250705us
W20260709 13:01:35.231607  6997 consensus_meta.cc:287] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 1.032s	user 0.015s	sys 0.001s
I20260709 13:01:35.282980  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:35.283602  1646 leader_election.cc:304] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
W20260709 13:01:35.271046  7315 consensus_meta.cc:287] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.995s	user 0.008s	sys 0.008s
I20260709 13:01:35.291507  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:35.288077  1598 raft_consensus.cc:2468] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 11.
I20260709 13:01:35.292960  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:35.289868  1646 leader_election.cc:304] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:35.296406  1678 raft_consensus.cc:2468] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 11.
I20260709 13:01:35.288025  1718 leader_election.cc:304] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:35.217116  7298 raft_consensus.cc:515] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:35.299058  1718 leader_election.cc:400] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 pre-election: Vote denied by peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261) with higher term. Message: Invalid argument: T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 11. Current term is 12.
I20260709 13:01:35.268872  7288 raft_consensus.cc:493] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:35.226986  7244 raft_consensus.cc:515] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:35.311146  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:35.317051  7362 consensus_queue.cc:1048] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 13:01:35.293722  6882 raft_consensus.cc:2804] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:01:35.318722  1718 leader_election.cc:304] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:35.382736  6882 raft_consensus.cc:493] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:35.383404  6882 raft_consensus.cc:3060] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Advancing to term 13
W20260709 13:01:35.383548  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 64910.9us
I20260709 13:01:35.385350  7388 leader_election.cc:290] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:35.386278  1718 leader_election.cc:304] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:35.387135  7325 consensus_queue.cc:1048] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 13:01:35.388450  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:35.388801  7325 consensus_queue.cc:1048] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 13:01:35.389343  1678 raft_consensus.cc:2393] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 14: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:01:35.390650  7227 raft_consensus.cc:493] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:35.300691  7143 raft_consensus.cc:2804] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:01:35.391271  7143 raft_consensus.cc:493] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:35.391710  7143 raft_consensus.cc:3060] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:01:35.391775  7289 raft_consensus.cc:493] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:35.391302  7227 raft_consensus.cc:515] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:35.392464  7289 raft_consensus.cc:515] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:35.392927  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:35.290958  6997 raft_consensus.cc:515] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:35.394737  7227 leader_election.cc:290] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:35.395277  7247 raft_consensus.cc:3055] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:01:35.395659  7247 raft_consensus.cc:740] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:01:35.396674  6997 leader_election.cc:290] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 17 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:35.396618  7247 consensus_queue.cc:260] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.12, Last appended by leader: 12, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:35.343796  7319 raft_consensus.cc:493] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:35.401291  7386 leader_election.cc:290] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:35.287076  7383 raft_consensus.cc:3060] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:01:35.372745  7288 raft_consensus.cc:515] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:35.404634  7288 leader_election.cc:290] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:35.318231  7376 consensus_queue.cc:1059] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 13:01:35.407778  7376 raft_consensus.cc:3055] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 13:01:35.408142  7376 raft_consensus.cc:740] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:01:35.409286  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:35.411301  1604 raft_consensus.cc:3055] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 LEADER]: Stepping down as leader of term 11
I20260709 13:01:35.411664  1604 raft_consensus.cc:740] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:01:35.412168  7366 leader_election.cc:290] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:35.412695  7301 leader_election.cc:290] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:35.412470  1604 consensus_queue.cc:260] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.10, Last appended by leader: 10, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:35.303282  7377 raft_consensus.cc:2749] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:01:35.386509  7362 raft_consensus.cc:2804] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:01:35.415222  7362 raft_consensus.cc:493] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:35.415745  7362 raft_consensus.cc:3060] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:01:35.394510  7370 raft_consensus.cc:3055] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:01:35.418908  7370 raft_consensus.cc:740] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:01:35.420063  7370 consensus_queue.cc:260] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:35.323758  7308 raft_consensus.cc:3055] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 13:01:35.421931  7308 raft_consensus.cc:740] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:01:35.423269  7308 consensus_queue.cc:260] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.11, Last appended by leader: 11, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:35.425251  1717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:35.400019  1672 raft_consensus.cc:2368] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 16. Current term is 17.
I20260709 13:01:35.401126  7319 raft_consensus.cc:515] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:35.433202  1753 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55848 (request call id 2795) took 1801 ms. Trace:
I20260709 13:01:35.433702  1753 rpcz_store.cc:276] 0709 13:01:33.631512 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:33.972916 (+341404us) service_pool.cc:224] Handling call
0709 13:01:33.973305 (+   389us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:35.019058 (+1045753us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:35.432918 (+413860us) spinlock_profiling.cc:243] Waited 1.04 s on lock 0x7b54001d47d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:01:35.433125 (+   207us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2397362560}
I20260709 13:01:35.434930  7319 leader_election.cc:290] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:35.437008  1747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55848 (request call id 2792) took 1806 ms. Trace:
I20260709 13:01:35.409086  7376 consensus_queue.cc:260] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.11, Last appended by leader: 11, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:35.437425  1747 rpcz_store.cc:276] 0709 13:01:33.630577 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:33.775895 (+145318us) service_pool.cc:224] Handling call
0709 13:01:33.922653 (+146758us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:34.978529 (+1055876us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:35.430321 (+451792us) spinlock_profiling.cc:243] Waited 1.05 s on lock 0x7b54001d60d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:01:35.436931 (+  6610us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2409622784}
I20260709 13:01:35.292433  7311 consensus_queue.cc:1048] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.088s
I20260709 13:01:35.287864  7126 raft_consensus.cc:515] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:35.363824  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 64271.5us
I20260709 13:01:35.398942  1718 leader_election.cc:304] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:35.390270  7325 raft_consensus.cc:2804] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:01:35.350390  7280 leader_election.cc:290] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:01:35.464677  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 65760.3us
I20260709 13:01:35.465041  6160 raft_consensus.cc:3060] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:01:35.493019  7307 raft_consensus.cc:493] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:35.493728  7307 raft_consensus.cc:515] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:35.494709  7233 raft_consensus.cc:493] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:35.495385  7233 raft_consensus.cc:515] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:35.496331  7307 leader_election.cc:290] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:35.497308  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:35.498728  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:35.499987  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:35.500793  1677 raft_consensus.cc:2393] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 16: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:01:35.501696  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:35.502499  1679 raft_consensus.cc:2468] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 15.
I20260709 13:01:35.503670  1718 leader_election.cc:304] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:35.467757  7247 raft_consensus.cc:3060] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Advancing to term 15
W20260709 13:01:35.467856  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:35.468216  7325 raft_consensus.cc:493] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:35.456844  7319 raft_consensus.cc:493] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:35.470805  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:35.562462  2330 raft_consensus.cc:2468] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 15.
I20260709 13:01:35.563920  7319 raft_consensus.cc:515] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:35.571834  1604 raft_consensus.cc:3060] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:01:35.476775  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:35.573089  1602 raft_consensus.cc:2468] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 15.
I20260709 13:01:35.477689  7376 raft_consensus.cc:3060] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:01:35.480993  7036 raft_consensus.cc:493] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:35.482002  6985 raft_consensus.cc:493] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:35.394894  7289 leader_election.cc:290] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:35.482717  7308 raft_consensus.cc:3060] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:01:35.456200  7377 raft_consensus.cc:493] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:35.589708  7377 raft_consensus.cc:515] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:35.498098  7233 leader_election.cc:290] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:35.580780  6876 raft_consensus.cc:493] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:35.593029  6876 raft_consensus.cc:515] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:35.595396  7377 leader_election.cc:290] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:35.595808  6876 leader_election.cc:290] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:35.532068  2328 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:35790 (request call id 21) took 1160 ms. Trace:
I20260709 13:01:35.597951  2328 rpcz_store.cc:276] 0709 13:01:34.382012 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:34.382165 (+   153us) service_pool.cc:224] Handling call
0709 13:01:35.531709 (+1149544us) spinlock_profiling.cc:243] Waited 67.5 ms on lock 0x7b5400504ea0. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42660dd5e7 00007f426615cc68 00007f426615c84a 00007f426b12a60a 00007f42646b07d4 00007f42646b0762 00007f42646b0691 00007f42646b060c 00007f42646af8c2 00007f426449d48c
0709 13:01:35.531949 (+   240us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":155208448}
I20260709 13:01:35.534929  7325 raft_consensus.cc:3060] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:01:35.538717  6160 raft_consensus.cc:2749] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:01:35.539224  6905 raft_consensus.cc:3060] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:01:35.541162  7126 leader_election.cc:290] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:35.543421  7164 raft_consensus.cc:3055] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 LEADER]: Stepping down as leader of term 11
I20260709 13:01:35.615161  7164 raft_consensus.cc:740] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:01:35.616231  7164 consensus_queue.cc:260] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:35.617853  7164 raft_consensus.cc:3060] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:01:35.545181  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:35.619056  1679 raft_consensus.cc:2393] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 14: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:01:35.552803  7347 leader_election.cc:290] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 15 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:35.555339  7307 raft_consensus.cc:2804] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:01:35.560168  7298 leader_election.cc:290] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:35.561375  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:35.579686  7319 leader_election.cc:290] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 13:01:35.506981  7067 consensus_meta.cc:287] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.543s	user 0.007s	sys 0.006s
I20260709 13:01:35.651542  7067 raft_consensus.cc:515] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:35.652802  2335 raft_consensus.cc:2468] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 16.
I20260709 13:01:35.653728  7067 leader_election.cc:290] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:35.584388  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:35.584848  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:35.655121  1598 raft_consensus.cc:1240] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 10. Current term is 11. Ops: []
I20260709 13:01:35.586382  7036 raft_consensus.cc:515] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:35.597062  6985 raft_consensus.cc:515] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:35.604418  2298 leader_election.cc:304] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
W20260709 13:01:35.660377  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 56108.6us
I20260709 13:01:35.621166  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:01:35.514765  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 61874.8us
I20260709 13:01:35.662490  2331 raft_consensus.cc:1240] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 13. Current term is 14. Ops: []
I20260709 13:01:35.585654  1646 leader_election.cc:304] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
W20260709 13:01:35.631083  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 437746us
I20260709 13:01:35.646162  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:35.575785  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:35.650732  7111 leader_election.cc:290] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 15 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:35.661516  2328 raft_consensus.cc:2468] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 14.
I20260709 13:01:35.669086  1604 raft_consensus.cc:1275] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 12 index: 10. (term mismatch)
I20260709 13:01:35.669648  1604 pending_rounds.cc:85] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a: Aborting all ops after (but not including) 9
I20260709 13:01:35.669930  1604 pending_rounds.cc:107] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a: Aborting uncommitted NO_OP operation due to leader change: 11.10
I20260709 13:01:35.670243  1604 raft_consensus.cc:2889] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:01:35.665998  1571 leader_election.cc:304] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:01:35.673080  1571 leader_election.cc:304] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:01:35.674849  2335 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39726 (request call id 1302) took 1232 ms. Trace:
I20260709 13:01:35.675570  2335 rpcz_store.cc:276] 0709 13:01:34.441947 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:34.442073 (+   126us) service_pool.cc:224] Handling call
0709 13:01:35.674608 (+1232535us) spinlock_profiling.cc:243] Waited 851 ms on lock 0x7b5400567f54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:01:35.674757 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1956472704}
I20260709 13:01:35.647022  7307 raft_consensus.cc:493] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:35.674777  7141 consensus_queue.cc:1048] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 13:01:35.683713  6160 raft_consensus.cc:493] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:01:35.700357  6160 raft_consensus.cc:515] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:35.648118  7244 leader_election.cc:290] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 17 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:35.703531  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:35.676079  7180 raft_consensus.cc:993] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1: : Instructing follower 83595ad0fd5940c0a2bfc7c61acd969a to start an election
I20260709 13:01:35.703800  7352 raft_consensus.cc:3060] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Advancing to term 15
W20260709 13:01:35.689522  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 103963us
W20260709 13:01:35.712106  1569 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.516s	user 0.019s	sys 0.059s
I20260709 13:01:35.712177  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 13 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:01:35.712694  1569 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.517s	user 0.019s	sys 0.059s
I20260709 13:01:35.713268  2335 raft_consensus.cc:2410] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for term 13 because replica has last-logged OpId of term: 12 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 9.
I20260709 13:01:35.673316  7319 raft_consensus.cc:493] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:35.724215  1717 leader_election.cc:304] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:35.726143  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:35.743521  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:35.744230  1748 raft_consensus.cc:3060] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:01:35.730160  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:35.731678  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:35.734474  1745 raft_consensus.cc:1275] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 10 index: 8. Preceding OpId from leader: term: 12 index: 9. (index mismatch)
I20260709 13:01:35.736619  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:35.747723  1752 raft_consensus.cc:2393] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 11: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:01:35.685946  7307 raft_consensus.cc:3060] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:01:35.676371  6876 raft_consensus.cc:2804] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:01:35.681465  7067 raft_consensus.cc:2749] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:01:35.682793  7126 raft_consensus.cc:493] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:35.738067  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 16 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:35.749867  1717 leader_election.cc:304] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:35.751857  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:35.707500  7287 raft_consensus.cc:493] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:35.752382  7135 raft_consensus.cc:2749] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:01:35.714607  6643 raft_consensus.cc:1081] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 12 LEADER]: Signalling peer 83595ad0fd5940c0a2bfc7c61acd969a to start an election
I20260709 13:01:35.760512  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 13 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:35.700845  7233 raft_consensus.cc:493] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:35.758047  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:35.764999  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:35.768532  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:35.755110  1750 raft_consensus.cc:2393] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 15: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:01:35.747186  7298 raft_consensus.cc:2749] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:01:35.774227  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:35.779613  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:35.782074  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:35.782509  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:35.782995  2328 raft_consensus.cc:2468] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 15.
I20260709 13:01:35.784174  6160 leader_election.cc:290] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:01:35.774770  1571 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:35.777812  1679 raft_consensus.cc:2468] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 16.
I20260709 13:01:35.815464  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:35.791239  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 16 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:35.820261  1603 raft_consensus.cc:1275] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 16 index: 11. (index mismatch)
I20260709 13:01:35.820928  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:35.777712  1752 raft_consensus.cc:2468] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 15.
I20260709 13:01:35.757731  7311 consensus_queue.cc:1048] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.045s
I20260709 13:01:35.740168  7342 consensus_queue.cc:1059] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 10.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.062s
W20260709 13:01:35.771766  7374 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 12 current term: 12 is higher.
I20260709 13:01:35.757622  6876 raft_consensus.cc:493] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:35.769490  7180 raft_consensus.cc:493] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:35.752424  7319 raft_consensus.cc:515] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:35.778775  7364 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.519s	user 0.004s	sys 0.010s
I20260709 13:01:35.824111  1569 leader_election.cc:304] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:01:35.791585  7067 raft_consensus.cc:493] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:35.801271  1748 raft_consensus.cc:2468] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 15.
I20260709 13:01:35.757912  7326 consensus_queue.cc:1048] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.048s
I20260709 13:01:35.752804  7111 consensus_queue.cc:1048] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.039s
I20260709 13:01:35.827312  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
W20260709 13:01:35.900816  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 66891.7us
I20260709 13:01:35.901068  1597 raft_consensus.cc:3060] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:01:35.828032  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:35.775250  7390 consensus_queue.cc:1059] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 14.10, Next index: 11, Last known committed idx: 8, Time since last communication: 0.066s
I20260709 13:01:35.828274  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
W20260709 13:01:35.837647  7345 consensus_meta.cc:287] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.722s	user 0.015s	sys 0.000s
I20260709 13:01:35.770103  7338 consensus_queue.cc:1059] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 10.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.041s
I20260709 13:01:35.831179  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:01:35.878901  7364 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.619s	user 0.004s	sys 0.010s
I20260709 13:01:35.907768  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:35.791832  7347 raft_consensus.cc:493] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:01:35.833918  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:35.834791  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:35.837507  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:35.841275  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:35.772898  7126 raft_consensus.cc:515] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:35.883496  1599 raft_consensus.cc:1275] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 14 index: 7. Preceding OpId from leader: term: 15 index: 8. (index mismatch)
I20260709 13:01:35.802781  7233 raft_consensus.cc:515] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:35.902240  1600 raft_consensus.cc:3060] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:01:35.828632  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:35.830381  7298 raft_consensus.cc:493] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:01:35.903982  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 80005.7us
I20260709 13:01:35.929167  1569 leader_election.cc:304] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:01:35.930840  1569 leader_election.cc:304] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:01:35.776173  7287 raft_consensus.cc:515] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:35.831625  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:35.887645  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:35.910097  1747 raft_consensus.cc:3060] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:01:35.910667  1752 raft_consensus.cc:3060] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:01:35.911690  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:35.934888  1599 raft_consensus.cc:2468] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 15.
I20260709 13:01:35.881217  6876 raft_consensus.cc:3060] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:01:35.891628  6985 leader_election.cc:290] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:35.924033  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:35.940639  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:35.932832  1603 raft_consensus.cc:2468] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 12.
I20260709 13:01:35.888092  7377 raft_consensus.cc:493] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:35.960809  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:35.889424  7244 raft_consensus.cc:2749] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:01:35.911698  7219 raft_consensus.cc:2804] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:01:35.886291  7135 raft_consensus.cc:493] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:35.960098  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:35.960387  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:35.962145  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:35.950469  1569 leader_election.cc:304] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
W20260709 13:01:35.959614  7305 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.612s	user 0.054s	sys 0.088s
I20260709 13:01:35.970834  7380 raft_consensus.cc:493] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:35.971840  6160 raft_consensus.cc:493] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:35.973532  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:35.977423  1679 raft_consensus.cc:2368] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 15. Current term is 16.
I20260709 13:01:35.972549  7377 raft_consensus.cc:515] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } }
I20260709 13:01:36.034646  2331 raft_consensus.cc:3060] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Advancing to term 17
W20260709 13:01:36.041253  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 90925.9us
I20260709 13:01:36.034742  6160 raft_consensus.cc:515] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:35.982323  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:35.956805  7298 raft_consensus.cc:515] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:35.962988  7289 raft_consensus.cc:2749] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:01:35.962623  7288 raft_consensus.cc:493] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:35.995703  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:35.903910  7067 raft_consensus.cc:515] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:01:36.063975  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:36.065112  1598 raft_consensus.cc:2393] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 12: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:01:35.890929  7180 raft_consensus.cc:515] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:36.010850  1748 raft_consensus.cc:1240] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 14. Current term is 15. Ops: []
I20260709 13:01:35.955825  7392 raft_consensus.cc:2749] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 13:01:35.967936  7036 leader_election.cc:290] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:35.909447  7345 raft_consensus.cc:515] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:36.020834  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:35.978070  7219 raft_consensus.cc:493] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:35.924011  7347 raft_consensus.cc:515] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:36.025547  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:01:36.027665  1678 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.630s	user 0.018s	sys 0.014s
W20260709 13:01:36.136179  1678 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.739s	user 0.019s	sys 0.015s
I20260709 13:01:35.983194  7126 leader_election.cc:290] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:36.034941  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:36.137076  1599 raft_consensus.cc:2468] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 15.
I20260709 13:01:36.044579  1603 raft_consensus.cc:2376] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 11. Re-sending same reply.
I20260709 13:01:36.053381  1747 raft_consensus.cc:2468] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 16.
W20260709 13:01:36.033869  7305 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.686s	user 0.054s	sys 0.089s
I20260709 13:01:36.056488  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:36.000178  7135 raft_consensus.cc:515] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:36.113741  6160 leader_election.cc:290] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:01:36.114935  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 297668us
I20260709 13:01:36.033869  7398 raft_consensus.cc:493] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:36.153309  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:36.124369  7233 leader_election.cc:290] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:36.125452  7289 raft_consensus.cc:493] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:01:36.126816  7394 log.cc:927] Time spent T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.104s	user 0.001s	sys 0.000s
I20260709 13:01:36.127506  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:36.127983  7338 raft_consensus.cc:3055] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:01:36.129812  6976 raft_consensus.cc:493] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:36.130426  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:36.046923  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:36.178997  7135 leader_election.cc:290] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:36.180243  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:36.178925  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:36.182233  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 16 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:36.183406  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:36.184016  1674 raft_consensus.cc:2468] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 15.
I20260709 13:01:36.184877  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:01:36.177361  7164 consensus_meta.cc:287] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.559s	user 0.012s	sys 0.000s
I20260709 13:01:36.187288  1677 raft_consensus.cc:2468] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 12.
I20260709 13:01:36.188124  7164 raft_consensus.cc:493] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:36.188771  7164 raft_consensus.cc:515] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:36.190888  7347 leader_election.cc:290] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:36.197126  1680 raft_consensus.cc:1240] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 10. Current term is 11. Ops: []
I20260709 13:01:36.201306  1680 raft_consensus.cc:1240] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 11. Current term is 12. Ops: []
I20260709 13:01:36.048298  7111 raft_consensus.cc:2804] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:01:36.151707  1752 raft_consensus.cc:2468] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 17.
I20260709 13:01:36.208282  1680 raft_consensus.cc:1275] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 12 index: 10. (index mismatch)
I20260709 13:01:36.149360  7229 raft_consensus.cc:493] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:36.221321  1571 leader_election.cc:304] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:01:36.162758  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:36.139333  7219 raft_consensus.cc:3060] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:01:36.168134  1600 raft_consensus.cc:2468] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 11.
I20260709 13:01:36.168383  1598 raft_consensus.cc:2468] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 15.
I20260709 13:01:36.177958  6976 raft_consensus.cc:515] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:36.037295  7380 raft_consensus.cc:515] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:36.170258  7289 raft_consensus.cc:515] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:36.173623  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:36.249794  1596 raft_consensus.cc:3060] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:01:36.249893  1571 leader_election.cc:304] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:36.175850  7319 leader_election.cc:290] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:36.169625  7281 raft_consensus.cc:493] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
W20260709 13:01:36.177256  7396 log.cc:927] Time spent T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.148s	user 0.000s	sys 0.001s
I20260709 13:01:36.207219  7111 raft_consensus.cc:493] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
W20260709 13:01:36.203106  7362 consensus_meta.cc:287] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.787s	user 0.014s	sys 0.004s
W20260709 13:01:36.217257  6882 consensus_meta.cc:287] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.833s	user 0.015s	sys 0.002s
I20260709 13:01:36.217478  7345 leader_election.cc:290] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 12 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:36.219122  7377 leader_election.cc:290] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:36.096673  7288 raft_consensus.cc:515] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:36.170670  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:36.226400  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:36.264562  1598 raft_consensus.cc:2468] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 15.
I20260709 13:01:36.267163  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:36.268064  1750 raft_consensus.cc:3060] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:01:36.271323  1599 raft_consensus.cc:3060] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:01:36.263777  1603 raft_consensus.cc:2468] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 14.
I20260709 13:01:36.254345  7392 raft_consensus.cc:2804] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader election won for term 15
I20260709 13:01:36.152398  2331 raft_consensus.cc:2468] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 17.
I20260709 13:01:36.261855  7111 raft_consensus.cc:3060] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:01:36.240058  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 16 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:36.243719  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:36.244896  1601 raft_consensus.cc:2393] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 15: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
W20260709 13:01:36.224969  7383 consensus_meta.cc:287] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.823s	user 0.006s	sys 0.009s
W20260709 13:01:36.245716  7325 consensus_meta.cc:287] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.646s	user 0.014s	sys 0.001s
I20260709 13:01:36.245862  7380 leader_election.cc:290] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:36.162513  7398 raft_consensus.cc:515] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:36.239092  7305 consensus_queue.cc:1048] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.008s
I20260709 13:01:36.170048  7338 raft_consensus.cc:740] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
W20260709 13:01:36.159299  7247 consensus_meta.cc:287] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.626s	user 0.013s	sys 0.003s
I20260709 13:01:36.250793  1597 raft_consensus.cc:2468] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 15.
I20260709 13:01:36.314342  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:36.315212  1597 raft_consensus.cc:3060] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:01:36.246628  7164 leader_election.cc:290] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:36.157863  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:36.258602  1679 raft_consensus.cc:2468] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 9.
I20260709 13:01:36.227981  7374 raft_consensus.cc:2749] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:01:36.291209  1600 raft_consensus.cc:2468] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 15.
W20260709 13:01:36.296548  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 394531us
I20260709 13:01:36.226375  7229 raft_consensus.cc:515] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:36.302120  7287 leader_election.cc:290] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:36.308547  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:36.295531  7392 raft_consensus.cc:697] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:01:36.259881  7281 raft_consensus.cc:515] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:36.270807  6882 raft_consensus.cc:515] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:36.329602  2301 leader_election.cc:304] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:01:36.301040  7383 raft_consensus.cc:2749] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:01:36.332145  7370 raft_consensus.cc:3060] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Advancing to term 11
W20260709 13:01:36.318064  6905 consensus_meta.cc:287] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.706s	user 0.014s	sys 0.000s
I20260709 13:01:36.343899  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:36.353754  2329 raft_consensus.cc:1275] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 11 index: 11. Preceding OpId from leader: term: 13 index: 12. (index mismatch)
I20260709 13:01:36.355561  1600 raft_consensus.cc:1240] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 10. Current term is 11. Ops: [10.7-10.7]
I20260709 13:01:36.356159  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:36.357244  1600 raft_consensus.cc:1240] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 10. Current term is 11. Ops: []
I20260709 13:01:36.363406  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:36.376089  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:36.376161  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:36.384644  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:36.386032  1598 raft_consensus.cc:1240] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 14. Current term is 15. Ops: [10.8-14.10]
I20260709 13:01:36.386237  1674 raft_consensus.cc:1240] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 12. Current term is 13. Ops: []
I20260709 13:01:36.388231  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:36.387862  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:36.394071  1599 raft_consensus.cc:2468] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 17.
I20260709 13:01:36.263988  7362 raft_consensus.cc:515] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:36.300561  7325 raft_consensus.cc:515] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:36.345939  1597 raft_consensus.cc:2468] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 17.
I20260709 13:01:36.407616  1596 raft_consensus.cc:2468] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 16.
I20260709 13:01:36.404784  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:36.381027  6876 raft_consensus.cc:515] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:36.413872  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:36.355984  7287 raft_consensus.cc:2749] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 13:01:36.415397  1673 raft_consensus.cc:2468] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 11.
I20260709 13:01:36.364311  7298 leader_election.cc:290] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:36.422861  2328 raft_consensus.cc:2393] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 16: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:01:36.397962  7067 leader_election.cc:290] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:01:36.415337  7396 log.cc:927] Time spent T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.061s	user 0.000s	sys 0.000s
I20260709 13:01:36.430063  1718 leader_election.cc:304] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:36.444016  1754 raft_consensus.cc:1240] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 13. Current term is 14. Ops: []
I20260709 13:01:36.449306  1718 leader_election.cc:304] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:36.452266  1751 raft_consensus.cc:2393] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 12: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:01:36.454094  7364 raft_consensus.cc:2749] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:01:36.456193  7362 leader_election.cc:290] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 14 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:36.457808  6976 leader_election.cc:290] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:01:36.467196  7408 log.cc:927] Time spent T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.117s	user 0.000s	sys 0.001s
I20260709 13:01:36.472520  7281 leader_election.cc:290] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:36.476576  7229 leader_election.cc:290] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:36.478929  6882 leader_election.cc:290] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 13 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:36.482883  7289 leader_election.cc:290] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:36.401310  7338 consensus_queue.cc:260] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.7, Last appended by leader: 7, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:36.491734  7338 raft_consensus.cc:3060] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:01:36.492060  7325 leader_election.cc:290] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 12 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:36.467337  7288 leader_election.cc:290] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:36.425096  7180 leader_election.cc:290] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 13:01:36.513892  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 64675.6us
I20260709 13:01:36.522001  1718 leader_election.cc:304] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
W20260709 13:01:36.529127  1646 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:36.530893  1646 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:36.545895  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:36.539340  1646 leader_election.cc:304] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:36.531014  1718 leader_election.cc:304] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
W20260709 13:01:36.553781  1569 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.503s	user 0.005s	sys 0.020s
W20260709 13:01:36.554296  1569 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.503s	user 0.005s	sys 0.020s
W20260709 13:01:36.535328  7352 consensus_meta.cc:287] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.794s	user 0.006s	sys 0.009s
I20260709 13:01:36.554697  1718 leader_election.cc:304] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:36.515377  7362 raft_consensus.cc:2749] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260709 13:01:36.522944  7307 consensus_meta.cc:287] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.752s	user 0.017s	sys 0.000s
I20260709 13:01:36.557724  1646 leader_election.cc:304] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:36.534878  7325 raft_consensus.cc:2749] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
W20260709 13:01:36.565575  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 411372us
I20260709 13:01:36.584245  1644 leader_election.cc:304] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:01:36.559748  1569 leader_election.cc:400] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 15 election: Vote denied by peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261) with higher term. Message: Invalid argument: T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 15. Current term is 16.
I20260709 13:01:36.585186  1675 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52778 (request call id 2372) took 1445 ms. Trace:
I20260709 13:01:36.585623  1569 leader_election.cc:403] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 15 election: Cancelling election due to peer responding with higher term
I20260709 13:01:36.586655  7281 raft_consensus.cc:2749] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Leader election lost for term 15. Reason: Vote denied by peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261) with higher term. Message: Invalid argument: T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 15. Current term is 16.
I20260709 13:01:36.587119  2330 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49082 (request call id 1037) took 1004 ms. Trace:
I20260709 13:01:36.588404  1569 leader_election.cc:304] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
W20260709 13:01:36.618746  2298 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:36.629021  1569 leader_election.cc:304] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
W20260709 13:01:36.624505  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 70025.1us
I20260709 13:01:36.631769  1750 raft_consensus.cc:2468] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 12.
I20260709 13:01:36.515415  7392 consensus_queue.cc:237] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.7, Last appended by leader: 7, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:36.559321  7364 raft_consensus.cc:2804] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Leader pre-election won for term 15
W20260709 13:01:36.595881  7143 consensus_meta.cc:287] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 1.204s	user 0.018s	sys 0.000s
I20260709 13:01:36.630554  1569 leader_election.cc:304] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:01:36.583192  7352 raft_consensus.cc:2749] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:01:36.587620  2330 rpcz_store.cc:276] 0709 13:01:35.583055 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:35.583274 (+   219us) service_pool.cc:224] Handling call
0709 13:01:35.587916 (+  4642us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:35.588374 (+   458us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:35.588413 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:01:35.677872 (+ 89459us) spinlock_profiling.cc:243] Waited 47.8 ms on lock 0x7b2c00021c88. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4264a5ba11 00007f4264a6b372 00007f426619da01 00007f426616243a 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2
0709 13:01:35.678421 (+   549us) log.cc:844] Serialized 26 byte log entry
0709 13:01:36.146267 (+467846us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:01:36.582796 (+436529us) log.cc:844] Serialized 14 byte log entry
0709 13:01:36.586614 (+  3818us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:36.586719 (+   105us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:36.586765 (+    46us) raft_consensus.cc:1685] finished
0709 13:01:36.586890 (+   125us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:36.587047 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":467089,"raft.queue_time_us":3883,"raft.run_cpu_time_us":257,"raft.run_wall_time_us":277,"spinlock_wait_cycles":110003968,"thread_start_us":825122,"threads_started":2,"wal-append.queue_time_us":716723}
I20260709 13:01:36.660871  1718 leader_election.cc:304] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
W20260709 13:01:36.639468  7308 consensus_meta.cc:287] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 1.021s	user 0.017s	sys 0.004s
W20260709 13:01:36.646098  7408 log.cc:927] Time spent T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.063s	user 0.000s	sys 0.001s
I20260709 13:01:36.585855  1675 rpcz_store.cc:276] 0709 13:01:35.140141 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:35.140352 (+   211us) service_pool.cc:224] Handling call
0709 13:01:35.154138 (+ 13786us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:01:35.155890 (+  1752us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:35.155937 (+    47us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:01:35.166158 (+ 10221us) log.cc:844] Serialized 52 byte log entry
0709 13:01:35.252311 (+ 86153us) spinlock_profiling.cc:243] Waited 5.62 ms on lock 0x7b48002ac558. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426b08baa1 00007f426b082b37 00007f426b0829ca 00007f426b17a26f 00007f426b179f37 00007f426b18e55c 00007f426b18e508 00007f426b18e488 00007f426b18e443 00007f426b18d6ec 00007f426a0b7a96
0709 13:01:35.708018 (+455707us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:01:36.155983 (+447965us) log.cc:844] Serialized 14 byte log entry
0709 13:01:36.559319 (+403336us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:36.559423 (+   104us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:36.559474 (+    51us) raft_consensus.cc:1685] finished
0709 13:01:36.584858 (+ 25384us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:36.585094 (+   236us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1138,"prepare.queue_time_us":479314,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":52,"raft.queue_time_us":322800,"raft.run_cpu_time_us":2310,"raft.run_wall_time_us":89908,"spinlock_wait_cycles":12931456,"thread_start_us":1269207,"threads_started":3,"wal-append.queue_time_us":745626}
I20260709 13:01:36.612769  7316 raft_consensus.cc:2804] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:01:36.631313  7289 raft_consensus.cc:2749] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:01:36.607605  6876 leader_election.cc:290] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 16 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:36.681407  1718 leader_election.cc:304] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:36.580157  7247 consensus_queue.cc:1059] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 13.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.092s
I20260709 13:01:36.632095  7362 raft_consensus.cc:2804] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:01:36.634833  7281 raft_consensus.cc:2749] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:01:36.692830  2330 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36"
dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
mode: GRACEFUL
new_leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
 from {username='slave'} at 127.0.0.1:35790
I20260709 13:01:36.720778  2330 raft_consensus.cc:606] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Received request to transfer leadership to TS 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:36.721185  2330 raft_consensus.cc:614] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260709 13:01:36.662006  7376 consensus_meta.cc:287] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 1.076s	user 0.018s	sys 0.000s
I20260709 13:01:36.589547  7307 raft_consensus.cc:515] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:36.701105  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:36.702598  7316 raft_consensus.cc:493] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:36.714087  1569 leader_election.cc:304] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:36.644604  7364 raft_consensus.cc:493] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:36.716564  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:36.716964  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 14 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:36.744468  2335 raft_consensus.cc:2468] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 15.
I20260709 13:01:36.717180  2333 raft_consensus.cc:2468] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 14.
I20260709 13:01:36.714985  1718 leader_election.cc:304] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:36.719794  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:36.671253  7067 raft_consensus.cc:2749] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:01:36.678555  7143 raft_consensus.cc:515] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:36.724493  7416 log.cc:927] Time spent T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.064s	user 0.000s	sys 0.000s
W20260709 13:01:36.735345  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 151182us
I20260709 13:01:36.737486  2328 raft_consensus.cc:1240] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 16. Current term is 17. Ops: []
I20260709 13:01:36.738065  1749 raft_consensus.cc:2410] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for term 17 because replica has last-logged OpId of term: 16 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 10.
I20260709 13:01:36.740188  1569 leader_election.cc:304] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:01:36.746096  2332 raft_consensus.cc:2468] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 17.
I20260709 13:01:36.681077  1598 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31"
dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
 from {username='slave'} at 127.0.0.1:36462
I20260709 13:01:36.770848  1598 raft_consensus.cc:493] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:01:36.771314  1598 raft_consensus.cc:3060] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:01:36.720248  7325 raft_consensus.cc:2749] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:01:36.775707  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:36.750746  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:01:36.757699  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 200074us
W20260709 13:01:36.772109  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 57207.9us
I20260709 13:01:36.788676  1646 leader_election.cc:304] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:36.790601  1600 raft_consensus.cc:2468] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 17.
I20260709 13:01:36.706278  7135 raft_consensus.cc:3060] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:01:36.793037  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:36.793905  1644 leader_election.cc:304] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:01:36.797510  1745 raft_consensus.cc:2393] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 16: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:01:36.797819  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:36.744452  7316 raft_consensus.cc:3060] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Advancing to term 16
W20260709 13:01:36.800004  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 59944.7us
I20260709 13:01:36.801568  1680 raft_consensus.cc:1240] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 13. Current term is 14. Ops: [13.10-13.10]
I20260709 13:01:36.744252  7301 raft_consensus.cc:2749] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:01:36.748098  7364 raft_consensus.cc:3060] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:01:36.719956  7374 consensus_queue.cc:1048] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.072s
I20260709 13:01:36.809226  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:36.808295  1598 raft_consensus.cc:515] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:36.835538  1677 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52778 (request call id 2402) took 1260 ms. Trace:
I20260709 13:01:36.718902  7362 raft_consensus.cc:493] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:36.806674  1569 leader_election.cc:304] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:01:36.835974  1677 rpcz_store.cc:276] 0709 13:01:35.575337 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:35.575485 (+   148us) service_pool.cc:224] Handling call
0709 13:01:36.835328 (+1259843us) spinlock_profiling.cc:243] Waited 522 ms on lock 0x7b54000226d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:01:36.835468 (+   140us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1199839104}
I20260709 13:01:36.836493  7362 raft_consensus.cc:3060] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:01:36.837332  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:36.837976  1677 raft_consensus.cc:2468] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 16.
I20260709 13:01:36.820303  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:36.839771  1746 raft_consensus.cc:3060] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:01:36.840453  1598 leader_election.cc:290] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:36.814303  1674 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52766 (request call id 2821) took 1177 ms. Trace:
I20260709 13:01:36.790366  7376 raft_consensus.cc:2804] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 13:01:36.772007  6882 raft_consensus.cc:2804] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Leader election won for term 16
I20260709 13:01:36.748893  7413 raft_consensus.cc:493] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:36.807093  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:36.805881  1643 leader_election.cc:304] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:36.814579  7424 raft_consensus.cc:493] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:36.863274  7424 raft_consensus.cc:515] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } }
I20260709 13:01:36.814795  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:36.865010  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:36.865722  7424 leader_election.cc:290] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:36.866129  7376 raft_consensus.cc:697] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 17 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
W20260709 13:01:36.867020  7415 log.cc:927] Time spent T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.173s	user 0.000s	sys 0.000s
I20260709 13:01:36.747139  7219 raft_consensus.cc:515] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:36.834465  7316 raft_consensus.cc:515] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:36.866140  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:36.867882  1749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55848 (request call id 2809) took 1095 ms. Trace:
I20260709 13:01:36.868170  1672 raft_consensus.cc:2393] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 16: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:01:36.868274  1749 rpcz_store.cc:276] 0709 13:01:35.772855 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:35.773881 (+  1026us) service_pool.cc:224] Handling call
0709 13:01:36.867672 (+1093791us) spinlock_profiling.cc:243] Waited 955 ms on lock 0x7b54002f49d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:01:36.867805 (+   133us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2196315008}
I20260709 13:01:36.869156  7219 leader_election.cc:290] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:36.869570  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:36.869984  1604 raft_consensus.cc:2468] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 11.
I20260709 13:01:36.870754  6997 raft_consensus.cc:2804] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Leader election won for term 17
I20260709 13:01:36.867311  7376 consensus_queue.cc:237] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.11, Last appended by leader: 11, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:36.871289  6997 raft_consensus.cc:697] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:01:36.871661  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:36.872310  1680 raft_consensus.cc:2468] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 12.
I20260709 13:01:36.872224  6997 consensus_queue.cc:237] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.9, Last appended by leader: 9, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:36.873368  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:36.873967  1680 raft_consensus.cc:3060] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:01:36.875312  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:36.877779  2298 leader_election.cc:304] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:01:36.861955  7413 raft_consensus.cc:515] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:36.874234  7419 raft_consensus.cc:2749] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:01:36.831560  1602 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40154 (request call id 2533) took 1004 ms. Trace:
I20260709 13:01:36.852429  7370 raft_consensus.cc:2749] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:01:36.882113  1751 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55848 (request call id 2813) took 1067 ms. Trace:
W20260709 13:01:36.882351  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 76755.9us
I20260709 13:01:36.846900  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:36.821645  1679 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52766 (request call id 2830) took 1049 ms. Trace:
I20260709 13:01:36.883558  1643 leader_election.cc:304] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:01:36.843937  1674 rpcz_store.cc:276] 0709 13:01:35.637276 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:36.185826 (+548550us) service_pool.cc:224] Handling call
0709 13:01:36.186215 (+   389us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:36.399936 (+213721us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:36.814031 (+414095us) spinlock_profiling.cc:243] Waited 200 ms on lock 0x7b5400091354. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:01:36.814233 (+   202us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":459459840}
I20260709 13:01:36.854453  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:36.870227  1749 raft_consensus.cc:2468] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 15.
I20260709 13:01:36.884075  1679 rpcz_store.cc:276] 0709 13:01:35.771835 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:36.179929 (+408094us) service_pool.cc:224] Handling call
0709 13:01:36.821448 (+641519us) spinlock_profiling.cc:243] Waited 77.5 ms on lock 0x7b540025add4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:01:36.821575 (+   127us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":178334592}
I20260709 13:01:36.854406  2299 leader_election.cc:304] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:01:36.886525  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:36.849959  1569 leader_election.cc:304] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:01:36.858616  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:01:36.858995  7425 log.cc:927] Time spent T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.082s	user 0.001s	sys 0.000s
I20260709 13:01:36.858943  7181 raft_consensus.cc:3055] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:01:36.860608  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 14 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:36.861104  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:36.905073  1675 raft_consensus.cc:2468] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 14.
I20260709 13:01:36.862637  7336 raft_consensus.cc:493] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:36.863631  7307 leader_election.cc:290] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:36.796413  7421 raft_consensus.cc:2804] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Leader election won for term 16
I20260709 13:01:36.833200  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:36.869107  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:36.845700  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:36.742161  7398 leader_election.cc:290] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:36.887099  1679 raft_consensus.cc:3060] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:01:36.857067  2329 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2"
dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
mode: GRACEFUL
new_leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
 from {username='slave'} at 127.0.0.1:35790
I20260709 13:01:36.909655  1643 leader_election.cc:304] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:01:36.911321  1750 raft_consensus.cc:2468] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 15.
I20260709 13:01:36.852764  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:36.881404  1602 rpcz_store.cc:276] 0709 13:01:35.827574 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:35.827783 (+   209us) service_pool.cc:224] Handling call
0709 13:01:35.887105 (+ 59322us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:36.161552 (+274447us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 13:01:36.175682 (+ 14130us) log.cc:844] Serialized 14 byte log entry
0709 13:01:36.582124 (+406442us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:01:36.582207 (+    83us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:01:36.582282 (+    75us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:36.831279 (+248997us) spinlock_profiling.cc:243] Waited 274 ms on lock 0x7b540006d6d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:01:36.831332 (+    53us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:36.831487 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":14697,"prepare.run_cpu_time_us":80,"prepare.run_wall_time_us":82,"raft.queue_time_us":197,"raft.run_cpu_time_us":184,"raft.run_wall_time_us":6596,"spinlock_wait_cycles":630353152,"thread_start_us":405927,"threads_started":1,"wal-append.queue_time_us":541015}
I20260709 13:01:36.887903  1751 rpcz_store.cc:276] 0709 13:01:35.814330 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:35.820636 (+  6306us) service_pool.cc:224] Handling call
0709 13:01:36.881909 (+1061273us) spinlock_profiling.cc:243] Waited 630 ms on lock 0x7b54001d4cd4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:01:36.882045 (+   136us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1449400704}
I20260709 13:01:36.917075  1677 raft_consensus.cc:2468] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 16.
W20260709 13:01:36.901813  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 47502.3us
I20260709 13:01:36.918113  2299 leader_election.cc:304] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
W20260709 13:01:36.902643  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 52778.4us
I20260709 13:01:36.902791  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:36.885908  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:36.794516  1600 raft_consensus.cc:1240] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 10. Current term is 11. Ops: []
I20260709 13:01:36.927057  1643 leader_election.cc:304] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:01:36.920177  1569 leader_election.cc:304] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:01:36.915824  2329 raft_consensus.cc:606] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 9 LEADER]: Received request to transfer leadership to TS 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:36.869125  7316 leader_election.cc:290] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 16 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:36.900534  7418 raft_consensus.cc:2749] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:01:36.940644  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:36.948753  1643 leader_election.cc:304] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:36.953918  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: true dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:36.908284  7180 raft_consensus.cc:2804] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:01:36.958590  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:36.903820  7181 raft_consensus.cc:740] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:01:36.961151  1747 raft_consensus.cc:2393] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 17: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:01:36.913877  7370 raft_consensus.cc:2804] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:01:36.901417  6882 raft_consensus.cc:697] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:01:36.955505  7419 raft_consensus.cc:2804] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:01:36.979842  7419 raft_consensus.cc:493] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Starting leader election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:01:36.978012  7347 consensus_queue.cc:1059] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 13:01:36.980726  7347 raft_consensus.cc:3055] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:01:36.980212  6882 consensus_queue.cc:237] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.8, Last appended by leader: 8, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:36.981069  7347 raft_consensus.cc:740] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:01:36.982074  7347 consensus_queue.cc:260] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.9, Last appended by leader: 9, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:36.925897  7219 raft_consensus.cc:493] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:36.983274  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:36.983515  1643 leader_election.cc:304] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:01:36.983430  7219 raft_consensus.cc:515] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:36.984091  1750 raft_consensus.cc:2368] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 11. Current term is 12.
W20260709 13:01:36.985410  7431 log.cc:927] Time spent T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.075s	user 0.000s	sys 0.000s
I20260709 13:01:36.986049  7219 leader_election.cc:290] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:36.961455  7432 raft_consensus.cc:2804] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:01:36.987092  7432 raft_consensus.cc:493] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:36.987478  7432 raft_consensus.cc:3060] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:01:36.987488  7347 raft_consensus.cc:3060] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:01:36.925930  7305 raft_consensus.cc:2749] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 13:01:36.964643  7352 raft_consensus.cc:2749] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:01:36.928493  7374 consensus_queue.cc:1059] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.048s
I20260709 13:01:36.990969  7374 raft_consensus.cc:3055] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 LEADER]: Stepping down as leader of term 16
I20260709 13:01:36.963829  1748 raft_consensus.cc:2468] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 14.
I20260709 13:01:36.992398  1604 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40154 (request call id 2529) took 1170 ms. Trace:
I20260709 13:01:36.992828  1604 rpcz_store.cc:276] 0709 13:01:35.822035 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:35.822456 (+   421us) service_pool.cc:224] Handling call
0709 13:01:36.992196 (+1169740us) spinlock_profiling.cc:243] Waited 937 ms on lock 0x7b54004222d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:01:36.992322 (+   126us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2156078976}
I20260709 13:01:36.977335  6985 consensus_queue.cc:1059] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 10.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 13:01:36.977439  7424 consensus_queue.cc:1059] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true }, Status: INVALID_TERM, Last received: 8.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 13:01:36.963215  7180 raft_consensus.cc:493] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:36.995723  7180 raft_consensus.cc:3060] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:01:36.979091  7370 raft_consensus.cc:493] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:36.998704  7370 raft_consensus.cc:3060] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:01:36.999969  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: true dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:37.000379  2301 leader_election.cc:304] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
W20260709 13:01:37.001276  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 45260.3us
I20260709 13:01:37.001951  7414 raft_consensus.cc:3055] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 13:01:36.986500  7434 raft_consensus.cc:2804] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Leader election won for term 17
I20260709 13:01:37.002410  7414 raft_consensus.cc:740] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:01:37.002314  7170 consensus_queue.cc:1048] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 13:01:37.002960  7434 raft_consensus.cc:697] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:01:36.980229  7419 raft_consensus.cc:3060] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:01:36.942174  7337 raft_consensus.cc:2804] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:01:37.004456  7337 raft_consensus.cc:493] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Starting leader election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:01:37.003952  7434 consensus_queue.cc:237] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.10, Last appended by leader: 10, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:37.004877  7337 raft_consensus.cc:3060] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:01:36.948264  7398 raft_consensus.cc:493] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:01:36.942036  7293 raft_consensus.cc:2749] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:01:37.006161  7398 raft_consensus.cc:515] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:36.962178  1569 leader_election.cc:304] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 13:01:37.003532  7414 consensus_queue.cc:260] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.11, Last appended by leader: 11, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:36.925099  7435 raft_consensus.cc:2804] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:01:37.000634  1753 raft_consensus.cc:3060] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:01:36.991334  7374 raft_consensus.cc:740] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:01:37.022671  2301 leader_election.cc:304] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
W20260709 13:01:37.008481  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 46401.8us
I20260709 13:01:36.941560  7421 raft_consensus.cc:697] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:01:37.021571  1718 leader_election.cc:304] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:36.946853  7301 raft_consensus.cc:2804] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:01:37.014482  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:37.030128  1678 raft_consensus.cc:3060] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Advancing to term 16
W20260709 13:01:37.019891  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 226053us
I20260709 13:01:37.032359  1644 leader_election.cc:304] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:01:37.035069  7036 raft_consensus.cc:2804] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:01:37.034614  7181 consensus_queue.cc:260] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:37.035595  7036 raft_consensus.cc:493] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:37.036083  7036 raft_consensus.cc:3060] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:01:37.036196  7181 raft_consensus.cc:3060] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:01:37.029135  1718 leader_election.cc:304] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:36.925307  7336 raft_consensus.cc:515] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:37.050989  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:37.051384  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:37.054806  2331 raft_consensus.cc:1240] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 15. Current term is 16. Ops: []
I20260709 13:01:37.035355  7421 consensus_queue.cc:237] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 15.13, Last appended by leader: 13, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:37.055943  2328 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23"
dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
mode: GRACEFUL
new_leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
 from {username='slave'} at 127.0.0.1:35790
I20260709 13:01:37.056818  2328 raft_consensus.cc:606] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Received request to transfer leadership to TS 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:37.057272  2328 raft_consensus.cc:614] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 13:01:37.046164  1679 raft_consensus.cc:2468] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 12.
I20260709 13:01:37.018671  7305 raft_consensus.cc:2804] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:01:37.048504  1718 leader_election.cc:304] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:37.019555  7435 raft_consensus.cc:493] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Starting leader election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:01:36.955086  1680 raft_consensus.cc:2468] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 12.
W20260709 13:01:37.074896  1673 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52778 (request call id 2379) took 1862 ms (client timeout 1638 ms). Trace:
W20260709 13:01:37.075340  1673 rpcz_store.cc:269] 0709 13:01:35.212444 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:35.213916 (+  1472us) service_pool.cc:224] Handling call
0709 13:01:37.074668 (+1860752us) spinlock_profiling.cc:243] Waited 1.16 s on lock 0x7b54001f6154. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:01:37.074811 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2671008512}
I20260709 13:01:37.066998  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:37.029786  7301 raft_consensus.cc:493] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:37.043977  7325 raft_consensus.cc:2749] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:01:37.047937  7424 raft_consensus.cc:493] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:37.087908  1753 raft_consensus.cc:2468] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 13.
I20260709 13:01:37.108829  1646 leader_election.cc:304] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:01:37.112744  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:37.113478  1753 raft_consensus.cc:3060] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:01:37.060164  7414 raft_consensus.cc:3060] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Advancing to term 14
W20260709 13:01:37.114219  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:37.123193  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:37.123967  1750 raft_consensus.cc:2468] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 12.
I20260709 13:01:37.093335  1746 raft_consensus.cc:2468] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 13.
I20260709 13:01:37.066480  7377 raft_consensus.cc:2804] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:01:37.127051  1749 raft_consensus.cc:1240] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 12. Current term is 13. Ops: [12.9-12.9]
I20260709 13:01:37.067790  7305 raft_consensus.cc:493] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:37.131795  1678 raft_consensus.cc:2468] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 16.
I20260709 13:01:37.077735  7435 raft_consensus.cc:3060] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:01:37.167721  1671 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52778 (request call id 2370) took 2035 ms. Trace:
I20260709 13:01:37.170303  1601 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40154 (request call id 2532) took 1343 ms. Trace:
I20260709 13:01:37.170867  1601 rpcz_store.cc:276] 0709 13:01:35.827250 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:35.828378 (+  1128us) service_pool.cc:224] Handling call
0709 13:01:37.170007 (+1341629us) spinlock_profiling.cc:243] Waited 327 ms on lock 0x7b54000f0d54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:01:37.170212 (+   205us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":752836352}
I20260709 13:01:37.176857  1753 raft_consensus.cc:2468] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 16.
I20260709 13:01:37.107784  7301 raft_consensus.cc:3060] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:01:37.166594  1646 leader_election.cc:304] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:37.107930  7418 raft_consensus.cc:493] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:01:37.185500  1751 raft_consensus.cc:1240] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 17 LEADER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 16. Current term is 17. Ops: []
I20260709 13:01:37.192677  7395 raft_consensus.cc:493] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:37.193317  7395 raft_consensus.cc:515] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:37.196616  7395 leader_election.cc:290] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:37.197302  7164 raft_consensus.cc:3055] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:01:37.197862  7164 raft_consensus.cc:740] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:01:37.204283  1754 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43156 (request call id 1144) took 1628 ms. Trace:
I20260709 13:01:37.204869  1754 rpcz_store.cc:276] 0709 13:01:35.576306 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:35.576450 (+   144us) service_pool.cc:224] Handling call
0709 13:01:35.576828 (+   378us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:36.447313 (+870485us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:37.203878 (+756565us) spinlock_profiling.cc:243] Waited 867 ms on lock 0x7b54000e4554. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:01:37.204169 (+   291us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1993385728}
I20260709 13:01:37.207907  1745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55848 (request call id 2814) took 1370 ms. Trace:
I20260709 13:01:37.208491  1745 rpcz_store.cc:276] 0709 13:01:35.837099 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:35.837256 (+   157us) service_pool.cc:224] Handling call
0709 13:01:37.207569 (+1370313us) spinlock_profiling.cc:243] Waited 880 ms on lock 0x7b5400019fd4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:01:37.207804 (+   235us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2024639232}
I20260709 13:01:37.213213  1571 leader_election.cc:304] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:01:37.214671  7395 raft_consensus.cc:2749] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:01:37.220892  2330 raft_consensus.cc:3055] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:01:37.221448  2330 raft_consensus.cc:740] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:01:37.234977  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:37.236343  1747 raft_consensus.cc:2393] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 14: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:01:37.237860  2333 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39726 (request call id 1306) took 1653 ms. Trace:
I20260709 13:01:37.244232  2333 rpcz_store.cc:276] 0709 13:01:35.584296 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:35.584526 (+   230us) service_pool.cc:224] Handling call
0709 13:01:37.237639 (+1653113us) spinlock_profiling.cc:243] Waited 1.04 s on lock 0x7b5400551754. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:01:37.237776 (+   137us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2394194304}
I20260709 13:01:37.246959  7164 consensus_queue.cc:260] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.10, Last appended by leader: 10, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:37.249856  2330 consensus_queue.cc:260] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:37.258015  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 44933.5us
I20260709 13:01:37.259492  1571 leader_election.cc:304] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:37.260731  7298 raft_consensus.cc:2804] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:01:37.261567  7298 raft_consensus.cc:697] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:01:37.263810  7298 consensus_queue.cc:237] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:01:37.266129  1644 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.107.196:46625 timed out after 2.234s (SENT) [suppressed 17 similar messages]
W20260709 13:01:37.267757  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:37.272202  1571 leader_election.cc:304] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:37.168202  1671 rpcz_store.cc:276] 0709 13:01:35.132630 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:35.132866 (+   236us) service_pool.cc:224] Handling call
0709 13:01:35.133375 (+   509us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:01:35.137212 (+  3837us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:35.137261 (+    49us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:01:35.143143 (+  5882us) log.cc:844] Serialized 52 byte log entry
0709 13:01:35.305616 (+162473us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:01:35.775924 (+470308us) log.cc:844] Serialized 14 byte log entry
0709 13:01:35.776398 (+   474us) log.cc:844] Serialized 14 byte log entry
0709 13:01:36.109299 (+332901us) spinlock_profiling.cc:243] Waited 7.93 ms on lock 0x7b1800473440. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f4262fefc0a 00007f426c72f219 00007f4263077a40 00007f42630724e4 00007f4263072455 00007f426615a060 00007f426616c716 00007f426617d287 00007f426617d208
0709 13:01:36.986406 (+877107us) spinlock_profiling.cc:243] Waited 4.93 ms on lock 0x7b1800473440. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f4263057e07 00007f426beb2ab9 00007f426307469b 00007f426307293f 00007f4263072455 00007f426615a060 00007f426616c716 00007f426617d287 00007f426617d208
0709 13:01:37.167299 (+180893us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:37.167408 (+   109us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:37.167457 (+    49us) raft_consensus.cc:1685] finished
0709 13:01:37.167472 (+    15us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:37.167645 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":329630,"prepare.queue_time_us":1253334,"prepare.run_cpu_time_us":79,"prepare.run_wall_time_us":81,"raft.queue_time_us":344448,"raft.run_cpu_time_us":518,"raft.run_wall_time_us":33304,"spinlock_wait_cycles":29566720,"thread_start_us":793239,"threads_started":3,"wal-append.queue_time_us":798627}
W20260709 13:01:37.277469  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 110996us
I20260709 13:01:37.127367  7143 leader_election.cc:290] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 14 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:37.135990  7422 raft_consensus.cc:2764] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Leader pre-election decision vote started in defunct term 15: won
I20260709 13:01:37.164969  7305 raft_consensus.cc:3060] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:01:37.286789  1569 leader_election.cc:304] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:01:37.165306  7373 raft_consensus.cc:2749] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:01:37.119508  7424 raft_consensus.cc:515] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:37.284988  7304 raft_consensus.cc:2804] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Leader election won for term 13
I20260709 13:01:37.178979  7390 raft_consensus.cc:493] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:01:37.289325  7319 raft_consensus.cc:493] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:37.280616  7383 raft_consensus.cc:2804] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:01:37.135987  7377 raft_consensus.cc:493] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:37.289921  7398 leader_election.cc:290] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:37.281241  7439 raft_consensus.cc:2749] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:01:37.179415  7438 consensus_queue.cc:1059] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.178s
I20260709 13:01:37.308897  7219 raft_consensus.cc:2804] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader pre-election won for term 15
W20260709 13:01:37.331326  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 74058.5us
I20260709 13:01:37.126605  7374 consensus_queue.cc:260] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.12, Last appended by leader: 12, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:37.343689  1717 leader_election.cc:304] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:37.313562  7383 raft_consensus.cc:493] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:37.317662  7377 raft_consensus.cc:3060] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:01:37.349606  1717 leader_election.cc:304] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:37.352415  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:37.354085  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:37.354962  1672 raft_consensus.cc:2468] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 12.
I20260709 13:01:37.357944  1569 leader_election.cc:304] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:01:37.359481  1717 leader_election.cc:304] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:37.360448  1671 raft_consensus.cc:1275] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 12 index: 8. (index mismatch)
I20260709 13:01:37.361881  1677 raft_consensus.cc:2368] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 11. Current term is 12.
W20260709 13:01:37.366819  2301 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:37.374898  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:37.286203  7418 raft_consensus.cc:515] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:37.379058  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:37.327919  7304 raft_consensus.cc:697] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:01:37.340970  7398 raft_consensus.cc:493] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:01:37.345467  7219 raft_consensus.cc:493] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:37.426860  7325 raft_consensus.cc:2764] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Leader pre-election decision vote started in defunct term 15: won
I20260709 13:01:37.427050  7219 raft_consensus.cc:3060] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:01:37.427752  7325 raft_consensus.cc:2804] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:01:37.428179  7325 raft_consensus.cc:493] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:37.428583  7325 raft_consensus.cc:3060] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:01:37.431618  7424 leader_election.cc:290] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:37.315907  7390 raft_consensus.cc:515] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:37.438910  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:37.438679  7398 raft_consensus.cc:515] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:37.440435  2332 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7"
dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
mode: GRACEFUL
new_leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
 from {username='slave'} at 127.0.0.1:35790
I20260709 13:01:37.441043  2332 raft_consensus.cc:606] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 16 LEADER]: Received request to transfer leadership to TS 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:37.441721  7398 leader_election.cc:290] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:01:37.411234  1643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:37.442855   431 auto_leader_rebalancer.cc:452] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260709 13:01:37.442986  7336 leader_election.cc:290] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:37.444475  7304 consensus_queue.cc:237] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.8, Last appended by leader: 8, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:37.449309  7347 raft_consensus.cc:2804] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader pre-election won for term 16
W20260709 13:01:37.453965  7441 log.cc:927] Time spent T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.148s	user 0.000s	sys 0.000s
I20260709 13:01:37.449909  7347 raft_consensus.cc:493] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:37.456205  7347 raft_consensus.cc:3060] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:01:37.385200  1672 raft_consensus.cc:2468] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 11.
I20260709 13:01:37.316807  7319 raft_consensus.cc:515] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:37.349032  7383 raft_consensus.cc:3060] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:01:37.308298  7364 raft_consensus.cc:515] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:37.414361  1748 raft_consensus.cc:1240] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 14. Current term is 15. Ops: []
I20260709 13:01:37.360175  7442 raft_consensus.cc:2749] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20260709 13:01:37.422750  1570 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:37.426226  7418 leader_election.cc:290] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:37.445565   431 auto_leader_rebalancer.cc:527] All tables' leader rebalancing finished this round
I20260709 13:01:37.487905  7432 raft_consensus.cc:515] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:37.428784  7370 raft_consensus.cc:515] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:37.495805  7370 leader_election.cc:290] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 16 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:37.496536  7370 raft_consensus.cc:2804] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:01:37.497084  7370 raft_consensus.cc:493] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:37.497498  7370 raft_consensus.cc:3060] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Advancing to term 13
W20260709 13:01:37.497920  7180 consensus_meta.cc:287] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.502s	user 0.006s	sys 0.006s
I20260709 13:01:37.500288  7364 leader_election.cc:290] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 15 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:37.499773  7180 raft_consensus.cc:515] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:37.501525  1752 raft_consensus.cc:2468] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 15.
I20260709 13:01:37.503104  7180 leader_election.cc:290] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 18 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:37.504087  7319 leader_election.cc:290] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:37.505815  7432 leader_election.cc:290] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 16 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:37.420248  7247 consensus_queue.cc:1048] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.086s
I20260709 13:01:37.484613  7141 consensus_queue.cc:1059] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.023s
I20260709 13:01:37.507864  7141 raft_consensus.cc:3055] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 13:01:37.508319  7141 raft_consensus.cc:740] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:01:37.516239  1646 leader_election.cc:304] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:37.523245  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:37.509397  7141 consensus_queue.cc:260] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 13.10, Last appended by leader: 10, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:37.525468  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 476953us
W20260709 13:01:37.479589  7338 consensus_meta.cc:287] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.978s	user 0.015s	sys 0.002s
I20260709 13:01:37.530742  1718 leader_election.cc:304] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
W20260709 13:01:37.485119  7111 consensus_meta.cc:287] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 1.184s	user 0.013s	sys 0.002s
I20260709 13:01:37.559263  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:37.560321  1746 raft_consensus.cc:2468] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 11.
I20260709 13:01:37.564983  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:37.566015  1746 raft_consensus.cc:2393] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 16: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:01:37.569486  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:37.543784  1718 leader_election.cc:304] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
W20260709 13:01:37.571460  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 55325us
I20260709 13:01:37.571259  1718 leader_election.cc:304] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:37.572507  1646 leader_election.cc:304] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
W20260709 13:01:37.574342  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 308197us
I20260709 13:01:37.577561  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:37.583140  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:37.533776  7413 leader_election.cc:290] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:37.544243  7307 raft_consensus.cc:2749] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:01:37.571586  7432 raft_consensus.cc:493] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:37.519315  7337 raft_consensus.cc:515] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:37.579195  7141 raft_consensus.cc:3060] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:01:37.579246  7434 raft_consensus.cc:2804] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:01:37.583618  7247 raft_consensus.cc:2804] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Leader election won for term 16
W20260709 13:01:37.587571  7419 consensus_meta.cc:287] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.581s	user 0.014s	sys 0.000s
W20260709 13:01:37.590510  7447 log.cc:927] Time spent T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.055s	user 0.001s	sys 0.000s
I20260709 13:01:37.579434  7442 raft_consensus.cc:2804] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:01:37.554888  7111 raft_consensus.cc:515] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:37.605814  2299 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.598s	user 0.016s	sys 0.008s
I20260709 13:01:37.643951  1674 raft_consensus.cc:1240] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 14. Current term is 15. Ops: []
W20260709 13:01:37.668020  2299 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.660s	user 0.018s	sys 0.010s
W20260709 13:01:37.668814  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 661207us
W20260709 13:01:37.669145  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 309765us
I20260709 13:01:37.652981  1646 leader_election.cc:304] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
W20260709 13:01:37.606565  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 248726us
W20260709 13:01:37.679427  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:37.767401  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 97256.1us
W20260709 13:01:37.699352  1644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:37.666872  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 394708us
I20260709 13:01:37.865792  1571 leader_election.cc:304] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:01:37.868655  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:37.868754  1717 leader_election.cc:304] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:37.871141  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:37.871922  2329 raft_consensus.cc:3060] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:01:37.872866  2332 raft_consensus.cc:2393] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 14: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:01:37.671164  2299 leader_election.cc:304] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:37.874598  1717 leader_election.cc:304] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:37.876235  1571 leader_election.cc:400] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 election: Vote denied by peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693) with higher term. Message: Invalid argument: T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 11. Current term is 12.
I20260709 13:01:37.876641  1571 leader_election.cc:403] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 election: Cancelling election due to peer responding with higher term
W20260709 13:01:37.653172  7446 log.cc:927] Time spent T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.173s	user 0.001s	sys 0.000s
I20260709 13:01:37.599236  7338 raft_consensus.cc:2804] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Leader election won for term 16
W20260709 13:01:37.877856  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 225050us
W20260709 13:01:37.715700  7181 consensus_meta.cc:287] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.679s	user 0.015s	sys 0.002s
W20260709 13:01:37.885419  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 214350us
I20260709 13:01:37.881978  1717 leader_election.cc:304] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:37.606014  7434 raft_consensus.cc:493] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:37.678761  7319 raft_consensus.cc:493] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:37.887053  7336 raft_consensus.cc:2804] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:01:37.887758  2299 leader_election.cc:304] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:37.713958  7200 raft_consensus.cc:493] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
W20260709 13:01:37.894016  1570 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:37.895990  2301 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:37.912568  2301 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:37.806516  7414 consensus_meta.cc:287] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.680s	user 0.010s	sys 0.004s
I20260709 13:01:37.884065  7307 raft_consensus.cc:2804] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Leader pre-election won for term 16
W20260709 13:01:37.884720  7036 consensus_meta.cc:287] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.839s	user 0.013s	sys 0.000s
I20260709 13:01:37.903671  1644 leader_election.cc:304] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:01:37.900015  1717 leader_election.cc:304] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:37.604496  7432 raft_consensus.cc:515] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:37.881106  7442 raft_consensus.cc:697] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 12 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:01:37.896214  7434 raft_consensus.cc:3060] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Advancing to term 13
W20260709 13:01:37.893921  7301 consensus_meta.cc:287] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.608s	user 0.007s	sys 0.006s
I20260709 13:01:37.943883  7461 raft_consensus.cc:3060] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:01:37.945431  7417 raft_consensus.cc:493] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:37.954885  7462 raft_consensus.cc:493] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:01:37.671985  7419 raft_consensus.cc:515] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:37.975754  7417 raft_consensus.cc:515] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:37.975903  7462 raft_consensus.cc:515] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:37.963745  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 60167.8us
I20260709 13:01:37.977624  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:37.978058  7417 leader_election.cc:290] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:37.978329  7462 leader_election.cc:290] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:37.945235  7200 raft_consensus.cc:515] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:38.003357  7111 leader_election.cc:290] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 17 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:38.003847  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:37.962018  7376 raft_consensus.cc:2749] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:01:37.997807  7434 raft_consensus.cc:515] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } }
I20260709 13:01:38.011698  1601 raft_consensus.cc:1240] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 16. Current term is 17. Ops: []
I20260709 13:01:38.012768  7434 leader_election.cc:290] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 13 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:38.018289  1679 raft_consensus.cc:1240] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 15. Current term is 16. Ops: []
I20260709 13:01:37.959800  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:38.023451  1680 raft_consensus.cc:2393] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 16: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:01:37.978338  1678 raft_consensus.cc:3060] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Advancing to term 18
I20260709 13:01:37.957751  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:37.902344  7364 raft_consensus.cc:2804] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 13:01:37.910879  7319 raft_consensus.cc:515] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:37.945807  7307 raft_consensus.cc:493] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:37.972407  7432 leader_election.cc:290] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:37.973340  7442 consensus_queue.cc:237] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:01:37.974043  7135 consensus_meta.cc:287] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 1.175s	user 0.011s	sys 0.003s
I20260709 13:01:37.917541  7458 raft_consensus.cc:2749] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:01:37.892505  7451 raft_consensus.cc:2804] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 13:01:37.974874  7301 raft_consensus.cc:515] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:37.980386  6882 raft_consensus.cc:493] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:37.980461  7456 raft_consensus.cc:493] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:37.891940  7395 raft_consensus.cc:493] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:37.984359  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
W20260709 13:01:37.954274  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 54369.8us
I20260709 13:01:38.084170  2333 raft_consensus.cc:2376] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 14. Re-sending same reply.
I20260709 13:01:37.988472  2332 raft_consensus.cc:1275] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 15 index: 13. Preceding OpId from leader: term: 16 index: 14. (index mismatch)
I20260709 13:01:37.989032  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:37.974953  7421 raft_consensus.cc:2749] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 13:01:38.029978  1671 raft_consensus.cc:2468] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 14.
W20260709 13:01:38.035504  1644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:38.074074  7219 consensus_meta.cc:287] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.646s	user 0.014s	sys 0.000s
I20260709 13:01:38.054358  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:38.056851  7364 raft_consensus.cc:697] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [term 11 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:01:38.059433  7319 leader_election.cc:290] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:38.060549  1598 raft_consensus.cc:1240] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 12. Current term is 13. Ops: []
I20260709 13:01:37.911350  7336 raft_consensus.cc:493] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:38.002193  7200 leader_election.cc:290] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:38.086545  1748 raft_consensus.cc:2468] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 15.
I20260709 13:01:37.935638  1718 leader_election.cc:304] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:38.066257  7307 raft_consensus.cc:3060] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:01:37.892154  7460 raft_consensus.cc:2749] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 13:01:38.090250  1644 leader_election.cc:304] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:01:37.961799  7036 raft_consensus.cc:515] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:38.077097  7135 raft_consensus.cc:2749] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:01:38.105571  1599 raft_consensus.cc:2393] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 18: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:01:38.051367  7287 consensus_queue.cc:1059] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 16.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.038s
I20260709 13:01:38.088909  7451 raft_consensus.cc:697] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 11 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:01:38.128181  2301 leader_election.cc:304] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
W20260709 13:01:38.132282  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 196830us
W20260709 13:01:38.105727  7362 consensus_meta.cc:287] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 1.268s	user 0.017s	sys 0.000s
I20260709 13:01:37.917112  7180 raft_consensus.cc:2749] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:01:37.869518  7247 raft_consensus.cc:697] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 16 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:01:38.104660  7458 raft_consensus.cc:493] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:38.110101  7336 raft_consensus.cc:3060] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:01:37.917019  7424 consensus_queue.cc:1059] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 15.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.241s
I20260709 13:01:37.917018  7338 raft_consensus.cc:697] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 16 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:01:38.166910  1746 raft_consensus.cc:2468] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 14.
I20260709 13:01:38.136130  7450 raft_consensus.cc:493] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:01:38.087749  6882 raft_consensus.cc:515] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:38.162108  7164 raft_consensus.cc:3060] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Advancing to term 15
W20260709 13:01:38.173628  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 297520us
I20260709 13:01:38.147601  7421 raft_consensus.cc:2804] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:01:38.139185  7413 raft_consensus.cc:493] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:38.092787  7395 raft_consensus.cc:515] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:38.105971  7219 raft_consensus.cc:515] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:01:38.160805  7463 log.cc:927] Time spent T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.078s	user 0.000s	sys 0.000s
I20260709 13:01:38.146620  1678 raft_consensus.cc:2468] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 18.
I20260709 13:01:38.092526  7456 raft_consensus.cc:515] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:38.187433  1571 leader_election.cc:304] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:01:38.178115  6985 raft_consensus.cc:2804] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:01:38.151183  7357 raft_consensus.cc:493] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:01:38.148257  7319 raft_consensus.cc:2749] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 13:01:38.166092  7111 raft_consensus.cc:493] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:38.173702  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:01:38.187080  7452 log.cc:927] Time spent T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.305s	user 0.000s	sys 0.001s
I20260709 13:01:38.195829  1571 leader_election.cc:304] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:38.183133  7436 raft_consensus.cc:493] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:38.169075  7362 raft_consensus.cc:515] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:38.216974  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:38.181758  7458 raft_consensus.cc:515] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:38.198931  7421 raft_consensus.cc:697] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 14 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:01:38.200049  6985 raft_consensus.cc:493] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:38.229211  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:38.227087  1604 raft_consensus.cc:1240] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 LEADER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 14. Current term is 15. Ops: []
I20260709 13:01:38.228382  2329 raft_consensus.cc:2468] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 16.
I20260709 13:01:38.229199  1673 raft_consensus.cc:3055] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:01:38.211189  7111 raft_consensus.cc:515] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:01:38.239182  7467 log.cc:927] Time spent T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.054s	user 0.000s	sys 0.001s
I20260709 13:01:38.240051  7219 leader_election.cc:290] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 15 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:38.240595  7111 leader_election.cc:290] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:38.240795  7219 raft_consensus.cc:2804] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Leader election won for term 13
I20260709 13:01:38.210462  7417 raft_consensus.cc:2749] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:01:38.241434  7219 raft_consensus.cc:697] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:01:38.187426  7450 raft_consensus.cc:515] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:38.245051  7450 leader_election.cc:290] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:01:38.250162  7370 consensus_meta.cc:287] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.752s	user 0.012s	sys 0.001s
I20260709 13:01:38.235019  1605 raft_consensus.cc:1240] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 12. Current term is 13. Ops: []
I20260709 13:01:38.232563  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:38.237170  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:38.238484  1673 raft_consensus.cc:740] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:01:38.252204  7458 leader_election.cc:290] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:38.199923  7364 consensus_queue.cc:237] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.7, Last appended by leader: 7, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:38.211023  7357 raft_consensus.cc:515] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:38.195484  7413 raft_consensus.cc:515] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:38.255153  7421 consensus_queue.cc:237] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.11, Last appended by leader: 11, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:38.255887  7362 leader_election.cc:290] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 15 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:01:38.256626  7305 consensus_meta.cc:287] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.941s	user 0.012s	sys 0.000s
I20260709 13:01:38.234557  7247 consensus_queue.cc:237] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.12, Last appended by leader: 12, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:38.258286  7325 consensus_meta.cc:287] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.829s	user 0.015s	sys 0.000s
I20260709 13:01:38.259034  7390 leader_election.cc:290] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:38.261117  7200 raft_consensus.cc:3055] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 LEADER]: Stepping down as leader of term 15
W20260709 13:01:38.261390  7377 consensus_meta.cc:287] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.874s	user 0.008s	sys 0.006s
I20260709 13:01:38.214493  7436 raft_consensus.cc:515] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:01:38.264807  1676 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52778 (request call id 2406) took 1469 ms (client timeout 1823 ms). Trace:
I20260709 13:01:38.379443  1751 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55840 (request call id 2773) took 1433 ms. Trace:
I20260709 13:01:38.231844  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
W20260709 13:01:38.215489  7383 consensus_meta.cc:287] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.721s	user 0.013s	sys 0.000s
I20260709 13:01:38.272310  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:38.382901  1602 raft_consensus.cc:2410] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for term 13 because replica has last-logged OpId of term: 12 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 8.
W20260709 13:01:38.274132  7435 consensus_meta.cc:287] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.991s	user 0.004s	sys 0.011s
W20260709 13:01:38.274971  7445 log.cc:927] Time spent T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.862s	user 0.002s	sys 0.002s
I20260709 13:01:38.276010  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:38.387701  1600 raft_consensus.cc:2468] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 11.
I20260709 13:01:38.278841  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:38.235379  6985 raft_consensus.cc:3060] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:01:38.283859  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:38.393080  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:38.394004  1597 raft_consensus.cc:3060] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:01:38.286741  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:38.286911  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:38.280403  7456 leader_election.cc:290] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:38.312338  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:01:38.403185  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:38.312633  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:38.400614  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:38.316357  7307 raft_consensus.cc:515] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:01:38.320914  7473 log.cc:927] Time spent T 00000000000000000000000000000000 P 919f6392d4f44fd68987e2c5e7ec37ca: Append to log took a long time: real 0.083s	user 0.000s	sys 0.001s
I20260709 13:01:38.280681  7219 consensus_queue.cc:237] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:38.323439  1596 raft_consensus.cc:1240] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 14. Current term is 15. Ops: [10.8-14.10]
I20260709 13:01:38.323704  2337 raft_consensus.cc:1275] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
W20260709 13:01:38.324077  7141 consensus_meta.cc:287] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.718s	user 0.014s	sys 0.000s
W20260709 13:01:38.324581  7347 consensus_meta.cc:287] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.867s	user 0.010s	sys 0.005s
I20260709 13:01:38.312083  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:38.477494  2328 raft_consensus.cc:2368] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 15. Current term is 16.
I20260709 13:01:38.226636  7451 consensus_queue.cc:237] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.7, Last appended by leader: 7, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:38.328224  7395 leader_election.cc:290] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:38.487586  7347 raft_consensus.cc:515] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:38.279415  7111 raft_consensus.cc:493] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:38.508687  7111 raft_consensus.cc:515] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:38.279608  7370 raft_consensus.cc:515] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:38.511684  7370 leader_election.cc:290] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:38.331446  7413 leader_election.cc:290] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:38.333333  7357 leader_election.cc:290] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:38.232056  7338 consensus_queue.cc:237] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 14.7, Last appended by leader: 7, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:38.313037  7305 raft_consensus.cc:515] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:38.517601  7305 leader_election.cc:290] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 10 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:38.310525  7325 raft_consensus.cc:515] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:38.341409  7036 leader_election.cc:290] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 16 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:38.519196  1747 raft_consensus.cc:1240] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 12. Current term is 13. Ops: []
I20260709 13:01:38.312031  7301 leader_election.cc:290] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 18 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:38.344420  1753 raft_consensus.cc:1275] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 16 index: 11. Preceding OpId from leader: term: 17 index: 11. (term mismatch)
I20260709 13:01:38.345629  1748 raft_consensus.cc:1275] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 15 index: 9. Preceding OpId from leader: term: 17 index: 10. (index mismatch)
I20260709 13:01:38.524845  1604 raft_consensus.cc:2393] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 16: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:01:38.346830  1754 raft_consensus.cc:1275] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 16 index: 9. (index mismatch)
I20260709 13:01:38.348152  1745 raft_consensus.cc:1240] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 14. Current term is 15. Ops: [14.8-14.8]
I20260709 13:01:38.526959  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:38.527637  1745 raft_consensus.cc:2468] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 11.
I20260709 13:01:38.346880  1673 consensus_queue.cc:260] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 14.8, Last appended by leader: 8, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:38.528746  1673 raft_consensus.cc:3060] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:01:38.531469  1597 raft_consensus.cc:2468] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 15.
W20260709 13:01:38.344468  7448 log.cc:927] Time spent T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.961s	user 0.003s	sys 0.000s
I20260709 13:01:38.533967  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:38.350078  1598 raft_consensus.cc:1275] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 16 index: 9. (index mismatch)
I20260709 13:01:38.379894  1751 rpcz_store.cc:276] 0709 13:01:36.945673 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:36.945814 (+   141us) service_pool.cc:224] Handling call
0709 13:01:36.946097 (+   283us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:37.186083 (+239986us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:38.379133 (+1193050us) spinlock_profiling.cc:243] Waited 239 ms on lock 0x7b5400385254. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:01:38.379365 (+   232us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":549787520}
W20260709 13:01:38.378679  1676 rpcz_store.cc:269] 0709 13:01:36.795910 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:36.796092 (+   182us) service_pool.cc:224] Handling call
0709 13:01:38.264538 (+1468446us) spinlock_profiling.cc:243] Waited 1 s on lock 0x7b54001f5254. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:01:38.264721 (+   183us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2308824320}
I20260709 13:01:38.384774  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:38.391201  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:38.537417  1680 raft_consensus.cc:3060] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:01:38.395979  2333 raft_consensus.cc:2393] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 16: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:01:38.375555  7200 raft_consensus.cc:740] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:01:38.401270  1602 raft_consensus.cc:1275] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 17 index: 11. (index mismatch)
I20260709 13:01:38.409041  1600 raft_consensus.cc:1275] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 15 index: 9. Preceding OpId from leader: term: 17 index: 10. (index mismatch)
I20260709 13:01:38.409989  6882 leader_election.cc:290] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:38.411738  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:38.415869  7374 raft_consensus.cc:3060] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:01:38.419226  7337 leader_election.cc:290] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 15 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:38.421442  7432 raft_consensus.cc:3055] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 13:01:38.423101  2330 raft_consensus.cc:3060] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:01:38.425040  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:38.560796  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:38.561784  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:38.386685  7383 raft_consensus.cc:515] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } }
I20260709 13:01:38.433322  7436 leader_election.cc:290] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:38.390734  7435 raft_consensus.cc:515] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:38.379336  7377 raft_consensus.cc:515] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:38.437970  7461 raft_consensus.cc:2749] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader election lost for term 11. Reason: Vote denied by peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693) with higher term. Message: Invalid argument: T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 11. Current term is 12.
I20260709 13:01:38.403331  1605 raft_consensus.cc:2468] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 16.
I20260709 13:01:38.440574  1679 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52778 (request call id 2415) took 1425 ms. Trace:
I20260709 13:01:38.443240  2301 leader_election.cc:304] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:38.552932  7370 raft_consensus.cc:493] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:38.576581  1679 rpcz_store.cc:276] 0709 13:01:37.015211 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:37.061531 (+ 46320us) service_pool.cc:224] Handling call
0709 13:01:37.061844 (+   313us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:38.019383 (+957539us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:38.440159 (+420776us) spinlock_profiling.cc:243] Waited 956 ms on lock 0x7b5400163854. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:01:38.440444 (+   285us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2198752128}
W20260709 13:01:38.577221  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 134084us
I20260709 13:01:38.465370  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:38.475999  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:38.556841  7305 raft_consensus.cc:493] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:38.581209  1569 leader_election.cc:304] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:01:38.466761  7437 raft_consensus.cc:493] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:38.471761  7364 raft_consensus.cc:493] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:38.472133  7135 raft_consensus.cc:493] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:38.519945  1603 raft_consensus.cc:1240] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 9. Current term is 10. Ops: []
I20260709 13:01:38.524865  1753 pending_rounds.cc:85] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701: Aborting all ops after (but not including) 10
I20260709 13:01:38.586948  1753 pending_rounds.cc:107] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701: Aborting uncommitted NO_OP operation due to leader change: 16.11
I20260709 13:01:38.587344  1753 raft_consensus.cc:2889] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:01:38.588210  1569 leader_election.cc:304] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:01:38.533661  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:38.591244  1673 raft_consensus.cc:1275] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 15 index: 8. (term mismatch)
I20260709 13:01:38.591653  1673 pending_rounds.cc:85] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d: Aborting all ops after (but not including) 7
I20260709 13:01:38.591879  1673 pending_rounds.cc:107] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d: Aborting uncommitted NO_OP operation due to leader change: 14.8
I20260709 13:01:38.591764  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:38.533706  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:38.541285  1750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43156 (request call id 1157) took 1073 ms. Trace:
W20260709 13:01:38.586853  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 306646us
I20260709 13:01:38.595247  1675 raft_consensus.cc:1240] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 15. Current term is 16. Ops: []
I20260709 13:01:38.593220  1680 raft_consensus.cc:2468] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 16.
I20260709 13:01:38.593359  1750 rpcz_store.cc:276] 0709 13:01:37.467462 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:37.469468 (+  2006us) service_pool.cc:224] Handling call
0709 13:01:37.469957 (+   489us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:37.515345 (+ 45388us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:37.515379 (+    34us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:01:37.515423 (+    44us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:01:37.515463 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:38.541027 (+1025564us) spinlock_profiling.cc:243] Waited 45.1 ms on lock 0x7b54003700d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:01:38.541084 (+    57us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:38.541222 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":103792512}
I20260709 13:01:38.541503  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:38.532927  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:38.598260  1597 raft_consensus.cc:2393] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 17: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:01:38.550007  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:38.508718  7347 leader_election.cc:290] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 16 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:38.572043  7298 raft_consensus.cc:493] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:38.560808  1674 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52778 (request call id 2413) took 1547 ms. Trace:
I20260709 13:01:38.612533  1674 rpcz_store.cc:276] 0709 13:01:37.012991 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:37.013250 (+   259us) service_pool.cc:224] Handling call
0709 13:01:37.013609 (+   359us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:37.671674 (+658065us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:38.560556 (+888882us) spinlock_profiling.cc:243] Waited 630 ms on lock 0x7b54000ff354. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:01:38.560740 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1448113024}
I20260709 13:01:38.565598  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:38.567062  2299 leader_election.cc:304] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:01:38.540030  7459 raft_consensus.cc:493] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:01:38.614876  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 47957.2us
I20260709 13:01:38.560349  7398 raft_consensus.cc:493] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:38.631110  7398 raft_consensus.cc:515] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:38.632830  1672 raft_consensus.cc:2468] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 15.
I20260709 13:01:38.633205  2330 raft_consensus.cc:2468] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 13.
I20260709 13:01:38.584977  7293 raft_consensus.cc:2804] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Leader pre-election won for term 15
W20260709 13:01:38.633904  2330 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49082 (request call id 1049) took 1680 ms (client timeout 1620 ms). Trace:
I20260709 13:01:38.633999  7293 raft_consensus.cc:493] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:38.634344  7307 leader_election.cc:290] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:38.634524  7293 raft_consensus.cc:3060] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:01:38.635004  7456 raft_consensus.cc:493] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:38.635565  7456 raft_consensus.cc:515] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:38.634356  2330 rpcz_store.cc:269] 0709 13:01:36.953021 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:36.953611 (+   590us) service_pool.cc:224] Handling call
0709 13:01:37.220701 (+267090us) spinlock_profiling.cc:243] Waited 73.5 ms on lock 0x7b5400504ea0. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42660d9dc0 00007f4266165e18 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6
0709 13:01:38.422633 (+1201932us) spinlock_profiling.cc:243] Waited 254 ms on lock 0x7b58007d81d0. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42660be0aa 00007f4266140447 00007f4266158fe7 00007f426615a7de 00007f426616627c 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c
0709 13:01:38.633807 (+211174us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":753037824}
I20260709 13:01:38.574630  1752 raft_consensus.cc:1275] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 10 index: 8. Preceding OpId from leader: term: 13 index: 9. (index mismatch)
I20260709 13:01:38.447149  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:38.639097  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:38.639324  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:38.602644  7417 raft_consensus.cc:2804] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Leader election won for term 16
I20260709 13:01:38.585800  1749 raft_consensus.cc:2393] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 13: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:01:38.560422  7432 raft_consensus.cc:740] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:01:38.592419  1673 raft_consensus.cc:2889] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:01:38.637180  7477 raft_consensus.cc:2804] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:01:38.236167  7469 raft_consensus.cc:493] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:38.602843  1751 raft_consensus.cc:2468] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 12.
W20260709 13:01:38.669332  7444 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 15 current term: 15 is higher.
I20260709 13:01:38.606817  1745 raft_consensus.cc:3060] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:01:38.556010  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:38.607625  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:38.580054  6605 raft_consensus.cc:493] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:38.683710  6605 raft_consensus.cc:515] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:38.684659  1745 raft_consensus.cc:1275] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 15 index: 13. Preceding OpId from leader: term: 16 index: 14. (index mismatch)
I20260709 13:01:38.686100  6605 leader_election.cc:290] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:38.685737  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:38.687345  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:38.688025  1745 raft_consensus.cc:2393] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 16: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:01:38.688294  7337 consensus_queue.cc:1048] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20260709 13:01:38.688766  1644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:38.689528  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:38.690153  1745 raft_consensus.cc:3060] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:01:38.590210  7370 raft_consensus.cc:515] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:38.690232  7337 consensus_queue.cc:1048] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 13:01:38.688025  1646 leader_election.cc:304] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:01:38.692512  7370 leader_election.cc:290] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:38.693215  7469 raft_consensus.cc:515] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:38.693889  7164 raft_consensus.cc:2749] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:01:38.694662  7164 consensus_queue.cc:1048] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 13:01:38.695669  7469 leader_election.cc:290] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:38.670249  7477 raft_consensus.cc:493] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:38.697145  7477 raft_consensus.cc:3060] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:01:38.615406  7181 raft_consensus.cc:2749] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 13:01:38.613515  7305 raft_consensus.cc:515] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:38.699178  1745 raft_consensus.cc:2468] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 13.
I20260709 13:01:38.700902  1745 raft_consensus.cc:1240] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 14. Current term is 15. Ops: []
I20260709 13:01:38.701335  7305 leader_election.cc:290] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:01:38.703140  7336 consensus_meta.cc:287] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.519s	user 0.009s	sys 0.003s
I20260709 13:01:38.703631  7336 raft_consensus.cc:515] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:38.709066  7432 consensus_queue.cc:260] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 16.13, Last appended by leader: 13, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:38.710589  7432 raft_consensus.cc:3060] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:01:38.674594  7325 leader_election.cc:290] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 13 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:38.720811  7435 leader_election.cc:290] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 16 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:38.723961  7337 consensus_queue.cc:1048] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 13:01:38.724538  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:38.725736  7336 leader_election.cc:290] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 12 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:38.595165  7437 raft_consensus.cc:515] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:38.729954  7437 leader_election.cc:290] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:38.731897  7377 leader_election.cc:290] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 12 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:38.731668  6985 raft_consensus.cc:515] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:38.732796  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:38.734035  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:38.740126  7392 raft_consensus.cc:493] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:38.740825  7392 raft_consensus.cc:515] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:38.741798  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:38.745817  7460 raft_consensus.cc:493] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:38.746623  7460 raft_consensus.cc:515] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:38.594846  7364 raft_consensus.cc:515] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:38.640417  7298 raft_consensus.cc:515] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:38.744069  7336 consensus_queue.cc:1059] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 14.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.009s
I20260709 13:01:38.611582  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:38.673178  7417 raft_consensus.cc:697] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:01:38.753589  1605 raft_consensus.cc:2393] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 13: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:01:38.754027  7417 consensus_queue.cc:237] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.10, Last appended by leader: 10, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:38.754923  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:38.756556  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:38.756803  7337 consensus_queue.cc:1059] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 12.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 13:01:38.757369  1605 raft_consensus.cc:2468] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 11.
I20260709 13:01:38.616554  2299 leader_election.cc:304] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
W20260709 13:01:38.758857  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 142368us
I20260709 13:01:38.759469  7475 raft_consensus.cc:2804] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 13:01:38.617198  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:38.760082  7475 raft_consensus.cc:697] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 18 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:01:38.760355  1753 raft_consensus.cc:3060] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:01:38.761071  7475 consensus_queue.cc:237] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 16.13, Last appended by leader: 13, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:38.603690  7135 raft_consensus.cc:515] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:38.763656  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:38.764269  7135 leader_election.cc:290] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:38.586314  7423 consensus_queue.cc:1048] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.030s
I20260709 13:01:38.628891  7383 leader_election.cc:290] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 13 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:38.637775  1596 raft_consensus.cc:3060] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:01:38.639712  1752 raft_consensus.cc:3060] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:01:38.584539  2336 raft_consensus.cc:1240] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 15. Current term is 16. Ops: []
I20260709 13:01:38.639884  1746 raft_consensus.cc:2468] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 13.
I20260709 13:01:38.675850  1600 raft_consensus.cc:2468] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 17.
W20260709 13:01:38.766458  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 47392.3us
I20260709 13:01:38.767691  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:38.574575  1599 raft_consensus.cc:1275] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 12 index: 9. Preceding OpId from leader: term: 13 index: 9. (term mismatch)
I20260709 13:01:38.586864  7468 raft_consensus.cc:493] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:38.768767  1599 pending_rounds.cc:85] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a: Aborting all ops after (but not including) 8
I20260709 13:01:38.769057  1599 pending_rounds.cc:107] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a: Aborting uncommitted NO_OP operation due to leader change: 12.9
I20260709 13:01:38.769289  1599 raft_consensus.cc:2889] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:01:38.664558  1673 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52766 (request call id 2845) took 1308 ms. Trace:
I20260709 13:01:38.770381  1673 rpcz_store.cc:276] 0709 13:01:37.356617 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:37.356802 (+   185us) service_pool.cc:224] Handling call
0709 13:01:37.357201 (+   399us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:38.229074 (+871873us) spinlock_profiling.cc:243] Waited 30.7 ms on lock 0x7b5400124420. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42660d9dc0 00007f42661641c0 00007f4266165231 00007f4266161d8f 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c
0709 13:01:38.346392 (+117318us) spinlock_profiling.cc:243] Waited 5.49 ms on lock 0x7b4c00120dc0. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426442ea20 00007f426443fa29 00007f4264405644 00007f426442cf2d 00007f426442cb93 00007f426615d18b 00007f426615dcb5 00007f4266158f30 00007f426615a7de 00007f4266164b49 00007f4266165416
0709 13:01:38.664263 (+317871us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:38.664490 (+   227us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":83348608}
I20260709 13:01:38.592941  1748 raft_consensus.cc:2468] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 14.
I20260709 13:01:38.649621  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:38.609061  7247 raft_consensus.cc:493] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:38.604825  1750 raft_consensus.cc:1240] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 15. Current term is 16. Ops: []
I20260709 13:01:38.662812  7459 raft_consensus.cc:515] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:38.661705  7347 raft_consensus.cc:493] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:38.682969  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:38.752585  7298 leader_election.cc:290] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:38.582322  7395 raft_consensus.cc:493] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:01:38.758117  7164 raft_consensus.cc:3055] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:01:38.765028  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:38.765313  7437 consensus_queue.cc:1048] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 13:01:38.744050  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:38.773023  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:01:38.775835  1646 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:38.780310  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:38.786020  1718 leader_election.cc:400] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 15 election: Vote denied by peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625) with higher term. Message: Invalid argument: T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 15. Current term is 16.
I20260709 13:01:38.768330  1746 raft_consensus.cc:2468] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 14.
I20260709 13:01:38.791494  1571 leader_election.cc:304] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:01:38.800326  1753 raft_consensus.cc:2468] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 16.
I20260709 13:01:38.610703  1569 leader_election.cc:400] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 election: Vote denied by peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261) with higher term. Message: Invalid argument: T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 11. Current term is 12.
I20260709 13:01:38.801873  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:38.803960  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:38.792673  2298 leader_election.cc:304] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:01:38.806529  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:38.831630  1751 raft_consensus.cc:3060] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:01:38.831959  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:38.832767  1597 raft_consensus.cc:2468] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 13.
I20260709 13:01:38.828270  1752 raft_consensus.cc:2468] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 15.
I20260709 13:01:38.603749  7461 consensus_queue.cc:1048] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.186s
I20260709 13:01:38.806102  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:38.820608  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:01:38.836508  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:38.611127  7200 consensus_queue.cc:260] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:38.808109  1599 raft_consensus.cc:1240] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 14. Current term is 15. Ops: [14.9-14.9]
W20260709 13:01:38.839864  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 47294.1us
I20260709 13:01:38.832791  1748 raft_consensus.cc:2468] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 10.
I20260709 13:01:38.840853  2298 leader_election.cc:304] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:01:38.832082  2330 raft_consensus.cc:2468] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 11.
I20260709 13:01:38.839421  1718 leader_election.cc:403] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 15 election: Cancelling election due to peer responding with higher term
W20260709 13:01:38.831738  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 234604us
I20260709 13:01:38.846102  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:38.846786  1604 raft_consensus.cc:2393] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 15: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
W20260709 13:01:38.846918  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 61128.3us
I20260709 13:01:38.834414  1752 raft_consensus.cc:1275] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
W20260709 13:01:38.853873  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 62676.8us
I20260709 13:01:38.787739  6985 leader_election.cc:290] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 17 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:38.855762  1571 leader_election.cc:304] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:38.872334  1571 leader_election.cc:304] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:38.873267  1571 leader_election.cc:304] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 13:01:38.874653  7469 raft_consensus.cc:2804] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Leader election won for term 13
I20260709 13:01:38.875540  7301 raft_consensus.cc:2804] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:01:38.875185  7293 raft_consensus.cc:515] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:38.876008  7301 raft_consensus.cc:493] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:38.876386  7301 raft_consensus.cc:3060] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:01:38.816798  7357 consensus_queue.cc:1059] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.072s
I20260709 13:01:38.804199  7460 leader_election.cc:290] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:38.868218  6605 raft_consensus.cc:2804] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:01:38.868693  7342 raft_consensus.cc:2804] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:01:38.851516  1718 leader_election.cc:304] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:38.871207  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:38.858806  1596 raft_consensus.cc:2468] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 16.
I20260709 13:01:38.880679  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:38.895184  1673 raft_consensus.cc:3060] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:01:38.895087  1674 raft_consensus.cc:2393] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 18: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:01:38.880852  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:38.893524  1751 raft_consensus.cc:2468] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 16.
I20260709 13:01:38.896301  1753 raft_consensus.cc:3060] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:01:38.801934  7305 raft_consensus.cc:493] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:38.882467  1571 leader_election.cc:304] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:38.884093  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:38.885862  1599 raft_consensus.cc:1275] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 16 index: 12. Preceding OpId from leader: term: 17 index: 12. (term mismatch)
I20260709 13:01:38.904023  1599 pending_rounds.cc:85] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a: Aborting all ops after (but not including) 11
I20260709 13:01:38.904350  1599 pending_rounds.cc:107] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a: Aborting uncommitted NO_OP operation due to leader change: 16.12
I20260709 13:01:38.904686  1599 raft_consensus.cc:2889] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:01:38.823277  7164 raft_consensus.cc:740] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:01:38.820537  7364 leader_election.cc:290] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:01:38.902588  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 51156.6us
I20260709 13:01:38.912895  1718 leader_election.cc:304] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:38.914958  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:38.852304  7442 raft_consensus.cc:2749] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Leader election lost for term 15. Reason: Vote denied by peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625) with higher term. Message: Invalid argument: T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 15. Current term is 16.
I20260709 13:01:38.875200  7469 raft_consensus.cc:697] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:01:38.820736  7347 raft_consensus.cc:515] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:38.879379  7298 raft_consensus.cc:2749] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:01:38.816574  7247 raft_consensus.cc:515] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:38.833829  7395 raft_consensus.cc:515] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:38.883750  6976 raft_consensus.cc:493] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:38.809725  7468 raft_consensus.cc:515] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:38.853389  7316 consensus_queue.cc:1059] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 15.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.066s
I20260709 13:01:38.912778  7422 raft_consensus.cc:493] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:38.947788  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:38.948100  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:38.960078  2333 raft_consensus.cc:2393] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 16: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:01:38.894291  7342 raft_consensus.cc:493] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:38.890405  6605 raft_consensus.cc:493] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:38.895555  7336 raft_consensus.cc:493] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:38.950170  1673 raft_consensus.cc:2468] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 15.
I20260709 13:01:38.869066  7451 consensus_queue.cc:1048] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.070s
I20260709 13:01:38.890115  7483 raft_consensus.cc:2804] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:01:38.945547  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:38.912453  7377 raft_consensus.cc:2804] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:01:38.939394  7374 raft_consensus.cc:2804] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader election won for term 15
I20260709 13:01:38.971899  1747 raft_consensus.cc:2393] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 15: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:01:38.945477  7135 raft_consensus.cc:2749] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 13:01:38.997623  1753 raft_consensus.cc:2468] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 12.
I20260709 13:01:38.999758  1602 raft_consensus.cc:2468] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 12.
I20260709 13:01:38.917765  7477 raft_consensus.cc:515] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:38.913589  7305 raft_consensus.cc:515] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:38.962365  7342 raft_consensus.cc:3060] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:01:38.949407  7383 consensus_queue.cc:1059] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 15.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.052s
I20260709 13:01:38.964258  7308 raft_consensus.cc:493] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:39.023473  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:39.019335  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:38.985718  7377 raft_consensus.cc:493] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:38.990113  7483 raft_consensus.cc:493] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Starting leader election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:01:38.955510  7364 consensus_queue.cc:1059] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 14.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.055s
I20260709 13:01:38.987283  7293 leader_election.cc:290] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 15 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:38.972111  7442 raft_consensus.cc:493] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:39.000397  6605 raft_consensus.cc:3060] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:01:38.958616  6976 raft_consensus.cc:515] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:01:39.025547  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 105281us
I20260709 13:01:39.009397  7200 raft_consensus.cc:3060] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:01:38.971354  7422 raft_consensus.cc:515] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:39.015319  7374 raft_consensus.cc:697] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:01:39.020995  6882 raft_consensus.cc:3055] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 LEADER]: Stepping down as leader of term 15
I20260709 13:01:39.049664  2330 raft_consensus.cc:1275] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 16 index: 13. (index mismatch)
I20260709 13:01:38.991406  7336 raft_consensus.cc:515] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:39.060541  1751 raft_consensus.cc:1240] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 14. Current term is 15. Ops: []
I20260709 13:01:39.028115  7377 raft_consensus.cc:3060] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:01:39.025245  7135 raft_consensus.cc:493] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:39.052197  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:39.060642  1646 leader_election.cc:304] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
W20260709 13:01:39.062630  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 145692us
I20260709 13:01:39.066390  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:39.049232  7383 raft_consensus.cc:3055] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 13:01:39.068181  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:39.069481  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:39.070828  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:39.028803  7483 raft_consensus.cc:3060] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:01:39.068599  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:39.072261  1676 raft_consensus.cc:2393] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 13: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:01:39.074023  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:39.074725  1604 raft_consensus.cc:2468] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 11.
I20260709 13:01:39.075100  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:39.075729  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:39.076794  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:39.076797  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:39.080116  1604 raft_consensus.cc:2393] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 12: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:01:39.061158  7364 raft_consensus.cc:3055] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:01:39.081912  1673 raft_consensus.cc:3060] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:01:39.026305  7308 raft_consensus.cc:515] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:39.083647  1679 raft_consensus.cc:1275] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 10 index: 7. Preceding OpId from leader: term: 11 index: 8. (index mismatch)
I20260709 13:01:39.084964  1643 leader_election.cc:304] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:39.089915  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:39.059424  6882 raft_consensus.cc:740] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:01:39.114792  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 16 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:39.115270  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:39.115924  1599 raft_consensus.cc:2393] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 10: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:01:39.116034  1679 raft_consensus.cc:2393] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 16: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:01:39.069797  7383 raft_consensus.cc:740] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:01:39.216306  7308 leader_election.cc:290] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:39.216610  7477 leader_election.cc:290] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 12 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:01:39.175755  2299 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:39.217212  7383 consensus_queue.cc:260] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.10, Last appended by leader: 10, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:39.218277  7247 leader_election.cc:290] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:39.218616  7383 raft_consensus.cc:3060] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Advancing to term 16
W20260709 13:01:39.179409  7486 log.cc:927] Time spent T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.105s	user 0.000s	sys 0.000s
W20260709 13:01:39.183085  7485 log.cc:927] Time spent T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.155s	user 0.000s	sys 0.000s
W20260709 13:01:39.183925  7482 log.cc:927] Time spent T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.171s	user 0.001s	sys 0.000s
I20260709 13:01:39.185303  7468 leader_election.cc:290] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:39.188017  7336 leader_election.cc:290] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:39.189622  6976 leader_election.cc:290] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:39.072913  7135 raft_consensus.cc:515] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:39.225344  7135 leader_election.cc:290] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:01:39.226212  1718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:39.237566  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:39.238389  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:39.238492  2332 raft_consensus.cc:2468] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 14.
I20260709 13:01:39.248268  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 12 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:39.249552  1672 raft_consensus.cc:2410] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for term 12 because replica has last-logged OpId of term: 10 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 6.
I20260709 13:01:39.026401  7164 consensus_queue.cc:260] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.9, Last appended by leader: 9, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:01:39.267719  1644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:39.052356  7442 raft_consensus.cc:515] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:39.035648  7469 consensus_queue.cc:237] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:39.203051  6882 consensus_queue.cc:260] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.8, Last appended by leader: 8, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:39.088095  7316 raft_consensus.cc:2804] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:01:39.207171  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:39.208072  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:39.209241  1678 raft_consensus.cc:1240] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 12. Current term is 13. Ops: [12.8-12.8]
I20260709 13:01:39.209909  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:39.210704  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:39.211153  7347 leader_election.cc:290] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:39.087023  7364 raft_consensus.cc:740] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:01:39.068271  7417 consensus_queue.cc:1048] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.080s
I20260709 13:01:39.212458  7422 leader_election.cc:290] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:39.212154  7374 consensus_queue.cc:237] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.8, Last appended by leader: 8, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:39.214397  7305 leader_election.cc:290] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:39.215502  7498 raft_consensus.cc:493] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:39.273658  1673 raft_consensus.cc:2468] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 13.
I20260709 13:01:39.273821  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:39.281664  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:01:39.269759  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 184188us
I20260709 13:01:39.287927  2337 raft_consensus.cc:1240] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 14. Current term is 15. Ops: []
I20260709 13:01:39.304165  2328 raft_consensus.cc:2468] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 14.
I20260709 13:01:39.289043  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:39.294123  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 18 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:39.291287  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:39.305290  1748 raft_consensus.cc:3060] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:01:39.305521  1752 raft_consensus.cc:3060] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:01:39.300034  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:39.270972  6976 raft_consensus.cc:2749] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20260709 13:01:39.279489  7432 consensus_meta.cc:287] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.568s	user 0.005s	sys 0.008s
I20260709 13:01:39.271322  7435 consensus_queue.cc:1059] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 14.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.026s
I20260709 13:01:39.298010  7364 consensus_queue.cc:260] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:39.254496  7483 raft_consensus.cc:515] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:39.255463  7456 leader_election.cc:290] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:01:39.286743  7219 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.521s	user 0.010s	sys 0.011s
I20260709 13:01:39.306919  1747 raft_consensus.cc:2468] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 14.
I20260709 13:01:39.317975  1752 raft_consensus.cc:2468] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 15.
I20260709 13:01:39.293184  7316 raft_consensus.cc:697] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:01:39.302047  6882 raft_consensus.cc:3060] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:01:39.322156  6605 raft_consensus.cc:515] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:39.280452  7342 raft_consensus.cc:515] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:39.324012  1674 raft_consensus.cc:2468] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 12.
W20260709 13:01:39.327819  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 449341us
I20260709 13:01:39.344072  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:39.331427  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:39.332393  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:39.332594  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:39.332998  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:39.336498  1748 raft_consensus.cc:2468] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 17.
I20260709 13:01:39.348501  1751 raft_consensus.cc:2393] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 13: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:01:39.348464  1754 raft_consensus.cc:2468] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 12.
I20260709 13:01:39.386127  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
W20260709 13:01:39.328852  7219 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.563s	user 0.010s	sys 0.011s
I20260709 13:01:39.384950  1748 raft_consensus.cc:1240] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 14. Current term is 15. Ops: []
I20260709 13:01:39.394310  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:39.406399  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:39.407369  1747 raft_consensus.cc:2468] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 15.
I20260709 13:01:39.311815  7498 raft_consensus.cc:515] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:39.407516  1604 raft_consensus.cc:1240] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 9. Current term is 10. Ops: []
I20260709 13:01:39.408589  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:39.351680  7364 raft_consensus.cc:3060] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:01:39.413007  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 18 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:39.409981  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:39.415159  1599 raft_consensus.cc:2410] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 for term 18 because replica has last-logged OpId of term: 15 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 10.
I20260709 13:01:39.331339  7301 raft_consensus.cc:515] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:39.420385  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:01:39.422883  7499 log.cc:927] Time spent T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.088s	user 0.001s	sys 0.000s
I20260709 13:01:39.423105  7342 leader_election.cc:290] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 15 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:39.424330  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:01:39.425827  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:39.392838  7419 leader_election.cc:290] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 16 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:39.406360  7496 raft_consensus.cc:493] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:39.448679  1675 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52778 (request call id 2411) took 2630 ms. Trace:
I20260709 13:01:39.449136  1675 rpcz_store.cc:276] 0709 13:01:36.801514 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:36.929930 (+128416us) service_pool.cc:224] Handling call
0709 13:01:36.956839 (+ 26909us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:38.610135 (+1653296us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:39.430191 (+820056us) spinlock_profiling.cc:243] Waited 1.64 s on lock 0x7b54001f4854. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:01:39.430464 (+   273us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3767010560}
I20260709 13:01:39.450165  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:39.450238  2301 leader_election.cc:304] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:39.430038  7308 raft_consensus.cc:493] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:39.501793  7442 leader_election.cc:290] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:39.502092  7483 leader_election.cc:290] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 14 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:39.513370  2301 leader_election.cc:304] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:01:39.451426  7419 raft_consensus.cc:493] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:39.515239  2301 leader_election.cc:304] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:39.500234  7432 raft_consensus.cc:2804] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Leader pre-election won for term 15
W20260709 13:01:39.519071  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 434132us
I20260709 13:01:39.520478  2301 leader_election.cc:304] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:01:39.525941  7377 raft_consensus.cc:515] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:39.530421  7377 leader_election.cc:290] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:39.513307  7308 raft_consensus.cc:515] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:39.532513  7301 leader_election.cc:290] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 14 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:39.533576  7308 leader_election.cc:290] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:01:39.538158  7502 log.cc:927] Time spent T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.129s	user 0.001s	sys 0.000s
I20260709 13:01:39.538995  6605 leader_election.cc:290] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 12 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:39.499379  7316 consensus_queue.cc:237] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:39.540370  7498 leader_election.cc:290] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:39.541036  7490 raft_consensus.cc:3060] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:01:39.516237  7357 raft_consensus.cc:2749] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:01:39.451713  7496 raft_consensus.cc:515] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:39.549244  7496 leader_election.cc:290] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:39.549572  7337 raft_consensus.cc:3055] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:01:39.549969  7337 raft_consensus.cc:740] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:01:39.550335  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:39.552063  7111 leader_election.cc:290] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:39.525801  7342 raft_consensus.cc:2804] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 13:01:39.553249  7342 raft_consensus.cc:697] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 16 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:01:39.556355  1754 raft_consensus.cc:1240] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 14. Current term is 15. Ops: []
I20260709 13:01:39.559442  2332 raft_consensus.cc:2468] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 17.
I20260709 13:01:39.525087  7432 raft_consensus.cc:493] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:01:39.571081  7432 raft_consensus.cc:3060] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:01:39.526597  7422 consensus_queue.cc:1059] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 12.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.048s
I20260709 13:01:39.575517  7422 raft_consensus.cc:3055] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:01:39.575963  7422 raft_consensus.cc:740] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:01:39.577026  7422 consensus_queue.cc:260] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.7, Last appended by leader: 7, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:39.522372  7419 raft_consensus.cc:515] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:39.579927  1569 leader_election.cc:304] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:01:39.583706  7111 consensus_queue.cc:1048] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 13:01:39.585608  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:39.580005  7419 leader_election.cc:290] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:39.587141  7495 raft_consensus.cc:2749] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:01:39.590415  1569 leader_election.cc:304] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:01:39.593070  7370 raft_consensus.cc:2749] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 13:01:39.593466  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:39.594173  1747 raft_consensus.cc:1240] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 16. Current term is 17. Ops: [16.14-16.14]
I20260709 13:01:39.601802  1749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55848 (request call id 2844) took 1250 ms. Trace:
I20260709 13:01:39.601536  7111 consensus_queue.cc:1059] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 11.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 13:01:39.554422  7342 consensus_queue.cc:237] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.12, Last appended by leader: 12, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:39.602253  1749 rpcz_store.cc:276] 0709 13:01:38.351341 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:38.465022 (+113681us) service_pool.cc:224] Handling call
0709 13:01:39.601554 (+1136532us) spinlock_profiling.cc:243] Waited 3.83 ms on lock 0x7b54000190d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:01:39.601728 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":8813952}
I20260709 13:01:39.603116  7370 raft_consensus.cc:3055] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:01:39.603641  7370 raft_consensus.cc:740] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:01:39.604775  7370 consensus_queue.cc:260] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.8, Last appended by leader: 8, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } }
I20260709 13:01:39.605659  7337 consensus_queue.cc:260] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.12, Last appended by leader: 12, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:39.606758  1745 raft_consensus.cc:1240] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 12. Current term is 13. Ops: []
I20260709 13:01:39.606660  7370 raft_consensus.cc:3060] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:01:39.607291  7337 raft_consensus.cc:3060] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:01:39.609484  1745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55840 (request call id 2778) took 1080 ms. Trace:
I20260709 13:01:39.616160  7495 raft_consensus.cc:493] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:39.616814  7495 raft_consensus.cc:515] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:39.622814  1646 leader_election.cc:304] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:39.625022  7338 raft_consensus.cc:2804] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:01:39.625762  7338 raft_consensus.cc:493] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:39.625887  7200 consensus_queue.cc:1059] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 13:01:39.626394  7338 raft_consensus.cc:3060] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:01:39.627034  7200 raft_consensus.cc:3055] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:01:39.627434  7200 raft_consensus.cc:740] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:01:39.628607  7200 consensus_queue.cc:260] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.12, Last appended by leader: 12, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:39.629906  7200 raft_consensus.cc:3060] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Advancing to term 15
W20260709 13:01:39.632006  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 69396.3us
I20260709 13:01:39.633635  2301 leader_election.cc:304] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:39.563158  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:39.634953  2330 raft_consensus.cc:2368] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 11. Current term is 12.
W20260709 13:01:39.636080  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 113244us
I20260709 13:01:39.609880  1745 rpcz_store.cc:276] 0709 13:01:38.529150 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:38.702640 (+173490us) service_pool.cc:224] Handling call
0709 13:01:39.606535 (+903895us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:39.607246 (+   711us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:39.609280 (+  2034us) spinlock_profiling.cc:243] Waited 903 ms on lock 0x7b54000190d0. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661618ad 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:01:39.609416 (+   136us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2077812608}
I20260709 13:01:39.642053  2301 leader_election.cc:304] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:39.657209  1644 leader_election.cc:304] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:39.658918  1569 leader_election.cc:304] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:39.660547  1569 leader_election.cc:304] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:39.657338  1643 leader_election.cc:304] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:39.663928  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:39.664788  1749 raft_consensus.cc:3060] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:01:39.666124  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:39.667987  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:39.668874  1752 raft_consensus.cc:3060] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:01:39.624993  7370 raft_consensus.cc:493] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:39.666981  1747 raft_consensus.cc:2468] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 14.
I20260709 13:01:39.671293  1569 leader_election.cc:304] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:39.673369  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:39.676718  2301 leader_election.cc:304] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:39.677063  2330 raft_consensus.cc:2468] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 16.
I20260709 13:01:39.679601  1569 leader_election.cc:304] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:39.683254  1569 leader_election.cc:304] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:39.654402  7450 raft_consensus.cc:2804] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 13:01:39.695586  1569 leader_election.cc:304] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:39.696877  1643 leader_election.cc:304] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:01:39.706367  1752 raft_consensus.cc:2468] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 12.
I20260709 13:01:39.655443  7497 raft_consensus.cc:493] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:39.659433  7422 raft_consensus.cc:3060] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:01:39.709508  1644 leader_election.cc:304] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:01:39.711216  2301 leader_election.cc:304] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:39.701627  6985 raft_consensus.cc:2804] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:01:39.685110  7510 raft_consensus.cc:2804] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Leader election won for term 15
W20260709 13:01:39.712052  1645 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:39.684981  7496 raft_consensus.cc:2749] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:01:39.698905  7219 raft_consensus.cc:2749] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 13:01:39.729344  7423 raft_consensus.cc:2804] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:01:39.729861  7423 raft_consensus.cc:493] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:39.721575  1749 raft_consensus.cc:2468] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 15.
I20260709 13:01:39.687201  7111 raft_consensus.cc:2749] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:01:39.686192  7483 raft_consensus.cc:2749] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 13:01:39.687026  7468 raft_consensus.cc:2749] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:01:39.700079  7419 raft_consensus.cc:2804] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:01:39.699795  7305 raft_consensus.cc:2749] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:01:39.732893  2301 leader_election.cc:304] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:39.720747  6985 raft_consensus.cc:493] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:39.716017  7450 raft_consensus.cc:697] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 17 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:01:39.720993  7181 raft_consensus.cc:2804] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:01:39.701166  7370 raft_consensus.cc:515] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:39.730222  7423 raft_consensus.cc:3060] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:01:39.730072  7469 raft_consensus.cc:2749] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
W20260709 13:01:39.758167  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 62707.4us
I20260709 13:01:39.759135  1569 leader_election.cc:304] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:39.760669  7219 raft_consensus.cc:2804] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Leader election won for term 17
I20260709 13:01:39.761209  7219 raft_consensus.cc:697] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:01:39.762275  1750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55848 (request call id 2857) took 1006 ms. Trace:
I20260709 13:01:39.762180  7219 consensus_queue.cc:237] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 16.13, Last appended by leader: 13, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:39.762848  1750 rpcz_store.cc:276] 0709 13:01:38.756062 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:38.850898 (+ 94836us) service_pool.cc:224] Handling call
0709 13:01:38.851300 (+   402us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:38.879292 (+ 27992us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:38.879332 (+    40us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:01:38.879843 (+   511us) log.cc:844] Serialized 26 byte log entry
0709 13:01:39.595842 (+715999us) spinlock_profiling.cc:243] Waited 1.48 ms on lock 0x7b1800471ee0. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f4263057e07 00007f426beb2ab9 00007f426307469b 00007f426307293f 00007f4263074c02 00007f4265cfc20b 00007f4265d01d23 00007f4265d020d1 00007f426611da0f
0709 13:01:39.727948 (+132106us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:01:39.728002 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:39.728140 (+   138us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:39.748966 (+ 20826us) raft_consensus.cc:1685] finished
0709 13:01:39.748992 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:39.762177 (+ 13185us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":27524,"raft.queue_time_us":32525,"raft.run_cpu_time_us":150,"raft.run_wall_time_us":11492,"spinlock_wait_cycles":3413376,"thread_start_us":131652,"threads_started":1,"wal-append.queue_time_us":845732}
I20260709 13:01:39.767913  7164 raft_consensus.cc:3060] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:01:39.720429  7497 raft_consensus.cc:515] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:39.772744  7497 leader_election.cc:290] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:39.731017  7510 raft_consensus.cc:697] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 15 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:01:39.740365  6882 raft_consensus.cc:2804] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Leader election won for term 15
I20260709 13:01:39.786618  7423 raft_consensus.cc:515] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:39.788128  7370 leader_election.cc:290] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:39.788833  7392 leader_election.cc:290] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:39.789235  7423 leader_election.cc:290] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 17 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:39.789124  7450 consensus_queue.cc:237] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 15.13, Last appended by leader: 13, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:39.753564  6985 raft_consensus.cc:3060] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:01:39.792479  6882 raft_consensus.cc:697] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:01:39.793496  6882 consensus_queue.cc:237] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.9, Last appended by leader: 9, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:39.795179  7510 consensus_queue.cc:237] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.11, Last appended by leader: 11, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:39.779661  7495 leader_election.cc:290] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:39.732061  7509 consensus_queue.cc:1059] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 15.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.087s
I20260709 13:01:39.804253  7509 raft_consensus.cc:3055] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 LEADER]: Stepping down as leader of term 16
I20260709 13:01:39.804682  7509 raft_consensus.cc:740] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:01:39.749442  7419 raft_consensus.cc:493] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:39.806731  7419 raft_consensus.cc:3060] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:01:39.775887  7308 raft_consensus.cc:2804] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 13:01:39.809257  7308 raft_consensus.cc:697] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 15 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
W20260709 13:01:39.810623  1718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:39.810351  7308 consensus_queue.cc:237] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.11, Last appended by leader: 11, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:39.818729  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:39.819531  2328 raft_consensus.cc:3060] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:01:39.832139  7304 raft_consensus.cc:493] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:39.832998  7304 raft_consensus.cc:515] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:39.836323  7304 leader_election.cc:290] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:39.838958  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 16 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:39.839933  1752 raft_consensus.cc:2468] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 15.
I20260709 13:01:39.843115  1646 leader_election.cc:304] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:39.844364  7304 raft_consensus.cc:2804] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:01:39.845223  7304 raft_consensus.cc:493] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:39.848099  7304 raft_consensus.cc:3060] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Advancing to term 16
W20260709 13:01:39.859313  7513 log.cc:927] Time spent T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.079s	user 0.001s	sys 0.000s
I20260709 13:01:39.863139  7432 raft_consensus.cc:515] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:39.864064  6985 raft_consensus.cc:515] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:39.865623  7432 leader_election.cc:290] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 15 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:39.866914  2328 raft_consensus.cc:2468] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 17.
I20260709 13:01:39.805625  7509 consensus_queue.cc:260] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 16.14, Last appended by leader: 14, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:39.870697  1677 raft_consensus.cc:2468] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 13.
I20260709 13:01:39.871726  7509 raft_consensus.cc:3060] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:01:39.873036  7490 raft_consensus.cc:2749] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:01:39.825198  7505 raft_consensus.cc:493] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:39.874292  7505 raft_consensus.cc:515] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:39.876386  7395 leader_election.cc:290] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:39.876744  7505 leader_election.cc:290] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:01:39.877079  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 50238.4us
I20260709 13:01:39.880906  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:39.881134  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:39.882418  1603 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40154 (request call id 2550) took 1312 ms. Trace:
I20260709 13:01:39.882794  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:39.882843  1603 rpcz_store.cc:276] 0709 13:01:38.570359 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:38.724186 (+153827us) service_pool.cc:224] Handling call
0709 13:01:39.882198 (+1158012us) spinlock_profiling.cc:243] Waited 268 ms on lock 0x7b54000f0d54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:01:39.882349 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":616833280}
I20260709 13:01:39.883646  2331 raft_consensus.cc:1275] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 14 index: 10. Preceding OpId from leader: term: 16 index: 11. (index mismatch)
W20260709 13:01:39.884106  1717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:39.882949  1569 leader_election.cc:304] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:01:39.885702  7395 raft_consensus.cc:2804] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:01:39.886087  7395 raft_consensus.cc:493] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:39.886499  7395 raft_consensus.cc:3060] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:01:39.887383  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:39.887555  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:39.887971  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:39.888523  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:39.816654  7181 raft_consensus.cc:697] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 12 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:01:39.890264  1644 leader_election.cc:304] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:01:39.891021  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:39.888037  1680 raft_consensus.cc:2468] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 11.
I20260709 13:01:39.895083  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:39.907599  1569 leader_election.cc:304] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:01:39.907904  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:39.908468  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:39.909348  1602 raft_consensus.cc:2410] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for term 13 because replica has last-logged OpId of term: 12 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 7.
I20260709 13:01:39.897099  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
W20260709 13:01:39.900753  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:39.883666  1601 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 2649) took 1536 ms. Trace:
I20260709 13:01:39.911254  1601 rpcz_store.cc:276] 0709 13:01:38.347680 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:38.351574 (+  3894us) service_pool.cc:224] Handling call
0709 13:01:38.351888 (+   314us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:38.696519 (+344631us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:38.696559 (+    40us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:01:38.696623 (+    64us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:01:38.696660 (+    37us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:39.883388 (+1186728us) spinlock_profiling.cc:243] Waited 344 ms on lock 0x7b540028a854. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:01:39.883440 (+    52us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:39.883562 (+   122us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":791987072}
I20260709 13:01:39.912425  1752 raft_consensus.cc:3060] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:01:39.921267  1752 raft_consensus.cc:2468] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 15.
I20260709 13:01:39.922314  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:39.922528  7495 consensus_queue.cc:1048] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 13:01:39.923923  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:39.924217  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:39.924729  7164 raft_consensus.cc:2804] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Leader election won for term 17
I20260709 13:01:39.924898  2331 raft_consensus.cc:2468] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 16.
I20260709 13:01:39.925351  7164 raft_consensus.cc:697] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:01:39.926321  7164 consensus_queue.cc:237] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.12, Last appended by leader: 12, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:39.866241  6985 leader_election.cc:290] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 15 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:39.924729  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:39.930058  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 14 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:39.931763  1677 raft_consensus.cc:2468] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 11.
W20260709 13:01:39.929998  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 233229us
I20260709 13:01:39.936746  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:39.937636  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:39.933243  1644 leader_election.cc:304] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:01:39.938212  1603 raft_consensus.cc:3060] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:01:39.939834  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:39.941640  1602 raft_consensus.cc:2376] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 12. Re-sending same reply.
I20260709 13:01:39.948596  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:39.950393  1601 raft_consensus.cc:1275] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 11 index: 8. (index mismatch)
I20260709 13:01:39.924285  7298 raft_consensus.cc:2804] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:01:39.952333  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:39.954718  1601 raft_consensus.cc:1275] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 10 index: 7. Preceding OpId from leader: term: 11 index: 8. (index mismatch)
I20260709 13:01:39.955824  1602 raft_consensus.cc:2393] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 11: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:01:39.966747  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:39.962867  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:39.967464  1602 raft_consensus.cc:2468] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 12.
W20260709 13:01:39.957290  7364 consensus_meta.cc:287] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.531s	user 0.011s	sys 0.002s
I20260709 13:01:39.969789  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 17 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:39.967741  1601 raft_consensus.cc:2468] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 15.
I20260709 13:01:39.971810  1602 raft_consensus.cc:2468] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 16.
I20260709 13:01:39.974989  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:39.953473  7392 raft_consensus.cc:493] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:39.980352  1598 raft_consensus.cc:2393] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 11: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:01:39.961589  7316 raft_consensus.cc:2804] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:01:39.982820  1679 raft_consensus.cc:2468] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 13.
I20260709 13:01:39.957912  7298 raft_consensus.cc:493] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:39.963524  7505 raft_consensus.cc:493] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:39.990724  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:39.991484  1605 raft_consensus.cc:2468] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 19.
I20260709 13:01:39.980818  7181 consensus_queue.cc:237] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:39.998642  1643 leader_election.cc:304] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:01:40.042677  7509 raft_consensus.cc:2749] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 13:01:39.989291  7298 raft_consensus.cc:3060] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Advancing to term 12
W20260709 13:01:40.055996  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 57516.5us
I20260709 13:01:40.057242  1643 leader_election.cc:304] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:01:39.988018  7316 raft_consensus.cc:493] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:40.054530  2301 leader_election.cc:304] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:40.061123  1643 leader_election.cc:304] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:40.069916  7308 raft_consensus.cc:2804] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 13:01:40.064152  7423 raft_consensus.cc:2804] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:01:40.072274  7308 raft_consensus.cc:697] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 15 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:01:40.072451  7423 raft_consensus.cc:493] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Starting leader election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:01:40.072957  7423 raft_consensus.cc:3060] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:01:40.073396  7308 consensus_queue.cc:237] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 13.10, Last appended by leader: 10, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:40.075801  7456 raft_consensus.cc:3060] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:01:40.078593  1643 leader_election.cc:304] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:01:39.987674  7392 raft_consensus.cc:515] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:40.019178  7338 raft_consensus.cc:515] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:40.123459  1678 raft_consensus.cc:1240] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 12. Current term is 13. Ops: []
I20260709 13:01:40.070941  7316 raft_consensus.cc:3060] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:01:40.011288  7200 consensus_queue.cc:1048] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.070s
I20260709 13:01:39.994141  7505 raft_consensus.cc:515] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:40.010998  6882 consensus_queue.cc:1048] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.073s
I20260709 13:01:40.054215  7520 consensus_queue.cc:1048] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.117s
I20260709 13:01:40.180377  2337 raft_consensus.cc:1240] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 17 LEADER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 16. Current term is 17. Ops: [16.14-16.14]
I20260709 13:01:40.103350  7509 raft_consensus.cc:2749] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20260709 13:01:40.164433  7374 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.660s	user 0.005s	sys 0.009s
I20260709 13:01:40.185138  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:40.183429  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:40.188511  1601 raft_consensus.cc:2368] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for earlier term 13. Current term is 14.
I20260709 13:01:40.188472  1605 raft_consensus.cc:2468] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 10.
W20260709 13:01:40.194957  7523 log.cc:927] Time spent T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.132s	user 0.000s	sys 0.001s
I20260709 13:01:40.202618  1603 raft_consensus.cc:2468] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 16.
I20260709 13:01:40.231208  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:40.234651  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:01:40.188066  7374 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.684s	user 0.006s	sys 0.009s
W20260709 13:01:40.237922  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 291738us
W20260709 13:01:40.242686  1717 connection.cc:256] RPC call timeout handler was delayed by -1.1001804399999742s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17820000000000003s for a response
I20260709 13:01:40.235965  1718 leader_election.cc:304] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:40.254148  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:01:40.263382  7526 log.cc:927] Time spent T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.100s	user 0.000s	sys 0.000s
W20260709 13:01:40.265228  7524 log.cc:927] Time spent T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.153s	user 0.000s	sys 0.001s
I20260709 13:01:40.271623  7505 leader_election.cc:290] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:40.274752  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 15 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:40.276108  1597 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 1424) took 1539 ms. Trace:
I20260709 13:01:40.276510  1597 rpcz_store.cc:276] 0709 13:01:38.736913 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:38.848238 (+111325us) service_pool.cc:224] Handling call
0709 13:01:38.848520 (+   282us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:39.028542 (+180022us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:39.028597 (+    55us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:01:39.028659 (+    62us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:01:39.028704 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:40.275811 (+1247107us) spinlock_profiling.cc:243] Waited 180 ms on lock 0x7b54001009d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:01:40.275884 (+    73us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:40.276029 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":413465856}
I20260709 13:01:40.278332  1569 leader_election.cc:304] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:01:40.280884  1569 leader_election.cc:304] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 13:01:40.281908  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 15 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:40.282764  1747 raft_consensus.cc:2393] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 15: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:01:40.281813  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:40.291081  1597 raft_consensus.cc:2468] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 15.
I20260709 13:01:40.296357  1674 raft_consensus.cc:1275] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
I20260709 13:01:40.300166  1597 raft_consensus.cc:1275] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 14 index: 7. Preceding OpId from leader: term: 16 index: 8. (index mismatch)
I20260709 13:01:40.274847  7364 raft_consensus.cc:2749] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:01:40.308763  1646 leader_election.cc:304] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:01:40.313187  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:40.314098  1597 raft_consensus.cc:2468] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 12.
I20260709 13:01:40.320895  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 16 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:40.322710  7316 raft_consensus.cc:515] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:40.303870  7477 raft_consensus.cc:2804] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:01:40.297714  7495 raft_consensus.cc:2804] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Leader pre-election won for term 12
W20260709 13:01:40.327016  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 95874.8us
I20260709 13:01:40.332593  1674 raft_consensus.cc:1275] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 10 index: 7. Preceding OpId from leader: term: 11 index: 8. (index mismatch)
I20260709 13:01:40.340685  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:40.342862  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:40.342152  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:40.344802  1597 raft_consensus.cc:2468] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 15.
I20260709 13:01:40.325511  7496 raft_consensus.cc:3060] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Advancing to term 15
W20260709 13:01:40.355000  1643 leader_election.cc:336] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 1.789s (SENT)
I20260709 13:01:40.356626  1643 leader_election.cc:304] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:40.327811  7477 raft_consensus.cc:493] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:40.363749  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 17 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:40.310721  7219 consensus_queue.cc:1048] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.037s
I20260709 13:01:40.329365  7495 raft_consensus.cc:493] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Starting leader election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:01:40.370698  1597 raft_consensus.cc:1275] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
I20260709 13:01:40.373041  7337 raft_consensus.cc:2749] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260709 13:01:40.364732  7419 consensus_meta.cc:287] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.557s	user 0.009s	sys 0.005s
W20260709 13:01:40.395409  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:40.400537  7456 raft_consensus.cc:2749] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20260709 13:01:40.410801  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 174885us
I20260709 13:01:40.362317  7442 raft_consensus.cc:493] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:40.415112  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:40.416699  1603 raft_consensus.cc:2393] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 13: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:01:40.373282  7477 raft_consensus.cc:3060] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:01:40.369787  7374 raft_consensus.cc:493] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:40.419796  1680 raft_consensus.cc:1240] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 LEADER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 16. Current term is 17. Ops: []
W20260709 13:01:40.417414  1717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:40.375674  7495 raft_consensus.cc:3060] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:01:40.410256  7419 raft_consensus.cc:515] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:40.452771  7419 leader_election.cc:290] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 16 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:40.455044  1674 raft_consensus.cc:2393] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 16: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
W20260709 13:01:40.435307  7304 consensus_meta.cc:287] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.584s	user 0.013s	sys 0.002s
I20260709 13:01:40.455914  7304 raft_consensus.cc:515] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:40.457355  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:40.458099  7304 leader_election.cc:290] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 16 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:40.458775  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:40.458842  7539 consensus_queue.cc:1048] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 13:01:40.459663  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
W20260709 13:01:40.460064  1643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:40.447767  7540 consensus_queue.cc:1059] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 15.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.013s
I20260709 13:01:40.403348  7521 raft_consensus.cc:493] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:40.463174  1671 raft_consensus.cc:1240] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 15. Current term is 16. Ops: [15.8-15.8]
I20260709 13:01:40.463078  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:40.471371  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:40.472265  1597 raft_consensus.cc:2468] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 17.
I20260709 13:01:40.476890  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:40.402386  7423 raft_consensus.cc:515] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:40.423362  7325 consensus_queue.cc:1048] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.058s
W20260709 13:01:40.475497  7395 consensus_meta.cc:287] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.589s	user 0.012s	sys 0.001s
I20260709 13:01:40.469762  7496 raft_consensus.cc:2749] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:01:40.435530  7374 raft_consensus.cc:515] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:40.486024  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:40.486714  2329 raft_consensus.cc:3060] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:01:40.425541  7442 raft_consensus.cc:515] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:40.470952  7521 raft_consensus.cc:515] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:40.493160  7539 raft_consensus.cc:493] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:01:40.495251  7495 raft_consensus.cc:515] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:40.499781  2329 raft_consensus.cc:2468] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 16.
I20260709 13:01:40.504843  7298 raft_consensus.cc:515] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:40.510105  7521 leader_election.cc:290] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:40.510681  7491 raft_consensus.cc:3055] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 13:01:40.507800  7539 raft_consensus.cc:515] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:40.511256  7491 raft_consensus.cc:740] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:01:40.512789  7442 leader_election.cc:290] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:40.512440  7491 consensus_queue.cc:260] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.8, Last appended by leader: 8, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:40.510012  7495 leader_election.cc:290] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 12 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:40.513124  7539 leader_election.cc:290] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:40.516232  7374 leader_election.cc:290] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:40.488118  7395 raft_consensus.cc:515] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:40.521642  7398 leader_election.cc:290] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:40.522049  7491 raft_consensus.cc:3060] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:01:40.522805  7298 leader_election.cc:290] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 12 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:40.523940  7395 leader_election.cc:290] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:40.520159  7423 leader_election.cc:290] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 17 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:40.525552  7392 leader_election.cc:290] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:40.528074  7477 raft_consensus.cc:515] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:40.596743  1604 raft_consensus.cc:2410] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for term 13 because replica has last-logged OpId of term: 12 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 7.
I20260709 13:01:40.597935  1601 raft_consensus.cc:2393] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 12: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:01:40.601325  1605 raft_consensus.cc:1275] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
I20260709 13:01:40.604197  1605 raft_consensus.cc:1275] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 15 index: 10. (index mismatch)
W20260709 13:01:40.611478  1718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:40.615756  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:40.617309  1676 raft_consensus.cc:3060] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:01:40.614148  1717 leader_election.cc:304] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:40.625123  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:40.626456  1680 raft_consensus.cc:2368] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 16. Current term is 17.
I20260709 13:01:40.631753  7477 leader_election.cc:290] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:40.631954  7545 raft_consensus.cc:493] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:40.632470  7462 consensus_queue.cc:1048] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 13:01:40.633277  7545 raft_consensus.cc:515] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:40.639030  7545 leader_election.cc:290] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:40.640161  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:40.640955  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:40.615783  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:40.642117  2333 raft_consensus.cc:1240] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 14. Current term is 15. Ops: []
I20260709 13:01:40.644572  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:40.645509  2330 raft_consensus.cc:1275] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 16 index: 13. Preceding OpId from leader: term: 17 index: 14. (index mismatch)
I20260709 13:01:40.646318  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:40.648123  7539 raft_consensus.cc:2749] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 13:01:40.649963  7491 raft_consensus.cc:493] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:40.645502  1672 raft_consensus.cc:3060] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:01:40.649963  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:40.656881  7423 raft_consensus.cc:493] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:01:40.658001  7423 raft_consensus.cc:515] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:40.660061  7542 consensus_queue.cc:1048] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 13:01:40.661365  7423 leader_election.cc:290] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:40.656785  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:39.595279   431 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
W20260709 13:01:40.668232  1596 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40166 (request call id 2653) took 2260 ms (client timeout 1867 ms). Trace:
W20260709 13:01:40.668884  1596 rpcz_store.cc:269] 0709 13:01:38.407473 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:38.469898 (+ 62425us) service_pool.cc:224] Handling call
0709 13:01:40.667615 (+2197717us) spinlock_profiling.cc:243] Waited 58.9 ms on lock 0x7b540028b754. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:01:40.667944 (+   329us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":135566080}
I20260709 13:01:40.668907  1600 raft_consensus.cc:1240] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 15. Current term is 16. Ops: [15.10-15.10]
I20260709 13:01:40.675130  2328 raft_consensus.cc:2393] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 16: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:01:40.678237  7477 consensus_queue.cc:1059] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 14.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 13:01:40.679627  7491 raft_consensus.cc:515] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:40.684526  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:40.684551  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
W20260709 13:01:40.723955  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 110443us
I20260709 13:01:40.745409  7491 leader_election.cc:290] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:40.755133  1718 leader_election.cc:304] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:40.758083  1718 leader_election.cc:400] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 election: Vote denied by peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625) with higher term. Message: Invalid argument: T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 11. Current term is 12.
I20260709 13:01:40.763115  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:40.764273  7392 consensus_queue.cc:1048] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
W20260709 13:01:40.765676  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 55439.7us
I20260709 13:01:40.761888  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:40.767771  1718 leader_election.cc:403] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 election: Cancelling election due to peer responding with higher term
I20260709 13:01:40.770912  1603 raft_consensus.cc:2376] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 12. Re-sending same reply.
I20260709 13:01:40.771059  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 16 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:40.778610  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:40.783699  7462 raft_consensus.cc:2749] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:01:40.785053  7459 leader_election.cc:290] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:40.796563  7338 leader_election.cc:290] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 13 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:40.798730  7491 raft_consensus.cc:493] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:40.803544  7460 raft_consensus.cc:3055] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:01:40.809607  1605 raft_consensus.cc:1275] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 16 index: 13. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
I20260709 13:01:40.811172  1676 raft_consensus.cc:2468] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 12.
I20260709 13:01:40.812492  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:40.815338  7316 leader_election.cc:290] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 17 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:40.815737  1600 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40154 (request call id 2543) took 2247 ms. Trace:
I20260709 13:01:40.817608  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:40.827705  1752 raft_consensus.cc:2410] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for term 16 because replica has last-logged OpId of term: 15 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 9.
I20260709 13:01:40.827629  1717 leader_election.cc:304] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:40.835404  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:40.838317  2336 raft_consensus.cc:3060] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:01:40.841301  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:40.841717  7414 raft_consensus.cc:2749] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Leader election lost for term 11. Reason: Vote denied by peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625) with higher term. Message: Invalid argument: T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 11. Current term is 12.
W20260709 13:01:40.836973  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 79250.1us
I20260709 13:01:40.837188  1600 rpcz_store.cc:276] 0709 13:01:38.568692 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:38.767515 (+198823us) service_pool.cc:224] Handling call
0709 13:01:40.668096 (+1900581us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:40.670007 (+  1911us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:40.815440 (+145433us) spinlock_profiling.cc:243] Waited 1.85 s on lock 0x7b540028b750. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661618ad 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:01:40.815632 (+   192us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4243773696}
I20260709 13:01:40.849434  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:40.858378  7462 raft_consensus.cc:2749] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:01:40.862929  1602 raft_consensus.cc:3060] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:01:40.863480  2337 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39726 (request call id 1343) took 1188 ms. Trace:
I20260709 13:01:40.863947  2337 rpcz_store.cc:276] 0709 13:01:39.675003 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:39.697069 (+ 22066us) service_pool.cc:224] Handling call
0709 13:01:39.697524 (+   455us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:40.183825 (+486301us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:40.862859 (+679034us) spinlock_profiling.cc:243] Waited 482 ms on lock 0x7b5400567f54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:01:40.863243 (+   384us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1109516544}
I20260709 13:01:40.834961  7460 raft_consensus.cc:740] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
W20260709 13:01:40.873015  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:40.876420  1674 raft_consensus.cc:1275] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
I20260709 13:01:40.851172  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:40.878482  1596 raft_consensus.cc:3060] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:01:40.878966  7491 raft_consensus.cc:515] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:40.881971  7491 leader_election.cc:290] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:40.888439  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 12 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:40.890381  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:40.891547  1754 raft_consensus.cc:2468] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 14.
W20260709 13:01:40.892294  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 157849us
I20260709 13:01:40.869580  7460 consensus_queue.cc:260] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:40.895833  7460 raft_consensus.cc:3060] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:01:40.904150  7316 raft_consensus.cc:493] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:40.905150  7316 raft_consensus.cc:515] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:40.906698  7395 raft_consensus.cc:3055] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:01:40.906970  7337 raft_consensus.cc:493] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:40.907200  7395 raft_consensus.cc:740] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:01:40.907951  7337 raft_consensus.cc:515] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:40.908489  7462 consensus_queue.cc:1048] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 13:01:40.908603  7395 consensus_queue.cc:260] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:40.910815  7338 raft_consensus.cc:493] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:40.911520  7338 raft_consensus.cc:515] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:40.912370  7538 log.cc:927] Time spent T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.477s	user 0.000s	sys 0.001s
I20260709 13:01:40.913884  1717 leader_election.cc:304] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:40.914587  7338 leader_election.cc:290] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:40.918009  2330 raft_consensus.cc:1240] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 16. Current term is 17. Ops: [16.13-16.13]
I20260709 13:01:40.918273  1745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43156 (request call id 1179) took 1260 ms. Trace:
I20260709 13:01:40.918771  1745 rpcz_store.cc:276] 0709 13:01:39.658123 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:39.658269 (+   146us) service_pool.cc:224] Handling call
0709 13:01:39.658637 (+   368us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:39.777135 (+118498us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:39.777169 (+    34us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:01:39.777214 (+    45us) raft_consensus.cc:1645] Marking committed up to 13
0709 13:01:39.777257 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:40.917611 (+1140354us) spinlock_profiling.cc:243] Waited 118 ms on lock 0x7b54000bd454. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:01:40.917717 (+   106us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:40.917993 (+   276us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":271887616}
I20260709 13:01:40.920244  1678 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52766 (request call id 2869) took 1033 ms. Trace:
I20260709 13:01:40.920691  1678 rpcz_store.cc:276] 0709 13:01:39.886299 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:39.887085 (+   786us) service_pool.cc:224] Handling call
0709 13:01:39.904126 (+ 17041us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:40.123918 (+219792us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:40.919700 (+795782us) spinlock_profiling.cc:243] Waited 219 ms on lock 0x7b54000d9154. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:01:40.920008 (+   308us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":503842176}
I20260709 13:01:40.921645  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:40.922791  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:40.923702  2330 raft_consensus.cc:2468] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 14.
I20260709 13:01:40.925583  7316 leader_election.cc:290] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:01:40.927210  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:40.928638  1753 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43156 (request call id 1177) took 1638 ms. Trace:
W20260709 13:01:40.928905  7534 log.cc:927] Time spent T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.481s	user 0.001s	sys 0.004s
I20260709 13:01:40.929139  1753 rpcz_store.cc:276] 0709 13:01:39.289800 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:39.290539 (+   739us) service_pool.cc:224] Handling call
0709 13:01:39.290893 (+   354us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:39.441288 (+150395us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:39.441340 (+    52us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:01:39.441407 (+    67us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:01:39.441455 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:40.928095 (+1486640us) spinlock_profiling.cc:243] Waited 150 ms on lock 0x7b54001d5bd4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:01:40.928170 (+    75us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:40.928421 (+   251us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":345107840}
I20260709 13:01:40.932013  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:40.933065  1747 raft_consensus.cc:2393] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 12: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:01:40.933320  1679 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52766 (request call id 2867) took 1047 ms. Trace:
I20260709 13:01:40.933718  1679 rpcz_store.cc:276] 0709 13:01:39.885619 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:39.887234 (+  1615us) service_pool.cc:224] Handling call
0709 13:01:40.932883 (+1045649us) spinlock_profiling.cc:243] Waited 94.1 ms on lock 0x7b54000d8254. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:01:40.933063 (+   180us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":216438016}
I20260709 13:01:40.937052  1678 raft_consensus.cc:1275] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 15 index: 12. (term mismatch)
I20260709 13:01:40.937695  1678 pending_rounds.cc:85] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d: Aborting all ops after (but not including) 11
I20260709 13:01:40.938078  1678 pending_rounds.cc:107] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d: Aborting uncommitted NO_OP operation due to leader change: 14.12
I20260709 13:01:40.937498  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 14 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:40.938779  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:40.939611  1753 raft_consensus.cc:2468] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 12.
I20260709 13:01:40.939913  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 17 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:40.942055  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:40.942781  2333 raft_consensus.cc:2468] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 14.
I20260709 13:01:40.942878  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:40.943861  1749 raft_consensus.cc:2368] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 14. Current term is 15.
I20260709 13:01:40.910280  7395 raft_consensus.cc:3060] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:01:40.928663  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:40.943989  6882 raft_consensus.cc:493] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:40.929071  7337 leader_election.cc:290] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:40.947440  1677 raft_consensus.cc:2468] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 15.
I20260709 13:01:40.947615  6882 raft_consensus.cc:515] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:40.878293  1674 raft_consensus.cc:1275] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 16 index: 13. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
I20260709 13:01:40.952659  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:40.954593  1678 raft_consensus.cc:2889] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:01:40.954766  7307 raft_consensus.cc:2749] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 13:01:40.955736  1745 raft_consensus.cc:1275] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 13 index: 7. Preceding OpId from leader: term: 15 index: 8. (index mismatch)
I20260709 13:01:40.956719  7377 raft_consensus.cc:493] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:40.957773  6882 leader_election.cc:290] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:40.957546  7377 raft_consensus.cc:515] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:40.959986  1569 leader_election.cc:400] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 16 pre-election: Vote denied by peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261) with higher term. Message: Invalid argument: T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 16. Current term is 17.
I20260709 13:01:40.960708  7377 leader_election.cc:290] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:40.957613  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:40.967193  1754 raft_consensus.cc:1275] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 15 index: 12. (index mismatch)
I20260709 13:01:40.967480  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:40.968492  7451 consensus_queue.cc:1059] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 13:01:40.964140  1569 leader_election.cc:403] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 16 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:01:40.972116  1718 leader_election.cc:304] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
W20260709 13:01:40.973887  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 60297.9us
I20260709 13:01:40.974756  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:40.981546  2298 leader_election.cc:304] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:01:40.981720  7489 raft_consensus.cc:493] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:01:40.989562  2336 raft_consensus.cc:2468] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 17.
I20260709 13:01:40.914242  1672 raft_consensus.cc:2468] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 13.
I20260709 13:01:41.009570  7489 raft_consensus.cc:515] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:40.987887  7181 consensus_queue.cc:1048] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 13:01:41.022189  1671 raft_consensus.cc:1240] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 12. Current term is 13. Ops: []
I20260709 13:01:40.989473  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:40.995863  1596 raft_consensus.cc:2468] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 12.
I20260709 13:01:40.992674  1644 leader_election.cc:304] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:01:41.009804  1717 leader_election.cc:304] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:41.021838  7497 raft_consensus.cc:2804] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:01:41.027302  7497 raft_consensus.cc:493] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:40.991856  7293 consensus_queue.cc:1048] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.036s
I20260709 13:01:41.027866  7497 raft_consensus.cc:3060] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:01:41.029260  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:41.029640  1602 raft_consensus.cc:2468] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 15.
I20260709 13:01:41.010751  7451 consensus_queue.cc:1243] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer 816fadcea8fc4893882f5f9b9f22a03d log is divergent from this leader: its last log entry 17.13 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 12
I20260709 13:01:41.027714  1643 leader_election.cc:304] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 13:01:41.036476  1680 raft_consensus.cc:2468] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 13.
I20260709 13:01:41.037797  7316 raft_consensus.cc:2804] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:01:41.055815  1644 leader_election.cc:304] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:01:41.043252  7316 raft_consensus.cc:493] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:41.062042  1753 raft_consensus.cc:1275] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 14 index: 7. Preceding OpId from leader: term: 16 index: 8. (index mismatch)
I20260709 13:01:41.061097  1497 catalog_manager.cc:5697] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 reported cstate change: term changed from 8 to 10, leader changed from 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194) to 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196). New cstate: current_term: 10 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 13:01:41.066303  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 222312us
I20260709 13:01:41.062575  7200 raft_consensus.cc:2804] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:01:41.062842  7437 raft_consensus.cc:2804] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Leader election won for term 17
I20260709 13:01:41.064538  7451 raft_consensus.cc:2749] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 13:01:41.070227  7200 raft_consensus.cc:493] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
W20260709 13:01:41.093437  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 65992.6us
I20260709 13:01:41.093940  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 16 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:41.095383  1745 raft_consensus.cc:3060] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:01:41.096133  1749 raft_consensus.cc:1275] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
W20260709 13:01:41.100103  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 90520.8us
I20260709 13:01:41.093508  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:41.089359  7437 raft_consensus.cc:697] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:01:41.079691  2331 raft_consensus.cc:1275] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20260709 13:01:41.064309  7316 raft_consensus.cc:3060] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:01:41.095718  7200 raft_consensus.cc:3060] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 10 FOLLOWER]: Advancing to term 11
W20260709 13:01:41.079487  7435 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.576s	user 0.020s	sys 0.015s
I20260709 13:01:41.110900  1569 leader_election.cc:304] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:41.102772  7364 consensus_queue.cc:1059] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 14.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.079s
I20260709 13:01:41.105707  1643 leader_election.cc:400] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 13 election: Vote denied by peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445) with higher term. Message: Invalid argument: T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for earlier term 13. Current term is 14.
I20260709 13:01:41.120877  1747 raft_consensus.cc:3060] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Advancing to term 16
W20260709 13:01:41.118774  7435 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.623s	user 0.020s	sys 0.016s
I20260709 13:01:41.134092  1749 raft_consensus.cc:1275] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 15 index: 10. (index mismatch)
I20260709 13:01:41.135762  7301 raft_consensus.cc:2749] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:01:41.135846  1717 leader_election.cc:304] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:41.139012  7483 raft_consensus.cc:3060] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Advancing to term 17
I20260709 13:01:41.159380  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:01:41.162771  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 63505.5us
W20260709 13:01:41.164117  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 57529.5us
W20260709 13:01:41.164719  1718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:41.171060  7325 raft_consensus.cc:2749] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:01:41.155587  7489 leader_election.cc:290] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:41.175724  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:41.177388  1569 leader_election.cc:304] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:01:41.155910  1497 catalog_manager.cc:5697] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 reported cstate change: term changed from 10 to 12. New cstate: current_term: 12 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 13:01:41.188474  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 60533.4us
I20260709 13:01:41.189874  1750 raft_consensus.cc:3055] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 13:01:41.190330  1750 raft_consensus.cc:740] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:01:41.200006  1745 raft_consensus.cc:2468] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 16.
W20260709 13:01:41.200176  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 218924us
I20260709 13:01:41.200798  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:41.166352  6882 consensus_queue.cc:1048] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.032s
I20260709 13:01:41.204412  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:41.207441  1754 raft_consensus.cc:3060] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:01:41.195264  7436 raft_consensus.cc:493] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:41.214419  1753 raft_consensus.cc:2468] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 16.
W20260709 13:01:41.224531  1501 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35458 (request call id 308) took 34299 ms (client timeout 15000 ms). Trace:
I20260709 13:01:41.226595  7392 consensus_queue.cc:1059] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 13.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 13:01:41.236908  7468 raft_consensus.cc:493] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:41.237725  7468 raft_consensus.cc:515] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } }
I20260709 13:01:41.239667  7413 raft_consensus.cc:2749] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:01:41.240156  7468 leader_election.cc:290] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:41.240834  7436 raft_consensus.cc:515] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:41.243974  1750 consensus_queue.cc:260] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.13, Last appended by leader: 13, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:41.247498  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
W20260709 13:01:41.252367  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 74998.7us
I20260709 13:01:41.253468  1643 leader_election.cc:304] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:01:41.254374  7338 raft_consensus.cc:2749] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:01:41.202378  7437 consensus_queue.cc:237] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:41.225077  1501 rpcz_store.cc:269] 0709 13:01:06.881742 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:06.881963 (+   221us) service_pool.cc:224] Handling call
0709 13:01:41.177167 (+34295204us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 13:01:32.302334 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 13:01:32.303175 (+   841us) write_op.cc:432] Acquiring schema lock in shared mode
0709 13:01:32.303242 (+    67us) write_op.cc:435] Acquired schema lock
0709 13:01:32.303254 (+    12us) tablet.cc:662] Decoding operations
0709 13:01:32.484814 (+181560us) write_op.cc:620] Acquiring the partition lock for write op
0709 13:01:32.497764 (+ 12950us) write_op.cc:641] Partition lock acquired for write op
0709 13:01:32.499549 (+  1785us) tablet.cc:685] Acquiring locks for 30 operations
0709 13:01:33.164223 (+664674us) tablet.cc:701] Row locks acquired
0709 13:01:33.164296 (+    73us) write_op.cc:260] PREPARE: finished
0709 13:01:33.164586 (+   290us) write_op.cc:270] Start()
0709 13:01:33.164824 (+   238us) write_op.cc:276] Timestamp: P: 1783602093164488 usec, L: 0
0709 13:01:33.164852 (+    28us) op_driver.cc:348] REPLICATION: starting
0709 13:01:33.209609 (+ 44757us) log.cc:844] Serialized 11720 byte log entry
0709 13:01:34.607725 (+1398116us) op_driver.cc:464] REPLICATION: finished
0709 13:01:35.243563 (+635838us) write_op.cc:301] APPLY: starting
0709 13:01:35.243684 (+   121us) tablet.cc:1367] starting BulkCheckPresence
0709 13:01:36.836660 (+1592976us) tablet.cc:1370] finished BulkCheckPresence
0709 13:01:36.836791 (+   131us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 13:01:37.991043 (+1154252us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 13:01:37.991599 (+   556us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=60,key_file_lookups=60,delta_file_lookups=60,mrs_lookups=0
0709 13:01:37.991653 (+    54us) write_op.cc:312] APPLY: finished
0709 13:01:37.992386 (+   733us) log.cc:844] Serialized 260 byte log entry
0709 13:01:38.229724 (+237338us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:01:38.380685 (+150961us) write_op.cc:454] Released schema lock
0709 13:01:38.393115 (+ 12430us) write_op.cc:341] FINISH: Updating metrics
0709 13:01:38.463182 (+ 70067us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:01:38.463237 (+    55us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":8615952,"reported_tablets":47,"child_traces":[["op",{"apply.queue_time_us":626478,"apply.run_cpu_time_us":82469,"apply.run_wall_time_us":3219545,"cfile_cache_hit":122,"cfile_cache_hit_bytes":129767,"num_ops":30,"prepare.queue_time_us":592041,"prepare.run_cpu_time_us":31829,"prepare.run_wall_time_us":1155738,"raft.queue_time_us":293537,"raft.run_cpu_time_us":14415,"raft.run_wall_time_us":650451,"replication_time_us":1408002,"spinlock_wait_cycles":23936,"thread_start_us":1419704,"threads_started":5,"wal-append.queue_time_us":562887,"wal-append.run_cpu_time_us":29686,"wal-append.run_wall_time_us":3149231}]]}
I20260709 13:01:41.214458  7459 raft_consensus.cc:2804] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:01:41.264492  7459 raft_consensus.cc:697] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:01:41.246150  1750 raft_consensus.cc:3060] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:01:41.272509  1747 raft_consensus.cc:2468] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 16.
W20260709 13:01:41.281948  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 48256.9us
W20260709 13:01:41.286021  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 84615us
I20260709 13:01:41.278151  7417 raft_consensus.cc:493] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:41.273878  7392 raft_consensus.cc:3055] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 LEADER]: Stepping down as leader of term 15
I20260709 13:01:41.291731  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:41.292214  7417 raft_consensus.cc:515] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:41.292390  7477 consensus_queue.cc:1059] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 13:01:41.278501  7364 raft_consensus.cc:493] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:41.293128  7392 raft_consensus.cc:740] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:01:41.265825  7459 consensus_queue.cc:237] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:41.295979  7435 consensus_queue.cc:1048] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.004s
I20260709 13:01:41.296778  1569 leader_election.cc:304] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:01:41.300612  1745 raft_consensus.cc:1275] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 15 index: 12. (index mismatch)
I20260709 13:01:41.304847  1752 raft_consensus.cc:1275] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
I20260709 13:01:41.305655  1753 raft_consensus.cc:1275] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 15 index: 13. Preceding OpId from leader: term: 17 index: 14. (index mismatch)
I20260709 13:01:41.306783  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:41.306974  7364 raft_consensus.cc:515] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:41.309664  7364 leader_election.cc:290] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:41.311848  7417 leader_election.cc:290] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:41.311465  1497 catalog_manager.cc:5697] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 reported cstate change: term changed from 8 to 11, leader changed from 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) to 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193). New cstate: current_term: 11 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 13:01:41.315753  7383 consensus_queue.cc:1048] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 13:01:41.323939  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:41.328480  1569 leader_election.cc:304] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 13:01:41.329363  7435 consensus_queue.cc:1048] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.005s
I20260709 13:01:41.334110  1717 leader_election.cc:304] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 13:01:41.340894  7364 consensus_queue.cc:1048] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 13:01:41.342684  7395 raft_consensus.cc:2804] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Leader election won for term 13
I20260709 13:01:41.342799  1569 leader_election.cc:304] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:01:41.343369  7395 raft_consensus.cc:697] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:01:41.343510  7337 consensus_queue.cc:1048] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 13:01:41.344547  7395 consensus_queue.cc:237] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:41.345613  7477 raft_consensus.cc:2804] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Leader election won for term 13
I20260709 13:01:41.346189  7477 raft_consensus.cc:697] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:01:41.347627  7477 consensus_queue.cc:237] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.9, Last appended by leader: 9, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:41.350729  1643 leader_election.cc:304] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:01:41.367717  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:41.369997  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:41.370853  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:41.371315  1751 raft_consensus.cc:2393] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 13: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:01:41.372373  7559 consensus_queue.cc:1059] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 12.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 13:01:41.374439  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 17 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:41.375238  7364 raft_consensus.cc:2804] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:01:41.375595  1717 leader_election.cc:304] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:41.375770  7364 raft_consensus.cc:493] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:41.376511  7364 raft_consensus.cc:3060] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Advancing to term 13
W20260709 13:01:41.408659  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 64090.8us
I20260709 13:01:41.321833  7497 raft_consensus.cc:515] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:41.413157  1751 raft_consensus.cc:1275] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 14 index: 10. Preceding OpId from leader: term: 16 index: 11. (index mismatch)
I20260709 13:01:41.424993  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:41.378013  7417 raft_consensus.cc:2804] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:01:41.408391  7569 consensus_queue.cc:1048] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 13:01:41.409992  7383 raft_consensus.cc:493] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:41.431569  1746 raft_consensus.cc:2468] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 13.
I20260709 13:01:41.432723  1717 leader_election.cc:304] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:41.433146  1752 raft_consensus.cc:1275] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 15 index: 9. (index mismatch)
I20260709 13:01:41.433714  1749 raft_consensus.cc:2393] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 16: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:01:41.436190  7497 leader_election.cc:290] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:41.437383  7337 raft_consensus.cc:2804] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:01:41.439368  7200 raft_consensus.cc:515] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:41.328881  7392 consensus_queue.cc:260] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 15.8, Last appended by leader: 8, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:41.440725  1677 raft_consensus.cc:1275] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 16 index: 13. (index mismatch)
W20260709 13:01:41.450767  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 123464us
I20260709 13:01:41.314152  1497 catalog_manager.cc:5697] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 reported cstate change: term changed from 9 to 10, leader changed from af520b67945d443b901bbeca37cc5701 (127.0.107.195) to 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196). New cstate: current_term: 10 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:01:41.457050  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:41.458983  2328 raft_consensus.cc:3060] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Advancing to term 16
I20260709 13:01:41.434037  7417 raft_consensus.cc:493] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:41.430347  7436 leader_election.cc:290] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:41.461503  7383 raft_consensus.cc:515] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:41.455317  7337 raft_consensus.cc:493] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:41.488744  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:41.497524  1754 raft_consensus.cc:2468] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 13.
I20260709 13:01:41.472640  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:01:41.471369  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 120845us
I20260709 13:01:41.500484  1747 raft_consensus.cc:1240] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 12. Current term is 13. Ops: [12.7-12.7]
W20260709 13:01:41.502785  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:41.503688  1750 raft_consensus.cc:1275] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 16 index: 13. Preceding OpId from leader: term: 17 index: 13. (term mismatch)
I20260709 13:01:41.504192  1750 pending_rounds.cc:85] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701: Aborting all ops after (but not including) 12
I20260709 13:01:41.504487  1750 pending_rounds.cc:107] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701: Aborting uncommitted NO_OP operation due to leader change: 16.13
I20260709 13:01:41.504750  1750 raft_consensus.cc:2889] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:01:41.466207  7540 raft_consensus.cc:2764] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Leader pre-election decision vote started in defunct term 15: won
I20260709 13:01:41.461779  7307 raft_consensus.cc:2804] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:01:41.430052  6882 consensus_queue.cc:1048] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.063s
I20260709 13:01:41.472155  7392 raft_consensus.cc:3060] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:01:41.476943  7483 raft_consensus.cc:2749] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Leader pre-election lost for term 16. Reason: Vote denied by peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261) with higher term. Message: Invalid argument: T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 16. Current term is 17.
I20260709 13:01:41.463338  7417 raft_consensus.cc:3060] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:01:41.526118  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:41.532840  1671 raft_consensus.cc:1275] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 11 index: 9. Preceding OpId from leader: term: 15 index: 11. (index mismatch)
W20260709 13:01:41.551959  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 124822us
I20260709 13:01:41.528424  1672 raft_consensus.cc:2468] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 9.
W20260709 13:01:41.560918  1718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:41.482698  7347 raft_consensus.cc:493] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:01:41.468297  7383 leader_election.cc:290] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:41.474292  7200 leader_election.cc:290] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 11 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:41.459784  7301 raft_consensus.cc:3055] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:01:41.506021  7337 raft_consensus.cc:3060] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Advancing to term 13
W20260709 13:01:41.579312  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 123479us
I20260709 13:01:41.517482  7497 raft_consensus.cc:493] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:41.526018  7307 raft_consensus.cc:493] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:41.517441  7559 consensus_queue.cc:1048] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.067s
W20260709 13:01:41.555939  7491 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 17 current term: 17 is higher.
I20260709 13:01:41.470418  7316 raft_consensus.cc:515] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:41.583467  1677 raft_consensus.cc:2468] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 13.
W20260709 13:01:41.584017  7554 log.cc:927] Time spent T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.290s	user 0.002s	sys 0.000s
I20260709 13:01:41.604712  7540 raft_consensus.cc:493] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:01:41.651106  7564 log.cc:927] Time spent T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.219s	user 0.001s	sys 0.000s
I20260709 13:01:41.650966  7540 raft_consensus.cc:515] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:41.651477  7316 leader_election.cc:290] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 16 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:41.645845  1646 leader_election.cc:304] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
W20260709 13:01:41.654790  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 164340us
I20260709 13:01:41.655892  7200 raft_consensus.cc:2804] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Leader election won for term 16
I20260709 13:01:41.655992  1646 leader_election.cc:304] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 13:01:41.656574  7200 raft_consensus.cc:697] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 16 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:01:41.657763  1646 leader_election.cc:304] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:41.657634  7200 consensus_queue.cc:237] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 13.7, Last appended by leader: 8, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:01:41.646528  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 99911.5us
W20260709 13:01:41.658979  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 65383.9us
W20260709 13:01:41.649971  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 58436.6us
I20260709 13:01:41.657831  7437 raft_consensus.cc:2804] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:01:41.647090  7497 raft_consensus.cc:515] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:41.583525  7301 raft_consensus.cc:740] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:01:41.662686  2328 raft_consensus.cc:2468] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 16.
I20260709 13:01:41.660599  7540 leader_election.cc:290] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:41.664211  1718 leader_election.cc:304] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:41.667197  1717 leader_election.cc:304] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 13:01:41.592626  7307 raft_consensus.cc:3060] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:01:41.668825  1646 leader_election.cc:304] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:41.677345  7301 consensus_queue.cc:260] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 12.7, Last appended by leader: 7, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:41.678653  7301 raft_consensus.cc:3060] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:01:41.679589  7565 consensus_queue.cc:1048] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 13:01:41.682395  1569 leader_election.cc:304] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:01:41.665127  7559 consensus_queue.cc:1048] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.038s
I20260709 13:01:41.483698  1497 catalog_manager.cc:5697] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 reported cstate change: term changed from 9 to 10, leader changed from 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) to 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196). New cstate: current_term: 10 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:01:41.661285  7437 raft_consensus.cc:493] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:41.710703  1571 leader_election.cc:304] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:01:41.709952  1569 leader_election.cc:304] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
W20260709 13:01:41.711356  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 47198.7us
I20260709 13:01:41.658944  7347 raft_consensus.cc:515] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:41.684216  7540 raft_consensus.cc:2804] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Leader election won for term 16
W20260709 13:01:41.719094  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 57538.6us
W20260709 13:01:41.723042  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:41.712966  7459 raft_consensus.cc:2804] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:01:41.712150  7437 raft_consensus.cc:3060] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:01:41.726271  1747 raft_consensus.cc:1275] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 16 index: 13. (index mismatch)
I20260709 13:01:41.727144  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:41.725968  7383 raft_consensus.cc:2804] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:01:41.717358  7567 raft_consensus.cc:2804] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:01:41.730283  1678 raft_consensus.cc:1275] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 15 index: 9. (index mismatch)
I20260709 13:01:41.729286  7383 raft_consensus.cc:493] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:41.718254  6882 raft_consensus.cc:2804] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Leader election won for term 16
I20260709 13:01:41.735136  1745 raft_consensus.cc:1275] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20260709 13:01:41.740375  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 16 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:41.734815  7575 consensus_queue.cc:1048] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.065s
I20260709 13:01:41.729640  7459 raft_consensus.cc:493] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:41.731887  1717 leader_election.cc:304] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
W20260709 13:01:41.736286  7562 log.cc:927] Time spent T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.317s	user 0.003s	sys 0.010s
I20260709 13:01:41.719980  7316 raft_consensus.cc:2804] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Leader election won for term 13
I20260709 13:01:41.741880  7568 raft_consensus.cc:2749] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:01:41.742233  1672 raft_consensus.cc:1275] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 16 index: 13. Preceding OpId from leader: term: 17 index: 14. (index mismatch)
I20260709 13:01:41.741040  1749 raft_consensus.cc:2468] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 15.
I20260709 13:01:41.725356  7540 raft_consensus.cc:697] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 16 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:01:41.740115  7383 raft_consensus.cc:3060] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Advancing to term 20
W20260709 13:01:41.790674  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 85223.3us
I20260709 13:01:41.732198  7567 raft_consensus.cc:493] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:41.724129  7435 consensus_queue.cc:1048] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.031s
I20260709 13:01:41.756269  7459 raft_consensus.cc:3060] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Advancing to term 11
W20260709 13:01:41.805614  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 143061us
I20260709 13:01:41.796418  7567 raft_consensus.cc:3060] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:01:41.804770  7325 raft_consensus.cc:493] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:41.888783  7347 leader_election.cc:290] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:41.896108  2298 leader_election.cc:304] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
W20260709 13:01:41.896569  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 237448us
I20260709 13:01:41.898665  7587 raft_consensus.cc:2804] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 13:01:41.899396  7587 raft_consensus.cc:697] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 16 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:01:41.900566  7587 consensus_queue.cc:237] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.8, Last appended by leader: 8, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:41.898187  7325 raft_consensus.cc:515] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:01:41.903050  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 106068us
W20260709 13:01:41.921659  7563 log.cc:927] Time spent T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.520s	user 0.010s	sys 0.014s
I20260709 13:01:41.889721  7477 raft_consensus.cc:493] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:41.927650  7477 raft_consensus.cc:515] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:41.926857  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 195048us
I20260709 13:01:41.932770  7581 raft_consensus.cc:493] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:41.933440  7581 raft_consensus.cc:515] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:41.934666  7540 consensus_queue.cc:237] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.12, Last appended by leader: 12, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:41.933032  1717 leader_election.cc:304] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:41.886132  7560 consensus_queue.cc:1048] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.210s
I20260709 13:01:41.953737  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:41.957683  1671 raft_consensus.cc:2468] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 13.
I20260709 13:01:41.967922  1678 raft_consensus.cc:1240] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 12. Current term is 13. Ops: []
I20260709 13:01:41.942626  7575 raft_consensus.cc:493] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:01:41.969897  7468 raft_consensus.cc:493] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:01:41.970028  7575 raft_consensus.cc:515] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:41.970593  7468 raft_consensus.cc:515] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:41.972616  7575 leader_election.cc:290] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:41.972867  7468 leader_election.cc:290] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:41.974704  1643 leader_election.cc:304] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:01:41.974998  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:41.977226  7590 raft_consensus.cc:493] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:41.978025  7590 raft_consensus.cc:515] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:41.965767  7301 consensus_queue.cc:1048] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.013s
I20260709 13:01:41.953284  7497 leader_election.cc:290] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:41.981490  7497 consensus_queue.cc:1048] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 13:01:41.982929  7497 raft_consensus.cc:3060] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:01:41.983969  7325 leader_election.cc:290] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:41.957685  7569 raft_consensus.cc:2749] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
W20260709 13:01:41.986940  7573 log.cc:927] Time spent T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.260s	user 0.001s	sys 0.000s
W20260709 13:01:41.988075  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 90237.1us
I20260709 13:01:41.988188  7582 consensus_queue.cc:1048] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 13:01:41.959664  7392 consensus_queue.cc:1048] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.031s
I20260709 13:01:41.988528  1497 catalog_manager.cc:5697] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 reported cstate change: term changed from 9 to 12. New cstate: current_term: 12 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:01:41.993767  1571 leader_election.cc:304] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:01:41.995419  1571 leader_election.cc:304] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 13:01:41.996232  1497 catalog_manager.cc:5697] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 reported cstate change: term changed from 7 to 11, leader changed from 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194) to 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196). New cstate: current_term: 11 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:01:41.999001  7483 raft_consensus.cc:2749] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:01:41.999678  1679 raft_consensus.cc:1275] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 13 index: 12. (index mismatch)
I20260709 13:01:41.999876  7483 raft_consensus.cc:2804] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:01:42.000396  7483 raft_consensus.cc:493] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:42.000773  7483 raft_consensus.cc:3060] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:01:42.001366  1569 leader_election.cc:304] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:01:42.002452  7392 raft_consensus.cc:2804] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:01:42.002847  7392 raft_consensus.cc:493] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Starting leader election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:01:42.003188  7392 raft_consensus.cc:3060] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:01:42.003661  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:42.004558  1671 raft_consensus.cc:2393] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 17: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
W20260709 13:01:42.007246  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 200743us
I20260709 13:01:42.009503  7496 raft_consensus.cc:3060] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:01:42.011439  7570 raft_consensus.cc:3055] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:01:42.011868  7570 raft_consensus.cc:740] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:01:42.012866  7570 consensus_queue.cc:260] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:41.930361  7477 leader_election.cc:290] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:42.028618  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:42.029389  1571 leader_election.cc:304] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:01:42.029546  2328 raft_consensus.cc:2468] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 10.
W20260709 13:01:42.022302  7364 consensus_meta.cc:287] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.613s	user 0.014s	sys 0.000s
I20260709 13:01:42.014021  7570 raft_consensus.cc:3060] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:01:42.038906  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:42.045249  1676 raft_consensus.cc:1275] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 12 index: 9. Preceding OpId from leader: term: 13 index: 10. (index mismatch)
I20260709 13:01:42.038982  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:42.022936  7590 leader_election.cc:290] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:01:42.071811  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:42.017179  7200 consensus_queue.cc:1048] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.043s
I20260709 13:01:42.029366  7307 raft_consensus.cc:515] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:01:42.100287  7417 consensus_meta.cc:287] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.515s	user 0.010s	sys 0.003s
W20260709 13:01:42.100772  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 167839us
I20260709 13:01:42.069074  7477 raft_consensus.cc:2749] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 13:01:42.115288  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:01:42.116050  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 101410us
I20260709 13:01:42.098667  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:01:42.122337  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 146116us
I20260709 13:01:41.998226  1497 catalog_manager.cc:5697] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 reported cstate change: term changed from 7 to 9. New cstate: current_term: 9 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:01:42.048501  7364 raft_consensus.cc:515] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:42.130641  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:42.119779  1678 raft_consensus.cc:2468] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 12.
I20260709 13:01:42.059568  7437 raft_consensus.cc:515] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:42.085709  7316 raft_consensus.cc:697] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
W20260709 13:01:42.142122  1718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:42.137113  1750 raft_consensus.cc:1240] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 16. Current term is 17. Ops: [16.8-16.8]
I20260709 13:01:42.144965  1571 leader_election.cc:304] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:01:42.080091  7459 raft_consensus.cc:515] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:42.155273  1680 raft_consensus.cc:1240] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 16. Current term is 17. Ops: []
I20260709 13:01:42.158975  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:42.160233  2336 raft_consensus.cc:2393] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 16: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:01:42.161306  7437 leader_election.cc:290] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 17 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:42.162499  1597 raft_consensus.cc:1240] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 LEADER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 12. Current term is 13. Ops: []
I20260709 13:01:42.164273  7567 raft_consensus.cc:515] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } }
I20260709 13:01:42.166970  1569 leader_election.cc:304] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:42.170691  7347 raft_consensus.cc:2749] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:01:42.145126  7301 raft_consensus.cc:493] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:42.176126  1717 leader_election.cc:304] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:42.127156  7417 raft_consensus.cc:515] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:42.158912  6882 raft_consensus.cc:697] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:01:42.218317  2331 raft_consensus.cc:1275] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20260709 13:01:42.265787  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:42.177096  7477 raft_consensus.cc:2749] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
W20260709 13:01:42.273427  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 54340.8us
W20260709 13:01:42.274765  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 59439.7us
I20260709 13:01:42.214018  7200 raft_consensus.cc:493] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:01:42.282829  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:42.283620  1678 raft_consensus.cc:3060] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:01:42.161832  1497 catalog_manager.cc:5697] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 reported cstate change: term changed from 8 to 10. New cstate: current_term: 10 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:01:42.214612  7497 raft_consensus.cc:2749] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:01:42.276520  1646 leader_election.cc:304] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:42.239634  7581 leader_election.cc:290] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:42.266659  7567 leader_election.cc:290] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 14 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:42.218272  7596 raft_consensus.cc:2749] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20260709 13:01:42.314690  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 161525us
W20260709 13:01:42.320295  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 43924.5us
I20260709 13:01:42.240252  7559 consensus_queue.cc:1048] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.045s
W20260709 13:01:42.324573  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 189260us
I20260709 13:01:42.240166  7383 raft_consensus.cc:515] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:42.218116  7301 raft_consensus.cc:515] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:42.323868  7337 consensus_meta.cc:287] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.662s	user 0.005s	sys 0.009s
I20260709 13:01:42.164287  7316 consensus_queue.cc:237] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:42.342815  7578 consensus_queue.cc:1048] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.022s
I20260709 13:01:42.343178  7417 leader_election.cc:290] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 14 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:42.343729  7579 consensus_queue.cc:1048] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.101s
W20260709 13:01:42.345520  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:42.344396  1717 leader_election.cc:304] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:42.321094  7395 consensus_queue.cc:1048] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.027s
I20260709 13:01:42.288131  7200 raft_consensus.cc:515] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:42.344345  7577 raft_consensus.cc:493] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:01:42.379547  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 52071.1us
W20260709 13:01:42.383386  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 259098us
I20260709 13:01:42.390070  2299 leader_election.cc:304] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
W20260709 13:01:42.391108  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 73120.7us
W20260709 13:01:42.392428  7607 log.cc:927] Time spent T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.113s	user 0.001s	sys 0.000s
W20260709 13:01:42.393842  7593 log.cc:927] Time spent T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.096s	user 0.000s	sys 0.000s
I20260709 13:01:42.378657  7417 raft_consensus.cc:493] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:42.394794  7417 raft_consensus.cc:515] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:42.398778  1571 leader_election.cc:400] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 14 election: Vote denied by peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693) with higher term. Message: Invalid argument: T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 14. Current term is 15.
I20260709 13:01:42.399160  1571 leader_election.cc:403] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 14 election: Cancelling election due to peer responding with higher term
I20260709 13:01:42.397182  7417 leader_election.cc:290] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:42.351649  7337 raft_consensus.cc:515] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:01:42.404366  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 60121.2us
I20260709 13:01:42.409684  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:42.380923  7307 leader_election.cc:290] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 17 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:42.380514  7596 raft_consensus.cc:2749] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:01:42.437938  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:42.441493  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:42.441761  1672 raft_consensus.cc:1275] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 16 index: 14. Preceding OpId from leader: term: 17 index: 14. (term mismatch)
I20260709 13:01:42.442373  1672 pending_rounds.cc:85] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d: Aborting all ops after (but not including) 13
I20260709 13:01:42.442721  1672 pending_rounds.cc:107] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d: Aborting uncommitted NO_OP operation due to leader change: 16.14
I20260709 13:01:42.443087  1672 raft_consensus.cc:2889] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:01:42.402311  7364 leader_election.cc:290] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 13 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:42.417768  7437 raft_consensus.cc:2804] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:01:42.405412  7580 raft_consensus.cc:2804] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:01:42.412705  7200 leader_election.cc:290] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:42.427426  7497 consensus_queue.cc:1048] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.054s
I20260709 13:01:42.432751  7468 raft_consensus.cc:2749] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader election lost for term 14. Reason: Vote denied by peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693) with higher term. Message: Invalid argument: T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 14. Current term is 15.
I20260709 13:01:42.435787  7491 raft_consensus.cc:493] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:42.472983  1717 leader_election.cc:304] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:42.471510  7459 leader_election.cc:290] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:42.410533  7577 raft_consensus.cc:515] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:42.474119  7580 raft_consensus.cc:493] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Starting leader election (detected failure of leader af520b67945d443b901bbeca37cc5701)
W20260709 13:01:42.524948  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 141108us
I20260709 13:01:42.526674  7591 raft_consensus.cc:493] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:42.527181  7615 raft_consensus.cc:493] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:42.531503  7497 raft_consensus.cc:2804] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Leader pre-election won for term 18
W20260709 13:01:42.519757  7392 consensus_meta.cc:287] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.516s	user 0.010s	sys 0.020s
I20260709 13:01:42.405148  6882 consensus_queue.cc:237] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.10, Last appended by leader: 10, Current term: 16, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:42.542008  1678 raft_consensus.cc:2468] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 14.
I20260709 13:01:42.520459  7364 raft_consensus.cc:493] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:42.536988  7580 raft_consensus.cc:3060] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:01:42.546816  1571 leader_election.cc:304] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:01:42.548658  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:42.551566  7337 leader_election.cc:290] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 13 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:01:42.556396  7597 log.cc:927] Time spent T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.395s	user 0.002s	sys 0.000s
I20260709 13:01:42.556123  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:42.523270  7491 raft_consensus.cc:515] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:42.573206  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:42.574050  1678 raft_consensus.cc:2393] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 16: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:01:42.545917  7605 raft_consensus.cc:493] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:42.546623  7497 raft_consensus.cc:493] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:42.567224  7392 raft_consensus.cc:515] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:01:42.584573  7608 log.cc:927] Time spent T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.165s	user 0.002s	sys 0.000s
I20260709 13:01:42.585088  1678 raft_consensus.cc:1240] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 16. Current term is 17. Ops: []
I20260709 13:01:42.569160  7591 raft_consensus.cc:515] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:42.592355  1679 raft_consensus.cc:2468] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 16.
I20260709 13:01:42.637595  1672 raft_consensus.cc:1240] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 11. Current term is 12. Ops: []
W20260709 13:01:42.581485  7483 consensus_meta.cc:287] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.580s	user 0.016s	sys 0.000s
I20260709 13:01:42.640081  1677 raft_consensus.cc:1240] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 LEADER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 12. Current term is 13. Ops: [12.7-12.7]
I20260709 13:01:42.581678  7468 raft_consensus.cc:2749] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:01:42.581177  7301 leader_election.cc:290] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:42.545796  7615 raft_consensus.cc:515] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:42.598313  7497 raft_consensus.cc:3060] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:01:42.572944  7364 raft_consensus.cc:515] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:42.647601  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:01:42.650761  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 114454us
I20260709 13:01:42.651964  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:42.655658  7616 raft_consensus.cc:493] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:42.656577  7616 raft_consensus.cc:515] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:42.661695  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:42.662580  1678 raft_consensus.cc:2393] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 16: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:01:42.663826  7616 leader_election.cc:290] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:42.669314  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 18 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:01:42.668198  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 218035us
W20260709 13:01:42.680347  7611 log.cc:927] Time spent T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.153s	user 0.000s	sys 0.001s
W20260709 13:01:42.681995  7603 log.cc:927] Time spent T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.127s	user 0.000s	sys 0.001s
I20260709 13:01:42.683100  1571 leader_election.cc:304] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:01:42.684085  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:42.684849  1571 leader_election.cc:304] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:01:42.685165  7392 leader_election.cc:290] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 14 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:42.702684  7625 raft_consensus.cc:493] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:42.685879  7392 raft_consensus.cc:2749] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:01:42.685817  7468 raft_consensus.cc:2749] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:01:42.642638  7483 raft_consensus.cc:515] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:42.721455  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:42.644572  7605 raft_consensus.cc:515] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:42.735404  7591 leader_election.cc:290] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:42.736850  7591 raft_consensus.cc:493] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:42.737396  7591 raft_consensus.cc:515] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:42.703490  7625 raft_consensus.cc:515] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:42.738603  7624 raft_consensus.cc:493] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:42.739243  7624 raft_consensus.cc:515] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:42.739853  7605 leader_election.cc:290] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:42.740294  7625 leader_election.cc:290] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:42.740877  7577 leader_election.cc:290] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:42.729647  7587 raft_consensus.cc:493] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:42.741566  7624 leader_election.cc:290] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:42.741797  7364 leader_election.cc:290] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:42.748179  7437 raft_consensus.cc:493] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:01:42.748719  7437 raft_consensus.cc:3060] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:01:42.748973  7615 leader_election.cc:290] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:42.750833  7483 leader_election.cc:290] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 15 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:01:42.726854  1718 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.107.194:46261 timed out after 1.787s (SENT) [suppressed 20 similar messages]
I20260709 13:01:42.751992  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:42.758234  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:01:42.861441  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:42.852087  1673 raft_consensus.cc:1240] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 12. Current term is 13. Ops: [12.7-12.7]
I20260709 13:01:42.788332  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:42.922050  1754 raft_consensus.cc:3060] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:01:42.826074  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 17 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:42.780123  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:01:42.927184  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 208158us
I20260709 13:01:42.759238  1601 raft_consensus.cc:1240] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 14. Current term is 15. Ops: []
I20260709 13:01:42.810084  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:01:42.779984  7570 consensus_meta.cc:287] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.731s	user 0.010s	sys 0.004s
I20260709 13:01:42.938163  1746 raft_consensus.cc:2468] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 13.
W20260709 13:01:42.749011  7496 consensus_meta.cc:287] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.739s	user 0.014s	sys 0.000s
I20260709 13:01:42.950630  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
W20260709 13:01:42.958537  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 179821us
I20260709 13:01:42.856612  7591 leader_election.cc:290] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:42.967023  2337 raft_consensus.cc:1275] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 13 index: 12. (index mismatch)
I20260709 13:01:42.968400  1671 raft_consensus.cc:2468] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 16.
I20260709 13:01:42.970968  1571 leader_election.cc:304] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:01:42.855475  7497 raft_consensus.cc:515] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:42.948681  7582 raft_consensus.cc:493] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:42.978574  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:01:42.983726  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 256848us
I20260709 13:01:42.956717  7596 raft_consensus.cc:493] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:01:42.982568  1643 leader_election.cc:336] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 1.534s (SENT)
I20260709 13:01:42.988857  1643 leader_election.cc:304] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:42.741832  7587 raft_consensus.cc:515] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:42.967027  7496 raft_consensus.cc:2749] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:01:43.005033  1754 raft_consensus.cc:2468] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 17.
I20260709 13:01:42.964707  7594 consensus_queue.cc:1048] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.031s
I20260709 13:01:43.010970  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:01:43.020464  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 44808.1us
I20260709 13:01:43.021821  1671 raft_consensus.cc:1240] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 11. Current term is 12. Ops: []
I20260709 13:01:42.993376  7577 raft_consensus.cc:2749] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:01:43.026544  7587 leader_election.cc:290] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:42.998783  7596 raft_consensus.cc:515] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:43.030756  7497 leader_election.cc:290] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 18 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 13:01:43.031679  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 60808.5us
I20260709 13:01:43.032428  7596 leader_election.cc:290] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:43.032972  1571 leader_election.cc:304] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:01:43.034202  7577 raft_consensus.cc:2749] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 13:01:43.035063  7630 raft_consensus.cc:493] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:43.035781  1571 leader_election.cc:304] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:01:43.035867  7630 raft_consensus.cc:515] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:43.037312  7577 raft_consensus.cc:2749] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:01:42.974337  7483 consensus_queue.cc:1048] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.035s
I20260709 13:01:43.007997  7600 raft_consensus.cc:2749] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:01:43.052142  7630 leader_election.cc:290] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:01:43.057593  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 75113.6us
I20260709 13:01:43.003240  7582 raft_consensus.cc:515] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:43.071349  1646 leader_election.cc:304] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:01:43.038261  7392 consensus_queue.cc:1048] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.024s
I20260709 13:01:43.078606  1753 raft_consensus.cc:1240] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 16. Current term is 17. Ops: []
I20260709 13:01:43.082732  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:43.083765  1598 raft_consensus.cc:2393] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 16: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:01:43.044787  7577 consensus_queue.cc:1048] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.033s
I20260709 13:01:43.099103  1753 raft_consensus.cc:1240] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 12. Current term is 13. Ops: []
I20260709 13:01:43.098904  1718 leader_election.cc:304] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:43.100907  1646 leader_election.cc:304] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:43.104571  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:43.105310  1605 raft_consensus.cc:3060] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Advancing to term 16
W20260709 13:01:43.106261  7588 log.cc:927] Time spent T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 1.119s	user 0.000s	sys 0.008s
I20260709 13:01:43.107857  1748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43156 (request call id 1198) took 1378 ms. Trace:
I20260709 13:01:43.108222  1748 rpcz_store.cc:276] 0709 13:01:41.730625 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:41.736767 (+  6142us) service_pool.cc:224] Handling call
0709 13:01:41.737168 (+   401us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:41.762716 (+ 25548us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:41.762767 (+    51us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:01:41.763559 (+   792us) log.cc:844] Serialized 26 byte log entry
0709 13:01:41.903247 (+139688us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:01:41.903321 (+    74us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:41.903428 (+   107us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:43.051805 (+1148377us) spinlock_profiling.cc:243] Waited 26.3 ms on lock 0x7b4800121ed8. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426b08baa1 00007f426b082b37 00007f426b0829ca 00007f426b17a26f 00007f426b179f37 00007f426b18e55c 00007f426b18e508 00007f426b18e488 00007f426b18e443 00007f426b18d6ec 00007f426a0b7a96
0709 13:01:43.105549 (+ 53744us) spinlock_profiling.cc:243] Waited 5.45 ms on lock 0x7b4400022b4c. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f4262f9bc85 00007f4262f9c6a2 00007f4265d0e1db 00007f4265d09aaf 00007f4265cfd86e 00007f4265cfcbb6 00007f4265cfc7d8 00007f4265d06221 00007f4265d061d9 00007f4265d06169
0709 13:01:43.107507 (+  1958us) raft_consensus.cc:1685] finished
0709 13:01:43.107529 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:43.107652 (+   123us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":9381,"raft.queue_time_us":24021,"raft.run_cpu_time_us":10328,"raft.run_wall_time_us":1290466,"spinlock_wait_cycles":73107584,"thread_start_us":163931,"threads_started":2,"wal-append.queue_time_us":184881}
I20260709 13:01:43.110780  7584 consensus_queue.cc:1059] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 11.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 13:01:43.111485  7587 consensus_queue.cc:1059] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 16.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 13:01:43.112043  7584 raft_consensus.cc:3055] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 13:01:43.112475  7584 raft_consensus.cc:740] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:01:43.112594  7587 raft_consensus.cc:3055] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 13:01:43.113051  7587 raft_consensus.cc:740] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:01:43.113611  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:43.114713  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:43.101912  7468 raft_consensus.cc:493] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:43.115347  1600 raft_consensus.cc:2376] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 16. Re-sending same reply.
I20260709 13:01:43.115520  7468 raft_consensus.cc:515] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:43.115893  2298 leader_election.cc:304] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:01:43.117448  7619 raft_consensus.cc:2749] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:01:43.118031  7468 leader_election.cc:290] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:01:43.118208  7595 log.cc:927] Time spent T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.961s	user 0.003s	sys 0.005s
I20260709 13:01:43.119994  1718 leader_election.cc:304] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:43.120774  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:43.121641  7364 raft_consensus.cc:2749] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:01:43.121996  1718 leader_election.cc:304] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:43.125279  1600 raft_consensus.cc:1275] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 15 index: 12. (index mismatch)
I20260709 13:01:43.122537  7364 raft_consensus.cc:2749] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:01:43.126060  7307 raft_consensus.cc:2804] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:01:43.126570  7307 raft_consensus.cc:493] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:43.110737  1748 raft_consensus.cc:1240] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 19. Current term is 20. Ops: []
I20260709 13:01:43.122658  7497 consensus_queue.cc:1059] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 10.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.002s
I20260709 13:01:43.122761  7383 leader_election.cc:290] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 20 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:43.113996  7587 consensus_queue.cc:260] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.9, Last appended by leader: 9, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:43.130244  1646 leader_election.cc:304] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 13:01:43.134467  1680 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57092 (request call id 1322) took 1095 ms. Trace:
I20260709 13:01:43.095101  7600 raft_consensus.cc:2749] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 13:01:43.135051  1680 rpcz_store.cc:276] 0709 13:01:42.034702 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:42.046360 (+ 11658us) service_pool.cc:224] Handling call
0709 13:01:42.070371 (+ 24011us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:42.155820 (+ 85449us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:43.130173 (+974353us) spinlock_profiling.cc:243] Waited 84.6 ms on lock 0x7b54001f5c54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:01:43.130451 (+   278us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":194616576}
I20260709 13:01:43.109997  7437 raft_consensus.cc:515] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:43.129160  7307 raft_consensus.cc:3060] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:01:43.147444  1605 raft_consensus.cc:2468] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 16.
I20260709 13:01:43.148707  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
W20260709 13:01:43.152218  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 93462.1us
I20260709 13:01:43.150556  1753 raft_consensus.cc:1275] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 11 index: 9. Preceding OpId from leader: term: 15 index: 11. (index mismatch)
I20260709 13:01:43.152695  1603 raft_consensus.cc:3060] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:01:43.163530  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:43.164325  1600 raft_consensus.cc:2468] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 11.
I20260709 13:01:43.165343  7587 raft_consensus.cc:3060] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:01:43.153920  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 16 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:43.113626  7584 consensus_queue.cc:260] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:43.190270  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:43.082162  7580 raft_consensus.cc:515] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:43.167464  1598 raft_consensus.cc:2410] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for term 16 because replica has last-logged OpId of term: 15 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 7.
I20260709 13:01:43.192616  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 18 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:43.199728  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:43.191253  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:43.201989  1679 raft_consensus.cc:2468] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 17.
I20260709 13:01:43.195376  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:43.203174  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:43.203573  1605 raft_consensus.cc:3060] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:01:43.206271  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:43.206666  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:43.206774  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:01:43.207334  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 77163.1us
I20260709 13:01:43.207840  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:43.137943  7586 consensus_queue.cc:1048] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.057s
I20260709 13:01:43.167121  7383 raft_consensus.cc:3055] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:01:43.206350  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:43.212134  1750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55848 (request call id 2902) took 1491 ms. Trace:
W20260709 13:01:43.210649  1643 leader_election.cc:336] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 16 election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 1.866s (SENT)
I20260709 13:01:43.207499  1679 raft_consensus.cc:2468] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 16.
I20260709 13:01:43.170358  7337 raft_consensus.cc:2804] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:01:43.201189  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:43.215124  1680 raft_consensus.cc:1275] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 15 index: 8. Preceding OpId from leader: term: 16 index: 9. (index mismatch)
I20260709 13:01:43.217829  1673 raft_consensus.cc:1275] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 16 index: 13. (index mismatch)
I20260709 13:01:43.218761  2333 raft_consensus.cc:2393] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 13: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:01:43.221464  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:43.216172  1750 rpcz_store.cc:276] 0709 13:01:41.720870 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:41.721063 (+   193us) service_pool.cc:224] Handling call
0709 13:01:41.730586 (+  9523us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:42.143572 (+412986us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:43.211803 (+1068231us) spinlock_profiling.cc:243] Waited 406 ms on lock 0x7b54001711d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:01:43.212043 (+   240us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":933984128}
I20260709 13:01:43.215871  7383 raft_consensus.cc:740] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:01:43.214818  7622 raft_consensus.cc:2804] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Leader election won for term 17
I20260709 13:01:43.225255  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 16 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:43.226668  1596 raft_consensus.cc:1240] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 13. Current term is 14. Ops: [13.9-13.9]
I20260709 13:01:43.243142  1597 raft_consensus.cc:2393] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 14: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:01:43.247596  1643 leader_election.cc:304] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:01:43.257624  1605 raft_consensus.cc:2468] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 17.
I20260709 13:01:43.259174  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:43.248804  7468 raft_consensus.cc:493] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:43.263586  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:43.271154  1599 raft_consensus.cc:2468] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 17.
I20260709 13:01:43.227861  7337 raft_consensus.cc:493] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:43.273437  1602 raft_consensus.cc:1275] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20260709 13:01:43.262462  7468 raft_consensus.cc:515] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:43.251979  7622 raft_consensus.cc:697] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:01:43.276823  7337 raft_consensus.cc:3060] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:01:43.284178   431 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:01:43.262486  7383 consensus_queue.cc:260] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 12.7, Last appended by leader: 7, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:43.301030  7383 raft_consensus.cc:3060] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:01:43.278469  7338 raft_consensus.cc:2749] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
W20260709 13:01:43.311129  7636 log.cc:927] Time spent T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.156s	user 0.000s	sys 0.001s
I20260709 13:01:43.313907  7307 raft_consensus.cc:515] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:43.318053  1601 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 1452) took 1299 ms. Trace:
I20260709 13:01:43.318590  1601 rpcz_store.cc:276] 0709 13:01:42.018791 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:42.024582 (+  5791us) service_pool.cc:224] Handling call
0709 13:01:42.038075 (+ 13493us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:42.938559 (+900484us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:43.317771 (+379212us) spinlock_profiling.cc:243] Waited 717 ms on lock 0x7b54001009d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:01:43.317980 (+   209us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1648022912}
I20260709 13:01:43.323448  7489 consensus_queue.cc:1059] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 14.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 13:01:43.324664  7489 raft_consensus.cc:3055] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:01:43.325058  7489 raft_consensus.cc:740] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:01:43.325369  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:01:43.325827  1719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:43.326134  1602 raft_consensus.cc:2468] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 16.
I20260709 13:01:43.339362  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:43.343743  1602 raft_consensus.cc:2393] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 15: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:01:43.308797  1603 raft_consensus.cc:2468] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 16.
W20260709 13:01:43.310365  7635 log.cc:927] Time spent T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.109s	user 0.000s	sys 0.001s
I20260709 13:01:43.327625  7600 consensus_queue.cc:1059] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 14.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 13:01:43.320122  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:43.331717  7489 consensus_queue.cc:260] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:43.396090  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 18 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:43.332949  2337 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49082 (request call id 1086) took 1117 ms. Trace:
I20260709 13:01:43.333868  1677 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52766 (request call id 2882) took 1650 ms. Trace:
I20260709 13:01:43.334160  7570 consensus_queue.cc:1059] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 16.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 13:01:43.401500  1677 rpcz_store.cc:276] 0709 13:01:41.683921 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:41.709853 (+ 25932us) service_pool.cc:224] Handling call
0709 13:01:41.721142 (+ 11289us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:42.640613 (+919471us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:43.333602 (+692989us) spinlock_profiling.cc:243] Waited 918 ms on lock 0x7b54000d9154. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:01:43.333801 (+   199us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2112305920}
W20260709 13:01:43.308233  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 110671us
I20260709 13:01:43.344998  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:43.394899  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:43.400717  2337 rpcz_store.cc:276] 0709 13:01:42.215756 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:42.215911 (+   155us) service_pool.cc:224] Handling call
0709 13:01:42.264585 (+ 48674us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:42.967517 (+702932us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:43.332677 (+365160us) spinlock_profiling.cc:243] Waited 700 ms on lock 0x7b54005b9fd4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:01:43.332883 (+   206us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1609332992}
I20260709 13:01:43.412586  1602 raft_consensus.cc:2468] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 15.
I20260709 13:01:43.414274  1602 raft_consensus.cc:1275] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 15 index: 8. Preceding OpId from leader: term: 16 index: 8. (term mismatch)
I20260709 13:01:43.414683  1602 pending_rounds.cc:85] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a: Aborting all ops after (but not including) 7
I20260709 13:01:43.394927  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:43.332664  7307 leader_election.cc:290] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 17 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:43.409829  1750 raft_consensus.cc:1275] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 13 index: 7. (index mismatch)
I20260709 13:01:43.410351  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:43.416088  1605 raft_consensus.cc:3060] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:01:43.334816  7565 consensus_queue.cc:1059] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 12.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20260709 13:01:43.411936  1643 leader_election.cc:336] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 16 election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 1.867s (SENT)
I20260709 13:01:43.418104  1643 leader_election.cc:304] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:01:43.409241  1601 raft_consensus.cc:2468] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 16.
I20260709 13:01:43.428527  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:43.429425  1680 raft_consensus.cc:3060] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 9 FOLLOWER]: Advancing to term 11
I20260709 13:01:43.340024  7628 raft_consensus.cc:493] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:43.436242  1752 raft_consensus.cc:1275] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 13 index: 7. Preceding OpId from leader: term: 16 index: 8. (index mismatch)
I20260709 13:01:43.414883  1602 pending_rounds.cc:107] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a: Aborting uncommitted NO_OP operation due to leader change: 15.8
I20260709 13:01:43.449793  1602 raft_consensus.cc:2889] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:01:43.453478  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:43.400135  7583 raft_consensus.cc:493] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:43.466498  1675 raft_consensus.cc:1240] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 10. Current term is 11. Ops: []
I20260709 13:01:43.468389  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:43.433950  7582 leader_election.cc:290] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:01:43.481407  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 135109us
I20260709 13:01:43.482210  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:43.371255  7586 consensus_queue.cc:1059] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 19.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.086s
I20260709 13:01:43.455904  7483 raft_consensus.cc:493] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:43.492412  1602 raft_consensus.cc:1240] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 16. Current term is 17. Ops: []
W20260709 13:01:43.470706  7635 log.cc:927] Time spent T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.055s	user 0.000s	sys 0.001s
I20260709 13:01:43.456746  7580 leader_election.cc:290] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 13 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:43.460683  7437 leader_election.cc:290] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 16 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:01:43.500290  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 130070us
I20260709 13:01:43.483594  7489 raft_consensus.cc:3060] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:01:43.520325  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:43.521312  1747 raft_consensus.cc:3060] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:01:43.500707  7496 raft_consensus.cc:2749] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 13:01:43.479514  7583 raft_consensus.cc:515] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:43.535027  1605 raft_consensus.cc:2468] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 13.
I20260709 13:01:43.536583  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:43.537492  1680 raft_consensus.cc:2468] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 11.
I20260709 13:01:43.458364  7628 raft_consensus.cc:515] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:43.539281  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 127382us
W20260709 13:01:43.542289  1643 leader_election.cc:336] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 16 election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 1.903s (SENT)
W20260709 13:01:43.542872  1718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:43.544046  1643 leader_election.cc:336] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 2.047s (SENT)
I20260709 13:01:43.544508  1643 leader_election.cc:304] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:01:43.544977  7628 leader_election.cc:290] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:01:43.545794  1643 leader_election.cc:336] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 14 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 2.047s (SENT)
I20260709 13:01:43.545850  7437 raft_consensus.cc:2749] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:01:43.546480  1643 leader_election.cc:304] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 13:01:43.537943  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 45261.8us
I20260709 13:01:43.501389  7483 raft_consensus.cc:515] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:43.553982  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:43.557049  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:43.558753  1603 raft_consensus.cc:1275] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 13 index: 7. (term mismatch)
I20260709 13:01:43.559285  1603 pending_rounds.cc:85] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a: Aborting all ops after (but not including) 6
I20260709 13:01:43.559556  1603 pending_rounds.cc:107] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a: Aborting uncommitted NO_OP operation due to leader change: 12.7
I20260709 13:01:43.559799  1603 raft_consensus.cc:2889] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:01:43.539557  7616 raft_consensus.cc:493] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:43.547458  7437 raft_consensus.cc:2749] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:01:43.566324  1747 raft_consensus.cc:2468] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 16.
W20260709 13:01:43.568176  7642 log.cc:927] Time spent T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.077s	user 0.000s	sys 0.001s
I20260709 13:01:43.568836  1601 raft_consensus.cc:1240] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 11. Current term is 12. Ops: [11.8-11.8]
I20260709 13:01:43.569353  7583 leader_election.cc:290] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:43.572649  1601 raft_consensus.cc:1240] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 12. Current term is 13. Ops: [12.7-12.7]
W20260709 13:01:43.574997  7641 log.cc:927] Time spent T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.068s	user 0.000s	sys 0.001s
I20260709 13:01:43.575776  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:43.578717  7616 raft_consensus.cc:515] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:43.579516  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:43.493276  7622 consensus_queue.cc:237] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.11, Last appended by leader: 11, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:43.580372  1598 raft_consensus.cc:2468] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 10.
I20260709 13:01:43.581789  7491 leader_election.cc:290] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:43.582854  7468 leader_election.cc:290] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:43.581257  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:43.585140  1601 raft_consensus.cc:2468] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 12.
I20260709 13:01:43.586012  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:43.586640  1601 raft_consensus.cc:2468] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 16.
I20260709 13:01:43.581478  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:43.584349  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:43.588363  1598 raft_consensus.cc:3060] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:01:43.603132  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:43.607153  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:43.527169  7655 consensus_queue.cc:1048] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.064s
I20260709 13:01:43.607153  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:43.608178  1749 raft_consensus.cc:2468] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 17.
I20260709 13:01:43.932884  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 17 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:43.708810  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 16 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:43.658511  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:43.947474  1749 raft_consensus.cc:3060] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Advancing to term 14
W20260709 13:01:43.864108  1719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:43.661275  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:43.952540  1747 raft_consensus.cc:3060] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:01:43.934428  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 16 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:43.953526  1753 raft_consensus.cc:2393] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 16: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:01:43.954728  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
W20260709 13:01:43.892971  7436 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.477s	user 0.011s	sys 0.028s
W20260709 13:01:43.957456  7436 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.542s	user 0.012s	sys 0.029s
I20260709 13:01:43.695271  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:43.959409  1603 raft_consensus.cc:2393] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 14: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:01:43.959793  1753 raft_consensus.cc:1275] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 12 index: 9. Preceding OpId from leader: term: 13 index: 10. (index mismatch)
I20260709 13:01:43.948612  1569 leader_election.cc:304] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
W20260709 13:01:43.680176  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 127231us
W20260709 13:01:43.919267  7337 consensus_meta.cc:287] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.551s	user 0.011s	sys 0.004s
I20260709 13:01:43.844233  7583 raft_consensus.cc:493] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
W20260709 13:01:43.969388  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 423534us
I20260709 13:01:43.960726  1603 raft_consensus.cc:1240] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 13. Current term is 14. Ops: []
I20260709 13:01:43.972666  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:43.979172  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:43.978922  1600 raft_consensus.cc:1275] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 15 index: 8. Preceding OpId from leader: term: 16 index: 9. (index mismatch)
I20260709 13:01:43.971171  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:43.997354  1747 raft_consensus.cc:2468] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 14.
W20260709 13:01:43.920199  7645 log.cc:927] Time spent T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.160s	user 0.001s	sys 0.000s
I20260709 13:01:44.000993  1601 raft_consensus.cc:1275] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
W20260709 13:01:43.767221  7648 log.cc:927] Time spent T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.155s	user 0.001s	sys 0.000s
I20260709 13:01:43.621340  7491 raft_consensus.cc:493] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:44.010205  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:44.011171  1753 raft_consensus.cc:2468] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 16.
I20260709 13:01:43.611891  7647 raft_consensus.cc:493] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:44.018923  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:44.005934  7583 raft_consensus.cc:515] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:44.020731  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:44.022050  7583 leader_election.cc:290] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:44.014264  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:43.921813  7575 raft_consensus.cc:493] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:01:43.684338  7599 raft_consensus.cc:493] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:43.846576  7483 leader_election.cc:290] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:44.018792  7647 raft_consensus.cc:515] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:44.000100  7586 raft_consensus.cc:493] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:44.010076  7491 raft_consensus.cc:515] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:44.024422  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:44.023877  1718 leader_election.cc:304] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:44.000033  7577 raft_consensus.cc:2804] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:01:43.974663  7337 raft_consensus.cc:515] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:44.064951  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:01:44.027964  7565 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.552s	user 0.011s	sys 0.011s
W20260709 13:01:44.080415  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 56597.8us
I20260709 13:01:44.080274  1753 raft_consensus.cc:2410] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for term 13 because replica has last-logged OpId of term: 12 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 6.
I20260709 13:01:44.109754  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
W20260709 13:01:44.110306  1605 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.557s	user 0.001s	sys 0.012s
W20260709 13:01:44.110795  1605 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.557s	user 0.001s	sys 0.012s
I20260709 13:01:44.111969  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:01:44.096560  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 116962us
I20260709 13:01:44.108238  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 16 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:44.027664  7559 raft_consensus.cc:493] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:44.029302  7659 raft_consensus.cc:493] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:44.134711  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:44.141634  1718 leader_election.cc:304] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:44.144552  1753 raft_consensus.cc:1240] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 15. Current term is 16. Ops: [15.9-15.9]
I20260709 13:01:44.109510  7307 raft_consensus.cc:2804] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:01:44.172833  1673 raft_consensus.cc:2393] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 14: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
W20260709 13:01:44.173501  7646 log.cc:927] Time spent T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.330s	user 0.002s	sys 0.000s
I20260709 13:01:44.180706  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:01:44.121784  7565 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.641s	user 0.014s	sys 0.013s
I20260709 13:01:44.112972  7577 raft_consensus.cc:697] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:01:44.209790  7437 raft_consensus.cc:493] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:44.235838  1599 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 1465) took 1008 ms. Trace:
I20260709 13:01:44.038610  7575 raft_consensus.cc:515] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:44.244146  1599 rpcz_store.cc:276] 0709 13:01:43.227060 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:43.275786 (+ 48726us) service_pool.cc:224] Handling call
0709 13:01:43.276263 (+   477us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:43.277131 (+   868us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:43.277168 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:01:43.288112 (+ 10944us) log.cc:844] Serialized 26 byte log entry
0709 13:01:43.447092 (+158980us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:01:43.447145 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:43.447230 (+    85us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:44.189538 (+742308us) raft_consensus.cc:1685] finished
0709 13:01:44.189570 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:44.235693 (+ 46123us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":101,"raft.run_cpu_time_us":153,"raft.run_wall_time_us":3471,"thread_start_us":158474,"threads_started":1,"wal-append.queue_time_us":250546}
I20260709 13:01:44.091543  7586 raft_consensus.cc:515] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:44.284395  2331 raft_consensus.cc:3060] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Advancing to term 11
W20260709 13:01:44.272287  7660 log.cc:927] Time spent T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.120s	user 0.000s	sys 0.000s
I20260709 13:01:44.260222  7437 raft_consensus.cc:515] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:44.181057  7307 raft_consensus.cc:493] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:44.108170  7599 raft_consensus.cc:515] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:44.190656  1749 raft_consensus.cc:2468] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 14.
I20260709 13:01:44.319792  7677 raft_consensus.cc:493] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:01:44.320667  7677 raft_consensus.cc:515] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:44.324384  7653 raft_consensus.cc:3055] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 13:01:44.324966  7653 raft_consensus.cc:740] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:01:44.328609  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:44.329668  1678 raft_consensus.cc:2393] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 13: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:01:44.254091  7578 raft_consensus.cc:493] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:44.251396  7622 consensus_queue.cc:1059] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 10.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.091s
I20260709 13:01:44.287302  7600 raft_consensus.cc:493] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:01:44.313963  7307 raft_consensus.cc:3060] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:01:44.326148  7653 consensus_queue.cc:260] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.11, Last appended by leader: 11, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:44.382416  1597 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 2698) took 1156 ms. Trace:
I20260709 13:01:44.330415  7337 leader_election.cc:290] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 14 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:44.332741  7565 consensus_queue.cc:1048] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.070s
I20260709 13:01:44.211081  7659 raft_consensus.cc:515] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:44.462492  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 351356us
I20260709 13:01:44.283315  7577 consensus_queue.cc:237] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.8, Last appended by leader: 8, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } }
W20260709 13:01:44.485793  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 346445us
I20260709 13:01:44.189364  7559 raft_consensus.cc:515] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:44.388146  1597 rpcz_store.cc:276] 0709 13:01:43.226059 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:43.260852 (+ 34793us) service_pool.cc:224] Handling call
0709 13:01:43.261289 (+   437us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:43.261752 (+   463us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:43.261787 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:01:43.262420 (+   633us) log.cc:844] Serialized 26 byte log entry
0709 13:01:43.392424 (+130004us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:01:43.392477 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:43.392565 (+    88us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:44.381976 (+989411us) raft_consensus.cc:1685] finished
0709 13:01:44.382041 (+    65us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:44.382305 (+   264us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":11004,"raft.run_cpu_time_us":116,"raft.run_wall_time_us":1591,"thread_start_us":129489,"threads_started":1,"wal-append.queue_time_us":343503}
W20260709 13:01:44.490864  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 310217us
W20260709 13:01:44.484967  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 227184us
I20260709 13:01:44.492858  1646 leader_election.cc:304] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:01:44.505293  1569 leader_election.cc:304] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:01:44.520548  2329 raft_consensus.cc:2393] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 16: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:01:44.383711  7578 raft_consensus.cc:515] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:44.383881  7600 raft_consensus.cc:515] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:44.511953  7660 log.cc:927] Time spent T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.073s	user 0.000s	sys 0.001s
I20260709 13:01:44.539834  1646 leader_election.cc:304] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:44.549166  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:44.555976  1571 leader_election.cc:304] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:44.546422  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:44.563992  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:44.565052  1605 raft_consensus.cc:2393] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 14: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
W20260709 13:01:44.566684  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 425038us
W20260709 13:01:44.570176  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 411406us
I20260709 13:01:44.571123  1643 leader_election.cc:304] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:44.545753  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:44.573598  1603 raft_consensus.cc:2376] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 16. Re-sending same reply.
I20260709 13:01:44.573918  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:44.576222  7436 raft_consensus.cc:2804] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Leader election won for term 16
I20260709 13:01:44.545583  7496 consensus_queue.cc:1048] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.035s
I20260709 13:01:44.536454  7337 raft_consensus.cc:2749] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:01:44.595279  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:44.595685  1602 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40154 (request call id 2585) took 1188 ms. Trace:
I20260709 13:01:44.541927  7583 consensus_queue.cc:1048] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.055s
I20260709 13:01:44.633934  1600 raft_consensus.cc:2376] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 17. Re-sending same reply.
I20260709 13:01:44.571019  7599 leader_election.cc:290] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:44.631420  1602 rpcz_store.cc:276] 0709 13:01:43.407571 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:43.516131 (+108560us) service_pool.cc:224] Handling call
0709 13:01:43.516718 (+   587us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:43.518618 (+  1900us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:43.518650 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:01:43.526288 (+  7638us) log.cc:844] Serialized 26 byte log entry
0709 13:01:43.597714 (+ 71426us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:01:43.598244 (+   530us) log.cc:844] Serialized 14 byte log entry
0709 13:01:43.598711 (+   467us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:43.598847 (+   136us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:44.595354 (+996507us) raft_consensus.cc:1685] finished
0709 13:01:44.595386 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:44.595591 (+   205us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":247,"prepare.run_cpu_time_us":92,"prepare.run_wall_time_us":96,"raft.queue_time_us":306,"raft.run_cpu_time_us":218,"raft.run_wall_time_us":15046,"spinlock_wait_cycles":14126464,"thread_start_us":40510,"threads_started":1,"wal-append.queue_time_us":477560}
I20260709 13:01:44.581244  7491 leader_election.cc:290] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 13:01:44.795454  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 1.54877e+06us
I20260709 13:01:44.663793  7647 leader_election.cc:290] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:44.689203  7677 leader_election.cc:290] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:01:44.872715  1598 consensus_meta.cc:287] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 1.284s	user 0.017s	sys 0.013s
W20260709 13:01:44.911015  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 355079us
W20260709 13:01:44.892194  1717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:44.915808  1717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:44.901945  1598 raft_consensus.cc:2468] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 20.
W20260709 13:01:44.952783  2299 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.548s	user 0.009s	sys 0.008s
I20260709 13:01:44.951817  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:01:44.958308  2299 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.554s	user 0.010s	sys 0.009s
W20260709 13:01:44.937430  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 367735us
W20260709 13:01:44.952917  1751 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.700s	user 0.004s	sys 0.027s
I20260709 13:01:44.959076  1596 raft_consensus.cc:2393] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 14: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
W20260709 13:01:44.962272  1751 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.719s	user 0.005s	sys 0.027s
I20260709 13:01:44.990351  1717 leader_election.cc:304] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:44.992141  1717 leader_election.cc:304] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 13:01:44.993171  2298 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.527s	user 0.024s	sys 0.026s
W20260709 13:01:44.993582  2298 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.527s	user 0.024s	sys 0.026s
I20260709 13:01:44.783191  7691 raft_consensus.cc:2804] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Leader pre-election won for term 17
W20260709 13:01:44.995393  1596 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40166 (request call id 2697) took 1748 ms (client timeout 1840 ms). Trace:
W20260709 13:01:45.005151  1596 rpcz_store.cc:269] 0709 13:01:43.225737 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:43.247162 (+ 21425us) service_pool.cc:224] Handling call
0709 13:01:44.951627 (+1704465us) spinlock_profiling.cc:243] Waited 1.21 ms on lock 0x7b5800003698. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f426303db71 00007f426303deee 0000000000375939 00007f426b0b11e0 00007f426b17a0c4 00007f426b17a131 00007f426b1280a0 00007f426b127984 00007f42646a26d4 00007f42646a2662
0709 13:01:44.973996 (+ 22369us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2811008}
I20260709 13:01:44.788187  7491 raft_consensus.cc:493] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:44.914256  1571 leader_election.cc:304] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
W20260709 13:01:44.997558  1598 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40154 (request call id 2596) took 1406 ms (client timeout 1815 ms). Trace:
W20260709 13:01:44.952373  7676 log.cc:927] Time spent T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.559s	user 0.000s	sys 0.002s
W20260709 13:01:44.948990  7489 consensus_meta.cc:287] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 1.428s	user 0.023s	sys 0.007s
I20260709 13:01:44.912233  7647 raft_consensus.cc:2804] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Leader election won for term 11
W20260709 13:01:45.007197  1598 rpcz_store.cc:269] 0709 13:01:43.573874 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:43.581182 (+  7308us) service_pool.cc:224] Handling call
0709 13:01:44.980067 (+1398885us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:01:44.887698  7698 raft_consensus.cc:2749] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
W20260709 13:01:45.012176  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 441078us
I20260709 13:01:45.021720  1643 leader_election.cc:304] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:44.917178  7437 leader_election.cc:290] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:44.937168  7436 raft_consensus.cc:697] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:01:45.023658  1643 leader_election.cc:304] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:44.944393  7575 leader_election.cc:290] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:44.974306  7667 raft_consensus.cc:493] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
W20260709 13:01:45.016471  1569 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.508s	user 0.025s	sys 0.032s
I20260709 13:01:44.911808  7586 leader_election.cc:290] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 13:01:44.943773  7655 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.714s	user 0.013s	sys 0.004s
I20260709 13:01:44.974375  7693 raft_consensus.cc:2764] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Leader pre-election decision vote started in defunct term 16: won
W20260709 13:01:45.038224  1569 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.530s	user 0.025s	sys 0.032s
I20260709 13:01:45.008989  7691 raft_consensus.cc:493] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:01:45.045239  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 539986us
I20260709 13:01:45.049608  1569 leader_election.cc:304] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:44.932922  7702 consensus_queue.cc:1048] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.126s
I20260709 13:01:44.996381  7689 raft_consensus.cc:493] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:01:44.935194  7540 consensus_queue.cc:1048] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.096s
W20260709 13:01:45.044258  7655 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.834s	user 0.014s	sys 0.004s
I20260709 13:01:45.054916  7691 raft_consensus.cc:3060] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:01:45.056175  7688 raft_consensus.cc:493] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:45.106271  7706 raft_consensus.cc:2749] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:01:45.111670  7688 raft_consensus.cc:515] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:45.081092  7713 raft_consensus.cc:493] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:45.113735  7713 raft_consensus.cc:515] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:45.054747  7496 consensus_queue.cc:1048] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.003s
I20260709 13:01:45.059427  7666 raft_consensus.cc:493] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:01:45.060066  7682 log.cc:927] Time spent T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.506s	user 0.001s	sys 0.001s
I20260709 13:01:45.115691  7666 raft_consensus.cc:515] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:45.118052  7666 leader_election.cc:290] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:45.060145  2336 raft_consensus.cc:2468] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 15.
I20260709 13:01:45.027325  7491 raft_consensus.cc:515] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:45.032958  7699 raft_consensus.cc:2749] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:01:45.003876  7483 consensus_queue.cc:1048] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.041s
I20260709 13:01:45.069480  7684 raft_consensus.cc:493] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:01:45.071043  7559 leader_election.cc:290] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:45.045646  7667 raft_consensus.cc:515] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:45.070798  7700 raft_consensus.cc:493] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
W20260709 13:01:45.076207  1646 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.507s	user 0.035s	sys 0.043s
I20260709 13:01:45.010174  7683 raft_consensus.cc:493] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:45.042014  7437 raft_consensus.cc:2749] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
W20260709 13:01:45.093578  2331 consensus_meta.cc:287] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.796s	user 0.011s	sys 0.017s
I20260709 13:01:45.096069  7687 raft_consensus.cc:493] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:45.096091  7540 raft_consensus.cc:2749] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:01:45.053231  7578 leader_election.cc:290] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:45.114359  7688 leader_election.cc:290] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:45.106184  7600 leader_election.cc:290] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:45.106877  7693 raft_consensus.cc:2804] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:01:45.108600  7702 consensus_queue.cc:1059] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 11.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 13:01:45.054834  7694 raft_consensus.cc:493] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:45.109527  1604 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 2686) took 1963 ms. Trace:
I20260709 13:01:45.110529  7436 consensus_queue.cc:237] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.9, Last appended by leader: 9, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:45.041136  1643 leader_election.cc:304] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
W20260709 13:01:45.144464  1646 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.576s	user 0.035s	sys 0.043s
W20260709 13:01:45.171698  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 603092us
I20260709 13:01:45.151024  2331 raft_consensus.cc:2468] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 11.
W20260709 13:01:45.175796  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 134773us
W20260709 13:01:45.145426  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 143231us
I20260709 13:01:45.119050  7677 raft_consensus.cc:493] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:45.090323  7689 raft_consensus.cc:515] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:45.164321  1673 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52766 (request call id 2894) took 1730 ms. Trace:
I20260709 13:01:45.180162  1643 leader_election.cc:304] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:01:45.201848  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:45.203869  7338 raft_consensus.cc:493] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:45.204656  7338 raft_consensus.cc:515] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:45.205961  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:45.207183  7338 leader_election.cc:290] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:45.208778  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:45.211565  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:45.212378  2335 raft_consensus.cc:2468] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 16.
I20260709 13:01:45.212651  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 18 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:45.200876  1673 rpcz_store.cc:276] 0709 13:01:43.427623 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:43.428411 (+   788us) service_pool.cc:224] Handling call
0709 13:01:45.157275 (+1728864us) spinlock_profiling.cc:243] Waited 714 ms on lock 0x7b54000d8254. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:01:45.157511 (+   236us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1642995584}
I20260709 13:01:45.168699  1604 rpcz_store.cc:276] 0709 13:01:43.146150 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:43.146724 (+   574us) service_pool.cc:224] Handling call
0709 13:01:43.160117 (+ 13393us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:43.160584 (+   467us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:43.160623 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:01:43.164401 (+  3778us) log.cc:844] Serialized 26 byte log entry
0709 13:01:43.249537 (+ 85136us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:01:43.279988 (+ 30451us) log.cc:844] Serialized 14 byte log entry
0709 13:01:45.108820 (+1828832us) spinlock_profiling.cc:243] Waited 4.9 ms on lock 0x7b1800452380. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f4265d0a4bc 00007f426305848b 00007f426beb2ab9 00007f426307469b 00007f426307293f 00007f4263072455 00007f426615a060 00007f426616c716 00007f426617d287
0709 13:01:45.109003 (+   183us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:45.109099 (+    96us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:45.109149 (+    50us) raft_consensus.cc:1685] finished
0709 13:01:45.109267 (+   118us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:45.109442 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":30596,"prepare.run_cpu_time_us":62,"prepare.run_wall_time_us":69,"raft.queue_time_us":2843,"raft.run_cpu_time_us":279,"raft.run_wall_time_us":6451,"spinlock_wait_cycles":11264768,"thread_start_us":247444,"threads_started":2,"wal-append.queue_time_us":268363,"wal-append.run_cpu_time_us":6035,"wal-append.run_wall_time_us":1147555}
I20260709 13:01:45.227814  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:01:45.228145  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 48105.8us
I20260709 13:01:45.168049  7693 raft_consensus.cc:493] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:45.159035  7687 raft_consensus.cc:515] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:45.234712  7338 raft_consensus.cc:2804] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Leader election won for term 13
I20260709 13:01:45.231701  7703 consensus_queue.cc:1059] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 10.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.013s
I20260709 13:01:45.235452  7338 raft_consensus.cc:697] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:01:45.236024  7710 raft_consensus.cc:493] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:45.236758  7710 raft_consensus.cc:515] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:45.236812  7338 consensus_queue.cc:237] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:45.218904  7666 consensus_queue.cc:1059] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 16.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.024s
I20260709 13:01:45.239660  7693 raft_consensus.cc:3060] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:01:45.250506  7570 raft_consensus.cc:3055] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:01:45.251088  7570 raft_consensus.cc:740] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
W20260709 13:01:45.251775  7704 log.cc:927] Time spent T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.178s	user 0.000s	sys 0.001s
I20260709 13:01:45.252197  7570 consensus_queue.cc:260] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.10, Last appended by leader: 10, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:45.253450  7570 raft_consensus.cc:3060] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:01:45.253989  1745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55840 (request call id 2832) took 1269 ms. Trace:
I20260709 13:01:45.254415  1745 rpcz_store.cc:276] 0709 13:01:43.984792 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:44.163966 (+179174us) service_pool.cc:224] Handling call
0709 13:01:44.164487 (+   521us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:44.466912 (+302425us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:44.466965 (+    53us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:01:44.470147 (+  3182us) log.cc:844] Serialized 26 byte log entry
0709 13:01:44.795531 (+325384us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:01:45.123781 (+328250us) log.cc:844] Serialized 14 byte log entry
0709 13:01:45.124229 (+   448us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:45.124349 (+   120us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:45.253712 (+129363us) raft_consensus.cc:1685] finished
0709 13:01:45.253740 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:45.253906 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":14646,"raft.queue_time_us":442030,"raft.run_cpu_time_us":316,"raft.run_wall_time_us":15774,"thread_start_us":788910,"threads_started":3,"wal-append.queue_time_us":599524}
W20260709 13:01:45.257454  7717 log.cc:927] Time spent T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.187s	user 0.000s	sys 0.001s
I20260709 13:01:45.259171  2334 raft_consensus.cc:1240] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 12. Current term is 13. Ops: []
I20260709 13:01:45.196763  6882 raft_consensus.cc:2804] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Leader pre-election won for term 17
W20260709 13:01:45.272433  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 280363us
I20260709 13:01:45.214640  7540 raft_consensus.cc:3055] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 11 LEADER]: Stepping down as leader of term 11
W20260709 13:01:45.266011  2333 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39726 (request call id 1373) took 2837 ms (client timeout 1916 ms). Trace:
I20260709 13:01:45.215065  7668 raft_consensus.cc:493] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
W20260709 13:01:45.235579  7716 log.cc:927] Time spent T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.161s	user 0.001s	sys 0.000s
I20260709 13:01:45.277621  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:45.246125  7616 leader_election.cc:290] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:01:45.278810  2333 rpcz_store.cc:269] 0709 13:01:42.420816 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:42.421022 (+   206us) service_pool.cc:224] Handling call
0709 13:01:45.258184 (+2837162us) spinlock_profiling.cc:243] Waited 778 ms on lock 0x7b54004880d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:01:45.258357 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1790561792}
I20260709 13:01:45.148101  7684 raft_consensus.cc:515] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:45.288862  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 374648us
I20260709 13:01:45.156103  7683 raft_consensus.cc:515] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:45.294623  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:45.295497  1571 leader_election.cc:304] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:45.296927  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:45.300372  1678 raft_consensus.cc:3060] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:01:45.300635  1671 raft_consensus.cc:2410] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for term 14 because replica has last-logged OpId of term: 13 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 9.
I20260709 13:01:45.197836  7559 consensus_queue.cc:1048] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.078s
I20260709 13:01:45.250547  7687 leader_election.cc:290] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:45.216184  7677 raft_consensus.cc:515] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:45.163007  7700 raft_consensus.cc:515] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:45.272368  6882 raft_consensus.cc:493] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:45.278577  7540 raft_consensus.cc:740] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:01:45.288420  7590 raft_consensus.cc:2749] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20260709 13:01:45.297564  1643 connection.cc:256] RPC call timeout handler was delayed by -1.2438208140000029s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18400000000000002s for a response
I20260709 13:01:45.197439  7694 raft_consensus.cc:515] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:45.274070  7698 consensus_queue.cc:1048] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.048s
I20260709 13:01:45.223215  7714 consensus_queue.cc:1048] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.102s
I20260709 13:01:45.316862  7468 raft_consensus.cc:2804] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:01:45.455799  6882 raft_consensus.cc:3060] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Advancing to term 17
W20260709 13:01:45.646669  1719 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.307s	user 0.024s	sys 0.059s
W20260709 13:01:45.657406  1719 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.317s	user 0.024s	sys 0.059s
W20260709 13:01:45.659459  1719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:45.673095  2298 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.671s	user 0.012s	sys 0.073s
I20260709 13:01:45.309288  7578 consensus_queue.cc:1059] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 13.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.308s
I20260709 13:01:45.291747  7668 raft_consensus.cc:515] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:45.476528  7687 raft_consensus.cc:493] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
W20260709 13:01:45.739785  1569 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.678s	user 0.051s	sys 0.131s
W20260709 13:01:45.758956  1569 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.698s	user 0.051s	sys 0.131s
W20260709 13:01:45.759549  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 698363us
I20260709 13:01:45.750634  2329 raft_consensus.cc:1275] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 16 index: 13. (index mismatch)
W20260709 13:01:45.680471  2298 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.678s	user 0.013s	sys 0.073s
W20260709 13:01:45.793948  1718 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.800s	user 0.038s	sys 0.062s
W20260709 13:01:45.794688  1718 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.801s	user 0.038s	sys 0.062s
W20260709 13:01:45.795058  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 801633us
I20260709 13:01:45.636250  7337 raft_consensus.cc:493] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:45.806972  1569 leader_election.cc:304] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
W20260709 13:01:45.685621  7647 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.641s	user 0.039s	sys 0.116s
W20260709 13:01:45.820161  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 53056.6us
W20260709 13:01:45.814723  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 527170us
I20260709 13:01:45.815573  1753 raft_consensus.cc:1240] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 16. Current term is 17. Ops: [16.8-16.8]
W20260709 13:01:45.750772  7307 consensus_meta.cc:287] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 1.345s	user 0.018s	sys 0.001s
I20260709 13:01:45.847891  7307 raft_consensus.cc:515] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:45.848938  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:45.849473  1749 raft_consensus.cc:1240] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 17. Current term is 18. Ops: []
I20260709 13:01:45.849895  1753 raft_consensus.cc:2393] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 14: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:01:45.850737  1717 leader_election.cc:304] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:45.851099  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:45.851835  1753 raft_consensus.cc:2468] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 16.
I20260709 13:01:45.852504  1717 leader_election.cc:304] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:45.853849  1717 leader_election.cc:304] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:45.855646  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:45.856144  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:45.856884  1752 raft_consensus.cc:2468] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 15.
I20260709 13:01:45.684401  7483 raft_consensus.cc:493] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:45.858596  1597 raft_consensus.cc:1240] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 19. Current term is 20. Ops: []
I20260709 13:01:45.861945  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:45.866365  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:45.690663  7468 raft_consensus.cc:697] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:01:45.873658  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:45.873744  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:45.738310  7688 raft_consensus.cc:493] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
W20260709 13:01:45.857812  7647 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.815s	user 0.039s	sys 0.117s
I20260709 13:01:45.884517  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:01:45.887615  7338 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.647s	user 0.021s	sys 0.021s
I20260709 13:01:45.858628  7728 raft_consensus.cc:493] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:01:45.898765  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:45.900314  1746 raft_consensus.cc:1275] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 14 index: 10. Preceding OpId from leader: term: 16 index: 11. (index mismatch)
I20260709 13:01:45.898285  1678 raft_consensus.cc:2468] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 17.
I20260709 13:01:45.880332  7710 leader_election.cc:290] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:01:45.888172  7338 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.647s	user 0.021s	sys 0.021s
I20260709 13:01:45.886767  7647 raft_consensus.cc:697] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:01:45.809203  7687 raft_consensus.cc:515] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:45.892127  7689 leader_election.cc:290] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:45.930938  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:45.932070  1604 raft_consensus.cc:2468] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 15.
I20260709 13:01:45.878201  7483 raft_consensus.cc:515] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:45.876374  7337 raft_consensus.cc:515] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:45.899571  7725 raft_consensus.cc:2749] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:01:45.907858  7684 leader_election.cc:290] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:45.886852  7578 consensus_queue.cc:1048] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.027s
I20260709 13:01:45.896926  7688 raft_consensus.cc:515] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:45.992272  2336 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49082 (request call id 1091) took 1936 ms (client timeout 2124 ms). Trace:
I20260709 13:01:45.994557  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:01:45.940749  7690 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.702s	user 0.030s	sys 0.072s
W20260709 13:01:45.993044  2336 rpcz_store.cc:269] 0709 13:01:44.042623 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:44.042743 (+   120us) service_pool.cc:224] Handling call
0709 13:01:45.978665 (+1935922us) spinlock_profiling.cc:243] Waited 913 ms on lock 0x7b54005843d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:01:45.978850 (+   185us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2100336512}
I20260709 13:01:45.880151  7540 consensus_queue.cc:260] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 11.8, Last appended by leader: 8, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:45.952059  7491 leader_election.cc:290] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:46.010450  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:46.016350  7586 consensus_queue.cc:1059] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 19.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 13:01:45.911841  7728 raft_consensus.cc:515] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:45.993378  7668 leader_election.cc:290] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:01:46.008862  7690 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.766s	user 0.030s	sys 0.073s
I20260709 13:01:46.035203  7667 leader_election.cc:290] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:46.011714  7631 raft_consensus.cc:2749] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:01:46.047586  7580 raft_consensus.cc:3055] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 13:01:46.048233  7580 raft_consensus.cc:740] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:01:46.049505  7580 consensus_queue.cc:260] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.13, Last appended by leader: 13, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:46.052531  7580 raft_consensus.cc:3060] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:01:46.012732  7468 consensus_queue.cc:237] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.7, Last appended by leader: 7, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:46.059051  1604 raft_consensus.cc:2468] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 17.
I20260709 13:01:46.051218  1746 raft_consensus.cc:1240] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 19. Current term is 20. Ops: []
I20260709 13:01:46.060475  7728 leader_election.cc:290] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:46.061141  7728 raft_consensus.cc:493] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:46.061614  7728 raft_consensus.cc:515] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:46.062151  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:46.062896  1746 raft_consensus.cc:2468] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 14.
I20260709 13:01:46.063886  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:46.067883  7337 leader_election.cc:290] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:46.079084  1749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43156 (request call id 1215) took 2072 ms. Trace:
I20260709 13:01:46.082392  7653 raft_consensus.cc:3060] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:01:46.082505  7683 leader_election.cc:290] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:46.079741  1749 rpcz_store.cc:276] 0709 13:01:44.007009 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:44.342053 (+335044us) service_pool.cc:224] Handling call
0709 13:01:44.393254 (+ 51201us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:45.850137 (+1456883us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:46.078719 (+228582us) spinlock_profiling.cc:243] Waited 1.46 s on lock 0x7b54000e4a54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:01:46.078986 (+   267us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3347760384}
I20260709 13:01:46.039798  7700 leader_election.cc:290] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:46.045425  7710 raft_consensus.cc:493] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:01:46.096928  7691 consensus_meta.cc:287] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.987s	user 0.016s	sys 0.000s
W20260709 13:01:46.094738  7570 consensus_meta.cc:287] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.841s	user 0.015s	sys 0.002s
I20260709 13:01:46.023221  7725 consensus_queue.cc:1059] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 10.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.071s
W20260709 13:01:46.095423  7693 consensus_meta.cc:287] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.855s	user 0.015s	sys 0.000s
I20260709 13:01:46.095971  7687 leader_election.cc:290] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:46.096514  7710 raft_consensus.cc:515] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:46.076083  7731 raft_consensus.cc:493] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:01:46.075832  7647 consensus_queue.cc:237] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:46.098868  7688 leader_election.cc:290] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:46.094420  7483 leader_election.cc:290] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:46.103861  7728 leader_election.cc:290] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:46.105306  6882 raft_consensus.cc:515] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:46.112028  7584 raft_consensus.cc:3060] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:01:46.091948  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:01:46.124271  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 317407us
I20260709 13:01:46.161198  1569 leader_election.cc:304] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
W20260709 13:01:46.128741  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 260823us
I20260709 13:01:46.090282  7578 raft_consensus.cc:493] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:46.129853  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:46.165195  1749 raft_consensus.cc:2393] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 13: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:01:46.168463  7738 raft_consensus.cc:2749] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:01:46.169905  1717 leader_election.cc:304] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:46.171581  1717 leader_election.cc:304] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:46.176776  1717 leader_election.cc:304] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:46.179059  1717 leader_election.cc:304] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:46.167348  1646 leader_election.cc:304] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:46.123874  7691 raft_consensus.cc:515] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:46.166023  7337 raft_consensus.cc:493] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:46.125129  7693 raft_consensus.cc:515] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:01:46.189116  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 172261us
I20260709 13:01:46.162228  7727 raft_consensus.cc:2749] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20260709 13:01:46.191565  1569 connection.cc:256] RPC call timeout handler was delayed by -1.4233055869999589s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2666s for a response
I20260709 13:01:46.192509  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:46.192651  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:46.197222  1605 raft_consensus.cc:2468] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 16.
I20260709 13:01:46.197386  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:01:46.182695  7669 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.235s	user 0.007s	sys 0.023s
I20260709 13:01:46.193948  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:46.198105  1749 raft_consensus.cc:2468] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 13.
I20260709 13:01:46.198616  1602 raft_consensus.cc:3060] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:01:46.184500  7700 raft_consensus.cc:2804] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:01:46.184279  7491 raft_consensus.cc:2804] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:01:46.129057  7731 raft_consensus.cc:515] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:46.224608  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 16 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:46.204041  1646 leader_election.cc:304] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:46.182428  7688 raft_consensus.cc:2749] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:01:46.207886  1717 leader_election.cc:304] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:46.210091  2298 leader_election.cc:304] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:01:46.211200  7683 raft_consensus.cc:493] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:46.205670  7338 raft_consensus.cc:2804] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:01:46.215782  6882 leader_election.cc:290] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 17 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:46.215719  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:46.229918  1598 raft_consensus.cc:2468] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 16.
I20260709 13:01:46.188593  7181 raft_consensus.cc:493] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:46.230763  1605 raft_consensus.cc:2368] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for earlier term 16. Current term is 17.
I20260709 13:01:46.177371  7712 consensus_queue.cc:1059] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 14.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.018s
I20260709 13:01:46.171706  7578 raft_consensus.cc:515] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:46.222713  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:46.236425  1599 raft_consensus.cc:2393] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 15: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:01:46.228137  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:46.233371  1717 leader_election.cc:304] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 13:01:46.203356  7669 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.259s	user 0.007s	sys 0.023s
I20260709 13:01:46.234536  7570 raft_consensus.cc:2804] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:01:46.234792  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:46.234970  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:46.229276  7338 raft_consensus.cc:493] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:46.245744  7743 raft_consensus.cc:2749] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
W20260709 13:01:46.246373  7736 log.cc:927] Time spent T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.063s	user 0.000s	sys 0.000s
I20260709 13:01:46.238162  7668 raft_consensus.cc:2804] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:01:46.229527  1602 raft_consensus.cc:2468] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 17.
I20260709 13:01:46.194839  7734 raft_consensus.cc:493] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:46.250592  1569 leader_election.cc:304] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:01:46.202184  7582 raft_consensus.cc:3055] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 16 LEADER]: Stepping down as leader of term 16
I20260709 13:01:46.289726  1717 leader_election.cc:304] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:46.201539  7468 raft_consensus.cc:493] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:46.206861  7741 raft_consensus.cc:2749] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 13:01:46.296268  1603 raft_consensus.cc:2376] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 17. Re-sending same reply.
I20260709 13:01:46.249929  7590 raft_consensus.cc:3060] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:01:46.201922  7700 raft_consensus.cc:493] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:46.325594  7700 raft_consensus.cc:3060] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:01:46.303740  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:46.306134  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:01:46.324342  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 73858.2us
I20260709 13:01:46.206449  7491 raft_consensus.cc:493] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:46.332257  1598 raft_consensus.cc:1275] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 16 index: 11. (index mismatch)
I20260709 13:01:46.290709  7338 raft_consensus.cc:3060] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:01:46.332715  1717 leader_election.cc:304] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:46.338459  1569 leader_election.cc:304] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:01:46.291044  7570 raft_consensus.cc:493] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:46.341120  1717 leader_election.cc:304] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:46.346263  1678 raft_consensus.cc:1275] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 12 index: 8. (index mismatch)
I20260709 13:01:46.301237  7668 raft_consensus.cc:493] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:46.351989  1717 leader_election.cc:304] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:46.287549  7181 raft_consensus.cc:515] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:46.306022  7582 raft_consensus.cc:740] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:01:46.320928  7743 raft_consensus.cc:2749] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:01:46.250672  7683 raft_consensus.cc:515] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:46.195909  7337 raft_consensus.cc:515] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } }
W20260709 13:01:46.373730  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 124063us
I20260709 13:01:46.375020  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:46.342123  7491 raft_consensus.cc:3060] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:01:46.373376  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:46.340097  7483 raft_consensus.cc:493] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:46.341204  7739 raft_consensus.cc:493] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:46.386507  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:46.344094  7570 raft_consensus.cc:3060] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:01:46.395821  7745 raft_consensus.cc:2749] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
W20260709 13:01:46.397905  7742 log.cc:927] Time spent T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.065s	user 0.000s	sys 0.000s
W20260709 13:01:46.399205  7686 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.266s	user 0.012s	sys 0.014s
W20260709 13:01:46.399667  7686 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.351s	user 0.012s	sys 0.014s
I20260709 13:01:46.348215  7714 raft_consensus.cc:493] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:46.361454  7668 raft_consensus.cc:3060] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:01:46.404235  1673 raft_consensus.cc:2393] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 11: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:01:46.389786  1646 leader_election.cc:304] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:46.350776  7741 raft_consensus.cc:2804] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 13:01:46.348047  7738 raft_consensus.cc:493] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:46.359989  7631 raft_consensus.cc:2804] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:01:46.310940  7734 raft_consensus.cc:515] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:46.330684  7468 raft_consensus.cc:515] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:46.381742  7687 raft_consensus.cc:2749] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:01:46.336570  7688 consensus_queue.cc:1048] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.046s
I20260709 13:01:46.426622  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:46.436241  1672 raft_consensus.cc:2468] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 12.
W20260709 13:01:46.436811  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:46.425992  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 45563.6us
I20260709 13:01:46.432013  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:46.439428  7729 consensus_queue.cc:1059] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 13.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.137s
I20260709 13:01:46.441015  1601 raft_consensus.cc:2468] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 12.
I20260709 13:01:46.441419  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:46.408687  7743 raft_consensus.cc:3060] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:01:46.417404  7483 raft_consensus.cc:515] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:46.453153  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 15 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:46.420183  7745 raft_consensus.cc:3055] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 LEADER]: Stepping down as leader of term 15
I20260709 13:01:46.427243  7631 raft_consensus.cc:493] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:46.458890  7483 leader_election.cc:290] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:46.432824  7737 raft_consensus.cc:493] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:46.435405  6882 raft_consensus.cc:493] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:46.433684  7741 raft_consensus.cc:697] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 20 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:01:46.441776  7691 leader_election.cc:290] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 17 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:46.471333  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 18 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:46.472755  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 15 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:46.446367  7710 leader_election.cc:290] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:46.481086  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 17 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:46.485026  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:46.485808  1754 raft_consensus.cc:2468] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 12.
I20260709 13:01:46.456452  7745 raft_consensus.cc:740] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:01:46.461915  7631 raft_consensus.cc:3060] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Advancing to term 17
W20260709 13:01:46.490835  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 138977us
I20260709 13:01:46.460057  7693 leader_election.cc:290] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 17 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:46.409308  7739 raft_consensus.cc:515] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:46.495064  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:46.497469  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:46.435930  7714 raft_consensus.cc:515] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:46.503649  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:46.506156  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:46.506980  1600 raft_consensus.cc:2468] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 15.
I20260709 13:01:46.479815  7580 raft_consensus.cc:493] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:46.509711  1674 raft_consensus.cc:2468] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 11.
I20260709 13:01:46.521690  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:46.524004  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:46.524933  1598 raft_consensus.cc:3060] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:01:46.510213  7714 leader_election.cc:290] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:46.475136  6882 raft_consensus.cc:515] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:01:46.512234  7746 log.cc:927] Time spent T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.138s	user 0.001s	sys 0.000s
I20260709 13:01:46.512566  7754 raft_consensus.cc:2804] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:01:46.571872  7754 raft_consensus.cc:493] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:01:46.572429  7754 raft_consensus.cc:3060] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:01:46.572990  7590 raft_consensus.cc:2749] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:01:46.575821  7337 leader_election.cc:290] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:46.455139  7582 consensus_queue.cc:260] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 16.8, Last appended by leader: 8, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:46.578496  7582 raft_consensus.cc:3060] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:01:46.526721  7580 raft_consensus.cc:515] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:46.580617  7745 consensus_queue.cc:260] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.9, Last appended by leader: 9, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:46.513767  7741 consensus_queue.cc:237] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.13, Last appended by leader: 13, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:46.582556  7745 raft_consensus.cc:3060] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:01:46.428253  7738 raft_consensus.cc:515] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:46.588034  7738 leader_election.cc:290] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:46.590535  6882 leader_election.cc:290] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:46.524016  1717 leader_election.cc:304] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
W20260709 13:01:46.511186  7747 log.cc:927] Time spent T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.150s	user 0.001s	sys 0.000s
I20260709 13:01:46.533511  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:46.479060  7737 raft_consensus.cc:515] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:46.537043  7674 log.cc:927] Time spent T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 1.653s	user 0.004s	sys 0.009s
W20260709 13:01:46.538513  2300 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:46.539260  7713 leader_election.cc:290] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:01:46.540948  7709 log.cc:927] Time spent T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 1.277s	user 0.002s	sys 0.000s
I20260709 13:01:46.510149  7752 consensus_queue.cc:1048] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 8, Time since last communication: 0.192s
I20260709 13:01:46.542202  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 17 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:46.543017  7731 leader_election.cc:290] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:46.543339  7761 raft_consensus.cc:493] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:46.544411  7734 leader_election.cc:290] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:46.544958  7683 leader_election.cc:290] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:46.545986  7578 leader_election.cc:290] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:46.592751  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:46.621937  1754 raft_consensus.cc:2393] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 17: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
W20260709 13:01:46.592967  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 69061.7us
I20260709 13:01:46.623328  1717 leader_election.cc:304] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 13:01:46.582108  7580 leader_election.cc:290] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:46.626703  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:46.631209  1598 raft_consensus.cc:2468] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 17.
I20260709 13:01:46.632040  7338 raft_consensus.cc:515] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:46.633745  7739 leader_election.cc:290] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:46.609761  7590 raft_consensus.cc:2804] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 13:01:46.634619  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:46.635489  1675 raft_consensus.cc:2393] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 11: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:01:46.644066  7570 raft_consensus.cc:515] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:46.614097  7668 raft_consensus.cc:515] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:46.653092  7668 leader_election.cc:290] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 16 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:46.657213  7658 raft_consensus.cc:3055] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 LEADER]: Stepping down as leader of term 13
I20260709 13:01:46.657675  7743 raft_consensus.cc:2749] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:01:46.657791  7658 raft_consensus.cc:740] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:01:46.658869  7658 consensus_queue.cc:260] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 13.9, Last appended by leader: 9, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:01:46.660014  1644 connection.cc:256] RPC call timeout handler was delayed by -1.5700157450000347s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15580000000000002s for a response
I20260709 13:01:46.660303  7658 raft_consensus.cc:3060] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Advancing to term 14
W20260709 13:01:46.660573  7711 log.cc:927] Time spent T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 1.565s	user 0.006s	sys 0.006s
I20260709 13:01:46.662464  7712 raft_consensus.cc:493] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:46.663017  7712 raft_consensus.cc:515] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:46.620958  7702 raft_consensus.cc:493] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:46.664760  7702 raft_consensus.cc:515] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } }
I20260709 13:01:46.665431  7712 leader_election.cc:290] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:46.667490  7702 leader_election.cc:290] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:46.625321  7761 raft_consensus.cc:515] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:46.669660  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:46.673374  1754 raft_consensus.cc:1240] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 12. Current term is 13. Ops: []
I20260709 13:01:46.671641  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:01:46.707566  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 84305.6us
I20260709 13:01:46.715431  1717 leader_election.cc:304] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:46.708622  1747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55840 (request call id 2828) took 2715 ms. Trace:
I20260709 13:01:46.717519  1747 rpcz_store.cc:276] 0709 13:01:43.960538 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:44.092493 (+131955us) service_pool.cc:224] Handling call
0709 13:01:44.140025 (+ 47532us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:44.483965 (+343940us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:44.484026 (+    61us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:01:44.494714 (+ 10688us) log.cc:844] Serialized 26 byte log entry
0709 13:01:44.917564 (+422850us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:01:45.279870 (+362306us) log.cc:844] Serialized 14 byte log entry
0709 13:01:45.281709 (+  1839us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:45.281800 (+    91us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:46.619695 (+1337895us) spinlock_profiling.cc:243] Waited 23.8 ms on lock 0x7b4800121ed8. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426b08baa1 00007f426b082b37 00007f426b0829ca 00007f426b17a26f 00007f426b179f37 00007f426b18e55c 00007f426b18e508 00007f426b18e488 00007f426b18e443 00007f426b18d6ec 00007f426a0b7a96
0709 13:01:46.659493 (+ 39798us) spinlock_profiling.cc:243] Waited 7 ms on lock 0x7b4400022b4c. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f4262f9bc85 00007f4262f9c6a2 00007f4265d0e1db 00007f4265d09aaf 00007f4265cfd86e 00007f4265cfcbb6 00007f4265cfc7d8 00007f4265d06221 00007f4265d061d9 00007f4265d06169
0709 13:01:46.675862 (+ 16369us) raft_consensus.cc:1685] finished
0709 13:01:46.675888 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:46.676014 (+   126us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":84335,"prepare.queue_time_us":698184,"prepare.run_cpu_time_us":68,"prepare.run_wall_time_us":70,"raft.queue_time_us":430214,"raft.run_cpu_time_us":8520,"raft.run_wall_time_us":2062305,"spinlock_wait_cycles":70793856,"thread_start_us":958102,"threads_started":3,"wal-append.queue_time_us":558120}
I20260709 13:01:46.723655  1569 leader_election.cc:304] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
W20260709 13:01:46.720686  7709 log.cc:927] Time spent T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.099s	user 0.001s	sys 0.000s
I20260709 13:01:46.732810  1745 raft_consensus.cc:1275] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 16 index: 11. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
W20260709 13:01:46.743391  1571 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:46.737331  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:46.711335  7697 raft_consensus.cc:2804] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:01:46.707180  7338 leader_election.cc:290] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 17 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:46.707494  7590 raft_consensus.cc:697] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 17 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:01:46.731402  7739 consensus_queue.cc:1048] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.040s
I20260709 13:01:46.771987  7590 consensus_queue.cc:237] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.8, Last appended by leader: 8, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:46.737468  7413 raft_consensus.cc:493] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:01:46.753191  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:46.755795  7737 leader_election.cc:290] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:46.757032  7744 raft_consensus.cc:493] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
W20260709 13:01:46.759793  1719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:46.737260  7728 raft_consensus.cc:2749] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:01:46.767691  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:01:46.769443  7719 log.cc:927] Time spent T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 1.688s	user 0.014s	sys 0.029s
I20260709 13:01:46.749823  7697 raft_consensus.cc:493] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:01:46.770761  7570 leader_election.cc:290] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 16 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:46.773029  1677 raft_consensus.cc:2468] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 17.
I20260709 13:01:46.704496  7761 leader_election.cc:290] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:46.752179  1569 leader_election.cc:304] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:01:46.804157  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:46.942926  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:46.806355  2299 leader_election.cc:304] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
W20260709 13:01:46.944639  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 138419us
I20260709 13:01:46.945664  2299 leader_election.cc:304] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
W20260709 13:01:46.849583  7711 log.cc:927] Time spent T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.100s	user 0.000s	sys 0.000s
I20260709 13:01:46.839859  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:46.949296  1599 raft_consensus.cc:2468] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 13.
I20260709 13:01:46.841753  1571 leader_election.cc:304] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:46.862840  1601 raft_consensus.cc:2468] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 12.
I20260709 13:01:46.942106  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:46.951252  1747 raft_consensus.cc:2468] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 15.
I20260709 13:01:46.738265  7702 raft_consensus.cc:2804] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 13:01:46.949753  2299 leader_election.cc:304] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:46.831844  7580 raft_consensus.cc:493] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:46.955225  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:46.968945  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:46.971501  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:46.972429  1601 raft_consensus.cc:3060] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:01:46.974375  1752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43156 (request call id 1216) took 1079 ms. Trace:
W20260709 13:01:46.943974  7584 consensus_meta.cc:287] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.778s	user 0.017s	sys 0.000s
W20260709 13:01:46.947711  7700 consensus_meta.cc:287] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.622s	user 0.015s	sys 0.001s
I20260709 13:01:46.907315  7582 raft_consensus.cc:493] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
W20260709 13:01:46.980897  2300 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:46.975880  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:46.947338  7697 raft_consensus.cc:3060] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:01:47.008862  1748 raft_consensus.cc:1275] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 12 index: 8. (index mismatch)
I20260709 13:01:46.955134  7575 raft_consensus.cc:2749] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:01:47.032708  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:47.034129  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:47.035020  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 15 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:46.975080  1752 rpcz_store.cc:276] 0709 13:01:45.895243 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:45.895497 (+   254us) service_pool.cc:224] Handling call
0709 13:01:45.903148 (+  7651us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:45.903421 (+   273us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 13:01:45.910508 (+  7087us) log.cc:844] Serialized 14 byte log entry
0709 13:01:46.721205 (+810697us) spinlock_profiling.cc:243] Waited 3.01 ms on lock 0x7b1800455740. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f4263057e07 00007f426beb2ab9 00007f426307469b 00007f426307293f 00007f4263074c02 00007f4265cfc20b 00007f4265d01d23 00007f4265d0223a 00007f426616ca2a
0709 13:01:46.973849 (+252644us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:01:46.973923 (+    74us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:01:46.974006 (+    83us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:46.974133 (+   127us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:46.974282 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":18669,"prepare.queue_time_us":6880,"prepare.run_cpu_time_us":91,"prepare.run_wall_time_us":5583,"raft.queue_time_us":22511,"spinlock_wait_cycles":6915968,"thread_start_us":233480,"threads_started":1}
I20260709 13:01:46.977854  7570 raft_consensus.cc:2749] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:01:46.974751  7724 raft_consensus.cc:2749] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:01:47.075644  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:46.839026  7413 raft_consensus.cc:515] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:47.006906  7577 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.894s	user 0.017s	sys 0.043s
I20260709 13:01:47.024129  7685 raft_consensus.cc:493] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:47.069870  1747 raft_consensus.cc:2393] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 18: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:01:46.978255  7489 raft_consensus.cc:2749] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 13:01:46.856120  7744 raft_consensus.cc:515] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:47.078794  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:47.116025  1752 raft_consensus.cc:3060] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:01:46.976765  7580 raft_consensus.cc:515] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:47.120744  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:47.146987  1747 raft_consensus.cc:2468] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 17.
W20260709 13:01:47.147193  7631 consensus_meta.cc:287] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.644s	user 0.011s	sys 0.003s
I20260709 13:01:47.147845  7631 raft_consensus.cc:515] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:47.148177  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:47.122876  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:01:47.124893  7768 log.cc:927] Time spent T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.052s	user 0.000s	sys 0.000s
I20260709 13:01:47.109279  7685 raft_consensus.cc:515] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:47.148355  7580 leader_election.cc:290] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:47.004508  7700 raft_consensus.cc:515] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:47.073243  7702 raft_consensus.cc:697] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 17 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:01:47.012254  7582 raft_consensus.cc:515] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:47.394301  1747 raft_consensus.cc:3060] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:01:47.327200  7677 leader_election.cc:290] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:01:47.324529  7692 log.cc:927] Time spent T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 2.333s	user 0.014s	sys 0.025s
W20260709 13:01:47.327661  1718 connection.cc:256] RPC call timeout handler was delayed by -1.7395602719999488s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1815s for a response
W20260709 13:01:47.381260  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:47.399564  7743 raft_consensus.cc:493] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:47.400254  7743 raft_consensus.cc:515] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:47.402849  1718 connection.cc:256] RPC call timeout handler was delayed by -1.4273902200000066s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2183s for a response
W20260709 13:01:47.403534  1718 connection.cc:256] RPC call timeout handler was delayed by -1.2829619879999541s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1698s for a response
W20260709 13:01:47.391264  1645 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:47.404201  7765 raft_consensus.cc:493] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:47.404982  7765 raft_consensus.cc:515] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:01:47.405450  1645 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:47.408057  7437 raft_consensus.cc:493] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:01:47.393357  7491 consensus_meta.cc:287] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.976s	user 0.011s	sys 0.006s
I20260709 13:01:47.408782  7491 raft_consensus.cc:515] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:47.410989  1749 raft_consensus.cc:1240] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 12. Current term is 13. Ops: []
W20260709 13:01:47.414280  7745 consensus_meta.cc:287] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.831s	user 0.014s	sys 0.001s
I20260709 13:01:47.281311  1754 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55840 (request call id 2835) took 1200 ms. Trace:
W20260709 13:01:47.441247  1718 connection.cc:256] RPC call timeout handler was delayed by -1.1228102369999533s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17190000000000003s for a response
W20260709 13:01:47.330953  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 489232us
W20260709 13:01:47.330659  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 581259us
I20260709 13:01:47.455631  1569 leader_election.cc:304] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:01:47.408711  7437 raft_consensus.cc:515] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:47.199986  1751 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55840 (request call id 2821) took 3612 ms. Trace:
W20260709 13:01:47.481942  1676 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.513s	user 0.019s	sys 0.007s
W20260709 13:01:47.482720  1676 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.513s	user 0.019s	sys 0.007s
I20260709 13:01:47.166178  1752 raft_consensus.cc:2468] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 16.
I20260709 13:01:47.492838  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:01:47.509483  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 559697us
I20260709 13:01:47.510648  2299 leader_election.cc:304] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:47.155408  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:47.419039  1754 rpcz_store.cc:276] 0709 13:01:46.081452 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:46.674260 (+592808us) service_pool.cc:224] Handling call
0709 13:01:46.720924 (+ 46664us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:46.721417 (+   493us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:46.721451 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:01:46.749555 (+ 28104us) log.cc:844] Serialized 26 byte log entry
0709 13:01:47.038747 (+289192us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:01:47.039041 (+   294us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:47.039160 (+   119us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:47.281054 (+241894us) raft_consensus.cc:1685] finished
0709 13:01:47.281076 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:47.281235 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":3554,"raft.run_cpu_time_us":187,"raft.run_wall_time_us":5876,"thread_start_us":283787,"threads_started":1,"wal-append.queue_time_us":439826}
W20260709 13:01:47.083443  7577 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.977s	user 0.017s	sys 0.043s
I20260709 13:01:47.420874  1672 raft_consensus.cc:2468] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 16.
W20260709 13:01:47.513584  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 58056.1us
I20260709 13:01:47.524719  1569 leader_election.cc:304] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:01:47.526841  1569 leader_election.cc:304] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:01:47.455881  1718 leader_election.cc:304] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:47.519356  1747 raft_consensus.cc:2468] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 17.
I20260709 13:01:47.530591  1747 raft_consensus.cc:1275] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 12 index: 8. (index mismatch)
W20260709 13:01:47.532123  2301 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:47.532871  2301 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:47.329022  7775 log.cc:927] Time spent T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.088s	user 0.001s	sys 0.000s
I20260709 13:01:47.520931  1571 leader_election.cc:304] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
W20260709 13:01:47.536011  1571 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:47.540406  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 84669.6us
I20260709 13:01:47.542096  1680 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52766 (request call id 2899) took 3820 ms. Trace:
I20260709 13:01:47.542711  1680 rpcz_store.cc:276] 0709 13:01:43.706965 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:43.707159 (+   194us) service_pool.cc:224] Handling call
0709 13:01:43.707622 (+   463us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:43.722489 (+ 14867us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:43.722544 (+    55us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:01:44.022993 (+300449us) log.cc:844] Serialized 26 byte log entry
0709 13:01:44.492275 (+469282us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:01:44.492336 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:44.492430 (+    94us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:47.323631 (+2831201us) spinlock_profiling.cc:243] Waited 21.9 ms on lock 0x7b4400022b4c. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f4262f9bc85 00007f4262f9c6a2 00007f4265d0e1db 00007f4265d09aaf 00007f4265cfd86e 00007f4265cfcbb6 00007f4265cfc7d8 00007f4265d06221 00007f4265d061d9 00007f4265d06169
0709 13:01:47.513164 (+189533us) raft_consensus.cc:1685] finished
0709 13:01:47.513198 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:47.527168 (+ 13970us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1090,"raft.queue_time_us":20702,"raft.run_cpu_time_us":270,"raft.run_wall_time_us":23385,"spinlock_wait_cycles":50407808,"thread_start_us":348159,"threads_started":1,"wal-append.queue_time_us":818837}
W20260709 13:01:47.278992  7766 log.cc:927] Time spent T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.129s	user 0.000s	sys 0.001s
I20260709 13:01:47.479235  1751 rpcz_store.cc:276] 0709 13:01:43.587725 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:44.048584 (+460859us) service_pool.cc:224] Handling call
0709 13:01:44.049137 (+   553us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:44.049493 (+   356us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:44.049531 (+    38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:01:44.092853 (+ 43322us) log.cc:844] Serialized 26 byte log entry
0709 13:01:44.964025 (+871172us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:01:45.301754 (+337729us) log.cc:844] Serialized 14 byte log entry
0709 13:01:45.301989 (+   235us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:45.302065 (+    76us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:46.768642 (+1466577us) spinlock_profiling.cc:243] Waited 23.9 ms on lock 0x7b4400022b4c. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f4262f9bc85 00007f4262f9c6a2 00007f4265d0e1db 00007f4265d09aaf 00007f4265cfd86e 00007f4265cfcbb6 00007f4265cfc7d8 00007f4265d06221 00007f4265d061d9 00007f4265d06169
0709 13:01:47.168629 (+399987us) raft_consensus.cc:1685] finished
0709 13:01:47.199669 (+ 31040us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:47.199886 (+   217us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":53408,"prepare.queue_time_us":843056,"prepare.run_cpu_time_us":101,"prepare.run_wall_time_us":6560,"raft.queue_time_us":15193,"raft.run_cpu_time_us":249,"raft.run_wall_time_us":129800,"spinlock_wait_cycles":54903552,"thread_start_us":984839,"threads_started":2,"wal-append.queue_time_us":921168}
I20260709 13:01:47.544140  1718 leader_election.cc:304] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
W20260709 13:01:47.550084  2300 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:47.331826  7687 raft_consensus.cc:2804] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:01:47.553982  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 15 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:47.555968  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 18 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:47.557155  1752 raft_consensus.cc:2393] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 18: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:01:47.240494  7489 raft_consensus.cc:493] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:47.555511  1754 raft_consensus.cc:2468] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 14.
I20260709 13:01:47.331492  7750 raft_consensus.cc:2804] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:01:47.526624  7739 raft_consensus.cc:2749] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:01:47.560441  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:47.531414  7580 raft_consensus.cc:2749] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:01:47.531942  7706 raft_consensus.cc:2749] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 13:01:47.539624  7686 raft_consensus.cc:2804] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:01:47.259613  7413 leader_election.cc:290] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:47.576829  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:47.548465  7647 raft_consensus.cc:2804] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:01:47.540011  7727 raft_consensus.cc:493] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:47.585098  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:47.605963  1752 raft_consensus.cc:2468] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 17.
I20260709 13:01:47.553413  7740 raft_consensus.cc:2749] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 13:01:47.598488  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:47.564637  7750 raft_consensus.cc:493] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
W20260709 13:01:47.578653  7768 log.cc:927] Time spent T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.053s	user 0.000s	sys 0.001s
I20260709 13:01:47.555181  7687 raft_consensus.cc:493] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:47.612928  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:47.619347  1754 raft_consensus.cc:1240] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 16. Current term is 17. Ops: []
I20260709 13:01:47.625850  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:47.626909  7307 leader_election.cc:290] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 18 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:47.635706  1675 raft_consensus.cc:2468] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 11.
I20260709 13:01:47.628479  7580 raft_consensus.cc:493] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:47.636904  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:47.636791  7580 raft_consensus.cc:515] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:47.626902  2331 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49082 (request call id 1088) took 3585 ms (client timeout 2760 ms). Trace:
W20260709 13:01:47.638667  2331 rpcz_store.cc:269] 0709 13:01:44.041446 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:44.078216 (+ 36770us) service_pool.cc:224] Handling call
0709 13:01:47.626664 (+3548448us) spinlock_profiling.cc:243] Waited 115 ms on lock 0x7b54006beed4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:01:47.626806 (+   142us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":265284864}
I20260709 13:01:47.639582  7580 leader_election.cc:290] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:47.629447  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:47.641896  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:47.642182  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:47.642629  7659 leader_election.cc:290] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:47.643095  1676 raft_consensus.cc:2410] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for term 13 because replica has last-logged OpId of term: 11 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 7.
I20260709 13:01:47.643698  1675 raft_consensus.cc:1240] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 16. Current term is 17. Ops: []
I20260709 13:01:47.644240  1676 raft_consensus.cc:1240] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 15. Current term is 16. Ops: []
I20260709 13:01:47.644490  1603 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 2722) took 1410 ms. Trace:
I20260709 13:01:47.641326  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:47.646915  7496 raft_consensus.cc:493] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:47.647571  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:47.647511  7496 raft_consensus.cc:515] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:47.648219  2329 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39738 (request call id 1087) took 2448 ms. Trace:
I20260709 13:01:47.648976  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:47.649714  1752 raft_consensus.cc:2468] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 15.
I20260709 13:01:47.649890  1676 raft_consensus.cc:1240] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 10. Current term is 11. Ops: []
I20260709 13:01:47.586777  7686 raft_consensus.cc:493] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:47.648628  2329 rpcz_store.cc:276] 0709 13:01:45.200121 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:45.200304 (+   183us) service_pool.cc:224] Handling call
0709 13:01:45.227379 (+ 27075us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:45.760775 (+533396us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:47.647905 (+1887130us) spinlock_profiling.cc:243] Waited 459 ms on lock 0x7b54007d2654. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:01:47.648115 (+   210us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1055803136}
I20260709 13:01:47.644907  1603 rpcz_store.cc:276] 0709 13:01:46.234336 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:46.337340 (+103004us) service_pool.cc:224] Handling call
0709 13:01:46.359666 (+ 22326us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:46.454705 (+ 95039us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 13:01:46.601200 (+146495us) log.cc:844] Serialized 14 byte log entry
0709 13:01:46.802869 (+201669us) log.cc:844] Serialized 14 byte log entry
0709 13:01:47.329905 (+527036us) spinlock_profiling.cc:243] Waited 54.2 ms on lock 0x7b180005ea80. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f4262fefc0a 00007f426c72f219 00007f4263077a40 00007f42630724e4 00007f4263072455 00007f426615a060 00007f426616c716 00007f426617d287 00007f426617d208
0709 13:01:47.330123 (+   218us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:01:47.330182 (+    59us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:01:47.330238 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:47.644207 (+313969us) spinlock_profiling.cc:243] Waited 94.7 ms on lock 0x7b54002175d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:01:47.644275 (+    68us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:47.644414 (+   139us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":456143,"prepare.queue_time_us":56012,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":36,"raft.queue_time_us":528707,"raft.run_cpu_time_us":283,"raft.run_wall_time_us":5074,"spinlock_wait_cycles":433903616,"thread_start_us":334279,"threads_started":2,"wal-append.queue_time_us":377034}
W20260709 13:01:47.622984  2301 connection.cc:256] RPC call timeout handler was delayed by -1.0586036589999708s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20299999999999999s for a response
I20260709 13:01:47.546815  7744 leader_election.cc:290] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:47.594319  7743 leader_election.cc:290] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:47.578959  7577 raft_consensus.cc:493] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:47.630686  7750 raft_consensus.cc:3060] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:01:47.668989  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:47.669766  2333 raft_consensus.cc:2468] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 17.
I20260709 13:01:47.675896  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:47.661509  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:47.682077  2329 raft_consensus.cc:2393] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 13: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:01:47.682968  7685 leader_election.cc:290] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:47.668576  7631 leader_election.cc:290] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 17 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:47.688897  7540 raft_consensus.cc:3060] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:01:47.661963  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:47.664259  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:47.665174  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:47.615512  7761 raft_consensus.cc:2749] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:01:47.691661  2332 raft_consensus.cc:2468] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 11.
I20260709 13:01:47.677378  7765 leader_election.cc:290] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:47.678658  2330 raft_consensus.cc:1275] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 14 index: 10. Preceding OpId from leader: term: 16 index: 11. (index mismatch)
I20260709 13:01:47.678992  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:47.679312  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:47.604147  7647 raft_consensus.cc:493] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:47.651680  7687 raft_consensus.cc:3060] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:01:47.627709  7684 raft_consensus.cc:493] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:47.658689  7686 raft_consensus.cc:3060] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:01:47.657339  7489 raft_consensus.cc:515] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:47.620406  7727 raft_consensus.cc:515] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:47.716378  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:47.720688  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:47.709946  7647 raft_consensus.cc:3060] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:01:47.725677  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 18 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:01:47.736667  1601 consensus_meta.cc:287] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.761s	user 0.014s	sys 0.001s
I20260709 13:01:47.737294  1601 raft_consensus.cc:2468] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 16.
I20260709 13:01:47.740468  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
W20260709 13:01:47.742161  7779 log.cc:927] Time spent T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.126s	user 0.000s	sys 0.000s
I20260709 13:01:47.742259  7489 leader_election.cc:290] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:47.745744  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:47.746099  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:47.746872  2333 raft_consensus.cc:2468] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 13.
I20260709 13:01:47.747323  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:47.747614  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:47.751232  1601 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36462 (request call id 1492) took 1025 ms. Trace:
I20260709 13:01:47.751785  1601 rpcz_store.cc:276] 0709 13:01:46.712345 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:46.971094 (+258749us) service_pool.cc:224] Handling call
0709 13:01:47.737927 (+766833us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:01:47.633713  7702 consensus_queue.cc:237] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.10, Last appended by leader: 10, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:47.754968  1673 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57092 (request call id 1332) took 1212 ms. Trace:
I20260709 13:01:47.771718  1673 rpcz_store.cc:276] 0709 13:01:46.542216 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:46.610505 (+ 68289us) service_pool.cc:224] Handling call
0709 13:01:46.670544 (+ 60039us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:46.670788 (+   244us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:46.670814 (+    26us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:01:46.670882 (+    68us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:01:46.670925 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:46.671003 (+    78us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:47.754705 (+1083702us) spinlock_profiling.cc:243] Waited 43.2 ms on lock 0x7b54000d7d50. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661618ad 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:01:47.754864 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":99293568}
I20260709 13:01:47.698668  7577 raft_consensus.cc:515] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:47.739812  7496 leader_election.cc:290] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:47.714857  7684 raft_consensus.cc:515] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:47.786649  7582 leader_election.cc:290] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:47.795063  7761 raft_consensus.cc:493] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
W20260709 13:01:47.806313  7754 consensus_meta.cc:287] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 1.233s	user 0.017s	sys 0.001s
I20260709 13:01:47.803000  7700 leader_election.cc:290] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:47.807448  7437 leader_election.cc:290] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:47.826094  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:47.827819  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:01:47.837141  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 123966us
W20260709 13:01:47.833526  7777 log.cc:927] Time spent T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.206s	user 0.000s	sys 0.001s
I20260709 13:01:47.852349  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:47.853250  1598 raft_consensus.cc:2468] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 15.
I20260709 13:01:47.862309  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:47.863556  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:47.863153  1603 raft_consensus.cc:2468] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 12.
I20260709 13:01:47.864423  1599 raft_consensus.cc:2468] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 16.
I20260709 13:01:47.860603  2298 leader_election.cc:304] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:47.824225  7761 raft_consensus.cc:515] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:47.868356  2298 leader_election.cc:304] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:01:47.869244  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:47.824545  7754 raft_consensus.cc:515] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:47.873775  7579 raft_consensus.cc:2804] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Leader pre-election won for term 13
W20260709 13:01:47.867405  7658 consensus_meta.cc:287] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 1.205s	user 0.015s	sys 0.004s
I20260709 13:01:47.885413  7688 raft_consensus.cc:3055] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 13:01:47.886885  7723 raft_consensus.cc:3055] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:01:47.887651  7761 leader_election.cc:290] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:47.891585  1603 raft_consensus.cc:1240] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 13. Current term is 14. Ops: []
I20260709 13:01:47.893555  1599 raft_consensus.cc:2468] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 20.
W20260709 13:01:47.899883  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 355834us
W20260709 13:01:47.900476  1643 connection.cc:256] RPC call timeout handler was delayed by -2.3975815080000302s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15580000000000002s for a response
I20260709 13:01:47.953303  1571 leader_election.cc:304] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
W20260709 13:01:47.954020  1643 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 1.840s (SENT) [suppressed 12 similar messages]
I20260709 13:01:47.936115  2298 leader_election.cc:304] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:01:47.954941  1718 leader_election.cc:304] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:47.878633  7745 raft_consensus.cc:493] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:47.956360  2298 leader_election.cc:304] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:01:47.959465  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:47.961825  1718 leader_election.cc:304] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:47.969521  7743 raft_consensus.cc:2749] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:01:47.969668  1718 leader_election.cc:304] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:47.970489  7743 consensus_queue.cc:1048] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 13:01:47.970937  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:47.971573  7685 raft_consensus.cc:2749] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:01:47.971724  2335 raft_consensus.cc:2468] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 17.
W20260709 13:01:47.974853  7708 log.cc:927] Time spent T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 2.901s	user 0.021s	sys 0.036s
I20260709 13:01:47.976491  1750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43156 (request call id 1214) took 3974 ms. Trace:
I20260709 13:01:47.977007  1750 rpcz_store.cc:276] 0709 13:01:44.001975 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:44.282046 (+280071us) service_pool.cc:224] Handling call
0709 13:01:44.282599 (+   553us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:01:44.482183 (+199584us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:44.482246 (+    63us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:01:44.512636 (+ 30390us) log.cc:844] Serialized 52 byte log entry
0709 13:01:44.996276 (+483640us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:01:44.996331 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:44.996436 (+   105us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:47.974147 (+2977711us) spinlock_profiling.cc:243] Waited 23.3 ms on lock 0x7b4400022b4c. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f4262f9bc85 00007f4262f9c6a2 00007f4265d0e1db 00007f4265d09aaf 00007f4265cfd86e 00007f4265cfcbb6 00007f4265cfc7d8 00007f4265d06221 00007f4265d061d9 00007f4265d06169
0709 13:01:47.976008 (+  1861us) raft_consensus.cc:1685] finished
0709 13:01:47.976048 (+    40us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:47.976399 (+   351us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":70922,"raft.queue_time_us":466850,"raft.run_cpu_time_us":204,"raft.run_wall_time_us":22965,"spinlock_wait_cycles":53673856,"thread_start_us":634728,"threads_started":2,"wal-append.queue_time_us":536822}
I20260709 13:01:47.980948  7577 leader_election.cc:290] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:47.981359  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:47.982596  7181 leader_election.cc:290] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:47.982661  2332 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f"
dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
mode: GRACEFUL
new_leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
 from {username='slave'} at 127.0.0.1:35790
I20260709 13:01:47.983520  2332 raft_consensus.cc:606] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 12 LEADER]: Received request to transfer leadership to TS 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:47.983847  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:47.980940  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:47.985173  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:47.985864  2337 raft_consensus.cc:2376] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 11. Re-sending same reply.
I20260709 13:01:47.986867  1675 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57092 (request call id 1342) took 1286 ms. Trace:
I20260709 13:01:47.987370  7694 leader_election.cc:290] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:01:47.987742  1605 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40166 (request call id 2714) took 1793 ms (client timeout 2074 ms). Trace:
I20260709 13:01:47.945401  7688 raft_consensus.cc:740] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:01:47.988185  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:01:47.988336  1605 rpcz_store.cc:269] 0709 13:01:46.194100 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:46.198096 (+  3996us) service_pool.cc:224] Handling call
0709 13:01:47.987521 (+1789425us) spinlock_profiling.cc:243] Waited 5.09 ms on lock 0x7b54000b1154. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:01:47.987670 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":11712256}
I20260709 13:01:47.989033  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:47.989744  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:47.989373  7688 consensus_queue.cc:260] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.8, Last appended by leader: 8, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:47.990965  7688 raft_consensus.cc:3060] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:01:47.991907  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
W20260709 13:01:47.998396  1596 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36462 (request call id 1476) took 2175 ms (client timeout 1856 ms). Trace:
W20260709 13:01:47.999058  1596 rpcz_store.cc:269] 0709 13:01:45.823365 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:45.842804 (+ 19439us) service_pool.cc:224] Handling call
0709 13:01:47.998179 (+2155375us) spinlock_profiling.cc:243] Waited 1.08 s on lock 0x7b54002995d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:01:47.998317 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2487524224}
I20260709 13:01:47.987288  1675 rpcz_store.cc:276] 0709 13:01:46.700270 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:47.644481 (+944211us) service_pool.cc:224] Handling call
0709 13:01:47.644826 (+   345us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:47.650337 (+  5511us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:47.650367 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:01:47.652685 (+  2318us) log.cc:844] Serialized 26 byte log entry
0709 13:01:47.815135 (+162450us) raft_consensus.cc:1645] Marking committed up to 14
0709 13:01:47.898194 (+ 83059us) log.cc:844] Serialized 14 byte log entry
0709 13:01:47.898607 (+   413us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:47.898718 (+   111us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:47.986606 (+ 87888us) raft_consensus.cc:1685] finished
0709 13:01:47.986635 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:47.986792 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8814,"prepare.queue_time_us":73159,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":36,"raft.queue_time_us":913,"raft.run_cpu_time_us":251,"raft.run_wall_time_us":7191,"thread_start_us":194762,"threads_started":2,"wal-append.queue_time_us":176026}
I20260709 13:01:48.000342  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:48.000722  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:48.001565  1605 raft_consensus.cc:2393] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 17: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:01:48.001644  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 18 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:48.002404  7631 raft_consensus.cc:493] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:48.002753  2334 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39738 (request call id 1093) took 2794 ms. Trace:
I20260709 13:01:48.003186  7631 raft_consensus.cc:515] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:48.003264  2334 rpcz_store.cc:276] 0709 13:01:45.208796 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:45.208972 (+   176us) service_pool.cc:224] Handling call
0709 13:01:45.258939 (+ 49967us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:45.259616 (+   677us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:48.002543 (+2742927us) spinlock_profiling.cc:243] Waited 49.5 ms on lock 0x7b54004880d0. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661618ad 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:01:48.002683 (+   140us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":113734912}
I20260709 13:01:48.004451  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:47.994366  7559 raft_consensus.cc:2804] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:01:48.005764  7631 leader_election.cc:290] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:48.006281  7559 raft_consensus.cc:493] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:48.006848  7559 raft_consensus.cc:3060] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:01:48.002494  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:01:48.009083  7793 log.cc:927] Time spent T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.116s	user 0.000s	sys 0.001s
W20260709 13:01:48.010870  7780 log.cc:927] Time spent T 00000000000000000000000000000000 P 919f6392d4f44fd68987e2c5e7ec37ca: Append to log took a long time: real 0.196s	user 0.001s	sys 0.000s
I20260709 13:01:47.994328  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:48.011543  7740 raft_consensus.cc:493] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:48.012238  7740 raft_consensus.cc:515] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:48.016423  7754 leader_election.cc:290] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 14 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:47.892000  7686 raft_consensus.cc:515] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:48.017536  1600 raft_consensus.cc:1240] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 11. Current term is 12. Ops: []
I20260709 13:01:48.018551  1679 raft_consensus.cc:1240] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 13. Current term is 14. Ops: []
I20260709 13:01:48.019246  1753 raft_consensus.cc:2468] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 12.
I20260709 13:01:48.019927  1749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43156 (request call id 1221) took 2102 ms. Trace:
I20260709 13:01:48.020618  1753 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55848 (request call id 2940) took 2137 ms. Trace:
I20260709 13:01:48.021013  1753 rpcz_store.cc:276] 0709 13:01:45.883664 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:45.884239 (+   575us) service_pool.cc:224] Handling call
0709 13:01:48.020412 (+2136173us) spinlock_profiling.cc:243] Waited 2.13 s on lock 0x7b5400324bd4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:01:48.020519 (+   107us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4902004608}
I20260709 13:01:48.020316  1749 rpcz_store.cc:276] 0709 13:01:45.917857 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:46.198920 (+281063us) service_pool.cc:224] Handling call
0709 13:01:46.199252 (+   332us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:47.411529 (+1212277us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:48.019657 (+608128us) spinlock_profiling.cc:243] Waited 1.21 s on lock 0x7b5400323cd4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:01:48.019852 (+   195us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2785703296}
I20260709 13:01:48.025151  7686 leader_election.cc:290] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 12 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:48.026748  1599 raft_consensus.cc:1275] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 19 index: 13. Preceding OpId from leader: term: 20 index: 14. (index mismatch)
I20260709 13:01:48.028445  1599 raft_consensus.cc:1240] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 16. Current term is 17. Ops: [16.8-16.8]
I20260709 13:01:48.028491  1749 raft_consensus.cc:1240] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 16. Current term is 17. Ops: []
W20260709 13:01:48.028991  1646 connection.cc:256] RPC call timeout handler was delayed by -1.1998282010000025s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 1.5s for a response
I20260709 13:01:48.029990  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:48.030145  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 18 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:48.030683  1745 raft_consensus.cc:3060] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Advancing to term 14
W20260709 13:01:48.031030  1677 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52766 (request call id 2911) took 1625 ms (client timeout 1962 ms). Trace:
W20260709 13:01:48.031456  1677 rpcz_store.cc:269] 0709 13:01:46.405565 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:46.405749 (+   184us) service_pool.cc:224] Handling call
0709 13:01:48.030840 (+1625091us) spinlock_profiling.cc:243] Waited 335 ms on lock 0x7b54000d8754. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:01:48.030963 (+   123us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":770019584}
I20260709 13:01:48.032007  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:01:48.033123  1604 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36462 (request call id 1487) took 2045 ms (client timeout 2156 ms). Trace:
I20260709 13:01:48.033507  1750 raft_consensus.cc:1240] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 17. Current term is 18. Ops: []
W20260709 13:01:48.033510  1604 rpcz_store.cc:269] 0709 13:01:45.987566 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:45.990354 (+  2788us) service_pool.cc:224] Handling call
0709 13:01:48.032944 (+2042590us) spinlock_profiling.cc:243] Waited 47.6 ms on lock 0x7b5400071054. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:01:48.033059 (+   115us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":109561216}
I20260709 13:01:48.036469  7745 raft_consensus.cc:515] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:48.036919  1753 raft_consensus.cc:1275] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 13 index: 7. (term mismatch)
I20260709 13:01:48.037333  1753 pending_rounds.cc:85] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701: Aborting all ops after (but not including) 6
I20260709 13:01:48.037616  7740 leader_election.cc:290] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:47.940029  7671 raft_consensus.cc:2804] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 13:01:48.039680  1600 raft_consensus.cc:1275] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 13 index: 7. (index mismatch)
I20260709 13:01:48.041010  1600 raft_consensus.cc:1240] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 13. Current term is 14. Ops: [13.9-13.9]
W20260709 13:01:48.041828  7697 consensus_meta.cc:287] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 1.010s	user 0.011s	sys 0.003s
I20260709 13:01:47.950270  7723 raft_consensus.cc:740] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:01:48.037577  1753 pending_rounds.cc:107] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701: Aborting uncommitted NO_OP operation due to leader change: 12.7
I20260709 13:01:48.042726  1753 raft_consensus.cc:2889] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:01:48.043329  7684 leader_election.cc:290] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:48.043221  7723 consensus_queue.cc:260] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:48.044243  1596 raft_consensus.cc:2468] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 17.
I20260709 13:01:48.044562  7723 raft_consensus.cc:3060] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:01:48.046015  1679 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52766 (request call id 2900) took 4337 ms. Trace:
I20260709 13:01:48.046258  1748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55840 (request call id 2817) took 4468 ms. Trace:
I20260709 13:01:48.047057  7468 leader_election.cc:290] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:48.046733  1748 rpcz_store.cc:276] 0709 13:01:43.577471 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:43.578120 (+   649us) service_pool.cc:224] Handling call
0709 13:01:43.578527 (+   407us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:47.009509 (+3430982us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:48.045973 (+1036464us) spinlock_profiling.cc:243] Waited 3.41 s on lock 0x7b5400116cd4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:01:48.046162 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":7839026432}
I20260709 13:01:47.932994  7579 raft_consensus.cc:493] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:48.050318  7579 raft_consensus.cc:3060] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:01:47.958860  7727 leader_election.cc:290] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:47.995560  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:48.046577  1679 rpcz_store.cc:276] 0709 13:01:43.716707 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:43.729475 (+ 12768us) service_pool.cc:224] Handling call
0709 13:01:47.023892 (+3294417us) spinlock_profiling.cc:243] Waited 12.5 ms on lock 0x7b5800086a98. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f426303db71 00007f426303da22 00007f426bf56c09 00007f426b0b0ee0 00007f426b17a0ab 00007f426b17a131 00007f426b13bfc0 00007f426b126929 00007f426b1263d2 00007f426469dbd4
0709 13:01:47.024198 (+   306us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:48.018952 (+994754us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:48.045650 (+ 26698us) spinlock_profiling.cc:243] Waited 994 ms on lock 0x7b54000226d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:01:48.045891 (+   241us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2314679040}
I20260709 13:01:48.005702  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 15 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:01:48.072186  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 119026us
I20260709 13:01:48.075171  1745 raft_consensus.cc:2468] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 14.
W20260709 13:01:48.076409  1645 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:48.085142  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 128848us
W20260709 13:01:48.085825  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 557304us
I20260709 13:01:48.086756  1569 leader_election.cc:304] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:01:48.089706  1602 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40154 (request call id 2614) took 1583 ms. Trace:
I20260709 13:01:48.085537  1717 leader_election.cc:304] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:48.094591  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 18 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:48.086298  2298 leader_election.cc:304] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:01:48.097762  1676 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52766 (request call id 2922) took 1336 ms. Trace:
I20260709 13:01:48.090306  1602 rpcz_store.cc:276] 0709 13:01:46.506678 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:46.597781 (+ 91103us) service_pool.cc:224] Handling call
0709 13:01:46.598197 (+   416us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:46.598540 (+   343us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 13:01:46.612872 (+ 14332us) log.cc:844] Serialized 14 byte log entry
0709 13:01:47.965992 (+1353120us) spinlock_profiling.cc:243] Waited 8.25 ms on lock 0x7b1800442c00. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f4265d0a4bc 00007f426305848b 00007f426beb2ab9 00007f426307469b 00007f426307293f 00007f4263072455 00007f4269b5a3c7 00007f426616c9a7 00007f426617c0bf
0709 13:01:47.966204 (+   212us) log.cc:844] Serialized 14 byte log entry
0709 13:01:48.089148 (+122944us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:01:48.089234 (+    86us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:01:48.089310 (+    76us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:48.089422 (+   112us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:48.089612 (+   190us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":604018,"prepare.queue_time_us":435580,"prepare.run_cpu_time_us":157,"prepare.run_wall_time_us":3542,"raft.queue_time_us":403340,"raft.run_cpu_time_us":110,"raft.run_wall_time_us":111,"spinlock_wait_cycles":18980608,"thread_start_us":825876,"threads_started":2,"wal-append.queue_time_us":416198}
W20260709 13:01:48.097565  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 117492us
I20260709 13:01:48.098237  1676 rpcz_store.cc:276] 0709 13:01:46.761121 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:47.651866 (+890745us) service_pool.cc:224] Handling call
0709 13:01:47.652234 (+   368us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:47.652544 (+   310us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 13:01:47.835144 (+182600us) log.cc:844] Serialized 14 byte log entry
0709 13:01:48.097308 (+262164us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:01:48.097378 (+    70us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:01:48.097450 (+    72us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:48.097535 (+    85us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:48.097684 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":235474,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":49,"thread_start_us":398461,"threads_started":2}
I20260709 13:01:48.105829  1646 leader_election.cc:304] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
W20260709 13:01:48.156097  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 69430.3us
I20260709 13:01:48.161957  7745 leader_election.cc:290] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:01:48.162133  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 64647.3us
I20260709 13:01:48.163813  7700 raft_consensus.cc:2804] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:01:48.164350  7700 raft_consensus.cc:493] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:48.164429  7706 raft_consensus.cc:3060] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:01:48.164790  7700 raft_consensus.cc:3060] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:01:47.994005  7491 leader_election.cc:290] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 13 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:48.166708  7307 consensus_queue.cc:1059] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 12.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 13:01:48.042294  7697 raft_consensus.cc:515] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:48.168056  7307 raft_consensus.cc:3055] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:01:48.168519  7307 raft_consensus.cc:740] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:01:48.169478  7307 consensus_queue.cc:260] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:48.084342  7631 raft_consensus.cc:493] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:48.170888  1746 raft_consensus.cc:3060] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:01:48.169962  7697 leader_election.cc:290] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 15 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:01:48.161948  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 76611.9us
I20260709 13:01:48.171864  1569 leader_election.cc:304] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:01:48.172763  2301 leader_election.cc:304] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:01:48.176136  1569 leader_election.cc:304] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:01:48.177259  7307 raft_consensus.cc:3060] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:01:48.177727  7671 raft_consensus.cc:697] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 16 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
W20260709 13:01:48.179209  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 92970.1us
I20260709 13:01:48.182644  2298 leader_election.cc:304] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:01:48.182873  7671 consensus_queue.cc:237] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:48.151693  7468 raft_consensus.cc:493] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:48.184834  7468 raft_consensus.cc:515] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:01:48.181842  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 76292.7us
I20260709 13:01:48.179822  7662 raft_consensus.cc:2804] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Leader pre-election won for term 21
W20260709 13:01:48.180325  1645 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:48.158780  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 75239.4us
I20260709 13:01:48.094412  7740 raft_consensus.cc:493] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:48.181381  7437 raft_consensus.cc:2804] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Leader election won for term 16
I20260709 13:01:48.177796  7577 raft_consensus.cc:2804] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:01:48.210819  7437 raft_consensus.cc:697] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 16 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:01:48.210852  7577 raft_consensus.cc:493] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Starting leader election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:01:48.211488  7577 raft_consensus.cc:3060] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:01:48.195307  7662 raft_consensus.cc:493] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:48.203045  7653 raft_consensus.cc:2749] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:01:48.213167  7489 raft_consensus.cc:2804] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:01:48.213645  7489 raft_consensus.cc:493] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Starting leader election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:01:48.094364  7728 raft_consensus.cc:2804] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:01:48.196766  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:01:48.194942  7750 consensus_meta.cc:287] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.526s	user 0.008s	sys 0.005s
I20260709 13:01:48.215225  7750 raft_consensus.cc:515] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:48.179962  7579 raft_consensus.cc:515] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:48.202898  1646 leader_election.cc:304] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:48.225899  7786 raft_consensus.cc:2804] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Leader election won for term 17
I20260709 13:01:48.226652  7786 raft_consensus.cc:697] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 17 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:01:48.227954  7786 consensus_queue.cc:237] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.13, Last appended by leader: 13, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:48.214075  7489 raft_consensus.cc:3060] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:01:48.235574  1646 leader_election.cc:304] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:48.212468  7727 consensus_queue.cc:1059] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 13.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 13:01:48.213166  7662 raft_consensus.cc:3060] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:01:48.196519  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:48.104501  7540 raft_consensus.cc:493] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:48.223656  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:48.223869  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:48.224691  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:01:48.261740  1747 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.702s	user 0.032s	sys 0.049s
W20260709 13:01:48.262413  1747 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.703s	user 0.032s	sys 0.049s
I20260709 13:01:48.263551  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:48.207198  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:48.265130  1602 raft_consensus.cc:2468] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 17.
I20260709 13:01:48.237464  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:01:48.240450  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 48909.4us
I20260709 13:01:48.108939  7778 raft_consensus.cc:2804] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:01:48.267807  1598 raft_consensus.cc:1240] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 12. Current term is 13. Ops: []
I20260709 13:01:48.094834  7658 consensus_queue.cc:1048] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.114s
I20260709 13:01:48.039716  7687 raft_consensus.cc:515] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:48.264258  1646 leader_election.cc:304] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:48.215845  7728 raft_consensus.cc:493] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:48.268652  2298 leader_election.cc:304] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:48.186484  7631 raft_consensus.cc:515] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:48.110656  7559 raft_consensus.cc:515] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:48.279886  2301 leader_election.cc:304] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
W20260709 13:01:48.282287  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 328240us
I20260709 13:01:48.324200  1678 raft_consensus.cc:1240] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 LEADER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 15. Current term is 16. Ops: []
W20260709 13:01:48.309969  1643 connection.cc:256] RPC call timeout handler was delayed by -2.2940815080000334s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1673s for a response
W20260709 13:01:48.337232  1643 connection.cc:256] RPC call timeout handler was delayed by -2.2670815080000466s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1703s for a response
W20260709 13:01:48.337656  1643 connection.cc:256] RPC call timeout handler was delayed by -2.2661815080000451s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1704s for a response
W20260709 13:01:48.338227  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 58441.2us
I20260709 13:01:48.340004  2301 leader_election.cc:304] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:01:48.349625  2301 leader_election.cc:304] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:01:48.268534  7754 raft_consensus.cc:2804] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:01:48.357529  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:48.261639  7697 raft_consensus.cc:493] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:01:48.224525  7707 consensus_queue.cc:1059] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 11.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.062s
W20260709 13:01:48.376309  1717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:48.267710  7778 raft_consensus.cc:493] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:48.234683  7740 raft_consensus.cc:515] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:48.382264  2329 raft_consensus.cc:2368] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 15. Current term is 16.
I20260709 13:01:48.286509  7728 raft_consensus.cc:3060] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:01:48.386970  7778 raft_consensus.cc:3060] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:01:48.376417  7754 raft_consensus.cc:493] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:48.335899  7694 raft_consensus.cc:2749] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:01:48.414726  7702 raft_consensus.cc:493] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:48.418128  2299 leader_election.cc:304] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:01:48.455816  2299 leader_election.cc:304] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:01:48.419162  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:01:48.457967  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 94016.1us
I20260709 13:01:48.419581  7631 leader_election.cc:290] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:48.419866  1678 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52766 (request call id 2907) took 2057 ms. Trace:
I20260709 13:01:48.278522  7684 consensus_queue.cc:1059] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 17.14, Next index: 15, Last known committed idx: 13, Time since last communication: 0.107s
I20260709 13:01:48.421188  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:48.422227  7559 leader_election.cc:290] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 16 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:48.424278  7750 leader_election.cc:290] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:48.374336  7585 raft_consensus.cc:2749] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:01:48.434264  7707 raft_consensus.cc:3055] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 13:01:48.431219  7700 raft_consensus.cc:515] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:48.368113  7181 raft_consensus.cc:2804] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:01:48.370275  7703 raft_consensus.cc:2749] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:01:48.409051  7753 raft_consensus.cc:2804] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:01:48.276532  7540 raft_consensus.cc:515] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:48.465431  1678 rpcz_store.cc:276] 0709 13:01:46.363577 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:46.364054 (+   477us) service_pool.cc:224] Handling call
0709 13:01:46.377822 (+ 13768us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:48.336287 (+1958465us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:48.419586 (+ 83299us) spinlock_profiling.cc:243] Waited 1.95 s on lock 0x7b5400092254. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:01:48.419793 (+   207us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4474998528}
I20260709 13:01:48.470314  2298 leader_election.cc:304] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
W20260709 13:01:48.454067  7647 consensus_meta.cc:287] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.700s	user 0.010s	sys 0.003s
I20260709 13:01:48.503273  2299 leader_election.cc:304] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:01:48.451614  7754 raft_consensus.cc:3060] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:01:48.465340  7468 leader_election.cc:290] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:48.385938  7697 raft_consensus.cc:515] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:48.536546  1746 raft_consensus.cc:2468] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 16.
I20260709 13:01:48.518208  1644 leader_election.cc:304] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:01:48.524374  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:48.498235  7707 raft_consensus.cc:740] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:01:48.533550  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:01:48.584893  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 66794.9us
I20260709 13:01:48.585191  1745 raft_consensus.cc:2468] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 13.
I20260709 13:01:48.540405  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:48.499819  7669 raft_consensus.cc:2749] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:01:48.541999  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:48.595654  1753 raft_consensus.cc:2468] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 17.
I20260709 13:01:48.547405  7703 raft_consensus.cc:2749] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:01:48.547564  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:48.374547  7437 consensus_queue.cc:237] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:48.551992  7720 consensus_queue.cc:1059] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 15.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 13:01:48.506927  7181 raft_consensus.cc:493] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:48.553333  7579 leader_election.cc:290] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 13 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:48.553923  7677 raft_consensus.cc:2804] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:01:48.496882  7653 consensus_queue.cc:1059] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 16.13, Next index: 14, Last known committed idx: 12, Time since last communication: 0.046s
I20260709 13:01:48.511214  7753 raft_consensus.cc:493] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:48.560240  7697 leader_election.cc:290] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:48.561661  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:48.599422  7740 leader_election.cc:290] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:48.515305  7690 consensus_queue.cc:1059] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 12.11, Next index: 12, Last known committed idx: 10, Time since last communication: 0.055s
I20260709 13:01:48.512523  7750 consensus_queue.cc:1048] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.059s
I20260709 13:01:48.570853  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:48.572784  7700 leader_election.cc:290] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 14 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:48.577775  7540 leader_election.cc:290] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:48.578946  7707 consensus_queue.cc:260] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 11.11, Last appended by leader: 11, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:48.580258  7687 leader_election.cc:290] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:01:48.580726  7723 consensus_meta.cc:287] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.535s	user 0.012s	sys 0.000s
I20260709 13:01:48.527375  7647 raft_consensus.cc:515] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:48.516774  7559 raft_consensus.cc:3055] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 17 LEADER]: Stepping down as leader of term 17
W20260709 13:01:48.588784  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 425585us
I20260709 13:01:48.586488  1644 leader_election.cc:304] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
W20260709 13:01:48.766651  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 180228us
I20260709 13:01:48.768159  1644 leader_election.cc:304] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:01:48.483314  7702 raft_consensus.cc:515] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:01:48.773957  7798 log.cc:927] Time spent T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.128s	user 0.000s	sys 0.000s
W20260709 13:01:48.620142  1717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:48.644054  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:48.599143  7600 raft_consensus.cc:2804] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:01:48.655141  1754 raft_consensus.cc:3060] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:01:48.567276  2334 raft_consensus.cc:1275] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 16 index: 11. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
I20260709 13:01:48.608245  7181 raft_consensus.cc:3060] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:01:48.669661  1747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55848 (request call id 2948) took 1819 ms. Trace:
W20260709 13:01:48.688964  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 45687.3us
I20260709 13:01:48.644433  7677 raft_consensus.cc:493] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:48.654389  7753 raft_consensus.cc:3060] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:01:48.745622  1746 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43156 (request call id 1228) took 1140 ms. Trace:
I20260709 13:01:48.683185  7671 raft_consensus.cc:3055] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 16 LEADER]: Stepping down as leader of term 16
W20260709 13:01:48.772647  7706 consensus_meta.cc:287] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.608s	user 0.015s	sys 0.000s
I20260709 13:01:48.772929  7687 raft_consensus.cc:493] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:48.773432  2329 raft_consensus.cc:2468] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 11.
W20260709 13:01:48.790082  2300 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:48.782965  1644 leader_election.cc:304] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:01:48.800096  2301 leader_election.cc:304] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:48.798235  1747 rpcz_store.cc:276] 0709 13:01:46.850744 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:47.536505 (+685761us) service_pool.cc:224] Handling call
0709 13:01:47.538372 (+  1867us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:47.538918 (+   546us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 13:01:48.262823 (+723905us) log.cc:844] Serialized 14 byte log entry
0709 13:01:48.608170 (+345347us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:01:48.643308 (+ 35138us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:01:48.643389 (+    81us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:48.643485 (+    96us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:48.669538 (+ 26053us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":875630,"prepare.run_cpu_time_us":76,"prepare.run_wall_time_us":78,"raft.queue_time_us":116,"raft.run_cpu_time_us":189,"raft.run_wall_time_us":4379,"thread_start_us":1048032,"threads_started":2}
I20260709 13:01:48.693584  7560 consensus_queue.cc:1059] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 16.13, Next index: 14, Last known committed idx: 12, Time since last communication: 0.092s
I20260709 13:01:48.798791  7600 raft_consensus.cc:493] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:01:48.797034  7700 raft_consensus.cc:493] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:01:48.815685  1643 connection.cc:256] RPC call timeout handler was delayed by -2.0789815080000267s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1912s for a response
W20260709 13:01:48.827024  1643 connection.cc:256] RPC call timeout handler was delayed by -2.028581508000002s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1968s for a response
I20260709 13:01:48.785708  7787 raft_consensus.cc:2804] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:01:48.805421  7707 raft_consensus.cc:3060] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:01:48.775714  7559 raft_consensus.cc:740] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:01:48.774626  7436 consensus_queue.cc:1048] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.033s
I20260709 13:01:48.784746  7786 raft_consensus.cc:3060] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:01:48.807051  1746 rpcz_store.cc:276] 0709 13:01:47.574876 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:47.612566 (+ 37690us) service_pool.cc:224] Handling call
0709 13:01:48.714699 (+1102133us) spinlock_profiling.cc:243] Waited 536 ms on lock 0x7b54003246d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:01:48.714905 (+   206us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1232170368}
I20260709 13:01:48.838267  1754 raft_consensus.cc:2468] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 16.
I20260709 13:01:48.812767  7671 raft_consensus.cc:740] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
W20260709 13:01:48.825884  7802 log.cc:927] Time spent T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.139s	user 0.000s	sys 0.001s
I20260709 13:01:48.808986  7677 raft_consensus.cc:3060] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Advancing to term 18
W20260709 13:01:48.814725  7688 consensus_meta.cc:287] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.813s	user 0.012s	sys 0.004s
I20260709 13:01:48.818073  7706 raft_consensus.cc:2749] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:01:48.823035  7687 raft_consensus.cc:515] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:01:48.816154  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 551924us
I20260709 13:01:48.823596  7653 raft_consensus.cc:2804] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:01:48.948807  1754 raft_consensus.cc:1240] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 15. Current term is 16. Ops: []
I20260709 13:01:49.045045  7697 raft_consensus.cc:493] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:48.934497  7559 consensus_queue.cc:260] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 17.14, Last appended by leader: 14, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:48.917641  1717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:49.065402  7559 raft_consensus.cc:3060] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:01:48.838711  7787 raft_consensus.cc:493] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:49.066845  7787 raft_consensus.cc:3060] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:01:49.071630  1679 raft_consensus.cc:1240] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 16. Current term is 17. Ops: [16.9-16.9]
W20260709 13:01:49.071969  2336 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.556s	user 0.022s	sys 0.035s
W20260709 13:01:49.072394  2336 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.557s	user 0.022s	sys 0.035s
I20260709 13:01:49.073931  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:49.019409  7702 leader_election.cc:290] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:49.062512  7697 raft_consensus.cc:515] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:49.018800  7647 leader_election.cc:290] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 16 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:48.973194  2301 leader_election.cc:304] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:01:48.827310  7600 raft_consensus.cc:3060] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:01:48.894328  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:01:48.943356  7803 log.cc:927] Time spent T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.066s	user 0.000s	sys 0.000s
W20260709 13:01:49.052721  2299 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.502s	user 0.002s	sys 0.010s
I20260709 13:01:48.947284  7740 raft_consensus.cc:493] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
W20260709 13:01:48.914495  1571 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.714s	user 0.022s	sys 0.032s
W20260709 13:01:49.099871  2299 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.549s	user 0.003s	sys 0.010s
W20260709 13:01:49.100129  1571 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.900s	user 0.025s	sys 0.045s
W20260709 13:01:49.100378  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 549854us
W20260709 13:01:49.100551  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 909329us
W20260709 13:01:48.917254  1569 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.717s	user 0.008s	sys 0.048s
W20260709 13:01:49.101150  1569 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.901s	user 0.012s	sys 0.060s
W20260709 13:01:49.101486  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 901568us
W20260709 13:01:49.010390  7489 consensus_meta.cc:287] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.736s	user 0.013s	sys 0.000s
W20260709 13:01:49.056995  1599 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.543s	user 0.019s	sys 0.030s
I20260709 13:01:49.061237  7653 raft_consensus.cc:493] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:48.984136  7671 consensus_queue.cc:260] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 16.13, Last appended by leader: 13, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:49.051994  1751 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55848 (request call id 2952) took 2200 ms. Trace:
W20260709 13:01:48.876623  1645 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:49.052873  2298 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.526s	user 0.000s	sys 0.017s
I20260709 13:01:48.979828  1746 raft_consensus.cc:1240] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 10. Current term is 11. Ops: []
I20260709 13:01:48.826642  7761 raft_consensus.cc:2804] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:01:49.090878  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:01:49.091387  7577 consensus_meta.cc:287] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.879s	user 0.009s	sys 0.010s
I20260709 13:01:48.949050  7307 raft_consensus.cc:493] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
W20260709 13:01:49.053200  7725 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.506s	user 0.009s	sys 0.008s
W20260709 13:01:49.103489  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 130416us
W20260709 13:01:49.060290  7727 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.774s	user 0.016s	sys 0.005s
W20260709 13:01:49.105512  1599 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.607s	user 0.020s	sys 0.031s
I20260709 13:01:48.875682  7808 raft_consensus.cc:493] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:48.962898  7684 raft_consensus.cc:3055] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 13:01:48.840651  7700 raft_consensus.cc:515] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:49.109913  7653 raft_consensus.cc:3060] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:01:49.091776  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:01:49.224867  7809 log.cc:927] Time spent T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.291s	user 0.000s	sys 0.001s
I20260709 13:01:49.123569  7585 raft_consensus.cc:2749] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20260709 13:01:49.208616  7754 consensus_meta.cc:287] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.644s	user 0.011s	sys 0.000s
I20260709 13:01:49.114841  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:49.079149  7718 raft_consensus.cc:2804] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:01:49.110524  1751 rpcz_store.cc:276] 0709 13:01:46.852025 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:47.549325 (+697300us) service_pool.cc:224] Handling call
0709 13:01:47.549822 (+   497us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:47.871346 (+321524us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 13:01:48.276072 (+404726us) log.cc:844] Serialized 14 byte log entry
0709 13:01:48.805694 (+529622us) log.cc:844] Serialized 14 byte log entry
0709 13:01:48.805913 (+   219us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:01:48.820756 (+ 14843us) log.cc:844] Serialized 26 byte log entry
0709 13:01:48.820996 (+   240us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:01:48.821627 (+   631us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:48.821726 (+    99us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:48.944198 (+122472us) spinlock_profiling.cc:243] Waited 9.71 ms on lock 0x7b18004554a0. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f4263075522 00007f42630777c1 00007f4263077779 00007f4263077709 00007f42630776d1 00007f42630769cd 000000000038f354 000000000038f199 00007f426305883a
0709 13:01:48.967071 (+ 22873us) spinlock_profiling.cc:243] Waited 2.49 ms on lock 0x7b18004554a0. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f42630753f3 00007f42630777c1 00007f4263077779 00007f4263077709 00007f42630776d1 00007f42630769cd 000000000038f354 000000000038f199 00007f426305883a
0709 13:01:49.051694 (+ 84623us) raft_consensus.cc:1685] finished
0709 13:01:49.051724 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:49.051904 (+   180us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":544254,"prepare.run_cpu_time_us":97,"prepare.run_wall_time_us":100,"raft.queue_time_us":89294,"raft.run_cpu_time_us":427,"raft.run_wall_time_us":7085,"spinlock_wait_cycles":28051072,"thread_start_us":1105939,"threads_started":3,"wal-append.queue_time_us":660823}
W20260709 13:01:49.111575  2298 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.585s	user 0.000s	sys 0.018s
I20260709 13:01:49.105527  7812 consensus_queue.cc:1059] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 16.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.025s
I20260709 13:01:49.094072  7688 raft_consensus.cc:493] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
W20260709 13:01:49.183400  7806 log.cc:927] Time spent T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.166s	user 0.001s	sys 0.000s
I20260709 13:01:48.958976  7714 raft_consensus.cc:3060] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:01:49.179723  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:49.180114  1599 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 2724) took 1278 ms. Trace:
W20260709 13:01:49.151564  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 46189.7us
I20260709 13:01:49.290623  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:49.296998  1751 raft_consensus.cc:2393] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 11: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:01:49.298861  1599 rpcz_store.cc:276] 0709 13:01:47.901951 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:48.029294 (+127343us) service_pool.cc:224] Handling call
0709 13:01:48.029586 (+   292us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:48.029822 (+   236us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 13:01:48.093971 (+ 64149us) log.cc:844] Serialized 14 byte log entry
0709 13:01:49.179633 (+1085662us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:01:49.179703 (+    70us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:01:49.179793 (+    90us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:49.179879 (+    86us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:49.180038 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":63851,"prepare.run_cpu_time_us":58,"prepare.run_wall_time_us":62,"raft.queue_time_us":574345,"raft.run_cpu_time_us":130,"raft.run_wall_time_us":133,"thread_start_us":1085127,"threads_started":2,"wal-append.queue_time_us":724787}
I20260709 13:01:49.128650  2301 leader_election.cc:304] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:48.970541  7579 raft_consensus.cc:3055] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 12 LEADER]: Stepping down as leader of term 12
W20260709 13:01:49.187232  7778 consensus_meta.cc:287] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.695s	user 0.010s	sys 0.005s
W20260709 13:01:49.188701  1569 connection.cc:256] RPC call timeout handler was delayed by -1.4207317539999735s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16820000000000002s for a response
W20260709 13:01:49.189673  7728 consensus_meta.cc:287] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.725s	user 0.007s	sys 0.007s
I20260709 13:01:49.190340  7671 raft_consensus.cc:3060] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:01:49.179116  7684 raft_consensus.cc:740] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:01:49.190896  7808 raft_consensus.cc:515] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:49.347275  7684 consensus_queue.cc:260] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:49.348457  7684 raft_consensus.cc:3060] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:01:49.348793  7808 leader_election.cc:290] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:49.247253  7754 raft_consensus.cc:515] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } }
I20260709 13:01:49.351642  7754 leader_election.cc:290] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 15 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:49.352325  7784 consensus_queue.cc:1048] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.011s
I20260709 13:01:49.191622  7697 leader_election.cc:290] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 13:01:49.192039  7807 log.cc:927] Time spent T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.093s	user 0.000s	sys 0.000s
I20260709 13:01:49.110225  7740 raft_consensus.cc:515] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:01:49.131163  7662 consensus_meta.cc:287] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.843s	user 0.009s	sys 0.004s
I20260709 13:01:49.192945  7706 raft_consensus.cc:493] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:49.113803  7582 consensus_queue.cc:1048] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.081s
I20260709 13:01:49.195060  7468 raft_consensus.cc:493] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
W20260709 13:01:49.198911  7810 log.cc:927] Time spent T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.217s	user 0.001s	sys 0.000s
I20260709 13:01:49.204300  7700 leader_election.cc:290] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:49.153278  7307 raft_consensus.cc:515] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:49.302881  7579 raft_consensus.cc:740] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:01:49.212462  7687 leader_election.cc:290] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:49.214021  7786 raft_consensus.cc:2749] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:01:49.214417  7686 raft_consensus.cc:2804] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:01:49.214763  7815 raft_consensus.cc:493] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
W20260709 13:01:49.231765  7811 log.cc:927] Time spent T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.260s	user 0.002s	sys 0.001s
I20260709 13:01:49.232583  1604 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 2725) took 1330 ms. Trace:
I20260709 13:01:49.388553  1604 rpcz_store.cc:276] 0709 13:01:47.902263 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:48.035391 (+133128us) service_pool.cc:224] Handling call
0709 13:01:48.035780 (+   389us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:48.108369 (+ 72589us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:48.108411 (+    42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:01:48.154631 (+ 46220us) log.cc:844] Serialized 26 byte log entry
0709 13:01:48.216530 (+ 61899us) spinlock_profiling.cc:243] Waited 10.9 ms on lock 0x7b1800273100. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f4263057e07 00007f426beb2ab9 00007f426307469b 00007f426307293f 00007f4263074c02 00007f4265cfc20b 00007f4265d01d23 00007f4265d020d1 00007f426611da0f
0709 13:01:48.609371 (+392841us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:01:48.609447 (+    76us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:48.609547 (+   100us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:49.232196 (+622649us) raft_consensus.cc:1685] finished
0709 13:01:49.232224 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:49.232503 (+   279us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":60469,"raft.queue_time_us":118365,"raft.run_cpu_time_us":96,"raft.run_wall_time_us":96,"spinlock_wait_cycles":24982272,"thread_start_us":390939,"threads_started":1,"wal-append.queue_time_us":713502}
W20260709 13:01:49.111285  1645 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:49.363833  7694 raft_consensus.cc:2804] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 13:01:49.404515  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:49.341706  7778 raft_consensus.cc:515] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:49.407115  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:49.117190  1646 leader_election.cc:304] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:49.118996  7702 raft_consensus.cc:493] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:49.248504  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:49.428570  1671 raft_consensus.cc:3060] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Advancing to term 17
W20260709 13:01:49.334363  1569 connection.cc:256] RPC call timeout handler was delayed by -1.3478317539999694s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1763s for a response
I20260709 13:01:49.336452  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 15 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:49.117969  7489 raft_consensus.cc:515] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:49.429114  1569 connection.cc:256] RPC call timeout handler was delayed by -1.2731317539999623s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18460000000000001s for a response
W20260709 13:01:49.429637  1569 connection.cc:256] RPC call timeout handler was delayed by -1.2514979540000013s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1507s for a response
W20260709 13:01:49.430063  1569 connection.cc:256] RPC call timeout handler was delayed by -1.2182317539999872s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1907s for a response
W20260709 13:01:49.430469  1569 connection.cc:256] RPC call timeout handler was delayed by -1.2029317539999624s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19240000000000002s for a response
W20260709 13:01:49.430859  1569 connection.cc:256] RPC call timeout handler was delayed by -1.1003317539999671s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20379999999999998s for a response
W20260709 13:01:49.431244  1569 connection.cc:256] RPC call timeout handler was delayed by -1.0562268769999719s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16100000000000003s for a response
I20260709 13:01:49.433074  1569 leader_election.cc:304] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:01:49.343498  7728 raft_consensus.cc:515] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:49.245697  7812 raft_consensus.cc:3055] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 LEADER]: Stepping down as leader of term 16
I20260709 13:01:49.124392  7577 raft_consensus.cc:515] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:49.285796  7540 raft_consensus.cc:493] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:49.131778  7813 consensus_queue.cc:1059] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 17.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.224s
I20260709 13:01:49.285061  7718 raft_consensus.cc:493] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Starting leader election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:01:49.285630  7688 raft_consensus.cc:515] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:01:49.361563  1676 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.530s	user 0.017s	sys 0.026s
W20260709 13:01:49.365904  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 237308us
I20260709 13:01:49.206230  7753 raft_consensus.cc:515] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:49.304553  7750 consensus_queue.cc:1048] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.068s
W20260709 13:01:49.387683  1644 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.571s	user 0.055s	sys 0.055s
W20260709 13:01:49.388010  1717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:49.239367  1718 leader_election.cc:304] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:49.395920  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 18 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:49.397652  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:49.399230  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:49.400857  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:49.402877  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
W20260709 13:01:49.404788  1571 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:49.396847  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:49.364234  7761 raft_consensus.cc:493] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:49.417550  1672 raft_consensus.cc:1275] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 14 index: 8. (index mismatch)
W20260709 13:01:49.126204  7725 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.589s	user 0.010s	sys 0.008s
W20260709 13:01:49.244547  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 774181us
I20260709 13:01:49.419730  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:49.420456  1745 raft_consensus.cc:1275] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 14 index: 8. (index mismatch)
I20260709 13:01:49.422199  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:49.377537  7700 raft_consensus.cc:493] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:01:49.425238  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 308130us
I20260709 13:01:49.406230  7560 raft_consensus.cc:493] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:01:49.426990  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:49.338802  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:01:49.182590  7727 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.906s	user 0.016s	sys 0.010s
I20260709 13:01:49.406716  2334 raft_consensus.cc:2468] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 13.
I20260709 13:01:49.363574  7662 raft_consensus.cc:515] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:49.437925  7786 raft_consensus.cc:2804] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:01:49.492832  7786 raft_consensus.cc:697] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:01:49.498885  7686 raft_consensus.cc:493] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:49.499614  7686 raft_consensus.cc:3060] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:01:49.503959  7778 leader_election.cc:290] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 14 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:49.406386  7657 raft_consensus.cc:493] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:49.504745  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 15 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:49.505192  7657 raft_consensus.cc:515] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:49.507751  7657 leader_election.cc:290] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:49.508607  7582 raft_consensus.cc:493] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:49.441812  7702 raft_consensus.cc:515] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:49.509193  7582 raft_consensus.cc:515] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:49.510716  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:49.511374  1602 raft_consensus.cc:3060] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:01:49.447667  7688 leader_election.cc:290] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:49.511972  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:01:49.447952  1676 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.617s	user 0.017s	sys 0.026s
W20260709 13:01:49.452844  1570 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.894s	user 0.005s	sys 0.039s
W20260709 13:01:49.454495  1644 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.638s	user 0.055s	sys 0.055s
I20260709 13:01:49.515246  2331 raft_consensus.cc:1240] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 20. Current term is 21. Ops: []
I20260709 13:01:49.456347  2335 raft_consensus.cc:1240] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 17. Current term is 18. Ops: []
I20260709 13:01:49.439018  1569 leader_election.cc:304] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:49.440337  1679 raft_consensus.cc:2393] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 15: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:01:49.370100  7706 raft_consensus.cc:515] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:49.305544  7181 raft_consensus.cc:515] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:49.441712  7817 raft_consensus.cc:2804] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:01:49.391086  7815 raft_consensus.cc:515] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:49.424257  7694 raft_consensus.cc:697] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 16 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:01:49.567938  7694 consensus_queue.cc:237] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:49.571206  2300 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:49.572078  7181 leader_election.cc:290] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 18 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:49.573580  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 18 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:49.574323  1679 raft_consensus.cc:2468] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 17.
I20260709 13:01:49.575598  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:49.491081  7653 raft_consensus.cc:515] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:49.515149  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 698589us
I20260709 13:01:49.577263  2335 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39738 (request call id 1107) took 1160 ms. Trace:
I20260709 13:01:49.458833  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:49.438627  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:49.578644  1751 raft_consensus.cc:2410] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for term 13 because replica has last-logged OpId of term: 11 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 7.
I20260709 13:01:49.463526  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:49.579308  1604 raft_consensus.cc:3060] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:01:49.463901  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:49.464290  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:49.464680  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:49.468935  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:49.581797  1674 raft_consensus.cc:2468] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 13.
I20260709 13:01:49.583024  1674 raft_consensus.cc:1240] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 14. Current term is 15. Ops: []
I20260709 13:01:49.373876  7468 raft_consensus.cc:515] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:49.474735  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:49.474743  1748 raft_consensus.cc:3060] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:01:49.476001  7577 leader_election.cc:290] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 12 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:49.400475  7822 raft_consensus.cc:493] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
W20260709 13:01:49.514676  1570 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.054s	user 0.006s	sys 0.039s
I20260709 13:01:49.602116  1644 leader_election.cc:304] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
W20260709 13:01:49.455912  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 216684us
I20260709 13:01:49.457432  1753 raft_consensus.cc:3060] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:01:49.452601  2336 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39738 (request call id 1105) took 1036 ms. Trace:
I20260709 13:01:49.656723  2336 rpcz_store.cc:276] 0709 13:01:48.416302 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:48.416658 (+   356us) service_pool.cc:224] Handling call
0709 13:01:48.417163 (+   505us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:48.449390 (+ 32227us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:48.449428 (+    38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:01:48.464396 (+ 14968us) log.cc:844] Serialized 26 byte log entry
0709 13:01:49.072781 (+608385us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:01:49.072827 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:49.073874 (+  1047us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:49.452346 (+378472us) raft_consensus.cc:1685] finished
0709 13:01:49.452372 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:49.452525 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":31627,"raft.queue_time_us":32344,"raft.run_cpu_time_us":155,"raft.run_wall_time_us":13085,"thread_start_us":556856,"threads_started":1,"wal-append.queue_time_us":904469}
I20260709 13:01:49.459225  1749 raft_consensus.cc:2468] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 17.
I20260709 13:01:49.546556  1602 raft_consensus.cc:2468] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 11.
I20260709 13:01:49.660020  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:49.660895  1749 raft_consensus.cc:2468] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 14.
I20260709 13:01:49.517880  1597 raft_consensus.cc:1275] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 15 index: 8. Preceding OpId from leader: term: 17 index: 9. (index mismatch)
I20260709 13:01:49.518771  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:49.664024  1605 raft_consensus.cc:2368] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 15. Current term is 16.
W20260709 13:01:49.526321  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 87388.8us
I20260709 13:01:49.529124  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:49.475630  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:49.667552  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:49.454123  7753 leader_election.cc:290] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 13 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:49.466127  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:49.540416  2331 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39738 (request call id 1100) took 1316 ms. Trace:
I20260709 13:01:49.671636  1672 raft_consensus.cc:2468] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 14.
I20260709 13:01:49.671571  2331 rpcz_store.cc:276] 0709 13:01:48.224174 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:48.225358 (+  1184us) service_pool.cc:224] Handling call
0709 13:01:48.260944 (+ 35586us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:49.515726 (+1254782us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:49.540134 (+ 24408us) spinlock_profiling.cc:243] Waited 1.23 s on lock 0x7b54005bca54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:01:49.540342 (+   208us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2835773696}
I20260709 13:01:49.463014  7816 raft_consensus.cc:493] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:49.482779  7579 consensus_queue.cc:260] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:49.471367  7700 raft_consensus.cc:515] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:49.549427  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:49.681988  1596 raft_consensus.cc:2468] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 12.
I20260709 13:01:49.550662  7815 leader_election.cc:290] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:49.456053  7489 leader_election.cc:290] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 18 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:49.513765  7745 raft_consensus.cc:3055] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 LEADER]: Stepping down as leader of term 13
I20260709 13:01:49.435369  7812 raft_consensus.cc:740] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:01:49.463240  7540 raft_consensus.cc:515] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:49.706949  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:49.707664  1596 raft_consensus.cc:3060] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Advancing to term 18
W20260709 13:01:49.707708  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 105682us
I20260709 13:01:49.472011  7560 raft_consensus.cc:515] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:49.476033  7727 raft_consensus.cc:493] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:49.494154  7662 leader_election.cc:290] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 21 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:49.511435  7702 leader_election.cc:290] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:49.484108  7740 leader_election.cc:290] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:49.457034  7784 raft_consensus.cc:2749] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:01:49.462360  7728 leader_election.cc:290] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 18 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:49.447140  7307 leader_election.cc:290] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:49.561851  7818 raft_consensus.cc:3055] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 LEADER]: Stepping down as leader of term 17
I20260709 13:01:49.542491  7750 raft_consensus.cc:3060] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 10 FOLLOWER]: Advancing to term 11
W20260709 13:01:49.562903  7823 log.cc:927] Time spent T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.127s	user 0.001s	sys 0.000s
I20260709 13:01:49.577651  2335 rpcz_store.cc:276] 0709 13:01:48.417123 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:48.417285 (+   162us) service_pool.cc:224] Handling call
0709 13:01:48.417634 (+   349us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:49.533927 (+1116293us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:49.576990 (+ 43063us) spinlock_profiling.cc:243] Waited 1.04 s on lock 0x7b54005802d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:01:49.577189 (+   199us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2387970432}
I20260709 13:01:49.471068  1675 raft_consensus.cc:3060] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:01:49.464959  7761 raft_consensus.cc:3060] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:01:49.585391  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:49.511430  7582 leader_election.cc:290] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:01:49.469377  1497 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:43476 (request call id 194) took 49996 ms (client timeout 15000 ms). Trace:
I20260709 13:01:49.586460  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:49.587594  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:49.754827  1751 raft_consensus.cc:3060] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:01:49.589185  7686 raft_consensus.cc:515] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:49.541343  7817 raft_consensus.cc:493] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:49.445987  7718 raft_consensus.cc:3060] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:01:49.596632  1604 raft_consensus.cc:2468] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 17.
I20260709 13:01:49.600200  7706 leader_election.cc:290] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:49.638090  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:49.740427  7818 raft_consensus.cc:740] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:01:49.648509  1673 raft_consensus.cc:3060] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Advancing to term 12
W20260709 13:01:49.652539  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 54243.5us
I20260709 13:01:49.665723  1603 raft_consensus.cc:2468] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 17.
I20260709 13:01:49.662178  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:01:49.637512  7437 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.587s	user 0.037s	sys 0.061s
W20260709 13:01:49.611634  7600 consensus_meta.cc:287] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.502s	user 0.011s	sys 0.001s
I20260709 13:01:49.705463  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:49.710011  1646 leader_election.cc:304] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
W20260709 13:01:49.773427  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 63515.4us
W20260709 13:01:49.716639  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 51334us
I20260709 13:01:49.773929  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:49.774473  1569 leader_election.cc:304] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:01:49.774750  1603 raft_consensus.cc:3060] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:01:49.771437  1643 leader_election.cc:304] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:01:49.769073  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
W20260709 13:01:49.717298  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 57491.1us
I20260709 13:01:49.718132  1671 raft_consensus.cc:2468] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 17.
I20260709 13:01:49.719324  2328 raft_consensus.cc:2393] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 18: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:01:49.721642  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:49.722286  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:49.724337  1747 raft_consensus.cc:2468] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 14.
I20260709 13:01:49.494122  7786 consensus_queue.cc:237] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.8, Last appended by leader: 8, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:49.733088  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 14 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:49.733790  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:49.741369  7754 consensus_queue.cc:1048] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.025s
I20260709 13:01:49.737185  1644 leader_election.cc:304] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:01:49.703429  7745 raft_consensus.cc:740] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
W20260709 13:01:49.770704  7437 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.726s	user 0.038s	sys 0.061s
I20260709 13:01:49.719146  7827 raft_consensus.cc:2749] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:01:49.731876  7579 raft_consensus.cc:3060] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:01:49.755659  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:49.781052  1596 raft_consensus.cc:2468] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 18.
I20260709 13:01:49.576519  1679 raft_consensus.cc:2468] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 14.
I20260709 13:01:49.704427  7816 raft_consensus.cc:515] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:49.731822  7727 raft_consensus.cc:515] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:01:49.769491  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 318031us
W20260709 13:01:49.769987  1645 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:49.796001  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 58922.7us
I20260709 13:01:49.796727  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:49.772104  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:49.797907  1677 raft_consensus.cc:2393] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 13: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:01:49.778681  1604 raft_consensus.cc:3060] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:01:49.781229  1602 raft_consensus.cc:3060] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:01:49.788625  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:49.780684  1597 raft_consensus.cc:3060] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:01:49.738114  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:49.804641  1754 raft_consensus.cc:3060] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:01:49.680819  7822 raft_consensus.cc:515] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:49.806499  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:49.807331  1599 raft_consensus.cc:2393] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 16: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:01:49.762224  7817 raft_consensus.cc:3060] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:01:49.813656  1680 raft_consensus.cc:2468] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 17.
I20260709 13:01:49.814682  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 14 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:49.779431  1569 leader_election.cc:304] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:01:49.816721  1569 leader_election.cc:304] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:49.818018  1569 leader_election.cc:304] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
W20260709 13:01:49.732437  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 292354us
I20260709 13:01:49.770387  1749 raft_consensus.cc:2468] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 16.
I20260709 13:01:49.792868  2298 leader_election.cc:304] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:01:49.797114  1571 leader_election.cc:304] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:01:49.822855  1571 leader_election.cc:304] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:49.823873  1569 leader_election.cc:304] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:01:49.778587  7577 raft_consensus.cc:2749] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:01:49.835054  1604 raft_consensus.cc:2468] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 16.
I20260709 13:01:49.835961  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:49.838150  1596 raft_consensus.cc:2468] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 18.
I20260709 13:01:49.838477  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:49.836076  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:49.839059  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 18 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:49.839246  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 17 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:01:49.841144  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 48365.7us
I20260709 13:01:49.839860  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:49.841572  1599 raft_consensus.cc:2410] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for term 17 because replica has last-logged OpId of term: 15 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 8.
I20260709 13:01:49.842092  1749 raft_consensus.cc:2468] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 16.
I20260709 13:01:49.841552  1604 raft_consensus.cc:2410] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for term 18 because replica has last-logged OpId of term: 16 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 7.
I20260709 13:01:49.843102  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:49.845659  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:49.781042  7753 raft_consensus.cc:2804] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:01:49.766346  7181 consensus_queue.cc:1059] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 12.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.040s
I20260709 13:01:49.789733  7740 raft_consensus.cc:493] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:49.865382  1601 raft_consensus.cc:2393] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 17: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:01:49.848891  1598 raft_consensus.cc:3060] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:01:49.864995  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:49.853602  1754 raft_consensus.cc:2468] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 18.
I20260709 13:01:49.869971  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:49.869900  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:49.870826  1749 raft_consensus.cc:3060] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:01:49.874673  1754 raft_consensus.cc:3060] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Advancing to term 18
W20260709 13:01:49.880851  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 109433us
I20260709 13:01:49.883526  1643 leader_election.cc:304] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:01:49.886139  1596 raft_consensus.cc:1275] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 13 index: 9. Preceding OpId from leader: term: 17 index: 11. (index mismatch)
I20260709 13:01:49.887250  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:49.808195  7582 raft_consensus.cc:3060] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:01:49.852205  1603 raft_consensus.cc:2468] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 21.
W20260709 13:01:49.810813  7787 consensus_meta.cc:287] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.743s	user 0.012s	sys 0.003s
I20260709 13:01:49.890650  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:49.815582  7658 raft_consensus.cc:2804] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:01:49.897045  2298 leader_election.cc:304] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:01:49.898541  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:49.899569  2331 raft_consensus.cc:1275] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 16 index: 13. Preceding OpId from leader: term: 17 index: 14. (index mismatch)
I20260709 13:01:49.892850  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 18 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:49.900830  1747 raft_consensus.cc:1240] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 17. Current term is 18. Ops: []
I20260709 13:01:49.901000  1680 raft_consensus.cc:2410] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for term 18 because replica has last-logged OpId of term: 16 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 7.
I20260709 13:01:49.840533  1602 raft_consensus.cc:2468] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 14.
W20260709 13:01:49.901886  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 445974us
I20260709 13:01:49.903018  1602 raft_consensus.cc:1240] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 10. Current term is 11. Ops: []
I20260709 13:01:49.895464  1604 raft_consensus.cc:1240] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 20. Current term is 21. Ops: [20.14-20.14]
I20260709 13:01:49.903283  1717 leader_election.cc:304] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:49.907509  1754 raft_consensus.cc:2468] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 18.
I20260709 13:01:49.913546  1754 raft_consensus.cc:1240] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 17. Current term is 18. Ops: []
W20260709 13:01:49.866320  7707 consensus_meta.cc:287] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 1.024s	user 0.009s	sys 0.006s
I20260709 13:01:49.825911  7808 raft_consensus.cc:2749] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
W20260709 13:01:49.886281  7677 consensus_meta.cc:287] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.768s	user 0.013s	sys 0.000s
I20260709 13:01:49.778419  7600 raft_consensus.cc:515] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:49.973654  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 70478.4us
I20260709 13:01:49.989673  1598 raft_consensus.cc:2468] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 13.
W20260709 13:01:50.006511  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 183754us
I20260709 13:01:50.009976  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:49.897615  7658 raft_consensus.cc:493] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:49.925338  7653 leader_election.cc:290] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 18 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:49.961279  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 17 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:49.929056  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:49.967293  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
W20260709 13:01:49.967916  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 70984.7us
I20260709 13:01:50.084050  7833 raft_consensus.cc:2749] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:01:49.968199  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:49.968214  7782 raft_consensus.cc:2749] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:01:49.925627  7728 raft_consensus.cc:2749] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:01:49.864375  7657 raft_consensus.cc:2804] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 13:01:49.927263  1599 raft_consensus.cc:1275] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 16 index: 12. (index mismatch)
I20260709 13:01:49.932472  1749 raft_consensus.cc:2468] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 14.
I20260709 13:01:49.807862  7687 consensus_queue.cc:1048] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.091s
I20260709 13:01:49.849256  7671 raft_consensus.cc:493] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:50.016350  1753 raft_consensus.cc:2468] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 13.
I20260709 13:01:50.105072  1746 raft_consensus.cc:1240] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 12. Current term is 13. Ops: []
I20260709 13:01:49.889647  7753 raft_consensus.cc:493] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:49.927788  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:49.967572  7820 raft_consensus.cc:2764] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Leader pre-election decision vote started in defunct term 17: won
I20260709 13:01:50.019968  1748 raft_consensus.cc:2468] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 16.
W20260709 13:01:49.897346  7714 consensus_meta.cc:287] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.595s	user 0.011s	sys 0.002s
W20260709 13:01:50.019915  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 208869us
W20260709 13:01:50.026782  1717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:50.086463  1754 raft_consensus.cc:1240] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 13. Current term is 14. Ops: [13.7-13.7]
W20260709 13:01:50.089007  1645 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:50.181010  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:50.086624  7830 raft_consensus.cc:2804] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 13:01:49.835680  7684 consensus_queue.cc:1059] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 10.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.165s
I20260709 13:01:49.764618  7812 consensus_queue.cc:260] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.9, Last appended by leader: 9, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:50.092470  1680 raft_consensus.cc:1240] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 17. Current term is 18. Ops: []
I20260709 13:01:49.995126  7843 raft_consensus.cc:2749] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:01:49.963315  2301 leader_election.cc:304] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
W20260709 13:01:50.109290  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 91867.4us
I20260709 13:01:49.896715  7740 raft_consensus.cc:515] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:50.011202  7307 raft_consensus.cc:493] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:01:50.174888  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 351025us
I20260709 13:01:50.199115  1571 leader_election.cc:304] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:49.926218  7787 raft_consensus.cc:515] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:50.083703  7658 raft_consensus.cc:3060] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:01:49.896991  7577 raft_consensus.cc:2804] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:01:50.178181  7714 raft_consensus.cc:2749] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:01:50.180826  1745 raft_consensus.cc:1240] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 15. Current term is 16. Ops: []
I20260709 13:01:50.110317  1749 raft_consensus.cc:1275] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 16 index: 12. (index mismatch)
W20260709 13:01:50.187266  2300 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:50.000168  7677 raft_consensus.cc:515] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:50.100615  7468 leader_election.cc:290] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:50.211930  1718 leader_election.cc:304] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:49.887439  7818 consensus_queue.cc:260] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.11, Last appended by leader: 11, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:50.125808  7753 raft_consensus.cc:3060] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:01:50.172186  7560 leader_election.cc:290] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:50.227954  1671 raft_consensus.cc:1240] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 11. Current term is 12. Ops: []
I20260709 13:01:50.214373  1571 leader_election.cc:304] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:50.109659  7700 leader_election.cc:290] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:49.972741  7745 consensus_queue.cc:260] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.8, Last appended by leader: 8, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:01:50.238049  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 354535us
I20260709 13:01:50.238984  1643 leader_election.cc:304] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:50.239286  1569 leader_election.cc:304] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:50.169822  7671 raft_consensus.cc:515] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:50.200453  7657 raft_consensus.cc:697] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 18 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:01:50.208038  7540 leader_election.cc:290] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:50.225996  2336 raft_consensus.cc:2468] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 12.
I20260709 13:01:50.216163  7577 raft_consensus.cc:493] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:50.247493  1569 leader_election.cc:400] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 15 election: Vote denied by peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625) with higher term. Message: Invalid argument: T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 15. Current term is 16.
I20260709 13:01:50.228685  7833 raft_consensus.cc:2804] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:01:50.228513  7686 leader_election.cc:290] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 17 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:50.212610  7307 raft_consensus.cc:515] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:50.303634  1645 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:50.200273  7181 consensus_queue.cc:1048] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.045s
I20260709 13:01:50.325173  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:50.374476  2336 raft_consensus.cc:2468] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 13.
I20260709 13:01:50.330562  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:50.333197  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:01:50.370827  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 178979us
I20260709 13:01:50.299403  7745 raft_consensus.cc:3060] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:01:50.378326  2328 raft_consensus.cc:2468] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 16.
I20260709 13:01:50.315941  7687 raft_consensus.cc:2804] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Leader election won for term 18
W20260709 13:01:50.387439  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 148521us
I20260709 13:01:50.388523  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
W20260709 13:01:50.389633  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 203985us
I20260709 13:01:50.319720  7833 raft_consensus.cc:493] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
W20260709 13:01:50.371392  7829 log.cc:927] Time spent T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.524s	user 0.000s	sys 0.001s
I20260709 13:01:50.396659  1749 raft_consensus.cc:1275] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 16 index: 13. (index mismatch)
W20260709 13:01:50.407919  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 70891.6us
W20260709 13:01:50.264971  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 50676us
I20260709 13:01:50.436131  1571 leader_election.cc:304] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:50.409329  1569 leader_election.cc:403] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 15 election: Cancelling election due to peer responding with higher term
I20260709 13:01:50.224476  7728 raft_consensus.cc:2804] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:01:50.424445  1754 raft_consensus.cc:1275] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 16 index: 13. Preceding OpId from leader: term: 17 index: 14. (index mismatch)
W20260709 13:01:50.431228  1597 consensus_meta.cc:287] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.627s	user 0.011s	sys 0.002s
W20260709 13:01:50.368643  1751 consensus_meta.cc:287] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.613s	user 0.011s	sys 0.000s
I20260709 13:01:50.249943  7820 raft_consensus.cc:2749] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:01:50.230598  7830 raft_consensus.cc:697] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 21 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:01:50.620314  7720 raft_consensus.cc:493] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:50.794029  7787 leader_election.cc:290] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 12 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:50.794836  7671 leader_election.cc:290] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:01:50.528635  1717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:50.769279  7854 log.cc:927] Time spent T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.085s	user 0.000s	sys 0.001s
I20260709 13:01:50.479975  7782 raft_consensus.cc:2749] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Leader election lost for term 15. Reason: Vote denied by peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625) with higher term. Message: Invalid argument: T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 15. Current term is 16.
I20260709 13:01:50.243271  7702 consensus_queue.cc:1059] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.301s
I20260709 13:01:50.609794  1671 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52766 (request call id 2929) took 1213 ms. Trace:
I20260709 13:01:50.586476  7677 leader_election.cc:290] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 18 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:50.594565  1746 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55848 (request call id 2963) took 1094 ms. Trace:
W20260709 13:01:50.836181  7684 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.603s	user 0.011s	sys 0.006s
I20260709 13:01:50.587139  7816 leader_election.cc:290] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:01:50.591322  7855 log.cc:927] Time spent T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.064s	user 0.000s	sys 0.000s
I20260709 13:01:50.617718  1745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55848 (request call id 2965) took 1116 ms. Trace:
I20260709 13:01:50.552240  7600 leader_election.cc:290] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 18 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:01:50.591677  7828 log.cc:927] Time spent T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.838s	user 0.006s	sys 0.001s
I20260709 13:01:50.624082  7657 consensus_queue.cc:237] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.12, Last appended by leader: 12, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:50.625221  7837 log.cc:927] Time spent T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.580s	user 0.001s	sys 0.001s
I20260709 13:01:50.233160  7852 consensus_queue.cc:1059] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 17.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.337s
I20260709 13:01:50.442725  7687 raft_consensus.cc:697] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:01:50.307667  7831 consensus_queue.cc:1059] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 17.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.298s
I20260709 13:01:50.441901  7728 raft_consensus.cc:493] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:01:50.593950  7839 log.cc:927] Time spent T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.378s	user 0.000s	sys 0.001s
I20260709 13:01:50.898068  7728 raft_consensus.cc:3060] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:01:50.587306  7835 raft_consensus.cc:493] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
W20260709 13:01:50.589325  7846 log.cc:927] Time spent T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.085s	user 0.000s	sys 0.000s
I20260709 13:01:50.428653  7577 raft_consensus.cc:3060] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:01:50.586948  7662 raft_consensus.cc:493] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:50.337575  7834 raft_consensus.cc:2804] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:01:50.336407  7850 consensus_queue.cc:1059] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 17.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.185s
I20260709 13:01:50.250917  7826 raft_consensus.cc:493] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
W20260709 13:01:50.395700  7836 log.cc:927] Time spent T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.291s	user 0.001s	sys 0.000s
I20260709 13:01:50.395939  7813 raft_consensus.cc:2749] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:01:50.721653  7819 raft_consensus.cc:493] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:50.339622  7727 leader_election.cc:290] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:50.422544  7822 leader_election.cc:290] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:50.423987  7833 raft_consensus.cc:3060] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:01:50.843082  7812 raft_consensus.cc:3060] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:01:50.822332  7725 raft_consensus.cc:493] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:50.858702  7830 consensus_queue.cc:237] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.13, Last appended by leader: 13, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:50.865939  7740 leader_election.cc:290] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 13:01:50.889166  7753 consensus_meta.cc:287] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.645s	user 0.009s	sys 0.000s
I20260709 13:01:50.890220  7778 raft_consensus.cc:3055] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:01:50.905489  7826 raft_consensus.cc:515] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:50.910154  7778 raft_consensus.cc:740] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:01:50.905097  7819 raft_consensus.cc:515] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:01:50.892954  7817 consensus_meta.cc:287] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 1.044s	user 0.012s	sys 0.000s
I20260709 13:01:50.901443  7834 raft_consensus.cc:493] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:50.894814  7818 raft_consensus.cc:3060] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Advancing to term 18
W20260709 13:01:50.897271  7761 consensus_meta.cc:287] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 1.133s	user 0.012s	sys 0.000s
I20260709 13:01:50.873464  7784 raft_consensus.cc:493] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:50.911877  7778 consensus_queue.cc:260] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.10, Last appended by leader: 10, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:50.913653  7826 leader_election.cc:290] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:50.913686  7819 leader_election.cc:290] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:50.912897  7834 raft_consensus.cc:3060] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:01:50.898887  7835 raft_consensus.cc:515] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:50.817667  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 570260us
I20260709 13:01:50.969805  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:50.972321  1603 raft_consensus.cc:3060] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:01:50.973176  1569 leader_election.cc:304] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
W20260709 13:01:49.756664  1497 rpcz_store.cc:269] 0709 13:00:59.473217 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:59.473694 (+   477us) service_pool.cc:224] Handling call
0709 13:00:59.958785 (+485091us) spinlock_profiling.cc:243] Waited 16.5 ms on lock 0x7b540041cb54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cc090 00007f426616af3a 00007f426bf211cc 00007f426c01f37e 00007f426a4e02a4 00007f426a4e0232 00007f426a4e0161 00007f426a4e00dc 00007f426a4df392 00007f426449d48c 00007f426449c8c6
0709 13:01:49.469255 (+49510470us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 13:01:42.880949 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 13:01:42.947390 (+ 66441us) write_op.cc:432] Acquiring schema lock in shared mode
0709 13:01:42.947461 (+    71us) write_op.cc:435] Acquired schema lock
0709 13:01:42.947482 (+    21us) tablet.cc:662] Decoding operations
0709 13:01:42.971646 (+ 24164us) write_op.cc:620] Acquiring the partition lock for write op
0709 13:01:42.987004 (+ 15358us) write_op.cc:641] Partition lock acquired for write op
0709 13:01:42.991764 (+  4760us) tablet.cc:685] Acquiring locks for 9 operations
0709 13:01:43.028150 (+ 36386us) tablet.cc:701] Row locks acquired
0709 13:01:43.028211 (+    61us) write_op.cc:260] PREPARE: finished
0709 13:01:43.028462 (+   251us) write_op.cc:270] Start()
0709 13:01:43.028674 (+   212us) write_op.cc:276] Timestamp: P: 1783602103028371 usec, L: 0
0709 13:01:43.028695 (+    21us) op_driver.cc:348] REPLICATION: starting
0709 13:01:43.029589 (+   894us) log.cc:844] Serialized 3601 byte log entry
0709 13:01:43.519688 (+490099us) op_driver.cc:464] REPLICATION: finished
0709 13:01:45.062255 (+1542567us) write_op.cc:301] APPLY: starting
0709 13:01:45.062425 (+   170us) tablet.cc:1367] starting BulkCheckPresence
0709 13:01:46.360491 (+1298066us) tablet.cc:1370] finished BulkCheckPresence
0709 13:01:46.384551 (+ 24060us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 13:01:47.129333 (+744782us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 13:01:47.129764 (+   431us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=18,key_file_lookups=18,delta_file_lookups=18,mrs_lookups=0
0709 13:01:47.129822 (+    58us) write_op.cc:312] APPLY: finished
0709 13:01:47.130315 (+   493us) log.cc:844] Serialized 89 byte log entry
0709 13:01:47.752899 (+622584us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:01:47.781593 (+ 28694us) write_op.cc:454] Released schema lock
0709 13:01:47.781740 (+   147us) write_op.cc:341] FINISH: Updating metrics
0709 13:01:47.808967 (+ 27227us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:01:47.811341 (+  2374us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":16318,"reported_tablets":36,"spinlock_wait_cycles":37972736,"child_traces":[["op",{"apply.queue_time_us":1494832,"apply.run_cpu_time_us":210283,"apply.run_wall_time_us":2746417,"cfile_cache_hit":38,"cfile_cache_hit_bytes":41819,"num_ops":9,"prepare.queue_time_us":134142,"prepare.run_cpu_time_us":25278,"prepare.run_wall_time_us":325936,"raft.queue_time_us":248929,"raft.run_cpu_time_us":8617,"raft.run_wall_time_us":153445,"replication_time_us":469078,"spinlock_wait_cycles":16256,"thread_start_us":827788,"threads_started":5,"wal-append.queue_time_us":851254,"wal-append.run_cpu_time_us":64847,"wal-append.run_wall_time_us":2415729}]]}
I20260709 13:01:50.818750  1571 leader_election.cc:304] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:50.443809  1597 raft_consensus.cc:2468] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 13.
I20260709 13:01:50.610841  1598 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40154 (request call id 2622) took 1148 ms. Trace:
W20260709 13:01:50.975571  1597 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36462 (request call id 1513) took 1321 ms (client timeout 1517 ms). Trace:
W20260709 13:01:50.976256  1597 rpcz_store.cc:269] 0709 13:01:49.654119 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:49.722018 (+ 67899us) service_pool.cc:224] Handling call
0709 13:01:50.975259 (+1253241us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:01:50.722581  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:50.975589  1598 rpcz_store.cc:276] 0709 13:01:49.461921 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:49.463996 (+  2075us) service_pool.cc:224] Handling call
0709 13:01:50.610598 (+1146602us) spinlock_profiling.cc:243] Waited 268 ms on lock 0x7b54002a3354. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:01:50.610743 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":615576448}
W20260709 13:01:50.431236  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 236142us
W20260709 13:01:50.693709  1645 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:50.983116  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 164311us
I20260709 13:01:50.985108  1571 leader_election.cc:304] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
W20260709 13:01:50.521708  1673 consensus_meta.cc:287] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.751s	user 0.007s	sys 0.004s
I20260709 13:01:50.989737  1673 raft_consensus.cc:2468] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 12.
I20260709 13:01:50.990105  1676 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52778 (request call id 2476) took 2159 ms. Trace:
W20260709 13:01:50.990692  1673 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52766 (request call id 2930) took 1593 ms (client timeout 1773 ms). Trace:
I20260709 13:01:50.518203  1678 raft_consensus.cc:2468] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 18.
W20260709 13:01:50.679746  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 255535us
I20260709 13:01:50.991161  1752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55840 (request call id 2859) took 1591 ms. Trace:
I20260709 13:01:50.992172  1674 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52766 (request call id 2926) took 1598 ms. Trace:
W20260709 13:01:50.634549  1718 leader_election.cc:336] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 17 pre-election: RPC error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Timed out: RequestConsensusVote RPC to 127.0.107.194:46261 timed out after 1.521s (SENT)
I20260709 13:01:50.993117  1643 leader_election.cc:304] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
W20260709 13:01:50.542239  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 119693us
I20260709 13:01:50.874030  1745 rpcz_store.cc:276] 0709 13:01:49.501106 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:49.506165 (+  5059us) service_pool.cc:224] Handling call
0709 13:01:50.129154 (+622989us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:50.212661 (+ 83507us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:50.617442 (+404781us) spinlock_profiling.cc:243] Waited 605 ms on lock 0x7b5400220950. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661618ad 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:01:50.617628 (+   186us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1392565632}
W20260709 13:01:50.442936  1675 consensus_meta.cc:287] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.697s	user 0.013s	sys 0.000s
I20260709 13:01:50.866338  1746 rpcz_store.cc:276] 0709 13:01:49.500414 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:49.506725 (+  6311us) service_pool.cc:224] Handling call
0709 13:01:50.104773 (+598048us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:50.105535 (+   762us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:50.594265 (+488730us) spinlock_profiling.cc:243] Waited 598 ms on lock 0x7b54001d42d0. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661618ad 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:01:50.594474 (+   209us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1374397312}
I20260709 13:01:50.481042  1751 raft_consensus.cc:2468] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 13.
I20260709 13:01:50.996771  1675 raft_consensus.cc:2468] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 13.
I20260709 13:01:50.997895  1751 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55848 (request call id 2970) took 1458 ms. Trace:
W20260709 13:01:50.715571  1677 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.519s	user 0.066s	sys 0.098s
I20260709 13:01:50.998417  1751 rpcz_store.cc:276] 0709 13:01:49.539675 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:49.587302 (+ 47627us) service_pool.cc:224] Handling call
0709 13:01:50.997502 (+1410200us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:01:50.999040  1677 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.805s	user 0.074s	sys 0.100s
W20260709 13:01:50.760421  1753 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.554s	user 0.012s	sys 0.021s
W20260709 13:01:51.000154  1753 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.794s	user 0.018s	sys 0.022s
I20260709 13:01:50.505198  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
W20260709 13:01:50.871289  7684 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.644s	user 0.011s	sys 0.006s
I20260709 13:01:51.001760  1753 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55840 (request call id 2863) took 1335 ms. Trace:
W20260709 13:01:50.887657  7579 consensus_meta.cc:287] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 1.079s	user 0.013s	sys 0.001s
I20260709 13:01:50.837019  7307 leader_election.cc:290] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:50.864015  1671 rpcz_store.cc:276] 0709 13:01:49.396640 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:50.094870 (+698230us) service_pool.cc:224] Handling call
0709 13:01:50.095284 (+   414us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:50.228515 (+133231us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:50.609451 (+380936us) spinlock_profiling.cc:243] Waited 132 ms on lock 0x7b540025add4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:01:50.609704 (+   253us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":304432000}
I20260709 13:01:51.003304  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:51.004254  7307 raft_consensus.cc:493] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:01:50.887207  2335 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39726 (request call id 1407) took 1043 ms. Trace:
I20260709 13:01:51.005620  1671 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131"
dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
mode: GRACEFUL
new_leader_uuid: "af520b67945d443b901bbeca37cc5701"
 from {username='slave'} at 127.0.0.1:43266
I20260709 13:01:51.005416  2335 rpcz_store.cc:276] 0709 13:01:49.844001 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:49.844189 (+   188us) service_pool.cc:224] Handling call
0709 13:01:49.844625 (+   436us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:49.960343 (+115718us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:49.960396 (+    53us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:01:49.971225 (+ 10829us) log.cc:844] Serialized 26 byte log entry
0709 13:01:50.331459 (+360234us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:01:50.331513 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:50.331601 (+    88us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:50.886854 (+555253us) raft_consensus.cc:1685] finished
0709 13:01:50.886892 (+    38us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:50.887106 (+   214us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":42198,"raft.queue_time_us":102763,"raft.run_cpu_time_us":211,"raft.run_wall_time_us":49781,"thread_start_us":235111,"threads_started":1,"wal-append.queue_time_us":521112}
I20260709 13:01:51.005302  7307 raft_consensus.cc:515] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:01:51.008036  7582 consensus_meta.cc:287] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 1.039s	user 0.009s	sys 0.005s
I20260709 13:01:51.008818  7582 raft_consensus.cc:2749] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:01:51.010331  7585 consensus_queue.cc:1059] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 10.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 13:01:51.011624  7307 leader_election.cc:290] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:50.903091  7702 raft_consensus.cc:3055] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 13:01:51.013409  1750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55848 (request call id 2964) took 1512 ms. Trace:
I20260709 13:01:51.013484  7702 raft_consensus.cc:740] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:01:50.822584  7686 raft_consensus.cc:2804] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:01:51.015070  7686 raft_consensus.cc:493] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:01:50.752986  2300 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:51.015792  7687 consensus_queue.cc:237] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.10, Last appended by leader: 10, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:51.015668  7702 consensus_queue.cc:260] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.14, Last appended by leader: 14, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:50.837795  2301 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.615s	user 0.000s	sys 0.017s
W20260709 13:01:51.019142  2301 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.796s	user 0.000s	sys 0.017s
I20260709 13:01:51.019311  7702 raft_consensus.cc:3060] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Advancing to term 21
W20260709 13:01:51.019944  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 1.05658e+06us
I20260709 13:01:51.022997  7647 raft_consensus.cc:2749] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:01:50.902570  7662 raft_consensus.cc:515] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:50.590643  7181 raft_consensus.cc:493] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
W20260709 13:01:50.991053  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:51.025027  7181 raft_consensus.cc:515] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:51.027755  7684 raft_consensus.cc:493] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:51.028669  7684 raft_consensus.cc:515] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:51.029235  7662 leader_election.cc:290] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:01:50.991289  1673 rpcz_store.cc:269] 0709 13:01:49.396941 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:49.439823 (+ 42882us) service_pool.cc:224] Handling call
0709 13:01:50.990552 (+1550729us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:01:51.031651  1644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:51.034343  1673 raft_consensus.cc:1275] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 15 index: 8. Preceding OpId from leader: term: 17 index: 9. (index mismatch)
I20260709 13:01:51.034657  7686 raft_consensus.cc:3060] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:01:51.041229  7647 consensus_queue.cc:1048] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 13:01:50.993815  1718 leader_election.cc:304] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:50.992668  1674 rpcz_store.cc:276] 0709 13:01:49.393834 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:49.583717 (+189883us) service_pool.cc:224] Handling call
0709 13:01:49.584063 (+   346us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:49.601283 (+ 17220us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:49.601319 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:01:49.680856 (+ 79537us) log.cc:844] Serialized 26 byte log entry
0709 13:01:49.925420 (+244564us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:01:49.925476 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:49.925553 (+    77us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:50.991526 (+1065973us) raft_consensus.cc:1685] finished
0709 13:01:50.991573 (+    47us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:50.991903 (+   330us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":36199,"raft.queue_time_us":17886,"raft.run_cpu_time_us":189,"raft.run_wall_time_us":31269,"thread_start_us":151991,"threads_started":1,"wal-append.queue_time_us":380874}
I20260709 13:01:51.038401  7437 raft_consensus.cc:3055] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 LEADER]: Stepping down as leader of term 17
W20260709 13:01:50.895581  7750 consensus_meta.cc:287] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 1.151s	user 0.012s	sys 0.003s
I20260709 13:01:51.043838  7437 raft_consensus.cc:740] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:01:51.059978  7437 consensus_queue.cc:260] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.10, Last appended by leader: 10, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:51.061252  2301 leader_election.cc:304] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:51.061504  7725 raft_consensus.cc:515] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:51.063203  7727 raft_consensus.cc:2804] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:01:51.063905  7727 raft_consensus.cc:493] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:51.064522  7727 raft_consensus.cc:3060] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:01:50.915539  7817 raft_consensus.cc:515] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:51.066040  1605 raft_consensus.cc:3060] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Advancing to term 14
W20260709 13:01:51.014192  1678 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52778 (request call id 2481) took 1926 ms (client timeout 1670 ms). Trace:
W20260709 13:01:51.067457  1678 rpcz_store.cc:269] 0709 13:01:49.097456 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:49.098151 (+   695us) service_pool.cc:224] Handling call
0709 13:01:51.013675 (+1915524us) spinlock_profiling.cc:243] Waited 987 ms on lock 0x7b54001f5254. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:01:51.013914 (+   239us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2270541056}
I20260709 13:01:51.068202  7714 raft_consensus.cc:3055] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 LEADER]: Stepping down as leader of term 17
I20260709 13:01:51.068763  7714 raft_consensus.cc:740] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:01:51.069494  7778 raft_consensus.cc:3060] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:01:51.070525  7830 raft_consensus.cc:2804] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Leader election won for term 18
W20260709 13:01:51.070781  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 75051.9us
I20260709 13:01:51.071245  7830 raft_consensus.cc:697] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 18 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:01:51.049706  7808 raft_consensus.cc:2804] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Leader election won for term 16
I20260709 13:01:51.073675  1674 raft_consensus.cc:1275] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 17 index: 11. (index mismatch)
I20260709 13:01:51.073819  7852 consensus_queue.cc:1048] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 13:01:51.074666  1674 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52778 (request call id 2495) took 1479 ms. Trace:
W20260709 13:01:51.075085  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 440536us
I20260709 13:01:51.073282  7808 raft_consensus.cc:697] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:01:50.837502  7720 raft_consensus.cc:515] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:51.078902  7740 raft_consensus.cc:2749] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:01:51.047572  7761 raft_consensus.cc:515] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:51.079267  7725 leader_election.cc:290] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:51.080682  7720 leader_election.cc:290] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:51.081332  7750 raft_consensus.cc:2749] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:01:51.006315  1680 raft_consensus.cc:1240] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 16 LEADER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 15. Current term is 16. Ops: []
I20260709 13:01:51.039067  7560 raft_consensus.cc:3055] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:01:51.047801  7753 raft_consensus.cc:515] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:51.069628  1678 raft_consensus.cc:1240] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 12. Current term is 13. Ops: []
W20260709 13:01:50.997999  1675 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52766 (request call id 2933) took 1599 ms (client timeout 1520 ms). Trace:
I20260709 13:01:51.075165  1674 rpcz_store.cc:276] 0709 13:01:49.595132 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:51.072484 (+1477352us) service_pool.cc:224] Handling call
0709 13:01:51.073065 (+   581us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:51.074226 (+  1161us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:51.074558 (+   332us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:01:50.991801  1752 rpcz_store.cc:276] 0709 13:01:49.399365 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:49.416573 (+ 17208us) service_pool.cc:224] Handling call
0709 13:01:49.417030 (+   457us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:01:49.417488 (+   458us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:49.417520 (+    32us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:01:49.436435 (+ 18915us) log.cc:844] Serialized 52 byte log entry
0709 13:01:49.735134 (+298699us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:01:49.735192 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:49.735293 (+   101us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:50.990396 (+1255103us) raft_consensus.cc:1685] finished
0709 13:01:50.990467 (+    71us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:50.990834 (+   367us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":4788,"raft.run_cpu_time_us":201,"raft.run_wall_time_us":14932,"thread_start_us":176327,"threads_started":1,"wal-append.queue_time_us":271547}
I20260709 13:01:51.076325  7813 raft_consensus.cc:2749] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
W20260709 13:01:51.056221  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 83530.2us
I20260709 13:01:51.077025  7600 consensus_queue.cc:1048] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 13:01:51.077461  1646 leader_election.cc:304] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:51.039299  7181 leader_election.cc:290] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:51.034052  7700 raft_consensus.cc:493] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:01:51.084412  1675 rpcz_store.cc:269] 0709 13:01:49.398641 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:49.418210 (+ 19569us) service_pool.cc:224] Handling call
0709 13:01:50.997846 (+1579636us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:01:51.084649  1673 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52778 (request call id 2491) took 1452 ms. Trace:
I20260709 13:01:51.093640  1673 rpcz_store.cc:276] 0709 13:01:49.593918 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:51.032442 (+1438524us) service_pool.cc:224] Handling call
0709 13:01:51.033536 (+  1094us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:51.037496 (+  3960us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:51.046169 (+  8673us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:01:51.099097  7826 consensus_queue.cc:1059] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 15.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 13:01:51.083984  7560 raft_consensus.cc:740] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:01:51.048231  7784 raft_consensus.cc:515] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:51.109766  7753 leader_election.cc:290] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 15 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:50.990815  1676 rpcz_store.cc:276] 0709 13:01:48.830051 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:48.830193 (+   142us) service_pool.cc:224] Handling call
0709 13:01:48.830715 (+   522us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:49.512953 (+682238us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:49.512984 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:01:49.513532 (+   548us) log.cc:844] Serialized 26 byte log entry
0709 13:01:49.778100 (+264568us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:01:49.778157 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:49.778264 (+   107us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:50.989434 (+1211170us) raft_consensus.cc:1685] finished
0709 13:01:50.989481 (+    47us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:50.989824 (+   343us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":179095,"raft.run_cpu_time_us":119,"raft.run_wall_time_us":3887,"thread_start_us":945727,"threads_started":2,"wal-append.queue_time_us":350608}
I20260709 13:01:51.091385  1671 raft_consensus.cc:606] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 LEADER]: Received request to transfer leadership to TS af520b67945d443b901bbeca37cc5701
I20260709 13:01:51.110729  7750 consensus_queue.cc:1059] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 13:01:51.111968  1676 raft_consensus.cc:1240] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 12. Current term is 13. Ops: []
I20260709 13:01:51.112169  7750 raft_consensus.cc:3055] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:01:51.112645  7750 raft_consensus.cc:740] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
W20260709 13:01:51.114588  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 210328us
I20260709 13:01:51.076489  7845 consensus_queue.cc:1059] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 10.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.040s
I20260709 13:01:51.088795  7714 consensus_queue.cc:260] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:51.117233  7835 leader_election.cc:290] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:51.002344  1753 rpcz_store.cc:276] 0709 13:01:49.666764 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:50.105047 (+438283us) service_pool.cc:224] Handling call
0709 13:01:50.105424 (+   377us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:50.105850 (+   426us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:50.105889 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:01:50.126626 (+ 20737us) log.cc:844] Serialized 26 byte log entry
0709 13:01:51.001040 (+874414us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:01:51.001180 (+   140us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:51.001326 (+   146us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:51.001391 (+    65us) raft_consensus.cc:1685] finished
0709 13:01:51.001406 (+    15us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:51.001628 (+   222us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1014,"raft.run_cpu_time_us":175,"raft.run_wall_time_us":61276,"thread_start_us":794456,"threads_started":1,"wal-append.queue_time_us":632724}
I20260709 13:01:51.118501  7714 raft_consensus.cc:3060] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:01:51.096478  1678 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52778 (request call id 2493) took 1497 ms. Trace:
I20260709 13:01:51.119922  1678 rpcz_store.cc:276] 0709 13:01:49.594564 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:51.068565 (+1474001us) service_pool.cc:224] Handling call
0709 13:01:51.069259 (+   694us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:51.089891 (+ 20632us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:51.091630 (+  1739us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:01:51.088162  7437 raft_consensus.cc:3060] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:01:51.110327  1569 leader_election.cc:304] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 13:01:51.137529  7863 consensus_queue.cc:1048] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.044s
I20260709 13:01:51.101671  7560 consensus_queue.cc:260] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:51.014098  1750 rpcz_store.cc:276] 0709 13:01:49.500769 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:49.507247 (+  6478us) service_pool.cc:224] Handling call
0709 13:01:49.507620 (+   373us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:49.793553 (+285933us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:49.793635 (+    82us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:01:49.794322 (+   687us) log.cc:844] Serialized 26 byte log entry
0709 13:01:50.214651 (+420329us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:01:50.214694 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:50.214768 (+    74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:51.012676 (+797908us) raft_consensus.cc:1685] finished
0709 13:01:51.012758 (+    82us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:51.013132 (+   374us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":4205,"raft.run_cpu_time_us":89,"raft.run_wall_time_us":89,"thread_start_us":666827,"threads_started":2,"wal-append.queue_time_us":485697}
I20260709 13:01:51.142380  7560 raft_consensus.cc:3060] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:01:51.143023  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:51.158454  1752 raft_consensus.cc:2393] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 17: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:01:51.072813  7830 consensus_queue.cc:237] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:51.160331  7869 raft_consensus.cc:3055] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 13:01:51.160982  7869 raft_consensus.cc:740] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:01:50.963225  1496 catalog_manager.cc:5697] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 11 to 13. New cstate: current_term: 13 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } }
I20260709 13:01:51.163524  1751 raft_consensus.cc:1240] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 13. Current term is 14. Ops: []
W20260709 13:01:51.164732  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 48951.3us
I20260709 13:01:51.164539  1496 catalog_manager.cc:5697] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 6 to 8. New cstate: current_term: 8 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } }
I20260709 13:01:51.167095  2298 leader_election.cc:304] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:51.168499  7819 raft_consensus.cc:2804] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:01:51.169015  7819 raft_consensus.cc:493] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:51.169478  7819 raft_consensus.cc:3060] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Advancing to term 15
W20260709 13:01:51.172513  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 81045.7us
I20260709 13:01:51.173575  2301 leader_election.cc:304] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:51.173915  7856 raft_consensus.cc:2804] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 13:01:51.174767  7585 raft_consensus.cc:2804] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:01:51.174818  7856 raft_consensus.cc:697] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 13 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:01:51.175720  7585 raft_consensus.cc:697] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 14 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:01:51.176069  7181 consensus_queue.cc:1048] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 13:01:51.177345  7585 consensus_queue.cc:237] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.10, Last appended by leader: 10, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:51.168792  7869 consensus_queue.cc:260] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.12, Last appended by leader: 12, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:51.179527  7856 consensus_queue.cc:237] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 12.12, Last appended by leader: 12, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:51.114287  7750 consensus_queue.cc:260] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 14.9, Last appended by leader: 9, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } }
I20260709 13:01:51.183687  7684 leader_election.cc:290] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:01:51.146536  1645 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:51.149250  1680 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57092 (request call id 1356) took 1709 ms. Trace:
I20260709 13:01:51.153241  1747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55848 (request call id 2967) took 1651 ms. Trace:
W20260709 13:01:51.118564  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 126054us
W20260709 13:01:51.194402  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 84226.1us
I20260709 13:01:51.049000  7816 raft_consensus.cc:2804] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Leader election won for term 13
I20260709 13:01:51.193954  1680 rpcz_store.cc:276] 0709 13:01:49.439570 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:51.005281 (+1565711us) service_pool.cc:224] Handling call
0709 13:01:51.005895 (+   614us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:51.084446 (+ 78551us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:51.148877 (+ 64431us) spinlock_profiling.cc:243] Waited 35.1 ms on lock 0x7b540028f8a0. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42660d9dc0 00007f4266165a02 00007f4266161dd6 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2
0709 13:01:51.149149 (+   272us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":80801536}
W20260709 13:01:51.151885  1718 leader_election.cc:336] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 17 pre-election: RPC error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Timed out: RequestConsensusVote RPC to 127.0.107.194:46261 timed out after 1.566s (SENT)
I20260709 13:01:51.109822  7784 leader_election.cc:290] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:51.144860  7761 leader_election.cc:290] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 17 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:51.149464  7700 raft_consensus.cc:515] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:51.114285  7754 consensus_queue.cc:1059] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 12.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.102s
I20260709 13:01:51.216475  7754 raft_consensus.cc:3055] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:01:51.216818  7700 leader_election.cc:290] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:51.216897  7754 raft_consensus.cc:740] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:01:51.208756  7813 raft_consensus.cc:3055] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 LEADER]: Stepping down as leader of term 15
W20260709 13:01:51.218703  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 141471us
I20260709 13:01:51.218935  7813 raft_consensus.cc:740] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:01:51.219368  7754 consensus_queue.cc:260] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.7, Last appended by leader: 7, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:51.220726  7753 raft_consensus.cc:2804] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Leader election won for term 13
I20260709 13:01:51.220418  7813 consensus_queue.cc:260] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.10, Last appended by leader: 10, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:51.221184  7754 raft_consensus.cc:3060] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:01:51.221349  7753 raft_consensus.cc:697] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:01:51.222602  7826 raft_consensus.cc:493] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:51.085723  7808 consensus_queue.cc:237] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.10, Last appended by leader: 10, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:01:51.223022  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 71306.5us
I20260709 13:01:51.222900  7753 consensus_queue.cc:237] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.8, Last appended by leader: 8, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:51.223506  7826 raft_consensus.cc:515] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:51.227805  7750 raft_consensus.cc:3060] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:01:51.229897  7826 leader_election.cc:290] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:51.230649  1751 raft_consensus.cc:1275] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 16 index: 12. (index mismatch)
I20260709 13:01:51.234619  7817 leader_election.cc:290] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:51.239277  7869 raft_consensus.cc:3060] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:01:51.195264  1747 rpcz_store.cc:276] 0709 13:01:49.501721 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:49.901612 (+399891us) service_pool.cc:224] Handling call
0709 13:01:49.901929 (+   317us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:49.902208 (+   279us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 13:01:49.998935 (+ 96727us) log.cc:844] Serialized 14 byte log entry
0709 13:01:51.034934 (+1035999us) spinlock_profiling.cc:243] Waited 8.15 ms on lock 0x7b18003f60a0. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f4263057e07 00007f426beb2ab9 00007f426307469b 00007f426307293f 00007f4263074c02 00007f4265cfc20b 00007f4265d01d23 00007f4265d0223a 00007f426616ca2a
0709 13:01:51.152317 (+117383us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:01:51.152439 (+   122us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:01:51.152606 (+   167us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:51.152748 (+   142us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:51.152973 (+   225us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":34316,"prepare.queue_time_us":96578,"prepare.run_cpu_time_us":8141,"prepare.run_wall_time_us":86760,"raft.queue_time_us":82525,"raft.run_cpu_time_us":145,"raft.run_wall_time_us":151,"spinlock_wait_cycles":18756352,"thread_start_us":11641,"threads_started":1,"wal-append.queue_time_us":1140936}
I20260709 13:01:51.249094  1717 leader_election.cc:304] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:51.257761  1603 raft_consensus.cc:2468] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 18.
I20260709 13:01:51.222337  7813 raft_consensus.cc:3060] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:01:51.211815  7816 raft_consensus.cc:697] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
W20260709 13:01:51.265141  1646 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:51.287355  1674 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52778 (request call id 2499) took 1691 ms. Trace:
I20260709 13:01:51.287969  1674 rpcz_store.cc:276] 0709 13:01:49.596273 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:51.085551 (+1489278us) service_pool.cc:224] Handling call
0709 13:01:51.210682 (+125131us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:51.286797 (+ 76115us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:51.286832 (+    35us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:01:51.286994 (+   162us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:01:51.287050 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:51.287128 (+    78us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:51.287270 (+   142us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:01:51.225976  7863 raft_consensus.cc:3055] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 LEADER]: Stepping down as leader of term 13
I20260709 13:01:51.292677  1673 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57092 (request call id 1357) took 1204 ms. Trace:
I20260709 13:01:51.294579  1605 raft_consensus.cc:2468] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 14.
W20260709 13:01:51.296700  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 47447.3us
I20260709 13:01:51.293788  1673 rpcz_store.cc:276] 0709 13:01:50.088717 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:51.134532 (+1045815us) service_pool.cc:224] Handling call
0709 13:01:51.226710 (+ 92178us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:51.292038 (+ 65328us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:51.292086 (+    48us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:01:51.292196 (+   110us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:01:51.292268 (+    72us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:51.292373 (+   105us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:51.292568 (+   195us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:01:51.273564  1496 catalog_manager.cc:5697] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 11 to 13, leader changed from 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194) to 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193). New cstate: current_term: 13 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } }
I20260709 13:01:51.304283  1496 catalog_manager.cc:5697] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 11 to 13, leader changed from 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) to 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196). New cstate: current_term: 13 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
W20260709 13:01:51.307464  7658 consensus_meta.cc:287] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 1.085s	user 0.013s	sys 0.013s
I20260709 13:01:51.308081  7658 raft_consensus.cc:515] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:51.309391  7647 raft_consensus.cc:493] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:51.310035  7647 raft_consensus.cc:515] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:51.315217  7816 consensus_queue.cc:237] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 12.7, Last appended by leader: 7, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:51.324820  7658 leader_election.cc:290] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 12 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:51.328295  7647 leader_election.cc:290] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:51.336171  7761 raft_consensus.cc:2804] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:01:51.336714  7761 raft_consensus.cc:493] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:51.337100  7761 raft_consensus.cc:3060] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:01:51.330209  7728 raft_consensus.cc:515] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:01:51.339108  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 90348us
W20260709 13:01:51.352061  1646 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:51.293715  7863 raft_consensus.cc:740] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:01:51.370512  1677 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52766 (request call id 2935) took 1944 ms. Trace:
W20260709 13:01:51.366302  2299 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.652s	user 0.000s	sys 0.028s
I20260709 13:01:51.381765  7686 raft_consensus.cc:515] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:51.384873  7686 leader_election.cc:290] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 15 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:51.386704  7687 consensus_queue.cc:1059] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 17.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 13:01:51.388381  7687 raft_consensus.cc:3055] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 LEADER]: Stepping down as leader of term 17
I20260709 13:01:51.389251  7707 raft_consensus.cc:493] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:51.389822  7830 raft_consensus.cc:493] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:51.389999  7707 raft_consensus.cc:515] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:51.390574  7830 raft_consensus.cc:515] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:51.391731  2328 raft_consensus.cc:1240] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 18 LEADER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 17. Current term is 18. Ops: []
I20260709 13:01:51.373951  1677 rpcz_store.cc:276] 0709 13:01:49.425958 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:50.106482 (+680524us) service_pool.cc:224] Handling call
0709 13:01:50.176176 (+ 69694us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:50.176525 (+   349us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 13:01:50.999946 (+823421us) log.cc:844] Serialized 14 byte log entry
0709 13:01:51.297192 (+297246us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:01:51.348496 (+ 51304us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:01:51.364194 (+ 15698us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:51.364298 (+   104us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:51.370402 (+  6104us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":56971,"prepare.queue_time_us":626719,"prepare.run_cpu_time_us":105,"prepare.run_wall_time_us":4615,"raft.queue_time_us":94144,"raft.run_cpu_time_us":216,"raft.run_wall_time_us":23920,"thread_start_us":1000897,"threads_started":2,"wal-append.queue_time_us":246978}
I20260709 13:01:51.388837  7687 raft_consensus.cc:740] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
W20260709 13:01:51.384918  7745 consensus_meta.cc:287] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.946s	user 0.014s	sys 0.008s
I20260709 13:01:51.393354  1672 raft_consensus.cc:2468] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 17.
W20260709 13:01:51.395399  2299 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.843s	user 0.000s	sys 0.029s
I20260709 13:01:51.268427  7307 consensus_queue.cc:1048] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.101s
I20260709 13:01:51.405296  7830 leader_election.cc:290] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:51.406912  1679 raft_consensus.cc:2468] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 18.
I20260709 13:01:51.340337  1496 catalog_manager.cc:5697] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 12 to 14. New cstate: current_term: 14 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } }
I20260709 13:01:51.415177  1496 catalog_manager.cc:5697] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 7 to 9. New cstate: current_term: 9 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } }
W20260709 13:01:51.427318  1605 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36462 (request call id 1521) took 1657 ms (client timeout 2177 ms). Trace:
W20260709 13:01:51.428017  1605 rpcz_store.cc:269] 0709 13:01:49.770344 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:49.796450 (+ 26106us) service_pool.cc:224] Handling call
0709 13:01:51.419107 (+1622657us) spinlock_profiling.cc:243] Waited 1.27 s on lock 0x7b54002792d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:01:51.427216 (+  8109us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2916116224}
I20260709 13:01:51.452755  2336 raft_consensus.cc:1240] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 17. Current term is 18. Ops: []
I20260709 13:01:51.453037  2331 raft_consensus.cc:1240] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 13 LEADER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 12. Current term is 13. Ops: []
W20260709 13:01:51.450649  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 195914us
I20260709 13:01:51.443131  1569 leader_election.cc:304] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
W20260709 13:01:51.429387  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 61496us
W20260709 13:01:51.445312  1718 leader_election.cc:336] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 17 pre-election: RPC error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Timed out: RequestConsensusVote RPC to 127.0.107.194:46261 timed out after 2.078s (SENT)
W20260709 13:01:51.469228  1644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:51.453758  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:51.454121  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:51.473542  2333 raft_consensus.cc:3060] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Advancing to term 12
W20260709 13:01:51.445129  7718 consensus_meta.cc:287] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 1.598s	user 0.014s	sys 0.010s
I20260709 13:01:51.474646  1718 leader_election.cc:304] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:51.467442  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:51.393958  7687 consensus_queue.cc:260] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 17.14, Last appended by leader: 14, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:51.478993  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:51.442737  7686 consensus_queue.cc:1059] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.043s
I20260709 13:01:51.436928  7707 leader_election.cc:290] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:51.444193  7728 leader_election.cc:290] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 17 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:51.488996  2334 raft_consensus.cc:3060] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Advancing to term 18
W20260709 13:01:51.495442  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:51.496822  7745 raft_consensus.cc:2804] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:01:51.497887  1597 raft_consensus.cc:2468] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 11.
I20260709 13:01:51.503774  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:51.508896  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:51.514688  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:51.442265  7863 consensus_queue.cc:260] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 13.7, Last appended by leader: 7, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:51.518595  7863 raft_consensus.cc:3060] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:01:51.518685  7577 raft_consensus.cc:515] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:51.520826  1679 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52778 (request call id 2501) took 1924 ms. Trace:
I20260709 13:01:51.521600  7577 leader_election.cc:290] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 18 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:51.518620  7834 raft_consensus.cc:515] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:51.523859  1645 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:51.525398  7834 leader_election.cc:290] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 17 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:51.526832  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:01:51.527860  2300 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:51.523919  2328 raft_consensus.cc:1240] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 16. Current term is 17. Ops: []
I20260709 13:01:51.529186  2299 leader_election.cc:304] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:01:51.530251  7662 raft_consensus.cc:2749] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 13:01:51.531401  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
W20260709 13:01:51.527149  1645 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:51.514851  1643 leader_election.cc:304] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
W20260709 13:01:51.536942  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:51.540729  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:51.554378  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:51.534736  2299 leader_election.cc:304] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:51.552935  1605 raft_consensus.cc:2468] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 14.
I20260709 13:01:51.533249  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:01:51.535223  1672 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52778 (request call id 2502) took 1938 ms (client timeout 1566 ms). Trace:
W20260709 13:01:51.568352  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 70150.3us
I20260709 13:01:51.521263  1679 rpcz_store.cc:276] 0709 13:01:49.596878 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:49.887154 (+290276us) service_pool.cc:224] Handling call
0709 13:01:51.520579 (+1633425us) spinlock_profiling.cc:243] Waited 1.31 s on lock 0x7b54000fe954. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:01:51.520745 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3002028288}
I20260709 13:01:51.540143  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:51.563558  2298 leader_election.cc:304] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:01:51.559851  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:51.550908  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:51.579244  2331 raft_consensus.cc:3060] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:01:51.556756  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:51.541198  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:51.555285  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:51.539415  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:51.582909  2333 raft_consensus.cc:2468] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 12.
W20260709 13:01:51.575675  1672 rpcz_store.cc:269] 0709 13:01:49.597190 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:49.700975 (+103785us) service_pool.cc:224] Handling call
0709 13:01:51.534967 (+1833992us) spinlock_profiling.cc:243] Waited 1.61 s on lock 0x7b54001e96d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:01:51.535137 (+   170us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3708516224}
I20260709 13:01:51.567030  7819 raft_consensus.cc:515] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:51.557013  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:51.550932  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:51.593967  1604 raft_consensus.cc:2393] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 18: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
W20260709 13:01:51.591080  7727 consensus_meta.cc:287] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.526s	user 0.009s	sys 0.007s
I20260709 13:01:51.598922  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:51.487931  7718 raft_consensus.cc:515] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:51.603698  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:01:51.604034  1646 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:51.597648  1674 raft_consensus.cc:2468] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 13.
I20260709 13:01:51.607612  1569 leader_election.cc:304] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:01:51.610224  1603 raft_consensus.cc:2468] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 12.
I20260709 13:01:51.609496  1747 raft_consensus.cc:2468] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 13.
I20260709 13:01:51.613385  7834 raft_consensus.cc:2804] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 13:01:51.613763  1753 raft_consensus.cc:2468] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 18.
I20260709 13:01:51.614135  7834 raft_consensus.cc:697] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 18 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:01:51.614725  7687 raft_consensus.cc:3060] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:01:51.615324  7834 consensus_queue.cc:237] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:51.616421  1719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:51.618029  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 103965us
I20260709 13:01:51.618801  1602 raft_consensus.cc:1240] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 14. Current term is 15. Ops: []
I20260709 13:01:51.609723  7307 raft_consensus.cc:2749] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:01:51.620030  7307 consensus_queue.cc:1048] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.000s
W20260709 13:01:51.620620  1717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:51.622462  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 87818.3us
I20260709 13:01:51.626194  7600 consensus_queue.cc:1059] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 17.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 13:01:51.626825  7818 raft_consensus.cc:2804] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Leader election won for term 17
I20260709 13:01:51.627247  7600 consensus_queue.cc:1059] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 12.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 13:01:51.627626  7818 raft_consensus.cc:697] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:01:51.628125  7600 consensus_queue.cc:1243] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer 3b879799899a499c8d409ee9f1277cd1 log is divergent from this leader: its last log entry 13.13 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 12
I20260709 13:01:51.629042  7818 consensus_queue.cc:237] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.9, Last appended by leader: 9, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:51.630674  7852 raft_consensus.cc:3055] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 LEADER]: Stepping down as leader of term 17
I20260709 13:01:51.598942  7727 raft_consensus.cc:515] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:51.631129  7852 raft_consensus.cc:740] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:01:51.631347  7850 raft_consensus.cc:3055] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:01:51.631893  7850 raft_consensus.cc:740] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:01:51.632128  7852 consensus_queue.cc:260] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:51.633646  7852 raft_consensus.cc:3060] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:01:51.633235  7850 consensus_queue.cc:260] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 12.12, Last appended by leader: 12, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:51.635663  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:51.618985  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:51.642227  1597 raft_consensus.cc:3060] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:01:51.641343  1677 raft_consensus.cc:2393] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 18: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:01:51.630638  7875 raft_consensus.cc:2749] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:01:51.423413  1496 catalog_manager.cc:5697] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 13 to 14, leader changed from 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) to 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194). New cstate: current_term: 14 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 13:01:51.591576  7745 raft_consensus.cc:697] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:01:51.630656  1598 raft_consensus.cc:2376] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 17. Re-sending same reply.
I20260709 13:01:51.629941  7600 consensus_queue.cc:1059] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 16.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 13:01:51.647712  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:51.649897  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:51.651245  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:51.653653  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:51.654866  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:01:51.657092  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 59008.5us
I20260709 13:01:51.667599  2328 raft_consensus.cc:1240] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 18 LEADER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 17. Current term is 18. Ops: []
I20260709 13:01:51.673457  2337 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "83226095f79a46dab445849b1febd425"
dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
mode: GRACEFUL
new_leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
 from {username='slave'} at 127.0.0.1:35790
I20260709 13:01:51.668234  2330 raft_consensus.cc:2468] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 13.
I20260709 13:01:51.674722  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:51.670289  2334 raft_consensus.cc:2468] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 18.
I20260709 13:01:51.680080  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:51.674185  2337 raft_consensus.cc:606] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 14 LEADER]: Received request to transfer leadership to TS 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:51.681700  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:51.685448  1752 raft_consensus.cc:2393] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 17: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:01:51.686518  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:51.687239  1752 raft_consensus.cc:3060] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Advancing to term 17
W20260709 13:01:51.690729  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 46164.1us
I20260709 13:01:51.692699  1496 catalog_manager.cc:5697] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 12 to 14, leader changed from 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194) to 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193). New cstate: current_term: 14 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } }
I20260709 13:01:51.685758  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:51.695488  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:51.696493  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:51.696914  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:51.697264  1754 raft_consensus.cc:3060] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:01:51.697602  2301 leader_election.cc:304] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:51.698585  7662 consensus_queue.cc:1048] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 13:01:51.698962  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:51.699854  7778 raft_consensus.cc:2804] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:01:51.700305  7778 raft_consensus.cc:493] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:51.700471  1750 raft_consensus.cc:2393] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 13: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:01:51.700675  7778 raft_consensus.cc:3060] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:01:51.701488  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:51.702203  1750 raft_consensus.cc:3060] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:01:51.702899  1605 raft_consensus.cc:2393] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 17: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:01:51.703101  1717 leader_election.cc:304] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:51.709400  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:51.705869  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:51.707242  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:51.703786  2332 raft_consensus.cc:1240] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 16. Current term is 17. Ops: []
I20260709 13:01:51.702812  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 18 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:51.715499  2301 leader_election.cc:304] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:51.699898  1748 raft_consensus.cc:2368] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 13. Current term is 14.
I20260709 13:01:51.725582  1597 raft_consensus.cc:2468] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 17.
W20260709 13:01:51.734225  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 236203us
I20260709 13:01:51.698186  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 14 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:51.733814  2298 leader_election.cc:304] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:01:51.703358  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:51.707623  1600 raft_consensus.cc:2468] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 13.
W20260709 13:01:51.750792  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 47876.2us
I20260709 13:01:51.751088  1746 raft_consensus.cc:2410] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for term 18 because replica has last-logged OpId of term: 16 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 7.
W20260709 13:01:51.767671  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 52286.6us
W20260709 13:01:51.738647  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 311183us
W20260709 13:01:51.738253  1569 connection.cc:256] RPC call timeout handler was delayed by -1.2199806290000197s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15200000000000002s for a response
W20260709 13:01:51.773720  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:51.706341  7850 raft_consensus.cc:3060] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Advancing to term 13
W20260709 13:01:51.783638  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 47940.2us
I20260709 13:01:51.784721  1747 raft_consensus.cc:3060] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:01:51.772022  1749 raft_consensus.cc:2468] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 13.
I20260709 13:01:51.758278  1596 raft_consensus.cc:2376] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 14. Re-sending same reply.
W20260709 13:01:51.788504  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 54778.9us
I20260709 13:01:51.783600  1569 leader_election.cc:304] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:01:51.789688  2298 leader_election.cc:304] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:01:51.679821  7808 raft_consensus.cc:2804] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Leader election won for term 13
I20260709 13:01:51.794273  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 18 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:51.795665  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:51.796753  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:51.799862  1571 leader_election.cc:400] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 election: Vote denied by peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693) with higher term. Message: Invalid argument: T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 13. Current term is 14.
I20260709 13:01:51.800319  1571 leader_election.cc:403] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 election: Cancelling election due to peer responding with higher term
I20260709 13:01:51.804096  2330 raft_consensus.cc:1240] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 11. Current term is 12. Ops: []
I20260709 13:01:51.804611  1644 leader_election.cc:304] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:01:51.802896  2298 leader_election.cc:304] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:01:51.807628  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:51.809999  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:51.811022  1601 raft_consensus.cc:2468] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 16.
I20260709 13:01:51.811169  1717 leader_election.cc:304] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:51.811524  1753 raft_consensus.cc:2468] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 16.
I20260709 13:01:51.812033  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:51.812670  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:51.813373  1753 raft_consensus.cc:3060] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:01:51.813843  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:51.814548  2334 raft_consensus.cc:2468] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 13.
I20260709 13:01:51.814826  7816 raft_consensus.cc:3055] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 LEADER]: Stepping down as leader of term 11
I20260709 13:01:51.814996  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:51.815323  7816 raft_consensus.cc:740] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:01:51.814371  7869 raft_consensus.cc:2764] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Leader pre-election decision vote started in defunct term 12: won
I20260709 13:01:51.815840  1599 raft_consensus.cc:2393] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 15: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:01:51.816461  7816 consensus_queue.cc:260] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.10, Last appended by leader: 10, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:51.888857  1571 leader_election.cc:304] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:51.842536  7875 raft_consensus.cc:2804] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:01:51.843048  7808 raft_consensus.cc:697] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:01:51.809602  7830 raft_consensus.cc:2749] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:01:51.695691  1496 catalog_manager.cc:5697] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 13 to 14, leader changed from 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) to 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196). New cstate: current_term: 14 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 13:01:51.834734  7819 leader_election.cc:290] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 15 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:51.845397  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:51.847388  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:51.906803  7647 raft_consensus.cc:2749] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:01:51.858405  1601 raft_consensus.cc:1275] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 16 index: 12. (index mismatch)
W20260709 13:01:51.841552  7879 log.cc:927] Time spent T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.223s	user 0.001s	sys 0.001s
I20260709 13:01:51.860936  2335 raft_consensus.cc:1240] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 17. Current term is 18. Ops: [17.12-17.12]
I20260709 13:01:51.838605  1604 raft_consensus.cc:1240] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 11. Current term is 12. Ops: []
I20260709 13:01:51.835666  1603 raft_consensus.cc:1240] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 17. Current term is 18. Ops: []
W20260709 13:01:51.833441  7880 log.cc:927] Time spent T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.219s	user 0.001s	sys 0.000s
I20260709 13:01:51.859521  1605 raft_consensus.cc:1240] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 17. Current term is 18. Ops: [17.9-17.9]
I20260709 13:01:51.846483  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:51.839931  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 17 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:01:51.862677  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 69906.4us
I20260709 13:01:51.849653  1675 raft_consensus.cc:1275] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
I20260709 13:01:51.842955  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:51.825158  7727 leader_election.cc:290] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 18 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:51.813395  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 17 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:01:51.855294  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 55761.7us
I20260709 13:01:51.850875  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:51.913251  7875 raft_consensus.cc:493] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:01:51.855901  7833 consensus_meta.cc:287] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.826s	user 0.013s	sys 0.005s
I20260709 13:01:51.812692  1717 leader_election.cc:304] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:51.819721  2301 leader_election.cc:304] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:51.817835  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:51.830961  1718 leader_election.cc:304] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 13:01:51.887717  1645 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:51.885411  1646 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:51.857751  2337 raft_consensus.cc:1240] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 17. Current term is 18. Ops: []
I20260709 13:01:51.844684  2332 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd"
dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
mode: GRACEFUL
new_leader_uuid: "af520b67945d443b901bbeca37cc5701"
 from {username='slave'} at 127.0.0.1:35790
W20260709 13:01:51.957665  1646 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:51.892452  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 89633.8us
W20260709 13:01:51.958151  1646 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:51.814760  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:51.819356  1598 raft_consensus.cc:2468] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 13.
W20260709 13:01:51.899137  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 128010us
I20260709 13:01:51.901038  2331 raft_consensus.cc:2468] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 12.
I20260709 13:01:51.899132  7816 raft_consensus.cc:3060] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:01:51.846256  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:51.962312  1644 leader_election.cc:304] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
W20260709 13:01:51.903237  7761 consensus_meta.cc:287] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.566s	user 0.014s	sys 0.003s
I20260709 13:01:51.909451  1750 raft_consensus.cc:2468] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 17.
I20260709 13:01:51.892241  7662 raft_consensus.cc:493] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:51.898670  7560 raft_consensus.cc:2804] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Leader pre-election won for term 14
W20260709 13:01:51.872007  7437 consensus_meta.cc:287] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.722s	user 0.015s	sys 0.002s
I20260709 13:01:51.915297  1752 raft_consensus.cc:2468] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 17.
I20260709 13:01:51.904888  7600 raft_consensus.cc:2804] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:01:51.703799  7745 consensus_queue.cc:237] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 11.11, Last appended by leader: 11, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:51.984189  1750 raft_consensus.cc:1240] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 16. Current term is 17. Ops: []
I20260709 13:01:51.848297  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
W20260709 13:01:51.923032  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 114653us
I20260709 13:01:51.969929  1672 raft_consensus.cc:2468] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 18.
W20260709 13:01:51.988296  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 60936.3us
I20260709 13:01:51.926334  1754 raft_consensus.cc:2468] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 18.
I20260709 13:01:51.846593  7750 raft_consensus.cc:2749] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader election lost for term 13. Reason: Vote denied by peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693) with higher term. Message: Invalid argument: T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 13. Current term is 14.
I20260709 13:01:51.931912  1596 raft_consensus.cc:2468] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 14.
I20260709 13:01:51.935436  1677 raft_consensus.cc:2393] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 12: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
W20260709 13:01:51.960729  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 141129us
I20260709 13:01:51.831951  1748 raft_consensus.cc:2468] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 13.
I20260709 13:01:51.957834  2332 raft_consensus.cc:606] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Received request to transfer leadership to TS af520b67945d443b901bbeca37cc5701
I20260709 13:01:51.992645  2332 raft_consensus.cc:614] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 13:01:51.964356  7577 raft_consensus.cc:2804] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Leader pre-election won for term 14
W20260709 13:01:51.972078  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 159508us
I20260709 13:01:51.966948  1746 raft_consensus.cc:2468] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 17.
I20260709 13:01:51.962826  7835 raft_consensus.cc:2764] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Leader pre-election decision vote started in defunct term 11: won
I20260709 13:01:51.957065  7875 raft_consensus.cc:3060] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:01:51.969818  1749 raft_consensus.cc:2393] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 17: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:01:51.989187  1571 leader_election.cc:304] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:51.988677  1600 raft_consensus.cc:2376] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 11. Re-sending same reply.
W20260709 13:01:52.007396  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:51.970396  7585 raft_consensus.cc:2804] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:01:51.974172  7560 raft_consensus.cc:493] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:52.018170  1598 raft_consensus.cc:1275] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 12 index: 12. Preceding OpId from leader: term: 13 index: 13. (index mismatch)
W20260709 13:01:52.019531  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 188643us
I20260709 13:01:51.996788  7863 raft_consensus.cc:2804] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Leader election won for term 18
I20260709 13:01:51.996532  7307 raft_consensus.cc:2804] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:01:51.970675  7707 consensus_queue.cc:1059] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 14.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.044s
I20260709 13:01:52.033298  1644 leader_election.cc:304] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:01:52.023382  1646 leader_election.cc:304] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:52.007287  7869 raft_consensus.cc:2804] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 13:01:51.973253  7761 raft_consensus.cc:515] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:52.009893  7600 raft_consensus.cc:697] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
W20260709 13:01:52.041141  1646 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:52.017987  7560 raft_consensus.cc:3060] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:01:52.014462  7585 raft_consensus.cc:493] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:52.042085  1750 raft_consensus.cc:1240] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 16. Current term is 17. Ops: []
I20260709 13:01:52.024600  7813 raft_consensus.cc:3055] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 LEADER]: Stepping down as leader of term 16
I20260709 13:01:52.034189  7817 raft_consensus.cc:2804] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Leader election won for term 17
I20260709 13:01:52.052608  1718 leader_election.cc:304] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:51.978475  7833 raft_consensus.cc:515] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:52.055061  1644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:51.996762  7582 consensus_queue.cc:1048] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 7, Time since last communication: 0.036s
I20260709 13:01:52.056689  1717 leader_election.cc:304] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:52.007004  7662 raft_consensus.cc:515] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:52.038908  7307 raft_consensus.cc:493] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:01:52.073634  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:52.068209  1569 leader_election.cc:304] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:01:52.072909  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
W20260709 13:01:52.080170  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:52.081856  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:52.047591  7753 raft_consensus.cc:2804] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:01:52.061337  7585 raft_consensus.cc:3060] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:01:52.069128  7863 raft_consensus.cc:697] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:01:52.036857  7714 consensus_queue.cc:1059] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 16.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.033s
I20260709 13:01:52.029214  7437 consensus_queue.cc:1059] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 16.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.040s
I20260709 13:01:52.041056  7778 raft_consensus.cc:515] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:52.062382  7813 raft_consensus.cc:740] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:01:52.065889  7817 raft_consensus.cc:697] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:01:52.090085  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:52.112952  1571 leader_election.cc:304] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:52.094234  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:52.118300  1675 raft_consensus.cc:2468] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 17.
I20260709 13:01:52.082780  7307 raft_consensus.cc:3060] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:01:52.087544  7707 raft_consensus.cc:3055] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:01:52.085959  7753 raft_consensus.cc:493] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:52.129681  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:52.110867  7686 raft_consensus.cc:2764] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Leader pre-election decision vote started in defunct term 12: won
I20260709 13:01:52.138816  1745 raft_consensus.cc:1240] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 13. Current term is 14. Ops: []
I20260709 13:01:52.135515  2301 leader_election.cc:304] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:52.144932  1673 raft_consensus.cc:1240] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 LEADER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 13. Current term is 14. Ops: []
I20260709 13:01:52.146212  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:52.138543  7718 leader_election.cc:290] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 17 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:52.151711  7600 consensus_queue.cc:237] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:52.152971  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
W20260709 13:01:52.158861  1570 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:52.160835  7778 leader_election.cc:290] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 14 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:52.127858  7707 raft_consensus.cc:740] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
W20260709 13:01:52.173326  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 120901us
I20260709 13:01:52.145129  7728 raft_consensus.cc:2804] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Leader election won for term 18
W20260709 13:01:52.179044  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 110860us
I20260709 13:01:52.174702  1675 raft_consensus.cc:2393] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 17: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:01:52.127458  7859 raft_consensus.cc:2804] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:01:52.090472  1747 raft_consensus.cc:2468] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 15.
I20260709 13:01:52.132202  7753 raft_consensus.cc:3060] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:01:52.189522  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 18 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:52.086508  7808 consensus_queue.cc:237] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.10, Last appended by leader: 10, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:52.192407  1718 leader_election.cc:304] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:52.186229  7863 consensus_queue.cc:237] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 17.14, Last appended by leader: 14, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:52.203730  2335 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2"
dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
mode: GRACEFUL
new_leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
 from {username='slave'} at 127.0.0.1:35790
W20260709 13:01:52.206707  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 111226us
I20260709 13:01:52.206983  1569 leader_election.cc:304] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:01:52.207443  2335 raft_consensus.cc:606] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Received request to transfer leadership to TS 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:52.207850  2335 raft_consensus.cc:614] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 13:01:52.209524  1747 raft_consensus.cc:1275] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 18 index: 13. (index mismatch)
I20260709 13:01:52.189079  7869 raft_consensus.cc:697] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 17 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:01:52.155947  7437 consensus_queue.cc:1059] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 16.11, Next index: 12, Last known committed idx: 10, Time since last communication: 0.045s
I20260709 13:01:52.216399  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:52.189061  7702 raft_consensus.cc:2764] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Leader pre-election decision vote started in defunct term 14: won
I20260709 13:01:52.218055  1753 raft_consensus.cc:2468] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 15.
I20260709 13:01:52.196646  7728 raft_consensus.cc:697] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:01:52.228806  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 17 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:52.229799  1747 raft_consensus.cc:1275] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 20 index: 14. Preceding OpId from leader: term: 21 index: 14. (term mismatch)
I20260709 13:01:52.203157  7577 raft_consensus.cc:493] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:52.207384  7859 raft_consensus.cc:697] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 12 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:01:52.154312  1496 catalog_manager.cc:5697] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 11 to 13. New cstate: current_term: 13 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } }
I20260709 13:01:52.216372  7819 raft_consensus.cc:2749] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:01:52.232004  1747 pending_rounds.cc:85] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701: Aborting all ops after (but not including) 13
W20260709 13:01:52.227165  7901 log.cc:927] Time spent T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.080s	user 0.001s	sys 0.000s
I20260709 13:01:52.138507  7817 consensus_queue.cc:237] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.9, Last appended by leader: 9, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:52.245093  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:52.246079  1753 raft_consensus.cc:1275] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 17 index: 10. Preceding OpId from leader: term: 18 index: 11. (index mismatch)
I20260709 13:01:52.245074  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:52.231815  7754 raft_consensus.cc:3055] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 LEADER]: Stepping down as leader of term 16
I20260709 13:01:52.254235  1677 raft_consensus.cc:3060] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:01:52.240967  1747 pending_rounds.cc:107] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701: Aborting uncommitted NO_OP operation due to leader change: 20.14
I20260709 13:01:52.259711  1747 raft_consensus.cc:2889] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:01:52.238188  7577 raft_consensus.cc:3060] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Advancing to term 14
W20260709 13:01:52.289029  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 244034us
W20260709 13:01:52.303550  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 54507.2us
I20260709 13:01:52.248410  7761 leader_election.cc:290] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 19 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:52.305645  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:52.306916  7902 raft_consensus.cc:2804] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 13:01:52.303395  7727 raft_consensus.cc:493] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:52.317058  7727 raft_consensus.cc:515] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:52.259289  7754 raft_consensus.cc:740] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
W20260709 13:01:52.323429  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 130987us
I20260709 13:01:52.328967  1677 raft_consensus.cc:2468] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 15.
W20260709 13:01:52.328461  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 73782.5us
I20260709 13:01:52.336733  2299 leader_election.cc:304] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 13:01:52.322424  7852 consensus_meta.cc:287] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.659s	user 0.015s	sys 0.000s
I20260709 13:01:52.309926  7662 leader_election.cc:290] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:52.334285  1747 raft_consensus.cc:1275] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
I20260709 13:01:52.321053  7833 leader_election.cc:290] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 14 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:52.349354  1643 leader_election.cc:400] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 16 election: Vote denied by peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445) with higher term. Message: Invalid argument: T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for earlier term 16. Current term is 17.
I20260709 13:01:52.349123  1718 leader_election.cc:304] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:52.349977  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:52.345989  1717 leader_election.cc:304] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:52.351119  1753 raft_consensus.cc:1275] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 16 index: 11. (index mismatch)
I20260709 13:01:52.333611  7913 raft_consensus.cc:493] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:52.386993  1598 raft_consensus.cc:1275] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 18 index: 13. (index mismatch)
I20260709 13:01:52.395210  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 18 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:52.395982  1678 raft_consensus.cc:3060] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:01:52.370999  7912 raft_consensus.cc:2749] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:01:52.366597  7902 raft_consensus.cc:697] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 17 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
W20260709 13:01:52.399037  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 62442.4us
I20260709 13:01:52.405138  1603 raft_consensus.cc:1275] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
I20260709 13:01:52.399223  7718 consensus_queue.cc:1048] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.009s
I20260709 13:01:52.374162  7728 consensus_queue.cc:237] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.9, Last appended by leader: 9, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:52.333334  7869 consensus_queue.cc:237] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.9, Last appended by leader: 9, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:52.371995  7859 consensus_queue.cc:237] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:52.319209  7707 consensus_queue.cc:260] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 14.13, Last appended by leader: 13, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:52.406052  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:52.417948  1601 raft_consensus.cc:1275] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 19 index: 13. Preceding OpId from leader: term: 21 index: 14. (index mismatch)
I20260709 13:01:52.438345  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:52.439047  7915 raft_consensus.cc:2749] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:01:52.440621  7916 raft_consensus.cc:2804] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:01:52.441133  7916 raft_consensus.cc:493] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:52.441502  7916 raft_consensus.cc:3060] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:01:52.442587  7908 consensus_queue.cc:1048] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.039s
I20260709 13:01:52.444585  7833 consensus_queue.cc:1059] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 16.13, Next index: 14, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 13:01:52.445847  7906 raft_consensus.cc:2804] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:01:52.450484  7910 raft_consensus.cc:2749] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
W20260709 13:01:52.450758  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 123660us
I20260709 13:01:52.450979  7902 consensus_queue.cc:237] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 16.13, Last appended by leader: 13, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:52.453239  1643 leader_election.cc:304] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:01:52.453621  1672 raft_consensus.cc:2468] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 18.
I20260709 13:01:52.451300  7307 raft_consensus.cc:515] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:52.454881  1675 raft_consensus.cc:1275] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 17 index: 10. Preceding OpId from leader: term: 18 index: 11. (index mismatch)
I20260709 13:01:52.456254  7307 leader_election.cc:290] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 14 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:01:52.454829  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 134956us
W20260709 13:01:52.460153  1644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:52.427510  7895 log.cc:927] Time spent T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.229s	user 0.000s	sys 0.001s
I20260709 13:01:52.465844  1678 raft_consensus.cc:2468] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 18.
I20260709 13:01:52.479555  7908 raft_consensus.cc:493] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:52.481747  7908 raft_consensus.cc:515] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:52.483644  7307 raft_consensus.cc:493] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:52.484216  7307 raft_consensus.cc:515] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:01:52.486354  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:52.434294  7727 leader_election.cc:290] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:52.492564  1678 raft_consensus.cc:1275] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 16 index: 11. (index mismatch)
I20260709 13:01:52.398692  7913 raft_consensus.cc:515] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:52.480022  7852 raft_consensus.cc:2749] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 13:01:52.399667  7754 consensus_queue.cc:260] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.11, Last appended by leader: 11, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:52.527552  7813 consensus_queue.cc:260] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 16.13, Last appended by leader: 13, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:52.530576  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 77535.8us
I20260709 13:01:52.531697  1643 leader_election.cc:304] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 13:01:52.532095  1719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:52.532588  7437 raft_consensus.cc:2749] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:01:52.533305  1643 leader_election.cc:304] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:01:52.534215  7437 raft_consensus.cc:2749] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20260709 13:01:52.534590  1646 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:52.535259  2336 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39726 (request call id 1421) took 1050 ms. Trace:
I20260709 13:01:52.535674  2336 rpcz_store.cc:276] 0709 13:01:51.484565 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:51.484721 (+   156us) service_pool.cc:224] Handling call
0709 13:01:51.496585 (+ 11864us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:51.507738 (+ 11153us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:51.507771 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:01:51.551997 (+ 44226us) log.cc:844] Serialized 26 byte log entry
0709 13:01:51.732652 (+180655us) spinlock_profiling.cc:243] Waited 13.4 ms on lock 0x7b18003b5900. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f4263057e07 00007f426beb2ab9 00007f426307469b 00007f426307293f 00007f4263074c02 00007f4265cfc20b 00007f4265d01d23 00007f4265d020d1 00007f426611da0f
0709 13:01:52.008549 (+275897us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:01:52.008604 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:52.008710 (+   106us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:52.534842 (+526132us) raft_consensus.cc:1685] finished
0709 13:01:52.534873 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:52.535158 (+   285us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5367,"raft.queue_time_us":12957,"raft.run_cpu_time_us":134,"raft.run_wall_time_us":3722,"spinlock_wait_cycles":30857984,"thread_start_us":275637,"threads_started":1,"wal-append.queue_time_us":572295}
I20260709 13:01:52.479758  7560 raft_consensus.cc:515] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:52.544001  1753 raft_consensus.cc:1275] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
I20260709 13:01:52.544723  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:52.400753  1496 catalog_manager.cc:5697] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 6 to 11, leader changed from 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194) to 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193). New cstate: current_term: 11 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: HEALTHY } } }
I20260709 13:01:52.574635  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:52.582751  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:01:52.582916  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 83911.8us
I20260709 13:01:52.583834  1753 raft_consensus.cc:2468] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 18.
I20260709 13:01:52.585568  1753 raft_consensus.cc:1275] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 11 index: 8. Preceding OpId from leader: term: 13 index: 9. (index mismatch)
I20260709 13:01:52.588974  1745 raft_consensus.cc:1240] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 13. Current term is 14. Ops: []
I20260709 13:01:52.540494  7307 leader_election.cc:290] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:52.589058  1643 leader_election.cc:304] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 13:01:52.591666  1718 leader_election.cc:336] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 15 pre-election: RPC error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Timed out: RequestConsensusVote RPC to 127.0.107.194:46261 timed out after 1.535s (SENT)
I20260709 13:01:52.592219  1718 leader_election.cc:304] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 13:01:52.593138  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 57538.5us
I20260709 13:01:52.595316  1646 leader_election.cc:304] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
W20260709 13:01:52.602010  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 96272.8us
I20260709 13:01:52.607422  7911 raft_consensus.cc:2749] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
W20260709 13:01:52.600790  7816 consensus_meta.cc:287] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.633s	user 0.009s	sys 0.009s
I20260709 13:01:52.610097  1643 leader_election.cc:304] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:52.610904  7600 raft_consensus.cc:2749] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:01:52.611104  7908 leader_election.cc:290] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:52.611850  7869 raft_consensus.cc:2749] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:01:52.613018  7714 raft_consensus.cc:2804] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Leader election won for term 15
I20260709 13:01:52.613582  7714 raft_consensus.cc:697] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
W20260709 13:01:52.614784  7875 consensus_meta.cc:287] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.591s	user 0.009s	sys 0.006s
I20260709 13:01:52.614557  7714 consensus_queue.cc:237] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 13.9, Last appended by leader: 9, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:52.615422  7875 raft_consensus.cc:515] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:52.614926  1496 catalog_manager.cc:5697] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 8 to 10. New cstate: current_term: 10 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } }
I20260709 13:01:52.619112  1643 leader_election.cc:304] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
W20260709 13:01:52.622771  1646 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:52.630383  2299 leader_election.cc:304] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:01:52.642915  1753 raft_consensus.cc:1240] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 16. Current term is 17. Ops: []
W20260709 13:01:52.648896  1645 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:52.623790  7925 log.cc:927] Time spent T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.098s	user 0.000s	sys 0.001s
I20260709 13:01:52.621896  7852 consensus_queue.cc:1048] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.047s
I20260709 13:01:52.619267  7560 leader_election.cc:290] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 14 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:01:52.659797  7585 consensus_meta.cc:287] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.550s	user 0.014s	sys 0.003s
W20260709 13:01:52.679930  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 88289.3us
I20260709 13:01:52.618036  7875 leader_election.cc:290] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 19 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:52.619307  7582 raft_consensus.cc:2749] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:01:52.704914  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:52.669761  7845 raft_consensus.cc:493] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:52.706051  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
W20260709 13:01:52.728628  1645 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:52.706290  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 43538.6us
I20260709 13:01:52.733438  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:01:52.733731  1718 leader_election.cc:336] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 18 election: RPC error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Timed out: RequestConsensusVote RPC to 127.0.107.194:46261 timed out after 1.670s (SENT)
I20260709 13:01:52.734285  1718 leader_election.cc:304] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:52.663429  7908 consensus_queue.cc:1048] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.044s
I20260709 13:01:52.735028  1717 leader_election.cc:304] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:52.739547  7905 raft_consensus.cc:2749] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:01:52.737231  7845 raft_consensus.cc:515] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:52.742412  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:52.743081  1678 raft_consensus.cc:3060] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:01:52.744531  1671 raft_consensus.cc:2468] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 12.
I20260709 13:01:52.747598  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:52.748385  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:52.667265  7834 consensus_queue.cc:1048] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.064s
I20260709 13:01:52.749768  2299 leader_election.cc:304] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 13:01:52.752781  7753 consensus_meta.cc:287] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.534s	user 0.009s	sys 0.003s
I20260709 13:01:52.753270  7753 raft_consensus.cc:515] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:52.753952  1602 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40154 (request call id 2665) took 1021 ms. Trace:
I20260709 13:01:52.754837  1680 raft_consensus.cc:2368] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 17. Current term is 18.
I20260709 13:01:52.755465  7702 raft_consensus.cc:2749] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 13:01:52.756654  1571 leader_election.cc:304] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:52.758946  1569 leader_election.cc:304] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
W20260709 13:01:52.759256  7850 consensus_meta.cc:287] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.958s	user 0.015s	sys 0.003s
I20260709 13:01:52.760252  2333 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:34804 (request call id 0) took 1122 ms. Trace:
I20260709 13:01:52.760674  2333 rpcz_store.cc:276] 0709 13:01:51.637823 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:51.660565 (+ 22742us) service_pool.cc:224] Handling call
0709 13:01:52.759926 (+1099361us) spinlock_profiling.cc:243] Waited 816 ms on lock 0x7b540073afd4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266168735 00007f42661609fa 00007f426b1284c9 00007f42646a4c54 00007f42646a4be2 00007f42646a4b11 00007f42646a4a8c 00007f42646a3d42 00007f426449d48c 00007f426449c8c6
0709 13:01:52.760153 (+   227us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1877698432}
I20260709 13:01:52.761113  1569 leader_election.cc:304] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:01:52.761278  7686 raft_consensus.cc:493] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:52.762699  7816 raft_consensus.cc:2749] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 13:01:52.701491  7919 consensus_queue.cc:1048] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.083s
I20260709 13:01:52.749009  1671 raft_consensus.cc:2468] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 13.
I20260709 13:01:52.617023  1496 catalog_manager.cc:5697] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 9 to 10, leader changed from 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) to 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194). New cstate: current_term: 10 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } }
I20260709 13:01:52.794726  7817 raft_consensus.cc:2764] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Leader pre-election decision vote started in defunct term 16: won
W20260709 13:01:52.795933  7928 log.cc:927] Time spent T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.117s	user 0.000s	sys 0.000s
I20260709 13:01:52.795012  1496 catalog_manager.cc:5697] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 9 to 12, leader changed from 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) to 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193). New cstate: current_term: 12 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 13:01:52.797052  7728 raft_consensus.cc:2804] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader election won for term 15
I20260709 13:01:52.797752  7728 raft_consensus.cc:697] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:01:52.797240  1496 catalog_manager.cc:5697] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 12 to 13, leader changed from 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) to af520b67945d443b901bbeca37cc5701 (127.0.107.195). New cstate: current_term: 13 leader_uuid: "af520b67945d443b901bbeca37cc5701" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
W20260709 13:01:52.800756  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 67062.5us
W20260709 13:01:52.794921  1570 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:52.695887  7916 raft_consensus.cc:515] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:52.827880  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 208927us
I20260709 13:01:52.830653  7916 leader_election.cc:290] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 14 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:52.831847  1754 raft_consensus.cc:2393] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 14: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
W20260709 13:01:52.831858  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 96948.6us
I20260709 13:01:52.820642  1676 raft_consensus.cc:1275] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
I20260709 13:01:52.798830  1571 leader_election.cc:304] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:52.833179  1717 leader_election.cc:304] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 13:01:52.730854  7936 raft_consensus.cc:2749] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:01:52.821353  1672 raft_consensus.cc:3060] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:01:52.696905  7863 consensus_queue.cc:1059] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 16.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.048s
I20260709 13:01:52.835017  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:52.835937  1753 raft_consensus.cc:2468] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 16.
I20260709 13:01:52.822117  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:52.838773  1674 raft_consensus.cc:3060] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:01:52.681241  7912 consensus_queue.cc:1048] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.062s
I20260709 13:01:52.825423  7817 consensus_queue.cc:1059] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 13.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 13:01:52.826392  1746 raft_consensus.cc:1240] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 17. Current term is 18. Ops: []
I20260709 13:01:52.798836  7728 consensus_queue.cc:237] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.8, Last appended by leader: 8, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:52.799819  7687 consensus_queue.cc:1048] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 13:01:52.827049  7727 raft_consensus.cc:493] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:52.800595  7745 consensus_queue.cc:1048] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 13:01:52.761940  7686 raft_consensus.cc:515] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:52.731148  7913 leader_election.cc:290] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:52.839059  1673 raft_consensus.cc:1275] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 12 index: 12. Preceding OpId from leader: term: 13 index: 13. (index mismatch)
I20260709 13:01:52.703529  7585 raft_consensus.cc:515] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:52.847255  1680 raft_consensus.cc:1275] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
I20260709 13:01:52.853121  1678 raft_consensus.cc:2468] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 14.
I20260709 13:01:52.861980  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:52.860190  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:52.928797  1600 raft_consensus.cc:3060] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:01:52.859051  2299 leader_election.cc:304] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
W20260709 13:01:52.884509  1645 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:52.894083  1752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55848 (request call id 2982) took 1205 ms. Trace:
W20260709 13:01:52.934866  1645 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:52.931737  1717 leader_election.cc:304] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:52.831734  7647 consensus_queue.cc:1048] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.064s
I20260709 13:01:52.934751  1752 rpcz_store.cc:276] 0709 13:01:51.688305 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:52.040967 (+352662us) service_pool.cc:224] Handling call
0709 13:01:52.066183 (+ 25216us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:52.243607 (+177424us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 13:01:52.243658 (+    51us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:01:52.302870 (+ 59212us) log.cc:844] Serialized 26 byte log entry
0709 13:01:52.407458 (+104588us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:01:52.419347 (+ 11889us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:52.419421 (+    74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:52.893730 (+474309us) raft_consensus.cc:1685] finished
0709 13:01:52.893765 (+    35us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:52.893984 (+   219us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":20877,"raft.queue_time_us":226874,"raft.run_cpu_time_us":205,"raft.run_wall_time_us":15072,"thread_start_us":219783,"threads_started":2,"wal-append.queue_time_us":164980}
I20260709 13:01:52.936702  7830 raft_consensus.cc:2804] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:01:52.937132  7830 raft_consensus.cc:493] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:01:52.934760  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 75824.2us
I20260709 13:01:52.938251  7647 raft_consensus.cc:493] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:01:52.938618  7929 log.cc:927] Time spent T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.197s	user 0.001s	sys 0.000s
I20260709 13:01:52.938854  7647 raft_consensus.cc:515] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:52.939109  2299 leader_election.cc:304] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:01:52.940609  1751 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43156 (request call id 1259) took 1247 ms. Trace:
I20260709 13:01:52.940518  7908 consensus_queue.cc:1048] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 13:01:52.941550  7560 raft_consensus.cc:2804] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:01:52.942075  7560 raft_consensus.cc:697] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 14 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
W20260709 13:01:52.894826  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 132398us
I20260709 13:01:52.943684  1569 leader_election.cc:304] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:52.944459  7913 raft_consensus.cc:2804] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:01:52.944904  7913 raft_consensus.cc:493] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:52.945320  7913 raft_consensus.cc:3060] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:01:52.945801  7687 raft_consensus.cc:2749] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:01:52.948046  7647 leader_election.cc:290] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:52.856899  7750 raft_consensus.cc:2804] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:01:52.754377  1602 rpcz_store.cc:276] 0709 13:01:51.735313 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:51.862662 (+127349us) service_pool.cc:224] Handling call
0709 13:01:51.892858 (+ 30196us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:01:51.920936 (+ 28078us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:51.920976 (+    40us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:01:52.011737 (+ 90761us) log.cc:844] Serialized 52 byte log entry
0709 13:01:52.232598 (+220861us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:01:52.289742 (+ 57144us) log.cc:844] Serialized 14 byte log entry
0709 13:01:52.753164 (+463422us) spinlock_profiling.cc:243] Waited 2.22 ms on lock 0x7b18003b4a00. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f4262fefc0a 00007f426c72f219 00007f4263077a40 00007f42630724e4 00007f4263072455 00007f426615a060 00007f426616c716 00007f426617d287 00007f426617d208
0709 13:01:52.753435 (+   271us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:52.753523 (+    88us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:52.753574 (+    51us) raft_consensus.cc:1685] finished
0709 13:01:52.753691 (+   117us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:52.753849 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":16307,"prepare.queue_time_us":68578,"prepare.run_cpu_time_us":56,"prepare.run_wall_time_us":58,"raft.queue_time_us":467673,"raft.run_cpu_time_us":276,"raft.run_wall_time_us":278,"spinlock_wait_cycles":5107968,"thread_start_us":188430,"threads_started":1,"wal-append.queue_time_us":293914}
I20260709 13:01:52.895792  7816 consensus_queue.cc:1059] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 13.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.008s
I20260709 13:01:52.949429  7750 raft_consensus.cc:493] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:52.863545  7869 raft_consensus.cc:2804] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:01:52.949944  7750 raft_consensus.cc:3060] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:01:52.950110  7647 raft_consensus.cc:3055] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 LEADER]: Stepping down as leader of term 13
I20260709 13:01:52.950212  7869 raft_consensus.cc:493] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:52.912865  7307 consensus_queue.cc:1059] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 15.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.038s
I20260709 13:01:52.950760  7647 raft_consensus.cc:740] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
W20260709 13:01:52.912966  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 83123.2us
I20260709 13:01:52.951277  7307 raft_consensus.cc:3055] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 13:01:52.951705  7307 raft_consensus.cc:740] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:01:52.951843  7647 consensus_queue.cc:260] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 13.12, Last appended by leader: 12, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:52.952713  7307 consensus_queue.cc:260] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 17.9, Last appended by leader: 9, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:52.953490  7647 raft_consensus.cc:3060] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:01:52.953784  1643 leader_election.cc:304] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:01:52.954710  7936 raft_consensus.cc:2764] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Leader pre-election decision vote started in defunct term 17: won
I20260709 13:01:52.955109  7585 leader_election.cc:290] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 17 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:01:52.937760  7927 log.cc:927] Time spent T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.199s	user 0.001s	sys 0.000s
I20260709 13:01:52.957648  7926 raft_consensus.cc:2804] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:01:52.958114  7926 raft_consensus.cc:493] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:52.897289  1596 raft_consensus.cc:1275] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
W20260709 13:01:52.913177  1718 leader_election.cc:336] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Timed out: RequestConsensusVote RPC to 127.0.107.194:46261 timed out after 1.717s (SENT)
I20260709 13:01:52.954216  7307 raft_consensus.cc:3060] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:01:52.831362  7916 raft_consensus.cc:2804] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Leader election won for term 14
W20260709 13:01:52.919996  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 85013us
I20260709 13:01:52.917953  7818 consensus_queue.cc:1059] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 13.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.008s
I20260709 13:01:52.919673  7702 consensus_queue.cc:1059] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 11.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.006s
I20260709 13:01:52.923570  7859 raft_consensus.cc:3055] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 LEADER]: Stepping down as leader of term 17
I20260709 13:01:52.937551  7830 raft_consensus.cc:3060] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:01:52.950863  7869 raft_consensus.cc:3060] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:01:53.008481  7845 leader_election.cc:290] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:52.984685  7727 raft_consensus.cc:515] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:53.010840  7702 raft_consensus.cc:3055] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 13:01:53.011319  7702 raft_consensus.cc:740] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:01:53.011030  7560 consensus_queue.cc:237] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:53.012318  7702 consensus_queue.cc:260] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 11.8, Last appended by leader: 8, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:53.013638  7702 raft_consensus.cc:3060] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:01:52.958498  7926 raft_consensus.cc:3060] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Advancing to term 13
W20260709 13:01:53.015187  7941 log.cc:927] Time spent T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.084s	user 0.000s	sys 0.000s
I20260709 13:01:53.001863  7859 raft_consensus.cc:740] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:01:53.017196  7859 consensus_queue.cc:260] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.12, Last appended by leader: 12, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:53.018669  7859 raft_consensus.cc:3060] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:01:52.827090  1496 catalog_manager.cc:5697] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 9 to 11. New cstate: current_term: 11 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 13:01:53.020404  7916 raft_consensus.cc:697] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 14 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
W20260709 13:01:53.030177  7948 log.cc:927] Time spent T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.087s	user 0.002s	sys 0.003s
W20260709 13:01:52.912426  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 70505.3us
I20260709 13:01:52.911803  1672 raft_consensus.cc:2468] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 17.
I20260709 13:01:52.959064  1569 leader_election.cc:304] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
W20260709 13:01:53.040170  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 81214.5us
I20260709 13:01:53.040727  7817 raft_consensus.cc:3060] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:01:53.041112  1569 leader_election.cc:304] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
W20260709 13:01:52.940670  7944 log.cc:927] Time spent T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.099s	user 0.000s	sys 0.000s
W20260709 13:01:53.043969  7942 log.cc:927] Time spent T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.101s	user 0.000s	sys 0.001s
I20260709 13:01:52.941090  1751 rpcz_store.cc:276] 0709 13:01:51.692999 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:52.154175 (+461176us) service_pool.cc:224] Handling call
0709 13:01:52.154627 (+   452us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:52.345404 (+190777us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:52.345442 (+    38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:01:52.395511 (+ 50069us) log.cc:844] Serialized 26 byte log entry
0709 13:01:52.601429 (+205918us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:01:52.601474 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:52.602260 (+   786us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:52.940334 (+338074us) raft_consensus.cc:1685] finished
0709 13:01:52.940360 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:52.940519 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":7622,"raft.queue_time_us":132483,"raft.run_cpu_time_us":192,"raft.run_wall_time_us":24134,"thread_start_us":290949,"threads_started":2,"wal-append.queue_time_us":237891}
I20260709 13:01:53.045513  7816 raft_consensus.cc:2749] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:01:53.046131  7819 raft_consensus.cc:3055] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 LEADER]: Stepping down as leader of term 13
I20260709 13:01:53.046582  7819 raft_consensus.cc:740] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:01:53.047549  7819 consensus_queue.cc:260] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 13.12, Last appended by leader: 12, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:53.048878  7819 raft_consensus.cc:3060] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:01:53.058874  7753 leader_election.cc:290] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 18 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:01:53.060566  7935 log.cc:927] Time spent T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.063s	user 0.000s	sys 0.000s
I20260709 13:01:53.059280  1569 leader_election.cc:304] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
W20260709 13:01:53.061170  7934 log.cc:927] Time spent T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.055s	user 0.001s	sys 0.000s
I20260709 13:01:53.062187  7816 raft_consensus.cc:2804] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Leader election won for term 18
I20260709 13:01:53.062765  7816 raft_consensus.cc:697] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:01:53.063685  7816 consensus_queue.cc:237] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.8, Last appended by leader: 8, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:01:52.998077  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 84922.3us
I20260709 13:01:53.000401  7818 raft_consensus.cc:3055] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 16 LEADER]: Stepping down as leader of term 16
I20260709 13:01:52.996744  1717 leader_election.cc:304] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:52.997895  1646 leader_election.cc:304] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:53.065850  1748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43156 (request call id 1265) took 1285 ms. Trace:
W20260709 13:01:53.069541  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 71751.1us
I20260709 13:01:53.069839  1600 raft_consensus.cc:2468] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 14.
I20260709 13:01:53.073372  2298 leader_election.cc:304] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
W20260709 13:01:53.074173  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 135150us
W20260709 13:01:53.075052  1719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:53.004885  7577 consensus_meta.cc:287] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.691s	user 0.007s	sys 0.007s
I20260709 13:01:53.079600  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:53.080324  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:53.076416  7745 consensus_queue.cc:1048] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.005s
I20260709 13:01:53.068863  7818 raft_consensus.cc:740] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:01:53.070302  1748 rpcz_store.cc:276] 0709 13:01:51.780755 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:52.234160 (+453405us) service_pool.cc:224] Handling call
0709 13:01:52.245158 (+ 10998us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:52.505762 (+260604us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:52.505796 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:01:52.568762 (+ 62966us) log.cc:844] Serialized 26 byte log entry
0709 13:01:52.670845 (+102083us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:01:52.670887 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:52.670971 (+    84us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:53.065607 (+394636us) raft_consensus.cc:1685] finished
0709 13:01:53.065629 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:53.065779 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":103492,"raft.queue_time_us":118003,"raft.run_cpu_time_us":185,"raft.run_wall_time_us":11564,"thread_start_us":261819,"threads_started":2,"wal-append.queue_time_us":304393}
W20260709 13:01:53.082732  1646 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:53.086424  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 89796.1us
I20260709 13:01:53.087878  7946 raft_consensus.cc:2804] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Leader pre-election won for term 17
W20260709 13:01:53.094647  7940 log.cc:927] Time spent T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.077s	user 0.000s	sys 0.000s
I20260709 13:01:53.078764  2299 leader_election.cc:304] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:01:53.079599  7902 raft_consensus.cc:2804] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:01:53.072293  7919 consensus_queue.cc:1048] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.015s
I20260709 13:01:53.099635  7753 consensus_queue.cc:1059] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 13:01:53.110158  1753 raft_consensus.cc:3060] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:01:53.120759  2336 raft_consensus.cc:1240] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 17 LEADER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 16. Current term is 17. Ops: []
I20260709 13:01:53.157315  2330 raft_consensus.cc:1275] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 17 index: 14. Preceding OpId from leader: term: 18 index: 15. (index mismatch)
I20260709 13:01:53.131852  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:53.157835  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:53.158516  1747 raft_consensus.cc:3060] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Advancing to term 19
W20260709 13:01:53.132771  1718 leader_election.cc:336] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 18 pre-election: RPC error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Timed out: RequestConsensusVote RPC to 127.0.107.194:46261 timed out after 1.726s (SENT)
I20260709 13:01:53.139003  7585 consensus_queue.cc:1048] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 13:01:53.141728  7875 consensus_queue.cc:1048] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 13:01:53.146216  1717 leader_election.cc:304] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 13:01:53.134374  7946 raft_consensus.cc:493] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:53.147989  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:01:53.155376  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 76721.6us
W20260709 13:01:53.168026  2299 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.107.194:46261 timed out after 1.657s (SENT) [suppressed 35 similar messages]
I20260709 13:01:53.159143  1718 leader_election.cc:304] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:53.160662  2329 raft_consensus.cc:1240] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 16. Current term is 17. Ops: []
I20260709 13:01:53.165582  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:53.160065  7912 raft_consensus.cc:2804] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 13:01:53.172108  1754 raft_consensus.cc:3060] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:01:53.176200  1674 raft_consensus.cc:2468] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 19.
I20260709 13:01:53.202979  1753 raft_consensus.cc:2468] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 18.
I20260709 13:01:53.203897  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:53.119961  7577 raft_consensus.cc:515] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:53.133445  7916 consensus_queue.cc:237] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 7, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:53.281489  1600 raft_consensus.cc:3060] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:01:53.180135  7946 raft_consensus.cc:3060] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Advancing to term 17
W20260709 13:01:53.281286  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 135230us
I20260709 13:01:53.180771  7945 raft_consensus.cc:493] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:53.191330  7902 raft_consensus.cc:697] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 14 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:01:53.288201  1754 raft_consensus.cc:2468] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 14.
I20260709 13:01:53.288269  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
W20260709 13:01:53.301411  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 183084us
I20260709 13:01:53.243069  7930 raft_consensus.cc:2804] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:01:53.243353  7911 raft_consensus.cc:2804] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 13:01:53.261317  7863 raft_consensus.cc:3055] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 LEADER]: Stepping down as leader of term 17
W20260709 13:01:53.227198  1646 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:53.212852  1747 raft_consensus.cc:2468] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 19.
I20260709 13:01:53.272640  7727 leader_election.cc:290] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:53.308063  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 15 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:53.301941  7945 raft_consensus.cc:515] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:53.282512  7686 leader_election.cc:290] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:53.209262  7582 raft_consensus.cc:493] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:53.256245  7915 raft_consensus.cc:2749] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:01:53.220315  7754 raft_consensus.cc:3060] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:01:53.323817  1598 raft_consensus.cc:2410] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for term 15 because replica has last-logged OpId of term: 14 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 7.
W20260709 13:01:53.326843  1571 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:53.209522  7912 raft_consensus.cc:697] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 17 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:01:53.273871  7750 raft_consensus.cc:515] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:53.331601  1718 leader_election.cc:336] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 17 election: RPC error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Timed out: RequestConsensusVote RPC to 127.0.107.194:46261 timed out after 1.778s (SENT)
I20260709 13:01:53.307996  7930 raft_consensus.cc:493] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:01:53.320977  7923 log.cc:927] Time spent T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.628s	user 0.000s	sys 0.004s
I20260709 13:01:53.315001  7863 raft_consensus.cc:740] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:01:53.219743  7908 consensus_queue.cc:1048] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.121s
I20260709 13:01:53.285040  7818 consensus_queue.cc:260] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 16.8, Last appended by leader: 8, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:01:53.370004  1645 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:53.359465  1600 raft_consensus.cc:2468] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 13.
I20260709 13:01:53.363811  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:01:53.366714  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 198663us
I20260709 13:01:53.371579  1747 raft_consensus.cc:1275] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
I20260709 13:01:53.371667  1605 raft_consensus.cc:2468] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 14.
I20260709 13:01:53.372828  2299 leader_election.cc:304] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:01:53.379184  1602 raft_consensus.cc:1240] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 18. Current term is 19. Ops: [18.13-18.13]
I20260709 13:01:53.380797  1753 raft_consensus.cc:1275] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 13 index: 9. Preceding OpId from leader: term: 15 index: 10. (index mismatch)
W20260709 13:01:53.380676  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 49098.5us
I20260709 13:01:53.257077  1496 catalog_manager.cc:5697] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 8 to 12. New cstate: current_term: 12 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20260709 13:01:53.387997  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 93769.5us
W20260709 13:01:53.384356  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 313715us
I20260709 13:01:53.372823  7915 raft_consensus.cc:493] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:53.366954  7930 raft_consensus.cc:3060] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Advancing to term 15
W20260709 13:01:53.392052  1718 leader_election.cc:336] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 17 pre-election: RPC error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Timed out: RequestConsensusVote RPC to 127.0.107.194:46261 timed out after 1.795s (SENT)
I20260709 13:01:53.395166  1718 leader_election.cc:304] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:53.396056  1569 leader_election.cc:400] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 17 election: Vote denied by peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261) with higher term. Message: Invalid argument: T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 17. Current term is 18.
I20260709 13:01:53.363000  7913 raft_consensus.cc:515] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:53.422793  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 22 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:53.416965  1569 leader_election.cc:403] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 17 election: Cancelling election due to peer responding with higher term
I20260709 13:01:53.387391  7875 raft_consensus.cc:2804] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Leader election won for term 19
W20260709 13:01:53.420696  1717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:53.359848  7582 raft_consensus.cc:515] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:53.365970  7869 raft_consensus.cc:515] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:53.365734  7926 raft_consensus.cc:515] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:53.443089  1604 raft_consensus.cc:2468] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 14.
I20260709 13:01:53.450337  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:53.453378  7945 leader_election.cc:290] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:53.454118  7910 raft_consensus.cc:3055] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 13:01:53.465790  7910 raft_consensus.cc:740] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:01:53.466922  7910 consensus_queue.cc:260] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 16.13, Last appended by leader: 13, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:53.467985  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:53.469111  7813 raft_consensus.cc:3060] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:01:53.468019  7910 raft_consensus.cc:3060] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:01:53.457875  7577 leader_election.cc:290] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 14 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:01:53.459195  1488 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet 983621a6fc5b4c9ea0159a648dfe7a36 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER failed with leader 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 3b879799899a499c8d409ee9f1277cd1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 16 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 13:01:53.483474  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:53.484328  1748 raft_consensus.cc:2393] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 19: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
W20260709 13:01:53.490022  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 346692us
W20260709 13:01:53.495039  7943 log.cc:927] Time spent T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.601s	user 0.000s	sys 0.002s
I20260709 13:01:53.497529  1745 raft_consensus.cc:1275] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 17 index: 10. (index mismatch)
W20260709 13:01:53.499352  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 111507us
W20260709 13:01:53.500032  1679 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57092 (request call id 1374) took 1654 ms (client timeout 1657 ms). Trace:
W20260709 13:01:53.500268  1718 leader_election.cc:336] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 18 pre-election: RPC error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Timed out: RequestConsensusVote RPC to 127.0.107.194:46261 timed out after 1.845s (SENT)
W20260709 13:01:53.500471  1679 rpcz_store.cc:269] 0709 13:01:51.845317 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:51.847040 (+  1723us) service_pool.cc:224] Handling call
0709 13:01:53.499803 (+1652763us) spinlock_profiling.cc:243] Waited 680 ms on lock 0x7b540025add4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:01:53.499969 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1564638464}
I20260709 13:01:53.500914  1718 leader_election.cc:304] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:53.502816  1677 raft_consensus.cc:1240] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 LEADER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 11. Current term is 12. Ops: [11.10-11.10]
I20260709 13:01:53.504117  2335 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a"
dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
mode: GRACEFUL
new_leader_uuid: "af520b67945d443b901bbeca37cc5701"
 from {username='slave'} at 127.0.0.1:35790
I20260709 13:01:53.504537  1677 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52766 (request call id 2956) took 1116 ms. Trace:
I20260709 13:01:53.504735  2335 raft_consensus.cc:606] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Received request to transfer leadership to TS af520b67945d443b901bbeca37cc5701
I20260709 13:01:53.505220  2335 raft_consensus.cc:614] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 13:01:53.504930  1677 rpcz_store.cc:276] 0709 13:01:52.388602 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:52.388742 (+   140us) service_pool.cc:224] Handling call
0709 13:01:53.502515 (+1113773us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:53.503219 (+   704us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:53.504327 (+  1108us) spinlock_profiling.cc:243] Waited 1.11 s on lock 0x7b540025add0. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661618ad 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:01:53.504473 (+   146us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2560097152}
W20260709 13:01:53.507381  7937 log.cc:927] Time spent T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.650s	user 0.002s	sys 0.000s
W20260709 13:01:53.510505  7307 consensus_meta.cc:287] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.513s	user 0.013s	sys 0.000s
W20260709 13:01:53.461804  7958 log.cc:927] Time spent T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.133s	user 0.003s	sys 0.000s
I20260709 13:01:53.464170  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
W20260709 13:01:53.464466  1570 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:53.512046  1598 raft_consensus.cc:1240] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 12. Current term is 13. Ops: []
I20260709 13:01:53.523185  2299 leader_election.cc:304] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:01:53.513664  1753 raft_consensus.cc:1240] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 13. Current term is 14. Ops: []
I20260709 13:01:53.397117  7915 raft_consensus.cc:515] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:53.529820  1604 raft_consensus.cc:2468] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 14.
W20260709 13:01:53.532763  1644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:53.533581  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 137665us
I20260709 13:01:53.531236  1646 leader_election.cc:304] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
W20260709 13:01:53.538195  1645 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:53.539628  1745 raft_consensus.cc:1275] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 16 index: 13. Preceding OpId from leader: term: 17 index: 14. (index mismatch)
I20260709 13:01:53.541260  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:53.541884  1604 raft_consensus.cc:3060] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:01:53.542057  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:53.536388  1749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55848 (request call id 2981) took 1848 ms. Trace:
I20260709 13:01:53.546793   431 auto_leader_rebalancer.cc:452] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260709 13:01:53.547643  7582 leader_election.cc:290] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:53.447171  7875 raft_consensus.cc:697] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 19 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:01:53.529207  7808 raft_consensus.cc:3060] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:01:53.559139  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:53.566847  1754 raft_consensus.cc:2468] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 21.
I20260709 13:01:53.567914  1678 raft_consensus.cc:1275] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 11 index: 11. Preceding OpId from leader: term: 12 index: 12. (index mismatch)
I20260709 13:01:53.533838  7585 raft_consensus.cc:2749] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 13:01:53.638018  1604 raft_consensus.cc:2468] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 19.
I20260709 13:01:53.680928  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:53.543458  1749 rpcz_store.cc:276] 0709 13:01:51.688004 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:52.029033 (+341029us) service_pool.cc:224] Handling call
0709 13:01:52.049707 (+ 20674us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:52.202683 (+152976us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:52.202719 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:01:52.234526 (+ 31807us) log.cc:844] Serialized 26 byte log entry
0709 13:01:52.426379 (+191853us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:01:52.426424 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:52.426520 (+    96us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:53.092397 (+665877us) spinlock_profiling.cc:243] Waited 3.2 ms on lock 0x7b240024b258. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f4262efb9c4 00007f4262ef9acb 00007f42630b4dc9 00007f42630b4a55 00007f42630b409d 00007f4265ce610e 00007f4265cfd88c 00007f4265cfcbb6 00007f4265cfc7d8 00007f4265d06221
0709 13:01:53.528690 (+436293us) raft_consensus.cc:1685] finished
0709 13:01:53.528730 (+    40us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:53.536307 (+  7577us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":61197,"raft.queue_time_us":186257,"raft.run_cpu_time_us":180,"raft.run_wall_time_us":13620,"spinlock_wait_cycles":7363968,"thread_start_us":251789,"threads_started":2,"wal-append.queue_time_us":182455}
I20260709 13:01:53.680054  1679 raft_consensus.cc:3055] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 LEADER]: Stepping down as leader of term 16
W20260709 13:01:53.688112  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 159604us
I20260709 13:01:53.679953  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:53.690407  1680 raft_consensus.cc:2468] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 18.
I20260709 13:01:53.692544  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:01:53.687487  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 152759us
I20260709 13:01:53.529541  7687 consensus_queue.cc:1048] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.038s
I20260709 13:01:53.695569  2335 raft_consensus.cc:1275] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 12 index: 10. (index mismatch)
I20260709 13:01:53.393736  7902 consensus_queue.cc:237] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:53.686581  1679 raft_consensus.cc:740] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:01:53.554026  7702 raft_consensus.cc:2749] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:01:53.680784   431 auto_leader_rebalancer.cc:527] All tables' leader rebalancing finished this round
W20260709 13:01:53.696910  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 196653us
I20260709 13:01:53.567032  7707 raft_consensus.cc:3060] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:01:53.564838  7647 consensus_queue.cc:1048] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.144s
I20260709 13:01:53.697774  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:53.448345  7863 consensus_queue.cc:260] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 17.10, Last appended by leader: 10, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:53.730888  1601 raft_consensus.cc:2468] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 14.
I20260709 13:01:53.556221  7307 raft_consensus.cc:2749] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:01:53.680440  7968 raft_consensus.cc:493] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:01:53.723052  1679 consensus_queue.cc:260] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 16.10, Last appended by leader: 10, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:53.701879  7745 raft_consensus.cc:493] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:53.697320  7845 raft_consensus.cc:2804] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Leader election won for term 18
I20260709 13:01:53.694998  7600 consensus_queue.cc:1048] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.019s
I20260709 13:01:53.727819  7910 raft_consensus.cc:493] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:53.723461  7560 raft_consensus.cc:493] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:01:53.791543  1601 raft_consensus.cc:1275] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 16 index: 9. Preceding OpId from leader: term: 17 index: 10. (index mismatch)
I20260709 13:01:53.737396  7906 raft_consensus.cc:493] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:53.803767  7745 raft_consensus.cc:515] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } }
I20260709 13:01:53.811376  7845 raft_consensus.cc:697] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 18 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:01:53.811801  7778 raft_consensus.cc:493] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
W20260709 13:01:53.812372  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:53.784456  7750 leader_election.cc:290] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 19 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:53.537421  7912 consensus_queue.cc:237] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.13, Last appended by leader: 13, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:53.797564  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 101653us
W20260709 13:01:53.808266  7817 consensus_meta.cc:287] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.767s	user 0.014s	sys 0.000s
I20260709 13:01:53.830876  1679 raft_consensus.cc:3060] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:01:53.753968  7585 raft_consensus.cc:493] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:01:53.831044  1601 raft_consensus.cc:1240] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 LEADER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 12. Current term is 13. Ops: []
I20260709 13:01:53.902740  7585 raft_consensus.cc:515] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:53.904723  1673 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57092 (request call id 1379) took 1172 ms. Trace:
I20260709 13:01:53.905102  7585 leader_election.cc:290] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:53.905164  1673 rpcz_store.cc:276] 0709 13:01:52.732596 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:52.740041 (+  7445us) service_pool.cc:224] Handling call
0709 13:01:52.740402 (+   361us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:52.927268 (+186866us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:53.904450 (+977182us) spinlock_profiling.cc:243] Waited 24.1 ms on lock 0x7b54001e7dd4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:01:53.904651 (+   201us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":55438208}
I20260709 13:01:53.834800  7977 raft_consensus.cc:493] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:53.907482  7977 raft_consensus.cc:515] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:53.909525  2298 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:53.909739  7977 leader_election.cc:290] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:53.911157  7818 raft_consensus.cc:3060] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Advancing to term 17
W20260709 13:01:53.915491  7956 log.cc:927] Time spent T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.487s	user 0.000s	sys 0.001s
I20260709 13:01:53.917155  1597 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 1557) took 1018 ms. Trace:
I20260709 13:01:53.917634  1597 rpcz_store.cc:276] 0709 13:01:52.898750 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:52.913733 (+ 14983us) service_pool.cc:224] Handling call
0709 13:01:52.924721 (+ 10988us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:52.931067 (+  6346us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:52.931103 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:01:52.984060 (+ 52957us) log.cc:844] Serialized 26 byte log entry
0709 13:01:53.313542 (+329482us) raft_consensus.cc:1645] Marking committed up to 13
0709 13:01:53.313602 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:53.313696 (+    94us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:53.916861 (+603165us) raft_consensus.cc:1685] finished
0709 13:01:53.916894 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:53.917071 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":18857,"raft.run_cpu_time_us":79,"raft.run_wall_time_us":78,"thread_start_us":325725,"threads_started":1,"wal-append.queue_time_us":380614}
I20260709 13:01:53.919768  7908 raft_consensus.cc:493] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
W20260709 13:01:53.920585  1719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:53.920392  7908 raft_consensus.cc:515] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:53.921094  1719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:53.922699  1646 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:53.925587  1717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:53.858142  7906 raft_consensus.cc:3060] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:01:53.761978  7964 consensus_queue.cc:1059] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 16.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.077s
I20260709 13:01:53.787544  7968 raft_consensus.cc:515] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:53.843863  1751 raft_consensus.cc:1240] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 16. Current term is 17. Ops: []
W20260709 13:01:53.815927  7859 consensus_meta.cc:287] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.797s	user 0.010s	sys 0.001s
I20260709 13:01:53.865309  7808 raft_consensus.cc:2749] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Leader election lost for term 17. Reason: Vote denied by peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261) with higher term. Message: Invalid argument: T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 17. Current term is 18.
I20260709 13:01:53.868855  7778 raft_consensus.cc:515] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:53.893788  7817 raft_consensus.cc:2749] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:01:53.946647  7915 leader_election.cc:290] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:53.894382  7845 consensus_queue.cc:237] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 16.8, Last appended by leader: 8, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:53.946729  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:53.894798  7961 consensus_queue.cc:1048] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.340s
I20260709 13:01:53.895610  7969 consensus_queue.cc:1048] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.188s
I20260709 13:01:53.896672  7955 raft_consensus.cc:493] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:53.961360  7745 leader_election.cc:290] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:53.961925  7955 raft_consensus.cc:515] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:53.962982  7977 raft_consensus.cc:493] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:01:53.963551  7977 raft_consensus.cc:515] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:53.964284  7955 leader_election.cc:290] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:53.965600  7977 leader_election.cc:290] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:53.943879  7913 leader_election.cc:290] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 14 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:53.857199  7875 consensus_queue.cc:237] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 17.14, Last appended by leader: 14, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:53.969462  7819 consensus_meta.cc:287] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.920s	user 0.010s	sys 0.003s
I20260709 13:01:53.969901  7908 leader_election.cc:290] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:53.707487  1496 catalog_manager.cc:5697] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 10 to 11, leader changed from 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) to af520b67945d443b901bbeca37cc5701 (127.0.107.195). New cstate: current_term: 11 leader_uuid: "af520b67945d443b901bbeca37cc5701" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 13:01:53.971670  1496 catalog_manager.cc:5697] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 7 to 9, leader changed from 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) to 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193). New cstate: current_term: 9 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:01:53.858374  7560 raft_consensus.cc:515] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:53.897521  7949 raft_consensus.cc:493] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:53.976830  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:01:53.899215  1645 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:53.901257  1569 leader_election.cc:304] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:01:53.832350  7910 raft_consensus.cc:515] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:53.903218  7965 log.cc:927] Time spent T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.391s	user 0.001s	sys 0.000s
I20260709 13:01:53.926188  2332 raft_consensus.cc:2468] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 15.
W20260709 13:01:53.838006  7830 consensus_meta.cc:287] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.783s	user 0.005s	sys 0.005s
I20260709 13:01:53.861558  7978 raft_consensus.cc:493] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:53.894304  7953 raft_consensus.cc:493] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:53.867093  7307 raft_consensus.cc:493] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:53.948323  1604 raft_consensus.cc:1240] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 16. Current term is 17. Ops: []
I20260709 13:01:53.919601  7687 consensus_queue.cc:1048] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.019s
I20260709 13:01:53.960247  7926 leader_election.cc:290] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 13 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:53.977499  2335 raft_consensus.cc:2468] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 18.
I20260709 13:01:53.982576  2336 raft_consensus.cc:1240] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 12. Current term is 13. Ops: []
W20260709 13:01:53.983669  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 82504.5us
I20260709 13:01:54.016793  1679 raft_consensus.cc:1275] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 17 index: 10. (term mismatch)
I20260709 13:01:54.017401  1679 pending_rounds.cc:85] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d: Aborting all ops after (but not including) 9
W20260709 13:01:54.015193  1571 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:54.018453  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:54.005842  1717 leader_election.cc:304] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:54.029600  1680 raft_consensus.cc:1275] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 16 index: 13. Preceding OpId from leader: term: 17 index: 14. (index mismatch)
I20260709 13:01:54.035523  1679 pending_rounds.cc:107] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d: Aborting uncommitted NO_OP operation due to leader change: 16.10
I20260709 13:01:54.035915  1679 raft_consensus.cc:2889] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:01:54.036310  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:54.037186  1600 raft_consensus.cc:2468] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 11.
I20260709 13:01:54.022552  1717 leader_election.cc:304] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:53.979069  7856 raft_consensus.cc:493] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:54.040493  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
W20260709 13:01:54.041846  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 280819us
I20260709 13:01:54.042868  1678 raft_consensus.cc:2468] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 15.
I20260709 13:01:54.043103  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 22 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:53.987521  7647 raft_consensus.cc:2764] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Leader pre-election decision vote started in defunct term 18: won
I20260709 13:01:53.986266  7818 raft_consensus.cc:493] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:01:54.068310  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
W20260709 13:01:54.072184  1718 leader_election.cc:336] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Timed out: RequestConsensusVote RPC to 127.0.107.194:46261 timed out after 1.983s (SENT)
W20260709 13:01:54.072326  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 342736us
I20260709 13:01:54.072779  1718 leader_election.cc:304] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:54.072968  1569 leader_election.cc:304] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:54.074023  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:54.035873  7913 raft_consensus.cc:2804] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:01:54.040488  7745 raft_consensus.cc:493] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:54.042349  1717 leader_election.cc:304] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:54.089442  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:54.098841  1601 raft_consensus.cc:2468] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 14.
I20260709 13:01:54.014216  7830 raft_consensus.cc:515] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:54.100298  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:54.045885  7915 raft_consensus.cc:2804] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 13:01:54.110635  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:54.116982  1746 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55840 (request call id 2894) took 1050 ms. Trace:
W20260709 13:01:54.118139  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 45991.4us
I20260709 13:01:54.122423  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:53.986143  7949 raft_consensus.cc:515] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:54.041105  7307 raft_consensus.cc:515] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:54.019963  7978 raft_consensus.cc:515] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:54.147532  2334 raft_consensus.cc:1240] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 13. Current term is 14. Ops: []
I20260709 13:01:54.155751  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:54.156894  1671 raft_consensus.cc:2393] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 15: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:01:54.043224  7953 raft_consensus.cc:515] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:54.120437  1746 rpcz_store.cc:276] 0709 13:01:53.066876 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:53.067089 (+   213us) service_pool.cc:224] Handling call
0709 13:01:53.067498 (+   409us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:53.108611 (+ 41113us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:53.108650 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:01:53.129318 (+ 20668us) log.cc:844] Serialized 26 byte log entry
0709 13:01:53.344385 (+215067us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:01:53.344429 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:53.344508 (+    79us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:54.116698 (+772190us) raft_consensus.cc:1685] finished
0709 13:01:54.116733 (+    35us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:54.116896 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":3941,"raft.run_cpu_time_us":156,"raft.run_wall_time_us":14833,"thread_start_us":153098,"threads_started":1,"wal-append.queue_time_us":229681}
I20260709 13:01:54.084928  7818 raft_consensus.cc:515] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:54.164549  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:54.159893  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:54.070086  7856 raft_consensus.cc:515] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:54.148216  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:54.188031  1605 raft_consensus.cc:2468] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 18.
I20260709 13:01:54.189455  7980 raft_consensus.cc:493] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:54.186333  7976 raft_consensus.cc:2749] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:01:54.193295  1600 raft_consensus.cc:2410] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for term 12 because replica has last-logged OpId of term: 11 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 9.
I20260709 13:01:54.145045  7728 consensus_queue.cc:1059] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.050s
I20260709 13:01:54.101339  7813 consensus_queue.cc:1048] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.051s
I20260709 13:01:54.205086  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:54.208374  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:54.124233  7745 raft_consensus.cc:515] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:54.208922  2298 leader_election.cc:304] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:01:54.215245  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:54.218962  1746 raft_consensus.cc:2393] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 14: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
W20260709 13:01:54.220183  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 175031us
I20260709 13:01:54.226274  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:54.227459  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:54.229051  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:54.230826  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:54.231881  1674 raft_consensus.cc:2368] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 18. Current term is 19.
I20260709 13:01:54.230957  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:54.235984  7910 leader_election.cc:290] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:54.237739  7830 leader_election.cc:290] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 14 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:54.238247  1601 raft_consensus.cc:2393] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 14: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:01:54.240100  7953 leader_election.cc:290] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:01:54.240465  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:54.214385  7968 leader_election.cc:290] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:54.243243  7307 leader_election.cc:290] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:54.208076  7647 raft_consensus.cc:2804] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:01:54.255365  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:54.256333  1601 raft_consensus.cc:3060] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:01:54.263715  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:54.261518  7702 raft_consensus.cc:2804] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:01:54.266981  7702 raft_consensus.cc:493] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:54.267454  7702 raft_consensus.cc:3060] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:01:54.271919  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:54.272637  1753 raft_consensus.cc:2468] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 14.
I20260709 13:01:54.218364  7980 raft_consensus.cc:515] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:54.273602  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:54.277397  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:54.273970  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:54.274317  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:54.281526  1678 raft_consensus.cc:2410] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for term 13 because replica has last-logged OpId of term: 12 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 7.
I20260709 13:01:54.277340  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:54.285925  1596 raft_consensus.cc:2468] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 14.
I20260709 13:01:54.288890  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
W20260709 13:01:54.291325  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 64077.8us
I20260709 13:01:54.265813  7912 raft_consensus.cc:2804] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:01:54.293470  7912 raft_consensus.cc:493] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:54.293788  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:54.294729  1677 raft_consensus.cc:2410] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for term 14 because replica has last-logged OpId of term: 13 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 7.
I20260709 13:01:54.295037  7307 consensus_queue.cc:1048] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 13:01:54.295625  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:54.296254  1677 raft_consensus.cc:2468] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 13.
I20260709 13:01:54.297187  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:54.297832  1677 raft_consensus.cc:2468] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 17.
I20260709 13:01:54.298996  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:54.227555  7913 raft_consensus.cc:697] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 14 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
W20260709 13:01:54.275157  7984 log.cc:927] Time spent T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.116s	user 0.000s	sys 0.000s
I20260709 13:01:54.301573  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:01:54.277837  7754 consensus_meta.cc:287] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.918s	user 0.012s	sys 0.002s
I20260709 13:01:54.276252  7980 leader_election.cc:290] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:01:54.312229  1718 connection.cc:256] RPC call timeout handler was delayed by -1.4153420360000268s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1825s for a response
I20260709 13:01:54.303783  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:01:54.410899  1646 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:54.448391  1644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:54.494272  1643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:54.432854  2298 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:54.504823  1719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:54.505589  1719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:54.506922  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 409875us
I20260709 13:01:54.449441  7728 raft_consensus.cc:493] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:54.283722  7560 leader_election.cc:290] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:54.521225  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:54.525352  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:54.586338  1679 raft_consensus.cc:2393] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 14: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:01:54.293824  7912 raft_consensus.cc:3060] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Advancing to term 13
W20260709 13:01:54.510878  7946 consensus_meta.cc:287] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 1.211s	user 0.016s	sys 0.000s
I20260709 13:01:54.633590  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:54.634801  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:54.449887  7993 raft_consensus.cc:493] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:54.300114  7813 consensus_queue.cc:1059] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 13.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.306s
I20260709 13:01:54.509697  7961 raft_consensus.cc:493] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:54.659864  1677 raft_consensus.cc:2376] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 17. Re-sending same reply.
I20260709 13:01:54.669608  1679 raft_consensus.cc:2468] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 15.
I20260709 13:01:54.507884  7728 raft_consensus.cc:515] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:54.674988  1718 leader_election.cc:336] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 pre-election: RPC error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Timed out: RequestConsensusVote RPC to 127.0.107.194:46261 timed out after 2.039s (SENT)
I20260709 13:01:54.687687  1678 raft_consensus.cc:2468] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 17.
I20260709 13:01:54.685474  1718 leader_election.cc:304] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:54.690244  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:54.701478  1601 raft_consensus.cc:2468] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 15.
I20260709 13:01:54.633579  7967 raft_consensus.cc:493] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:54.704303  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:54.703104  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:54.714164  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:54.735631  1679 raft_consensus.cc:2393] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 14: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
W20260709 13:01:54.688752  7930 consensus_meta.cc:287] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 1.248s	user 0.009s	sys 0.005s
I20260709 13:01:54.736910  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 14 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:54.657219  7817 raft_consensus.cc:493] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:54.657591  7916 raft_consensus.cc:493] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:01:54.741601  1677 raft_consensus.cc:2410] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for term 14 because replica has last-logged OpId of term: 13 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 6.
I20260709 13:01:54.449766  7913 consensus_queue.cc:237] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 13.12, Last appended by leader: 12, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:54.752470  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:54.701390  7813 raft_consensus.cc:493] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:54.699788  7600 raft_consensus.cc:493] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:54.753293  1679 raft_consensus.cc:2468] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 18.
I20260709 13:01:54.656533  7946 raft_consensus.cc:515] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:54.762090  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:54.766765  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:54.767477  1677 raft_consensus.cc:3060] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:01:54.739995  8006 raft_consensus.cc:493] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:54.745849  7754 raft_consensus.cc:3055] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 LEADER]: Stepping down as leader of term 13
I20260709 13:01:54.771981  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:54.775566  1601 raft_consensus.cc:2468] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 18.
I20260709 13:01:54.778069  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:01:54.787580  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 135588us
I20260709 13:01:54.682140  7961 raft_consensus.cc:515] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:54.791333  1675 raft_consensus.cc:1240] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 16. Current term is 17. Ops: [16.11-16.11]
I20260709 13:01:54.757791  7818 leader_election.cc:290] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:54.811309  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 15 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:54.799135  1596 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40154 (request call id 2667) took 1380 ms. Trace:
I20260709 13:01:54.740398  7967 raft_consensus.cc:515] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:54.795737  8003 raft_consensus.cc:2749] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:01:54.701911  7993 raft_consensus.cc:515] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:54.799010  7981 raft_consensus.cc:493] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:54.799921  7903 raft_consensus.cc:493] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:54.800403  7819 raft_consensus.cc:3055] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 LEADER]: Stepping down as leader of term 12
W20260709 13:01:54.801548  7988 log.cc:927] Time spent T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.091s	user 0.000s	sys 0.000s
I20260709 13:01:54.802960  1604 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40154 (request call id 2668) took 1377 ms. Trace:
I20260709 13:01:54.740891  7930 raft_consensus.cc:515] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:54.826014  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:54.804358  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:54.827706  1601 raft_consensus.cc:2393] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 14: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:01:54.805702  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:54.761662  7813 raft_consensus.cc:515] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:01:54.807842  7963 log.cc:927] Time spent T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 1.415s	user 0.002s	sys 0.020s
W20260709 13:01:54.830686  7992 log.cc:927] Time spent T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.157s	user 0.001s	sys 0.000s
I20260709 13:01:54.757352  7916 raft_consensus.cc:515] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:54.832043  1672 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57092 (request call id 1386) took 1370 ms. Trace:
I20260709 13:01:54.833065  7916 leader_election.cc:290] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:54.834029  1603 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 1559) took 1834 ms. Trace:
I20260709 13:01:54.834380  7993 leader_election.cc:290] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:54.810148  7647 raft_consensus.cc:493] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:54.835356  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:54.835597  7647 raft_consensus.cc:3060] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:01:54.836091  1601 raft_consensus.cc:2468] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 11.
W20260709 13:01:54.838548  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:54.832409  1672 rpcz_store.cc:276] 0709 13:01:53.461138 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:53.463233 (+  2095us) service_pool.cc:224] Handling call
0709 13:01:53.463604 (+   371us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:53.529924 (+ 66320us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:53.529958 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:01:53.544250 (+ 14292us) log.cc:844] Serialized 26 byte log entry
0709 13:01:54.045900 (+501650us) raft_consensus.cc:1645] Marking committed up to 13
0709 13:01:54.045944 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:54.046018 (+    74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:54.831801 (+785783us) raft_consensus.cc:1685] finished
0709 13:01:54.831826 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:54.831972 (+   146us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":207731,"raft.queue_time_us":66468,"raft.run_cpu_time_us":124,"raft.run_wall_time_us":11252,"thread_start_us":353225,"threads_started":1,"wal-append.queue_time_us":888609}
I20260709 13:01:54.838515  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:54.812227  1596 rpcz_store.cc:276] 0709 13:01:53.418939 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:53.467646 (+ 48707us) service_pool.cc:224] Handling call
0709 13:01:54.798944 (+1331298us) spinlock_profiling.cc:243] Waited 506 ms on lock 0x7b54000acb54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:01:54.799063 (+   119us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1164589824}
I20260709 13:01:54.766402  7949 leader_election.cc:290] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:01:54.819895  1718 connection.cc:256] RPC call timeout handler was delayed by -1.1948420360000114s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.207s for a response
I20260709 13:01:54.786661  7978 leader_election.cc:290] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:54.779309  7754 raft_consensus.cc:740] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:01:54.828177  1674 raft_consensus.cc:2410] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for term 18 because replica has last-logged OpId of term: 17 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 11.
I20260709 13:01:54.826656  1604 rpcz_store.cc:276] 0709 13:01:53.425399 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:53.698985 (+273586us) service_pool.cc:224] Handling call
0709 13:01:53.699249 (+   264us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:54.014546 (+315297us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:54.802662 (+788116us) spinlock_profiling.cc:243] Waited 249 ms on lock 0x7b54002d9454. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:01:54.802866 (+   204us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":572290816}
I20260709 13:01:54.824456  7819 raft_consensus.cc:740] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:01:54.841090  1601 raft_consensus.cc:3060] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:01:54.756557  7817 raft_consensus.cc:515] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:54.843606  1718 connection.cc:256] RPC call timeout handler was delayed by -1.1912420360000056s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2074s for a response
W20260709 13:01:54.890769  1644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:54.834252  1677 raft_consensus.cc:2468] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 19.
I20260709 13:01:54.834468  1603 rpcz_store.cc:276] 0709 13:01:52.999765 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:53.000292 (+   527us) service_pool.cc:224] Handling call
0709 13:01:53.000763 (+   471us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:53.001213 (+   450us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:53.001253 (+    40us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:01:53.066728 (+ 65475us) log.cc:844] Serialized 26 byte log entry
0709 13:01:53.347605 (+280877us) raft_consensus.cc:1645] Marking committed up to 13
0709 13:01:53.347682 (+    77us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:53.347785 (+   103us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:54.807207 (+1459422us) spinlock_profiling.cc:243] Waited 43.4 ms on lock 0x7b4400022b4c. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f4262f9bc85 00007f4262f9c6a2 00007f4265d0e1db 00007f4265d09aaf 00007f4265cfd86e 00007f4265cfcbb6 00007f4265cfc7d8 00007f4265d06221 00007f4265d061d9 00007f4265d06169
0709 13:01:54.833670 (+ 26463us) raft_consensus.cc:1685] finished
0709 13:01:54.833711 (+    41us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:54.833958 (+   247us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":16919,"raft.queue_time_us":2566,"raft.run_cpu_time_us":159,"raft.run_wall_time_us":159,"spinlock_wait_cycles":99906304,"thread_start_us":199526,"threads_started":1,"wal-append.queue_time_us":271838}
W20260709 13:01:54.886685  1718 connection.cc:256] RPC call timeout handler was delayed by -1.1855688880000343s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.46540000000000004s for a response
I20260709 13:01:54.812242  1679 raft_consensus.cc:2393] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 15: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
W20260709 13:01:54.897776  1718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:54.828147  7981 raft_consensus.cc:515] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:54.826823  7903 raft_consensus.cc:515] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:54.788200  8006 raft_consensus.cc:515] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:01:54.882869  7845 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.922s	user 0.009s	sys 0.025s
I20260709 13:01:54.766819  7600 raft_consensus.cc:515] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:54.816650  7745 leader_election.cc:290] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:54.872951  1598 raft_consensus.cc:1275] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
I20260709 13:01:54.872951  1604 raft_consensus.cc:1275] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 10 index: 7. Preceding OpId from leader: term: 13 index: 9. (index mismatch)
W20260709 13:01:54.898283  1718 connection.cc:256] RPC call timeout handler was delayed by -1.0664245920000326s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2074s for a response
I20260709 13:01:54.847898  7754 consensus_queue.cc:260] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.7, Last appended by leader: 7, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:54.912417  1601 raft_consensus.cc:2468] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 12.
W20260709 13:01:54.915141  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 408203us
W20260709 13:01:54.912813  7915 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.629s	user 0.013s	sys 0.019s
W20260709 13:01:54.899134  7845 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.939s	user 0.009s	sys 0.025s
I20260709 13:01:54.922886  1676 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57092 (request call id 1389) took 1460 ms. Trace:
I20260709 13:01:54.923089  1596 raft_consensus.cc:1240] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 16. Current term is 17. Ops: [10.8-16.11]
I20260709 13:01:54.897634  8003 raft_consensus.cc:493] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:01:54.929077  1646 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:54.924800  1676 rpcz_store.cc:276] 0709 13:01:53.462105 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:53.464722 (+  2617us) service_pool.cc:224] Handling call
0709 13:01:53.465026 (+   304us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:53.520508 (+ 55482us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:53.520545 (+    37us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:01:53.520593 (+    48us) raft_consensus.cc:1645] Marking committed up to 14
0709 13:01:53.520625 (+    32us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:54.922615 (+1401990us) spinlock_profiling.cc:243] Waited 55.2 ms on lock 0x7b5400163d54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:01:54.922668 (+    53us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:54.922804 (+   136us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":127065984}
W20260709 13:01:54.936075  1719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:54.903111  7582 raft_consensus.cc:493] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:54.905767  7818 raft_consensus.cc:493] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:54.913098  7993 raft_consensus.cc:493] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:54.948175  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:54.948917  1753 raft_consensus.cc:2468] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 12.
W20260709 13:01:54.951720  1673 consensus_meta.cc:287] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.669s	user 0.014s	sys 0.000s
W20260709 13:01:54.938486  7915 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.676s	user 0.013s	sys 0.020s
I20260709 13:01:54.923949  8009 raft_consensus.cc:493] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:54.952188  1673 raft_consensus.cc:2468] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 17.
I20260709 13:01:54.925112  7845 raft_consensus.cc:493] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:54.960574  1672 raft_consensus.cc:1275] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 16 index: 9. Preceding OpId from leader: term: 17 index: 10. (index mismatch)
I20260709 13:01:54.955778  1718 leader_election.cc:304] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 13:01:54.947527  7906 consensus_meta.cc:287] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 1.003s	user 0.015s	sys 0.000s
I20260709 13:01:54.938552  7916 raft_consensus.cc:493] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:54.957914  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 16 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:54.960279  7845 raft_consensus.cc:515] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:54.964892  7817 leader_election.cc:290] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:54.965750  7845 leader_election.cc:290] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:54.952546  7582 raft_consensus.cc:515] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:54.969101  7582 leader_election.cc:290] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:54.969952  7582 raft_consensus.cc:493] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
W20260709 13:01:54.971262  7702 consensus_meta.cc:287] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.703s	user 0.014s	sys 0.000s
I20260709 13:01:54.971786  7702 raft_consensus.cc:515] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:54.952982  7818 raft_consensus.cc:515] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:54.976322  7961 leader_election.cc:290] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:54.938819  8003 raft_consensus.cc:515] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:54.977106  7818 leader_election.cc:290] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:54.965706  7946 leader_election.cc:290] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 17 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:54.978920  8003 leader_election.cc:290] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:54.974301  7702 leader_election.cc:290] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 15 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:54.952044  7754 raft_consensus.cc:3060] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:01:54.963162  1745 raft_consensus.cc:2468] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 15.
W20260709 13:01:54.984858  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 60455.5us
I20260709 13:01:54.956481  7915 raft_consensus.cc:697] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 19 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:01:54.995342  7915 consensus_queue.cc:237] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.11, Last appended by leader: 11, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:54.984249  7582 raft_consensus.cc:515] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:54.998000  1645 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:54.992340  7981 leader_election.cc:290] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:54.993011  7916 raft_consensus.cc:515] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:54.998791  7582 leader_election.cc:290] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:55.022034  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:54.999202  8015 consensus_queue.cc:1059] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 15.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.038s
I20260709 13:01:55.000653  7967 leader_election.cc:290] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:55.001392  7813 leader_election.cc:290] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:55.002632  8006 leader_election.cc:290] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:54.956773  7993 raft_consensus.cc:515] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:55.002727  7912 raft_consensus.cc:515] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:55.004601  1747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43156 (request call id 1271) took 1894 ms. Trace:
I20260709 13:01:54.989180  7913 raft_consensus.cc:2749] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:01:55.028349  1747 rpcz_store.cc:276] 0709 13:01:53.110411 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:53.317620 (+207209us) service_pool.cc:224] Handling call
0709 13:01:53.317939 (+   319us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:53.372261 (+ 54322us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:55.004304 (+1632043us) spinlock_profiling.cc:243] Waited 44.2 ms on lock 0x7b5400019fd4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:01:55.004527 (+   223us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":101688064}
I20260709 13:01:55.029502  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:55.005458  1602 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 1558) took 2018 ms. Trace:
I20260709 13:01:54.984673  7906 raft_consensus.cc:515] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:54.922624  7819 consensus_queue.cc:260] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.12, Last appended by leader: 12, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:55.006263  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 50625.3us
I20260709 13:01:55.006616  1643 leader_election.cc:304] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:01:55.015167  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:55.020994  1717 leader_election.cc:304] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:54.963321  8009 raft_consensus.cc:515] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:55.022925  1569 leader_election.cc:304] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
W20260709 13:01:55.025442  1646 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:55.025791  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:55.030300  1602 rpcz_store.cc:276] 0709 13:01:52.986767 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:52.986923 (+   156us) service_pool.cc:224] Handling call
0709 13:01:53.004105 (+ 17182us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:53.379630 (+375525us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:55.005165 (+1625535us) spinlock_profiling.cc:243] Waited 375 ms on lock 0x7b5400071054. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:01:55.005381 (+   216us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":861782784}
W20260709 13:01:55.101092  1646 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:55.031579  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:55.034582  1673 raft_consensus.cc:2468] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 17.
I20260709 13:01:55.101649  1748 raft_consensus.cc:1240] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 16. Current term is 17. Ops: []
I20260709 13:01:55.034804  1674 raft_consensus.cc:1275] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 15 index: 8. Preceding OpId from leader: term: 18 index: 9. (index mismatch)
W20260709 13:01:55.102241  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 79400.1us
W20260709 13:01:55.036556  1719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:55.030964  1747 raft_consensus.cc:2468] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 14.
W20260709 13:01:55.101743  1646 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:55.104295  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:55.105880  1571 leader_election.cc:304] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:01:55.101833  1677 raft_consensus.cc:2468] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 17.
I20260709 13:01:55.109397  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:55.112296  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:55.112655  1604 raft_consensus.cc:1240] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 13. Current term is 14. Ops: []
I20260709 13:01:55.113090  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:55.113934  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 14 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:55.113948  1677 raft_consensus.cc:2468] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 11.
I20260709 13:01:55.116093  1598 raft_consensus.cc:1240] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 16. Current term is 17. Ops: [16.8-16.8]
I20260709 13:01:55.112365  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:55.111605  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:55.115765  1673 raft_consensus.cc:2468] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 13.
W20260709 13:01:55.037322  7911 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.687s	user 0.026s	sys 0.032s
I20260709 13:01:55.026813  7981 consensus_queue.cc:1048] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.079s
I20260709 13:01:55.121963  7906 leader_election.cc:290] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 17 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:55.127833  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:55.109978  8006 raft_consensus.cc:493] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:01:55.130328  1602 raft_consensus.cc:1275] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 12 index: 10. (index mismatch)
I20260709 13:01:55.037303  7961 raft_consensus.cc:493] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:55.120497  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:55.142027  1596 raft_consensus.cc:2468] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 18.
I20260709 13:01:55.193445  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 18 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:55.188508  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 15 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:55.195127  1677 raft_consensus.cc:2410] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for term 15 because replica has last-logged OpId of term: 13 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 6.
I20260709 13:01:55.196262  1745 raft_consensus.cc:2468] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 12.
I20260709 13:01:55.105993  7967 raft_consensus.cc:3055] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 13:01:55.110368  8003 raft_consensus.cc:2804] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:01:55.222833  8003 raft_consensus.cc:697] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 14 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:01:55.105242  7582 raft_consensus.cc:493] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:55.223613  7582 raft_consensus.cc:515] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:55.224046  8003 consensus_queue.cc:237] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 13.12, Last appended by leader: 12, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:55.228333  8013 log.cc:927] Time spent T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.267s	user 0.001s	sys 0.000s
W20260709 13:01:55.201546  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 92306.1us
W20260709 13:01:55.202478  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 96664.8us
I20260709 13:01:55.202996  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:01:55.203549  8014 log.cc:927] Time spent T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.101s	user 0.001s	sys 0.000s
I20260709 13:01:55.186408  7913 consensus_queue.cc:1048] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.017s
I20260709 13:01:55.184900  8009 leader_election.cc:290] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:55.112531  7647 raft_consensus.cc:515] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:55.204470  7921 raft_consensus.cc:493] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:55.205925  7912 leader_election.cc:290] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 13 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 13:01:55.272858  1488 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet 983621a6fc5b4c9ea0159a648dfe7a36 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER failed with leader 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 3b879799899a499c8d409ee9f1277cd1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 16 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 13:01:55.211083  7967 raft_consensus.cc:740] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:01:55.118743  8015 consensus_queue.cc:1059] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 16.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.088s
I20260709 13:01:55.122228  7845 raft_consensus.cc:2749] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 13:01:55.136571  8010 raft_consensus.cc:2749] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 13:01:55.134724  7903 leader_election.cc:290] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:01:55.126963  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 137893us
I20260709 13:01:55.181790  7778 leader_election.cc:290] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:55.108762  7577 raft_consensus.cc:2804] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:01:55.135483  1646 leader_election.cc:304] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
W20260709 13:01:55.291977  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 156593us
I20260709 13:01:55.292891  2329 raft_consensus.cc:1240] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 17 LEADER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 16. Current term is 17. Ops: []
I20260709 13:01:55.293133  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 22 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:55.213714  7947 raft_consensus.cc:493] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:55.219964  1751 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55848 (request call id 3002) took 1296 ms. Trace:
I20260709 13:01:55.257910  1569 leader_election.cc:304] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:01:55.115048  7819 raft_consensus.cc:3060] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:01:55.235646  1571 leader_election.cc:304] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:55.265841  7582 leader_election.cc:290] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:55.179263  7916 leader_election.cc:290] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:55.277287  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:55.277760  7830 raft_consensus.cc:493] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:55.278560  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:01:55.206456  7911 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.867s	user 0.029s	sys 0.032s
I20260709 13:01:55.308681  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:55.309811  1643 leader_election.cc:304] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 13:01:55.285400  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:55.288625  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:55.264626  7961 raft_consensus.cc:515] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:55.291188  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:55.278054  7921 raft_consensus.cc:515] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:55.138754  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:55.294760  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 15 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:55.290568  7903 consensus_queue.cc:1048] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.006s
I20260709 13:01:55.292744  1646 leader_election.cc:304] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:55.291496  7577 raft_consensus.cc:493] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:55.299466  8024 raft_consensus.cc:493] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:55.299732  8009 raft_consensus.cc:2804] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:01:55.297103  1751 rpcz_store.cc:276] 0709 13:01:53.923828 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:54.446939 (+523111us) service_pool.cc:224] Handling call
0709 13:01:54.510630 (+ 63691us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:54.519107 (+  8477us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:54.519154 (+    47us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:01:54.635895 (+116741us) log.cc:844] Serialized 26 byte log entry
0709 13:01:54.942710 (+306815us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:01:54.942756 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:54.942827 (+    71us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:55.219671 (+276844us) raft_consensus.cc:1685] finished
0709 13:01:55.219698 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:55.219866 (+   168us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":68850,"raft.run_cpu_time_us":197,"raft.run_wall_time_us":8336,"thread_start_us":232346,"threads_started":1,"wal-append.queue_time_us":357256}
I20260709 13:01:55.264068  8006 raft_consensus.cc:515] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:55.314378  7830 raft_consensus.cc:515] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:55.306161  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 48364.2us
I20260709 13:01:55.338114  8006 leader_election.cc:290] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:55.339241  7647 leader_election.cc:290] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 19 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:55.344599  7961 leader_election.cc:290] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:55.315413  8010 raft_consensus.cc:2804] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:01:55.345570  7600 leader_election.cc:290] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:55.316057  7993 leader_election.cc:290] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:55.346678  1569 leader_election.cc:304] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:01:55.347664  1569 leader_election.cc:304] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:01:55.348443  7950 consensus_queue.cc:1048] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 13:01:55.349493  8024 raft_consensus.cc:515] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:55.352146  8024 leader_election.cc:290] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:55.352768  7967 consensus_queue.cc:260] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.9, Last appended by leader: 9, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:55.353343  7863 raft_consensus.cc:3060] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:01:55.354673  7967 raft_consensus.cc:3060] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:01:55.355150  7727 raft_consensus.cc:2804] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:01:55.355595  7727 raft_consensus.cc:493] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:55.355968  7727 raft_consensus.cc:3060] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:01:55.363003  7600 raft_consensus.cc:2804] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Leader pre-election won for term 15
W20260709 13:01:55.363211  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 53481.1us
I20260709 13:01:55.363484  7600 raft_consensus.cc:493] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:55.363992  7600 raft_consensus.cc:3060] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Advancing to term 15
W20260709 13:01:55.306615  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 71077.6us
I20260709 13:01:55.298678  7869 leader_election.cc:290] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 17 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:55.315258  7911 raft_consensus.cc:697] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 11 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:01:55.330300  7582 raft_consensus.cc:2804] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:01:55.282047  7912 raft_consensus.cc:493] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:55.348582  1569 leader_election.cc:304] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:01:55.349980  7950 raft_consensus.cc:2804] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:01:55.316849  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:55.321595  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:55.332424  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:55.368726  2298 leader_election.cc:304] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:01:55.390408  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
W20260709 13:01:55.402665  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 54149.6us
I20260709 13:01:55.391570  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 22 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:55.327152  7577 raft_consensus.cc:3060] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:01:55.306018  7947 raft_consensus.cc:515] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:55.315650  8028 raft_consensus.cc:2804] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:01:55.407377  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 19 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:55.408281  1672 raft_consensus.cc:2410] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 for term 19 because replica has last-logged OpId of term: 17 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 11.
I20260709 13:01:55.401006  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:55.405138  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:55.399366  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:55.409425  1598 raft_consensus.cc:3060] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:01:55.332683  8009 raft_consensus.cc:493] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:01:55.415608  1602 raft_consensus.cc:2393] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 17: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:01:55.415337  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:55.319680  7778 consensus_queue.cc:1048] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.075s
I20260709 13:01:55.345659  8010 raft_consensus.cc:493] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Starting leader election (detected failure of leader af520b67945d443b901bbeca37cc5701)
W20260709 13:01:55.418219  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 96078.1us
I20260709 13:01:55.433624  1718 leader_election.cc:304] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:55.418865  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:55.404870  7953 raft_consensus.cc:493] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:55.405100  7950 raft_consensus.cc:493] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:01:55.442145  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:55.442507  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:01:55.443365  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 146193us
I20260709 13:01:55.424580  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:55.420318  1745 raft_consensus.cc:2468] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 17.
I20260709 13:01:55.426555  1643 leader_election.cc:304] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:55.395434  7582 raft_consensus.cc:493] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
W20260709 13:01:55.399603  8026 log.cc:927] Time spent T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.113s	user 0.001s	sys 0.000s
I20260709 13:01:55.424068  1571 leader_election.cc:304] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:01:55.423517  8009 raft_consensus.cc:3060] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:01:55.386340  7964 consensus_queue.cc:1048] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.047s
I20260709 13:01:55.414819  7919 raft_consensus.cc:2804] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:01:55.415392  7816 raft_consensus.cc:2804] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:01:55.447902  1718 leader_election.cc:304] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:55.417430  8028 raft_consensus.cc:493] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:55.453472  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:55.421248  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:55.459206  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:55.405493  7912 raft_consensus.cc:515] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:55.453907  7869 raft_consensus.cc:2804] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:01:55.389500  7993 consensus_queue.cc:1048] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.047s
I20260709 13:01:55.438405  8010 raft_consensus.cc:3060] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Advancing to term 18
W20260709 13:01:55.468923  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 44937.9us
I20260709 13:01:55.441778  8024 raft_consensus.cc:493] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:01:55.462735  1644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:55.481088  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
W20260709 13:01:55.482028  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 55592.8us
I20260709 13:01:55.446190  7950 raft_consensus.cc:3060] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:01:55.464007  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:01:55.483419  7875 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.514s	user 0.012s	sys 0.018s
I20260709 13:01:55.456466  7582 raft_consensus.cc:3060] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:01:55.499284  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:55.473956  7816 raft_consensus.cc:493] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:55.516993  1602 raft_consensus.cc:3060] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Advancing to term 15
W20260709 13:01:55.500787  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 52987us
I20260709 13:01:55.520502  1718 leader_election.cc:304] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:55.484351  7976 raft_consensus.cc:2749] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:01:55.498481  7981 raft_consensus.cc:2749] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:01:55.479297  7919 raft_consensus.cc:493] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:55.485153  7818 raft_consensus.cc:2749] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:01:55.438992  7953 raft_consensus.cc:515] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:55.525511  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:55.569888  1745 raft_consensus.cc:2468] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 17.
I20260709 13:01:55.479877  8028 raft_consensus.cc:3060] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:01:55.499641  7869 raft_consensus.cc:493] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:55.531064  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 16 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:55.531288  7830 leader_election.cc:290] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:55.534732  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:55.535915  7912 leader_election.cc:290] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:55.539518  7947 leader_election.cc:290] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:55.498792  7911 consensus_queue.cc:237] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:55.542553  8012 consensus_queue.cc:1048] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 13:01:55.544358  7913 raft_consensus.cc:493] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:55.546659  8006 raft_consensus.cc:2749] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 13:01:55.551656  2334 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49082 (request call id 1130) took 1573 ms. Trace:
W20260709 13:01:55.553655  7960 log.cc:927] Time spent T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 2.193s	user 0.010s	sys 0.020s
I20260709 13:01:55.546228  7946 raft_consensus.cc:493] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:55.555022  1718 leader_election.cc:304] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1
W20260709 13:01:55.517650  7875 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.548s	user 0.012s	sys 0.019s
W20260709 13:01:55.560230  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:55.614657  2334 rpcz_store.cc:276] 0709 13:01:53.978478 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:53.978643 (+   165us) service_pool.cc:224] Handling call
0709 13:01:54.030241 (+ 51598us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:54.153813 (+123572us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:55.551367 (+1397554us) spinlock_profiling.cc:243] Waited 114 ms on lock 0x7b5400645354. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:01:55.551580 (+   213us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":261741312}
W20260709 13:01:55.515820  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 94993us
I20260709 13:01:55.576145  7953 leader_election.cc:290] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:55.597117  2301 leader_election.cc:304] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:01:55.598627  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:55.527457  7816 raft_consensus.cc:3060] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:01:55.611582  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:55.572142  7919 raft_consensus.cc:3060] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:01:55.568753  1598 raft_consensus.cc:2468] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 13.
I20260709 13:01:55.625485  2301 leader_election.cc:304] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:01:55.579346  7869 raft_consensus.cc:3060] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Advancing to term 17
W20260709 13:01:55.628314  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 73417.1us
I20260709 13:01:55.622534  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:01:55.618530  8029 log.cc:927] Time spent T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.069s	user 0.001s	sys 0.000s
W20260709 13:01:55.631842  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 186390us
I20260709 13:01:55.519121  8024 raft_consensus.cc:515] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:55.634408  1671 raft_consensus.cc:2410] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for term 18 because replica has last-logged OpId of term: 17 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 9.
I20260709 13:01:55.573625  1602 raft_consensus.cc:2468] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 15.
I20260709 13:01:55.636101  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:55.637070  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:55.637851  1598 raft_consensus.cc:2393] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 15: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:01:55.638531  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:55.639662  7714 raft_consensus.cc:493] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:55.642297  7714 raft_consensus.cc:515] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:55.617815  7830 consensus_queue.cc:1059] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 16.12, Next index: 13, Last known committed idx: 11, Time since last communication: 0.022s
I20260709 13:01:55.640511  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:01:55.644634  1646 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:55.641417  8024 leader_election.cc:290] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:55.650694  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 19 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:55.645578  1602 raft_consensus.cc:2468] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 14.
I20260709 13:01:55.667308  1745 raft_consensus.cc:2410] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for term 19 because replica has last-logged OpId of term: 17 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 8.
I20260709 13:01:55.631785  8030 raft_consensus.cc:3060] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:01:55.689244  1717 leader_election.cc:304] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:55.725713  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 19 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:55.729380  1605 raft_consensus.cc:1240] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 18. Current term is 19. Ops: []
I20260709 13:01:55.691323  7981 raft_consensus.cc:493] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:55.694859  7727 raft_consensus.cc:515] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:55.695462  8010 raft_consensus.cc:515] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:55.699520  1718 leader_election.cc:304] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:55.699607  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 18 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:55.759240  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:55.759773  1679 raft_consensus.cc:2410] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for term 18 because replica has last-logged OpId of term: 16 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 7.
I20260709 13:01:55.631875  7921 leader_election.cc:290] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:01:55.716609  7754 consensus_meta.cc:287] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.729s	user 0.002s	sys 0.011s
I20260709 13:01:55.614193  7912 consensus_queue.cc:1048] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.032s
I20260709 13:01:55.619912  7913 raft_consensus.cc:515] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:55.692153  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 18 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:55.633288  7915 raft_consensus.cc:2804] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Leader pre-election won for term 16
W20260709 13:01:55.755358  2332 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49082 (request call id 1125) took 3500 ms (client timeout 1832 ms). Trace:
W20260709 13:01:55.783183  2332 rpcz_store.cc:269] 0709 13:01:52.228750 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:52.233119 (+  4369us) service_pool.cc:224] Handling call
0709 13:01:55.728421 (+3495302us) spinlock_profiling.cc:243] Waited 1.66 s on lock 0x7b5400627ad4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:01:55.728548 (+   127us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3826325760}
I20260709 13:01:55.633612  7875 raft_consensus.cc:2749] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:01:55.760139  1602 raft_consensus.cc:2468] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 17.
I20260709 13:01:55.787737  1602 raft_consensus.cc:1275] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 16 index: 8. Preceding OpId from leader: term: 18 index: 9. (index mismatch)
W20260709 13:01:55.760188  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 71035.2us
I20260709 13:01:55.765503  8006 raft_consensus.cc:2804] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Leader election won for term 15
W20260709 13:01:55.763129  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 63706.5us
I20260709 13:01:55.763985  1605 raft_consensus.cc:1275] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 10 index: 8. Preceding OpId from leader: term: 15 index: 10. (index mismatch)
I20260709 13:01:55.764145  1752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55840 (request call id 2893) took 2698 ms. Trace:
I20260709 13:01:55.627492  7946 raft_consensus.cc:515] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:55.763571  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:55.769191  1603 raft_consensus.cc:2410] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for term 18 because replica has last-logged OpId of term: 16 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 8.
I20260709 13:01:55.748422  1643 leader_election.cc:304] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:01:55.770244  1596 raft_consensus.cc:2393] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 18: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:01:55.771246  8010 leader_election.cc:290] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 18 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:55.755450  8024 raft_consensus.cc:493] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:55.755347  7964 raft_consensus.cc:493] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
W20260709 13:01:55.780253  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 151296us
I20260709 13:01:55.720109  7950 raft_consensus.cc:515] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:55.782379  1604 raft_consensus.cc:2393] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 13: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:01:55.767814  7976 raft_consensus.cc:2804] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:01:55.790874  1752 rpcz_store.cc:276] 0709 13:01:53.065874 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:53.066018 (+   144us) service_pool.cc:224] Handling call
0709 13:01:53.079744 (+ 13726us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:53.144364 (+ 64620us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:53.144412 (+    48us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:01:53.145002 (+   590us) log.cc:844] Serialized 26 byte log entry
0709 13:01:53.247449 (+102447us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:01:53.247510 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:53.247613 (+   103us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:55.552953 (+2305340us) spinlock_profiling.cc:243] Waited 78.1 ms on lock 0x7b4400022b4c. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f4262f9bc85 00007f4262f9c6a2 00007f4265d0e1db 00007f4265d09aaf 00007f4265cfd86e 00007f4265cfcbb6 00007f4265cfc7d8 00007f4265d06221 00007f4265d061d9 00007f4265d06169
0709 13:01:55.759817 (+206864us) raft_consensus.cc:1685] finished
0709 13:01:55.759846 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:55.764060 (+  4214us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":20364,"raft.queue_time_us":23245,"raft.run_cpu_time_us":127,"raft.run_wall_time_us":7915,"spinlock_wait_cycles":179573120,"thread_start_us":101975,"threads_started":1,"wal-append.queue_time_us":154455}
W20260709 13:01:55.793926  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 161182us
W20260709 13:01:55.797508  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 49161.4us
I20260709 13:01:55.807729  1718 leader_election.cc:304] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:55.795177  1745 raft_consensus.cc:2468] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 14.
W20260709 13:01:55.803192  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 134415us
I20260709 13:01:55.763885  7981 raft_consensus.cc:515] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:55.817018  1752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55840 (request call id 2901) took 1782 ms. Trace:
I20260709 13:01:55.817435  1752 rpcz_store.cc:276] 0709 13:01:54.034737 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:55.804076 (+1769339us) service_pool.cc:224] Handling call
0709 13:01:55.816391 (+ 12315us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:55.816652 (+   261us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:55.816691 (+    39us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:01:55.816756 (+    65us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:01:55.816801 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:55.816852 (+    51us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:55.816942 (+    90us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:01:55.822912  1598 raft_consensus.cc:1240] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 17. Current term is 18. Ops: [17.10-17.10]
W20260709 13:01:55.824492  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 188538us
I20260709 13:01:55.825654  2299 leader_election.cc:304] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:55.828052  2299 leader_election.cc:304] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:55.800426  7754 raft_consensus.cc:2749] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 13:01:55.829339  1752 raft_consensus.cc:1275] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 17 index: 9. Preceding OpId from leader: term: 18 index: 9. (term mismatch)
I20260709 13:01:55.790477  7915 raft_consensus.cc:493] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:55.829785  1752 pending_rounds.cc:85] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701: Aborting all ops after (but not including) 8
I20260709 13:01:55.830240  1752 pending_rounds.cc:107] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701: Aborting uncommitted NO_OP operation due to leader change: 17.9
I20260709 13:01:55.830560  1752 raft_consensus.cc:2889] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:01:55.831391  1752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55840 (request call id 2903) took 1786 ms. Trace:
I20260709 13:01:55.831780  1752 rpcz_store.cc:276] 0709 13:01:54.044847 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:55.818882 (+1774035us) service_pool.cc:224] Handling call
0709 13:01:55.819193 (+   311us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:55.831132 (+ 11939us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:55.831326 (+   194us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:01:55.834049  1752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55840 (request call id 2904) took 1760 ms. Trace:
I20260709 13:01:55.834561  1752 rpcz_store.cc:276] 0709 13:01:54.074151 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:55.832615 (+1758464us) service_pool.cc:224] Handling call
0709 13:01:55.833396 (+   781us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:55.833642 (+   246us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:55.833679 (+    37us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:01:55.833739 (+    60us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:01:55.833784 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:55.833853 (+    69us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:55.833984 (+   131us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:01:55.836704  1752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55840 (request call id 2906) took 1742 ms. Trace:
I20260709 13:01:55.821991  2298 leader_election.cc:304] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:01:55.839202  1752 rpcz_store.cc:276] 0709 13:01:54.094510 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:55.835760 (+1741250us) service_pool.cc:224] Handling call
0709 13:01:55.836101 (+   341us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:55.836327 (+   226us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:55.836363 (+    36us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:01:55.836417 (+    54us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:01:55.836464 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:55.836527 (+    63us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:55.836641 (+   114us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:01:55.806156  7976 raft_consensus.cc:493] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
W20260709 13:01:55.836086  7908 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.724s	user 0.013s	sys 0.022s
W20260709 13:01:55.849478  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:55.798523  7600 raft_consensus.cc:515] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:55.804368  7964 raft_consensus.cc:515] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:55.804140  8024 raft_consensus.cc:515] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:55.837802  7915 raft_consensus.cc:3060] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:01:55.859961  7916 raft_consensus.cc:3060] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Advancing to term 18
W20260709 13:01:55.860780  7954 log.cc:927] Time spent T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 2.522s	user 0.015s	sys 0.009s
I20260709 13:01:55.862802  7946 leader_election.cc:290] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:55.862720  7582 raft_consensus.cc:515] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:55.815778  7875 consensus_queue.cc:1059] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 17.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.037s
I20260709 13:01:55.806125  7835 consensus_queue.cc:1048] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.039s
W20260709 13:01:55.864542  7971 log.cc:927] Time spent T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 2.178s	user 0.007s	sys 0.006s
W20260709 13:01:55.865397  1754 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55840 (request call id 2895) took 2474 ms (client timeout 2898 ms). Trace:
W20260709 13:01:55.853873  7908 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.747s	user 0.013s	sys 0.023s
I20260709 13:01:55.847090  7976 raft_consensus.cc:3060] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Advancing to term 19
W20260709 13:01:55.880319  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 58402us
I20260709 13:01:55.881162  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:55.923108  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:55.946329  1751 raft_consensus.cc:1240] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 14. Current term is 15. Ops: [14.11-14.11]
I20260709 13:01:55.953860  1751 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43156 (request call id 1282) took 1843 ms. Trace:
I20260709 13:01:55.953624  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:55.933597  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:55.954619  1602 raft_consensus.cc:2468] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 15.
I20260709 13:01:55.891535  7950 leader_election.cc:290] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 18 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:01:55.895052  1678 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52778 (request call id 2526) took 1618 ms (client timeout 1502 ms). Trace:
I20260709 13:01:55.896628  7964 leader_election.cc:290] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:55.849644  8006 raft_consensus.cc:697] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 15 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:01:55.954223  1751 rpcz_store.cc:276] 0709 13:01:54.110750 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:55.945571 (+1834821us) service_pool.cc:224] Handling call
0709 13:01:55.946034 (+   463us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:55.953568 (+  7534us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:55.953796 (+   228us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:01:55.949208  1754 rpcz_store.cc:269] 0709 13:01:53.391798 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:53.391949 (+   151us) service_pool.cc:224] Handling call
0709 13:01:55.865122 (+2473173us) spinlock_profiling.cc:243] Waited 143 ms on lock 0x7b54000bd454. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:01:55.865305 (+   183us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":329257344}
W20260709 13:01:55.958256  1678 rpcz_store.cc:269] 0709 13:01:54.276568 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:54.288557 (+ 11989us) service_pool.cc:224] Handling call
0709 13:01:55.894841 (+1606284us) spinlock_profiling.cc:243] Waited 210 ms on lock 0x7b540025a154. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:01:55.894955 (+   114us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":483150208}
I20260709 13:01:55.949442  1646 leader_election.cc:304] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:55.955168  2336 raft_consensus.cc:2468] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 12.
I20260709 13:01:55.949951  2330 raft_consensus.cc:2376] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 15. Re-sending same reply.
I20260709 13:01:55.967573  1600 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40166 (request call id 2748) took 1929 ms. Trace:
I20260709 13:01:55.968533  1601 raft_consensus.cc:1275] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 12 index: 10. (term mismatch)
I20260709 13:01:55.969048  1601 pending_rounds.cc:85] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a: Aborting all ops after (but not including) 9
I20260709 13:01:55.969319  1601 pending_rounds.cc:107] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a: Aborting uncommitted NO_OP operation due to leader change: 11.10
I20260709 13:01:55.969615  1601 raft_consensus.cc:2889] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:01:55.963887  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:55.968016  1600 rpcz_store.cc:276] 0709 13:01:54.037886 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:54.042521 (+  4635us) service_pool.cc:224] Handling call
0709 13:01:55.967338 (+1924817us) spinlock_profiling.cc:243] Waited 118 ms on lock 0x7b54002995d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:01:55.967506 (+   168us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":271375616}
I20260709 13:01:55.977881  1673 raft_consensus.cc:1240] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 15. Current term is 16. Ops: [15.9-15.9]
I20260709 13:01:55.914165  7913 leader_election.cc:290] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:55.939905  7856 leader_election.cc:290] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:55.981256  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:55.957154  7686 raft_consensus.cc:493] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:55.906329  7600 leader_election.cc:290] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 15 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:55.984932  1754 raft_consensus.cc:2468] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 15.
I20260709 13:01:55.981446  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:01:55.908377  7967 consensus_meta.cc:287] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.553s	user 0.006s	sys 0.008s
W20260709 13:01:55.992676  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 164706us
I20260709 13:01:55.993815  2299 leader_election.cc:304] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:55.978894  1646 leader_election.cc:304] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
W20260709 13:01:55.985503  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 237068us
I20260709 13:01:55.997120  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:55.997651  1678 raft_consensus.cc:3060] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:01:55.926055  8045 raft_consensus.cc:2749] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 13:01:55.957279  8024 leader_election.cc:290] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:55.988243  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 18 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:56.006102  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:56.007122  1754 raft_consensus.cc:2393] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 17: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:01:55.991401  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:01:56.007871  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 206216us
W20260709 13:01:55.964936  7962 log.cc:927] Time spent T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 2.449s	user 0.009s	sys 0.013s
I20260709 13:01:55.987977  1751 raft_consensus.cc:2468] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 17.
I20260709 13:01:56.008762  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:56.001188  7952 raft_consensus.cc:493] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:55.980904  7727 leader_election.cc:290] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:56.014859  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 18 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:55.974895  7964 raft_consensus.cc:2804] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:01:56.011327  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:56.018746  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:56.035233  1748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55848 (request call id 3003) took 2096 ms. Trace:
I20260709 13:01:56.043417  2330 raft_consensus.cc:3060] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:01:56.029227  1750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55848 (request call id 2993) took 3205 ms. Trace:
I20260709 13:01:56.043370  1748 rpcz_store.cc:276] 0709 13:01:53.938346 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:54.451005 (+512659us) service_pool.cc:224] Handling call
0709 13:01:54.451484 (+   479us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:55.106572 (+655088us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:56.034915 (+928343us) spinlock_profiling.cc:243] Waited 650 ms on lock 0x7b54001c6e54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:01:56.035148 (+   233us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1494557952}
I20260709 13:01:56.044780  1750 rpcz_store.cc:276] 0709 13:01:52.823942 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:52.824160 (+   218us) service_pool.cc:224] Handling call
0709 13:01:52.824552 (+   392us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:52.831028 (+  6476us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:52.831060 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:01:52.840364 (+  9304us) log.cc:844] Serialized 26 byte log entry
0709 13:01:53.168608 (+328244us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:01:53.168688 (+    80us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:53.168766 (+    78us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:55.838682 (+2669916us) spinlock_profiling.cc:243] Waited 103 ms on lock 0x7b4400022b4c. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f4262f9bc85 00007f4262f9c6a2 00007f4265d0e1db 00007f4265d09aaf 00007f4265cfd86e 00007f4265cfcbb6 00007f4265cfc7d8 00007f4265d06221 00007f4265d061d9 00007f4265d06169
0709 13:01:56.028765 (+190083us) raft_consensus.cc:1685] finished
0709 13:01:56.028801 (+    36us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:56.029133 (+   332us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5969,"raft.queue_time_us":15970,"raft.run_cpu_time_us":146,"raft.run_wall_time_us":18201,"spinlock_wait_cycles":237375232,"thread_start_us":313571,"threads_started":1,"wal-append.queue_time_us":440395}
I20260709 13:01:55.939669  8028 raft_consensus.cc:515] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:56.009675  1718 leader_election.cc:304] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:56.108417  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:56.121263  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:56.122375  1750 raft_consensus.cc:2468] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 17.
I20260709 13:01:56.068626  2298 leader_election.cc:304] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:01:56.095799  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:56.095522  2332 raft_consensus.cc:1240] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 12. Current term is 13. Ops: [12.12-12.12]
I20260709 13:01:55.984632  8048 raft_consensus.cc:2804] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 13:01:56.131791  1678 raft_consensus.cc:2468] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 18.
I20260709 13:01:56.127584  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 19 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:56.143669  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:56.155875  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:56.156682  1678 raft_consensus.cc:2468] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 18.
W20260709 13:01:56.169807  1646 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:56.194250  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:56.199368  1751 raft_consensus.cc:2393] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 17: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:01:56.154853  1750 raft_consensus.cc:2468] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 16.
I20260709 13:01:56.202320  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:01:56.096825  8053 log.cc:927] Time spent T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.062s	user 0.002s	sys 0.000s
W20260709 13:01:56.095397  8044 log.cc:927] Time spent T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.063s	user 0.000s	sys 0.001s
I20260709 13:01:56.060074  7964 raft_consensus.cc:493] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:56.010998  7754 raft_consensus.cc:2804] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Leader pre-election won for term 15
W20260709 13:01:56.209918  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 216190us
I20260709 13:01:56.220463  2299 leader_election.cc:304] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:56.211582  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:56.035976  7830 raft_consensus.cc:2749] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20260709 13:01:56.214362  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 145885us
I20260709 13:01:56.016407  7582 leader_election.cc:290] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 16 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:56.243654  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:56.245868  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:56.246614  1678 raft_consensus.cc:3060] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:01:55.979441  7919 raft_consensus.cc:515] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:56.205533  7981 leader_election.cc:290] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:56.261953  2299 leader_election.cc:304] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:56.245265  1751 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55848 (request call id 3005) took 1326 ms. Trace:
I20260709 13:01:56.263669  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:56.271246  1751 rpcz_store.cc:276] 0709 13:01:54.918679 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:56.213597 (+1294918us) service_pool.cc:224] Handling call
0709 13:01:56.244741 (+ 31144us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:56.244945 (+   204us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:56.244969 (+    24us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:01:56.245014 (+    45us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:01:56.245045 (+    31us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:56.245096 (+    51us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:56.245202 (+   106us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:01:56.203349  7913 raft_consensus.cc:493] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:56.021458  7808 consensus_queue.cc:1059] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.200s
I20260709 13:01:56.219261  8045 raft_consensus.cc:2749] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:01:56.283859  2335 raft_consensus.cc:1275] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 12 index: 10. (index mismatch)
I20260709 13:01:56.267709  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:56.264322  8030 raft_consensus.cc:2749] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:01:56.021369  7952 raft_consensus.cc:515] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:56.265978  1646 leader_election.cc:304] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:56.222470  7754 raft_consensus.cc:493] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:56.213850  8056 raft_consensus.cc:493] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:56.218456  7819 raft_consensus.cc:493] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:56.239079  7875 raft_consensus.cc:2804] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:01:56.240427  7964 raft_consensus.cc:3060] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:01:56.299348  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:01:56.300446  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 290823us
W20260709 13:01:56.309713  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 43804.4us
I20260709 13:01:56.241144  8018 raft_consensus.cc:493] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:56.303651  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 16 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:56.311636  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:56.249676  8055 raft_consensus.cc:3060] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:01:56.028290  8006 consensus_queue.cc:237] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.8, Last appended by leader: 8, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:56.311856  1718 leader_election.cc:304] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:56.305452  2334 raft_consensus.cc:2393] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 12: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:01:56.265586  8048 raft_consensus.cc:697] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 13 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:01:56.265369  7835 raft_consensus.cc:2749] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
W20260709 13:01:56.316720  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:56.305020  7869 consensus_meta.cc:287] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.666s	user 0.013s	sys 0.002s
W20260709 13:01:56.292935  8009 consensus_meta.cc:287] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.808s	user 0.019s	sys 0.000s
I20260709 13:01:56.298372  7967 raft_consensus.cc:493] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:56.338404  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:56.312462  7875 raft_consensus.cc:493] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:56.341334  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:56.319540  1751 raft_consensus.cc:3060] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:01:56.290920  7908 raft_consensus.cc:3060] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:01:56.299190  7754 raft_consensus.cc:3060] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Advancing to term 15
W20260709 13:01:56.316910  7816 consensus_meta.cc:287] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.672s	user 0.015s	sys 0.002s
I20260709 13:01:56.318316  7600 raft_consensus.cc:2804] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:01:56.003166  7686 raft_consensus.cc:515] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:56.282503  7582 consensus_queue.cc:1059] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 16.13, Next index: 14, Last known committed idx: 12, Time since last communication: 0.050s
I20260709 13:01:56.355679  1678 raft_consensus.cc:2468] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 18.
I20260709 13:01:56.337673  7812 raft_consensus.cc:3055] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 16 LEADER]: Stepping down as leader of term 16
I20260709 13:01:56.306993  7913 raft_consensus.cc:515] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:56.315737  7819 raft_consensus.cc:515] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:56.367801  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:56.345875  7875 raft_consensus.cc:3060] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:01:56.373035  7686 leader_election.cc:290] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:56.355301  7600 raft_consensus.cc:493] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:56.347080  7967 raft_consensus.cc:515] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:56.312533  8056 raft_consensus.cc:515] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:56.379447  7600 raft_consensus.cc:3060] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:01:56.336236  8018 raft_consensus.cc:515] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:56.338330  7869 raft_consensus.cc:515] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:56.339830  8009 raft_consensus.cc:515] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:56.385012  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:56.386292  1750 raft_consensus.cc:2468] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 18.
I20260709 13:01:56.363348  7812 raft_consensus.cc:740] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:01:56.390997  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:56.397444  8009 leader_election.cc:290] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 13 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:56.398638  1672 raft_consensus.cc:1240] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 12. Current term is 13. Ops: []
I20260709 13:01:56.399536  2337 raft_consensus.cc:2468] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 18.
W20260709 13:01:56.380515  7577 consensus_meta.cc:287] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.957s	user 0.010s	sys 0.011s
I20260709 13:01:56.353916  7816 raft_consensus.cc:515] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:56.365115  7911 consensus_queue.cc:1048] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.026s
I20260709 13:01:56.407634  2330 raft_consensus.cc:2468] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 19.
I20260709 13:01:56.408608  2337 raft_consensus.cc:1240] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 16. Current term is 17. Ops: []
I20260709 13:01:56.411150  2336 raft_consensus.cc:1240] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 18. Current term is 19. Ops: [18.15-18.15]
I20260709 13:01:56.414541  1754 raft_consensus.cc:2410] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for term 18 because replica has last-logged OpId of term: 16 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 7.
W20260709 13:01:56.415974  8054 log.cc:927] Time spent T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.124s	user 0.000s	sys 0.001s
I20260709 13:01:56.416109  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:56.416791  7816 leader_election.cc:290] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 14 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:56.417353  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:56.417806  1680 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52766 (request call id 2988) took 1004 ms. Trace:
I20260709 13:01:56.418200  1680 rpcz_store.cc:276] 0709 13:01:55.416323 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:55.547888 (+131565us) service_pool.cc:224] Handling call
0709 13:01:55.548271 (+   383us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:55.548689 (+   418us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 13:01:55.780280 (+231591us) log.cc:844] Serialized 14 byte log entry
0709 13:01:55.982369 (+202089us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:01:55.983966 (+  1597us) log.cc:844] Serialized 26 byte log entry
0709 13:01:55.984180 (+   214us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:01:55.984223 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:55.985548 (+  1325us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:56.417572 (+432024us) raft_consensus.cc:1685] finished
0709 13:01:56.417593 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:56.417723 (+   130us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2866,"prepare.queue_time_us":10467,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":38,"raft.queue_time_us":13970,"raft.run_cpu_time_us":229,"raft.run_wall_time_us":5554,"thread_start_us":386033,"threads_started":2,"wal-append.queue_time_us":408549}
I20260709 13:01:56.418550  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
W20260709 13:01:56.409174  7968 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.547s	user 0.015s	sys 0.041s
W20260709 13:01:56.421854  7968 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.560s	user 0.015s	sys 0.042s
I20260709 13:01:56.413354  8048 consensus_queue.cc:237] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:56.431169  7952 leader_election.cc:290] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:56.433172  8056 leader_election.cc:290] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:56.433841  8056 raft_consensus.cc:2749] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:01:56.447122  7967 leader_election.cc:290] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:56.447862  7967 raft_consensus.cc:493] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:56.448539  7869 leader_election.cc:290] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 17 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 13:01:56.453446  8060 log.cc:927] Time spent T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.085s	user 0.000s	sys 0.000s
I20260709 13:01:56.404567  8028 leader_election.cc:290] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 15 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:56.456336  1679 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52766 (request call id 2991) took 1039 ms. Trace:
I20260709 13:01:56.456871  1679 rpcz_store.cc:276] 0709 13:01:55.417147 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:55.764532 (+347385us) service_pool.cc:224] Handling call
0709 13:01:55.764896 (+   364us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:55.779241 (+ 14345us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 13:01:55.787553 (+  8312us) log.cc:844] Serialized 14 byte log entry
0709 13:01:55.988574 (+201021us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:01:56.001721 (+ 13147us) log.cc:844] Serialized 26 byte log entry
0709 13:01:56.002153 (+   432us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:01:56.002242 (+    89us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:56.002480 (+   238us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:56.456024 (+453544us) raft_consensus.cc:1685] finished
0709 13:01:56.456062 (+    38us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:56.456247 (+   185us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":14051,"prepare.queue_time_us":9060,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":46,"raft.queue_time_us":5665,"raft.run_cpu_time_us":151,"raft.run_wall_time_us":2094,"spinlock_wait_cycles":34454784,"thread_start_us":163923,"threads_started":1,"wal-append.queue_time_us":543554}
I20260709 13:01:56.473202  1644 leader_election.cc:304] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 13:01:56.431172  8064 log.cc:927] Time spent T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.074s	user 0.000s	sys 0.001s
I20260709 13:01:56.490350  1644 leader_election.cc:304] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:56.450258  7819 leader_election.cc:290] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:56.502555  1644 leader_election.cc:304] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:01:56.472060  8045 raft_consensus.cc:3055] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 13:01:56.516688  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:56.518461  1678 raft_consensus.cc:2468] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 14.
W20260709 13:01:56.488680  7916 consensus_meta.cc:287] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.558s	user 0.010s	sys 0.004s
I20260709 13:01:56.521950  1751 raft_consensus.cc:2468] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 16.
W20260709 13:01:56.529808  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 163490us
W20260709 13:01:56.530032  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 218741us
I20260709 13:01:56.531549  1718 leader_election.cc:304] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:56.410059  7577 raft_consensus.cc:515] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } }
W20260709 13:01:56.525480  1746 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43156 (request call id 1284) took 2380 ms (client timeout 2041 ms). Trace:
W20260709 13:01:56.545486  1746 rpcz_store.cc:269] 0709 13:01:54.144701 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:54.263373 (+118672us) service_pool.cc:224] Handling call
0709 13:01:56.525160 (+2261787us) spinlock_profiling.cc:243] Waited 1.86 s on lock 0x7b5400019ad4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:01:56.525366 (+   206us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4277232512}
I20260709 13:01:56.504622  8009 raft_consensus.cc:2749] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:01:56.448554  7967 raft_consensus.cc:515] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:56.549186  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:56.558612  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:56.561739  1678 raft_consensus.cc:3060] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 11 FOLLOWER]: Advancing to term 13
I20260709 13:01:56.517464  8045 raft_consensus.cc:740] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:01:56.571887  8045 consensus_queue.cc:260] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.13, Last appended by leader: 13, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:56.567219  8009 raft_consensus.cc:2804] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:01:56.529170  7916 raft_consensus.cc:2749] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:01:56.559656  7850 raft_consensus.cc:2749] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:01:56.560915  8056 raft_consensus.cc:2749] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 13:01:56.588814  1748 raft_consensus.cc:2410] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 for term 19 because replica has last-logged OpId of term: 17 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 11.
I20260709 13:01:56.581717  8009 raft_consensus.cc:493] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:56.612308  1751 raft_consensus.cc:1240] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 15 LEADER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 14. Current term is 15. Ops: []
I20260709 13:01:56.555968  8068 consensus_queue.cc:1059] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 12.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.053s
W20260709 13:01:56.617991  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 196936us
I20260709 13:01:56.621191  2301 leader_election.cc:304] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
W20260709 13:01:56.622372  1646 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:56.628396  7921 raft_consensus.cc:2804] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Leader pre-election won for term 16
W20260709 13:01:56.615666  7915 consensus_meta.cc:287] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.645s	user 0.016s	sys 0.000s
I20260709 13:01:56.611123  8009 raft_consensus.cc:3060] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:01:56.632545  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:56.628865  7921 raft_consensus.cc:493] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:56.635218  1678 raft_consensus.cc:2468] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 13.
I20260709 13:01:56.621308  8006 raft_consensus.cc:493] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:56.638827  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:56.640268  2301 leader_election.cc:304] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:56.638520  7921 raft_consensus.cc:3060] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Advancing to term 16
W20260709 13:01:56.648948  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 146514us
I20260709 13:01:56.638321  7913 leader_election.cc:290] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:56.652981  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:56.654592  2301 leader_election.cc:304] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:01:56.644280  8075 raft_consensus.cc:493] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:56.648474  8018 leader_election.cc:290] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:56.623337  1646 leader_election.cc:304] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:56.662635  2301 leader_election.cc:304] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:56.661646  8084 raft_consensus.cc:493] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:56.675076  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:56.675572  8084 raft_consensus.cc:515] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:56.676499  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:56.677851  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:56.656972  8068 raft_consensus.cc:2804] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Leader pre-election won for term 18
W20260709 13:01:56.679602  8069 log.cc:927] Time spent T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.092s	user 0.001s	sys 0.000s
I20260709 13:01:56.681249  8018 consensus_queue.cc:1059] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 16.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
W20260709 13:01:56.681747  8071 log.cc:927] Time spent T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.096s	user 0.000s	sys 0.000s
I20260709 13:01:56.682490  8018 raft_consensus.cc:493] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:01:56.682837  8067 log.cc:927] Time spent T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.151s	user 0.001s	sys 0.000s
I20260709 13:01:56.683121  7987 raft_consensus.cc:493] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:01:56.683005  8018 raft_consensus.cc:515] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:56.683840  7987 raft_consensus.cc:515] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:56.685637  8018 leader_election.cc:290] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:56.681227  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:56.646584  8006 raft_consensus.cc:515] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:56.687402  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:56.688045  8018 raft_consensus.cc:2804] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Leader election won for term 16
I20260709 13:01:56.638021  7915 raft_consensus.cc:515] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:56.692830  1745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55840 (request call id 2902) took 2654 ms. Trace:
I20260709 13:01:56.699777  1752 raft_consensus.cc:1240] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 15. Current term is 16. Ops: []
I20260709 13:01:56.706038  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:56.710678  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:56.711155  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:56.697290  1745 rpcz_store.cc:276] 0709 13:01:54.038039 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:55.810665 (+1772626us) service_pool.cc:224] Handling call
0709 13:01:55.829187 (+ 18522us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:01:55.983627 (+154440us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:55.983659 (+    32us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:01:55.994807 (+ 11148us) log.cc:844] Serialized 52 byte log entry
0709 13:01:56.332582 (+337775us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:01:56.553156 (+220574us) log.cc:844] Serialized 14 byte log entry
0709 13:01:56.564342 (+ 11186us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:56.564462 (+   120us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:56.692495 (+128033us) raft_consensus.cc:1685] finished
0709 13:01:56.692524 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:56.692742 (+   218us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":31481,"prepare.queue_time_us":282948,"prepare.run_cpu_time_us":59,"prepare.run_wall_time_us":2271,"raft.queue_time_us":158369,"raft.run_cpu_time_us":331,"raft.run_wall_time_us":19397,"thread_start_us":544899,"threads_started":3,"wal-append.queue_time_us":411809}
I20260709 13:01:56.669845  8066 raft_consensus.cc:3060] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:01:56.716399  1644 leader_election.cc:304] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:56.719604  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:56.720885  7728 leader_election.cc:290] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:56.721325  1745 raft_consensus.cc:1275] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 14 index: 7. Preceding OpId from leader: term: 18 index: 9. (index mismatch)
I20260709 13:01:56.721997  1745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55848 (request call id 3010) took 1615 ms. Trace:
I20260709 13:01:56.723886  8018 raft_consensus.cc:697] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 16 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:01:56.724460  8083 raft_consensus.cc:493] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:01:56.725067  8083 raft_consensus.cc:515] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:56.725150  8018 consensus_queue.cc:237] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.8, Last appended by leader: 8, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:56.718636  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:56.722388  1745 rpcz_store.cc:276] 0709 13:01:55.106614 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:56.720652 (+1614038us) service_pool.cc:224] Handling call
0709 13:01:56.720973 (+   321us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:56.721721 (+   748us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:56.721938 (+   217us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:01:56.730180  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:56.729089  8090 raft_consensus.cc:2749] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:01:56.701231  8084 leader_election.cc:290] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:56.734061  1746 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55840 (request call id 2910) took 1709 ms. Trace:
I20260709 13:01:56.715848  8077 raft_consensus.cc:2804] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:01:56.737586  1745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55848 (request call id 3012) took 1623 ms. Trace:
I20260709 13:01:56.738487  7967 leader_election.cc:290] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:56.739604  8006 leader_election.cc:290] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:56.744187  1596 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40166 (request call id 2769) took 1170 ms. Trace:
I20260709 13:01:56.752120  1596 rpcz_store.cc:276] 0709 13:01:55.573517 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:55.635802 (+ 62285us) service_pool.cc:224] Handling call
0709 13:01:56.743985 (+1108183us) spinlock_profiling.cc:243] Waited 133 ms on lock 0x7b540028b754. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:01:56.744120 (+   135us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":305757056}
W20260709 13:01:56.744690  7976 consensus_meta.cc:287] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.772s	user 0.007s	sys 0.007s
I20260709 13:01:56.736543  1746 rpcz_store.cc:276] 0709 13:01:55.024296 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:56.546661 (+1522365us) service_pool.cc:224] Handling call
0709 13:01:56.547387 (+   726us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:56.547670 (+   283us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 13:01:56.641057 (+ 93387us) log.cc:844] Serialized 14 byte log entry
0709 13:01:56.733565 (+ 92508us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:01:56.733660 (+    95us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:01:56.733732 (+    72us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:56.733830 (+    98us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:56.733976 (+   146us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":98062,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":45,"raft.queue_time_us":88,"raft.run_cpu_time_us":101,"raft.run_wall_time_us":101,"thread_start_us":172801,"threads_started":2}
W20260709 13:01:56.753157  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 129900us
I20260709 13:01:56.748119  1745 rpcz_store.cc:276] 0709 13:01:55.114637 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:56.736770 (+1622133us) service_pool.cc:224] Handling call
0709 13:01:56.737077 (+   307us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:56.737276 (+   199us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:56.737301 (+    25us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:01:56.737348 (+    47us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:01:56.737389 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:56.737442 (+    53us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:56.737531 (+    89us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:01:56.715150  7577 leader_election.cc:290] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 16 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:56.753597  1718 leader_election.cc:304] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:56.754848  1745 raft_consensus.cc:1240] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 15. Current term is 16. Ops: []
I20260709 13:01:56.755075  1718 leader_election.cc:304] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:56.696346  8075 raft_consensus.cc:515] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:56.756346  1745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55840 (request call id 2916) took 1321 ms. Trace:
I20260709 13:01:56.756848  1718 leader_election.cc:304] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:56.757305  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:56.758901  1745 rpcz_store.cc:276] 0709 13:01:55.434593 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:56.754271 (+1319678us) service_pool.cc:224] Handling call
0709 13:01:56.754635 (+   364us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:56.756083 (+  1448us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:56.756288 (+   205us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:01:56.760401  1745 raft_consensus.cc:1275] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 16 index: 9. Preceding OpId from leader: term: 18 index: 10. (index mismatch)
I20260709 13:01:56.760890  1745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55840 (request call id 2917) took 1323 ms. Trace:
I20260709 13:01:56.761137  1745 rpcz_store.cc:276] 0709 13:01:55.437983 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:56.759885 (+1321902us) service_pool.cc:224] Handling call
0709 13:01:56.760093 (+   208us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:56.760662 (+   569us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:56.760850 (+   188us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:01:56.764564  8056 raft_consensus.cc:2749] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:01:56.762342  8006 raft_consensus.cc:2749] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:01:56.763594  1718 leader_election.cc:304] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 13:01:56.768321  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:56.748054  8077 raft_consensus.cc:493] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:01:56.779527  7959 log.cc:927] Time spent T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 3.259s	user 0.007s	sys 0.018s
W20260709 13:01:56.781086  8081 log.cc:927] Time spent T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.066s	user 0.001s	sys 0.000s
I20260709 13:01:56.781098  1599 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 1550) took 3947 ms. Trace:
I20260709 13:01:56.782013  1599 rpcz_store.cc:276] 0709 13:01:52.833355 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:52.833526 (+   171us) service_pool.cc:224] Handling call
0709 13:01:52.884507 (+ 50981us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:53.119021 (+234514us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:53.119064 (+    43us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:01:53.144231 (+ 25167us) log.cc:844] Serialized 26 byte log entry
0709 13:01:53.328998 (+184767us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:01:53.329414 (+   416us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:53.329509 (+    95us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:56.773468 (+3443959us) spinlock_profiling.cc:243] Waited 4.91 ms on lock 0x7b4400022b4c. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f4262f9bc85 00007f4262f9c6a2 00007f4265d0e1db 00007f4265d09aaf 00007f4265cfd86e 00007f4265cfcbb6 00007f4265cfc7d8 00007f4265d06221 00007f4265d061d9 00007f4265d06169
0709 13:01:56.780739 (+  7271us) raft_consensus.cc:1685] finished
0709 13:01:56.780761 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:56.781002 (+   241us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3933,"raft.queue_time_us":42265,"raft.run_cpu_time_us":69,"raft.run_wall_time_us":69,"spinlock_wait_cycles":11296128,"thread_start_us":381721,"threads_started":2,"wal-append.queue_time_us":235081}
I20260709 13:01:56.785249  7830 raft_consensus.cc:3055] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:01:56.785720  7830 raft_consensus.cc:740] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:01:56.759207  7967 raft_consensus.cc:2749] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 13:01:56.786712  7830 consensus_queue.cc:260] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:56.789062  7915 leader_election.cc:290] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:56.763039  8055 raft_consensus.cc:2749] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:01:56.798265  7830 raft_consensus.cc:3060] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:01:56.803303  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:56.778301  8006 raft_consensus.cc:2749] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 13:01:56.804610   431 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:01:56.804829  2329 raft_consensus.cc:1240] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 13 LEADER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 12. Current term is 13. Ops: [12.10-12.10]
I20260709 13:01:56.758263  7976 raft_consensus.cc:515] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:56.777902  8077 raft_consensus.cc:3060] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:01:56.801041  7930 leader_election.cc:290] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 15 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:56.786691  7600 raft_consensus.cc:515] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:56.809085  7981 raft_consensus.cc:3055] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 LEADER]: Stepping down as leader of term 13
I20260709 13:01:56.809336  7976 leader_election.cc:290] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 19 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:56.810366  7875 raft_consensus.cc:515] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:56.831696  2333 raft_consensus.cc:2368] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for earlier term 15. Current term is 16.
I20260709 13:01:56.831564  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:56.832537  1599 raft_consensus.cc:3060] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:01:56.833081  7875 leader_election.cc:290] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 16 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:56.814611  7981 raft_consensus.cc:740] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:01:56.835585  7981 consensus_queue.cc:260] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.11, Last appended by leader: 11, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:56.837069  7981 raft_consensus.cc:3060] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:01:56.838773  1603 raft_consensus.cc:1240] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 18. Current term is 19. Ops: [18.13-18.13]
I20260709 13:01:56.815979  8075 leader_election.cc:290] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:56.813102  7908 raft_consensus.cc:2749] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:01:56.813953  8083 leader_election.cc:290] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:56.827863  1596 raft_consensus.cc:1240] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 14. Current term is 15. Ops: []
I20260709 13:01:56.840520  2328 raft_consensus.cc:2468] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 17.
I20260709 13:01:56.830183  1673 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52766 (request call id 2993) took 1316 ms. Trace:
I20260709 13:01:56.841070  1673 rpcz_store.cc:276] 0709 13:01:55.514000 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:55.897353 (+383353us) service_pool.cc:224] Handling call
0709 13:01:55.897814 (+   461us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:55.978496 (+ 80682us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:56.829947 (+851451us) spinlock_profiling.cc:243] Waited 75.3 ms on lock 0x7b5400394ed4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:01:56.830119 (+   172us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":173106048}
I20260709 13:01:56.842551  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:56.843257  1673 raft_consensus.cc:2376] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 17. Re-sending same reply.
I20260709 13:01:56.844331  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:56.845538  1673 raft_consensus.cc:2393] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 13: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:01:56.851325  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:56.852571  1673 raft_consensus.cc:2410] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for term 19 because replica has last-logged OpId of term: 18 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 13.
I20260709 13:01:56.853691  1597 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 1561) took 2929 ms. Trace:
I20260709 13:01:56.854110  1597 rpcz_store.cc:276] 0709 13:01:53.924292 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:53.932635 (+  8343us) service_pool.cc:224] Handling call
0709 13:01:53.949631 (+ 16996us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:54.191038 (+241407us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:54.191088 (+    50us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:01:54.191132 (+    44us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:01:54.191176 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:56.853435 (+2662259us) spinlock_profiling.cc:243] Waited 241 ms on lock 0x7b5400216bd4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:01:56.853510 (+    75us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:56.853634 (+   124us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":554480896}
I20260709 13:01:56.861212  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:56.863351  1596 raft_consensus.cc:3060] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:01:56.863715  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:56.864588  1673 raft_consensus.cc:2468] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 13.
I20260709 13:01:56.867951  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:56.869419  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:56.875435  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:56.888579  1596 raft_consensus.cc:2468] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 16.
W20260709 13:01:56.873994  7970 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.536s	user 0.009s	sys 0.002s
I20260709 13:01:56.828760  7754 raft_consensus.cc:515] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:56.898921  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:56.888111  7921 raft_consensus.cc:515] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:56.900983  7714 leader_election.cc:290] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:56.904492  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:56.912400  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:56.913339  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:56.913628  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:56.866776  8035 consensus_queue.cc:1059] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.028s
I20260709 13:01:56.866580  7908 consensus_queue.cc:1059] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 17.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.028s
W20260709 13:01:56.916543  8043 log.cc:927] Time spent T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 1.056s	user 0.000s	sys 0.004s
I20260709 13:01:56.917557  1679 raft_consensus.cc:2393] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 18: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:01:56.919540  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:01:56.891803  7964 consensus_meta.cc:287] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.565s	user 0.009s	sys 0.006s
I20260709 13:01:56.926646  1599 raft_consensus.cc:2468] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 12.
W20260709 13:01:56.893962  7970 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.559s	user 0.009s	sys 0.003s
I20260709 13:01:56.933077  1673 raft_consensus.cc:2468] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 17.
I20260709 13:01:56.919552  7754 leader_election.cc:290] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 15 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:56.934767  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:56.935856  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:56.936767  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:01:56.937883  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 110362us
I20260709 13:01:56.943830  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 14 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:56.931241  7687 raft_consensus.cc:493] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:56.932152  7964 raft_consensus.cc:515] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:56.934191  1569 leader_election.cc:304] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:01:56.929224  7600 leader_election.cc:290] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 18 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:56.948508  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:56.951370  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:56.958091  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 19 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:56.952960  1569 leader_election.cc:304] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:56.959959  1569 leader_election.cc:304] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:56.963102  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:56.947819  8066 raft_consensus.cc:2749] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:01:56.966904  1674 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52766 (request call id 2989) took 1550 ms. Trace:
I20260709 13:01:56.967394  1674 rpcz_store.cc:276] 0709 13:01:55.416646 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:55.559007 (+142361us) service_pool.cc:224] Handling call
0709 13:01:55.559405 (+   398us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:55.559681 (+   276us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:55.559710 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:01:55.560240 (+   530us) log.cc:844] Serialized 26 byte log entry
0709 13:01:55.761362 (+201122us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:01:55.761417 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:55.761537 (+   120us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:56.915881 (+1154344us) spinlock_profiling.cc:243] Waited 8.7 ms on lock 0x7b4400022b4c. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f4262f9bc85 00007f4262f9c6a2 00007f4265d0e1db 00007f4265d09aaf 00007f4265cfd86e 00007f4265cfcbb6 00007f4265cfc7d8 00007f4265d06221 00007f4265d061d9 00007f4265d06169
0709 13:01:56.965729 (+ 49848us) raft_consensus.cc:1685] finished
0709 13:01:56.965757 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:56.966808 (+  1051us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":20010368,"thread_start_us":200689,"threads_started":1,"wal-append.queue_time_us":285699}
I20260709 13:01:56.973215  1679 raft_consensus.cc:2393] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 18: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:01:56.973802  1569 leader_election.cc:304] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:56.975647  1569 leader_election.cc:304] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
W20260709 13:01:56.952107  8063 thread.cc:641] apply [worker] (thread pool) Time spent creating pthread: real 0.536s	user 0.000s	sys 0.018s
I20260709 13:01:56.979102  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 17 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:56.980696  7968 raft_consensus.cc:493] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:56.963828  8009 raft_consensus.cc:515] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:56.981277  7968 raft_consensus.cc:515] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:56.983302  8009 leader_election.cc:290] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:56.983673  7968 leader_election.cc:290] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:56.986601  7819 raft_consensus.cc:2749] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 13:01:56.963155  7951 raft_consensus.cc:2804] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:01:56.987638  7951 raft_consensus.cc:493] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:56.988021  7951 raft_consensus.cc:3060] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:01:56.992105  1569 leader_election.cc:304] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:01:56.993319  1569 leader_election.cc:304] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 13:01:56.952992  7577 raft_consensus.cc:2749] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:01:56.996465  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:56.961512  7921 leader_election.cc:290] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 16 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:56.965092  7952 raft_consensus.cc:2764] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Leader election decision vote started in defunct term 15: won
I20260709 13:01:56.999658  1599 raft_consensus.cc:1240] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 14. Current term is 15. Ops: [14.12-14.12]
I20260709 13:01:56.998018  1569 leader_election.cc:304] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
W20260709 13:01:56.979403  8063 thread.cc:608] apply [worker] (thread pool) Time spent starting thread: real 0.564s	user 0.000s	sys 0.019s
I20260709 13:01:56.953118  7687 raft_consensus.cc:515] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:56.978727  7728 raft_consensus.cc:2804] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Leader election won for term 19
I20260709 13:01:57.015761  7728 raft_consensus.cc:697] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:01:57.007087  1673 raft_consensus.cc:1240] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 14. Current term is 15. Ops: []
I20260709 13:01:56.999257  7850 raft_consensus.cc:493] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:01:57.010092  1676 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52766 (request call id 2974) took 1995 ms. Trace:
I20260709 13:01:56.999066  7968 raft_consensus.cc:2804] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:01:57.018283  1676 rpcz_store.cc:276] 0709 13:01:55.014938 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:55.015069 (+   131us) service_pool.cc:224] Handling call
0709 13:01:55.027271 (+ 12202us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:55.046696 (+ 19425us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:55.097756 (+ 51060us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:01:55.108877 (+ 11121us) log.cc:844] Serialized 26 byte log entry
0709 13:01:56.941427 (+1832550us) spinlock_profiling.cc:243] Waited 11.1 ms on lock 0x7b180036cac0. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f4263057e07 00007f426beb2ab9 00007f426307469b 00007f426307293f 00007f4263074c02 00007f4265cfc20b 00007f4265d01d23 00007f4265d020d1 00007f426611da0f
0709 13:01:56.999078 (+ 57651us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:01:56.999124 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:56.999372 (+   248us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:57.009612 (+ 10240us) raft_consensus.cc:1685] finished
0709 13:01:57.009647 (+    35us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:57.009995 (+   348us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":80754,"raft.queue_time_us":69410,"raft.run_cpu_time_us":282,"raft.run_wall_time_us":16705,"spinlock_wait_cycles":25541888,"thread_start_us":56356,"threads_started":1,"wal-append.queue_time_us":1883211}
I20260709 13:01:57.004810  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:56.998507  7819 raft_consensus.cc:2804] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Leader election won for term 18
I20260709 13:01:57.020326  7968 raft_consensus.cc:493] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:57.046854  7968 raft_consensus.cc:3060] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:01:57.042591  7819 raft_consensus.cc:697] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:01:57.017366  7577 raft_consensus.cc:2804] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Leader election won for term 18
I20260709 13:01:57.051290  7577 raft_consensus.cc:697] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:01:57.051440  7819 consensus_queue.cc:237] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.11, Last appended by leader: 11, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:57.052330  7577 consensus_queue.cc:237] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.9, Last appended by leader: 9, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:57.053826  1599 raft_consensus.cc:1275] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 13 index: 10. (index mismatch)
I20260709 13:01:57.033629  7850 raft_consensus.cc:515] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:57.040352  2332 raft_consensus.cc:3060] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:01:57.060585  1748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43156 (request call id 1288) took 1107 ms. Trace:
I20260709 13:01:57.061146  1748 rpcz_store.cc:276] 0709 13:01:55.952904 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:56.050618 (+ 97714us) service_pool.cc:224] Handling call
0709 13:01:57.060284 (+1009666us) spinlock_profiling.cc:243] Waited 434 ms on lock 0x7b54001d5bd4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:01:57.060490 (+   206us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":997065472}
W20260709 13:01:57.063480  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 43824.4us
I20260709 13:01:57.064152  8036 raft_consensus.cc:3055] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 13:01:57.099998  1748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43156 (request call id 1291) took 1144 ms. Trace:
I20260709 13:01:57.100833  1748 rpcz_store.cc:276] 0709 13:01:55.955407 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:57.062422 (+1107015us) service_pool.cc:224] Handling call
0709 13:01:57.099356 (+ 36934us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:57.099588 (+   232us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:57.099619 (+    31us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:01:57.099670 (+    51us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:01:57.099707 (+    37us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:57.099765 (+    58us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:57.099917 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:01:57.106297  7812 consensus_queue.cc:260] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.11, Last appended by leader: 11, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:57.107666  1671 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52778 (request call id 2517) took 2834 ms (client timeout 1845 ms). Trace:
I20260709 13:01:57.108058  7812 raft_consensus.cc:3060] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Advancing to term 17
W20260709 13:01:57.108322  1671 rpcz_store.cc:269] 0709 13:01:54.273579 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:54.273706 (+   127us) service_pool.cc:224] Handling call
0709 13:01:57.107311 (+2833605us) spinlock_profiling.cc:243] Waited 1.35 s on lock 0x7b5400092754. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:01:57.107528 (+   217us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3113488768}
I20260709 13:01:57.112528  1677 raft_consensus.cc:1275] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 17 index: 10. Preceding OpId from leader: term: 18 index: 10. (term mismatch)
I20260709 13:01:57.112887  1677 pending_rounds.cc:85] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d: Aborting all ops after (but not including) 9
I20260709 13:01:57.113097  1677 pending_rounds.cc:107] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d: Aborting uncommitted NO_OP operation due to leader change: 17.10
I20260709 13:01:57.113315  1677 raft_consensus.cc:2889] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:01:57.110424  1671 raft_consensus.cc:1240] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 17. Current term is 18. Ops: []
I20260709 13:01:57.114905  1748 raft_consensus.cc:1275] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 14 index: 12. (index mismatch)
I20260709 13:01:57.115543  1748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43156 (request call id 1294) took 1159 ms. Trace:
I20260709 13:01:57.115790  1748 rpcz_store.cc:276] 0709 13:01:55.956100 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:57.101970 (+1145870us) service_pool.cc:224] Handling call
0709 13:01:57.102286 (+   316us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:57.115246 (+ 12960us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:57.115492 (+   246us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:01:57.033815  7976 raft_consensus.cc:493] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:57.119601  1671 raft_consensus.cc:1240] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 17. Current term is 18. Ops: [17.11-17.11]
I20260709 13:01:57.125463  1671 raft_consensus.cc:1275] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 19 index: 12. (index mismatch)
I20260709 13:01:57.064730  8036 raft_consensus.cc:740] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:01:57.135253  7968 raft_consensus.cc:515] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:57.032464  8077 raft_consensus.cc:515] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:57.126358  7976 raft_consensus.cc:515] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:57.140633  1671 raft_consensus.cc:1240] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 17. Current term is 18. Ops: [17.10-17.10]
I20260709 13:01:57.136245  7951 raft_consensus.cc:515] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:57.145046  2332 raft_consensus.cc:2468] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 19.
I20260709 13:01:57.146588  1599 raft_consensus.cc:1240] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 18. Current term is 19. Ops: []
I20260709 13:01:57.143759  7728 consensus_queue.cc:237] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 17.14, Last appended by leader: 14, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:57.154017  7968 leader_election.cc:290] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 19 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:57.157586  1678 raft_consensus.cc:2468] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 17.
I20260709 13:01:57.151265  1644 leader_election.cc:304] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:01:57.159551  7987 leader_election.cc:290] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:57.160209  7987 raft_consensus.cc:2804] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Leader election won for term 19
I20260709 13:01:57.160761  8036 consensus_queue.cc:260] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.12, Last appended by leader: 12, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:57.162384  8036 raft_consensus.cc:3060] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:01:57.167137  7951 leader_election.cc:290] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:57.171129  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 14 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:57.172709  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:57.172741  1748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55848 (request call id 3019) took 1168 ms. Trace:
I20260709 13:01:57.175848  1748 rpcz_store.cc:276] 0709 13:01:56.004201 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:57.116363 (+1112162us) service_pool.cc:224] Handling call
0709 13:01:57.116588 (+   225us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:57.116777 (+   189us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 13:01:57.127005 (+ 10228us) log.cc:844] Serialized 14 byte log entry
0709 13:01:57.172387 (+ 45382us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:01:57.172450 (+    63us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:01:57.172501 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:57.172569 (+    68us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:57.172677 (+   108us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":10150,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":44,"raft.queue_time_us":448,"thread_start_us":44995,"threads_started":1}
I20260709 13:01:57.178118  1748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55848 (request call id 3020) took 1166 ms. Trace:
I20260709 13:01:57.178403  1748 rpcz_store.cc:276] 0709 13:01:56.011840 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:57.177430 (+1165590us) service_pool.cc:224] Handling call
0709 13:01:57.177695 (+   265us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:57.177872 (+   177us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:57.177907 (+    35us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:01:57.177935 (+    28us) raft_consensus.cc:1645] Marking committed up to 14
0709 13:01:57.177955 (+    20us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:57.177990 (+    35us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:57.178067 (+    77us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:01:57.181635  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:57.182858  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:57.160755  7987 raft_consensus.cc:697] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:01:57.185452  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:57.184379  1676 raft_consensus.cc:1275] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 13 index: 7. Preceding OpId from leader: term: 14 index: 7. (term mismatch)
I20260709 13:01:57.187281  7987 consensus_queue.cc:237] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.12, Last appended by leader: 12, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:57.189456  1676 pending_rounds.cc:85] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d: Aborting all ops after (but not including) 6
I20260709 13:01:57.190582  1676 pending_rounds.cc:107] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d: Aborting uncommitted NO_OP operation due to leader change: 13.7
I20260709 13:01:57.190860  1676 raft_consensus.cc:2889] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:01:57.191016  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:57.191429  1754 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55840 (request call id 2919) took 1223 ms. Trace:
I20260709 13:01:57.191833  1754 rpcz_store.cc:276] 0709 13:01:55.968135 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:56.014560 (+ 46425us) service_pool.cc:224] Handling call
0709 13:01:57.191224 (+1176664us) spinlock_profiling.cc:243] Waited 371 ms on lock 0x7b54001c6954. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:01:57.191360 (+   136us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":853309312}
I20260709 13:01:57.197249  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:57.197554  1675 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52766 (request call id 2967) took 3636 ms. Trace:
I20260709 13:01:57.197953  1675 rpcz_store.cc:276] 0709 13:01:53.560867 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:53.561810 (+   943us) service_pool.cc:224] Handling call
0709 13:01:53.696440 (+134630us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:54.794183 (+1097743us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:57.197284 (+2403101us) spinlock_profiling.cc:243] Waited 1.09 s on lock 0x7b54000ff354. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:01:57.197489 (+   205us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2517062016}
I20260709 13:01:57.198626  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:57.184298  7964 leader_election.cc:290] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 18 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:57.204977  2336 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39726 (request call id 1434) took 1095 ms. Trace:
I20260709 13:01:57.205387  2336 rpcz_store.cc:276] 0709 13:01:56.109397 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:56.109607 (+   210us) service_pool.cc:224] Handling call
0709 13:01:56.410880 (+301273us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:56.413415 (+  2535us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:57.204764 (+791349us) spinlock_profiling.cc:243] Waited 301 ms on lock 0x7b540083f750. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661618ad 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:01:57.204906 (+   142us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":691693312}
I20260709 13:01:57.209424  1749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43156 (request call id 1277) took 3691 ms. Trace:
I20260709 13:01:57.209952  1749 rpcz_store.cc:276] 0709 13:01:53.518051 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:53.685243 (+167192us) service_pool.cc:224] Handling call
0709 13:01:55.866590 (+2181347us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:55.867081 (+   491us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:55.867123 (+    42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:01:55.867775 (+   652us) log.cc:844] Serialized 26 byte log entry
0709 13:01:56.300864 (+433089us) raft_consensus.cc:1645] Marking committed up to 13
0709 13:01:56.300912 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:56.300994 (+    82us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:56.600320 (+299326us) raft_consensus.cc:1685] finished
0709 13:01:56.600347 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:57.209063 (+608716us) spinlock_profiling.cc:243] Waited 2.16 s on lock 0x7b54000bd450. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661618ad 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:01:57.209327 (+   264us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4215,"raft.queue_time_us":173103,"raft.run_cpu_time_us":316,"raft.run_wall_time_us":64775,"spinlock_wait_cycles":4966098688,"thread_start_us":432470,"threads_started":1,"wal-append.queue_time_us":488354}
I20260709 13:01:57.215713  7976 leader_election.cc:290] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:57.217185  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:57.217897  2328 raft_consensus.cc:2468] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 17.
I20260709 13:01:57.218892  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:57.219343  1644 leader_election.cc:304] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:57.220201  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:01:57.222257  2299 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:57.220369  7964 raft_consensus.cc:2804] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:01:57.223183  2328 raft_consensus.cc:1240] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 13. Current term is 14. Ops: [13.11-13.11]
I20260709 13:01:57.223276  7964 raft_consensus.cc:493] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:57.223731  7964 raft_consensus.cc:3060] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:01:57.225991  2334 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49082 (request call id 1135) took 1346 ms. Trace:
I20260709 13:01:57.221320  1644 leader_election.cc:304] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:57.226382  2334 rpcz_store.cc:276] 0709 13:01:55.880037 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:55.925286 (+ 45249us) service_pool.cc:224] Handling call
0709 13:01:57.225729 (+1300443us) spinlock_profiling.cc:243] Waited 326 ms on lock 0x7b5400507b54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:01:57.225896 (+   167us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":750801152}
I20260709 13:01:57.221625  8077 leader_election.cc:290] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 18 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:57.221400  2333 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39726 (request call id 1437) took 1004 ms. Trace:
I20260709 13:01:57.229509  1569 leader_election.cc:304] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:01:57.229513  2333 rpcz_store.cc:276] 0709 13:01:56.217101 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:56.217297 (+   196us) service_pool.cc:224] Handling call
0709 13:01:57.221123 (+1003826us) spinlock_profiling.cc:243] Waited 586 ms on lock 0x7b5400627ad4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:01:57.221314 (+   191us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1347333888}
I20260709 13:01:57.237550  2337 raft_consensus.cc:1240] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 11. Current term is 12. Ops: []
I20260709 13:01:57.227962  8023 raft_consensus.cc:3060] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:01:57.239215  7728 raft_consensus.cc:2749] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:01:57.246850  7812 consensus_queue.cc:1059] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 17.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.024s
I20260709 13:01:57.275789  1746 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55840 (request call id 2915) took 1877 ms. Trace:
I20260709 13:01:57.281035  1746 rpcz_store.cc:276] 0709 13:01:55.398296 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:56.753675 (+1355379us) service_pool.cc:224] Handling call
0709 13:01:56.762327 (+  8652us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:56.767539 (+  5212us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:56.767573 (+    34us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:01:56.767615 (+    42us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:01:56.767660 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:57.275437 (+507777us) spinlock_profiling.cc:243] Waited 4.99 ms on lock 0x7b54001c8254. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:01:57.275507 (+    70us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:57.275700 (+   193us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":11481472}
I20260709 13:01:57.288493  1746 raft_consensus.cc:1275] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 16 index: 13. Preceding OpId from leader: term: 17 index: 14. (index mismatch)
I20260709 13:01:57.279078  7850 leader_election.cc:290] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:57.293362  1601 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 2759) took 2193 ms. Trace:
I20260709 13:01:57.293973  1601 rpcz_store.cc:276] 0709 13:01:55.099774 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:55.103057 (+  3283us) service_pool.cc:224] Handling call
0709 13:01:55.968045 (+864988us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:55.970181 (+  2136us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:57.293080 (+1322899us) spinlock_profiling.cc:243] Waited 864 ms on lock 0x7b54002995d0. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661618ad 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:01:57.293270 (+   190us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1988287872}
W20260709 13:01:57.300834  1488 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet 983621a6fc5b4c9ea0159a648dfe7a36 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER failed with leader 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 3b879799899a499c8d409ee9f1277cd1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 17 committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 13:01:57.303292  7728 raft_consensus.cc:493] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:57.304059  8068 raft_consensus.cc:493] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:57.382665  8010 raft_consensus.cc:493] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:57.383474  8010 raft_consensus.cc:515] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:57.386150  8010 leader_election.cc:290] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:57.360752  7577 raft_consensus.cc:493] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:57.387380  8068 raft_consensus.cc:3060] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:01:57.303972  7728 raft_consensus.cc:515] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:01:57.397753  8108 log.cc:927] Time spent T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.112s	user 0.001s	sys 0.000s
W20260709 13:01:57.442369  8111 log.cc:927] Time spent T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.064s	user 0.000s	sys 0.000s
I20260709 13:01:57.393062  7850 raft_consensus.cc:493] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:01:57.442253  8107 log.cc:927] Time spent T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.166s	user 0.001s	sys 0.000s
W20260709 13:01:57.440222  8112 log.cc:927] Time spent T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.054s	user 0.000s	sys 0.001s
I20260709 13:01:57.436085  7812 raft_consensus.cc:493] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:57.454911  2301 leader_election.cc:304] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:01:57.447890  7850 raft_consensus.cc:515] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:57.474177  8031 consensus_queue.cc:1048] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 13:01:57.476377  7850 leader_election.cc:290] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:01:57.477200  8093 log.cc:927] Time spent T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.573s	user 0.000s	sys 0.003s
I20260709 13:01:57.432116  7577 raft_consensus.cc:515] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:57.478093  7912 consensus_queue.cc:1048] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 13:01:57.481980  2335 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39726 (request call id 1436) took 1281 ms. Trace:
I20260709 13:01:57.482649  2335 rpcz_store.cc:276] 0709 13:01:56.200761 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:56.200995 (+   234us) service_pool.cc:224] Handling call
0709 13:01:56.213106 (+ 12111us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:56.284361 (+ 71255us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:57.479233 (+1194872us) spinlock_profiling.cc:243] Waited 70.3 ms on lock 0x7b54006beed4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:01:57.481892 (+  2659us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":161749888}
I20260709 13:01:57.513865  7728 leader_election.cc:290] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:57.479079  7991 raft_consensus.cc:2749] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:01:57.518034  1748 raft_consensus.cc:1240] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 16. Current term is 17. Ops: []
I20260709 13:01:57.519596  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:57.468858  7945 consensus_queue.cc:1048] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.048s
I20260709 13:01:57.525535  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:57.526387  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:57.526875  1748 raft_consensus.cc:2468] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 13.
I20260709 13:01:57.527132  1746 raft_consensus.cc:2468] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 17.
I20260709 13:01:57.528261  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:57.481559  7812 raft_consensus.cc:515] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } }
I20260709 13:01:57.532148  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:57.534837  2330 raft_consensus.cc:1275] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 13 index: 12. Preceding OpId from leader: term: 14 index: 13. (index mismatch)
I20260709 13:01:57.534632  1748 raft_consensus.cc:2468] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 14.
I20260709 13:01:57.538165  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:57.524317  7577 leader_election.cc:290] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:01:57.534665  8113 log.cc:927] Time spent T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.098s	user 0.001s	sys 0.000s
I20260709 13:01:57.528204  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:57.550199  1746 raft_consensus.cc:3060] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:01:57.548456  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:57.556162  1748 raft_consensus.cc:3060] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:01:57.556743  1646 leader_election.cc:304] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:01:57.561062  1646 leader_election.cc:304] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:57.562537  1646 leader_election.cc:304] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:57.566960  1718 leader_election.cc:304] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 13:01:57.563417  1646 leader_election.cc:304] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:57.569001  8009 raft_consensus.cc:2804] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:01:57.574185  1598 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40154 (request call id 2676) took 2182 ms. Trace:
I20260709 13:01:57.574920  1753 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43156 (request call id 1280) took 3619 ms. Trace:
I20260709 13:01:57.574638  1598 rpcz_store.cc:276] 0709 13:01:55.391499 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:55.727164 (+335665us) service_pool.cc:224] Handling call
0709 13:01:55.727619 (+   455us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:55.823385 (+ 95766us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:57.573860 (+1750475us) spinlock_profiling.cc:243] Waited 94.9 ms on lock 0x7b540029a4d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:01:57.574078 (+   218us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":218317568}
I20260709 13:01:57.575449  1753 rpcz_store.cc:276] 0709 13:01:53.955139 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:54.959754 (+1004615us) service_pool.cc:224] Handling call
0709 13:01:54.960150 (+   396us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:55.035286 (+ 75136us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:55.035322 (+    36us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:01:55.035364 (+    42us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:01:55.035398 (+    34us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:57.574632 (+2539234us) spinlock_profiling.cc:243] Waited 74.9 ms on lock 0x7b54001d51d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:01:57.574695 (+    63us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:57.574814 (+   119us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":172302592}
I20260709 13:01:57.576730  1718 leader_election.cc:304] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:57.580013  7946 raft_consensus.cc:3055] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 16 LEADER]: Stepping down as leader of term 16
I20260709 13:01:57.580466  7946 raft_consensus.cc:740] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:01:57.581463  7946 consensus_queue.cc:260] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 16.13, Last appended by leader: 13, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:57.584975  1752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43156 (request call id 1281) took 3477 ms. Trace:
I20260709 13:01:57.585430  7662 raft_consensus.cc:493] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
W20260709 13:01:57.564320  8109 log.cc:927] Time spent T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.188s	user 0.001s	sys 0.000s
I20260709 13:01:57.583148  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:57.588284  1718 leader_election.cc:304] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:57.585371  1752 rpcz_store.cc:276] 0709 13:01:54.107586 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:55.840922 (+1733336us) service_pool.cc:224] Handling call
0709 13:01:55.841551 (+   629us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:56.700214 (+858663us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:57.584726 (+884512us) spinlock_profiling.cc:243] Waited 858 ms on lock 0x7b54000e4554. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:01:57.584910 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1973059968}
I20260709 13:01:57.590006  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:57.561901  7850 raft_consensus.cc:2749] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:01:57.569164  7600 raft_consensus.cc:2804] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:01:57.606516  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:57.607399  1753 raft_consensus.cc:2393] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 19: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:01:57.609195  7945 raft_consensus.cc:493] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:57.609772  7945 raft_consensus.cc:515] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:57.612134  7945 leader_election.cc:290] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:57.612931  7945 raft_consensus.cc:2804] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:01:57.614595  7945 raft_consensus.cc:493] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:57.615074  7945 raft_consensus.cc:3060] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Advancing to term 15
W20260709 13:01:57.621764  1571 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:57.621169  7910 consensus_queue.cc:1059] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 16.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 13:01:57.622943  7915 raft_consensus.cc:3055] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 13:01:57.623347  7915 raft_consensus.cc:740] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:01:57.623474  1754 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43156 (request call id 1296) took 1009 ms. Trace:
I20260709 13:01:57.574270  8009 raft_consensus.cc:493] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:57.624311  7915 consensus_queue.cc:260] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 17.10, Last appended by leader: 10, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:57.624905  8009 raft_consensus.cc:3060] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:01:57.625718  7915 raft_consensus.cc:3060] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:01:57.623859  1754 rpcz_store.cc:276] 0709 13:01:56.614722 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:57.192608 (+577886us) service_pool.cc:224] Handling call
0709 13:01:57.192855 (+   247us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:57.193175 (+   320us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:57.193222 (+    47us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:01:57.196251 (+  3029us) log.cc:844] Serialized 26 byte log entry
0709 13:01:57.277130 (+ 80879us) raft_consensus.cc:1645] Marking committed up to 13
0709 13:01:57.277175 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:57.277269 (+    94us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:57.623197 (+345928us) raft_consensus.cc:1685] finished
0709 13:01:57.623218 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:57.623401 (+   183us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":89,"raft.run_cpu_time_us":139,"raft.run_wall_time_us":2271,"thread_start_us":37984,"threads_started":1,"wal-append.queue_time_us":167573}
I20260709 13:01:57.586050  7662 raft_consensus.cc:515] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:57.630872  7812 leader_election.cc:290] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:57.631481  7946 raft_consensus.cc:3060] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:01:57.633615  7662 leader_election.cc:290] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:57.633486  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:57.591202  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:57.635856  1752 raft_consensus.cc:2468] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 18.
I20260709 13:01:57.591152  1718 leader_election.cc:304] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 13:01:57.639139  1571 connection.cc:256] RPC call timeout handler was delayed by -1.7016363870000077s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2898s for a response
I20260709 13:01:57.595000  8096 raft_consensus.cc:2804] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 13:01:57.591339  7976 raft_consensus.cc:2749] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:01:57.598587  8003 raft_consensus.cc:2804] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Leader pre-election won for term 14
W20260709 13:01:57.640592  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 49556.4us
I20260709 13:01:57.647691  1718 leader_election.cc:304] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:57.638260  1746 raft_consensus.cc:2468] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 18.
I20260709 13:01:57.607193  7600 raft_consensus.cc:493] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:01:57.649289  8096 raft_consensus.cc:697] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 17 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:01:57.649595  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 19 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:57.652559  1753 raft_consensus.cc:2468] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 18.
I20260709 13:01:57.598568  8077 consensus_queue.cc:1059] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 15.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.047s
I20260709 13:01:57.613698  8068 raft_consensus.cc:515] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:57.642805  7577 raft_consensus.cc:493] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:57.659657  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:57.677114  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:57.660960  1599 raft_consensus.cc:1240] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 11. Current term is 12. Ops: []
I20260709 13:01:57.661904  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:57.678406  1753 raft_consensus.cc:2468] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 14.
I20260709 13:01:57.662997  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 19 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:57.646306  8003 raft_consensus.cc:493] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:01:57.664112  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:57.683383  1603 raft_consensus.cc:2393] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 17: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:01:57.665580  1600 raft_consensus.cc:1275] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 14 index: 7. (index mismatch)
I20260709 13:01:57.666714  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:57.685458  1602 raft_consensus.cc:2393] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 19: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:01:57.688629  1602 raft_consensus.cc:1275] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 15 index: 9. (index mismatch)
I20260709 13:01:57.688550  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:57.668640  8009 raft_consensus.cc:515] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:57.669353  1571 leader_election.cc:304] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:01:57.651592  7600 raft_consensus.cc:3060] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:01:57.645273  7910 raft_consensus.cc:2749] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:01:57.676127  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:57.657047  7728 raft_consensus.cc:493] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:57.663211  1748 raft_consensus.cc:2468] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 19.
I20260709 13:01:57.676658  7976 raft_consensus.cc:2749] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20260709 13:01:57.699083  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 51483.5us
I20260709 13:01:57.681592  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:57.698812  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:57.699932  1718 leader_election.cc:304] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:57.700182  1599 raft_consensus.cc:2393] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 15: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:01:57.668658  8077 raft_consensus.cc:3055] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 13:01:57.704262  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:57.691715  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:57.692673  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:57.705040  1674 raft_consensus.cc:3060] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:01:57.705971  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:57.707080  1597 raft_consensus.cc:2468] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 15.
I20260709 13:01:57.692775  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:57.707974  1748 raft_consensus.cc:2468] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 17.
I20260709 13:01:57.685036  8003 raft_consensus.cc:3060] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:01:57.690901  1643 leader_election.cc:304] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
W20260709 13:01:57.691715  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 128331us
W20260709 13:01:57.712596  1646 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:57.709085  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:57.698782  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:57.715463  1646 leader_election.cc:304] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:01:57.716564  1643 leader_election.cc:304] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:01:57.716753  1646 leader_election.cc:304] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:01:57.686751  7850 raft_consensus.cc:493] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:01:57.686926  7812 raft_consensus.cc:493] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:57.717919  1675 raft_consensus.cc:3055] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:01:57.720060  1675 raft_consensus.cc:740] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:01:57.701390  7981 raft_consensus.cc:2749] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:01:57.653743  8096 consensus_queue.cc:237] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.11, Last appended by leader: 11, Current term: 17, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:57.724892  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 16 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:57.726217  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:57.674355  7964 raft_consensus.cc:515] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:57.682634  7577 raft_consensus.cc:515] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:57.703768  8077 raft_consensus.cc:740] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:01:57.730576  1718 leader_election.cc:304] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:57.727190  1675 consensus_queue.cc:260] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.9, Last appended by leader: 9, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:57.732975  1675 raft_consensus.cc:3060] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:01:57.734908  8116 raft_consensus.cc:2804] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:01:57.735400  8116 raft_consensus.cc:493] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:57.735750  8116 raft_consensus.cc:3060] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:01:57.738415  7577 leader_election.cc:290] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:57.739238  1600 raft_consensus.cc:1275] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 13 index: 12. Preceding OpId from leader: term: 14 index: 13. (index mismatch)
I20260709 13:01:57.720108  7910 raft_consensus.cc:2749] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
W20260709 13:01:57.740064  1604 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40154 (request call id 2679) took 2341 ms (client timeout 1591 ms). Trace:
I20260709 13:01:57.740092  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:01:57.740556  1604 rpcz_store.cc:269] 0709 13:01:55.392247 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:55.420942 (+ 28695us) service_pool.cc:224] Handling call
0709 13:01:57.733862 (+2312920us) spinlock_profiling.cc:243] Waited 304 ms on lock 0x7b540029b3d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:01:57.734001 (+   139us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":700087552}
I20260709 13:01:57.741386  2337 raft_consensus.cc:2468] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 14.
I20260709 13:01:57.743124  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:57.744236  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:57.744872  8003 raft_consensus.cc:515] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:57.747259  8068 leader_election.cc:290] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 18 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:57.747805  8009 leader_election.cc:290] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 18 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:57.748654  8009 raft_consensus.cc:2749] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:01:57.749275  8009 raft_consensus.cc:2749] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:01:57.750247  1675 raft_consensus.cc:1275] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 15 index: 9. (term mismatch)
I20260709 13:01:57.750491  8003 leader_election.cc:290] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 14 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:57.750723  1675 pending_rounds.cc:85] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d: Aborting all ops after (but not including) 8
I20260709 13:01:57.751160  1675 pending_rounds.cc:107] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d: Aborting uncommitted NO_OP operation due to leader change: 14.9
I20260709 13:01:57.751461  1675 raft_consensus.cc:2889] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:01:57.752357  7976 raft_consensus.cc:2749] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 13:01:57.752619  8009 raft_consensus.cc:2749] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 13:01:57.754105  1718 leader_election.cc:304] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:57.755760  1718 leader_election.cc:304] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:57.758666  7981 raft_consensus.cc:493] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:57.759228  7981 raft_consensus.cc:515] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:57.760183  7976 raft_consensus.cc:2749] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:01:57.761076  7976 raft_consensus.cc:2749] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:01:57.761390  7981 leader_election.cc:290] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:57.763101  8077 consensus_queue.cc:260] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.11, Last appended by leader: 11, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:57.764459  8077 raft_consensus.cc:3060] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:01:57.764714  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 14 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:57.726583  7812 raft_consensus.cc:515] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:57.766076  7964 leader_election.cc:290] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 18 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:57.780439  1753 raft_consensus.cc:2410] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for term 18 because replica has last-logged OpId of term: 17 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 9.
I20260709 13:01:57.782377  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:57.783139  2337 raft_consensus.cc:3060] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Advancing to term 18
W20260709 13:01:57.784394  1750 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55848 (request call id 3024) took 1544 ms (client timeout 1533 ms). Trace:
I20260709 13:01:57.785151  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:57.786343  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:57.786923  1753 raft_consensus.cc:3060] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:01:57.788695  1679 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52778 (request call id 2542) took 1134 ms. Trace:
W20260709 13:01:57.789179  2300 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:57.789078  1679 rpcz_store.cc:276] 0709 13:01:56.654226 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:56.948269 (+294043us) service_pool.cc:224] Handling call
0709 13:01:57.788464 (+840195us) spinlock_profiling.cc:243] Waited 8.11 ms on lock 0x7b54000d9654. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:01:57.788626 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":18652032}
I20260709 13:01:57.790287  1673 raft_consensus.cc:1240] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 17. Current term is 18. Ops: [17.14-17.14]
I20260709 13:01:57.792379  1673 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57092 (request call id 1405) took 1465 ms. Trace:
I20260709 13:01:57.792747  1673 rpcz_store.cc:276] 0709 13:01:56.327066 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:57.018728 (+691662us) service_pool.cc:224] Handling call
0709 13:01:57.790010 (+771282us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:57.790740 (+   730us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:57.792157 (+  1417us) spinlock_profiling.cc:243] Waited 754 ms on lock 0x7b54000d9650. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661618ad 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:01:57.792314 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1734560512}
I20260709 13:01:57.767308  1671 raft_consensus.cc:2468] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 19.
I20260709 13:01:57.726698  7850 raft_consensus.cc:515] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:57.796173  1569 leader_election.cc:304] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:01:57.707865  7728 raft_consensus.cc:515] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:57.796311  7946 consensus_queue.cc:1048] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 13:01:57.798295  7728 leader_election.cc:290] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:57.731110  8023 raft_consensus.cc:2749] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:01:57.769927  8035 raft_consensus.cc:3055] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:01:57.812639  8035 raft_consensus.cc:740] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:01:57.813347  7987 raft_consensus.cc:3055] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 LEADER]: Stepping down as leader of term 18
I20260709 13:01:57.813735  7987 raft_consensus.cc:740] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:01:57.813577  8035 consensus_queue.cc:260] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.12, Last appended by leader: 12, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:57.814795  8035 raft_consensus.cc:3060] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:01:57.814611  7987 consensus_queue.cc:260] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.15, Last appended by leader: 15, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:57.815083  7964 raft_consensus.cc:493] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:01:57.815670  7987 raft_consensus.cc:3060] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:01:57.816157  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 14 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:57.817327  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:57.796482  7850 leader_election.cc:290] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:57.773525  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 20 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:57.819105  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:57.819694  1680 raft_consensus.cc:3060] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:01:57.784474  1604 raft_consensus.cc:1240] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 13. Current term is 14. Ops: []
W20260709 13:01:57.784894  1750 rpcz_store.cc:269] 0709 13:01:56.239820 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:56.267323 (+ 27503us) service_pool.cc:224] Handling call
0709 13:01:57.784199 (+1516876us) spinlock_profiling.cc:243] Waited 98.5 ms on lock 0x7b5400019fd4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:01:57.784317 (+   118us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":226521472}
I20260709 13:01:57.821638  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:57.822193  1750 raft_consensus.cc:3060] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:01:57.822979  1678 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57092 (request call id 1408) took 1482 ms. Trace:
I20260709 13:01:57.823347  1678 rpcz_store.cc:276] 0709 13:01:56.340586 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:56.638537 (+297951us) service_pool.cc:224] Handling call
0709 13:01:57.822799 (+1184262us) spinlock_profiling.cc:243] Waited 515 ms on lock 0x7b5400090954. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:01:57.822919 (+   120us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1184091904}
I20260709 13:01:57.824409  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:57.824713  2337 raft_consensus.cc:2468] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 18.
I20260709 13:01:57.800498  2332 raft_consensus.cc:1240] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 18. Current term is 19. Ops: [18.15-18.15]
I20260709 13:01:57.834506  2330 raft_consensus.cc:1240] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 17. Current term is 18. Ops: []
I20260709 13:01:57.762722  1718 leader_election.cc:304] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
W20260709 13:01:57.836647  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 74018.7us
I20260709 13:01:57.807257  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:57.807621  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 14 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:57.799045  7728 raft_consensus.cc:2804] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Leader election won for term 13
I20260709 13:01:57.811133  7952 consensus_queue.cc:1059] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 13.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20260709 13:01:57.812021  8039 log.cc:927] Time spent T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 1.743s	user 0.008s	sys 0.009s
I20260709 13:01:57.812187  8023 raft_consensus.cc:493] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:57.825003  1678 raft_consensus.cc:3060] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:01:57.829958  1750 raft_consensus.cc:2468] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 13.
I20260709 13:01:57.832818  1748 raft_consensus.cc:3060] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:01:57.767238  7812 leader_election.cc:290] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:57.841396  1675 raft_consensus.cc:2468] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 17.
I20260709 13:01:57.846930  1753 raft_consensus.cc:2468] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 15.
I20260709 13:01:57.853219  1718 leader_election.cc:304] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:57.859956  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 19 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:57.859995  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:57.867137  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:57.862074  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:57.885002  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:57.885682  1750 raft_consensus.cc:3060] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:01:57.855872  7307 raft_consensus.cc:2804] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 13:01:57.899503  1748 raft_consensus.cc:2468] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 16.
I20260709 13:01:57.899503  1678 raft_consensus.cc:2468] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 16.
I20260709 13:01:57.905441  1674 raft_consensus.cc:2468] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 19.
I20260709 13:01:57.908066  7945 raft_consensus.cc:515] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:57.891855  7915 raft_consensus.cc:2749] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:01:57.912005  7307 raft_consensus.cc:697] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 17 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:01:57.913309  7307 consensus_queue.cc:237] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 16.8, Last appended by leader: 8, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:57.855609  7946 consensus_queue.cc:1059] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 17.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.063s
I20260709 13:01:57.815696  7964 raft_consensus.cc:515] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:57.919755  7946 raft_consensus.cc:3055] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 17 LEADER]: Stepping down as leader of term 17
I20260709 13:01:57.920243  7946 raft_consensus.cc:740] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:01:57.920043  7600 raft_consensus.cc:515] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:57.920907  7964 leader_election.cc:290] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:57.866015  8023 raft_consensus.cc:515] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:57.921289  1676 raft_consensus.cc:1240] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 13. Current term is 14. Ops: []
I20260709 13:01:57.922251  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 16 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:57.922974  1679 raft_consensus.cc:2468] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 15.
I20260709 13:01:57.923538  8023 leader_election.cc:290] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:57.924258  1673 raft_consensus.cc:2393] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 15: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:01:57.927718  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:57.928470  2337 raft_consensus.cc:2393] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 13: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:01:57.929400  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:57.930689  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:57.931384  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:57.932479  1747 raft_consensus.cc:2410] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for term 18 because replica has last-logged OpId of term: 16 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 10.
I20260709 13:01:57.933820  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 16 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:01:57.934966  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 81848.4us
I20260709 13:01:57.935576  1747 raft_consensus.cc:1240] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 16. Current term is 17. Ops: []
I20260709 13:01:57.936057  1718 leader_election.cc:304] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:57.936935  1747 raft_consensus.cc:1240] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 18. Current term is 19. Ops: []
I20260709 13:01:57.932263  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 14 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:57.938081  8075 raft_consensus.cc:2749] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:01:57.939440  1753 raft_consensus.cc:1275] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 19 index: 13. (index mismatch)
I20260709 13:01:57.944037  7600 leader_election.cc:290] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 14 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:57.945792  7946 consensus_queue.cc:260] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 17.14, Last appended by leader: 14, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:57.948120  7946 raft_consensus.cc:3060] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:01:57.962119  1746 raft_consensus.cc:1240] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 17. Current term is 18. Ops: [17.14-17.14]
I20260709 13:01:57.965430  1680 raft_consensus.cc:2468] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 19.
I20260709 13:01:57.972016  1750 raft_consensus.cc:2468] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 15.
I20260709 13:01:57.976020  1754 raft_consensus.cc:1240] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 14. Current term is 15. Ops: [14.12-14.12]
I20260709 13:01:57.965090  2298 leader_election.cc:304] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:01:57.984956  2298 leader_election.cc:304] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:01:57.994838  2329 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39726 (request call id 1435) took 1856 ms. Trace:
I20260709 13:01:57.995256  2329 rpcz_store.cc:276] 0709 13:01:56.138222 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:56.207591 (+ 69369us) service_pool.cc:224] Handling call
0709 13:01:56.240528 (+ 32937us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:56.805255 (+564727us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:57.986745 (+1181490us) spinlock_profiling.cc:243] Waited 564 ms on lock 0x7b54005e3b54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:01:57.994743 (+  7998us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1296825216}
I20260709 13:01:57.997690  7919 leader_election.cc:290] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 12 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:57.998387  7662 raft_consensus.cc:2804] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:01:57.998739  7945 leader_election.cc:290] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 15 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:57.999017  7662 raft_consensus.cc:697] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 12 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:01:58.000130  7662 consensus_queue.cc:237] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:58.036227  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:58.019068  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:58.049958  1680 raft_consensus.cc:2468] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 19.
W20260709 13:01:58.114771  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 62042.9us
W20260709 13:01:58.149830  2299 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:57.983340  8068 raft_consensus.cc:493] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:58.003494  8046 raft_consensus.cc:2804] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Leader election won for term 16
W20260709 13:01:58.169080  2300 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:58.176469  1672 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52766 (request call id 2987) took 2763 ms. Trace:
I20260709 13:01:58.177068  1672 rpcz_store.cc:276] 0709 13:01:55.412953 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:55.449520 (+ 36567us) service_pool.cc:224] Handling call
0709 13:01:55.449874 (+   354us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:56.404487 (+954613us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:58.176033 (+1771546us) spinlock_profiling.cc:243] Waited 948 ms on lock 0x7b5400346f54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:01:58.176371 (+   338us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2181126016}
I20260709 13:01:58.038947  7913 raft_consensus.cc:493] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:58.038674  7976 raft_consensus.cc:493] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:01:58.002975  8050 consensus_queue.cc:1059] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 16.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.156s
I20260709 13:01:57.987461  8055 consensus_queue.cc:1059] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 15.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.171s
I20260709 13:01:58.007668  8003 consensus_queue.cc:1048] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.169s
I20260709 13:01:58.035991  7915 consensus_queue.cc:1048] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.143s
I20260709 13:01:58.009260  7945 consensus_queue.cc:1059] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 10.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.170s
W20260709 13:01:58.191457  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 474887us
I20260709 13:01:58.204433  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 14 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:58.009593  7902 consensus_queue.cc:1048] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.174s
W20260709 13:01:58.208585  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 491810us
I20260709 13:01:58.211346  1674 raft_consensus.cc:1240] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 18. Current term is 19. Ops: []
I20260709 13:01:58.147707  7910 consensus_queue.cc:1048] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.042s
I20260709 13:01:58.215440  1680 raft_consensus.cc:1275] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
I20260709 13:01:58.152402  7970 consensus_queue.cc:1059] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.044s
I20260709 13:01:58.219802  1646 leader_election.cc:304] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:58.189915  8117 raft_consensus.cc:493] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:01:58.249375  1643 leader_election.cc:304] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:01:58.257618  1646 leader_election.cc:304] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 13:01:58.172554  8068 raft_consensus.cc:515] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:58.267704  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:58.275463  1679 raft_consensus.cc:3060] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:01:58.276165  1643 leader_election.cc:304] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:01:58.278162  1674 raft_consensus.cc:1240] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 14. Current term is 15. Ops: []
I20260709 13:01:58.186219  1672 raft_consensus.cc:1275] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 16 index: 9. Preceding OpId from leader: term: 18 index: 10. (index mismatch)
W20260709 13:01:58.313078  1569 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.515s	user 0.007s	sys 0.015s
W20260709 13:01:58.313619  1569 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.515s	user 0.007s	sys 0.015s
I20260709 13:01:58.314726  1646 leader_election.cc:304] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1
W20260709 13:01:58.250216  8010 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.517s	user 0.021s	sys 0.035s
I20260709 13:01:58.193147  8031 consensus_queue.cc:1059] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 18.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.066s
I20260709 13:01:58.318199  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:58.191919  7913 raft_consensus.cc:515] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:58.338011  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 61956.1us
W20260709 13:01:58.332343  1646 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:58.247289  7585 raft_consensus.cc:3055] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 17 LEADER]: Stepping down as leader of term 17
W20260709 13:01:58.347126  2333 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.515s	user 0.024s	sys 0.013s
I20260709 13:01:58.247430  7945 consensus_queue.cc:1243] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer 3b879799899a499c8d409ee9f1277cd1 log is divergent from this leader: its last log entry 13.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 9
I20260709 13:01:58.192241  7976 raft_consensus.cc:515] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:01:58.352285  2333 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.520s	user 0.024s	sys 0.013s
W20260709 13:01:58.314007  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 515905us
I20260709 13:01:58.311921  7812 raft_consensus.cc:2804] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Leader election won for term 18
I20260709 13:01:58.363713  1751 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55848 (request call id 3006) took 3444 ms. Trace:
I20260709 13:01:58.380017  1751 rpcz_store.cc:276] 0709 13:01:54.918948 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:56.527170 (+1608222us) service_pool.cc:224] Handling call
0709 13:01:56.527602 (+   432us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:56.614404 (+ 86802us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:58.363324 (+1748920us) spinlock_profiling.cc:243] Waited 84.4 ms on lock 0x7b540031f454. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:01:58.363616 (+   292us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":194117888}
I20260709 13:01:58.312155  8023 raft_consensus.cc:2804] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Leader pre-election won for term 19
W20260709 13:01:58.373768  1643 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 1.567s (SENT) [suppressed 15 similar messages]
I20260709 13:01:58.385648  1674 raft_consensus.cc:3055] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:01:58.386189  1674 raft_consensus.cc:740] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:01:58.400719  1679 raft_consensus.cc:2468] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 16.
I20260709 13:01:58.401914  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:58.317694  8115 raft_consensus.cc:2749] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:01:58.274308  7600 consensus_queue.cc:1048] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.106s
I20260709 13:01:58.309734  7964 consensus_queue.cc:1048] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.107s
I20260709 13:01:58.355283  8009 raft_consensus.cc:2804] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Leader election won for term 15
I20260709 13:01:58.387328  1674 consensus_queue.cc:260] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.10, Last appended by leader: 10, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:58.427919  1674 raft_consensus.cc:3060] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:01:58.415930  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:58.353190  7585 raft_consensus.cc:740] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:01:58.418752  1569 leader_election.cc:304] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:01:58.420495  2335 raft_consensus.cc:2468] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 13.
I20260709 13:01:58.430351  1672 raft_consensus.cc:3060] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:01:58.348122  7850 consensus_queue.cc:1048] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.064s
I20260709 13:01:58.404036  7945 consensus_queue.cc:1048] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.032s
I20260709 13:01:58.422959  8045 raft_consensus.cc:3060] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:01:58.389536  8023 raft_consensus.cc:493] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:58.380553  8121 raft_consensus.cc:2749] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:01:58.324462  8117 raft_consensus.cc:515] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:01:58.497645  1752 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.538s	user 0.012s	sys 0.016s
W20260709 13:01:58.498183  1752 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.538s	user 0.012s	sys 0.016s
I20260709 13:01:58.453696  1718 leader_election.cc:304] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:58.458287  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:58.410698  7812 raft_consensus.cc:697] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
W20260709 13:01:58.450421  7577 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.651s	user 0.046s	sys 0.062s
I20260709 13:01:58.485845  1674 raft_consensus.cc:1275] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 13 index: 10. (term mismatch)
W20260709 13:01:58.489779  1571 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.796s	user 0.006s	sys 0.044s
I20260709 13:01:58.514473  1674 pending_rounds.cc:85] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d: Aborting all ops after (but not including) 9
W20260709 13:01:58.514596  1571 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.820s	user 0.009s	sys 0.044s
I20260709 13:01:58.514824  1674 pending_rounds.cc:107] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d: Aborting uncommitted NO_OP operation due to leader change: 12.10
W20260709 13:01:58.515069  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 820977us
W20260709 13:01:58.490738  8120 log.cc:927] Time spent T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.080s	user 0.001s	sys 0.000s
I20260709 13:01:58.492614  7687 leader_election.cc:290] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:58.492816  7976 leader_election.cc:290] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:58.494963  7915 raft_consensus.cc:493] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:01:58.474068  8023 raft_consensus.cc:3060] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Advancing to term 19
W20260709 13:01:58.499964  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 46408.5us
I20260709 13:01:58.533977  1718 leader_election.cc:304] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:58.515113  1674 raft_consensus.cc:2889] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:01:58.503758  8046 raft_consensus.cc:697] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 16 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:01:58.540133  8046 consensus_queue.cc:237] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 14.13, Last appended by leader: 13, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:58.499635  8124 raft_consensus.cc:3055] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 LEADER]: Stepping down as leader of term 15
I20260709 13:01:58.411805  8123 consensus_queue.cc:1059] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 15.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.103s
W20260709 13:01:58.510612  7577 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.712s	user 0.047s	sys 0.062s
I20260709 13:01:58.498900  8009 raft_consensus.cc:697] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:01:58.510111  8010 raft_consensus.cc:493] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:01:58.546654  1747 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.544s	user 0.006s	sys 0.026s
I20260709 13:01:58.549264  1718 leader_election.cc:304] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:58.551687  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:01:58.554847  1747 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.552s	user 0.006s	sys 0.026s
I20260709 13:01:58.555379  1675 raft_consensus.cc:2468] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 12.
W20260709 13:01:58.548856  7307 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.551s	user 0.002s	sys 0.031s
I20260709 13:01:58.511129  7600 raft_consensus.cc:493] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
W20260709 13:01:58.511085  7728 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.628s	user 0.010s	sys 0.018s
I20260709 13:01:58.521183  1672 raft_consensus.cc:2468] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 18.
W20260709 13:01:58.574960  1746 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.611s	user 0.023s	sys 0.024s
W20260709 13:01:58.546825  7662 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.509s	user 0.015s	sys 0.028s
I20260709 13:01:58.544970  7976 raft_consensus.cc:2749] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 13:01:58.546526  8124 raft_consensus.cc:740] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:01:58.542842  7687 raft_consensus.cc:493] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
W20260709 13:01:58.580626  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 424429us
W20260709 13:01:58.583961  1746 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.620s	user 0.023s	sys 0.024s
I20260709 13:01:58.579756  2299 leader_election.cc:304] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
W20260709 13:01:58.577363  7307 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.576s	user 0.002s	sys 0.031s
W20260709 13:01:58.577255  7728 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.711s	user 0.010s	sys 0.018s
W20260709 13:01:58.580353  7662 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.542s	user 0.016s	sys 0.028s
I20260709 13:01:58.544453  7915 raft_consensus.cc:515] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:58.604048  2298 leader_election.cc:304] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
W20260709 13:01:58.624897  7946 consensus_meta.cc:287] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.676s	user 0.009s	sys 0.006s
I20260709 13:01:58.615335  7707 raft_consensus.cc:3060] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:01:58.575397  8010 raft_consensus.cc:515] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:58.615597  7728 raft_consensus.cc:697] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:01:58.583706  7600 raft_consensus.cc:515] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:58.687767  8117 leader_election.cc:290] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:58.551091  7585 consensus_queue.cc:260] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.14, Last appended by leader: 14, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:58.645612  7307 raft_consensus.cc:3060] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:01:58.646881  8055 raft_consensus.cc:2804] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 13:01:58.616521  7687 raft_consensus.cc:515] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:58.693382  8113 log.cc:927] Time spent T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.053s	user 0.001s	sys 0.000s
I20260709 13:01:58.627566  7976 consensus_queue.cc:1048] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.069s
I20260709 13:01:58.640918  8050 consensus_queue.cc:1059] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 14.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.059s
I20260709 13:01:58.669659  7981 raft_consensus.cc:2804] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader pre-election won for term 15
W20260709 13:01:58.704878  2300 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:58.682513  8068 leader_election.cc:290] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 13:01:58.711723  8116 consensus_meta.cc:287] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.976s	user 0.008s	sys 0.006s
I20260709 13:01:58.623301  7812 consensus_queue.cc:237] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.13, Last appended by leader: 13, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:58.735471  1717 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.579s	user 0.006s	sys 0.009s
W20260709 13:01:58.746868  1717 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.593s	user 0.007s	sys 0.009s
W20260709 13:01:58.748184  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 599424us
I20260709 13:01:58.699383  1679 raft_consensus.cc:2468] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 18.
I20260709 13:01:58.712598  7981 raft_consensus.cc:493] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:58.730894  7585 raft_consensus.cc:3060] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:01:58.715142  8055 raft_consensus.cc:697] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 19 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:01:58.701231  8119 consensus_queue.cc:1048] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.033s
I20260709 13:01:58.733724  7913 leader_election.cc:290] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:01:58.762861  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 344183us
I20260709 13:01:58.735049  8145 raft_consensus.cc:493] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:58.744240  7902 raft_consensus.cc:2749] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20260709 13:01:58.766820  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 393010us
I20260709 13:01:58.764074  1569 leader_election.cc:304] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:01:58.759902  7981 raft_consensus.cc:3060] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Advancing to term 15
W20260709 13:01:58.765002  8130 log.cc:927] Time spent T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.068s	user 0.000s	sys 0.001s
I20260709 13:01:58.702915  8009 consensus_queue.cc:237] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:58.760983  8077 raft_consensus.cc:493] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:01:58.749300  8116 raft_consensus.cc:515] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:01:58.796094  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 279648us
I20260709 13:01:58.732486  8140 consensus_queue.cc:1059] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 16.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.060s
I20260709 13:01:58.698170  8124 consensus_queue.cc:260] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.9, Last appended by leader: 9, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:58.764444  8050 raft_consensus.cc:3055] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [term 14 LEADER]: Stepping down as leader of term 14
W20260709 13:01:58.838131  1643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:58.841986  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 238001us
I20260709 13:01:58.848719  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:01:58.826792  8050 raft_consensus.cc:740] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:01:58.846589  1745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43156 (request call id 1287) took 2893 ms. Trace:
I20260709 13:01:58.845629  8115 raft_consensus.cc:493] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
W20260709 13:01:58.848073  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 222492us
I20260709 13:01:58.862833  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:58.823873  8058 raft_consensus.cc:493] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:58.839088  8137 raft_consensus.cc:493] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:58.872833  1604 raft_consensus.cc:2410] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 for term 15 because replica has last-logged OpId of term: 13 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 10.
I20260709 13:01:58.854178  1745 rpcz_store.cc:276] 0709 13:01:55.952672 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:56.761713 (+809041us) service_pool.cc:224] Handling call
0709 13:01:56.761908 (+   195us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:56.777387 (+ 15479us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 13:01:56.796665 (+ 19278us) log.cc:844] Serialized 14 byte log entry
0709 13:01:56.890075 (+ 93410us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:01:56.890164 (+    89us) raft_consensus.cc:1645] Marking committed up to 14
0709 13:01:56.890238 (+    74us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:58.846241 (+1956003us) spinlock_profiling.cc:243] Waited 14.8 ms on lock 0x7b54000e4a54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:01:58.846306 (+    65us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:58.846504 (+   198us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6206,"prepare.queue_time_us":19939,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":44,"raft.queue_time_us":6715,"raft.run_cpu_time_us":321,"raft.run_wall_time_us":26717,"spinlock_wait_cycles":34081152,"thread_start_us":35419,"threads_started":1,"wal-append.queue_time_us":126385,"wal-append.run_cpu_time_us":1209,"wal-append.run_wall_time_us":1001230}
I20260709 13:01:58.866164  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:58.875078  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:58.767251  7728 consensus_queue.cc:237] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 11.8, Last appended by leader: 8, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:58.876530  1672 raft_consensus.cc:2393] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 13: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:01:58.854686  8124 raft_consensus.cc:3060] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Advancing to term 16
W20260709 13:01:58.884948  8135 log.cc:927] Time spent T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.082s	user 0.001s	sys 0.001s
I20260709 13:01:58.887691  1747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55848 (request call id 3041) took 1173 ms. Trace:
I20260709 13:01:58.883693  2299 leader_election.cc:304] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
W20260709 13:01:58.890918  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 126909us
I20260709 13:01:58.891880  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:58.888139  1747 rpcz_store.cc:276] 0709 13:01:57.714070 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:57.937636 (+223566us) service_pool.cc:224] Handling call
0709 13:01:57.937982 (+   346us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:01:57.968936 (+ 30954us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:57.968984 (+    48us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:01:57.969835 (+   851us) log.cc:844] Serialized 52 byte log entry
0709 13:01:58.575388 (+605553us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:01:58.575442 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:58.575543 (+   101us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:58.887362 (+311819us) raft_consensus.cc:1685] finished
0709 13:01:58.887389 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:58.887588 (+   199us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":34998,"raft.queue_time_us":66021,"raft.run_cpu_time_us":144,"raft.run_wall_time_us":7066,"thread_start_us":552553,"threads_started":1,"wal-append.queue_time_us":731735}
I20260709 13:01:58.892844  1672 raft_consensus.cc:2368] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 17. Current term is 18.
I20260709 13:01:58.827654  8077 raft_consensus.cc:515] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:58.800884  8145 raft_consensus.cc:515] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:58.898761  2333 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39726 (request call id 1462) took 1097 ms. Trace:
W20260709 13:01:58.910187  1717 leader_election.cc:336] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 1.591s (SENT)
W20260709 13:01:58.712186  7662 consensus_queue.cc:1175] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "816fadcea8fc4893882f5f9b9f22a03d" responder_term: 18 status { last_received { term: 16 index: 13 } last_committed_idx: 12 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 17. Current term is 18. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
W20260709 13:01:58.902724  1645 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:58.910791  1717 leader_election.cc:304] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:01:58.873704  7915 leader_election.cc:290] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:58.909853  2333 rpcz_store.cc:276] 0709 13:01:57.801032 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:57.830728 (+ 29696us) service_pool.cc:224] Handling call
0709 13:01:57.831130 (+   402us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:57.831552 (+   422us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:57.831587 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:01:57.832184 (+   597us) log.cc:844] Serialized 26 byte log entry
0709 13:01:58.357452 (+525268us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:01:58.357516 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:58.357619 (+   103us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:58.898545 (+540926us) raft_consensus.cc:1685] finished
0709 13:01:58.898568 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:58.898694 (+   126us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":10700,"raft.run_cpu_time_us":151,"raft.run_wall_time_us":3938,"thread_start_us":520248,"threads_started":1,"wal-append.queue_time_us":765823}
I20260709 13:01:58.903775  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:01:58.909021  8136 log.cc:927] Time spent T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.075s	user 0.001s	sys 0.001s
W20260709 13:01:58.915467  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 89936us
I20260709 13:01:58.922402  1571 leader_election.cc:304] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 13:01:58.891897  8154 raft_consensus.cc:493] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:58.891952  8155 raft_consensus.cc:493] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:58.925168  8162 raft_consensus.cc:493] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:58.887962  8068 consensus_queue.cc:1059] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 13.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.040s
I20260709 13:01:58.878361  8115 raft_consensus.cc:515] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:58.937170  8046 consensus_queue.cc:1059] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 18.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.016s
I20260709 13:01:58.920607  8010 leader_election.cc:290] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:58.926662  7577 raft_consensus.cc:2804] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:01:58.865581  8055 consensus_queue.cc:237] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.11, Last appended by leader: 11, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:58.954653  2331 raft_consensus.cc:2468] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 19.
I20260709 13:01:58.926548  7687 leader_election.cc:290] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:58.918422  8146 raft_consensus.cc:2749] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:01:58.973140  1601 raft_consensus.cc:1240] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 LEADER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 17. Current term is 18. Ops: []
I20260709 13:01:58.982792  1601 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40154 (request call id 2694) took 1314 ms. Trace:
I20260709 13:01:58.983198  1601 rpcz_store.cc:276] 0709 13:01:57.667907 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:57.685687 (+ 17780us) service_pool.cc:224] Handling call
0709 13:01:57.686129 (+   442us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:58.972968 (+1286839us) spinlock_profiling.cc:243] Waited 1.47 ms on lock 0x7b54002e66a0. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42660d9dc0 00007f42661641c0 00007f4266165231 00007f4266161d8f 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c
0709 13:01:58.982387 (+  9419us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:58.982705 (+   318us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3377024}
I20260709 13:01:58.888844  8058 raft_consensus.cc:515] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:58.951649  8169 raft_consensus.cc:2804] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 13:01:58.881628  8137 raft_consensus.cc:515] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:58.954337  8165 raft_consensus.cc:2749] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 13:01:58.942247  7600 leader_election.cc:290] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:58.986480  1674 raft_consensus.cc:2468] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 13.
I20260709 13:01:58.953873  8168 consensus_queue.cc:1048] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.018s
I20260709 13:01:58.974676  8138 raft_consensus.cc:493] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:58.985096  8068 raft_consensus.cc:3055] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 13:01:58.994417  8169 raft_consensus.cc:493] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:58.969113  7577 raft_consensus.cc:493] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:59.002977  8138 raft_consensus.cc:515] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:58.919981  8050 consensus_queue.cc:260] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.12, Last appended by leader: 12, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:58.988941  8023 raft_consensus.cc:515] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:58.951987  8163 consensus_queue.cc:1059] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.050s
I20260709 13:01:58.940521  8155 raft_consensus.cc:515] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:59.012742  1569 leader_election.cc:304] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
W20260709 13:01:59.012758  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 65727.1us
I20260709 13:01:59.003815  8068 raft_consensus.cc:740] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:01:58.961743  8162 raft_consensus.cc:515] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:58.943306  8154 raft_consensus.cc:515] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:59.022719  1571 leader_election.cc:304] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
W20260709 13:01:59.034435  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 150849us
I20260709 13:01:59.035419  1571 leader_election.cc:304] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:59.010773  8169 raft_consensus.cc:3060] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:01:59.012135  7577 raft_consensus.cc:3060] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:01:59.028671  1569 leader_election.cc:304] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:59.036367  1571 leader_election.cc:304] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
W20260709 13:01:59.033483  8150 log.cc:927] Time spent T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.141s	user 0.001s	sys 0.000s
I20260709 13:01:59.013681  7952 raft_consensus.cc:3055] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 LEADER]: Stepping down as leader of term 16
I20260709 13:01:59.022224  8045 raft_consensus.cc:493] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:01:59.032918  7915 raft_consensus.cc:493] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:01:59.110582  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 74283.9us
I20260709 13:01:59.125435  1571 leader_election.cc:304] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
W20260709 13:01:59.109256  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 73117.9us
W20260709 13:01:59.112632  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 84051.6us
W20260709 13:01:59.030716  8156 log.cc:927] Time spent T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.072s	user 0.000s	sys 0.001s
I20260709 13:01:59.036753  8116 leader_election.cc:290] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 15 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:59.031368  8175 raft_consensus.cc:2749] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 13:01:59.107601  8010 raft_consensus.cc:2804] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Leader pre-election won for term 19
W20260709 13:01:59.140882  1646 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:59.107389  7687 raft_consensus.cc:2749] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:01:59.133131  1571 leader_election.cc:304] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:01:59.113595  7952 raft_consensus.cc:740] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:01:59.143991  1571 leader_election.cc:304] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:01:59.137746  1569 leader_election.cc:304] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:01:59.145803  1569 leader_election.cc:304] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:01:59.147377  8115 leader_election.cc:290] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:59.107391  8077 leader_election.cc:290] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:59.132349  1677 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52766 (request call id 2985) took 3720 ms. Trace:
I20260709 13:01:59.114588  8176 raft_consensus.cc:2804] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:01:59.128328  8050 raft_consensus.cc:3060] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:01:59.171255  1677 rpcz_store.cc:276] 0709 13:01:55.412381 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:55.413621 (+  1240us) service_pool.cc:224] Handling call
0709 13:01:57.112237 (+1698616us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:57.113746 (+  1509us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:59.132110 (+2018364us) spinlock_profiling.cc:243] Waited 1.7 s on lock 0x7b5400092750. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661618ad 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:01:59.132270 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3905257216}
I20260709 13:01:59.173147  1677 raft_consensus.cc:1240] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 14. Current term is 15. Ops: [14.7-14.7]
I20260709 13:01:59.139667  2299 leader_election.cc:304] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:01:59.141269  7728 raft_consensus.cc:2804] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:01:59.141182  8010 raft_consensus.cc:493] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:59.127708  8045 raft_consensus.cc:515] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:59.130301  8179 raft_consensus.cc:2749] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 13:01:59.164831  8157 raft_consensus.cc:2749] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:01:59.156582  8175 raft_consensus.cc:2749] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:01:59.128959  7915 raft_consensus.cc:515] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:59.171720  8171 raft_consensus.cc:2804] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Leader election won for term 16
I20260709 13:01:59.187785  1569 leader_election.cc:304] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 13:01:59.170538  7687 raft_consensus.cc:2804] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader pre-election won for term 16
W20260709 13:01:59.190615  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 51269.9us
I20260709 13:01:59.177791  7728 raft_consensus.cc:493] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:59.182721  8180 consensus_queue.cc:1048] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.007s
I20260709 13:01:59.177554  8176 raft_consensus.cc:493] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:59.184087  8010 raft_consensus.cc:3060] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:01:59.198587  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:01:59.200332  1569 leader_election.cc:304] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:01:59.204798  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:01:59.245162  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 45062.1us
W20260709 13:01:59.207176  8181 log.cc:927] Time spent T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.112s	user 0.001s	sys 0.000s
I20260709 13:01:59.208398  8165 raft_consensus.cc:3055] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:01:59.211545  8154 leader_election.cc:290] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:59.215337  8162 leader_election.cc:290] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:01:59.218884  8172 log.cc:927] Time spent T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.207s	user 0.000s	sys 0.001s
W20260709 13:01:59.221958  1488 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet 983621a6fc5b4c9ea0159a648dfe7a36 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER failed with leader 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 3b879799899a499c8d409ee9f1277cd1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 18 committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
W20260709 13:01:59.222623  8174 log.cc:927] Time spent T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.087s	user 0.001s	sys 0.000s
W20260709 13:01:59.222959  1748 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55840 (request call id 2928) took 1585 ms (client timeout 1912 ms). Trace:
W20260709 13:01:59.262054  1748 rpcz_store.cc:269] 0709 13:01:57.637257 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:57.708812 (+ 71555us) service_pool.cc:224] Handling call
0709 13:01:59.222744 (+1513932us) spinlock_profiling.cc:243] Waited 118 ms on lock 0x7b54000190d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:01:59.222885 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":270716672}
I20260709 13:01:59.226295  8155 leader_election.cc:290] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:01:59.226950  8164 log.cc:927] Time spent T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.202s	user 0.000s	sys 0.001s
I20260709 13:01:59.201848  8176 raft_consensus.cc:3060] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:01:59.203060  8178 raft_consensus.cc:2804] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Leader election won for term 19
I20260709 13:01:59.228418  8045 leader_election.cc:290] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:01:59.276412  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 132462us
W20260709 13:01:59.234908  8134 log.cc:927] Time spent T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.472s	user 0.000s	sys 0.002s
I20260709 13:01:59.235306  8115 raft_consensus.cc:493] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:59.245405  8171 raft_consensus.cc:697] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:01:59.200891  8077 raft_consensus.cc:493] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:59.201447  7728 raft_consensus.cc:3060] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Advancing to term 18
W20260709 13:01:59.243117  2300 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:59.192739  7952 consensus_queue.cc:260] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.11, Last appended by leader: 11, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:59.243916  1680 raft_consensus.cc:1275] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 14 index: 13. Preceding OpId from leader: term: 16 index: 14. (index mismatch)
I20260709 13:01:59.270290  8137 leader_election.cc:290] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:59.247583  8157 raft_consensus.cc:2804] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:01:59.247619  8129 consensus_queue.cc:1059] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 12.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 13:01:59.197513  8046 raft_consensus.cc:2804] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 13:01:59.197909  7687 raft_consensus.cc:493] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:01:59.228902  1673 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52766 (request call id 3010) took 1373 ms (client timeout 1744 ms). Trace:
W20260709 13:01:59.240116  2331 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39738 (request call id 1139) took 2484 ms (client timeout 1630 ms). Trace:
I20260709 13:01:59.249459  8145 leader_election.cc:290] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:59.249594  7946 raft_consensus.cc:493] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:59.280252  1671 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52778 (request call id 2564) took 1592 ms. Trace:
I20260709 13:01:59.315296  1671 rpcz_store.cc:276] 0709 13:01:57.688157 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:57.688306 (+   149us) service_pool.cc:224] Handling call
0709 13:01:59.279991 (+1591685us) spinlock_profiling.cc:243] Waited 77.9 ms on lock 0x7b54001f5254. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:01:59.280174 (+   183us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":179133696}
I20260709 13:01:59.259536  8165 raft_consensus.cc:740] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:01:59.282276  1571 leader_election.cc:304] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 13:01:59.246320  1569 leader_election.cc:304] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:01:59.290602  7981 raft_consensus.cc:515] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:59.284026  7307 consensus_meta.cc:287] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.578s	user 0.013s	sys 0.001s
W20260709 13:01:59.320259  2301 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:59.275326  8178 raft_consensus.cc:697] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
W20260709 13:01:59.309010  1673 rpcz_store.cc:269] 0709 13:01:57.856487 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:57.861754 (+  5267us) service_pool.cc:224] Handling call
0709 13:01:59.228665 (+1366911us) spinlock_profiling.cc:243] Waited 55.3 ms on lock 0x7b54000d9154. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:01:59.228816 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":127261568}
W20260709 13:01:59.309105  2331 rpcz_store.cc:269] 0709 13:01:56.755464 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:56.790740 (+ 35276us) service_pool.cc:224] Handling call
0709 13:01:59.239895 (+2449155us) spinlock_profiling.cc:243] Waited 2.15 s on lock 0x7b54005802d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:01:59.240016 (+   121us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4945480064}
I20260709 13:01:59.294255  8157 raft_consensus.cc:493] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:59.282496  8115 raft_consensus.cc:515] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:59.323565  7307 raft_consensus.cc:2749] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:01:59.325032  7981 leader_election.cc:290] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 15 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:59.353647  8171 consensus_queue.cc:237] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 14.9, Last appended by leader: 9, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } }
W20260709 13:01:59.355466  7987 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 13 current term: 13 is higher.
I20260709 13:01:59.357574  8157 raft_consensus.cc:3060] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:01:59.353260  8115 leader_election.cc:290] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:59.359364  8129 raft_consensus.cc:2804] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader election won for term 15
W20260709 13:01:59.366796  7585 consensus_meta.cc:287] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.604s	user 0.009s	sys 0.006s
I20260709 13:01:59.368099  8129 raft_consensus.cc:697] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
W20260709 13:01:59.368562  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 122318us
I20260709 13:01:59.369184  8129 consensus_queue.cc:237] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:59.370347  7915 leader_election.cc:290] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:59.309544  8023 leader_election.cc:290] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:59.375407  8003 raft_consensus.cc:3055] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 13:01:59.392292  8179 raft_consensus.cc:493] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:59.392580  8003 raft_consensus.cc:740] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:01:59.392943  8179 raft_consensus.cc:515] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:59.393556  8003 consensus_queue.cc:260] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.11, Last appended by leader: 11, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:59.342734  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:59.342986  8165 consensus_queue.cc:260] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.12, Last appended by leader: 12, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:59.344848  8126 log.cc:927] Time spent T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.799s	user 0.001s	sys 0.002s
I20260709 13:01:59.345724  7919 raft_consensus.cc:3055] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 13:01:59.310949  7687 raft_consensus.cc:3060] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:01:59.347628  8178 consensus_queue.cc:237] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:59.427471  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:59.285147  8077 raft_consensus.cc:515] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:59.321173  7946 raft_consensus.cc:515] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:59.436708  7946 leader_election.cc:290] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:59.371847  8185 consensus_queue.cc:1048] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.066s
I20260709 13:01:59.391189  7745 raft_consensus.cc:493] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:59.342346  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:59.394768  8168 raft_consensus.cc:493] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:59.439127  8168 raft_consensus.cc:515] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:59.384905  8138 leader_election.cc:290] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:59.371642  8175 consensus_queue.cc:1048] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 9, Time since last communication: 0.069s
I20260709 13:01:59.441365  8168 leader_election.cc:290] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:59.395613  8179 leader_election.cc:290] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:01:59.371601  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 89999.5us
I20260709 13:01:59.413044  7919 raft_consensus.cc:740] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:01:59.443188  7919 consensus_queue.cc:260] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:59.443532  1571 leader_election.cc:304] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:01:59.444594  7919 raft_consensus.cc:3060] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Advancing to term 19
W20260709 13:01:59.308321  7707 consensus_meta.cc:287] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.624s	user 0.013s	sys 0.006s
I20260709 13:01:59.445848  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 18 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:59.446270  7707 raft_consensus.cc:2749] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:01:59.399019  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:59.376652  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:59.448103  2336 raft_consensus.cc:2468] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 14.
I20260709 13:01:59.447237  8010 raft_consensus.cc:515] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:59.449195  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:59.378824  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:59.427878  8117 raft_consensus.cc:493] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:59.450966  2337 raft_consensus.cc:2468] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 19.
I20260709 13:01:59.450766  8117 raft_consensus.cc:515] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:59.322947  8046 raft_consensus.cc:697] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 18 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:01:59.422758  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:59.452453  8169 raft_consensus.cc:515] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:59.452937  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:59.453608  1747 raft_consensus.cc:3060] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:01:59.422904  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:59.454768  1748 raft_consensus.cc:2468] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 17.
I20260709 13:01:59.425369  2334 raft_consensus.cc:1275] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 17 index: 14. Preceding OpId from leader: term: 19 index: 15. (index mismatch)
I20260709 13:01:59.455519  8010 leader_election.cc:290] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 19 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:59.456287  8010 raft_consensus.cc:2749] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:01:59.428155  1745 raft_consensus.cc:2468] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 15.
I20260709 13:01:59.432274  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:59.433367  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:59.434367  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 20 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:59.392880  8176 raft_consensus.cc:515] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:59.373108  7981 consensus_queue.cc:1048] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.069s
I20260709 13:01:59.448199  2332 raft_consensus.cc:2468] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 17.
I20260709 13:01:59.403019  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:01:59.449954  8165 raft_consensus.cc:3060] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:01:59.452536  8046 consensus_queue.cc:237] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.11, Last appended by leader: 11, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:59.456820  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:59.430284  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:59.478881  2337 raft_consensus.cc:2468] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 17.
I20260709 13:01:59.479445  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 15 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:59.479642  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:59.480240  1750 raft_consensus.cc:2468] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 13.
I20260709 13:01:59.480623  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 19 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:59.481184  1752 raft_consensus.cc:3060] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:01:59.402272  7307 raft_consensus.cc:493] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:59.371243  8188 raft_consensus.cc:2749] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:01:59.424296  8003 raft_consensus.cc:3060] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:01:59.458864  8058 leader_election.cc:290] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:59.429739  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:59.455714  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:01:59.489499  1748 raft_consensus.cc:2393] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 14: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:01:59.461611  1718 leader_election.cc:304] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:59.428102  8055 raft_consensus.cc:493] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:59.464376  2336 raft_consensus.cc:1240] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 17. Current term is 18. Ops: []
I20260709 13:01:59.494761  2301 leader_election.cc:304] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:59.501804  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:59.502532  2301 leader_election.cc:304] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:59.466784  1746 raft_consensus.cc:2468] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 18.
I20260709 13:01:59.468492  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:59.469748  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:59.504407  1745 raft_consensus.cc:2468] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 13.
I20260709 13:01:59.504977  1750 raft_consensus.cc:2393] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 16: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:01:59.475028  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:59.438802  7745 raft_consensus.cc:515] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:59.430266  7915 raft_consensus.cc:493] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:59.483479  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:59.509690  1598 raft_consensus.cc:2368] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for earlier term 12. Current term is 13.
I20260709 13:01:59.486550  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 19 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:59.452962  8117 leader_election.cc:290] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:59.487898  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:59.490495  1718 leader_election.cc:304] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:59.494863  1646 leader_election.cc:304] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:01:59.515578  1646 leader_election.cc:304] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:59.495527  1747 raft_consensus.cc:2468] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 15.
I20260709 13:01:59.500303  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 16 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:59.517972  1597 raft_consensus.cc:2410] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for term 16 because replica has last-logged OpId of term: 15 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 8.
I20260709 13:01:59.489270  2328 raft_consensus.cc:2468] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 15.
I20260709 13:01:59.500959  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:59.507701  2301 leader_election.cc:304] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:59.448577  7577 raft_consensus.cc:515] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:59.528515  1752 raft_consensus.cc:2468] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 19.
I20260709 13:01:59.516745  1646 leader_election.cc:304] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
W20260709 13:01:59.507162  8028 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.264s	user 0.011s	sys 0.022s
I20260709 13:01:59.532615  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:59.537777  1603 raft_consensus.cc:2393] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 15: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:01:59.539369  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:01:59.539850  8173 log.cc:927] Time spent T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.427s	user 0.000s	sys 0.005s
I20260709 13:01:59.540076  1597 raft_consensus.cc:2468] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 17.
I20260709 13:01:59.551851  8176 leader_election.cc:290] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 15 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:59.508306  8138 raft_consensus.cc:2804] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:01:59.558804  8138 raft_consensus.cc:493] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:59.559289  8138 raft_consensus.cc:3060] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:01:59.560405  1754 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43156 (request call id 1304) took 1726 ms. Trace:
I20260709 13:01:59.560956  1754 rpcz_store.cc:276] 0709 13:01:57.834117 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:57.957275 (+123158us) service_pool.cc:224] Handling call
0709 13:01:57.975695 (+ 18420us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:57.976476 (+   781us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:59.560116 (+1583640us) spinlock_profiling.cc:243] Waited 18 ms on lock 0x7b540019bed0. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661618ad 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:01:59.560310 (+   194us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":41335680}
I20260709 13:01:59.565506  7577 leader_election.cc:290] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 14 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:59.512827  8050 raft_consensus.cc:493] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:59.566757  8050 raft_consensus.cc:515] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:59.569111  8050 leader_election.cc:290] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:01:59.571170  1646 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:59.501762  8169 leader_election.cc:290] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 22 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:59.508520  8145 raft_consensus.cc:2804] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:01:59.608687  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:59.609305  2337 raft_consensus.cc:1240] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 15. Current term is 16. Ops: []
I20260709 13:01:59.609457  2330 raft_consensus.cc:2468] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 16.
W20260709 13:01:59.605990  2299 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:59.606350  1605 raft_consensus.cc:3060] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:01:59.608188  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:59.612922  2336 raft_consensus.cc:2393] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 14: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:01:59.511322  8146 raft_consensus.cc:493] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:01:59.615453  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:01:59.535782  8028 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.318s	user 0.011s	sys 0.022s
I20260709 13:01:59.488379  7307 raft_consensus.cc:515] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:59.531353  7585 raft_consensus.cc:2804] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:01:59.534715  7987 raft_consensus.cc:2804] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:01:59.529675  7946 raft_consensus.cc:2749] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:01:59.613729  1600 raft_consensus.cc:1240] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 12. Current term is 13. Ops: [12.10-12.10]
I20260709 13:01:59.531914  8115 raft_consensus.cc:2804] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:01:59.535449  8117 raft_consensus.cc:2804] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:01:59.536084  7707 raft_consensus.cc:2804] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:01:59.633908  1751 raft_consensus.cc:2410] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for term 19 because replica has last-logged OpId of term: 17 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 9.
I20260709 13:01:59.647802  2328 raft_consensus.cc:1275] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 16 index: 11. Preceding OpId from leader: term: 19 index: 13. (index mismatch)
I20260709 13:01:59.508697  8055 raft_consensus.cc:515] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:59.650235  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:59.610579  7945 raft_consensus.cc:493] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:59.606942  8154 raft_consensus.cc:3055] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:01:59.614639  1754 raft_consensus.cc:1275] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 16 index: 9. (index mismatch)
I20260709 13:01:59.660604  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:59.531927  7728 raft_consensus.cc:515] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:59.708724  7307 leader_election.cc:290] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:59.557816  8068 consensus_queue.cc:260] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 13.13, Last appended by leader: 13, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:59.715148  1676 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52766 (request call id 3009) took 1865 ms. Trace:
I20260709 13:01:59.715700  1676 rpcz_store.cc:276] 0709 13:01:57.850278 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:57.850459 (+   181us) service_pool.cc:224] Handling call
0709 13:01:57.850841 (+   382us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:57.921736 (+ 70895us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:59.714791 (+1793055us) spinlock_profiling.cc:243] Waited 70.2 ms on lock 0x7b54001f3e54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:01:59.715058 (+   267us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":161457408}
I20260709 13:01:59.711488  8031 raft_consensus.cc:493] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:59.719396  8031 raft_consensus.cc:515] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:59.720578  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:01:59.720820  8055 leader_election.cc:290] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:59.721302  1747 raft_consensus.cc:2468] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 16.
I20260709 13:01:59.721941  8031 leader_election.cc:290] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:59.632026  8179 consensus_queue.cc:1059] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 18.11, Next index: 12, Last known committed idx: 10, Time since last communication: 0.086s
I20260709 13:01:59.724119  8157 raft_consensus.cc:515] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:59.725538  7902 raft_consensus.cc:3055] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 13:01:59.725981  7902 raft_consensus.cc:740] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
W20260709 13:01:59.727792  1570 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:59.728837  2301 leader_election.cc:304] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:59.733760  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:59.735108  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:59.653067  8117 raft_consensus.cc:493] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:59.653226  8115 raft_consensus.cc:493] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:59.653986  7707 raft_consensus.cc:493] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:59.738040  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:59.739629  1598 raft_consensus.cc:2393] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 20: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:01:59.740206  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:59.743551  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 16 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:59.726245  8157 leader_election.cc:290] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 20 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:59.531677  7915 raft_consensus.cc:515] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:59.759996  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:59.760620  1598 raft_consensus.cc:3060] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:01:59.775904  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:59.655700  7987 raft_consensus.cc:493] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:01:59.747371  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:59.735813  8055 raft_consensus.cc:493] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:01:59.748844  8192 log.cc:927] Time spent T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.128s	user 0.000s	sys 0.000s
I20260709 13:01:59.749361  7856 raft_consensus.cc:2804] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:01:59.749893  7902 consensus_queue.cc:260] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 17.14, Last appended by leader: 14, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:59.624902  8146 raft_consensus.cc:515] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:59.787220  1597 raft_consensus.cc:3060] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:01:59.687281  8154 raft_consensus.cc:740] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:01:59.743481  8068 raft_consensus.cc:3060] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:01:59.817233  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:59.792574  7987 raft_consensus.cc:3060] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:01:59.741402  8117 raft_consensus.cc:3060] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:01:59.741942  7707 raft_consensus.cc:3060] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:01:59.706398  7945 raft_consensus.cc:515] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:01:59.788138  8144 log.cc:927] Time spent T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.966s	user 0.003s	sys 0.000s
I20260709 13:01:59.809793  7856 raft_consensus.cc:493] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:59.818074  1603 raft_consensus.cc:2393] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 15: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
W20260709 13:01:59.797320  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 306798us
I20260709 13:01:59.864034  1718 leader_election.cc:304] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:01:59.821326  7745 leader_election.cc:290] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:01:59.823825  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:59.872411  1596 raft_consensus.cc:2368] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for earlier term 13. Current term is 14.
I20260709 13:01:59.873489  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 18 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:59.814028  7946 raft_consensus.cc:493] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:01:59.831055  1679 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57092 (request call id 1420) took 1373 ms (client timeout 1635 ms). Trace:
W20260709 13:01:59.878902  1679 rpcz_store.cc:269] 0709 13:01:58.457630 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:58.457882 (+   252us) service_pool.cc:224] Handling call
0709 13:01:59.830864 (+1372982us) spinlock_profiling.cc:243] Waited 182 ms on lock 0x7b540025b054. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:01:59.830984 (+   120us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":419313536}
I20260709 13:01:59.835282  1748 raft_consensus.cc:1240] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 17. Current term is 18. Ops: []
W20260709 13:01:59.841318  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:59.847280  7307 raft_consensus.cc:493] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:01:59.880824  7307 raft_consensus.cc:515] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:59.700309  1605 raft_consensus.cc:2468] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 15.
I20260709 13:01:59.851251  1598 raft_consensus.cc:2468] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 18.
I20260709 13:01:59.741192  8115 raft_consensus.cc:3060] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:01:59.859318  2298 leader_election.cc:304] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 13:01:59.816119  8126 log.cc:927] Time spent T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.053s	user 0.000s	sys 0.000s
I20260709 13:01:59.825440  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:59.882727  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:59.843222  7902 raft_consensus.cc:3060] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:01:59.832598  1678 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57092 (request call id 1413) took 1619 ms. Trace:
I20260709 13:01:59.882793  1602 raft_consensus.cc:2468] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 18.
I20260709 13:01:59.867818  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:59.883549  1598 raft_consensus.cc:2468] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 19.
I20260709 13:01:59.798489  8055 raft_consensus.cc:515] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:01:59.885526  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 16 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:59.884783  1678 rpcz_store.cc:276] 0709 13:01:58.213156 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:58.219361 (+  6205us) service_pool.cc:224] Handling call
0709 13:01:59.280181 (+1060820us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:59.280486 (+   305us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:59.280516 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:01:59.283740 (+  3224us) log.cc:844] Serialized 26 byte log entry
0709 13:01:59.430990 (+147250us) raft_consensus.cc:1645] Marking committed up to 14
0709 13:01:59.431029 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:59.431102 (+    73us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:59.831513 (+400411us) raft_consensus.cc:1685] finished
0709 13:01:59.831545 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:59.832249 (+   704us) spinlock_profiling.cc:243] Waited 1.06 s on lock 0x7b54001f5250. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661618ad 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:01:59.832512 (+   263us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2438329088,"thread_start_us":138413,"threads_started":1,"wal-append.queue_time_us":195960}
W20260709 13:01:59.890698  2300 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:59.890607  1596 raft_consensus.cc:2410] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for term 16 because replica has last-logged OpId of term: 15 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 8.
I20260709 13:01:59.860056  8148 raft_consensus.cc:493] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:59.891994  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:59.905289  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:59.906101  1602 raft_consensus.cc:2468] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 14.
I20260709 13:01:59.907209  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:01:59.880893  8096 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 2.145s	user 0.014s	sys 0.015s
I20260709 13:01:59.903296  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:59.908092  1602 raft_consensus.cc:2393] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 14: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:01:59.893239  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:59.920617  1598 raft_consensus.cc:2468] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 16.
I20260709 13:01:59.878450  8058 raft_consensus.cc:2804] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:01:59.916981  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:01:59.917613  7307 leader_election.cc:290] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:59.919632  1597 raft_consensus.cc:2468] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 18.
I20260709 13:01:59.887107  1569 leader_election.cc:304] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
W20260709 13:01:59.951133  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 64115.8us
I20260709 13:01:59.921358  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:59.868995  7856 raft_consensus.cc:3060] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:01:59.921566  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:59.958243  8127 raft_consensus.cc:2804] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader pre-election won for term 15
W20260709 13:01:59.938135  8096 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 2.186s	user 0.015s	sys 0.015s
W20260709 13:01:59.927565  2300 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:59.940640  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:59.921890  8148 raft_consensus.cc:515] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:59.903601  8140 raft_consensus.cc:2749] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
W20260709 13:01:59.922623  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 237279us
I20260709 13:01:59.885102  7946 raft_consensus.cc:515] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:01:59.964991  1569 leader_election.cc:304] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:01:59.957801  7307 raft_consensus.cc:493] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:01:59.958775  8127 raft_consensus.cc:493] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:02:00.002003  2301 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:00.003343  1749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43156 (request call id 1297) took 3388 ms. Trace:
I20260709 13:01:59.996424  7307 raft_consensus.cc:515] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:00.010811  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 15 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:00.011782  1600 raft_consensus.cc:3060] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:01:59.965061  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 19 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:59.966610  7745 consensus_queue.cc:1048] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.015s
I20260709 13:01:59.985500  8077 leader_election.cc:290] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:01:59.986400  8055 leader_election.cc:290] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:01:59.988644  2329 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49082 (request call id 1158) took 1095 ms. Trace:
I20260709 13:01:59.989149  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:00.025717  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:00.049201  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:59.955148  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:00.073526  2329 rpcz_store.cc:276] 0709 13:01:58.893700 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:58.894167 (+   467us) service_pool.cc:224] Handling call
0709 13:01:58.911038 (+ 16871us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:58.924621 (+ 13583us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:58.924657 (+    36us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:01:58.924725 (+    68us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:01:58.924759 (+    34us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:59.988363 (+1063604us) spinlock_profiling.cc:243] Waited 13.3 ms on lock 0x7b54005b9fd4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:01:59.988422 (+    59us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:59.988571 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":30683904}
I20260709 13:02:00.076781  1597 raft_consensus.cc:2468] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 14.
I20260709 13:02:00.068291  1748 raft_consensus.cc:1275] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 17 index: 10. Preceding OpId from leader: term: 18 index: 10. (term mismatch)
I20260709 13:02:00.077651  1748 pending_rounds.cc:85] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701: Aborting all ops after (but not including) 9
I20260709 13:02:00.077934  1748 pending_rounds.cc:107] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701: Aborting uncommitted NO_OP operation due to leader change: 17.10
I20260709 13:02:00.077971  2332 raft_consensus.cc:2468] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 15.
I20260709 13:02:00.075294  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:00.028687  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:00.079299  2333 raft_consensus.cc:1240] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 12 LEADER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 11. Current term is 12. Ops: [11.10-11.10]
I20260709 13:02:00.078287  1748 raft_consensus.cc:2889] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:01:59.967458  8129 raft_consensus.cc:493] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:00.078217  2298 leader_election.cc:304] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:02:00.042562  1644 leader_election.cc:304] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:02:00.090626  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:00.083221  1754 raft_consensus.cc:2393] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 15: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:02:00.091840  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:00.091879  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:00.093065  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 19 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:00.081871  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:00.043092  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:00.003885  1749 rpcz_store.cc:276] 0709 13:01:56.615110 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:57.229101 (+613991us) service_pool.cc:224] Handling call
0709 13:01:57.786599 (+557498us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:57.787089 (+   490us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:57.787123 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:01:57.787650 (+   527us) log.cc:844] Serialized 26 byte log entry
0709 13:01:58.213333 (+425683us) raft_consensus.cc:1645] Marking committed up to 14
0709 13:01:58.709035 (+495702us) log.cc:844] Serialized 14 byte log entry
0709 13:01:58.718487 (+  9452us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:58.718611 (+   124us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:59.344190 (+625579us) spinlock_profiling.cc:243] Waited 2.56 ms on lock 0x7b4400022b4c. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f4262f9bc85 00007f4262f9c6a2 00007f4265d0e1db 00007f4265d09aaf 00007f4265cfd86e 00007f4265cfcbb6 00007f4265cfc7d8 00007f4265d06221 00007f4265d061d9 00007f4265d06169
0709 13:01:59.511725 (+167535us) raft_consensus.cc:1685] finished
0709 13:01:59.530115 (+ 18390us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:00.003103 (+472988us) spinlock_profiling.cc:243] Waited 535 ms on lock 0x7b5400019fd0. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661618ad 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:02:00.003256 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":778,"prepare.queue_time_us":475747,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":35,"raft.queue_time_us":21866,"raft.run_cpu_time_us":307,"raft.run_wall_time_us":38845,"spinlock_wait_cycles":1237231744,"thread_start_us":880064,"threads_started":2,"wal-append.queue_time_us":636699}
W20260709 13:02:00.062194  2301 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:00.086961  7307 leader_election.cc:290] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:00.096552  7307 raft_consensus.cc:493] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:00.097340  7307 raft_consensus.cc:515] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:00.099406  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 56992.6us
I20260709 13:02:00.101114  7307 leader_election.cc:290] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:00.029822  8127 raft_consensus.cc:3060] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:02:00.088223  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:01:59.947800  8154 consensus_queue.cc:260] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.8, Last appended by leader: 8, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:00.093948  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:00.112885  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:00.113644  1748 raft_consensus.cc:2393] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 15: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:02:00.094640  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:01:59.966940  8201 consensus_queue.cc:1059] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.145s
W20260709 13:02:00.119575  1718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:00.120522  1676 raft_consensus.cc:1240] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 17. Current term is 18. Ops: []
I20260709 13:02:00.082991  7728 leader_election.cc:290] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 18 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:00.045882  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:00.123821  2337 raft_consensus.cc:3060] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:02:00.123916  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:00.105201  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:00.088382  7910 raft_consensus.cc:3055] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 13:02:00.070688  8031 raft_consensus.cc:493] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:00.122179  1718 leader_election.cc:400] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 17 pre-election: Vote denied by peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261) with higher term. Message: Invalid argument: T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 17. Current term is 18.
I20260709 13:02:00.133394  1718 leader_election.cc:403] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 17 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:02:00.030043  7981 raft_consensus.cc:2804] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader pre-election won for term 15
W20260709 13:02:00.110589  2335 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.711s	user 0.027s	sys 0.028s
W20260709 13:02:00.136572  2335 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.809s	user 0.027s	sys 0.028s
I20260709 13:01:59.970409  8058 raft_consensus.cc:493] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:00.140188  1678 raft_consensus.cc:1240] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 17. Current term is 18. Ops: []
W20260709 13:02:00.119789  7687 consensus_meta.cc:287] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.679s	user 0.013s	sys 0.000s
I20260709 13:02:00.113029  8009 raft_consensus.cc:2804] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:02:00.118046  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:00.124980  1603 raft_consensus.cc:2393] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 19: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:02:00.127408  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:00.128805  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:00.135932  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:00.138054  1643 leader_election.cc:400] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 12 election: Vote denied by peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445) with higher term. Message: Invalid argument: T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for earlier term 12. Current term is 13.
I20260709 13:02:00.160005  1671 raft_consensus.cc:2468] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 17.
I20260709 13:02:00.160038  1675 raft_consensus.cc:2468] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 17.
I20260709 13:02:00.087457  8055 raft_consensus.cc:493] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:00.116698  1754 raft_consensus.cc:1240] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 13. Current term is 14. Ops: []
I20260709 13:02:00.135552  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:00.164522  1748 raft_consensus.cc:1240] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 18. Current term is 19. Ops: [18.10-18.10]
I20260709 13:02:00.146176  1674 raft_consensus.cc:1240] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 17. Current term is 18. Ops: [17.11-17.11]
I20260709 13:02:00.133620  7910 raft_consensus.cc:740] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:02:00.150769  1680 raft_consensus.cc:1275] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 18 index: 11. Preceding OpId from leader: term: 19 index: 12. (index mismatch)
I20260709 13:02:00.144758  8058 raft_consensus.cc:3060] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:02:00.116747  1600 raft_consensus.cc:2468] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 15.
I20260709 13:02:00.183154  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:00.184708  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:00.159432  1602 raft_consensus.cc:2393] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 22: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:02:00.159492  1673 raft_consensus.cc:2468] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 18.
I20260709 13:02:00.085561  8023 consensus_queue.cc:1059] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 15.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.037s
I20260709 13:02:00.166390  1643 leader_election.cc:304] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:02:00.170123  1748 raft_consensus.cc:1275] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 16 index: 10. (index mismatch)
I20260709 13:02:00.166085  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:02:00.190301  1570 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:00.136802  8140 raft_consensus.cc:2804] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:02:00.136472  7981 raft_consensus.cc:493] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:00.192689  7981 raft_consensus.cc:3060] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:02:00.194766  7945 leader_election.cc:290] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:00.132548  7915 leader_election.cc:290] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:00.113343  7812 consensus_queue.cc:1048] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.046s
I20260709 13:02:00.150761  8031 raft_consensus.cc:515] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:00.103793  8129 raft_consensus.cc:515] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:00.160655  8201 raft_consensus.cc:3060] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Advancing to term 18
I20260709 13:02:00.201280  8031 leader_election.cc:290] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:00.159930  8154 raft_consensus.cc:3060] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Advancing to term 13
W20260709 13:02:00.202375  8197 log.cc:927] Time spent T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.210s	user 0.000s	sys 0.000s
W20260709 13:02:00.204102  8199 log.cc:927] Time spent T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.068s	user 0.000s	sys 0.000s
W20260709 13:02:00.205773  8200 log.cc:927] Time spent T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.087s	user 0.000s	sys 0.000s
I20260709 13:02:00.191751  7910 consensus_queue.cc:260] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.10, Last appended by leader: 10, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:00.206918  7910 raft_consensus.cc:3060] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 11 FOLLOWER]: Advancing to term 12
W20260709 13:02:00.209575  8204 log.cc:927] Time spent T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.089s	user 0.000s	sys 0.000s
I20260709 13:02:00.213649  7946 leader_election.cc:290] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:00.171860  8146 leader_election.cc:290] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:00.214685  8214 consensus_queue.cc:1059] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 10.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 13:02:00.215355  8146 raft_consensus.cc:493] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:00.216213  8146 raft_consensus.cc:515] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } }
I20260709 13:02:00.219026  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:00.219715  7952 raft_consensus.cc:3060] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:02:00.190515  1603 raft_consensus.cc:2468] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 15.
I20260709 13:02:00.220670  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 19 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:00.221482  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:00.221998  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:00.159016  8009 raft_consensus.cc:493] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:00.222893  2298 leader_election.cc:304] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:00.223466  8009 raft_consensus.cc:3060] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Advancing to term 16
W20260709 13:02:00.225096  1645 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:00.225695  8031 raft_consensus.cc:2749] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:02:00.226470  8146 leader_election.cc:290] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:00.227070  8146 raft_consensus.cc:3055] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:02:00.227150  8148 leader_election.cc:290] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:00.227468  8146 raft_consensus.cc:740] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:02:00.229163  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:00.230602  1603 raft_consensus.cc:2368] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 17. Current term is 18.
I20260709 13:02:00.155599  2337 raft_consensus.cc:2468] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 14.
I20260709 13:02:00.255903  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:00.256731  1748 raft_consensus.cc:3060] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:02:00.166227  7687 raft_consensus.cc:515] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:00.264938  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
W20260709 13:02:00.269447  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 103145us
W20260709 13:02:00.270448  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 47662.1us
I20260709 13:02:00.271477  1643 leader_election.cc:400] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 13 pre-election: Vote denied by peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445) with higher term. Message: Invalid argument: T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for earlier term 13. Current term is 14.
I20260709 13:02:00.272749  1571 leader_election.cc:304] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
W20260709 13:02:00.229003  8202 log.cc:927] Time spent T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.178s	user 0.001s	sys 0.000s
I20260709 13:02:00.276190  1643 leader_election.cc:403] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 13 pre-election: Cancelling election due to peer responding with higher term
W20260709 13:02:00.299407  1719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:00.299925  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 291875us
W20260709 13:02:00.301761  1646 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:00.188863  8055 raft_consensus.cc:515] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:00.228466  8146 consensus_queue.cc:260] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 14.7, Last appended by leader: 7, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:02:00.298074  8165 consensus_meta.cc:287] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.814s	user 0.008s	sys 0.005s
W20260709 13:02:00.297204  8138 consensus_meta.cc:287] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.729s	user 0.014s	sys 0.000s
I20260709 13:02:00.308986  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:00.311640  1753 raft_consensus.cc:1275] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 16 index: 13. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
I20260709 13:02:00.288754  7745 raft_consensus.cc:2749] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 13:02:00.304203  1601 raft_consensus.cc:1240] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 14. Current term is 16. Ops: [14.9-14.9]
W20260709 13:02:00.303642  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 294533us
I20260709 13:02:00.305956  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:00.328640  1745 raft_consensus.cc:2468] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 15.
I20260709 13:02:00.330019  1745 raft_consensus.cc:1240] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 19 LEADER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 18. Current term is 19. Ops: []
W20260709 13:02:00.345654  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 206334us
I20260709 13:02:00.277969  7728 consensus_queue.cc:1048] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.046s
W20260709 13:02:00.315682  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 215143us
I20260709 13:02:00.302379  1748 raft_consensus.cc:2468] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 18.
I20260709 13:02:00.352739  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:00.372834  1600 raft_consensus.cc:2468] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 15.
I20260709 13:02:00.302881  8198 consensus_queue.cc:1059] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 13.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.028s
I20260709 13:02:00.302325  1646 leader_election.cc:304] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
W20260709 13:02:00.331828  8213 log.cc:927] Time spent T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.101s	user 0.001s	sys 0.000s
W20260709 13:02:00.374948  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 72707.3us
I20260709 13:02:00.297000  8119 consensus_queue.cc:1059] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 16.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.036s
I20260709 13:02:00.386880  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:00.388092  1600 raft_consensus.cc:2393] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 15: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
W20260709 13:02:00.395637  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:00.335572  8210 raft_consensus.cc:2749] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:02:00.414968  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:00.371345  8146 raft_consensus.cc:3060] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:02:00.422760  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:00.427528  8217 raft_consensus.cc:493] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:00.428333  8217 raft_consensus.cc:515] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:00.429672  8219 raft_consensus.cc:3060] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Advancing to term 14
I20260709 13:02:00.370963  7946 raft_consensus.cc:493] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
W20260709 13:02:00.432193  2300 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:00.333659  8138 raft_consensus.cc:515] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:00.440107  2335 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39738 (request call id 1145) took 1241 ms. Trace:
I20260709 13:02:00.442655  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 20 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:00.445322  8055 leader_election.cc:290] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:00.446112  1680 raft_consensus.cc:1240] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 17. Current term is 18. Ops: [17.14-17.14]
I20260709 13:02:00.449533  7856 raft_consensus.cc:515] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:00.469425  1677 raft_consensus.cc:1240] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 16. Current term is 17. Ops: [16.11-16.11]
I20260709 13:02:00.454834  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:00.489977  1600 raft_consensus.cc:2468] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 16.
I20260709 13:02:00.466426  2335 rpcz_store.cc:276] 0709 13:01:59.199141 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:59.200778 (+  1637us) service_pool.cc:224] Handling call
0709 13:01:59.201237 (+   459us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:59.278047 (+ 76810us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:59.278104 (+    57us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:01:59.278876 (+   772us) log.cc:844] Serialized 26 byte log entry
0709 13:02:00.137047 (+858171us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:02:00.137108 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:00.137201 (+    93us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:00.439839 (+302638us) raft_consensus.cc:1685] finished
0709 13:02:00.439866 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:00.440018 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":17896,"raft.run_cpu_time_us":100,"raft.run_wall_time_us":100,"thread_start_us":885633,"threads_started":2,"wal-append.queue_time_us":995476}
I20260709 13:02:00.467183  1674 raft_consensus.cc:2410] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 for term 20 because replica has last-logged OpId of term: 18 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 14.
I20260709 13:02:00.467833  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 19 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:00.475310  1680 raft_consensus.cc:1240] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 18. Current term is 19. Ops: [18.10-18.10]
W20260709 13:02:00.485399  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 91111.1us
I20260709 13:02:00.458770  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:00.422773  8165 raft_consensus.cc:493] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:02:00.496428  1672 raft_consensus.cc:1240] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 18. Current term is 19. Ops: []
I20260709 13:02:00.496276  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:00.499528  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:00.462783  8198 raft_consensus.cc:493] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:00.451650  8201 raft_consensus.cc:2749] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Leader pre-election lost for term 17. Reason: Vote denied by peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261) with higher term. Message: Invalid argument: T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 17. Current term is 18.
I20260709 13:02:00.510185  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 19 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:02:00.501183  7987 consensus_meta.cc:287] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.635s	user 0.015s	sys 0.005s
I20260709 13:02:00.516512  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:00.495699  8154 raft_consensus.cc:3055] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 LEADER]: Stepping down as leader of term 13
I20260709 13:02:00.521697  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:00.522010  1674 raft_consensus.cc:2393] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 15: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:02:00.503640  8210 raft_consensus.cc:2804] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Leader election won for term 19
I20260709 13:02:00.537701  1680 raft_consensus.cc:2393] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 15: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:02:00.541631  1596 raft_consensus.cc:2468] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 13.
W20260709 13:02:00.534952  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 231078us
I20260709 13:02:00.472337  7946 raft_consensus.cc:515] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:00.551664  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 20 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:00.577482  1680 raft_consensus.cc:1275] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 16 index: 10. (index mismatch)
W20260709 13:02:00.556529  7707 consensus_meta.cc:287] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.672s	user 0.005s	sys 0.015s
I20260709 13:02:00.504024  8165 raft_consensus.cc:515] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:00.524132  8154 raft_consensus.cc:740] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
W20260709 13:02:00.593775  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 322447us
W20260709 13:02:00.597987  1717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:00.602648  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:00.511932  8198 raft_consensus.cc:515] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:00.575266  8210 raft_consensus.cc:697] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:02:00.607362  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:00.611785  1680 raft_consensus.cc:3060] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Advancing to term 18
W20260709 13:02:00.613951  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 115130us
I20260709 13:02:00.597987  1643 leader_election.cc:304] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:02:00.616976  8220 consensus_queue.cc:1048] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 13:02:00.617460  1643 leader_election.cc:304] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:02:00.521090  7987 raft_consensus.cc:515] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:00.619026  1676 raft_consensus.cc:1240] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 13. Current term is 14. Ops: []
I20260709 13:02:00.619004  1643 leader_election.cc:304] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 13:02:00.619791  8220 consensus_queue.cc:1048] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 13:02:00.620334  7987 leader_election.cc:290] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 14 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:00.621109  7987 raft_consensus.cc:2749] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:02:00.621537  1643 leader_election.cc:304] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:00.621780  7987 raft_consensus.cc:2804] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Leader election won for term 18
I20260709 13:02:00.603601  1600 raft_consensus.cc:2468] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 16.
I20260709 13:02:00.623854  1643 leader_election.cc:304] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:02:00.622607  1717 leader_election.cc:304] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:00.623170  8023 consensus_queue.cc:1048] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.005s
I20260709 13:02:00.628051  7856 leader_election.cc:290] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 17 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:00.626116  8233 consensus_queue.cc:1048] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 7, Time since last communication: 0.006s
I20260709 13:02:00.630532  7687 leader_election.cc:290] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 16 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:00.633296  1717 leader_election.cc:304] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:00.638753  1717 leader_election.cc:304] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:00.639338  1680 raft_consensus.cc:2468] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 18.
I20260709 13:02:00.638823  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:00.646862  1675 raft_consensus.cc:2393] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 15: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:02:00.647316  1600 raft_consensus.cc:1240] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 12. Current term is 13. Ops: [12.10-12.10]
I20260709 13:02:00.592523  7707 raft_consensus.cc:515] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:00.580529  8228 consensus_queue.cc:1059] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 13.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.058s
I20260709 13:02:00.588517  8225 consensus_queue.cc:1048] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.059s
I20260709 13:02:00.622380  7987 raft_consensus.cc:697] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:02:00.645152  8023 raft_consensus.cc:2804] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:02:00.683874  7946 leader_election.cc:290] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:02:00.686895  8234 log.cc:927] Time spent T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.056s	user 0.001s	sys 0.000s
I20260709 13:02:00.689332  8138 leader_election.cc:290] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 18 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:00.626214  8058 raft_consensus.cc:515] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:00.635078  7910 raft_consensus.cc:2749] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:02:00.669656  1600 raft_consensus.cc:1240] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 18. Current term is 19. Ops: [18.9-18.9]
W20260709 13:02:00.698827  1645 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:00.656834  1717 leader_election.cc:304] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:02:00.667869  8220 raft_consensus.cc:3060] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:02:00.726732  8241 raft_consensus.cc:2804] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Leader election won for term 18
I20260709 13:02:00.727356  8241 raft_consensus.cc:697] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 18 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:02:00.728401  8241 consensus_queue.cc:237] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 16.8, Last appended by leader: 8, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:00.693423  7987 consensus_queue.cc:237] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.11, Last appended by leader: 11, Current term: 18, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:00.697463  7981 consensus_meta.cc:287] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.504s	user 0.010s	sys 0.002s
I20260709 13:02:00.733261  7981 raft_consensus.cc:515] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:00.735564  7981 leader_election.cc:290] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 15 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:00.737277  8198 leader_election.cc:290] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:00.742030  1569 leader_election.cc:304] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
W20260709 13:02:00.725382  8229 log.cc:927] Time spent T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.053s	user 0.001s	sys 0.000s
I20260709 13:02:00.652638  8236 raft_consensus.cc:2749] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:02:00.742578  7687 raft_consensus.cc:493] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:00.742995  8217 leader_election.cc:290] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:00.744143  8228 consensus_queue.cc:1059] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 16.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 13:02:00.746832  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:00.746984  1675 raft_consensus.cc:1275] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 18 index: 14. Preceding OpId from leader: term: 19 index: 15. (index mismatch)
I20260709 13:02:00.747442  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 16 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:00.645773  8227 consensus_queue.cc:1059] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 18.11, Next index: 12, Last known committed idx: 10, Time since last communication: 0.083s
W20260709 13:02:00.752328  8224 log.cc:927] Time spent T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.109s	user 0.000s	sys 0.000s
I20260709 13:02:00.753720  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
W20260709 13:02:00.822834  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 53268.7us
W20260709 13:02:00.766625  7952 consensus_meta.cc:287] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.546s	user 0.012s	sys 0.004s
I20260709 13:02:00.675501  8023 raft_consensus.cc:493] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:00.836606  8023 raft_consensus.cc:3060] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:02:00.843044  1599 raft_consensus.cc:2410] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for term 18 because replica has last-logged OpId of term: 16 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 10.
W20260709 13:02:00.843954  2300 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:00.851119  8023 raft_consensus.cc:515] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:00.852451  8058 leader_election.cc:290] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 18 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:00.855400  8023 leader_election.cc:290] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:02:00.859851  7902 consensus_meta.cc:287] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.951s	user 0.010s	sys 0.007s
I20260709 13:02:00.861979  7707 leader_election.cc:290] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 20 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:00.864488  8219 raft_consensus.cc:2749] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Leader pre-election lost for term 13. Reason: Vote denied by peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445) with higher term. Message: Invalid argument: T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for earlier term 13. Current term is 14.
I20260709 13:02:00.869537  8165 leader_election.cc:290] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:02:00.872741  8003 consensus_meta.cc:287] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 1.371s	user 0.007s	sys 0.007s
I20260709 13:02:00.873876  1746 raft_consensus.cc:1275] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
W20260709 13:02:00.875643  8117 consensus_meta.cc:287] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 1.006s	user 0.016s	sys 0.001s
I20260709 13:02:00.876236  8117 raft_consensus.cc:515] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:00.877914  1752 raft_consensus.cc:2368] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 17. Current term is 18.
I20260709 13:02:00.880429  8117 leader_election.cc:290] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 18 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:00.884361  1747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55848 (request call id 3060) took 1445 ms. Trace:
I20260709 13:02:00.875635  8220 raft_consensus.cc:2749] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:02:00.885499  1605 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40166 (request call id 2783) took 1407 ms. Trace:
I20260709 13:02:00.885987  1605 rpcz_store.cc:276] 0709 13:01:59.477938 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:59.478343 (+   405us) service_pool.cc:224] Handling call
0709 13:02:00.885257 (+1406914us) spinlock_profiling.cc:243] Waited 86.5 ms on lock 0x7b54001e0354. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:02:00.885415 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":199024512}
I20260709 13:02:00.888271  1605 raft_consensus.cc:1275] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 15 index: 9. Preceding OpId from leader: term: 16 index: 9. (term mismatch)
I20260709 13:02:00.888649  1605 pending_rounds.cc:85] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a: Aborting all ops after (but not including) 8
I20260709 13:02:00.888932  1605 pending_rounds.cc:107] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a: Aborting uncommitted NO_OP operation due to leader change: 15.9
I20260709 13:02:00.889235  1605 raft_consensus.cc:2889] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:02:00.890087  1605 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 2806) took 1066 ms. Trace:
I20260709 13:02:00.753893  8210 consensus_queue.cc:237] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.9, Last appended by leader: 10, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:00.724850  7910 raft_consensus.cc:3060] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:02:00.824610  1569 leader_election.cc:304] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
W20260709 13:02:00.900177  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 75639.7us
I20260709 13:02:00.901233  1569 leader_election.cc:304] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
W20260709 13:02:00.756392  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 99611.2us
I20260709 13:02:00.826922  8242 raft_consensus.cc:2749] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 13:02:00.827523  8163 raft_consensus.cc:493] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:00.830762  8227 consensus_queue.cc:1243] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer af520b67945d443b901bbeca37cc5701 log is divergent from this leader: its last log entry 19.12 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 10
I20260709 13:02:00.765776  8055 raft_consensus.cc:2804] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Leader pre-election won for term 16
W20260709 13:02:00.909016  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:00.916877  1605 rpcz_store.cc:276] 0709 13:01:59.823355 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:00.887323 (+1063968us) service_pool.cc:224] Handling call
0709 13:02:00.887766 (+   443us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:00.889772 (+  2006us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:00.889999 (+   227us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:02:00.979805  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 78681.7us
W20260709 13:02:00.991143  1646 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:00.991950  1678 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.526s	user 0.029s	sys 0.073s
W20260709 13:02:01.180557  1678 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.715s	user 0.030s	sys 0.074s
I20260709 13:02:00.884905  1747 rpcz_store.cc:276] 0709 13:01:59.438728 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:59.722221 (+283493us) service_pool.cc:224] Handling call
0709 13:01:59.727023 (+  4802us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:59.862653 (+135630us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 13:02:00.134712 (+272059us) log.cc:844] Serialized 14 byte log entry
0709 13:02:00.625154 (+490442us) log.cc:844] Serialized 14 byte log entry
0709 13:02:00.625359 (+   205us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:00.625412 (+    53us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:02:00.625465 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:00.883989 (+258524us) spinlock_profiling.cc:243] Waited 135 ms on lock 0x7b54001711d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:02:00.884071 (+    82us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:00.884268 (+   197us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":14185,"prepare.queue_time_us":462807,"prepare.run_cpu_time_us":70,"prepare.run_wall_time_us":123,"raft.queue_time_us":23174,"raft.run_cpu_time_us":236,"raft.run_wall_time_us":10534,"spinlock_wait_cycles":330900608,"thread_start_us":726230,"threads_started":3,"wal-append.queue_time_us":292488}
W20260709 13:02:01.100358  1717 leader_election.cc:336] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 17 election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 1.975s (SENT)
W20260709 13:02:01.182250  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 81995.6us
W20260709 13:02:00.935202  8127 consensus_meta.cc:287] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.816s	user 0.016s	sys 0.000s
I20260709 13:02:01.190415  1569 leader_election.cc:304] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:02:00.768079  7687 raft_consensus.cc:515] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:01.259070  7687 leader_election.cc:290] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:01.173272  8023 raft_consensus.cc:493] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:01.268217  8023 raft_consensus.cc:515] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:01.209270  7952 consensus_queue.cc:1048] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 7, Time since last communication: 0.062s
I20260709 13:02:01.174711  8198 consensus_queue.cc:1059] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 18.11, Next index: 12, Last known committed idx: 10, Time since last communication: 0.131s
I20260709 13:02:01.216140  8242 raft_consensus.cc:2804] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Leader election won for term 18
I20260709 13:02:01.275671  8242 raft_consensus.cc:697] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:02:01.277004  8242 consensus_queue.cc:237] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.9, Last appended by leader: 9, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:01.192301  8227 raft_consensus.cc:493] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:01.278829  8227 raft_consensus.cc:515] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:01.281713  8227 leader_election.cc:290] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:01.176426  8157 raft_consensus.cc:2749] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 13:02:01.245028  7728 raft_consensus.cc:493] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:01.287171  7728 raft_consensus.cc:515] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:01.289980  7728 leader_election.cc:290] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:02:01.261402  8068 consensus_meta.cc:287] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 1.410s	user 0.005s	sys 0.009s
I20260709 13:02:01.219822  8127 raft_consensus.cc:515] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:01.187609  8178 raft_consensus.cc:2749] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 13:02:01.184057  7812 raft_consensus.cc:493] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:01.184731  8163 raft_consensus.cc:515] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:01.173780  8055 raft_consensus.cc:493] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:01.252508  1605 raft_consensus.cc:1240] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 LEADER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 18. Current term is 19. Ops: []
I20260709 13:02:01.316327  2334 raft_consensus.cc:2468] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 14.
W20260709 13:02:01.324215  2300 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:01.335630  1605 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 1608) took 1495 ms. Trace:
I20260709 13:02:01.336194  1605 rpcz_store.cc:276] 0709 13:01:59.840502 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:01.196042 (+1355540us) service_pool.cc:224] Handling call
0709 13:02:01.252230 (+ 56188us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:01.334711 (+ 82481us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:01.335408 (+   697us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:01.339990  1598 raft_consensus.cc:2368] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for earlier term 14. Current term is 15.
I20260709 13:02:01.338711  8055 raft_consensus.cc:3060] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:02:01.333178  7812 raft_consensus.cc:515] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:01.399382  8003 raft_consensus.cc:493] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:01.427827  8163 leader_election.cc:290] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:01.429896  8127 leader_election.cc:290] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 15 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:01.403898  8023 leader_election.cc:290] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:01.469420  8003 raft_consensus.cc:515] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:01.491245  8003 leader_election.cc:290] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:01.498818  7812 leader_election.cc:290] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:02:01.545187  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 362089us
W20260709 13:02:01.582700  1717 leader_election.cc:336] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 2.016s (SENT)
I20260709 13:02:01.583312  1717 leader_election.cc:304] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:01.583593  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 15 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:01.557413  7585 raft_consensus.cc:493] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:01.621028  7585 raft_consensus.cc:3060] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:02:01.574337  1600 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 2805) took 1751 ms. Trace:
I20260709 13:02:01.622723  1600 rpcz_store.cc:276] 0709 13:01:59.823059 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:00.730487 (+907428us) service_pool.cc:224] Handling call
0709 13:02:00.730949 (+   462us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:02:00.731433 (+   484us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:00.731467 (+    34us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:02:00.732210 (+   743us) log.cc:844] Serialized 52 byte log entry
0709 13:02:01.131798 (+399588us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:02:01.544305 (+412507us) log.cc:844] Serialized 14 byte log entry
0709 13:02:01.544861 (+   556us) log.cc:844] Serialized 14 byte log entry
0709 13:02:01.573965 (+ 29104us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:01.574064 (+    99us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:01.574109 (+    45us) raft_consensus.cc:1685] finished
0709 13:02:01.574121 (+    12us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:01.574269 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":28811,"raft.queue_time_us":37017,"raft.run_cpu_time_us":469,"raft.run_wall_time_us":14756,"thread_start_us":750906,"threads_started":2,"wal-append.queue_time_us":750675}
W20260709 13:02:01.554090  8237 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.728s	user 0.000s	sys 0.009s
W20260709 13:02:01.572947  8241 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.820s	user 0.010s	sys 0.032s
I20260709 13:02:01.585403  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:02:01.629714  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 47087.4us
I20260709 13:02:01.630229  1680 raft_consensus.cc:2468] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 13.
I20260709 13:02:01.552394  1605 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 1610) took 1711 ms. Trace:
I20260709 13:02:01.588778  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
W20260709 13:02:01.631624  2299 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:01.632041  1672 raft_consensus.cc:3060] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Advancing to term 15
I20260709 13:02:01.599448  1601 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 2779) took 2138 ms. Trace:
I20260709 13:02:01.632797  1605 rpcz_store.cc:276] 0709 13:01:59.841221 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:01.534304 (+1693083us) service_pool.cc:224] Handling call
0709 13:02:01.551574 (+ 17270us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:01.551866 (+   292us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:01.551908 (+    42us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:01.551972 (+    64us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:02:01.552034 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:01.552113 (+    79us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:01.552296 (+   183us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:02:01.629505  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:01.631384  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:01.628046  1569 leader_election.cc:304] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d
W20260709 13:02:01.584210  1677 consensus_meta.cc:287] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.719s	user 0.010s	sys 0.003s
I20260709 13:02:01.636909  1569 leader_election.cc:304] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
W20260709 13:02:01.596756  1646 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:01.635296  1680 raft_consensus.cc:2376] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 18. Re-sending same reply.
I20260709 13:02:01.639791  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:01.640609  1680 raft_consensus.cc:2468] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 19.
I20260709 13:02:01.637063  1677 raft_consensus.cc:2468] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 14.
I20260709 13:02:01.641647  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:01.643755  1569 leader_election.cc:304] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
W20260709 13:02:01.604841  1596 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40166 (request call id 2812) took 1753 ms (client timeout 1801 ms). Trace:
W20260709 13:02:01.645023  1596 rpcz_store.cc:269] 0709 13:01:59.851368 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:59.916725 (+ 65357us) service_pool.cc:224] Handling call
0709 13:02:01.604593 (+1687868us) spinlock_profiling.cc:243] Waited 520 ms on lock 0x7b54003edfd4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:02:01.604748 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1195468160}
I20260709 13:02:01.632555  1601 rpcz_store.cc:276] 0709 13:01:59.460679 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:59.460818 (+   139us) service_pool.cc:224] Handling call
0709 13:01:59.461155 (+   337us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:00.328099 (+866944us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:01.599025 (+1270926us) spinlock_profiling.cc:243] Waited 843 ms on lock 0x7b54000ab4d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:02:01.599337 (+   312us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1937883904}
I20260709 13:02:01.647579  1601 raft_consensus.cc:1240] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 13. Current term is 14. Ops: []
I20260709 13:02:01.648409  1601 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 1619) took 1767 ms. Trace:
I20260709 13:02:01.648875  1601 rpcz_store.cc:276] 0709 13:01:59.884049 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:01.646934 (+1762885us) service_pool.cc:224] Handling call
0709 13:02:01.647324 (+   390us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:01.648087 (+   763us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:01.648333 (+   246us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:01.655388  1596 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 1618) took 1782 ms. Trace:
I20260709 13:02:01.655969  1596 rpcz_store.cc:276] 0709 13:01:59.872911 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:01.646201 (+1773290us) service_pool.cc:224] Handling call
0709 13:02:01.654661 (+  8460us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:01.654929 (+   268us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:01.654967 (+    38us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:01.655041 (+    74us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:02:01.655109 (+    68us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:01.655181 (+    72us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:01.655309 (+   128us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:02:01.845829  1643 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.195s	user 0.009s	sys 0.003s
W20260709 13:02:01.686653  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 47837.4us
I20260709 13:02:01.684306  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 19 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:01.886409  1604 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 2780) took 2425 ms. Trace:
I20260709 13:02:01.867455  8262 raft_consensus.cc:493] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:01.852891  1602 raft_consensus.cc:1275] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 14 index: 13. Preceding OpId from leader: term: 16 index: 14. (index mismatch)
I20260709 13:02:01.693434  1603 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 1612) took 1851 ms. Trace:
W20260709 13:02:01.728694  7910 consensus_meta.cc:287] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.584s	user 0.012s	sys 0.001s
I20260709 13:02:01.895512  7910 raft_consensus.cc:2749] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260709 13:02:01.704586  8247 log.cc:927] Time spent T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.066s	user 0.001s	sys 0.000s
I20260709 13:02:01.687619  7981 raft_consensus.cc:2749] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 13:02:01.903213  1596 raft_consensus.cc:1240] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 19. Current term is 20. Ops: []
W20260709 13:02:01.635706  8237 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.802s	user 0.001s	sys 0.009s
I20260709 13:02:01.904505  1596 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40154 (request call id 2699) took 1852 ms. Trace:
I20260709 13:02:01.664454  8227 raft_consensus.cc:2749] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:02:01.731992  8145 raft_consensus.cc:493] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:01.723500  8190 raft_consensus.cc:493] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:01.688153  8163 raft_consensus.cc:493] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:01.626871  8115 consensus_meta.cc:287] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 1.709s	user 0.018s	sys 0.002s
I20260709 13:02:01.922356  1677 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52766 (request call id 3029) took 1447 ms. Trace:
I20260709 13:02:01.921998  8163 raft_consensus.cc:515] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:01.922209  8115 raft_consensus.cc:515] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:01.922945  1677 rpcz_store.cc:276] 0709 13:02:00.475245 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:00.496037 (+ 20792us) service_pool.cc:224] Handling call
0709 13:02:01.921799 (+1425762us) spinlock_profiling.cc:243] Waited 367 ms on lock 0x7b54000d7d54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:02:01.922113 (+   314us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":843340160}
I20260709 13:02:01.925426  8163 leader_election.cc:290] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:01.702010  8068 raft_consensus.cc:493] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:01.687873  8244 log.cc:927] Time spent T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.123s	user 0.000s	sys 0.000s
I20260709 13:02:01.930783  1605 raft_consensus.cc:1240] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 21. Current term is 22. Ops: []
I20260709 13:02:01.790520  8148 raft_consensus.cc:3055] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 15 LEADER]: Stepping down as leader of term 15
I20260709 13:02:01.765221  8123 raft_consensus.cc:3055] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 LEADER]: Stepping down as leader of term 18
I20260709 13:02:01.932528  1671 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52766 (request call id 3022) took 1472 ms. Trace:
I20260709 13:02:01.932732  8123 raft_consensus.cc:740] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:02:01.932363  8190 raft_consensus.cc:515] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:01.789503  8211 raft_consensus.cc:493] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:01.801751  8140 raft_consensus.cc:493] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Starting leader election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:02:01.795671  8129 leader_election.cc:290] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:01.932353  8148 raft_consensus.cc:740] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
W20260709 13:02:01.739786  1752 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55840 (request call id 2942) took 1711 ms (client timeout 2110 ms). Trace:
I20260709 13:02:01.942773  8068 raft_consensus.cc:515] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:01.943102  1752 rpcz_store.cc:269] 0709 13:02:00.028180 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:00.091620 (+ 63440us) service_pool.cc:224] Handling call
0709 13:02:01.739457 (+1647837us) spinlock_profiling.cc:243] Waited 784 ms on lock 0x7b540019e6d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:02:01.739689 (+   232us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1804171904}
I20260709 13:02:01.662789  7687 raft_consensus.cc:2764] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader pre-election decision vote started in defunct term 14: won
I20260709 13:02:01.573052  7728 raft_consensus.cc:493] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:01.683611  7812 raft_consensus.cc:493] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:01.948589  1750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55848 (request call id 3054) took 2511 ms. Trace:
I20260709 13:02:01.636672  7915 raft_consensus.cc:493] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:02:01.951566  7915 raft_consensus.cc:515] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:02:01.724602  7987 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.987s	user 0.189s	sys 0.435s
I20260709 13:02:01.936353  8211 raft_consensus.cc:515] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:01.955024  7987 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.219s	user 0.190s	sys 0.436s
I20260709 13:02:01.649480  8003 raft_consensus.cc:2749] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20260709 13:02:01.627022  8009 consensus_meta.cc:287] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 1.403s	user 0.019s	sys 0.009s
I20260709 13:02:01.957572  8211 leader_election.cc:290] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:01.685400  8245 raft_consensus.cc:493] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:01.695044  2301 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:01.957901  8009 raft_consensus.cc:515] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:01.768182  8154 consensus_queue.cc:260] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.10, Last appended by leader: 10, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:01.967926  1605 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 1623) took 2016 ms. Trace:
I20260709 13:02:01.968850  8154 raft_consensus.cc:3060] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:02:01.969441  7307 raft_consensus.cc:3055] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 17 LEADER]: Stepping down as leader of term 17
W20260709 13:02:01.742065  8210 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.847s	user 0.003s	sys 0.015s
W20260709 13:02:01.971052  8210 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.076s	user 0.003s	sys 0.016s
I20260709 13:02:01.970945  1673 raft_consensus.cc:2468] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 16.
W20260709 13:02:01.725487  8146 consensus_meta.cc:287] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 1.257s	user 0.016s	sys 0.000s
I20260709 13:02:01.972869  8129 raft_consensus.cc:493] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:01.973320  8146 raft_consensus.cc:493] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:01.849851  1600 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 1616) took 1983 ms. Trace:
I20260709 13:02:01.973884  8129 raft_consensus.cc:515] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:02:01.973807  2334 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39726 (request call id 1467) took 2366 ms (client timeout 1823 ms). Trace:
I20260709 13:02:01.887130  1604 rpcz_store.cc:276] 0709 13:01:59.461020 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:59.461336 (+   316us) service_pool.cc:224] Handling call
0709 13:01:59.461681 (+   345us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:01:59.481534 (+ 19853us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:59.481569 (+    35us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:01:59.486174 (+  4605us) log.cc:844] Serialized 52 byte log entry
0709 13:01:59.885751 (+399577us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:01:59.885794 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:01.885262 (+1999468us) spinlock_profiling.cc:243] Waited 19.4 ms on lock 0x7b540029aed4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:02:01.885405 (+   143us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:01.885516 (+   111us) raft_consensus.cc:1685] finished
0709 13:02:01.885541 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:01.886129 (+   588us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":2239,"raft.run_cpu_time_us":150,"raft.run_wall_time_us":2395,"spinlock_wait_cycles":44626304,"thread_start_us":311680,"threads_started":1,"wal-append.queue_time_us":517639,"wal-append.run_cpu_time_us":2073,"wal-append.run_wall_time_us":1018359}
I20260709 13:02:01.892118  1602 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 1611) took 2050 ms. Trace:
I20260709 13:02:01.889953  1680 raft_consensus.cc:1275] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 16 index: 11. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
W20260709 13:02:01.891235  1643 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.240s	user 0.010s	sys 0.003s
I20260709 13:02:01.894866  1603 rpcz_store.cc:276] 0709 13:01:59.841858 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:01.630985 (+1789127us) service_pool.cc:224] Handling call
0709 13:02:01.692736 (+ 61751us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:01.693003 (+   267us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:01.693038 (+    35us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:01.693096 (+    58us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:02:01.693145 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:01.693216 (+    71us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:01.693349 (+   133us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:01.900861  8235 raft_consensus.cc:493] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:01.949102  1750 rpcz_store.cc:276] 0709 13:01:59.436688 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:59.534571 (+ 97883us) service_pool.cc:224] Handling call
0709 13:01:59.616034 (+ 81463us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:59.810327 (+194293us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:59.810380 (+    53us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:01:59.810481 (+   101us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:01:59.810534 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:01.947903 (+2137369us) spinlock_profiling.cc:243] Waited 194 ms on lock 0x7b54000e4054. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:02:01.948022 (+   119us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:01.948298 (+   276us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":446064384}
I20260709 13:02:01.905154  1596 rpcz_store.cc:276] 0709 13:02:00.052363 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:01.900705 (+1848342us) service_pool.cc:224] Handling call
0709 13:02:01.901527 (+   822us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:01.903970 (+  2443us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:01.904375 (+   405us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:01.910014  8145 raft_consensus.cc:3060] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:02:01.747041  8055 raft_consensus.cc:515] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:02:01.912774  2300 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:01.917045  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 273192us
I20260709 13:02:01.925904  8115 leader_election.cc:290] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 14 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:02:01.646492  8241 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.894s	user 0.011s	sys 0.032s
I20260709 13:02:01.992830  8235 raft_consensus.cc:515] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:01.994395  8068 leader_election.cc:290] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:01.995100  8055 leader_election.cc:290] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:01.996340  8235 leader_election.cc:290] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:01.907171  7910 raft_consensus.cc:493] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:01.938627  8140 raft_consensus.cc:3060] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Advancing to term 15
W20260709 13:02:01.998767  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 1.37487e+06us
I20260709 13:02:01.999161  7910 raft_consensus.cc:515] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:01.946349  1672 raft_consensus.cc:2468] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 15.
W20260709 13:02:01.979220  2334 rpcz_store.cc:269] 0709 13:01:59.607253 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:59.608445 (+  1192us) service_pool.cc:224] Handling call
0709 13:02:01.973205 (+2364760us) spinlock_profiling.cc:243] Waited 1.7 s on lock 0x7b54005e01d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:02:01.973418 (+   213us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3909703040}
I20260709 13:02:01.995158  8068 raft_consensus.cc:493] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:01.947005  8262 raft_consensus.cc:515] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:02.004307  8068 raft_consensus.cc:515] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:01.947425  7728 raft_consensus.cc:515] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:01.974263  8146 raft_consensus.cc:515] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:01.936372  7745 raft_consensus.cc:3055] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 LEADER]: Stepping down as leader of term 18
I20260709 13:02:02.008762  7745 raft_consensus.cc:740] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:02:02.009140  8146 leader_election.cc:290] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:02.010456  7745 consensus_queue.cc:260] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 18.10, Last appended by leader: 10, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:02.012161  8190 leader_election.cc:290] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:02.012949  8190 raft_consensus.cc:493] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:02:02.013476  8009 leader_election.cc:290] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 16 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:01.919787  8261 raft_consensus.cc:493] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:02.001716  1502 catalog_manager.cc:5697] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 reported cstate change: term changed from 11 to 13. New cstate: current_term: 13 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 13:02:02.015066  8261 raft_consensus.cc:515] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:02.015470  1569 leader_election.cc:304] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:02:02.016402  1502 catalog_manager.cc:5697] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 reported cstate change: term changed from 10 to 13. New cstate: current_term: 13 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:02:01.960239  8245 raft_consensus.cc:515] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:02.017858  1569 leader_election.cc:304] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:02:02.018754  7981 raft_consensus.cc:2749] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:02:02.019969  8245 leader_election.cc:290] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:01.970135  7307 raft_consensus.cc:740] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
W20260709 13:02:02.023113  1673 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57092 (request call id 1428) took 1608 ms (client timeout 1927 ms). Trace:
I20260709 13:02:01.975518  1599 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40166 (request call id 2793) took 2242 ms. Trace:
I20260709 13:02:02.023669  1676 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52766 (request call id 3023) took 1562 ms. Trace:
I20260709 13:02:02.023854  1599 rpcz_store.cc:276] 0709 13:01:59.733338 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:59.733481 (+   143us) service_pool.cc:224] Handling call
0709 13:02:01.974987 (+2241506us) spinlock_profiling.cc:243] Waited 1.11 s on lock 0x7b54002175d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:02:01.975208 (+   221us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2548486016}
I20260709 13:02:02.024689  1676 rpcz_store.cc:276] 0709 13:02:00.462388 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:00.462848 (+   460us) service_pool.cc:224] Handling call
0709 13:02:00.463166 (+   318us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:00.619553 (+156387us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:02.023006 (+1403453us) spinlock_profiling.cc:243] Waited 156 ms on lock 0x7b54001e87d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:02:02.023407 (+   401us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":357954432}
I20260709 13:02:02.025440  1601 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 1621) took 2105 ms. Trace:
I20260709 13:02:01.979487  1602 rpcz_store.cc:276] 0709 13:01:59.841545 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:01.631456 (+1789911us) service_pool.cc:224] Handling call
0709 13:02:01.851473 (+220017us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:01.891740 (+ 40267us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:01.892011 (+   271us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:02:01.982837  1598 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40166 (request call id 2816) took 2093 ms (client timeout 1884 ms). Trace:
I20260709 13:02:02.022116  7307 consensus_queue.cc:260] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.12, Last appended by leader: 12, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:01.983822  1746 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55840 (request call id 2935) took 1915 ms. Trace:
I20260709 13:02:01.978912  1600 rpcz_store.cc:276] 0709 13:01:59.866545 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:01.642072 (+1775527us) service_pool.cc:224] Handling call
0709 13:02:01.848429 (+206357us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:01.849072 (+   643us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:01.849144 (+    72us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:01.849234 (+    90us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:02:01.849290 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:01.849388 (+    98us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:01.849594 (+   206us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:01.949669  7812 raft_consensus.cc:515] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:02.006156  8262 leader_election.cc:290] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:01.989642  8129 leader_election.cc:290] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:01.935760  8123 consensus_queue.cc:260] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.11, Last appended by leader: 11, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:01.933094  1671 rpcz_store.cc:276] 0709 13:02:00.460340 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:00.460791 (+   451us) service_pool.cc:224] Handling call
0709 13:02:00.461193 (+   402us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:00.590789 (+129596us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:00.590826 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:00.591634 (+   808us) log.cc:844] Serialized 26 byte log entry
0709 13:02:00.755598 (+163964us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:02:00.755646 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:00.755720 (+    74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:01.784133 (+1028413us) spinlock_profiling.cc:243] Waited 9.88 ms on lock 0x7b48002ac558. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426b08baa1 00007f426b082b37 00007f426b0829ca 00007f426b17a26f 00007f426b179f37 00007f426b18e55c 00007f426b18e508 00007f426b18e488 00007f426b18e443 00007f426b18d6ec 00007f426a0b7a96
0709 13:02:01.931790 (+147657us) raft_consensus.cc:1685] finished
0709 13:02:01.931849 (+    59us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:01.932301 (+   452us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":124041,"raft.run_cpu_time_us":45173,"raft.run_wall_time_us":1194484,"spinlock_wait_cycles":22715776,"thread_start_us":255514,"threads_started":2,"wal-append.queue_time_us":910488}
I20260709 13:02:02.026540  7728 leader_election.cc:290] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:02:02.027185  1598 rpcz_store.cc:269] 0709 13:01:59.882555 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:59.921306 (+ 38751us) service_pool.cc:224] Handling call
0709 13:02:01.975452 (+2054146us) spinlock_profiling.cc:243] Waited 1.38 s on lock 0x7b540028ad54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:02:01.975752 (+   300us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3178815360}
I20260709 13:02:02.025969  1601 rpcz_store.cc:276] 0709 13:01:59.919920 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:01.650727 (+1730807us) service_pool.cc:224] Handling call
0709 13:02:01.651115 (+   388us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:01.920429 (+269314us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:01.927597 (+  7168us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:01.927668 (+    71us) raft_consensus.cc:1645] Marking committed up to 14
0709 13:02:01.927775 (+   107us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:02.024929 (+ 97154us) spinlock_profiling.cc:243] Waited 255 ms on lock 0x7b54002170d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:02:02.024990 (+    61us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:02.025227 (+   237us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":587393408}
I20260709 13:02:01.968544  1605 rpcz_store.cc:276] 0709 13:01:59.951153 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:01.854964 (+1903811us) service_pool.cc:224] Handling call
0709 13:02:01.855608 (+   644us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:01.931412 (+ 75804us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:01.967421 (+ 36009us) spinlock_profiling.cc:243] Waited 51 ms on lock 0x7b54002157d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:02:01.967801 (+   380us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":117320576}
I20260709 13:02:02.043406  8068 leader_election.cc:290] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:02.044140  7910 leader_election.cc:290] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:01.980012  8163 raft_consensus.cc:493] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:02.045303  8163 raft_consensus.cc:515] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:02.046865  8123 raft_consensus.cc:3060] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:02:02.023685  1604 raft_consensus.cc:1240] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 17. Current term is 18. Ops: []
I20260709 13:02:02.049163  1604 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40154 (request call id 2705) took 1967 ms. Trace:
I20260709 13:02:02.049600  1604 rpcz_store.cc:276] 0709 13:02:00.081678 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:02.022676 (+1940998us) service_pool.cc:224] Handling call
0709 13:02:02.023330 (+   654us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:02.048723 (+ 25393us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:02.049021 (+   298us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:02:02.050180  1717 leader_election.cc:336] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 15 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 1.983s (SENT)
I20260709 13:02:01.961063  1678 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52766 (request call id 3024) took 1418 ms. Trace:
I20260709 13:02:02.031579  1596 raft_consensus.cc:1240] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 17. Current term is 18. Ops: []
W20260709 13:02:02.023660  1673 rpcz_store.cc:269] 0709 13:02:00.414327 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:00.414534 (+   207us) service_pool.cc:224] Handling call
0709 13:02:02.022840 (+1608306us) spinlock_profiling.cc:243] Waited 1.55 s on lock 0x7b54000230d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:02:02.023011 (+   171us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3559271296}
I20260709 13:02:01.941829  8148 consensus_queue.cc:260] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:02.028617  7307 raft_consensus.cc:3060] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:02:02.034215  7812 leader_election.cc:290] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:02.065011  8148 raft_consensus.cc:3060] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:02:02.065477  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:02.030953  1746 rpcz_store.cc:276] 0709 13:02:00.022059 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:00.024179 (+  2120us) service_pool.cc:224] Handling call
0709 13:02:00.084710 (+ 60531us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:00.874319 (+789609us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:01.936245 (+1061926us) spinlock_profiling.cc:243] Waited 789 ms on lock 0x7b540035ff54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:02:01.937398 (+  1153us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1813656832}
I20260709 13:02:02.017858  8261 leader_election.cc:290] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:02.066255  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:02.059173  1596 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40154 (request call id 2701) took 2006 ms. Trace:
I20260709 13:02:02.067335  1601 raft_consensus.cc:3060] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:02:02.067752  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:02.068497  1600 raft_consensus.cc:3060] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Advancing to term 18
I20260709 13:02:02.013729  8190 raft_consensus.cc:515] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:02.071661  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 19 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:02.072372  1598 raft_consensus.cc:2468] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 18.
I20260709 13:02:02.065481  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:02.073477  8190 leader_election.cc:290] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:02.067041  1596 rpcz_store.cc:276] 0709 13:02:00.052906 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:02.030502 (+1977596us) service_pool.cc:224] Handling call
0709 13:02:02.031167 (+   665us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:02.058885 (+ 27718us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:02.059109 (+   224us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:02.062250  7745 raft_consensus.cc:3060] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:02:02.062721  7981 raft_consensus.cc:2804] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:02:02.063156  7915 leader_election.cc:290] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:02.063738  1717 leader_election.cc:304] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:02.063831  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:02.087001  1602 raft_consensus.cc:2393] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 13: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
W20260709 13:02:02.087167  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 85146.4us
W20260709 13:02:02.087651  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 149679us
I20260709 13:02:02.089365  1643 leader_election.cc:304] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:02:02.051563  1678 rpcz_store.cc:276] 0709 13:02:00.464712 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:00.465060 (+   348us) service_pool.cc:224] Handling call
0709 13:02:00.465479 (+   419us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:02:01.180982 (+715503us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 13:02:01.544788 (+363806us) log.cc:844] Serialized 14 byte log entry
0709 13:02:01.870235 (+325447us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:02:01.871734 (+  1499us) log.cc:844] Serialized 52 byte log entry
0709 13:02:01.871922 (+   188us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:02:01.871957 (+    35us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:01.872130 (+   173us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:01.874038 (+  1908us) raft_consensus.cc:1685] finished
0709 13:02:01.874063 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:01.882342 (+  8279us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":304685,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":48,"raft.queue_time_us":120095,"raft.run_cpu_time_us":502,"raft.run_wall_time_us":9922,"spinlock_wait_cycles":35584,"thread_start_us":1350734,"threads_started":3,"wal-append.queue_time_us":307055}
I20260709 13:02:02.090632  7812 raft_consensus.cc:2749] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
W20260709 13:02:02.105086  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 55186.9us
I20260709 13:02:02.105792  8211 raft_consensus.cc:2749] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
W20260709 13:02:02.106353  1717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:02.107494  1717 leader_election.cc:336] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 17 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 2.200s (SENT)
I20260709 13:02:02.108035  1717 leader_election.cc:304] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:02.110817  7915 raft_consensus.cc:2749] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:02:02.111702  1601 raft_consensus.cc:2468] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 20.
I20260709 13:02:02.083190  7981 raft_consensus.cc:697] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:02:02.123697  7981 consensus_queue.cc:237] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.12, Last appended by leader: 12, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:02.128594  1646 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:02.110893  8268 raft_consensus.cc:493] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:02.130525  8268 raft_consensus.cc:515] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:02.133224  8268 leader_election.cc:290] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:02:02.142840  1646 leader_election.cc:336] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 15 election: RPC error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Timed out: RequestConsensusVote RPC to 127.0.107.195:40693 timed out after 1.504s (SENT)
I20260709 13:02:02.152722  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 15 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:02.156190  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:02.157212  2334 raft_consensus.cc:2468] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 17.
I20260709 13:02:02.157289  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:02.158278  2332 raft_consensus.cc:2468] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 19.
I20260709 13:02:02.158810  2335 raft_consensus.cc:1275] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 15 index: 12. (term mismatch)
I20260709 13:02:02.158977  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 19 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:02.159533  1678 raft_consensus.cc:1275] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 18 index: 15. Preceding OpId from leader: term: 19 index: 15. (term mismatch)
I20260709 13:02:02.159936  1678 pending_rounds.cc:85] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d: Aborting all ops after (but not including) 14
I20260709 13:02:02.160102  2336 raft_consensus.cc:2410] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for term 19 because replica has last-logged OpId of term: 17 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 13.
I20260709 13:02:02.160199  1678 pending_rounds.cc:107] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d: Aborting uncommitted NO_OP operation due to leader change: 18.15
I20260709 13:02:02.160327  2329 raft_consensus.cc:1275] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 17 index: 14. Preceding OpId from leader: term: 18 index: 14. (term mismatch)
I20260709 13:02:02.160573  1678 raft_consensus.cc:2889] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:02:02.160800  2329 pending_rounds.cc:85] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1: Aborting all ops after (but not including) 13
I20260709 13:02:02.161221  2329 pending_rounds.cc:107] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1: Aborting uncommitted NO_OP operation due to leader change: 17.14
I20260709 13:02:02.161476  2329 raft_consensus.cc:2889] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:02:02.159307  2335 pending_rounds.cc:85] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1: Aborting all ops after (but not including) 11
I20260709 13:02:02.162634  2335 pending_rounds.cc:107] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1: Aborting uncommitted NO_OP operation due to leader change: 14.12
I20260709 13:02:02.162940  2335 raft_consensus.cc:2889] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:02:02.163100  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:02.162745  1502 catalog_manager.cc:5697] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 reported cstate change: term changed from 12 to 14. New cstate: current_term: 14 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:02:02.163970  2337 raft_consensus.cc:2468] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 19.
I20260709 13:02:02.165109  1502 catalog_manager.cc:5697] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 reported cstate change: term changed from 12 to 14. New cstate: current_term: 14 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 13:02:02.168968  8140 raft_consensus.cc:515] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:02.172209  1569 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:02.172904  1603 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 1624) took 2050 ms. Trace:
I20260709 13:02:02.102165  8163 leader_election.cc:290] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:02.173306  1603 rpcz_store.cc:276] 0709 13:02:00.122638 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:01.997006 (+1874368us) service_pool.cc:224] Handling call
0709 13:02:01.997735 (+   729us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:01.998313 (+   578us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:01.998351 (+    38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:01.999204 (+   853us) log.cc:844] Serialized 26 byte log entry
0709 13:02:02.097333 (+ 98129us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:02:02.097362 (+    29us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:02.097427 (+    65us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:02.172648 (+ 75221us) raft_consensus.cc:1685] finished
0709 13:02:02.172673 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:02.172832 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":97596,"threads_started":1,"wal-append.queue_time_us":171145}
I20260709 13:02:02.181475  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:02.187923  1673 raft_consensus.cc:1240] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 13. Current term is 14. Ops: []
I20260709 13:02:02.189873  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:02:02.202054  1646 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:02.203426  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 114323us
I20260709 13:02:02.204738  1643 leader_election.cc:304] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:02:02.206748  1643 leader_election.cc:304] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:02:02.213177  1569 leader_election.cc:304] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:02:02.213943  8163 consensus_queue.cc:1048] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 13:02:02.214489  8260 consensus_queue.cc:1048] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 13:02:02.216228  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:02.216428  8260 raft_consensus.cc:2804] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:02:02.216868  8260 raft_consensus.cc:493] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:02.217151  2335 raft_consensus.cc:2393] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 16: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:02:02.217310  8260 raft_consensus.cc:3060] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Advancing to term 14
W20260709 13:02:02.218155  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:02.218961  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 19 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:02.219197  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:02.219569  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:02.219650  2337 raft_consensus.cc:2468] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 18.
I20260709 13:02:02.220242  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:02.220387  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:02.221136  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:02.221400  2335 raft_consensus.cc:2393] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 18: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:02:02.220623  2332 raft_consensus.cc:2393] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 18: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:02:02.219767  2336 raft_consensus.cc:3060] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:02:02.223248  2331 raft_consensus.cc:1240] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 17. Current term is 18. Ops: [17.12-17.12]
I20260709 13:02:02.224660  8237 raft_consensus.cc:2749] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:02:02.225728  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:02.226389  2331 raft_consensus.cc:2468] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 15.
I20260709 13:02:02.240370  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:02.243153  1600 raft_consensus.cc:2468] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 18.
I20260709 13:02:02.238082  8268 consensus_queue.cc:1059] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 21.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.006s
W20260709 13:02:02.244256  7585 consensus_meta.cc:287] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.622s	user 0.011s	sys 0.015s
I20260709 13:02:02.237687  7307 raft_consensus.cc:493] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:02.249187  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:02.249241  8190 raft_consensus.cc:3055] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 13:02:02.250028  8190 raft_consensus.cc:740] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:02:02.250954  8270 raft_consensus.cc:493] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:02.251123  8266 raft_consensus.cc:493] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:02.251267  8190 consensus_queue.cc:260] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.14, Last appended by leader: 14, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:02.251745  8270 raft_consensus.cc:515] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:02.251889  8266 raft_consensus.cc:515] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:02.252928  8190 raft_consensus.cc:3060] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:02:02.245848  8237 raft_consensus.cc:2804] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Leader election won for term 15
I20260709 13:02:02.257450  8237 raft_consensus.cc:697] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:02:02.258208  8145 raft_consensus.cc:515] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:02.258749  8237 consensus_queue.cc:237] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 7, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:02.261126  8145 leader_election.cc:290] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 19 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:02:02.261619  1570 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:02.262171  8140 leader_election.cc:290] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 15 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:02.269016  2336 raft_consensus.cc:2468] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 18.
I20260709 13:02:02.271840  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:02.262627  7585 raft_consensus.cc:515] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:02.273752  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:02.274890  1746 raft_consensus.cc:2393] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 18: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:02:02.275372  8129 raft_consensus.cc:493] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:02:02.276269  8129 raft_consensus.cc:515] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:02.262202  8266 leader_election.cc:290] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:02.254330  8270 leader_election.cc:290] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:02.282831  8129 leader_election.cc:290] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:02.284896  1502 catalog_manager.cc:5697] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 reported cstate change: term changed from 9 to 10, leader changed from 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) to 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196). New cstate: current_term: 10 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:02:02.292092  7307 raft_consensus.cc:515] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:02.290109  1678 raft_consensus.cc:2468] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 17.
I20260709 13:02:02.297969  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:02.300038  2332 raft_consensus.cc:2393] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 17: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:02:02.300369  7307 leader_election.cc:290] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:02.299877  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 19 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:02.301744  1746 raft_consensus.cc:2410] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for term 19 because replica has last-logged OpId of term: 17 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 13.
I20260709 13:02:02.301980  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:02.305791  1750 raft_consensus.cc:1275] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 15 index: 12. (index mismatch)
I20260709 13:02:02.305706  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:02.306545  1746 raft_consensus.cc:2376] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 15. Re-sending same reply.
I20260709 13:02:02.307580  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:02.308435  1752 raft_consensus.cc:3060] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:02:02.311549  1754 raft_consensus.cc:1240] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 15. Current term is 16. Ops: []
I20260709 13:02:02.311700  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:02.312603  1751 raft_consensus.cc:3060] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:02:02.319000  1600 raft_consensus.cc:2393] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 15: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:02:02.319957  2298 leader_election.cc:304] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
W20260709 13:02:02.320827  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 81995.1us
W20260709 13:02:02.327138  8273 log.cc:927] Time spent T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.082s	user 0.001s	sys 0.000s
I20260709 13:02:02.327473  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:02.327584  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:02.328284  1600 raft_consensus.cc:3060] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:02:02.328398  1747 raft_consensus.cc:2468] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 19.
W20260709 13:02:02.329344  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:02.330971  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:02.331797  7585 leader_election.cc:290] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 16 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:02.331966  1747 raft_consensus.cc:2393] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 16: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:02:02.334039  1754 raft_consensus.cc:3055] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 13:02:02.334571  1754 raft_consensus.cc:740] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:02:02.334334  1752 raft_consensus.cc:2468] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 14.
W20260709 13:02:02.351352  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 80572.4us
I20260709 13:02:02.342971  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:02.343602  7307 raft_consensus.cc:493] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:02:02.344089  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:02.345371  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:02.345762  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:02.347039  8268 consensus_queue.cc:1048] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 13:02:02.362255  2337 raft_consensus.cc:2468] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 18.
I20260709 13:02:02.347771  2330 raft_consensus.cc:1240] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 15. Current term is 16. Ops: []
I20260709 13:02:02.348292  2329 raft_consensus.cc:1240] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 14. Current term is 15. Ops: []
I20260709 13:02:02.349274  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:02.349551  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 17 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:02.355090  8140 raft_consensus.cc:2804] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Leader election won for term 20
W20260709 13:02:02.371876  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 165181us
I20260709 13:02:02.382164  7307 raft_consensus.cc:515] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:02.391800  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 19 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:02.393043  1747 raft_consensus.cc:2410] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for term 19 because replica has last-logged OpId of term: 17 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 13.
I20260709 13:02:02.388267  8140 raft_consensus.cc:697] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 20 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:02:02.395291  8140 consensus_queue.cc:237] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.13, Last appended by leader: 13, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:02.396636  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:02:02.392175  1645 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:02.400766  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:02.398084  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:02.403190  2336 raft_consensus.cc:2368] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for earlier term 16. Current term is 17.
I20260709 13:02:02.371570  8129 raft_consensus.cc:493] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:02.413471  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:02.417826  1600 raft_consensus.cc:2468] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 15.
I20260709 13:02:02.406486  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 19 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:02.404099  1749 raft_consensus.cc:2468] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 17.
I20260709 13:02:02.407461  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:02.413450  1643 leader_election.cc:304] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:02.414642  1569 leader_election.cc:304] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:02:02.415623  7307 leader_election.cc:290] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:02.380493  8145 raft_consensus.cc:493] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:02:02.417850  2328 raft_consensus.cc:2468] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 14.
I20260709 13:02:02.421011  2330 raft_consensus.cc:2468] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 19.
I20260709 13:02:02.423386  2334 raft_consensus.cc:2410] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for term 19 because replica has last-logged OpId of term: 17 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 13.
W20260709 13:02:02.423382  1717 leader_election.cc:336] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 14 election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 1.631s (SENT)
I20260709 13:02:02.430348  1751 raft_consensus.cc:2468] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 14.
I20260709 13:02:02.432209  1717 leader_election.cc:304] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 13:02:02.429782  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 74598.8us
I20260709 13:02:02.405962  1754 consensus_queue.cc:260] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 17.9, Last appended by leader: 9, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:02.440040  1597 raft_consensus.cc:2468] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 14.
I20260709 13:02:02.444686  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:02.445381  2330 raft_consensus.cc:2468] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 18.
I20260709 13:02:02.452279  7307 raft_consensus.cc:2749] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 13:02:02.456497  1754 raft_consensus.cc:3060] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:02:02.433769  8148 raft_consensus.cc:2749] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 13:02:02.443236  8123 raft_consensus.cc:2804] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader election won for term 15
I20260709 13:02:02.437151  8145 raft_consensus.cc:515] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:02.443296  7728 raft_consensus.cc:493] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:02.433466  8129 raft_consensus.cc:515] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:02.466843  7687 consensus_queue.cc:1048] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 13:02:02.472985  2298 leader_election.cc:304] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:02:02.442945  8259 consensus_queue.cc:1059] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 13.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.025s
W20260709 13:02:02.486935  1570 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:02.471171  8123 raft_consensus.cc:697] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
W20260709 13:02:02.514760  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 101464us
I20260709 13:02:02.475350  7728 raft_consensus.cc:515] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:02:02.556490  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 96836.2us
I20260709 13:02:02.537405  1502 catalog_manager.cc:5697] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 reported cstate change: term changed from 8 to 12. New cstate: current_term: 12 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 13:02:02.564849  1646 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:02.570948  2298 leader_election.cc:304] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
W20260709 13:02:02.583487  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 122770us
W20260709 13:02:02.573796  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 186221us
I20260709 13:02:02.584960  7307 raft_consensus.cc:493] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:02.604152  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:02.599025  1717 leader_election.cc:304] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 13:02:02.568701  8278 log.cc:927] Time spent T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.081s	user 0.001s	sys 0.000s
I20260709 13:02:02.608335  2328 raft_consensus.cc:2368] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 15. Current term is 16.
I20260709 13:02:02.599809  2299 leader_election.cc:304] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:02.617470  7307 raft_consensus.cc:515] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:02.623939  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:02.624744  1600 raft_consensus.cc:3060] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:02:02.604969  8068 raft_consensus.cc:2804] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 13:02:02.649367  8270 raft_consensus.cc:2749] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
W20260709 13:02:02.650707  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 93366.1us
W20260709 13:02:02.650887  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 52049.2us
W20260709 13:02:02.651535  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 44899.8us
I20260709 13:02:02.659304  8129 leader_election.cc:290] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:02.659107  1502 catalog_manager.cc:5697] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 reported cstate change: term changed from 10 to 14. New cstate: current_term: 14 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:02:02.662825  8145 leader_election.cc:290] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:02.587324  8123 consensus_queue.cc:237] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.11, Last appended by leader: 11, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:02.666181  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 16 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:02.617411  8163 raft_consensus.cc:2804] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Leader election won for term 15
W20260709 13:02:02.671527  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:02.668184  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:02:02.672402  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 226840us
I20260709 13:02:02.673427  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:02.680505  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:02.681918  1754 raft_consensus.cc:2468] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 15.
I20260709 13:02:02.680945  1600 raft_consensus.cc:2468] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 16.
I20260709 13:02:02.687259  1750 raft_consensus.cc:1240] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 13. Current term is 14. Ops: []
I20260709 13:02:02.637961  7981 raft_consensus.cc:3055] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 LEADER]: Stepping down as leader of term 13
I20260709 13:02:02.699466  2328 raft_consensus.cc:1275] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 18 index: 14. Preceding OpId from leader: term: 19 index: 15. (index mismatch)
I20260709 13:02:02.701215  7307 leader_election.cc:290] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:02.711069  2332 raft_consensus.cc:1275] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 15 index: 12. (term mismatch)
I20260709 13:02:02.712476  2332 pending_rounds.cc:85] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1: Aborting all ops after (but not including) 11
I20260709 13:02:02.712751  2332 pending_rounds.cc:107] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1: Aborting uncommitted NO_OP operation due to leader change: 14.12
I20260709 13:02:02.713021  2332 raft_consensus.cc:2889] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:02:02.713762  1571 leader_election.cc:400] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 17 pre-election: Vote denied by peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693) with higher term. Message: Invalid argument: T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 17. Current term is 18.
I20260709 13:02:02.714097  1571 leader_election.cc:403] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 17 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:02:02.714458  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:02.714939  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:02.715818  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 15 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:02.716498  2335 raft_consensus.cc:2468] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 14.
I20260709 13:02:02.717500  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:02.720048  8148 consensus_queue.cc:1048] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 13:02:02.685508  8237 raft_consensus.cc:493] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:02:02.721482  8237 raft_consensus.cc:515] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:02.697413  8163 raft_consensus.cc:697] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 15 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:02:02.711069  1502 catalog_manager.cc:5697] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 reported cstate change: term changed from 9 to 13. New cstate: current_term: 13 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:02:02.723433  8163 consensus_queue.cc:237] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 14.13, Last appended by leader: 13, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:02.724246  8237 leader_election.cc:290] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:02.679927  8068 raft_consensus.cc:697] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 18 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:02:02.725641  8068 consensus_queue.cc:237] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.12, Last appended by leader: 12, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:02.726907  8227 consensus_queue.cc:1059] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 13:02:02.679557  8280 raft_consensus.cc:2749] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20260709 13:02:02.729192  8260 consensus_meta.cc:287] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.511s	user 0.014s	sys 0.002s
I20260709 13:02:02.702138  7981 raft_consensus.cc:740] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:02:02.748853  8260 raft_consensus.cc:515] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:02.749770  7981 consensus_queue.cc:260] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.8, Last appended by leader: 8, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:02.751778  8260 leader_election.cc:290] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 14 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:02.737782  1748 raft_consensus.cc:1240] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 13. Current term is 14. Ops: []
I20260709 13:02:02.737847  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:02.752770  2332 raft_consensus.cc:3060] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:02:02.722311  8148 consensus_queue.cc:1059] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 15.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.031s
I20260709 13:02:02.739619  1745 raft_consensus.cc:1240] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 13. Current term is 14. Ops: []
I20260709 13:02:02.738162  1750 raft_consensus.cc:1217] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 19 LEADER]: Deduplicated request from leader. Original: 17.10->[18.11-18.11]   Dedup: 18.11->[]
I20260709 13:02:02.754535  1750 raft_consensus.cc:1240] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 19 LEADER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 18. Current term is 19. Ops: [18.11-18.11]
I20260709 13:02:02.754729  1747 raft_consensus.cc:2393] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 15: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
W20260709 13:02:02.747295  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 46888.2us
I20260709 13:02:02.751735  7981 raft_consensus.cc:3060] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Advancing to term 14
W20260709 13:02:02.735564  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 136153us
W20260709 13:02:02.756075  1718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:02.763550  8285 raft_consensus.cc:2749] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Leader pre-election lost for term 17. Reason: Vote denied by peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693) with higher term. Message: Invalid argument: T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 17. Current term is 18.
I20260709 13:02:02.766691  8284 raft_consensus.cc:3055] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 LEADER]: Stepping down as leader of term 13
I20260709 13:02:02.774279  1569 leader_election.cc:304] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:02:02.779398  1750 raft_consensus.cc:1240] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 18. Current term is 19. Ops: [18.10-18.10]
I20260709 13:02:02.781016  1748 raft_consensus.cc:1275] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 15 index: 12. (index mismatch)
I20260709 13:02:02.776288  8284 raft_consensus.cc:740] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:02:02.760136  8237 consensus_queue.cc:1059] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 16.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.022s
W20260709 13:02:02.785692  8190 consensus_meta.cc:287] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.532s	user 0.017s	sys 0.000s
I20260709 13:02:02.793838  2299 leader_election.cc:304] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:02:02.796060  8285 raft_consensus.cc:493] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:02.796913  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:02.805078  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 14 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:02.808132  1671 raft_consensus.cc:1275] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 15 index: 8. Preceding OpId from leader: term: 18 index: 10. (index mismatch)
W20260709 13:02:02.810832  1496 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35444 (request call id 262) took 40765 ms (client timeout 15000 ms). Trace:
I20260709 13:02:02.811579  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:02.812808  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:02.864409  1748 raft_consensus.cc:2468] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 16.
I20260709 13:02:02.757030  7812 consensus_queue.cc:1059] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.030s
I20260709 13:02:02.787029  8260 raft_consensus.cc:2804] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Leader pre-election won for term 18
W20260709 13:02:02.834476  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 134385us
W20260709 13:02:02.855044  1717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:02.866871  1717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:02.866384  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 145562us
I20260709 13:02:02.875849  1680 raft_consensus.cc:3060] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:02:02.875936  1745 raft_consensus.cc:1275] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 16 index: 11. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
W20260709 13:02:02.875079  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 145663us
I20260709 13:02:02.874118  2298 leader_election.cc:304] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:02:02.790285  8270 consensus_queue.cc:1059] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.069s
I20260709 13:02:02.880930  1753 raft_consensus.cc:2410] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for term 13 because replica has last-logged OpId of term: 11 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 9.
I20260709 13:02:02.879426  1646 leader_election.cc:304] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:02.888887  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 19 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:02.870246  8260 raft_consensus.cc:493] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:02.890189  1748 raft_consensus.cc:2410] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 for term 19 because replica has last-logged OpId of term: 17 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 11.
I20260709 13:02:02.897298  1672 raft_consensus.cc:1275] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 12 index: 12. Preceding OpId from leader: term: 14 index: 13. (index mismatch)
I20260709 13:02:02.893973  8190 raft_consensus.cc:2804] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Leader election won for term 18
W20260709 13:02:02.900600  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 118114us
W20260709 13:02:02.907860  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:02.917637  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:02.917303  8284 consensus_queue.cc:260] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.9, Last appended by leader: 9, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:02.922335  8284 raft_consensus.cc:3060] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Advancing to term 14
W20260709 13:02:02.922546  8291 log.cc:927] Time spent T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.054s	user 0.001s	sys 0.000s
I20260709 13:02:02.891307  7745 consensus_queue.cc:1048] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.035s
I20260709 13:02:02.928135  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:02.917752  8190 raft_consensus.cc:697] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 18 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:02:02.931656  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:02.932384  1748 raft_consensus.cc:3060] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:02:02.941370  1643 leader_election.cc:304] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:02:02.891685  8260 raft_consensus.cc:3060] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:02:02.942870  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:02.949999  1596 raft_consensus.cc:1275] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 18 index: 11. Preceding OpId from leader: term: 19 index: 12. (index mismatch)
I20260709 13:02:02.951952  2332 raft_consensus.cc:2468] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 15.
I20260709 13:02:02.873705  8285 raft_consensus.cc:515] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:02.951975  2299 leader_election.cc:304] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 13:02:02.940857  8292 log.cc:927] Time spent T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.143s	user 0.000s	sys 0.001s
I20260709 13:02:02.957587  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 17 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:02.957244  8190 consensus_queue.cc:237] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 17.14, Last appended by leader: 14, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:02.958854  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:02.959709  1751 raft_consensus.cc:2393] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 18: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:02:02.957037  7745 raft_consensus.cc:493] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:02.918627  8268 raft_consensus.cc:2804] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Leader election won for term 16
W20260709 13:02:02.963884  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 47801us
I20260709 13:02:02.964890  1603 raft_consensus.cc:2468] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 16.
I20260709 13:02:02.989197  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:02.918385  8148 consensus_queue.cc:1048] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.067s
I20260709 13:02:02.995370  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:02.993741  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 17 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:02:03.001338  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 67766us
I20260709 13:02:03.009892  1751 raft_consensus.cc:1275] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 12 index: 10. (term mismatch)
W20260709 13:02:03.010861  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 131747us
I20260709 13:02:02.992067  7585 raft_consensus.cc:2804] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:02:02.987898  8154 raft_consensus.cc:493] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:03.013945  1596 raft_consensus.cc:1240] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 14. Current term is 15. Ops: [14.13-14.13]
I20260709 13:02:02.989758  8268 raft_consensus.cc:697] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 16 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:02:03.010500  1751 pending_rounds.cc:85] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701: Aborting all ops after (but not including) 9
I20260709 13:02:03.027787  1751 pending_rounds.cc:107] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701: Aborting uncommitted NO_OP operation due to leader change: 11.10
I20260709 13:02:03.028060  1751 raft_consensus.cc:2889] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:02:03.032387  1680 raft_consensus.cc:2468] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 14.
W20260709 13:02:03.032523  1597 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40154 (request call id 2700) took 2979 ms (client timeout 1631 ms). Trace:
W20260709 13:02:03.035490  1645 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:03.033144  1597 rpcz_store.cc:269] 0709 13:02:00.052654 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:00.753333 (+700679us) service_pool.cc:224] Handling call
0709 13:02:03.011987 (+2258654us) spinlock_profiling.cc:243] Waited 1.48 s on lock 0x7b54002a1f54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:02:03.032441 (+ 20454us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3399572224}
I20260709 13:02:03.037873  1604 raft_consensus.cc:1240] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 14. Current term is 15. Ops: [14.13-14.13]
I20260709 13:02:03.039137  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:03.039790  1753 raft_consensus.cc:3060] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:02:03.043424  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:03.044543  1597 raft_consensus.cc:2393] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 18: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:02:03.032582  2301 leader_election.cc:304] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:02.991437  7745 raft_consensus.cc:515] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:03.015101  7585 raft_consensus.cc:493] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:03.037611  1601 raft_consensus.cc:1240] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 15. Current term is 16. Ops: [15.9-15.9]
I20260709 13:02:03.014400  8237 raft_consensus.cc:493] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
W20260709 13:02:03.049513  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 118049us
I20260709 13:02:03.052753  8311 raft_consensus.cc:3055] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:02:03.051748  8154 raft_consensus.cc:515] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:03.059919  8311 raft_consensus.cc:740] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:02:03.061088  8311 consensus_queue.cc:260] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.10, Last appended by leader: 10, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:03.061993  8154 leader_election.cc:290] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:02:03.063313  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 159110us
I20260709 13:02:03.053773  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:03.070279  1603 raft_consensus.cc:2393] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 19: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
W20260709 13:02:03.073896  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:03.074786  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:03.051852  7585 raft_consensus.cc:3060] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:02:03.081259  8140 raft_consensus.cc:2804] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:02:03.081697  8140 raft_consensus.cc:493] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:02:03.087285  8296 log.cc:927] Time spent T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.068s	user 0.000s	sys 0.001s
I20260709 13:02:03.088665  7745 leader_election.cc:290] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:03.089064  8309 raft_consensus.cc:3055] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 18 LEADER]: Stepping down as leader of term 18
I20260709 13:02:03.089609  8309 raft_consensus.cc:740] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:02:03.089782  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:03.090775  8309 consensus_queue.cc:260] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 18.9, Last appended by leader: 9, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:03.092666  8309 raft_consensus.cc:3060] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Advancing to term 19
W20260709 13:02:03.075992  8299 log.cc:927] Time spent T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.146s	user 0.000s	sys 0.001s
I20260709 13:02:03.082077  8140 raft_consensus.cc:3060] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Advancing to term 16
W20260709 13:02:03.109760  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 110915us
W20260709 13:02:03.112848  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 112044us
I20260709 13:02:03.077726  8317 raft_consensus.cc:2804] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Leader pre-election won for term 20
W20260709 13:02:02.863497  1496 rpcz_store.cc:269] 0709 13:01:22.078082 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:22.151913 (+ 73831us) service_pool.cc:224] Handling call
0709 13:02:02.805201 (+40653288us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 13:01:55.186278 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 13:01:55.278547 (+ 92269us) write_op.cc:432] Acquiring schema lock in shared mode
0709 13:01:55.278610 (+    63us) write_op.cc:435] Acquired schema lock
0709 13:01:55.278629 (+    19us) tablet.cc:662] Decoding operations
0709 13:01:55.367395 (+ 88766us) write_op.cc:620] Acquiring the partition lock for write op
0709 13:01:55.405722 (+ 38327us) write_op.cc:641] Partition lock acquired for write op
0709 13:01:55.414475 (+  8753us) tablet.cc:685] Acquiring locks for 19 operations
0709 13:01:55.643425 (+228950us) tablet.cc:701] Row locks acquired
0709 13:01:55.643488 (+    63us) write_op.cc:260] PREPARE: finished
0709 13:01:55.643702 (+   214us) write_op.cc:270] Start()
0709 13:01:55.643909 (+   207us) write_op.cc:276] Timestamp: P: 1783602115643629 usec, L: 0
0709 13:01:55.643942 (+    33us) op_driver.cc:348] REPLICATION: starting
0709 13:01:55.644730 (+   788us) log.cc:844] Serialized 7477 byte log entry
0709 13:01:56.413071 (+768341us) op_driver.cc:464] REPLICATION: finished
0709 13:01:56.909023 (+495952us) spinlock_profiling.cc:243] Waited 43.5 ms on lock 0x7b4400022b4c. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f4262f9bc85 00007f4262f9c6a2 00007f4265d0e1db 00007f4265d09aaf 00007f42630580ac 00007f426beb2ab9 00007f426307469b 00007f426307293f 00007f4263074c02 00007f4269b98ef0
0709 13:01:56.990916 (+ 81893us) write_op.cc:301] APPLY: starting
0709 13:01:56.999108 (+  8192us) tablet.cc:1367] starting BulkCheckPresence
0709 13:01:57.160055 (+160947us) tablet.cc:1370] finished BulkCheckPresence
0709 13:01:57.160112 (+    57us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 13:01:57.294746 (+134634us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 13:01:57.295338 (+   592us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=38,key_file_lookups=38,delta_file_lookups=38,mrs_lookups=0
0709 13:01:57.295399 (+    61us) write_op.cc:312] APPLY: finished
0709 13:01:57.296005 (+   606us) log.cc:844] Serialized 172 byte log entry
0709 13:01:57.297327 (+  1322us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:01:57.297568 (+   241us) write_op.cc:454] Released schema lock
0709 13:01:57.297674 (+   106us) write_op.cc:341] FINISH: Updating metrics
0709 13:01:57.302802 (+  5128us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:01:57.302872 (+    70us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":472149,"reported_tablets":48,"spinlock_wait_cycles":16384,"child_traces":[["op",{"apply.queue_time_us":575410,"apply.run_cpu_time_us":31883,"apply.run_wall_time_us":310981,"cfile_cache_hit":78,"cfile_cache_hit_bytes":83699,"num_ops":19,"prepare.queue_time_us":538538,"prepare.run_cpu_time_us":36494,"prepare.run_wall_time_us":764344,"raft.queue_time_us":351583,"raft.run_cpu_time_us":21157,"raft.run_wall_time_us":636374,"replication_time_us":744126,"spinlock_wait_cycles":100108032,"thread_start_us":1472515,"threads_started":4,"wal-append.queue_time_us":278188,"wal-append.run_cpu_time_us":27190,"wal-append.run_wall_time_us":2390873}]]}
I20260709 13:02:03.130048  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:03.131078  1597 raft_consensus.cc:3060] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:02:03.130327  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:03.131493  1748 raft_consensus.cc:2468] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 17.
I20260709 13:02:03.119514  8311 raft_consensus.cc:3060] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:02:03.114714  8270 raft_consensus.cc:493] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:03.136749  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:03.096019  8237 raft_consensus.cc:515] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:03.138835  7745 consensus_queue.cc:1048] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.006s
I20260709 13:02:03.146731  7728 leader_election.cc:290] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:02:03.147166  8304 log.cc:927] Time spent T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.097s	user 0.001s	sys 0.000s
I20260709 13:02:03.140534  8154 consensus_queue.cc:1048] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.008s
I20260709 13:02:03.132759  8310 raft_consensus.cc:2804] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Leader election won for term 13
I20260709 13:02:03.184378  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:03.185206  1600 raft_consensus.cc:2368] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 19. Current term is 20.
I20260709 13:02:03.170957  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:03.187564  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:03.189103  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:02:03.190789  1718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:03.191927  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:03.193151  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 17 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:03.186163  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 17 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:03.199079  1600 raft_consensus.cc:2468] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 16.
I20260709 13:02:03.201856  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:02:03.143786  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 79037.3us
I20260709 13:02:03.127951  8317 raft_consensus.cc:493] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:03.142998  8270 raft_consensus.cc:515] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:03.153622  2301 leader_election.cc:304] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:02:03.154331  8285 leader_election.cc:290] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:02:03.158949  8302 log.cc:927] Time spent T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.090s	user 0.000s	sys 0.001s
I20260709 13:02:03.137447  8268 consensus_queue.cc:237] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:03.163110  8237 leader_election.cc:290] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:02:03.170881  1488 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet 983621a6fc5b4c9ea0159a648dfe7a36 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER failed with leader 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 3b879799899a499c8d409ee9f1277cd1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 18 committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 13:02:03.171389  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:03.172329  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:03.225060  1605 raft_consensus.cc:2468] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 13.
I20260709 13:02:03.173148  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:02:03.173508  1570 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:03.226400  1570 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:03.173838  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 15 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:03.173743  8316 consensus_queue.cc:1048] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.033s
I20260709 13:02:03.174185  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:03.175740  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:03.176582  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:03.177453  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:03.178273  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:03.231513  1674 raft_consensus.cc:2410] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 for term 15 because replica has last-logged OpId of term: 12 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 11.
I20260709 13:02:03.229959  1598 raft_consensus.cc:2468] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 15.
I20260709 13:02:03.178206  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:03.232577  7307 raft_consensus.cc:493] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:03.176589  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 20 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:03.211476  1753 raft_consensus.cc:2468] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 20.
I20260709 13:02:03.232573  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 19 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:03.200811  1569 leader_election.cc:304] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:02:03.233229  7307 raft_consensus.cc:515] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:03.216446  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:03.222669  8260 raft_consensus.cc:515] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:03.182797  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:03.233927  1718 leader_election.cc:304] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
W20260709 13:02:03.248121  1571 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:03.234457  2332 raft_consensus.cc:2393] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 16: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
W20260709 13:02:03.234491  1719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:03.214586  8317 raft_consensus.cc:3060] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Advancing to term 20
W20260709 13:02:03.236181  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 84301.2us
I20260709 13:02:03.236564  1596 raft_consensus.cc:1275] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 20 index: 14. (index mismatch)
I20260709 13:02:03.230827  1679 raft_consensus.cc:2468] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 16.
I20260709 13:02:03.243523  7687 raft_consensus.cc:2804] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:02:03.243626  1672 raft_consensus.cc:2468] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 18.
I20260709 13:02:03.245754  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 20 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:02:03.246985  1488 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 76410.3us
W20260709 13:02:03.256142  1569 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:03.263976  1674 raft_consensus.cc:1240] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 13. Current term is 14. Ops: []
W20260709 13:02:03.264637  1571 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:03.232841  8262 raft_consensus.cc:2749] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:02:03.268694  1597 raft_consensus.cc:2468] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 16.
I20260709 13:02:03.263180  1677 raft_consensus.cc:2468] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 13.
I20260709 13:02:03.271016  7728 raft_consensus.cc:493] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:03.271222  7687 raft_consensus.cc:697] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:02:03.277899  1601 raft_consensus.cc:2468] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 15.
W20260709 13:02:03.279294  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 125904us
I20260709 13:02:03.286767  1679 raft_consensus.cc:1240] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 18. Current term is 19. Ops: []
I20260709 13:02:03.283416  1571 leader_election.cc:304] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:02:03.272348  2299 leader_election.cc:304] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:03.312446  2299 leader_election.cc:304] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:02:03.238155  7585 raft_consensus.cc:515] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:03.318296  2299 leader_election.cc:304] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:02:03.320075  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:03.324258  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 20 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:02:03.315496  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 86990.5us
I20260709 13:02:03.334903  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:03.328377  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:03.288661  8145 raft_consensus.cc:3055] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 13:02:03.342224  8145 raft_consensus.cc:740] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
W20260709 13:02:03.342274  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 277433us
I20260709 13:02:03.343551  1646 leader_election.cc:304] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:03.343346  8145 consensus_queue.cc:260] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.12, Last appended by leader: 12, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:03.344635  1646 leader_election.cc:304] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:03.346540  8262 consensus_queue.cc:1059] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 13.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 13:02:03.346751  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 16 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:03.348268  8325 raft_consensus.cc:2804] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:02:03.348595  1749 raft_consensus.cc:1275] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 16 index: 9. Preceding OpId from leader: term: 18 index: 10. (index mismatch)
I20260709 13:02:03.348831  8325 raft_consensus.cc:493] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:03.349301  8325 raft_consensus.cc:3060] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:02:03.350854  8319 raft_consensus.cc:2804] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:02:03.351430  8319 raft_consensus.cc:697] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:02:03.352329  8319 consensus_queue.cc:237] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 11.8, Last appended by leader: 8, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:03.354261  1675 raft_consensus.cc:2410] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for term 20 because replica has last-logged OpId of term: 18 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 8.
I20260709 13:02:03.275835  7728 raft_consensus.cc:515] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:02:03.355216  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 71922.1us
I20260709 13:02:03.356128  1571 leader_election.cc:304] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:02:03.261058  8190 consensus_queue.cc:1048] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.061s
I20260709 13:02:03.357143  1604 raft_consensus.cc:2368] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 17. Current term is 18.
I20260709 13:02:03.357820  1750 raft_consensus.cc:2393] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 16: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:02:03.344755  8145 raft_consensus.cc:3060] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Advancing to term 20
W20260709 13:02:03.362142  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 134474us
I20260709 13:02:03.364249  7307 leader_election.cc:290] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:03.272713  8300 raft_consensus.cc:3055] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 LEADER]: Stepping down as leader of term 15
I20260709 13:02:03.313987  8280 raft_consensus.cc:2749] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:02:03.367305  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:03.301908  8310 raft_consensus.cc:697] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 13 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:02:03.335340  8307 raft_consensus.cc:2804] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:02:03.335872  8068 raft_consensus.cc:2804] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:02:03.381683  1745 raft_consensus.cc:2468] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 14.
I20260709 13:02:03.288631  8317 raft_consensus.cc:515] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:03.363607  8163 raft_consensus.cc:3055] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 13 LEADER]: Stepping down as leader of term 13
W20260709 13:02:03.399471  1718 leader_election.cc:336] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Timed out: RequestConsensusVote RPC to 127.0.107.194:46261 timed out after 2.881s (SENT)
I20260709 13:02:03.364279  7745 raft_consensus.cc:2749] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 13:02:03.396839  7307 consensus_queue.cc:1059] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 13.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 13:02:03.307497  8281 consensus_queue.cc:1048] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.076s
I20260709 13:02:03.408363  2301 leader_election.cc:304] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:02:03.373935  8300 raft_consensus.cc:740] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
W20260709 13:02:03.398273  1643 connection.cc:256] RPC call timeout handler was delayed by -2.8515018030000192s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3145s for a response
W20260709 13:02:03.417343  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 61376.1us
I20260709 13:02:03.410741  1718 leader_election.cc:304] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
W20260709 13:02:03.420588  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 100748us
I20260709 13:02:03.421273  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:03.422181  1751 raft_consensus.cc:2468] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 18.
I20260709 13:02:03.393671  8307 raft_consensus.cc:493] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:03.391475  8068 raft_consensus.cc:493] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:03.426288  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:03.427734  1671 raft_consensus.cc:1240] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 19. Current term is 20. Ops: []
I20260709 13:02:03.429513  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:03.430013  2299 leader_election.cc:304] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:03.426553  1571 leader_election.cc:304] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:03.277026  7687 consensus_queue.cc:237] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.10, Last appended by leader: 10, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:03.405180  8270 leader_election.cc:290] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:03.405110  8163 raft_consensus.cc:740] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:02:03.404512  7728 leader_election.cc:290] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:03.439064  1746 raft_consensus.cc:1240] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 15. Current term is 16. Ops: []
I20260709 13:02:03.404177  8298 raft_consensus.cc:493] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:02:03.436092  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:03.436745  7307 consensus_queue.cc:1059] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 13.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.013s
I20260709 13:02:03.452426  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:03.450598  1672 raft_consensus.cc:2468] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 19.
W20260709 13:02:03.457098  1645 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:03.455123  1749 raft_consensus.cc:2376] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 14. Re-sending same reply.
W20260709 13:02:03.463532  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:03.427304  8068 raft_consensus.cc:3060] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:02:03.466424  1747 raft_consensus.cc:1275] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 14 index: 7. Preceding OpId from leader: term: 15 index: 7. (term mismatch)
W20260709 13:02:03.467926  1643 connection.cc:256] RPC call timeout handler was delayed by -2.3055167399999732s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1884s for a response
I20260709 13:02:03.428066  8307 raft_consensus.cc:3060] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Advancing to term 17
W20260709 13:02:03.453374  7981 consensus_meta.cc:287] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.689s	user 0.012s	sys 0.003s
W20260709 13:02:03.468326  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 67445.9us
I20260709 13:02:03.473886  1673 raft_consensus.cc:2468] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 14.
I20260709 13:02:03.467001  1747 pending_rounds.cc:85] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701: Aborting all ops after (but not including) 6
I20260709 13:02:03.476382  1747 pending_rounds.cc:107] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701: Aborting uncommitted NO_OP operation due to leader change: 14.7
I20260709 13:02:03.476660  1747 raft_consensus.cc:2889] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:02:03.477876  7307 raft_consensus.cc:2749] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:02:03.483383  8330 consensus_queue.cc:1048] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.112s
I20260709 13:02:03.483809  8328 raft_consensus.cc:2804] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:02:03.484338  8328 raft_consensus.cc:493] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:03.484688  8328 raft_consensus.cc:3060] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:02:03.485427  8323 consensus_queue.cc:1048] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.094s
I20260709 13:02:03.485548  8298 raft_consensus.cc:515] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:03.487413  8294 raft_consensus.cc:2749] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20260709 13:02:03.488169  1644 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.107.196:46625 timed out after 1.772s (SENT) [suppressed 55 similar messages]
W20260709 13:02:03.488854  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 104917us
I20260709 13:02:03.467829  7745 raft_consensus.cc:2804] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:02:03.496807  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:03.498644  2301 leader_election.cc:304] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:02:03.508939  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 17 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:02:03.519400  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 46996.4us
I20260709 13:02:03.519469  8327 raft_consensus.cc:493] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:03.523140  8317 leader_election.cc:290] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 20 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:03.496340  8270 raft_consensus.cc:3055] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:02:03.522930  1717 leader_election.cc:304] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:03.522377   431 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:02:03.506052  7745 raft_consensus.cc:493] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:03.535902  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:02:03.534663  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 199076us
I20260709 13:02:03.517733  8280 raft_consensus.cc:2804] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:02:03.538110  1673 raft_consensus.cc:2468] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 16.
I20260709 13:02:03.541989  7307 raft_consensus.cc:2749] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20260709 13:02:03.542043  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 50548.8us
I20260709 13:02:03.551883  8298 leader_election.cc:290] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:03.497162  8300 consensus_queue.cc:260] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.10, Last appended by leader: 10, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:03.552706  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:03.553468  1674 raft_consensus.cc:2468] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 19.
I20260709 13:02:03.553802  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:03.527478  8270 raft_consensus.cc:740] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:02:03.540896  7745 raft_consensus.cc:3060] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:02:03.556420  8334 raft_consensus.cc:2749] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:02:03.557377  8334 raft_consensus.cc:2804] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 13:02:03.557925  8334 raft_consensus.cc:697] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 17 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:02:03.558480  8280 raft_consensus.cc:493] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:03.559032  8280 raft_consensus.cc:3060] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:02:03.520012  8163 consensus_queue.cc:260] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.10, Last appended by leader: 10, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:03.564718  7585 leader_election.cc:290] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 20 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:03.565917  8163 raft_consensus.cc:3060] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:02:03.523164  8327 raft_consensus.cc:515] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:03.578303  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 19 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:03.524559  8260 leader_election.cc:290] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 18 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:03.591696  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:03.547451  8323 raft_consensus.cc:493] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:03.543813  8336 raft_consensus.cc:493] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:03.533438  8330 consensus_queue.cc:1059] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 13.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.058s
I20260709 13:02:03.610301  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 17 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:02:03.610953  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 181125us
I20260709 13:02:03.616123  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:02:03.641695  2299 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:03.586246  8154 consensus_queue.cc:1048] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.027s
W20260709 13:02:03.647359  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 124632us
W20260709 13:02:03.653129  1644 leader_election.cc:336] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625): Timed out: RequestConsensusVote RPC to 127.0.107.196:46625 timed out after 1.592s (SENT)
W20260709 13:02:03.650329  1717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:03.657368  1717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:03.655457  1644 leader_election.cc:304] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:02:03.668711  1571 leader_election.cc:304] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:02:03.527127  8310 consensus_queue.cc:237] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.8, Last appended by leader: 8, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:03.556206  8270 consensus_queue.cc:260] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 14.13, Last appended by leader: 13, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:03.681406  1571 leader_election.cc:304] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:02:03.608592  8323 raft_consensus.cc:515] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:03.685415  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
W20260709 13:02:03.689563  1644 leader_election.cc:336] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625): Timed out: RequestConsensusVote RPC to 127.0.107.196:46625 timed out after 1.618s (SENT)
W20260709 13:02:03.709789  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 327603us
I20260709 13:02:03.683588  8311 raft_consensus.cc:2749] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:02:03.716773  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:03.608556  8336 raft_consensus.cc:515] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:03.687172  8259 raft_consensus.cc:2749] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:02:03.727334  1678 raft_consensus.cc:3060] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:02:03.707783  8260 raft_consensus.cc:2749] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20260709 13:02:03.724589  8331 log.cc:927] Time spent T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.256s	user 0.002s	sys 0.003s
W20260709 13:02:03.740895  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 59635us
I20260709 13:02:03.745884  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:03.733608  8270 raft_consensus.cc:3060] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:02:03.715525  8237 consensus_queue.cc:1048] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.026s
I20260709 13:02:03.750859  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 14 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:03.745137  1644 leader_election.cc:304] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:02:03.754112  1569 leader_election.cc:304] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:02:03.757552  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:03.760169  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:03.761096  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:03.762163  2329 raft_consensus.cc:1275] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 15 index: 12. (index mismatch)
I20260709 13:02:03.767704  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:03.768646  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:03.769774  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:03.769774  1571 leader_election.cc:304] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:02:03.770278  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:03.770975  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:03.771605  2332 raft_consensus.cc:2393] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 18: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:02:03.772486  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 16 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:03.779054  2336 raft_consensus.cc:1275] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 11 index: 11. Preceding OpId from leader: term: 14 index: 13. (index mismatch)
I20260709 13:02:03.783586  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:03.788872  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:03.738803  8341 consensus_queue.cc:1059] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 18.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.044s
I20260709 13:02:03.789758  2329 raft_consensus.cc:2376] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 15. Re-sending same reply.
I20260709 13:02:03.683420  8334 consensus_queue.cc:237] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.11, Last appended by leader: 11, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:03.795316  1571 leader_election.cc:304] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:03.777964  7728 raft_consensus.cc:493] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:03.806286  8340 log.cc:927] Time spent T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.091s	user 0.001s	sys 0.000s
I20260709 13:02:03.807575  8350 raft_consensus.cc:493] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:03.808193  8350 raft_consensus.cc:515] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:03.813535  1644 leader_election.cc:400] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 16 election: Vote denied by peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625) with higher term. Message: Invalid argument: T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for earlier term 16. Current term is 17.
I20260709 13:02:03.813867  1644 leader_election.cc:403] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 16 election: Cancelling election due to peer responding with higher term
I20260709 13:02:03.756757  7981 consensus_queue.cc:1048] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.047s
I20260709 13:02:03.815548  8311 consensus_queue.cc:1048] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.013s
I20260709 13:02:03.815678  7728 raft_consensus.cc:515] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:03.800254  8259 raft_consensus.cc:2804] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:02:03.818976  8259 raft_consensus.cc:493] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:03.819339  8259 raft_consensus.cc:3060] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:02:03.820976  8320 raft_consensus.cc:3060] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:02:03.818172  7728 leader_election.cc:290] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:02:03.833391  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 90507.2us
I20260709 13:02:03.783015  8319 raft_consensus.cc:2804] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:02:03.766646  8260 consensus_queue.cc:1048] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.078s
I20260709 13:02:03.843266  8336 leader_election.cc:290] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:03.845589  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:03.847046  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:02:03.856106  1645 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:03.856912  8327 leader_election.cc:290] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:03.859133  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:03.860157  8323 leader_election.cc:290] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:03.860565  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:03.861318  1671 raft_consensus.cc:2468] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 15.
I20260709 13:02:03.865603  8329 raft_consensus.cc:3055] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:02:03.866132  8329 raft_consensus.cc:740] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:02:03.867172  8329 consensus_queue.cc:260] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 14.13, Last appended by leader: 13, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:03.868353  8329 raft_consensus.cc:3060] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:02:03.776487  8294 consensus_queue.cc:1048] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.048s
I20260709 13:02:03.874629  7307 raft_consensus.cc:3055] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 15 LEADER]: Stepping down as leader of term 15
W20260709 13:02:03.875571  8345 log.cc:927] Time spent T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.074s	user 0.000s	sys 0.000s
I20260709 13:02:03.876852  8350 leader_election.cc:290] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:03.848717  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:03.799520  8154 raft_consensus.cc:2749] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20260709 13:02:03.880079  1717 leader_election.cc:336] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 1.588s (SENT)
I20260709 13:02:03.875111  7307 raft_consensus.cc:740] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:02:03.898782  1646 leader_election.cc:304] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:03.891459  1644 leader_election.cc:304] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 13:02:03.901381  1719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:03.901875  1719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:03.903690  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:03.907472  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:03.908166  1746 raft_consensus.cc:2468] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 18.
W20260709 13:02:03.908823  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 46153.8us
I20260709 13:02:03.910976  2299 leader_election.cc:304] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 13:02:03.912158  1717 leader_election.cc:336] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 18 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 1.592s (SENT)
I20260709 13:02:03.916188  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:02:03.898563  8140 consensus_meta.cc:287] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.778s	user 0.015s	sys 0.001s
W20260709 13:02:03.920288  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 125114us
I20260709 13:02:03.920363  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:03.927977  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:03.935734  1752 raft_consensus.cc:3060] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:02:03.928272  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 20 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:03.838604  8319 raft_consensus.cc:493] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:02:03.929514  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:03.945591  1748 raft_consensus.cc:2468] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 15.
I20260709 13:02:03.930332  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:03.932327  1646 leader_election.cc:304] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:03.928789  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:03.873925  8307 raft_consensus.cc:515] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:03.944679  1605 raft_consensus.cc:2468] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 22.
I20260709 13:02:03.931136  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:03.940275  1745 raft_consensus.cc:2410] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 for term 20 because replica has last-logged OpId of term: 19 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 11.
I20260709 13:02:03.940495  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:03.947121  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:03.949743  1675 raft_consensus.cc:2468] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 15.
I20260709 13:02:03.957612  8327 raft_consensus.cc:2804] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:02:03.958181  8327 raft_consensus.cc:493] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:03.958626  8327 raft_consensus.cc:3060] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:02:03.960355  8294 raft_consensus.cc:493] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:03.960892  8294 raft_consensus.cc:515] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:03.947257  8350 raft_consensus.cc:2749] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 13:02:03.962651  1604 raft_consensus.cc:2468] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 15.
I20260709 13:02:03.936563  7728 raft_consensus.cc:2804] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:02:03.963588  7728 raft_consensus.cc:493] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:03.963943  7728 raft_consensus.cc:3060] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:02:03.917101  8298 raft_consensus.cc:2804] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:02:03.971349  8298 raft_consensus.cc:493] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:03.971722  8298 raft_consensus.cc:3060] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:02:03.974620  8269 consensus_queue.cc:1059] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 19.15, Next index: 16, Last known committed idx: 14, Time since last communication: 0.034s
I20260709 13:02:03.948310  8319 raft_consensus.cc:3060] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:02:03.978821  1671 raft_consensus.cc:2393] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 17: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:02:03.913277  8145 raft_consensus.cc:493] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:03.984450  8145 raft_consensus.cc:515] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:03.955184  7307 consensus_queue.cc:260] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.9, Last appended by leader: 9, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:02:03.984475  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 101541us
I20260709 13:02:04.003703  1752 raft_consensus.cc:2468] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 16.
I20260709 13:02:04.005059  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:04.004927  1678 raft_consensus.cc:2468] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 18.
I20260709 13:02:04.006031  1752 raft_consensus.cc:2468] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 22.
I20260709 13:02:03.923167  8323 raft_consensus.cc:3055] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 18 LEADER]: Stepping down as leader of term 18
W20260709 13:02:04.011940  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 176874us
W20260709 13:02:04.015599  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 297011us
I20260709 13:02:04.019245  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:04.017673  1717 leader_election.cc:304] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
W20260709 13:02:04.023161  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 112359us
I20260709 13:02:04.024534  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:02:04.008018  8325 consensus_meta.cc:287] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.658s	user 0.009s	sys 0.010s
W20260709 13:02:04.027937  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 111351us
W20260709 13:02:04.040978  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 104942us
I20260709 13:02:04.041851  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:04.038882  1717 leader_election.cc:304] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:04.044902  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:04.042730  1675 raft_consensus.cc:2468] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 19.
I20260709 13:02:04.042200  1569 leader_election.cc:304] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:02:04.011500  8323 raft_consensus.cc:740] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:02:04.050989  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:03.943082  8140 raft_consensus.cc:515] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:04.056887  1749 raft_consensus.cc:2468] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 15.
I20260709 13:02:04.059589  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:03.974239  8328 raft_consensus.cc:515] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:04.070838  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:04.071811  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:04.061730  1717 leader_election.cc:304] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:04.073627  1717 leader_election.cc:304] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:04.067107  1752 raft_consensus.cc:1275] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 16 index: 12. Preceding OpId from leader: term: 18 index: 13. (index mismatch)
I20260709 13:02:04.058590  8154 raft_consensus.cc:493] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:04.063987  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:04.068217  7307 raft_consensus.cc:3060] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:02:04.040642  8336 raft_consensus.cc:2804] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:02:04.069255  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:04.108601  1717 leader_election.cc:304] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:04.122273  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:04.123216  1750 raft_consensus.cc:3060] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Advancing to term 18
W20260709 13:02:04.123471  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 47908.6us
I20260709 13:02:04.127652  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:04.128386  1605 raft_consensus.cc:2468] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 18.
I20260709 13:02:04.130985  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:04.131667  1600 raft_consensus.cc:2468] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 22.
I20260709 13:02:04.128083  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 17 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:04.124773  1569 leader_election.cc:304] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:02:04.137512  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:04.138146  1678 raft_consensus.cc:2468] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 18.
I20260709 13:02:04.139508  1754 raft_consensus.cc:1240] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 18. Current term is 19. Ops: []
I20260709 13:02:04.069602  8270 raft_consensus.cc:2804] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:02:04.108042  8330 raft_consensus.cc:2804] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:02:04.144793  1599 raft_consensus.cc:1240] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 14. Current term is 15. Ops: [14.13-14.13]
I20260709 13:02:04.109393  8336 raft_consensus.cc:493] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:04.159032  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:04.161373  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:04.161381  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:04.043401  8325 raft_consensus.cc:515] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:04.163654  1752 raft_consensus.cc:2468] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 19.
I20260709 13:02:04.165094  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:04.163657  1672 raft_consensus.cc:2468] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 16.
I20260709 13:02:04.167292  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:02:04.140829  8280 consensus_meta.cc:287] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.581s	user 0.011s	sys 0.009s
I20260709 13:02:04.173231  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:04.176347  8303 consensus_queue.cc:1048] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.129s
I20260709 13:02:04.179037  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:04.182529  8326 raft_consensus.cc:2804] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:02:04.183207  8326 raft_consensus.cc:697] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:02:04.184130  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:04.184487  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 17 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:04.184288  8326 consensus_queue.cc:237] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.8, Last appended by leader: 8, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:04.186715  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:04.187179  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 19 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:04.187731  1672 raft_consensus.cc:2376] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 18. Re-sending same reply.
I20260709 13:02:04.186367  1643 leader_election.cc:304] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
W20260709 13:02:04.186576  8359 log.cc:927] Time spent T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.060s	user 0.001s	sys 0.000s
I20260709 13:02:04.144937  8145 leader_election.cc:290] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:04.162214  8330 raft_consensus.cc:493] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:04.167954  8336 raft_consensus.cc:3060] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:02:04.132267  8154 raft_consensus.cc:515] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:04.217207  1674 raft_consensus.cc:2368] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 18. Current term is 19.
I20260709 13:02:04.221616  8350 raft_consensus.cc:493] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:02:04.146130  8270 raft_consensus.cc:493] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:04.234426  8360 raft_consensus.cc:2749] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 13:02:04.238754  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:04.234483  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:04.222553  8330 raft_consensus.cc:3060] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:02:04.225886  1748 raft_consensus.cc:2468] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 16.
I20260709 13:02:04.200817  2301 leader_election.cc:304] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:02:04.201980  8280 raft_consensus.cc:515] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:04.228102  8154 leader_election.cc:290] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:04.166828  8357 raft_consensus.cc:2749] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:02:04.228860  7307 raft_consensus.cc:2804] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:02:04.229528  1596 raft_consensus.cc:2468] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 13.
I20260709 13:02:04.273331  7307 raft_consensus.cc:493] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:04.273797  7307 raft_consensus.cc:3060] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Advancing to term 16
W20260709 13:02:04.232483  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 124061us
I20260709 13:02:04.261885  1599 raft_consensus.cc:2393] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 15: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:02:04.226747  8344 raft_consensus.cc:2804] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 13:02:04.280475  2328 raft_consensus.cc:2410] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for term 14 because replica has last-logged OpId of term: 12 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 9.
I20260709 13:02:04.225327  8320 raft_consensus.cc:2749] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Leader election lost for term 16. Reason: Vote denied by peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625) with higher term. Message: Invalid argument: T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for earlier term 16. Current term is 17.
I20260709 13:02:04.276513  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:02:04.277673  1717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:04.282881  1752 raft_consensus.cc:1275] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 18 index: 13. Preceding OpId from leader: term: 20 index: 14. (index mismatch)
W20260709 13:02:04.283290  1717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:04.278810  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 92844.9us
I20260709 13:02:04.260254  8270 raft_consensus.cc:3060] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:02:04.266127  8303 raft_consensus.cc:2749] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:02:04.173126  8323 consensus_queue.cc:260] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:04.283963  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 83309.2us
I20260709 13:02:04.285347  1718 leader_election.cc:304] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:04.286763  1676 raft_consensus.cc:2468] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 17.
I20260709 13:02:04.286854  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:04.288251  2301 leader_election.cc:304] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:02:04.260780  8145 consensus_queue.cc:1048] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.022s
I20260709 13:02:04.293145  1753 raft_consensus.cc:3060] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:02:04.293344  1746 pending_rounds.cc:85] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701: Aborting all ops after (but not including) 8
I20260709 13:02:04.290743  1602 raft_consensus.cc:2393] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 20: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
W20260709 13:02:04.291460  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 166864us
I20260709 13:02:04.237232  8350 raft_consensus.cc:515] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:04.295648  2301 leader_election.cc:304] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:04.293660  1746 pending_rounds.cc:107] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701: Aborting uncommitted NO_OP operation due to leader change: 17.9
I20260709 13:02:04.295503  7307 raft_consensus.cc:515] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:04.296754  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:04.298945  1569 leader_election.cc:304] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:02:04.299443  1599 raft_consensus.cc:2410] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for term 19 because replica has last-logged OpId of term: 18 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 10.
I20260709 13:02:04.297567  2301 leader_election.cc:304] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:02:04.287343  8354 raft_consensus.cc:2749] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:02:04.302213  1643 leader_election.cc:400] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 14 pre-election: Vote denied by peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445) with higher term. Message: Invalid argument: T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for earlier term 14. Current term is 15.
I20260709 13:02:04.306587  1643 leader_election.cc:403] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 14 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:02:04.297780  1746 raft_consensus.cc:2889] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:02:04.309966  8371 raft_consensus.cc:2749] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:02:04.311746  1718 leader_election.cc:304] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:04.313627  1718 leader_election.cc:304] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:04.314910  8328 leader_election.cc:290] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 19 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:04.315683  8328 raft_consensus.cc:2749] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 13:02:04.316561  8328 raft_consensus.cc:2804] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:02:04.316998  8328 raft_consensus.cc:493] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:04.317402  8328 raft_consensus.cc:3060] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:02:04.318272  8325 leader_election.cc:290] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:04.319418  1718 leader_election.cc:304] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:04.319815  8280 leader_election.cc:290] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 13 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:04.320199  8371 raft_consensus.cc:2804] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 13:02:04.320763  8371 raft_consensus.cc:697] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 18 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
W20260709 13:02:04.321961  8259 consensus_meta.cc:287] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.502s	user 0.007s	sys 0.011s
I20260709 13:02:04.321779  8371 consensus_queue.cc:237] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.13, Last appended by leader: 13, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:04.322851  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:04.276010  8319 raft_consensus.cc:515] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:04.324172  8350 leader_election.cc:290] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:04.324643  1718 leader_election.cc:304] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:04.325799  8319 leader_election.cc:290] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 20 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:04.326248  1718 leader_election.cc:304] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 13:02:04.327163  8363 log.cc:927] Time spent T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.061s	user 0.000s	sys 0.001s
W20260709 13:02:04.329262  8335 log.cc:927] Time spent T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.809s	user 0.003s	sys 0.004s
I20260709 13:02:04.330250  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:04.330792  8145 raft_consensus.cc:2804] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:02:04.330930  2337 raft_consensus.cc:2468] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 18.
I20260709 13:02:04.331274  8145 raft_consensus.cc:493] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:04.331732  8145 raft_consensus.cc:3060] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:02:04.334781  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:04.339314  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:04.341703  1750 raft_consensus.cc:2468] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 18.
W20260709 13:02:04.344576  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 45731us
I20260709 13:02:04.346304  7307 leader_election.cc:290] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:04.346357  1569 leader_election.cc:304] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:02:04.344954  2298 leader_election.cc:304] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:02:04.351136  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:04.351655  8154 raft_consensus.cc:2749] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:02:04.352097  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:04.352573  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:04.353441  8284 consensus_queue.cc:1048] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 13:02:04.354065  8280 raft_consensus.cc:2749] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:02:04.355469  8325 raft_consensus.cc:2749] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Leader pre-election lost for term 14. Reason: Vote denied by peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445) with higher term. Message: Invalid argument: T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for earlier term 14. Current term is 15.
I20260709 13:02:04.356550  2336 raft_consensus.cc:3060] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:02:04.360219  8307 leader_election.cc:290] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 17 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:04.360966  8307 raft_consensus.cc:2804] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:02:04.361344  8307 raft_consensus.cc:493] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:04.361680  8307 raft_consensus.cc:3060] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:02:04.301837  8334 raft_consensus.cc:2804] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Leader pre-election won for term 23
W20260709 13:02:04.367483  2300 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:04.296331  8320 raft_consensus.cc:493] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:02:04.316258  8140 leader_election.cc:290] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 16 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:04.369220  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
W20260709 13:02:04.370841  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 72918.1us
I20260709 13:02:04.296463  8368 raft_consensus.cc:2749] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:02:04.307816  8323 raft_consensus.cc:3060] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:02:04.384963  1753 raft_consensus.cc:1275] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 17 index: 14. Preceding OpId from leader: term: 18 index: 15. (index mismatch)
I20260709 13:02:04.388120  1599 raft_consensus.cc:1275] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 16 index: 12. (index mismatch)
I20260709 13:02:04.341759  7728 raft_consensus.cc:515] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:04.407573  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:04.408488  1602 raft_consensus.cc:2393] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 13: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:02:04.409246  7728 leader_election.cc:290] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 15 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:04.342099  8259 raft_consensus.cc:515] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:04.411664  8259 leader_election.cc:290] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 15 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:02:04.411989  8068 consensus_meta.cc:287] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.932s	user 0.009s	sys 0.009s
I20260709 13:02:04.412544  8068 raft_consensus.cc:515] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:04.418756  8350 consensus_queue.cc:1048] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 13:02:04.420683  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:04.421166  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:04.422462  2332 raft_consensus.cc:1240] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 19. Current term is 20. Ops: [19.15-19.15]
W20260709 13:02:04.422614  8163 consensus_meta.cc:287] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.840s	user 0.016s	sys 0.006s
I20260709 13:02:04.391132  8327 raft_consensus.cc:515] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:04.424021  2333 raft_consensus.cc:1240] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 15. Current term is 16. Ops: [15.12-15.12]
I20260709 13:02:04.425424  8327 leader_election.cc:290] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 14 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:04.425684  1644 leader_election.cc:304] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:02:04.367190  8154 raft_consensus.cc:2749] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:02:04.372812  8334 raft_consensus.cc:493] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:04.426669  2331 raft_consensus.cc:2468] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 14.
I20260709 13:02:04.427016  8334 raft_consensus.cc:3060] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:02:04.428247  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:02:04.430050  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 373742us
I20260709 13:02:04.430673  8372 raft_consensus.cc:2749] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:02:04.431177  1644 leader_election.cc:304] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:02:04.303617  8357 consensus_queue.cc:1048] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.123s
I20260709 13:02:04.433383  8376 raft_consensus.cc:2749] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:02:04.438201  2336 raft_consensus.cc:2468] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 19.
I20260709 13:02:04.441511  8298 raft_consensus.cc:515] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:04.450757  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 47185.2us
I20260709 13:02:04.458949  8330 raft_consensus.cc:515] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:02:04.460086  1645 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:04.461555  8330 leader_election.cc:290] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:04.462662  8336 raft_consensus.cc:515] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:04.464519  1754 raft_consensus.cc:1240] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 16. Current term is 17. Ops: []
W20260709 13:02:04.465142  8329 consensus_meta.cc:287] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.596s	user 0.013s	sys 0.000s
I20260709 13:02:04.450791  1600 raft_consensus.cc:2368] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for earlier term 14. Current term is 15.
W20260709 13:02:04.432348  7745 consensus_meta.cc:287] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.876s	user 0.013s	sys 0.002s
I20260709 13:02:04.432015  8327 raft_consensus.cc:2749] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
W20260709 13:02:04.468116  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 75814.9us
I20260709 13:02:04.468703  1605 raft_consensus.cc:2468] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 14.
I20260709 13:02:04.469580  1569 leader_election.cc:304] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:02:04.476712  1718 leader_election.cc:304] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:02:04.482425  1672 raft_consensus.cc:1275] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 15 index: 12. (index mismatch)
I20260709 13:02:04.486166  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:04.489398  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:04.489881  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:04.490906  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:04.492027  1605 raft_consensus.cc:2410] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for term 14 because replica has last-logged OpId of term: 13 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 8.
I20260709 13:02:04.492714  1718 leader_election.cc:304] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:04.493624  1671 raft_consensus.cc:2368] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 18. Current term is 19.
I20260709 13:02:04.494484  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 15 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:04.493032  1571 leader_election.cc:304] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:04.498566  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:04.499370  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:04.499982  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:04.504308  1677 raft_consensus.cc:3060] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:02:04.466643  8336 leader_election.cc:290] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 17 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:04.517654  1717 leader_election.cc:304] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:04.534291  7307 raft_consensus.cc:2749] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:02:04.542704  1675 raft_consensus.cc:1240] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 18. Current term is 19. Ops: []
I20260709 13:02:04.544706  1678 raft_consensus.cc:1240] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 18. Current term is 19. Ops: []
I20260709 13:02:04.547694  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:04.547813  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 15 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:04.548347  1676 raft_consensus.cc:3060] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:02:04.533627  1717 leader_election.cc:304] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:04.551371  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:04.489434  8329 raft_consensus.cc:2749] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:02:04.485198  8344 raft_consensus.cc:697] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 16 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:02:04.556289  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:04.562541  1717 leader_election.cc:304] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:04.556094  7307 raft_consensus.cc:2804] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 13:02:04.386006  8320 raft_consensus.cc:515] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:04.561816  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:04.563591  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:04.563608  7307 raft_consensus.cc:493] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:04.567023  7307 raft_consensus.cc:3060] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:02:04.511026  7728 raft_consensus.cc:2804] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:02:04.571830  1646 leader_election.cc:304] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
W20260709 13:02:04.564847  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 54767.9us
I20260709 13:02:04.570504  1643 leader_election.cc:304] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:04.573351  1718 leader_election.cc:304] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:04.574576  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:04.486222  8270 raft_consensus.cc:515] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:04.560812  8330 raft_consensus.cc:2804] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:02:04.582491  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 16 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:04.577294  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:04.582841  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:04.583577  1603 raft_consensus.cc:2393] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 16: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:02:04.579636  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:04.485860  7745 raft_consensus.cc:515] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:04.590340  1718 leader_election.cc:304] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:04.590992  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 17 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:04.592235  1643 leader_election.cc:304] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:02:04.594898  2330 raft_consensus.cc:2393] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 20: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:02:04.571452  8372 raft_consensus.cc:2804] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:02:04.571188  7728 raft_consensus.cc:493] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Starting leader election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:02:04.575240  8356 raft_consensus.cc:493] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:04.610010  1601 raft_consensus.cc:1240] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 15. Current term is 16. Ops: []
I20260709 13:02:04.614714  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:04.615355  1671 raft_consensus.cc:3060] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:02:04.582307  8330 raft_consensus.cc:493] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:04.582772  8327 raft_consensus.cc:2804] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:02:04.657383  1643 leader_election.cc:304] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:02:04.619535  1603 raft_consensus.cc:1275] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 16 index: 12. Preceding OpId from leader: term: 18 index: 13. (index mismatch)
I20260709 13:02:04.622831  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:04.624056  8329 raft_consensus.cc:2804] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:02:04.624811  1718 leader_election.cc:304] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:04.656873  8327 raft_consensus.cc:493] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Starting leader election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:02:04.580056  8326 consensus_queue.cc:1059] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 18.14, Next index: 15, Last known committed idx: 12, Time since last communication: 0.039s
I20260709 13:02:04.635444  7307 raft_consensus.cc:515] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:04.587447  8310 raft_consensus.cc:2749] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:02:04.631812  1751 raft_consensus.cc:2468] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 15.
I20260709 13:02:04.663643  8327 raft_consensus.cc:3060] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:02:04.606750  8372 raft_consensus.cc:493] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:04.622411  7728 raft_consensus.cc:3060] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:02:04.658591  8330 raft_consensus.cc:3060] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:02:04.587028  8317 consensus_queue.cc:1059] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 16.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.069s
I20260709 13:02:04.651055  8259 raft_consensus.cc:493] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:04.662034  8329 raft_consensus.cc:493] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:04.667102  8376 raft_consensus.cc:3060] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:02:04.650578  8356 raft_consensus.cc:515] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:04.686076  7307 leader_election.cc:290] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 23 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:04.697934  1603 raft_consensus.cc:1275] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
W20260709 13:02:04.701669  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 82256.9us
I20260709 13:02:04.696051  8127 raft_consensus.cc:2804] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:02:04.765483  8127 raft_consensus.cc:493] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:04.766005  8127 raft_consensus.cc:3060] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:02:04.700649  8259 raft_consensus.cc:515] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:04.701876  8329 raft_consensus.cc:3060] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:02:04.652881  1678 raft_consensus.cc:2468] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 13.
W20260709 13:02:04.750025  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 125389us
I20260709 13:02:04.780470  1718 leader_election.cc:304] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:02:04.722055  1675 raft_consensus.cc:1240] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 18. Current term is 19. Ops: []
I20260709 13:02:04.774513  2299 leader_election.cc:304] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:02:04.677595  8372 raft_consensus.cc:3060] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:02:04.735545  2332 raft_consensus.cc:2468] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 14.
I20260709 13:02:04.792043  7307 raft_consensus.cc:2749] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:02:04.794675  1718 leader_election.cc:304] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:04.795895  1677 raft_consensus.cc:2468] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 18.
I20260709 13:02:04.797418  1718 leader_election.cc:304] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1
W20260709 13:02:04.719739  1488 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet 983621a6fc5b4c9ea0159a648dfe7a36 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER failed with leader 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 3b879799899a499c8d409ee9f1277cd1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 18 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
W20260709 13:02:04.802453  1488 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 82829.4us
I20260709 13:02:04.823105  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:04.824359  1678 raft_consensus.cc:3060] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:02:04.701393  7745 leader_election.cc:290] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 16 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:04.831946  1643 leader_election.cc:304] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:02:04.840894  1605 raft_consensus.cc:1275] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 13 index: 7. Preceding OpId from leader: term: 16 index: 8. (index mismatch)
I20260709 13:02:04.685243  8145 raft_consensus.cc:515] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:04.714941  8298 leader_election.cc:290] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:04.718479  8320 leader_election.cc:290] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:04.849602  7307 raft_consensus.cc:2804] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:02:04.850069  7307 raft_consensus.cc:493] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:04.850462  7307 raft_consensus.cc:3060] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:02:04.772480  8310 raft_consensus.cc:493] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:02:04.865676  1678 raft_consensus.cc:2468] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 19.
I20260709 13:02:04.820262  8280 raft_consensus.cc:2804] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 13:02:04.878110  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:04.901669  1675 raft_consensus.cc:1240] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 19. Current term is 20. Ops: []
I20260709 13:02:04.885360  2298 leader_election.cc:304] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:04.905332  1749 raft_consensus.cc:3060] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:02:04.909505  8341 raft_consensus.cc:2749] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:02:04.910784  8269 consensus_queue.cc:1048] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 13:02:04.685041  8344 consensus_queue.cc:237] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.10, Last appended by leader: 10, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:02:04.960855  1645 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:04.960247  1675 raft_consensus.cc:1275] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 15 index: 11. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
I20260709 13:02:04.719650  8354 consensus_queue.cc:1059] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 18.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.164s
I20260709 13:02:04.974282  1749 raft_consensus.cc:2468] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 19.
I20260709 13:02:04.985277  1671 raft_consensus.cc:2468] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 17.
I20260709 13:02:04.849148  8371 raft_consensus.cc:2749] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:02:04.882925  8259 leader_election.cc:290] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:04.873740  8390 raft_consensus.cc:2749] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:02:04.994117  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
W20260709 13:02:04.942720  8378 log.cc:927] Time spent T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.060s	user 0.001s	sys 0.000s
I20260709 13:02:04.838897  8368 consensus_queue.cc:1059] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.148s
I20260709 13:02:05.002462  1749 raft_consensus.cc:3060] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Advancing to term 15
W20260709 13:02:04.985642  8307 consensus_meta.cc:287] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.623s	user 0.009s	sys 0.004s
W20260709 13:02:05.006984  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 502050us
I20260709 13:02:05.009085  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:05.010830  1675 raft_consensus.cc:3060] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:02:04.984325  8298 raft_consensus.cc:493] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
W20260709 13:02:05.011559  1748 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.661s	user 0.037s	sys 0.051s
W20260709 13:02:05.311174  1748 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.967s	user 0.038s	sys 0.051s
W20260709 13:02:05.173553  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 265169us
I20260709 13:02:05.312199  1748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55840 (request call id 2984) took 1273 ms. Trace:
I20260709 13:02:04.905534  8316 consensus_queue.cc:1048] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.097s
I20260709 13:02:04.972512  7307 raft_consensus.cc:515] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:04.883064  8310 raft_consensus.cc:515] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } }
I20260709 13:02:04.998272  8270 leader_election.cc:290] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 14 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 13:02:05.075961  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 280028us
I20260709 13:02:04.975997  8280 raft_consensus.cc:697] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 13 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
W20260709 13:02:05.012290  8384 log.cc:927] Time spent T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.221s	user 0.001s	sys 0.000s
I20260709 13:02:05.100128  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 15 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
W20260709 13:02:05.062770  8377 log.cc:927] Time spent T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.382s	user 0.001s	sys 0.000s
W20260709 13:02:05.138429  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 151159us
I20260709 13:02:05.336650  1671 raft_consensus.cc:3060] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:02:05.338173  2299 leader_election.cc:304] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:02:05.313565  8336 raft_consensus.cc:2749] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
W20260709 13:02:05.163321  8382 log.cc:927] Time spent T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.466s	user 0.001s	sys 0.000s
I20260709 13:02:05.013643  8307 raft_consensus.cc:515] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:05.341578  1750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55840 (request call id 2985) took 1302 ms. Trace:
I20260709 13:02:05.343037  8307 leader_election.cc:290] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 16 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:05.343689  8307 raft_consensus.cc:493] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:05.343588  8280 consensus_queue.cc:237] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.10, Last appended by leader: 10, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:05.344300  8307 raft_consensus.cc:515] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:04.985194  8317 raft_consensus.cc:493] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:02:04.994700  8320 raft_consensus.cc:2749] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:02:05.352165  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:05.354014  1569 leader_election.cc:304] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:02:05.342235  7307 leader_election.cc:290] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:05.355142  1747 raft_consensus.cc:1275] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 14 index: 13. Preceding OpId from leader: term: 15 index: 14. (index mismatch)
I20260709 13:02:05.354135  1751 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43156 (request call id 1339) took 1449 ms. Trace:
I20260709 13:02:05.351044  1643 leader_election.cc:304] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:05.341953  1750 rpcz_store.cc:276] 0709 13:02:04.039471 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:04.342392 (+302921us) service_pool.cc:224] Handling call
0709 13:02:04.342785 (+   393us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:04.343152 (+   367us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 13:02:04.374727 (+ 31575us) log.cc:844] Serialized 14 byte log entry
0709 13:02:04.576166 (+201439us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:04.582071 (+  5905us) log.cc:844] Serialized 26 byte log entry
0709 13:02:04.582215 (+   144us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:02:04.582253 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:04.582326 (+    73us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:05.341301 (+758975us) raft_consensus.cc:1685] finished
0709 13:02:05.341330 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:05.341501 (+   171us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":37369,"prepare.queue_time_us":31575,"prepare.run_cpu_time_us":63,"prepare.run_wall_time_us":66,"raft.queue_time_us":32343,"raft.run_cpu_time_us":214,"raft.run_wall_time_us":9447,"thread_start_us":169204,"threads_started":1,"wal-append.queue_time_us":211403}
I20260709 13:02:05.357920  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:05.359175  1605 raft_consensus.cc:2368] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 17. Current term is 18.
I20260709 13:02:05.360319  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:05.296327  1749 raft_consensus.cc:2468] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 15.
I20260709 13:02:05.360949  1605 raft_consensus.cc:3060] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Advancing to term 16
W20260709 13:02:05.296727  1676 consensus_meta.cc:287] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.734s	user 0.015s	sys 0.002s
I20260709 13:02:05.361778  1676 raft_consensus.cc:2468] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 16.
W20260709 13:02:05.298727  8334 consensus_meta.cc:287] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.871s	user 0.005s	sys 0.016s
I20260709 13:02:04.914659  8140 consensus_queue.cc:1059] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 19.15, Next index: 16, Last known committed idx: 14, Time since last communication: 0.145s
W20260709 13:02:05.313360  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 516267us
I20260709 13:02:05.367080  1718 leader_election.cc:304] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:05.356146  1751 rpcz_store.cc:276] 0709 13:02:03.904381 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:03.904530 (+   149us) service_pool.cc:224] Handling call
0709 13:02:05.353871 (+1449341us) spinlock_profiling.cc:243] Waited 521 ms on lock 0x7b54000e3b54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:02:05.354028 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1198957824}
I20260709 13:02:05.317308  1571 leader_election.cc:304] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:05.312649  1748 rpcz_store.cc:276] 0709 13:02:04.039105 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:04.336291 (+297186us) service_pool.cc:224] Handling call
0709 13:02:04.336625 (+   334us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:04.336882 (+   257us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 13:02:04.343616 (+  6734us) log.cc:844] Serialized 14 byte log entry
0709 13:02:05.311715 (+968099us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:05.311793 (+    78us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:02:05.311862 (+    69us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:05.311965 (+   103us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:05.312123 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":14091,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":44,"thread_start_us":967669,"threads_started":1}
W20260709 13:02:05.310400  7728 consensus_meta.cc:287] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.612s	user 0.013s	sys 0.004s
W20260709 13:02:05.331928  8389 log.cc:927] Time spent T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.337s	user 0.000s	sys 0.014s
I20260709 13:02:05.326668  8298 raft_consensus.cc:515] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:02:05.329303  8329 consensus_meta.cc:287] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.558s	user 0.008s	sys 0.010s
I20260709 13:02:05.348021  8310 leader_election.cc:290] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:05.365563  1671 raft_consensus.cc:2468] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 15.
I20260709 13:02:05.374089  8334 raft_consensus.cc:515] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:05.347427  8342 log.cc:927] Time spent T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 1.662s	user 0.008s	sys 0.004s
W20260709 13:02:05.429360  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 78434.6us
I20260709 13:02:05.430291  1643 leader_election.cc:304] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:05.380208  8336 raft_consensus.cc:2749] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:02:05.323870  8390 raft_consensus.cc:493] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:05.373225  1718 leader_election.cc:304] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 13:02:05.384760  8376 consensus_meta.cc:287] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.657s	user 0.006s	sys 0.012s
I20260709 13:02:05.442046  8376 raft_consensus.cc:2749] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:02:05.441792  8390 raft_consensus.cc:515] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:02:05.442476  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 69359.8us
I20260709 13:02:05.443476  1718 leader_election.cc:304] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:05.444428  8387 raft_consensus.cc:493] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:05.379277  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:05.383078  8356 leader_election.cc:290] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:02:05.325511  8323 consensus_meta.cc:287] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.884s	user 0.011s	sys 0.011s
I20260709 13:02:05.375968  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:05.379943  7728 raft_consensus.cc:515] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:05.352669  8307 leader_election.cc:290] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:05.381657  1751 raft_consensus.cc:1275] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 16 index: 12. (index mismatch)
I20260709 13:02:05.350646  8317 raft_consensus.cc:515] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:05.367347  8303 raft_consensus.cc:2749] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 13:02:05.414661  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 16 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
W20260709 13:02:05.357738  8327 consensus_meta.cc:287] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.631s	user 0.015s	sys 0.002s
W20260709 13:02:05.422695  1644 leader_election.cc:336] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 14 pre-election: RPC error from VoteRequest() call to peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625): Timed out: RequestConsensusVote RPC to 127.0.107.196:46625 timed out after 1.554s (SENT)
W20260709 13:02:05.355671  8328 consensus_meta.cc:287] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 1.038s	user 0.009s	sys 0.012s
I20260709 13:02:05.314044  8341 consensus_queue.cc:1048] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.102s
W20260709 13:02:05.437517  8127 consensus_meta.cc:287] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.671s	user 0.011s	sys 0.006s
I20260709 13:02:05.440929  8334 leader_election.cc:290] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 23 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:05.445331  1748 raft_consensus.cc:3060] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:02:05.445660  2337 raft_consensus.cc:2393] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 23: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:02:05.444995  1718 leader_election.cc:304] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:02:05.486117  8395 consensus_queue.cc:1059] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 12.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.026s
I20260709 13:02:05.486936  8404 raft_consensus.cc:493] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:02:05.487032  8400 consensus_queue.cc:1059] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 18.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.061s
I20260709 13:02:05.487528  8270 raft_consensus.cc:2749] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:02:05.487751  1718 leader_election.cc:304] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 13:02:05.487622  8404 raft_consensus.cc:515] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:05.488466  8393 raft_consensus.cc:2749] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:02:05.489210  8393 raft_consensus.cc:2764] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Leader pre-election decision vote started in defunct term 16: won
I20260709 13:02:05.489645  1718 leader_election.cc:400] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 18 pre-election: Vote denied by peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261) with higher term. Message: Invalid argument: T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 18. Current term is 19.
I20260709 13:02:05.489800  7728 leader_election.cc:290] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 20 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:05.490043  1718 leader_election.cc:403] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 18 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:02:05.470952  8376 raft_consensus.cc:2749] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 13:02:05.427767  8329 raft_consensus.cc:515] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:05.431736  8145 leader_election.cc:290] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 19 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:05.445047  8387 raft_consensus.cc:515] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:05.494978  1680 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52766 (request call id 3068) took 2086 ms. Trace:
I20260709 13:02:05.444931  8390 leader_election.cc:290] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 13:02:05.480091  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 57482.6us
I20260709 13:02:05.480705  1751 raft_consensus.cc:1240] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 15. Current term is 16. Ops: [15.7-15.7]
I20260709 13:02:05.462494  8154 raft_consensus.cc:2804] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Leader election won for term 18
W20260709 13:02:05.472525  8330 consensus_meta.cc:287] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.776s	user 0.009s	sys 0.007s
I20260709 13:02:05.486593  8402 raft_consensus.cc:2749] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:02:05.415652  8326 raft_consensus.cc:2749] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:02:05.488106  8400 raft_consensus.cc:3055] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 LEADER]: Stepping down as leader of term 15
I20260709 13:02:05.565315  1748 raft_consensus.cc:2468] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 17.
I20260709 13:02:05.388940  8354 raft_consensus.cc:2804] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 13:02:05.476209  1605 raft_consensus.cc:2468] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 16.
W20260709 13:02:05.521350  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 182012us
I20260709 13:02:05.520470  1680 rpcz_store.cc:276] 0709 13:02:03.408676 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:03.408807 (+   131us) service_pool.cc:224] Handling call
0709 13:02:03.409198 (+   391us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:03.432077 (+ 22879us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:03.432116 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:03.432695 (+   579us) log.cc:844] Serialized 26 byte log entry
0709 13:02:03.584147 (+151452us) raft_consensus.cc:1645] Marking committed up to 14
0709 13:02:03.584214 (+    67us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:03.584316 (+   102us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:05.346905 (+1762589us) spinlock_profiling.cc:243] Waited 21.4 ms on lock 0x7b24003fa058. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f4262efb9c4 00007f4262ef9acb 00007f42630b4dc9 00007f42630b4a55 00007f42630b409d 00007f4265ce610e 00007f4265cfd88c 00007f4265cfcbb6 00007f4265cfc7d8 00007f4265d06221
0709 13:02:05.494539 (+147634us) raft_consensus.cc:1685] finished
0709 13:02:05.494573 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:05.494888 (+   315us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":10839,"raft.queue_time_us":14940,"raft.run_cpu_time_us":185,"raft.run_wall_time_us":16422,"spinlock_wait_cycles":49190272,"thread_start_us":102537,"threads_started":1,"wal-append.queue_time_us":151570}
W20260709 13:02:05.583781  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 228390us
I20260709 13:02:05.584038  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:05.490345  8404 leader_election.cc:290] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:05.543030  2335 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39726 (request call id 1498) took 1028 ms. Trace:
W20260709 13:02:05.553972  8388 log.cc:927] Time spent T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.120s	user 0.001s	sys 0.001s
I20260709 13:02:05.509977  8270 raft_consensus.cc:2749] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Leader pre-election lost for term 18. Reason: Vote denied by peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261) with higher term. Message: Invalid argument: T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 18. Current term is 19.
I20260709 13:02:05.589323  2335 rpcz_store.cc:276] 0709 13:02:04.514395 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:04.514561 (+   166us) service_pool.cc:224] Handling call
0709 13:02:04.558181 (+ 43620us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:04.558457 (+   276us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:04.558492 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:04.577834 (+ 19342us) log.cc:844] Serialized 26 byte log entry
0709 13:02:05.021545 (+443711us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:02:05.021605 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:05.021727 (+   122us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:05.542719 (+520992us) raft_consensus.cc:1685] finished
0709 13:02:05.542751 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:05.542954 (+   203us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":411739,"threads_started":1,"wal-append.queue_time_us":890129}
I20260709 13:02:05.550693  1643 leader_election.cc:304] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:05.542902  8387 leader_election.cc:290] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:02:05.545239  8348 log.cc:927] Time spent T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 1.800s	user 0.005s	sys 0.000s
I20260709 13:02:05.547806  8317 leader_election.cc:290] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:05.502682  8298 leader_election.cc:290] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:05.486243  8386 raft_consensus.cc:2749] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:02:05.586486  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:05.624399  1753 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55840 (request call id 2987) took 1584 ms. Trace:
I20260709 13:02:05.624765  1680 raft_consensus.cc:3060] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:02:05.477658  8327 raft_consensus.cc:515] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:05.582567  1747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43156 (request call id 1343) took 1677 ms. Trace:
I20260709 13:02:05.592541  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:05.527379  8284 raft_consensus.cc:493] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:05.594666  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:05.595731  1748 raft_consensus.cc:1275] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 16 index: 9. Preceding OpId from leader: term: 19 index: 10. (index mismatch)
I20260709 13:02:05.483937  8328 raft_consensus.cc:515] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:05.497426  8127 raft_consensus.cc:515] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:05.602361  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 19 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:05.606931  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:05.642275  1676 raft_consensus.cc:2468] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 18.
I20260709 13:02:05.608002  8190 raft_consensus.cc:3055] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 13:02:05.556134  8330 raft_consensus.cc:515] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:05.592941  8370 log.cc:927] Time spent T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.295s	user 0.008s	sys 0.003s
I20260709 13:02:05.577872  8400 raft_consensus.cc:740] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
W20260709 13:02:05.615170  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 152061us
W20260709 13:02:05.616714  8398 log.cc:927] Time spent T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.317s	user 0.000s	sys 0.001s
I20260709 13:02:05.616634  8350 raft_consensus.cc:3055] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 13:02:05.598682  8354 raft_consensus.cc:697] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 17 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:02:05.621376  8404 raft_consensus.cc:2749] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
W20260709 13:02:05.621716  8372 consensus_meta.cc:287] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.781s	user 0.007s	sys 0.008s
I20260709 13:02:05.560268  8154 raft_consensus.cc:697] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:02:05.675551  8154 consensus_queue.cc:237] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.11, Last appended by leader: 11, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:05.624979  1753 rpcz_store.cc:276] 0709 13:02:04.040088 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:04.487784 (+447696us) service_pool.cc:224] Handling call
0709 13:02:04.488227 (+   443us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:04.494161 (+  5934us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:04.494206 (+    45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:04.494894 (+   688us) log.cc:844] Serialized 26 byte log entry
0709 13:02:04.839849 (+344955us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:02:04.839913 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:04.841068 (+  1155us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:05.553106 (+712038us) spinlock_profiling.cc:243] Waited 11.8 ms on lock 0x7b4400022b4c. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f4262f9bc85 00007f4262f9c6a2 00007f4265d0e1db 00007f4265d09aaf 00007f4265cfd86e 00007f4265cfcbb6 00007f4265cfc7d8 00007f4265d06221 00007f4265d061d9 00007f4265d06169
0709 13:02:05.623965 (+ 70859us) raft_consensus.cc:1685] finished
0709 13:02:05.623990 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:05.624304 (+   314us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":19950,"raft.run_cpu_time_us":174,"raft.run_wall_time_us":23087,"spinlock_wait_cycles":27084672,"thread_start_us":267835,"threads_started":1,"wal-append.queue_time_us":515560}
I20260709 13:02:05.632730  1672 raft_consensus.cc:1240] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 18. Current term is 19. Ops: []
I20260709 13:02:05.636914  1747 rpcz_store.cc:276] 0709 13:02:03.905353 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:03.975420 (+ 70067us) service_pool.cc:224] Handling call
0709 13:02:05.354791 (+1379371us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:05.355506 (+   715us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:05.578017 (+222511us) spinlock_profiling.cc:243] Waited 1.35 s on lock 0x7b54000e3b50. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661618ad 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:02:05.582461 (+  4444us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3115997952}
I20260709 13:02:05.640095  1748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55848 (request call id 3096) took 1031 ms. Trace:
I20260709 13:02:05.679081  1748 rpcz_store.cc:276] 0709 13:02:04.608598 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:05.578910 (+970312us) service_pool.cc:224] Handling call
0709 13:02:05.585105 (+  6195us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:05.639697 (+ 54592us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:05.640018 (+   321us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:05.645180  2335 raft_consensus.cc:2393] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 19: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:02:05.645924  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:05.680130  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 19 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:05.680737  1605 raft_consensus.cc:3060] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:02:05.665923  8327 leader_election.cc:290] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 16 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:05.666494  1643 leader_election.cc:304] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:02:05.669588  8344 raft_consensus.cc:493] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:05.662827  8350 raft_consensus.cc:740] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:02:05.671947  8354 consensus_queue.cc:237] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:05.673841  8372 raft_consensus.cc:515] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:05.694054  1599 raft_consensus.cc:1240] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 19. Current term is 20. Ops: []
I20260709 13:02:05.746796  1605 raft_consensus.cc:2468] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 14.
W20260709 13:02:05.761770  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 444762us
W20260709 13:02:05.723235  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 203765us
W20260709 13:02:05.749662  2331 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39726 (request call id 1493) took 1997 ms (client timeout 1554 ms). Trace:
I20260709 13:02:05.724052  1644 leader_election.cc:304] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:05.731084  1676 raft_consensus.cc:1240] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 19. Current term is 20. Ops: []
I20260709 13:02:05.766865  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:05.682379  1672 raft_consensus.cc:1275] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
I20260709 13:02:05.747529  2328 raft_consensus.cc:1275] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
I20260709 13:02:05.716687  1675 raft_consensus.cc:2468] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 15.
I20260709 13:02:05.763803  1717 leader_election.cc:304] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 13:02:05.765170  2331 rpcz_store.cc:269] 0709 13:02:03.749170 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:03.749303 (+   133us) service_pool.cc:224] Handling call
0709 13:02:05.746783 (+1997480us) spinlock_profiling.cc:243] Waited 658 ms on lock 0x7b54005e3b54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:02:05.746933 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1514437376}
W20260709 13:02:05.778553  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 112238us
I20260709 13:02:05.779944  1643 leader_election.cc:304] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:05.698143  1751 raft_consensus.cc:1240] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 18. Current term is 19. Ops: []
I20260709 13:02:05.687305  1680 raft_consensus.cc:2468] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 16.
I20260709 13:02:05.694118  1749 raft_consensus.cc:1240] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 15. Current term is 16. Ops: []
I20260709 13:02:05.776218  1644 leader_election.cc:304] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:02:05.783190  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:05.784061  1605 raft_consensus.cc:2368] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for earlier term 14. Current term is 15.
I20260709 13:02:05.650736  8190 raft_consensus.cc:740] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:02:05.787230  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 17 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:05.788208  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:05.788899  1605 raft_consensus.cc:3060] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:02:05.788252  1676 raft_consensus.cc:2410] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for term 17 because replica has last-logged OpId of term: 15 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 8.
I20260709 13:02:05.659353  7745 raft_consensus.cc:493] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:05.729069  8280 raft_consensus.cc:493] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:05.794683  1754 raft_consensus.cc:2393] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 14: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:02:05.797361  1717 leader_election.cc:400] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 17 pre-election: Vote denied by peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445) with higher term. Message: Invalid argument: T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 17. Current term is 18.
I20260709 13:02:05.798264  1717 leader_election.cc:403] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 17 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:02:05.793246  8414 raft_consensus.cc:493] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:05.799806  8414 raft_consensus.cc:515] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:05.801525  1745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43156 (request call id 1344) took 1896 ms. Trace:
I20260709 13:02:05.782702  8404 raft_consensus.cc:2749] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:02:05.801947  1745 rpcz_store.cc:276] 0709 13:02:03.905502 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:03.978118 (+ 72616us) service_pool.cc:224] Handling call
0709 13:02:05.348946 (+1370828us) spinlock_profiling.cc:243] Waited 21.9 ms on lock 0x7b58001ffe98. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f426303db71 00007f426303da22 00007f426bf56c09 00007f426b0b0ee0 00007f426b17a0ab 00007f426b17a131 00007f426b13bfc0 00007f426b126929 00007f426b1263d2 00007f426469dbd4
0709 13:02:05.349196 (+   250us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:05.349598 (+   402us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:05.349637 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:05.350218 (+   581us) log.cc:844] Serialized 26 byte log entry
0709 13:02:05.583322 (+233104us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:02:05.744276 (+160954us) log.cc:844] Serialized 14 byte log entry
0709 13:02:05.749216 (+  4940us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:05.749342 (+   126us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:05.801181 (+ 51839us) raft_consensus.cc:1685] finished
0709 13:02:05.801296 (+   115us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:05.801451 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1446,"prepare.queue_time_us":23526,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":37,"raft.queue_time_us":8711,"raft.run_cpu_time_us":290,"raft.run_wall_time_us":24717,"spinlock_wait_cycles":50460928,"thread_start_us":367394,"threads_started":2,"wal-append.queue_time_us":388931}
I20260709 13:02:05.803820  8280 raft_consensus.cc:515] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:05.804446  8390 raft_consensus.cc:2749] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 13:02:05.805368  8334 raft_consensus.cc:493] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:05.805943  8334 raft_consensus.cc:515] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:05.785387  8270 raft_consensus.cc:493] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:05.806865  8270 raft_consensus.cc:515] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:05.808180  8334 leader_election.cc:290] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:05.809141  8270 leader_election.cc:290] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:05.811520  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:05.812712  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 15 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:05.813593  2331 raft_consensus.cc:3060] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:02:05.653288  8298 raft_consensus.cc:493] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:02:05.775426  8386 raft_consensus.cc:493] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:05.806160  8280 leader_election.cc:290] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:05.782869  8317 raft_consensus.cc:2749] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
W20260709 13:02:05.841049  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 43869us
I20260709 13:02:05.841886  1717 leader_election.cc:304] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:05.842926  1717 leader_election.cc:304] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:05.837446  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:05.844329  1676 raft_consensus.cc:2468] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 19.
I20260709 13:02:05.836308  7728 raft_consensus.cc:493] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:05.845767  8145 raft_consensus.cc:2749] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Leader pre-election lost for term 17. Reason: Vote denied by peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445) with higher term. Message: Invalid argument: T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 17. Current term is 18.
I20260709 13:02:05.845824  7728 raft_consensus.cc:515] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:05.846737  8145 raft_consensus.cc:2804] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 13:02:05.847273  8145 raft_consensus.cc:697] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 16 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:02:05.848050  8379 raft_consensus.cc:2804] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:02:05.848425  7728 leader_election.cc:290] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:05.848215  8145 consensus_queue.cc:237] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 7, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:05.783069  8400 consensus_queue.cc:260] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.12, Last appended by leader: 12, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:05.848655  8379 raft_consensus.cc:697] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 14 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:02:05.659626  8387 consensus_queue.cc:1048] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.136s
W20260709 13:02:05.839514  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 59665.4us
I20260709 13:02:05.853425  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 16 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:05.809165  8328 leader_election.cc:290] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:05.793695  8402 raft_consensus.cc:493] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:05.854147  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:05.857234  1605 raft_consensus.cc:2468] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 16.
I20260709 13:02:05.858731  1643 leader_election.cc:304] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:05.859678  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:05.810316  8326 raft_consensus.cc:2764] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Leader election decision vote started in defunct term 14: won
I20260709 13:02:05.855356  1676 raft_consensus.cc:2410] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 for term 16 because replica has last-logged OpId of term: 13 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 10.
I20260709 13:02:05.853864  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:05.735046  8344 raft_consensus.cc:515] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:05.658991  8284 raft_consensus.cc:515] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:05.850451  8400 raft_consensus.cc:3060] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:02:05.802420  8414 leader_election.cc:290] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:05.867946  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:05.854358  8404 raft_consensus.cc:2749] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:02:05.869643  8317 raft_consensus.cc:2749] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
W20260709 13:02:05.879169  8407 log.cc:927] Time spent T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.413s	user 0.004s	sys 0.000s
W20260709 13:02:05.871702  8413 log.cc:927] Time spent T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.062s	user 0.000s	sys 0.000s
I20260709 13:02:05.873946  1599 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 2837) took 1420 ms. Trace:
I20260709 13:02:05.880868  1603 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 2839) took 1426 ms. Trace:
I20260709 13:02:05.881645  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:05.883126  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 19 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:05.881284  1603 rpcz_store.cc:276] 0709 13:02:04.454200 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:04.819047 (+364847us) service_pool.cc:224] Handling call
0709 13:02:04.819496 (+   449us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:04.819894 (+   398us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:04.819934 (+    40us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:04.844955 (+ 25021us) log.cc:844] Serialized 26 byte log entry
0709 13:02:05.463850 (+618895us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:02:05.463898 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:05.463975 (+    77us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:05.878512 (+414537us) spinlock_profiling.cc:243] Waited 5.87 ms on lock 0x7b4400022b4c. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f4262f9bc85 00007f4262f9c6a2 00007f4265d0e1db 00007f4265d09aaf 00007f4265cfd86e 00007f4265cfcbb6 00007f4265cfc7d8 00007f4265d06221 00007f4265d061d9 00007f4265d06169
0709 13:02:05.880460 (+  1948us) raft_consensus.cc:1685] finished
0709 13:02:05.880485 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:05.880797 (+   312us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6661,"raft.queue_time_us":81691,"raft.run_cpu_time_us":8795,"raft.run_wall_time_us":77078,"spinlock_wait_cycles":13491072,"thread_start_us":166166,"threads_started":1,"wal-append.queue_time_us":552986}
I20260709 13:02:05.886797  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:05.887521  1603 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40154 (request call id 2752) took 1009 ms. Trace:
I20260709 13:02:05.887828  1603 rpcz_store.cc:276] 0709 13:02:04.878334 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:05.886523 (+1008189us) service_pool.cc:224] Handling call
0709 13:02:05.887459 (+   936us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:05.888881  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:05.889206  1717 leader_election.cc:304] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:05.880946  1599 rpcz_store.cc:276] 0709 13:02:04.453570 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:04.766707 (+313137us) service_pool.cc:224] Handling call
0709 13:02:04.767216 (+   509us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:05.775647 (+1008431us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:05.873666 (+ 98019us) spinlock_profiling.cc:243] Waited 927 ms on lock 0x7b540006d6d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:02:05.873854 (+   188us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2132026240}
I20260709 13:02:05.890584  8270 raft_consensus.cc:2749] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:02:05.890950  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:05.890861  8376 consensus_queue.cc:1048] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 7, Time since last communication: 0.022s
I20260709 13:02:05.891705  1599 raft_consensus.cc:3060] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:02:05.897740  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 15 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:05.863641  8319 consensus_queue.cc:1048] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.036s
I20260709 13:02:05.902909  8127 leader_election.cc:290] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 20 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:05.904870  8372 leader_election.cc:290] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 14 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:05.905627  8329 leader_election.cc:290] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 19 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:05.910796  1605 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40154 (request call id 2757) took 1030 ms. Trace:
I20260709 13:02:05.889417  1603 raft_consensus.cc:3060] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:02:05.912087  1605 rpcz_store.cc:276] 0709 13:02:04.879992 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:05.867409 (+987417us) service_pool.cc:224] Handling call
0709 13:02:05.910706 (+ 43297us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:05.913013  1643 leader_election.cc:400] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 14 election: Vote denied by peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445) with higher term. Message: Invalid argument: T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for earlier term 14. Current term is 15.
W20260709 13:02:05.913846  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 48201.2us
I20260709 13:02:05.914783  1602 raft_consensus.cc:2410] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for term 15 because replica has last-logged OpId of term: 13 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 8.
W20260709 13:02:05.917092  1643 connection.cc:256] RPC call timeout handler was delayed by -1.2903132079999864s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15300000000000002s for a response
W20260709 13:02:05.911221  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 66797.9us
I20260709 13:02:05.863876  7745 raft_consensus.cc:515] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:05.919977  8323 consensus_queue.cc:1048] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 13:02:05.920490  7745 leader_election.cc:290] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:05.905645  8414 raft_consensus.cc:493] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:05.864195  8402 raft_consensus.cc:515] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:05.815023  8350 consensus_queue.cc:260] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.15, Last appended by leader: 15, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:05.841437  8386 raft_consensus.cc:515] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:05.923168  8402 leader_election.cc:290] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:05.921176  8414 raft_consensus.cc:515] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:05.923722  8406 consensus_queue.cc:1175] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "af520b67945d443b901bbeca37cc5701" responder_term: 18 status { last_received { term: 17 index: 14 } last_committed_idx: 14 error { code: PRECEDING_ENTRY_DIDNT_MATCH status { code: ILLEGAL_STATE message: "Log matching property violated. Preceding OpId in replica: term: 17 index: 14. Preceding OpId from leader: term: 18 index: 15. (index mismatch)" } } last_received_current_leader { term: 0 index: 0 } }
I20260709 13:02:05.924196  8379 consensus_queue.cc:237] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 13.9, Last appended by leader: 9, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:05.923079  2301 leader_election.cc:304] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:05.925319  8344 leader_election.cc:290] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:05.915747  1718 leader_election.cc:304] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:05.926851  8408 raft_consensus.cc:2804] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:02:05.927137  8414 leader_election.cc:290] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:05.927340  8408 raft_consensus.cc:493] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Starting leader election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:02:05.927774  8408 raft_consensus.cc:3060] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:02:05.930912  1750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43156 (request call id 1357) took 1352 ms. Trace:
I20260709 13:02:05.931341  1750 rpcz_store.cc:276] 0709 13:02:04.578179 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:05.463364 (+885185us) service_pool.cc:224] Handling call
0709 13:02:05.478962 (+ 15598us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:05.479494 (+   532us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 13:02:05.526452 (+ 46958us) log.cc:844] Serialized 14 byte log entry
0709 13:02:05.859582 (+333130us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:05.881919 (+ 22337us) log.cc:844] Serialized 26 byte log entry
0709 13:02:05.882089 (+   170us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:02:05.882126 (+    37us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:05.882196 (+    70us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:05.930668 (+ 48472us) raft_consensus.cc:1685] finished
0709 13:02:05.930695 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:05.930838 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":9448,"prepare.queue_time_us":46623,"prepare.run_cpu_time_us":52,"prepare.run_wall_time_us":55,"raft.queue_time_us":3933,"raft.run_cpu_time_us":290,"raft.run_wall_time_us":16033,"thread_start_us":260610,"threads_started":1,"wal-append.queue_time_us":341469}
I20260709 13:02:05.933468  8344 raft_consensus.cc:2804] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 13:02:05.934026  8344 raft_consensus.cc:697] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 16 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:02:05.934881  8127 raft_consensus.cc:493] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:05.935088  8344 consensus_queue.cc:237] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.7, Last appended by leader: 7, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:05.935475  8127 raft_consensus.cc:515] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:05.922318  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 21 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:05.938834  8127 leader_election.cc:290] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:05.939635  1676 raft_consensus.cc:2410] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for term 21 because replica has last-logged OpId of term: 19 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 11.
I20260709 13:02:05.940797  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:05.842092  8298 raft_consensus.cc:515] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:02:05.951236  1643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:05.956152  2299 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:05.951973  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:05.962473  1605 raft_consensus.cc:2468] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 14.
I20260709 13:02:05.964762  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:05.966563  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:05.953783  1599 raft_consensus.cc:2468] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 16.
I20260709 13:02:05.865299  8280 raft_consensus.cc:493] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:05.971004  1599 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40154 (request call id 2755) took 1091 ms. Trace:
I20260709 13:02:05.971101  8280 raft_consensus.cc:515] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:05.971452  1599 rpcz_store.cc:276] 0709 13:02:04.879365 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:05.890677 (+1011312us) service_pool.cc:224] Handling call
0709 13:02:05.970938 (+ 80261us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:05.972491  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:05.973296  8317 raft_consensus.cc:493] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:02:05.973655  8280 leader_election.cc:290] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:05.945169  1603 raft_consensus.cc:2468] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 23.
I20260709 13:02:05.974128  8317 raft_consensus.cc:515] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:05.978472  8317 leader_election.cc:290] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:05.979972  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:05.981011  1603 raft_consensus.cc:2393] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 16: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:02:05.982060  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:05.983044  1603 raft_consensus.cc:2393] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 23: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:02:05.983613  1603 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40154 (request call id 2759) took 1103 ms. Trace:
I20260709 13:02:05.984411  1680 raft_consensus.cc:2393] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 19: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:02:05.984162  1603 rpcz_store.cc:276] 0709 13:02:04.880566 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:05.981769 (+1101203us) service_pool.cc:224] Handling call
0709 13:02:05.983499 (+  1730us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:05.986752  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:05.987766  1599 raft_consensus.cc:3060] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:02:05.985767  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:05.992173  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 19 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:05.993517  1603 raft_consensus.cc:2468] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 20.
I20260709 13:02:06.004105  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:06.007439  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:06.008311  1680 raft_consensus.cc:2468] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 20.
I20260709 13:02:06.009418  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 15 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:06.016695  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:06.018059  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:06.019086  1603 raft_consensus.cc:2393] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 19: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:02:06.019796  1676 raft_consensus.cc:2368] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 16. Current term is 17.
I20260709 13:02:06.020999  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:06.023886  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:06.024173  1717 leader_election.cc:304] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:06.025797  1717 leader_election.cc:304] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:06.028352  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:06.075384  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:06.075747  1605 raft_consensus.cc:2468] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 15.
I20260709 13:02:06.076757  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:06.077538  1605 raft_consensus.cc:3060] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:02:06.032663  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:06.024926  8154 raft_consensus.cc:493] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:06.034654  8402 raft_consensus.cc:2749] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 13:02:06.036499  1599 raft_consensus.cc:2468] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 19.
I20260709 13:02:06.036789  8420 raft_consensus.cc:493] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:06.038268  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:06.092294  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:06.038704  8145 raft_consensus.cc:2804] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Leader pre-election won for term 15
W20260709 13:02:06.039681  1570 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:06.040071  8330 leader_election.cc:290] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 19 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:06.041684  7687 raft_consensus.cc:493] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:05.993709  8320 consensus_queue.cc:1059] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.049s
I20260709 13:02:06.043505  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:06.043985  8284 leader_election.cc:290] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:06.101495  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:06.047396  1569 leader_election.cc:304] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:02:06.047688  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:06.049639  8294 leader_election.cc:290] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:06.104871  1604 raft_consensus.cc:1240] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 13. Current term is 14. Ops: []
I20260709 13:02:06.105345  8300 raft_consensus.cc:3060] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:02:06.105618  1604 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 2840) took 1651 ms. Trace:
I20260709 13:02:06.107316  8408 raft_consensus.cc:515] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:06.108099  1601 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 1651) took 2117 ms. Trace:
W20260709 13:02:06.102833  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 55564.9us
I20260709 13:02:06.108597  1601 rpcz_store.cc:276] 0709 13:02:03.990799 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:03.991373 (+   574us) service_pool.cc:224] Handling call
0709 13:02:04.017934 (+ 26561us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:04.615696 (+597762us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:06.107806 (+1492110us) spinlock_profiling.cc:243] Waited 592 ms on lock 0x7b5400215cd4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:02:06.108028 (+   222us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1361814912}
W20260709 13:02:06.052225  8385 log.cc:927] Time spent T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 1.341s	user 0.000s	sys 0.004s
I20260709 13:02:06.111936  8352 raft_consensus.cc:2804] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader election won for term 15
I20260709 13:02:06.061976  8386 leader_election.cc:290] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:06.112596  8352 raft_consensus.cc:697] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:02:06.112669  8298 leader_election.cc:290] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:06.114101  8294 consensus_queue.cc:1059] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 19.15, Next index: 16, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 13:02:06.114506  8386 consensus_queue.cc:1048] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 13:02:06.115412  1601 raft_consensus.cc:1240] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 15. Current term is 16. Ops: []
I20260709 13:02:06.115792  1746 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55848 (request call id 3090) took 2091 ms. Trace:
I20260709 13:02:06.095510  8145 raft_consensus.cc:493] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:06.116483  8145 raft_consensus.cc:3060] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:02:06.116868  1601 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 2842) took 1661 ms. Trace:
W20260709 13:02:06.070011  1643 leader_election.cc:336] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 14 election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 1.530s (SENT)
I20260709 13:02:06.073184  2299 leader_election.cc:304] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:02:06.074817  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:06.018606  8414 raft_consensus.cc:493] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:06.088296  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:02:06.120445  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 47423.1us
I20260709 13:02:06.118937  1601 rpcz_store.cc:276] 0709 13:02:04.455098 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:06.110197 (+1655099us) service_pool.cc:224] Handling call
0709 13:02:06.114903 (+  4706us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:06.115908 (+  1005us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:06.116112 (+   204us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:06.089856  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:06.082755  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 16 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:06.096719  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:06.123817  2330 raft_consensus.cc:2468] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 19.
I20260709 13:02:05.910708  8190 consensus_queue.cc:260] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 19.15, Last appended by leader: 15, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:06.120471  1602 raft_consensus.cc:2468] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 18.
I20260709 13:02:06.127926  1676 raft_consensus.cc:2410] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 for term 21 because replica has last-logged OpId of term: 19 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 14.
W20260709 13:02:06.127858  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 180699us
W20260709 13:02:06.101045  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 121958us
I20260709 13:02:06.046377  8309 raft_consensus.cc:3055] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 16 LEADER]: Stepping down as leader of term 16
I20260709 13:02:06.102784  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:06.093855  8154 raft_consensus.cc:515] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:06.096364  8420 raft_consensus.cc:515] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:06.103379  7687 raft_consensus.cc:515] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:06.104233  8320 consensus_queue.cc:1243] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer 83595ad0fd5940c0a2bfc7c61acd969a log is divergent from this leader: its last log entry 18.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 9
I20260709 13:02:06.106012  1604 rpcz_store.cc:276] 0709 13:02:04.454526 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:06.104294 (+1649768us) service_pool.cc:224] Handling call
0709 13:02:06.104682 (+   388us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:06.105309 (+   627us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:06.105532 (+   223us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:06.061383  8329 raft_consensus.cc:493] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:06.110231  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:06.138461  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:06.117424  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:06.025663  1718 leader_election.cc:304] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
W20260709 13:02:06.139541  1570 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:06.139441  1671 raft_consensus.cc:2393] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 19: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:02:06.142004  1671 raft_consensus.cc:1275] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 13 index: 11. (index mismatch)
I20260709 13:02:06.130249  8309 raft_consensus.cc:740] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:02:06.142658  8420 leader_election.cc:290] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:06.143450  8309 consensus_queue.cc:260] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.9, Last appended by leader: 9, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:06.129941  8260 raft_consensus.cc:493] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:06.144676  8309 raft_consensus.cc:3060] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:02:06.145071  8260 raft_consensus.cc:515] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:06.146643  7687 leader_election.cc:290] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:06.146958  8330 raft_consensus.cc:2804] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:02:06.147414  8330 raft_consensus.cc:493] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:06.147782  8330 raft_consensus.cc:3060] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:02:06.121294  2299 leader_election.cc:304] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:02:06.116245  1746 rpcz_store.cc:276] 0709 13:02:04.024154 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:04.285307 (+261153us) service_pool.cc:224] Handling call
0709 13:02:04.293126 (+  7819us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:04.307701 (+ 14575us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:04.307741 (+    40us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:04.357973 (+ 50232us) log.cc:844] Serialized 26 byte log entry
0709 13:02:04.559613 (+201640us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:02:04.559658 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:04.559728 (+    70us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:06.051381 (+1491653us) spinlock_profiling.cc:243] Waited 11.2 ms on lock 0x7b4400022b4c. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f4262f9bc85 00007f4262f9c6a2 00007f4265d0e1db 00007f4265d09aaf 00007f4265cfd86e 00007f4265cfcbb6 00007f4265cfc7d8 00007f4265d06221 00007f4265d061d9 00007f4265d06169
0709 13:02:06.115389 (+ 64008us) raft_consensus.cc:1685] finished
0709 13:02:06.115417 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:06.115706 (+   289us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":25751424,"thread_start_us":74772,"threads_started":1,"wal-append.queue_time_us":284870}
I20260709 13:02:06.150486  1680 raft_consensus.cc:1275] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 16 index: 13. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
I20260709 13:02:06.130304  2331 raft_consensus.cc:2468] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 15.
I20260709 13:02:06.130518  1643 leader_election.cc:304] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:02:06.130496  1677 raft_consensus.cc:2468] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 19.
I20260709 13:02:06.132627  1599 raft_consensus.cc:1240] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 19. Current term is 20. Ops: []
I20260709 13:02:06.133395  1602 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 1662) took 1260 ms. Trace:
I20260709 13:02:06.113757  8352 consensus_queue.cc:237] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 13.13, Last appended by leader: 13, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:06.135596  1601 raft_consensus.cc:1275] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 13 index: 9. Preceding OpId from leader: term: 14 index: 9. (term mismatch)
W20260709 13:02:06.136799  8421 log.cc:927] Time spent T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.143s	user 0.002s	sys 0.003s
I20260709 13:02:06.134702  8269 raft_consensus.cc:493] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:02:06.155100  1601 pending_rounds.cc:85] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a: Aborting all ops after (but not including) 8
W20260709 13:02:06.140023  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 114501us
I20260709 13:02:06.158051  1601 pending_rounds.cc:107] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a: Aborting uncommitted NO_OP operation due to leader change: 13.9
W20260709 13:02:06.032114  8422 log.cc:927] Time spent T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.060s	user 0.001s	sys 0.000s
I20260709 13:02:06.150776  2299 leader_election.cc:304] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:02:06.130901  8354 raft_consensus.cc:2804] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 13:02:06.152366  1599 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40154 (request call id 2758) took 1272 ms. Trace:
I20260709 13:02:06.151954  1602 rpcz_store.cc:276] 0709 13:02:04.872936 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:06.132550 (+1259614us) service_pool.cc:224] Handling call
0709 13:02:06.132854 (+   304us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:06.133059 (+   205us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:06.133094 (+    35us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:06.133146 (+    52us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:02:06.133184 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:06.133242 (+    58us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:06.133337 (+    95us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:06.066171  1679 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52766 (request call id 3066) took 2658 ms. Trace:
I20260709 13:02:06.172340  2299 leader_election.cc:304] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:02:06.158370  1601 raft_consensus.cc:2889] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:02:06.144685  8154 leader_election.cc:290] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:06.159999  8260 leader_election.cc:290] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:06.149518  8329 raft_consensus.cc:515] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:06.162528  8427 consensus_queue.cc:1059] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 18.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.006s
I20260709 13:02:06.160048  8408 leader_election.cc:290] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 15 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:02:06.163678  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 93786.7us
I20260709 13:02:06.156441  8280 raft_consensus.cc:2804] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:02:06.157681  8298 raft_consensus.cc:2749] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 13:02:06.176038  1718 leader_election.cc:400] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 election: Vote denied by peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261) with higher term. Message: Invalid argument: T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 16. Current term is 17.
I20260709 13:02:06.203079  1718 leader_election.cc:403] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 election: Cancelling election due to peer responding with higher term
W20260709 13:02:06.203655  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 45359.4us
I20260709 13:02:06.204455  1718 leader_election.cc:304] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:06.205765  1718 leader_election.cc:304] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:02:06.206689  8425 raft_consensus.cc:2804] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 13:02:06.172858  1679 rpcz_store.cc:276] 0709 13:02:03.407964 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:03.408115 (+   151us) service_pool.cc:224] Handling call
0709 13:02:03.427420 (+ 19305us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:03.611532 (+184112us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 13:02:03.611583 (+    51us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:03.671121 (+ 59538us) log.cc:844] Serialized 26 byte log entry
0709 13:02:04.117333 (+446212us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:02:04.117380 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:06.065563 (+1948183us) spinlock_profiling.cc:243] Waited 172 ms on lock 0x7b54000d7d54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:02:06.065637 (+    74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:06.065705 (+    68us) raft_consensus.cc:1685] finished
0709 13:02:06.065724 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:06.066056 (+   332us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":680,"raft.run_cpu_time_us":226,"raft.run_wall_time_us":31467,"spinlock_wait_cycles":396121856,"thread_start_us":402048,"threads_started":1,"wal-append.queue_time_us":537700,"wal-append.run_cpu_time_us":3073,"wal-append.run_wall_time_us":1102228}
W20260709 13:02:06.178265  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 65384.8us
I20260709 13:02:06.171849  1599 rpcz_store.cc:276] 0709 13:02:04.880282 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:06.131912 (+1251630us) service_pool.cc:224] Handling call
0709 13:02:06.132424 (+   512us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:06.152006 (+ 19582us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:06.152283 (+   277us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:06.179446  1605 raft_consensus.cc:2468] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 14.
I20260709 13:02:06.210379  1718 leader_election.cc:304] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:06.183795  8433 raft_consensus.cc:493] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:06.218523  8379 raft_consensus.cc:2749] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 13:02:06.218714  8433 raft_consensus.cc:515] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:06.221024  1718 leader_election.cc:304] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:02:06.184816  8307 consensus_queue.cc:1048] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 13:02:06.185611  8354 raft_consensus.cc:697] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 19 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:02:06.133059  8414 raft_consensus.cc:515] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:06.225493  8328 consensus_queue.cc:1048] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 13:02:06.227762  8414 leader_election.cc:290] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:06.171561  8269 raft_consensus.cc:515] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:06.223659  8372 raft_consensus.cc:2749] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Leader election lost for term 16. Reason: Vote denied by peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261) with higher term. Message: Invalid argument: T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 16. Current term is 17.
I20260709 13:02:06.207870  8320 raft_consensus.cc:493] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:06.186424  1601 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 2844) took 1730 ms. Trace:
I20260709 13:02:06.231042  1601 rpcz_store.cc:276] 0709 13:02:04.455644 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:06.122966 (+1667322us) service_pool.cc:224] Handling call
0709 13:02:06.134261 (+ 11295us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:06.186121 (+ 51860us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:06.186357 (+   236us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:06.233029  1602 raft_consensus.cc:1275] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 13 index: 12. Preceding OpId from leader: term: 15 index: 14. (index mismatch)
I20260709 13:02:06.201086  8145 raft_consensus.cc:515] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:06.182822  8334 raft_consensus.cc:2804] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:02:06.212478  1569 leader_election.cc:304] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:02:06.224921  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:06.230566  8269 leader_election.cc:290] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:06.223668  1718 leader_election.cc:304] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:06.207810  8280 raft_consensus.cc:493] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:06.256613  8280 raft_consensus.cc:3060] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:02:06.250874  1599 raft_consensus.cc:1275] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 13 index: 11. (index mismatch)
I20260709 13:02:06.260721  1601 raft_consensus.cc:1275] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
I20260709 13:02:06.222209  8433 leader_election.cc:290] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:06.223348  8402 raft_consensus.cc:2749] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:02:06.222942  8344 raft_consensus.cc:2749] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 13:02:06.263422  1601 raft_consensus.cc:1240] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 18. Current term is 19. Ops: [10.8-18.12]
I20260709 13:02:06.237850  8379 raft_consensus.cc:2749] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
W20260709 13:02:06.264205  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 79953.9us
I20260709 13:02:06.242981  8334 raft_consensus.cc:493] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:06.268319  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:06.270705  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:06.270836  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:06.251099  8414 raft_consensus.cc:2804] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader election won for term 15
W20260709 13:02:06.271987  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 48422.9us
I20260709 13:02:06.270278  1569 leader_election.cc:304] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:02:06.273289  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:06.277686  1598 raft_consensus.cc:1275] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 16 index: 11. (index mismatch)
I20260709 13:02:06.277963  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:06.279665  8414 raft_consensus.cc:697] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:02:06.280444  8145 leader_election.cc:290] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 15 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:06.239924  8320 raft_consensus.cc:515] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:06.281375  1601 raft_consensus.cc:1240] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 13. Current term is 14. Ops: []
I20260709 13:02:06.283223  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:06.283382  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:06.284426  1718 leader_election.cc:304] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:02:06.267489  8425 raft_consensus.cc:697] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [term 16 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:02:06.294792  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:06.276033  8334 raft_consensus.cc:3060] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:02:06.280750  8414 consensus_queue.cc:237] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.10, Last appended by leader: 10, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:06.299835  8425 consensus_queue.cc:237] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.12, Last appended by leader: 12, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:06.294137  8402 raft_consensus.cc:2749] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 13:02:06.275866  8371 raft_consensus.cc:2804] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 13:02:06.306999  1673 raft_consensus.cc:2468] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 16.
I20260709 13:02:06.309228  1718 leader_election.cc:304] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:02:06.310544  8310 raft_consensus.cc:493] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:06.311965  8310 raft_consensus.cc:515] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:06.313529  8379 raft_consensus.cc:2749] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 13:02:06.290603  8400 raft_consensus.cc:2804] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:02:06.291090  8145 raft_consensus.cc:493] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:06.328694  8320 leader_election.cc:290] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:06.349761  8330 raft_consensus.cc:515] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:06.353996  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:06.354702  1678 raft_consensus.cc:3055] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 LEADER]: Stepping down as leader of term 15
I20260709 13:02:06.355136  1678 raft_consensus.cc:740] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:02:06.330582  8329 leader_election.cc:290] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:06.354892  2330 raft_consensus.cc:2468] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 15.
I20260709 13:02:06.355947  1678 consensus_queue.cc:260] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 15.7, Last appended by leader: 7, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:06.365031  1678 raft_consensus.cc:3060] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:02:06.332638  8371 raft_consensus.cc:697] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 19 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:02:06.332114  8400 raft_consensus.cc:493] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:06.380630  1678 raft_consensus.cc:1275] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 15 index: 7. Preceding OpId from leader: term: 16 index: 7. (term mismatch)
I20260709 13:02:06.381099  1678 pending_rounds.cc:85] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d: Aborting all ops after (but not including) 6
I20260709 13:02:06.381341  1678 pending_rounds.cc:107] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d: Aborting uncommitted NO_OP operation due to leader change: 15.7
I20260709 13:02:06.381579  1678 raft_consensus.cc:2889] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:02:06.376400  1718 leader_election.cc:304] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:06.351359  8145 raft_consensus.cc:515] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:06.383824  8329 raft_consensus.cc:2804] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:02:06.384264  8329 raft_consensus.cc:493] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:06.384694  8329 raft_consensus.cc:3060] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:02:06.384455  8372 consensus_queue.cc:1048] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 13:02:06.385893  8145 leader_election.cc:290] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:06.390683  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:06.391993  1678 raft_consensus.cc:2468] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 16.
I20260709 13:02:06.307457  8354 consensus_queue.cc:237] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 17.14, Last appended by leader: 14, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:06.424288  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:06.766862  1679 raft_consensus.cc:3060] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Advancing to term 21
W20260709 13:02:06.767570  2298 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:06.769531  2298 leader_election.cc:304] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:02:06.563782  1674 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52766 (request call id 3075) took 2095 ms. Trace:
I20260709 13:02:06.770516  8403 raft_consensus.cc:2804] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 13:02:06.771394  8403 raft_consensus.cc:697] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 23 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:02:06.425247  8371 consensus_queue.cc:237] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 18.9, Last appended by leader: 9, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:06.772445  8403 consensus_queue.cc:237] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.14, Last appended by leader: 14, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:06.609004  2336 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39738 (request call id 1178) took 2268 ms (client timeout 1646 ms). Trace:
W20260709 13:02:06.559521  1754 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55840 (request call id 2988) took 1985 ms (client timeout 1761 ms). Trace:
I20260709 13:02:06.562884  1604 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 1663) took 1202 ms. Trace:
W20260709 13:02:06.728981  1644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:06.505590  1646 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:06.552768  1749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55848 (request call id 3093) took 1974 ms. Trace:
W20260709 13:02:06.779693  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 749600us
W20260709 13:02:06.702845  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 210017us
W20260709 13:02:06.476873  1643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:06.785744  1678 raft_consensus.cc:3055] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 LEADER]: Stepping down as leader of term 13
I20260709 13:02:06.786268  1678 raft_consensus.cc:740] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:02:06.787170  1678 consensus_queue.cc:260] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 13.9, Last appended by leader: 9, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:02:06.523870  2299 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:06.371426  8400 raft_consensus.cc:3060] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:02:06.610404  2329 raft_consensus.cc:1240] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 18. Current term is 19. Ops: [18.14-18.14]
W20260709 13:02:06.560793  2337 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39738 (request call id 1175) took 2222 ms (client timeout 2124 ms). Trace:
W20260709 13:02:06.774895  2336 rpcz_store.cc:269] 0709 13:02:04.341055 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:04.341239 (+   184us) service_pool.cc:224] Handling call
0709 13:02:06.608721 (+2267482us) spinlock_profiling.cc:243] Waited 4.68 ms on lock 0x7b54007d2654. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:02:06.608905 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":10775296}
W20260709 13:02:06.775954  1754 rpcz_store.cc:269] 0709 13:02:04.573734 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:04.574221 (+   487us) service_pool.cc:224] Handling call
0709 13:02:06.559270 (+1985049us) spinlock_profiling.cc:243] Waited 1.22 s on lock 0x7b5400116cd4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:02:06.559431 (+   161us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2805409664}
I20260709 13:02:06.780156  1749 rpcz_store.cc:276] 0709 13:02:04.578569 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:05.471577 (+893008us) service_pool.cc:224] Handling call
0709 13:02:05.472011 (+   434us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:05.782067 (+310056us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:06.552451 (+770384us) spinlock_profiling.cc:243] Waited 222 ms on lock 0x7b540019bed4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:02:06.552683 (+   232us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":510570752}
I20260709 13:02:06.770967  1674 rpcz_store.cc:276] 0709 13:02:04.468802 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:04.468938 (+   136us) service_pool.cc:224] Handling call
0709 13:02:04.473404 (+  4466us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:04.584213 (+110809us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:04.584252 (+    39us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:04.584291 (+    39us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:02:04.584322 (+    31us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:06.563453 (+1979131us) spinlock_profiling.cc:243] Waited 111 ms on lock 0x7b54001e96d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:02:06.563527 (+    74us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:06.563698 (+   171us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":254402432}
W20260709 13:02:06.792682  2337 rpcz_store.cc:269] 0709 13:02:04.338700 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:04.339049 (+   349us) service_pool.cc:224] Handling call
0709 13:02:06.560542 (+2221493us) spinlock_profiling.cc:243] Waited 1.11 s on lock 0x7b54005bca54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:02:06.560699 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2543295360}
W20260709 13:02:06.801350  1571 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.528s	user 0.111s	sys 0.177s
W20260709 13:02:06.801694  1571 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.529s	user 0.111s	sys 0.177s
W20260709 13:02:06.802192  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 529870us
I20260709 13:02:06.802780  1678 raft_consensus.cc:3060] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Advancing to term 14
W20260709 13:02:06.811939  1570 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:06.784147  8353 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.910s	user 0.001s	sys 0.021s
I20260709 13:02:06.777058  1604 rpcz_store.cc:276] 0709 13:02:05.360523 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:06.135681 (+775158us) service_pool.cc:224] Handling call
0709 13:02:06.144260 (+  8579us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:06.147826 (+  3566us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:06.147862 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:06.148408 (+   546us) log.cc:844] Serialized 26 byte log entry
0709 13:02:06.264152 (+115744us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:02:06.264195 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:06.562385 (+298190us) spinlock_profiling.cc:243] Waited 3.17 ms on lock 0x7b54002792d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:02:06.562457 (+    72us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:06.562511 (+    54us) raft_consensus.cc:1685] finished
0709 13:02:06.562527 (+    16us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:06.562798 (+   271us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":7969,"raft.run_cpu_time_us":116,"raft.run_wall_time_us":1309,"spinlock_wait_cycles":7279872,"thread_start_us":113857,"threads_started":1,"wal-append.queue_time_us":108154}
I20260709 13:02:06.797057  1718 leader_election.cc:304] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
W20260709 13:02:06.552203  8443 log.cc:927] Time spent T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.091s	user 0.001s	sys 0.000s
I20260709 13:02:06.797336  8303 raft_consensus.cc:493] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:06.727622  8310 leader_election.cc:290] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:06.775727  8145 raft_consensus.cc:493] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:02:06.824297  1751 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55848 (request call id 3095) took 2244 ms. Trace:
I20260709 13:02:06.404518  8330 leader_election.cc:290] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 21 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:06.784817  8402 raft_consensus.cc:493] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:06.433928  8284 raft_consensus.cc:493] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:06.561782  8338 raft_consensus.cc:3055] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 13:02:06.824877  1751 rpcz_store.cc:276] 0709 13:02:04.580261 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:05.527122 (+946861us) service_pool.cc:224] Handling call
0709 13:02:05.541249 (+ 14127us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:05.780833 (+239584us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:06.824001 (+1043168us) spinlock_profiling.cc:243] Waited 157 ms on lock 0x7b54001d47d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:02:06.824226 (+   225us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":360357248}
I20260709 13:02:06.770097  8334 raft_consensus.cc:515] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:06.827525  1678 raft_consensus.cc:1275] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 13 index: 9. Preceding OpId from leader: term: 14 index: 10. (index mismatch)
W20260709 13:02:06.815742  8353 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 2.044s	user 0.001s	sys 0.022s
I20260709 13:02:06.798043  8307 raft_consensus.cc:493] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:06.818915  2298 leader_election.cc:304] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 13:02:06.827539  1570 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:06.840199  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:06.841195  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:06.465730  8280 raft_consensus.cc:515] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:06.840751  8163 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.510s	user 0.024s	sys 0.030s
W20260709 13:02:06.874468  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 55681.5us
I20260709 13:02:06.874105  1598 raft_consensus.cc:1275] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 13 index: 7. Preceding OpId from leader: term: 16 index: 8. (index mismatch)
I20260709 13:02:06.873355  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:06.877691  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:06.887044  1604 raft_consensus.cc:3060] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:02:06.887953  1601 raft_consensus.cc:2468] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 16.
I20260709 13:02:06.824307  8300 consensus_queue.cc:1048] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.051s
I20260709 13:02:06.827432  8368 consensus_queue.cc:1059] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 15.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.049s
I20260709 13:02:06.834905  8338 raft_consensus.cc:740] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:02:06.887482  1603 raft_consensus.cc:3060] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:02:06.904498  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:06.884052  8341 raft_consensus.cc:2749] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:02:06.831010  8303 raft_consensus.cc:515] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:06.907927  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:06.833261  8145 raft_consensus.cc:515] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:06.845247  8320 consensus_queue.cc:1048] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.060s
I20260709 13:02:06.839548  8310 consensus_queue.cc:1048] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.064s
I20260709 13:02:06.847656  8372 raft_consensus.cc:493] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:06.910595  1717 leader_election.cc:304] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1
W20260709 13:02:06.912885  1600 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40166 (request call id 2825) took 2681 ms (client timeout 1530 ms). Trace:
I20260709 13:02:06.889333  8344 raft_consensus.cc:2804] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Leader pre-election won for term 17
W20260709 13:02:06.879837  8163 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.549s	user 0.024s	sys 0.030s
I20260709 13:02:06.836262  8402 raft_consensus.cc:515] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:06.846179  8330 raft_consensus.cc:493] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:06.917505  1600 rpcz_store.cc:269] 0709 13:02:04.231941 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:04.232171 (+   230us) service_pool.cc:224] Handling call
0709 13:02:06.907949 (+2675778us) spinlock_profiling.cc:243] Waited 207 ms on lock 0x7b54002a1f54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:02:06.912801 (+  4852us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":476989696}
I20260709 13:02:06.924536  1603 raft_consensus.cc:2468] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 21.
I20260709 13:02:06.925876  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:06.926564  1603 raft_consensus.cc:2468] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 13.
I20260709 13:02:06.931465  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:06.932209  1600 raft_consensus.cc:2468] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 16.
I20260709 13:02:06.933399  1603 raft_consensus.cc:1240] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 19. Current term is 20. Ops: [19.10-19.10]
I20260709 13:02:06.843119  8284 raft_consensus.cc:515] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:06.877792  8307 raft_consensus.cc:515] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:06.941303  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:06.918272  8344 raft_consensus.cc:493] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:06.967061  1679 raft_consensus.cc:2468] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 21.
I20260709 13:02:06.973765  1597 raft_consensus.cc:2468] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 15.
W20260709 13:02:06.951318  8447 log.cc:927] Time spent T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.051s	user 0.001s	sys 0.000s
I20260709 13:02:06.951850  8344 raft_consensus.cc:3060] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Advancing to term 17
W20260709 13:02:06.990772  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 194059us
I20260709 13:02:06.940148  8163 consensus_queue.cc:1048] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.046s
I20260709 13:02:06.937587  8330 raft_consensus.cc:515] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:06.996002  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 85539.2us
I20260709 13:02:06.984812  8453 raft_consensus.cc:493] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:06.934352  8372 raft_consensus.cc:515] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:07.005187  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 90775.6us
W20260709 13:02:07.007372  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 171665us
I20260709 13:02:06.972944  8300 consensus_queue.cc:1059] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true }, Status: INVALID_TERM, Last received: 8.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.031s
I20260709 13:02:07.012352  2298 leader_election.cc:304] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:07.015239  1643 leader_election.cc:304] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:02:06.943630  8338 consensus_queue.cc:260] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:07.021708  1717 leader_election.cc:304] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:02:07.001807  8310 raft_consensus.cc:493] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:07.027338  8334 leader_election.cc:290] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 20 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:07.053002  8456 raft_consensus.cc:493] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
W20260709 13:02:07.053403  8445 log.cc:927] Time spent T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.119s	user 0.000s	sys 0.001s
I20260709 13:02:07.031647  8425 raft_consensus.cc:2749] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:02:07.039089  8454 raft_consensus.cc:493] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:07.010816  8453 raft_consensus.cc:515] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:02:07.052146  8448 log.cc:927] Time spent T 00000000000000000000000000000000 P 919f6392d4f44fd68987e2c5e7ec37ca: Append to log took a long time: real 0.140s	user 0.001s	sys 0.001s
I20260709 13:02:07.047919  8338 raft_consensus.cc:3060] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:02:07.048023  8280 leader_election.cc:290] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 19 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:07.048967  8460 raft_consensus.cc:493] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:07.047853  8461 raft_consensus.cc:2804] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:02:07.035257  8310 raft_consensus.cc:515] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:07.062047  8457 raft_consensus.cc:493] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:07.075764  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 60624.3us
I20260709 13:02:07.055655  8455 raft_consensus.cc:493] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:02:07.082192  1604 raft_consensus.cc:2468] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 15.
W20260709 13:02:07.071944  8451 log.cc:927] Time spent T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.072s	user 0.001s	sys 0.000s
W20260709 13:02:07.089484  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 67999.3us
W20260709 13:02:07.090500  8329 consensus_meta.cc:287] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.705s	user 0.014s	sys 0.003s
I20260709 13:02:07.071354  8284 leader_election.cc:290] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:07.082547  8466 raft_consensus.cc:493] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:07.100895  8466 raft_consensus.cc:515] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:07.058735  8456 raft_consensus.cc:515] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:07.103421  8466 leader_election.cc:290] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:02:07.095651  8458 log.cc:927] Time spent T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.060s	user 0.001s	sys 0.000s
I20260709 13:02:07.104233  8456 leader_election.cc:290] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:07.088322  8465 raft_consensus.cc:493] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:02:07.105485  8465 raft_consensus.cc:515] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:07.083330  8461 raft_consensus.cc:493] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:07.092404  8455 raft_consensus.cc:515] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:07.089883  1643 leader_election.cc:304] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:02:07.088893  8402 leader_election.cc:290] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:02:07.092888  8449 log.cc:927] Time spent T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.153s	user 0.001s	sys 0.000s
I20260709 13:02:07.095486  8303 leader_election.cc:290] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:07.127862  1751 raft_consensus.cc:1275] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 15 index: 11. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
I20260709 13:02:07.071412  8454 raft_consensus.cc:515] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:07.094419  8307 leader_election.cc:290] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:02:07.136787  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 47026.3us
I20260709 13:02:07.164091  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:07.164923  1748 raft_consensus.cc:3060] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:02:07.166828  1745 raft_consensus.cc:1240] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 14 LEADER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 13. Current term is 14. Ops: []
I20260709 13:02:07.167649  1604 raft_consensus.cc:1240] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 14. Current term is 15. Ops: [14.9-14.9]
I20260709 13:02:07.168301  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:07.125317  8466 raft_consensus.cc:493] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:07.172070  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:07.178925  1751 raft_consensus.cc:2468] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 18.
I20260709 13:02:07.087062  8460 raft_consensus.cc:515] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:07.132119  8452 log.cc:927] Time spent T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.150s	user 0.001s	sys 0.000s
I20260709 13:02:07.174896  8372 leader_election.cc:290] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:07.175227  1746 raft_consensus.cc:1240] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 18. Current term is 19. Ops: []
W20260709 13:02:07.176110  8462 log.cc:927] Time spent T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.085s	user 0.000s	sys 0.000s
I20260709 13:02:07.112713  8464 raft_consensus.cc:2804] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:02:07.171433  1754 raft_consensus.cc:2393] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 23: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:02:07.179850  1753 raft_consensus.cc:1275] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 11 index: 8. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
I20260709 13:02:07.176246  8453 leader_election.cc:290] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:07.091046  8457 raft_consensus.cc:515] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:07.180852  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:07.107179  8329 raft_consensus.cc:515] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:07.131551  8456 raft_consensus.cc:493] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:07.132392  8469 raft_consensus.cc:493] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:07.134735  2300 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:07.133183  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 19 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:07.133914  1599 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 2855) took 1178 ms. Trace:
I20260709 13:02:07.123158  8461 raft_consensus.cc:3060] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:02:07.186883  1745 raft_consensus.cc:1240] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 14. Current term is 15. Ops: []
I20260709 13:02:07.163075  8303 consensus_queue.cc:1059] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.026s
W20260709 13:02:07.187366  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 94006.5us
I20260709 13:02:07.199863  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:07.183053  8468 raft_consensus.cc:493] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:07.188329  2333 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49082 (request call id 1194) took 2763 ms. Trace:
I20260709 13:02:07.199118  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:07.137552  8402 raft_consensus.cc:493] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:02:07.168360  8330 leader_election.cc:290] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:07.274516  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:07.135151  8477 raft_consensus.cc:2804] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:02:07.275718  8477 raft_consensus.cc:493] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:07.276206  8477 raft_consensus.cc:3060] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:02:07.194051  1599 rpcz_store.cc:276] 0709 13:02:05.955499 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:06.262112 (+306613us) service_pool.cc:224] Handling call
0709 13:02:06.262480 (+   368us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:02:06.271043 (+  8563us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:06.271068 (+    25us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:02:06.275077 (+  4009us) log.cc:844] Serialized 52 byte log entry
0709 13:02:06.513171 (+238094us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:02:06.526687 (+ 13516us) log.cc:844] Serialized 14 byte log entry
0709 13:02:06.816496 (+289809us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:06.816615 (+   119us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:07.133601 (+316986us) raft_consensus.cc:1685] finished
0709 13:02:07.133633 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:07.133827 (+   194us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":307861,"prepare.queue_time_us":13198,"prepare.run_cpu_time_us":74,"prepare.run_wall_time_us":75,"raft.queue_time_us":298045,"raft.run_cpu_time_us":312,"raft.run_wall_time_us":10508,"thread_start_us":212171,"threads_started":1,"wal-append.queue_time_us":616425}
I20260709 13:02:07.196302  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:07.277062  1754 raft_consensus.cc:2468] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 18.
I20260709 13:02:07.278054  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:07.278277  8237 raft_consensus.cc:3055] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 LEADER]: Stepping down as leader of term 18
I20260709 13:02:07.278779  8237 raft_consensus.cc:740] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:02:07.249398  8329 leader_election.cc:290] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:07.279937  8237 consensus_queue.cc:260] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:07.281155  8237 raft_consensus.cc:3060] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:02:07.266038  8400 raft_consensus.cc:515] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:07.283569  8145 leader_election.cc:290] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:07.284844  8400 leader_election.cc:290] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 19 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:07.286042  8190 raft_consensus.cc:3060] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:02:07.287271  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:07.287909  1596 raft_consensus.cc:3060] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:02:07.288800  8068 leader_election.cc:290] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 19 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:07.291071  8068 raft_consensus.cc:493] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:02:07.226643  8478 raft_consensus.cc:2804] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:02:07.197929  8456 raft_consensus.cc:515] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:07.198073  8469 raft_consensus.cc:515] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:07.199432  8467 log.cc:927] Time spent T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.071s	user 0.001s	sys 0.000s
I20260709 13:02:07.181988  8466 raft_consensus.cc:515] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:07.302780  8068 raft_consensus.cc:515] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:07.313513  8466 leader_election.cc:290] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:07.192629  8372 consensus_queue.cc:1048] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.042s
I20260709 13:02:07.313879  8068 leader_election.cc:290] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:07.315239  1754 raft_consensus.cc:3060] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:02:07.234023  8455 leader_election.cc:290] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:07.203218  2333 rpcz_store.cc:276] 0709 13:02:04.468628 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:04.472291 (+  3663us) service_pool.cc:224] Handling call
0709 13:02:04.472684 (+   393us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:02:04.481639 (+  8955us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:04.481677 (+    38us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:02:04.711445 (+229768us) spinlock_profiling.cc:243] Waited 8.01 ms on lock 0x7b18000dff00. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f4263075522 00007f42630777c1 00007f4263077779 00007f4263077709 00007f42630776d1 00007f42630769cd 000000000038f354 000000000038f199 00007f426305883a
0709 13:02:06.566710 (+1855265us) spinlock_profiling.cc:243] Waited 3.44 ms on lock 0x7b2c00021c88. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4264a5ba11 00007f4264a6b372 00007f426619da01 00007f426616243a 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2
0709 13:02:06.567610 (+   900us) log.cc:844] Serialized 52 byte log entry
0709 13:02:06.836202 (+268592us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:02:06.836245 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:06.836334 (+    89us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:07.181299 (+344965us) raft_consensus.cc:1685] finished
0709 13:02:07.181324 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:07.181479 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8439,"raft.queue_time_us":8778,"raft.run_cpu_time_us":131,"raft.run_wall_time_us":5515,"spinlock_wait_cycles":26325504,"thread_start_us":268141,"threads_started":1,"wal-append.queue_time_us":375028}
I20260709 13:02:07.317144  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:07.318542  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:07.249419  8454 leader_election.cc:290] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:02:07.253682  1645 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:07.248530  8471 raft_consensus.cc:493] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:02:07.320124  8471 raft_consensus.cc:515] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:07.321000  8456 leader_election.cc:290] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:07.322605  8471 leader_election.cc:290] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:07.251746  8457 leader_election.cc:290] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:07.323071  1675 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52766 (request call id 3090) took 1574 ms. Trace:
I20260709 13:02:07.323473  1675 rpcz_store.cc:276] 0709 13:02:05.748684 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:05.783467 (+ 34783us) service_pool.cc:224] Handling call
0709 13:02:05.783816 (+   349us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:05.860624 (+ 76808us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:05.860649 (+    25us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:05.860679 (+    30us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:02:05.860700 (+    21us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:07.322818 (+1462118us) spinlock_profiling.cc:243] Waited 76.6 ms on lock 0x7b5400163854. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:02:07.322872 (+    54us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:07.323001 (+   129us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":176235776}
I20260709 13:02:07.250613  8310 leader_election.cc:290] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:07.323081  1678 raft_consensus.cc:1275] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
W20260709 13:02:07.329481  2332 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39738 (request call id 1180) took 1745 ms (client timeout 1633 ms). Trace:
I20260709 13:02:07.296808  8300 raft_consensus.cc:3055] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 13 LEADER]: Stepping down as leader of term 13
I20260709 13:02:07.330091  1674 raft_consensus.cc:1240] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 18. Current term is 19. Ops: []
I20260709 13:02:07.330350  8300 raft_consensus.cc:740] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:02:07.331558  8300 consensus_queue.cc:260] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.9, Last appended by leader: 9, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:02:07.255060  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 53719.2us
I20260709 13:02:07.337513  1717 leader_election.cc:304] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:07.256134  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 19 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:07.256496  8473 consensus_queue.cc:1048] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 13:02:07.257175  8404 raft_consensus.cc:3055] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 LEADER]: Stepping down as leader of term 19
I20260709 13:02:07.258721  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 18 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:07.259147  8460 leader_election.cc:290] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:07.261554  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:07.259418  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:07.252830  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:07.250226  8465 leader_election.cc:290] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:07.264900  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:07.257206  1753 raft_consensus.cc:1240] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 16. Current term is 17. Ops: []
W20260709 13:02:07.319139  8381 log.cc:927] Time spent T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 2.325s	user 0.017s	sys 0.000s
I20260709 13:02:07.361605  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:07.254307  8468 raft_consensus.cc:515] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:07.200356  8464 raft_consensus.cc:697] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 14 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:02:07.107832  8463 consensus_queue.cc:1059] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 19.12, Next index: 13, Last known committed idx: 10, Time since last communication: 0.086s
I20260709 13:02:07.200618  1745 raft_consensus.cc:2468] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 14.
I20260709 13:02:07.367385  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:07.187687  1748 raft_consensus.cc:2468] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 19.
I20260709 13:02:07.367959  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 16 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:07.369179  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 18 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:07.369773  1748 raft_consensus.cc:2468] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 17.
I20260709 13:02:07.370625  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
W20260709 13:02:07.296036  1643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:07.299530  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:07.298103  8478 raft_consensus.cc:493] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:02:07.305734  2301 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:07.311354  8469 leader_election.cc:290] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:07.254967  1746 raft_consensus.cc:2468] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 18.
W20260709 13:02:07.329931  2332 rpcz_store.cc:269] 0709 13:02:05.583615 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:05.583757 (+   142us) service_pool.cc:224] Handling call
0709 13:02:07.329244 (+1745487us) spinlock_profiling.cc:243] Waited 510 ms on lock 0x7b54004e02d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:02:07.329399 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1172839680}
I20260709 13:02:07.343163  1754 raft_consensus.cc:2468] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 20.
I20260709 13:02:07.386248  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:07.353483  1596 raft_consensus.cc:2468] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 20.
I20260709 13:02:07.388065  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:07.368057  1745 raft_consensus.cc:2376] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 19. Re-sending same reply.
I20260709 13:02:07.354346  1675 raft_consensus.cc:1275] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
I20260709 13:02:07.359407  2337 raft_consensus.cc:2468] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 13.
I20260709 13:02:07.374974  1599 raft_consensus.cc:2393] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 19: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:02:07.356256  8379 raft_consensus.cc:2804] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 13:02:07.379601  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:07.262329  8344 raft_consensus.cc:515] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:07.355574  8404 raft_consensus.cc:740] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:02:07.401394  1679 raft_consensus.cc:1275] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 13 index: 7. Preceding OpId from leader: term: 16 index: 8. (index mismatch)
I20260709 13:02:07.297593  8402 raft_consensus.cc:515] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } }
I20260709 13:02:07.392656  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:07.388036  2298 leader_election.cc:304] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
W20260709 13:02:07.413776  2298 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:07.382941  2301 leader_election.cc:304] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:02:07.397701  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:07.398681  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:07.384732  8311 raft_consensus.cc:493] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:02:07.404706  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:07.405803  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:07.403609  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:07.406903  8344 leader_election.cc:290] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 17 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:07.407505  1749 raft_consensus.cc:2368] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for earlier term 16. Current term is 17.
I20260709 13:02:07.394551  8316 raft_consensus.cc:493] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:02:07.408411  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:07.414386  8379 raft_consensus.cc:697] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 21 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:02:07.436179  8298 consensus_queue.cc:1059] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.022s
I20260709 13:02:07.415126  1571 leader_election.cc:304] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:02:07.458979  8298 consensus_queue.cc:1243] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer af520b67945d443b901bbeca37cc5701 log is divergent from this leader: its last log entry 17.12 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 10
I20260709 13:02:07.440837  8460 raft_consensus.cc:493] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:07.459095  8379 consensus_queue.cc:237] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.12, Last appended by leader: 12, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:07.459976  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:07.460083  8460 raft_consensus.cc:515] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:07.461401  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:07.436540  2335 raft_consensus.cc:2468] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 20.
I20260709 13:02:07.464341  8262 raft_consensus.cc:2749] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 13:02:07.464497  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:07.465416  1749 raft_consensus.cc:2468] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 14.
W20260709 13:02:07.465725  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 82945.4us
W20260709 13:02:07.454816  1673 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52778 (request call id 2592) took 3733 ms (client timeout 2129 ms). Trace:
W20260709 13:02:07.467473  1673 rpcz_store.cc:269] 0709 13:02:03.721465 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:03.737167 (+ 15702us) service_pool.cc:224] Handling call
0709 13:02:07.454538 (+3717371us) spinlock_profiling.cc:243] Waited 2.48 s on lock 0x7b5400091354. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:02:07.454694 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5709136768}
I20260709 13:02:07.467752  8262 consensus_queue.cc:1048] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 13:02:07.468904  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:07.427734  8338 raft_consensus.cc:2804] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 13:02:07.453948  1672 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52766 (request call id 3087) took 1790 ms. Trace:
I20260709 13:02:07.471510  1672 rpcz_store.cc:276] 0709 13:02:05.663500 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:05.768645 (+105145us) service_pool.cc:224] Handling call
0709 13:02:05.769020 (+   375us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:06.026726 (+257706us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:06.026761 (+    35us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:06.026806 (+    45us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:02:06.026839 (+    33us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:07.453574 (+1426735us) spinlock_profiling.cc:243] Waited 258 ms on lock 0x7b5400092254. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:02:07.453644 (+    70us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:07.453851 (+   207us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":592393600}
I20260709 13:02:07.473230  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:07.473366  8460 leader_election.cc:290] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:07.474370  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:07.475508  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:07.476145  1672 raft_consensus.cc:2468] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 15.
I20260709 13:02:07.462116  1597 raft_consensus.cc:2468] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 15.
I20260709 13:02:07.434654  1751 raft_consensus.cc:2468] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 16.
I20260709 13:02:07.396859  8068 consensus_queue.cc:1048] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.012s
I20260709 13:02:07.363739  8471 consensus_queue.cc:1048] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.050s
W20260709 13:02:07.488488  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 73501.7us
I20260709 13:02:07.388002  8341 raft_consensus.cc:493] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:07.470184  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:07.362524  8461 raft_consensus.cc:515] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:07.491559  1673 raft_consensus.cc:3060] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:02:07.479967  2330 raft_consensus.cc:1240] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 19. Current term is 20. Ops: [19.15-19.15]
I20260709 13:02:07.366304  8457 raft_consensus.cc:493] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:07.461647  1674 raft_consensus.cc:2393] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 19: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:02:07.436455  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:07.428092  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:07.496464  1676 raft_consensus.cc:2393] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 20: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:02:07.483866  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:07.501364  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:07.505086  1746 raft_consensus.cc:2393] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 23: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:02:07.507129  1644 leader_election.cc:304] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:07.509374  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:07.510396  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:07.511381  1596 raft_consensus.cc:3060] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:02:07.521915  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:07.485548  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:07.525039  1749 raft_consensus.cc:2468] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 18.
W20260709 13:02:07.440829  2299 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:07.455757  1602 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 1665) took 2044 ms. Trace:
I20260709 13:02:07.526328  1602 rpcz_store.cc:276] 0709 13:02:05.413196 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:06.175229 (+762033us) service_pool.cc:224] Handling call
0709 13:02:06.175605 (+   376us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:06.233698 (+ 58093us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:07.455399 (+1221701us) spinlock_profiling.cc:243] Waited 56.9 ms on lock 0x7b54000acb54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:02:07.455672 (+   273us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":130861952}
I20260709 13:02:07.527312  2299 leader_election.cc:304] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:07.528139  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:07.477164  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:07.387135  8300 raft_consensus.cc:3060] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:02:07.437099  7687 raft_consensus.cc:493] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:07.532344  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:07.380031  8412 raft_consensus.cc:493] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:07.538087  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:07.476199  8338 raft_consensus.cc:697] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 20 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:02:07.539120  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 15 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
W20260709 13:02:07.533414  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 121220us
I20260709 13:02:07.477442  8459 raft_consensus.cc:493] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:07.529242  1672 raft_consensus.cc:2468] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 18.
I20260709 13:02:07.539844  2336 raft_consensus.cc:2376] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 15. Re-sending same reply.
I20260709 13:02:07.539573  1745 raft_consensus.cc:2393] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 16: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:02:07.430218  8402 leader_election.cc:290] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:07.542600  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:07.490504  8311 raft_consensus.cc:515] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:07.544169  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 16 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:07.385226  8478 raft_consensus.cc:3060] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:02:07.546232  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 17 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:07.546743  1602 raft_consensus.cc:2393] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 16: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:02:07.547413  2334 raft_consensus.cc:2410] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for term 16 because replica has last-logged OpId of term: 14 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 12.
I20260709 13:02:07.501370  8284 raft_consensus.cc:2749] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 13:02:07.556876  1752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43156 (request call id 1352) took 3569 ms. Trace:
I20260709 13:02:07.564850  1752 rpcz_store.cc:276] 0709 13:02:03.987488 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:04.283711 (+296223us) service_pool.cc:224] Handling call
0709 13:02:04.284060 (+   349us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:04.296940 (+ 12880us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:04.296982 (+    42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:04.301833 (+  4851us) log.cc:844] Serialized 26 byte log entry
0709 13:02:04.527072 (+225239us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:02:04.527121 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:04.527197 (+    76us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:07.317754 (+2790557us) spinlock_profiling.cc:243] Waited 3.62 ms on lock 0x7b4400022b4c. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f4262f9bc85 00007f4262f9c6a2 00007f4265d0e1db 00007f4265d09aaf 00007f4265cfd86e 00007f4265cfcbb6 00007f4265cfc7d8 00007f4265d06221 00007f4265d061d9 00007f4265d06169
0709 13:02:07.540992 (+223238us) raft_consensus.cc:1685] finished
0709 13:02:07.541028 (+    36us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:07.556756 (+ 15728us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6694,"raft.queue_time_us":8877,"raft.run_cpu_time_us":164,"raft.run_wall_time_us":6528,"spinlock_wait_cycles":8330624,"thread_start_us":183201,"threads_started":1,"wal-append.queue_time_us":471877}
I20260709 13:02:07.552357  1673 raft_consensus.cc:2468] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 20.
I20260709 13:02:07.561319  2333 raft_consensus.cc:1275] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 13 index: 13. Preceding OpId from leader: term: 15 index: 14. (index mismatch)
I20260709 13:02:07.563055  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:07.571201  1753 raft_consensus.cc:2468] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 19.
I20260709 13:02:07.430703  8310 consensus_queue.cc:1059] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 15.12, Next index: 13, Last known committed idx: 11, Time since last communication: 0.111s
I20260709 13:02:07.541702  1646 leader_election.cc:304] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:07.563803  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:02:07.557389  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 145962us
I20260709 13:02:07.572816  1602 raft_consensus.cc:2393] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 20: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:02:07.537281  8368 raft_consensus.cc:2749] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:02:07.577517  1569 leader_election.cc:304] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:02:07.585575  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:07.593297  1674 raft_consensus.cc:2368] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 19. Current term is 20.
I20260709 13:02:07.494061  8316 raft_consensus.cc:515] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:07.587591  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:07.606189  1598 raft_consensus.cc:1275] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 13 index: 9. Preceding OpId from leader: term: 14 index: 10. (index mismatch)
I20260709 13:02:07.589359  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:07.582762  1718 leader_election.cc:304] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:07.607003  1746 raft_consensus.cc:2468] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 18.
I20260709 13:02:07.607551  8468 leader_election.cc:290] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:07.607869  8463 raft_consensus.cc:2804] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:02:07.608059  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:07.608384  8463 raft_consensus.cc:493] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:07.608847  8463 raft_consensus.cc:3060] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:02:07.609531  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:07.610208  1746 raft_consensus.cc:2468] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 12.
I20260709 13:02:07.586791  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:07.636736  1752 raft_consensus.cc:2393] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 19: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:02:07.596560  1597 raft_consensus.cc:1240] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 15. Current term is 16. Ops: []
I20260709 13:02:07.588462  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 19 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:07.615114  1679 raft_consensus.cc:1275] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 16 index: 13. Preceding OpId from leader: term: 19 index: 15. (index mismatch)
I20260709 13:02:07.641152  1597 raft_consensus.cc:1275] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 23 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 21 index: 14. Preceding OpId from leader: term: 23 index: 15. (index mismatch)
I20260709 13:02:07.615624  8484 raft_consensus.cc:2804] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:02:07.616348  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:07.642494  2331 raft_consensus.cc:2468] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 19.
I20260709 13:02:07.643716  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:07.619323  8463 raft_consensus.cc:515] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:07.619848  1601 raft_consensus.cc:1240] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 18. Current term is 19. Ops: [10.8-18.12]
I20260709 13:02:07.506671  8457 raft_consensus.cc:515] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:07.648711  8434 raft_consensus.cc:3055] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 13:02:07.649326  8434 raft_consensus.cc:740] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:02:07.650537  8434 consensus_queue.cc:260] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:07.652484  8434 raft_consensus.cc:3060] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:02:07.624461  1717 leader_election.cc:304] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:02:07.625283  8350 raft_consensus.cc:3060] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:02:07.555531  8412 raft_consensus.cc:515] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:07.502418  8341 raft_consensus.cc:515] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:07.674093  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:07.674806  1673 raft_consensus.cc:1240] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 19. Current term is 20. Ops: []
I20260709 13:02:07.627990  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:07.675485  1751 raft_consensus.cc:2468] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 17.
I20260709 13:02:07.676410  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:07.627803  8338 consensus_queue.cc:237] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:07.549564  7687 raft_consensus.cc:515] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:07.629055  8464 consensus_queue.cc:237] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.8, Last appended by leader: 8, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:07.589998  8402 raft_consensus.cc:493] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
W20260709 13:02:07.685745  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 115436us
W20260709 13:02:07.630869  8436 log.cc:927] Time spent T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 1.260s	user 0.000s	sys 0.005s
I20260709 13:02:07.567718  8460 consensus_queue.cc:1048] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.063s
I20260709 13:02:07.631104  8358 raft_consensus.cc:2749] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:02:07.590168  8303 raft_consensus.cc:2804] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Leader pre-election won for term 19
W20260709 13:02:07.606621  8373 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 2.758s	user 0.023s	sys 0.014s
W20260709 13:02:07.635370  1644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:07.452203  8404 consensus_queue.cc:260] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 19.10, Last appended by leader: 10, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:07.656165  1717 leader_election.cc:304] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:07.650681  8457 leader_election.cc:290] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:07.563623  2330 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc"
dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
mode: GRACEFUL
new_leader_uuid: "af520b67945d443b901bbeca37cc5701"
 from {username='slave'} at 127.0.0.1:35790
W20260709 13:02:07.661764  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 60110.8us
I20260709 13:02:07.778499  1646 leader_election.cc:304] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 13:02:07.677208  1751 raft_consensus.cc:2393] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 19: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:02:07.628135  8461 leader_election.cc:290] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:07.678686  1597 raft_consensus.cc:1275] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 16 index: 7. (index mismatch)
I20260709 13:02:07.679133  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:07.673022  1596 raft_consensus.cc:2468] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 17.
I20260709 13:02:07.681393  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:07.636293  8310 raft_consensus.cc:3055] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 LEADER]: Stepping down as leader of term 15
I20260709 13:02:07.654637  8484 raft_consensus.cc:493] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:02:07.797698  8311 leader_election.cc:290] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:07.797977  8484 raft_consensus.cc:3060] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:02:07.798555  8311 consensus_queue.cc:1059] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 13:02:07.799525  8311 raft_consensus.cc:2804] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Leader election won for term 19
I20260709 13:02:07.800004  8311 raft_consensus.cc:697] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:02:07.736335  1599 raft_consensus.cc:2468] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 16.
I20260709 13:02:07.800954  8311 consensus_queue.cc:237] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:07.557284  8459 raft_consensus.cc:515] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:07.802598  2331 raft_consensus.cc:2468] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 20.
I20260709 13:02:07.745561  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:07.803565  1746 raft_consensus.cc:2393] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 23: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:02:07.804517  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 17 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:07.805317  1746 raft_consensus.cc:2393] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 17: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:02:07.608492  8472 consensus_queue.cc:1048] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.244s
W20260709 13:02:07.755838  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 108551us
I20260709 13:02:07.809934  1569 leader_election.cc:304] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:02:07.811131  8420 consensus_queue.cc:1059] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 18.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 13:02:07.811846  7687 leader_election.cc:290] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:07.812320  8386 consensus_queue.cc:1048] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 13:02:07.818287  8481 raft_consensus.cc:493] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:07.818964  8481 raft_consensus.cc:515] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:07.821491  8481 leader_election.cc:290] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:07.756731  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 20 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:07.681154  8468 consensus_queue.cc:1048] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.094s
W20260709 13:02:07.775360  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 119325us
I20260709 13:02:07.778664  7307 raft_consensus.cc:2804] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:02:07.828662  7307 raft_consensus.cc:493] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:07.829064  7307 raft_consensus.cc:3060] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Advancing to term 16
W20260709 13:02:07.780890  8373 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 3.015s	user 0.023s	sys 0.014s
I20260709 13:02:07.779884  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:07.732461  8473 raft_consensus.cc:493] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:07.663038  8471 raft_consensus.cc:2804] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 13:02:07.759274  8490 raft_consensus.cc:493] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:02:07.826130  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:07.544708  8323 consensus_queue.cc:1048] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.091s
I20260709 13:02:07.661052  8477 raft_consensus.cc:515] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:07.833689  8294 raft_consensus.cc:2749] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:02:07.836596  8316 leader_election.cc:290] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 13:02:07.838546  1645 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:07.847985  1751 raft_consensus.cc:2368] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 for earlier term 15. Current term is 16.
I20260709 13:02:07.911320  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 16 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:07.757553  8303 raft_consensus.cc:493] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:02:07.927088  8303 raft_consensus.cc:3060] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Advancing to term 19
W20260709 13:02:07.824005  8487 log.cc:927] Time spent T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.062s	user 0.000s	sys 0.001s
I20260709 13:02:07.934744  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 19 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:07.936023  1749 raft_consensus.cc:2410] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for term 19 because replica has last-logged OpId of term: 18 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 12.
I20260709 13:02:07.937165  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:07.946197  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:07.947399  2334 raft_consensus.cc:2393] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 16: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:02:07.948906  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:07.787339  8298 raft_consensus.cc:493] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:02:07.806113  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:07.905649  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:07.969885  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:07.793915  8310 raft_consensus.cc:740] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:02:07.744403  8402 raft_consensus.cc:515] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:07.970191  2333 raft_consensus.cc:3060] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:02:07.925156  1605 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 1668) took 2368 ms. Trace:
W20260709 13:02:07.896796  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 57946.2us
I20260709 13:02:07.780205  8334 consensus_queue.cc:1048] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.122s
I20260709 13:02:08.263743  1675 raft_consensus.cc:1275] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 13 index: 13. Preceding OpId from leader: term: 15 index: 14. (index mismatch)
I20260709 13:02:08.264552  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:08.276551  1718 leader_election.cc:304] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:02:08.166888  8371 raft_consensus.cc:493] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:07.990161  7307 raft_consensus.cc:515] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:08.281186  8371 raft_consensus.cc:515] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:02:08.278825  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 500658us
W20260709 13:02:08.117149  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 347628us
I20260709 13:02:08.276101  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:08.263159  8472 raft_consensus.cc:493] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:08.282083  1751 raft_consensus.cc:2393] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 16: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:02:08.276115  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:07.900665  8481 raft_consensus.cc:493] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:08.287792  7307 leader_election.cc:290] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:08.285631  1605 rpcz_store.cc:276] 0709 13:02:05.556839 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:06.214669 (+657830us) service_pool.cc:224] Handling call
0709 13:02:06.215014 (+   345us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:06.215539 (+   525us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:06.215576 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:06.216322 (+   746us) log.cc:844] Serialized 26 byte log entry
0709 13:02:06.281873 (+ 65551us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:02:06.281946 (+    73us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:06.282026 (+    80us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:07.630220 (+1348194us) spinlock_profiling.cc:243] Waited 1.93 ms on lock 0x7b4400022b4c. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f4262f9bc85 00007f4262f9c6a2 00007f4265d0e1db 00007f4265d09aaf 00007f4265cfd86e 00007f4265cfcbb6 00007f4265cfc7d8 00007f4265d06221 00007f4265d061d9 00007f4265d06169
0709 13:02:07.913249 (+283029us) raft_consensus.cc:1685] finished
0709 13:02:07.913285 (+    36us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:07.925058 (+ 11773us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":117,"raft.run_cpu_time_us":160,"raft.run_wall_time_us":160,"spinlock_wait_cycles":4441344,"thread_start_us":59340,"threads_started":1,"wal-append.queue_time_us":132781}
I20260709 13:02:08.294597  1569 leader_election.cc:400] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 19 election: Vote denied by peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261) with higher term. Message: Invalid argument: T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 19. Current term is 20.
I20260709 13:02:08.295065  1569 leader_election.cc:403] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 19 election: Cancelling election due to peer responding with higher term
W20260709 13:02:08.283963  1678 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.543s	user 0.011s	sys 0.026s
I20260709 13:02:07.974081  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:02:08.295965  1678 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.555s	user 0.011s	sys 0.026s
I20260709 13:02:08.296492  1746 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55840 (request call id 3006) took 1005 ms. Trace:
I20260709 13:02:08.297029  1746 rpcz_store.cc:276] 0709 13:02:07.291151 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:07.973819 (+682668us) service_pool.cc:224] Handling call
0709 13:02:08.296387 (+322568us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:07.970830  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:08.300156  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:08.300971  1746 raft_consensus.cc:2468] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 16.
I20260709 13:02:08.185137  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 20 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:08.183735  2328 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49082 (request call id 1195) took 3714 ms. Trace:
I20260709 13:02:07.847783  8404 raft_consensus.cc:3060] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:02:08.198971  2299 leader_election.cc:304] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:08.288595  1646 leader_election.cc:304] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 13:02:08.043303  1671 raft_consensus.cc:1240] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 13. Current term is 14. Ops: [13.11-13.11]
I20260709 13:02:08.289613  1644 leader_election.cc:304] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:02:08.305356  1753 raft_consensus.cc:2410] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 for term 20 because replica has last-logged OpId of term: 18 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 13.
I20260709 13:02:07.897202  8461 raft_consensus.cc:493] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:08.302611  2328 rpcz_store.cc:276] 0709 13:02:04.469026 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:04.469582 (+   556us) service_pool.cc:224] Handling call
0709 13:02:05.747061 (+1277479us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:05.769349 (+ 22288us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:08.183408 (+2414059us) spinlock_profiling.cc:243] Waited 1.28 s on lock 0x7b54005e3b50. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661618ad 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:02:08.183643 (+   235us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2938509568}
I20260709 13:02:08.325472  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:08.327760  7307 raft_consensus.cc:493] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:08.330653  1747 raft_consensus.cc:1240] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 14 LEADER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 13. Current term is 14. Ops: []
I20260709 13:02:08.330845  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 21 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:08.330144  7307 raft_consensus.cc:515] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:08.333824  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 19 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:08.274544  8341 leader_election.cc:290] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:08.334914  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 19 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:08.340952  1753 raft_consensus.cc:2410] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for term 19 because replica has last-logged OpId of term: 18 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 7.
I20260709 13:02:08.341187  8360 raft_consensus.cc:2749] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Leader election lost for term 19. Reason: Vote denied by peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261) with higher term. Message: Invalid argument: T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 19. Current term is 20.
I20260709 13:02:08.341421  1753 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55840 (request call id 2996) took 1071 ms. Trace:
I20260709 13:02:08.341838  1753 rpcz_store.cc:276] 0709 13:02:07.270385 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:08.334646 (+1064261us) service_pool.cc:224] Handling call
0709 13:02:08.341351 (+  6705us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:08.342399  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:08.343107  1753 raft_consensus.cc:1240] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 19. Current term is 20. Ops: [19.10-19.10]
I20260709 13:02:08.335709  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:08.040639  8298 raft_consensus.cc:515] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:02:08.297199  8300 consensus_meta.cc:287] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.752s	user 0.008s	sys 0.005s
I20260709 13:02:08.337368  1746 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55840 (request call id 3001) took 1065 ms. Trace:
I20260709 13:02:07.915553  8471 raft_consensus.cc:697] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 15 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:02:08.337991  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 19 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:08.283970  8463 leader_election.cc:290] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:08.292030  8469 raft_consensus.cc:2764] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Leader pre-election decision vote started in defunct term 18: won
I20260709 13:02:08.344630  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:02:08.336709  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 137959us
I20260709 13:02:08.107920  8468 raft_consensus.cc:3055] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:02:08.346787  8154 consensus_queue.cc:1059] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 19.15, Next index: 16, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 13:02:08.347105  1753 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55848 (request call id 3114) took 1079 ms. Trace:
I20260709 13:02:08.349993  8403 raft_consensus.cc:2749] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:02:08.176554  8412 leader_election.cc:290] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:07.905002  8358 consensus_queue.cc:1048] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.428s
I20260709 13:02:08.351245  1746 rpcz_store.cc:276] 0709 13:02:07.271863 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:08.330537 (+1058674us) service_pool.cc:224] Handling call
0709 13:02:08.337292 (+  6755us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:08.388273  1747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55848 (request call id 3099) took 1224 ms. Trace:
I20260709 13:02:08.388720  1747 rpcz_store.cc:276] 0709 13:02:07.163967 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:07.164088 (+   121us) service_pool.cc:224] Handling call
0709 13:02:07.174914 (+ 10826us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:08.335740 (+1160826us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:08.387991 (+ 52251us) spinlock_profiling.cc:243] Waited 1.15 s on lock 0x7b5400116cd4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:02:08.388194 (+   203us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2656132992}
I20260709 13:02:08.352505  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 16 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:08.353209  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 16 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:08.337131  8461 raft_consensus.cc:515] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:08.390756  1754 raft_consensus.cc:2393] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 16: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:02:07.907335  8473 raft_consensus.cc:515] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:08.391855  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:08.392506  8461 leader_election.cc:290] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:07.958503  8490 raft_consensus.cc:515] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:08.392953  1749 raft_consensus.cc:2410] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for term 19 because replica has last-logged OpId of term: 16 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 10.
I20260709 13:02:08.393051  8473 leader_election.cc:290] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:08.360002  2329 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39726 (request call id 1497) took 3888 ms. Trace:
I20260709 13:02:08.360071  7307 leader_election.cc:290] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:08.394153  1746 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55840 (request call id 2992) took 1125 ms. Trace:
I20260709 13:02:08.394735  1746 rpcz_store.cc:276] 0709 13:02:07.268803 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:08.388631 (+1119828us) service_pool.cc:224] Handling call
0709 13:02:08.393551 (+  4920us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:08.393794 (+   243us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:08.393824 (+    30us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:08.393875 (+    51us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:02:08.393912 (+    37us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:08.393975 (+    63us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:08.394086 (+   111us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:08.395522  1747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55840 (request call id 2994) took 1126 ms. Trace:
I20260709 13:02:08.396607  1746 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55840 (request call id 3003) took 1124 ms. Trace:
I20260709 13:02:08.364816  8468 raft_consensus.cc:740] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
W20260709 13:02:08.348234  8498 log.cc:927] Time spent T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.092s	user 0.000s	sys 0.001s
I20260709 13:02:08.396978  1746 rpcz_store.cc:276] 0709 13:02:07.272371 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:08.395706 (+1123335us) service_pool.cc:224] Handling call
0709 13:02:08.396003 (+   297us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:08.396210 (+   207us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:08.396235 (+    25us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:08.396275 (+    40us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:02:08.396323 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:08.396396 (+    73us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:08.396520 (+   124us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:08.392534  1754 raft_consensus.cc:2468] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 19.
I20260709 13:02:08.373622  8300 raft_consensus.cc:2804] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:02:08.413803  8298 leader_election.cc:290] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:02:08.415447  8485 log.cc:927] Time spent T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.907s	user 0.004s	sys 0.002s
I20260709 13:02:08.415795  8454 raft_consensus.cc:3055] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 13:02:08.416416  8454 raft_consensus.cc:740] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:02:08.416647  8490 leader_election.cc:290] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:08.416764  1604 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 2864) took 1247 ms. Trace:
W20260709 13:02:08.399386  1643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:08.417214  1604 rpcz_store.cc:276] 0709 13:02:07.169402 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:07.169546 (+   144us) service_pool.cc:224] Handling call
0709 13:02:07.175062 (+  5516us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:07.175496 (+   434us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:07.175536 (+    40us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:07.176154 (+   618us) log.cc:844] Serialized 26 byte log entry
0709 13:02:07.488316 (+312162us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:02:07.488391 (+    75us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:07.509786 (+ 21395us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:08.414776 (+904990us) spinlock_profiling.cc:243] Waited 4.07 ms on lock 0x7b4400022b4c. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f4262f9bc85 00007f4262f9c6a2 00007f4265d0e1db 00007f4265d09aaf 00007f4265cfd86e 00007f4265cfcbb6 00007f4265cfc7d8 00007f4265d06221 00007f4265d061d9 00007f4265d06169
0709 13:02:08.416424 (+  1648us) raft_consensus.cc:1685] finished
0709 13:02:08.416446 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:08.416689 (+   243us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1285,"raft.run_cpu_time_us":151,"raft.run_wall_time_us":1989,"spinlock_wait_cycles":9371392,"thread_start_us":311667,"threads_started":1,"wal-append.queue_time_us":331700}
I20260709 13:02:08.418409  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:08.419124  8474 raft_consensus.cc:493] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:08.419684  8474 raft_consensus.cc:515] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:08.325547  8472 raft_consensus.cc:515] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:08.376209  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:08.423424  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 17 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:02:08.377614  8478 consensus_meta.cc:287] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.787s	user 0.009s	sys 0.000s
I20260709 13:02:08.381893  2333 raft_consensus.cc:2468] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 16.
I20260709 13:02:08.382688  8477 leader_election.cc:290] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 14 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:08.384526  8371 leader_election.cc:290] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:08.387151  8459 leader_election.cc:290] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:08.387563  8402 leader_election.cc:290] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:08.280144  8310 consensus_queue.cc:260] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:08.362690  2299 leader_election.cc:304] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:08.366422  1753 rpcz_store.cc:276] 0709 13:02:07.267401 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:08.342509 (+1075108us) service_pool.cc:224] Handling call
0709 13:02:08.342845 (+   336us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:08.346798 (+  3953us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:08.347043 (+   245us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:08.338169  8481 raft_consensus.cc:515] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:08.393703  2329 rpcz_store.cc:276] 0709 13:02:04.471309 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:04.471445 (+   136us) service_pool.cc:224] Handling call
0709 13:02:06.610097 (+2138652us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:06.798066 (+187969us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:08.359773 (+1561707us) spinlock_profiling.cc:243] Waited 2.14 s on lock 0x7b54007d2650. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661618ad 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:02:08.359933 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4919936896}
I20260709 13:02:08.413867  8300 raft_consensus.cc:493] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:02:08.363839  8501 log.cc:927] Time spent T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.109s	user 0.000s	sys 0.001s
I20260709 13:02:08.415874  1754 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55840 (request call id 3013) took 1008 ms. Trace:
I20260709 13:02:08.419613  8471 consensus_queue.cc:237] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.8, Last appended by leader: 8, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:08.351738  8360 consensus_queue.cc:1048] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.022s
I20260709 13:02:08.424330  1599 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40154 (request call id 2780) took 1080 ms. Trace:
I20260709 13:02:08.482617  1599 rpcz_store.cc:276] 0709 13:02:07.344091 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:07.390194 (+ 46103us) service_pool.cc:224] Handling call
0709 13:02:08.424164 (+1033970us) spinlock_profiling.cc:243] Waited 323 ms on lock 0x7b54002a3354. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:02:08.424267 (+   103us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":743553280}
W20260709 13:02:08.424602  1643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:08.411870  8508 raft_consensus.cc:493] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
W20260709 13:02:08.361191  8484 consensus_meta.cc:287] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.563s	user 0.011s	sys 0.004s
W20260709 13:02:08.375538  8434 consensus_meta.cc:287] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.722s	user 0.014s	sys 0.000s
I20260709 13:02:08.493665  1752 raft_consensus.cc:2468] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 14.
I20260709 13:02:08.395917  1747 rpcz_store.cc:276] 0709 13:02:07.269383 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:08.389823 (+1120440us) service_pool.cc:224] Handling call
0709 13:02:08.394964 (+  5141us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:08.395213 (+   249us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:08.395240 (+    27us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:08.395284 (+    44us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:02:08.395320 (+    36us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:08.395368 (+    48us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:08.395466 (+    98us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:08.425712  2334 raft_consensus.cc:2468] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 15.
I20260709 13:02:08.397962  1750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55840 (request call id 3005) took 1107 ms. Trace:
W20260709 13:02:08.456744  8237 consensus_meta.cc:287] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 1.175s	user 0.010s	sys 0.006s
I20260709 13:02:08.477192  2332 raft_consensus.cc:1240] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 18. Current term is 19. Ops: []
I20260709 13:02:08.498891  2331 raft_consensus.cc:2468] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 16.
I20260709 13:02:08.417379  8454 consensus_queue.cc:260] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.12, Last appended by leader: 12, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:08.483958  8459 raft_consensus.cc:3060] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:02:08.422353  8474 leader_election.cc:290] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:08.503434  1603 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40154 (request call id 2786) took 1143 ms. Trace:
I20260709 13:02:08.479264  1754 rpcz_store.cc:276] 0709 13:02:07.407483 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:08.414859 (+1007376us) service_pool.cc:224] Handling call
0709 13:02:08.415230 (+   371us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:08.415449 (+   219us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:08.415477 (+    28us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:08.415536 (+    59us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:02:08.415584 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:08.415645 (+    61us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:08.415813 (+   168us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:08.504599  1673 raft_consensus.cc:2410] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 for term 20 because replica has last-logged OpId of term: 18 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 13.
I20260709 13:02:08.505483  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:02:08.481006  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 118516us
I20260709 13:02:08.506737  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:08.507426  1674 raft_consensus.cc:2468] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 18.
I20260709 13:02:08.508481  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:08.509105  1674 raft_consensus.cc:2468] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 17.
I20260709 13:02:08.510015  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:08.512708  8481 leader_election.cc:290] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 13:02:08.459015  8373 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.520s	user 0.016s	sys 0.014s
I20260709 13:02:08.480522  8300 raft_consensus.cc:3060] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:02:08.496726  1750 rpcz_store.cc:276] 0709 13:02:07.290794 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:08.397132 (+1106338us) service_pool.cc:224] Handling call
0709 13:02:08.397451 (+   319us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:08.397634 (+   183us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:08.397660 (+    26us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:08.397703 (+    43us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:02:08.397740 (+    37us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:08.397794 (+    54us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:08.397892 (+    98us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:08.504060  1603 rpcz_store.cc:276] 0709 13:02:07.360030 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:07.617329 (+257299us) service_pool.cc:224] Handling call
0709 13:02:07.617692 (+   363us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:07.661675 (+ 43983us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:07.661715 (+    40us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:07.662245 (+   530us) log.cc:844] Serialized 26 byte log entry
0709 13:02:07.898848 (+236603us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:02:07.898908 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:07.899138 (+   230us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:08.501445 (+602307us) raft_consensus.cc:1685] finished
0709 13:02:08.501482 (+    37us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:08.503338 (+  1856us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":61334,"raft.queue_time_us":44435,"raft.run_cpu_time_us":157,"raft.run_wall_time_us":10633,"thread_start_us":163591,"threads_started":1,"wal-append.queue_time_us":433530}
I20260709 13:02:08.458482  8478 raft_consensus.cc:515] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:02:08.520263  1488 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet 983621a6fc5b4c9ea0159a648dfe7a36 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER failed with leader 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 3b879799899a499c8d409ee9f1277cd1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 18 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 13:02:08.524639  1749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55848 (request call id 3116) took 1256 ms. Trace:
I20260709 13:02:08.525229  1749 rpcz_store.cc:276] 0709 13:02:07.267955 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:08.337717 (+1069762us) service_pool.cc:224] Handling call
0709 13:02:08.524317 (+186600us) spinlock_profiling.cc:243] Waited 38.5 ms on lock 0x7b5400220954. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:02:08.524554 (+   237us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":88588672}
I20260709 13:02:08.532964  8320 raft_consensus.cc:3055] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 LEADER]: Stepping down as leader of term 18
I20260709 13:02:08.533551  8320 raft_consensus.cc:740] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:02:08.534757  8320 consensus_queue.cc:260] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.12, Last appended by leader: 12, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:08.536808  8320 raft_consensus.cc:3060] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:02:08.540344  1745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55840 (request call id 3004) took 1267 ms. Trace:
I20260709 13:02:08.540650  1746 raft_consensus.cc:1275] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 16 index: 11. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
I20260709 13:02:08.540767  1745 rpcz_store.cc:276] 0709 13:02:07.272617 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:07.397341 (+124724us) service_pool.cc:224] Handling call
0709 13:02:08.540110 (+1142769us) spinlock_profiling.cc:243] Waited 67.8 ms on lock 0x7b54001d56d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:02:08.540264 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":155953536}
I20260709 13:02:08.543260  1753 raft_consensus.cc:1240] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 15. Current term is 16. Ops: []
I20260709 13:02:08.544526  1746 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55840 (request call id 3012) took 1137 ms. Trace:
I20260709 13:02:08.544958  1746 rpcz_store.cc:276] 0709 13:02:07.407151 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:08.412531 (+1005380us) service_pool.cc:224] Handling call
0709 13:02:08.540229 (+127698us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:08.541091 (+   862us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:08.542230 (+  1139us) spinlock_profiling.cc:243] Waited 127 ms on lock 0x7b5400220950. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661618ad 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:02:08.542423 (+   193us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":293053824}
W20260709 13:02:08.548970  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 116419us
I20260709 13:02:08.557397  8472 leader_election.cc:290] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:08.572795  1671 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57092 (request call id 1480) took 1252 ms. Trace:
I20260709 13:02:08.573406  1671 rpcz_store.cc:276] 0709 13:02:07.320556 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:07.322598 (+  2042us) service_pool.cc:224] Handling call
0709 13:02:07.322967 (+   369us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:08.304425 (+981458us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:08.572333 (+267908us) spinlock_profiling.cc:243] Waited 720 ms on lock 0x7b540025add4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:02:08.572678 (+   345us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1656504064}
I20260709 13:02:08.575346  2331 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39738 (request call id 1189) took 1016 ms. Trace:
I20260709 13:02:08.575888  2331 rpcz_store.cc:276] 0709 13:02:07.559343 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:08.418134 (+858791us) service_pool.cc:224] Handling call
0709 13:02:08.575091 (+156957us) spinlock_profiling.cc:243] Waited 79.7 ms on lock 0x7b540045bfd4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:02:08.575254 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":183398144}
I20260709 13:02:08.580515  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:08.581499  1673 raft_consensus.cc:2468] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 19.
I20260709 13:02:08.592697  1752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55840 (request call id 2993) took 1323 ms. Trace:
I20260709 13:02:08.593335  1752 rpcz_store.cc:276] 0709 13:02:07.269109 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:07.825754 (+556645us) service_pool.cc:224] Handling call
0709 13:02:08.592448 (+766694us) spinlock_profiling.cc:243] Waited 591 ms on lock 0x7b54001d60d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:02:08.592603 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1359387008}
I20260709 13:02:08.603647  1717 leader_election.cc:304] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:08.606300  8402 consensus_queue.cc:1048] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 13:02:08.608063  8402 consensus_queue.cc:1048] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 13:02:08.609772  8402 raft_consensus.cc:2804] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 13:02:08.617493  1753 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43156 (request call id 1364) took 1183 ms. Trace:
I20260709 13:02:08.618062  1753 rpcz_store.cc:276] 0709 13:02:07.434131 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:08.476734 (+1042603us) service_pool.cc:224] Handling call
0709 13:02:08.542969 (+ 66235us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:08.543800 (+   831us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:08.617225 (+ 73425us) spinlock_profiling.cc:243] Waited 65.9 ms on lock 0x7b54001d56d0. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661618ad 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:02:08.617412 (+   187us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":151639424}
I20260709 13:02:08.633349  8310 raft_consensus.cc:3060] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:02:08.646713  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 20 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:08.633993  1717 leader_election.cc:304] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 13:02:08.641270  8481 raft_consensus.cc:493] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:08.647859  8481 raft_consensus.cc:515] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:08.631538  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:08.651386  8481 leader_election.cc:290] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:08.656117  2301 leader_election.cc:400] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 15 election: Vote denied by peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693) with higher term. Message: Invalid argument: T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 for earlier term 15. Current term is 16.
I20260709 13:02:08.668776  2301 leader_election.cc:403] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 15 election: Cancelling election due to peer responding with higher term
I20260709 13:02:08.675843  8323 raft_consensus.cc:2749] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Leader election lost for term 15. Reason: Vote denied by peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693) with higher term. Message: Invalid argument: T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 for earlier term 15. Current term is 16.
I20260709 13:02:08.676692  8481 raft_consensus.cc:493] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:08.677789  8481 raft_consensus.cc:515] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:08.680652  8481 leader_election.cc:290] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:08.642361  1671 raft_consensus.cc:1275] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 18 index: 9. Preceding OpId from leader: term: 19 index: 10. (index mismatch)
I20260709 13:02:08.642822  8402 raft_consensus.cc:697] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 17 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:02:08.650131  1746 raft_consensus.cc:2468] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 15.
I20260709 13:02:08.685096  8379 raft_consensus.cc:2804] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:02:08.687826  8379 raft_consensus.cc:493] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:02:08.688339  8379 raft_consensus.cc:3060] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:02:08.692862  1746 raft_consensus.cc:1275] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 17 index: 14. Preceding OpId from leader: term: 19 index: 15. (index mismatch)
I20260709 13:02:08.705971  8434 consensus_queue.cc:1048] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 13:02:08.707294  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:08.707592  8478 leader_election.cc:290] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 17 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:08.708521  1680 raft_consensus.cc:1240] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 16. Current term is 17. Ops: []
I20260709 13:02:08.498016  8468 consensus_queue.cc:260] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.9, Last appended by leader: 9, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:08.498427  8504 raft_consensus.cc:493] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:02:08.494874  8477 raft_consensus.cc:493] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:08.717809  1747 raft_consensus.cc:1275] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20260709 13:02:08.499651  8484 raft_consensus.cc:515] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:08.719698  1598 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40154 (request call id 2792) took 1175 ms. Trace:
I20260709 13:02:08.724226  2301 leader_election.cc:304] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:02:08.711385  1750 raft_consensus.cc:1275] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 21 index: 14. Preceding OpId from leader: term: 23 index: 15. (index mismatch)
I20260709 13:02:08.751806  1750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43156 (request call id 1372) took 1239 ms. Trace:
I20260709 13:02:08.729854  1674 raft_consensus.cc:3060] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Advancing to term 17
W20260709 13:02:08.752303  1571 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.107.195:40693 timed out after 1.773s (SENT) [suppressed 22 similar messages]
I20260709 13:02:08.750068  1747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43156 (request call id 1367) took 1310 ms. Trace:
I20260709 13:02:08.752760  1718 leader_election.cc:304] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:08.753111  1747 rpcz_store.cc:276] 0709 13:02:07.439189 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:08.496323 (+1057134us) service_pool.cc:224] Handling call
0709 13:02:08.717383 (+221060us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:08.743185 (+ 25802us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:08.749980 (+  6795us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:08.754148  7307 raft_consensus.cc:493] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:08.752183  1750 rpcz_store.cc:276] 0709 13:02:07.512635 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:08.514321 (+1001686us) service_pool.cc:224] Handling call
0709 13:02:08.710867 (+196546us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:08.751510 (+ 40643us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:08.751743 (+   233us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:08.683920  8402 consensus_queue.cc:237] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.9, Last appended by leader: 9, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:08.766641  2301 leader_election.cc:304] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:02:08.765028  7307 raft_consensus.cc:515] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:08.781383  1679 raft_consensus.cc:1240] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 13. Current term is 14. Ops: []
I20260709 13:02:08.499854  8508 raft_consensus.cc:515] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:08.768958  2299 leader_election.cc:304] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:02:08.767720  8497 raft_consensus.cc:2749] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:02:08.751317  1598 rpcz_store.cc:276] 0709 13:02:07.547506 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:07.684498 (+136992us) service_pool.cc:224] Handling call
0709 13:02:07.748383 (+ 63885us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:07.797147 (+ 48764us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 13:02:07.899432 (+102285us) log.cc:844] Serialized 14 byte log entry
0709 13:02:08.352783 (+453351us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:08.405408 (+ 52625us) log.cc:844] Serialized 26 byte log entry
0709 13:02:08.405629 (+   221us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:02:08.405679 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:08.405773 (+    94us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:08.719414 (+313641us) raft_consensus.cc:1685] finished
0709 13:02:08.719442 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:08.719618 (+   176us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":40715,"prepare.queue_time_us":102049,"prepare.run_cpu_time_us":69,"prepare.run_wall_time_us":71,"raft.queue_time_us":51473,"raft.run_cpu_time_us":262,"raft.run_wall_time_us":7437,"thread_start_us":291586,"threads_started":1,"wal-append.queue_time_us":528435}
I20260709 13:02:08.787972  2301 leader_election.cc:304] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
W20260709 13:02:08.787022  8350 consensus_meta.cc:287] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 1.122s	user 0.008s	sys 0.008s
I20260709 13:02:08.784446  8323 raft_consensus.cc:2749] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 13:02:08.798643  2330 raft_consensus.cc:606] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Received request to transfer leadership to TS af520b67945d443b901bbeca37cc5701
W20260709 13:02:08.799799  1602 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.664s	user 0.032s	sys 0.029s
W20260709 13:02:08.800086  1672 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.545s	user 0.027s	sys 0.062s
W20260709 13:02:08.800671  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 511358us
W20260709 13:02:08.796285  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 508014us
I20260709 13:02:08.788887  8280 raft_consensus.cc:2749] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:02:08.791301  8461 raft_consensus.cc:2749] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:02:08.754379  8490 consensus_queue.cc:1059] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 13.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.046s
I20260709 13:02:08.754895  8477 raft_consensus.cc:515] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:08.750887  8504 raft_consensus.cc:515] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:08.802063  2330 raft_consensus.cc:614] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 13:02:08.802359  7307 leader_election.cc:290] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:08.798681  8294 raft_consensus.cc:3055] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 LEADER]: Stepping down as leader of term 19
W20260709 13:02:08.802712  1602 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.667s	user 0.033s	sys 0.029s
W20260709 13:02:08.805698  1645 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:08.827225  1645 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:08.812366  1646 leader_election.cc:304] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 13:02:08.800119  8127 raft_consensus.cc:493] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:08.850715  1751 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55840 (request call id 3011) took 1444 ms. Trace:
I20260709 13:02:08.851132  1751 rpcz_store.cc:276] 0709 13:02:07.406683 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:07.911016 (+504333us) service_pool.cc:224] Handling call
0709 13:02:08.850407 (+939391us) spinlock_profiling.cc:243] Waited 370 ms on lock 0x7b54000e3b54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:02:08.850640 (+   233us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":850970496}
I20260709 13:02:08.801154  1718 leader_election.cc:304] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:08.854018  1748 raft_consensus.cc:1240] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 15. Current term is 16. Ops: [15.14-15.14]
W20260709 13:02:08.815848  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 471864us
I20260709 13:02:08.858692  1646 leader_election.cc:400] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 16 election: Vote denied by peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693) with higher term. Message: Invalid argument: T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for earlier term 16. Current term is 17.
I20260709 13:02:08.859143  1646 leader_election.cc:403] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 16 election: Cancelling election due to peer responding with higher term
I20260709 13:02:08.860563  1646 leader_election.cc:304] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 13:02:08.854241  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 53221.9us
I20260709 13:02:08.864588  1646 leader_election.cc:304] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:08.867321  1646 leader_election.cc:304] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 13:02:08.808425  1644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:08.826419  1602 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 1684) took 1533 ms. Trace:
I20260709 13:02:08.868937  1646 leader_election.cc:304] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:08.895691  8484 leader_election.cc:290] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 14 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:08.896473  8484 raft_consensus.cc:2804] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:02:08.896917  8484 raft_consensus.cc:493] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:02:08.897282  8484 raft_consensus.cc:3060] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:02:08.904556  8508 leader_election.cc:290] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:08.905434  8468 raft_consensus.cc:3060] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:02:08.840420  8294 raft_consensus.cc:740] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:02:08.907382  8294 consensus_queue.cc:260] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 19.15, Last appended by leader: 15, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:08.908841  8294 raft_consensus.cc:3060] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:02:08.817085  8350 consensus_queue.cc:1059] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 12.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.096s
I20260709 13:02:08.905404  8508 raft_consensus.cc:2804] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:02:08.911586  8508 raft_consensus.cc:493] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:02:08.911964  8508 raft_consensus.cc:3060] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:02:08.818689  8500 consensus_queue.cc:1059] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 15.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.094s
I20260709 13:02:08.801378  8497 raft_consensus.cc:2804] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:02:08.912968  8497 raft_consensus.cc:493] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:08.913414  8497 raft_consensus.cc:3060] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:02:08.918833  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:08.920572  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:02:08.816926  8303 consensus_meta.cc:287] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.889s	user 0.013s	sys 0.003s
I20260709 13:02:08.921594  8303 raft_consensus.cc:515] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:08.923364  8504 leader_election.cc:290] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:08.868932  1602 rpcz_store.cc:276] 0709 13:02:07.294001 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:07.600186 (+306185us) service_pool.cc:224] Handling call
0709 13:02:07.600593 (+   407us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:07.600876 (+   283us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 13:02:08.135506 (+534630us) log.cc:844] Serialized 14 byte log entry
0709 13:02:08.826010 (+690504us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:08.826076 (+    66us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:02:08.826135 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:08.826209 (+    74us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:08.826335 (+   126us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":173205,"prepare.run_cpu_time_us":29,"prepare.run_wall_time_us":30,"raft.queue_time_us":15778,"thread_start_us":1160471,"threads_started":2}
I20260709 13:02:08.870226  8414 consensus_queue.cc:1048] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
W20260709 13:02:08.802981  1672 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.548s	user 0.027s	sys 0.062s
I20260709 13:02:08.875710  8237 raft_consensus.cc:2804] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:02:08.925958  8237 raft_consensus.cc:493] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:02:08.926414  8237 raft_consensus.cc:3060] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:02:08.878584  8127 raft_consensus.cc:515] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:08.883627  1679 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52766 (request call id 3106) took 1401 ms. Trace:
I20260709 13:02:08.885097  1680 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52778 (request call id 2642) took 1702 ms. Trace:
I20260709 13:02:08.933954  1680 rpcz_store.cc:276] 0709 13:02:07.182811 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:07.182982 (+   171us) service_pool.cc:224] Handling call
0709 13:02:07.455953 (+272971us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:08.729954 (+1274001us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:08.884366 (+154412us) spinlock_profiling.cc:243] Waited 1.25 s on lock 0x7b5400091354. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:02:08.884883 (+   517us) spinlock_profiling.cc:243] Waited 266 ms on lock 0x7b5400091350. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661618ad 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:02:08.885023 (+   140us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3494191744}
W20260709 13:02:08.934813  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 79789.3us
I20260709 13:02:08.935285  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 16 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:08.935954  1680 raft_consensus.cc:2468] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 15.
I20260709 13:02:08.889622  1748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43156 (request call id 1373) took 1374 ms. Trace:
I20260709 13:02:08.823939  8190 raft_consensus.cc:493] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:08.936789  1748 rpcz_store.cc:276] 0709 13:02:07.515124 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:08.539206 (+1024082us) service_pool.cc:224] Handling call
0709 13:02:08.853646 (+314440us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:08.854650 (+  1004us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:08.889351 (+ 34701us) spinlock_profiling.cc:243] Waited 314 ms on lock 0x7b54000e3b50. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661618ad 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:02:08.889530 (+   179us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":722284416}
W20260709 13:02:08.810899  8145 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.730s	user 0.044s	sys 0.049s
I20260709 13:02:08.932325  1679 rpcz_store.cc:276] 0709 13:02:07.482354 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:07.640204 (+157850us) service_pool.cc:224] Handling call
0709 13:02:07.640644 (+   440us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:08.781927 (+1141283us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:08.883277 (+101350us) spinlock_profiling.cc:243] Waited 1.14 s on lock 0x7b5400058cd4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:02:08.883507 (+   230us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2624203136}
I20260709 13:02:08.942175  1678 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52766 (request call id 3104) took 1600 ms. Trace:
I20260709 13:02:08.959297  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 20 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:08.960215  1680 raft_consensus.cc:2410] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for term 20 because replica has last-logged OpId of term: 18 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 8.
I20260709 13:02:08.959129  1678 rpcz_store.cc:276] 0709 13:02:07.341430 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:07.341567 (+   137us) service_pool.cc:224] Handling call
0709 13:02:07.364443 (+ 22876us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:07.366595 (+  2152us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 13:02:07.473746 (+107151us) spinlock_profiling.cc:243] Waited 15.7 ms on lock 0x7b180034a020. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f4263057e07 00007f426beb2ab9 00007f426307469b 00007f426307293f 00007f4263072455 00007f4269b5a3c7 00007f426616c9a7 00007f426617c0bf 00007f426617c038
0709 13:02:07.653322 (+179576us) log.cc:844] Serialized 14 byte log entry
0709 13:02:08.728890 (+1075568us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:08.742968 (+ 14078us) log.cc:844] Serialized 26 byte log entry
0709 13:02:08.743186 (+   218us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:02:08.743242 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:08.743316 (+    74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:08.935711 (+192395us) raft_consensus.cc:1685] finished
0709 13:02:08.941980 (+  6269us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:08.942111 (+   131us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":14450,"prepare.queue_time_us":43105,"prepare.run_cpu_time_us":56,"prepare.run_wall_time_us":61,"raft.queue_time_us":91128,"raft.run_cpu_time_us":231,"raft.run_wall_time_us":14359,"spinlock_wait_cycles":36075776,"thread_start_us":1166963,"threads_started":3,"wal-append.queue_time_us":714069}
I20260709 13:02:08.924226  8303 leader_election.cc:290] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:08.811951  8478 raft_consensus.cc:493] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:08.849598  8310 raft_consensus.cc:493] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:08.943565  8434 consensus_queue.cc:1048] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.032s
I20260709 13:02:08.936510  8298 raft_consensus.cc:2804] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Leader election won for term 19
I20260709 13:02:08.932121  8477 leader_election.cc:290] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:08.976063  8190 raft_consensus.cc:515] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:08.984086  8472 raft_consensus.cc:493] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
W20260709 13:02:08.985324  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 49148.1us
I20260709 13:02:08.985424  8403 consensus_queue.cc:1048] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 13:02:08.932152  8127 leader_election.cc:290] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:08.986166  8507 raft_consensus.cc:3055] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 13:02:08.986711  8368 raft_consensus.cc:3055] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 13:02:08.988106  8514 raft_consensus.cc:493] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:08.940613  8145 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.882s	user 0.045s	sys 0.050s
I20260709 13:02:08.938220  8506 raft_consensus.cc:2804] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:02:08.940955  8496 consensus_queue.cc:1048] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.059s
I20260709 13:02:09.008697  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:09.038857  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:02:09.027243  8311 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.643s	user 0.018s	sys 0.023s
I20260709 13:02:09.049046  2335 raft_consensus.cc:2468] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 16.
I20260709 13:02:09.051640  2328 raft_consensus.cc:2410] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for term 20 because replica has last-logged OpId of term: 18 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 14.
I20260709 13:02:09.056162  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:09.060389  1569 leader_election.cc:304] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:02:09.052137  2333 raft_consensus.cc:1240] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 19. Current term is 20. Ops: [19.15-19.15]
W20260709 13:02:09.048370  8404 consensus_meta.cc:287] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.717s	user 0.010s	sys 0.004s
I20260709 13:02:09.043699  8477 raft_consensus.cc:2804] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:02:09.069514  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:09.034822  8507 raft_consensus.cc:740] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
W20260709 13:02:09.058765  8311 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.675s	user 0.018s	sys 0.023s
W20260709 13:02:09.072770  1719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:09.072885  1675 raft_consensus.cc:2410] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for term 20 because replica has last-logged OpId of term: 19 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 9.
I20260709 13:02:09.050608  8323 raft_consensus.cc:493] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
W20260709 13:02:09.075181  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 210522us
I20260709 13:02:09.035506  8298 raft_consensus.cc:697] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 19 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:02:09.043095  8368 raft_consensus.cc:740] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:02:08.997759  8478 raft_consensus.cc:515] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:09.092555  1718 leader_election.cc:304] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:09.116215  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:09.116962  2333 raft_consensus.cc:2468] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 20.
I20260709 13:02:09.045169  8472 raft_consensus.cc:515] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:09.022050  1674 raft_consensus.cc:2468] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 17.
I20260709 13:02:09.068013  8127 raft_consensus.cc:2804] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Leader pre-election won for term 17
W20260709 13:02:09.114624  1677 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.571s	user 0.003s	sys 0.010s
I20260709 13:02:09.043047  8310 raft_consensus.cc:515] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:09.116899  1718 leader_election.cc:304] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 13:02:09.095494  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 343376us
I20260709 13:02:09.044297  8300 raft_consensus.cc:515] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:09.057637  8280 consensus_queue.cc:1059] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 14.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.037s
I20260709 13:02:09.071311  8477 raft_consensus.cc:493] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
W20260709 13:02:09.122773  1677 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.579s	user 0.004s	sys 0.010s
I20260709 13:02:09.126466  2330 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:35790 (request call id 31) took 1590 ms. Trace:
I20260709 13:02:09.058342  8514 raft_consensus.cc:515] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:02:09.189481  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 72849us
I20260709 13:02:09.190805  1718 leader_election.cc:304] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:09.184639  2330 rpcz_store.cc:276] 0709 13:02:07.498734 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:07.505357 (+  6623us) service_pool.cc:224] Handling call
0709 13:02:09.089086 (+1583729us) spinlock_profiling.cc:243] Waited 1.02 s on lock 0x7b540065ed54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f426615c853 00007f426b12a60a 00007f42646b07d4 00007f42646b0762 00007f42646b0691 00007f42646b060c 00007f42646af8c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:02:09.089221 (+   135us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2352045312}
I20260709 13:02:09.121773  8145 raft_consensus.cc:2804] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:02:09.196696  1571 leader_election.cc:304] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:02:09.215752  1571 leader_election.cc:304] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 13:02:09.122112  8127 raft_consensus.cc:493] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:09.232645  8127 raft_consensus.cc:3060] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:02:09.122256  8474 raft_consensus.cc:2749] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Leader election lost for term 16. Reason: Vote denied by peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693) with higher term. Message: Invalid argument: T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for earlier term 16. Current term is 17.
W20260709 13:02:09.177340  8371 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.516s	user 0.005s	sys 0.009s
I20260709 13:02:09.210474  1676 raft_consensus.cc:2393] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 21: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
W20260709 13:02:09.199599  1643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:09.241803  1643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:09.242394  1643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:09.217072  1673 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.626s	user 0.008s	sys 0.009s
W20260709 13:02:09.246470  1673 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.656s	user 0.008s	sys 0.009s
W20260709 13:02:09.224068  1754 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.717s	user 0.004s	sys 0.015s
I20260709 13:02:09.187520  8481 raft_consensus.cc:2749] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20260709 13:02:09.227509  1569 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:09.118798  8323 raft_consensus.cc:515] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:09.180141  8517 raft_consensus.cc:493] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
W20260709 13:02:09.224566  8372 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.879s	user 0.009s	sys 0.013s
W20260709 13:02:09.254387  1754 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.748s	user 0.004s	sys 0.015s
W20260709 13:02:09.211159  8338 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.612s	user 0.008s	sys 0.008s
I20260709 13:02:09.194830  8477 raft_consensus.cc:3060] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:02:09.208457  8145 raft_consensus.cc:493] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:09.210889  8360 raft_consensus.cc:493] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:09.179575  8508 raft_consensus.cc:515] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:09.215266  8373 raft_consensus.cc:2749] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
W20260709 13:02:09.277380  1745 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.670s	user 0.007s	sys 0.015s
I20260709 13:02:09.257592  1571 leader_election.cc:304] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
W20260709 13:02:09.262475  8372 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.928s	user 0.011s	sys 0.013s
W20260709 13:02:09.319152  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 61719.5us
W20260709 13:02:09.317709  1745 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.711s	user 0.008s	sys 0.016s
W20260709 13:02:09.262955  8338 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.677s	user 0.008s	sys 0.009s
I20260709 13:02:09.320133  1571 leader_election.cc:304] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:02:09.261531  7745 raft_consensus.cc:2804] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Leader pre-election won for term 13
W20260709 13:02:09.226333  8457 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.614s	user 0.010s	sys 0.012s
W20260709 13:02:09.279348  8459 consensus_meta.cc:287] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.769s	user 0.015s	sys 0.000s
I20260709 13:02:09.277791  8145 raft_consensus.cc:3060] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:02:09.315618  8357 raft_consensus.cc:493] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
W20260709 13:02:09.368475  8519 log.cc:927] Time spent T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.249s	user 0.001s	sys 0.000s
W20260709 13:02:09.370623  8457 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.758s	user 0.011s	sys 0.012s
I20260709 13:02:09.389132  8300 leader_election.cc:290] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 21 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:09.323958  7745 raft_consensus.cc:493] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:09.390170  8357 raft_consensus.cc:515] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:09.334203  8154 raft_consensus.cc:2749] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
W20260709 13:02:09.394059  1644 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.523s	user 0.024s	sys 0.057s
I20260709 13:02:09.369863  8513 raft_consensus.cc:493] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:09.397457  1644 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.527s	user 0.024s	sys 0.058s
I20260709 13:02:09.265010  8517 raft_consensus.cc:515] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:02:09.397001  1749 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.729s	user 0.012s	sys 0.020s
W20260709 13:02:09.400661  1749 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.733s	user 0.013s	sys 0.020s
W20260709 13:02:09.399749  1646 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.506s	user 0.012s	sys 0.018s
I20260709 13:02:09.376332  8459 raft_consensus.cc:2749] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
W20260709 13:02:09.403000  1646 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.509s	user 0.013s	sys 0.018s
W20260709 13:02:09.404978  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 536378us
I20260709 13:02:09.409595  1644 leader_election.cc:304] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:02:09.413499  1646 leader_election.cc:304] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
W20260709 13:02:09.415721  1645 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.526s	user 0.033s	sys 0.052s
W20260709 13:02:09.416712  1645 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.527s	user 0.034s	sys 0.052s
I20260709 13:02:09.392498  7728 raft_consensus.cc:2749] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 13:02:09.397101  7745 raft_consensus.cc:3060] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:02:09.278359  8507 consensus_queue.cc:260] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.11, Last appended by leader: 11, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:09.318426  8360 raft_consensus.cc:515] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:09.263908  8298 consensus_queue.cc:237] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 17.14, Last appended by leader: 14, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:09.437397  8311 raft_consensus.cc:2749] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
W20260709 13:02:09.440296  8520 log.cc:927] Time spent T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.317s	user 0.001s	sys 0.000s
I20260709 13:02:09.430227  8300 raft_consensus.cc:2804] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:02:09.451308  8310 leader_election.cc:290] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:09.499616  8395 consensus_queue.cc:1059] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 13.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.016s
W20260709 13:02:09.484364  8523 log.cc:927] Time spent T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.078s	user 0.000s	sys 0.000s
W20260709 13:02:09.516943  8320 consensus_meta.cc:287] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.980s	user 0.013s	sys 0.004s
W20260709 13:02:09.468578  8354 consensus_queue.cc:1175] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" responder_term: 13 status { last_received { term: 13 index: 11 } last_committed_idx: 10 last_received_current_leader { term: 13 index: 11 } }
I20260709 13:02:09.511252  8514 leader_election.cc:290] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:09.516918  8154 raft_consensus.cc:3055] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 LEADER]: Stepping down as leader of term 13
I20260709 13:02:09.496032  2299 leader_election.cc:304] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:02:09.447420  8404 raft_consensus.cc:493] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:02:09.518841  8154 raft_consensus.cc:740] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:02:09.519327  8404 raft_consensus.cc:515] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:09.402313  8513 raft_consensus.cc:515] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:09.519834  8496 raft_consensus.cc:3055] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 13:02:09.520368  8496 raft_consensus.cc:740] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:02:09.521675  8404 leader_election.cc:290] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:09.521303  8496 consensus_queue.cc:260] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 15.14, Last appended by leader: 14, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:09.522614  8454 raft_consensus.cc:3060] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:02:09.523777  1675 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52778 (request call id 2649) took 1023 ms. Trace:
I20260709 13:02:09.523996  8300 raft_consensus.cc:493] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:09.524533  8300 raft_consensus.cc:3060] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:02:09.524245  1675 rpcz_store.cc:276] 0709 13:02:08.500024 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:08.505170 (+  5146us) service_pool.cc:224] Handling call
0709 13:02:09.523550 (+1018380us) spinlock_profiling.cc:243] Waited 567 ms on lock 0x7b5400092754. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:02:09.523705 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1302991232}
I20260709 13:02:09.525708  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:09.468741  8472 leader_election.cc:290] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:09.483480  8508 leader_election.cc:290] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:09.506940  8360 leader_election.cc:290] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:09.449966  7728 raft_consensus.cc:493] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:09.474964  8468 raft_consensus.cc:493] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:09.519938  8154 consensus_queue.cc:260] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 13.12, Last appended by leader: 12, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:09.492223  8515 raft_consensus.cc:493] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:09.533587  1644 leader_election.cc:304] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:02:09.534785  1675 raft_consensus.cc:2393] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 14: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:02:09.505249  1752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43156 (request call id 1375) took 1345 ms. Trace:
W20260709 13:02:09.468884  1671 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.533s	user 0.001s	sys 0.025s
W20260709 13:02:09.430863  8471 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.528s	user 0.012s	sys 0.020s
W20260709 13:02:09.574014  1671 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.638s	user 0.001s	sys 0.040s
I20260709 13:02:09.536139  1675 raft_consensus.cc:1240] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 18. Current term is 19. Ops: []
I20260709 13:02:09.584292  1677 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52766 (request call id 3101) took 2262 ms. Trace:
I20260709 13:02:09.576642  1569 leader_election.cc:304] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:02:09.574831  1752 rpcz_store.cc:276] 0709 13:02:08.160391 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:08.595753 (+435362us) service_pool.cc:224] Handling call
0709 13:02:08.596144 (+   391us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:02:08.596634 (+   490us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:08.596675 (+    41us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:02:08.597475 (+   800us) log.cc:844] Serialized 52 byte log entry
0709 13:02:09.088700 (+491225us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:02:09.119555 (+ 30855us) log.cc:844] Serialized 14 byte log entry
0709 13:02:09.120051 (+   496us) log.cc:844] Serialized 14 byte log entry
0709 13:02:09.122280 (+  2229us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:09.122363 (+    83us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:09.504959 (+382596us) raft_consensus.cc:1685] finished
0709 13:02:09.504988 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:09.505172 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3536,"prepare.queue_time_us":118268,"prepare.run_cpu_time_us":129,"prepare.run_wall_time_us":9061,"raft.queue_time_us":13752,"raft.run_cpu_time_us":270,"raft.run_wall_time_us":3028,"spinlock_wait_cycles":192770816,"thread_start_us":490710,"threads_started":1,"wal-append.queue_time_us":810374}
I20260709 13:02:09.588579  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:02:09.593415  8471 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.690s	user 0.012s	sys 0.031s
W20260709 13:02:09.489308  8237 consensus_meta.cc:287] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.562s	user 0.009s	sys 0.001s
I20260709 13:02:09.529616  8154 raft_consensus.cc:3060] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:02:09.516853  8478 leader_election.cc:290] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:09.521724  8513 leader_election.cc:290] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:09.575721  8404 raft_consensus.cc:2804] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:02:09.528105  8357 leader_election.cc:290] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:09.575237  8507 raft_consensus.cc:3060] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:02:09.609972  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:09.559331  8508 raft_consensus.cc:493] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:09.611573  2301 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:09.630214  2330 raft_consensus.cc:3060] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:02:09.504503  8517 leader_election.cc:290] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:09.587607  8496 raft_consensus.cc:3060] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:02:09.590063  8500 raft_consensus.cc:2804] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:02:09.589602  8472 raft_consensus.cc:493] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:09.647190  8472 raft_consensus.cc:515] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:09.650038  8472 leader_election.cc:290] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:09.625481  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:09.653378  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:09.576233  8338 raft_consensus.cc:493] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:02:09.588965  8504 raft_consensus.cc:493] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:02:09.654762  8338 raft_consensus.cc:515] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:09.654896  8504 raft_consensus.cc:515] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:09.657229  8338 leader_election.cc:290] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:09.657846  2335 raft_consensus.cc:2468] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 19.
I20260709 13:02:09.602955  8395 raft_consensus.cc:3060] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:02:09.660804  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:09.530020  8127 raft_consensus.cc:515] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:09.526408  8477 raft_consensus.cc:515] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:09.527802  7728 raft_consensus.cc:515] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:09.528281  8468 raft_consensus.cc:515] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:09.530946  8515 raft_consensus.cc:515] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:09.623724  8404 raft_consensus.cc:493] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:09.676075  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
W20260709 13:02:09.663964  8523 log.cc:927] Time spent T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.058s	user 0.000s	sys 0.000s
I20260709 13:02:09.676447  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:02:09.663297  8484 consensus_meta.cc:287] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.766s	user 0.014s	sys 0.001s
I20260709 13:02:09.584888  1677 rpcz_store.cc:276] 0709 13:02:07.322327 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:07.326231 (+  3904us) service_pool.cc:224] Handling call
0709 13:02:07.340045 (+ 13814us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:07.340285 (+   240us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 13:02:07.362790 (+ 22505us) log.cc:844] Serialized 14 byte log entry
0709 13:02:08.543239 (+1180449us) spinlock_profiling.cc:243] Waited 2.54 ms on lock 0x7b180034c3c0. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f4263057e07 00007f426beb2ab9 00007f426307469b 00007f426307293f 00007f4263074c02 00007f4265cfc20b 00007f4265d01d23 00007f4265d0223a 00007f426616ca2a
0709 13:02:09.583430 (+1040191us) log.cc:844] Serialized 14 byte log entry
0709 13:02:09.583799 (+   369us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:09.583867 (+    68us) raft_consensus.cc:1645] Marking committed up to 13
0709 13:02:09.583937 (+    70us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:09.584035 (+    98us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:09.584208 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6414,"prepare.queue_time_us":309896,"prepare.run_cpu_time_us":119,"prepare.run_wall_time_us":5858,"raft.queue_time_us":19936,"spinlock_wait_cycles":13262464,"thread_start_us":1019190,"threads_started":2,"wal-append.queue_time_us":2016705}
I20260709 13:02:09.714015  1680 raft_consensus.cc:2468] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 16.
I20260709 13:02:09.611099  8237 raft_consensus.cc:515] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:09.717415  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:09.717746  2299 leader_election.cc:304] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:02:09.718116  1680 raft_consensus.cc:2468] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 19.
I20260709 13:02:09.719266  2336 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f"
dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
mode: GRACEFUL
new_leader_uuid: "af520b67945d443b901bbeca37cc5701"
 from {username='slave'} at 127.0.0.1:35790
I20260709 13:02:09.719427  2299 leader_election.cc:304] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:02:09.615491  8373 consensus_queue.cc:1059] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 18.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.059s
W20260709 13:02:09.713177  8379 consensus_meta.cc:287] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 1.024s	user 0.017s	sys 0.001s
W20260709 13:02:09.713886  8294 consensus_meta.cc:287] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.805s	user 0.013s	sys 0.006s
I20260709 13:02:09.730870  2330 raft_consensus.cc:2468] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 14.
I20260709 13:02:09.676252  8404 raft_consensus.cc:3060] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:02:09.732805  1644 leader_election.cc:304] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:02:09.663838  8508 raft_consensus.cc:515] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:09.719590  8537 raft_consensus.cc:493] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
W20260709 13:02:09.745141  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 425269us
W20260709 13:02:09.744935  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 271520us
I20260709 13:02:09.730578  8403 raft_consensus.cc:2804] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:02:09.746816  1571 leader_election.cc:304] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:02:09.732064  8472 raft_consensus.cc:2804] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:02:09.715603  8484 raft_consensus.cc:515] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:09.762789  1673 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52778 (request call id 2653) took 1261 ms. Trace:
W20260709 13:02:09.744166  8522 log.cc:927] Time spent T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.085s	user 0.000s	sys 0.000s
I20260709 13:02:09.736861  8514 raft_consensus.cc:2804] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:02:09.768733  1677 raft_consensus.cc:1240] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 18. Current term is 19. Ops: []
I20260709 13:02:09.771870  1571 leader_election.cc:304] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
W20260709 13:02:09.774978  8528 log.cc:927] Time spent T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.062s	user 0.000s	sys 0.001s
I20260709 13:02:09.778667  8360 raft_consensus.cc:493] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:09.779258  8360 raft_consensus.cc:515] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:09.781541  8360 leader_election.cc:290] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:09.782056  8360 raft_consensus.cc:2749] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
W20260709 13:02:09.783119  8542 log.cc:927] Time spent T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.050s	user 0.000s	sys 0.000s
I20260709 13:02:09.785481  8484 leader_election.cc:290] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:02:09.786502  8538 log.cc:927] Time spent T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.113s	user 0.000s	sys 0.001s
I20260709 13:02:09.787750  8500 raft_consensus.cc:493] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:02:09.788297  8500 raft_consensus.cc:3060] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Advancing to term 16
W20260709 13:02:09.789117  8531 log.cc:927] Time spent T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.067s	user 0.000s	sys 0.000s
I20260709 13:02:09.751587  8294 raft_consensus.cc:3055] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 LEADER]: Stepping down as leader of term 18
I20260709 13:02:09.790671  8294 raft_consensus.cc:740] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:02:09.791617  8237 leader_election.cc:290] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 15 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:09.793586  1675 raft_consensus.cc:1240] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 18. Current term is 19. Ops: []
I20260709 13:02:09.748272  8403 raft_consensus.cc:493] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
W20260709 13:02:09.763697  8497 consensus_meta.cc:287] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.850s	user 0.009s	sys 0.008s
I20260709 13:02:09.798174  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:09.798555  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:09.734786  8379 raft_consensus.cc:515] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:09.803865  8300 raft_consensus.cc:515] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:09.763659  1673 rpcz_store.cc:276] 0709 13:02:08.501256 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:08.582314 (+ 81058us) service_pool.cc:224] Handling call
0709 13:02:08.590256 (+  7942us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:08.590516 (+   260us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 13:02:09.247057 (+656541us) log.cc:844] Serialized 14 byte log entry
0709 13:02:09.762024 (+514967us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:09.762069 (+    45us) raft_consensus.cc:1645] Marking committed up to 14
0709 13:02:09.762112 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:09.762187 (+    75us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:09.762320 (+   133us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":43053,"prepare.queue_time_us":856873,"prepare.run_cpu_time_us":54,"prepare.run_wall_time_us":2254,"raft.queue_time_us":44076,"thread_start_us":1127322,"threads_started":2}
I20260709 13:02:09.811097  8506 raft_consensus.cc:493] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:09.859139  8506 raft_consensus.cc:3060] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:02:09.860368  1671 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52778 (request call id 2655) took 1356 ms. Trace:
I20260709 13:02:09.753347  8537 raft_consensus.cc:515] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:09.860809  1671 rpcz_store.cc:276] 0709 13:02:08.503652 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:08.681982 (+178330us) service_pool.cc:224] Handling call
0709 13:02:08.682503 (+   521us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:08.934659 (+252156us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:08.934695 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:08.935243 (+   548us) log.cc:844] Serialized 26 byte log entry
0709 13:02:09.575106 (+639863us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:02:09.575156 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:09.855971 (+280815us) spinlock_profiling.cc:243] Waited 3.07 ms on lock 0x7b48002ac558. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426b08baa1 00007f426b082b37 00007f426b0829ca 00007f426b17a26f 00007f426b179f37 00007f426b18e55c 00007f426b18e508 00007f426b18e488 00007f426b18e443 00007f426b18d6ec 00007f426a0b7a96
0709 13:02:09.859985 (+  4014us) spinlock_profiling.cc:243] Waited 252 ms on lock 0x7b54001f6154. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:02:09.860056 (+    71us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:09.860114 (+    58us) raft_consensus.cc:1685] finished
0709 13:02:09.860127 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:09.860292 (+   165us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":123105,"raft.run_cpu_time_us":8645,"raft.run_wall_time_us":795443,"spinlock_wait_cycles":586274944,"thread_start_us":639403,"threads_started":1,"wal-append.queue_time_us":731961}
I20260709 13:02:09.792066  8294 consensus_queue.cc:260] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.10, Last appended by leader: 10, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:09.862951  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:09.863188  8294 raft_consensus.cc:3060] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:02:09.863988  8537 leader_election.cc:290] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:09.834784  8403 raft_consensus.cc:3060] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:02:09.815603  1598 raft_consensus.cc:1275] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 16 index: 13. (index mismatch)
I20260709 13:02:09.818768  8508 leader_election.cc:290] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:09.820031  8395 raft_consensus.cc:2749] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:02:09.820894  8514 raft_consensus.cc:697] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:02:09.868307  8514 consensus_queue.cc:237] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 13.12, Last appended by leader: 12, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:09.869443  1643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:09.869796  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 16 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:09.822888  8477 leader_election.cc:290] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 18 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:09.823962  7728 leader_election.cc:290] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:09.824800  8127 leader_election.cc:290] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 17 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:09.825300  8368 consensus_queue.cc:260] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.11, Last appended by leader: 11, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:09.832858  8512 log.cc:927] Time spent T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.745s	user 0.004s	sys 0.005s
I20260709 13:02:09.757894  8472 raft_consensus.cc:493] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:02:09.819960  8515 leader_election.cc:290] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:09.846875  8497 raft_consensus.cc:515] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:09.850627  8379 leader_election.cc:290] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:09.852845  8300 leader_election.cc:290] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 16 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:02:09.853513  2298 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:09.808199  8323 leader_election.cc:290] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 13:02:09.854221  8532 log.cc:927] Time spent T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.362s	user 0.001s	sys 0.000s
I20260709 13:02:09.889366  1599 raft_consensus.cc:2468] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 14.
I20260709 13:02:09.876796  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:02:09.875411  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 75822.9us
I20260709 13:02:09.886847  1680 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52766 (request call id 3113) took 1015 ms. Trace:
I20260709 13:02:09.886770  1671 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52766 (request call id 3117) took 1001 ms. Trace:
I20260709 13:02:09.892475  1671 rpcz_store.cc:276] 0709 13:02:08.872691 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:09.873127 (+1000436us) service_pool.cc:224] Handling call
0709 13:02:09.873407 (+   280us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:09.873635 (+   228us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:09.873767 (+   132us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:09.873818 (+    51us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:02:09.873883 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:09.873961 (+    78us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:09.874091 (+   130us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:02:09.897936  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 707571us
I20260709 13:02:09.898849  1718 leader_election.cc:304] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 13:02:09.899398  8145 consensus_meta.cc:287] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.516s	user 0.010s	sys 0.001s
I20260709 13:02:09.899883  8145 raft_consensus.cc:515] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:09.897949  1753 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55840 (request call id 3022) took 1282 ms. Trace:
I20260709 13:02:09.905635  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:09.905905  1675 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52766 (request call id 3112) took 1043 ms. Trace:
I20260709 13:02:09.904748  1753 rpcz_store.cc:276] 0709 13:02:08.615088 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:08.696922 (+ 81834us) service_pool.cc:224] Handling call
0709 13:02:08.697198 (+   276us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:08.697401 (+   203us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 13:02:08.740003 (+ 42602us) log.cc:844] Serialized 14 byte log entry
0709 13:02:09.853591 (+1113588us) spinlock_profiling.cc:243] Waited 12.5 ms on lock 0x7b4400022b4c. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f4262f9bc85 00007f4262f9c6a2 00007f4265d0e1db 00007f4265d09aaf 00007f4265cfd86e 00007f4265cfcbb6 00007f4265cfc7d8 00007f4265d06221 00007f4265d061d9 00007f4265d06169
0709 13:02:09.897413 (+ 43822us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:09.897500 (+    87us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:02:09.897572 (+    72us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:09.897683 (+   111us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:09.897842 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":712544,"prepare.queue_time_us":42354,"prepare.run_cpu_time_us":58,"prepare.run_wall_time_us":60,"spinlock_wait_cycles":28648320,"thread_start_us":419553,"threads_started":1,"wal-append.queue_time_us":751401}
I20260709 13:02:09.907385  2333 raft_consensus.cc:2393] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 21: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:02:09.904076  8497 leader_election.cc:290] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 21 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:09.906296  1675 rpcz_store.cc:276] 0709 13:02:08.862116 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:09.615485 (+753369us) service_pool.cc:224] Handling call
0709 13:02:09.663529 (+ 48044us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:09.794003 (+130474us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:09.905630 (+111627us) spinlock_profiling.cc:243] Waited 130 ms on lock 0x7b54001f4854. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:02:09.905838 (+   208us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":298718080}
I20260709 13:02:09.911849  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:09.879393  8472 raft_consensus.cc:3060] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:02:09.922521  1672 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57092 (request call id 1495) took 1839 ms. Trace:
I20260709 13:02:09.923044  1672 rpcz_store.cc:276] 0709 13:02:08.083364 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:08.184229 (+100865us) service_pool.cc:224] Handling call
0709 13:02:08.184689 (+   460us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:08.254073 (+ 69384us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:08.254115 (+    42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:08.254806 (+   691us) log.cc:844] Serialized 26 byte log entry
0709 13:02:08.925565 (+670759us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:02:08.925613 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:08.925698 (+    85us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:09.832220 (+906522us) spinlock_profiling.cc:243] Waited 7.65 ms on lock 0x7b4400022b4c. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f4262f9bc85 00007f4262f9c6a2 00007f4265d0e1db 00007f4265d09aaf 00007f4265cfd86e 00007f4265cfcbb6 00007f4265cfc7d8 00007f4265d06221 00007f4265d061d9 00007f4265d06169
0709 13:02:09.917216 (+ 84996us) raft_consensus.cc:1685] finished
0709 13:02:09.917244 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:09.922410 (+  5166us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":119157,"raft.run_cpu_time_us":65,"raft.run_wall_time_us":66,"spinlock_wait_cycles":17603840,"thread_start_us":670293,"threads_started":1,"wal-append.queue_time_us":776518}
I20260709 13:02:09.902292  8145 leader_election.cc:290] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 20 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:09.904853  8468 leader_election.cc:290] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:09.959981  1749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43156 (request call id 1377) took 1771 ms. Trace:
I20260709 13:02:09.963680  1749 rpcz_store.cc:276] 0709 13:02:08.188352 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:08.648461 (+460109us) service_pool.cc:224] Handling call
0709 13:02:08.648857 (+   396us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:08.666827 (+ 17970us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:08.666870 (+    43us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:08.667571 (+   701us) log.cc:844] Serialized 26 byte log entry
0709 13:02:09.401931 (+734360us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:02:09.430234 (+ 28303us) log.cc:844] Serialized 14 byte log entry
0709 13:02:09.959528 (+529294us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:09.959632 (+   104us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:09.959691 (+    59us) raft_consensus.cc:1685] finished
0709 13:02:09.959705 (+    14us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:09.959874 (+   169us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":494898,"prepare.queue_time_us":28040,"prepare.run_cpu_time_us":67,"prepare.run_wall_time_us":2812,"raft.queue_time_us":9528,"raft.run_cpu_time_us":123,"raft.run_wall_time_us":124,"thread_start_us":733806,"threads_started":1,"wal-append.queue_time_us":947945}
I20260709 13:02:09.965240  1745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43156 (request call id 1376) took 1787 ms. Trace:
W20260709 13:02:09.961006  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 69867.5us
I20260709 13:02:09.968539  1745 rpcz_store.cc:276] 0709 13:02:08.178240 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:08.605962 (+427722us) service_pool.cc:224] Handling call
0709 13:02:08.606283 (+   321us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:08.606534 (+   251us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 13:02:08.606802 (+   268us) log.cc:844] Serialized 14 byte log entry
0709 13:02:09.964165 (+1357363us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:09.964864 (+   699us) raft_consensus.cc:1645] Marking committed up to 13
0709 13:02:09.964931 (+    67us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:09.965022 (+    91us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:09.965149 (+   127us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":599031,"prepare.queue_time_us":6331,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":45,"thread_start_us":757806,"threads_started":2,"wal-append.queue_time_us":1018491}
I20260709 13:02:09.922829  8372 raft_consensus.cc:2749] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:02:09.970752  1746 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43156 (request call id 1382) took 1333 ms. Trace:
W20260709 13:02:09.973361  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 70688us
I20260709 13:02:09.971567  1675 raft_consensus.cc:1240] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 15. Current term is 16. Ops: [15.14-15.14]
W20260709 13:02:09.959527  7745 consensus_meta.cc:287] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.510s	user 0.010s	sys 0.003s
I20260709 13:02:09.925578  8461 raft_consensus.cc:493] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:09.966356  8473 raft_consensus.cc:2749] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:02:09.976435  8524 raft_consensus.cc:493] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:09.977025  8524 raft_consensus.cc:515] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:09.979719  8524 leader_election.cc:290] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:09.959784  8477 raft_consensus.cc:493] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:09.980731  8477 raft_consensus.cc:515] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:09.964340  8550 raft_consensus.cc:2749] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:02:09.987915  8395 consensus_queue.cc:1059] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 18.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 13:02:09.923880  8403 raft_consensus.cc:515] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:09.928218  8456 raft_consensus.cc:2804] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 13:02:09.973479  1746 rpcz_store.cc:276] 0709 13:02:08.637544 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:08.739105 (+101561us) service_pool.cc:224] Handling call
0709 13:02:08.749483 (+ 10378us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:08.749786 (+   303us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 13:02:09.215383 (+465597us) log.cc:844] Serialized 14 byte log entry
0709 13:02:09.970007 (+754624us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:09.970090 (+    83us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:02:09.970174 (+    84us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:09.970282 (+   108us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:09.970665 (+   383us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":339743,"prepare.queue_time_us":33514,"prepare.run_cpu_time_us":103,"prepare.run_wall_time_us":103,"thread_start_us":858454,"threads_started":2,"wal-append.queue_time_us":543430}
W20260709 13:02:09.996807  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 56826.5us
I20260709 13:02:09.997727  1718 leader_election.cc:304] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:09.996876  7307 consensus_queue.cc:1059] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
W20260709 13:02:09.971794  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 200067us
I20260709 13:02:10.002338  1571 leader_election.cc:304] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:02:10.019723  1754 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55848 (request call id 3121) took 2575 ms. Trace:
I20260709 13:02:09.990366  8395 raft_consensus.cc:3055] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 LEADER]: Stepping down as leader of term 18
I20260709 13:02:10.021173  1754 rpcz_store.cc:276] 0709 13:02:07.443890 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:08.505142 (+1061252us) service_pool.cc:224] Handling call
0709 13:02:08.505589 (+   447us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:08.505901 (+   312us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 13:02:08.506333 (+   432us) log.cc:844] Serialized 14 byte log entry
0709 13:02:10.019258 (+1512925us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:10.019341 (+    83us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:02:10.019408 (+    67us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:10.019503 (+    95us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:10.019644 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":655663,"prepare.queue_time_us":2767,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":50,"raft.queue_time_us":704161,"raft.run_cpu_time_us":252,"raft.run_wall_time_us":24778,"thread_start_us":856607,"threads_started":2,"wal-append.queue_time_us":1086225}
W20260709 13:02:10.024850  1718 connection.cc:256] RPC call timeout handler was delayed by -1.0636101890000305s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1514s for a response
I20260709 13:02:10.026252  1718 leader_election.cc:304] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:10.029501  8403 leader_election.cc:290] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 16 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:10.027204  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:10.036981  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:10.038748  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:10.041481  1675 raft_consensus.cc:2410] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for term 20 because replica has last-logged OpId of term: 18 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 12.
I20260709 13:02:10.020241  8145 raft_consensus.cc:2804] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:02:10.044250  8565 raft_consensus.cc:3055] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 13:02:10.044898  8456 raft_consensus.cc:697] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 16 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
W20260709 13:02:10.053718  1719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:10.048661  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 46433.3us
I20260709 13:02:09.995592  7745 raft_consensus.cc:515] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:10.020598  8300 raft_consensus.cc:493] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:09.996325  8461 raft_consensus.cc:515] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:10.055588  1571 leader_election.cc:304] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:02:10.003957  7728 consensus_queue.cc:1059] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 18.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.042s
I20260709 13:02:10.065893  1571 leader_election.cc:304] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:02:10.066699  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:10.067337  1571 leader_election.cc:304] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:02:10.055074  8481 raft_consensus.cc:493] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:10.068423  8481 raft_consensus.cc:515] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:10.069496  8190 leader_election.cc:290] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:09.891389  1680 rpcz_store.cc:276] 0709 13:02:08.871416 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:09.723259 (+851843us) service_pool.cc:224] Handling call
0709 13:02:09.723630 (+   371us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:09.723946 (+   316us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 13:02:09.736822 (+ 12876us) log.cc:844] Serialized 14 byte log entry
0709 13:02:09.882526 (+145704us) log.cc:844] Serialized 14 byte log entry
0709 13:02:09.886489 (+  3963us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:09.886542 (+    53us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:02:09.886583 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:09.886664 (+    81us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:09.886785 (+   121us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":7551,"prepare.queue_time_us":14993,"prepare.run_cpu_time_us":85,"prepare.run_wall_time_us":4047,"raft.queue_time_us":12270,"raft.run_cpu_time_us":156,"raft.run_wall_time_us":6086,"thread_start_us":137286,"threads_started":1}
I20260709 13:02:10.071743  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:10.071959  7745 leader_election.cc:290] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:10.072371  1680 raft_consensus.cc:2468] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 17.
I20260709 13:02:10.072647  1602 raft_consensus.cc:1240] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 12. Current term is 13. Ops: []
I20260709 13:02:10.073300  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:10.073875  1680 raft_consensus.cc:2468] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 16.
W20260709 13:02:10.076105  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 49998.5us
I20260709 13:02:10.077347  1718 leader_election.cc:304] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:10.077586  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 15 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:10.022528  8395 raft_consensus.cc:740] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:02:10.046315  8145 raft_consensus.cc:493] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:10.051455  8565 raft_consensus.cc:740] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:02:10.067373  1675 raft_consensus.cc:2468] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 19.
I20260709 13:02:10.154839  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 19 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:10.081452  1571 leader_election.cc:304] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
W20260709 13:02:10.153805  1719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:10.152062  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
W20260709 13:02:10.157313  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 76010.5us
I20260709 13:02:10.158476  1571 leader_election.cc:304] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:02:10.053579  8558 raft_consensus.cc:2749] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:02:10.161289  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:10.161623  8565 consensus_queue.cc:260] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:10.163018  8565 raft_consensus.cc:3060] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:02:10.163760  1675 raft_consensus.cc:1240] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 17. Current term is 18. Ops: []
I20260709 13:02:10.165365  8481 leader_election.cc:290] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:10.166021  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:10.166952  1753 raft_consensus.cc:2468] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 16.
I20260709 13:02:10.166711  8395 consensus_queue.cc:260] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.10, Last appended by leader: 10, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:10.168174  8395 raft_consensus.cc:3060] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:02:10.064653  8300 raft_consensus.cc:515] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:10.153388  8145 raft_consensus.cc:3060] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:02:10.172420  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:10.173482  1752 raft_consensus.cc:2393] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 16: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:02:10.173871  8562 raft_consensus.cc:493] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:02:10.177548  8562 raft_consensus.cc:515] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:10.175153  8454 raft_consensus.cc:2804] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:02:10.175280  8402 raft_consensus.cc:493] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:02:10.179107  8454 raft_consensus.cc:493] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:02:10.179751  8454 raft_consensus.cc:3060] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:02:10.179502  8402 raft_consensus.cc:515] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:10.176105  7728 raft_consensus.cc:493] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:10.181600  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:10.182021  7728 raft_consensus.cc:515] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:10.183537  1752 raft_consensus.cc:3060] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:02:10.186261  8560 consensus_queue.cc:1059] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 13.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 13:02:10.117790  8127 raft_consensus.cc:2749] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:02:10.055891  8500 raft_consensus.cc:515] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:10.117429  8517 raft_consensus.cc:2749] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:02:10.191556  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:10.192631  1749 raft_consensus.cc:2468] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 16.
W20260709 13:02:10.195163  1676 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57092 (request call id 1500) took 1366 ms (client timeout 1636 ms). Trace:
W20260709 13:02:10.195734  1676 rpcz_store.cc:269] 0709 13:02:08.829009 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:08.920234 (+ 91225us) service_pool.cc:224] Handling call
0709 13:02:10.194891 (+1274657us) spinlock_profiling.cc:243] Waited 289 ms on lock 0x7b54001f5254. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:02:10.195081 (+   190us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":664823680}
I20260709 13:02:10.198247  8477 leader_election.cc:290] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:10.191726  8471 raft_consensus.cc:493] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:10.210119  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 132943us
I20260709 13:02:10.212137  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 17 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:10.216683  1718 leader_election.cc:304] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:10.213584  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:10.222185  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:10.223141  1749 raft_consensus.cc:2468] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 20.
I20260709 13:02:10.229614  1673 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52766 (request call id 3116) took 1357 ms. Trace:
W20260709 13:02:10.231098  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 72807.8us
I20260709 13:02:10.230082  1673 rpcz_store.cc:276] 0709 13:02:08.872335 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:09.872311 (+999976us) service_pool.cc:224] Handling call
0709 13:02:09.872756 (+   445us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:02:09.881410 (+  8654us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:09.881441 (+    31us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:02:09.885838 (+  4397us) log.cc:844] Serialized 52 byte log entry
0709 13:02:10.024235 (+138397us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:02:10.059628 (+ 35393us) log.cc:844] Serialized 14 byte log entry
0709 13:02:10.060248 (+   620us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:10.060413 (+   165us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:10.179049 (+118636us) spinlock_profiling.cc:243] Waited 1.78 ms on lock 0x7b180044e660. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f42630753f3 00007f42630777c1 00007f4263077779 00007f4263077709 00007f42630776d1 00007f42630769cd 000000000038f354 000000000038f199 00007f426305883a
0709 13:02:10.214093 (+ 35044us) raft_consensus.cc:1685] finished
0709 13:02:10.229339 (+ 15246us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:10.229538 (+   199us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8136,"prepare.queue_time_us":37222,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":44,"raft.queue_time_us":8527,"spinlock_wait_cycles":45383040,"thread_start_us":63700,"threads_started":1,"wal-append.queue_time_us":143949}
I20260709 13:02:10.232897  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 16 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:10.234017  1673 raft_consensus.cc:2410] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for term 16 because replica has last-logged OpId of term: 14 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 8.
I20260709 13:02:10.120249  8372 raft_consensus.cc:2804] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:02:10.231794  1571 leader_election.cc:304] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 13:02:10.242827  1571 leader_election.cc:304] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:02:10.244145  1571 leader_election.cc:304] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:02:10.245267  1571 leader_election.cc:304] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:02:10.240650  1718 leader_election.cc:304] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:10.247548  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:10.248754  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:10.201401  8414 raft_consensus.cc:2749] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:02:10.251636  1673 raft_consensus.cc:2468] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 17.
I20260709 13:02:10.253707  1749 raft_consensus.cc:1217] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Deduplicated request from leader. Original: 14.10->[16.11-18.12]   Dedup: 16.11->[18.12-18.12]
I20260709 13:02:10.264770  1749 raft_consensus.cc:1240] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 18. Current term is 19. Ops: [16.11-18.12]
I20260709 13:02:10.265290  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:10.266032  1673 raft_consensus.cc:2468] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 16.
I20260709 13:02:10.214524  8524 raft_consensus.cc:493] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:10.279866  1673 raft_consensus.cc:1275] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 21 index: 13. (index mismatch)
I20260709 13:02:10.280449  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:10.229070  8481 raft_consensus.cc:493] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:10.287117  1673 raft_consensus.cc:1275] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 13 index: 9. Preceding OpId from leader: term: 14 index: 9. (term mismatch)
I20260709 13:02:10.286923  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:10.287525  1673 pending_rounds.cc:85] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d: Aborting all ops after (but not including) 8
I20260709 13:02:10.287765  1673 pending_rounds.cc:107] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d: Aborting uncommitted NO_OP operation due to leader change: 13.9
I20260709 13:02:10.288085  1673 raft_consensus.cc:2889] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:02:10.251219  1718 leader_election.cc:304] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:10.214813  8477 raft_consensus.cc:493] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:02:10.119855  8371 consensus_queue.cc:1048] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.129s
I20260709 13:02:10.269042  1752 raft_consensus.cc:2468] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 21.
I20260709 13:02:10.235100  8517 raft_consensus.cc:2749] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
W20260709 13:02:10.284394  1570 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:10.248732  8372 raft_consensus.cc:493] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:02:10.250823  8127 raft_consensus.cc:2804] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Leader pre-election won for term 17
W20260709 13:02:10.305127  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 59956us
I20260709 13:02:10.254165  8379 raft_consensus.cc:2749] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:02:10.250679  8560 raft_consensus.cc:2749] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
W20260709 13:02:10.303493  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 116784us
I20260709 13:02:10.267774  8516 raft_consensus.cc:493] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:10.315071  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 17 candidate_status { last_received { term: 16 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:10.302068  8414 raft_consensus.cc:3060] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Advancing to term 16
W20260709 13:02:10.311731  8404 consensus_meta.cc:287] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.572s	user 0.012s	sys 0.004s
I20260709 13:02:10.280579  8294 raft_consensus.cc:2749] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:02:10.279970  8500 leader_election.cc:290] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 16 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:10.326265  1675 raft_consensus.cc:1240] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 14. Current term is 15. Ops: []
I20260709 13:02:10.326862  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:10.250689  8568 raft_consensus.cc:2804] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Leader pre-election won for term 18
W20260709 13:02:10.331634  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 81203.6us
I20260709 13:02:10.235829  8471 raft_consensus.cc:515] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:10.310607  8344 raft_consensus.cc:2804] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:02:10.359380  1677 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52766 (request call id 3118) took 1486 ms. Trace:
I20260709 13:02:10.370323  1677 rpcz_store.cc:276] 0709 13:02:08.873006 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:09.872400 (+999394us) service_pool.cc:224] Handling call
0709 13:02:09.884386 (+ 11986us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:09.884640 (+   254us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:09.884664 (+    24us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:09.887378 (+  2714us) log.cc:844] Serialized 26 byte log entry
0709 13:02:10.215210 (+327832us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:02:10.215253 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:10.215324 (+    71us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:10.359075 (+143751us) raft_consensus.cc:1685] finished
0709 13:02:10.359114 (+    39us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:10.359301 (+   187us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":250388,"threads_started":1,"wal-append.queue_time_us":400832}
I20260709 13:02:10.340157  1749 raft_consensus.cc:3060] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:02:10.309633  8360 raft_consensus.cc:2749] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:02:10.320389  8372 raft_consensus.cc:3060] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:02:10.377223  1718 leader_election.cc:304] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 13:02:10.349839  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:10.255738  8456 consensus_queue.cc:237] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 14.13, Last appended by leader: 13, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:10.336184  8516 raft_consensus.cc:515] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:10.342752  8404 raft_consensus.cc:515] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:10.352998  2328 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39738 (request call id 1201) took 2271 ms (client timeout 2371 ms). Trace:
I20260709 13:02:10.306830  8477 raft_consensus.cc:515] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:10.384605  2328 rpcz_store.cc:269] 0709 13:02:08.081369 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:08.344312 (+262943us) service_pool.cc:224] Handling call
0709 13:02:10.352730 (+2008418us) spinlock_profiling.cc:243] Waited 690 ms on lock 0x7b540065ed54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:02:10.352914 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1586240256}
W20260709 13:02:10.353479  1717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:10.343755  8568 raft_consensus.cc:493] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:02:10.360287  1674 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57092 (request call id 1504) took 1530 ms. Trace:
I20260709 13:02:10.360522  8379 consensus_queue.cc:1048] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 13:02:10.376044  8371 consensus_queue.cc:1059] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.031s
I20260709 13:02:10.332755  8562 leader_election.cc:290] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:10.361527  7728 leader_election.cc:290] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:10.323350  8127 raft_consensus.cc:493] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:10.375386  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:10.305514  8481 raft_consensus.cc:515] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:10.294929  8524 raft_consensus.cc:515] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:10.406426  8516 leader_election.cc:290] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:10.407598  8371 raft_consensus.cc:3055] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 13:02:10.413232  8371 raft_consensus.cc:740] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:02:10.408109  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:10.415359  1751 raft_consensus.cc:2468] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 17.
I20260709 13:02:10.403537  8568 raft_consensus.cc:3060] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:02:10.416334  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:10.417032  1751 raft_consensus.cc:2468] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 17.
I20260709 13:02:10.417927  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:10.418784  8524 leader_election.cc:290] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:10.408780  8404 leader_election.cc:290] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 21 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:10.400871  1674 rpcz_store.cc:276] 0709 13:02:08.829870 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:09.122337 (+292467us) service_pool.cc:224] Handling call
0709 13:02:09.185987 (+ 63650us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:02:09.186661 (+   674us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:09.186705 (+    44us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:02:09.197693 (+ 10988us) log.cc:844] Serialized 52 byte log entry
0709 13:02:09.604610 (+406917us) raft_consensus.cc:1645] Marking committed up to 14
0709 13:02:09.684797 (+ 80187us) log.cc:844] Serialized 14 byte log entry
0709 13:02:09.761663 (+ 76866us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:09.761761 (+    98us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:10.355614 (+593853us) spinlock_profiling.cc:243] Waited 23.4 ms on lock 0x7b180034a0e0. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f42630753f3 00007f42630777c1 00007f4263077779 00007f4263077709 00007f42630776d1 00007f42630769cd 000000000038f354 000000000038f199 00007f426305883a
0709 13:02:10.359891 (+  4277us) raft_consensus.cc:1685] finished
0709 13:02:10.360043 (+   152us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:10.360210 (+   167us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":87004,"prepare.queue_time_us":67824,"prepare.run_cpu_time_us":10628,"prepare.run_wall_time_us":51722,"raft.queue_time_us":50511,"raft.run_cpu_time_us":370,"raft.run_wall_time_us":11886,"spinlock_wait_cycles":53790464,"thread_start_us":372528,"threads_started":1,"wal-append.queue_time_us":544504}
I20260709 13:02:10.409621  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:10.410091  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:10.421604  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:10.421898  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:10.410614  8127 raft_consensus.cc:3060] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:02:10.369261  8344 raft_consensus.cc:493] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:10.424492  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:10.430805  8344 raft_consensus.cc:3060] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:02:10.431382  1752 raft_consensus.cc:2393] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 21: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:02:10.432510  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:10.433876  1747 raft_consensus.cc:3060] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Advancing to term 19
W20260709 13:02:10.439677  8506 consensus_meta.cc:287] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.580s	user 0.014s	sys 0.000s
I20260709 13:02:10.440351  8506 raft_consensus.cc:515] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:10.440982  1751 raft_consensus.cc:3060] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:02:10.360296  8471 leader_election.cc:290] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:10.410830  1671 raft_consensus.cc:1240] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 19. Current term is 21. Ops: []
I20260709 13:02:10.411486  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:10.444406  1746 raft_consensus.cc:2393] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 16: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:02:10.445070  8477 leader_election.cc:290] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:10.445626  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:10.425439  8504 leader_election.cc:290] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:10.449132  8481 leader_election.cc:290] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:10.450027  1746 raft_consensus.cc:2468] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 20.
I20260709 13:02:10.451717  1746 raft_consensus.cc:1275] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 18 index: 14. Preceding OpId from leader: term: 19 index: 15. (index mismatch)
W20260709 13:02:10.452878  8521 log.cc:927] Time spent T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 1.237s	user 0.002s	sys 0.002s
W20260709 13:02:10.463244  2337 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49082 (request call id 1219) took 2942 ms (client timeout 1984 ms). Trace:
W20260709 13:02:10.463724  2337 rpcz_store.cc:269] 0709 13:02:07.520357 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:07.521664 (+  1307us) service_pool.cc:224] Handling call
0709 13:02:10.462969 (+2941305us) spinlock_profiling.cc:243] Waited 1.86 s on lock 0x7b54004f88d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:02:10.463150 (+   181us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4272735488}
I20260709 13:02:10.414597  8371 consensus_queue.cc:260] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.11, Last appended by leader: 11, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:10.477161  8371 raft_consensus.cc:3060] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:02:10.479396  8506 leader_election.cc:290] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 18 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:10.398926  8461 leader_election.cc:290] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:10.458106  7745 raft_consensus.cc:3055] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 LEADER]: Stepping down as leader of term 18
I20260709 13:02:10.485409  7745 raft_consensus.cc:740] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
W20260709 13:02:10.485847  8525 log.cc:927] Time spent T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 1.003s	user 0.002s	sys 0.004s
I20260709 13:02:10.486781  7745 consensus_queue.cc:260] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 18.9, Last appended by leader: 9, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:10.488456  7745 raft_consensus.cc:3060] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:02:10.473707  1646 leader_election.cc:304] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:10.497527  1646 leader_election.cc:304] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:10.499173  8468 raft_consensus.cc:2804] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:02:10.475575  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:10.499711  8468 raft_consensus.cc:493] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
W20260709 13:02:10.500029  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:10.500265  8468 raft_consensus.cc:3060] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:02:10.503216  8506 raft_consensus.cc:2804] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:02:10.503720  8506 raft_consensus.cc:493] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Starting leader election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:02:10.504216  8506 raft_consensus.cc:3060] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:02:10.507340  1679 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57092 (request call id 1498) took 1678 ms. Trace:
I20260709 13:02:10.507763  1679 rpcz_store.cc:276] 0709 13:02:08.828546 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:08.942233 (+113687us) service_pool.cc:224] Handling call
0709 13:02:08.973804 (+ 31571us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:09.195076 (+221272us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:09.195111 (+    35us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:09.195161 (+    50us) raft_consensus.cc:1645] Marking committed up to 14
0709 13:02:09.195202 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:10.507037 (+1311835us) spinlock_profiling.cc:243] Waited 221 ms on lock 0x7b54001e82d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:02:10.507100 (+    63us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:10.507256 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":508526464}
W20260709 13:02:10.511170  8472 consensus_meta.cc:287] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.549s	user 0.012s	sys 0.005s
I20260709 13:02:10.511801  8472 raft_consensus.cc:515] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:10.450333  8402 leader_election.cc:290] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:10.500275  1746 raft_consensus.cc:2468] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 19.
W20260709 13:02:10.519881  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 143047us
I20260709 13:02:10.522013  1749 raft_consensus.cc:2468] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 19.
W20260709 13:02:10.523172  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 674751us
I20260709 13:02:10.535899  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:10.536170  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 16 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:10.537048  1674 raft_consensus.cc:2393] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 16: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:02:10.537928  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
W20260709 13:02:10.530418  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 611122us
I20260709 13:02:10.531322  1646 leader_election.cc:304] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 13:02:10.523350  8461 raft_consensus.cc:2749] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:02:10.524538  8504 raft_consensus.cc:493] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:10.539922  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:02:10.542433  2301 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:10.543308  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:10.543905  1644 leader_election.cc:304] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:02:10.556216  1718 leader_election.cc:304] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:10.568437  1749 raft_consensus.cc:2468] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 19.
I20260709 13:02:10.540642  1747 raft_consensus.cc:2468] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 19.
I20260709 13:02:10.546482  1680 raft_consensus.cc:2393] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 21: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:02:10.550734  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:10.572342  1718 leader_election.cc:304] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:10.565771  1746 raft_consensus.cc:2468] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 19.
I20260709 13:02:10.546217  8404 raft_consensus.cc:493] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:10.556485  8513 raft_consensus.cc:493] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:10.552352  1751 raft_consensus.cc:2468] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 15.
I20260709 13:02:10.550249  8490 raft_consensus.cc:493] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:10.567045  8537 raft_consensus.cc:2804] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:02:10.574620  1718 leader_election.cc:304] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:10.570690  8508 raft_consensus.cc:2804] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:02:10.578384  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:10.585616  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:02:10.610271  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 289944us
I20260709 13:02:10.612057  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:10.615084  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:10.619227  1674 raft_consensus.cc:2368] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 20. Current term is 21.
I20260709 13:02:10.615515  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:10.600982  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:10.622037  2335 raft_consensus.cc:2468] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 19.
I20260709 13:02:10.623440  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:10.603632  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 16 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:10.603204  2331 raft_consensus.cc:1275] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 18 index: 15. Preceding OpId from leader: term: 19 index: 15. (term mismatch)
I20260709 13:02:10.626394  2331 pending_rounds.cc:85] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1: Aborting all ops after (but not including) 14
I20260709 13:02:10.626724  2331 pending_rounds.cc:107] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1: Aborting uncommitted NO_OP operation due to leader change: 18.15
I20260709 13:02:10.627059  2331 raft_consensus.cc:2889] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:02:10.628722  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:10.610061  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:10.630846  2329 raft_consensus.cc:2410] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for term 16 because replica has last-logged OpId of term: 15 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 13.
W20260709 13:02:10.579406  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 605087us
I20260709 13:02:10.635995  1672 raft_consensus.cc:2393] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 19: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:02:10.637037  2328 raft_consensus.cc:2393] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 16: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:02:10.550789  8145 raft_consensus.cc:515] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:10.645638  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:10.647709  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:10.649858  1571 leader_election.cc:304] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:02:10.650326  2332 raft_consensus.cc:2468] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 19.
I20260709 13:02:10.648368  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:10.651005  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:10.651696  2330 raft_consensus.cc:2410] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for term 20 because replica has last-logged OpId of term: 18 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 12.
I20260709 13:02:10.651866  2329 raft_consensus.cc:2468] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 16.
I20260709 13:02:10.653272  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:10.655604  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:10.656721  2331 raft_consensus.cc:2393] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 16: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:02:10.652788  2335 raft_consensus.cc:2468] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 19.
I20260709 13:02:10.568656  8504 raft_consensus.cc:515] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:10.577270  8481 raft_consensus.cc:2804] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:02:10.663494  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 18 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:10.582476  8338 raft_consensus.cc:493] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:10.578042  8471 raft_consensus.cc:2804] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:02:10.610105  8508 raft_consensus.cc:493] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:02:10.671701  8454 raft_consensus.cc:515] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:10.672832  8154 raft_consensus.cc:2749] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:02:10.673398  1571 leader_election.cc:304] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:02:10.674522  8481 raft_consensus.cc:493] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:10.675753  8472 leader_election.cc:290] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 20 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:10.616664  8565 consensus_queue.cc:1048] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.071s
I20260709 13:02:10.585891  8404 raft_consensus.cc:515] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:10.680064  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:10.689196  8471 raft_consensus.cc:493] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:10.717229  1605 raft_consensus.cc:1275] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 16 index: 8. Preceding OpId from leader: term: 19 index: 10. (index mismatch)
I20260709 13:02:10.681136  8395 consensus_queue.cc:1048] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
W20260709 13:02:10.683424  1717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:10.610749  8497 raft_consensus.cc:493] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:10.720378  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:10.686611  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:10.599073  8537 raft_consensus.cc:493] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:10.580642  8496 raft_consensus.cc:493] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:02:10.557101  1646 leader_election.cc:304] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:10.696344  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:10.633378  8524 consensus_queue.cc:1048] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.071s
I20260709 13:02:10.696597  8190 raft_consensus.cc:493] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:10.705775  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:10.705111  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:10.705731  1596 raft_consensus.cc:1240] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 19. Current term is 20. Ops: []
I20260709 13:02:10.617821  8461 raft_consensus.cc:2804] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:02:10.704746  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:10.707304  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:10.710748  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 19 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:10.755512  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:10.678959  8145 leader_election.cc:290] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:10.759821  1680 raft_consensus.cc:1275] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 15 index: 9. (term mismatch)
I20260709 13:02:10.598696  8513 raft_consensus.cc:515] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:10.713191  2332 raft_consensus.cc:3060] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:02:10.680688  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:10.716746  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:10.719106  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:10.719653  8471 raft_consensus.cc:3060] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:02:10.719813  1753 raft_consensus.cc:2393] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 19: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:02:10.721200  1604 raft_consensus.cc:2468] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 19.
I20260709 13:02:10.763828  1598 raft_consensus.cc:2468] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 16.
I20260709 13:02:10.699402  8508 raft_consensus.cc:3060] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:02:10.738251  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 15 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:10.748682  8537 raft_consensus.cc:3060] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:02:10.707628  8481 raft_consensus.cc:3060] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:02:10.752234  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:10.708022  8358 raft_consensus.cc:3060] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:02:10.756676  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:10.757825  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:10.755203  1601 raft_consensus.cc:2468] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 18.
I20260709 13:02:10.759143  8461 raft_consensus.cc:493] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:10.776687  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:10.759871  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:10.778644  1597 raft_consensus.cc:2468] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 19.
I20260709 13:02:10.780231  1597 raft_consensus.cc:1275] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
I20260709 13:02:10.781733  1597 raft_consensus.cc:1275] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 15 index: 9. (index mismatch)
I20260709 13:02:10.760293  1680 pending_rounds.cc:85] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d: Aborting all ops after (but not including) 8
I20260709 13:02:10.782644  1680 pending_rounds.cc:107] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d: Aborting uncommitted NO_OP operation due to leader change: 14.9
I20260709 13:02:10.782989  1680 raft_consensus.cc:2889] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:02:10.711879  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:10.590727  8490 raft_consensus.cc:515] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:10.691488  8338 raft_consensus.cc:515] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:10.767040  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:10.768353  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:10.769135  1600 raft_consensus.cc:2410] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for term 15 because replica has last-logged OpId of term: 14 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 8.
I20260709 13:02:10.787422  1599 raft_consensus.cc:2368] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 16. Current term is 17.
W20260709 13:02:10.738729  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 181798us
I20260709 13:02:10.758828  8514 raft_consensus.cc:2804] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:02:10.737540  8568 raft_consensus.cc:515] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:10.788192  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:10.804718  1603 raft_consensus.cc:2468] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 18.
I20260709 13:02:10.804978  2330 raft_consensus.cc:1240] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 19. Current term is 20. Ops: []
W20260709 13:02:10.788174  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 82732.5us
I20260709 13:02:10.739202  8127 raft_consensus.cc:515] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:10.807458  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:10.808308  1678 raft_consensus.cc:2393] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 19: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:02:10.803233  2335 raft_consensus.cc:1240] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 13. Current term is 14. Ops: []
I20260709 13:02:10.802771  1646 leader_election.cc:304] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:10.785436  1597 raft_consensus.cc:1275] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 17 index: 10. (index mismatch)
I20260709 13:02:10.752825  8497 raft_consensus.cc:515] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:10.762022  8190 raft_consensus.cc:515] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:10.758622  8496 raft_consensus.cc:515] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:10.837293  1717 leader_election.cc:304] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:10.814352  2332 raft_consensus.cc:2468] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 21.
I20260709 13:02:10.831722  8338 leader_election.cc:290] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:10.835036  1677 raft_consensus.cc:2468] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 18.
I20260709 13:02:10.834996  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 16 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:10.856673  2331 raft_consensus.cc:2393] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 19: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:02:10.836633  1754 raft_consensus.cc:1240] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 15. Current term is 16. Ops: []
I20260709 13:02:10.789594  8514 raft_consensus.cc:493] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:10.851238  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:10.852713  8190 leader_election.cc:290] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:10.858187  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:10.786336  8461 raft_consensus.cc:3060] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:02:10.867650  1752 raft_consensus.cc:2393] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 18: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
W20260709 13:02:10.871583  1719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:10.879379  1679 raft_consensus.cc:2468] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 16.
I20260709 13:02:10.889899  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:10.890110  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:11.184165  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:11.006377  2328 raft_consensus.cc:2468] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 14.
I20260709 13:02:10.891196  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
W20260709 13:02:11.005031  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 392275us
I20260709 13:02:11.188676  1751 raft_consensus.cc:3060] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Advancing to term 17
W20260709 13:02:11.010392  8468 consensus_meta.cc:287] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.509s	user 0.014s	sys 0.000s
I20260709 13:02:11.189508  8468 raft_consensus.cc:515] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } }
I20260709 13:02:10.889899  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:10.901217  8514 raft_consensus.cc:3060] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:02:11.190704  1599 raft_consensus.cc:2468] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 19.
I20260709 13:02:11.192080  8468 leader_election.cc:290] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 17 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:11.195390  1717 leader_election.cc:304] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 13:02:11.022832  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 211106us
I20260709 13:02:11.196933  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:11.197582  1646 leader_election.cc:304] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:11.198266  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:11.199576  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 18 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:02:10.944481  1570 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:11.184165  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:11.184705  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:11.204686  1596 raft_consensus.cc:2468] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 20.
I20260709 13:02:11.187337  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:11.211850  1597 raft_consensus.cc:2393] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 16: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:02:11.199990  1718 leader_election.cc:304] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:02:11.215096  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:02:11.219677  2298 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.585s	user 0.021s	sys 0.031s
W20260709 13:02:11.220139  2298 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.585s	user 0.021s	sys 0.031s
W20260709 13:02:11.220934  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 586343us
I20260709 13:02:11.221447  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:11.222847  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:11.223941  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:11.224243  8190 consensus_queue.cc:1048] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 13:02:11.220561  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:11.227835  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 16 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:11.230863  2298 leader_election.cc:304] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:11.231437  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:11.232699  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:11.232517  8334 raft_consensus.cc:2749] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:02:11.227986  1717 leader_election.cc:304] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:11.233950  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 16 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:11.235944  1679 raft_consensus.cc:1275] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 15 index: 9. Preceding OpId from leader: term: 17 index: 10. (index mismatch)
I20260709 13:02:10.835084  8477 raft_consensus.cc:2804] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:02:11.245142  1603 raft_consensus.cc:2468] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 14.
I20260709 13:02:11.238108  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:11.246228  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:11.237079  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:02:11.243538  1571 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:11.247398  1596 raft_consensus.cc:2410] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for term 14 because replica has last-logged OpId of term: 13 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 10.
I20260709 13:02:11.249295  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:10.892221  8524 raft_consensus.cc:2749] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 13:02:11.252389  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:11.257221  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:10.996238  8504 leader_election.cc:290] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:11.258376  1596 raft_consensus.cc:2468] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 16.
I20260709 13:02:11.264663  8597 raft_consensus.cc:493] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:11.264998  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:11.265230  8597 raft_consensus.cc:515] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:11.265632  1596 raft_consensus.cc:2468] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 19.
W20260709 13:02:11.266947  8586 log.cc:927] Time spent T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.433s	user 0.000s	sys 0.001s
I20260709 13:02:11.262049  1751 raft_consensus.cc:2468] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 17.
I20260709 13:02:11.269438  1754 raft_consensus.cc:2393] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 16: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:02:11.248178  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:11.247588  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:11.260987  1643 leader_election.cc:304] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:02:11.271189  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:11.271335  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:11.271857  1596 raft_consensus.cc:2468] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 14.
I20260709 13:02:11.272562  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:11.272485  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:11.236068  8585 raft_consensus.cc:493] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:11.245380  8477 raft_consensus.cc:493] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:11.240851  8490 leader_election.cc:290] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:11.275538  1571 leader_election.cc:304] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:11.236141  8404 leader_election.cc:290] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:11.258140  8145 raft_consensus.cc:2749] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:02:11.235765  8468 consensus_queue.cc:1048] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.041s
I20260709 13:02:11.271175  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:11.313202  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:02:11.314056  8587 log.cc:927] Time spent T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.278s	user 0.000s	sys 0.001s
I20260709 13:02:11.238968  8588 raft_consensus.cc:493] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:11.315398  8588 raft_consensus.cc:515] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:11.317824  8588 leader_election.cc:290] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:11.280823  8524 raft_consensus.cc:2749] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 13:02:11.307539  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:11.291266  8477 raft_consensus.cc:3060] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:02:11.273545  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:11.306586  8513 leader_election.cc:290] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:11.331627  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:11.261952  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 17 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:11.333477  1600 raft_consensus.cc:2410] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for term 17 because replica has last-logged OpId of term: 16 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 7.
I20260709 13:02:11.311332  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:11.331575  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 17 candidate_status { last_received { term: 16 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:11.325989  1601 raft_consensus.cc:2468] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 19.
I20260709 13:02:11.335511  2330 raft_consensus.cc:2468] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 19.
I20260709 13:02:11.335234  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:11.336853  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:11.340791  1601 raft_consensus.cc:2468] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 20.
I20260709 13:02:11.338312  1603 raft_consensus.cc:2468] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 19.
I20260709 13:02:11.344024  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:02:11.344452  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 107388us
I20260709 13:02:11.344743  1596 raft_consensus.cc:2468] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 21.
I20260709 13:02:11.356596  2298 leader_election.cc:304] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:02:11.356774  2335 raft_consensus.cc:2468] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 16.
I20260709 13:02:11.366779  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:11.367512  1601 raft_consensus.cc:3060] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:02:11.368883  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:11.291643  8585 raft_consensus.cc:515] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:11.374766  8334 raft_consensus.cc:493] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:11.375353  8334 raft_consensus.cc:515] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:11.376766  8585 leader_election.cc:290] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:11.377588  8334 leader_election.cc:290] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:11.367324  8497 leader_election.cc:290] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:11.378890  1599 raft_consensus.cc:2393] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 20: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:02:11.330217  8145 raft_consensus.cc:2749] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 13:02:11.388197  8497 raft_consensus.cc:493] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:11.388751  8497 raft_consensus.cc:515] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:11.391011  8497 leader_election.cc:290] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:11.343137  8496 leader_election.cc:290] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:11.392473  8514 raft_consensus.cc:515] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:02:11.400264  8506 consensus_meta.cc:287] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.895s	user 0.014s	sys 0.000s
I20260709 13:02:11.400768  8506 raft_consensus.cc:515] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:11.401516  1601 raft_consensus.cc:2468] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 21.
I20260709 13:02:11.364118  8568 leader_election.cc:290] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 18 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:11.407744  8514 leader_election.cc:290] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 18 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:11.404350  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:11.405148  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 17 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:02:11.409104  8414 consensus_meta.cc:287] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 1.068s	user 0.014s	sys 0.000s
W20260709 13:02:11.386375  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 158655us
I20260709 13:02:11.409044  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 17 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:02:11.410009  8471 consensus_meta.cc:287] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.637s	user 0.007s	sys 0.000s
I20260709 13:02:11.409951  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:11.410310  2298 leader_election.cc:304] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:11.343648  8504 raft_consensus.cc:2804] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:02:11.353029  8127 leader_election.cc:290] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 17 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:11.269454  8154 consensus_queue.cc:1059] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.097s
W20260709 13:02:11.412704  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 137365us
W20260709 13:02:11.405851  8372 consensus_meta.cc:287] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.971s	user 0.009s	sys 0.002s
I20260709 13:02:11.269552  8338 consensus_queue.cc:1059] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 12.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.097s
I20260709 13:02:11.416777  1717 leader_election.cc:304] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:11.428223  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 16 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:11.442034  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:11.441601  2298 leader_election.cc:304] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:11.421981  1597 raft_consensus.cc:2468] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 16.
I20260709 13:02:11.410255  8190 raft_consensus.cc:493] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:11.380424  8591 raft_consensus.cc:2764] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Leader pre-election decision vote started in defunct term 13: won
I20260709 13:02:11.425972  8395 raft_consensus.cc:493] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:11.471683  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:11.427170  8585 raft_consensus.cc:2804] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Leader election won for term 17
I20260709 13:02:11.455054  1717 leader_election.cc:400] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 election: Vote denied by peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445) with higher term. Message: Invalid argument: T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 16. Current term is 17.
W20260709 13:02:11.470322  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 272949us
I20260709 13:02:11.496985  1717 leader_election.cc:403] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 election: Cancelling election due to peer responding with higher term
I20260709 13:02:11.424952  8414 raft_consensus.cc:2749] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20260709 13:02:11.483609  8601 log.cc:927] Time spent T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.154s	user 0.000s	sys 0.000s
I20260709 13:02:11.483157  8477 raft_consensus.cc:515] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:11.484161  8597 leader_election.cc:290] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:11.484439  8190 raft_consensus.cc:515] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:11.485716  2298 leader_election.cc:304] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:02:11.486713  1597 raft_consensus.cc:1275] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 13 index: 12. Preceding OpId from leader: term: 14 index: 13. (index mismatch)
I20260709 13:02:11.491171  8300 leader_election.cc:290] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:11.438755  8334 raft_consensus.cc:2749] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:02:11.426985  8504 raft_consensus.cc:493] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:11.497406  1646 leader_election.cc:304] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:11.503296  1598 raft_consensus.cc:2468] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 16.
I20260709 13:02:11.450285  8490 raft_consensus.cc:2749] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:02:11.452126  8568 consensus_queue.cc:1048] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.052s
W20260709 13:02:11.499322  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 238606us
I20260709 13:02:11.451033  8602 raft_consensus.cc:2764] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Leader pre-election decision vote started in defunct term 18: won
I20260709 13:02:11.455724  8496 raft_consensus.cc:2749] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 13:02:11.435407  8471 raft_consensus.cc:515] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:11.449602  8372 raft_consensus.cc:515] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:11.509187  1646 leader_election.cc:304] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:11.520696  1646 leader_election.cc:304] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:11.521148  8608 raft_consensus.cc:2804] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Leader election won for term 19
W20260709 13:02:11.521270  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 321504us
I20260709 13:02:11.521683  1646 leader_election.cc:304] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
W20260709 13:02:11.500236  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 58523.4us
I20260709 13:02:11.521857  8608 raft_consensus.cc:697] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 19 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:02:11.453078  8145 consensus_queue.cc:1048] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.062s
I20260709 13:02:11.522939  1717 leader_election.cc:304] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:11.504734  8585 raft_consensus.cc:697] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:02:11.503383  8605 raft_consensus.cc:2749] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Leader election lost for term 16. Reason: Vote denied by peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445) with higher term. Message: Invalid argument: T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 16. Current term is 17.
I20260709 13:02:11.516068  1643 leader_election.cc:304] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:02:11.509784  8504 raft_consensus.cc:3060] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:02:11.510722  8338 raft_consensus.cc:2804] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Leader pre-election won for term 17
W20260709 13:02:11.514961  8358 consensus_meta.cc:287] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.729s	user 0.012s	sys 0.001s
I20260709 13:02:11.526183  1718 leader_election.cc:304] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:02:11.528329  1717 leader_election.cc:304] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:02:11.498034  8395 raft_consensus.cc:515] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:11.515336  8300 raft_consensus.cc:2804] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Leader election won for term 19
I20260709 13:02:11.532717  1750 raft_consensus.cc:2393] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 19: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:02:11.533257  1718 leader_election.cc:304] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:02:11.534334  1746 raft_consensus.cc:2368] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 for earlier term 16. Current term is 17.
W20260709 13:02:11.525575  8537 consensus_meta.cc:287] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.737s	user 0.009s	sys 0.004s
I20260709 13:02:11.536706  1643 leader_election.cc:304] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:11.529956  8456 raft_consensus.cc:2749] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:02:11.537756  8602 raft_consensus.cc:2804] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:02:11.538301  8602 raft_consensus.cc:493] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:11.538708  8602 raft_consensus.cc:3060] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:02:11.526981  8468 raft_consensus.cc:2804] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:02:11.540890  8468 raft_consensus.cc:493] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Starting leader election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:02:11.541280  8468 raft_consensus.cc:3060] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:02:11.541376  8581 consensus_queue.cc:1048] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 13:02:11.543059  8581 raft_consensus.cc:2749] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20260709 13:02:11.544427  8508 consensus_meta.cc:287] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.767s	user 0.012s	sys 0.000s
I20260709 13:02:11.544549  8477 leader_election.cc:290] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 21 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:11.545092  8508 raft_consensus.cc:515] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:11.530069  8338 raft_consensus.cc:493] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:11.547051  8338 raft_consensus.cc:3060] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:02:11.547936  8190 leader_election.cc:290] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:11.535161  1717 leader_election.cc:304] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:11.519732  8506 leader_election.cc:290] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 20 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:11.534642  8358 raft_consensus.cc:2749] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:02:11.562255  8372 leader_election.cc:290] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 19 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:11.562474  8300 raft_consensus.cc:697] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:02:11.534546  8604 raft_consensus.cc:2749] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:02:11.563719  8600 raft_consensus.cc:3060] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:02:11.563791  8300 consensus_queue.cc:237] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.10, Last appended by leader: 10, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:11.566507  8537 raft_consensus.cc:515] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:11.567783  1676 raft_consensus.cc:2393] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 18: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:02:11.569473  2337 raft_consensus.cc:1275] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 14 index: 13. Preceding OpId from leader: term: 16 index: 14. (index mismatch)
W20260709 13:02:11.569726  8344 consensus_meta.cc:287] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 1.138s	user 0.013s	sys 0.001s
I20260709 13:02:11.572194  8360 raft_consensus.cc:493] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:11.572778  8360 raft_consensus.cc:515] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:11.576040  1643 leader_election.cc:304] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:11.576941  8597 raft_consensus.cc:2749] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:02:11.576642  8585 consensus_queue.cc:237] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.12, Last appended by leader: 12, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:11.577692  8597 raft_consensus.cc:2804] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:02:11.578089  8597 raft_consensus.cc:493] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:11.580318  1643 leader_election.cc:304] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:02:11.578426  8597 raft_consensus.cc:3060] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:02:11.582163  8608 consensus_queue.cc:237] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 18.9, Last appended by leader: 9, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:02:11.567823  8481 consensus_meta.cc:287] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.765s	user 0.009s	sys 0.002s
I20260709 13:02:11.586937  8481 raft_consensus.cc:515] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:11.589488  8481 leader_election.cc:290] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 18 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:11.570223  8344 raft_consensus.cc:515] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:11.590342  8471 leader_election.cc:290] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 17 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:11.591131  1749 raft_consensus.cc:1240] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 19. Current term is 20. Ops: []
W20260709 13:02:11.591519  8584 log.cc:927] Time spent T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.934s	user 0.003s	sys 0.000s
I20260709 13:02:11.591935  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:11.592589  2337 raft_consensus.cc:3060] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:02:11.592654  1749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55848 (request call id 3147) took 1121 ms. Trace:
I20260709 13:02:11.593397  1747 raft_consensus.cc:2368] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 17. Current term is 18.
I20260709 13:02:11.593875  1748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55848 (request call id 3125) took 1379 ms. Trace:
I20260709 13:02:11.595470  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:11.595877  1571 leader_election.cc:400] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 17 pre-election: Vote denied by peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693) with higher term. Message: Invalid argument: T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 17. Current term is 18.
I20260709 13:02:11.596457  1571 leader_election.cc:403] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 17 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:02:11.594260  1748 rpcz_store.cc:276] 0709 13:02:10.214450 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:10.215144 (+   694us) service_pool.cc:224] Handling call
0709 13:02:10.215466 (+   322us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:10.234778 (+ 19312us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:10.234825 (+    47us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:10.247371 (+ 12546us) log.cc:844] Serialized 26 byte log entry
0709 13:02:10.598620 (+351249us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:02:10.598667 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:10.598749 (+    82us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:11.590673 (+991924us) spinlock_profiling.cc:243] Waited 26.8 ms on lock 0x7b4400022b4c. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f4262f9bc85 00007f4262f9c6a2 00007f4265d0e1db 00007f4265d09aaf 00007f4265cfd86e 00007f4265cfcbb6 00007f4265cfc7d8 00007f4265d06221 00007f4265d061d9 00007f4265d06169
0709 13:02:11.593508 (+  2835us) raft_consensus.cc:1685] finished
0709 13:02:11.593550 (+    42us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:11.593805 (+   255us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":50,"raft.queue_time_us":13556,"raft.run_cpu_time_us":189,"raft.run_wall_time_us":12812,"spinlock_wait_cycles":61537664,"thread_start_us":289229,"threads_started":1,"wal-append.queue_time_us":389812}
I20260709 13:02:11.598671  8127 raft_consensus.cc:3060] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Advancing to term 18
I20260709 13:02:11.603684  8344 leader_election.cc:290] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 20 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:11.593096  1749 rpcz_store.cc:276] 0709 13:02:10.471073 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:10.583356 (+112283us) service_pool.cc:224] Handling call
0709 13:02:10.584124 (+   768us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:11.591517 (+1007393us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:11.592374 (+   857us) spinlock_profiling.cc:243] Waited 1.01 s on lock 0x7b54000195d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:02:11.592585 (+   211us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2316538112}
I20260709 13:02:11.614421  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:11.605288  8360 leader_election.cc:290] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:11.607303  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:11.618309  2328 raft_consensus.cc:3060] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:02:11.608297  2299 leader_election.cc:304] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:02:11.609257  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:11.562577  8395 leader_election.cc:290] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:11.610893  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:11.596109  1747 raft_consensus.cc:2468] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 18.
I20260709 13:02:11.612752  1673 raft_consensus.cc:1240] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 17. Current term is 18. Ops: []
I20260709 13:02:11.620819  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 20 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:11.531705  8513 raft_consensus.cc:2804] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Leader election won for term 15
I20260709 13:02:11.624048  1643 leader_election.cc:304] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:11.629788  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:11.635476  1643 leader_election.cc:304] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:02:11.553436  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 17 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:02:11.561787  7745 consensus_meta.cc:287] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 1.073s	user 0.017s	sys 0.000s
I20260709 13:02:11.597980  8461 raft_consensus.cc:515] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:11.630995  1646 leader_election.cc:304] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:11.639624  1747 raft_consensus.cc:2393] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 18: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:02:11.682916  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:11.656512  1643 leader_election.cc:304] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:02:11.683648  1748 raft_consensus.cc:3060] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:02:11.644542  1602 raft_consensus.cc:2468] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 19.
I20260709 13:02:11.643982  1600 raft_consensus.cc:1240] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 16. Current term is 17. Ops: []
I20260709 13:02:11.626482  8508 leader_election.cc:290] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 20 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:11.685216  1643 leader_election.cc:304] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:02:11.685400  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 18 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:11.686996  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:11.687727  1747 raft_consensus.cc:3060] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:02:11.691300  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:11.692061  2335 raft_consensus.cc:3060] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:02:11.702940  2335 raft_consensus.cc:2468] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 20.
I20260709 13:02:11.643857  8344 raft_consensus.cc:2749] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
W20260709 13:02:11.643672  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 108609us
I20260709 13:02:11.628994  8506 raft_consensus.cc:2804] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Leader pre-election won for term 15
W20260709 13:02:11.706774  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 75938.8us
I20260709 13:02:11.641103  8515 raft_consensus.cc:2749] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:02:11.711040  1747 raft_consensus.cc:2468] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 17.
I20260709 13:02:11.662355  1749 raft_consensus.cc:2468] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 20.
I20260709 13:02:11.638355  8591 raft_consensus.cc:493] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:11.718276  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:11.724964  1749 raft_consensus.cc:3060] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:02:11.718644  1646 leader_election.cc:304] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:11.628320  8358 consensus_queue.cc:1059] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 17.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.083s
I20260709 13:02:11.727012  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:11.711607  1717 leader_election.cc:304] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:11.728240  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:11.709054  8514 raft_consensus.cc:2804] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 13:02:11.640053  8617 raft_consensus.cc:2749] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:02:11.682682  8513 raft_consensus.cc:697] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:02:11.713521  8461 leader_election.cc:290] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 20 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:11.740300  1717 leader_election.cc:304] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:11.632630  8607 consensus_queue.cc:1059] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.094s
I20260709 13:02:11.711956  8508 raft_consensus.cc:2804] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:02:11.711534  8609 raft_consensus.cc:493] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:11.712388  8506 raft_consensus.cc:493] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Starting leader election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:02:11.644124  8334 consensus_queue.cc:1059] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 17.13, Next index: 14, Last known committed idx: 12, Time since last communication: 0.085s
I20260709 13:02:11.726321  8404 raft_consensus.cc:2804] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Leader election won for term 21
I20260709 13:02:11.744001  1749 raft_consensus.cc:2468] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 18.
I20260709 13:02:11.741151  1748 raft_consensus.cc:2468] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 17.
I20260709 13:02:11.793094  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:11.793834  1571 leader_election.cc:304] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:11.794108  1748 raft_consensus.cc:3060] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:02:11.761325  2333 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39726 (request call id 1520) took 2197 ms. Trace:
I20260709 13:02:11.795408  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:11.797014  2333 rpcz_store.cc:276] 0709 13:02:09.563586 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:09.662046 (+ 98460us) service_pool.cc:224] Handling call
0709 13:02:11.761053 (+2099007us) spinlock_profiling.cc:243] Waited 231 ms on lock 0x7b5400567f54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:02:11.761239 (+   186us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":530188160}
I20260709 13:02:11.786875  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:11.732520  8514 raft_consensus.cc:493] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Starting leader election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:02:11.796506  8508 raft_consensus.cc:493] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:11.801523  8623 raft_consensus.cc:3055] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 13:02:11.801904  8514 raft_consensus.cc:3060] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:02:11.802945  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:11.802261  8508 raft_consensus.cc:3060] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:02:11.816936  2337 raft_consensus.cc:2468] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 17.
I20260709 13:02:11.793615  8506 raft_consensus.cc:3060] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:02:11.741319  8344 raft_consensus.cc:2804] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Leader pre-election won for term 21
W20260709 13:02:11.822400  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 103946us
I20260709 13:02:11.783133  8617 raft_consensus.cc:2804] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Leader election won for term 20
I20260709 13:02:11.733232  8599 raft_consensus.cc:493] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
W20260709 13:02:11.825105  1571 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:11.800220  8588 raft_consensus.cc:2804] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Leader election won for term 17
W20260709 13:02:11.828148  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 217780us
I20260709 13:02:11.752326  8515 consensus_queue.cc:1048] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.072s
I20260709 13:02:11.803834  8623 raft_consensus.cc:740] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:02:11.759536  1717 leader_election.cc:304] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:11.832065  1748 raft_consensus.cc:2468] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 21.
I20260709 13:02:11.741288  8338 raft_consensus.cc:515] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:11.863590  7307 raft_consensus.cc:493] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:11.864219  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 151239us
W20260709 13:02:11.899058  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 139692us
I20260709 13:02:11.806620  8461 raft_consensus.cc:2749] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:02:11.861975  1646 leader_election.cc:304] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:11.858724  8496 raft_consensus.cc:2764] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Leader election decision vote started in defunct term 16: won
I20260709 13:02:11.863837  8626 raft_consensus.cc:3055] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 19 LEADER]: Stepping down as leader of term 19
I20260709 13:02:11.895980  2328 raft_consensus.cc:2468] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 20.
I20260709 13:02:11.925523  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:11.898840  7307 raft_consensus.cc:515] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:11.909603  1569 leader_election.cc:304] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:02:11.927345  1602 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40154 (request call id 2813) took 1208 ms. Trace:
I20260709 13:02:11.730234  8591 raft_consensus.cc:515] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:11.860723  8344 raft_consensus.cc:493] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:11.824846  8404 raft_consensus.cc:697] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:02:11.925549  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:11.929677  1569 leader_election.cc:304] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
W20260709 13:02:11.945163  8619 log.cc:927] Time spent T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.118s	user 0.001s	sys 0.000s
I20260709 13:02:11.927743  1602 rpcz_store.cc:276] 0709 13:02:10.718570 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:10.766781 (+ 48211us) service_pool.cc:224] Handling call
0709 13:02:11.927158 (+1160377us) spinlock_profiling.cc:243] Waited 858 ms on lock 0x7b54002797d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:02:11.927282 (+   124us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1972753152}
I20260709 13:02:11.944850  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
W20260709 13:02:11.947163  8621 log.cc:927] Time spent T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.143s	user 0.001s	sys 0.000s
I20260709 13:02:11.948112  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:11.895717  8599 raft_consensus.cc:515] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:11.946422  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:11.897946  8617 raft_consensus.cc:697] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
W20260709 13:02:11.954823  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 55791.1us
I20260709 13:02:11.912432  8145 raft_consensus.cc:2749] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 13:02:11.961369  7307 leader_election.cc:290] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:11.827943  8609 raft_consensus.cc:515] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:11.864499  8504 raft_consensus.cc:515] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:11.952867  1717 leader_election.cc:304] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:11.966921  7307 raft_consensus.cc:2764] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Leader pre-election decision vote started in defunct term 18: won
I20260709 13:02:11.913286  8588 raft_consensus.cc:697] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:02:11.835903  8513 consensus_queue.cc:237] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.8, Last appended by leader: 8, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:11.941243  8344 raft_consensus.cc:3060] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:02:11.925808  8626 raft_consensus.cc:740] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:02:11.950762  8599 leader_election.cc:290] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:11.961643  8582 raft_consensus.cc:2749] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20260709 13:02:11.981943  8613 log.cc:927] Time spent T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.225s	user 0.001s	sys 0.001s
I20260709 13:02:11.973330  8360 raft_consensus.cc:493] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:11.997016  8622 log.cc:927] Time spent T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.200s	user 0.001s	sys 0.000s
I20260709 13:02:11.997175  8294 consensus_queue.cc:1059] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 14.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 13:02:11.998693  8154 raft_consensus.cc:493] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:11.999244  8358 raft_consensus.cc:3055] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:02:12.003249  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:12.005415  8609 leader_election.cc:290] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:12.013059  1745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43156 (request call id 1386) took 1855 ms. Trace:
I20260709 13:02:11.999238  8154 raft_consensus.cc:515] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:12.013490  1745 rpcz_store.cc:276] 0709 13:02:10.157523 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:10.158409 (+   886us) service_pool.cc:224] Handling call
0709 13:02:10.158917 (+   508us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:10.176100 (+ 17183us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:10.176151 (+    51us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:10.176859 (+   708us) log.cc:844] Serialized 26 byte log entry
0709 13:02:12.000088 (+1823229us) spinlock_profiling.cc:243] Waited 58.8 ms on lock 0x7b54002cb3a0. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cc090 00007f42660d96b8 00007f426616292b 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c
0709 13:02:12.000158 (+    70us) raft_consensus.cc:1645] Marking committed up to 14
0709 13:02:12.000236 (+    78us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:12.000323 (+    87us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:12.000371 (+    48us) raft_consensus.cc:1685] finished
0709 13:02:12.000382 (+    11us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:12.012984 (+ 12602us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":16627,"raft.queue_time_us":18292,"raft.run_cpu_time_us":133,"raft.run_wall_time_us":134,"spinlock_wait_cycles":135195008,"thread_start_us":68655,"threads_started":1,"wal-append.queue_time_us":60064,"wal-append.run_cpu_time_us":3754,"wal-append.run_wall_time_us":1090127}
I20260709 13:02:12.015854  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:12.016476  8588 consensus_queue.cc:237] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.12, Last appended by leader: 12, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:12.020062  8617 consensus_queue.cc:237] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.12, Last appended by leader: 12, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:12.023589  8591 leader_election.cc:290] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:12.025751  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:12.026528  8368 raft_consensus.cc:3060] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:02:12.027172  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:12.028028  1598 raft_consensus.cc:2393] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 13: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
W20260709 13:02:12.029366  1677 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52778 (request call id 2678) took 1415 ms (client timeout 1611 ms). Trace:
I20260709 13:02:12.029963  1754 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55840 (request call id 3028) took 1195 ms. Trace:
W20260709 13:02:12.029788  1677 rpcz_store.cc:269] 0709 13:02:10.613651 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:10.669543 (+ 55892us) service_pool.cc:224] Handling call
0709 13:02:12.029191 (+1359648us) spinlock_profiling.cc:243] Waited 96.7 ms on lock 0x7b5400090954. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:02:12.029299 (+   108us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":222452096}
I20260709 13:02:12.030324  1600 raft_consensus.cc:1275] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 13 index: 12. Preceding OpId from leader: term: 16 index: 14. (index mismatch)
I20260709 13:02:12.030591  1754 rpcz_store.cc:276] 0709 13:02:10.834909 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:10.857886 (+ 22977us) service_pool.cc:224] Handling call
0709 13:02:12.029718 (+1171832us) spinlock_profiling.cc:243] Waited 72.4 ms on lock 0x7b54001d56d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:02:12.029897 (+   179us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":166531456}
I20260709 13:02:12.032172  1754 raft_consensus.cc:1240] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 16. Current term is 17. Ops: []
I20260709 13:02:12.032872  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:12.035125  8504 leader_election.cc:290] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 20 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:12.035498  8338 leader_election.cc:290] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 17 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:02:12.023670  1753 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55848 (request call id 3124) took 1810 ms (client timeout 1631 ms). Trace:
I20260709 13:02:12.040659  2334 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49082 (request call id 1221) took 3034 ms. Trace:
W20260709 13:02:12.038527  1753 rpcz_store.cc:269] 0709 13:02:10.213083 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:10.213260 (+   177us) service_pool.cc:224] Handling call
0709 13:02:12.023440 (+1810180us) spinlock_profiling.cc:243] Waited 498 ms on lock 0x7b540035ff54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:02:12.023587 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1145129856}
I20260709 13:02:12.041088  2334 rpcz_store.cc:276] 0709 13:02:09.005988 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:09.006143 (+   155us) service_pool.cc:224] Handling call
0709 13:02:09.053661 (+ 47518us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:09.253419 (+199758us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 13:02:09.628368 (+374949us) log.cc:844] Serialized 14 byte log entry
0709 13:02:09.761918 (+133550us) log.cc:844] Serialized 14 byte log entry
0709 13:02:10.354509 (+592591us) spinlock_profiling.cc:243] Waited 29.4 ms on lock 0x7b18003e2540. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f4263075522 00007f42630777c1 00007f4263077779 00007f4263077709 00007f42630776d1 00007f42630769cd 000000000038f354 000000000038f199 00007f426305883a
0709 13:02:10.359544 (+  5035us) spinlock_profiling.cc:243] Waited 13.9 ms on lock 0x7b18003e2540. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f42630753f3 00007f42630777c1 00007f4263077779 00007f4263077709 00007f42630776d1 00007f42630769cd 000000000038f354 000000000038f199 00007f426305883a
0709 13:02:10.517945 (+158401us) spinlock_profiling.cc:243] Waited 2.47 ms on lock 0x7b18003e2540. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f4262fefc0a 00007f426c72f219 00007f4263077a40 00007f42630724e4 00007f4263072455 00007f426615a060 00007f426616c716 00007f426617d287 00007f426617d208
0709 13:02:10.518112 (+   167us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:10.518170 (+    58us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:02:10.518230 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:12.040406 (+1522176us) spinlock_profiling.cc:243] Waited 200 ms on lock 0x7b5400645354. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:02:12.040451 (+    45us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:12.040595 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":202498,"prepare.queue_time_us":532872,"prepare.run_cpu_time_us":86,"prepare.run_wall_time_us":88,"raft.queue_time_us":761021,"raft.run_cpu_time_us":259,"raft.run_wall_time_us":11975,"spinlock_wait_cycles":564217472,"thread_start_us":481432,"threads_started":2,"wal-append.queue_time_us":236997,"wal-append.run_cpu_time_us":1757,"wal-append.run_wall_time_us":1025061}
I20260709 13:02:12.044940  8627 raft_consensus.cc:2804] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Leader election won for term 17
I20260709 13:02:12.045735  8627 raft_consensus.cc:697] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:02:12.047015  7745 raft_consensus.cc:3055] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 LEADER]: Stepping down as leader of term 17
W20260709 13:02:12.048434  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 88142.8us
I20260709 13:02:12.007403  8360 raft_consensus.cc:515] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:12.050374  8564 raft_consensus.cc:493] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:12.051230  8564 raft_consensus.cc:515] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:12.047322  8627 consensus_queue.cc:237] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.10, Last appended by leader: 10, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } }
I20260709 13:02:12.052992  8360 leader_election.cc:290] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:12.054702  8626 consensus_queue.cc:260] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.15, Last appended by leader: 15, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:12.015558  8404 consensus_queue.cc:237] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.15, Last appended by leader: 15, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:12.057018  1600 raft_consensus.cc:2468] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 14.
I20260709 13:02:12.057762  1718 leader_election.cc:304] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:02:11.999719  8358 raft_consensus.cc:740] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:02:12.079523  7307 raft_consensus.cc:2749] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20260709 13:02:12.062554  1645 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:12.079931  1718 leader_election.cc:400] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 20 pre-election: Vote denied by peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261) with higher term. Message: Invalid argument: T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 20. Current term is 21.
I20260709 13:02:12.013285  8537 leader_election.cc:290] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 18 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:12.047436  7745 raft_consensus.cc:740] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:02:12.116365  7745 consensus_queue.cc:260] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.10, Last appended by leader: 10, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:12.119000  8358 consensus_queue.cc:260] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.8, Last appended by leader: 8, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:12.100329  8145 consensus_queue.cc:1048] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.015s
I20260709 13:02:12.123459  1718 leader_election.cc:304] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:02:12.127683  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:12.129837  1673 raft_consensus.cc:2393] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 18: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:02:12.130932  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:12.131748  1673 raft_consensus.cc:2393] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 19: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:02:12.132509  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:12.133092  1673 raft_consensus.cc:3060] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:02:12.140408  8609 raft_consensus.cc:2749] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:02:12.138166  1718 leader_election.cc:304] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:02:12.118269  7745 raft_consensus.cc:3060] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:02:12.146379  1745 raft_consensus.cc:1240] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 14. Current term is 15. Ops: []
I20260709 13:02:12.149412  7307 consensus_queue.cc:1048] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.001s
W20260709 13:02:12.157143  8127 consensus_meta.cc:287] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.558s	user 0.014s	sys 0.001s
W20260709 13:02:12.302316  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:12.306959  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 342556us
I20260709 13:02:12.362478  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
W20260709 13:02:12.349318  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 487492us
W20260709 13:02:12.361503  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 206105us
I20260709 13:02:12.370725  1745 raft_consensus.cc:1240] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 15. Current term is 16. Ops: [15.11-15.11]
I20260709 13:02:12.370880  1646 leader_election.cc:304] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:12.413558  8360 raft_consensus.cc:493] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
W20260709 13:02:12.414395  8468 consensus_meta.cc:287] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.872s	user 0.006s	sys 0.013s
I20260709 13:02:12.414940  8360 raft_consensus.cc:515] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:12.415328  8468 raft_consensus.cc:515] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:12.417235  8514 consensus_meta.cc:287] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.593s	user 0.014s	sys 0.006s
I20260709 13:02:12.418181  8514 raft_consensus.cc:515] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:02:12.420591  8600 consensus_meta.cc:287] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.856s	user 0.017s	sys 0.003s
I20260709 13:02:12.421559  8600 raft_consensus.cc:2749] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:02:12.422616  1678 raft_consensus.cc:3060] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Advancing to term 19
W20260709 13:02:12.423053  8602 consensus_meta.cc:287] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.884s	user 0.030s	sys 0.013s
I20260709 13:02:12.424031  8602 raft_consensus.cc:515] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:02:12.433107  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 60482.6us
W20260709 13:02:12.435318  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 59604.5us
W20260709 13:02:12.434613  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 65438.3us
W20260709 13:02:12.441780  8508 consensus_meta.cc:287] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.616s	user 0.015s	sys 0.007s
I20260709 13:02:12.442719  8508 raft_consensus.cc:515] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:12.445693  1677 raft_consensus.cc:1240] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 18. Current term is 19. Ops: []
W20260709 13:02:12.446240  8597 consensus_meta.cc:287] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.865s	user 0.013s	sys 0.007s
I20260709 13:02:12.447327  8597 raft_consensus.cc:515] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:12.449709  1753 raft_consensus.cc:1240] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 18. Current term is 19. Ops: []
I20260709 13:02:12.451992  8127 raft_consensus.cc:2749] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Leader pre-election lost for term 17. Reason: Vote denied by peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693) with higher term. Message: Invalid argument: T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 17. Current term is 18.
I20260709 13:02:12.443251  8154 leader_election.cc:290] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:12.459077  1646 leader_election.cc:304] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:12.463205  8537 raft_consensus.cc:2804] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Leader election won for term 18
I20260709 13:02:12.464236  8537 raft_consensus.cc:697] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 18 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:02:12.467157  8303 raft_consensus.cc:2804] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Leader election won for term 21
I20260709 13:02:12.467887  8303 raft_consensus.cc:697] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:02:12.471012  8360 leader_election.cc:290] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:02:12.471705  1571 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:12.472116  1679 raft_consensus.cc:2410] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 for term 20 because replica has last-logged OpId of term: 18 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 13.
I20260709 13:02:12.479213  1673 raft_consensus.cc:2468] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 17.
I20260709 13:02:12.473901  8360 raft_consensus.cc:493] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:02:12.482889  8360 raft_consensus.cc:515] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:12.483206  8504 raft_consensus.cc:493] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:02:12.478281  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:12.484511  8504 raft_consensus.cc:515] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:12.485682  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:12.469455  8303 consensus_queue.cc:237] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 19.10, Last appended by leader: 10, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:12.481566  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:12.489147  8500 raft_consensus.cc:493] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:12.498060  8597 leader_election.cc:290] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 20 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:02:12.500555  2300 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:12.487617  8514 leader_election.cc:290] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 22 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:12.526813  1678 raft_consensus.cc:2468] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 19.
I20260709 13:02:12.531716  8360 leader_election.cc:290] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:02:12.535653  1603 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40166 (request call id 2884) took 1305 ms (client timeout 1654 ms). Trace:
W20260709 13:02:12.536343  1603 rpcz_store.cc:269] 0709 13:02:11.229710 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:11.247342 (+ 17632us) service_pool.cc:224] Handling call
0709 13:02:12.535110 (+1287768us) spinlock_profiling.cc:243] Waited 45.7 ms on lock 0x7b54002d9454. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:02:12.535353 (+   243us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":105013504}
I20260709 13:02:12.543247  1673 raft_consensus.cc:2468] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 17.
I20260709 13:02:12.544574  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:12.545789  1673 raft_consensus.cc:2393] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 20: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:02:12.546838  8609 raft_consensus.cc:2749] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:02:12.546948  8500 raft_consensus.cc:515] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:12.549944  1718 leader_election.cc:304] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 13:02:12.551159  8585 raft_consensus.cc:2804] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Leader pre-election won for term 17
W20260709 13:02:12.551820  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 54154.1us
I20260709 13:02:12.553171  1718 leader_election.cc:304] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:12.538127  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:12.466428  8537 consensus_queue.cc:237] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.12, Last appended by leader: 12, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:12.579694  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:12.579917  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:12.572441  8468 leader_election.cc:290] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:12.583230  8504 leader_election.cc:290] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:12.603315  8650 raft_consensus.cc:2749] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
W20260709 13:02:12.603837  1676 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52778 (request call id 2680) took 1989 ms (client timeout 2063 ms). Trace:
I20260709 13:02:12.584847  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:12.590525  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:12.594566  1597 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 1712) took 1186 ms. Trace:
I20260709 13:02:12.595664  1718 leader_election.cc:304] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:12.599496  8500 leader_election.cc:290] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:02:12.599732  8506 consensus_meta.cc:287] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.735s	user 0.013s	sys 0.012s
I20260709 13:02:12.574126  8602 leader_election.cc:290] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 19 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 13:02:12.606887  8344 consensus_meta.cc:287] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.616s	user 0.017s	sys 0.009s
I20260709 13:02:12.602145  8652 raft_consensus.cc:2804] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:02:12.608031  1680 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52778 (request call id 2671) took 1996 ms. Trace:
I20260709 13:02:12.615192  1751 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55840 (request call id 3032) took 1380 ms. Trace:
W20260709 13:02:12.617139  1752 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55848 (request call id 3143) took 2206 ms (client timeout 2168 ms). Trace:
W20260709 13:02:12.619082  1746 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43156 (request call id 1393) took 2074 ms (client timeout 1721 ms). Trace:
W20260709 13:02:12.620783  1750 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55848 (request call id 3135) took 2214 ms (client timeout 1654 ms). Trace:
I20260709 13:02:12.623059  1673 raft_consensus.cc:2393] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 18: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
W20260709 13:02:12.628456  2331 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39726 (request call id 1531) took 1917 ms (client timeout 2006 ms). Trace:
I20260709 13:02:12.632671  1749 raft_consensus.cc:1240] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 17. Current term is 18. Ops: []
W20260709 13:02:12.637153  1676 rpcz_store.cc:269] 0709 13:02:10.614304 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:10.814862 (+200558us) service_pool.cc:224] Handling call
0709 13:02:12.603181 (+1788319us) spinlock_profiling.cc:243] Waited 380 ms on lock 0x7b54000230d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:02:12.603521 (+   340us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":875069184}
I20260709 13:02:12.645149  1599 raft_consensus.cc:2468] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 18.
W20260709 13:02:12.651175  2331 rpcz_store.cc:269] 0709 13:02:10.710717 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:10.711378 (+   661us) service_pool.cc:224] Handling call
0709 13:02:12.627800 (+1916422us) spinlock_profiling.cc:243] Waited 72 ms on lock 0x7b5400551754. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:02:12.628116 (+   316us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":165660032}
W20260709 13:02:12.650362  1750 rpcz_store.cc:269] 0709 13:02:10.406625 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:10.409218 (+  2593us) service_pool.cc:224] Handling call
0709 13:02:12.620224 (+2211006us) spinlock_profiling.cc:243] Waited 1.11 s on lock 0x7b5400071554. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:02:12.620539 (+   315us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2558783872}
W20260709 13:02:12.649959  1746 rpcz_store.cc:269] 0709 13:02:10.544901 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:10.578083 (+ 33182us) service_pool.cc:224] Handling call
0709 13:02:12.618457 (+2040374us) spinlock_profiling.cc:243] Waited 918 ms on lock 0x7b540019bed4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:02:12.618764 (+   307us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2110357760}
W20260709 13:02:12.649248  1752 rpcz_store.cc:269] 0709 13:02:10.409968 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:10.432182 (+ 22214us) service_pool.cc:224] Handling call
0709 13:02:12.616390 (+2184208us) spinlock_profiling.cc:243] Waited 434 ms on lock 0x7b540006fc54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:02:12.616736 (+   346us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":999239552}
I20260709 13:02:12.668915  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:12.647606  1680 rpcz_store.cc:276] 0709 13:02:10.611259 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:10.640966 (+ 29707us) service_pool.cc:224] Handling call
0709 13:02:10.641636 (+   670us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:10.783608 (+141972us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:12.607222 (+1823614us) spinlock_profiling.cc:243] Waited 118 ms on lock 0x7b54001e87d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:02:12.607684 (+   462us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":271047424}
I20260709 13:02:12.646554  8652 raft_consensus.cc:493] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
W20260709 13:02:12.676613  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 81256.7us
I20260709 13:02:12.639045  1597 rpcz_store.cc:276] 0709 13:02:11.372577 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:11.507718 (+135141us) service_pool.cc:224] Handling call
0709 13:02:11.508148 (+   430us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:11.570718 (+ 62570us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:11.570759 (+    41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:11.571316 (+   557us) log.cc:844] Serialized 26 byte log entry
0709 13:02:11.714932 (+143616us) raft_consensus.cc:1645] Marking committed up to 14
0709 13:02:11.714968 (+    36us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:12.506182 (+791214us) spinlock_profiling.cc:243] Waited 62.3 ms on lock 0x7b54002170d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:02:12.506312 (+   130us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:12.506800 (+   488us) raft_consensus.cc:1685] finished
0709 13:02:12.506820 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:12.558664 (+ 51844us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":143292160,"thread_start_us":143173,"threads_started":1,"wal-append.queue_time_us":210265}
I20260709 13:02:12.652058  8344 raft_consensus.cc:515] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:12.679684  8652 raft_consensus.cc:3060] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:02:12.703838  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:12.647992  1751 rpcz_store.cc:276] 0709 13:02:11.234530 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:11.314325 (+ 79795us) service_pool.cc:224] Handling call
0709 13:02:11.321866 (+  7541us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:11.328109 (+  6243us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:11.328160 (+    51us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:11.352359 (+ 24199us) log.cc:844] Serialized 26 byte log entry
0709 13:02:11.449161 (+ 96802us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:02:11.449225 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:12.614237 (+1165012us) spinlock_profiling.cc:243] Waited 4.6 ms on lock 0x7b5400220954. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:02:12.614329 (+    92us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:12.614416 (+    87us) raft_consensus.cc:1685] finished
0709 13:02:12.614449 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:12.614895 (+   446us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1143,"raft.queue_time_us":30350,"raft.run_cpu_time_us":237,"raft.run_wall_time_us":50386,"spinlock_wait_cycles":10574720,"thread_start_us":49804,"threads_started":1,"wal-append.queue_time_us":141090}
I20260709 13:02:12.704753  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:12.659531  8506 raft_consensus.cc:515] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:12.712546  8414 raft_consensus.cc:493] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:12.713654  8651 raft_consensus.cc:2749] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 13:02:12.714322  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 18 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:12.714526  8414 raft_consensus.cc:515] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:12.715916  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:12.716718  1602 raft_consensus.cc:3060] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:02:12.722069  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:12.713816  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:12.727370  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:12.734797  1675 raft_consensus.cc:1240] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 18 LEADER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 17. Current term is 18. Ops: []
I20260709 13:02:12.727828  1718 leader_election.cc:304] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:12.740063  1673 raft_consensus.cc:2468] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 14.
I20260709 13:02:12.742261  1676 raft_consensus.cc:2468] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 20.
I20260709 13:02:12.747735  1747 raft_consensus.cc:2393] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 21: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:02:12.748565  1718 leader_election.cc:304] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:12.750901  8402 raft_consensus.cc:2804] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:02:12.751325  8402 raft_consensus.cc:493] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:12.751597  8344 leader_election.cc:290] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 21 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:12.751663  8402 raft_consensus.cc:3060] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:02:12.752560  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 17 candidate_status { last_received { term: 16 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:12.754012  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:12.754824  1680 raft_consensus.cc:2468] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 19.
I20260709 13:02:12.755960  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:12.756772  1680 raft_consensus.cc:2393] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 17: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:02:12.757700  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:12.758590  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:12.759081  8414 leader_election.cc:290] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:12.759675  1718 leader_election.cc:304] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:12.760643  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:12.761775  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:12.762245  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:12.763341  2333 raft_consensus.cc:2393] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 21: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:02:12.763321  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:12.764256  2334 raft_consensus.cc:3060] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:02:12.746901  1603 raft_consensus.cc:2376] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 19. Re-sending same reply.
I20260709 13:02:12.767295  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:12.768826  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 16 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:12.770107  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:12.762115  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
W20260709 13:02:12.771468  2332 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49082 (request call id 1229) took 1895 ms (client timeout 1984 ms). Trace:
W20260709 13:02:12.773949  2332 rpcz_store.cc:269] 0709 13:02:10.851017 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:12.637548 (+1786531us) service_pool.cc:224] Handling call
0709 13:02:12.745833 (+108285us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:12.787359  1680 raft_consensus.cc:2468] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 14.
I20260709 13:02:12.790871  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:12.796159  8602 raft_consensus.cc:2804] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 13:02:12.797068  8602 raft_consensus.cc:697] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 19 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
W20260709 13:02:12.801935  8368 consensus_meta.cc:287] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.775s	user 0.008s	sys 0.018s
I20260709 13:02:12.807547  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:12.814656  8323 raft_consensus.cc:493] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:12.815277  8323 raft_consensus.cc:515] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:12.815685  8651 raft_consensus.cc:2804] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 13:02:12.816529  8344 consensus_queue.cc:1048] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 13:02:12.822927  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:12.825104  1680 raft_consensus.cc:1240] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 15. Current term is 16. Ops: [15.14-15.14]
I20260709 13:02:12.796484  2329 raft_consensus.cc:1240] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 15. Current term is 16. Ops: []
W20260709 13:02:12.834230  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 110004us
I20260709 13:02:12.809760  1673 raft_consensus.cc:1240] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 LEADER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 16. Current term is 17. Ops: []
I20260709 13:02:12.857049  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:12.816602  8651 raft_consensus.cc:697] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 17 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:02:12.861670  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:12.862080  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:12.861090  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:12.866650  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:12.869900  1673 raft_consensus.cc:1240] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 18. Current term is 19. Ops: []
I20260709 13:02:12.863873  1597 raft_consensus.cc:2376] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 14. Re-sending same reply.
I20260709 13:02:12.880471  2334 raft_consensus.cc:2468] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 17.
I20260709 13:02:12.881517  8559 raft_consensus.cc:493] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:12.882331  8559 raft_consensus.cc:515] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:02:12.884248  8637 log.cc:927] Time spent T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.843s	user 0.002s	sys 0.000s
I20260709 13:02:12.888985  8358 raft_consensus.cc:3060] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:02:12.890561  2336 raft_consensus.cc:606] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Received request to transfer leadership to TS af520b67945d443b901bbeca37cc5701
I20260709 13:02:12.891068  2336 raft_consensus.cc:614] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 13:02:12.891295  8564 leader_election.cc:290] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:12.878496  8650 raft_consensus.cc:493] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:12.864835  8360 raft_consensus.cc:493] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:12.898123  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:12.898324  8360 raft_consensus.cc:515] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:12.900928  8360 leader_election.cc:290] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:12.902644  8650 raft_consensus.cc:515] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:12.909194  8645 log.cc:927] Time spent T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.375s	user 0.000s	sys 0.001s
I20260709 13:02:12.911250  1718 leader_election.cc:304] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:02:12.896768  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:12.863224  8344 raft_consensus.cc:493] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:02:12.912917  2337 raft_consensus.cc:2468] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 14.
I20260709 13:02:12.917292  1674 raft_consensus.cc:1240] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 15. Current term is 16. Ops: []
I20260709 13:02:12.918561  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
W20260709 13:02:12.918694  1569 leader_election.cc:336] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 19 election: RPC error from VoteRequest() call to peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625): Timed out: RequestConsensusVote RPC to 127.0.107.196:46625 timed out after 1.984s (SENT)
I20260709 13:02:12.920028  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
W20260709 13:02:12.920929  1569 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:12.921844  8461 raft_consensus.cc:493] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:12.922634  8461 raft_consensus.cc:515] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:12.923290  1672 raft_consensus.cc:1240] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 19. Current term is 20. Ops: []
I20260709 13:02:12.924592  1672 raft_consensus.cc:1240] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 21. Current term is 22. Ops: []
I20260709 13:02:12.925824  8461 leader_election.cc:290] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:12.867349  8602 consensus_queue.cc:237] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.12, Last appended by leader: 12, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:12.927131  8454 leader_election.cc:290] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 19 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:12.927645  1598 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40154 (request call id 2826) took 1023 ms. Trace:
I20260709 13:02:12.926180  1672 raft_consensus.cc:1240] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 14. Current term is 15. Ops: []
W20260709 13:02:12.932673  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:12.928082  1598 rpcz_store.cc:276] 0709 13:02:11.903971 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:12.028599 (+124628us) service_pool.cc:224] Handling call
0709 13:02:12.028986 (+   387us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:02:12.077295 (+ 48309us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:12.077342 (+    47us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:02:12.107873 (+ 30531us) log.cc:844] Serialized 52 byte log entry
0709 13:02:12.384348 (+276475us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:02:12.384457 (+   109us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:12.384635 (+   178us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:12.908348 (+523713us) spinlock_profiling.cc:243] Waited 28.8 ms on lock 0x7b4400022b4c. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f4262f9bc85 00007f4262f9c6a2 00007f4265d0e1db 00007f4265d09aaf 00007f4265cfd86e 00007f4265cfcbb6 00007f4265cfc7d8 00007f4265d06221 00007f4265d061d9 00007f4265d06169
0709 13:02:12.914754 (+  6406us) raft_consensus.cc:1685] finished
0709 13:02:12.914774 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:12.927439 (+ 12665us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":47722,"raft.queue_time_us":50477,"raft.run_cpu_time_us":219,"raft.run_wall_time_us":24847,"spinlock_wait_cycles":66176256,"thread_start_us":20458,"threads_started":1,"wal-append.queue_time_us":207621}
I20260709 13:02:12.941479  8650 leader_election.cc:290] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:12.937017  1603 raft_consensus.cc:2393] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 20: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:02:12.932763  8609 raft_consensus.cc:493] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:12.951473  1602 raft_consensus.cc:2468] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 17.
W20260709 13:02:12.956847  8472 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.763s	user 0.004s	sys 0.008s
W20260709 13:02:12.957414  8472 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.763s	user 0.004s	sys 0.008s
I20260709 13:02:12.942662  8506 leader_election.cc:290] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 15 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:12.958760  8360 raft_consensus.cc:493] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:12.964684  8648 raft_consensus.cc:2749] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:02:12.960953  8559 leader_election.cc:290] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:12.967721  1718 leader_election.cc:304] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:12.969723  1718 leader_election.cc:304] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
W20260709 13:02:12.970696  7745 consensus_meta.cc:287] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.789s	user 0.015s	sys 0.003s
I20260709 13:02:12.970572  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 17 candidate_status { last_received { term: 16 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:12.971390  1597 raft_consensus.cc:2468] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 16.
I20260709 13:02:12.972419  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:12.973860  8323 leader_election.cc:290] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:12.974606  8609 raft_consensus.cc:515] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:12.975556  8603 raft_consensus.cc:3055] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:02:12.975997  8603 raft_consensus.cc:740] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:02:12.977232  8609 leader_election.cc:290] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:12.977186  8603 consensus_queue.cc:260] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.10, Last appended by leader: 10, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:12.981199  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:12.982474  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:12.981962  8651 consensus_queue.cc:237] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.12, Last appended by leader: 12, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:12.983547  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:12.984176  1598 raft_consensus.cc:3060] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:02:12.969882  8650 raft_consensus.cc:2804] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 13:02:12.985913  1677 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57092 (request call id 1520) took 1353 ms. Trace:
I20260709 13:02:12.986363  1677 rpcz_store.cc:276] 0709 13:02:11.632714 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:12.031043 (+398329us) service_pool.cc:224] Handling call
0709 13:02:12.031354 (+   311us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:12.446251 (+414897us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:12.985591 (+539340us) spinlock_profiling.cc:243] Waited 414 ms on lock 0x7b5400090954. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:02:12.985817 (+   226us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":952233216}
W20260709 13:02:12.988036  1488 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet 983621a6fc5b4c9ea0159a648dfe7a36 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER failed with leader 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 3b879799899a499c8d409ee9f1277cd1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 19 committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 13:02:12.982081  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:13.000821  1751 raft_consensus.cc:2468] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 16.
I20260709 13:02:12.986537  8344 raft_consensus.cc:515] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:13.002408  8454 consensus_queue.cc:1048] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
W20260709 13:02:13.004465  8610 log.cc:927] Time spent T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.960s	user 0.000s	sys 0.011s
I20260709 13:02:13.002555  8650 raft_consensus.cc:697] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 16 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
W20260709 13:02:13.008064  8620 log.cc:927] Time spent T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.989s	user 0.009s	sys 0.010s
I20260709 13:02:13.011425  8650 consensus_queue.cc:237] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:13.012324  8344 leader_election.cc:290] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:13.020720  8360 raft_consensus.cc:515] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:13.021123  1671 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52766 (request call id 3119) took 3586 ms. Trace:
I20260709 13:02:13.021687  1671 rpcz_store.cc:276] 0709 13:02:09.434852 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:09.912766 (+477914us) service_pool.cc:224] Handling call
0709 13:02:09.913202 (+   436us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:10.443800 (+530598us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:13.020579 (+2576779us) spinlock_profiling.cc:243] Waited 498 ms on lock 0x7b5400163d54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:02:13.020924 (+   345us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1144476416}
I20260709 13:02:13.023355  8360 leader_election.cc:290] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:13.025758  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:13.026814  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:02:12.993083  8513 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.809s	user 0.002s	sys 0.016s
I20260709 13:02:13.030462  1597 raft_consensus.cc:3060] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:02:13.036851  1602 raft_consensus.cc:2468] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 15.
W20260709 13:02:13.041361  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 51597.4us
I20260709 13:02:13.041580  8603 raft_consensus.cc:3060] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:02:13.041519  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:13.043138  2301 leader_election.cc:304] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:13.043702  8402 raft_consensus.cc:515] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:13.044149  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:13.044775  8609 raft_consensus.cc:2804] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 13:02:13.045466  8609 raft_consensus.cc:697] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 20 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:02:13.046821  8609 consensus_queue.cc:237] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.14, Last appended by leader: 14, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:13.050697  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 81114.5us
I20260709 13:02:13.052417  1752 raft_consensus.cc:2468] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 19.
I20260709 13:02:13.052567  1718 leader_election.cc:304] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:13.053891  1718 leader_election.cc:304] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:13.054639  1604 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40154 (request call id 2803) took 2344 ms. Trace:
I20260709 13:02:13.054733  8454 raft_consensus.cc:2804] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 13:02:13.055534  8454 raft_consensus.cc:697] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 20 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:02:13.056929  8454 consensus_queue.cc:237] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 17.14, Last appended by leader: 14, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:13.058267  8633 log.cc:927] Time spent T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 1.016s	user 0.004s	sys 0.015s
I20260709 13:02:13.058674  1603 raft_consensus.cc:3060] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:02:13.055215  1604 rpcz_store.cc:276] 0709 13:02:10.710160 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:10.767380 (+ 57220us) service_pool.cc:224] Handling call
0709 13:02:10.786731 (+ 19351us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:10.787142 (+   411us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 13:02:10.832620 (+ 45478us) log.cc:844] Serialized 14 byte log entry
0709 13:02:11.517946 (+685326us) log.cc:844] Serialized 14 byte log entry
0709 13:02:11.518145 (+   199us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:11.518736 (+   591us) log.cc:844] Serialized 26 byte log entry
0709 13:02:11.518880 (+   144us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:02:11.518914 (+    34us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:11.518989 (+    75us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:11.998457 (+479468us) spinlock_profiling.cc:243] Waited 1.82 ms on lock 0x7b18002a9520. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f4263075522 00007f42630777c1 00007f4263077779 00007f4263077709 00007f42630776d1 00007f42630769cd 000000000038f354 000000000038f199 00007f426305883a
0709 13:02:12.044153 (+ 45696us) spinlock_profiling.cc:243] Waited 4.67 ms on lock 0x7b18002a9520. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f42630753f3 00007f42630777c1 00007f4263077779 00007f4263077709 00007f42630776d1 00007f42630769cd 000000000038f354 000000000038f199 00007f426305883a
0709 13:02:12.634809 (+590656us) spinlock_profiling.cc:243] Waited 3.22 ms on lock 0x7b4800003cd8. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426b08baa1 00007f426b082b37 00007f426b0829ca 00007f426b17a26f 00007f426b179f37 00007f426b18e55c 00007f426b18e508 00007f426b18e488 00007f426b18e443 00007f426b18d6ec 00007f426a0b7a96
0709 13:02:13.003518 (+368709us) spinlock_profiling.cc:243] Waited 30.5 ms on lock 0x7b4400022b4c. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f4262f9bc85 00007f4262f9c6a2 00007f4265d0e1db 00007f4265d09aaf 00007f4265cfd86e 00007f4265cfcbb6 00007f4265cfc7d8 00007f4265d06221 00007f4265d061d9 00007f4265d06169
0709 13:02:13.054105 (+ 50587us) raft_consensus.cc:1685] finished
0709 13:02:13.054138 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:13.054350 (+   212us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":18922,"prepare.queue_time_us":66743,"prepare.run_cpu_time_us":88,"prepare.run_wall_time_us":90,"raft.queue_time_us":2754,"raft.run_cpu_time_us":4281,"raft.run_wall_time_us":1152353,"spinlock_wait_cycles":92414848,"thread_start_us":482069,"threads_started":1,"wal-append.queue_time_us":629575}
W20260709 13:02:13.055581  2299 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:13.055872  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 56993.4us
I20260709 13:02:13.054723  1605 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40154 (request call id 2799) took 2348 ms. Trace:
I20260709 13:02:13.065827  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:13.068511  1596 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 1715) took 1684 ms. Trace:
I20260709 13:02:13.071031  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:13.071918  1745 raft_consensus.cc:2468] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 18.
I20260709 13:02:13.072065  1646 leader_election.cc:304] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:13.073450  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
W20260709 13:02:13.034045  8513 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.982s	user 0.002s	sys 0.017s
I20260709 13:02:13.073475  8607 raft_consensus.cc:2804] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Leader election won for term 19
I20260709 13:02:13.068537  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:13.080850  8403 raft_consensus.cc:2804] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:02:13.081450  2336 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:35790 (request call id 32) took 3470 ms. Trace:
I20260709 13:02:13.082360  8344 raft_consensus.cc:3060] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:02:13.069027  1596 rpcz_store.cc:276] 0709 13:02:11.384337 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:11.512611 (+128274us) service_pool.cc:224] Handling call
0709 13:02:11.517917 (+  5306us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:11.528923 (+ 11006us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:11.528957 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:11.573288 (+ 44331us) log.cc:844] Serialized 26 byte log entry
0709 13:02:11.711335 (+138047us) raft_consensus.cc:1645] Marking committed up to 13
0709 13:02:11.732948 (+ 21613us) log.cc:844] Serialized 14 byte log entry
0709 13:02:11.746003 (+ 13055us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:11.746129 (+   126us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:12.018350 (+272221us) spinlock_profiling.cc:243] Waited 2.28 ms on lock 0x7b180037daa0. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f42630753f3 00007f42630777c1 00007f4263077779 00007f4263077709 00007f42630776d1 00007f42630769cd 000000000038f354 000000000038f199 00007f426305883a
0709 13:02:12.768778 (+750428us) spinlock_profiling.cc:243] Waited 2.61 ms on lock 0x7b4800003cd8. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426b08baa1 00007f426b082b37 00007f426b0829ca 00007f426b17a26f 00007f426b179f37 00007f426b18e55c 00007f426b18e508 00007f426b18e488 00007f426b18e443 00007f426b18d6ec 00007f426a0b7a96
0709 13:02:13.007148 (+238370us) spinlock_profiling.cc:243] Waited 37.9 ms on lock 0x7b4400022b4c. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f4262f9bc85 00007f4262f9c6a2 00007f4265d0e1db 00007f4265d09aaf 00007f4265cfd86e 00007f4265cfcbb6 00007f4265cfc7d8 00007f4265d06221 00007f4265d061d9 00007f4265d06169
0709 13:02:13.067963 (+ 60815us) raft_consensus.cc:1685] finished
0709 13:02:13.067986 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:13.068228 (+   242us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":12657,"prepare.queue_time_us":30600,"prepare.run_cpu_time_us":55,"prepare.run_wall_time_us":64,"raft.queue_time_us":18043,"raft.run_cpu_time_us":2256,"raft.run_wall_time_us":1249891,"spinlock_wait_cycles":98469376,"thread_start_us":137655,"threads_started":1,"wal-append.queue_time_us":221944}
I20260709 13:02:13.083982  1753 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55848 (request call id 3155) took 1555 ms. Trace:
I20260709 13:02:13.064296  1605 rpcz_store.cc:276] 0709 13:02:10.706619 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:10.706835 (+   216us) service_pool.cc:224] Handling call
0709 13:02:10.707171 (+   336us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:10.717700 (+ 10529us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:13.031729 (+2314029us) spinlock_profiling.cc:243] Waited 9.7 ms on lock 0x7b5400279cd4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:02:13.054467 (+ 22738us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":22318976}
I20260709 13:02:13.092326  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:02:13.059764  8588 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.845s	user 0.004s	sys 0.009s
I20260709 13:02:13.081970  2301 leader_election.cc:304] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:02:13.093152  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:13.103446  1754 raft_consensus.cc:3060] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Advancing to term 20
W20260709 13:02:13.104583  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 50849.4us
I20260709 13:02:13.093609  1646 leader_election.cc:304] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:13.091831  2336 rpcz_store.cc:276] 0709 13:02:09.610672 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:09.719078 (+108406us) service_pool.cc:224] Handling call
0709 13:02:13.081125 (+3362047us) spinlock_profiling.cc:243] Waited 3.11 s on lock 0x7b54005f3a54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f426615c853 00007f426b12a60a 00007f42646b07d4 00007f42646b0762 00007f42646b0691 00007f42646b060c 00007f42646af8c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:02:13.081281 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":7158486912}
I20260709 13:02:13.106750  1604 raft_consensus.cc:2468] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 19.
W20260709 13:02:13.116269  2300 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:13.092512  1753 rpcz_store.cc:276] 0709 13:02:11.528663 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:12.043028 (+514365us) service_pool.cc:224] Handling call
0709 13:02:12.043422 (+   394us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:12.450245 (+406823us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:13.083658 (+633413us) spinlock_profiling.cc:243] Waited 383 ms on lock 0x7b54001c6954. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:02:13.083890 (+   232us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":882026368}
I20260709 13:02:13.118319  8402 leader_election.cc:290] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 18 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:13.108659  8403 raft_consensus.cc:493] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:02:13.126883  8403 raft_consensus.cc:3060] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:02:13.128213  1679 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57092 (request call id 1518) took 1515 ms. Trace:
I20260709 13:02:13.128656  1679 rpcz_store.cc:276] 0709 13:02:11.613256 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:11.620486 (+  7230us) service_pool.cc:224] Handling call
0709 13:02:13.127934 (+1507448us) spinlock_profiling.cc:243] Waited 798 ms on lock 0x7b54000596d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:02:13.128139 (+   205us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1835546880}
I20260709 13:02:13.130115  8607 raft_consensus.cc:697] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 19 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:02:13.130988  8373 raft_consensus.cc:3055] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 LEADER]: Stepping down as leader of term 15
I20260709 13:02:13.131482  8373 raft_consensus.cc:740] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:02:13.131376  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:13.132405  1745 raft_consensus.cc:2393] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 18: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:02:13.131691  8607 consensus_queue.cc:237] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:13.132704  8373 consensus_queue.cc:260] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.11, Last appended by leader: 11, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:13.133963  8373 raft_consensus.cc:3060] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:02:13.138185  1597 raft_consensus.cc:2468] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 21.
I20260709 13:02:13.115525  1749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55840 (request call id 3034) took 1873 ms. Trace:
I20260709 13:02:13.134022  1745 raft_consensus.cc:1275] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 18 index: 10. Preceding OpId from leader: term: 19 index: 11. (index mismatch)
W20260709 13:02:13.139285  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 49601.5us
W20260709 13:02:13.109865  8588 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.090s	user 0.005s	sys 0.009s
I20260709 13:02:13.139891  1749 rpcz_store.cc:276] 0709 13:02:11.241936 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:11.819210 (+577274us) service_pool.cc:224] Handling call
0709 13:02:12.631937 (+812727us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:12.664781 (+ 32844us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:13.112123 (+447342us) spinlock_profiling.cc:243] Waited 813 ms on lock 0x7b540006fc50. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661618ad 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:02:13.115425 (+  3302us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1868849024}
I20260709 13:02:13.082002  8551 raft_consensus.cc:3055] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 LEADER]: Stepping down as leader of term 15
I20260709 13:02:13.145787  8514 raft_consensus.cc:2804] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:02:13.148279  8514 raft_consensus.cc:493] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:02:13.154358  1602 raft_consensus.cc:2468] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 15.
I20260709 13:02:13.163864  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:13.157197  8514 raft_consensus.cc:3060] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:02:13.164796  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:13.158227  1646 leader_election.cc:304] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:02:13.164577  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:13.202982  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:13.212668  1598 raft_consensus.cc:2410] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for term 17 because replica has last-logged OpId of term: 16 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 10.
I20260709 13:02:13.121629  8652 raft_consensus.cc:515] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:13.166756  8551 raft_consensus.cc:740] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:02:13.164646  8513 raft_consensus.cc:493] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:13.206923  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:13.216879  1597 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 1721) took 1300 ms. Trace:
I20260709 13:02:13.260378  1597 rpcz_store.cc:276] 0709 13:02:11.911326 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:13.145903 (+1234577us) service_pool.cc:224] Handling call
0709 13:02:13.180205 (+ 34302us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:13.196271 (+ 16066us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:13.196320 (+    49us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:13.210917 (+ 14597us) raft_consensus.cc:1645] Marking committed up to 14
0709 13:02:13.210970 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:13.211064 (+    94us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:13.211223 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:13.219113  2330 raft_consensus.cc:2468] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 19.
I20260709 13:02:13.233156  1749 raft_consensus.cc:1275] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 17 index: 11. (index mismatch)
I20260709 13:02:13.214026  8504 raft_consensus.cc:2749] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 13:02:13.248185  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
W20260709 13:02:13.249107  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 104786us
I20260709 13:02:13.202320  8648 consensus_queue.cc:1059] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 21.13, Next index: 14, Last known committed idx: 12, Time since last communication: 0.026s
I20260709 13:02:13.252647  8623 consensus_queue.cc:260] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:13.253788  8496 raft_consensus.cc:493] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:13.258622  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 176890us
I20260709 13:02:13.262552  1598 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40166 (request call id 2912) took 1193 ms. Trace:
I20260709 13:02:13.263375  1673 raft_consensus.cc:3060] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:02:13.265779  1598 rpcz_store.cc:276] 0709 13:02:12.066092 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:12.949857 (+883765us) service_pool.cc:224] Handling call
0709 13:02:13.259266 (+309409us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:13.279635  1752 raft_consensus.cc:2393] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 19: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
W20260709 13:02:13.353574  1569 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:13.404927  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:13.304539  8623 raft_consensus.cc:3060] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:02:13.434113  2301 leader_election.cc:304] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
W20260709 13:02:13.463780  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 198200us
I20260709 13:02:13.275415  8496 raft_consensus.cc:515] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:13.466023  1749 raft_consensus.cc:1275] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 18 index: 9. Preceding OpId from leader: term: 19 index: 10. (index mismatch)
I20260709 13:02:13.376176  8190 raft_consensus.cc:2804] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 13:02:13.278613  8513 raft_consensus.cc:515] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:13.438293  7745 raft_consensus.cc:493] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
W20260709 13:02:13.475023  2299 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:13.469864  8564 raft_consensus.cc:493] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:13.476403  8354 raft_consensus.cc:493] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:13.477030  8627 consensus_queue.cc:1059] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 18.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.199s
W20260709 13:02:13.477985  8592 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.329s	user 0.013s	sys 0.024s
I20260709 13:02:13.478921  1605 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40154 (request call id 2828) took 1574 ms. Trace:
I20260709 13:02:13.480448  8585 raft_consensus.cc:493] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Starting leader election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:02:13.481696  8651 raft_consensus.cc:3055] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 21 LEADER]: Stepping down as leader of term 21
W20260709 13:02:13.483736  8669 log.cc:927] Time spent T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.208s	user 0.001s	sys 0.000s
W20260709 13:02:13.484989  8670 log.cc:927] Time spent T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.206s	user 0.000s	sys 0.000s
I20260709 13:02:13.487962  1596 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40154 (request call id 2829) took 1583 ms. Trace:
W20260709 13:02:13.493458  8668 log.cc:927] Time spent T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.536s	user 0.003s	sys 0.000s
I20260709 13:02:13.501029  8626 raft_consensus.cc:3060] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:02:13.504297  1675 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57092 (request call id 1516) took 1891 ms. Trace:
I20260709 13:02:13.510826  8472 raft_consensus.cc:2749] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:02:13.512565  8508 leader_election.cc:290] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:13.516247  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:02:13.535552  8673 log.cc:927] Time spent T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.061s	user 0.000s	sys 0.000s
I20260709 13:02:13.537726  1674 raft_consensus.cc:1240] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 21. Current term is 22. Ops: []
I20260709 13:02:13.211534  8602 consensus_queue.cc:1059] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 14.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.169s
I20260709 13:02:13.564986  1603 raft_consensus.cc:2468] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 22.
I20260709 13:02:13.577806  1718 leader_election.cc:304] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:13.443166  8648 raft_consensus.cc:493] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
W20260709 13:02:13.587786  2299 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:13.588580  2300 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:13.587949  8592 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.548s	user 0.013s	sys 0.024s
I20260709 13:02:13.605260  1596 rpcz_store.cc:276] 0709 13:02:11.904707 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:13.121794 (+1217087us) service_pool.cc:224] Handling call
0709 13:02:13.122362 (+   568us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:13.122882 (+   520us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:13.122911 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:13.123517 (+   606us) log.cc:844] Serialized 26 byte log entry
0709 13:02:13.277413 (+153896us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:02:13.277472 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:13.277564 (+    92us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:13.487719 (+210155us) raft_consensus.cc:1685] finished
0709 13:02:13.487742 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:13.487893 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":792,"raft.run_cpu_time_us":103,"raft.run_wall_time_us":103,"thread_start_us":153396,"threads_started":1,"wal-append.queue_time_us":257658}
I20260709 13:02:13.601019  8651 raft_consensus.cc:740] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:02:13.615396  1753 raft_consensus.cc:1275] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 16 index: 9. Preceding OpId from leader: term: 21 index: 11. (index mismatch)
I20260709 13:02:13.616158  8646 raft_consensus.cc:493] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:13.618490  8564 raft_consensus.cc:515] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:13.621764  1570 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:13.581240  8402 raft_consensus.cc:493] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:13.581535  1605 rpcz_store.cc:276] 0709 13:02:11.904465 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:13.144251 (+1239786us) service_pool.cc:224] Handling call
0709 13:02:13.181024 (+ 36773us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:13.214721 (+ 33697us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:13.214774 (+    53us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:13.221170 (+  6396us) log.cc:844] Serialized 26 byte log entry
0709 13:02:13.437934 (+216764us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:02:13.437988 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:13.438071 (+    83us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:13.478534 (+ 40463us) raft_consensus.cc:1685] finished
0709 13:02:13.478579 (+    45us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:13.478780 (+   201us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8156,"raft.queue_time_us":10491,"raft.run_cpu_time_us":177,"raft.run_wall_time_us":8218,"thread_start_us":180880,"threads_started":1,"wal-append.queue_time_us":229299}
I20260709 13:02:13.625082  1675 rpcz_store.cc:276] 0709 13:02:11.612413 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:11.612600 (+   187us) service_pool.cc:224] Handling call
0709 13:02:12.608847 (+996247us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:12.735216 (+126369us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:13.502885 (+767669us) spinlock_profiling.cc:243] Waited 126 ms on lock 0x7b54000230d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:02:13.503963 (+  1078us) spinlock_profiling.cc:243] Waited 996 ms on lock 0x7b54000230d0. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661618ad 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:02:13.504184 (+   221us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2580285056}
I20260709 13:02:13.650934  8648 raft_consensus.cc:515] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:13.637962  1750 raft_consensus.cc:2368] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 17. Current term is 18.
I20260709 13:02:13.599695  8585 raft_consensus.cc:3060] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:02:13.640956  1604 raft_consensus.cc:1240] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 21. Current term is 22. Ops: []
W20260709 13:02:13.642868  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 65200.9us
I20260709 13:02:13.643810  1603 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40166 (request call id 2921) took 1133 ms. Trace:
I20260709 13:02:13.579020  8634 consensus_queue.cc:1048] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.052s
I20260709 13:02:13.599426  8190 raft_consensus.cc:697] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 21 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:02:13.649190  8602 raft_consensus.cc:3055] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:02:13.650377  8609 raft_consensus.cc:2804] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:02:13.581132  8468 consensus_queue.cc:1048] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.054s
W20260709 13:02:13.634348  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 328787us
W20260709 13:02:13.658456  1754 consensus_meta.cc:287] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.517s	user 0.012s	sys 0.000s
I20260709 13:02:13.730253  1754 raft_consensus.cc:2468] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 20.
I20260709 13:02:13.708626  1603 rpcz_store.cc:276] 0709 13:02:12.504063 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:12.951552 (+447489us) service_pool.cc:224] Handling call
0709 13:02:13.637448 (+685896us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:13.637291  8650 raft_consensus.cc:493] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:13.596318  8354 raft_consensus.cc:515] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:13.509830  8551 consensus_queue.cc:260] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 15.14, Last appended by leader: 14, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:13.637210  8504 consensus_queue.cc:1048] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.091s
W20260709 13:02:13.746585  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 220189us
I20260709 13:02:13.716270  8360 raft_consensus.cc:493] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:13.716084  8602 raft_consensus.cc:740] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:02:13.730822  8609 raft_consensus.cc:493] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:13.722872  8652 leader_election.cc:290] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 19 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:13.641065  7745 raft_consensus.cc:515] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:13.763615  1673 raft_consensus.cc:2468] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 18.
I20260709 13:02:13.765954  8190 consensus_queue.cc:237] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 19.15, Last appended by leader: 15, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:13.766865  8564 leader_election.cc:290] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:13.767835  2337 raft_consensus.cc:1275] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
W20260709 13:02:13.768522  1602 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.512s	user 0.063s	sys 0.087s
W20260709 13:02:13.768877  1602 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.512s	user 0.063s	sys 0.087s
I20260709 13:02:13.769646  1602 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 2919) took 1287 ms. Trace:
I20260709 13:02:13.770823  8145 raft_consensus.cc:493] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:13.771294  1747 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55840 (request call id 3038) took 2319 ms (client timeout 2298 ms). Trace:
I20260709 13:02:13.771390  8145 raft_consensus.cc:515] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:13.771976  8354 leader_election.cc:290] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 13:02:13.771797  1747 rpcz_store.cc:269] 0709 13:02:11.451818 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:11.752730 (+300912us) service_pool.cc:224] Handling call
0709 13:02:13.771018 (+2018288us) spinlock_profiling.cc:243] Waited 920 ms on lock 0x7b5400384d54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:02:13.771197 (+   179us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2115045760}
W20260709 13:02:13.774477  8550 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.528s	user 0.024s	sys 0.051s
W20260709 13:02:13.774912  8550 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.722s	user 0.025s	sys 0.051s
I20260709 13:02:13.770016  1602 rpcz_store.cc:276] 0709 13:02:12.481926 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:13.155373 (+673447us) service_pool.cc:224] Handling call
0709 13:02:13.169163 (+ 13790us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:13.210303 (+ 41140us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:13.210353 (+    50us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:13.224698 (+ 14345us) log.cc:844] Serialized 26 byte log entry
0709 13:02:13.769262 (+544564us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:02:13.769306 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:13.769386 (+    80us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:13.769437 (+    51us) raft_consensus.cc:1685] finished
0709 13:02:13.769454 (+    17us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:13.769579 (+   125us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":32468,"raft.run_cpu_time_us":250,"raft.run_wall_time_us":21802,"thread_start_us":512632,"threads_started":1,"wal-append.queue_time_us":383879}
I20260709 13:02:13.775367  8360 raft_consensus.cc:515] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:02:13.776325  8373 consensus_meta.cc:287] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.631s	user 0.010s	sys 0.002s
W20260709 13:02:13.776540  8671 log.cc:927] Time spent T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.152s	user 0.001s	sys 0.000s
W20260709 13:02:13.777798  8681 log.cc:927] Time spent T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.167s	user 0.000s	sys 0.000s
I20260709 13:02:13.777925  8360 leader_election.cc:290] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:13.779019  8496 leader_election.cc:290] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 13:02:13.779886  8678 log.cc:927] Time spent T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.098s	user 0.000s	sys 0.001s
I20260709 13:02:13.780620  8323 raft_consensus.cc:493] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:13.781167  8323 raft_consensus.cc:515] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:13.782130  8607 raft_consensus.cc:3055] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 18 LEADER]: Stepping down as leader of term 18
I20260709 13:02:13.782567  8607 raft_consensus.cc:740] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
W20260709 13:02:13.779328  8603 consensus_meta.cc:287] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.737s	user 0.008s	sys 0.007s
I20260709 13:02:13.784231  2301 leader_election.cc:400] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 16 election: Vote denied by peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693) with higher term. Message: Invalid argument: T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 for earlier term 16. Current term is 17.
I20260709 13:02:13.784240  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
W20260709 13:02:13.787025  8672 log.cc:927] Time spent T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.134s	user 0.000s	sys 0.000s
I20260709 13:02:13.736148  8402 raft_consensus.cc:515] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:13.789314  8323 leader_election.cc:290] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:13.786489  1599 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40154 (request call id 2830) took 1878 ms. Trace:
I20260709 13:02:13.790788  1745 raft_consensus.cc:1240] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 20. Current term is 21. Ops: []
I20260709 13:02:13.791026  1599 rpcz_store.cc:276] 0709 13:02:11.908248 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:11.947704 (+ 39456us) service_pool.cc:224] Handling call
0709 13:02:13.786219 (+1838515us) spinlock_profiling.cc:243] Waited 505 ms on lock 0x7b54001027d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:02:13.786376 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1160845696}
I20260709 13:02:13.773983  8145 leader_election.cc:290] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:13.793450  8609 raft_consensus.cc:3060] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Advancing to term 20
W20260709 13:02:13.824739  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 182509us
W20260709 13:02:13.837354  1569 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:13.742830  8646 raft_consensus.cc:515] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:13.827486  8551 raft_consensus.cc:3060] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:02:13.762647  8650 raft_consensus.cc:515] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:13.870247  2299 leader_election.cc:304] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:13.839349  8651 consensus_queue.cc:260] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 21.13, Last appended by leader: 13, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:13.871371  8651 raft_consensus.cc:3060] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:02:13.871563  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:13.872357  8190 raft_consensus.cc:2804] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:02:13.872802  8190 raft_consensus.cc:493] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:02:13.873181  8190 raft_consensus.cc:3060] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Advancing to term 15
W20260709 13:02:13.874364  1644 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.107.196:46625 timed out after 1.558s (SENT) [suppressed 40 similar messages]
I20260709 13:02:13.876003  8650 leader_election.cc:290] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 13:02:13.877290  8404 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.375s	user 0.013s	sys 0.009s
W20260709 13:02:13.877663  8404 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.428s	user 0.013s	sys 0.009s
I20260709 13:02:13.879567  7745 leader_election.cc:290] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:02:13.882704  1600 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36462 (request call id 1722) took 1850 ms (client timeout 1691 ms). Trace:
W20260709 13:02:13.883191  1600 rpcz_store.cc:269] 0709 13:02:12.031873 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:12.032633 (+   760us) service_pool.cc:224] Handling call
0709 13:02:13.882263 (+1849630us) spinlock_profiling.cc:243] Waited 23.5 ms on lock 0x7b54002995d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:02:13.882457 (+   194us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":54030592}
W20260709 13:02:13.884130  1502 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:43476 (request call id 195) took 51987 ms (client timeout 15000 ms). Trace:
W20260709 13:02:13.903256  8675 log.cc:927] Time spent T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.162s	user 0.000s	sys 0.000s
W20260709 13:02:13.904711  8683 log.cc:927] Time spent T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.075s	user 0.000s	sys 0.000s
W20260709 13:02:13.906379  1643 leader_election.cc:336] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 15 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 1.630s (SENT)
I20260709 13:02:13.907043  1643 leader_election.cc:304] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
W20260709 13:02:13.908310  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 169437us
W20260709 13:02:13.908599  8692 log.cc:927] Time spent T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.066s	user 0.000s	sys 0.000s
I20260709 13:02:13.906183  1604 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40154 (request call id 2831) took 1997 ms. Trace:
I20260709 13:02:13.910761  1604 rpcz_store.cc:276] 0709 13:02:11.908517 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:13.143058 (+1234541us) service_pool.cc:224] Handling call
0709 13:02:13.637888 (+494830us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:13.713174 (+ 75286us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:13.905894 (+192720us) spinlock_profiling.cc:243] Waited 480 ms on lock 0x7b5400301950. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661618ad 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:02:13.906105 (+   211us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1105029376}
W20260709 13:02:13.911388  1570 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:13.863027  8358 consensus_meta.cc:287] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.973s	user 0.016s	sys 0.001s
I20260709 13:02:13.863782  8402 leader_election.cc:290] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:02:13.884542  1502 rpcz_store.cc:269] 0709 13:01:21.896907 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:21.960762 (+ 63855us) service_pool.cc:224] Handling call
0709 13:02:13.883823 (+51923061us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 13:02:02.957345 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 13:02:02.958220 (+   875us) write_op.cc:432] Acquiring schema lock in shared mode
0709 13:02:02.958308 (+    88us) write_op.cc:435] Acquired schema lock
0709 13:02:02.958342 (+    34us) tablet.cc:662] Decoding operations
0709 13:02:03.063092 (+104750us) write_op.cc:620] Acquiring the partition lock for write op
0709 13:02:03.063192 (+   100us) write_op.cc:641] Partition lock acquired for write op
0709 13:02:03.063224 (+    32us) tablet.cc:685] Acquiring locks for 8 operations
0709 13:02:03.064203 (+   979us) tablet.cc:701] Row locks acquired
0709 13:02:03.064261 (+    58us) write_op.cc:260] PREPARE: finished
0709 13:02:03.064486 (+   225us) write_op.cc:270] Start()
0709 13:02:03.064676 (+   190us) write_op.cc:276] Timestamp: P: 1783602123064405 usec, L: 0
0709 13:02:03.064700 (+    24us) op_driver.cc:348] REPLICATION: starting
0709 13:02:03.094707 (+ 30007us) log.cc:844] Serialized 3218 byte log entry
0709 13:02:03.547527 (+452820us) op_driver.cc:464] REPLICATION: finished
0709 13:02:03.800488 (+252961us) write_op.cc:301] APPLY: starting
0709 13:02:03.822292 (+ 21804us) tablet.cc:1367] starting BulkCheckPresence
0709 13:02:04.009595 (+187303us) tablet.cc:1370] finished BulkCheckPresence
0709 13:02:04.049508 (+ 39913us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 13:02:04.667625 (+618117us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 13:02:04.771043 (+103418us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=16,key_file_lookups=16,delta_file_lookups=16,mrs_lookups=0
0709 13:02:04.771097 (+    54us) write_op.cc:312] APPLY: finished
0709 13:02:04.771542 (+   445us) log.cc:844] Serialized 81 byte log entry
0709 13:02:06.409542 (+1638000us) spinlock_profiling.cc:243] Waited 24.2 ms on lock 0x7b4400022b4c. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f4262f9bc85 00007f4262f9c6a2 00007f4265d0e1db 00007f4265d09aaf 00007f42630580ac 00007f426beb2ab9 00007f426307469b 00007f426307293f 00007f4263074c02 00007f4265cfc20b
0709 13:02:06.899637 (+490095us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:02:06.906559 (+  6922us) write_op.cc:454] Released schema lock
0709 13:02:06.915220 (+  8661us) write_op.cc:341] FINISH: Updating metrics
0709 13:02:06.971752 (+ 56532us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:02:06.985374 (+ 13622us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":2389909,"reported_tablets":36,"child_traces":[["op",{"apply.queue_time_us":180964,"apply.run_cpu_time_us":43611,"apply.run_wall_time_us":3158469,"cfile_cache_hit":34,"cfile_cache_hit_bytes":37631,"num_ops":8,"prepare.queue_time_us":217023,"prepare.run_cpu_time_us":14319,"prepare.run_wall_time_us":237333,"raft.queue_time_us":186440,"raft.run_cpu_time_us":25147,"raft.run_wall_time_us":253447,"replication_time_us":443697,"spinlock_wait_cycles":55706880,"thread_start_us":2427298,"threads_started":5,"wal-append.queue_time_us":2273921,"wal-append.run_cpu_time_us":23409,"wal-append.run_wall_time_us":2376937}]]}
I20260709 13:02:13.925880  8648 leader_election.cc:290] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:13.910405  8629 raft_consensus.cc:2749] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:02:13.929493  1569 leader_election.cc:304] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
W20260709 13:02:13.910478  8632 log.cc:927] Time spent T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 1.902s	user 0.004s	sys 0.002s
W20260709 13:02:13.931876  1643 leader_election.cc:336] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 1.631s (SENT)
I20260709 13:02:13.934499  1643 leader_election.cc:304] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:02:13.936349  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:13.936543  8507 raft_consensus.cc:493] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:02:13.936255  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 17 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:13.937366  8507 raft_consensus.cc:515] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:13.938004  1749 raft_consensus.cc:2468] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 16.
I20260709 13:02:13.937808  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 17 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:13.938643  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 20 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:13.939548  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:13.940963  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 17 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:13.941550  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:13.941772  8472 consensus_queue.cc:1059] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 18.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 13:02:13.944017  8472 raft_consensus.cc:3055] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 13:02:13.944375  8472 raft_consensus.cc:740] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:02:13.945362  8472 consensus_queue.cc:260] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.12, Last appended by leader: 12, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:13.947909  1643 leader_election.cc:336] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 16 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 1.669s (SENT)
I20260709 13:02:13.948616  1643 leader_election.cc:304] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:13.950737  8696 raft_consensus.cc:2749] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:02:13.951598  8696 raft_consensus.cc:2749] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:02:13.955133  1569 leader_election.cc:304] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:02:13.956673  1569 leader_election.cc:304] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
W20260709 13:02:13.957182  8664 log.cc:927] Time spent T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 1.099s	user 0.003s	sys 0.007s
I20260709 13:02:13.957847  8360 raft_consensus.cc:2804] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:02:13.958334  8360 raft_consensus.cc:493] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:13.958725  8360 raft_consensus.cc:3060] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Advancing to term 15
W20260709 13:02:13.959782  8685 log.cc:927] Time spent T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.123s	user 0.000s	sys 0.001s
W20260709 13:02:13.961319  2299 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:13.963083  8677 log.cc:927] Time spent T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.089s	user 0.001s	sys 0.000s
I20260709 13:02:13.963953  8507 leader_election.cc:290] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 13:02:13.964841  8344 consensus_meta.cc:287] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.841s	user 0.015s	sys 0.000s
I20260709 13:02:13.965464  8344 raft_consensus.cc:2749] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:02:13.965973  8700 raft_consensus.cc:2804] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:02:13.916412  8602 consensus_queue.cc:260] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:13.964788  7745 raft_consensus.cc:2804] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:02:13.966782  8513 leader_election.cc:290] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:13.967206  7745 raft_consensus.cc:493] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:13.967684  7745 raft_consensus.cc:3060] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Advancing to term 21
W20260709 13:02:13.977249  1643 leader_election.cc:336] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 1.806s (SENT)
I20260709 13:02:13.978471  8700 raft_consensus.cc:493] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:13.979444  8513 raft_consensus.cc:493] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:13.982810  8700 raft_consensus.cc:3060] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:02:13.983072  8513 raft_consensus.cc:515] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:13.984676  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:13.985554  8513 leader_election.cc:290] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:02:13.985719  8403 consensus_meta.cc:287] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.858s	user 0.006s	sys 0.006s
I20260709 13:02:13.986752  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:13.987188  8646 leader_election.cc:290] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:13.980154  8323 consensus_queue.cc:1059] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 17.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 13:02:13.967692  8602 raft_consensus.cc:3060] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:02:13.988133  8323 consensus_queue.cc:1243] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer 816fadcea8fc4893882f5f9b9f22a03d log is divergent from this leader: its last log entry 18.13 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 12
I20260709 13:02:13.988866  8323 raft_consensus.cc:3055] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 13:02:13.989413  8323 raft_consensus.cc:740] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
W20260709 13:02:13.980813  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:13.988813  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 17 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:13.991247  1751 raft_consensus.cc:1240] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 15. Current term is 16. Ops: []
I20260709 13:02:13.991873  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:13.991765  8323 consensus_queue.cc:260] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.12, Last appended by leader: 12, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:13.991381  1754 raft_consensus.cc:2393] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 17: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:02:13.993487  8323 raft_consensus.cc:3060] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:02:13.994630  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:13.994810  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:13.995409  1680 raft_consensus.cc:2393] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 15: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:02:13.998070  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:14.001194  1751 raft_consensus.cc:2393] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 20: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:02:14.001513  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:14.002493  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:14.003811  1644 leader_election.cc:304] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:02:14.005026  1676 raft_consensus.cc:1275] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 18 index: 12. Preceding OpId from leader: term: 19 index: 13. (index mismatch)
I20260709 13:02:14.005007  8696 raft_consensus.cc:2749] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
W20260709 13:02:14.006204  8514 consensus_meta.cc:287] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.802s	user 0.011s	sys 0.004s
I20260709 13:02:14.006624  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:14.007290  8472 raft_consensus.cc:3060] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:02:14.007421  1679 raft_consensus.cc:2468] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 14.
I20260709 13:02:14.008735  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:14.009492  1673 raft_consensus.cc:2468] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 13.
I20260709 13:02:14.010982  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:14.013193  1598 raft_consensus.cc:1275] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 20 index: 13. (index mismatch)
I20260709 13:02:13.981349  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 17 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:14.014855  1604 raft_consensus.cc:2468] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 16.
I20260709 13:02:14.016417  1596 raft_consensus.cc:1275] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 18 index: 10. Preceding OpId from leader: term: 21 index: 11. (index mismatch)
I20260709 13:02:14.017489  8700 raft_consensus.cc:515] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:14.018707  1602 raft_consensus.cc:2368] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for earlier term 19. Current term is 20.
W20260709 13:02:14.021421  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 64957.3us
I20260709 13:02:14.022324  1598 raft_consensus.cc:1275] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 18 index: 10. Preceding OpId from leader: term: 19 index: 11. (index mismatch)
I20260709 13:02:14.023731  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:14.024415  1600 raft_consensus.cc:3060] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:02:14.024068  8550 consensus_queue.cc:1059] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 16.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 13:02:14.025209  8550 consensus_queue.cc:1243] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer 816fadcea8fc4893882f5f9b9f22a03d log is divergent from this leader: its last log entry 17.11 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 10
I20260709 13:02:14.026016  8358 raft_consensus.cc:3055] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 LEADER]: Stepping down as leader of term 16
I20260709 13:02:14.026422  8358 raft_consensus.cc:740] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
W20260709 13:02:14.027586  1719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:14.029192  1596 raft_consensus.cc:1275] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 18 index: 9. Preceding OpId from leader: term: 19 index: 10. (index mismatch)
I20260709 13:02:14.036876  1745 raft_consensus.cc:1240] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 18. Current term is 19. Ops: []
I20260709 13:02:14.037081  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:14.037751  1752 raft_consensus.cc:2468] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 19.
I20260709 13:02:14.033877  1494 catalog_manager.cc:5697] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 8 to 11, leader changed from 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194) to af520b67945d443b901bbeca37cc5701 (127.0.107.195). New cstate: current_term: 11 leader_uuid: "af520b67945d443b901bbeca37cc5701" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } }
I20260709 13:02:14.039661  1494 catalog_manager.cc:5697] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 9 to 13. New cstate: current_term: 13 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } }
I20260709 13:02:13.997989  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:14.041373  1680 raft_consensus.cc:2393] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 19: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:02:14.042003  1494 catalog_manager.cc:5697] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 7 to 12. New cstate: current_term: 12 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } }
I20260709 13:02:14.040632  1753 raft_consensus.cc:1275] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 18 index: 13. (index mismatch)
I20260709 13:02:14.028644  8358 consensus_queue.cc:260] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.10, Last appended by leader: 10, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } }
I20260709 13:02:14.044231  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:14.046069  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:14.047050  1677 raft_consensus.cc:1240] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 17. Current term is 18. Ops: []
I20260709 13:02:14.049568  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 17 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:14.051301  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 17 candidate_status { last_received { term: 16 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:02:14.060174  1644 leader_election.cc:336] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 16 election: RPC error from VoteRequest() call to peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625): Timed out: RequestConsensusVote RPC to 127.0.107.196:46625 timed out after 1.592s (SENT)
I20260709 13:02:14.039992  8190 raft_consensus.cc:515] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:14.075346  1674 raft_consensus.cc:2393] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 20: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:02:14.083257  1644 leader_election.cc:304] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:14.085863  7745 raft_consensus.cc:515] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:14.081066  1679 raft_consensus.cc:2468] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 16.
W20260709 13:02:14.029217  8623 consensus_meta.cc:287] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.586s	user 0.005s	sys 0.007s
I20260709 13:02:14.091030  1598 raft_consensus.cc:2468] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 16.
I20260709 13:02:14.093037  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:14.093721  1747 raft_consensus.cc:2468] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 16.
I20260709 13:02:14.095697  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:14.096853  1646 leader_election.cc:304] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:14.097940  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 18 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:14.098627  1677 raft_consensus.cc:2468] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 17.
I20260709 13:02:14.099174  8537 raft_consensus.cc:2804] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:02:14.099057  8513 consensus_queue.cc:1059] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 18.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 13:02:14.099612  8537 raft_consensus.cc:493] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:14.074683  8404 consensus_queue.cc:1048] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.034s
I20260709 13:02:14.096274  1494 catalog_manager.cc:5697] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 9 to 12, leader changed from 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) to 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194). New cstate: current_term: 12 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 13:02:14.100383  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:14.107695  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:14.107633  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:14.111922  1604 raft_consensus.cc:2393] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 20: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:02:14.111829  1749 raft_consensus.cc:2468] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 20.
W20260709 13:02:14.100616  1643 leader_election.cc:336] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 1.883s (SENT)
I20260709 13:02:14.107250  2299 leader_election.cc:304] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:02:14.093034  1718 leader_election.cc:304] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:14.100646  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:14.109150  8190 leader_election.cc:290] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 15 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:13.986217  8403 raft_consensus.cc:515] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:14.118893  1605 raft_consensus.cc:2393] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 18: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:02:14.006721  8514 raft_consensus.cc:515] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:14.100000  8537 raft_consensus.cc:3060] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:02:14.127278  2298 leader_election.cc:304] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 13:02:14.131135  1643 leader_election.cc:336] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 14 election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 1.959s (SENT)
I20260709 13:02:14.112353  8496 raft_consensus.cc:493] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:14.105187  8700 leader_election.cc:290] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 20 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:14.131906  7745 leader_election.cc:290] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 21 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:14.103188  8698 raft_consensus.cc:493] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:14.118485  8646 raft_consensus.cc:2804] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 13:02:14.135313  2299 leader_election.cc:304] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:02:14.121501  8652 raft_consensus.cc:493] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:14.100085  8504 consensus_queue.cc:1059] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.031s
I20260709 13:02:14.125442  8358 raft_consensus.cc:3060] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:02:14.125316  8696 raft_consensus.cc:2749] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 13:02:14.144481  2299 leader_election.cc:304] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:02:14.140450  7745 consensus_queue.cc:1059] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 18.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 13:02:14.130959  8303 raft_consensus.cc:493] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:14.099723  8585 raft_consensus.cc:515] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:14.132579  8623 raft_consensus.cc:2804] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Leader pre-election won for term 15
W20260709 13:02:14.147815  1718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:14.132989  8344 raft_consensus.cc:493] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:14.122138  8461 consensus_queue.cc:1048] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.020s
I20260709 13:02:14.126394  8404 consensus_queue.cc:1048] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.020s
W20260709 13:02:14.159842  1718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:14.159859  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 83677.9us
I20260709 13:02:14.144616  8609 raft_consensus.cc:515] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:14.164214  1750 raft_consensus.cc:2393] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 20: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:02:14.165179  8609 leader_election.cc:290] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 20 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:14.151335  8623 raft_consensus.cc:493] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:14.170168  1599 raft_consensus.cc:2468] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 16.
I20260709 13:02:14.170828  8603 raft_consensus.cc:2749] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 13:02:14.170197  8623 raft_consensus.cc:3060] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:02:14.171978  8585 leader_election.cc:290] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 17 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:14.150823  8646 raft_consensus.cc:697] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 18 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:02:14.174574  8344 raft_consensus.cc:515] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:14.141955  8496 raft_consensus.cc:515] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:14.175464  8646 consensus_queue.cc:237] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.11, Last appended by leader: 11, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:14.146585  8507 raft_consensus.cc:2804] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:02:14.180425  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:02:14.180668  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 53529.1us
I20260709 13:02:14.156364  8700 raft_consensus.cc:3055] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 LEADER]: Stepping down as leader of term 18
W20260709 13:02:14.194111  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 134038us
I20260709 13:02:14.184882  8368 raft_consensus.cc:2804] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:02:14.203449  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:14.187613  8507 raft_consensus.cc:493] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:02:14.204180  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 59804.9us
I20260709 13:02:14.153716  8652 raft_consensus.cc:515] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:14.205771  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 114422us
I20260709 13:02:14.154690  8303 raft_consensus.cc:515] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:14.206782  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:14.219830  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:14.222163  8360 raft_consensus.cc:515] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:14.199272  8403 leader_election.cc:290] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 17 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:14.196934  8700 raft_consensus.cc:740] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:02:14.198724  8368 raft_consensus.cc:493] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:14.158414  8698 raft_consensus.cc:515] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:14.212366  8507 raft_consensus.cc:3060] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:02:14.193027  8609 consensus_queue.cc:1048] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.031s
I20260709 13:02:14.202963  8190 raft_consensus.cc:493] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:14.289121  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:02:14.290427  2299 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:14.298907  1643 leader_election.cc:336] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 14 election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 2.016s (SENT)
I20260709 13:02:14.207386  8706 consensus_queue.cc:1059] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 19.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.067s
W20260709 13:02:14.307317  1570 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:14.313588  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:14.316812  1678 raft_consensus.cc:2410] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for term 20 because replica has last-logged OpId of term: 19 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 14.
I20260709 13:02:14.318099  1643 leader_election.cc:304] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:14.319005  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 18 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:14.319629  2334 raft_consensus.cc:2468] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 17.
I20260709 13:02:14.176709  1494 catalog_manager.cc:5697] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 10 to 12. New cstate: current_term: 12 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 13:02:14.295967  8461 raft_consensus.cc:493] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:14.325987  1644 leader_election.cc:336] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 15 pre-election: RPC error from VoteRequest() call to peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625): Timed out: RequestConsensusVote RPC to 127.0.107.196:46625 timed out after 1.703s (SENT)
I20260709 13:02:14.339148  1718 leader_election.cc:304] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:14.339474  1644 leader_election.cc:304] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:14.327466  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:14.329155  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:02:14.338340  2299 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:14.329511  8698 leader_election.cc:290] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:14.330039  8373 raft_consensus.cc:3055] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 LEADER]: Stepping down as leader of term 19
I20260709 13:02:14.330919  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:14.331048  8700 consensus_queue.cc:260] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.12, Last appended by leader: 12, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:14.331642  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:14.331781  1748 raft_consensus.cc:1240] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 18. Current term is 19. Ops: []
W20260709 13:02:14.333664  1571 leader_election.cc:336] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 16 pre-election: RPC error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Timed out: RequestConsensusVote RPC to 127.0.107.195:40693 timed out after 1.773s (SENT)
I20260709 13:02:14.348495  1571 leader_election.cc:304] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:02:14.299189  8368 raft_consensus.cc:3060] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:02:14.335429  1753 raft_consensus.cc:2410] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for term 15 because replica has last-logged OpId of term: 14 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 8.
I20260709 13:02:14.330058  2333 raft_consensus.cc:1275] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 16 index: 13. Preceding OpId from leader: term: 20 index: 15. (index mismatch)
I20260709 13:02:14.326862  2332 raft_consensus.cc:1275] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 17 index: 14. Preceding OpId from leader: term: 20 index: 15. (index mismatch)
I20260709 13:02:14.316067  8506 raft_consensus.cc:493] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:02:14.351826  1749 raft_consensus.cc:2393] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 19: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:02:14.345894  1754 raft_consensus.cc:1240] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 17. Current term is 18. Ops: []
I20260709 13:02:14.347620  2331 raft_consensus.cc:1240] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 16. Current term is 17. Ops: []
I20260709 13:02:14.334275  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:14.347339  8373 raft_consensus.cc:740] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:02:14.355894  8551 raft_consensus.cc:2749] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:02:14.355705  8373 consensus_queue.cc:260] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 19.15, Last appended by leader: 15, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:14.357096  8373 raft_consensus.cc:3060] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:02:14.358952  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:14.359741  2334 raft_consensus.cc:2393] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 19: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:02:14.359982  2329 raft_consensus.cc:2393] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 20: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:02:14.360579  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:14.360844  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 16 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:14.361356  2334 raft_consensus.cc:2393] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 20: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:02:14.363947  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:14.365160  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:14.366582  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:14.367324  8303 leader_election.cc:290] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:14.367799  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:14.368996  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:02:14.369321  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:14.369742  2328 raft_consensus.cc:2393] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 15: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:02:14.343289  8323 raft_consensus.cc:493] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:02:14.371230  8323 raft_consensus.cc:515] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:14.372859  1745 raft_consensus.cc:1240] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 19 LEADER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 18. Current term is 19. Ops: []
I20260709 13:02:14.373589  8323 leader_election.cc:290] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:14.343748  8514 leader_election.cc:290] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:14.375906  8360 leader_election.cc:290] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 15 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:14.376777  8496 leader_election.cc:290] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:14.378135  2301 leader_election.cc:304] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:14.378862  8623 raft_consensus.cc:515] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:14.381105  8623 leader_election.cc:290] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 15 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:14.382532  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:14.383204  2332 raft_consensus.cc:2468] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 19.
W20260709 13:02:14.383291  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 176176us
W20260709 13:02:14.384140  1643 leader_election.cc:336] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 16 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 2.030s (SENT)
W20260709 13:02:14.385666  1643 leader_election.cc:336] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 2.051s (SENT)
I20260709 13:02:14.387257  2330 raft_consensus.cc:1275] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 13 index: 12. Preceding OpId from leader: term: 14 index: 13. (index mismatch)
I20260709 13:02:14.388324  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:14.388630  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:14.388890  2330 raft_consensus.cc:2376] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 17. Re-sending same reply.
I20260709 13:02:14.389618  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:14.389904  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:14.390257  1675 raft_consensus.cc:2468] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 18.
I20260709 13:02:14.391249  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:14.391935  1678 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52778 (request call id 2693) took 2255 ms. Trace:
I20260709 13:02:14.384157  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:14.356807  8506 raft_consensus.cc:515] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:14.354489  1600 raft_consensus.cc:2468] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 20.
I20260709 13:02:14.394573  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:14.395453  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:14.392340  1678 rpcz_store.cc:276] 0709 13:02:12.136604 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:12.569719 (+433115us) service_pool.cc:224] Handling call
0709 13:02:14.391719 (+1822000us) spinlock_profiling.cc:243] Waited 1.7 s on lock 0x7b540025b054. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:02:14.391869 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3919641856}
I20260709 13:02:14.362186  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:14.406226  2334 raft_consensus.cc:2393] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 20: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:02:14.346805  8602 raft_consensus.cc:2804] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:02:14.345502  8706 raft_consensus.cc:493] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:14.348731  8504 raft_consensus.cc:2749] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 13:02:14.434019  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:14.335882  8190 raft_consensus.cc:515] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:14.436316  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 18 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:14.404673  1569 leader_election.cc:304] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:02:14.444820  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:14.414722  8700 raft_consensus.cc:3060] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:02:14.445806  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:14.449621  1679 raft_consensus.cc:2468] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 14.
I20260709 13:02:14.449105  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:14.350252  8703 raft_consensus.cc:2749] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:02:14.454458  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:14.456212  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:14.445487  1671 raft_consensus.cc:1275] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 18 index: 14. Preceding OpId from leader: term: 20 index: 15. (index mismatch)
I20260709 13:02:14.450116  2328 raft_consensus.cc:2393] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 19: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:02:14.449393  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:14.405232  8360 raft_consensus.cc:493] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:02:14.397212  8698 raft_consensus.cc:493] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:14.454537  2334 raft_consensus.cc:2393] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 20: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:02:14.459919  1673 raft_consensus.cc:3060] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:02:14.459820  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 17 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:14.460294  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:14.461019  2334 raft_consensus.cc:3060] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:02:14.461570  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:14.461912  1678 raft_consensus.cc:2410] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for term 17 because replica has last-logged OpId of term: 15 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 10.
I20260709 13:02:14.462195  2336 raft_consensus.cc:3060] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:02:14.451331  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
W20260709 13:02:14.453181  1569 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:14.463125  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 137241us
I20260709 13:02:14.437583  8602 raft_consensus.cc:493] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:02:14.465863  1644 leader_election.cc:336] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625): Timed out: RequestConsensusVote RPC to 127.0.107.196:46625 timed out after 1.751s (SENT)
W20260709 13:02:14.468021  1569 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:14.410657  8711 consensus_queue.cc:1059] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 16.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.056s
I20260709 13:02:14.469128  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:14.469749  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
W20260709 13:02:14.468739  1569 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:14.470585  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:02:14.471451  1569 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:14.444586  8496 raft_consensus.cc:2804] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:02:14.453814  8585 raft_consensus.cc:2804] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:02:14.472225  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:14.475613  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:14.409662  8461 raft_consensus.cc:515] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:14.478783  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:14.480780  1678 raft_consensus.cc:3060] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:02:14.481132  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:14.482796  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:14.456967  8537 raft_consensus.cc:515] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:14.489233  8710 log.cc:927] Time spent T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.056s	user 0.000s	sys 0.001s
I20260709 13:02:14.485790  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:14.490847  1749 raft_consensus.cc:3060] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:02:14.371304  8607 consensus_queue.cc:260] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 18.9, Last appended by leader: 9, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:14.485651  2330 raft_consensus.cc:2376] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 21. Re-sending same reply.
W20260709 13:02:14.492288  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 114317us
I20260709 13:02:14.470027  8602 raft_consensus.cc:3060] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Advancing to term 18
W20260709 13:02:14.474715  8708 log.cc:927] Time spent T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.150s	user 0.001s	sys 0.000s
I20260709 13:02:14.497360  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:14.454103  8609 consensus_queue.cc:1048] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.029s
I20260709 13:02:14.475031  8496 raft_consensus.cc:493] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:14.499699  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:14.502583  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 22 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:14.504693  2301 leader_election.cc:304] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:02:14.476481  8585 raft_consensus.cc:493] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:14.506479  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:14.499574  8537 leader_election.cc:290] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 20 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:14.484571  8360 raft_consensus.cc:515] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:02:14.508329  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 102864us
I20260709 13:02:14.466415  8698 raft_consensus.cc:515] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:14.509056  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:14.515407  1643 leader_election.cc:304] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:02:14.516361  1673 raft_consensus.cc:2468] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 15.
W20260709 13:02:14.517959  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 52158.8us
I20260709 13:02:14.518187  8490 raft_consensus.cc:493] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:14.518756  8490 raft_consensus.cc:515] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:02:14.523138  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 189644us
W20260709 13:02:14.545905  1571 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:14.526840  8652 leader_election.cc:290] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:14.527298  8623 raft_consensus.cc:2749] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:02:14.530292  8360 leader_election.cc:290] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:14.531932  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:14.533967  8626 raft_consensus.cc:3055] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 LEADER]: Stepping down as leader of term 20
I20260709 13:02:14.503787  8496 raft_consensus.cc:3060] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:02:14.512380  8585 raft_consensus.cc:3060] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:02:14.495236  8706 raft_consensus.cc:515] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:14.511199  2298 leader_election.cc:304] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:02:14.525856  2328 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c"
dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
mode: GRACEFUL
new_leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
 from {username='slave'} at 127.0.0.1:35790
I20260709 13:02:14.543228  8190 leader_election.cc:290] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:14.543082  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 22 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:14.546577  2333 raft_consensus.cc:2468] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 18.
I20260709 13:02:14.547624  2332 raft_consensus.cc:2468] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 18.
I20260709 13:02:14.558917  8607 raft_consensus.cc:3060] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:02:14.549391  1677 raft_consensus.cc:1275] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20260709 13:02:14.555411  1746 raft_consensus.cc:2468] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 22.
W20260709 13:02:14.556114  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:14.556087  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:14.561733  2334 raft_consensus.cc:2468] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 20.
I20260709 13:02:14.469885  1494 catalog_manager.cc:5697] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 12 to 16, leader changed from 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) to 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193). New cstate: current_term: 16 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:02:14.583287  1671 raft_consensus.cc:2468] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 21.
I20260709 13:02:14.567981  8506 leader_election.cc:290] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:14.568729  8626 raft_consensus.cc:740] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
W20260709 13:02:14.605039  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 100509us
I20260709 13:02:14.594282  1678 raft_consensus.cc:2468] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 20.
I20260709 13:02:14.595319  1749 raft_consensus.cc:2468] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 19.
W20260709 13:02:14.581223  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 73316.4us
I20260709 13:02:14.610095  2298 leader_election.cc:304] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:02:14.611580  8623 raft_consensus.cc:2804] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:02:14.611994  8623 raft_consensus.cc:493] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:14.585320  2332 raft_consensus.cc:1275] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 18 index: 14. Preceding OpId from leader: term: 19 index: 15. (index mismatch)
I20260709 13:02:14.612368  8623 raft_consensus.cc:3060] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:02:14.585807  2331 raft_consensus.cc:1275] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 18 index: 13. (index mismatch)
I20260709 13:02:14.602295  2336 raft_consensus.cc:2468] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 20.
I20260709 13:02:14.580739  1596 raft_consensus.cc:1275] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 17 index: 11. (index mismatch)
I20260709 13:02:14.614295  2333 raft_consensus.cc:1240] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 19. Current term is 20. Ops: []
I20260709 13:02:14.582906  2328 raft_consensus.cc:606] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [term 16 LEADER]: Received request to transfer leadership to TS 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 13:02:14.577095  8712 log.cc:927] Time spent T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.108s	user 0.000s	sys 0.000s
I20260709 13:02:14.612388  2298 leader_election.cc:304] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 13:02:14.629585  1646 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:14.629802  1754 raft_consensus.cc:1275] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 19 index: 15. Preceding OpId from leader: term: 21 index: 16. (index mismatch)
I20260709 13:02:14.630121  2298 leader_election.cc:304] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:02:14.581898  8711 raft_consensus.cc:2804] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 13:02:14.574316  8461 leader_election.cc:290] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 13:02:14.615447  8472 consensus_meta.cc:287] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.608s	user 0.012s	sys 0.003s
W20260709 13:02:14.636385  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 91366.1us
I20260709 13:02:14.576555  8706 leader_election.cc:290] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:14.576162  8550 raft_consensus.cc:493] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:02:14.576016  8714 raft_consensus.cc:493] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:14.639603  1646 leader_election.cc:304] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:14.577214  8698 leader_election.cc:290] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:14.635072  1494 catalog_manager.cc:5697] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 13 to 15. New cstate: current_term: 15 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 13:02:14.642994  2301 leader_election.cc:304] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:02:14.615685  8468 raft_consensus.cc:2804] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:02:14.650949  2335 raft_consensus.cc:2468] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 13.
I20260709 13:02:14.656725  1718 leader_election.cc:304] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:02:14.677671  1718 leader_election.cc:304] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:14.574420  8368 raft_consensus.cc:515] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:14.665858  1644 leader_election.cc:336] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625): Timed out: RequestConsensusVote RPC to 127.0.107.196:46625 timed out after 2.006s (SENT)
I20260709 13:02:14.590276  8507 raft_consensus.cc:515] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:14.674755  2337 raft_consensus.cc:2468] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 19.
I20260709 13:02:14.636061  8403 raft_consensus.cc:2749] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:02:14.682070  1679 raft_consensus.cc:1275] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 19 index: 15. Preceding OpId from leader: term: 21 index: 16. (index mismatch)
I20260709 13:02:14.658802  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:14.739784  2333 raft_consensus.cc:2393] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 18: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:02:14.677181  1569 leader_election.cc:304] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:02:14.729053  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:14.648355  8711 raft_consensus.cc:697] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 17 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:02:14.746847  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:14.674283  8537 raft_consensus.cc:2804] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Leader election won for term 19
I20260709 13:02:14.675254  8468 raft_consensus.cc:493] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Starting leader election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:02:14.751933  2328 raft_consensus.cc:1240] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 17. Current term is 18. Ops: [17.13-17.13]
W20260709 13:02:14.760098  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 249615us
I20260709 13:02:14.685541  8694 raft_consensus.cc:2749] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
W20260709 13:02:14.760901  1718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:14.728456  8373 raft_consensus.cc:493] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:14.763351  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 120510us
W20260709 13:02:14.765535  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 99971.6us
W20260709 13:02:14.767210  1644 leader_election.cc:336] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625): Timed out: RequestConsensusVote RPC to 127.0.107.196:46625 timed out after 1.646s (SENT)
W20260709 13:02:14.766625  1718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:14.784351  2329 raft_consensus.cc:2368] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for earlier term 16. Current term is 17.
I20260709 13:02:14.736907  8698 raft_consensus.cc:2804] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 13:02:14.737560  8358 raft_consensus.cc:493] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:14.678164  8461 consensus_queue.cc:1048] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 12, Time since last communication: 0.081s
I20260709 13:02:14.678308  8609 consensus_queue.cc:1059] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.081s
I20260709 13:02:14.787685  8354 raft_consensus.cc:2749] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:02:14.742111  8360 raft_consensus.cc:493] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:14.794919  8698 raft_consensus.cc:697] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 20 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:02:14.763281  8468 raft_consensus.cc:3060] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:02:14.683871  8714 raft_consensus.cc:515] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:14.801136  2301 leader_election.cc:304] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:02:14.802500  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:14.809581  8698 consensus_queue.cc:237] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:14.738859  8550 raft_consensus.cc:515] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:14.810403  8507 leader_election.cc:290] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 17 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:14.813503  8711 consensus_queue.cc:237] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:14.816318  1705 heartbeater.cc:646] Failed to heartbeat to 127.0.107.254:33643 (5 consecutive failures): Timed out: Failed to send heartbeat to master: TSHeartbeat RPC to 127.0.107.254:33643 timed out after 15.000s (SENT)
I20260709 13:02:14.817425  1643 leader_election.cc:304] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:02:14.818615  8358 raft_consensus.cc:515] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:14.797020  8706 raft_consensus.cc:2749] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:02:14.726819  8626 consensus_queue.cc:260] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 20.13, Last appended by leader: 13, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:14.789484  8373 raft_consensus.cc:515] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:14.839399  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
W20260709 13:02:14.842845  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 165828us
I20260709 13:02:14.827371  8603 raft_consensus.cc:2749] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:02:14.838687  8504 raft_consensus.cc:2804] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Leader pre-election won for term 16
W20260709 13:02:14.859023  1644 leader_election.cc:336] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625): Timed out: RequestConsensusVote RPC to 127.0.107.196:46625 timed out after 2.127s (SENT)
W20260709 13:02:14.869590  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 206218us
I20260709 13:02:14.876426  2299 leader_election.cc:304] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:02:14.831369  8360 raft_consensus.cc:515] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:14.881739  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:14.892287  8490 leader_election.cc:290] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:14.892524  8461 consensus_queue.cc:1048] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 13:02:14.897768  8550 leader_election.cc:290] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:14.900735  1643 leader_election.cc:304] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:02:14.902367  1643 leader_election.cc:304] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:02:14.904906  1643 leader_election.cc:304] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:02:14.905048  8303 raft_consensus.cc:2749] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 13:02:14.906055  8688 raft_consensus.cc:2804] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Leader election won for term 22
I20260709 13:02:14.906720  8688 raft_consensus.cc:697] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:02:14.907655  8688 consensus_queue.cc:237] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 21.13, Last appended by leader: 13, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:14.908838  8323 raft_consensus.cc:3055] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 13:02:14.909246  8323 raft_consensus.cc:740] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:02:14.910156  8323 consensus_queue.cc:260] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.12, Last appended by leader: 12, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:14.869544  8626 raft_consensus.cc:3060] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:02:14.869289  8504 raft_consensus.cc:493] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Starting leader election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:02:14.762253  1494 catalog_manager.cc:5697] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 8 to 13, leader changed from 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) to 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193). New cstate: current_term: 13 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 13:02:14.915910  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 56956.3us
I20260709 13:02:14.920805  8714 leader_election.cc:290] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:14.919987  8368 leader_election.cc:290] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 18 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:14.920609  8507 raft_consensus.cc:2804] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Leader pre-election won for term 19
W20260709 13:02:14.874832  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 108087us
I20260709 13:02:14.881283  8730 raft_consensus.cc:493] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:14.951447  1571 leader_election.cc:304] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:14.948146  2299 leader_election.cc:304] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:02:14.976855  8726 raft_consensus.cc:493] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:14.977003  1717 leader_election.cc:304] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:02:14.870400  8585 raft_consensus.cc:515] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:14.982784  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:14.977628  8726 raft_consensus.cc:515] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:14.952787  1569 leader_election.cc:304] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
W20260709 13:02:14.947197  8705 log.cc:927] Time spent T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.070s	user 0.000s	sys 0.001s
I20260709 13:02:14.935207  8504 raft_consensus.cc:3060] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:02:14.988366  1751 raft_consensus.cc:2468] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 16.
W20260709 13:02:14.988921  1644 leader_election.cc:336] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 18 election: RPC error from VoteRequest() call to peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625): Timed out: RequestConsensusVote RPC to 127.0.107.196:46625 timed out after 2.213s (SENT)
I20260709 13:02:14.992919  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:14.953452  8507 raft_consensus.cc:493] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:14.952293  8323 raft_consensus.cc:3060] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:02:15.004648  1643 leader_election.cc:400] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 election: Vote denied by peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445) with higher term. Message: Invalid argument: T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for earlier term 19. Current term is 20.
I20260709 13:02:15.008309  1644 leader_election.cc:304] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:02:15.009094  1600 raft_consensus.cc:2393] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 15: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:02:15.022408  1643 leader_election.cc:403] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 election: Cancelling election due to peer responding with higher term
I20260709 13:02:15.024044  1599 raft_consensus.cc:1240] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 20. Current term is 21. Ops: []
W20260709 13:02:15.011387  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 60055us
I20260709 13:02:14.986912  8490 raft_consensus.cc:2749] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
W20260709 13:02:15.024127  8724 log.cc:927] Time spent T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.134s	user 0.000s	sys 0.001s
I20260709 13:02:15.008666  8651 raft_consensus.cc:2804] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 13:02:15.014463  8507 raft_consensus.cc:3060] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:02:15.048235  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 17 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:15.048863  1599 raft_consensus.cc:3060] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Advancing to term 17
W20260709 13:02:15.045079  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 55969.1us
I20260709 13:02:15.046641  1571 leader_election.cc:400] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 17 pre-election: Vote denied by peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693) with higher term. Message: Invalid argument: T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 17. Current term is 18.
I20260709 13:02:15.050426  1571 leader_election.cc:403] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 17 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:02:15.032271  8354 raft_consensus.cc:2804] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:02:15.033389  8706 raft_consensus.cc:2804] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:02:15.058179  2298 leader_election.cc:304] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:15.050709  8607 consensus_queue.cc:1048] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.010s
I20260709 13:02:15.060672  8607 consensus_queue.cc:1243] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer 83595ad0fd5940c0a2bfc7c61acd969a log is divergent from this leader: its last log entry 18.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 9
I20260709 13:02:15.060791  8358 leader_election.cc:290] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:15.061820  1494 catalog_manager.cc:5697] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 9 to 19, leader changed from af520b67945d443b901bbeca37cc5701 (127.0.107.195) to 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196). New cstate: current_term: 19 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 13:02:15.067384  8360 leader_election.cc:290] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:15.068042  8354 raft_consensus.cc:493] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:02:15.068511  8354 raft_consensus.cc:3060] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:02:15.068764  8726 leader_election.cc:290] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:15.069356  8706 raft_consensus.cc:493] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:15.069840  8706 raft_consensus.cc:3060] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:02:15.009822  8496 raft_consensus.cc:515] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:15.060077  8651 raft_consensus.cc:697] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 18 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:02:15.075793  2298 leader_election.cc:304] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:15.046954  8732 raft_consensus.cc:2749] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
W20260709 13:02:15.081717  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 167678us
W20260709 13:02:15.093866  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 109814us
I20260709 13:02:15.130160  1600 raft_consensus.cc:1240] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 13. Current term is 14. Ops: []
W20260709 13:02:15.090735  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 142760us
W20260709 13:02:15.090463  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 261085us
W20260709 13:02:15.132465  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 86314.2us
I20260709 13:02:15.132983  7745 raft_consensus.cc:3060] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:02:14.986302  8730 raft_consensus.cc:515] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:15.127632  1646 leader_election.cc:304] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:15.064656  1494 catalog_manager.cc:5697] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 14 to 18, leader changed from 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194) to 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196). New cstate: current_term: 18 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 13:02:15.073230  8634 raft_consensus.cc:2749] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 13:02:15.162335  2299 leader_election.cc:304] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
W20260709 13:02:15.176389  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 258819us
W20260709 13:02:15.174800  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 96989.2us
I20260709 13:02:15.187482  1599 raft_consensus.cc:2468] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 17.
I20260709 13:02:15.091686  8490 raft_consensus.cc:2749] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Leader election lost for term 19. Reason: Vote denied by peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445) with higher term. Message: Invalid argument: T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for earlier term 19. Current term is 20.
I20260709 13:02:15.167541  2301 leader_election.cc:304] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:02:15.172885  1571 leader_election.cc:304] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:02:15.090226  8537 raft_consensus.cc:697] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:02:15.198175  1750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43156 (request call id 1396) took 1265 ms. Trace:
I20260709 13:02:15.198319  1752 raft_consensus.cc:1240] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 20 LEADER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 19. Current term is 20. Ops: []
W20260709 13:02:15.177415  8602 consensus_meta.cc:287] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.673s	user 0.007s	sys 0.006s
I20260709 13:02:15.189280  8373 leader_election.cc:290] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:15.093315  8623 raft_consensus.cc:515] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:15.199863  1750 rpcz_store.cc:276] 0709 13:02:13.932920 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:13.938341 (+  5421us) service_pool.cc:224] Handling call
0709 13:02:15.197940 (+1259599us) spinlock_profiling.cc:243] Waited 225 ms on lock 0x7b54001c6e54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:02:15.198104 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":517045504}
I20260709 13:02:15.218299  8537 consensus_queue.cc:237] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.12, Last appended by leader: 12, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:15.219211  8732 raft_consensus.cc:3060] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Advancing to term 21
W20260709 13:02:15.240818  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 52078.9us
I20260709 13:02:15.222734  8696 raft_consensus.cc:2804] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:02:15.289367  8696 raft_consensus.cc:493] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:15.290048  8696 raft_consensus.cc:3060] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:02:15.199154  7745 raft_consensus.cc:2749] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Leader pre-election lost for term 17. Reason: Vote denied by peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693) with higher term. Message: Invalid argument: T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 17. Current term is 18.
I20260709 13:02:15.212551  8741 raft_consensus.cc:2749] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:02:15.245255  8585 leader_election.cc:290] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 14 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:15.226397  8602 raft_consensus.cc:515] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:15.244340  8373 raft_consensus.cc:493] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:15.300060  1747 raft_consensus.cc:1240] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 17. Current term is 18. Ops: []
I20260709 13:02:15.299777  8373 raft_consensus.cc:515] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:15.201345  8368 raft_consensus.cc:2749] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 13:02:15.198764  8190 raft_consensus.cc:2804] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:02:15.214234  8634 consensus_queue.cc:1048] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.045s
W20260709 13:02:15.259459  8737 log.cc:927] Time spent T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.093s	user 0.001s	sys 0.000s
I20260709 13:02:15.261273  8750 raft_consensus.cc:2804] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 13:02:15.303651  8750 raft_consensus.cc:697] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 15 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:02:15.305027  8750 consensus_queue.cc:237] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.11, Last appended by leader: 11, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:15.266276  8748 raft_consensus.cc:2749] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 13:02:15.242740  8651 consensus_queue.cc:237] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 16.9, Last appended by leader: 9, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:02:15.272089  8736 log.cc:927] Time spent T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.113s	user 0.000s	sys 0.000s
I20260709 13:02:15.334496  8703 raft_consensus.cc:493] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:15.335444  8703 raft_consensus.cc:515] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:15.302078  8602 leader_election.cc:290] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 18 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:15.245911  1494 catalog_manager.cc:5697] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 14 to 17, leader changed from 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) to 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194). New cstate: current_term: 17 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } }
W20260709 13:02:15.360045  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 187380us
I20260709 13:02:15.361047  1494 catalog_manager.cc:5697] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 10 to 13, leader changed from 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) to 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193). New cstate: current_term: 13 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20260709 13:02:15.366490  8700 consensus_meta.cc:287] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.908s	user 0.017s	sys 0.001s
I20260709 13:02:15.367429  1494 catalog_manager.cc:5697] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 14 to 17, leader changed from 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) to 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193). New cstate: current_term: 17 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:02:15.371088  8623 leader_election.cc:290] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 17 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:15.372531  1571 leader_election.cc:304] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
W20260709 13:02:15.363057  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 120869us
I20260709 13:02:15.377516  8602 raft_consensus.cc:493] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:15.386226  8504 raft_consensus.cc:515] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:15.394582  8730 leader_election.cc:290] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:02:15.418998  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 46574us
I20260709 13:02:15.421605  8496 leader_election.cc:290] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 17 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:15.422995  1571 leader_election.cc:304] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:15.427349  1571 leader_election.cc:304] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:02:15.441864  1571 leader_election.cc:304] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:02:15.449014  7745 raft_consensus.cc:2749] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:02:15.450377  1571 leader_election.cc:304] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
W20260709 13:02:15.451042  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 247294us
W20260709 13:02:15.451717  1644 leader_election.cc:336] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 20 election: RPC error from VoteRequest() call to peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625): Timed out: RequestConsensusVote RPC to 127.0.107.196:46625 timed out after 1.558s (SENT)
I20260709 13:02:15.452376  1644 leader_election.cc:304] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:02:15.429445  8585 raft_consensus.cc:2804] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:02:15.456724  8504 leader_election.cc:290] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 16 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:15.457206  8585 raft_consensus.cc:493] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:15.473506  8585 raft_consensus.cc:3060] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:02:15.436239  8752 raft_consensus.cc:493] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:15.465458  8468 consensus_meta.cc:287] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.635s	user 0.009s	sys 0.006s
I20260709 13:02:15.467831  8504 raft_consensus.cc:2749] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 13:02:15.422703  8602 raft_consensus.cc:515] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:15.512105  7745 raft_consensus.cc:3060] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Advancing to term 19
W20260709 13:02:15.512105  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 61866.6us
I20260709 13:02:15.471868  8703 leader_election.cc:290] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:15.457670  8551 raft_consensus.cc:2749] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:02:15.454802  8373 leader_election.cc:290] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:15.437261  8700 raft_consensus.cc:2749] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:02:15.525420  8602 leader_election.cc:290] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:15.473515  8623 consensus_queue.cc:1059] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 20.15, Next index: 16, Last known committed idx: 14, Time since last communication: 0.052s
I20260709 13:02:15.536628  7745 raft_consensus.cc:2749] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:02:15.483937  8752 raft_consensus.cc:515] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:02:15.521330  8758 log.cc:927] Time spent T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.087s	user 0.000s	sys 0.002s
I20260709 13:02:15.513672  1571 leader_election.cc:304] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:02:15.538484  8496 raft_consensus.cc:3055] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 13:02:15.514711  8468 raft_consensus.cc:515] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:15.576222  8468 leader_election.cc:290] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 17 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:15.543011  8753 consensus_queue.cc:1059] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 13.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.034s
I20260709 13:02:15.578084  8753 raft_consensus.cc:3055] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 13:02:15.549638  8373 raft_consensus.cc:493] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:02:15.550468  8551 raft_consensus.cc:493] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:15.551162  1569 leader_election.cc:304] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:02:15.551964  8700 raft_consensus.cc:493] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:15.543435  2298 leader_election.cc:304] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:02:15.586892  1752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55848 (request call id 3192) took 1197 ms. Trace:
W20260709 13:02:15.587319  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 44018.6us
I20260709 13:02:15.587427  1752 rpcz_store.cc:276] 0709 13:02:14.389925 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:14.591394 (+201469us) service_pool.cc:224] Handling call
0709 13:02:15.198100 (+606706us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:15.200634 (+  2534us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:15.586639 (+386005us) spinlock_profiling.cc:243] Waited 607 ms on lock 0x7b54001c6e50. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661618ad 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:02:15.586799 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1395185408}
I20260709 13:02:15.587653  2331 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39726 (request call id 1551) took 1228 ms. Trace:
I20260709 13:02:15.592396  2331 rpcz_store.cc:276] 0709 13:02:14.359181 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:14.581727 (+222546us) service_pool.cc:224] Handling call
0709 13:02:14.585460 (+  3733us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:14.613931 (+ 28471us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:15.581611 (+967680us) spinlock_profiling.cc:243] Waited 3.35 ms on lock 0x7b540072e550. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661618ad 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:02:15.587569 (+  5958us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":7697664}
W20260709 13:02:15.564175  1746 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55848 (request call id 3166) took 2528 ms (client timeout 1748 ms). Trace:
W20260709 13:02:15.602313  1746 rpcz_store.cc:269] 0709 13:02:13.035551 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:13.783860 (+748309us) service_pool.cc:224] Handling call
0709 13:02:15.563889 (+1780029us) spinlock_profiling.cc:243] Waited 566 ms on lock 0x7b54002f49d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:02:15.564059 (+   170us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1302445440}
I20260709 13:02:15.582991  1596 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 2936) took 1537 ms. Trace:
I20260709 13:02:15.606290  1596 rpcz_store.cc:276] 0709 13:02:14.045389 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:14.074933 (+ 29544us) service_pool.cc:224] Handling call
0709 13:02:14.086302 (+ 11369us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:14.614485 (+528183us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:15.582708 (+968223us) spinlock_profiling.cc:243] Waited 494 ms on lock 0x7b54003edad4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:02:15.582914 (+   206us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1136786304}
W20260709 13:02:15.571985  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 406349us
W20260709 13:02:15.583058  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 421004us
I20260709 13:02:15.583619  7745 raft_consensus.cc:2804] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Leader election won for term 20
I20260709 13:02:15.627291  7745 raft_consensus.cc:697] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
W20260709 13:02:15.584326  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 70813.7us
I20260709 13:02:15.631134  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
W20260709 13:02:15.671200  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 43854.5us
I20260709 13:02:15.673241  7745 consensus_queue.cc:237] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.10, Last appended by leader: 10, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:15.683090  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 17 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:15.684041  1599 raft_consensus.cc:3060] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:02:15.684929  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:15.685770  1600 raft_consensus.cc:3060] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Advancing to term 17
I20260709 13:02:15.689015  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:15.691757  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:15.692734  1602 raft_consensus.cc:2468] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 23.
W20260709 13:02:15.671133  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 43942.8us
I20260709 13:02:15.706008  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:15.578557  8753 raft_consensus.cc:740] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:02:15.710296  2336 raft_consensus.cc:1240] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 17 LEADER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 16. Current term is 17. Ops: [16.13-16.13]
I20260709 13:02:15.578940  8496 raft_consensus.cc:740] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:02:15.586673  8714 raft_consensus.cc:3060] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:02:15.714807  1747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55840 (request call id 3053) took 1328 ms. Trace:
I20260709 13:02:15.715232  1747 rpcz_store.cc:276] 0709 13:02:14.386643 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:14.405165 (+ 18522us) service_pool.cc:224] Handling call
0709 13:02:14.447772 (+ 42607us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:15.300536 (+852764us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:15.714536 (+414000us) spinlock_profiling.cc:243] Waited 852 ms on lock 0x7b5400323cd4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:02:15.714723 (+   187us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1960404352}
I20260709 13:02:15.671255  8698 raft_consensus.cc:493] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:02:15.716660  8698 raft_consensus.cc:515] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:15.723424  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 18 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
W20260709 13:02:15.711647  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 106559us
W20260709 13:02:15.711395  1643 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.578s	user 0.063s	sys 0.039s
I20260709 13:02:15.724290  2330 raft_consensus.cc:3060] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Advancing to term 18
W20260709 13:02:15.724514  1643 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.591s	user 0.063s	sys 0.039s
W20260709 13:02:15.725039  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 591855us
I20260709 13:02:15.672510  8750 raft_consensus.cc:2804] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 13:02:15.721575  2332 raft_consensus.cc:1275] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 16 index: 13. Preceding OpId from leader: term: 20 index: 15. (index mismatch)
W20260709 13:02:15.713826  1644 leader_election.cc:336] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 21 election: RPC error from VoteRequest() call to peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625): Timed out: RequestConsensusVote RPC to 127.0.107.196:46625 timed out after 2.536s (SENT)
I20260709 13:02:15.726110  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:15.737581  1602 raft_consensus.cc:2468] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 15.
I20260709 13:02:15.741220  1599 raft_consensus.cc:2468] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 17.
I20260709 13:02:15.742668  2298 leader_election.cc:304] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
W20260709 13:02:15.726413  1569 leader_election.cc:336] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 17 pre-election: RPC error from VoteRequest() call to peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625): Timed out: RequestConsensusVote RPC to 127.0.107.196:46625 timed out after 1.720s (SENT)
I20260709 13:02:15.750906  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:02:15.718716  8354 consensus_meta.cc:287] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.650s	user 0.018s	sys 0.003s
I20260709 13:02:15.757716  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
W20260709 13:02:15.764638  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 50892.6us
W20260709 13:02:15.765471  1644 leader_election.cc:336] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 17 pre-election: RPC error from VoteRequest() call to peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625): Timed out: RequestConsensusVote RPC to 127.0.107.196:46625 timed out after 1.834s (SENT)
I20260709 13:02:15.776139  1599 raft_consensus.cc:1275] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 18 index: 12. Preceding OpId from leader: term: 19 index: 13. (index mismatch)
I20260709 13:02:15.670073  8373 raft_consensus.cc:515] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:15.669135  8585 raft_consensus.cc:515] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:15.811712  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 18 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:15.674794  8551 raft_consensus.cc:515] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:15.827023  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:15.673547  8700 raft_consensus.cc:515] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:15.797605  1569 leader_election.cc:304] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:02:15.811512  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:15.851089  2331 raft_consensus.cc:2393] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 20: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:02:15.853027  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:15.789918  8354 raft_consensus.cc:515] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:15.859911  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 45878.9us
I20260709 13:02:15.827724  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 17 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:02:15.809574  8696 consensus_meta.cc:287] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.519s	user 0.018s	sys 0.000s
I20260709 13:02:15.861420  1601 raft_consensus.cc:2410] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for term 17 because replica has last-logged OpId of term: 15 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 10.
I20260709 13:02:15.827947  2334 raft_consensus.cc:2376] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 18. Re-sending same reply.
W20260709 13:02:15.766212  1643 connection.cc:256] RPC call timeout handler was delayed by -1.3014094989999876s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2239s for a response
I20260709 13:02:15.729734  8697 consensus_queue.cc:1059] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 15.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.122s
I20260709 13:02:15.860011  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:15.707566  8711 consensus_queue.cc:1048] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.058s
I20260709 13:02:15.886853  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:15.864989  1599 raft_consensus.cc:2393] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 21: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:02:15.865618  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:15.888278  1602 raft_consensus.cc:2468] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 19.
I20260709 13:02:15.790120  8360 raft_consensus.cc:493] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:15.868942  2336 raft_consensus.cc:2368] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for earlier term 19. Current term is 20.
I20260709 13:02:15.880381  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:15.877918  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:15.881819  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:15.890950  2333 raft_consensus.cc:2393] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 15: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:02:15.758545  8468 consensus_queue.cc:1048] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.102s
I20260709 13:02:15.885329  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:15.886314  8373 leader_election.cc:290] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:15.887712  1673 raft_consensus.cc:2393] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 20: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:02:15.882141  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:15.898810  2335 raft_consensus.cc:2376] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 20. Re-sending same reply.
I20260709 13:02:15.892094  1679 raft_consensus.cc:1240] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 16. Current term is 17. Ops: []
I20260709 13:02:15.893065  1672 raft_consensus.cc:1275] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
W20260709 13:02:15.869108  8706 consensus_meta.cc:287] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.778s	user 0.014s	sys 0.001s
I20260709 13:02:15.914999  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:02:15.916291  1643 leader_election.cc:336] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 20 election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 1.673s (SENT)
I20260709 13:02:15.898283  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:15.898679  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:15.866716  8754 raft_consensus.cc:493] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:15.928098  1674 raft_consensus.cc:2393] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 18: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:02:15.920560  1671 raft_consensus.cc:2468] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 15.
I20260709 13:02:15.915877  1675 raft_consensus.cc:2468] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 18.
I20260709 13:02:15.933363  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:15.929416  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:15.952131  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:15.892154  8730 raft_consensus.cc:2749] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:02:15.901947  8360 raft_consensus.cc:515] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } }
W20260709 13:02:15.947317  8723 log.cc:927] Time spent T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 1.045s	user 0.014s	sys 0.041s
I20260709 13:02:15.958031  8700 leader_election.cc:290] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:15.960873  8755 consensus_queue.cc:1059] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 13:02:15.957412  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:15.888685  8753 consensus_queue.cc:260] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 13.12, Last appended by leader: 12, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:15.995930  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:15.997470  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:15.870698  8506 consensus_queue.cc:1048] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.083s
I20260709 13:02:15.869235  8696 raft_consensus.cc:515] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:02:15.974790  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 58603.1us
I20260709 13:02:15.722905  1494 catalog_manager.cc:5697] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 13 to 15. New cstate: current_term: 15 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } }
I20260709 13:02:15.934571  8698 leader_election.cc:290] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:15.931085  8743 raft_consensus.cc:2804] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 13:02:15.982546  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:15.925679  8741 consensus_queue.cc:1048] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.075s
I20260709 13:02:15.987299  8752 leader_election.cc:290] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:02:15.987355  8757 log.cc:927] Time spent T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.415s	user 0.000s	sys 0.001s
W20260709 13:02:15.988196  8759 log.cc:927] Time spent T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.268s	user 0.000s	sys 0.001s
I20260709 13:02:15.988519  8733 raft_consensus.cc:493] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:15.995990  8753 raft_consensus.cc:3060] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:02:15.922379  8551 leader_election.cc:290] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:15.999277  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:15.998422  2331 raft_consensus.cc:2468] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 14.
W20260709 13:02:16.006304  1643 leader_election.cc:336] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 18 election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 1.704s (SENT)
W20260709 13:02:16.014762  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 297056us
I20260709 13:02:16.029500  2301 leader_election.cc:304] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:02:15.805408  1600 raft_consensus.cc:2468] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 17.
W20260709 13:02:15.973717  8626 consensus_meta.cc:287] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 1.039s	user 0.009s	sys 0.012s
W20260709 13:02:16.054946  2299 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:15.998222  8760 log.cc:927] Time spent T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.065s	user 0.000s	sys 0.000s
I20260709 13:02:15.951581  8754 raft_consensus.cc:515] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:16.057379  1602 raft_consensus.cc:2468] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 19.
I20260709 13:02:15.932158  8706 raft_consensus.cc:515] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:15.974090  8354 leader_election.cc:290] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 15 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 13:02:16.044896  8732 consensus_meta.cc:287] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.790s	user 0.017s	sys 0.009s
I20260709 13:02:16.068984  1603 raft_consensus.cc:2468] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 18.
I20260709 13:02:16.091753  1643 leader_election.cc:304] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
W20260709 13:02:16.096434  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 354157us
W20260709 13:02:16.100893  8507 consensus_meta.cc:287] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 1.028s	user 0.013s	sys 0.007s
I20260709 13:02:16.101591  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 18 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:02:16.106166  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 379965us
W20260709 13:02:16.107067  1569 leader_election.cc:336] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 17 pre-election: RPC error from VoteRequest() call to peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625): Timed out: RequestConsensusVote RPC to 127.0.107.196:46625 timed out after 1.791s (SENT)
I20260709 13:02:16.078061  8743 raft_consensus.cc:697] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 17 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:02:16.107487  1569 leader_election.cc:304] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:02:16.106258  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
W20260709 13:02:16.108119  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 78814.8us
I20260709 13:02:16.109795  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:16.115411  1603 raft_consensus.cc:2468] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 19.
I20260709 13:02:16.115694  8768 raft_consensus.cc:493] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:16.116338  8768 raft_consensus.cc:515] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:16.116595  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:16.117412  1603 raft_consensus.cc:2468] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 15.
I20260709 13:02:16.118607  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:16.119374  1603 raft_consensus.cc:2468] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 16.
I20260709 13:02:16.111361  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:16.120589  1600 raft_consensus.cc:2376] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 18. Re-sending same reply.
I20260709 13:02:16.126914  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:16.112982  1718 leader_election.cc:304] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:02:16.130198  1748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55848 (request call id 3191) took 1740 ms. Trace:
I20260709 13:02:16.134289  2330 raft_consensus.cc:2468] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 18.
I20260709 13:02:16.135604  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:16.151751  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:16.152531  1601 raft_consensus.cc:2393] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 14: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:02:16.153380  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 17 candidate_status { last_received { term: 16 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:16.153882  1601 raft_consensus.cc:3060] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:02:16.137003  7745 raft_consensus.cc:2749] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:02:16.139478  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:16.158891  1603 raft_consensus.cc:2468] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 23.
I20260709 13:02:16.097898  8551 consensus_queue.cc:1059] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.043s
I20260709 13:02:16.082049  8733 raft_consensus.cc:515] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:16.141397  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 18 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:16.169028  1600 raft_consensus.cc:2410] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for term 18 because replica has last-logged OpId of term: 17 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 9.
I20260709 13:02:16.108219  8732 raft_consensus.cc:2749] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:02:16.130770  1748 rpcz_store.cc:276] 0709 13:02:14.389692 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:14.566649 (+176957us) service_pool.cc:224] Handling call
0709 13:02:14.567037 (+   388us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:14.567472 (+   435us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:14.567505 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:14.568140 (+   635us) log.cc:844] Serialized 26 byte log entry
0709 13:02:14.645427 (+ 77287us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:02:14.645491 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:14.645588 (+    97us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:15.670077 (+1024489us) spinlock_profiling.cc:243] Waited 6.77 ms on lock 0x7b4400022b4c. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f4262f9bc85 00007f4262f9c6a2 00007f4265d0e1db 00007f4265d09aaf 00007f4265cfd86e 00007f4265cfcbb6 00007f4265cfc7d8 00007f4265d06221 00007f4265d061d9 00007f4265d06169
0709 13:02:16.112576 (+442499us) raft_consensus.cc:1685] finished
0709 13:02:16.112601 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:16.130079 (+ 17478us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":14133,"raft.run_cpu_time_us":99,"raft.run_wall_time_us":99,"spinlock_wait_cycles":15562496,"thread_start_us":76802,"threads_started":1,"wal-append.queue_time_us":224876}
W20260709 13:02:16.108711  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 238028us
W20260709 13:02:16.140236  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 261466us
I20260709 13:02:16.159231  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 17 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:16.163066  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:16.177068  2331 raft_consensus.cc:2468] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 17.
I20260709 13:02:16.169373  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
W20260709 13:02:16.183364  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 177194us
I20260709 13:02:16.146277  8354 raft_consensus.cc:2749] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20260709 13:02:16.184320  1643 leader_election.cc:336] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 17 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 1.834s (SENT)
I20260709 13:02:16.192165  1643 leader_election.cc:304] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:16.184679  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:16.149696  8585 leader_election.cc:290] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 21 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:16.147825  8750 raft_consensus.cc:697] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 17 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:02:16.188050  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:16.209858  1600 raft_consensus.cc:2468] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 16.
I20260709 13:02:16.147344  8703 raft_consensus.cc:2749] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 13:02:16.109982  8507 raft_consensus.cc:515] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:16.177477  8756 raft_consensus.cc:493] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:16.198273  2330 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "87b86d46388a41cd90374c203cb27914"
dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
mode: GRACEFUL
new_leader_uuid: "af520b67945d443b901bbeca37cc5701"
 from {username='slave'} at 127.0.0.1:35790
I20260709 13:02:16.186636  1604 raft_consensus.cc:2468] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 19.
I20260709 13:02:16.206647  1676 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52778 (request call id 2713) took 2200 ms. Trace:
I20260709 13:02:16.211400  1676 rpcz_store.cc:276] 0709 13:02:14.006738 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:14.006871 (+   133us) service_pool.cc:224] Handling call
0709 13:02:14.391918 (+385047us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:14.440668 (+ 48750us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 13:02:14.440713 (+    45us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:14.440758 (+    45us) raft_consensus.cc:1645] Marking committed up to 14
0709 13:02:14.440800 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:14.440872 (+    72us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:16.205956 (+1765084us) spinlock_profiling.cc:243] Waited 385 ms on lock 0x7b540025b050. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661618ad 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:02:16.206571 (+   615us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":42013,"raft.queue_time_us":42165,"raft.run_cpu_time_us":205,"raft.run_wall_time_us":17728,"spinlock_wait_cycles":885124352}
I20260709 13:02:16.175387  8360 leader_election.cc:290] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:16.211419  2330 raft_consensus.cc:606] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Received request to transfer leadership to TS af520b67945d443b901bbeca37cc5701
I20260709 13:02:16.214272  2330 raft_consensus.cc:614] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 13:02:16.148970  8768 leader_election.cc:290] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:02:16.185163  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 78160.7us
I20260709 13:02:16.235210  2331 raft_consensus.cc:1275] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 16 index: 12. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20260709 13:02:16.239185  1598 raft_consensus.cc:2468] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 21.
I20260709 13:02:16.241335  8733 leader_election.cc:290] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:16.243532  8696 leader_election.cc:290] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 17 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:16.245237  8754 leader_election.cc:290] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:16.245767  2332 raft_consensus.cc:1240] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 17. Current term is 18. Ops: []
I20260709 13:02:16.248195  8694 raft_consensus.cc:3055] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 13:02:16.209793  8607 consensus_queue.cc:1048] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.049s
I20260709 13:02:16.133328  1494 catalog_manager.cc:5697] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 11 to 13. New cstate: current_term: 13 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } }
W20260709 13:02:16.256210  8718 log.cc:927] Time spent T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 1.519s	user 0.020s	sys 0.009s
I20260709 13:02:16.258222  8648 raft_consensus.cc:2749] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:02:16.259548  2330 raft_consensus.cc:1240] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 14. Current term is 15. Ops: []
I20260709 13:02:16.260164  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:16.330120  1600 raft_consensus.cc:3060] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:02:16.261205  8706 leader_election.cc:290] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 17 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:16.261943  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:16.262933  8537 raft_consensus.cc:493] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:02:16.263571  1603 raft_consensus.cc:2468] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 17.
I20260709 13:02:16.263656  1597 raft_consensus.cc:1240] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 16. Current term is 17. Ops: []
I20260709 13:02:16.334075  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:16.334791  1603 raft_consensus.cc:2468] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 18.
I20260709 13:02:16.143556  8698 consensus_queue.cc:1048] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.042s
I20260709 13:02:16.264453  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:16.265455  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
W20260709 13:02:16.267554  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 121274us
I20260709 13:02:16.249722  8743 consensus_queue.cc:237] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 15.14, Last appended by leader: 14, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:16.336591  1605 raft_consensus.cc:2468] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 19.
I20260709 13:02:16.340456  1748 raft_consensus.cc:1240] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 16. Current term is 17. Ops: []
I20260709 13:02:16.341048  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:16.308779  8776 raft_consensus.cc:2749] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:02:16.307251  2298 leader_election.cc:304] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:02:16.345804  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:16.346462  1746 raft_consensus.cc:3060] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Advancing to term 17
W20260709 13:02:16.310467  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 129613us
I20260709 13:02:16.274221  8507 leader_election.cc:290] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 19 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:16.276336  8506 raft_consensus.cc:493] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:16.277570  8773 raft_consensus.cc:493] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:16.338734  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:16.305359  1717 leader_election.cc:304] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 13:02:16.307289  1569 leader_election.cc:336] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 17 election: RPC error from VoteRequest() call to peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625): Timed out: RequestConsensusVote RPC to 127.0.107.196:46625 timed out after 1.812s (SENT)
I20260709 13:02:16.342787  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:16.357339  1749 raft_consensus.cc:2468] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 17.
I20260709 13:02:16.343493  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 18 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:16.320894  8694 raft_consensus.cc:740] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
W20260709 13:02:16.363636  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 58460.7us
I20260709 13:02:16.364516  1717 leader_election.cc:304] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:16.335937  8344 leader_election.cc:290] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:16.365899  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:16.367208  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 18 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:16.270273  8756 raft_consensus.cc:515] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:16.371621  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 187428us
I20260709 13:02:16.370878  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:16.352398  1644 leader_election.cc:304] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:16.376672  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 17 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:16.377271  1748 raft_consensus.cc:3060] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:02:16.375156  1643 leader_election.cc:304] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 13:02:16.354584  1600 raft_consensus.cc:2468] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 16.
I20260709 13:02:16.321018  8775 consensus_queue.cc:1048] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.041s
I20260709 13:02:16.374574  1605 raft_consensus.cc:2468] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 18.
I20260709 13:02:16.376075  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 21 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:16.389503  1746 raft_consensus.cc:2468] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 17.
I20260709 13:02:16.391404  1718 leader_election.cc:304] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 13:02:16.393683  1600 raft_consensus.cc:1275] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 16 index: 11. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
I20260709 13:02:16.396485  8733 raft_consensus.cc:2804] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 13:02:16.405587  8733 raft_consensus.cc:493] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:16.405787  8190 raft_consensus.cc:493] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Starting leader election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:02:16.406119  8733 raft_consensus.cc:3060] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:02:16.396301  8651 consensus_queue.cc:1048] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.015s
I20260709 13:02:16.412652  8784 consensus_queue.cc:1048] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 13:02:16.414537  8784 raft_consensus.cc:2749] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 13:02:16.396643  8648 raft_consensus.cc:2749] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 13:02:16.415280  8784 raft_consensus.cc:2804] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:02:16.415679  8784 raft_consensus.cc:493] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:16.416076  8784 raft_consensus.cc:3060] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:02:16.421759  8694 consensus_queue.cc:260] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.13, Last appended by leader: 13, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:16.406378  8190 raft_consensus.cc:3060] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:02:16.423133  8694 raft_consensus.cc:3060] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:02:16.374147  8769 raft_consensus.cc:2804] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 13:02:16.425485  8769 raft_consensus.cc:697] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 17 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:02:16.427933  1605 raft_consensus.cc:1240] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 16. Current term is 17. Ops: []
I20260709 13:02:16.430842  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 17 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:16.431550  1679 raft_consensus.cc:3060] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:02:16.343731  8537 raft_consensus.cc:515] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:16.433820  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 126701us
W20260709 13:02:16.434624  1569 leader_election.cc:336] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 election: RPC error from VoteRequest() call to peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625): Timed out: RequestConsensusVote RPC to 127.0.107.196:46625 timed out after 1.838s (SENT)
I20260709 13:02:16.434870  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 18 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:16.435060  1569 leader_election.cc:304] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:02:16.376621  8779 raft_consensus.cc:2749] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20260709 13:02:16.430868  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 78589us
I20260709 13:02:16.336987  8750 consensus_queue.cc:237] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 15.14, Last appended by leader: 14, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:16.439616  1677 raft_consensus.cc:1275] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 15 index: 12. (index mismatch)
I20260709 13:02:16.440114  1673 raft_consensus.cc:1240] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 19. Current term is 20. Ops: []
I20260709 13:02:16.441210  1600 raft_consensus.cc:1240] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 15. Current term is 16. Ops: []
I20260709 13:02:16.367439  8506 raft_consensus.cc:515] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:16.447783  1745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55848 (request call id 3190) took 2058 ms. Trace:
I20260709 13:02:16.368273  8773 raft_consensus.cc:515] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:16.448340  1605 pending_rounds.cc:85] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a: Aborting all ops after (but not including) 9
I20260709 13:02:16.450862  1601 raft_consensus.cc:2468] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 17.
I20260709 13:02:16.442154  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:16.454888  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 22 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:16.455701  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 21 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:16.398316  8344 raft_consensus.cc:493] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:16.579698  2299 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:16.589347  1674 raft_consensus.cc:2410] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 for term 21 because replica has last-logged OpId of term: 18 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 13.
W20260709 13:02:16.590986  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 156469us
I20260709 13:02:16.584221  1678 raft_consensus.cc:3060] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:02:16.584404  1603 raft_consensus.cc:1240] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 16. Current term is 17. Ops: []
I20260709 13:02:16.541931  1745 rpcz_store.cc:276] 0709 13:02:14.389329 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:14.393938 (+  4609us) service_pool.cc:224] Handling call
0709 13:02:14.436403 (+ 42465us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:14.441248 (+  4845us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:14.441279 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:14.459323 (+ 18044us) log.cc:844] Serialized 26 byte log entry
0709 13:02:14.505798 (+ 46475us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:02:14.505843 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:14.507209 (+  1366us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:16.181566 (+1674357us) spinlock_profiling.cc:243] Waited 49.2 ms on lock 0x7b4400022b4c. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f4262f9bc85 00007f4262f9c6a2 00007f4265d0e1db 00007f4265d09aaf 00007f4265cfd86e 00007f4265cfcbb6 00007f4265cfc7d8 00007f4265d06221 00007f4265d061d9 00007f4265d06169
0709 13:02:16.440293 (+258727us) raft_consensus.cc:1685] finished
0709 13:02:16.440320 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:16.447701 (+  7381us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":113290368,"thread_start_us":11520,"threads_started":1,"wal-append.queue_time_us":147571}
W20260709 13:02:16.602620  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 153537us
I20260709 13:02:16.613102  1601 raft_consensus.cc:1275] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 22 index: 14. (index mismatch)
I20260709 13:02:16.614693  1601 raft_consensus.cc:1275] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 15 index: 12. (index mismatch)
I20260709 13:02:16.615808  1601 raft_consensus.cc:1240] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 20. Current term is 21. Ops: []
I20260709 13:02:16.624827  1644 leader_election.cc:400] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 election: Vote denied by peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625) with higher term. Message: Invalid argument: T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for earlier term 19. Current term is 20.
I20260709 13:02:16.626251  1644 leader_election.cc:304] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
W20260709 13:02:16.634636  2329 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39726 (request call id 1541) took 2231 ms (client timeout 1592 ms). Trace:
W20260709 13:02:16.635311  2329 rpcz_store.cc:269] 0709 13:02:14.355878 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:14.360579 (+  4701us) service_pool.cc:224] Handling call
0709 13:02:16.587132 (+2226553us) spinlock_profiling.cc:243] Waited 399 ms on lock 0x7b54005e01d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:02:16.587292 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":917108096}
I20260709 13:02:16.581302  1605 pending_rounds.cc:107] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a: Aborting uncommitted NO_OP operation due to leader change: 18.10
I20260709 13:02:16.656069  1605 raft_consensus.cc:2889] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:02:16.447443  8706 raft_consensus.cc:2749] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 13:02:16.635188  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:16.659530  1673 raft_consensus.cc:3060] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:02:16.662425  1569 leader_election.cc:304] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:02:16.666812  1672 raft_consensus.cc:3060] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:02:16.447274  8468 raft_consensus.cc:493] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:02:16.435645  8752 consensus_queue.cc:1048] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.176s
I20260709 13:02:16.453056  8607 raft_consensus.cc:493] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:02:16.683071  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:16.683862  1746 raft_consensus.cc:2468] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 19.
I20260709 13:02:16.683921  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:16.684316  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:16.684594  1751 raft_consensus.cc:2468] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 18.
I20260709 13:02:16.684970  1749 raft_consensus.cc:3060] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:02:16.686811  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 17 candidate_status { last_received { term: 16 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:16.680022  1569 leader_election.cc:304] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:02:16.695003  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:16.697397  1749 raft_consensus.cc:2468] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 17.
I20260709 13:02:16.697180  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:16.636487  8780 raft_consensus.cc:493] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:16.701871  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 308962us
I20260709 13:02:16.715191  1678 raft_consensus.cc:2468] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 15.
I20260709 13:02:16.636997  8711 raft_consensus.cc:493] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:16.609972  8373 raft_consensus.cc:493] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:16.702214  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:16.713325  1676 raft_consensus.cc:2393] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 17: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:02:16.703269  1674 raft_consensus.cc:2468] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 14.
I20260709 13:02:16.716449  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:16.723466  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:16.725909  1678 raft_consensus.cc:2468] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 18.
I20260709 13:02:16.726912  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 19 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:16.726989  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:16.729521  1749 raft_consensus.cc:3060] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:02:16.729501  1678 raft_consensus.cc:2468] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 18.
I20260709 13:02:16.731130  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:16.728143  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:16.728545  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:16.440318  8769 consensus_queue.cc:237] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.11, Last appended by leader: 11, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:16.691603  8551 raft_consensus.cc:2749] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 13:02:16.739980  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:16.690166  8732 raft_consensus.cc:493] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:16.732707  1751 raft_consensus.cc:2468] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 18.
I20260709 13:02:16.733265  1718 leader_election.cc:304] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:16.729498  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:16.636652  8344 raft_consensus.cc:515] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:16.731851  1678 raft_consensus.cc:2393] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 16: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
W20260709 13:02:16.732018  8755 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.741s	user 0.009s	sys 0.068s
I20260709 13:02:16.756497  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:16.718029  8768 raft_consensus.cc:493] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:16.758551  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 132457us
I20260709 13:02:16.759003  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:16.764565  1749 raft_consensus.cc:2468] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 20.
I20260709 13:02:16.765623  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:16.768504  1644 leader_election.cc:304] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:16.766564  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:16.769804  1746 raft_consensus.cc:1240] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 18. Current term is 19. Ops: []
I20260709 13:02:16.770354  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:16.772778  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 19 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:16.773718  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:16.775452  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:16.718272  8779 raft_consensus.cc:493] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:16.773239  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:16.777956  1746 raft_consensus.cc:2468] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 18.
I20260709 13:02:16.772058  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 20 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:16.742043  8373 raft_consensus.cc:515] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:16.782734  8373 leader_election.cc:290] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:16.782879  1671 raft_consensus.cc:1240] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 LEADER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 18. Current term is 19. Ops: []
I20260709 13:02:16.694134  8468 raft_consensus.cc:515] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:16.776152  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 17 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:16.785043  1718 leader_election.cc:304] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:16.790534  1718 leader_election.cc:304] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:02:16.697206  8607 raft_consensus.cc:515] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:02:16.766564  8755 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.776s	user 0.009s	sys 0.068s
I20260709 13:02:16.801756  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 18 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:16.801433  1644 leader_election.cc:304] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:02:16.808512  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:16.809379  1750 raft_consensus.cc:2393] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 17: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:02:16.776459  8786 raft_consensus.cc:2804] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:02:16.442128  1494 catalog_manager.cc:5697] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 10 to 13, leader changed from 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194) to af520b67945d443b901bbeca37cc5701 (127.0.107.195). New cstate: current_term: 13 leader_uuid: "af520b67945d443b901bbeca37cc5701" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } }
I20260709 13:02:16.817440  1671 raft_consensus.cc:1240] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 14. Current term is 15. Ops: []
I20260709 13:02:16.818284  1676 raft_consensus.cc:2410] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for term 20 because replica has last-logged OpId of term: 19 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 8.
I20260709 13:02:16.803591  1673 raft_consensus.cc:2468] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 19.
I20260709 13:02:16.800393  8698 raft_consensus.cc:2804] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:02:16.731876  8711 raft_consensus.cc:515] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:16.817052  8789 log.cc:927] Time spent T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.171s	user 0.000s	sys 0.001s
I20260709 13:02:16.835922  1644 leader_election.cc:304] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:16.814329  8650 raft_consensus.cc:3060] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:02:16.839924  8797 raft_consensus.cc:2749] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 13:02:16.766594  8732 raft_consensus.cc:515] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:16.840718  8797 raft_consensus.cc:2749] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:02:16.814805  8786 raft_consensus.cc:493] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:16.841602  8786 raft_consensus.cc:3060] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:02:16.807525  8755 consensus_queue.cc:1243] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer af520b67945d443b901bbeca37cc5701 log is divergent from this leader: its last log entry 20.15 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14
I20260709 13:02:16.797257  8771 raft_consensus.cc:2804] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:02:16.776706  8781 consensus_queue.cc:1048] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.049s
I20260709 13:02:16.836576  8506 leader_election.cc:290] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:16.730497  8780 raft_consensus.cc:515] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:16.847986  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:16.858246  1748 raft_consensus.cc:2468] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 17.
I20260709 13:02:16.862532  1671 raft_consensus.cc:2410] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for term 20 because replica has last-logged OpId of term: 19 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 12.
I20260709 13:02:16.798568  8648 consensus_queue.cc:1059] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 16.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.036s
I20260709 13:02:16.834293  8698 raft_consensus.cc:493] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:16.854677  1679 raft_consensus.cc:2468] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 17.
I20260709 13:02:16.851053  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 20 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:16.799719  8779 raft_consensus.cc:515] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:16.792379  8768 raft_consensus.cc:515] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:16.871280  8782 log.cc:927] Time spent T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.408s	user 0.001s	sys 0.001s
I20260709 13:02:16.872221  8698 raft_consensus.cc:3060] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:02:16.872325  8772 consensus_queue.cc:1059] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.022s
I20260709 13:02:16.873381  8788 raft_consensus.cc:2749] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:02:16.874842  8344 leader_election.cc:290] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:16.881553  1672 raft_consensus.cc:2468] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 18.
W20260709 13:02:16.870946  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 48062.1us
I20260709 13:02:16.842576  8732 leader_election.cc:290] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:02:16.906466  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 66662.4us
I20260709 13:02:16.865913  8506 raft_consensus.cc:493] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
W20260709 13:02:16.902889  8714 consensus_meta.cc:287] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 1.156s	user 0.012s	sys 0.004s
W20260709 13:02:16.915513  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 125177us
W20260709 13:02:16.909651  2301 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:16.920029  1569 leader_election.cc:304] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:02:16.912967  8648 raft_consensus.cc:493] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:16.922287  1718 leader_election.cc:304] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:16.930104  1569 leader_election.cc:304] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
W20260709 13:02:16.922428  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 193866us
I20260709 13:02:16.937059  1569 leader_election.cc:304] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:02:16.920557  8714 raft_consensus.cc:2749] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:02:16.941082  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 22 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:16.943845  1600 raft_consensus.cc:2410] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 for term 22 because replica has last-logged OpId of term: 19 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 14.
I20260709 13:02:16.929011  8773 leader_election.cc:290] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:16.934315  8732 raft_consensus.cc:2804] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:02:16.947192  2299 leader_election.cc:304] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:02:16.937268  8551 raft_consensus.cc:493] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:16.960457  1602 raft_consensus.cc:2468] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 15.
I20260709 13:02:16.959106  2299 leader_election.cc:304] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:02:16.959391  1646 leader_election.cc:304] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:16.964761  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:16.965587  1679 raft_consensus.cc:2468] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 20.
I20260709 13:02:16.970829  8767 raft_consensus.cc:2804] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 13:02:16.971555  8767 raft_consensus.cc:697] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 15 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
W20260709 13:02:16.972267  8800 log.cc:927] Time spent T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.052s	user 0.000s	sys 0.000s
I20260709 13:02:16.972675  8767 consensus_queue.cc:237] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:16.973594  8727 consensus_queue.cc:1059] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 19.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.127s
I20260709 13:02:16.957554  8790 raft_consensus.cc:493] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:16.875116  1494 catalog_manager.cc:5697] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 12 to 15, leader changed from 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) to 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196). New cstate: current_term: 15 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 13:02:16.941622  8781 raft_consensus.cc:2804] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 13:02:16.970990  8778 raft_consensus.cc:493] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:16.921217  8506 raft_consensus.cc:515] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:16.971366  8783 raft_consensus.cc:2804] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Leader election won for term 20
I20260709 13:02:16.985050  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:16.985783  2299 leader_election.cc:304] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
W20260709 13:02:16.988766  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 66615.1us
I20260709 13:02:16.990351  8585 raft_consensus.cc:493] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:16.990875  1718 leader_election.cc:304] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:16.991106  8585 raft_consensus.cc:515] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:16.978675  1646 leader_election.cc:304] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:16.993129  8788 raft_consensus.cc:2804] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:02:16.993672  8788 raft_consensus.cc:493] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:16.994132  8788 raft_consensus.cc:3060] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:02:17.007933  8472 raft_consensus.cc:2804] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 13:02:16.975328  8551 raft_consensus.cc:515] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:16.978708  8607 leader_election.cc:290] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:16.976738  8790 raft_consensus.cc:515] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:16.921188  8804 consensus_queue.cc:1048] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.048s
I20260709 13:02:16.938587  8648 raft_consensus.cc:515] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:17.008038  8797 raft_consensus.cc:2804] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:02:17.057222  8797 raft_consensus.cc:493] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:17.057643  8797 raft_consensus.cc:3060] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:02:17.059422  8468 leader_election.cc:290] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:17.063345  8801 consensus_queue.cc:1048] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.142s
I20260709 13:02:17.006773  8727 raft_consensus.cc:493] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:17.069382  8727 raft_consensus.cc:515] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:17.005726  1646 leader_election.cc:304] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:17.072213  8806 raft_consensus.cc:2804] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Leader election won for term 17
I20260709 13:02:17.071563  1494 catalog_manager.cc:5697] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 8 to 10. New cstate: current_term: 10 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 13:02:17.072503  7745 raft_consensus.cc:3055] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 LEADER]: Stepping down as leader of term 17
I20260709 13:02:17.072907  8806 raft_consensus.cc:697] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:02:17.072975  7745 raft_consensus.cc:740] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
W20260709 13:02:17.074833  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 69262.4us
I20260709 13:02:17.074018  7745 consensus_queue.cc:260] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.13, Last appended by leader: 13, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:16.958246  8732 raft_consensus.cc:493] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Starting leader election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:02:17.077522  7745 raft_consensus.cc:3060] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:02:17.080353  1748 raft_consensus.cc:1275] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 18 index: 12. Preceding OpId from leader: term: 19 index: 13. (index mismatch)
W20260709 13:02:17.088948  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 98235.8us
I20260709 13:02:17.094069  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 18 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:02:17.045037  8190 consensus_meta.cc:287] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.621s	user 0.014s	sys 0.000s
I20260709 13:02:17.075239  8778 raft_consensus.cc:515] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:17.096271  8190 raft_consensus.cc:515] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:17.046325  8804 raft_consensus.cc:2804] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:02:17.098562  8190 leader_election.cc:290] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 17 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:17.098759  8804 raft_consensus.cc:493] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:17.056653  8648 leader_election.cc:290] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:17.099239  8804 raft_consensus.cc:3060] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:02:17.082424  8732 raft_consensus.cc:3060] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:02:17.035244  8472 raft_consensus.cc:697] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 19 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:02:17.106719  8472 consensus_queue.cc:237] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:17.047820  8799 log.cc:927] Time spent T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.144s	user 0.001s	sys 0.000s
I20260709 13:02:17.114524  1750 raft_consensus.cc:1275] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 21 index: 13. Preceding OpId from leader: term: 22 index: 14. (index mismatch)
I20260709 13:02:17.074081  8806 consensus_queue.cc:237] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.11, Last appended by leader: 11, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:17.041190  8714 raft_consensus.cc:2804] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Leader pre-election won for term 19
W20260709 13:02:17.116640  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 131400us
I20260709 13:02:17.056531  8506 leader_election.cc:290] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:17.077812  8781 raft_consensus.cc:697] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 18 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:02:17.119434  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:17.120723  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:17.121948  1746 raft_consensus.cc:2468] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 21.
I20260709 13:02:17.037431  8551 leader_election.cc:290] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:17.120785  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:17.054733  8783 raft_consensus.cc:697] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:02:17.102233  8803 raft_consensus.cc:2804] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:02:17.122262  1671 raft_consensus.cc:2468] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 19.
I20260709 13:02:17.098330  8778 leader_election.cc:290] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:17.128226  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:17.130023  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:17.040984  8780 leader_election.cc:290] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:17.133755  2299 leader_election.cc:304] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
W20260709 13:02:17.125324  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 184464us
I20260709 13:02:17.125561  2334 raft_consensus.cc:2468] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 19.
I20260709 13:02:17.141657  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 17 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:17.144104  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:17.146310  2301 leader_election.cc:304] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:02:17.118923  8714 raft_consensus.cc:493] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:17.152081  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:17.159209  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:17.135965  8803 raft_consensus.cc:493] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:17.152746  1750 raft_consensus.cc:2468] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 13.
I20260709 13:02:17.167156  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 18 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:17.139393  8711 leader_election.cc:290] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:17.168073  1646 leader_election.cc:304] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:17.143601  8650 raft_consensus.cc:2749] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:02:17.169268  2337 raft_consensus.cc:2393] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 18: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:02:17.169618  7745 consensus_queue.cc:1059] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 14.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 13:02:17.173450  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:17.152303  8714 raft_consensus.cc:3060] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:02:17.175184  7745 raft_consensus.cc:3055] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:02:17.177171  7745 raft_consensus.cc:740] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:02:17.175285  2334 raft_consensus.cc:2468] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 19.
I20260709 13:02:17.185878  1605 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 2945) took 1122 ms. Trace:
I20260709 13:02:17.186024  7745 consensus_queue.cc:260] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:17.189913  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:17.190893  2334 raft_consensus.cc:2468] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 13.
I20260709 13:02:17.176137  2301 leader_election.cc:304] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:02:17.192054  1753 raft_consensus.cc:3060] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:02:17.186652  1605 rpcz_store.cc:276] 0709 13:02:16.063141 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:16.434720 (+371579us) service_pool.cc:224] Handling call
0709 13:02:16.447560 (+ 12840us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:02:16.680814 (+233254us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:16.680866 (+    52us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:02:16.681668 (+   802us) log.cc:844] Serialized 52 byte log entry
0709 13:02:16.941146 (+259478us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:02:16.957602 (+ 16456us) log.cc:844] Serialized 14 byte log entry
0709 13:02:16.989900 (+ 32298us) log.cc:844] Serialized 14 byte log entry
0709 13:02:16.990180 (+   280us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:16.992487 (+  2307us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:17.177665 (+185178us) raft_consensus.cc:1685] finished
0709 13:02:17.177690 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:17.185785 (+  8095us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":32054,"prepare.queue_time_us":16626,"prepare.run_cpu_time_us":97,"prepare.run_wall_time_us":100,"raft.queue_time_us":354207,"raft.run_cpu_time_us":510,"raft.run_wall_time_us":45141,"thread_start_us":202276,"threads_started":2,"wal-append.queue_time_us":345989}
I20260709 13:02:17.172034  8790 leader_election.cc:290] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:17.167923  8803 raft_consensus.cc:3060] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:02:17.147464  8755 consensus_queue.cc:1048] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.028s
I20260709 13:02:17.217113  7745 raft_consensus.cc:3060] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:02:17.202962  2301 leader_election.cc:304] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:17.203622  8810 raft_consensus.cc:2749] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
W20260709 13:02:17.203713  1717 leader_election.cc:336] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 21 election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 1.578s (SENT)
I20260709 13:02:17.239267  1717 leader_election.cc:304] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:17.172276  8768 leader_election.cc:290] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:17.214911  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:17.217197  8781 consensus_queue.cc:237] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.12, Last appended by leader: 12, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:17.253509  8767 raft_consensus.cc:2804] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Leader pre-election won for term 14
W20260709 13:02:17.254001  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 50364.4us
I20260709 13:02:17.254401  8819 raft_consensus.cc:2749] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
W20260709 13:02:17.254848  1717 leader_election.cc:336] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 18 election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 1.535s (SENT)
I20260709 13:02:17.254012  8767 raft_consensus.cc:493] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:17.255898  8767 raft_consensus.cc:3060] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:02:17.228480  8783 consensus_queue.cc:237] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.12, Last appended by leader: 12, Current term: 20, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:17.092267  1494 catalog_manager.cc:5697] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 11 to 13, leader changed from af520b67945d443b901bbeca37cc5701 (127.0.107.195) to 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193). New cstate: current_term: 13 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:02:17.259111  2337 raft_consensus.cc:1240] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 16. Current term is 17. Ops: []
I20260709 13:02:17.178642  8779 leader_election.cc:290] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:17.172686  8506 consensus_queue.cc:1048] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.044s
I20260709 13:02:17.232398  8814 consensus_queue.cc:1059] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 18.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.061s
W20260709 13:02:17.222652  8694 consensus_meta.cc:287] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.799s	user 0.012s	sys 0.003s
W20260709 13:02:17.234040  8753 consensus_meta.cc:287] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 1.153s	user 0.014s	sys 0.000s
I20260709 13:02:17.218973  8468 raft_consensus.cc:2804] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 13:02:17.195859  8780 raft_consensus.cc:2804] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:02:17.243985  1644 leader_election.cc:304] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:02:17.251756  1750 raft_consensus.cc:2468] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 19.
I20260709 13:02:17.260442  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
W20260709 13:02:17.255899  1717 leader_election.cc:336] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 18 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 1.509s (SENT)
I20260709 13:02:17.244588  7745 consensus_queue.cc:1048] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.018s
I20260709 13:02:17.231845  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:17.274546  1754 raft_consensus.cc:2410] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for term 18 because replica has last-logged OpId of term: 16 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 12.
W20260709 13:02:17.287847  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 43966.9us
I20260709 13:02:17.289116  1644 leader_election.cc:304] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:02:17.217850  8727 leader_election.cc:290] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:17.293186  1753 raft_consensus.cc:2468] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 19.
I20260709 13:02:17.270419  1604 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40154 (request call id 2850) took 1409 ms. Trace:
W20260709 13:02:17.308416  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 140510us
I20260709 13:02:17.296231  1604 rpcz_store.cc:276] 0709 13:02:15.861438 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:16.373231 (+511793us) service_pool.cc:224] Handling call
0709 13:02:16.387176 (+ 13945us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:16.387415 (+   239us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 13:02:16.722329 (+334914us) log.cc:844] Serialized 14 byte log entry
0709 13:02:16.809073 (+ 86744us) log.cc:844] Serialized 14 byte log entry
0709 13:02:17.074669 (+265596us) spinlock_profiling.cc:243] Waited 3.67 ms on lock 0x7b18001df2c0. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f4263057e07 00007f426beb2ab9 00007f426307469b 00007f426307293f 00007f4263072455 00007f426615a060 00007f426616c716 00007f426617d287 00007f426617d208
0709 13:02:17.223275 (+148606us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:17.223340 (+    65us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:02:17.270125 (+ 46785us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:17.270212 (+    87us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:17.270350 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":16237,"prepare.queue_time_us":60415,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":91,"raft.queue_time_us":5851,"raft.run_cpu_time_us":131,"raft.run_wall_time_us":8188,"spinlock_wait_cycles":24042752,"thread_start_us":492412,"threads_started":3,"wal-append.queue_time_us":144148}
W20260709 13:02:17.270529  8784 consensus_meta.cc:287] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.854s	user 0.008s	sys 0.007s
I20260709 13:02:17.268224  8468 raft_consensus.cc:493] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:17.326462  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
W20260709 13:02:17.329528  1570 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:17.295109  8733 consensus_meta.cc:287] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.888s	user 0.015s	sys 0.000s
I20260709 13:02:17.344040  2336 raft_consensus.cc:2468] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 14.
I20260709 13:02:17.361810  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:02:17.363606  1570 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:17.407122  1673 raft_consensus.cc:2468] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 20.
W20260709 13:02:17.358806  1717 leader_election.cc:336] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 1.555s (SENT)
I20260709 13:02:17.359455  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 18 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:17.324263  8773 raft_consensus.cc:2804] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Leader election won for term 19
I20260709 13:02:17.331693  8468 raft_consensus.cc:3060] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:02:17.322165  8786 raft_consensus.cc:515] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:17.344763  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:17.264533  8788 raft_consensus.cc:515] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:17.267004  8824 consensus_queue.cc:1059] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 14.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.066s
I20260709 13:02:17.343567  8733 raft_consensus.cc:515] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:17.423996  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 143468us
I20260709 13:02:17.407925  1717 leader_election.cc:304] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:17.411619  1674 raft_consensus.cc:2468] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 17.
I20260709 13:02:17.331453  8784 raft_consensus.cc:515] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:17.321856  8794 raft_consensus.cc:2749] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20260709 13:02:17.429013  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 156836us
I20260709 13:02:17.368937  8820 raft_consensus.cc:2749] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20260709 13:02:17.432538  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 73837.8us
W20260709 13:02:17.433961  1717 leader_election.cc:336] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 1.813s (SENT)
I20260709 13:02:17.434412  1717 leader_election.cc:304] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 13:02:17.439692  1717 leader_election.cc:336] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 1.889s (SENT)
I20260709 13:02:17.440955  1717 leader_election.cc:304] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:17.447999  1677 raft_consensus.cc:1240] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 16. Current term is 17. Ops: []
I20260709 13:02:17.424124  8773 raft_consensus.cc:697] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 19 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:02:17.440272  8779 raft_consensus.cc:2749] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:02:17.364360  1494 catalog_manager.cc:5697] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 10 to 12. New cstate: current_term: 12 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
W20260709 13:02:17.456166  1717 leader_election.cc:336] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 1.932s (SENT)
I20260709 13:02:17.457659  1717 leader_election.cc:304] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:17.443536  8694 raft_consensus.cc:2749] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:02:17.442399  1569 leader_election.cc:304] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:02:17.479655  8732 raft_consensus.cc:515] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:17.481988  8732 leader_election.cc:290] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 15 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:17.485404  1672 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57092 (request call id 1545) took 1052 ms. Trace:
I20260709 13:02:17.479549  8733 leader_election.cc:290] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 22 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:17.485908  1672 rpcz_store.cc:276] 0709 13:02:16.432753 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:16.434610 (+  1857us) service_pool.cc:224] Handling call
0709 13:02:17.485185 (+1050575us) spinlock_profiling.cc:243] Waited 231 ms on lock 0x7b54000fe454. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:02:17.485319 (+   134us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":532063616}
W20260709 13:02:17.488866  8698 consensus_meta.cc:287] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.575s	user 0.010s	sys 0.001s
I20260709 13:02:17.488546  8496 consensus_queue.cc:260] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.15, Last appended by leader: 15, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:17.497246  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:17.500325  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:17.458828  8779 raft_consensus.cc:2749] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:02:17.456902  8827 raft_consensus.cc:493] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:02:17.509215  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 18 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:02:17.499843  8813 log.cc:927] Time spent T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.178s	user 0.000s	sys 0.001s
I20260709 13:02:17.515069  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:17.520143  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 18 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:17.524693  8714 raft_consensus.cc:515] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:17.528362  8714 leader_election.cc:290] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 19 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:17.467813  8781 raft_consensus.cc:2749] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 13:02:17.528996  8815 raft_consensus.cc:3055] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 13:02:17.529667  8815 raft_consensus.cc:740] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
W20260709 13:02:17.530655  8823 log.cc:927] Time spent T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.181s	user 0.000s	sys 0.001s
I20260709 13:02:17.530761  8815 consensus_queue.cc:260] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 19.15, Last appended by leader: 15, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:17.532711  8826 log.cc:927] Time spent T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.153s	user 0.001s	sys 0.000s
I20260709 13:02:17.536832  2333 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "d21485c094584689917b2a97b2d3c624"
dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
mode: GRACEFUL
new_leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
 from {username='slave'} at 127.0.0.1:35790
W20260709 13:02:17.532712  2336 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49082 (request call id 1258) took 1648 ms (client timeout 1871 ms). Trace:
W20260709 13:02:17.538492  2336 rpcz_store.cc:269] 0709 13:02:15.884645 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:15.911888 (+ 27243us) service_pool.cc:224] Handling call
0709 13:02:17.532514 (+1620626us) spinlock_profiling.cc:243] Waited 1.34 s on lock 0x7b5400443754. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:02:17.532636 (+   122us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3082048384}
I20260709 13:02:17.541899  1717 leader_election.cc:304] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:17.503356  8732 consensus_queue.cc:1059] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 16.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.045s
I20260709 13:02:17.536075  8773 consensus_queue.cc:237] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.12, Last appended by leader: 12, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:17.563699  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:17.565641  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
W20260709 13:02:17.546185  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 103966us
I20260709 13:02:17.548879  1603 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40154 (request call id 2849) took 1691 ms. Trace:
I20260709 13:02:17.613726  1675 raft_consensus.cc:1275] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 19 index: 14. (index mismatch)
I20260709 13:02:17.617635  1603 rpcz_store.cc:276] 0709 13:02:15.857347 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:16.356718 (+499371us) service_pool.cc:224] Handling call
0709 13:02:16.584137 (+227419us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:16.600772 (+ 16635us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:17.548583 (+947811us) spinlock_profiling.cc:243] Waited 227 ms on lock 0x7b54004222d0. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661618ad 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:02:17.548787 (+   204us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":522184064}
I20260709 13:02:17.577062  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 18 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:17.619215  1672 raft_consensus.cc:2376] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 18. Re-sending same reply.
I20260709 13:02:17.616401  1750 raft_consensus.cc:3060] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Advancing to term 21
W20260709 13:02:17.547647  1598 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40154 (request call id 2836) took 1917 ms (client timeout 1578 ms). Trace:
W20260709 13:02:17.600253  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 53938.9us
W20260709 13:02:17.621940  1598 rpcz_store.cc:269] 0709 13:02:15.630563 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:15.630732 (+   169us) service_pool.cc:224] Handling call
0709 13:02:17.547409 (+1916677us) spinlock_profiling.cc:243] Waited 563 ms on lock 0x7b54002990d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:02:17.547544 (+   135us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1295596928}
I20260709 13:02:17.561029  8694 consensus_queue.cc:1048] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.073s
I20260709 13:02:17.631897  1569 leader_election.cc:304] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:02:17.522272  8496 raft_consensus.cc:3060] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:02:17.529022  8714 raft_consensus.cc:2804] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Leader pre-election won for term 21
W20260709 13:02:17.563441  8822 log.cc:927] Time spent T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.312s	user 0.003s	sys 0.000s
W20260709 13:02:17.596566  8832 log.cc:927] Time spent T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.130s	user 0.000s	sys 0.001s
I20260709 13:02:17.558727  8825 raft_consensus.cc:493] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:17.566425  8801 raft_consensus.cc:493] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:17.648418  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:17.638850  8781 consensus_queue.cc:1059] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 16.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.007s
I20260709 13:02:17.544646  8788 leader_election.cc:290] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:17.504743  8698 raft_consensus.cc:515] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:17.651844  1674 raft_consensus.cc:3060] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:02:17.573711  8779 raft_consensus.cc:2749] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:02:17.658211  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:17.658654  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 19 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:17.659224  1672 raft_consensus.cc:3060] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:02:17.659601  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:17.660423  1745 raft_consensus.cc:2410] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 for term 19 because replica has last-logged OpId of term: 18 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 12.
I20260709 13:02:17.661237  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:02:17.668507  8838 log.cc:927] Time spent T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.054s	user 0.000s	sys 0.000s
I20260709 13:02:17.670951  1673 raft_consensus.cc:1275] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 15 index: 12. (index mismatch)
I20260709 13:02:17.670610  8801 raft_consensus.cc:515] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:17.637981  8733 consensus_queue.cc:1059] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 15.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.020s
I20260709 13:02:17.514853  8827 raft_consensus.cc:515] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:17.679564  8824 raft_consensus.cc:2804] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:02:17.680011  8824 raft_consensus.cc:493] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:17.680361  8824 raft_consensus.cc:3060] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:02:17.681025  8827 leader_election.cc:290] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:17.682112  8698 leader_election.cc:290] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 19 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:17.682559  8468 raft_consensus.cc:515] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:02:17.678829  8831 log.cc:927] Time spent T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.181s	user 0.001s	sys 0.000s
I20260709 13:02:17.656971  2299 leader_election.cc:304] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:02:17.684677  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 19 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:17.688900  8468 leader_election.cc:290] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 22 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:02:17.692138  8804 consensus_meta.cc:287] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.592s	user 0.009s	sys 0.006s
I20260709 13:02:17.655295  8825 raft_consensus.cc:515] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:17.649323  8714 raft_consensus.cc:493] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:17.678791  8801 leader_election.cc:290] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:17.696906  8733 raft_consensus.cc:3055] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 13:02:17.699540  8767 raft_consensus.cc:515] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:17.743343  8733 raft_consensus.cc:740] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:02:17.702841  1672 raft_consensus.cc:2468] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 19.
I20260709 13:02:17.693159  1680 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57092 (request call id 1555) took 1151 ms. Trace:
I20260709 13:02:17.696621  1750 raft_consensus.cc:2468] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 21.
I20260709 13:02:18.042894  1750 raft_consensus.cc:1275] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 18 index: 10. Preceding OpId from leader: term: 20 index: 11. (index mismatch)
I20260709 13:02:18.044340  1750 raft_consensus.cc:1240] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 18 LEADER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 17. Current term is 18. Ops: [17.13-17.13]
I20260709 13:02:17.828330  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:18.048743  1745 raft_consensus.cc:1240] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 20 LEADER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 19. Current term is 20. Ops: [19.15-19.15]
I20260709 13:02:17.864548  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:18.050832  1753 raft_consensus.cc:2468] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 16.
I20260709 13:02:18.052167  1753 raft_consensus.cc:1240] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 21. Current term is 22. Ops: [21.16-21.16]
W20260709 13:02:18.052325  1602 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40154 (request call id 2868) took 2053 ms (client timeout 2203 ms). Trace:
W20260709 13:02:18.052789  1602 rpcz_store.cc:269] 0709 13:02:15.998739 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:16.209249 (+210510us) service_pool.cc:224] Handling call
0709 13:02:18.052138 (+1842889us) spinlock_profiling.cc:243] Waited 628 ms on lock 0x7b54003edfd4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:02:18.052256 (+   118us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1443372288}
I20260709 13:02:18.050201  1745 raft_consensus.cc:1275] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 14 index: 13. Preceding OpId from leader: term: 17 index: 15. (index mismatch)
W20260709 13:02:17.887535  1599 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40154 (request call id 2846) took 2097 ms (client timeout 2589 ms). Trace:
W20260709 13:02:18.058621  1599 rpcz_store.cc:269] 0709 13:02:15.790644 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:15.848774 (+ 58130us) service_pool.cc:224] Handling call
0709 13:02:17.887286 (+2038512us) spinlock_profiling.cc:243] Waited 10.7 ms on lock 0x7b5400071054. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:02:17.887460 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":24723328}
I20260709 13:02:17.894351  1754 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55840 (request call id 3067) took 1093 ms. Trace:
I20260709 13:02:18.062135  1754 rpcz_store.cc:276] 0709 13:02:16.801317 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:16.801456 (+   139us) service_pool.cc:224] Handling call
0709 13:02:17.894096 (+1092640us) spinlock_profiling.cc:243] Waited 467 ms on lock 0x7b54001d51d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:02:17.894276 (+   180us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1074125696}
I20260709 13:02:17.894452  1674 raft_consensus.cc:2468] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 15.
I20260709 13:02:17.900441  1747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55840 (request call id 3064) took 1128 ms. Trace:
I20260709 13:02:17.930522  8845 raft_consensus.cc:3055] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 LEADER]: Stepping down as leader of term 16
I20260709 13:02:18.064193  1747 rpcz_store.cc:276] 0709 13:02:16.772050 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:16.772195 (+   145us) service_pool.cc:224] Handling call
0709 13:02:16.772608 (+   413us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:16.772893 (+   285us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:16.772934 (+    41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:17.857804 (+1084870us) spinlock_profiling.cc:243] Waited 14.5 ms on lock 0x7b2c000bc788. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4264a5ba11 00007f4264a6b372 00007f426619da01 00007f426616243a 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2
0709 13:02:17.858404 (+   600us) log.cc:844] Serialized 26 byte log entry
0709 13:02:17.897755 (+ 39351us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:02:17.897800 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:17.898077 (+   277us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:17.900095 (+  2018us) raft_consensus.cc:1685] finished
0709 13:02:17.900120 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:17.900360 (+   240us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":33302784,"thread_start_us":38071,"threads_started":1,"wal-append.queue_time_us":39112}
I20260709 13:02:18.065253  8845 raft_consensus.cc:740] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:02:18.066817  8845 consensus_queue.cc:260] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.11, Last appended by leader: 11, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:18.069450  8845 raft_consensus.cc:3060] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:02:17.554956  1494 catalog_manager.cc:5697] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 9 to 11. New cstate: current_term: 11 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 13:02:18.072431  1494 catalog_manager.cc:5697] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 12 to 17. New cstate: current_term: 17 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } }
I20260709 13:02:17.985630  1676 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52766 (request call id 3167) took 1267 ms. Trace:
I20260709 13:02:18.074602  1676 rpcz_store.cc:276] 0709 13:02:16.717815 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:16.771727 (+ 53912us) service_pool.cc:224] Handling call
0709 13:02:17.985357 (+1213630us) spinlock_profiling.cc:243] Waited 37.5 ms on lock 0x7b54000d8754. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:02:17.985542 (+   185us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":86288768}
W20260709 13:02:17.909911  2298 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:18.260758  8820 raft_consensus.cc:493] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:17.775517  2332 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39738 (request call id 1248) took 1675 ms. Trace:
I20260709 13:02:17.822165  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:17.821328  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
W20260709 13:02:17.710426  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 53583.4us
I20260709 13:02:18.433653  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:17.822000  2301 leader_election.cc:304] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:02:17.765305  8727 raft_consensus.cc:493] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:18.446233  8727 raft_consensus.cc:515] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:17.817579  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:17.879113  2330 raft_consensus.cc:1240] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 15 LEADER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 14. Current term is 15. Ops: []
I20260709 13:02:17.719588  8714 raft_consensus.cc:3060] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:02:17.898047  8344 raft_consensus.cc:2804] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:02:17.902149  8507 raft_consensus.cc:493] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:18.477535  8507 raft_consensus.cc:515] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:18.480611  8507 leader_election.cc:290] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:17.901293  8585 leader_election.cc:290] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:17.898277  8626 raft_consensus.cc:3055] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 19 LEADER]: Stepping down as leader of term 19
W20260709 13:02:18.484098  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 662490us
I20260709 13:02:18.484498  8626 raft_consensus.cc:740] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:02:17.855186  8764 raft_consensus.cc:3055] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 13:02:18.486100  8764 raft_consensus.cc:740] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:02:17.840332  8368 raft_consensus.cc:493] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:02:17.748453  8733 consensus_queue.cc:260] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.9, Last appended by leader: 9, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:17.763994  8756 leader_election.cc:290] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:17.718228  8763 raft_consensus.cc:2804] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 13:02:18.591455  8763 raft_consensus.cc:697] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 18 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:02:18.592680  8763 consensus_queue.cc:237] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:17.747543  8767 leader_election.cc:290] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 14 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:17.883049  8771 raft_consensus.cc:493] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:18.596725  8771 raft_consensus.cc:3060] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:02:18.556897  8764 consensus_queue.cc:260] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.13, Last appended by leader: 13, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:18.598133  8764 raft_consensus.cc:3060] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:02:17.649780  8784 leader_election.cc:290] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 17 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:17.649264  8786 leader_election.cc:290] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:17.773291  8792 raft_consensus.cc:3055] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 13:02:18.601140  8792 raft_consensus.cc:740] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:02:18.602324  8792 consensus_queue.cc:260] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 16.7, Last appended by leader: 7, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:18.603637  8792 raft_consensus.cc:3060] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Advancing to term 17
W20260709 13:02:17.715374  8797 consensus_meta.cc:287] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.653s	user 0.010s	sys 0.005s
I20260709 13:02:17.742650  8803 raft_consensus.cc:515] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:17.761967  8825 leader_election.cc:290] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:17.717156  8804 raft_consensus.cc:515] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:18.623638  8804 leader_election.cc:290] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 21 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:17.774236  8824 raft_consensus.cc:515] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:18.626858  2337 raft_consensus.cc:2393] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 21: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
W20260709 13:02:18.628508  2337 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49082 (request call id 1265) took 1406 ms (client timeout 1596 ms). Trace:
W20260709 13:02:18.628932  2337 rpcz_store.cc:269] 0709 13:02:17.222074 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:17.324060 (+101986us) service_pool.cc:224] Handling call
0709 13:02:18.628284 (+1304224us) spinlock_profiling.cc:243] Waited 1.2 s on lock 0x7b540083f754. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:02:18.628438 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2767664384}
W20260709 13:02:18.029587  8814 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.707s	user 0.007s	sys 0.005s
W20260709 13:02:18.630898  8814 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.326s	user 0.007s	sys 0.005s
I20260709 13:02:18.631335  8814 consensus_queue.cc:1243] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer 816fadcea8fc4893882f5f9b9f22a03d log is divergent from this leader: its last log entry 19.13 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 12
I20260709 13:02:18.632805  7745 raft_consensus.cc:3055] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 LEADER]: Stepping down as leader of term 18
I20260709 13:02:18.633147  7745 raft_consensus.cc:740] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:02:18.031088  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:18.633905  7745 consensus_queue.cc:260] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.12, Last appended by leader: 12, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:18.634737  1749 raft_consensus.cc:3060] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:02:18.635131  7745 raft_consensus.cc:3060] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:02:18.039428  1680 rpcz_store.cc:276] 0709 13:02:16.542181 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:16.590184 (+ 48003us) service_pool.cc:224] Handling call
0709 13:02:16.654154 (+ 63970us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:16.804676 (+150522us) raft_consensus.cc:1534] Early marking committed up to index 15
0709 13:02:17.070967 (+266291us) spinlock_profiling.cc:243] Waited 11.3 ms on lock 0x7b18002b9840. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f4265d0a4bc 00007f426305848b 00007f426beb2ab9 00007f426307469b 00007f426307293f 00007f4263072455 00007f4269b5a3c7 00007f426616c9a7 00007f426617c0bf
0709 13:02:17.071143 (+   176us) log.cc:844] Serialized 14 byte log entry
0709 13:02:17.134954 (+ 63811us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:17.144982 (+ 10028us) log.cc:844] Serialized 26 byte log entry
0709 13:02:17.145163 (+   181us) raft_consensus.cc:1645] Marking committed up to 15
0709 13:02:17.145193 (+    30us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:17.145263 (+    70us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:17.535851 (+390588us) spinlock_profiling.cc:243] Waited 28.2 ms on lock 0x7b4400022b4c. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f4262f9bc85 00007f4262f9c6a2 00007f4265d0e1db 00007f4265d09aaf 00007f4265cfd86e 00007f4265cfcbb6 00007f4265cfc7d8 00007f4265d06221 00007f4265d061d9 00007f4265d06169
0709 13:02:17.692934 (+157083us) raft_consensus.cc:1685] finished
0709 13:02:17.692960 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:17.693091 (+   131us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":22400,"prepare.queue_time_us":98365,"prepare.run_cpu_time_us":65,"prepare.run_wall_time_us":2169,"raft.queue_time_us":166666,"raft.run_cpu_time_us":255,"raft.run_wall_time_us":6682,"spinlock_wait_cycles":90793088,"thread_start_us":202719,"threads_started":3,"wal-append.queue_time_us":119147}
I20260709 13:02:18.087186  1643 leader_election.cc:304] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:18.092005  1718 leader_election.cc:304] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
W20260709 13:02:18.643137  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 551515us
I20260709 13:02:18.644151  1718 leader_election.cc:304] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:18.645078  8771 raft_consensus.cc:515] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:18.100924  1677 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57092 (request call id 1553) took 1659 ms. Trace:
I20260709 13:02:18.112069  8537 leader_election.cc:290] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:02:18.155146  1570 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:18.647078  1570 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:18.646617  1677 rpcz_store.cc:276] 0709 13:02:16.441225 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:16.538947 (+ 97722us) service_pool.cc:224] Handling call
0709 13:02:16.539420 (+   473us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:17.449380 (+909960us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:18.100564 (+651184us) spinlock_profiling.cc:243] Waited 909 ms on lock 0x7b54001e82d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:02:18.100843 (+   279us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2089913728}
I20260709 13:02:18.168404  8794 raft_consensus.cc:493] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:18.647791  8771 leader_election.cc:290] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 15 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:18.648151  8794 raft_consensus.cc:515] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:18.649537  8771 raft_consensus.cc:493] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:18.180866  8854 raft_consensus.cc:493] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:18.650336  8845 raft_consensus.cc:493] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:18.650744  8854 raft_consensus.cc:515] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:18.652999  8854 leader_election.cc:290] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:18.194629  8730 raft_consensus.cc:3055] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:02:18.653158  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:18.653610  8730 raft_consensus.cc:740] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:02:18.654415  1598 raft_consensus.cc:1240] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 20. Current term is 21. Ops: []
I20260709 13:02:18.198462  1752 raft_consensus.cc:1275] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 16 index: 13. Preceding OpId from leader: term: 17 index: 13. (term mismatch)
I20260709 13:02:18.655043  1752 pending_rounds.cc:85] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701: Aborting all ops after (but not including) 12
I20260709 13:02:18.655289  1752 pending_rounds.cc:107] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701: Aborting uncommitted NO_OP operation due to leader change: 16.13
I20260709 13:02:18.655532  1752 raft_consensus.cc:2889] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:02:18.657050  1752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43156 (request call id 1422) took 1528 ms. Trace:
I20260709 13:02:18.184680  8833 raft_consensus.cc:493] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:18.657354  1752 rpcz_store.cc:276] 0709 13:02:17.128422 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:17.128545 (+   123us) service_pool.cc:224] Handling call
0709 13:02:18.197946 (+1069401us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:18.656000 (+458054us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:18.656862 (+   862us) spinlock_profiling.cc:243] Waited 1.06 s on lock 0x7b54001d51d0. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661618ad 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:02:18.656994 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2435713280}
I20260709 13:02:18.657774  8833 raft_consensus.cc:515] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:18.221022  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:18.659003  1747 raft_consensus.cc:2393] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 19: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:02:18.659966  8833 leader_election.cc:290] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:18.660100  1747 raft_consensus.cc:1240] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 20. Current term is 21. Ops: []
W20260709 13:02:18.237257  1720 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.539s	user 0.029s	sys 0.014s
I20260709 13:02:18.261617  8820 raft_consensus.cc:515] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:18.661139  1605 raft_consensus.cc:1240] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 16. Current term is 17. Ops: []
I20260709 13:02:18.647852  8848 raft_consensus.cc:2804] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:02:18.662020  8848 raft_consensus.cc:493] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:18.662380  8848 raft_consensus.cc:3060] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Advancing to term 20
W20260709 13:02:18.661144  1720 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.963s	user 0.030s	sys 0.014s
I20260709 13:02:18.283718  8798 raft_consensus.cc:493] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:02:18.664006  8798 raft_consensus.cc:515] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:18.663530  8820 leader_election.cc:290] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:18.665618  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:18.335520  2336 raft_consensus.cc:2376] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 14. Re-sending same reply.
I20260709 13:02:18.666289  1605 raft_consensus.cc:2468] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 13.
I20260709 13:02:18.338230  8829 consensus_queue.cc:1059] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 13:02:18.342661  2334 raft_consensus.cc:2393] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 17: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:02:18.669340  2334 raft_consensus.cc:1240] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 16. Current term is 17. Ops: []
I20260709 13:02:18.409158  1596 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40154 (request call id 2857) took 2475 ms. Trace:
I20260709 13:02:18.671051  1596 rpcz_store.cc:276] 0709 13:02:15.933877 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:16.380537 (+446660us) service_pool.cc:224] Handling call
0709 13:02:16.380865 (+   328us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:02:16.381330 (+   465us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:16.381364 (+    34us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:02:16.429463 (+ 48099us) log.cc:844] Serialized 52 byte log entry
0709 13:02:16.713275 (+283812us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:02:16.758548 (+ 45273us) log.cc:844] Serialized 14 byte log entry
0709 13:02:16.758983 (+   435us) log.cc:844] Serialized 14 byte log entry
0709 13:02:16.759211 (+   228us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:16.759322 (+   111us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:17.777221 (+1017899us) spinlock_profiling.cc:243] Waited 15.2 ms on lock 0x7b180005ecc0. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f42630432a6 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426b08baa1 00007f426b082b37 00007f426b0829ca 00007f426b17a26f
0709 13:02:17.801632 (+ 24411us) spinlock_profiling.cc:243] Waited 4.89 ms on lock 0x7b180005ecc0. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f4263075522 00007f42630777c1 00007f4263077779 00007f4263077709 00007f42630776d1 00007f42630769cd 000000000038f354 000000000038f199 00007f426305883a
0709 13:02:17.898817 (+ 97185us) spinlock_profiling.cc:243] Waited 10.5 ms on lock 0x7b4800003cd8. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426b08baa1 00007f426b082b37 00007f426b0829ca 00007f426b17a26f 00007f426b179f37 00007f426b18e55c 00007f426b18e508 00007f426b18e488 00007f426b18e443 00007f426b18d6ec 00007f426a0b7a96
0709 13:02:18.405151 (+506334us) spinlock_profiling.cc:243] Waited 1.61 ms on lock 0x7b180005ecc0. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f42630753f3 00007f42630777c1 00007f4263077779 00007f4263077709 00007f42630776d1 00007f42630769cd 000000000038f354 000000000038f199 00007f426305883a
0709 13:02:18.408825 (+  3674us) raft_consensus.cc:1685] finished
0709 13:02:18.408858 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:18.409060 (+   202us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":133331,"prepare.run_cpu_time_us":98,"prepare.run_wall_time_us":98,"raft.queue_time_us":15976,"raft.run_cpu_time_us":4959,"raft.run_wall_time_us":1142717,"spinlock_wait_cycles":74154880,"thread_start_us":117122,"threads_started":1,"wal-append.queue_time_us":370705}
W20260709 13:02:18.411942  1571 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.893s	user 0.003s	sys 0.007s
I20260709 13:02:18.422710  8772 raft_consensus.cc:493] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:18.673861  1571 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.155s	user 0.004s	sys 0.008s
I20260709 13:02:18.435523  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 22 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:18.674001  8772 raft_consensus.cc:515] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:18.439126  8623 raft_consensus.cc:2749] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:02:18.460390  2330 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49082 (request call id 1261) took 1239 ms. Trace:
I20260709 13:02:18.676333  2330 rpcz_store.cc:276] 0709 13:02:17.220704 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:17.539151 (+318447us) service_pool.cc:224] Handling call
0709 13:02:17.563943 (+ 24792us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:17.878925 (+314982us) spinlock_profiling.cc:243] Waited 3.09 ms on lock 0x7b54005e4320. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42660d9dc0 00007f42661641c0 00007f4266165231 00007f4266161d8f 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c
0709 13:02:18.460014 (+581089us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:18.460297 (+   283us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":7100800}
I20260709 13:02:18.462046  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 21 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:18.467759  8815 raft_consensus.cc:3060] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:02:18.480736  8403 consensus_queue.cc:1048] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 13:02:18.679111  2335 raft_consensus.cc:2410] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for term 21 because replica has last-logged OpId of term: 18 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 12.
I20260709 13:02:18.482323  8354 consensus_queue.cc:1048] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
W20260709 13:02:18.486852  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 225992us
I20260709 13:02:18.681167  8775 raft_consensus.cc:493] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:18.681926  8775 raft_consensus.cc:515] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:18.682682  2298 leader_election.cc:304] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:18.488085  8585 raft_consensus.cc:493] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:18.438916  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:02:18.489332  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 309730us
I20260709 13:02:18.685832  8820 consensus_queue.cc:1048] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 13:02:18.492093  8344 raft_consensus.cc:493] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:02:18.686800  8344 raft_consensus.cc:3060] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:02:18.492548  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:18.499194  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:18.689260  1751 raft_consensus.cc:3060] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:02:18.500437  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 21 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:18.690881  1754 raft_consensus.cc:2468] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 20.
I20260709 13:02:18.493920  8714 raft_consensus.cc:515] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:18.502745  2301 leader_election.cc:304] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
W20260709 13:02:18.693938  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 191407us
I20260709 13:02:18.330080  2332 rpcz_store.cc:276] 0709 13:02:16.100294 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:16.309712 (+209418us) service_pool.cc:224] Handling call
0709 13:02:16.329139 (+ 19427us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:16.329398 (+   259us) raft_consensus.cc:1534] Early marking committed up to index 15
0709 13:02:16.337568 (+  8170us) log.cc:844] Serialized 14 byte log entry
0709 13:02:17.774797 (+1437229us) spinlock_profiling.cc:243] Waited 1.14 ms on lock 0x7b180015d4e0. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f4265d0a4bc 00007f426305848b 00007f426beb2ab9 00007f426307469b 00007f426307293f 00007f4263074c02 00007f4265cfc20b 00007f4265d01d23 00007f4265d0223a
0709 13:02:17.775047 (+   250us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:17.775111 (+    64us) raft_consensus.cc:1645] Marking committed up to 15
0709 13:02:17.775176 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:17.775259 (+    83us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:17.775415 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":8814,"prepare.run_cpu_time_us":63,"prepare.run_wall_time_us":67,"raft.queue_time_us":1369,"raft.run_cpu_time_us":118,"raft.run_wall_time_us":119,"spinlock_wait_cycles":2628480,"thread_start_us":32865,"threads_started":1,"wal-append.queue_time_us":95949,"wal-append.run_cpu_time_us":2486,"wal-append.run_wall_time_us":1394650}
I20260709 13:02:18.509032  8727 leader_election.cc:290] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:18.517815  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:02:18.527860  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 190933us
W20260709 13:02:18.733188  1717 leader_election.cc:336] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 17 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 1.952s (SENT)
I20260709 13:02:18.733700  1717 leader_election.cc:304] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:18.532744  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:18.548714  1753 raft_consensus.cc:1275] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 20 index: 13. (index mismatch)
W20260709 13:02:18.715278  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 96239.4us
I20260709 13:02:18.526394  8368 raft_consensus.cc:515] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:18.589718  8733 raft_consensus.cc:3060] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:02:17.880724  8780 raft_consensus.cc:493] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:02:18.750833  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 664019us
I20260709 13:02:18.751274  8780 raft_consensus.cc:3060] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:02:18.751705  1643 leader_election.cc:304] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
W20260709 13:02:18.605036  1750 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.550s	user 0.040s	sys 0.088s
W20260709 13:02:18.753290  1750 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.698s	user 0.040s	sys 0.088s
I20260709 13:02:18.614715  8797 raft_consensus.cc:515] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:18.618316  1748 raft_consensus.cc:1275] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 18 index: 12. Preceding OpId from leader: term: 19 index: 13. (index mismatch)
I20260709 13:02:18.619782  8803 leader_election.cc:290] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 19 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:18.650874  8845 raft_consensus.cc:515] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:18.253121  8768 raft_consensus.cc:3055] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 13:02:18.491489  1676 raft_consensus.cc:1275] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 18 index: 10. Preceding OpId from leader: term: 20 index: 11. (index mismatch)
I20260709 13:02:18.503849  1745 raft_consensus.cc:1275] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
I20260709 13:02:18.697881  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:18.778483  1596 raft_consensus.cc:3060] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:02:18.709355  1569 leader_election.cc:304] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:02:18.714802  1747 raft_consensus.cc:1275] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 19 index: 14. (index mismatch)
I20260709 13:02:18.731398  1602 raft_consensus.cc:2393] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 21: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:02:18.630336  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:18.737715  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:02:18.741997  1717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:18.751452  8775 leader_election.cc:290] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:18.747934  8602 raft_consensus.cc:2749] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:02:18.739800  8833 raft_consensus.cc:493] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
W20260709 13:02:18.795816  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 59373.9us
I20260709 13:02:18.748531  8354 raft_consensus.cc:493] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:02:18.739385  8551 raft_consensus.cc:493] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:18.749485  8368 leader_election.cc:290] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:18.806932  8551 raft_consensus.cc:515] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:18.808120  8798 leader_election.cc:290] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:18.809900  8551 leader_election.cc:290] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:18.811020  8714 leader_election.cc:290] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 21 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:18.810952  8507 consensus_queue.cc:1059] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 16.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 13:02:18.756000  2330 raft_consensus.cc:1217] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Deduplicated request from leader. Original: 16.13->[18.14-19.15]   Dedup: 18.14->[19.15-19.15]
I20260709 13:02:18.812784  2330 raft_consensus.cc:1240] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 19. Current term is 20. Ops: [18.14-19.15]
I20260709 13:02:18.761489  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:18.768430  1603 raft_consensus.cc:1275] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 15 index: 14. Preceding OpId from leader: term: 17 index: 15. (index mismatch)
I20260709 13:02:18.769511  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:18.815434  1598 raft_consensus.cc:3060] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:02:18.772251  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:02:18.774864  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:18.775008  1750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55848 (request call id 3228) took 1216 ms. Trace:
I20260709 13:02:18.816267  1750 rpcz_store.cc:276] 0709 13:02:17.558920 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:18.045911 (+486991us) service_pool.cc:224] Handling call
0709 13:02:18.046219 (+   308us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:18.046432 (+   213us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 13:02:18.055010 (+  8578us) log.cc:844] Serialized 14 byte log entry
0709 13:02:18.774536 (+719526us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:18.774613 (+    77us) raft_consensus.cc:1645] Marking committed up to 13
0709 13:02:18.774679 (+    66us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:18.774767 (+    88us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:18.774930 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":20638,"prepare.queue_time_us":128848,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":46,"raft.queue_time_us":21728,"raft.run_cpu_time_us":156,"raft.run_wall_time_us":11131,"thread_start_us":698425,"threads_started":1,"wal-append.queue_time_us":556469}
I20260709 13:02:18.777213  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:18.684198  8585 raft_consensus.cc:515] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:18.822091  1604 raft_consensus.cc:1275] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 15 index: 11. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
I20260709 13:02:18.743279  8767 consensus_queue.cc:1048] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.070s
I20260709 13:02:18.825693  8797 leader_election.cc:290] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 20 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:18.779258  8829 consensus_queue.cc:1243] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer af520b67945d443b901bbeca37cc5701 log is divergent from this leader: its last log entry 20.15 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14
I20260709 13:02:18.738641  8817 raft_consensus.cc:493] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:18.826823  8817 raft_consensus.cc:515] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:18.829046  8817 leader_election.cc:290] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:18.830222  8772 leader_election.cc:290] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:18.744280  8820 consensus_queue.cc:1048] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.069s
I20260709 13:02:18.791010  2332 raft_consensus.cc:1275] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 18 index: 12. Preceding OpId from leader: term: 20 index: 13. (index mismatch)
I20260709 13:02:18.753798  8651 consensus_queue.cc:1059] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 16.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.078s
I20260709 13:02:18.767524  8504 raft_consensus.cc:3060] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:02:18.739118  8814 raft_consensus.cc:493] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:18.750451  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:18.802145  8354 raft_consensus.cc:515] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:18.832866  8814 raft_consensus.cc:515] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:18.841207  8354 leader_election.cc:290] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:18.818248  1671 raft_consensus.cc:3060] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:02:18.841569  8814 leader_election.cc:290] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:18.777943  8768 raft_consensus.cc:740] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:02:18.842945  8768 consensus_queue.cc:260] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 16.8, Last appended by leader: 8, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:18.844309  8768 raft_consensus.cc:3060] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:02:18.778592  8776 raft_consensus.cc:493] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:02:18.776014  8784 raft_consensus.cc:2749] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:02:18.846949  2337 raft_consensus.cc:2468] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 15.
I20260709 13:02:18.740702  8756 consensus_queue.cc:1059] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 19.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.056s
I20260709 13:02:18.819880  8769 raft_consensus.cc:3055] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 13:02:18.738617  8496 raft_consensus.cc:2804] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:02:18.850615  8769 raft_consensus.cc:740] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:02:18.850785  8496 raft_consensus.cc:493] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:18.851224  8496 raft_consensus.cc:3060] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:02:18.851716  8769 consensus_queue.cc:260] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 16.14, Last appended by leader: 14, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:18.853178  8756 raft_consensus.cc:3055] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 13:02:18.853669  8756 raft_consensus.cc:740] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:02:18.854596  8756 consensus_queue.cc:260] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.16, Last appended by leader: 16, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:18.855959  8756 raft_consensus.cc:3060] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:02:18.771287  8748 consensus_queue.cc:1059] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 20.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.086s
I20260709 13:02:18.858479  8748 raft_consensus.cc:3055] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 13:02:18.858837  8748 raft_consensus.cc:740] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:02:18.858896  8626 consensus_queue.cc:260] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.15, Last appended by leader: 15, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:18.860399  8626 raft_consensus.cc:3060] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:02:18.870842  1643 leader_election.cc:304] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:02:18.833643  1605 raft_consensus.cc:2468] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 17.
I20260709 13:02:18.843365  8585 leader_election.cc:290] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:02:18.812598  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 103377us
I20260709 13:02:18.819018  1746 raft_consensus.cc:2468] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 20.
I20260709 13:02:18.849064  1604 raft_consensus.cc:1240] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 17. Current term is 18. Ops: []
I20260709 13:02:18.767705  8824 leader_election.cc:290] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 18 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:18.849439  1646 leader_election.cc:304] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:18.771135  8794 leader_election.cc:290] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:18.887204  1643 leader_election.cc:304] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:18.888304  8798 raft_consensus.cc:2804] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Leader election won for term 16
I20260709 13:02:18.888866  8798 raft_consensus.cc:697] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:02:18.889647  1644 leader_election.cc:304] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:02:18.889889  8798 consensus_queue.cc:237] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.9, Last appended by leader: 9, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:18.890547  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:18.891372  2331 raft_consensus.cc:2393] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 15: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:02:18.891413  8607 consensus_queue.cc:1048] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 13:02:18.892365  8634 raft_consensus.cc:2804] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:02:18.892452  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:18.892897  8634 raft_consensus.cc:493] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:02:18.893374  8634 raft_consensus.cc:3060] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:02:18.888890  1643 leader_election.cc:304] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
W20260709 13:02:18.896623  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 47310.2us
I20260709 13:02:18.654538  8730 consensus_queue.cc:260] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 14.13, Last appended by leader: 13, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:18.876500  1569 leader_election.cc:304] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:02:18.745448  7745 raft_consensus.cc:493] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:18.836367  8714 raft_consensus.cc:2749] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 13:02:18.905954  8354 consensus_queue.cc:1048] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.028s
I20260709 13:02:18.899724  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:18.929340  8845 leader_election.cc:290] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:18.846302  8646 raft_consensus.cc:493] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:18.871302  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:02:18.910279  1718 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.107.194:46261 timed out after 1.756s (SENT) [suppressed 36 similar messages]
I20260709 13:02:18.910885  8794 consensus_queue.cc:1048] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 13:02:18.945493  8794 consensus_queue.cc:1243] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer af520b67945d443b901bbeca37cc5701 log is divergent from this leader: its last log entry 17.12 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 10
I20260709 13:02:18.945621  8646 raft_consensus.cc:515] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:18.864935  8776 raft_consensus.cc:515] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:18.948135  8646 leader_election.cc:290] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:18.948460  8776 leader_election.cc:290] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:18.949697  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:18.897784  8403 raft_consensus.cc:2749] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:02:18.955964  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:18.956740  1602 raft_consensus.cc:2393] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 21: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:02:18.957337  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:02:18.957566  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 81193.8us
I20260709 13:02:18.957985  1603 raft_consensus.cc:2468] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 20.
I20260709 13:02:18.917723  1751 raft_consensus.cc:2468] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 22.
I20260709 13:02:18.919029  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:18.920183  8730 raft_consensus.cc:3060] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:02:18.794866  8833 raft_consensus.cc:515] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:18.908876  8797 raft_consensus.cc:2804] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:02:18.924734  8854 raft_consensus.cc:493] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:18.860044  8748 consensus_queue.cc:260] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.14, Last appended by leader: 14, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:18.932874  1748 raft_consensus.cc:1275] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 15 index: 11. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
I20260709 13:02:18.932817  1598 raft_consensus.cc:2468] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 14.
I20260709 13:02:19.012059  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 23 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:18.944624  1596 raft_consensus.cc:2468] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 17.
I20260709 13:02:18.968853  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 21 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:19.013589  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:19.014129  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:18.909883  7745 raft_consensus.cc:515] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:19.016283  1599 raft_consensus.cc:1275] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 15 index: 12. (index mismatch)
I20260709 13:02:18.867697  1749 raft_consensus.cc:2468] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 14.
W20260709 13:02:18.992784  1719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:19.019835  1749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43156 (request call id 1435) took 1204 ms. Trace:
I20260709 13:02:19.020347  1749 rpcz_store.cc:276] 0709 13:02:17.815483 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:18.030748 (+215265us) service_pool.cc:224] Handling call
0709 13:02:19.019740 (+988992us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:02:19.021394  1718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:19.022197  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 22 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:19.023999  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:02:19.024366  1570 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:19.024842  1673 raft_consensus.cc:2393] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 20: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:02:18.982959  1569 leader_election.cc:304] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:02:19.009727  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:19.010526  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 19 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:19.018779  8829 raft_consensus.cc:2749] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:02:19.028882  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 18 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:19.030133  1676 raft_consensus.cc:3060] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:02:19.030530  1571 leader_election.cc:304] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:02:19.018695  7745 leader_election.cc:290] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:02:19.027839  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 45055.9us
W20260709 13:02:19.028875  2300 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:19.034322  1679 raft_consensus.cc:2393] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 19: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:02:19.030613  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:19.028179  1680 raft_consensus.cc:2468] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 18.
W20260709 13:02:19.040627  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:19.034746  1569 leader_election.cc:304] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
W20260709 13:02:18.863063  8623 consensus_queue.cc:1175] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "816fadcea8fc4893882f5f9b9f22a03d" responder_term: 21 status { last_received { term: 21 index: 16 } last_committed_idx: 15 last_received_current_leader { term: 21 index: 16 } }
I20260709 13:02:18.973699  8784 raft_consensus.cc:2804] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Leader election won for term 17
I20260709 13:02:18.974936  8817 raft_consensus.cc:2804] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:02:18.990420  8797 raft_consensus.cc:493] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:19.049934  1644 leader_election.cc:304] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:02:19.086179  2333 raft_consensus.cc:606] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Received request to transfer leadership to TS 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:19.086818  2333 raft_consensus.cc:614] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 13:02:19.051429  7745 raft_consensus.cc:2749] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 13:02:18.650012  8771 raft_consensus.cc:515] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:18.989277  8814 raft_consensus.cc:493] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:19.088958  8814 raft_consensus.cc:515] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:19.090317  8771 leader_election.cc:290] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:19.078882  2331 raft_consensus.cc:2468] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 20.
I20260709 13:02:19.047240  2332 raft_consensus.cc:1240] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 18 LEADER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 17. Current term is 18. Ops: [17.13-17.13]
I20260709 13:02:19.094663  8814 leader_election.cc:290] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:18.958729  8607 consensus_queue.cc:1048] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.093s
I20260709 13:02:19.098918  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:19.097440  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:19.024912  8694 raft_consensus.cc:2764] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Leader pre-election decision vote started in defunct term 13: won
W20260709 13:02:19.103175  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 214498us
I20260709 13:02:19.103847  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 18 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
W20260709 13:02:19.072800  1775 debug-util.cc:398] Leaking SignalData structure 0x7b0800360b20 after lost signal to thread 431
W20260709 13:02:19.106348  1775 debug-util.cc:398] Leaking SignalData structure 0x7b0800360bc0 after lost signal to thread 434
W20260709 13:02:19.107839  1775 debug-util.cc:398] Leaking SignalData structure 0x7b08002c9dc0 after lost signal to thread 1537
W20260709 13:02:19.109802  1775 debug-util.cc:398] Leaking SignalData structure 0x7b08001b3600 after lost signal to thread 1629
I20260709 13:02:18.989351  8354 raft_consensus.cc:493] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:19.062376  1571 leader_election.cc:304] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 13:02:19.100561  2328 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39726 (request call id 1572) took 1956 ms. Trace:
I20260709 13:02:18.989204  8772 consensus_queue.cc:1048] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.095s
W20260709 13:02:19.248580  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 182646us
W20260709 13:02:19.267819  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 219272us
I20260709 13:02:19.268697  1571 leader_election.cc:304] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:02:19.087298  8804 raft_consensus.cc:493] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:19.275301  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 18 candidate_status { last_received { term: 16 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:19.252226  8880 raft_consensus.cc:2749] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:02:19.096470  8797 raft_consensus.cc:3060] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:02:19.099713  8817 raft_consensus.cc:697] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:02:19.036454  8845 raft_consensus.cc:2804] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Leader election won for term 19
W20260709 13:02:19.113886  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 64099.9us
W20260709 13:02:19.168622  1717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:19.290825  1717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:19.099056  8784 raft_consensus.cc:697] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
W20260709 13:02:19.200414  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:19.256116  1775 debug-util.cc:398] Leaking SignalData structure 0x7b0800098f80 after lost signal to thread 1630
W20260709 13:02:19.293852  1775 debug-util.cc:398] Leaking SignalData structure 0x7b080012f660 after lost signal to thread 1704
I20260709 13:02:19.296345  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:02:19.296474  1775 debug-util.cc:398] Leaking SignalData structure 0x7b08003a1cc0 after lost signal to thread 1778
W20260709 13:02:19.297497  1775 debug-util.cc:398] Leaking SignalData structure 0x7b08003448c0 after lost signal to thread 1779
I20260709 13:02:19.274539  8551 raft_consensus.cc:2804] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Leader pre-election won for term 19
W20260709 13:02:19.259778  2298 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:19.077319  8504 raft_consensus.cc:2749] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:02:19.085557  8824 raft_consensus.cc:2804] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader election won for term 15
W20260709 13:02:19.257422  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 153391us
I20260709 13:02:19.302237  8845 raft_consensus.cc:697] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:02:19.033553  8854 raft_consensus.cc:515] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:19.110369  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 331411us
I20260709 13:02:19.276822  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:19.280733  1571 leader_election.cc:304] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:02:19.249554  8748 raft_consensus.cc:3060] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:02:19.267375  2328 rpcz_store.cc:276] 0709 13:02:17.144351 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:17.144480 (+   129us) service_pool.cc:224] Handling call
0709 13:02:17.154383 (+  9903us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:17.194634 (+ 40251us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:17.194670 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:17.199071 (+  4401us) log.cc:844] Serialized 26 byte log entry
0709 13:02:18.033514 (+834443us) spinlock_profiling.cc:243] Waited 16.7 ms on lock 0x7b18003efbc0. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f4263057e07 00007f426beb2ab9 00007f426307469b 00007f426307293f 00007f4263074c02 00007f4265cfc20b 00007f4265d01d23 00007f4265d020d1 00007f426611da0f
0709 13:02:18.529610 (+496096us) raft_consensus.cc:1645] Marking committed up to 13
0709 13:02:18.770293 (+240683us) log.cc:844] Serialized 14 byte log entry
0709 13:02:19.085996 (+315703us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:19.086097 (+   101us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:19.086139 (+    42us) raft_consensus.cc:1685] finished
0709 13:02:19.100355 (+ 14216us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:19.100496 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":57529,"prepare.queue_time_us":81755,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":51,"raft.queue_time_us":81239,"raft.run_cpu_time_us":300,"raft.run_wall_time_us":38870,"spinlock_wait_cycles":38415616,"thread_start_us":1030753,"threads_started":3,"wal-append.queue_time_us":1294935}
I20260709 13:02:19.284817  8870 raft_consensus.cc:493] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:19.300207  1775 debug-util.cc:398] Leaking SignalData structure 0x7b08000c08e0 after lost signal to thread 2361
I20260709 13:02:19.301016  2337 raft_consensus.cc:2410] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for term 21 because replica has last-logged OpId of term: 20 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 15.
I20260709 13:02:19.256287  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
W20260709 13:02:19.327539  1775 debug-util.cc:398] Leaking SignalData structure 0x7b080035b200 after lost signal to thread 2362
I20260709 13:02:19.308590  1671 raft_consensus.cc:2468] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 21.
I20260709 13:02:19.328742  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:19.281792  8771 raft_consensus.cc:2749] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:02:19.248132  8780 raft_consensus.cc:515] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:19.262581  8814 consensus_queue.cc:1048] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.037s
I20260709 13:02:19.099184  8634 raft_consensus.cc:515] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:19.321740  1676 raft_consensus.cc:2468] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 18.
I20260709 13:02:19.319083  8824 raft_consensus.cc:697] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:02:19.306797  8551 raft_consensus.cc:493] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:19.277328  8496 raft_consensus.cc:515] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:19.283239  8804 raft_consensus.cc:515] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:19.338816  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 58182.6us
W20260709 13:02:19.352466  8848 consensus_meta.cc:287] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.690s	user 0.012s	sys 0.000s
I20260709 13:02:19.340941  2330 raft_consensus.cc:2393] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 21: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:02:19.344753  7745 raft_consensus.cc:2749] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:02:19.364567  1571 leader_election.cc:304] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:02:19.341475  1605 raft_consensus.cc:2468] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 21.
I20260709 13:02:19.353066  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:19.366279  7745 raft_consensus.cc:2749] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:02:19.366848  2329 raft_consensus.cc:2393] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 20: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:02:19.355170  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:02:19.150569  1570 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:19.368786  1570 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:19.360648  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 18 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:19.340095  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 22 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:19.370638  1571 leader_election.cc:304] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:02:19.373309  1672 raft_consensus.cc:1240] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 15. Current term is 16. Ops: []
I20260709 13:02:19.361860  8551 raft_consensus.cc:3060] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:02:19.354245  8764 raft_consensus.cc:493] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:19.375036  1571 leader_election.cc:304] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:02:19.376147  7745 raft_consensus.cc:2749] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20260709 13:02:19.377324  8868 log.cc:927] Time spent T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.114s	user 0.001s	sys 0.000s
W20260709 13:02:19.378165  8864 log.cc:927] Time spent T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.082s	user 0.000s	sys 0.000s
W20260709 13:02:19.378831  8867 log.cc:927] Time spent T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.081s	user 0.001s	sys 0.000s
I20260709 13:02:19.326068  8796 consensus_queue.cc:1048] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 11, Time since last communication: 0.039s
I20260709 13:02:19.385946  8551 raft_consensus.cc:515] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:19.395841  8771 raft_consensus.cc:3060] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:02:19.397447  8551 leader_election.cc:290] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 19 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:19.397660  8780 leader_election.cc:290] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 20 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:19.376447  1571 leader_election.cc:304] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:02:19.400915  8634 leader_election.cc:290] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 16 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:19.394882  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 21 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:19.403375  2328 raft_consensus.cc:2410] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for term 21 because replica has last-logged OpId of term: 18 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 7.
I20260709 13:02:19.404547  2328 raft_consensus.cc:1240] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 14. Current term is 15. Ops: []
I20260709 13:02:19.325160  8354 raft_consensus.cc:515] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:19.416430  8354 leader_election.cc:290] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:19.417007  8764 raft_consensus.cc:515] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:19.418673  8756 consensus_meta.cc:287] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.562s	user 0.008s	sys 0.004s
I20260709 13:02:19.419533  8764 leader_election.cc:290] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:19.356860  8820 consensus_queue.cc:1048] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.039s
I20260709 13:02:19.344336  8815 consensus_queue.cc:1059] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 16.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.029s
I20260709 13:02:19.430478  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:02:19.441447  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 65090.2us
I20260709 13:02:19.443698  1571 leader_election.cc:304] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:02:19.357023  8767 consensus_queue.cc:1048] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.033s
I20260709 13:02:19.445145  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:19.382241  8804 leader_election.cc:290] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:19.449848  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:19.351928  8870 raft_consensus.cc:515] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:19.501019  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:19.501925  1676 raft_consensus.cc:3060] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:02:19.362857  8727 consensus_queue.cc:1059] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.078s
W20260709 13:02:19.482098  1718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:19.350571  8845 consensus_queue.cc:237] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.9, Last appended by leader: 9, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:19.430678  8634 raft_consensus.cc:493] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:19.373835  8848 raft_consensus.cc:515] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:19.477785  1571 leader_election.cc:304] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:02:19.522626  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 22 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:19.534426  1569 leader_election.cc:304] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:02:19.535461  1674 raft_consensus.cc:3060] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Advancing to term 22
W20260709 13:02:19.524546  2301 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:19.447494  8551 raft_consensus.cc:2749] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:02:19.445139  8833 leader_election.cc:290] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:19.536666  1569 leader_election.cc:304] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
W20260709 13:02:19.539638  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 62031.5us
I20260709 13:02:19.540496  1571 leader_election.cc:304] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
W20260709 13:02:19.527880  1752 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.572s	user 0.015s	sys 0.041s
W20260709 13:02:19.543963  1752 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.588s	user 0.016s	sys 0.041s
I20260709 13:02:19.499877  8714 raft_consensus.cc:2749] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:02:19.362264  8784 consensus_queue.cc:237] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 16.7, Last appended by leader: 7, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:19.519465  8880 raft_consensus.cc:493] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:19.605086  8854 leader_election.cc:290] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:19.604782  8880 raft_consensus.cc:515] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:02:19.605907  8344 consensus_meta.cc:287] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.919s	user 0.009s	sys 0.004s
I20260709 13:02:19.608479  8880 leader_election.cc:290] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:19.610402  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:19.608433  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:19.611501  2337 raft_consensus.cc:2468] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 18.
I20260709 13:02:19.612427  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:19.612848  8870 leader_election.cc:290] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:19.606602  8344 raft_consensus.cc:515] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:19.617843  8786 raft_consensus.cc:3055] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 13:02:19.618083  8344 leader_election.cc:290] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 18 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:19.619755  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:19.553468  1676 raft_consensus.cc:2468] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 16.
W20260709 13:02:19.560194  1646 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.587s	user 0.002s	sys 0.018s
I20260709 13:02:19.624312  8756 raft_consensus.cc:3055] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 13:02:19.624776  8756 raft_consensus.cc:740] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
W20260709 13:02:19.604653  2300 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:19.374842  8817 consensus_queue.cc:237] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 13.12, Last appended by leader: 12, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:19.610715  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
W20260709 13:02:19.621711  1646 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.665s	user 0.002s	sys 0.029s
W20260709 13:02:19.628343  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 280451us
I20260709 13:02:19.630018  2298 leader_election.cc:304] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:02:19.634254  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:19.635003  1678 raft_consensus.cc:2393] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 19: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
W20260709 13:02:19.629663  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 717992us
I20260709 13:02:19.625741  8756 consensus_queue.cc:260] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.15, Last appended by leader: 15, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:19.598713  8626 consensus_meta.cc:287] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.719s	user 0.010s	sys 0.003s
I20260709 13:02:19.645406  1678 raft_consensus.cc:1240] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 17. Current term is 18. Ops: [17.13-17.13]
I20260709 13:02:19.544377  8814 raft_consensus.cc:2749] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 13:02:19.545519  7745 raft_consensus.cc:2749] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:02:19.642719  2298 leader_election.cc:304] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:02:19.673506  2298 leader_election.cc:304] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:02:19.545030  8585 raft_consensus.cc:2749] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 13:02:19.649530  1646 leader_election.cc:304] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:19.550607  1571 leader_election.cc:304] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 13:02:19.658223  1671 raft_consensus.cc:1275] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 19 index: 15. Preceding OpId from leader: term: 20 index: 15. (term mismatch)
I20260709 13:02:19.646900  8833 raft_consensus.cc:2749] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:02:19.640038  8344 raft_consensus.cc:493] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
W20260709 13:02:19.643276  8768 consensus_meta.cc:287] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.799s	user 0.011s	sys 0.003s
I20260709 13:02:19.629518  8786 raft_consensus.cc:740] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:02:19.682206  1671 pending_rounds.cc:85] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d: Aborting all ops after (but not including) 14
I20260709 13:02:19.687201  1671 pending_rounds.cc:107] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d: Aborting uncommitted NO_OP operation due to leader change: 19.15
I20260709 13:02:19.687582  1671 raft_consensus.cc:2889] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 13:02:19.683938  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 133495us
I20260709 13:02:19.690979  1571 leader_election.cc:304] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:02:19.692725  7745 raft_consensus.cc:2749] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:02:19.678982  1674 raft_consensus.cc:2410] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for term 22 because replica has last-logged OpId of term: 21 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 15.
I20260709 13:02:19.703783  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:19.704627  1674 raft_consensus.cc:2468] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 20.
I20260709 13:02:19.706190  8833 raft_consensus.cc:2804] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Leader election won for term 21
I20260709 13:02:19.706825  8833 raft_consensus.cc:697] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:02:19.707858  8833 consensus_queue.cc:237] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.14, Last appended by leader: 14, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:19.699019  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:19.710608  8344 raft_consensus.cc:515] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:19.711064  8496 leader_election.cc:290] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 18 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:19.711020  1601 raft_consensus.cc:2376] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 20. Re-sending same reply.
I20260709 13:02:19.712733  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:19.713375  1601 raft_consensus.cc:2468] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 19.
I20260709 13:02:19.713680  8344 leader_election.cc:290] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:19.714268  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:19.715911  1601 raft_consensus.cc:2468] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 19.
I20260709 13:02:19.716857  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:19.717486  1601 raft_consensus.cc:2468] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 21.
I20260709 13:02:19.718335  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:19.718708  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:19.719172  1601 raft_consensus.cc:2393] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 17: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:02:19.719944  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 19 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:19.721201  8848 leader_election.cc:290] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 20 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:19.721727  1601 raft_consensus.cc:2410] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for term 19 because replica has last-logged OpId of term: 16 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 10.
I20260709 13:02:19.722728  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:19.538499  8634 raft_consensus.cc:515] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:19.723241  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:19.724395  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:19.724385  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:19.725346  1601 raft_consensus.cc:3060] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:02:19.725416  8634 leader_election.cc:290] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:19.725494  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:19.726091  8354 consensus_queue.cc:1048] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 13:02:19.534184  8824 consensus_queue.cc:237] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.10, Last appended by leader: 10, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:02:19.826073  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:19.682258  8820 raft_consensus.cc:2804] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Leader election won for term 14
W20260709 13:02:19.686688  8733 consensus_meta.cc:287] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.937s	user 0.008s	sys 0.004s
I20260709 13:02:19.837356  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:19.837713  2328 raft_consensus.cc:1275] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 18 index: 13. (index mismatch)
W20260709 13:02:19.839550  1717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:19.858814  1717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:19.839989  1675 raft_consensus.cc:1240] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 18. Current term is 19. Ops: []
I20260709 13:02:19.727041  8403 raft_consensus.cc:2804] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 13:02:19.683847  8756 raft_consensus.cc:3060] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:02:19.869244  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 18 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:02:19.862104  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 325705us
W20260709 13:02:19.852010  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 202681us
I20260709 13:02:19.903976  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:02:19.875965  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 276365us
I20260709 13:02:19.895865  1601 raft_consensus.cc:2468] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 22.
W20260709 13:02:19.895989  1570 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:19.911816  1643 leader_election.cc:336] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 20 election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 2.239s (SENT)
I20260709 13:02:19.856705  8794 raft_consensus.cc:2804] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Leader election won for term 22
I20260709 13:02:19.923337  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:19.924237  1601 raft_consensus.cc:3060] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:02:19.917671  1675 pending_rounds.cc:85] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d: Aborting all ops after (but not including) 14
I20260709 13:02:19.860800  8764 raft_consensus.cc:493] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:19.930518  1675 pending_rounds.cc:107] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d: Aborting uncommitted NO_OP operation due to leader change: 19.15
I20260709 13:02:19.930989  1675 raft_consensus.cc:2889] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:02:19.920842  1569 leader_election.cc:304] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:02:19.943426  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:19.943571  1601 raft_consensus.cc:2468] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 20.
I20260709 13:02:19.947082  2334 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39726 (request call id 1579) took 1459 ms. Trace:
I20260709 13:02:19.947170  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:19.944440  1569 leader_election.cc:304] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:02:19.949317  1674 raft_consensus.cc:1240] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 14. Current term is 15. Ops: []
I20260709 13:02:19.949514  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:02:19.951037  2298 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:19.916383  8403 raft_consensus.cc:697] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 17 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:02:19.972522  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:19.976292  1751 raft_consensus.cc:1217] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Deduplicated request from leader. Original: 14.10->[16.11-20.13]   Dedup: 16.11->[18.12-20.13]
I20260709 13:02:19.947984  2337 raft_consensus.cc:2468] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 21.
I20260709 13:02:20.007385  1751 pending_rounds.cc:85] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701: Aborting all ops after (but not including) 11
I20260709 13:02:20.007668  1751 pending_rounds.cc:107] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701: Aborting uncommitted NO_OP operation due to leader change: 17.12
I20260709 13:02:20.008021  1751 raft_consensus.cc:2889] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:02:19.922164  8504 raft_consensus.cc:493] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:19.947628  2334 rpcz_store.cc:276] 0709 13:02:18.495666 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:18.670327 (+174661us) service_pool.cc:224] Handling call
0709 13:02:18.670699 (+   372us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:18.706500 (+ 35801us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 13:02:18.990450 (+283950us) log.cc:844] Serialized 14 byte log entry
0709 13:02:19.440195 (+449745us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:19.440270 (+    75us) raft_consensus.cc:1645] Marking committed up to 13
0709 13:02:19.440339 (+    69us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:19.946817 (+506478us) spinlock_profiling.cc:243] Waited 35.5 ms on lock 0x7b540072e554. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:02:19.946871 (+    54us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:19.947014 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":39876,"prepare.queue_time_us":124216,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":45,"raft.queue_time_us":4954,"raft.run_cpu_time_us":206,"raft.run_wall_time_us":51664,"spinlock_wait_cycles":81715072,"thread_start_us":633162,"threads_started":2,"wal-append.queue_time_us":608536}
I20260709 13:02:19.905339  8820 raft_consensus.cc:697] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 14 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:02:19.877051  8814 consensus_queue.cc:1048] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.077s
I20260709 13:02:19.925922  8703 raft_consensus.cc:493] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:20.047034  2335 raft_consensus.cc:1240] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 19. Current term is 20. Ops: []
W20260709 13:02:20.046917  2329 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39738 (request call id 1253) took 1614 ms (client timeout 1618 ms). Trace:
W20260709 13:02:20.056572  2329 rpcz_store.cc:269] 0709 13:02:18.432633 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:18.438554 (+  5921us) service_pool.cc:224] Handling call
0709 13:02:20.046621 (+1608067us) spinlock_profiling.cc:243] Waited 678 ms on lock 0x7b54006bfb54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:02:20.046844 (+   223us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1558588160}
I20260709 13:02:20.058357  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:20.043087  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 18 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:20.050345  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:19.932943  8794 raft_consensus.cc:697] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
W20260709 13:02:20.060554  1717 leader_election.cc:336] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 2.202s (SENT)
W20260709 13:02:20.061482  1717 leader_election.cc:336] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 2.203s (SENT)
I20260709 13:02:20.061120  8794 consensus_queue.cc:237] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 20.13, Last appended by leader: 13, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:02:20.056097  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 111781us
I20260709 13:02:20.065135  1569 leader_election.cc:304] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
W20260709 13:02:20.066609  1717 leader_election.cc:336] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 2.267s (SENT)
I20260709 13:02:20.067272  1717 leader_election.cc:304] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:20.071425  8551 raft_consensus.cc:2804] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:02:20.072021  8551 raft_consensus.cc:493] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:20.072500  8551 raft_consensus.cc:3060] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:02:20.075390  1569 leader_election.cc:304] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:02:20.084177  8714 raft_consensus.cc:2749] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 13:02:20.086908  1569 leader_election.cc:304] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:02:20.046216  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:20.089263  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:20.103026  1601 raft_consensus.cc:2468] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 22.
I20260709 13:02:20.002322  8646 raft_consensus.cc:493] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:20.105286  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:20.106217  1605 raft_consensus.cc:3060] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:02:19.950204  8507 raft_consensus.cc:2804] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:02:20.111881  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:20.119385  1746 raft_consensus.cc:1240] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 19. Current term is 20. Ops: []
I20260709 13:02:20.123617  1752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43156 (request call id 1445) took 1252 ms. Trace:
I20260709 13:02:20.124171  1752 rpcz_store.cc:276] 0709 13:02:18.870709 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:18.870829 (+   120us) service_pool.cc:224] Handling call
0709 13:02:18.898608 (+ 27779us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:18.955046 (+ 56438us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:18.955081 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:18.955616 (+   535us) log.cc:844] Serialized 26 byte log entry
0709 13:02:19.647452 (+691836us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:02:19.647512 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:20.123100 (+475588us) spinlock_profiling.cc:243] Waited 56.1 ms on lock 0x7b54001d51d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:02:20.123178 (+    78us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:20.123242 (+    64us) raft_consensus.cc:1685] finished
0709 13:02:20.123261 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:20.123536 (+   275us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1320,"raft.run_cpu_time_us":165,"raft.run_wall_time_us":30575,"spinlock_wait_cycles":128925056,"thread_start_us":691471,"threads_started":1,"wal-append.queue_time_us":697364}
I20260709 13:02:20.003332  8585 raft_consensus.cc:2749] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:02:20.131722  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:20.047041  8344 raft_consensus.cc:493] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:20.054689  8768 raft_consensus.cc:493] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:20.100566  2299 leader_election.cc:304] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:20.101981  1569 leader_election.cc:304] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1
W20260709 13:02:20.245249  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 143479us
W20260709 13:02:20.218050  2300 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:20.051995  8848 raft_consensus.cc:493] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:20.009747  8354 raft_consensus.cc:493] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:20.237188  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:19.917340  8786 consensus_queue.cc:260] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 16.14, Last appended by leader: 14, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:20.241395  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:19.945577  8764 raft_consensus.cc:515] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:20.254890  1752 raft_consensus.cc:1275] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 18 index: 13. (term mismatch)
I20260709 13:02:20.265048  1752 pending_rounds.cc:85] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701: Aborting all ops after (but not including) 12
I20260709 13:02:20.265347  1752 pending_rounds.cc:107] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701: Aborting uncommitted NO_OP operation due to leader change: 17.13
I20260709 13:02:20.265592  1752 raft_consensus.cc:2889] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:02:20.258653  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:20.260653  1601 raft_consensus.cc:2468] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 17.
I20260709 13:02:20.104328  8780 raft_consensus.cc:3055] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 13:02:20.116065  8714 raft_consensus.cc:2749] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:02:20.286587  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:20.287796  1601 raft_consensus.cc:2393] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 17: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
W20260709 13:02:20.276206  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:20.098891  8895 raft_consensus.cc:2804] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:02:20.286180  1672 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52766 (request call id 3184) took 1318 ms. Trace:
W20260709 13:02:20.288203  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 342999us
I20260709 13:02:20.234966  8650 raft_consensus.cc:2749] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:02:20.304636  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:20.306088  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:20.319291  1752 raft_consensus.cc:3060] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Advancing to term 18
I20260709 13:02:20.307199  1672 rpcz_store.cc:276] 0709 13:02:18.967572 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:18.967734 (+   162us) service_pool.cc:224] Handling call
0709 13:02:19.015316 (+ 47582us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:19.373780 (+358464us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:20.285801 (+912021us) spinlock_profiling.cc:243] Waited 358 ms on lock 0x7b54000591d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:02:20.286086 (+   285us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":823174016}
W20260709 13:02:20.307979  2301 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:20.309633  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 215269us
I20260709 13:02:20.055577  8504 raft_consensus.cc:515] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:20.320768  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:20.321040  2299 leader_election.cc:304] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:02:20.321702  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:20.324211  1601 raft_consensus.cc:2393] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 15: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:02:20.239545  8496 raft_consensus.cc:493] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:20.326577  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:02:20.326736  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 260313us
W20260709 13:02:20.327680  1717 leader_election.cc:336] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 2.589s (SENT)
W20260709 13:02:20.327930  2301 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:20.328150  1717 leader_election.cc:304] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 13:02:20.330013  1717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:20.330484  1717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:20.332557  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:02:20.333477  1719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:20.109654  8703 raft_consensus.cc:515] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:20.113366  8646 raft_consensus.cc:515] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:20.261201  8880 raft_consensus.cc:493] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:20.340786  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
W20260709 13:02:20.345100  2298 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:20.262378  8870 raft_consensus.cc:2804] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Leader election won for term 18
I20260709 13:02:20.307091  8895 raft_consensus.cc:493] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:20.298985  8780 raft_consensus.cc:740] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:02:20.322995  8786 raft_consensus.cc:3060] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:02:20.335206  8815 raft_consensus.cc:2804] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Leader election won for term 21
W20260709 13:02:20.358263  8730 consensus_meta.cc:287] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 1.342s	user 0.017s	sys 0.002s
I20260709 13:02:20.264374  8848 raft_consensus.cc:515] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:20.284778  8354 raft_consensus.cc:515] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:20.263444  8768 raft_consensus.cc:515] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:20.351104  8694 raft_consensus.cc:2749] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:02:20.250240  8344 raft_consensus.cc:515] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:20.390484  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:20.421646  1601 raft_consensus.cc:2393] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 19: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:02:20.401876  1604 raft_consensus.cc:2468] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 15.
W20260709 13:02:20.416440  8899 log.cc:927] Time spent T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.071s	user 0.000s	sys 0.000s
I20260709 13:02:20.417125  8914 raft_consensus.cc:493] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
W20260709 13:02:20.417528  8891 log.cc:927] Time spent T 00000000000000000000000000000000 P 919f6392d4f44fd68987e2c5e7ec37ca: Append to log took a long time: real 0.489s	user 0.001s	sys 0.000s
I20260709 13:02:20.418382  8848 leader_election.cc:290] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 13:02:20.418671  8898 log.cc:927] Time spent T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.119s	user 0.001s	sys 0.000s
W20260709 13:02:20.419390  8897 log.cc:927] Time spent T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.168s	user 0.000s	sys 0.001s
I20260709 13:02:20.451113  1745 raft_consensus.cc:3060] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:02:20.384899  8895 raft_consensus.cc:3060] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:02:20.386757  8623 raft_consensus.cc:493] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:20.387950  8870 raft_consensus.cc:697] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:02:20.392979  8784 raft_consensus.cc:493] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:20.353822  8496 raft_consensus.cc:515] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:20.381292  8880 raft_consensus.cc:515] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:20.577867  8880 leader_election.cc:290] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:20.578110  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
W20260709 13:02:20.413648  8894 log.cc:927] Time spent T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.508s	user 0.000s	sys 0.001s
I20260709 13:02:20.579128  1748 raft_consensus.cc:2393] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 19: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:02:20.354791  8820 consensus_queue.cc:237] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.10, Last appended by leader: 10, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:20.456861  8914 raft_consensus.cc:515] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:20.457783  8876 raft_consensus.cc:2804] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Leader pre-election won for term 21
W20260709 13:02:20.458832  8902 log.cc:927] Time spent T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.062s	user 0.001s	sys 0.000s
I20260709 13:02:20.417928  8780 consensus_queue.cc:260] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.12, Last appended by leader: 12, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:20.460724  8768 leader_election.cc:290] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:20.431411  8650 raft_consensus.cc:493] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:20.462080  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:20.427856  8733 raft_consensus.cc:493] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:20.467731  8703 leader_election.cc:290] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:20.469642  8848 raft_consensus.cc:493] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:20.427680  8909 raft_consensus.cc:493] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
W20260709 13:02:20.431447  8886 log.cc:927] Time spent T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.578s	user 0.001s	sys 0.001s
W20260709 13:02:20.447147  8901 log.cc:927] Time spent T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.437s	user 0.001s	sys 0.000s
W20260709 13:02:20.465938  8797 consensus_meta.cc:287] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 1.170s	user 0.011s	sys 0.000s
I20260709 13:02:20.472312  1747 raft_consensus.cc:2468] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 19.
I20260709 13:02:20.407050  8815 raft_consensus.cc:697] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 21 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:02:20.474990  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 19 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:20.481469  1745 raft_consensus.cc:2468] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 20.
I20260709 13:02:20.338392  8403 consensus_queue.cc:237] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 16.14, Last appended by leader: 14, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:20.489670  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 18 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:20.427500  8714 raft_consensus.cc:493] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:20.531114  1752 raft_consensus.cc:2468] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 18.
I20260709 13:02:20.731974  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:20.732954  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:20.609512  8730 consensus_queue.cc:1059] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 15.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.093s
I20260709 13:02:20.598083  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:20.599529  1644 leader_election.cc:304] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
W20260709 13:02:20.736603  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 137296us
W20260709 13:02:20.738270  1605 consensus_meta.cc:287] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.631s	user 0.012s	sys 0.000s
I20260709 13:02:20.738687  1605 raft_consensus.cc:2468] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 16.
I20260709 13:02:20.587515  8814 raft_consensus.cc:493] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:20.579591  8833 raft_consensus.cc:493] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:20.595322  8876 raft_consensus.cc:493] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Starting leader election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:02:20.665350  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:20.569918  8623 raft_consensus.cc:515] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:20.694651  1643 leader_election.cc:304] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:02:20.700929  2333 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:35790 (request call id 35) took 3454 ms. Trace:
I20260709 13:02:20.614411  8733 raft_consensus.cc:515] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:20.587553  8784 raft_consensus.cc:515] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:20.715631  1717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:20.752213  1571 leader_election.cc:304] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
W20260709 13:02:20.753055  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 62622.2us
I20260709 13:02:20.731539  2332 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39738 (request call id 1259) took 2024 ms. Trace:
W20260709 13:02:20.579161  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 258173us
I20260709 13:02:20.734942  1746 raft_consensus.cc:1240] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 16. Current term is 18. Ops: []
I20260709 13:02:20.734093  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:20.668352  8780 raft_consensus.cc:3060] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:02:20.639627  8848 raft_consensus.cc:515] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:20.762648  8880 raft_consensus.cc:2804] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Leader election won for term 20
I20260709 13:02:20.763247  8880 raft_consensus.cc:697] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 20 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:02:20.764246  8848 leader_election.cc:290] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:02:20.765195  8904 log.cc:927] Time spent T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.313s	user 0.000s	sys 0.001s
I20260709 13:02:20.768435  1597 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 2957) took 1070 ms. Trace:
I20260709 13:02:20.769140  1678 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52778 (request call id 2740) took 1443 ms. Trace:
W20260709 13:02:20.769817  8916 log.cc:927] Time spent T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.123s	user 0.000s	sys 0.000s
I20260709 13:02:20.769604  1678 rpcz_store.cc:276] 0709 13:02:19.326014 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:19.659380 (+333366us) service_pool.cc:224] Handling call
0709 13:02:19.659817 (+   437us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:19.660324 (+   507us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:19.660377 (+    53us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:19.685386 (+ 25009us) log.cc:844] Serialized 26 byte log entry
0709 13:02:20.118335 (+432949us) raft_consensus.cc:1645] Marking committed up to 13
0709 13:02:20.118399 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:20.118520 (+   121us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:20.768890 (+650370us) raft_consensus.cc:1685] finished
0709 13:02:20.768917 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:20.769071 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":148974,"raft.queue_time_us":857,"raft.run_cpu_time_us":161,"raft.run_wall_time_us":9294,"thread_start_us":263702,"threads_started":1,"wal-append.queue_time_us":553084}
I20260709 13:02:20.771723  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:20.743947  8768 raft_consensus.cc:493] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:20.772754  8646 leader_election.cc:290] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:20.747970  8814 raft_consensus.cc:515] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:20.775260  8814 leader_election.cc:290] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:20.739866  1748 raft_consensus.cc:2468] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 21.
I20260709 13:02:20.778111  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:20.777086  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:20.768821  1597 rpcz_store.cc:276] 0709 13:02:19.697614 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:19.697767 (+   153us) service_pool.cc:224] Handling call
0709 13:02:19.838096 (+140329us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:19.914611 (+ 76515us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:19.914662 (+    51us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:19.931161 (+ 16499us) log.cc:844] Serialized 26 byte log entry
0709 13:02:20.445681 (+514520us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:02:20.507491 (+ 61810us) log.cc:844] Serialized 14 byte log entry
0709 13:02:20.507833 (+   342us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:20.507929 (+    96us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:20.768185 (+260256us) raft_consensus.cc:1685] finished
0709 13:02:20.768210 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:20.768351 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":77339,"prepare.queue_time_us":124245,"raft.queue_time_us":78498,"raft.run_cpu_time_us":372,"raft.run_wall_time_us":16042,"spinlock_wait_cycles":34384000,"thread_start_us":390840,"threads_started":1,"wal-append.queue_time_us":592765}
I20260709 13:02:20.781611  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:20.782294  1601 raft_consensus.cc:2468] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 19.
I20260709 13:02:20.740008  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:20.664249  8797 raft_consensus.cc:515] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:02:20.785590  1717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:20.742677  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:20.614642  8650 raft_consensus.cc:515] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:20.699100  8887 raft_consensus.cc:493] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:20.745824  1603 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 2956) took 1048 ms. Trace:
I20260709 13:02:20.754122  1643 leader_election.cc:304] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 13:02:20.787320  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:20.787220  1603 rpcz_store.cc:276] 0709 13:02:19.697219 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:19.697366 (+   147us) service_pool.cc:224] Handling call
0709 13:02:19.697868 (+   502us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:19.698176 (+   308us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:19.698216 (+    40us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:19.710077 (+ 11861us) log.cc:844] Serialized 26 byte log entry
0709 13:02:19.929890 (+219813us) raft_consensus.cc:1645] Marking committed up to 14
0709 13:02:19.929929 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:19.930000 (+    71us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:20.745558 (+815558us) raft_consensus.cc:1685] finished
0709 13:02:20.745584 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:20.745749 (+   165us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":219411,"threads_started":1,"wal-append.queue_time_us":612481}
I20260709 13:02:20.788280  1605 raft_consensus.cc:2376] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 16. Re-sending same reply.
I20260709 13:02:20.788641  1643 leader_election.cc:304] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 13:02:20.735996  8886 log.cc:927] Time spent T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.090s	user 0.000s	sys 0.001s
I20260709 13:02:20.755915  2299 leader_election.cc:304] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:20.756248  2332 rpcz_store.cc:276] 0709 13:02:18.707304 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:18.831900 (+124596us) service_pool.cc:224] Handling call
0709 13:02:18.832276 (+   376us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:19.091436 (+259160us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:20.731129 (+1639693us) spinlock_profiling.cc:243] Waited 215 ms on lock 0x7b54004880d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:02:20.731442 (+   313us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":493720320}
I20260709 13:02:20.748832  2333 rpcz_store.cc:276] 0709 13:02:17.246262 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:17.523801 (+277539us) service_pool.cc:224] Handling call
0709 13:02:20.700639 (+3176838us) spinlock_profiling.cc:243] Waited 1.55 s on lock 0x7b54005b9fd4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f426615c853 00007f426b12a60a 00007f42646b07d4 00007f42646b0762 00007f42646b0691 00007f42646b060c 00007f42646af8c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:02:20.700832 (+   193us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3563126656}
W20260709 13:02:20.758071  1719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:20.791770  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 19 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:20.792984  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 18 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:20.793581  1749 raft_consensus.cc:2468] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 17.
I20260709 13:02:20.795954  8344 leader_election.cc:290] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:20.784905  1602 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 2955) took 1088 ms. Trace:
I20260709 13:02:20.796379  8784 leader_election.cc:290] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:02:20.797868  8917 log.cc:927] Time spent T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.071s	user 0.000s	sys 0.000s
I20260709 13:02:20.796645  1602 rpcz_store.cc:276] 0709 13:02:19.696839 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:19.697018 (+   179us) service_pool.cc:224] Handling call
0709 13:02:19.708321 (+ 11303us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:19.708716 (+   395us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:19.708749 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:19.709269 (+   520us) log.cc:844] Serialized 26 byte log entry
0709 13:02:19.875417 (+166148us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:02:19.924566 (+ 49149us) log.cc:844] Serialized 14 byte log entry
0709 13:02:19.924808 (+   242us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:19.924896 (+    88us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:20.784664 (+859768us) raft_consensus.cc:1685] finished
0709 13:02:20.784692 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:20.784834 (+   142us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":49427,"prepare.run_cpu_time_us":75,"prepare.run_wall_time_us":9527,"raft.queue_time_us":3151,"raft.run_cpu_time_us":46,"raft.run_wall_time_us":46,"spinlock_wait_cycles":5707520,"thread_start_us":165658,"threads_started":1,"wal-append.queue_time_us":160375}
I20260709 13:02:20.640213  8909 raft_consensus.cc:515] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:20.800246  8650 leader_election.cc:290] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:20.800700  8909 leader_election.cc:290] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:20.800954  8887 raft_consensus.cc:515] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:20.801467  1600 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 1766) took 1851 ms. Trace:
I20260709 13:02:20.803256  8887 leader_election.cc:290] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:20.803812  8922 raft_consensus.cc:2749] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 13:02:20.802016  1600 rpcz_store.cc:276] 0709 13:02:18.950358 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:18.973591 (+ 23233us) service_pool.cc:224] Handling call
0709 13:02:19.032096 (+ 58505us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:19.051371 (+ 19275us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 13:02:19.326295 (+274924us) log.cc:844] Serialized 14 byte log entry
0709 13:02:19.489052 (+162757us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:19.504368 (+ 15316us) log.cc:844] Serialized 26 byte log entry
0709 13:02:19.504664 (+   296us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:02:19.505134 (+   470us) log.cc:844] Serialized 14 byte log entry
0709 13:02:19.539974 (+ 34840us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:19.540262 (+   288us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:20.801135 (+1260873us) raft_consensus.cc:1685] finished
0709 13:02:20.801173 (+    38us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:20.801352 (+   179us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":53462,"prepare.queue_time_us":64448,"prepare.run_cpu_time_us":132,"prepare.run_wall_time_us":8044,"raft.queue_time_us":54624,"raft.run_cpu_time_us":311,"raft.run_wall_time_us":3846,"thread_start_us":385074,"threads_started":2,"wal-append.queue_time_us":350737}
I20260709 13:02:20.804730  8928 raft_consensus.cc:3060] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:02:20.805024  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:20.764213  8880 consensus_queue.cc:237] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 18.9, Last appended by leader: 9, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:20.805925  1600 raft_consensus.cc:2468] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 19.
W20260709 13:02:20.801663  8905 log.cc:927] Time spent T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.369s	user 0.001s	sys 0.000s
I20260709 13:02:20.807591  8914 leader_election.cc:290] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:20.809422  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 18 candidate_status { last_received { term: 16 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:20.826629  1752 raft_consensus.cc:2468] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 17.
W20260709 13:02:20.827085  2331 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39726 (request call id 1588) took 1958 ms (client timeout 2332 ms). Trace:
W20260709 13:02:20.827534  2331 rpcz_store.cc:269] 0709 13:02:18.868454 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:18.892173 (+ 23719us) service_pool.cc:224] Handling call
0709 13:02:20.826777 (+1934604us) spinlock_profiling.cc:243] Waited 186 ms on lock 0x7b540077d654. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:02:20.826902 (+   125us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":427245312}
I20260709 13:02:20.719708  8870 consensus_queue.cc:237] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 15.14, Last appended by leader: 14, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:20.829763  2301 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.500s	user 0.029s	sys 0.067s
W20260709 13:02:20.830179  2301 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.501s	user 0.029s	sys 0.067s
W20260709 13:02:20.830346  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 74529.9us
I20260709 13:02:20.831488  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:20.811184  8354 leader_election.cc:290] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:20.772758  8768 raft_consensus.cc:515] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:20.837333  8507 raft_consensus.cc:493] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:20.837854  8507 raft_consensus.cc:3060] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:02:20.838860  1599 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 2953) took 1145 ms. Trace:
W20260709 13:02:20.839725  2299 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:20.839255  1599 rpcz_store.cc:276] 0709 13:02:19.693818 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:19.694190 (+   372us) service_pool.cc:224] Handling call
0709 13:02:19.694743 (+   553us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:19.695083 (+   340us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:19.695122 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:19.695807 (+   685us) log.cc:844] Serialized 26 byte log entry
0709 13:02:19.933323 (+237516us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:02:19.933382 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:19.934111 (+   729us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:20.838602 (+904491us) raft_consensus.cc:1685] finished
0709 13:02:20.838626 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:20.838776 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":74415,"threads_started":1,"wal-append.queue_time_us":544140}
I20260709 13:02:20.745144  8703 raft_consensus.cc:2804] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:02:20.812834  8496 leader_election.cc:290] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:20.748134  8876 raft_consensus.cc:3060] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:02:20.815028  8797 leader_election.cc:290] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 21 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:20.741703  8714 raft_consensus.cc:515] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:20.816757  8756 consensus_meta.cc:287] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.924s	user 0.012s	sys 0.001s
W20260709 13:02:20.777064  8551 consensus_meta.cc:287] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.704s	user 0.016s	sys 0.000s
I20260709 13:02:20.818496  8769 raft_consensus.cc:3060] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:02:20.857862  8551 raft_consensus.cc:515] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:20.859239  1745 raft_consensus.cc:1240] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 17. Current term is 18. Ops: []
I20260709 13:02:20.776080  8504 leader_election.cc:290] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:02:20.853806  8748 consensus_meta.cc:287] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 1.523s	user 0.010s	sys 0.009s
I20260709 13:02:20.822033  1672 raft_consensus.cc:1240] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 20. Current term is 21. Ops: [20.11-20.11]
I20260709 13:02:20.861531  8551 leader_election.cc:290] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 19 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:02:20.847996  8771 consensus_meta.cc:287] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 1.452s	user 0.016s	sys 0.000s
I20260709 13:02:20.866909  8771 raft_consensus.cc:2749] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20260709 13:02:20.867247  2298 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:20.794490  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 23 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:20.827970  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 21 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:02:20.822889  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 70813.6us
I20260709 13:02:20.872287  1752 raft_consensus.cc:1275] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 20 index: 14. Preceding OpId from leader: term: 21 index: 15. (index mismatch)
I20260709 13:02:20.832270  2331 raft_consensus.cc:2468] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 21.
I20260709 13:02:20.834913  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:02:20.873500  1571 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:20.840864  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 18 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:20.844609  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:20.874854  1597 raft_consensus.cc:2468] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 22.
I20260709 13:02:20.829778  8926 raft_consensus.cc:2749] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
W20260709 13:02:20.839378  8885 log.cc:927] Time spent T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 1.301s	user 0.014s	sys 0.011s
I20260709 13:02:20.876261  1717 leader_election.cc:304] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:20.843923  8921 raft_consensus.cc:2749] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:02:20.875159  1643 leader_election.cc:304] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:02:20.856315  8623 leader_election.cc:290] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:20.750901  8833 raft_consensus.cc:515] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:20.837864  8768 leader_election.cc:290] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:20.869145  8714 leader_election.cc:290] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:20.842310  8703 raft_consensus.cc:493] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:20.902212  8703 raft_consensus.cc:3060] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:02:20.871098  8893 raft_consensus.cc:493] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:20.903851  1672 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52766 (request call id 3193) took 1399 ms. Trace:
I20260709 13:02:20.904387  1672 rpcz_store.cc:276] 0709 13:02:19.504129 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:20.382329 (+878200us) service_pool.cc:224] Handling call
0709 13:02:20.424755 (+ 42426us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:20.860344 (+435589us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:20.903434 (+ 43090us) spinlock_profiling.cc:243] Waited 397 ms on lock 0x7b54000fda54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:02:20.903761 (+   327us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":913399552}
I20260709 13:02:20.876259  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 19 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:20.874250  1599 raft_consensus.cc:2468] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 17.
I20260709 13:02:20.908123  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:20.909111  1599 raft_consensus.cc:2393] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 14: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:02:20.909653  1599 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40154 (request call id 2875) took 1041 ms. Trace:
I20260709 13:02:20.880874  1598 raft_consensus.cc:2468] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 17.
I20260709 13:02:20.834329  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:20.882781  1571 leader_election.cc:304] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
W20260709 13:02:20.887449  8906 log.cc:927] Time spent T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.086s	user 0.000s	sys 0.000s
I20260709 13:02:20.884828  1753 raft_consensus.cc:2468] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 17.
I20260709 13:02:20.896131  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:20.891810  1643 leader_election.cc:304] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
W20260709 13:02:20.931681  1717 leader_election.cc:336] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 18 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 2.199s (SENT)
I20260709 13:02:20.923378  1596 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 1767) took 1828 ms. Trace:
I20260709 13:02:20.910044  1599 rpcz_store.cc:276] 0709 13:02:19.868199 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:20.907762 (+1039563us) service_pool.cc:224] Handling call
0709 13:02:20.909562 (+  1800us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:20.933130  1596 rpcz_store.cc:276] 0709 13:02:19.094850 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:19.095047 (+   197us) service_pool.cc:224] Handling call
0709 13:02:19.095422 (+   375us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:19.095676 (+   254us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:19.095706 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:19.098601 (+  2895us) log.cc:844] Serialized 26 byte log entry
0709 13:02:19.331355 (+232754us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:02:19.331394 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:19.331464 (+    70us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:20.821501 (+1490037us) spinlock_profiling.cc:243] Waited 8.65 ms on lock 0x7b4400022b4c. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f4262f9bc85 00007f4262f9c6a2 00007f4265d0e1db 00007f4265d09aaf 00007f4265cfd86e 00007f4265cfcbb6 00007f4265cfc7d8 00007f4265d06221 00007f4265d061d9 00007f4265d06169
0709 13:02:20.922972 (+101471us) raft_consensus.cc:1685] finished
0709 13:02:20.922999 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:20.923301 (+   302us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":19906432,"thread_start_us":40213,"threads_started":1,"wal-append.queue_time_us":105544}
I20260709 13:02:20.934336  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:20.927078  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:20.932025  1597 raft_consensus.cc:2393] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 20: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:02:20.936115  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:20.936568  1672 raft_consensus.cc:2393] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 17: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:02:20.937726  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:20.937041  1599 raft_consensus.cc:3060] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:02:20.939325  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:02:20.910153  8923 log.cc:927] Time spent T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.066s	user 0.000s	sys 0.000s
W20260709 13:02:20.943080  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 60424.3us
W20260709 13:02:20.944851  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 53159.5us
I20260709 13:02:20.942734  1672 raft_consensus.cc:2468] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 14.
I20260709 13:02:20.945744  1571 leader_election.cc:304] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:02:20.948119  1571 leader_election.cc:304] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:02:20.892288  8756 raft_consensus.cc:493] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:02:20.950729  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:20.952310  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:20.957162  1596 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40154 (request call id 2873) took 1091 ms. Trace:
I20260709 13:02:20.957726  1596 rpcz_store.cc:276] 0709 13:02:19.867560 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:20.942556 (+1074996us) service_pool.cc:224] Handling call
0709 13:02:20.957073 (+ 14517us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:02:20.949820  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 117550us
I20260709 13:02:20.960579  2328 raft_consensus.cc:1240] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 17 LEADER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 16. Current term is 17. Ops: []
I20260709 13:02:20.961145  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:20.913064  8833 leader_election.cc:290] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:20.914577  8786 raft_consensus.cc:2804] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 13:02:20.972455  8932 raft_consensus.cc:493] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:20.976276  8932 raft_consensus.cc:515] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:20.980331  8932 leader_election.cc:290] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:20.984383  8928 raft_consensus.cc:2749] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:02:20.987900  8895 raft_consensus.cc:515] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:20.996069  1672 raft_consensus.cc:2468] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 15.
I20260709 13:02:20.953994  8730 raft_consensus.cc:2804] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 13:02:20.969914  1599 raft_consensus.cc:2468] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 18.
I20260709 13:02:20.991729  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 18 candidate_status { last_received { term: 16 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:20.999703  1599 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36462 (request call id 1773) took 1047 ms. Trace:
I20260709 13:02:20.999876  1597 raft_consensus.cc:2468] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 17.
I20260709 13:02:21.000217  1599 rpcz_store.cc:276] 0709 13:02:19.952057 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:20.935739 (+983682us) service_pool.cc:224] Handling call
0709 13:02:20.999603 (+ 63864us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:20.967969  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 21 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:20.993568  2333 raft_consensus.cc:2468] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 17.
I20260709 13:02:20.963546  8914 raft_consensus.cc:2804] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:02:20.954734  8768 raft_consensus.cc:493] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:21.001560  8914 raft_consensus.cc:493] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Starting leader election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:02:21.002162  8914 raft_consensus.cc:3060] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:02:21.001794  8768 raft_consensus.cc:515] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:21.005795  8768 leader_election.cc:290] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:20.841773  8815 consensus_queue.cc:237] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 19.15, Last appended by leader: 15, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:20.946871  8926 raft_consensus.cc:2749] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:02:21.009545  1596 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40166 (request call id 2969) took 1307 ms. Trace:
I20260709 13:02:21.014686  1596 rpcz_store.cc:276] 0709 13:02:19.701708 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:20.958848 (+1257140us) service_pool.cc:224] Handling call
0709 13:02:21.009450 (+ 50602us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:20.888200  8919 consensus_queue.cc:1048] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.119s
I20260709 13:02:21.025435  8507 raft_consensus.cc:515] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:21.029271  8507 leader_election.cc:290] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 21 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:21.036554  8876 raft_consensus.cc:515] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:21.040061  1680 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52778 (request call id 2743) took 1688 ms. Trace:
I20260709 13:02:21.042666  2335 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39726 (request call id 1585) took 2545 ms. Trace:
I20260709 13:02:21.043165  2335 rpcz_store.cc:276] 0709 13:02:18.497690 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:18.747104 (+249414us) service_pool.cc:224] Handling call
0709 13:02:20.046794 (+1299690us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:20.056016 (+  9222us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:21.042348 (+986332us) spinlock_profiling.cc:243] Waited 1.3 s on lock 0x7b54006bfb50. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661618ad 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:02:21.042551 (+   203us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2989648256}
W20260709 13:02:21.044152  8929 log.cc:927] Time spent T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.223s	user 0.001s	sys 0.000s
I20260709 13:02:21.046689  1751 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55848 (request call id 3254) took 1075 ms. Trace:
I20260709 13:02:21.017831  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:02:21.019330  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 71326.4us
I20260709 13:02:21.048629  1571 leader_election.cc:304] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:02:20.917090  8893 raft_consensus.cc:515] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:20.992893  8833 raft_consensus.cc:2749] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 13:02:20.943742  8714 consensus_queue.cc:1059] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 18.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.051s
I20260709 13:02:20.997237  8786 raft_consensus.cc:697] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 16 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:02:21.018572  2299 leader_election.cc:304] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:02:20.971839  8756 raft_consensus.cc:515] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:21.050043  1599 raft_consensus.cc:2393] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 17: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
W20260709 13:02:21.051254  1717 leader_election.cc:336] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 1.756s (SENT)
I20260709 13:02:21.023623  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:02:21.079231  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 60796.7us
I20260709 13:02:21.089080  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:21.090212  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 18 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:21.056378  1571 leader_election.cc:304] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:02:21.050930  2301 leader_election.cc:304] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:02:21.040639  1680 rpcz_store.cc:276] 0709 13:02:19.351682 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:19.827747 (+476065us) service_pool.cc:224] Handling call
0709 13:02:19.828260 (+   513us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:02:19.854917 (+ 26657us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:19.854968 (+    51us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:02:19.879428 (+ 24460us) log.cc:844] Serialized 52 byte log entry
0709 13:02:20.251546 (+372118us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:02:20.643906 (+392360us) log.cc:844] Serialized 14 byte log entry
0709 13:02:20.644491 (+   585us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:20.644610 (+   119us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:21.039704 (+395094us) raft_consensus.cc:1685] finished
0709 13:02:21.039741 (+    37us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:21.039965 (+   224us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":63506,"prepare.queue_time_us":128276,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":45,"raft.queue_time_us":28442,"raft.run_cpu_time_us":198,"raft.run_wall_time_us":52279,"spinlock_wait_cycles":6777216,"thread_start_us":564732,"threads_started":2,"wal-append.queue_time_us":849702}
I20260709 13:02:21.091789  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:20.999274  8730 raft_consensus.cc:493] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:21.023070  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:21.095526  1597 raft_consensus.cc:2376] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 16. Re-sending same reply.
I20260709 13:02:21.096211  1571 leader_election.cc:304] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 13:02:21.096407  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:21.015977  8354 raft_consensus.cc:493] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:21.047173  1751 rpcz_store.cc:276] 0709 13:02:19.971707 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:19.973724 (+  2017us) service_pool.cc:224] Handling call
0709 13:02:19.974211 (+   487us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 13:02:20.051221 (+ 77010us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 13:02:20.236635 (+185414us) log.cc:844] Serialized 14 byte log entry
0709 13:02:20.727129 (+490494us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:02:20.734159 (+  7030us) log.cc:844] Serialized 52 byte log entry
0709 13:02:20.734359 (+   200us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:02:20.734408 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:20.734516 (+   108us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:21.043297 (+308781us) spinlock_profiling.cc:243] Waited 8.87 ms on lock 0x7b4400022b4c. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f4262f9bc85 00007f4262f9c6a2 00007f4265d0e1db 00007f4265d09aaf 00007f4265cfd86e 00007f4265cfcbb6 00007f4265cfc7d8 00007f4265d06221 00007f4265d061d9 00007f4265d06169
0709 13:02:21.046220 (+  2923us) raft_consensus.cc:1685] finished
0709 13:02:21.046365 (+   145us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:21.046593 (+   228us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":19017,"prepare.queue_time_us":353053,"prepare.run_cpu_time_us":69,"prepare.run_wall_time_us":72,"raft.queue_time_us":14167,"raft.run_cpu_time_us":317,"raft.run_wall_time_us":57452,"spinlock_wait_cycles":20396288,"thread_start_us":418399,"threads_started":2,"wal-append.queue_time_us":518157}
I20260709 13:02:21.099539  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:21.100457  1602 raft_consensus.cc:2393] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 21: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
W20260709 13:02:21.100858  8933 log.cc:927] Time spent T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.152s	user 0.000s	sys 0.000s
I20260709 13:02:21.101408  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:21.102149  1602 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40166 (request call id 2960) took 1403 ms. Trace:
W20260709 13:02:21.102900  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 96834.7us
I20260709 13:02:21.102980  1602 rpcz_store.cc:276] 0709 13:02:19.698828 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:21.101148 (+1402320us) service_pool.cc:224] Handling call
0709 13:02:21.102087 (+   939us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:21.104275  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:21.105513  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:21.105875  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:21.106259  1602 raft_consensus.cc:2393] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 14: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:02:21.107394  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:21.060392  8551 raft_consensus.cc:2749] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:02:21.108556  1599 raft_consensus.cc:2393] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 19: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:02:21.106786  1597 raft_consensus.cc:2468] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 17.
I20260709 13:02:21.116843  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 17 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:21.117442  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 21 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:21.117641  1602 raft_consensus.cc:2393] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 17: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:02:21.092725  8934 raft_consensus.cc:2804] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:02:21.160902  8934 raft_consensus.cc:493] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:21.161545  8934 raft_consensus.cc:3060] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:02:21.115710  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:21.109349  8354 raft_consensus.cc:515] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:21.166065  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:21.167804  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 19 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:21.118086  1597 raft_consensus.cc:2468] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 20.
I20260709 13:02:21.173790  8727 raft_consensus.cc:493] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:02:21.174582  8727 raft_consensus.cc:515] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:21.175235  1597 raft_consensus.cc:1275] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 15 index: 11. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
I20260709 13:02:21.175989  1597 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 2970) took 1474 ms. Trace:
I20260709 13:02:21.176375  1597 rpcz_store.cc:276] 0709 13:02:19.702019 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:21.174396 (+1472377us) service_pool.cc:224] Handling call
0709 13:02:21.174842 (+   446us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:21.175689 (+   847us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:21.175913 (+   224us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:21.177695  8727 leader_election.cc:290] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:21.178035  1597 raft_consensus.cc:1275] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 18 index: 12. Preceding OpId from leader: term: 19 index: 13. (index mismatch)
I20260709 13:02:21.113700  1571 leader_election.cc:304] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:21.198709  8829 raft_consensus.cc:493] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
W20260709 13:02:21.199038  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 85490.3us
I20260709 13:02:21.199514  8829 raft_consensus.cc:515] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:21.200706  8551 raft_consensus.cc:2804] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:02:21.201292  8551 raft_consensus.cc:493] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:21.201869  8551 raft_consensus.cc:3060] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:02:21.202673  8824 consensus_queue.cc:1059] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 17.13, Next index: 14, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 13:02:21.204193  8824 consensus_queue.cc:1048] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 13:02:21.119086  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 18 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:21.207096  1599 raft_consensus.cc:2410] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for term 18 because replica has last-logged OpId of term: 15 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 10.
I20260709 13:02:21.201887  1571 leader_election.cc:304] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:02:21.209641  1571 leader_election.cc:304] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:02:21.206418  8824 raft_consensus.cc:3055] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 LEADER]: Stepping down as leader of term 17
W20260709 13:02:21.120864  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 70092.3us
I20260709 13:02:21.211098  8824 raft_consensus.cc:740] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:02:21.212163  2301 leader_election.cc:304] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:21.212945  1679 raft_consensus.cc:1240] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 LEADER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 16. Current term is 17. Ops: []
I20260709 13:02:21.213138  2301 leader_election.cc:304] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:02:21.213985  8727 raft_consensus.cc:2804] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 13:02:21.121460  8935 raft_consensus.cc:2749] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 13:02:21.214721  8727 raft_consensus.cc:697] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 18 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:02:21.215229  8935 raft_consensus.cc:2804] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:02:21.215747  8935 raft_consensus.cc:493] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
W20260709 13:02:21.212622  1677 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52766 (request call id 3191) took 1938 ms (client timeout 1733 ms). Trace:
I20260709 13:02:21.110461  8914 raft_consensus.cc:515] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:02:21.216428  1677 rpcz_store.cc:269] 0709 13:02:19.274647 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:19.274912 (+   265us) service_pool.cc:224] Handling call
0709 13:02:21.212355 (+1937443us) spinlock_profiling.cc:243] Waited 647 ms on lock 0x7b54001f4d54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:02:21.212542 (+   187us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1488021888}
I20260709 13:02:21.216073  8727 consensus_queue.cc:237] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.11, Last appended by leader: 11, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:21.135135  1598 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40166 (request call id 2954) took 1438 ms. Trace:
I20260709 13:02:21.217538  1598 rpcz_store.cc:276] 0709 13:02:19.696287 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:19.696474 (+   187us) service_pool.cc:224] Handling call
0709 13:02:21.134905 (+1438431us) spinlock_profiling.cc:243] Waited 1.01 s on lock 0x7b54002a1f54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:02:21.135050 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2326317952}
I20260709 13:02:21.218761  8914 leader_election.cc:290] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 20 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:02:21.219319  1719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:21.137778  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 21 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:21.057897  8820 consensus_queue.cc:1048] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.053s
I20260709 13:02:21.220507  1673 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52778 (request call id 2749) took 1822 ms. Trace:
I20260709 13:02:21.220960  1673 rpcz_store.cc:276] 0709 13:02:19.397720 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:19.627735 (+230015us) service_pool.cc:224] Handling call
0709 13:02:21.137577 (+1509842us) spinlock_profiling.cc:243] Waited 6.24 ms on lock 0x7b5800086a98. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f426303db71 00007f426303da22 00007f426bf56c09 00007f426b0b0ee0 00007f426b17a0ab 00007f426b17a131 00007f426b1280a0 00007f426b127984 00007f42646a26d4 00007f42646a2662
0709 13:02:21.220438 (+ 82861us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":14340864}
I20260709 13:02:21.143141  8895 leader_election.cc:290] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 19 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:21.144073  8817 raft_consensus.cc:3055] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 LEADER]: Stepping down as leader of term 15
I20260709 13:02:21.145901  8756 leader_election.cc:290] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:21.147869  8876 leader_election.cc:290] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 21 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:21.108619  8833 raft_consensus.cc:2749] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20260709 13:02:21.150859  8925 log.cc:927] Time spent T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.333s	user 0.002s	sys 0.004s
I20260709 13:02:21.118075  8714 raft_consensus.cc:2804] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:02:21.158844  8786 consensus_queue.cc:237] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:21.103188  8730 raft_consensus.cc:3060] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:02:21.114934  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 17 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:21.258453  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:21.259289  2336 raft_consensus.cc:2468] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 22.
I20260709 13:02:21.190016  2328 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39738 (request call id 1264) took 1360 ms. Trace:
I20260709 13:02:21.169551  1680 raft_consensus.cc:1240] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 17. Current term is 18. Ops: []
I20260709 13:02:21.193888  1600 raft_consensus.cc:1275] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 20 index: 13. (index mismatch)
I20260709 13:02:21.214100  1597 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 2971) took 1511 ms. Trace:
I20260709 13:02:21.219265  1598 raft_consensus.cc:1240] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 LEADER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 14. Current term is 15. Ops: []
I20260709 13:02:21.224131  8354 leader_election.cc:290] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:21.218082  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:21.214174  1679 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52778 (request call id 2752) took 1815 ms. Trace:
I20260709 13:02:21.271066  1679 rpcz_store.cc:276] 0709 13:02:19.398773 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:19.945624 (+546851us) service_pool.cc:224] Handling call
0709 13:02:21.212730 (+1267106us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:21.213368 (+   638us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:21.213993 (+   625us) spinlock_profiling.cc:243] Waited 1.21 s on lock 0x7b54001f4d50. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661618ad 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:02:21.214109 (+   116us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2783565056}
I20260709 13:02:21.225286  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:21.272401  1753 raft_consensus.cc:2376] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 14. Re-sending same reply.
I20260709 13:02:21.226001  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:21.239830  1571 leader_election.cc:304] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:02:21.242702  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:21.273526  1754 raft_consensus.cc:2468] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 18.
I20260709 13:02:21.274573  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:21.245895  8870 raft_consensus.cc:2749] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 13:02:21.275313  1673 raft_consensus.cc:2468] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 17.
I20260709 13:02:21.276399  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:21.246901  8623 raft_consensus.cc:493] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:21.244102  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:21.287068  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:21.287788  2329 raft_consensus.cc:2410] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for term 21 because replica has last-logged OpId of term: 20 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 14.
I20260709 13:02:21.253554  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 19 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:21.254465  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:21.289256  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:21.290066  2335 raft_consensus.cc:3060] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:02:21.255300  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:02:21.256933  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:21.261497  1598 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40154 (request call id 2872) took 1396 ms. Trace:
I20260709 13:02:21.256356  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:21.290881  1598 rpcz_store.cc:276] 0709 13:02:19.865063 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:21.218629 (+1353566us) service_pool.cc:224] Handling call
0709 13:02:21.219036 (+   407us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:21.261168 (+ 42132us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:21.261417 (+   249us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:21.261785  1597 rpcz_store.cc:276] 0709 13:02:19.702338 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:21.177297 (+1474959us) service_pool.cc:224] Handling call
0709 13:02:21.177675 (+   378us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:21.213790 (+ 36115us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:21.214026 (+   236us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:21.262818  1600 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 2972) took 1525 ms. Trace:
I20260709 13:02:21.261049  2328 rpcz_store.cc:276] 0709 13:02:19.830051 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:19.857507 (+ 27456us) service_pool.cc:224] Handling call
0709 13:02:19.858342 (+   835us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:20.961176 (+1102834us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:21.189720 (+228544us) spinlock_profiling.cc:243] Waited 1.1 s on lock 0x7b54005046d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:02:21.189935 (+   215us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2535459584}
I20260709 13:02:21.292685  1598 raft_consensus.cc:1240] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 LEADER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 18. Current term is 19. Ops: []
W20260709 13:02:21.293334  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:21.293516  1598 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40154 (request call id 2874) took 1425 ms. Trace:
W20260709 13:02:21.293812  2298 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:21.136791  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:21.292866  1600 rpcz_store.cc:276] 0709 13:02:19.737203 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:21.193152 (+1455949us) service_pool.cc:224] Handling call
0709 13:02:21.193452 (+   300us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:21.262501 (+ 69049us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:21.262736 (+   235us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:21.295711  1676 raft_consensus.cc:2393] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 20: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:02:21.268654  1601 raft_consensus.cc:2393] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 17: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:02:21.269407  1680 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57092 (request call id 1574) took 1602 ms. Trace:
I20260709 13:02:21.293869  1598 rpcz_store.cc:276] 0709 13:02:19.867898 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:21.291966 (+1424068us) service_pool.cc:224] Handling call
0709 13:02:21.292408 (+   442us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:21.293254 (+   846us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:21.293458 (+   204us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:21.296684  1680 rpcz_store.cc:276] 0709 13:02:19.667156 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:21.168829 (+1501673us) service_pool.cc:224] Handling call
0709 13:02:21.169290 (+   461us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:21.269029 (+ 99739us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:21.269320 (+   291us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:21.216356  8935 raft_consensus.cc:3060] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:02:21.223567  8817 raft_consensus.cc:740] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:02:21.226346  1569 leader_election.cc:304] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:02:21.276645  2299 leader_election.cc:304] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
W20260709 13:02:21.276284  1488 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet 983621a6fc5b4c9ea0159a648dfe7a36 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER failed with leader 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 3b879799899a499c8d409ee9f1277cd1 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 21 committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
W20260709 13:02:21.314586  1717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:21.289101  1746 raft_consensus.cc:2393] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 20: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:02:21.223598  8820 raft_consensus.cc:493] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:02:21.317696  1644 leader_election.cc:304] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:02:21.318113  1677 raft_consensus.cc:1275] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 18 index: 13. (index mismatch)
I20260709 13:02:21.318189  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:21.304097  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:21.304955  1601 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40154 (request call id 2871) took 1440 ms. Trace:
I20260709 13:02:21.305203  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:21.305022  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:21.306524  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:21.307324  1676 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52778 (request call id 2755) took 1899 ms. Trace:
I20260709 13:02:21.212620  8824 consensus_queue.cc:260] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:21.256443  8714 raft_consensus.cc:493] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:21.344710  8714 raft_consensus.cc:3060] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:02:21.347121  8824 raft_consensus.cc:3060] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:02:21.356323  8764 leader_election.cc:290] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:21.357103   431 auto_leader_rebalancer.cc:452] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 13:02:21.359313  8703 raft_consensus.cc:515] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:21.360592   431 auto_leader_rebalancer.cc:527] All tables' leader rebalancing finished this round
I20260709 13:02:21.298123  1598 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 1771) took 1346 ms. Trace:
I20260709 13:02:21.315899  1597 raft_consensus.cc:1240] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 16. Current term is 17. Ops: []
I20260709 13:02:21.364154  1598 rpcz_store.cc:276] 0709 13:02:19.951543 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:21.297025 (+1345482us) service_pool.cc:224] Handling call
0709 13:02:21.297423 (+   398us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:21.297683 (+   260us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:21.297721 (+    38us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:21.297783 (+    62us) raft_consensus.cc:1645] Marking committed up to 13
0709 13:02:21.297837 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:21.297910 (+    73us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:21.298038 (+   128us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:21.318858  1677 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57092 (request call id 1578) took 1633 ms. Trace:
I20260709 13:02:21.365875  1677 rpcz_store.cc:276] 0709 13:02:19.685025 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:21.317502 (+1632477us) service_pool.cc:224] Handling call
0709 13:02:21.317805 (+   303us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:21.318616 (+   811us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:21.318802 (+   186us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:21.321640  1676 rpcz_store.cc:276] 0709 13:02:19.407837 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:19.621121 (+213284us) service_pool.cc:224] Handling call
0709 13:02:21.136576 (+1515455us) spinlock_profiling.cc:243] Waited 4.13 ms on lock 0x7b5800086a98. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f426303db71 00007f426303da22 00007f426bf56c09 00007f426b0b0ee0 00007f426b17a0ab 00007f426b17a131 00007f426b1280a0 00007f426b127984 00007f42646a26d4 00007f42646a2662
0709 13:02:21.307227 (+170651us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":9506176}
W20260709 13:02:21.322446  1717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:21.319157  2329 raft_consensus.cc:2468] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 15.
I20260709 13:02:21.329619  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 18 candidate_status { last_received { term: 16 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:21.330878  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:21.331739  2334 raft_consensus.cc:1275] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 20 index: 14. Preceding OpId from leader: term: 21 index: 15. (index mismatch)
I20260709 13:02:21.319782  1601 rpcz_store.cc:276] 0709 13:02:19.864756 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:21.094953 (+1230197us) service_pool.cc:224] Handling call
0709 13:02:21.304875 (+209922us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:21.244967  8772 consensus_queue.cc:1059] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.041s
I20260709 13:02:21.343715  8876 raft_consensus.cc:2804] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:02:21.344166  8928 raft_consensus.cc:2804] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 13:02:21.304420  8833 raft_consensus.cc:2749] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:02:21.371187  1677 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57092 (request call id 1579) took 1664 ms. Trace:
I20260709 13:02:21.377521  8817 consensus_queue.cc:260] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:21.313308  8623 raft_consensus.cc:515] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:21.396193  8829 leader_election.cc:290] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:02:21.396891  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 122644us
I20260709 13:02:21.365339  1678 raft_consensus.cc:2468] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 21.
I20260709 13:02:21.448235  1571 leader_election.cc:304] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 13:02:21.448292  1671 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52778 (request call id 2736) took 2127 ms. Trace:
I20260709 13:02:21.399562  1597 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40154 (request call id 2878) took 1477 ms. Trace:
I20260709 13:02:21.450834  1597 rpcz_store.cc:276] 0709 13:02:19.931393 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:21.292644 (+1361251us) service_pool.cc:224] Handling call
0709 13:02:21.315645 (+ 23001us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:21.399149 (+ 83504us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:21.399470 (+   321us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:21.407226  1603 raft_consensus.cc:1240] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 18. Current term is 19. Ops: [18.12-18.12]
W20260709 13:02:21.408941  8911 log.cc:927] Time spent T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.942s	user 0.002s	sys 0.001s
I20260709 13:02:21.372391  8820 raft_consensus.cc:515] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:21.411845  1672 raft_consensus.cc:1240] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 15. Current term is 16. Ops: []
I20260709 13:02:21.469297  1603 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40154 (request call id 2880) took 1416 ms. Trace:
I20260709 13:02:21.312054  1673 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57092 (request call id 1577) took 1635 ms. Trace:
I20260709 13:02:21.470788  1673 rpcz_store.cc:276] 0709 13:02:19.676371 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:21.310958 (+1634587us) service_pool.cc:224] Handling call
0709 13:02:21.311377 (+   419us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:21.311631 (+   254us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:21.311661 (+    30us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:21.311724 (+    63us) raft_consensus.cc:1645] Marking committed up to 13
0709 13:02:21.311768 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:21.311835 (+    67us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:21.311977 (+   142us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:21.403411  8876 raft_consensus.cc:493] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:21.423590  2335 raft_consensus.cc:2468] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 19.
I20260709 13:02:21.450608  1671 rpcz_store.cc:276] 0709 13:02:19.321116 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:19.329725 (+  8609us) service_pool.cc:224] Handling call
0709 13:02:19.330021 (+   296us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:19.688202 (+358181us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:21.447776 (+1759574us) spinlock_profiling.cc:243] Waited 328 ms on lock 0x7b54000596d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:02:21.448075 (+   299us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":754054912}
I20260709 13:02:21.533357  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 18 candidate_status { last_received { term: 16 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:21.534912  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 23 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:21.454384  1605 raft_consensus.cc:1240] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 18. Current term is 19. Ops: []
I20260709 13:02:21.469923  1603 rpcz_store.cc:276] 0709 13:02:20.052608 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:21.406221 (+1353613us) service_pool.cc:224] Handling call
0709 13:02:21.406844 (+   623us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:21.468843 (+ 61999us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:21.469201 (+   358us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:21.481177  2298 leader_election.cc:304] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:21.536450  1605 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40154 (request call id 2886) took 1157 ms. Trace:
I20260709 13:02:21.536844  1605 rpcz_store.cc:276] 0709 13:02:20.379412 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:21.453648 (+1074236us) service_pool.cc:224] Handling call
0709 13:02:21.454102 (+   454us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:21.536170 (+ 82068us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:21.536386 (+   216us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:02:21.537070  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 56031.6us
I20260709 13:02:21.487100  8926 raft_consensus.cc:3055] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 LEADER]: Stepping down as leader of term 20
I20260709 13:02:21.486066  8784 consensus_queue.cc:1048] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.002s
I20260709 13:02:21.488888  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:21.491827  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:21.494971  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:21.498556  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:21.500599  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:21.500712  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:02:21.503235  1604 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36462 (request call id 1762) took 2604 ms (client timeout 1686 ms). Trace:
I20260709 13:02:21.664114  1597 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40154 (request call id 2887) took 1278 ms. Trace:
W20260709 13:02:21.506052  8930 log.cc:927] Time spent T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.685s	user 0.002s	sys 0.000s
I20260709 13:02:21.506575  8727 consensus_queue.cc:1048] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 13:02:21.508673  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:21.509809  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 22 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:21.667797  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:21.513721  8829 raft_consensus.cc:493] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:21.517650  1646 leader_election.cc:304] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
W20260709 13:02:21.670188  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 152731us
W20260709 13:02:21.672277  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:21.674290  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:21.520246  8764 raft_consensus.cc:493] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:21.521456  8772 consensus_queue.cc:1048] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 13:02:21.524472  8634 consensus_queue.cc:1048] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 13:02:21.417830  1677 rpcz_store.cc:276] 0709 13:02:19.706509 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:21.367127 (+1660618us) service_pool.cc:224] Handling call
0709 13:02:21.367536 (+   409us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:21.370734 (+  3198us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:21.370780 (+    46us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:21.370843 (+    63us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:02:21.370892 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:21.370976 (+    84us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:21.371117 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:21.411648  8928 raft_consensus.cc:493] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:21.695207  8928 raft_consensus.cc:3060] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:02:21.695050  8764 raft_consensus.cc:515] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:21.697404  8764 leader_election.cc:290] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:21.697993  8368 raft_consensus.cc:493] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:21.461903  1717 leader_election.cc:304] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:21.698553  8368 raft_consensus.cc:515] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:21.699083  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 237394us
I20260709 13:02:21.700896  8368 leader_election.cc:290] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:21.671269  8551 raft_consensus.cc:515] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:02:21.703392  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:21.597067  2301 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:21.645677  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:21.641204  8926 raft_consensus.cc:740] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
W20260709 13:02:21.664417  1604 rpcz_store.cc:269] 0709 13:02:18.899214 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:18.899377 (+   163us) service_pool.cc:224] Handling call
0709 13:02:21.502939 (+2603562us) spinlock_profiling.cc:243] Waited 1.48 s on lock 0x7b54002a1cd4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:02:21.503102 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3392552320}
I20260709 13:02:21.707716  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:21.642402  8817 raft_consensus.cc:3060] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:02:21.708348  1604 raft_consensus.cc:2468] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 15.
I20260709 13:02:21.709414  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:21.666352  2333 raft_consensus.cc:2393] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 20: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:02:21.505828  8776 raft_consensus.cc:493] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:21.711927  8776 raft_consensus.cc:515] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:21.422307  8914 raft_consensus.cc:3055] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 19 LEADER]: Stepping down as leader of term 19
I20260709 13:02:21.714610  8914 raft_consensus.cc:740] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:02:21.665584  1597 rpcz_store.cc:276] 0709 13:02:20.385976 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:21.641371 (+1255395us) service_pool.cc:224] Handling call
0709 13:02:21.663309 (+ 21938us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:21.663612 (+   303us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:21.663652 (+    40us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:21.663715 (+    63us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:02:21.663775 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:21.663848 (+    73us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:21.664023 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:21.510756  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:21.519454  8932 raft_consensus.cc:493] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:21.671984  1646 leader_election.cc:304] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:02:21.679513  1672 raft_consensus.cc:2468] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 16.
W20260709 13:02:21.720338  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 48502.7us
I20260709 13:02:21.720790  1672 raft_consensus.cc:1275] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 19 index: 15. Preceding OpId from leader: term: 21 index: 16. (index mismatch)
I20260709 13:02:21.688304  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:21.693374  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:21.714180  8776 leader_election.cc:290] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:21.643445  8876 raft_consensus.cc:3060] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:02:21.647859  8895 raft_consensus.cc:2804] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 13:02:21.709452  8845 raft_consensus.cc:2764] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Leader pre-election decision vote started in defunct term 16: won
I20260709 13:02:21.705719  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 18 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:21.710925  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:21.685002  8796 raft_consensus.cc:2749] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:02:21.723861  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:21.735519  1597 raft_consensus.cc:2468] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 16.
I20260709 13:02:21.722827  8932 raft_consensus.cc:515] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } }
I20260709 13:02:21.725035  8764 raft_consensus.cc:2804] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Leader pre-election won for term 18
W20260709 13:02:21.732046  1599 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.505s	user 0.024s	sys 0.076s
W20260709 13:02:21.750240  1599 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.524s	user 0.024s	sys 0.077s
I20260709 13:02:21.704520  8551 leader_election.cc:290] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 18 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:21.690471  8829 raft_consensus.cc:515] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:21.646450  8871 raft_consensus.cc:3060] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:02:21.728281  8895 raft_consensus.cc:493] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:21.728524  8634 raft_consensus.cc:2749] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:02:21.767541  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:21.771294  1597 raft_consensus.cc:3060] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:02:21.768743  8954 raft_consensus.cc:493] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:21.715615  8914 consensus_queue.cc:260] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.13, Last appended by leader: 13, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:21.731870  8880 consensus_queue.cc:1059] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 19.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.023s
I20260709 13:02:21.805691  1678 raft_consensus.cc:2376] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 16. Re-sending same reply.
I20260709 13:02:21.749917  8764 raft_consensus.cc:493] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:02:21.807442  8931 thread.cc:641] apply [worker] (thread pool) Time spent creating pthread: real 0.986s	user 0.043s	sys 0.085s
I20260709 13:02:21.771467  8895 raft_consensus.cc:3060] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Advancing to term 22
W20260709 13:02:21.817028  8931 thread.cc:608] apply [worker] (thread pool) Time spent starting thread: real 0.996s	user 0.043s	sys 0.086s
I20260709 13:02:21.813102  8764 raft_consensus.cc:3060] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:02:21.738183  8926 consensus_queue.cc:260] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 20.11, Last appended by leader: 11, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:21.838678  8958 raft_consensus.cc:493] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:21.845758  8958 raft_consensus.cc:515] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:21.848675  8958 leader_election.cc:290] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:02:21.817205  8784 consensus_queue.cc:1175] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "af520b67945d443b901bbeca37cc5701" responder_term: 19 status { last_received { term: 19 index: 13 } last_committed_idx: 12 last_received_current_leader { term: 19 index: 13 } }
I20260709 13:02:21.829346  8914 raft_consensus.cc:3060] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Advancing to term 20
W20260709 13:02:21.854367  8948 log.cc:927] Time spent T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.162s	user 0.000s	sys 0.001s
I20260709 13:02:21.856005  8703 leader_election.cc:290] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 21 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:02:21.863997  8786 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.500s	user 0.000s	sys 0.011s
W20260709 13:02:21.864627  8786 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.501s	user 0.000s	sys 0.011s
I20260709 13:02:21.865370  8786 raft_consensus.cc:493] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:21.866055  8786 raft_consensus.cc:515] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:21.807245  8954 raft_consensus.cc:515] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:21.873036  1674 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52778 (request call id 2754) took 2473 ms. Trace:
W20260709 13:02:21.874841  1717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:21.856344  8623 leader_election.cc:290] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:21.878378  1676 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57092 (request call id 1583) took 2170 ms. Trace:
I20260709 13:02:21.878902  1597 raft_consensus.cc:2468] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 21.
I20260709 13:02:21.863163  8824 raft_consensus.cc:493] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:21.883493  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:21.904754  1671 raft_consensus.cc:2468] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 20.
I20260709 13:02:21.903220  1717 leader_election.cc:304] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:21.906677  2299 leader_election.cc:304] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
W20260709 13:02:21.906963  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 176486us
I20260709 13:02:21.869274  8820 leader_election.cc:290] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:21.877897  8958 raft_consensus.cc:493] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:21.910353  8824 raft_consensus.cc:515] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:21.919361  8824 leader_election.cc:290] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:21.924214  1602 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 2959) took 2225 ms. Trace:
I20260709 13:02:21.924705  1602 rpcz_store.cc:276] 0709 13:02:19.698514 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:21.125170 (+1426656us) service_pool.cc:224] Handling call
0709 13:02:21.125533 (+   363us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:21.127596 (+  2063us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:21.127633 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:21.128180 (+   547us) log.cc:844] Serialized 26 byte log entry
0709 13:02:21.324226 (+196046us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:02:21.325211 (+   985us) log.cc:844] Serialized 14 byte log entry
0709 13:02:21.330963 (+  5752us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:21.331080 (+   117us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:21.923951 (+592871us) raft_consensus.cc:1685] finished
0709 13:02:21.923979 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:21.924133 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5352,"prepare.queue_time_us":302,"prepare.run_cpu_time_us":63,"prepare.run_wall_time_us":48,"raft.queue_time_us":322712,"raft.run_cpu_time_us":278,"raft.run_wall_time_us":284,"thread_start_us":195535,"threads_started":1,"wal-append.queue_time_us":314854}
I20260709 13:02:21.927438  8829 leader_election.cc:290] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:21.878567  8504 raft_consensus.cc:2804] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Leader pre-election won for term 20
W20260709 13:02:21.928856  8935 consensus_meta.cc:287] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.578s	user 0.015s	sys 0.000s
I20260709 13:02:21.880172  8368 consensus_queue.cc:1048] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.054s
I20260709 13:02:21.940124  8954 leader_election.cc:290] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:21.933105  8820 raft_consensus.cc:2804] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:02:21.962942  8820 raft_consensus.cc:493] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:21.963467  8820 raft_consensus.cc:3060] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:02:21.976383  8786 leader_election.cc:290] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:21.979029  8932 leader_election.cc:290] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:21.979838  8932 raft_consensus.cc:493] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:21.980466  8932 raft_consensus.cc:515] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:21.909080  8703 raft_consensus.cc:493] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
W20260709 13:02:21.994052  1749 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.574s	user 0.011s	sys 0.021s
W20260709 13:02:21.994501  1749 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.575s	user 0.012s	sys 0.021s
I20260709 13:02:21.933409  8958 raft_consensus.cc:515] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:21.994810  8771 raft_consensus.cc:2749] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
W20260709 13:02:22.031399  1569 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.717s	user 0.017s	sys 0.020s
W20260709 13:02:22.031795  1569 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.717s	user 0.017s	sys 0.020s
W20260709 13:02:22.032071  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 806166us
I20260709 13:02:21.997946  8786 raft_consensus.cc:493] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:21.936733  8928 raft_consensus.cc:515] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:22.039136  1671 raft_consensus.cc:1275] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 13 index: 13. Preceding OpId from leader: term: 18 index: 15. (index mismatch)
I20260709 13:02:22.046969  1678 raft_consensus.cc:1275] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 16 index: 9. Preceding OpId from leader: term: 19 index: 10. (index mismatch)
I20260709 13:02:22.042191  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:21.989516  8935 raft_consensus.cc:515] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:22.026786  8814 raft_consensus.cc:493] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:22.052294  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 18 candidate_status { last_received { term: 16 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
W20260709 13:02:22.041707  8730 consensus_meta.cc:287] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.770s	user 0.014s	sys 0.001s
I20260709 13:02:21.999977  8703 raft_consensus.cc:515] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:22.058025  1673 raft_consensus.cc:2368] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 22. Current term is 23.
I20260709 13:02:22.058259  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:02:22.059572  1569 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:22.045953  8932 leader_election.cc:290] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:22.046195  8954 raft_consensus.cc:493] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:22.069792  2331 raft_consensus.cc:1240] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 19. Current term is 20. Ops: []
I20260709 13:02:22.068568  1569 leader_election.cc:304] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:02:22.049808  8968 raft_consensus.cc:3060] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:02:22.053457  8786 raft_consensus.cc:515] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:02:22.088014  8714 consensus_meta.cc:287] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.743s	user 0.011s	sys 0.005s
W20260709 13:02:22.094270  8947 log.cc:927] Time spent T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.715s	user 0.005s	sys 0.010s
W20260709 13:02:22.071833  1718 connection.cc:256] RPC call timeout handler was delayed by -1.1306467569999654s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.25520000000000004s for a response
I20260709 13:02:22.097996  1603 raft_consensus.cc:1240] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 16. Current term is 17. Ops: []
W20260709 13:02:22.098315  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:22.101191  1675 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52778 (request call id 2744) took 2745 ms. Trace:
I20260709 13:02:22.101635  1675 rpcz_store.cc:276] 0709 13:02:19.356176 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:19.916870 (+560694us) service_pool.cc:224] Handling call
0709 13:02:19.917389 (+   519us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:20.004563 (+ 87174us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:20.004613 (+    50us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:21.045968 (+1041355us) spinlock_profiling.cc:243] Waited 8.03 ms on lock 0x7b2c00010828. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4264a5ba11 00007f4264a6b372 00007f426619da01 00007f426616243a 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2
0709 13:02:21.046544 (+   576us) log.cc:844] Serialized 26 byte log entry
0709 13:02:21.244021 (+197477us) raft_consensus.cc:1645] Marking committed up to 15
0709 13:02:21.262886 (+ 18865us) log.cc:844] Serialized 14 byte log entry
0709 13:02:21.263326 (+   440us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:21.263471 (+   145us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:21.645667 (+382196us) spinlock_profiling.cc:243] Waited 3.55 ms on lock 0x7b18003f4480. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f42630753f3 00007f42630777c1 00007f4263077779 00007f4263077709 00007f42630776d1 00007f42630769cd 000000000038f354 000000000038f199 00007f426305883a
0709 13:02:22.031145 (+385478us) spinlock_profiling.cc:243] Waited 18.8 ms on lock 0x7b4400022b4c. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f4262f9bc85 00007f4262f9c6a2 00007f4265d0e1db 00007f4265d09aaf 00007f4265cfd86e 00007f4265cfcbb6 00007f4265cfc7d8 00007f4265d06221 00007f4265d061d9 00007f4265d06169
0709 13:02:22.100909 (+ 69764us) raft_consensus.cc:1685] finished
0709 13:02:22.100935 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:22.101105 (+   170us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":19128,"prepare.run_cpu_time_us":93,"prepare.run_wall_time_us":17444,"raft.queue_time_us":4100,"raft.run_cpu_time_us":129,"raft.run_wall_time_us":130,"spinlock_wait_cycles":69943040,"thread_start_us":197022,"threads_started":1,"wal-append.queue_time_us":331953}
I20260709 13:02:22.058014  8814 raft_consensus.cc:515] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:22.061575  8730 raft_consensus.cc:515] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:22.077369  8952 raft_consensus.cc:2804] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader pre-election won for term 16
W20260709 13:02:22.112768  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 209764us
I20260709 13:02:22.115168  1717 leader_election.cc:304] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:22.115386  1603 raft_consensus.cc:1240] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 18. Current term is 19. Ops: []
I20260709 13:02:22.108711  8714 raft_consensus.cc:515] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:22.120659  8714 leader_election.cc:290] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 18 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:22.124681  2298 leader_election.cc:304] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:02:22.127424  8928 leader_election.cc:290] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 23 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:22.127736  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 23 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:22.128695  1605 raft_consensus.cc:2468] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 22.
I20260709 13:02:22.128882  8935 leader_election.cc:290] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 20 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:22.129912  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 18 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:22.130544  8703 leader_election.cc:290] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:22.130877  1598 raft_consensus.cc:1240] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 17. Current term is 18. Ops: [17.12-17.12]
I20260709 13:02:22.131217  1601 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 1774) took 2178 ms. Trace:
I20260709 13:02:22.168591  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:22.134617  8958 leader_election.cc:290] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:22.168457  1601 rpcz_store.cc:276] 0709 13:02:19.952318 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:21.372650 (+1420332us) service_pool.cc:224] Handling call
0709 13:02:21.419713 (+ 47063us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:21.420210 (+   497us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:21.420255 (+    45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:21.420970 (+   715us) log.cc:844] Serialized 26 byte log entry
0709 13:02:21.874325 (+453355us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:02:22.069405 (+195080us) log.cc:844] Serialized 14 byte log entry
0709 13:02:22.069779 (+   374us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:22.069869 (+    90us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:22.130945 (+ 61076us) raft_consensus.cc:1685] finished
0709 13:02:22.130974 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:22.131142 (+   168us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":278908,"prepare.run_cpu_time_us":67,"prepare.run_wall_time_us":67,"raft.queue_time_us":250583,"raft.run_cpu_time_us":369,"raft.run_wall_time_us":21020,"thread_start_us":387833,"threads_started":2,"wal-append.queue_time_us":604162}
I20260709 13:02:22.076800  8954 raft_consensus.cc:515] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:22.140311  8733 leader_election.cc:290] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:22.140133  8820 raft_consensus.cc:515] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:22.146757  8971 consensus_queue.cc:1048] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.033s
I20260709 13:02:22.148772  8871 raft_consensus.cc:2749] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:02:22.149852  8814 leader_election.cc:290] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:21.900529  1676 rpcz_store.cc:276] 0709 13:02:19.707848 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:21.368191 (+1660343us) service_pool.cc:224] Handling call
0709 13:02:21.368562 (+   371us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:21.368863 (+   301us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 13:02:21.416303 (+ 47440us) log.cc:844] Serialized 14 byte log entry
0709 13:02:21.501982 (+ 85679us) spinlock_profiling.cc:243] Waited 6.03 ms on lock 0x7b18003f72a0. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f4263057e07 00007f426beb2ab9 00007f426307469b 00007f426307293f 00007f4263074c02 00007f4265cfc20b 00007f4265d01d23 00007f4265d0223a 00007f426616ca2a
0709 13:02:21.877996 (+376014us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:21.878060 (+    64us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:02:21.878113 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:21.878187 (+    74us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:21.878314 (+   127us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3904,"prepare.queue_time_us":49794,"prepare.run_cpu_time_us":63,"prepare.run_wall_time_us":64,"raft.queue_time_us":8329,"spinlock_wait_cycles":55215744,"thread_start_us":367442,"threads_started":1}
I20260709 13:02:22.127453  2333 raft_consensus.cc:1275] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 17 index: 15. Preceding OpId from leader: term: 18 index: 15. (term mismatch)
I20260709 13:02:22.218255  2333 pending_rounds.cc:85] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1: Aborting all ops after (but not including) 14
I20260709 13:02:22.218549  2333 pending_rounds.cc:107] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1: Aborting uncommitted NO_OP operation due to leader change: 17.15
I20260709 13:02:22.218806  2333 raft_consensus.cc:2889] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:02:22.155229  8876 raft_consensus.cc:515] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:22.156001  8634 raft_consensus.cc:3055] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 LEADER]: Stepping down as leader of term 19
I20260709 13:02:22.230634  8634 raft_consensus.cc:740] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:02:22.231182  1598 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 1772) took 2279 ms. Trace:
I20260709 13:02:22.231602  1598 rpcz_store.cc:276] 0709 13:02:19.951805 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:21.365457 (+1413652us) service_pool.cc:224] Handling call
0709 13:02:21.365958 (+   501us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:22.168312 (+802354us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:22.230908 (+ 62596us) spinlock_profiling.cc:243] Waited 765 ms on lock 0x7b54002161d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:02:22.231109 (+   201us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1759316224}
I20260709 13:02:22.231976  8634 consensus_queue.cc:260] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.13, Last appended by leader: 13, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:22.234875  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:22.235126  8814 raft_consensus.cc:493] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:22.235656  8814 raft_consensus.cc:515] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:22.236138  8727 consensus_queue.cc:1059] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 18.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 13:02:22.237331  8727 raft_consensus.cc:3055] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 13:02:22.237706  8727 raft_consensus.cc:740] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:02:22.238042  8814 leader_election.cc:290] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:02:22.159439  1719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:22.162833  1603 raft_consensus.cc:1275] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
I20260709 13:02:22.164626  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 19 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:22.164969  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:22.169153  1605 raft_consensus.cc:2468] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 16.
W20260709 13:02:22.137257  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:22.086983  8824 consensus_queue.cc:1048] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.033s
I20260709 13:02:22.126420  8368 raft_consensus.cc:2804] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:02:22.252383  2337 raft_consensus.cc:2410] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for term 19 because replica has last-logged OpId of term: 15 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 13.
W20260709 13:02:22.174957  8969 log.cc:927] Time spent T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.080s	user 0.000s	sys 0.001s
I20260709 13:02:22.175446  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:22.256098  2332 raft_consensus.cc:2468] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 15.
I20260709 13:02:22.177529  1599 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 2975) took 2314 ms. Trace:
W20260709 13:02:22.179850  8946 log.cc:927] Time spent T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.836s	user 0.000s	sys 0.002s
I20260709 13:02:22.258244  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:22.258545  8368 raft_consensus.cc:493] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:22.258919  1601 raft_consensus.cc:2376] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 17. Re-sending same reply.
I20260709 13:02:22.259220  8368 raft_consensus.cc:3060] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:02:22.259506  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 19 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:22.259931  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:22.260236  1752 raft_consensus.cc:2468] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 18.
I20260709 13:02:22.260535  1601 raft_consensus.cc:2376] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 22. Re-sending same reply.
I20260709 13:02:22.261560  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:22.263542  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:21.898864  1674 rpcz_store.cc:276] 0709 13:02:19.399379 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:20.100182 (+700803us) service_pool.cc:224] Handling call
0709 13:02:20.131635 (+ 31453us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:20.131902 (+   267us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 13:02:20.259783 (+127881us) log.cc:844] Serialized 14 byte log entry
0709 13:02:21.489583 (+1229800us) spinlock_profiling.cc:243] Waited 2.63 ms on lock 0x7b180042aa20. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f4263057e07 00007f426beb2ab9 00007f426307469b 00007f426307293f 00007f4263074c02 00007f4265cfc20b 00007f4265d01d23 00007f4265d0223a 00007f426616ca2a
0709 13:02:21.872556 (+382973us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:21.872628 (+    72us) raft_consensus.cc:1645] Marking committed up to 13
0709 13:02:21.872688 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:21.872798 (+   110us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:21.872958 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":15034,"prepare.queue_time_us":133144,"prepare.run_cpu_time_us":52,"prepare.run_wall_time_us":55,"raft.queue_time_us":8407,"raft.run_cpu_time_us":171,"raft.run_wall_time_us":15027,"spinlock_wait_cycles":6050688,"thread_start_us":379669,"threads_started":1,"wal-append.queue_time_us":1508856}
I20260709 13:02:22.264194  1605 raft_consensus.cc:2468] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 15.
I20260709 13:02:22.181937  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:22.186623  2301 leader_election.cc:304] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:02:22.262316  1601 raft_consensus.cc:2393] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 17: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:02:22.265093  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 19 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:02:22.265398  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 78927.9us
I20260709 13:02:22.266047  1605 raft_consensus.cc:2393] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 19: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:02:22.191634  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:22.192371  8978 raft_consensus.cc:493] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
W20260709 13:02:22.192466  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 77410.7us
I20260709 13:02:22.256956  1599 rpcz_store.cc:276] 0709 13:02:19.863567 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:21.207843 (+1344276us) service_pool.cc:224] Handling call
0709 13:02:21.208388 (+   545us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 13:02:21.208939 (+   551us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:21.208989 (+    50us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 13:02:21.210038 (+  1049us) log.cc:844] Serialized 78 byte log entry
0709 13:02:21.751066 (+541028us) raft_consensus.cc:1645] Marking committed up to 14
0709 13:02:21.800779 (+ 49713us) log.cc:844] Serialized 14 byte log entry
0709 13:02:21.801210 (+   431us) log.cc:844] Serialized 14 byte log entry
0709 13:02:21.805333 (+  4123us) log.cc:844] Serialized 14 byte log entry
0709 13:02:21.991642 (+186309us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:21.992054 (+   412us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:22.105025 (+112971us) spinlock_profiling.cc:243] Waited 30.2 ms on lock 0x7b18002639e0. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f42630753f3 00007f42630777c1 00007f4263077779 00007f4263077709 00007f42630776d1 00007f42630769cd 000000000038f354 000000000038f199 00007f426305883a
0709 13:02:22.132593 (+ 27568us) spinlock_profiling.cc:243] Waited 3.25 ms on lock 0x7b18002639e0. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f42630753f3 00007f42630777c1 00007f4263077779 00007f4263077709 00007f42630776d1 00007f42630769cd 000000000038f354 000000000038f199 00007f426305883a
0709 13:02:22.149664 (+ 17071us) spinlock_profiling.cc:243] Waited 3.82 ms on lock 0x7b18002639e0. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f42630753f3 00007f42630777c1 00007f4263077779 00007f4263077709 00007f42630776d1 00007f42630769cd 000000000038f354 000000000038f199 00007f426305883a
0709 13:02:22.177269 (+ 27605us) raft_consensus.cc:1685] finished
0709 13:02:22.177298 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:22.177459 (+   161us) inbound_call.cc:177] Queueing success response
0709 13:02:22.204019 (+ 26560us) spinlock_profiling.cc:243] Waited 5.39 ms on lock 0x7b18002639e0. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f42630753f3 00007f42630777c1 00007f4263077779 00007f4263077709 00007f42630776d1 00007f42630769cd 000000000038f354 000000000038f199 00007f426305883a
Metrics: {"mutex_wait_us":92812,"prepare.queue_time_us":128308,"prepare.run_cpu_time_us":152,"prepare.run_wall_time_us":154,"raft.queue_time_us":118825,"raft.run_cpu_time_us":602,"raft.run_wall_time_us":24860,"spinlock_wait_cycles":98051200,"thread_start_us":699085,"threads_started":2,"wal-append.queue_time_us":625438}
I20260709 13:02:22.193125  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:22.270678  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:22.193629  8733 consensus_queue.cc:1048] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 13:02:22.195060  8730 leader_election.cc:290] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 22 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:22.195678  1596 raft_consensus.cc:2393] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 22: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:02:22.193104  8974 raft_consensus.cc:2804] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:02:22.193073  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 19 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:22.277762  8974 raft_consensus.cc:493] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:22.277998  1680 raft_consensus.cc:2468] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 18.
I20260709 13:02:22.279795  2299 leader_election.cc:304] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
W20260709 13:02:22.199111  8900 log.cc:927] Time spent T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 1.803s	user 0.005s	sys 0.023s
I20260709 13:02:22.202052  8786 leader_election.cc:290] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:22.280928  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 20 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:02:22.206162  8934 consensus_meta.cc:287] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 1.044s	user 0.013s	sys 0.003s
I20260709 13:02:22.206643  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:22.283677  2331 raft_consensus.cc:3060] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:02:22.208249  8893 leader_election.cc:290] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:22.210635  8954 leader_election.cc:290] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:22.224339  1569 leader_election.cc:304] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:02:22.226203  8771 consensus_queue.cc:1048] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 13:02:22.229952  8504 raft_consensus.cc:493] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Starting leader election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:02:22.250878  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:22.253839  8634 raft_consensus.cc:3060] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:02:22.254254  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:22.259227  1749 raft_consensus.cc:1275] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 15 index: 9. Preceding OpId from leader: term: 16 index: 10. (index mismatch)
I20260709 13:02:22.322676  1753 raft_consensus.cc:2468] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 22.
W20260709 13:02:22.234278  8817 consensus_meta.cc:287] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.523s	user 0.014s	sys 0.000s
I20260709 13:02:22.270856  2336 raft_consensus.cc:2468] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 17.
I20260709 13:02:22.193852  1718 leader_election.cc:304] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 13:02:22.333325  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 139633us
I20260709 13:02:22.334463  1718 leader_election.cc:304] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:22.266247  2301 leader_election.cc:304] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:02:22.291581  8871 raft_consensus.cc:2804] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:02:22.271442  1599 raft_consensus.cc:3060] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:02:22.273512  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:22.272269  8935 raft_consensus.cc:2804] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:02:22.340744  1605 raft_consensus.cc:2376] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 22. Re-sending same reply.
I20260709 13:02:22.276144  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 17 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:22.342007  1598 raft_consensus.cc:2468] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 16.
I20260709 13:02:22.277380  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 21 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:22.117066  8952 raft_consensus.cc:493] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:22.282325  2334 raft_consensus.cc:2393] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 21: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:02:22.253188  8814 raft_consensus.cc:493] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:22.274950  8978 raft_consensus.cc:515] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:22.287267  8876 leader_election.cc:290] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 18 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:22.289175  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:22.349444  1750 raft_consensus.cc:1275] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 18 index: 9. Preceding OpId from leader: term: 20 index: 10. (index mismatch)
I20260709 13:02:22.290728  8926 raft_consensus.cc:3060] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:02:22.278254  8974 raft_consensus.cc:3060] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:02:22.293149  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:22.357877  1680 raft_consensus.cc:3060] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:02:22.293664  1752 raft_consensus.cc:1275] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 22 index: 14. (index mismatch)
I20260709 13:02:22.358317  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:22.315165  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
W20260709 13:02:22.317502  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 93280us
I20260709 13:02:22.361833  2330 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39726 (request call id 1589) took 3465 ms. Trace:
I20260709 13:02:22.362365  2330 rpcz_store.cc:276] 0709 13:02:18.896325 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:19.375641 (+479316us) service_pool.cc:224] Handling call
0709 13:02:19.445269 (+ 69628us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:19.510420 (+ 65151us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 13:02:19.510482 (+    62us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:19.519603 (+  9121us) log.cc:844] Serialized 26 byte log entry
0709 13:02:19.934032 (+414429us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:02:19.934087 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:19.934171 (+    84us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:22.172910 (+2238739us) spinlock_profiling.cc:243] Waited 5.22 ms on lock 0x7b4400022b4c. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f4262f9bc85 00007f4262f9c6a2 00007f4265d0e1db 00007f4265d09aaf 00007f4265cfd86e 00007f4265cfcbb6 00007f4265cfc7d8 00007f4265d06221 00007f4265d061d9 00007f4265d06169
0709 13:02:22.361469 (+188559us) raft_consensus.cc:1685] finished
0709 13:02:22.361521 (+    52us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:22.361753 (+   232us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":123046,"raft.queue_time_us":68605,"raft.run_cpu_time_us":163,"raft.run_wall_time_us":14636,"spinlock_wait_cycles":12014848,"thread_start_us":306287,"threads_started":1,"wal-append.queue_time_us":700467}
I20260709 13:02:22.284898  8764 raft_consensus.cc:515] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:22.299561  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 21 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:22.326753  1643 leader_election.cc:304] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 13:02:22.367096  1747 raft_consensus.cc:3060] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:02:22.367985  1643 leader_election.cc:304] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
W20260709 13:02:22.328190  1644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:22.337482  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 71371.8us
I20260709 13:02:22.369210  2301 leader_election.cc:304] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:02:22.270088  1717 leader_election.cc:304] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:22.350991  1718 leader_election.cc:304] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
W20260709 13:02:22.370498  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 100555us
I20260709 13:02:22.370926  1718 leader_election.cc:304] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:22.372164  1718 leader_election.cc:304] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:22.275437  8895 raft_consensus.cc:515] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:22.343986  8623 raft_consensus.cc:2749] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 13:02:22.331115  8504 raft_consensus.cc:3060] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:02:22.338378  8786 raft_consensus.cc:2749] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:02:22.380026  2301 leader_election.cc:304] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:02:22.344668  8952 raft_consensus.cc:3060] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:02:22.380095  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:22.380587  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:22.300902  8934 raft_consensus.cc:515] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:22.368916  1569 leader_election.cc:304] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:02:22.396430  2301 leader_election.cc:304] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:02:22.400219  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:22.400226  1678 raft_consensus.cc:1275] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 13 index: 12. Preceding OpId from leader: term: 14 index: 13. (index mismatch)
I20260709 13:02:22.407074  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:22.275580  8727 consensus_queue.cc:260] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.13, Last appended by leader: 13, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:22.349004  8871 raft_consensus.cc:493] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:22.378424  8965 raft_consensus.cc:3060] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:02:22.344295  8935 raft_consensus.cc:493] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:22.349370  8733 raft_consensus.cc:2749] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 13:02:22.379823  1643 leader_election.cc:304] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:02:22.462144  1604 raft_consensus.cc:2393] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 22: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:02:22.381784  8976 raft_consensus.cc:2749] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:02:22.378834  8794 raft_consensus.cc:2804] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:02:22.391182  8786 raft_consensus.cc:2749] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 13:02:22.382227  8784 raft_consensus.cc:3060] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:02:22.409394  1745 raft_consensus.cc:1240] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 17. Current term is 18. Ops: []
I20260709 13:02:22.411180  8979 consensus_queue.cc:1059] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 14.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 13:02:22.375798  8932 raft_consensus.cc:2749] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:02:22.411895  8876 raft_consensus.cc:3060] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:02:22.414618  8817 raft_consensus.cc:2804] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Leader election won for term 19
I20260709 13:02:22.415040  8954 raft_consensus.cc:493] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:22.415449  2301 leader_election.cc:304] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:02:22.415681  8623 raft_consensus.cc:2749] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:02:22.416217  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 22 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:02:22.523437  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 108147us
I20260709 13:02:22.418233  8978 leader_election.cc:290] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:22.349637  8814 raft_consensus.cc:515] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:22.423014  8893 raft_consensus.cc:2749] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:02:22.425532  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:22.427887  1597 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40154 (request call id 2888) took 2037 ms. Trace:
W20260709 13:02:22.428539  1570 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:22.430459  1596 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40154 (request call id 2897) took 1973 ms (client timeout 2124 ms). Trace:
I20260709 13:02:22.432493  1602 raft_consensus.cc:1240] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 21. Current term is 22. Ops: []
I20260709 13:02:22.453423  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:02:22.467125  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 87430.4us
I20260709 13:02:22.375509  2329 raft_consensus.cc:2468] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 21.
I20260709 13:02:22.483071  2335 raft_consensus.cc:1240] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 14. Current term is 15. Ops: []
W20260709 13:02:22.484202  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:22.485244  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 20 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:22.377825  1599 raft_consensus.cc:2468] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 17.
I20260709 13:02:22.459453  8870 consensus_queue.cc:1048] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.024s
I20260709 13:02:22.485906  8772 raft_consensus.cc:3060] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:02:22.467873  8794 raft_consensus.cc:493] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:22.481083  8935 raft_consensus.cc:3060] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:02:22.516752  8979 consensus_queue.cc:1243] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer 83595ad0fd5940c0a2bfc7c61acd969a log is divergent from this leader: its last log entry 15.11 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 9
I20260709 13:02:22.631227  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:22.508010  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 19 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:22.512041  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 17 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:22.463742  8871 raft_consensus.cc:3060] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:02:22.649319  1598 raft_consensus.cc:1240] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 17. Current term is 18. Ops: []
I20260709 13:02:22.653540  1598 raft_consensus.cc:1275] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 22 index: 14. (index mismatch)
I20260709 13:02:22.541270  1680 raft_consensus.cc:2468] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 18.
I20260709 13:02:22.622303  1746 raft_consensus.cc:1240] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 19. Current term is 20. Ops: []
I20260709 13:02:22.560535  1603 raft_consensus.cc:2368] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for earlier term 21. Current term is 22.
I20260709 13:02:22.655055  1643 leader_election.cc:304] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:02:22.561128  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:22.622871  1597 rpcz_store.cc:276] 0709 13:02:20.390550 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:21.902310 (+1511760us) service_pool.cc:224] Handling call
0709 13:02:21.910898 (+  8588us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:21.911303 (+   405us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:21.911336 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:21.931762 (+ 20426us) log.cc:844] Serialized 26 byte log entry
0709 13:02:22.168057 (+236295us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:02:22.168103 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:22.168177 (+    74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:22.427563 (+259386us) raft_consensus.cc:1685] finished
0709 13:02:22.427602 (+    39us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:22.427798 (+   196us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":687,"raft.run_cpu_time_us":131,"raft.run_wall_time_us":131,"thread_start_us":173689,"threads_started":1,"wal-append.queue_time_us":395677}
I20260709 13:02:22.628773  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 21 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:22.655691  1598 raft_consensus.cc:1275] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 15 index: 9. Preceding OpId from leader: term: 16 index: 10. (index mismatch)
I20260709 13:02:22.556242  8921 raft_consensus.cc:2749] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 13:02:22.664891  1599 raft_consensus.cc:1240] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 14. Current term is 15. Ops: []
I20260709 13:02:22.633796  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:22.681380  1604 raft_consensus.cc:2468] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 16.
I20260709 13:02:22.656700  1745 raft_consensus.cc:3060] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Advancing to term 23
W20260709 13:02:22.656906  1596 rpcz_store.cc:269] 0709 13:02:20.457504 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:21.023285 (+565781us) service_pool.cc:224] Handling call
0709 13:02:22.430183 (+1406898us) spinlock_profiling.cc:243] Waited 1.11 s on lock 0x7b540006d6d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:02:22.430366 (+   183us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2547522816}
I20260709 13:02:22.690483  1596 raft_consensus.cc:1240] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 20. Current term is 21. Ops: [10.8-20.13]
I20260709 13:02:22.555883  8797 raft_consensus.cc:493] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:02:22.696950  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:22.699645  1602 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 2978) took 2208 ms. Trace:
I20260709 13:02:22.700062  1602 rpcz_store.cc:276] 0709 13:02:20.491478 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:21.932594 (+1441116us) service_pool.cc:224] Handling call
0709 13:02:22.432223 (+499629us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:22.657015 (+224792us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:22.699450 (+ 42435us) spinlock_profiling.cc:243] Waited 499 ms on lock 0x7b540006d6d0. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661618ad 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:02:22.699581 (+   131us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1148635520}
I20260709 13:02:22.656684  1643 leader_election.cc:304] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:22.668916  1598 raft_consensus.cc:1275] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 18 index: 9. Preceding OpId from leader: term: 20 index: 10. (index mismatch)
I20260709 13:02:22.522708  8727 raft_consensus.cc:3060] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:02:22.637638  8794 raft_consensus.cc:3060] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:02:22.674678  8972 raft_consensus.cc:2804] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:02:22.630847  8981 raft_consensus.cc:2764] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Leader pre-election decision vote started in defunct term 16: won
I20260709 13:02:22.739802  1745 raft_consensus.cc:2468] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 23.
I20260709 13:02:22.546876  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
W20260709 13:02:22.753585  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 357909us
W20260709 13:02:22.759447  1644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:22.770699  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:22.771343  1680 raft_consensus.cc:3060] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:02:22.776870  2331 raft_consensus.cc:2468] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 23.
W20260709 13:02:22.785036  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 46892us
I20260709 13:02:22.792083  1680 raft_consensus.cc:2468] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 21.
I20260709 13:02:22.526469  8817 raft_consensus.cc:697] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
W20260709 13:02:22.793465  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 421942us
I20260709 13:02:22.795653  1605 raft_consensus.cc:2393] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 19: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:02:22.795922  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:22.799304  1680 raft_consensus.cc:3060] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:02:22.800493  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:22.801324  1745 raft_consensus.cc:2468] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 23.
I20260709 13:02:22.526981  8954 raft_consensus.cc:515] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:22.812803  1680 raft_consensus.cc:2468] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 22.
I20260709 13:02:22.766741  8972 raft_consensus.cc:493] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:22.835860  1717 leader_election.cc:304] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:22.852984  1717 leader_election.cc:304] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:22.724244  8797 raft_consensus.cc:515] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:22.786954  8820 leader_election.cc:290] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 21 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:22.904088  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:22.914712  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 18 candidate_status { last_received { term: 16 index: 7 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:22.917186  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:02:22.896684  8504 consensus_meta.cc:287] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.507s	user 0.011s	sys 0.002s
W20260709 13:02:22.922984  1747 consensus_meta.cc:287] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.555s	user 0.012s	sys 0.000s
I20260709 13:02:23.109715   431 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:02:22.919528  1746 raft_consensus.cc:2393] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 18: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:02:23.128086  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:23.149878  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:02:23.134112  1718 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.761s	user 0.030s	sys 0.035s
I20260709 13:02:22.925431  8703 raft_consensus.cc:2749] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:02:22.843107  8733 consensus_queue.cc:1059] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 16.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.097s
I20260709 13:02:23.170058  8771 raft_consensus.cc:493] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:23.137684  1747 raft_consensus.cc:2410] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for term 21 because replica has last-logged OpId of term: 20 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 10.
I20260709 13:02:23.145826  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
W20260709 13:02:23.162808  8368 consensus_meta.cc:287] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.903s	user 0.014s	sys 0.005s
I20260709 13:02:23.146658  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 18 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:23.140491  1604 raft_consensus.cc:1275] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 16 index: 11. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
I20260709 13:02:23.146849  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:02:23.144977  2301 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.586s	user 0.036s	sys 0.070s
I20260709 13:02:22.898260  8764 leader_election.cc:290] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 18 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 13:02:23.135555  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:23.145408  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:23.133782  1602 raft_consensus.cc:1275] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 19 index: 15. Preceding OpId from leader: term: 21 index: 16. (index mismatch)
W20260709 13:02:23.293810  2301 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.735s	user 0.036s	sys 0.073s
W20260709 13:02:23.126417  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 469804us
I20260709 13:02:23.133782  1605 raft_consensus.cc:1275] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 16 index: 14. Preceding OpId from leader: term: 17 index: 15. (index mismatch)
I20260709 13:02:22.944003  8786 raft_consensus.cc:3055] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:02:23.158280  1569 leader_election.cc:304] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:02:23.019205  8797 leader_election.cc:290] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:22.965077  8814 leader_election.cc:290] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:23.349524  1599 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 2998) took 1104 ms. Trace:
I20260709 13:02:22.932641  8817 consensus_queue.cc:237] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.13, Last appended by leader: 13, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:22.931111  8504 raft_consensus.cc:515] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:02:23.352226  8784 consensus_meta.cc:287] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.857s	user 0.009s	sys 0.007s
I20260709 13:02:22.974608  8880 raft_consensus.cc:493] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:23.352898  8784 raft_consensus.cc:2749] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:02:22.933202  8893 consensus_queue.cc:1059] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 15.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.025s
I20260709 13:02:22.907501  8895 leader_election.cc:290] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 22 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:02:23.063169  8926 consensus_meta.cc:287] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.706s	user 0.017s	sys 0.000s
I20260709 13:02:22.953987  8934 leader_election.cc:290] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 15 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 13:02:23.059044  8952 consensus_meta.cc:287] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.636s	user 0.011s	sys 0.003s
I20260709 13:02:23.357760  8952 raft_consensus.cc:515] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:23.361718  8952 leader_election.cc:290] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 16 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:23.363332  1601 raft_consensus.cc:2410] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for term 21 because replica has last-logged OpId of term: 20 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 10.
I20260709 13:02:23.026777  8954 leader_election.cc:290] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:02:23.043967  8965 consensus_meta.cc:287] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.556s	user 0.012s	sys 0.000s
I20260709 13:02:23.365131  8965 raft_consensus.cc:2749] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20260709 13:02:22.945125  8968 consensus_meta.cc:287] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.859s	user 0.013s	sys 0.000s
I20260709 13:02:22.849745  8972 raft_consensus.cc:3060] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:02:23.369529  1596 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 3001) took 1120 ms. Trace:
W20260709 13:02:23.061007  8974 consensus_meta.cc:287] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.685s	user 0.013s	sys 0.001s
I20260709 13:02:23.371299  8974 raft_consensus.cc:515] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:23.370074  1596 rpcz_store.cc:276] 0709 13:02:22.249780 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:22.691548 (+441768us) service_pool.cc:224] Handling call
0709 13:02:22.691933 (+   385us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:22.693975 (+  2042us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 13:02:22.694011 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:22.788778 (+ 94767us) spinlock_profiling.cc:243] Waited 14.1 ms on lock 0x7b2c00000658. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4264a5ba11 00007f4264a6b372 00007f426619da01 00007f426616243a 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2
0709 13:02:22.789260 (+   482us) log.cc:844] Serialized 26 byte log entry
0709 13:02:23.279402 (+490142us) spinlock_profiling.cc:243] Waited 7.24 ms on lock 0x7b4800003cd8. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426b08baa1 00007f426b082b37 00007f426b0829ca 00007f426b17a26f 00007f426b179f37 00007f426b18e55c 00007f426b18e508 00007f426b18e488 00007f426b18e443 00007f426b18d6ec 00007f426a0b7a96
0709 13:02:23.310102 (+ 30700us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:02:23.336193 (+ 26091us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:23.364790 (+ 28597us) spinlock_profiling.cc:243] Waited 23.3 ms on lock 0x7b54002e20a0. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42660d7dbe 00007f42660e2930 00007f42660e28d8 00007f42660e2858 00007f42660e2813 00007f42660e1a9c 00007f426bfcd626 00007f426bfc7dd8 00007f426611e7cb 00007f42661212c0
0709 13:02:23.368885 (+  4095us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:23.368964 (+    79us) raft_consensus.cc:1685] finished
0709 13:02:23.368984 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:23.369274 (+   290us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":123,"raft.run_cpu_time_us":2389,"raft.run_wall_time_us":583194,"spinlock_wait_cycles":102572544,"thread_start_us":109058,"threads_started":1,"wal-append.queue_time_us":141966}
I20260709 13:02:22.848737  8976 consensus_queue.cc:1059] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.074s
I20260709 13:02:23.375320  8976 consensus_queue.cc:1243] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer 83595ad0fd5940c0a2bfc7c61acd969a log is divergent from this leader: its last log entry 17.13 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 11
I20260709 13:02:23.376292  8976 raft_consensus.cc:3055] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 13:02:23.376901  8976 raft_consensus.cc:740] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
W20260709 13:02:23.015197  8983 log.cc:927] Time spent T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.057s	user 0.000s	sys 0.000s
I20260709 13:02:23.377734  8786 raft_consensus.cc:740] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:02:23.378558  8730 raft_consensus.cc:493] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:22.833214  8978 consensus_queue.cc:1059] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.099s
I20260709 13:02:23.379638  8730 raft_consensus.cc:515] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:22.978642  8982 log.cc:927] Time spent T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.120s	user 0.001s	sys 0.000s
W20260709 13:02:23.213141  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 422049us
W20260709 13:02:23.215098  1718 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.842s	user 0.032s	sys 0.036s
I20260709 13:02:23.227669  9006 raft_consensus.cc:493] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:22.939010  8634 consensus_meta.cc:287] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.606s	user 0.011s	sys 0.001s
I20260709 13:02:23.269457  8771 raft_consensus.cc:515] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:23.274525  8870 raft_consensus.cc:493] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:23.276002  8932 raft_consensus.cc:2804] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Leader pre-election won for term 23
W20260709 13:02:23.276785  1644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:23.279608  1600 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 1770) took 3328 ms. Trace:
I20260709 13:02:23.283085  8368 raft_consensus.cc:515] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:23.286636  8981 raft_consensus.cc:493] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:23.288125  8979 raft_consensus.cc:2749] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:02:23.290285  9011 raft_consensus.cc:493] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:23.400585  9011 raft_consensus.cc:515] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:23.402717  1598 raft_consensus.cc:2468] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 18.
W20260709 13:02:23.302479  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 374009us
I20260709 13:02:23.291229  8733 raft_consensus.cc:3055] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 13:02:23.403620  8954 raft_consensus.cc:493] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:23.404176  8733 raft_consensus.cc:740] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:02:23.404697  1717 leader_election.cc:304] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:23.303215  8996 raft_consensus.cc:493] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:23.405963  9006 raft_consensus.cc:515] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:23.407248  8996 raft_consensus.cc:515] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:23.407783  1717 leader_election.cc:304] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:23.304044  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 19 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:23.409775  9012 raft_consensus.cc:2804] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:02:23.410290  9012 raft_consensus.cc:493] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:23.410727  9012 raft_consensus.cc:3060] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:02:23.411068  1675 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57092 (request call id 1595) took 1221 ms. Trace:
I20260709 13:02:23.302690  1750 raft_consensus.cc:2468] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 14.
I20260709 13:02:23.411466  1675 rpcz_store.cc:276] 0709 13:02:22.189369 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:22.190908 (+  1539us) service_pool.cc:224] Handling call
0709 13:02:22.191245 (+   337us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:22.191673 (+   428us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:22.191709 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:22.192239 (+   530us) log.cc:844] Serialized 26 byte log entry
0709 13:02:22.502892 (+310653us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:02:22.502969 (+    77us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:22.503076 (+   107us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:23.393550 (+890474us) raft_consensus.cc:1685] finished
0709 13:02:23.410527 (+ 16977us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:23.410819 (+   292us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":4611,"raft.run_cpu_time_us":97,"raft.run_wall_time_us":98,"thread_start_us":310168,"threads_started":1,"wal-append.queue_time_us":558509}
I20260709 13:02:23.412649  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:23.414028  8926 raft_consensus.cc:2804] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Leader election won for term 22
I20260709 13:02:23.323201  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
W20260709 13:02:23.333839  8772 consensus_meta.cc:287] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.702s	user 0.006s	sys 0.010s
I20260709 13:02:23.414757  8926 raft_consensus.cc:697] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:02:23.415169  1602 raft_consensus.cc:3060] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:02:23.416483  8926 consensus_queue.cc:237] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.15, Last appended by leader: 15, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:23.327575  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:23.354242  1643 leader_election.cc:304] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1
W20260709 13:02:23.420193  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 66211.4us
I20260709 13:02:23.421535  8870 raft_consensus.cc:515] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:23.376958  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 219308us
I20260709 13:02:23.422966  8748 consensus_queue.cc:1048] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 13:02:23.424762  8870 leader_election.cc:290] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:23.426749  1752 raft_consensus.cc:3060] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:02:23.427795  8764 raft_consensus.cc:2749] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:02:23.350173  1599 rpcz_store.cc:276] 0709 13:02:22.244777 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:22.680617 (+435840us) service_pool.cc:224] Handling call
0709 13:02:22.681043 (+   426us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:22.685765 (+  4722us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:22.685798 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:22.699967 (+ 14169us) log.cc:844] Serialized 26 byte log entry
0709 13:02:23.116373 (+416406us) raft_consensus.cc:1645] Marking committed up to 13
0709 13:02:23.117344 (+   971us) log.cc:844] Serialized 14 byte log entry
0709 13:02:23.265597 (+148253us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:23.300204 (+ 34607us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:23.311599 (+ 11395us) raft_consensus.cc:1685] finished
0709 13:02:23.311612 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:23.349229 (+ 37617us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":94384,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":49,"raft.queue_time_us":166408,"raft.run_cpu_time_us":385,"raft.run_wall_time_us":20937,"thread_start_us":563077,"threads_started":2,"wal-append.queue_time_us":353194}
W20260709 13:02:22.992202  8985 log.cc:927] Time spent T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.063s	user 0.000s	sys 0.001s
I20260709 13:02:23.436895  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:23.437659  1680 raft_consensus.cc:3060] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:02:23.437433  8972 raft_consensus.cc:515] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:23.383147  1679 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57092 (request call id 1597) took 1192 ms. Trace:
I20260709 13:02:23.420226  9016 raft_consensus.cc:2764] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Leader election decision vote started in defunct term 17: won
I20260709 13:02:23.446529  9006 leader_election.cc:290] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:23.446611  1679 rpcz_store.cc:276] 0709 13:02:22.190102 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:22.193221 (+  3119us) service_pool.cc:224] Handling call
0709 13:02:22.193602 (+   381us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:22.193958 (+   356us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:22.193990 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:22.194500 (+   510us) log.cc:844] Serialized 26 byte log entry
0709 13:02:22.678684 (+484184us) raft_consensus.cc:1645] Marking committed up to 15
0709 13:02:22.678861 (+   177us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:22.679230 (+   369us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:23.382567 (+703337us) raft_consensus.cc:1685] finished
0709 13:02:23.382615 (+    48us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:23.382905 (+   290us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":2522,"raft.run_cpu_time_us":55,"raft.run_wall_time_us":55,"thread_start_us":483759,"threads_started":1,"wal-append.queue_time_us":732796}
I20260709 13:02:22.922678  8989 raft_consensus.cc:493] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:02:23.449121  8996 leader_election.cc:290] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:23.449587  8989 raft_consensus.cc:515] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:23.451224  1751 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55848 (request call id 3265) took 1207 ms. Trace:
I20260709 13:02:23.452548  8989 leader_election.cc:290] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:23.353567  8880 raft_consensus.cc:515] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:23.239289  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 113459us
W20260709 13:02:23.388512  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 1.01683e+06us
W20260709 13:02:23.346066  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 788274us
I20260709 13:02:23.397372  8978 consensus_queue.cc:1243] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer 83595ad0fd5940c0a2bfc7c61acd969a log is divergent from this leader: its last log entry 19.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 9
I20260709 13:02:23.393784  8784 raft_consensus.cc:493] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:02:23.399644  1597 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 2992) took 1156 ms. Trace:
I20260709 13:02:23.399295  8981 raft_consensus.cc:515] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:23.402624  8968 raft_consensus.cc:2749] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:02:23.302671  1747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55848 (request call id 3274) took 1003 ms. Trace:
I20260709 13:02:23.404661  8954 raft_consensus.cc:515] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:23.304217  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:23.415092  8772 raft_consensus.cc:2749] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
W20260709 13:02:23.131145  1676 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.723s	user 0.013s	sys 0.051s
I20260709 13:02:23.421950  2298 leader_election.cc:304] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:23.379552  8786 consensus_queue.cc:260] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.10, Last appended by leader: 10, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:23.414211  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:23.422052  1643 leader_election.cc:304] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:23.393990  8932 raft_consensus.cc:493] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:23.392716  1600 rpcz_store.cc:276] 0709 13:02:19.951299 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:21.295853 (+1344554us) service_pool.cc:224] Handling call
0709 13:02:21.306108 (+ 10255us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:22.202588 (+896480us) spinlock_profiling.cc:243] Waited 12 ms on lock 0x7b18000073c0. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f4265d0a4bc 00007f426305848b 00007f426beb2ab9 00007f426307469b 00007f426307293f 00007f4263072455 00007f426615a060 00007f426615abfb 00007f426616564b
0709 13:02:22.202751 (+   163us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 13:02:22.202780 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:22.203361 (+   581us) log.cc:844] Serialized 26 byte log entry
0709 13:02:22.666973 (+463612us) raft_consensus.cc:1645] Marking committed up to 13
0709 13:02:22.667019 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:22.667093 (+    74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:23.118203 (+451110us) raft_consensus.cc:1685] finished
0709 13:02:23.118279 (+    76us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:23.278983 (+160704us) spinlock_profiling.cc:243] Waited 9.82 ms on lock 0x7b54000acb50. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661618ad 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:02:23.279278 (+   295us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":112290,"raft.run_cpu_time_us":203,"raft.run_wall_time_us":14874,"spinlock_wait_cycles":50238080,"thread_start_us":579067,"threads_started":2,"wal-append.queue_time_us":686985}
I20260709 13:02:23.440467  8972 leader_election.cc:290] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 17 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:23.442734  8974 leader_election.cc:290] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 18 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:23.443609  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:23.444185  8504 leader_election.cc:290] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 20 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:23.451642  1751 rpcz_store.cc:276] 0709 13:02:22.243780 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:22.243900 (+   120us) service_pool.cc:224] Handling call
0709 13:02:22.244226 (+   326us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:02:22.244637 (+   411us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:22.244664 (+    27us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:02:22.268216 (+ 23552us) log.cc:844] Serialized 52 byte log entry
0709 13:02:22.698090 (+429874us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:02:22.698135 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:22.698215 (+    80us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:23.450654 (+752439us) raft_consensus.cc:1685] finished
0709 13:02:23.450703 (+    49us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:23.450947 (+   244us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":8562,"raft.run_cpu_time_us":147,"raft.run_wall_time_us":6739,"thread_start_us":421157,"threads_started":1,"wal-append.queue_time_us":571230}
I20260709 13:02:23.396881  9010 raft_consensus.cc:493] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:23.461735  1747 rpcz_store.cc:276] 0709 13:02:22.299073 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:22.299237 (+   164us) service_pool.cc:224] Handling call
0709 13:02:23.302504 (+1003267us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:23.457181  8368 leader_election.cc:290] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 16 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 13:02:23.480501  1676 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.072s	user 0.018s	sys 0.051s
I20260709 13:02:23.457001  8771 leader_election.cc:290] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:23.472023  8932 raft_consensus.cc:3060] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:02:23.495662  1676 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52766 (request call id 3210) took 1089 ms. Trace:
I20260709 13:02:23.496130  1676 rpcz_store.cc:276] 0709 13:02:22.406199 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:22.406778 (+   579us) service_pool.cc:224] Handling call
0709 13:02:22.407132 (+   354us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:22.407382 (+   250us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:22.407407 (+    25us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:22.407964 (+   557us) log.cc:844] Serialized 26 byte log entry
0709 13:02:23.495154 (+1087190us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:02:23.495216 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:23.495332 (+   116us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:23.495399 (+    67us) raft_consensus.cc:1685] finished
0709 13:02:23.495414 (+    15us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:23.495595 (+   181us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":1077772,"threads_started":1,"wal-append.queue_time_us":630878}
I20260709 13:02:23.500331  1751 raft_consensus.cc:1275] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 16 index: 14. Preceding OpId from leader: term: 17 index: 15. (index mismatch)
I20260709 13:02:23.502641  9010 raft_consensus.cc:515] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:23.505291  9010 leader_election.cc:290] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:23.506145  8880 leader_election.cc:290] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:23.506779  1747 raft_consensus.cc:1275] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
W20260709 13:02:23.508167  8794 consensus_meta.cc:287] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.743s	user 0.019s	sys 0.008s
I20260709 13:02:23.508481  1747 raft_consensus.cc:1275] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 16 index: 11. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
I20260709 13:02:23.508658  8794 raft_consensus.cc:515] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:23.509920  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 21 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:23.492501  8996 raft_consensus.cc:2749] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:02:23.511377  1678 raft_consensus.cc:1240] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 15. Current term is 16. Ops: []
I20260709 13:02:23.512069  9011 leader_election.cc:290] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:23.515655  1749 raft_consensus.cc:2468] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 19.
I20260709 13:02:23.406102  8733 consensus_queue.cc:260] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.12, Last appended by leader: 12, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:23.483212  8952 raft_consensus.cc:493] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
W20260709 13:02:23.520135  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 169929us
I20260709 13:02:23.457718  1597 rpcz_store.cc:276] 0709 13:02:22.243325 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:22.657513 (+414188us) service_pool.cc:224] Handling call
0709 13:02:22.657813 (+   300us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:22.658033 (+   220us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 13:02:23.217467 (+559434us) log.cc:844] Serialized 14 byte log entry
0709 13:02:23.398907 (+181440us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:23.398991 (+    84us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:02:23.399105 (+   114us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:23.399250 (+   145us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:23.399421 (+   171us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":61385,"prepare.queue_time_us":190082,"prepare.run_cpu_time_us":106,"prepare.run_wall_time_us":5041,"raft.queue_time_us":2679,"raft.run_cpu_time_us":202,"raft.run_wall_time_us":203,"thread_start_us":596266,"threads_started":2}
W20260709 13:02:23.534322  8935 consensus_meta.cc:287] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.836s	user 0.019s	sys 0.000s
I20260709 13:02:23.535007  8935 raft_consensus.cc:515] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:23.538097  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 116425us
I20260709 13:02:23.538877  8748 raft_consensus.cc:3060] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:02:23.539451  8786 raft_consensus.cc:3060] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:02:23.541469  8976 consensus_queue.cc:260] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.13, Last appended by leader: 13, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:23.542537  1603 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 2991) took 1235 ms. Trace:
I20260709 13:02:23.542724  1746 raft_consensus.cc:2468] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 17.
I20260709 13:02:23.543542  8976 raft_consensus.cc:3060] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:02:23.544039  2299 leader_election.cc:304] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
W20260709 13:02:23.547305  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 125537us
I20260709 13:02:23.549908  1643 leader_election.cc:400] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 21 election: Vote denied by peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445) with higher term. Message: Invalid argument: T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for earlier term 21. Current term is 22.
I20260709 13:02:23.551610  8733 raft_consensus.cc:3060] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:02:23.551923  8972 consensus_queue.cc:1059] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 13:02:23.553136  8972 raft_consensus.cc:3055] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 LEADER]: Stepping down as leader of term 17
I20260709 13:02:23.553540  8972 raft_consensus.cc:740] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:02:23.554997  1646 leader_election.cc:304] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 13:02:23.555759  1750 raft_consensus.cc:2393] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 18: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:02:23.557125  8968 consensus_queue.cc:1059] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 13:02:23.557646  1718 leader_election.cc:304] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:02:23.559902  8952 raft_consensus.cc:515] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } }
I20260709 13:02:23.543110  1603 rpcz_store.cc:276] 0709 13:02:22.243084 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:22.655600 (+412516us) service_pool.cc:224] Handling call
0709 13:02:22.655942 (+   342us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:22.656209 (+   267us) raft_consensus.cc:1534] Early marking committed up to index 16
0709 13:02:22.719277 (+ 63068us) log.cc:844] Serialized 14 byte log entry
0709 13:02:23.294651 (+575374us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:23.339473 (+ 44822us) raft_consensus.cc:1645] Marking committed up to 16
0709 13:02:23.388821 (+ 49348us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:23.431727 (+ 42906us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:23.478422 (+ 46695us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2966,"prepare.queue_time_us":109711,"prepare.run_cpu_time_us":81,"prepare.run_wall_time_us":62,"raft.queue_time_us":157216,"raft.run_cpu_time_us":168,"raft.run_wall_time_us":167,"thread_start_us":536253,"threads_started":2,"wal-append.queue_time_us":336292}
I20260709 13:02:23.560297  8972 consensus_queue.cc:260] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 17.8, Last appended by leader: 8, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:23.518898  8784 raft_consensus.cc:515] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:23.565826  8784 leader_election.cc:290] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:23.565932  8730 leader_election.cc:290] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:23.568085  1601 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40166 (request call id 2988) took 1325 ms. Trace:
I20260709 13:02:23.568496  1601 rpcz_store.cc:276] 0709 13:02:22.242322 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:22.270243 (+ 27921us) service_pool.cc:224] Handling call
0709 13:02:23.567788 (+1297545us) spinlock_profiling.cc:243] Waited 1.02 s on lock 0x7b54002d9454. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:02:23.567989 (+   201us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2344668032}
I20260709 13:02:23.550320  1643 leader_election.cc:403] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 21 election: Cancelling election due to peer responding with higher term
W20260709 13:02:23.546334  8876 consensus_meta.cc:287] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 1.030s	user 0.017s	sys 0.003s
I20260709 13:02:23.623214  8876 raft_consensus.cc:2749] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20260709 13:02:23.589465  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 62416.9us
I20260709 13:02:23.624734  2299 leader_election.cc:304] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:02:23.626816  9002 raft_consensus.cc:2804] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 13:02:23.627425  9002 raft_consensus.cc:697] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 21 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:02:23.628448  9002 consensus_queue.cc:237] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 19.15, Last appended by leader: 15, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:23.589885  8954 leader_election.cc:290] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:23.590396  9001 raft_consensus.cc:2804] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 13:02:23.590739  8979 raft_consensus.cc:2749] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:02:23.591702  8880 raft_consensus.cc:2804] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:02:23.592782  8972 raft_consensus.cc:3060] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:02:23.593269  8921 raft_consensus.cc:3055] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 13:02:23.587598  8772 consensus_queue.cc:1059] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 19.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 13:02:23.585747  8623 consensus_queue.cc:1048] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 13:02:23.644040  9011 raft_consensus.cc:3055] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 LEADER]: Stepping down as leader of term 19
I20260709 13:02:23.644577  9011 raft_consensus.cc:740] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:02:23.584656  8934 consensus_queue.cc:1048] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 13:02:23.595913  1718 leader_election.cc:304] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
W20260709 13:02:23.646422  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 50713.5us
I20260709 13:02:23.597611  1646 leader_election.cc:304] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:02:23.599905  8935 leader_election.cc:290] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 19 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:23.583734  1644 leader_election.cc:304] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:23.570730  8952 leader_election.cc:290] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:23.647168  9001 raft_consensus.cc:697] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 18 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:02:23.656333  8772 raft_consensus.cc:2749] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
W20260709 13:02:23.603430  2301 connection.cc:256] RPC call timeout handler was delayed by -1.0365840759999969s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 1.5s for a response
I20260709 13:02:23.605095  8893 raft_consensus.cc:493] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:23.610567  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:23.610853  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:23.613754  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:23.612426  8368 raft_consensus.cc:493] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:23.631901  1680 raft_consensus.cc:2468] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 16.
I20260709 13:02:23.659252  1749 raft_consensus.cc:2468] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 18.
I20260709 13:02:23.530031  9010 raft_consensus.cc:493] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:23.643903  8871 consensus_meta.cc:287] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.859s	user 0.007s	sys 0.013s
W20260709 13:02:23.639662  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 90186.3us
I20260709 13:02:23.645744  9011 consensus_queue.cc:260] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 19.15, Last appended by leader: 15, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:23.676323  9001 consensus_queue.cc:237] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 16.14, Last appended by leader: 14, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:23.677062  1604 raft_consensus.cc:1240] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 17. Current term is 18. Ops: [17.12-17.12]
I20260709 13:02:23.677690  9011 raft_consensus.cc:3060] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:02:23.679942  8764 raft_consensus.cc:493] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:23.680513  8764 raft_consensus.cc:515] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:23.682338  8914 consensus_queue.cc:1048] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 13:02:23.684185  8914 consensus_queue.cc:1048] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 13:02:23.684438  8981 leader_election.cc:290] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:23.665416  8784 raft_consensus.cc:2749] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Leader election lost for term 21. Reason: Vote denied by peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445) with higher term. Message: Invalid argument: T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for earlier term 21. Current term is 22.
I20260709 13:02:23.685835  8784 consensus_queue.cc:1059] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 14.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 13:02:23.686686  8934 consensus_queue.cc:1059] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 15.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 13:02:23.686920  8784 consensus_queue.cc:1059] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true }, Status: INVALID_TERM, Last received: 8.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 13:02:23.687723  8934 raft_consensus.cc:3055] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 13:02:23.688098  8797 raft_consensus.cc:3055] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 LEADER]: Stepping down as leader of term 20
I20260709 13:02:23.688606  8797 raft_consensus.cc:740] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:02:23.664070  1569 leader_election.cc:304] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:02:23.691608  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:23.692140  1748 raft_consensus.cc:3055] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 13:02:23.692421  1745 raft_consensus.cc:2468] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 19.
I20260709 13:02:23.692610  1748 raft_consensus.cc:740] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:02:23.693529  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:23.654537  1718 leader_election.cc:304] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
W20260709 13:02:23.655308  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 57864.4us
I20260709 13:02:23.641269  8921 raft_consensus.cc:740] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:02:23.637665  8748 raft_consensus.cc:2749] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:02:23.698472  1745 raft_consensus.cc:2468] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 20.
I20260709 13:02:23.611824  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 21 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:23.636749  8733 raft_consensus.cc:493] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:23.662562  1752 raft_consensus.cc:2410] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for term 18 because replica has last-logged OpId of term: 17 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 11.
I20260709 13:02:23.660459  8504 raft_consensus.cc:2749] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:02:23.636821  8880 raft_consensus.cc:493] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:23.637735  1602 raft_consensus.cc:2468] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 18.
I20260709 13:02:23.666954  9014 raft_consensus.cc:2749] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 13:02:23.666857  8893 raft_consensus.cc:515] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:23.668359  8954 raft_consensus.cc:493] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:02:23.668957  8952 raft_consensus.cc:493] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:23.669401  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:23.724720  1746 raft_consensus.cc:3060] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Advancing to term 21
I20260709 13:02:23.669880  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:23.671314  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:23.672070  8368 raft_consensus.cc:515] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:23.719473  1747 raft_consensus.cc:2468] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 20.
I20260709 13:02:23.673738  9006 raft_consensus.cc:493] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:02:23.674916  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:23.675532  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:23.675761  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:23.687175  8634 raft_consensus.cc:3055] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 LEADER]: Stepping down as leader of term 14
W20260709 13:02:23.652055  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 68463us
W20260709 13:02:23.700320  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 61348.5us
I20260709 13:02:23.663128  8817 consensus_queue.cc:1048] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.032s
I20260709 13:02:23.668426  1751 raft_consensus.cc:2468] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 17.
I20260709 13:02:23.718314  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:23.718132  8921 consensus_queue.cc:260] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.12, Last appended by leader: 12, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:23.718825  1646 leader_election.cc:304] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:02:23.660518  8794 leader_election.cc:290] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 16 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:23.718143  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:23.720403  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:23.723851  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
W20260709 13:02:23.741081  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:23.752046  9002 raft_consensus.cc:493] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:23.719236  8870 raft_consensus.cc:2804] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:02:23.672857  9010 raft_consensus.cc:515] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:23.688127  8934 raft_consensus.cc:740] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
W20260709 13:02:23.764021  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 47036.2us
I20260709 13:02:23.721536  1748 consensus_queue.cc:260] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 18.10, Last appended by leader: 10, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:23.760911  8772 consensus_queue.cc:1048] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.004s
I20260709 13:02:23.765302  8921 raft_consensus.cc:3060] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:02:23.798707  8895 raft_consensus.cc:493] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:23.767913  1643 leader_election.cc:304] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:02:23.763687  8634 raft_consensus.cc:740] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:02:23.770105  8703 consensus_queue.cc:1048] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 13:02:23.771658  9006 raft_consensus.cc:515] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:23.772437  1600 raft_consensus.cc:2393] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 16: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:02:23.772817  8954 raft_consensus.cc:515] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:23.773543  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:23.774268  1599 raft_consensus.cc:2468] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 16.
I20260709 13:02:23.774415  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:23.775030  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 17 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:23.714838  8880 raft_consensus.cc:3060] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:02:23.669435  8730 raft_consensus.cc:493] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:23.712410  8871 raft_consensus.cc:515] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:23.821540  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 102937us
I20260709 13:02:23.785849  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:23.788594  1571 leader_election.cc:304] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 13:02:23.789717  1644 leader_election.cc:304] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:02:23.790027  1748 raft_consensus.cc:3060] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:02:23.790565  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:23.790676  8771 consensus_queue.cc:1059] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 13:02:23.791019  1753 raft_consensus.cc:1275] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 14 index: 10. Preceding OpId from leader: term: 15 index: 11. (index mismatch)
I20260709 13:02:23.825387  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:23.768550  8914 consensus_queue.cc:1059] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 21.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 13:02:23.801189  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:23.797187  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:23.719595  8981 raft_consensus.cc:493] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:23.778419  8870 raft_consensus.cc:493] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:23.809430  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:23.809868  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:23.809993  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:23.783612  8733 raft_consensus.cc:515] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:23.846673  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 192493us
I20260709 13:02:23.848718  1718 leader_election.cc:304] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:02:23.850656  8932 raft_consensus.cc:515] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:23.851120  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 18 candidate_status { last_received { term: 16 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:23.852456  1599 raft_consensus.cc:2393] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 18: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:02:23.853077  8932 leader_election.cc:290] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 23 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:23.854867  8893 leader_election.cc:290] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:23.855185  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:23.855623  8733 leader_election.cc:290] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:23.856163  1749 raft_consensus.cc:2468] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 18.
I20260709 13:02:23.857085  8871 leader_election.cc:290] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 19 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:23.857486  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 19 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:23.858810  1596 raft_consensus.cc:2468] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 23.
W20260709 13:02:23.862967  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 95265.7us
I20260709 13:02:23.864085  8817 raft_consensus.cc:3055] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:02:23.864598  8817 raft_consensus.cc:740] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
W20260709 13:02:23.867555  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 79124.4us
I20260709 13:02:23.799070  9016 raft_consensus.cc:2749] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:02:23.814895  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:23.826990  1569 leader_election.cc:304] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:02:23.689762  8797 consensus_queue.cc:260] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 20.13, Last appended by leader: 13, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:23.757241  8952 raft_consensus.cc:515] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:23.826678  1680 raft_consensus.cc:2393] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 16: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:02:23.719889  8978 raft_consensus.cc:2749] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:02:23.801321  8764 leader_election.cc:290] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:23.875945  1751 raft_consensus.cc:2410] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for term 19 because replica has last-logged OpId of term: 18 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 13.
I20260709 13:02:23.830224  8730 raft_consensus.cc:515] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:23.822386  9002 raft_consensus.cc:515] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:23.834959  2332 raft_consensus.cc:2393] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 16: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:02:23.834767  8504 raft_consensus.cc:2749] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 13:02:23.835690  8772 raft_consensus.cc:2749] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:02:23.836182  8914 raft_consensus.cc:3055] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 LEADER]: Stepping down as leader of term 21
I20260709 13:02:23.837982  8634 consensus_queue.cc:260] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 14.13, Last appended by leader: 13, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:23.838603  9015 raft_consensus.cc:493] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:23.839278  8895 raft_consensus.cc:515] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:23.920411  8895 leader_election.cc:290] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:23.832471  8870 raft_consensus.cc:3060] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:02:23.924880  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:23.925665  1603 raft_consensus.cc:2468] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 17.
W20260709 13:02:23.848793  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 59237.8us
I20260709 13:02:23.927863  1644 leader_election.cc:304] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
W20260709 13:02:23.928776  9018 log.cc:927] Time spent T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.366s	user 0.001s	sys 0.000s
I20260709 13:02:23.857089  8934 consensus_queue.cc:260] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:23.813936  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:23.929744  8934 raft_consensus.cc:3060] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:02:23.817015  1750 raft_consensus.cc:2468] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 17.
I20260709 13:02:23.858354  1749 raft_consensus.cc:2468] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 18.
I20260709 13:02:23.936781  1604 raft_consensus.cc:2468] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 14.
I20260709 13:02:23.811153  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:23.865756  8817 consensus_queue.cc:260] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 14.13, Last appended by leader: 13, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:23.889684  1678 raft_consensus.cc:1240] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 15. Current term is 16. Ops: []
I20260709 13:02:23.891155  1597 raft_consensus.cc:1275] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 16 index: 10. (index mismatch)
I20260709 13:02:23.891294  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:23.884423  8981 raft_consensus.cc:515] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:23.892259  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 19 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:23.942467  1673 raft_consensus.cc:2410] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 for term 21 because replica has last-logged OpId of term: 19 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 12.
I20260709 13:02:23.893361  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:02:23.894394  8727 consensus_meta.cc:287] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 1.169s	user 0.014s	sys 0.007s
I20260709 13:02:23.895136  8634 raft_consensus.cc:3060] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:02:23.896100  8954 leader_election.cc:290] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:23.901072  9012 raft_consensus.cc:515] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:23.886202  8914 raft_consensus.cc:740] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:02:23.906857  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:23.981385  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:23.907788  1571 leader_election.cc:304] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:02:23.908218  8730 leader_election.cc:290] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:23.908782  1598 raft_consensus.cc:3060] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:02:23.912178  8952 leader_election.cc:290] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:23.871662  8771 raft_consensus.cc:2804] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 13:02:23.914938  9035 raft_consensus.cc:2804] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Leader election won for term 16
I20260709 13:02:23.916307  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:23.884567  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:23.917133  9015 raft_consensus.cc:515] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:23.917536  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:02:23.918051  2298 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:23.842913  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:23.893059  8764 consensus_queue.cc:1048] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 13:02:24.019471  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
W20260709 13:02:23.968474  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 69865.9us
W20260709 13:02:23.948127  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 62984.5us
I20260709 13:02:23.990741  2332 raft_consensus.cc:2468] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 19.
I20260709 13:02:24.023451  9035 raft_consensus.cc:697] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:02:24.027079  8727 consensus_queue.cc:1048] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 13:02:24.027616  9035 consensus_queue.cc:237] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 14.13, Last appended by leader: 13, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:24.028805  8952 raft_consensus.cc:2749] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 13:02:24.031594  8914 consensus_queue.cc:260] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 21.11, Last appended by leader: 11, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:23.980429  9012 leader_election.cc:290] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 17 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:24.033349  8914 raft_consensus.cc:3060] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:02:24.033782  1646 leader_election.cc:304] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:02:24.036468  8772 consensus_queue.cc:1048] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 13:02:23.976018  8981 leader_election.cc:290] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:02:23.982630  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 54933.4us
W20260709 13:02:24.044020  1644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:23.982756  2333 raft_consensus.cc:2468] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 17.
I20260709 13:02:23.990610  1748 raft_consensus.cc:1275] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 18 index: 10. Preceding OpId from leader: term: 19 index: 10. (term mismatch)
I20260709 13:02:24.044874  1748 pending_rounds.cc:85] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701: Aborting all ops after (but not including) 9
I20260709 13:02:23.994411  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:23.995595  9011 raft_consensus.cc:2804] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Leader election won for term 23
W20260709 13:02:23.996068  2300 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:23.996191  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:23.998932  8817 raft_consensus.cc:3060] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:02:24.064944  9006 leader_election.cc:290] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:02:23.985652  2301 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.107.195:40693 timed out after 1.738s (SENT) [suppressed 32 similar messages]
W20260709 13:02:24.011324  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 103706us
I20260709 13:02:24.012795  9002 leader_election.cc:290] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:24.017447  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:24.018617  1671 raft_consensus.cc:2393] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 16: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:02:23.972559  1746 raft_consensus.cc:2468] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 21.
I20260709 13:02:24.045789  1750 raft_consensus.cc:3060] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:02:23.986348  8771 raft_consensus.cc:493] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 23 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
W20260709 13:02:24.048760  1644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:24.083101  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:24.085400  1745 raft_consensus.cc:2410] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 for term 21 because replica has last-logged OpId of term: 19 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 12.
I20260709 13:02:24.061872  8797 raft_consensus.cc:3060] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:02:24.062655  2337 raft_consensus.cc:2468] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 15.
I20260709 13:02:24.094272  8771 raft_consensus.cc:3060] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 23 FOLLOWER]: Advancing to term 24
I20260709 13:02:24.069564  1748 pending_rounds.cc:107] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701: Aborting uncommitted NO_OP operation due to leader change: 18.10
I20260709 13:02:24.094878  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:24.118957  1748 raft_consensus.cc:2889] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:02:24.095649  2298 leader_election.cc:304] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:02:24.119189  2333 raft_consensus.cc:2468] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 15.
I20260709 13:02:24.097119  1646 leader_election.cc:304] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:02:24.100224  8870 raft_consensus.cc:515] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:24.104076  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:24.107574  9011 raft_consensus.cc:697] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:02:24.109329  9015 leader_election.cc:290] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:24.111159  9012 raft_consensus.cc:493] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:24.123953  2330 raft_consensus.cc:2468] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 21.
I20260709 13:02:24.111477  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 18 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:24.084795  8996 consensus_queue.cc:1048] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.027s
I20260709 13:02:24.084035  2332 raft_consensus.cc:2468] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 20.
I20260709 13:02:24.133667  2331 raft_consensus.cc:2410] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for term 18 because replica has last-logged OpId of term: 17 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 11.
W20260709 13:02:24.127079  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 300388us
I20260709 13:02:24.135721  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:24.086395  8932 raft_consensus.cc:2749] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:02:24.136669  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:24.140177  1748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55840 (request call id 3097) took 1240 ms. Trace:
W20260709 13:02:24.126122  8926 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.580s	user 0.009s	sys 0.009s
I20260709 13:02:24.145869  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 19 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:24.142812  1748 rpcz_store.cc:276] 0709 13:02:22.900309 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:22.902580 (+  2271us) service_pool.cc:224] Handling call
0709 13:02:22.903030 (+   450us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:23.691776 (+788746us) spinlock_profiling.cc:243] Waited 558 ms on lock 0x7b5400252720. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42660d9dc0 00007f42661641c0 00007f4266165231 00007f4266161d8f 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c
0709 13:02:24.125620 (+433844us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:24.140098 (+ 14478us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1284170112}
W20260709 13:02:24.150832  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 55355.9us
I20260709 13:02:24.117712  8772 raft_consensus.cc:2749] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 13:02:24.157879  2337 raft_consensus.cc:2468] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 20.
I20260709 13:02:24.158631  1571 leader_election.cc:304] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:02:24.159425  2337 raft_consensus.cc:1240] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 14. Current term is 15. Ops: []
I20260709 13:02:24.160558  1571 leader_election.cc:304] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:24.161597  9008 raft_consensus.cc:2749] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 13:02:24.162565  9008 raft_consensus.cc:2804] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:02:24.163044  9008 raft_consensus.cc:493] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:24.163439  9008 raft_consensus.cc:3060] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:02:24.163599  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 22 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:24.166528  9012 raft_consensus.cc:515] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:24.167147  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 20 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:24.167927  1671 raft_consensus.cc:2468] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 19.
W20260709 13:02:24.168341  1570 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:24.168892  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:24.169103  9012 leader_election.cc:290] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:24.169800  1679 raft_consensus.cc:2393] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 20: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
W20260709 13:02:24.171070  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 148073us
I20260709 13:02:24.172904  1718 leader_election.cc:304] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:24.175494  8368 leader_election.cc:290] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:24.176847  9010 leader_election.cc:290] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:24.176530  9011 consensus_queue.cc:237] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.16, Last appended by leader: 16, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:24.136210  8965 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 19 current term: 19 is higher.
I20260709 13:02:24.182657  8968 raft_consensus.cc:3060] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:02:24.151494  9016 raft_consensus.cc:2749] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:02:24.184562  9016 raft_consensus.cc:493] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:24.185096  1754 raft_consensus.cc:1240] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 17. Current term is 18. Ops: []
I20260709 13:02:24.176563  1752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55840 (request call id 3103) took 1033 ms. Trace:
I20260709 13:02:24.155900  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:24.162359  1571 leader_election.cc:304] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:24.215080  1598 raft_consensus.cc:2468] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 18.
I20260709 13:02:24.217841  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:02:24.149801  8926 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.604s	user 0.010s	sys 0.009s
I20260709 13:02:24.186972  1752 rpcz_store.cc:276] 0709 13:02:23.143632 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:23.146024 (+  2392us) service_pool.cc:224] Handling call
0709 13:02:24.176354 (+1030330us) spinlock_profiling.cc:243] Waited 126 ms on lock 0x7b54001d56d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:02:24.176493 (+   139us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":289772288}
I20260709 13:02:24.227458  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:24.163587  8727 raft_consensus.cc:2804] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:02:24.225584  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:24.226677  2330 raft_consensus.cc:2393] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 16: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:02:24.226565  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:02:24.224723  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 62474.7us
I20260709 13:02:24.233342  1671 raft_consensus.cc:2468] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 17.
I20260709 13:02:24.239305  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:24.241629  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:02:24.244668  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 147755us
I20260709 13:02:24.246964  2331 raft_consensus.cc:1240] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 18. Current term is 19. Ops: [18.15-18.15]
W20260709 13:02:24.246954  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 266018us
I20260709 13:02:24.245787  1673 raft_consensus.cc:2468] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 17.
W20260709 13:02:24.233556  8972 consensus_meta.cc:287] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.586s	user 0.014s	sys 0.000s
I20260709 13:02:24.226125  9010 raft_consensus.cc:493] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:24.256728  1718 leader_election.cc:304] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:02:24.239069  8727 raft_consensus.cc:493] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:02:24.268256  2301 leader_election.cc:304] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:02:24.264389  1676 raft_consensus.cc:2468] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 21.
W20260709 13:02:24.256426  9037 log.cc:927] Time spent T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.098s	user 0.001s	sys 0.000s
W20260709 13:02:24.332161  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 74455.1us
I20260709 13:02:24.318596  8814 raft_consensus.cc:2804] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:02:24.270360  9012 raft_consensus.cc:2749] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:02:24.185104  9016 raft_consensus.cc:515] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:24.360659  9016 leader_election.cc:290] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:24.363749  8870 leader_election.cc:290] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 18 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:24.270809  8727 raft_consensus.cc:3060] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:02:24.365075  2334 raft_consensus.cc:2468] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 16.
W20260709 13:02:24.370054  8880 consensus_meta.cc:287] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.545s	user 0.008s	sys 0.002s
I20260709 13:02:24.351409  8814 raft_consensus.cc:493] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:24.373344  1646 leader_election.cc:304] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:02:24.375366  1569 leader_election.cc:304] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:02:24.366257  9010 raft_consensus.cc:515] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:24.386425  1601 raft_consensus.cc:2410] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 for term 15 because replica has last-logged OpId of term: 14 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 12.
W20260709 13:02:24.406298  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 57512.9us
I20260709 13:02:24.387857  8968 raft_consensus.cc:2749] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:02:24.389405  8771 raft_consensus.cc:515] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 24 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:24.395131  2328 raft_consensus.cc:2468] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 22.
I20260709 13:02:24.397241  1750 raft_consensus.cc:2468] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 19.
I20260709 13:02:24.408533  2298 leader_election.cc:304] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d
W20260709 13:02:24.421301  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 168463us
I20260709 13:02:24.406366  8814 raft_consensus.cc:3060] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Advancing to term 19
W20260709 13:02:24.427466  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 52221.7us
W20260709 13:02:24.431824  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 58610.8us
I20260709 13:02:24.433387  1605 raft_consensus.cc:1240] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 24 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 23. Current term is 24. Ops: []
I20260709 13:02:24.427567  8870 raft_consensus.cc:493] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:24.439607  2298 leader_election.cc:304] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:24.441624  1646 leader_election.cc:304] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 13:02:24.428992  9006 raft_consensus.cc:2749] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:02:24.451143  1569 leader_election.cc:304] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:02:24.406172  8880 raft_consensus.cc:515] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:24.451964  8771 leader_election.cc:290] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 24 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:24.453213  8870 raft_consensus.cc:515] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:24.453941  8880 leader_election.cc:290] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:24.455649  8870 leader_election.cc:290] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:24.438476  8994 raft_consensus.cc:493] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:02:24.456529  8994 raft_consensus.cc:515] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:24.459693  9010 leader_election.cc:290] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:24.460168  2298 leader_election.cc:304] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:24.432581  8797 raft_consensus.cc:2749] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 13:02:24.438704  8368 raft_consensus.cc:2749] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
W20260709 13:02:24.462733  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 204299us
I20260709 13:02:24.448907  1718 leader_election.cc:304] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 13:02:24.461875  1644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:24.466538  1571 leader_election.cc:304] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:02:24.462188  1675 raft_consensus.cc:1240] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 18. Current term is 19. Ops: []
I20260709 13:02:24.451815  8871 raft_consensus.cc:2804] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:02:24.477620  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:24.464457  8634 raft_consensus.cc:2804] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:02:24.489358  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:24.469827  8871 raft_consensus.cc:493] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:02:24.487253  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 19 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
W20260709 13:02:24.492621  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 224557us
I20260709 13:02:24.470818  9001 raft_consensus.cc:2749] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
W20260709 13:02:24.510462  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 44048.1us
I20260709 13:02:24.504410  8871 raft_consensus.cc:3060] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:02:24.511590  1571 leader_election.cc:304] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:24.511814  1679 raft_consensus.cc:1275] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 19 index: 15. Preceding OpId from leader: term: 21 index: 16. (index mismatch)
I20260709 13:02:24.512603  1571 leader_election.cc:304] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:02:24.515033  9015 raft_consensus.cc:2804] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:02:24.515661  9015 raft_consensus.cc:493] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:24.515045  8817 raft_consensus.cc:2749] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:02:24.516098  9015 raft_consensus.cc:3060] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:02:24.519347  1604 raft_consensus.cc:1275] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 16 index: 14. Preceding OpId from leader: term: 18 index: 15. (index mismatch)
W20260709 13:02:24.520707  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 79336.9us
I20260709 13:02:24.521723  1718 leader_election.cc:304] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:24.523321  1571 leader_election.cc:304] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:02:24.478519  8994 leader_election.cc:290] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:24.524116  1718 leader_election.cc:304] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:02:24.524621  8817 raft_consensus.cc:2749] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:02:24.524838  8932 raft_consensus.cc:2749] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:02:24.525004  8764 raft_consensus.cc:493] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:24.505079  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 20 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:24.526054  1571 leader_election.cc:304] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:24.525744  8764 raft_consensus.cc:515] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:24.526909  8817 raft_consensus.cc:2804] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:02:24.527504  8817 raft_consensus.cc:493] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:24.507670  1569 leader_election.cc:304] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:02:24.527860  8817 raft_consensus.cc:3060] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:02:24.528788  8764 leader_election.cc:290] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:24.525741  8932 raft_consensus.cc:2749] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:02:24.529053  1569 leader_election.cc:304] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:02:24.530861  1571 leader_election.cc:304] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:24.505405  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:24.502553  8634 raft_consensus.cc:493] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:24.470443  9038 raft_consensus.cc:2804] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:02:24.475618  8921 raft_consensus.cc:2749] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 13:02:24.532397  1676 raft_consensus.cc:2468] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 20.
I20260709 13:02:24.487416  8771 raft_consensus.cc:2804] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:02:24.477963  8968 raft_consensus.cc:2749] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:02:24.530373  1569 leader_election.cc:304] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
W20260709 13:02:24.540263  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 98805.8us
I20260709 13:02:24.548367  1646 leader_election.cc:304] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:24.540138  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:24.531901  1571 leader_election.cc:304] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:24.541630  2298 leader_election.cc:304] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:24.541692  1600 raft_consensus.cc:2468] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 21.
I20260709 13:02:24.504060  9036 consensus_queue.cc:1059] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 23.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.039s
I20260709 13:02:24.535676  9010 raft_consensus.cc:2764] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Leader pre-election decision vote started in defunct term 17: won
I20260709 13:02:24.535637  8634 raft_consensus.cc:3060] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:02:24.570582  1754 raft_consensus.cc:1240] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 16. Current term is 17. Ops: [16.10-16.10]
I20260709 13:02:24.536902  9038 raft_consensus.cc:493] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:02:24.547124  9041 log.cc:927] Time spent T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.097s	user 0.001s	sys 0.001s
I20260709 13:02:24.543241  8876 consensus_queue.cc:1048] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.030s
I20260709 13:02:24.573673  1745 raft_consensus.cc:1240] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 22. Current term is 23. Ops: [20.13-22.14]
I20260709 13:02:24.506245  9008 raft_consensus.cc:515] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:24.576586  8871 raft_consensus.cc:515] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:24.543675  8771 raft_consensus.cc:493] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:24.582410  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:24.595556  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:24.586562  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:24.554774  8914 raft_consensus.cc:2804] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:02:24.560451  8921 consensus_queue.cc:1048] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.021s
I20260709 13:02:24.599323  1748 raft_consensus.cc:1240] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 18. Current term is 19. Ops: []
I20260709 13:02:24.558355  9012 raft_consensus.cc:493] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:24.609490  1601 raft_consensus.cc:1240] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 14. Current term is 15. Ops: []
I20260709 13:02:24.579150  8870 raft_consensus.cc:2804] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:02:24.596271  1751 raft_consensus.cc:2468] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 20.
I20260709 13:02:24.548079  8994 consensus_queue.cc:1048] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.040s
I20260709 13:02:24.608882  9036 raft_consensus.cc:2749] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 13:02:24.611464  9015 raft_consensus.cc:515] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:24.553952  9001 consensus_queue.cc:1048] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.048s
I20260709 13:02:24.581928  8981 raft_consensus.cc:2804] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 13:02:24.582147  9038 raft_consensus.cc:3060] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:02:24.628361  9039 raft_consensus.cc:2804] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:02:24.598829  8771 raft_consensus.cc:3060] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:02:24.600252  8914 raft_consensus.cc:493] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:24.620936  9012 raft_consensus.cc:515] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:24.612670  8880 consensus_queue.cc:1059] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 15.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.052s
I20260709 13:02:24.630069  1495 catalog_manager.cc:5697] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 reported cstate change: term changed from 14 to 15, leader changed from 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) to 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194). New cstate: current_term: 15 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } }
I20260709 13:02:24.622363  9010 raft_consensus.cc:493] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:24.631983  8968 raft_consensus.cc:493] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:24.634474  9008 leader_election.cc:290] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 15 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:24.612495  9016 consensus_queue.cc:1059] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 19.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.054s
I20260709 13:02:24.632396  8871 leader_election.cc:290] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 16 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 13:02:24.667775  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 107249us
I20260709 13:02:24.671674  8727 raft_consensus.cc:515] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:24.630200  9015 leader_election.cc:290] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 18 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:02:24.682232  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 137608us
W20260709 13:02:24.685834  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 161903us
W20260709 13:02:24.715763  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 155631us
W20260709 13:02:24.691745  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 139438us
I20260709 13:02:24.614108  8870 raft_consensus.cc:493] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:24.638324  8981 raft_consensus.cc:493] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:24.588281  8989 consensus_queue.cc:1048] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.039s
I20260709 13:02:24.649130  9039 raft_consensus.cc:493] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Starting leader election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:02:24.688692  8914 raft_consensus.cc:3060] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:02:24.729513  2299 leader_election.cc:304] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:02:24.739039  9054 raft_consensus.cc:2804] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:02:24.739596  9054 raft_consensus.cc:493] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Starting leader election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:02:24.739995  9054 raft_consensus.cc:3060] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:02:24.744509  8952 raft_consensus.cc:2804] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Leader pre-election won for term 18
W20260709 13:02:24.745261  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 203862us
I20260709 13:02:24.748415  2298 leader_election.cc:304] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:02:24.752923  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:24.724722  8870 raft_consensus.cc:3060] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:02:24.750938  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:24.724798  8981 raft_consensus.cc:3060] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:02:24.761833  1675 raft_consensus.cc:2393] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 16: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:02:24.713913  8968 raft_consensus.cc:515] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:24.731829  9039 raft_consensus.cc:3060] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:02:24.758265  9052 raft_consensus.cc:2804] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Leader pre-election won for term 22
W20260709 13:02:24.781560  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 45870.3us
W20260709 13:02:24.777055  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 59108.2us
I20260709 13:02:24.786026  2301 leader_election.cc:304] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
W20260709 13:02:24.788707  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 66103.2us
I20260709 13:02:24.749620  8871 consensus_queue.cc:1059] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 18.13, Next index: 14, Last known committed idx: 12, Time since last communication: 0.023s
I20260709 13:02:24.757941  9010 raft_consensus.cc:515] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:24.766325  9015 consensus_queue.cc:1048] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.070s
W20260709 13:02:24.858160  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 72324.8us
I20260709 13:02:24.844731  8921 raft_consensus.cc:493] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:24.849730  9066 consensus_queue.cc:1059] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 14.11, Next index: 12, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 13:02:24.860404  1718 leader_election.cc:304] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:24.781028  9052 raft_consensus.cc:493] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
W20260709 13:02:24.871094  9046 log.cc:927] Time spent T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.234s	user 0.000s	sys 0.000s
W20260709 13:02:24.872575  9049 log.cc:927] Time spent T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.170s	user 0.001s	sys 0.000s
I20260709 13:02:24.873106  9034 raft_consensus.cc:3055] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 13:02:24.873606  9034 raft_consensus.cc:740] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:02:24.864128  8871 raft_consensus.cc:2749] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260709 13:02:24.916443  1644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:24.876622  9052 raft_consensus.cc:3060] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Advancing to term 22
W20260709 13:02:24.904131  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 140688us
I20260709 13:02:24.888818  8764 raft_consensus.cc:2749] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20260709 13:02:24.943145  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 82876.9us
I20260709 13:02:24.889966  8921 raft_consensus.cc:515] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:24.939357  9012 leader_election.cc:290] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 13:02:24.963250  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 212632us
W20260709 13:02:24.965758  2298 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:24.891482  9034 consensus_queue.cc:260] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.13, Last appended by leader: 13, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:24.971903  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:24.970062  8727 leader_election.cc:290] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 18 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:24.976239  9034 raft_consensus.cc:3060] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:02:24.979316  9038 raft_consensus.cc:515] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:24.983289  9038 leader_election.cc:290] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 20 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:24.984769  8968 leader_election.cc:290] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:24.986100  8772 raft_consensus.cc:493] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
W20260709 13:02:24.986595  9002 consensus_queue.cc:1175] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "af520b67945d443b901bbeca37cc5701" responder_term: 18 status { last_received { term: 18 index: 13 } last_committed_idx: 12 last_received_current_leader { term: 18 index: 13 } }
I20260709 13:02:24.986780  8772 raft_consensus.cc:515] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:24.989763  8772 leader_election.cc:290] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:24.990006  1601 raft_consensus.cc:2468] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 16.
I20260709 13:02:24.997318  9036 raft_consensus.cc:3055] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:02:24.997742  9036 raft_consensus.cc:740] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:02:25.000391  2298 leader_election.cc:304] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:02:25.007558  8871 raft_consensus.cc:2804] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 13:02:25.009559  8871 raft_consensus.cc:697] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 16 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
W20260709 13:02:25.010006  9062 log.cc:927] Time spent T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.051s	user 0.000s	sys 0.000s
I20260709 13:02:25.010667  8871 consensus_queue.cc:237] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:25.023567  9063 log.cc:927] Time spent T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.062s	user 0.000s	sys 0.000s
W20260709 13:02:25.026173  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 263214us
I20260709 13:02:25.026367  9058 consensus_queue.cc:1048] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.001s
W20260709 13:02:25.040130  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 146114us
W20260709 13:02:25.045572  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 74900.3us
I20260709 13:02:25.049089  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:24.989976  8634 raft_consensus.cc:515] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:25.059350  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 20 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:25.059928  1603 raft_consensus.cc:2393] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 18: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:02:25.060079  1678 raft_consensus.cc:3060] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:02:25.006428  8817 raft_consensus.cc:515] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:25.062912  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 93088.6us
W20260709 13:02:25.050081  8814 consensus_meta.cc:287] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.609s	user 0.014s	sys 0.002s
I20260709 13:02:25.064157  2301 leader_election.cc:304] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:02:25.065165  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 20 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:25.067772  1676 raft_consensus.cc:1275] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 16 index: 14. Preceding OpId from leader: term: 18 index: 15. (index mismatch)
I20260709 13:02:25.068609  1674 raft_consensus.cc:2410] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for term 20 because replica has last-logged OpId of term: 18 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 11.
I20260709 13:02:25.074623  9010 leader_election.cc:290] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:25.075440  8921 leader_election.cc:290] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:02:25.080190  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 77285.3us
I20260709 13:02:25.081846  1676 raft_consensus.cc:1275] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 14 index: 13. Preceding OpId from leader: term: 16 index: 14. (index mismatch)
I20260709 13:02:25.085556  2301 leader_election.cc:304] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:02:25.087322  9061 raft_consensus.cc:493] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:02:25.092511  9061 raft_consensus.cc:515] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:25.061043  8771 raft_consensus.cc:515] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:25.040244  9036 consensus_queue.cc:260] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:25.078769  8814 raft_consensus.cc:515] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:25.112854  8368 raft_consensus.cc:2804] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 13:02:25.126232  1603 raft_consensus.cc:1240] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 24 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 23. Current term is 24. Ops: []
I20260709 13:02:25.095387  9010 consensus_queue.cc:1048] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.028s
I20260709 13:02:25.131016  1600 raft_consensus.cc:2368] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 for earlier term 18. Current term is 19.
I20260709 13:02:25.134143  1718 leader_election.cc:400] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 22 election: Vote denied by peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261) with higher term. Message: Invalid argument: T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 22. Current term is 23.
I20260709 13:02:25.129312  1644 leader_election.cc:304] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:02:25.134560  1718 leader_election.cc:403] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 22 election: Cancelling election due to peer responding with higher term
I20260709 13:02:25.137405  1569 leader_election.cc:304] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:02:25.126832  9036 raft_consensus.cc:3060] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:02:25.117544  9070 consensus_queue.cc:1048] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.043s
I20260709 13:02:25.139780  8914 raft_consensus.cc:515] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:25.130813  9058 raft_consensus.cc:493] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:25.153316  9058 raft_consensus.cc:515] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:25.154554  9012 raft_consensus.cc:2749] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Leader election lost for term 22. Reason: Vote denied by peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261) with higher term. Message: Invalid argument: T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 22. Current term is 23.
I20260709 13:02:25.138305  8368 raft_consensus.cc:697] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 19 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:02:25.155714  9058 leader_election.cc:290] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:25.156841  9072 raft_consensus.cc:493] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:25.157464  9072 raft_consensus.cc:515] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:25.159006  9061 leader_election.cc:290] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:25.159885  9072 leader_election.cc:290] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:25.146591  8880 raft_consensus.cc:2804] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:02:25.160742  8880 raft_consensus.cc:493] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Starting leader election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:02:25.161176  8880 raft_consensus.cc:3060] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:02:25.154662  8914 leader_election.cc:290] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 21 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:25.156459  8368 consensus_queue.cc:237] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.13, Last appended by leader: 13, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:25.167659  1678 raft_consensus.cc:2468] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 20.
I20260709 13:02:25.169750  9065 raft_consensus.cc:2749] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:02:25.193157  9054 raft_consensus.cc:515] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:25.194536  9068 raft_consensus.cc:493] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:25.203073  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:02:25.199178  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 59334.3us
W20260709 13:02:25.200774  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 67218.9us
I20260709 13:02:25.172031  9058 raft_consensus.cc:2804] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:02:25.207397  1569 leader_election.cc:304] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:02:25.208734  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 19 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:25.209754  1675 raft_consensus.cc:2393] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 19: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:02:25.213384  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:25.206807  9006 raft_consensus.cc:493] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:25.169879  8921 consensus_queue.cc:1059] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 23.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.038s
I20260709 13:02:25.214067  1597 raft_consensus.cc:2468] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 18.
I20260709 13:02:25.217610  9010 raft_consensus.cc:493] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:25.222743  9068 raft_consensus.cc:515] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:25.223461  9010 raft_consensus.cc:515] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:25.224198  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:25.224823  1603 raft_consensus.cc:3060] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:02:25.226342  8968 raft_consensus.cc:493] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:25.227368  8968 raft_consensus.cc:515] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:25.229681  8968 leader_election.cc:290] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:25.226796  9076 raft_consensus.cc:493] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:25.230753  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 17 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:25.230785  9076 raft_consensus.cc:515] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:25.225744  9010 leader_election.cc:290] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:25.233062  8952 raft_consensus.cc:493] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:25.222060  9006 raft_consensus.cc:515] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:25.233503  9076 leader_election.cc:290] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:25.236661  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:25.237227  1602 raft_consensus.cc:3060] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:02:25.237329  1596 raft_consensus.cc:2468] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 16.
I20260709 13:02:25.230250  8797 raft_consensus.cc:3055] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 LEADER]: Stepping down as leader of term 22
I20260709 13:02:25.236521  9006 leader_election.cc:290] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:25.245359  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
W20260709 13:02:25.247090  9056 log.cc:927] Time spent T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.503s	user 0.003s	sys 0.002s
I20260709 13:02:25.248440  8504 raft_consensus.cc:3055] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 LEADER]: Stepping down as leader of term 16
I20260709 13:02:25.233078  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:25.207935  9058 raft_consensus.cc:493] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:25.271458  8871 raft_consensus.cc:493] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:25.233594  8952 raft_consensus.cc:3060] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:02:25.272166  8871 raft_consensus.cc:515] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:25.215098  9061 raft_consensus.cc:2749] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:02:25.277372  1598 raft_consensus.cc:1240] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 20. Current term is 21. Ops: [20.10-20.10]
I20260709 13:02:25.250993  8797 raft_consensus.cc:740] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:02:25.278748  9054 leader_election.cc:290] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 21 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:25.279443  8797 consensus_queue.cc:260] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 22.14, Last appended by leader: 14, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:25.280623  8797 raft_consensus.cc:3060] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:02:25.282900  8634 leader_election.cc:290] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 20 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:25.287398  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:25.288825  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:25.290198  1673 raft_consensus.cc:1240] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 19. Current term is 20. Ops: []
I20260709 13:02:25.290547  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 19 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:25.290848  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:25.267139  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:25.200124  8870 raft_consensus.cc:515] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:25.296435  2298 leader_election.cc:304] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:02:25.226116  9068 leader_election.cc:290] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:25.310717  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:25.323764  1602 raft_consensus.cc:2468] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 17.
I20260709 13:02:25.325459  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:25.326133  1680 raft_consensus.cc:3060] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Advancing to term 18
W20260709 13:02:25.277449  9074 log.cc:927] Time spent T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.108s	user 0.000s	sys 0.000s
I20260709 13:02:25.325165  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 18 candidate_status { last_received { term: 16 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:25.329773  1679 raft_consensus.cc:2410] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for term 18 because replica has last-logged OpId of term: 17 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 7.
I20260709 13:02:25.332741  1677 raft_consensus.cc:1240] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 16. Current term is 17. Ops: []
I20260709 13:02:25.329191  8634 raft_consensus.cc:493] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:25.333532  8634 raft_consensus.cc:515] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:25.267880  8504 raft_consensus.cc:740] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:02:25.290750  9058 raft_consensus.cc:3060] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:02:25.336674  9080 raft_consensus.cc:2804] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:02:25.337221  9080 raft_consensus.cc:493] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:25.337282  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 23 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:25.337709  9080 raft_consensus.cc:3060] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:02:25.336045  8634 leader_election.cc:290] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:25.338694  1601 raft_consensus.cc:1240] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 18. Current term is 19. Ops: []
I20260709 13:02:25.333978  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:25.349278  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:25.349941  1678 raft_consensus.cc:2468] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 21.
I20260709 13:02:25.349886  1677 raft_consensus.cc:2468] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 19.
W20260709 13:02:25.356530  2301 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:25.357424  2301 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:25.359050  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:25.233645  9010 consensus_queue.cc:1059] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 15.14, Next index: 15, Last known committed idx: 13, Time since last communication: 0.094s
I20260709 13:02:25.364974  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:25.325151  8871 leader_election.cc:290] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:25.363826  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:25.373064  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:25.374087  1603 raft_consensus.cc:2468] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 16.
I20260709 13:02:25.375618  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:25.364662  1643 leader_election.cc:304] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:02:25.381538  1675 raft_consensus.cc:2393] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 20: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:02:25.352412  9016 raft_consensus.cc:493] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:25.385710  1603 raft_consensus.cc:2468] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 15.
I20260709 13:02:25.389904  1643 leader_election.cc:304] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:25.327251  8880 raft_consensus.cc:515] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:25.329394  9039 raft_consensus.cc:515] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:25.394351  1643 leader_election.cc:304] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:02:25.366920  8814 leader_election.cc:290] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 19 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:25.395720  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:25.335441  8504 consensus_queue.cc:260] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.10, Last appended by leader: 10, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:25.382499  9082 raft_consensus.cc:493] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:25.402942  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:25.405864  1680 raft_consensus.cc:2468] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 18.
I20260709 13:02:25.392625  8914 raft_consensus.cc:2804] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:02:25.417543  1671 raft_consensus.cc:1240] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 20. Current term is 21. Ops: []
I20260709 13:02:25.392596  8921 raft_consensus.cc:493] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:25.428792  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:25.434176  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:25.434405  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:25.435215  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:25.395401  8634 consensus_queue.cc:1059] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 18.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.031s
I20260709 13:02:25.436146  2330 raft_consensus.cc:3060] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:02:25.435482  2334 raft_consensus.cc:1275] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 19 index: 14. (index mismatch)
I20260709 13:02:25.438944  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:25.440182  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:25.440578  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 22 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:25.441082  2334 raft_consensus.cc:3060] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:02:25.441947  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:25.428555  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:25.442766  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 18 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:25.443753  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:25.444648  2337 raft_consensus.cc:3060] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:02:25.435257  2333 raft_consensus.cc:2468] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 16.
I20260709 13:02:25.447432  9012 raft_consensus.cc:493] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:25.448043  9012 raft_consensus.cc:515] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:25.448674  2333 raft_consensus.cc:1275] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 20 index: 15. Preceding OpId from leader: term: 22 index: 16. (index mismatch)
I20260709 13:02:25.446825  8772 raft_consensus.cc:2749] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 13:02:25.449489  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:25.396481  8968 consensus_queue.cc:1059] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 19.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.054s
I20260709 13:02:25.451442  8968 raft_consensus.cc:3055] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 13:02:25.451826  8968 raft_consensus.cc:740] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:02:25.452778  8968 consensus_queue.cc:260] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.10, Last appended by leader: 10, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:25.458271  8870 leader_election.cc:290] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 19 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:25.461025  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 23 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:25.401326  9011 raft_consensus.cc:2749] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:02:25.462325  8968 raft_consensus.cc:3060] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:02:25.462397  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:25.467195  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:25.467577  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:25.477692  2333 raft_consensus.cc:2468] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 20.
I20260709 13:02:25.485747  2334 raft_consensus.cc:2468] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 19.
W20260709 13:02:25.537508  1718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:25.391165  9016 raft_consensus.cc:515] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:25.573982  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:25.576894  2330 raft_consensus.cc:2468] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 18.
I20260709 13:02:25.419528  8914 raft_consensus.cc:493] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:02:25.468838  8981 consensus_meta.cc:287] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.700s	user 0.013s	sys 0.001s
I20260709 13:02:25.579751  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:25.576217  8764 raft_consensus.cc:493] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:25.430814  8504 raft_consensus.cc:3060] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:02:25.463158  8994 raft_consensus.cc:3055] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 23 LEADER]: Stepping down as leader of term 23
I20260709 13:02:25.586577  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:25.587882  1603 raft_consensus.cc:2468] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 21.
I20260709 13:02:25.492274  9011 raft_consensus.cc:493] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:25.468000  8880 leader_election.cc:290] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 21 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:02:25.577656  9052 consensus_meta.cc:287] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.633s	user 0.005s	sys 0.010s
I20260709 13:02:25.605886  2330 raft_consensus.cc:2393] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 15: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:02:25.419704  9082 raft_consensus.cc:515] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:25.583843  9012 leader_election.cc:290] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:25.584997  8914 raft_consensus.cc:3060] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:02:25.677521  1672 raft_consensus.cc:2410] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for term 23 because replica has last-logged OpId of term: 22 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 12.
I20260709 13:02:25.657600  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:25.679258  2328 raft_consensus.cc:3060] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:02:25.590602  8994 raft_consensus.cc:740] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
W20260709 13:02:25.706287  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 312154us
I20260709 13:02:25.583124  8921 raft_consensus.cc:515] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:25.710772  2329 raft_consensus.cc:2393] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 21: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:02:25.711194  2330 raft_consensus.cc:1240] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 19 LEADER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 18. Current term is 19. Ops: []
I20260709 13:02:25.649937  9002 raft_consensus.cc:493] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:25.592720  8981 raft_consensus.cc:515] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:25.725171  1596 raft_consensus.cc:1240] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 17. Current term is 18. Ops: []
I20260709 13:02:25.604157  8764 raft_consensus.cc:515] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:25.616163  9011 raft_consensus.cc:515] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:25.678555  9012 raft_consensus.cc:493] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:25.732825  2334 raft_consensus.cc:1275] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 21 index: 16. Preceding OpId from leader: term: 23 index: 17. (index mismatch)
I20260709 13:02:25.709964  8797 raft_consensus.cc:493] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:25.740445  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:25.749495  2328 raft_consensus.cc:2468] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 21.
I20260709 13:02:25.738636  2337 raft_consensus.cc:2468] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 20.
I20260709 13:02:25.752717  1599 raft_consensus.cc:2393] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 22: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:02:25.648645  9052 raft_consensus.cc:515] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:25.761493  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
W20260709 13:02:25.757566  9036 consensus_meta.cc:287] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.611s	user 0.009s	sys 0.004s
I20260709 13:02:25.767565  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 19 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:25.773576  1644 leader_election.cc:304] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:02:25.785925  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:25.730703  9002 raft_consensus.cc:515] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:25.794283  1644 leader_election.cc:304] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:02:25.794478  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:25.797549  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:25.802903  2328 raft_consensus.cc:3060] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Advancing to term 21
W20260709 13:02:25.798641  8952 consensus_meta.cc:287] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.526s	user 0.009s	sys 0.005s
W20260709 13:02:25.799336  9060 log.cc:927] Time spent T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 1.053s	user 0.007s	sys 0.007s
I20260709 13:02:25.810739  8952 raft_consensus.cc:515] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:25.801961  1569 leader_election.cc:304] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:02:25.805124  8764 leader_election.cc:290] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:25.812073  2336 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49082 (request call id 1299) took 2120 ms. Trace:
I20260709 13:02:25.812501  1602 raft_consensus.cc:1240] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 24 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 23. Current term is 24. Ops: []
I20260709 13:02:25.813210  8764 raft_consensus.cc:493] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:25.813578  1604 raft_consensus.cc:1240] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 17. Current term is 18. Ops: []
W20260709 13:02:25.814110  9083 log.cc:927] Time spent T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.088s	user 0.000s	sys 0.000s
I20260709 13:02:25.813773  8764 raft_consensus.cc:515] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:25.816267  8764 leader_election.cc:290] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:25.816817  8764 consensus_queue.cc:1059] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 13:02:25.806792  8880 raft_consensus.cc:2804] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Leader election won for term 21
I20260709 13:02:25.817993  8880 raft_consensus.cc:697] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 21 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:02:25.819108  8880 consensus_queue.cc:237] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.13, Last appended by leader: 13, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:25.824806  8771 leader_election.cc:290] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 21 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:25.825981  2333 raft_consensus.cc:2468] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 19.
I20260709 13:02:25.828142  8932 raft_consensus.cc:3055] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 13:02:25.828590  8932 raft_consensus.cc:740] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:02:25.755645  8797 raft_consensus.cc:515] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:25.784601  9039 leader_election.cc:290] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 18 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:25.836409  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:25.834064  2331 raft_consensus.cc:2468] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 16.
I20260709 13:02:25.841629  2335 raft_consensus.cc:2468] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 15.
I20260709 13:02:25.849941  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:25.880223  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 20 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:02:25.882085  1643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:25.882732  1643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:25.883121  1643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:25.883773  1754 raft_consensus.cc:2468] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 17.
I20260709 13:02:25.886907  9101 raft_consensus.cc:493] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:25.887578  9101 raft_consensus.cc:515] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:25.889969  9101 leader_election.cc:290] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:25.890538  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 19 candidate_status { last_received { term: 17 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:25.891359  1599 raft_consensus.cc:2468] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 18.
I20260709 13:02:25.755084  9012 raft_consensus.cc:515] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:25.898677  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:25.900314  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:02:25.904078  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 112868us
I20260709 13:02:25.884903  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:25.905596  1754 raft_consensus.cc:2468] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 18.
I20260709 13:02:25.906517  1745 raft_consensus.cc:2393] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 18: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:02:25.907609  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 19 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:02:25.908735  1643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:25.909029  2330 raft_consensus.cc:1275] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 14 index: 13. Preceding OpId from leader: term: 16 index: 14. (index mismatch)
I20260709 13:02:25.831022  9095 raft_consensus.cc:493] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:25.896414  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:25.911206  2328 raft_consensus.cc:2468] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 21.
I20260709 13:02:25.911700  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:25.832962  8981 leader_election.cc:290] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 22 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:25.812476  2336 rpcz_store.cc:276] 0709 13:02:23.691097 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:23.717971 (+ 26874us) service_pool.cc:224] Handling call
0709 13:02:23.718583 (+   612us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:24.139319 (+420736us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:24.139368 (+    49us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:24.255673 (+116305us) log.cc:844] Serialized 26 byte log entry
0709 13:02:24.739331 (+483658us) raft_consensus.cc:1645] Marking committed up to 14
0709 13:02:24.739383 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:24.739549 (+   166us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:25.798380 (+1058831us) spinlock_profiling.cc:243] Waited 14.8 ms on lock 0x7b4400022b4c. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f4262f9bc85 00007f4262f9c6a2 00007f4265d0e1db 00007f4265d09aaf 00007f4265cfd86e 00007f4265cfcbb6 00007f4265cfc7d8 00007f4265d06221 00007f4265d061d9 00007f4265d06169
0709 13:02:25.811559 (+ 13179us) raft_consensus.cc:1685] finished
0709 13:02:25.811587 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:25.811853 (+   266us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":40756,"raft.run_cpu_time_us":160,"raft.run_wall_time_us":20257,"spinlock_wait_cycles":34148224,"thread_start_us":745948,"threads_started":2,"wal-append.queue_time_us":401829}
I20260709 13:02:25.834481  9052 leader_election.cc:290] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 22 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:25.913504  2331 raft_consensus.cc:1240] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 18. Current term is 19. Ops: [18.15-18.15]
I20260709 13:02:25.831303  9096 raft_consensus.cc:493] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:02:25.831128  9016 leader_election.cc:290] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:25.921095  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:25.800712  9093 raft_consensus.cc:2804] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:02:25.910686  1753 raft_consensus.cc:2376] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 21. Re-sending same reply.
I20260709 13:02:25.792788  8814 consensus_queue.cc:1048] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 11, Time since last communication: 0.115s
I20260709 13:02:25.803138  8634 consensus_queue.cc:1059] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 18.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.107s
I20260709 13:02:25.763051  8994 consensus_queue.cc:260] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 23.15, Last appended by leader: 15, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:25.921756  1749 raft_consensus.cc:2468] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 19.
I20260709 13:02:25.928707  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 24 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:25.925882  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 20 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:25.829562  8932 consensus_queue.cc:260] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.10, Last appended by leader: 10, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:25.931777  1754 raft_consensus.cc:2468] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 19.
I20260709 13:02:25.931631  1745 raft_consensus.cc:2468] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 23.
I20260709 13:02:25.935091  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:25.936050  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 19 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:25.938246  2330 raft_consensus.cc:2393] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 19: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:02:25.939217  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:25.921015  9103 raft_consensus.cc:493] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:25.940860  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:25.927546  9096 raft_consensus.cc:515] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:25.945276  8797 leader_election.cc:290] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:02:25.946837  9058 consensus_meta.cc:287] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.611s	user 0.010s	sys 0.003s
I20260709 13:02:25.947654  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:25.960835  1754 raft_consensus.cc:2468] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 21.
I20260709 13:02:25.975754  1745 raft_consensus.cc:2468] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 17.
I20260709 13:02:25.948305  9002 leader_election.cc:290] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:25.948203  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:25.950117  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:25.935288  9088 raft_consensus.cc:493] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:25.939935  9100 raft_consensus.cc:493] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:25.951203  8634 consensus_queue.cc:1243] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer 3b879799899a499c8d409ee9f1277cd1 log is divergent from this leader: its last log entry 19.14 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 13
I20260709 13:02:25.984934  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:26.001963  1754 raft_consensus.cc:3060] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:02:25.919453  9095 raft_consensus.cc:515] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:26.004179  2337 raft_consensus.cc:1240] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 20. Current term is 21. Ops: []
I20260709 13:02:25.926986  9093 raft_consensus.cc:493] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:25.937731  8994 raft_consensus.cc:3060] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Advancing to term 24
I20260709 13:02:25.958665  8932 raft_consensus.cc:3060] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:02:25.989840  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:26.007259  1745 raft_consensus.cc:2468] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 20.
I20260709 13:02:25.979496  1718 leader_election.cc:304] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:25.973604  9103 raft_consensus.cc:515] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:26.010162  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:26.001952  9088 raft_consensus.cc:515] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:26.015986  1753 raft_consensus.cc:3060] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Advancing to term 24
I20260709 13:02:26.016796  9103 leader_election.cc:290] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:25.995436  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:26.017680  1749 raft_consensus.cc:2468] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 20.
I20260709 13:02:26.018760  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:26.019569  1749 raft_consensus.cc:2393] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 18: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:02:25.958382  8968 raft_consensus.cc:493] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:26.021430  1718 leader_election.cc:304] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:25.997008  2330 raft_consensus.cc:1240] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 20. Current term is 21. Ops: []
I20260709 13:02:26.009465  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:26.023378  1745 raft_consensus.cc:2468] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 19.
I20260709 13:02:26.010138  9082 leader_election.cc:290] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:26.025713  9104 raft_consensus.cc:2804] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 13:02:26.025808  8968 raft_consensus.cc:515] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:26.026401  9104 raft_consensus.cc:493] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:26.026846  9104 raft_consensus.cc:3060] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:02:26.029824  9101 raft_consensus.cc:2804] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:02:26.034770  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:26.035734  1718 leader_election.cc:304] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:26.036501  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:26.036118  9016 raft_consensus.cc:3055] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 18 LEADER]: Stepping down as leader of term 18
I20260709 13:02:26.037406  2329 raft_consensus.cc:1240] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 18. Current term is 19. Ops: [15.12-18.13]
I20260709 13:02:26.036177  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 23 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:26.038504  1749 raft_consensus.cc:2393] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 23: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:02:26.042529  9016 raft_consensus.cc:740] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
W20260709 13:02:26.008538  9051 log.cc:927] Time spent T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 1.269s	user 0.000s	sys 0.015s
I20260709 13:02:26.043485  1750 raft_consensus.cc:2410] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 for term 20 because replica has last-logged OpId of term: 18 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 11.
I20260709 13:02:26.044821  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:26.030259  9101 raft_consensus.cc:493] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:25.983626  9058 raft_consensus.cc:515] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:26.018244  9088 leader_election.cc:290] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:26.006800  9093 raft_consensus.cc:3060] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:02:26.048384  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
W20260709 13:02:26.051505  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 257411us
I20260709 13:02:26.052476  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:26.049520  9101 raft_consensus.cc:3060] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:02:26.054401  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:26.057830  1603 raft_consensus.cc:2393] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 16: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:02:26.058759  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:26.057376  1749 raft_consensus.cc:3060] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:02:26.063932  1644 leader_election.cc:304] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:02:26.064765  1644 leader_election.cc:304] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:02:26.005887  9100 raft_consensus.cc:515] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:26.072722  1753 raft_consensus.cc:2468] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 24.
I20260709 13:02:26.073720  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:26.074316  1753 raft_consensus.cc:3060] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Advancing to term 18
W20260709 13:02:26.075452  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 165913us
I20260709 13:02:26.071482  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:26.077631  1643 leader_election.cc:304] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 13:02:26.080847  1754 raft_consensus.cc:2468] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 18.
I20260709 13:02:26.076138  9016 consensus_queue.cc:260] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.13, Last appended by leader: 13, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:26.083698  9016 raft_consensus.cc:3060] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Advancing to term 19
W20260709 13:02:26.084414  9080 consensus_meta.cc:287] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.746s	user 0.014s	sys 0.000s
I20260709 13:02:26.084913  9080 raft_consensus.cc:515] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:26.086238  2332 raft_consensus.cc:2368] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 18. Current term is 19.
I20260709 13:02:26.090583  8968 leader_election.cc:290] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:26.091296  8968 raft_consensus.cc:493] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:26.091879  8968 raft_consensus.cc:515] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:26.052345  9012 leader_election.cc:290] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:26.095350  9012 raft_consensus.cc:2804] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 13:02:26.095881  9012 raft_consensus.cc:697] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 18 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:02:26.096946  9012 consensus_queue.cc:237] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 16.8, Last appended by leader: 8, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:02:26.109807  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 74171.7us
I20260709 13:02:26.109445  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:26.114338  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 19 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:26.127612  1754 raft_consensus.cc:2468] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 20.
I20260709 13:02:26.068002  8914 raft_consensus.cc:515] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:26.116070  9058 leader_election.cc:290] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 16 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:26.116869  9100 leader_election.cc:290] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:26.118206  9095 leader_election.cc:290] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:26.072024  9088 raft_consensus.cc:2804] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Leader election won for term 19
I20260709 13:02:26.110682  8634 raft_consensus.cc:2804] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:02:26.124992  9096 leader_election.cc:290] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:02:26.146251  2301 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:26.108160  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:26.112371  8880 raft_consensus.cc:2804] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Leader election won for term 16
I20260709 13:02:26.133443  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
W20260709 13:02:26.130673  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 287454us
I20260709 13:02:26.135912  9104 raft_consensus.cc:515] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:26.137537  8914 leader_election.cc:290] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 17 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:26.139540  8968 leader_election.cc:290] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:02:26.142226  2300 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:26.145941  8634 raft_consensus.cc:493] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:02:26.156412  8634 raft_consensus.cc:3060] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:02:26.143476  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 21 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:26.144681  9088 raft_consensus.cc:697] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:02:26.158640  9088 consensus_queue.cc:237] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.13, Last appended by leader: 13, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:26.158968  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
W20260709 13:02:26.108186  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 43520.2us
I20260709 13:02:26.144706  1753 raft_consensus.cc:2468] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 18.
I20260709 13:02:26.161540  1644 leader_election.cc:304] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:02:26.150769  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:26.159231  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:26.162088  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 20 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:26.163272  1753 raft_consensus.cc:3060] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:02:26.163432  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 17 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:26.162530  1644 leader_election.cc:304] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:02:26.164367  1746 raft_consensus.cc:2410] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for term 17 because replica has last-logged OpId of term: 16 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 9.
I20260709 13:02:26.163157  1751 raft_consensus.cc:2468] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 15.
W20260709 13:02:26.147234  1718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:26.165895  9107 raft_consensus.cc:2804] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Leader election won for term 21
I20260709 13:02:26.174772  1746 raft_consensus.cc:1240] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 23. Current term is 24. Ops: []
I20260709 13:02:26.182103  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:26.186194  1748 raft_consensus.cc:1275] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 18 index: 13. Preceding OpId from leader: term: 19 index: 14. (index mismatch)
I20260709 13:02:26.178068  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:26.195186  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:26.166728  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 20 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:02:26.221823  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 59434us
I20260709 13:02:26.178897  2298 leader_election.cc:400] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 18 election: Vote denied by peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445) with higher term. Message: Invalid argument: T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 for earlier term 18. Current term is 19.
I20260709 13:02:26.223547  2298 leader_election.cc:403] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 18 election: Cancelling election due to peer responding with higher term
I20260709 13:02:26.182605  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:26.224803  2333 raft_consensus.cc:1240] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 21. Current term is 22. Ops: []
I20260709 13:02:26.183526  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:26.177721  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:26.179327  1745 raft_consensus.cc:2393] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 22: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:02:26.226125  1680 raft_consensus.cc:2468] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 21.
I20260709 13:02:26.226855  1602 raft_consensus.cc:2393] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 20: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:02:26.179916  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
W20260709 13:02:26.228188  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 64827.2us
I20260709 13:02:26.227835  1676 raft_consensus.cc:1240] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 16. Current term is 17. Ops: []
I20260709 13:02:26.228694  1750 raft_consensus.cc:3060] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Advancing to term 20
W20260709 13:02:26.221374  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 58501.5us
I20260709 13:02:26.235446  1754 raft_consensus.cc:3060] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:02:26.227187  1605 raft_consensus.cc:2468] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 19.
I20260709 13:02:26.237326  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:26.238143  1747 raft_consensus.cc:2393] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 17: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
W20260709 13:02:26.245735  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 67114us
I20260709 13:02:26.245954  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:26.259567  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 24 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:26.260936  1604 raft_consensus.cc:2393] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 18: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:02:26.261881  1754 raft_consensus.cc:2410] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for term 21 because replica has last-logged OpId of term: 20 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 9.
I20260709 13:02:26.264233  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:26.269805  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:26.270970  1597 raft_consensus.cc:2393] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 18: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:02:26.275193  1671 raft_consensus.cc:1240] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 19. Current term is 20. Ops: []
I20260709 13:02:26.180330  8893 raft_consensus.cc:493] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:26.191892  8817 leader_election.cc:290] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 20 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:26.292215  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:26.228047  9107 raft_consensus.cc:697] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:02:26.283573  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:26.286638  1753 raft_consensus.cc:2468] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 20.
I20260709 13:02:26.171451  8880 raft_consensus.cc:697] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
W20260709 13:02:26.295820  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 65758.2us
W20260709 13:02:26.288483  1717 connection.cc:256] RPC call timeout handler was delayed by -1.2422194630000263s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16690000000000002s for a response
I20260709 13:02:26.297281  1643 leader_election.cc:304] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:02:26.298604  2329 raft_consensus.cc:2393] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 18: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:02:26.299659  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:26.300295  2329 raft_consensus.cc:2468] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 15.
I20260709 13:02:26.301297  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:26.301884  2329 raft_consensus.cc:2468] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 21.
I20260709 13:02:26.302770  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:26.304179  2329 raft_consensus.cc:1240] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 20. Current term is 21. Ops: []
I20260709 13:02:26.305262  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:26.305936  1602 raft_consensus.cc:2468] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 17.
I20260709 13:02:26.307583  1753 raft_consensus.cc:1275] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 19 index: 15. Preceding OpId from leader: term: 23 index: 17. (index mismatch)
I20260709 13:02:26.309446  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:26.264040  9016 raft_consensus.cc:3060] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:02:26.312458  1599 raft_consensus.cc:2393] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 19: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
W20260709 13:02:26.315228  1717 connection.cc:256] RPC call timeout handler was delayed by -1.1117194629999858s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1814s for a response
W20260709 13:02:26.315723  1717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:26.316084  1717 connection.cc:256] RPC call timeout handler was delayed by -1.0289194630000225s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1906s for a response
I20260709 13:02:26.317164  1717 leader_election.cc:304] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:26.320478  1643 leader_election.cc:304] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:26.269933  9082 raft_consensus.cc:2749] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Leader election lost for term 18. Reason: Vote denied by peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445) with higher term. Message: Invalid argument: T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 for earlier term 18. Current term is 19.
I20260709 13:02:26.319119  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:26.325251  1597 raft_consensus.cc:2393] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 21: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:02:26.286962  9104 leader_election.cc:290] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 22 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 13:02:26.325078  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 94731.7us
I20260709 13:02:26.321677  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:26.330307  1717 leader_election.cc:304] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:26.332155  1750 raft_consensus.cc:2468] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 20.
I20260709 13:02:26.333480  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:26.286557  9053 consensus_queue.cc:1059] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 17.15, Next index: 16, Last known committed idx: 14, Time since last communication: 0.054s
I20260709 13:02:26.334149  1597 raft_consensus.cc:3060] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:02:26.339908  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:26.340852  2329 raft_consensus.cc:2393] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 19: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:02:26.341399  9061 raft_consensus.cc:3055] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 LEADER]: Stepping down as leader of term 18
I20260709 13:02:26.341823  9061 raft_consensus.cc:740] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:02:26.342794  9061 consensus_queue.cc:260] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 18.15, Last appended by leader: 15, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:26.343966  9061 raft_consensus.cc:3060] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:02:26.344338  8880 consensus_queue.cc:237] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:26.184662  1495 catalog_manager.cc:5697] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 reported cstate change: term changed from 11 to 16, leader changed from 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) to 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194). New cstate: current_term: 16 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 13:02:26.327440  8932 raft_consensus.cc:493] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:26.350900  8932 raft_consensus.cc:515] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:26.340062  1672 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52778 (request call id 2782) took 1058 ms. Trace:
I20260709 13:02:26.354365  1672 rpcz_store.cc:276] 0709 13:02:25.281293 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:25.336852 (+ 55559us) service_pool.cc:224] Handling call
0709 13:02:26.339820 (+1002968us) spinlock_profiling.cc:243] Waited 339 ms on lock 0x7b5400162e54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:02:26.339965 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":780223232}
I20260709 13:02:26.354013  8797 raft_consensus.cc:493] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:26.362233  9096 raft_consensus.cc:2749] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:02:26.335417  8968 raft_consensus.cc:2749] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:02:26.363003  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:26.364522  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:26.370857  8504 raft_consensus.cc:493] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:02:26.370174  8771 raft_consensus.cc:2749] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:02:26.371538  8504 raft_consensus.cc:515] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:26.371959  9080 leader_election.cc:290] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 19 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:26.363039  8932 leader_election.cc:290] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:26.370903  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:26.374526  1598 raft_consensus.cc:2368] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 19. Current term is 20.
I20260709 13:02:26.322072  8893 raft_consensus.cc:515] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:26.378831  2336 raft_consensus.cc:2468] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 20.
I20260709 13:02:26.357338  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:26.380540  2330 raft_consensus.cc:2468] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 24.
I20260709 13:02:26.380648  1604 raft_consensus.cc:2468] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 22.
I20260709 13:02:26.362632  8797 raft_consensus.cc:515] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:02:26.395255  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 603755us
I20260709 13:02:26.397670  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:26.398983  1604 raft_consensus.cc:2393] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 22: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:02:26.399987  1569 leader_election.cc:304] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d
W20260709 13:02:26.410840  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 90505us
I20260709 13:02:26.414908  8968 consensus_queue.cc:1059] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 20.15, Next index: 16, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 13:02:26.416097  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:26.416353  8968 raft_consensus.cc:3055] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 13:02:26.416957  1602 raft_consensus.cc:2468] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 21.
W20260709 13:02:26.421751  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:26.423113  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:26.424024  1599 raft_consensus.cc:3060] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:02:26.427129  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:26.415197  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:26.432643  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 17 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:26.433480  1596 raft_consensus.cc:2376] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 17. Re-sending same reply.
W20260709 13:02:26.434849  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:26.434898  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:26.417074  8968 raft_consensus.cc:740] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:02:26.437218  8968 consensus_queue.cc:260] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.15, Last appended by leader: 15, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:26.439378  8968 raft_consensus.cc:3060] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:02:26.462762  1745 raft_consensus.cc:1240] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 16. Current term is 17. Ops: [16.10-16.10]
I20260709 13:02:26.464638  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:26.476341  9011 leader_election.cc:290] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:26.352231  9107 consensus_queue.cc:237] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.15, Last appended by leader: 15, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:26.492260  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:26.492537  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:26.493157  2333 raft_consensus.cc:2468] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 19.
I20260709 13:02:26.493950  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:26.494867  2329 raft_consensus.cc:2393] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 21: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:02:26.494051  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:02:26.492389  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 162252us
I20260709 13:02:26.496127  1717 leader_election.cc:304] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:26.497563  1717 leader_election.cc:304] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:26.495024  1718 leader_election.cc:304] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:02:26.498785  9096 raft_consensus.cc:2749] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:02:26.499531  9096 raft_consensus.cc:2764] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Leader pre-election decision vote started in defunct term 23: won
I20260709 13:02:26.499938  8772 raft_consensus.cc:3055] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 20 LEADER]: Stepping down as leader of term 20
I20260709 13:02:26.500253  8797 leader_election.cc:290] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:26.502400  8932 raft_consensus.cc:3060] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:02:26.504664  9082 raft_consensus.cc:3055] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 13:02:26.505061  9082 raft_consensus.cc:740] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:02:26.505961  9082 consensus_queue.cc:260] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 17.15, Last appended by leader: 15, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:26.474917  8504 leader_election.cc:290] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:26.507292  9082 raft_consensus.cc:3060] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:02:26.470062  8893 leader_election.cc:290] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:26.507941  1644 leader_election.cc:304] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:02:26.509629  1644 leader_election.cc:304] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:02:26.515091  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:26.515836  2328 raft_consensus.cc:2368] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 19. Current term is 20.
I20260709 13:02:26.517035  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:26.517863  2336 raft_consensus.cc:2393] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 21: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:02:26.523406  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:26.524109  1745 raft_consensus.cc:3060] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:02:26.524441  1747 raft_consensus.cc:1240] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 22. Current term is 23. Ops: [20.13-22.14]
I20260709 13:02:26.525115  9115 consensus_queue.cc:1059] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 21.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20260709 13:02:26.525738  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:26.526417  1754 raft_consensus.cc:2468] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 19.
I20260709 13:02:26.527305  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:26.527520  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:26.395258  8921 leader_election.cc:290] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:26.527840  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:26.528391  9008 raft_consensus.cc:2764] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Leader pre-election decision vote started in defunct term 15: won
I20260709 13:02:26.528656  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:26.529354  1753 raft_consensus.cc:3060] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:02:26.530232  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 19 candidate_status { last_received { term: 17 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:26.530941  1747 raft_consensus.cc:2468] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 18.
I20260709 13:02:26.528487  1752 raft_consensus.cc:2468] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 20.
I20260709 13:02:26.396538  9103 raft_consensus.cc:2804] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:02:26.536840  9103 raft_consensus.cc:493] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:26.537673  1601 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 1814) took 1498 ms. Trace:
I20260709 13:02:26.537886  9117 raft_consensus.cc:2804] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:02:26.538549  9117 raft_consensus.cc:493] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:26.538980  9117 raft_consensus.cc:3060] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:02:26.539404  2332 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49082 (request call id 1317) took 1116 ms. Trace:
I20260709 13:02:26.539783  2332 rpcz_store.cc:276] 0709 13:02:25.423235 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:25.423488 (+   253us) service_pool.cc:224] Handling call
0709 13:02:26.539178 (+1115690us) spinlock_profiling.cc:243] Waited 643 ms on lock 0x7b54005f3a54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:02:26.539310 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1478528384}
I20260709 13:02:26.538086  1601 rpcz_store.cc:276] 0709 13:02:25.039287 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:25.039443 (+   156us) service_pool.cc:224] Handling call
0709 13:02:25.039835 (+   392us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:25.349361 (+309526us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:26.537395 (+1188034us) spinlock_profiling.cc:243] Waited 299 ms on lock 0x7b54000f0d54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:02:26.537599 (+   204us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":687095168}
I20260709 13:02:26.500326  8772 raft_consensus.cc:740] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:02:26.542687  8772 consensus_queue.cc:260] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 20.10, Last appended by leader: 10, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:26.543736  9119 raft_consensus.cc:3055] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 LEADER]: Stepping down as leader of term 21
I20260709 13:02:26.543881  8772 raft_consensus.cc:3060] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:02:26.544270  9119 raft_consensus.cc:740] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
W20260709 13:02:26.545774  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 145975us
I20260709 13:02:26.545359  9119 consensus_queue.cc:260] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 21.16, Last appended by leader: 16, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:26.547115  1569 leader_election.cc:304] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:02:26.547600  8771 consensus_queue.cc:1048] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20260709 13:02:26.548524  9016 raft_consensus.cc:2749] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:02:26.548736  1569 leader_election.cc:304] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:02:26.549154  8771 raft_consensus.cc:2749] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:02:26.549582  1680 raft_consensus.cc:2468] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 21.
I20260709 13:02:26.549778  1569 leader_election.cc:304] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:02:26.537170  9115 raft_consensus.cc:2804] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:02:26.550809  9115 raft_consensus.cc:493] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:26.549933  8771 raft_consensus.cc:2804] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Leader election won for term 19
I20260709 13:02:26.551223  9115 raft_consensus.cc:3060] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:02:26.551700  8771 raft_consensus.cc:697] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
W20260709 13:02:26.554278  2300 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:26.472241  8952 leader_election.cc:290] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 18 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:26.560552  9110 consensus_queue.cc:1048] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.000s
I20260709 13:02:26.564339  1753 raft_consensus.cc:2468] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 20.
I20260709 13:02:26.537245  9103 raft_consensus.cc:3060] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:02:26.553786  8771 consensus_queue.cc:237] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:26.573449  9117 raft_consensus.cc:515] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:26.573971  1749 raft_consensus.cc:2468] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 19.
I20260709 13:02:26.579831  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:26.580164  2301 leader_election.cc:304] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:26.583021  1753 raft_consensus.cc:2468] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 22.
I20260709 13:02:26.587391  1569 leader_election.cc:304] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:02:26.565289  9101 raft_consensus.cc:515] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:26.617486  1745 raft_consensus.cc:2468] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 21.
W20260709 13:02:26.578872  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 81446.6us
I20260709 13:02:26.618891  1717 leader_election.cc:304] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:26.623235  1599 raft_consensus.cc:2468] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 17.
I20260709 13:02:26.624076  1603 raft_consensus.cc:1240] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 17. Current term is 18. Ops: []
I20260709 13:02:26.555349  9119 raft_consensus.cc:3060] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:02:26.640904  2301 leader_election.cc:304] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:26.642364  1597 raft_consensus.cc:2468] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 19.
I20260709 13:02:26.645221  1751 raft_consensus.cc:2393] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 20: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:02:26.602864  9064 raft_consensus.cc:493] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:26.616498  9008 raft_consensus.cc:2804] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Leader election won for term 18
I20260709 13:02:26.616844  8952 raft_consensus.cc:2804] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Leader election won for term 20
I20260709 13:02:26.647843  2301 leader_election.cc:304] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:26.617595  9016 raft_consensus.cc:493] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:26.663306  2299 leader_election.cc:304] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
W20260709 13:02:26.669167  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 50426.4us
I20260709 13:02:26.626262  9001 raft_consensus.cc:2764] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Leader pre-election decision vote started in defunct term 17: won
I20260709 13:02:26.661432  1569 leader_election.cc:304] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:02:26.670928  2298 leader_election.cc:304] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:02:26.577121  8764 consensus_queue.cc:1059] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 20.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.062s
I20260709 13:02:26.676148  1717 leader_election.cc:304] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:02:26.680107  1569 leader_election.cc:400] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 18 election: Vote denied by peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625) with higher term. Message: Invalid argument: T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 18. Current term is 19.
I20260709 13:02:26.620965  9067 consensus_queue.cc:1059] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 18.12, Next index: 13, Last known committed idx: 11, Time since last communication: 0.050s
I20260709 13:02:26.681576  1569 leader_election.cc:403] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 18 election: Cancelling election due to peer responding with higher term
I20260709 13:02:26.639552  9111 raft_consensus.cc:2749] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 13:02:26.685015  8934 raft_consensus.cc:2804] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 13:02:26.700397  8504 raft_consensus.cc:493] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:26.683363  9054 raft_consensus.cc:2804] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 13:02:26.701246  9054 raft_consensus.cc:493] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:26.701713  9054 raft_consensus.cc:3060] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:02:26.701089  8504 raft_consensus.cc:515] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:26.704319  8504 leader_election.cc:290] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:26.718636  9118 raft_consensus.cc:493] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:02:26.719051  9054 raft_consensus.cc:515] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:26.719399  9118 raft_consensus.cc:515] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:26.722684  8504 raft_consensus.cc:493] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:02:26.723168  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:26.723287  8504 raft_consensus.cc:515] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:26.724104  1603 raft_consensus.cc:3060] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:02:26.725138  1569 leader_election.cc:304] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:02:26.725683  8504 leader_election.cc:290] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:02:26.728150  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 62594.3us
I20260709 13:02:26.729455  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:26.729760  2301 leader_election.cc:304] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:02:26.730346  1672 raft_consensus.cc:2410] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for term 17 because replica has last-logged OpId of term: 16 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 9.
I20260709 13:02:26.732005  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:26.732787  1672 raft_consensus.cc:2393] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 20: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
W20260709 13:02:26.734057  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 176436us
W20260709 13:02:26.735579  1571 leader_election.cc:336] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Timed out: RequestConsensusVote RPC to 127.0.107.195:40693 timed out after 1.557s (SENT)
I20260709 13:02:26.736068  1571 leader_election.cc:304] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:02:26.738371  1603 raft_consensus.cc:2468] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 22.
I20260709 13:02:26.738973  8764 raft_consensus.cc:3055] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 13:02:26.750077  8764 raft_consensus.cc:740] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:02:26.750512  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:26.751317  8764 consensus_queue.cc:260] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.15, Last appended by leader: 15, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:26.752261  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:26.752957  1675 raft_consensus.cc:3060] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:02:26.753296  8764 raft_consensus.cc:3060] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Advancing to term 21
W20260709 13:02:26.739252  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 63242.5us
I20260709 13:02:26.755286  1717 leader_election.cc:304] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:02:26.756346  9012 raft_consensus.cc:2804] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:02:26.756757  9012 raft_consensus.cc:493] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:26.757107  9012 raft_consensus.cc:3060] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:02:26.757797  1717 leader_election.cc:304] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:26.685443  9015 raft_consensus.cc:2804] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader election won for term 15
I20260709 13:02:26.759310  9095 raft_consensus.cc:2749] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:02:26.763321  1675 raft_consensus.cc:2468] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 18.
I20260709 13:02:26.740479  9104 raft_consensus.cc:493] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:26.796262  9104 raft_consensus.cc:515] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:26.741238  1672 raft_consensus.cc:1275] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 20 index: 15. Preceding OpId from leader: term: 22 index: 16. (index mismatch)
I20260709 13:02:26.680990  9016 raft_consensus.cc:515] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:26.742995  1751 raft_consensus.cc:1240] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 19. Current term is 20. Ops: []
I20260709 13:02:26.677845  9002 consensus_queue.cc:1059] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 17.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.072s
I20260709 13:02:26.675380  9080 raft_consensus.cc:2804] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:02:26.748039  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 21 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:26.748319  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 23 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:26.814985  1671 raft_consensus.cc:2410] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for term 23 because replica has last-logged OpId of term: 22 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 12.
I20260709 13:02:26.749389  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 17 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:02:26.763675  2300 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:26.764812  1745 raft_consensus.cc:1275] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 19 index: 15. Preceding OpId from leader: term: 21 index: 16. (index mismatch)
W20260709 13:02:26.823736  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:26.765121  9101 leader_election.cc:290] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 20 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:26.623042  8634 raft_consensus.cc:515] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:26.625005  8981 consensus_queue.cc:1048] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.050s
I20260709 13:02:26.852452  1672 raft_consensus.cc:1240] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 17. Current term is 18. Ops: []
I20260709 13:02:26.765924  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:26.853189  1749 raft_consensus.cc:3060] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:02:26.647877  8797 consensus_queue.cc:1048] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.119s
I20260709 13:02:26.767143  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:26.866520  1753 raft_consensus.cc:2468] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 19.
I20260709 13:02:26.769198  1752 raft_consensus.cc:1240] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 17. Current term is 18. Ops: []
I20260709 13:02:26.769550  9012 raft_consensus.cc:515] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:26.869724  9012 leader_election.cc:290] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 15 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:26.770496  1747 raft_consensus.cc:1240] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 20. Current term is 21. Ops: []
I20260709 13:02:26.870944  1754 raft_consensus.cc:1240] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 14. Current term is 15. Ops: []
I20260709 13:02:26.697242  8921 consensus_queue.cc:1059] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.075s
I20260709 13:02:26.844311  8814 raft_consensus.cc:2749] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:02:26.872761  9065 raft_consensus.cc:3055] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 13:02:26.873271  9065 raft_consensus.cc:740] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:02:26.772828  9067 raft_consensus.cc:3055] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 13:02:26.874617  9067 raft_consensus.cc:740] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:02:26.874536  9065 consensus_queue.cc:260] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.13, Last appended by leader: 13, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:26.875686  9067 consensus_queue.cc:260] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.12, Last appended by leader: 12, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:26.739571  9096 raft_consensus.cc:2804] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:02:26.773476  8994 raft_consensus.cc:2749] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:02:26.670467  9064 raft_consensus.cc:515] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:26.877246  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:26.878010  1671 raft_consensus.cc:2468] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 21.
I20260709 13:02:26.878968  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:26.879278  9064 leader_election.cc:290] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 13:02:26.774748  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 45091.7us
I20260709 13:02:26.879736  1671 raft_consensus.cc:2393] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 20: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:02:26.673574  9008 raft_consensus.cc:697] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:02:26.780638  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
W20260709 13:02:26.781595  9093 consensus_meta.cc:287] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.727s	user 0.015s	sys 0.000s
I20260709 13:02:26.781936  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:26.881603  9093 raft_consensus.cc:515] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:26.881839  1679 raft_consensus.cc:3060] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:02:26.882911  1674 raft_consensus.cc:2393] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 21: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:02:26.883656  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:26.884047  9093 leader_election.cc:290] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 21 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:26.884356  1675 raft_consensus.cc:2468] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 22.
I20260709 13:02:26.676766  8952 raft_consensus.cc:697] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:02:26.885996  8952 consensus_queue.cc:237] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.13, Last appended by leader: 13, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:26.886965  9008 consensus_queue.cc:237] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.12, Last appended by leader: 12, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:26.786193  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 95171.1us
W20260709 13:02:26.889514  2300 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:26.790571  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 65561.4us
I20260709 13:02:26.881842  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:26.722244  9054 leader_election.cc:290] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 22 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:26.844832  1676 raft_consensus.cc:2393] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 17: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:02:26.802942  8817 raft_consensus.cc:2749] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Leader election lost for term 18. Reason: Vote denied by peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625) with higher term. Message: Invalid argument: T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 18. Current term is 19.
I20260709 13:02:26.843880  8880 raft_consensus.cc:493] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:26.877087  9096 raft_consensus.cc:493] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:26.788817  1597 raft_consensus.cc:1240] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 16. Current term is 17. Ops: []
I20260709 13:02:26.810945  9015 raft_consensus.cc:697] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:02:26.922716  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:26.813783  9080 raft_consensus.cc:493] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:26.903483  2301 leader_election.cc:304] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:26.906657  1646 leader_election.cc:304] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:26.908300  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:26.846243  9061 raft_consensus.cc:2804] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Leader election won for term 20
I20260709 13:02:26.843186  9111 raft_consensus.cc:493] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:26.914904  1569 leader_election.cc:304] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
W20260709 13:02:26.918073  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 160418us
I20260709 13:02:26.910632  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:02:26.920161  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 184703us
I20260709 13:02:26.944696  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 17 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:26.945940  1671 raft_consensus.cc:2393] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 21: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:02:26.893630  8814 raft_consensus.cc:493] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:26.956063  9117 leader_election.cc:290] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 21 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:26.955900  8814 raft_consensus.cc:515] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } }
I20260709 13:02:26.956933  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:26.959187  9067 raft_consensus.cc:3060] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:02:26.963038  9015 consensus_queue.cc:237] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.11, Last appended by leader: 11, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:26.958158  8814 leader_election.cc:290] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:26.816248  9115 raft_consensus.cc:515] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:26.950345  9002 consensus_queue.cc:1059] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 17.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 13:02:26.950968  9053 consensus_queue.cc:1059] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 13:02:26.951902  1571 leader_election.cc:304] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:02:26.894119  9103 raft_consensus.cc:515] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:26.954553  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:26.920940  8893 raft_consensus.cc:2804] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 13:02:26.898766  8934 raft_consensus.cc:697] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 19 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:02:26.974453  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:26.976183  1678 raft_consensus.cc:2410] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for term 21 because replica has last-logged OpId of term: 20 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 9.
I20260709 13:02:26.926972  9096 raft_consensus.cc:3060] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Advancing to term 18
W20260709 13:02:27.025017  9082 consensus_meta.cc:287] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.517s	user 0.019s	sys 0.000s
I20260709 13:02:26.946310  1717 leader_election.cc:304] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:26.974819  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:26.977083  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:27.031755  1676 raft_consensus.cc:3060] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:02:26.970065  1679 raft_consensus.cc:2468] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 22.
W20260709 13:02:26.972488  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 295344us
I20260709 13:02:26.953845  2334 raft_consensus.cc:1275] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 18 index: 12. Preceding OpId from leader: term: 21 index: 14. (index mismatch)
I20260709 13:02:26.935290  9080 raft_consensus.cc:3060] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Advancing to term 21
W20260709 13:02:26.971428  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 64685.2us
I20260709 13:02:26.985551  2331 raft_consensus.cc:3060] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:02:26.926456  8880 raft_consensus.cc:515] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:26.980027  9061 raft_consensus.cc:697] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:02:26.916811  8994 consensus_queue.cc:1059] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 19.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.059s
I20260709 13:02:26.987351  8771 raft_consensus.cc:3060] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Advancing to term 20
W20260709 13:02:26.987129  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 72257.5us
I20260709 13:02:26.967478  9054 raft_consensus.cc:2804] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:02:26.965588  9104 leader_election.cc:290] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:26.988168  1674 raft_consensus.cc:1240] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 20. Current term is 21. Ops: []
I20260709 13:02:26.969347  8932 raft_consensus.cc:2749] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:02:26.928664  9124 raft_consensus.cc:493] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:02:26.917171  9001 consensus_queue.cc:1048] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.059s
I20260709 13:02:26.910130  1749 raft_consensus.cc:2468] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 19.
I20260709 13:02:27.022825  1677 raft_consensus.cc:2393] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 16: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:02:27.035598  1646 leader_election.cc:304] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 13:02:26.985729  1571 leader_election.cc:304] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 13:02:27.036471  2299 leader_election.cc:304] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:02:27.026466  8934 consensus_queue.cc:237] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.12, Last appended by leader: 12, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:27.039505  1718 leader_election.cc:304] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:26.987421  8926 raft_consensus.cc:2804] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Leader election won for term 18
I20260709 13:02:27.075484  9104 raft_consensus.cc:2804] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 13:02:27.075912  9104 raft_consensus.cc:493] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:27.076264  9104 raft_consensus.cc:3060] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:02:27.044003  2337 raft_consensus.cc:2468] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 18.
I20260709 13:02:27.019696  9012 raft_consensus.cc:493] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:27.068892  8814 raft_consensus.cc:493] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:27.083498  9012 raft_consensus.cc:515] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:27.083912  8814 raft_consensus.cc:515] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:27.030195  8914 raft_consensus.cc:2749] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:02:27.062400  9101 raft_consensus.cc:2749] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:02:26.979545  9088 raft_consensus.cc:2804] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 13:02:27.087304  9088 raft_consensus.cc:493] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:27.087852  9088 raft_consensus.cc:3060] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Advancing to term 22
W20260709 13:02:27.089442  1644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:27.053202  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 109077us
I20260709 13:02:27.057138  1679 raft_consensus.cc:1240] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 19. Current term is 20. Ops: []
I20260709 13:02:27.091388  1717 leader_election.cc:304] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 13:02:27.091691  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 56261.7us
I20260709 13:02:27.067368  1643 leader_election.cc:304] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:27.061023  9054 raft_consensus.cc:493] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:02:27.049764  8968 consensus_meta.cc:287] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.610s	user 0.012s	sys 0.000s
I20260709 13:02:27.095517  1747 raft_consensus.cc:1275] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 18 index: 13. Preceding OpId from leader: term: 19 index: 14. (index mismatch)
I20260709 13:02:27.095633  1569 leader_election.cc:304] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:02:27.097769  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
W20260709 13:02:27.098489  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 112861us
I20260709 13:02:27.098678  1747 raft_consensus.cc:3060] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Advancing to term 21
I20260709 13:02:27.099633  1571 leader_election.cc:304] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:02:27.101573  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:27.102618  1717 leader_election.cc:304] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:27.092516  1646 leader_election.cc:304] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
W20260709 13:02:27.155032  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 62638.9us
I20260709 13:02:27.068677  9039 consensus_queue.cc:1048] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.035s
I20260709 13:02:27.104003  8914 raft_consensus.cc:2804] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Leader election won for term 20
I20260709 13:02:27.104049  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:27.104384  9054 raft_consensus.cc:3060] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:02:27.104813  9012 leader_election.cc:290] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:27.106868  8926 raft_consensus.cc:697] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:02:27.107326  8764 raft_consensus.cc:2749] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 13:02:27.108469  9096 raft_consensus.cc:515] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } }
I20260709 13:02:27.112969  9058 raft_consensus.cc:2804] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:02:27.115588  8981 raft_consensus.cc:2804] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Leader election won for term 22
I20260709 13:02:27.116458  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:27.117113  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:27.120621  1571 leader_election.cc:304] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:02:27.121220  8817 raft_consensus.cc:2749] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:02:27.125056  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:27.127072  2331 raft_consensus.cc:2468] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 16.
I20260709 13:02:27.044574  9103 leader_election.cc:290] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:26.976630  9111 raft_consensus.cc:515] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:27.147449  1569 leader_election.cc:304] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:02:27.142424  1750 raft_consensus.cc:2468] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 21.
I20260709 13:02:27.061849  9116 raft_consensus.cc:493] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
W20260709 13:02:27.143002  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 75813.2us
I20260709 13:02:27.309640  1643 leader_election.cc:304] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
W20260709 13:02:27.141142  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 106226us
I20260709 13:02:27.142900  1676 raft_consensus.cc:2468] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 20.
I20260709 13:02:27.152001  1674 raft_consensus.cc:1275] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 16 index: 8. Preceding OpId from leader: term: 18 index: 9. (index mismatch)
I20260709 13:02:27.148259  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:27.162493  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:27.315189  1749 raft_consensus.cc:3060] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Advancing to term 22
I20260709 13:02:27.061546  9107 consensus_queue.cc:1048] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.091s
I20260709 13:02:27.104218  9118 leader_election.cc:290] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:27.066561  9136 consensus_queue.cc:1059] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 17.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.121s
I20260709 13:02:27.096567  9124 raft_consensus.cc:515] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:27.303696  2337 raft_consensus.cc:2468] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 15.
I20260709 13:02:27.303898  1753 raft_consensus.cc:2468] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 20.
I20260709 13:02:27.304279  2330 raft_consensus.cc:2393] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 19: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:02:27.196772  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:02:27.307395  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 186942us
I20260709 13:02:27.340395  1571 leader_election.cc:304] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:02:27.132689  9101 raft_consensus.cc:493] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:27.317981  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:27.341500  1751 raft_consensus.cc:3060] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:02:27.146306  8772 raft_consensus.cc:2749] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:02:27.347900  8989 raft_consensus.cc:3055] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 13:02:27.348512  8989 raft_consensus.cc:740] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:02:27.258414  9016 leader_election.cc:290] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:02:27.324914  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 222489us
I20260709 13:02:27.327919  2299 leader_election.cc:304] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:02:27.353892  1717 leader_election.cc:304] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:27.329257  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:27.145979  9069 raft_consensus.cc:2749] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
W20260709 13:02:27.234139  2300 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:27.101475  8814 leader_election.cc:290] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:27.305678  9058 raft_consensus.cc:493] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:27.165750  9120 raft_consensus.cc:2804] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 13:02:27.358860  1676 raft_consensus.cc:2468] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 15.
I20260709 13:02:27.365414  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 17 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:27.215795  8797 raft_consensus.cc:493] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:27.306671  8952 raft_consensus.cc:493] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:27.380677  1753 raft_consensus.cc:1275] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 19 index: 13. Preceding OpId from leader: term: 21 index: 14. (index mismatch)
I20260709 13:02:27.336758  9012 raft_consensus.cc:2749] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 13:02:27.349738  8989 consensus_queue.cc:260] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:27.400177  1675 raft_consensus.cc:1275] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 18 index: 13. Preceding OpId from leader: term: 19 index: 14. (index mismatch)
I20260709 13:02:27.404121  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:27.405447  1747 raft_consensus.cc:2468] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 21.
I20260709 13:02:27.414106  1717 leader_election.cc:304] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:27.345963  9103 raft_consensus.cc:2804] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Leader election won for term 17
I20260709 13:02:27.385486  8968 raft_consensus.cc:2749] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:02:27.337287  9001 consensus_queue.cc:1059] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 19.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.072s
I20260709 13:02:27.387053  9067 raft_consensus.cc:2804] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 13:02:27.345585  8914 raft_consensus.cc:697] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 20 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:02:27.240204  9061 consensus_queue.cc:237] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.10, Last appended by leader: 10, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:27.396260  8817 raft_consensus.cc:2804] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:02:27.399864  9039 raft_consensus.cc:2804] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Leader election won for term 18
I20260709 13:02:27.330180  9116 raft_consensus.cc:515] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:27.414338  9120 raft_consensus.cc:493] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:27.370340  9101 raft_consensus.cc:515] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:02:27.472729  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 341235us
I20260709 13:02:27.423535  9058 raft_consensus.cc:3060] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Advancing to term 18
W20260709 13:02:27.472352  9035 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.570s	user 0.023s	sys 0.032s
W20260709 13:02:27.488758  9035 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.598s	user 0.023s	sys 0.032s
W20260709 13:02:27.458351  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 44386.9us
I20260709 13:02:27.425693  9115 leader_election.cc:290] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 16 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:27.396943  9072 consensus_queue.cc:1048] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.069s
I20260709 13:02:27.479939  1646 leader_election.cc:304] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:27.428669  8932 consensus_queue.cc:1059] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 17.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.057s
W20260709 13:02:27.482993  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 173544us
I20260709 13:02:27.489867  1717 leader_election.cc:304] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:27.464326  9136 raft_consensus.cc:2749] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:02:27.489907  1749 raft_consensus.cc:2468] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 22.
W20260709 13:02:27.502374  1672 consensus_meta.cc:287] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.533s	user 0.013s	sys 0.001s
I20260709 13:02:27.486514  1718 leader_election.cc:304] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:27.468976  9067 raft_consensus.cc:493] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:27.482648  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 19 candidate_status { last_received { term: 17 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:27.515458  1748 raft_consensus.cc:3060] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:02:27.479424  9103 raft_consensus.cc:697] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 17 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
W20260709 13:02:27.507119  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 166904us
I20260709 13:02:27.472522  9008 raft_consensus.cc:2749] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:02:27.511696  1717 leader_election.cc:304] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:27.509684  8989 raft_consensus.cc:3060] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:02:27.506918  9107 raft_consensus.cc:2804] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:02:27.474020  8817 raft_consensus.cc:493] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
W20260709 13:02:27.539320  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 392137us
I20260709 13:02:27.542997  2330 raft_consensus.cc:1275] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 16 index: 12. Preceding OpId from leader: term: 18 index: 13. (index mismatch)
I20260709 13:02:27.478307  8926 consensus_queue.cc:237] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.10, Last appended by leader: 10, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:27.551246  9096 leader_election.cc:290] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 18 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:27.551980  9096 raft_consensus.cc:2749] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20260709 13:02:27.552779  9132 log.cc:927] Time spent T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.357s	user 0.000s	sys 0.001s
I20260709 13:02:27.553704  9065 raft_consensus.cc:3060] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:02:27.555073  1746 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55840 (request call id 3135) took 1526 ms. Trace:
I20260709 13:02:27.555476  1746 rpcz_store.cc:276] 0709 13:02:26.028375 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:26.236486 (+208111us) service_pool.cc:224] Handling call
0709 13:02:26.236880 (+   394us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:26.237342 (+   462us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 13:02:26.616765 (+379423us) log.cc:844] Serialized 14 byte log entry
0709 13:02:26.976462 (+359697us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:27.018279 (+ 41817us) log.cc:844] Serialized 26 byte log entry
0709 13:02:27.018536 (+   257us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:02:27.021389 (+  2853us) log.cc:844] Serialized 14 byte log entry
0709 13:02:27.021667 (+   278us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:27.021771 (+   104us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:27.554827 (+533056us) raft_consensus.cc:1685] finished
0709 13:02:27.554848 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:27.554993 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":85134,"prepare.run_cpu_time_us":87,"prepare.run_wall_time_us":91,"raft.queue_time_us":300,"raft.run_cpu_time_us":371,"raft.run_wall_time_us":18157,"thread_start_us":626830,"threads_started":2,"wal-append.queue_time_us":430084}
W20260709 13:02:27.557278  9134 log.cc:927] Time spent T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.157s	user 0.001s	sys 0.000s
I20260709 13:02:27.535539  9136 raft_consensus.cc:2749] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:02:27.559576  9101 leader_election.cc:290] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:27.513840  1672 raft_consensus.cc:2468] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 22.
I20260709 13:02:27.484434  9120 raft_consensus.cc:3060] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:02:27.452275  8797 raft_consensus.cc:515] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:27.485442  8968 raft_consensus.cc:493] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:27.456609  8952 raft_consensus.cc:515] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:27.500691  9039 raft_consensus.cc:697] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:02:27.522382  1571 leader_election.cc:304] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:02:27.522655  1749 raft_consensus.cc:1275] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 18 index: 13. Preceding OpId from leader: term: 19 index: 14. (index mismatch)
I20260709 13:02:27.503379  8880 leader_election.cc:290] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:27.599869  8797 leader_election.cc:290] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:27.533372  1717 leader_election.cc:304] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:27.600553  1751 raft_consensus.cc:2468] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 21.
W20260709 13:02:27.601078  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 67831.9us
W20260709 13:02:27.602092  9142 log.cc:927] Time spent T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.140s	user 0.000s	sys 0.001s
I20260709 13:02:27.601990  1717 leader_election.cc:304] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 13:02:27.603384  1717 leader_election.cc:304] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:27.512351  1643 leader_election.cc:304] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
W20260709 13:02:27.604635  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 92433.5us
I20260709 13:02:27.539062  8817 raft_consensus.cc:3060] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Advancing to term 20
W20260709 13:02:27.611527  9144 log.cc:927] Time spent T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.143s	user 0.000s	sys 0.000s
I20260709 13:02:27.572036  9069 raft_consensus.cc:493] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:27.613137  9069 raft_consensus.cc:515] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:27.536262  9140 raft_consensus.cc:493] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:27.614770  9140 raft_consensus.cc:515] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:27.615465  9069 leader_election.cc:290] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:27.617131  9140 leader_election.cc:290] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:27.536607  9107 raft_consensus.cc:493] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:27.539664  8932 raft_consensus.cc:2804] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:02:27.517297  9067 raft_consensus.cc:3060] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:02:27.581539  1748 raft_consensus.cc:2468] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 19.
W20260709 13:02:27.589720  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 103383us
I20260709 13:02:27.599191  9039 consensus_queue.cc:237] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 17.15, Last appended by leader: 15, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:27.605343  1643 leader_election.cc:304] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1
W20260709 13:02:27.655507  2300 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:27.621768  8932 raft_consensus.cc:493] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:27.660063  9157 raft_consensus.cc:2749] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:02:27.624584  8797 raft_consensus.cc:2749] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:02:27.660128  8932 raft_consensus.cc:3060] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:02:27.664268  1597 raft_consensus.cc:1240] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 17. Current term is 18. Ops: []
W20260709 13:02:27.671866  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 149643us
I20260709 13:02:27.687487  1571 leader_election.cc:304] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:02:27.644806  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:27.626664  2299 leader_election.cc:304] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 13:02:27.691299  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 64805.1us
I20260709 13:02:27.692484  2299 leader_election.cc:304] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:27.693753  9072 raft_consensus.cc:2749] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:02:27.703927  8880 raft_consensus.cc:2804] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Leader election won for term 17
I20260709 13:02:27.704736  8880 raft_consensus.cc:697] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 17 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
W20260709 13:02:27.707249  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 102044us
I20260709 13:02:27.709306  9072 consensus_queue.cc:1048] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 13:02:27.706064  8880 consensus_queue.cc:237] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:27.724057  9072 raft_consensus.cc:493] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:27.724639  9072 raft_consensus.cc:515] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:27.728376  9072 leader_election.cc:290] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:27.731240  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 20 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:27.629904  1717 leader_election.cc:304] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:27.665241  1718 leader_election.cc:304] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
W20260709 13:02:27.742414  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 55089.7us
I20260709 13:02:27.747427  1571 leader_election.cc:304] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1
W20260709 13:02:27.740123  1643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:27.747464  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 202888us
W20260709 13:02:27.751086  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 85978.6us
I20260709 13:02:27.752400  1718 leader_election.cc:400] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 19 pre-election: Vote denied by peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625) with higher term. Message: Invalid argument: T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 19. Current term is 20.
I20260709 13:02:27.752748  1718 leader_election.cc:403] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 19 pre-election: Cancelling election due to peer responding with higher term
W20260709 13:02:27.600871  9141 log.cc:927] Time spent T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.122s	user 0.001s	sys 0.001s
I20260709 13:02:27.753645  1718 leader_election.cc:304] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 13:02:27.758873  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 129114us
I20260709 13:02:27.763038  1717 leader_election.cc:400] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 19 pre-election: Vote denied by peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445) with higher term. Message: Invalid argument: T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 19. Current term is 20.
I20260709 13:02:27.763760  1571 leader_election.cc:304] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:02:27.759711  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 20 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:27.764868  1597 raft_consensus.cc:2410] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 for term 20 because replica has last-logged OpId of term: 18 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 14.
I20260709 13:02:27.766320  1717 leader_election.cc:304] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:27.658516  9136 raft_consensus.cc:2749] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:02:27.754422  1569 leader_election.cc:304] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:02:27.658850  9012 raft_consensus.cc:2804] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:02:27.775493  8981 raft_consensus.cc:697] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:02:27.772646  1571 leader_election.cc:304] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:02:27.741663  9140 raft_consensus.cc:3060] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:02:27.750466  9107 raft_consensus.cc:3060] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:02:27.774196  1569 leader_election.cc:304] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:02:27.657300  8893 raft_consensus.cc:493] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:27.786966  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:27.746343  9150 raft_consensus.cc:2749] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:02:27.740038  8994 raft_consensus.cc:2749] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:02:27.800997  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:27.770483  9096 raft_consensus.cc:2749] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:02:27.754793  9101 raft_consensus.cc:2749] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:02:27.596521  8968 raft_consensus.cc:515] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:27.770229  9157 raft_consensus.cc:2749] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:02:27.770273  9115 raft_consensus.cc:493] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:27.542680  8914 consensus_queue.cc:237] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.14, Last appended by leader: 14, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:27.777642  9008 raft_consensus.cc:2804] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:02:27.866885  9008 raft_consensus.cc:493] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:27.867327  9008 raft_consensus.cc:3060] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:02:27.867801  8634 leader_election.cc:290] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 15 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:27.773032  9035 raft_consensus.cc:2749] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 13:02:27.808262  9116 leader_election.cc:290] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:27.776516  9152 raft_consensus.cc:493] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:27.810567  8981 consensus_queue.cc:237] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 20.11, Last appended by leader: 11, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:27.811030  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:27.799465  9136 consensus_queue.cc:1059] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 20.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.012s
I20260709 13:02:27.813803  8764 raft_consensus.cc:3055] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 13:02:27.814960  9154 raft_consensus.cc:2749] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Leader pre-election lost for term 19. Reason: Vote denied by peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625) with higher term. Message: Invalid argument: T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 19. Current term is 20.
W20260709 13:02:27.815511  9143 log.cc:927] Time spent T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.235s	user 0.000s	sys 0.001s
I20260709 13:02:27.773656  9012 raft_consensus.cc:493] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:27.819773  9096 raft_consensus.cc:2749] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
W20260709 13:02:27.826983  9147 log.cc:927] Time spent T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.278s	user 0.000s	sys 0.001s
I20260709 13:02:27.888950  9012 raft_consensus.cc:3060] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:02:27.832003  8932 raft_consensus.cc:515] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:27.833316  1754 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55848 (request call id 3322) took 1062 ms. Trace:
I20260709 13:02:27.835029  9161 consensus_queue.cc:1048] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 13:02:27.894356  1754 rpcz_store.cc:276] 0709 13:02:26.771071 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:26.771202 (+   131us) service_pool.cc:224] Handling call
0709 13:02:26.771488 (+   286us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:26.871448 (+ 99960us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:27.833011 (+961563us) spinlock_profiling.cc:243] Waited 99.2 ms on lock 0x7b54001d60d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:02:27.833240 (+   229us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":228237056}
W20260709 13:02:27.836535  9104 consensus_meta.cc:287] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.760s	user 0.014s	sys 0.000s
I20260709 13:02:27.896607  9104 raft_consensus.cc:515] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:02:27.841984  1644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:27.802917  8926 raft_consensus.cc:2804] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Leader election won for term 16
I20260709 13:02:27.843793  2299 leader_election.cc:304] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 13:02:27.901228  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 57621us
I20260709 13:02:27.885545  8764 raft_consensus.cc:740] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
W20260709 13:02:27.857656  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 85601.6us
I20260709 13:02:27.871385  9115 raft_consensus.cc:515] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:02:27.902695  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 136528us
I20260709 13:02:27.904140  1571 leader_election.cc:304] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 13:02:27.860126  8817 raft_consensus.cc:515] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:27.906396  8817 leader_election.cc:290] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 20 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:27.861128  8968 leader_election.cc:290] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:27.789935  9124 leader_election.cc:290] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:02:27.870646  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 102655us
W20260709 13:02:27.854316  9080 consensus_meta.cc:287] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.793s	user 0.008s	sys 0.007s
I20260709 13:02:27.909637  9080 raft_consensus.cc:515] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:27.910655  8926 raft_consensus.cc:697] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:02:27.911945  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:27.911765  8926 consensus_queue.cc:237] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 14.13, Last appended by leader: 13, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:27.914829  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 22 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:02:27.877228  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 96030us
I20260709 13:02:27.915552  1676 raft_consensus.cc:2468] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 21.
I20260709 13:02:27.813831  9153 raft_consensus.cc:2804] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 24 FOLLOWER]: Leader election won for term 24
I20260709 13:02:27.773769  9103 consensus_queue.cc:237] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.9, Last appended by leader: 9, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:27.897650  8932 leader_election.cc:290] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 20 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:27.799472  8893 raft_consensus.cc:3060] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Advancing to term 24
I20260709 13:02:27.922559  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:27.917285  2333 raft_consensus.cc:2393] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 21: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:02:27.919328  1571 leader_election.cc:304] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:02:27.920898  2298 leader_election.cc:304] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:27.925927  1676 raft_consensus.cc:2393] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 21: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:02:27.926920  1718 leader_election.cc:304] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:02:27.928385  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:27.925756  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:27.929069  1571 leader_election.cc:304] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:02:27.931396  8968 raft_consensus.cc:2749] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:02:27.932248  1718 leader_election.cc:304] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:27.933115  8968 consensus_queue.cc:1059] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 20.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20260709 13:02:27.933911  8968 consensus_queue.cc:1059] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 19.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 13:02:27.934739  9096 raft_consensus.cc:2804] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 13:02:27.934904  8968 consensus_queue.cc:1048] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 13:02:27.935266  9096 raft_consensus.cc:697] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 20 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:02:27.936249  9096 consensus_queue.cc:237] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.12, Last appended by leader: 12, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:27.941380  1718 leader_election.cc:304] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:27.936712  8968 raft_consensus.cc:3055] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 13:02:27.942525  8968 raft_consensus.cc:740] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:02:27.943819  8932 raft_consensus.cc:2804] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 13:02:27.944360  8932 raft_consensus.cc:493] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:27.944864  8932 raft_consensus.cc:3060] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:02:27.954053  8968 consensus_queue.cc:260] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.13, Last appended by leader: 13, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:27.961591  8968 raft_consensus.cc:3060] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Advancing to term 20
W20260709 13:02:27.916441  9054 consensus_meta.cc:287] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.575s	user 0.014s	sys 0.000s
I20260709 13:02:27.962611  2336 raft_consensus.cc:2393] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 15: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:02:27.916538  8994 raft_consensus.cc:2804] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 13:02:27.929033  2329 raft_consensus.cc:2468] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 24.
W20260709 13:02:27.924293  9088 consensus_meta.cc:287] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.836s	user 0.015s	sys 0.000s
I20260709 13:02:27.966420  1571 leader_election.cc:304] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:02:27.963948  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:27.925376  9153 raft_consensus.cc:697] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 24 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
W20260709 13:02:27.969080  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 48309.3us
I20260709 13:02:27.924364  8817 raft_consensus.cc:2804] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Leader election won for term 18
I20260709 13:02:27.968923  9039 raft_consensus.cc:2749] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:02:27.978974  9039 raft_consensus.cc:3060] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:02:27.967815  9035 raft_consensus.cc:2804] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader pre-election won for term 16
W20260709 13:02:27.971553  8771 consensus_meta.cc:287] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.902s	user 0.012s	sys 0.002s
I20260709 13:02:27.962972  8952 leader_election.cc:290] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:27.980610  1571 leader_election.cc:304] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:02:27.969987  9080 leader_election.cc:290] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 21 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:27.981195  8764 consensus_queue.cc:260] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.10, Last appended by leader: 10, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:28.007764  9002 raft_consensus.cc:3055] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 13:02:28.008046  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:28.008232  9002 raft_consensus.cc:740] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:02:28.009274  9002 consensus_queue.cc:260] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.14, Last appended by leader: 14, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:28.010063  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 20 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:28.010757  1747 raft_consensus.cc:2376] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 20. Re-sending same reply.
I20260709 13:02:28.013830  1677 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52766 (request call id 3254) took 1232 ms. Trace:
I20260709 13:02:27.984740  8817 raft_consensus.cc:697] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:02:27.886989  9152 raft_consensus.cc:515] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:28.019683  1745 raft_consensus.cc:1275] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 19 index: 14. (index mismatch)
I20260709 13:02:27.996897  9035 raft_consensus.cc:493] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:28.019935  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:28.034008  1747 raft_consensus.cc:2393] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 18: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:02:27.981856  8994 raft_consensus.cc:697] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 22 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:02:28.021509  9172 raft_consensus.cc:2749] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:02:28.022526  8952 raft_consensus.cc:493] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:28.018613  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:27.971345  9161 consensus_queue.cc:1048] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 11, Time since last communication: 0.030s
I20260709 13:02:28.035413  8952 raft_consensus.cc:515] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:28.036128  9161 consensus_queue.cc:1243] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer af520b67945d443b901bbeca37cc5701 log is divergent from this leader: its last log entry 18.13 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 11
I20260709 13:02:28.028105  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:28.036935  1603 raft_consensus.cc:2468] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 17.
I20260709 13:02:28.037910  8952 leader_election.cc:290] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:28.027899  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 22 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:28.029629  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:28.038743  1604 raft_consensus.cc:2468] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 21.
I20260709 13:02:28.039803  1604 raft_consensus.cc:1275] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 16 index: 8. Preceding OpId from leader: term: 18 index: 9. (index mismatch)
I20260709 13:02:28.040867  1604 raft_consensus.cc:1240] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 20. Current term is 21. Ops: [20.10-20.10]
I20260709 13:02:28.042287  1604 raft_consensus.cc:1240] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 19. Current term is 20. Ops: []
I20260709 13:02:28.030357  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:28.043397  1749 raft_consensus.cc:1275] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 16 index: 12. Preceding OpId from leader: term: 18 index: 13. (index mismatch)
I20260709 13:02:28.014582  9174 raft_consensus.cc:3055] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 13:02:28.015028  9002 raft_consensus.cc:3060] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:02:28.014245  1677 rpcz_store.cc:276] 0709 13:02:26.781478 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:26.781640 (+   162us) service_pool.cc:224] Handling call
0709 13:02:28.013594 (+1231954us) spinlock_profiling.cc:243] Waited 139 ms on lock 0x7b5400058cd4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:02:28.013737 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":319605504}
I20260709 13:02:28.054196  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:28.060516  1605 raft_consensus.cc:3060] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:02:28.054246  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:28.061568  1598 raft_consensus.cc:3060] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:02:28.059338  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:28.038041  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:27.972707  9054 raft_consensus.cc:515] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:28.029481  9153 consensus_queue.cc:237] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 23.15, Last appended by leader: 15, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:28.001595  8771 raft_consensus.cc:2749] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:02:28.002707  9159 raft_consensus.cc:2804] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Leader pre-election won for term 20
W20260709 13:02:28.070392  1750 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55848 (request call id 3314) took 1545 ms (client timeout 2046 ms). Trace:
I20260709 13:02:28.070744  9159 raft_consensus.cc:493] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Starting leader election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:02:28.071233  9159 raft_consensus.cc:3060] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:02:28.072357  1746 raft_consensus.cc:1275] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 20 index: 15. Preceding OpId from leader: term: 21 index: 16. (index mismatch)
I20260709 13:02:28.053808  9172 raft_consensus.cc:493] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:28.075925  2333 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39726 (request call id 1641) took 1109 ms. Trace:
I20260709 13:02:28.076386  2333 rpcz_store.cc:276] 0709 13:02:26.966896 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:26.967026 (+   130us) service_pool.cc:224] Handling call
0709 13:02:28.075692 (+1108666us) spinlock_profiling.cc:243] Waited 896 ms on lock 0x7b540045b5d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:02:28.075849 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2060297984}
I20260709 13:02:28.076406  9172 raft_consensus.cc:515] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:28.062781  1603 raft_consensus.cc:2468] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 22.
I20260709 13:02:28.045234  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:28.035343  8994 consensus_queue.cc:237] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 21.16, Last appended by leader: 16, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:28.051105  1675 raft_consensus.cc:1240] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 20 LEADER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 19. Current term is 20. Ops: []
I20260709 13:02:28.062665  1600 raft_consensus.cc:2468] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 17.
I20260709 13:02:28.067389  9054 leader_election.cc:290] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 21 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:28.058878  8817 consensus_queue.cc:237] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:28.069075  2328 raft_consensus.cc:2393] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 21: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
W20260709 13:02:28.045293  9156 log.cc:927] Time spent T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.173s	user 0.000s	sys 0.001s
I20260709 13:02:28.090288  1596 raft_consensus.cc:1240] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 19. Current term is 20. Ops: []
W20260709 13:02:28.070878  1750 rpcz_store.cc:269] 0709 13:02:26.524654 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:26.527030 (+  2376us) service_pool.cc:224] Handling call
0709 13:02:28.070151 (+1543121us) spinlock_profiling.cc:243] Waited 580 ms on lock 0x7b54003237d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:02:28.070298 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1334192896}
I20260709 13:02:27.979979  9088 raft_consensus.cc:515] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:28.055250  9167 log.cc:927] Time spent T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.153s	user 0.001s	sys 0.000s
I20260709 13:02:28.043179  9035 raft_consensus.cc:3060] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:02:28.107200  1752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43156 (request call id 1506) took 1012 ms. Trace:
I20260709 13:02:28.045516  9174 raft_consensus.cc:740] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:02:28.122694  1752 rpcz_store.cc:276] 0709 13:02:27.094892 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:27.134485 (+ 39593us) service_pool.cc:224] Handling call
0709 13:02:27.159831 (+ 25346us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:27.378539 (+218708us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:27.378577 (+    38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:27.409832 (+ 31255us) log.cc:844] Serialized 26 byte log entry
0709 13:02:27.582460 (+172628us) raft_consensus.cc:1645] Marking committed up to 16
0709 13:02:27.768999 (+186539us) log.cc:844] Serialized 14 byte log entry
0709 13:02:27.873742 (+104743us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:28.093644 (+219902us) spinlock_profiling.cc:243] Waited 182 ms on lock 0x7b5400220454. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:02:28.093699 (+    55us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:28.093751 (+    52us) raft_consensus.cc:1685] finished
0709 13:02:28.106886 (+ 13135us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:28.107111 (+   225us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":11532,"prepare.queue_time_us":222722,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":50,"raft.queue_time_us":85372,"raft.run_cpu_time_us":248,"raft.run_wall_time_us":12383,"spinlock_wait_cycles":418909056,"thread_start_us":385610,"threads_started":3,"wal-append.queue_time_us":200347}
I20260709 13:02:28.093372  9136 raft_consensus.cc:493] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:28.119647  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 322340us
I20260709 13:02:28.131254  1679 raft_consensus.cc:2393] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 22: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:02:28.131834  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:28.140164  1603 raft_consensus.cc:1275] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 15 index: 11. Preceding OpId from leader: term: 19 index: 13. (index mismatch)
I20260709 13:02:28.140486  1598 raft_consensus.cc:2468] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 15.
I20260709 13:02:28.140504  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:28.141302  1604 raft_consensus.cc:1240] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 14. Current term is 15. Ops: []
I20260709 13:02:28.146876  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 19 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:28.148638  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:28.149135  1605 raft_consensus.cc:2468] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 18.
I20260709 13:02:28.149354  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:28.149915  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 21 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:28.150823  1678 raft_consensus.cc:2410] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for term 21 because replica has last-logged OpId of term: 20 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 9.
I20260709 13:02:28.125946  9148 raft_consensus.cc:493] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:28.155205  8893 raft_consensus.cc:515] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:28.156471  9088 leader_election.cc:290] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 22 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:28.157778  9174 consensus_queue.cc:260] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.15, Last appended by leader: 15, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:28.158982  9152 leader_election.cc:290] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:28.159448  9180 raft_consensus.cc:3060] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:02:28.172362  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:28.139813  8952 raft_consensus.cc:493] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:28.192951  9008 raft_consensus.cc:515] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:28.195622  9008 leader_election.cc:290] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 16 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:02:28.205703  9135 log.cc:927] Time spent T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.868s	user 0.007s	sys 0.009s
I20260709 13:02:28.207932  1673 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57092 (request call id 1628) took 2027 ms. Trace:
I20260709 13:02:28.208395  1673 rpcz_store.cc:276] 0709 13:02:26.180653 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:26.182428 (+  1775us) service_pool.cc:224] Handling call
0709 13:02:26.182856 (+   428us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:26.639988 (+457132us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:26.640025 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:26.648656 (+  8631us) log.cc:844] Serialized 26 byte log entry
0709 13:02:27.021012 (+372356us) raft_consensus.cc:1645] Marking committed up to 14
0709 13:02:27.021051 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:27.021135 (+    84us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:28.204363 (+1183228us) spinlock_profiling.cc:243] Waited 7.62 ms on lock 0x7b4400022b4c. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f4262f9bc85 00007f4262f9c6a2 00007f4265d0e1db 00007f4265d09aaf 00007f4265cfd86e 00007f4265cfcbb6 00007f4265cfc7d8 00007f4265d06221 00007f4265d061d9 00007f4265d06169
0709 13:02:28.207467 (+  3104us) raft_consensus.cc:1685] finished
0709 13:02:28.207504 (+    37us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:28.207846 (+   342us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":186684,"raft.queue_time_us":146536,"raft.run_cpu_time_us":227,"raft.run_wall_time_us":29445,"spinlock_wait_cycles":17516928,"thread_start_us":683143,"threads_started":2,"wal-append.queue_time_us":463150}
I20260709 13:02:28.213742  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:28.216375  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 22 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:28.151568  9115 leader_election.cc:290] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:28.223721  1675 raft_consensus.cc:1275] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 17 index: 10. Preceding OpId from leader: term: 20 index: 11. (index mismatch)
I20260709 13:02:28.165409  9111 leader_election.cc:290] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:28.228792  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:28.229442  1673 raft_consensus.cc:3060] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:02:28.221549  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:28.132827  9136 raft_consensus.cc:515] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:28.182817  9072 raft_consensus.cc:493] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:28.184274  8952 raft_consensus.cc:515] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:28.235107  8932 raft_consensus.cc:515] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:28.226336  9012 raft_consensus.cc:515] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:28.263046  9172 leader_election.cc:290] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:28.265818  9072 raft_consensus.cc:515] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:28.175163  9148 raft_consensus.cc:515] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:28.166533  9133 consensus_queue.cc:1048] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.064s
W20260709 13:02:28.292697  1717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:28.294339  1601 raft_consensus.cc:1240] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 17. Current term is 18. Ops: []
I20260709 13:02:28.438468  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:28.442770  1672 raft_consensus.cc:2393] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 21: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:02:28.440361  1673 raft_consensus.cc:2468] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 16.
W20260709 13:02:28.445032  2328 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39726 (request call id 1639) took 1479 ms (client timeout 1829 ms). Trace:
W20260709 13:02:28.378412  9058 consensus_meta.cc:287] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.890s	user 0.012s	sys 0.004s
W20260709 13:02:28.363605  1644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:28.413600  1605 raft_consensus.cc:2468] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 19.
W20260709 13:02:28.418694  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 121074us
W20260709 13:02:28.445890  2328 rpcz_store.cc:269] 0709 13:02:26.966064 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:26.966207 (+   143us) service_pool.cc:224] Handling call
0709 13:02:28.296720 (+1330513us) spinlock_profiling.cc:243] Waited 1.04 s on lock 0x7b540065e0d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:02:28.444956 (+148236us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2387575552}
W20260709 13:02:28.467195  2299 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:28.473912  2299 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:28.469705  1746 raft_consensus.cc:1240] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 19. Current term is 20. Ops: []
I20260709 13:02:28.470099  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:28.475979  1749 raft_consensus.cc:2393] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 21: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:02:28.470744  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:28.476670  1747 raft_consensus.cc:2393] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 21: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:02:28.476891  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:02:28.445559  9067 consensus_meta.cc:287] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.789s	user 0.012s	sys 0.004s
I20260709 13:02:28.480834  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:28.481544  1749 raft_consensus.cc:2468] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 21.
I20260709 13:02:28.477849  1745 raft_consensus.cc:2468] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 14.
W20260709 13:02:28.445271  9140 consensus_meta.cc:287] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.647s	user 0.010s	sys 0.005s
I20260709 13:02:28.478897  1753 raft_consensus.cc:1240] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 17. Current term is 18. Ops: []
I20260709 13:02:28.384991  9174 raft_consensus.cc:3060] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:02:28.483119  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:28.483673  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:28.485733  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:28.498970  1751 raft_consensus.cc:2468] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 17.
I20260709 13:02:28.499028  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:28.486485  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:28.500190  1747 raft_consensus.cc:2393] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 22: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:02:28.500010  1750 raft_consensus.cc:2468] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 19.
I20260709 13:02:28.488401  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 22 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:28.489456  8921 raft_consensus.cc:493] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:28.490110  9002 raft_consensus.cc:493] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:28.505797  1748 raft_consensus.cc:1275] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 19 index: 13. (index mismatch)
W20260709 13:02:28.477088  8968 consensus_meta.cc:287] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.500s	user 0.013s	sys 0.002s
I20260709 13:02:28.511709  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:28.507484  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 19 candidate_status { last_received { term: 17 index: 8 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:28.512490  1601 raft_consensus.cc:3060] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:02:28.444314  9015 raft_consensus.cc:493] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:28.519898  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:28.520419  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:28.490664  9069 raft_consensus.cc:493] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:28.510366  1753 raft_consensus.cc:1240] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 17. Current term is 18. Ops: []
I20260709 13:02:28.512516  1602 raft_consensus.cc:2368] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for earlier term 15. Current term is 16.
I20260709 13:02:28.521097  1746 raft_consensus.cc:2468] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 23.
I20260709 13:02:28.527000  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 23 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:28.510214  9140 raft_consensus.cc:2749] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:02:28.536353  1604 raft_consensus.cc:1275] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 16 index: 13. (index mismatch)
W20260709 13:02:28.531666  9184 log.cc:927] Time spent T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.090s	user 0.000s	sys 0.004s
I20260709 13:02:28.546123  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:28.480358  9058 raft_consensus.cc:515] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:28.553351  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:28.506649  9067 raft_consensus.cc:515] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:28.558365  1604 raft_consensus.cc:2468] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 23.
I20260709 13:02:28.560910  1601 raft_consensus.cc:2468] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 19.
I20260709 13:02:28.513706  8921 raft_consensus.cc:515] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:28.515486  9002 raft_consensus.cc:515] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:28.571982  1603 raft_consensus.cc:1240] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 17. Current term is 18. Ops: []
I20260709 13:02:28.581789  1674 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57092 (request call id 1640) took 1217 ms. Trace:
I20260709 13:02:28.582422  1674 rpcz_store.cc:276] 0709 13:02:27.364171 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:27.379335 (+ 15164us) service_pool.cc:224] Handling call
0709 13:02:27.410498 (+ 31163us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:28.167752 (+757254us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:28.167801 (+    49us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:28.167860 (+    59us) raft_consensus.cc:1645] Marking committed up to 16
0709 13:02:28.167910 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:28.581440 (+413530us) spinlock_profiling.cc:243] Waited 757 ms on lock 0x7b5400164254. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:02:28.581516 (+    76us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:28.581698 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1741750144}
I20260709 13:02:28.583315  1746 raft_consensus.cc:1217] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Deduplicated request from leader. Original: 12.11->[15.12-19.14]   Dedup: 15.12->[17.13-19.14]
I20260709 13:02:28.583969  1746 raft_consensus.cc:1240] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 19. Current term is 20. Ops: [15.12-19.14]
I20260709 13:02:28.585237  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 19 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:28.586886  9171 raft_consensus.cc:493] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:28.587518  9171 raft_consensus.cc:515] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:28.594573  9183 log.cc:927] Time spent T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.117s	user 0.000s	sys 0.005s
I20260709 13:02:28.534922  9069 raft_consensus.cc:515] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:28.614276  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 171889us
I20260709 13:02:28.609157  1717 leader_election.cc:304] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:02:28.529182  9015 raft_consensus.cc:515] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:28.616032  1646 leader_election.cc:304] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:28.614349  8893 leader_election.cc:290] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 24 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:28.620524  8772 consensus_queue.cc:1048] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 13:02:28.625183  1646 leader_election.cc:304] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 13:02:28.627599  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:28.627771  1717 leader_election.cc:304] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:28.573369  8932 leader_election.cc:290] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 22 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:28.597384  8952 leader_election.cc:290] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:28.642086  1753 raft_consensus.cc:1275] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 17 index: 10. Preceding OpId from leader: term: 20 index: 11. (index mismatch)
I20260709 13:02:28.675338  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:28.597229  9172 raft_consensus.cc:493] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:02:28.677707  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:28.678395  2332 raft_consensus.cc:2468] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 24.
I20260709 13:02:28.678617  1571 leader_election.cc:304] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:02:28.674386  1749 raft_consensus.cc:3060] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:02:28.629102  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:28.641957  1601 raft_consensus.cc:2393] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 22: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:02:28.677023  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:28.677244  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:28.681013  2328 raft_consensus.cc:2468] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 19.
W20260709 13:02:28.679530  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 48716.2us
I20260709 13:02:28.682878  1717 leader_election.cc:304] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:28.684422  1571 leader_election.cc:304] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:02:28.688407  1571 leader_election.cc:304] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:02:28.691478  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:28.621371  9011 raft_consensus.cc:2804] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Leader election won for term 21
I20260709 13:02:28.700587  1717 leader_election.cc:304] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:02:28.628576  9111 raft_consensus.cc:2804] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:02:28.678834  8968 raft_consensus.cc:3060] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:02:28.735634  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:28.710585  1571 leader_election.cc:304] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:02:28.722806  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 22 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:28.744783  1605 raft_consensus.cc:3060] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:02:28.709017  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:28.746923  1597 raft_consensus.cc:2393] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 19: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:02:28.626176  9104 leader_election.cc:290] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 23 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:28.687412  9161 raft_consensus.cc:2749] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:02:28.721657  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:02:28.716424  9120 consensus_meta.cc:287] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 1.120s	user 0.010s	sys 0.006s
I20260709 13:02:28.628602  9088 consensus_queue.cc:1048] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.091s
I20260709 13:02:28.736575  1601 raft_consensus.cc:3060] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Advancing to term 23
W20260709 13:02:28.722896  9159 consensus_meta.cc:287] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.651s	user 0.014s	sys 0.000s
I20260709 13:02:28.700951  8952 raft_consensus.cc:2749] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 13:02:28.751785  1750 raft_consensus.cc:1275] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 20 index: 11. Preceding OpId from leader: term: 22 index: 12. (index mismatch)
I20260709 13:02:28.747393  1749 raft_consensus.cc:1275] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 17 index: 15. Preceding OpId from leader: term: 18 index: 16. (index mismatch)
I20260709 13:02:28.738827  9111 raft_consensus.cc:493] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:28.696044  8771 consensus_queue.cc:1048] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.056s
W20260709 13:02:28.751626  9185 log.cc:927] Time spent T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.304s	user 0.000s	sys 0.001s
W20260709 13:02:28.750075  9039 consensus_meta.cc:287] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.767s	user 0.018s	sys 0.022s
I20260709 13:02:27.952181   431 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:02:28.734917  9012 leader_election.cc:290] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 18 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:28.687799  9172 raft_consensus.cc:515] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:28.774720  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 293145us
W20260709 13:02:28.752564  9107 consensus_meta.cc:287] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.952s	user 0.010s	sys 0.012s
I20260709 13:02:28.706193  9140 consensus_queue.cc:1048] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.058s
I20260709 13:02:28.697598  9096 consensus_queue.cc:1048] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.055s
I20260709 13:02:28.763104  9072 leader_election.cc:290] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:28.764575  9104 raft_consensus.cc:493] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:28.839147  2301 leader_election.cc:304] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:02:28.821825  9111 raft_consensus.cc:3060] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:02:28.829025  9169 raft_consensus.cc:2749] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 13:02:28.828047  8772 raft_consensus.cc:2804] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Leader election won for term 21
I20260709 13:02:28.862640  2301 leader_election.cc:304] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:02:28.847123  8952 raft_consensus.cc:2749] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:02:28.847599  9039 raft_consensus.cc:2749] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:02:28.880221  1601 raft_consensus.cc:2468] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 23.
W20260709 13:02:28.860952  9187 log.cc:927] Time spent T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.385s	user 0.002s	sys 0.000s
I20260709 13:02:28.852895  9065 raft_consensus.cc:493] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:28.883517  9193 log.cc:927] Time spent T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.133s	user 0.000s	sys 0.001s
W20260709 13:02:28.884500  9188 log.cc:927] Time spent T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.134s	user 0.000s	sys 0.000s
I20260709 13:02:28.774082  9161 consensus_queue.cc:1059] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.093s
I20260709 13:02:28.912729  9161 consensus_queue.cc:1243] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer af520b67945d443b901bbeca37cc5701 log is divergent from this leader: its last log entry 18.13 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 11
I20260709 13:02:28.767922  9120 raft_consensus.cc:515] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:28.882339  2301 leader_election.cc:304] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:02:28.916662  1718 leader_election.cc:304] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:28.918558  1602 raft_consensus.cc:2468] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 17.
I20260709 13:02:28.920241  2301 leader_election.cc:304] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:28.883260  9072 raft_consensus.cc:2749] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 13:02:28.852514  9148 leader_election.cc:290] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:02:28.901546  2299 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:28.878651  9152 raft_consensus.cc:493] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:28.906163  9133 raft_consensus.cc:493] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:28.900557  9067 leader_election.cc:290] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 22 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:28.872627  8934 raft_consensus.cc:2804] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 13:02:28.773267  9159 raft_consensus.cc:515] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:02:28.888552  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 188235us
I20260709 13:02:28.860689  9196 raft_consensus.cc:2749] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:02:28.870811  9012 raft_consensus.cc:2804] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:02:28.926142  1605 raft_consensus.cc:2468] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 22.
I20260709 13:02:28.855918  9107 raft_consensus.cc:515] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:02:28.890512  9189 log.cc:927] Time spent T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.349s	user 0.002s	sys 0.000s
I20260709 13:02:28.894723  8772 raft_consensus.cc:697] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
W20260709 13:02:28.932368  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 221974us
I20260709 13:02:28.932117  1718 leader_election.cc:304] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:28.932096  9011 raft_consensus.cc:697] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:02:28.913805  9058 leader_election.cc:290] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 18 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:28.918195  9065 raft_consensus.cc:515] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:28.940173  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 315222us
I20260709 13:02:28.948983  1600 raft_consensus.cc:2468] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 20.
I20260709 13:02:28.951053  9199 raft_consensus.cc:2749] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 13:02:28.953871  1646 leader_election.cc:304] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:28.954097  8994 raft_consensus.cc:2804] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:02:28.954706  8994 raft_consensus.cc:493] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:28.954852  9148 raft_consensus.cc:2804] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Leader election won for term 19
I20260709 13:02:28.955253  8994 raft_consensus.cc:3060] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:02:28.955539  9148 raft_consensus.cc:697] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:02:28.956687  9148 consensus_queue.cc:237] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 17.8, Last appended by leader: 8, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:28.959239  8772 consensus_queue.cc:237] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.13, Last appended by leader: 13, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:28.960768  9200 raft_consensus.cc:2804] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:02:28.961403  9200 raft_consensus.cc:493] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:28.961905  9200 raft_consensus.cc:3060] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:02:28.969367  8797 consensus_queue.cc:1059] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 14.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 13:02:28.970620  8797 raft_consensus.cc:3055] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:02:28.937911  1717 leader_election.cc:304] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:02:28.867288  9104 raft_consensus.cc:515] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:28.972433  1717 leader_election.cc:304] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:02:28.973182  9104 leader_election.cc:290] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:28.973608  9159 leader_election.cc:290] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 20 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:28.973836  9104 raft_consensus.cc:2804] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 13:02:28.935174  9002 leader_election.cc:290] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:28.943342  8921 leader_election.cc:290] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:28.939795  9012 raft_consensus.cc:493] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:28.970968  8797 raft_consensus.cc:740] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
W20260709 13:02:28.971113  1644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:28.974535  1717 leader_election.cc:304] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:28.982743  1646 leader_election.cc:304] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:28.927371  9169 raft_consensus.cc:2804] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 13:02:28.938760  9198 raft_consensus.cc:493] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:28.935263  9152 raft_consensus.cc:515] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:29.001421  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:29.002384  2328 raft_consensus.cc:2393] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 24: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:02:29.002712  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:28.985476  8932 raft_consensus.cc:2804] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 13:02:28.996531  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:29.009087  1601 raft_consensus.cc:2468] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 17.
I20260709 13:02:28.974464  9104 raft_consensus.cc:697] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [term 15 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:02:28.991199  9012 raft_consensus.cc:3060] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:02:29.007501  1752 raft_consensus.cc:2393] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 18: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:02:28.998137  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 20 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:28.994807  9202 raft_consensus.cc:2804] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:02:28.995234  1717 leader_election.cc:304] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:29.028424  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:29.037317  1748 raft_consensus.cc:1275] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
I20260709 13:02:28.990047  8797 consensus_queue.cc:260] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:29.011791  1646 leader_election.cc:304] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:28.990455  9169 raft_consensus.cc:493] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:28.936110  9133 raft_consensus.cc:515] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:29.017805  8934 raft_consensus.cc:697] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 22 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:02:29.033052  2298 leader_election.cc:304] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
W20260709 13:02:29.071437  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 59734.8us
I20260709 13:02:28.993498  9204 consensus_queue.cc:1048] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.053s
I20260709 13:02:29.073032  1646 leader_election.cc:304] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:29.073364  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 21 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:02:29.074532  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 154480us
I20260709 13:02:29.025731  9202 raft_consensus.cc:493] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Starting leader election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:02:29.026965  8932 raft_consensus.cc:697] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 18 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:02:29.029798  9203 raft_consensus.cc:2749] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 13:02:29.086987  1747 raft_consensus.cc:2468] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 20.
I20260709 13:02:29.094834  1571 leader_election.cc:304] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:29.068967  9169 raft_consensus.cc:3060] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Advancing to term 25
I20260709 13:02:28.995257  9198 raft_consensus.cc:515] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:29.108096  2301 leader_election.cc:304] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:02:29.111424  9207 consensus_queue.cc:1048] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.070s
I20260709 13:02:29.079165  9115 raft_consensus.cc:2804] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:02:29.153596  1751 raft_consensus.cc:1217] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Deduplicated request from leader. Original: 12.11->[15.12-19.14]   Dedup: 15.12->[17.13-19.14]
I20260709 13:02:29.153430  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:29.173600  1751 raft_consensus.cc:1240] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 19. Current term is 20. Ops: [15.12-19.14]
I20260709 13:02:29.157095  1569 leader_election.cc:304] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:02:29.174938  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:29.151139  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:29.175966  1604 raft_consensus.cc:2468] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 20.
I20260709 13:02:29.175872  1603 raft_consensus.cc:2393] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 23: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:02:29.073762  9199 raft_consensus.cc:2804] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:02:29.149395  1752 raft_consensus.cc:1275] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 18 index: 13. Preceding OpId from leader: term: 20 index: 14. (index mismatch)
W20260709 13:02:29.099661  9180 consensus_meta.cc:287] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.907s	user 0.016s	sys 0.000s
W20260709 13:02:29.183238  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 188167us
W20260709 13:02:29.263710  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 182908us
W20260709 13:02:29.279114  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 49601.2us
I20260709 13:02:29.086196  9088 consensus_queue.cc:1059] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 17.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.154s
I20260709 13:02:29.283908  1569 leader_election.cc:304] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:02:29.185935  9115 raft_consensus.cc:493] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:29.109469  9197 raft_consensus.cc:2804] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Leader election won for term 20
W20260709 13:02:29.294523  1718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:29.107369  9201 raft_consensus.cc:2804] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Leader election won for term 23
W20260709 13:02:29.160905  9190 log.cc:927] Time spent T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.588s	user 0.003s	sys 0.000s
I20260709 13:02:29.082463  9205 consensus_queue.cc:1048] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.158s
I20260709 13:02:29.229027  9058 raft_consensus.cc:2749] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 13:02:29.086372  9202 raft_consensus.cc:3060] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:02:29.111603  8797 raft_consensus.cc:3060] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:02:29.125100  9067 raft_consensus.cc:2804] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:02:29.155627  9154 raft_consensus.cc:493] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:02:29.328255  1718 leader_election.cc:304] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:02:29.184507  9199 raft_consensus.cc:493] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:29.109319  9039 consensus_queue.cc:1048] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.211s
I20260709 13:02:29.100052  9153 consensus_queue.cc:1048] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.217s
I20260709 13:02:29.362932  9209 raft_consensus.cc:493] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:02:29.172441  9213 raft_consensus.cc:2804] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 13:02:29.363653  9209 raft_consensus.cc:515] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:29.364228  9213 raft_consensus.cc:493] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:29.364668  9213 raft_consensus.cc:3060] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Advancing to term 24
I20260709 13:02:29.366593  9067 raft_consensus.cc:493] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:29.367043  9067 raft_consensus.cc:3060] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:02:29.331996  9197 raft_consensus.cc:697] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:02:29.368801  9197 consensus_queue.cc:237] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.10, Last appended by leader: 10, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:29.366149  9209 leader_election.cc:290] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:29.241768  9180 raft_consensus.cc:2749] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:02:29.370829  9209 raft_consensus.cc:493] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:29.370649  9154 raft_consensus.cc:515] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:29.371381  9209 raft_consensus.cc:515] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:29.327551  9115 raft_consensus.cc:3060] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:02:29.318599  8926 raft_consensus.cc:2804] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Leader election won for term 16
I20260709 13:02:29.087288  9011 consensus_queue.cc:237] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 19.15, Last appended by leader: 15, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:29.320014  9120 leader_election.cc:290] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 23 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:29.325995  9107 leader_election.cc:290] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 21 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:29.346030  9212 raft_consensus.cc:493] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:02:29.388114  9212 raft_consensus.cc:515] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:29.150274  9012 raft_consensus.cc:515] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:29.393079  9012 leader_election.cc:290] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 18 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:29.282821  9181 consensus_queue.cc:1059] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.063s
I20260709 13:02:29.394809  9181 consensus_queue.cc:1243] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer af520b67945d443b901bbeca37cc5701 log is divergent from this leader: its last log entry 18.13 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 11
I20260709 13:02:29.372985  9154 leader_election.cc:290] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:29.395972  9212 leader_election.cc:290] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:29.118646  8921 consensus_queue.cc:1059] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 17.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.225s
I20260709 13:02:29.142812  8893 consensus_queue.cc:1048] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.197s
I20260709 13:02:29.382750  8926 raft_consensus.cc:697] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
W20260709 13:02:29.403256  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 278786us
I20260709 13:02:29.352870  9065 leader_election.cc:290] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:29.404124  1599 raft_consensus.cc:2468] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 20.
I20260709 13:02:29.131116  8932 consensus_queue.cc:237] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.11, Last appended by leader: 11, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } }
I20260709 13:02:29.435382  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:29.438366  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:29.442569  2336 raft_consensus.cc:2393] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 16: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:02:29.440598  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:29.443475  2333 raft_consensus.cc:2393] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 15: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:02:29.377132  9199 raft_consensus.cc:3060] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Advancing to term 18
W20260709 13:02:29.434110  9190 log.cc:927] Time spent T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.079s	user 0.000s	sys 0.000s
I20260709 13:02:29.317668  9104 consensus_queue.cc:237] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:29.432562  9152 leader_election.cc:290] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:29.146178  1495 catalog_manager.cc:5697] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 reported cstate change: term changed from 13 to 15. New cstate: current_term: 15 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 13:02:29.436235  9215 raft_consensus.cc:493] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:29.428553  9207 consensus_queue.cc:1048] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.049s
W20260709 13:02:29.475023  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 146997us
I20260709 13:02:29.324929  8934 consensus_queue.cc:237] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 20.13, Last appended by leader: 13, Current term: 22, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:29.478156  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 23 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:29.426335  9213 raft_consensus.cc:515] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:02:29.480190  1643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:29.483939  2333 raft_consensus.cc:2410] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for term 23 because replica has last-logged OpId of term: 22 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 15.
I20260709 13:02:29.478801  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:29.486994  1717 leader_election.cc:304] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:29.489353  2328 raft_consensus.cc:2393] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 21: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:02:29.499004  2328 raft_consensus.cc:1240] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 24 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 23. Current term is 24. Ops: []
I20260709 13:02:29.490790  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:29.514981  2334 raft_consensus.cc:1275] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 16 index: 13. (index mismatch)
I20260709 13:02:29.514860  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:29.516659  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:29.517390  1601 raft_consensus.cc:3060] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:02:29.502252  8926 consensus_queue.cc:237] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.13, Last appended by leader: 13, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:29.519984  9218 raft_consensus.cc:2804] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 13:02:29.520558  9218 raft_consensus.cc:697] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 22 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:02:29.487054  9012 raft_consensus.cc:493] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:29.524217  2334 raft_consensus.cc:1275] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 20 index: 15. Preceding OpId from leader: term: 21 index: 16. (index mismatch)
W20260709 13:02:29.525511  1644 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.107.196:46625 timed out after 1.645s (SENT) [suppressed 13 similar messages]
I20260709 13:02:29.521508  9218 consensus_queue.cc:237] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 20.10, Last appended by leader: 10, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:02:29.528951  9200 consensus_meta.cc:287] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.566s	user 0.010s	sys 0.001s
I20260709 13:02:29.530648  9213 leader_election.cc:290] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 24 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:29.488137  9215 raft_consensus.cc:515] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:02:29.528828  9217 log.cc:927] Time spent T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.162s	user 0.000s	sys 0.002s
W20260709 13:02:29.544114  1571 leader_election.cc:336] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 18 pre-election: RPC error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Timed out: RequestConsensusVote RPC to 127.0.107.195:40693 timed out after 1.528s (SENT)
I20260709 13:02:29.554891  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:29.492396  9209 leader_election.cc:290] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:29.480211  9208 consensus_queue.cc:1059] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 17.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.071s
I20260709 13:02:29.558085  9198 leader_election.cc:290] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:02:29.559015  9210 log.cc:927] Time spent T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.174s	user 0.001s	sys 0.001s
I20260709 13:02:29.548172  8817 raft_consensus.cc:493] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:29.559711  9148 consensus_queue.cc:1059] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 19.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
W20260709 13:02:29.541350  8994 consensus_meta.cc:287] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.585s	user 0.012s	sys 0.004s
W20260709 13:02:29.563169  9035 consensus_meta.cc:287] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 1.434s	user 0.009s	sys 0.008s
I20260709 13:02:29.551625  9201 raft_consensus.cc:697] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 23 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:02:29.568434  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:29.571198  9216 raft_consensus.cc:493] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:29.564877  1643 leader_election.cc:304] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:02:29.571713  9227 raft_consensus.cc:2749] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:02:29.558414  9214 raft_consensus.cc:493] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:29.590127  1643 leader_election.cc:400] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 15 election: Vote denied by peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445) with higher term. Message: Invalid argument: T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for earlier term 15. Current term is 16.
I20260709 13:02:29.547564  9012 raft_consensus.cc:515] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:29.590572  1643 leader_election.cc:403] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 15 election: Cancelling election due to peer responding with higher term
I20260709 13:02:29.580468  1601 raft_consensus.cc:2468] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 18.
I20260709 13:02:29.571023  9222 raft_consensus.cc:493] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:29.580807  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:29.596489  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:02:29.569087  8968 consensus_meta.cc:287] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.821s	user 0.007s	sys 0.008s
I20260709 13:02:29.593676  1717 leader_election.cc:304] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:29.552417  9200 raft_consensus.cc:515] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:29.589921  9213 raft_consensus.cc:2749] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:02:29.626390  9150 raft_consensus.cc:493] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:29.631377  9150 raft_consensus.cc:515] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:29.632593  9107 consensus_queue.cc:1048] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 13:02:29.576666  8994 raft_consensus.cc:515] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:29.627902  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 102489us
I20260709 13:02:29.579349  8817 raft_consensus.cc:515] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:29.620929  8968 raft_consensus.cc:2749] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:02:29.643934  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:29.629601  1643 leader_election.cc:304] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 13:02:29.631640  9200 leader_election.cc:290] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 19 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:29.594367  9039 raft_consensus.cc:493] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:29.623077  9153 raft_consensus.cc:493] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:29.624563  2331 raft_consensus.cc:1275] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 19 index: 14. (index mismatch)
I20260709 13:02:29.582873  9035 raft_consensus.cc:515] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:29.591849  9088 consensus_queue.cc:1059] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 17.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.037s
I20260709 13:02:29.666656  1643 leader_election.cc:304] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:02:29.625267  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:29.623358  9204 raft_consensus.cc:2749] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Leader election lost for term 15. Reason: Vote denied by peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445) with higher term. Message: Invalid argument: T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for earlier term 15. Current term is 16.
W20260709 13:02:29.672014  1600 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40166 (request call id 3050) took 1534 ms (client timeout 1523 ms). Trace:
I20260709 13:02:29.673285  1750 raft_consensus.cc:1275] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 15 index: 12. (index mismatch)
I20260709 13:02:29.704370  2330 raft_consensus.cc:3060] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:02:29.705785  1643 leader_election.cc:304] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
W20260709 13:02:29.726727  1600 rpcz_store.cc:269] 0709 13:02:28.137983 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:28.138189 (+   206us) service_pool.cc:224] Handling call
0709 13:02:29.671858 (+1533669us) spinlock_profiling.cc:243] Waited 799 ms on lock 0x7b5400279cd4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:02:29.671945 (+    87us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1838059392}
I20260709 13:02:29.733335  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:29.733680  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:29.594849  9214 raft_consensus.cc:515] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:29.734453  1680 raft_consensus.cc:2468] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 19.
I20260709 13:02:29.672659  9213 raft_consensus.cc:2804] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:02:29.735299  1598 raft_consensus.cc:1240] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 15. Current term is 16. Ops: []
I20260709 13:02:29.731256  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:29.774194  1674 raft_consensus.cc:2393] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 23: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:02:29.734423  2331 raft_consensus.cc:2468] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 19.
I20260709 13:02:29.730999  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:29.738278  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:29.729079  9011 raft_consensus.cc:2749] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 13:02:29.743196  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 22 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
W20260709 13:02:29.745787  9233 log.cc:927] Time spent T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.110s	user 0.000s	sys 0.000s
I20260709 13:02:29.746881  9231 raft_consensus.cc:2804] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 13:02:29.747349  9223 raft_consensus.cc:493] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:29.612597  9222 raft_consensus.cc:515] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:02:29.741235  9169 consensus_meta.cc:287] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.509s	user 0.010s	sys 0.003s
W20260709 13:02:29.748683  9230 log.cc:927] Time spent T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.164s	user 0.000s	sys 0.000s
I20260709 13:02:29.749195  2337 raft_consensus.cc:1275] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 17 index: 15. Preceding OpId from leader: term: 18 index: 16. (index mismatch)
I20260709 13:02:29.749440  2330 raft_consensus.cc:2468] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 16.
I20260709 13:02:29.786553  2330 raft_consensus.cc:1275] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 14 index: 13. Preceding OpId from leader: term: 16 index: 14. (index mismatch)
I20260709 13:02:29.647358  9215 leader_election.cc:290] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:29.752910  9067 raft_consensus.cc:515] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:29.755743  8994 leader_election.cc:290] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 16 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:29.758566  9214 leader_election.cc:290] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:29.758224  9199 raft_consensus.cc:515] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:29.788513  2332 raft_consensus.cc:1240] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 19. Current term is 20. Ops: []
I20260709 13:02:29.788561  2334 raft_consensus.cc:1240] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 17. Current term is 18. Ops: []
I20260709 13:02:29.789844  9199 leader_election.cc:290] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 18 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:29.791354  9067 leader_election.cc:290] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 20 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:29.764405  2335 raft_consensus.cc:2468] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 19.
I20260709 13:02:29.766597  9220 raft_consensus.cc:493] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:29.771699  9235 log.cc:927] Time spent T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.065s	user 0.001s	sys 0.000s
I20260709 13:02:29.792260  9220 raft_consensus.cc:515] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:29.590595  9216 raft_consensus.cc:515] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:29.776608  1718 leader_election.cc:304] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:29.794790  9220 leader_election.cc:290] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:29.777908  2331 raft_consensus.cc:1240] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 15. Current term is 16. Ops: []
I20260709 13:02:29.795053  9216 leader_election.cc:290] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:29.706152  8968 raft_consensus.cc:493] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:29.706696  9232 raft_consensus.cc:493] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:29.778875  1678 raft_consensus.cc:3060] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:02:29.796389  9232 raft_consensus.cc:515] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:29.794912  9215 raft_consensus.cc:2804] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:02:29.783728  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:29.732403  9148 raft_consensus.cc:2804] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:02:29.805682  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 19 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:29.806361  2331 raft_consensus.cc:2468] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 18.
I20260709 13:02:29.803287  1718 leader_election.cc:304] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:29.772535  9201 consensus_queue.cc:237] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 21.11, Last appended by leader: 11, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:29.796316  8968 raft_consensus.cc:515] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:02:29.813736  9111 consensus_meta.cc:287] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.918s	user 0.002s	sys 0.013s
I20260709 13:02:29.814744  9232 leader_election.cc:290] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:29.802873  9223 raft_consensus.cc:515] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:29.815745  8968 leader_election.cc:290] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:29.816500  8817 leader_election.cc:290] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:29.728955  9153 raft_consensus.cc:515] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:29.817857  9223 leader_election.cc:290] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:29.822479  1718 leader_election.cc:304] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:29.773676  9213 raft_consensus.cc:493] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:02:29.740816  9150 leader_election.cc:290] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:29.825556  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:29.726147  9039 raft_consensus.cc:515] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:29.826834  1672 raft_consensus.cc:1275] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 18 index: 12. Preceding OpId from leader: term: 20 index: 13. (index mismatch)
I20260709 13:02:29.826792  2329 raft_consensus.cc:2468] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 18.
I20260709 13:02:29.827759  9213 raft_consensus.cc:3060] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:02:29.828356  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 21 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:29.798882  9215 raft_consensus.cc:493] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
W20260709 13:02:29.802861  9174 consensus_meta.cc:287] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 1.292s	user 0.014s	sys 0.003s
I20260709 13:02:29.830716  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:29.835930  2335 raft_consensus.cc:2468] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 20.
I20260709 13:02:29.843286  9154 raft_consensus.cc:493] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:29.806118  9148 raft_consensus.cc:493] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:29.846446  1678 raft_consensus.cc:2468] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 22.
I20260709 13:02:29.847041  9172 leader_election.cc:290] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:29.846737  9154 raft_consensus.cc:515] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:29.805610  9231 raft_consensus.cc:697] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 18 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:02:29.849906  9035 leader_election.cc:290] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 16 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:29.824663  9200 raft_consensus.cc:2804] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Leader election won for term 19
W20260709 13:02:29.857519  1599 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40154 (request call id 2940) took 1833 ms (client timeout 1943 ms). Trace:
W20260709 13:02:29.858011  1599 rpcz_store.cc:269] 0709 13:02:28.024360 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:28.024574 (+   214us) service_pool.cc:224] Handling call
0709 13:02:29.857319 (+1832745us) spinlock_profiling.cc:243] Waited 1.29 s on lock 0x7b54000ac8d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:02:29.857449 (+   130us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2970496384}
I20260709 13:02:29.844046  9215 raft_consensus.cc:3060] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:02:29.845876  9216 raft_consensus.cc:2804] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:02:29.854265  1718 leader_election.cc:304] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:29.860868  1597 raft_consensus.cc:1275] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 16 index: 11. Preceding OpId from leader: term: 20 index: 13. (index mismatch)
I20260709 13:02:29.863476  2331 raft_consensus.cc:1275] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
I20260709 13:02:29.867048  1644 leader_election.cc:304] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:29.851539  9148 raft_consensus.cc:3060] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:02:29.860560  9204 raft_consensus.cc:493] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:29.814314  9111 raft_consensus.cc:515] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:29.860805  9088 raft_consensus.cc:493] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
W20260709 13:02:29.907519  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 53329.1us
W20260709 13:02:29.909022  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 59834.6us
I20260709 13:02:29.880178  1569 leader_election.cc:304] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:02:29.805869  9169 raft_consensus.cc:515] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:29.924485  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 44569.4us
I20260709 13:02:29.926162  2328 raft_consensus.cc:1275] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 19 index: 13. Preceding OpId from leader: term: 21 index: 14. (index mismatch)
I20260709 13:02:29.926378  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:29.925256  1569 leader_election.cc:304] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:02:29.923048  1718 leader_election.cc:304] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
W20260709 13:02:29.931267  1718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:29.861056  9153 leader_election.cc:290] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:29.907145  9216 raft_consensus.cc:493] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:29.930727  1569 leader_election.cc:304] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:02:29.905485  9200 raft_consensus.cc:697] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 19 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:02:29.911926  9174 raft_consensus.cc:2804] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:02:29.930271  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:29.934719  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:29.941351  2328 raft_consensus.cc:2393] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 16: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:02:29.941296  9226 raft_consensus.cc:493] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:29.942020  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:29.942742  2330 raft_consensus.cc:2468] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 21.
I20260709 13:02:29.928730  9035 raft_consensus.cc:2749] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:02:29.929718  8981 raft_consensus.cc:2749] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:02:29.934070  8932 raft_consensus.cc:2804] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:02:29.908039  8934 raft_consensus.cc:493] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:02:29.944986  1718 leader_election.cc:304] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:29.948889  9197 consensus_queue.cc:1048] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 13:02:29.948828  9150 consensus_queue.cc:1059] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 13:02:29.927119  9204 raft_consensus.cc:515] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:29.928803  9088 raft_consensus.cc:515] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:29.950560  9200 consensus_queue.cc:237] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.12, Last appended by leader: 12, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:29.946816  9174 raft_consensus.cc:493] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:29.938737  9216 raft_consensus.cc:3060] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:02:29.952870  1495 catalog_manager.cc:5697] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 reported cstate change: term changed from 13 to 15. New cstate: current_term: 15 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 13:02:29.956034  9111 leader_election.cc:290] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 18 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:29.957752  8932 raft_consensus.cc:493] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:29.960984  9153 consensus_queue.cc:1048] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 13:02:29.963053  9222 leader_election.cc:290] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:29.932435  9172 raft_consensus.cc:493] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:29.933171  9012 leader_election.cc:290] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:29.940447  9061 raft_consensus.cc:2804] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:02:29.945197  9011 raft_consensus.cc:493] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:29.930403  9231 consensus_queue.cc:237] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.10, Last appended by leader: 10, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:29.931975  8817 consensus_queue.cc:1048] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.040s
I20260709 13:02:29.933640  2331 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51"
dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
mode: GRACEFUL
new_leader_uuid: "af520b67945d443b901bbeca37cc5701"
 from {username='slave'} at 127.0.0.1:35790
I20260709 13:02:30.398599  2331 raft_consensus.cc:606] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Received request to transfer leadership to TS af520b67945d443b901bbeca37cc5701
I20260709 13:02:30.399087  2331 raft_consensus.cc:614] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 13:02:29.970192  8797 raft_consensus.cc:493] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:30.333005  8932 raft_consensus.cc:3060] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:02:30.296420  9174 raft_consensus.cc:3060] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:02:29.974789  9244 raft_consensus.cc:2804] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 13:02:30.091926  9246 raft_consensus.cc:2749] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:02:29.942050  9226 raft_consensus.cc:515] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:30.284950  9223 raft_consensus.cc:493] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:30.147920  8934 raft_consensus.cc:515] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:30.403426  9169 leader_election.cc:290] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 25 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:30.407334  8797 raft_consensus.cc:515] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:30.411849  2331 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443"
dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
mode: GRACEFUL
new_leader_uuid: "af520b67945d443b901bbeca37cc5701"
 from {username='slave'} at 127.0.0.1:35790
I20260709 13:02:30.447379  2331 raft_consensus.cc:606] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Received request to transfer leadership to TS af520b67945d443b901bbeca37cc5701
W20260709 13:02:30.447527  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 499545us
I20260709 13:02:30.447930  2331 raft_consensus.cc:614] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 13:02:30.415141  9171 leader_election.cc:290] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:30.419641  9133 leader_election.cc:290] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:30.421870  8764 raft_consensus.cc:3060] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:02:30.448849  1569 leader_election.cc:304] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
W20260709 13:02:30.423497  9243 log.cc:927] Time spent T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.051s	user 0.000s	sys 0.000s
I20260709 13:02:30.382084  9104 raft_consensus.cc:493] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:30.439536  1644 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.533s	user 0.023s	sys 0.040s
I20260709 13:02:30.391374  9061 raft_consensus.cc:493] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:02:30.399418  9115 consensus_meta.cc:287] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 1.013s	user 0.005s	sys 0.010s
W20260709 13:02:30.455696  1644 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.550s	user 0.023s	sys 0.041s
W20260709 13:02:30.456341  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 589654us
I20260709 13:02:30.457737  1644 leader_election.cc:304] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 13:02:30.443549  9228 log.cc:927] Time spent T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.817s	user 0.009s	sys 0.002s
I20260709 13:02:30.444643  9015 leader_election.cc:290] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:30.431354  9244 raft_consensus.cc:493] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:30.462366  9216 raft_consensus.cc:515] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:30.465224  9216 leader_election.cc:290] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 19 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:30.470016  1644 leader_election.cc:304] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:02:30.392236  9172 raft_consensus.cc:515] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:30.451097  9218 raft_consensus.cc:493] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:30.398960  9011 raft_consensus.cc:515] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:30.453194  9169 raft_consensus.cc:493] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:30.467823  9246 raft_consensus.cc:2749] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:02:30.460253  9061 raft_consensus.cc:3060] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:02:30.455276  9136 leader_election.cc:290] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:30.473294  9015 raft_consensus.cc:2749] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 13:02:30.471141  9244 raft_consensus.cc:3060] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:02:30.471712  9111 raft_consensus.cc:493] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:30.450754  9223 raft_consensus.cc:515] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:02:30.485762  9202 consensus_meta.cc:287] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 1.103s	user 0.012s	sys 0.004s
I20260709 13:02:30.467705  9115 raft_consensus.cc:515] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:30.474267  9222 consensus_queue.cc:1048] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.016s
I20260709 13:02:30.474748  9107 consensus_queue.cc:1059] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 23.17, Next index: 18, Last known committed idx: 15, Time since last communication: 0.016s
I20260709 13:02:30.505271  9107 raft_consensus.cc:2749] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:02:30.505997  9107 raft_consensus.cc:3055] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 LEADER]: Stepping down as leader of term 23
I20260709 13:02:30.506325  9107 raft_consensus.cc:740] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:02:30.507718  9172 leader_election.cc:290] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:30.493137  9008 raft_consensus.cc:493] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:30.510545  9008 raft_consensus.cc:515] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:30.511243  9035 raft_consensus.cc:3055] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 LEADER]: Stepping down as leader of term 15
I20260709 13:02:30.511412  8797 leader_election.cc:290] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:30.511714  9035 raft_consensus.cc:740] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:02:30.512748  9008 leader_election.cc:290] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:30.512657  9035 consensus_queue.cc:260] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.12, Last appended by leader: 12, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:30.513763  9035 raft_consensus.cc:3060] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:02:30.515089  9011 leader_election.cc:290] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:30.507372  9107 consensus_queue.cc:260] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 23.17, Last appended by leader: 17, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:30.519246  1750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55840 (request call id 3166) took 1511 ms. Trace:
I20260709 13:02:30.519711  9107 raft_consensus.cc:3060] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Advancing to term 24
I20260709 13:02:30.519624  1750 rpcz_store.cc:276] 0709 13:02:29.008310 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:29.008498 (+   188us) service_pool.cc:224] Handling call
0709 13:02:29.036605 (+ 28107us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:29.726809 (+690204us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:30.518945 (+792136us) spinlock_profiling.cc:243] Waited 637 ms on lock 0x7b5400019ad4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:02:30.519174 (+   229us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1464001920}
I20260709 13:02:30.524633  9115 leader_election.cc:290] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 18 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:30.527318  9226 leader_election.cc:290] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:30.526085  1747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55840 (request call id 3173) took 1360 ms. Trace:
I20260709 13:02:30.530922  9204 leader_election.cc:290] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:30.529426  1747 rpcz_store.cc:276] 0709 13:02:29.165617 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:29.165819 (+   202us) service_pool.cc:224] Handling call
0709 13:02:29.166315 (+   496us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:29.166837 (+   522us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 13:02:29.166887 (+    50us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:29.167576 (+   689us) log.cc:844] Serialized 26 byte log entry
0709 13:02:29.445741 (+278165us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:02:29.445816 (+    75us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:29.445927 (+   111us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:30.442855 (+996928us) spinlock_profiling.cc:243] Waited 5.38 ms on lock 0x7b4400022b4c. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f4262f9bc85 00007f4262f9c6a2 00007f4265d0e1db 00007f4265d09aaf 00007f4265cfd86e 00007f4265cfcbb6 00007f4265cfc7d8 00007f4265d06221 00007f4265d061d9 00007f4265d06169
0709 13:02:30.525751 (+ 82896us) raft_consensus.cc:1685] finished
0709 13:02:30.525779 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:30.526012 (+   233us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":19493,"raft.queue_time_us":273738,"raft.run_cpu_time_us":249,"raft.run_wall_time_us":22785,"spinlock_wait_cycles":12377472,"thread_start_us":275874,"threads_started":1,"wal-append.queue_time_us":398103}
I20260709 13:02:30.478089  9153 consensus_queue.cc:1048] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.015s
I20260709 13:02:30.504280  2331 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f"
dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
mode: GRACEFUL
new_leader_uuid: "af520b67945d443b901bbeca37cc5701"
 from {username='slave'} at 127.0.0.1:35790
I20260709 13:02:30.467046  9104 raft_consensus.cc:515] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:30.531936  8934 leader_election.cc:290] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:30.524149  8932 raft_consensus.cc:515] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:30.545159  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:30.547448  9213 consensus_meta.cc:287] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.698s	user 0.013s	sys 0.000s
I20260709 13:02:30.548302  9039 leader_election.cc:290] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:30.551524  1598 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 3054) took 2176 ms. Trace:
W20260709 13:02:30.549118  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 79238.9us
I20260709 13:02:30.553558  1597 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40154 (request call id 2954) took 1861 ms. Trace:
I20260709 13:02:30.559053  1597 rpcz_store.cc:276] 0709 13:02:28.692344 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:28.748860 (+ 56516us) service_pool.cc:224] Handling call
0709 13:02:29.858581 (+1109721us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:29.910942 (+ 52361us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:30.553308 (+642366us) spinlock_profiling.cc:243] Waited 1.11 s on lock 0x7b54000ac8d0. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661618ad 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:02:30.553476 (+   168us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2552514944}
I20260709 13:02:30.484874  9169 raft_consensus.cc:515] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:30.557401  1598 rpcz_store.cc:276] 0709 13:02:28.374889 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:28.438355 (+ 63466us) service_pool.cc:224] Handling call
0709 13:02:28.475749 (+ 37394us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:29.773867 (+1298118us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:30.551158 (+777291us) spinlock_profiling.cc:243] Waited 1.26 s on lock 0x7b540006dbd4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:02:30.551389 (+   231us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2897628928}
I20260709 13:02:30.563747  1599 raft_consensus.cc:1240] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 17. Current term is 18. Ops: []
I20260709 13:02:30.564714  1644 leader_election.cc:304] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:30.565892  1597 raft_consensus.cc:1275] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20260709 13:02:30.568081  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:30.568578  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:30.568835  1597 raft_consensus.cc:2468] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 19.
I20260709 13:02:30.569375  1601 raft_consensus.cc:2393] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 20: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:02:30.548139  9213 raft_consensus.cc:515] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:30.554670  9104 leader_election.cc:290] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:30.482724  9218 raft_consensus.cc:515] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:30.551817  9180 raft_consensus.cc:493] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:30.588649  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:30.592092  1598 raft_consensus.cc:3060] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 23 FOLLOWER]: Advancing to term 24
I20260709 13:02:30.577087  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 16 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:02:30.590551  1644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:30.592878  1597 raft_consensus.cc:2468] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 19.
I20260709 13:02:30.578675  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:30.593814  1602 raft_consensus.cc:3060] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:02:30.594043  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 19 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:30.588478  1754 raft_consensus.cc:2468] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 18.
I20260709 13:02:30.577543  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:30.588320  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:30.595597  1597 raft_consensus.cc:1275] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 15 index: 10. (index mismatch)
I20260709 13:02:30.596163  1599 raft_consensus.cc:2393] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 19: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:02:30.578246  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:30.579479  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:30.576782  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 21 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:30.499017  9111 raft_consensus.cc:515] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:30.561771  9088 leader_election.cc:290] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:30.499290  9202 raft_consensus.cc:515] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:30.607539  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:30.608125  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:30.621353  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 19 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:30.597118  1604 raft_consensus.cc:2376] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 20. Re-sending same reply.
I20260709 13:02:30.493952  9250 consensus_queue.cc:1048] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.084s
W20260709 13:02:30.602982  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 137093us
I20260709 13:02:30.615283  9252 raft_consensus.cc:493] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:30.615792  9104 raft_consensus.cc:493] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:30.588733  9226 raft_consensus.cc:493] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:30.554759  9159 consensus_queue.cc:1048] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.053s
I20260709 13:02:30.626992  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 21 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:02:30.630097  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 186158us
I20260709 13:02:30.655043  1598 raft_consensus.cc:2468] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 24.
I20260709 13:02:30.616302  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:30.782950  1747 raft_consensus.cc:3060] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Advancing to term 24
I20260709 13:02:30.599946  9012 raft_consensus.cc:493] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:30.590111  9115 raft_consensus.cc:493] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:30.616362  8764 raft_consensus.cc:493] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:30.656157  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:30.571679  9107 consensus_queue.cc:1048] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.058s
I20260709 13:02:30.657051  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 20 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:30.575192  9260 raft_consensus.cc:2749] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 13:02:30.796130  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:02:30.718564  1645 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.703s	user 0.021s	sys 0.014s
I20260709 13:02:30.631193  9035 raft_consensus.cc:493] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:30.749413  1644 leader_election.cc:336] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 21 election: RPC error from VoteRequest() call to peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625): Timed out: RequestConsensusVote RPC to 127.0.107.196:46625 timed out after 1.829s (SENT)
I20260709 13:02:30.728868  9254 raft_consensus.cc:493] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:30.800091  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 50767.6us
W20260709 13:02:30.800365  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 69054.1us
I20260709 13:02:30.676225  9216 raft_consensus.cc:493] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:30.657737  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 19 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:30.802130  2328 raft_consensus.cc:3060] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:02:30.658187  1753 raft_consensus.cc:2393] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 21: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:02:30.684490  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:30.676755  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:30.743121  1602 raft_consensus.cc:2468] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 18.
W20260709 13:02:30.718407  9262 log.cc:927] Time spent T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.061s	user 0.000s	sys 0.000s
I20260709 13:02:30.781168  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:02:30.651683  9148 consensus_meta.cc:287] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.728s	user 0.013s	sys 0.000s
I20260709 13:02:30.607503  9180 raft_consensus.cc:515] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:30.779742  9169 leader_election.cc:290] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:30.729293  9219 raft_consensus.cc:3055] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 LEADER]: Stepping down as leader of term 19
I20260709 13:02:30.798468  1752 raft_consensus.cc:2393] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 20: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:02:30.697669  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:30.819094  2335 raft_consensus.cc:3060] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 24 FOLLOWER]: Advancing to term 25
I20260709 13:02:30.803158  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:30.806249  1750 raft_consensus.cc:3060] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:02:30.675917  9136 raft_consensus.cc:493] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:02:30.819662  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:30.808949  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:30.652285  9252 raft_consensus.cc:515] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:30.811472  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 16 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:30.811789  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:30.812855  2331 raft_consensus.cc:606] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 19 LEADER]: Received request to transfer leadership to TS af520b67945d443b901bbeca37cc5701
W20260709 13:02:30.796999  1645 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.868s	user 0.022s	sys 0.025s
W20260709 13:02:30.810230  9215 consensus_meta.cc:287] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.899s	user 0.009s	sys 0.005s
I20260709 13:02:30.833299  1675 raft_consensus.cc:1275] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 20 index: 10. Preceding OpId from leader: term: 22 index: 11. (index mismatch)
I20260709 13:02:30.656711  9104 raft_consensus.cc:515] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:30.798167  9231 raft_consensus.cc:493] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:02:30.801528  8764 raft_consensus.cc:515] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:30.848826  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:30.849080  1679 raft_consensus.cc:3055] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 LEADER]: Stepping down as leader of term 17
I20260709 13:02:30.849493  1679 raft_consensus.cc:740] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:02:30.850080  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 22 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:30.850545  8764 leader_election.cc:290] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:30.850732  1678 raft_consensus.cc:2376] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 22. Re-sending same reply.
I20260709 13:02:30.850459  1679 consensus_queue.cc:260] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.11, Last appended by leader: 11, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } }
I20260709 13:02:30.851789  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:30.852023  1679 raft_consensus.cc:3060] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:02:30.797811  9012 raft_consensus.cc:515] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:30.816836  9148 raft_consensus.cc:515] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:30.836443  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:30.854245  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:30.829713  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:30.807734  9216 raft_consensus.cc:515] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:30.857746  9216 leader_election.cc:290] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:30.857997  1674 raft_consensus.cc:1275] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 21 index: 16. Preceding OpId from leader: term: 22 index: 17. (index mismatch)
I20260709 13:02:30.859736  9061 raft_consensus.cc:515] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:30.861481  1674 raft_consensus.cc:1275] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 20 index: 13. Preceding OpId from leader: term: 22 index: 14. (index mismatch)
I20260709 13:02:30.862123  9061 leader_election.cc:290] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 20 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:30.852973  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:30.869920  1678 raft_consensus.cc:2468] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 22.
I20260709 13:02:30.677351  9226 raft_consensus.cc:515] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:30.843416  9088 raft_consensus.cc:493] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:02:30.863521  1679 raft_consensus.cc:1275] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
I20260709 13:02:30.865898  9104 leader_election.cc:290] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:30.867063  1753 raft_consensus.cc:2468] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 22.
I20260709 13:02:30.873700  2328 raft_consensus.cc:2468] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 19.
I20260709 13:02:30.876292  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:30.721621  9244 raft_consensus.cc:515] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:30.818594  9219 raft_consensus.cc:740] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:02:30.901033  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:30.798062  9115 raft_consensus.cc:515] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:30.903414  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:30.920472  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:30.909759  1718 leader_election.cc:304] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:30.906152  1750 raft_consensus.cc:2468] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 22.
I20260709 13:02:30.828799  1747 raft_consensus.cc:2468] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 24.
I20260709 13:02:30.927501  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 19 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:30.916301  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:30.803396  9035 raft_consensus.cc:515] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:30.928983  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:30.929581  1680 raft_consensus.cc:2468] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 19.
I20260709 13:02:30.930471  9035 leader_election.cc:290] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:30.930711  2335 raft_consensus.cc:2468] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 25.
I20260709 13:02:30.847301  9169 raft_consensus.cc:493] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:02:30.923324  1678 raft_consensus.cc:2393] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 21: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:02:30.931583  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:30.924845  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:30.932102  1718 leader_election.cc:304] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:30.931648  9169 raft_consensus.cc:515] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:30.932974  1750 raft_consensus.cc:2393] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 18: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:02:30.932252  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 21 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:30.930492  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:30.934226  1678 raft_consensus.cc:2468] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 20.
I20260709 13:02:30.934700  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:30.935485  1747 raft_consensus.cc:2410] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 for term 19 because replica has last-logged OpId of term: 18 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 12.
I20260709 13:02:30.935310  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 24 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:30.936201  1678 raft_consensus.cc:2393] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 24: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:02:30.935806  1750 raft_consensus.cc:2468] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 20.
I20260709 13:02:30.936389  9216 raft_consensus.cc:2804] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 25 FOLLOWER]: Leader election won for term 25
I20260709 13:02:30.937130  9216 raft_consensus.cc:697] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 25 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:02:30.937707  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:30.938920  1750 raft_consensus.cc:2468] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 19.
I20260709 13:02:30.938549  9216 consensus_queue.cc:237] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 23.15, Last appended by leader: 15, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:30.940599  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 19 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:30.934960  9169 leader_election.cc:290] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:30.850481  9252 leader_election.cc:290] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:30.945542  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:30.949113  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:30.949856  1596 raft_consensus.cc:2468] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 21.
I20260709 13:02:30.951581  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 21 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:30.953594  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:30.954308  1603 raft_consensus.cc:1275] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 16 index: 7. Preceding OpId from leader: term: 19 index: 9. (index mismatch)
I20260709 13:02:30.946152  1677 raft_consensus.cc:3060] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:02:30.953598  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:30.915359  9088 raft_consensus.cc:515] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:30.957751  9169 consensus_queue.cc:1048] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 13:02:30.958920  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:30.839583  9215 raft_consensus.cc:515] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:30.960201  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:30.960983  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:30.961454  9215 leader_election.cc:290] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 20 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:30.961627  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:30.890122  9213 leader_election.cc:290] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:30.961644  1598 raft_consensus.cc:2468] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 20.
I20260709 13:02:30.962299  1599 raft_consensus.cc:3060] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:02:30.802030  9254 raft_consensus.cc:515] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:30.949083  1674 raft_consensus.cc:2468] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 21.
I20260709 13:02:30.967381  9254 leader_election.cc:290] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:30.970144  1680 raft_consensus.cc:2468] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 16.
I20260709 13:02:30.923771  9111 leader_election.cc:290] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:30.971657  1601 raft_consensus.cc:1240] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 17. Current term is 18. Ops: []
I20260709 13:02:30.973511  9180 leader_election.cc:290] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:30.973902  1599 raft_consensus.cc:2468] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 18.
W20260709 13:02:30.975449  9275 log.cc:927] Time spent T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.054s	user 0.001s	sys 0.000s
I20260709 13:02:30.977262  9202 leader_election.cc:290] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 21 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:30.983372  1749 raft_consensus.cc:2393] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 22: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:02:30.954197  1597 raft_consensus.cc:2468] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 21.
I20260709 13:02:30.987619  1596 raft_consensus.cc:2368] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for earlier term 19. Current term is 20.
I20260709 13:02:30.988636  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:30.837999  9136 raft_consensus.cc:515] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:30.993407  9111 consensus_queue.cc:1048] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 13:02:30.994736  1718 leader_election.cc:304] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:02:30.945077  9104 raft_consensus.cc:2804] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 13:02:30.877808  9231 raft_consensus.cc:515] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:31.004575  1718 leader_election.cc:304] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:31.008427  1752 raft_consensus.cc:2468] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 21.
I20260709 13:02:31.013759  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 18 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:31.009212  9254 raft_consensus.cc:2749] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:02:31.015650  9252 raft_consensus.cc:2804] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:02:31.012800  9215 raft_consensus.cc:493] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:31.025048  1748 raft_consensus.cc:2410] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 for term 20 because replica has last-logged OpId of term: 19 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 13.
I20260709 13:02:31.026552  1571 leader_election.cc:304] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:31.013343  9104 raft_consensus.cc:697] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 19 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:02:31.038316  1674 raft_consensus.cc:1275] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 18 index: 10. Preceding OpId from leader: term: 22 index: 12. (index mismatch)
I20260709 13:02:31.027961  9252 raft_consensus.cc:493] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:31.028406  9175 raft_consensus.cc:3055] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 LEADER]: Stepping down as leader of term 19
I20260709 13:02:31.031683  9061 raft_consensus.cc:2804] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Leader pre-election won for term 20
W20260709 13:02:31.047008  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 106252us
I20260709 13:02:31.048771  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:31.049358  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 19 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:31.049552  1674 raft_consensus.cc:2368] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 20. Current term is 21.
I20260709 13:02:31.063064  1674 raft_consensus.cc:1275] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 17 index: 10. Preceding OpId from leader: term: 18 index: 11. (index mismatch)
I20260709 13:02:31.044723  9252 raft_consensus.cc:3060] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:02:31.031564  9215 raft_consensus.cc:515] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:31.064411  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:31.027021  9219 consensus_queue.cc:260] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 19.11, Last appended by leader: 11, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:31.062320  9244 leader_election.cc:290] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 22 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:31.045837  9175 raft_consensus.cc:740] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:02:31.047097  9061 raft_consensus.cc:493] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:31.065207  9254 raft_consensus.cc:493] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:31.082496  9254 raft_consensus.cc:515] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:31.084801  9254 leader_election.cc:290] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:02:31.086011  9276 log.cc:927] Time spent T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.052s	user 0.000s	sys 0.000s
W20260709 13:02:31.087239  9271 log.cc:927] Time spent T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.071s	user 0.000s	sys 0.001s
I20260709 13:02:31.085947  9175 consensus_queue.cc:260] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.14, Last appended by leader: 14, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:31.088920  9175 raft_consensus.cc:3060] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Advancing to term 20
W20260709 13:02:31.089488  9273 log.cc:927] Time spent T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.061s	user 0.001s	sys 0.000s
I20260709 13:02:31.089800  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:31.072052  9104 consensus_queue.cc:237] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 17.15, Last appended by leader: 15, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:31.073282  9136 leader_election.cc:290] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:31.074625  2329 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49082 (request call id 1348) took 1219 ms. Trace:
W20260709 13:02:31.075021  9272 log.cc:927] Time spent T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.069s	user 0.001s	sys 0.000s
W20260709 13:02:31.075433  9120 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 18 current term: 18 is higher.
I20260709 13:02:31.077795  9231 leader_election.cc:290] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:31.096333  1747 raft_consensus.cc:3060] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:02:31.071412  9061 raft_consensus.cc:3060] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:02:31.109947  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:31.102479  2329 rpcz_store.cc:276] 0709 13:02:29.854892 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:29.856107 (+  1215us) service_pool.cc:224] Handling call
0709 13:02:29.909280 (+ 53173us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:30.472091 (+562811us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 13:02:30.575468 (+103377us) log.cc:844] Serialized 14 byte log entry
0709 13:02:30.631585 (+ 56117us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:30.631663 (+    78us) raft_consensus.cc:1645] Marking committed up to 14
0709 13:02:30.631714 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:31.074300 (+442586us) spinlock_profiling.cc:243] Waited 558 ms on lock 0x7b5400605ed4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:02:31.074360 (+    60us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:31.074525 (+   165us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":15753,"prepare.queue_time_us":145106,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":42,"raft.queue_time_us":9083,"raft.run_cpu_time_us":164,"raft.run_wall_time_us":23202,"spinlock_wait_cycles":1284087552,"thread_start_us":149173,"threads_started":2,"wal-append.queue_time_us":185224}
I20260709 13:02:31.144218  1677 raft_consensus.cc:2468] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 22.
I20260709 13:02:31.120762  1747 raft_consensus.cc:2468] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 16.
I20260709 13:02:31.068965  9209 consensus_queue.cc:1048] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.044s
I20260709 13:02:31.148249  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:31.127830  9219 raft_consensus.cc:3060] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Advancing to term 20
W20260709 13:02:31.149268  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 81607.4us
I20260709 13:02:31.152491  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:31.153174  1747 raft_consensus.cc:3060] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:02:31.156028  9154 leader_election.cc:290] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:31.168908  9154 raft_consensus.cc:493] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:31.169515  9154 raft_consensus.cc:515] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:31.171355  1747 raft_consensus.cc:2468] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 16.
I20260709 13:02:31.171891  9154 leader_election.cc:290] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:31.172911  1571 leader_election.cc:304] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:31.174006  9152 raft_consensus.cc:2804] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Leader election won for term 16
I20260709 13:02:31.174573  9152 raft_consensus.cc:697] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:02:31.175561  9152 consensus_queue.cc:237] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.9, Last appended by leader: 9, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:31.176980  8932 leader_election.cc:290] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 21 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:31.180606  8934 consensus_queue.cc:1048] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 13:02:31.181160  9215 leader_election.cc:290] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:31.183300  8981 consensus_queue.cc:1048] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 13:02:31.184624  1569 leader_election.cc:400] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 20 election: Vote denied by peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261) with higher term. Message: Invalid argument: T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 20. Current term is 21.
I20260709 13:02:31.185062  9207 raft_consensus.cc:493] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:31.179440  2299 leader_election.cc:304] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
W20260709 13:02:31.180130  9174 consensus_meta.cc:287] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.739s	user 0.016s	sys 0.000s
I20260709 13:02:31.188185  9212 raft_consensus.cc:3055] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 13:02:31.188735  9212 raft_consensus.cc:740] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:02:31.188870  9174 raft_consensus.cc:515] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:31.193209  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:31.194900  1747 raft_consensus.cc:3060] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:02:31.196743  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:31.198484  1751 raft_consensus.cc:2393] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 19: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:02:31.204458  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:31.206909  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:31.207206  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:31.185808  9207 raft_consensus.cc:515] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:31.209496  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:31.215410  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:31.216038  1673 raft_consensus.cc:3060] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:02:31.205657  2299 leader_election.cc:304] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:02:31.217257  1674 raft_consensus.cc:2393] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 22: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:02:31.219352  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:31.221930  2299 leader_election.cc:304] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:02:31.199296  9058 consensus_queue.cc:1048] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.022s
I20260709 13:02:31.214111  9209 raft_consensus.cc:2804] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 13:02:31.226872  9209 raft_consensus.cc:493] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:02:31.227483  9209 raft_consensus.cc:3060] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:02:31.234146  9174 leader_election.cc:290] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 19 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:02:31.236446  1754 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55840 (request call id 3149) took 3229 ms (client timeout 1528 ms). Trace:
W20260709 13:02:31.236961  1754 rpcz_store.cc:269] 0709 13:02:28.006705 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:28.007700 (+   995us) service_pool.cc:224] Handling call
0709 13:02:31.236148 (+3228448us) spinlock_profiling.cc:243] Waited 2.58 s on lock 0x7b540031f454. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:02:31.236304 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5935605504}
I20260709 13:02:31.201483  9252 raft_consensus.cc:515] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:31.240756  9252 leader_election.cc:290] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 21 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:31.231572  9282 raft_consensus.cc:2804] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 13:02:31.242408  1746 raft_consensus.cc:1275] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 17 index: 10. Preceding OpId from leader: term: 18 index: 11. (index mismatch)
I20260709 13:02:31.242588  9282 raft_consensus.cc:493] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:31.243342  9282 raft_consensus.cc:3060] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:02:31.245317  1746 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55840 (request call id 3160) took 2551 ms. Trace:
I20260709 13:02:31.258677  1746 rpcz_store.cc:276] 0709 13:02:28.693602 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:28.693784 (+   182us) service_pool.cc:224] Handling call
0709 13:02:31.241916 (+2548132us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:31.242892 (+   976us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:31.245009 (+  2117us) spinlock_profiling.cc:243] Waited 2.55 s on lock 0x7b540031f450. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661618ad 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:02:31.245180 (+   171us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5862081408}
I20260709 13:02:31.190058  9212 consensus_queue.cc:260] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 17.15, Last appended by leader: 15, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:31.252058  9226 leader_election.cc:290] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:31.261175  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:31.271081  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:02:31.265633  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 43859.9us
I20260709 13:02:31.264703  1747 raft_consensus.cc:2468] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 20.
I20260709 13:02:31.277324  1569 leader_election.cc:304] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:02:31.284679  1673 raft_consensus.cc:2468] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 20.
I20260709 13:02:31.299435  9035 raft_consensus.cc:2749] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:02:31.299049  1495 catalog_manager.cc:5697] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 reported cstate change: term changed from 9 to 12, leader changed from 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) to 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194). New cstate: current_term: 12 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } }
I20260709 13:02:31.301127  1495 catalog_manager.cc:5697] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 reported cstate change: term changed from 7 to 15, leader changed from 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194) to af520b67945d443b901bbeca37cc5701 (127.0.107.195). New cstate: current_term: 15 leader_uuid: "af520b67945d443b901bbeca37cc5701" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: HEALTHY } } }
I20260709 13:02:31.304185  9207 leader_election.cc:290] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:31.304352  9088 leader_election.cc:290] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:31.271061  8981 consensus_queue.cc:1048] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.034s
I20260709 13:02:31.307315  9212 raft_consensus.cc:3060] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:02:31.276515  9253 raft_consensus.cc:493] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:31.312167  9253 raft_consensus.cc:515] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } }
I20260709 13:02:31.314045  1672 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52778 (request call id 2811) took 1584 ms. Trace:
I20260709 13:02:31.323594  1672 rpcz_store.cc:276] 0709 13:02:29.729689 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:29.729829 (+   140us) service_pool.cc:224] Handling call
0709 13:02:29.735694 (+  5865us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:29.830221 (+ 94527us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:31.313770 (+1483549us) spinlock_profiling.cc:243] Waited 90.9 ms on lock 0x7b5400091354. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:02:31.313973 (+   203us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":209012096}
I20260709 13:02:31.327210  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:31.328306  1754 raft_consensus.cc:2468] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 15.
I20260709 13:02:31.328991  2336 raft_consensus.cc:2393] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 21: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:02:31.336428  1495 catalog_manager.cc:5697] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 reported cstate change: term changed from 13 to 16. New cstate: current_term: 16 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 13:02:31.338727  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:31.339725  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:31.340284  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:31.340437  1672 raft_consensus.cc:2376] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 22. Re-sending same reply.
I20260709 13:02:31.340874  1754 raft_consensus.cc:3060] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:02:31.344954  1495 catalog_manager.cc:5697] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 reported cstate change: term changed from 10 to 16, leader changed from 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) to 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194). New cstate: current_term: 16 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
W20260709 13:02:31.351186  1494 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35444 (request call id 263) took 45055 ms (client timeout 15000 ms). Trace:
I20260709 13:02:31.352329  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:31.353273  1750 raft_consensus.cc:3060] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Advancing to term 24
W20260709 13:02:31.355257  1570 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:31.358889  9140 raft_consensus.cc:493] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:31.359607  9140 raft_consensus.cc:515] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:31.362637  8981 raft_consensus.cc:493] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:31.363348  8981 raft_consensus.cc:515] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:31.366664  9207 raft_consensus.cc:493] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:31.352255  1495 catalog_manager.cc:5697] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 reported cstate change: term changed from 13 to 15, leader changed from 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194) to af520b67945d443b901bbeca37cc5701 (127.0.107.195). New cstate: current_term: 15 leader_uuid: "af520b67945d443b901bbeca37cc5701" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: HEALTHY } } }
I20260709 13:02:31.367717  1717 leader_election.cc:304] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:02:31.369592  1717 leader_election.cc:304] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:31.370579  8968 consensus_queue.cc:1048] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 13:02:31.371037  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
W20260709 13:02:31.307389  9266 log.cc:927] Time spent T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.510s	user 0.003s	sys 0.000s
I20260709 13:02:31.372352  9218 leader_election.cc:290] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:31.373018  8968 raft_consensus.cc:2804] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:02:31.373517  9218 raft_consensus.cc:3060] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:02:31.373670  8968 raft_consensus.cc:493] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:31.374272  8968 raft_consensus.cc:3060] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:02:31.374374  1717 leader_election.cc:304] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:02:31.370051  1495 catalog_manager.cc:5697] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 reported cstate change: term changed from 9 to 12, leader changed from 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) to af520b67945d443b901bbeca37cc5701 (127.0.107.195). New cstate: current_term: 12 leader_uuid: "af520b67945d443b901bbeca37cc5701" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:02:31.377641  9232 raft_consensus.cc:2804] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:02:31.377760  9012 leader_election.cc:290] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:31.378317  9232 raft_consensus.cc:493] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:31.378870  9232 raft_consensus.cc:3060] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:02:31.383355  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 21 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:31.384303  1752 raft_consensus.cc:2393] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 21: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:02:31.385499  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:31.379148  1495 catalog_manager.cc:5697] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 reported cstate change: term changed from 5 to 13, leader changed from 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194) to 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196). New cstate: current_term: 13 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 13:02:31.388155  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:31.388458  9253 leader_election.cc:290] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:31.390142  1495 catalog_manager.cc:5697] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 reported cstate change: term changed from 13 to 14, leader changed from 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194) to af520b67945d443b901bbeca37cc5701 (127.0.107.195). New cstate: current_term: 14 leader_uuid: "af520b67945d443b901bbeca37cc5701" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: HEALTHY } } }
I20260709 13:02:31.393471  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:31.394135  1751 raft_consensus.cc:2468] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 18.
I20260709 13:02:31.393023  1495 catalog_manager.cc:5697] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 reported cstate change: term changed from 11 to 14, leader changed from 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) to 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193). New cstate: current_term: 14 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 13:02:31.394484  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:31.395135  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:31.396093  9250 raft_consensus.cc:493] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:31.395536  1495 catalog_manager.cc:5697] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 reported cstate change: term changed from 13 to 16, leader changed from 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) to af520b67945d443b901bbeca37cc5701 (127.0.107.195). New cstate: current_term: 16 leader_uuid: "af520b67945d443b901bbeca37cc5701" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: HEALTHY } } }
I20260709 13:02:31.396862  9250 raft_consensus.cc:515] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:31.395787  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:31.398552  1749 raft_consensus.cc:2393] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 22: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:02:31.398810  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:31.399521  1746 raft_consensus.cc:2468] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 20.
I20260709 13:02:31.399724  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:31.400444  9223 leader_election.cc:290] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:31.278417  8934 raft_consensus.cc:2804] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:02:31.401227  1749 raft_consensus.cc:1240] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 17. Current term is 18. Ops: []
I20260709 13:02:31.401485  8934 raft_consensus.cc:493] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:02:31.402184  8934 raft_consensus.cc:3060] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:02:31.404629  1749 raft_consensus.cc:1240] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 19 LEADER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 18. Current term is 19. Ops: [18.16-18.16]
I20260709 13:02:31.395766  1751 raft_consensus.cc:3060] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:02:31.406208  1749 raft_consensus.cc:1240] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 18. Current term is 19. Ops: [18.14-18.14]
I20260709 13:02:31.407310  2332 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39726 (request call id 1644) took 1971 ms. Trace:
I20260709 13:02:31.407817  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:31.407023  1495 catalog_manager.cc:5697] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 reported cstate change: term changed from 8 to 10. New cstate: current_term: 10 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 13:02:31.408474  1673 raft_consensus.cc:2468] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 15.
I20260709 13:02:31.408303  1749 raft_consensus.cc:1240] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 20. Current term is 21. Ops: [20.11-20.11]
I20260709 13:02:31.412281  1749 raft_consensus.cc:1217] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Deduplicated request from leader. Original: 12.11->[15.12-19.14]   Dedup: 15.12->[17.13-19.14]
I20260709 13:02:31.412436  1747 raft_consensus.cc:1240] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 19. Current term is 20. Ops: []
I20260709 13:02:31.409271  1495 catalog_manager.cc:5697] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 reported cstate change: term changed from 12 to 14. New cstate: current_term: 14 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 13:02:31.416069  9115 leader_election.cc:290] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:31.417177  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 22 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:31.417842  2330 raft_consensus.cc:2468] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 21.
I20260709 13:02:31.418843  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:31.419634  2335 raft_consensus.cc:2468] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 19.
I20260709 13:02:31.420380  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:31.420763  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:31.421100  2337 raft_consensus.cc:2468] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 21.
I20260709 13:02:31.421729  2333 raft_consensus.cc:2393] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 22: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
W20260709 13:02:31.419768  1644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:31.422721  1644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:31.423482  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:31.367471  9207 raft_consensus.cc:515] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:31.426223  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 19 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:31.412817  1749 raft_consensus.cc:1240] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 19. Current term is 20. Ops: [15.12-19.14]
I20260709 13:02:31.430470  1754 raft_consensus.cc:2468] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 18.
W20260709 13:02:31.432862  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:31.434809  1646 leader_election.cc:304] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:31.347096  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:02:31.436285  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 54012us
I20260709 13:02:31.438791  1571 leader_election.cc:304] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:02:31.417758  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:31.407814  2332 rpcz_store.cc:276] 0709 13:02:29.435620 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:29.435755 (+   135us) service_pool.cc:224] Handling call
0709 13:02:29.444736 (+  8981us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:29.789007 (+344271us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:31.407018 (+1618011us) spinlock_profiling.cc:243] Waited 344 ms on lock 0x7b54004880d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:02:31.407228 (+   210us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":790507904}
I20260709 13:02:31.461361  1717 leader_election.cc:304] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:02:31.456000  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:31.456993  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:31.468377  2328 raft_consensus.cc:3060] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:02:31.455533  1569 leader_election.cc:304] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:02:31.482421  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 21 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:31.483500  2330 raft_consensus.cc:2410] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for term 21 because replica has last-logged OpId of term: 19 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 13.
W20260709 13:02:31.483572  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 90329.2us
I20260709 13:02:31.465837  1718 leader_election.cc:304] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:31.485522  2337 raft_consensus.cc:2393] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 20: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:02:31.486265  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:31.493266  1750 raft_consensus.cc:2468] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 24.
I20260709 13:02:31.485183  1718 leader_election.cc:304] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:31.496104  2334 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39726 (request call id 1655) took 1925 ms. Trace:
I20260709 13:02:31.500084  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:31.503530  1751 raft_consensus.cc:2468] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 21.
I20260709 13:02:31.495242  9203 raft_consensus.cc:2804] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 13:02:31.499498  1643 leader_election.cc:304] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
W20260709 13:02:31.550006  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 50623.9us
I20260709 13:02:31.550773  1643 leader_election.cc:304] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1
W20260709 13:02:31.508055  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 52664.1us
I20260709 13:02:31.522527  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:31.523721  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:31.523942  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 21 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:31.552948  1749 raft_consensus.cc:1240] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 25 LEADER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 24. Current term is 25. Ops: []
W20260709 13:02:31.530352  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 95705.4us
I20260709 13:02:31.466809  8814 raft_consensus.cc:493] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:31.553922  1646 leader_election.cc:304] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:02:31.558832  1750 raft_consensus.cc:1240] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 20. Current term is 21. Ops: [20.14-20.14]
I20260709 13:02:31.559689  1646 leader_election.cc:304] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:02:31.561060  1646 leader_election.cc:304] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:31.537537  1753 raft_consensus.cc:2468] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 21.
I20260709 13:02:31.534168  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:31.562763  1673 raft_consensus.cc:2468] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 18.
I20260709 13:02:31.495054  9252 raft_consensus.cc:493] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:31.536192  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:31.503055  9104 raft_consensus.cc:493] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:31.546099  1599 raft_consensus.cc:1275] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
I20260709 13:02:31.565485  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 19 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:31.546792  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:31.547266  1752 raft_consensus.cc:1240] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 20. Current term is 21. Ops: []
I20260709 13:02:31.502519  9227 raft_consensus.cc:2804] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Leader pre-election won for term 22
W20260709 13:02:31.547859  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 62800.6us
I20260709 13:02:31.572464  1718 leader_election.cc:304] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:31.506042  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 21 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:31.574084  2335 raft_consensus.cc:3060] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:02:31.496029  8952 raft_consensus.cc:2804] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 13:02:31.546149  2334 rpcz_store.cc:276] 0709 13:02:29.570472 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:29.570715 (+   243us) service_pool.cc:224] Handling call
0709 13:02:29.571070 (+   355us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:29.789208 (+218138us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:31.495811 (+1706603us) spinlock_profiling.cc:243] Waited 217 ms on lock 0x7b5400645354. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:02:31.496034 (+   223us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":499775744}
I20260709 13:02:31.550807  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:31.552738  1596 raft_consensus.cc:2410] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for term 20 because replica has last-logged OpId of term: 19 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 12.
I20260709 13:02:31.553155  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 19 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:31.507089  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 23 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:31.578377  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:31.555119  2328 raft_consensus.cc:2468] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 19.
I20260709 13:02:31.485349  9223 consensus_queue.cc:1048] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.072s
I20260709 13:02:31.557765  1643 leader_election.cc:304] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:02:31.538829  9088 raft_consensus.cc:2804] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:02:31.552533  1569 leader_election.cc:304] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:02:31.567528  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:31.565529  1753 raft_consensus.cc:1240] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 19. Current term is 20. Ops: [19.14-19.14]
I20260709 13:02:31.570858  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:31.531827  9253 raft_consensus.cc:2804] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Leader election won for term 20
I20260709 13:02:31.594144  1753 raft_consensus.cc:1240] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 23. Current term is 24. Ops: []
I20260709 13:02:31.578810  1601 raft_consensus.cc:2468] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 18.
W20260709 13:02:31.351763  1494 rpcz_store.cc:269] 0709 13:01:46.295445 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:46.296189 (+   744us) service_pool.cc:224] Handling call
0709 13:02:31.350582 (+45054393us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 13:02:19.091673 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 13:02:19.092564 (+   891us) write_op.cc:432] Acquiring schema lock in shared mode
0709 13:02:19.092631 (+    67us) write_op.cc:435] Acquired schema lock
0709 13:02:19.092655 (+    24us) tablet.cc:662] Decoding operations
0709 13:02:19.094743 (+  2088us) write_op.cc:620] Acquiring the partition lock for write op
0709 13:02:19.094853 (+   110us) write_op.cc:641] Partition lock acquired for write op
0709 13:02:19.094885 (+    32us) tablet.cc:685] Acquiring locks for 22 operations
0709 13:02:19.407355 (+312470us) tablet.cc:701] Row locks acquired
0709 13:02:19.407415 (+    60us) write_op.cc:260] PREPARE: finished
0709 13:02:19.407642 (+   227us) write_op.cc:270] Start()
0709 13:02:19.407816 (+   174us) write_op.cc:276] Timestamp: P: 1783602139407566 usec, L: 0
0709 13:02:19.407836 (+    20us) op_driver.cc:348] REPLICATION: starting
0709 13:02:19.408772 (+   936us) log.cc:844] Serialized 8616 byte log entry
0709 13:02:20.820600 (+1411828us) op_driver.cc:464] REPLICATION: finished
0709 13:02:21.221865 (+401265us) spinlock_profiling.cc:243] Waited 10.7 ms on lock 0x7b4400022b4c. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f4262f9bc85 00007f4262f9c6a2 00007f4265d0e1db 00007f4265d09aaf 00007f42630580ac 00007f426beb2ab9 00007f426307469b 00007f426307293f 00007f4263074c02 00007f4269b98ef0
0709 13:02:22.054064 (+832199us) write_op.cc:301] APPLY: starting
0709 13:02:22.061204 (+  7140us) tablet.cc:1367] starting BulkCheckPresence
0709 13:02:23.025087 (+963883us) tablet.cc:1370] finished BulkCheckPresence
0709 13:02:23.025148 (+    61us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 13:02:23.597895 (+572747us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 13:02:23.606547 (+  8652us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=44,key_file_lookups=44,delta_file_lookups=44,mrs_lookups=0
0709 13:02:23.606666 (+   119us) write_op.cc:312] APPLY: finished
0709 13:02:23.607852 (+  1186us) log.cc:844] Serialized 196 byte log entry
0709 13:02:23.791516 (+183664us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:02:23.792131 (+   615us) write_op.cc:454] Released schema lock
0709 13:02:23.792242 (+   111us) write_op.cc:341] FINISH: Updating metrics
0709 13:02:23.792997 (+   755us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:02:23.793039 (+    42us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":1468874,"reported_tablets":48,"child_traces":[["op",{"apply.queue_time_us":1188957,"apply.run_cpu_time_us":67990,"apply.run_wall_time_us":1757086,"cfile_cache_hit":90,"cfile_cache_hit_bytes":96263,"num_ops":22,"prepare.queue_time_us":465315,"prepare.run_cpu_time_us":21540,"prepare.run_wall_time_us":433592,"raft.queue_time_us":361536,"raft.run_cpu_time_us":130250,"raft.run_wall_time_us":1022169,"replication_time_us":1412552,"spinlock_wait_cycles":24532224,"thread_start_us":1964201,"threads_started":5,"wal-append.queue_time_us":799759,"wal-append.run_cpu_time_us":25190,"wal-append.run_wall_time_us":2930762}]]}
I20260709 13:02:31.599133  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:31.600274  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 19 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:31.579079  1605 raft_consensus.cc:2468] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 20.
I20260709 13:02:31.518644  9197 consensus_queue.cc:1059] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 17.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.057s
I20260709 13:02:31.509423  8772 consensus_queue.cc:1059] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 15.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.049s
I20260709 13:02:31.581871  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:31.569453  9224 raft_consensus.cc:2749] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 13:02:31.508773  8981 leader_election.cc:290] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:31.653964  2331 raft_consensus.cc:1275] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 19 index: 14. Preceding OpId from leader: term: 20 index: 15. (index mismatch)
I20260709 13:02:31.507498  9107 consensus_queue.cc:1059] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.060s
I20260709 13:02:31.593942  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:31.504386  9181 consensus_queue.cc:1059] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 17.15, Next index: 16, Last known committed idx: 14, Time since last communication: 0.047s
I20260709 13:02:31.596416  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 21 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:31.671667  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:31.672245  1601 raft_consensus.cc:3060] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:02:31.596470  1753 raft_consensus.cc:1240] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 15. Current term is 16. Ops: []
I20260709 13:02:31.547720  9140 leader_election.cc:290] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:31.606290  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 21 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:31.569710  9213 raft_consensus.cc:2804] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 24 FOLLOWER]: Leader election won for term 24
I20260709 13:02:31.524958  9120 consensus_queue.cc:1048] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.047s
I20260709 13:02:31.566407  9174 raft_consensus.cc:2804] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:02:31.575726  9227 raft_consensus.cc:493] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:31.616010  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:31.712427  1598 raft_consensus.cc:2410] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 for term 16 because replica has last-logged OpId of term: 15 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 11.
I20260709 13:02:31.622556  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:31.623664  2335 raft_consensus.cc:2468] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 21.
W20260709 13:02:31.630609  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 78195.4us
I20260709 13:02:31.632735  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:31.715657  2328 raft_consensus.cc:2393] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 19: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:02:31.640829  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 23 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:31.716441  1596 raft_consensus.cc:2468] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 22.
I20260709 13:02:31.526377  9152 consensus_queue.cc:1059] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 17.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.058s
I20260709 13:02:31.651253  9133 raft_consensus.cc:493] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:31.642387  1603 raft_consensus.cc:1275] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 18 index: 9. Preceding OpId from leader: term: 22 index: 11. (index mismatch)
I20260709 13:02:31.533372  9035 consensus_queue.cc:1059] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 19.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.060s
I20260709 13:02:31.732698  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 18 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:31.733506  2328 raft_consensus.cc:2393] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 18: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:02:31.668466  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 21 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:31.582746  9203 raft_consensus.cc:697] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 18 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:02:31.583707  8952 raft_consensus.cc:493] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:31.605044  9214 raft_consensus.cc:2804] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 13:02:31.753810  9214 raft_consensus.cc:493] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:02:31.754276  9214 raft_consensus.cc:3060] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:02:31.763298  9035 raft_consensus.cc:3055] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 LEADER]: Stepping down as leader of term 19
I20260709 13:02:31.763823  9035 raft_consensus.cc:740] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:02:31.764771  9035 consensus_queue.cc:260] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.10, Last appended by leader: 10, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:31.767267  9152 raft_consensus.cc:3055] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 LEADER]: Stepping down as leader of term 20
I20260709 13:02:31.767732  9152 raft_consensus.cc:740] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:02:31.768610  9152 consensus_queue.cc:260] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 20.11, Last appended by leader: 11, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:31.770016  9152 raft_consensus.cc:3060] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:02:31.677536  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:31.693079  9246 raft_consensus.cc:3060] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:02:31.771214  2333 raft_consensus.cc:2393] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 22: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:02:31.772302  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:31.772946  2333 raft_consensus.cc:3060] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:02:31.678084  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:31.692723  9202 raft_consensus.cc:2804] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:02:31.688541  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 24 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:31.599862  9088 raft_consensus.cc:493] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:02:31.609601  9253 raft_consensus.cc:697] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:02:31.573686  8814 raft_consensus.cc:515] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:31.708446  1748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43156 (request call id 1517) took 1083 ms. Trace:
I20260709 13:02:31.579519  9252 raft_consensus.cc:515] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:31.573055  9255 consensus_queue.cc:1048] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.095s
I20260709 13:02:31.674190  9226 raft_consensus.cc:2804] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 13:02:31.667418  9115 raft_consensus.cc:2804] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:02:31.653255  9058 raft_consensus.cc:2804] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:02:31.723251  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:31.826356  1605 raft_consensus.cc:3060] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:02:31.724918  1596 raft_consensus.cc:1275] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 18 index: 12. Preceding OpId from leader: term: 19 index: 13. (index mismatch)
I20260709 13:02:31.783268  9202 raft_consensus.cc:493] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:02:31.737408  9287 log.cc:927] Time spent T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.450s	user 0.000s	sys 0.013s
I20260709 13:02:31.577713  9104 raft_consensus.cc:515] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:31.749514  9250 leader_election.cc:290] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:31.859058  9250 raft_consensus.cc:3055] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 17 LEADER]: Stepping down as leader of term 17
I20260709 13:02:31.858745  9203 consensus_queue.cc:237] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.12, Last appended by leader: 12, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:31.860306  9253 consensus_queue.cc:237] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 18.15, Last appended by leader: 15, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:31.780084  9133 raft_consensus.cc:515] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:31.864880  9133 leader_election.cc:290] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:31.868111  9252 leader_election.cc:290] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:31.753288  8952 raft_consensus.cc:3060] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Advancing to term 23
W20260709 13:02:31.880149  9282 consensus_meta.cc:287] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.636s	user 0.013s	sys 0.000s
I20260709 13:02:31.880807  9282 raft_consensus.cc:515] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:31.836294  9202 raft_consensus.cc:3060] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:02:31.684244  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:02:31.779944  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 222347us
I20260709 13:02:31.883785  1643 leader_election.cc:304] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:02:31.884014  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:31.884675  2331 raft_consensus.cc:2468] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 19.
I20260709 13:02:31.884729  1643 leader_election.cc:304] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 13:02:31.885638  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:31.886123  1643 leader_election.cc:400] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 election: Vote denied by peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445) with higher term. Message: Invalid argument: T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for earlier term 19. Current term is 20.
I20260709 13:02:31.886456  1643 leader_election.cc:403] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 election: Cancelling election due to peer responding with higher term
I20260709 13:02:31.784664  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:31.792387  1600 raft_consensus.cc:2393] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 24: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:02:31.766239  9035 raft_consensus.cc:3060] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:02:31.704977  9174 raft_consensus.cc:493] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:02:31.721542  9227 raft_consensus.cc:3060] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Advancing to term 22
W20260709 13:02:31.742998  9061 consensus_meta.cc:287] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.599s	user 0.013s	sys 0.000s
I20260709 13:02:31.712450  9213 raft_consensus.cc:697] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 24 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:02:31.805210  1748 rpcz_store.cc:276] 0709 13:02:30.625440 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:30.625809 (+   369us) service_pool.cc:224] Handling call
0709 13:02:31.708139 (+1082330us) spinlock_profiling.cc:243] Waited 226 ms on lock 0x7b54003700d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:02:31.708354 (+   215us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":520802560}
I20260709 13:02:31.832523  9058 raft_consensus.cc:493] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:31.738718  9181 consensus_queue.cc:1243] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer af520b67945d443b901bbeca37cc5701 log is divergent from this leader: its last log entry 19.16 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14
W20260709 13:02:31.820094  1570 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:31.828013  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:31.731429  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:02:31.823720  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 262857us
I20260709 13:02:31.797063  9088 raft_consensus.cc:3060] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Advancing to term 16
W20260709 13:02:31.833142  9212 consensus_meta.cc:287] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.525s	user 0.010s	sys 0.006s
I20260709 13:02:31.846741  9207 leader_election.cc:290] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:31.944298  1748 raft_consensus.cc:1240] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 21. Current term is 22. Ops: [21.16-21.16]
I20260709 13:02:31.852317  8814 leader_election.cc:290] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:31.859651  9250 raft_consensus.cc:740] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:02:31.752475  2301 leader_election.cc:304] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:02:31.888866  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:31.890241  2334 raft_consensus.cc:2368] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for earlier term 21. Current term is 22.
W20260709 13:02:31.891126  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:31.829649  9175 consensus_meta.cc:287] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.736s	user 0.009s	sys 0.004s
W20260709 13:02:31.952281  9232 consensus_meta.cc:287] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.573s	user 0.007s	sys 0.006s
I20260709 13:02:31.953006  1602 raft_consensus.cc:2468] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 20.
W20260709 13:02:31.830420  9219 consensus_meta.cc:287] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.679s	user 0.015s	sys 0.000s
I20260709 13:02:31.899744  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:31.953152  9232 raft_consensus.cc:515] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:31.955193  2328 raft_consensus.cc:1240] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 20. Current term is 21. Ops: []
I20260709 13:02:31.955464  1673 raft_consensus.cc:2468] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 20.
W20260709 13:02:31.917670  9209 consensus_meta.cc:287] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.689s	user 0.012s	sys 0.004s
I20260709 13:02:31.957302  9209 raft_consensus.cc:515] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:31.911973  9058 raft_consensus.cc:3060] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:02:31.906080  9061 raft_consensus.cc:515] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:32.010528  1605 raft_consensus.cc:1275] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 21 index: 16. Preceding OpId from leader: term: 22 index: 17. (index mismatch)
I20260709 13:02:31.898015  9174 raft_consensus.cc:3060] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Advancing to term 20
W20260709 13:02:31.914630  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 269902us
I20260709 13:02:32.006577  1746 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43156 (request call id 1528) took 1220 ms. Trace:
I20260709 13:02:32.034890  1746 rpcz_store.cc:276] 0709 13:02:30.786088 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:31.400176 (+614088us) service_pool.cc:224] Handling call
0709 13:02:31.400500 (+   324us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:31.400793 (+   293us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 13:02:31.615865 (+215072us) log.cc:844] Serialized 14 byte log entry
0709 13:02:32.006147 (+390282us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:32.006223 (+    76us) raft_consensus.cc:1645] Marking committed up to 14
0709 13:02:32.006284 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:32.006352 (+    68us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:32.006508 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":242398,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":10670,"raft.queue_time_us":96,"thread_start_us":604439,"threads_started":2}
I20260709 13:02:32.050554  1746 raft_consensus.cc:1240] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 18 LEADER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 17. Current term is 18. Ops: []
I20260709 13:02:31.994585  9212 raft_consensus.cc:2749] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:02:32.024008  2333 raft_consensus.cc:2468] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 20.
W20260709 13:02:32.073454  9292 log.cc:927] Time spent T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.107s	user 0.000s	sys 0.001s
I20260709 13:02:32.074648  9054 consensus_queue.cc:1048] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 13:02:32.083552  9115 raft_consensus.cc:493] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:02:32.084254  9115 raft_consensus.cc:3060] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:02:32.087239  9205 raft_consensus.cc:3055] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 LEADER]: Stepping down as leader of term 18
I20260709 13:02:32.087674  9205 raft_consensus.cc:740] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
W20260709 13:02:32.092317  9285 log.cc:927] Time spent T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.477s	user 0.002s	sys 0.000s
I20260709 13:02:32.098143  9069 leader_election.cc:290] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:32.104190  9295 raft_consensus.cc:2804] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 13:02:32.104766  9295 raft_consensus.cc:493] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:32.105162  9295 raft_consensus.cc:3060] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Advancing to term 22
W20260709 13:02:32.107030  1645 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:32.104773  2330 raft_consensus.cc:1275] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 13 index: 12. Preceding OpId from leader: term: 16 index: 14. (index mismatch)
I20260709 13:02:32.136489  2333 raft_consensus.cc:1240] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 18. Current term is 19. Ops: [18.16-18.16]
I20260709 13:02:32.141657  2330 raft_consensus.cc:1240] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 25 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 24. Current term is 25. Ops: []
I20260709 13:02:32.017760  1601 raft_consensus.cc:2468] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 18.
I20260709 13:02:32.143399  2330 raft_consensus.cc:1240] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 18. Current term is 19. Ops: [18.14-18.14]
I20260709 13:02:32.064742  1746 raft_consensus.cc:1275] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 19 index: 14. Preceding OpId from leader: term: 20 index: 15. (index mismatch)
I20260709 13:02:32.155150  2330 raft_consensus.cc:1240] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 22. Current term is 23. Ops: []
I20260709 13:02:32.149346  2298 leader_election.cc:304] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:02:32.109681  9296 raft_consensus.cc:2804] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Leader election won for term 18
I20260709 13:02:32.003611  9255 consensus_queue.cc:1059] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 17.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.142s
W20260709 13:02:32.151708  9294 log.cc:927] Time spent T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.083s	user 0.000s	sys 0.004s
W20260709 13:02:32.193498  1570 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:32.194077  1570 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:32.210088  2298 leader_election.cc:304] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d
W20260709 13:02:32.175838  8934 consensus_meta.cc:287] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.773s	user 0.015s	sys 0.000s
I20260709 13:02:32.227643  1679 raft_consensus.cc:1240] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 20. Current term is 21. Ops: []
I20260709 13:02:32.229806  1569 leader_election.cc:304] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
W20260709 13:02:32.235512  1753 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.510s	user 0.003s	sys 0.011s
W20260709 13:02:32.235909  1753 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.510s	user 0.003s	sys 0.011s
I20260709 13:02:32.236707  1569 leader_election.cc:304] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:02:32.177647  9282 leader_election.cc:290] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 22 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:32.172243  9299 raft_consensus.cc:2749] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Leader election lost for term 19. Reason: Vote denied by peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445) with higher term. Message: Invalid argument: T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for earlier term 19. Current term is 20.
I20260709 13:02:32.051959  9213 consensus_queue.cc:237] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 22.14, Last appended by leader: 14, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:32.222599  9002 raft_consensus.cc:2804] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 13:02:32.056136  9250 consensus_queue.cc:260] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 17.10, Last appended by leader: 10, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:32.271152  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 24 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:32.272042  1680 raft_consensus.cc:2468] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 23.
I20260709 13:02:32.265534  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:32.237627  9175 raft_consensus.cc:2804] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:02:32.261762  8981 raft_consensus.cc:2804] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:02:32.285323  1680 raft_consensus.cc:1275] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 14 index: 13. Preceding OpId from leader: term: 16 index: 14. (index mismatch)
I20260709 13:02:32.287247  1753 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55840 (request call id 3191) took 1199 ms. Trace:
W20260709 13:02:32.287278  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 398233us
W20260709 13:02:32.286921  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 534797us
I20260709 13:02:32.217669  9214 raft_consensus.cc:515] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:32.287806  1753 rpcz_store.cc:276] 0709 13:02:31.087659 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:31.675627 (+587968us) service_pool.cc:224] Handling call
0709 13:02:31.675946 (+   319us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:31.676191 (+   245us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 13:02:31.708701 (+ 32510us) log.cc:844] Serialized 14 byte log entry
0709 13:02:32.272796 (+564095us) log.cc:844] Serialized 14 byte log entry
0709 13:02:32.286858 (+ 14062us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:32.286921 (+    63us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:02:32.286980 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:32.287052 (+    72us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:32.287182 (+   130us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":39962,"prepare.queue_time_us":108888,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":38,"raft.queue_time_us":4536,"raft.run_cpu_time_us":152,"raft.run_wall_time_us":8035,"thread_start_us":510764,"threads_started":1}
I20260709 13:02:32.229341  8934 raft_consensus.cc:515] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:32.292795  2301 leader_election.cc:304] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:02:32.300918  9296 raft_consensus.cc:697] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:02:32.301977  9296 consensus_queue.cc:237] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.10, Last appended by leader: 10, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:32.303367  9219 consensus_queue.cc:1059] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 13:02:32.180943  9205 consensus_queue.cc:260] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.14, Last appended by leader: 14, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:32.287050  8772 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.582s	user 0.041s	sys 0.042s
I20260709 13:02:32.331295  1749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55840 (request call id 3188) took 1351 ms. Trace:
W20260709 13:02:32.293497  8968 consensus_meta.cc:287] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.917s	user 0.012s	sys 0.007s
I20260709 13:02:32.282662  9175 raft_consensus.cc:493] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:02:32.292908  9246 consensus_meta.cc:287] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.522s	user 0.008s	sys 0.010s
I20260709 13:02:32.298277  9250 raft_consensus.cc:3060] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:02:32.281392  9002 raft_consensus.cc:697] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 18 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:02:32.285076  8981 raft_consensus.cc:493] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:32.338857  1745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43156 (request call id 1522) took 1556 ms. Trace:
I20260709 13:02:32.333721  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:32.324070  9282 raft_consensus.cc:2804] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 24 FOLLOWER]: Leader election won for term 24
W20260709 13:02:32.327744  8772 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.639s	user 0.041s	sys 0.043s
I20260709 13:02:32.359006  1745 rpcz_store.cc:276] 0709 13:02:30.782396 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:30.783370 (+   974us) service_pool.cc:224] Handling call
0709 13:02:30.783805 (+   435us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:30.795346 (+ 11541us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 13:02:31.241530 (+446184us) spinlock_profiling.cc:243] Waited 2.4 ms on lock 0x7b180031a8c0. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f4263057e07 00007f426beb2ab9 00007f426307469b 00007f426307293f 00007f4263072455 00007f4269b5a3c7 00007f426616c9a7 00007f426617c0bf 00007f426617c038
0709 13:02:31.456722 (+215192us) log.cc:844] Serialized 14 byte log entry
0709 13:02:31.725144 (+268422us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:31.738289 (+ 13145us) log.cc:844] Serialized 26 byte log entry
0709 13:02:31.738552 (+   263us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:02:31.738604 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:31.738705 (+   101us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:32.329086 (+590381us) raft_consensus.cc:1685] finished
0709 13:02:32.338579 (+  9493us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:32.338763 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":30224,"prepare.queue_time_us":754492,"prepare.run_cpu_time_us":52,"prepare.run_wall_time_us":53,"raft.queue_time_us":18496,"raft.run_cpu_time_us":358,"raft.run_wall_time_us":60651,"spinlock_wait_cycles":5527936,"thread_start_us":394625,"threads_started":2,"wal-append.queue_time_us":437876}
I20260709 13:02:32.311189  2301 leader_election.cc:304] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:32.330925  9209 leader_election.cc:290] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 23 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:32.296387  9104 leader_election.cc:290] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:32.343631  9175 raft_consensus.cc:3060] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:02:32.353808  9246 raft_consensus.cc:2749] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:02:32.362223  8981 raft_consensus.cc:3060] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:02:32.331766  9255 consensus_queue.cc:1059] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.040s
I20260709 13:02:32.379760  1675 raft_consensus.cc:1275] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 14 index: 13. Preceding OpId from leader: term: 16 index: 14. (index mismatch)
I20260709 13:02:32.353559  9061 leader_election.cc:290] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 20 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:32.340739  8968 raft_consensus.cc:515] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:02:32.378456  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 67369.5us
I20260709 13:02:32.331799  1749 rpcz_store.cc:276] 0709 13:02:30.979399 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:31.553815 (+574416us) service_pool.cc:224] Handling call
0709 13:02:31.584208 (+ 30393us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:31.584439 (+   231us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 13:02:32.005461 (+421022us) log.cc:844] Serialized 14 byte log entry
0709 13:02:32.330871 (+325410us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:32.330936 (+    65us) raft_consensus.cc:1645] Marking committed up to 13
0709 13:02:32.330996 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:32.331076 (+    80us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:32.331220 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":145124,"prepare.run_cpu_time_us":52,"prepare.run_wall_time_us":54,"raft.queue_time_us":28681,"thread_start_us":691913,"threads_started":2}
W20260709 13:02:32.406490  1570 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:32.415608  2301 leader_election.cc:304] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:02:32.416824  1680 raft_consensus.cc:3055] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 LEADER]: Stepping down as leader of term 15
I20260709 13:02:32.424305  1680 raft_consensus.cc:740] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
W20260709 13:02:32.425071  1598 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.629s	user 0.000s	sys 0.008s
I20260709 13:02:32.426422  1748 raft_consensus.cc:1240] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 18 LEADER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 17. Current term is 18. Ops: []
W20260709 13:02:32.426043  1598 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.630s	user 0.000s	sys 0.008s
I20260709 13:02:32.375344  9282 raft_consensus.cc:697] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 24 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:02:32.427891  9054 raft_consensus.cc:2804] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:02:32.431740  9054 raft_consensus.cc:493] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:32.432157  9054 raft_consensus.cc:3060] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Advancing to term 21
W20260709 13:02:32.432350  9297 log.cc:927] Time spent T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.148s	user 0.000s	sys 0.000s
I20260709 13:02:32.393926  9209 raft_consensus.cc:2804] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 13:02:32.394330  9259 raft_consensus.cc:493] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:32.394104  9308 raft_consensus.cc:3055] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 LEADER]: Stepping down as leader of term 17
I20260709 13:02:32.434636  2301 leader_election.cc:304] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
W20260709 13:02:32.425530  9218 consensus_meta.cc:287] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 1.051s	user 0.017s	sys 0.000s
W20260709 13:02:32.430025  9152 consensus_meta.cc:287] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.660s	user 0.017s	sys 0.000s
I20260709 13:02:32.447268  2301 leader_election.cc:304] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:32.441411  1680 consensus_queue.cc:260] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.9, Last appended by leader: 9, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:32.498868  1680 raft_consensus.cc:3060] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:02:32.500164  1569 leader_election.cc:304] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
W20260709 13:02:32.443794  9202 consensus_meta.cc:287] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.561s	user 0.014s	sys 0.002s
I20260709 13:02:32.453581  9209 raft_consensus.cc:697] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 22 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
W20260709 13:02:32.540964  9301 log.cc:927] Time spent T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.075s	user 0.000s	sys 0.000s
I20260709 13:02:32.541798  9247 raft_consensus.cc:493] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:32.542663  9247 raft_consensus.cc:515] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:32.545070  9311 raft_consensus.cc:2804] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 13:02:32.545722  9311 raft_consensus.cc:697] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 16 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:02:32.443038  9308 raft_consensus.cc:740] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
W20260709 13:02:32.547087  9307 log.cc:927] Time spent T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.109s	user 0.000s	sys 0.000s
I20260709 13:02:32.547353  9308 consensus_queue.cc:260] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.12, Last appended by leader: 12, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:32.548588  9308 raft_consensus.cc:3060] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:02:32.550709  8772 raft_consensus.cc:493] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
W20260709 13:02:32.551251  1644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:32.551762  1644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:32.551412  8772 raft_consensus.cc:515] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:32.552438  1644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:32.552875  1644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:32.515508  9255 consensus_queue.cc:1243] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer 3b879799899a499c8d409ee9f1277cd1 log is divergent from this leader: its last log entry 16.13 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 12
I20260709 13:02:32.531141  9202 raft_consensus.cc:515] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:32.561472  9202 leader_election.cc:290] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 21 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:02:32.563331  9305 log.cc:927] Time spent T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.138s	user 0.000s	sys 0.000s
I20260709 13:02:32.446162  9259 raft_consensus.cc:515] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:02:32.567811  2336 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39726 (request call id 1654) took 3001 ms (client timeout 1829 ms). Trace:
W20260709 13:02:32.568253  2336 rpcz_store.cc:269] 0709 13:02:29.566476 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:29.567176 (+   700us) service_pool.cc:224] Handling call
0709 13:02:32.567527 (+3000351us) spinlock_profiling.cc:243] Waited 1.73 s on lock 0x7b540065e0d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:02:32.567699 (+   172us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3989394816}
I20260709 13:02:32.467998  9218 raft_consensus.cc:2749] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20260709 13:02:32.538103  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:32.571895  1645 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:32.567452  1746 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55840 (request call id 3193) took 1134 ms. Trace:
I20260709 13:02:32.573287  1746 rpcz_store.cc:276] 0709 13:02:31.433502 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:32.214711 (+781209us) service_pool.cc:224] Handling call
0709 13:02:32.215088 (+   377us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:32.226677 (+ 11589us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:32.226713 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:32.262020 (+ 35307us) log.cc:844] Serialized 26 byte log entry
0709 13:02:32.451700 (+189680us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:02:32.451768 (+    68us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:32.451884 (+   116us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:32.567152 (+115268us) raft_consensus.cc:1685] finished
0709 13:02:32.567181 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:32.567371 (+   190us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":17753,"raft.run_cpu_time_us":182,"raft.run_wall_time_us":21352,"thread_start_us":159959,"threads_started":1,"wal-append.queue_time_us":289281}
W20260709 13:02:32.575076  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:32.580811  2299 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:32.531747  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 241964us
I20260709 13:02:32.586930  2332 raft_consensus.cc:1275] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 19 index: 15. Preceding OpId from leader: term: 21 index: 16. (index mismatch)
I20260709 13:02:32.588469  1754 raft_consensus.cc:1240] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 19. Current term is 20. Ops: []
I20260709 13:02:32.439327  9002 consensus_queue.cc:237] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.13, Last appended by leader: 13, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:32.592528  9303 log.cc:927] Time spent T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.212s	user 0.000s	sys 0.001s
I20260709 13:02:32.519573  9296 raft_consensus.cc:3055] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 LEADER]: Stepping down as leader of term 19
W20260709 13:02:32.615907  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 168890us
I20260709 13:02:32.617188  2301 leader_election.cc:304] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:02:32.532716  9260 raft_consensus.cc:493] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
W20260709 13:02:32.674816  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 57797.7us
I20260709 13:02:32.676126  2301 leader_election.cc:304] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:02:32.687338  1680 raft_consensus.cc:1275] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 15 index: 9. Preceding OpId from leader: term: 16 index: 10. (index mismatch)
I20260709 13:02:32.694134  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
W20260709 13:02:32.700047  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 200086us
I20260709 13:02:32.700187  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:32.701638  1680 raft_consensus.cc:3060] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:02:32.704597  9072 raft_consensus.cc:2804] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 13:02:32.705266  9072 raft_consensus.cc:697] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 20 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:02:32.711267  8968 leader_election.cc:290] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 20 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:32.715287  9150 consensus_queue.cc:1048] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 13:02:32.715112  9311 consensus_queue.cc:237] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:32.716910  9150 consensus_queue.cc:1048] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 13:02:32.724725  9148 leader_election.cc:290] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 21 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:32.517627  9061 consensus_queue.cc:1059] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.077s
I20260709 13:02:32.727025  9061 raft_consensus.cc:3055] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 LEADER]: Stepping down as leader of term 20
I20260709 13:02:32.727403  9061 raft_consensus.cc:740] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:02:32.728308  9061 consensus_queue.cc:260] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 20.11, Last appended by leader: 11, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:32.729626  9061 raft_consensus.cc:3060] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:02:32.731525  9197 raft_consensus.cc:3055] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 LEADER]: Stepping down as leader of term 18
I20260709 13:02:32.731987  9197 raft_consensus.cc:740] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:02:32.732918  9197 consensus_queue.cc:260] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.16, Last appended by leader: 16, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:32.734047  9197 raft_consensus.cc:3060] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:02:32.736162  2333 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49082 (request call id 1358) took 1015 ms. Trace:
I20260709 13:02:32.736547  2333 rpcz_store.cc:276] 0709 13:02:31.721191 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:32.147170 (+425979us) service_pool.cc:224] Handling call
0709 13:02:32.147495 (+   325us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:32.147943 (+   448us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:32.147975 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:32.176963 (+ 28988us) log.cc:844] Serialized 26 byte log entry
0709 13:02:32.396966 (+220003us) raft_consensus.cc:1645] Marking committed up to 13
0709 13:02:32.397023 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:32.397123 (+   100us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:32.735911 (+338788us) raft_consensus.cc:1685] finished
0709 13:02:32.735935 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:32.736092 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":10637,"raft.run_cpu_time_us":182,"raft.run_wall_time_us":3405,"thread_start_us":151128,"threads_started":1,"wal-append.queue_time_us":482850}
I20260709 13:02:32.746749  9331 raft_consensus.cc:493] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:32.747392  9331 raft_consensus.cc:515] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:32.749886  9331 leader_election.cc:290] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:32.593698  9322 raft_consensus.cc:493] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:32.751014  9322 raft_consensus.cc:515] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:32.753693  9322 leader_election.cc:290] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:02:32.754779  9227 consensus_meta.cc:287] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.844s	user 0.010s	sys 0.007s
I20260709 13:02:32.755300  9227 raft_consensus.cc:515] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:32.757450  9227 leader_election.cc:290] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 22 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:32.758080  9232 leader_election.cc:290] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 20 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:32.766285  2332 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39726 (request call id 1670) took 1147 ms. Trace:
I20260709 13:02:32.774786  2332 rpcz_store.cc:276] 0709 13:02:31.619221 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:31.619982 (+   761us) service_pool.cc:224] Handling call
0709 13:02:32.569736 (+949754us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:32.587385 (+ 17649us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:32.765988 (+178603us) spinlock_profiling.cc:243] Waited 893 ms on lock 0x7b540065e0d0. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661618ad 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:02:32.766197 (+   209us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2053179264}
I20260709 13:02:32.780054  1748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55848 (request call id 3357) took 1515 ms. Trace:
I20260709 13:02:32.780514  9300 raft_consensus.cc:493] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:02:32.780452  9054 raft_consensus.cc:515] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:32.781344  9300 raft_consensus.cc:515] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:32.780469  1748 rpcz_store.cc:276] 0709 13:02:31.264796 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:31.996173 (+731377us) service_pool.cc:224] Handling call
0709 13:02:31.996513 (+   340us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:32.426914 (+430401us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:32.779785 (+352871us) spinlock_profiling.cc:243] Waited 430 ms on lock 0x7b54000bd954. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:02:32.779983 (+   198us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":988618112}
I20260709 13:02:32.782922  8772 leader_election.cc:290] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:32.783787  9259 leader_election.cc:290] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:32.608505  9296 raft_consensus.cc:740] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:02:32.764637  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 25 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:02:32.734282  9088 consensus_meta.cc:287] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.785s	user 0.013s	sys 0.005s
I20260709 13:02:32.784932  9296 consensus_queue.cc:260] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.14, Last appended by leader: 14, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:32.785082  9088 raft_consensus.cc:515] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:32.785570  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 22 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:32.786083  9296 raft_consensus.cc:3060] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Advancing to term 20
W20260709 13:02:32.761524  9319 log.cc:927] Time spent T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.367s	user 0.001s	sys 0.001s
I20260709 13:02:32.768236  1754 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55848 (request call id 3361) took 1496 ms. Trace:
I20260709 13:02:32.769809  8893 raft_consensus.cc:2749] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:02:32.770766  8934 leader_election.cc:290] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 17 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:32.771560  9209 consensus_queue.cc:237] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.15, Last appended by leader: 15, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:32.773422  9328 log.cc:927] Time spent T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.074s	user 0.000s	sys 0.000s
I20260709 13:02:32.682797  9214 leader_election.cc:290] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 22 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:32.597633  9282 consensus_queue.cc:237] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 23.17, Last appended by leader: 17, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:32.766131  9035 consensus_meta.cc:287] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.878s	user 0.010s	sys 0.001s
I20260709 13:02:32.792745  1717 leader_election.cc:304] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:32.786353  1673 raft_consensus.cc:1240] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 15. Current term is 16. Ops: []
I20260709 13:02:32.803128  1754 rpcz_store.cc:276] 0709 13:02:31.271792 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:32.105355 (+833563us) service_pool.cc:224] Handling call
0709 13:02:32.105706 (+   351us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:32.589024 (+483318us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:32.767976 (+178952us) spinlock_profiling.cc:243] Waited 483 ms on lock 0x7b54000e4f54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:02:32.768168 (+   192us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1110141824}
W20260709 13:02:32.804519  1570 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:32.804659  1646 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:32.828328  1571 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:32.837108  1601 raft_consensus.cc:3055] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 24 LEADER]: Stepping down as leader of term 24
I20260709 13:02:32.837503  1601 raft_consensus.cc:740] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
W20260709 13:02:32.818614  8952 consensus_meta.cc:287] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.938s	user 0.015s	sys 0.000s
I20260709 13:02:32.679490  9226 raft_consensus.cc:493] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:32.876854  9226 raft_consensus.cc:3060] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:02:32.878921  2335 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49082 (request call id 1350) took 1237 ms. Trace:
I20260709 13:02:32.879480  2335 rpcz_store.cc:276] 0709 13:02:31.641826 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:31.922034 (+280208us) service_pool.cc:224] Handling call
0709 13:02:31.962206 (+ 40172us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:31.963432 (+  1226us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 13:02:32.394909 (+431477us) log.cc:844] Serialized 14 byte log entry
0709 13:02:32.435799 (+ 40890us) log.cc:844] Serialized 14 byte log entry
0709 13:02:32.443924 (+  8125us) log.cc:844] Serialized 14 byte log entry
0709 13:02:32.544550 (+100626us) spinlock_profiling.cc:243] Waited 10.6 ms on lock 0x7b18002f1a00. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f4263075522 00007f42630777c1 00007f4263077779 00007f4263077709 00007f42630776d1 00007f42630769cd 000000000038f354 000000000038f199 00007f426305883a
0709 13:02:32.878237 (+333687us) spinlock_profiling.cc:243] Waited 28.1 ms on lock 0x7b18002f1a00. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f4265d0a4bc 00007f426305848b 00007f426beb2ab9 00007f426307469b 00007f426307293f 00007f4263072455 00007f426615a060 00007f426616c716 00007f426617d287
0709 13:02:32.878385 (+   148us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:32.878468 (+    83us) raft_consensus.cc:1645] Marking committed up to 14
0709 13:02:32.878552 (+    84us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:32.878664 (+   112us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:32.878812 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":49051,"prepare.queue_time_us":1153778,"prepare.run_cpu_time_us":116,"prepare.run_wall_time_us":121,"raft.queue_time_us":111451,"raft.run_cpu_time_us":464,"raft.run_wall_time_us":31035,"spinlock_wait_cycles":248475008,"thread_start_us":530949,"threads_started":3,"wal-append.queue_time_us":199636}
I20260709 13:02:32.831532  9213 raft_consensus.cc:493] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:32.804440  9336 raft_consensus.cc:493] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:32.841992  9088 leader_election.cc:290] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 16 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:32.842773  9054 leader_election.cc:290] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 21 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:32.842926  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:32.882941  9336 raft_consensus.cc:515] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:32.884195  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:32.884336  9214 consensus_queue.cc:1048] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 13:02:32.846410  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 22 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:32.885886  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:32.886389  9212 raft_consensus.cc:2804] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:02:32.887176  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:32.838510  1601 consensus_queue.cc:260] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 24.16, Last appended by leader: 16, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:32.888150  1601 raft_consensus.cc:3060] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 24 FOLLOWER]: Advancing to term 25
I20260709 13:02:32.888310  9212 raft_consensus.cc:493] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:02:32.887928  1718 leader_election.cc:304] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:32.888944  9212 raft_consensus.cc:3060] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:02:32.788506  9244 consensus_queue.cc:1048] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.101s
W20260709 13:02:32.849779  9058 consensus_meta.cc:287] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.855s	user 0.013s	sys 0.000s
I20260709 13:02:32.854197  1746 raft_consensus.cc:1240] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 21. Current term is 22. Ops: []
I20260709 13:02:32.854830  1680 raft_consensus.cc:2468] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 22.
I20260709 13:02:32.891371  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:32.891688  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:32.892547  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:32.893091  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:32.854979  9247 leader_election.cc:290] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:32.855890  1603 raft_consensus.cc:1275] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 22 index: 14. (index mismatch)
I20260709 13:02:32.894223  9058 raft_consensus.cc:515] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:32.894606  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:32.894807  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:32.895241  1602 raft_consensus.cc:3060] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:02:32.857143  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:32.896255  1597 raft_consensus.cc:2410] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for term 21 because replica has last-logged OpId of term: 20 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 10.
I20260709 13:02:32.896458  1718 leader_election.cc:304] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:32.896889  1599 raft_consensus.cc:2393] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 16: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:02:32.893702  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 16 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
W20260709 13:02:32.819360  9295 consensus_meta.cc:287] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.711s	user 0.011s	sys 0.005s
I20260709 13:02:32.898448  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:32.860352  1752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55840 (request call id 3185) took 2045 ms. Trace:
I20260709 13:02:32.861456  1754 raft_consensus.cc:1240] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 18 LEADER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 17. Current term is 18. Ops: []
I20260709 13:02:32.899627  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 22 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:32.899008  9295 raft_consensus.cc:515] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:32.900424  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:32.900776  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:32.901656  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:32.901963  9295 leader_election.cc:290] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 22 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:32.706308  9072 consensus_queue.cc:237] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.13, Last appended by leader: 13, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:32.862834  9215 raft_consensus.cc:2749] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:02:32.863337  1747 raft_consensus.cc:1240] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 19. Current term is 20. Ops: []
I20260709 13:02:32.903607  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:32.904779  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:32.905319  1747 raft_consensus.cc:2468] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 16.
I20260709 13:02:32.864825  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:32.899013  1752 rpcz_store.cc:276] 0709 13:02:30.815109 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:31.429474 (+614365us) service_pool.cc:224] Handling call
0709 13:02:31.547034 (+117560us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:31.566731 (+ 19697us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:32.860111 (+1293380us) spinlock_profiling.cc:243] Waited 117 ms on lock 0x7b54001711d0. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661618ad 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:02:32.860276 (+   165us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":269621632}
I20260709 13:02:32.739466  9152 raft_consensus.cc:2804] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 13:02:32.866353  1745 raft_consensus.cc:1275] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 19 index: 15. Preceding OpId from leader: term: 21 index: 16. (index mismatch)
I20260709 13:02:32.872241  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:32.873991  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:32.875206  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:02:32.875785  9325 log.cc:927] Time spent T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.131s	user 0.001s	sys 0.000s
W20260709 13:02:32.876263  9115 consensus_meta.cc:287] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.792s	user 0.008s	sys 0.007s
W20260709 13:02:32.841322  9329 log.cc:927] Time spent T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.106s	user 0.000s	sys 0.000s
W20260709 13:02:32.844471  9120 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 16 current term: 16 is higher.
I20260709 13:02:32.891525  9213 raft_consensus.cc:515] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:32.893694  1746 raft_consensus.cc:3060] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Advancing to term 21
W20260709 13:02:32.855039  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 62462.3us
W20260709 13:02:32.916286  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 240433us
I20260709 13:02:32.898530  9232 raft_consensus.cc:2804] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 13:02:32.862352  1748 raft_consensus.cc:1240] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 19. Current term is 20. Ops: []
I20260709 13:02:32.908002  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:32.943533  1752 raft_consensus.cc:2468] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 19.
I20260709 13:02:32.944687  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:32.865135  9175 raft_consensus.cc:515] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } }
I20260709 13:02:32.902235  1754 raft_consensus.cc:2468] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 24.
I20260709 13:02:32.906172  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 23 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:32.947562  1747 raft_consensus.cc:1240] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 21. Current term is 22. Ops: []
W20260709 13:02:32.948287  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:32.949254  1747 raft_consensus.cc:1217] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 19 LEADER]: Deduplicated request from leader. Original: 16.14->[17.15-18.16]   Dedup: 17.15->[18.16-18.16]
I20260709 13:02:32.930976  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:32.913371  1646 leader_election.cc:304] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:32.915158  1718 leader_election.cc:304] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:02:32.930984  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 25 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:32.950796  1745 raft_consensus.cc:2468] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 24.
I20260709 13:02:32.952771  1747 raft_consensus.cc:1240] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 19 LEADER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 18. Current term is 19. Ops: [17.15-18.16]
I20260709 13:02:32.936575  1717 leader_election.cc:304] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 13:02:32.895435  9214 raft_consensus.cc:2804] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 13:02:32.937711  1674 raft_consensus.cc:2468] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 16.
I20260709 13:02:32.938395  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 24 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:32.938699  1671 raft_consensus.cc:2393] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 22: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:02:32.945286  1752 raft_consensus.cc:2468] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 20.
I20260709 13:02:32.939885  2301 leader_election.cc:304] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:32.949371  1601 raft_consensus.cc:1275] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 25 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 24 index: 16. Preceding OpId from leader: term: 25 index: 16. (term mismatch)
I20260709 13:02:32.929795  1678 raft_consensus.cc:2468] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 24.
W20260709 13:02:32.913792  9174 consensus_meta.cc:287] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.856s	user 0.013s	sys 0.001s
W20260709 13:02:32.910830  9253 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.642s	user 0.010s	sys 0.036s
I20260709 13:02:32.959467  1601 pending_rounds.cc:85] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a: Aborting all ops after (but not including) 15
I20260709 13:02:32.960547  1745 raft_consensus.cc:1275] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 15 index: 9. Preceding OpId from leader: term: 16 index: 10. (index mismatch)
W20260709 13:02:32.906286  9330 log.cc:927] Time spent T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.159s	user 0.001s	sys 0.000s
I20260709 13:02:32.882851  8952 raft_consensus.cc:515] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:32.962718  2299 leader_election.cc:304] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:32.965113  1601 pending_rounds.cc:107] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a: Aborting uncommitted NO_OP operation due to leader change: 24.16
I20260709 13:02:32.943003  8934 raft_consensus.cc:493] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:32.937959  8994 consensus_queue.cc:1048] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 7, Time since last communication: 0.028s
I20260709 13:02:32.966930  9133 raft_consensus.cc:493] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:32.967589  9133 raft_consensus.cc:515] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:32.962304  9334 raft_consensus.cc:493] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:32.968681  9334 raft_consensus.cc:515] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:32.969939  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 19 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:32.955242  9215 raft_consensus.cc:493] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
W20260709 13:02:32.970978  9253 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.708s	user 0.013s	sys 0.037s
I20260709 13:02:32.971393  9215 raft_consensus.cc:515] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:32.971789  9227 raft_consensus.cc:2749] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 13:02:32.972548  9227 raft_consensus.cc:2804] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:02:32.972889  9227 raft_consensus.cc:493] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:32.973208  9227 raft_consensus.cc:3060] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:02:32.973443  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:32.974615  9232 raft_consensus.cc:697] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 21 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:02:32.975826  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:32.975667  9232 consensus_queue.cc:237] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.13, Last appended by leader: 13, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:32.977105  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:32.977866  1603 raft_consensus.cc:2393] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 20: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:02:32.978726  1598 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40154 (request call id 2987) took 1429 ms. Trace:
I20260709 13:02:32.979717  9175 leader_election.cc:290] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 19 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:32.965468  1601 raft_consensus.cc:2889] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 25 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:02:32.979144  1598 rpcz_store.cc:276] 0709 13:02:31.549313 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:31.724249 (+174936us) service_pool.cc:224] Handling call
0709 13:02:31.790628 (+ 66379us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:31.794619 (+  3991us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:31.794675 (+    56us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:31.795414 (+   739us) log.cc:844] Serialized 26 byte log entry
0709 13:02:32.427455 (+632041us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:02:32.481064 (+ 53609us) log.cc:844] Serialized 14 byte log entry
0709 13:02:32.688569 (+207505us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:32.689449 (+   880us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:32.978506 (+289057us) raft_consensus.cc:1685] finished
0709 13:02:32.978530 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:32.978662 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":53132,"prepare.run_cpu_time_us":63,"prepare.run_wall_time_us":66,"raft.queue_time_us":65466,"raft.run_cpu_time_us":302,"raft.run_wall_time_us":9628,"thread_start_us":837167,"threads_started":2,"wal-append.queue_time_us":721182}
I20260709 13:02:32.974205  1599 raft_consensus.cc:2468] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 16.
I20260709 13:02:32.982534  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 23 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:32.929795  9152 raft_consensus.cc:493] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:32.983338  1601 raft_consensus.cc:2468] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 22.
I20260709 13:02:32.983448  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:02:32.986577  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 73353.9us
I20260709 13:02:32.983466  1717 leader_election.cc:304] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:32.916880  9202 raft_consensus.cc:493] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
W20260709 13:02:32.992030  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:32.922394  9336 leader_election.cc:290] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:32.992998  2299 leader_election.cc:304] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:32.993376  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
W20260709 13:02:32.993829  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 80873.6us
I20260709 13:02:32.995991  2299 leader_election.cc:304] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:02:32.997834  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:32.998127  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
W20260709 13:02:32.999446  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 59699.4us
I20260709 13:02:32.999608  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:32.987038  1601 raft_consensus.cc:1240] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 15. Current term is 16. Ops: [15.10-15.10]
I20260709 13:02:32.988942  1646 leader_election.cc:304] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:32.738883  9260 raft_consensus.cc:515] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:33.005968  1601 raft_consensus.cc:1240] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 17. Current term is 18. Ops: [17.13-17.13]
I20260709 13:02:32.970106  9133 leader_election.cc:290] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:33.002381  2337 raft_consensus.cc:1275] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 17 index: 15. Preceding OpId from leader: term: 19 index: 16. (index mismatch)
I20260709 13:02:33.011096  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 22 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:33.011394  1604 raft_consensus.cc:2393] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 23: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:02:32.990821  9152 raft_consensus.cc:3060] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 23 FOLLOWER]: Advancing to term 24
I20260709 13:02:33.016204  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:33.002946  2333 raft_consensus.cc:1275] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 25 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 23 index: 15. Preceding OpId from leader: term: 25 index: 16. (index mismatch)
I20260709 13:02:32.966269  8934 raft_consensus.cc:515] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:33.004408  1717 leader_election.cc:304] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:02:33.005146  2334 raft_consensus.cc:1275] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 18 index: 12. Preceding OpId from leader: term: 19 index: 13. (index mismatch)
I20260709 13:02:32.991979  9295 raft_consensus.cc:493] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:33.037029  1717 leader_election.cc:304] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:33.003125  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 16 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
W20260709 13:02:33.040030  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:33.001574  9335 raft_consensus.cc:2749] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 13:02:32.973500  9215 leader_election.cc:290] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:32.938530  9115 raft_consensus.cc:515] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:33.012827  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 22 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:33.002914  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 22 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:32.984383  9054 raft_consensus.cc:2749] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:02:32.984517  9174 raft_consensus.cc:515] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:32.986352  9214 raft_consensus.cc:697] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 19 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:02:33.075626  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:33.075376  9214 consensus_queue.cc:237] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:33.076649  9293 log.cc:927] Time spent T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.779s	user 0.000s	sys 0.003s
I20260709 13:02:33.077059  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:33.078872  2332 raft_consensus.cc:2410] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for term 22 because replica has last-logged OpId of term: 21 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 13.
I20260709 13:02:33.078296  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 21 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
W20260709 13:02:33.079362  9310 log.cc:927] Time spent T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.329s	user 0.000s	sys 0.001s
I20260709 13:02:33.080368  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 17 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:33.047482  9295 raft_consensus.cc:515] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:33.080833  1604 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40154 (request call id 2978) took 1533 ms (client timeout 1517 ms). Trace:
W20260709 13:02:33.081269  1604 rpcz_store.cc:269] 0709 13:02:31.546892 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:31.550556 (+  3664us) service_pool.cc:224] Handling call
0709 13:02:33.080599 (+1530043us) spinlock_profiling.cc:243] Waited 1.43 s on lock 0x7b5400301954. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:02:33.080751 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3279106816}
I20260709 13:02:33.081637  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 23 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:33.022240  9133 raft_consensus.cc:2804] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 13:02:33.082746  1597 raft_consensus.cc:1240] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 23 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 22. Current term is 23. Ops: []
I20260709 13:02:33.023561  9334 leader_election.cc:290] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:33.024330  9226 raft_consensus.cc:515] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:33.024926  9300 leader_election.cc:290] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:33.026336  8952 leader_election.cc:290] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 23 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:33.027127  9203 consensus_queue.cc:1048] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20260709 13:02:33.028501  9252 consensus_queue.cc:1048] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20260709 13:02:32.995236  9323 raft_consensus.cc:2804] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Leader election won for term 21
I20260709 13:02:33.034471  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:33.000270  2301 leader_election.cc:304] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
W20260709 13:02:33.041680  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:33.007486  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:32.992707  1603 raft_consensus.cc:1240] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 19. Current term is 20. Ops: []
I20260709 13:02:33.008639  9250 raft_consensus.cc:2764] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Leader election decision vote started in defunct term 21: won
I20260709 13:02:33.009155  9244 raft_consensus.cc:2749] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20260709 13:02:33.058779  1718 leader_election.cc:336] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Timed out: RequestConsensusVote RPC to 127.0.107.194:46261 timed out after 1.919s (SENT)
W20260709 13:02:33.066152  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 331627us
I20260709 13:02:33.092492  9220 raft_consensus.cc:2749] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:02:33.067049  1571 leader_election.cc:304] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:02:33.048192  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 22 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:33.046538  1717 leader_election.cc:304] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:33.074064  8934 leader_election.cc:290] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:33.074231  9205 raft_consensus.cc:3060] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:02:33.083628  1751 raft_consensus.cc:1240] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 19. Current term is 20. Ops: []
I20260709 13:02:33.120060  9133 raft_consensus.cc:697] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 22 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:02:33.083801  1671 raft_consensus.cc:2410] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for term 23 because replica has last-logged OpId of term: 22 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 13.
I20260709 13:02:33.124245  9323 raft_consensus.cc:697] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 21 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:02:33.021301  9061 raft_consensus.cc:493] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:33.147583  9133 consensus_queue.cc:237] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.15, Last appended by leader: 15, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:33.148092  9061 raft_consensus.cc:515] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:33.086027  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 22 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:33.150714  9215 raft_consensus.cc:2749] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 13:02:33.134785  9339 raft_consensus.cc:2749] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 13:02:33.151403  9174 leader_election.cc:290] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 20 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:33.086046  2299 leader_election.cc:304] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:33.152704  9213 leader_election.cc:290] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:02:33.152966  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 67127.5us
I20260709 13:02:33.003535  9202 raft_consensus.cc:515] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:33.090056  2331 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49082 (request call id 1353) took 1438 ms. Trace:
I20260709 13:02:33.090970  1602 raft_consensus.cc:2468] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 21.
I20260709 13:02:33.039855  9218 raft_consensus.cc:2749] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:02:33.093964  1679 raft_consensus.cc:1240] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 18. Current term is 19. Ops: []
W20260709 13:02:33.095010  1645 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:33.048595  8994 raft_consensus.cc:2804] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 13:02:33.047315  9159 consensus_queue.cc:1059] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 18.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.057s
I20260709 13:02:33.047668  9120 consensus_queue.cc:1059] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 20.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.052s
I20260709 13:02:33.096570  1643 leader_election.cc:304] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:02:33.103833  1746 raft_consensus.cc:2468] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 21.
I20260709 13:02:33.123711  1598 raft_consensus.cc:1275] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 18 index: 11. (index mismatch)
W20260709 13:02:33.125061  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 252069us
W20260709 13:02:33.126350  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 126231us
I20260709 13:02:33.049185  9201 consensus_queue.cc:1048] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.068s
W20260709 13:02:33.138538  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 92160.6us
W20260709 13:02:33.132860  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 65958.3us
I20260709 13:02:33.128265  9334 raft_consensus.cc:2749] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:02:33.136792  1646 leader_election.cc:304] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:02:33.127956  9311 raft_consensus.cc:493] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:33.154601  9346 raft_consensus.cc:2749] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 13:02:33.155284  2331 rpcz_store.cc:276] 0709 13:02:31.651620 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:31.962375 (+310755us) service_pool.cc:224] Handling call
0709 13:02:31.962694 (+   319us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:31.962929 (+   235us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 13:02:32.376652 (+413723us) log.cc:844] Serialized 14 byte log entry
0709 13:02:33.088252 (+711600us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:33.088334 (+    82us) raft_consensus.cc:1645] Marking committed up to 13
0709 13:02:33.088421 (+    87us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:33.089860 (+  1439us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:33.089990 (+   130us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":509163,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":50,"raft.queue_time_us":86107,"raft.run_cpu_time_us":100,"raft.run_wall_time_us":100,"thread_start_us":1077602,"threads_started":3,"wal-append.queue_time_us":357889}
W20260709 13:02:33.126132  9002 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.533s	user 0.012s	sys 0.023s
I20260709 13:02:33.121647  8952 raft_consensus.cc:493] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:33.163520  8994 raft_consensus.cc:493] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:02:33.167421  2301 leader_election.cc:304] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:02:33.133742  2335 raft_consensus.cc:2468] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 21.
I20260709 13:02:33.170830  1717 leader_election.cc:304] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:33.128599  1718 leader_election.cc:304] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:02:33.169896  1571 leader_election.cc:304] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
W20260709 13:02:33.172689  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 76252.7us
I20260709 13:02:33.154598  9226 leader_election.cc:290] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 23 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:33.137566  9300 raft_consensus.cc:493] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
W20260709 13:02:33.128865  8981 consensus_meta.cc:287] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.711s	user 0.009s	sys 0.005s
W20260709 13:02:33.188257  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 51630.4us
W20260709 13:02:33.207453  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 148797us
W20260709 13:02:33.174950  9308 consensus_meta.cc:287] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.626s	user 0.014s	sys 0.000s
I20260709 13:02:33.162097  9148 consensus_queue.cc:1059] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.026s
I20260709 13:02:33.168488  8968 raft_consensus.cc:2764] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Leader pre-election decision vote started in defunct term 20: won
I20260709 13:02:33.199201  1643 leader_election.cc:304] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:02:33.173753  9213 raft_consensus.cc:3055] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 LEADER]: Stepping down as leader of term 21
I20260709 13:02:33.089107  9306 consensus_queue.cc:1048] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.086s
W20260709 13:02:33.185312  9002 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.585s	user 0.013s	sys 0.023s
I20260709 13:02:33.210805  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:33.211061  1717 leader_election.cc:304] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:33.214221  1646 leader_election.cc:304] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:02:33.206259  2301 leader_election.cc:304] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:02:33.218215  1571 leader_election.cc:304] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:02:33.186991  9295 leader_election.cc:290] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:33.205144  9088 raft_consensus.cc:2749] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 13:02:33.148161  9323 consensus_queue.cc:237] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.10, Last appended by leader: 10, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:33.202255  9339 raft_consensus.cc:3055] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 LEADER]: Stepping down as leader of term 20
I20260709 13:02:33.200680  8994 raft_consensus.cc:3060] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:02:33.205410  9244 raft_consensus.cc:2749] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:02:33.222045  1717 leader_election.cc:304] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:33.189970  9351 raft_consensus.cc:2804] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:02:33.209841  9346 raft_consensus.cc:3060] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:02:33.216214  9159 raft_consensus.cc:2804] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:02:33.243940  9159 raft_consensus.cc:493] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:33.244464  9159 raft_consensus.cc:3060] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:02:33.208976  9061 leader_election.cc:290] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:33.246642  9002 raft_consensus.cc:2749] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:02:33.234254  9308 raft_consensus.cc:3060] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:02:33.216969  9213 raft_consensus.cc:740] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:02:33.200558  8952 raft_consensus.cc:515] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:33.250316  9213 consensus_queue.cc:260] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 21.16, Last appended by leader: 16, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:33.233307  1646 leader_election.cc:304] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:33.206826  9218 consensus_queue.cc:1048] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.024s
I20260709 13:02:33.252094  9213 raft_consensus.cc:3060] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:02:33.252224  1646 leader_election.cc:304] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:33.252653  8952 leader_election.cc:290] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:33.220913  9120 consensus_queue.cc:1048] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.032s
I20260709 13:02:33.253343  9148 raft_consensus.cc:2804] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:02:33.253775  9148 raft_consensus.cc:493] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:33.254153  9148 raft_consensus.cc:3060] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:02:33.178903  9311 raft_consensus.cc:515] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:33.286651  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 17 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:33.288846  9295 consensus_queue.cc:1059] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 19.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 13:02:33.234781  9260 leader_election.cc:290] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:33.290625  9115 leader_election.cc:290] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:33.215200  8772 consensus_queue.cc:1059] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.023s
I20260709 13:02:33.230465  9207 raft_consensus.cc:493] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:33.292819  9207 raft_consensus.cc:515] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:33.251034  9061 raft_consensus.cc:493] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:33.212191  9203 raft_consensus.cc:493] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:02:33.212226  9300 raft_consensus.cc:515] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:33.254884  1645 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:33.294984  9203 raft_consensus.cc:515] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:33.209704  9252 raft_consensus.cc:2749] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 13:02:33.254663  9331 raft_consensus.cc:2804] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 13:02:33.298764  9331 raft_consensus.cc:493] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:33.299331  9331 raft_consensus.cc:3060] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 24 FOLLOWER]: Advancing to term 25
I20260709 13:02:33.300772  9205 raft_consensus.cc:493] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:33.254225  1646 leader_election.cc:304] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:33.266999  9202 leader_election.cc:290] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:33.223734  9246 raft_consensus.cc:2804] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Leader pre-election won for term 23
W20260709 13:02:33.268461  9344 log.cc:927] Time spent T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.058s	user 0.000s	sys 0.000s
I20260709 13:02:33.232412  9253 raft_consensus.cc:2804] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 13:02:33.228129  9232 raft_consensus.cc:2749] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:02:33.233295  9339 raft_consensus.cc:740] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:02:33.270114  9247 consensus_queue.cc:1048] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 13:02:33.271636  9327 raft_consensus.cc:3055] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 17 LEADER]: Stepping down as leader of term 17
I20260709 13:02:33.210124  8981 raft_consensus.cc:515] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:33.340976  9327 raft_consensus.cc:740] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:02:33.342417  9327 consensus_queue.cc:260] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.13, Last appended by leader: 13, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:33.343618  8981 leader_election.cc:290] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 21 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:33.213106  9174 consensus_queue.cc:1048] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.020s
I20260709 13:02:33.344055  9327 raft_consensus.cc:3060] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:02:33.344792  9346 raft_consensus.cc:2749] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:02:33.274679  9201 raft_consensus.cc:2749] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
W20260709 13:02:33.276962  9348 log.cc:927] Time spent T 00000000000000000000000000000000 P 919f6392d4f44fd68987e2c5e7ec37ca: Append to log took a long time: real 0.061s	user 0.000s	sys 0.000s
I20260709 13:02:33.311733  9203 leader_election.cc:290] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:33.316377  9306 raft_consensus.cc:2804] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Leader election won for term 21
I20260709 13:02:33.353354  9306 raft_consensus.cc:697] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:02:33.279027  1717 leader_election.cc:304] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:33.247022  9351 raft_consensus.cc:493] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:33.354537  9351 raft_consensus.cc:3060] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:02:33.355195  9203 raft_consensus.cc:2749] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 13:02:33.354843  9306 consensus_queue.cc:237] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.13, Last appended by leader: 13, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:02:33.354861  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 75991.3us
I20260709 13:02:33.357483  1717 leader_election.cc:304] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:33.289242  1675 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52766 (request call id 3274) took 2276 ms. Trace:
I20260709 13:02:33.256603  1571 leader_election.cc:304] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:02:33.358873  9203 raft_consensus.cc:2749] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 13:02:33.298103  9300 leader_election.cc:290] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:33.228238  9336 consensus_queue.cc:1059] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 21.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.042s
W20260709 13:02:33.359653  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 103182us
I20260709 13:02:33.360347  8981 raft_consensus.cc:2749] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:02:33.360651  9336 raft_consensus.cc:3055] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 LEADER]: Stepping down as leader of term 21
I20260709 13:02:33.361164  9336 raft_consensus.cc:740] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:02:33.358613  1675 rpcz_store.cc:276] 0709 13:02:31.012945 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:31.013231 (+   286us) service_pool.cc:224] Handling call
0709 13:02:31.013679 (+   448us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:32.392563 (+1378884us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:33.288872 (+896309us) spinlock_profiling.cc:243] Waited 1.37 s on lock 0x7b5400058cd4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:02:33.289117 (+   245us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3142195584}
W20260709 13:02:33.303023  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 48897.5us
I20260709 13:02:33.303314  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 19 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:33.363359  1604 raft_consensus.cc:2468] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 18.
I20260709 13:02:33.367302  1746 raft_consensus.cc:1275] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 24 index: 15. (index mismatch)
I20260709 13:02:33.368973  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:33.369781  9336 consensus_queue.cc:260] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.15, Last appended by leader: 15, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:33.318419  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 112474us
I20260709 13:02:33.316048  9246 raft_consensus.cc:493] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:33.268189  9212 raft_consensus.cc:515] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:33.338764  9352 log.cc:927] Time spent T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.161s	user 0.001s	sys 0.001s
I20260709 13:02:33.339504  9219 raft_consensus.cc:3055] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 LEADER]: Stepping down as leader of term 19
I20260709 13:02:33.339977  9207 leader_election.cc:290] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:33.370719  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:02:33.371690  9320 log.cc:927] Time spent T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.628s	user 0.000s	sys 0.003s
I20260709 13:02:33.372108  1753 raft_consensus.cc:1275] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 15 index: 9. Preceding OpId from leader: term: 18 index: 11. (index mismatch)
I20260709 13:02:33.386875  1717 leader_election.cc:304] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:33.391695  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:02:33.335523  9350 log.cc:927] Time spent T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.077s	user 0.003s	sys 0.003s
I20260709 13:02:33.294873  9061 raft_consensus.cc:515] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:33.373003  1673 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52766 (request call id 3298) took 1224 ms. Trace:
I20260709 13:02:33.409147  1673 rpcz_store.cc:276] 0709 13:02:32.148686 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:32.148862 (+   176us) service_pool.cc:224] Handling call
0709 13:02:32.177509 (+ 28647us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:32.832878 (+655369us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:33.372711 (+539833us) spinlock_profiling.cc:243] Waited 609 ms on lock 0x7b54001f6154. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:02:33.372926 (+   215us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1400434944}
I20260709 13:02:33.327435  9253 raft_consensus.cc:493] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:33.375409  9058 leader_election.cc:290] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 16 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:33.377457  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:33.379010  1747 raft_consensus.cc:1240] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 19. Current term is 20. Ops: [17.8-19.9]
I20260709 13:02:33.317899  2298 leader_election.cc:304] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:33.379515  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:33.416827  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 19 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:33.380494  1643 leader_election.cc:304] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:02:33.392377  1753 raft_consensus.cc:2468] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 21.
I20260709 13:02:33.421139  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:33.372874  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:33.301553  9205 raft_consensus.cc:515] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:33.395294  1748 raft_consensus.cc:1240] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 21. Current term is 22. Ops: [21.14-21.14]
I20260709 13:02:33.395715  9212 leader_election.cc:290] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 20 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:33.396235  9219 raft_consensus.cc:740] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:02:33.396706  9300 consensus_queue.cc:1048] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 13:02:33.389312  9201 consensus_queue.cc:1048] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.009s
I20260709 13:02:33.398308  9174 consensus_queue.cc:1059] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 14.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 13:02:33.385655  9152 raft_consensus.cc:515] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:33.398684  9107 consensus_queue.cc:1048] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 13:02:33.390939  1745 raft_consensus.cc:2468] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 22.
I20260709 13:02:33.454770  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:33.457870  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 19 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:33.459671  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:33.460280  1745 raft_consensus.cc:3060] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Advancing to term 21
W20260709 13:02:33.461637  9312 log.cc:927] Time spent T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.692s	user 0.001s	sys 0.004s
I20260709 13:02:33.462971  1750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55848 (request call id 3347) took 2482 ms. Trace:
I20260709 13:02:33.464067  9152 leader_election.cc:290] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 24 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:33.464838  9088 consensus_queue.cc:1048] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 13:02:33.463344  1750 rpcz_store.cc:276] 0709 13:02:30.980604 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:31.559615 (+579011us) service_pool.cc:224] Handling call
0709 13:02:31.559941 (+   326us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:31.907074 (+347133us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 13:02:31.907123 (+    49us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:31.921625 (+ 14502us) log.cc:844] Serialized 26 byte log entry
0709 13:02:32.300013 (+378388us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:02:32.300062 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:32.773833 (+473771us) spinlock_profiling.cc:243] Waited 347 ms on lock 0x7b5400360954. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:02:32.773891 (+    58us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:33.434369 (+660478us) spinlock_profiling.cc:243] Waited 31 ms on lock 0x7b4400022b4c. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f4262f9bc85 00007f4262f9c6a2 00007f4265d0e1db 00007f4265d09aaf 00007f4265cfd86e 00007f4265cfcbb6 00007f4265cfc7d8 00007f4265d06221 00007f4265d061d9 00007f4265d06169
0709 13:02:33.462709 (+ 28340us) raft_consensus.cc:1685] finished
0709 13:02:33.462736 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:33.462900 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":631,"raft.run_cpu_time_us":182,"raft.run_wall_time_us":8203,"spinlock_wait_cycles":868971264,"thread_start_us":306276,"threads_started":1,"wal-append.queue_time_us":779998}
I20260709 13:02:33.467062  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 22 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:02:33.469043  9313 log.cc:927] Time spent T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.724s	user 0.000s	sys 0.003s
I20260709 13:02:33.474376  1753 raft_consensus.cc:1275] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 20 index: 13. Preceding OpId from leader: term: 22 index: 14. (index mismatch)
I20260709 13:02:33.459772  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:33.488904  1745 raft_consensus.cc:2468] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 21.
I20260709 13:02:33.405704  9322 raft_consensus.cc:3055] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 LEADER]: Stepping down as leader of term 21
W20260709 13:02:33.406076  9318 log.cc:927] Time spent T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.664s	user 0.001s	sys 0.002s
W20260709 13:02:33.351033  9353 log.cc:927] Time spent T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.078s	user 0.005s	sys 0.001s
I20260709 13:02:33.407186  9226 raft_consensus.cc:3055] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 19 LEADER]: Stepping down as leader of term 19
W20260709 13:02:33.407760  9356 log.cc:927] Time spent T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.072s	user 0.000s	sys 0.000s
I20260709 13:02:33.374543  9339 consensus_queue.cc:260] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.14, Last appended by leader: 14, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:33.389810  9246 raft_consensus.cc:3060] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Advancing to term 23
W20260709 13:02:33.412371  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 202465us
I20260709 13:02:33.412964  1717 leader_election.cc:304] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 13:02:33.502946  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 90133.9us
I20260709 13:02:33.274341  9227 raft_consensus.cc:515] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:33.422168  1749 raft_consensus.cc:2468] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 21.
I20260709 13:02:33.423668  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:33.426368  9115 raft_consensus.cc:2749] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 13:02:33.427014  9295 consensus_queue.cc:1059] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 19.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 13:02:33.427862  9244 raft_consensus.cc:2749] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 13:02:33.434417  9336 raft_consensus.cc:3060] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:02:33.437304  9061 leader_election.cc:290] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:02:33.438701  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 121070us
I20260709 13:02:33.493779  9322 raft_consensus.cc:740] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:02:33.494326  9226 raft_consensus.cc:740] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:02:33.450282  9343 raft_consensus.cc:2749] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:02:33.415795  9253 raft_consensus.cc:3060] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 24 FOLLOWER]: Advancing to term 25
I20260709 13:02:33.451385  9308 raft_consensus.cc:2749] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:02:33.451699  9202 consensus_queue.cc:1059] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 16.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.052s
I20260709 13:02:33.448275  8994 raft_consensus.cc:515] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:33.395037  1605 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 1865) took 1317 ms. Trace:
W20260709 13:02:33.410941  1571 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:33.659945  1571 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:33.661330  8994 leader_election.cc:290] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 22 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:33.662930  9061 raft_consensus.cc:493] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:33.663767  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:33.664165  9061 raft_consensus.cc:515] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:33.658903  1605 rpcz_store.cc:276] 0709 13:02:32.077825 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:32.124505 (+ 46680us) service_pool.cc:224] Handling call
0709 13:02:32.172348 (+ 47843us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:32.181661 (+  9313us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:32.181711 (+    50us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:32.182334 (+   623us) log.cc:844] Serialized 26 byte log entry
0709 13:02:32.416277 (+233943us) raft_consensus.cc:1645] Marking committed up to 16
0709 13:02:32.416354 (+    77us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:32.416452 (+    98us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:33.370664 (+954212us) spinlock_profiling.cc:243] Waited 29.5 ms on lock 0x7b4400022b4c. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f4262f9bc85 00007f4262f9c6a2 00007f4265d0e1db 00007f4265d09aaf 00007f4265cfd86e 00007f4265cfcbb6 00007f4265cfc7d8 00007f4265d06221 00007f4265d061d9 00007f4265d06169
0709 13:02:33.394631 (+ 23967us) raft_consensus.cc:1685] finished
0709 13:02:33.394682 (+    51us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:33.394963 (+   281us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":38285,"raft.run_cpu_time_us":167,"raft.run_wall_time_us":2665,"spinlock_wait_cycles":67919232,"thread_start_us":181617,"threads_started":1,"wal-append.queue_time_us":530717}
W20260709 13:02:33.603693  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:33.668468  9061 leader_election.cc:290] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:02:33.668887  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:33.669102  9061 raft_consensus.cc:493] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:02:33.607760  1596 raft_consensus.cc:1240] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 24 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 23. Current term is 24. Ops: []
I20260709 13:02:33.609560  1604 raft_consensus.cc:1275] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
I20260709 13:02:33.609659  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:33.514417  9174 raft_consensus.cc:3060] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:02:33.430850  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:33.404626  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:33.676385  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:33.676717  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:33.678000  1678 raft_consensus.cc:2468] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 21.
I20260709 13:02:33.501570  1718 leader_election.cc:304] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:33.682987  9148 raft_consensus.cc:515] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:33.683753  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:02:33.685194  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 183679us
I20260709 13:02:33.431165  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:33.687032  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:33.688022  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:33.688947  1674 raft_consensus.cc:3060] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Advancing to term 21
W20260709 13:02:33.433799  9197 consensus_meta.cc:287] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.693s	user 0.009s	sys 0.006s
I20260709 13:02:33.691608  9203 raft_consensus.cc:2749] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 13:02:33.640604  2328 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49082 (request call id 1352) took 1997 ms. Trace:
I20260709 13:02:33.692240  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:33.693750  1600 raft_consensus.cc:1275] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 18 index: 16. Preceding OpId from leader: term: 19 index: 16. (term mismatch)
I20260709 13:02:33.694156  1600 pending_rounds.cc:85] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a: Aborting all ops after (but not including) 15
I20260709 13:02:33.692724  2328 rpcz_store.cc:276] 0709 13:02:31.642346 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:31.956232 (+313886us) service_pool.cc:224] Handling call
0709 13:02:31.956625 (+   393us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:31.964996 (+  8371us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:31.965038 (+    42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:31.965665 (+   627us) log.cc:844] Serialized 26 byte log entry
0709 13:02:32.340761 (+375096us) raft_consensus.cc:1645] Marking committed up to 13
0709 13:02:32.340823 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:32.340928 (+   105us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:33.405113 (+1064185us) spinlock_profiling.cc:243] Waited 34.1 ms on lock 0x7b4400022b4c. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f4262f9bc85 00007f4262f9c6a2 00007f4265d0e1db 00007f4265d09aaf 00007f4265cfd86e 00007f4265cfcbb6 00007f4265cfc7d8 00007f4265d06221 00007f4265d061d9 00007f4265d06169
0709 13:02:33.639597 (+234484us) raft_consensus.cc:1685] finished
0709 13:02:33.639651 (+    54us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:33.640200 (+   549us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":7560,"raft.queue_time_us":10619,"raft.run_cpu_time_us":169,"raft.run_wall_time_us":11026,"spinlock_wait_cycles":78463744,"thread_start_us":286347,"threads_started":1,"wal-append.queue_time_us":734495}
I20260709 13:02:33.694516  1600 pending_rounds.cc:107] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a: Aborting uncommitted NO_OP operation due to leader change: 18.16
I20260709 13:02:33.522730  2298 leader_election.cc:304] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:02:33.645062  1646 leader_election.cc:304] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:33.631578  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 19 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:33.696938  1599 raft_consensus.cc:2468] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 18.
I20260709 13:02:33.673897  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:33.489624  1748 raft_consensus.cc:3060] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:02:33.714665  1673 raft_consensus.cc:3060] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Advancing to term 19
W20260709 13:02:33.715972  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 193353us
I20260709 13:02:33.716444  8934 raft_consensus.cc:2804] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:02:33.717092  8934 raft_consensus.cc:493] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:33.717522  8934 raft_consensus.cc:3060] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:02:33.679123  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 17 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:33.680066  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:33.720050  1603 raft_consensus.cc:2468] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 18.
I20260709 13:02:33.721258  9295 raft_consensus.cc:2804] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 13:02:33.721947  9295 raft_consensus.cc:493] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:33.722527  9295 raft_consensus.cc:3060] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:02:33.724380  9282 raft_consensus.cc:493] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:33.725364  9282 raft_consensus.cc:515] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:33.725479  9226 consensus_queue.cc:260] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.10, Last appended by leader: 10, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:02:33.727779  1643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:33.559623  9322 consensus_queue.cc:260] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 21.14, Last appended by leader: 14, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:33.728950  9226 raft_consensus.cc:3060] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:02:33.729041  9252 raft_consensus.cc:493] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:02:33.730544  9322 raft_consensus.cc:3060] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:02:33.730294  9252 raft_consensus.cc:515] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:33.701682  9115 raft_consensus.cc:493] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:02:33.679878  9159 raft_consensus.cc:515] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:33.685765  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:33.690073  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:33.694960  1600 raft_consensus.cc:2889] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:02:33.624656  1746 raft_consensus.cc:2393] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 19: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:02:33.699311  1718 leader_election.cc:304] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:02:33.699517  9148 leader_election.cc:290] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 20 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:33.697978  1677 raft_consensus.cc:2410] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for term 19 because replica has last-logged OpId of term: 17 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 9.
I20260709 13:02:33.669961  9061 raft_consensus.cc:515] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:02:33.702195  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 57689.6us
W20260709 13:02:33.549335  9296 consensus_meta.cc:287] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.732s	user 0.011s	sys 0.006s
I20260709 13:02:33.454809  9331 raft_consensus.cc:515] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:33.768023  1680 raft_consensus.cc:2410] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for term 20 because replica has last-logged OpId of term: 19 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 13.
W20260709 13:02:33.562342  9332 log.cc:927] Time spent T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.820s	user 0.004s	sys 0.000s
I20260709 13:02:33.538605  9358 raft_consensus.cc:993] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1: : Instructing follower af520b67945d443b901bbeca37cc5701 to start an election
I20260709 13:02:33.713712  9205 leader_election.cc:290] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:33.734207  9115 raft_consensus.cc:515] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:33.745990  1604 raft_consensus.cc:2468] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 15.
I20260709 13:02:33.748895  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:33.752202  9282 leader_election.cc:290] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:33.761011  9061 leader_election.cc:290] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:33.763746  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
W20260709 13:02:33.764909  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:33.752193  9252 leader_election.cc:290] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:33.784247  9343 raft_consensus.cc:493] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:33.784087  9214 raft_consensus.cc:2749] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:02:33.785398  9343 raft_consensus.cc:515] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:33.786612  1752 raft_consensus.cc:3060] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:02:33.784015  9331 leader_election.cc:290] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 25 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:33.790624  1598 raft_consensus.cc:1240] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 16. Current term is 17. Ops: []
I20260709 13:02:33.786633  9258 consensus_queue.cc:1048] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.011s
I20260709 13:02:33.794471  9358 raft_consensus.cc:1081] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 19 LEADER]: Signalling peer af520b67945d443b901bbeca37cc5701 to start an election
W20260709 13:02:33.778620  9202 consensus_queue.cc:1175] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "af520b67945d443b901bbeca37cc5701" responder_term: 22 status { last_received { term: 19 index: 13 } last_committed_idx: 13 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 21. Current term is 22. Ops: [21.14-21.14]" } } last_received_current_leader { term: 0 index: 0 } }
I20260709 13:02:33.784193  9107 raft_consensus.cc:493] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:02:33.797676  9107 raft_consensus.cc:515] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:33.798728  9252 raft_consensus.cc:493] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:33.799554  9252 raft_consensus.cc:515] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:33.800534  9107 leader_election.cc:290] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:33.806972  9296 raft_consensus.cc:493] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:33.807857  9296 raft_consensus.cc:515] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:33.809305  1672 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57092 (request call id 1680) took 1236 ms. Trace:
I20260709 13:02:33.813093  1679 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52766 (request call id 3306) took 1388 ms. Trace:
I20260709 13:02:33.814906  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:33.815960  1750 raft_consensus.cc:3060] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Advancing to term 25
I20260709 13:02:33.817723  1676 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52766 (request call id 3292) took 2544 ms. Trace:
W20260709 13:02:33.819765  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 120867us
I20260709 13:02:33.821475  1718 leader_election.cc:304] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:33.823781  9327 raft_consensus.cc:493] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:02:33.824594  9327 raft_consensus.cc:515] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:33.814813  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:33.813711  1679 rpcz_store.cc:276] 0709 13:02:32.424751 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:32.514514 (+ 89763us) service_pool.cc:224] Handling call
0709 13:02:32.530324 (+ 15810us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:33.159072 (+628748us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:33.812474 (+653402us) spinlock_profiling.cc:243] Waited 564 ms on lock 0x7b54000226d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:02:33.812840 (+   366us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1296305024}
I20260709 13:02:33.809796  1672 rpcz_store.cc:276] 0709 13:02:32.572683 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:32.572942 (+   259us) service_pool.cc:224] Handling call
0709 13:02:32.602904 (+ 29962us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:33.040799 (+437895us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:33.040848 (+    49us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:33.040891 (+    43us) raft_consensus.cc:1645] Marking committed up to 15
0709 13:02:33.040924 (+    33us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:33.808594 (+767670us) spinlock_profiling.cc:243] Waited 438 ms on lock 0x7b54001e82d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:02:33.808685 (+    91us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:33.809007 (+   322us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1006983936}
I20260709 13:02:33.835167  9214 raft_consensus.cc:2764] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Leader pre-election decision vote started in defunct term 15: won
I20260709 13:02:33.838838  9115 leader_election.cc:290] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:33.840951  9252 leader_election.cc:290] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:33.847308  1601 raft_consensus.cc:2468] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 20.
I20260709 13:02:33.826560  9227 leader_election.cc:290] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 19 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:33.851729  9351 raft_consensus.cc:515] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:33.853018  9296 leader_election.cc:290] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:33.873201  1752 raft_consensus.cc:2468] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 20.
I20260709 13:02:33.848630  9159 leader_election.cc:290] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 17 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:33.879185  9351 leader_election.cc:290] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 17 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:02:33.882409  9347 log.cc:927] Time spent T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.735s	user 0.002s	sys 0.000s
W20260709 13:02:33.886291  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 65104.4us
I20260709 13:02:33.862872  8934 raft_consensus.cc:515] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:33.818121  1676 rpcz_store.cc:276] 0709 13:02:31.272742 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:31.272925 (+   183us) service_pool.cc:224] Handling call
0709 13:02:31.273504 (+   579us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:02:31.328114 (+ 54610us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 13:02:31.494969 (+166855us) log.cc:844] Serialized 14 byte log entry
0709 13:02:32.283657 (+788688us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:02:32.327605 (+ 43948us) log.cc:844] Serialized 52 byte log entry
0709 13:02:32.327836 (+   231us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:02:32.328280 (+   444us) log.cc:844] Serialized 14 byte log entry
0709 13:02:32.328641 (+   361us) log.cc:844] Serialized 14 byte log entry
0709 13:02:32.902632 (+573991us) spinlock_profiling.cc:243] Waited 2.31 ms on lock 0x7b180031d680. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f4263057e07 00007f426beb2ab9 00007f426307469b 00007f426307293f 00007f4263072455 00007f426615a060 00007f426616c716 00007f426617d287 00007f426617d208
0709 13:02:33.075999 (+173367us) spinlock_profiling.cc:243] Waited 115 ms on lock 0x7b4400022b4c. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f4262f9bc85 00007f4262f9c6a2 00007f4265d0e1db 00007f4265d09aaf 00007f4265cfd86e 00007f4265cfcbb6 00007f4265cfc7d8 00007f4265d06221 00007f4265d061d9 00007f4265d06169
0709 13:02:33.816969 (+740970us) spinlock_profiling.cc:243] Waited 10.6 ms on lock 0x7b5400121720. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42660dd733 00007f4266162dbf 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c
0709 13:02:33.817050 (+    81us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:33.817231 (+   181us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:33.817300 (+    69us) raft_consensus.cc:1685] finished
0709 13:02:33.817315 (+    15us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:33.817481 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":128026,"prepare.queue_time_us":190171,"prepare.run_cpu_time_us":214,"prepare.run_wall_time_us":2328,"raft.queue_time_us":492470,"raft.run_cpu_time_us":740,"raft.run_wall_time_us":107092,"spinlock_wait_cycles":294210048,"thread_start_us":989777,"threads_started":3,"wal-append.queue_time_us":698298}
I20260709 13:02:33.897375  1673 raft_consensus.cc:2468] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 19.
I20260709 13:02:33.898589  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 22 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:02:33.904170  1646 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:33.905514  1718 leader_election.cc:304] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:33.906985  1718 leader_election.cc:304] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:02:33.908004  1680 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52778 (request call id 2840) took 1081 ms. Trace:
I20260709 13:02:33.908262  9214 raft_consensus.cc:2804] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 13:02:33.908789  9214 raft_consensus.cc:493] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:33.908505  1680 rpcz_store.cc:276] 0709 13:02:32.828014 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:32.900548 (+ 72534us) service_pool.cc:224] Handling call
0709 13:02:33.907702 (+1007154us) spinlock_profiling.cc:243] Waited 866 ms on lock 0x7b54000fe454. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:02:33.907915 (+   213us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1992190848}
I20260709 13:02:33.909409  9214 raft_consensus.cc:3060] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:02:33.910704  9252 raft_consensus.cc:2749] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 13:02:33.916355  1679 raft_consensus.cc:1240] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 18. Current term is 19. Ops: [18.12-18.12]
I20260709 13:02:33.903968  2298 leader_election.cc:304] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:02:33.930629  9219 consensus_queue.cc:260] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 19.9, Last appended by leader: 9, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:33.932864  9219 raft_consensus.cc:3060] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:02:33.941054  1751 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55840 (request call id 3187) took 2962 ms. Trace:
I20260709 13:02:33.941591  1751 rpcz_store.cc:276] 0709 13:02:30.978884 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:31.546594 (+567710us) service_pool.cc:224] Handling call
0709 13:02:31.709098 (+162504us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:33.146788 (+1437690us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:33.815212 (+668424us) spinlock_profiling.cc:243] Waited 1.37 s on lock 0x7b54003700d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:02:33.940578 (+125366us) spinlock_profiling.cc:243] Waited 162 ms on lock 0x7b54003700d0. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661618ad 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:02:33.940818 (+   240us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3535167488}
I20260709 13:02:33.919795  9174 raft_consensus.cc:2749] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:02:33.955417  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:02:33.956841  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 79838.2us
I20260709 13:02:33.959793  9282 raft_consensus.cc:2804] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 13:02:33.960716  9282 raft_consensus.cc:697] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 20 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:02:33.970739  1674 raft_consensus.cc:2468] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 21.
I20260709 13:02:33.974700  1675 raft_consensus.cc:1275] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 17 index: 10. Preceding OpId from leader: term: 18 index: 11. (index mismatch)
I20260709 13:02:33.974946  1753 raft_consensus.cc:1240] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 22. Current term is 23. Ops: []
I20260709 13:02:33.975404  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 24 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:33.975572  1675 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52778 (request call id 2851) took 1102 ms. Trace:
I20260709 13:02:33.976239  1675 rpcz_store.cc:276] 0709 13:02:32.872794 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:33.937918 (+1065124us) service_pool.cc:224] Handling call
0709 13:02:33.946833 (+  8915us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:33.975236 (+ 28403us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:33.975476 (+   240us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:33.978039  1675 raft_consensus.cc:1240] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 20. Current term is 21. Ops: [20.13-20.13]
I20260709 13:02:33.983960  1596 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40154 (request call id 2994) took 1147 ms. Trace:
I20260709 13:02:33.984485  1596 rpcz_store.cc:276] 0709 13:02:32.836587 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:32.836855 (+   268us) service_pool.cc:224] Handling call
0709 13:02:32.837126 (+   271us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:33.670219 (+833093us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:33.983554 (+313335us) spinlock_profiling.cc:243] Waited 770 ms on lock 0x7b540006d6d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:02:33.983856 (+   302us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1771371904}
I20260709 13:02:33.985321  9253 raft_consensus.cc:515] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:33.985956  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:33.991214  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:33.994019  1676 raft_consensus.cc:1275] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 21 index: 11. Preceding OpId from leader: term: 23 index: 12. (index mismatch)
I20260709 13:02:33.994880  1676 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52778 (request call id 2844) took 1165 ms. Trace:
I20260709 13:02:33.995230  1676 rpcz_store.cc:276] 0709 13:02:32.829242 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:33.902601 (+1073359us) service_pool.cc:224] Handling call
0709 13:02:33.954780 (+ 52179us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:33.994549 (+ 39769us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:33.994777 (+   228us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:33.996572  1600 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40154 (request call id 2995) took 1159 ms. Trace:
I20260709 13:02:33.996840  1676 raft_consensus.cc:1240] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 19. Current term is 20. Ops: []
I20260709 13:02:33.997080  1600 rpcz_store.cc:276] 0709 13:02:32.836849 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:32.837320 (+   471us) service_pool.cc:224] Handling call
0709 13:02:32.856538 (+ 19218us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:33.747151 (+890613us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:33.996204 (+249053us) spinlock_profiling.cc:243] Waited 837 ms on lock 0x7b54000f1754. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:02:33.996496 (+   292us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1924454272}
I20260709 13:02:33.998862  9343 leader_election.cc:290] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:33.999408  1597 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 3087) took 1159 ms. Trace:
I20260709 13:02:33.999523  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 19 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
W20260709 13:02:33.999904  9213 consensus_meta.cc:287] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.747s	user 0.017s	sys 0.007s
I20260709 13:02:33.999907  1597 rpcz_store.cc:276] 0709 13:02:32.839676 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:32.994321 (+154645us) service_pool.cc:224] Handling call
0709 13:02:33.082534 (+ 88213us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:33.086260 (+  3726us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:33.998945 (+912685us) spinlock_profiling.cc:243] Waited 65.1 ms on lock 0x7b5400301950. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661618ad 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:02:33.999156 (+   211us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":149717376}
I20260709 13:02:34.005303  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:34.006824  1643 leader_election.cc:304] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:02:34.010504  9250 raft_consensus.cc:2749] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 13:02:34.012856  2332 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca"
dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
mode: GRACEFUL
new_leader_uuid: "af520b67945d443b901bbeca37cc5701"
 from {username='slave'} at 127.0.0.1:35790
I20260709 13:02:34.013928  2332 raft_consensus.cc:606] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Received request to transfer leadership to TS af520b67945d443b901bbeca37cc5701
I20260709 13:02:34.015092  2332 raft_consensus.cc:614] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 13:02:34.013743  1600 raft_consensus.cc:2468] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 19.
I20260709 13:02:34.019452  9253 leader_election.cc:290] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 25 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:34.020278  9327 leader_election.cc:290] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:34.020979  2298 leader_election.cc:304] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:02:34.021823  1675 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52778 (request call id 2852) took 1111 ms. Trace:
I20260709 13:02:33.988689  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:34.023535  9311 leader_election.cc:290] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:34.025400  9232 raft_consensus.cc:3055] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 13:02:34.025938  9232 raft_consensus.cc:740] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:02:34.023593  1596 raft_consensus.cc:3060] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:02:34.027432  9232 consensus_queue.cc:260] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.10, Last appended by leader: 10, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:34.029346  9232 raft_consensus.cc:3060] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:02:34.029906  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:34.030948  1605 raft_consensus.cc:1240] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 25 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 24. Current term is 25. Ops: []
I20260709 13:02:34.032572  9253 raft_consensus.cc:493] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:34.033083  2335 raft_consensus.cc:1240] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 21. Current term is 22. Ops: []
I20260709 13:02:34.033371  9253 raft_consensus.cc:515] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:34.034060  1603 raft_consensus.cc:1240] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 18. Current term is 19. Ops: [18.12-18.12]
I20260709 13:02:34.035948  9253 leader_election.cc:290] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:33.972683  1680 raft_consensus.cc:1275] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 18 index: 13. (index mismatch)
I20260709 13:02:34.037324  1680 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52778 (request call id 2850) took 1164 ms. Trace:
I20260709 13:02:34.037550  1679 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52778 (request call id 2835) took 1212 ms. Trace:
I20260709 13:02:34.038017  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:34.037681  2332 raft_consensus.cc:1240] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 20 LEADER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 19. Current term is 20. Ops: [19.14-19.14]
I20260709 13:02:34.037899  1680 rpcz_store.cc:276] 0709 13:02:32.872510 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:33.910067 (+1037557us) service_pool.cc:224] Handling call
0709 13:02:33.929454 (+ 19387us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:34.037016 (+107562us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:34.037251 (+   235us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:34.022275  1675 rpcz_store.cc:276] 0709 13:02:32.910872 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:33.977099 (+1066227us) service_pool.cc:224] Handling call
0709 13:02:33.977654 (+   555us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:33.978421 (+   767us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:34.021746 (+ 43325us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:34.038044  1679 rpcz_store.cc:276] 0709 13:02:32.824944 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:33.869782 (+1044838us) service_pool.cc:224] Handling call
0709 13:02:33.898141 (+ 28359us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:33.920485 (+ 22344us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:34.037291 (+116806us) spinlock_profiling.cc:243] Waited 2.51 ms on lock 0x7b54000d7350. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661618ad 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:02:34.037457 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5768064}
I20260709 13:02:34.023523  1600 raft_consensus.cc:1275] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 18 index: 11. (index mismatch)
I20260709 13:02:34.040782  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:34.057447  1676 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57092 (request call id 1686) took 1050 ms. Trace:
I20260709 13:02:34.069440  1676 rpcz_store.cc:276] 0709 13:02:33.006567 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:33.997782 (+991215us) service_pool.cc:224] Handling call
0709 13:02:33.998227 (+   445us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:34.056930 (+ 58703us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:34.056975 (+    45us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:34.057044 (+    69us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:02:34.057097 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:34.057194 (+    97us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:34.057366 (+   172us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:34.063576  1679 raft_consensus.cc:1240] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 16. Current term is 17. Ops: []
I20260709 13:02:34.064819  9054 raft_consensus.cc:2804] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:02:34.052983  9282 consensus_queue.cc:237] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.14, Last appended by leader: 14, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:34.074174  1752 raft_consensus.cc:2393] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 20: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:02:34.076524  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
W20260709 13:02:34.078212  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 130482us
W20260709 13:02:34.083050  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 176315us
W20260709 13:02:34.083364  1644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:34.085099  1718 leader_election.cc:304] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:02:34.087893  1750 raft_consensus.cc:2468] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 25.
I20260709 13:02:34.086128  9213 raft_consensus.cc:493] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:34.092402  1596 raft_consensus.cc:2468] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 19.
I20260709 13:02:34.093925  9250 consensus_queue.cc:1048] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 13:02:34.096549  9069 consensus_queue.cc:1059] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 13:02:34.097628  9069 consensus_queue.cc:1243] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer 83595ad0fd5940c0a2bfc7c61acd969a log is divergent from this leader: its last log entry 15.12 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 10
I20260709 13:02:34.098464  1748 raft_consensus.cc:2468] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 22.
I20260709 13:02:34.100111  8934 leader_election.cc:290] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 19 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:34.103554  1718 leader_election.cc:304] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:02:34.074326  9054 raft_consensus.cc:493] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:34.105633  1677 raft_consensus.cc:2468] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 16.
I20260709 13:02:34.116684  1674 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52766 (request call id 3311) took 1214 ms. Trace:
I20260709 13:02:34.121093  1673 raft_consensus.cc:1240] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 15. Current term is 16. Ops: []
I20260709 13:02:34.117244  1674 rpcz_store.cc:276] 0709 13:02:32.902349 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:33.971923 (+1069574us) service_pool.cc:224] Handling call
0709 13:02:34.089016 (+117093us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:34.111405 (+ 22389us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:34.111460 (+    55us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:34.116257 (+  4797us) raft_consensus.cc:1645] Marking committed up to 16
0709 13:02:34.116344 (+    87us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:34.116437 (+    93us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:34.116601 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:34.124591  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:02:34.134579  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:34.098335  9069 raft_consensus.cc:3055] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 13:02:34.142264  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:34.144239  9250 raft_consensus.cc:493] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:34.144960  9250 raft_consensus.cc:515] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:34.150938  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:34.151682  1748 raft_consensus.cc:2468] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 21.
I20260709 13:02:34.152877  9250 leader_election.cc:290] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:34.154018  8968 raft_consensus.cc:2749] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:02:34.154397  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 22 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:34.141814  2298 leader_election.cc:304] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:02:34.155951  1678 raft_consensus.cc:2468] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 22.
I20260709 13:02:34.158557  9295 raft_consensus.cc:515] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:34.160135  1675 raft_consensus.cc:1240] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 21. Current term is 22. Ops: []
I20260709 13:02:34.160372  1596 raft_consensus.cc:2410] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for term 22 because replica has last-logged OpId of term: 20 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 10.
I20260709 13:02:34.161453  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:34.162863  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 19 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:34.162949  1671 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52766 (request call id 3310) took 1368 ms. Trace:
I20260709 13:02:34.163472  1671 rpcz_store.cc:276] 0709 13:02:32.795507 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:33.803695 (+1008188us) service_pool.cc:224] Handling call
0709 13:02:33.804470 (+   775us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:33.805248 (+   778us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:33.805287 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:33.806014 (+   727us) log.cc:844] Serialized 26 byte log entry
0709 13:02:33.900616 (+ 94602us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:02:33.900661 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:33.900736 (+    75us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:34.162542 (+261806us) raft_consensus.cc:1685] finished
0709 13:02:34.162573 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:34.162726 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":18250,"raft.run_cpu_time_us":185,"raft.run_wall_time_us":187,"thread_start_us":94144,"threads_started":1,"wal-append.queue_time_us":247859}
I20260709 13:02:34.165983  1671 raft_consensus.cc:1240] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 22. Current term is 23. Ops: []
I20260709 13:02:34.167536  1676 raft_consensus.cc:1275] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 16 index: 13. (term mismatch)
I20260709 13:02:34.168030  1676 pending_rounds.cc:85] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d: Aborting all ops after (but not including) 12
I20260709 13:02:34.168352  1676 pending_rounds.cc:107] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d: Aborting uncommitted NO_OP operation due to leader change: 17.13
I20260709 13:02:34.168649  1676 raft_consensus.cc:2889] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 13:02:34.169589  1643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:34.169781  1596 raft_consensus.cc:2393] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 19: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:02:34.139060  9213 raft_consensus.cc:515] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:34.173045  9311 raft_consensus.cc:2804] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 13:02:34.173115  9213 leader_election.cc:290] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:34.173802  9311 raft_consensus.cc:697] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 19 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:02:34.174784  1644 leader_election.cc:304] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:02:34.175010  9311 consensus_queue.cc:237] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 17.15, Last appended by leader: 15, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:34.175879  1677 raft_consensus.cc:1240] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 21. Current term is 22. Ops: []
I20260709 13:02:34.176803  1644 leader_election.cc:304] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:34.177819  1677 raft_consensus.cc:1275] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 20 index: 15. Preceding OpId from leader: term: 22 index: 16. (index mismatch)
I20260709 13:02:34.179076  9213 raft_consensus.cc:2749] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:02:34.178988  9333 consensus_queue.cc:1059] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 19.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.058s
I20260709 13:02:34.179703  1671 raft_consensus.cc:1275] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 19 index: 14. Preceding OpId from leader: term: 20 index: 14. (term mismatch)
I20260709 13:02:34.180051  1600 raft_consensus.cc:1275] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 18 index: 13. (index mismatch)
I20260709 13:02:34.180099  9213 raft_consensus.cc:2749] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:02:34.180452  9333 raft_consensus.cc:3055] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 13:02:34.180910  9333 raft_consensus.cc:740] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:02:34.180976  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:34.181581  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:34.171018  9295 leader_election.cc:290] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 22 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:34.180235  1671 pending_rounds.cc:85] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d: Aborting all ops after (but not including) 13
I20260709 13:02:34.182104  9333 consensus_queue.cc:260] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.14, Last appended by leader: 14, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:34.184590  9333 raft_consensus.cc:3060] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:02:34.170974  1676 raft_consensus.cc:1240] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 25 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 24. Current term is 25. Ops: []
I20260709 13:02:34.189888  1644 leader_election.cc:400] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 21 election: Vote denied by peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625) with higher term. Message: Invalid argument: T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for earlier term 21. Current term is 22.
I20260709 13:02:34.190572  1671 pending_rounds.cc:107] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d: Aborting uncommitted NO_OP operation due to leader change: 19.14
I20260709 13:02:34.190934  1671 raft_consensus.cc:2889] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 13:02:34.182821  1645 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:34.206112  1604 raft_consensus.cc:2468] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 22.
I20260709 13:02:34.181974  1745 raft_consensus.cc:2468] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 20.
I20260709 13:02:34.225831  9322 raft_consensus.cc:493] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:34.226670  9322 raft_consensus.cc:515] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:34.118696  9054 raft_consensus.cc:3060] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Advancing to term 20
W20260709 13:02:34.230720  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 127304us
I20260709 13:02:34.154922  9069 raft_consensus.cc:740] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:02:34.238741  1675 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57092 (request call id 1691) took 1029 ms. Trace:
I20260709 13:02:34.239289  1675 rpcz_store.cc:276] 0709 13:02:33.209115 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:34.053423 (+844308us) service_pool.cc:224] Handling call
0709 13:02:34.079831 (+ 26408us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:34.160609 (+ 80778us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:34.238459 (+ 77850us) spinlock_profiling.cc:243] Waited 80 ms on lock 0x7b54000d9654. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:02:34.238675 (+   216us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":184096512}
I20260709 13:02:34.244241  9322 leader_election.cc:290] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:34.246862  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 17 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:34.259799  2335 raft_consensus.cc:1240] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 18 LEADER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 17. Current term is 18. Ops: [17.13-17.13]
I20260709 13:02:34.183005  9214 raft_consensus.cc:515] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:34.262578  1750 raft_consensus.cc:1275] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 19 index: 13. Preceding OpId from leader: term: 21 index: 14. (index mismatch)
I20260709 13:02:34.266451  1571 leader_election.cc:304] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:34.276648  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
W20260709 13:02:34.278119  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:34.296236  1748 raft_consensus.cc:2393] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 22: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
W20260709 13:02:34.296521  1643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:34.237481  9069 consensus_queue.cc:260] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.12, Last appended by leader: 12, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:34.305733  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:34.236605  9219 consensus_queue.cc:1048] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.055s
I20260709 13:02:34.310595  1646 leader_election.cc:304] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
W20260709 13:02:34.311683  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 91339us
I20260709 13:02:34.312970  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:02:34.321538  9323 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.752s	user 0.010s	sys 0.025s
W20260709 13:02:34.322067  9323 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.758s	user 0.010s	sys 0.025s
I20260709 13:02:34.327246  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
W20260709 13:02:34.331519  2298 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:34.289467  8994 consensus_queue.cc:1059] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 16.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.028s
I20260709 13:02:34.352496  1646 leader_election.cc:304] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:34.358245  2332 raft_consensus.cc:2368] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for earlier term 19. Current term is 20.
I20260709 13:02:34.350406  9213 raft_consensus.cc:2804] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Leader election won for term 20
I20260709 13:02:34.381616  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:34.382664  2333 raft_consensus.cc:2393] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 23: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:02:34.382843  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:34.384473  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:34.385283  2330 raft_consensus.cc:1240] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 20. Current term is 21. Ops: [20.15-20.15]
I20260709 13:02:34.386209  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
W20260709 13:02:34.388345  9320 log.cc:927] Time spent T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.060s	user 0.000s	sys 0.000s
W20260709 13:02:34.366341  9246 consensus_meta.cc:287] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.850s	user 0.016s	sys 0.001s
I20260709 13:02:34.376678  1680 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52766 (request call id 3315) took 1191 ms. Trace:
W20260709 13:02:34.378881  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:34.380702  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:34.364147  8934 consensus_queue.cc:1048] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.001s
W20260709 13:02:34.397207  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 46122.5us
I20260709 13:02:34.386344  9336 raft_consensus.cc:2804] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 13:02:34.365739  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:34.329020  9358 consensus_queue.cc:1059] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 22.17, Next index: 18, Last known committed idx: 16, Time since last communication: 0.032s
I20260709 13:02:34.354692  9069 raft_consensus.cc:3060] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:02:34.401436  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:34.394263  1680 rpcz_store.cc:276] 0709 13:02:33.185403 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:34.053425 (+868022us) service_pool.cc:224] Handling call
0709 13:02:34.065818 (+ 12393us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:34.066294 (+   476us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:34.066332 (+    38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:34.067205 (+   873us) log.cc:844] Serialized 26 byte log entry
0709 13:02:34.250124 (+182919us) raft_consensus.cc:1645] Marking committed up to 15
0709 13:02:34.250178 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:34.250280 (+   102us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:34.376419 (+126139us) raft_consensus.cc:1685] finished
0709 13:02:34.376442 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:34.376591 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":182465,"threads_started":1,"wal-append.queue_time_us":284418}
I20260709 13:02:34.403208  1646 leader_election.cc:304] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:34.405884  1605 raft_consensus.cc:1217] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Deduplicated request from leader. Original: 12.10->[14.11-16.12]   Dedup: 14.11->[16.12-16.12]
I20260709 13:02:34.412978  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:34.413578  1605 raft_consensus.cc:1240] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 16. Current term is 17. Ops: [14.11-16.12]
I20260709 13:02:34.392969  9253 raft_consensus.cc:493] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:34.401060  9246 raft_consensus.cc:515] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:34.405555  9336 raft_consensus.cc:493] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:34.401537  9213 raft_consensus.cc:697] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 20 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:02:34.440892  1571 leader_election.cc:304] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:34.444638  1672 raft_consensus.cc:1240] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 22. Current term is 23. Ops: [22.11-22.11]
I20260709 13:02:34.437067  9336 raft_consensus.cc:3060] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:02:34.450100  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:34.422968  8994 consensus_queue.cc:1059] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 21.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.034s
I20260709 13:02:34.430469  9258 consensus_queue.cc:1048] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.020s
I20260709 13:02:34.431977  9327 raft_consensus.cc:493] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:02:34.460129  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:34.455262  9253 raft_consensus.cc:515] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:34.468792  9253 leader_election.cc:290] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:34.432133  9282 consensus_queue.cc:1059] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.020s
I20260709 13:02:34.465098  1680 raft_consensus.cc:2468] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 22.
I20260709 13:02:34.452174  9214 leader_election.cc:290] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 22 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:34.481349  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:34.477698  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:34.482807  1673 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52778 (request call id 2836) took 1649 ms. Trace:
I20260709 13:02:34.481071  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:34.483354  1673 rpcz_store.cc:276] 0709 13:02:32.825227 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:33.901325 (+1076098us) service_pool.cc:224] Handling call
0709 13:02:34.114662 (+213337us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:34.122642 (+  7980us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:34.474629 (+351987us) spinlock_profiling.cc:243] Waited 191 ms on lock 0x7b54001f6650. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661618ad 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:02:34.474778 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":440442112}
I20260709 13:02:34.472203  9219 raft_consensus.cc:2804] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 13:02:34.491387  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 22 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:34.495371  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:34.499577  2334 raft_consensus.cc:1240] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 16. Current term is 17. Ops: []
I20260709 13:02:34.500911  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:34.502580  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:34.503610  1678 raft_consensus.cc:2393] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 16: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:02:34.504419  2332 raft_consensus.cc:1275] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 15 index: 14. Preceding OpId from leader: term: 18 index: 15. (index mismatch)
I20260709 13:02:34.505568  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:34.506621  9327 raft_consensus.cc:515] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:34.509508  9327 leader_election.cc:290] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:34.477808  9250 consensus_queue.cc:1059] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 22.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.033s
I20260709 13:02:34.517128  9343 raft_consensus.cc:3055] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 13:02:34.517565  9343 raft_consensus.cc:740] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:02:34.493028  9219 raft_consensus.cc:493] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:34.519276  9219 raft_consensus.cc:3060] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:02:34.519989  1672 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52778 (request call id 2832) took 1696 ms. Trace:
I20260709 13:02:34.520427  1672 rpcz_store.cc:276] 0709 13:02:32.823990 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:33.840939 (+1016949us) service_pool.cc:224] Handling call
0709 13:02:33.841539 (+   600us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:34.447115 (+605576us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:34.519673 (+ 72558us) spinlock_profiling.cc:243] Waited 603 ms on lock 0x7b5400163354. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:02:34.519922 (+   249us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1387058944}
I20260709 13:02:34.525692  2299 leader_election.cc:304] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
W20260709 13:02:34.526782  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 73883.8us
I20260709 13:02:34.526943  9282 raft_consensus.cc:2804] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 13:02:34.527495  9282 raft_consensus.cc:493] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:34.527900  9282 raft_consensus.cc:3060] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:02:34.528080  1754 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55848 (request call id 3388) took 1154 ms. Trace:
I20260709 13:02:34.528402  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:34.502617  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:34.528563  1754 rpcz_store.cc:276] 0709 13:02:33.373974 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:33.389785 (+ 15811us) service_pool.cc:224] Handling call
0709 13:02:33.390237 (+   452us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:33.414551 (+ 24314us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:33.414605 (+    54us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:34.195270 (+780665us) spinlock_profiling.cc:243] Waited 5.08 ms on lock 0x7b2c000bc788. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4264a5ba11 00007f4264a6b372 00007f426619da01 00007f426616243a 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2
0709 13:02:34.195981 (+   711us) log.cc:844] Serialized 26 byte log entry
0709 13:02:34.429248 (+233267us) raft_consensus.cc:1645] Marking committed up to 15
0709 13:02:34.429293 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:34.435349 (+  6056us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:34.527529 (+ 92180us) raft_consensus.cc:1685] finished
0709 13:02:34.527553 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:34.527851 (+   298us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":18943,"raft.queue_time_us":19186,"raft.run_cpu_time_us":193,"raft.run_wall_time_us":10074,"spinlock_wait_cycles":11693440,"thread_start_us":225125,"threads_started":1,"wal-append.queue_time_us":329566}
I20260709 13:02:34.537142  8952 raft_consensus.cc:2804] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Leader election won for term 21
I20260709 13:02:34.537756  8952 raft_consensus.cc:697] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
W20260709 13:02:34.539215  9335 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.542s	user 0.013s	sys 0.039s
I20260709 13:02:34.538905  8952 consensus_queue.cc:237] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 19.11, Last appended by leader: 11, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:02:34.539639  9335 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.660s	user 0.013s	sys 0.039s
I20260709 13:02:34.540671  9335 raft_consensus.cc:3055] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 13:02:34.541043  9335 raft_consensus.cc:740] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:02:34.541229  9244 raft_consensus.cc:2804] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 13:02:34.542644  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:34.543284  2333 raft_consensus.cc:2376] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 20. Re-sending same reply.
I20260709 13:02:34.546967  2330 raft_consensus.cc:2410] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for term 17 because replica has last-logged OpId of term: 16 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 12.
W20260709 13:02:34.547202  9379 log.cc:927] Time spent T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.056s	user 0.000s	sys 0.000s
I20260709 13:02:34.547716  9209 raft_consensus.cc:3055] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 13:02:34.548170  9209 raft_consensus.cc:740] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:02:34.548941  2329 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39738 (request call id 1330) took 1338 ms. Trace:
I20260709 13:02:34.549275  9209 consensus_queue.cc:260] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 21.16, Last appended by leader: 16, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:34.550580  9334 consensus_queue.cc:1059] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 21.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.088s
I20260709 13:02:34.549409  2329 rpcz_store.cc:276] 0709 13:02:33.210769 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:33.211033 (+   264us) service_pool.cc:224] Handling call
0709 13:02:33.225081 (+ 14048us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:33.225582 (+   501us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:33.225625 (+    43us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:33.950266 (+724641us) spinlock_profiling.cc:243] Waited 3.82 ms on lock 0x7b2c00021c88. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4264a5ba11 00007f4264a6b372 00007f426619da01 00007f426616243a 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2
0709 13:02:33.994793 (+ 44527us) log.cc:844] Serialized 26 byte log entry
0709 13:02:34.320111 (+325318us) raft_consensus.cc:1645] Marking committed up to 14
0709 13:02:34.327789 (+  7678us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:34.340542 (+ 12753us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:34.548302 (+207760us) raft_consensus.cc:1685] finished
0709 13:02:34.548328 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:34.548720 (+   392us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":3403,"raft.run_cpu_time_us":115,"raft.run_wall_time_us":3911,"spinlock_wait_cycles":8792320,"thread_start_us":300347,"threads_started":1,"wal-append.queue_time_us":450745}
I20260709 13:02:34.528241  9333 consensus_queue.cc:1048] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.023s
I20260709 13:02:34.552434  9213 consensus_queue.cc:237] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.14, Last appended by leader: 14, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:34.530618  8934 raft_consensus.cc:3055] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 13:02:34.533059  9088 raft_consensus.cc:2804] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 25 FOLLOWER]: Leader election won for term 25
W20260709 13:02:34.549185  9356 log.cc:927] Time spent T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.092s	user 0.000s	sys 0.000s
I20260709 13:02:34.487329  9054 raft_consensus.cc:515] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:34.535225  9327 raft_consensus.cc:3055] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 LEADER]: Stepping down as leader of term 22
W20260709 13:02:34.559403  9383 log.cc:927] Time spent T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.132s	user 0.001s	sys 0.000s
W20260709 13:02:34.579716  1644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:34.597128  2330 raft_consensus.cc:1240] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 19. Current term is 20. Ops: [19.13-19.13]
I20260709 13:02:34.561520  8934 raft_consensus.cc:740] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:02:34.592430  2334 raft_consensus.cc:2468] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 19.
I20260709 13:02:34.596254  9209 raft_consensus.cc:3060] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:02:34.542055  9335 consensus_queue.cc:260] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 22.17, Last appended by leader: 17, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:34.613824  2336 raft_consensus.cc:2468] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 20.
I20260709 13:02:34.576501  9327 raft_consensus.cc:740] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:02:34.579335  9088 raft_consensus.cc:697] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 25 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:02:34.619714  1718 leader_election.cc:304] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
W20260709 13:02:34.634341  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 226745us
I20260709 13:02:34.562111  9343 consensus_queue.cc:260] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 16.13, Last appended by leader: 13, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:34.643649  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:34.629261  9335 raft_consensus.cc:3060] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Advancing to term 23
W20260709 13:02:34.651891  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 204128us
W20260709 13:02:34.654384  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 215692us
I20260709 13:02:34.660379  1646 leader_election.cc:304] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:34.648895  9232 raft_consensus.cc:2749] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20260709 13:02:34.657099  1569 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.107.196:46625 timed out after 1.906s (SENT) [suppressed 12 similar messages]
I20260709 13:02:34.659765  9054 leader_election.cc:290] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 20 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:34.665939  9295 raft_consensus.cc:2804] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Leader pre-election won for term 21
W20260709 13:02:34.683298  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 63828us
W20260709 13:02:34.708640  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:34.712740  9253 consensus_queue.cc:1059] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 19.14, Next index: 15, Last known committed idx: 13, Time since last communication: 0.001s
W20260709 13:02:34.714596  9382 log.cc:927] Time spent T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.198s	user 0.000s	sys 0.002s
W20260709 13:02:34.718015  1643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:34.702852  9295 raft_consensus.cc:493] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:34.675314  9327 consensus_queue.cc:260] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 22.14, Last appended by leader: 14, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:34.735201  9327 raft_consensus.cc:3060] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:02:34.735260  1644 leader_election.cc:304] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:34.736439  9409 raft_consensus.cc:2749] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 13:02:34.660288  8934 consensus_queue.cc:260] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 24.18, Last appended by leader: 18, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:34.729430  9343 raft_consensus.cc:3060] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Advancing to term 17
W20260709 13:02:34.768894  1643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:34.733777  9295 raft_consensus.cc:3060] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Advancing to term 21
W20260709 13:02:34.783950  1719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:34.785668  1643 leader_election.cc:304] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:02:34.788034  1643 leader_election.cc:304] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:02:34.789045  9409 raft_consensus.cc:2804] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:02:34.789541  9409 raft_consensus.cc:493] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Starting leader election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:02:34.682189  9088 consensus_queue.cc:237] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 23.17, Last appended by leader: 17, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:34.789940  9409 raft_consensus.cc:3060] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Advancing to term 19
W20260709 13:02:34.797411  2300 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:34.785403  9396 consensus_queue.cc:1048] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.082s
I20260709 13:02:34.799620  9396 raft_consensus.cc:2804] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:02:34.800127  9396 raft_consensus.cc:493] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:34.800565  9396 raft_consensus.cc:3060] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:02:34.801234  1644 leader_election.cc:304] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:34.802224  9323 raft_consensus.cc:2749] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:02:34.807251  9409 raft_consensus.cc:515] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:34.808876  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:34.810338  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:34.811163  1748 raft_consensus.cc:3060] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:02:34.811695  9398 raft_consensus.cc:3055] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 LEADER]: Stepping down as leader of term 19
I20260709 13:02:34.812783  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:34.813408  1675 raft_consensus.cc:2468] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 18.
I20260709 13:02:34.813686  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:34.814692  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:34.814998  1747 raft_consensus.cc:1275] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 19 index: 10. Preceding OpId from leader: term: 21 index: 11. (index mismatch)
I20260709 13:02:34.812160  9398 raft_consensus.cc:740] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:02:34.815351  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:34.816388  1672 raft_consensus.cc:2393] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 21: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:02:34.817378  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 25 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:34.789213  1643 leader_election.cc:304] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:02:34.833225  1748 raft_consensus.cc:2468] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 17.
I20260709 13:02:34.756666  9232 consensus_queue.cc:1048] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 8, Time since last communication: 0.071s
I20260709 13:02:34.838272  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 24 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:34.851801  1678 raft_consensus.cc:3060] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Advancing to term 24
I20260709 13:02:34.839643  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:02:34.839931  9381 log.cc:927] Time spent T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.394s	user 0.005s	sys 0.000s
I20260709 13:02:34.840205  9409 leader_election.cc:290] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:34.843693  9339 raft_consensus.cc:3060] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Advancing to term 21
W20260709 13:02:34.843945  1571 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:34.844816  9412 raft_consensus.cc:493] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:34.843684  9407 raft_consensus.cc:493] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:34.832057  8968 consensus_queue.cc:1048] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.015s
I20260709 13:02:34.850847  2298 leader_election.cc:304] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:02:34.879577  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
W20260709 13:02:34.879592  1571 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:34.880373  1748 raft_consensus.cc:1240] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 16. Current term is 17. Ops: []
W20260709 13:02:34.880486  1571 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:34.875339  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:34.890790  9398 consensus_queue.cc:260] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.14, Last appended by leader: 14, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:34.891485  9282 raft_consensus.cc:515] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:34.892700  9398 raft_consensus.cc:3060] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Advancing to term 20
W20260709 13:02:34.895386  1571 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:34.897657  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 46974.6us
I20260709 13:02:34.817473  9219 raft_consensus.cc:515] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:34.869249  8934 raft_consensus.cc:3060] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 24 FOLLOWER]: Advancing to term 25
I20260709 13:02:34.873116  9226 raft_consensus.cc:493] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:34.873111  9232 consensus_queue.cc:1243] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer 816fadcea8fc4893882f5f9b9f22a03d log is divergent from this leader: its last log entry 17.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
I20260709 13:02:34.906723  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:34.910125  1754 raft_consensus.cc:2393] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 20: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:02:34.912214  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:34.894245  9282 leader_election.cc:290] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 22 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:34.935896  1748 raft_consensus.cc:1240] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 25 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 24. Current term is 25. Ops: []
I20260709 13:02:34.957608  1604 raft_consensus.cc:2468] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 18.
I20260709 13:02:34.904434  9409 raft_consensus.cc:2804] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Leader pre-election won for term 16
W20260709 13:02:34.965912  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 116945us
I20260709 13:02:34.971304  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:34.980679  1644 leader_election.cc:304] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:34.982684  1604 raft_consensus.cc:2468] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 18.
I20260709 13:02:34.979058  1605 raft_consensus.cc:1275] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 19 index: 13. Preceding OpId from leader: term: 21 index: 14. (index mismatch)
I20260709 13:02:34.992075  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:34.898830  9336 raft_consensus.cc:515] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:02:34.989080  9390 log.cc:927] Time spent T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.052s	user 0.000s	sys 0.001s
I20260709 13:02:34.996371  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:35.002103  1748 raft_consensus.cc:2393] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 22: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:02:34.996320  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:35.004067  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:35.006676  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:34.968272  9409 raft_consensus.cc:493] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
W20260709 13:02:35.009208  2298 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:34.894783  9407 raft_consensus.cc:515] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:35.015463  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 23 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:35.048621  2301 leader_election.cc:304] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:02:35.020140  1717 leader_election.cc:304] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 13:02:35.025385  2300 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:35.031096  9358 raft_consensus.cc:493] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:02:34.988416  9333 raft_consensus.cc:493] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:02:35.051335  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 22 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:02:35.052201  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 82844.2us
I20260709 13:02:34.958979  9226 raft_consensus.cc:515] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:35.067751  1605 raft_consensus.cc:2393] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 22: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:02:35.048179  9409 raft_consensus.cc:3060] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:02:35.099124  1748 raft_consensus.cc:1275] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 20 index: 15. Preceding OpId from leader: term: 22 index: 16. (index mismatch)
I20260709 13:02:35.097198  1644 leader_election.cc:304] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:35.051036  9088 raft_consensus.cc:2749] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 13:02:34.935891  9412 raft_consensus.cc:515] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:35.125348  9323 raft_consensus.cc:2749] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
W20260709 13:02:35.132988  9422 log.cc:927] Time spent T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.201s	user 0.002s	sys 0.000s
I20260709 13:02:35.134873  9412 leader_election.cc:290] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:02:35.135353  9413 log.cc:927] Time spent T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.137s	user 0.000s	sys 0.000s
I20260709 13:02:35.136996  9220 raft_consensus.cc:2804] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 13:02:35.137893  9220 raft_consensus.cc:697] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 22 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:02:35.139206  9220 consensus_queue.cc:237] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.15, Last appended by leader: 15, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:35.102639  9334 raft_consensus.cc:2749] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:02:35.079805  9351 raft_consensus.cc:493] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:35.141309  9405 consensus_queue.cc:1059] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.236s
W20260709 13:02:35.143141  9414 log.cc:927] Time spent T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.143s	user 0.000s	sys 0.001s
I20260709 13:02:35.138140  1752 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f"
dest_uuid: "af520b67945d443b901bbeca37cc5701"
 from {username='slave'} at 127.0.0.1:43156
I20260709 13:02:35.143843  1752 raft_consensus.cc:493] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:02:35.144317  1752 raft_consensus.cc:3060] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:02:35.144971  9219 leader_election.cc:290] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 22 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:35.146453  9424 raft_consensus.cc:493] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:35.147089  9424 raft_consensus.cc:515] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:35.149328  9232 consensus_queue.cc:1048] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 13:02:35.158810  1601 raft_consensus.cc:2410] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 for term 22 because replica has last-logged OpId of term: 20 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 13.
I20260709 13:02:35.160246  9226 leader_election.cc:290] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:35.165483  1749 raft_consensus.cc:1240] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 19. Current term is 20. Ops: []
I20260709 13:02:35.166565  9253 raft_consensus.cc:493] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:35.166955  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:35.167187  9253 raft_consensus.cc:515] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:35.168638  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:35.169620  1751 raft_consensus.cc:2393] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 22: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:02:35.141970  9351 raft_consensus.cc:515] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:35.173012  9351 leader_election.cc:290] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:02:35.174654  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 344037us
I20260709 13:02:35.175813  2301 leader_election.cc:304] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:02:35.177667  9334 raft_consensus.cc:2749] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 13:02:35.068819  9358 raft_consensus.cc:515] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:35.181437  9358 leader_election.cc:290] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:35.182099  9232 consensus_queue.cc:1059] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 18.12, Next index: 13, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 13:02:35.183804  9336 leader_election.cc:290] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 22 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:35.183877  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:35.184733  1603 raft_consensus.cc:2468] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 20.
I20260709 13:02:35.081480  9402 consensus_queue.cc:1048] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.327s
I20260709 13:02:35.121022  1597 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 1885) took 1317 ms. Trace:
W20260709 13:02:35.201323  2300 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:35.081295  9072 raft_consensus.cc:493] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:02:35.243587  1571 leader_election.cc:304] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:35.211606  1597 rpcz_store.cc:276] 0709 13:02:33.803902 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:34.002010 (+198108us) service_pool.cc:224] Handling call
0709 13:02:34.002490 (+   480us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:02:34.003135 (+   645us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:34.003186 (+    51us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:02:34.004095 (+   909us) log.cc:844] Serialized 52 byte log entry
0709 13:02:34.229984 (+225889us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:02:34.230032 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:34.230113 (+    81us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:34.839211 (+609098us) spinlock_profiling.cc:243] Waited 3.84 ms on lock 0x7b4400022b4c. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f4262f9bc85 00007f4262f9c6a2 00007f4265d0e1db 00007f4265d09aaf 00007f4265cfd86e 00007f4265cfcbb6 00007f4265cfc7d8 00007f4265d06221 00007f4265d061d9 00007f4265d06169
0709 13:02:35.100399 (+261188us) raft_consensus.cc:1685] finished
0709 13:02:35.100427 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:35.120898 (+ 20471us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":2014,"raft.run_cpu_time_us":146,"raft.run_wall_time_us":146,"spinlock_wait_cycles":8834816,"thread_start_us":225400,"threads_started":1,"wal-append.queue_time_us":386718}
I20260709 13:02:35.275331  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:35.275879  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 19 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:35.085070  9333 raft_consensus.cc:515] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:35.279979  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:35.280958  1749 raft_consensus.cc:2468] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 20.
I20260709 13:02:35.281677  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 25 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:35.286602  1745 raft_consensus.cc:2393] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 19: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
W20260709 13:02:35.294694  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:35.295432  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:35.283789  2301 leader_election.cc:304] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
W20260709 13:02:35.297437  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 200473us
W20260709 13:02:35.303339  2300 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:35.308565  1644 leader_election.cc:400] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 election: Vote denied by peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625) with higher term. Message: Invalid argument: T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for earlier term 19. Current term is 20.
I20260709 13:02:35.312551  1678 raft_consensus.cc:2468] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 24.
I20260709 13:02:35.272477  9358 raft_consensus.cc:493] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:35.305992  1752 raft_consensus.cc:515] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:35.321234  1748 raft_consensus.cc:1240] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 19. Current term is 20. Ops: []
I20260709 13:02:35.279739  9336 raft_consensus.cc:2804] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Leader election won for term 17
I20260709 13:02:35.326126  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:35.326326  1571 leader_election.cc:304] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:35.329914  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 20 candidate_status { last_received { term: 19 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:35.308646  9226 raft_consensus.cc:493] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:02:35.334421  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:35.335260  1750 raft_consensus.cc:3060] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Advancing to term 22
W20260709 13:02:35.340453  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 349074us
I20260709 13:02:35.340623  1752 leader_election.cc:290] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 20 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:02:35.341825  9396 consensus_meta.cc:287] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.522s	user 0.007s	sys 0.007s
I20260709 13:02:35.372310  1749 raft_consensus.cc:2393] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 20: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:02:35.327543  9334 raft_consensus.cc:2749] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260709 13:02:35.373167  1643 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.500s	user 0.005s	sys 0.017s
W20260709 13:02:35.382100  1643 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.509s	user 0.005s	sys 0.017s
W20260709 13:02:35.382673  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 593838us
I20260709 13:02:35.290869  9072 raft_consensus.cc:515] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:35.343655  9421 raft_consensus.cc:2749] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 13:02:35.334467  9226 raft_consensus.cc:515] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:02:35.407559  1643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:35.387673  9426 raft_consensus.cc:3055] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 13:02:35.371731  9427 raft_consensus.cc:2804] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Leader pre-election won for term 21
W20260709 13:02:35.414170  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 395680us
W20260709 13:02:35.414306  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 61383.7us
W20260709 13:02:35.415275  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 63301.8us
I20260709 13:02:35.378994  9424 leader_election.cc:290] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 13:02:35.424682  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 157597us
I20260709 13:02:35.348790  9358 raft_consensus.cc:515] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:35.409922  9426 raft_consensus.cc:740] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:02:35.416818  9427 raft_consensus.cc:493] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:35.438910  9219 consensus_queue.cc:1059] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 13:02:35.439563  9427 raft_consensus.cc:3060] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:02:35.439937  9219 raft_consensus.cc:3055] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 24 LEADER]: Stepping down as leader of term 24
I20260709 13:02:35.440338  9219 raft_consensus.cc:740] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:02:35.441375  9219 consensus_queue.cc:260] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 24.15, Last appended by leader: 15, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:35.437891  9426 consensus_queue.cc:260] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.13, Last appended by leader: 13, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:02:35.443235  9411 log.cc:927] Time spent T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.599s	user 0.000s	sys 0.015s
I20260709 13:02:35.443823  9426 raft_consensus.cc:3060] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:02:35.444271  9244 raft_consensus.cc:493] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:35.444837  9244 raft_consensus.cc:3060] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Advancing to term 22
W20260709 13:02:35.446141  9408 log.cc:927] Time spent T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.600s	user 0.010s	sys 0.004s
I20260709 13:02:35.400990  9438 raft_consensus.cc:3055] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 23 LEADER]: Stepping down as leader of term 23
I20260709 13:02:35.446972  9438 raft_consensus.cc:740] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:02:35.448148  9438 consensus_queue.cc:260] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 23.12, Last appended by leader: 12, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:35.449828  9219 raft_consensus.cc:3060] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 24 FOLLOWER]: Advancing to term 25
I20260709 13:02:35.434681  1750 raft_consensus.cc:2468] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 22.
I20260709 13:02:35.436236  9333 leader_election.cc:290] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:35.436504  1643 leader_election.cc:304] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:02:35.387380  9396 raft_consensus.cc:515] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:35.457563  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 159036us
I20260709 13:02:35.411289  9213 raft_consensus.cc:493] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:35.450830  9392 log.cc:927] Time spent T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.928s	user 0.006s	sys 0.008s
I20260709 13:02:35.472622  1643 leader_election.cc:304] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 13:02:35.475916  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:35.479604  1643 leader_election.cc:304] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
W20260709 13:02:35.485935  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 49246.1us
W20260709 13:02:35.485874  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 49445.6us
I20260709 13:02:35.486977  1671 raft_consensus.cc:2393] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 19: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:02:35.458276  9429 raft_consensus.cc:493] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:02:35.476256  9407 leader_election.cc:290] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:35.486619  9250 raft_consensus.cc:3060] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:02:35.513495  1718 leader_election.cc:304] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:35.481961  9322 raft_consensus.cc:2804] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:02:35.465799  9213 raft_consensus.cc:515] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:35.491485  9421 raft_consensus.cc:493] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:35.487298  9339 raft_consensus.cc:493] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:35.512588  1717 leader_election.cc:304] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:35.469808  9333 consensus_queue.cc:1048] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 11, Time since last communication: 0.029s
I20260709 13:02:35.518838  1643 leader_election.cc:304] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:02:35.522059  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 20 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:35.525723  1643 leader_election.cc:304] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 13:02:35.520300  9425 log.cc:927] Time spent T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.067s	user 0.001s	sys 0.000s
W20260709 13:02:35.532191  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 43688.9us
I20260709 13:02:35.517561  9322 raft_consensus.cc:493] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:35.511125  9424 consensus_queue.cc:1059] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 20.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.017s
W20260709 13:02:35.577579  2298 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:35.534260  9295 consensus_meta.cc:287] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.750s	user 0.015s	sys 0.003s
I20260709 13:02:35.517970  9351 consensus_queue.cc:1059] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 19.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.020s
W20260709 13:02:35.597152  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 109540us
I20260709 13:02:35.535722  9323 raft_consensus.cc:2749] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
W20260709 13:02:35.587397  9327 consensus_meta.cc:287] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.852s	user 0.017s	sys 0.003s
I20260709 13:02:35.588802  9396 leader_election.cc:290] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:02:35.589367  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 68429.2us
I20260709 13:02:35.590636  9420 raft_consensus.cc:2804] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Leader pre-election won for term 19
W20260709 13:02:35.590974  9419 log.cc:927] Time spent T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.208s	user 0.000s	sys 0.001s
W20260709 13:02:35.591646  9444 log.cc:927] Time spent T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.074s	user 0.001s	sys 0.000s
I20260709 13:02:35.525779  9429 raft_consensus.cc:515] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:35.594421  9427 raft_consensus.cc:515] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:35.523160  9400 consensus_queue.cc:1048] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.058s
I20260709 13:02:35.549906  9322 raft_consensus.cc:3060] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Advancing to term 19
W20260709 13:02:35.600445  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 88025.4us
I20260709 13:02:35.597991  9088 raft_consensus.cc:2749] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 13:02:35.602495  1718 leader_election.cc:304] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:02:35.530507  9339 raft_consensus.cc:515] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:35.603018  8934 consensus_meta.cc:287] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.694s	user 0.016s	sys 0.000s
W20260709 13:02:35.618254  1719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:35.627007  9423 raft_consensus.cc:493] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:35.627681  9423 raft_consensus.cc:515] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:35.604864  9420 raft_consensus.cc:493] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:35.628700  9420 raft_consensus.cc:3060] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:02:35.627044  9439 consensus_queue.cc:1059] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 17.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.142s
I20260709 13:02:35.632047  9446 consensus_queue.cc:1048] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 13:02:35.633708  9446 raft_consensus.cc:2804] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 13:02:35.634092  9446 raft_consensus.cc:493] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
W20260709 13:02:35.635082  9428 log.cc:927] Time spent T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.149s	user 0.000s	sys 0.000s
W20260709 13:02:35.636531  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 55527.2us
I20260709 13:02:35.638583  1644 leader_election.cc:304] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:02:35.642863  9072 leader_election.cc:290] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:35.644518  9429 leader_election.cc:290] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:35.644901  9427 leader_election.cc:290] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 21 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:35.645112  9429 raft_consensus.cc:3060] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:02:35.645900  9400 raft_consensus.cc:2749] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 13:02:35.523370  9421 raft_consensus.cc:515] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:35.648347  9421 leader_election.cc:290] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:35.612550  9432 raft_consensus.cc:493] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:02:35.649384  9432 raft_consensus.cc:515] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:35.651177  9244 raft_consensus.cc:515] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:35.625703  9226 leader_election.cc:290] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:02:35.626267  9434 log.cc:927] Time spent T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.201s	user 0.001s	sys 0.000s
I20260709 13:02:35.654991  9333 consensus_queue.cc:1048] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20260709 13:02:35.658530  9446 raft_consensus.cc:3060] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:02:35.659684  9213 leader_election.cc:290] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:02:35.660840  9398 consensus_meta.cc:287] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.768s	user 0.015s	sys 0.004s
I20260709 13:02:35.660976  9358 leader_election.cc:290] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:02:35.606103  9209 consensus_meta.cc:287] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.990s	user 0.013s	sys 0.005s
I20260709 13:02:35.662155  9439 raft_consensus.cc:2749] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:02:35.662787  9432 leader_election.cc:290] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:35.663231  9439 raft_consensus.cc:3055] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 13:02:35.663709  9439 raft_consensus.cc:740] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:02:35.664949  9439 consensus_queue.cc:260] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.12, Last appended by leader: 12, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } }
W20260709 13:02:35.666667  9435 consensus_queue.cc:1175] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" responder_term: 19 status { last_received { term: 17 index: 11 } last_committed_idx: 11 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 18. Current term is 19. Ops: [18.12-18.12]" } } last_received_current_leader { term: 0 index: 0 } }
I20260709 13:02:35.667558  9226 raft_consensus.cc:493] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:35.668124  9400 raft_consensus.cc:493] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:35.668156  9226 raft_consensus.cc:515] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:35.668673  9400 raft_consensus.cc:515] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:35.671172  9400 leader_election.cc:290] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:35.653430  9244 leader_election.cc:290] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 22 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:35.671644  9213 raft_consensus.cc:493] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:35.682336  9335 consensus_meta.cc:287] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 1.022s	user 0.014s	sys 0.002s
I20260709 13:02:35.682837  9213 raft_consensus.cc:515] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } }
I20260709 13:02:35.685101  9213 leader_election.cc:290] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:35.675977  9226 leader_election.cc:290] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:02:35.632354  9433 log.cc:927] Time spent T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.095s	user 0.000s	sys 0.001s
W20260709 13:02:35.690659  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 165084us
I20260709 13:02:35.691956  1643 leader_election.cc:304] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:02:35.678498  9420 raft_consensus.cc:515] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:35.680917  9339 leader_election.cc:290] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:35.602548  9295 raft_consensus.cc:515] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:35.703732  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 19 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:35.705427  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:35.705632  9448 raft_consensus.cc:493] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:35.706835  9448 raft_consensus.cc:515] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:35.707616  2332 raft_consensus.cc:1275] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 25 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 24 index: 18. Preceding OpId from leader: term: 25 index: 18. (term mismatch)
I20260709 13:02:35.708003  2332 pending_rounds.cc:85] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1: Aborting all ops after (but not including) 17
I20260709 13:02:35.708251  2332 pending_rounds.cc:107] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1: Aborting uncommitted NO_OP operation due to leader change: 24.18
I20260709 13:02:35.708487  2332 raft_consensus.cc:2889] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 25 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:02:35.709197  9448 leader_election.cc:290] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:35.709816  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:35.710096  9420 leader_election.cc:290] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:35.706054  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:35.712767  2332 raft_consensus.cc:2468] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 22.
I20260709 13:02:35.705580  1750 raft_consensus.cc:1240] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 21. Current term is 22. Ops: []
I20260709 13:02:35.703362  9226 raft_consensus.cc:2804] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:02:35.701565  9327 raft_consensus.cc:493] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:35.721637  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:35.723336  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:35.699146  9336 raft_consensus.cc:697] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:02:35.726240  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:35.737386  9327 raft_consensus.cc:515] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:35.725670  1746 raft_consensus.cc:2393] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 22: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:02:35.698900  9432 consensus_queue.cc:1048] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.023s
I20260709 13:02:35.719062  9226 raft_consensus.cc:493] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:02:35.714381  9439 raft_consensus.cc:3060] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:02:35.734843  1752 raft_consensus.cc:2393] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 20: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
W20260709 13:02:35.768525  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 45603.7us
W20260709 13:02:35.771311  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:35.715027  9322 raft_consensus.cc:515] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:35.791682  1749 raft_consensus.cc:1275] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 18 index: 13. Preceding OpId from leader: term: 20 index: 14. (index mismatch)
I20260709 13:02:35.787791  9054 raft_consensus.cc:493] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:35.804301  2331 raft_consensus.cc:2468] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 16.
I20260709 13:02:35.804709  9054 raft_consensus.cc:515] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:35.805902  9322 leader_election.cc:290] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:35.777974  9253 leader_election.cc:290] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:35.808456  9054 leader_election.cc:290] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:35.809885  9446 raft_consensus.cc:515] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:35.820183  9327 leader_election.cc:290] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:02:35.799084  9409 consensus_meta.cc:287] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.560s	user 0.017s	sys 0.000s
I20260709 13:02:35.821112  9446 leader_election.cc:290] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 23 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:35.821298  9409 raft_consensus.cc:515] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:35.824353  9409 leader_election.cc:290] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 16 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:35.799891  9358 raft_consensus.cc:493] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:02:35.826872  9358 raft_consensus.cc:515] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:35.829270  9358 leader_election.cc:290] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:35.837255  9336 consensus_queue.cc:237] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.12, Last appended by leader: 12, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:35.782490  9295 leader_election.cc:290] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 21 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:02:35.853211  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:35.796715  9343 consensus_meta.cc:287] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 1.020s	user 0.014s	sys 0.004s
I20260709 13:02:35.790037  9226 raft_consensus.cc:3060] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Advancing to term 20
W20260709 13:02:35.866894  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 47510.3us
I20260709 13:02:35.854537  9054 raft_consensus.cc:493] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:02:35.868840  9054 raft_consensus.cc:515] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:35.872555  9426 raft_consensus.cc:493] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:35.873170  9244 raft_consensus.cc:493] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:35.873514  9432 raft_consensus.cc:493] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:35.873761  9244 raft_consensus.cc:515] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:35.874039  9432 raft_consensus.cc:515] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:35.873171  9426 raft_consensus.cc:515] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:35.865957  2328 raft_consensus.cc:2468] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 17.
I20260709 13:02:35.893321  1644 leader_election.cc:304] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
W20260709 13:02:35.894811  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 295498us
I20260709 13:02:35.896728  2331 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2"
dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
mode: GRACEFUL
new_leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
 from {username='slave'} at 127.0.0.1:35790
I20260709 13:02:35.897387  2331 raft_consensus.cc:606] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Received request to transfer leadership to TS 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:02:35.897262  1718 leader_election.cc:304] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:35.897909  2331 raft_consensus.cc:614] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 13:02:35.883365  9429 raft_consensus.cc:2749] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20260709 13:02:35.901324  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 63426.3us
I20260709 13:02:35.896281  1644 leader_election.cc:304] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 13:02:35.901400  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:35.900959  9327 raft_consensus.cc:2804] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 13:02:35.903740  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:35.905963  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:35.906627  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:35.907337  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 21 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:35.909590  2301 leader_election.cc:304] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:02:35.910578  1679 raft_consensus.cc:2393] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 22: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:02:35.910696  1673 raft_consensus.cc:1217] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Deduplicated request from leader. Original: 12.8->[14.9-18.11]   Dedup: 14.9->[16.10-18.11]
I20260709 13:02:35.894928  9343 raft_consensus.cc:493] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:35.911330  1673 raft_consensus.cc:1240] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 18. Current term is 19. Ops: [14.9-18.11]
I20260709 13:02:35.894541  9295 consensus_queue.cc:1048] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 15, Time since last communication: 0.014s
I20260709 13:02:35.915230  1678 raft_consensus.cc:1240] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 18. Current term is 19. Ops: [18.13-18.13]
I20260709 13:02:35.916949  1672 raft_consensus.cc:1240] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 18. Current term is 19. Ops: []
I20260709 13:02:35.918926  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:35.918608  9343 raft_consensus.cc:515] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:35.919634  1675 raft_consensus.cc:2468] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 16.
I20260709 13:02:35.916837  9358 raft_consensus.cc:2749] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 13:02:35.923264  9250 raft_consensus.cc:2749] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:02:35.923840  9244 leader_election.cc:290] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:35.925580  9244 raft_consensus.cc:493] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:35.926821  9446 raft_consensus.cc:493] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:35.927018  9253 raft_consensus.cc:493] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:02:35.927423  9446 raft_consensus.cc:515] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:35.927600  9253 raft_consensus.cc:515] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:35.929694  9253 leader_election.cc:290] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:35.930063  9429 raft_consensus.cc:493] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:35.926054  9244 raft_consensus.cc:515] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:35.929685  9446 leader_election.cc:290] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:35.932602  9446 raft_consensus.cc:493] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:35.933118  9446 raft_consensus.cc:515] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:35.933837  9244 leader_election.cc:290] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:35.906752  9322 raft_consensus.cc:2804] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:02:35.934484  9429 raft_consensus.cc:515] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:35.935451  9446 leader_election.cc:290] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:35.935882  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:35.937146  9429 leader_election.cc:290] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 13:02:35.937711  2336 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39726 (request call id 1688) took 1557 ms (client timeout 1769 ms). Trace:
I20260709 13:02:35.907930  9327 raft_consensus.cc:493] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:35.940261  9327 raft_consensus.cc:3060] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Advancing to term 23
W20260709 13:02:35.938264  2336 rpcz_store.cc:269] 0709 13:02:34.379958 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:34.380198 (+   240us) service_pool.cc:224] Handling call
0709 13:02:35.937487 (+1557289us) spinlock_profiling.cc:243] Waited 44.8 ms on lock 0x7b540072e554. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:02:35.937617 (+   130us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":103103360}
I20260709 13:02:35.941830  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:35.931768  1605 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40166 (request call id 3116) took 1028 ms. Trace:
I20260709 13:02:35.935400  9322 raft_consensus.cc:493] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:02:35.943996  8934 raft_consensus.cc:493] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:35.944233  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:35.944670  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 19 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:35.944638  8934 raft_consensus.cc:515] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:35.945683  2329 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39738 (request call id 1332) took 1356 ms. Trace:
I20260709 13:02:35.946074  2329 rpcz_store.cc:276] 0709 13:02:34.589467 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:35.942680 (+1353213us) service_pool.cc:224] Handling call
0709 13:02:35.945592 (+  2912us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:02:35.946924  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:35.947336  9209 raft_consensus.cc:493] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:35.908921  9453 raft_consensus.cc:2749] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:02:35.947892  9209 raft_consensus.cc:515] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:35.948868  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:35.949457  9432 leader_election.cc:290] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:35.944243  9322 raft_consensus.cc:3060] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:02:35.951151  9054 leader_election.cc:290] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:35.943339  1605 rpcz_store.cc:276] 0709 13:02:34.903726 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:34.988565 (+ 84839us) service_pool.cc:224] Handling call
0709 13:02:35.931539 (+942974us) spinlock_profiling.cc:243] Waited 45.9 ms on lock 0x7b540006c2d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:02:35.931696 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":105581824}
I20260709 13:02:35.937907  9336 consensus_queue.cc:1048] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 13:02:35.926944  9246 leader_election.cc:290] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 23 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:35.962489  2332 raft_consensus.cc:1275] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 18 index: 13. Preceding OpId from leader: term: 21 index: 14. (index mismatch)
I20260709 13:02:35.960448  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:35.960615  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:35.962425  1599 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 3106) took 1065 ms. Trace:
I20260709 13:02:35.971283  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:35.972304  1672 raft_consensus.cc:3060] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:02:35.937708  9253 raft_consensus.cc:493] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:35.974064  1679 raft_consensus.cc:2393] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 19: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
W20260709 13:02:35.984771  1601 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36462 (request call id 1883) took 2215 ms (client timeout 2812 ms). Trace:
W20260709 13:02:35.985293  1601 rpcz_store.cc:269] 0709 13:02:33.769453 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:33.874822 (+105369us) service_pool.cc:224] Handling call
0709 13:02:35.984493 (+2109671us) spinlock_profiling.cc:243] Waited 1.26 s on lock 0x7b54001009d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:02:35.984672 (+   179us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2888737024}
I20260709 13:02:35.987013  9253 raft_consensus.cc:515] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:35.987852  2334 raft_consensus.cc:1275] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 19 index: 14. (index mismatch)
I20260709 13:02:35.989750  9253 leader_election.cc:290] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:35.990780  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: true dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:35.991436  2334 raft_consensus.cc:3055] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 13:02:35.991717  2334 raft_consensus.cc:740] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:02:35.992553  2334 consensus_queue.cc:260] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.13, Last appended by leader: 13, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:35.993567  2334 raft_consensus.cc:3060] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:02:35.993489  9226 raft_consensus.cc:515] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:35.970685  1599 rpcz_store.cc:276] 0709 13:02:34.897431 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:34.903043 (+  5612us) service_pool.cc:224] Handling call
0709 13:02:34.936409 (+ 33366us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:35.138519 (+202110us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 13:02:35.138934 (+   415us) log.cc:844] Serialized 14 byte log entry
0709 13:02:35.292436 (+153502us) log.cc:844] Serialized 14 byte log entry
0709 13:02:35.292668 (+   232us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:35.292722 (+    54us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:02:35.292787 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:35.448748 (+155961us) spinlock_profiling.cc:243] Waited 5.53 ms on lock 0x7b180043b820. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f42630753f3 00007f42630777c1 00007f4263077779 00007f4263077709 00007f42630776d1 00007f42630769cd 000000000038f354 000000000038f199 00007f426305883a
0709 13:02:35.961670 (+512922us) spinlock_profiling.cc:243] Waited 13.8 ms on lock 0x7b180043b820. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f42630432a6 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419
0709 13:02:35.962171 (+   501us) spinlock_profiling.cc:243] Waited 199 ms on lock 0x7b54004222d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:02:35.962225 (+    54us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:35.962359 (+   134us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":20623,"prepare.queue_time_us":5097,"prepare.run_cpu_time_us":69,"prepare.run_wall_time_us":69,"raft.queue_time_us":1252,"raft.run_cpu_time_us":292,"raft.run_wall_time_us":11441,"spinlock_wait_cycles":585577088,"thread_start_us":152618,"threads_started":1,"wal-append.queue_time_us":239011}
I20260709 13:02:35.994923  1600 raft_consensus.cc:1275] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 20 index: 14. Preceding OpId from leader: term: 21 index: 14. (term mismatch)
I20260709 13:02:35.994824  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:35.995443  1600 pending_rounds.cc:85] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a: Aborting all ops after (but not including) 13
I20260709 13:02:35.995716  1600 pending_rounds.cc:107] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a: Aborting uncommitted NO_OP operation due to leader change: 20.14
I20260709 13:02:35.995962  1600 raft_consensus.cc:2889] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:02:35.996809  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:35.997242  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 21 candidate_status { last_received { term: 16 index: 7 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:35.998072  1676 raft_consensus.cc:2410] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for term 21 because replica has last-logged OpId of term: 19 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 7.
I20260709 13:02:35.998373  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 26 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:35.999032  1680 raft_consensus.cc:2468] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 20.
I20260709 13:02:36.000185  1679 raft_consensus.cc:1240] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 22. Current term is 23. Ops: [22.11-22.11]
I20260709 13:02:36.000329  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:36.001243  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 26 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:36.002534  9295 raft_consensus.cc:493] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:36.003166  9295 raft_consensus.cc:515] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:36.005226  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:36.005488  9295 leader_election.cc:290] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:36.005882  1677 raft_consensus.cc:2468] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 23.
I20260709 13:02:36.006125  1678 raft_consensus.cc:1240] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 24 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 23. Current term is 24. Ops: [23.12-23.12]
I20260709 13:02:35.999738  2337 raft_consensus.cc:2468] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 25.
I20260709 13:02:35.957923  9426 leader_election.cc:290] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:36.007203  9069 raft_consensus.cc:3055] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 13:02:36.007112  9450 consensus_queue.cc:1059] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 15.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 13:02:36.007771  9069 raft_consensus.cc:740] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:02:36.008446  9336 raft_consensus.cc:3055] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 LEADER]: Stepping down as leader of term 15
I20260709 13:02:36.018016  1672 raft_consensus.cc:2468] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 23.
I20260709 13:02:36.019510  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:36.021162  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:36.021925  2336 raft_consensus.cc:2468] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 18.
I20260709 13:02:36.037078  1718 leader_election.cc:304] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:36.008864  9336 raft_consensus.cc:740] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:02:36.043788  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 23 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:36.045888  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:36.046818  1718 leader_election.cc:304] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:36.044669  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 23 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:36.050015  1718 leader_election.cc:304] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:36.051156  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 21 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:36.051796  1718 leader_election.cc:304] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:36.043720  9435 raft_consensus.cc:2804] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:02:36.008889  9069 consensus_queue.cc:260] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 21.16, Last appended by leader: 16, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:36.041698  9402 consensus_queue.cc:1059] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.014s
I20260709 13:02:36.050616  9453 raft_consensus.cc:2804] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 13:02:36.082402  1674 raft_consensus.cc:1240] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 18. Current term is 19. Ops: []
I20260709 13:02:36.088706  9453 raft_consensus.cc:493] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:36.089179  9453 raft_consensus.cc:3060] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Advancing to term 24
I20260709 13:02:36.080758  9226 leader_election.cc:290] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 20 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:36.077777  9429 raft_consensus.cc:2749] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
W20260709 13:02:36.090386  1671 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52766 (request call id 3324) took 1269 ms (client timeout 1665 ms). Trace:
W20260709 13:02:36.091040  1671 rpcz_store.cc:269] 0709 13:02:34.812943 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:34.814353 (+  1410us) service_pool.cc:224] Handling call
0709 13:02:36.081746 (+1267393us) spinlock_profiling.cc:243] Waited 663 ms on lock 0x7b54001e82d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:02:36.081973 (+   227us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1524075904}
I20260709 13:02:36.045248  9426 consensus_queue.cc:1059] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 17.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.035s
I20260709 13:02:36.062964  9435 raft_consensus.cc:493] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:36.048691  9432 consensus_queue.cc:1059] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 18.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.034s
I20260709 13:02:36.055446  9424 raft_consensus.cc:2804] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Leader election won for term 23
W20260709 13:02:36.103366  2298 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:36.143170  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 94371.3us
I20260709 13:02:36.111533  9437 raft_consensus.cc:3055] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 20 LEADER]: Stepping down as leader of term 20
I20260709 13:02:36.117763  9438 raft_consensus.cc:3060] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Advancing to term 24
W20260709 13:02:36.121021  1571 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:36.125316  1753 raft_consensus.cc:1275] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 17 index: 15. Preceding OpId from leader: term: 19 index: 16. (index mismatch)
I20260709 13:02:36.075062  9339 raft_consensus.cc:493] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:36.146509  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 49089.6us
I20260709 13:02:36.144198  9437 raft_consensus.cc:740] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:02:36.105018  9435 raft_consensus.cc:3060] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:02:36.106606  2330 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39738 (request call id 1331) took 1549 ms. Trace:
I20260709 13:02:36.147688  1718 leader_election.cc:304] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:02:36.053128  8968 consensus_queue.cc:1059] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 20.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.041s
I20260709 13:02:36.148171  2330 rpcz_store.cc:276] 0709 13:02:34.567286 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:34.574740 (+  7454us) service_pool.cc:224] Handling call
0709 13:02:34.594326 (+ 19586us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:34.597720 (+  3394us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:36.106226 (+1508506us) spinlock_profiling.cc:243] Waited 2.41 ms on lock 0x7b54006ef854. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:02:36.106510 (+   284us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5544832}
I20260709 13:02:36.102850  9402 consensus_queue.cc:1243] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer 816fadcea8fc4893882f5f9b9f22a03d log is divergent from this leader: its last log entry 17.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
W20260709 13:02:36.146390  1571 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:36.144893  1646 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:36.142690  9336 consensus_queue.cc:260] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.11, Last appended by leader: 11, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:02:36.310250  2299 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:36.181391  1718 leader_election.cc:304] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:02:36.217439  1674 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57092 (request call id 1702) took 1202 ms. Trace:
I20260709 13:02:36.273190  1600 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40154 (request call id 3008) took 2178 ms. Trace:
W20260709 13:02:36.330936  1571 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:36.209228  2328 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49082 (request call id 1371) took 1712 ms (client timeout 1753 ms). Trace:
I20260709 13:02:36.331761  1600 rpcz_store.cc:276] 0709 13:02:34.095149 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:34.200924 (+105775us) service_pool.cc:224] Handling call
0709 13:02:35.994533 (+1793609us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:35.996446 (+  1913us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:36.272931 (+276485us) spinlock_profiling.cc:243] Waited 1.76 s on lock 0x7b54001009d0. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661618ad 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:02:36.273093 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4047775616}
I20260709 13:02:36.241696  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:02:36.331638  1571 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:36.332017  2328 rpcz_store.cc:269] 0709 13:02:34.496472 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:34.502269 (+  5797us) service_pool.cc:224] Handling call
0709 13:02:36.209013 (+1706744us) spinlock_profiling.cc:243] Waited 1.33 s on lock 0x7b54006b6cd4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:02:36.209153 (+   140us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3070354176}
I20260709 13:02:36.194339  2334 raft_consensus.cc:2468] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 20.
I20260709 13:02:36.330363  1674 rpcz_store.cc:276] 0709 13:02:35.015013 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:35.016513 (+  1500us) service_pool.cc:224] Handling call
0709 13:02:36.082171 (+1065658us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:36.088881 (+  6710us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:36.217170 (+128289us) spinlock_profiling.cc:243] Waited 1.07 s on lock 0x7b54001e82d0. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661618ad 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:02:36.217366 (+   196us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2450921728}
W20260709 13:02:36.329589  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 148428us
I20260709 13:02:36.336597  1718 leader_election.cc:304] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:02:36.242530  9209 leader_election.cc:290] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:36.150759  9452 raft_consensus.cc:3055] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 13:02:36.348429  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:36.348785  9452 raft_consensus.cc:740] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:02:36.247954  1598 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 1895) took 1420 ms. Trace:
I20260709 13:02:36.194018  1569 leader_election.cc:304] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:02:36.350033  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:36.351933  1603 raft_consensus.cc:3060] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:02:36.347409  1718 leader_election.cc:304] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:02:36.350087  1598 rpcz_store.cc:276] 0709 13:02:34.827077 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:36.217675 (+1390598us) service_pool.cc:224] Handling call
0709 13:02:36.218101 (+   426us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:36.218339 (+   238us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:36.218375 (+    36us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:36.218425 (+    50us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:02:36.247645 (+ 29220us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:36.247724 (+    79us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:36.247871 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:36.356046  1718 leader_election.cc:304] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
W20260709 13:02:36.245143  9465 log.cc:927] Time spent T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.156s	user 0.001s	sys 0.000s
I20260709 13:02:36.397948  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 21 candidate_status { last_received { term: 19 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:36.358352  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 19 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:02:36.390728  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:36.353926  9402 raft_consensus.cc:2749] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:02:36.399474  1604 raft_consensus.cc:2410] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for term 19 because replica has last-logged OpId of term: 18 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 12.
I20260709 13:02:36.209975  9327 raft_consensus.cc:515] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:36.399749  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:36.400435  1601 raft_consensus.cc:3060] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:02:36.396550  9447 raft_consensus.cc:2804] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 13:02:36.198746  9437 consensus_queue.cc:260] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.15, Last appended by leader: 15, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:36.396395  9452 consensus_queue.cc:260] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.13, Last appended by leader: 13, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:36.357712  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
W20260709 13:02:36.401862  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 45961.5us
I20260709 13:02:36.354225  9246 consensus_queue.cc:1048] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.069s
I20260709 13:02:36.422933  1718 leader_election.cc:304] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:36.176551  9424 raft_consensus.cc:697] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 23 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:02:36.351372  9439 raft_consensus.cc:493] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:36.150867  9405 consensus_queue.cc:1048] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.266s
I20260709 13:02:36.178871  8968 raft_consensus.cc:2749] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 13:02:36.418824  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:36.418709  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:36.147296  9339 raft_consensus.cc:515] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:36.346382  9244 raft_consensus.cc:2749] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260709 13:02:36.425278  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:36.425661  1605 raft_consensus.cc:2393] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 22: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:02:36.400547  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:36.431223  1601 raft_consensus.cc:2468] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 19.
W20260709 13:02:36.455675  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:36.455793  1604 raft_consensus.cc:2468] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 19.
I20260709 13:02:36.456367  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:36.241077  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:36.425956  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:36.262838  2298 leader_election.cc:304] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
W20260709 13:02:36.454864  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 264334us
I20260709 13:02:36.457001  1601 raft_consensus.cc:3060] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:02:36.185927  9426 consensus_queue.cc:1048] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.237s
I20260709 13:02:36.484588  1605 pending_rounds.cc:85] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a: Aborting all ops after (but not including) 10
I20260709 13:02:36.487284  1605 pending_rounds.cc:107] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a: Aborting uncommitted NO_OP operation due to leader change: 20.11
I20260709 13:02:36.487586  1605 raft_consensus.cc:2889] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:02:36.359491  9451 raft_consensus.cc:493] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:36.451505  1602 raft_consensus.cc:2468] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 19.
I20260709 13:02:36.421787  9447 raft_consensus.cc:493] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:36.421473  9214 raft_consensus.cc:2749] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 13:02:36.487525  1599 raft_consensus.cc:2468] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 20.
I20260709 13:02:36.453472  9402 raft_consensus.cc:2804] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:02:36.493458  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
W20260709 13:02:36.494607  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 232056us
W20260709 13:02:36.506801  2298 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:36.457913  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:36.485656  1598 raft_consensus.cc:2468] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 20.
W20260709 13:02:36.502576  2299 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:36.302418  9322 raft_consensus.cc:515] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:36.457499  9250 consensus_queue.cc:1175] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "af520b67945d443b901bbeca37cc5701" responder_term: 20 status { last_received { term: 20 index: 15 } last_committed_idx: 15 last_received_current_leader { term: 20 index: 15 } }
I20260709 13:02:36.450330  9398 raft_consensus.cc:2749] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 13:02:36.507776  1597 raft_consensus.cc:3060] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:02:36.491279  9452 raft_consensus.cc:3060] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:02:36.491053  9437 raft_consensus.cc:3060] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:02:36.503968  9447 raft_consensus.cc:3060] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 25 FOLLOWER]: Advancing to term 26
I20260709 13:02:36.525477  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:36.457873  9439 raft_consensus.cc:515] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:36.502148  9220 raft_consensus.cc:2749] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:02:36.524096  1569 leader_election.cc:304] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:02:36.524581  1601 raft_consensus.cc:2468] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 19.
I20260709 13:02:36.519934  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:36.529277  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:36.508987  9402 raft_consensus.cc:493] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:02:36.539254  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:36.491215  9432 consensus_queue.cc:1059] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 18.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.032s
I20260709 13:02:36.540057  1598 raft_consensus.cc:2468] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 16.
W20260709 13:02:36.520813  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 149365us
I20260709 13:02:36.547251  1603 raft_consensus.cc:2468] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 19.
I20260709 13:02:36.516888  9327 leader_election.cc:290] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 23 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:02:36.581256  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 156367us
I20260709 13:02:36.543568  1602 raft_consensus.cc:2368] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for earlier term 20. Current term is 21.
I20260709 13:02:36.540462  9253 raft_consensus.cc:493] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:36.585983  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:36.584956  1598 raft_consensus.cc:1275] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 17 index: 15. Preceding OpId from leader: term: 19 index: 16. (index mismatch)
I20260709 13:02:36.509352  9451 raft_consensus.cc:515] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:36.543222  9402 raft_consensus.cc:3060] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:02:36.585321  1718 leader_election.cc:304] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:36.592614  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:36.582477  9450 raft_consensus.cc:2749] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:02:36.596127  9426 raft_consensus.cc:2804] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 13:02:36.590281  9253 raft_consensus.cc:515] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:36.608026  9426 raft_consensus.cc:697] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 20 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:02:36.609071  9426 consensus_queue.cc:237] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.13, Last appended by leader: 13, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:36.610040  9253 leader_election.cc:290] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:02:36.596442  9469 log.cc:927] Time spent T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.142s	user 0.000s	sys 0.001s
I20260709 13:02:36.597239  9446 raft_consensus.cc:493] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:36.611173  9446 raft_consensus.cc:515] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:36.613445  9446 leader_election.cc:290] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:36.597908  9214 raft_consensus.cc:493] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:36.599419  9421 raft_consensus.cc:3055] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 13:02:36.614295  9421 raft_consensus.cc:740] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:02:36.614305  9214 raft_consensus.cc:515] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:36.615330  9421 consensus_queue.cc:260] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 22.11, Last appended by leader: 11, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:36.615990  2298 leader_election.cc:304] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:02:36.616602  9421 raft_consensus.cc:3060] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:02:36.617261  2298 leader_election.cc:304] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
W20260709 13:02:36.600682  9454 log.cc:927] Time spent T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.825s	user 0.000s	sys 0.009s
I20260709 13:02:36.543017  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 22 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:36.584836  9226 raft_consensus.cc:493] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
W20260709 13:02:36.603325  9440 log.cc:927] Time spent T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 1.060s	user 0.016s	sys 0.003s
I20260709 13:02:36.592163  9424 consensus_queue.cc:237] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 22.12, Last appended by leader: 12, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:36.617637  9214 leader_election.cc:290] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 13:02:36.601251  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 94409.9us
I20260709 13:02:36.620668  1601 raft_consensus.cc:2410] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for term 22 because replica has last-logged OpId of term: 20 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 12.
I20260709 13:02:36.619560  1599 raft_consensus.cc:2468] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 25.
I20260709 13:02:36.620699  2298 leader_election.cc:304] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
W20260709 13:02:36.618011  2301 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:36.628819  8893 raft_consensus.cc:2804] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:02:36.628896  9439 leader_election.cc:290] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:36.629340  8893 raft_consensus.cc:493] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:36.629761  8893 raft_consensus.cc:3060] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:02:36.621625  9335 raft_consensus.cc:2804] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:02:36.630896  9226 raft_consensus.cc:515] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:36.635442  9478 raft_consensus.cc:493] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 25 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:02:36.623554  9209 raft_consensus.cc:2804] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:02:36.662171  1597 raft_consensus.cc:2468] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 19.
I20260709 13:02:36.657356  9335 raft_consensus.cc:493] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Starting leader election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:02:36.692442  1569 leader_election.cc:304] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:02:36.663344  9209 raft_consensus.cc:493] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
W20260709 13:02:36.707648  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 87056.7us
I20260709 13:02:36.709800  1596 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 3114) took 1807 ms. Trace:
I20260709 13:02:36.698760  9335 raft_consensus.cc:3060] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:02:36.704056  9209 raft_consensus.cc:3060] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:02:36.698021  9339 leader_election.cc:290] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:36.710266  1596 rpcz_store.cc:276] 0709 13:02:34.902600 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:35.078924 (+176324us) service_pool.cc:224] Handling call
0709 13:02:35.108022 (+ 29098us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:35.187483 (+ 79461us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 13:02:35.187539 (+    56us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:35.239981 (+ 52442us) log.cc:844] Serialized 26 byte log entry
0709 13:02:35.433612 (+193631us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:02:35.433663 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:35.433745 (+    82us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:36.592104 (+1158359us) spinlock_profiling.cc:243] Waited 37 ms on lock 0x7b4400022b4c. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f4262f9bc85 00007f4262f9c6a2 00007f4265d0e1db 00007f4265d09aaf 00007f4265cfd86e 00007f4265cfcbb6 00007f4265cfc7d8 00007f4265d06221 00007f4265d061d9 00007f4265d06169
0709 13:02:36.700796 (+108692us) raft_consensus.cc:1685] finished
0709 13:02:36.700824 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:36.709698 (+  8874us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":602,"raft.run_cpu_time_us":239,"raft.run_wall_time_us":12240,"spinlock_wait_cycles":85175680,"thread_start_us":111197,"threads_started":1,"wal-append.queue_time_us":200083}
I20260709 13:02:36.674899  9478 raft_consensus.cc:515] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:36.716022  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:36.718111  1597 raft_consensus.cc:2393] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 21: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:02:36.709869  9322 leader_election.cc:290] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 17 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:36.714951  9398 raft_consensus.cc:2749] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:02:36.728103  9343 leader_election.cc:290] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:36.773113  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: true dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:36.728157  9451 leader_election.cc:290] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:36.731595  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:36.735587  9439 raft_consensus.cc:493] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:36.735745  1719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:36.736390  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:36.737562  9476 raft_consensus.cc:493] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:36.738065  1571 leader_election.cc:304] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
W20260709 13:02:36.807617  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 69685.1us
I20260709 13:02:36.741866  9220 consensus_queue.cc:1048] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.000s
W20260709 13:02:36.743809  9410 log.cc:927] Time spent T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 1.895s	user 0.021s	sys 0.048s
I20260709 13:02:36.744475  1748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43156 (request call id 1547) took 1581 ms. Trace:
I20260709 13:02:36.749782  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:36.814281  1603 raft_consensus.cc:2393] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 25: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:02:36.742630  9282 consensus_queue.cc:1048] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.016s
I20260709 13:02:36.761330  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:36.813863  1748 rpcz_store.cc:276] 0709 13:02:35.162882 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:35.164254 (+  1372us) service_pool.cc:224] Handling call
0709 13:02:35.164691 (+   437us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:35.321688 (+156997us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:36.744225 (+1422537us) spinlock_profiling.cc:243] Waited 156 ms on lock 0x7b54001d56d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:02:36.744409 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":359748480}
I20260709 13:02:36.817756  2337 raft_consensus.cc:3060] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Advancing to term 17
W20260709 13:02:36.770743  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 80128us
I20260709 13:02:36.742301  2333 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39726 (request call id 1699) took 1046 ms. Trace:
I20260709 13:02:36.776072  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:02:36.819897  1719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:36.782701  9435 consensus_meta.cc:287] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.633s	user 0.002s	sys 0.010s
W20260709 13:02:36.783499  1645 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:36.784252  9409 raft_consensus.cc:493] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:02:36.786983  9322 raft_consensus.cc:493] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:36.850404  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:36.791713  8934 leader_election.cc:290] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 13:02:36.800441  1643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:36.856719  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:36.857928  1718 leader_election.cc:304] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
W20260709 13:02:36.804877  9470 log.cc:927] Time spent T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.626s	user 0.014s	sys 0.000s
I20260709 13:02:36.805444  9429 raft_consensus.cc:3055] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 13:02:36.782716  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:36.812096  1597 raft_consensus.cc:3060] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:02:36.815716  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:36.865101  2328 raft_consensus.cc:3060] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:02:36.817972  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:36.827854  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 21 candidate_status { last_received { term: 16 index: 7 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:36.866990  1603 raft_consensus.cc:2410] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for term 21 because replica has last-logged OpId of term: 19 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 7.
I20260709 13:02:36.820360  1605 raft_consensus.cc:2468] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 19.
W20260709 13:02:36.852102  9453 consensus_meta.cc:287] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.758s	user 0.015s	sys 0.001s
I20260709 13:02:36.846594  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:36.879709  1680 raft_consensus.cc:1240] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 21. Current term is 22. Ops: []
I20260709 13:02:36.844952  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:36.882957  1752 raft_consensus.cc:2468] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 16.
I20260709 13:02:36.880494  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:36.818692  2333 rpcz_store.cc:276] 0709 13:02:35.695757 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:35.695919 (+   162us) service_pool.cc:224] Handling call
0709 13:02:35.940257 (+244338us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:35.943454 (+  3197us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:35.943478 (+    24us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:35.943514 (+    36us) raft_consensus.cc:1645] Marking committed up to 14
0709 13:02:35.943536 (+    22us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:35.943586 (+    50us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:36.742112 (+798526us) spinlock_profiling.cc:243] Waited 235 ms on lock 0x7b540072e550. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661618ad 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:02:36.742234 (+   122us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2770,"raft.queue_time_us":5989,"raft.run_cpu_time_us":126,"raft.run_wall_time_us":126,"spinlock_wait_cycles":541573376}
I20260709 13:02:36.868954  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:36.883806  1599 raft_consensus.cc:2468] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 22.
I20260709 13:02:36.882077  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:36.882381  1598 raft_consensus.cc:2468] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 16.
I20260709 13:02:36.886054  1718 leader_election.cc:304] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:36.887202  1603 pending_rounds.cc:85] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a: Aborting all ops after (but not including) 13
I20260709 13:02:36.887549  1603 pending_rounds.cc:107] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a: Aborting uncommitted NO_OP operation due to leader change: 18.14
I20260709 13:02:36.887790  1603 raft_consensus.cc:2889] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:02:36.849372  9451 raft_consensus.cc:493] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:36.884992  1602 raft_consensus.cc:2468] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 23.
I20260709 13:02:36.855144  1717 leader_election.cc:304] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:36.859622  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:36.891476  1750 raft_consensus.cc:2468] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 16.
I20260709 13:02:36.892318  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:36.892463  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 21 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:36.893113  1750 raft_consensus.cc:3060] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:02:36.893364  1745 raft_consensus.cc:2410] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for term 20 because replica has last-logged OpId of term: 18 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 11.
I20260709 13:02:36.860528  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:36.818573  9253 raft_consensus.cc:2804] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Leader election won for term 22
I20260709 13:02:36.817245  9250 raft_consensus.cc:493] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:36.851332  9438 consensus_meta.cc:287] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.702s	user 0.001s	sys 0.013s
I20260709 13:02:36.819185  2298 leader_election.cc:304] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:02:36.893638  9446 raft_consensus.cc:3060] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:02:36.860044  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 21 candidate_status { last_received { term: 19 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:36.930161  1646 leader_election.cc:304] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:36.930418  1674 raft_consensus.cc:1240] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 20. Current term is 21. Ops: [20.14-20.14]
I20260709 13:02:36.888489  9453 raft_consensus.cc:515] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:36.934563  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:36.864310  9429 raft_consensus.cc:740] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:02:36.900056  1597 raft_consensus.cc:2468] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 20.
I20260709 13:02:36.939330  9246 raft_consensus.cc:493] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:36.940008  9246 raft_consensus.cc:515] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:36.940255  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:36.859819  9322 raft_consensus.cc:515] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:36.941958  9226 leader_election.cc:290] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:36.942404  9246 leader_election.cc:290] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:36.942597  9226 raft_consensus.cc:493] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:36.943150  9246 raft_consensus.cc:2804] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:02:36.943137  9226 raft_consensus.cc:515] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:36.943820  9246 raft_consensus.cc:493] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:02:36.944355  9246 raft_consensus.cc:3060] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:02:36.944963  9322 leader_election.cc:290] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:36.945572  9226 leader_election.cc:290] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:36.854856  9439 raft_consensus.cc:515] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:02:36.946789  9481 log.cc:927] Time spent T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.087s	user 0.000s	sys 0.001s
I20260709 13:02:36.948465  9439 leader_election.cc:290] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:36.948850  1604 raft_consensus.cc:1240] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 18. Current term is 19. Ops: [18.13-18.13]
I20260709 13:02:36.949158  1746 raft_consensus.cc:2468] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 20.
I20260709 13:02:36.853799  9409 raft_consensus.cc:515] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:36.950161  1604 raft_consensus.cc:1240] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 18. Current term is 19. Ops: []
I20260709 13:02:36.863969  9432 consensus_queue.cc:1048] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.087s
I20260709 13:02:36.911062  9484 consensus_queue.cc:1059] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 18.11, Next index: 12, Last known committed idx: 10, Time since last communication: 0.071s
I20260709 13:02:36.910629  1601 raft_consensus.cc:1240] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 18. Current term is 19. Ops: [16.10-18.11]
I20260709 13:02:36.884222  1754 raft_consensus.cc:2468] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 22.
I20260709 13:02:36.906869  2337 raft_consensus.cc:2468] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 17.
I20260709 13:02:36.906701  1717 leader_election.cc:304] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:36.854353  9483 raft_consensus.cc:493] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:36.971181  1750 raft_consensus.cc:2468] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 21.
I20260709 13:02:36.926761  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 21 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:02:36.926942  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 71212.5us
I20260709 13:02:36.929425  9423 leader_election.cc:290] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 13:02:36.930193  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 111154us
I20260709 13:02:36.931789  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:36.933434  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:36.935461  1752 raft_consensus.cc:2410] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for term 19 because replica has last-logged OpId of term: 18 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 12.
I20260709 13:02:36.930847  1751 raft_consensus.cc:2468] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 20.
I20260709 13:02:36.864718  8968 raft_consensus.cc:2749] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:02:36.999084  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:36.937173  9054 raft_consensus.cc:2804] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 13:02:36.951869  9478 leader_election.cc:290] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:36.850425  9435 raft_consensus.cc:515] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:36.939313  9429 consensus_queue.cc:260] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 18.11, Last appended by leader: 11, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:02:36.952785  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 47944.8us
I20260709 13:02:36.910926  1753 raft_consensus.cc:2393] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 22: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
W20260709 13:02:36.955257  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 48700.8us
I20260709 13:02:37.000777  1644 leader_election.cc:304] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:02:36.953285  9453 leader_election.cc:290] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 24 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:36.956063  9432 raft_consensus.cc:2804] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:02:36.956833  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:37.018921  1754 raft_consensus.cc:2468] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 23.
I20260709 13:02:36.957298  9409 leader_election.cc:290] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:36.889257  9447 raft_consensus.cc:515] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:36.972716  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:36.880708  9476 raft_consensus.cc:515] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:36.988741  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:36.992978  1599 raft_consensus.cc:2410] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for term 21 because replica has last-logged OpId of term: 19 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 13.
I20260709 13:02:36.994401  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:36.997875  9489 raft_consensus.cc:493] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:36.868304  9343 consensus_queue.cc:1048] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.091s
W20260709 13:02:37.003125  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 73094.3us
I20260709 13:02:37.002836  9054 raft_consensus.cc:493] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 25 FOLLOWER]: Starting leader election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:02:37.007834  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:37.036339  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:37.037257  1672 raft_consensus.cc:2393] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 23: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:02:37.029774  1646 leader_election.cc:304] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
W20260709 13:02:37.013569  1717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:37.015009  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:37.017194  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:37.039045  1749 raft_consensus.cc:1240] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 26 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 25. Current term is 26. Ops: []
I20260709 13:02:37.039114  1750 raft_consensus.cc:2468] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 22.
I20260709 13:02:36.952798  9253 raft_consensus.cc:697] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:02:37.044094  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:37.022465  9423 raft_consensus.cc:493] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:37.048831  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:37.016964  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:37.020534  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:36.899900  9214 raft_consensus.cc:2804] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:02:37.002738  9220 raft_consensus.cc:493] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:37.024910  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 139051us
I20260709 13:02:37.026237  2337 raft_consensus.cc:2393] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 17: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:02:36.954655  9451 raft_consensus.cc:515] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:37.033633  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:37.034057  2328 raft_consensus.cc:2468] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 23.
I20260709 13:02:37.034740  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:36.999781  2332 raft_consensus.cc:2468] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 22.
I20260709 13:02:37.039623  9054 raft_consensus.cc:3060] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 25 FOLLOWER]: Advancing to term 26
I20260709 13:02:37.041041  1717 leader_election.cc:304] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 13:02:37.022670  9432 raft_consensus.cc:493] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:37.073758  9432 raft_consensus.cc:3060] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:02:37.074687  9476 leader_election.cc:290] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:37.075585  9451 leader_election.cc:290] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:36.960301  9069 raft_consensus.cc:3060] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:02:37.024292  9437 raft_consensus.cc:2804] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Leader election won for term 17
I20260709 13:02:36.990149  9250 raft_consensus.cc:515] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:37.084498  9250 leader_election.cc:290] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:37.085093  9453 raft_consensus.cc:2804] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 13:02:37.085497  9453 raft_consensus.cc:493] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:37.085551  9482 raft_consensus.cc:3055] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 LEADER]: Stepping down as leader of term 18
I20260709 13:02:37.085866  9453 raft_consensus.cc:3060] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:02:37.086230  9482 raft_consensus.cc:740] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:02:37.086691  1671 raft_consensus.cc:2468] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 16.
I20260709 13:02:37.087323  9482 consensus_queue.cc:260] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.11, Last appended by leader: 11, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:02:37.088778  2299 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:37.088970  9482 raft_consensus.cc:3060] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:02:37.091159  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:37.092865  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:37.093406  1752 raft_consensus.cc:2468] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 20.
I20260709 13:02:37.095254  9072 raft_consensus.cc:493] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:37.095818  9072 raft_consensus.cc:515] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:37.046517  1643 leader_election.cc:304] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:37.017509  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:37.044764  1599 raft_consensus.cc:2468] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 20.
I20260709 13:02:37.049973  1569 leader_election.cc:304] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:02:37.051268  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:37.094250  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:37.051397  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:37.051640  1745 raft_consensus.cc:2393] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 20: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:02:37.101217  1602 raft_consensus.cc:3060] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 23 FOLLOWER]: Advancing to term 24
I20260709 13:02:37.048162  1646 leader_election.cc:304] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 13:02:37.055169  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:37.071082  1644 leader_election.cc:304] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:02:37.055671  9423 raft_consensus.cc:515] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:37.057495  1747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43156 (request call id 1558) took 1333 ms. Trace:
I20260709 13:02:37.035477  9429 raft_consensus.cc:3060] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:02:37.061707  1674 raft_consensus.cc:2468] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 22.
I20260709 13:02:37.044397  9409 raft_consensus.cc:2804] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:02:37.063102  9253 consensus_queue.cc:237] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 20.11, Last appended by leader: 11, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:37.023450  8893 raft_consensus.cc:515] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:37.064400  9220 raft_consensus.cc:515] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:37.065711  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:37.071317  1679 raft_consensus.cc:1240] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 15. Current term is 16. Ops: []
I20260709 13:02:37.111956  1750 raft_consensus.cc:2468] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 20.
I20260709 13:02:37.072602  1717 leader_election.cc:304] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:36.991927  9483 raft_consensus.cc:515] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:37.052202  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:37.101830  9214 raft_consensus.cc:493] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:37.114076  9476 raft_consensus.cc:493] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:37.114485  9214 raft_consensus.cc:3060] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:02:37.114940  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:37.115636  1677 raft_consensus.cc:2468] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 23.
I20260709 13:02:37.118016  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:37.119493  9476 raft_consensus.cc:515] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:37.121886  9476 leader_election.cc:290] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:37.123904  9220 leader_election.cc:290] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:37.126005  8893 leader_election.cc:290] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 17 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:37.127058  9072 leader_election.cc:290] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 13:02:37.128331  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 80292us
I20260709 13:02:37.128635  9447 leader_election.cc:290] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 26 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:37.045724  9489 raft_consensus.cc:515] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:37.131026  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 84653.9us
I20260709 13:02:37.131489  9489 leader_election.cc:290] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:02:37.131942  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:37.132105  9489 raft_consensus.cc:2804] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Leader election won for term 21
I20260709 13:02:37.132839  9489 raft_consensus.cc:697] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:02:37.110965  9246 raft_consensus.cc:515] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:37.133980  9474 raft_consensus.cc:2749] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
W20260709 13:02:37.135387  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 64415.7us
I20260709 13:02:37.136147  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:37.137332  1644 leader_election.cc:304] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:02:37.137962  2334 raft_consensus.cc:1240] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 16. Current term is 17. Ops: []
I20260709 13:02:37.138942  2331 raft_consensus.cc:2468] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 20.
I20260709 13:02:37.143646  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 17 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:37.118587  1671 raft_consensus.cc:2468] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 16.
I20260709 13:02:37.132041  1643 leader_election.cc:304] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:02:37.145602  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:37.146878  2332 raft_consensus.cc:2410] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for term 17 because replica has last-logged OpId of term: 16 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 13.
W20260709 13:02:37.146893  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 74400.3us
I20260709 13:02:37.104696  1747 rpcz_store.cc:276] 0709 13:02:35.724542 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:35.724806 (+   264us) service_pool.cc:224] Handling call
0709 13:02:35.725241 (+   435us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:35.785145 (+ 59904us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 13:02:35.838761 (+ 53616us) log.cc:844] Serialized 14 byte log entry
0709 13:02:35.955315 (+116554us) spinlock_profiling.cc:243] Waited 4.73 ms on lock 0x7b1800341080. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f4265d0a4bc 00007f426305848b 00007f426beb2ab9 00007f426307469b 00007f426307293f 00007f4263074c02 00007f4265cfc20b 00007f4265d01d23 00007f4265d0223a
0709 13:02:35.960467 (+  5152us) log.cc:844] Serialized 14 byte log entry
0709 13:02:35.960709 (+   242us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:35.961307 (+   598us) log.cc:844] Serialized 26 byte log entry
0709 13:02:35.961460 (+   153us) raft_consensus.cc:1645] Marking committed up to 14
0709 13:02:35.961697 (+   237us) log.cc:844] Serialized 14 byte log entry
0709 13:02:35.961867 (+   170us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:35.962022 (+   155us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:37.029180 (+1067158us) spinlock_profiling.cc:243] Waited 15.4 ms on lock 0x7b1800341080. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f42630753f3 00007f42630777c1 00007f4263077779 00007f4263077709 00007f42630776d1 00007f42630769cd 000000000038f354 000000000038f199 00007f426305883a
0709 13:02:37.057265 (+ 28085us) raft_consensus.cc:1685] finished
0709 13:02:37.057292 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:37.057424 (+   132us) inbound_call.cc:177] Queueing success response
0709 13:02:37.076662 (+ 19238us) spinlock_profiling.cc:243] Waited 2.44 ms on lock 0x7b1800341080. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f42630753f3 00007f42630777c1 00007f4263077779 00007f4263077709 00007f42630776d1 00007f42630769cd 000000000038f354 000000000038f199 00007f426305883a
Metrics: {"mutex_wait_us":3945,"prepare.queue_time_us":1163770,"prepare.run_cpu_time_us":108,"prepare.run_wall_time_us":1456,"raft.queue_time_us":13252,"raft.run_cpu_time_us":547,"raft.run_wall_time_us":18944,"spinlock_wait_cycles":51991040,"thread_start_us":37543,"threads_started":1,"wal-append.queue_time_us":29229}
I20260709 13:02:37.150048  1602 raft_consensus.cc:2468] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 24.
I20260709 13:02:37.152061  1604 raft_consensus.cc:1275] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 19 index: 14. Preceding OpId from leader: term: 20 index: 15. (index mismatch)
I20260709 13:02:37.153318  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:37.040134  9402 raft_consensus.cc:515] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:37.160390  1604 raft_consensus.cc:1240] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 18. Current term is 19. Ops: []
I20260709 13:02:37.162551  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:37.156747  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 23 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:37.157402  1643 leader_election.cc:304] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:02:37.157729  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:37.111304  9409 raft_consensus.cc:493] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:37.158983  9220 consensus_queue.cc:1059] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 18.13, Next index: 14, Last known committed idx: 12, Time since last communication: 0.023s
I20260709 13:02:37.183089  9474 raft_consensus.cc:2804] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Leader election won for term 19
I20260709 13:02:37.184116  9402 leader_election.cc:290] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 19 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:37.116619  8994 consensus_queue.cc:1059] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 18.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.068s
I20260709 13:02:37.184998  9483 leader_election.cc:290] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:37.185631  9483 raft_consensus.cc:3055] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 13:02:37.166023  1747 raft_consensus.cc:2468] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 22.
I20260709 13:02:37.101662  9437 raft_consensus.cc:697] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 17 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:02:37.169016  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:37.169510  1746 raft_consensus.cc:2410] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for term 23 because replica has last-logged OpId of term: 22 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 10.
I20260709 13:02:37.163388  1599 raft_consensus.cc:2468] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 21.
I20260709 13:02:37.174666  9336 raft_consensus.cc:3060] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:02:37.174669  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 21 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:37.146456  8934 raft_consensus.cc:493] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:37.187093  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:37.209913  1747 raft_consensus.cc:2468] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 22.
I20260709 13:02:37.168874  1643 leader_election.cc:304] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:02:37.206712  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 22 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:37.178653  9409 raft_consensus.cc:3060] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:02:37.206579  1717 leader_election.cc:304] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:37.213976  1717 leader_election.cc:304] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:37.155642  9447 raft_consensus.cc:493] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:37.150244  9439 raft_consensus.cc:2749] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:02:37.150040  9295 raft_consensus.cc:2804] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 13:02:37.159508  9250 raft_consensus.cc:2804] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Leader election won for term 23
W20260709 13:02:37.214788  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 46018.5us
I20260709 13:02:37.221582  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:37.222210  1643 leader_election.cc:304] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:37.224061  1643 leader_election.cc:304] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:02:37.225456  1643 leader_election.cc:304] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 13:02:37.215237  1750 raft_consensus.cc:2468] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 21.
I20260709 13:02:37.133975  9489 consensus_queue.cc:237] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.13, Last appended by leader: 14, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:37.185954  9483 raft_consensus.cc:740] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:02:37.216858  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:37.218571  1717 leader_election.cc:304] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:37.208192  9492 raft_consensus.cc:2804] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:02:37.210268  9474 raft_consensus.cc:697] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:02:37.231393  1747 raft_consensus.cc:2468] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 17.
I20260709 13:02:37.240068  1646 leader_election.cc:304] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 13:02:37.240658  1717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:37.228165  1643 leader_election.cc:400] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 20 election: Vote denied by peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445) with higher term. Message: Invalid argument: T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for earlier term 20. Current term is 21.
I20260709 13:02:37.243299  1643 leader_election.cc:403] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 20 election: Cancelling election due to peer responding with higher term
I20260709 13:02:37.219072  9226 raft_consensus.cc:2804] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:02:37.219462  9244 raft_consensus.cc:2749] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:02:37.245399  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:37.246125  1750 raft_consensus.cc:3060] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:02:37.221455  9295 raft_consensus.cc:493] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:37.247125  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:37.247895  1753 raft_consensus.cc:2393] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 19: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:02:37.250000  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 22 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:37.250890  1751 raft_consensus.cc:2410] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for term 22 because replica has last-logged OpId of term: 20 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 12.
I20260709 13:02:37.250769  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:37.207540  9402 consensus_queue.cc:1059] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 18.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.026s
I20260709 13:02:37.230762  9424 raft_consensus.cc:2804] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:02:37.231230  9327 raft_consensus.cc:2749] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 13:02:37.216289  8934 raft_consensus.cc:515] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:37.286696  1753 raft_consensus.cc:1275] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 19 index: 14. Preceding OpId from leader: term: 20 index: 15. (index mismatch)
I20260709 13:02:37.231036  9491 raft_consensus.cc:2804] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Leader election won for term 19
I20260709 13:02:37.212580  9335 raft_consensus.cc:515] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:37.222400  9447 raft_consensus.cc:515] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } }
I20260709 13:02:37.230782  9322 raft_consensus.cc:2804] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Leader election won for term 19
I20260709 13:02:37.252149  9214 raft_consensus.cc:515] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:37.239220  9426 raft_consensus.cc:2804] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 13:02:37.238191  9492 raft_consensus.cc:493] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:37.256417  9295 raft_consensus.cc:3060] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:02:37.247644  9226 raft_consensus.cc:493] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:37.278700  9424 raft_consensus.cc:493] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:37.373487  9424 raft_consensus.cc:3060] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Advancing to term 21
W20260709 13:02:37.374152  9494 log.cc:927] Time spent T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.157s	user 0.000s	sys 0.001s
W20260709 13:02:37.376484  9493 log.cc:927] Time spent T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.327s	user 0.001s	sys 0.000s
I20260709 13:02:37.372920  9226 raft_consensus.cc:3060] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:02:37.383315  9246 leader_election.cc:290] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 17 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:37.384222  9246 raft_consensus.cc:493] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:37.384742  9246 raft_consensus.cc:515] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:37.386678  9358 consensus_queue.cc:1048] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
W20260709 13:02:37.407122  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 309155us
I20260709 13:02:37.410629  2299 leader_election.cc:304] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:37.412596  2299 leader_election.cc:304] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:37.416383  1750 raft_consensus.cc:2468] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 23.
I20260709 13:02:37.288753  9452 raft_consensus.cc:493] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:37.424578  9246 leader_election.cc:290] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:37.442961  9246 raft_consensus.cc:3060] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:02:37.442799  9452 raft_consensus.cc:515] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:37.387550  9226 raft_consensus.cc:515] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:37.445341  9452 leader_election.cc:290] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:37.446229  9226 leader_election.cc:290] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 21 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:37.446931  9226 raft_consensus.cc:2804] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:02:37.447331  9226 raft_consensus.cc:493] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:37.447667  9226 raft_consensus.cc:3060] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:02:37.450744  9486 raft_consensus.cc:493] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:37.451350  9486 raft_consensus.cc:515] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:37.453562  9486 leader_election.cc:290] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:37.458086  9246 raft_consensus.cc:2749] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Leader election lost for term 20. Reason: Vote denied by peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445) with higher term. Message: Invalid argument: T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for earlier term 20. Current term is 21.
I20260709 13:02:37.472247  9226 raft_consensus.cc:515] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:37.476969  9447 leader_election.cc:290] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:37.481873  9474 consensus_queue.cc:237] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 18.9, Last appended by leader: 9, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:37.487663  9335 leader_election.cc:290] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 20 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:37.488600  9335 raft_consensus.cc:2804] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:02:37.489032  9335 raft_consensus.cc:493] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:02:37.489485  9335 raft_consensus.cc:3060] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:02:37.495666  8934 leader_election.cc:290] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:37.496328  8934 raft_consensus.cc:2804] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Leader pre-election won for term 18
W20260709 13:02:37.435437  1719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:37.439062  9069 raft_consensus.cc:493] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:37.434682  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 206830us
W20260709 13:02:37.514700  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 274898us
I20260709 13:02:37.515372  1643 leader_election.cc:304] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:02:37.371622  9492 raft_consensus.cc:3060] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Advancing to term 20
W20260709 13:02:37.517710  1719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:37.519716  9486 raft_consensus.cc:2804] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Leader election won for term 19
I20260709 13:02:37.520896  1643 leader_election.cc:304] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:37.391793  9426 raft_consensus.cc:697] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 24 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:02:37.536691  1646 leader_election.cc:304] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:02:37.391472  9322 raft_consensus.cc:697] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:02:37.409574  9491 raft_consensus.cc:697] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 19 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:02:37.552448  1646 leader_election.cc:304] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 13:02:37.496711  8934 raft_consensus.cc:493] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:37.510741  9501 raft_consensus.cc:493] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:37.563166  2301 leader_election.cc:304] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:37.590699  2301 leader_election.cc:304] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:02:37.550916  9452 raft_consensus.cc:493] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:37.596395  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:37.598496  2301 leader_election.cc:304] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
W20260709 13:02:37.599200  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 389778us
I20260709 13:02:37.599494  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:37.559414  8934 raft_consensus.cc:3060] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:02:37.391402  9437 consensus_queue.cc:237] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.14, Last appended by leader: 14, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:37.574607  9438 raft_consensus.cc:493] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:37.605384  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:37.606361  1748 raft_consensus.cc:3060] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Advancing to term 17
W20260709 13:02:37.608084  1717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:37.546744  9069 raft_consensus.cc:515] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:37.575573  9250 raft_consensus.cc:697] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:02:37.591288  9226 leader_election.cc:290] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 21 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:37.556859  9453 raft_consensus.cc:515] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:37.613219  9232 raft_consensus.cc:3055] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 13:02:37.598424  9423 leader_election.cc:290] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:37.632071  1644 leader_election.cc:304] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:02:37.639181  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:37.639984  2331 raft_consensus.cc:2468] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 23.
W20260709 13:02:37.621582  9432 consensus_meta.cc:287] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.547s	user 0.009s	sys 0.004s
I20260709 13:02:37.647969  1750 raft_consensus.cc:1240] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 22. Current term is 23. Ops: []
I20260709 13:02:37.597586  9054 raft_consensus.cc:515] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:37.642846  9429 consensus_meta.cc:287] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.525s	user 0.011s	sys 0.001s
I20260709 13:02:37.608021  9452 raft_consensus.cc:515] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:37.627568  9232 raft_consensus.cc:740] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:02:37.615317  9514 raft_consensus.cc:3060] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:02:37.592279  9501 raft_consensus.cc:515] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:37.661252  9432 raft_consensus.cc:515] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:37.687716  1752 raft_consensus.cc:2410] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for term 19 because replica has last-logged OpId of term: 18 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 10.
I20260709 13:02:37.639637  9511 raft_consensus.cc:2749] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 13:02:37.660713  1748 raft_consensus.cc:2468] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 17.
I20260709 13:02:37.689215  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
W20260709 13:02:37.689764  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 80037.4us
I20260709 13:02:37.645673  9517 raft_consensus.cc:2804] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:02:37.708449  9517 raft_consensus.cc:493] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:37.708953  9517 raft_consensus.cc:3060] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:02:37.620227  9438 raft_consensus.cc:515] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:37.711691  9069 leader_election.cc:290] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:37.712229  9438 leader_election.cc:290] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:37.647027  9322 consensus_queue.cc:237] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 18.15, Last appended by leader: 15, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:37.713737  9054 leader_election.cc:290] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 26 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:37.714823  9453 leader_election.cc:290] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 23 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:37.690672  2298 leader_election.cc:304] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:02:37.689018  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:37.695605  9232 consensus_queue.cc:260] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 18.9, Last appended by leader: 9, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:37.686669  9491 consensus_queue.cc:237] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 17.10, Last appended by leader: 10, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:37.663868  9246 raft_consensus.cc:2804] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Leader pre-election won for term 23
W20260709 13:02:37.687000  9209 consensus_meta.cc:287] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.965s	user 0.011s	sys 0.003s
I20260709 13:02:37.668270  9518 raft_consensus.cc:493] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:37.703586  9482 consensus_meta.cc:287] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.614s	user 0.014s	sys 0.000s
W20260709 13:02:37.704509  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 670755us
W20260709 13:02:37.705735  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:37.755746  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:37.756122  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:37.706593  9452 leader_election.cc:290] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:37.707659  9339 raft_consensus.cc:2804] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Leader election won for term 19
I20260709 13:02:37.723803  1749 raft_consensus.cc:2468] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 24.
I20260709 13:02:37.733983  9453 raft_consensus.cc:493] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:37.760519  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:37.668769  9424 raft_consensus.cc:515] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:37.735117  1601 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40154 (request call id 3018) took 1004 ms. Trace:
I20260709 13:02:37.656431  9486 raft_consensus.cc:697] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:02:37.668939  9489 raft_consensus.cc:2804] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Leader pre-election won for term 21
W20260709 13:02:37.743479  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 95946.8us
W20260709 13:02:37.743423  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 331049us
I20260709 13:02:37.787148  1601 rpcz_store.cc:276] 0709 13:02:36.730763 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:36.730887 (+   124us) service_pool.cc:224] Handling call
0709 13:02:36.731217 (+   330us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:02:36.952490 (+221273us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:37.734364 (+781874us) spinlock_profiling.cc:243] Waited 179 ms on lock 0x7b54003edfd4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:02:37.735039 (+   675us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":411929984}
I20260709 13:02:37.717497  9432 leader_election.cc:290] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 20 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:37.762601  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 20 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:02:37.724844  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 172548us
I20260709 13:02:37.748289  9506 raft_consensus.cc:493] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:37.775583  9453 raft_consensus.cc:515] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:02:37.776757  1571 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:37.811081  1571 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:37.811975  1571 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:37.812412  1571 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:37.777169  1746 raft_consensus.cc:2468] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 22.
I20260709 13:02:37.746230  9250 consensus_queue.cc:237] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 21.16, Last appended by leader: 16, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:37.739156  9246 raft_consensus.cc:493] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:37.757696  9405 raft_consensus.cc:3055] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 13:02:37.748387  9504 raft_consensus.cc:2749] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:02:37.661633  9426 consensus_queue.cc:237] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 22.14, Last appended by leader: 14, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:37.796015  1646 leader_election.cc:304] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
W20260709 13:02:37.797115  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 106613us
I20260709 13:02:37.788419  1644 leader_election.cc:304] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:02:37.754679  9209 raft_consensus.cc:515] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:37.757941  9518 raft_consensus.cc:515] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:37.806627  9232 raft_consensus.cc:3060] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:02:37.747668  9069 consensus_queue.cc:1059] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 18.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.050s
I20260709 13:02:37.814958  9506 raft_consensus.cc:515] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:37.831916  9209 leader_election.cc:290] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 20 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:37.807448  9486 consensus_queue.cc:237] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.12, Last appended by leader: 12, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:37.846527  9521 raft_consensus.cc:2749] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 13:02:37.848778  9501 leader_election.cc:290] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:37.850461  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:37.822157  9246 raft_consensus.cc:3060] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:02:37.748530  9510 raft_consensus.cc:3060] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:02:37.816061  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 23 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:37.820654  1749 raft_consensus.cc:1275] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 18 index: 13. Preceding OpId from leader: term: 20 index: 15. (index mismatch)
I20260709 13:02:37.821754  1748 raft_consensus.cc:2393] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 21: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:02:37.819471  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 22 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:37.824875  1571 leader_election.cc:304] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:37.855211  1751 raft_consensus.cc:2410] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 for term 22 because replica has last-logged OpId of term: 20 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 14.
I20260709 13:02:37.855576  1571 leader_election.cc:304] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:37.856278  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:37.856994  1751 raft_consensus.cc:2468] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 22.
I20260709 13:02:37.857451  1571 leader_election.cc:304] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:37.826618  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:37.858310  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:37.796620  9489 raft_consensus.cc:493] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:02:37.827354  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:37.816756  9530 raft_consensus.cc:2804] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 13:02:37.799010  9339 raft_consensus.cc:697] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
W20260709 13:02:37.820675  9446 consensus_meta.cc:287] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.861s	user 0.010s	sys 0.006s
W20260709 13:02:37.834124  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 45805.1us
I20260709 13:02:37.835139  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 20 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:37.821913  9405 raft_consensus.cc:740] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:02:37.835734  9506 leader_election.cc:290] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:37.837426  9518 leader_election.cc:290] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:37.839100  2333 raft_consensus.cc:1240] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 19. Current term is 20. Ops: [19.13-19.13]
I20260709 13:02:37.839265  9424 leader_election.cc:290] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 21 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:37.839875  2328 raft_consensus.cc:1240] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 26 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 25. Current term is 26. Ops: []
I20260709 13:02:37.839895  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 23 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:37.943913  2330 raft_consensus.cc:2410] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for term 23 because replica has last-logged OpId of term: 21 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 13.
W20260709 13:02:37.840477  9336 consensus_meta.cc:287] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.624s	user 0.008s	sys 0.004s
I20260709 13:02:37.840286  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 26 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:37.841192  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 18 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:37.841984  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:37.843806  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 21 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:02:37.845825  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 49941.5us
I20260709 13:02:37.845845  9226 raft_consensus.cc:2804] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 13:02:37.846899  1569 leader_election.cc:304] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:02:37.851235  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:37.809321  9514 raft_consensus.cc:2749] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:02:37.828199  1601 raft_consensus.cc:1275] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 21 index: 15. Preceding OpId from leader: term: 22 index: 16. (index mismatch)
I20260709 13:02:37.885907  1751 raft_consensus.cc:2393] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 20: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:02:37.947319  1646 leader_election.cc:304] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:37.895047  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:37.962743  1680 raft_consensus.cc:2468] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 22.
I20260709 13:02:37.915190  1749 raft_consensus.cc:1240] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 18. Current term is 19. Ops: []
I20260709 13:02:37.899384  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:37.898818  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 22 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:37.890450  9450 raft_consensus.cc:2804] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:02:37.963289  1646 leader_election.cc:304] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:02:37.968887  9529 raft_consensus.cc:2804] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Leader election won for term 16
W20260709 13:02:37.969475  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 122746us
I20260709 13:02:37.963186  9435 leader_election.cc:290] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 17 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:37.971716  9435 consensus_queue.cc:1059] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 25.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 13:02:37.972934  9432 raft_consensus.cc:3055] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 25 LEADER]: Stepping down as leader of term 25
I20260709 13:02:37.973338  9432 raft_consensus.cc:740] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:02:37.929893  9517 raft_consensus.cc:515] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:37.974289  9432 consensus_queue.cc:260] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 25.16, Last appended by leader: 16, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:37.975637  9432 raft_consensus.cc:3060] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 25 FOLLOWER]: Advancing to term 26
I20260709 13:02:37.976822  9517 leader_election.cc:290] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 18 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:37.898057  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:37.932288  9437 raft_consensus.cc:493] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:37.866935  1571 leader_election.cc:304] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:37.859273  1602 raft_consensus.cc:2468] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 22.
I20260709 13:02:37.858642  2332 raft_consensus.cc:2393] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 23: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:02:37.887375  9489 raft_consensus.cc:3060] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:02:37.913965  9322 raft_consensus.cc:493] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:37.890304  9412 raft_consensus.cc:2804] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 13:02:37.940759  9527 raft_consensus.cc:493] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:37.960283  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:37.858310  2301 leader_election.cc:304] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:37.928830  9530 raft_consensus.cc:493] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:37.967656  1749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55840 (request call id 3231) took 1038 ms. Trace:
I20260709 13:02:37.970844  9522 raft_consensus.cc:2749] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:02:37.923957  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:38.007496  1749 rpcz_store.cc:276] 0709 13:02:36.929748 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:37.890662 (+960914us) service_pool.cc:224] Handling call
0709 13:02:37.891080 (+   418us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:37.967310 (+ 76230us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:37.967594 (+   284us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:38.011086  1676 raft_consensus.cc:2468] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 22.
W20260709 13:02:37.925328  1719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:38.011885  1749 raft_consensus.cc:1240] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 20. Current term is 21. Ops: [20.14-20.14]
I20260709 13:02:38.012596  1749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43156 (request call id 1568) took 1053 ms. Trace:
I20260709 13:02:37.992216  2329 raft_consensus.cc:2393] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 18: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:02:37.978890  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:37.946513  9226 raft_consensus.cc:493] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:02:37.989271  1717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:37.959443  1600 raft_consensus.cc:2468] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 16.
I20260709 13:02:38.006515  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:38.020655  2332 raft_consensus.cc:2468] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 20.
W20260709 13:02:38.007652  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 149510us
I20260709 13:02:37.965873  9482 raft_consensus.cc:2804] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 13:02:38.006592  9412 raft_consensus.cc:493] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Starting leader election (detected failure of leader af520b67945d443b901bbeca37cc5701)
W20260709 13:02:37.979226  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:37.965552  9506 raft_consensus.cc:2804] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 13:02:38.037876  9506 raft_consensus.cc:493] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:38.038426  9506 raft_consensus.cc:3060] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:02:38.014381  2328 raft_consensus.cc:2393] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 21: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:02:37.946044  9446 raft_consensus.cc:2749] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:02:38.048307  9446 raft_consensus.cc:493] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:38.049024  9446 raft_consensus.cc:515] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:37.968923  9450 raft_consensus.cc:493] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:38.050784  9450 raft_consensus.cc:3060] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:02:38.052947  1745 raft_consensus.cc:2468] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 19.
I20260709 13:02:38.032070  2332 raft_consensus.cc:1275] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 19 index: 13. Preceding OpId from leader: term: 20 index: 14. (index mismatch)
I20260709 13:02:38.025588  1718 leader_election.cc:304] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:38.004084  9474 consensus_queue.cc:1048] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.035s
I20260709 13:02:38.012902  1749 rpcz_store.cc:276] 0709 13:02:36.959640 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:38.011245 (+1051605us) service_pool.cc:224] Handling call
0709 13:02:38.011632 (+   387us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:38.012311 (+   679us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:38.012537 (+   226us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:38.035286  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:37.992144  9529 raft_consensus.cc:697] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:02:37.995685  8934 raft_consensus.cc:515] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:38.019228  8994 raft_consensus.cc:3055] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 13:02:38.020769  9226 raft_consensus.cc:3060] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Advancing to term 24
I20260709 13:02:38.055280  9412 raft_consensus.cc:3060] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:02:38.056273  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:38.080502  1603 raft_consensus.cc:2468] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 16.
I20260709 13:02:38.016202  9517 raft_consensus.cc:2804] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 13:02:38.009351  9530 raft_consensus.cc:3060] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:02:38.028698  2333 raft_consensus.cc:1240] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 19. Current term is 20. Ops: []
W20260709 13:02:38.033306  9335 consensus_meta.cc:287] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.543s	user 0.015s	sys 0.000s
I20260709 13:02:38.004266  9514 consensus_queue.cc:1059] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 0.0, Next index: 18, Last known committed idx: 15, Time since last communication: 0.029s
I20260709 13:02:38.092612  2334 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39726 (request call id 1712) took 1276 ms. Trace:
I20260709 13:02:38.085034  9517 raft_consensus.cc:493] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:38.100137  9517 raft_consensus.cc:3060] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:02:38.015097  9527 raft_consensus.cc:515] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:37.964044  9426 consensus_queue.cc:1059] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 18.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.072s
I20260709 13:02:38.060899  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:38.003692  9437 raft_consensus.cc:515] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:38.066190  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
W20260709 13:02:38.035157  9409 consensus_meta.cc:287] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.803s	user 0.009s	sys 0.005s
I20260709 13:02:38.058413  9424 raft_consensus.cc:2804] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:02:38.068608  9529 consensus_queue.cc:237] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:38.069335  9453 leader_election.cc:290] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:38.071233  9518 consensus_queue.cc:1059] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 18.13, Next index: 14, Last known committed idx: 12, Time since last communication: 0.031s
W20260709 13:02:38.069542  9525 log.cc:927] Time spent T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.149s	user 0.000s	sys 0.001s
I20260709 13:02:38.007788  9322 raft_consensus.cc:515] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:38.070776  9537 log.cc:927] Time spent T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.054s	user 0.001s	sys 0.000s
I20260709 13:02:38.050812  1752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55848 (request call id 3426) took 1160 ms. Trace:
W20260709 13:02:38.062618  9492 consensus_meta.cc:287] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.506s	user 0.013s	sys 0.000s
W20260709 13:02:38.086227  1718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:38.175294  1718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:38.034905  9054 consensus_queue.cc:1048] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 11, Time since last communication: 0.045s
I20260709 13:02:38.093118  2334 rpcz_store.cc:276] 0709 13:02:36.816570 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:36.816758 (+   188us) service_pool.cc:224] Handling call
0709 13:02:36.817116 (+   358us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:37.138396 (+321280us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:38.092295 (+953899us) spinlock_profiling.cc:243] Waited 321 ms on lock 0x7b54006beed4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:02:38.092528 (+   233us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":737748224}
I20260709 13:02:38.071548  8994 raft_consensus.cc:740] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:02:38.177794  8994 consensus_queue.cc:260] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 18.15, Last appended by leader: 15, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:38.105001  1754 raft_consensus.cc:1275] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20260709 13:02:38.111629  1750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43156 (request call id 1565) took 1183 ms. Trace:
I20260709 13:02:38.180634  1750 rpcz_store.cc:276] 0709 13:02:36.927759 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:37.534571 (+606812us) service_pool.cc:224] Handling call
0709 13:02:37.535025 (+   454us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:37.648476 (+113451us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:38.111316 (+462840us) spinlock_profiling.cc:243] Waited 113 ms on lock 0x7b5400385254. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:02:38.111545 (+   229us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":259343744}
W20260709 13:02:38.129860  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 159280us
W20260709 13:02:38.059540  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 88158.3us
I20260709 13:02:38.182605  9482 raft_consensus.cc:493] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:38.183135  9482 raft_consensus.cc:3060] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:02:38.183184  1646 leader_election.cc:304] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:38.140118  1746 raft_consensus.cc:1240] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 21. Current term is 22. Ops: []
I20260709 13:02:38.184419  8893 consensus_queue.cc:1059] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
W20260709 13:02:38.056223  9295 consensus_meta.cc:287] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.646s	user 0.011s	sys 0.002s
I20260709 13:02:38.147724  1673 raft_consensus.cc:2410] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 for term 22 because replica has last-logged OpId of term: 20 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 14.
I20260709 13:02:38.185393  8893 raft_consensus.cc:3055] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 13:02:38.185905  8893 raft_consensus.cc:740] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:02:38.185427  9295 raft_consensus.cc:515] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:38.141196  2298 leader_election.cc:304] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:02:38.186841  8893 consensus_queue.cc:260] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.16, Last appended by leader: 16, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:38.187673  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 115426us
I20260709 13:02:38.188032  8893 raft_consensus.cc:3060] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:02:38.188467  2298 leader_election.cc:304] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:02:38.188652  9295 leader_election.cc:290] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 23 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:38.189777  2298 leader_election.cc:304] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
W20260709 13:02:38.140146  9534 log.cc:927] Time spent T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.136s	user 0.000s	sys 0.001s
I20260709 13:02:38.166692  1752 rpcz_store.cc:276] 0709 13:02:36.890683 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:37.093988 (+203305us) service_pool.cc:224] Handling call
0709 13:02:38.050529 (+956541us) spinlock_profiling.cc:243] Waited 584 ms on lock 0x7b5400116cd4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:02:38.050741 (+   212us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1342429568}
I20260709 13:02:38.176087  9054 consensus_queue.cc:1243] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer af520b67945d443b901bbeca37cc5701 log is divergent from this leader: its last log entry 18.13 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 11
I20260709 13:02:38.190485  9533 raft_consensus.cc:2749] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 13:02:38.190639  9282 raft_consensus.cc:2749] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 13:02:38.148211  2328 consensus_queue.cc:237] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 22.14, Last appended by leader: 13, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:38.193504  1745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55848 (request call id 3429) took 1300 ms. Trace:
I20260709 13:02:38.193682  1751 raft_consensus.cc:1240] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 18. Current term is 19. Ops: []
I20260709 13:02:38.090533  9335 raft_consensus.cc:515] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:38.193778  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 327076us
I20260709 13:02:38.128257  9424 raft_consensus.cc:493] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:38.065686  9339 consensus_queue.cc:237] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.11, Last appended by leader: 11, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } }
I20260709 13:02:38.198238  1602 raft_consensus.cc:2468] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 20.
I20260709 13:02:38.200111  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:38.198832  1646 leader_election.cc:304] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:38.204255  2298 leader_election.cc:304] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:02:38.203174  1675 raft_consensus.cc:2468] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 17.
I20260709 13:02:38.237648  1646 leader_election.cc:304] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 13:02:38.204133  1745 rpcz_store.cc:276] 0709 13:02:36.892808 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:37.135859 (+243051us) service_pool.cc:224] Handling call
0709 13:02:38.193233 (+1057374us) spinlock_profiling.cc:243] Waited 916 ms on lock 0x7b540031f454. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:02:38.193392 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2107838848}
I20260709 13:02:38.139725  9409 raft_consensus.cc:515] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:02:38.272163  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:38.273458  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:38.282176  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:38.282474  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:38.244102  2332 raft_consensus.cc:3060] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:02:38.208062  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 22 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:38.251034  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 18 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:38.202617  9220 raft_consensus.cc:2749] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20260709 13:02:38.345427  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:38.347702  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 110236us
I20260709 13:02:38.226536  2298 leader_election.cc:304] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:02:38.361550  1605 raft_consensus.cc:2468] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 24.
I20260709 13:02:38.205601  9524 raft_consensus.cc:2804] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Leader election won for term 17
W20260709 13:02:38.222465  9539 log.cc:927] Time spent T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.132s	user 0.000s	sys 0.001s
W20260709 13:02:38.380929  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 154592us
I20260709 13:02:38.382881  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:38.393960  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:38.393116  2298 leader_election.cc:304] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:02:38.245589  9518 raft_consensus.cc:2804] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 13:02:38.197451  9492 raft_consensus.cc:515] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } }
I20260709 13:02:38.239632  9446 leader_election.cc:290] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:38.219136  9424 raft_consensus.cc:3060] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:02:38.395643  9542 raft_consensus.cc:2804] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 13:02:38.260527  9250 raft_consensus.cc:2804] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 25 FOLLOWER]: Leader pre-election won for term 26
W20260709 13:02:38.361788  9500 log.cc:927] Time spent T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.156s	user 0.001s	sys 0.000s
W20260709 13:02:38.416440  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 233649us
I20260709 13:02:38.442741  1569 leader_election.cc:304] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:02:38.370293  9535 raft_consensus.cc:493] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:38.447068  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:38.396428  9510 raft_consensus.cc:2749] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:02:38.244024  9295 consensus_queue.cc:1059] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 16.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.207s
I20260709 13:02:38.405028  9533 raft_consensus.cc:2804] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:02:38.415953  9521 raft_consensus.cc:493] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:38.407361  9518 raft_consensus.cc:493] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:38.416440  9250 raft_consensus.cc:493] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:38.417889  9220 raft_consensus.cc:493] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:38.415548  9542 raft_consensus.cc:493] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:38.418619  9282 raft_consensus.cc:2749] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 13:02:38.474967  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:38.487108  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
W20260709 13:02:38.488907  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 258220us
I20260709 13:02:38.415799  9491 consensus_queue.cc:1059] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 21.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.081s
I20260709 13:02:38.468053  9533 raft_consensus.cc:493] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:02:38.504997  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 460435us
I20260709 13:02:38.485114  9518 raft_consensus.cc:3060] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Advancing to term 24
W20260709 13:02:38.509974  2299 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:38.510767  2336 raft_consensus.cc:2468] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 19.
I20260709 13:02:38.486872  9214 leader_election.cc:290] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 21 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:38.491950  9250 raft_consensus.cc:3060] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 25 FOLLOWER]: Advancing to term 26
I20260709 13:02:38.491114  9542 raft_consensus.cc:3060] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:02:38.458364  9535 raft_consensus.cc:515] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:38.514609  9533 raft_consensus.cc:3060] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Advancing to term 18
W20260709 13:02:38.550926  1643 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.993s	user 0.018s	sys 0.050s
W20260709 13:02:38.551311  1643 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.993s	user 0.018s	sys 0.050s
W20260709 13:02:38.551599  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 994082us
W20260709 13:02:38.555060  9545 log.cc:927] Time spent T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.101s	user 0.000s	sys 0.000s
I20260709 13:02:38.556818  9527 leader_election.cc:290] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:38.530087  9483 consensus_queue.cc:260] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.14, Last appended by leader: 14, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:38.532091  9535 leader_election.cc:290] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:38.515779  9513 raft_consensus.cc:3055] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 LEADER]: Stepping down as leader of term 16
I20260709 13:02:38.534217  1747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55848 (request call id 3432) took 1633 ms. Trace:
I20260709 13:02:38.538687  9547 raft_consensus.cc:493] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:38.485242  9521 raft_consensus.cc:515] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:38.495568  9220 raft_consensus.cc:515] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:38.540512  9335 leader_election.cc:290] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 17 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:38.524308  9549 raft_consensus.cc:2804] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:02:38.547739  9492 leader_election.cc:290] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 20 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:38.547885  9253 raft_consensus.cc:2764] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Leader pre-election decision vote started in defunct term 18: won
I20260709 13:02:38.549907  9409 leader_election.cc:290] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 17 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:38.518374  8934 leader_election.cc:290] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 18 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:38.591223  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:38.575479  2298 leader_election.cc:304] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:02:38.598621  1747 rpcz_store.cc:276] 0709 13:02:36.900801 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:37.242968 (+342167us) service_pool.cc:224] Handling call
0709 13:02:37.256554 (+ 13586us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:37.257040 (+   486us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:37.257085 (+    45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:38.053742 (+796657us) spinlock_profiling.cc:243] Waited 13.3 ms on lock 0x7b18004aa0a0. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f42630432a6 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4264a5ba11 00007f4264a6b372 00007f426619da01 00007f426616243a
0709 13:02:38.180905 (+127163us) spinlock_profiling.cc:243] Waited 6.55 ms on lock 0x7b2c000bc788. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4264a5ba11 00007f4264a6b372 00007f426619da01 00007f426616243a 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2
0709 13:02:38.181445 (+   540us) log.cc:844] Serialized 26 byte log entry
0709 13:02:38.457653 (+276208us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:02:38.457727 (+    74us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:38.457842 (+   115us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:38.533950 (+ 76108us) raft_consensus.cc:1685] finished
0709 13:02:38.533992 (+    42us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:38.534143 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":6256,"raft.run_cpu_time_us":176,"raft.run_wall_time_us":17952,"spinlock_wait_cycles":45636736,"thread_start_us":270181,"threads_started":1,"wal-append.queue_time_us":339494}
I20260709 13:02:38.562340  1673 raft_consensus.cc:1240] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 LEADER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 18. Current term is 19. Ops: [18.13-18.13]
W20260709 13:02:38.630769  1718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:38.564728  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 23 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:38.563596  1674 raft_consensus.cc:1240] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 22. Current term is 23. Ops: [22.11-22.11]
I20260709 13:02:38.564299  1676 raft_consensus.cc:1240] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 LEADER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 18. Current term is 19. Ops: []
W20260709 13:02:38.646533  9432 consensus_meta.cc:287] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.657s	user 0.016s	sys 0.006s
I20260709 13:02:38.615540  9336 raft_consensus.cc:493] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
W20260709 13:02:38.687125  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 111853us
W20260709 13:02:38.660549  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:38.621896  9214 raft_consensus.cc:493] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:38.557397  9437 leader_election.cc:290] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:38.616106  9513 raft_consensus.cc:740] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:02:38.724684  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 23 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:38.703639  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:38.558323  9491 raft_consensus.cc:3055] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 LEADER]: Stepping down as leader of term 21
I20260709 13:02:38.634835  9549 raft_consensus.cc:493] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:02:38.711709  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:38.635182  9453 raft_consensus.cc:493] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:38.732605  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:38.733505  1679 raft_consensus.cc:2393] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 23: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:02:38.735735  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:02:38.701524  9246 consensus_meta.cc:287] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.816s	user 0.018s	sys 0.011s
I20260709 13:02:38.680203  9446 raft_consensus.cc:493] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
W20260709 13:02:38.697158  9450 consensus_meta.cc:287] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.646s	user 0.012s	sys 0.003s
I20260709 13:02:38.701254  9335 raft_consensus.cc:2804] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 13:02:38.748349  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 22 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:02:38.743695  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:38.749856  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 26 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:38.757788  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:38.758606  1673 raft_consensus.cc:3060] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Advancing to term 24
W20260709 13:02:38.767097  9482 consensus_meta.cc:287] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.583s	user 0.011s	sys 0.002s
I20260709 13:02:38.767567  9482 raft_consensus.cc:515] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:38.770021  9482 leader_election.cc:290] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 23 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:38.770843  1602 raft_consensus.cc:2393] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 23: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:02:38.751397  9246 raft_consensus.cc:515] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:38.735139  9214 raft_consensus.cc:515] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:38.776011  9220 leader_election.cc:290] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:38.777261  9246 leader_election.cc:290] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 23 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:38.778756  9335 raft_consensus.cc:697] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 20 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
W20260709 13:02:38.751530  9530 consensus_meta.cc:287] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.650s	user 0.012s	sys 0.002s
W20260709 13:02:38.780303  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 337856us
I20260709 13:02:38.733187  9491 raft_consensus.cc:740] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:02:38.781205  1569 leader_election.cc:304] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:02:38.781497  9491 consensus_queue.cc:260] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 21.16, Last appended by leader: 16, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:38.782930  9491 raft_consensus.cc:3060] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:02:38.734751  9432 consensus_queue.cc:1048] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.044s
I20260709 13:02:38.733403  9549 raft_consensus.cc:3060] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:02:38.756994  9450 raft_consensus.cc:515] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:38.779903  9335 consensus_queue.cc:237] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.14, Last appended by leader: 14, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:38.778251  9559 raft_consensus.cc:493] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:38.749037  9453 raft_consensus.cc:515] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:38.779978  9530 raft_consensus.cc:515] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:38.775097  9554 log.cc:927] Time spent T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.151s	user 0.000s	sys 0.000s
I20260709 13:02:38.622394  9547 raft_consensus.cc:515] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:38.792970  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:38.789863  1603 raft_consensus.cc:2393] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 21: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:02:38.799724  2332 raft_consensus.cc:2468] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 23.
I20260709 13:02:38.787268  1679 raft_consensus.cc:1217] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 19 LEADER]: Deduplicated request from leader. Original: 12.8->[14.9-18.11]   Dedup: 14.9->[16.10-18.11]
I20260709 13:02:38.809181  1679 raft_consensus.cc:1240] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 19 LEADER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 18. Current term is 19. Ops: [14.9-18.11]
I20260709 13:02:38.778273  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:38.792122  1569 leader_election.cc:304] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:02:38.757153  9446 raft_consensus.cc:515] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:38.777765  9214 leader_election.cc:290] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:38.814837  1676 raft_consensus.cc:2393] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 21: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:02:38.728361  9524 raft_consensus.cc:697] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
W20260709 13:02:38.800243  9506 consensus_meta.cc:287] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.761s	user 0.015s	sys 0.001s
I20260709 13:02:38.840760  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:38.841450  2329 raft_consensus.cc:2468] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 21.
W20260709 13:02:38.792989  8893 consensus_meta.cc:287] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.604s	user 0.014s	sys 0.000s
W20260709 13:02:38.801255  9517 consensus_meta.cc:287] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.701s	user 0.011s	sys 0.004s
I20260709 13:02:38.835445  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:38.804673  9547 leader_election.cc:290] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:02:38.786391  9412 consensus_meta.cc:287] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.697s	user 0.008s	sys 0.000s
I20260709 13:02:38.786550  9253 raft_consensus.cc:3060] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:02:38.837397  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:38.793953  9453 leader_election.cc:290] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:38.802765  9521 leader_election.cc:290] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:38.796389  9530 leader_election.cc:290] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 23 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:38.789356  9450 leader_election.cc:290] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 21 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:38.865598  1644 leader_election.cc:304] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
W20260709 13:02:38.848948  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 165762us
I20260709 13:02:38.861083  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:02:38.867393  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 45182.1us
I20260709 13:02:38.867574  1676 raft_consensus.cc:2468] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 21.
I20260709 13:02:38.868239  1569 leader_election.cc:304] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:02:38.859311  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:38.790253  9559 raft_consensus.cc:515] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:38.871979  1679 raft_consensus.cc:2393] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 21: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:02:38.748945  9336 raft_consensus.cc:515] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:02:38.850890  1571 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:38.874670  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:02:38.875569  1571 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:38.876078  1571 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:38.876592  1571 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:38.875901  1679 raft_consensus.cc:2468] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 21.
W20260709 13:02:38.915343  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:38.847652  9506 raft_consensus.cc:515] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:38.877940  9453 consensus_queue.cc:1059] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 19.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.113s
I20260709 13:02:38.853024  9517 raft_consensus.cc:515] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:38.934193  9226 consensus_meta.cc:287] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.859s	user 0.012s	sys 0.006s
I20260709 13:02:38.861481  9412 raft_consensus.cc:515] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:39.043664  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:39.074170  1679 raft_consensus.cc:2468] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 24.
I20260709 13:02:38.798830  9513 consensus_queue.cc:260] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 16.13, Last appended by leader: 13, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:39.044914  1569 leader_election.cc:304] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:02:38.876945  9529 raft_consensus.cc:2749] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:02:39.050889  1673 raft_consensus.cc:2468] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 24.
I20260709 13:02:38.873777  9482 raft_consensus.cc:493] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:39.058549  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:02:39.121191  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 243663us
I20260709 13:02:38.932127  9181 raft_consensus.cc:2749] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
W20260709 13:02:39.066388  2298 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:38.933372  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 52358.2us
I20260709 13:02:38.878254  9542 raft_consensus.cc:515] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:39.138965  9542 leader_election.cc:290] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 23 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:02:39.123270  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:39.123518  1676 raft_consensus.cc:2393] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 20: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:02:39.140833  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:39.141016  9453 raft_consensus.cc:3055] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 13:02:39.141500  9453 raft_consensus.cc:740] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:02:39.142474  9453 consensus_queue.cc:260] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 19.16, Last appended by leader: 16, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:39.143635  9453 raft_consensus.cc:3060] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Advancing to term 20
W20260709 13:02:39.144698  9518 consensus_meta.cc:287] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.629s	user 0.010s	sys 0.002s
I20260709 13:02:39.145164  9518 raft_consensus.cc:515] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:39.146505  9506 leader_election.cc:290] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 22 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:39.123730  9226 raft_consensus.cc:515] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:39.147547  1680 raft_consensus.cc:2393] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 22: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:02:39.140843  9549 raft_consensus.cc:515] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:38.984371  9524 consensus_queue.cc:237] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.10, Last appended by leader: 10, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:39.149166  9226 leader_election.cc:290] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 24 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:39.149657  9549 leader_election.cc:290] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 21 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:39.149991  1672 raft_consensus.cc:2393] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 24: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:02:39.151489  9559 leader_election.cc:290] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:02:39.146574  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:39.153411  9446 leader_election.cc:290] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:02:39.160827  9508 log.cc:927] Time spent T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 1.496s	user 0.003s	sys 0.008s
W20260709 13:02:39.161177  9250 consensus_meta.cc:287] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.635s	user 0.011s	sys 0.005s
I20260709 13:02:39.161582  9250 raft_consensus.cc:515] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:39.163157  1753 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43156 (request call id 1563) took 2260 ms. Trace:
W20260709 13:02:39.163439  9533 consensus_meta.cc:287] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.613s	user 0.008s	sys 0.004s
I20260709 13:02:39.163887  9533 raft_consensus.cc:515] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:39.163579  1753 rpcz_store.cc:276] 0709 13:02:36.903047 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:37.316785 (+413738us) service_pool.cc:224] Handling call
0709 13:02:38.092647 (+775862us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:38.093047 (+   400us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:38.093092 (+    45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:38.126623 (+ 33531us) log.cc:844] Serialized 26 byte log entry
0709 13:02:38.224405 (+ 97782us) raft_consensus.cc:1645] Marking committed up to 15
0709 13:02:38.224475 (+    70us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:38.224576 (+   101us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:38.557380 (+332804us) raft_consensus.cc:1685] finished
0709 13:02:38.557407 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:39.162860 (+605453us) spinlock_profiling.cc:243] Waited 684 ms on lock 0x7b54001c6e50. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661618ad 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:02:39.163079 (+   219us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":7012,"raft.run_cpu_time_us":103,"raft.run_wall_time_us":103,"spinlock_wait_cycles":1574110464,"thread_start_us":97287,"threads_started":1,"wal-append.queue_time_us":362306}
I20260709 13:02:39.063540  9521 raft_consensus.cc:2804] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Leader election won for term 23
I20260709 13:02:39.165951  9521 raft_consensus.cc:697] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:02:39.166945  9521 consensus_queue.cc:237] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 21.14, Last appended by leader: 14, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:39.171018  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:39.124236  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:39.171860  1679 raft_consensus.cc:3060] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:02:39.133955  1644 leader_election.cc:304] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:02:39.134953  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:39.135541  9482 raft_consensus.cc:515] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:39.047252  9486 raft_consensus.cc:3055] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 25 LEADER]: Stepping down as leader of term 25
W20260709 13:02:39.180742  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 117081us
W20260709 13:02:39.180835  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 47023.4us
I20260709 13:02:39.181681  1674 raft_consensus.cc:2368] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 25. Current term is 26.
W20260709 13:02:39.088114  9555 log.cc:927] Time spent T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.227s	user 0.000s	sys 0.006s
W20260709 13:02:39.094065  9424 consensus_meta.cc:287] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.645s	user 0.019s	sys 0.000s
W20260709 13:02:39.194085  2336 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39738 (request call id 1355) took 1353 ms (client timeout 1741 ms). Trace:
W20260709 13:02:39.200531  2336 rpcz_store.cc:269] 0709 13:02:37.841039 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:37.841698 (+   659us) service_pool.cc:224] Handling call
0709 13:02:39.193856 (+1352158us) spinlock_profiling.cc:243] Waited 541 ms on lock 0x7b54007602d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:02:39.194001 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1245077376}
I20260709 13:02:39.203079  1673 raft_consensus.cc:1275] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 22 index: 12. Preceding OpId from leader: term: 23 index: 13. (index mismatch)
I20260709 13:02:39.143859  9336 leader_election.cc:290] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:39.186264  9486 raft_consensus.cc:740] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:02:39.192364  9506 raft_consensus.cc:2804] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 13:02:39.226397  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:39.191977  9561 consensus_queue.cc:1059] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 18.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.052s
I20260709 13:02:39.220604  9506 raft_consensus.cc:493] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:39.240785  1673 raft_consensus.cc:2468] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 21.
W20260709 13:02:39.243213  1643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:39.244786  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:39.247578  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:39.249555  2337 raft_consensus.cc:2393] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 24: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:02:39.232039  9506 raft_consensus.cc:3060] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:02:39.231719  9453 raft_consensus.cc:493] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:39.245211  9489 consensus_meta.cc:287] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 1.236s	user 0.009s	sys 0.009s
I20260709 13:02:39.206023  9424 raft_consensus.cc:515] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:39.248452  9461 negotiation.cc:336] Failed RPC negotiation. Trace:
0709 13:02:35.929593 (+     0us) reactor.cc:730] Submitting negotiation task for client connection to 127.0.107.193:46445 (local address 127.0.0.1:54246)
0709 13:02:36.091358 (+161765us) negotiation.cc:107] Waiting for socket to connect
0709 13:02:36.091462 (+   104us) client_negotiation.cc:175] Beginning negotiation
0709 13:02:36.138400 (+ 46938us) client_negotiation.cc:262] Sending NEGOTIATE NegotiatePB request
0709 13:02:39.151030 (+3012630us) negotiation.cc:326] Negotiation complete: Timed out: Client connection negotiation failed: client connection to 127.0.107.193:46445: received 0 of 4 requested bytes
Metrics: {"client-negotiator.queue_time_us":113634,"thread_start_us":71307,"threads_started":1}
I20260709 13:02:39.244881  9486 consensus_queue.cc:260] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 25.18, Last appended by leader: 18, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:39.271482  9486 raft_consensus.cc:3060] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 25 FOLLOWER]: Advancing to term 26
I20260709 13:02:39.272069  2329 raft_consensus.cc:1275] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 13 index: 12. Preceding OpId from leader: term: 17 index: 15. (index mismatch)
I20260709 13:02:39.272675  9405 consensus_queue.cc:260] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.13, Last appended by leader: 13, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:39.274031  9405 raft_consensus.cc:3060] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:02:39.277513  9412 leader_election.cc:290] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 23 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:39.278524  9489 raft_consensus.cc:515] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:39.281004  9489 leader_election.cc:290] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 21 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:39.283737  9482 leader_election.cc:290] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:39.287946  1679 raft_consensus.cc:2468] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 17.
I20260709 13:02:39.288771  1677 raft_consensus.cc:2393] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 21: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:02:39.294556  1678 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57092 (request call id 1715) took 2211 ms. Trace:
W20260709 13:02:39.252622  1643 connection.cc:256] RPC call timeout handler was delayed by -1.868251430999976s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1841s for a response
I20260709 13:02:39.298691  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:39.301309  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:39.295075  1678 rpcz_store.cc:276] 0709 13:02:37.083092 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:37.083262 (+   170us) service_pool.cc:224] Handling call
0709 13:02:37.083643 (+   381us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:37.138138 (+ 54495us) raft_consensus.cc:1534] Early marking committed up to index 16
0709 13:02:37.167891 (+ 29753us) log.cc:844] Serialized 14 byte log entry
0709 13:02:37.553191 (+385300us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:37.553251 (+    60us) raft_consensus.cc:1645] Marking committed up to 16
0709 13:02:37.553308 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:39.294177 (+1740869us) spinlock_profiling.cc:243] Waited 54.2 ms on lock 0x7b54000596d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:02:39.294258 (+    81us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:39.294468 (+   210us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":21937,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":2981,"raft.queue_time_us":85552,"raft.run_cpu_time_us":144,"raft.run_wall_time_us":5323,"spinlock_wait_cycles":124737408,"thread_start_us":384926,"threads_started":1,"wal-append.queue_time_us":494850,"wal-append.run_cpu_time_us":9253,"wal-append.run_wall_time_us":1078141}
I20260709 13:02:39.317382  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:39.321825  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:39.322675  2336 raft_consensus.cc:3060] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:02:39.327452  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:39.287652  9250 leader_election.cc:290] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 26 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:39.268980  9533 leader_election.cc:290] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 18 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:39.292977  9253 raft_consensus.cc:2749] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
W20260709 13:02:39.331928   431 auto_leader_rebalancer.cc:438] leader transfer for tablet f6028a708cb344fbb08ab8056921d9e7 from 83595ad0fd5940c0a2bfc7c61acd969a to 816fadcea8fc4893882f5f9b9f22a03d failed: Timed out: Client connection negotiation failed: client connection to 127.0.107.193:46445: received 0 of 4 requested bytes
I20260709 13:02:39.269270  9453 raft_consensus.cc:515] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:39.340086   431 auto_leader_rebalancer.cc:452] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 13:02:39.330953  9517 leader_election.cc:290] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 23 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:39.346071  1671 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57092 (request call id 1722) took 1449 ms. Trace:
I20260709 13:02:39.348311  2336 raft_consensus.cc:2468] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 21.
I20260709 13:02:39.347815  1604 raft_consensus.cc:2468] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 21.
I20260709 13:02:39.318236  9489 consensus_queue.cc:1048] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 12, Time since last communication: 0.025s
W20260709 13:02:39.344820  9491 consensus_meta.cc:287] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.560s	user 0.007s	sys 0.007s
I20260709 13:02:39.356727  1644 leader_election.cc:304] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:02:39.360492  1676 raft_consensus.cc:2468] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 22.
I20260709 13:02:39.348028  1671 rpcz_store.cc:276] 0709 13:02:37.896814 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:37.898268 (+  1454us) service_pool.cc:224] Handling call
0709 13:02:37.914413 (+ 16145us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:37.914729 (+   316us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 13:02:37.927423 (+ 12694us) log.cc:844] Serialized 14 byte log entry
0709 13:02:38.553499 (+626076us) spinlock_profiling.cc:243] Waited 15.4 ms on lock 0x7b18003492a0. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f42630753f3 00007f42630777c1 00007f4263077779 00007f4263077709 00007f42630776d1 00007f42630769cd 000000000038f354 000000000038f199 00007f426305883a
0709 13:02:39.167139 (+613640us) spinlock_profiling.cc:243] Waited 1.98 ms on lock 0x7b18003492a0. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f4263057e07 00007f426beb2ab9 00007f426307469b 00007f426307293f 00007f4263072455 00007f4269b5a3c7 00007f426616c9a7 00007f426617c0bf 00007f426617c038
0709 13:02:39.341431 (+174292us) log.cc:844] Serialized 14 byte log entry
0709 13:02:39.345704 (+  4273us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:39.345760 (+    56us) raft_consensus.cc:1645] Marking committed up to 14
0709 13:02:39.345811 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:39.345887 (+    76us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:39.346007 (+   120us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3948,"prepare.queue_time_us":33067,"prepare.run_cpu_time_us":98,"prepare.run_wall_time_us":92,"raft.queue_time_us":4392,"raft.run_cpu_time_us":130,"raft.run_wall_time_us":7394,"spinlock_wait_cycles":40085504,"thread_start_us":437414,"threads_started":2,"wal-append.queue_time_us":438633}
W20260709 13:02:39.371541  1748 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55848 (request call id 3443) took 1840 ms (client timeout 2210 ms). Trace:
W20260709 13:02:39.372027  1748 rpcz_store.cc:269] 0709 13:02:37.531436 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:37.688917 (+157481us) service_pool.cc:224] Handling call
0709 13:02:39.371290 (+1682373us) spinlock_profiling.cc:243] Waited 132 ms on lock 0x7b54001c7354. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:02:39.371464 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":302853888}
W20260709 13:02:39.374950  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:39.381690  9424 leader_election.cc:290] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 21 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:39.382230  1603 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40154 (request call id 3054) took 1073 ms. Trace:
I20260709 13:02:39.382947  1602 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40154 (request call id 3051) took 1074 ms. Trace:
I20260709 13:02:39.383375  1602 rpcz_store.cc:276] 0709 13:02:38.308288 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:38.402920 (+ 94632us) service_pool.cc:224] Handling call
0709 13:02:39.382724 (+979804us) spinlock_profiling.cc:243] Waited 283 ms on lock 0x7b5400071054. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:02:39.382870 (+   146us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":650784512}
I20260709 13:02:39.384749  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:39.385538  1602 raft_consensus.cc:2468] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 19.
I20260709 13:02:39.386014  1602 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40154 (request call id 3055) took 1076 ms. Trace:
I20260709 13:02:39.382756  1603 rpcz_store.cc:276] 0709 13:02:38.309289 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:38.452827 (+143538us) service_pool.cc:224] Handling call
0709 13:02:39.381963 (+929136us) spinlock_profiling.cc:243] Waited 303 ms on lock 0x7b54002797d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:02:39.382156 (+   193us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":697305472}
W20260709 13:02:39.386861  1645 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:39.387665  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:39.389046  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:39.389626  1603 raft_consensus.cc:3060] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Advancing to term 23
I20260709 13:02:39.392316  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:39.392977  1605 raft_consensus.cc:2468] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 20.
I20260709 13:02:39.393836  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:39.394951  1605 raft_consensus.cc:2410] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 for term 21 because replica has last-logged OpId of term: 19 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 12.
I20260709 13:02:39.386303  1602 rpcz_store.cc:276] 0709 13:02:38.309580 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:39.384423 (+1074843us) service_pool.cc:224] Handling call
0709 13:02:39.385958 (+  1535us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:39.395694  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:39.396335  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:39.396919  1602 raft_consensus.cc:2468] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 20.
I20260709 13:02:39.397012  1605 raft_consensus.cc:3055] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 LEADER]: Stepping down as leader of term 22
I20260709 13:02:39.397351  1605 raft_consensus.cc:740] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:02:39.397403  1602 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40154 (request call id 3050) took 1111 ms. Trace:
I20260709 13:02:39.397794  1602 rpcz_store.cc:276] 0709 13:02:38.307812 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:39.396079 (+1088267us) service_pool.cc:224] Handling call
0709 13:02:39.397334 (+  1255us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:39.398260  1605 consensus_queue.cc:260] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 22.12, Last appended by leader: 12, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:39.399070  1602 raft_consensus.cc:1275] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 19 index: 13. Preceding OpId from leader: term: 20 index: 14. (index mismatch)
I20260709 13:02:39.399307  1605 raft_consensus.cc:3060] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Advancing to term 23
W20260709 13:02:39.399717  1717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:39.399909  1599 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40154 (request call id 3053) took 1091 ms. Trace:
I20260709 13:02:39.400297  1599 rpcz_store.cc:276] 0709 13:02:38.308976 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:39.398950 (+1089974us) service_pool.cc:224] Handling call
0709 13:02:39.399286 (+   336us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:39.399501 (+   215us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:39.399526 (+    25us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:39.399571 (+    45us) raft_consensus.cc:1645] Marking committed up to 16
0709 13:02:39.399618 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:39.399674 (+    56us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:39.399838 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:39.399731  1602 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40154 (request call id 3048) took 1114 ms. Trace:
I20260709 13:02:39.404661  9453 leader_election.cc:290] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:39.404659  1602 rpcz_store.cc:276] 0709 13:02:38.284936 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:39.398415 (+1113479us) service_pool.cc:224] Handling call
0709 13:02:39.398764 (+   349us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:39.399485 (+   721us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:39.399678 (+   193us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:39.407181  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:39.407509  2299 leader_election.cc:304] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 13:02:39.384714  1719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:39.409552  1605 raft_consensus.cc:1275] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 23 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 22 index: 12. Preceding OpId from leader: term: 23 index: 13. (index mismatch)
I20260709 13:02:39.402215  1599 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40154 (request call id 3056) took 1092 ms. Trace:
I20260709 13:02:39.412379  1599 rpcz_store.cc:276] 0709 13:02:38.309888 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:39.401457 (+1091569us) service_pool.cc:224] Handling call
0709 13:02:39.401731 (+   274us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:39.401916 (+   185us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:39.401944 (+    28us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:39.401984 (+    40us) raft_consensus.cc:1645] Marking committed up to 13
0709 13:02:39.402016 (+    32us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:39.402067 (+    51us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:39.402162 (+    95us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:39.413708  1605 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40154 (request call id 3047) took 1129 ms. Trace:
I20260709 13:02:39.414160  1605 rpcz_store.cc:276] 0709 13:02:38.284612 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:39.396590 (+1111978us) service_pool.cc:224] Handling call
0709 13:02:39.396885 (+   295us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:39.413465 (+ 16580us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:39.413653 (+   188us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:39.423206  1748 raft_consensus.cc:1217] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Deduplicated request from leader. Original: 12.11->[15.12-20.15]   Dedup: 15.12->[17.13-20.15]
I20260709 13:02:39.423727  1748 pending_rounds.cc:85] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701: Aborting all ops after (but not including) 12
I20260709 13:02:39.423962  1748 pending_rounds.cc:107] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701: Aborting uncommitted NO_OP operation due to leader change: 18.13
I20260709 13:02:39.424188  1748 raft_consensus.cc:2889] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:02:39.425073  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:39.426553  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 17 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:39.427482  1750 raft_consensus.cc:2393] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 17: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:02:39.444070  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:39.359433  9521 raft_consensus.cc:2804] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Leader election won for term 21
I20260709 13:02:39.453055  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 21 candidate_status { last_received { term: 19 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:39.453634  1752 raft_consensus.cc:3060] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:02:39.455202  2329 raft_consensus.cc:1275] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 21 index: 16. Preceding OpId from leader: term: 23 index: 17. (index mismatch)
I20260709 13:02:39.456308  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:39.457064  1747 raft_consensus.cc:3060] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:02:39.458227  1603 raft_consensus.cc:2468] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 23.
I20260709 13:02:39.459975  1754 raft_consensus.cc:1240] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 20. Current term is 21. Ops: []
I20260709 13:02:39.462213  1746 raft_consensus.cc:1275] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 18 index: 13. Preceding OpId from leader: term: 21 index: 14. (index mismatch)
I20260709 13:02:39.462229  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:39.462800  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:39.447436  9533 raft_consensus.cc:2804] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 13:02:39.460028  9424 raft_consensus.cc:493] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:39.469726  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 22 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:39.470067  9424 raft_consensus.cc:515] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:39.470709  1746 raft_consensus.cc:3060] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Advancing to term 22
W20260709 13:02:39.472622  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 167342us
I20260709 13:02:39.475365  2299 leader_election.cc:304] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:39.477639  2299 leader_election.cc:304] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:02:39.480459  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:39.481360  1750 raft_consensus.cc:3060] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:02:39.483877  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:39.484658  1754 raft_consensus.cc:2468] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 20.
I20260709 13:02:39.452744  9576 raft_consensus.cc:493] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:39.496544  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:39.497288  1753 raft_consensus.cc:3060] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Advancing to term 26
I20260709 13:02:39.444192  9486 raft_consensus.cc:493] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:39.467392  9575 raft_consensus.cc:493] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:39.467763  9521 raft_consensus.cc:697] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 21 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:02:39.469758  9533 raft_consensus.cc:493] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:39.478878  9476 raft_consensus.cc:493] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:39.482555  9209 raft_consensus.cc:2749] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:02:39.478999  9072 raft_consensus.cc:2749] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 13:02:39.591535  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:39.465965  9501 consensus_queue.cc:1059] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 15.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.041s
W20260709 13:02:39.616891  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:39.591796  9524 raft_consensus.cc:493] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:39.591207  9533 raft_consensus.cc:3060] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Advancing to term 24
W20260709 13:02:39.635699  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:39.636281  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:39.590396  9569 raft_consensus.cc:493] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:39.598647  9564 raft_consensus.cc:493] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:39.495628  9511 consensus_queue.cc:1048] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 14, Time since last communication: 0.131s
I20260709 13:02:39.643638  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:39.654841  1752 raft_consensus.cc:2468] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 21.
I20260709 13:02:39.655318  1754 raft_consensus.cc:3060] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Advancing to term 23
W20260709 13:02:39.661115  1675 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57092 (request call id 1724) took 1763 ms (client timeout 1658 ms). Trace:
I20260709 13:02:39.644532  1746 raft_consensus.cc:2468] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 22.
W20260709 13:02:39.646575  2298 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:39.661563  1675 rpcz_store.cc:269] 0709 13:02:37.897545 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:37.899060 (+  1515us) service_pool.cc:224] Handling call
0709 13:02:39.660902 (+1761842us) spinlock_profiling.cc:243] Waited 238 ms on lock 0x7b5400346a54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:02:39.661032 (+   130us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":548411264}
I20260709 13:02:39.662971  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:39.664278  1746 raft_consensus.cc:2468] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 19.
I20260709 13:02:39.505879  9576 raft_consensus.cc:515] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:39.659914  9564 raft_consensus.cc:515] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:39.655968  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:39.590758  9575 raft_consensus.cc:515] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:39.609315  9476 raft_consensus.cc:515] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:39.509779  9486 raft_consensus.cc:515] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:39.653266  9524 raft_consensus.cc:515] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:39.685532  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:39.686055  1752 raft_consensus.cc:3060] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:02:39.690832  2336 raft_consensus.cc:1275] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 21 index: 14. Preceding OpId from leader: term: 23 index: 15. (index mismatch)
I20260709 13:02:39.651110  9569 raft_consensus.cc:515] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:39.698256  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:39.694912   431 auto_leader_rebalancer.cc:527] All tables' leader rebalancing finished this round
I20260709 13:02:39.700121  1746 raft_consensus.cc:3060] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 16 FOLLOWER]: Advancing to term 17
W20260709 13:02:39.705981  1717 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.107.193:46445 timed out after 1.658s (SENT) [suppressed 11 similar messages]
W20260709 13:02:39.707259  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:39.708467  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 216723us
I20260709 13:02:39.711490  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:39.712126  1675 raft_consensus.cc:3060] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:02:39.713559  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:39.714170  1671 raft_consensus.cc:3060] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:02:39.715374  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:39.715987  1673 raft_consensus.cc:3060] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:02:39.717818  1679 raft_consensus.cc:1240] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 LEADER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 18. Current term is 19. Ops: []
I20260709 13:02:39.718509  1752 raft_consensus.cc:2468] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 23.
I20260709 13:02:39.719158  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:39.719748  1679 raft_consensus.cc:3060] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:02:39.724826  1750 raft_consensus.cc:2410] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for term 20 because replica has last-logged OpId of term: 18 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 11.
I20260709 13:02:39.726243  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:39.719482  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:39.738919  1749 raft_consensus.cc:2410] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 for term 21 because replica has last-logged OpId of term: 19 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 12.
I20260709 13:02:39.739341  9574 raft_consensus.cc:493] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:39.738785  1494 catalog_manager.cc:5697] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 11 to 14, leader changed from af520b67945d443b901bbeca37cc5701 (127.0.107.195) to 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194). New cstate: current_term: 14 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } }
I20260709 13:02:39.739890  9574 raft_consensus.cc:515] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:39.742192  1754 raft_consensus.cc:2468] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 23.
I20260709 13:02:39.743633  1754 raft_consensus.cc:1275] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 18 index: 11. Preceding OpId from leader: term: 19 index: 11. (term mismatch)
I20260709 13:02:39.744215  1754 pending_rounds.cc:85] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701: Aborting all ops after (but not including) 10
I20260709 13:02:39.744447  1754 pending_rounds.cc:107] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701: Aborting uncommitted NO_OP operation due to leader change: 18.11
I20260709 13:02:39.744670  1754 raft_consensus.cc:2889] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:02:39.745079  9486 leader_election.cc:290] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:39.745319  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:39.749867  1754 raft_consensus.cc:1275] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 18 index: 13. Preceding OpId from leader: term: 19 index: 13. (term mismatch)
I20260709 13:02:39.750268  1754 pending_rounds.cc:85] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701: Aborting all ops after (but not including) 12
I20260709 13:02:39.750528  1754 pending_rounds.cc:107] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701: Aborting uncommitted NO_OP operation due to leader change: 18.13
I20260709 13:02:39.750801  1754 raft_consensus.cc:2889] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:02:39.752195  1747 raft_consensus.cc:2468] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 23.
I20260709 13:02:39.759546  9576 leader_election.cc:290] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:39.761125  9564 leader_election.cc:290] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:02:39.762535  1672 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57092 (request call id 1721) took 1866 ms (client timeout 1878 ms). Trace:
W20260709 13:02:39.763047  1672 rpcz_store.cc:269] 0709 13:02:37.896377 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:37.897715 (+  1338us) service_pool.cc:224] Handling call
0709 13:02:39.762304 (+1864589us) spinlock_profiling.cc:243] Waited 1.17 s on lock 0x7b5400163d54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:02:39.762466 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2697096576}
I20260709 13:02:39.763919  1747 raft_consensus.cc:1275] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 18 index: 9. Preceding OpId from leader: term: 19 index: 10. (index mismatch)
W20260709 13:02:39.766569  2301 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:39.767320  1673 raft_consensus.cc:2468] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 23.
I20260709 13:02:39.768337  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:39.685148  9521 consensus_queue.cc:237] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.16, Last appended by leader: 15, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:39.780288  1746 raft_consensus.cc:2468] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 17.
I20260709 13:02:39.785349  1753 raft_consensus.cc:2468] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 26.
I20260709 13:02:39.772390  9569 leader_election.cc:290] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 13:02:39.727452  9584 log.cc:927] Time spent T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.053s	user 0.001s	sys 0.000s
I20260709 13:02:39.772568  9072 raft_consensus.cc:3055] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 13:02:39.705708  9522 consensus_queue.cc:1048] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.060s
I20260709 13:02:39.772768  9575 leader_election.cc:290] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:39.773972  9424 leader_election.cc:290] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:39.774302  9524 leader_election.cc:290] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:39.754539  1754 raft_consensus.cc:1240] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 26 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 25. Current term is 26. Ops: [25.18-25.18]
W20260709 13:02:39.802014  9506 consensus_meta.cc:287] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.533s	user 0.012s	sys 0.001s
I20260709 13:02:39.802536  9506 raft_consensus.cc:515] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:39.803726  1754 raft_consensus.cc:1275] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 22 index: 14. Preceding OpId from leader: term: 22 index: 15. (index mismatch)
I20260709 13:02:39.798130  9072 raft_consensus.cc:740] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:02:39.765306  1747 raft_consensus.cc:1275] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 20 index: 15. Preceding OpId from leader: term: 23 index: 17. (index mismatch)
I20260709 13:02:39.769654  1675 raft_consensus.cc:2468] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 18.
I20260709 13:02:39.804987  9072 consensus_queue.cc:260] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:39.768294  1752 raft_consensus.cc:1240] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 19. Current term is 20. Ops: []
I20260709 13:02:39.786394  1746 raft_consensus.cc:1240] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 16. Current term is 17. Ops: []
I20260709 13:02:39.755479  1671 raft_consensus.cc:2468] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 17.
I20260709 13:02:39.793792  1679 raft_consensus.cc:2468] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 23.
I20260709 13:02:39.795113  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:39.807125  1678 raft_consensus.cc:3060] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:02:39.797698  1753 raft_consensus.cc:1275] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 19 index: 13. Preceding OpId from leader: term: 23 index: 15. (index mismatch)
W20260709 13:02:39.818323  1600 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40166 (request call id 3129) took 3472 ms (client timeout 1841 ms). Trace:
I20260709 13:02:39.818740  1680 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52778 (request call id 2886) took 1194 ms. Trace:
I20260709 13:02:39.819344  1678 raft_consensus.cc:2468] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 18.
I20260709 13:02:39.819193  1680 rpcz_store.cc:276] 0709 13:02:38.624242 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:38.724915 (+100673us) service_pool.cc:224] Handling call
0709 13:02:39.818537 (+1093622us) spinlock_profiling.cc:243] Waited 392 ms on lock 0x7b54001e96d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:02:39.818677 (+   140us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":902561664}
W20260709 13:02:39.818783  1600 rpcz_store.cc:269] 0709 13:02:36.345620 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:36.354930 (+  9310us) service_pool.cc:224] Handling call
0709 13:02:39.818128 (+3463198us) spinlock_profiling.cc:243] Waited 1.51 s on lock 0x7b540029aed4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:02:39.818247 (+   119us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3471100544}
I20260709 13:02:39.815385  9506 leader_election.cc:290] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 22 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:39.822839  9072 raft_consensus.cc:3060] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 15 FOLLOWER]: Advancing to term 16
W20260709 13:02:39.830183  1674 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52778 (request call id 2880) took 1266 ms (client timeout 1598 ms). Trace:
W20260709 13:02:39.833613  1674 rpcz_store.cc:269] 0709 13:02:38.564034 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:38.703264 (+139230us) service_pool.cc:224] Handling call
0709 13:02:39.829965 (+1126701us) spinlock_profiling.cc:243] Waited 452 ms on lock 0x7b5400162e54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:02:39.830116 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1039045504}
W20260709 13:02:39.835047  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:39.836611  9483 raft_consensus.cc:3060] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:02:39.836644  9476 leader_election.cc:290] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:02:39.840793  1604 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40154 (request call id 3049) took 1555 ms (client timeout 1785 ms). Trace:
W20260709 13:02:39.841231  1604 rpcz_store.cc:269] 0709 13:02:38.285243 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:38.792665 (+507422us) service_pool.cc:224] Handling call
0709 13:02:39.840582 (+1047917us) spinlock_profiling.cc:243] Waited 540 ms on lock 0x7b54000ac8d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:02:39.840700 (+   118us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1241555200}
W20260709 13:02:39.845721  1717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:39.846328  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:39.857329  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 735507us
I20260709 13:02:39.863155  1646 leader_election.cc:304] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:02:39.864131  2298 leader_election.cc:304] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:02:39.865056  9529 raft_consensus.cc:2804] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 24 FOLLOWER]: Leader election won for term 24
I20260709 13:02:39.864813  9450 consensus_queue.cc:1048] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 13:02:39.874063  1717 leader_election.cc:304] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:39.878937  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:39.883502  1748 raft_consensus.cc:1240] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 23 LEADER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 22. Current term is 23. Ops: []
I20260709 13:02:39.884593  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 25 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:39.887817  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 20 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:39.888273  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:39.888741  1752 raft_consensus.cc:2410] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for term 20 because replica has last-logged OpId of term: 16 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 9.
I20260709 13:02:39.889025  1747 raft_consensus.cc:2393] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 20: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:02:39.889472  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:39.889607  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:39.892870  1746 raft_consensus.cc:2393] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 21: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:02:39.894016  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:39.895004  1753 raft_consensus.cc:2393] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 23: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:02:39.899049  1596 raft_consensus.cc:2393] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 21: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:02:39.932502  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:39.933372  1747 raft_consensus.cc:2468] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 19.
I20260709 13:02:39.906221  1646 leader_election.cc:304] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:39.958796  1745 raft_consensus.cc:1275] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 17 index: 10. Preceding OpId from leader: term: 22 index: 12. (index mismatch)
I20260709 13:02:39.865607  9529 raft_consensus.cc:697] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 24 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:02:39.969945  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:39.970642  1754 raft_consensus.cc:3060] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Advancing to term 22
I20260709 13:02:39.971673  1604 raft_consensus.cc:1275] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 24 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 22 index: 14. Preceding OpId from leader: term: 24 index: 15. (index mismatch)
I20260709 13:02:39.964396  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:39.881577  9515 raft_consensus.cc:2749] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 13:02:39.964396  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:39.975155  1598 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40154 (request call id 3042) took 1766 ms. Trace:
I20260709 13:02:39.943140  1646 leader_election.cc:304] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:39.979970  1646 leader_election.cc:304] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:39.982257  1603 raft_consensus.cc:1275] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 19 index: 14. Preceding OpId from leader: term: 20 index: 15. (index mismatch)
I20260709 13:02:39.982546  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:39.985761  1597 raft_consensus.cc:2468] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 19.
I20260709 13:02:39.986830  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:39.983738  1605 raft_consensus.cc:1240] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 20. Current term is 21. Ops: [20.15-20.15]
I20260709 13:02:39.983892  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:39.988013  1599 raft_consensus.cc:3060] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:02:39.990062  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 24 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:39.990243  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:39.990550  1748 raft_consensus.cc:2393] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 21: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:02:39.990787  1602 raft_consensus.cc:2468] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 23.
I20260709 13:02:39.964834  9524 raft_consensus.cc:2804] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Leader election won for term 21
I20260709 13:02:39.991753  9524 raft_consensus.cc:697] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:02:39.992776  9524 consensus_queue.cc:237] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 19.9, Last appended by leader: 9, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:39.994668  9522 raft_consensus.cc:2804] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Leader election won for term 22
I20260709 13:02:39.995245  9522 raft_consensus.cc:697] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:02:39.995872  9322 leader_election.cc:290] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:39.996165  9522 consensus_queue.cc:237] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 19.11, Last appended by leader: 11, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:39.990904  1601 raft_consensus.cc:3060] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Advancing to term 24
I20260709 13:02:39.998819  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:39.999806  1602 raft_consensus.cc:3060] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Advancing to term 20
W20260709 13:02:40.003769  1645 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:39.891840  9409 consensus_queue.cc:1048] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.113s
I20260709 13:02:39.963788  9594 consensus_queue.cc:1048] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.044s
I20260709 13:02:40.010103  1599 raft_consensus.cc:2468] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 21.
I20260709 13:02:40.011861  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:40.012511  1596 raft_consensus.cc:3060] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Advancing to term 18
W20260709 13:02:40.017566  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:39.975548  1598 rpcz_store.cc:276] 0709 13:02:38.208330 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:38.280619 (+ 72289us) service_pool.cc:224] Handling call
0709 13:02:38.378354 (+ 97735us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:38.378633 (+   279us) raft_consensus.cc:1534] Early marking committed up to index 16
0709 13:02:38.821307 (+442674us) log.cc:844] Serialized 14 byte log entry
0709 13:02:39.274312 (+453005us) spinlock_profiling.cc:243] Waited 1.1 ms on lock 0x7b4400022b4c. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f4262f9bc85 00007f4262f9c6a2 00007f4265d0e1db 00007f4265d09aaf 00007f4265cfd86e 00007f4265cfcbb6 00007f4265cfc7d8 00007f4265d06221 00007f4265d061d9 00007f4265d06169
0709 13:02:39.847794 (+573482us) spinlock_profiling.cc:243] Waited 26.1 ms on lock 0x7b18003db160. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f4262fefc0a 00007f426c72f219 00007f4263077a40 00007f42630724e4 00007f4263072455 00007f426615a060 00007f426616c716 00007f426617d287 00007f426617d208
0709 13:02:39.964034 (+116240us) log.cc:844] Serialized 14 byte log entry
0709 13:02:39.974762 (+ 10728us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:39.974848 (+    86us) raft_consensus.cc:1645] Marking committed up to 16
0709 13:02:39.974900 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:39.974967 (+    67us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:39.975079 (+   112us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":153729,"prepare.queue_time_us":490907,"prepare.run_cpu_time_us":9005,"prepare.run_wall_time_us":149617,"raft.queue_time_us":865369,"raft.run_cpu_time_us":125,"raft.run_wall_time_us":125,"spinlock_wait_cycles":62601344,"thread_start_us":618619,"threads_started":3,"wal-append.queue_time_us":391138}
I20260709 13:02:39.971983  9426 raft_consensus.cc:2804] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:02:40.029404  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
W20260709 13:02:40.028741  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 984400us
I20260709 13:02:40.031036  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:40.031965  9529 consensus_queue.cc:237] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 21.11, Last appended by leader: 11, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:40.032806  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:40.039191  1603 raft_consensus.cc:2393] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 23: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
W20260709 13:02:40.028681  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 48871.6us
I20260709 13:02:40.033987  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:40.040570  1604 raft_consensus.cc:2393] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 21: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:02:40.040676  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:40.042088  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:40.040025  1646 leader_election.cc:304] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 13:02:40.042766  1599 raft_consensus.cc:2468] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 21.
I20260709 13:02:40.038277  1602 raft_consensus.cc:2468] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 20.
I20260709 13:02:40.044987  1601 raft_consensus.cc:2468] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 24.
I20260709 13:02:40.043638  1752 raft_consensus.cc:2468] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 23.
I20260709 13:02:40.046597  1754 raft_consensus.cc:2468] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 22.
I20260709 13:02:40.048090  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:40.045164  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:40.043725  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:40.049439  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 22 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:40.049635  1602 raft_consensus.cc:2468] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 19.
I20260709 13:02:40.053772  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:40.053973  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 18 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:40.059139  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:40.059369  1596 raft_consensus.cc:2468] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 18.
I20260709 13:02:40.026856  9594 raft_consensus.cc:493] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:40.062498  1751 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55840 (request call id 3236) took 3125 ms. Trace:
I20260709 13:02:40.062943  1751 rpcz_store.cc:276] 0709 13:02:36.937565 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:38.006537 (+1068972us) service_pool.cc:224] Handling call
0709 13:02:38.193432 (+186895us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:38.204713 (+ 11281us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:40.062226 (+1857513us) spinlock_profiling.cc:243] Waited 174 ms on lock 0x7b540031f450. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661618ad 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:02:40.062395 (+   169us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":400716672}
I20260709 13:02:40.064978  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:40.065855  1603 raft_consensus.cc:2468] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 21.
I20260709 13:02:40.033413  9506 raft_consensus.cc:2804] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:02:40.068137  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 17 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:40.040462  9426 raft_consensus.cc:493] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:40.070235  9564 consensus_queue.cc:1048] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 13:02:40.075923  8968 raft_consensus.cc:493] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:40.076937  1646 leader_election.cc:304] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:40.077634  9563 consensus_queue.cc:1048] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 13:02:40.104522  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:40.106870  1604 raft_consensus.cc:2468] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 25.
I20260709 13:02:40.102803  9426 raft_consensus.cc:3060] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:02:40.094744  9322 consensus_queue.cc:1048] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.013s
I20260709 13:02:40.107455  8968 raft_consensus.cc:515] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:40.109247  9486 raft_consensus.cc:2804] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Leader election won for term 17
I20260709 13:02:40.100903  9506 raft_consensus.cc:493] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:40.028422  9593 consensus_queue.cc:1048] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.085s
I20260709 13:02:40.107491  2332 raft_consensus.cc:1275] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 20 index: 14. Preceding OpId from leader: term: 21 index: 14. (term mismatch)
I20260709 13:02:40.118808  2332 pending_rounds.cc:85] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1: Aborting all ops after (but not including) 13
W20260709 13:02:40.118523  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:40.106065  1646 leader_election.cc:304] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:40.024701  9533 raft_consensus.cc:515] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:40.114897  1676 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52778 (request call id 2898) took 1234 ms. Trace:
I20260709 13:02:40.123219  1676 rpcz_store.cc:276] 0709 13:02:38.879915 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:39.140535 (+260620us) service_pool.cc:224] Handling call
0709 13:02:40.114259 (+973724us) spinlock_profiling.cc:243] Waited 219 ms on lock 0x7b5400347e54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:02:40.114380 (+   121us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":503771392}
I20260709 13:02:40.066875  9409 raft_consensus.cc:2804] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Leader election won for term 23
I20260709 13:02:40.117007  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:40.124785  1603 raft_consensus.cc:2393] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 21: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:02:40.117863  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 21 candidate_status { last_received { term: 19 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:40.125681  1598 raft_consensus.cc:3060] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Advancing to term 21
I20260709 13:02:40.119089  2332 pending_rounds.cc:107] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1: Aborting uncommitted NO_OP operation due to leader change: 20.14
I20260709 13:02:40.114663  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:40.126562  2332 raft_consensus.cc:2889] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:02:40.127359  1604 raft_consensus.cc:2393] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 24: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:02:40.114802  9322 consensus_queue.cc:1243] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer af520b67945d443b901bbeca37cc5701 log is divergent from this leader: its last log entry 16.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 9
I20260709 13:02:40.134788  1646 leader_election.cc:304] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:40.101132  9594 raft_consensus.cc:515] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:40.116487  9476 raft_consensus.cc:493] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:40.118928  9506 raft_consensus.cc:3060] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:02:40.122804  9486 raft_consensus.cc:697] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:02:40.131588  9335 raft_consensus.cc:493] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:40.132428  9564 raft_consensus.cc:2804] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 26 FOLLOWER]: Leader election won for term 26
I20260709 13:02:40.132117  9522 consensus_queue.cc:1059] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 19.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.011s
I20260709 13:02:40.140115  9409 raft_consensus.cc:697] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 23 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:02:40.142671  9489 raft_consensus.cc:2804] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Leader election won for term 22
I20260709 13:02:40.132795  9563 consensus_queue.cc:1048] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.021s
W20260709 13:02:40.162263  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:40.137923  9226 consensus_queue.cc:1048] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.020s
I20260709 13:02:40.157727  9564 raft_consensus.cc:697] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 26 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:02:40.179776  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 17 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:40.148590  9599 consensus_queue.cc:1048] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.038s
I20260709 13:02:40.149242  9476 raft_consensus.cc:515] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:40.190117  9486 consensus_queue.cc:237] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.11, Last appended by leader: 11, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:40.195317  8968 leader_election.cc:290] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:40.194712  9409 consensus_queue.cc:237] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 22.16, Last appended by leader: 16, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:40.155783  9335 raft_consensus.cc:515] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:40.183720  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 44243.7us
I20260709 13:02:40.177209  9489 raft_consensus.cc:697] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 22 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:02:40.181478  9250 raft_consensus.cc:3055] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 LEADER]: Stepping down as leader of term 19
I20260709 13:02:40.207605  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:40.209740  1600 raft_consensus.cc:2468] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 23.
W20260709 13:02:40.216435  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:40.218098  1750 pending_rounds.cc:85] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701: Aborting all ops after (but not including) 9
I20260709 13:02:40.218461  1750 pending_rounds.cc:107] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701: Aborting uncommitted NO_OP operation due to leader change: 16.10
I20260709 13:02:40.218700  1750 raft_consensus.cc:2889] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 13:02:40.220057  1719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:40.222409  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:40.223153  1751 raft_consensus.cc:1275] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 17 index: 11. (index mismatch)
I20260709 13:02:40.209498  9250 raft_consensus.cc:740] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:02:40.236217  1600 raft_consensus.cc:1275] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 18 index: 9. Preceding OpId from leader: term: 19 index: 10. (index mismatch)
W20260709 13:02:40.238308  1717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:40.238348  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 374458us
I20260709 13:02:40.232722  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:40.242204  2298 leader_election.cc:304] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:02:40.244622  1604 raft_consensus.cc:2393] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 18: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:02:40.248941  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:40.240463  8893 raft_consensus.cc:493] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:40.250859  9588 raft_consensus.cc:2804] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 13:02:40.255163  9588 raft_consensus.cc:697] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 23 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:02:40.256162  9588 consensus_queue.cc:237] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 21.16, Last appended by leader: 16, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:40.260623  9250 consensus_queue.cc:260] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.13, Last appended by leader: 13, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:40.261500  9072 consensus_queue.cc:1059] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 19.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 13:02:40.262130  9250 raft_consensus.cc:3060] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:02:40.262404  9072 raft_consensus.cc:3055] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 13:02:40.262825  9072 raft_consensus.cc:740] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:02:40.263073  9594 leader_election.cc:290] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:40.251605  9533 leader_election.cc:290] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 24 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:40.263696  9072 consensus_queue.cc:260] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.15, Last appended by leader: 15, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:40.251343  9483 consensus_queue.cc:1048] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.013s
I20260709 13:02:40.265005  9072 raft_consensus.cc:3060] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:02:40.244256  9282 raft_consensus.cc:2804] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:02:40.238735  9478 consensus_queue.cc:1059] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 22.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.013s
I20260709 13:02:40.265486  9282 raft_consensus.cc:493] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:40.265908  9282 raft_consensus.cc:3060] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:02:40.266093  9599 raft_consensus.cc:3055] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 LEADER]: Stepping down as leader of term 22
I20260709 13:02:40.270774  8893 raft_consensus.cc:515] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:40.283663  9599 raft_consensus.cc:740] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:02:40.284322  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:40.284935  2329 raft_consensus.cc:3060] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:02:40.284648  9599 consensus_queue.cc:260] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 22.16, Last appended by leader: 16, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:40.285328  8893 leader_election.cc:290] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:40.231150  9564 consensus_queue.cc:237] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 24.15, Last appended by leader: 15, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:40.286108  9599 raft_consensus.cc:3060] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:02:40.274266  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 22 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:40.297545  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:40.298622  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:40.299213  1599 raft_consensus.cc:2468] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 16.
I20260709 13:02:40.274864  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:40.276314  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:40.278671  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:40.300843  1596 raft_consensus.cc:2393] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 24: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:02:40.300940  1602 raft_consensus.cc:3060] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 25 FOLLOWER]: Advancing to term 26
I20260709 13:02:40.280933  9335 leader_election.cc:290] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:40.281620  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:40.312701  1604 raft_consensus.cc:1275] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 18 index: 15. Preceding OpId from leader: term: 19 index: 16. (index mismatch)
I20260709 13:02:40.286773  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:40.313890  2334 raft_consensus.cc:2393] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 18: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:02:40.271728  9476 leader_election.cc:290] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:40.261129  9574 leader_election.cc:290] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:40.296370  1751 raft_consensus.cc:1240] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 16. Current term is 17. Ops: []
I20260709 13:02:40.302491  1603 raft_consensus.cc:2393] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 24: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:02:40.312899  1596 raft_consensus.cc:1240] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 19. Current term is 20. Ops: []
I20260709 13:02:40.249512  9489 consensus_queue.cc:237] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.15, Last appended by leader: 15, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:40.296023  1717 leader_election.cc:304] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:40.314716  1599 raft_consensus.cc:1275] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 19 index: 11. (index mismatch)
I20260709 13:02:40.318738  1598 raft_consensus.cc:2468] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 21.
I20260709 13:02:40.320892  2331 raft_consensus.cc:2393] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 24: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:02:40.321870  2301 leader_election.cc:304] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:40.336053  9523 raft_consensus.cc:2804] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 13:02:40.336808  9523 raft_consensus.cc:697] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 23 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:02:40.338308  9613 consensus_queue.cc:1048] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 13:02:40.340341  1676 raft_consensus.cc:1275] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 22 index: 14. Preceding OpId from leader: term: 22 index: 15. (index mismatch)
I20260709 13:02:40.342916  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:40.343549  1676 raft_consensus.cc:3060] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Advancing to term 24
I20260709 13:02:40.329931  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:40.343056  1673 raft_consensus.cc:1240] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 22. Current term is 23. Ops: []
I20260709 13:02:40.346144  1600 raft_consensus.cc:3060] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:02:40.346861  8893 raft_consensus.cc:493] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:40.348371  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:40.349403  1751 raft_consensus.cc:2468] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 23.
I20260709 13:02:40.350284  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 22 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:40.348353  1717 leader_election.cc:304] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:40.352568  1717 leader_election.cc:304] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:40.353567  2301 leader_election.cc:304] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
W20260709 13:02:40.331166  1571 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:40.358165  9605 log.cc:927] Time spent T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.111s	user 0.001s	sys 0.000s
I20260709 13:02:40.360706  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 22 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:40.347469  8893 raft_consensus.cc:515] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:40.364804  8893 leader_election.cc:290] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:40.366675  9424 raft_consensus.cc:2804] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 13:02:40.367268  9424 raft_consensus.cc:697] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 21 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:02:40.368253  9424 consensus_queue.cc:237] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.14, Last appended by leader: 14, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:40.370056  9558 raft_consensus.cc:2804] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 13:02:40.370572  9558 raft_consensus.cc:493] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:40.370925  9558 raft_consensus.cc:3060] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Advancing to term 24
I20260709 13:02:40.371248  9412 raft_consensus.cc:3055] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 LEADER]: Stepping down as leader of term 18
I20260709 13:02:40.371692  9412 raft_consensus.cc:740] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:02:40.373132  1751 raft_consensus.cc:1240] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 18. Current term is 19. Ops: []
I20260709 13:02:40.361881  9594 raft_consensus.cc:2804] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 13:02:40.380242  1676 raft_consensus.cc:2468] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 24.
I20260709 13:02:40.382469  1602 raft_consensus.cc:2468] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 26.
I20260709 13:02:40.384763  1569 leader_election.cc:304] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:02:40.388927  1674 raft_consensus.cc:1240] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 23 LEADER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 22. Current term is 23. Ops: []
I20260709 13:02:40.332284  9483 raft_consensus.cc:493] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:40.391932  1717 leader_election.cc:304] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:40.352907  9574 raft_consensus.cc:2804] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:02:40.399718  1717 leader_election.cc:304] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:40.402176  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:40.402652  9450 raft_consensus.cc:2749] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:02:40.403514  8968 raft_consensus.cc:2749] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 13:02:40.404304  9412 consensus_queue.cc:260] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.13, Last appended by leader: 13, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:40.405855  9412 raft_consensus.cc:3060] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:02:40.415547  8994 raft_consensus.cc:3060] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:02:40.419265  9523 consensus_queue.cc:237] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 21.16, Last appended by leader: 16, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:40.421351  1677 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57092 (request call id 1718) took 2578 ms (client timeout 2643 ms). Trace:
W20260709 13:02:40.421798  1677 rpcz_store.cc:269] 0709 13:02:37.843324 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:37.843518 (+   194us) service_pool.cc:224] Handling call
0709 13:02:40.421106 (+2577588us) spinlock_profiling.cc:243] Waited 1.34 s on lock 0x7b54001e7dd4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:02:40.421259 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3089966336}
I20260709 13:02:40.356537  9446 consensus_queue.cc:1048] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.074s
I20260709 13:02:40.436952  9518 leader_election.cc:290] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 24 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:40.436646  9483 raft_consensus.cc:515] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:40.439054  9483 leader_election.cc:290] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:40.440689  9513 raft_consensus.cc:3060] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:02:40.441738  9426 raft_consensus.cc:515] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:40.446895  9476 raft_consensus.cc:2804] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 13:02:40.447314  9476 raft_consensus.cc:493] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:40.447362  9437 consensus_queue.cc:1048] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 13:02:40.447849  9476 raft_consensus.cc:3060] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:02:40.453612  2301 leader_election.cc:304] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:02:40.400938  1717 leader_election.cc:304] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
W20260709 13:02:40.455123  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:40.456970  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 56162.3us
W20260709 13:02:40.462459  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 73148.9us
I20260709 13:02:40.331439  9335 consensus_queue.cc:1059] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.063s
I20260709 13:02:40.387432  9594 raft_consensus.cc:493] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:40.468063  2332 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39726 (request call id 1728) took 1657 ms. Trace:
I20260709 13:02:40.470017  2332 rpcz_store.cc:276] 0709 13:02:38.810504 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:38.810675 (+   171us) service_pool.cc:224] Handling call
0709 13:02:38.810971 (+   296us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:40.131413 (+1320442us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:40.467767 (+336354us) spinlock_profiling.cc:243] Waited 1.3 s on lock 0x7b54004880d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:02:40.467974 (+   207us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2980044672}
I20260709 13:02:40.474292  1571 leader_election.cc:304] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:40.373574  9336 consensus_queue.cc:1059] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 16.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.103s
I20260709 13:02:40.472296  1717 leader_election.cc:304] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:40.480137  1754 raft_consensus.cc:1240] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 19. Current term is 20. Ops: [19.13-19.13]
I20260709 13:02:40.480605  1571 leader_election.cc:304] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:40.481472  9502 raft_consensus.cc:493] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:40.450841  9574 raft_consensus.cc:493] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:40.486306  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:40.486510  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:40.488159  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:02:40.490051  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 117061us
I20260709 13:02:40.527441  1598 raft_consensus.cc:2468] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 22.
I20260709 13:02:40.472352  9594 raft_consensus.cc:3060] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Advancing to term 24
I20260709 13:02:40.482141  9339 raft_consensus.cc:2804] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:02:40.563545  1597 raft_consensus.cc:2468] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 19.
I20260709 13:02:40.537897  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:40.526141  9335 raft_consensus.cc:493] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:40.483140  9446 raft_consensus.cc:2804] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 13:02:40.505074  1643 leader_election.cc:304] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:40.489697  9550 raft_consensus.cc:2804] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 23 FOLLOWER]: Leader election won for term 23
W20260709 13:02:40.541615  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 69442.4us
I20260709 13:02:40.489893  9478 consensus_queue.cc:1048] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.093s
W20260709 13:02:40.591214  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 101539us
I20260709 13:02:40.592676  1643 leader_election.cc:304] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:02:40.586499  2329 raft_consensus.cc:2468] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 22.
I20260709 13:02:40.531836  9518 raft_consensus.cc:493] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:02:40.532882  9181 raft_consensus.cc:3055] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 LEADER]: Stepping down as leader of term 16
I20260709 13:02:40.500062  9609 raft_consensus.cc:493] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
W20260709 13:02:40.593155  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 242867us
I20260709 13:02:40.605023  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:40.605886  1747 raft_consensus.cc:2393] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 22: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:02:40.549938  1598 raft_consensus.cc:1275] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 19 index: 13. (index mismatch)
I20260709 13:02:40.526213  9450 consensus_queue.cc:1059] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.085s
I20260709 13:02:40.597498  1569 leader_election.cc:304] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:02:40.484777  9282 raft_consensus.cc:515] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:40.566016  9339 raft_consensus.cc:493] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:40.598896  1717 leader_election.cc:304] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:40.599952  9522 raft_consensus.cc:2749] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 13:02:40.600610  9511 raft_consensus.cc:2749] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:02:40.484899  9483 raft_consensus.cc:2749] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:02:40.555751  9327 raft_consensus.cc:493] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:40.611057  2299 leader_election.cc:304] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:02:40.526178  9502 raft_consensus.cc:515] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:40.603193  9518 raft_consensus.cc:515] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:40.589082  1599 raft_consensus.cc:3055] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 LEADER]: Stepping down as leader of term 16
I20260709 13:02:40.575366  9574 raft_consensus.cc:3060] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:02:40.601259  9446 raft_consensus.cc:697] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 20 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:02:40.652261  1599 raft_consensus.cc:740] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:02:40.654639  2299 leader_election.cc:304] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:40.602197  9181 raft_consensus.cc:740] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
W20260709 13:02:40.617971  9616 log.cc:927] Time spent T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.136s	user 0.001s	sys 0.000s
I20260709 13:02:40.588738  9336 consensus_queue.cc:1059] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 16.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.035s
I20260709 13:02:40.663648  2299 leader_election.cc:304] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:40.642937  9453 raft_consensus.cc:2804] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 13:02:40.659065  9530 raft_consensus.cc:2749] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 13:02:40.623625  9339 raft_consensus.cc:3060] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:02:40.589157  9335 raft_consensus.cc:515] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:40.659991  1600 raft_consensus.cc:2468] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 17.
I20260709 13:02:40.677716  9517 raft_consensus.cc:2804] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 13:02:40.682001  9517 raft_consensus.cc:697] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 17 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:02:40.682776  9518 leader_election.cc:290] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:40.683100  9517 consensus_queue.cc:237] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 16.14, Last appended by leader: 14, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:40.676188  9453 raft_consensus.cc:493] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:40.666133  9529 consensus_queue.cc:1048] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 13, Time since last communication: 0.026s
I20260709 13:02:40.700067  9558 raft_consensus.cc:515] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:40.625617  9609 raft_consensus.cc:515] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:40.716387  9282 leader_election.cc:290] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 21 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:40.717231  9609 leader_election.cc:290] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:40.717682  9426 leader_election.cc:290] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 20 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:40.670864  9483 raft_consensus.cc:2804] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 13:02:40.719353  9483 raft_consensus.cc:697] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 18 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:02:40.689888  9453 raft_consensus.cc:3060] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:02:40.720428  9483 consensus_queue.cc:237] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 16.14, Last appended by leader: 14, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:40.659732  9327 raft_consensus.cc:515] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:40.659601  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:40.670113  2299 leader_election.cc:304] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
W20260709 13:02:40.723992  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 54009.7us
I20260709 13:02:40.606470  9506 raft_consensus.cc:515] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:02:40.697973  1644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:40.709163  9478 raft_consensus.cc:2749] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:02:40.729592  9530 raft_consensus.cc:2804] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 13:02:40.729880  9358 consensus_queue.cc:1048] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 13:02:40.702841  9181 consensus_queue.cc:260] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.10, Last appended by leader: 10, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:40.730657  8893 consensus_queue.cc:1059] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 22.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20260709 13:02:40.729898  2299 leader_election.cc:304] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:02:40.731638  9358 raft_consensus.cc:3055] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 13:02:40.731706  9181 raft_consensus.cc:3060] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:02:40.732197  9358 raft_consensus.cc:740] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:02:40.732475  9518 raft_consensus.cc:2749] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:02:40.733276  2299 leader_election.cc:304] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:02:40.710183  9446 consensus_queue.cc:237] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.13, Last appended by leader: 13, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:40.734129  8893 raft_consensus.cc:2804] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 13:02:40.734267  2299 leader_election.cc:304] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:40.737176  9590 raft_consensus.cc:2749] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:02:40.736652  8893 raft_consensus.cc:697] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 24 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:02:40.738078  9335 leader_election.cc:290] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:02:40.738376  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:40.738780  9590 raft_consensus.cc:493] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:02:40.730120  9530 raft_consensus.cc:697] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [term 18 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:02:40.739465  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:40.739512  9590 raft_consensus.cc:515] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:40.740458  9530 consensus_queue.cc:237] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 16.13, Last appended by leader: 13, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:40.741241  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:40.738838  9072 raft_consensus.cc:493] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:40.719115  9558 leader_election.cc:290] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 24 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:40.661559  1599 consensus_queue.cc:260] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 16.14, Last appended by leader: 14, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:40.748425  2331 raft_consensus.cc:1240] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 20. Current term is 21. Ops: [20.14-20.14]
I20260709 13:02:40.750383  1599 raft_consensus.cc:3060] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:02:40.780227  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:40.709374  9502 leader_election.cc:290] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:40.736567  9506 leader_election.cc:290] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 20 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:40.770021  1597 raft_consensus.cc:3055] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 LEADER]: Stepping down as leader of term 20
I20260709 13:02:40.786302  1597 raft_consensus.cc:740] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:02:40.785727  8893 consensus_queue.cc:237] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 22.16, Last appended by leader: 16, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:40.787164  9358 consensus_queue.cc:260] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 22.16, Last appended by leader: 16, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:40.788663  9358 raft_consensus.cc:3060] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:02:40.794636  9327 leader_election.cc:290] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:40.799609  2330 raft_consensus.cc:1275] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 18 index: 15. Preceding OpId from leader: term: 19 index: 16. (index mismatch)
I20260709 13:02:40.800995  1602 raft_consensus.cc:1275] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 17 index: 11. (index mismatch)
I20260709 13:02:40.801801  9515 raft_consensus.cc:493] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:40.781442  2331 raft_consensus.cc:2468] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 26.
I20260709 13:02:40.748761  9476 raft_consensus.cc:515] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:40.802523  9515 raft_consensus.cc:515] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:40.759775  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:40.804006  1598 raft_consensus.cc:2468] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 23.
W20260709 13:02:40.804705  1719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:40.805634  9476 leader_election.cc:290] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 22 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:40.814247  9533 raft_consensus.cc:493] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:40.814698  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:40.815393  1753 raft_consensus.cc:3060] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Advancing to term 24
I20260709 13:02:40.746517  9529 raft_consensus.cc:3055] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 LEADER]: Stepping down as leader of term 16
I20260709 13:02:40.821980  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:02:40.823762  2298 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.576s	user 0.004s	sys 0.012s
W20260709 13:02:40.824206  2298 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.576s	user 0.004s	sys 0.012s
I20260709 13:02:40.813640  1597 consensus_queue.cc:260] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 20.16, Last appended by leader: 16, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:40.824545  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 576785us
I20260709 13:02:40.825516  1597 raft_consensus.cc:3060] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:02:40.827430  1753 raft_consensus.cc:2468] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 24.
I20260709 13:02:40.828994  2301 leader_election.cc:304] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:40.830828  9558 raft_consensus.cc:2804] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 13:02:40.832365  9558 raft_consensus.cc:697] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 24 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:02:40.833621  9558 consensus_queue.cc:237] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.16, Last appended by leader: 16, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:40.825400  2298 leader_election.cc:304] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:02:40.835695  9518 raft_consensus.cc:2804] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 13:02:40.836238  9518 raft_consensus.cc:697] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 24 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:02:40.838238  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 22 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:40.839000  2328 raft_consensus.cc:2468] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 21.
W20260709 13:02:40.844895  2298 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:40.886087  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:40.887382  2331 raft_consensus.cc:2468] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 20.
I20260709 13:02:40.746994  9609 consensus_queue.cc:1048] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.074s
I20260709 13:02:40.893157  1597 raft_consensus.cc:1275] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 20 index: 16. Preceding OpId from leader: term: 21 index: 17. (index mismatch)
I20260709 13:02:40.837182  9518 consensus_queue.cc:237] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 22.17, Last appended by leader: 17, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:40.749168  9072 raft_consensus.cc:515] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:40.912339  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:40.917124  1672 raft_consensus.cc:2393] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 24: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:02:40.924893  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:40.821709  9529 raft_consensus.cc:740] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:02:40.926662  2331 raft_consensus.cc:2468] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 23.
I20260709 13:02:40.940311  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:40.941578  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:40.958616  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:40.959955  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:40.884989  9282 raft_consensus.cc:493] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:40.814817  9533 raft_consensus.cc:515] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:40.988552  2329 raft_consensus.cc:2468] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 21.
W20260709 13:02:40.976617  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 379374us
I20260709 13:02:40.972532  1676 raft_consensus.cc:2468] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 22.
W20260709 13:02:40.981534  1571 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:41.003672  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:40.971725   431 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
W20260709 13:02:41.015993  1750 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.534s	user 0.010s	sys 0.002s
W20260709 13:02:41.045986  1750 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.564s	user 0.011s	sys 0.003s
I20260709 13:02:41.048594  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:41.049363  1597 raft_consensus.cc:3060] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 23 FOLLOWER]: Advancing to term 24
I20260709 13:02:40.978080  9335 raft_consensus.cc:493] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:41.071599  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:02:41.054723  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 437908us
I20260709 13:02:41.085323  1597 raft_consensus.cc:2468] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 24.
I20260709 13:02:41.089875  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:41.090791  1600 raft_consensus.cc:2468] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 23.
I20260709 13:02:41.092556  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:41.094343  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:41.096311  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:41.097000  1600 raft_consensus.cc:3060] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 23 FOLLOWER]: Advancing to term 24
I20260709 13:02:41.103343  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:41.104233  1596 raft_consensus.cc:2468] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 19.
I20260709 13:02:41.105020  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:41.114742  1598 raft_consensus.cc:1240] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 20. Current term is 21. Ops: [20.15-20.15]
W20260709 13:02:41.116168  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 501691us
I20260709 13:02:41.117982  1643 leader_election.cc:304] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:02:40.989028  9594 raft_consensus.cc:515] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:41.119753  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:41.120561  1753 raft_consensus.cc:2468] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 20.
W20260709 13:02:41.122742  1751 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.654s	user 0.049s	sys 0.035s
W20260709 13:02:41.124584  1751 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.656s	user 0.049s	sys 0.036s
I20260709 13:02:40.993731  9282 raft_consensus.cc:515] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:41.017503  9339 raft_consensus.cc:515] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:41.301651  1600 raft_consensus.cc:2468] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 24.
W20260709 13:02:41.219944  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 173538us
W20260709 13:02:41.203552  9424 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.519s	user 0.005s	sys 0.017s
I20260709 13:02:41.233072  1753 raft_consensus.cc:1240] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 21. Current term is 22. Ops: []
I20260709 13:02:41.079107  9574 raft_consensus.cc:515] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:02:41.229148  9517 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.507s	user 0.002s	sys 0.016s
I20260709 13:02:41.109993  9335 raft_consensus.cc:515] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:41.314581  9614 log.cc:927] Time spent T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.241s	user 0.005s	sys 0.007s
I20260709 13:02:41.219601  9590 leader_election.cc:290] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:02:41.282188  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:41.355348  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:41.286509  1601 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.674s	user 0.020s	sys 0.019s
W20260709 13:02:41.356192  1601 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.744s	user 0.020s	sys 0.019s
I20260709 13:02:41.357120  1601 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 3152) took 1202 ms. Trace:
W20260709 13:02:41.258368  1717 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.633s	user 0.002s	sys 0.019s
W20260709 13:02:41.358722  1717 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.733s	user 0.004s	sys 0.019s
W20260709 13:02:41.292171  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 174399us
I20260709 13:02:41.360266  1643 leader_election.cc:304] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
W20260709 13:02:41.293303  1599 consensus_meta.cc:287] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.542s	user 0.009s	sys 0.007s
I20260709 13:02:41.361631  1599 raft_consensus.cc:1275] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 16 index: 14. Preceding OpId from leader: term: 17 index: 15. (index mismatch)
W20260709 13:02:41.156764  1605 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.651s	user 0.011s	sys 0.006s
W20260709 13:02:41.109162  9587 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.554s	user 0.015s	sys 0.000s
W20260709 13:02:41.229297  9588 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.572s	user 0.000s	sys 0.019s
W20260709 13:02:41.297436  9426 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.559s	user 0.024s	sys 0.026s
W20260709 13:02:41.359112  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 734287us
I20260709 13:02:41.119400  9529 consensus_queue.cc:260] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 16.14, Last appended by leader: 14, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:41.228488  9450 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.565s	user 0.004s	sys 0.015s
W20260709 13:02:41.331220  9517 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.609s	user 0.003s	sys 0.016s
W20260709 13:02:41.321852  9424 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.678s	user 0.006s	sys 0.018s
W20260709 13:02:41.363104  1605 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.858s	user 0.012s	sys 0.011s
W20260709 13:02:41.329757  9550 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.706s	user 0.014s	sys 0.015s
W20260709 13:02:41.369603  1717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:41.374155  1746 raft_consensus.cc:1240] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 19. Current term is 20. Ops: []
I20260709 13:02:41.298064  9609 raft_consensus.cc:2749] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:02:41.357558  1601 rpcz_store.cc:276] 0709 13:02:40.154931 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:40.155094 (+   163us) service_pool.cc:224] Handling call
0709 13:02:40.155433 (+   339us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:40.155667 (+   234us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 13:02:40.203961 (+ 48294us) log.cc:844] Serialized 14 byte log entry
0709 13:02:41.356629 (+1152668us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:41.356712 (+    83us) raft_consensus.cc:1645] Marking committed up to 14
0709 13:02:41.356777 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:41.356866 (+    89us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:41.357044 (+   178us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2920,"prepare.queue_time_us":4740,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":35,"raft.queue_time_us":83982,"raft.run_cpu_time_us":43,"raft.run_wall_time_us":44,"spinlock_wait_cycles":37447424,"thread_start_us":1196165,"threads_started":3,"wal-append.queue_time_us":680874}
I20260709 13:02:41.366760  1753 raft_consensus.cc:1240] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 22. Current term is 23. Ops: []
W20260709 13:02:41.367372  9588 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.723s	user 0.000s	sys 0.020s
I20260709 13:02:41.387349  1717 leader_election.cc:304] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
W20260709 13:02:41.373704  9450 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.715s	user 0.004s	sys 0.016s
W20260709 13:02:41.367208  9587 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.888s	user 0.016s	sys 0.000s
W20260709 13:02:41.367660  9426 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.630s	user 0.024s	sys 0.027s
I20260709 13:02:41.398974  2337 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39726 (request call id 1748) took 1123 ms. Trace:
I20260709 13:02:41.400955  1751 raft_consensus.cc:1240] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 21 LEADER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 20. Current term is 21. Ops: [20.15-20.15]
W20260709 13:02:41.409410  9626 log.cc:927] Time spent T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.134s	user 0.001s	sys 0.000s
W20260709 13:02:41.410279  9627 log.cc:927] Time spent T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.108s	user 0.000s	sys 0.000s
W20260709 13:02:41.412268  9624 log.cc:927] Time spent T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.084s	user 0.000s	sys 0.000s
W20260709 13:02:41.412708  9453 consensus_meta.cc:287] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.692s	user 0.013s	sys 0.006s
W20260709 13:02:41.385773  9550 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.769s	user 0.015s	sys 0.016s
W20260709 13:02:41.414512  1571 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:41.414988  1571 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:41.415472  1571 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:41.418459  2299 leader_election.cc:304] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:02:41.419322  2336 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39726 (request call id 1744) took 1717 ms. Trace:
I20260709 13:02:41.419862  2336 rpcz_store.cc:276] 0709 13:02:39.702186 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:39.702349 (+   163us) service_pool.cc:224] Handling call
0709 13:02:39.702781 (+   432us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:39.703300 (+   519us) raft_consensus.cc:1534] Early marking committed up to index 16
0709 13:02:39.711993 (+  8693us) log.cc:844] Serialized 14 byte log entry
0709 13:02:40.032695 (+320702us) spinlock_profiling.cc:243] Waited 4.28 ms on lock 0x7b180031d980. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f42630753f3 00007f42630777c1 00007f4263077779 00007f4263077709 00007f42630776d1 00007f42630769cd 000000000038f354 000000000038f199 00007f426305883a
0709 13:02:40.228609 (+195914us) spinlock_profiling.cc:243] Waited 36.1 ms on lock 0x7b180031d980. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f4262fefc0a 00007f426c72f219 00007f426a03658f 00007f4263075571 00007f42630777c1 00007f4263077779 00007f4263077709 00007f42630776d1 00007f42630769cd
0709 13:02:40.420629 (+192020us) spinlock_profiling.cc:243] Waited 16 ms on lock 0x7b180031d980. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f4263057e07 00007f426beb2ab9 00007f426307469b 00007f426307293f 00007f4263074c02 00007f4265cfc20b 00007f4265d01d23 00007f4265d0223a 00007f426616ca2a
0709 13:02:40.867446 (+446817us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:40.911989 (+ 44543us) log.cc:844] Serialized 26 byte log entry
0709 13:02:40.912168 (+   179us) raft_consensus.cc:1645] Marking committed up to 16
0709 13:02:40.912200 (+    32us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:40.912277 (+    77us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:41.418975 (+506698us) raft_consensus.cc:1685] finished
0709 13:02:41.419008 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:41.419200 (+   192us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6528,"prepare.queue_time_us":8474,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":40,"raft.queue_time_us":31823,"raft.run_cpu_time_us":269,"raft.run_wall_time_us":14844,"spinlock_wait_cycles":129701376,"thread_start_us":420472,"threads_started":1,"wal-append.queue_time_us":1390764}
I20260709 13:02:41.379978  9524 raft_consensus.cc:2804] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 13:02:41.429360  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:41.430068  1599 raft_consensus.cc:3060] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:02:41.401857  2337 rpcz_store.cc:276] 0709 13:02:40.275933 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:40.282127 (+  6194us) service_pool.cc:224] Handling call
0709 13:02:40.282500 (+   373us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:40.282911 (+   411us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 13:02:40.282943 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:40.283475 (+   532us) log.cc:844] Serialized 26 byte log entry
0709 13:02:40.451288 (+167813us) raft_consensus.cc:1645] Marking committed up to 13
0709 13:02:40.451354 (+    66us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:40.451546 (+   192us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:41.398639 (+947093us) raft_consensus.cc:1685] finished
0709 13:02:41.398677 (+    38us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:41.398878 (+   201us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":3163,"raft.run_cpu_time_us":78,"raft.run_wall_time_us":77,"thread_start_us":166868,"threads_started":1,"wal-append.queue_time_us":763617}
W20260709 13:02:41.434720  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:41.434890  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
W20260709 13:02:41.424186  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 51676.7us
W20260709 13:02:41.437242  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 95773.3us
I20260709 13:02:41.437284  1643 leader_election.cc:304] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 13:02:41.443481  1571 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:41.443202  2299 leader_election.cc:304] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:02:41.391896  9513 consensus_queue.cc:1048] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.067s
I20260709 13:02:41.463240  1643 leader_election.cc:304] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
W20260709 13:02:41.404973  9621 log.cc:927] Time spent T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.359s	user 0.001s	sys 0.001s
I20260709 13:02:41.405093  9450 consensus_queue.cc:1243] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer af520b67945d443b901bbeca37cc5701 log is divergent from this leader: its last log entry 15.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 9
I20260709 13:02:41.477425  1643 leader_election.cc:304] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:41.471678  1569 leader_election.cc:304] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:02:41.484738  1569 leader_election.cc:304] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:02:41.425607  9550 raft_consensus.cc:697] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 23 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:02:41.461125  8893 raft_consensus.cc:2749] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:02:41.473966  9517 raft_consensus.cc:2804] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 13:02:41.491559  9517 raft_consensus.cc:493] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:41.491163  9550 consensus_queue.cc:237] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 22.11, Last appended by leader: 11, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:41.492130  9517 raft_consensus.cc:3060] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:02:41.495503  9574 leader_election.cc:290] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 20 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:41.496523  9339 leader_election.cc:290] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 20 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:41.480299  9502 raft_consensus.cc:2749] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:02:41.413199  9453 raft_consensus.cc:515] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:41.501798  9453 leader_election.cc:290] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 22 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:41.504755  9335 leader_election.cc:290] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:41.506111  1748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55840 (request call id 3254) took 1178 ms. Trace:
I20260709 13:02:41.506639  1748 rpcz_store.cc:276] 0709 13:02:40.327936 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:40.328086 (+   150us) service_pool.cc:224] Handling call
0709 13:02:40.328476 (+   390us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:02:40.328926 (+   450us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:40.328956 (+    30us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:02:40.348018 (+ 19062us) log.cc:844] Serialized 52 byte log entry
0709 13:02:40.819952 (+471934us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:02:40.820205 (+   253us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:40.820355 (+   150us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:41.505765 (+685410us) raft_consensus.cc:1685] finished
0709 13:02:41.505807 (+    42us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:41.506023 (+   216us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":603,"raft.run_cpu_time_us":138,"raft.run_wall_time_us":20074,"thread_start_us":365952,"threads_started":1,"wal-append.queue_time_us":1113323}
I20260709 13:02:41.470963  9072 leader_election.cc:290] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:41.514078  9594 leader_election.cc:290] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 24 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:41.439181  9524 raft_consensus.cc:493] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:41.515218  9524 raft_consensus.cc:3060] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Advancing to term 24
I20260709 13:02:41.521298  9483 raft_consensus.cc:493] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:41.522174  9483 raft_consensus.cc:515] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:41.523855  2298 leader_election.cc:304] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:02:41.479426  9563 raft_consensus.cc:493] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:41.525257  9563 raft_consensus.cc:515] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:41.525841  9483 leader_election.cc:290] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:41.526716  9530 raft_consensus.cc:2804] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:02:41.527245  9530 raft_consensus.cc:493] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:41.527699  9530 raft_consensus.cc:3060] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Advancing to term 21
W20260709 13:02:41.527699  2298 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:41.534052  1597 raft_consensus.cc:3055] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 LEADER]: Stepping down as leader of term 16
I20260709 13:02:41.534629  1597 raft_consensus.cc:740] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:02:41.535693  1597 consensus_queue.cc:260] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 16.14, Last appended by leader: 14, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:41.537024  1597 raft_consensus.cc:3060] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Advancing to term 17
W20260709 13:02:41.541255  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 63965.6us
I20260709 13:02:41.541306  9648 raft_consensus.cc:2749] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 13:02:41.542542  9648 consensus_queue.cc:1048] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 13:02:41.525619  8893 consensus_queue.cc:1059] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 19.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 13:02:41.545225  9563 leader_election.cc:290] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:41.549002  9282 leader_election.cc:290] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:41.462765  9529 raft_consensus.cc:3060] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:02:41.537408  1754 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55848 (request call id 3474) took 1332 ms. Trace:
I20260709 13:02:41.470101  9649 raft_consensus.cc:2804] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:02:41.510253  1748 raft_consensus.cc:1275] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 23 index: 12. Preceding OpId from leader: term: 24 index: 12. (term mismatch)
I20260709 13:02:41.561486  1748 pending_rounds.cc:85] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701: Aborting all ops after (but not including) 11
I20260709 13:02:41.552964  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:02:41.557150  1719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:41.562202  1600 raft_consensus.cc:2468] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 19.
I20260709 13:02:41.555191  1754 rpcz_store.cc:276] 0709 13:02:40.205177 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:40.205295 (+   118us) service_pool.cc:224] Handling call
0709 13:02:40.223597 (+ 18302us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:40.480626 (+257029us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:41.537039 (+1056413us) spinlock_profiling.cc:243] Waited 256 ms on lock 0x7b54000bd954. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:02:41.537303 (+   264us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":589556992}
I20260709 13:02:41.560115  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 19 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:41.561826  1748 pending_rounds.cc:107] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701: Aborting uncommitted NO_OP operation due to leader change: 23.12
I20260709 13:02:41.564932  1748 raft_consensus.cc:2889] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:02:41.542722  1643 leader_election.cc:304] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:02:41.563966  2298 leader_election.cc:304] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
W20260709 13:02:41.571499  1750 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.525s	user 0.011s	sys 0.003s
I20260709 13:02:41.579702  1750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43156 (request call id 1587) took 1237 ms. Trace:
I20260709 13:02:41.581326  1605 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40154 (request call id 3078) took 1097 ms. Trace:
I20260709 13:02:41.581737  1600 raft_consensus.cc:1275] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 24 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 22 index: 16. Preceding OpId from leader: term: 24 index: 17. (index mismatch)
I20260709 13:02:41.584019  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:41.584527  1597 raft_consensus.cc:1275] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 16 index: 14. Preceding OpId from leader: term: 17 index: 15. (index mismatch)
I20260709 13:02:41.592885  1605 rpcz_store.cc:276] 0709 13:02:40.484188 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:40.484325 (+   137us) service_pool.cc:224] Handling call
0709 13:02:40.484661 (+   336us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:41.380707 (+896046us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:41.380748 (+    41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:41.381430 (+   682us) log.cc:844] Serialized 26 byte log entry
0709 13:02:41.456915 (+ 75485us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:02:41.456967 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:41.457054 (+    87us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:41.581081 (+124027us) raft_consensus.cc:1685] finished
0709 13:02:41.581105 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:41.581255 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":20339,"raft.queue_time_us":47861,"raft.run_cpu_time_us":154,"raft.run_wall_time_us":16119,"thread_start_us":922850,"threads_started":2,"wal-append.queue_time_us":107300}
I20260709 13:02:41.598812  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 24 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:41.599720  1754 raft_consensus.cc:2393] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 24: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:02:41.591761  9450 raft_consensus.cc:2764] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Leader pre-election decision vote started in defunct term 16: won
I20260709 13:02:41.600811  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:41.600977  9633 raft_consensus.cc:2804] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:02:41.601503  9633 raft_consensus.cc:493] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Starting leader election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:02:41.601914  9633 raft_consensus.cc:3060] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:02:41.601855  1605 raft_consensus.cc:1275] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 16 index: 13. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
I20260709 13:02:41.601509  1754 raft_consensus.cc:2468] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 22.
I20260709 13:02:41.594161  9563 raft_consensus.cc:2749] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 13:02:41.604295  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 22 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:41.600353  9530 raft_consensus.cc:515] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:41.594822  1748 raft_consensus.cc:3060] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:02:41.605222  1754 raft_consensus.cc:2468] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 21.
I20260709 13:02:41.559813  9606 raft_consensus.cc:2749] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 13:02:41.486655  9590 raft_consensus.cc:493] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:41.557734  9649 raft_consensus.cc:493] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:41.639412  9649 raft_consensus.cc:3060] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:02:41.612598  1599 raft_consensus.cc:2468] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 20.
I20260709 13:02:41.641911  9530 leader_election.cc:290] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 21 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:41.592857  9609 consensus_queue.cc:1059] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 17.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.035s
I20260709 13:02:41.556617  9645 consensus_queue.cc:1059] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 21.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.086s
I20260709 13:02:41.643224  9609 raft_consensus.cc:3055] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 LEADER]: Stepping down as leader of term 19
I20260709 13:02:41.643385  9645 raft_consensus.cc:3055] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 21 LEADER]: Stepping down as leader of term 21
I20260709 13:02:41.643635  9609 raft_consensus.cc:740] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:02:41.643857  9645 raft_consensus.cc:740] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:02:41.644589  9609 consensus_queue.cc:260] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.13, Last appended by leader: 13, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:41.616300  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 23 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:41.617729  1646 leader_election.cc:304] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:41.617897  9533 leader_election.cc:290] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:41.605432  1750 rpcz_store.cc:276] 0709 13:02:40.342218 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:40.403340 (+ 61122us) service_pool.cc:224] Handling call
0709 13:02:40.403701 (+   361us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:40.403930 (+   229us) raft_consensus.cc:1534] Early marking committed up to index 16
0709 13:02:41.046665 (+642735us) log.cc:844] Serialized 14 byte log entry
0709 13:02:41.579139 (+532474us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:41.579225 (+    86us) raft_consensus.cc:1645] Marking committed up to 16
0709 13:02:41.579298 (+    73us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:41.579403 (+   105us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:41.579593 (+   190us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":64409,"prepare.queue_time_us":171843,"prepare.run_cpu_time_us":96,"prepare.run_wall_time_us":1335,"raft.queue_time_us":2020,"thread_start_us":1096341,"threads_started":2}
W20260709 13:02:41.626125  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 141543us
I20260709 13:02:41.626816  9517 raft_consensus.cc:515] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:41.592236  9483 raft_consensus.cc:2804] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:02:41.633066  1751 raft_consensus.cc:1275] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 15 index: 11. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
I20260709 13:02:41.652231  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:41.647190  9590 raft_consensus.cc:515] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:41.659662  2301 leader_election.cc:304] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:02:41.663554  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:41.664448  1673 raft_consensus.cc:2393] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 22: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:02:41.646027  9609 raft_consensus.cc:3060] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:02:41.653159  9606 raft_consensus.cc:2804] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 13:02:41.644737  9645 consensus_queue.cc:260] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 21.11, Last appended by leader: 11, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:41.647411  9618 raft_consensus.cc:3055] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 13:02:41.670081  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 20 candidate_status { last_received { term: 19 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:41.648517  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:41.665961  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:41.671959  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:41.656953  9483 raft_consensus.cc:493] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:41.661499  9590 leader_election.cc:290] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:41.675422  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:41.666292  2301 leader_election.cc:304] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:02:41.622406  1643 leader_election.cc:304] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:41.645044  1748 raft_consensus.cc:2468] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 20.
I20260709 13:02:41.681115  1680 raft_consensus.cc:1240] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 LEADER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 21. Current term is 22. Ops: []
I20260709 13:02:41.681944  9523 raft_consensus.cc:2749] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:02:41.682837  1604 raft_consensus.cc:3060] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:02:41.663935  9501 raft_consensus.cc:493] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:41.686015  1602 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40166 (request call id 3168) took 1153 ms. Trace:
I20260709 13:02:41.684999  9588 raft_consensus.cc:2955] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 22 LEADER]: Committing config change with OpId 22.15: config changed from index 5 to 15, VOTER 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) evicted. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 13:02:41.686394  9501 raft_consensus.cc:515] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:41.686472  1602 rpcz_store.cc:276] 0709 13:02:40.533159 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:40.534967 (+  1808us) service_pool.cc:224] Handling call
0709 13:02:40.577540 (+ 42573us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:40.801481 (+223941us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:41.685752 (+884271us) spinlock_profiling.cc:243] Waited 223 ms on lock 0x7b540006dbd4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:02:41.685941 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":513088384}
I20260709 13:02:41.689092  9501 leader_election.cc:290] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:41.689590  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:41.690032  9513 raft_consensus.cc:2749] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:02:41.690377  1677 raft_consensus.cc:2393] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 23: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:02:41.691807  9450 raft_consensus.cc:493] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:41.692309  9450 raft_consensus.cc:515] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:41.692775  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:41.693373  1602 raft_consensus.cc:3060] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:02:41.665671  9322 consensus_queue.cc:1048] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.029s
I20260709 13:02:41.675038  9483 raft_consensus.cc:3060] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:02:41.695046  9634 consensus_queue.cc:1048] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 13:02:41.695022  1717 leader_election.cc:304] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:41.697964  9594 raft_consensus.cc:2804] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 13:02:41.698505  9594 raft_consensus.cc:697] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 20 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:02:41.698947  2330 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39726 (request call id 1730) took 2886 ms. Trace:
I20260709 13:02:41.699394  9594 consensus_queue.cc:237] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.12, Last appended by leader: 13, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:41.700565  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:41.701201  1598 raft_consensus.cc:3060] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:02:41.699361  2330 rpcz_store.cc:276] 0709 13:02:38.812646 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:38.812776 (+   130us) service_pool.cc:224] Handling call
0709 13:02:38.817589 (+  4813us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:40.800047 (+1982458us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:41.698624 (+898577us) spinlock_profiling.cc:243] Waited 1.98 s on lock 0x7b5400627ad4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:02:41.698839 (+   215us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4559343488}
I20260709 13:02:41.673244  9666 raft_consensus.cc:2749] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 13:02:41.707070  9517 leader_election.cc:290] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 22 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:41.707435  9450 leader_election.cc:290] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:02:41.707674  9644 log.cc:927] Time spent T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.293s	user 0.000s	sys 0.003s
I20260709 13:02:41.649737  9563 consensus_queue.cc:1048] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 8, Time since last communication: 0.028s
I20260709 13:02:41.709383  9501 consensus_queue.cc:1059] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 22.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 13:02:41.709951  1747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55848 (request call id 3480) took 1236 ms. Trace:
I20260709 13:02:41.670382  9606 raft_consensus.cc:493] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:41.703760  1646 leader_election.cc:304] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:41.710397  1747 rpcz_store.cc:276] 0709 13:02:40.473358 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:40.614341 (+140983us) service_pool.cc:224] Handling call
0709 13:02:40.614686 (+   345us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:40.615067 (+   381us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:40.615095 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:40.615567 (+   472us) log.cc:844] Serialized 26 byte log entry
0709 13:02:41.074564 (+458997us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:02:41.074626 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:41.074725 (+    99us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:41.706994 (+632269us) spinlock_profiling.cc:243] Waited 10.1 ms on lock 0x7b4400022b4c. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f4262f9bc85 00007f4262f9c6a2 00007f4265d0e1db 00007f4265d09aaf 00007f4265cfd86e 00007f4265cfcbb6 00007f4265cfc7d8 00007f4265d06221 00007f4265d061d9 00007f4265d06169
0709 13:02:41.709598 (+  2604us) raft_consensus.cc:1685] finished
0709 13:02:41.709622 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:41.709880 (+   258us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":79,"raft.run_cpu_time_us":132,"raft.run_wall_time_us":26338,"spinlock_wait_cycles":23123840,"thread_start_us":458518,"threads_started":1,"wal-append.queue_time_us":758055}
I20260709 13:02:41.718852  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:41.718989  2301 leader_election.cc:304] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:02:41.719842  1747 raft_consensus.cc:2393] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 20: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:02:41.692036  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:41.720793  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 19 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:41.674697  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:41.673065  9618 raft_consensus.cc:740] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:02:41.711244  9358 raft_consensus.cc:3055] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 22 LEADER]: Stepping down as leader of term 22
W20260709 13:02:41.777889  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 45547.5us
I20260709 13:02:41.778261  1602 raft_consensus.cc:2468] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 20.
I20260709 13:02:41.777164  1598 raft_consensus.cc:2468] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 22.
I20260709 13:02:41.780113  1748 raft_consensus.cc:1275] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 21 index: 15. Preceding OpId from leader: term: 24 index: 17. (index mismatch)
I20260709 13:02:41.781854  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 24 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:41.802906  1751 raft_consensus.cc:2393] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 26: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:02:41.734401  9606 raft_consensus.cc:3060] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:02:41.804114  1717 leader_election.cc:304] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:41.751389  9530 raft_consensus.cc:2749] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:02:41.808365  1717 leader_election.cc:304] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:41.779729  9648 raft_consensus.cc:2804] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Leader election won for term 20
I20260709 13:02:41.820282  1674 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57092 (request call id 1737) took 1470 ms. Trace:
I20260709 13:02:41.820782  1674 rpcz_store.cc:276] 0709 13:02:40.349729 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:40.350168 (+   439us) service_pool.cc:224] Handling call
0709 13:02:40.350482 (+   314us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:40.389374 (+ 38892us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:41.819924 (+1430550us) spinlock_profiling.cc:243] Waited 38.2 ms on lock 0x7b540025b054. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:02:41.820194 (+   270us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":87946112}
I20260709 13:02:41.757879  9645 raft_consensus.cc:3060] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Advancing to term 22
W20260709 13:02:41.828325  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 173275us
W20260709 13:02:41.809120  1571 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:41.753887  9426 raft_consensus.cc:493] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:41.833597  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:41.834312  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:41.835801  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:41.836781  1754 raft_consensus.cc:2393] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 22: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
W20260709 13:02:41.829993  1571 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:41.776079  9501 raft_consensus.cc:493] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:41.798015  9358 raft_consensus.cc:740] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:02:41.845156  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
W20260709 13:02:41.843371  1718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:41.751693  9666 consensus_queue.cc:1059] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 19.14, Next index: 15, Last known committed idx: 13, Time since last communication: 0.089s
I20260709 13:02:41.847869  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:41.862246  2332 raft_consensus.cc:2468] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 16.
I20260709 13:02:41.849061  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 18 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:41.845960  1754 raft_consensus.cc:3060] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:02:41.816186  8968 raft_consensus.cc:2804] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 13:02:41.861047  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 20 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:41.864461  2337 raft_consensus.cc:2368] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 20. Current term is 21.
I20260709 13:02:41.816443  9432 raft_consensus.cc:2804] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 13:02:41.875478  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:41.879825  2330 raft_consensus.cc:2393] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 20: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:02:41.882853  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:41.844939  9648 raft_consensus.cc:697] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:02:41.860919  9589 raft_consensus.cc:2804] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:02:41.898182  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:41.886447  1718 leader_election.cc:304] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:02:41.891970  8968 raft_consensus.cc:697] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 22 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:02:41.901443  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:41.904471  1674 raft_consensus.cc:2393] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 21: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:02:41.905849  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:02:41.895785  9628 log.cc:927] Time spent T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.060s	user 0.000s	sys 0.000s
I20260709 13:02:41.909734  1718 leader_election.cc:304] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 13:02:41.909112  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 286933us
I20260709 13:02:41.879673  9666 consensus_queue.cc:1243] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer af520b67945d443b901bbeca37cc5701 log is divergent from this leader: its last log entry 21.15 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 13
W20260709 13:02:41.895288  9665 log.cc:927] Time spent T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.052s	user 0.001s	sys 0.000s
I20260709 13:02:41.913821  1643 leader_election.cc:304] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:41.913151  1718 leader_election.cc:304] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:02:41.918924  1676 raft_consensus.cc:1240] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 16. Current term is 17. Ops: [16.14-16.14]
I20260709 13:02:41.923396  1754 raft_consensus.cc:2468] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 21.
I20260709 13:02:41.869592  9501 raft_consensus.cc:515] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:41.875604  9530 consensus_queue.cc:1059] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 22.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.034s
I20260709 13:02:41.931695  1718 leader_election.cc:304] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 13:02:41.925865  1643 leader_election.cc:304] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 13:02:41.927335  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:02:41.933540  1569 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:41.897117  9432 raft_consensus.cc:697] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 20 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:02:41.858754  9426 raft_consensus.cc:515] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:41.934823  1569 leader_election.cc:304] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:02:41.908453  9476 raft_consensus.cc:2749] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:02:41.918290  9594 raft_consensus.cc:2749] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:02:41.939767  1718 leader_election.cc:304] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:41.936354  9666 consensus_queue.cc:1048] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.007s
I20260709 13:02:41.940812  9648 consensus_queue.cc:237] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.11, Last appended by leader: 11, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:41.938576  9453 raft_consensus.cc:2764] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Leader pre-election decision vote started in defunct term 20: won
I20260709 13:02:41.953778  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:42.021884  2337 raft_consensus.cc:2393] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 17: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:02:41.981805  9501 leader_election.cc:290] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:41.947412  8968 consensus_queue.cc:237] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.13, Last appended by leader: 13, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:42.023181  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:41.930150  9513 raft_consensus.cc:2749] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:02:41.925381  9424 raft_consensus.cc:2749] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 13:02:42.032493  2328 raft_consensus.cc:2410] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for term 21 because replica has last-logged OpId of term: 20 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 14.
I20260709 13:02:41.947309  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:42.034418  2329 raft_consensus.cc:2410] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for term 21 because replica has last-logged OpId of term: 20 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 14.
I20260709 13:02:41.948433  2332 raft_consensus.cc:1240] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 17. Current term is 18. Ops: []
I20260709 13:02:41.949079  9432 consensus_queue.cc:237] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 19.11, Last appended by leader: 11, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:42.000965  9589 raft_consensus.cc:493] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:42.038559  9589 raft_consensus.cc:3060] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Advancing to term 20
W20260709 13:02:42.040736  9671 log.cc:927] Time spent T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.095s	user 0.000s	sys 0.003s
W20260709 13:02:42.042641  9670 log.cc:927] Time spent T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.104s	user 0.003s	sys 0.000s
W20260709 13:02:42.044159  9524 consensus_meta.cc:287] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.528s	user 0.008s	sys 0.005s
I20260709 13:02:42.044750  9524 raft_consensus.cc:515] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:02:41.969424  1571 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:42.050307  1571 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:42.023759  2331 raft_consensus.cc:1217] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Deduplicated request from leader. Original: 13.12->[14.13-16.14]   Dedup: 16.14->[]
I20260709 13:02:42.051606  2331 raft_consensus.cc:1240] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 16. Current term is 17. Ops: [14.13-16.14]
I20260709 13:02:41.987661  1746 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55848 (request call id 3481) took 1514 ms. Trace:
I20260709 13:02:42.003983  1747 raft_consensus.cc:1275] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 21 index: 16. Preceding OpId from leader: term: 23 index: 17. (index mismatch)
I20260709 13:02:42.052918  1746 rpcz_store.cc:276] 0709 13:02:40.473646 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:40.704469 (+230823us) service_pool.cc:224] Handling call
0709 13:02:40.704936 (+   467us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:41.386746 (+681810us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:41.987291 (+600545us) spinlock_profiling.cc:243] Waited 669 ms on lock 0x7b5400116cd4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:02:41.987566 (+   275us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1539095424}
I20260709 13:02:41.964658  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:42.044948  1494 catalog_manager.cc:5697] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 10 to 16, leader changed from 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194) to af520b67945d443b901bbeca37cc5701 (127.0.107.195). New cstate: current_term: 16 leader_uuid: "af520b67945d443b901bbeca37cc5701" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } }
I20260709 13:02:42.059038  2336 raft_consensus.cc:2393] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 26: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:02:41.965677  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
W20260709 13:02:42.068254  1571 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:42.072810  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 133212us
I20260709 13:02:42.074668  1718 leader_election.cc:304] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:41.973124  1569 leader_election.cc:304] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d
W20260709 13:02:42.054270  9620 log.cc:927] Time spent T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 1.066s	user 0.003s	sys 0.000s
W20260709 13:02:41.954934  9679 log.cc:927] Time spent T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.052s	user 0.000s	sys 0.000s
I20260709 13:02:42.084280  2334 raft_consensus.cc:1240] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 17 LEADER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 16. Current term is 17. Ops: []
I20260709 13:02:41.945340  9530 consensus_queue.cc:1243] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer 816fadcea8fc4893882f5f9b9f22a03d log is divergent from this leader: its last log entry 23.17 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 16
I20260709 13:02:41.947055  9437 raft_consensus.cc:2764] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 24 FOLLOWER]: Leader pre-election decision vote started in defunct term 23: won
I20260709 13:02:42.086234  1718 leader_election.cc:304] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 13:02:42.107235  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 134282us
I20260709 13:02:42.127688  1569 leader_election.cc:304] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:02:42.107596  1603 raft_consensus.cc:1275] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 19 index: 13. Preceding OpId from leader: term: 20 index: 14. (index mismatch)
I20260709 13:02:42.025822  9598 raft_consensus.cc:3055] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 LEADER]: Stepping down as leader of term 16
I20260709 13:02:41.985616  9515 leader_election.cc:290] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:02:42.119043  9681 log.cc:927] Time spent T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.069s	user 0.000s	sys 0.003s
I20260709 13:02:42.010762  9426 leader_election.cc:290] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:41.955320  9550 raft_consensus.cc:2749] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 24 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:02:41.977072  9609 consensus_queue.cc:1048] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.109s
I20260709 13:02:42.139240  2334 raft_consensus.cc:1240] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 20. Current term is 21. Ops: [18.15-20.16]
W20260709 13:02:42.144338  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 58208.2us
I20260709 13:02:42.106827  9214 raft_consensus.cc:2804] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 13:02:42.136034  9658 raft_consensus.cc:2749] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:02:42.155754  2334 raft_consensus.cc:1240] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 18 LEADER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 16. Current term is 18. Ops: []
I20260709 13:02:41.956063  9563 consensus_queue.cc:1048] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.177s
I20260709 13:02:42.103493  9524 leader_election.cc:290] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 24 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:42.107805  9529 raft_consensus.cc:2804] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:02:42.107468  9647 raft_consensus.cc:493] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:42.157195  1718 leader_election.cc:304] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:42.147269  9214 raft_consensus.cc:493] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:02:42.158034  9594 consensus_queue.cc:1048] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 13:02:42.160310  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:42.082094  9476 raft_consensus.cc:2749] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 13:02:42.155544  9636 raft_consensus.cc:493] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:42.132589  9598 raft_consensus.cc:740] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
W20260709 13:02:42.151489  9675 log.cc:927] Time spent T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.166s	user 0.000s	sys 0.006s
W20260709 13:02:42.264303  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 318719us
I20260709 13:02:42.175094  9529 raft_consensus.cc:493] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:42.076474  9634 raft_consensus.cc:493] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:42.107173  9513 consensus_queue.cc:1059] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 19.11, Next index: 12, Last known committed idx: 8, Time since last communication: 0.054s
I20260709 13:02:42.081768  9511 consensus_queue.cc:1048] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.068s
I20260709 13:02:42.155161  9426 raft_consensus.cc:2749] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:02:42.163457  9453 consensus_queue.cc:1048] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
W20260709 13:02:42.322314  9633 consensus_meta.cc:287] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.720s	user 0.010s	sys 0.011s
W20260709 13:02:42.353864  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 226266us
I20260709 13:02:42.315568  9214 raft_consensus.cc:3060] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Advancing to term 25
I20260709 13:02:42.361305  9598 consensus_queue.cc:260] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.10, Last appended by leader: 10, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:42.361987  9476 raft_consensus.cc:2804] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 13:02:42.371838  1752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55848 (request call id 3475) took 2166 ms. Trace:
I20260709 13:02:42.341493  9529 raft_consensus.cc:3060] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:02:42.378571  1750 raft_consensus.cc:1275] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 22 index: 16. Preceding OpId from leader: term: 23 index: 17. (index mismatch)
I20260709 13:02:42.340438  9636 raft_consensus.cc:515] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:02:42.384384  9483 consensus_meta.cc:287] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.688s	user 0.015s	sys 0.011s
I20260709 13:02:42.351774  9634 raft_consensus.cc:515] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:42.323123  9647 raft_consensus.cc:515] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:42.382025  2336 raft_consensus.cc:1275] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 15 index: 14. Preceding OpId from leader: term: 21 index: 17. (index mismatch)
I20260709 13:02:42.385301  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:42.392139  1569 leader_election.cc:400] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 20 pre-election: Vote denied by peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625) with higher term. Message: Invalid argument: T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 20. Current term is 21.
I20260709 13:02:42.460496  1569 leader_election.cc:403] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 20 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:02:42.388528  2301 leader_election.cc:304] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
W20260709 13:02:42.461403  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 73004.8us
I20260709 13:02:42.439206  1752 rpcz_store.cc:276] 0709 13:02:40.205480 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:40.206151 (+   671us) service_pool.cc:224] Handling call
0709 13:02:40.206538 (+   387us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:40.206962 (+   424us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:40.206999 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:40.209796 (+  2797us) log.cc:844] Serialized 26 byte log entry
0709 13:02:41.680734 (+1470938us) spinlock_profiling.cc:243] Waited 1.78 ms on lock 0x7b180037a740. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f4263057e07 00007f426beb2ab9 00007f426307469b 00007f426307293f 00007f4263074c02 00007f4265cfc20b 00007f4265d01d23 00007f4265d020d1 00007f426611da0f
0709 13:02:41.834964 (+154230us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:02:41.834999 (+    35us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:41.835231 (+   232us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:42.117609 (+282378us) spinlock_profiling.cc:243] Waited 35.3 ms on lock 0x7b4400022b4c. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f4262f9bc85 00007f4262f9c6a2 00007f4265d0e1db 00007f4265d09aaf 00007f4265cfd86e 00007f4265cfcbb6 00007f4265cfc7d8 00007f4265d06221 00007f4265d061d9 00007f4265d06169
0709 13:02:42.371536 (+253927us) raft_consensus.cc:1685] finished
0709 13:02:42.371563 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:42.371726 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2230,"raft.queue_time_us":2229,"raft.run_cpu_time_us":136,"raft.run_wall_time_us":6648,"spinlock_wait_cycles":85346304,"thread_start_us":154001,"threads_started":1,"wal-append.queue_time_us":1710752}
W20260709 13:02:42.444476  1570 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:42.382076  9606 consensus_meta.cc:287] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.557s	user 0.013s	sys 0.002s
I20260709 13:02:42.379721  9530 raft_consensus.cc:493] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:42.390849  9682 negotiation.cc:336] Failed RPC negotiation. Trace:
0709 13:02:41.675445 (+     0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:54246 (local address 127.0.107.193:46445)
0709 13:02:41.943231 (+267786us) server_negotiation.cc:207] Beginning negotiation
0709 13:02:41.943273 (+    42us) server_negotiation.cc:400] Waiting for connection header
0709 13:02:41.943328 (+    55us) server_negotiation.cc:408] Connection header received
0709 13:02:41.943545 (+   217us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0709 13:02:41.943575 (+    30us) server_negotiation.cc:462] Received NEGOTIATE request from client
0709 13:02:41.943907 (+   332us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0709 13:02:41.944426 (+   519us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54246: BlockingWrite error: write error: Broken pipe (error 32)
Metrics: {"server-negotiator.queue_time_us":266928,"thread_start_us":104858,"threads_started":1}
I20260709 13:02:42.446704  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:42.360272  9625 raft_consensus.cc:493] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:42.393415  9649 consensus_meta.cc:287] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.753s	user 0.008s	sys 0.009s
I20260709 13:02:42.457808  1748 raft_consensus.cc:1275] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 26 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 24 index: 15. Preceding OpId from leader: term: 26 index: 16. (index mismatch)
W20260709 13:02:42.457778  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 300823us
I20260709 13:02:42.383610  9226 consensus_queue.cc:1048] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.075s
I20260709 13:02:42.479642  1718 leader_election.cc:304] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:42.493623  1718 leader_election.cc:304] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:42.495702  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:02:42.496244  1571 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:42.496196  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:42.496596  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 22 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:42.497236  1671 raft_consensus.cc:2393] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 23: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:02:42.497795  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:42.498764  1675 raft_consensus.cc:2410] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for term 23 because replica has last-logged OpId of term: 22 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 14.
I20260709 13:02:42.458665  9476 raft_consensus.cc:697] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 21 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:02:42.494840  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:42.475227  9598 raft_consensus.cc:3060] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:02:42.475732  9668 raft_consensus.cc:2804] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 13:02:42.515800  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:42.466940  9658 raft_consensus.cc:493] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:42.508385  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:42.433408  9633 raft_consensus.cc:515] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:42.519873  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 53089.2us
I20260709 13:02:42.519938  9609 raft_consensus.cc:493] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:42.527333  9476 consensus_queue.cc:237] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.10, Last appended by leader: 10, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:42.476068  9606 raft_consensus.cc:515] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:42.521785  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:42.560779  1680 raft_consensus.cc:2468] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 22.
I20260709 13:02:42.523715  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 24 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:42.481822  9530 raft_consensus.cc:515] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:42.476960  9518 consensus_queue.cc:1048] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.067s
I20260709 13:02:42.468930  9625 raft_consensus.cc:515] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:42.561937  1678 raft_consensus.cc:2468] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 23.
W20260709 13:02:42.562454  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 68945.7us
I20260709 13:02:42.596334  1751 raft_consensus.cc:1217] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 21 LEADER]: Deduplicated request from leader. Original: 17.13->[19.14-20.15]   Dedup: 19.14->[20.15-20.15]
I20260709 13:02:42.597160  1751 raft_consensus.cc:1240] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 21 LEADER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 20. Current term is 21. Ops: [19.14-20.15]
I20260709 13:02:42.466657  9676 consensus_queue.cc:1048] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.050s
I20260709 13:02:42.458369  9483 raft_consensus.cc:515] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:42.508744  9424 raft_consensus.cc:2804] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 13:02:42.598412  1718 leader_election.cc:304] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:42.487551  9649 raft_consensus.cc:515] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:42.604063  1598 raft_consensus.cc:1240] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 19. Current term is 20. Ops: []
W20260709 13:02:42.604785  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 213047us
I20260709 13:02:42.620522  1672 raft_consensus.cc:2368] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 21. Current term is 22.
I20260709 13:02:42.621781  1569 leader_election.cc:304] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:02:42.627935  9606 leader_election.cc:290] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 22 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:42.571861  9645 consensus_queue.cc:1048] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 13, Time since last communication: 0.057s
I20260709 13:02:42.629558  9633 leader_election.cc:290] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 20 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:42.612066  9698 raft_consensus.cc:2749] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 26 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 13:02:42.571578  9609 raft_consensus.cc:515] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:42.637398  9618 consensus_queue.cc:260] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.15, Last appended by leader: 15, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:42.640347  9483 leader_election.cc:290] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 20 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 13:02:42.642498  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 44250.7us
I20260709 13:02:42.643435  1718 leader_election.cc:304] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:42.671818  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:42.672775  1750 raft_consensus.cc:2468] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 20.
I20260709 13:02:42.675033  2328 raft_consensus.cc:1240] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 19. Current term is 20. Ops: [19.16-19.16]
I20260709 13:02:42.586238  9668 raft_consensus.cc:697] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 21 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:02:42.634181  9609 leader_election.cc:290] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:42.704452  1747 raft_consensus.cc:1240] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 19. Current term is 20. Ops: []
W20260709 13:02:42.709923  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 66615.5us
I20260709 13:02:42.711362  2301 leader_election.cc:304] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:42.604456  9588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:34804 (request call id 11) took 4643 ms. Trace:
I20260709 13:02:42.737517  1677 raft_consensus.cc:1275] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 24 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 22 index: 14. Preceding OpId from leader: term: 24 index: 15. (index mismatch)
I20260709 13:02:42.738101  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:42.738849  1674 raft_consensus.cc:3060] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:02:42.613813  9666 raft_consensus.cc:493] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:42.746217  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:42.747468  1673 raft_consensus.cc:2410] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for term 20 because replica has last-logged OpId of term: 19 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 12.
I20260709 13:02:42.678714  9598 raft_consensus.cc:3060] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Advancing to term 21
I20260709 13:02:42.676052  9489 raft_consensus.cc:493] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:42.692700  9618 raft_consensus.cc:3060] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:02:42.759907  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:42.620162  9625 leader_election.cc:290] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:42.763547  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 23 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:42.717342  1750 raft_consensus.cc:2955] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Committing config change with OpId 22.15: config changed from index 5 to 15, VOTER 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) evicted. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 13:02:42.738693  9501 raft_consensus.cc:2804] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Leader pre-election won for term 22
W20260709 13:02:42.756705  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 135069us
I20260709 13:02:42.757925  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 24 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:42.761173  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:42.768977  1676 raft_consensus.cc:3060] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Advancing to term 24
I20260709 13:02:42.703974  9424 raft_consensus.cc:697] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [term 17 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:02:42.764328  1679 raft_consensus.cc:1275] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 19 index: 13. Preceding OpId from leader: term: 20 index: 14. (index mismatch)
W20260709 13:02:42.703603  9687 log.cc:927] Time spent T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.200s	user 0.001s	sys 0.001s
I20260709 13:02:42.565025  9658 raft_consensus.cc:515] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:42.738526  8968 raft_consensus.cc:2804] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 13:02:42.770182  1569 leader_election.cc:304] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
W20260709 13:02:42.771602  1571 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:42.738883  9693 raft_consensus.cc:2749] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:02:42.752790  9609 raft_consensus.cc:493] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:42.782541  1571 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:42.790225  1674 raft_consensus.cc:2468] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 22.
I20260709 13:02:42.782816  1569 leader_election.cc:304] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
W20260709 13:02:42.793321  1571 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:42.777364  9501 raft_consensus.cc:493] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:42.806411  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:42.807130  1597 raft_consensus.cc:2468] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 22.
I20260709 13:02:42.808880  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 23 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:42.798725  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:42.809240  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:42.810034  1605 raft_consensus.cc:2393] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 20: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:02:42.801343  1604 raft_consensus.cc:1275] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 20 index: 13. (index mismatch)
I20260709 13:02:42.812086  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:42.812811  1604 raft_consensus.cc:2468] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 16.
I20260709 13:02:42.813894  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:42.803956  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 24 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:42.804986  1601 raft_consensus.cc:3055] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 LEADER]: Stepping down as leader of term 21
I20260709 13:02:42.815714  1601 raft_consensus.cc:740] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:02:42.784158  9633 raft_consensus.cc:3060] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:02:42.821826  1599 raft_consensus.cc:1275] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 26 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 22 index: 14. Preceding OpId from leader: term: 26 index: 16. (index mismatch)
I20260709 13:02:42.844024  1599 raft_consensus.cc:1275] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 19 index: 9. Preceding OpId from leader: term: 21 index: 10. (index mismatch)
I20260709 13:02:42.840369  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 22 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:42.846170  1600 raft_consensus.cc:2468] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 21.
I20260709 13:02:42.841835  1597 raft_consensus.cc:1240] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 19. Current term is 20. Ops: [17.10-19.11]
I20260709 13:02:42.784734  8968 raft_consensus.cc:493] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:42.816699  1601 consensus_queue.cc:260] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 21.12, Last appended by leader: 12, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:42.848510  1601 raft_consensus.cc:3060] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:02:42.761658  9489 raft_consensus.cc:515] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:42.847608  9687 log.cc:927] Time spent T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.056s	user 0.001s	sys 0.000s
I20260709 13:02:42.759092  9666 raft_consensus.cc:515] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:42.857889  1678 raft_consensus.cc:2468] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 22.
I20260709 13:02:42.820595  9501 raft_consensus.cc:3060] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:02:42.867605  1679 raft_consensus.cc:1240] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 24 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 23. Current term is 24. Ops: [23.13-23.13]
I20260709 13:02:42.795737  9609 raft_consensus.cc:515] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:42.902895  1674 raft_consensus.cc:1275] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 19 index: 13. Preceding OpId from leader: term: 20 index: 13. (term mismatch)
W20260709 13:02:42.868072  1570 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:42.852066  8968 raft_consensus.cc:3060] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:02:42.909061  1601 raft_consensus.cc:1275] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 21 index: 12. Preceding OpId from leader: term: 22 index: 12. (term mismatch)
I20260709 13:02:42.877285  1753 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55840 (request call id 3256) took 2108 ms. Trace:
I20260709 13:02:42.909548  1753 rpcz_store.cc:276] 0709 13:02:40.769271 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:41.394099 (+624828us) service_pool.cc:224] Handling call
0709 13:02:41.410148 (+ 16049us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:02:41.469950 (+ 59802us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:41.469992 (+    42us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:02:41.479401 (+  9409us) log.cc:844] Serialized 52 byte log entry
0709 13:02:42.498673 (+1019272us) spinlock_profiling.cc:243] Waited 64.6 ms on lock 0x7b180040bdc0. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f4263057e07 00007f426beb2ab9 00007f426307469b 00007f426307293f 00007f4263074c02 00007f4265cfc20b 00007f4265d01d23 00007f4265d020d1 00007f426611da0f
0709 13:02:42.766203 (+267530us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:02:42.767239 (+  1036us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:42.767380 (+   141us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:42.876900 (+109520us) raft_consensus.cc:1685] finished
0709 13:02:42.876934 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:42.877198 (+   264us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":59223,"raft.queue_time_us":59870,"raft.run_cpu_time_us":145,"raft.run_wall_time_us":6921,"spinlock_wait_cycles":148499328,"thread_start_us":266684,"threads_started":1,"wal-append.queue_time_us":1378904}
I20260709 13:02:42.911041  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
W20260709 13:02:42.885934  9688 log.cc:927] Time spent T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.207s	user 0.000s	sys 0.001s
W20260709 13:02:42.870566  9695 log.cc:927] Time spent T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.107s	user 0.001s	sys 0.000s
I20260709 13:02:42.891252  9636 leader_election.cc:290] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:42.846602  9676 raft_consensus.cc:493] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:42.899422  1673 raft_consensus.cc:3055] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 19 LEADER]: Stepping down as leader of term 19
I20260709 13:02:42.904433  1674 pending_rounds.cc:85] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d: Aborting all ops after (but not including) 12
I20260709 13:02:42.830066  1676 raft_consensus.cc:2468] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 24.
I20260709 13:02:42.894155  1745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55848 (request call id 3477) took 2441 ms. Trace:
I20260709 13:02:42.913734  1673 raft_consensus.cc:740] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:02:42.913802  9676 raft_consensus.cc:515] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:42.913867  1674 pending_rounds.cc:107] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d: Aborting uncommitted NO_OP operation due to leader change: 19.13
I20260709 13:02:42.914774  1674 raft_consensus.cc:2889] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:02:42.911919  2333 raft_consensus.cc:2376] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 19. Re-sending same reply.
I20260709 13:02:42.914845  1673 consensus_queue.cc:260] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 19.11, Last appended by leader: 11, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:42.916275  1673 raft_consensus.cc:3060] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:02:42.916400  9676 leader_election.cc:290] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:42.914265  1745 rpcz_store.cc:276] 0709 13:02:40.453087 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:40.467263 (+ 14176us) service_pool.cc:224] Handling call
0709 13:02:40.482749 (+ 15486us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:02:40.557600 (+ 74851us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:40.557656 (+    56us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:02:42.370237 (+1812581us) spinlock_profiling.cc:243] Waited 53.4 ms on lock 0x7b2c000bc788. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4264a5ba11 00007f4264a6b372 00007f426619da01 00007f426616243a 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2
0709 13:02:42.370973 (+   736us) log.cc:844] Serialized 52 byte log entry
0709 13:02:42.497712 (+126739us) raft_consensus.cc:1645] Marking committed up to 16
0709 13:02:42.678163 (+180451us) log.cc:844] Serialized 14 byte log entry
0709 13:02:42.711975 (+ 33812us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:42.712058 (+    83us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:42.893729 (+181671us) raft_consensus.cc:1685] finished
0709 13:02:42.893869 (+   140us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:42.894067 (+   198us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":96014,"prepare.queue_time_us":137275,"prepare.run_cpu_time_us":28,"prepare.run_wall_time_us":29,"raft.queue_time_us":94046,"raft.run_cpu_time_us":446,"raft.run_wall_time_us":59160,"spinlock_wait_cycles":122892160,"thread_start_us":191344,"threads_started":2,"wal-append.queue_time_us":472009}
I20260709 13:02:42.916205  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:42.918486  1601 pending_rounds.cc:85] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a: Aborting all ops after (but not including) 11
I20260709 13:02:42.918826  1601 pending_rounds.cc:107] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a: Aborting uncommitted NO_OP operation due to leader change: 21.12
I20260709 13:02:42.919075  1601 raft_consensus.cc:2889] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:02:42.923471  8968 raft_consensus.cc:515] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:42.927448  1673 raft_consensus.cc:1275] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 19 index: 11. Preceding OpId from leader: term: 20 index: 12. (index mismatch)
I20260709 13:02:42.928483  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:42.928740  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:42.930590  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:42.932278  1600 raft_consensus.cc:1275] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 24 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 21 index: 15. Preceding OpId from leader: term: 24 index: 17. (index mismatch)
I20260709 13:02:42.935560  1600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:42.936432  1600 raft_consensus.cc:2468] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 16.
I20260709 13:02:42.939653  9647 leader_election.cc:290] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:02:43.084111  1644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:43.137351  1672 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52766 (request call id 3328) took 1485 ms (client timeout 1708 ms). Trace:
W20260709 13:02:43.137876  1672 rpcz_store.cc:269] 0709 13:02:41.651791 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:41.651973 (+   182us) service_pool.cc:224] Handling call
0709 13:02:43.137107 (+1485134us) spinlock_profiling.cc:243] Waited 966 ms on lock 0x7b5400394ed4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:02:43.137282 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2220891904}
I20260709 13:02:42.823869  9668 consensus_queue.cc:237] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 19.16, Last appended by leader: 16, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:42.951068  9634 leader_election.cc:290] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:02:43.191053  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 408495us
I20260709 13:02:43.130808  9676 raft_consensus.cc:493] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:42.950798  9530 leader_election.cc:290] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:43.192283  1569 leader_election.cc:304] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:02:43.149509  9647 raft_consensus.cc:493] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:02:42.739881  9588 rpcz_store.cc:276] 0709 13:02:37.928724 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:38.045789 (+117065us) service_pool.cc:224] Handling call
0709 13:02:39.388736 (+1342947us) spinlock_profiling.cc:243] Waited 128 ms on lock 0x7b4c00332c80. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426442ea20 00007f426443fa29 00007f4264405644 00007f426442cf2d 00007f426442cb93 00007f42660bb1af 00007f42660baf3e 00007f4266140888 00007f426615d75c 00007f426615e76a 00007f426615d91b
0709 13:02:39.410062 (+ 21326us) log.cc:844] Serialized 501 byte log entry
0709 13:02:42.571972 (+3161910us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":566746,"raft.queue_time_us":122464,"raft.run_cpu_time_us":1722,"raft.run_wall_time_us":122624,"spinlock_wait_cycles":294201216,"thread_start_us":212744,"threads_started":2,"wal-append.queue_time_us":265082,"wal-append.run_cpu_time_us":2152,"wal-append.run_wall_time_us":1059604}
I20260709 13:02:42.899330  9424 consensus_queue.cc:237] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.10, Last appended by leader: 10, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:43.207212  9609 leader_election.cc:290] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:43.208745  9658 leader_election.cc:290] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:43.209262  9648 raft_consensus.cc:493] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:43.205307  9666 leader_election.cc:290] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:02:43.229161  1748 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.685s	user 0.005s	sys 0.002s
W20260709 13:02:43.254853  1748 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.711s	user 0.005s	sys 0.002s
I20260709 13:02:43.199536  9645 raft_consensus.cc:493] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:43.260982  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 68843us
I20260709 13:02:43.199450  9704 raft_consensus.cc:2749] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:02:43.236193  9647 raft_consensus.cc:515] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:43.267205  1569 leader_election.cc:304] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
W20260709 13:02:43.263235  1718 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.542s	user 0.041s	sys 0.075s
W20260709 13:02:43.269296  1718 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.553s	user 0.042s	sys 0.075s
W20260709 13:02:43.269788  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 554381us
I20260709 13:02:43.220252  8968 leader_election.cc:290] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 23 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:43.270287  1569 leader_election.cc:304] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
W20260709 13:02:43.260053  9589 consensus_meta.cc:287] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 1.221s	user 0.015s	sys 0.004s
I20260709 13:02:43.228536  9649 leader_election.cc:290] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 20 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:43.260982  9609 raft_consensus.cc:493] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:43.261744  9658 raft_consensus.cc:493] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:43.287961  1569 leader_election.cc:304] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:02:43.222314  9676 raft_consensus.cc:515] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:43.293085  9529 consensus_meta.cc:287] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.828s	user 0.013s	sys 0.002s
W20260709 13:02:43.312544  9437 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 20 current term: 20 is higher.
I20260709 13:02:43.321709  9703 raft_consensus.cc:2749] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 13:02:43.272357  9645 raft_consensus.cc:515] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } }
I20260709 13:02:43.297942  9489 leader_election.cc:290] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:43.331350  1488 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 983621a6fc5b4c9ea0159a648dfe7a36 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 12)
I20260709 13:02:43.302271  9709 raft_consensus.cc:2749] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
W20260709 13:02:43.334722  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:43.335325  1598 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40154 (request call id 3090) took 1641 ms. Trace:
I20260709 13:02:43.367172  1598 rpcz_store.cc:276] 0709 13:02:41.693506 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:41.780892 (+ 87386us) service_pool.cc:224] Handling call
0709 13:02:41.781264 (+   372us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:42.620963 (+839699us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:43.335017 (+714054us) spinlock_profiling.cc:243] Waited 823 ms on lock 0x7b5400215cd4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:02:43.335242 (+   225us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1892566784}
W20260709 13:02:43.367738  9618 consensus_meta.cc:287] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.602s	user 0.008s	sys 0.005s
I20260709 13:02:43.367791  1569 leader_election.cc:304] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:02:43.339253  9647 leader_election.cc:290] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:43.263399  9648 raft_consensus.cc:515] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:43.314236  9598 consensus_meta.cc:287] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.556s	user 0.013s	sys 0.000s
I20260709 13:02:43.330541  9704 consensus_queue.cc:1059] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.001s
W20260709 13:02:43.359292  1599 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.514s	user 0.019s	sys 0.011s
W20260709 13:02:43.359704  1597 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.510s	user 0.008s	sys 0.017s
I20260709 13:02:43.302721  9721 raft_consensus.cc:493] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:43.284709  9589 raft_consensus.cc:515] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:02:43.398895  1597 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.549s	user 0.008s	sys 0.017s
I20260709 13:02:43.369268  1598 raft_consensus.cc:1240] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 20. Current term is 21. Ops: [20.15-20.15]
W20260709 13:02:43.402096  1599 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.556s	user 0.019s	sys 0.012s
I20260709 13:02:43.369954  2335 raft_consensus.cc:1275] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 20 index: 15. Preceding OpId from leader: term: 21 index: 15. (term mismatch)
I20260709 13:02:43.404400  2335 pending_rounds.cc:85] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1: Aborting all ops after (but not including) 14
I20260709 13:02:43.404675  2335 pending_rounds.cc:107] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1: Aborting uncommitted NO_OP operation due to leader change: 20.15
I20260709 13:02:43.404923  2335 raft_consensus.cc:2889] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:02:43.308952  9720 raft_consensus.cc:2749] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 23 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260709 13:02:43.409689  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:43.418538  1488 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 138547us
I20260709 13:02:43.303251  9658 raft_consensus.cc:515] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:02:43.444254  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 76568.8us
I20260709 13:02:43.299160  9609 raft_consensus.cc:515] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:43.344389  9529 raft_consensus.cc:515] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:43.407760  9724 raft_consensus.cc:3060] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 25 FOLLOWER]: Advancing to term 26
I20260709 13:02:43.458604  1604 raft_consensus.cc:2468] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 16.
I20260709 13:02:43.423880  9704 consensus_queue.cc:1243] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer 3b879799899a499c8d409ee9f1277cd1 log is divergent from this leader: its last log entry 16.12 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 11
I20260709 13:02:43.445499  9647 raft_consensus.cc:493] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:43.422392  9703 raft_consensus.cc:493] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 23 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:02:43.413334  9723 raft_consensus.cc:2749] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 24 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 13:02:43.415112  9598 raft_consensus.cc:2749] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Leader pre-election lost for term 20. Reason: Vote denied by peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625) with higher term. Message: Invalid argument: T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 20. Current term is 21.
W20260709 13:02:43.459522  1718 connection.cc:256] RPC call timeout handler was delayed by -2.2238381380000192s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2515s for a response
W20260709 13:02:43.498224  1718 connection.cc:256] RPC call timeout handler was delayed by -1.6811094830000002s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1514s for a response
I20260709 13:02:43.481546  1569 leader_election.cc:304] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
W20260709 13:02:43.499678  1571 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.649s	user 0.000s	sys 0.015s
W20260709 13:02:43.500030  1571 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.649s	user 0.000s	sys 0.015s
I20260709 13:02:43.408985  9721 raft_consensus.cc:515] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:43.498857  1718 connection.cc:256] RPC call timeout handler was delayed by -1.5956094829999756s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16090000000000002s for a response
W20260709 13:02:43.505769  1718 connection.cc:256] RPC call timeout handler was delayed by -1.5065094830000021s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1708s for a response
W20260709 13:02:43.506476  1718 connection.cc:256] RPC call timeout handler was delayed by -1.4903094829999759s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1726s for a response
W20260709 13:02:43.507054  1718 connection.cc:256] RPC call timeout handler was delayed by -1.4903094829999759s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1726s for a response
W20260709 13:02:43.507575  1718 connection.cc:256] RPC call timeout handler was delayed by -1.4714094830000022s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17470000000000002s for a response
W20260709 13:02:43.507969  1718 connection.cc:256] RPC call timeout handler was delayed by -1.4714094830000022s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17470000000000002s for a response
W20260709 13:02:43.508349  1718 connection.cc:256] RPC call timeout handler was delayed by -1.4273094829999877s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1796s for a response
W20260709 13:02:43.508736  1718 connection.cc:256] RPC call timeout handler was delayed by -1.3994094829999995s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1827s for a response
W20260709 13:02:43.509121  1718 connection.cc:256] RPC call timeout handler was delayed by -1.3904094829999849s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1837s for a response
W20260709 13:02:43.509496  1718 connection.cc:256] RPC call timeout handler was delayed by -1.3904094829999849s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1837s for a response
W20260709 13:02:43.509897  1718 connection.cc:256] RPC call timeout handler was delayed by -1.2671094830000129s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19740000000000002s for a response
W20260709 13:02:43.515004  1570 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:43.521953  1601 raft_consensus.cc:2468] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 21.
I20260709 13:02:43.499605  9513 raft_consensus.cc:493] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:43.497926  9719 log.cc:927] Time spent T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.091s	user 0.001s	sys 0.000s
I20260709 13:02:43.502864  9437 raft_consensus.cc:493] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:43.474298  9704 raft_consensus.cc:493] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:43.523375  9723 raft_consensus.cc:2749] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20260709 13:02:43.510284  1718 connection.cc:256] RPC call timeout handler was delayed by -1.2275094830000057s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20179999999999998s for a response
W20260709 13:02:43.539942  2332 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.685s	user 0.011s	sys 0.017s
I20260709 13:02:43.539677  9598 raft_consensus.cc:2749] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 13:02:43.545010  9437 raft_consensus.cc:515] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:43.549760  1646 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:43.496371  9489 raft_consensus.cc:493] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:43.528694  9722 raft_consensus.cc:3055] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 LEADER]: Stepping down as leader of term 17
I20260709 13:02:43.495697  9647 raft_consensus.cc:515] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:02:43.552580  2332 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.702s	user 0.012s	sys 0.017s
I20260709 13:02:43.565150  2332 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39726 (request call id 1756) took 1182 ms. Trace:
W20260709 13:02:43.552420  1718 connection.cc:256] RPC call timeout handler was delayed by -1.1631284140000275s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2965s for a response
W20260709 13:02:43.541870  9214 consensus_meta.cc:287] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 1.085s	user 0.011s	sys 0.004s
I20260709 13:02:43.535156  9609 leader_election.cc:290] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:43.499215  9703 raft_consensus.cc:515] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:02:43.566964  1718 connection.cc:256] RPC call timeout handler was delayed by -1.1375094829999739s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2118s for a response
W20260709 13:02:43.567272  2298 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:43.570741  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
W20260709 13:02:43.588805  1718 connection.cc:256] RPC call timeout handler was delayed by -1.1060094829999798s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21530000000000002s for a response
I20260709 13:02:43.590723  1718 leader_election.cc:304] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:02:43.593950  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:43.595968  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:43.600651  1605 raft_consensus.cc:2468] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 19.
I20260709 13:02:43.558130  9542 raft_consensus.cc:493] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:43.558540  9723 raft_consensus.cc:493] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:43.558310  9728 raft_consensus.cc:493] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:43.559895  9722 raft_consensus.cc:740] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:02:43.558243  9560 raft_consensus.cc:493] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:43.555416  9589 leader_election.cc:290] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 20 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:43.544119  9513 raft_consensus.cc:515] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:43.623765  9647 leader_election.cc:290] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:43.598938  9476 raft_consensus.cc:2749] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:02:43.616050  9728 raft_consensus.cc:515] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:43.630000  9658 leader_election.cc:290] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:43.631413  9437 leader_election.cc:290] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:02:43.635368  9696 log.cc:927] Time spent T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 1.132s	user 0.003s	sys 0.003s
I20260709 13:02:43.638872  9588 raft_consensus.cc:493] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:02:43.639508  9588 raft_consensus.cc:515] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:43.641052  9703 leader_election.cc:290] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:43.641711  9588 leader_election.cc:290] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:43.643091  2298 leader_election.cc:304] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:02:43.644042  9714 raft_consensus.cc:493] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:43.644903  9710 raft_consensus.cc:2749] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:02:43.645741  9710 consensus_queue.cc:1048] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20260709 13:02:43.586683  9214 raft_consensus.cc:515] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:02:43.647856  2337 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49082 (request call id 1383) took 1750 ms (client timeout 2027 ms). Trace:
I20260709 13:02:43.648465  9214 leader_election.cc:290] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 25 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:43.563290  9489 raft_consensus.cc:515] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:43.650207  9501 consensus_meta.cc:287] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.709s	user 0.007s	sys 0.003s
I20260709 13:02:43.623920  9513 leader_election.cc:290] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:43.600700  1598 raft_consensus.cc:2468] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 21.
I20260709 13:02:43.644637  9714 raft_consensus.cc:515] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:43.686689  1569 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:43.691009  1747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43156 (request call id 1607) took 1152 ms. Trace:
I20260709 13:02:43.691553  1747 rpcz_store.cc:276] 0709 13:02:42.538949 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:42.541150 (+  2201us) service_pool.cc:224] Handling call
0709 13:02:42.541517 (+   367us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:42.705092 (+163575us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:43.690668 (+985576us) spinlock_profiling.cc:243] Waited 163 ms on lock 0x7b54000e3b54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:02:43.690919 (+   251us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":374222592}
I20260709 13:02:43.650614  9529 leader_election.cc:290] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 16 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:43.652343  9722 consensus_queue.cc:260] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:43.653100  2334 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39726 (request call id 1751) took 1272 ms. Trace:
I20260709 13:02:43.644897  9721 leader_election.cc:290] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:43.608215  9732 consensus_queue.cc:1059] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 19.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.014s
I20260709 13:02:43.654893  9489 leader_election.cc:290] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:02:43.648320  2337 rpcz_store.cc:269] 0709 13:02:41.897754 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:41.897935 (+   181us) service_pool.cc:224] Handling call
0709 13:02:43.647644 (+1749709us) spinlock_profiling.cc:243] Waited 59.1 ms on lock 0x7b54006beed4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:02:43.647787 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":135998336}
I20260709 13:02:43.610333  9645 leader_election.cc:290] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:43.655831  9588 consensus_queue.cc:1048] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 13:02:43.657416  2336 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39726 (request call id 1760) took 1106 ms. Trace:
I20260709 13:02:43.726296  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:43.658262  9560 raft_consensus.cc:515] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:43.603335  1718 leader_election.cc:304] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:43.600013  9676 leader_election.cc:290] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:43.553050  9704 raft_consensus.cc:515] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:43.659221  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 112494us
I20260709 13:02:43.734331  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 25 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:43.735909  1745 raft_consensus.cc:1275] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 18 index: 11. Preceding OpId from leader: term: 20 index: 12. (index mismatch)
I20260709 13:02:43.737298  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:43.661583  9530 consensus_queue.cc:1048] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 13:02:43.662286  2331 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39726 (request call id 1761) took 1023 ms. Trace:
I20260709 13:02:43.663671  9358 consensus_queue.cc:260] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 22.16, Last appended by leader: 16, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:43.668051  1746 raft_consensus.cc:1240] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 20 LEADER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 19. Current term is 20. Ops: []
I20260709 13:02:43.722385  9658 raft_consensus.cc:493] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:43.669401  1645 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:43.670404  1494 catalog_manager.cc:5697] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 13 to 17, leader changed from 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) to af520b67945d443b901bbeca37cc5701 (127.0.107.195). New cstate: current_term: 17 leader_uuid: "af520b67945d443b901bbeca37cc5701" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } }
I20260709 13:02:43.758040  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:02:43.674001  9702 log.cc:927] Time spent T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.494s	user 0.001s	sys 0.000s
W20260709 13:02:43.674911  9733 log.cc:927] Time spent T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.070s	user 0.001s	sys 0.000s
W20260709 13:02:43.677631  9633 consensus_meta.cc:287] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.824s	user 0.009s	sys 0.004s
I20260709 13:02:43.678915  9728 leader_election.cc:290] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:43.602015  9618 raft_consensus.cc:493] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:43.776126  1678 raft_consensus.cc:1275] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 21 index: 16. Preceding OpId from leader: term: 23 index: 17. (index mismatch)
W20260709 13:02:43.698717  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:43.699100  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:43.638742  9668 raft_consensus.cc:493] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:43.712625  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 22 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:43.780761  1597 raft_consensus.cc:3060] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:02:43.709322  2334 rpcz_store.cc:276] 0709 13:02:42.380789 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:42.382555 (+  1766us) service_pool.cc:224] Handling call
0709 13:02:42.382960 (+   405us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:42.383249 (+   289us) raft_consensus.cc:1534] Early marking committed up to index 15
0709 13:02:42.485633 (+102384us) log.cc:844] Serialized 14 byte log entry
0709 13:02:42.852067 (+366434us) log.cc:844] Serialized 14 byte log entry
0709 13:02:43.649923 (+797856us) spinlock_profiling.cc:243] Waited 3.17 ms on lock 0x7b18004a02c0. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f4262fefc0a 00007f426c72f219 00007f4263077a40 00007f42630724e4 00007f4263072455 00007f4269b5a3c7 00007f426616c9a7 00007f426617c0bf 00007f426617c038
0709 13:02:43.650152 (+   229us) log.cc:844] Serialized 14 byte log entry
0709 13:02:43.652601 (+  2449us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:43.652665 (+    64us) raft_consensus.cc:1645] Marking committed up to 15
0709 13:02:43.652732 (+    67us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:43.652807 (+    75us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:43.653023 (+   216us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":876779,"prepare.queue_time_us":2155294,"prepare.run_cpu_time_us":93,"prepare.run_wall_time_us":95,"raft.queue_time_us":14662,"raft.run_cpu_time_us":378,"raft.run_wall_time_us":7806,"spinlock_wait_cycles":7299328,"thread_start_us":330973,"threads_started":2,"wal-append.queue_time_us":391396}
I20260709 13:02:43.715765  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 22 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:43.726969  1750 raft_consensus.cc:2468] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 22.
I20260709 13:02:43.726577  2336 rpcz_store.cc:276] 0709 13:02:42.551203 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:42.551352 (+   149us) service_pool.cc:224] Handling call
0709 13:02:42.551723 (+   371us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:42.552003 (+   280us) raft_consensus.cc:1534] Early marking committed up to index 15
0709 13:02:42.759880 (+207877us) log.cc:844] Serialized 14 byte log entry
0709 13:02:43.656823 (+896943us) log.cc:844] Serialized 14 byte log entry
0709 13:02:43.657042 (+   219us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:43.657102 (+    60us) raft_consensus.cc:1645] Marking committed up to 15
0709 13:02:43.657152 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:43.657229 (+    77us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:43.657348 (+   119us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":738190,"prepare.queue_time_us":1837070,"prepare.run_cpu_time_us":73,"prepare.run_wall_time_us":75,"raft.queue_time_us":5938,"raft.run_cpu_time_us":246,"raft.run_wall_time_us":247,"thread_start_us":322307,"threads_started":2,"wal-append.queue_time_us":523610}
I20260709 13:02:43.730854  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:02:43.731956  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 128777us
I20260709 13:02:43.754756  9214 raft_consensus.cc:493] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:43.783406  2299 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:43.783623  1718 leader_election.cc:304] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:02:43.738296  1745 raft_consensus.cc:2393] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 23: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:02:43.708724  9741 raft_consensus.cc:493] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:02:43.715943  9703 raft_consensus.cc:493] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:43.668099  1752 raft_consensus.cc:1275] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 16 index: 7. Preceding OpId from leader: term: 21 index: 10. (index mismatch)
W20260709 13:02:43.754401  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 194310us
I20260709 13:02:43.755793  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 24 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:43.791812  1718 leader_election.cc:304] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:43.792464  1602 raft_consensus.cc:2410] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for term 24 because replica has last-logged OpId of term: 23 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 16.
I20260709 13:02:43.756309  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 25 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:43.793334  1753 raft_consensus.cc:2393] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 25: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:02:43.727846  9722 raft_consensus.cc:3060] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:02:43.756944  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 21 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:43.760869  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:43.761408  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:43.797683  1671 raft_consensus.cc:2393] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 24: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:02:43.763082  1672 raft_consensus.cc:1275] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 16 index: 13. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
I20260709 13:02:43.759915  1751 raft_consensus.cc:2468] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 26.
I20260709 13:02:43.767002  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:43.768071  1674 raft_consensus.cc:1275] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 21 index: 16. Preceding OpId from leader: term: 23 index: 17. (index mismatch)
I20260709 13:02:43.769842  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 22 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:43.798476  1673 raft_consensus.cc:2468] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 26.
I20260709 13:02:43.798966  1605 raft_consensus.cc:2468] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 21.
I20260709 13:02:43.771538  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:43.608482  9723 raft_consensus.cc:515] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:43.748209  2331 rpcz_store.cc:276] 0709 13:02:42.639399 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:42.640160 (+   761us) service_pool.cc:224] Handling call
0709 13:02:42.640639 (+   479us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 13:02:42.641122 (+   483us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 13:02:42.796205 (+155083us) log.cc:844] Serialized 14 byte log entry
0709 13:02:43.259479 (+463274us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 13:02:43.272906 (+ 13427us) log.cc:844] Serialized 78 byte log entry
0709 13:02:43.273278 (+   372us) raft_consensus.cc:1645] Marking committed up to 15
0709 13:02:43.659328 (+386050us) log.cc:844] Serialized 14 byte log entry
0709 13:02:43.661844 (+  2516us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:43.661948 (+   104us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:43.661988 (+    40us) raft_consensus.cc:1685] finished
0709 13:02:43.661998 (+    10us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:43.662201 (+   203us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":387817,"prepare.queue_time_us":1397592,"prepare.run_cpu_time_us":64,"prepare.run_wall_time_us":71,"raft.queue_time_us":9332,"raft.run_cpu_time_us":361,"raft.run_wall_time_us":11500,"thread_start_us":547445,"threads_started":2,"wal-append.queue_time_us":673417}
I20260709 13:02:43.711658  9501 raft_consensus.cc:515] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:43.754557  9732 raft_consensus.cc:2804] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 13:02:43.783775  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 19 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:43.606525  9542 raft_consensus.cc:515] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:43.771759  9358 raft_consensus.cc:3060] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:02:43.777151  9633 raft_consensus.cc:2749] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:02:43.815236  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:43.803296  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:43.819242  1671 raft_consensus.cc:1275] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 14 index: 13. Preceding OpId from leader: term: 17 index: 15. (index mismatch)
I20260709 13:02:43.768332  9658 raft_consensus.cc:515] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:43.820932  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:43.821221  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:43.821579  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:43.821866  1676 raft_consensus.cc:2468] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 16.
I20260709 13:02:43.817607  1673 raft_consensus.cc:2955] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Committing config change with OpId 22.15: config changed from index 5 to 15, VOTER 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) evicted. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 13:02:43.830868  1672 raft_consensus.cc:2468] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 17.
W20260709 13:02:43.833425  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:43.824375  1675 raft_consensus.cc:1275] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 24 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 22 index: 17. Preceding OpId from leader: term: 24 index: 18. (index mismatch)
I20260709 13:02:43.802983  9589 raft_consensus.cc:2804] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:02:43.795948  9647 raft_consensus.cc:2804] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 13:02:43.786131  9618 raft_consensus.cc:515] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:43.830225  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 112837us
I20260709 13:02:43.824537  1598 raft_consensus.cc:2468] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 18.
I20260709 13:02:43.836627  1675 raft_consensus.cc:1275] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 24 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 22 index: 16. Preceding OpId from leader: term: 24 index: 17. (index mismatch)
I20260709 13:02:43.819120  1718 leader_election.cc:304] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:43.838558  1678 raft_consensus.cc:1275] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 16 index: 14. Preceding OpId from leader: term: 18 index: 15. (index mismatch)
I20260709 13:02:43.794801  9214 raft_consensus.cc:515] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:43.837836  9589 raft_consensus.cc:493] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:43.854007  9589 raft_consensus.cc:3060] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:02:43.854621  9714 leader_election.cc:290] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:43.855868  9560 leader_election.cc:290] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:43.817845  9645 consensus_queue.cc:1048] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.035s
W20260709 13:02:43.856952  1569 leader_election.cc:336] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 17 pre-election: RPC error from VoteRequest() call to peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625): Timed out: RequestConsensusVote RPC to 127.0.107.196:46625 timed out after 2.027s (SENT)
I20260709 13:02:43.857460  1569 leader_election.cc:304] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:02:43.793921  9741 raft_consensus.cc:515] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:43.792008  9668 raft_consensus.cc:515] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:43.859643  9741 leader_election.cc:290] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:43.859709  9668 leader_election.cc:290] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:43.861954  9618 leader_election.cc:290] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:43.853634  1718 leader_election.cc:304] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:43.864283  9658 leader_election.cc:290] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:43.866218  9501 leader_election.cc:290] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 22 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:43.863094  1597 raft_consensus.cc:2468] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 22.
I20260709 13:02:43.868345  1674 raft_consensus.cc:1240] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 17 LEADER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 16. Current term is 17. Ops: []
I20260709 13:02:43.867923  9424 raft_consensus.cc:2804] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 13:02:43.869693  9424 raft_consensus.cc:493] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:43.870922  1674 raft_consensus.cc:1240] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 24 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 22. Current term is 24. Ops: []
I20260709 13:02:43.871059  9658 raft_consensus.cc:2749] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:02:43.871114  9424 raft_consensus.cc:3060] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:02:43.870162  9589 raft_consensus.cc:515] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:43.873454  1674 raft_consensus.cc:1275] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 20 index: 11. Preceding OpId from leader: term: 22 index: 12. (index mismatch)
I20260709 13:02:43.843152  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:43.815243  9732 raft_consensus.cc:493] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:43.878782  9732 raft_consensus.cc:3060] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 26 FOLLOWER]: Advancing to term 27
I20260709 13:02:43.880483  1676 raft_consensus.cc:1240] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 23 LEADER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 22. Current term is 23. Ops: []
I20260709 13:02:43.882093  1676 raft_consensus.cc:1240] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 LEADER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 16. Current term is 17. Ops: []
W20260709 13:02:43.883193  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 228148us
I20260709 13:02:43.884873  9704 leader_election.cc:290] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:43.885023  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 21 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:43.895929  1676 raft_consensus.cc:1240] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 21 LEADER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 20. Current term is 21. Ops: []
I20260709 13:02:43.888238  1646 leader_election.cc:304] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:43.889132  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:43.891352  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 20 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:43.892382  1679 raft_consensus.cc:1240] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 24 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 22. Current term is 24. Ops: []
I20260709 13:02:43.796036  9703 raft_consensus.cc:515] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:43.892926  8968 raft_consensus.cc:2804] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 13:02:43.893618  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:43.902709  1747 raft_consensus.cc:1275] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 20 index: 15. Preceding OpId from leader: term: 22 index: 16. (index mismatch)
W20260709 13:02:43.923498  2299 leader_election.cc:336] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Timed out: RequestConsensusVote RPC to 127.0.107.194:46261 timed out after 1.878s (SENT)
I20260709 13:02:43.894644  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:43.897945  1597 raft_consensus.cc:2468] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 20.
I20260709 13:02:43.887439  9226 consensus_queue.cc:1048] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
W20260709 13:02:43.935103  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 81602.1us
I20260709 13:02:43.935209  9704 raft_consensus.cc:493] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:43.945926  1718 leader_election.cc:304] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:43.946157  1680 raft_consensus.cc:2468] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 20.
I20260709 13:02:43.948511  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:43.950603  1672 raft_consensus.cc:2393] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 21: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:02:43.934906  9589 leader_election.cc:290] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 21 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:43.953864  1674 raft_consensus.cc:2410] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for term 20 because replica has last-logged OpId of term: 19 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 9.
W20260709 13:02:43.952886  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 64760.2us
I20260709 13:02:43.839852  9647 raft_consensus.cc:493] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:43.934628  8968 raft_consensus.cc:493] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
W20260709 13:02:43.962217  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 61549.1us
I20260709 13:02:43.964443  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 22 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:43.965164  1750 raft_consensus.cc:2468] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 21.
I20260709 13:02:43.968516  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:43.969362  1676 raft_consensus.cc:2368] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 23. Current term is 24.
I20260709 13:02:43.971352  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 21 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:43.963783  8968 raft_consensus.cc:3060] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:02:44.015267  1748 raft_consensus.cc:2410] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for term 21 because replica has last-logged OpId of term: 18 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 11.
I20260709 13:02:43.970199  1671 raft_consensus.cc:2468] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 22.
I20260709 13:02:43.956830  9489 raft_consensus.cc:2804] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 13:02:44.025801  1675 raft_consensus.cc:2468] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 23.
I20260709 13:02:43.977846  1750 raft_consensus.cc:1275] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 19 index: 11. Preceding OpId from leader: term: 22 index: 12. (index mismatch)
I20260709 13:02:44.019964  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 19 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:44.026988  1597 raft_consensus.cc:2468] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 18.
I20260709 13:02:44.020357  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 24 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:43.964025  9647 raft_consensus.cc:3060] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:02:44.034482  1597 raft_consensus.cc:1275] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 18 index: 11. Preceding OpId from leader: term: 20 index: 12. (index mismatch)
I20260709 13:02:44.034825  9757 raft_consensus.cc:2804] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Leader election won for term 19
W20260709 13:02:44.037078  9752 log.cc:927] Time spent T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.092s	user 0.000s	sys 0.000s
I20260709 13:02:44.039130  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 21 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:44.023325  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 25 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:44.042419  1671 raft_consensus.cc:2468] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 24.
W20260709 13:02:44.016942  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 50362.2us
W20260709 13:02:44.039224  9749 log.cc:927] Time spent T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.074s	user 0.000s	sys 0.000s
I20260709 13:02:44.044221  2298 leader_election.cc:304] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:02:44.027652  1748 raft_consensus.cc:2468] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 23.
I20260709 13:02:44.045689  1748 raft_consensus.cc:1240] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 19. Current term is 20. Ops: []
W20260709 13:02:44.047056  9672 log.cc:927] Time spent T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 2.070s	user 0.005s	sys 0.010s
I20260709 13:02:44.048923  1749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55848 (request call id 3478) took 3583 ms. Trace:
W20260709 13:02:44.058727  1570 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:43.949548  9704 raft_consensus.cc:515] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:44.042550  9647 raft_consensus.cc:515] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:44.062027  9704 leader_election.cc:290] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:43.970577  9750 consensus_queue.cc:1048] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.097s
I20260709 13:02:44.085502  2298 leader_election.cc:304] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:02:44.087708  1671 raft_consensus.cc:1240] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 16. Current term is 17. Ops: []
I20260709 13:02:44.088201  2298 leader_election.cc:304] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
W20260709 13:02:44.089680  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 143909us
I20260709 13:02:44.090853  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:44.026808  9489 raft_consensus.cc:493] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:02:44.066080  2300 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:44.066991  9723 leader_election.cc:290] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:02:44.071216  9754 log.cc:927] Time spent T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.114s	user 0.000s	sys 0.000s
I20260709 13:02:44.049407  1749 rpcz_store.cc:276] 0709 13:02:40.465331 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:40.466156 (+   825us) service_pool.cc:224] Handling call
0709 13:02:40.466607 (+   451us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:02:40.532168 (+ 65561us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:40.532221 (+    53us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:02:41.460378 (+928157us) spinlock_profiling.cc:243] Waited 7.28 ms on lock 0x7b4800121ed8. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426b08baa1 00007f426b082b37 00007f426b0829ca 00007f426b17a26f 00007f426b179f37 00007f426b18e55c 00007f426b18e508 00007f426b18e488 00007f426b18e443 00007f426b18d6ec 00007f426a0b7a96
0709 13:02:41.505006 (+ 44628us) spinlock_profiling.cc:243] Waited 79.1 ms on lock 0x7b2c000bc788. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4264a5ba11 00007f4264a6b372 00007f426619da01 00007f426616243a 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2
0709 13:02:41.505945 (+   939us) log.cc:844] Serialized 52 byte log entry
0709 13:02:41.854707 (+348762us) raft_consensus.cc:1645] Marking committed up to 13
0709 13:02:41.854753 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:41.854830 (+    77us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:43.956984 (+2102154us) spinlock_profiling.cc:243] Waited 37.3 ms on lock 0x7b4400022b4c. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f4262f9bc85 00007f4262f9c6a2 00007f4265d0e1db 00007f4265d09aaf 00007f4265cfd86e 00007f4265cfcbb6 00007f4265cfc7d8 00007f4265d06221 00007f4265d061d9 00007f4265d06169
0709 13:02:44.048610 (+ 91626us) raft_consensus.cc:1685] finished
0709 13:02:44.048646 (+    36us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:44.048838 (+   192us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":57658,"raft.queue_time_us":65550,"raft.run_cpu_time_us":3568,"raft.run_wall_time_us":919217,"spinlock_wait_cycles":284287872,"thread_start_us":348195,"threads_started":1,"wal-append.queue_time_us":427495}
I20260709 13:02:44.103783  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:44.104422  1602 raft_consensus.cc:3060] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:02:44.095153  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:44.097604  1605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 22 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:44.107502  1605 raft_consensus.cc:2468] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 21.
I20260709 13:02:44.092209  1494 catalog_manager.cc:5697] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 10 to 18. New cstate: current_term: 18 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } }
I20260709 13:02:44.100035  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 22 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:44.109127  1599 raft_consensus.cc:2410] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 for term 22 because replica has last-logged OpId of term: 20 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 13.
W20260709 13:02:44.110954  1569 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:44.112591  1671 raft_consensus.cc:1240] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 19. Current term is 20. Ops: []
I20260709 13:02:44.090039  8994 raft_consensus.cc:2804] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Leader pre-election won for term 18
W20260709 13:02:44.125860  1570 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:44.063542  9647 leader_election.cc:290] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 22 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:44.099334  9710 raft_consensus.cc:2804] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:02:44.096783  9542 leader_election.cc:290] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:02:44.116988  9724 consensus_meta.cc:287] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.643s	user 0.016s	sys 0.000s
I20260709 13:02:44.165752  1673 raft_consensus.cc:1275] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 24 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 21 index: 11. Preceding OpId from leader: term: 24 index: 12. (index mismatch)
I20260709 13:02:44.100670  9489 raft_consensus.cc:3060] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Advancing to term 24
I20260709 13:02:44.166693  9647 consensus_queue.cc:1048] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 13:02:44.169589  1602 raft_consensus.cc:2468] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 22.
I20260709 13:02:44.143963  9589 raft_consensus.cc:493] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:44.171664  9589 raft_consensus.cc:515] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:44.101123  9714 raft_consensus.cc:2804] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:02:44.174787  9714 raft_consensus.cc:493] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:44.175411  9714 raft_consensus.cc:3060] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:02:44.127009  2298 leader_election.cc:304] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:02:44.183197  9589 leader_election.cc:290] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:44.195981  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 22 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:44.196971  2336 raft_consensus.cc:2468] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 21.
I20260709 13:02:44.145330  8994 raft_consensus.cc:493] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:02:44.200716  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 73889.6us
I20260709 13:02:44.153226  9710 raft_consensus.cc:493] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:44.204540  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:44.205252  2298 leader_election.cc:304] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:02:44.206131  9618 raft_consensus.cc:2804] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 13:02:44.206640  9757 raft_consensus.cc:697] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 19 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:02:44.206676  9618 raft_consensus.cc:697] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 22 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:02:44.208174  9618 consensus_queue.cc:237] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.15, Last appended by leader: 15, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:44.209244  9757 consensus_queue.cc:237] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.13, Last appended by leader: 13, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:44.202816  2336 raft_consensus.cc:1240] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 17. Current term is 18. Ops: [15.12-17.13]
I20260709 13:02:44.192826  9724 raft_consensus.cc:2749] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 13:02:44.191239  9209 raft_consensus.cc:493] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:02:44.228645  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:44.228866  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:44.201061  9214 leader_election.cc:290] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:44.225449  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 19 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:44.203675  8994 raft_consensus.cc:3060] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:02:44.237694  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:44.205080  9750 raft_consensus.cc:2804] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 13:02:44.194613  9501 raft_consensus.cc:493] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:44.238615  2335 raft_consensus.cc:2393] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 21: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:02:44.211628  9710 raft_consensus.cc:3060] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:02:44.226065  9589 raft_consensus.cc:493] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:44.239632  9750 raft_consensus.cc:493] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:44.239779  9703 leader_election.cc:290] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:44.252372  9750 raft_consensus.cc:3060] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:02:44.255604  9732 raft_consensus.cc:515] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:44.260944  9732 leader_election.cc:290] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 27 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:44.316080  1604 raft_consensus.cc:2468] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 26.
I20260709 13:02:44.323598  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:44.327637  1676 raft_consensus.cc:1240] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 24 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 23. Current term is 24. Ops: []
I20260709 13:02:44.329411  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:44.331009  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:02:44.333449  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:44.340412  1646 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:44.343659  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 22 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:44.345793  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:44.346768  1748 raft_consensus.cc:3060] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Advancing to term 22
I20260709 13:02:44.247959  9589 raft_consensus.cc:515] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:44.365757  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 166877us
I20260709 13:02:44.358120  9589 leader_election.cc:290] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:44.393491  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:44.359687  9529 raft_consensus.cc:493] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:44.406327  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:44.407680  1674 raft_consensus.cc:2393] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 16: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:02:44.409349  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 25 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:44.410586  1601 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36462 (request call id 1953) took 1458 ms. Trace:
W20260709 13:02:44.410913  1718 leader_election.cc:336] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Timed out: RequestConsensusVote RPC to 127.0.107.194:46261 timed out after 2.018s (SENT)
I20260709 13:02:44.411116  9724 raft_consensus.cc:493] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:44.413553  9724 raft_consensus.cc:515] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:44.429059  9489 raft_consensus.cc:515] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:44.442648  9633 raft_consensus.cc:493] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:44.443917  9633 raft_consensus.cc:515] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:44.447259  9633 leader_election.cc:290] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:44.447325  1605 raft_consensus.cc:1275] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 19 index: 16. Preceding OpId from leader: term: 21 index: 17. (index mismatch)
I20260709 13:02:44.447934  1718 leader_election.cc:304] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:44.451490  1748 raft_consensus.cc:2468] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 22.
I20260709 13:02:44.457288  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 18 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:02:44.448465  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 369036us
I20260709 13:02:44.417109  9759 raft_consensus.cc:493] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:44.236766  9424 raft_consensus.cc:515] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:44.471621  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:44.247625  9501 raft_consensus.cc:515] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:44.472328  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:44.479259  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 19 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:44.480792  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 21 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:44.481696  1673 raft_consensus.cc:2468] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 20.
I20260709 13:02:44.483040  2335 raft_consensus.cc:2468] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 23.
I20260709 13:02:44.484481  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:44.484905  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:44.485066  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:44.235699  9209 raft_consensus.cc:515] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:44.485929  1597 raft_consensus.cc:3060] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 23 FOLLOWER]: Advancing to term 24
I20260709 13:02:44.486056  1569 leader_election.cc:304] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
W20260709 13:02:44.486475  2300 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:44.488292  2331 raft_consensus.cc:3060] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:02:44.490226  9704 raft_consensus.cc:2804] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 13:02:44.490608  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 19 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:44.490955  9704 raft_consensus.cc:493] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:44.491514  9704 raft_consensus.cc:3060] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:02:44.485355  2337 raft_consensus.cc:2468] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 25.
I20260709 13:02:44.486066  1671 raft_consensus.cc:2468] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 17.
I20260709 13:02:44.528059  9757 raft_consensus.cc:2749] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:02:44.535449  1750 raft_consensus.cc:1240] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 21. Current term is 22. Ops: []
I20260709 13:02:44.535351  1646 leader_election.cc:304] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:44.536861  9424 leader_election.cc:290] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 22 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:44.550928  9226 raft_consensus.cc:2804] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 13:02:44.551681  9226 raft_consensus.cc:493] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:44.499799  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 22 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:44.411140  1601 rpcz_store.cc:276] 0709 13:02:42.927598 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:42.930247 (+  2649us) service_pool.cc:224] Handling call
0709 13:02:44.359583 (+1429336us) spinlock_profiling.cc:243] Waited 581 ms on lock 0x7b54000f1754. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:02:44.386258 (+ 26675us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1335947520}
I20260709 13:02:44.552240  9226 raft_consensus.cc:3060] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 26 FOLLOWER]: Advancing to term 27
I20260709 13:02:44.557045  9724 leader_election.cc:290] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:44.557991  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:44.560972  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:44.562109  1674 raft_consensus.cc:2393] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 24: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:02:44.562973  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 25 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:44.563424  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:44.563889  1672 raft_consensus.cc:2468] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 24.
I20260709 13:02:44.564378  1677 raft_consensus.cc:2368] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 21. Current term is 22.
I20260709 13:02:44.565250  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:44.566118  1679 raft_consensus.cc:2368] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 23. Current term is 24.
I20260709 13:02:44.552810  9529 raft_consensus.cc:515] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:44.575979  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 25 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:44.583359  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:44.583711  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:44.584198  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:44.574220  9529 leader_election.cc:290] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:44.582105  1671 raft_consensus.cc:2468] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 24.
I20260709 13:02:44.589588  2331 raft_consensus.cc:2468] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 20.
I20260709 13:02:44.590091  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:44.588876  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 19 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:44.588850  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 23 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:44.597266  9476 raft_consensus.cc:3055] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 13:02:44.597802  9476 raft_consensus.cc:740] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:02:44.597970  9489 leader_election.cc:290] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 24 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:44.599035  9489 consensus_queue.cc:1048] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 13:02:44.602929  1672 raft_consensus.cc:2468] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 18.
I20260709 13:02:44.603612  8994 raft_consensus.cc:515] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:44.605432  9530 raft_consensus.cc:493] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:44.598968  9777 consensus_queue.cc:1059] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 19.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 13:02:44.606174  9530 raft_consensus.cc:515] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:44.616533  1597 raft_consensus.cc:2410] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for term 24 because replica has last-logged OpId of term: 23 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 12.
I20260709 13:02:44.614250  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:02:44.627379  1570 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:44.628759  9358 consensus_meta.cc:287] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.804s	user 0.019s	sys 0.005s
I20260709 13:02:44.631251  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:44.632807  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:44.634698  1677 raft_consensus.cc:2393] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 21: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:02:44.635546  9714 raft_consensus.cc:515] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:44.636610  9530 leader_election.cc:290] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:44.554009  9759 raft_consensus.cc:515] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:44.638501  9771 log.cc:927] Time spent T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.050s	user 0.001s	sys 0.000s
I20260709 13:02:44.638679  2336 raft_consensus.cc:2368] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 16. Current term is 17.
I20260709 13:02:44.640793  9759 leader_election.cc:290] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:44.632318  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:44.640700  9776 raft_consensus.cc:3055] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 13:02:44.617611  8994 leader_election.cc:290] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 18 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:44.644372  9776 raft_consensus.cc:740] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:02:44.647909  9501 leader_election.cc:290] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:44.648741  1600 raft_consensus.cc:1275] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 24 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 22 index: 17. Preceding OpId from leader: term: 24 index: 18. (index mismatch)
I20260709 13:02:44.649287  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:44.658344  2330 raft_consensus.cc:1275] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 22 index: 16. Preceding OpId from leader: term: 23 index: 17. (index mismatch)
I20260709 13:02:44.658802  9714 leader_election.cc:290] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 17 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:44.656359  1679 raft_consensus.cc:2468] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 20.
I20260709 13:02:44.660501  1601 raft_consensus.cc:2393] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 20: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:02:44.632833  9209 leader_election.cc:290] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:44.658450  9776 consensus_queue.cc:260] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 19.16, Last appended by leader: 16, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:44.662797  9776 raft_consensus.cc:3060] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:02:44.679165  9666 raft_consensus.cc:493] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:44.679591  9704 raft_consensus.cc:515] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:44.688001  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:44.689596  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:44.690620  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:44.691668  1753 raft_consensus.cc:2393] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 27: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:02:44.693317  1753 raft_consensus.cc:1275] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 19 index: 16. Preceding OpId from leader: term: 21 index: 17. (index mismatch)
I20260709 13:02:44.694219  1674 raft_consensus.cc:2468] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 22.
I20260709 13:02:44.696576  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:44.697921  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:44.703333  2328 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39726 (request call id 1759) took 2152 ms. Trace:
I20260709 13:02:44.703848  2328 rpcz_store.cc:276] 0709 13:02:42.550840 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:42.551009 (+   169us) service_pool.cc:224] Handling call
0709 13:02:42.619969 (+ 68960us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:42.675632 (+ 55663us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:44.702563 (+2026931us) spinlock_profiling.cc:243] Waited 23.2 ms on lock 0x7b5400627ad4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:02:44.703053 (+   490us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":53362048}
W20260709 13:02:44.705528  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 55791us
I20260709 13:02:44.706713  9589 raft_consensus.cc:493] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:43.565609  2332 rpcz_store.cc:276] 0709 13:02:42.385923 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:42.386530 (+   607us) service_pool.cc:224] Handling call
0709 13:02:42.386916 (+   386us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:42.394203 (+  7287us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:42.394246 (+    43us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:42.431478 (+ 37232us) log.cc:844] Serialized 26 byte log entry
0709 13:02:42.762864 (+331386us) raft_consensus.cc:1645] Marking committed up to 14
0709 13:02:43.564520 (+801656us) log.cc:844] Serialized 14 byte log entry
0709 13:02:43.564779 (+   259us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:43.564871 (+    92us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:43.564920 (+    49us) raft_consensus.cc:1685] finished
0709 13:02:43.564936 (+    16us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:43.565078 (+   142us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6746,"raft.queue_time_us":7757,"raft.run_cpu_time_us":195,"raft.run_wall_time_us":13267,"thread_start_us":1000516,"threads_started":2,"wal-append.queue_time_us":448931}
I20260709 13:02:44.707151  9594 consensus_queue.cc:1048] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 13:02:44.707378  9589 raft_consensus.cc:515] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } }
I20260709 13:02:44.716657  9358 raft_consensus.cc:493] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:44.720587  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:44.721443  1597 raft_consensus.cc:3060] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:02:44.722595  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:44.723388  1599 raft_consensus.cc:2376] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 21. Re-sending same reply.
I20260709 13:02:44.724474  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:44.725124  1599 raft_consensus.cc:3060] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 16 FOLLOWER]: Advancing to term 18
I20260709 13:02:44.730993  9226 raft_consensus.cc:515] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:44.731442  1754 raft_consensus.cc:1240] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 22. Current term is 23. Ops: []
I20260709 13:02:44.729979  9648 leader_election.cc:290] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:44.701092  9704 leader_election.cc:290] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 22 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:44.706079  1718 leader_election.cc:304] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:02:44.717514  9358 raft_consensus.cc:515] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:44.690382  1603 raft_consensus.cc:2468] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 17.
I20260709 13:02:44.743582  9634 raft_consensus.cc:493] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:44.744405  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:44.744839  9714 raft_consensus.cc:493] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
W20260709 13:02:44.745666  2299 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:44.754577  9598 raft_consensus.cc:493] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:44.754954  9760 raft_consensus.cc:2749] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
W20260709 13:02:44.756032  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 50119.2us
W20260709 13:02:44.764037  9722 consensus_meta.cc:287] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.962s	user 0.015s	sys 0.011s
I20260709 13:02:44.765169  1604 raft_consensus.cc:1275] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 20 index: 12. (index mismatch)
I20260709 13:02:44.767280  2330 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39726 (request call id 1762) took 2127 ms. Trace:
I20260709 13:02:44.767642  1601 raft_consensus.cc:1240] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 21. Current term is 22. Ops: []
I20260709 13:02:44.679948  9666 raft_consensus.cc:515] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:44.776057  2301 leader_election.cc:304] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:02:44.775094  2330 rpcz_store.cc:276] 0709 13:02:42.639782 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:42.641698 (+  1916us) service_pool.cc:224] Handling call
0709 13:02:42.671487 (+ 29789us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:44.658857 (+1987370us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:44.766955 (+108098us) spinlock_profiling.cc:243] Waited 1.97 s on lock 0x7b540065ed54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:02:44.767211 (+   256us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4540223232}
W20260709 13:02:44.798161  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 43798.9us
W20260709 13:02:44.797679  2299 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.107.194:46261 timed out after 2.172s (SENT) [suppressed 36 similar messages]
I20260709 13:02:44.803958  1598 raft_consensus.cc:2410] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 for term 21 because replica has last-logged OpId of term: 19 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 12.
I20260709 13:02:44.806720  1603 raft_consensus.cc:1240] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 21. Current term is 22. Ops: []
I20260709 13:02:44.808678  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 25 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:44.771175  9714 raft_consensus.cc:515] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:44.815361  9710 consensus_meta.cc:287] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.568s	user 0.007s	sys 0.011s
I20260709 13:02:44.815948  9710 raft_consensus.cc:515] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:44.817597  9714 leader_election.cc:290] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:44.818840  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:44.818674  9598 raft_consensus.cc:515] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:44.819473  1598 raft_consensus.cc:3060] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:02:44.821254  9598 leader_election.cc:290] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:44.822024  8994 raft_consensus.cc:2749] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:02:44.777071  9634 raft_consensus.cc:515] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:44.826730  8968 consensus_meta.cc:287] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.797s	user 0.010s	sys 0.009s
I20260709 13:02:44.826872  9634 leader_election.cc:290] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:44.827461  9634 raft_consensus.cc:493] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:44.827261  8968 raft_consensus.cc:515] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:44.828011  9634 raft_consensus.cc:515] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:44.828526  9515 raft_consensus.cc:493] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:44.829349  9515 raft_consensus.cc:515] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:44.829937  8968 leader_election.cc:290] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 23 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:44.832041  9515 leader_election.cc:290] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:44.828485  1748 raft_consensus.cc:2468] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 23.
W20260709 13:02:44.834579  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:44.841164  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 62437.4us
I20260709 13:02:44.802779  9589 leader_election.cc:290] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 13:02:44.844803  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 63533.7us
I20260709 13:02:44.845885  1599 raft_consensus.cc:2468] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 18.
I20260709 13:02:44.855038  1597 raft_consensus.cc:2468] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 17.
I20260709 13:02:44.818423  9710 leader_election.cc:290] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 20 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:44.834345  9666 leader_election.cc:290] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:44.846657  9634 leader_election.cc:290] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:44.862305  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:44.865796  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:44.866752  1597 raft_consensus.cc:3060] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 26 FOLLOWER]: Advancing to term 27
I20260709 13:02:44.864512  2298 leader_election.cc:304] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:02:44.869846  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:44.863919  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 24 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:44.872946  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:44.880021  1674 raft_consensus.cc:2368] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 for earlier term 26. Current term is 27.
I20260709 13:02:44.884888  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:44.855756  9760 consensus_queue.cc:1048] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.028s
I20260709 13:02:44.889169  9358 leader_election.cc:290] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:44.892403  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
W20260709 13:02:44.891214  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 119917us
I20260709 13:02:44.898461  9634 consensus_queue.cc:1048] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 13:02:44.903290  1599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:44.907123  1597 raft_consensus.cc:2468] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 27.
I20260709 13:02:44.904210  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:44.909144  1675 raft_consensus.cc:2376] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 22. Re-sending same reply.
I20260709 13:02:44.910279  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 24 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:44.923890  2330 raft_consensus.cc:3060] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:02:44.904726  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
W20260709 13:02:44.929922  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 392582us
I20260709 13:02:44.935880  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:44.933530  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:44.942097  1597 raft_consensus.cc:1240] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 24 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 23. Current term is 24. Ops: []
I20260709 13:02:44.911899  9634 raft_consensus.cc:493] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:44.933737  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:44.960125  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:44.961007  1675 raft_consensus.cc:3060] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:02:44.972534  1679 raft_consensus.cc:2468] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 23.
I20260709 13:02:44.933837  9226 leader_election.cc:290] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 27 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:44.978003  1746 raft_consensus.cc:2393] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 22: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:02:44.978421  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:44.941642  9594 consensus_queue.cc:1048] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.038s
I20260709 13:02:44.980748  9634 raft_consensus.cc:515] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:44.981874  9730 log.cc:927] Time spent T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 1.349s	user 0.006s	sys 0.000s
I20260709 13:02:44.983989  9634 leader_election.cc:290] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:44.985857  9588 raft_consensus.cc:2804] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 13:02:44.986307  9588 raft_consensus.cc:493] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:44.986683  9588 raft_consensus.cc:3060] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 26 FOLLOWER]: Advancing to term 27
W20260709 13:02:44.986886  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 45066.9us
I20260709 13:02:44.987056  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:44.987762  2328 raft_consensus.cc:3060] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:02:44.988766  1569 leader_election.cc:304] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
W20260709 13:02:44.989646  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 94545.6us
I20260709 13:02:44.941401  9648 raft_consensus.cc:493] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:44.999852  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 27 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:45.000064  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:45.003834  2298 leader_election.cc:304] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
W20260709 13:02:45.006281  2300 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:45.005152  1646 leader_election.cc:304] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:45.009837  2298 leader_election.cc:304] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:02:45.015635  1679 raft_consensus.cc:2468] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 22.
W20260709 13:02:45.018352  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 141820us
I20260709 13:02:45.017035  1675 raft_consensus.cc:2468] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 23.
I20260709 13:02:45.024004  9618 consensus_queue.cc:1048] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.006s
I20260709 13:02:45.026047  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 18 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:02:45.026615  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 63867.4us
I20260709 13:02:45.027172  2332 raft_consensus.cc:2393] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 18: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:02:45.028947  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:45.030403  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:02:45.031149  9782 log.cc:927] Time spent T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.125s	user 0.001s	sys 0.000s
I20260709 13:02:45.031805  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:45.031946  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:45.032545  2337 raft_consensus.cc:3060] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:02:45.032930  1754 raft_consensus.cc:2468] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 20.
I20260709 13:02:45.037766  1677 raft_consensus.cc:2393] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 22: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:02:45.038937  9709 raft_consensus.cc:2749] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:02:45.040400  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:45.040966  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 25 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:45.041812  1672 raft_consensus.cc:3060] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 24 FOLLOWER]: Advancing to term 25
W20260709 13:02:45.040400  1645 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:45.044023  1675 raft_consensus.cc:1240] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 21. Current term is 22. Ops: []
I20260709 13:02:45.024187  9513 consensus_queue.cc:1059] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 18.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.037s
I20260709 13:02:45.052528  1596 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40154 (request call id 3097) took 2210 ms. Trace:
I20260709 13:02:45.052989  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:45.054921  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:45.055091  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:45.055806  1673 raft_consensus.cc:3060] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:02:45.056372  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:45.057714  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:45.058842  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:45.059752  9668 raft_consensus.cc:2804] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 13:02:45.060176  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:45.060364  9668 raft_consensus.cc:697] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 17 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:02:45.060875  1679 raft_consensus.cc:2468] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 16.
I20260709 13:02:45.061654  9668 consensus_queue.cc:237] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:45.066948  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:02:45.068220  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 79702.8us
I20260709 13:02:45.066414  1598 raft_consensus.cc:2468] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 21.
I20260709 13:02:45.066941  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:45.074775  1680 raft_consensus.cc:3060] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:02:45.058746  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:45.072475  2337 raft_consensus.cc:2468] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 22.
W20260709 13:02:45.077152  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 48603.6us
I20260709 13:02:45.053045  1596 rpcz_store.cc:276] 0709 13:02:42.800049 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:42.801940 (+  1891us) service_pool.cc:224] Handling call
0709 13:02:42.802363 (+   423us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:43.245006 (+442643us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:43.245061 (+    55us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:43.245991 (+   930us) log.cc:844] Serialized 26 byte log entry
0709 13:02:43.458809 (+212818us) raft_consensus.cc:1645] Marking committed up to 13
0709 13:02:43.458874 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:43.458942 (+    68us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:44.810818 (+1351876us) spinlock_profiling.cc:243] Waited 6.08 ms on lock 0x7b4400022b4c. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f4262f9bc85 00007f4262f9c6a2 00007f4265d0e1db 00007f4265d09aaf 00007f4265cfd86e 00007f4265cfcbb6 00007f4265cfc7d8 00007f4265d06221 00007f4265d061d9 00007f4265d06169
0709 13:02:44.991609 (+180791us) raft_consensus.cc:1685] finished
0709 13:02:44.991641 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:45.010818 (+ 19177us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":136028,"raft.run_cpu_time_us":228,"raft.run_wall_time_us":190870,"spinlock_wait_cycles":13981440,"thread_start_us":621951,"threads_started":2,"wal-append.queue_time_us":273892}
W20260709 13:02:45.082116  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 113952us
W20260709 13:02:45.081516  9750 consensus_meta.cc:287] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.807s	user 0.017s	sys 0.007s
I20260709 13:02:45.083657  9750 raft_consensus.cc:515] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:45.085916  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:45.086651  2333 raft_consensus.cc:3060] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Advancing to term 24
I20260709 13:02:45.086189  9588 raft_consensus.cc:515] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:45.088274  9776 raft_consensus.cc:493] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:45.088716  9588 leader_election.cc:290] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 27 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:45.088814  9776 raft_consensus.cc:515] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:45.090963  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 19 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:45.091331  9750 leader_election.cc:290] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 22 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:45.016299  9648 raft_consensus.cc:515] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:45.092339  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:45.093647  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 25 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:45.098381  2330 raft_consensus.cc:2468] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 23.
I20260709 13:02:45.109565  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 24 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:45.111503  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:45.111065  2328 raft_consensus.cc:2468] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 21.
I20260709 13:02:45.112373  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:45.113153  2337 raft_consensus.cc:2393] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 23: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:02:45.083102  2336 raft_consensus.cc:2468] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 20.
I20260709 13:02:45.097162  1643 leader_election.cc:304] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:02:45.114933  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 24 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:45.117676  1718 leader_election.cc:304] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
W20260709 13:02:45.095963  9772 log.cc:927] Time spent T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.061s	user 0.000s	sys 0.001s
I20260709 13:02:45.095059  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 19 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:45.119964  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:45.124099  2334 raft_consensus.cc:3060] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 25 FOLLOWER]: Advancing to term 27
I20260709 13:02:45.125676  1673 raft_consensus.cc:2468] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 18.
I20260709 13:02:45.096202  9780 raft_consensus.cc:2804] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 13:02:45.096969  9226 raft_consensus.cc:493] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:45.095721  9783 raft_consensus.cc:2804] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 13:02:45.094337  9648 leader_election.cc:290] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:45.131968  2330 raft_consensus.cc:1275] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 19 index: 14. Preceding OpId from leader: term: 22 index: 16. (index mismatch)
I20260709 13:02:45.097647  9542 raft_consensus.cc:493] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:45.140858  9785 raft_consensus.cc:2749] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:02:45.096737  9760 consensus_queue.cc:1059] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 19.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.037s
I20260709 13:02:45.147064  9424 raft_consensus.cc:2749] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 13:02:45.147356  1680 raft_consensus.cc:2468] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 17.
I20260709 13:02:45.147670  9776 leader_election.cc:290] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:45.118407  9714 consensus_queue.cc:1048] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.034s
I20260709 13:02:45.097738  9700 consensus_queue.cc:1048] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.054s
I20260709 13:02:45.134792  9783 raft_consensus.cc:493] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:45.167248  1494 catalog_manager.cc:5697] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 14 to 16. New cstate: current_term: 16 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
W20260709 13:02:45.168417  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 88154.8us
I20260709 13:02:45.139216  9780 raft_consensus.cc:697] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 18 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:02:45.172670  1600 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 1959) took 2041 ms. Trace:
I20260709 13:02:45.180455  2333 raft_consensus.cc:2468] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 24.
I20260709 13:02:45.181028  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 22 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:45.181967  2331 raft_consensus.cc:2393] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 22: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:02:45.181111  1600 rpcz_store.cc:276] 0709 13:02:43.131542 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:43.134667 (+  3125us) service_pool.cc:224] Handling call
0709 13:02:43.135123 (+   456us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:44.656833 (+1521710us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:45.172253 (+515420us) spinlock_profiling.cc:243] Waited 1.51 s on lock 0x7b54000ac3d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:02:45.172554 (+   301us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3480677120}
I20260709 13:02:45.133364  9358 consensus_queue.cc:1048] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.034s
I20260709 13:02:45.179116  9226 raft_consensus.cc:515] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:45.187728  1494 catalog_manager.cc:5697] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 12 to 14, leader changed from 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) to af520b67945d443b901bbeca37cc5701 (127.0.107.195). New cstate: current_term: 14 leader_uuid: "af520b67945d443b901bbeca37cc5701" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } }
I20260709 13:02:45.193687  9226 leader_election.cc:290] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:45.193205  1494 catalog_manager.cc:5697] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 12 to 17, leader changed from 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) to 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193). New cstate: current_term: 17 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20260709 13:02:45.196661  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 99891.7us
I20260709 13:02:45.196838  9542 raft_consensus.cc:515] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:45.197767  1643 leader_election.cc:304] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:02:45.199149  1672 raft_consensus.cc:2468] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 25.
I20260709 13:02:45.200062  9542 leader_election.cc:290] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:45.204051  1748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43156 (request call id 1614) took 1039 ms. Trace:
I20260709 13:02:45.204479  1748 rpcz_store.cc:276] 0709 13:02:44.164207 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:44.467627 (+303420us) service_pool.cc:224] Handling call
0709 13:02:45.203811 (+736184us) spinlock_profiling.cc:243] Waited 351 ms on lock 0x7b54001c6e54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:02:45.203941 (+   130us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":806955264}
I20260709 13:02:45.195502  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:45.215199  1717 leader_election.cc:304] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:45.215477  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 22 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:45.182376  9783 raft_consensus.cc:3060] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:02:45.218904  1494 catalog_manager.cc:5697] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 13 to 15, leader changed from 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) to 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196). New cstate: current_term: 15 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 13:02:45.219606  1718 leader_election.cc:304] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:45.156358  9750 consensus_queue.cc:1048] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.031s
W20260709 13:02:45.222820  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 47717.2us
I20260709 13:02:45.224596  2334 raft_consensus.cc:2468] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 27.
I20260709 13:02:45.231913  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:45.232568  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:45.185160  8994 consensus_queue.cc:1048] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.040s
I20260709 13:02:45.225449  1644 leader_election.cc:304] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:02:45.216202  9226 consensus_queue.cc:1048] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.010s
I20260709 13:02:45.230831  1494 catalog_manager.cc:5697] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 19 to 23. New cstate: current_term: 23 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 13:02:45.235734  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:02:45.237383  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 49319.5us
I20260709 13:02:45.239049  1754 raft_consensus.cc:1240] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 25 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 24. Current term is 25. Ops: []
I20260709 13:02:45.241343  2301 leader_election.cc:304] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:02:45.242810  1750 raft_consensus.cc:2468] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 18.
I20260709 13:02:45.234817  9785 raft_consensus.cc:2749] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 13:02:45.250388  1746 raft_consensus.cc:1240] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 19. Current term is 20. Ops: []
I20260709 13:02:45.251622  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:45.254228  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:45.239583  9760 raft_consensus.cc:2804] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 13:02:45.258864  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:45.253042  9700 raft_consensus.cc:2749] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:02:45.225795  9704 consensus_queue.cc:1059] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.027s
I20260709 13:02:45.263064  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:45.264008  1753 raft_consensus.cc:2393] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 20: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:02:45.266556  9744 consensus_queue.cc:1059] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 13:02:45.269901  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 27 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:45.271307  9609 consensus_queue.cc:1048] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 12, Time since last communication: 0.032s
I20260709 13:02:45.271495  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:45.272930  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 23 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:45.273777  1747 raft_consensus.cc:1240] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 23. Current term is 24. Ops: []
I20260709 13:02:45.274155  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 21 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:45.275816  1646 leader_election.cc:304] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:45.278650  9700 raft_consensus.cc:2804] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 13:02:45.278726  9594 raft_consensus.cc:3055] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 13:02:45.279168  9700 raft_consensus.cc:493] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:45.279460  9594 raft_consensus.cc:740] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:02:45.280961  9594 consensus_queue.cc:260] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.14, Last appended by leader: 14, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:45.283301  1746 raft_consensus.cc:1275] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 20 index: 15. Preceding OpId from leader: term: 22 index: 16. (index mismatch)
I20260709 13:02:45.301558  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 22 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:45.291671  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:45.279680  9700 raft_consensus.cc:3060] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Advancing to term 24
I20260709 13:02:45.255481  9424 raft_consensus.cc:2804] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:02:45.299134  1494 catalog_manager.cc:5697] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 12 to 14, leader changed from 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194) to 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193). New cstate: current_term: 14 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: HEALTHY } } }
I20260709 13:02:45.253400  9757 consensus_queue.cc:1059] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.041s
I20260709 13:02:45.232646  9476 consensus_queue.cc:260] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.14, Last appended by leader: 14, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:45.306685  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 87406.2us
I20260709 13:02:45.241882  9515 consensus_queue.cc:1048] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.019s
I20260709 13:02:45.252938  9589 consensus_queue.cc:1048] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 8, Time since last communication: 0.042s
I20260709 13:02:45.256966  9598 consensus_queue.cc:1048] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.048s
I20260709 13:02:45.303555  9714 raft_consensus.cc:2749] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:02:45.323329  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:45.339942  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:45.341116  1753 raft_consensus.cc:2393] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 27: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:02:45.279878  9226 consensus_queue.cc:1059] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 19.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.043s
W20260709 13:02:45.344458  1495 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35464 (request call id 288) took 66580 ms (client timeout 15000 ms). Trace:
I20260709 13:02:45.344841  1494 catalog_manager.cc:5697] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 9 to 13. New cstate: current_term: 13 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
W20260709 13:02:45.346622  1571 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:45.346951  1494 catalog_manager.cc:5697] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 6 to 16, leader changed from af520b67945d443b901bbeca37cc5701 (127.0.107.195) to 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193). New cstate: current_term: 16 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } }
I20260709 13:02:45.348894  1571 leader_election.cc:304] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:02:45.350363  9744 raft_consensus.cc:2749] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:02:45.352263  9589 consensus_queue.cc:1059] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 13:02:45.294512  9513 consensus_queue.cc:1048] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.033s
I20260709 13:02:45.357657  9515 consensus_queue.cc:1048] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 13:02:45.358829  1751 raft_consensus.cc:2393] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 21: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:02:45.359869  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 22 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:45.360993  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 23 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:45.361850  1751 raft_consensus.cc:2410] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 for term 23 because replica has last-logged OpId of term: 20 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 14.
I20260709 13:02:45.295049  9780 consensus_queue.cc:237] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 16.13, Last appended by leader: 13, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:45.365152  8968 consensus_queue.cc:1059] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 22.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260709 13:02:45.362367  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 126471us
I20260709 13:02:45.309360  9424 raft_consensus.cc:493] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:45.349774  1494 catalog_manager.cc:5697] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 17 to 19. New cstate: current_term: 19 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:02:45.366302  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
W20260709 13:02:45.368299  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 45729.9us
I20260709 13:02:45.368132  1571 leader_election.cc:304] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
W20260709 13:02:45.370675  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 129543us
I20260709 13:02:45.374579  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 18 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:45.370453  9424 raft_consensus.cc:3060] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:02:45.377805  1604 raft_consensus.cc:3060] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:02:45.379798  9502 raft_consensus.cc:3055] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 LEADER]: Stepping down as leader of term 19
I20260709 13:02:45.380263  9502 raft_consensus.cc:740] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:02:45.381412  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:45.381748  2301 leader_election.cc:304] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:02:45.382247  1596 raft_consensus.cc:2393] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 27: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:02:45.382651  9703 raft_consensus.cc:3055] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 LEADER]: Stepping down as leader of term 19
I20260709 13:02:45.383082  9703 raft_consensus.cc:740] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:02:45.383337  1601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 22 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:45.384176  1601 raft_consensus.cc:2393] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 22: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:02:45.384106  9703 consensus_queue.cc:260] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 19.12, Last appended by leader: 12, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } }
I20260709 13:02:45.388561  2301 leader_election.cc:304] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
W20260709 13:02:45.409418  2300 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:45.414790  1494 catalog_manager.cc:5697] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 14 to 18. New cstate: current_term: 18 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 13:02:45.381433  9502 consensus_queue.cc:260] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 19.16, Last appended by leader: 16, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:45.441849  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:45.442571  9710 consensus_queue.cc:1048] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20260709 13:02:45.442793  1602 raft_consensus.cc:2468] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 21.
I20260709 13:02:45.391597  9515 raft_consensus.cc:3055] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 LEADER]: Stepping down as leader of term 19
I20260709 13:02:45.425516  9668 raft_consensus.cc:2804] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:02:45.425997  9750 raft_consensus.cc:2749] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:02:45.467561  2298 leader_election.cc:304] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
W20260709 13:02:45.469523  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 197990us
I20260709 13:02:45.387694  9776 consensus_queue.cc:1048] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.058s
I20260709 13:02:45.396778  9714 consensus_queue.cc:1048] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.062s
I20260709 13:02:45.480166  1603 raft_consensus.cc:1275] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 19 index: 14. Preceding OpId from leader: term: 22 index: 16. (index mismatch)
W20260709 13:02:45.476178  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 198551us
I20260709 13:02:45.439935  9744 raft_consensus.cc:2804] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:02:45.444128  9502 raft_consensus.cc:3060] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:02:45.456398  9515 raft_consensus.cc:740] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:02:45.463608  9668 raft_consensus.cc:493] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:45.498384  1494 catalog_manager.cc:5697] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 15 to 17. New cstate: current_term: 17 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: HEALTHY } } }
I20260709 13:02:45.487408  2301 leader_election.cc:304] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:45.478859  9704 raft_consensus.cc:493] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:02:45.481208  9703 raft_consensus.cc:3060] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Advancing to term 20
W20260709 13:02:45.493649  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 230986us
I20260709 13:02:45.525517  1604 raft_consensus.cc:2468] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 20.
W20260709 13:02:45.505353  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 238002us
I20260709 13:02:45.487013  9744 raft_consensus.cc:493] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
W20260709 13:02:45.528504  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 46707.9us
I20260709 13:02:45.532166  1494 catalog_manager.cc:5697] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 13 to 15. New cstate: current_term: 15 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } }
I20260709 13:02:45.537320  2301 leader_election.cc:304] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:02:45.545819  2301 leader_election.cc:304] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:02:45.501439  9668 raft_consensus.cc:3060] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Advancing to term 21
W20260709 13:02:45.554328  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 86889.8us
I20260709 13:02:45.544818  1717 leader_election.cc:304] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:45.527653  9710 raft_consensus.cc:2804] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 13:02:45.531646  9594 raft_consensus.cc:3060] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:02:45.531786  9744 raft_consensus.cc:3060] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:02:45.501214  9618 consensus_queue.cc:1048] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.051s
I20260709 13:02:45.573729  9710 raft_consensus.cc:493] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:45.578733  9634 raft_consensus.cc:493] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:45.494262  9709 consensus_queue.cc:1059] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 24.12, Next index: 13, Last known committed idx: 11, Time since last communication: 0.063s
I20260709 13:02:45.578817  9710 raft_consensus.cc:3060] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:02:45.583307  1643 leader_election.cc:304] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:02:45.530184  9760 raft_consensus.cc:697] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 22 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
W20260709 13:02:45.579473  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 83342us
I20260709 13:02:45.565604  1494 catalog_manager.cc:5697] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 13 to 16, leader changed from af520b67945d443b901bbeca37cc5701 (127.0.107.195) to 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194). New cstate: current_term: 16 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } }
I20260709 13:02:45.527750  9598 consensus_queue.cc:1059] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.033s
I20260709 13:02:45.547631  9780 raft_consensus.cc:2764] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 24 FOLLOWER]: Leader pre-election decision vote started in defunct term 23: won
I20260709 13:02:45.534202  9807 raft_consensus.cc:3055] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 13:02:45.605597  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:45.607748  1604 raft_consensus.cc:2393] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 18: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
W20260709 13:02:45.606356  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 70489.2us
I20260709 13:02:45.610548  1718 leader_election.cc:304] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:45.618605  1602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 25 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:45.619314  1602 raft_consensus.cc:2468] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 24.
I20260709 13:02:45.623056  1605 raft_consensus.cc:1275] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 15 index: 11. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
I20260709 13:02:45.609338  9812 raft_consensus.cc:3055] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 23 LEADER]: Stepping down as leader of term 23
I20260709 13:02:45.607363  9807 raft_consensus.cc:740] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:02:45.612547  9808 raft_consensus.cc:3055] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 13:02:45.565653  9704 raft_consensus.cc:515] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:45.628832  1494 catalog_manager.cc:5697] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 10 to 16, leader changed from 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194) to af520b67945d443b901bbeca37cc5701 (127.0.107.195). New cstate: current_term: 16 leader_uuid: "af520b67945d443b901bbeca37cc5701" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } }
I20260709 13:02:45.617674  9757 raft_consensus.cc:493] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:45.624959  9476 raft_consensus.cc:3060] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Advancing to term 21
W20260709 13:02:45.345033  1495 rpcz_store.cc:269] 0709 13:01:38.676776 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:38.677083 (+   307us) service_pool.cc:224] Handling call
0709 13:02:45.250372 (+66573289us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 13:02:32.601935 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 13:02:32.742779 (+140844us) write_op.cc:432] Acquiring schema lock in shared mode
0709 13:02:32.742846 (+    67us) write_op.cc:435] Acquired schema lock
0709 13:02:32.742871 (+    25us) tablet.cc:662] Decoding operations
0709 13:02:32.744352 (+  1481us) write_op.cc:620] Acquiring the partition lock for write op
0709 13:02:32.744491 (+   139us) write_op.cc:641] Partition lock acquired for write op
0709 13:02:32.744520 (+    29us) tablet.cc:685] Acquiring locks for 16 operations
0709 13:02:32.746019 (+  1499us) tablet.cc:701] Row locks acquired
0709 13:02:32.746084 (+    65us) write_op.cc:260] PREPARE: finished
0709 13:02:32.746291 (+   207us) write_op.cc:270] Start()
0709 13:02:32.823883 (+ 77592us) write_op.cc:276] Timestamp: P: 1783602152746212 usec, L: 0
0709 13:02:32.830321 (+  6438us) op_driver.cc:348] REPLICATION: starting
0709 13:02:32.865426 (+ 35105us) log.cc:844] Serialized 6312 byte log entry
0709 13:02:33.545795 (+680369us) op_driver.cc:464] REPLICATION: finished
0709 13:02:33.767876 (+222081us) write_op.cc:301] APPLY: starting
0709 13:02:33.768091 (+   215us) tablet.cc:1367] starting BulkCheckPresence
0709 13:02:33.903377 (+135286us) tablet.cc:1370] finished BulkCheckPresence
0709 13:02:33.928770 (+ 25393us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 13:02:34.384953 (+456183us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 13:02:34.385428 (+   475us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=32,key_file_lookups=32,delta_file_lookups=32,mrs_lookups=0
0709 13:02:34.385494 (+    66us) write_op.cc:312] APPLY: finished
0709 13:02:34.476010 (+ 90516us) log.cc:844] Serialized 148 byte log entry
0709 13:02:34.503056 (+ 27046us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:02:34.503594 (+   538us) write_op.cc:454] Released schema lock
0709 13:02:34.503715 (+   121us) write_op.cc:341] FINISH: Updating metrics
0709 13:02:34.504424 (+   709us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:02:34.504464 (+    40us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":6552661,"reported_tablets":47,"child_traces":[["op",{"apply.queue_time_us":221566,"apply.run_cpu_time_us":36217,"apply.run_wall_time_us":736595,"cfile_cache_hit":66,"cfile_cache_hit_bytes":71135,"num_ops":16,"prepare.queue_time_us":450154,"prepare.run_cpu_time_us":23264,"prepare.run_wall_time_us":618888,"raft.queue_time_us":192028,"raft.run_cpu_time_us":12166,"raft.run_wall_time_us":19528,"replication_time_us":715141,"thread_start_us":720034,"threads_started":4,"wal-append.queue_time_us":306434,"wal-append.run_cpu_time_us":42979,"wal-append.run_wall_time_us":2427846}]]}
I20260709 13:02:45.658713  1601 raft_consensus.cc:1275] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 20 index: 13. Preceding OpId from leader: term: 22 index: 14. (index mismatch)
I20260709 13:02:45.663352  1494 catalog_manager.cc:5697] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 13 to 20, leader changed from af520b67945d443b901bbeca37cc5701 (127.0.107.195) to 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196). New cstate: current_term: 20 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 13:02:45.566474  9515 consensus_queue.cc:260] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.10, Last appended by leader: 10, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:45.635030  9542 raft_consensus.cc:2749] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 13:02:45.632426  9812 raft_consensus.cc:740] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
W20260709 13:02:45.642593  9801 log.cc:927] Time spent T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.121s	user 0.001s	sys 0.000s
I20260709 13:02:45.618036  9714 consensus_queue.cc:1048] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.037s
I20260709 13:02:45.638586  9780 raft_consensus.cc:2749] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 13:02:45.674095  1494 catalog_manager.cc:5697] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 10 to 14, leader changed from 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) to 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193). New cstate: current_term: 14 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 13:02:45.675976  9515 raft_consensus.cc:3060] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:02:45.638715  9808 raft_consensus.cc:740] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
W20260709 13:02:45.695905  9802 log.cc:927] Time spent T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.136s	user 0.000s	sys 0.000s
W20260709 13:02:45.696856  9803 log.cc:927] Time spent T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.151s	user 0.000s	sys 0.001s
I20260709 13:02:45.638424  9816 raft_consensus.cc:2749] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:02:45.678215  9820 raft_consensus.cc:2804] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 13:02:45.640309  9760 consensus_queue.cc:237] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 21.11, Last appended by leader: 11, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:02:45.741932  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 183192us
I20260709 13:02:45.684965  9809 raft_consensus.cc:2804] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 27 FOLLOWER]: Leader election won for term 27
W20260709 13:02:45.748514  1646 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:45.657591  9634 raft_consensus.cc:515] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:45.754355  9709 raft_consensus.cc:493] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:45.731184  9588 consensus_queue.cc:1048] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.024s
I20260709 13:02:45.755239  9709 raft_consensus.cc:515] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:45.649887  9810 consensus_queue.cc:1048] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.088s
I20260709 13:02:45.717808  9226 consensus_queue.cc:1059] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 21.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.033s
W20260709 13:02:45.763934  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 218300us
I20260709 13:02:45.765805  2301 leader_election.cc:304] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:02:45.750413  1494 catalog_manager.cc:5697] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 14 to 16, leader changed from 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) to 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196). New cstate: current_term: 16 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 13:02:45.669970  9783 raft_consensus.cc:515] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:45.668597  9757 raft_consensus.cc:515] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:45.782586  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:45.782912  1644 leader_election.cc:304] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:02:45.783339  1754 raft_consensus.cc:3060] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Advancing to term 24
I20260709 13:02:45.786651  1753 raft_consensus.cc:1240] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 21. Current term is 22. Ops: []
I20260709 13:02:45.757050  9820 raft_consensus.cc:697] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 24 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:02:45.806252  1718 leader_election.cc:304] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:45.806681  8968 consensus_queue.cc:1048] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.009s
I20260709 13:02:45.768021  9809 raft_consensus.cc:697] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 27 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:02:45.819698  9489 raft_consensus.cc:493] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:45.827194  9826 consensus_queue.cc:1048] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.016s
I20260709 13:02:45.827732  9489 raft_consensus.cc:515] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:45.829154  9819 consensus_queue.cc:1048] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.109s
I20260709 13:02:45.830796  9815 raft_consensus.cc:2804] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 13:02:45.831284  9815 raft_consensus.cc:493] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:45.831686  9815 raft_consensus.cc:3060] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Advancing to term 24
I20260709 13:02:45.792130  1494 catalog_manager.cc:5697] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 14 to 18, leader changed from af520b67945d443b901bbeca37cc5701 (127.0.107.195) to 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193). New cstate: current_term: 18 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } }
I20260709 13:02:45.838810  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:45.839365  9821 raft_consensus.cc:2749] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
W20260709 13:02:45.839995  9813 log.cc:927] Time spent T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.126s	user 0.001s	sys 0.001s
I20260709 13:02:45.840273  9226 raft_consensus.cc:2804] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 24 FOLLOWER]: Leader election won for term 24
I20260709 13:02:45.840849  9226 raft_consensus.cc:697] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 24 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:02:45.843667  9808 consensus_queue.cc:260] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 21.15, Last appended by leader: 15, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:45.844549  9757 leader_election.cc:290] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:45.845837  9808 raft_consensus.cc:3060] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:02:45.839556  1745 raft_consensus.cc:2468] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 22.
I20260709 13:02:45.846588  9513 raft_consensus.cc:3055] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 21 LEADER]: Stepping down as leader of term 21
W20260709 13:02:45.846067  1495 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35458 (request call id 311) took 22300 ms (client timeout 15000 ms). Trace:
W20260709 13:02:45.895251  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 144802us
I20260709 13:02:45.847220  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:45.905488  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:02:45.911666  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 557455us
I20260709 13:02:45.830323  9489 leader_election.cc:290] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:02:45.909484  9811 log.cc:927] Time spent T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.062s	user 0.001s	sys 0.005s
I20260709 13:02:45.918888  1718 leader_election.cc:304] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:45.835968  8994 raft_consensus.cc:493] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:45.937351  1753 raft_consensus.cc:3060] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 26 FOLLOWER]: Advancing to term 27
I20260709 13:02:45.926376  8968 raft_consensus.cc:2804] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 13:02:45.938674  8968 raft_consensus.cc:493] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:02:45.939119  8968 raft_consensus.cc:3060] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:02:45.940204  1494 catalog_manager.cc:5697] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 11 to 18. New cstate: current_term: 18 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: HEALTHY } } }
I20260709 13:02:45.941210  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:45.892760  9513 raft_consensus.cc:740] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
W20260709 13:02:45.942633  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 408498us
W20260709 13:02:45.945955  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 399783us
W20260709 13:02:45.943092  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 114044us
I20260709 13:02:45.952199  1494 catalog_manager.cc:5697] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 12 to 15, leader changed from 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) to 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193). New cstate: current_term: 15 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:02:45.955711  1644 leader_election.cc:304] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:02:45.956219  1494 catalog_manager.cc:5697] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 11 to 14, leader changed from 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) to 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193). New cstate: current_term: 14 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:02:45.842028  9226 consensus_queue.cc:237] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 22.16, Last appended by leader: 16, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:45.961208  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 342974us
I20260709 13:02:45.964329  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:45.967579  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 18 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:02:45.897948  1495 rpcz_store.cc:269] 0709 13:02:23.546312 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:45.845775 (+22299463us) service_pool.cc:211] Skipping call since client already timed out
0709 13:02:45.845950 (+   175us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260709 13:02:45.969751  1494 catalog_manager.cc:5697] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 10 to 12. New cstate: current_term: 12 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
W20260709 13:02:45.972232  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 61614.8us
I20260709 13:02:45.966992  9542 raft_consensus.cc:2804] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 27 FOLLOWER]: Leader election won for term 27
I20260709 13:02:45.984390  8968 raft_consensus.cc:515] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:02:45.977771  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 363953us
W20260709 13:02:45.971851  9424 consensus_meta.cc:287] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.594s	user 0.015s	sys 0.000s
I20260709 13:02:45.982666  1717 leader_election.cc:304] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:45.987885  1569 leader_election.cc:304] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:02:45.996874  1643 leader_election.cc:304] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:02:45.813298  9807 consensus_queue.cc:260] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 21.11, Last appended by leader: 11, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:45.944949  8994 raft_consensus.cc:515] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:46.000038  2298 leader_election.cc:304] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:02:46.018692  8968 leader_election.cc:290] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 23 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 13:02:46.020042  9824 log.cc:927] Time spent T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.086s	user 0.001s	sys 0.000s
I20260709 13:02:46.020335  2337 raft_consensus.cc:1240] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 21. Current term is 22. Ops: []
I20260709 13:02:46.008569  1718 leader_election.cc:304] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:46.001526  1494 catalog_manager.cc:5697] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 3 to 16. New cstate: current_term: 16 leader_uuid: "af520b67945d443b901bbeca37cc5701" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } }
W20260709 13:02:46.032496  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 60079.5us
I20260709 13:02:45.995741  9704 leader_election.cc:290] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:02:46.017236  9806 log.cc:927] Time spent T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.244s	user 0.002s	sys 0.000s
I20260709 13:02:46.002381  9722 raft_consensus.cc:3055] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 23 LEADER]: Stepping down as leader of term 23
W20260709 13:02:46.043184  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 46714.2us
I20260709 13:02:46.044063  8968 raft_consensus.cc:2804] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 13:02:46.046962  8968 raft_consensus.cc:697] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 21 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:02:46.003332  9542 raft_consensus.cc:697] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 27 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:02:45.922871  9812 consensus_queue.cc:260] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 23.13, Last appended by leader: 13, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:46.058179  2299 leader_election.cc:304] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
W20260709 13:02:46.063632  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 89313.2us
I20260709 13:02:46.065310  9835 raft_consensus.cc:2804] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 13:02:46.017416  9809 consensus_queue.cc:237] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 25.16, Last appended by leader: 16, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:46.066356  9832 raft_consensus.cc:2804] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 13:02:46.067061  9832 raft_consensus.cc:697] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 23 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:02:46.068094  9832 consensus_queue.cc:237] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 21.16, Last appended by leader: 16, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:46.065374  9513 consensus_queue.cc:260] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.14, Last appended by leader: 14, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:46.007596  9424 raft_consensus.cc:515] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:46.045745  9807 raft_consensus.cc:3060] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:02:46.085176  1718 leader_election.cc:304] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 13:02:46.086218  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 112872us
W20260709 13:02:46.062325  9805 log.cc:927] Time spent T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.159s	user 0.000s	sys 0.002s
I20260709 13:02:46.051555  9810 raft_consensus.cc:2749] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:02:46.050175  9722 raft_consensus.cc:740] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:02:46.017904  9820 consensus_queue.cc:237] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 22.11, Last appended by leader: 11, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:02:46.203976  9833 log.cc:927] Time spent T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.133s	user 0.000s	sys 0.000s
I20260709 13:02:46.195935  9829 raft_consensus.cc:493] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:46.076303  9835 raft_consensus.cc:697] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 20 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:02:46.055959  9757 consensus_queue.cc:1059] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 23.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.054s
I20260709 13:02:46.083374  9783 leader_election.cc:290] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 22 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:46.216581  9818 raft_consensus.cc:2749] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260709 13:02:46.217478  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 144491us
I20260709 13:02:46.189982  9704 raft_consensus.cc:2749] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
W20260709 13:02:46.062961  1495 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35464 (request call id 290) took 21536 ms (client timeout 15000 ms). Trace:
I20260709 13:02:46.381294  1752 raft_consensus.cc:2393] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 18: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
W20260709 13:02:46.369370  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 292304us
I20260709 13:02:46.358623  1569 leader_election.cc:304] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:02:46.095158  8968 consensus_queue.cc:237] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.13, Last appended by leader: 13, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:46.391085  9700 consensus_meta.cc:287] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 1.065s	user 0.014s	sys 0.008s
I20260709 13:02:46.198508  9542 consensus_queue.cc:237] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 25.18, Last appended by leader: 18, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:46.388726  1754 consensus_meta.cc:287] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.600s	user 0.015s	sys 0.000s
I20260709 13:02:46.446406  1754 raft_consensus.cc:2468] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 24.
I20260709 13:02:46.408208  1643 leader_election.cc:304] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:46.410837  1602 raft_consensus.cc:1275] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 16 index: 13. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
I20260709 13:02:46.411455  9589 raft_consensus.cc:493] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:46.435246  2299 leader_election.cc:400] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 26 pre-election: Vote denied by peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261) with higher term. Message: Invalid argument: T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 for earlier term 26. Current term is 27.
W20260709 13:02:46.406548  9502 consensus_meta.cc:287] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.912s	user 0.008s	sys 0.018s
I20260709 13:02:46.381475  9594 raft_consensus.cc:2749] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
W20260709 13:02:46.440303  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:46.424427  1603 raft_consensus.cc:1275] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20260709 13:02:46.085935  9826 consensus_queue.cc:1048] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.098s
W20260709 13:02:46.409664  9744 consensus_meta.cc:287] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.840s	user 0.019s	sys 0.001s
I20260709 13:02:46.446857  1753 raft_consensus.cc:2468] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 27.
W20260709 13:02:46.417227  9703 consensus_meta.cc:287] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.888s	user 0.015s	sys 0.007s
I20260709 13:02:46.450652  2299 leader_election.cc:403] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 26 pre-election: Cancelling election due to peer responding with higher term
W20260709 13:02:46.454200  1645 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.532s	user 0.009s	sys 0.022s
W20260709 13:02:46.454684  1717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:46.461561  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 22 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:46.461130  1646 leader_election.cc:304] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1
W20260709 13:02:46.474731  1645 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.552s	user 0.010s	sys 0.022s
W20260709 13:02:46.478116  1645 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:46.479444  1754 raft_consensus.cc:3060] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:02:46.480293  1596 raft_consensus.cc:1240] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 21. Current term is 22. Ops: [21.11-21.11]
W20260709 13:02:46.498766  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 405315us
I20260709 13:02:46.496948  1603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 23 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
W20260709 13:02:46.445559  1495 rpcz_store.cc:269] 0709 13:02:24.513471 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:46.025537 (+21512066us) service_pool.cc:211] Skipping call since client already timed out
0709 13:02:46.049414 (+ 23877us) inbound_call.cc:177] Queueing failure response
Metrics: {}
W20260709 13:02:46.511909  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 127840us
I20260709 13:02:46.516422  1569 leader_election.cc:304] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 13:02:46.510779  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:46.523559  1600 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 1989) took 1180 ms. Trace:
I20260709 13:02:46.465724  9589 raft_consensus.cc:515] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:46.455271  9700 raft_consensus.cc:515] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:46.503149  9502 raft_consensus.cc:2749] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 13:02:46.449688  9829 raft_consensus.cc:515] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:46.543619  1754 raft_consensus.cc:2468] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 22.
I20260709 13:02:46.506898  9666 raft_consensus.cc:2749] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 27 FOLLOWER]: Leader pre-election lost for term 26. Reason: Vote denied by peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261) with higher term. Message: Invalid argument: T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 for earlier term 26. Current term is 27.
I20260709 13:02:46.547384  1643 leader_election.cc:304] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:02:46.528241  1600 rpcz_store.cc:276] 0709 13:02:45.343602 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:45.343798 (+   196us) service_pool.cc:224] Handling call
0709 13:02:45.347058 (+  3260us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:45.347569 (+   511us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:45.347597 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:45.348103 (+   506us) log.cc:844] Serialized 26 byte log entry
0709 13:02:45.494670 (+146567us) raft_consensus.cc:1645] Marking committed up to 16
0709 13:02:45.494744 (+    74us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:45.494854 (+   110us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:46.523292 (+1028438us) raft_consensus.cc:1685] finished
0709 13:02:46.523323 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:46.523480 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1368,"raft.run_cpu_time_us":110,"raft.run_wall_time_us":110,"thread_start_us":146035,"threads_started":1,"wal-append.queue_time_us":320756}
I20260709 13:02:46.479931  9744 raft_consensus.cc:515] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:02:46.557965  1719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:46.542207  9703 raft_consensus.cc:2804] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:02:46.562249  1603 raft_consensus.cc:2468] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 22.
I20260709 13:02:46.563092  1646 leader_election.cc:304] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:46.544766  9830 raft_consensus.cc:2804] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 13:02:46.549429  9826 raft_consensus.cc:493] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:46.555326  9837 log.cc:927] Time spent T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.293s	user 0.000s	sys 0.008s
W20260709 13:02:46.570503  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 54287us
I20260709 13:02:46.507249  9722 consensus_queue.cc:260] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 23.12, Last appended by leader: 12, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:46.545313  9818 consensus_queue.cc:1048] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.026s
I20260709 13:02:46.564699  9703 raft_consensus.cc:493] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:46.568575  9502 raft_consensus.cc:2804] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Leader election won for term 22
W20260709 13:02:46.583890  9808 consensus_meta.cc:287] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.691s	user 0.021s	sys 0.008s
I20260709 13:02:46.573180  9810 raft_consensus.cc:2749] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
W20260709 13:02:46.595640  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 48396.2us
I20260709 13:02:46.568735  9830 raft_consensus.cc:493] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
W20260709 13:02:46.577461  9815 consensus_meta.cc:287] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.745s	user 0.014s	sys 0.002s
W20260709 13:02:46.587092  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 225140us
W20260709 13:02:46.619634  9515 consensus_meta.cc:287] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.894s	user 0.015s	sys 0.008s
I20260709 13:02:46.632038  9815 raft_consensus.cc:515] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:46.633183  2299 leader_election.cc:304] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
W20260709 13:02:46.584564  9668 consensus_meta.cc:287] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.978s	user 0.009s	sys 0.009s
I20260709 13:02:46.634747  9668 raft_consensus.cc:515] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:46.586792  9710 consensus_meta.cc:287] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.974s	user 0.014s	sys 0.010s
W20260709 13:02:46.612686  9476 consensus_meta.cc:287] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.939s	user 0.012s	sys 0.011s
I20260709 13:02:46.635054  2299 leader_election.cc:304] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:02:46.574105  9826 raft_consensus.cc:515] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:46.614790  9703 raft_consensus.cc:3060] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:02:46.584231  9844 raft_consensus.cc:493] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
W20260709 13:02:46.640188  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 135252us
I20260709 13:02:46.630312  1569 leader_election.cc:304] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:02:46.583745  9823 consensus_queue.cc:1059] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 21.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.159s
I20260709 13:02:46.647338  9700 leader_election.cc:290] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 24 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:02:46.647913  9842 log.cc:927] Time spent T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.063s	user 0.001s	sys 0.000s
I20260709 13:02:46.650527  9834 consensus_queue.cc:1048] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.102s
I20260709 13:02:46.636297  9722 raft_consensus.cc:3060] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 23 FOLLOWER]: Advancing to term 24
I20260709 13:02:46.614089  9841 raft_consensus.cc:493] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:46.667713  2337 raft_consensus.cc:1240] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 20. Current term is 21. Ops: []
W20260709 13:02:46.641038  9831 log.cc:927] Time spent T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.089s	user 0.000s	sys 0.001s
I20260709 13:02:46.670862  8968 raft_consensus.cc:493] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:46.630978  9825 raft_consensus.cc:493] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:46.636464  9830 raft_consensus.cc:3060] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:02:46.636133  9502 raft_consensus.cc:697] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 22 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:02:46.639632  9816 raft_consensus.cc:2749] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260709 13:02:46.726298  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 82870.9us
I20260709 13:02:46.549554  9835 consensus_queue.cc:237] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.13, Last appended by leader: 13, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:46.672277  9809 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.603s	user 0.007s	sys 0.000s
I20260709 13:02:46.635885  9840 raft_consensus.cc:2804] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:02:46.675709  9666 raft_consensus.cc:2764] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Leader election decision vote started in defunct term 22: won
I20260709 13:02:46.679142  9815 leader_election.cc:290] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 24 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:46.653419  9710 raft_consensus.cc:515] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:46.735539  8968 raft_consensus.cc:515] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:46.769969  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:46.776779  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:02:46.779814  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 144939us
I20260709 13:02:46.801615  1748 raft_consensus.cc:2468] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 21.
W20260709 13:02:46.779451  9809 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.707s	user 0.008s	sys 0.000s
I20260709 13:02:46.770251  9760 raft_consensus.cc:493] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:46.806224  1643 leader_election.cc:304] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:02:46.806517  1597 raft_consensus.cc:2468] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 21.
I20260709 13:02:46.680006  9844 raft_consensus.cc:515] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:46.812160  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:46.812732  2299 leader_election.cc:304] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 13:02:46.813555  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 203302us
I20260709 13:02:46.815539  1597 raft_consensus.cc:2468] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 20.
I20260709 13:02:46.821492  1751 raft_consensus.cc:1240] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 25 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 24. Current term is 25. Ops: []
I20260709 13:02:46.823870  8968 leader_election.cc:290] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:46.784405  9589 leader_election.cc:290] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:46.797384  9840 raft_consensus.cc:493] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Starting leader election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:02:46.750097  9841 raft_consensus.cc:515] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:46.846936  1569 leader_election.cc:304] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:02:46.772866  9825 raft_consensus.cc:515] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:46.835954  9666 raft_consensus.cc:2749] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:02:46.853540  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 24 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:46.851204  2335 raft_consensus.cc:1240] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 21. Current term is 22. Ops: []
I20260709 13:02:46.836433  9840 raft_consensus.cc:3060] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:02:46.850744  9744 leader_election.cc:290] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 19 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:46.857044  9817 raft_consensus.cc:493] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:46.857477  9836 log.cc:927] Time spent T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.319s	user 0.000s	sys 0.002s
I20260709 13:02:46.857867  9853 raft_consensus.cc:3055] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 LEADER]: Stepping down as leader of term 21
W20260709 13:02:46.860770  9862 log.cc:927] Time spent T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.052s	user 0.000s	sys 0.000s
W20260709 13:02:46.861754  9858 log.cc:927] Time spent T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.060s	user 0.000s	sys 0.000s
W20260709 13:02:46.863327  9846 log.cc:927] Time spent T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.253s	user 0.001s	sys 0.000s
I20260709 13:02:46.864414  9844 leader_election.cc:290] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:02:46.913997  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 73637.7us
W20260709 13:02:46.866349  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 70370.4us
I20260709 13:02:46.824757  9760 raft_consensus.cc:515] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:46.867312  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 54684.4us
W20260709 13:02:46.868448  9859 log.cc:927] Time spent T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.073s	user 0.000s	sys 0.000s
I20260709 13:02:46.870184  9841 leader_election.cc:290] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:46.851938  9502 consensus_queue.cc:237] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.10, Last appended by leader: 10, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:46.872216  9668 leader_election.cc:290] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 21 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:46.872560  9825 leader_election.cc:290] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:02:46.873641  9807 consensus_meta.cc:287] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.783s	user 0.010s	sys 0.002s
I20260709 13:02:46.874033  9829 leader_election.cc:290] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:46.843852  9226 raft_consensus.cc:2804] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Leader election won for term 22
I20260709 13:02:46.842239  9700 raft_consensus.cc:493] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:46.901153  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 77407.9us
I20260709 13:02:46.865756  9710 leader_election.cc:290] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 22 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:46.851573  9871 raft_consensus.cc:2749] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 13:02:46.918292  2299 leader_election.cc:304] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:02:46.927296  1751 raft_consensus.cc:1240] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 24 LEADER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 23. Current term is 24. Ops: [23.12-23.12]
I20260709 13:02:46.905714  9853 raft_consensus.cc:740] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:02:46.935117  9844 consensus_queue.cc:1059] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 24.12, Next index: 13, Last known committed idx: 11, Time since last communication: 0.001s
W20260709 13:02:46.952920  1719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:46.945813  9873 raft_consensus.cc:2804] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 24 FOLLOWER]: Leader pre-election won for term 25
W20260709 13:02:46.957073  1570 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:46.969056  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 50891us
I20260709 13:02:46.969837  9668 raft_consensus.cc:2749] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 13:02:46.957057  1569 leader_election.cc:400] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 21 pre-election: Vote denied by peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261) with higher term. Message: Invalid argument: T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 21. Current term is 22.
I20260709 13:02:46.945930  9841 raft_consensus.cc:493] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:46.906013  9817 raft_consensus.cc:515] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:46.982220  1598 raft_consensus.cc:1275] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 27 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 25 index: 16. Preceding OpId from leader: term: 27 index: 17. (index mismatch)
I20260709 13:02:46.984289  1643 leader_election.cc:304] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:02:46.986938  1597 raft_consensus.cc:3060] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:02:46.976409  9873 raft_consensus.cc:493] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 24 FOLLOWER]: Starting leader election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:02:46.988806  9873 raft_consensus.cc:3060] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 24 FOLLOWER]: Advancing to term 25
W20260709 13:02:46.993553  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 163697us
W20260709 13:02:46.994896  1645 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:46.992386  9760 leader_election.cc:290] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:46.994371  9871 raft_consensus.cc:493] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:46.973197  9226 raft_consensus.cc:697] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 22 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:02:46.997509  1605 raft_consensus.cc:1275] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 17 index: 11. (index mismatch)
I20260709 13:02:46.998302  1604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 23 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:46.968912  9700 raft_consensus.cc:515] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:47.002333  2299 leader_election.cc:304] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:02:47.010574  1598 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 22 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:47.011548  1604 raft_consensus.cc:2468] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 22.
I20260709 13:02:46.990249  9817 leader_election.cc:290] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:47.015405  2299 leader_election.cc:304] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:02:47.013697  9542 raft_consensus.cc:2804] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Leader pre-election won for term 21
W20260709 13:02:47.020989  1717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:47.021158  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 74314.4us
W20260709 13:02:46.997862  9872 log.cc:927] Time spent T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.060s	user 0.000s	sys 0.001s
W20260709 13:02:47.022281  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 54453.5us
I20260709 13:02:47.021914  9853 consensus_queue.cc:260] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.14, Last appended by leader: 14, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:47.010116  9841 raft_consensus.cc:515] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:47.017156  9666 raft_consensus.cc:2749] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 13:02:47.032492  1569 leader_election.cc:400] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 23 election: Vote denied by peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261) with higher term. Message: Invalid argument: T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 23. Current term is 24.
I20260709 13:02:47.026417  9830 raft_consensus.cc:515] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Starting leader election with config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:47.033751  1569 leader_election.cc:304] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:02:47.021741  9542 raft_consensus.cc:493] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:47.034947  9542 raft_consensus.cc:3060] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:02:47.039016  1643 leader_election.cc:304] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 13:02:47.040722  9648 raft_consensus.cc:2804] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:02:47.041085  9648 raft_consensus.cc:493] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:47.041430  9648 raft_consensus.cc:3060] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:02:46.998003  9871 raft_consensus.cc:515] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:47.047366  9880 raft_consensus.cc:2749] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:02:47.047098  9783 consensus_queue.cc:1048] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 13:02:47.047147  9830 leader_election.cc:290] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 23 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:47.030359  9841 leader_election.cc:290] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:47.047761  1597 raft_consensus.cc:1275] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 21 index: 11. Preceding OpId from leader: term: 22 index: 12. (index mismatch)
I20260709 13:02:47.026793  9853 raft_consensus.cc:3060] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:02:47.051091  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:46.999982  9502 raft_consensus.cc:493] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:47.028231  9700 leader_election.cc:290] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:47.054522  9502 raft_consensus.cc:515] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:47.056160  1748 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55840 (request call id 3272) took 1827 ms (client timeout 2092 ms). Trace:
I20260709 13:02:47.056869  9502 leader_election.cc:290] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:47.062621  1643 leader_election.cc:304] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:02:47.064987  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:47.065809  9871 leader_election.cc:290] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:47.065984  1673 raft_consensus.cc:2410] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 for term 20 because replica has last-logged OpId of term: 19 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 15.
W20260709 13:02:47.067682  9785 consensus_queue.cc:1175] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "3b879799899a499c8d409ee9f1277cd1" responder_term: 21 status { last_received { term: 21 index: 14 } last_committed_idx: 14 last_received_current_leader { term: 21 index: 14 } }
I20260709 13:02:47.069494  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:47.070384  1673 raft_consensus.cc:2410] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 for term 20 because replica has last-logged OpId of term: 19 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 15.
I20260709 13:02:47.070988  9700 raft_consensus.cc:2749] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 13:02:47.073428  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 23 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:47.074640  8994 leader_election.cc:290] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:47.077095  1673 raft_consensus.cc:1240] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 24 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 23. Current term is 24. Ops: [23.17-23.17]
I20260709 13:02:47.078827  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 19 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" is_pre_election: true
I20260709 13:02:47.078994  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 25 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:47.079723  1597 raft_consensus.cc:2410] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 for term 19 because replica has last-logged OpId of term: 17 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 12.
I20260709 13:02:47.080542  9812 raft_consensus.cc:3060] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Advancing to term 24
I20260709 13:02:47.080789  1597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:47.081496  1597 raft_consensus.cc:3060] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Advancing to term 22
W20260709 13:02:47.056629  1748 rpcz_store.cc:269] 0709 13:02:45.228782 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:45.263528 (+ 34746us) service_pool.cc:224] Handling call
0709 13:02:47.055932 (+1792404us) spinlock_profiling.cc:243] Waited 1.49 s on lock 0x7b54001d56d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:02:47.056060 (+   128us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3437016960}
I20260709 13:02:47.073555  1643 leader_election.cc:304] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1
W20260709 13:02:47.094072  1646 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:47.094084  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 23 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:47.046829  9588 raft_consensus.cc:2749] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 13:02:47.110404  1676 raft_consensus.cc:1240] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 22. Current term is 23. Ops: []
I20260709 13:02:47.136935  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 25 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:47.111359  2335 raft_consensus.cc:1240] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 21. Current term is 22. Ops: [21.15-21.15]
I20260709 13:02:47.112586  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:47.116050  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 24 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
W20260709 13:02:47.117007  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 43842.8us
I20260709 13:02:47.118620  9515 raft_consensus.cc:493] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:47.125561  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:47.131191  1597 raft_consensus.cc:2468] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 22.
W20260709 13:02:47.131722  1645 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:47.132028  9598 raft_consensus.cc:3055] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 24 LEADER]: Stepping down as leader of term 24
I20260709 13:02:47.132896  1746 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55848 (request call id 3537) took 1099 ms. Trace:
I20260709 13:02:47.064110  9634 leader_election.cc:290] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:47.166819  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:47.167869  1671 raft_consensus.cc:1275] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20260709 13:02:47.167266  2298 leader_election.cc:304] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:02:47.168254  2330 raft_consensus.cc:2393] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 27: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:02:47.134492  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:47.145244  1748 raft_consensus.cc:2393] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 24: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
W20260709 13:02:47.161480  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 52705.6us
I20260709 13:02:47.167650  1678 raft_consensus.cc:1275] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 16 index: 13. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
I20260709 13:02:47.171913  9515 raft_consensus.cc:515] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:47.179571  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:47.181576  9515 leader_election.cc:290] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:47.183620  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 24 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:47.184283  2330 raft_consensus.cc:2468] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 23.
I20260709 13:02:47.175577  1643 leader_election.cc:304] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:47.164860  1746 rpcz_store.cc:276] 0709 13:02:46.033787 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:46.040510 (+  6723us) service_pool.cc:224] Handling call
0709 13:02:46.056028 (+ 15518us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 13:02:46.362080 (+306052us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:46.362137 (+    57us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 13:02:46.440815 (+ 78678us) log.cc:844] Serialized 78 byte log entry
0709 13:02:46.639145 (+198330us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:02:46.639212 (+    67us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:46.639305 (+    93us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:47.120739 (+481434us) spinlock_profiling.cc:243] Waited 11.7 ms on lock 0x7b240024b2e8. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f4262efb9c4 00007f4262ef9acb 00007f42630b4dc9 00007f42630b4a55 00007f42630b3fb0 00007f4265d2a488 00007f4265d2abe8 00007f4265d02398 00007f4265cfd76a 00007f4265cfcbb6
0709 13:02:47.132524 (+ 11785us) raft_consensus.cc:1685] finished
0709 13:02:47.132550 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:47.132802 (+   252us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":220142,"raft.queue_time_us":19540,"raft.run_cpu_time_us":187,"raft.run_wall_time_us":44237,"spinlock_wait_cycles":26905472,"thread_start_us":197835,"threads_started":1,"wal-append.queue_time_us":283846}
W20260709 13:02:47.194497  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 160868us
I20260709 13:02:47.195863  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:47.198699  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:02:47.201828  9703 consensus_meta.cc:287] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.517s	user 0.012s	sys 0.002s
I20260709 13:02:47.202348  9703 raft_consensus.cc:515] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:47.204692  9703 leader_election.cc:290] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 18 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:47.205467  1604 raft_consensus.cc:1240] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 17. Current term is 18. Ops: []
I20260709 13:02:47.206202  9873 raft_consensus.cc:515] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:47.208317  1745 raft_consensus.cc:1240] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 23. Current term is 24. Ops: []
I20260709 13:02:47.208523  9873 leader_election.cc:290] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 25 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:47.210700  1569 leader_election.cc:304] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:02:47.167030  9598 raft_consensus.cc:740] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:02:47.189134  8994 raft_consensus.cc:2804] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 13:02:47.220963  1754 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55848 (request call id 3539) took 1145 ms. Trace:
/apache/dev/git/kudu/src/kudu/master/auto_leader_rebalancer-test.cc:278: Failure
Failed
Bad status: Illegal state: leader_transfer_task size should be 0, but 5
I20260709 13:02:47.225979  1754 rpcz_store.cc:276] 0709 13:02:46.075613 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:46.548957 (+473344us) service_pool.cc:224] Handling call
0709 13:02:46.550191 (+  1234us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:46.551898 (+  1707us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 13:02:46.569088 (+ 17190us) log.cc:844] Serialized 14 byte log entry
0709 13:02:47.043347 (+474259us) spinlock_profiling.cc:243] Waited 4.62 ms on lock 0x7b1800412660. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f4263057e07 00007f426beb2ab9 00007f426307469b 00007f426307293f 00007f4263074c02 00007f4265cfc20b 00007f4265d01d23 00007f4265d0223a 00007f426616ca2a
0709 13:02:47.198033 (+154686us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:47.198095 (+    62us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:02:47.198164 (+    69us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:47.198255 (+    91us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:47.220895 (+ 22640us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":16369,"prepare.run_cpu_time_us":58,"prepare.run_wall_time_us":61,"raft.queue_time_us":5498,"raft.run_cpu_time_us":152,"raft.run_wall_time_us":8601,"spinlock_wait_cycles":10619648,"thread_start_us":149016,"threads_started":1}
I20260709 13:02:47.220072  9783 raft_consensus.cc:2749] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:02:47.240845  9873 raft_consensus.cc:2749] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260709 13:02:47.229708  9750 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.552s	user 0.011s	sys 0.018s
I20260709 13:02:47.244004  9542 raft_consensus.cc:515] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:47.251286  1754 raft_consensus.cc:1240] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 21. Current term is 22. Ops: []
I20260709 13:02:47.254931  1746 raft_consensus.cc:1275] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 27 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 23 index: 17. Preceding OpId from leader: term: 27 index: 19. (index mismatch)
W20260709 13:02:47.256418  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 45842.2us
I20260709 13:02:47.257926  2331 raft_consensus.cc:1240] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 20. Current term is 21. Ops: [20.14-20.14]
/apache/dev/git/kudu/src/kudu/util/test_util.cc:402: Failure
Failed
Timed out waiting for assertion to pass.
I20260709 13:02:47.229200  8994 raft_consensus.cc:697] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 22 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
W20260709 13:02:47.259212  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 83761.2us
I20260709 13:02:47.213119  9226 consensus_queue.cc:237] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.10, Last appended by leader: 10, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:02:47.253960  9750 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.581s	user 0.011s	sys 0.018s
I20260709 13:02:47.290791  1747 raft_consensus.cc:1240] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 17. Current term is 18. Ops: []
I20260709 13:02:47.272679  1746 raft_consensus.cc:1240] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 21. Current term is 22. Ops: []
I20260709 13:02:47.275540  1643 leader_election.cc:304] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 13:02:47.298041  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:47.299115  2331 raft_consensus.cc:2368] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 23. Current term is 24.
I20260709 13:02:47.300819  2330 raft_consensus.cc:1240] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 20 LEADER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 19. Current term is 20. Ops: []
W20260709 13:02:47.290060  1752 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55840 (request call id 3269) took 2062 ms (client timeout 1762 ms). Trace:
I20260709 13:02:47.288740  1596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "83595ad0fd5940c0a2bfc7c61acd969a"
I20260709 13:02:47.307211  1596 raft_consensus.cc:3060] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Advancing to term 18
W20260709 13:02:47.303443  1752 rpcz_store.cc:269] 0709 13:02:45.227893 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:45.228026 (+   133us) service_pool.cc:224] Handling call
0709 13:02:47.289737 (+2061711us) spinlock_profiling.cc:243] Waited 767 ms on lock 0x7b5400324bd4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:02:47.289948 (+   211us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1765073152}
W20260709 13:02:47.286240  9849 log.cc:927] Time spent T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.754s	user 0.003s	sys 0.000s
I20260709 13:02:47.314391  1753 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55848 (request call id 3540) took 1181 ms. Trace:
I20260709 13:02:47.314239  1569 leader_election.cc:304] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:02:47.287587  9513 raft_consensus.cc:3060] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Advancing to term 22
W20260709 13:02:47.316341  1718 leader_election.cc:336] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 18 pre-election: RPC error from VoteRequest() call to peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625): Timed out: RequestConsensusVote RPC to 127.0.107.196:46625 timed out after 2.518s (SENT)
I20260709 13:02:47.253981  9424 leader_election.cc:290] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 18 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:47.318392  1753 rpcz_store.cc:276] 0709 13:02:46.133601 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:46.803049 (+669448us) service_pool.cc:224] Handling call
0709 13:02:46.835643 (+ 32594us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:46.850660 (+ 15017us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:46.850706 (+    46us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:47.133031 (+282325us) spinlock_profiling.cc:243] Waited 3.77 ms on lock 0x7b2c000bc788. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4264a5ba11 00007f4264a6b372 00007f426619da01 00007f426616243a 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2
0709 13:02:47.133497 (+   466us) log.cc:844] Serialized 26 byte log entry
0709 13:02:47.272271 (+138774us) raft_consensus.cc:1645] Marking committed up to 15
0709 13:02:47.272325 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:47.272441 (+   116us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:47.296302 (+ 23861us) raft_consensus.cc:1685] finished
0709 13:02:47.296321 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:47.314278 (+ 17957us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":4617,"raft.run_cpu_time_us":165,"raft.run_wall_time_us":2884,"spinlock_wait_cycles":8671488,"thread_start_us":138288,"threads_started":1,"wal-append.queue_time_us":153479}
I20260709 13:02:47.321830  1718 leader_election.cc:304] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:47.328810  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:47.283896  9709 leader_election.cc:290] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:47.314623  9815 raft_consensus.cc:493] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:47.336514  1569 leader_election.cc:304] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:02:47.339745  2299 leader_election.cc:304] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:47.326988  9873 raft_consensus.cc:2804] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 13:02:47.337863  9814 raft_consensus.cc:2749] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:02:47.337728  9823 raft_consensus.cc:2804] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:02:47.291054  9598 consensus_queue.cc:260] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 24.12, Last appended by leader: 12, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:47.347114  9709 raft_consensus.cc:2749] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 13:02:47.345085  9873 raft_consensus.cc:493] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 24 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
W20260709 13:02:47.360107  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 54795.6us
I20260709 13:02:47.341579  9815 raft_consensus.cc:515] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:47.391207  9892 log.cc:927] Time spent T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.050s	user 0.001s	sys 0.000s
I20260709 13:02:47.347754  8994 consensus_queue.cc:237] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 21.17, Last appended by leader: 17, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:47.396555  2334 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39726 (request call id 1779) took 1581 ms. Trace:
I20260709 13:02:47.397184   431 tablet_server.cc:179] TabletServer@127.0.107.193:0 shutting down...
W20260709 13:02:47.400779  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 75538.2us
I20260709 13:02:47.363543  9776 raft_consensus.cc:2749] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 13:02:47.404177  1596 raft_consensus.cc:2468] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 18.
I20260709 13:02:47.366999  9598 raft_consensus.cc:3060] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 24 FOLLOWER]: Advancing to term 25
I20260709 13:02:47.397650  2334 rpcz_store.cc:276] 0709 13:02:45.814880 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:45.821082 (+  6202us) service_pool.cc:224] Handling call
0709 13:02:45.821482 (+   400us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:02:45.913535 (+ 92053us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:45.913575 (+    40us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:02:45.940996 (+ 27421us) log.cc:844] Serialized 52 byte log entry
0709 13:02:46.056620 (+115624us) raft_consensus.cc:1645] Marking committed up to 15
0709 13:02:46.083846 (+ 27226us) log.cc:844] Serialized 14 byte log entry
0709 13:02:46.390626 (+306780us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:46.390732 (+   106us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:47.026289 (+635557us) spinlock_profiling.cc:243] Waited 2.88 ms on lock 0x7b48002301d8. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426b08baa1 00007f426b082b37 00007f426b0829ca 00007f426b17a26f 00007f426b179f37 00007f426b18e55c 00007f426b18e508 00007f426b18e488 00007f426b18e443 00007f426b18d6ec 00007f426a0b7a96
0709 13:02:47.387914 (+361625us) spinlock_profiling.cc:243] Waited 10.3 ms on lock 0x7b18002cf260. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f42630753f3 00007f42630777c1 00007f4263077779 00007f4263077709 00007f42630776d1 00007f42630769cd 000000000038f354 000000000038f199 00007f426305883a
0709 13:02:47.396195 (+  8281us) raft_consensus.cc:1685] finished
0709 13:02:47.396241 (+    46us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:47.396437 (+   196us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":349302,"prepare.queue_time_us":34188,"prepare.run_cpu_time_us":10273,"prepare.run_wall_time_us":265085,"raft.queue_time_us":391745,"raft.run_cpu_time_us":10098,"raft.run_wall_time_us":1117887,"spinlock_wait_cycles":30264320,"thread_start_us":89315,"threads_started":1,"wal-append.queue_time_us":472437}
I20260709 13:02:47.367249  9873 raft_consensus.cc:3060] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 24 FOLLOWER]: Advancing to term 25
I20260709 13:02:47.419229  9542 leader_election.cc:290] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 21 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:47.440454  9823 raft_consensus.cc:493] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:47.441064  9823 raft_consensus.cc:3060] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Advancing to term 19
W20260709 13:02:47.434924  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 66568.4us
I20260709 13:02:47.448573  2299 leader_election.cc:304] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:02:47.437273  1569 leader_election.cc:304] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
W20260709 13:02:47.454350  1718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:47.453748  9821 raft_consensus.cc:2749] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
W20260709 13:02:47.461118  9899 log.cc:927] Time spent T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.052s	user 0.000s	sys 0.000s
W20260709 13:02:47.435823  9896 log.cc:927] Time spent T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.068s	user 0.000s	sys 0.000s
I20260709 13:02:47.462496  9709 raft_consensus.cc:2804] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:02:47.462972  9709 raft_consensus.cc:493] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:47.463390  9709 raft_consensus.cc:3060] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:02:47.402695  9840 raft_consensus.cc:515] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:47.480602  1601 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36462 (request call id 1998) took 1095 ms. Trace:
I20260709 13:02:47.481201  1601 rpcz_store.cc:276] 0709 13:02:46.385688 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:46.385852 (+   164us) service_pool.cc:224] Handling call
0709 13:02:46.419558 (+ 33706us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:47.077872 (+658314us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:47.077922 (+    50us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:47.077967 (+    45us) raft_consensus.cc:1645] Marking committed up to 15
0709 13:02:47.078001 (+    34us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:47.480224 (+402223us) spinlock_profiling.cc:243] Waited 658 ms on lock 0x7b54002a1cd4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:02:47.480310 (+    86us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:47.480509 (+   199us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1514045312}
I20260709 13:02:47.482393  9840 leader_election.cc:290] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 21 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:47.485847  1599 raft_consensus.cc:1240] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 20. Current term is 21. Ops: []
I20260709 13:02:47.480186  1569 leader_election.cc:304] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:02:47.480186  9815 leader_election.cc:290] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:47.492650  9358 consensus_queue.cc:1059] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 22.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 13:02:47.492834  9588 consensus_queue.cc:1048] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 13:02:47.494050  9875 raft_consensus.cc:3055] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 13:02:47.494616  9875 raft_consensus.cc:740] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:02:47.495795  9875 consensus_queue.cc:260] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.15, Last appended by leader: 15, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:47.497793  9875 raft_consensus.cc:3060] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:02:47.498893  1718 leader_election.cc:304] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:47.499452  2337 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39726 (request call id 1786) took 1429 ms. Trace:
I20260709 13:02:47.500029  2337 rpcz_store.cc:276] 0709 13:02:46.070469 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:46.070611 (+   142us) service_pool.cc:224] Handling call
0709 13:02:46.091812 (+ 21201us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:46.732610 (+640798us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:47.499069 (+766459us) spinlock_profiling.cc:243] Waited 576 ms on lock 0x7b540072e554. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:02:47.499340 (+   271us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1324358912}
I20260709 13:02:47.509348  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:47.509878  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 22 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:47.510371  2333 raft_consensus.cc:2468] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 21.
I20260709 13:02:47.515264  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:47.515872  1746 raft_consensus.cc:3060] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:02:47.523922  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 19 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:47.526894  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:47.527706  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 23 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:47.526966  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:47.530077  1750 raft_consensus.cc:2410] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 for term 19 because replica has last-logged OpId of term: 17 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 12.
I20260709 13:02:47.484385  9780 consensus_queue.cc:1059] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 21.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.021s
I20260709 13:02:47.535888  1745 raft_consensus.cc:1240] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 22. Current term is 23. Ops: []
I20260709 13:02:47.533545  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:47.529358  2329 raft_consensus.cc:2376] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 22. Re-sending same reply.
I20260709 13:02:47.527729  2335 raft_consensus.cc:2376] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 21. Re-sending same reply.
I20260709 13:02:47.518930  9815 raft_consensus.cc:2749] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 13:02:47.537496  2331 raft_consensus.cc:2376] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 23. Re-sending same reply.
I20260709 13:02:47.532258  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 24 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:47.526109  2333 raft_consensus.cc:1275] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 27 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 25 index: 16. Preceding OpId from leader: term: 27 index: 17. (index mismatch)
I20260709 13:02:47.520932  9714 raft_consensus.cc:493] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:47.502750  9776 consensus_queue.cc:1048] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.040s
I20260709 13:02:47.566146  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 18 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:47.568833  1718 leader_election.cc:400] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 23 election: Vote denied by peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625) with higher term. Message: Invalid argument: T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 23. Current term is 24.
I20260709 13:02:47.569216  1718 leader_election.cc:403] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 23 election: Cancelling election due to peer responding with higher term
I20260709 13:02:47.535909  9840 raft_consensus.cc:2749] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 13:02:47.570657  1750 raft_consensus.cc:2393] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 18: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
W20260709 13:02:47.573716  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 93649.9us
I20260709 13:02:47.576275  1569 leader_election.cc:304] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:02:47.574553  1718 leader_election.cc:304] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:47.580876  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:47.582782  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:47.569885  9714 raft_consensus.cc:515] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:47.616030  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:47.586616  2330 raft_consensus.cc:2393] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 21: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:02:47.585984  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:47.573928  9869 raft_consensus.cc:2749] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Leader election lost for term 23. Reason: Vote denied by peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625) with higher term. Message: Invalid argument: T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 23. Current term is 24.
I20260709 13:02:47.620275  2334 raft_consensus.cc:2376] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 20. Re-sending same reply.
I20260709 13:02:47.619647  2337 raft_consensus.cc:1275] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 18 index: 12. Preceding OpId from leader: term: 21 index: 14. (index mismatch)
I20260709 13:02:47.571126  9838 consensus_queue.cc:1059] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 23.17, Next index: 18, Last known committed idx: 16, Time since last communication: 0.047s
I20260709 13:02:47.627913  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:47.582813  9878 raft_consensus.cc:2764] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Leader pre-election decision vote started in defunct term 19: won
I20260709 13:02:47.586251  9815 raft_consensus.cc:2804] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 13:02:47.632227  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 23 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:47.637339  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 23 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:02:47.639331  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 63209.3us
I20260709 13:02:47.576591  9814 consensus_queue.cc:1059] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.056s
I20260709 13:02:47.626510  9424 raft_consensus.cc:493] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:47.684507  9424 raft_consensus.cc:515] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:47.686663  1569 leader_election.cc:304] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:02:47.686997  9424 leader_election.cc:290] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:02:47.688051  9812 consensus_meta.cc:287] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.607s	user 0.015s	sys 0.000s
I20260709 13:02:47.688239  1569 leader_election.cc:304] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:02:47.689611  9826 leader_election.cc:290] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:47.690652  9840 raft_consensus.cc:2749] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 13:02:47.645442  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:47.633157  9714 leader_election.cc:290] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:47.636898  9815 raft_consensus.cc:493] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:47.652276  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:02:47.654364  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 379743us
W20260709 13:02:47.655814  1570 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:47.703012  1570 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:47.656901  1644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:47.658836  1749 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55840 (request call id 3278) took 2427 ms (client timeout 1848 ms). Trace:
W20260709 13:02:47.659334  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:47.659505  1645 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:47.661616  1747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55848 (request call id 3526) took 1899 ms. Trace:
I20260709 13:02:47.664999  2301 leader_election.cc:304] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
W20260709 13:02:47.670913  9853 consensus_meta.cc:287] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.620s	user 0.012s	sys 0.003s
I20260709 13:02:47.671813  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:47.674906  1571 leader_election.cc:304] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:02:47.678663  9878 consensus_queue.cc:1059] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 22.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 13:02:47.681176  9666 raft_consensus.cc:3055] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 23 LEADER]: Stepping down as leader of term 23
I20260709 13:02:47.682428  9785 raft_consensus.cc:493] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:47.704313  1749 rpcz_store.cc:269] 0709 13:02:45.231639 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:45.241973 (+ 10334us) service_pool.cc:224] Handling call
0709 13:02:47.658593 (+2416620us) spinlock_profiling.cc:243] Waited 744 ms on lock 0x7b54001711d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:02:47.658734 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1710057728}
W20260709 13:02:47.714775  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 241369us
W20260709 13:02:47.718480  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 53600.6us
I20260709 13:02:47.715852  9812 raft_consensus.cc:493] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:47.723961  1569 leader_election.cc:304] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:02:47.696995  9835 raft_consensus.cc:493] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:47.711840  9815 raft_consensus.cc:3060] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:02:47.712589  1747 rpcz_store.cc:276] 0709 13:02:45.762260 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:45.762391 (+   131us) service_pool.cc:224] Handling call
0709 13:02:47.290557 (+1528166us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:47.295657 (+  5100us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:47.661387 (+365730us) spinlock_profiling.cc:243] Waited 1.51 s on lock 0x7b5400324bd0. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661618ad 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:02:47.661548 (+   161us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3472592768}
W20260709 13:02:47.748325  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:47.740784  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:47.757848  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 21 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:47.737596  9618 raft_consensus.cc:2749] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
W20260709 13:02:47.737912  9648 consensus_meta.cc:287] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.696s	user 0.015s	sys 0.001s
I20260709 13:02:47.711733  9814 consensus_queue.cc:1243] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer 3b879799899a499c8d409ee9f1277cd1 log is divergent from this leader: its last log entry 20.14 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 13
I20260709 13:02:47.745551  9812 raft_consensus.cc:515] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:47.762418  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 25 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:02:47.774192  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 46429.7us
I20260709 13:02:47.763136  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:47.775069  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 25 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:47.773160  1644 leader_election.cc:304] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:02:47.761616  9645 raft_consensus.cc:493] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:47.779618  9645 raft_consensus.cc:515] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:47.747408  9853 consensus_queue.cc:1048] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 15, Time since last communication: 0.034s
I20260709 13:02:47.781906  9645 leader_election.cc:290] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:47.761335  9835 raft_consensus.cc:515] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:47.783043  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 22 candidate_status { last_received { term: 19 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:47.737181  9666 raft_consensus.cc:740] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:02:47.784312  9666 consensus_queue.cc:260] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 23.17, Last appended by leader: 17, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:47.785147  9835 leader_election.cc:290] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:47.785909  9666 raft_consensus.cc:3060] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Advancing to term 24
I20260709 13:02:47.785607  9873 raft_consensus.cc:515] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:47.786361  1747 raft_consensus.cc:1240] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 25 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 24. Current term is 25. Ops: []
I20260709 13:02:47.788808  9812 leader_election.cc:290] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:47.762640  9840 raft_consensus.cc:2749] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:02:47.791883  9840 raft_consensus.cc:3060] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:02:47.761685  9838 consensus_queue.cc:1059] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 17.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.035s
I20260709 13:02:47.793293  9709 raft_consensus.cc:515] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:47.746703  9703 raft_consensus.cc:493] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:47.794423  1596 raft_consensus.cc:1240] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 18. Current term is 19. Ops: []
I20260709 13:02:47.795044  9838 raft_consensus.cc:3055] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 13:02:47.794821  9703 raft_consensus.cc:515] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:47.776607  1749 raft_consensus.cc:2468] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 24.
I20260709 13:02:47.795449  9838 raft_consensus.cc:740] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:02:47.747219  9704 raft_consensus.cc:493] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:47.800855  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 126080us
I20260709 13:02:47.802069  2301 leader_election.cc:304] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:02:47.802207  1571 leader_election.cc:304] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:02:47.791510  9823 raft_consensus.cc:515] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:47.799479  1644 leader_election.cc:304] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:02:47.720760  9804 consensus_queue.cc:1048] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.052s
I20260709 13:02:47.805627  1746 raft_consensus.cc:2468] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 21.
I20260709 13:02:47.761812  9869 raft_consensus.cc:493] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:47.814131  1753 raft_consensus.cc:1240] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 20. Current term is 21. Ops: []
I20260709 13:02:47.816394  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:47.815521  1749 raft_consensus.cc:3055] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 13:02:47.817134  1749 raft_consensus.cc:740] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:02:47.827391  1672 raft_consensus.cc:1240] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 18. Current term is 19. Ops: []
I20260709 13:02:47.755400  9785 raft_consensus.cc:515] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:47.827855  9869 raft_consensus.cc:515] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:47.837692  9869 leader_election.cc:290] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 13:02:47.838724  9903 log.cc:927] Time spent T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.213s	user 0.001s	sys 0.000s
W20260709 13:02:47.841270  9905 log.cc:927] Time spent T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.082s	user 0.000s	sys 0.000s
I20260709 13:02:47.842927  9838 consensus_queue.cc:260] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 17.15, Last appended by leader: 15, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:47.844800  9703 leader_election.cc:290] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:47.812584  9618 raft_consensus.cc:2804] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 13:02:47.845340  9618 raft_consensus.cc:493] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:47.845505  9703 raft_consensus.cc:2749] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:02:47.845764  9618 raft_consensus.cc:3060] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 24 FOLLOWER]: Advancing to term 25
I20260709 13:02:47.774637  9648 raft_consensus.cc:515] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:47.847915  1749 consensus_queue.cc:260] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.14, Last appended by leader: 14, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:47.849123  9648 leader_election.cc:290] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:02:47.849303  9908 log.cc:927] Time spent T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.088s	user 0.000s	sys 0.001s
I20260709 13:02:47.849375  1749 raft_consensus.cc:3060] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:02:47.809952  9704 raft_consensus.cc:515] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:47.804843  9853 raft_consensus.cc:2749] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 13:02:47.852072  9704 leader_election.cc:290] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:47.813047  9542 consensus_queue.cc:1059] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 21.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.040s
I20260709 13:02:47.854848  2301 leader_election.cc:304] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:02:47.855314  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 23 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:47.855690  9823 leader_election.cc:290] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:47.856254  1745 raft_consensus.cc:2410] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for term 23 because replica has last-logged OpId of term: 21 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 10.
I20260709 13:02:47.857858  8994 raft_consensus.cc:2804] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 13:02:47.858446  8994 raft_consensus.cc:697] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 21 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:02:47.859571  8994 consensus_queue.cc:237] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.15, Last appended by leader: 15, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:47.863688  9513 consensus_queue.cc:1059] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 17.12, Next index: 13, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 13:02:47.865729  9873 leader_election.cc:290] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 25 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:47.854871  1679 raft_consensus.cc:1240] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 18. Current term is 19. Ops: [18.15-18.15]
W20260709 13:02:47.848495  9906 log.cc:927] Time spent T 00000000000000000000000000000000 P 919f6392d4f44fd68987e2c5e7ec37ca: Append to log took a long time: real 0.124s	user 0.001s	sys 0.000s
I20260709 13:02:47.877666  1745 raft_consensus.cc:1275] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 19 index: 10. Preceding OpId from leader: term: 22 index: 11. (index mismatch)
I20260709 13:02:47.884567  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 25 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:47.834930  9810 raft_consensus.cc:2749] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 13:02:47.890023  1717 leader_election.cc:304] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:47.893870  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:47.829535  9776 consensus_queue.cc:1059] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 18.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.059s
W20260709 13:02:47.884111  9890 log.cc:927] Time spent T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.053s	user 0.000s	sys 0.001s
I20260709 13:02:47.909639  9709 leader_election.cc:290] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 19 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:47.911535  9542 raft_consensus.cc:3055] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 21 LEADER]: Stepping down as leader of term 21
I20260709 13:02:47.912067  9542 raft_consensus.cc:740] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:02:47.919806  9594 consensus_queue.cc:1048] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
W20260709 13:02:47.929261  2298 consensus_peers.cc:597] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:02:47.953433  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 45646.2us
I20260709 13:02:47.954780  1717 leader_election.cc:304] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:47.963397  1717 leader_election.cc:304] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 13:02:47.925506  9812 raft_consensus.cc:2804] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 13:02:47.974805  1717 leader_election.cc:304] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:47.913260  9542 consensus_queue.cc:260] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 21.17, Last appended by leader: 17, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:47.942863  9840 raft_consensus.cc:2749] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:02:47.956851  9804 raft_consensus.cc:3055] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 LEADER]: Stepping down as leader of term 17
W20260709 13:02:47.963665  1718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:47.971120  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 247345us
W20260709 13:02:47.992149  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 99865.8us
I20260709 13:02:47.974102  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:47.993579  1718 leader_election.cc:304] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:47.995251  2329 raft_consensus.cc:2468] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 22.
I20260709 13:02:47.942363  9821 consensus_queue.cc:1059] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 16.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.046s
I20260709 13:02:47.973325  9709 raft_consensus.cc:493] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:47.998656  2298 consensus_peers.cc:597] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:02:47.973407  9869 raft_consensus.cc:2749] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:02:47.985863  9812 raft_consensus.cc:493] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:48.033155  9812 raft_consensus.cc:3060] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:02:48.003353  9513 raft_consensus.cc:493] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:48.034634  9513 raft_consensus.cc:515] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:48.035162  1569 leader_election.cc:400] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 16 election: Vote denied by peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625) with higher term. Message: Invalid argument: T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 16. Current term is 17.
I20260709 13:02:48.035617  1569 leader_election.cc:403] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 16 election: Cancelling election due to peer responding with higher term
I20260709 13:02:47.988071  9594 raft_consensus.cc:2804] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 13:02:47.963070  9814 consensus_queue.cc:1048] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.040s
W20260709 13:02:48.032460  2298 consensus_peers.cc:597] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:02:48.048599  1749 raft_consensus.cc:1275] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 19 index: 14. Preceding OpId from leader: term: 20 index: 14. (term mismatch)
I20260709 13:02:48.054809  1749 pending_rounds.cc:85] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701: Aborting all ops after (but not including) 13
I20260709 13:02:47.997575  9804 raft_consensus.cc:740] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:02:48.056507  9780 consensus_queue.cc:1059] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 18.14, Next index: 15, Last known committed idx: 13, Time since last communication: 0.014s
W20260709 13:02:48.047546  9875 consensus_meta.cc:287] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.538s	user 0.009s	sys 0.005s
I20260709 13:02:48.056164  1749 pending_rounds.cc:107] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701: Aborting uncommitted NO_OP operation due to leader change: 19.14
I20260709 13:02:48.081549  1749 raft_consensus.cc:2889] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 13:02:48.074364  2298 consensus_peers.cc:597] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:02:48.038983  9851 raft_consensus.cc:2804] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Leader pre-election won for term 23
W20260709 13:02:48.083798  2298 consensus_peers.cc:597] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:02:48.050874  9542 raft_consensus.cc:3060] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:02:48.002804  9783 consensus_queue.cc:1048] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.073s
I20260709 13:02:48.089664  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 23 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:48.046128  9594 raft_consensus.cc:493] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:48.059366  9814 consensus_queue.cc:1243] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer 83595ad0fd5940c0a2bfc7c61acd969a log is divergent from this leader: its last log entry 16.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 9
I20260709 13:02:48.056916  9873 raft_consensus.cc:493] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:02:48.075845  9840 raft_consensus.cc:2749] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Leader election lost for term 16. Reason: Vote denied by peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625) with higher term. Message: Invalid argument: T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 16. Current term is 17.
W20260709 13:02:48.097777  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 62810.9us
I20260709 13:02:48.071918  9921 raft_consensus.cc:493] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:48.045853  9709 raft_consensus.cc:515] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:02:48.095736  2298 consensus_peers.cc:597] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:02:48.104327  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:48.087504  9845 raft_consensus.cc:2804] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 24 FOLLOWER]: Leader pre-election won for term 25
W20260709 13:02:48.108213  2298 consensus_peers.cc:597] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:02:48.110713  1673 raft_consensus.cc:2393] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 23: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:02:48.111581  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:48.111697  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:48.111581  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:48.112895  1680 raft_consensus.cc:2410] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for term 20 because replica has last-logged OpId of term: 19 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 9.
I20260709 13:02:48.072763  9869 consensus_queue.cc:1048] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.041s
I20260709 13:02:48.115957  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:48.116649  1569 leader_election.cc:304] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:02:48.112252  1671 raft_consensus.cc:2376] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 22. Re-sending same reply.
I20260709 13:02:48.120674  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 26 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:48.121807  1746 raft_consensus.cc:2410] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for term 26 because replica has last-logged OpId of term: 24 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 11.
I20260709 13:02:48.124172  1672 raft_consensus.cc:1275] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 17 index: 11. (index mismatch)
I20260709 13:02:48.090015  9914 raft_consensus.cc:2804] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 13:02:48.117427  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:48.134150  1674 raft_consensus.cc:2393] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 23: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:02:48.093037  9594 raft_consensus.cc:3060] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:02:48.112578  1678 raft_consensus.cc:2468] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 22.
I20260709 13:02:48.100005  9513 leader_election.cc:290] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:02:48.115543  2298 leader_election.cc:336] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:02:48.151710  2298 leader_election.cc:304] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:02:48.156184  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:48.156324  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:02:48.168066  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 59346.1us
I20260709 13:02:48.169203  1569 leader_election.cc:304] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:02:48.105829  9873 raft_consensus.cc:515] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:02:48.171911  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 61668.4us
I20260709 13:02:48.171090  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 24 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:48.133701  9914 raft_consensus.cc:493] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:48.172766  1674 raft_consensus.cc:2376] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 24 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 24. Re-sending same reply.
W20260709 13:02:48.157182  9424 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 20 current term: 20 is higher.
W20260709 13:02:48.186131  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 192732us
W20260709 13:02:48.172731  2298 consensus_peers.cc:597] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:02:48.208714  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:02:48.209481  2298 consensus_peers.cc:597] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:02:48.163690  9840 raft_consensus.cc:2804] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Leader election won for term 20
I20260709 13:02:48.114326  9921 raft_consensus.cc:515] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:48.258459  1679 raft_consensus.cc:2468] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 20.
I20260709 13:02:48.216133  1673 raft_consensus.cc:1275] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 18 index: 12. Preceding OpId from leader: term: 22 index: 14. (index mismatch)
I20260709 13:02:48.259716  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:48.237316  1600 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40154 (request call id 3121) took 1557 ms. Trace:
I20260709 13:02:48.260607  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:48.261289  1745 raft_consensus.cc:2468] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 22.
I20260709 13:02:48.244004  9709 leader_election.cc:290] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:48.161558  9821 consensus_queue.cc:1048] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.045s
I20260709 13:02:48.248009  9873 leader_election.cc:290] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:48.249634  9785 leader_election.cc:290] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:48.271701  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:48.272086  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
W20260709 13:02:48.250200  1570 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:48.272696  1679 raft_consensus.cc:3060] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:02:48.190008  9914 raft_consensus.cc:3060] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Advancing to term 23
W20260709 13:02:48.251058  9919 log.cc:927] Time spent T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.161s	user 0.000s	sys 0.000s
I20260709 13:02:48.251406  9814 raft_consensus.cc:2804] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 13:02:48.152379  9804 consensus_queue.cc:260] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.12, Last appended by leader: 12, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:48.281111  1646 leader_election.cc:304] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:48.281519  9804 raft_consensus.cc:3060] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Advancing to term 18
W20260709 13:02:48.251973  9913 log.cc:927] Time spent T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.167s	user 0.001s	sys 0.000s
I20260709 13:02:48.188162  9878 raft_consensus.cc:2749] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
W20260709 13:02:48.253751  9918 log.cc:927] Time spent T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.156s	user 0.000s	sys 0.000s
I20260709 13:02:48.254519  9845 raft_consensus.cc:493] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 24 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
W20260709 13:02:48.254642  9915 log.cc:927] Time spent T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.098s	user 0.001s	sys 0.000s
I20260709 13:02:48.255168  9703 raft_consensus.cc:493] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:48.255694  9704 raft_consensus.cc:493] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:48.256395  1680 raft_consensus.cc:1240] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 20. Current term is 21. Ops: [20.13-20.13]
I20260709 13:02:48.262197  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:48.264757  1675 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57092 (request call id 1786) took 1169 ms. Trace:
W20260709 13:02:48.263903  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 46315.1us
I20260709 13:02:48.260859  1600 rpcz_store.cc:276] 0709 13:02:46.680496 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:46.680682 (+   186us) service_pool.cc:224] Handling call
0709 13:02:46.727908 (+ 47226us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:46.820608 (+ 92700us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:46.820665 (+    57us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:46.844997 (+ 24332us) log.cc:844] Serialized 26 byte log entry
0709 13:02:47.647241 (+802244us) spinlock_profiling.cc:243] Waited 6.76 ms on lock 0x7b18003e5600. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f4263057e07 00007f426beb2ab9 00007f426307469b 00007f426307293f 00007f4263074c02 00007f4265cfc20b 00007f4265d01d23 00007f4265d020d1 00007f426611da0f
0709 13:02:48.007017 (+359776us) raft_consensus.cc:1645] Marking committed up to 13
0709 13:02:48.007088 (+    71us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:48.007198 (+   110us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:48.236926 (+229728us) raft_consensus.cc:1685] finished
0709 13:02:48.236953 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:48.237226 (+   273us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":187466,"raft.run_cpu_time_us":157,"raft.run_wall_time_us":4247,"spinlock_wait_cycles":15537920,"thread_start_us":405824,"threads_started":2,"wal-append.queue_time_us":1337875}
I20260709 13:02:48.266238  9815 raft_consensus.cc:515] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:48.266983  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 467780us
I20260709 13:02:48.349042  1644 leader_election.cc:304] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:02:48.273486  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:48.274577  9645 consensus_queue.cc:1059] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 20.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.002s
I20260709 13:02:48.227136  9780 raft_consensus.cc:2749] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
W20260709 13:02:48.287823  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:48.345590  1675 rpcz_store.cc:276] 0709 13:02:47.089520 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:47.089681 (+   161us) service_pool.cc:224] Handling call
0709 13:02:47.103191 (+ 13510us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:47.331640 (+228449us) raft_consensus.cc:1534] Early marking committed up to index 17
0709 13:02:47.520769 (+189129us) log.cc:844] Serialized 14 byte log entry
0709 13:02:48.078825 (+558056us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:48.079434 (+   609us) log.cc:844] Serialized 26 byte log entry
0709 13:02:48.079695 (+   261us) raft_consensus.cc:1645] Marking committed up to 17
0709 13:02:48.079734 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:48.079873 (+   139us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:48.258715 (+178842us) raft_consensus.cc:1685] finished
0709 13:02:48.258739 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:48.258884 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":82777,"prepare.queue_time_us":97242,"prepare.run_cpu_time_us":52,"prepare.run_wall_time_us":53,"raft.queue_time_us":119989,"raft.run_cpu_time_us":237,"raft.run_wall_time_us":3407,"thread_start_us":811139,"threads_started":4,"wal-append.queue_time_us":497220}
I20260709 13:02:48.304911  1671 raft_consensus.cc:2468] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 22.
I20260709 13:02:48.275758  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:48.331743  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 25 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
W20260709 13:02:48.332319  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 51302.7us
I20260709 13:02:48.276062  9709 raft_consensus.cc:493] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:48.335628  9862 log.cc:927] Time spent T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.061s	user 0.000s	sys 0.000s
I20260709 13:02:48.499353  1673 raft_consensus.cc:2376] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 25 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 25. Re-sending same reply.
W20260709 13:02:48.420013  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 445447us
I20260709 13:02:48.443208  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:48.475438  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:48.498850  1718 leader_election.cc:304] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:48.500666  1750 raft_consensus.cc:2368] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 21. Current term is 22.
W20260709 13:02:48.436416  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 87501.1us
I20260709 13:02:48.440236  9700 raft_consensus.cc:2804] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Leader pre-election won for term 23
W20260709 13:02:48.436323  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 83826.7us
I20260709 13:02:48.331822  9666 raft_consensus.cc:493] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:48.336545  9845 raft_consensus.cc:3060] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 24 FOLLOWER]: Advancing to term 25
I20260709 13:02:48.515270  1717 leader_election.cc:304] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:48.341315  9814 raft_consensus.cc:697] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 22 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:02:48.523401  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 23 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:48.524202  9835 raft_consensus.cc:3055] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [term 18 LEADER]: Stepping down as leader of term 18
W20260709 13:02:48.530675  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 158782us
I20260709 13:02:48.531623  9851 raft_consensus.cc:493] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Starting leader election (detected failure of leader af520b67945d443b901bbeca37cc5701)
W20260709 13:02:48.532554  9924 log.cc:927] Time spent T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.056s	user 0.000s	sys 0.000s
W20260709 13:02:48.532963  1645 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:48.541567  9598 raft_consensus.cc:3055] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 LEADER]: Stepping down as leader of term 22
I20260709 13:02:48.510417  9934 consensus_queue.cc:1048] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.066s
I20260709 13:02:48.444468  9783 raft_consensus.cc:2804] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 13:02:48.543157  9776 raft_consensus.cc:3055] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 13:02:48.543804  1754 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43156 (request call id 1639) took 1028 ms. Trace:
W20260709 13:02:48.545204  9929 log.cc:927] Time spent T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.199s	user 0.000s	sys 0.000s
I20260709 13:02:48.617177  1754 rpcz_store.cc:276] 0709 13:02:47.515174 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:47.515318 (+   144us) service_pool.cc:224] Handling call
0709 13:02:47.528288 (+ 12970us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:47.530738 (+  2450us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 13:02:47.530790 (+    52us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:47.540967 (+ 10177us) log.cc:844] Serialized 26 byte log entry
0709 13:02:47.768183 (+227216us) raft_consensus.cc:1645] Marking committed up to 14
0709 13:02:47.768236 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:47.768315 (+    79us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:48.543440 (+775125us) raft_consensus.cc:1685] finished
0709 13:02:48.543478 (+    38us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:48.543711 (+   233us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1741,"raft.queue_time_us":2541,"raft.run_cpu_time_us":252,"raft.run_wall_time_us":28901,"thread_start_us":194919,"threads_started":1,"wal-append.queue_time_us":431306}
I20260709 13:02:48.553890  9838 raft_consensus.cc:3060] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:02:48.557066  9709 raft_consensus.cc:515] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:48.573167  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:48.628764  1675 raft_consensus.cc:2393] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 24: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
W20260709 13:02:48.575729  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 471724us
W20260709 13:02:48.579071  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 63990.9us
I20260709 13:02:48.583658  9645 raft_consensus.cc:493] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:48.609686  9869 raft_consensus.cc:2804] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 13:02:48.352195  9703 raft_consensus.cc:515] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:48.589876  2299 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.545s	user 0.015s	sys 0.033s
W20260709 13:02:48.637707  2299 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.593s	user 0.015s	sys 0.033s
I20260709 13:02:48.589448  9424 raft_consensus.cc:2804] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 13:02:48.612438  9598 raft_consensus.cc:740] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:02:48.600278  9851 raft_consensus.cc:3060] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Advancing to term 23
W20260709 13:02:48.582188  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 83510.9us
I20260709 13:02:48.578939  9666 raft_consensus.cc:515] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:48.436671  9704 raft_consensus.cc:515] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:48.583621  9835 raft_consensus.cc:740] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:02:48.637478  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:48.621430  1671 raft_consensus.cc:2468] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 22.
W20260709 13:02:48.546277  9917 log.cc:927] Time spent T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.103s	user 0.000s	sys 0.000s
W20260709 13:02:48.627704  9812 consensus_meta.cc:287] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.594s	user 0.014s	sys 0.000s
I20260709 13:02:48.579679  9923 consensus_queue.cc:1059] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 20.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.055s
I20260709 13:02:48.614638  9783 raft_consensus.cc:493] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:02:48.631379  1720 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.507s	user 0.016s	sys 0.025s
I20260709 13:02:48.617354  9780 raft_consensus.cc:493] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:48.616921  9776 raft_consensus.cc:740] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:02:48.649294  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:48.616598  9358 raft_consensus.cc:493] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
W20260709 13:02:48.637006  9618 consensus_meta.cc:287] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.791s	user 0.017s	sys 0.000s
I20260709 13:02:48.628899  9873 raft_consensus.cc:493] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
W20260709 13:02:48.655697  2298 consensus_peers.cc:597] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:02:48.666584  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:02:48.725126  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 69528.7us
I20260709 13:02:48.668160  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 19 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:48.721695  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
W20260709 13:02:48.671686  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:48.640359  9826 consensus_queue.cc:1059] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 }, Status: INVALID_TERM, Last received: 20.13, Next index: 14, Last known committed idx: 12, Time since last communication: 0.027s
I20260709 13:02:48.681578  9815 leader_election.cc:290] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 22 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:48.745628  9815 raft_consensus.cc:493] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:48.747727  1750 raft_consensus.cc:1275] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 22 index: 16. Preceding OpId from leader: term: 24 index: 17. (index mismatch)
I20260709 13:02:48.751206  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 23 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:48.752583  9542 consensus_queue.cc:1048] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 13:02:48.753274  9814 consensus_queue.cc:237] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.14, Last appended by leader: 14, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:48.756098  1750 raft_consensus.cc:1240] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 22. Current term is 23. Ops: []
I20260709 13:02:48.757400  9878 raft_consensus.cc:493] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:48.758157  9878 raft_consensus.cc:515] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:48.765235  9878 leader_election.cc:290] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:48.773608  9358 raft_consensus.cc:515] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:48.727156  1675 raft_consensus.cc:2468] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 24.
I20260709 13:02:48.728295  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:48.736748  9618 raft_consensus.cc:515] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:48.737965  1671 raft_consensus.cc:1275] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 24 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 20 index: 10. Preceding OpId from leader: term: 24 index: 12. (index mismatch)
I20260709 13:02:48.659063  9935 raft_consensus.cc:493] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:48.781118  2331 raft_consensus.cc:2468] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 21.
W20260709 13:02:48.665699  9594 consensus_meta.cc:287] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.505s	user 0.012s	sys 0.000s
W20260709 13:02:48.789038  1678 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.531s	user 0.016s	sys 0.029s
I20260709 13:02:48.790541  1673 raft_consensus.cc:1275] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 21 index: 11. Preceding OpId from leader: term: 22 index: 12. (index mismatch)
I20260709 13:02:48.682971  9783 raft_consensus.cc:3060] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Advancing to term 24
W20260709 13:02:48.796676  1678 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.539s	user 0.016s	sys 0.029s
I20260709 13:02:48.799311  1676 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57092 (request call id 1796) took 1033 ms. Trace:
I20260709 13:02:48.646930  9645 raft_consensus.cc:515] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:48.771908  9934 consensus_queue.cc:1048] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.007s
W20260709 13:02:48.791553  9917 log.cc:927] Time spent T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.108s	user 0.000s	sys 0.001s
I20260709 13:02:48.789856  1571 leader_election.cc:400] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 21 pre-election: Vote denied by peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693) with higher term. Message: Invalid argument: T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 21. Current term is 22.
I20260709 13:02:48.725288  9921 leader_election.cc:290] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:48.814090  1571 leader_election.cc:403] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 21 pre-election: Cancelling election due to peer responding with higher term
W20260709 13:02:48.790309  9840 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.538s	user 0.019s	sys 0.033s
I20260709 13:02:48.804670  1676 rpcz_store.cc:276] 0709 13:02:47.765488 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:47.765646 (+   158us) service_pool.cc:224] Handling call
0709 13:02:47.782695 (+ 17049us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:47.806002 (+ 23307us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:47.806053 (+    51us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:47.847019 (+ 40966us) log.cc:844] Serialized 26 byte log entry
0709 13:02:48.148637 (+301618us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:02:48.148679 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:48.148759 (+    80us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:48.280750 (+131991us) spinlock_profiling.cc:243] Waited 1.43 ms on lock 0x7b48002ac558. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426b08baa1 00007f426b082b37 00007f426b0829ca 00007f426b17a26f 00007f426b179f37 00007f426b18e55c 00007f426b18e508 00007f426b18e488 00007f426b18e443 00007f426b18d6ec 00007f426a0b7a96
0709 13:02:48.777992 (+497242us) raft_consensus.cc:1685] finished
0709 13:02:48.778025 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:48.799218 (+ 21193us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":40279,"raft.queue_time_us":23815,"raft.run_cpu_time_us":4485,"raft.run_wall_time_us":469429,"spinlock_wait_cycles":3283840,"thread_start_us":301181,"threads_started":1,"wal-append.queue_time_us":722367}
I20260709 13:02:48.669634  9812 raft_consensus.cc:515] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:48.794528  9878 raft_consensus.cc:493] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:48.746304  9815 raft_consensus.cc:515] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:48.736135  9780 raft_consensus.cc:515] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:48.822533  9840 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.570s	user 0.019s	sys 0.033s
W20260709 13:02:48.858002  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 68397.2us
W20260709 13:02:48.835877  9875 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.562s	user 0.024s	sys 0.050s
I20260709 13:02:48.861279  1748 raft_consensus.cc:1240] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 21. Current term is 22. Ops: []
I20260709 13:02:48.866851  2335 raft_consensus.cc:1275] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 27 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 25 index: 18. Preceding OpId from leader: term: 27 index: 19. (index mismatch)
I20260709 13:02:48.788828  9873 raft_consensus.cc:515] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:48.842207  9921 raft_consensus.cc:3060] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Advancing to term 22
I20260709 13:02:48.887476  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:48.891274  2329 raft_consensus.cc:2393] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 20: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:02:48.805310  9935 raft_consensus.cc:515] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:48.893419  1570 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:48.798935  9598 consensus_queue.cc:260] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 22.12, Last appended by leader: 12, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:02:48.897656  2300 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.635s	user 0.014s	sys 0.031s
W20260709 13:02:48.901677  2300 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.639s	user 0.014s	sys 0.031s
W20260709 13:02:48.898000  1751 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.546s	user 0.015s	sys 0.028s
W20260709 13:02:48.863104  9875 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.593s	user 0.025s	sys 0.050s
I20260709 13:02:48.859334  9840 raft_consensus.cc:697] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:02:48.855335  8994 raft_consensus.cc:493] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:48.903426  1570 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:48.903441  1751 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.557s	user 0.015s	sys 0.029s
I20260709 13:02:48.862918  9920 raft_consensus.cc:493] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:48.918982  1644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:48.919395  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:48.921942  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:48.924943  2329 raft_consensus.cc:2468] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 18.
I20260709 13:02:48.825122  9835 consensus_queue.cc:260] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:48.933493  9835 raft_consensus.cc:3060] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:02:48.938668  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:48.939358  9618 leader_election.cc:290] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 25 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:48.939502  2335 raft_consensus.cc:2468] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 21.
I20260709 13:02:48.940776  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 23 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:48.941962  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:48.942953  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 19 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:48.943137  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:48.944051  9812 leader_election.cc:290] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 22 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:48.944190  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:48.945112  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:48.945210  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:48.945750  2332 raft_consensus.cc:1813] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Handling vote request from an unknown peer 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:48.944335  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:48.856623  9878 raft_consensus.cc:515] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:48.946774  9815 leader_election.cc:290] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:48.946385  2332 raft_consensus.cc:2410] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for term 23 because replica has last-logged OpId of term: 22 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 14.
I20260709 13:02:48.947623  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:48.949795  2335 raft_consensus.cc:2468] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 27.
I20260709 13:02:48.950131  9878 leader_election.cc:290] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:48.951040  9704 leader_election.cc:290] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:48.952665  9645 leader_election.cc:290] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:48.953545  9666 leader_election.cc:290] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:48.955782  9709 leader_election.cc:290] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:48.904798  9700 raft_consensus.cc:493] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:48.956686  9700 raft_consensus.cc:3060] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:02:48.915771  9358 leader_election.cc:290] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:48.926751  9875 raft_consensus.cc:493] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:48.959702  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 22 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:48.959578  9875 raft_consensus.cc:515] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:48.845613  9776 consensus_queue.cc:260] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 18.15, Last appended by leader: 15, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:48.960924  9776 raft_consensus.cc:3060] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:02:48.961128  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 21 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:48.961845  1746 raft_consensus.cc:2468] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 20.
I20260709 13:02:48.962146  9840 consensus_queue.cc:237] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.14, Last appended by leader: 14, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:48.965324  9922 raft_consensus.cc:3055] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 20 LEADER]: Stepping down as leader of term 20
I20260709 13:02:48.965776  9922 raft_consensus.cc:740] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:02:48.943002  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:48.966838  9922 consensus_queue.cc:260] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.15, Last appended by leader: 15, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:48.967376  9703 leader_election.cc:290] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:48.968361  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:02:48.950680  9878 consensus_queue.cc:1175] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "af520b67945d443b901bbeca37cc5701" responder_term: 23 status { last_received { term: 22 index: 12 } last_committed_idx: 12 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 22. Current term is 23. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260709 13:02:48.927834  9920 raft_consensus.cc:515] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:48.973723  1746 raft_consensus.cc:1275] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 21 index: 11. Preceding OpId from leader: term: 22 index: 11. (term mismatch)
I20260709 13:02:48.974145  1746 pending_rounds.cc:85] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701: Aborting all ops after (but not including) 10
I20260709 13:02:48.974395  1746 pending_rounds.cc:107] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701: Aborting uncommitted NO_OP operation due to leader change: 21.11
I20260709 13:02:48.974680  1746 raft_consensus.cc:2889] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 13:02:48.919437  9914 consensus_meta.cc:287] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.637s	user 0.015s	sys 0.000s
I20260709 13:02:48.976240  1569 leader_election.cc:304] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:02:48.971697  1571 leader_election.cc:304] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:48.824637  9594 raft_consensus.cc:515] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:48.977698  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:48.983712  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 25 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:48.987524  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:48.995925  1754 raft_consensus.cc:2468] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 20.
I20260709 13:02:48.997833  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:49.002686  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:49.006419  1569 leader_election.cc:304] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:02:49.009984  2334 raft_consensus.cc:1240] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 25 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 24. Current term is 25. Ops: []
W20260709 13:02:49.013360  1679 consensus_meta.cc:287] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.715s	user 0.013s	sys 0.001s
I20260709 13:02:49.013911  1679 raft_consensus.cc:2468] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 23.
I20260709 13:02:49.018492  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 22 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:49.021245  2329 raft_consensus.cc:2468] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 21.
I20260709 13:02:48.985759  9938 raft_consensus.cc:493] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:49.029704  9938 raft_consensus.cc:515] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:49.032076  9938 leader_election.cc:290] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:48.925016  8994 raft_consensus.cc:515] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:48.990451  9914 raft_consensus.cc:515] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:49.008034  9780 leader_election.cc:290] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:48.991786  9875 leader_election.cc:290] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:48.995199  9878 raft_consensus.cc:2804] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:02:49.025352  1569 leader_election.cc:304] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:02:49.026369  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:48.981562  9542 consensus_queue.cc:1048] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.045s
I20260709 13:02:48.993992  9703 raft_consensus.cc:2804] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:02:49.057385  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 22 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:49.057782  1747 raft_consensus.cc:1240] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 22. Current term is 23. Ops: []
I20260709 13:02:49.065102  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:49.065038  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:49.055704  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 21 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:49.076543  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 28 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:48.991573  9830 consensus_queue.cc:1048] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.068s
W20260709 13:02:49.078625  1645 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:49.081456  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:02:49.082669  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 57471us
I20260709 13:02:49.084630  1569 leader_election.cc:304] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
W20260709 13:02:49.040429  9804 consensus_meta.cc:287] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.758s	user 0.009s	sys 0.004s
I20260709 13:02:49.087833  1745 raft_consensus.cc:1240] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 20. Current term is 21. Ops: []
I20260709 13:02:49.091439  1569 leader_election.cc:304] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:02:49.087028  2301 leader_election.cc:304] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:49.098774  1672 raft_consensus.cc:1240] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 18. Current term is 19. Ops: [18.15-18.15]
I20260709 13:02:49.094230  1674 raft_consensus.cc:2468] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 18.
I20260709 13:02:49.052495  9935 leader_election.cc:290] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:49.039245  9821 raft_consensus.cc:493] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:49.042277  8994 leader_election.cc:290] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:49.044548  9914 leader_election.cc:290] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 23 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:49.054733  9709 raft_consensus.cc:2804] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 13:02:49.067705  9703 raft_consensus.cc:493] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:49.051188  9873 leader_election.cc:290] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:49.068152  9878 raft_consensus.cc:493] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:49.106727  1749 raft_consensus.cc:1240] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 22. Current term is 23. Ops: []
I20260709 13:02:49.077785  9424 raft_consensus.cc:697] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 20 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:02:49.113561  1672 raft_consensus.cc:1240] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 18. Current term is 19. Ops: []
I20260709 13:02:49.089351  9838 raft_consensus.cc:493] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:49.099318  9704 raft_consensus.cc:3060] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:02:49.117949  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:49.090718  9941 raft_consensus.cc:2749] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:02:49.074566  9853 consensus_queue.cc:1059] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.039s
W20260709 13:02:49.119570  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:49.120148  1678 raft_consensus.cc:1275] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 21 index: 17. Preceding OpId from leader: term: 22 index: 18. (index mismatch)
I20260709 13:02:49.116271  9703 raft_consensus.cc:3060] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:02:49.038388  9851 raft_consensus.cc:515] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:49.123871  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 25 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:49.114108  9709 raft_consensus.cc:493] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Starting leader election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:02:49.121037  9878 raft_consensus.cc:3060] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:02:49.125412  9873 raft_consensus.cc:2804] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 13:02:49.143201  1749 raft_consensus.cc:1275] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 20 index: 15. Preceding OpId from leader: term: 21 index: 16. (index mismatch)
I20260709 13:02:49.125599  8994 raft_consensus.cc:2804] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:02:49.112887  9821 raft_consensus.cc:515] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:49.129060  9838 raft_consensus.cc:515] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:49.153826  9838 leader_election.cc:290] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:49.144939  9873 raft_consensus.cc:493] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:49.158871  9873 raft_consensus.cc:3060] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 27 FOLLOWER]: Advancing to term 28
I20260709 13:02:49.148748  9424 consensus_queue.cc:237] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 19.16, Last appended by leader: 16, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:49.138142  9853 raft_consensus.cc:3055] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 24 LEADER]: Stepping down as leader of term 24
I20260709 13:02:49.163241  9853 raft_consensus.cc:740] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:02:49.164358  9853 consensus_queue.cc:260] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 24.17, Last appended by leader: 17, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:49.165709  9853 raft_consensus.cc:3060] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 24 FOLLOWER]: Advancing to term 25
I20260709 13:02:49.138047  9666 consensus_queue.cc:1059] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 22.16, Next index: 17, Last known committed idx: 14, Time since last communication: 0.030s
I20260709 13:02:49.168882  9666 raft_consensus.cc:3055] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 13:02:49.169333  9666 raft_consensus.cc:740] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:02:49.171124  9594 leader_election.cc:290] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 23 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:49.140287  9709 raft_consensus.cc:3060] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:02:49.175714  9878 raft_consensus.cc:515] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:49.177479  9920 leader_election.cc:290] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:49.178642  1748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55848 (request call id 3562) took 1300 ms. Trace:
I20260709 13:02:49.179108  1748 rpcz_store.cc:276] 0709 13:02:47.878508 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:47.894938 (+ 16430us) service_pool.cc:224] Handling call
0709 13:02:47.926337 (+ 31399us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:48.868505 (+942168us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:49.178362 (+309857us) spinlock_profiling.cc:243] Waited 935 ms on lock 0x7b540035ff54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:02:49.178573 (+   211us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2150236032}
W20260709 13:02:49.147078  9845 consensus_meta.cc:287] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.561s	user 0.010s	sys 0.005s
I20260709 13:02:49.149708  9922 raft_consensus.cc:3060] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:02:49.147389  8994 raft_consensus.cc:493] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:49.203405  9780 consensus_queue.cc:1048] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20260709 13:02:49.204730  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 26 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:49.205547  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 24 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:49.207814  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:49.199558  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 21 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:49.209596  1673 raft_consensus.cc:2468] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 22.
I20260709 13:02:49.217379  2328 raft_consensus.cc:2468] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 18.
I20260709 13:02:49.192399  9878 leader_election.cc:290] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 21 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:49.219317  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 28 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:02:49.220046  1717 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.579s	user 0.003s	sys 0.024s
W20260709 13:02:49.220495  1717 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.580s	user 0.003s	sys 0.024s
W20260709 13:02:49.224648  1718 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.573s	user 0.022s	sys 0.037s
W20260709 13:02:49.226392  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:49.226347  1718 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.575s	user 0.022s	sys 0.037s
W20260709 13:02:49.227360  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 580679us
I20260709 13:02:49.228497  1718 leader_election.cc:304] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:49.209822  8994 raft_consensus.cc:3060] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:02:49.221499  1745 raft_consensus.cc:2468] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 25.
I20260709 13:02:49.280550  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 24 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:49.184798  9666 consensus_queue.cc:260] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 22.16, Last appended by leader: 16, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:49.263180  9598 raft_consensus.cc:3060] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:02:49.208797  9873 raft_consensus.cc:515] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 28 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:49.285702  1718 leader_election.cc:304] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:02:49.200464  9845 raft_consensus.cc:515] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:49.263803  9838 raft_consensus.cc:493] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:49.285161  1643 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.686s	user 0.013s	sys 0.041s
W20260709 13:02:49.302846  1644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:49.309365  1643 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.710s	user 0.013s	sys 0.042s
I20260709 13:02:49.295256  1571 leader_election.cc:304] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:49.276700  9878 raft_consensus.cc:493] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:49.303821  2335 raft_consensus.cc:2468] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 19.
I20260709 13:02:49.273093  9757 raft_consensus.cc:2804] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Leader pre-election won for term 23
W20260709 13:02:49.331355  1570 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:49.370967  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:49.375488  9838 raft_consensus.cc:515] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:49.375988  1643 leader_election.cc:336] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 20 election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:02:49.376202  9851 leader_election.cc:290] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 23 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:02:49.399278  2300 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:49.464113  2300 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:49.408926  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:49.450260  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:02:49.455112  2299 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:49.466023  1673 raft_consensus.cc:2468] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 17.
W20260709 13:02:49.472268  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 186767us
I20260709 13:02:49.473392  1718 leader_election.cc:304] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:02:49.482582  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:49.483435  1680 raft_consensus.cc:2468] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 20.
I20260709 13:02:49.376784  9920 raft_consensus.cc:2804] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 13:02:49.448221  9941 raft_consensus.cc:493] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:49.455137  9937 raft_consensus.cc:2749] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:02:49.452013  9757 raft_consensus.cc:493] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:02:49.510769  2299 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:49.504223  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 193827us
I20260709 13:02:49.513283  1673 raft_consensus.cc:1240] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 22. Current term is 23. Ops: [22.12-22.12]
W20260709 13:02:49.485957  9700 consensus_meta.cc:287] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.529s	user 0.013s	sys 0.000s
I20260709 13:02:49.516772  9941 raft_consensus.cc:515] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:49.470453  9666 raft_consensus.cc:3060] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:02:49.534009  9941 leader_election.cc:290] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:49.461740  9776 raft_consensus.cc:493] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:49.478910  9869 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.811s	user 0.007s	sys 0.029s
I20260709 13:02:49.495559  9920 raft_consensus.cc:493] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:49.479096  9812 raft_consensus.cc:493] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:02:49.511986  9960 raft_consensus.cc:3060] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:02:49.537803  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 21 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:49.570685  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:02:49.502893  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 411700us
I20260709 13:02:49.571370  1679 raft_consensus.cc:1813] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Handling vote request from an unknown peer 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:49.571892  1679 raft_consensus.cc:2393] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 23: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:02:49.551198  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:49.490228  9851 raft_consensus.cc:493] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:49.572944  1680 raft_consensus.cc:2468] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 22.
I20260709 13:02:49.516002  9757 raft_consensus.cc:3060] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:02:49.573796  1748 raft_consensus.cc:2410] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for term 21 because replica has last-logged OpId of term: 20 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 11.
W20260709 13:02:49.540135  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 272304us
I20260709 13:02:49.566800  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 23 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:49.468962  9878 raft_consensus.cc:515] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:49.568674  2336 raft_consensus.cc:2468] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 19.
I20260709 13:02:49.572213  1569 leader_election.cc:304] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:02:49.563068  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:49.579216  1673 raft_consensus.cc:1240] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 25 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 24. Current term is 25. Ops: [24.12-24.12]
W20260709 13:02:49.558168  9783 consensus_meta.cc:287] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.741s	user 0.010s	sys 0.005s
I20260709 13:02:49.580544  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
W20260709 13:02:49.581070  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 84991.8us
W20260709 13:02:49.567234  9869 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.909s	user 0.008s	sys 0.030s
I20260709 13:02:49.582038  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:49.584746  1671 raft_consensus.cc:2468] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 18.
I20260709 13:02:49.583122  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:49.585676  1678 raft_consensus.cc:1240] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 24 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 23. Current term is 24. Ops: [22.11-23.12]
I20260709 13:02:49.583212  1569 leader_election.cc:304] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:02:49.588271  1569 leader_election.cc:304] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
W20260709 13:02:49.589192  1645 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:49.574483  9949 log.cc:927] Time spent T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.366s	user 0.000s	sys 0.002s
I20260709 13:02:49.590842  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:49.591176  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:49.595690  1675 raft_consensus.cc:2393] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 21: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:02:49.592216  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:49.593303  9840 raft_consensus.cc:2804] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 13:02:49.593954  1672 raft_consensus.cc:2393] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 19: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:02:49.567453  9920 raft_consensus.cc:3060] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 25 FOLLOWER]: Advancing to term 26
I20260709 13:02:49.597164  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:49.598201  1673 raft_consensus.cc:2393] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 19: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:02:49.596124  2299 leader_election.cc:304] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:02:49.599535  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:49.600116  1680 raft_consensus.cc:3060] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:02:49.600880  2299 leader_election.cc:304] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:02:49.602391  1679 raft_consensus.cc:2393] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 25: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:02:49.604730  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 21 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:49.599227  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:49.574703  9812 raft_consensus.cc:515] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:49.537767  9700 raft_consensus.cc:515] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:49.612932  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:49.622006  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:49.573271  9776 raft_consensus.cc:515] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:49.672391  9845 leader_election.cc:290] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 25 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:49.672827  9776 leader_election.cc:290] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:49.576439  9851 raft_consensus.cc:515] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:49.678682  9851 leader_election.cc:290] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:02:49.678774  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:49.679260  9851 raft_consensus.cc:493] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:49.679788  9851 raft_consensus.cc:515] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:49.682096  9851 leader_election.cc:290] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:49.590957  9969 raft_consensus.cc:2749] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 13:02:49.584501  9869 raft_consensus.cc:697] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 18 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:02:49.643646  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 19 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:49.653477  9700 leader_election.cc:290] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 23 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:49.653770  9812 leader_election.cc:290] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:02:49.654678  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 72868.3us
W20260709 13:02:49.655489  9958 log.cc:927] Time spent T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.315s	user 0.000s	sys 0.001s
W20260709 13:02:49.657783  9951 log.cc:927] Time spent T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.162s	user 0.000s	sys 0.000s
I20260709 13:02:49.776374  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:49.662705  9956 raft_consensus.cc:2749] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 13:02:49.612879  9963 consensus_queue.cc:1048] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.054s
I20260709 13:02:49.590775  9783 raft_consensus.cc:515] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:49.698324  9875 raft_consensus.cc:493] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:49.650940  2299 leader_election.cc:304] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:02:49.704391  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 26 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:49.711138  9873 leader_election.cc:290] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 28 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:49.600164  9840 raft_consensus.cc:493] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:49.718216  9971 raft_consensus.cc:493] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
W20260709 13:02:49.726035  9921 consensus_meta.cc:287] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.828s	user 0.012s	sys 0.001s
W20260709 13:02:49.792342  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 141564us
I20260709 13:02:49.793558  2299 leader_election.cc:304] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:02:49.616845  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:49.797374  2299 leader_election.cc:304] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:02:49.611616  9941 raft_consensus.cc:2804] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Leader election won for term 18
I20260709 13:02:49.743196  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
W20260709 13:02:49.748082  1717 consensus_peers.cc:597] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:02:49.799697  2331 raft_consensus.cc:3060] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Advancing to term 23
W20260709 13:02:49.799757  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 51742.1us
W20260709 13:02:49.775795  9704 consensus_meta.cc:287] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.644s	user 0.014s	sys 0.001s
I20260709 13:02:49.740590  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:49.759560  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:49.777395  1672 raft_consensus.cc:2468] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 24.
I20260709 13:02:49.783322  1673 raft_consensus.cc:2468] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 25.
I20260709 13:02:49.806465  1679 raft_consensus.cc:3060] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:02:49.812541  1677 raft_consensus.cc:1240] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 24 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 23. Current term is 24. Ops: []
I20260709 13:02:49.815364  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 23 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:49.817153  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:49.818886  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 21 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:49.775238  9851 raft_consensus.cc:493] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:49.801005  1717 leader_election.cc:336] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 13:02:49.823359  1717 leader_election.cc:336] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 13:02:49.824119  1717 consensus_peers.cc:597] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:02:49.824222  1672 raft_consensus.cc:1240] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 25 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 24. Current term is 25. Ops: []
W20260709 13:02:49.749307  1646 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:49.765911  9542 raft_consensus.cc:493] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:02:49.789930  1680 raft_consensus.cc:2468] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 18.
I20260709 13:02:49.780772  9878 leader_election.cc:290] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:02:49.828184  1717 leader_election.cc:336] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 13:02:49.834364  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:49.831686  1718 leader_election.cc:304] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
W20260709 13:02:49.838593  1717 leader_election.cc:336] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 13:02:49.839080  1570 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:49.786983  9840 raft_consensus.cc:3060] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:02:49.842701  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 22 candidate_status { last_received { term: 19 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:02:49.839352  1717 consensus_peers.cc:597] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:02:49.821341  9709 consensus_meta.cc:287] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.646s	user 0.012s	sys 0.006s
I20260709 13:02:49.811832  9921 raft_consensus.cc:2749] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Leader pre-election lost for term 21. Reason: Vote denied by peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693) with higher term. Message: Invalid argument: T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 21. Current term is 22.
I20260709 13:02:49.774462  9957 consensus_queue.cc:1048] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.062s
W20260709 13:02:49.849607  1717 consensus_peers.cc:597] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:02:49.822670  9588 raft_consensus.cc:2804] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 13:02:49.820817  9704 raft_consensus.cc:2749] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:02:49.833508  9941 raft_consensus.cc:697] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:02:49.836699  9956 raft_consensus.cc:2749] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 13:02:49.787328  9875 raft_consensus.cc:515] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:49.867492  2330 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.662s	user 0.088s	sys 0.160s
W20260709 13:02:49.871111  2330 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.666s	user 0.089s	sys 0.161s
I20260709 13:02:49.875734  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:02:49.876477  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 44958.6us
I20260709 13:02:49.877391  1718 leader_election.cc:304] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:49.802851  9971 raft_consensus.cc:515] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:49.875320  9941 consensus_queue.cc:237] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 17.15, Last appended by leader: 15, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:49.833549  9542 raft_consensus.cc:515] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:49.860508  9588 raft_consensus.cc:493] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:49.861140  9873 raft_consensus.cc:493] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:49.861536  9838 leader_election.cc:290] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:49.881902  9424 raft_consensus.cc:2749] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:02:49.900840  1752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55848 (request call id 3560) took 2028 ms. Trace:
I20260709 13:02:49.849604  9845 consensus_queue.cc:1059] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 21.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.029s
I20260709 13:02:49.853690  9709 raft_consensus.cc:515] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:49.917122  9974 log.cc:927] Time spent T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.057s	user 0.001s	sys 0.000s
I20260709 13:02:49.945151  9952 raft_consensus.cc:493] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:49.945832  9972 log.cc:927] Time spent T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.082s	user 0.000s	sys 0.000s
W20260709 13:02:49.946761  2298 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 53 similar messages]
I20260709 13:02:49.949622  9875 leader_election.cc:290] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:49.961160  1679 raft_consensus.cc:2468] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 23.
I20260709 13:02:49.879163  9945 raft_consensus.cc:2749] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
W20260709 13:02:49.971897  1674 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52778 (request call id 2957) took 1679 ms (client timeout 1938 ms). Trace:
I20260709 13:02:49.837481  9851 raft_consensus.cc:515] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:49.911940  9588 raft_consensus.cc:3060] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Advancing to term 23
W20260709 13:02:49.975790  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 98540us
W20260709 13:02:50.004941  2300 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:49.944392  9853 consensus_meta.cc:287] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.778s	user 0.012s	sys 0.004s
I20260709 13:02:49.860687  9869 consensus_queue.cc:237] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.12, Last appended by leader: 12, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:02:49.999336  1674 rpcz_store.cc:269] 0709 13:02:48.257587 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:48.260632 (+  3045us) service_pool.cc:224] Handling call
0709 13:02:49.937100 (+1676468us) spinlock_profiling.cc:243] Waited 744 ms on lock 0x7b54001f6154. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:02:49.937272 (+   172us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1711241600}
W20260709 13:02:50.010947  9598 consensus_meta.cc:287] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.703s	user 0.014s	sys 0.000s
I20260709 13:02:50.011305  9815 raft_consensus.cc:3055] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 13:02:50.011915  9815 raft_consensus.cc:740] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:02:50.012115  9851 leader_election.cc:290] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:50.013185  9815 consensus_queue.cc:260] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.13, Last appended by leader: 13, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:49.923081  9873 raft_consensus.cc:515] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:50.015992  9873 leader_election.cc:290] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:50.018591  9704 raft_consensus.cc:493] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:50.019294  1718 leader_election.cc:304] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:50.019392  9704 raft_consensus.cc:515] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:50.021100  9424 raft_consensus.cc:2804] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:02:50.021510  9424 raft_consensus.cc:493] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:50.021855  9424 raft_consensus.cc:3060] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:02:50.022137  9704 leader_election.cc:290] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:50.018559  9815 raft_consensus.cc:3060] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:02:49.993053  9921 raft_consensus.cc:493] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:50.038743  2331 raft_consensus.cc:2468] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 23.
W20260709 13:02:50.043313  1643 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.530s	user 0.066s	sys 0.102s
W20260709 13:02:50.043720  1643 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.531s	user 0.066s	sys 0.102s
W20260709 13:02:50.044066  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 531729us
W20260709 13:02:50.045634  1643 leader_election.cc:336] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:02:50.046191  1643 leader_election.cc:304] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 13:02:50.037487  9666 consensus_meta.cc:287] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.505s	user 0.010s	sys 0.003s
I20260709 13:02:50.049419  1718 leader_election.cc:304] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 13:02:49.942624  1752 rpcz_store.cc:276] 0709 13:02:47.872812 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:47.893638 (+ 20826us) service_pool.cc:224] Handling call
0709 13:02:47.899922 (+  6284us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:02:47.953393 (+ 53471us) raft_consensus.cc:1534] Early marking committed up to index 17
0709 13:02:48.475808 (+522415us) log.cc:844] Serialized 14 byte log entry
0709 13:02:49.133675 (+657867us) log.cc:844] Serialized 14 byte log entry
0709 13:02:49.198701 (+ 65026us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:02:49.199636 (+   935us) log.cc:844] Serialized 52 byte log entry
0709 13:02:49.205655 (+  6019us) raft_consensus.cc:1645] Marking committed up to 18
0709 13:02:49.211239 (+  5584us) log.cc:844] Serialized 14 byte log entry
0709 13:02:49.212442 (+  1203us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:49.212546 (+   104us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:49.900440 (+687894us) raft_consensus.cc:1685] finished
0709 13:02:49.900479 (+    39us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:49.900729 (+   250us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":149121,"prepare.queue_time_us":331575,"prepare.run_cpu_time_us":134,"prepare.run_wall_time_us":137,"raft.queue_time_us":178917,"raft.run_cpu_time_us":391,"raft.run_wall_time_us":11376,"spinlock_wait_cycles":5896448,"thread_start_us":970229,"threads_started":2,"wal-append.queue_time_us":738227}
W20260709 13:02:50.047155  9703 consensus_meta.cc:287] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.909s	user 0.015s	sys 0.008s
I20260709 13:02:50.055120  9703 raft_consensus.cc:515] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:50.058077  9703 leader_election.cc:290] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 19 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:50.042482  9783 leader_election.cc:290] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 24 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:02:50.060742  9922 consensus_meta.cc:287] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.851s	user 0.014s	sys 0.005s
I20260709 13:02:50.062873  1676 raft_consensus.cc:1275] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 20 index: 15. Preceding OpId from leader: term: 21 index: 16. (index mismatch)
I20260709 13:02:50.043069  9598 raft_consensus.cc:493] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:50.063602  9840 raft_consensus.cc:515] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:50.064019  9598 raft_consensus.cc:515] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:50.067659  9542 leader_election.cc:290] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:50.069347  9757 raft_consensus.cc:515] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:50.112094  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 64648.1us
I20260709 13:02:50.113281  1718 leader_election.cc:304] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 13:02:50.117257  2299 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:50.036405  9957 consensus_queue.cc:1048] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 11, Time since last communication: 0.038s
I20260709 13:02:50.053719  9920 raft_consensus.cc:515] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:50.131075  9594 raft_consensus.cc:2804] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 25 FOLLOWER]: Leader election won for term 25
I20260709 13:02:50.131755  9594 raft_consensus.cc:697] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 25 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:02:50.132870  9920 leader_election.cc:290] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 26 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:50.132784  9594 consensus_queue.cc:237] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 24.12, Last appended by leader: 12, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:50.132925  1718 leader_election.cc:304] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:02:50.072804  9598 leader_election.cc:290] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:49.976629  9952 raft_consensus.cc:515] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:50.146518  1643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:50.046249  9921 raft_consensus.cc:515] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:50.138336  1569 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.531s	user 0.011s	sys 0.045s
W20260709 13:02:50.147331  1569 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.540s	user 0.011s	sys 0.046s
I20260709 13:02:50.050539  9424 raft_consensus.cc:515] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } }
I20260709 13:02:50.114789  9703 raft_consensus.cc:493] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:50.149518  2336 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49082 (request call id 1418) took 1202 ms. Trace:
I20260709 13:02:50.150991  1718 leader_election.cc:304] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:50.152293  1718 leader_election.cc:304] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:02:50.148018  9920 raft_consensus.cc:493] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:02:50.153275  9941 raft_consensus.cc:493] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:50.153609  9920 raft_consensus.cc:515] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:50.153774  9937 raft_consensus.cc:2749] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:02:50.154021  9941 raft_consensus.cc:515] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } }
I20260709 13:02:50.155732  9757 leader_election.cc:290] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 23 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:50.155982  9920 leader_election.cc:290] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:50.156297  1569 leader_election.cc:304] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:02:50.156456  9941 leader_election.cc:290] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:50.157917  1569 leader_election.cc:304] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 13:02:50.149925  2336 rpcz_store.cc:276] 0709 13:02:48.941268 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:48.943916 (+  2648us) service_pool.cc:224] Handling call
0709 13:02:50.143576 (+1199660us) spinlock_profiling.cc:243] Waited 509 ms on lock 0x7b54006b6cd4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:02:50.143739 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1169924480}
I20260709 13:02:50.135712  9934 raft_consensus.cc:2804] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 13:02:50.135008  9618 raft_consensus.cc:2749] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 13:02:50.159355  1718 leader_election.cc:304] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 13:02:50.167125  9960 consensus_meta.cc:287] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.585s	user 0.011s	sys 0.004s
I20260709 13:02:50.135936  9840 leader_election.cc:290] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 22 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:02:50.141597  9814 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.346s	user 0.020s	sys 0.024s
I20260709 13:02:50.174782  1569 leader_election.cc:304] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 13:02:50.148697  9815 raft_consensus.cc:2749] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 13:02:50.176565  1718 leader_election.cc:304] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:50.146555  9542 raft_consensus.cc:493] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:50.173161  9931 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.492s	user 0.044s	sys 0.049s
I20260709 13:02:50.178114  9955 consensus_queue.cc:1048] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.003s
I20260709 13:02:50.183290  9955 raft_consensus.cc:2804] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 13:02:50.183821  9955 raft_consensus.cc:493] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:50.184273  9955 raft_consensus.cc:3060] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Advancing to term 25
I20260709 13:02:50.185194  9542 raft_consensus.cc:515] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:50.164019  9598 raft_consensus.cc:2804] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:02:50.153586  9971 leader_election.cc:290] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:50.187079  9598 raft_consensus.cc:493] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:50.187518  9598 raft_consensus.cc:3060] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:02:50.187932  9542 leader_election.cc:290] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:50.178323  9934 raft_consensus.cc:697] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 22 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
W20260709 13:02:50.191737  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 245073us
I20260709 13:02:50.153577  9873 raft_consensus.cc:493] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:50.162655  9757 raft_consensus.cc:2749] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:02:50.194739  1569 leader_election.cc:304] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
W20260709 13:02:50.179903  9814 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.392s	user 0.021s	sys 0.024s
I20260709 13:02:50.175246  9960 raft_consensus.cc:2749] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:02:50.175565  9853 raft_consensus.cc:2749] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:02:50.204146  1569 leader_election.cc:304] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
W20260709 13:02:50.205695  2298 consensus_peers.cc:597] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:02:50.191481  9931 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.511s	user 0.044s	sys 0.049s
I20260709 13:02:50.178701  9937 raft_consensus.cc:2749] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:02:50.178215  9941 raft_consensus.cc:2804] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:02:50.161415  9703 raft_consensus.cc:515] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:02:50.199908  8994 consensus_meta.cc:287] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.922s	user 0.014s	sys 0.001s
I20260709 13:02:50.196080  9920 raft_consensus.cc:2764] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Leader pre-election decision vote started in defunct term 21: won
I20260709 13:02:50.217674  1569 leader_election.cc:304] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:02:50.198798  9709 leader_election.cc:290] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 22 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:50.210538  9840 raft_consensus.cc:2749] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:02:50.224834  1569 leader_election.cc:304] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:02:50.217656  9941 raft_consensus.cc:493] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:50.219925  9812 raft_consensus.cc:2749] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:02:50.201961  9873 raft_consensus.cc:515] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:50.219691  9979 raft_consensus.cc:2749] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 13:02:50.237937  1569 leader_election.cc:304] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:02:50.231058  9709 raft_consensus.cc:493] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:50.245630  1677 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52778 (request call id 2961) took 1806 ms. Trace:
I20260709 13:02:50.245843  9709 raft_consensus.cc:515] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:50.246091  1677 rpcz_store.cc:276] 0709 13:02:48.439045 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:48.752505 (+313460us) service_pool.cc:224] Handling call
0709 13:02:48.752868 (+   363us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:49.813102 (+1060234us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:50.245331 (+432229us) spinlock_profiling.cc:243] Waited 1.06 s on lock 0x7b5400347e54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:02:50.245552 (+   221us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2437355904}
I20260709 13:02:50.247496  9840 raft_consensus.cc:2749] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 13:02:50.248075  9873 leader_election.cc:290] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:50.248337  9709 leader_election.cc:290] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:50.226142  9851 raft_consensus.cc:2804] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:02:50.249094  9851 raft_consensus.cc:493] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:50.249563  9851 raft_consensus.cc:3060] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:02:50.243065  9704 raft_consensus.cc:2749] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 13:02:50.254644  9934 consensus_queue.cc:237] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.14, Last appended by leader: 14, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:50.257356  9703 leader_election.cc:290] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:50.258050  9937 raft_consensus.cc:493] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:50.258642  9937 raft_consensus.cc:515] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:50.260905  9937 leader_election.cc:290] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:50.261348  9851 raft_consensus.cc:515] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:50.263336  9424 leader_election.cc:290] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 21 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:50.264132  9851 leader_election.cc:290] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 19 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:02:50.268391  9981 log.cc:927] Time spent T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.095s	user 0.000s	sys 0.000s
I20260709 13:02:50.268497  9853 consensus_queue.cc:1059] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 22.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 13:02:50.269526  9845 raft_consensus.cc:3055] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 LEADER]: Stepping down as leader of term 22
I20260709 13:02:50.269963  9845 raft_consensus.cc:740] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
W20260709 13:02:50.270273  9978 log.cc:927] Time spent T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.077s	user 0.000s	sys 0.000s
W20260709 13:02:50.271322  9954 log.cc:927] Time spent T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.767s	user 0.005s	sys 0.000s
I20260709 13:02:50.233032  9936 raft_consensus.cc:493] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:50.275508  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:50.279893  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:50.280277  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:50.280740  2330 raft_consensus.cc:2468] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 20.
I20260709 13:02:50.280979  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 23 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:50.280918  2331 raft_consensus.cc:3060] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:02:50.281895  2328 raft_consensus.cc:2468] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 22.
I20260709 13:02:50.285279  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 22 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:50.286021  2328 raft_consensus.cc:3060] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:02:50.234464  9941 raft_consensus.cc:3060] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:02:50.224591  8994 raft_consensus.cc:515] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:50.302007  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:50.302718  2336 raft_consensus.cc:3060] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 27 FOLLOWER]: Advancing to term 28
I20260709 13:02:50.313467  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:50.314546  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:50.314139  2335 raft_consensus.cc:3060] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:02:50.313071  9952 leader_election.cc:290] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:50.315866  2331 raft_consensus.cc:2468] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 19.
I20260709 13:02:50.321173  2330 raft_consensus.cc:1240] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 20. Current term is 21. Ops: []
W20260709 13:02:50.320299  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 143880us
I20260709 13:02:50.324882  1718 leader_election.cc:304] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:50.325735  2328 raft_consensus.cc:2468] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 22.
I20260709 13:02:50.327072  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 24 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:50.328619  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:50.311978  9931 raft_consensus.cc:493] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:50.355666  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:50.356331  2333 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39726 (request call id 1795) took 1694 ms. Trace:
I20260709 13:02:50.356633  1753 raft_consensus.cc:2468] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 20.
I20260709 13:02:50.272311  9936 raft_consensus.cc:515] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:50.272509  9845 consensus_queue.cc:260] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 22.12, Last appended by leader: 12, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:50.356905  2333 rpcz_store.cc:276] 0709 13:02:48.662094 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:48.662235 (+   141us) service_pool.cc:224] Handling call
0709 13:02:48.662714 (+   479us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:48.775241 (+112527us) raft_consensus.cc:1534] Early marking committed up to index 17
0709 13:02:49.204570 (+429329us) log.cc:844] Serialized 14 byte log entry
0709 13:02:50.128802 (+924232us) spinlock_profiling.cc:243] Waited 11.7 ms on lock 0x7b180010c840. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f4265d0a4bc 00007f426305848b 00007f426beb2ab9 00007f426307469b 00007f426307293f 00007f4263074c02 00007f4265cfc20b 00007f4265d01d23 00007f4265d0223a
0709 13:02:50.129056 (+   254us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:50.129128 (+    72us) raft_consensus.cc:1645] Marking committed up to 17
0709 13:02:50.129202 (+    74us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:50.355962 (+226760us) spinlock_profiling.cc:243] Waited 112 ms on lock 0x7b540065ed54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:02:50.356049 (+    87us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:50.356232 (+   183us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":700114,"prepare.run_cpu_time_us":67,"prepare.run_wall_time_us":68,"raft.queue_time_us":1574,"raft.run_cpu_time_us":115,"raft.run_wall_time_us":115,"spinlock_wait_cycles":285123072,"thread_start_us":751029,"threads_started":2,"wal-append.queue_time_us":579361}
I20260709 13:02:50.331102  9937 raft_consensus.cc:2804] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 13:02:50.332405  9921 leader_election.cc:290] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:50.327988  9960 consensus_queue.cc:1048] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.041s
W20260709 13:02:50.372747  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 76718.2us
I20260709 13:02:50.354985  9757 raft_consensus.cc:493] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:50.376080  1569 leader_election.cc:304] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:02:50.377815  1569 leader_election.cc:304] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:02:50.381857  1569 leader_election.cc:304] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:02:50.402452  2336 raft_consensus.cc:2468] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 28.
I20260709 13:02:50.404747  1569 leader_election.cc:304] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:02:50.361341  9931 raft_consensus.cc:515] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:50.416488  9931 leader_election.cc:290] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:50.405875  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:50.379762  9845 raft_consensus.cc:3060] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:02:50.380087  9921 raft_consensus.cc:2749] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:02:50.410326  1752 raft_consensus.cc:1240] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 21. Current term is 22. Ops: []
I20260709 13:02:50.385782  9851 raft_consensus.cc:2749] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 13:02:50.420794  1753 raft_consensus.cc:2468] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 23.
I20260709 13:02:50.427079  9936 leader_election.cc:290] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:50.422227  1569 leader_election.cc:304] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:02:50.386180  9937 raft_consensus.cc:697] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 23 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:02:50.433297  1569 leader_election.cc:304] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:02:50.435460  1569 leader_election.cc:304] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:02:50.435283  9709 raft_consensus.cc:2804] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:02:50.437517  9709 raft_consensus.cc:493] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:50.437999  9709 raft_consensus.cc:3060] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:02:50.437606  9937 consensus_queue.cc:237] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 22.12, Last appended by leader: 12, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:50.407923  9979 raft_consensus.cc:2804] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:02:50.382236  9757 raft_consensus.cc:515] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:50.442242  9821 leader_election.cc:290] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:50.443836  8994 leader_election.cc:290] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 21 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:50.444721  9955 raft_consensus.cc:515] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:50.447137  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 22 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:50.447474  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:50.418289  9588 raft_consensus.cc:515] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:50.436862  1569 leader_election.cc:304] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:02:50.450852  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:50.432139  9703 raft_consensus.cc:2804] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Leader election won for term 19
I20260709 13:02:50.458834  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:50.438479  9598 raft_consensus.cc:515] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:50.440202  9873 raft_consensus.cc:2804] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 13:02:50.459435  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:50.464731  1752 raft_consensus.cc:2393] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 22: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:02:50.465127  1750 raft_consensus.cc:2393] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 20: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:02:50.460505  1569 leader_election.cc:304] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:02:50.447638  9840 raft_consensus.cc:2804] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Leader election won for term 19
I20260709 13:02:50.481302  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:50.467861  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 19 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:50.448635  9979 raft_consensus.cc:493] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:50.471910  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 26 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:50.471992  9955 leader_election.cc:290] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 25 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:50.468925  9703 raft_consensus.cc:697] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:02:50.463886  9921 raft_consensus.cc:2804] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Leader election won for term 22
I20260709 13:02:50.487720  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 23 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:50.487778  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:50.488421  9851 raft_consensus.cc:2804] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 28 FOLLOWER]: Leader election won for term 28
I20260709 13:02:50.494266  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:50.494696  9757 leader_election.cc:290] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:50.495560  9851 raft_consensus.cc:697] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 28 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:02:50.496632  9851 consensus_queue.cc:237] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 25.16, Last appended by leader: 16, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:50.490964  9840 raft_consensus.cc:697] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:02:50.499497  9840 consensus_queue.cc:237] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:50.484285  9979 raft_consensus.cc:3060] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:02:50.466917  9873 raft_consensus.cc:493] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:50.505604  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:50.506266  1752 raft_consensus.cc:3060] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Advancing to term 19
W20260709 13:02:50.491801  1717 consensus_peers.cc:597] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:02:50.508369  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 22 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:50.510655  1746 raft_consensus.cc:2393] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 23: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
W20260709 13:02:50.510053  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 653238us
I20260709 13:02:50.480822  9598 leader_election.cc:290] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 18 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:02:50.511842  1717 consensus_peers.cc:597] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:02:50.516188  1746 raft_consensus.cc:1275] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 19 index: 14. Preceding OpId from leader: term: 20 index: 15. (index mismatch)
I20260709 13:02:50.505067  9873 raft_consensus.cc:3060] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:02:50.523828  9588 leader_election.cc:290] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 23 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:02:50.516651  1717 consensus_peers.cc:597] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:02:50.537511  9920 consensus_queue.cc:1059] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.006s
I20260709 13:02:50.546983  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:50.512755  9921 raft_consensus.cc:697] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:02:50.559624  1746 raft_consensus.cc:2393] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 21: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
W20260709 13:02:50.564446  1717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:50.564486  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:50.565989  1750 raft_consensus.cc:2393] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 25: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
W20260709 13:02:50.567696  1717 consensus_peers.cc:597] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:02:50.571503  1717 leader_election.cc:336] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 22 election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:02:50.571501  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:50.573896  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 27 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:50.578158  2335 raft_consensus.cc:2468] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 22.
W20260709 13:02:50.579674  1717 consensus_peers.cc:597] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:02:50.582592  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:02:50.582127  1717 leader_election.cc:336] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 22 election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 13:02:50.584582  1717 leader_election.cc:336] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:02:50.580260  1569 leader_election.cc:304] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:02:50.585987  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
W20260709 13:02:50.586917  1717 leader_election.cc:336] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 13:02:50.588543  1717 leader_election.cc:336] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 23 election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 13:02:50.589253  1717 leader_election.cc:336] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 23 election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:02:50.591348  9921 consensus_queue.cc:237] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.13, Last appended by leader: 13, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:50.590240  1717 leader_election.cc:336] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 18 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:02:50.585757  9703 consensus_queue.cc:237] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:50.617379  1646 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:50.622982  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:50.623677  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:50.623760  1750 raft_consensus.cc:2468] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 24.
I20260709 13:02:50.624547  1746 raft_consensus.cc:2468] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 23.
I20260709 13:02:50.624547  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:50.628579  1751 raft_consensus.cc:3060] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:02:50.628209  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:50.645464  9709 raft_consensus.cc:515] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:50.648748  9709 leader_election.cc:290] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 21 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:50.650060  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:50.650774  2335 raft_consensus.cc:3060] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:02:50.651757  1646 leader_election.cc:304] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:50.652901  9830 raft_consensus.cc:2804] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:02:50.653283  9830 raft_consensus.cc:493] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:50.653405  1646 leader_election.cc:304] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:50.653661  9830 raft_consensus.cc:3060] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:02:50.654255  9785 raft_consensus.cc:493] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:50.654835  9785 raft_consensus.cc:515] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:50.656981  9785 leader_election.cc:290] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:50.657567  9785 raft_consensus.cc:2804] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 13:02:50.658121  9785 raft_consensus.cc:493] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:50.656797  9513 consensus_queue.cc:1059] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 21.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 13:02:50.658840  1752 raft_consensus.cc:2468] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 19.
I20260709 13:02:50.659503  2335 raft_consensus.cc:2468] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 21.
I20260709 13:02:50.659777  1751 raft_consensus.cc:2468] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 21.
I20260709 13:02:50.665594  1569 leader_election.cc:304] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:02:50.668013  1569 leader_election.cc:304] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:02:50.668754  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:50.613207  9920 raft_consensus.cc:2804] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Leader election won for term 22
I20260709 13:02:50.672942  2301 leader_election.cc:304] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
W20260709 13:02:50.697841  1645 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:50.698652  2301 leader_election.cc:304] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:50.702750  2301 leader_election.cc:304] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:50.662415  9785 raft_consensus.cc:3060] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Advancing to term 24
I20260709 13:02:50.612682  9941 raft_consensus.cc:515] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:50.728135  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 60235.8us
I20260709 13:02:50.729076  1569 leader_election.cc:304] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
W20260709 13:02:50.708756  1644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:50.722819  9598 raft_consensus.cc:3055] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 LEADER]: Stepping down as leader of term 20
W20260709 13:02:50.705305  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 102230us
I20260709 13:02:50.734251  1676 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57092 (request call id 1802) took 1633 ms. Trace:
I20260709 13:02:50.735723  1676 rpcz_store.cc:276] 0709 13:02:49.100786 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:49.111460 (+ 10674us) service_pool.cc:224] Handling call
0709 13:02:49.121270 (+  9810us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:50.063213 (+941943us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:50.733992 (+670779us) spinlock_profiling.cc:243] Waited 941 ms on lock 0x7b54000230d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:02:50.734183 (+   191us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2165305088}
I20260709 13:02:50.732975  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 24 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:50.731885  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:50.737303  2330 raft_consensus.cc:3060] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Advancing to term 24
I20260709 13:02:50.747378  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:50.701157  9709 raft_consensus.cc:2749] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:02:50.703948  8994 raft_consensus.cc:2804] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 13:02:50.726272  9750 raft_consensus.cc:2804] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 13:02:50.763387  2328 raft_consensus.cc:2468] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 23.
I20260709 13:02:50.745939  9666 raft_consensus.cc:2804] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 13:02:50.765493  9984 raft_consensus.cc:493] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:50.735209  9598 raft_consensus.cc:740] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
W20260709 13:02:50.774572  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 45536.3us
I20260709 13:02:50.747592  9703 raft_consensus.cc:2749] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:02:50.747130  9875 raft_consensus.cc:493] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:50.760701  8994 raft_consensus.cc:493] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:50.739807  9840 consensus_queue.cc:1048] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.043s
I20260709 13:02:50.783540 10005 raft_consensus.cc:2749] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:02:50.778236  9984 raft_consensus.cc:515] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:50.757045  9936 raft_consensus.cc:493] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:50.763319  9819 raft_consensus.cc:493] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:50.765662  9750 raft_consensus.cc:493] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:50.758194  9845 raft_consensus.cc:493] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:50.781932  1569 leader_election.cc:304] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:02:50.778780  9700 raft_consensus.cc:3055] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 LEADER]: Stepping down as leader of term 21
I20260709 13:02:50.776823  9666 raft_consensus.cc:697] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 21 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
W20260709 13:02:50.881661  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 99865.5us
I20260709 13:02:50.795756  8994 raft_consensus.cc:3060] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 24 FOLLOWER]: Advancing to term 25
I20260709 13:02:50.883638  1569 leader_election.cc:304] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
W20260709 13:02:50.880606  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 210063us
I20260709 13:02:50.894469  1569 leader_election.cc:304] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:02:50.876277  9750 raft_consensus.cc:3060] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Advancing to term 24
I20260709 13:02:50.880107  9700 raft_consensus.cc:740] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:02:50.795189  9875 raft_consensus.cc:515] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:50.882313  9851 raft_consensus.cc:2749] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:02:50.910256  1569 leader_election.cc:304] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:02:50.908166  9920 raft_consensus.cc:697] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:02:50.883237  9845 raft_consensus.cc:515] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:50.878419  9936 raft_consensus.cc:515] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:50.879949  9819 raft_consensus.cc:515] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:50.929454  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:50.930075  2331 raft_consensus.cc:1240] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 28 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 27. Current term is 28. Ops: [27.17-27.17]
I20260709 13:02:50.907079 10005 raft_consensus.cc:2749] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:02:50.933440  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:50.935202  2333 raft_consensus.cc:3060] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:02:50.908506  9703 raft_consensus.cc:2804] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 23 FOLLOWER]: Leader election won for term 23
I20260709 13:02:50.934002  9821 consensus_queue.cc:1059] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 24.18, Next index: 19, Last known committed idx: 17, Time since last communication: 0.042s
I20260709 13:02:50.937506  2330 raft_consensus.cc:2468] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 24.
I20260709 13:02:50.955341  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:50.946969  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:50.943603 10003 raft_consensus.cc:2804] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Leader election won for term 21
W20260709 13:02:50.987403 10004 log.cc:927] Time spent T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.081s	user 0.000s	sys 0.000s
I20260709 13:02:51.119637  2332 raft_consensus.cc:2368] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 18. Current term is 19.
I20260709 13:02:50.922252  9588 raft_consensus.cc:493] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:50.946696  9853 raft_consensus.cc:493] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:50.936015  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:51.137058  2331 raft_consensus.cc:3060] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:02:51.047542  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 22 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:51.048182  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 19 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:51.048952  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 26 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:51.139096  1678 raft_consensus.cc:3060] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 25 FOLLOWER]: Advancing to term 26
I20260709 13:02:51.049119  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:51.049728  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 27 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:51.050024  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:02:51.045957  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 114047us
I20260709 13:02:51.140421  1680 raft_consensus.cc:2468] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 24.
I20260709 13:02:51.139842  1672 raft_consensus.cc:2468] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 20.
I20260709 13:02:51.134124  2328 raft_consensus.cc:1240] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 24 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 23. Current term is 24. Ops: []
I20260709 13:02:51.113363  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:50.957540  9666 consensus_queue.cc:237] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.14, Last appended by leader: 14, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:51.143795  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:51.144415  1675 raft_consensus.cc:3060] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:02:51.123050  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:51.123992  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:51.124758  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:51.148975  1673 raft_consensus.cc:2468] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 20.
I20260709 13:02:50.926383  9920 consensus_queue.cc:237] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.14, Last appended by leader: 14, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:51.145256  2298 leader_election.cc:336] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:02:51.150624  1674 raft_consensus.cc:2468] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 22.
W20260709 13:02:51.155390  1646 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:51.129520  2333 raft_consensus.cc:2468] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 22.
I20260709 13:02:51.162894  9953 raft_consensus.cc:493] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:51.176554  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:51.177477  1673 raft_consensus.cc:2468] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 17.
W20260709 13:02:51.152938  9979 consensus_meta.cc:287] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.644s	user 0.009s	sys 0.005s
I20260709 13:02:51.165151 10005 raft_consensus.cc:493] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:50.977451  9700 consensus_queue.cc:260] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.14, Last appended by leader: 14, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:51.182915  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:51.187872  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:51.191133  1675 raft_consensus.cc:2468] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 19.
I20260709 13:02:51.192430  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:51.193069  1674 raft_consensus.cc:2393] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 25: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:02:51.205591  2331 raft_consensus.cc:2468] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 23.
I20260709 13:02:51.141240  9588 raft_consensus.cc:515] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:51.145184  9853 raft_consensus.cc:515] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:51.216120  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:51.216346  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:51.217177  1675 raft_consensus.cc:3060] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 24 FOLLOWER]: Advancing to term 25
I20260709 13:02:51.153124  9830 raft_consensus.cc:515] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:51.216859  1673 raft_consensus.cc:2468] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 24.
I20260709 13:02:51.222390  1671 raft_consensus.cc:2368] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 20. Current term is 21.
W20260709 13:02:51.224102  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 49924.6us
W20260709 13:02:51.226454 10001 log.cc:927] Time spent T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.194s	user 0.000s	sys 0.005s
I20260709 13:02:51.227260 10010 consensus_queue.cc:1048] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20260709 13:02:51.227893  9830 leader_election.cc:290] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 21 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:02:51.230016  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 316743us
I20260709 13:02:51.230971  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:51.231782  2335 raft_consensus.cc:2393] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 25: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:02:51.232182  1718 leader_election.cc:304] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:51.232999  9955 consensus_queue.cc:1059] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20260709 13:02:51.233505  1718 leader_election.cc:400] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 18 pre-election: Vote denied by peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625) with higher term. Message: Invalid argument: T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 18. Current term is 19.
I20260709 13:02:51.233843  1718 leader_election.cc:403] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 18 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:02:51.233892  9955 consensus_queue.cc:1059] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 25.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20260709 13:02:51.234987  1718 leader_election.cc:304] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:51.235157  9955 raft_consensus.cc:2749] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:02:51.235937  9955 raft_consensus.cc:3055] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 23 LEADER]: Stepping down as leader of term 23
I20260709 13:02:51.236326  9955 raft_consensus.cc:740] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:02:51.237136  9955 consensus_queue.cc:260] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 23.17, Last appended by leader: 17, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:51.237510  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:51.238001  9814 raft_consensus.cc:2749] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Leader pre-election lost for term 18. Reason: Vote denied by peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625) with higher term. Message: Invalid argument: T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 18. Current term is 19.
I20260709 13:02:51.238359  9955 raft_consensus.cc:3060] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Advancing to term 24
I20260709 13:02:51.238749  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
W20260709 13:02:51.238946  1717 leader_election.cc:336] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:02:51.239444  9814 raft_consensus.cc:3055] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 27 LEADER]: Stepping down as leader of term 27
W20260709 13:02:51.239790  1717 leader_election.cc:336] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 23 election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 13:02:51.240444  1717 leader_election.cc:336] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:02:51.235918  9750 raft_consensus.cc:515] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:51.242898  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:51.243387  9594 raft_consensus.cc:2804] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 13:02:51.243494  1680 raft_consensus.cc:2376] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 18. Re-sending same reply.
I20260709 13:02:51.244069  9594 raft_consensus.cc:697] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 22 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:02:51.244606  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:51.245209  1680 raft_consensus.cc:3060] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:02:51.244940  9594 consensus_queue.cc:237] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.14, Last appended by leader: 14, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:51.239799  9814 raft_consensus.cc:740] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:02:51.248201  1675 raft_consensus.cc:2468] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 25.
I20260709 13:02:51.247967  9814 consensus_queue.cc:260] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 27.17, Last appended by leader: 17, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:51.249166  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 19 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:51.249274  9814 raft_consensus.cc:3060] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 27 FOLLOWER]: Advancing to term 28
I20260709 13:02:51.250053  2336 raft_consensus.cc:1240] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 21. Current term is 22. Ops: [19.13-21.14]
I20260709 13:02:51.251652  2334 raft_consensus.cc:2468] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 23.
I20260709 13:02:51.251929  1675 raft_consensus.cc:2468] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 18.
I20260709 13:02:51.292358  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:51.293725  2331 raft_consensus.cc:1240] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 21. Current term is 22. Ops: []
I20260709 13:02:51.294426  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:02:51.291116  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 50088.8us
I20260709 13:02:51.298242  1679 raft_consensus.cc:2410] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for term 19 because replica has last-logged OpId of term: 17 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 11.
I20260709 13:02:51.292586  1680 raft_consensus.cc:2468] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 18.
I20260709 13:02:51.217550  1644 leader_election.cc:304] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:02:51.196251 10005 raft_consensus.cc:515] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:51.308897  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 91458.5us
I20260709 13:02:51.310791  1569 leader_election.cc:304] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:02:51.306140  2299 leader_election.cc:304] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:02:51.306485  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:51.307384  2330 raft_consensus.cc:1240] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 20. Current term is 21. Ops: []
I20260709 13:02:51.313539  1674 raft_consensus.cc:3060] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 24 FOLLOWER]: Advancing to term 25
I20260709 13:02:51.326226  2335 raft_consensus.cc:1240] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 22. Current term is 23. Ops: []
I20260709 13:02:51.330206  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:51.331423  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:51.332165  1569 leader_election.cc:304] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:02:51.332185  1672 raft_consensus.cc:2468] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 20.
I20260709 13:02:51.335012  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:51.336278  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 19 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:51.339104 10012 raft_consensus.cc:2804] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Leader election won for term 23
I20260709 13:02:51.339850 10012 raft_consensus.cc:697] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 23 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:02:51.340932 10012 consensus_queue.cc:237] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.14, Last appended by leader: 14, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:51.343668  1673 raft_consensus.cc:2468] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 20.
W20260709 13:02:51.296322 10002 log.cc:927] Time spent T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.176s	user 0.000s	sys 0.001s
I20260709 13:02:51.345615  1644 leader_election.cc:304] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:02:51.245210  9750 leader_election.cc:290] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 24 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:51.189612  9979 raft_consensus.cc:515] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:51.296501  9700 raft_consensus.cc:3060] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:02:51.183508  9953 raft_consensus.cc:515] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:51.352391  1675 raft_consensus.cc:2468] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 23.
I20260709 13:02:51.330598  9945 raft_consensus.cc:493] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:51.345618  9785 consensus_meta.cc:287] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.603s	user 0.009s	sys 0.004s
I20260709 13:02:51.342752  9704 raft_consensus.cc:2804] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 24 FOLLOWER]: Leader pre-election won for term 25
W20260709 13:02:51.364825  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 196773us
I20260709 13:02:51.332597  9875 leader_election.cc:290] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:51.347201  9821 raft_consensus.cc:2804] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:02:51.368336  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 25 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:51.378564  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:51.368852  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:51.343952  9984 leader_election.cc:290] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:51.382997  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:51.387881  1673 raft_consensus.cc:2468] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 24.
I20260709 13:02:51.389719  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:51.390479  1673 raft_consensus.cc:2468] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 23.
I20260709 13:02:51.349711  9703 raft_consensus.cc:697] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 23 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:02:51.391678  9703 consensus_queue.cc:237] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 22.14, Last appended by leader: 14, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:51.395818  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:51.398313  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:51.399066  1673 raft_consensus.cc:2468] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 25.
I20260709 13:02:51.400028  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:51.402078  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
W20260709 13:02:51.387334  2298 leader_election.cc:336] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:02:51.403187  2298 leader_election.cc:304] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:02:51.404021  9953 leader_election.cc:290] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:51.354358  9845 leader_election.cc:290] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:51.404501  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 22 candidate_status { last_received { term: 21 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:51.405751  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:51.373602  9952 raft_consensus.cc:493] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:51.406940  1672 raft_consensus.cc:2410] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for term 19 because replica has last-logged OpId of term: 18 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 13.
W20260709 13:02:51.408330  2298 leader_election.cc:336] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:02:51.411859  2298 leader_election.cc:304] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:02:51.406303  1675 raft_consensus.cc:3060] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:02:51.367750  9921 raft_consensus.cc:2804] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:02:51.416179  9750 raft_consensus.cc:2749] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:02:51.366149  9783 raft_consensus.cc:2804] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 24 FOLLOWER]: Leader election won for term 24
I20260709 13:02:51.419708  9588 leader_election.cc:290] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:51.375743  9936 leader_election.cc:290] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:51.451970  1674 raft_consensus.cc:2468] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 25.
I20260709 13:02:51.378859  9704 raft_consensus.cc:493] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:02:51.422643  9873 consensus_meta.cc:287] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.878s	user 0.021s	sys 0.001s
I20260709 13:02:51.384495  9785 raft_consensus.cc:515] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 24 FOLLOWER]: Starting leader election with config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:51.410147  9875 raft_consensus.cc:2749] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:02:51.470260  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:51.471616  2336 raft_consensus.cc:2393] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 24: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:02:51.473321  1680 raft_consensus.cc:2393] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 24: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
W20260709 13:02:51.476421  8994 consensus_meta.cc:287] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.566s	user 0.008s	sys 0.003s
I20260709 13:02:51.476918  8994 raft_consensus.cc:515] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:51.478209  2331 raft_consensus.cc:2393] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 25: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:02:51.422956  9952 raft_consensus.cc:515] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:51.462193  9783 raft_consensus.cc:697] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 24 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:02:51.479645  9783 consensus_queue.cc:237] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 21.16, Last appended by leader: 16, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:51.480435  9952 leader_election.cc:290] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:51.426816  9598 consensus_queue.cc:260] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.15, Last appended by leader: 15, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:51.481490  9598 raft_consensus.cc:3060] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:02:51.481945  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:51.483186  2336 raft_consensus.cc:2393] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 23: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:02:51.428028  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:51.438391  1678 raft_consensus.cc:2468] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 26.
W20260709 13:02:51.448828  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 138677us
I20260709 13:02:51.385222  9945 raft_consensus.cc:515] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:51.492780  9945 leader_election.cc:290] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:51.495584  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:51.385442  9821 raft_consensus.cc:493] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:51.455440  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:51.498689  9821 raft_consensus.cc:3060] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:02:51.498992  1677 raft_consensus.cc:2468] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 23.
I20260709 13:02:51.500694  1677 raft_consensus.cc:1240] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 18. Current term is 19. Ops: []
W20260709 13:02:51.452041  2298 leader_election.cc:336] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 13:02:51.502769  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 50794.6us
I20260709 13:02:51.504021  9941 leader_election.cc:290] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 20 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:51.429566  9979 leader_election.cc:290] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 19 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:51.506592  9618 raft_consensus.cc:3055] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 13:02:51.507051  9618 raft_consensus.cc:740] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:02:51.508071  9618 consensus_queue.cc:260] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 24.18, Last appended by leader: 18, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:51.509501  9618 raft_consensus.cc:3060] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 24 FOLLOWER]: Advancing to term 25
I20260709 13:02:51.464447  1675 raft_consensus.cc:2468] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 21.
I20260709 13:02:51.512120  9819 leader_election.cc:290] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:51.467621  9785 leader_election.cc:290] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 24 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:51.460999  9704 raft_consensus.cc:3060] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 24 FOLLOWER]: Advancing to term 25
W20260709 13:02:51.498045  1717 leader_election.cc:336] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 21 election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:02:51.517236  1678 raft_consensus.cc:2468] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 19.
I20260709 13:02:51.526060  1678 raft_consensus.cc:1240] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 25 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 24. Current term is 25. Ops: []
I20260709 13:02:51.453116  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 21 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:51.521553  1673 raft_consensus.cc:3060] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:02:51.436862  9921 raft_consensus.cc:493] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:51.531962  1674 raft_consensus.cc:2410] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 for term 21 because replica has last-logged OpId of term: 19 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 15.
I20260709 13:02:51.453538 10003 raft_consensus.cc:697] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
W20260709 13:02:51.543305  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 344019us
W20260709 13:02:51.532946  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 200943us
I20260709 13:02:51.560391  1672 raft_consensus.cc:1240] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 17. Current term is 18. Ops: [15.10-17.11]
W20260709 13:02:51.550838  1645 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:51.561647  1569 leader_election.cc:304] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:02:51.563751  2299 leader_election.cc:304] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
W20260709 13:02:51.559028  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 324227us
I20260709 13:02:51.513257  9821 raft_consensus.cc:515] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:51.523496  1717 consensus_peers.cc:597] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:02:51.571637  1718 leader_election.cc:304] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:51.580593  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:51.578868  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:02:51.576380  1643 leader_election.cc:336] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:02:51.581310  1745 raft_consensus.cc:3060] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:02:51.582671  1643 leader_election.cc:304] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:02:51.558458  1673 raft_consensus.cc:2468] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 23.
I20260709 13:02:51.584187  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:51.584816  1750 raft_consensus.cc:3060] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Advancing to term 24
I20260709 13:02:51.585916  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:51.588820  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:51.590122  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 22 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:02:51.578415  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 55007.1us
W20260709 13:02:51.598266  1643 leader_election.cc:336] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:02:51.599001  1643 leader_election.cc:304] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:02:51.605607  1718 leader_election.cc:304] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 13:02:51.606932  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 45415.5us
I20260709 13:02:51.607969  1569 leader_election.cc:304] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
W20260709 13:02:51.609640  1643 leader_election.cc:336] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 18 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:02:51.610320  1643 leader_election.cc:304] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
W20260709 13:02:51.612371  2298 leader_election.cc:336] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 18 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:02:51.614921  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:02:51.616966  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 53324.8us
I20260709 13:02:51.617416  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 25 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:51.617761  1569 leader_election.cc:304] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:02:51.540951  9921 raft_consensus.cc:3060] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:02:51.613104  2298 leader_election.cc:304] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:51.622622  9979 raft_consensus.cc:2749] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 13:02:51.626214  8994 leader_election.cc:290] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 25 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:51.549476  9588 raft_consensus.cc:493] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:51.632236  1646 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:51.627166  1676 raft_consensus.cc:2468] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 22.
W20260709 13:02:51.619864  1643 consensus_peers.cc:597] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:02:51.627640  1569 leader_election.cc:304] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:02:51.629246  1749 raft_consensus.cc:2410] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for term 25 because replica has last-logged OpId of term: 23 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 16.
I20260709 13:02:51.604241  9952 raft_consensus.cc:2749] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:02:51.625914 10020 raft_consensus.cc:2749] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:02:51.630626  9953 raft_consensus.cc:2749] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:02:51.596915 10010 raft_consensus.cc:493] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
W20260709 13:02:51.649433  1643 consensus_peers.cc:597] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:02:51.517755  9873 raft_consensus.cc:515] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:51.616947  9937 raft_consensus.cc:2749] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
W20260709 13:02:51.626039  1717 leader_election.cc:336] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 25 election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:02:51.631667  9960 raft_consensus.cc:2804] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 13:02:51.618793 10005 leader_election.cc:290] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:02:51.675184  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 58490.5us
W20260709 13:02:51.687110  1643 leader_election.cc:336] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:02:51.687676  1643 leader_election.cc:304] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:51.559074  9819 consensus_queue.cc:1059] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 18.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.083s
I20260709 13:02:51.626415 10007 raft_consensus.cc:2749] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:02:51.643235  1569 leader_election.cc:304] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1
W20260709 13:02:51.696017  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 52935.9us
I20260709 13:02:51.696846  1569 leader_election.cc:304] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 13:02:51.678857  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:51.681224  1674 raft_consensus.cc:1240] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 22. Current term is 23. Ops: []
I20260709 13:02:51.684144  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:51.698565  1746 raft_consensus.cc:2468] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 23.
I20260709 13:02:51.641364  9979 raft_consensus.cc:2804] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Leader pre-election won for term 21
W20260709 13:02:51.686879  1717 leader_election.cc:336] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:02:51.691530  1673 raft_consensus.cc:1240] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 22. Current term is 23. Ops: [20.13-22.14]
I20260709 13:02:51.653048  9941 raft_consensus.cc:2804] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 13:02:51.707167  9941 raft_consensus.cc:493] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:51.707653  9941 raft_consensus.cc:3060] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:02:51.708475  1746 raft_consensus.cc:1240] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 20. Current term is 21. Ops: [20.15-20.15]
W20260709 13:02:51.693519  1643 leader_election.cc:336] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 22 election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 13:02:51.711279  1643 consensus_peers.cc:597] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:02:51.711880  1643 leader_election.cc:336] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:02:51.712249  1751 raft_consensus.cc:1275] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 19 index: 14. (index mismatch)
I20260709 13:02:51.650707  9869 consensus_queue.cc:1059] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.050s
I20260709 13:02:51.714367  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:51.715041  9869 raft_consensus.cc:3055] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 13:02:51.715230  1751 raft_consensus.cc:2393] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 23: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:02:51.715451  9869 raft_consensus.cc:740] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:02:51.716538  9869 consensus_queue.cc:260] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 20.17, Last appended by leader: 17, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:51.646646  9931 consensus_queue.cc:1059] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 21.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.050s
I20260709 13:02:51.718230  9931 raft_consensus.cc:3055] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 13:02:51.718592  9931 raft_consensus.cc:740] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:02:51.719547  9931 consensus_queue.cc:260] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.14, Last appended by leader: 14, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:51.712261  1643 leader_election.cc:304] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:02:51.721086  9931 raft_consensus.cc:3060] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Advancing to term 22
W20260709 13:02:51.722107  1643 consensus_peers.cc:597] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:02:51.650367  9757 consensus_queue.cc:1059] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.053s
I20260709 13:02:51.723117  9757 raft_consensus.cc:3055] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 13:02:51.705161  1569 leader_election.cc:304] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
W20260709 13:02:51.723153  1643 leader_election.cc:336] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:02:51.724128  1643 leader_election.cc:304] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:51.724941  9785 raft_consensus.cc:2749] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 13:02:51.699354  9953 raft_consensus.cc:2749] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:02:51.736001  1569 leader_election.cc:304] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:02:51.739071  1745 raft_consensus.cc:2468] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 21.
I20260709 13:02:51.699518  9960 raft_consensus.cc:697] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 23 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:02:51.739439 10011 raft_consensus.cc:2804] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:02:51.745010 10011 raft_consensus.cc:493] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:51.745213  9873 leader_election.cc:290] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 23 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:51.744809  9960 consensus_queue.cc:237] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 21.15, Last appended by leader: 15, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:51.739629 10024 raft_consensus.cc:2749] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 13:02:51.723637  9757 raft_consensus.cc:740] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:02:51.648232  9588 raft_consensus.cc:515] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:51.745441 10011 raft_consensus.cc:3060] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:02:51.670024  9424 consensus_queue.cc:1059] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 18.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.071s
W20260709 13:02:51.754449  1643 leader_election.cc:336] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 21 election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:02:51.759505  1643 leader_election.cc:304] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:02:51.702860  9853 leader_election.cc:290] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:51.748713  1569 leader_election.cc:304] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:02:51.699720  9945 raft_consensus.cc:2749] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:02:51.703771  9979 raft_consensus.cc:493] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Starting leader election (detected failure of leader af520b67945d443b901bbeca37cc5701)
W20260709 13:02:51.747506  9955 consensus_meta.cc:287] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.502s	user 0.010s	sys 0.001s
I20260709 13:02:51.730480 10005 raft_consensus.cc:2804] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:02:51.741603  9920 raft_consensus.cc:2804] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Leader election won for term 19
I20260709 13:02:51.742506 10020 raft_consensus.cc:3060] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 24 FOLLOWER]: Advancing to term 25
I20260709 13:02:51.772166  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 23 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:51.765198  1569 leader_election.cc:304] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:02:51.780239  1569 leader_election.cc:304] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:02:51.773680  9709 raft_consensus.cc:2804] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:02:51.781292  9709 raft_consensus.cc:493] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Starting leader election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:02:51.781708  9709 raft_consensus.cc:3060] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Advancing to term 21
W20260709 13:02:51.762742  1643 consensus_peers.cc:597] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:02:51.756832  9873 raft_consensus.cc:2804] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Leader election won for term 18
I20260709 13:02:51.781975  9588 leader_election.cc:290] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:51.782429  9873 raft_consensus.cc:697] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:02:51.783133  9757 consensus_queue.cc:260] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 22.15, Last appended by leader: 15, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:51.783480  9873 consensus_queue.cc:237] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.10, Last appended by leader: 10, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:51.784590  9757 raft_consensus.cc:3060] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Advancing to term 23
W20260709 13:02:51.787582  1643 consensus_peers.cc:597] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:02:51.788198  1643 leader_election.cc:336] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:02:51.788615  1643 leader_election.cc:304] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
W20260709 13:02:51.790462  1643 leader_election.cc:336] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:02:51.790730 10018 raft_consensus.cc:2804] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 13:02:51.790920  1643 leader_election.cc:304] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:51.791230 10018 raft_consensus.cc:493] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:51.791623 10018 raft_consensus.cc:3060] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 24 FOLLOWER]: Advancing to term 25
W20260709 13:02:51.792209  1643 leader_election.cc:336] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:02:51.792644  1643 leader_election.cc:304] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:51.781595  1569 leader_election.cc:304] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:02:51.769876  9953 raft_consensus.cc:2749] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 13:02:51.764729 10023 consensus_queue.cc:1048] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.054s
I20260709 13:02:51.801888  1569 leader_election.cc:304] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 13:02:51.677301 10010 raft_consensus.cc:515] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:51.773696  9979 raft_consensus.cc:3060] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:02:51.811975 10023 raft_consensus.cc:2804] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 13:02:51.812405  9920 raft_consensus.cc:697] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:02:51.812954  9840 raft_consensus.cc:2804] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 25 FOLLOWER]: Leader election won for term 25
I20260709 13:02:51.777158 10005 raft_consensus.cc:493] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:51.810297  1750 raft_consensus.cc:2468] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 24.
W20260709 13:02:51.817046  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 199445us
I20260709 13:02:51.811398  1569 leader_election.cc:304] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:02:51.805158  9703 raft_consensus.cc:2804] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Leader election won for term 20
I20260709 13:02:51.824044  1569 leader_election.cc:304] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 13:02:51.821586  2299 leader_election.cc:304] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:51.807104 10024 raft_consensus.cc:2749] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 13:02:51.827191  2299 leader_election.cc:304] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: af520b67945d443b901bbeca37cc5701
W20260709 13:02:51.818140  9814 consensus_meta.cc:287] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.568s	user 0.015s	sys 0.000s
I20260709 13:02:51.810034  9952 raft_consensus.cc:2749] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
W20260709 13:02:51.830592  1643 consensus_peers.cc:597] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:02:51.878670  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 58543.6us
I20260709 13:02:51.877655  1749 raft_consensus.cc:1275] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 28 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 27 index: 17. Preceding OpId from leader: term: 28 index: 17. (term mismatch)
W20260709 13:02:51.876596  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 264326us
I20260709 13:02:51.880311  1749 pending_rounds.cc:85] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701: Aborting all ops after (but not including) 16
I20260709 13:02:51.880652  1749 pending_rounds.cc:107] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701: Aborting uncommitted NO_OP operation due to leader change: 27.17
I20260709 13:02:51.880909  1749 raft_consensus.cc:2889] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 28 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 13:02:51.877128  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 50082.9us
W20260709 13:02:51.883330  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 313564us
I20260709 13:02:51.819568 10023 raft_consensus.cc:493] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:51.885035  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
W20260709 13:02:51.875345  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 51396.3us
I20260709 13:02:51.885888  1718 leader_election.cc:304] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:51.888077  2330 raft_consensus.cc:1240] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 22. Current term is 23. Ops: []
W20260709 13:02:51.879776  1643 leader_election.cc:336] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:02:51.882234  9703 raft_consensus.cc:697] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:02:51.909072  1569 leader_election.cc:304] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:02:51.821749 10005 raft_consensus.cc:3060] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Advancing to term 18
W20260709 13:02:51.967012  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 81296.5us
I20260709 13:02:51.826043  9588 raft_consensus.cc:2749] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:02:51.829051  9598 raft_consensus.cc:2804] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 26 FOLLOWER]: Leader election won for term 26
I20260709 13:02:51.837579 10012 raft_consensus.cc:2749] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 13:02:51.952451  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:51.955549  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:52.038300  1679 raft_consensus.cc:2468] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 19.
I20260709 13:02:51.956542  1677 raft_consensus.cc:1275] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 18 index: 13. (index mismatch)
I20260709 13:02:51.907081  9998 raft_consensus.cc:2804] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 13:02:52.040601  9998 raft_consensus.cc:493] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 23 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:52.041157  9998 raft_consensus.cc:3060] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 23 FOLLOWER]: Advancing to term 24
I20260709 13:02:51.829347  9840 raft_consensus.cc:697] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 25 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:02:52.047511  9840 consensus_queue.cc:237] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 24.17, Last appended by leader: 17, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:51.747038 10003 consensus_queue.cc:237] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 19.16, Last appended by leader: 16, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:51.891372  1750 raft_consensus.cc:3060] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Advancing to term 25
I20260709 13:02:51.889359  9819 raft_consensus.cc:493] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:52.054879  9819 raft_consensus.cc:515] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:51.960673  1643 leader_election.cc:304] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:02:51.825889  9934 raft_consensus.cc:493] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:51.962841  8994 raft_consensus.cc:2804] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 13:02:51.826364  9945 consensus_queue.cc:1059] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 24.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.132s
I20260709 13:02:52.014168  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 19 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:52.058749  9934 raft_consensus.cc:515] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:02:52.017164  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 108257us
I20260709 13:02:52.060523  1569 leader_election.cc:304] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:02:51.967324  9814 raft_consensus.cc:2804] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 13:02:52.061664  9934 leader_election.cc:290] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:52.061792  9814 raft_consensus.cc:697] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 23 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:02:51.959465  9821 leader_election.cc:290] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 19 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:51.911629 10023 raft_consensus.cc:3060] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 23 FOLLOWER]: Advancing to term 24
I20260709 13:02:52.059522  1671 raft_consensus.cc:3060] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Advancing to term 19
W20260709 13:02:52.014786  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 45806.5us
W20260709 13:02:52.073979  1718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:52.063551  1674 raft_consensus.cc:3060] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 24 FOLLOWER]: Advancing to term 25
W20260709 13:02:52.068333  2300 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:52.068905  8994 raft_consensus.cc:697] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 23 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
W20260709 13:02:52.069867  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 190191us
I20260709 13:02:52.085616  8994 consensus_queue.cc:237] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.14, Last appended by leader: 14, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:52.062947  9814 consensus_queue.cc:237] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 22.16, Last appended by leader: 16, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:52.089344  2301 leader_election.cc:304] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:02:52.090939  2301 leader_election.cc:304] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:02:52.076754  9588 raft_consensus.cc:493] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:52.097596  1671 raft_consensus.cc:2468] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 19.
I20260709 13:02:52.076493  9783 raft_consensus.cc:2749] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 13:02:52.057823  9703 consensus_queue.cc:237] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 19.16, Last appended by leader: 16, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:52.083962  1750 raft_consensus.cc:2468] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 25.
I20260709 13:02:52.107436  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:52.107628  2299 leader_election.cc:304] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:02:52.070149 10022 consensus_queue.cc:1059] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.067s
I20260709 13:02:52.110622  2299 leader_election.cc:304] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:02:52.108428  2301 leader_election.cc:304] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:52.101267  9851 raft_consensus.cc:493] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:52.022053  9920 consensus_queue.cc:237] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:52.142062  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:52.136957  1678 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57092 (request call id 1820) took 1081 ms. Trace:
W20260709 13:02:52.144003  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 83596.7us
I20260709 13:02:52.143278  1678 rpcz_store.cc:276] 0709 13:02:51.055113 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:51.527083 (+471970us) service_pool.cc:224] Handling call
0709 13:02:51.590586 (+ 63503us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:51.591064 (+   478us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:51.591101 (+    37us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:51.591142 (+    41us) raft_consensus.cc:1645] Marking committed up to 14
0709 13:02:51.591180 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:51.591234 (+    54us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:52.136787 (+545553us) spinlock_profiling.cc:243] Waited 63.1 ms on lock 0x7b540025add0. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661618ad 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:02:52.136894 (+   107us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":145232640}
I20260709 13:02:52.146678  1569 leader_election.cc:304] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
W20260709 13:02:52.130282 10033 log.cc:927] Time spent T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.172s	user 0.001s	sys 0.000s
W20260709 13:02:52.134811 10030 log.cc:927] Time spent T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.252s	user 0.000s	sys 0.000s
I20260709 13:02:52.122382  1674 raft_consensus.cc:2468] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 25.
I20260709 13:02:52.150499  1569 leader_election.cc:400] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 20 pre-election: Vote denied by peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261) with higher term. Message: Invalid argument: T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 20. Current term is 21.
I20260709 13:02:52.154022  1569 leader_election.cc:403] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 20 pre-election: Cancelling election due to peer responding with higher term
W20260709 13:02:52.152158 10027 log.cc:927] Time spent T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.225s	user 0.001s	sys 0.001s
I20260709 13:02:52.142273 10010 leader_election.cc:290] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:52.152626  9709 raft_consensus.cc:515] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:52.130746 10048 raft_consensus.cc:2804] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 13:02:52.127859  9873 raft_consensus.cc:2749] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:02:52.226085  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
W20260709 13:02:52.154712  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 79638.3us
I20260709 13:02:52.229688  1718 leader_election.cc:304] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:52.184927  1750 raft_consensus.cc:1275] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 21 index: 14. Preceding OpId from leader: term: 22 index: 15. (index mismatch)
W20260709 13:02:52.149655 10016 log.cc:927] Time spent T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.805s	user 0.007s	sys 0.000s
I20260709 13:02:52.104622  9588 raft_consensus.cc:515] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:52.235807  1718 leader_election.cc:304] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 13:02:52.167505 10015 log.cc:927] Time spent T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.817s	user 0.000s	sys 0.005s
I20260709 13:02:52.169823 10010 raft_consensus.cc:2804] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Leader pre-election won for term 20
W20260709 13:02:52.248934  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 138570us
I20260709 13:02:52.167505 10020 raft_consensus.cc:2749] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 13:02:52.184132 10047 raft_consensus.cc:2749] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 13:02:52.167057 10043 raft_consensus.cc:2749] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:02:52.243045  9873 raft_consensus.cc:493] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:52.246099  1718 leader_election.cc:304] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:52.175539 10035 raft_consensus.cc:2804] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 13:02:52.233744 10003 raft_consensus.cc:2749] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:02:52.183651  9757 raft_consensus.cc:493] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:52.182930 10048 raft_consensus.cc:493] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:52.130733  9851 raft_consensus.cc:515] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:52.260516  9588 leader_election.cc:290] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:52.261288 10050 raft_consensus.cc:493] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:52.254179 10010 raft_consensus.cc:493] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:02:52.262738 10032 log.cc:927] Time spent T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.316s	user 0.000s	sys 0.004s
I20260709 13:02:52.232950  9709 leader_election.cc:290] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 21 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:52.165588  9979 raft_consensus.cc:515] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:52.243219  9931 raft_consensus.cc:2804] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 25 FOLLOWER]: Leader election won for term 25
W20260709 13:02:52.250743 10042 log.cc:927] Time spent T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.116s	user 0.003s	sys 0.000s
I20260709 13:02:52.265327  2330 raft_consensus.cc:1275] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 19 index: 13. Preceding OpId from leader: term: 22 index: 15. (index mismatch)
W20260709 13:02:52.250654 10026 log.cc:927] Time spent T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.271s	user 0.004s	sys 0.000s
W20260709 13:02:52.250527 10037 log.cc:927] Time spent T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.150s	user 0.000s	sys 0.004s
W20260709 13:02:52.248087 10040 log.cc:927] Time spent T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.127s	user 0.003s	sys 0.000s
W20260709 13:02:52.250108 10039 log.cc:927] Time spent T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.074s	user 0.001s	sys 0.002s
I20260709 13:02:52.282146  1718 leader_election.cc:304] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 13:02:52.157449 10044 consensus_queue.cc:1048] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.108s
I20260709 13:02:52.248329 10022 raft_consensus.cc:2749] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
W20260709 13:02:52.264757  9618 consensus_meta.cc:287] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.755s	user 0.011s	sys 0.006s
I20260709 13:02:52.301790  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 26 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:52.318740  1671 raft_consensus.cc:2410] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 for term 26 because replica has last-logged OpId of term: 24 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 16.
I20260709 13:02:52.319013 10052 raft_consensus.cc:493] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:52.319883  9704 consensus_meta.cc:287] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.770s	user 0.014s	sys 0.001s
I20260709 13:02:52.280963  9840 raft_consensus.cc:493] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:52.303444 10010 raft_consensus.cc:3060] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:02:52.320463  9704 raft_consensus.cc:515] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:52.320734  9840 raft_consensus.cc:515] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:52.152222  9598 raft_consensus.cc:697] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 26 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
W20260709 13:02:52.301992 10031 log.cc:927] Time spent T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.167s	user 0.002s	sys 0.001s
I20260709 13:02:52.301509 10050 raft_consensus.cc:515] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:52.297593 10046 raft_consensus.cc:2804] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 13:02:52.290717  9424 raft_consensus.cc:2804] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 13:02:52.291249 10020 raft_consensus.cc:3060] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Advancing to term 21
I20260709 13:02:52.291020 10048 raft_consensus.cc:3060] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:02:52.324556  1718 leader_election.cc:304] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:52.319583 10052 raft_consensus.cc:515] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:52.294715  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 144455us
I20260709 13:02:52.362115  1672 raft_consensus.cc:1240] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 18. Current term is 19. Ops: []
I20260709 13:02:52.363651  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
W20260709 13:02:52.314850  9941 consensus_meta.cc:287] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.607s	user 0.008s	sys 0.008s
I20260709 13:02:52.314314  9931 raft_consensus.cc:697] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 25 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:02:52.299860  9757 raft_consensus.cc:515] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:52.368036  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 43641.9us
I20260709 13:02:52.374727  1718 leader_election.cc:304] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 13:02:52.369509  2298 leader_election.cc:336] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 21 election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:02:52.369745  1676 raft_consensus.cc:2393] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 19: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:02:52.370748  1672 raft_consensus.cc:1275] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 18 index: 12. Preceding OpId from leader: term: 23 index: 15. (index mismatch)
I20260709 13:02:52.313316 10003 raft_consensus.cc:493] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:52.377254  1671 raft_consensus.cc:1275] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 18 index: 14. Preceding OpId from leader: term: 19 index: 15. (index mismatch)
I20260709 13:02:52.375524  1675 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57092 (request call id 1816) took 1321 ms. Trace:
I20260709 13:02:52.371012  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:52.378752  1675 rpcz_store.cc:276] 0709 13:02:51.054077 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:51.511851 (+457774us) service_pool.cc:224] Handling call
0709 13:02:51.512250 (+   399us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:51.512724 (+   474us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:51.512764 (+    40us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:51.513374 (+   610us) log.cc:844] Serialized 26 byte log entry
0709 13:02:51.771385 (+258011us) raft_consensus.cc:1645] Marking committed up to 17
0709 13:02:51.771427 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:51.771499 (+    72us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:52.375294 (+603795us) raft_consensus.cc:1685] finished
0709 13:02:52.375320 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:52.375454 (+   134us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":10527,"raft.run_cpu_time_us":146,"raft.run_wall_time_us":5725,"thread_start_us":257640,"threads_started":1,"wal-append.queue_time_us":310396}
I20260709 13:02:52.287704  9873 raft_consensus.cc:515] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:52.346719  9955 raft_consensus.cc:2804] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 13:02:52.353305 10046 raft_consensus.cc:493] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:52.384061  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:52.386554  1718 leader_election.cc:304] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:52.393771  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:52.396710  1672 raft_consensus.cc:2393] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 25: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
W20260709 13:02:52.381402 10041 log.cc:927] Time spent T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.135s	user 0.001s	sys 0.003s
W20260709 13:02:52.402096  1717 consensus_peers.cc:597] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:02:52.537607  9819 leader_election.cc:290] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:52.392426 10046 raft_consensus.cc:3060] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Advancing to term 24
I20260709 13:02:52.382323  9955 raft_consensus.cc:493] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:52.744025  1746 raft_consensus.cc:3055] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 13:02:52.740247  9873 leader_election.cc:290] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:52.373564  9598 consensus_queue.cc:237] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 24.12, Last appended by leader: 12, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:52.399905  9814 raft_consensus.cc:2749] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:02:52.390924 10022 raft_consensus.cc:2749] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:02:52.363224 10058 raft_consensus.cc:493] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:52.424367  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:52.902863  1671 raft_consensus.cc:1813] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 24 FOLLOWER]: Handling vote request from an unknown peer 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:52.903530  1671 raft_consensus.cc:2393] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 24: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
W20260709 13:02:52.737815  2298 leader_election.cc:336] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:02:52.743350  1677 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52778 (request call id 2982) took 1554 ms. Trace:
W20260709 13:02:52.904362  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 506355us
I20260709 13:02:52.747479  1749 raft_consensus.cc:3055] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [term 17 LEADER]: Stepping down as leader of term 17
W20260709 13:02:52.751366  2300 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:52.905694  1674 raft_consensus.cc:1240] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 26 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 25. Current term is 26. Ops: []
I20260709 13:02:52.905699  1749 raft_consensus.cc:740] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:02:52.386502 10003 raft_consensus.cc:515] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:52.460670  9931 consensus_queue.cc:237] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 23.13, Last appended by leader: 13, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:52.373099  9941 raft_consensus.cc:515] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:02:52.649506  2301 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.537s	user 0.007s	sys 0.014s
W20260709 13:02:53.170331  2301 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.058s	user 0.011s	sys 0.018s
W20260709 13:02:53.174849  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 1.06678e+06us
I20260709 13:02:53.175762 10050 leader_election.cc:290] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:53.176597 10046 raft_consensus.cc:515] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:52.660538  9704 leader_election.cc:290] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 25 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:02:52.742009 10023 consensus_meta.cc:287] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.661s	user 0.011s	sys 0.002s
I20260709 13:02:52.809049  1746 raft_consensus.cc:740] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:02:53.179056 10023 raft_consensus.cc:515] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:53.180141  1746 consensus_queue.cc:260] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 21.14, Last appended by leader: 14, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:53.181159  2299 leader_election.cc:304] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:02:53.181528  1746 raft_consensus.cc:3060] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Advancing to term 22
W20260709 13:02:52.798659  9921 consensus_meta.cc:287] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 1.156s	user 0.014s	sys 0.009s
I20260709 13:02:53.182267 10047 raft_consensus.cc:2804] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 25 FOLLOWER]: Leader election won for term 25
I20260709 13:02:53.182926 10047 raft_consensus.cc:697] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 25 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:02:53.182744  9921 raft_consensus.cc:515] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } }
I20260709 13:02:53.183892 10047 consensus_queue.cc:237] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 23.17, Last appended by leader: 17, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:53.185141  9921 leader_election.cc:290] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 21 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:52.851023  9955 raft_consensus.cc:3060] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Advancing to term 25
I20260709 13:02:52.751120  9588 raft_consensus.cc:493] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:53.190794  9588 raft_consensus.cc:515] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:53.193228  9588 leader_election.cc:290] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:53.194612  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:53.195950  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 23 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:53.196993  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:53.199191  1676 raft_consensus.cc:1275] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 21 index: 14. Preceding OpId from leader: term: 23 index: 15. (index mismatch)
I20260709 13:02:53.200599  1676 raft_consensus.cc:1275] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 25 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 24 index: 17. Preceding OpId from leader: term: 25 index: 18. (index mismatch)
W20260709 13:02:52.798290 10018 consensus_meta.cc:287] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 1.006s	user 0.018s	sys 0.004s
I20260709 13:02:53.202338 10018 raft_consensus.cc:515] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:53.204771 10018 leader_election.cc:290] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 25 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:02:52.794633 10011 consensus_meta.cc:287] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 1.024s	user 0.007s	sys 0.013s
I20260709 13:02:53.205252  9955 raft_consensus.cc:515] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:53.205641 10011 raft_consensus.cc:515] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:53.208137 10011 leader_election.cc:290] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 20 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:52.904001  1677 rpcz_store.cc:276] 0709 13:02:51.190731 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:51.817300 (+626569us) service_pool.cc:224] Handling call
0709 13:02:51.956093 (+138793us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:52.039503 (+ 83410us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:52.651777 (+612274us) spinlock_profiling.cc:243] Waited 138 ms on lock 0x7b54001f6150. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661618ad 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:02:52.743255 (+ 91478us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":318535552}
W20260709 13:02:52.748672  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 362351us
I20260709 13:02:52.909255  1678 raft_consensus.cc:1240] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 23 LEADER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 22. Current term is 23. Ops: []
I20260709 13:02:52.910467  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:02:52.802979  9945 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.722s	user 0.009s	sys 0.012s
I20260709 13:02:52.796166  9934 raft_consensus.cc:2804] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 13:02:53.234557 10022 consensus_queue.cc:1059] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 18.12, Next index: 13, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 13:02:53.237670 10003 leader_election.cc:290] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:52.737824  9709 raft_consensus.cc:493] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
W20260709 13:02:52.850368 10049 log.cc:927] Time spent T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.546s	user 0.010s	sys 0.000s
I20260709 13:02:52.743860  9875 raft_consensus.cc:3055] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 13:02:52.757297  9594 raft_consensus.cc:3055] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 13:02:53.252235  9594 raft_consensus.cc:740] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:02:53.253252  9594 consensus_queue.cc:260] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 22.15, Last appended by leader: 15, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:53.257766  9869 raft_consensus.cc:3060] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:02:53.247934  1673 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52778 (request call id 2976) took 2072 ms. Trace:
I20260709 13:02:52.936769  1674 raft_consensus.cc:1275] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 20 index: 11. Preceding OpId from leader: term: 23 index: 13. (index mismatch)
I20260709 13:02:52.397419 10035 raft_consensus.cc:697] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 19 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:02:52.761461  9703 raft_consensus.cc:493] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:52.940649  9666 raft_consensus.cc:493] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:52.941620  1749 consensus_queue.cc:260] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 17.11, Last appended by leader: 11, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:52.943488  9840 leader_election.cc:290] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:02:52.958619  9998 consensus_meta.cc:287] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.917s	user 0.015s	sys 0.006s
I20260709 13:02:52.958191  9814 raft_consensus.cc:493] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
W20260709 13:02:52.990497 10051 log.cc:927] Time spent T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.479s	user 0.000s	sys 0.010s
I20260709 13:02:52.920003 10058 raft_consensus.cc:515] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:52.992069  9851 leader_election.cc:290] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:52.991780  9873 raft_consensus.cc:493] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:53.023839  9920 raft_consensus.cc:493] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:53.006177 10052 leader_election.cc:290] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:52.998088  9979 leader_election.cc:290] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 21 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:02:53.018646 10048 consensus_meta.cc:287] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.637s	user 0.015s	sys 0.000s
I20260709 13:02:53.300931 10048 raft_consensus.cc:515] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:02:53.033547 10059 log.cc:927] Time spent T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.113s	user 0.001s	sys 0.000s
W20260709 13:02:53.034346 10057 log.cc:927] Time spent T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.116s	user 0.000s	sys 0.000s
I20260709 13:02:53.003326  9757 leader_election.cc:290] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:53.058892  1646 leader_election.cc:304] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
W20260709 13:02:53.069024 10005 consensus_meta.cc:287] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 1.032s	user 0.010s	sys 0.010s
I20260709 13:02:53.305675 10072 raft_consensus.cc:2749] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260709 13:02:53.305918  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 247224us
W20260709 13:02:53.068058 10020 consensus_meta.cc:287] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.700s	user 0.012s	sys 0.003s
I20260709 13:02:53.306787 10020 raft_consensus.cc:2749] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Leader pre-election lost for term 20. Reason: Vote denied by peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261) with higher term. Message: Invalid argument: T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 20. Current term is 21.
I20260709 13:02:53.306921  1646 leader_election.cc:304] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:02:53.067564 10009 raft_consensus.cc:493] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:53.280632  9594 raft_consensus.cc:3060] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:02:53.307813 10009 raft_consensus.cc:515] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:53.309145  1646 leader_election.cc:304] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:53.310029  1646 leader_election.cc:304] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 13:02:53.086330 10073 raft_consensus.cc:493] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:53.108970  1569 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:53.310906 10073 raft_consensus.cc:515] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:53.313489 10073 leader_election.cc:290] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:53.313670  1569 leader_election.cc:304] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:02:53.103786 10044 raft_consensus.cc:493] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:53.314483  9920 raft_consensus.cc:515] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:53.314751 10044 raft_consensus.cc:515] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:53.316094  9785 raft_consensus.cc:2804] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Leader election won for term 21
I20260709 13:02:53.316711  9785 raft_consensus.cc:697] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:02:53.316922  9920 leader_election.cc:290] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:53.317049 10044 leader_election.cc:290] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:02:53.120944 10064 log.cc:927] Time spent T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.065s	user 0.001s	sys 0.000s
I20260709 13:02:53.317572  9873 raft_consensus.cc:515] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:53.136147 10066 consensus_queue.cc:1048] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.003s
I20260709 13:02:53.122594  9941 leader_election.cc:290] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 23 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:53.142750  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:53.319813  9873 leader_election.cc:290] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:53.319896  1751 raft_consensus.cc:2468] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 18.
I20260709 13:02:53.143179  1680 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57092 (request call id 1826) took 1843 ms. Trace:
I20260709 13:02:53.320730  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:53.141886  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:53.145520  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 23 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:53.313566 10052 consensus_queue.cc:1059] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.008s
I20260709 13:02:53.321679  1750 raft_consensus.cc:1813] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Handling vote request from an unknown peer 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 13:02:53.143980  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 79885.7us
I20260709 13:02:53.158934  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:53.322459  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:53.307763 10058 leader_election.cc:290] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:53.323625  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:53.314198 10073 consensus_queue.cc:1048] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 11, Time since last communication: 0.010s
W20260709 13:02:53.323941  1643 leader_election.cc:336] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 18 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:02:53.324455  1643 leader_election.cc:304] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:53.324899  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:53.146476  1748 raft_consensus.cc:3055] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 13:02:53.181411 10023 leader_election.cc:290] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 24 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:53.320984  1680 rpcz_store.cc:276] 0709 13:02:51.299721 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:52.020914 (+721193us) service_pool.cc:224] Handling call
0709 13:02:52.021361 (+   447us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:52.021774 (+   413us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:52.034570 (+ 12796us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:52.533022 (+498452us) spinlock_profiling.cc:243] Waited 33.3 ms on lock 0x7b2c00010828. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4264a5ba11 00007f4264a6b372 00007f426619da01 00007f426616243a 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2
0709 13:02:52.745282 (+212260us) log.cc:844] Serialized 26 byte log entry
0709 13:02:53.022292 (+277010us) raft_consensus.cc:1645] Marking committed up to 16
0709 13:02:53.104633 (+ 82341us) log.cc:844] Serialized 14 byte log entry
0709 13:02:53.104912 (+   279us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:53.105000 (+    88us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:53.142831 (+ 37831us) raft_consensus.cc:1685] finished
0709 13:02:53.142956 (+   125us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:53.143108 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":9745,"prepare.queue_time_us":94969,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":50,"raft.queue_time_us":1065,"raft.run_cpu_time_us":326,"raft.run_wall_time_us":41248,"spinlock_wait_cycles":76546432,"thread_start_us":161487,"threads_started":2,"wal-append.queue_time_us":176799}
I20260709 13:02:53.326023  1748 raft_consensus.cc:740] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:02:53.159667  9960 raft_consensus.cc:493] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:52.816174  9424 raft_consensus.cc:697] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 23 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:02:53.327562 10072 raft_consensus.cc:2749] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:02:53.327430  9960 raft_consensus.cc:515] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:53.193253  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:53.328482  1671 raft_consensus.cc:2468] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 24.
I20260709 13:02:53.328142  9424 consensus_queue.cc:237] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 22.12, Last appended by leader: 12, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:53.175657  2301 leader_election.cc:304] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:02:53.329707  9960 leader_election.cc:290] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:53.168820 10083 raft_consensus.cc:493] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
W20260709 13:02:53.329986  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 154492us
W20260709 13:02:52.820179  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 1.11357e+06us
I20260709 13:02:53.285570  9814 raft_consensus.cc:515] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:53.217013  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 24 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:53.220634  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:53.332044  1672 raft_consensus.cc:3060] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:02:53.330511 10083 raft_consensus.cc:515] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:53.332772  9814 leader_election.cc:290] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:53.230918 10056 consensus_queue.cc:1048] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 13:02:53.234866  9934 raft_consensus.cc:697] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 21 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:02:53.325536  1752 raft_consensus.cc:2468] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 21.
I20260709 13:02:52.923727  1675 raft_consensus.cc:1240] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 18. Current term is 19. Ops: []
I20260709 13:02:53.249514  1746 raft_consensus.cc:1275] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 21 index: 14. Preceding OpId from leader: term: 22 index: 14. (term mismatch)
I20260709 13:02:53.335937  1746 pending_rounds.cc:85] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701: Aborting all ops after (but not including) 13
W20260709 13:02:53.325788  1643 consensus_peers.cc:597] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:02:53.335263  9934 consensus_queue.cc:237] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.12, Last appended by leader: 12, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } }
I20260709 13:02:53.259130  1673 rpcz_store.cc:276] 0709 13:02:51.175209 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:51.718850 (+543641us) service_pool.cc:224] Handling call
0709 13:02:51.719203 (+   353us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:51.719598 (+   395us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:51.719632 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:51.720191 (+   559us) log.cc:844] Serialized 26 byte log entry
0709 13:02:51.911938 (+191747us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:02:51.911986 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:51.912066 (+    80us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:53.232972 (+1320906us) spinlock_profiling.cc:243] Waited 5.04 ms on lock 0x7b5400266c20. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42660d8aaa 00007f42660d7dd9 00007f42660e2930 00007f42660e28d8 00007f42660e2858 00007f42660e2813 00007f42660e1a9c 00007f426bfcd626 00007f426bfc7dd8 00007f426611e7cb
0709 13:02:53.238574 (+  5602us) raft_consensus.cc:1685] finished
0709 13:02:53.238604 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:53.247836 (+  9232us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":3303,"raft.run_cpu_time_us":121,"raft.run_wall_time_us":121,"spinlock_wait_cycles":11588352,"thread_start_us":191281,"threads_started":1,"wal-append.queue_time_us":381779}
W20260709 13:02:53.339412  1643 leader_election.cc:336] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 25 election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:02:53.275744  9666 raft_consensus.cc:515] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:53.276018 10035 consensus_queue.cc:237] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 17.15, Last appended by leader: 15, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:53.275590  9937 consensus_queue.cc:1059] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 14.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 13:02:53.288460  9955 leader_election.cc:290] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 25 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:53.284345  9998 raft_consensus.cc:515] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 24 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:53.343410  9666 leader_election.cc:290] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:53.277828 10046 leader_election.cc:290] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 24 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:53.290797  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:53.281690  9703 raft_consensus.cc:515] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:53.345606  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:53.345793  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 25 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:53.294919 10024 raft_consensus.cc:493] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:53.346750 10024 raft_consensus.cc:515] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:53.349211 10024 leader_election.cc:290] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:02:53.066424 10010 consensus_meta.cc:287] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.746s	user 0.019s	sys 0.000s
I20260709 13:02:53.349910 10010 raft_consensus.cc:515] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:53.317749  9785 consensus_queue.cc:237] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.10, Last appended by leader: 10, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:53.305514 10005 raft_consensus.cc:515] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:53.313827  9830 raft_consensus.cc:2749] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
W20260709 13:02:53.342273  2301 connection.cc:256] RPC call timeout handler was delayed by -1.0205170950000024s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 1.5s for a response
I20260709 13:02:53.354095 10011 raft_consensus.cc:493] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:53.354586 10011 raft_consensus.cc:515] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:53.356541 10011 leader_election.cc:290] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:53.346324  1753 raft_consensus.cc:2468] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 21.
I20260709 13:02:53.357053 10011 raft_consensus.cc:493] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:53.357534 10011 raft_consensus.cc:515] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:53.357776  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 22 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:53.354115 10005 leader_election.cc:290] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 18 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:53.358807  1753 raft_consensus.cc:2410] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for term 22 because replica has last-logged OpId of term: 20 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 13.
I20260709 13:02:53.335518  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 23 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:53.360987 10011 leader_election.cc:290] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:53.360910  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 21 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:53.321637 10009 leader_election.cc:290] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:53.360145  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:53.330772  9821 raft_consensus.cc:2749] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 13:02:53.201876  1676 raft_consensus.cc:1275] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 19 index: 16. Preceding OpId from leader: term: 20 index: 17. (index mismatch)
I20260709 13:02:53.198117  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:53.321179  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:02:53.213999  9945 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.135s	user 0.016s	sys 0.026s
I20260709 13:02:53.371174  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:53.372488  2330 raft_consensus.cc:1240] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 21 LEADER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 20. Current term is 21. Ops: [20.15-20.15]
I20260709 13:02:53.373190  2333 raft_consensus.cc:1275] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 28 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 25 index: 16. Preceding OpId from leader: term: 28 index: 17. (index mismatch)
I20260709 13:02:53.373697  2329 raft_consensus.cc:1275] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 18 index: 12. Preceding OpId from leader: term: 22 index: 14. (index mismatch)
I20260709 13:02:53.251658  9875 raft_consensus.cc:740] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:02:53.337957 10056 consensus_queue.cc:1059] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 18.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 13:02:53.371799  1751 raft_consensus.cc:2468] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 23.
I20260709 13:02:53.375115  9703 leader_election.cc:290] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:53.340829  1644 leader_election.cc:304] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:02:53.375470  9875 consensus_queue.cc:260] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:53.341334  1750 raft_consensus.cc:2393] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 24: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:02:53.376986  9875 raft_consensus.cc:3060] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:02:53.377130  9998 leader_election.cc:290] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 24 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:53.377740  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:53.378512  1750 raft_consensus.cc:2468] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 24.
I20260709 13:02:53.379581  1749 raft_consensus.cc:3060] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:02:53.379647  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 26 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:53.380690  1750 raft_consensus.cc:2410] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 for term 26 because replica has last-logged OpId of term: 24 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 16.
I20260709 13:02:53.381507  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 24 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:53.383004  9830 raft_consensus.cc:2749] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 13:02:53.260226  9709 raft_consensus.cc:515] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:53.323573  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:53.384053 10048 leader_election.cc:290] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 22 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:53.384832  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:02:53.385049  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 479736us
I20260709 13:02:53.385526  1747 raft_consensus.cc:2468] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 23.
I20260709 13:02:53.352232 10010 leader_election.cc:290] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 20 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:53.368685  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:53.369832  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:53.386734  1673 raft_consensus.cc:3060] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:02:53.386972  2331 raft_consensus.cc:2393] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 20: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:02:53.372630  2334 raft_consensus.cc:1275] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 16 index: 12. Preceding OpId from leader: term: 19 index: 14. (index mismatch)
I20260709 13:02:53.336314  1746 pending_rounds.cc:107] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701: Aborting uncommitted NO_OP operation due to leader change: 21.14
I20260709 13:02:53.388578 10036 raft_consensus.cc:3055] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 13:02:53.388808  1746 raft_consensus.cc:2889] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:02:53.389051 10036 raft_consensus.cc:740] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
W20260709 13:02:53.389631  9812 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 22 current term: 22 is higher.
I20260709 13:02:53.389693  1746 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55840 (request call id 3334) took 1680 ms. Trace:
I20260709 13:02:53.390194  1746 rpcz_store.cc:276] 0709 13:02:51.709733 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:51.710055 (+   322us) service_pool.cc:224] Handling call
0709 13:02:51.710359 (+   304us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:52.617874 (+907515us) spinlock_profiling.cc:243] Waited 69.9 ms on lock 0x7b540010e8a0. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42660d9dc0 00007f42661641c0 00007f4266165231 00007f4266161d8f 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c
0709 13:02:53.389408 (+771534us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:53.389629 (+   221us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":160878848}
W20260709 13:02:53.390983  2298 leader_election.cc:336] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:02:53.391399  9851 consensus_queue.cc:1048] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 13:02:53.393483  9851 raft_consensus.cc:2749] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 13:02:53.371191  1754 raft_consensus.cc:1275] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 20 index: 17. Preceding OpId from leader: term: 21 index: 17. (term mismatch)
I20260709 13:02:53.394261  1754 pending_rounds.cc:85] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701: Aborting all ops after (but not including) 16
I20260709 13:02:53.394675  1754 pending_rounds.cc:107] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701: Aborting uncommitted NO_OP operation due to leader change: 20.17
I20260709 13:02:53.394766  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:53.395107  1754 raft_consensus.cc:2889] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:02:53.395475  1747 raft_consensus.cc:2468] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 17.
I20260709 13:02:53.385555  9709 leader_election.cc:290] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:53.367803  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:53.375473  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:53.398964  1678 raft_consensus.cc:2468] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 25.
I20260709 13:02:53.391463  2298 leader_election.cc:304] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:02:53.341988  1643 leader_election.cc:304] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 13:02:53.394629  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 81062us
I20260709 13:02:53.398162  1747 raft_consensus.cc:1275] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 19 index: 14. Preceding OpId from leader: term: 21 index: 15. (index mismatch)
I20260709 13:02:53.401221  1569 leader_election.cc:304] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
W20260709 13:02:53.401372  2298 leader_election.cc:336] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:02:53.334400 10083 leader_election.cc:290] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:02:53.403729  2298 leader_election.cc:336] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 13:02:53.404126  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 94191.1us
I20260709 13:02:53.363090  1752 raft_consensus.cc:2410] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for term 21 because replica has last-logged OpId of term: 20 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 11.
I20260709 13:02:53.382356  1750 raft_consensus.cc:2410] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 for term 24 because replica has last-logged OpId of term: 22 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 14.
I20260709 13:02:53.368520  1672 raft_consensus.cc:2468] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 20.
I20260709 13:02:53.402925  2299 leader_election.cc:304] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
W20260709 13:02:53.436408  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 97081.4us
W20260709 13:02:53.437496  1643 leader_election.cc:336] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:02:53.441972  1752 raft_consensus.cc:1275] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 23 index: 17. Preceding OpId from leader: term: 24 index: 17. (term mismatch)
I20260709 13:02:53.442480  1752 pending_rounds.cc:85] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701: Aborting all ops after (but not including) 16
I20260709 13:02:53.442768  1752 pending_rounds.cc:107] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701: Aborting uncommitted NO_OP operation due to leader change: 23.17
I20260709 13:02:53.443073  1752 raft_consensus.cc:2889] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:02:53.443125  1754 raft_consensus.cc:1275] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 22 index: 15. Preceding OpId from leader: term: 23 index: 15. (term mismatch)
I20260709 13:02:53.443687  1754 pending_rounds.cc:85] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701: Aborting all ops after (but not including) 14
I20260709 13:02:53.443924  1754 pending_rounds.cc:107] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701: Aborting uncommitted NO_OP operation due to leader change: 22.15
I20260709 13:02:53.444165  1754 raft_consensus.cc:2889] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:02:53.435294  2299 leader_election.cc:304] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:02:53.438073  1643 leader_election.cc:304] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
W20260709 13:02:53.452431  1643 leader_election.cc:336] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:02:53.457387  1673 raft_consensus.cc:2468] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 20.
I20260709 13:02:53.461170  2331 raft_consensus.cc:1240] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 19. Current term is 20. Ops: []
I20260709 13:02:53.463043  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
W20260709 13:02:53.463721  1643 leader_election.cc:336] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 18 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:02:53.463856  2334 raft_consensus.cc:2368] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 19. Current term is 20.
W20260709 13:02:53.468528  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 257528us
I20260709 13:02:53.470307  1643 leader_election.cc:304] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:53.479652  1749 raft_consensus.cc:1275] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 18 index: 11. (term mismatch)
I20260709 13:02:53.480182  1749 pending_rounds.cc:85] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701: Aborting all ops after (but not including) 10
I20260709 13:02:53.480563  1749 pending_rounds.cc:107] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701: Aborting uncommitted NO_OP operation due to leader change: 17.11
I20260709 13:02:53.481115  1749 raft_consensus.cc:2889] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 13:02:53.482672  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 120631us
I20260709 13:02:53.484560  1571 leader_election.cc:304] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:53.487601  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:53.491715  9703 raft_consensus.cc:2804] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:02:53.492202  9703 raft_consensus.cc:493] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:53.492606  9703 raft_consensus.cc:3060] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:02:53.447507 10010 raft_consensus.cc:2749] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:02:53.459410 10012 raft_consensus.cc:2749] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 13:02:53.447865 10056 raft_consensus.cc:493] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:53.495589 10056 raft_consensus.cc:515] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:53.496929  9709 raft_consensus.cc:2749] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
W20260709 13:02:53.508462  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 107408us
W20260709 13:02:53.508761  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 73634.8us
I20260709 13:02:53.509593  1569 leader_election.cc:304] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:02:53.487653  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:53.513543  2331 raft_consensus.cc:2368] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 23. Current term is 24.
I20260709 13:02:53.515645  1751 raft_consensus.cc:2393] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 22: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:02:53.448020 10072 raft_consensus.cc:493] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:53.510743  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:53.448654 10024 raft_consensus.cc:493] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:53.520462  1571 leader_election.cc:304] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:02:53.510898  2299 leader_election.cc:304] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
W20260709 13:02:53.524186  2298 leader_election.cc:336] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:02:53.517926  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 23 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:53.448494  9830 raft_consensus.cc:2804] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 24 FOLLOWER]: Leader election won for term 24
I20260709 13:02:53.526464  2328 raft_consensus.cc:2468] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 22.
I20260709 13:02:53.459093  9945 raft_consensus.cc:2804] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 13:02:53.528425  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 23 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:53.464053  9783 raft_consensus.cc:2749] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:02:53.529356  1569 leader_election.cc:304] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:02:53.530354  2330 raft_consensus.cc:1275] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 19 index: 16. Preceding OpId from leader: term: 21 index: 17. (index mismatch)
I20260709 13:02:53.521776  1571 leader_election.cc:304] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:53.531387  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
W20260709 13:02:53.532492  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 68852.3us
I20260709 13:02:53.532647  1749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55840 (request call id 3342) took 1140 ms. Trace:
I20260709 13:02:53.568354  2334 raft_consensus.cc:1240] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 19 LEADER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 18. Current term is 19. Ops: []
I20260709 13:02:53.568441  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:02:53.569859  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 59070.9us
I20260709 13:02:53.568297  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:53.570670  2335 raft_consensus.cc:1275] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 21 index: 14. Preceding OpId from leader: term: 22 index: 15. (index mismatch)
I20260709 13:02:53.573524  2336 raft_consensus.cc:1813] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 24 FOLLOWER]: Handling vote request from an unknown peer 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:53.575974  2336 raft_consensus.cc:2393] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 24: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:02:53.576365  2331 raft_consensus.cc:2393] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 23: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:02:53.577342  2333 raft_consensus.cc:2393] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 25: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:02:53.575635  1749 rpcz_store.cc:276] 0709 13:02:52.392559 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:52.404230 (+ 11671us) service_pool.cc:224] Handling call
0709 13:02:52.747307 (+343077us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:53.351846 (+604539us) spinlock_profiling.cc:243] Waited 73.9 ms on lock 0x7b5400132a20. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42660d7a1d 00007f42660be1a4 00007f4266140447 00007f4266158fe7 00007f426615a7de 00007f4266164b49 00007f4266165416 00007f4266161d8f 00007f426616183e 00007f426b126419
0709 13:02:53.378610 (+ 26764us) spinlock_profiling.cc:243] Waited 25.9 ms on lock 0x7b580081b6d0. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42660be0aa 00007f4266140447 00007f4266158fe7 00007f426615a7de 00007f4266164b49 00007f4266165416 00007f4266161d8f 00007f426616183e 00007f426b126419 00007f426469dbd4
0709 13:02:53.532261 (+153651us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:53.532577 (+   316us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":229571840}
I20260709 13:02:53.571162  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:53.586205  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 23 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:53.466832  9922 raft_consensus.cc:493] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:53.479288  9700 raft_consensus.cc:493] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
W20260709 13:02:53.587244  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 65570.5us
W20260709 13:02:53.588757  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 59501.3us
I20260709 13:02:53.389959 10036 consensus_queue.cc:260] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 24.17, Last appended by leader: 17, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:53.615180  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:53.616473  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 21 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:53.616798  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:53.613806  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
W20260709 13:02:53.620399  2298 leader_election.cc:336] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 19 election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:02:53.620757  1672 raft_consensus.cc:3060] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Advancing to term 24
I20260709 13:02:53.617604  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:53.618119  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 23 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:53.621937  1676 raft_consensus.cc:2468] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 21.
I20260709 13:02:53.610695  1571 leader_election.cc:304] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:53.627355  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:53.627367  1680 raft_consensus.cc:1275] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 18 index: 11. (index mismatch)
I20260709 13:02:53.620757  1678 raft_consensus.cc:2393] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 21: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:02:53.637640  1679 raft_consensus.cc:3060] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 24 FOLLOWER]: Advancing to term 25
I20260709 13:02:53.640398  1571 leader_election.cc:304] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:02:53.641414  1571 leader_election.cc:304] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:53.641623  2331 raft_consensus.cc:1275] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 24 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 21 index: 16. Preceding OpId from leader: term: 24 index: 17. (index mismatch)
I20260709 13:02:53.627319  1748 consensus_queue.cc:260] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 22.14, Last appended by leader: 14, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:53.527338 10048 raft_consensus.cc:2749] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 13:02:53.569730  9819 raft_consensus.cc:493] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:53.572511  9945 raft_consensus.cc:493] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:53.647015  1571 leader_election.cc:304] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:53.647974  1748 raft_consensus.cc:3060] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:02:53.648131  2333 raft_consensus.cc:1275] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 18 index: 15. Preceding OpId from leader: term: 20 index: 17. (index mismatch)
I20260709 13:02:53.603976  9937 raft_consensus.cc:493] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:53.637454  1672 raft_consensus.cc:2468] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 24.
I20260709 13:02:53.527562  9821 raft_consensus.cc:493] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:53.608166  9666 raft_consensus.cc:2804] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Leader election won for term 20
W20260709 13:02:53.656369  2299 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:53.657142  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:53.606006  9955 raft_consensus.cc:493] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:53.660415  1569 leader_election.cc:304] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:02:53.661087  2333 raft_consensus.cc:1275] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 21 index: 14. Preceding OpId from leader: term: 23 index: 15. (index mismatch)
I20260709 13:02:53.661146 10066 raft_consensus.cc:493] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:53.662783  1571 leader_election.cc:304] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:02:53.662937 10082 consensus_queue.cc:1048] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 13:02:53.663434  9840 raft_consensus.cc:2804] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:02:53.695984  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 25 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:53.586867 10071 raft_consensus.cc:2749] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:02:53.527732 10005 raft_consensus.cc:2749] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 13:02:53.518610  9812 consensus_queue.cc:1059] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.109s
I20260709 13:02:53.674641  9945 raft_consensus.cc:3060] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 25 FOLLOWER]: Advancing to term 26
I20260709 13:02:53.608090 10073 raft_consensus.cc:2804] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 13:02:53.568985 10072 raft_consensus.cc:515] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:53.684667  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:53.533838 10006 consensus_queue.cc:1048] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.168s
I20260709 13:02:53.639569  9709 raft_consensus.cc:493] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:53.656515 10023 consensus_queue.cc:1048] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.055s
I20260709 13:02:53.638357  9830 raft_consensus.cc:697] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 24 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:02:53.526232  9853 consensus_queue.cc:1048] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.129s
I20260709 13:02:53.572109 10052 consensus_queue.cc:1048] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.105s
I20260709 13:02:53.699044 10066 raft_consensus.cc:515] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:53.723770  9920 consensus_queue.cc:1048] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.010s
I20260709 13:02:53.735591 10066 leader_election.cc:290] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:53.735301  9830 consensus_queue.cc:237] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.15, Last appended by leader: 15, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:53.698249  9955 raft_consensus.cc:515] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:53.715626 10018 raft_consensus.cc:2749] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:02:53.739478  9955 leader_election.cc:290] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:53.741257  1571 leader_election.cc:304] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:02:53.741439  9598 consensus_queue.cc:1048] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 13:02:53.651786 10020 raft_consensus.cc:3060] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 23 FOLLOWER]: Advancing to term 24
I20260709 13:02:53.742146  9598 consensus_queue.cc:1243] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer af520b67945d443b901bbeca37cc5701 log is divergent from this leader: its last log entry 15.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 9
I20260709 13:02:53.745502  9598 consensus_queue.cc:1059] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 13:02:53.646550  9998 raft_consensus.cc:2804] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 13:02:53.746372  9598 raft_consensus.cc:3055] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 LEADER]: Stepping down as leader of term 19
I20260709 13:02:53.746712  9998 raft_consensus.cc:493] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:53.746762  9598 raft_consensus.cc:740] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:02:53.747241  9998 raft_consensus.cc:3060] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:02:53.747871  9598 consensus_queue.cc:260] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.15, Last appended by leader: 15, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:53.749152  9598 raft_consensus.cc:3060] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:02:53.656380  9979 raft_consensus.cc:2804] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 13:02:53.574530 10024 raft_consensus.cc:515] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:53.750525  9979 raft_consensus.cc:493] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 24 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:53.751056  9979 raft_consensus.cc:3060] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 24 FOLLOWER]: Advancing to term 25
I20260709 13:02:53.751441  9709 raft_consensus.cc:515] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:53.752292 10024 leader_election.cc:290] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:53.704612  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:53.753710  9709 leader_election.cc:290] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:53.668968  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:53.683538  9821 raft_consensus.cc:515] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:53.674692  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 68117.5us
I20260709 13:02:53.708541  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:53.758524  9821 leader_election.cc:290] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:53.618892  9700 raft_consensus.cc:515] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:53.760948  9700 leader_election.cc:290] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:53.627508  9873 consensus_queue.cc:1048] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.066s
I20260709 13:02:53.716411  1747 raft_consensus.cc:1275] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 22 index: 15. Preceding OpId from leader: term: 23 index: 15. (term mismatch)
I20260709 13:02:53.762643  1747 pending_rounds.cc:85] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701: Aborting all ops after (but not including) 14
I20260709 13:02:53.762918  1747 pending_rounds.cc:107] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701: Aborting uncommitted NO_OP operation due to leader change: 22.15
I20260709 13:02:53.763170  1747 raft_consensus.cc:2889] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:02:53.682492  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:53.637419  9922 raft_consensus.cc:515] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:53.607483 10070 raft_consensus.cc:2804] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 13:02:53.720824  1672 raft_consensus.cc:3060] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:02:53.764923 10070 raft_consensus.cc:493] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:53.765372 10070 raft_consensus.cc:3060] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 25 FOLLOWER]: Advancing to term 26
I20260709 13:02:53.765803  9922 leader_election.cc:290] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:53.695583 10094 consensus_queue.cc:1048] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.021s
I20260709 13:02:53.766571  9785 consensus_queue.cc:1048] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20260709 13:02:53.703500 10048 raft_consensus.cc:2804] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 24 FOLLOWER]: Leader election won for term 24
I20260709 13:02:53.607196  9934 consensus_queue.cc:1048] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.086s
I20260709 13:02:53.767755 10048 raft_consensus.cc:697] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 24 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:02:53.696559  9840 raft_consensus.cc:493] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:53.768903  9840 raft_consensus.cc:3060] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:02:53.768702 10048 consensus_queue.cc:237] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 22.12, Last appended by leader: 12, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:53.702934  9666 raft_consensus.cc:697] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [term 20 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:02:53.772544  9666 consensus_queue.cc:237] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:53.773017  1672 raft_consensus.cc:2468] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 18.
I20260709 13:02:53.760715 10063 raft_consensus.cc:2804] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 13:02:53.775328 10063 raft_consensus.cc:697] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 20 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:02:53.775954 10072 leader_election.cc:290] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:53.776078  1673 raft_consensus.cc:1275] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 18 index: 14. Preceding OpId from leader: term: 19 index: 15. (index mismatch)
I20260709 13:02:53.697077 10036 raft_consensus.cc:3060] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 24 FOLLOWER]: Advancing to term 25
I20260709 13:02:53.776602 10044 consensus_queue.cc:1048] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20260709 13:02:53.743258  9594 raft_consensus.cc:3055] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 25 LEADER]: Stepping down as leader of term 25
I20260709 13:02:53.777339  9594 raft_consensus.cc:740] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:02:53.778275  9594 consensus_queue.cc:260] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 25.13, Last appended by leader: 13, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:53.743366  9941 raft_consensus.cc:2749] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 13:02:53.744256  1569 leader_election.cc:400] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 19 election: Vote denied by peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625) with higher term. Message: Invalid argument: T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 19. Current term is 20.
I20260709 13:02:53.786345  1569 leader_election.cc:403] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 19 election: Cancelling election due to peer responding with higher term
I20260709 13:02:53.760705  9979 raft_consensus.cc:515] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:53.675014  9819 raft_consensus.cc:515] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:53.678016  9937 raft_consensus.cc:515] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:53.787081  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 390156us
I20260709 13:02:53.742660 10005 consensus_queue.cc:1048] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 13:02:53.690361  2301 leader_election.cc:304] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:02:53.775568  1674 raft_consensus.cc:2393] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 23: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:02:53.781509  1748 raft_consensus.cc:1275] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 22 index: 14. Preceding OpId from leader: term: 23 index: 15. (index mismatch)
I20260709 13:02:53.781811  1679 raft_consensus.cc:2410] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for term 25 because replica has last-logged OpId of term: 24 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 12.
I20260709 13:02:53.812150  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:53.718359 10074 raft_consensus.cc:493] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:53.813928  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:53.814393  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:53.814658  1676 raft_consensus.cc:2468] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 21.
I20260709 13:02:53.815668  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:02:53.816449  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 126296us
I20260709 13:02:53.817337  2301 leader_election.cc:304] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:02:53.784621  1678 raft_consensus.cc:2468] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 22.
W20260709 13:02:53.788885  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 44819.6us
I20260709 13:02:53.789947  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:53.794399  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 22 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:53.796376  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 22 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:53.795038  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 26 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:53.802963  2331 raft_consensus.cc:1240] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 24 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 23. Current term is 24. Ops: []
I20260709 13:02:53.807644  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:53.719069 10073 raft_consensus.cc:493] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:53.846118  2333 raft_consensus.cc:1275] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 19 index: 14. Preceding OpId from leader: term: 23 index: 16. (index mismatch)
I20260709 13:02:53.819243  1748 pending_rounds.cc:85] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701: Aborting all ops after (but not including) 9
I20260709 13:02:53.849939  1748 pending_rounds.cc:107] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701: Aborting uncommitted NO_OP operation due to leader change: 15.10
I20260709 13:02:53.850736  2334 raft_consensus.cc:2468] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 21.
I20260709 13:02:53.851522  1569 leader_election.cc:400] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 23 pre-election: Vote denied by peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625) with higher term. Message: Invalid argument: T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 23. Current term is 24.
I20260709 13:02:53.731455 10056 leader_election.cc:290] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:53.779580  9594 raft_consensus.cc:3060] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 25 FOLLOWER]: Advancing to term 26
I20260709 13:02:53.851950  1569 leader_election.cc:403] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 23 pre-election: Cancelling election due to peer responding with higher term
W20260709 13:02:53.926959  1570 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:53.851382  1678 raft_consensus.cc:1240] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 19. Current term is 20. Ops: [19.15-19.15]
I20260709 13:02:53.851403  1748 raft_consensus.cc:2889] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 13:02:53.913260  1717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:53.878971  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 61745.7us
I20260709 13:02:53.886246  1672 raft_consensus.cc:1240] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 25 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 24. Current term is 25. Ops: []
I20260709 13:02:53.852438  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:53.816520  1718 leader_election.cc:304] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 13:02:53.875910  1753 raft_consensus.cc:2468] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 24.
I20260709 13:02:53.997588  2301 leader_election.cc:304] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:02:53.804075 10072 raft_consensus.cc:493] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:53.787598 10058 raft_consensus.cc:3055] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 LEADER]: Stepping down as leader of term 18
W20260709 13:02:54.017968  1643 leader_election.cc:336] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 21 election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 13:02:54.018050  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 201721us
W20260709 13:02:54.023912  1643 leader_election.cc:336] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:02:54.024644  1643 leader_election.cc:304] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 13:02:54.034451  1643 leader_election.cc:336] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:02:54.021512  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 25 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:53.818539 10005 consensus_queue.cc:1243] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer 3b879799899a499c8d409ee9f1277cd1 log is divergent from this leader: its last log entry 16.12 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 11
I20260709 13:02:53.884234 10035 raft_consensus.cc:2804] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 13:02:53.856832 10044 raft_consensus.cc:493] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:54.005213 10071 raft_consensus.cc:493] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:54.035833 10024 raft_consensus.cc:2749] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:02:54.017477 10047 raft_consensus.cc:2804] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 13:02:54.017067 10020 raft_consensus.cc:2749] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 13:02:53.877184 10073 raft_consensus.cc:3060] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:02:54.038321 10093 raft_consensus.cc:2804] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 13:02:54.040606  2331 raft_consensus.cc:2468] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 24.
I20260709 13:02:54.048398 10093 raft_consensus.cc:493] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:53.912171 10060 raft_consensus.cc:493] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:54.044155 10044 raft_consensus.cc:515] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:54.043048 10035 raft_consensus.cc:493] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:54.044508 10071 raft_consensus.cc:515] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:54.049785 10035 raft_consensus.cc:3060] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Advancing to term 24
I20260709 13:02:53.776345 10063 consensus_queue.cc:237] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 18.15, Last appended by leader: 15, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:54.043159  9920 raft_consensus.cc:493] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:54.051227 10060 raft_consensus.cc:515] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:54.051561  9920 raft_consensus.cc:515] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:54.047560  9998 raft_consensus.cc:515] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:54.053771  9920 leader_election.cc:290] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:02:54.041265  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 81277.2us
I20260709 13:02:53.873082 10070 raft_consensus.cc:515] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:54.055310  9869 raft_consensus.cc:493] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:54.055883  9998 leader_election.cc:290] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 22 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:53.866081  9703 raft_consensus.cc:515] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:54.056622 10070 leader_election.cc:290] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 26 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:54.055915  9869 raft_consensus.cc:515] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:54.050926 10044 leader_election.cc:290] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:02:54.059366  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 208068us
I20260709 13:02:53.809504 10009 raft_consensus.cc:3060] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:02:54.058120  9703 leader_election.cc:290] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 20 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:53.881879  9821 raft_consensus.cc:2749] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 13:02:53.863979  9979 leader_election.cc:290] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 25 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:53.809072  9812 consensus_queue.cc:1048] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 10, Time since last communication: 0.229s
I20260709 13:02:53.817816  9851 consensus_queue.cc:1048] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.230s
W20260709 13:02:54.066617  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 364462us
I20260709 13:02:53.865422  9819 leader_election.cc:290] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:53.875669  9937 leader_election.cc:290] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:54.040549  1643 leader_election.cc:304] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:02:54.019941 10058 raft_consensus.cc:740] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:02:54.048969 10093 raft_consensus.cc:3060] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:02:54.062953  2301 leader_election.cc:304] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
W20260709 13:02:54.227460  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 167363us
I20260709 13:02:54.228652  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:54.228837  1569 leader_election.cc:304] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:02:54.175741  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 22 candidate_status { last_received { term: 19 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:02:54.227366  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 168390us
W20260709 13:02:54.233111  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 170505us
I20260709 13:02:54.187371 10071 leader_election.cc:290] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:54.206496  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 26 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:54.237722  2301 leader_election.cc:304] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
W20260709 13:02:54.167171  2299 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:54.053697 10060 leader_election.cc:290] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:54.045450 10047 raft_consensus.cc:493] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:54.245977  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:54.246279  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:54.079361  9998 raft_consensus.cc:3060] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Advancing to term 24
I20260709 13:02:54.247783  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:54.242069  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:02:54.249760  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 215422us
I20260709 13:02:54.250646  1673 raft_consensus.cc:1275] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 25 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 21 index: 16. Preceding OpId from leader: term: 25 index: 18. (index mismatch)
I20260709 13:02:54.250806  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:53.883721 10074 raft_consensus.cc:515] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:54.262554  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:54.264932  1569 leader_election.cc:304] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
W20260709 13:02:54.265337  1754 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.509s	user 0.028s	sys 0.020s
W20260709 13:02:54.266395  1754 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.510s	user 0.028s	sys 0.020s
I20260709 13:02:54.269598  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 22 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:54.271847  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 25 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:54.276845  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:54.277168  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:54.279592  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:54.280491  1753 raft_consensus.cc:3060] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Advancing to term 22
W20260709 13:02:54.283597   431 thread.cc:527] Waited for 1000ms trying to join with heartbeat (tid 1630)
I20260709 13:02:54.272426  2336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:54.303648  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:54.305879  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 23 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:54.032871 10072 raft_consensus.cc:515] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:54.078100 10056 consensus_queue.cc:1048] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.174s
I20260709 13:02:54.242874  9869 leader_election.cc:290] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:54.322189  1745 raft_consensus.cc:2410] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 for term 23 because replica has last-logged OpId of term: 19 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 12.
I20260709 13:02:54.305897  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:54.243944  9821 raft_consensus.cc:2749] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 13:02:54.248978  9598 raft_consensus.cc:493] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:54.332073  1718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:54.369511  1718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:54.237051 10070 consensus_queue.cc:1048] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.089s
I20260709 13:02:54.306316  9920 raft_consensus.cc:493] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:54.252308  9588 raft_consensus.cc:2804] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 13:02:54.250674 10047 raft_consensus.cc:3060] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Advancing to term 24
I20260709 13:02:54.360209  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 23 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:54.363451  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 25 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:54.401862  1751 raft_consensus.cc:2468] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 24.
I20260709 13:02:54.307688  1569 leader_election.cc:304] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:02:54.248261  9703 consensus_queue.cc:1048] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.116s
I20260709 13:02:54.248692 10046 consensus_queue.cc:1048] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.111s
I20260709 13:02:54.251482  9979 raft_consensus.cc:2804] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 13:02:54.385464  1718 leader_election.cc:304] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:54.405851  1753 raft_consensus.cc:2468] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 22.
W20260709 13:02:54.356124  9424 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.960s	user 0.016s	sys 0.062s
W20260709 13:02:54.411978  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 104402us
I20260709 13:02:54.357229 10044 raft_consensus.cc:2749] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 13:02:54.424737  1718 leader_election.cc:304] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
W20260709 13:02:54.387777 10048 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.506s	user 0.015s	sys 0.004s
W20260709 13:02:54.427987  9424 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.032s	user 0.016s	sys 0.063s
I20260709 13:02:54.429051 10020 raft_consensus.cc:2749] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:02:54.429520  9979 raft_consensus.cc:493] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:54.572172  9979 raft_consensus.cc:3060] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:02:54.553944 10052 consensus_queue.cc:1048] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 13:02:54.413272  9588 raft_consensus.cc:493] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:54.577467  9588 raft_consensus.cc:3060] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 24 FOLLOWER]: Advancing to term 25
I20260709 13:02:54.420961  9920 raft_consensus.cc:515] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:54.402041  9869 consensus_queue.cc:1048] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 13, Time since last communication: 0.179s
I20260709 13:02:54.426424  1718 leader_election.cc:304] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:54.388015  9819 consensus_queue.cc:1048] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.128s
W20260709 13:02:54.515157  1494 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35444 (request call id 264) took 37027 ms (client timeout 15000 ms). Trace:
W20260709 13:02:54.519970 10103 log.cc:927] Time spent T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.161s	user 0.003s	sys 0.001s
I20260709 13:02:54.399094  9853 raft_consensus.cc:2749] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:02:54.356755  9785 consensus_queue.cc:1048] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.062s
W20260709 13:02:54.545822 10102 log.cc:927] Time spent T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.239s	user 0.000s	sys 0.000s
I20260709 13:02:54.289317 10054 consensus_queue.cc:1048] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.130s
W20260709 13:02:54.616340  1494 rpcz_store.cc:269] 0709 13:02:17.487296 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:31.597491 (+14110195us) service_pool.cc:224] Handling call
0709 13:02:54.514942 (+22917451us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 13:02:46.937042 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 13:02:46.991211 (+ 54169us) write_op.cc:432] Acquiring schema lock in shared mode
0709 13:02:46.991279 (+    68us) write_op.cc:435] Acquired schema lock
0709 13:02:46.991301 (+    22us) tablet.cc:662] Decoding operations
0709 13:02:47.042932 (+ 51631us) write_op.cc:620] Acquiring the partition lock for write op
0709 13:02:47.043034 (+   102us) write_op.cc:641] Partition lock acquired for write op
0709 13:02:47.043062 (+    28us) tablet.cc:685] Acquiring locks for 27 operations
0709 13:02:47.114000 (+ 70938us) tablet.cc:701] Row locks acquired
0709 13:02:47.114083 (+    83us) write_op.cc:260] PREPARE: finished
0709 13:02:47.114361 (+   278us) write_op.cc:270] Start()
0709 13:02:47.187150 (+ 72789us) write_op.cc:276] Timestamp: P: 1783602167114277 usec, L: 0
0709 13:02:47.192194 (+  5044us) op_driver.cc:348] REPLICATION: starting
0709 13:02:47.345345 (+153151us) log.cc:844] Serialized 10557 byte log entry
0709 13:02:48.789533 (+1444188us) op_driver.cc:464] REPLICATION: finished
0709 13:02:49.594701 (+805168us) write_op.cc:301] APPLY: starting
0709 13:02:49.614062 (+ 19361us) tablet.cc:1367] starting BulkCheckPresence
0709 13:02:50.529450 (+915388us) tablet.cc:1370] finished BulkCheckPresence
0709 13:02:50.529678 (+   228us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 13:02:53.139208 (+2609530us) spinlock_profiling.cc:243] Waited 11.5 ms on lock 0x7b5c000c668c. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426c354331 00007f426c354221 00007f426c34ef6c 00007f4269aaf30b 00007f4269aaefd3 00007f4269bade34 00007f4269b9997c 00007f4269b9dd61 00007f4269b9dd19 00007f4269b9dca9 00007f4269b9dc71
0709 13:02:53.174363 (+ 35155us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 13:02:53.258906 (+ 84543us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=54,key_file_lookups=54,delta_file_lookups=54,mrs_lookups=0
0709 13:02:53.258986 (+    80us) write_op.cc:312] APPLY: finished
0709 13:02:53.259664 (+   678us) log.cc:844] Serialized 236 byte log entry
0709 13:02:53.701428 (+441764us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:02:53.724970 (+ 23542us) write_op.cc:454] Released schema lock
0709 13:02:53.734522 (+  9552us) write_op.cc:341] FINISH: Updating metrics
0709 13:02:53.760244 (+ 25722us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:02:53.760286 (+    42us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":7732101,"reported_tablets":48,"spinlock_wait_cycles":71936,"child_traces":[["op",{"apply.queue_time_us":744021,"apply.run_cpu_time_us":191582,"apply.run_wall_time_us":4186425,"cfile_cache_hit":110,"cfile_cache_hit_bytes":117203,"num_ops":27,"prepare.queue_time_us":385572,"prepare.run_cpu_time_us":34594,"prepare.run_wall_time_us":683398,"raft.queue_time_us":606445,"raft.run_cpu_time_us":15025,"raft.run_wall_time_us":626800,"replication_time_us":1465191,"spinlock_wait_cycles":26490240,"thread_start_us":1337528,"threads_started":5,"wal-append.queue_time_us":834541,"wal-append.run_cpu_time_us":22904,"wal-append.run_wall_time_us":1730573}]]}
I20260709 13:02:54.546923  1571 leader_election.cc:304] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
W20260709 13:02:54.630879  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 84048.7us
I20260709 13:02:54.631721  1746 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55840 (request call id 3362) took 1297 ms. Trace:
I20260709 13:02:54.373469 10058 consensus_queue.cc:260] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 18.16, Last appended by leader: 16, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:54.379992  9598 raft_consensus.cc:515] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:54.632949 10023 consensus_queue.cc:1059] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20260709 13:02:54.633769 10023 consensus_queue.cc:1243] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Peer 3b879799899a499c8d409ee9f1277cd1 log is divergent from this leader: its last log entry 19.16 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 15
I20260709 13:02:54.554738 10083 consensus_queue.cc:1059] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20260709 13:02:54.634658  9598 leader_election.cc:290] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:54.635416 10083 raft_consensus.cc:3055] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 23 LEADER]: Stepping down as leader of term 23
I20260709 13:02:54.635974 10083 raft_consensus.cc:740] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:02:54.636989 10083 consensus_queue.cc:260] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 23.17, Last appended by leader: 17, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:54.632308  1746 rpcz_store.cc:276] 0709 13:02:53.333947 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:53.393280 (+ 59333us) service_pool.cc:224] Handling call
0709 13:02:53.393583 (+   303us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:53.394049 (+   466us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:53.394081 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:53.396379 (+  2298us) log.cc:844] Serialized 26 byte log entry
0709 13:02:53.991778 (+595399us) raft_consensus.cc:1645] Marking committed up to 14
0709 13:02:53.991852 (+    74us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:53.991973 (+   121us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:54.519158 (+527185us) spinlock_profiling.cc:243] Waited 4.28 ms on lock 0x7b4400022b4c. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f4262f9bc85 00007f4262f9c6a2 00007f4265d0e1db 00007f4265d09aaf 00007f4265cfd86e 00007f4265cfcbb6 00007f4265cfc7d8 00007f4265d06221 00007f4265d061d9 00007f4265d06169
0709 13:02:54.631317 (+112159us) raft_consensus.cc:1685] finished
0709 13:02:54.631352 (+    35us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:54.631637 (+   285us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":155,"raft.run_cpu_time_us":106,"raft.run_wall_time_us":107,"spinlock_wait_cycles":9853184,"thread_start_us":472449,"threads_started":1,"wal-append.queue_time_us":761331}
I20260709 13:02:54.643285  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 27 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:54.635473  9598 raft_consensus.cc:2804] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Leader election won for term 22
W20260709 13:02:54.611430  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 185123us
I20260709 13:02:54.646492  1718 leader_election.cc:304] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
W20260709 13:02:54.566632 10048 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.691s	user 0.016s	sys 0.004s
I20260709 13:02:54.639976  9869 raft_consensus.cc:2749] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 13:02:54.638320 10083 raft_consensus.cc:3060] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Advancing to term 24
W20260709 13:02:54.671708 10035 consensus_meta.cc:287] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.621s	user 0.010s	sys 0.005s
I20260709 13:02:54.651156  9424 raft_consensus.cc:2749] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
W20260709 13:02:54.681020  1717 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.646s	user 0.038s	sys 0.040s
W20260709 13:02:54.697381  1717 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.662s	user 0.038s	sys 0.040s
W20260709 13:02:54.698710  1752 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.609s	user 0.032s	sys 0.039s
I20260709 13:02:54.672473 10035 raft_consensus.cc:515] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:54.700907  1752 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.612s	user 0.032s	sys 0.039s
I20260709 13:02:54.659265  9598 raft_consensus.cc:697] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
W20260709 13:02:54.677278 10063 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.624s	user 0.006s	sys 0.038s
W20260709 13:02:54.675561  9814 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.807s	user 0.026s	sys 0.054s
W20260709 13:02:54.705060  9875 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.608s	user 0.018s	sys 0.059s
W20260709 13:02:54.683946 10036 consensus_meta.cc:287] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.907s	user 0.009s	sys 0.012s
W20260709 13:02:54.715389  2298 consensus_peers.cc:597] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:02:54.719606  2298 leader_election.cc:336] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 20 election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 13:02:54.723407  2298 leader_election.cc:336] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 24 election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:02:54.660742 10094 consensus_queue.cc:1059] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 24.12, Next index: 13, Last known committed idx: 9, Time since last communication: 0.045s
I20260709 13:02:54.723979  2298 leader_election.cc:304] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
W20260709 13:02:54.705494  9830 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.568s	user 0.039s	sys 0.040s
W20260709 13:02:54.729609  2298 leader_election.cc:336] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 20 election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:02:54.730855  2335 raft_consensus.cc:1275] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 25 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 24 index: 17. Preceding OpId from leader: term: 25 index: 18. (index mismatch)
W20260709 13:02:54.724692  9875 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.628s	user 0.018s	sys 0.059s
I20260709 13:02:54.705698  9931 raft_consensus.cc:2804] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Leader pre-election won for term 25
W20260709 13:02:54.717691 10063 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.664s	user 0.006s	sys 0.039s
W20260709 13:02:54.718546  9814 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.850s	user 0.027s	sys 0.054s
W20260709 13:02:54.735357  2298 leader_election.cc:336] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:02:54.813041  2298 leader_election.cc:304] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
W20260709 13:02:54.814107  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 78811.8us
W20260709 13:02:54.731365  1570 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:54.814965  2298 leader_election.cc:336] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 13:02:54.732118  1717 consensus_peers.cc:597] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:02:54.815845  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 87565.9us
W20260709 13:02:54.816077  2298 leader_election.cc:336] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 18 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:02:54.816527  2298 leader_election.cc:304] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 13:02:54.801751  1748 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.722s	user 0.020s	sys 0.050s
W20260709 13:02:54.820806  1748 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.741s	user 0.020s	sys 0.050s
I20260709 13:02:54.765246  1677 raft_consensus.cc:2468] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 25.
W20260709 13:02:54.825150  2298 leader_election.cc:336] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 13:02:54.804737  1646 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.524s	user 0.000s	sys 0.010s
W20260709 13:02:54.811787  2299 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.522s	user 0.012s	sys 0.054s
I20260709 13:02:54.750262  2335 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49082 (request call id 1452) took 1183 ms. Trace:
W20260709 13:02:54.827601  2299 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.537s	user 0.012s	sys 0.054s
W20260709 13:02:54.826887  1646 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.546s	user 0.000s	sys 0.011s
W20260709 13:02:54.829421  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 598794us
I20260709 13:02:54.827468  2335 rpcz_store.cc:276] 0709 13:02:53.567071 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:53.575430 (+  8359us) service_pool.cc:224] Handling call
0709 13:02:53.621219 (+ 45789us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:54.735574 (+1114355us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:54.749921 (+ 14347us) spinlock_profiling.cc:243] Waited 1.11 s on lock 0x7b54004f88d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:02:54.750165 (+   244us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2551563648}
W20260709 13:02:54.834964  1718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:54.828748  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 538948us
I20260709 13:02:54.838997  1674 raft_consensus.cc:1240] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 17. Current term is 18. Ops: []
W20260709 13:02:54.848112 10112 log.cc:927] Time spent T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.081s	user 0.000s	sys 0.000s
I20260709 13:02:54.847927  9666 raft_consensus.cc:493] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:54.734501  9830 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.598s	user 0.039s	sys 0.040s
W20260709 13:02:54.765409  9840 consensus_meta.cc:287] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.996s	user 0.015s	sys 0.000s
I20260709 13:02:54.752398 10046 raft_consensus.cc:2749] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 13:02:54.853806  1754 raft_consensus.cc:1275] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 19 index: 10. Preceding OpId from leader: term: 21 index: 11. (index mismatch)
I20260709 13:02:54.760641 10072 leader_election.cc:290] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:54.871397  1680 raft_consensus.cc:1275] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 16 index: 14. Preceding OpId from leader: term: 18 index: 15. (index mismatch)
I20260709 13:02:54.760352 10074 leader_election.cc:290] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:54.833140 10063 raft_consensus.cc:2749] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:02:54.829100 10036 raft_consensus.cc:2749] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
W20260709 13:02:54.880484  2298 consensus_peers.cc:597] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:02:54.751852  9920 leader_election.cc:290] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:02:54.856381  9945 consensus_meta.cc:287] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 1.135s	user 0.009s	sys 0.007s
W20260709 13:02:54.920928  1673 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.556s	user 0.013s	sys 0.048s
W20260709 13:02:54.922598  1673 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.558s	user 0.013s	sys 0.048s
I20260709 13:02:54.923141  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:54.925084  1754 raft_consensus.cc:2468] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 25.
W20260709 13:02:54.885363 10111 log.cc:927] Time spent T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.079s	user 0.001s	sys 0.000s
I20260709 13:02:54.938196  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
W20260709 13:02:54.951217  2301 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.663s	user 0.048s	sys 0.043s
W20260709 13:02:54.952716  2301 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.665s	user 0.048s	sys 0.043s
I20260709 13:02:54.911566  9830 raft_consensus.cc:493] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:54.801553  9598 consensus_queue.cc:237] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.15, Last appended by leader: 15, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:54.960541 10104 consensus_queue.cc:1048] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.000s
W20260709 13:02:54.975804  1569 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.548s	user 0.001s	sys 0.041s
I20260709 13:02:54.976541  1675 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52778 (request call id 2994) took 1183 ms. Trace:
I20260709 13:02:54.976997  1675 rpcz_store.cc:276] 0709 13:02:53.793108 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:53.923719 (+130611us) service_pool.cc:224] Handling call
0709 13:02:53.924215 (+   496us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:53.935643 (+ 11428us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:53.935690 (+    47us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:53.936381 (+   691us) log.cc:844] Serialized 26 byte log entry
0709 13:02:54.115907 (+179526us) spinlock_profiling.cc:243] Waited 904 us on lock 0x7b48002ac558. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426b08baa1 00007f426b082b37 00007f426b0829ca 00007f426b17a26f 00007f426b179f37 00007f426b18e55c 00007f426b18e508 00007f426b18e488 00007f426b18e443 00007f426b18d6ec 00007f426a0b7a96
0709 13:02:54.391358 (+275451us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:02:54.391430 (+    72us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:54.391547 (+   117us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:54.969515 (+577968us) raft_consensus.cc:1685] finished
0709 13:02:54.969542 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:54.976458 (+  6916us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":10836,"raft.queue_time_us":12859,"raft.run_cpu_time_us":2514,"raft.run_wall_time_us":178558,"spinlock_wait_cycles":2079616,"thread_start_us":454546,"threads_started":1,"wal-append.queue_time_us":795404}
W20260709 13:02:54.980077  1569 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.555s	user 0.001s	sys 0.042s
I20260709 13:02:54.876649  9840 raft_consensus.cc:515] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:02:54.941514  9998 consensus_meta.cc:287] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.674s	user 0.010s	sys 0.003s
I20260709 13:02:54.969071  1646 leader_election.cc:304] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:54.986027  1677 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52778 (request call id 2989) took 1283 ms. Trace:
I20260709 13:02:54.995344  1677 rpcz_store.cc:276] 0709 13:02:53.702701 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:53.814103 (+111402us) service_pool.cc:224] Handling call
0709 13:02:54.985769 (+1171666us) spinlock_profiling.cc:243] Waited 565 ms on lock 0x7b54000596d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:02:54.985938 (+   169us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1299166976}
I20260709 13:02:54.918203 10011 consensus_queue.cc:1059] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 17.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.053s
I20260709 13:02:54.877411  9666 raft_consensus.cc:515] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:54.950744 10094 raft_consensus.cc:493] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:54.957629  9869 raft_consensus.cc:493] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:54.938746  9821 consensus_queue.cc:1048] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.049s
I20260709 13:02:54.965312 10035 leader_election.cc:290] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 24 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:54.941802  9945 raft_consensus.cc:515] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:55.012312 10109 log.cc:927] Time spent T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.061s	user 0.000s	sys 0.000s
I20260709 13:02:55.016036 10124 raft_consensus.cc:3055] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 13:02:55.026671  9931 raft_consensus.cc:493] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:55.027225  9931 raft_consensus.cc:3060] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Advancing to term 25
I20260709 13:02:55.027841  9840 leader_election.cc:290] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 18 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:55.029834  9945 leader_election.cc:290] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 26 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:55.026352  9979 raft_consensus.cc:515] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:55.002300  9998 raft_consensus.cc:2749] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 24 FOLLOWER]: Leader pre-election lost for term 23. Reason: Vote denied by peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625) with higher term. Message: Invalid argument: T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 23. Current term is 24.
W20260709 13:02:55.017261 10129 log.cc:927] Time spent T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.159s	user 0.000s	sys 0.001s
I20260709 13:02:55.021955 10124 raft_consensus.cc:740] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:02:55.006966 10094 raft_consensus.cc:515] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:55.048801 10058 raft_consensus.cc:3060] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:02:55.056100 10124 consensus_queue.cc:260] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:55.080969 10124 raft_consensus.cc:3060] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Advancing to term 18
W20260709 13:02:55.055240 10116 log.cc:927] Time spent T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.053s	user 0.000s	sys 0.000s
W20260709 13:02:55.085458 10073 consensus_meta.cc:287] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 1.038s	user 0.017s	sys 0.000s
I20260709 13:02:55.086210 10073 raft_consensus.cc:515] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:55.058288 10094 leader_election.cc:290] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 13:02:55.065639 10093 consensus_meta.cc:287] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.817s	user 0.003s	sys 0.008s
I20260709 13:02:55.091282 10093 raft_consensus.cc:515] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:55.109490 10093 leader_election.cc:290] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 22 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:55.089514 10073 leader_election.cc:290] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 22 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:02:55.170042  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 748322us
I20260709 13:02:55.173120  1569 leader_election.cc:304] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
W20260709 13:02:55.182044  1643 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.901s	user 0.010s	sys 0.031s
W20260709 13:02:55.182713  1643 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.902s	user 0.010s	sys 0.031s
I20260709 13:02:55.187040  1676 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52766 (request call id 3450) took 1564 ms. Trace:
W20260709 13:02:55.187198  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 218161us
I20260709 13:02:55.194718  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:55.187721  1676 rpcz_store.cc:276] 0709 13:02:53.622582 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:53.816603 (+194021us) service_pool.cc:224] Handling call
0709 13:02:53.816999 (+   396us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 13:02:53.925789 (+108790us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 13:02:54.313189 (+387400us) log.cc:844] Serialized 14 byte log entry
0709 13:02:54.856205 (+543016us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 13:02:54.880995 (+ 24790us) log.cc:844] Serialized 78 byte log entry
0709 13:02:54.881211 (+   216us) raft_consensus.cc:1645] Marking committed up to 13
0709 13:02:54.881548 (+   337us) log.cc:844] Serialized 14 byte log entry
0709 13:02:54.905015 (+ 23467us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:54.905143 (+   128us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:55.186052 (+280909us) raft_consensus.cc:1685] finished
0709 13:02:55.186130 (+    78us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:55.186697 (+   567us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":37757,"prepare.queue_time_us":706148,"prepare.run_cpu_time_us":117,"prepare.run_wall_time_us":9096,"raft.queue_time_us":136421,"raft.run_cpu_time_us":576,"raft.run_wall_time_us":48203,"spinlock_wait_cycles":4352,"thread_start_us":759228,"threads_started":2,"wal-append.queue_time_us":659767}
I20260709 13:02:55.201159  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 25 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:55.203146  1679 raft_consensus.cc:1217] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Deduplicated request from leader. Original: 18.12->[20.13-22.14]   Dedup: 22.14->[]
I20260709 13:02:55.204260  1679 raft_consensus.cc:1240] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 22. Current term is 23. Ops: [20.13-22.14]
I20260709 13:02:55.204995  1672 raft_consensus.cc:1240] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 24 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 23. Current term is 24. Ops: [22.12-23.13]
I20260709 13:02:55.206147  1679 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52778 (request call id 2992) took 1414 ms. Trace:
I20260709 13:02:55.206622  1679 rpcz_store.cc:276] 0709 13:02:53.791996 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:53.881783 (+ 89787us) service_pool.cc:224] Handling call
0709 13:02:55.202346 (+1320563us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:02:55.204680 (+  2334us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:55.205928 (+  1248us) spinlock_profiling.cc:243] Waited 1.32 s on lock 0x7b5400091350. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661618ad 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:02:55.206094 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3036834048}
I20260709 13:02:55.210011  1672 raft_consensus.cc:1275] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 25 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 22 index: 12. Preceding OpId from leader: term: 25 index: 14. (index mismatch)
I20260709 13:02:55.215907  1672 raft_consensus.cc:1275] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 21 index: 13. (index mismatch)
W20260709 13:02:55.226781  1570 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:54.961938  1717 consensus_peers.cc:597] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:02:55.235133  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 273224us
W20260709 13:02:55.243472  1718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:55.244441  1718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:55.246379  1747 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.773s	user 0.012s	sys 0.004s
W20260709 13:02:55.250234  1747 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.777s	user 0.013s	sys 0.004s
I20260709 13:02:55.247402  1748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55840 (request call id 3369) took 1432 ms. Trace:
I20260709 13:02:55.252204  1748 rpcz_store.cc:276] 0709 13:02:53.815168 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:53.818543 (+  3375us) service_pool.cc:224] Handling call
0709 13:02:53.818942 (+   399us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:02:54.017455 (+198513us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:54.017493 (+    38us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:02:54.018206 (+   713us) log.cc:844] Serialized 52 byte log entry
0709 13:02:54.821189 (+802983us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:02:54.821238 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:54.821322 (+    84us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:55.246701 (+425379us) raft_consensus.cc:1685] finished
0709 13:02:55.246762 (+    61us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:55.247109 (+   347us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":21899,"raft.run_cpu_time_us":170,"raft.run_wall_time_us":13241,"thread_start_us":741193,"threads_started":1,"wal-append.queue_time_us":986244}
I20260709 13:02:55.266964  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 22 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:55.248955  1752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55848 (request call id 3624) took 1460 ms. Trace:
W20260709 13:02:55.258811  1643 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 128 similar messages]
I20260709 13:02:55.329102  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 22 candidate_status { last_received { term: 19 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:54.975850  9830 raft_consensus.cc:515] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:55.336095  9960 raft_consensus.cc:2804] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 13:02:55.337692  9960 raft_consensus.cc:493] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
W20260709 13:02:55.334900 10009 consensus_meta.cc:287] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 1.256s	user 0.018s	sys 0.001s
I20260709 13:02:55.338340  9960 raft_consensus.cc:3060] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Advancing to term 22
W20260709 13:02:55.070045 10047 consensus_meta.cc:287] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.513s	user 0.007s	sys 0.006s
I20260709 13:02:55.383466  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 23 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:55.395306  1680 raft_consensus.cc:1275] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 26 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 21 index: 11. Preceding OpId from leader: term: 26 index: 13. (index mismatch)
I20260709 13:02:55.269685  1752 rpcz_store.cc:276] 0709 13:02:53.788807 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:53.789675 (+   868us) service_pool.cc:224] Handling call
0709 13:02:54.087627 (+297952us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:54.088112 (+   485us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:54.088155 (+    43us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:54.088887 (+   732us) log.cc:844] Serialized 26 byte log entry
0709 13:02:54.702323 (+613436us) raft_consensus.cc:1645] Marking committed up to 16
0709 13:02:54.702376 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:54.702486 (+   110us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:54.961523 (+259037us) raft_consensus.cc:1685] finished
0709 13:02:54.961548 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:55.248295 (+286747us) spinlock_profiling.cc:243] Waited 298 ms on lock 0x7b5400019fd0. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661618ad 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:02:55.248710 (+   415us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":29523,"raft.run_cpu_time_us":135,"raft.run_wall_time_us":136,"spinlock_wait_cycles":684650880,"thread_start_us":613025,"threads_started":1,"wal-append.queue_time_us":870845}
I20260709 13:02:55.400700  9666 leader_election.cc:290] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:55.402526 10083 raft_consensus.cc:493] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:55.403504 10083 raft_consensus.cc:515] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:55.404232  9979 leader_election.cc:290] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 23 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:55.010412  9869 raft_consensus.cc:515] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:55.407095 10024 raft_consensus.cc:2804] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 13:02:55.407686 10024 raft_consensus.cc:493] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:55.408090 10024 raft_consensus.cc:3060] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 25 FOLLOWER]: Advancing to term 26
I20260709 13:02:55.385953 10009 raft_consensus.cc:2749] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Leader election lost for term 19. Reason: Vote denied by peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625) with higher term. Message: Invalid argument: T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 19. Current term is 20.
I20260709 13:02:55.409631  9869 leader_election.cc:290] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:55.410718   431 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 13:02:55.411249  9830 leader_election.cc:290] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:55.411839  9703 raft_consensus.cc:493] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:55.412843  9703 raft_consensus.cc:515] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:55.414659 10023 raft_consensus.cc:493] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:55.414851  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 78582.3us
I20260709 13:02:55.415622 10023 raft_consensus.cc:515] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:55.419976  1492 maintenance_manager.cc:419] P 919f6392d4f44fd68987e2c5e7ec37ca: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=0.033338
W20260709 13:02:55.406605  9875 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.617s	user 0.022s	sys 0.036s
W20260709 13:02:55.421252  9875 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.654s	user 0.022s	sys 0.037s
W20260709 13:02:55.423467  2335 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.528s	user 0.023s	sys 0.038s
I20260709 13:02:55.426537  9840 raft_consensus.cc:2749] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260709 13:02:55.431000  2335 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.568s	user 0.024s	sys 0.038s
I20260709 13:02:55.434964  1747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55840 (request call id 3373) took 1173 ms. Trace:
I20260709 13:02:55.435407  1747 rpcz_store.cc:276] 0709 13:02:54.261699 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:54.261822 (+   123us) service_pool.cc:224] Handling call
0709 13:02:54.355450 (+ 93628us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:54.376891 (+ 21441us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:54.376947 (+    56us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:54.379233 (+  2286us) log.cc:844] Serialized 26 byte log entry
0709 13:02:54.472837 (+ 93604us) spinlock_profiling.cc:243] Waited 8.55 ms on lock 0x7b18003bd160. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f4263057e07 00007f426beb2ab9 00007f426307469b 00007f426307293f 00007f4263074c02 00007f4265cfc20b 00007f4265d01d23 00007f4265d020d1 00007f426611da0f
0709 13:02:55.251115 (+778278us) raft_consensus.cc:1645] Marking committed up to 16
0709 13:02:55.251260 (+   145us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:55.251430 (+   170us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:55.434588 (+183158us) raft_consensus.cc:1685] finished
0709 13:02:55.434616 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:55.434896 (+   280us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":27507,"raft.queue_time_us":23095,"raft.run_cpu_time_us":208,"raft.run_wall_time_us":24693,"spinlock_wait_cycles":19663744,"thread_start_us":777639,"threads_started":1,"wal-append.queue_time_us":1032103}
I20260709 13:02:55.442889  9998 raft_consensus.cc:493] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:55.443706  9998 raft_consensus.cc:515] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:55.446602 10124 raft_consensus.cc:493] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:55.447415 10124 raft_consensus.cc:515] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:55.448598  1569 leader_election.cc:304] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:02:55.449721  9840 raft_consensus.cc:2749] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 13:02:55.452157  1569 leader_election.cc:304] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:02:55.452808  9998 leader_election.cc:290] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:55.452778 10047 raft_consensus.cc:515] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:55.458021 10047 leader_election.cc:290] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 24 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:02:55.453528  9814 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.658s	user 0.016s	sys 0.011s
W20260709 13:02:55.461390  9814 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.666s	user 0.016s	sys 0.011s
W20260709 13:02:55.461782  1746 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.522s	user 0.003s	sys 0.007s
W20260709 13:02:55.462256  1746 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.638s	user 0.004s	sys 0.007s
I20260709 13:02:55.469916  9998 raft_consensus.cc:2804] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Leader election won for term 22
I20260709 13:02:55.470927  9998 raft_consensus.cc:697] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:02:55.472536 10056 raft_consensus.cc:493] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:55.473537 10056 raft_consensus.cc:515] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:55.474913  9998 consensus_queue.cc:237] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.14, Last appended by leader: 14, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:55.476872 10056 leader_election.cc:290] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:55.478991  2334 raft_consensus.cc:1240] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 24 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 23. Current term is 24. Ops: []
I20260709 13:02:55.478893  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:55.480000  2329 raft_consensus.cc:2468] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 19.
I20260709 13:02:55.482800  2331 raft_consensus.cc:1240] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 24 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 23. Current term is 24. Ops: []
I20260709 13:02:55.486120  1569 leader_election.cc:304] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 13:02:55.487435  9840 raft_consensus.cc:2804] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Leader election won for term 18
I20260709 13:02:55.488059  9840 raft_consensus.cc:697] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:02:55.489547  9840 consensus_queue.cc:237] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.13, Last appended by leader: 13, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:55.491118  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:55.418273 10023 leader_election.cc:290] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:55.450335 10124 leader_election.cc:290] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:55.497059 10083 leader_election.cc:290] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:02:55.518673  1673 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.574s	user 0.013s	sys 0.018s
W20260709 13:02:55.539851  1673 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.613s	user 0.014s	sys 0.019s
I20260709 13:02:55.545324  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:55.522886  9814 raft_consensus.cc:3055] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 13:02:55.568817  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 23 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:55.572093  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:55.561067 10023 raft_consensus.cc:493] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:55.565131  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:55.560954  9814 raft_consensus.cc:740] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:02:55.580816  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:55.591394  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:02:55.591164  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 364797us
I20260709 13:02:55.596194  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 27 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:02:55.586637 10052 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.530s	user 0.018s	sys 0.025s
W20260709 13:02:55.608705  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 122837us
I20260709 13:02:55.613696  2331 raft_consensus.cc:2468] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 25.
I20260709 13:02:55.617792  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:55.618657  1675 raft_consensus.cc:2468] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 25.
W20260709 13:02:55.622484  2332 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.636s	user 0.018s	sys 0.049s
W20260709 13:02:55.623085  2332 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.710s	user 0.018s	sys 0.049s
W20260709 13:02:55.614423 10052 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.602s	user 0.019s	sys 0.025s
I20260709 13:02:55.634825  2334 raft_consensus.cc:2468] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 20.
I20260709 13:02:55.638659  1718 leader_election.cc:304] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 13:02:55.638522 10023 raft_consensus.cc:515] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:02:55.640511  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 47695.9us
I20260709 13:02:55.641477 10023 leader_election.cc:290] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:02:55.648074 10143 log.cc:927] Time spent T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.063s	user 0.001s	sys 0.000s
I20260709 13:02:55.655925  1569 leader_election.cc:304] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:02:55.657367  9979 consensus_queue.cc:1048] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 13:02:55.659418  9814 consensus_queue.cc:260] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 24.12, Last appended by leader: 12, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:55.659945  9979 raft_consensus.cc:3060] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:02:55.661394  9814 raft_consensus.cc:3060] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Advancing to term 25
I20260709 13:02:55.664757   431 tablet_replica.cc:333] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a: stopping tablet replica
W20260709 13:02:55.666761  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 60138.7us
W20260709 13:02:55.670802  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 64622.7us
I20260709 13:02:55.671386  1673 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57092 (request call id 1854) took 1397 ms. Trace:
W20260709 13:02:55.671741  1717 leader_election.cc:336] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:02:55.672314  1717 leader_election.cc:304] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:55.671828  1673 rpcz_store.cc:276] 0709 13:02:54.274060 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:54.274201 (+   141us) service_pool.cc:224] Handling call
0709 13:02:54.274680 (+   479us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:54.274942 (+   262us) raft_consensus.cc:1534] Early marking committed up to index 15
0709 13:02:54.926172 (+651230us) log.cc:844] Serialized 14 byte log entry
0709 13:02:55.591237 (+665065us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:55.602695 (+ 11458us) raft_consensus.cc:1645] Marking committed up to 15
0709 13:02:55.609453 (+  6758us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:55.671113 (+ 61660us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:55.671317 (+   204us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":357165,"prepare.run_cpu_time_us":27,"prepare.run_wall_time_us":28,"raft.queue_time_us":5113,"raft.run_cpu_time_us":254,"raft.run_wall_time_us":15053,"thread_start_us":1183971,"threads_started":2}
I20260709 13:02:55.673722 10094 raft_consensus.cc:2804] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 25 FOLLOWER]: Leader election won for term 25
I20260709 13:02:55.674409 10094 raft_consensus.cc:697] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 25 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
W20260709 13:02:55.675227  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 94924us
I20260709 13:02:55.687273  2329 raft_consensus.cc:1275] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 24 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 22 index: 15. Preceding OpId from leader: term: 24 index: 16. (index mismatch)
I20260709 13:02:55.689075 10083 raft_consensus.cc:2749] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:02:55.695510  9979 raft_consensus.cc:2749] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:02:55.712441  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:55.717922  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:55.718688  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 19 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:55.718972  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:55.714418  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:55.721675  1752 raft_consensus.cc:3060] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:02:55.725664  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 19 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:55.675752 10094 consensus_queue.cc:237] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 24.17, Last appended by leader: 17, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:55.727093  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:02:55.715306  1717 consensus_peers.cc:597] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:02:55.728886  1717 leader_election.cc:336] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:02:55.729075  9783 raft_consensus.cc:493] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:55.729502  1717 leader_election.cc:304] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:55.676141  9703 leader_election.cc:290] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:55.729770  9783 raft_consensus.cc:515] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:55.730899  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:02:55.731225  1717 leader_election.cc:336] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:02:55.731747  1673 raft_consensus.cc:2468] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 20.
W20260709 13:02:55.732026 10141 log.cc:927] Time spent T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.147s	user 0.000s	sys 0.003s
I20260709 13:02:55.731784  1717 leader_election.cc:304] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:55.732420  9783 leader_election.cc:290] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:55.733476 10083 raft_consensus.cc:2749] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:02:55.734342 10083 raft_consensus.cc:2749] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:02:55.725504  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:02:55.709865  1643 leader_election.cc:336] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:02:55.773643  1643 leader_election.cc:304] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:55.775466  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:55.791960  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:55.795439  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:55.797544  1672 raft_consensus.cc:1275] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 17 index: 15. Preceding OpId from leader: term: 19 index: 16. (index mismatch)
W20260709 13:02:55.795682  9594 consensus_meta.cc:287] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 1.778s	user 0.022s	sys 0.004s
W20260709 13:02:55.799785  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 68642.7us
I20260709 13:02:55.806000  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:55.804817  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:02:55.799103  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 156814us
I20260709 13:02:55.818750  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:55.809171  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:55.826911  1673 raft_consensus.cc:3060] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 25 FOLLOWER]: Advancing to term 26
I20260709 13:02:55.830147  1679 raft_consensus.cc:3060] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:02:55.830315  9840 raft_consensus.cc:493] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:55.831252  9840 raft_consensus.cc:515] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:55.834231  9840 leader_election.cc:290] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:02:55.798385 10058 consensus_meta.cc:287] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.723s	user 0.014s	sys 0.006s
I20260709 13:02:55.799995 10071 raft_consensus.cc:2749] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
W20260709 13:02:55.837001  1717 leader_election.cc:336] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:02:55.811048  9960 raft_consensus.cc:515] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:55.876904  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 60460.5us
W20260709 13:02:55.880120  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 182233us
I20260709 13:02:55.898928 10122 raft_consensus.cc:493] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:55.899839 10122 raft_consensus.cc:515] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:55.901799 10138 consensus_queue.cc:1048] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.023s
I20260709 13:02:55.904754  1679 raft_consensus.cc:2468] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 22.
W20260709 13:02:55.908361  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 134378us
W20260709 13:02:55.916262 10153 log.cc:927] Time spent T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.151s	user 0.001s	sys 0.000s
I20260709 13:02:55.916807  9960 leader_election.cc:290] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 22 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:02:55.922263  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 245340us
I20260709 13:02:55.927561 10127 raft_consensus.cc:493] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:55.928295 10127 raft_consensus.cc:515] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:55.930474 10076 raft_consensus.cc:493] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:55.931289 10076 raft_consensus.cc:515] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:55.874172  9869 raft_consensus.cc:493] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:02:55.936678  9920 raft_consensus.cc:493] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:55.937213  9869 raft_consensus.cc:515] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:55.937531  9920 raft_consensus.cc:515] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:55.940167  9920 leader_election.cc:290] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:02:55.923444  1643 leader_election.cc:336] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 13:02:55.948181  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 140285us
W20260709 13:02:55.949227  1643 leader_election.cc:336] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:02:55.953588  1643 leader_election.cc:304] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:02:55.922246  9424 raft_consensus.cc:493] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:55.961877  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 27 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:02:55.963841  1717 consensus_peers.cc:597] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:02:55.983429  1717 leader_election.cc:336] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:02:55.983737 10076 leader_election.cc:290] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:55.984361  1717 leader_election.cc:304] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:55.985273 10122 leader_election.cc:290] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:02:55.991549  9931 consensus_meta.cc:287] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.964s	user 0.015s	sys 0.009s
I20260709 13:02:55.992374  9931 raft_consensus.cc:515] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:55.992527  1679 raft_consensus.cc:1275] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 18 index: 14. Preceding OpId from leader: term: 20 index: 15. (index mismatch)
I20260709 13:02:56.001398 10115 raft_consensus.cc:2749] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 13:02:55.964588  9424 raft_consensus.cc:515] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:56.004043 10127 leader_election.cc:290] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:02:56.006947  1717 consensus_peers.cc:597] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:02:56.007341  1749 raft_consensus.cc:1275] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 26 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 25 index: 13. Preceding OpId from leader: term: 26 index: 13. (term mismatch)
I20260709 13:02:56.008563  2332 raft_consensus.cc:1275] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 18 index: 14. Preceding OpId from leader: term: 19 index: 15. (index mismatch)
W20260709 13:02:56.013725  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 62649.8us
W20260709 13:02:56.015412 10159 log.cc:927] Time spent T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.095s	user 0.000s	sys 0.001s
W20260709 13:02:56.017848  9588 consensus_meta.cc:287] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 1.434s	user 0.019s	sys 0.006s
W20260709 13:02:56.018554 10152 log.cc:927] Time spent T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.259s	user 0.000s	sys 0.003s
I20260709 13:02:56.020605  9931 leader_election.cc:290] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 25 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:56.025938  1749 pending_rounds.cc:85] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701: Aborting all ops after (but not including) 12
W20260709 13:02:56.035915  1717 consensus_peers.cc:597] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:02:56.039165  9588 raft_consensus.cc:515] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:56.052727  1752 raft_consensus.cc:2468] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 22.
I20260709 13:02:56.006132  9424 leader_election.cc:290] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:56.055106  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:56.062512  2336 raft_consensus.cc:1240] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 25 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 24. Current term is 25. Ops: [24.17-24.17]
I20260709 13:02:56.061307  1749 pending_rounds.cc:107] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701: Aborting uncommitted NO_OP operation due to leader change: 25.13
I20260709 13:02:56.067147  1749 raft_consensus.cc:2889] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 26 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:02:56.061765  1569 leader_election.cc:304] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:02:56.069420  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:02:56.068106  1717 consensus_peers.cc:597] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:02:56.079805  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:56.083779  9588 leader_election.cc:290] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 25 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:56.089440  2335 pending_rounds.cc:85] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1: Aborting all ops after (but not including) 11
I20260709 13:02:56.089980  2335 pending_rounds.cc:107] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1: Aborting uncommitted NO_OP operation due to leader change: 16.12
I20260709 13:02:56.090301  2335 raft_consensus.cc:2889] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:02:56.097321 10138 consensus_queue.cc:1048] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.010s
W20260709 13:02:56.116375  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 249971us
I20260709 13:02:56.077999  9869 leader_election.cc:290] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:56.134835  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 25 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
W20260709 13:02:56.114176  1498 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35464 (request call id 286) took 126748 ms (client timeout 15000 ms). Trace:
W20260709 13:02:56.157930  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 108663us
I20260709 13:02:56.160194  1569 leader_election.cc:304] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:02:56.171618  1673 raft_consensus.cc:2468] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 26.
I20260709 13:02:56.162492  9703 raft_consensus.cc:2749] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:02:56.144651  9979 raft_consensus.cc:2749] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:02:56.179443   431 raft_consensus.cc:2243] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 LEADER]: Raft consensus shutting down.
I20260709 13:02:56.182089  2299 leader_election.cc:304] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:02:56.185796  1745 raft_consensus.cc:1240] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 25 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 24. Current term is 25. Ops: []
I20260709 13:02:56.193907  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 26 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:56.194921  1679 raft_consensus.cc:2468] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 25.
W20260709 13:02:56.187505  9814 consensus_meta.cc:287] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.526s	user 0.017s	sys 0.002s
I20260709 13:02:56.202749  1678 raft_consensus.cc:1275] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 24 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 20 index: 11. Preceding OpId from leader: term: 24 index: 13. (index mismatch)
I20260709 13:02:56.206956  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 24 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:56.212283  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 25 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:56.220142  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 20 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:02:56.147162  1498 rpcz_store.cc:269] 0709 13:00:49.365438 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:49.365751 (+   313us) service_pool.cc:224] Handling call
0709 13:02:56.113784 (+126748033us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":145512,"reported_tablets":47}
W20260709 13:02:56.203178 10024 consensus_meta.cc:287] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.795s	user 0.016s	sys 0.004s
W20260709 13:02:56.224766  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 149467us
I20260709 13:02:56.234560  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 26 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:56.252967  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:56.251827  1569 leader_election.cc:304] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:02:56.277654  1676 raft_consensus.cc:1275] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
I20260709 13:02:56.264914  1753 raft_consensus.cc:1240] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 21. Current term is 22. Ops: []
I20260709 13:02:56.278465  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 24 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:56.396031  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:56.279665  9588 raft_consensus.cc:2804] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 26 FOLLOWER]: Leader election won for term 26
I20260709 13:02:56.256494  9700 raft_consensus.cc:493] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:56.287725  1571 leader_election.cc:304] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:56.207901 10171 consensus_queue.cc:1059] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 20.11, Next index: 12, Last known committed idx: 10, Time since last communication: 0.063s
I20260709 13:02:56.397578  9513 raft_consensus.cc:2749] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:02:56.398160  9853 consensus_queue.cc:1059] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 23.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.158s
I20260709 13:02:56.234109 10024 raft_consensus.cc:515] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:56.517928  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 266381us
W20260709 13:02:56.501904  1491 tablet_replica.cc:1406] Time spent applying in-flights took a long time: real 0.207s	user 0.000s	sys 0.008s
W20260709 13:02:56.503695  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 560342us
I20260709 13:02:56.507746 10054 consensus_queue.cc:1048] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.295s
I20260709 13:02:56.508010  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 23 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:56.508773  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:02:56.509292  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 269301us
I20260709 13:02:56.510555  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:56.484819  9814 raft_consensus.cc:493] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
W20260709 13:02:56.531036  1643 leader_election.cc:336] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:02:56.539762  2337 raft_consensus.cc:3060] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Advancing to term 23
W20260709 13:02:56.513285  2298 consensus_peers.cc:597] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:02:56.541986  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:02:56.546826  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 285438us
I20260709 13:02:56.554584  1643 leader_election.cc:304] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
W20260709 13:02:56.559109  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 355225us
I20260709 13:02:56.565905 10073 raft_consensus.cc:2749] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:02:56.567276  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:56.568892  1750 raft_consensus.cc:1240] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 21. Current term is 22. Ops: [21.15-21.15]
I20260709 13:02:56.572456  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:56.580729  9869 consensus_queue.cc:1048] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 13:02:56.582278  1569 leader_election.cc:304] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:02:56.570626 10167 consensus_queue.cc:1048] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 13:02:56.607373  1747 raft_consensus.cc:1240] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 21. Current term is 22. Ops: []
I20260709 13:02:56.545558  9814 raft_consensus.cc:515] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:56.616039  1754 raft_consensus.cc:1275] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 21 index: 14. Preceding OpId from leader: term: 22 index: 15. (index mismatch)
W20260709 13:02:56.617682 10173 log.cc:927] Time spent T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.335s	user 0.000s	sys 0.001s
I20260709 13:02:56.617991  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:56.622867  1753 raft_consensus.cc:1275] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 25 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 23 index: 17. Preceding OpId from leader: term: 25 index: 18. (index mismatch)
I20260709 13:02:56.608004 10052 raft_consensus.cc:2804] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Leader election won for term 22
W20260709 13:02:56.627856 10170 log.cc:927] Time spent T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.184s	user 0.001s	sys 0.000s
W20260709 13:02:56.629233 10157 log.cc:927] Time spent T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.662s	user 0.000s	sys 0.003s
I20260709 13:02:56.617612  1747 raft_consensus.cc:1275] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 19 index: 14. Preceding OpId from leader: term: 22 index: 16. (index mismatch)
W20260709 13:02:56.630695  2298 leader_election.cc:336] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 26 election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:02:56.630959 10171 raft_consensus.cc:493] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:56.564776  9700 raft_consensus.cc:515] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:56.645607  2337 raft_consensus.cc:2410] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for term 23 because replica has last-logged OpId of term: 22 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 14.
I20260709 13:02:56.655105  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 20 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:56.637012  9594 raft_consensus.cc:493] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:56.681519  1679 raft_consensus.cc:1275] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 21 index: 14. Preceding OpId from leader: term: 22 index: 15. (index mismatch)
W20260709 13:02:56.681771  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 51165.6us
W20260709 13:02:56.682744  2298 leader_election.cc:336] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 13:02:56.657974 10174 log.cc:927] Time spent T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.128s	user 0.001s	sys 0.001s
I20260709 13:02:56.683352  2298 leader_election.cc:304] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:56.658490 10052 raft_consensus.cc:697] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
W20260709 13:02:56.705997  2298 consensus_peers.cc:597] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:02:56.711740 10170 log.cc:927] Time spent T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.052s	user 0.001s	sys 0.000s
W20260709 13:02:56.731587  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 171366us
I20260709 13:02:56.684827  9703 consensus_queue.cc:1059] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 21.17, Next index: 18, Last known committed idx: 16, Time since last communication: 0.037s
I20260709 13:02:56.727746  9588 raft_consensus.cc:697] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 26 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:02:56.734851  9703 raft_consensus.cc:3055] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 LEADER]: Stepping down as leader of term 21
I20260709 13:02:56.709820 10009 raft_consensus.cc:2749] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:02:56.738087  2337 raft_consensus.cc:3055] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 13:02:56.739390  2337 raft_consensus.cc:740] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:02:56.656306 10171 raft_consensus.cc:515] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:56.738024  1569 leader_election.cc:304] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:02:56.749590  1569 leader_election.cc:304] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:02:56.735292  9703 raft_consensus.cc:740] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:02:56.751565  9703 consensus_queue.cc:260] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 21.17, Last appended by leader: 17, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:56.700511  9594 raft_consensus.cc:515] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:02:56.759720  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 191187us
I20260709 13:02:56.711501  9920 consensus_queue.cc:1048] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.033s
I20260709 13:02:56.766809 10189 raft_consensus.cc:2749] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 13:02:56.769989 10189 raft_consensus.cc:493] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:56.753602  9703 raft_consensus.cc:3060] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:02:56.808683 10189 raft_consensus.cc:515] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:56.823882  9700 leader_election.cc:290] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:02:56.827276 10165 log.cc:927] Time spent T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.764s	user 0.005s	sys 0.000s
I20260709 13:02:56.836565 10167 raft_consensus.cc:493] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:56.837455 10181 log.cc:927] Time spent T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.207s	user 0.001s	sys 0.000s
I20260709 13:02:56.837764  9931 raft_consensus.cc:493] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
W20260709 13:02:56.844870 10177 log.cc:927] Time spent T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.216s	user 0.001s	sys 0.000s
I20260709 13:02:56.847468 10052 consensus_queue.cc:237] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 20.13, Last appended by leader: 13, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:56.848834 10054 consensus_queue.cc:1048] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.001s
W20260709 13:02:56.850656 10172 log.cc:927] Time spent T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.267s	user 0.001s	sys 0.001s
I20260709 13:02:56.852269  2337 consensus_queue.cc:260] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.15, Last appended by leader: 15, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:56.853559 10127 raft_consensus.cc:2749] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
W20260709 13:02:56.863312 10147 log.cc:927] Time spent T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.933s	user 0.005s	sys 0.000s
I20260709 13:02:56.864355 10171 leader_election.cc:290] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:56.867504  1748 raft_consensus.cc:3055] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 23 LEADER]: Stepping down as leader of term 23
I20260709 13:02:56.764432 10024 leader_election.cc:290] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 26 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:56.735477  9588 consensus_queue.cc:237] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 24.18, Last appended by leader: 18, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:56.893620  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 144262us
I20260709 13:02:56.805461  9979 raft_consensus.cc:2749] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
W20260709 13:02:56.902878  1717 leader_election.cc:336] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:02:56.907014  1748 raft_consensus.cc:740] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:02:56.889603 10073 consensus_queue.cc:1048] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.024s
I20260709 13:02:56.911492  1749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55840 (request call id 3371) took 2670 ms. Trace:
I20260709 13:02:56.915513  1749 rpcz_store.cc:276] 0709 13:02:54.240700 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:54.260440 (+ 19740us) service_pool.cc:224] Handling call
0709 13:02:54.260882 (+   442us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:56.116233 (+1855351us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:56.895591 (+779358us) spinlock_profiling.cc:243] Waited 1.75 s on lock 0x7b54001c7d54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:02:56.911381 (+ 15790us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4015758208}
I20260709 13:02:56.919723  1569 leader_election.cc:304] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
W20260709 13:02:56.917658  1717 leader_election.cc:336] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:02:56.895380  9931 raft_consensus.cc:515] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:56.890012 10167 raft_consensus.cc:515] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:56.957297  9830 raft_consensus.cc:3055] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 LEADER]: Stepping down as leader of term 23
I20260709 13:02:56.966174  9830 raft_consensus.cc:740] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:02:56.958597  9513 raft_consensus.cc:2749] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 13:02:56.967809 10167 leader_election.cc:290] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:02:56.958947  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 435487us
W20260709 13:02:56.969295  1643 leader_election.cc:336] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:02:56.959923  9869 raft_consensus.cc:493] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:56.960543  9783 raft_consensus.cc:493] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:02:56.960671  9931 leader_election.cc:290] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:56.974838  1643 leader_election.cc:304] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
W20260709 13:02:56.976521  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 65762.5us
I20260709 13:02:56.977123   431 raft_consensus.cc:2272] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Raft consensus is shut down!
I20260709 13:02:56.977489  1569 leader_election.cc:304] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:02:56.977945  9920 raft_consensus.cc:2749] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 13:02:56.978421 10127 raft_consensus.cc:2749] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 13:02:56.960775  1644 leader_election.cc:304] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:56.921209  9814 leader_election.cc:290] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:56.962028 10180 raft_consensus.cc:493] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:56.963523  1491 tablet_replica.cc:1411] Time spent flushing the Log queue took a long time: real 0.434s	user 0.009s	sys 0.011s
I20260709 13:02:56.957384  9812 raft_consensus.cc:493] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
W20260709 13:02:56.988863  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 226347us
I20260709 13:02:56.991524  1748 consensus_queue.cc:260] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 23.13, Last appended by leader: 13, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:57.016048  1569 leader_election.cc:304] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
W20260709 13:02:57.020305  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 59699.6us
I20260709 13:02:57.025156  1748 raft_consensus.cc:3060] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Advancing to term 24
I20260709 13:02:56.967746  9830 consensus_queue.cc:260] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 23.17, Last appended by leader: 17, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:57.004343 10076 raft_consensus.cc:2749] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 13:02:56.991362  9783 raft_consensus.cc:515] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:56.992257  9869 raft_consensus.cc:515] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:57.039120  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 69915.7us
I20260709 13:02:56.996623  9812 raft_consensus.cc:515] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:56.996414 10180 raft_consensus.cc:515] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:57.055817  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:57.055967  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:57.056780  1745 raft_consensus.cc:2468] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 22.
I20260709 13:02:57.062450 10104 raft_consensus.cc:2804] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:02:57.063022 10104 raft_consensus.cc:493] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:02:57.065557  1644 leader_election.cc:304] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:02:57.046597 10189 leader_election.cc:290] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:57.049062 10127 raft_consensus.cc:2804] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 13:02:57.071916 10076 raft_consensus.cc:2804] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:02:57.072431 10076 raft_consensus.cc:493] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:02:57.072877 10076 raft_consensus.cc:3060] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:02:57.073318  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:02:57.075206 10196 log.cc:927] Time spent T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.069s	user 0.000s	sys 0.000s
I20260709 13:02:57.076697  1675 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57092 (request call id 1872) took 1128 ms. Trace:
I20260709 13:02:57.077064  1675 rpcz_store.cc:276] 0709 13:02:55.948125 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:55.948356 (+   231us) service_pool.cc:224] Handling call
0709 13:02:55.956305 (+  7949us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:56.221114 (+264809us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:56.221151 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:56.263126 (+ 41975us) log.cc:844] Serialized 26 byte log entry
0709 13:02:56.800751 (+537625us) raft_consensus.cc:1645] Marking committed up to 15
0709 13:02:56.800835 (+    84us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:56.800984 (+   149us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:57.076432 (+275448us) raft_consensus.cc:1685] finished
0709 13:02:57.076458 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:57.076620 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":929325,"raft.queue_time_us":267353,"raft.run_cpu_time_us":214,"raft.run_wall_time_us":21370,"thread_start_us":287361,"threads_started":1,"wal-append.queue_time_us":473355}
I20260709 13:02:57.059062 10071 raft_consensus.cc:493] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 24 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:02:57.079000 10071 raft_consensus.cc:515] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:02:57.079715 10202 log.cc:927] Time spent T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.061s	user 0.001s	sys 0.000s
I20260709 13:02:57.082063 10071 leader_election.cc:290] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:02:57.082248  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 302501us
I20260709 13:02:57.086131  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:57.084067  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:57.070062  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 26 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:57.042203 10131 consensus_queue.cc:1059] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 23.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.029s
I20260709 13:02:57.092970  2333 raft_consensus.cc:2393] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 25: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:02:57.065408 10104 raft_consensus.cc:3060] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:02:57.090793  1569 leader_election.cc:304] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:02:57.090844  9830 raft_consensus.cc:3060] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Advancing to term 24
I20260709 13:02:57.098603  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:57.101646  2328 raft_consensus.cc:1275] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 25 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 22 index: 16. Preceding OpId from leader: term: 25 index: 18. (index mismatch)
W20260709 13:02:57.102608  1719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:57.071293 10127 raft_consensus.cc:493] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:02:57.102941  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 290263us
I20260709 13:02:57.103734 10201 raft_consensus.cc:493] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:57.106019 10072 raft_consensus.cc:493] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
W20260709 13:02:57.116304 10194 log.cc:927] Time spent T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.084s	user 0.001s	sys 0.000s
I20260709 13:02:57.135557  9783 leader_election.cc:290] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:02:57.136739 10200 log.cc:927] Time spent T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.092s	user 0.000s	sys 0.000s
I20260709 13:02:57.141474  2337 raft_consensus.cc:3060] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:02:57.142305  2331 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49082 (request call id 1467) took 1022 ms. Trace:
I20260709 13:02:57.145093 10076 raft_consensus.cc:515] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:57.102487  1718 leader_election.cc:304] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 13:02:57.165201  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 62890.8us
W20260709 13:02:57.167013  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 127009us
I20260709 13:02:57.167543  9814 raft_consensus.cc:2749] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 13:02:57.174997  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:02:57.178925 10205 log.cc:927] Time spent T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.090s	user 0.000s	sys 0.001s
I20260709 13:02:57.158121 10047 raft_consensus.cc:493] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:57.183848  9869 leader_election.cc:290] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:57.183955 10047 raft_consensus.cc:515] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:57.184986 10076 leader_election.cc:290] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 21 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:57.186358   431 tablet_replica.cc:333] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a: stopping tablet replica
I20260709 13:02:57.160147 10127 raft_consensus.cc:3060] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 25 FOLLOWER]: Advancing to term 26
I20260709 13:02:57.188373   431 raft_consensus.cc:2243] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Raft consensus shutting down.
I20260709 13:02:57.191938  9812 leader_election.cc:290] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:57.192441  2301 leader_election.cc:304] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:02:57.193403  2330 raft_consensus.cc:2468] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 24.
I20260709 13:02:57.194466 10199 raft_consensus.cc:2749] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 13:02:57.195077 10072 raft_consensus.cc:515] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:57.203428 10072 leader_election.cc:290] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:57.204629 10047 leader_election.cc:290] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:57.164471  2331 rpcz_store.cc:276] 0709 13:02:56.119908 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:56.120079 (+   171us) service_pool.cc:224] Handling call
0709 13:02:56.127843 (+  7764us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:02:56.178715 (+ 50872us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:56.178754 (+    39us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:02:56.186233 (+  7479us) log.cc:844] Serialized 52 byte log entry
0709 13:02:56.331983 (+145750us) raft_consensus.cc:1645] Marking committed up to 13
0709 13:02:56.843880 (+511897us) spinlock_profiling.cc:243] Waited 13.9 ms on lock 0x7b4400022b4c. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f4262f9bc85 00007f4262f9c6a2 00007f4265d0e1db 00007f4265d09aaf 00007f4265cfd86e 00007f4265cfcbb6 00007f4265cfc7d8 00007f4265d06221 00007f4265d061d9 00007f4265d06169
0709 13:02:56.915014 (+ 71134us) log.cc:844] Serialized 14 byte log entry
0709 13:02:56.915607 (+   593us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:56.915704 (+    97us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:57.137491 (+221787us) spinlock_profiling.cc:243] Waited 10.7 ms on lock 0x7b240034acd8. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f4262efb9c4 00007f4262ef9acb 00007f42630b4dc9 00007f42630b4a55 00007f42630b3fb0 00007f4265d2a488 00007f4265d2abe8 00007f4265d02398 00007f4265cfd76a 00007f4265cfcbb6
0709 13:02:57.141983 (+  4492us) raft_consensus.cc:1685] finished
0709 13:02:57.142015 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:57.142214 (+   199us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":304997,"prepare.run_cpu_time_us":57,"prepare.run_wall_time_us":59,"raft.queue_time_us":31059,"raft.run_cpu_time_us":377,"raft.run_wall_time_us":19628,"spinlock_wait_cycles":56798336,"thread_start_us":522921,"threads_started":2,"wal-append.queue_time_us":387274}
I20260709 13:02:57.162400 10201 raft_consensus.cc:515] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:57.203451  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:57.207219  1679 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52766 (request call id 3479) took 1002 ms. Trace:
I20260709 13:02:57.212289  1679 rpcz_store.cc:276] 0709 13:02:56.204538 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:56.204740 (+   202us) service_pool.cc:224] Handling call
0709 13:02:56.222402 (+ 17662us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:56.682066 (+459664us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:57.206920 (+524854us) spinlock_profiling.cc:243] Waited 459 ms on lock 0x7b5400163854. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:02:57.207123 (+   203us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1055048064}
I20260709 13:02:57.208025  2336 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49082 (request call id 1470) took 1009 ms. Trace:
I20260709 13:02:57.209278  2329 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49082 (request call id 1469) took 1033 ms. Trace:
I20260709 13:02:57.220615 10180 leader_election.cc:290] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:57.224140 10201 leader_election.cc:290] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:57.225571 10024 consensus_queue.cc:1059] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 24.17, Next index: 18, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 13:02:57.168200 10073 consensus_queue.cc:1048] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.059s
I20260709 13:02:57.227134 10131 raft_consensus.cc:3055] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 24 LEADER]: Stepping down as leader of term 24
I20260709 13:02:57.227674 10131 raft_consensus.cc:740] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:02:57.229270  9594 leader_election.cc:290] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:57.228915 10131 consensus_queue.cc:260] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 24.17, Last appended by leader: 17, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:57.213986  2329 rpcz_store.cc:276] 0709 13:02:56.177710 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:56.180146 (+  2436us) service_pool.cc:224] Handling call
0709 13:02:56.186144 (+  5998us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:02:56.208995 (+ 22851us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:56.209060 (+    65us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:02:56.227505 (+ 18445us) log.cc:844] Serialized 52 byte log entry
0709 13:02:56.518754 (+291249us) raft_consensus.cc:1645] Marking committed up to 15
0709 13:02:56.518941 (+   187us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:56.519161 (+   220us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:56.836435 (+317274us) spinlock_profiling.cc:243] Waited 12 ms on lock 0x7b4400022b4c. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f4262f9bc85 00007f4262f9c6a2 00007f4265d0e1db 00007f4265d09aaf 00007f4265cfd86e 00007f4265cfcbb6 00007f4265cfc7d8 00007f4265d06221 00007f4265d061d9 00007f4265d06169
0709 13:02:57.208061 (+371626us) spinlock_profiling.cc:243] Waited 6.44 ms on lock 0x7b240034acd8. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f4262efb9c4 00007f4262ef9acb 00007f42630b4dc9 00007f42630b4a55 00007f42630b3fb0 00007f4265d2a488 00007f4265d2abe8 00007f4265d02398 00007f4265cfd76a 00007f4265cfcbb6
0709 13:02:57.208997 (+   936us) raft_consensus.cc:1685] finished
0709 13:02:57.209029 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:57.209205 (+   176us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":9059,"raft.queue_time_us":20981,"raft.run_cpu_time_us":154,"raft.run_wall_time_us":16965,"spinlock_wait_cycles":42529408,"thread_start_us":124345,"threads_started":1,"wal-append.queue_time_us":339246}
W20260709 13:02:57.232671  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 144185us
W20260709 13:02:57.240186  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 149539us
I20260709 13:02:57.241271  1569 leader_election.cc:304] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:02:57.243191  2301 leader_election.cc:304] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:02:57.248381  1569 leader_election.cc:304] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:02:57.253847  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:57.254670  1679 raft_consensus.cc:2468] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 18.
I20260709 13:02:57.254769  9424 consensus_queue.cc:1048] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 13:02:57.256489  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:57.257248  2331 raft_consensus.cc:2468] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 23.
W20260709 13:02:57.262699  2298 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:57.259763  1569 leader_election.cc:304] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:02:57.213600  2336 rpcz_store.cc:276] 0709 13:02:56.198342 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:56.198548 (+   206us) service_pool.cc:224] Handling call
0709 13:02:56.212998 (+ 14450us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:56.491151 (+278153us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:56.501965 (+ 10814us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:56.539832 (+ 37867us) log.cc:844] Serialized 26 byte log entry
0709 13:02:56.773152 (+233320us) raft_consensus.cc:1645] Marking committed up to 16
0709 13:02:56.773246 (+    94us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:56.773385 (+   139us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:57.207742 (+434357us) raft_consensus.cc:1685] finished
0709 13:02:57.207772 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:57.207949 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":227795,"raft.run_cpu_time_us":217,"raft.run_wall_time_us":25573,"thread_start_us":185271,"threads_started":1,"wal-append.queue_time_us":424609}
I20260709 13:02:57.285946 10182 raft_consensus.cc:493] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:57.287066 10182 raft_consensus.cc:515] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } }
I20260709 13:02:57.290949 10182 leader_election.cc:290] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:57.293736  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:57.293906  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 26 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:57.292785  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:57.297080  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:57.297407  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 27 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:57.298137  1676 raft_consensus.cc:2468] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 26.
I20260709 13:02:57.299960  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:57.300638  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:57.300897  1676 raft_consensus.cc:2468] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 18.
I20260709 13:02:57.304354  2332 raft_consensus.cc:1240] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 22. Current term is 23. Ops: []
I20260709 13:02:57.307397  1748 raft_consensus.cc:1275] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 23 index: 13. Preceding OpId from leader: term: 24 index: 13. (term mismatch)
I20260709 13:02:57.308058  1748 pending_rounds.cc:85] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701: Aborting all ops after (but not including) 12
I20260709 13:02:57.308418  1748 pending_rounds.cc:107] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701: Aborting uncommitted NO_OP operation due to leader change: 23.13
I20260709 13:02:57.308986  1748 raft_consensus.cc:2889] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:02:57.313537  1646 leader_election.cc:304] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:57.247292 10201 raft_consensus.cc:493] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:57.320503  2299 leader_election.cc:304] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:02:57.332129  1748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55840 (request call id 3396) took 1094 ms. Trace:
I20260709 13:02:57.254966 10073 raft_consensus.cc:2706] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Received pre-election callback for term 19 while not running: Illegal state: RaftConsensus is not running: State = Stopping
I20260709 13:02:57.238647  9979 raft_consensus.cc:2749] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 13:02:57.332863  1748 rpcz_store.cc:276] 0709 13:02:56.237242 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:56.237423 (+   181us) service_pool.cc:224] Handling call
0709 13:02:56.257459 (+ 20036us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:56.867341 (+609882us) spinlock_profiling.cc:243] Waited 42.8 ms on lock 0x7b54002cefa0. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42660d9dc0 00007f42661641c0 00007f4266165231 00007f4266161d8f 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c
0709 13:02:57.330912 (+463571us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:57.332033 (+  1121us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":98528000}
I20260709 13:02:57.357872   431 raft_consensus.cc:2272] T a8c04bcdff52470cb06667e46356e653 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Raft consensus is shut down!
I20260709 13:02:57.366927  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:57.370122  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:57.334980 10071 raft_consensus.cc:2804] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 13:02:57.326015  9812 raft_consensus.cc:493] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
W20260709 13:02:57.342700 10115 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 24 current term: 24 is higher.
I20260709 13:02:57.387198  1674 raft_consensus.cc:1240] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 26 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 25. Current term is 26. Ops: []
W20260709 13:02:57.374176  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 157740us
W20260709 13:02:57.400913  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 141339us
I20260709 13:02:57.401938  1569 leader_election.cc:304] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:02:57.370426 10052 raft_consensus.cc:2749] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:02:57.368925 10073 raft_consensus.cc:2804] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:02:57.399391  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 22 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:57.391000 10047 raft_consensus.cc:2804] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:02:57.391772 10071 raft_consensus.cc:493] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:57.392055  9424 raft_consensus.cc:493] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:57.413594  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 101716us
I20260709 13:02:57.420886  2337 raft_consensus.cc:1275] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 21 index: 15. Preceding OpId from leader: term: 22 index: 16. (index mismatch)
I20260709 13:02:57.427225  1718 leader_election.cc:304] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:57.430949  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 22 candidate_status { last_received { term: 21 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:57.358143 10201 raft_consensus.cc:515] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:57.436147  2301 leader_election.cc:304] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:02:57.412838 10047 raft_consensus.cc:493] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:57.420598 10071 raft_consensus.cc:3060] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Advancing to term 23
W20260709 13:02:57.444304  1644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:57.414047 10073 raft_consensus.cc:493] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:57.450103  1569 leader_election.cc:304] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:02:57.443992 10063 raft_consensus.cc:2749] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:02:57.444190  9869 consensus_queue.cc:1059] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 22.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.016s
I20260709 13:02:57.407430  9812 raft_consensus.cc:515] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:57.465788  1717 leader_election.cc:336] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 22 election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:02:57.433885  9979 raft_consensus.cc:2749] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:02:57.539237  9979 raft_consensus.cc:493] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:57.539393  1570 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:57.539867  9979 raft_consensus.cc:515] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:57.468329 10201 leader_election.cc:290] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 13:02:57.469641  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 48775.4us
I20260709 13:02:57.543617  2301 leader_election.cc:304] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:02:57.545442  2301 leader_election.cc:304] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:02:57.545605 10225 raft_consensus.cc:2749] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 13:02:57.472052  9853 raft_consensus.cc:3055] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 23 LEADER]: Stepping down as leader of term 23
I20260709 13:02:57.472309  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:57.473613  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:57.474409 10224 raft_consensus.cc:2804] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 13:02:57.475718  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:57.443627  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8c04bcdff52470cb06667e46356e653" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:02:57.447599  1644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:57.437144  9594 raft_consensus.cc:2804] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 13:02:57.485289  1718 leader_election.cc:304] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:02:57.484318 10073 raft_consensus.cc:3060] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:02:57.488277  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:57.493270 10131 raft_consensus.cc:3060] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 24 FOLLOWER]: Advancing to term 25
I20260709 13:02:57.495282  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:02:57.499799 10218 log.cc:927] Time spent T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.055s	user 0.000s	sys 0.000s
W20260709 13:02:57.513171 10211 log.cc:927] Time spent T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.095s	user 0.000s	sys 0.000s
I20260709 13:02:57.516263  1491 maintenance_manager.cc:643] P 919f6392d4f44fd68987e2c5e7ec37ca: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 2.095s	user 0.057s	sys 0.028s Metrics: {"bytes_written":5774,"delete_count":0,"lbm_write_time_us":1303,"lbm_writes_lt_1ms":6,"reinsert_count":0,"thread_start_us":174119,"threads_started":1,"update_count":214,"wal-append.queue_time_us":332578}
I20260709 13:02:57.447096 10047 raft_consensus.cc:3060] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:02:57.536789  1717 leader_election.cc:304] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:57.538223  9869 raft_consensus.cc:3055] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 13:02:57.429199  9424 raft_consensus.cc:515] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:57.570631 10224 raft_consensus.cc:493] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:02:57.582594  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 99882.3us
I20260709 13:02:57.695420  1718 leader_election.cc:304] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
W20260709 13:02:57.695818  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 150525us
W20260709 13:02:57.695271  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 229597us
W20260709 13:02:57.697597  1717 leader_election.cc:336] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:02:57.567106  9853 raft_consensus.cc:740] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:02:57.572607  9594 raft_consensus.cc:493] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:57.572790  9979 leader_election.cc:290] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:57.703399  1718 leader_election.cc:304] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 13:02:57.698897  1717 leader_election.cc:336] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:02:57.573009 10201 raft_consensus.cc:2749] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 13:02:57.684757 10094 raft_consensus.cc:2749] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 13:02:57.685408  9869 raft_consensus.cc:740] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:02:57.727733  2301 leader_election.cc:304] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:02:57.582813 10104 raft_consensus.cc:515] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:57.700729 10224 raft_consensus.cc:3060] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 24 FOLLOWER]: Advancing to term 25
W20260709 13:02:57.748746  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 298852us
I20260709 13:02:57.713167  9594 raft_consensus.cc:3060] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Advancing to term 24
I20260709 13:02:57.705648 10171 raft_consensus.cc:2749] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 13:02:57.775455  1569 leader_election.cc:304] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
W20260709 13:02:57.775604  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 47980.4us
I20260709 13:02:57.719872 10167 raft_consensus.cc:2804] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:02:57.718649 10063 raft_consensus.cc:493] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:02:57.793215  2301 leader_election.cc:304] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:02:57.779057  1492 maintenance_manager.cc:419] P 919f6392d4f44fd68987e2c5e7ec37ca: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.288885
I20260709 13:02:57.754985 10054 raft_consensus.cc:2804] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 13:02:57.798830 10054 raft_consensus.cc:493] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:57.799079 10104 leader_election.cc:290] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 20 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:57.799292 10054 raft_consensus.cc:3060] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 25 FOLLOWER]: Advancing to term 26
I20260709 13:02:57.787490 10201 raft_consensus.cc:2749] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:02:57.808096 10182 raft_consensus.cc:493] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:02:57.808892 10182 raft_consensus.cc:515] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:57.810868  2334 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49082 (request call id 1462) took 1801 ms. Trace:
I20260709 13:02:57.811350  2334 rpcz_store.cc:276] 0709 13:02:56.009265 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:56.009448 (+   183us) service_pool.cc:224] Handling call
0709 13:02:56.009867 (+   419us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:56.010524 (+   657us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:56.011126 (+   602us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:56.012254 (+  1128us) log.cc:844] Serialized 26 byte log entry
0709 13:02:56.192249 (+179995us) raft_consensus.cc:1645] Marking committed up to 17
0709 13:02:56.490175 (+297926us) log.cc:844] Serialized 14 byte log entry
0709 13:02:56.747868 (+257693us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:56.747992 (+   124us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:57.769011 (+1021019us) spinlock_profiling.cc:243] Waited 19.9 ms on lock 0x7b240034acd8. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f4262efb9c4 00007f4262ef9acb 00007f42630b4dc9 00007f42630b4a55 00007f42630b3fb0 00007f4265d2a488 00007f4265d2abe8 00007f4265d02398 00007f4265cfd76a 00007f4265cfcbb6
0709 13:02:57.810525 (+ 41514us) raft_consensus.cc:1685] finished
0709 13:02:57.810634 (+   109us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:57.810762 (+   128us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":20961,"prepare.queue_time_us":438945,"prepare.run_cpu_time_us":55,"prepare.run_wall_time_us":59,"raft.queue_time_us":84288,"raft.run_cpu_time_us":321,"raft.run_wall_time_us":7535,"spinlock_wait_cycles":45813120,"thread_start_us":651975,"threads_started":3,"wal-append.queue_time_us":218746}
I20260709 13:02:57.814008 10182 leader_election.cc:290] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:57.813817  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:57.813866  1674 raft_consensus.cc:1275] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 26 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 24 index: 18. Preceding OpId from leader: term: 26 index: 19. (index mismatch)
W20260709 13:02:57.798134  9830 consensus_meta.cc:287] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.644s	user 0.016s	sys 0.000s
I20260709 13:02:57.755239 10226 raft_consensus.cc:493] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:57.789944  9703 consensus_meta.cc:287] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.905s	user 0.007s	sys 0.010s
W20260709 13:02:57.863922  2298 leader_election.cc:336] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:02:57.782147 10167 raft_consensus.cc:493] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:02:57.875815 10239 raft_consensus.cc:2749] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
W20260709 13:02:57.857004 10222 log.cc:927] Time spent T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.108s	user 0.000s	sys 0.000s
I20260709 13:02:57.880333 10138 raft_consensus.cc:2749] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 13:02:57.861693 10052 consensus_queue.cc:1048] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.045s
I20260709 13:02:57.906998 10182 consensus_queue.cc:1048] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
W20260709 13:02:57.907771 10229 log.cc:927] Time spent T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.092s	user 0.000s	sys 0.000s
I20260709 13:02:57.902971  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 25 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:57.914623  1747 raft_consensus.cc:2393] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 25: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:02:57.890573 10167 raft_consensus.cc:3060] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:02:57.926810 10239 raft_consensus.cc:493] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:57.898972 10204 consensus_queue.cc:1059] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 21.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.029s
I20260709 13:02:57.819715  9979 consensus_queue.cc:1048] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 10, Time since last communication: 0.082s
I20260709 13:02:57.798207  9869 consensus_queue.cc:260] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.15, Last appended by leader: 15, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:57.934852  9812 leader_election.cc:290] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:57.893452 10094 raft_consensus.cc:493] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:57.944304  9853 consensus_queue.cc:260] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 23.17, Last appended by leader: 17, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:57.945130 10094 raft_consensus.cc:515] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:57.901934 10076 raft_consensus.cc:493] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:02:57.913280  9588 consensus_queue.cc:1048] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.029s
I20260709 13:02:57.943902 10204 raft_consensus.cc:3055] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 13:02:57.945482  9853 raft_consensus.cc:3060] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Advancing to term 24
I20260709 13:02:57.942344 10239 raft_consensus.cc:515] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:57.821448 10063 raft_consensus.cc:515] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:57.948192 10204 raft_consensus.cc:740] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
W20260709 13:02:57.986830  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 272481us
W20260709 13:02:57.962921 10127 consensus_meta.cc:287] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.762s	user 0.013s	sys 0.003s
I20260709 13:02:57.994777  1718 leader_election.cc:304] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:02:57.896694 10226 raft_consensus.cc:515] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:58.001840  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 25 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:58.002916  1748 raft_consensus.cc:2410] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 for term 25 because replica has last-logged OpId of term: 23 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 16.
I20260709 13:02:57.973856   431 tablet_replica.cc:333] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a: stopping tablet replica
I20260709 13:02:57.987725  9869 raft_consensus.cc:3060] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:02:58.022749  1718 leader_election.cc:304] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:58.027539 10239 leader_election.cc:290] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:57.992849 10203 consensus_queue.cc:1048] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.035s
I20260709 13:02:58.029836 10204 consensus_queue.cc:260] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 21.16, Last appended by leader: 16, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:58.032508 10115 raft_consensus.cc:2749] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:02:58.011397 10236 consensus_queue.cc:1059] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.032s
I20260709 13:02:58.036873 10094 leader_election.cc:290] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:58.037475 10094 raft_consensus.cc:3055] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 25 LEADER]: Stepping down as leader of term 25
I20260709 13:02:58.037798 10094 raft_consensus.cc:740] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:02:57.955741 10076 raft_consensus.cc:515] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:57.994100 10171 raft_consensus.cc:493] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:58.073892   431 raft_consensus.cc:2243] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 LEADER]: Raft consensus shutting down.
I20260709 13:02:58.090533 10094 consensus_queue.cc:260] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 25.18, Last appended by leader: 18, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:58.091900 10076 leader_election.cc:290] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:58.092085 10094 raft_consensus.cc:3060] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 25 FOLLOWER]: Advancing to term 26
I20260709 13:02:58.006501  9594 raft_consensus.cc:515] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:58.090415   431 raft_consensus.cc:2272] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Raft consensus is shut down!
W20260709 13:02:58.099875  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 307893us
I20260709 13:02:58.102691 10242 raft_consensus.cc:493] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:58.103348  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:58.104477 10247 raft_consensus.cc:493] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:58.124708 10247 raft_consensus.cc:515] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:58.127161 10247 leader_election.cc:290] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:58.114604 10226 leader_election.cc:290] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:58.043452  9812 raft_consensus.cc:2749] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 13:02:58.005348 10127 raft_consensus.cc:515] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:58.129992 10127 leader_election.cc:290] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 26 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:02:58.118222  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 89450.5us
I20260709 13:02:58.135288 10242 raft_consensus.cc:515] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:58.136616 10247 raft_consensus.cc:493] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:58.136909  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 26 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:58.137284 10247 raft_consensus.cc:515] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:58.137715  1677 raft_consensus.cc:2468] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 25.
I20260709 13:02:58.138027 10242 leader_election.cc:290] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:58.139596 10247 leader_election.cc:290] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:58.050107 10054 raft_consensus.cc:515] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:58.141778  1718 leader_election.cc:304] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
W20260709 13:02:58.147680  1570 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:58.147920 10233 log.cc:927] Time spent T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.289s	user 0.002s	sys 0.005s
I20260709 13:02:58.119132  9424 leader_election.cc:290] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:58.156028  9594 leader_election.cc:290] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 24 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:58.156574  1751 raft_consensus.cc:1240] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 23. Current term is 24. Ops: []
I20260709 13:02:58.157734 10063 leader_election.cc:290] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:58.173709  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:02:58.177666 10071 consensus_meta.cc:287] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.711s	user 0.015s	sys 0.000s
I20260709 13:02:58.198606 10071 raft_consensus.cc:515] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:58.201196  1678 raft_consensus.cc:2393] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 23: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
W20260709 13:02:58.180981  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 387521us
W20260709 13:02:58.191702  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 50029.2us
W20260709 13:02:58.196604  1570 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:58.196928 10247 raft_consensus.cc:2804] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 13:02:58.208818  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 20 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:02:58.209247  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 88305.3us
I20260709 13:02:58.206925  1718 leader_election.cc:304] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:58.211001  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:58.073012 10171 raft_consensus.cc:515] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:58.215471  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 21 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:58.215922  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:58.216784  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:58.216861  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:58.217597  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:58.212996 10247 raft_consensus.cc:493] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:58.211140 10063 raft_consensus.cc:493] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:02:58.241914 10242 consensus_queue.cc:1048] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 13:02:58.242157 10063 raft_consensus.cc:515] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:58.243883  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:58.244797 10063 leader_election.cc:290] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:58.229202  1671 raft_consensus.cc:2468] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 24.
I20260709 13:02:58.248077 10054 leader_election.cc:290] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 26 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:58.194749  9814 raft_consensus.cc:2804] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 13:02:58.237998  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:58.253093  2331 raft_consensus.cc:2468] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 25.
I20260709 13:02:58.237897  1644 leader_election.cc:304] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:02:58.238582  2328 raft_consensus.cc:1240] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 24 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 23. Current term is 24. Ops: []
I20260709 13:02:58.238750 10247 raft_consensus.cc:3060] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Advancing to term 25
I20260709 13:02:58.239678  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:58.254765  2330 raft_consensus.cc:3060] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:02:58.240499  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 20 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:58.255765  2301 leader_election.cc:304] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
W20260709 13:02:58.261592  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 51359.8us
I20260709 13:02:58.277422 10246 raft_consensus.cc:493] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:58.277586  9979 raft_consensus.cc:493] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:58.278223  9979 raft_consensus.cc:515] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:58.278223 10246 raft_consensus.cc:515] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:58.280913  9979 leader_election.cc:290] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:58.281814  9979 raft_consensus.cc:493] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:58.282176 10246 leader_election.cc:290] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:58.282358  9979 raft_consensus.cc:515] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:58.284106   431 tablet_replica.cc:333] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a: stopping tablet replica
I20260709 13:02:58.284289 10054 raft_consensus.cc:493] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:58.285002  9979 leader_election.cc:290] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:58.284950 10054 raft_consensus.cc:515] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } }
W20260709 13:02:58.286182  1569 connection.cc:256] RPC call timeout handler was delayed by -1.613542644000006s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1635s for a response
I20260709 13:02:58.286784  9979 raft_consensus.cc:493] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:58.288087 10054 leader_election.cc:290] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:58.288597 10171 leader_election.cc:290] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:02:58.288995  1569 connection.cc:256] RPC call timeout handler was delayed by -1.2859426439999879s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19990000000000002s for a response
I20260709 13:02:58.289242 10246 raft_consensus.cc:493] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:58.290267 10246 raft_consensus.cc:515] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:58.291567  1569 leader_election.cc:304] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:02:58.293313 10052 consensus_queue.cc:1048] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 13:02:58.295367 10223 raft_consensus.cc:2749] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 13:02:58.292547 10171 raft_consensus.cc:493] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:02:58.296346 10171 raft_consensus.cc:515] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:58.298147  1569 leader_election.cc:304] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:02:58.299994 10054 raft_consensus.cc:2749] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:02:58.301091 10171 leader_election.cc:290] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:58.302300 10246 leader_election.cc:290] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:58.302492  1569 leader_election.cc:304] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:02:58.288051  9979 raft_consensus.cc:515] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:02:58.303448 10234 log.cc:927] Time spent T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.313s	user 0.000s	sys 0.000s
I20260709 13:02:58.303458 10246 raft_consensus.cc:2749] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 13:02:58.308735  9979 leader_election.cc:290] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:58.309954  1569 leader_election.cc:304] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:02:58.312700  9979 raft_consensus.cc:2749] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 13:02:58.314352  1569 leader_election.cc:304] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:02:58.316509  9979 raft_consensus.cc:2749] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:02:58.331364 10246 consensus_queue.cc:1048] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 13:02:58.271687 10076 raft_consensus.cc:2749] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 13:02:58.286059   431 raft_consensus.cc:2243] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Raft consensus shutting down.
I20260709 13:02:58.254060  9594 raft_consensus.cc:2804] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 13:02:58.332707  9814 raft_consensus.cc:493] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 26 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:02:58.334919 10246 consensus_queue.cc:1243] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer 3b879799899a499c8d409ee9f1277cd1 log is divergent from this leader: its last log entry 21.15 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14
I20260709 13:02:58.188989 10224 raft_consensus.cc:515] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:58.341678  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 86025.7us
W20260709 13:02:58.341480 10047 consensus_meta.cc:287] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.645s	user 0.012s	sys 0.000s
W20260709 13:02:58.342370 10131 consensus_meta.cc:287] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.636s	user 0.008s	sys 0.005s
I20260709 13:02:58.351658   431 raft_consensus.cc:2272] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Raft consensus is shut down!
I20260709 13:02:58.339766  9594 raft_consensus.cc:493] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:58.271487 10226 consensus_queue.cc:1048] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.077s
I20260709 13:02:58.344928  9814 raft_consensus.cc:3060] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 26 FOLLOWER]: Advancing to term 27
I20260709 13:02:58.316356 10167 raft_consensus.cc:515] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:58.361294  2336 raft_consensus.cc:1240] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 25 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 24. Current term is 25. Ops: [24.16-24.16]
I20260709 13:02:58.347311 10245 raft_consensus.cc:2749] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 13:02:58.363350  9594 raft_consensus.cc:3060] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 25 FOLLOWER]: Advancing to term 26
W20260709 13:02:58.369381 10073 consensus_meta.cc:287] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.681s	user 0.010s	sys 0.005s
I20260709 13:02:58.361778  9979 raft_consensus.cc:493] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:58.376811 10237 raft_consensus.cc:493] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:02:58.350526 10047 raft_consensus.cc:515] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:58.381120 10063 raft_consensus.cc:493] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:58.395634  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:58.380226  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 23 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:02:58.392560  1646 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:58.381281  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:58.396850  2337 raft_consensus.cc:2468] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 21.
I20260709 13:02:58.395689  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 21 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:58.373593 10249 raft_consensus.cc:493] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:58.398782  2335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 23 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:58.401033  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 25 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:02:58.402320  1717 leader_election.cc:336] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:02:58.405582  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:02:58.399349  2298 consensus_peers.cc:597] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 13:02:58.403252  1717 leader_election.cc:304] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 13:02:58.410667  1717 leader_election.cc:336] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:02:58.411093  1717 leader_election.cc:304] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:58.413888  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 21 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:58.414291  2333 raft_consensus.cc:2393] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 19: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
W20260709 13:02:58.412497  1717 leader_election.cc:336] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:02:58.415853  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 25 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:58.417150  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:58.417603  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 25 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:58.418198  2329 raft_consensus.cc:2468] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 24.
I20260709 13:02:58.417160  1569 leader_election.cc:304] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:02:58.416114  1717 leader_election.cc:304] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 13:02:58.421960  1717 leader_election.cc:336] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 13:02:58.424672  1717 leader_election.cc:336] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:02:58.425756  1717 leader_election.cc:304] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 13:02:58.415709  2298 consensus_peers.cc:597] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 13:02:58.429430  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 26 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:58.431077  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 21 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:58.425562 10223 raft_consensus.cc:2804] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Leader pre-election won for term 23
W20260709 13:02:58.436857  1717 leader_election.cc:336] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:02:58.414741 10171 raft_consensus.cc:2749] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:02:58.440181  2337 pending_rounds.cc:85] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1: Aborting all ops after (but not including) 14
I20260709 13:02:58.440261  1569 leader_election.cc:304] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:02:58.394754 10073 raft_consensus.cc:515] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:58.442101  2337 pending_rounds.cc:107] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1: Aborting uncommitted NO_OP operation due to leader change: 21.15
I20260709 13:02:58.445142  2337 raft_consensus.cc:2889] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:02:58.422890 10115 raft_consensus.cc:2749] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:02:58.445546  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 26 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:58.449589  1674 raft_consensus.cc:2410] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for term 26 because replica has last-logged OpId of term: 24 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 12.
I20260709 13:02:58.451474  1569 leader_election.cc:304] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:02:58.399333 10237 raft_consensus.cc:515] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:58.401276  9979 raft_consensus.cc:515] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:58.416749  9424 raft_consensus.cc:2749] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:02:58.436829 10083 raft_consensus.cc:2749] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:02:58.461545  1569 leader_election.cc:304] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:02:58.465481 10127 raft_consensus.cc:2804] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 13:02:58.465754 10224 leader_election.cc:290] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 25 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:58.468315  2330 raft_consensus.cc:2468] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 20.
I20260709 13:02:58.437108 10223 raft_consensus.cc:493] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:58.469938  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:58.401844 10063 raft_consensus.cc:515] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:58.409286 10249 raft_consensus.cc:515] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:58.481484  2334 raft_consensus.cc:1240] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 19. Current term is 20. Ops: [19.15-19.15]
I20260709 13:02:58.462635 10216 raft_consensus.cc:2749] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
W20260709 13:02:58.487433  1717 leader_election.cc:336] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:02:58.492470 10249 leader_election.cc:290] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:58.494834  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:58.510916  1569 leader_election.cc:304] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:02:58.476179 10127 raft_consensus.cc:493] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:58.514535  1675 raft_consensus.cc:2468] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 25.
I20260709 13:02:58.482299 10223 raft_consensus.cc:3060] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:02:58.520915  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 23 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:58.498842 10246 raft_consensus.cc:2749] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 13:02:58.535410  1569 leader_election.cc:304] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:02:58.539930  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 22 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:58.527814 10127 raft_consensus.cc:3060] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 24 FOLLOWER]: Advancing to term 25
I20260709 13:02:58.532393 10054 raft_consensus.cc:2749] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 13:02:58.566644 10167 leader_election.cc:290] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 19 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:58.564193 10241 raft_consensus.cc:2749] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
W20260709 13:02:58.583715  1717 leader_election.cc:336] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:02:58.596383 10237 leader_election.cc:290] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:58.604198 10047 leader_election.cc:290] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 19 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:58.607517  9979 leader_election.cc:290] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:02:58.611588  2301 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:58.615954  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 24 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:58.594420 10204 raft_consensus.cc:3060] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:02:58.626251  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
W20260709 13:02:58.628803  1717 leader_election.cc:336] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 13:02:58.637036 10252 log.cc:927] Time spent T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.266s	user 0.002s	sys 0.000s
I20260709 13:02:58.660749  2331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:58.664229  1679 raft_consensus.cc:1240] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 20. Current term is 21. Ops: []
W20260709 13:02:58.671010  9869 consensus_meta.cc:287] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.648s	user 0.012s	sys 0.003s
I20260709 13:02:58.704840  2328 raft_consensus.cc:1240] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 18. Current term is 19. Ops: []
I20260709 13:02:58.706133  1718 leader_election.cc:304] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:58.658174 10073 leader_election.cc:290] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 21 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:58.712054  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 26 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:02:58.717928  1717 leader_election.cc:336] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:02:58.718115  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 23 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:58.720361  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 26 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:58.717768  1718 leader_election.cc:304] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:58.717214 10063 leader_election.cc:290] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:58.725301  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 25 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:58.727195  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 23 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:58.727839  1747 raft_consensus.cc:2468] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 22.
I20260709 13:02:58.728855  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 21 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:02:58.718721  1717 leader_election.cc:336] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:02:58.729933  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:58.685375 10104 raft_consensus.cc:493] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:58.737471  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 25 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:58.734818  1718 leader_election.cc:304] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:58.685169  9853 raft_consensus.cc:493] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
W20260709 13:02:58.731923  1717 leader_election.cc:336] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 19 election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:02:58.711179 10226 raft_consensus.cc:493] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:58.713546 10255 log.cc:927] Time spent T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.051s	user 0.000s	sys 0.000s
I20260709 13:02:58.717577  9869 raft_consensus.cc:493] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:58.749114  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14588fa8cfd6467e989746e326a0ed31" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:02:58.744508  1717 leader_election.cc:336] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 13:02:58.753088  1717 leader_election.cc:336] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:02:58.753324  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:58.721558 10190 raft_consensus.cc:3060] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:02:58.759605  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a93d4b974fee4d02afbaa26d69bfeb07" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 22 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:58.760281  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef5897d71124435db68c88060b095421" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:58.760926  1679 raft_consensus.cc:3060] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:02:58.736182  9424 raft_consensus.cc:2749] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 13:02:58.674415 10247 raft_consensus.cc:515] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:58.766919  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 56004.5us
I20260709 13:02:58.748008 10171 raft_consensus.cc:2804] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Leader pre-election won for term 24
W20260709 13:02:58.771591  1643 consensus_peers.cc:597] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:02:58.773378  1680 raft_consensus.cc:1240] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 19. Current term is 20. Ops: []
W20260709 13:02:58.788028  1643 leader_election.cc:336] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:02:58.796002  1643 leader_election.cc:304] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:02:58.735526 10216 consensus_queue.cc:1059] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 20.17, Next index: 18, Last known committed idx: 15, Time since last communication: 0.032s
I20260709 13:02:58.745992 10230 consensus_queue.cc:1059] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 23.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.030s
I20260709 13:02:58.822278  1749 raft_consensus.cc:1275] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 20 index: 13. Preceding OpId from leader: term: 22 index: 14. (index mismatch)
I20260709 13:02:58.823304  1750 raft_consensus.cc:2393] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 25: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:02:58.748716 10104 raft_consensus.cc:515] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:58.752966  9853 raft_consensus.cc:515] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:58.802547 10171 raft_consensus.cc:493] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
W20260709 13:02:58.853775  1675 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52778 (request call id 3010) took 1557 ms (client timeout 1733 ms). Trace:
W20260709 13:02:58.854251  1675 rpcz_store.cc:269] 0709 13:02:57.296600 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:57.296785 (+   185us) service_pool.cc:224] Handling call
0709 13:02:58.853541 (+1556756us) spinlock_profiling.cc:243] Waited 1.06 s on lock 0x7b5400347e54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:02:58.853672 (+   131us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2449244032}
I20260709 13:02:58.845964  1679 raft_consensus.cc:2468] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 19.
I20260709 13:02:58.764736  9869 raft_consensus.cc:515] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:58.860687  1679 raft_consensus.cc:1240] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 22. Current term is 23. Ops: [22.15-22.15]
I20260709 13:02:58.758044 10226 raft_consensus.cc:515] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:02:58.825599  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 82081.9us
I20260709 13:02:58.847615  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 25 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:58.865157 10071 leader_election.cc:290] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 23 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:58.867647 10239 raft_consensus.cc:3055] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 LEADER]: Stepping down as leader of term 20
I20260709 13:02:58.868089 10239 raft_consensus.cc:740] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:02:58.869016 10239 consensus_queue.cc:260] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 20.17, Last appended by leader: 17, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:58.871258 10226 leader_election.cc:290] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:58.862273  9869 leader_election.cc:290] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 13:02:58.877943  1570 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:58.883522  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:58.900154 10256 log.cc:927] Time spent T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.291s	user 0.003s	sys 0.000s
I20260709 13:02:58.907755 10071 raft_consensus.cc:493] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:58.908439 10071 raft_consensus.cc:515] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:58.920893  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:58.921936  1673 raft_consensus.cc:2468] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 22.
I20260709 13:02:58.803140  9594 raft_consensus.cc:515] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:58.922855  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 24 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:58.842751 10131 raft_consensus.cc:2749] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 13:02:58.928543  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 24 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:58.847949 10230 raft_consensus.cc:3055] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 23 LEADER]: Stepping down as leader of term 23
I20260709 13:02:58.913751 10171 raft_consensus.cc:3060] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Advancing to term 24
I20260709 13:02:58.916707 10239 raft_consensus.cc:3060] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Advancing to term 21
W20260709 13:02:58.921007 10052 raft_consensus.cc:670] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a: failed to trigger leader election: Illegal state: RaftConsensus is not running: State = Stopping
I20260709 13:02:58.940564  1748 raft_consensus.cc:2393] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 26: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
W20260709 13:02:58.942528  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 154577us
I20260709 13:02:58.922633 10226 consensus_queue.cc:1059] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 19.15, Next index: 16, Last known committed idx: 14, Time since last communication: 0.022s
I20260709 13:02:58.933970 10230 raft_consensus.cc:740] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:02:58.968868  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 24 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:58.929960 10260 consensus_queue.cc:1059] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 21.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.041s
I20260709 13:02:58.988875 10071 leader_election.cc:290] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:02:58.999825  1643 leader_election.cc:336] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:02:59.002928  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:59.006124  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:59.009137  1749 raft_consensus.cc:2393] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 22: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:02:58.996600  9979 raft_consensus.cc:3055] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 LEADER]: Stepping down as leader of term 19
I20260709 13:02:59.015548  1751 raft_consensus.cc:1240] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Rejecting Update request from peer 3b879799899a499c8d409ee9f1277cd1 for earlier term 21. Current term is 22. Ops: []
W20260709 13:02:59.020166  1646 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:59.023788 10247 leader_election.cc:290] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 25 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:59.025624  9853 leader_election.cc:290] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:59.041182  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 25 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
W20260709 13:02:59.044894  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 45141.2us
I20260709 13:02:59.019037  9979 raft_consensus.cc:740] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:02:59.052093  1718 leader_election.cc:304] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:02:59.053588  1718 leader_election.cc:304] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:59.055260 10267 raft_consensus.cc:2749] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 13:02:59.056068 10267 raft_consensus.cc:2804] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 13:02:59.056516 10267 raft_consensus.cc:697] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [term 19 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:02:59.057554 10267 consensus_queue.cc:237] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.11, Last appended by leader: 11, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:59.063572  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 25 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:59.094803  1644 leader_election.cc:304] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
W20260709 13:02:59.106825  9814 consensus_meta.cc:287] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.738s	user 0.010s	sys 0.006s
I20260709 13:02:59.031589 10230 consensus_queue.cc:260] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 23.15, Last appended by leader: 15, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:59.115891  1644 leader_election.cc:304] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:02:59.111788  9814 raft_consensus.cc:515] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:59.125530 10277 raft_consensus.cc:2804] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 13:02:59.125769  1750 raft_consensus.cc:1240] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 27 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 26. Current term is 27. Ops: []
I20260709 13:02:59.126027 10277 raft_consensus.cc:493] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 25 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:02:59.126523 10277 raft_consensus.cc:3060] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 25 FOLLOWER]: Advancing to term 26
I20260709 13:02:59.126993  1644 leader_election.cc:304] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:02:59.128161  9853 raft_consensus.cc:2804] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Leader election won for term 20
W20260709 13:02:59.128662  1644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:59.128682  9853 raft_consensus.cc:697] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 20 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:02:59.130003  9853 consensus_queue.cc:237] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.15, Last appended by leader: 15, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:59.135629 10104 leader_election.cc:290] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:59.141693 10230 raft_consensus.cc:3060] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Advancing to term 24
I20260709 13:02:59.155479  9814 leader_election.cc:290] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 27 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:59.156646 10190 raft_consensus.cc:2749] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:02:59.163069  1753 raft_consensus.cc:1240] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 22. Current term is 23. Ops: []
I20260709 13:02:59.163650  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:59.164919  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:59.165639  1749 raft_consensus.cc:2393] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 22: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:02:59.166102  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 23 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:59.167284  1753 raft_consensus.cc:2410] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for term 23 because replica has last-logged OpId of term: 20 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 12.
W20260709 13:02:59.168290 10262 log.cc:927] Time spent T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.339s	user 0.000s	sys 0.002s
W20260709 13:02:59.158613 10223 consensus_meta.cc:287] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.620s	user 0.015s	sys 0.000s
I20260709 13:02:59.112087 10275 raft_consensus.cc:2749] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 13:02:59.135020  9979 consensus_queue.cc:260] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 19.15, Last appended by leader: 15, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:59.161929  1747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:59.229894  1747 raft_consensus.cc:2410] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for term 23 because replica has last-logged OpId of term: 22 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 14.
I20260709 13:02:59.174969 10054 raft_consensus.cc:493] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:59.188900 10277 raft_consensus.cc:515] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:59.173841  9594 leader_election.cc:290] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 26 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:59.237217  9979 raft_consensus.cc:3060] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Advancing to term 20
W20260709 13:02:59.240983 10127 consensus_meta.cc:287] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.661s	user 0.017s	sys 0.000s
I20260709 13:02:59.241483 10127 raft_consensus.cc:515] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:59.235457  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 50829.6us
I20260709 13:02:59.244959   431 tablet_replica.cc:333] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a: stopping tablet replica
I20260709 13:02:59.231316 10054 raft_consensus.cc:515] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:59.252331 10054 leader_election.cc:290] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:59.254449 10277 leader_election.cc:290] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 26 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:59.259790  9594 raft_consensus.cc:493] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:59.260376  9594 raft_consensus.cc:515] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:59.260907   431 raft_consensus.cc:2243] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 24 FOLLOWER]: Raft consensus shutting down.
I20260709 13:02:59.261745   431 raft_consensus.cc:2272] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a [term 24 FOLLOWER]: Raft consensus is shut down!
I20260709 13:02:59.229148 10223 raft_consensus.cc:515] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:59.266108 10223 leader_election.cc:290] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 23 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:59.268368   431 tablet_replica.cc:333] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a: stopping tablet replica
I20260709 13:02:59.268445 10052 raft_consensus.cc:3055] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 LEADER]: Stepping down as leader of term 22
I20260709 13:02:59.275017 10052 raft_consensus.cc:740] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:02:59.276036 10052 consensus_queue.cc:260] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.15, Last appended by leader: 15, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:59.277179 10052 raft_consensus.cc:3060] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:02:59.288566  1680 raft_consensus.cc:1240] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 22. Current term is 23. Ops: [22.15-22.15]
I20260709 13:02:59.295066  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:59.305644  9594 leader_election.cc:290] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:59.308029  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 26 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:59.308686  1671 raft_consensus.cc:3060] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 25 FOLLOWER]: Advancing to term 26
I20260709 13:02:59.311074  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 27 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:59.315555 10127 leader_election.cc:290] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 25 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:59.317071   431 raft_consensus.cc:2243] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 23 FOLLOWER]: Raft consensus shutting down.
I20260709 13:02:59.317970   431 raft_consensus.cc:2272] T 6f6cc48def184bcc907c18c98277187f P 83595ad0fd5940c0a2bfc7c61acd969a [term 23 FOLLOWER]: Raft consensus is shut down!
I20260709 13:02:59.319705  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:02:59.322451  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 620350us
I20260709 13:02:59.325906  1569 leader_election.cc:304] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:02:59.326293  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 23 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:59.327077  1750 raft_consensus.cc:3060] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:02:59.332022 10224 raft_consensus.cc:493] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:02:59.332686 10224 raft_consensus.cc:515] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:59.335592 10225 raft_consensus.cc:493] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:02:59.336227 10225 raft_consensus.cc:515] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:59.344576  2301 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:59.345283  1674 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52766 (request call id 3482) took 1081 ms. Trace:
I20260709 13:02:59.346673 10201 raft_consensus.cc:493] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:59.346616  9588 raft_consensus.cc:493] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:02:59.345734  1674 rpcz_store.cc:276] 0709 13:02:58.263928 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:58.654739 (+390811us) service_pool.cc:224] Handling call
0709 13:02:58.655176 (+   437us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:02:58.664830 (+  9654us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:58.665214 (+   384us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:02:58.720677 (+ 55463us) log.cc:844] Serialized 52 byte log entry
0709 13:02:58.839705 (+119028us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:02:58.921852 (+ 82147us) log.cc:844] Serialized 14 byte log entry
0709 13:02:58.922106 (+   254us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:58.922189 (+    83us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:59.273305 (+351116us) spinlock_profiling.cc:243] Waited 3.99 ms on lock 0x7b18004ab7e0. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f4263075522 00007f42630777c1 00007f4263077779 00007f4263077709 00007f42630776d1 00007f42630769cd 000000000038f354 000000000038f199 00007f426305883a
0709 13:02:59.300714 (+ 27409us) spinlock_profiling.cc:243] Waited 3.54 ms on lock 0x7b18004ab7e0. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f42630753f3 00007f42630777c1 00007f4263077779 00007f4263077709 00007f42630776d1 00007f42630769cd 000000000038f354 000000000038f199 00007f426305883a
0709 13:02:59.345014 (+ 44300us) raft_consensus.cc:1685] finished
0709 13:02:59.345042 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:59.345212 (+   170us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":14090,"prepare.queue_time_us":94364,"prepare.run_cpu_time_us":61,"prepare.run_wall_time_us":63,"raft.queue_time_us":42748,"raft.run_cpu_time_us":239,"raft.run_wall_time_us":10053,"spinlock_wait_cycles":17327104,"thread_start_us":70066,"threads_started":1,"wal-append.queue_time_us":234458}
I20260709 13:02:59.347365 10201 raft_consensus.cc:515] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:59.347605  9588 raft_consensus.cc:515] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:59.351100 10271 log.cc:927] Time spent T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.336s	user 0.001s	sys 0.003s
I20260709 13:02:59.345160 10277 consensus_queue.cc:1059] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 22.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.017s
I20260709 13:02:59.381263 10277 raft_consensus.cc:3055] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 24 LEADER]: Stepping down as leader of term 24
I20260709 13:02:59.381767 10277 raft_consensus.cc:740] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
W20260709 13:02:59.385053  1570 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:59.386570  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:59.387549  1748 raft_consensus.cc:2468] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 23.
W20260709 13:02:59.394615  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:59.419541  1675 raft_consensus.cc:1275] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 22 index: 14. (index mismatch)
I20260709 13:02:59.321477 10171 raft_consensus.cc:515] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:59.382764 10277 consensus_queue.cc:260] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 24.16, Last appended by leader: 16, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:59.427626 10287 log.cc:927] Time spent T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.109s	user 0.000s	sys 0.000s
I20260709 13:02:59.400094 10239 raft_consensus.cc:493] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:59.435763  1675 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52766 (request call id 3497) took 1018 ms. Trace:
I20260709 13:02:59.473630  1675 rpcz_store.cc:276] 0709 13:02:58.417289 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:58.855331 (+438042us) service_pool.cc:224] Handling call
0709 13:02:59.418057 (+562726us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:59.420185 (+  2128us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:59.435486 (+ 15301us) spinlock_profiling.cc:243] Waited 563 ms on lock 0x7b54001f3e50. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661618ad 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:02:59.435677 (+   191us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1293774080}
I20260709 13:02:59.418565 10223 raft_consensus.cc:2749] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
W20260709 13:02:59.463337  1568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:59.465335 10268 raft_consensus.cc:493] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:59.465632  1718 leader_election.cc:304] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:59.475224 10268 raft_consensus.cc:515] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:59.480235 10094 raft_consensus.cc:2804] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 13:02:59.480756 10094 raft_consensus.cc:493] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:59.480417 10239 raft_consensus.cc:515] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:59.481122 10094 raft_consensus.cc:3060] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:02:59.482982 10239 leader_election.cc:290] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:59.400573 10225 leader_election.cc:290] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:02:59.495841  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 170176us
W20260709 13:02:59.497123  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 117593us
I20260709 13:02:59.497843  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:59.498656  2330 raft_consensus.cc:2468] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 24.
I20260709 13:02:59.500236  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 23 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:59.500923  2337 raft_consensus.cc:3060] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Advancing to term 23
I20260709 13:02:59.510632  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:59.498646  1571 leader_election.cc:304] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:02:59.513971  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 25 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:59.514585  2329 raft_consensus.cc:3060] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 24 FOLLOWER]: Advancing to term 25
I20260709 13:02:59.479839  1494 catalog_manager.cc:5697] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 15 to 21, leader changed from af520b67945d443b901bbeca37cc5701 (127.0.107.195) to 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194). New cstate: current_term: 21 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } }
I20260709 13:02:59.519467  2301 leader_election.cc:304] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:02:59.519781  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 27 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:59.514842  1571 leader_election.cc:304] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:02:59.506641 10277 raft_consensus.cc:3060] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 24 FOLLOWER]: Advancing to term 25
I20260709 13:02:59.523804  1676 raft_consensus.cc:1240] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 26 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 25. Current term is 26. Ops: []
I20260709 13:02:59.526880  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:59.524204  1494 catalog_manager.cc:5697] T ef5897d71124435db68c88060b095421 P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 14 to 16, leader changed from 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194) to 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193). New cstate: current_term: 16 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } }
I20260709 13:02:59.531150  1494 catalog_manager.cc:5697] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 16 to 19, leader changed from af520b67945d443b901bbeca37cc5701 (127.0.107.195) to 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194). New cstate: current_term: 19 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } }
I20260709 13:02:59.549657  1494 catalog_manager.cc:5697] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 12 to 20, leader changed from 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194) to 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193). New cstate: current_term: 20 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 13:02:59.553869  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:59.521617 10239 raft_consensus.cc:2749] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:02:59.554816  1673 raft_consensus.cc:2468] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 24.
I20260709 13:02:59.537626  2301 leader_election.cc:304] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:59.509119 10216 consensus_queue.cc:1059] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.023s
I20260709 13:02:59.566460  1718 leader_election.cc:304] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:59.561985  1494 catalog_manager.cc:5697] T a3accbb733bb4cafa69428319664f239 P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 16 to 19, leader changed from 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) to af520b67945d443b901bbeca37cc5701 (127.0.107.195). New cstate: current_term: 19 leader_uuid: "af520b67945d443b901bbeca37cc5701" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 13:02:59.532229 10127 raft_consensus.cc:493] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:59.533756 10123 raft_consensus.cc:2749] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 13:02:59.569298  1494 catalog_manager.cc:5697] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 15 to 19. New cstate: current_term: 19 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 13:02:59.574913  9588 leader_election.cc:290] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:59.574066  1494 catalog_manager.cc:5697] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 15 to 22, leader changed from 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194) to 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193). New cstate: current_term: 22 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: HEALTHY } } }
I20260709 13:02:59.576249 10230 raft_consensus.cc:493] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:59.576911 10230 raft_consensus.cc:515] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:59.576428  1494 catalog_manager.cc:5697] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 23 to 25, leader changed from 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) to af520b67945d443b901bbeca37cc5701 (127.0.107.195). New cstate: current_term: 25 leader_uuid: "af520b67945d443b901bbeca37cc5701" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 13:02:59.536024 10054 raft_consensus.cc:2749] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
W20260709 13:02:59.578575  1499 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35464 (request call id 287) took 104555 ms (client timeout 15000 ms). Trace:
I20260709 13:02:59.579191 10268 leader_election.cc:290] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:02:59.579062  1499 rpcz_store.cc:269] 0709 13:01:14.980525 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:14.984387 (+  3862us) service_pool.cc:224] Handling call
0709 13:02:59.529823 (+104545436us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1482418,"reported_tablets":47}
I20260709 13:02:59.583750 10171 leader_election.cc:290] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 24 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:59.591521  1644 leader_election.cc:304] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:02:59.579185  1494 catalog_manager.cc:5697] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 14 to 16. New cstate: current_term: 16 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: HEALTHY } } }
I20260709 13:02:59.591641  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:02:59.593272  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 264352us
I20260709 13:02:59.593357  1748 raft_consensus.cc:2468] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 26.
I20260709 13:02:59.596483  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:02:59.594569  1494 catalog_manager.cc:5697] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 14 to 16. New cstate: current_term: 16 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 13:02:59.600692  1671 raft_consensus.cc:2468] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 26.
I20260709 13:02:59.603494  2337 raft_consensus.cc:2468] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 23.
W20260709 13:02:59.605997  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 278759us
I20260709 13:02:59.569396 10063 raft_consensus.cc:2804] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 13:02:59.602154  1718 leader_election.cc:304] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:02:59.609990  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:59.612916  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:59.572332  9424 raft_consensus.cc:2804] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 13:02:59.610160  1494 catalog_manager.cc:5697] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 18 to 22. New cstate: current_term: 22 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 13:02:59.616228  2301 leader_election.cc:304] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
W20260709 13:02:59.619948  2298 leader_election.cc:336] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:02:59.620630  2298 leader_election.cc:304] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:02:59.606932  2299 leader_election.cc:304] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:02:59.618757  1494 catalog_manager.cc:5697] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 13 to 16, leader changed from 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) to 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194). New cstate: current_term: 16 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 13:02:59.620002 10237 raft_consensus.cc:493] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:59.625161  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:59.625408 10237 raft_consensus.cc:515] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:59.608546 10076 raft_consensus.cc:2804] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 13:02:59.626488 10076 raft_consensus.cc:493] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:59.626838 10076 raft_consensus.cc:3060] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 24 FOLLOWER]: Advancing to term 25
I20260709 13:02:59.627190 10245 raft_consensus.cc:2804] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 13:02:59.627642 10245 raft_consensus.cc:493] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 25 FOLLOWER]: Starting leader election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:02:59.628028 10245 raft_consensus.cc:3060] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 25 FOLLOWER]: Advancing to term 26
W20260709 13:02:59.631556  2298 leader_election.cc:336] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 18 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:02:59.632123  2298 leader_election.cc:304] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:02:59.623824  1494 catalog_manager.cc:5697] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 13 to 16. New cstate: current_term: 16 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20260709 13:02:59.637910  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 99418.9us
I20260709 13:02:59.607343 10171 raft_consensus.cc:493] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
W20260709 13:02:59.641599  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 113932us
I20260709 13:02:59.642655  1571 leader_election.cc:304] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:02:59.617141 10063 raft_consensus.cc:493] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:59.607519 10127 raft_consensus.cc:515] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:59.608027 10201 leader_election.cc:290] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:59.623003 10268 raft_consensus.cc:2804] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 26 FOLLOWER]: Leader election won for term 26
I20260709 13:02:59.621212  9424 raft_consensus.cc:493] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:02:59.637292 10225 raft_consensus.cc:2749] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
W20260709 13:02:59.679067  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 47553.4us
W20260709 13:02:59.681066  2298 consensus_peers.cc:597] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:02:59.651702 10063 raft_consensus.cc:3060] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Advancing to term 24
W20260709 13:02:59.688489  1570 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:59.665544  1494 catalog_manager.cc:5697] T f6028a708cb344fbb08ab8056921d9e7 P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 19 to 21, leader changed from 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) to 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196). New cstate: current_term: 21 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 13:02:59.689930  1673 raft_consensus.cc:2468] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 24.
I20260709 13:02:59.692607  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 23 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:59.663748  9424 raft_consensus.cc:3060] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Advancing to term 24
I20260709 13:02:59.695730  1718 leader_election.cc:304] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
W20260709 13:02:59.690907 10204 consensus_meta.cc:287] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 1.043s	user 0.019s	sys 0.003s
I20260709 13:02:59.660068 10054 consensus_queue.cc:1059] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 25.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.041s
I20260709 13:02:59.676584 10268 raft_consensus.cc:697] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 26 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:02:59.709313  1494 catalog_manager.cc:5697] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 16 to 20. New cstate: current_term: 20 leader_uuid: "af520b67945d443b901bbeca37cc5701" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } }
I20260709 13:02:59.652407 10239 consensus_queue.cc:1048] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.040s
I20260709 13:02:59.691838 10185 raft_consensus.cc:2749] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:02:59.652213 10171 raft_consensus.cc:515] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:59.723577  2330 raft_consensus.cc:2468] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 22.
W20260709 13:02:59.745203  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 76261.1us
W20260709 13:02:59.734994 10292 log.cc:927] Time spent T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.142s	user 0.001s	sys 0.000s
I20260709 13:02:59.743325  1494 catalog_manager.cc:5697] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 15 to 19, leader changed from 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) to af520b67945d443b901bbeca37cc5701 (127.0.107.195). New cstate: current_term: 19 leader_uuid: "af520b67945d443b901bbeca37cc5701" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 13:02:59.696236 10230 leader_election.cc:290] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:59.731779 10127 leader_election.cc:290] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:59.711539 10115 raft_consensus.cc:2804] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 13:02:59.748891  1750 raft_consensus.cc:2468] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 23.
I20260709 13:02:59.747677   431 tablet_replica.cc:333] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a: stopping tablet replica
I20260709 13:02:59.757239  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:59.758064  2337 raft_consensus.cc:2393] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 24: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:02:59.751212  1494 catalog_manager.cc:5697] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 14 to 15, leader changed from 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) to 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196). New cstate: current_term: 15 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 13:02:59.737819 10076 raft_consensus.cc:515] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:59.759007  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:59.719162 10223 raft_consensus.cc:2749] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260709 13:02:59.797734  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 144829us
I20260709 13:02:59.799378  1494 catalog_manager.cc:5697] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 8 to 22. New cstate: current_term: 22 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 13:02:59.804176  1569 leader_election.cc:304] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:02:59.747617 10123 raft_consensus.cc:493] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:59.805346  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d9ac76dc2c445edb5a25cf9087998bb" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:02:59.807847  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 165309us
I20260709 13:02:59.811177  1571 leader_election.cc:304] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:02:59.802368  1494 catalog_manager.cc:5697] T 1d9ac76dc2c445edb5a25cf9087998bb P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 16 to 18, leader changed from 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) to 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193). New cstate: current_term: 18 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:02:59.813259  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:59.807647  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:59.817978  1671 raft_consensus.cc:1275] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 18 index: 11. Preceding OpId from leader: term: 19 index: 12. (index mismatch)
I20260709 13:02:59.818334  1644 leader_election.cc:304] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:02:59.814098  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:59.820863  2332 raft_consensus.cc:3060] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Advancing to term 24
I20260709 13:02:59.819227  2330 raft_consensus.cc:2468] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 22.
I20260709 13:02:59.822563  1494 catalog_manager.cc:5697] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 18 to 19, leader changed from 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) to 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194). New cstate: current_term: 19 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } }
I20260709 13:02:59.827286  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:59.825871  1494 catalog_manager.cc:5697] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 12 to 22. New cstate: current_term: 22 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:02:59.830029  2337 raft_consensus.cc:1275] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 18 index: 14. Preceding OpId from leader: term: 20 index: 16. (index mismatch)
I20260709 13:02:59.816860 10123 raft_consensus.cc:515] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:59.832948  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 22 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:59.833398 10171 leader_election.cc:290] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:59.831058 10204 raft_consensus.cc:493] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:59.835616 10204 raft_consensus.cc:515] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:59.837836 10204 leader_election.cc:290] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:02:59.839599  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:02:59.844182   431 raft_consensus.cc:2243] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 23 FOLLOWER]: Raft consensus shutting down.
I20260709 13:02:59.801146 10115 raft_consensus.cc:697] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 24 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:02:59.768586 10295 raft_consensus.cc:493] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:02:59.849058  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:02:59.851534  1571 leader_election.cc:304] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
W20260709 13:02:59.854959  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 158933us
I20260709 13:02:59.854385  1494 catalog_manager.cc:5697] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 15 to 18, leader changed from 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) to 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196). New cstate: current_term: 18 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
W20260709 13:02:59.847409 10052 consensus_meta.cc:287] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.570s	user 0.018s	sys 0.006s
I20260709 13:02:59.864442   431 raft_consensus.cc:2272] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [term 23 FOLLOWER]: Raft consensus is shut down!
I20260709 13:02:59.865272  2329 raft_consensus.cc:2468] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 25.
I20260709 13:02:59.858212  1494 catalog_manager.cc:5697] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 14 to 16. New cstate: current_term: 16 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:02:59.893002  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:02:59.894196  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:02:59.963454  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 94648.8us
I20260709 13:02:59.965232  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:02:59.859601 10237 leader_election.cc:290] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:02:59.850090 10298 raft_consensus.cc:2804] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 23 FOLLOWER]: Leader election won for term 23
W20260709 13:02:59.960377  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 82365.5us
I20260709 13:02:59.982838 10247 consensus_queue.cc:1048] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 13:02:59.852234 10268 consensus_queue.cc:237] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 24.12, Last appended by leader: 12, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:59.990315 10076 leader_election.cc:290] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 25 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:02:59.992197 10063 raft_consensus.cc:515] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:59.859390 10295 raft_consensus.cc:515] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:59.994473 10052 consensus_queue.cc:1048] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.018s
I20260709 13:02:59.867529 10230 consensus_queue.cc:1048] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.106s
I20260709 13:02:59.996117 10242 raft_consensus.cc:493] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:59.940106 10245 raft_consensus.cc:515] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:02:59.996694 10242 raft_consensus.cc:515] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:02:59.998981 10115 consensus_queue.cc:237] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 23.17, Last appended by leader: 17, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:02:59.966581  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 111780us
I20260709 13:03:00.002158  1571 leader_election.cc:304] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:02:59.971647  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:03:00.003728  2337 raft_consensus.cc:2393] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 24: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:02:59.996704 10295 leader_election.cc:290] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:03:00.006914  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:03:00.007917  1675 raft_consensus.cc:2393] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 26: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:02:59.971660  1494 catalog_manager.cc:5697] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 16 to 18. New cstate: current_term: 18 leader_uuid: "af520b67945d443b901bbeca37cc5701" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } }
I20260709 13:02:59.876669  9588 raft_consensus.cc:2749] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 13:02:59.867353 10304 raft_consensus.cc:2804] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Leader pre-election won for term 22
W20260709 13:02:59.994236 10094 consensus_meta.cc:287] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.513s	user 0.007s	sys 0.007s
I20260709 13:03:00.015476  2329 raft_consensus.cc:2393] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 24: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:03:00.016265  2332 raft_consensus.cc:2468] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 24.
I20260709 13:03:00.016533  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:03:00.021026  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:03:00.046705  2330 raft_consensus.cc:2393] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 24: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:03:00.027907  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cc48def184bcc907c18c98277187f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:03:00.049197  1748 raft_consensus.cc:3060] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Advancing to term 24
I20260709 13:03:00.028434  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:03:00.050695  2337 raft_consensus.cc:3060] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 24 FOLLOWER]: Advancing to term 25
W20260709 13:03:00.041990  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 349890us
I20260709 13:03:00.038736  1494 catalog_manager.cc:5697] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a reported cstate change: term changed from 13 to 16, leader changed from 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) to 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193). New cstate: current_term: 16 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 13:03:00.054692  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:03:00.055127  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 239974us
I20260709 13:02:59.964874 10303 raft_consensus.cc:2749] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:03:00.081804  1569 leader_election.cc:304] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:03:00.083792  2330 raft_consensus.cc:1240] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 18. Current term is 19. Ops: [18.14-18.14]
I20260709 13:03:00.089078  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:03:00.093720  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:03:00.094417  1671 raft_consensus.cc:2468] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 24.
I20260709 13:03:00.103351  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:03:00.104151  1678 raft_consensus.cc:2393] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 26: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
W20260709 13:03:00.037454 10300 log.cc:927] Time spent T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.063s	user 0.000s	sys 0.000s
I20260709 13:03:00.105952  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:02:59.976132 10305 raft_consensus.cc:2749] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:03:00.020298 10304 raft_consensus.cc:493] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Starting leader election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:03:00.036475 10245 leader_election.cc:290] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 26 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:03:00.023044 10063 leader_election.cc:290] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 24 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:03:00.040414 10224 leader_election.cc:290] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:03:00.039337 10306 raft_consensus.cc:493] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:03:00.141139  1748 raft_consensus.cc:2468] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 24.
I20260709 13:03:00.052510 10094 raft_consensus.cc:515] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:03:00.107612 10298 raft_consensus.cc:697] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 23 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:03:00.115274 10242 leader_election.cc:290] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:03:00.120599 10304 raft_consensus.cc:3060] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:03:00.086460  1491 maintenance_manager.cc:643] P 919f6392d4f44fd68987e2c5e7ec37ca: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 2.278s	user 0.132s	sys 0.083s Metrics: {"cfile_cache_miss":9,"cfile_cache_miss_bytes":26841,"cfile_init":4,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"dirs.queue_time_us":461523,"dirs.run_cpu_time_us":558,"dirs.run_wall_time_us":1080,"lbm_read_time_us":21223,"lbm_reads_10-100_ms":1,"lbm_reads_lt_1ms":24,"lbm_write_time_us":1788,"lbm_writes_lt_1ms":11,"peak_mem_usage":279555,"reinsert_count":0,"thread_start_us":435714,"threads_started":3,"update_count":226,"wal-append.queue_time_us":63002}
I20260709 13:03:00.144301 10201 raft_consensus.cc:493] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
W20260709 13:03:00.164602  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 187084us
I20260709 13:03:00.133643 10311 raft_consensus.cc:2749] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 13:03:00.150506 10303 raft_consensus.cc:2804] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 25 FOLLOWER]: Leader election won for term 25
I20260709 13:03:00.151374 10076 raft_consensus.cc:493] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:00.222968 10076 raft_consensus.cc:515] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:03:00.224984 10239 raft_consensus.cc:3055] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 25 LEADER]: Stepping down as leader of term 25
I20260709 13:03:00.225448 10239 raft_consensus.cc:740] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:03:00.225605 10076 leader_election.cc:290] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:03:00.166668 10052 consensus_queue.cc:1059] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 17.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.015s
W20260709 13:03:00.187122  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 69485.7us
I20260709 13:03:00.227414 10201 raft_consensus.cc:515] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:03:00.228044 10094 leader_election.cc:290] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 23 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:03:00.145565 10306 raft_consensus.cc:515] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:03:00.230494 10306 leader_election.cc:290] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:03:00.230576 10201 leader_election.cc:290] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:03:00.231437  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:03:00.232260  2330 raft_consensus.cc:2468] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 24.
I20260709 13:03:00.188371  2337 raft_consensus.cc:2468] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 25.
I20260709 13:03:00.179435 10123 leader_election.cc:290] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:03:00.193475  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 191431us
I20260709 13:03:00.237455  1571 leader_election.cc:304] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:03:00.211927  1749 raft_consensus.cc:2468] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 26.
I20260709 13:03:00.206934   431 tablet_replica.cc:333] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a: stopping tablet replica
I20260709 13:03:00.221678  1492 maintenance_manager.cc:419] P 919f6392d4f44fd68987e2c5e7ec37ca: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260709 13:03:00.240439  1491 maintenance_manager.cc:643] P 919f6392d4f44fd68987e2c5e7ec37ca: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {}
I20260709 13:03:00.206375  9979 raft_consensus.cc:493] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:00.238705 10054 raft_consensus.cc:493] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:03:00.244519  1571 raft_consensus.cc:2686] T 1f1329a6c1e442bcba5945a2d1e65e85 P 83595ad0fd5940c0a2bfc7c61acd969a: unable to run election callback: Service unavailable: Thread pool token was shut down
I20260709 13:03:00.233551 10303 raft_consensus.cc:697] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 25 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:03:00.250694  1571 leader_election.cc:304] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:03:00.252102  1571 leader_election.cc:304] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
W20260709 13:03:00.242501  9424 consensus_meta.cc:287] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.526s	user 0.006s	sys 0.007s
I20260709 13:03:00.246217 10054 raft_consensus.cc:515] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:03:00.254673 10242 consensus_queue.cc:1059] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 26.13, Next index: 14, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 13:03:00.256582 10321 raft_consensus.cc:3060] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 25 FOLLOWER]: Advancing to term 26
I20260709 13:03:00.262218 10076 raft_consensus.cc:493] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:00.262809 10076 raft_consensus.cc:515] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:03:00.263293 10298 consensus_queue.cc:237] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 21.17, Last appended by leader: 17, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:03:00.267009 10303 consensus_queue.cc:237] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 22.16, Last appended by leader: 16, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:03:00.267494  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:03:00.268296  2337 raft_consensus.cc:2468] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 25.
I20260709 13:03:00.268749 10076 leader_election.cc:290] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:03:00.270148 10242 raft_consensus.cc:2749] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:03:00.248565  9979 raft_consensus.cc:515] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:03:00.279070  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 198564us
W20260709 13:03:00.280221  2298 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 102 similar messages]
I20260709 13:03:00.281634 10304 raft_consensus.cc:515] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:03:00.290042  2301 leader_election.cc:304] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:03:00.291126  2301 leader_election.cc:304] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:03:00.292088 10123 raft_consensus.cc:2804] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 13:03:00.292696 10123 raft_consensus.cc:697] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 24 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:03:00.226367 10239 consensus_queue.cc:260] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 25.18, Last appended by leader: 18, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:03:00.294687 10115 raft_consensus.cc:493] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:00.295873 10239 raft_consensus.cc:3060] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 25 FOLLOWER]: Advancing to term 26
I20260709 13:03:00.295408 10115 raft_consensus.cc:515] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:03:00.296402 10201 raft_consensus.cc:2804] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 26 FOLLOWER]: Leader election won for term 26
I20260709 13:03:00.297006 10201 raft_consensus.cc:697] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 26 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:03:00.297988 10201 consensus_queue.cc:237] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 25.18, Last appended by leader: 18, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:03:00.300634 10312 raft_consensus.cc:3055] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 26 LEADER]: Stepping down as leader of term 26
I20260709 13:03:00.301162 10312 raft_consensus.cc:740] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:03:00.303180  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:03:00.303892  1746 raft_consensus.cc:2468] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 24.
I20260709 13:03:00.303892 10268 raft_consensus.cc:493] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:03:00.305378 10268 raft_consensus.cc:515] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:03:00.306771  9424 raft_consensus.cc:515] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:03:00.307781 10268 leader_election.cc:290] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:03:00.307834  1752 raft_consensus.cc:3060] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Advancing to term 25
I20260709 13:03:00.298581 10115 leader_election.cc:290] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:03:00.318710  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:03:00.302172 10312 consensus_queue.cc:260] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 26.13, Last appended by leader: 13, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:03:00.359589 10312 raft_consensus.cc:3060] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 26 FOLLOWER]: Advancing to term 27
I20260709 13:03:00.318912  1646 leader_election.cc:304] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:03:00.352451  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:03:00.364362  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 49907.5us
I20260709 13:03:00.293884 10123 consensus_queue.cc:237] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.15, Last appended by leader: 15, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:03:00.356202 10216 raft_consensus.cc:493] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:00.369477  1748 raft_consensus.cc:2468] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 25.
I20260709 13:03:00.372094  1646 leader_election.cc:304] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:03:00.367483  1752 raft_consensus.cc:2468] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 25.
I20260709 13:03:00.364184 10302 raft_consensus.cc:2804] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 13:03:00.396369 10310 raft_consensus.cc:493] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:03:00.397023 10310 raft_consensus.cc:515] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:03:00.400784  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:03:00.394016 10294 raft_consensus.cc:493] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:03:00.402933 10294 raft_consensus.cc:515] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:03:00.405308 10294 leader_election.cc:290] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:03:00.406632 10310 leader_election.cc:290] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:03:00.373787 10216 raft_consensus.cc:515] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:03:00.391659 10302 raft_consensus.cc:493] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:00.411204 10302 raft_consensus.cc:3060] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 24 FOLLOWER]: Advancing to term 25
I20260709 13:03:00.412068  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:03:00.388135 10076 raft_consensus.cc:2804] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 13:03:00.408743 10309 raft_consensus.cc:3055] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 LEADER]: Stepping down as leader of term 18
I20260709 13:03:00.414759 10076 raft_consensus.cc:493] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:00.414850 10309 raft_consensus.cc:740] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:03:00.391386 10286 raft_consensus.cc:493] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:03:00.415748 10286 raft_consensus.cc:515] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:03:00.410425 10054 leader_election.cc:290] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:03:00.419292 10247 raft_consensus.cc:493] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:03:00.419394  9979 leader_election.cc:290] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:03:00.419902 10247 raft_consensus.cc:515] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:03:00.415251 10076 raft_consensus.cc:3060] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 25 FOLLOWER]: Advancing to term 26
I20260709 13:03:00.426842  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6028a708cb344fbb08ab8056921d9e7" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:03:00.426695  1677 raft_consensus.cc:2468] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 22.
I20260709 13:03:00.433956  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:03:00.439541  1752 raft_consensus.cc:2468] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 22.
W20260709 13:03:00.439903  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 223310us
I20260709 13:03:00.436511  1644 leader_election.cc:304] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:03:00.438476  2299 leader_election.cc:304] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:03:00.422257 10247 leader_election.cc:290] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:03:00.445194  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:03:00.446664  2299 leader_election.cc:304] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:03:00.448254  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 21 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:03:00.453459  1673 raft_consensus.cc:2393] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 21: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:03:00.452080  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:03:00.442368  2301 leader_election.cc:304] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:03:00.454118  1754 raft_consensus.cc:2468] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 22.
I20260709 13:03:00.458563   431 raft_consensus.cc:2243] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 26 FOLLOWER]: Raft consensus shutting down.
I20260709 13:03:00.457209  1718 leader_election.cc:304] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:03:00.461717  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:03:00.436969  9979 raft_consensus.cc:493] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:00.418392 10286 leader_election.cc:290] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:03:00.465386  2301 leader_election.cc:304] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:03:00.442632  9424 leader_election.cc:290] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 24 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:03:00.463084  1718 leader_election.cc:304] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:03:00.469455  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:03:00.448109 10063 raft_consensus.cc:2749] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 13:03:00.464061 10047 raft_consensus.cc:2804] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 13:03:00.458974 10230 raft_consensus.cc:2804] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 25 FOLLOWER]: Leader election won for term 25
I20260709 13:03:00.484534  1718 leader_election.cc:304] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:03:00.490285  1718 leader_election.cc:304] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:03:00.464516 10224 raft_consensus.cc:2804] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 13:03:00.465451 10247 raft_consensus.cc:2749] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 13:03:00.495940  1718 leader_election.cc:304] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:03:00.484100 10047 raft_consensus.cc:493] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:00.496665  9424 raft_consensus.cc:2804] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 13:03:00.480413 10201 raft_consensus.cc:2804] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 13:03:00.483304 10321 raft_consensus.cc:2749] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 13:03:00.504197  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 23 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:03:00.489581 10115 raft_consensus.cc:2749] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:03:00.469981  9979 raft_consensus.cc:515] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:03:00.493597 10224 raft_consensus.cc:493] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:03:00.517598  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 77839us
I20260709 13:03:00.500149 10307 raft_consensus.cc:2749] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:03:00.500821  9424 raft_consensus.cc:493] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:03:00.520848  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:03:00.521402  9424 raft_consensus.cc:3060] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:03:00.524957  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 27 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:03:00.525746   431 raft_consensus.cc:2272] T f8ab62ffb73246dc9bcf39f7bee51147 P 83595ad0fd5940c0a2bfc7c61acd969a [term 26 FOLLOWER]: Raft consensus is shut down!
I20260709 13:03:00.526190  1746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:03:00.526646  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 22 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:03:00.527477  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:03:00.528132  1671 raft_consensus.cc:2468] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 22.
I20260709 13:03:00.528299  1750 raft_consensus.cc:1240] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 22. Current term is 23. Ops: [22.14-22.14]
I20260709 13:03:00.499627 10047 raft_consensus.cc:3060] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 24 FOLLOWER]: Advancing to term 25
I20260709 13:03:00.532876  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e527450b1d6c4cb79e9f6d3c87d4ad23" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:03:00.534315  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:03:00.536406  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:03:00.538270  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:03:00.539413  2337 raft_consensus.cc:2368] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 24. Current term is 25.
W20260709 13:03:00.508875 10329 log.cc:927] Time spent T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.125s	user 0.001s	sys 0.000s
I20260709 13:03:00.507283 10094 raft_consensus.cc:2804] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 13:03:00.502187 10230 raft_consensus.cc:697] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 25 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:03:00.503674 10201 raft_consensus.cc:493] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
W20260709 13:03:00.543704  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 140724us
I20260709 13:03:00.547438  1677 raft_consensus.cc:1275] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 24 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 23 index: 17. Preceding OpId from leader: term: 24 index: 18. (index mismatch)
W20260709 13:03:00.560102  2298 consensus_peers.cc:597] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260709 13:03:00.561172  2298 leader_election.cc:336] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:03:00.561635  2298 leader_election.cc:304] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:03:00.576222 10302 raft_consensus.cc:515] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:03:00.577766  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:03:00.578786  1752 raft_consensus.cc:2393] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 22: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:03:00.584275 10302 leader_election.cc:290] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 25 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:03:00.586968  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:03:00.593016  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5108457796a949b7b753ab22b575b7a0" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:03:00.603932  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:03:00.604638  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:03:00.605567  2337 raft_consensus.cc:3060] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 24 FOLLOWER]: Advancing to term 25
W20260709 13:03:00.556347  1643 leader_election.cc:336] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:03:00.530795 10224 raft_consensus.cc:3060] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:03:00.611318  1643 leader_election.cc:304] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:03:00.614116  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 25 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
W20260709 13:03:00.615711  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 54590.8us
W20260709 13:03:00.622958  2298 leader_election.cc:336] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:03:00.623581  2298 leader_election.cc:304] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:03:00.604665  1675 raft_consensus.cc:2468] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 23.
I20260709 13:03:00.467522 10309 consensus_queue.cc:260] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:03:00.634652 10267 raft_consensus.cc:493] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
W20260709 13:03:00.627733  1571 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:00.647243  1754 raft_consensus.cc:2393] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 25: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:03:00.647149  2337 raft_consensus.cc:2468] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 25.
I20260709 13:03:00.541550 10268 raft_consensus.cc:493] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:03:00.875916 10230 consensus_queue.cc:237] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 23.17, Last appended by leader: 17, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:03:00.656875 10306 raft_consensus.cc:2749] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:03:00.752166 10304 leader_election.cc:290] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 22 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:03:00.772272 10324 log.cc:927] Time spent T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.272s	user 0.000s	sys 0.001s
I20260709 13:03:00.619474 10054 raft_consensus.cc:493] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:00.612586 10094 raft_consensus.cc:697] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 24 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
W20260709 13:03:00.773528 10330 log.cc:927] Time spent T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.257s	user 0.000s	sys 0.000s
I20260709 13:03:00.557235 10201 raft_consensus.cc:3060] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:03:00.622781 10303 raft_consensus.cc:493] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:03:00.619673 10321 raft_consensus.cc:493] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:03:00.980014  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 326966us
I20260709 13:03:00.972455 10063 raft_consensus.cc:2749] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:03:01.008917 10303 raft_consensus.cc:515] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:03:01.015030  1571 leader_election.cc:304] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:03:01.013798  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 20 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:03:01.008209  9920 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.504s	user 0.000s	sys 0.030s
I20260709 13:03:01.005597 10304 raft_consensus.cc:493] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:01.050650 10094 consensus_queue.cc:237] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 23.15, Last appended by leader: 15, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:03:00.985113 10267 raft_consensus.cc:515] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:03:01.074024  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 517857us
I20260709 13:03:01.029304 10216 leader_election.cc:290] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:03:01.008819 10321 raft_consensus.cc:515] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:03:00.991143 10268 raft_consensus.cc:515] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:03:00.996376 10054 raft_consensus.cc:515] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:03:01.076112  1643 leader_election.cc:336] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:03:01.092757  1643 leader_election.cc:304] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 13:03:01.090685  1673 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.539s	user 0.018s	sys 0.021s
W20260709 13:03:01.095134  1673 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.544s	user 0.018s	sys 0.021s
I20260709 13:03:01.066597 10304 raft_consensus.cc:515] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:03:01.118357  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 495553us
I20260709 13:03:01.097111 10302 raft_consensus.cc:2749] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 13:03:01.100514  9853 raft_consensus.cc:2749] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:03:01.096817  9979 leader_election.cc:290] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:03:01.109441 10239 raft_consensus.cc:2749] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
W20260709 13:03:01.129694  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 114826us
W20260709 13:03:01.124195  2298 leader_election.cc:336] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 19 election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:03:01.130663  2298 leader_election.cc:304] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:03:01.124676   431 tablet_replica.cc:333] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a: stopping tablet replica
I20260709 13:03:01.132526  1571 leader_election.cc:304] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
W20260709 13:03:01.131944  2298 leader_election.cc:336] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:03:01.133912 10341 raft_consensus.cc:2749] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 13:03:01.137859 10216 raft_consensus.cc:2749] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260709 13:03:01.138154 10076 consensus_meta.cc:287] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d: Time spent flushing consensus metadata: real 0.714s	user 0.013s	sys 0.006s
I20260709 13:03:01.138563  1571 leader_election.cc:304] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:03:01.138684 10076 raft_consensus.cc:515] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:03:01.139348  9979 raft_consensus.cc:2749] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:03:01.141673  9979 consensus_queue.cc:1059] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 20.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 13:03:01.142900  1571 leader_election.cc:304] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:03:01.121753 10303 leader_election.cc:290] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:03:01.147575  1571 leader_election.cc:304] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
W20260709 13:03:01.135973 10047 consensus_meta.cc:287] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.607s	user 0.012s	sys 0.004s
W20260709 13:03:01.145496 10312 consensus_meta.cc:287] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a: Time spent flushing consensus metadata: real 0.785s	user 0.015s	sys 0.004s
I20260709 13:03:01.155005 10303 raft_consensus.cc:2749] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:03:01.162189 10304 leader_election.cc:290] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:03:01.146627 10216 raft_consensus.cc:3055] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 LEADER]: Stepping down as leader of term 22
I20260709 13:03:01.149603  9979 raft_consensus.cc:2749] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 13:03:01.173180  1747 raft_consensus.cc:1275] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 21 index: 17. Preceding OpId from leader: term: 23 index: 18. (index mismatch)
I20260709 13:03:01.171871  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:03:01.154605 10290 raft_consensus.cc:493] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:03:01.174858  1751 raft_consensus.cc:3060] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 25 FOLLOWER]: Advancing to term 26
I20260709 13:03:01.156333 10322 raft_consensus.cc:493] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:01.167594 10303 raft_consensus.cc:493] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:01.210098  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 27 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:03:01.213795  1754 raft_consensus.cc:2410] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for term 27 because replica has last-logged OpId of term: 25 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 17.
I20260709 13:03:01.169978 10216 raft_consensus.cc:740] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:03:01.222151   431 raft_consensus.cc:2243] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Raft consensus shutting down.
I20260709 13:03:01.154372 10047 raft_consensus.cc:515] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:03:01.219707 10307 raft_consensus.cc:493] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:03:01.271003  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 176689us
W20260709 13:03:01.281759  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 253513us
I20260709 13:03:01.277004  1678 raft_consensus.cc:1275] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 26 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 24 index: 17. Preceding OpId from leader: term: 26 index: 19. (index mismatch)
I20260709 13:03:01.182292 10290 raft_consensus.cc:515] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:03:01.265245 10347 raft_consensus.cc:493] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:03:01.287388  1643 leader_election.cc:336] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:03:01.208199 10322 raft_consensus.cc:515] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:03:01.224882 10303 raft_consensus.cc:515] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:03:01.298843  1643 leader_election.cc:304] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
W20260709 13:03:01.302408  1643 leader_election.cc:336] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 13:03:01.289291  9424 consensus_meta.cc:287] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701: Time spent flushing consensus metadata: real 0.767s	user 0.012s	sys 0.004s
I20260709 13:03:01.303923  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 21 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:03:01.304399 10344 consensus_queue.cc:1048] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 16, Time since last communication: 0.007s
I20260709 13:03:01.304268  9424 raft_consensus.cc:515] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:03:01.306746  9424 leader_election.cc:290] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 23 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:03:01.303658 10301 raft_consensus.cc:2749] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
W20260709 13:03:01.309557  1643 leader_election.cc:336] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:03:01.310174  1643 leader_election.cc:304] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:03:01.301151  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:03:01.311079 10304 raft_consensus.cc:2749] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
W20260709 13:03:01.312553  1643 leader_election.cc:336] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:03:01.313050  1643 leader_election.cc:304] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:03:01.313788  1751 raft_consensus.cc:2468] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 26.
I20260709 13:03:01.313921 10304 raft_consensus.cc:2749] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
W20260709 13:03:01.315059  1643 leader_election.cc:336] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 21 election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:03:01.316350  1643 leader_election.cc:304] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:03:01.317317 10294 raft_consensus.cc:2749] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
W20260709 13:03:01.318960  1643 leader_election.cc:336] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:03:01.319855  1643 leader_election.cc:304] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:03:01.320801 10294 raft_consensus.cc:2749] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
W20260709 13:03:01.322259  1643 leader_election.cc:336] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:03:01.322798  1643 leader_election.cc:304] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:03:01.323153  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 21 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:03:01.323854 10294 raft_consensus.cc:3060] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Advancing to term 19
W20260709 13:03:01.331286  1643 leader_election.cc:336] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:03:01.335072  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 22 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:03:01.299623 10335 log.cc:927] Time spent T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.188s	user 0.001s	sys 0.000s
I20260709 13:03:01.277421 10268 leader_election.cc:290] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:03:01.333632  1643 leader_election.cc:304] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
W20260709 13:03:01.342254  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 52981.6us
I20260709 13:03:01.338343   431 raft_consensus.cc:2272] T 14588fa8cfd6467e989746e326a0ed31 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Raft consensus is shut down!
I20260709 13:03:01.335834  1672 raft_consensus.cc:2468] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 21.
I20260709 13:03:01.330417 10321 leader_election.cc:290] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:03:01.347669  1643 leader_election.cc:336] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 20 election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 13:03:01.356894  1643 leader_election.cc:336] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 13:03:01.358108  1643 leader_election.cc:336] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:03:01.277211 10307 raft_consensus.cc:515] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:03:01.357018  1718 leader_election.cc:304] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:03:01.358783  1643 leader_election.cc:304] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:03:01.339458 10267 leader_election.cc:290] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:03:01.361804  1643 leader_election.cc:336] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 23 election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 13:03:01.346619 10224 consensus_meta.cc:287] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1: Time spent flushing consensus metadata: real 0.688s	user 0.015s	sys 0.001s
W20260709 13:03:01.345423 10336 log.cc:927] Time spent T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.216s	user 0.001s	sys 0.001s
I20260709 13:03:01.364980  1643 leader_election.cc:304] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:03:01.331151 10347 raft_consensus.cc:515] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:03:01.374747   431 tablet_replica.cc:333] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a: stopping tablet replica
I20260709 13:03:01.374992 10306 raft_consensus.cc:493] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:01.375676 10306 raft_consensus.cc:515] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:03:01.376281   431 raft_consensus.cc:2243] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 26 FOLLOWER]: Raft consensus shutting down.
I20260709 13:03:01.376718 10347 leader_election.cc:290] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:03:01.377194   431 pending_rounds.cc:70] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a: Trying to abort 1 pending ops.
I20260709 13:03:01.377311 10347 raft_consensus.cc:493] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:01.377534   431 pending_rounds.cc:77] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a: Aborting op as it isn't in flight: id { term: 24 index: 18 } timestamp: 7305634453550555136 op_type: NO_OP noop_request { }
I20260709 13:03:01.377967   431 raft_consensus.cc:2889] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 26 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260709 13:03:01.378368   431 raft_consensus.cc:2272] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [term 26 FOLLOWER]: Raft consensus is shut down!
I20260709 13:03:01.379968 10303 leader_election.cc:290] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:03:01.296298 10216 consensus_queue.cc:260] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 22.14, Last appended by leader: 14, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:03:01.381592 10216 raft_consensus.cc:3060] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Advancing to term 23
W20260709 13:03:01.385794 10338 log.cc:927] Time spent T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.118s	user 0.000s	sys 0.000s
I20260709 13:03:01.349020 10354 raft_consensus.cc:493] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:01.387506 10354 raft_consensus.cc:515] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:03:01.389770  9853 raft_consensus.cc:2749] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 13:03:01.391258 10047 leader_election.cc:290] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 25 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:03:01.363163 10321 raft_consensus.cc:493] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:01.392619 10321 raft_consensus.cc:515] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:03:01.397015 10307 leader_election.cc:290] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:03:01.389917 10354 leader_election.cc:290] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:03:01.377842 10347 raft_consensus.cc:515] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:03:01.408700  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 51793.8us
I20260709 13:03:01.355878 10351 raft_consensus.cc:2749] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:03:01.410979  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87b86d46388a41cd90374c203cb27914" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:03:01.412395  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:03:01.363735 10310 raft_consensus.cc:2749] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 13:03:01.413241  2330 raft_consensus.cc:2368] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 24. Current term is 25.
W20260709 13:03:01.415017  2298 leader_election.cc:336] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:03:01.416802  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3accbb733bb4cafa69428319664f239" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:03:01.417189  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:03:01.417596  2332 raft_consensus.cc:3060] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:03:01.418056  2337 raft_consensus.cc:2393] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 25: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:03:01.418677  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:03:01.415546  2298 leader_election.cc:304] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
W20260709 13:03:01.421470  2298 leader_election.cc:336] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:03:01.421921  2298 leader_election.cc:304] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:03:01.378319 10306 leader_election.cc:290] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:03:01.404233 10246 raft_consensus.cc:493] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:01.425447  1718 leader_election.cc:400] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 24 election: Vote denied by peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625) with higher term. Message: Invalid argument: T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 24. Current term is 25.
W20260709 13:03:01.427688  2298 leader_election.cc:336] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 24 election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:03:01.428654  1718 leader_election.cc:403] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 24 election: Cancelling election due to peer responding with higher term
W20260709 13:03:01.429808  2298 leader_election.cc:336] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:03:01.430297  2298 leader_election.cc:304] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:03:01.414758 10268 raft_consensus.cc:2804] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 13:03:01.440680 10268 raft_consensus.cc:493] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:01.441138 10268 raft_consensus.cc:3060] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:03:01.447696 10115 raft_consensus.cc:2749] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 25 FOLLOWER]: Leader election lost for term 24. Reason: Vote denied by peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625) with higher term. Message: Invalid argument: T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 24. Current term is 25.
I20260709 13:03:01.451512 10322 leader_election.cc:290] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:03:01.453630 10290 leader_election.cc:290] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:03:01.455598   431 tablet_replica.cc:333] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a: stopping tablet replica
I20260709 13:03:01.456760   431 raft_consensus.cc:2243] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Raft consensus shutting down.
I20260709 13:03:01.456954 10294 raft_consensus.cc:2749] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:03:01.457458   431 pending_rounds.cc:70] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a: Trying to abort 1 pending ops.
I20260709 13:03:01.457753   431 pending_rounds.cc:77] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a: Aborting op as it isn't in flight: id { term: 19 index: 10 } timestamp: 7305634437059641344 op_type: NO_OP noop_request { }
I20260709 13:03:01.458150   431 raft_consensus.cc:2889] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260709 13:03:01.459985 10076 leader_election.cc:290] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 26 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:03:01.461329 10047 raft_consensus.cc:3060] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:03:01.400036 10224 raft_consensus.cc:515] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:03:01.439951 10321 leader_election.cc:290] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:03:01.425642 10245 raft_consensus.cc:2749] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 13:03:01.464380   431 raft_consensus.cc:2272] T a93d4b974fee4d02afbaa26d69bfeb07 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Raft consensus is shut down!
I20260709 13:03:01.470752 10346 raft_consensus.cc:493] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:03:01.473502 10346 raft_consensus.cc:515] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:03:01.480819  1644 leader_election.cc:304] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:03:01.481827 10071 raft_consensus.cc:2804] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 25 FOLLOWER]: Leader election won for term 25
I20260709 13:03:01.482340 10071 raft_consensus.cc:697] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 25 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:03:01.483286 10071 consensus_queue.cc:237] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 23.15, Last appended by leader: 15, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:03:01.484539 10201 raft_consensus.cc:515] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:03:01.471827  9979 raft_consensus.cc:493] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:01.485793  2334 raft_consensus.cc:2393] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 23: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:03:01.485939  9979 raft_consensus.cc:515] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:03:01.487946 10224 leader_election.cc:290] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 23 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:03:01.488591 10346 leader_election.cc:290] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:03:01.488839  2329 raft_consensus.cc:2393] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 23: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:03:01.428793 10306 raft_consensus.cc:2749] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:03:01.432572 10246 raft_consensus.cc:515] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:03:01.499271 10347 leader_election.cc:290] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:03:01.500943 10246 leader_election.cc:290] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:03:01.504546   431 tablet_replica.cc:333] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a: stopping tablet replica
I20260709 13:03:01.469080  2332 raft_consensus.cc:2468] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 23.
I20260709 13:03:01.505990  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 26 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:03:01.497330  2328 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39726 (request call id 1867) took 1469 ms. Trace:
I20260709 13:03:01.511554  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ebc67196b4f4119981cdb499e3350cc" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:03:01.510746  2328 rpcz_store.cc:276] 0709 13:03:00.028303 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:00.028605 (+   302us) service_pool.cc:224] Handling call
0709 13:03:00.092095 (+ 63490us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:03:00.120101 (+ 28006us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:00.120138 (+    37us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:03:00.143676 (+ 23538us) log.cc:844] Serialized 52 byte log entry
0709 13:03:00.368463 (+224787us) raft_consensus.cc:1645] Marking committed up to 14
0709 13:03:00.368511 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:00.368593 (+    82us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:01.496021 (+1127428us) spinlock_profiling.cc:243] Waited 6.11 ms on lock 0x7b240034ad68. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f4262efb9c4 00007f4262ef9acb 00007f42630b4dc9 00007f42630b4a55 00007f42630b3fb0 00007f4265d2a488 00007f4265d2abe8 00007f4265d02398 00007f4265cfd76a 00007f4265cfcbb6
0709 13:03:01.496905 (+   884us) raft_consensus.cc:1685] finished
0709 13:03:01.496934 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:01.497240 (+   306us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":17578,"raft.queue_time_us":25316,"raft.run_cpu_time_us":126,"raft.run_wall_time_us":2920,"spinlock_wait_cycles":14052736,"thread_start_us":144082,"threads_started":1,"wal-append.queue_time_us":307770}
I20260709 13:03:01.513902  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:03:01.507975  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 20 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:03:01.508174  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 22 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:03:01.517022  1752 raft_consensus.cc:2410] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 for term 26 because replica has last-logged OpId of term: 24 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 15.
I20260709 13:03:01.517683  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 21 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:03:01.522066  2334 raft_consensus.cc:1275] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 21 index: 17. Preceding OpId from leader: term: 23 index: 18. (index mismatch)
I20260709 13:03:01.524113  2329 raft_consensus.cc:1240] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 22. Current term is 23. Ops: []
I20260709 13:03:01.528158  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:03:01.488595  9979 leader_election.cc:290] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:03:01.516698 10246 raft_consensus.cc:493] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:01.533134   431 raft_consensus.cc:2243] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 LEADER]: Raft consensus shutting down.
I20260709 13:03:01.534279  2334 raft_consensus.cc:1240] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 18. Current term is 19. Ops: [18.14-18.14]
I20260709 13:03:01.534201  2332 raft_consensus.cc:1275] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 25 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 22 index: 16. Preceding OpId from leader: term: 25 index: 17. (index mismatch)
I20260709 13:03:01.538130  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 23 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:03:01.541181  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 27 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:03:01.548730 10268 raft_consensus.cc:515] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:03:01.533274 10246 raft_consensus.cc:515] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:03:01.552901 10246 leader_election.cc:290] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:03:01.554323 10054 leader_election.cc:290] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:03:01.565527  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58d4fdd02c6d4667aa718fb146a76505" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 21 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:03:01.568321  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 24 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:03:01.570624  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:03:01.571611  1748 raft_consensus.cc:2468] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 23.
I20260709 13:03:01.575147  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 21 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:03:01.580014  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 22 candidate_status { last_received { term: 19 index: 9 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:03:01.582828  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 23 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:03:01.584265 10047 raft_consensus.cc:2749] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:03:01.585187  2337 raft_consensus.cc:2468] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 21.
I20260709 13:03:01.586056  1646 leader_election.cc:304] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:03:01.556346 10201 leader_election.cc:290] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 23 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:03:01.587666  9853 raft_consensus.cc:2804] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 26 FOLLOWER]: Leader election won for term 26
I20260709 13:03:01.588327  9853 raft_consensus.cc:697] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 26 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:03:01.589419  9853 consensus_queue.cc:237] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 24.17, Last appended by leader: 17, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:03:01.590817  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:03:01.591730 10268 leader_election.cc:290] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 23 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:03:01.594487  1748 raft_consensus.cc:1275] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 25 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 23 index: 17. Preceding OpId from leader: term: 25 index: 17. (term mismatch)
I20260709 13:03:01.594985  1748 pending_rounds.cc:85] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701: Aborting all ops after (but not including) 16
I20260709 13:03:01.595289  1748 pending_rounds.cc:107] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701: Aborting uncommitted NO_OP operation due to leader change: 23.17
I20260709 13:03:01.595603  1748 raft_consensus.cc:2889] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 25 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:03:01.598623  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 22 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:03:01.599327  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:03:01.603947  2332 raft_consensus.cc:2368] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for earlier term 22. Current term is 23.
W20260709 13:03:01.602608  2298 leader_election.cc:336] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:03:01.612401  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a40e423ed583480280352b261913276c" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 27 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:03:01.602823  1571 leader_election.cc:304] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
W20260709 13:03:01.620173  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 220092us
W20260709 13:03:01.606634  2298 consensus_peers.cc:597] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 13:03:01.599153 10246 raft_consensus.cc:493] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:03:01.629787  2298 consensus_peers.cc:597] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 13:03:01.593026 10360 consensus_queue.cc:1048] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.034s
I20260709 13:03:01.621968 10347 raft_consensus.cc:2804] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 23 FOLLOWER]: Leader pre-election won for term 24
W20260709 13:03:01.633826  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 216978us
I20260709 13:03:01.635170  1718 leader_election.cc:304] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:03:01.632454 10363 raft_consensus.cc:493] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:03:01.625345 10216 consensus_queue.cc:1048] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 14, Time since last communication: 0.013s
I20260709 13:03:01.642490 10363 raft_consensus.cc:515] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:03:01.642764  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:03:01.643594  1680 raft_consensus.cc:2468] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 25.
I20260709 13:03:01.645426 10363 leader_election.cc:290] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:03:01.634588 10347 raft_consensus.cc:493] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:01.646502 10347 raft_consensus.cc:3060] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 23 FOLLOWER]: Advancing to term 24
I20260709 13:03:01.646421  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:03:01.648877  1673 raft_consensus.cc:1275] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 26 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 24 index: 12. Preceding OpId from leader: term: 26 index: 13. (index mismatch)
I20260709 13:03:01.629840 10246 raft_consensus.cc:515] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } }
I20260709 13:03:01.655820  1752 raft_consensus.cc:1275] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 26 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 25 index: 18. Preceding OpId from leader: term: 26 index: 19. (index mismatch)
I20260709 13:03:01.657712 10246 leader_election.cc:290] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:03:01.658808  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:03:01.659220  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:03:01.659511  1673 raft_consensus.cc:2468] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 23.
I20260709 13:03:01.660064  1748 raft_consensus.cc:2393] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 23: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:03:01.660780  1644 leader_election.cc:400] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 22 election: Vote denied by peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625) with higher term. Message: Invalid argument: T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for earlier term 22. Current term is 23.
I20260709 13:03:01.661124  1644 leader_election.cc:403] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 22 election: Cancelling election due to peer responding with higher term
I20260709 13:03:01.661799 10286 raft_consensus.cc:3060] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Advancing to term 23
W20260709 13:03:01.636596  1717 leader_election.cc:336] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:03:01.658011  1569 leader_election.cc:304] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:03:01.664201  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:03:01.666985  2334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:03:01.667580  2334 raft_consensus.cc:3060] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 25 FOLLOWER]: Advancing to term 26
I20260709 13:03:01.663750  1717 leader_election.cc:304] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:03:01.672876  1569 leader_election.cc:304] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:03:01.681207  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:03:01.689424  1569 leader_election.cc:304] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
W20260709 13:03:01.663218  1643 leader_election.cc:336] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:03:01.690970  1643 leader_election.cc:304] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
W20260709 13:03:01.692401  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 57351.9us
W20260709 13:03:01.692961  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 58643.3us
I20260709 13:03:01.693574  1718 leader_election.cc:400] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 24 pre-election: Vote denied by peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625) with higher term. Message: Invalid argument: T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 for earlier term 24. Current term is 25.
I20260709 13:03:01.696569  1569 leader_election.cc:304] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:03:01.699062  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:03:01.696014  1718 leader_election.cc:304] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:03:01.696236  1644 leader_election.cc:304] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:03:01.700516  1677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e26344297fff44bb8d2099b8e64d7df2" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:03:01.709901  2337 raft_consensus.cc:1240] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 25 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 24. Current term is 25. Ops: []
I20260709 13:03:01.711153  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:03:01.711961  2332 raft_consensus.cc:2393] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 23: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
W20260709 13:03:01.711714  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 48539.7us
I20260709 13:03:01.707965  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 21 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:03:01.690299 10345 raft_consensus.cc:2749] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:03:01.731138  1676 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52778 (request call id 3073) took 1210 ms. Trace:
W20260709 13:03:01.733069  1570 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:01.733283  1676 rpcz_store.cc:276] 0709 13:03:00.520408 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:00.520565 (+   157us) service_pool.cc:224] Handling call
0709 13:03:01.726781 (+1206216us) spinlock_profiling.cc:243] Waited 902 ms on lock 0x7b540025b054. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:03:01.731059 (+  4278us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2074470144}
I20260709 13:03:01.731613  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "453513ba44f34adb8b436305ee51fe3d" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 24 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:03:01.727070  2298 leader_election.cc:336] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:03:01.733655  1569 leader_election.cc:304] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:03:01.728957  1718 leader_election.cc:304] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:03:01.735277  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c81785db7d654e4288fcaeacf4db2b49" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 27 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:03:01.737239  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:03:01.738749  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 23 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:03:01.739918  1679 raft_consensus.cc:2410] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a for term 23 because replica has last-logged OpId of term: 22 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 17.
I20260709 13:03:01.742671  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:03:01.745754 10296 raft_consensus.cc:2749] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 13:03:01.735016 10295 raft_consensus.cc:493] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:03:01.735106 10054 raft_consensus.cc:2749] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:03:01.738085  1680 raft_consensus.cc:2468] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 23.
W20260709 13:03:01.746734  2298 consensus_peers.cc:597] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:03:01.695905 10268 raft_consensus.cc:2749] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 13:03:01.695649 10294 raft_consensus.cc:493] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:03:01.700866  9979 raft_consensus.cc:2749] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20260709 13:03:01.758848  2298 leader_election.cc:336] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 23 election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:03:01.767325  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 23 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:03:01.772176  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84e78b66ceec4f759ae9e101a5469e4c" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 22 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:03:01.728348 10115 raft_consensus.cc:2749] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 13:03:01.731671 10277 raft_consensus.cc:2749] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:03:01.733512 10353 raft_consensus.cc:2804] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 13:03:01.780880  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e37c14b11849208b5c9c51f32e1702" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 22 candidate_status { last_received { term: 19 index: 9 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:03:01.735455 10216 raft_consensus.cc:2749] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
W20260709 13:03:01.786794  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 57936.6us
I20260709 13:03:01.747416 10346 raft_consensus.cc:2804] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 13:03:01.789417  2334 raft_consensus.cc:2468] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 26.
I20260709 13:03:01.791213  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 26 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:03:01.732666 10359 consensus_queue.cc:1048] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.053s
I20260709 13:03:01.728585 10246 consensus_queue.cc:1059] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 22.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.057s
I20260709 13:03:01.793838  1676 raft_consensus.cc:2410] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 for term 26 because replica has last-logged OpId of term: 24 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 15.
I20260709 13:03:01.732894 10360 consensus_queue.cc:1048] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.056s
I20260709 13:03:01.796428  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:03:01.794809  1718 leader_election.cc:304] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 13:03:01.784982 10295 raft_consensus.cc:515] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:03:01.790362 10346 raft_consensus.cc:493] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:01.799880 10346 raft_consensus.cc:3060] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:03:01.799966  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03daa4b346f54f1da820d1ff6cd1ff86" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:03:01.800671  1671 raft_consensus.cc:3060] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 24 FOLLOWER]: Advancing to term 25
I20260709 13:03:01.800732 10247 raft_consensus.cc:2804] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 13:03:01.801267 10247 raft_consensus.cc:493] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:03:01.801792 10247 raft_consensus.cc:3060] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Advancing to term 25
I20260709 13:03:01.801855 10364 raft_consensus.cc:2804] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 13:03:01.802340 10364 raft_consensus.cc:493] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:03:01.802757 10364 raft_consensus.cc:3060] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Advancing to term 24
I20260709 13:03:01.803551 10357 consensus_queue.cc:1048] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.069s
I20260709 13:03:01.805114 10340 consensus_queue.cc:1048] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 16, Time since last communication: 0.031s
I20260709 13:03:01.801347 10295 leader_election.cc:290] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:03:01.790931 10353 raft_consensus.cc:697] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 23 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:03:01.779865 10294 raft_consensus.cc:515] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:03:01.829571  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:03:01.841892  1569 leader_election.cc:304] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:03:01.850102  2299 leader_election.cc:304] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:03:01.850481  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86abe5695d5643f888190301b88dacef" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:03:01.816341  1718 leader_election.cc:304] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:03:01.859776  9424 raft_consensus.cc:2804] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 13:03:01.860280  9424 raft_consensus.cc:493] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:01.860680  9424 raft_consensus.cc:3060] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Advancing to term 24
W20260709 13:03:01.819170 10370 log.cc:927] Time spent T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.053s	user 0.000s	sys 0.000s
I20260709 13:03:01.822746 10372 consensus_queue.cc:1048] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 9, Time since last communication: 0.044s
I20260709 13:03:01.824782 10353 consensus_queue.cc:237] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 22.14, Last appended by leader: 14, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:03:01.828192 10309 raft_consensus.cc:3060] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:03:01.829046 10294 leader_election.cc:290] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:03:01.831619 10203 raft_consensus.cc:493] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:01.873118 10203 raft_consensus.cc:515] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:03:01.875578 10340 raft_consensus.cc:2749] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 13:03:01.876078  1680 raft_consensus.cc:2393] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 26: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
I20260709 13:03:01.835951 10347 raft_consensus.cc:515] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:03:01.880497 10369 log.cc:927] Time spent T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.084s	user 0.002s	sys 0.000s
I20260709 13:03:01.880836 10303 raft_consensus.cc:2749] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:03:01.871784 10216 raft_consensus.cc:3055] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 LEADER]: Stepping down as leader of term 22
I20260709 13:03:01.884014 10216 raft_consensus.cc:740] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:03:01.884944 10216 consensus_queue.cc:260] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 22.16, Last appended by leader: 16, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:03:01.891374  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 49620.5us
I20260709 13:03:01.892920  1569 leader_election.cc:304] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:03:01.893798 10303 raft_consensus.cc:2749] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:03:01.894423 10364 raft_consensus.cc:515] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:03:01.895568  1569 leader_election.cc:304] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
W20260709 13:03:01.896219  1570 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:01.896361  1569 raft_consensus.cc:2686] T c81785db7d654e4288fcaeacf4db2b49 P 83595ad0fd5940c0a2bfc7c61acd969a: unable to run election callback: Service unavailable: Thread pool token was shut down
I20260709 13:03:01.896759 10364 leader_election.cc:290] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 24 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:03:01.897457  1569 leader_election.cc:304] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:03:01.899237   431 raft_consensus.cc:2272] T 1809d674329a492dbea0b60a4703fe51 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Raft consensus is shut down!
W20260709 13:03:01.901031  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 51093.2us
I20260709 13:03:01.902957 10239 raft_consensus.cc:2749] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:03:01.903811 10203 leader_election.cc:290] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:03:01.903986  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:03:01.904106 10347 leader_election.cc:290] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 24 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:03:01.904672  1675 raft_consensus.cc:3060] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Advancing to term 24
W20260709 13:03:01.906860  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 140147us
I20260709 13:03:01.907898  1671 raft_consensus.cc:2468] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 25.
I20260709 13:03:01.916590  2299 leader_election.cc:304] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
W20260709 13:03:01.914991  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 54394us
I20260709 13:03:01.916970  1569 leader_election.cc:304] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:03:01.922991  2299 leader_election.cc:304] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:03:01.928058  2301 leader_election.cc:304] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:03:01.931563  9424 raft_consensus.cc:515] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:03:01.968595  1569 leader_election.cc:304] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:03:01.970528  1569 leader_election.cc:304] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:03:01.914160 10216 raft_consensus.cc:3060] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Advancing to term 23
W20260709 13:03:01.973012  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 208925us
I20260709 13:03:01.976294  1671 raft_consensus.cc:1240] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 25 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 24. Current term is 25. Ops: [24.18-24.18]
W20260709 13:03:01.977100  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 54266us
I20260709 13:03:01.978955  1676 raft_consensus.cc:1240] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 25 LEADER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 24. Current term is 25. Ops: []
I20260709 13:03:01.981549  2299 leader_election.cc:304] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:03:01.982172  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 26 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:03:01.986545  1675 raft_consensus.cc:2468] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 24.
I20260709 13:03:01.989151  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:03:01.989677  9424 leader_election.cc:290] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 24 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:03:01.973919  1569 leader_election.cc:304] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:03:01.971508 10347 raft_consensus.cc:2749] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 13:03:01.991865 10247 raft_consensus.cc:515] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:03:01.987803 10063 raft_consensus.cc:3060] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 25 FOLLOWER]: Advancing to term 26
I20260709 13:03:01.998190   431 tablet_replica.cc:333] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a: stopping tablet replica
W20260709 13:03:02.006933  2298 leader_election.cc:336] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:03:02.006976 10298 raft_consensus.cc:493] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:01.999418 10347 raft_consensus.cc:493] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:01.943678 10286 raft_consensus.cc:2749] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Leader election lost for term 22. Reason: Vote denied by peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625) with higher term. Message: Invalid argument: T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d for earlier term 22. Current term is 23.
I20260709 13:03:02.003940 10357 consensus_queue.cc:1059] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20260709 13:03:02.018704 10357 raft_consensus.cc:3055] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 13:03:02.019134 10357 raft_consensus.cc:740] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:03:02.020107 10357 consensus_queue.cc:260] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 24.18, Last appended by leader: 18, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:03:02.020738 10346 raft_consensus.cc:515] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:03:02.021515 10357 raft_consensus.cc:3060] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Advancing to term 25
I20260709 13:03:01.930307 10185 raft_consensus.cc:2749] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 13:03:01.994161 10247 leader_election.cc:290] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 25 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:03:01.979437 10303 raft_consensus.cc:493] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:03:02.022737 10364 consensus_queue.cc:1175] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "816fadcea8fc4893882f5f9b9f22a03d" responder_term: 25 status { last_received { term: 23 index: 17 } last_committed_idx: 17 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 24. Current term is 25. Ops: [24.18-24.18]" } } last_received_current_leader { term: 0 index: 0 } }
I20260709 13:03:02.023351 10303 raft_consensus.cc:515] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:03:02.025960 10303 leader_election.cc:290] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:03:01.973527 10322 raft_consensus.cc:2749] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:03:02.033330  2329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:03:02.034648  2330 raft_consensus.cc:1275] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 25 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 23 index: 15. Preceding OpId from leader: term: 25 index: 16. (index mismatch)
I20260709 13:03:02.015511  1569 leader_election.cc:304] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:03:01.988840 10047 raft_consensus.cc:2749] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 13:03:01.979091  9920 raft_consensus.cc:2749] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:03:02.043603  1753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 27 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:03:02.016629 10298 raft_consensus.cc:515] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:03:02.043749  1752 raft_consensus.cc:1275] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 22 index: 15. Preceding OpId from leader: term: 24 index: 16. (index mismatch)
I20260709 13:03:02.017509 10347 raft_consensus.cc:515] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:03:02.045938  2328 raft_consensus.cc:1275] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 25 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 23 index: 17. Preceding OpId from leader: term: 25 index: 18. (index mismatch)
I20260709 13:03:02.047683 10063 raft_consensus.cc:2749] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 13:03:02.050940  1754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:03:02.051779  1569 leader_election.cc:304] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:03:02.052275   431 raft_consensus.cc:2243] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Raft consensus shutting down.
I20260709 13:03:02.054061  1754 raft_consensus.cc:2393] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 83595ad0fd5940c0a2bfc7c61acd969a in current term 24: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:03:02.055487  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0615de9bff034ecb9ed6173433128ff5" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 27 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:03:02.064992  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 83557us
I20260709 13:03:02.044201 10294 raft_consensus.cc:493] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:03:02.045785 10309 raft_consensus.cc:2749] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:03:02.078781  2301 leader_election.cc:304] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:03:02.078168  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 22 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:03:02.080960  1680 raft_consensus.cc:3060] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:03:02.082319  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:03:02.077526  1571 leader_election.cc:304] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:03:02.054100 10239 raft_consensus.cc:2749] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:03:02.112993   431 raft_consensus.cc:2272] T 2ebc67196b4f4119981cdb499e3350cc P 83595ad0fd5940c0a2bfc7c61acd969a [term 17 FOLLOWER]: Raft consensus is shut down!
I20260709 13:03:02.046254 10286 consensus_queue.cc:1048] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.032s
I20260709 13:03:02.073712 10185 raft_consensus.cc:2804] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 25 FOLLOWER]: Leader election won for term 25
I20260709 13:03:02.083977  2328 raft_consensus.cc:2393] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 25: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:03:02.045882 10247 consensus_queue.cc:1059] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.032s
I20260709 13:03:02.117205  1678 raft_consensus.cc:2468] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 23.
I20260709 13:03:02.075685 10303 raft_consensus.cc:2804] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 24 FOLLOWER]: Leader election won for term 24
I20260709 13:03:02.083915 10290 raft_consensus.cc:493] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:02.112910  9920 raft_consensus.cc:2749] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260709 13:03:02.133726  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 106717us
I20260709 13:03:02.073693 10294 raft_consensus.cc:515] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:03:02.131716 10185 raft_consensus.cc:697] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 25 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:03:02.125886 10224 raft_consensus.cc:2749] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 13:03:02.085224 10353 raft_consensus.cc:493] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:03:02.148751  1571 leader_election.cc:304] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
W20260709 13:03:02.139889  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 53732.5us
I20260709 13:03:02.161082  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
W20260709 13:03:02.162295  1643 leader_election.cc:336] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 26 election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:03:02.162878  1643 leader_election.cc:304] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:03:02.163843  1679 raft_consensus.cc:2393] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 24: Already voted for candidate 83595ad0fd5940c0a2bfc7c61acd969a in this term.
I20260709 13:03:02.124158 10341 consensus_queue.cc:1048] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.019s
I20260709 13:03:02.132946 10247 consensus_queue.cc:1243] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer 816fadcea8fc4893882f5f9b9f22a03d log is divergent from this leader: its last log entry 25.16 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 15
I20260709 13:03:02.142517 10346 leader_election.cc:290] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 22 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:03:02.163929 10353 raft_consensus.cc:515] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:03:02.166373 10230 consensus_queue.cc:1048] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.006s
I20260709 13:03:02.175596 10347 leader_election.cc:290] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:03:02.176334  1671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 28 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:03:02.198552  1671 raft_consensus.cc:2468] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 26.
I20260709 13:03:02.147590 10303 raft_consensus.cc:697] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 24 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:03:02.202289  1569 leader_election.cc:304] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:03:02.180382 10302 raft_consensus.cc:2749] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 13:03:02.184859  1754 raft_consensus.cc:1240] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Rejecting Update request from peer 83595ad0fd5940c0a2bfc7c61acd969a for earlier term 23. Current term is 24. Ops: []
I20260709 13:03:02.184221  1675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "983621a6fc5b4c9ea0159a648dfe7a36" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:03:02.183779  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 28 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:03:02.168826  1643 leader_election.cc:336] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:03:02.210851  1752 raft_consensus.cc:2468] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 27.
I20260709 13:03:02.212301  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f1329a6c1e442bcba5945a2d1e65e85" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:03:02.213538  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78cabb03415041ca9c1a5a400e506131" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:03:02.211176  1643 leader_election.cc:304] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:03:02.184633 10309 raft_consensus.cc:2749] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
W20260709 13:03:02.226264  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 57480.1us
I20260709 13:03:02.135412 10290 raft_consensus.cc:515] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:03:02.227432  1643 consensus_peers.cc:597] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:03:02.229542  1643 leader_election.cc:336] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:03:02.230012  1643 leader_election.cc:304] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:03:02.218144 10347 raft_consensus.cc:2804] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 13:03:02.233199  1748 raft_consensus.cc:1275] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 26 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 24 index: 17. Preceding OpId from leader: term: 26 index: 18. (index mismatch)
I20260709 13:03:02.236666  1745 raft_consensus.cc:1275] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 25 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 23 index: 17. Preceding OpId from leader: term: 25 index: 18. (index mismatch)
I20260709 13:03:02.246932  1750 raft_consensus.cc:3055] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 24 LEADER]: Stepping down as leader of term 24
W20260709 13:03:02.251143 10379 log.cc:927] Time spent T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.093s	user 0.000s	sys 0.000s
I20260709 13:03:02.235598 10185 consensus_queue.cc:237] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 23.17, Last appended by leader: 17, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:03:02.243243 10071 raft_consensus.cc:2749] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 13:03:02.261960 10303 consensus_queue.cc:237] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.15, Last appended by leader: 15, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:03:02.263903   431 tablet_replica.cc:333] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a: stopping tablet replica
I20260709 13:03:02.264997   431 raft_consensus.cc:2243] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 24 FOLLOWER]: Raft consensus shutting down.
W20260709 13:03:02.237803  1717 consensus_peers.cc:597] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 13:03:02.267920   431 raft_consensus.cc:2272] T 983621a6fc5b4c9ea0159a648dfe7a36 P 83595ad0fd5940c0a2bfc7c61acd969a [term 24 FOLLOWER]: Raft consensus is shut down!
W20260709 13:03:02.268004  1717 leader_election.cc:336] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 25 election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:03:02.268566  1717 leader_election.cc:304] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:03:02.235859 10347 raft_consensus.cc:493] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:02.267099  1646 leader_election.cc:304] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:03:02.270558 10347 raft_consensus.cc:3060] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 27 FOLLOWER]: Advancing to term 28
W20260709 13:03:02.238271  1643 leader_election.cc:336] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:03:02.274643 10386 raft_consensus.cc:2749] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 13:03:02.247324  1750 raft_consensus.cc:740] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:03:02.276571  1750 consensus_queue.cc:260] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 24.16, Last appended by leader: 16, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:03:02.279691  1750 raft_consensus.cc:3060] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Advancing to term 25
I20260709 13:03:02.280395 10301 raft_consensus.cc:2749] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 13:03:02.241396 10346 consensus_queue.cc:1059] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: INVALID_TERM, Last received: 23.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 13:03:02.281235  1643 leader_election.cc:304] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:03:02.283356 10346 raft_consensus.cc:3055] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 23 LEADER]: Stepping down as leader of term 23
I20260709 13:03:02.283807 10346 raft_consensus.cc:740] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:03:02.283840 10351 raft_consensus.cc:2749] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
W20260709 13:03:02.285138  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 46915.7us
I20260709 13:03:02.284842 10346 consensus_queue.cc:260] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 23.15, Last appended by leader: 15, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:03:02.285836 10298 leader_election.cc:290] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:03:02.286538 10346 raft_consensus.cc:3060] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 23 FOLLOWER]: Advancing to term 24
I20260709 13:03:02.288232 10353 leader_election.cc:290] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:03:02.228060 10310 raft_consensus.cc:2749] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:03:02.293200  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:03:02.289664  1680 raft_consensus.cc:2468] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 22.
I20260709 13:03:02.283565 10071 consensus_queue.cc:1048] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 13:03:02.293902  1679 raft_consensus.cc:2468] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 19.
I20260709 13:03:02.288853   431 tablet_replica.cc:333] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a: stopping tablet replica
I20260709 13:03:02.291987  1675 raft_consensus.cc:2468] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 25.
W20260709 13:03:02.270185  1717 leader_election.cc:336] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 26 election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 13:03:02.294175  1643 leader_election.cc:336] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:03:02.303117  1643 leader_election.cc:304] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:03:02.295220  1569 leader_election.cc:304] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
W20260709 13:03:02.303253  1717 leader_election.cc:336] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 27 election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:03:02.304327  1717 leader_election.cc:304] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:03:02.296805 10290 leader_election.cc:290] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:03:02.307528  1717 leader_election.cc:336] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:03:02.308106  1717 leader_election.cc:304] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 13:03:02.312629  1717 leader_election.cc:336] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:03:02.313228  1717 leader_election.cc:304] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 13:03:02.314800  1717 leader_election.cc:336] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:03:02.315342  1717 leader_election.cc:304] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 13:03:02.321199  1717 consensus_peers.cc:597] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:03:02.321810  1717 leader_election.cc:336] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 23 election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:03:02.322233  1717 leader_election.cc:304] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:03:02.318058  1569 leader_election.cc:304] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
W20260709 13:03:02.326786  1720 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:02.328408  1643 leader_election.cc:336] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:03:02.328891  1643 leader_election.cc:304] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 13:03:02.330225  1717 leader_election.cc:336] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:03:02.331048  1750 raft_consensus.cc:1275] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 25 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 24 index: 16. Preceding OpId from leader: term: 25 index: 16. (term mismatch)
I20260709 13:03:02.331540  1750 pending_rounds.cc:85] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701: Aborting all ops after (but not including) 15
I20260709 13:03:02.331683  1717 leader_election.cc:304] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:03:02.332350   431 raft_consensus.cc:2243] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 23 FOLLOWER]: Raft consensus shutting down.
I20260709 13:03:02.332335  1750 pending_rounds.cc:107] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701: Aborting uncommitted NO_OP operation due to leader change: 24.16
I20260709 13:03:02.346524  1750 raft_consensus.cc:2889] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 25 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:03:02.339449 10359 raft_consensus.cc:2804] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:03:02.311473 10216 raft_consensus.cc:2804] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Leader election won for term 22
I20260709 13:03:02.357302 10216 raft_consensus.cc:697] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:03:02.358364 10216 consensus_queue.cc:237] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 20.17, Last appended by leader: 17, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:03:02.317421 10307 raft_consensus.cc:2749] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 13:03:02.332037 10230 raft_consensus.cc:2749] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 13:03:02.328024 10353 raft_consensus.cc:2749] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:03:02.316833 10267 raft_consensus.cc:2749] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 13:03:02.379148  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 22 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:03:02.379851  2337 raft_consensus.cc:3060] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:03:02.381438  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 26 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:03:02.382182  1745 raft_consensus.cc:2468] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 25.
W20260709 13:03:02.350121  1717 leader_election.cc:336] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:03:02.356240   431 raft_consensus.cc:2272] T 453513ba44f34adb8b436305ee51fe3d P 83595ad0fd5940c0a2bfc7c61acd969a [term 23 FOLLOWER]: Raft consensus is shut down!
I20260709 13:03:02.356673 10247 raft_consensus.cc:3060] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:03:02.323204 10317 raft_consensus.cc:2749] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:03:02.329972 10386 raft_consensus.cc:2749] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 13:03:02.416322  2337 raft_consensus.cc:2468] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 22.
I20260709 13:03:02.350768 10359 raft_consensus.cc:493] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:02.422851  2301 leader_election.cc:304] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:03:02.424830   431 tablet_replica.cc:333] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a: stopping tablet replica
I20260709 13:03:02.429629 10294 leader_election.cc:290] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:03:02.330704 10310 consensus_queue.cc:1048] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.077s
I20260709 13:03:02.449553   431 raft_consensus.cc:2243] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Raft consensus shutting down.
I20260709 13:03:02.450685   431 raft_consensus.cc:2272] T 03e37c14b11849208b5c9c51f32e1702 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Raft consensus is shut down!
I20260709 13:03:02.453598 10247 raft_consensus.cc:2749] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:03:02.453924   431 tablet_replica.cc:333] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a: stopping tablet replica
I20260709 13:03:02.436957 10359 raft_consensus.cc:3060] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:03:02.455128 10347 raft_consensus.cc:515] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:03:02.460752 10347 leader_election.cc:290] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 28 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:03:02.455341   431 raft_consensus.cc:2243] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 LEADER]: Raft consensus shutting down.
I20260709 13:03:02.462117   431 raft_consensus.cc:2272] T c6ab3300be1e4a59b64b14d50da193ca P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Raft consensus is shut down!
I20260709 13:03:02.469728   431 tablet_replica.cc:333] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a: stopping tablet replica
I20260709 13:03:02.475739  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 28 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:03:02.476619  1748 raft_consensus.cc:3060] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 27 FOLLOWER]: Advancing to term 28
I20260709 13:03:02.485962  2332 raft_consensus.cc:1275] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 22 index: 14. Preceding OpId from leader: term: 23 index: 15. (index mismatch)
W20260709 13:03:02.492446  1717 consensus_peers.cc:597] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:03:02.495645  1717 leader_election.cc:336] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:03:02.496330  1717 leader_election.cc:304] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:03:02.497465 10247 raft_consensus.cc:2749] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
W20260709 13:03:02.499156  1717 leader_election.cc:336] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 23 election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 13:03:02.500535  1717 leader_election.cc:336] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 13:03:02.504426  1717 leader_election.cc:336] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 24 election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:03:02.505228  1717 leader_election.cc:304] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 13:03:02.507668  1717 leader_election.cc:336] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 24 election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:03:02.483412 10360 raft_consensus.cc:493] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:02.525315 10267 consensus_queue.cc:1048] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.001s
W20260709 13:03:02.527323 10115 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 25 current term: 25 is higher.
I20260709 13:03:02.535596  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:03:02.536393  2328 raft_consensus.cc:2468] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 19.
I20260709 13:03:02.503014 10340 raft_consensus.cc:2804] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 13:03:02.558895  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:03:02.564118  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 235792us
W20260709 13:03:02.568678  2298 consensus_peers.cc:597] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 13:03:02.510281 10386 raft_consensus.cc:2749] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 13:03:02.578493  1748 raft_consensus.cc:2468] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 28.
I20260709 13:03:02.582288  1718 leader_election.cc:304] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:03:02.583225  2336 raft_consensus.cc:1275] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 26 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 21 index: 16. Preceding OpId from leader: term: 26 index: 18. (index mismatch)
I20260709 13:03:02.590876  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 28 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:03:02.591981  1678 raft_consensus.cc:3060] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 26 FOLLOWER]: Advancing to term 28
I20260709 13:03:02.602677  1571 leader_election.cc:304] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
W20260709 13:03:02.607779  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 99213.4us
I20260709 13:03:02.615020  1679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 26 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:03:02.621309 10347 raft_consensus.cc:2804] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 28 FOLLOWER]: Leader election won for term 28
I20260709 13:03:02.570355 10301 raft_consensus.cc:2749] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:03:02.630020  1678 raft_consensus.cc:2468] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 28.
I20260709 13:03:02.546571 10360 raft_consensus.cc:515] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:03:02.640426  1718 leader_election.cc:304] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 13:03:02.623095 10307 raft_consensus.cc:2749] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
W20260709 13:03:02.643623  2298 consensus_peers.cc:597] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 13:03:02.626629 10347 raft_consensus.cc:697] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 28 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:03:02.646764  1718 leader_election.cc:304] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 13:03:02.648335 10317 raft_consensus.cc:2804] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 25 FOLLOWER]: Leader election won for term 25
I20260709 13:03:02.648926 10317 raft_consensus.cc:697] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 25 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:03:02.652609  2337 raft_consensus.cc:1240] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 25 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 24. Current term is 25. Ops: []
I20260709 13:03:02.643589 10267 raft_consensus.cc:493] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:03:02.655627 10267 raft_consensus.cc:515] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:03:02.655954  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 66825.9us
I20260709 13:03:02.658001 10267 leader_election.cc:290] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 13:03:02.662873  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 54100.2us
W20260709 13:03:02.699205  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 116940us
W20260709 13:03:02.701990  1717 leader_election.cc:336] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:03:02.709470  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63cf5ab930f24f239fde96a128656333" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:03:02.707626  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 44505.6us
I20260709 13:03:02.638753 10286 consensus_queue.cc:1048] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.046s
W20260709 13:03:02.715737  1717 consensus_peers.cc:597] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 13:03:02.726112 10347 consensus_queue.cc:237] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 26.13, Last appended by leader: 13, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:03:02.703398 10267 raft_consensus.cc:2804] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Leader election won for term 23
W20260709 13:03:02.731163  1717 consensus_peers.cc:597] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 13:03:02.649984 10317 consensus_queue.cc:237] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 24.16, Last appended by leader: 16, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:03:02.724407 10340 raft_consensus.cc:493] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 25 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:03:02.751125  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:03:02.752990  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:03:02.753631  1672 raft_consensus.cc:2468] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 22.
I20260709 13:03:02.759963   431 raft_consensus.cc:2243] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 25 LEADER]: Raft consensus shutting down.
I20260709 13:03:02.744987 10267 raft_consensus.cc:697] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 23 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:03:02.765908  1569 leader_election.cc:304] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:03:02.732903 10373 consensus_queue.cc:1048] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.023s
I20260709 13:03:02.755939 10340 raft_consensus.cc:3060] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 25 FOLLOWER]: Advancing to term 26
I20260709 13:03:02.809551  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:03:02.809765  1643 leader_election.cc:336] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:03:02.810360  1751 raft_consensus.cc:2468] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 23.
I20260709 13:03:02.810707  1643 leader_election.cc:304] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:03:02.811918  1718 leader_election.cc:304] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:03:02.821033 10216 raft_consensus.cc:2804] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 13:03:02.821653 10216 raft_consensus.cc:493] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:02.822109 10216 raft_consensus.cc:3060] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 23 FOLLOWER]: Advancing to term 24
I20260709 13:03:02.751907 10359 raft_consensus.cc:515] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:03:02.836951  9853 raft_consensus.cc:2749] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:03:02.839043 10247 raft_consensus.cc:2749] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
W20260709 13:03:02.832674 10401 log.cc:927] Time spent T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.109s	user 0.000s	sys 0.001s
I20260709 13:03:02.851621 10267 consensus_queue.cc:237] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 21.16, Last appended by leader: 16, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:03:02.798748 10360 leader_election.cc:290] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:03:02.867216  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 101427us
I20260709 13:03:02.869827  2337 raft_consensus.cc:1275] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 20 index: 17. Preceding OpId from leader: term: 22 index: 18. (index mismatch)
W20260709 13:03:02.878259  1643 leader_election.cc:336] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:03:02.878489  1676 raft_consensus.cc:1275] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 20 index: 17. Preceding OpId from leader: term: 22 index: 18. (index mismatch)
I20260709 13:03:02.878822  1643 leader_election.cc:304] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:03:02.879709 10304 raft_consensus.cc:2749] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
W20260709 13:03:02.876586  2298 consensus_peers.cc:597] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260709 13:03:02.893041  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 173249us
I20260709 13:03:02.896240  1672 raft_consensus.cc:1275] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 24 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 21 index: 14. Preceding OpId from leader: term: 24 index: 16. (index mismatch)
I20260709 13:03:02.914085 10360 consensus_queue.cc:1048] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 13:03:02.915732 10360 consensus_queue.cc:1048] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 13:03:02.917203 10360 consensus_queue.cc:1048] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20260709 13:03:02.938670 10359 leader_election.cc:290] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 20 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:03:02.957098  1750 raft_consensus.cc:1275] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 22 index: 15. Preceding OpId from leader: term: 24 index: 16. (index mismatch)
W20260709 13:03:02.964704  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 70861.8us
I20260709 13:03:02.931592 10411 raft_consensus.cc:493] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:02.982918 10411 raft_consensus.cc:515] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:03:02.986052  1672 raft_consensus.cc:1275] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 25 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 23 index: 17. Preceding OpId from leader: term: 25 index: 18. (index mismatch)
W20260709 13:03:02.980043  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 56964us
I20260709 13:03:02.956998 10216 raft_consensus.cc:515] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:03:02.995872  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:03:02.998275  1674 raft_consensus.cc:3060] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:03:02.964207 10347 raft_consensus.cc:493] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:03:03.005435  1643 leader_election.cc:336] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:03:03.006078  1643 leader_election.cc:304] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:03:03.007680 10302 raft_consensus.cc:2749] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:03:03.015378  1750 raft_consensus.cc:1275] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 25 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 24 index: 18. Preceding OpId from leader: term: 25 index: 18. (term mismatch)
I20260709 13:03:03.015816  1750 pending_rounds.cc:85] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701: Aborting all ops after (but not including) 17
I20260709 13:03:03.016108  1750 pending_rounds.cc:107] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701: Aborting uncommitted NO_OP operation due to leader change: 24.18
I20260709 13:03:03.016362  1750 raft_consensus.cc:2889] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 25 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:03:03.021322  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:03:03.021977  2330 raft_consensus.cc:3060] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Advancing to term 20
W20260709 13:03:03.026918  1717 leader_election.cc:336] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:03:03.031778 10411 leader_election.cc:290] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:03:03.027398  1717 leader_election.cc:304] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:03:03.035815  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 27 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:03:03.037459  1717 consensus_peers.cc:597] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:03:03.042702  2330 raft_consensus.cc:2468] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 20.
I20260709 13:03:03.042928  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b917a1e9c9aa40a0afce5a12c343eacd" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 27 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:03:03.044116  1717 consensus_peers.cc:597] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 13:03:03.054770  1569 leader_election.cc:304] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:03:03.058789 10384 raft_consensus.cc:493] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:03:03.062894 10384 raft_consensus.cc:515] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:03:03.065395 10384 leader_election.cc:290] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:03:03.056394 10417 raft_consensus.cc:2749] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 13:03:03.072484  1569 leader_election.cc:304] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:03:03.024317 10245 consensus_queue.cc:1048] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.051s
I20260709 13:03:03.081136  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:03:03.042099 10317 raft_consensus.cc:2749] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:03:03.018033 10303 consensus_queue.cc:1048] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.039s
I20260709 13:03:03.020845 10347 raft_consensus.cc:515] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:03:03.121521 10216 leader_election.cc:290] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 24 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:03:03.125136  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:03:03.125808  1676 raft_consensus.cc:3060] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Advancing to term 24
I20260709 13:03:03.088019 10411 raft_consensus.cc:2804] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Leader election won for term 20
W20260709 13:03:03.123126 10406 log.cc:927] Time spent T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.057s	user 0.000s	sys 0.001s
W20260709 13:03:03.143632  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 91785.6us
W20260709 13:03:03.150146  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 131278us
I20260709 13:03:03.077106 10203 consensus_queue.cc:1048] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.062s
W20260709 13:03:03.175086  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 58443.4us
I20260709 13:03:03.179209  1745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:03:03.179859  1745 raft_consensus.cc:3060] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Advancing to term 24
I20260709 13:03:03.159610 10411 raft_consensus.cc:697] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:03:03.183115 10411 consensus_queue.cc:237] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:03:03.191027  1674 raft_consensus.cc:2468] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 20.
I20260709 13:03:03.171705 10340 raft_consensus.cc:515] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:03:03.213723 10340 leader_election.cc:290] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 26 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:03:03.215225  1676 raft_consensus.cc:2468] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 24.
I20260709 13:03:03.216156  2332 raft_consensus.cc:1240] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 26 FOLLOWER]: Rejecting Update request from peer 816fadcea8fc4893882f5f9b9f22a03d for earlier term 25. Current term is 26. Ops: []
I20260709 13:03:03.217499 10216 raft_consensus.cc:493] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:03.218200 10216 raft_consensus.cc:515] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:03:03.221470  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:03:03.230829 10347 leader_election.cc:290] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:03:03.235160  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:03:03.238763  2299 leader_election.cc:304] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:03:03.251883 10277 consensus_queue.cc:1059] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 25.18, Next index: 19, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 13:03:03.253135 10277 raft_consensus.cc:3055] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 25 LEADER]: Stepping down as leader of term 25
I20260709 13:03:03.253526 10277 raft_consensus.cc:740] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
W20260709 13:03:03.254866  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 82996us
I20260709 13:03:03.261138  1569 leader_election.cc:304] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
I20260709 13:03:03.264603  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 24 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:03:03.266240 10414 raft_consensus.cc:2804] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 24 FOLLOWER]: Leader election won for term 24
I20260709 13:03:03.266871 10414 raft_consensus.cc:697] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 24 LEADER]: Becoming Leader. State: Replica: 83595ad0fd5940c0a2bfc7c61acd969a, State: Running, Role: LEADER
I20260709 13:03:03.267822 10414 consensus_queue.cc:237] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 22.14, Last appended by leader: 14, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:03:03.273425   431 raft_consensus.cc:2272] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 83595ad0fd5940c0a2bfc7c61acd969a [term 25 FOLLOWER]: Raft consensus is shut down!
W20260709 13:03:03.300580  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 61937.7us
I20260709 13:03:03.308768  2332 raft_consensus.cc:1275] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 25 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 22 index: 15. Preceding OpId from leader: term: 25 index: 17. (index mismatch)
I20260709 13:03:03.310699  1674 raft_consensus.cc:1275] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 21 index: 16. Preceding OpId from leader: term: 23 index: 17. (index mismatch)
I20260709 13:03:03.316672  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 27 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:03:03.320992  2337 raft_consensus.cc:1275] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 21 index: 16. Preceding OpId from leader: term: 23 index: 17. (index mismatch)
W20260709 13:03:03.328027  2298 leader_election.cc:336] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:03:03.328881  2298 leader_election.cc:304] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:03:03.351365 10216 leader_election.cc:290] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:03:03.311153 10341 raft_consensus.cc:2749] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 13:03:03.330389 10267 consensus_queue.cc:1048] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.041s
I20260709 13:03:03.299341 10277 consensus_queue.cc:260] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 25.18, Last appended by leader: 18, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:03:03.343597 10247 consensus_queue.cc:1048] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.032s
I20260709 13:03:03.372906 10307 consensus_queue.cc:1048] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 13:03:03.375675  1745 raft_consensus.cc:2468] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 24.
I20260709 13:03:03.377441 10306 raft_consensus.cc:2749] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:03:03.385877  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d66b92688a42e3ad231959a644db56" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 24 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:03:03.324818 10424 log.cc:927] Time spent T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.087s	user 0.001s	sys 0.000s
I20260709 13:03:03.394134  1673 raft_consensus.cc:1275] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 25 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 24 index: 16. Preceding OpId from leader: term: 25 index: 17. (index mismatch)
I20260709 13:03:03.399725  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 27 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:03:03.400749  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 55117.2us
W20260709 13:03:03.388254  1717 consensus_peers.cc:597] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260709 13:03:03.433770  1717 consensus_peers.cc:597] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260709 13:03:03.439891  1717 consensus_peers.cc:597] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 13:03:03.412573  1750 raft_consensus.cc:1275] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 28 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 26 index: 13. Preceding OpId from leader: term: 28 index: 14. (index mismatch)
I20260709 13:03:03.409417 10347 raft_consensus.cc:493] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:03.459506 10409 consensus_queue.cc:1048] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 13:03:03.485333   431 tablet_replica.cc:333] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a: stopping tablet replica
W20260709 13:03:03.491125  2298 consensus_peers.cc:597] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 13:03:03.519191   431 raft_consensus.cc:2243] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 25 FOLLOWER]: Raft consensus shutting down.
I20260709 13:03:03.528998  1673 raft_consensus.cc:1275] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 28 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 26 index: 13. Preceding OpId from leader: term: 28 index: 14. (index mismatch)
I20260709 13:03:03.541607   431 raft_consensus.cc:2272] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 83595ad0fd5940c0a2bfc7c61acd969a [term 25 FOLLOWER]: Raft consensus is shut down!
I20260709 13:03:03.502301 10347 raft_consensus.cc:515] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:03:03.540444 10277 raft_consensus.cc:3060] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 25 FOLLOWER]: Advancing to term 26
I20260709 13:03:03.547339 10267 raft_consensus.cc:493] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:03:03.545699 10412 consensus_queue.cc:1048] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.023s
I20260709 13:03:03.602223  1571 leader_election.cc:304] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701
I20260709 13:03:03.624336  1571 leader_election.cc:304] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:03:03.629019  9920 raft_consensus.cc:2749] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 13:03:03.629133 10345 raft_consensus.cc:2749] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 13:03:03.632465  1500 catalog_manager.cc:5697] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d reported cstate change: term changed from 9 to 10, leader changed from 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) to 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194). New cstate: current_term: 10 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: HEALTHY } } }
I20260709 13:03:03.639492  1500 catalog_manager.cc:5697] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d reported cstate change: term changed from 12 to 14, leader changed from 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) to af520b67945d443b901bbeca37cc5701 (127.0.107.195). New cstate: current_term: 14 leader_uuid: "af520b67945d443b901bbeca37cc5701" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
W20260709 13:03:03.641018 10425 log.cc:927] Time spent T 00000000000000000000000000000000 P 919f6392d4f44fd68987e2c5e7ec37ca: Append to log took a long time: real 0.126s	user 0.001s	sys 0.000s
W20260709 13:03:03.642144 10430 log.cc:927] Time spent T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a: Append to log took a long time: real 0.076s	user 0.000s	sys 0.000s
I20260709 13:03:03.608242 10411 raft_consensus.cc:493] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:03.651103 10411 raft_consensus.cc:515] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:03:03.653908 10411 leader_election.cc:290] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:03:03.655658  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:03:03.655849 10345 consensus_queue.cc:1048] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.001s
W20260709 13:03:03.635010 10427 log.cc:927] Time spent T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.111s	user 0.000s	sys 0.000s
W20260709 13:03:03.637975 10423 log.cc:927] Time spent T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.079s	user 0.000s	sys 0.000s
W20260709 13:03:03.662679  1643 consensus_peers.cc:597] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 13:03:03.645624  1500 catalog_manager.cc:5697] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d reported cstate change: term changed from 12 to 14. New cstate: current_term: 14 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:03:03.677531  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:03:03.678241  2328 raft_consensus.cc:2468] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 23.
I20260709 13:03:03.608006 10410 raft_consensus.cc:493] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:03.690821  1749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:03:03.691854  1749 raft_consensus.cc:2468] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 23.
I20260709 13:03:03.571957 10267 raft_consensus.cc:515] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:03:03.672123 10347 leader_election.cc:290] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:03:03.711795  1571 leader_election.cc:304] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701; no voters: 
W20260709 13:03:03.729682 10433 log.cc:927] Time spent T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.121s	user 0.000s	sys 0.000s
W20260709 13:03:03.777388  2298 consensus_peers.cc:597] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
W20260709 13:03:03.779706  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 68012.9us
W20260709 13:03:03.779311  2298 consensus_peers.cc:597] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 13:03:03.784767  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7e5e9266614676a9c05c398cf7ba9e" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:03:03.784739  2298 consensus_peers.cc:597] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260709 13:03:03.790465  2298 consensus_peers.cc:597] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 13:03:03.790591 10304 raft_consensus.cc:493] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:03:03.788533 10303 raft_consensus.cc:2804] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 13:03:03.792188 10303 raft_consensus.cc:493] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:03.792558 10303 raft_consensus.cc:3060] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 23 FOLLOWER]: Advancing to term 24
W20260709 13:03:03.773926 10423 log.cc:927] Time spent T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.083s	user 0.000s	sys 0.001s
I20260709 13:03:03.806803  1571 leader_election.cc:304] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
W20260709 13:03:03.806228  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 76581.5us
W20260709 13:03:03.809439  1643 consensus_peers.cc:597] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 13:03:03.783727 10411 raft_consensus.cc:493] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:03.712435 10410 raft_consensus.cc:515] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:03:03.833498  1672 raft_consensus.cc:1275] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 24 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 22 index: 14. Preceding OpId from leader: term: 24 index: 15. (index mismatch)
I20260709 13:03:03.835965  1680 raft_consensus.cc:1275] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 18 index: 14. Preceding OpId from leader: term: 20 index: 15. (index mismatch)
I20260709 13:03:03.791183 10304 raft_consensus.cc:515] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:03:03.854611 10216 consensus_queue.cc:1048] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 13:03:03.837102 10359 raft_consensus.cc:2749] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:03:03.843640 10411 raft_consensus.cc:515] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:03:03.897059  1643 consensus_peers.cc:597] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 13:03:03.865908  9920 consensus_queue.cc:1048] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.034s
I20260709 13:03:03.918517 10267 leader_election.cc:290] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:03:03.919205  2333 raft_consensus.cc:1275] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 20 index: 15. (index mismatch)
I20260709 13:03:03.922833   431 tablet_replica.cc:333] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a: stopping tablet replica
I20260709 13:03:03.924054   431 raft_consensus.cc:2243] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Raft consensus shutting down.
I20260709 13:03:03.924772   431 raft_consensus.cc:2272] T 58d4fdd02c6d4667aa718fb146a76505 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Raft consensus is shut down!
I20260709 13:03:03.928490 10410 leader_election.cc:290] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:03:03.932068  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:03:03.935197 10304 leader_election.cc:290] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:03:03.939639 10411 leader_election.cc:290] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:03:03.943312  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "388afdca3ce84bf0a8ce3331ed0d9b2f" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:03:03.945410  1569 leader_election.cc:304] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
W20260709 13:03:03.955438  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 120554us
I20260709 13:03:03.966555  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:03:03.967417  1673 raft_consensus.cc:2468] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 23.
I20260709 13:03:03.982028  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:03:03.964504 10418 log.cc:927] Time spent T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.055s	user 0.000s	sys 0.000s
I20260709 13:03:03.954916 10411 raft_consensus.cc:2749] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:03:03.948534 10245 raft_consensus.cc:493] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:03:03.948088 10203 raft_consensus.cc:493] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:03:03.996397  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:03:03.983166  2333 raft_consensus.cc:2468] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 19.
I20260709 13:03:04.008890  1752 raft_consensus.cc:2393] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 24: Already voted for candidate af520b67945d443b901bbeca37cc5701 in this term.
I20260709 13:03:03.955677 10414 consensus_queue.cc:1048] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.053s
I20260709 13:03:04.033021 10277 raft_consensus.cc:493] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:04.033655 10277 raft_consensus.cc:515] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:03:04.035914 10277 leader_election.cc:290] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:03:04.011644 10393 raft_consensus.cc:493] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:04.058138  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:03:04.009419 10203 raft_consensus.cc:515] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:03:04.010283 10245 raft_consensus.cc:515] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:03:04.065054   431 tablet_replica.cc:333] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a: stopping tablet replica
I20260709 13:03:04.094203   431 raft_consensus.cc:2243] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 24 FOLLOWER]: Raft consensus shutting down.
I20260709 13:03:04.094890   431 raft_consensus.cc:2272] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [term 24 FOLLOWER]: Raft consensus is shut down!
I20260709 13:03:04.053092 10411 raft_consensus.cc:493] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:04.052127 10303 raft_consensus.cc:515] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:03:04.137764  1718 leader_election.cc:304] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:03:04.063817 10393 raft_consensus.cc:515] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:03:04.149681  2332 raft_consensus.cc:2468] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 26.
I20260709 13:03:04.148670 10353 raft_consensus.cc:2804] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:03:04.175988  1569 leader_election.cc:304] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
W20260709 13:03:04.178289  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 300658us
I20260709 13:03:04.105260 10411 raft_consensus.cc:515] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:03:04.243109  1569 raft_consensus.cc:2686] T bb255e8a61e7458babbbdf0183773a81 P 83595ad0fd5940c0a2bfc7c61acd969a: unable to run election callback: Service unavailable: Thread pool token was shut down
W20260709 13:03:04.282678  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 106839us
W20260709 13:03:04.253541  1717 consensus_peers.cc:597] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 13:03:04.270783 10303 leader_election.cc:290] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 24 election: Requested vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:03:04.275282 10393 leader_election.cc:290] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:03:04.152992 10353 raft_consensus.cc:493] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
W20260709 13:03:04.305140  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 274110us
W20260709 13:03:04.309734  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 56281.2us
I20260709 13:03:04.318676  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:03:04.319706  1678 raft_consensus.cc:2468] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 26.
I20260709 13:03:04.293529   431 tablet_replica.cc:333] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a: stopping tablet replica
I20260709 13:03:04.327337   431 raft_consensus.cc:2243] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 24 FOLLOWER]: Raft consensus shutting down.
I20260709 13:03:04.334082   431 raft_consensus.cc:2272] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 24 FOLLOWER]: Raft consensus is shut down!
I20260709 13:03:04.321430 10353 raft_consensus.cc:3060] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:03:04.321463 10245 leader_election.cc:290] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:03:04.348374  2299 leader_election.cc:304] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:03:04.394807  1500 catalog_manager.cc:5697] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d reported cstate change: term changed from 8 to 14, leader changed from af520b67945d443b901bbeca37cc5701 (127.0.107.195) to 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194). New cstate: current_term: 14 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: HEALTHY } } }
I20260709 13:03:04.360559 10245 raft_consensus.cc:2804] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 13:03:04.417886   431 tablet_replica.cc:333] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a: stopping tablet replica
I20260709 13:03:04.419018   431 raft_consensus.cc:2243] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Raft consensus shutting down.
I20260709 13:03:04.419688   431 raft_consensus.cc:2272] T 127630d60f944b899508c92ee18fdddf P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Raft consensus is shut down!
I20260709 13:03:04.421082  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:03:04.421484 10449 log.cc:927] Time spent T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701: Append to log took a long time: real 0.372s	user 0.000s	sys 0.010s
I20260709 13:03:04.421926  1750 raft_consensus.cc:2468] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 22.
W20260709 13:03:04.426857  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 86315.4us
W20260709 13:03:04.428696  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 99935.8us
W20260709 13:03:04.409749 10457 log.cc:927] Time spent T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.056s	user 0.000s	sys 0.000s
I20260709 13:03:04.409911 10245 raft_consensus.cc:493] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:04.446751   431 tablet_replica.cc:333] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a: stopping tablet replica
I20260709 13:03:04.448099   431 raft_consensus.cc:2243] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 24 LEADER]: Raft consensus shutting down.
I20260709 13:03:04.441738 10412 raft_consensus.cc:493] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:04.454092 10412 raft_consensus.cc:515] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:03:04.460170 10246 raft_consensus.cc:493] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:04.460779 10246 raft_consensus.cc:515] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } }
I20260709 13:03:04.461615 10411 leader_election.cc:290] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:03:04.462468   431 raft_consensus.cc:2272] T b9bb5648969b4299a776843d72043448 P 83595ad0fd5940c0a2bfc7c61acd969a [term 24 FOLLOWER]: Raft consensus is shut down!
I20260709 13:03:04.479522  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 27 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:03:04.463095 10246 leader_election.cc:290] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:03:04.484786  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:03:04.485404  2328 raft_consensus.cc:3060] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Advancing to term 24
I20260709 13:03:04.469249  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:03:04.470567 10245 raft_consensus.cc:3060] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 26 FOLLOWER]: Advancing to term 27
I20260709 13:03:04.471560  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 23 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:03:04.414961 10203 leader_election.cc:290] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:03:04.477720  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:03:04.496606 10408 raft_consensus.cc:493] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:03:04.510732 10408 raft_consensus.cc:515] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:03:04.513115 10408 leader_election.cc:290] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:03:04.504803  2298 consensus_peers.cc:597] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 13:03:04.523675 10267 raft_consensus.cc:493] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:04.524166 10267 raft_consensus.cc:515] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:03:04.526798  1752 raft_consensus.cc:1275] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Refusing update from remote peer 83595ad0fd5940c0a2bfc7c61acd969a: Log matching property violated. Preceding OpId in replica: term: 20 index: 13. Preceding OpId from leader: term: 24 index: 15. (index mismatch)
I20260709 13:03:04.529902  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:03:04.530486  1751 raft_consensus.cc:3060] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Advancing to term 24
I20260709 13:03:04.536602  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:03:04.537248  1752 raft_consensus.cc:2468] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 24.
I20260709 13:03:04.542776  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 23 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:03:04.544533 10412 leader_election.cc:290] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:03:04.549480 10267 leader_election.cc:290] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:03:04.567756 10246 raft_consensus.cc:493] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:04.568463 10246 raft_consensus.cc:515] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:03:04.570873 10246 leader_election.cc:290] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:03:04.572652  1494 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35444 (request call id 265) took 24569 ms (client timeout 15000 ms). Trace:
W20260709 13:03:04.577178  1569 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 80880.3us
I20260709 13:03:04.578187  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13031f9e955e4f5fb289c79ee2c46108" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:03:04.495869  2301 leader_election.cc:304] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:03:04.597648  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 23 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
W20260709 13:03:04.597934  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 102168us
I20260709 13:03:04.591872   431 tablet_replica.cc:333] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a: stopping tablet replica
W20260709 13:03:04.599774  2298 leader_election.cc:336] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:03:04.599927 10267 raft_consensus.cc:493] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:03:04.602284 10063 raft_consensus.cc:2804] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 13:03:04.604542 10063 raft_consensus.cc:493] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:03:04.605034 10063 raft_consensus.cc:3060] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Advancing to term 23
W20260709 13:03:04.614269  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 99889.2us
I20260709 13:03:04.620489  1644 leader_election.cc:304] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
W20260709 13:03:04.621841  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 268520us
I20260709 13:03:04.631891  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 27 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:03:04.590946  1494 rpcz_store.cc:269] 0709 13:02:40.003695 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:54.654779 (+14651084us) service_pool.cc:224] Handling call
0709 13:02:55.179625 (+524846us) spinlock_profiling.cc:243] Waited 6.24 ms on lock 0x7b5400002490. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f426303db71 00007f426303e0b2 00007f426bfb7a89 00007f426bf4fda6 00007f426bf21088 00007f426c01f37e 00007f426a4e02a4 00007f426a4e0232 00007f426a4e0161 00007f426a4e00dc
0709 13:03:04.572534 (+9392909us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 13:03:00.625887 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 13:03:00.865399 (+239512us) write_op.cc:432] Acquiring schema lock in shared mode
0709 13:03:00.865484 (+    85us) write_op.cc:435] Acquired schema lock
0709 13:03:00.865511 (+    27us) tablet.cc:662] Decoding operations
0709 13:03:01.056407 (+190896us) write_op.cc:620] Acquiring the partition lock for write op
0709 13:03:01.086749 (+ 30342us) write_op.cc:641] Partition lock acquired for write op
0709 13:03:01.090493 (+  3744us) tablet.cc:685] Acquiring locks for 25 operations
0709 13:03:01.314505 (+224012us) tablet.cc:701] Row locks acquired
0709 13:03:01.314561 (+    56us) write_op.cc:260] PREPARE: finished
0709 13:03:01.314772 (+   211us) write_op.cc:270] Start()
0709 13:03:01.314955 (+   183us) write_op.cc:276] Timestamp: P: 1783602181314700 usec, L: 0
0709 13:03:01.314993 (+    38us) op_driver.cc:348] REPLICATION: starting
0709 13:03:01.315870 (+   877us) log.cc:844] Serialized 9828 byte log entry
0709 13:03:01.746376 (+430506us) op_driver.cc:464] REPLICATION: finished
0709 13:03:02.116157 (+369781us) write_op.cc:301] APPLY: starting
0709 13:03:02.116328 (+   171us) tablet.cc:1367] starting BulkCheckPresence
0709 13:03:02.466292 (+349964us) tablet.cc:1370] finished BulkCheckPresence
0709 13:03:02.466427 (+   135us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 13:03:03.102259 (+635832us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 13:03:03.102720 (+   461us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=50,key_file_lookups=50,delta_file_lookups=0,mrs_lookups=0
0709 13:03:03.102768 (+    48us) write_op.cc:312] APPLY: finished
0709 13:03:03.103358 (+   590us) log.cc:844] Serialized 220 byte log entry
0709 13:03:03.320757 (+217399us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:03:03.363080 (+ 42323us) write_op.cc:454] Released schema lock
0709 13:03:03.363224 (+   144us) write_op.cc:341] FINISH: Updating metrics
0709 13:03:03.363873 (+   649us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:03:03.363906 (+    33us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":54669,"reported_tablets":48,"spinlock_wait_cycles":14361984,"child_traces":[["op",{"apply.queue_time_us":333204,"apply.run_cpu_time_us":82260,"apply.run_wall_time_us":1251336,"cfile_cache_hit":102,"cfile_cache_hit_bytes":108827,"cfile_init":1,"lbm_read_time_us":17785,"lbm_reads_10-100_ms":1,"lbm_reads_lt_1ms":3,"num_ops":25,"prepare.queue_time_us":297648,"prepare.run_cpu_time_us":45517,"prepare.run_wall_time_us":896142,"raft.queue_time_us":229631,"raft.run_cpu_time_us":14344,"raft.run_wall_time_us":175602,"replication_time_us":431149,"spinlock_wait_cycles":1920,"thread_start_us":837758,"threads_started":5,"wal-append.queue_time_us":473973,"wal-append.run_cpu_time_us":20726,"wal-append.run_wall_time_us":1187200}]]}
I20260709 13:03:04.639243  2328 raft_consensus.cc:2468] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 24.
I20260709 13:03:04.640110  1751 raft_consensus.cc:2468] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83595ad0fd5940c0a2bfc7c61acd969a in term 24.
I20260709 13:03:04.604563 10267 raft_consensus.cc:515] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:03:04.652136   431 raft_consensus.cc:2243] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Raft consensus shutting down.
I20260709 13:03:04.650246  1571 leader_election.cc:304] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:03:04.660545  1569 leader_election.cc:304] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 
W20260709 13:03:04.661209  1569 raft_consensus.cc:2686] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 83595ad0fd5940c0a2bfc7c61acd969a: unable to run election callback: Service unavailable: Thread pool token was shut down
I20260709 13:03:04.629161 10462 raft_consensus.cc:2804] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 13:03:04.659946  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592fa647ae2545e09ad5ee25f8cb658e" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 23 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:03:04.661309  1571 leader_election.cc:304] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:03:04.657778 10345 raft_consensus.cc:2749] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:03:04.674026 10466 raft_consensus.cc:493] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 24 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:03:04.674657 10466 raft_consensus.cc:515] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:03:04.678939 10466 leader_election.cc:290] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:03:04.668768  2298 leader_election.cc:336] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:03:04.669762  1569 leader_election.cc:304] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:03:04.684168  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:03:04.685040  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e947690dcb3849cb9f9731123d363b91" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:03:04.683532  1571 leader_election.cc:304] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:03:04.696221 10447 raft_consensus.cc:2749] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:03:04.699102  2330 raft_consensus.cc:2393] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 26: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:03:04.665886 10462 raft_consensus.cc:493] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:04.707679 10462 raft_consensus.cc:3060] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 26 FOLLOWER]: Advancing to term 27
I20260709 13:03:04.701232   431 pending_rounds.cc:70] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a: Trying to abort 1 pending ops.
I20260709 13:03:04.722370   431 pending_rounds.cc:77] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a: Aborting op as it isn't in flight: id { term: 19 index: 15 } timestamp: 7305634490801647616 op_type: NO_OP noop_request { }
I20260709 13:03:04.722846   431 raft_consensus.cc:2889] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260709 13:03:04.681963 10298 raft_consensus.cc:2749] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:03:04.722098 10462 raft_consensus.cc:515] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:03:04.726727 10462 leader_election.cc:290] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 27 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:03:04.727886 10063 raft_consensus.cc:515] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:03:04.656018 10359 consensus_queue.cc:1048] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.029s
I20260709 13:03:04.703054 10360 raft_consensus.cc:2749] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 13:03:04.732865 10267 leader_election.cc:290] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:03:04.723222   431 raft_consensus.cc:2272] T 84e78b66ceec4f759ae9e101a5469e4c P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Raft consensus is shut down!
I20260709 13:03:04.760717 10063 leader_election.cc:290] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 23 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:03:04.742688  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
W20260709 13:03:04.746354  2298 consensus_peers.cc:597] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 13:03:04.748178 10353 raft_consensus.cc:515] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:03:04.722111 10245 raft_consensus.cc:515] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:03:04.767444  1571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 64068.8us
I20260709 13:03:04.771477  1676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:03:04.772218  1676 raft_consensus.cc:2468] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 26.
I20260709 13:03:04.765370   431 tablet_replica.cc:333] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a: stopping tablet replica
W20260709 13:03:04.771875  1717 leader_election.cc:336] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:03:04.778501  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:03:04.779178  2328 raft_consensus.cc:2468] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 25.
I20260709 13:03:04.783545 10353 leader_election.cc:290] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 20 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:03:04.786365  2298 leader_election.cc:336] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:03:04.786924  2298 leader_election.cc:304] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
W20260709 13:03:04.788365  2298 consensus_peers.cc:597] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 13:03:04.790902   431 raft_consensus.cc:2243] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Raft consensus shutting down.
I20260709 13:03:04.791643   431 raft_consensus.cc:2272] T 5108457796a949b7b753ab22b575b7a0 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Raft consensus is shut down!
I20260709 13:03:04.794727 10204 raft_consensus.cc:2749] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 13:03:04.799063  2330 raft_consensus.cc:2393] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 27: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:03:04.816574  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ab3300be1e4a59b64b14d50da193ca" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:03:04.818037  1751 raft_consensus.cc:3060] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Advancing to term 23
W20260709 13:03:04.821352  2298 leader_election.cc:336] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 23 election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:03:04.832523  1718 leader_election.cc:304] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:03:04.833577 10353 raft_consensus.cc:2804] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 13:03:04.833989 10353 raft_consensus.cc:493] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 25 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:03:04.834342 10353 raft_consensus.cc:3060] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 25 FOLLOWER]: Advancing to term 26
I20260709 13:03:04.839346   431 tablet_replica.cc:333] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a: stopping tablet replica
I20260709 13:03:04.840451   431 raft_consensus.cc:2243] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 23 FOLLOWER]: Raft consensus shutting down.
I20260709 13:03:04.841140   431 raft_consensus.cc:2272] T 77d66b92688a42e3ad231959a644db56 P 83595ad0fd5940c0a2bfc7c61acd969a [term 23 FOLLOWER]: Raft consensus is shut down!
I20260709 13:03:04.844621 10353 raft_consensus.cc:515] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:03:04.851734  1751 raft_consensus.cc:2468] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 23.
I20260709 13:03:04.864302  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:03:04.865773  1673 raft_consensus.cc:2468] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 23.
W20260709 13:03:04.867010  1643 consensus_peers.cc:597] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 13:03:04.869971  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1809d674329a492dbea0b60a4703fe51" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:03:04.870887  2328 raft_consensus.cc:3060] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:03:04.878844  1718 leader_election.cc:304] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1
W20260709 13:03:04.879264  2298 leader_election.cc:336] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 27 election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:03:04.882095  2328 raft_consensus.cc:2468] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 20.
I20260709 13:03:04.886015 10245 leader_election.cc:290] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 27 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:03:04.893532 10353 leader_election.cc:290] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 26 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:03:04.881896  2301 leader_election.cc:304] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:03:04.898784  1673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:03:04.901420  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f3cdc0abae4f74aad5e4a15cb02e1b" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:03:04.902140  2328 raft_consensus.cc:3060] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 25 FOLLOWER]: Advancing to term 26
I20260709 13:03:04.910329  1673 raft_consensus.cc:2393] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b879799899a499c8d409ee9f1277cd1 in current term 27: Already voted for candidate 816fadcea8fc4893882f5f9b9f22a03d in this term.
W20260709 13:03:04.917760  1717 consensus_peers.cc:597] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 13:03:04.918797  2328 raft_consensus.cc:2468] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 26.
I20260709 13:03:04.922740  2299 leader_election.cc:304] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:03:04.922273  1718 leader_election.cc:304] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:03:04.904738 10267 raft_consensus.cc:2804] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 13:03:04.926775  1718 leader_election.cc:304] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:03:04.927825 10247 raft_consensus.cc:2804] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 13:03:04.928375 10247 raft_consensus.cc:697] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 20 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:03:04.928581 10353 raft_consensus.cc:2804] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 26 FOLLOWER]: Leader election won for term 26
I20260709 13:03:04.929311 10353 raft_consensus.cc:697] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 26 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:03:04.929388 10247 consensus_queue.cc:237] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.14, Last appended by leader: 14, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:03:04.920593 10063 raft_consensus.cc:2804] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 13:03:04.940250 10063 raft_consensus.cc:697] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 23 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:03:04.942525 10452 raft_consensus.cc:2749] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 13:03:04.935089   431 tablet_replica.cc:333] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a: stopping tablet replica
I20260709 13:03:04.935478 10267 raft_consensus.cc:493] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 25 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
W20260709 13:03:04.954939  1643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 87072us
W20260709 13:03:04.960726  1643 consensus_peers.cc:597] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 13:03:04.941684 10063 consensus_queue.cc:237] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.15, Last appended by leader: 15, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:03:04.953670 10267 raft_consensus.cc:3060] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 25 FOLLOWER]: Advancing to term 26
I20260709 13:03:04.966482   431 raft_consensus.cc:2243] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 28 LEADER]: Raft consensus shutting down.
I20260709 13:03:04.973268  1718 leader_election.cc:304] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
W20260709 13:03:04.975067  1717 leader_election.cc:336] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:03:04.976011 10357 raft_consensus.cc:2804] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 13:03:04.964094 10353 consensus_queue.cc:237] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 25.17, Last appended by leader: 17, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:03:04.992247  1643 leader_election.cc:336] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:03:04.992798  1643 leader_election.cc:304] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:03:04.994029 10310 raft_consensus.cc:3060] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Advancing to term 24
I20260709 13:03:04.984330 10357 raft_consensus.cc:493] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:03:04.999571  1643 consensus_peers.cc:597] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 13:03:05.000898 10357 raft_consensus.cc:3060] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Advancing to term 25
I20260709 13:03:05.026068   431 raft_consensus.cc:2272] T dce8db7565da4e13a82ffbb9c39fde78 P 83595ad0fd5940c0a2bfc7c61acd969a [term 28 FOLLOWER]: Raft consensus is shut down!
I20260709 13:03:05.029721   431 tablet_replica.cc:333] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a: stopping tablet replica
W20260709 13:03:05.035382  1717 leader_election.cc:336] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 13:03:05.038050  1717 leader_election.cc:336] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 20 election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:03:05.040032   431 raft_consensus.cc:2243] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 26 FOLLOWER]: Raft consensus shutting down.
I20260709 13:03:05.040829   431 raft_consensus.cc:2272] T 0615de9bff034ecb9ed6173433128ff5 P 83595ad0fd5940c0a2bfc7c61acd969a [term 26 FOLLOWER]: Raft consensus is shut down!
W20260709 13:03:05.043342  1717 leader_election.cc:336] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 26 election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:03:05.044452   431 tablet_replica.cc:333] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a: stopping tablet replica
I20260709 13:03:05.045447   431 raft_consensus.cc:2243] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 23 LEADER]: Raft consensus shutting down.
I20260709 13:03:05.046739   431 raft_consensus.cc:2272] T cd44a2ebdc764e898f25ad5390f4bffd P 83595ad0fd5940c0a2bfc7c61acd969a [term 23 FOLLOWER]: Raft consensus is shut down!
W20260709 13:03:05.034225  1643 leader_election.cc:336] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:03:05.042598 10310 raft_consensus.cc:2749] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 13:03:05.063524 10267 raft_consensus.cc:515] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:03:05.052531  1643 consensus_peers.cc:597] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 13:03:05.093410 10267 leader_election.cc:290] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 26 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:03:05.094532  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:03:05.095521  2328 raft_consensus.cc:2393] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 26: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:03:05.095105 10357 raft_consensus.cc:515] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:03:05.097553  1643 consensus_peers.cc:597] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 13:03:05.115661 10357 leader_election.cc:290] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 25 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 13:03:05.114547  1643 leader_election.cc:336] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:03:05.122418  1643 leader_election.cc:304] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:03:05.123225   431 tablet_replica.cc:333] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a: stopping tablet replica
I20260709 13:03:05.124343 10310 raft_consensus.cc:2749] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 13:03:05.124939   431 raft_consensus.cc:2243] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 LEADER]: Raft consensus shutting down.
I20260709 13:03:05.126260   431 raft_consensus.cc:2272] T 83226095f79a46dab445849b1febd425 P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Raft consensus is shut down!
W20260709 13:03:05.126683  1717 leader_election.cc:336] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 25 election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 13:03:05.127444  1643 leader_election.cc:336] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 27 election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:03:05.127907  1643 leader_election.cc:304] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:03:05.128732 10310 raft_consensus.cc:2749] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 13:03:05.138799  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:03:05.151576 10414 raft_consensus.cc:493] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:05.152333 10414 raft_consensus.cc:515] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:03:05.155395  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb255e8a61e7458babbbdf0183773a81" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:03:05.155591  1678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:03:05.156137  1680 raft_consensus.cc:3060] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 24 FOLLOWER]: Advancing to term 25
I20260709 13:03:05.160185 10414 leader_election.cc:290] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:03:05.175747  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a92588e9316e4b4aa1691e8669c839d3" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:03:05.177739 10414 raft_consensus.cc:493] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:05.181775 10414 raft_consensus.cc:515] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:03:05.188577  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 20 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:03:05.191401 10414 leader_election.cc:290] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:03:05.192032  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 20 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:03:05.193841  1569 leader_election.cc:304] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:03:05.195197 10414 raft_consensus.cc:2749] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:03:05.196565   431 tablet_replica.cc:333] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a: stopping tablet replica
I20260709 13:03:05.197862 10063 raft_consensus.cc:493] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:05.198457 10063 raft_consensus.cc:515] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:03:05.196776  1500 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35458 (request call id 309) took 91753 ms (client timeout 15000 ms). Trace:
I20260709 13:03:05.200816 10063 leader_election.cc:290] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:03:05.201480  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:03:05.201836  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:03:05.211226  1748 raft_consensus.cc:2468] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 26.
I20260709 13:03:05.200197  1494 catalog_manager.cc:5697] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 reported cstate change: term changed from 15 to 22. New cstate: current_term: 22 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 13:03:05.212420  2301 leader_election.cc:304] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:03:05.213269 10393 raft_consensus.cc:2804] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 13:03:05.213676 10393 raft_consensus.cc:493] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:05.214278 10393 raft_consensus.cc:3060] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 26 FOLLOWER]: Advancing to term 27
I20260709 13:03:05.217553   431 raft_consensus.cc:2243] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Raft consensus shutting down.
I20260709 13:03:05.218236   431 pending_rounds.cc:70] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a: Trying to abort 3 pending ops.
I20260709 13:03:05.218528   431 pending_rounds.cc:77] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a: Aborting op as it isn't in flight: id { term: 20 index: 12 } timestamp: 7305634455743971328 op_type: NO_OP noop_request { }
I20260709 13:03:05.224316   431 raft_consensus.cc:2889] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260709 13:03:05.224687   431 pending_rounds.cc:77] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a: Aborting op as it isn't in flight: id { term: 19 index: 11 } timestamp: 7305634438271397888 op_type: NO_OP noop_request { }
I20260709 13:03:05.225029   431 raft_consensus.cc:2889] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260709 13:03:05.225301   431 pending_rounds.cc:77] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a: Aborting op as it isn't in flight: id { term: 17 index: 10 } timestamp: 7305634398118977536 op_type: NO_OP noop_request { }
I20260709 13:03:05.225616   431 raft_consensus.cc:2889] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260709 13:03:05.225975   431 raft_consensus.cc:2272] T fe7e5e9266614676a9c05c398cf7ba9e P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Raft consensus is shut down!
I20260709 13:03:05.221015  1494 catalog_manager.cc:5697] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 reported cstate change: term changed from 15 to 27, leader changed from 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) to 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194). New cstate: current_term: 27 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 13:03:05.204236  1571 leader_election.cc:304] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:03:05.230716 10414 raft_consensus.cc:2749] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:03:05.222643  1680 raft_consensus.cc:2468] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 25.
I20260709 13:03:05.223011  1674 raft_consensus.cc:2468] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 26.
I20260709 13:03:05.234354  1718 leader_election.cc:304] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:03:05.235806  1718 leader_election.cc:304] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1
I20260709 13:03:05.236819 10317 raft_consensus.cc:2804] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 25 FOLLOWER]: Leader election won for term 25
I20260709 13:03:05.237340 10317 raft_consensus.cc:697] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 25 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:03:05.237540 10267 raft_consensus.cc:2804] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 26 FOLLOWER]: Leader election won for term 26
I20260709 13:03:05.238199 10267 raft_consensus.cc:697] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 26 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:03:05.239319 10267 consensus_queue.cc:237] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 25.18, Last appended by leader: 18, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:03:05.210584  1500 rpcz_store.cc:269] 0709 13:01:33.443439 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:33.445458 (+  2019us) service_pool.cc:224] Handling call
0709 13:03:05.196651 (+91751193us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 13:03:04.859749 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 13:03:04.875553 (+ 15804us) write_op.cc:432] Acquiring schema lock in shared mode
0709 13:03:04.875615 (+    62us) write_op.cc:435] Acquired schema lock
0709 13:03:04.875634 (+    19us) tablet.cc:662] Decoding operations
0709 13:03:04.876282 (+   648us) write_op.cc:620] Acquiring the partition lock for write op
0709 13:03:04.876374 (+    92us) write_op.cc:641] Partition lock acquired for write op
0709 13:03:04.876404 (+    30us) tablet.cc:685] Acquiring locks for 4 operations
0709 13:03:04.876836 (+   432us) tablet.cc:701] Row locks acquired
0709 13:03:04.876855 (+    19us) write_op.cc:260] PREPARE: finished
0709 13:03:04.877005 (+   150us) write_op.cc:270] Start()
0709 13:03:04.877139 (+   134us) write_op.cc:276] Timestamp: P: 1783602184876958 usec, L: 0
0709 13:03:04.877161 (+    22us) op_driver.cc:348] REPLICATION: starting
0709 13:03:04.884968 (+  7807us) log.cc:844] Serialized 1696 byte log entry
0709 13:03:05.023261 (+138293us) op_driver.cc:464] REPLICATION: finished
0709 13:03:05.091849 (+ 68588us) spinlock_profiling.cc:243] Waited 11.3 ms on lock 0x7b18001ddd60. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f42630753f3 00007f42630777c1 00007f4263077779 00007f4263077709 00007f42630776d1 00007f42630769cd 000000000038f354 000000000038f199 00007f426305883a
0709 13:03:05.092024 (+   175us) write_op.cc:301] APPLY: starting
0709 13:03:05.092111 (+    87us) tablet.cc:1367] starting BulkCheckPresence
0709 13:03:05.102800 (+ 10689us) tablet.cc:1370] finished BulkCheckPresence
0709 13:03:05.102824 (+    24us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 13:03:05.107313 (+  4489us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 13:03:05.111128 (+  3815us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=8,key_file_lookups=8,delta_file_lookups=0,mrs_lookups=0
0709 13:03:05.121092 (+  9964us) write_op.cc:312] APPLY: finished
0709 13:03:05.121376 (+   284us) log.cc:844] Serialized 49 byte log entry
0709 13:03:05.121577 (+   201us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:03:05.121715 (+   138us) write_op.cc:454] Released schema lock
0709 13:03:05.121774 (+    59us) write_op.cc:341] FINISH: Updating metrics
0709 13:03:05.122305 (+   531us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:03:05.122342 (+    37us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":833673,"reported_tablets":47,"spinlock_wait_cycles":1313920,"child_traces":[["op",{"apply.queue_time_us":55896,"apply.run_cpu_time_us":7019,"apply.run_wall_time_us":30236,"cfile_cache_hit":18,"cfile_cache_hit_bytes":20879,"num_ops":4,"prepare.queue_time_us":297659,"prepare.run_cpu_time_us":18549,"prepare.run_wall_time_us":87121,"raft.queue_time_us":32624,"raft.run_cpu_time_us":5127,"raft.run_wall_time_us":37650,"replication_time_us":145914,"spinlock_wait_cycles":26072448,"thread_start_us":230462,"threads_started":4,"wal-append.queue_time_us":78236}]]}
I20260709 13:03:05.234336  1494 catalog_manager.cc:5697] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 reported cstate change: term changed from 16 to 20, leader changed from 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) to 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196). New cstate: current_term: 20 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:03:05.254134  1494 catalog_manager.cc:5697] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 reported cstate change: term changed from 21 to 23. New cstate: current_term: 23 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:03:05.255817  1494 catalog_manager.cc:5697] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 reported cstate change: term changed from 18 to 22, leader changed from 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) to 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193). New cstate: current_term: 22 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 13:03:05.257571  1494 catalog_manager.cc:5697] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 reported cstate change: term changed from 22 to 26. New cstate: current_term: 26 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:03:05.259884  1494 catalog_manager.cc:5697] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 reported cstate change: config changed from index 5 to 15, term changed from 22 to 25, leader changed from 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) to <none>, VOTER 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) evicted. New cstate: current_term: 25 committed_config { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 13:03:05.250190 10393 raft_consensus.cc:515] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:03:05.270471 10393 leader_election.cc:290] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 27 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:03:05.271526  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:03:05.238350 10317 consensus_queue.cc:237] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 23.15, Last appended by leader: 15, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:03:05.278594  1494 catalog_manager.cc:5697] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 reported cstate change: term changed from 25 to 28, leader changed from af520b67945d443b901bbeca37cc5701 (127.0.107.195) to 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193). New cstate: current_term: 28 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 13:03:05.279539  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513043a9b96c4f70adad53ac4b5d01dc" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
W20260709 13:03:05.280731  1717 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 218 similar messages]
I20260709 13:03:05.284822  1674 raft_consensus.cc:3060] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 26 FOLLOWER]: Advancing to term 27
I20260709 13:03:05.296113  1674 raft_consensus.cc:2468] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 27.
I20260709 13:03:05.298377  2299 leader_election.cc:304] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: af520b67945d443b901bbeca37cc5701
I20260709 13:03:05.296229   431 tablet_replica.cc:333] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a: stopping tablet replica
I20260709 13:03:05.287271  1494 catalog_manager.cc:5697] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 reported cstate change: term changed from 15 to 24. New cstate: current_term: 24 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:03:05.300118  1494 catalog_manager.cc:5697] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 reported cstate change: term changed from 12 to 19, leader changed from 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) to 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193). New cstate: current_term: 19 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 13:03:05.301786  1494 catalog_manager.cc:5697] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 reported cstate change: term changed from 17 to 23, leader changed from 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) to af520b67945d443b901bbeca37cc5701 (127.0.107.195). New cstate: current_term: 23 leader_uuid: "af520b67945d443b901bbeca37cc5701" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 13:03:05.303166  1494 catalog_manager.cc:5697] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 reported cstate change: term changed from 16 to 20. New cstate: current_term: 20 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:03:05.304510  1494 catalog_manager.cc:5697] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 reported cstate change: term changed from 16 to 23, leader changed from 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) to 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194). New cstate: current_term: 23 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 13:03:05.306593   431 raft_consensus.cc:2243] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 26 FOLLOWER]: Raft consensus shutting down.
I20260709 13:03:05.305995  1494 catalog_manager.cc:5697] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 reported cstate change: term changed from 16 to 18, leader changed from 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194) to 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196). New cstate: current_term: 18 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 13:03:05.323312  2298 consensus_peers.cc:597] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 13:03:05.308574  1494 catalog_manager.cc:5697] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 reported cstate change: term changed from 20 to 24. New cstate: current_term: 24 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:03:05.306687 10393 raft_consensus.cc:2804] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 27 FOLLOWER]: Leader election won for term 27
I20260709 13:03:05.326220  1494 catalog_manager.cc:5697] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 reported cstate change: term changed from 16 to 17, leader changed from 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) to 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194). New cstate: current_term: 17 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 13:03:05.327176 10393 raft_consensus.cc:697] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 27 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:03:05.327781  1494 catalog_manager.cc:5697] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 reported cstate change: term changed from 22 to 25. New cstate: current_term: 25 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 13:03:05.328261 10393 consensus_queue.cc:237] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 25.18, Last appended by leader: 18, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:03:05.329190  1494 catalog_manager.cc:5697] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 reported cstate change: term changed from 20 to 22, leader changed from 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) to 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196). New cstate: current_term: 22 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:03:05.341459  1494 catalog_manager.cc:5697] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 reported cstate change: term changed from 18 to 20, leader changed from 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) to 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194). New cstate: current_term: 20 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
W20260709 13:03:05.345729  1643 consensus_peers.cc:597] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 13:03:05.345021  1494 catalog_manager.cc:5697] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 reported cstate change: term changed from 16 to 25, leader changed from 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194) to 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196). New cstate: current_term: 25 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:03:05.352064  1494 catalog_manager.cc:5697] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 reported cstate change: term changed from 18 to 20, leader changed from 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) to 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196). New cstate: current_term: 20 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:03:05.364090   431 raft_consensus.cc:2272] T e947690dcb3849cb9f9731123d363b91 P 83595ad0fd5940c0a2bfc7c61acd969a [term 26 FOLLOWER]: Raft consensus is shut down!
I20260709 13:03:05.367452   431 tablet_replica.cc:333] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a: stopping tablet replica
I20260709 13:03:05.368368   431 raft_consensus.cc:2243] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Raft consensus shutting down.
I20260709 13:03:05.368968   431 pending_rounds.cc:70] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a: Trying to abort 1 pending ops.
I20260709 13:03:05.369212   431 pending_rounds.cc:77] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a: Aborting op as it isn't in flight: id { term: 19 index: 12 } timestamp: 7305634441395777536 op_type: NO_OP noop_request { }
I20260709 13:03:05.369573   431 raft_consensus.cc:2889] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260709 13:03:05.369946   431 raft_consensus.cc:2272] T 13031f9e955e4f5fb289c79ee2c46108 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Raft consensus is shut down!
I20260709 13:03:05.369784  1494 catalog_manager.cc:5697] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 reported cstate change: term changed from 16 to 19, leader changed from 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) to 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196). New cstate: current_term: 19 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:03:05.371654  1494 catalog_manager.cc:5697] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 reported cstate change: term changed from 12 to 23, leader changed from af520b67945d443b901bbeca37cc5701 (127.0.107.195) to 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196). New cstate: current_term: 23 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:03:05.377956  1494 catalog_manager.cc:5697] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 reported cstate change: term changed from 13 to 26. New cstate: current_term: 26 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:03:05.379483  1494 catalog_manager.cc:5697] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 reported cstate change: term changed from 13 to 21, leader changed from 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) to 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196). New cstate: current_term: 21 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:03:05.395262 10267 raft_consensus.cc:493] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:03:05.396003 10267 raft_consensus.cc:515] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:03:05.402341  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:03:05.408470   431 tablet_replica.cc:333] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a: stopping tablet replica
I20260709 13:03:05.409627   431 raft_consensus.cc:2243] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Raft consensus shutting down.
W20260709 13:03:05.419795  2298 consensus_peers.cc:597] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 13:03:05.423427  2333 raft_consensus.cc:1275] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 19 index: 14. Preceding OpId from leader: term: 20 index: 15. (index mismatch)
I20260709 13:03:05.425546 10317 consensus_queue.cc:1048] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 13:03:05.439261   431 pending_rounds.cc:70] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a: Trying to abort 1 pending ops.
I20260709 13:03:05.439684   431 pending_rounds.cc:77] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a: Aborting op as it isn't in flight: id { term: 18 index: 14 } timestamp: 7305634469334306816 op_type: NO_OP noop_request { }
I20260709 13:03:05.440073   431 raft_consensus.cc:2889] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260709 13:03:05.440447   431 raft_consensus.cc:2272] T e26344297fff44bb8d2099b8e64d7df2 P 83595ad0fd5940c0a2bfc7c61acd969a [term 18 FOLLOWER]: Raft consensus is shut down!
W20260709 13:03:05.441716  1717 leader_election.cc:336] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:03:05.442389  1717 leader_election.cc:304] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:03:05.406831 10267 leader_election.cc:290] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:03:05.447634 10267 raft_consensus.cc:2749] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 13:03:05.484006   431 tablet_replica.cc:333] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a: stopping tablet replica
W20260709 13:03:05.492034  1717 consensus_peers.cc:597] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:03:05.499300  1717 consensus_peers.cc:597] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:03:05.510684  2298 consensus_peers.cc:597] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 13:03:05.522619  2328 raft_consensus.cc:1275] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 26 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 25 index: 17. Preceding OpId from leader: term: 26 index: 18. (index mismatch)
I20260709 13:03:05.561033   431 raft_consensus.cc:2243] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 24 LEADER]: Raft consensus shutting down.
I20260709 13:03:05.612763   431 raft_consensus.cc:2272] T 73dc0a9696be41f78e22f1f77a6eee72 P 83595ad0fd5940c0a2bfc7c61acd969a [term 24 FOLLOWER]: Raft consensus is shut down!
I20260709 13:03:05.621266   431 tablet_replica.cc:333] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a: stopping tablet replica
I20260709 13:03:05.629513   431 raft_consensus.cc:2243] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Raft consensus shutting down.
I20260709 13:03:05.629123 10353 consensus_queue.cc:1048] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.002s
W20260709 13:03:05.623081  2298 consensus_peers.cc:597] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:03:05.643548   431 raft_consensus.cc:2272] T 32dc176efd53437ab7735ea8c6ac4a0d P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Raft consensus is shut down!
I20260709 13:03:05.654155   431 tablet_replica.cc:333] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a: stopping tablet replica
I20260709 13:03:05.656244   431 raft_consensus.cc:2243] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 24 LEADER]: Raft consensus shutting down.
I20260709 13:03:05.665940  1494 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:43476 (request call id 199) took 3283 ms. Trace:
W20260709 13:03:05.667866  1643 consensus_peers.cc:597] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 13:03:05.690625   431 raft_consensus.cc:2272] T 65285571995c4ed5b593041598056a22 P 83595ad0fd5940c0a2bfc7c61acd969a [term 24 FOLLOWER]: Raft consensus is shut down!
I20260709 13:03:05.704540   431 tablet_replica.cc:333] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a: stopping tablet replica
I20260709 13:03:05.706449   431 raft_consensus.cc:2243] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 LEADER]: Raft consensus shutting down.
W20260709 13:03:05.715693  1571 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:05.731309  1674 raft_consensus.cc:1275] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 25 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 23 index: 15. Preceding OpId from leader: term: 25 index: 16. (index mismatch)
I20260709 13:03:05.678891  1494 rpcz_store.cc:276] 0709 13:03:02.382399 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:04.666703 (+2284304us) service_pool.cc:224] Handling call
0709 13:03:05.665663 (+998960us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 13:03:05.389269 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 13:03:05.389852 (+   583us) write_op.cc:432] Acquiring schema lock in shared mode
0709 13:03:05.389908 (+    56us) write_op.cc:435] Acquired schema lock
0709 13:03:05.389932 (+    24us) tablet.cc:662] Decoding operations
0709 13:03:05.391794 (+  1862us) write_op.cc:620] Acquiring the partition lock for write op
0709 13:03:05.391852 (+    58us) write_op.cc:641] Partition lock acquired for write op
0709 13:03:05.391887 (+    35us) tablet.cc:685] Acquiring locks for 25 operations
0709 13:03:05.393858 (+  1971us) tablet.cc:701] Row locks acquired
0709 13:03:05.393871 (+    13us) write_op.cc:260] PREPARE: finished
0709 13:03:05.394016 (+   145us) write_op.cc:270] Start()
0709 13:03:05.394137 (+   121us) write_op.cc:276] Timestamp: P: 1783602185393968 usec, L: 0
0709 13:03:05.394152 (+    15us) op_driver.cc:348] REPLICATION: starting
0709 13:03:05.395037 (+   885us) log.cc:844] Serialized 9695 byte log entry
0709 13:03:05.398012 (+  2975us) op_driver.cc:464] REPLICATION: finished
0709 13:03:05.398538 (+   526us) write_op.cc:301] APPLY: starting
0709 13:03:05.398632 (+    94us) tablet.cc:1367] starting BulkCheckPresence
0709 13:03:05.429158 (+ 30526us) tablet.cc:1370] finished BulkCheckPresence
0709 13:03:05.429270 (+   112us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 13:03:05.462069 (+ 32799us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 13:03:05.486079 (+ 24010us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=50,key_file_lookups=50,delta_file_lookups=0,mrs_lookups=0
0709 13:03:05.486137 (+    58us) write_op.cc:312] APPLY: finished
0709 13:03:05.486681 (+   544us) log.cc:844] Serialized 220 byte log entry
0709 13:03:05.487607 (+   926us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:03:05.496084 (+  8477us) write_op.cc:454] Released schema lock
0709 13:03:05.496170 (+    86us) write_op.cc:341] FINISH: Updating metrics
0709 13:03:05.496656 (+   486us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:03:05.496687 (+    31us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":36,"spinlock_wait_cycles":3328,"child_traces":[["op",{"apply.queue_time_us":273,"apply.run_cpu_time_us":29001,"apply.run_wall_time_us":98049,"cfile_cache_hit":100,"cfile_cache_hit_bytes":104700,"num_ops":25,"prepare.queue_time_us":126,"prepare.run_cpu_time_us":6084,"prepare.run_wall_time_us":12093,"replication_time_us":3710,"spinlock_wait_cycles":342912}]]}
I20260709 13:03:05.734213 10353 consensus_queue.cc:1048] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 13:03:05.738305  1752 raft_consensus.cc:1275] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 22 index: 15. Preceding OpId from leader: term: 23 index: 16. (index mismatch)
I20260709 13:03:05.740026   431 raft_consensus.cc:2272] T 2a521fb6c3e7489d9694fd81cadeb33f P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Raft consensus is shut down!
I20260709 13:03:05.749164 10063 consensus_queue.cc:1048] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.003s
I20260709 13:03:05.760708   431 tablet_replica.cc:333] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a: stopping tablet replica
I20260709 13:03:05.762224   431 raft_consensus.cc:2243] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Raft consensus shutting down.
I20260709 13:03:05.768730   431 pending_rounds.cc:70] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a: Trying to abort 1 pending ops.
W20260709 13:03:05.794507  1717 consensus_peers.cc:597] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
W20260709 13:03:05.795372  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 47927.8us
I20260709 13:03:05.773416   431 pending_rounds.cc:77] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a: Aborting op as it isn't in flight: id { term: 18 index: 16 } timestamp: 7305634488371961856 op_type: NO_OP noop_request { }
I20260709 13:03:05.819273   431 raft_consensus.cc:2889] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260709 13:03:05.827855   431 raft_consensus.cc:2272] T d21485c094584689917b2a97b2d3c624 P 83595ad0fd5940c0a2bfc7c61acd969a [term 19 FOLLOWER]: Raft consensus is shut down!
I20260709 13:03:05.855746  1680 raft_consensus.cc:1275] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 27 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 25 index: 18. Preceding OpId from leader: term: 27 index: 19. (index mismatch)
I20260709 13:03:05.881876   431 tablet_replica.cc:333] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a: stopping tablet replica
I20260709 13:03:05.883227   431 raft_consensus.cc:2243] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Raft consensus shutting down.
I20260709 13:03:05.885258   431 raft_consensus.cc:2272] T a92588e9316e4b4aa1691e8669c839d3 P 83595ad0fd5940c0a2bfc7c61acd969a [term 21 FOLLOWER]: Raft consensus is shut down!
I20260709 13:03:05.891508 10458 consensus_queue.cc:1048] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 13:03:05.890352   431 tablet_replica.cc:333] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a: stopping tablet replica
W20260709 13:03:05.908840  1487 catalog_manager.cc:4755] TS 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 983621a6fc5b4c9ea0159a648dfe7a36: Remote error: Service unavailable: service kudu.tserver.TabletServerAdminService not registered on TabletServer
I20260709 13:03:05.902204   431 raft_consensus.cc:2243] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 26 FOLLOWER]: Raft consensus shutting down.
I20260709 13:03:05.916592   431 pending_rounds.cc:70] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a: Trying to abort 1 pending ops.
I20260709 13:03:05.916872   431 pending_rounds.cc:77] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a: Aborting op as it isn't in flight: id { term: 23 index: 12 } timestamp: 7305634453881225216 op_type: NO_OP noop_request { }
I20260709 13:03:05.917289   431 raft_consensus.cc:2889] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 26 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260709 13:03:05.917734   431 raft_consensus.cc:2272] T b917a1e9c9aa40a0afce5a12c343eacd P 83595ad0fd5940c0a2bfc7c61acd969a [term 26 FOLLOWER]: Raft consensus is shut down!
I20260709 13:03:05.920881  2328 raft_consensus.cc:1240] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 27 LEADER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 26. Current term is 27. Ops: []
I20260709 13:03:05.923054 10409 consensus_queue.cc:1059] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 13:03:05.924021 10409 consensus_queue.cc:1243] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [LEADER]: Peer 3b879799899a499c8d409ee9f1277cd1 log is divergent from this leader: its last log entry 27.19 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 17
I20260709 13:03:05.924728 10317 raft_consensus.cc:3055] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 26 LEADER]: Stepping down as leader of term 26
I20260709 13:03:05.925148 10317 raft_consensus.cc:740] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:03:05.926383 10317 consensus_queue.cc:260] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 26.19, Last appended by leader: 19, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:03:05.927862 10317 raft_consensus.cc:3060] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 26 FOLLOWER]: Advancing to term 27
W20260709 13:03:05.930006  1717 consensus_peers.cc:597] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 13:03:05.945779  1752 raft_consensus.cc:1275] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 27 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 26 index: 19. Preceding OpId from leader: term: 27 index: 19. (term mismatch)
I20260709 13:03:05.946267  1752 pending_rounds.cc:85] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701: Aborting all ops after (but not including) 18
I20260709 13:03:05.947067  1752 pending_rounds.cc:107] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701: Aborting uncommitted NO_OP operation due to leader change: 26.19
I20260709 13:03:05.947363  1752 raft_consensus.cc:2889] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 27 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:03:05.952240  1495 catalog_manager.cc:5697] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 reported cstate change: term changed from 10 to 20. New cstate: current_term: 20 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:03:05.961701   431 tablet_replica.cc:333] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a: stopping tablet replica
I20260709 13:03:05.962459  1495 catalog_manager.cc:5697] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 reported cstate change: term changed from 14 to 22, leader changed from af520b67945d443b901bbeca37cc5701 (127.0.107.195) to 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196). New cstate: current_term: 22 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:03:05.967936 10393 consensus_queue.cc:1048] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 13:03:05.962850   431 raft_consensus.cc:2243] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 26 FOLLOWER]: Raft consensus shutting down.
I20260709 13:03:05.977993   431 pending_rounds.cc:70] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a: Trying to abort 2 pending ops.
I20260709 13:03:05.978257   431 pending_rounds.cc:77] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a: Aborting op as it isn't in flight: id { term: 22 index: 14 } timestamp: 7305634365703692288 op_type: NO_OP noop_request { }
I20260709 13:03:05.978737   431 raft_consensus.cc:2889] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 26 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260709 13:03:05.979063   431 pending_rounds.cc:77] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a: Aborting op as it isn't in flight: id { term: 20 index: 13 } timestamp: 7305634333124009984 op_type: NO_OP noop_request { }
I20260709 13:03:05.983878   431 raft_consensus.cc:2889] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 26 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260709 13:03:05.990566   431 raft_consensus.cc:2272] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 83595ad0fd5940c0a2bfc7c61acd969a [term 26 FOLLOWER]: Raft consensus is shut down!
I20260709 13:03:05.995160  1495 catalog_manager.cc:5697] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 reported cstate change: term changed from 16 to 18. New cstate: current_term: 18 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:03:05.997169   431 tablet_replica.cc:333] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a: stopping tablet replica
I20260709 13:03:06.006656  1672 raft_consensus.cc:1240] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 27 FOLLOWER]: Rejecting Update request from peer af520b67945d443b901bbeca37cc5701 for earlier term 26. Current term is 27. Ops: []
I20260709 13:03:06.011144  1495 catalog_manager.cc:5697] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 reported cstate change: config changed from index 15 to 5, term changed from 25 to 22, leader changed from <none> to 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196), VOTER 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) added. New cstate: current_term: 22 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:03:06.014245  1495 catalog_manager.cc:5697] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 reported cstate change: term changed from 16 to 21. New cstate: current_term: 21 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 13:03:06.016010  1495 catalog_manager.cc:5697] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 reported cstate change: term changed from 8 to 19, leader changed from 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194) to 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196). New cstate: current_term: 19 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:03:06.025039 10414 raft_consensus.cc:493] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:03:06.014149  1717 consensus_peers.cc:597] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:03:06.025800 10414 raft_consensus.cc:515] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:03:06.032610  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 21 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:03:06.035177 10414 leader_election.cc:290] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:03:06.035846  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" candidate_term: 21 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:03:06.040086  1569 leader_election.cc:304] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83595ad0fd5940c0a2bfc7c61acd969a; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:03:06.050850 10414 raft_consensus.cc:2749] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
W20260709 13:03:06.053139  1717 consensus_peers.cc:597] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 13:03:06.058642   431 raft_consensus.cc:2243] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Raft consensus shutting down.
I20260709 13:03:06.059576   431 raft_consensus.cc:2272] T dd3f391a609743de886c525de483fc29 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Raft consensus is shut down!
I20260709 13:03:06.059463  1495 catalog_manager.cc:5697] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 reported cstate change: term changed from 14 to 21. New cstate: current_term: 21 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 13:03:06.066911   431 tablet_replica.cc:333] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a: stopping tablet replica
I20260709 13:03:06.075868   431 raft_consensus.cc:2243] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Raft consensus shutting down.
I20260709 13:03:06.076921   431 raft_consensus.cc:2272] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Raft consensus is shut down!
I20260709 13:03:06.080636   431 tablet_replica.cc:333] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a: stopping tablet replica
I20260709 13:03:06.081568   431 raft_consensus.cc:2243] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 28 LEADER]: Raft consensus shutting down.
I20260709 13:03:06.087783 10453 raft_consensus.cc:493] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:03:06.088639 10453 raft_consensus.cc:515] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:03:06.092849  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:03:06.093667  1751 raft_consensus.cc:2468] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 24.
I20260709 13:03:06.091322 10453 leader_election.cc:290] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:03:06.095072  1646 leader_election.cc:304] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:03:06.097016 10453 raft_consensus.cc:2804] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 13:03:06.097406 10453 raft_consensus.cc:493] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 24 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:03:06.097769 10453 raft_consensus.cc:3060] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 24 FOLLOWER]: Advancing to term 25
W20260709 13:03:06.099447  1571 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:06.104780   431 raft_consensus.cc:2272] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 83595ad0fd5940c0a2bfc7c61acd969a [term 28 FOLLOWER]: Raft consensus is shut down!
W20260709 13:03:06.108567  1503 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35458 (request call id 307) took 144909 ms (client timeout 15000 ms). Trace:
I20260709 13:03:06.112663   431 tablet_replica.cc:333] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a: stopping tablet replica
W20260709 13:03:06.109084  1503 rpcz_store.cc:269] 0709 13:00:41.199035 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:41.210699 (+ 11664us) service_pool.cc:224] Handling call
0709 13:03:06.108282 (+144897583us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":169999,"reported_tablets":47}
I20260709 13:03:06.120049 10453 raft_consensus.cc:515] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:03:06.121541   431 raft_consensus.cc:2243] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Raft consensus shutting down.
I20260709 13:03:06.123514  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9bb5648969b4299a776843d72043448" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:03:06.124105   431 pending_rounds.cc:70] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a: Trying to abort 1 pending ops.
I20260709 13:03:06.135398  1751 raft_consensus.cc:3060] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Advancing to term 25
I20260709 13:03:06.135815   431 pending_rounds.cc:77] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a: Aborting op as it isn't in flight: id { term: 21 index: 17 } timestamp: 7305634446695034880 op_type: NO_OP noop_request { }
I20260709 13:03:06.136620   431 raft_consensus.cc:2889] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260709 13:03:06.137068   431 raft_consensus.cc:2272] T 592fa647ae2545e09ad5ee25f8cb658e P 83595ad0fd5940c0a2bfc7c61acd969a [term 22 FOLLOWER]: Raft consensus is shut down!
W20260709 13:03:06.140611  1643 leader_election.cc:336] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:03:06.122399 10453 leader_election.cc:290] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 25 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:03:06.150315  1643 leader_election.cc:336] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 25 election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:03:06.151765  1751 raft_consensus.cc:2468] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 25.
I20260709 13:03:06.153537  1646 leader_election.cc:304] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:03:06.155378 10453 raft_consensus.cc:2804] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 25 FOLLOWER]: Leader election won for term 25
I20260709 13:03:06.156159 10453 raft_consensus.cc:697] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 25 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:03:06.157496 10453 consensus_queue.cc:237] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 24.16, Last appended by leader: 16, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:03:06.158721   431 tablet_replica.cc:333] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a: stopping tablet replica
I20260709 13:03:06.160089 10463 raft_consensus.cc:493] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:03:06.162258 10463 raft_consensus.cc:515] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:03:06.163429  1717 consensus_peers.cc:597] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260709 13:03:06.168143  2298 leader_election.cc:336] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:03:06.168780 10463 leader_election.cc:290] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:03:06.169920   431 raft_consensus.cc:2243] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Raft consensus shutting down.
I20260709 13:03:06.170866   431 raft_consensus.cc:2272] T 63cf5ab930f24f239fde96a128656333 P 83595ad0fd5940c0a2bfc7c61acd969a [term 20 FOLLOWER]: Raft consensus is shut down!
I20260709 13:03:06.178948  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:03:06.180838  2301 leader_election.cc:304] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:03:06.182601 10463 raft_consensus.cc:2749] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
W20260709 13:03:06.186494  2298 consensus_peers.cc:597] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
W20260709 13:03:06.198832  1717 consensus_peers.cc:597] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260709 13:03:06.255627  1717 consensus_peers.cc:597] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
W20260709 13:03:06.323302  2298 consensus_peers.cc:597] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
I20260709 13:03:06.348537 10463 raft_consensus.cc:493] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:03:06.349426 10463 raft_consensus.cc:515] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:03:06.356642  2298 leader_election.cc:336] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:03:06.360643  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 23 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:03:06.362934  2299 leader_election.cc:304] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:03:06.366214 10393 raft_consensus.cc:2749] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:03:06.368793 10463 leader_election.cc:290] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:03:06.396775  1494 catalog_manager.cc:5697] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 reported cstate change: term changed from 22 to 25, leader changed from 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) to 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194). New cstate: current_term: 25 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 13:03:06.401034  1494 catalog_manager.cc:5697] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 reported cstate change: term changed from 16 to 22. New cstate: current_term: 22 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } }
I20260709 13:03:06.402673  1494 catalog_manager.cc:5697] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 reported cstate change: term changed from 21 to 25, leader changed from 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194) to 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196). New cstate: current_term: 25 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 13:03:06.415407  1494 catalog_manager.cc:5697] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 reported cstate change: term changed from 19 to 23, leader changed from 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) to 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193). New cstate: current_term: 23 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 13:03:06.418248  1494 catalog_manager.cc:5697] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 reported cstate change: term changed from 22 to 25, leader changed from 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) to af520b67945d443b901bbeca37cc5701 (127.0.107.195). New cstate: current_term: 25 leader_uuid: "af520b67945d443b901bbeca37cc5701" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: HEALTHY } } }
I20260709 13:03:06.427474 10317 raft_consensus.cc:493] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:03:06.428133 10317 raft_consensus.cc:515] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:03:06.432932 10317 leader_election.cc:290] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:03:06.426584  1494 catalog_manager.cc:5697] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 reported cstate change: term changed from 19 to 23. New cstate: current_term: 23 leader_uuid: "af520b67945d443b901bbeca37cc5701" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:03:06.435745  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:03:06.436595  2333 raft_consensus.cc:2468] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 28.
I20260709 13:03:06.437830  1718 leader_election.cc:304] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:03:06.438796 10317 raft_consensus.cc:2804] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 13:03:06.439196 10317 raft_consensus.cc:493] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 28 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:03:06.439584 10317 raft_consensus.cc:3060] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 28 FOLLOWER]: Advancing to term 29
I20260709 13:03:06.446107 10393 raft_consensus.cc:493] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:03:06.446867 10393 raft_consensus.cc:515] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:03:06.437242  1494 catalog_manager.cc:5697] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 reported cstate change: term changed from 18 to 26, leader changed from 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194) to af520b67945d443b901bbeca37cc5701 (127.0.107.195). New cstate: current_term: 26 leader_uuid: "af520b67945d443b901bbeca37cc5701" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: HEALTHY } } }
I20260709 13:03:06.457857  1494 catalog_manager.cc:5697] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 reported cstate change: term changed from 17 to 26. New cstate: current_term: 26 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } }
I20260709 13:03:06.467358  1494 catalog_manager.cc:5697] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 reported cstate change: term changed from 10 to 18, leader changed from 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194) to af520b67945d443b901bbeca37cc5701 (127.0.107.195). New cstate: current_term: 18 leader_uuid: "af520b67945d443b901bbeca37cc5701" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: HEALTHY } } }
I20260709 13:03:06.459537  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 24 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:03:06.461786 10317 raft_consensus.cc:515] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 29 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:03:06.465277 10364 raft_consensus.cc:493] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:03:06.475983 10364 raft_consensus.cc:515] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:03:06.478864 10317 leader_election.cc:290] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 29 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:03:06.481505  1752 raft_consensus.cc:2468] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 23.
I20260709 13:03:06.484688  2301 leader_election.cc:304] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:03:06.487577 10458 raft_consensus.cc:2804] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 13:03:06.488045 10458 raft_consensus.cc:493] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:03:06.488479 10458 raft_consensus.cc:3060] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Advancing to term 24
W20260709 13:03:06.465991  1717 leader_election.cc:336] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:03:06.456403 10393 leader_election.cc:290] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:03:06.491191  1717 consensus_peers.cc:597] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 13:03:06.493376  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 24 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:03:06.493884 10364 leader_election.cc:290] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:03:06.479136  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dce8db7565da4e13a82ffbb9c39fde78" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:03:06.501245  2333 raft_consensus.cc:3060] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 28 FOLLOWER]: Advancing to term 29
W20260709 13:03:06.453480  2298 consensus_peers.cc:597] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
W20260709 13:03:06.502094  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 48693.6us
W20260709 13:03:06.504851  1717 consensus_peers.cc:597] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 13:03:06.507612  1494 catalog_manager.cc:5697] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 reported cstate change: term changed from 16 to 17, leader changed from 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) to 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194). New cstate: current_term: 17 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } }
I20260709 13:03:06.509933  1494 catalog_manager.cc:5697] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 reported cstate change: term changed from 15 to 22, leader changed from af520b67945d443b901bbeca37cc5701 (127.0.107.195) to 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194). New cstate: current_term: 22 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } }
W20260709 13:03:06.534164  1717 leader_election.cc:336] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 29 election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:03:06.511642  1494 catalog_manager.cc:5697] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 reported cstate change: term changed from 17 to 20, leader changed from af520b67945d443b901bbeca37cc5701 (127.0.107.195) to 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194). New cstate: current_term: 20 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } }
W20260709 13:03:06.543526  2298 leader_election.cc:336] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 13:03:06.539254  1717 leader_election.cc:336] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 13:03:06.552330  2298 consensus_peers.cc:597] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
I20260709 13:03:06.557057  1494 catalog_manager.cc:5697] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 reported cstate change: term changed from 10 to 18, leader changed from 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) to 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194). New cstate: current_term: 18 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } }
W20260709 13:03:06.564360  1495 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:43476 (request call id 198) took 27605 ms (client timeout 15000 ms). Trace:
I20260709 13:03:06.568964  2333 raft_consensus.cc:2468] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 29.
I20260709 13:03:06.567526  1494 catalog_manager.cc:5697] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 reported cstate change: term changed from 17 to 24. New cstate: current_term: 24 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } }
I20260709 13:03:06.575361  1494 catalog_manager.cc:5697] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 reported cstate change: term changed from 21 to 25. New cstate: current_term: 25 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 13:03:06.574816  1718 leader_election.cc:304] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:03:06.578825 10364 raft_consensus.cc:2804] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 29 FOLLOWER]: Leader election won for term 29
I20260709 13:03:06.579727 10364 raft_consensus.cc:697] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 29 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:03:06.580753 10364 consensus_queue.cc:237] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 28.17, Last appended by leader: 17, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:03:06.583765 10462 raft_consensus.cc:493] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:03:06.584408 10462 raft_consensus.cc:515] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:03:06.598536  1643 connection.cc:573] client connection to 127.0.107.193:46445 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260709 13:03:06.602591 10462 leader_election.cc:290] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:03:06.604142  2330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 23 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:03:06.604873  2330 raft_consensus.cc:2468] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 22.
I20260709 13:03:06.606556  1644 leader_election.cc:304] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
W20260709 13:03:06.564858  1495 rpcz_store.cc:269] 0709 13:02:39.134564 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:46.553231 (+7418667us) service_pool.cc:224] Handling call
0709 13:03:06.559536 (+20006305us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 13:03:06.159610 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 13:03:06.160441 (+   831us) write_op.cc:432] Acquiring schema lock in shared mode
0709 13:03:06.160503 (+    62us) write_op.cc:435] Acquired schema lock
0709 13:03:06.160523 (+    20us) tablet.cc:662] Decoding operations
0709 13:03:06.161676 (+  1153us) write_op.cc:620] Acquiring the partition lock for write op
0709 13:03:06.161779 (+   103us) write_op.cc:641] Partition lock acquired for write op
0709 13:03:06.161811 (+    32us) tablet.cc:685] Acquiring locks for 7 operations
0709 13:03:06.187401 (+ 25590us) tablet.cc:701] Row locks acquired
0709 13:03:06.187464 (+    63us) write_op.cc:260] PREPARE: finished
0709 13:03:06.187698 (+   234us) write_op.cc:270] Start()
0709 13:03:06.187867 (+   169us) write_op.cc:276] Timestamp: P: 1783602186187602 usec, L: 0
0709 13:03:06.187909 (+    42us) op_driver.cc:348] REPLICATION: starting
0709 13:03:06.188737 (+   828us) log.cc:844] Serialized 2898 byte log entry
0709 13:03:06.215928 (+ 27191us) op_driver.cc:464] REPLICATION: finished
0709 13:03:06.294983 (+ 79055us) write_op.cc:301] APPLY: starting
0709 13:03:06.295105 (+   122us) tablet.cc:1367] starting BulkCheckPresence
0709 13:03:06.303868 (+  8763us) tablet.cc:1370] finished BulkCheckPresence
0709 13:03:06.304014 (+   146us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 13:03:06.335373 (+ 31359us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 13:03:06.335762 (+   389us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=14,key_file_lookups=14,delta_file_lookups=0,mrs_lookups=0
0709 13:03:06.335823 (+    61us) write_op.cc:312] APPLY: finished
0709 13:03:06.336257 (+   434us) log.cc:844] Serialized 73 byte log entry
0709 13:03:06.336498 (+   241us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:03:06.336643 (+   145us) write_op.cc:454] Released schema lock
0709 13:03:06.336735 (+    92us) write_op.cc:341] FINISH: Updating metrics
0709 13:03:06.337352 (+   617us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:03:06.337383 (+    31us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":72977,"reported_tablets":36,"child_traces":[["op",{"apply.queue_time_us":78656,"apply.run_cpu_time_us":12579,"apply.run_wall_time_us":42324,"cfile_cache_hit":30,"cfile_cache_hit_bytes":33443,"num_ops":7,"prepare.queue_time_us":55782,"prepare.run_cpu_time_us":5235,"prepare.run_wall_time_us":55471,"replication_time_us":27746,"thread_start_us":96856,"threads_started":2}]]}
I20260709 13:03:06.597479  1494 catalog_manager.cc:5697] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 reported cstate change: term changed from 19 to 23, leader changed from 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) to af520b67945d443b901bbeca37cc5701 (127.0.107.195). New cstate: current_term: 23 leader_uuid: "af520b67945d443b901bbeca37cc5701" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:03:06.611963 10462 raft_consensus.cc:2804] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 13:03:06.612670  1494 catalog_manager.cc:5697] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 reported cstate change: term changed from 21 to 24, leader changed from 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194) to 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193). New cstate: current_term: 24 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } }
I20260709 13:03:06.621170  1494 catalog_manager.cc:5697] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 reported cstate change: term changed from 16 to 19, leader changed from 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) to af520b67945d443b901bbeca37cc5701 (127.0.107.195). New cstate: current_term: 19 leader_uuid: "af520b67945d443b901bbeca37cc5701" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: HEALTHY } } }
I20260709 13:03:06.614358 10462 raft_consensus.cc:493] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
W20260709 13:03:06.646375  1717 consensus_peers.cc:597] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 13:03:06.632339  1494 catalog_manager.cc:5697] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 reported cstate change: term changed from 17 to 24, leader changed from 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194) to af520b67945d443b901bbeca37cc5701 (127.0.107.195). New cstate: current_term: 24 leader_uuid: "af520b67945d443b901bbeca37cc5701" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: HEALTHY } } }
W20260709 13:03:06.646984  1717 consensus_peers.cc:597] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:03:06.647447  1717 consensus_peers.cc:597] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 13:03:06.647687  1494 catalog_manager.cc:5697] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 reported cstate change: term changed from 16 to 21, leader changed from 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) to af520b67945d443b901bbeca37cc5701 (127.0.107.195). New cstate: current_term: 21 leader_uuid: "af520b67945d443b901bbeca37cc5701" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } }
I20260709 13:03:06.655654  1752 raft_consensus.cc:1275] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 25 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 24 index: 16. Preceding OpId from leader: term: 25 index: 17. (index mismatch)
I20260709 13:03:06.632964 10462 raft_consensus.cc:3060] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:03:06.658968 10373 consensus_queue.cc:1048] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 13:03:06.655961  1494 catalog_manager.cc:5697] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 reported cstate change: term changed from 14 to 22, leader changed from 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194) to af520b67945d443b901bbeca37cc5701 (127.0.107.195). New cstate: current_term: 22 leader_uuid: "af520b67945d443b901bbeca37cc5701" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: HEALTHY } } }
I20260709 13:03:06.695287 10462 raft_consensus.cc:515] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:03:06.696084  2298 consensus_peers.cc:597] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:03:06.696969  2298 consensus_peers.cc:597] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 13:03:06.656592 10458 raft_consensus.cc:515] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:03:06.700402  2332 raft_consensus.cc:2393] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af520b67945d443b901bbeca37cc5701 in current term 24: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:03:06.703668  1718 leader_election.cc:304] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:03:06.699924  1494 catalog_manager.cc:5697] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 reported cstate change: term changed from 18 to 21, leader changed from af520b67945d443b901bbeca37cc5701 (127.0.107.195) to 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194). New cstate: current_term: 21 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } }
I20260709 13:03:06.704576 10364 raft_consensus.cc:3060] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Advancing to term 24
I20260709 13:03:06.709254 10462 leader_election.cc:290] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 23 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:03:06.706491  2298 leader_election.cc:336] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 24 election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111)
I20260709 13:03:06.708451  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83226095f79a46dab445849b1febd425" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 23 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:03:06.721347  2332 raft_consensus.cc:3060] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Advancing to term 23
W20260709 13:03:06.727217  1643 leader_election.cc:336] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111)
I20260709 13:03:06.723528  1494 catalog_manager.cc:5697] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 reported cstate change: config changed from index 5 to 15, term changed from 22 to 25, leader changed from 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) to af520b67945d443b901bbeca37cc5701 (127.0.107.195), VOTER 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) evicted. New cstate: current_term: 25 leader_uuid: "af520b67945d443b901bbeca37cc5701" committed_config { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:03:06.734020 10502 raft_consensus.cc:493] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:03:06.735917 10502 raft_consensus.cc:515] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:03:06.747452  1494 catalog_manager.cc:5697] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 reported cstate change: term changed from 14 to 26. New cstate: current_term: 26 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 13:03:06.749217  1494 catalog_manager.cc:5697] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 reported cstate change: term changed from 14 to 20. New cstate: current_term: 20 leader_uuid: "af520b67945d443b901bbeca37cc5701" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: HEALTHY } } }
I20260709 13:03:06.732487 10364 raft_consensus.cc:2749] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 13:03:06.770783 10502 leader_election.cc:290] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:03:06.773357  1751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:03:06.776500 10458 leader_election.cc:290] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 24 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:03:06.777817  1748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd44a2ebdc764e898f25ad5390f4bffd" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 24 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:03:06.786692  1494 catalog_manager.cc:5697] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 reported cstate change: term changed from 19 to 21. New cstate: current_term: 21 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } }
I20260709 13:03:06.789625  1494 catalog_manager.cc:5697] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 reported cstate change: term changed from 19 to 20, leader changed from 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194) to af520b67945d443b901bbeca37cc5701 (127.0.107.195). New cstate: current_term: 20 leader_uuid: "af520b67945d443b901bbeca37cc5701" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: HEALTHY } } }
I20260709 13:03:06.794495  1494 catalog_manager.cc:5697] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 reported cstate change: term changed from 16 to 28, leader changed from af520b67945d443b901bbeca37cc5701 (127.0.107.195) to 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193). New cstate: current_term: 28 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } }
I20260709 13:03:06.796130  1494 catalog_manager.cc:5697] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 reported cstate change: term changed from 18 to 23, leader changed from 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) to af520b67945d443b901bbeca37cc5701 (127.0.107.195). New cstate: current_term: 23 leader_uuid: "af520b67945d443b901bbeca37cc5701" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: HEALTHY } } }
I20260709 13:03:06.753881  2332 raft_consensus.cc:2468] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 23.
I20260709 13:03:06.797706  1494 catalog_manager.cc:5697] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 reported cstate change: term changed from 20 to 25, leader changed from 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) to 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193). New cstate: current_term: 25 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 13:03:06.798700  1644 leader_election.cc:304] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:03:06.799628 10502 raft_consensus.cc:2804] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Leader election won for term 23
W20260709 13:03:06.779042  1643 consensus_peers.cc:597] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 13:03:06.800135 10502 raft_consensus.cc:697] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:03:06.803865  1748 raft_consensus.cc:2468] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 24.
I20260709 13:03:06.804356 10502 consensus_queue.cc:237] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 22.18, Last appended by leader: 18, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:03:06.806420  2301 leader_election.cc:304] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:03:06.807721 10458 raft_consensus.cc:2804] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 13:03:06.808250 10458 raft_consensus.cc:697] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 24 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:03:06.810467 10458 consensus_queue.cc:237] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 23.18, Last appended by leader: 18, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:03:06.818882 10364 raft_consensus.cc:493] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:03:06.819730 10364 raft_consensus.cc:515] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:03:06.831197 10463 raft_consensus.cc:493] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:03:06.831837 10463 raft_consensus.cc:515] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:03:06.836820  1717 leader_election.cc:336] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111)
I20260709 13:03:06.836862 10364 leader_election.cc:290] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:03:06.841944 10463 leader_election.cc:290] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:03:06.878669 10364 raft_consensus.cc:493] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:03:06.879578 10364 raft_consensus.cc:515] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:03:06.886027  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:03:06.887655 10462 raft_consensus.cc:493] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:03:06.890939 10462 raft_consensus.cc:515] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:03:06.891758  1674 raft_consensus.cc:2468] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 20.
I20260709 13:03:06.893044 10462 leader_election.cc:290] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:03:06.893527  2299 leader_election.cc:304] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:03:06.894920 10393 raft_consensus.cc:2804] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:03:06.895303 10393 raft_consensus.cc:493] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:03:06.895669 10393 raft_consensus.cc:3060] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:03:06.898856  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:03:06.911393  2298 leader_election.cc:336] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111)
W20260709 13:03:06.915817  1717 leader_election.cc:336] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111)
I20260709 13:03:06.916157 10393 raft_consensus.cc:515] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:03:06.917513  2332 raft_consensus.cc:2393] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 816fadcea8fc4893882f5f9b9f22a03d in current term 21: Already voted for candidate 3b879799899a499c8d409ee9f1277cd1 in this term.
I20260709 13:03:06.917532 10364 leader_election.cc:290] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 13:03:06.928216  2298 consensus_peers.cc:597] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 13:03:06.932787  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a521fb6c3e7489d9694fd81cadeb33f" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:03:06.943538  1680 raft_consensus.cc:3060] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Advancing to term 21
W20260709 13:03:06.946015  2298 leader_election.cc:336] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 21 election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111)
I20260709 13:03:06.948314  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:03:06.950177  1672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:03:06.950843  1672 raft_consensus.cc:2468] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 24.
I20260709 13:03:06.924382 10393 leader_election.cc:290] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 21 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 13:03:06.963640  1643 consensus_peers.cc:597] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 13:03:06.964506  1643 consensus_peers.cc:597] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:03:06.963698  2298 consensus_peers.cc:597] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 13:03:06.965394  1680 raft_consensus.cc:2468] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 21.
I20260709 13:03:06.966892  2299 leader_election.cc:304] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:03:06.968261 10393 raft_consensus.cc:2804] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 13:03:06.969062 10393 raft_consensus.cc:697] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 21 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:03:06.970198 10393 consensus_queue.cc:237] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.15, Last appended by leader: 15, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:03:06.982723  1718 leader_election.cc:304] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:03:06.984525  1718 leader_election.cc:304] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:03:06.986079 10484 raft_consensus.cc:2749] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 13:03:06.985180  1674 consensus_queue.cc:237] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 26.16, Last appended by leader: 15, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } }
I20260709 13:03:06.987273 10484 raft_consensus.cc:2804] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 13:03:06.987653 10484 raft_consensus.cc:493] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:03:06.988058 10484 raft_consensus.cc:3060] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Advancing to term 25
W20260709 13:03:06.995718  1643 leader_election.cc:336] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111)
I20260709 13:03:06.996251  1643 leader_election.cc:304] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:03:06.999382 10508 raft_consensus.cc:2749] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
W20260709 13:03:07.000900  1643 leader_election.cc:336] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 23 election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111)
W20260709 13:03:07.007906  1643 leader_election.cc:336] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111)
I20260709 13:03:07.008360  1643 leader_election.cc:304] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:03:07.009131 10373 raft_consensus.cc:2749] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:03:07.022117 10373 raft_consensus.cc:493] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:03:07.022881 10373 raft_consensus.cc:515] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:03:07.028275  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 29 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:03:07.037968  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 53544.3us
I20260709 13:03:07.033439 10373 leader_election.cc:290] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:03:07.043721  1752 raft_consensus.cc:1275] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 26 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 26 index: 16. Preceding OpId from leader: term: 26 index: 17. (index mismatch)
W20260709 13:03:07.043931  1644 consensus_peers.cc:597] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d -> Peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625): Couldn't send request to peer 3b879799899a499c8d409ee9f1277cd1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7ebbb6d7c6c4478e95972de2f78d3f80. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:03:07.060474  2333 raft_consensus.cc:1275] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 29 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 28 index: 17. Preceding OpId from leader: term: 29 index: 18. (index mismatch)
W20260709 13:03:07.057931  1643 consensus_peers.cc:597] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 13:03:07.065044 10317 consensus_queue.cc:1048] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20260709 13:03:07.053570 10373 consensus_queue.cc:1048] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.021s
W20260709 13:03:07.080413  1643 consensus_peers.cc:597] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 13:03:07.097684  1643 leader_election.cc:336] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111)
I20260709 13:03:07.098284  1643 leader_election.cc:304] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:03:07.106540 10510 raft_consensus.cc:493] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:03:07.107656 10510 raft_consensus.cc:515] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:03:07.102567 10373 raft_consensus.cc:2749] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 13:03:07.115700 10510 leader_election.cc:290] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:03:07.117559 10508 raft_consensus.cc:2955] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 26 LEADER]: Committing config change with OpId 26.17: config changed from index -1 to 17, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } } }
I20260709 13:03:07.111068 10484 raft_consensus.cc:515] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:03:07.126658  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:03:07.130180  1643 leader_election.cc:336] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111)
I20260709 13:03:07.131209  1750 raft_consensus.cc:2468] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 24.
I20260709 13:03:07.131274  1752 raft_consensus.cc:2955] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 26 FOLLOWER]: Committing config change with OpId 26.17: config changed from index -1 to 17, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } } }
I20260709 13:03:07.133566  1646 leader_election.cc:304] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:03:07.134577 10373 raft_consensus.cc:2804] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 13:03:07.134946 10373 raft_consensus.cc:493] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 24 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:03:07.135300 10373 raft_consensus.cc:3060] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 24 FOLLOWER]: Advancing to term 25
I20260709 13:03:07.146472 10373 raft_consensus.cc:515] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:03:07.149919  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65285571995c4ed5b593041598056a22" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:03:07.150097 10373 leader_election.cc:290] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 25 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:03:07.150655  1750 raft_consensus.cc:3060] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Advancing to term 25
W20260709 13:03:07.167128  1643 leader_election.cc:336] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 25 election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111)
W20260709 13:03:07.169560  1717 consensus_peers.cc:597] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 13:03:07.169775  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73dc0a9696be41f78e22f1f77a6eee72" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:03:07.170526  1674 raft_consensus.cc:3060] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 24 FOLLOWER]: Advancing to term 25
W20260709 13:03:07.173897  1717 leader_election.cc:336] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 25 election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111)
I20260709 13:03:07.177201  1750 raft_consensus.cc:2468] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 25.
I20260709 13:03:07.149871 10484 leader_election.cc:290] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 25 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:03:07.178818  1646 leader_election.cc:304] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:03:07.179708 10373 raft_consensus.cc:2804] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 25 FOLLOWER]: Leader election won for term 25
I20260709 13:03:07.180519 10373 raft_consensus.cc:697] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 25 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:03:07.181607 10373 consensus_queue.cc:237] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 24.15, Last appended by leader: 15, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:03:07.193612  1680 consensus_queue.cc:237] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 22.11, Last appended by leader: 10, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } }
W20260709 13:03:07.211047  1717 consensus_peers.cc:597] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:03:07.216588  1643 consensus_peers.cc:597] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:03:07.243492  1487 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 91268.1us
I20260709 13:03:07.255112  1674 raft_consensus.cc:2468] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 25.
I20260709 13:03:07.256754  1718 leader_election.cc:304] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:03:07.258167 10520 raft_consensus.cc:2804] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 25 FOLLOWER]: Leader election won for term 25
I20260709 13:03:07.258773 10520 raft_consensus.cc:697] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 25 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:03:07.259815 10520 consensus_queue.cc:237] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 24.13, Last appended by leader: 13, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:03:07.259955  1487 catalog_manager.cc:4755] TS 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 983621a6fc5b4c9ea0159a648dfe7a36: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111)
W20260709 13:03:07.283783  1644 consensus_peers.cc:597] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d -> Peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625): Couldn't send request to peer 3b879799899a499c8d409ee9f1277cd1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 32dc176efd53437ab7735ea8c6ac4a0d. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:03:07.296872  1752 raft_consensus.cc:1275] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 22 index: 11. Preceding OpId from leader: term: 22 index: 12. (index mismatch)
I20260709 13:03:07.299646  2330 raft_consensus.cc:1275] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 22 index: 18. Preceding OpId from leader: term: 23 index: 19. (index mismatch)
I20260709 13:03:07.304474 10510 consensus_queue.cc:1048] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20260709 13:03:07.311839 10510 consensus_queue.cc:1048] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
W20260709 13:03:07.315260  1643 consensus_peers.cc:597] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 13:03:07.327953 10063 raft_consensus.cc:493] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:07.328696 10063 raft_consensus.cc:515] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:03:07.334671 10063 leader_election.cc:290] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:03:07.336514  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 25 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:03:07.337320  1750 raft_consensus.cc:2468] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 24.
W20260709 13:03:07.340874  2298 leader_election.cc:336] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111)
W20260709 13:03:07.349231  1643 consensus_peers.cc:597] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 13:03:07.351306  2301 leader_election.cc:304] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:03:07.352278 10463 raft_consensus.cc:2804] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 13:03:07.352722 10463 raft_consensus.cc:493] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:07.353084 10463 raft_consensus.cc:3060] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 24 FOLLOWER]: Advancing to term 25
I20260709 13:03:07.372350 10463 raft_consensus.cc:515] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:03:07.382659  2298 leader_election.cc:336] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 25 election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111)
W20260709 13:03:07.396811  2298 consensus_peers.cc:597] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 13:03:07.397773 10463 leader_election.cc:290] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 25 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:03:07.407986  1752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d4ba8d6f4f947fabf8a2609af29c3d3" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 25 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:03:07.408695  1752 raft_consensus.cc:3060] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Advancing to term 25
I20260709 13:03:07.411458  1499 catalog_manager.cc:5697] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 reported cstate change: term changed from 15 to 20, leader changed from 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) to af520b67945d443b901bbeca37cc5701 (127.0.107.195). New cstate: current_term: 20 leader_uuid: "af520b67945d443b901bbeca37cc5701" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: HEALTHY } } }
I20260709 13:03:07.420048 10462 raft_consensus.cc:2955] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 22 LEADER]: Committing config change with OpId 22.12: config changed from index -1 to 12, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } } }
I20260709 13:03:07.422896  1494 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35464 (request call id 292) took 2495 ms. Trace:
I20260709 13:03:07.425557  1753 raft_consensus.cc:1275] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 23 index: 18. Preceding OpId from leader: term: 24 index: 19. (index mismatch)
I20260709 13:03:07.426556  1752 raft_consensus.cc:2468] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 25.
I20260709 13:03:07.427531 10463 consensus_queue.cc:1048] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 13:03:07.414150  1499 catalog_manager.cc:5697] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 reported cstate change: term changed from 20 to 22. New cstate: current_term: 22 leader_uuid: "af520b67945d443b901bbeca37cc5701" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: HEALTHY } } }
I20260709 13:03:07.425942  1494 rpcz_store.cc:276] 0709 13:03:04.927751 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:05.772342 (+844591us) service_pool.cc:224] Handling call
0709 13:03:07.422793 (+1650451us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 13:03:06.966544 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 13:03:06.967244 (+   700us) write_op.cc:432] Acquiring schema lock in shared mode
0709 13:03:06.967282 (+    38us) write_op.cc:435] Acquired schema lock
0709 13:03:06.967302 (+    20us) tablet.cc:662] Decoding operations
0709 13:03:06.969871 (+  2569us) write_op.cc:620] Acquiring the partition lock for write op
0709 13:03:06.969964 (+    93us) write_op.cc:641] Partition lock acquired for write op
0709 13:03:06.970002 (+    38us) tablet.cc:685] Acquiring locks for 30 operations
0709 13:03:07.004053 (+ 34051us) tablet.cc:701] Row locks acquired
0709 13:03:07.004075 (+    22us) write_op.cc:260] PREPARE: finished
0709 13:03:07.004251 (+   176us) write_op.cc:270] Start()
0709 13:03:07.004379 (+   128us) write_op.cc:276] Timestamp: P: 1783602187004190 usec, L: 0
0709 13:03:07.004402 (+    23us) op_driver.cc:348] REPLICATION: starting
0709 13:03:07.005295 (+   893us) log.cc:844] Serialized 11741 byte log entry
0709 13:03:07.071725 (+ 66430us) op_driver.cc:464] REPLICATION: finished
0709 13:03:07.157329 (+ 85604us) write_op.cc:301] APPLY: starting
0709 13:03:07.157429 (+   100us) tablet.cc:1367] starting BulkCheckPresence
0709 13:03:07.280511 (+123082us) tablet.cc:1370] finished BulkCheckPresence
0709 13:03:07.280679 (+   168us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 13:03:07.319553 (+ 38874us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 13:03:07.319966 (+   413us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=60,key_file_lookups=60,delta_file_lookups=0,mrs_lookups=0
0709 13:03:07.321047 (+  1081us) write_op.cc:312] APPLY: finished
0709 13:03:07.322243 (+  1196us) log.cc:844] Serialized 260 byte log entry
0709 13:03:07.322478 (+   235us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:03:07.322771 (+   293us) write_op.cc:454] Released schema lock
0709 13:03:07.322857 (+    86us) write_op.cc:341] FINISH: Updating metrics
0709 13:03:07.323343 (+   486us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:03:07.323372 (+    29us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":63350,"reported_tablets":47,"child_traces":[["op",{"apply.queue_time_us":85304,"apply.run_cpu_time_us":42708,"apply.run_wall_time_us":165988,"cfile_cache_hit":122,"cfile_cache_hit_bytes":129767,"num_ops":30,"prepare.queue_time_us":102796,"prepare.run_cpu_time_us":7996,"prepare.run_wall_time_us":55314,"replication_time_us":67127,"thread_start_us":163346,"threads_started":2}]]}
I20260709 13:03:07.436254  1753 raft_consensus.cc:2955] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Committing config change with OpId 22.12: config changed from index -1 to 12, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } } }
I20260709 13:03:07.457300  2301 leader_election.cc:304] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:03:07.463865 10063 raft_consensus.cc:2804] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 25 FOLLOWER]: Leader election won for term 25
I20260709 13:03:07.464447 10063 raft_consensus.cc:697] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 25 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:03:07.465444 10063 consensus_queue.cc:237] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 22.16, Last appended by leader: 16, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:03:07.505910 10520 raft_consensus.cc:493] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
I20260709 13:03:07.506567 10520 raft_consensus.cc:515] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:03:07.508884 10520 leader_election.cc:290] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:03:07.523512  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 29 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d" is_pre_election: true
I20260709 13:03:07.524343  1674 raft_consensus.cc:2468] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 28.
W20260709 13:03:07.528506  1499 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35464 (request call id 291) took 18791 ms (client timeout 15000 ms). Trace:
I20260709 13:03:07.531622  1718 leader_election.cc:304] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:03:07.536391 10364 raft_consensus.cc:2804] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 13:03:07.536845 10364 raft_consensus.cc:493] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 28 FOLLOWER]: Starting leader election (detected failure of leader 83595ad0fd5940c0a2bfc7c61acd969a)
W20260709 13:03:07.537051  1717 leader_election.cc:336] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111)
I20260709 13:03:07.537253 10364 raft_consensus.cc:3060] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 28 FOLLOWER]: Advancing to term 29
W20260709 13:03:07.538877  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 44738.2us
I20260709 13:03:07.543807  1680 consensus_queue.cc:237] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 22.11, Last appended by leader: 10, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } }
I20260709 13:03:07.553637 10453 raft_consensus.cc:493] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:03:07.533406  1499 rpcz_store.cc:269] 0709 13:02:48.737090 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:59.581177 (+10844087us) service_pool.cc:224] Handling call
0709 13:03:07.528371 (+7947194us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 13:03:07.451122 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 13:03:07.451764 (+   642us) write_op.cc:432] Acquiring schema lock in shared mode
0709 13:03:07.451808 (+    44us) write_op.cc:435] Acquired schema lock
0709 13:03:07.451823 (+    15us) tablet.cc:662] Decoding operations
0709 13:03:07.452279 (+   456us) write_op.cc:620] Acquiring the partition lock for write op
0709 13:03:07.452360 (+    81us) write_op.cc:641] Partition lock acquired for write op
0709 13:03:07.452389 (+    29us) tablet.cc:685] Acquiring locks for 2 operations
0709 13:03:07.452782 (+   393us) tablet.cc:701] Row locks acquired
0709 13:03:07.452801 (+    19us) write_op.cc:260] PREPARE: finished
0709 13:03:07.452968 (+   167us) write_op.cc:270] Start()
0709 13:03:07.453085 (+   117us) write_op.cc:276] Timestamp: P: 1783602187452916 usec, L: 0
0709 13:03:07.453102 (+    17us) op_driver.cc:348] REPLICATION: starting
0709 13:03:07.453820 (+   718us) log.cc:844] Serialized 929 byte log entry
0709 13:03:07.457613 (+  3793us) op_driver.cc:464] REPLICATION: finished
0709 13:03:07.458018 (+   405us) write_op.cc:301] APPLY: starting
0709 13:03:07.458127 (+   109us) tablet.cc:1367] starting BulkCheckPresence
0709 13:03:07.459573 (+  1446us) tablet.cc:1370] finished BulkCheckPresence
0709 13:03:07.459661 (+    88us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 13:03:07.460958 (+  1297us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 13:03:07.461140 (+   182us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=4,key_file_lookups=4,delta_file_lookups=0,mrs_lookups=0
0709 13:03:07.461186 (+    46us) write_op.cc:312] APPLY: finished
0709 13:03:07.461376 (+   190us) log.cc:844] Serialized 33 byte log entry
0709 13:03:07.461529 (+   153us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:03:07.461619 (+    90us) write_op.cc:454] Released schema lock
0709 13:03:07.461660 (+    41us) write_op.cc:341] FINISH: Updating metrics
0709 13:03:07.462091 (+   431us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:03:07.462122 (+    31us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":5288,"reported_tablets":47,"spinlock_wait_cycles":17024,"child_traces":[["op",{"apply.queue_time_us":128,"apply.run_cpu_time_us":4023,"apply.run_wall_time_us":4022,"cfile_cache_hit":8,"cfile_cache_hit_bytes":8376,"num_ops":2,"prepare.queue_time_us":143,"prepare.run_cpu_time_us":3057,"prepare.run_wall_time_us":6224,"replication_time_us":4368}]]}
I20260709 13:03:07.554270 10453 raft_consensus.cc:515] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:03:07.560880  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 28 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:03:07.561627  2333 raft_consensus.cc:2468] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 27.
W20260709 13:03:07.567220  2298 consensus_peers.cc:597] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 13:03:07.556748 10453 leader_election.cc:290] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:03:07.574856  1644 leader_election.cc:304] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 
I20260709 13:03:07.581715 10511 raft_consensus.cc:2804] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 13:03:07.582186 10511 raft_consensus.cc:493] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:07.582551 10511 raft_consensus.cc:3060] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 27 FOLLOWER]: Advancing to term 28
I20260709 13:03:07.596719  1674 raft_consensus.cc:1275] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 20 index: 15. Preceding OpId from leader: term: 21 index: 16. (index mismatch)
I20260709 13:03:07.599243 10458 consensus_queue.cc:1048] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 13:03:07.620007  1752 raft_consensus.cc:1275] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 25 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 24 index: 15. Preceding OpId from leader: term: 25 index: 16. (index mismatch)
W20260709 13:03:07.625541   431 thread.cc:527] Waited for 1000ms trying to join with diag-logger (tid 1626)
I20260709 13:03:07.645478 10539 ts_tablet_manager.cc:933] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 3b879799899a499c8d409ee9f1277cd1: Initiating tablet copy from peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 13:03:07.648204  1644 consensus_peers.cc:597] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d -> Peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625): Couldn't send request to peer 3b879799899a499c8d409ee9f1277cd1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 127630d60f944b899508c92ee18fdddf. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:03:07.633958 10462 consensus_queue.cc:1048] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.010s
I20260709 13:03:07.650609  1752 raft_consensus.cc:1275] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 22 index: 11. Preceding OpId from leader: term: 22 index: 12. (index mismatch)
I20260709 13:03:07.652613 10506 consensus_queue.cc:1048] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
W20260709 13:03:07.658187  1643 consensus_peers.cc:597] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:03:07.658759  1643 leader_election.cc:336] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111)
I20260709 13:03:07.670058 10511 raft_consensus.cc:515] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:03:07.643363 10537 log.cc:927] Time spent T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d: Append to log took a long time: real 0.062s	user 0.001s	sys 0.000s
W20260709 13:03:07.685781  1643 consensus_peers.cc:597] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 13:03:07.686311 10511 leader_election.cc:290] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 28 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:03:07.690662  2332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8ab62ffb73246dc9bcf39f7bee51147" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 28 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:03:07.691303 10539 tablet_copy_client.cc:323] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Beginning tablet copy session from remote peer at address 127.0.107.194:46261
I20260709 13:03:07.691447  2332 raft_consensus.cc:3060] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 27 FOLLOWER]: Advancing to term 28
I20260709 13:03:07.696447  1690 tablet_copy_service.cc:140] P 816fadcea8fc4893882f5f9b9f22a03d: Received BeginTabletCopySession request for tablet 7ebbb6d7c6c4478e95972de2f78d3f80 from peer 3b879799899a499c8d409ee9f1277cd1 ({username='slave'} at 127.0.0.1:57092)
I20260709 13:03:07.697149  1690 tablet_copy_service.cc:161] P 816fadcea8fc4893882f5f9b9f22a03d: Beginning new tablet copy session on tablet 7ebbb6d7c6c4478e95972de2f78d3f80 from peer 3b879799899a499c8d409ee9f1277cd1 at {username='slave'} at 127.0.0.1:57092: session id = 3b879799899a499c8d409ee9f1277cd1-7ebbb6d7c6c4478e95972de2f78d3f80
I20260709 13:03:07.692133 10364 raft_consensus.cc:515] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:03:07.713680 10364 leader_election.cc:290] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 29 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 13:03:07.722740  1717 leader_election.cc:336] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 29 election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111)
I20260709 13:03:07.724817  2332 raft_consensus.cc:2468] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816fadcea8fc4893882f5f9b9f22a03d in term 28.
W20260709 13:03:07.727092  1643 leader_election.cc:336] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 28 election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111)
I20260709 13:03:07.728593  1644 leader_election.cc:304] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:03:07.729737 10508 raft_consensus.cc:2804] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 28 FOLLOWER]: Leader election won for term 28
I20260709 13:03:07.730376 10508 raft_consensus.cc:697] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 28 LEADER]: Becoming Leader. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:03:07.732182  1674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19ce8a0cd7c54eafbb0e61fc6e305b62" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 29 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:03:07.734349  1674 raft_consensus.cc:3060] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 28 FOLLOWER]: Advancing to term 29
I20260709 13:03:07.745482  1690 tablet_copy_source_session.cc:215] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 13:03:07.751601 10539 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ebbb6d7c6c4478e95972de2f78d3f80. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:07.733546 10508 consensus_queue.cc:237] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 25.18, Last appended by leader: 18, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:03:07.805361  1674 raft_consensus.cc:2468] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 29.
I20260709 13:03:07.826777 10544 ts_tablet_manager.cc:933] T 32dc176efd53437ab7735ea8c6ac4a0d P 3b879799899a499c8d409ee9f1277cd1: Initiating tablet copy from peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 13:03:07.828725  1643 consensus_peers.cc:597] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 13:03:07.831228  1718 leader_election.cc:304] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:03:07.832661 10364 raft_consensus.cc:2804] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 29 FOLLOWER]: Leader election won for term 29
I20260709 13:03:07.833268 10364 raft_consensus.cc:697] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 29 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:03:07.834271 10364 consensus_queue.cc:237] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 28.14, Last appended by leader: 14, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:03:07.837909  1496 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35464 (request call id 289) took 64342 ms (client timeout 15000 ms). Trace:
W20260709 13:03:07.838410  1496 rpcz_store.cc:269] 0709 13:02:03.505625 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:03.523362 (+ 17737us) service_pool.cc:224] Handling call
0709 13:03:07.837660 (+64314298us) inbound_call.cc:177] Queueing success response
Metrics: {"reported_tablets":47}
I20260709 13:03:07.851523 10536 raft_consensus.cc:2955] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 22 LEADER]: Committing config change with OpId 22.12: config changed from index -1 to 12, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } } }
I20260709 13:03:07.875198  1751 raft_consensus.cc:2955] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Committing config change with OpId 22.12: config changed from index -1 to 12, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } } }
I20260709 13:03:07.877300  1499 catalog_manager.cc:5697] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 reported cstate change: term changed from 28 to 29, leader changed from 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) to af520b67945d443b901bbeca37cc5701 (127.0.107.195). New cstate: current_term: 29 leader_uuid: "af520b67945d443b901bbeca37cc5701" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:03:07.881753  1674 raft_consensus.cc:1275] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 25 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 24 index: 13. Preceding OpId from leader: term: 25 index: 14. (index mismatch)
W20260709 13:03:07.882820  1717 consensus_peers.cc:597] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 13:03:07.890378 10544 tablet_copy_client.cc:323] T 32dc176efd53437ab7735ea8c6ac4a0d P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Beginning tablet copy session from remote peer at address 127.0.107.194:46261
I20260709 13:03:07.892611  1690 tablet_copy_service.cc:140] P 816fadcea8fc4893882f5f9b9f22a03d: Received BeginTabletCopySession request for tablet 32dc176efd53437ab7735ea8c6ac4a0d from peer 3b879799899a499c8d409ee9f1277cd1 ({username='slave'} at 127.0.0.1:57092)
I20260709 13:03:07.893249  1690 tablet_copy_service.cc:161] P 816fadcea8fc4893882f5f9b9f22a03d: Beginning new tablet copy session on tablet 32dc176efd53437ab7735ea8c6ac4a0d from peer 3b879799899a499c8d409ee9f1277cd1 at {username='slave'} at 127.0.0.1:57092: session id = 3b879799899a499c8d409ee9f1277cd1-32dc176efd53437ab7735ea8c6ac4a0d
I20260709 13:03:07.896365 10353 consensus_queue.cc:1048] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 13:03:07.902915 10539 tablet_copy_client.cc:806] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Starting download of 0 data blocks...
I20260709 13:03:07.906466  1499 catalog_manager.cc:5697] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 reported cstate change: term changed from 25 to 27, leader changed from 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194) to 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196). New cstate: current_term: 27 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
W20260709 13:03:07.908156  1717 consensus_peers.cc:597] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 13:03:07.915156 10539 tablet_copy_client.cc:670] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Starting download of 1 WAL segments...
I20260709 13:03:07.922034 10539 tablet_copy_client.cc:538] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 13:03:07.924988  1499 catalog_manager.cc:5697] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 reported cstate change: term changed from 22 to 23, leader changed from 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) to 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196). New cstate: current_term: 23 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 13:03:07.926789  1499 catalog_manager.cc:5697] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 reported cstate change: term changed from 25 to 26, leader changed from 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) to af520b67945d443b901bbeca37cc5701 (127.0.107.195). New cstate: current_term: 26 leader_uuid: "af520b67945d443b901bbeca37cc5701" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:03:07.928654  1499 catalog_manager.cc:5697] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 reported cstate change: term changed from 24 to 25, leader changed from 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) to af520b67945d443b901bbeca37cc5701 (127.0.107.195). New cstate: current_term: 25 leader_uuid: "af520b67945d443b901bbeca37cc5701" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: HEALTHY } } }
I20260709 13:03:07.936920  1690 tablet_copy_source_session.cc:215] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 13:03:07.961968 10544 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32dc176efd53437ab7735ea8c6ac4a0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:07.957150  1499 catalog_manager.cc:5697] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 reported cstate change: term changed from 23 to 24, leader changed from 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) to 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196). New cstate: current_term: 24 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
W20260709 13:03:07.977469  2298 consensus_peers.cc:597] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 13:03:07.990345 10539 tablet_bootstrap.cc:492] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 3b879799899a499c8d409ee9f1277cd1: Bootstrap starting.
I20260709 13:03:08.011263  1752 raft_consensus.cc:1275] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 25 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 22 index: 16. Preceding OpId from leader: term: 25 index: 17. (index mismatch)
W20260709 13:03:08.026275  2298 consensus_peers.cc:597] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:03:08.027019  2298 consensus_peers.cc:597] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:03:08.038640  1717 consensus_peers.cc:597] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 13:03:08.043411 10547 consensus_queue.cc:1048] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 13:03:08.048605  1499 catalog_manager.cc:5697] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 reported cstate change: term changed from 19 to 20, leader changed from 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) to af520b67945d443b901bbeca37cc5701 (127.0.107.195). New cstate: current_term: 20 leader_uuid: "af520b67945d443b901bbeca37cc5701" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:03:08.050662  1499 catalog_manager.cc:5697] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 reported cstate change: config changed from index -1 to 17, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New cstate: current_term: 26 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } } }
I20260709 13:03:08.052405  1499 catalog_manager.cc:5697] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 reported cstate change: term changed from 22 to 25. New cstate: current_term: 25 leader_uuid: "af520b67945d443b901bbeca37cc5701" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: HEALTHY } } }
W20260709 13:03:08.088328  1643 consensus_peers.cc:597] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 13:03:08.094722  1751 consensus_queue.cc:237] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.14, Last appended by leader: 13, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } }
I20260709 13:03:08.054042  1499 catalog_manager.cc:5697] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 reported cstate change: term changed from 24 to 25, leader changed from 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) to 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194). New cstate: current_term: 25 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } }
I20260709 13:03:08.103922 10544 tablet_copy_client.cc:806] T 32dc176efd53437ab7735ea8c6ac4a0d P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Starting download of 0 data blocks...
I20260709 13:03:08.134482 10544 tablet_copy_client.cc:670] T 32dc176efd53437ab7735ea8c6ac4a0d P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Starting download of 1 WAL segments...
I20260709 13:03:08.139953 10544 tablet_copy_client.cc:538] T 32dc176efd53437ab7735ea8c6ac4a0d P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20260709 13:03:08.147009  1498 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35458 (request call id 312) took 20795 ms (client timeout 15000 ms). Trace:
W20260709 13:03:08.154311  2298 consensus_peers.cc:597] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:03:08.155448  1717 consensus_peers.cc:597] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:03:08.149848  1498 rpcz_store.cc:269] 0709 13:02:47.369174 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:56.507831 (+9138657us) service_pool.cc:224] Handling call
0709 13:02:57.123693 (+615862us) spinlock_profiling.cc:243] Waited 7.74 ms on lock 0x7b5400002490. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f426303db71 00007f426303e0b2 00007f426bfb7a89 00007f426bf4fda6 00007f426bf21088 00007f426c01f37e 00007f426a4e02a4 00007f426a4e0232 00007f426a4e0161 00007f426a4e00dc
0709 13:02:57.254176 (+130483us) spinlock_profiling.cc:243] Waited 29.4 ms on lock 0x7b540041cb54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cc090 00007f426616af3a 00007f426bf211cc 00007f426c01f37e 00007f426a4e02a4 00007f426a4e0232 00007f426a4e0161 00007f426a4e00dc 00007f426a4df392 00007f426449d48c 00007f426449c8c6
0709 13:03:08.146751 (+10892575us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1208464,"reported_tablets":47,"spinlock_wait_cycles":85423232}
W20260709 13:03:08.155107  2298 consensus_peers.cc:597] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 13:03:08.165673  2332 raft_consensus.cc:1275] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 28 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 25 index: 18. Preceding OpId from leader: term: 28 index: 19. (index mismatch)
I20260709 13:03:08.167850 10533 consensus_queue.cc:1048] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
W20260709 13:03:08.213390  1718 consensus_peers.cc:597] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 -> Peer 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625): Couldn't send request to peer 3b879799899a499c8d409ee9f1277cd1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a92588e9316e4b4aa1691e8669c839d3. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:03:08.242846 10539 log.cc:927] Time spent T 7ebbb6d7c6c4478e95972de2f78d3f80 P 3b879799899a499c8d409ee9f1277cd1: Append to log took a long time: real 0.052s	user 0.000s	sys 0.004s
I20260709 13:03:08.281260  1680 raft_consensus.cc:1275] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 20 index: 14. Preceding OpId from leader: term: 20 index: 15. (index mismatch)
W20260709 13:03:08.305289  1717 consensus_peers.cc:597] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 13:03:08.311594 10544 tablet_bootstrap.cc:492] T 32dc176efd53437ab7735ea8c6ac4a0d P 3b879799899a499c8d409ee9f1277cd1: Bootstrap starting.
I20260709 13:03:08.316929 10557 ts_tablet_manager.cc:933] T 127630d60f944b899508c92ee18fdddf P 3b879799899a499c8d409ee9f1277cd1: Initiating tablet copy from peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 13:03:08.327700  1643 consensus_peers.cc:597] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 13:03:08.328104  1717 consensus_peers.cc:597] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 13:03:08.328905  1717 consensus_peers.cc:597] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 13:03:08.334194 10520 consensus_queue.cc:1048] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.015s
W20260709 13:03:08.360357  1643 consensus_peers.cc:597] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 13:03:08.394753 10557 tablet_copy_client.cc:323] T 127630d60f944b899508c92ee18fdddf P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Beginning tablet copy session from remote peer at address 127.0.107.194:46261
I20260709 13:03:08.396524  1690 tablet_copy_service.cc:140] P 816fadcea8fc4893882f5f9b9f22a03d: Received BeginTabletCopySession request for tablet 127630d60f944b899508c92ee18fdddf from peer 3b879799899a499c8d409ee9f1277cd1 ({username='slave'} at 127.0.0.1:57092)
I20260709 13:03:08.397127  1690 tablet_copy_service.cc:161] P 816fadcea8fc4893882f5f9b9f22a03d: Beginning new tablet copy session on tablet 127630d60f944b899508c92ee18fdddf from peer 3b879799899a499c8d409ee9f1277cd1 at {username='slave'} at 127.0.0.1:57092: session id = 3b879799899a499c8d409ee9f1277cd1-127630d60f944b899508c92ee18fdddf
W20260709 13:03:08.405722  1487 catalog_manager.cc:4755] TS 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 983621a6fc5b4c9ea0159a648dfe7a36: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111)
I20260709 13:03:08.419391  1690 tablet_copy_source_session.cc:215] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 13:03:08.425872 10520 raft_consensus.cc:2955] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 20 LEADER]: Committing config change with OpId 20.15: config changed from index -1 to 15, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } } }
I20260709 13:03:08.426774  1680 raft_consensus.cc:2955] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Committing config change with OpId 20.15: config changed from index -1 to 15, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } } }
I20260709 13:03:08.420858 10539 tablet_bootstrap.cc:492] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 3b879799899a499c8d409ee9f1277cd1: Bootstrap replayed 1/1 log segments. Stats: ops{read=17 overwritten=0 applied=17 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:03:08.443409 10557 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 127630d60f944b899508c92ee18fdddf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:08.455161  1676 raft_consensus.cc:1275] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 29 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 28 index: 14. Preceding OpId from leader: term: 29 index: 15. (index mismatch)
I20260709 13:03:08.456681 10539 tablet_bootstrap.cc:492] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 3b879799899a499c8d409ee9f1277cd1: Bootstrap complete.
I20260709 13:03:08.457603 10484 consensus_queue.cc:1048] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 13:03:08.467794 10539 ts_tablet_manager.cc:1403] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 3b879799899a499c8d409ee9f1277cd1: Time spent bootstrapping tablet: real 0.478s	user 0.055s	sys 0.018s
W20260709 13:03:08.536768  1717 consensus_peers.cc:597] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 13:03:08.552769 10557 tablet_copy_client.cc:806] T 127630d60f944b899508c92ee18fdddf P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Starting download of 0 data blocks...
I20260709 13:03:08.553638 10557 tablet_copy_client.cc:670] T 127630d60f944b899508c92ee18fdddf P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Starting download of 1 WAL segments...
I20260709 13:03:08.554502 10539 raft_consensus.cc:359] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 3b879799899a499c8d409ee9f1277cd1 [term 26 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } }
I20260709 13:03:08.555563 10539 raft_consensus.cc:740] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 3b879799899a499c8d409ee9f1277cd1 [term 26 LEARNER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Initialized, Role: LEARNER
I20260709 13:03:08.558544 10539 consensus_queue.cc:260] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 17, Last appended: 26.17, Last appended by leader: 17, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } }
I20260709 13:03:08.569598 10557 tablet_copy_client.cc:538] T 127630d60f944b899508c92ee18fdddf P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 13:03:08.585327 10539 ts_tablet_manager.cc:1434] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 3b879799899a499c8d409ee9f1277cd1: Time spent starting tablet: real 0.106s	user 0.010s	sys 0.000s
I20260709 13:03:08.595363  1690 tablet_copy_service.cc:342] P 816fadcea8fc4893882f5f9b9f22a03d: Request end of tablet copy session 3b879799899a499c8d409ee9f1277cd1-7ebbb6d7c6c4478e95972de2f78d3f80 received from {username='slave'} at 127.0.0.1:57092
I20260709 13:03:08.595847  1690 tablet_copy_service.cc:434] P 816fadcea8fc4893882f5f9b9f22a03d: ending tablet copy session 3b879799899a499c8d409ee9f1277cd1-7ebbb6d7c6c4478e95972de2f78d3f80 on tablet 7ebbb6d7c6c4478e95972de2f78d3f80 with peer 3b879799899a499c8d409ee9f1277cd1
I20260709 13:03:08.623684  1499 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35464 (request call id 293) took 1036 ms. Trace:
W20260709 13:03:08.626242   431 thread.cc:527] Waited for 2000ms trying to join with diag-logger (tid 1626)
I20260709 13:03:08.629936 10544 tablet_bootstrap.cc:492] T 32dc176efd53437ab7735ea8c6ac4a0d P 3b879799899a499c8d409ee9f1277cd1: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:03:08.624182  1499 rpcz_store.cc:276] 0709 13:03:07.586856 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:07.587101 (+   245us) service_pool.cc:224] Handling call
0709 13:03:08.623605 (+1036504us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 13:03:08.339336 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 13:03:08.340010 (+   674us) write_op.cc:432] Acquiring schema lock in shared mode
0709 13:03:08.340053 (+    43us) write_op.cc:435] Acquired schema lock
0709 13:03:08.340074 (+    21us) tablet.cc:662] Decoding operations
0709 13:03:08.341199 (+  1125us) write_op.cc:620] Acquiring the partition lock for write op
0709 13:03:08.341299 (+   100us) write_op.cc:641] Partition lock acquired for write op
0709 13:03:08.341331 (+    32us) tablet.cc:685] Acquiring locks for 10 operations
0709 13:03:08.342228 (+   897us) tablet.cc:701] Row locks acquired
0709 13:03:08.342248 (+    20us) write_op.cc:260] PREPARE: finished
0709 13:03:08.342426 (+   178us) write_op.cc:270] Start()
0709 13:03:08.342583 (+   157us) write_op.cc:276] Timestamp: P: 1783602188342363 usec, L: 0
0709 13:03:08.342604 (+    21us) op_driver.cc:348] REPLICATION: starting
0709 13:03:08.353060 (+ 10456us) log.cc:844] Serialized 4062 byte log entry
0709 13:03:08.424134 (+ 71074us) op_driver.cc:464] REPLICATION: finished
0709 13:03:08.508792 (+ 84658us) write_op.cc:301] APPLY: starting
0709 13:03:08.508895 (+   103us) tablet.cc:1367] starting BulkCheckPresence
0709 13:03:08.583369 (+ 74474us) tablet.cc:1370] finished BulkCheckPresence
0709 13:03:08.583499 (+   130us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 13:03:08.608085 (+ 24586us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 13:03:08.608387 (+   302us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=20,key_file_lookups=20,delta_file_lookups=0,mrs_lookups=0
0709 13:03:08.608439 (+    52us) write_op.cc:312] APPLY: finished
0709 13:03:08.608766 (+   327us) log.cc:844] Serialized 97 byte log entry
0709 13:03:08.608958 (+   192us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:03:08.609117 (+   159us) write_op.cc:454] Released schema lock
0709 13:03:08.609171 (+    54us) write_op.cc:341] FINISH: Updating metrics
0709 13:03:08.609687 (+   516us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:03:08.609723 (+    36us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":114228,"reported_tablets":13,"child_traces":[["op",{"apply.queue_time_us":84370,"apply.run_cpu_time_us":15312,"apply.run_wall_time_us":100873,"cfile_cache_hit":42,"cfile_cache_hit_bytes":46007,"num_ops":10,"prepare.queue_time_us":181273,"prepare.run_cpu_time_us":4511,"prepare.run_wall_time_us":14091,"replication_time_us":81347,"thread_start_us":218130,"threads_started":2}]]}
I20260709 13:03:08.631207 10544 tablet_bootstrap.cc:492] T 32dc176efd53437ab7735ea8c6ac4a0d P 3b879799899a499c8d409ee9f1277cd1: Bootstrap complete.
I20260709 13:03:08.635254 10539 ts_tablet_manager.cc:933] T a92588e9316e4b4aa1691e8669c839d3 P 3b879799899a499c8d409ee9f1277cd1: Initiating tablet copy from peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:03:08.648046 10544 ts_tablet_manager.cc:1403] T 32dc176efd53437ab7735ea8c6ac4a0d P 3b879799899a499c8d409ee9f1277cd1: Time spent bootstrapping tablet: real 0.337s	user 0.057s	sys 0.030s
I20260709 13:03:08.654156 10544 raft_consensus.cc:359] T 32dc176efd53437ab7735ea8c6ac4a0d P 3b879799899a499c8d409ee9f1277cd1 [term 22 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } }
I20260709 13:03:08.655135 10544 raft_consensus.cc:740] T 32dc176efd53437ab7735ea8c6ac4a0d P 3b879799899a499c8d409ee9f1277cd1 [term 22 LEARNER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Initialized, Role: LEARNER
I20260709 13:03:08.655854 10544 consensus_queue.cc:260] T 32dc176efd53437ab7735ea8c6ac4a0d P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 22.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } }
I20260709 13:03:08.658869 10544 ts_tablet_manager.cc:1434] T 32dc176efd53437ab7735ea8c6ac4a0d P 3b879799899a499c8d409ee9f1277cd1: Time spent starting tablet: real 0.010s	user 0.009s	sys 0.000s
I20260709 13:03:08.661254  1690 tablet_copy_service.cc:342] P 816fadcea8fc4893882f5f9b9f22a03d: Request end of tablet copy session 3b879799899a499c8d409ee9f1277cd1-32dc176efd53437ab7735ea8c6ac4a0d received from {username='slave'} at 127.0.0.1:57092
I20260709 13:03:08.661639  1690 tablet_copy_service.cc:434] P 816fadcea8fc4893882f5f9b9f22a03d: ending tablet copy session 3b879799899a499c8d409ee9f1277cd1-32dc176efd53437ab7735ea8c6ac4a0d on tablet 32dc176efd53437ab7735ea8c6ac4a0d with peer 3b879799899a499c8d409ee9f1277cd1
I20260709 13:03:08.636952 10557 tablet_bootstrap.cc:492] T 127630d60f944b899508c92ee18fdddf P 3b879799899a499c8d409ee9f1277cd1: Bootstrap starting.
I20260709 13:03:08.715095 10539 tablet_copy_client.cc:323] T a92588e9316e4b4aa1691e8669c839d3 P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Beginning tablet copy session from remote peer at address 127.0.107.195:40693
I20260709 13:03:08.717413  1764 tablet_copy_service.cc:140] P af520b67945d443b901bbeca37cc5701: Received BeginTabletCopySession request for tablet a92588e9316e4b4aa1691e8669c839d3 from peer 3b879799899a499c8d409ee9f1277cd1 ({username='slave'} at 127.0.0.1:43156)
I20260709 13:03:08.718111  1764 tablet_copy_service.cc:161] P af520b67945d443b901bbeca37cc5701: Beginning new tablet copy session on tablet a92588e9316e4b4aa1691e8669c839d3 from peer 3b879799899a499c8d409ee9f1277cd1 at {username='slave'} at 127.0.0.1:43156: session id = 3b879799899a499c8d409ee9f1277cd1-a92588e9316e4b4aa1691e8669c839d3
I20260709 13:03:08.728199  1764 tablet_copy_source_session.cc:215] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 13:03:08.732066 10539 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a92588e9316e4b4aa1691e8669c839d3. 1 dirs total, 0 dirs full, 0 dirs failed
W20260709 13:03:08.751896  1717 consensus_peers.cc:597] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 13:03:08.789011 10539 tablet_copy_client.cc:806] T a92588e9316e4b4aa1691e8669c839d3 P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Starting download of 0 data blocks...
I20260709 13:03:08.789880 10539 tablet_copy_client.cc:670] T a92588e9316e4b4aa1691e8669c839d3 P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Starting download of 1 WAL segments...
W20260709 13:03:08.795768  1717 consensus_peers.cc:597] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 13:03:08.808501 10539 tablet_copy_client.cc:538] T a92588e9316e4b4aa1691e8669c839d3 P 3b879799899a499c8d409ee9f1277cd1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 13:03:08.832116 10539 tablet_bootstrap.cc:492] T a92588e9316e4b4aa1691e8669c839d3 P 3b879799899a499c8d409ee9f1277cd1: Bootstrap starting.
W20260709 13:03:08.835153  2298 consensus_peers.cc:597] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:03:08.841212  1717 consensus_peers.cc:597] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:03:08.850965  1497 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35458 (request call id 310) took 69415 ms (client timeout 15000 ms). Trace:
W20260709 13:03:08.851552  1497 rpcz_store.cc:269] 0709 13:01:59.435900 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:59.446689 (+ 10789us) service_pool.cc:224] Handling call
0709 13:03:08.850717 (+69404028us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1050216,"reported_tablets":47}
W20260709 13:03:08.864652  2298 consensus_peers.cc:597] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260709 13:03:08.890622  2332 raft_consensus.cc:1217] T 32dc176efd53437ab7735ea8c6ac4a0d P 3b879799899a499c8d409ee9f1277cd1 [term 22 LEARNER]: Deduplicated request from leader. Original: 22.11->[22.12-22.12]   Dedup: 22.12->[]
W20260709 13:03:08.917023  2298 consensus_peers.cc:597] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:03:08.924206  1717 consensus_peers.cc:597] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 13:03:08.963256  1499 catalog_manager.cc:5697] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 reported cstate change: term changed from 20 to 25, leader changed from af520b67945d443b901bbeca37cc5701 (127.0.107.195) to 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194). New cstate: current_term: 25 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } }
I20260709 13:03:08.964988  1499 catalog_manager.cc:5697] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 reported cstate change: term changed from 28 to 29, leader changed from 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) to af520b67945d443b901bbeca37cc5701 (127.0.107.195). New cstate: current_term: 29 leader_uuid: "af520b67945d443b901bbeca37cc5701" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: HEALTHY } } }
I20260709 13:03:08.966764  1499 catalog_manager.cc:5697] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 reported cstate change: config changed from index -1 to 12, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New cstate: current_term: 22 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } } }
I20260709 13:03:08.973374  1499 catalog_manager.cc:5697] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 reported cstate change: config changed from index -1 to 12, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New cstate: current_term: 22 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } } }
I20260709 13:03:08.975147  1499 catalog_manager.cc:5697] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 reported cstate change: term changed from 19 to 25, leader changed from af520b67945d443b901bbeca37cc5701 (127.0.107.195) to 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196). New cstate: current_term: 25 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 13:03:08.976951  1499 catalog_manager.cc:5697] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 reported cstate change: config changed from index -1 to 15, NON_VOTER 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) added. New cstate: current_term: 20 leader_uuid: "af520b67945d443b901bbeca37cc5701" committed_config { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20260709 13:03:08.996590  1717 consensus_peers.cc:597] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:03:08.999763  1717 consensus_peers.cc:597] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:03:09.010959  2298 consensus_peers.cc:597] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:03:09.014008  1502 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:43476 (request call id 197) took 59953 ms (client timeout 15000 ms). Trace:
W20260709 13:03:09.021173  1502 rpcz_store.cc:269] 0709 13:02:09.069409 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:13.922084 (+4852675us) service_pool.cc:224] Handling call
0709 13:03:09.013741 (+55091657us) inbound_call.cc:177] Queueing success response
Metrics: {"reported_tablets":36}
I20260709 13:03:09.016299 10557 tablet_bootstrap.cc:492] T 127630d60f944b899508c92ee18fdddf P 3b879799899a499c8d409ee9f1277cd1: Bootstrap replayed 1/1 log segments. Stats: ops{read=13 overwritten=1 applied=12 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:03:09.029855 10557 tablet_bootstrap.cc:492] T 127630d60f944b899508c92ee18fdddf P 3b879799899a499c8d409ee9f1277cd1: Bootstrap complete.
I20260709 13:03:09.031023 10557 ts_tablet_manager.cc:1403] T 127630d60f944b899508c92ee18fdddf P 3b879799899a499c8d409ee9f1277cd1: Time spent bootstrapping tablet: real 0.394s	user 0.046s	sys 0.020s
I20260709 13:03:09.047017 10557 raft_consensus.cc:359] T 127630d60f944b899508c92ee18fdddf P 3b879799899a499c8d409ee9f1277cd1 [term 22 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } }
I20260709 13:03:09.052676 10557 raft_consensus.cc:740] T 127630d60f944b899508c92ee18fdddf P 3b879799899a499c8d409ee9f1277cd1 [term 22 LEARNER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Initialized, Role: LEARNER
W20260709 13:03:09.056457  2298 consensus_peers.cc:597] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 13:03:09.057372 10539 tablet_bootstrap.cc:492] T a92588e9316e4b4aa1691e8669c839d3 P 3b879799899a499c8d409ee9f1277cd1: Bootstrap replayed 1/1 log segments. Stats: ops{read=17 overwritten=2 applied=15 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:03:09.053455 10557 consensus_queue.cc:260] T 127630d60f944b899508c92ee18fdddf P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 22.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } }
I20260709 13:03:09.066563 10539 tablet_bootstrap.cc:492] T a92588e9316e4b4aa1691e8669c839d3 P 3b879799899a499c8d409ee9f1277cd1: Bootstrap complete.
I20260709 13:03:09.077777  2333 raft_consensus.cc:1217] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 3b879799899a499c8d409ee9f1277cd1 [term 26 LEARNER]: Deduplicated request from leader. Original: 26.16->[26.17-26.17]   Dedup: 26.17->[]
I20260709 13:03:09.093578 10539 ts_tablet_manager.cc:1403] T a92588e9316e4b4aa1691e8669c839d3 P 3b879799899a499c8d409ee9f1277cd1: Time spent bootstrapping tablet: real 0.262s	user 0.052s	sys 0.019s
I20260709 13:03:09.096261 10557 ts_tablet_manager.cc:1434] T 127630d60f944b899508c92ee18fdddf P 3b879799899a499c8d409ee9f1277cd1: Time spent starting tablet: real 0.065s	user 0.006s	sys 0.001s
I20260709 13:03:09.115043 10539 raft_consensus.cc:359] T a92588e9316e4b4aa1691e8669c839d3 P 3b879799899a499c8d409ee9f1277cd1 [term 20 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } }
I20260709 13:03:09.118032 10539 raft_consensus.cc:740] T a92588e9316e4b4aa1691e8669c839d3 P 3b879799899a499c8d409ee9f1277cd1 [term 20 LEARNER]: Becoming Follower/Learner. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Initialized, Role: LEARNER
I20260709 13:03:09.118956 10539 consensus_queue.cc:260] T a92588e9316e4b4aa1691e8669c839d3 P 3b879799899a499c8d409ee9f1277cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 15, Last appended: 20.15, Last appended by leader: 15, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } }
I20260709 13:03:09.124773 10539 ts_tablet_manager.cc:1434] T a92588e9316e4b4aa1691e8669c839d3 P 3b879799899a499c8d409ee9f1277cd1: Time spent starting tablet: real 0.028s	user 0.004s	sys 0.003s
I20260709 13:03:09.128057  1764 tablet_copy_service.cc:342] P af520b67945d443b901bbeca37cc5701: Request end of tablet copy session 3b879799899a499c8d409ee9f1277cd1-a92588e9316e4b4aa1691e8669c839d3 received from {username='slave'} at 127.0.0.1:43156
I20260709 13:03:09.128660  1764 tablet_copy_service.cc:434] P af520b67945d443b901bbeca37cc5701: ending tablet copy session 3b879799899a499c8d409ee9f1277cd1-a92588e9316e4b4aa1691e8669c839d3 on tablet a92588e9316e4b4aa1691e8669c839d3 with peer 3b879799899a499c8d409ee9f1277cd1
I20260709 13:03:09.141990  1690 tablet_copy_service.cc:342] P 816fadcea8fc4893882f5f9b9f22a03d: Request end of tablet copy session 3b879799899a499c8d409ee9f1277cd1-127630d60f944b899508c92ee18fdddf received from {username='slave'} at 127.0.0.1:57092
I20260709 13:03:09.142411  1690 tablet_copy_service.cc:434] P 816fadcea8fc4893882f5f9b9f22a03d: ending tablet copy session 3b879799899a499c8d409ee9f1277cd1-127630d60f944b899508c92ee18fdddf on tablet 127630d60f944b899508c92ee18fdddf with peer 3b879799899a499c8d409ee9f1277cd1
I20260709 13:03:09.182991  1680 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "127630d60f944b899508c92ee18fdddf"
dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:39628
I20260709 13:03:09.183799  1680 raft_consensus.cc:606] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 22 LEADER]: Received request to transfer leadership
I20260709 13:03:09.190224  1680 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "127630d60f944b899508c92ee18fdddf"
dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:39628
I20260709 13:03:09.191731  1680 raft_consensus.cc:606] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 22 LEADER]: Received request to transfer leadership
I20260709 13:03:09.195884  2333 raft_consensus.cc:1217] T a92588e9316e4b4aa1691e8669c839d3 P 3b879799899a499c8d409ee9f1277cd1 [term 20 LEARNER]: Deduplicated request from leader. Original: 20.14->[20.15-20.15]   Dedup: 20.15->[]
I20260709 13:03:09.210132  1680 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "127630d60f944b899508c92ee18fdddf"
dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:39628
I20260709 13:03:09.210759  1680 raft_consensus.cc:606] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 22 LEADER]: Received request to transfer leadership
W20260709 13:03:09.218601  1551 auto_rebalancer.cc:893] Removing replace marker failed: Service unavailable: leader transfer in progress
W20260709 13:03:09.219059  1551 auto_rebalancer.cc:901] Could not move replica: Remote error: Service unavailable: leadership transfer for 127630d60f944b899508c92ee18fdddf already in progress
W20260709 13:03:09.219369  1551 auto_rebalancer.cc:330] scheduled replica move failed to complete: Remote error: Service unavailable: leadership transfer for 127630d60f944b899508c92ee18fdddf already in progress
W20260709 13:03:09.235980  2298 consensus_peers.cc:597] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:03:09.246093  1501 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:43476 (request call id 196) took 87077 ms (client timeout 15000 ms). Trace:
W20260709 13:03:09.250455  1501 rpcz_store.cc:269] 0709 13:01:42.168986 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:42.171625 (+  2639us) service_pool.cc:224] Handling call
0709 13:03:09.245847 (+87074222us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":9805,"reported_tablets":36}
I20260709 13:03:09.248915  1500 catalog_manager.cc:5697] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d reported cstate change: term changed from 18 to 22, leader changed from 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) to 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193). New cstate: current_term: 22 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
W20260709 13:03:09.250011  1643 consensus_peers.cc:597] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:03:09.310283  1643 consensus_peers.cc:597] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:03:09.317003  1717 consensus_peers.cc:597] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 13:03:09.325423 10536 raft_consensus.cc:993] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d: : Instructing follower af520b67945d443b901bbeca37cc5701 to start an election
I20260709 13:03:09.326053 10511 raft_consensus.cc:1081] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 22 LEADER]: Signalling peer af520b67945d443b901bbeca37cc5701 to start an election
I20260709 13:03:09.328595  1752 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "127630d60f944b899508c92ee18fdddf"
dest_uuid: "af520b67945d443b901bbeca37cc5701"
 from {username='slave'} at 127.0.0.1:55848
I20260709 13:03:09.329406  1752 raft_consensus.cc:493] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:03:09.343190  1752 raft_consensus.cc:3060] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:03:09.356389  1503 catalog_manager.cc:5697] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 reported cstate change: term changed from 15 to 20, leader changed from 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) to 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193). New cstate: current_term: 20 leader_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
W20260709 13:03:09.389772  1643 consensus_peers.cc:597] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:03:09.412482  2298 consensus_peers.cc:597] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 13:03:09.415683  1752 raft_consensus.cc:515] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Starting forced leader election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } }
I20260709 13:03:09.418679  1752 leader_election.cc:290] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 23 election: Requested vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:03:09.420043  1680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "127630d60f944b899508c92ee18fdddf" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: true dest_uuid: "816fadcea8fc4893882f5f9b9f22a03d"
I20260709 13:03:09.424245  1680 raft_consensus.cc:3055] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 22 LEADER]: Stepping down as leader of term 22
I20260709 13:03:09.429836  2333 raft_consensus.cc:1217] T 127630d60f944b899508c92ee18fdddf P 3b879799899a499c8d409ee9f1277cd1 [term 22 LEARNER]: Deduplicated request from leader. Original: 22.11->[22.12-22.12]   Dedup: 22.12->[]
I20260709 13:03:09.424723  1680 raft_consensus.cc:740] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 816fadcea8fc4893882f5f9b9f22a03d, State: Running, Role: LEADER
I20260709 13:03:09.434366  1680 consensus_queue.cc:260] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 22.12, Last appended by leader: 12, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } }
W20260709 13:03:09.421970  1717 consensus_peers.cc:597] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 13:03:09.447266 10533 raft_consensus.cc:1064] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d: attempting to promote NON_VOTER 3b879799899a499c8d409ee9f1277cd1 to VOTER
I20260709 13:03:09.451752 10533 consensus_queue.cc:237] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 22.12, Last appended by leader: 10, Current term: 22, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:03:09.452243  1717 leader_election.cc:336] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 23 election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111)
I20260709 13:03:09.445852  1680 raft_consensus.cc:3060] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:03:09.490775  1503 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:43476 (request call id 200) took 3529 ms. Trace:
W20260709 13:03:09.494521  2298 consensus_peers.cc:597] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260709 13:03:09.496873  1500 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35458 (request call id 313) took 5050 ms. Trace:
I20260709 13:03:09.500753  1500 rpcz_store.cc:276] 0709 13:03:04.446565 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:05.267436 (+820871us) service_pool.cc:224] Handling call
0709 13:03:09.496780 (+4229344us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 13:03:09.278780 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 13:03:09.279467 (+   687us) write_op.cc:432] Acquiring schema lock in shared mode
0709 13:03:09.279508 (+    41us) write_op.cc:435] Acquired schema lock
0709 13:03:09.279523 (+    15us) tablet.cc:662] Decoding operations
0709 13:03:09.279910 (+   387us) write_op.cc:620] Acquiring the partition lock for write op
0709 13:03:09.279978 (+    68us) write_op.cc:641] Partition lock acquired for write op
0709 13:03:09.280010 (+    32us) tablet.cc:685] Acquiring locks for 1 operations
0709 13:03:09.280259 (+   249us) tablet.cc:701] Row locks acquired
0709 13:03:09.280274 (+    15us) write_op.cc:260] PREPARE: finished
0709 13:03:09.280458 (+   184us) write_op.cc:270] Start()
0709 13:03:09.280608 (+   150us) write_op.cc:276] Timestamp: P: 1783602189280407 usec, L: 0
0709 13:03:09.280632 (+    24us) op_driver.cc:348] REPLICATION: starting
0709 13:03:09.281379 (+   747us) log.cc:844] Serialized 536 byte log entry
0709 13:03:09.283880 (+  2501us) op_driver.cc:464] REPLICATION: finished
0709 13:03:09.291805 (+  7925us) write_op.cc:301] APPLY: starting
0709 13:03:09.291895 (+    90us) tablet.cc:1367] starting BulkCheckPresence
0709 13:03:09.292833 (+   938us) tablet.cc:1370] finished BulkCheckPresence
0709 13:03:09.292926 (+    93us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 13:03:09.293621 (+   695us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 13:03:09.293822 (+   201us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=2,key_file_lookups=2,delta_file_lookups=0,mrs_lookups=0
0709 13:03:09.293868 (+    46us) write_op.cc:312] APPLY: finished
0709 13:03:09.294102 (+   234us) log.cc:844] Serialized 25 byte log entry
0709 13:03:09.294274 (+   172us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:03:09.294368 (+    94us) write_op.cc:454] Released schema lock
0709 13:03:09.294407 (+    39us) write_op.cc:341] FINISH: Updating metrics
0709 13:03:09.321188 (+ 26781us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:03:09.321220 (+    32us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":54332,"reported_tablets":47,"spinlock_wait_cycles":1408,"child_traces":[["op",{"apply.queue_time_us":7681,"apply.run_cpu_time_us":3098,"apply.run_wall_time_us":23510,"cfile_cache_hit":4,"cfile_cache_hit_bytes":4188,"num_ops":1,"prepare.queue_time_us":637,"prepare.run_cpu_time_us":2939,"prepare.run_wall_time_us":4802,"replication_time_us":3095}]]}
I20260709 13:03:09.511891  2333 raft_consensus.cc:1275] T 32dc176efd53437ab7735ea8c6ac4a0d P 3b879799899a499c8d409ee9f1277cd1 [term 22 LEARNER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 22 index: 12. Preceding OpId from leader: term: 22 index: 13. (index mismatch)
W20260709 13:03:09.516470  1643 consensus_peers.cc:597] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 13:03:09.519550  1752 raft_consensus.cc:1275] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 22 index: 12. Preceding OpId from leader: term: 22 index: 13. (index mismatch)
I20260709 13:03:09.519629 10533 consensus_queue.cc:1048] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 13:03:09.521739 10533 consensus_queue.cc:1048] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260709 13:03:09.532835  1717 consensus_peers.cc:597] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 13:03:09.540297 10536 raft_consensus.cc:1064] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d: attempting to promote NON_VOTER 3b879799899a499c8d409ee9f1277cd1 to VOTER
I20260709 13:03:09.497563  1503 rpcz_store.cc:276] 0709 13:03:05.961345 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:06.120376 (+159031us) service_pool.cc:224] Handling call
0709 13:03:09.490653 (+3370277us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 13:03:09.397372 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 13:03:09.398032 (+   660us) write_op.cc:432] Acquiring schema lock in shared mode
0709 13:03:09.398091 (+    59us) write_op.cc:435] Acquired schema lock
0709 13:03:09.398114 (+    23us) tablet.cc:662] Decoding operations
0709 13:03:09.398584 (+   470us) write_op.cc:620] Acquiring the partition lock for write op
0709 13:03:09.398670 (+    86us) write_op.cc:641] Partition lock acquired for write op
0709 13:03:09.398698 (+    28us) tablet.cc:685] Acquiring locks for 1 operations
0709 13:03:09.398917 (+   219us) tablet.cc:701] Row locks acquired
0709 13:03:09.398935 (+    18us) write_op.cc:260] PREPARE: finished
0709 13:03:09.399080 (+   145us) write_op.cc:270] Start()
0709 13:03:09.399184 (+   104us) write_op.cc:276] Timestamp: P: 1783602189399036 usec, L: 0
0709 13:03:09.399205 (+    21us) op_driver.cc:348] REPLICATION: starting
0709 13:03:09.399869 (+   664us) log.cc:844] Serialized 536 byte log entry
0709 13:03:09.402115 (+  2246us) op_driver.cc:464] REPLICATION: finished
0709 13:03:09.402475 (+   360us) write_op.cc:301] APPLY: starting
0709 13:03:09.402559 (+    84us) tablet.cc:1367] starting BulkCheckPresence
0709 13:03:09.403494 (+   935us) tablet.cc:1370] finished BulkCheckPresence
0709 13:03:09.403614 (+   120us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 13:03:09.404357 (+   743us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 13:03:09.404525 (+   168us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=2,key_file_lookups=2,delta_file_lookups=0,mrs_lookups=0
0709 13:03:09.404601 (+    76us) write_op.cc:312] APPLY: finished
0709 13:03:09.404800 (+   199us) log.cc:844] Serialized 25 byte log entry
0709 13:03:09.404959 (+   159us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:03:09.405046 (+    87us) write_op.cc:454] Released schema lock
0709 13:03:09.405083 (+    37us) write_op.cc:341] FINISH: Updating metrics
0709 13:03:09.405458 (+   375us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:03:09.405487 (+    29us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":19774,"reported_tablets":16,"child_traces":[["op",{"apply.queue_time_us":94,"apply.run_cpu_time_us":2987,"apply.run_wall_time_us":2986,"cfile_cache_hit":4,"cfile_cache_hit_bytes":4188,"num_ops":1,"prepare.queue_time_us":1084,"prepare.run_cpu_time_us":2781,"prepare.run_wall_time_us":2781,"replication_time_us":2777}]]}
I20260709 13:03:09.507606  1680 raft_consensus.cc:2468] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 23.
I20260709 13:03:09.547799 10536 consensus_queue.cc:237] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 26.17, Last appended by leader: 15, Current term: 26, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:03:09.552341  1643 consensus_peers.cc:597] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 13:03:09.555955  1718 leader_election.cc:304] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701; no voters: 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:03:09.561926 10520 raft_consensus.cc:2804] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 13:03:09.563158 10520 raft_consensus.cc:697] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 23 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:03:09.564141 10520 consensus_queue.cc:237] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 22.12, Last appended by leader: 12, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } }
W20260709 13:03:09.572978  1643 consensus_peers.cc:597] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 13:03:09.591660  2332 raft_consensus.cc:1275] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 3b879799899a499c8d409ee9f1277cd1 [term 26 LEARNER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 26 index: 17. Preceding OpId from leader: term: 26 index: 18. (index mismatch)
I20260709 13:03:09.593429 10533 consensus_queue.cc:1048] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
W20260709 13:03:09.626798   431 thread.cc:527] Waited for 3000ms trying to join with diag-logger (tid 1626)
W20260709 13:03:09.626545  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 60873.8us
W20260709 13:03:09.668543  1643 consensus_peers.cc:597] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 13:03:09.682816  1751 raft_consensus.cc:1275] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 26 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 26 index: 17. Preceding OpId from leader: term: 26 index: 18. (index mismatch)
I20260709 13:03:09.686146 10580 raft_consensus.cc:2955] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 22 LEADER]: Committing config change with OpId 22.13: config changed from index 12 to 13, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 13:03:09.692178 10579 consensus_queue.cc:1048] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 13:03:09.693085  2332 raft_consensus.cc:2955] T 32dc176efd53437ab7735ea8c6ac4a0d P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Committing config change with OpId 22.13: config changed from index 12 to 13, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 13:03:09.698617  1500 catalog_manager.cc:5697] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 reported cstate change: term changed from 22 to 23, leader changed from 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194) to af520b67945d443b901bbeca37cc5701 (127.0.107.195). New cstate: current_term: 23 leader_uuid: "af520b67945d443b901bbeca37cc5701" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260709 13:03:09.719025  1753 raft_consensus.cc:2955] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Committing config change with OpId 22.13: config changed from index 12 to 13, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 13:03:09.744653  1503 catalog_manager.cc:5697] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d reported cstate change: term changed from 22 to 23, leader changed from 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) to 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194). New cstate: current_term: 23 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:03:09.749478  1503 catalog_manager.cc:5697] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d reported cstate change: term changed from 25 to 28, leader changed from 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) to 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194). New cstate: current_term: 28 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:03:09.752101  1503 catalog_manager.cc:5697] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d reported cstate change: term changed from 20 to 21, leader changed from 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) to 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196). New cstate: current_term: 21 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 13:03:09.778630 10560 raft_consensus.cc:1064] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701: attempting to promote NON_VOTER 3b879799899a499c8d409ee9f1277cd1 to VOTER
I20260709 13:03:09.779901 10580 raft_consensus.cc:2955] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 26 LEADER]: Committing config change with OpId 26.18: config changed from index 17 to 18, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 13:03:09.782207 10560 consensus_queue.cc:237] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.15, Last appended by leader: 13, Current term: 20, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:03:09.794970  1752 raft_consensus.cc:2955] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 26 FOLLOWER]: Committing config change with OpId 26.18: config changed from index 17 to 18, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 13:03:09.802878  1500 catalog_manager.cc:5697] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 reported cstate change: config changed from index 12 to 13, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New cstate: current_term: 22 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 13:03:09.833801  2332 raft_consensus.cc:1275] T a92588e9316e4b4aa1691e8669c839d3 P 3b879799899a499c8d409ee9f1277cd1 [term 20 LEARNER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 20 index: 15. Preceding OpId from leader: term: 20 index: 16. (index mismatch)
I20260709 13:03:09.838234 10520 consensus_queue.cc:1048] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
W20260709 13:03:09.850077  1717 consensus_peers.cc:597] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:03:09.859423  1717 consensus_peers.cc:597] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 13:03:09.874238  1672 raft_consensus.cc:1275] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 20 index: 15. Preceding OpId from leader: term: 20 index: 16. (index mismatch)
I20260709 13:03:09.879535 10560 consensus_queue.cc:1048] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 13:03:09.910933  2332 raft_consensus.cc:2955] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 3b879799899a499c8d409ee9f1277cd1 [term 26 FOLLOWER]: Committing config change with OpId 26.18: config changed from index 17 to 18, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 13:03:09.933346  1499 catalog_manager.cc:5697] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 reported cstate change: config changed from index 17 to 18, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New cstate: current_term: 26 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 13:03:09.950738 10520 raft_consensus.cc:2955] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 20 LEADER]: Committing config change with OpId 20.16: config changed from index 15 to 16, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
W20260709 13:03:09.967438  2298 consensus_peers.cc:597] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 13:03:09.971894  1672 consensus_queue.cc:237] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 26.18, Last appended by leader: 15, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:03:09.972965  2333 raft_consensus.cc:3060] T 127630d60f944b899508c92ee18fdddf P 3b879799899a499c8d409ee9f1277cd1 [term 22 LEARNER]: Advancing to term 23
W20260709 13:03:09.981750  2298 consensus_peers.cc:597] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 13:03:09.976820  1680 raft_consensus.cc:2955] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Committing config change with OpId 20.16: config changed from index 15 to 16, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 13:03:10.015697  2332 raft_consensus.cc:2955] T a92588e9316e4b4aa1691e8669c839d3 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Committing config change with OpId 20.16: config changed from index 15 to 16, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 13:03:10.022469  2330 raft_consensus.cc:1275] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 3b879799899a499c8d409ee9f1277cd1 [term 26 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 26 index: 18. Preceding OpId from leader: term: 26 index: 19. (index mismatch)
I20260709 13:03:10.022851  1501 catalog_manager.cc:5697] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 reported cstate change: config changed from index 15 to 16, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New cstate: current_term: 20 leader_uuid: "af520b67945d443b901bbeca37cc5701" committed_config { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:03:10.027621 10511 consensus_queue.cc:1048] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 13:03:10.030602  1753 raft_consensus.cc:1275] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 26 FOLLOWER]: Refusing update from remote peer 816fadcea8fc4893882f5f9b9f22a03d: Log matching property violated. Preceding OpId in replica: term: 26 index: 18. Preceding OpId from leader: term: 26 index: 19. (index mismatch)
W20260709 13:03:10.030978  1643 consensus_peers.cc:597] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 13:03:10.032400 10511 consensus_queue.cc:1048] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20260709 13:03:10.039053 10536 raft_consensus.cc:2955] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 26 LEADER]: Committing config change with OpId 26.19: config changed from index 18 to 19, VOTER 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) evicted. New config: { opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 13:03:10.051927  2330 raft_consensus.cc:2955] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 3b879799899a499c8d409ee9f1277cd1 [term 26 FOLLOWER]: Committing config change with OpId 26.19: config changed from index 18 to 19, VOTER 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) evicted. New config: { opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 13:03:10.064043  1752 raft_consensus.cc:2955] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 26 FOLLOWER]: Committing config change with OpId 26.19: config changed from index 18 to 19, VOTER 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) evicted. New config: { opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 13:03:10.099756  2333 raft_consensus.cc:1275] T 127630d60f944b899508c92ee18fdddf P 3b879799899a499c8d409ee9f1277cd1 [term 23 LEARNER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 22 index: 12. Preceding OpId from leader: term: 23 index: 13. (index mismatch)
I20260709 13:03:10.111457 10520 consensus_queue.cc:1048] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 13:03:10.115370  1488 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 7ebbb6d7c6c4478e95972de2f78d3f80 with cas_config_opid_index 18: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 13:03:10.108533  1500 catalog_manager.cc:5697] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 reported cstate change: config changed from index 18 to 19, VOTER 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) evicted. New cstate: current_term: 26 leader_uuid: "816fadcea8fc4893882f5f9b9f22a03d" committed_config { opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 13:03:10.144511 10484 raft_consensus.cc:1064] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701: attempting to promote NON_VOTER 3b879799899a499c8d409ee9f1277cd1 to VOTER
W20260709 13:03:10.145140 10484 raft_consensus.cc:1068] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701: Unable to promote non-voter 3b879799899a499c8d409ee9f1277cd1: Illegal state: Leader has not yet committed an operation in its own term
I20260709 13:03:10.155510  1680 raft_consensus.cc:1275] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 22 index: 12. Preceding OpId from leader: term: 23 index: 13. (index mismatch)
W20260709 13:03:10.157378  1717 consensus_peers.cc:597] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 13:03:10.203864  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 46549.9us
W20260709 13:03:10.204768  1717 consensus_peers.cc:597] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 13:03:10.212023  1753 consensus_queue.cc:237] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 20.16, Last appended by leader: 13, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:03:10.221525 10520 consensus_queue.cc:1048] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
W20260709 13:03:10.227665  1643 consensus_peers.cc:597] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 13:03:10.237754  2330 raft_consensus.cc:1275] T a92588e9316e4b4aa1691e8669c839d3 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 20 index: 16. Preceding OpId from leader: term: 20 index: 17. (index mismatch)
I20260709 13:03:10.242129 10520 consensus_queue.cc:1048] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 13:03:10.256769 10560 raft_consensus.cc:2955] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 20 LEADER]: Committing config change with OpId 20.17: config changed from index 16 to 17, VOTER 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) evicted. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 13:03:10.264434  1673 raft_consensus.cc:1275] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 20 index: 16. Preceding OpId from leader: term: 20 index: 17. (index mismatch)
I20260709 13:03:10.272672  2330 raft_consensus.cc:2955] T a92588e9316e4b4aa1691e8669c839d3 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Committing config change with OpId 20.17: config changed from index 16 to 17, VOTER 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) evicted. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 13:03:10.276557 10520 consensus_queue.cc:1048] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
W20260709 13:03:10.287446  2298 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111) [suppressed 458 similar messages]
W20260709 13:03:10.301926  2298 consensus_peers.cc:597] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:03:10.332445  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 46850.2us
W20260709 13:03:10.335282  1487 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 58008.8us
W20260709 13:03:10.335196  1643 consensus_peers.cc:597] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 13:03:10.340804  1490 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet a92588e9316e4b4aa1691e8669c839d3 with cas_config_opid_index 16: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 13:03:10.304558  1680 raft_consensus.cc:2955] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Committing config change with OpId 20.17: config changed from index 16 to 17, VOTER 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) evicted. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
I20260709 13:03:10.358807  1500 catalog_manager.cc:5697] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 reported cstate change: config changed from index 16 to 17, VOTER 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193) evicted. New cstate: current_term: 20 leader_uuid: "af520b67945d443b901bbeca37cc5701" committed_config { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 13:03:10.361850  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 59121.1us
W20260709 13:03:10.405788  1487 catalog_manager.cc:4755] TS 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 7ebbb6d7c6c4478e95972de2f78d3f80: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111)
W20260709 13:03:10.467784  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 85548.6us
W20260709 13:03:10.533051  2301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 136089us
W20260709 13:03:10.545781  2298 consensus_peers.cc:597] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:03:10.581466  2298 consensus_peers.cc:597] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:03:10.627414   431 thread.cc:527] Waited for 4000ms trying to join with diag-logger (tid 1626)
W20260709 13:03:10.627740  1717 consensus_peers.cc:597] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260709 13:03:10.636858 10602 raft_consensus.cc:1064] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701: attempting to promote NON_VOTER 3b879799899a499c8d409ee9f1277cd1 to VOTER
I20260709 13:03:10.641269 10602 consensus_queue.cc:237] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 23.13, Last appended by leader: 12, Current term: 23, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:03:10.648416  1643 consensus_peers.cc:597] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:03:10.651618  1717 consensus_peers.cc:597] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 13:03:10.694026  1672 raft_consensus.cc:1275] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 23 index: 13. Preceding OpId from leader: term: 23 index: 14. (index mismatch)
W20260709 13:03:11.314375  1643 consensus_peers.cc:597] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:03:11.320248  1643 consensus_peers.cc:597] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:03:11.342710  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 650040us
I20260709 13:03:11.351392 10353 consensus_queue.cc:1048] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
W20260709 13:03:11.410501  1487 catalog_manager.cc:4755] TS 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet a92588e9316e4b4aa1691e8669c839d3: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111)
I20260709 13:03:11.419839  2330 raft_consensus.cc:1275] T 127630d60f944b899508c92ee18fdddf P 3b879799899a499c8d409ee9f1277cd1 [term 23 LEARNER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 23 index: 13. Preceding OpId from leader: term: 23 index: 14. (index mismatch)
W20260709 13:03:11.492713  2299 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 50834.9us
W20260709 13:03:11.490103  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 144788us
W20260709 13:03:11.551169  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 52252.5us
I20260709 13:03:11.551596 10353 consensus_queue.cc:1048] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.006s
I20260709 13:03:11.600059 10613 raft_consensus.cc:2955] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 23 LEADER]: Committing config change with OpId 23.14: config changed from index 12 to 14, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
W20260709 13:03:11.612516  2298 consensus_peers.cc:597] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:03:11.614565  1646 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:40693, user_credentials={real_user=slave}} blocked reactor thread for 58248.4us
W20260709 13:03:11.615518  2298 consensus_peers.cc:597] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:03:11.618212  2298 consensus_peers.cc:597] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20260709 13:03:11.622779  2337 raft_consensus.cc:2955] T 127630d60f944b899508c92ee18fdddf P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Committing config change with OpId 23.14: config changed from index 12 to 14, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
W20260709 13:03:11.628050   431 thread.cc:527] Waited for 5000ms trying to join with diag-logger (tid 1626)
I20260709 13:03:11.642625  1672 raft_consensus.cc:2955] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Committing config change with OpId 23.14: config changed from index 12 to 14, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } } }
W20260709 13:03:11.645213  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 53849.3us
W20260709 13:03:11.661484  1717 consensus_peers.cc:597] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:03:11.662101  1717 consensus_peers.cc:597] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 13:03:11.669485  1499 catalog_manager.cc:5697] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 reported cstate change: config changed from index 12 to 14, 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196) changed from NON_VOTER to VOTER. New cstate: current_term: 23 leader_uuid: "af520b67945d443b901bbeca37cc5701" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 13:03:11.704282  1717 consensus_peers.cc:597] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 13:03:11.705199  1717 consensus_peers.cc:597] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:03:11.705680  1717 consensus_peers.cc:597] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:03:11.720667  1534 debug-util.cc:398] Leaking SignalData structure 0x7b080036fe60 after lost signal to thread 434
W20260709 13:03:11.722182  1534 debug-util.cc:398] Leaking SignalData structure 0x7b0800277ee0 after lost signal to thread 1537
W20260709 13:03:11.724061  1534 debug-util.cc:398] Leaking SignalData structure 0x7b0800040de0 after lost signal to thread 1704
W20260709 13:03:11.725637  1534 debug-util.cc:398] Leaking SignalData structure 0x7b0800372dc0 after lost signal to thread 1778
W20260709 13:03:11.727402  1534 debug-util.cc:398] Leaking SignalData structure 0x7b08003cda40 after lost signal to thread 2361
W20260709 13:03:11.728238  1534 debug-util.cc:398] Leaking SignalData structure 0x7b0800381f40 after lost signal to thread 10576
W20260709 13:03:11.728608  1534 debug-util.cc:398] Leaking SignalData structure 0x7b08001031c0 after lost signal to thread 10577
W20260709 13:03:11.729130  1534 debug-util.cc:398] Leaking SignalData structure 0x7b0800346de0 after lost signal to thread 10589
W20260709 13:03:11.800527  2298 consensus_peers.cc:597] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:03:11.847347  1717 consensus_peers.cc:597] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:03:11.859637  2298 consensus_peers.cc:597] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:03:11.889278  1717 consensus_peers.cc:597] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:03:11.909268  2298 consensus_peers.cc:597] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:03:11.917430  1717 consensus_peers.cc:597] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:03:11.942696  1717 consensus_peers.cc:597] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:03:11.966720  1717 consensus_peers.cc:597] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:03:11.977016  2298 consensus_peers.cc:597] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:03:12.024556  1717 consensus_peers.cc:597] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:03:12.116420  2298 consensus_peers.cc:597] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:03:12.158126  2298 consensus_peers.cc:597] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:03:12.176730  1643 consensus_peers.cc:597] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:03:12.191529  2298 consensus_peers.cc:597] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:03:12.227578  1643 consensus_peers.cc:597] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:03:12.266681  1643 consensus_peers.cc:597] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:03:12.379848  1643 consensus_peers.cc:597] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:03:12.427975  1644 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 47210.5us
W20260709 13:03:12.450623  1643 consensus_peers.cc:597] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:03:12.456974  1643 consensus_peers.cc:597] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:03:12.462042  1717 consensus_peers.cc:597] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:03:12.475016  1717 consensus_peers.cc:597] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:03:12.493821  1717 consensus_peers.cc:597] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:03:12.628597   431 thread.cc:527] Waited for 6000ms trying to join with diag-logger (tid 1626)
W20260709 13:03:12.880178  1643 consensus_peers.cc:597] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:03:12.903766  1717 consensus_peers.cc:597] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:03:12.937546  1643 consensus_peers.cc:597] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:03:13.046998  2298 consensus_peers.cc:597] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:03:13.122520  2298 consensus_peers.cc:597] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:03:13.140669  1643 consensus_peers.cc:597] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:03:13.154898  1487 catalog_manager.cc:4755] TS 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 7ebbb6d7c6c4478e95972de2f78d3f80: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111)
W20260709 13:03:13.255678  2298 consensus_peers.cc:597] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:03:13.307241  1643 consensus_peers.cc:597] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:03:13.351955  2298 consensus_peers.cc:597] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:03:13.370834  2298 consensus_peers.cc:597] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:03:13.388526  1717 consensus_peers.cc:597] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:03:13.566262  1717 consensus_peers.cc:597] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:03:13.598511  2298 consensus_peers.cc:597] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:03:13.629245   431 thread.cc:527] Waited for 7000ms trying to join with diag-logger (tid 1626)
W20260709 13:03:13.666522  1717 consensus_peers.cc:597] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:03:13.758970  1717 consensus_peers.cc:597] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:03:13.836916  1643 consensus_peers.cc:597] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:03:13.841252  1717 consensus_peers.cc:597] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:03:13.844851  1643 consensus_peers.cc:597] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:03:13.869019  2298 consensus_peers.cc:597] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:03:13.913293  1717 consensus_peers.cc:597] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:03:14.118906  2298 consensus_peers.cc:597] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:03:14.165544  1717 consensus_peers.cc:597] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:03:14.419977  1717 consensus_peers.cc:597] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:03:14.440083  2298 consensus_peers.cc:597] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:03:14.442773  1717 consensus_peers.cc:597] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:03:14.462850  1717 consensus_peers.cc:597] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:03:14.500514  1643 consensus_peers.cc:597] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:03:14.548401  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 80546.2us
W20260709 13:03:14.565750  2298 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 96624.1us
W20260709 13:03:14.628650  2298 consensus_peers.cc:597] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:03:14.631102   431 thread.cc:527] Waited for 8000ms trying to join with diag-logger (tid 1626)
W20260709 13:03:14.644062  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:46625, user_credentials={real_user=slave}} blocked reactor thread for 120352us
W20260709 13:03:14.658322  1487 catalog_manager.cc:4755] TS 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 983621a6fc5b4c9ea0159a648dfe7a36: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111)
W20260709 13:03:14.660490  1717 consensus_peers.cc:597] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:03:14.682502  1717 consensus_peers.cc:597] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:03:14.696677  1717 consensus_peers.cc:597] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:03:14.706075  1717 consensus_peers.cc:597] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:03:14.714641  2298 consensus_peers.cc:597] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:03:14.748342  1643 consensus_peers.cc:597] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:03:14.761826  2298 consensus_peers.cc:597] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:03:14.762631  2298 consensus_peers.cc:597] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:03:14.763265  2298 consensus_peers.cc:597] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:03:14.892817  2298 consensus_peers.cc:597] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:03:14.938647  1643 consensus_peers.cc:597] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:03:14.978466  1643 consensus_peers.cc:597] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:03:14.993702  1717 consensus_peers.cc:597] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:03:15.001004  1643 consensus_peers.cc:597] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:03:15.016628  1643 consensus_peers.cc:597] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:03:15.046511  1643 consensus_peers.cc:597] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:03:15.123104  1717 consensus_peers.cc:597] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:03:15.224426  1643 consensus_peers.cc:597] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:03:15.231659  1717 consensus_peers.cc:597] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:03:15.298440  1717 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111) [suppressed 428 similar messages]
W20260709 13:03:15.304068  1717 consensus_peers.cc:597] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:03:15.348941  1643 consensus_peers.cc:597] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:03:15.419919  2298 consensus_peers.cc:597] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:03:15.660629  1643 consensus_peers.cc:597] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:03:15.715901   431 thread.cc:527] Waited for 9000ms trying to join with diag-logger (tid 1626)
W20260709 13:03:15.806916  2298 consensus_peers.cc:597] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:03:15.808815  2298 consensus_peers.cc:597] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:03:15.886202  2298 consensus_peers.cc:597] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:03:15.896004  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 50478us
W20260709 13:03:15.900025  1717 consensus_peers.cc:597] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:03:15.945008  1717 consensus_peers.cc:597] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:03:15.943557  2298 consensus_peers.cc:597] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:03:15.962486  2298 consensus_peers.cc:597] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:03:16.034888  1717 consensus_peers.cc:597] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:03:16.397428  1717 consensus_peers.cc:597] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:03:16.458362  1717 consensus_peers.cc:597] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:03:16.485651  1643 consensus_peers.cc:597] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:03:16.489125  1717 consensus_peers.cc:597] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:03:16.516418  2298 consensus_peers.cc:597] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 13:03:16.558151  1717 consensus_peers.cc:597] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:03:16.594678  1643 consensus_peers.cc:597] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:03:16.717072   431 thread.cc:527] Waited for 10000ms trying to join with diag-logger (tid 1626)
W20260709 13:03:16.897979  1717 consensus_peers.cc:597] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:03:16.898847  1717 consensus_peers.cc:597] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:03:16.909979  1717 consensus_peers.cc:597] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:03:16.910656  1717 consensus_peers.cc:597] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:03:16.989283  2298 consensus_peers.cc:597] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:03:16.994568  2298 consensus_peers.cc:597] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 13:03:17.002923  2298 consensus_peers.cc:597] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 13:03:17.072423  1643 consensus_peers.cc:597] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:03:17.073431  1643 consensus_peers.cc:597] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:03:17.096513  2298 consensus_peers.cc:597] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:03:17.172765  2298 consensus_peers.cc:597] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:03:17.211073  1717 consensus_peers.cc:597] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:03:17.277839  1717 consensus_peers.cc:597] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:03:17.318032  2298 consensus_peers.cc:597] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 13:03:17.369021  1643 consensus_peers.cc:597] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:03:17.378679  2298 consensus_peers.cc:597] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:03:17.384909  1717 consensus_peers.cc:597] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:03:17.385669  1717 consensus_peers.cc:597] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:03:17.416261  1643 consensus_peers.cc:597] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:03:17.621356  1643 consensus_peers.cc:597] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:03:17.631105  1643 consensus_peers.cc:597] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:03:17.642259  1643 consensus_peers.cc:597] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:03:17.670131  2298 consensus_peers.cc:597] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 13:03:17.742449  1643 consensus_peers.cc:597] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:03:17.775365  1717 consensus_peers.cc:597] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:03:17.784780   431 thread.cc:527] Waited for 11000ms trying to join with diag-logger (tid 1626)
W20260709 13:03:17.807880  1643 consensus_peers.cc:597] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:03:17.884420  1717 consensus_peers.cc:597] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:03:18.065954  1717 consensus_peers.cc:597] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:03:18.072445  2298 consensus_peers.cc:597] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:03:18.254974  2298 consensus_peers.cc:597] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:03:18.301203  1643 consensus_peers.cc:597] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:03:18.395056  1717 consensus_peers.cc:597] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:03:18.464236  1717 consensus_peers.cc:597] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:03:18.509531  1717 consensus_peers.cc:597] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 13:03:18.570487  2298 consensus_peers.cc:597] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:03:18.601300  2298 consensus_peers.cc:597] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:03:18.677400  1717 consensus_peers.cc:597] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 13:03:18.717664  2298 consensus_peers.cc:597] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 13:03:18.762320  2298 consensus_peers.cc:597] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 13:03:18.864771  1717 consensus_peers.cc:597] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260709 13:03:18.910769   431 tablet_server.cc:196] TabletServer@127.0.107.193:0 shutdown complete.
W20260709 13:03:18.951838  2298 consensus_peers.cc:597] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 13:03:19.010362  1717 consensus_peers.cc:597] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 13:03:19.035614  1717 consensus_peers.cc:597] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:03:19.042599  1643 consensus_peers.cc:597] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:03:19.137728  1643 consensus_peers.cc:597] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:03:19.162662  1643 consensus_peers.cc:597] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:03:19.253193  1717 consensus_peers.cc:597] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:03:19.288596  1717 consensus_peers.cc:597] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:03:19.594839  1717 consensus_peers.cc:597] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:03:19.610374  2298 consensus_peers.cc:597] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 13:03:19.624595  1487 catalog_manager.cc:4755] TS 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 7ebbb6d7c6c4478e95972de2f78d3f80: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111)
W20260709 13:03:19.631543  2298 consensus_peers.cc:597] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 13:03:19.660017  2298 consensus_peers.cc:597] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 13:03:19.676402  1717 consensus_peers.cc:597] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 13:03:19.698719  1643 consensus_peers.cc:597] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 13:03:19.701723  1717 consensus_peers.cc:597] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 13:03:19.709236  2298 consensus_peers.cc:597] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 13:03:19.727947  2298 consensus_peers.cc:597] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 13:03:19.728535  2298 consensus_peers.cc:597] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20260709 13:03:19.740798   431 tablet_server.cc:179] TabletServer@127.0.107.194:0 shutting down...
W20260709 13:03:19.758487  1717 consensus_peers.cc:597] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:03:19.763515  2298 consensus_peers.cc:597] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 13:03:19.793953  1718 consensus_peers.cc:597] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:03:19.802642  2299 consensus_peers.cc:597] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:03:19.815908  1643 consensus_peers.cc:597] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 13:03:19.818991  1718 consensus_peers.cc:597] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:03:19.894455  1718 consensus_peers.cc:597] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:03:19.896375  2298 consensus_peers.cc:597] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 13:03:19.907891  2299 consensus_peers.cc:597] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:03:19.911882  2299 consensus_peers.cc:597] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:03:19.909734  1718 consensus_peers.cc:597] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:03:19.916392  1717 consensus_peers.cc:597] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:03:19.918669  2299 consensus_peers.cc:597] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:03:19.944720  1643 consensus_peers.cc:597] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 13:03:19.947798  1718 consensus_peers.cc:597] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:03:19.952571  2299 consensus_peers.cc:597] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:03:19.976615  1643 consensus_peers.cc:597] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 13:03:19.981451  2299 consensus_peers.cc:597] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:03:19.998903  2299 consensus_peers.cc:597] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:03:20.000479  1718 consensus_peers.cc:597] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:03:20.031924  2299 consensus_peers.cc:597] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:03:20.034835  1718 consensus_peers.cc:597] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:03:20.036532  1718 consensus_peers.cc:597] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:03:20.036556  1717 consensus_peers.cc:597] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 13:03:20.100883  1718 consensus_peers.cc:597] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:03:20.105387  2299 consensus_peers.cc:597] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:03:20.116070  1718 consensus_peers.cc:597] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:03:20.117841  1718 consensus_peers.cc:597] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:03:20.118543  1718 consensus_peers.cc:597] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:03:20.123606   431 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 13:03:20.126390   431 tablet_replica.cc:333] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d: stopping tablet replica
I20260709 13:03:20.128170   431 raft_consensus.cc:2243] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Raft consensus shutting down.
I20260709 13:03:20.129603   431 raft_consensus.cc:2272] T ef5897d71124435db68c88060b095421 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Raft consensus is shut down!
W20260709 13:03:20.130923  1718 consensus_peers.cc:597] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:03:20.133347   431 tablet_replica.cc:333] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d: stopping tablet replica
I20260709 13:03:20.143525   431 raft_consensus.cc:2243] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Raft consensus shutting down.
I20260709 13:03:20.144515   431 raft_consensus.cc:2272] T a8c04bcdff52470cb06667e46356e653 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Raft consensus is shut down!
I20260709 13:03:20.148350   431 tablet_replica.cc:333] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d: stopping tablet replica
I20260709 13:03:20.149412   431 raft_consensus.cc:2243] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 LEADER]: Raft consensus shutting down.
I20260709 13:03:20.151654   431 raft_consensus.cc:2272] T 41166f283aea41b0b3f5be60e8234443 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Raft consensus is shut down!
W20260709 13:03:20.155921  1643 consensus_peers.cc:597] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20260709 13:03:20.161537   431 tablet_replica.cc:333] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d: stopping tablet replica
I20260709 13:03:20.162628   431 raft_consensus.cc:2243] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Raft consensus shutting down.
I20260709 13:03:20.163363   431 raft_consensus.cc:2272] T f6028a708cb344fbb08ab8056921d9e7 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Raft consensus is shut down!
I20260709 13:03:20.166623   431 tablet_replica.cc:333] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d: stopping tablet replica
I20260709 13:03:20.171239   431 raft_consensus.cc:2243] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 22 LEADER]: Raft consensus shutting down.
W20260709 13:03:20.171808  2299 consensus_peers.cc:597] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:03:20.175151   431 raft_consensus.cc:2272] T 79ed08f2339a4b1ead1e70d78eaec2ec P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Raft consensus is shut down!
W20260709 13:03:20.175521  1718 consensus_peers.cc:597] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:03:20.178392   431 tablet_replica.cc:333] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d: stopping tablet replica
I20260709 13:03:20.191841   431 raft_consensus.cc:2243] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Raft consensus shutting down.
I20260709 13:03:20.192632   431 raft_consensus.cc:2272] T 1f1329a6c1e442bcba5945a2d1e65e85 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Raft consensus is shut down!
I20260709 13:03:20.196909   431 tablet_replica.cc:333] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d: stopping tablet replica
I20260709 13:03:20.197849   431 raft_consensus.cc:2243] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Raft consensus shutting down.
W20260709 13:03:20.197989  2299 consensus_peers.cc:597] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:03:20.198628   431 raft_consensus.cc:2272] T 453513ba44f34adb8b436305ee51fe3d P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Raft consensus is shut down!
I20260709 13:03:20.202049   431 tablet_replica.cc:333] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d: stopping tablet replica
I20260709 13:03:20.207306   431 raft_consensus.cc:2243] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 LEADER]: Raft consensus shutting down.
I20260709 13:03:20.215601   431 raft_consensus.cc:2272] T a93d4b974fee4d02afbaa26d69bfeb07 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Raft consensus is shut down!
I20260709 13:03:20.225483   431 tablet_replica.cc:333] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d: stopping tablet replica
I20260709 13:03:20.226665   431 raft_consensus.cc:2243] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 17 LEADER]: Raft consensus shutting down.
I20260709 13:03:20.239711   431 raft_consensus.cc:2272] T 2ebc67196b4f4119981cdb499e3350cc P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Raft consensus is shut down!
W20260709 13:03:20.239668  1718 consensus_peers.cc:597] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:03:20.244118  2299 consensus_peers.cc:597] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:03:20.246801  2299 consensus_peers.cc:597] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:03:20.248944  1718 consensus_peers.cc:597] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:03:20.251296   431 tablet_replica.cc:333] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d: stopping tablet replica
I20260709 13:03:20.252486   431 raft_consensus.cc:2243] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 LEADER]: Raft consensus shutting down.
I20260709 13:03:20.254155   431 raft_consensus.cc:2272] T 03e37c14b11849208b5c9c51f32e1702 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Raft consensus is shut down!
I20260709 13:03:20.257653   431 tablet_replica.cc:333] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d: stopping tablet replica
I20260709 13:03:20.259295   431 raft_consensus.cc:2243] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 28 LEADER]: Raft consensus shutting down.
I20260709 13:03:20.260923   431 raft_consensus.cc:2272] T f8ab62ffb73246dc9bcf39f7bee51147 P 816fadcea8fc4893882f5f9b9f22a03d [term 28 FOLLOWER]: Raft consensus is shut down!
I20260709 13:03:20.264138   431 tablet_replica.cc:333] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d: stopping tablet replica
I20260709 13:03:20.265136   431 raft_consensus.cc:2243] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 26 LEADER]: Raft consensus shutting down.
I20260709 13:03:20.273058   431 raft_consensus.cc:2272] T 344b4152823f4d618345b14ffd579400 P 816fadcea8fc4893882f5f9b9f22a03d [term 26 FOLLOWER]: Raft consensus is shut down!
I20260709 13:03:20.276516   431 tablet_replica.cc:333] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d: stopping tablet replica
I20260709 13:03:20.277601   431 raft_consensus.cc:2243] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 26 FOLLOWER]: Raft consensus shutting down.
I20260709 13:03:20.278473   431 raft_consensus.cc:2272] T c81785db7d654e4288fcaeacf4db2b49 P 816fadcea8fc4893882f5f9b9f22a03d [term 26 FOLLOWER]: Raft consensus is shut down!
I20260709 13:03:20.281840   431 tablet_replica.cc:333] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d: stopping tablet replica
I20260709 13:03:20.282888   431 raft_consensus.cc:2243] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 25 FOLLOWER]: Raft consensus shutting down.
I20260709 13:03:20.284942   431 raft_consensus.cc:2272] T 983621a6fc5b4c9ea0159a648dfe7a36 P 816fadcea8fc4893882f5f9b9f22a03d [term 25 FOLLOWER]: Raft consensus is shut down!
I20260709 13:03:20.288406   431 tablet_replica.cc:333] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d: stopping tablet replica
I20260709 13:03:20.289400   431 raft_consensus.cc:2243] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 25 FOLLOWER]: Raft consensus shutting down.
I20260709 13:03:20.290302   431 raft_consensus.cc:2272] T 78cabb03415041ca9c1a5a400e506131 P 816fadcea8fc4893882f5f9b9f22a03d [term 25 FOLLOWER]: Raft consensus is shut down!
W20260709 13:03:20.296723  1643 consensus_peers.cc:597] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20260709 13:03:20.298017   431 tablet_replica.cc:333] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d: stopping tablet replica
I20260709 13:03:20.299291   431 raft_consensus.cc:2243] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 25 LEADER]: Raft consensus shutting down.
W20260709 13:03:20.299801  2299 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 474 similar messages]
I20260709 13:03:20.300729   431 raft_consensus.cc:2272] T b9bb5648969b4299a776843d72043448 P 816fadcea8fc4893882f5f9b9f22a03d [term 25 FOLLOWER]: Raft consensus is shut down!
I20260709 13:03:20.304015   431 tablet_replica.cc:333] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d: stopping tablet replica
I20260709 13:03:20.304944   431 raft_consensus.cc:2243] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Raft consensus shutting down.
I20260709 13:03:20.305994   431 raft_consensus.cc:2272] T 84e78b66ceec4f759ae9e101a5469e4c P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Raft consensus is shut down!
I20260709 13:03:20.309535   431 tablet_replica.cc:333] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d: stopping tablet replica
I20260709 13:03:20.310485   431 raft_consensus.cc:2243] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 LEADER]: Raft consensus shutting down.
I20260709 13:03:20.311879   431 raft_consensus.cc:2272] T 5108457796a949b7b753ab22b575b7a0 P 816fadcea8fc4893882f5f9b9f22a03d [term 17 FOLLOWER]: Raft consensus is shut down!
I20260709 13:03:20.315084   431 tablet_replica.cc:333] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d: stopping tablet replica
I20260709 13:03:20.316044   431 raft_consensus.cc:2243] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 25 FOLLOWER]: Raft consensus shutting down.
I20260709 13:03:20.316782   431 raft_consensus.cc:2272] T 03daa4b346f54f1da820d1ff6cd1ff86 P 816fadcea8fc4893882f5f9b9f22a03d [term 25 FOLLOWER]: Raft consensus is shut down!
I20260709 13:03:20.327353   431 tablet_replica.cc:333] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d: stopping tablet replica
I20260709 13:03:20.338711   431 raft_consensus.cc:2243] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Raft consensus shutting down.
I20260709 13:03:20.340093   431 raft_consensus.cc:2272] T 127630d60f944b899508c92ee18fdddf P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Raft consensus is shut down!
W20260709 13:03:20.343197  1717 consensus_peers.cc:597] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 13:03:20.344153  1643 consensus_peers.cc:597] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260709 13:03:20.344799   431 tablet_replica.cc:333] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d: stopping tablet replica
I20260709 13:03:20.345840   431 raft_consensus.cc:2243] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Raft consensus shutting down.
I20260709 13:03:20.359338   431 raft_consensus.cc:2272] T 87b86d46388a41cd90374c203cb27914 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Raft consensus is shut down!
I20260709 13:03:20.363518   431 tablet_replica.cc:333] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d: stopping tablet replica
I20260709 13:03:20.364796   431 raft_consensus.cc:2243] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 LEADER]: Raft consensus shutting down.
I20260709 13:03:20.366634   431 raft_consensus.cc:2272] T 58d4fdd02c6d4667aa718fb146a76505 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Raft consensus is shut down!
I20260709 13:03:20.371656   431 tablet_replica.cc:333] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d: stopping tablet replica
I20260709 13:03:20.372932   431 raft_consensus.cc:2243] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 25 FOLLOWER]: Raft consensus shutting down.
I20260709 13:03:20.374060   431 raft_consensus.cc:2272] T bb255e8a61e7458babbbdf0183773a81 P 816fadcea8fc4893882f5f9b9f22a03d [term 25 FOLLOWER]: Raft consensus is shut down!
I20260709 13:03:20.415434   431 tablet_replica.cc:333] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d: stopping tablet replica
I20260709 13:03:20.416515   431 raft_consensus.cc:2243] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 27 FOLLOWER]: Raft consensus shutting down.
I20260709 13:03:20.417263   431 raft_consensus.cc:2272] T 513043a9b96c4f70adad53ac4b5d01dc P 816fadcea8fc4893882f5f9b9f22a03d [term 27 FOLLOWER]: Raft consensus is shut down!
I20260709 13:03:20.420629   431 tablet_replica.cc:333] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d: stopping tablet replica
I20260709 13:03:20.421523   431 raft_consensus.cc:2243] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 25 FOLLOWER]: Raft consensus shutting down.
I20260709 13:03:20.422399   431 raft_consensus.cc:2272] T 0615de9bff034ecb9ed6173433128ff5 P 816fadcea8fc4893882f5f9b9f22a03d [term 25 FOLLOWER]: Raft consensus is shut down!
I20260709 13:03:20.501824   431 tablet_replica.cc:333] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d: stopping tablet replica
I20260709 13:03:20.502961   431 raft_consensus.cc:2243] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Raft consensus shutting down.
I20260709 13:03:20.503798   431 raft_consensus.cc:2272] T 86abe5695d5643f888190301b88dacef P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Raft consensus is shut down!
W20260709 13:03:20.505133  2298 consensus_peers.cc:597] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:03:20.545825  1717 consensus_peers.cc:597] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20260709 13:03:20.597663   431 tablet_replica.cc:333] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d: stopping tablet replica
I20260709 13:03:20.601383   431 raft_consensus.cc:2243] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 LEADER]: Raft consensus shutting down.
I20260709 13:03:20.603636   431 raft_consensus.cc:2272] T 83226095f79a46dab445849b1febd425 P 816fadcea8fc4893882f5f9b9f22a03d [term 23 FOLLOWER]: Raft consensus is shut down!
W20260709 13:03:20.618417  2298 consensus_peers.cc:597] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20260709 13:03:20.661304   431 tablet_replica.cc:333] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d: stopping tablet replica
I20260709 13:03:20.662654   431 raft_consensus.cc:2243] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 25 LEADER]: Raft consensus shutting down.
I20260709 13:03:20.714509 10682 raft_consensus.cc:493] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:03:20.715386 10682 raft_consensus.cc:515] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:03:20.721187  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3f391a609743de886c525de483fc29" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:03:20.720957 10682 leader_election.cc:290] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:03:20.723500   431 raft_consensus.cc:2272] T e4f4b860aab3464383ce8f9d990f4a4a P 816fadcea8fc4893882f5f9b9f22a03d [term 25 FOLLOWER]: Raft consensus is shut down!
W20260709 13:03:20.724298  1643 leader_election.cc:336] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111)
I20260709 13:03:20.725582  1644 leader_election.cc:304] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:03:20.727510 10682 raft_consensus.cc:2749] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:03:20.732683   431 tablet_replica.cc:333] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d: stopping tablet replica
I20260709 13:03:20.733778   431 raft_consensus.cc:2243] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Raft consensus shutting down.
I20260709 13:03:20.734864   431 raft_consensus.cc:2272] T 2a521fb6c3e7489d9694fd81cadeb33f P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Raft consensus is shut down!
I20260709 13:03:20.744195   431 tablet_replica.cc:333] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d: stopping tablet replica
I20260709 13:03:20.745139   431 raft_consensus.cc:2243] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 24 FOLLOWER]: Raft consensus shutting down.
I20260709 13:03:20.745760   431 raft_consensus.cc:2272] T e947690dcb3849cb9f9731123d363b91 P 816fadcea8fc4893882f5f9b9f22a03d [term 24 FOLLOWER]: Raft consensus is shut down!
I20260709 13:03:20.757467   431 tablet_replica.cc:333] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d: stopping tablet replica
I20260709 13:03:20.762573   431 raft_consensus.cc:2243] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Raft consensus shutting down.
I20260709 13:03:20.763370   431 raft_consensus.cc:2272] T 13031f9e955e4f5fb289c79ee2c46108 P 816fadcea8fc4893882f5f9b9f22a03d [term 21 FOLLOWER]: Raft consensus is shut down!
I20260709 13:03:20.766628   431 tablet_replica.cc:333] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d: stopping tablet replica
I20260709 13:03:20.767509   431 raft_consensus.cc:2243] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Raft consensus shutting down.
I20260709 13:03:20.768119   431 raft_consensus.cc:2272] T e26344297fff44bb8d2099b8e64d7df2 P 816fadcea8fc4893882f5f9b9f22a03d [term 18 FOLLOWER]: Raft consensus is shut down!
W20260709 13:03:20.773145  1717 consensus_peers.cc:597] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20260709 13:03:20.779140   431 tablet_replica.cc:333] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d: stopping tablet replica
I20260709 13:03:20.780107   431 raft_consensus.cc:2243] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Raft consensus shutting down.
I20260709 13:03:20.781131   431 raft_consensus.cc:2272] T fe7e5e9266614676a9c05c398cf7ba9e P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Raft consensus is shut down!
I20260709 13:03:20.792361   431 tablet_replica.cc:333] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d: stopping tablet replica
I20260709 13:03:20.793228   431 raft_consensus.cc:2243] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 27 LEADER]: Raft consensus shutting down.
I20260709 13:03:20.794703   431 raft_consensus.cc:2272] T 1c2f5fa72dee4ec58979ace307d2acc3 P 816fadcea8fc4893882f5f9b9f22a03d [term 27 FOLLOWER]: Raft consensus is shut down!
I20260709 13:03:20.797912   431 tablet_replica.cc:333] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d: stopping tablet replica
I20260709 13:03:20.798846   431 raft_consensus.cc:2243] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Raft consensus shutting down.
I20260709 13:03:20.800135   431 raft_consensus.cc:2272] T a92588e9316e4b4aa1691e8669c839d3 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Raft consensus is shut down!
I20260709 13:03:20.815477   431 tablet_replica.cc:333] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d: stopping tablet replica
I20260709 13:03:20.816413   431 raft_consensus.cc:2243] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 26 FOLLOWER]: Raft consensus shutting down.
I20260709 13:03:20.817252   431 raft_consensus.cc:2272] T a40e423ed583480280352b261913276c P 816fadcea8fc4893882f5f9b9f22a03d [term 26 FOLLOWER]: Raft consensus is shut down!
I20260709 13:03:20.820380   431 tablet_replica.cc:333] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d: stopping tablet replica
I20260709 13:03:20.821254   431 raft_consensus.cc:2243] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 25 FOLLOWER]: Raft consensus shutting down.
I20260709 13:03:20.821900   431 raft_consensus.cc:2272] T 73dc0a9696be41f78e22f1f77a6eee72 P 816fadcea8fc4893882f5f9b9f22a03d [term 25 FOLLOWER]: Raft consensus is shut down!
I20260709 13:03:20.829412 10678 raft_consensus.cc:493] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 3b879799899a499c8d409ee9f1277cd1)
I20260709 13:03:20.830065 10678 raft_consensus.cc:515] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:03:20.832938 10678 leader_election.cc:290] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:03:20.835076  2328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21485c094584689917b2a97b2d3c624" candidate_uuid: "816fadcea8fc4893882f5f9b9f22a03d" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:03:20.837451   431 tablet_replica.cc:333] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d: stopping tablet replica
I20260709 13:03:20.838414   431 raft_consensus.cc:2243] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 22 LEADER]: Raft consensus shutting down.
I20260709 13:03:20.841640   431 raft_consensus.cc:2272] T 32dc176efd53437ab7735ea8c6ac4a0d P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Raft consensus is shut down!
I20260709 13:03:20.846012   431 tablet_replica.cc:333] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d: stopping tablet replica
I20260709 13:03:20.846949   431 raft_consensus.cc:2243] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 25 LEADER]: Raft consensus shutting down.
I20260709 13:03:20.848124   431 raft_consensus.cc:2272] T 65285571995c4ed5b593041598056a22 P 816fadcea8fc4893882f5f9b9f22a03d [term 25 FOLLOWER]: Raft consensus is shut down!
I20260709 13:03:20.867754   431 tablet_replica.cc:333] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d: stopping tablet replica
I20260709 13:03:20.868734   431 raft_consensus.cc:2243] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 LEADER]: Raft consensus shutting down.
I20260709 13:03:20.870005   431 raft_consensus.cc:2272] T 63cf5ab930f24f239fde96a128656333 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Raft consensus is shut down!
I20260709 13:03:20.873231   431 tablet_replica.cc:333] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d: stopping tablet replica
I20260709 13:03:20.874838   431 raft_consensus.cc:2243] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Raft consensus shutting down.
I20260709 13:03:20.878212   431 raft_consensus.cc:2272] T dd3f391a609743de886c525de483fc29 P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Raft consensus is shut down!
I20260709 13:03:20.887708   431 tablet_replica.cc:333] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d: stopping tablet replica
I20260709 13:03:20.888574   431 raft_consensus.cc:2243] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 20 LEADER]: Raft consensus shutting down.
I20260709 13:03:20.890056   431 raft_consensus.cc:2272] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 816fadcea8fc4893882f5f9b9f22a03d [term 20 FOLLOWER]: Raft consensus is shut down!
I20260709 13:03:20.893195   431 tablet_replica.cc:333] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d: stopping tablet replica
I20260709 13:03:20.894040   431 raft_consensus.cc:2243] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 29 FOLLOWER]: Raft consensus shutting down.
W20260709 13:03:20.906705  1717 consensus_peers.cc:597] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20260709 13:03:20.910590   431 raft_consensus.cc:2272] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P 816fadcea8fc4893882f5f9b9f22a03d [term 29 FOLLOWER]: Raft consensus is shut down!
I20260709 13:03:20.913988   431 tablet_replica.cc:333] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d: stopping tablet replica
I20260709 13:03:20.919153   431 raft_consensus.cc:2243] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Raft consensus shutting down.
I20260709 13:03:20.920599   431 raft_consensus.cc:2272] T 592fa647ae2545e09ad5ee25f8cb658e P 816fadcea8fc4893882f5f9b9f22a03d [term 22 FOLLOWER]: Raft consensus is shut down!
W20260709 13:03:20.926132  1643 leader_election.cc:336] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111)
I20260709 13:03:20.931262  1643 leader_election.cc:304] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 816fadcea8fc4893882f5f9b9f22a03d; no voters: 3b879799899a499c8d409ee9f1277cd1, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:03:20.923749   431 tablet_replica.cc:333] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d: stopping tablet replica
I20260709 13:03:20.935031   431 raft_consensus.cc:2243] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Raft consensus shutting down.
I20260709 13:03:20.935999   431 raft_consensus.cc:2272] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Raft consensus is shut down!
I20260709 13:03:20.936538 10682 raft_consensus.cc:2706] T d21485c094584689917b2a97b2d3c624 P 816fadcea8fc4893882f5f9b9f22a03d [term 19 FOLLOWER]: Received pre-election callback for term 20 while not running: Illegal state: RaftConsensus is not running: State = Stopped
I20260709 13:03:20.941605   431 tablet_replica.cc:333] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d: stopping tablet replica
I20260709 13:03:20.942924   431 raft_consensus.cc:2243] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 26 FOLLOWER]: Raft consensus shutting down.
I20260709 13:03:20.944199   431 raft_consensus.cc:2272] T b917a1e9c9aa40a0afce5a12c343eacd P 816fadcea8fc4893882f5f9b9f22a03d [term 26 FOLLOWER]: Raft consensus is shut down!
I20260709 13:03:20.972476   431 tablet_replica.cc:333] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d: stopping tablet replica
I20260709 13:03:20.995832   431 raft_consensus.cc:2243] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 26 LEADER]: Raft consensus shutting down.
I20260709 13:03:20.999753   431 raft_consensus.cc:2272] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 816fadcea8fc4893882f5f9b9f22a03d [term 26 FOLLOWER]: Raft consensus is shut down!
W20260709 13:03:21.011644  1717 consensus_peers.cc:597] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:03:21.184202  1717 consensus_peers.cc:597] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:03:21.195504  2298 consensus_peers.cc:597] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20260709 13:03:21.237521 10685 raft_consensus.cc:493] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:03:21.238215 10685 raft_consensus.cc:515] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:03:21.244671  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:03:21.248334  2299 leader_election.cc:336] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:03:21.248838  2299 leader_election.cc:304] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:03:21.249871 10686 raft_consensus.cc:2749] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:03:21.250912 10685 leader_election.cc:290] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:03:21.285000  2298 consensus_peers.cc:597] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20260709 13:03:21.290923 10685 raft_consensus.cc:493] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:03:21.291678 10685 raft_consensus.cc:515] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:03:21.294334 10685 leader_election.cc:290] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 13:03:21.301558  2299 leader_election.cc:336] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 18 pre-election: RPC error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 13:03:21.302780  2298 leader_election.cc:336] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 18 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111)
I20260709 13:03:21.303342  2298 leader_election.cc:304] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:03:21.343076 10669 raft_consensus.cc:2749] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20260709 13:03:21.352819  1717 consensus_peers.cc:597] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20260709 13:03:21.366315 10676 raft_consensus.cc:493] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:03:21.367137 10676 raft_consensus.cc:515] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:03:21.369761 10676 leader_election.cc:290] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 13:03:21.374646  1718 leader_election.cc:336] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 18 pre-election: RPC error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:03:21.400256 10685 raft_consensus.cc:493] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:03:21.402627 10685 raft_consensus.cc:515] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:03:21.410086 10685 leader_election.cc:290] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 13:03:21.412194  2298 consensus_peers.cc:597] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:03:21.406611  1717 leader_election.cc:336] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 18 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111)
I20260709 13:03:21.427729  1717 leader_election.cc:304] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 13:03:21.419847  2299 connection.cc:769] client connection to 127.0.107.194:46261 send error: Network error: sendmsg error: Broken pipe (error 32)
I20260709 13:03:21.429342 10651 raft_consensus.cc:2749] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:03:21.453634 10677 raft_consensus.cc:493] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:03:21.454294 10677 raft_consensus.cc:515] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:03:21.459815  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:03:21.462265 10676 raft_consensus.cc:493] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:03:21.462978 10676 raft_consensus.cc:515] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:03:21.470875 10676 leader_election.cc:290] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:03:21.472553 10677 leader_election.cc:290] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:03:21.473451  2299 leader_election.cc:336] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111)
W20260709 13:03:21.474826  2298 consensus_peers.cc:597] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20260709 13:03:21.507259 10651 raft_consensus.cc:493] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:03:21.508107 10651 raft_consensus.cc:515] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:03:21.510739 10651 leader_election.cc:290] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:03:21.532757  1717 consensus_peers.cc:597] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 13:03:21.544080  2299 leader_election.cc:336] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111)
I20260709 13:03:21.544699  2299 leader_election.cc:304] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
W20260709 13:03:21.551589  2298 leader_election.cc:336] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111)
I20260709 13:03:21.560812  2298 leader_election.cc:304] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 13:03:21.553339  1718 leader_election.cc:336] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111)
I20260709 13:03:21.568065 10685 raft_consensus.cc:2749] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 13:03:21.569350 10685 raft_consensus.cc:2749] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
W20260709 13:03:21.573112  1717 leader_election.cc:336] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111)
I20260709 13:03:21.574738 10688 raft_consensus.cc:493] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:03:21.583595 10688 raft_consensus.cc:515] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:03:21.586185 10688 leader_election.cc:290] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:03:21.590260 10651 raft_consensus.cc:493] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:03:21.617095 10651 raft_consensus.cc:515] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:03:21.619340 10651 leader_election.cc:290] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 13:03:21.625286  1718 leader_election.cc:336] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111)
I20260709 13:03:21.626188 10677 raft_consensus.cc:493] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:03:21.635651 10677 raft_consensus.cc:515] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:03:21.639273 10677 leader_election.cc:290] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:03:21.610188  1717 leader_election.cc:304] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:03:21.630277  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:03:21.640354  2333 raft_consensus.cc:2468] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 23.
W20260709 13:03:21.642001  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 81969.1us
I20260709 13:03:21.642616 10688 raft_consensus.cc:2749] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:03:21.651537  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:03:21.652331  1750 raft_consensus.cc:2468] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 23.
I20260709 13:03:21.656172  2301 leader_election.cc:304] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:03:21.657269 10677 raft_consensus.cc:2804] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 13:03:21.657723 10677 raft_consensus.cc:493] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:03:21.658165 10677 raft_consensus.cc:3060] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Advancing to term 24
I20260709 13:03:21.676268 10676 raft_consensus.cc:493] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:03:21.676950 10676 raft_consensus.cc:515] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:03:21.679445 10676 leader_election.cc:290] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:03:21.712240  1718 leader_election.cc:336] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111)
I20260709 13:03:21.715253 10685 raft_consensus.cc:493] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:03:21.717953 10685 raft_consensus.cc:515] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:03:21.721913  2298 consensus_peers.cc:597] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20260709 13:03:21.733565 10677 raft_consensus.cc:515] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:03:21.757129 10688 raft_consensus.cc:493] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:03:21.775779 10688 raft_consensus.cc:515] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:03:21.776760 10677 leader_election.cc:290] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 24 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:03:21.778029  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41166f283aea41b0b3f5be60e8234443" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:03:21.778720  1750 raft_consensus.cc:3060] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Advancing to term 24
I20260709 13:03:21.787030 10688 leader_election.cc:290] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:03:21.773120  2299 leader_election.cc:336] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111)
W20260709 13:03:21.762107  1718 leader_election.cc:336] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111)
I20260709 13:03:21.759049 10685 leader_election.cc:290] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:03:21.799278  1718 leader_election.cc:304] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
W20260709 13:03:21.808815  1718 leader_election.cc:336] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111)
W20260709 13:03:21.820221  1718 leader_election.cc:336] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111)
I20260709 13:03:21.826241  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:03:21.823762  2298 leader_election.cc:336] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111)
I20260709 13:03:21.826968  2337 raft_consensus.cc:2468] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 26.
I20260709 13:03:21.823909  1750 raft_consensus.cc:2468] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 24.
I20260709 13:03:21.828464 10651 raft_consensus.cc:2764] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Leader pre-election decision vote started in defunct term 23: won
W20260709 13:03:21.826020  1717 consensus_peers.cc:597] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20260709 13:03:21.831653  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:03:21.863165  1717 leader_election.cc:336] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111)
I20260709 13:03:21.863700  1717 leader_election.cc:304] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:03:21.863843  2337 raft_consensus.cc:2468] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 22.
I20260709 13:03:21.866551  2301 leader_election.cc:304] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:03:21.875628 10669 raft_consensus.cc:2804] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 13:03:21.876219 10669 raft_consensus.cc:697] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 24 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:03:21.877187 10669 consensus_queue.cc:237] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 23.15, Last appended by leader: 15, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:03:21.890341  1718 leader_election.cc:304] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:03:21.891345 10651 raft_consensus.cc:2804] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 13:03:21.891793 10651 raft_consensus.cc:493] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 26 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:03:21.892246 10651 raft_consensus.cc:3060] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 26 FOLLOWER]: Advancing to term 27
W20260709 13:03:21.891417  2298 consensus_peers.cc:597] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20260709 13:03:21.904531 10651 raft_consensus.cc:515] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:03:21.906780  1718 leader_election.cc:304] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:03:21.907774 10669 raft_consensus.cc:493] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:03:21.908059 10688 raft_consensus.cc:2749] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:03:21.908380 10669 raft_consensus.cc:515] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:03:21.909082  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 45993.5us
W20260709 13:03:21.909646  1717 consensus_peers.cc:597] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:03:21.910041  1717 leader_election.cc:336] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111)
I20260709 13:03:21.910475  1717 leader_election.cc:304] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:03:21.910766 10669 leader_election.cc:290] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:03:21.911223 10688 raft_consensus.cc:2749] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:03:21.910203  1499 catalog_manager.cc:5697] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 reported cstate change: term changed from 23 to 24, leader changed from 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194) to 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196). New cstate: current_term: 24 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:03:21.917492 10651 leader_election.cc:290] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 27 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:03:21.918514 10690 raft_consensus.cc:493] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:03:21.919144 10690 raft_consensus.cc:515] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:03:21.916803 10688 raft_consensus.cc:2804] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 13:03:21.920032 10688 raft_consensus.cc:493] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:03:21.920454 10688 raft_consensus.cc:3060] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:03:21.921494 10690 leader_election.cc:290] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:03:21.922247 10690 raft_consensus.cc:493] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:03:21.922798 10690 raft_consensus.cc:515] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:03:21.925060 10690 leader_election.cc:290] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 13:03:21.937417  2298 leader_election.cc:336] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111)
W20260709 13:03:21.943751  2299 leader_election.cc:336] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111)
I20260709 13:03:21.944304  2299 leader_election.cc:304] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:03:21.949747 10669 raft_consensus.cc:2749] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 13:03:21.950764 10688 raft_consensus.cc:515] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:03:21.951738  2299 leader_election.cc:336] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 24 election: RPC error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111)
W20260709 13:03:21.960068  1717 consensus_peers.cc:597] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:03:21.964339  1717 consensus_peers.cc:597] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20260709 13:03:21.964573  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "344b4152823f4d618345b14ffd579400" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:03:21.965183  2337 raft_consensus.cc:3060] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 26 FOLLOWER]: Advancing to term 27
I20260709 13:03:21.970212  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
W20260709 13:03:21.978544  1718 leader_election.cc:336] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 27 election: RPC error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111)
W20260709 13:03:21.984262  1718 leader_election.cc:336] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111)
W20260709 13:03:21.984844  1718 leader_election.cc:336] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111)
W20260709 13:03:21.985693  1718 leader_election.cc:336] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 23 election: RPC error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111)
I20260709 13:03:21.990697 10688 leader_election.cc:290] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 23 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:03:21.997861  1718 leader_election.cc:304] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 3b879799899a499c8d409ee9f1277cd1, 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:03:21.994254  2337 raft_consensus.cc:2468] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 27.
I20260709 13:03:22.002183 10669 raft_consensus.cc:493] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
W20260709 13:03:21.997170  2299 leader_election.cc:336] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111)
I20260709 13:03:22.002933  2299 leader_election.cc:304] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:03:22.002887 10669 raft_consensus.cc:515] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:03:22.000633  1717 leader_election.cc:336] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111)
I20260709 13:03:22.003909 10688 raft_consensus.cc:2749] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 13:03:22.003993  1717 leader_election.cc:304] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:03:22.006206 10669 leader_election.cc:290] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:03:22.006701 10685 raft_consensus.cc:2749] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:03:22.006949 10688 raft_consensus.cc:2749] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 13:03:22.008906  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
W20260709 13:03:22.009894  1718 consensus_peers.cc:597] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 13:03:22.013839  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ed08f2339a4b1ead1e70d78eaec2ec" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:03:22.015376  2337 raft_consensus.cc:3060] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:03:22.024791  1718 leader_election.cc:304] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
W20260709 13:03:22.027221  1717 consensus_peers.cc:597] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260709 13:03:22.027949 10651 raft_consensus.cc:2804] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 27 FOLLOWER]: Leader election won for term 27
I20260709 13:03:22.028458 10651 raft_consensus.cc:697] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 27 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:03:22.029304 10651 consensus_queue.cc:237] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 26.18, Last appended by leader: 18, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:03:22.039273  2299 leader_election.cc:336] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111)
I20260709 13:03:22.039822  2299 leader_election.cc:304] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:03:22.041234 10669 raft_consensus.cc:2749] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 13:03:22.044021 10651 raft_consensus.cc:493] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:03:22.044585 10651 raft_consensus.cc:515] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:03:22.049502 10651 leader_election.cc:290] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:03:22.055878  1717 leader_election.cc:336] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111)
I20260709 13:03:22.060783  2337 raft_consensus.cc:2468] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 23.
W20260709 13:03:22.066001  1718 leader_election.cc:336] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111)
I20260709 13:03:22.066581  1718 leader_election.cc:304] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:03:22.068722 10669 raft_consensus.cc:493] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:03:22.069379 10669 raft_consensus.cc:515] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:03:22.071905 10669 leader_election.cc:290] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:03:22.075084  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:03:22.075887  1750 raft_consensus.cc:2468] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 25.
I20260709 13:03:22.079196  2301 leader_election.cc:304] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:03:22.082257  1501 catalog_manager.cc:5697] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 reported cstate change: term changed from 26 to 27, leader changed from 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194) to af520b67945d443b901bbeca37cc5701 (127.0.107.195). New cstate: current_term: 27 leader_uuid: "af520b67945d443b901bbeca37cc5701" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 13:03:22.091241 10651 raft_consensus.cc:2749] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:03:22.092687  1718 leader_election.cc:304] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:03:22.095551 10651 raft_consensus.cc:2804] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 13:03:22.096119 10651 raft_consensus.cc:697] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 23 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:03:22.096555 10669 raft_consensus.cc:2804] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 13:03:22.097008 10669 raft_consensus.cc:493] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 25 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:03:22.097492 10669 raft_consensus.cc:3060] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 25 FOLLOWER]: Advancing to term 26
I20260709 13:03:22.097052 10651 consensus_queue.cc:237] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 22.16, Last appended by leader: 16, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:03:22.102612 10677 raft_consensus.cc:493] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 3b879799899a499c8d409ee9f1277cd1 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:03:22.103204 10677 raft_consensus.cc:515] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 3b879799899a499c8d409ee9f1277cd1 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:03:22.105350 10677 leader_election.cc:290] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:03:22.106829 10669 raft_consensus.cc:515] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:03:22.108832 10669 leader_election.cc:290] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 26 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:03:22.109459  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:03:22.111112  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4f4b860aab3464383ce8f9d990f4a4a" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701"
I20260709 13:03:22.111702  1750 raft_consensus.cc:3060] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 25 FOLLOWER]: Advancing to term 26
I20260709 13:03:22.122586  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:03:22.123359  2333 raft_consensus.cc:2468] T 32dc176efd53437ab7735ea8c6ac4a0d P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 22.
W20260709 13:03:22.128484  2299 leader_election.cc:336] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111)
W20260709 13:03:22.132023  2299 leader_election.cc:336] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111)
I20260709 13:03:22.132555  2299 leader_election.cc:304] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:03:22.133566 10669 raft_consensus.cc:2749] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 3b879799899a499c8d409ee9f1277cd1 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
W20260709 13:03:22.134358  2298 consensus_peers.cc:597] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 13:03:22.132128  1717 consensus_peers.cc:597] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 13:03:22.137373  2299 leader_election.cc:336] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 26 election: RPC error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111)
I20260709 13:03:22.134253  1750 raft_consensus.cc:2468] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 26.
I20260709 13:03:22.146420  2301 leader_election.cc:304] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:03:22.148777 10669 raft_consensus.cc:2804] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 26 FOLLOWER]: Leader election won for term 26
W20260709 13:03:22.155108  2298 consensus_peers.cc:597] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20260709 13:03:22.183560 10669 raft_consensus.cc:697] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 26 LEADER]: Becoming Leader. State: Replica: 3b879799899a499c8d409ee9f1277cd1, State: Running, Role: LEADER
I20260709 13:03:22.190994 10669 consensus_queue.cc:237] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 25.16, Last appended by leader: 16, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:03:22.196529  1501 catalog_manager.cc:5697] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 reported cstate change: term changed from 22 to 23, leader changed from 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194) to af520b67945d443b901bbeca37cc5701 (127.0.107.195). New cstate: current_term: 23 leader_uuid: "af520b67945d443b901bbeca37cc5701" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20260709 13:03:22.225211  2299 consensus_peers.cc:597] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 13:03:22.226773  1501 catalog_manager.cc:5697] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 reported cstate change: term changed from 25 to 26, leader changed from 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194) to 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196). New cstate: current_term: 26 leader_uuid: "3b879799899a499c8d409ee9f1277cd1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 13:03:22.252067  1717 consensus_peers.cc:597] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20260709 13:03:22.263883 10690 raft_consensus.cc:493] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
W20260709 13:03:22.264364  1717 consensus_peers.cc:597] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20260709 13:03:22.264559 10690 raft_consensus.cc:515] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:03:22.267100 10690 leader_election.cc:290] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
W20260709 13:03:22.267777  2299 consensus_peers.cc:597] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 13:03:22.278649 10651 raft_consensus.cc:493] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:03:22.279274 10651 raft_consensus.cc:515] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:03:22.281258 10651 leader_election.cc:290] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:03:22.284904  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:03:22.285663  2333 raft_consensus.cc:2468] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 27.
W20260709 13:03:22.304975  2298 consensus_peers.cc:597] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20260709 13:03:22.305284 10690 raft_consensus.cc:493] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:03:22.305945 10690 raft_consensus.cc:515] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:03:22.312923  1717 leader_election.cc:336] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111)
I20260709 13:03:22.314060 10690 leader_election.cc:290] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:03:22.315462  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1" is_pre_election: true
I20260709 13:03:22.316277  2333 raft_consensus.cc:2468] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 3b879799899a499c8d409ee9f1277cd1 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 26.
W20260709 13:03:22.318946  1718 leader_election.cc:336] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111)
I20260709 13:03:22.320663  1718 leader_election.cc:304] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:03:22.326679 10651 raft_consensus.cc:2804] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 13:03:22.327090 10651 raft_consensus.cc:493] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 27 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:03:22.327440 10651 raft_consensus.cc:3060] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 27 FOLLOWER]: Advancing to term 28
I20260709 13:03:22.335850 10690 raft_consensus.cc:493] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:03:22.336509 10690 raft_consensus.cc:515] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:03:22.338913 10690 leader_election.cc:290] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 13:03:22.345039  1717 leader_election.cc:336] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111)
W20260709 13:03:22.348845  1717 consensus_peers.cc:597] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20260709 13:03:22.353145  1718 leader_election.cc:304] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:03:22.355062 10676 raft_consensus.cc:2804] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 13:03:22.355460 10676 raft_consensus.cc:493] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 26 FOLLOWER]: Starting leader election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:03:22.355849 10676 raft_consensus.cc:3060] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 26 FOLLOWER]: Advancing to term 27
W20260709 13:03:22.358942  2299 consensus_peers.cc:597] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 13:03:22.361608 10651 raft_consensus.cc:515] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:03:22.364454 10651 leader_election.cc:290] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 28 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:03:22.368968  1750 raft_consensus.cc:1275] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 23 index: 15. Preceding OpId from leader: term: 24 index: 16. (index mismatch)
W20260709 13:03:22.374373  2299 consensus_peers.cc:597] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 13:03:22.377878 10669 consensus_queue.cc:1048] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
W20260709 13:03:22.391135  2298 consensus_peers.cc:597] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20260709 13:03:22.391703 10676 raft_consensus.cc:515] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:03:22.399283  2333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c2f5fa72dee4ec58979ace307d2acc3" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:03:22.399964  2333 raft_consensus.cc:3060] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 27 FOLLOWER]: Advancing to term 28
W20260709 13:03:22.400823  1718 consensus_peers.cc:597] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:03:22.401436  1718 leader_election.cc:336] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111)
W20260709 13:03:22.411448   431 thread.cc:527] Waited for 1000ms trying to join with diag-logger (tid 1701)
I20260709 13:03:22.418963  2333 raft_consensus.cc:2468] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 28.
I20260709 13:03:22.419243 10676 leader_election.cc:290] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 27 election: Requested vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), 3b879799899a499c8d409ee9f1277cd1 (127.0.107.196:46625)
I20260709 13:03:22.424175  2337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ebbb6d7c6c4478e95972de2f78d3f80" candidate_uuid: "af520b67945d443b901bbeca37cc5701" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "3b879799899a499c8d409ee9f1277cd1"
I20260709 13:03:22.424856  2337 raft_consensus.cc:3060] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 3b879799899a499c8d409ee9f1277cd1 [term 26 FOLLOWER]: Advancing to term 27
W20260709 13:03:22.433362  1718 leader_election.cc:336] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111)
I20260709 13:03:22.435868  2337 raft_consensus.cc:2468] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 3b879799899a499c8d409ee9f1277cd1 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af520b67945d443b901bbeca37cc5701 in term 27.
I20260709 13:03:22.438776  1718 leader_election.cc:304] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 13:03:22.456846  1718 consensus_peers.cc:597] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:03:22.457505  2299 consensus_peers.cc:597] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:03:22.457701  1718 leader_election.cc:336] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111)
I20260709 13:03:22.458309  1718 leader_election.cc:304] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:03:22.457711 10676 raft_consensus.cc:2749] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
W20260709 13:03:22.461582  1717 consensus_peers.cc:597] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20260709 13:03:22.462615 10676 raft_consensus.cc:2749] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
W20260709 13:03:22.463927  1718 leader_election.cc:336] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 28 election: RPC error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111)
I20260709 13:03:22.470259  2337 raft_consensus.cc:1275] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 27 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 26 index: 18. Preceding OpId from leader: term: 27 index: 19. (index mismatch)
I20260709 13:03:22.475055  1718 leader_election.cc:304] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 
I20260709 13:03:22.475915 10690 raft_consensus.cc:2804] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 27 FOLLOWER]: Leader election won for term 27
I20260709 13:03:22.476436 10690 raft_consensus.cc:697] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 27 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:03:22.477504 10690 consensus_queue.cc:237] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 26.19, Last appended by leader: 19, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:03:22.481493  1718 leader_election.cc:304] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d
I20260709 13:03:22.482504 10704 raft_consensus.cc:2804] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 28 FOLLOWER]: Leader election won for term 28
I20260709 13:03:22.483151 10704 raft_consensus.cc:697] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 28 LEADER]: Becoming Leader. State: Replica: af520b67945d443b901bbeca37cc5701, State: Running, Role: LEADER
I20260709 13:03:22.485834 10704 consensus_queue.cc:237] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 27.19, Last appended by leader: 19, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:03:22.489768  2298 consensus_peers.cc:597] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20260709 13:03:22.494643 10676 consensus_queue.cc:1048] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
W20260709 13:03:22.500061  1718 consensus_peers.cc:597] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:03:22.504101  2298 consensus_peers.cc:597] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 13:03:22.502424  1718 consensus_peers.cc:597] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:03:22.506282  1718 leader_election.cc:336] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 27 election: RPC error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111)
I20260709 13:03:22.530489  2333 raft_consensus.cc:1275] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 22 index: 16. Preceding OpId from leader: term: 23 index: 17. (index mismatch)
I20260709 13:03:22.532697 10688 consensus_queue.cc:1048] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
W20260709 13:03:22.536008  1718 consensus_peers.cc:597] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:03:22.545951  2299 consensus_peers.cc:597] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:03:22.546571  2298 consensus_peers.cc:597] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20260709 13:03:22.545225  1501 catalog_manager.cc:5697] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 reported cstate change: term changed from 26 to 27, leader changed from 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194) to af520b67945d443b901bbeca37cc5701 (127.0.107.195). New cstate: current_term: 27 leader_uuid: "af520b67945d443b901bbeca37cc5701" committed_config { opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20260709 13:03:22.563927  1718 consensus_peers.cc:597] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 13:03:22.565385  2299 consensus_peers.cc:597] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:03:22.570550  1718 consensus_peers.cc:597] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 13:03:22.574522  1501 catalog_manager.cc:5697] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 reported cstate change: term changed from 27 to 28, leader changed from 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194) to af520b67945d443b901bbeca37cc5701 (127.0.107.195). New cstate: current_term: 28 leader_uuid: "af520b67945d443b901bbeca37cc5701" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20260709 13:03:22.586541  2299 consensus_peers.cc:597] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:03:22.609674  1718 consensus_peers.cc:597] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:03:22.619356  2299 consensus_peers.cc:597] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 13:03:22.628598  2299 consensus_peers.cc:597] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:03:22.638938  1718 consensus_peers.cc:597] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:03:22.665125  2299 consensus_peers.cc:597] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 13:03:22.671031  1751 raft_consensus.cc:1275] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 26 FOLLOWER]: Refusing update from remote peer 3b879799899a499c8d409ee9f1277cd1: Log matching property violated. Preceding OpId in replica: term: 25 index: 16. Preceding OpId from leader: term: 26 index: 17. (index mismatch)
I20260709 13:03:22.674014 10701 consensus_queue.cc:1048] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
W20260709 13:03:22.683763  2299 consensus_peers.cc:597] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:03:22.692358  1718 consensus_peers.cc:597] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:03:22.712309  2299 consensus_peers.cc:597] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:03:22.719113  1718 consensus_peers.cc:597] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:03:22.722117  1718 consensus_peers.cc:597] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:03:22.773211  1718 consensus_peers.cc:597] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:03:22.779124  2299 consensus_peers.cc:597] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:03:22.831292  1718 consensus_peers.cc:597] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:03:22.859745  2298 consensus_peers.cc:597] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:03:22.868598  1718 consensus_peers.cc:597] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:03:22.929082  1487 catalog_manager.cc:4755] TS 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 983621a6fc5b4c9ea0159a648dfe7a36: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111)
W20260709 13:03:22.948117  1718 consensus_peers.cc:597] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 13:03:22.950026  2299 consensus_peers.cc:597] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:03:22.952741  1718 consensus_peers.cc:597] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:03:22.972671  1717 consensus_peers.cc:597] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 13:03:22.978039  1717 consensus_peers.cc:597] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 13:03:22.993009  1718 consensus_peers.cc:597] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 13:03:23.047398  1718 consensus_peers.cc:597] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 13:03:23.055699  2333 raft_consensus.cc:1275] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 28 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 27 index: 19. Preceding OpId from leader: term: 28 index: 20. (index mismatch)
I20260709 13:03:23.061684 10703 consensus_queue.cc:1048] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
I20260709 13:03:23.062079  2333 raft_consensus.cc:1275] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 3b879799899a499c8d409ee9f1277cd1 [term 27 FOLLOWER]: Refusing update from remote peer af520b67945d443b901bbeca37cc5701: Log matching property violated. Preceding OpId in replica: term: 26 index: 19. Preceding OpId from leader: term: 27 index: 20. (index mismatch)
I20260709 13:03:23.069383 10704 consensus_queue.cc:1048] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
W20260709 13:03:23.113478  2298 consensus_peers.cc:597] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20260709 13:03:23.125793 10677 raft_consensus.cc:493] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:03:23.126416 10677 raft_consensus.cc:515] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:03:23.129007 10677 leader_election.cc:290] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 13:03:23.141358  2298 leader_election.cc:336] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 18 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111)
W20260709 13:03:23.142689  2299 leader_election.cc:336] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 18 pre-election: RPC error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111)
I20260709 13:03:23.143193  2299 leader_election.cc:304] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:03:23.148468 10685 raft_consensus.cc:2749] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20260709 13:03:23.171712  1717 consensus_peers.cc:597] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20260709 13:03:23.224363 10677 raft_consensus.cc:493] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:03:23.225023 10677 raft_consensus.cc:515] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:03:23.232066  2298 leader_election.cc:336] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111)
W20260709 13:03:23.240943  2299 leader_election.cc:336] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111)
I20260709 13:03:23.241499  2299 leader_election.cc:304] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:03:23.242367 10685 raft_consensus.cc:2749] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 13:03:23.242594 10677 leader_election.cc:290] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:03:23.312788 10690 raft_consensus.cc:493] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:03:23.313649 10690 raft_consensus.cc:515] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:03:23.327557 10690 leader_election.cc:290] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 13:03:23.341113  1717 leader_election.cc:336] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 18 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111)
W20260709 13:03:23.342618  1718 leader_election.cc:336] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 18 pre-election: RPC error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111)
I20260709 13:03:23.343209  1718 leader_election.cc:304] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:03:23.344537 10690 raft_consensus.cc:2749] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20260709 13:03:23.411981   431 thread.cc:527] Waited for 2000ms trying to join with diag-logger (tid 1701)
I20260709 13:03:23.413612 10690 raft_consensus.cc:493] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:03:23.414269 10690 raft_consensus.cc:515] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:03:23.428383  1718 leader_election.cc:336] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111)
W20260709 13:03:23.429744  1717 leader_election.cc:336] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111)
I20260709 13:03:23.430306  1717 leader_election.cc:304] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:03:23.437573 10703 raft_consensus.cc:2749] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:03:23.433808 10690 leader_election.cc:290] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 13:03:23.523981  1717 consensus_peers.cc:597] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20260709 13:03:23.545104 10717 raft_consensus.cc:493] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:03:23.545789 10717 raft_consensus.cc:515] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:03:23.548435 10717 leader_election.cc:290] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 13:03:23.560088  1718 leader_election.cc:336] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111)
W20260709 13:03:23.567716  1717 leader_election.cc:336] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111)
I20260709 13:03:23.568223  1717 leader_election.cc:304] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:03:23.569032 10690 raft_consensus.cc:2749] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:03:23.614675 10717 raft_consensus.cc:493] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:03:23.615303 10717 raft_consensus.cc:515] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:03:23.617753 10717 leader_election.cc:290] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 13:03:23.625530  1717 leader_election.cc:336] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111)
W20260709 13:03:23.626307  1718 leader_election.cc:336] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111)
I20260709 13:03:23.626768  1718 leader_election.cc:304] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:03:23.628454 10717 raft_consensus.cc:2749] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260709 13:03:23.649362  1717 consensus_peers.cc:597] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 13:03:23.672842  1717 consensus_peers.cc:597] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 13:03:23.688199  1717 consensus_peers.cc:597] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 13:03:23.734193  1717 consensus_peers.cc:597] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:03:23.772927  2298 consensus_peers.cc:597] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:03:23.778306  2298 consensus_peers.cc:597] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20260709 13:03:23.824239 10685 raft_consensus.cc:493] T 32dc176efd53437ab7735ea8c6ac4a0d P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:03:23.824908 10685 raft_consensus.cc:515] T 32dc176efd53437ab7735ea8c6ac4a0d P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:03:23.831110  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:03:23.830616 10685 leader_election.cc:290] T 32dc176efd53437ab7735ea8c6ac4a0d P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:03:23.831909  1750 raft_consensus.cc:2468] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 22.
I20260709 13:03:23.833530 10717 raft_consensus.cc:493] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:03:23.834072 10717 raft_consensus.cc:515] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:03:23.834903  2299 leader_election.cc:336] T 32dc176efd53437ab7735ea8c6ac4a0d P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111)
I20260709 13:03:23.837543 10717 leader_election.cc:290] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 13:03:23.843295  1718 leader_election.cc:336] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111)
W20260709 13:03:23.843688  2298 leader_election.cc:336] T 32dc176efd53437ab7735ea8c6ac4a0d P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111)
I20260709 13:03:23.844496  2298 leader_election.cc:304] T 32dc176efd53437ab7735ea8c6ac4a0d P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:03:23.845328 10685 raft_consensus.cc:2749] T 32dc176efd53437ab7735ea8c6ac4a0d P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
W20260709 13:03:23.843966  1717 leader_election.cc:336] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111)
I20260709 13:03:23.846014  1717 leader_election.cc:304] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:03:23.849838 10717 raft_consensus.cc:2749] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
W20260709 13:03:23.859539  2298 consensus_peers.cc:597] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 13:03:23.871573  2298 consensus_peers.cc:597] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20260709 13:03:23.891528 10685 raft_consensus.cc:493] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:03:23.892213 10685 raft_consensus.cc:515] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:03:23.894552 10685 leader_election.cc:290] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 13:03:23.902267  2299 leader_election.cc:336] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111)
W20260709 13:03:23.911094  2298 leader_election.cc:336] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111)
I20260709 13:03:23.911590  2298 leader_election.cc:304] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:03:23.912648 10685 raft_consensus.cc:2749] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
W20260709 13:03:23.920415  1717 consensus_peers.cc:597] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:03:24.082371  2298 consensus_peers.cc:597] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260709 13:03:24.110124  1717 consensus_peers.cc:597] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20260709 13:03:24.226908 10685 raft_consensus.cc:493] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:03:24.229219 10685 raft_consensus.cc:515] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:03:24.233327 10685 leader_election.cc:290] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 13:03:24.262897  2299 leader_election.cc:336] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111)
W20260709 13:03:24.266705  2298 leader_election.cc:336] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111)
I20260709 13:03:24.267333  2298 leader_election.cc:304] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:03:24.270902 10685 raft_consensus.cc:2749] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
W20260709 13:03:24.276046  2298 consensus_peers.cc:597] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 13:03:24.358733  1717 consensus_peers.cc:597] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20260709 13:03:24.361402 10719 raft_consensus.cc:493] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:03:24.362211 10719 raft_consensus.cc:515] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:03:24.378718  1717 leader_election.cc:336] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111)
W20260709 13:03:24.381737  1718 leader_election.cc:336] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111)
I20260709 13:03:24.382452  1718 leader_election.cc:304] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:03:24.383378 10690 raft_consensus.cc:2749] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:03:24.379725 10719 leader_election.cc:290] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 13:03:24.396667  2299 consensus_peers.cc:597] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:03:24.411245  2298 consensus_peers.cc:597] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 13:03:24.413405   431 thread.cc:527] Waited for 3000ms trying to join with diag-logger (tid 1701)
W20260709 13:03:24.426119  1717 consensus_peers.cc:597] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 13:03:24.506345  1717 consensus_peers.cc:597] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 13:03:24.608799  1717 consensus_peers.cc:597] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 13:03:24.620963  1718 consensus_peers.cc:597] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 13:03:24.629264 10719 raft_consensus.cc:493] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:03:24.629958 10719 raft_consensus.cc:515] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:03:24.645622 10719 leader_election.cc:290] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 13:03:24.651033  1718 leader_election.cc:336] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111)
W20260709 13:03:24.652271  1717 leader_election.cc:336] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111)
I20260709 13:03:24.652797  1717 leader_election.cc:304] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:03:24.653666 10719 raft_consensus.cc:2749] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
W20260709 13:03:24.736491  2299 consensus_peers.cc:597] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:03:24.737951  1718 consensus_peers.cc:597] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:03:24.742249  1718 consensus_peers.cc:597] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:03:24.744882  1717 consensus_peers.cc:597] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 13:03:24.746914  1717 consensus_peers.cc:597] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 13:03:24.766322  2298 consensus_peers.cc:597] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260709 13:03:24.794870  2298 consensus_peers.cc:597] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260709 13:03:24.829856  2299 consensus_peers.cc:597] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:03:24.851935  1717 consensus_peers.cc:597] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 13:03:24.852257  2298 consensus_peers.cc:597] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 13:03:24.861161  1717 consensus_peers.cc:597] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 13:03:24.862047  2299 consensus_peers.cc:597] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:03:24.863678  1718 consensus_peers.cc:597] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:03:24.890776  2299 consensus_peers.cc:597] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 13:03:24.927846 10685 raft_consensus.cc:493] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:03:24.928465 10685 raft_consensus.cc:515] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:03:24.930858 10685 leader_election.cc:290] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 13:03:24.935935  2299 leader_election.cc:336] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 18 pre-election: RPC error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111)
W20260709 13:03:24.936411  2298 leader_election.cc:336] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 18 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111)
I20260709 13:03:24.936924  2298 leader_election.cc:304] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:03:24.937953 10685 raft_consensus.cc:2749] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20260709 13:03:24.948026  2298 consensus_peers.cc:597] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260709 13:03:24.979480  1718 consensus_peers.cc:597] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:03:24.991384  2299 consensus_peers.cc:597] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:03:25.051407  1718 consensus_peers.cc:597] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:03:25.056373  2298 consensus_peers.cc:597] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20260709 13:03:25.064541 10722 raft_consensus.cc:493] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:03:25.065225 10722 raft_consensus.cc:515] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:03:25.067589 10722 leader_election.cc:290] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 13:03:25.076274  1717 leader_election.cc:336] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111)
W20260709 13:03:25.078608  1718 leader_election.cc:336] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111)
I20260709 13:03:25.079416  1718 leader_election.cc:304] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:03:25.080279 10722 raft_consensus.cc:2749] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260709 13:03:25.083841  1718 consensus_peers.cc:597] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:03:25.094698  2299 consensus_peers.cc:597] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:03:25.096127  2298 consensus_peers.cc:597] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260709 13:03:25.098642  2299 consensus_peers.cc:597] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:03:25.104573  2299 consensus_peers.cc:597] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:03:25.117959  1718 consensus_peers.cc:597] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:03:25.159618  2299 consensus_peers.cc:597] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:03:25.170130  1718 consensus_peers.cc:597] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:03:25.192368  2299 consensus_peers.cc:597] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:03:25.257872  1718 consensus_peers.cc:597] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:03:25.258600  1718 consensus_peers.cc:597] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:03:25.264057  2299 consensus_peers.cc:597] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:03:25.285264  1718 consensus_peers.cc:597] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:03:25.296030  1718 consensus_peers.cc:597] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:03:25.307512  2299 consensus_peers.cc:597] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:03:25.315856  2298 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111) [suppressed 726 similar messages]
W20260709 13:03:25.357873  1718 consensus_peers.cc:597] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:03:25.392915  1718 consensus_peers.cc:597] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:03:25.401033  2299 consensus_peers.cc:597] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:03:25.415995   431 thread.cc:527] Waited for 4000ms trying to join with diag-logger (tid 1701)
W20260709 13:03:25.417171  2298 consensus_peers.cc:597] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 13:03:25.493429  1718 consensus_peers.cc:597] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:03:25.515555  2298 consensus_peers.cc:597] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 13:03:25.523038  2299 consensus_peers.cc:597] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:03:25.550884  1717 consensus_peers.cc:597] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 13:03:25.557439  1718 consensus_peers.cc:597] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:03:25.645340  1717 consensus_peers.cc:597] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20260709 13:03:25.656625 10723 raft_consensus.cc:493] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:03:25.657438 10723 raft_consensus.cc:515] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:03:25.670645 10723 leader_election.cc:290] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 13:03:25.730896  1718 consensus_peers.cc:597] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:03:25.731498  1718 leader_election.cc:336] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111)
W20260709 13:03:25.746201  1717 leader_election.cc:336] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111)
I20260709 13:03:25.747011  1717 leader_election.cc:304] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 13:03:25.751632  1718 consensus_peers.cc:597] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:03:25.787462  1718 consensus_peers.cc:597] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 13:03:25.789497 10725 raft_consensus.cc:2749] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
W20260709 13:03:25.790920  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 44782.3us
I20260709 13:03:25.841899 10722 raft_consensus.cc:493] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:03:25.842638 10722 raft_consensus.cc:515] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:03:25.845850 10722 leader_election.cc:290] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 13:03:25.858676  1718 leader_election.cc:336] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111)
I20260709 13:03:25.869488 10720 raft_consensus.cc:493] T 32dc176efd53437ab7735ea8c6ac4a0d P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:03:25.870571 10720 raft_consensus.cc:515] T 32dc176efd53437ab7735ea8c6ac4a0d P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:03:25.882715  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:03:25.883644  1750 raft_consensus.cc:2468] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 22.
I20260709 13:03:25.889729 10720 leader_election.cc:290] T 32dc176efd53437ab7735ea8c6ac4a0d P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:03:25.896795  2298 leader_election.cc:336] T 32dc176efd53437ab7735ea8c6ac4a0d P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111)
I20260709 13:03:25.900524 10720 raft_consensus.cc:493] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:03:25.901252 10720 raft_consensus.cc:515] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:03:25.932340  1717 consensus_peers.cc:597] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260709 13:03:25.933787  1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46445, user_credentials={real_user=slave}} blocked reactor thread for 44688.3us
I20260709 13:03:25.950548 10720 leader_election.cc:290] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:03:25.951915 10725 raft_consensus.cc:493] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:03:25.953299 10725 raft_consensus.cc:515] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:03:25.957211 10725 leader_election.cc:290] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 13:03:25.976255  1717 leader_election.cc:336] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111)
I20260709 13:03:25.979986  1717 leader_election.cc:304] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:03:25.981331 10725 raft_consensus.cc:2749] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260709 13:03:25.979213  1718 leader_election.cc:336] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111)
W20260709 13:03:25.986539  2299 leader_election.cc:336] T 32dc176efd53437ab7735ea8c6ac4a0d P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111)
I20260709 13:03:25.994029  2299 leader_election.cc:304] T 32dc176efd53437ab7735ea8c6ac4a0d P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 13:03:25.991495  2298 leader_election.cc:336] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111)
W20260709 13:03:26.002103  1717 leader_election.cc:336] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111)
I20260709 13:03:26.002760  1717 leader_election.cc:304] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:03:26.004534 10720 raft_consensus.cc:2749] T 32dc176efd53437ab7735ea8c6ac4a0d P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:03:26.003985 10725 raft_consensus.cc:2749] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:03:26.015118 10722 raft_consensus.cc:493] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:03:26.015929 10722 raft_consensus.cc:515] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:03:26.018815 10722 leader_election.cc:290] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 13:03:26.025449  1718 leader_election.cc:336] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 18 pre-election: RPC error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111)
W20260709 13:03:26.036347  1717 leader_election.cc:336] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 18 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111)
I20260709 13:03:26.041109  1717 leader_election.cc:304] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:03:26.044346 10723 raft_consensus.cc:2749] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20260709 13:03:26.047650  2299 leader_election.cc:336] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111)
I20260709 13:03:26.048226  2299 leader_election.cc:304] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:03:26.053439 10720 raft_consensus.cc:2749] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
W20260709 13:03:26.072713  1717 consensus_peers.cc:597] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260709 13:03:26.106072  1717 consensus_peers.cc:597] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20260709 13:03:26.124106 10720 raft_consensus.cc:493] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:03:26.133778 10720 raft_consensus.cc:515] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:03:26.139250 10720 leader_election.cc:290] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 13:03:26.132851  2298 consensus_peers.cc:597] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 13:03:26.163432  2298 leader_election.cc:336] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111)
W20260709 13:03:26.183276  2298 consensus_peers.cc:597] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 13:03:26.184890  1717 consensus_peers.cc:597] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 13:03:26.192085  2299 leader_election.cc:336] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111)
I20260709 13:03:26.192806  2299 leader_election.cc:304] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:03:26.193842 10720 raft_consensus.cc:2749] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
W20260709 13:03:26.261384  2298 consensus_peers.cc:597] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260709 13:03:26.309933  1717 consensus_peers.cc:597] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260709 13:03:26.313527  1717 consensus_peers.cc:597] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 13:03:26.415473  1717 consensus_peers.cc:597] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 13:03:26.419052   431 thread.cc:527] Waited for 5000ms trying to join with diag-logger (tid 1701)
I20260709 13:03:26.423949 10685 raft_consensus.cc:493] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:03:26.424892 10685 raft_consensus.cc:515] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:03:26.464687 10685 leader_election.cc:290] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 13:03:26.486624  2298 leader_election.cc:336] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111)
W20260709 13:03:26.493898  2299 leader_election.cc:336] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111)
I20260709 13:03:26.499723  2299 leader_election.cc:304] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 13:03:26.492535  2298 consensus_peers.cc:597] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20260709 13:03:26.514549 10685 raft_consensus.cc:2749] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:03:26.550323 10728 raft_consensus.cc:493] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:03:26.551385 10728 raft_consensus.cc:515] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:03:26.553997 10728 leader_election.cc:290] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 13:03:26.576584  1718 leader_election.cc:336] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111)
W20260709 13:03:26.592190  1717 leader_election.cc:336] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111)
I20260709 13:03:26.592832  1717 leader_election.cc:304] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:03:26.596344 10723 raft_consensus.cc:2749] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
W20260709 13:03:26.644126  1717 consensus_peers.cc:597] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 13:03:26.706095  2298 consensus_peers.cc:597] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20260709 13:03:26.718678  2298 consensus_peers.cc:597] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260709 13:03:26.780351  2298 consensus_peers.cc:597] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260709 13:03:26.873054  1718 consensus_peers.cc:597] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:03:26.898304  1717 consensus_peers.cc:597] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 13:03:26.910560  1717 consensus_peers.cc:597] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 13:03:27.001770  2298 consensus_peers.cc:597] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20260709 13:03:27.050472  1718 consensus_peers.cc:597] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:03:27.056294  1717 consensus_peers.cc:597] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:03:27.083652  1717 consensus_peers.cc:597] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260709 13:03:27.084380  1717 consensus_peers.cc:597] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 13:03:27.173563  1718 consensus_peers.cc:597] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:03:27.235467  2299 consensus_peers.cc:597] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:03:27.249328  2298 consensus_peers.cc:597] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260709 13:03:27.260097  1717 consensus_peers.cc:597] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260709 13:03:27.280148  2299 consensus_peers.cc:597] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:03:27.284196  2299 consensus_peers.cc:597] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 13:03:27.317296 10723 raft_consensus.cc:493] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:03:27.318007 10723 raft_consensus.cc:515] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:03:27.319833  1717 consensus_peers.cc:597] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260709 13:03:27.330868  1718 leader_election.cc:336] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111)
W20260709 13:03:27.332913  1717 leader_election.cc:336] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111)
I20260709 13:03:27.333465  1717 leader_election.cc:304] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:03:27.334888 10728 raft_consensus.cc:2749] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 13:03:27.334064 10723 leader_election.cc:290] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 13:03:27.339058  1717 consensus_peers.cc:597] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 13:03:27.351154  2298 consensus_peers.cc:597] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20260709 13:03:27.361032  1718 consensus_peers.cc:597] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:03:27.367763  2299 consensus_peers.cc:597] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:03:27.368708  2298 consensus_peers.cc:597] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20260709 13:03:27.418612 10728 raft_consensus.cc:493] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
W20260709 13:03:27.419591   431 thread.cc:527] Waited for 6000ms trying to join with diag-logger (tid 1701)
I20260709 13:03:27.419402 10728 raft_consensus.cc:515] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:03:27.425621  1717 leader_election.cc:336] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111)
I20260709 13:03:27.422324 10728 leader_election.cc:290] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 13:03:27.464352  1718 leader_election.cc:336] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111)
I20260709 13:03:27.465049  1718 leader_election.cc:304] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:03:27.466554 10731 raft_consensus.cc:2749] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
W20260709 13:03:27.469128  1718 consensus_peers.cc:597] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:03:27.496456  1718 consensus_peers.cc:597] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:03:27.506493  2298 consensus_peers.cc:597] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20260709 13:03:27.515568  2299 consensus_peers.cc:597] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:03:27.530891  1718 consensus_peers.cc:597] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:03:27.531538  1718 consensus_peers.cc:597] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:03:27.542732  1718 consensus_peers.cc:597] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:03:27.550103  2299 consensus_peers.cc:597] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:03:27.600498  2299 consensus_peers.cc:597] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:03:27.614634  2299 consensus_peers.cc:597] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:03:27.622157  2298 consensus_peers.cc:597] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260709 13:03:27.648893  1718 consensus_peers.cc:597] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:03:27.708738  2299 consensus_peers.cc:597] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:03:27.749143  2299 consensus_peers.cc:597] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:03:27.760807  2299 consensus_peers.cc:597] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:03:27.770282  2299 consensus_peers.cc:597] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:03:27.793497  1718 consensus_peers.cc:597] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 13:03:27.799175 10731 raft_consensus.cc:493] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:03:27.799939 10731 raft_consensus.cc:515] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:03:27.811100  1717 leader_election.cc:336] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111)
W20260709 13:03:27.812755  1718 leader_election.cc:336] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111)
I20260709 13:03:27.813230  1718 leader_election.cc:304] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:03:27.814036 10732 raft_consensus.cc:2749] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:03:27.814255 10731 leader_election.cc:290] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 13:03:27.815824  2299 consensus_peers.cc:597] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:03:27.821875  1718 consensus_peers.cc:597] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:03:27.844485  2299 consensus_peers.cc:597] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:03:27.884667  1718 consensus_peers.cc:597] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:03:27.886081  1487 catalog_manager.cc:4755] TS 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 7ebbb6d7c6c4478e95972de2f78d3f80: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111)
W20260709 13:03:27.925978  1718 consensus_peers.cc:597] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:03:27.977736  1718 consensus_peers.cc:597] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:03:28.001273  2298 consensus_peers.cc:597] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 13:03:28.011853  1717 consensus_peers.cc:597] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260709 13:03:28.074000  2298 consensus_peers.cc:597] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 13:03:28.081279  1718 consensus_peers.cc:597] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:03:28.142679  1717 consensus_peers.cc:597] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20260709 13:03:28.149943 10685 raft_consensus.cc:493] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:03:28.159358 10685 raft_consensus.cc:515] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:03:28.162179 10685 leader_election.cc:290] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 13:03:28.171780  2299 leader_election.cc:336] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111)
W20260709 13:03:28.179570  1718 consensus_peers.cc:597] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:03:28.181712  2298 leader_election.cc:336] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111)
I20260709 13:03:28.185240  2298 leader_election.cc:304] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 13:03:28.193243  2299 consensus_peers.cc:597] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 13:03:28.205221 10735 raft_consensus.cc:2749] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 13:03:28.253544 10732 raft_consensus.cc:493] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:03:28.254215 10732 raft_consensus.cc:515] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:03:28.256953 10732 leader_election.cc:290] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 13:03:28.300458  1718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:46261, user_credentials={real_user=slave}} blocked reactor thread for 51413.2us
W20260709 13:03:28.306915  1718 consensus_peers.cc:597] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:03:28.313496  1718 consensus_peers.cc:597] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:03:28.318332  1717 leader_election.cc:336] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111)
W20260709 13:03:28.359676  1718 consensus_peers.cc:597] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 13:03:28.361260 10735 raft_consensus.cc:493] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:03:28.369645 10735 raft_consensus.cc:515] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:03:28.372668 10735 leader_election.cc:290] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
I20260709 13:03:28.417783 10735 raft_consensus.cc:493] T 32dc176efd53437ab7735ea8c6ac4a0d P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:03:28.418895 10735 raft_consensus.cc:515] T 32dc176efd53437ab7735ea8c6ac4a0d P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } attrs { replace: true } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:03:28.420118   431 thread.cc:527] Waited for 7000ms trying to join with diag-logger (tid 1701)
I20260709 13:03:28.423977  1750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32dc176efd53437ab7735ea8c6ac4a0d" candidate_uuid: "3b879799899a499c8d409ee9f1277cd1" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "af520b67945d443b901bbeca37cc5701" is_pre_election: true
I20260709 13:03:28.425030  1750 raft_consensus.cc:2468] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b879799899a499c8d409ee9f1277cd1 in term 22.
I20260709 13:03:28.426739 10735 leader_election.cc:290] T 32dc176efd53437ab7735ea8c6ac4a0d P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:03:28.432410  2298 leader_election.cc:336] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111)
W20260709 13:03:28.437057  1718 leader_election.cc:336] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111)
I20260709 13:03:28.451205  1718 leader_election.cc:304] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:03:28.461130 10730 raft_consensus.cc:2749] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
W20260709 13:03:28.464975  2298 consensus_peers.cc:597] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 13:03:28.490646  1717 consensus_peers.cc:597] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 13:03:28.491978  2299 leader_election.cc:336] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111)
I20260709 13:03:28.495057  2299 leader_election.cc:304] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
W20260709 13:03:28.490324  2298 leader_election.cc:336] T 32dc176efd53437ab7735ea8c6ac4a0d P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111)
I20260709 13:03:28.502723 10735 raft_consensus.cc:2749] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
W20260709 13:03:28.511704  1717 consensus_peers.cc:597] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260709 13:03:28.519678  2299 leader_election.cc:336] T 32dc176efd53437ab7735ea8c6ac4a0d P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111)
I20260709 13:03:28.551000  2299 leader_election.cc:304] T 32dc176efd53437ab7735ea8c6ac4a0d P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1, af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:03:28.561543 10735 raft_consensus.cc:2749] T 32dc176efd53437ab7735ea8c6ac4a0d P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
W20260709 13:03:28.686895  1717 consensus_peers.cc:597] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20260709 13:03:28.692224  1717 consensus_peers.cc:597] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 13:03:28.694324  2298 consensus_peers.cc:597] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 13:03:28.790103  1717 consensus_peers.cc:597] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20260709 13:03:28.795253 10735 raft_consensus.cc:493] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:03:28.795971 10735 raft_consensus.cc:515] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:03:28.798336 10735 leader_election.cc:290] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 13:03:28.804695  2299 leader_election.cc:336] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111)
W20260709 13:03:28.811053  2298 leader_election.cc:336] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111)
I20260709 13:03:28.811609  2298 leader_election.cc:304] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:03:28.812894 10735 raft_consensus.cc:2749] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
W20260709 13:03:28.895103  2298 consensus_peers.cc:597] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20260709 13:03:29.062832  1717 consensus_peers.cc:597] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20260709 13:03:29.076131 10741 raft_consensus.cc:493] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:03:29.076968 10741 raft_consensus.cc:515] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
I20260709 13:03:29.079530 10741 leader_election.cc:290] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 13:03:29.091459  1718 leader_election.cc:336] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111)
I20260709 13:03:29.092903 10737 raft_consensus.cc:493] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:03:29.099205 10737 raft_consensus.cc:515] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:03:29.102609 10737 leader_election.cc:290] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 13:03:29.109951  2298 leader_election.cc:336] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 18 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111)
W20260709 13:03:29.129042  1717 leader_election.cc:336] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111)
W20260709 13:03:29.132956  2299 leader_election.cc:336] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 18 pre-election: RPC error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111)
I20260709 13:03:29.137466  2299 leader_election.cc:304] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:03:29.135406  1717 leader_election.cc:304] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:03:29.138731 10741 raft_consensus.cc:2749] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260709 13:03:29.139990  1717 consensus_peers.cc:597] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20260709 13:03:29.140520 10737 raft_consensus.cc:2749] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20260709 13:03:29.154193  2298 consensus_peers.cc:597] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20260709 13:03:29.164637 10741 raft_consensus.cc:493] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:03:29.165326 10741 raft_consensus.cc:515] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } }
W20260709 13:03:29.175595  1717 leader_election.cc:336] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 18 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111)
W20260709 13:03:29.184376  1718 leader_election.cc:336] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 18 pre-election: RPC error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111)
I20260709 13:03:29.184864  1718 leader_election.cc:304] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af520b67945d443b901bbeca37cc5701; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:03:29.185631 10730 raft_consensus.cc:2749] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:03:29.188040 10741 leader_election.cc:290] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 13:03:29.191147  1717 consensus_peers.cc:597] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260709 13:03:29.203205  2298 consensus_peers.cc:597] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20260709 13:03:29.292665   431 tablet_server.cc:196] TabletServer@127.0.107.194:0 shutdown complete.
W20260709 13:03:29.301127  2298 consensus_peers.cc:597] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20260709 13:03:29.305557  1718 consensus_peers.cc:597] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:03:29.357566  2298 consensus_peers.cc:597] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20260709 13:03:29.388965  1717 consensus_peers.cc:597] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260709 13:03:29.405068  1718 consensus_peers.cc:597] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260709 13:03:29.507609   431 tablet_server.cc:179] TabletServer@127.0.107.195:0 shutting down...
W20260709 13:03:29.536803  2298 consensus_peers.cc:597] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20260709 13:03:29.544144  2301 consensus_peers.cc:597] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 -> Peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Couldn't send request to peer af520b67945d443b901bbeca37cc5701. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:03:29.552436  1717 consensus_peers.cc:597] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 13:03:29.553140  1717 consensus_peers.cc:597] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 13:03:29.553597  1717 consensus_peers.cc:597] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20260709 13:03:29.595898  1717 consensus_peers.cc:597] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 13:03:29.598784  2299 consensus_peers.cc:597] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:03:29.620208  2301 consensus_peers.cc:597] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 -> Peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Couldn't send request to peer af520b67945d443b901bbeca37cc5701. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:03:29.642110  1718 consensus_peers.cc:597] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 13:03:29.643925   431 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 13:03:29.651153   431 tablet_replica.cc:333] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701: stopping tablet replica
I20260709 13:03:29.654765   431 raft_consensus.cc:2243] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [term 19 LEADER]: Raft consensus shutting down.
I20260709 13:03:29.656530   431 raft_consensus.cc:2272] T ef5897d71124435db68c88060b095421 P af520b67945d443b901bbeca37cc5701 [term 19 FOLLOWER]: Raft consensus is shut down!
I20260709 13:03:29.659999   431 tablet_replica.cc:333] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701: stopping tablet replica
I20260709 13:03:29.660908   431 raft_consensus.cc:2243] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 23 LEADER]: Raft consensus shutting down.
I20260709 13:03:29.662919   431 raft_consensus.cc:2272] T a3accbb733bb4cafa69428319664f239 P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Raft consensus is shut down!
I20260709 13:03:29.665964   431 tablet_replica.cc:333] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701: stopping tablet replica
I20260709 13:03:29.666877   431 raft_consensus.cc:2243] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 18 LEADER]: Raft consensus shutting down.
I20260709 13:03:29.668295   431 raft_consensus.cc:2272] T a8c04bcdff52470cb06667e46356e653 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Raft consensus is shut down!
I20260709 13:03:29.671139   431 tablet_replica.cc:333] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701: stopping tablet replica
I20260709 13:03:29.672021   431 raft_consensus.cc:2243] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Raft consensus shutting down.
I20260709 13:03:29.673621   431 raft_consensus.cc:2272] T 41166f283aea41b0b3f5be60e8234443 P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Raft consensus is shut down!
W20260709 13:03:29.675498  2301 consensus_peers.cc:597] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 -> Peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Couldn't send request to peer af520b67945d443b901bbeca37cc5701. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:03:29.677068   431 tablet_replica.cc:333] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701: stopping tablet replica
I20260709 13:03:29.678092   431 raft_consensus.cc:2243] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Raft consensus shutting down.
I20260709 13:03:29.679333   431 raft_consensus.cc:2272] T 1d9ac76dc2c445edb5a25cf9087998bb P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Raft consensus is shut down!
I20260709 13:03:29.682330   431 tablet_replica.cc:333] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701: stopping tablet replica
I20260709 13:03:29.683171   431 raft_consensus.cc:2243] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Raft consensus shutting down.
I20260709 13:03:29.684284   431 raft_consensus.cc:2272] T 6f6cc48def184bcc907c18c98277187f P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Raft consensus is shut down!
I20260709 13:03:29.687033   431 tablet_replica.cc:333] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701: stopping tablet replica
I20260709 13:03:29.687939   431 raft_consensus.cc:2243] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 23 LEADER]: Raft consensus shutting down.
I20260709 13:03:29.689805   431 raft_consensus.cc:2272] T 79ed08f2339a4b1ead1e70d78eaec2ec P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Raft consensus is shut down!
I20260709 13:03:29.692615   431 tablet_replica.cc:333] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701: stopping tablet replica
I20260709 13:03:29.693572   431 raft_consensus.cc:2243] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 23 LEADER]: Raft consensus shutting down.
I20260709 13:03:29.696319   431 raft_consensus.cc:2272] T 1f1329a6c1e442bcba5945a2d1e65e85 P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Raft consensus is shut down!
I20260709 13:03:29.699636   431 tablet_replica.cc:333] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701: stopping tablet replica
I20260709 13:03:29.700454   431 raft_consensus.cc:2243] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 27 LEADER]: Raft consensus shutting down.
I20260709 13:03:29.701882   431 raft_consensus.cc:2272] T 344b4152823f4d618345b14ffd579400 P af520b67945d443b901bbeca37cc5701 [term 27 FOLLOWER]: Raft consensus is shut down!
I20260709 13:03:29.705018   431 tablet_replica.cc:333] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701: stopping tablet replica
I20260709 13:03:29.705856   431 raft_consensus.cc:2243] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Raft consensus shutting down.
I20260709 13:03:29.706558   431 raft_consensus.cc:2272] T 14588fa8cfd6467e989746e326a0ed31 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Raft consensus is shut down!
I20260709 13:03:29.709590   431 tablet_replica.cc:333] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701: stopping tablet replica
I20260709 13:03:29.710552   431 raft_consensus.cc:2243] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Raft consensus shutting down.
I20260709 13:03:29.711318   431 raft_consensus.cc:2272] T a93d4b974fee4d02afbaa26d69bfeb07 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Raft consensus is shut down!
I20260709 13:03:29.714246   431 tablet_replica.cc:333] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701: stopping tablet replica
I20260709 13:03:29.715075   431 raft_consensus.cc:2243] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 20 LEADER]: Raft consensus shutting down.
I20260709 13:03:29.716598   431 raft_consensus.cc:2272] T 1809d674329a492dbea0b60a4703fe51 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Raft consensus is shut down!
I20260709 13:03:29.719478   431 tablet_replica.cc:333] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701: stopping tablet replica
I20260709 13:03:29.720427   431 raft_consensus.cc:2243] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Raft consensus shutting down.
I20260709 13:03:29.721292   431 raft_consensus.cc:2272] T 03e37c14b11849208b5c9c51f32e1702 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Raft consensus is shut down!
W20260709 13:03:29.721705  2301 consensus_peers.cc:597] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 -> Peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Couldn't send request to peer af520b67945d443b901bbeca37cc5701. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:03:29.724085   431 tablet_replica.cc:333] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701: stopping tablet replica
I20260709 13:03:29.725011   431 raft_consensus.cc:2243] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 26 LEADER]: Raft consensus shutting down.
I20260709 13:03:29.726722   431 raft_consensus.cc:2272] T 72f3cdc0abae4f74aad5e4a15cb02e1b P af520b67945d443b901bbeca37cc5701 [term 26 FOLLOWER]: Raft consensus is shut down!
I20260709 13:03:29.730703   431 tablet_replica.cc:333] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701: stopping tablet replica
I20260709 13:03:29.731496   431 raft_consensus.cc:2243] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 22 LEADER]: Raft consensus shutting down.
W20260709 13:03:29.732836  2301 consensus_peers.cc:597] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 -> Peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Couldn't send request to peer af520b67945d443b901bbeca37cc5701. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:03:29.732939   431 raft_consensus.cc:2272] T 453513ba44f34adb8b436305ee51fe3d P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Raft consensus is shut down!
W20260709 13:03:29.736286  2298 consensus_peers.cc:597] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20260709 13:03:29.740222   431 tablet_replica.cc:333] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701: stopping tablet replica
I20260709 13:03:29.741206   431 raft_consensus.cc:2243] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Raft consensus shutting down.
I20260709 13:03:29.741982   431 raft_consensus.cc:2272] T c6ab3300be1e4a59b64b14d50da193ca P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Raft consensus is shut down!
W20260709 13:03:29.745049  2301 consensus_peers.cc:597] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 -> Peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Couldn't send request to peer af520b67945d443b901bbeca37cc5701. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:03:29.745967   431 tablet_replica.cc:333] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701: stopping tablet replica
I20260709 13:03:29.747478   431 raft_consensus.cc:2243] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 25 FOLLOWER]: Raft consensus shutting down.
I20260709 13:03:29.748291   431 raft_consensus.cc:2272] T 03daa4b346f54f1da820d1ff6cd1ff86 P af520b67945d443b901bbeca37cc5701 [term 25 FOLLOWER]: Raft consensus is shut down!
W20260709 13:03:29.750293  2301 consensus_peers.cc:597] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 -> Peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Couldn't send request to peer af520b67945d443b901bbeca37cc5701. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:03:29.752344   431 tablet_replica.cc:333] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701: stopping tablet replica
I20260709 13:03:29.753182   431 raft_consensus.cc:2243] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 23 LEADER]: Raft consensus shutting down.
I20260709 13:03:29.755771   431 raft_consensus.cc:2272] T 127630d60f944b899508c92ee18fdddf P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Raft consensus is shut down!
I20260709 13:03:29.758570   431 tablet_replica.cc:333] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701: stopping tablet replica
I20260709 13:03:29.759423   431 raft_consensus.cc:2243] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Raft consensus shutting down.
I20260709 13:03:29.760316   431 raft_consensus.cc:2272] T 5108457796a949b7b753ab22b575b7a0 P af520b67945d443b901bbeca37cc5701 [term 17 FOLLOWER]: Raft consensus is shut down!
I20260709 13:03:29.764273   431 tablet_replica.cc:333] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701: stopping tablet replica
I20260709 13:03:29.765196   431 raft_consensus.cc:2243] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 25 FOLLOWER]: Raft consensus shutting down.
I20260709 13:03:29.765957   431 raft_consensus.cc:2272] T 78cabb03415041ca9c1a5a400e506131 P af520b67945d443b901bbeca37cc5701 [term 25 FOLLOWER]: Raft consensus is shut down!
I20260709 13:03:29.776925   431 tablet_replica.cc:333] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701: stopping tablet replica
I20260709 13:03:29.778717   431 raft_consensus.cc:2243] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 25 FOLLOWER]: Raft consensus shutting down.
I20260709 13:03:29.779449   431 raft_consensus.cc:2272] T b9bb5648969b4299a776843d72043448 P af520b67945d443b901bbeca37cc5701 [term 25 FOLLOWER]: Raft consensus is shut down!
I20260709 13:03:29.782276   431 tablet_replica.cc:333] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701: stopping tablet replica
W20260709 13:03:29.785943  2299 consensus_peers.cc:597] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:03:29.786486  2301 consensus_peers.cc:597] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 -> Peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Couldn't send request to peer af520b67945d443b901bbeca37cc5701. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:03:29.790218   431 raft_consensus.cc:2243] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 25 LEADER]: Raft consensus shutting down.
I20260709 13:03:29.792775   431 raft_consensus.cc:2272] T 983621a6fc5b4c9ea0159a648dfe7a36 P af520b67945d443b901bbeca37cc5701 [term 25 FOLLOWER]: Raft consensus is shut down!
W20260709 13:03:29.795773  1717 consensus_peers.cc:597] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20260709 13:03:29.796799   431 tablet_replica.cc:333] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701: stopping tablet replica
I20260709 13:03:29.797953   431 raft_consensus.cc:2243] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Raft consensus shutting down.
I20260709 13:03:29.798717   431 raft_consensus.cc:2272] T 58d4fdd02c6d4667aa718fb146a76505 P af520b67945d443b901bbeca37cc5701 [term 18 FOLLOWER]: Raft consensus is shut down!
I20260709 13:03:29.801673   431 tablet_replica.cc:333] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701: stopping tablet replica
I20260709 13:03:29.802476   431 raft_consensus.cc:2243] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 25 LEADER]: Raft consensus shutting down.
I20260709 13:03:29.804329   431 raft_consensus.cc:2272] T bb255e8a61e7458babbbdf0183773a81 P af520b67945d443b901bbeca37cc5701 [term 25 FOLLOWER]: Raft consensus is shut down!
I20260709 13:03:29.807126   431 tablet_replica.cc:333] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701: stopping tablet replica
I20260709 13:03:29.807973   431 raft_consensus.cc:2243] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Raft consensus shutting down.
I20260709 13:03:29.808801   431 raft_consensus.cc:2272] T e527450b1d6c4cb79e9f6d3c87d4ad23 P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Raft consensus is shut down!
I20260709 13:03:29.811944   431 tablet_replica.cc:333] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701: stopping tablet replica
I20260709 13:03:29.812999   431 raft_consensus.cc:2243] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Raft consensus shutting down.
I20260709 13:03:29.813798   431 raft_consensus.cc:2272] T 87b86d46388a41cd90374c203cb27914 P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Raft consensus is shut down!
W20260709 13:03:29.814956  2298 consensus_peers.cc:597] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20260709 13:03:29.816581   431 tablet_replica.cc:333] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701: stopping tablet replica
I20260709 13:03:29.817520   431 raft_consensus.cc:2243] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 28 LEADER]: Raft consensus shutting down.
I20260709 13:03:29.818995   431 raft_consensus.cc:2272] T 1c2f5fa72dee4ec58979ace307d2acc3 P af520b67945d443b901bbeca37cc5701 [term 28 FOLLOWER]: Raft consensus is shut down!
I20260709 13:03:29.821884   431 tablet_replica.cc:333] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701: stopping tablet replica
I20260709 13:03:29.822744   431 raft_consensus.cc:2243] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 25 FOLLOWER]: Raft consensus shutting down.
I20260709 13:03:29.823617   431 raft_consensus.cc:2272] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P af520b67945d443b901bbeca37cc5701 [term 25 FOLLOWER]: Raft consensus is shut down!
I20260709 13:03:29.826676   431 tablet_replica.cc:333] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701: stopping tablet replica
I20260709 13:03:29.827515   431 raft_consensus.cc:2243] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 24 LEADER]: Raft consensus shutting down.
I20260709 13:03:29.828895   431 raft_consensus.cc:2272] T e947690dcb3849cb9f9731123d363b91 P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Raft consensus is shut down!
I20260709 13:03:29.831882   431 tablet_replica.cc:333] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701: stopping tablet replica
I20260709 13:03:29.832800   431 raft_consensus.cc:2243] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 21 LEADER]: Raft consensus shutting down.
I20260709 13:03:29.834632   431 raft_consensus.cc:2272] T 13031f9e955e4f5fb289c79ee2c46108 P af520b67945d443b901bbeca37cc5701 [term 21 FOLLOWER]: Raft consensus is shut down!
W20260709 13:03:29.835399  2298 consensus_peers.cc:597] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20260709 13:03:29.838079   431 tablet_replica.cc:333] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701: stopping tablet replica
I20260709 13:03:29.839126   431 raft_consensus.cc:2243] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 26 FOLLOWER]: Raft consensus shutting down.
I20260709 13:03:29.839942   431 raft_consensus.cc:2272] T e4f4b860aab3464383ce8f9d990f4a4a P af520b67945d443b901bbeca37cc5701 [term 26 FOLLOWER]: Raft consensus is shut down!
I20260709 13:03:29.843173   431 tablet_replica.cc:333] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701: stopping tablet replica
I20260709 13:03:29.843986   431 raft_consensus.cc:2243] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 20 LEADER]: Raft consensus shutting down.
I20260709 13:03:29.845403   431 raft_consensus.cc:2272] T fe7e5e9266614676a9c05c398cf7ba9e P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Raft consensus is shut down!
I20260709 13:03:29.848443   431 tablet_replica.cc:333] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701: stopping tablet replica
I20260709 13:03:29.849236   431 raft_consensus.cc:2243] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 23 LEADER]: Raft consensus shutting down.
I20260709 13:03:29.850759   431 raft_consensus.cc:2272] T 86abe5695d5643f888190301b88dacef P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Raft consensus is shut down!
I20260709 13:03:29.853459   431 tablet_replica.cc:333] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701: stopping tablet replica
I20260709 13:03:29.854321   431 raft_consensus.cc:2243] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 23 LEADER]: Raft consensus shutting down.
I20260709 13:03:29.856031   431 raft_consensus.cc:2272] T 77d66b92688a42e3ad231959a644db56 P af520b67945d443b901bbeca37cc5701 [term 23 FOLLOWER]: Raft consensus is shut down!
I20260709 13:03:29.859197   431 tablet_replica.cc:333] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701: stopping tablet replica
W20260709 13:03:29.860023  2301 consensus_peers.cc:597] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 -> Peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Couldn't send request to peer af520b67945d443b901bbeca37cc5701. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:03:29.860075   431 raft_consensus.cc:2243] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 29 LEADER]: Raft consensus shutting down.
I20260709 13:03:29.861763   431 raft_consensus.cc:2272] T dce8db7565da4e13a82ffbb9c39fde78 P af520b67945d443b901bbeca37cc5701 [term 29 FOLLOWER]: Raft consensus is shut down!
I20260709 13:03:29.865073   431 tablet_replica.cc:333] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701: stopping tablet replica
W20260709 13:03:29.867879  2299 consensus_peers.cc:597] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260709 13:03:29.879781   431 raft_consensus.cc:2243] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 25 LEADER]: Raft consensus shutting down.
I20260709 13:03:29.881786   431 raft_consensus.cc:2272] T 0615de9bff034ecb9ed6173433128ff5 P af520b67945d443b901bbeca37cc5701 [term 25 FOLLOWER]: Raft consensus is shut down!
I20260709 13:03:29.885036   431 tablet_replica.cc:333] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701: stopping tablet replica
W20260709 13:03:29.864815  2301 consensus_peers.cc:597] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 -> Peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Couldn't send request to peer af520b67945d443b901bbeca37cc5701. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:03:29.885883   431 raft_consensus.cc:2243] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Raft consensus shutting down.
I20260709 13:03:29.887343   431 raft_consensus.cc:2272] T cd44a2ebdc764e898f25ad5390f4bffd P af520b67945d443b901bbeca37cc5701 [term 24 FOLLOWER]: Raft consensus is shut down!
I20260709 13:03:29.892232   431 tablet_replica.cc:333] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701: stopping tablet replica
I20260709 13:03:29.892879   431 raft_consensus.cc:2243] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 20 LEADER]: Raft consensus shutting down.
I20260709 13:03:29.894016   431 raft_consensus.cc:2272] T a92588e9316e4b4aa1691e8669c839d3 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Raft consensus is shut down!
I20260709 13:03:29.896338   431 tablet_replica.cc:333] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701: stopping tablet replica
I20260709 13:03:29.896936   431 raft_consensus.cc:2243] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 26 FOLLOWER]: Raft consensus shutting down.
I20260709 13:03:29.897631   431 raft_consensus.cc:2272] T a40e423ed583480280352b261913276c P af520b67945d443b901bbeca37cc5701 [term 26 FOLLOWER]: Raft consensus is shut down!
I20260709 13:03:29.900053   431 tablet_replica.cc:333] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701: stopping tablet replica
I20260709 13:03:29.900632   431 raft_consensus.cc:2243] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 26 LEADER]: Raft consensus shutting down.
I20260709 13:03:29.902151   431 raft_consensus.cc:2272] T b917a1e9c9aa40a0afce5a12c343eacd P af520b67945d443b901bbeca37cc5701 [term 26 FOLLOWER]: Raft consensus is shut down!
I20260709 13:03:29.904687   431 tablet_replica.cc:333] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701: stopping tablet replica
I20260709 13:03:29.905284   431 raft_consensus.cc:2243] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 27 LEADER]: Raft consensus shutting down.
I20260709 13:03:29.907114   431 raft_consensus.cc:2272] T 7ebbb6d7c6c4478e95972de2f78d3f80 P af520b67945d443b901bbeca37cc5701 [term 27 FOLLOWER]: Raft consensus is shut down!
I20260709 13:03:29.909451   431 tablet_replica.cc:333] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701: stopping tablet replica
I20260709 13:03:29.910037   431 raft_consensus.cc:2243] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 25 FOLLOWER]: Raft consensus shutting down.
I20260709 13:03:29.911077   431 raft_consensus.cc:2272] T 65285571995c4ed5b593041598056a22 P af520b67945d443b901bbeca37cc5701 [term 25 FOLLOWER]: Raft consensus is shut down!
I20260709 13:03:29.914486   431 tablet_replica.cc:333] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701: stopping tablet replica
I20260709 13:03:29.915328   431 raft_consensus.cc:2243] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Raft consensus shutting down.
I20260709 13:03:29.916733   431 raft_consensus.cc:2272] T 32dc176efd53437ab7735ea8c6ac4a0d P af520b67945d443b901bbeca37cc5701 [term 22 FOLLOWER]: Raft consensus is shut down!
I20260709 13:03:29.919894   431 tablet_replica.cc:333] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701: stopping tablet replica
I20260709 13:03:29.920670   431 raft_consensus.cc:2243] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 27 FOLLOWER]: Raft consensus shutting down.
I20260709 13:03:29.921453   431 raft_consensus.cc:2272] T 513043a9b96c4f70adad53ac4b5d01dc P af520b67945d443b901bbeca37cc5701 [term 27 FOLLOWER]: Raft consensus is shut down!
I20260709 13:03:29.924297   431 tablet_replica.cc:333] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701: stopping tablet replica
I20260709 13:03:29.925074   431 raft_consensus.cc:2243] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 25 LEADER]: Raft consensus shutting down.
I20260709 13:03:29.927832   431 raft_consensus.cc:2272] T 73dc0a9696be41f78e22f1f77a6eee72 P af520b67945d443b901bbeca37cc5701 [term 25 FOLLOWER]: Raft consensus is shut down!
I20260709 13:03:29.930660   431 tablet_replica.cc:333] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701: stopping tablet replica
I20260709 13:03:29.931352   431 raft_consensus.cc:2243] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 29 LEADER]: Raft consensus shutting down.
I20260709 13:03:29.934090   431 raft_consensus.cc:2272] T 19ce8a0cd7c54eafbb0e61fc6e305b62 P af520b67945d443b901bbeca37cc5701 [term 29 FOLLOWER]: Raft consensus is shut down!
I20260709 13:03:29.937146   431 tablet_replica.cc:333] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701: stopping tablet replica
I20260709 13:03:29.937819   431 raft_consensus.cc:2243] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Raft consensus shutting down.
I20260709 13:03:29.938548   431 raft_consensus.cc:2272] T 63cf5ab930f24f239fde96a128656333 P af520b67945d443b901bbeca37cc5701 [term 20 FOLLOWER]: Raft consensus is shut down!
W20260709 13:03:29.955933  2301 consensus_peers.cc:597] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 -> Peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Couldn't send request to peer af520b67945d443b901bbeca37cc5701. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:03:29.963953  2301 consensus_peers.cc:597] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 -> Peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Couldn't send request to peer af520b67945d443b901bbeca37cc5701. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:03:29.972096  2299 consensus_peers.cc:597] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:03:29.978685  2299 consensus_peers.cc:597] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:03:30.027693  2301 consensus_peers.cc:597] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 -> Peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Couldn't send request to peer af520b67945d443b901bbeca37cc5701. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:03:30.058948  2298 consensus_peers.cc:597] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20260709 13:03:30.084619  2299 consensus_peers.cc:597] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:03:30.088375  2299 consensus_peers.cc:597] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:03:30.099732  2301 consensus_peers.cc:597] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 -> Peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Couldn't send request to peer af520b67945d443b901bbeca37cc5701. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.195:40693: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 13:03:30.143129  2299 consensus_peers.cc:597] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:03:30.179071  2299 consensus_peers.cc:597] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:03:30.253044  2299 consensus_peers.cc:597] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:03:30.263124  2298 consensus_peers.cc:597] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 13:03:30.266527  2299 consensus_peers.cc:597] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:03:30.302371  2299 consensus_peers.cc:597] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:03:30.350162  2298 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111) [suppressed 697 similar messages]
W20260709 13:03:30.390321  2298 consensus_peers.cc:597] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20260709 13:03:30.462239  2298 consensus_peers.cc:597] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 13:03:30.485915  2299 consensus_peers.cc:597] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:03:30.512035  2299 consensus_peers.cc:597] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:03:30.622478  2299 consensus_peers.cc:597] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 -> Peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Couldn't send request to peer 816fadcea8fc4893882f5f9b9f22a03d. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260709 13:03:30.801296 10743 raft_consensus.cc:493] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:03:30.802119 10743 raft_consensus.cc:515] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:03:30.804570 10743 leader_election.cc:290] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:03:30.809661  2301 leader_election.cc:336] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Network error: Client connection negotiation failed: client connection to 127.0.107.195:40693: connect: Connection refused (error 111)
I20260709 13:03:30.851408 10743 raft_consensus.cc:493] T a92588e9316e4b4aa1691e8669c839d3 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:03:30.852360 10743 raft_consensus.cc:515] T a92588e9316e4b4aa1691e8669c839d3 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:03:30.855343 10743 leader_election.cc:290] T a92588e9316e4b4aa1691e8669c839d3 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:03:30.866462 10744 raft_consensus.cc:493] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:03:30.870734 10685 raft_consensus.cc:493] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:03:30.886762 10685 raft_consensus.cc:515] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:03:30.886780 10744 raft_consensus.cc:515] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:03:30.888837 10685 leader_election.cc:290] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:03:30.883834  2301 leader_election.cc:336] T a92588e9316e4b4aa1691e8669c839d3 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Network error: Client connection negotiation failed: client connection to 127.0.107.195:40693: connect: Connection refused (error 111)
I20260709 13:03:30.889873 10744 leader_election.cc:290] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:03:30.898399  2299 leader_election.cc:336] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111)
I20260709 13:03:30.899101  2299 leader_election.cc:304] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
W20260709 13:03:30.898370  2301 leader_election.cc:336] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Network error: Client connection negotiation failed: client connection to 127.0.107.195:40693: connect: Connection refused (error 111)
W20260709 13:03:30.900434  2299 leader_election.cc:336] T a92588e9316e4b4aa1691e8669c839d3 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111)
I20260709 13:03:30.901379  2299 leader_election.cc:304] T a92588e9316e4b4aa1691e8669c839d3 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
W20260709 13:03:30.901831  2301 leader_election.cc:336] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Network error: Client connection negotiation failed: client connection to 127.0.107.195:40693: connect: Connection refused (error 111)
I20260709 13:03:30.900933 10743 raft_consensus.cc:2749] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 13:03:30.902666 10685 raft_consensus.cc:2749] T a92588e9316e4b4aa1691e8669c839d3 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
W20260709 13:03:30.908357  2299 leader_election.cc:336] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111)
I20260709 13:03:30.910761  2299 leader_election.cc:304] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:03:30.911978 10685 raft_consensus.cc:2749] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 13:03:30.912132 10743 raft_consensus.cc:493] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:03:30.914278 10743 raft_consensus.cc:515] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:03:30.916918 10743 leader_election.cc:290] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:03:30.913798  2298 leader_election.cc:336] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111)
I20260709 13:03:30.921459  2298 leader_election.cc:304] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:03:30.922240 10685 raft_consensus.cc:2749] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
W20260709 13:03:30.922068  2301 leader_election.cc:336] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Network error: Client connection negotiation failed: client connection to 127.0.107.195:40693: connect: Connection refused (error 111)
W20260709 13:03:30.925967  2299 leader_election.cc:336] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111)
I20260709 13:03:30.926425  2299 leader_election.cc:304] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:03:30.927157 10743 raft_consensus.cc:2749] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 13:03:31.042511 10743 raft_consensus.cc:493] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:03:31.043191 10743 raft_consensus.cc:515] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:03:31.045675 10743 leader_election.cc:290] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:03:31.050148  2299 leader_election.cc:336] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111)
W20260709 13:03:31.050657  2301 leader_election.cc:336] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Network error: Client connection negotiation failed: client connection to 127.0.107.195:40693: connect: Connection refused (error 111)
I20260709 13:03:31.051143 10743 raft_consensus.cc:493] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:03:31.052423 10685 raft_consensus.cc:493] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:03:31.051754 10743 raft_consensus.cc:515] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:03:31.053103 10685 raft_consensus.cc:515] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:03:31.054965 10743 leader_election.cc:290] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:03:31.055202 10685 leader_election.cc:290] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:03:31.051146  2301 leader_election.cc:304] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:03:31.056579 10685 raft_consensus.cc:2749] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
W20260709 13:03:31.060657  2298 leader_election.cc:336] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111)
I20260709 13:03:31.061509 10685 raft_consensus.cc:493] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:03:31.062041 10685 raft_consensus.cc:515] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
W20260709 13:03:31.062803  2299 leader_election.cc:336] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111)
I20260709 13:03:31.063714 10743 raft_consensus.cc:493] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:03:31.064162 10685 leader_election.cc:290] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
I20260709 13:03:31.064291 10743 raft_consensus.cc:515] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:03:31.067102 10743 leader_election.cc:290] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:03:31.071511  2301 leader_election.cc:336] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Network error: Client connection negotiation failed: client connection to 127.0.107.195:40693: connect: Connection refused (error 111)
W20260709 13:03:31.071525  2298 leader_election.cc:336] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111)
I20260709 13:03:31.072171  2301 leader_election.cc:304] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
W20260709 13:03:31.072391  2298 leader_election.cc:336] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111)
I20260709 13:03:31.073124 10743 raft_consensus.cc:2749] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
W20260709 13:03:31.073167  2301 leader_election.cc:336] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Network error: Client connection negotiation failed: client connection to 127.0.107.195:40693: connect: Connection refused (error 111)
I20260709 13:03:31.074177  2301 leader_election.cc:304] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:03:31.074905 10685 raft_consensus.cc:2749] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 13:03:31.079869 10743 raft_consensus.cc:493] T 127630d60f944b899508c92ee18fdddf P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
W20260709 13:03:31.080049  2301 leader_election.cc:336] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Network error: Client connection negotiation failed: client connection to 127.0.107.195:40693: connect: Connection refused (error 111)
I20260709 13:03:31.080538  2301 leader_election.cc:304] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:03:31.080475 10743 raft_consensus.cc:515] T 127630d60f944b899508c92ee18fdddf P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } attrs { replace: true } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:03:31.081375 10685 raft_consensus.cc:2749] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
W20260709 13:03:31.081977  2301 leader_election.cc:336] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Network error: Client connection negotiation failed: client connection to 127.0.107.195:40693: connect: Connection refused (error 111)
I20260709 13:03:31.082417  2301 leader_election.cc:304] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:03:31.083184 10685 raft_consensus.cc:2749] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
W20260709 13:03:31.090864  2299 leader_election.cc:336] T 127630d60f944b899508c92ee18fdddf P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111)
W20260709 13:03:31.091897  2298 leader_election.cc:336] T 127630d60f944b899508c92ee18fdddf P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111)
I20260709 13:03:31.092374  2298 leader_election.cc:304] T 127630d60f944b899508c92ee18fdddf P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:03:31.093369 10685 raft_consensus.cc:2749] T 127630d60f944b899508c92ee18fdddf P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 13:03:31.094189 10743 leader_election.cc:290] T 127630d60f944b899508c92ee18fdddf P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:03:31.098867  2301 leader_election.cc:336] T 127630d60f944b899508c92ee18fdddf P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Network error: Client connection negotiation failed: client connection to 127.0.107.195:40693: connect: Connection refused (error 111)
W20260709 13:03:31.112391  2298 consensus_peers.cc:597] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260709 13:03:31.203615  2298 consensus_peers.cc:597] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20260709 13:03:31.238533   431 tablet_server.cc:196] TabletServer@127.0.107.195:0 shutdown complete.
I20260709 13:03:31.262274 10743 raft_consensus.cc:493] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 3b879799899a499c8d409ee9f1277cd1 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:03:31.263113 10743 raft_consensus.cc:515] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 3b879799899a499c8d409ee9f1277cd1 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:03:31.265893 10743 leader_election.cc:290] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:03:31.274298  2299 leader_election.cc:336] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111)
W20260709 13:03:31.275017  2301 leader_election.cc:336] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Network error: Client connection negotiation failed: client connection to 127.0.107.195:40693: connect: Connection refused (error 111)
I20260709 13:03:31.275667  2301 leader_election.cc:304] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, af520b67945d443b901bbeca37cc5701
I20260709 13:03:31.276474 10743 raft_consensus.cc:2749] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 3b879799899a499c8d409ee9f1277cd1 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 13:03:31.285570 10685 raft_consensus.cc:493] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader af520b67945d443b901bbeca37cc5701)
I20260709 13:03:31.286181 10685 raft_consensus.cc:515] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "af520b67945d443b901bbeca37cc5701" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 40693 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:03:31.288455 10685 leader_election.cc:290] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693)
W20260709 13:03:31.297151  2298 leader_election.cc:336] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111)
W20260709 13:03:31.297916  2301 leader_election.cc:336] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer af520b67945d443b901bbeca37cc5701 (127.0.107.195:40693): Network error: Client connection negotiation failed: client connection to 127.0.107.195:40693: connect: Connection refused (error 111)
I20260709 13:03:31.298480  2301 leader_election.cc:304] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 83595ad0fd5940c0a2bfc7c61acd969a, af520b67945d443b901bbeca37cc5701
I20260709 13:03:31.299295 10743 raft_consensus.cc:2749] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 13:03:31.378353   431 tablet_server.cc:179] TabletServer@127.0.107.196:0 shutting down...
I20260709 13:03:31.410990 10685 raft_consensus.cc:493] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 816fadcea8fc4893882f5f9b9f22a03d)
I20260709 13:03:31.411679 10685 raft_consensus.cc:515] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83595ad0fd5940c0a2bfc7c61acd969a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46445 } } peers { permanent_uuid: "816fadcea8fc4893882f5f9b9f22a03d" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 46261 } } peers { permanent_uuid: "3b879799899a499c8d409ee9f1277cd1" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 46625 } attrs { promote: false } }
I20260709 13:03:31.417593 10685 leader_election.cc:290] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445), 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261)
W20260709 13:03:31.418521  2298 leader_election.cc:336] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111)
W20260709 13:03:31.428202  2299 leader_election.cc:336] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 816fadcea8fc4893882f5f9b9f22a03d (127.0.107.194:46261): Network error: Client connection negotiation failed: client connection to 127.0.107.194:46261: connect: Connection refused (error 111)
I20260709 13:03:31.428737  2299 leader_election.cc:304] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b879799899a499c8d409ee9f1277cd1; no voters: 816fadcea8fc4893882f5f9b9f22a03d, 83595ad0fd5940c0a2bfc7c61acd969a
I20260709 13:03:31.429612 10685 raft_consensus.cc:2749] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 13:03:31.456408   431 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 13:03:31.457603   431 tablet_replica.cc:333] T a92588e9316e4b4aa1691e8669c839d3 P 3b879799899a499c8d409ee9f1277cd1: stopping tablet replica
I20260709 13:03:31.458886   431 raft_consensus.cc:2243] T a92588e9316e4b4aa1691e8669c839d3 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Raft consensus shutting down.
W20260709 13:03:31.459508  2298 consensus_peers.cc:597] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 -> Peer 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): Couldn't send request to peer 83595ad0fd5940c0a2bfc7c61acd969a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20260709 13:03:31.460014   431 raft_consensus.cc:2272] T a92588e9316e4b4aa1691e8669c839d3 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Raft consensus is shut down!
I20260709 13:03:31.464329   431 tablet_replica.cc:333] T 127630d60f944b899508c92ee18fdddf P 3b879799899a499c8d409ee9f1277cd1: stopping tablet replica
I20260709 13:03:31.465467   431 raft_consensus.cc:2243] T 127630d60f944b899508c92ee18fdddf P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Raft consensus shutting down.
I20260709 13:03:31.466370   431 raft_consensus.cc:2272] T 127630d60f944b899508c92ee18fdddf P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Raft consensus is shut down!
I20260709 13:03:31.469763   431 tablet_replica.cc:333] T 32dc176efd53437ab7735ea8c6ac4a0d P 3b879799899a499c8d409ee9f1277cd1: stopping tablet replica
I20260709 13:03:31.470705   431 raft_consensus.cc:2243] T 32dc176efd53437ab7735ea8c6ac4a0d P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Raft consensus shutting down.
I20260709 13:03:31.471585   431 raft_consensus.cc:2272] T 32dc176efd53437ab7735ea8c6ac4a0d P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Raft consensus is shut down!
I20260709 13:03:31.475073   431 tablet_replica.cc:333] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 3b879799899a499c8d409ee9f1277cd1: stopping tablet replica
I20260709 13:03:31.476064   431 raft_consensus.cc:2243] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 3b879799899a499c8d409ee9f1277cd1 [term 27 FOLLOWER]: Raft consensus shutting down.
I20260709 13:03:31.476718   431 raft_consensus.cc:2272] T 7ebbb6d7c6c4478e95972de2f78d3f80 P 3b879799899a499c8d409ee9f1277cd1 [term 27 FOLLOWER]: Raft consensus is shut down!
I20260709 13:03:31.480206   431 tablet_replica.cc:333] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1: stopping tablet replica
I20260709 13:03:31.481103   431 raft_consensus.cc:2243] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 24 LEADER]: Raft consensus shutting down.
I20260709 13:03:31.482467   431 raft_consensus.cc:2272] T 6f6cc48def184bcc907c18c98277187f P 3b879799899a499c8d409ee9f1277cd1 [term 24 FOLLOWER]: Raft consensus is shut down!
I20260709 13:03:31.485637   431 tablet_replica.cc:333] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1: stopping tablet replica
I20260709 13:03:31.486476   431 raft_consensus.cc:2243] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 26 FOLLOWER]: Raft consensus shutting down.
I20260709 13:03:31.487391   431 raft_consensus.cc:2272] T 72f3cdc0abae4f74aad5e4a15cb02e1b P 3b879799899a499c8d409ee9f1277cd1 [term 26 FOLLOWER]: Raft consensus is shut down!
I20260709 13:03:31.490477   431 tablet_replica.cc:333] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1: stopping tablet replica
I20260709 13:03:31.491339   431 raft_consensus.cc:2243] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Raft consensus shutting down.
I20260709 13:03:31.492115   431 raft_consensus.cc:2272] T 83226095f79a46dab445849b1febd425 P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Raft consensus is shut down!
I20260709 13:03:31.495013   431 tablet_replica.cc:333] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1: stopping tablet replica
I20260709 13:03:31.495843   431 raft_consensus.cc:2243] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 25 LEADER]: Raft consensus shutting down.
I20260709 13:03:31.497113   431 raft_consensus.cc:2272] T 78cabb03415041ca9c1a5a400e506131 P 3b879799899a499c8d409ee9f1277cd1 [term 25 FOLLOWER]: Raft consensus is shut down!
I20260709 13:03:31.500321   431 tablet_replica.cc:333] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1: stopping tablet replica
I20260709 13:03:31.501174   431 raft_consensus.cc:2243] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Raft consensus shutting down.
I20260709 13:03:31.501853   431 raft_consensus.cc:2272] T 388afdca3ce84bf0a8ce3331ed0d9b2f P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Raft consensus is shut down!
I20260709 13:03:31.504942   431 tablet_replica.cc:333] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1: stopping tablet replica
I20260709 13:03:31.505739   431 raft_consensus.cc:2243] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 25 LEADER]: Raft consensus shutting down.
I20260709 13:03:31.507133   431 raft_consensus.cc:2272] T 4d4ba8d6f4f947fabf8a2609af29c3d3 P 3b879799899a499c8d409ee9f1277cd1 [term 25 FOLLOWER]: Raft consensus is shut down!
I20260709 13:03:31.510143   431 tablet_replica.cc:333] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1: stopping tablet replica
I20260709 13:03:31.511021   431 raft_consensus.cc:2243] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 25 LEADER]: Raft consensus shutting down.
I20260709 13:03:31.512346   431 raft_consensus.cc:2272] T 03daa4b346f54f1da820d1ff6cd1ff86 P 3b879799899a499c8d409ee9f1277cd1 [term 25 FOLLOWER]: Raft consensus is shut down!
I20260709 13:03:31.515689   431 tablet_replica.cc:333] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1: stopping tablet replica
I20260709 13:03:31.516608   431 raft_consensus.cc:2243] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [term 20 LEADER]: Raft consensus shutting down.
I20260709 13:03:31.518004   431 raft_consensus.cc:2272] T 84e78b66ceec4f759ae9e101a5469e4c P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Raft consensus is shut down!
I20260709 13:03:31.521209   431 tablet_replica.cc:333] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1: stopping tablet replica
I20260709 13:03:31.522064   431 raft_consensus.cc:2243] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 20 LEADER]: Raft consensus shutting down.
I20260709 13:03:31.523422   431 raft_consensus.cc:2272] T 1d9ac76dc2c445edb5a25cf9087998bb P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Raft consensus is shut down!
I20260709 13:03:31.526650   431 tablet_replica.cc:333] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1: stopping tablet replica
I20260709 13:03:31.527534   431 raft_consensus.cc:2243] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 24 LEADER]: Raft consensus shutting down.
I20260709 13:03:31.528874   431 raft_consensus.cc:2272] T 41166f283aea41b0b3f5be60e8234443 P 3b879799899a499c8d409ee9f1277cd1 [term 24 FOLLOWER]: Raft consensus is shut down!
I20260709 13:03:31.531847   431 tablet_replica.cc:333] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1: stopping tablet replica
I20260709 13:03:31.532662   431 raft_consensus.cc:2243] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 21 LEADER]: Raft consensus shutting down.
I20260709 13:03:31.534152   431 raft_consensus.cc:2272] T 2a521fb6c3e7489d9694fd81cadeb33f P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Raft consensus is shut down!
I20260709 13:03:31.537343   431 tablet_replica.cc:333] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1: stopping tablet replica
I20260709 13:03:31.538218   431 raft_consensus.cc:2243] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Raft consensus shutting down.
I20260709 13:03:31.538908   431 raft_consensus.cc:2272] T 86abe5695d5643f888190301b88dacef P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Raft consensus is shut down!
I20260709 13:03:31.541775   431 tablet_replica.cc:333] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1: stopping tablet replica
I20260709 13:03:31.542620   431 raft_consensus.cc:2243] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 26 LEADER]: Raft consensus shutting down.
I20260709 13:03:31.544029   431 raft_consensus.cc:2272] T e4f4b860aab3464383ce8f9d990f4a4a P 3b879799899a499c8d409ee9f1277cd1 [term 26 FOLLOWER]: Raft consensus is shut down!
I20260709 13:03:31.546962   431 tablet_replica.cc:333] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1: stopping tablet replica
I20260709 13:03:31.547780   431 raft_consensus.cc:2243] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Raft consensus shutting down.
I20260709 13:03:31.548468   431 raft_consensus.cc:2272] T 79ed08f2339a4b1ead1e70d78eaec2ec P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Raft consensus is shut down!
I20260709 13:03:31.551417   431 tablet_replica.cc:333] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1: stopping tablet replica
I20260709 13:03:31.552211   431 raft_consensus.cc:2243] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 28 FOLLOWER]: Raft consensus shutting down.
I20260709 13:03:31.552976   431 raft_consensus.cc:2272] T 1c2f5fa72dee4ec58979ace307d2acc3 P 3b879799899a499c8d409ee9f1277cd1 [term 28 FOLLOWER]: Raft consensus is shut down!
I20260709 13:03:31.556207   431 tablet_replica.cc:333] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1: stopping tablet replica
I20260709 13:03:31.557106   431 raft_consensus.cc:2243] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 26 LEADER]: Raft consensus shutting down.
I20260709 13:03:31.558347   431 raft_consensus.cc:2272] T a40e423ed583480280352b261913276c P 3b879799899a499c8d409ee9f1277cd1 [term 26 FOLLOWER]: Raft consensus is shut down!
I20260709 13:03:31.561268   431 tablet_replica.cc:333] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1: stopping tablet replica
I20260709 13:03:31.562027   431 raft_consensus.cc:2243] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 21 LEADER]: Raft consensus shutting down.
I20260709 13:03:31.563256   431 raft_consensus.cc:2272] T 14588fa8cfd6467e989746e326a0ed31 P 3b879799899a499c8d409ee9f1277cd1 [term 21 FOLLOWER]: Raft consensus is shut down!
I20260709 13:03:31.566170   431 tablet_replica.cc:333] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1: stopping tablet replica
I20260709 13:03:31.567003   431 raft_consensus.cc:2243] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 27 LEADER]: Raft consensus shutting down.
I20260709 13:03:31.568272   431 raft_consensus.cc:2272] T 513043a9b96c4f70adad53ac4b5d01dc P 3b879799899a499c8d409ee9f1277cd1 [term 27 FOLLOWER]: Raft consensus is shut down!
I20260709 13:03:31.571218   431 tablet_replica.cc:333] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1: stopping tablet replica
I20260709 13:03:31.572000   431 raft_consensus.cc:2243] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 23 LEADER]: Raft consensus shutting down.
I20260709 13:03:31.573230   431 raft_consensus.cc:2272] T f6028a708cb344fbb08ab8056921d9e7 P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Raft consensus is shut down!
I20260709 13:03:31.576228   431 tablet_replica.cc:333] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1: stopping tablet replica
I20260709 13:03:31.578146   431 raft_consensus.cc:2243] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 23 LEADER]: Raft consensus shutting down.
I20260709 13:03:31.579478   431 raft_consensus.cc:2272] T c6ab3300be1e4a59b64b14d50da193ca P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Raft consensus is shut down!
I20260709 13:03:31.582330   431 tablet_replica.cc:333] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1: stopping tablet replica
I20260709 13:03:31.583137   431 raft_consensus.cc:2243] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 19 LEADER]: Raft consensus shutting down.
I20260709 13:03:31.584306   431 raft_consensus.cc:2272] T d21485c094584689917b2a97b2d3c624 P 3b879799899a499c8d409ee9f1277cd1 [term 19 FOLLOWER]: Raft consensus is shut down!
I20260709 13:03:31.587165   431 tablet_replica.cc:333] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1: stopping tablet replica
I20260709 13:03:31.587954   431 raft_consensus.cc:2243] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 26 LEADER]: Raft consensus shutting down.
I20260709 13:03:31.589340   431 raft_consensus.cc:2272] T c81785db7d654e4288fcaeacf4db2b49 P 3b879799899a499c8d409ee9f1277cd1 [term 26 FOLLOWER]: Raft consensus is shut down!
I20260709 13:03:31.592288   431 tablet_replica.cc:333] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1: stopping tablet replica
I20260709 13:03:31.593101   431 raft_consensus.cc:2243] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Raft consensus shutting down.
I20260709 13:03:31.593776   431 raft_consensus.cc:2272] T 2ebc67196b4f4119981cdb499e3350cc P 3b879799899a499c8d409ee9f1277cd1 [term 17 FOLLOWER]: Raft consensus is shut down!
I20260709 13:03:31.596571   431 tablet_replica.cc:333] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1: stopping tablet replica
I20260709 13:03:31.597389   431 raft_consensus.cc:2243] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 28 FOLLOWER]: Raft consensus shutting down.
I20260709 13:03:31.597998   431 raft_consensus.cc:2272] T f8ab62ffb73246dc9bcf39f7bee51147 P 3b879799899a499c8d409ee9f1277cd1 [term 28 FOLLOWER]: Raft consensus is shut down!
I20260709 13:03:31.602516   431 tablet_replica.cc:333] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1: stopping tablet replica
I20260709 13:03:31.603328   431 raft_consensus.cc:2243] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 22 LEADER]: Raft consensus shutting down.
I20260709 13:03:31.608199   431 raft_consensus.cc:2272] T 592fa647ae2545e09ad5ee25f8cb658e P 3b879799899a499c8d409ee9f1277cd1 [term 22 FOLLOWER]: Raft consensus is shut down!
I20260709 13:03:31.610975   431 tablet_replica.cc:333] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1: stopping tablet replica
I20260709 13:03:31.611768   431 raft_consensus.cc:2243] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 20 LEADER]: Raft consensus shutting down.
I20260709 13:03:31.613160   431 raft_consensus.cc:2272] T dd3f391a609743de886c525de483fc29 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Raft consensus is shut down!
I20260709 13:03:31.615926   431 tablet_replica.cc:333] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1: stopping tablet replica
I20260709 13:03:31.616695   431 raft_consensus.cc:2243] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 24 LEADER]: Raft consensus shutting down.
I20260709 13:03:31.617928   431 raft_consensus.cc:2272] T e527450b1d6c4cb79e9f6d3c87d4ad23 P 3b879799899a499c8d409ee9f1277cd1 [term 24 FOLLOWER]: Raft consensus is shut down!
I20260709 13:03:31.620733   431 tablet_replica.cc:333] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1: stopping tablet replica
I20260709 13:03:31.621524   431 raft_consensus.cc:2243] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 23 LEADER]: Raft consensus shutting down.
I20260709 13:03:31.625476   431 raft_consensus.cc:2272] T 87b86d46388a41cd90374c203cb27914 P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Raft consensus is shut down!
I20260709 13:03:31.628314   431 tablet_replica.cc:333] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1: stopping tablet replica
I20260709 13:03:31.629077   431 raft_consensus.cc:2243] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 18 LEADER]: Raft consensus shutting down.
I20260709 13:03:31.632701   431 raft_consensus.cc:2272] T e26344297fff44bb8d2099b8e64d7df2 P 3b879799899a499c8d409ee9f1277cd1 [term 18 FOLLOWER]: Raft consensus is shut down!
I20260709 13:03:31.635561   431 tablet_replica.cc:333] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1: stopping tablet replica
I20260709 13:03:31.636291   431 raft_consensus.cc:2243] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 25 FOLLOWER]: Raft consensus shutting down.
I20260709 13:03:31.636910   431 raft_consensus.cc:2272] T 983621a6fc5b4c9ea0159a648dfe7a36 P 3b879799899a499c8d409ee9f1277cd1 [term 25 FOLLOWER]: Raft consensus is shut down!
I20260709 13:03:31.640079   431 tablet_replica.cc:333] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1: stopping tablet replica
I20260709 13:03:31.640843   431 raft_consensus.cc:2243] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 27 FOLLOWER]: Raft consensus shutting down.
I20260709 13:03:31.641592   431 raft_consensus.cc:2272] T 344b4152823f4d618345b14ffd579400 P 3b879799899a499c8d409ee9f1277cd1 [term 27 FOLLOWER]: Raft consensus is shut down!
I20260709 13:03:31.644376   431 tablet_replica.cc:333] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1: stopping tablet replica
I20260709 13:03:31.645123   431 raft_consensus.cc:2243] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 29 FOLLOWER]: Raft consensus shutting down.
I20260709 13:03:31.645768   431 raft_consensus.cc:2272] T dce8db7565da4e13a82ffbb9c39fde78 P 3b879799899a499c8d409ee9f1277cd1 [term 29 FOLLOWER]: Raft consensus is shut down!
I20260709 13:03:31.648151   431 tablet_replica.cc:333] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1: stopping tablet replica
I20260709 13:03:31.648674   431 raft_consensus.cc:2243] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 24 LEADER]: Raft consensus shutting down.
I20260709 13:03:31.649681   431 raft_consensus.cc:2272] T cd44a2ebdc764e898f25ad5390f4bffd P 3b879799899a499c8d409ee9f1277cd1 [term 24 FOLLOWER]: Raft consensus is shut down!
I20260709 13:03:31.651742   431 tablet_replica.cc:333] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1: stopping tablet replica
I20260709 13:03:31.652289   431 raft_consensus.cc:2243] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Raft consensus shutting down.
I20260709 13:03:31.652755   431 raft_consensus.cc:2272] T 1809d674329a492dbea0b60a4703fe51 P 3b879799899a499c8d409ee9f1277cd1 [term 20 FOLLOWER]: Raft consensus is shut down!
I20260709 13:03:31.654778   431 tablet_replica.cc:333] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1: stopping tablet replica
I20260709 13:03:31.655267   431 raft_consensus.cc:2243] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Raft consensus shutting down.
I20260709 13:03:31.655831   431 raft_consensus.cc:2272] T a3accbb733bb4cafa69428319664f239 P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Raft consensus is shut down!
I20260709 13:03:31.658002   431 tablet_replica.cc:333] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1: stopping tablet replica
I20260709 13:03:31.658536   431 raft_consensus.cc:2243] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Raft consensus shutting down.
I20260709 13:03:31.659148   431 raft_consensus.cc:2272] T 77d66b92688a42e3ad231959a644db56 P 3b879799899a499c8d409ee9f1277cd1 [term 23 FOLLOWER]: Raft consensus is shut down!
W20260709 13:03:31.856565  2358 debug-util.cc:398] Leaking SignalData structure 0x7b0800050320 after lost signal to thread 434
W20260709 13:03:31.858039  2358 debug-util.cc:398] Leaking SignalData structure 0x7b08001682c0 after lost signal to thread 1537
W20260709 13:03:31.859611  2358 debug-util.cc:398] Leaking SignalData structure 0x7b08002f5860 after lost signal to thread 2361
W20260709 13:03:32.734835   431 thread.cc:527] Waited for 1000ms trying to join with diag-logger (tid 2358)
I20260709 13:03:33.398162   431 tablet_server.cc:196] TabletServer@127.0.107.196:0 shutdown complete.
I20260709 13:03:33.487769   431 master.cc:562] Master@127.0.107.254:33643 shutting down...
W20260709 13:03:33.540041  1551 auto_rebalancer.cc:873] Removing replace marker failed: Catalog manager is not initialized. State: Closing
W20260709 13:03:33.540606  1551 auto_rebalancer.cc:901] Could not move replica: Service unavailable: Catalog manager is not initialized. State: Closing
W20260709 13:03:33.540879  1551 auto_rebalancer.cc:330] scheduled replica move failed to complete: Service unavailable: Catalog manager is not initialized. State: Closing
W20260709 13:03:39.356484  1487 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111) [suppressed 140 similar messages]
W20260709 13:03:39.361125  1487 catalog_manager.cc:4755] TS 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 983621a6fc5b4c9ea0159a648dfe7a36: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111)
W20260709 13:03:44.324506  1487 catalog_manager.cc:4755] TS 83595ad0fd5940c0a2bfc7c61acd969a (127.0.107.193:46445): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 7ebbb6d7c6c4478e95972de2f78d3f80: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46445: connect: Connection refused (error 111)
I20260709 13:03:44.444317   431 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 919f6392d4f44fd68987e2c5e7ec37ca [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:03:44.445660   431 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 919f6392d4f44fd68987e2c5e7ec37ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:03:44.446134   431 tablet_replica.cc:333] T 00000000000000000000000000000000 P 919f6392d4f44fd68987e2c5e7ec37ca: stopping tablet replica
I20260709 13:03:44.610924   431 master.cc:584] Master@127.0.107.254:33643 shutdown complete.
I20260709 13:03:44.675181   431 test_util.cc:182] -----------------------------------------------
I20260709 13:03:44.675482   431 test_util.cc:183] Had failures, leaving test files at /tmp/dist-test-taskakPCWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783601878574137-431-0
[  FAILED  ] LeaderRebalancerTest.AddTserver (312564 ms)
[ RUN      ] LeaderRebalancerTest.RestartTserver
I20260709 13:03:44.679478   431 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.107.254:40861
I20260709 13:03:44.680737   431 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 13:03:44.687268 10753 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:03:44.687961 10756 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:03:44.695197 10754 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:03:44.695993   431 server_base.cc:1061] running on GCE node
I20260709 13:03:44.702287   431 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 13:03:44.702574   431 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 13:03:44.702771   431 hybrid_clock.cc:648] HybridClock initialized: now 1783602224702736 us; error 0 us; skew 500 ppm
I20260709 13:03:44.703382   431 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 13:03:44.706168   431 webserver.cc:533] Webserver started at http://127.0.107.254:41807/ using document root <none> and password file <none>
I20260709 13:03:44.706763   431 fs_manager.cc:362] Metadata directory not provided
I20260709 13:03:44.706983   431 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 13:03:44.707297   431 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 13:03:44.708820   431 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskakPCWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783601878574137-431-0/minicluster-data/master-0-root/instance:
uuid: "c669bee8ad644d0d956c148b33886b51"
format_stamp: "Formatted at 2026-07-09 13:03:44 on dist-test-slave-nvvq"
I20260709 13:03:44.713783   431 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.006s	sys 0.000s
I20260709 13:03:44.717489 10761 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:03:44.718308   431 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.003s	sys 0.000s
I20260709 13:03:44.718636   431 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskakPCWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783601878574137-431-0/minicluster-data/master-0-root
uuid: "c669bee8ad644d0d956c148b33886b51"
format_stamp: "Formatted at 2026-07-09 13:03:44 on dist-test-slave-nvvq"
I20260709 13:03:44.718940   431 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskakPCWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783601878574137-431-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskakPCWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783601878574137-431-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskakPCWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783601878574137-431-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 13:03:44.777026   431 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 13:03:44.778743   431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 13:03:44.829366   431 rpc_server.cc:307] RPC server started. Bound to: 127.0.107.254:40861
I20260709 13:03:44.829476 10812 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.107.254:40861 every 8 connection(s)
I20260709 13:03:44.835018 10813 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:44.850178 10813 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c669bee8ad644d0d956c148b33886b51: Bootstrap starting.
I20260709 13:03:44.856623 10813 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c669bee8ad644d0d956c148b33886b51: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:44.861806 10813 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c669bee8ad644d0d956c148b33886b51: No bootstrap required, opened a new log
I20260709 13:03:44.864584 10813 raft_consensus.cc:359] T 00000000000000000000000000000000 P c669bee8ad644d0d956c148b33886b51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c669bee8ad644d0d956c148b33886b51" member_type: VOTER }
I20260709 13:03:44.865099 10813 raft_consensus.cc:385] T 00000000000000000000000000000000 P c669bee8ad644d0d956c148b33886b51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:44.865299 10813 raft_consensus.cc:740] T 00000000000000000000000000000000 P c669bee8ad644d0d956c148b33886b51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c669bee8ad644d0d956c148b33886b51, State: Initialized, Role: FOLLOWER
I20260709 13:03:44.865866 10813 consensus_queue.cc:260] T 00000000000000000000000000000000 P c669bee8ad644d0d956c148b33886b51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c669bee8ad644d0d956c148b33886b51" member_type: VOTER }
I20260709 13:03:44.866353 10813 raft_consensus.cc:399] T 00000000000000000000000000000000 P c669bee8ad644d0d956c148b33886b51 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260709 13:03:44.866657 10813 raft_consensus.cc:493] T 00000000000000000000000000000000 P c669bee8ad644d0d956c148b33886b51 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260709 13:03:44.866895 10813 raft_consensus.cc:3060] T 00000000000000000000000000000000 P c669bee8ad644d0d956c148b33886b51 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:03:44.874356 10813 raft_consensus.cc:515] T 00000000000000000000000000000000 P c669bee8ad644d0d956c148b33886b51 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c669bee8ad644d0d956c148b33886b51" member_type: VOTER }
I20260709 13:03:44.874891 10813 leader_election.cc:304] T 00000000000000000000000000000000 P c669bee8ad644d0d956c148b33886b51 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c669bee8ad644d0d956c148b33886b51; no voters: 
I20260709 13:03:44.876201 10813 leader_election.cc:290] T 00000000000000000000000000000000 P c669bee8ad644d0d956c148b33886b51 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20260709 13:03:44.876772 10816 raft_consensus.cc:2804] T 00000000000000000000000000000000 P c669bee8ad644d0d956c148b33886b51 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:03:44.878468 10816 raft_consensus.cc:697] T 00000000000000000000000000000000 P c669bee8ad644d0d956c148b33886b51 [term 1 LEADER]: Becoming Leader. State: Replica: c669bee8ad644d0d956c148b33886b51, State: Running, Role: LEADER
I20260709 13:03:44.879316 10816 consensus_queue.cc:237] T 00000000000000000000000000000000 P c669bee8ad644d0d956c148b33886b51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c669bee8ad644d0d956c148b33886b51" member_type: VOTER }
I20260709 13:03:44.879535 10813 sys_catalog.cc:565] T 00000000000000000000000000000000 P c669bee8ad644d0d956c148b33886b51 [sys.catalog]: configured and running, proceeding with master startup.
I20260709 13:03:44.887388 10817 sys_catalog.cc:455] T 00000000000000000000000000000000 P c669bee8ad644d0d956c148b33886b51 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c669bee8ad644d0d956c148b33886b51" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c669bee8ad644d0d956c148b33886b51" member_type: VOTER } }
I20260709 13:03:44.888069 10817 sys_catalog.cc:458] T 00000000000000000000000000000000 P c669bee8ad644d0d956c148b33886b51 [sys.catalog]: This master's current role is: LEADER
I20260709 13:03:44.887926 10818 sys_catalog.cc:455] T 00000000000000000000000000000000 P c669bee8ad644d0d956c148b33886b51 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c669bee8ad644d0d956c148b33886b51. Latest consensus state: current_term: 1 leader_uuid: "c669bee8ad644d0d956c148b33886b51" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c669bee8ad644d0d956c148b33886b51" member_type: VOTER } }
I20260709 13:03:44.888649 10818 sys_catalog.cc:458] T 00000000000000000000000000000000 P c669bee8ad644d0d956c148b33886b51 [sys.catalog]: This master's current role is: LEADER
I20260709 13:03:44.890831 10824 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260709 13:03:44.900072 10824 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260709 13:03:44.901254   431 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260709 13:03:44.911037 10824 catalog_manager.cc:1383] Generated new cluster ID: 2d5c8c46587f4660be2611d1a73d7004
I20260709 13:03:44.911300 10824 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260709 13:03:44.932792 10824 catalog_manager.cc:1406] Generated new certificate authority record
I20260709 13:03:44.934312 10824 catalog_manager.cc:1540] Loading token signing keys...
I20260709 13:03:44.953456 10824 catalog_manager.cc:6070] T 00000000000000000000000000000000 P c669bee8ad644d0d956c148b33886b51: Generated new TSK 0
I20260709 13:03:44.954100 10824 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260709 13:03:44.968137   431 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 13:03:44.976126 10834 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:03:44.977082 10835 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:03:44.979086 10837 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:03:44.983855   431 server_base.cc:1061] running on GCE node
I20260709 13:03:44.988632   431 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 13:03:44.988893   431 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 13:03:44.989073   431 hybrid_clock.cc:648] HybridClock initialized: now 1783602224989051 us; error 0 us; skew 500 ppm
I20260709 13:03:44.989732   431 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 13:03:44.992466   431 webserver.cc:533] Webserver started at http://127.0.107.193:44315/ using document root <none> and password file <none>
I20260709 13:03:44.993031   431 fs_manager.cc:362] Metadata directory not provided
I20260709 13:03:44.993278   431 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 13:03:44.993593   431 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 13:03:44.995167   431 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskakPCWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783601878574137-431-0/minicluster-data/ts-0-root/instance:
uuid: "186b12638a87410487467df721a3cbda"
format_stamp: "Formatted at 2026-07-09 13:03:44 on dist-test-slave-nvvq"
I20260709 13:03:45.000351   431 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.005s	sys 0.000s
I20260709 13:03:45.004283 10842 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:03:45.005311   431 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.001s	sys 0.000s
I20260709 13:03:45.005699   431 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskakPCWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783601878574137-431-0/minicluster-data/ts-0-root
uuid: "186b12638a87410487467df721a3cbda"
format_stamp: "Formatted at 2026-07-09 13:03:44 on dist-test-slave-nvvq"
I20260709 13:03:45.006142   431 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskakPCWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783601878574137-431-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskakPCWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783601878574137-431-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskakPCWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783601878574137-431-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 13:03:45.023166   431 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 13:03:45.024597   431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 13:03:45.026348   431 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 13:03:45.029280   431 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 13:03:45.029476   431 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:03:45.029801   431 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 13:03:45.030018   431 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:03:45.082685   431 rpc_server.cc:307] RPC server started. Bound to: 127.0.107.193:38967
I20260709 13:03:45.082836 10904 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.107.193:38967 every 8 connection(s)
I20260709 13:03:45.089416   431 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 13:03:45.099246 10909 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:03:45.100096 10910 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:03:45.102231 10912 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:03:45.104574   431 server_base.cc:1061] running on GCE node
I20260709 13:03:45.109802   431 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 13:03:45.110167   431 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 13:03:45.110364 10905 heartbeater.cc:344] Connected to a master server at 127.0.107.254:40861
I20260709 13:03:45.110364   431 hybrid_clock.cc:648] HybridClock initialized: now 1783602225110346 us; error 0 us; skew 500 ppm
I20260709 13:03:45.111020 10905 heartbeater.cc:461] Registering TS with master...
I20260709 13:03:45.111423   431 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 13:03:45.112023 10905 heartbeater.cc:507] Master 127.0.107.254:40861 requested a full tablet report, sending...
I20260709 13:03:45.114348   431 webserver.cc:533] Webserver started at http://127.0.107.194:40023/ using document root <none> and password file <none>
I20260709 13:03:45.114877 10778 ts_manager.cc:194] Registered new tserver with Master: 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:03:45.115196   431 fs_manager.cc:362] Metadata directory not provided
I20260709 13:03:45.115551   431 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 13:03:45.115922   431 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 13:03:45.117865 10778 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:59598
I20260709 13:03:45.117815   431 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskakPCWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783601878574137-431-0/minicluster-data/ts-1-root/instance:
uuid: "10ca5ee68ad64b3283b9d235352c9d23"
format_stamp: "Formatted at 2026-07-09 13:03:45 on dist-test-slave-nvvq"
I20260709 13:03:45.123742   431 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.001s	sys 0.004s
I20260709 13:03:45.127544 10917 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:03:45.128355   431 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.004s	sys 0.000s
I20260709 13:03:45.128675   431 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskakPCWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783601878574137-431-0/minicluster-data/ts-1-root
uuid: "10ca5ee68ad64b3283b9d235352c9d23"
format_stamp: "Formatted at 2026-07-09 13:03:45 on dist-test-slave-nvvq"
I20260709 13:03:45.129005   431 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskakPCWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783601878574137-431-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskakPCWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783601878574137-431-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskakPCWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783601878574137-431-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 13:03:45.140429   431 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 13:03:45.141938   431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 13:03:45.143911   431 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 13:03:45.146667   431 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 13:03:45.146911   431 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:03:45.147168   431 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 13:03:45.147342   431 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:03:45.200260   431 rpc_server.cc:307] RPC server started. Bound to: 127.0.107.194:44903
I20260709 13:03:45.200413 10979 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.107.194:44903 every 8 connection(s)
I20260709 13:03:45.206318   431 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 13:03:45.216785 10983 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:03:45.217219 10984 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:03:45.220384   431 server_base.cc:1061] running on GCE node
W20260709 13:03:45.221439 10986 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:03:45.221511 10980 heartbeater.cc:344] Connected to a master server at 127.0.107.254:40861
I20260709 13:03:45.222364 10980 heartbeater.cc:461] Registering TS with master...
I20260709 13:03:45.222921   431 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 13:03:45.223217   431 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 13:03:45.223399   431 hybrid_clock.cc:648] HybridClock initialized: now 1783602225223382 us; error 0 us; skew 500 ppm
I20260709 13:03:45.223446 10980 heartbeater.cc:507] Master 127.0.107.254:40861 requested a full tablet report, sending...
I20260709 13:03:45.224089   431 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 13:03:45.226315 10778 ts_manager.cc:194] Registered new tserver with Master: 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:03:45.227500   431 webserver.cc:533] Webserver started at http://127.0.107.195:46683/ using document root <none> and password file <none>
I20260709 13:03:45.228246   431 fs_manager.cc:362] Metadata directory not provided
I20260709 13:03:45.228525   431 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 13:03:45.228668 10778 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:59608
I20260709 13:03:45.228905   431 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 13:03:45.230973   431 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskakPCWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783601878574137-431-0/minicluster-data/ts-2-root/instance:
uuid: "873357742d5843579556ee8166f23f47"
format_stamp: "Formatted at 2026-07-09 13:03:45 on dist-test-slave-nvvq"
I20260709 13:03:45.236351   431 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.001s	sys 0.003s
I20260709 13:03:45.240278 10991 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:03:45.241181   431 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.000s	sys 0.001s
I20260709 13:03:45.241529   431 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskakPCWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783601878574137-431-0/minicluster-data/ts-2-root
uuid: "873357742d5843579556ee8166f23f47"
format_stamp: "Formatted at 2026-07-09 13:03:45 on dist-test-slave-nvvq"
I20260709 13:03:45.241858   431 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskakPCWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783601878574137-431-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskakPCWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783601878574137-431-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskakPCWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783601878574137-431-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 13:03:45.263077   431 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 13:03:45.264732   431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 13:03:45.266669   431 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 13:03:45.269243   431 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 13:03:45.269477   431 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:03:45.269734   431 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 13:03:45.269919   431 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:03:45.326233   431 rpc_server.cc:307] RPC server started. Bound to: 127.0.107.195:45807
I20260709 13:03:45.326333 11053 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.107.195:45807 every 8 connection(s)
I20260709 13:03:45.332784   431 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 13:03:45.343117 11057 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:03:45.344169 11058 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:03:45.347039   431 server_base.cc:1061] running on GCE node
W20260709 13:03:45.347144 11060 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:03:45.348712   431 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 13:03:45.349094   431 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 13:03:45.349124 11054 heartbeater.cc:344] Connected to a master server at 127.0.107.254:40861
I20260709 13:03:45.349313   431 hybrid_clock.cc:648] HybridClock initialized: now 1783602225349293 us; error 0 us; skew 500 ppm
I20260709 13:03:45.349757 11054 heartbeater.cc:461] Registering TS with master...
I20260709 13:03:45.350266   431 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 13:03:45.350871 11054 heartbeater.cc:507] Master 127.0.107.254:40861 requested a full tablet report, sending...
I20260709 13:03:45.353511   431 webserver.cc:533] Webserver started at http://127.0.107.196:44755/ using document root <none> and password file <none>
I20260709 13:03:45.353731 10778 ts_manager.cc:194] Registered new tserver with Master: 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:03:45.354202   431 fs_manager.cc:362] Metadata directory not provided
I20260709 13:03:45.354525   431 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 13:03:45.354830   431 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 13:03:45.355587 10778 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:59610
I20260709 13:03:45.356453   431 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskakPCWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783601878574137-431-0/minicluster-data/ts-3-root/instance:
uuid: "0172fed22b6b4890a7fdc953fa128af4"
format_stamp: "Formatted at 2026-07-09 13:03:45 on dist-test-slave-nvvq"
I20260709 13:03:45.361912   431 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.006s	sys 0.000s
I20260709 13:03:45.366654 11065 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:03:45.367477   431 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.000s	sys 0.003s
I20260709 13:03:45.367760   431 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskakPCWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783601878574137-431-0/minicluster-data/ts-3-root
uuid: "0172fed22b6b4890a7fdc953fa128af4"
format_stamp: "Formatted at 2026-07-09 13:03:45 on dist-test-slave-nvvq"
I20260709 13:03:45.368028   431 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskakPCWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783601878574137-431-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskakPCWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783601878574137-431-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskakPCWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783601878574137-431-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 13:03:45.380551   431 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 13:03:45.382370   431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 13:03:45.384217   431 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 13:03:45.387100   431 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 13:03:45.387356   431 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:03:45.387676   431 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 13:03:45.387849   431 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:03:45.442998   431 rpc_server.cc:307] RPC server started. Bound to: 127.0.107.196:38507
I20260709 13:03:45.443113 11127 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.107.196:38507 every 8 connection(s)
I20260709 13:03:45.461385 11128 heartbeater.cc:344] Connected to a master server at 127.0.107.254:40861
I20260709 13:03:45.461938 11128 heartbeater.cc:461] Registering TS with master...
I20260709 13:03:45.462929 11128 heartbeater.cc:507] Master 127.0.107.254:40861 requested a full tablet report, sending...
I20260709 13:03:45.465055 10778 ts_manager.cc:194] Registered new tserver with Master: 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:03:45.465185   431 internal_mini_cluster.cc:371] 4 TS(s) registered with all masters after 0.017982821s
I20260709 13:03:45.466811 10778 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:59614
I20260709 13:03:45.495321 10778 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59616:
name: "test-workload"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
  rows: "\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}""\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}"
  indirect_data: """"
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I20260709 13:03:45.787207 11093 tablet_service.cc:1511] Processing CreateTablet for tablet 6a9353d1d230416296627d7cf8b1d425 (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260709 13:03:45.789161 11093 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a9353d1d230416296627d7cf8b1d425. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:45.791749 11092 tablet_service.cc:1511] Processing CreateTablet for tablet 0f9df2bc281f4866b6da5da94a0defc7 (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260709 13:03:45.793787 11092 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f9df2bc281f4866b6da5da94a0defc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:45.798511 11091 tablet_service.cc:1511] Processing CreateTablet for tablet 6f46a5102fd64b45bf1c4dff863d2b4d (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260709 13:03:45.800443 11091 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f46a5102fd64b45bf1c4dff863d2b4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:45.812320 11090 tablet_service.cc:1511] Processing CreateTablet for tablet d7ce47b9a92d43dd88d95bcda5038597 (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260709 13:03:45.816031 11018 tablet_service.cc:1511] Processing CreateTablet for tablet 7839ca2d2c13431a85f44d7c458c6351 (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260709 13:03:45.816401 10945 tablet_service.cc:1511] Processing CreateTablet for tablet 7839ca2d2c13431a85f44d7c458c6351 (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260709 13:03:45.818074 11018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7839ca2d2c13431a85f44d7c458c6351. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:45.818574 10945 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7839ca2d2c13431a85f44d7c458c6351. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:45.818089 11089 tablet_service.cc:1511] Processing CreateTablet for tablet 4f9eb06ed01f4ce68be3af8a8434665b (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260709 13:03:45.819952 11089 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f9eb06ed01f4ce68be3af8a8434665b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:45.819842 11017 tablet_service.cc:1511] Processing CreateTablet for tablet 6f46a5102fd64b45bf1c4dff863d2b4d (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260709 13:03:45.822044 11090 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7ce47b9a92d43dd88d95bcda5038597. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:45.821593 10944 tablet_service.cc:1511] Processing CreateTablet for tablet 0f9df2bc281f4866b6da5da94a0defc7 (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260709 13:03:45.823704 10944 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f9df2bc281f4866b6da5da94a0defc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:45.823861 11019 tablet_service.cc:1511] Processing CreateTablet for tablet 6a9353d1d230416296627d7cf8b1d425 (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260709 13:03:45.826637 11016 tablet_service.cc:1511] Processing CreateTablet for tablet 4c2a35bb99374e93b573cf870d04faaa (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260709 13:03:45.826776 10943 tablet_service.cc:1511] Processing CreateTablet for tablet 6f46a5102fd64b45bf1c4dff863d2b4d (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260709 13:03:45.828738 10943 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f46a5102fd64b45bf1c4dff863d2b4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:45.831067 11015 tablet_service.cc:1511] Processing CreateTablet for tablet 4f9eb06ed01f4ce68be3af8a8434665b (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260709 13:03:45.832196 10942 tablet_service.cc:1511] Processing CreateTablet for tablet 4c2a35bb99374e93b573cf870d04faaa (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260709 13:03:45.833765 10942 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c2a35bb99374e93b573cf870d04faaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:45.834726 10941 tablet_service.cc:1511] Processing CreateTablet for tablet d7ce47b9a92d43dd88d95bcda5038597 (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260709 13:03:45.836284 10941 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7ce47b9a92d43dd88d95bcda5038597. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:45.836848 11088 tablet_service.cc:1511] Processing CreateTablet for tablet e5ba72ccd854430baadbc636dfd17136 (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260709 13:03:45.838660 11088 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5ba72ccd854430baadbc636dfd17136. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:45.842414 11014 tablet_service.cc:1511] Processing CreateTablet for tablet e5ba72ccd854430baadbc636dfd17136 (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260709 13:03:45.843739 10940 tablet_service.cc:1511] Processing CreateTablet for tablet e5ba72ccd854430baadbc636dfd17136 (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260709 13:03:45.847500 10940 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5ba72ccd854430baadbc636dfd17136. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:45.845745 10939 tablet_service.cc:1511] Processing CreateTablet for tablet 57b6a2418c1342e7924195e9c1dbd357 (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260709 13:03:45.850597 10939 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57b6a2418c1342e7924195e9c1dbd357. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:45.850355 11012 tablet_service.cc:1511] Processing CreateTablet for tablet 7709387f4d3d4f9992d97a0fbca39a40 (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260709 13:03:45.850937 10938 tablet_service.cc:1511] Processing CreateTablet for tablet c51782cd9d064ec0949e84e24dd4de67 (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260709 13:03:45.843847 11013 tablet_service.cc:1511] Processing CreateTablet for tablet 57b6a2418c1342e7924195e9c1dbd357 (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260709 13:03:45.852453 10938 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c51782cd9d064ec0949e84e24dd4de67. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:45.855068 11015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f9eb06ed01f4ce68be3af8a8434665b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:45.858952 11017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f46a5102fd64b45bf1c4dff863d2b4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:45.860524 11087 tablet_service.cc:1511] Processing CreateTablet for tablet c51782cd9d064ec0949e84e24dd4de67 (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260709 13:03:45.862406 11087 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c51782cd9d064ec0949e84e24dd4de67. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:45.862882 11019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a9353d1d230416296627d7cf8b1d425. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:45.867183 11013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57b6a2418c1342e7924195e9c1dbd357. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:45.869843 11012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7709387f4d3d4f9992d97a0fbca39a40. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:45.875093 11016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c2a35bb99374e93b573cf870d04faaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:45.881408 11014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5ba72ccd854430baadbc636dfd17136. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:45.888356 10937 tablet_service.cc:1511] Processing CreateTablet for tablet 525bfdbd6d934afd823f7d3ee3f9d052 (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260709 13:03:45.889503 11086 tablet_service.cc:1511] Processing CreateTablet for tablet 7709387f4d3d4f9992d97a0fbca39a40 (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260709 13:03:45.890344 10937 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 525bfdbd6d934afd823f7d3ee3f9d052. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:45.891363 11086 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7709387f4d3d4f9992d97a0fbca39a40. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:45.896165 11011 tablet_service.cc:1511] Processing CreateTablet for tablet 525bfdbd6d934afd823f7d3ee3f9d052 (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260709 13:03:45.898064 11011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 525bfdbd6d934afd823f7d3ee3f9d052. 1 dirs total, 0 dirs full, 0 dirs failed
W20260709 13:03:45.899183 10826 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 13:03:45.931365 11085 tablet_service.cc:1511] Processing CreateTablet for tablet 525bfdbd6d934afd823f7d3ee3f9d052 (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260709 13:03:45.933384 11085 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 525bfdbd6d934afd823f7d3ee3f9d052. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:46.003129 10936 tablet_service.cc:1511] Processing CreateTablet for tablet 34064dc7bf3747dbb3876dd824ee0688 (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260709 13:03:46.004381 11016 tablet_service.cc:1511] Processing CreateTablet for tablet 34064dc7bf3747dbb3876dd824ee0688 (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260709 13:03:46.004940 10936 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34064dc7bf3747dbb3876dd824ee0688. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:46.005957 11016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34064dc7bf3747dbb3876dd824ee0688. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:46.006948 10870 tablet_service.cc:1511] Processing CreateTablet for tablet 6a9353d1d230416296627d7cf8b1d425 (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260709 13:03:46.007742 10868 tablet_service.cc:1511] Processing CreateTablet for tablet 0f9df2bc281f4866b6da5da94a0defc7 (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260709 13:03:46.008603 10870 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a9353d1d230416296627d7cf8b1d425. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:46.009537 10867 tablet_service.cc:1511] Processing CreateTablet for tablet 4c2a35bb99374e93b573cf870d04faaa (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260709 13:03:46.011130 11084 tablet_service.cc:1511] Processing CreateTablet for tablet 8a69f32c53f24a50b7852d143665d7b4 (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260709 13:03:46.011874 10869 tablet_service.cc:1511] Processing CreateTablet for tablet 7839ca2d2c13431a85f44d7c458c6351 (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260709 13:03:46.012722 11084 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a69f32c53f24a50b7852d143665d7b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:46.013500 10869 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7839ca2d2c13431a85f44d7c458c6351. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:46.013294 10865 tablet_service.cc:1511] Processing CreateTablet for tablet 4f9eb06ed01f4ce68be3af8a8434665b (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260709 13:03:46.015957 10864 tablet_service.cc:1511] Processing CreateTablet for tablet 57b6a2418c1342e7924195e9c1dbd357 (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260709 13:03:46.017534 10864 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57b6a2418c1342e7924195e9c1dbd357. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:46.011880 10866 tablet_service.cc:1511] Processing CreateTablet for tablet d7ce47b9a92d43dd88d95bcda5038597 (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260709 13:03:46.031847 10863 tablet_service.cc:1511] Processing CreateTablet for tablet c51782cd9d064ec0949e84e24dd4de67 (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260709 13:03:46.044822 10862 tablet_service.cc:1511] Processing CreateTablet for tablet 7709387f4d3d4f9992d97a0fbca39a40 (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260709 13:03:46.041958 10861 tablet_service.cc:1511] Processing CreateTablet for tablet 34064dc7bf3747dbb3876dd824ee0688 (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260709 13:03:46.031857 10868 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f9df2bc281f4866b6da5da94a0defc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:46.049499 10867 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c2a35bb99374e93b573cf870d04faaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:46.050967 10862 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7709387f4d3d4f9992d97a0fbca39a40. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:46.051860 10866 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7ce47b9a92d43dd88d95bcda5038597. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:46.054831 10863 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c51782cd9d064ec0949e84e24dd4de67. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:46.058823 10861 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34064dc7bf3747dbb3876dd824ee0688. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:46.060081 10865 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f9eb06ed01f4ce68be3af8a8434665b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:46.063472 11148 tablet_bootstrap.cc:492] T 4c2a35bb99374e93b573cf870d04faaa P 10ca5ee68ad64b3283b9d235352c9d23: Bootstrap starting.
I20260709 13:03:46.078356 10944 tablet_service.cc:1511] Processing CreateTablet for tablet 8a69f32c53f24a50b7852d143665d7b4 (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260709 13:03:46.080132 10944 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a69f32c53f24a50b7852d143665d7b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:46.099144 11149 tablet_bootstrap.cc:492] T 4c2a35bb99374e93b573cf870d04faaa P 873357742d5843579556ee8166f23f47: Bootstrap starting.
I20260709 13:03:46.100162 10863 tablet_service.cc:1511] Processing CreateTablet for tablet 8a69f32c53f24a50b7852d143665d7b4 (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260709 13:03:46.102109 10863 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a69f32c53f24a50b7852d143665d7b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:46.121384 10905 heartbeater.cc:499] Master 127.0.107.254:40861 was elected leader, sending a full tablet report...
I20260709 13:03:46.122532 11015 tablet_service.cc:1511] Processing CreateTablet for tablet 7757ecbf402e47bfbc93d0447741e26c (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260709 13:03:46.136740 11015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7757ecbf402e47bfbc93d0447741e26c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:46.140755 11149 tablet_bootstrap.cc:654] T 4c2a35bb99374e93b573cf870d04faaa P 873357742d5843579556ee8166f23f47: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:46.151826 11148 tablet_bootstrap.cc:654] T 4c2a35bb99374e93b573cf870d04faaa P 10ca5ee68ad64b3283b9d235352c9d23: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:46.155916 11151 tablet_bootstrap.cc:492] T 4c2a35bb99374e93b573cf870d04faaa P 186b12638a87410487467df721a3cbda: Bootstrap starting.
I20260709 13:03:46.160076 10941 tablet_service.cc:1511] Processing CreateTablet for tablet d71ca3a1ba6b48478aaf62741bf3cf6a (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260709 13:03:46.161932 10941 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d71ca3a1ba6b48478aaf62741bf3cf6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:46.164028 11093 tablet_service.cc:1511] Processing CreateTablet for tablet 7757ecbf402e47bfbc93d0447741e26c (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260709 13:03:46.164227 10867 tablet_service.cc:1511] Processing CreateTablet for tablet 7757ecbf402e47bfbc93d0447741e26c (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260709 13:03:46.180109 11152 tablet_bootstrap.cc:492] T 4f9eb06ed01f4ce68be3af8a8434665b P 0172fed22b6b4890a7fdc953fa128af4: Bootstrap starting.
I20260709 13:03:46.181185 10867 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7757ecbf402e47bfbc93d0447741e26c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:46.191349 11017 tablet_service.cc:1511] Processing CreateTablet for tablet d71ca3a1ba6b48478aaf62741bf3cf6a (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260709 13:03:46.193289 11017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d71ca3a1ba6b48478aaf62741bf3cf6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:46.194839 11093 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7757ecbf402e47bfbc93d0447741e26c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:46.205874 11148 tablet_bootstrap.cc:492] T 4c2a35bb99374e93b573cf870d04faaa P 10ca5ee68ad64b3283b9d235352c9d23: No bootstrap required, opened a new log
I20260709 13:03:46.207126 11148 ts_tablet_manager.cc:1403] T 4c2a35bb99374e93b573cf870d04faaa P 10ca5ee68ad64b3283b9d235352c9d23: Time spent bootstrapping tablet: real 0.144s	user 0.007s	sys 0.012s
I20260709 13:03:46.209761 11151 tablet_bootstrap.cc:654] T 4c2a35bb99374e93b573cf870d04faaa P 186b12638a87410487467df721a3cbda: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:46.212576 11148 raft_consensus.cc:359] T 4c2a35bb99374e93b573cf870d04faaa P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:46.215581 11148 raft_consensus.cc:385] T 4c2a35bb99374e93b573cf870d04faaa P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:46.221401 11148 raft_consensus.cc:740] T 4c2a35bb99374e93b573cf870d04faaa P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10ca5ee68ad64b3283b9d235352c9d23, State: Initialized, Role: FOLLOWER
I20260709 13:03:46.222792 11148 consensus_queue.cc:260] T 4c2a35bb99374e93b573cf870d04faaa P 10ca5ee68ad64b3283b9d235352c9d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:46.309783 10864 tablet_service.cc:1511] Processing CreateTablet for tablet e96e80ba9ac34d0998395515e5c8c41a (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260709 13:03:46.314304 10940 tablet_service.cc:1511] Processing CreateTablet for tablet e96e80ba9ac34d0998395515e5c8c41a (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260709 13:03:46.315667 10864 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e96e80ba9ac34d0998395515e5c8c41a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:46.320345 10980 heartbeater.cc:499] Master 127.0.107.254:40861 was elected leader, sending a full tablet report...
I20260709 13:03:46.320845 10940 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e96e80ba9ac34d0998395515e5c8c41a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:46.343154 11014 tablet_service.cc:1511] Processing CreateTablet for tablet e96e80ba9ac34d0998395515e5c8c41a (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260709 13:03:46.344820 11088 tablet_service.cc:1511] Processing CreateTablet for tablet d71ca3a1ba6b48478aaf62741bf3cf6a (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260709 13:03:46.347823 11093 tablet_service.cc:1511] Processing CreateTablet for tablet 836251ebc2b74f5da88b8a8b2338e2a5 (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260709 13:03:46.348194 11148 ts_tablet_manager.cc:1434] T 4c2a35bb99374e93b573cf870d04faaa P 10ca5ee68ad64b3283b9d235352c9d23: Time spent starting tablet: real 0.140s	user 0.005s	sys 0.009s
I20260709 13:03:46.349216 11088 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d71ca3a1ba6b48478aaf62741bf3cf6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:46.345633 11014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e96e80ba9ac34d0998395515e5c8c41a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:46.352715 11093 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 836251ebc2b74f5da88b8a8b2338e2a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:46.353511 11148 tablet_bootstrap.cc:492] T 0f9df2bc281f4866b6da5da94a0defc7 P 10ca5ee68ad64b3283b9d235352c9d23: Bootstrap starting.
I20260709 13:03:46.361627 11152 tablet_bootstrap.cc:700] T 4f9eb06ed01f4ce68be3af8a8434665b P 0172fed22b6b4890a7fdc953fa128af4: Time spent opening tablet: real 0.130s	user 0.004s	sys 0.000s
I20260709 13:03:46.361827 11054 heartbeater.cc:499] Master 127.0.107.254:40861 was elected leader, sending a full tablet report...
I20260709 13:03:46.365553 11152 tablet_bootstrap.cc:654] T 4f9eb06ed01f4ce68be3af8a8434665b P 0172fed22b6b4890a7fdc953fa128af4: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:46.380496 11149 tablet_bootstrap.cc:492] T 4c2a35bb99374e93b573cf870d04faaa P 873357742d5843579556ee8166f23f47: No bootstrap required, opened a new log
I20260709 13:03:46.382023 11149 ts_tablet_manager.cc:1403] T 4c2a35bb99374e93b573cf870d04faaa P 873357742d5843579556ee8166f23f47: Time spent bootstrapping tablet: real 0.283s	user 0.015s	sys 0.005s
I20260709 13:03:46.399454 11149 raft_consensus.cc:359] T 4c2a35bb99374e93b573cf870d04faaa P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:46.401187 11149 raft_consensus.cc:385] T 4c2a35bb99374e93b573cf870d04faaa P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:46.401643 11149 raft_consensus.cc:740] T 4c2a35bb99374e93b573cf870d04faaa P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 873357742d5843579556ee8166f23f47, State: Initialized, Role: FOLLOWER
I20260709 13:03:46.406941 11149 consensus_queue.cc:260] T 4c2a35bb99374e93b573cf870d04faaa P 873357742d5843579556ee8166f23f47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
W20260709 13:03:46.409571 10763 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.0.107.195:45807, user_credentials={real_user=slave}} blocked reactor thread for 60375.7us
I20260709 13:03:46.426666 10866 tablet_service.cc:1511] Processing CreateTablet for tablet 836251ebc2b74f5da88b8a8b2338e2a5 (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260709 13:03:46.428874 10866 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 836251ebc2b74f5da88b8a8b2338e2a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:46.449218 11154 raft_consensus.cc:493] T 4c2a35bb99374e93b573cf870d04faaa P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:46.450295 11154 raft_consensus.cc:515] T 4c2a35bb99374e93b573cf870d04faaa P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:46.453616 10944 tablet_service.cc:1511] Processing CreateTablet for tablet 836251ebc2b74f5da88b8a8b2338e2a5 (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
W20260709 13:03:46.456336 10981 tablet.cc:2406] T 4c2a35bb99374e93b573cf870d04faaa P 10ca5ee68ad64b3283b9d235352c9d23: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260709 13:03:46.455542 10944 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 836251ebc2b74f5da88b8a8b2338e2a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:46.459086 11154 leader_election.cc:290] T 4c2a35bb99374e93b573cf870d04faaa P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:03:46.455960 11017 tablet_service.cc:1511] Processing CreateTablet for tablet 9a1716c0adf34bee8a4e99341ba5788f (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260709 13:03:46.465998 11017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a1716c0adf34bee8a4e99341ba5788f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:46.470276 11128 heartbeater.cc:499] Master 127.0.107.254:40861 was elected leader, sending a full tablet report...
I20260709 13:03:46.506908 11151 tablet_bootstrap.cc:492] T 4c2a35bb99374e93b573cf870d04faaa P 186b12638a87410487467df721a3cbda: No bootstrap required, opened a new log
I20260709 13:03:46.506887 11152 tablet_bootstrap.cc:492] T 4f9eb06ed01f4ce68be3af8a8434665b P 0172fed22b6b4890a7fdc953fa128af4: No bootstrap required, opened a new log
I20260709 13:03:46.508798 11152 ts_tablet_manager.cc:1403] T 4f9eb06ed01f4ce68be3af8a8434665b P 0172fed22b6b4890a7fdc953fa128af4: Time spent bootstrapping tablet: real 0.329s	user 0.020s	sys 0.036s
I20260709 13:03:46.518735 11151 ts_tablet_manager.cc:1403] T 4c2a35bb99374e93b573cf870d04faaa P 186b12638a87410487467df721a3cbda: Time spent bootstrapping tablet: real 0.363s	user 0.015s	sys 0.023s
I20260709 13:03:46.521746 11085 tablet_service.cc:1511] Processing CreateTablet for tablet 9a1716c0adf34bee8a4e99341ba5788f (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260709 13:03:46.523304 11152 raft_consensus.cc:359] T 4f9eb06ed01f4ce68be3af8a8434665b P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:46.524283 11152 raft_consensus.cc:385] T 4f9eb06ed01f4ce68be3af8a8434665b P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:46.524564 11085 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a1716c0adf34bee8a4e99341ba5788f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:46.524590 11152 raft_consensus.cc:740] T 4f9eb06ed01f4ce68be3af8a8434665b P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0172fed22b6b4890a7fdc953fa128af4, State: Initialized, Role: FOLLOWER
I20260709 13:03:46.528254 11149 ts_tablet_manager.cc:1434] T 4c2a35bb99374e93b573cf870d04faaa P 873357742d5843579556ee8166f23f47: Time spent starting tablet: real 0.146s	user 0.017s	sys 0.033s
I20260709 13:03:46.533982 11152 consensus_queue.cc:260] T 4f9eb06ed01f4ce68be3af8a8434665b P 0172fed22b6b4890a7fdc953fa128af4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:46.536877 11148 tablet_bootstrap.cc:654] T 0f9df2bc281f4866b6da5da94a0defc7 P 10ca5ee68ad64b3283b9d235352c9d23: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:46.543013 11149 tablet_bootstrap.cc:492] T 4f9eb06ed01f4ce68be3af8a8434665b P 873357742d5843579556ee8166f23f47: Bootstrap starting.
I20260709 13:03:46.553730 10865 tablet_service.cc:1511] Processing CreateTablet for tablet 9a1716c0adf34bee8a4e99341ba5788f (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260709 13:03:46.564916 11152 ts_tablet_manager.cc:1434] T 4f9eb06ed01f4ce68be3af8a8434665b P 0172fed22b6b4890a7fdc953fa128af4: Time spent starting tablet: real 0.056s	user 0.012s	sys 0.015s
I20260709 13:03:46.566941 11152 tablet_bootstrap.cc:492] T d7ce47b9a92d43dd88d95bcda5038597 P 0172fed22b6b4890a7fdc953fa128af4: Bootstrap starting.
I20260709 13:03:46.528524 11151 raft_consensus.cc:359] T 4c2a35bb99374e93b573cf870d04faaa P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:46.567771 11151 raft_consensus.cc:385] T 4c2a35bb99374e93b573cf870d04faaa P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:46.568176 11151 raft_consensus.cc:740] T 4c2a35bb99374e93b573cf870d04faaa P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 186b12638a87410487467df721a3cbda, State: Initialized, Role: FOLLOWER
I20260709 13:03:46.571130 10865 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a1716c0adf34bee8a4e99341ba5788f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:46.589900 11084 tablet_service.cc:1511] Processing CreateTablet for tablet a64b7e99d6c54934a7de6f6dffcf5e94 (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260709 13:03:46.594174 11084 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a64b7e99d6c54934a7de6f6dffcf5e94. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:46.606518 11162 raft_consensus.cc:493] T 4f9eb06ed01f4ce68be3af8a8434665b P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:46.607383 11162 raft_consensus.cc:515] T 4f9eb06ed01f4ce68be3af8a8434665b P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:46.612093 10940 tablet_service.cc:1511] Processing CreateTablet for tablet a64b7e99d6c54934a7de6f6dffcf5e94 (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260709 13:03:46.613379 11148 tablet_bootstrap.cc:492] T 0f9df2bc281f4866b6da5da94a0defc7 P 10ca5ee68ad64b3283b9d235352c9d23: No bootstrap required, opened a new log
I20260709 13:03:46.613785 10940 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a64b7e99d6c54934a7de6f6dffcf5e94. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:46.614110 11148 ts_tablet_manager.cc:1403] T 0f9df2bc281f4866b6da5da94a0defc7 P 10ca5ee68ad64b3283b9d235352c9d23: Time spent bootstrapping tablet: real 0.262s	user 0.014s	sys 0.011s
I20260709 13:03:46.618347 11148 raft_consensus.cc:359] T 0f9df2bc281f4866b6da5da94a0defc7 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:46.619422 11029 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c2a35bb99374e93b573cf870d04faaa" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:03:46.620429 11029 raft_consensus.cc:2468] T 4c2a35bb99374e93b573cf870d04faaa P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 0.
I20260709 13:03:46.621963 10919 leader_election.cc:304] T 4c2a35bb99374e93b573cf870d04faaa P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:03:46.623461 11154 raft_consensus.cc:2804] T 4c2a35bb99374e93b573cf870d04faaa P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:03:46.626658 11154 raft_consensus.cc:493] T 4c2a35bb99374e93b573cf870d04faaa P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:46.627049 11154 raft_consensus.cc:3060] T 4c2a35bb99374e93b573cf870d04faaa P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:03:46.627545 10941 tablet_service.cc:1511] Processing CreateTablet for tablet e89e640169e14e2d98f968f76027daa7 (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260709 13:03:46.629317 10941 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e89e640169e14e2d98f968f76027daa7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:46.629272 10880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c2a35bb99374e93b573cf870d04faaa" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:03:46.630088 10864 tablet_service.cc:1511] Processing CreateTablet for tablet e89e640169e14e2d98f968f76027daa7 (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260709 13:03:46.631675 10864 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e89e640169e14e2d98f968f76027daa7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:46.619403 11148 raft_consensus.cc:385] T 0f9df2bc281f4866b6da5da94a0defc7 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:46.633579 11148 raft_consensus.cc:740] T 0f9df2bc281f4866b6da5da94a0defc7 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10ca5ee68ad64b3283b9d235352c9d23, State: Initialized, Role: FOLLOWER
I20260709 13:03:46.635407 11162 leader_election.cc:290] T 4f9eb06ed01f4ce68be3af8a8434665b P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:03:46.593178 11151 consensus_queue.cc:260] T 4c2a35bb99374e93b573cf870d04faaa P 186b12638a87410487467df721a3cbda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:46.634761 11148 consensus_queue.cc:260] T 0f9df2bc281f4866b6da5da94a0defc7 P 10ca5ee68ad64b3283b9d235352c9d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:46.637915 10880 raft_consensus.cc:2468] T 4c2a35bb99374e93b573cf870d04faaa P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 0.
I20260709 13:03:46.645164 11015 tablet_service.cc:1511] Processing CreateTablet for tablet e89e640169e14e2d98f968f76027daa7 (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260709 13:03:46.646899 11015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e89e640169e14e2d98f968f76027daa7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:46.648474 11154 raft_consensus.cc:515] T 4c2a35bb99374e93b573cf870d04faaa P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:46.659307 10936 tablet_service.cc:1511] Processing CreateTablet for tablet 72b111ad7b5f4753ab1d2b5ee1114840 (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260709 13:03:46.661365 10936 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 72b111ad7b5f4753ab1d2b5ee1114840. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:46.665335 10866 tablet_service.cc:1511] Processing CreateTablet for tablet 72b111ad7b5f4753ab1d2b5ee1114840 (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260709 13:03:46.667132 10866 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 72b111ad7b5f4753ab1d2b5ee1114840. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:46.659406 11016 tablet_service.cc:1511] Processing CreateTablet for tablet 868e92339c2f4508af0afbd07c814d64 (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260709 13:03:46.691244 11149 tablet_bootstrap.cc:654] T 4f9eb06ed01f4ce68be3af8a8434665b P 873357742d5843579556ee8166f23f47: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:46.690830 11016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 868e92339c2f4508af0afbd07c814d64. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:46.699043 10869 tablet_service.cc:1511] Processing CreateTablet for tablet 868e92339c2f4508af0afbd07c814d64 (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260709 13:03:46.700754 10869 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 868e92339c2f4508af0afbd07c814d64. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:46.699425 11154 leader_election.cc:290] T 4c2a35bb99374e93b573cf870d04faaa P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 1 election: Requested vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:03:46.703861 10939 tablet_service.cc:1511] Processing CreateTablet for tablet 8f1466124dbd4f6ebb3c4d62ca48bf96 (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260709 13:03:46.705513 10939 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f1466124dbd4f6ebb3c4d62ca48bf96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:46.703873 11029 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c2a35bb99374e93b573cf870d04faaa" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
I20260709 13:03:46.707898 11029 raft_consensus.cc:3060] T 4c2a35bb99374e93b573cf870d04faaa P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:03:46.698983 11017 tablet_service.cc:1511] Processing CreateTablet for tablet a64b7e99d6c54934a7de6f6dffcf5e94 (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260709 13:03:46.714720 11017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a64b7e99d6c54934a7de6f6dffcf5e94. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:46.720468 11148 ts_tablet_manager.cc:1434] T 0f9df2bc281f4866b6da5da94a0defc7 P 10ca5ee68ad64b3283b9d235352c9d23: Time spent starting tablet: real 0.106s	user 0.011s	sys 0.011s
I20260709 13:03:46.714200 11152 tablet_bootstrap.cc:654] T d7ce47b9a92d43dd88d95bcda5038597 P 0172fed22b6b4890a7fdc953fa128af4: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:46.746001 11154 raft_consensus.cc:493] T 0f9df2bc281f4866b6da5da94a0defc7 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:46.752731 11148 tablet_bootstrap.cc:492] T 7839ca2d2c13431a85f44d7c458c6351 P 10ca5ee68ad64b3283b9d235352c9d23: Bootstrap starting.
I20260709 13:03:46.756031 10880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c2a35bb99374e93b573cf870d04faaa" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:03:46.758925 11014 tablet_service.cc:1511] Processing CreateTablet for tablet 8f1466124dbd4f6ebb3c4d62ca48bf96 (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260709 13:03:46.788467 10880 raft_consensus.cc:3060] T 4c2a35bb99374e93b573cf870d04faaa P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:03:46.789309 11014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f1466124dbd4f6ebb3c4d62ca48bf96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:46.759563 11029 raft_consensus.cc:2468] T 4c2a35bb99374e93b573cf870d04faaa P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 1.
I20260709 13:03:46.793406 10919 leader_election.cc:304] T 4c2a35bb99374e93b573cf870d04faaa P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:03:46.762683 11154 raft_consensus.cc:515] T 0f9df2bc281f4866b6da5da94a0defc7 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:46.771039 10870 tablet_service.cc:1511] Processing CreateTablet for tablet 2567c2d84e4648d7ba5dc23a3df32817 (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260709 13:03:46.803578 10870 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2567c2d84e4648d7ba5dc23a3df32817. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:46.804893 11154 leader_election.cc:290] T 0f9df2bc281f4866b6da5da94a0defc7 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:03:46.806888 11088 tablet_service.cc:1511] Processing CreateTablet for tablet 72b111ad7b5f4753ab1d2b5ee1114840 (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260709 13:03:46.808516 11088 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 72b111ad7b5f4753ab1d2b5ee1114840. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:46.809855 10879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f9df2bc281f4866b6da5da94a0defc7" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:03:46.785543 11028 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f9eb06ed01f4ce68be3af8a8434665b" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:03:46.813635 11012 tablet_service.cc:1511] Processing CreateTablet for tablet 2567c2d84e4648d7ba5dc23a3df32817 (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260709 13:03:46.787685 11151 ts_tablet_manager.cc:1434] T 4c2a35bb99374e93b573cf870d04faaa P 186b12638a87410487467df721a3cbda: Time spent starting tablet: real 0.268s	user 0.010s	sys 0.006s
I20260709 13:03:46.796540 11168 raft_consensus.cc:2804] T 4c2a35bb99374e93b573cf870d04faaa P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Leader election won for term 1
W20260709 13:03:46.815153 10921 leader_election.cc:343] T 0f9df2bc281f4866b6da5da94a0defc7 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 186b12638a87410487467df721a3cbda (127.0.107.193:38967): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:03:46.784978 10944 tablet_service.cc:1511] Processing CreateTablet for tablet 2567c2d84e4648d7ba5dc23a3df32817 (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260709 13:03:46.815764 11168 raft_consensus.cc:697] T 4c2a35bb99374e93b573cf870d04faaa P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 LEADER]: Becoming Leader. State: Replica: 10ca5ee68ad64b3283b9d235352c9d23, State: Running, Role: LEADER
I20260709 13:03:46.816036 11093 tablet_service.cc:1511] Processing CreateTablet for tablet 868e92339c2f4508af0afbd07c814d64 (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260709 13:03:46.800408 10880 raft_consensus.cc:2468] T 4c2a35bb99374e93b573cf870d04faaa P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 1.
I20260709 13:03:46.815378 11012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2567c2d84e4648d7ba5dc23a3df32817. 1 dirs total, 0 dirs full, 0 dirs failed
W20260709 13:03:46.821494 11067 leader_election.cc:343] T 4f9eb06ed01f4ce68be3af8a8434665b P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 873357742d5843579556ee8166f23f47 (127.0.107.195:45807): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:03:46.816983 10944 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2567c2d84e4648d7ba5dc23a3df32817. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:46.817535 11093 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 868e92339c2f4508af0afbd07c814d64. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:46.816994 11168 consensus_queue.cc:237] T 4c2a35bb99374e93b573cf870d04faaa P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:46.829790 11091 tablet_service.cc:1511] Processing CreateTablet for tablet 8f1466124dbd4f6ebb3c4d62ca48bf96 (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260709 13:03:46.832418 11151 tablet_bootstrap.cc:492] T c51782cd9d064ec0949e84e24dd4de67 P 186b12638a87410487467df721a3cbda: Bootstrap starting.
I20260709 13:03:46.845505 11089 tablet_service.cc:1511] Processing CreateTablet for tablet d270b9e5481245d9b5340a454a7cd10b (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260709 13:03:46.848146 10865 tablet_service.cc:1511] Processing CreateTablet for tablet d270b9e5481245d9b5340a454a7cd10b (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260709 13:03:46.849851 10865 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d270b9e5481245d9b5340a454a7cd10b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:46.834332 11148 tablet_bootstrap.cc:654] T 7839ca2d2c13431a85f44d7c458c6351 P 10ca5ee68ad64b3283b9d235352c9d23: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:46.832412 11152 tablet_bootstrap.cc:492] T d7ce47b9a92d43dd88d95bcda5038597 P 0172fed22b6b4890a7fdc953fa128af4: No bootstrap required, opened a new log
I20260709 13:03:46.855773 10941 tablet_service.cc:1511] Processing CreateTablet for tablet d270b9e5481245d9b5340a454a7cd10b (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260709 13:03:46.859953 10941 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d270b9e5481245d9b5340a454a7cd10b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:46.861065 11149 tablet_bootstrap.cc:492] T 4f9eb06ed01f4ce68be3af8a8434665b P 873357742d5843579556ee8166f23f47: No bootstrap required, opened a new log
I20260709 13:03:46.861747 11149 ts_tablet_manager.cc:1403] T 4f9eb06ed01f4ce68be3af8a8434665b P 873357742d5843579556ee8166f23f47: Time spent bootstrapping tablet: real 0.319s	user 0.020s	sys 0.014s
I20260709 13:03:46.866490 10880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f9eb06ed01f4ce68be3af8a8434665b" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
W20260709 13:03:46.868660 11069 leader_election.cc:343] T 4f9eb06ed01f4ce68be3af8a8434665b P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 186b12638a87410487467df721a3cbda (127.0.107.193:38967): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:03:46.869225 11069 leader_election.cc:304] T 4f9eb06ed01f4ce68be3af8a8434665b P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4; no voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47
I20260709 13:03:46.870193 11162 raft_consensus.cc:2749] T 4f9eb06ed01f4ce68be3af8a8434665b P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:03:46.874737 11089 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d270b9e5481245d9b5340a454a7cd10b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:46.875433 11149 raft_consensus.cc:359] T 4f9eb06ed01f4ce68be3af8a8434665b P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:46.876355 11149 raft_consensus.cc:385] T 4f9eb06ed01f4ce68be3af8a8434665b P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:46.876900 11149 raft_consensus.cc:740] T 4f9eb06ed01f4ce68be3af8a8434665b P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 873357742d5843579556ee8166f23f47, State: Initialized, Role: FOLLOWER
I20260709 13:03:46.885628 11018 tablet_service.cc:1511] Processing CreateTablet for tablet a610d1a71f52432e88fe56130f921fa2 (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260709 13:03:46.886770 11086 tablet_service.cc:1511] Processing CreateTablet for tablet a610d1a71f52432e88fe56130f921fa2 (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260709 13:03:46.887326 11018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a610d1a71f52432e88fe56130f921fa2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:46.899700 11091 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f1466124dbd4f6ebb3c4d62ca48bf96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:46.902107 10869 tablet_service.cc:1511] Processing CreateTablet for tablet a610d1a71f52432e88fe56130f921fa2 (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260709 13:03:46.902889 11086 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a610d1a71f52432e88fe56130f921fa2. 1 dirs total, 0 dirs full, 0 dirs failed
W20260709 13:03:46.903345 10826 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 13:03:46.903867 10869 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a610d1a71f52432e88fe56130f921fa2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:46.880861 11152 ts_tablet_manager.cc:1403] T d7ce47b9a92d43dd88d95bcda5038597 P 0172fed22b6b4890a7fdc953fa128af4: Time spent bootstrapping tablet: real 0.314s	user 0.016s	sys 0.005s
I20260709 13:03:46.908866 11152 raft_consensus.cc:359] T d7ce47b9a92d43dd88d95bcda5038597 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:46.909673 11152 raft_consensus.cc:385] T d7ce47b9a92d43dd88d95bcda5038597 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:46.909989 11152 raft_consensus.cc:740] T d7ce47b9a92d43dd88d95bcda5038597 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0172fed22b6b4890a7fdc953fa128af4, State: Initialized, Role: FOLLOWER
I20260709 13:03:46.913801 10940 tablet_service.cc:1511] Processing CreateTablet for tablet bac7e5040f984ba2b7146bd36af7267e (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260709 13:03:46.914606 11149 consensus_queue.cc:260] T 4f9eb06ed01f4ce68be3af8a8434665b P 873357742d5843579556ee8166f23f47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:46.920581 11149 ts_tablet_manager.cc:1434] T 4f9eb06ed01f4ce68be3af8a8434665b P 873357742d5843579556ee8166f23f47: Time spent starting tablet: real 0.058s	user 0.008s	sys 0.001s
I20260709 13:03:46.922320 11015 tablet_service.cc:1511] Processing CreateTablet for tablet bac7e5040f984ba2b7146bd36af7267e (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260709 13:03:46.923949 11015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bac7e5040f984ba2b7146bd36af7267e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:46.927788 10864 tablet_service.cc:1511] Processing CreateTablet for tablet 65bf0a62f4d84ef18dd6b23e082724b8 (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260709 13:03:46.914750 11151 tablet_bootstrap.cc:654] T c51782cd9d064ec0949e84e24dd4de67 P 186b12638a87410487467df721a3cbda: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:46.939289 10777 catalog_manager.cc:5697] T 4c2a35bb99374e93b573cf870d04faaa P 10ca5ee68ad64b3283b9d235352c9d23 reported cstate change: term changed from 0 to 1, leader changed from <none> to 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194). New cstate: current_term: 1 leader_uuid: "10ca5ee68ad64b3283b9d235352c9d23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } health_report { overall_health: UNKNOWN } } }
I20260709 13:03:46.942662 11149 tablet_bootstrap.cc:492] T 6a9353d1d230416296627d7cf8b1d425 P 873357742d5843579556ee8166f23f47: Bootstrap starting.
I20260709 13:03:46.944237 10936 tablet_service.cc:1511] Processing CreateTablet for tablet 65bf0a62f4d84ef18dd6b23e082724b8 (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260709 13:03:46.915539 10940 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bac7e5040f984ba2b7146bd36af7267e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:46.954713 11013 tablet_service.cc:1511] Processing CreateTablet for tablet 65bf0a62f4d84ef18dd6b23e082724b8 (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260709 13:03:46.965399 11103 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f9df2bc281f4866b6da5da94a0defc7" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
W20260709 13:03:46.967746 10920 leader_election.cc:343] T 0f9df2bc281f4866b6da5da94a0defc7 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:03:46.968325 10920 leader_election.cc:304] T 0f9df2bc281f4866b6da5da94a0defc7 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23; no voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda
I20260709 13:03:46.969923 11168 raft_consensus.cc:2749] T 0f9df2bc281f4866b6da5da94a0defc7 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:03:46.970863 11152 consensus_queue.cc:260] T d7ce47b9a92d43dd88d95bcda5038597 P 0172fed22b6b4890a7fdc953fa128af4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:46.986513 11085 tablet_service.cc:1511] Processing CreateTablet for tablet 5d439326f23747eaab3c1e7da2d6b785 (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260709 13:03:46.987468 11084 tablet_service.cc:1511] Processing CreateTablet for tablet bac7e5040f984ba2b7146bd36af7267e (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260709 13:03:46.988267 11085 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d439326f23747eaab3c1e7da2d6b785. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:46.992020 10870 tablet_service.cc:1511] Processing CreateTablet for tablet 5d439326f23747eaab3c1e7da2d6b785 (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260709 13:03:46.999702 11013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65bf0a62f4d84ef18dd6b23e082724b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:47.002851 10864 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65bf0a62f4d84ef18dd6b23e082724b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:47.013911 10941 tablet_service.cc:1511] Processing CreateTablet for tablet 5d439326f23747eaab3c1e7da2d6b785 (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260709 13:03:47.015856 10941 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d439326f23747eaab3c1e7da2d6b785. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:46.974352 11152 ts_tablet_manager.cc:1434] T d7ce47b9a92d43dd88d95bcda5038597 P 0172fed22b6b4890a7fdc953fa128af4: Time spent starting tablet: real 0.069s	user 0.006s	sys 0.000s
I20260709 13:03:47.022070 11152 tablet_bootstrap.cc:492] T 0f9df2bc281f4866b6da5da94a0defc7 P 0172fed22b6b4890a7fdc953fa128af4: Bootstrap starting.
I20260709 13:03:47.026203 10936 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65bf0a62f4d84ef18dd6b23e082724b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:47.048283 10870 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d439326f23747eaab3c1e7da2d6b785. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:47.051086 11149 tablet_bootstrap.cc:654] T 6a9353d1d230416296627d7cf8b1d425 P 873357742d5843579556ee8166f23f47: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:47.057659 11151 tablet_bootstrap.cc:492] T c51782cd9d064ec0949e84e24dd4de67 P 186b12638a87410487467df721a3cbda: No bootstrap required, opened a new log
I20260709 13:03:47.058715 11084 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bac7e5040f984ba2b7146bd36af7267e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:47.069887 11162 raft_consensus.cc:493] T d7ce47b9a92d43dd88d95bcda5038597 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:47.070705 11162 raft_consensus.cc:515] T d7ce47b9a92d43dd88d95bcda5038597 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:47.074916 10880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d7ce47b9a92d43dd88d95bcda5038597" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:03:47.074960 11162 leader_election.cc:290] T d7ce47b9a92d43dd88d95bcda5038597 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:03:47.078393 11018 tablet_service.cc:1511] Processing CreateTablet for tablet 555217d45d9b4ee492dbb3ddaf0f7c02 (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
W20260709 13:03:47.080343 11069 leader_election.cc:343] T d7ce47b9a92d43dd88d95bcda5038597 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 186b12638a87410487467df721a3cbda (127.0.107.193:38967): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:03:47.087963 11148 tablet_bootstrap.cc:492] T 7839ca2d2c13431a85f44d7c458c6351 P 10ca5ee68ad64b3283b9d235352c9d23: No bootstrap required, opened a new log
I20260709 13:03:47.058288 11151 ts_tablet_manager.cc:1403] T c51782cd9d064ec0949e84e24dd4de67 P 186b12638a87410487467df721a3cbda: Time spent bootstrapping tablet: real 0.226s	user 0.021s	sys 0.035s
I20260709 13:03:47.099841 11152 tablet_bootstrap.cc:654] T 0f9df2bc281f4866b6da5da94a0defc7 P 0172fed22b6b4890a7fdc953fa128af4: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:47.100595 11148 ts_tablet_manager.cc:1403] T 7839ca2d2c13431a85f44d7c458c6351 P 10ca5ee68ad64b3283b9d235352c9d23: Time spent bootstrapping tablet: real 0.356s	user 0.023s	sys 0.057s
I20260709 13:03:47.080453 11018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 555217d45d9b4ee492dbb3ddaf0f7c02. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:47.114897 11089 tablet_service.cc:1511] Processing CreateTablet for tablet 555217d45d9b4ee492dbb3ddaf0f7c02 (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260709 13:03:47.116542 11089 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 555217d45d9b4ee492dbb3ddaf0f7c02. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:47.124202 11151 raft_consensus.cc:359] T c51782cd9d064ec0949e84e24dd4de67 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:47.125120 11151 raft_consensus.cc:385] T c51782cd9d064ec0949e84e24dd4de67 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:47.125475 11151 raft_consensus.cc:740] T c51782cd9d064ec0949e84e24dd4de67 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 186b12638a87410487467df721a3cbda, State: Initialized, Role: FOLLOWER
I20260709 13:03:47.123901 11016 tablet_service.cc:1511] Processing CreateTablet for tablet b4cd3d7dfbbf4096908ff11e9dc57141 (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260709 13:03:47.126794 11016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4cd3d7dfbbf4096908ff11e9dc57141. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:47.127441 11149 tablet_bootstrap.cc:492] T 6a9353d1d230416296627d7cf8b1d425 P 873357742d5843579556ee8166f23f47: No bootstrap required, opened a new log
I20260709 13:03:47.128258 11149 ts_tablet_manager.cc:1403] T 6a9353d1d230416296627d7cf8b1d425 P 873357742d5843579556ee8166f23f47: Time spent bootstrapping tablet: real 0.186s	user 0.036s	sys 0.017s
I20260709 13:03:47.134491 10944 tablet_service.cc:1511] Processing CreateTablet for tablet b4cd3d7dfbbf4096908ff11e9dc57141 (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260709 13:03:47.134995 11088 tablet_service.cc:1511] Processing CreateTablet for tablet b4cd3d7dfbbf4096908ff11e9dc57141 (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260709 13:03:47.136315 10944 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4cd3d7dfbbf4096908ff11e9dc57141. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:47.136715 11088 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4cd3d7dfbbf4096908ff11e9dc57141. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:47.146420 11151 consensus_queue.cc:260] T c51782cd9d064ec0949e84e24dd4de67 P 186b12638a87410487467df721a3cbda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:47.139109 10869 tablet_service.cc:1511] Processing CreateTablet for tablet 555217d45d9b4ee492dbb3ddaf0f7c02 (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260709 13:03:47.145916 11148 raft_consensus.cc:359] T 7839ca2d2c13431a85f44d7c458c6351 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:47.159494 11148 raft_consensus.cc:385] T 7839ca2d2c13431a85f44d7c458c6351 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:47.159132 11149 raft_consensus.cc:359] T 6a9353d1d230416296627d7cf8b1d425 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:47.159929 11148 raft_consensus.cc:740] T 7839ca2d2c13431a85f44d7c458c6351 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10ca5ee68ad64b3283b9d235352c9d23, State: Initialized, Role: FOLLOWER
I20260709 13:03:47.160132 11149 raft_consensus.cc:385] T 6a9353d1d230416296627d7cf8b1d425 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:47.160537 11149 raft_consensus.cc:740] T 6a9353d1d230416296627d7cf8b1d425 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 873357742d5843579556ee8166f23f47, State: Initialized, Role: FOLLOWER
I20260709 13:03:47.161150 11148 consensus_queue.cc:260] T 7839ca2d2c13431a85f44d7c458c6351 P 10ca5ee68ad64b3283b9d235352c9d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:47.164546 10870 tablet_service.cc:1511] Processing CreateTablet for tablet 63d44f6ba6434ab2870fe3190f4d7c2f (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260709 13:03:47.167495 10939 tablet_service.cc:1511] Processing CreateTablet for tablet 63d44f6ba6434ab2870fe3190f4d7c2f (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260709 13:03:47.169572 10939 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 63d44f6ba6434ab2870fe3190f4d7c2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:47.170202 11015 tablet_service.cc:1511] Processing CreateTablet for tablet 63d44f6ba6434ab2870fe3190f4d7c2f (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260709 13:03:47.171355 10869 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 555217d45d9b4ee492dbb3ddaf0f7c02. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:47.173110 11015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 63d44f6ba6434ab2870fe3190f4d7c2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:47.175659 11151 ts_tablet_manager.cc:1434] T c51782cd9d064ec0949e84e24dd4de67 P 186b12638a87410487467df721a3cbda: Time spent starting tablet: real 0.077s	user 0.009s	sys 0.000s
I20260709 13:03:47.177145 11151 tablet_bootstrap.cc:492] T 7709387f4d3d4f9992d97a0fbca39a40 P 186b12638a87410487467df721a3cbda: Bootstrap starting.
I20260709 13:03:47.186671 11149 consensus_queue.cc:260] T 6a9353d1d230416296627d7cf8b1d425 P 873357742d5843579556ee8166f23f47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:47.187023 10955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d7ce47b9a92d43dd88d95bcda5038597" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
W20260709 13:03:47.189558 11066 leader_election.cc:343] T d7ce47b9a92d43dd88d95bcda5038597 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:03:47.190138 11066 leader_election.cc:304] T d7ce47b9a92d43dd88d95bcda5038597 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda
I20260709 13:03:47.191951 11162 raft_consensus.cc:2749] T d7ce47b9a92d43dd88d95bcda5038597 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:03:47.193747 10870 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 63d44f6ba6434ab2870fe3190f4d7c2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:47.194638 11091 tablet_service.cc:1511] Processing CreateTablet for tablet 5de5f0581e3b41dea729363d6ca563bb (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260709 13:03:47.204460 11152 tablet_bootstrap.cc:492] T 0f9df2bc281f4866b6da5da94a0defc7 P 0172fed22b6b4890a7fdc953fa128af4: No bootstrap required, opened a new log
I20260709 13:03:47.206135 11152 ts_tablet_manager.cc:1403] T 0f9df2bc281f4866b6da5da94a0defc7 P 0172fed22b6b4890a7fdc953fa128af4: Time spent bootstrapping tablet: real 0.184s	user 0.003s	sys 0.016s
I20260709 13:03:47.209666 10938 tablet_service.cc:1511] Processing CreateTablet for tablet 5de5f0581e3b41dea729363d6ca563bb (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260709 13:03:47.204631 10865 tablet_service.cc:1511] Processing CreateTablet for tablet 5de5f0581e3b41dea729363d6ca563bb (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260709 13:03:47.219820 10865 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5de5f0581e3b41dea729363d6ca563bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:47.213107 11152 raft_consensus.cc:359] T 0f9df2bc281f4866b6da5da94a0defc7 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:47.214367 11091 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5de5f0581e3b41dea729363d6ca563bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:47.226030 11152 raft_consensus.cc:385] T 0f9df2bc281f4866b6da5da94a0defc7 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:47.218611 10938 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5de5f0581e3b41dea729363d6ca563bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:47.240097 11151 tablet_bootstrap.cc:654] T 7709387f4d3d4f9992d97a0fbca39a40 P 186b12638a87410487467df721a3cbda: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:47.247061 11013 tablet_service.cc:1511] Processing CreateTablet for tablet 0e5d327630e5438b878035d28efe6c0c (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260709 13:03:47.248849 11013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e5d327630e5438b878035d28efe6c0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:47.221534 11149 ts_tablet_manager.cc:1434] T 6a9353d1d230416296627d7cf8b1d425 P 873357742d5843579556ee8166f23f47: Time spent starting tablet: real 0.093s	user 0.005s	sys 0.014s
I20260709 13:03:47.205454 11148 ts_tablet_manager.cc:1434] T 7839ca2d2c13431a85f44d7c458c6351 P 10ca5ee68ad64b3283b9d235352c9d23: Time spent starting tablet: real 0.093s	user 0.010s	sys 0.000s
I20260709 13:03:47.226425 11152 raft_consensus.cc:740] T 0f9df2bc281f4866b6da5da94a0defc7 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0172fed22b6b4890a7fdc953fa128af4, State: Initialized, Role: FOLLOWER
I20260709 13:03:47.255659 10864 tablet_service.cc:1511] Processing CreateTablet for tablet 0e5d327630e5438b878035d28efe6c0c (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260709 13:03:47.257380 10864 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e5d327630e5438b878035d28efe6c0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:47.256942 11152 consensus_queue.cc:260] T 0f9df2bc281f4866b6da5da94a0defc7 P 0172fed22b6b4890a7fdc953fa128af4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:47.255941 11148 tablet_bootstrap.cc:492] T 6f46a5102fd64b45bf1c4dff863d2b4d P 10ca5ee68ad64b3283b9d235352c9d23: Bootstrap starting.
I20260709 13:03:47.263885 11084 tablet_service.cc:1511] Processing CreateTablet for tablet 0e5d327630e5438b878035d28efe6c0c (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260709 13:03:47.265527 11084 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e5d327630e5438b878035d28efe6c0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:47.271185 11152 ts_tablet_manager.cc:1434] T 0f9df2bc281f4866b6da5da94a0defc7 P 0172fed22b6b4890a7fdc953fa128af4: Time spent starting tablet: real 0.065s	user 0.007s	sys 0.002s
I20260709 13:03:47.273062 11152 tablet_bootstrap.cc:492] T 6a9353d1d230416296627d7cf8b1d425 P 0172fed22b6b4890a7fdc953fa128af4: Bootstrap starting.
I20260709 13:03:47.274154 11158 raft_consensus.cc:493] T 4f9eb06ed01f4ce68be3af8a8434665b P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:47.274845 11158 raft_consensus.cc:515] T 4f9eb06ed01f4ce68be3af8a8434665b P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:47.260874 11149 tablet_bootstrap.cc:492] T 525bfdbd6d934afd823f7d3ee3f9d052 P 873357742d5843579556ee8166f23f47: Bootstrap starting.
I20260709 13:03:47.281579 11158 leader_election.cc:290] T 4f9eb06ed01f4ce68be3af8a8434665b P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:03:47.301136 11018 tablet_service.cc:1511] Processing CreateTablet for tablet 32147c6d2d5a477aa38c597c7f78df5b (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260709 13:03:47.303146 11148 tablet_bootstrap.cc:654] T 6f46a5102fd64b45bf1c4dff863d2b4d P 10ca5ee68ad64b3283b9d235352c9d23: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:47.319259 10939 tablet_service.cc:1511] Processing CreateTablet for tablet 32147c6d2d5a477aa38c597c7f78df5b (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260709 13:03:47.320894 10939 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32147c6d2d5a477aa38c597c7f78df5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:47.320163 11088 tablet_service.cc:1511] Processing CreateTablet for tablet 32147c6d2d5a477aa38c597c7f78df5b (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260709 13:03:47.338613 10940 tablet_service.cc:1511] Processing CreateTablet for tablet 11de93cd9b9d457489ff1cb80f976c9a (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260709 13:03:47.340314 10940 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11de93cd9b9d457489ff1cb80f976c9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:47.341871 10866 tablet_service.cc:1511] Processing CreateTablet for tablet 11de93cd9b9d457489ff1cb80f976c9a (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260709 13:03:47.343339 10866 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11de93cd9b9d457489ff1cb80f976c9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:47.347983 11168 consensus_queue.cc:1048] T 4c2a35bb99374e93b573cf870d04faaa P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:03:47.356241 11088 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32147c6d2d5a477aa38c597c7f78df5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:47.369304 11018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32147c6d2d5a477aa38c597c7f78df5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:47.374672 11152 tablet_bootstrap.cc:654] T 6a9353d1d230416296627d7cf8b1d425 P 0172fed22b6b4890a7fdc953fa128af4: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:47.376703 11086 tablet_service.cc:1511] Processing CreateTablet for tablet 87cd8f5097624c8692f1d83544d8881e (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260709 13:03:47.378525 11016 tablet_service.cc:1511] Processing CreateTablet for tablet 11de93cd9b9d457489ff1cb80f976c9a (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260709 13:03:47.380364 11016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11de93cd9b9d457489ff1cb80f976c9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:47.380729 11086 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 87cd8f5097624c8692f1d83544d8881e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:47.386102 10867 tablet_service.cc:1511] Processing CreateTablet for tablet 87cd8f5097624c8692f1d83544d8881e (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260709 13:03:47.387894 10867 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 87cd8f5097624c8692f1d83544d8881e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:47.415729 10936 tablet_service.cc:1511] Processing CreateTablet for tablet 87cd8f5097624c8692f1d83544d8881e (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260709 13:03:47.417482 10936 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 87cd8f5097624c8692f1d83544d8881e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:47.423398 11158 raft_consensus.cc:493] T 6a9353d1d230416296627d7cf8b1d425 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:47.424086 11158 raft_consensus.cc:515] T 6a9353d1d230416296627d7cf8b1d425 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:47.427256 11158 leader_election.cc:290] T 6a9353d1d230416296627d7cf8b1d425 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:03:47.428951 11151 tablet_bootstrap.cc:492] T 7709387f4d3d4f9992d97a0fbca39a40 P 186b12638a87410487467df721a3cbda: No bootstrap required, opened a new log
I20260709 13:03:47.429630 11151 ts_tablet_manager.cc:1403] T 7709387f4d3d4f9992d97a0fbca39a40 P 186b12638a87410487467df721a3cbda: Time spent bootstrapping tablet: real 0.253s	user 0.022s	sys 0.024s
I20260709 13:03:47.431936 11148 tablet_bootstrap.cc:492] T 6f46a5102fd64b45bf1c4dff863d2b4d P 10ca5ee68ad64b3283b9d235352c9d23: No bootstrap required, opened a new log
I20260709 13:03:47.444097 11170 raft_consensus.cc:493] T c51782cd9d064ec0949e84e24dd4de67 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:47.447089 11170 raft_consensus.cc:515] T c51782cd9d064ec0949e84e24dd4de67 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:47.453307 11015 tablet_service.cc:1511] Processing CreateTablet for tablet d371c5e27bd843469d2e2e453734cb4a (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260709 13:03:47.436687 11149 tablet_bootstrap.cc:654] T 525bfdbd6d934afd823f7d3ee3f9d052 P 873357742d5843579556ee8166f23f47: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:47.455428 11015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d371c5e27bd843469d2e2e453734cb4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:47.456223 11148 ts_tablet_manager.cc:1403] T 6f46a5102fd64b45bf1c4dff863d2b4d P 10ca5ee68ad64b3283b9d235352c9d23: Time spent bootstrapping tablet: real 0.201s	user 0.021s	sys 0.010s
I20260709 13:03:47.463219 11151 raft_consensus.cc:359] T 7709387f4d3d4f9992d97a0fbca39a40 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:47.464118 11151 raft_consensus.cc:385] T 7709387f4d3d4f9992d97a0fbca39a40 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:47.464457 11151 raft_consensus.cc:740] T 7709387f4d3d4f9992d97a0fbca39a40 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 186b12638a87410487467df721a3cbda, State: Initialized, Role: FOLLOWER
I20260709 13:03:47.472777 11151 consensus_queue.cc:260] T 7709387f4d3d4f9992d97a0fbca39a40 P 186b12638a87410487467df721a3cbda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:47.475911 11152 tablet_bootstrap.cc:492] T 6a9353d1d230416296627d7cf8b1d425 P 0172fed22b6b4890a7fdc953fa128af4: No bootstrap required, opened a new log
I20260709 13:03:47.476754 11152 ts_tablet_manager.cc:1403] T 6a9353d1d230416296627d7cf8b1d425 P 0172fed22b6b4890a7fdc953fa128af4: Time spent bootstrapping tablet: real 0.204s	user 0.012s	sys 0.014s
I20260709 13:03:47.484089 11152 raft_consensus.cc:359] T 6a9353d1d230416296627d7cf8b1d425 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:47.492242 11151 ts_tablet_manager.cc:1434] T 7709387f4d3d4f9992d97a0fbca39a40 P 186b12638a87410487467df721a3cbda: Time spent starting tablet: real 0.062s	user 0.011s	sys 0.007s
I20260709 13:03:47.468358 11170 leader_election.cc:290] T c51782cd9d064ec0949e84e24dd4de67 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:03:47.507920 11091 tablet_service.cc:1511] Processing CreateTablet for tablet d371c5e27bd843469d2e2e453734cb4a (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260709 13:03:47.509392 11102 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a9353d1d230416296627d7cf8b1d425" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:03:47.509891 11103 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f9eb06ed01f4ce68be3af8a8434665b" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:03:47.533476 11103 raft_consensus.cc:2468] T 4f9eb06ed01f4ce68be3af8a8434665b P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 0.
I20260709 13:03:47.533834 11091 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d371c5e27bd843469d2e2e453734cb4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:47.515981 10879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f9eb06ed01f4ce68be3af8a8434665b" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:03:47.516785 10878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a9353d1d230416296627d7cf8b1d425" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:03:47.495862 11162 raft_consensus.cc:493] T 0f9df2bc281f4866b6da5da94a0defc7 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:47.536093 11162 raft_consensus.cc:515] T 0f9df2bc281f4866b6da5da94a0defc7 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:47.538252 10936 tablet_service.cc:1511] Processing CreateTablet for tablet bb7d67be500947b3aef4c4245e33671c (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260709 13:03:47.539798 10936 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb7d67be500947b3aef4c4245e33671c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:47.539675 10955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f9df2bc281f4866b6da5da94a0defc7" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
W20260709 13:03:47.540125 10995 leader_election.cc:343] T 4f9eb06ed01f4ce68be3af8a8434665b P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 186b12638a87410487467df721a3cbda (127.0.107.193:38967): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:03:47.540463 10955 raft_consensus.cc:2468] T 0f9df2bc281f4866b6da5da94a0defc7 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 0.
W20260709 13:03:47.541047 10995 leader_election.cc:343] T 6a9353d1d230416296627d7cf8b1d425 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 186b12638a87410487467df721a3cbda (127.0.107.193:38967): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:03:47.526280 11189 consensus_queue.cc:1048] T 4c2a35bb99374e93b573cf870d04faaa P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:03:47.541989 11066 leader_election.cc:304] T 0f9df2bc281f4866b6da5da94a0defc7 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 10ca5ee68ad64b3283b9d235352c9d23; no voters: 
I20260709 13:03:47.526677 10866 tablet_service.cc:1511] Processing CreateTablet for tablet d371c5e27bd843469d2e2e453734cb4a (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260709 13:03:47.532563 11151 tablet_bootstrap.cc:492] T 57b6a2418c1342e7924195e9c1dbd357 P 186b12638a87410487467df721a3cbda: Bootstrap starting.
I20260709 13:03:47.529845 11018 tablet_service.cc:1511] Processing CreateTablet for tablet bb7d67be500947b3aef4c4245e33671c (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260709 13:03:47.494478 11152 raft_consensus.cc:385] T 6a9353d1d230416296627d7cf8b1d425 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:47.464347 11148 raft_consensus.cc:359] T 6f46a5102fd64b45bf1c4dff863d2b4d P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:47.554252 11148 raft_consensus.cc:385] T 6f46a5102fd64b45bf1c4dff863d2b4d P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:47.554695 11148 raft_consensus.cc:740] T 6f46a5102fd64b45bf1c4dff863d2b4d P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10ca5ee68ad64b3283b9d235352c9d23, State: Initialized, Role: FOLLOWER
I20260709 13:03:47.548097 10994 leader_election.cc:304] T 4f9eb06ed01f4ce68be3af8a8434665b P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47; no voters: 186b12638a87410487467df721a3cbda
I20260709 13:03:47.550894 11018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb7d67be500947b3aef4c4245e33671c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:47.556563 11152 raft_consensus.cc:740] T 6a9353d1d230416296627d7cf8b1d425 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0172fed22b6b4890a7fdc953fa128af4, State: Initialized, Role: FOLLOWER
I20260709 13:03:47.557369 11189 raft_consensus.cc:493] T 7839ca2d2c13431a85f44d7c458c6351 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:47.554486 10865 tablet_service.cc:1511] Processing CreateTablet for tablet 732c4adeeff6422497aa7db4a82f535b (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260709 13:03:47.568667 11162 leader_election.cc:290] T 0f9df2bc281f4866b6da5da94a0defc7 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:03:47.571092 10866 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d371c5e27bd843469d2e2e453734cb4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:47.574913 11158 raft_consensus.cc:2804] T 4f9eb06ed01f4ce68be3af8a8434665b P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:03:47.576875 11189 raft_consensus.cc:515] T 7839ca2d2c13431a85f44d7c458c6351 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:47.584947 10944 tablet_service.cc:1511] Processing CreateTablet for tablet 732c4adeeff6422497aa7db4a82f535b (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260709 13:03:47.585301 11088 tablet_service.cc:1511] Processing CreateTablet for tablet bb7d67be500947b3aef4c4245e33671c (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260709 13:03:47.586973 11088 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb7d67be500947b3aef4c4245e33671c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:47.588722 10879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7839ca2d2c13431a85f44d7c458c6351" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:03:47.569083 11148 consensus_queue.cc:260] T 6f46a5102fd64b45bf1c4dff863d2b4d P 10ca5ee68ad64b3283b9d235352c9d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:47.593029 11013 tablet_service.cc:1511] Processing CreateTablet for tablet 732c4adeeff6422497aa7db4a82f535b (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260709 13:03:47.594614 11013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 732c4adeeff6422497aa7db4a82f535b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:47.596448 11029 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7839ca2d2c13431a85f44d7c458c6351" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
W20260709 13:03:47.598121 10919 leader_election.cc:343] T 7839ca2d2c13431a85f44d7c458c6351 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 873357742d5843579556ee8166f23f47 (127.0.107.195:45807): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:03:47.598979 11189 leader_election.cc:290] T 7839ca2d2c13431a85f44d7c458c6351 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:03:47.582989 11158 raft_consensus.cc:493] T 4f9eb06ed01f4ce68be3af8a8434665b P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:47.601014 11158 raft_consensus.cc:3060] T 4f9eb06ed01f4ce68be3af8a8434665b P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:03:47.602046 11148 ts_tablet_manager.cc:1434] T 6f46a5102fd64b45bf1c4dff863d2b4d P 10ca5ee68ad64b3283b9d235352c9d23: Time spent starting tablet: real 0.144s	user 0.008s	sys 0.000s
I20260709 13:03:47.602221 10944 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 732c4adeeff6422497aa7db4a82f535b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:47.603466 11148 tablet_bootstrap.cc:492] T 525bfdbd6d934afd823f7d3ee3f9d052 P 10ca5ee68ad64b3283b9d235352c9d23: Bootstrap starting.
I20260709 13:03:47.613729 11148 tablet_bootstrap.cc:654] T 525bfdbd6d934afd823f7d3ee3f9d052 P 10ca5ee68ad64b3283b9d235352c9d23: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:47.608134 11152 consensus_queue.cc:260] T 6a9353d1d230416296627d7cf8b1d425 P 0172fed22b6b4890a7fdc953fa128af4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:47.615383 11102 raft_consensus.cc:2468] T 6a9353d1d230416296627d7cf8b1d425 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 0.
I20260709 13:03:47.618609 11152 ts_tablet_manager.cc:1434] T 6a9353d1d230416296627d7cf8b1d425 P 0172fed22b6b4890a7fdc953fa128af4: Time spent starting tablet: real 0.141s	user 0.014s	sys 0.000s
I20260709 13:03:47.619235 10865 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 732c4adeeff6422497aa7db4a82f535b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:47.623265 11089 tablet_service.cc:1511] Processing CreateTablet for tablet e5a5f5a608504650b48fd27f163faed6 (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260709 13:03:47.627298 11089 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5a5f5a608504650b48fd27f163faed6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:47.579267 11162 raft_consensus.cc:2804] T 0f9df2bc281f4866b6da5da94a0defc7 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:03:47.630825 11189 raft_consensus.cc:493] T 6f46a5102fd64b45bf1c4dff863d2b4d P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:47.640307 11189 raft_consensus.cc:515] T 6f46a5102fd64b45bf1c4dff863d2b4d P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:47.640771 10879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f9df2bc281f4866b6da5da94a0defc7" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:03:47.642916 11189 leader_election.cc:290] T 6f46a5102fd64b45bf1c4dff863d2b4d P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
W20260709 13:03:47.633142 10921 leader_election.cc:343] T 7839ca2d2c13431a85f44d7c458c6351 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 186b12638a87410487467df721a3cbda (127.0.107.193:38967): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:03:47.643739 10921 leader_election.cc:304] T 7839ca2d2c13431a85f44d7c458c6351 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23; no voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47
I20260709 13:03:47.644038 11028 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f46a5102fd64b45bf1c4dff863d2b4d" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:03:47.633383 11162 raft_consensus.cc:493] T 0f9df2bc281f4866b6da5da94a0defc7 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:47.645341 11162 raft_consensus.cc:3060] T 0f9df2bc281f4866b6da5da94a0defc7 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:03:47.646728 11102 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f46a5102fd64b45bf1c4dff863d2b4d" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
W20260709 13:03:47.648486 11069 leader_election.cc:343] T 0f9df2bc281f4866b6da5da94a0defc7 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 186b12638a87410487467df721a3cbda (127.0.107.193:38967): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:03:47.649454 10867 tablet_service.cc:1511] Processing CreateTablet for tablet e5a5f5a608504650b48fd27f163faed6 (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260709 13:03:47.650605 11189 raft_consensus.cc:2749] T 7839ca2d2c13431a85f44d7c458c6351 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260709 13:03:47.650640 10920 leader_election.cc:343] T 6f46a5102fd64b45bf1c4dff863d2b4d P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:03:47.651084 10867 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5a5f5a608504650b48fd27f163faed6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:47.639268 11152 tablet_bootstrap.cc:492] T 6f46a5102fd64b45bf1c4dff863d2b4d P 0172fed22b6b4890a7fdc953fa128af4: Bootstrap starting.
W20260709 13:03:47.661804 10919 leader_election.cc:343] T 6f46a5102fd64b45bf1c4dff863d2b4d P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 873357742d5843579556ee8166f23f47 (127.0.107.195:45807): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:03:47.662591 10919 leader_election.cc:304] T 6f46a5102fd64b45bf1c4dff863d2b4d P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23; no voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47
I20260709 13:03:47.665705 11189 raft_consensus.cc:2749] T 6f46a5102fd64b45bf1c4dff863d2b4d P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:03:47.677762 10955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c51782cd9d064ec0949e84e24dd4de67" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:03:47.680249 11102 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c51782cd9d064ec0949e84e24dd4de67" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
W20260709 13:03:47.682298 10843 leader_election.cc:343] T c51782cd9d064ec0949e84e24dd4de67 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:03:47.682737 11158 raft_consensus.cc:515] T 4f9eb06ed01f4ce68be3af8a8434665b P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
W20260709 13:03:47.683990 10845 leader_election.cc:343] T c51782cd9d064ec0949e84e24dd4de67 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:03:47.685262 10845 leader_election.cc:304] T c51782cd9d064ec0949e84e24dd4de67 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 186b12638a87410487467df721a3cbda; no voters: 0172fed22b6b4890a7fdc953fa128af4, 10ca5ee68ad64b3283b9d235352c9d23
I20260709 13:03:47.685767 11151 tablet_bootstrap.cc:654] T 57b6a2418c1342e7924195e9c1dbd357 P 186b12638a87410487467df721a3cbda: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:47.685356 10938 tablet_service.cc:1511] Processing CreateTablet for tablet e5a5f5a608504650b48fd27f163faed6 (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260709 13:03:47.695117 10938 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5a5f5a608504650b48fd27f163faed6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:47.686939 10879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f9eb06ed01f4ce68be3af8a8434665b" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:03:47.691149 11016 tablet_service.cc:1511] Processing CreateTablet for tablet f371d2565dd84667b5613848604949bb (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260709 13:03:47.692229 11084 tablet_service.cc:1511] Processing CreateTablet for tablet f371d2565dd84667b5613848604949bb (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260709 13:03:47.699352 11170 raft_consensus.cc:2749] T c51782cd9d064ec0949e84e24dd4de67 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:03:47.700013 11084 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f371d2565dd84667b5613848604949bb. 1 dirs total, 0 dirs full, 0 dirs failed
W20260709 13:03:47.703759 10995 leader_election.cc:343] T 4f9eb06ed01f4ce68be3af8a8434665b P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 186b12638a87410487467df721a3cbda (127.0.107.193:38967): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:03:47.712692 11162 raft_consensus.cc:515] T 0f9df2bc281f4866b6da5da94a0defc7 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:47.721776 10940 tablet_service.cc:1511] Processing CreateTablet for tablet a154cc8b181f460abf4880064379948b (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260709 13:03:47.723371 10940 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a154cc8b181f460abf4880064379948b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:47.725350 11016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f371d2565dd84667b5613848604949bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:47.728300 11015 tablet_service.cc:1511] Processing CreateTablet for tablet a154cc8b181f460abf4880064379948b (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260709 13:03:47.731489 10864 tablet_service.cc:1511] Processing CreateTablet for tablet f371d2565dd84667b5613848604949bb (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260709 13:03:47.732954 10864 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f371d2565dd84667b5613848604949bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:47.733803 11015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a154cc8b181f460abf4880064379948b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:47.739737 11152 tablet_bootstrap.cc:654] T 6f46a5102fd64b45bf1c4dff863d2b4d P 0172fed22b6b4890a7fdc953fa128af4: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:47.747948 10879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f9df2bc281f4866b6da5da94a0defc7" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
W20260709 13:03:47.749620 11069 leader_election.cc:343] T 0f9df2bc281f4866b6da5da94a0defc7 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 186b12638a87410487467df721a3cbda (127.0.107.193:38967): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:03:47.749944 11149 tablet_bootstrap.cc:492] T 525bfdbd6d934afd823f7d3ee3f9d052 P 873357742d5843579556ee8166f23f47: No bootstrap required, opened a new log
I20260709 13:03:47.750771 11149 ts_tablet_manager.cc:1403] T 525bfdbd6d934afd823f7d3ee3f9d052 P 873357742d5843579556ee8166f23f47: Time spent bootstrapping tablet: real 0.491s	user 0.012s	sys 0.012s
I20260709 13:03:47.752002 11148 tablet_bootstrap.cc:492] T 525bfdbd6d934afd823f7d3ee3f9d052 P 10ca5ee68ad64b3283b9d235352c9d23: No bootstrap required, opened a new log
I20260709 13:03:47.752592 11148 ts_tablet_manager.cc:1403] T 525bfdbd6d934afd823f7d3ee3f9d052 P 10ca5ee68ad64b3283b9d235352c9d23: Time spent bootstrapping tablet: real 0.149s	user 0.033s	sys 0.016s
I20260709 13:03:47.755033 10955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f9df2bc281f4866b6da5da94a0defc7" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:03:47.755964 10955 raft_consensus.cc:3060] T 0f9df2bc281f4866b6da5da94a0defc7 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:03:47.759979 11162 leader_election.cc:290] T 0f9df2bc281f4866b6da5da94a0defc7 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 1 election: Requested vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:03:47.685473 11158 leader_election.cc:290] T 4f9eb06ed01f4ce68be3af8a8434665b P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 1 election: Requested vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:03:47.767585 11149 raft_consensus.cc:359] T 525bfdbd6d934afd823f7d3ee3f9d052 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:47.770239 11149 raft_consensus.cc:385] T 525bfdbd6d934afd823f7d3ee3f9d052 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:47.775060 11149 raft_consensus.cc:740] T 525bfdbd6d934afd823f7d3ee3f9d052 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 873357742d5843579556ee8166f23f47, State: Initialized, Role: FOLLOWER
I20260709 13:03:47.776530 11149 consensus_queue.cc:260] T 525bfdbd6d934afd823f7d3ee3f9d052 P 873357742d5843579556ee8166f23f47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
W20260709 13:03:47.779042 10994 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 231231us
I20260709 13:03:47.781160 11102 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f9eb06ed01f4ce68be3af8a8434665b" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:03:47.781769 11102 raft_consensus.cc:3060] T 4f9eb06ed01f4ce68be3af8a8434665b P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:03:47.785574 11149 ts_tablet_manager.cc:1434] T 525bfdbd6d934afd823f7d3ee3f9d052 P 873357742d5843579556ee8166f23f47: Time spent starting tablet: real 0.034s	user 0.009s	sys 0.000s
I20260709 13:03:47.792708 11102 raft_consensus.cc:2468] T 4f9eb06ed01f4ce68be3af8a8434665b P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 1.
I20260709 13:03:47.794482 11148 raft_consensus.cc:359] T 525bfdbd6d934afd823f7d3ee3f9d052 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:47.802770 11148 raft_consensus.cc:385] T 525bfdbd6d934afd823f7d3ee3f9d052 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:47.803074 11148 raft_consensus.cc:740] T 525bfdbd6d934afd823f7d3ee3f9d052 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10ca5ee68ad64b3283b9d235352c9d23, State: Initialized, Role: FOLLOWER
I20260709 13:03:47.803885 11148 consensus_queue.cc:260] T 525bfdbd6d934afd823f7d3ee3f9d052 P 10ca5ee68ad64b3283b9d235352c9d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:47.795642 10994 leader_election.cc:304] T 6a9353d1d230416296627d7cf8b1d425 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47; no voters: 186b12638a87410487467df721a3cbda
I20260709 13:03:47.806414 11148 ts_tablet_manager.cc:1434] T 525bfdbd6d934afd823f7d3ee3f9d052 P 10ca5ee68ad64b3283b9d235352c9d23: Time spent starting tablet: real 0.054s	user 0.009s	sys 0.000s
I20260709 13:03:47.814519 11158 raft_consensus.cc:2804] T 6a9353d1d230416296627d7cf8b1d425 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:03:47.815794 11158 raft_consensus.cc:493] T 6a9353d1d230416296627d7cf8b1d425 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:47.816264 11158 raft_consensus.cc:3060] T 6a9353d1d230416296627d7cf8b1d425 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:03:47.816365 11148 tablet_bootstrap.cc:492] T d7ce47b9a92d43dd88d95bcda5038597 P 10ca5ee68ad64b3283b9d235352c9d23: Bootstrap starting.
I20260709 13:03:47.798748 10955 raft_consensus.cc:2468] T 0f9df2bc281f4866b6da5da94a0defc7 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 1.
I20260709 13:03:47.807299 11149 tablet_bootstrap.cc:492] T 57b6a2418c1342e7924195e9c1dbd357 P 873357742d5843579556ee8166f23f47: Bootstrap starting.
I20260709 13:03:47.815040 10994 leader_election.cc:304] T 4f9eb06ed01f4ce68be3af8a8434665b P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47; no voters: 186b12638a87410487467df721a3cbda
I20260709 13:03:47.843863 11198 raft_consensus.cc:2804] T 4f9eb06ed01f4ce68be3af8a8434665b P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:03:47.835189 11066 leader_election.cc:304] T 0f9df2bc281f4866b6da5da94a0defc7 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 10ca5ee68ad64b3283b9d235352c9d23; no voters: 186b12638a87410487467df721a3cbda
I20260709 13:03:47.848320 11196 raft_consensus.cc:2804] T 0f9df2bc281f4866b6da5da94a0defc7 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:03:47.848888 11196 raft_consensus.cc:697] T 0f9df2bc281f4866b6da5da94a0defc7 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 LEADER]: Becoming Leader. State: Replica: 0172fed22b6b4890a7fdc953fa128af4, State: Running, Role: LEADER
I20260709 13:03:47.850692 11086 tablet_service.cc:1511] Processing CreateTablet for tablet a154cc8b181f460abf4880064379948b (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260709 13:03:47.852206 11086 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a154cc8b181f460abf4880064379948b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:47.857919 11149 tablet_bootstrap.cc:654] T 57b6a2418c1342e7924195e9c1dbd357 P 873357742d5843579556ee8166f23f47: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:47.849885 11196 consensus_queue.cc:237] T 0f9df2bc281f4866b6da5da94a0defc7 P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:47.872879 11170 raft_consensus.cc:493] T 7709387f4d3d4f9992d97a0fbca39a40 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:47.873679 11170 raft_consensus.cc:515] T 7709387f4d3d4f9992d97a0fbca39a40 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:47.883186 11170 leader_election.cc:290] T 7709387f4d3d4f9992d97a0fbca39a40 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:03:47.884461 11102 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7709387f4d3d4f9992d97a0fbca39a40" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
W20260709 13:03:47.886358 10845 leader_election.cc:343] T 7709387f4d3d4f9992d97a0fbca39a40 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:03:47.889598 11158 raft_consensus.cc:515] T 6a9353d1d230416296627d7cf8b1d425 P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:47.904577 10867 tablet_service.cc:1511] Processing CreateTablet for tablet 4b37f8fca8dc42718b2a42f0883a956f (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260709 13:03:47.906301 10867 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b37f8fca8dc42718b2a42f0883a956f. 1 dirs total, 0 dirs full, 0 dirs failed
W20260709 13:03:47.907013 10826 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 13:03:47.914084 11151 tablet_bootstrap.cc:492] T 57b6a2418c1342e7924195e9c1dbd357 P 186b12638a87410487467df721a3cbda: No bootstrap required, opened a new log
I20260709 13:03:47.913861 10936 tablet_service.cc:1511] Processing CreateTablet for tablet 4b37f8fca8dc42718b2a42f0883a956f (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260709 13:03:47.924315 10936 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b37f8fca8dc42718b2a42f0883a956f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:47.925117 10775 catalog_manager.cc:5697] T 0f9df2bc281f4866b6da5da94a0defc7 P 0172fed22b6b4890a7fdc953fa128af4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196). New cstate: current_term: 1 leader_uuid: "0172fed22b6b4890a7fdc953fa128af4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } health_report { overall_health: UNKNOWN } } }
I20260709 13:03:47.935514 11013 tablet_service.cc:1511] Processing CreateTablet for tablet 4b37f8fca8dc42718b2a42f0883a956f (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260709 13:03:47.931378 11151 ts_tablet_manager.cc:1403] T 57b6a2418c1342e7924195e9c1dbd357 P 186b12638a87410487467df721a3cbda: Time spent bootstrapping tablet: real 0.399s	user 0.016s	sys 0.021s
I20260709 13:03:47.935514 11148 tablet_bootstrap.cc:654] T d7ce47b9a92d43dd88d95bcda5038597 P 10ca5ee68ad64b3283b9d235352c9d23: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:47.939393 11198 raft_consensus.cc:697] T 4f9eb06ed01f4ce68be3af8a8434665b P 873357742d5843579556ee8166f23f47 [term 1 LEADER]: Becoming Leader. State: Replica: 873357742d5843579556ee8166f23f47, State: Running, Role: LEADER
I20260709 13:03:47.944696 11013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b37f8fca8dc42718b2a42f0883a956f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:47.944833 11198 consensus_queue.cc:237] T 4f9eb06ed01f4ce68be3af8a8434665b P 873357742d5843579556ee8166f23f47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:47.952728 11086 tablet_service.cc:1511] Processing CreateTablet for tablet 5d156d6ae2ca4fa59a1e387edf0fea39 (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260709 13:03:47.954129 11086 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d156d6ae2ca4fa59a1e387edf0fea39. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:47.956583 11158 leader_election.cc:290] T 6a9353d1d230416296627d7cf8b1d425 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 1 election: Requested vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:03:47.957754 10878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a9353d1d230416296627d7cf8b1d425" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:03:47.959523 11152 tablet_bootstrap.cc:492] T 6f46a5102fd64b45bf1c4dff863d2b4d P 0172fed22b6b4890a7fdc953fa128af4: No bootstrap required, opened a new log
W20260709 13:03:47.959589 10995 leader_election.cc:343] T 6a9353d1d230416296627d7cf8b1d425 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 186b12638a87410487467df721a3cbda (127.0.107.193:38967): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:03:47.960321 11152 ts_tablet_manager.cc:1403] T 6f46a5102fd64b45bf1c4dff863d2b4d P 0172fed22b6b4890a7fdc953fa128af4: Time spent bootstrapping tablet: real 0.338s	user 0.037s	sys 0.034s
I20260709 13:03:47.961114 11151 raft_consensus.cc:359] T 57b6a2418c1342e7924195e9c1dbd357 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:47.961980 11151 raft_consensus.cc:385] T 57b6a2418c1342e7924195e9c1dbd357 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:47.963876 11152 raft_consensus.cc:359] T 6f46a5102fd64b45bf1c4dff863d2b4d P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:47.964628 11152 raft_consensus.cc:385] T 6f46a5102fd64b45bf1c4dff863d2b4d P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:47.964908 11152 raft_consensus.cc:740] T 6f46a5102fd64b45bf1c4dff863d2b4d P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0172fed22b6b4890a7fdc953fa128af4, State: Initialized, Role: FOLLOWER
I20260709 13:03:47.965791 11152 consensus_queue.cc:260] T 6f46a5102fd64b45bf1c4dff863d2b4d P 0172fed22b6b4890a7fdc953fa128af4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:47.967707 11102 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a9353d1d230416296627d7cf8b1d425" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:03:47.968348 11102 raft_consensus.cc:3060] T 6a9353d1d230416296627d7cf8b1d425 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:03:47.962327 11151 raft_consensus.cc:740] T 57b6a2418c1342e7924195e9c1dbd357 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 186b12638a87410487467df721a3cbda, State: Initialized, Role: FOLLOWER
I20260709 13:03:47.989672 11151 consensus_queue.cc:260] T 57b6a2418c1342e7924195e9c1dbd357 P 186b12638a87410487467df721a3cbda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:48.002398 11149 tablet_bootstrap.cc:492] T 57b6a2418c1342e7924195e9c1dbd357 P 873357742d5843579556ee8166f23f47: No bootstrap required, opened a new log
I20260709 13:03:47.995692 10865 tablet_service.cc:1511] Processing CreateTablet for tablet 5d156d6ae2ca4fa59a1e387edf0fea39 (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260709 13:03:48.008430 11149 ts_tablet_manager.cc:1403] T 57b6a2418c1342e7924195e9c1dbd357 P 873357742d5843579556ee8166f23f47: Time spent bootstrapping tablet: real 0.202s	user 0.023s	sys 0.032s
I20260709 13:03:48.009433 10865 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d156d6ae2ca4fa59a1e387edf0fea39. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:47.991142 11196 raft_consensus.cc:493] T 6f46a5102fd64b45bf1c4dff863d2b4d P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:48.010183 11196 raft_consensus.cc:515] T 6f46a5102fd64b45bf1c4dff863d2b4d P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:48.011826 11016 tablet_service.cc:1511] Processing CreateTablet for tablet 8c160ddfe9b34bdaa3835d0915c1e06f (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260709 13:03:48.012542 11196 leader_election.cc:290] T 6f46a5102fd64b45bf1c4dff863d2b4d P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:03:47.995358 11152 ts_tablet_manager.cc:1434] T 6f46a5102fd64b45bf1c4dff863d2b4d P 0172fed22b6b4890a7fdc953fa128af4: Time spent starting tablet: real 0.035s	user 0.011s	sys 0.001s
I20260709 13:03:48.014681 11152 tablet_bootstrap.cc:492] T 7709387f4d3d4f9992d97a0fbca39a40 P 0172fed22b6b4890a7fdc953fa128af4: Bootstrap starting.
I20260709 13:03:48.016712 11016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c160ddfe9b34bdaa3835d0915c1e06f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:48.016559 10955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f46a5102fd64b45bf1c4dff863d2b4d" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:03:48.017460 10955 raft_consensus.cc:2468] T 6f46a5102fd64b45bf1c4dff863d2b4d P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 0.
I20260709 13:03:48.003048 11151 ts_tablet_manager.cc:1434] T 57b6a2418c1342e7924195e9c1dbd357 P 186b12638a87410487467df721a3cbda: Time spent starting tablet: real 0.060s	user 0.008s	sys 0.000s
I20260709 13:03:48.018932 11066 leader_election.cc:304] T 6f46a5102fd64b45bf1c4dff863d2b4d P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 10ca5ee68ad64b3283b9d235352c9d23; no voters: 
I20260709 13:03:48.019989 11151 tablet_bootstrap.cc:492] T d7ce47b9a92d43dd88d95bcda5038597 P 186b12638a87410487467df721a3cbda: Bootstrap starting.
I20260709 13:03:48.041106 11196 raft_consensus.cc:2804] T 6f46a5102fd64b45bf1c4dff863d2b4d P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:03:48.044866 11196 raft_consensus.cc:493] T 6f46a5102fd64b45bf1c4dff863d2b4d P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:48.044914 11028 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f46a5102fd64b45bf1c4dff863d2b4d" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:03:48.045430 11196 raft_consensus.cc:3060] T 6f46a5102fd64b45bf1c4dff863d2b4d P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:03:48.050711 11149 raft_consensus.cc:359] T 57b6a2418c1342e7924195e9c1dbd357 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:48.051582 11149 raft_consensus.cc:385] T 57b6a2418c1342e7924195e9c1dbd357 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:48.051730 11188 raft_consensus.cc:493] T 525bfdbd6d934afd823f7d3ee3f9d052 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:48.051942 11149 raft_consensus.cc:740] T 57b6a2418c1342e7924195e9c1dbd357 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 873357742d5843579556ee8166f23f47, State: Initialized, Role: FOLLOWER
I20260709 13:03:47.990423 11102 raft_consensus.cc:2468] T 6a9353d1d230416296627d7cf8b1d425 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 1.
I20260709 13:03:48.052825 11149 consensus_queue.cc:260] T 57b6a2418c1342e7924195e9c1dbd357 P 873357742d5843579556ee8166f23f47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:48.054136 10994 leader_election.cc:304] T 6a9353d1d230416296627d7cf8b1d425 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47; no voters: 186b12638a87410487467df721a3cbda
I20260709 13:03:48.055471 11198 raft_consensus.cc:2804] T 6a9353d1d230416296627d7cf8b1d425 P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:03:48.055780 11149 ts_tablet_manager.cc:1434] T 57b6a2418c1342e7924195e9c1dbd357 P 873357742d5843579556ee8166f23f47: Time spent starting tablet: real 0.047s	user 0.006s	sys 0.002s
I20260709 13:03:48.052366 11188 raft_consensus.cc:515] T 525bfdbd6d934afd823f7d3ee3f9d052 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:48.052794 11151 tablet_bootstrap.cc:654] T d7ce47b9a92d43dd88d95bcda5038597 P 186b12638a87410487467df721a3cbda: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:48.059657 11188 leader_election.cc:290] T 525bfdbd6d934afd823f7d3ee3f9d052 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:03:48.059828 11028 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "525bfdbd6d934afd823f7d3ee3f9d052" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:03:48.060559 11028 raft_consensus.cc:2468] T 525bfdbd6d934afd823f7d3ee3f9d052 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 0.
I20260709 13:03:48.062032 10919 leader_election.cc:304] T 525bfdbd6d934afd823f7d3ee3f9d052 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:03:47.996531 10940 tablet_service.cc:1511] Processing CreateTablet for tablet 5d156d6ae2ca4fa59a1e387edf0fea39 (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260709 13:03:48.062974 11188 raft_consensus.cc:2804] T 525bfdbd6d934afd823f7d3ee3f9d052 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:03:48.063369 11188 raft_consensus.cc:493] T 525bfdbd6d934afd823f7d3ee3f9d052 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:48.063794 11102 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "525bfdbd6d934afd823f7d3ee3f9d052" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:03:48.064899 11149 tablet_bootstrap.cc:492] T 6f46a5102fd64b45bf1c4dff863d2b4d P 873357742d5843579556ee8166f23f47: Bootstrap starting.
W20260709 13:03:48.065749 10920 leader_election.cc:343] T 525bfdbd6d934afd823f7d3ee3f9d052 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:03:48.063787 10940 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d156d6ae2ca4fa59a1e387edf0fea39. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:48.072357 11152 tablet_bootstrap.cc:654] T 7709387f4d3d4f9992d97a0fbca39a40 P 0172fed22b6b4890a7fdc953fa128af4: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:48.072021 11084 tablet_service.cc:1511] Processing CreateTablet for tablet 8c160ddfe9b34bdaa3835d0915c1e06f (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260709 13:03:48.079458 11084 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c160ddfe9b34bdaa3835d0915c1e06f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:48.056020 11198 raft_consensus.cc:697] T 6a9353d1d230416296627d7cf8b1d425 P 873357742d5843579556ee8166f23f47 [term 1 LEADER]: Becoming Leader. State: Replica: 873357742d5843579556ee8166f23f47, State: Running, Role: LEADER
I20260709 13:03:48.076221 10864 tablet_service.cc:1511] Processing CreateTablet for tablet 8c160ddfe9b34bdaa3835d0915c1e06f (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260709 13:03:48.063719 11188 raft_consensus.cc:3060] T 525bfdbd6d934afd823f7d3ee3f9d052 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:03:48.082479 10864 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c160ddfe9b34bdaa3835d0915c1e06f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:48.082792 11148 tablet_bootstrap.cc:492] T d7ce47b9a92d43dd88d95bcda5038597 P 10ca5ee68ad64b3283b9d235352c9d23: No bootstrap required, opened a new log
I20260709 13:03:48.082264 11198 consensus_queue.cc:237] T 6a9353d1d230416296627d7cf8b1d425 P 873357742d5843579556ee8166f23f47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:48.083410 11148 ts_tablet_manager.cc:1403] T d7ce47b9a92d43dd88d95bcda5038597 P 10ca5ee68ad64b3283b9d235352c9d23: Time spent bootstrapping tablet: real 0.267s	user 0.019s	sys 0.003s
I20260709 13:03:48.068210 11196 raft_consensus.cc:515] T 6f46a5102fd64b45bf1c4dff863d2b4d P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:48.081535 11149 tablet_bootstrap.cc:654] T 6f46a5102fd64b45bf1c4dff863d2b4d P 873357742d5843579556ee8166f23f47: Neither blocks nor log segments found. Creating new log.
W20260709 13:03:48.103714 11067 leader_election.cc:343] T 6f46a5102fd64b45bf1c4dff863d2b4d P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 873357742d5843579556ee8166f23f47 (127.0.107.195:45807): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:03:48.109220 10944 tablet_service.cc:1511] Processing CreateTablet for tablet ff13ae9a47c7455f962406c6d131a3ff (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260709 13:03:48.111244 10944 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff13ae9a47c7455f962406c6d131a3ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:48.116663 11148 raft_consensus.cc:359] T d7ce47b9a92d43dd88d95bcda5038597 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:48.117538 11148 raft_consensus.cc:385] T d7ce47b9a92d43dd88d95bcda5038597 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:48.117852 11148 raft_consensus.cc:740] T d7ce47b9a92d43dd88d95bcda5038597 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10ca5ee68ad64b3283b9d235352c9d23, State: Initialized, Role: FOLLOWER
I20260709 13:03:48.119765 11028 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7709387f4d3d4f9992d97a0fbca39a40" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
W20260709 13:03:48.121526 10844 leader_election.cc:343] T 7709387f4d3d4f9992d97a0fbca39a40 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 873357742d5843579556ee8166f23f47 (127.0.107.195:45807): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:03:48.122051 10844 leader_election.cc:304] T 7709387f4d3d4f9992d97a0fbca39a40 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 186b12638a87410487467df721a3cbda; no voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47
I20260709 13:03:48.123596 11170 raft_consensus.cc:2749] T 7709387f4d3d4f9992d97a0fbca39a40 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:03:48.135932 11015 tablet_service.cc:1511] Processing CreateTablet for tablet ff13ae9a47c7455f962406c6d131a3ff (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260709 13:03:48.138052 11015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff13ae9a47c7455f962406c6d131a3ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:48.139195 10955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f46a5102fd64b45bf1c4dff863d2b4d" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:03:48.139530 11028 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f46a5102fd64b45bf1c4dff863d2b4d" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
I20260709 13:03:48.139992 10955 raft_consensus.cc:3060] T 6f46a5102fd64b45bf1c4dff863d2b4d P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Advancing to term 1
W20260709 13:03:48.141016 11067 leader_election.cc:343] T 6f46a5102fd64b45bf1c4dff863d2b4d P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 873357742d5843579556ee8166f23f47 (127.0.107.195:45807): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:03:48.144313 11196 leader_election.cc:290] T 6f46a5102fd64b45bf1c4dff863d2b4d P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 1 election: Requested vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:03:48.147360 10867 tablet_service.cc:1511] Processing CreateTablet for tablet a5299b8d044f41b7b68057e63d17b159 (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260709 13:03:48.148880 10867 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a5299b8d044f41b7b68057e63d17b159. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:48.149370 11086 tablet_service.cc:1511] Processing CreateTablet for tablet ff13ae9a47c7455f962406c6d131a3ff (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260709 13:03:48.153256 11086 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff13ae9a47c7455f962406c6d131a3ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:48.154563 11148 consensus_queue.cc:260] T d7ce47b9a92d43dd88d95bcda5038597 P 10ca5ee68ad64b3283b9d235352c9d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:48.157137 11148 ts_tablet_manager.cc:1434] T d7ce47b9a92d43dd88d95bcda5038597 P 10ca5ee68ad64b3283b9d235352c9d23: Time spent starting tablet: real 0.073s	user 0.000s	sys 0.007s
I20260709 13:03:48.151399 11188 raft_consensus.cc:515] T 525bfdbd6d934afd823f7d3ee3f9d052 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:48.152055 10955 raft_consensus.cc:2468] T 6f46a5102fd64b45bf1c4dff863d2b4d P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 1.
I20260709 13:03:48.150195 11151 tablet_bootstrap.cc:492] T d7ce47b9a92d43dd88d95bcda5038597 P 186b12638a87410487467df721a3cbda: No bootstrap required, opened a new log
I20260709 13:03:48.163035 11148 tablet_bootstrap.cc:492] T c51782cd9d064ec0949e84e24dd4de67 P 10ca5ee68ad64b3283b9d235352c9d23: Bootstrap starting.
I20260709 13:03:48.164888 11168 raft_consensus.cc:493] T d7ce47b9a92d43dd88d95bcda5038597 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:48.174587 11028 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "525bfdbd6d934afd823f7d3ee3f9d052" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
I20260709 13:03:48.175375 11028 raft_consensus.cc:3060] T 525bfdbd6d934afd823f7d3ee3f9d052 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:03:48.166994 11151 ts_tablet_manager.cc:1403] T d7ce47b9a92d43dd88d95bcda5038597 P 186b12638a87410487467df721a3cbda: Time spent bootstrapping tablet: real 0.147s	user 0.010s	sys 0.008s
I20260709 13:03:48.180615 11013 tablet_service.cc:1511] Processing CreateTablet for tablet a5299b8d044f41b7b68057e63d17b159 (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260709 13:03:48.174638 11168 raft_consensus.cc:515] T d7ce47b9a92d43dd88d95bcda5038597 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:48.200925 11168 leader_election.cc:290] T d7ce47b9a92d43dd88d95bcda5038597 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:03:48.201116 11151 raft_consensus.cc:359] T d7ce47b9a92d43dd88d95bcda5038597 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:48.202055 11151 raft_consensus.cc:385] T d7ce47b9a92d43dd88d95bcda5038597 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:48.178759 11066 leader_election.cc:304] T 6f46a5102fd64b45bf1c4dff863d2b4d P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 10ca5ee68ad64b3283b9d235352c9d23; no voters: 873357742d5843579556ee8166f23f47
I20260709 13:03:48.203409 11196 raft_consensus.cc:2804] T 6f46a5102fd64b45bf1c4dff863d2b4d P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:03:48.203945 11196 raft_consensus.cc:697] T 6f46a5102fd64b45bf1c4dff863d2b4d P 0172fed22b6b4890a7fdc953fa128af4 [term 1 LEADER]: Becoming Leader. State: Replica: 0172fed22b6b4890a7fdc953fa128af4, State: Running, Role: LEADER
I20260709 13:03:48.181022 10777 catalog_manager.cc:5697] T 4f9eb06ed01f4ce68be3af8a8434665b P 873357742d5843579556ee8166f23f47 reported cstate change: term changed from 0 to 1, leader changed from <none> to 873357742d5843579556ee8166f23f47 (127.0.107.195). New cstate: current_term: 1 leader_uuid: "873357742d5843579556ee8166f23f47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } health_report { overall_health: UNKNOWN } } }
I20260709 13:03:48.204893 11196 consensus_queue.cc:237] T 6f46a5102fd64b45bf1c4dff863d2b4d P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:48.208593 10878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d7ce47b9a92d43dd88d95bcda5038597" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:03:48.209167 11102 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d7ce47b9a92d43dd88d95bcda5038597" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:03:48.210132 11102 raft_consensus.cc:2468] T d7ce47b9a92d43dd88d95bcda5038597 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 0.
I20260709 13:03:48.211838 10920 leader_election.cc:304] T d7ce47b9a92d43dd88d95bcda5038597 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 10ca5ee68ad64b3283b9d235352c9d23; no voters: 
I20260709 13:03:48.213142 11168 raft_consensus.cc:2804] T d7ce47b9a92d43dd88d95bcda5038597 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:03:48.213738 11168 raft_consensus.cc:493] T d7ce47b9a92d43dd88d95bcda5038597 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:48.214246 11168 raft_consensus.cc:3060] T d7ce47b9a92d43dd88d95bcda5038597 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:03:48.216706 11089 tablet_service.cc:1511] Processing CreateTablet for tablet c31881534c5e4f0089c82057c1746652 (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260709 13:03:48.218168 11089 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c31881534c5e4f0089c82057c1746652. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:48.219537 10936 tablet_service.cc:1511] Processing CreateTablet for tablet a5299b8d044f41b7b68057e63d17b159 (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260709 13:03:48.189913 11198 raft_consensus.cc:493] T 57b6a2418c1342e7924195e9c1dbd357 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:48.221915 11198 raft_consensus.cc:515] T 57b6a2418c1342e7924195e9c1dbd357 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:48.225350 10879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57b6a2418c1342e7924195e9c1dbd357" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:03:48.225706 11013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a5299b8d044f41b7b68057e63d17b159. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:48.226085 10879 raft_consensus.cc:2468] T 57b6a2418c1342e7924195e9c1dbd357 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 0.
I20260709 13:03:48.226820 11198 leader_election.cc:290] T 57b6a2418c1342e7924195e9c1dbd357 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:03:48.228623 10995 leader_election.cc:304] T 57b6a2418c1342e7924195e9c1dbd357 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:03:48.230195 10865 tablet_service.cc:1511] Processing CreateTablet for tablet c31881534c5e4f0089c82057c1746652 (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260709 13:03:48.231637 10865 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c31881534c5e4f0089c82057c1746652. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:48.231822 10939 tablet_service.cc:1511] Processing CreateTablet for tablet c31881534c5e4f0089c82057c1746652 (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260709 13:03:48.233881 11198 raft_consensus.cc:2804] T 57b6a2418c1342e7924195e9c1dbd357 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:03:48.234388 11198 raft_consensus.cc:493] T 57b6a2418c1342e7924195e9c1dbd357 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:48.234830 11198 raft_consensus.cc:3060] T 57b6a2418c1342e7924195e9c1dbd357 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:03:48.185071 11152 tablet_bootstrap.cc:492] T 7709387f4d3d4f9992d97a0fbca39a40 P 0172fed22b6b4890a7fdc953fa128af4: No bootstrap required, opened a new log
I20260709 13:03:48.244794 11152 ts_tablet_manager.cc:1403] T 7709387f4d3d4f9992d97a0fbca39a40 P 0172fed22b6b4890a7fdc953fa128af4: Time spent bootstrapping tablet: real 0.230s	user 0.020s	sys 0.014s
I20260709 13:03:48.246618 10936 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a5299b8d044f41b7b68057e63d17b159. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:48.202387 11151 raft_consensus.cc:740] T d7ce47b9a92d43dd88d95bcda5038597 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 186b12638a87410487467df721a3cbda, State: Initialized, Role: FOLLOWER
I20260709 13:03:48.251452 10939 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c31881534c5e4f0089c82057c1746652. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:48.255211 11188 leader_election.cc:290] T 525bfdbd6d934afd823f7d3ee3f9d052 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 1 election: Requested vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:03:48.255358 11102 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "525bfdbd6d934afd823f7d3ee3f9d052" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:03:48.257936 11018 tablet_service.cc:1511] Processing CreateTablet for tablet acc241e7273d49149984968596242712 (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260709 13:03:48.259608 10866 tablet_service.cc:1511] Processing CreateTablet for tablet acc241e7273d49149984968596242712 (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260709 13:03:48.275252 10866 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet acc241e7273d49149984968596242712. 1 dirs total, 0 dirs full, 0 dirs failed
W20260709 13:03:48.268757 10920 leader_election.cc:343] T 525bfdbd6d934afd823f7d3ee3f9d052 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:03:48.252631 11151 consensus_queue.cc:260] T d7ce47b9a92d43dd88d95bcda5038597 P 186b12638a87410487467df721a3cbda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:48.275305 11018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet acc241e7273d49149984968596242712. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:48.278932 11028 raft_consensus.cc:2468] T 525bfdbd6d934afd823f7d3ee3f9d052 P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 1.
I20260709 13:03:48.269955 11091 tablet_service.cc:1511] Processing CreateTablet for tablet acc241e7273d49149984968596242712 (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260709 13:03:48.286490 11091 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet acc241e7273d49149984968596242712. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:48.282840 10878 raft_consensus.cc:2468] T d7ce47b9a92d43dd88d95bcda5038597 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 0.
I20260709 13:03:48.308992 10919 leader_election.cc:304] T 525bfdbd6d934afd823f7d3ee3f9d052 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47; no voters: 0172fed22b6b4890a7fdc953fa128af4
I20260709 13:03:48.310868 11188 raft_consensus.cc:2804] T 525bfdbd6d934afd823f7d3ee3f9d052 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:03:48.311384 11188 raft_consensus.cc:697] T 525bfdbd6d934afd823f7d3ee3f9d052 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 LEADER]: Becoming Leader. State: Replica: 10ca5ee68ad64b3283b9d235352c9d23, State: Running, Role: LEADER
I20260709 13:03:48.312260 11188 consensus_queue.cc:237] T 525bfdbd6d934afd823f7d3ee3f9d052 P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:48.319149 10940 tablet_service.cc:1511] Processing CreateTablet for tablet 75492f86fde046c3900df3eac8292fd4 (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260709 13:03:48.322556 11016 tablet_service.cc:1511] Processing CreateTablet for tablet 75492f86fde046c3900df3eac8292fd4 (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260709 13:03:48.327522 11016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 75492f86fde046c3900df3eac8292fd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:48.323943 11168 raft_consensus.cc:515] T d7ce47b9a92d43dd88d95bcda5038597 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:48.325194 11151 ts_tablet_manager.cc:1434] T d7ce47b9a92d43dd88d95bcda5038597 P 186b12638a87410487467df721a3cbda: Time spent starting tablet: real 0.128s	user 0.010s	sys 0.000s
I20260709 13:03:48.328220 11084 tablet_service.cc:1511] Processing CreateTablet for tablet 75492f86fde046c3900df3eac8292fd4 (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260709 13:03:48.323333 10940 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 75492f86fde046c3900df3eac8292fd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:48.338431 11084 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 75492f86fde046c3900df3eac8292fd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:48.339913 11198 raft_consensus.cc:515] T 57b6a2418c1342e7924195e9c1dbd357 P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:48.332572 11151 tablet_bootstrap.cc:492] T 0f9df2bc281f4866b6da5da94a0defc7 P 186b12638a87410487467df721a3cbda: Bootstrap starting.
I20260709 13:03:48.353149 10878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57b6a2418c1342e7924195e9c1dbd357" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:03:48.353957 10878 raft_consensus.cc:3060] T 57b6a2418c1342e7924195e9c1dbd357 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:03:48.353747 10775 catalog_manager.cc:5697] T 6f46a5102fd64b45bf1c4dff863d2b4d P 0172fed22b6b4890a7fdc953fa128af4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196). New cstate: current_term: 1 leader_uuid: "0172fed22b6b4890a7fdc953fa128af4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } health_report { overall_health: HEALTHY } } }
I20260709 13:03:48.368211 11168 leader_election.cc:290] T d7ce47b9a92d43dd88d95bcda5038597 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 1 election: Requested vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:03:48.369059 11102 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d7ce47b9a92d43dd88d95bcda5038597" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
W20260709 13:03:48.369369 10995 consensus_peers.cc:597] T 4f9eb06ed01f4ce68be3af8a8434665b P 873357742d5843579556ee8166f23f47 -> Peer 186b12638a87410487467df721a3cbda (127.0.107.193:38967): Couldn't send request to peer 186b12638a87410487467df721a3cbda. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:03:48.369839 11102 raft_consensus.cc:3060] T d7ce47b9a92d43dd88d95bcda5038597 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:03:48.370741 10879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d7ce47b9a92d43dd88d95bcda5038597" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:03:48.371448 10879 raft_consensus.cc:3060] T d7ce47b9a92d43dd88d95bcda5038597 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:03:48.354002 11148 tablet_bootstrap.cc:654] T c51782cd9d064ec0949e84e24dd4de67 P 10ca5ee68ad64b3283b9d235352c9d23: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:48.342267 11198 leader_election.cc:290] T 57b6a2418c1342e7924195e9c1dbd357 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 1 election: Requested vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:03:48.382555 11196 consensus_queue.cc:1048] T 0f9df2bc281f4866b6da5da94a0defc7 P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:03:48.386860 11152 raft_consensus.cc:359] T 7709387f4d3d4f9992d97a0fbca39a40 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:48.387745 11152 raft_consensus.cc:385] T 7709387f4d3d4f9992d97a0fbca39a40 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:48.388060 11152 raft_consensus.cc:740] T 7709387f4d3d4f9992d97a0fbca39a40 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0172fed22b6b4890a7fdc953fa128af4, State: Initialized, Role: FOLLOWER
I20260709 13:03:48.383028 11149 tablet_bootstrap.cc:492] T 6f46a5102fd64b45bf1c4dff863d2b4d P 873357742d5843579556ee8166f23f47: No bootstrap required, opened a new log
I20260709 13:03:48.358966 10778 catalog_manager.cc:5697] T 6a9353d1d230416296627d7cf8b1d425 P 873357742d5843579556ee8166f23f47 reported cstate change: term changed from 0 to 1, leader changed from <none> to 873357742d5843579556ee8166f23f47 (127.0.107.195). New cstate: current_term: 1 leader_uuid: "873357742d5843579556ee8166f23f47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } health_report { overall_health: UNKNOWN } } }
I20260709 13:03:48.390462 11152 consensus_queue.cc:260] T 7709387f4d3d4f9992d97a0fbca39a40 P 0172fed22b6b4890a7fdc953fa128af4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:48.391716 11102 raft_consensus.cc:2468] T d7ce47b9a92d43dd88d95bcda5038597 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 1.
I20260709 13:03:48.400058 10879 raft_consensus.cc:2468] T d7ce47b9a92d43dd88d95bcda5038597 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 1.
I20260709 13:03:48.400396 10920 leader_election.cc:304] T d7ce47b9a92d43dd88d95bcda5038597 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 10ca5ee68ad64b3283b9d235352c9d23; no voters: 
I20260709 13:03:48.401294 11188 raft_consensus.cc:2804] T d7ce47b9a92d43dd88d95bcda5038597 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:03:48.401808 11188 raft_consensus.cc:697] T d7ce47b9a92d43dd88d95bcda5038597 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 LEADER]: Becoming Leader. State: Replica: 10ca5ee68ad64b3283b9d235352c9d23, State: Running, Role: LEADER
I20260709 13:03:48.401994 11198 consensus_queue.cc:1048] T 4f9eb06ed01f4ce68be3af8a8434665b P 873357742d5843579556ee8166f23f47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:03:48.402784 11188 consensus_queue.cc:237] T d7ce47b9a92d43dd88d95bcda5038597 P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:48.412650 10867 tablet_service.cc:1511] Processing CreateTablet for tablet 1e4f2241a94a41f38e481137f868b83d (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260709 13:03:48.414242 10775 catalog_manager.cc:5697] T 525bfdbd6d934afd823f7d3ee3f9d052 P 10ca5ee68ad64b3283b9d235352c9d23 reported cstate change: term changed from 0 to 1, leader changed from <none> to 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194). New cstate: current_term: 1 leader_uuid: "10ca5ee68ad64b3283b9d235352c9d23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } health_report { overall_health: UNKNOWN } } }
I20260709 13:03:48.425849 10867 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e4f2241a94a41f38e481137f868b83d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:48.435514 11152 ts_tablet_manager.cc:1434] T 7709387f4d3d4f9992d97a0fbca39a40 P 0172fed22b6b4890a7fdc953fa128af4: Time spent starting tablet: real 0.181s	user 0.006s	sys 0.003s
I20260709 13:03:48.436578 11148 tablet_bootstrap.cc:492] T c51782cd9d064ec0949e84e24dd4de67 P 10ca5ee68ad64b3283b9d235352c9d23: No bootstrap required, opened a new log
I20260709 13:03:48.436902 11152 tablet_bootstrap.cc:492] T c51782cd9d064ec0949e84e24dd4de67 P 0172fed22b6b4890a7fdc953fa128af4: Bootstrap starting.
I20260709 13:03:48.437179 11148 ts_tablet_manager.cc:1403] T c51782cd9d064ec0949e84e24dd4de67 P 10ca5ee68ad64b3283b9d235352c9d23: Time spent bootstrapping tablet: real 0.279s	user 0.011s	sys 0.007s
W20260709 13:03:48.438624 11069 consensus_peers.cc:597] T 0f9df2bc281f4866b6da5da94a0defc7 P 0172fed22b6b4890a7fdc953fa128af4 -> Peer 186b12638a87410487467df721a3cbda (127.0.107.193:38967): Couldn't send request to peer 186b12638a87410487467df721a3cbda. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:03:48.399407 11149 ts_tablet_manager.cc:1403] T 6f46a5102fd64b45bf1c4dff863d2b4d P 873357742d5843579556ee8166f23f47: Time spent bootstrapping tablet: real 0.330s	user 0.049s	sys 0.033s
I20260709 13:03:48.393370 10878 raft_consensus.cc:2468] T 57b6a2418c1342e7924195e9c1dbd357 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 1.
I20260709 13:03:48.445964 10954 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57b6a2418c1342e7924195e9c1dbd357" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:03:48.447098 10953 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57b6a2418c1342e7924195e9c1dbd357" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:03:48.451421 10939 tablet_service.cc:1511] Processing CreateTablet for tablet 1e4f2241a94a41f38e481137f868b83d (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
W20260709 13:03:48.456785 10992 leader_election.cc:343] T 57b6a2418c1342e7924195e9c1dbd357 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:03:48.457064 10939 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e4f2241a94a41f38e481137f868b83d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:48.453711 11149 raft_consensus.cc:359] T 6f46a5102fd64b45bf1c4dff863d2b4d P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:48.455461 11015 tablet_service.cc:1511] Processing CreateTablet for tablet 1e4f2241a94a41f38e481137f868b83d (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260709 13:03:48.464514 11015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e4f2241a94a41f38e481137f868b83d. 1 dirs total, 0 dirs full, 0 dirs failed
W20260709 13:03:48.465092 10992 leader_election.cc:343] T 57b6a2418c1342e7924195e9c1dbd357 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:03:48.469432 10995 leader_election.cc:304] T 57b6a2418c1342e7924195e9c1dbd357 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47; no voters: 10ca5ee68ad64b3283b9d235352c9d23
I20260709 13:03:48.470911 11198 raft_consensus.cc:2804] T 57b6a2418c1342e7924195e9c1dbd357 P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:03:48.471437 11198 raft_consensus.cc:697] T 57b6a2418c1342e7924195e9c1dbd357 P 873357742d5843579556ee8166f23f47 [term 1 LEADER]: Becoming Leader. State: Replica: 873357742d5843579556ee8166f23f47, State: Running, Role: LEADER
I20260709 13:03:48.472352 11198 consensus_queue.cc:237] T 57b6a2418c1342e7924195e9c1dbd357 P 873357742d5843579556ee8166f23f47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:48.465875 11149 raft_consensus.cc:385] T 6f46a5102fd64b45bf1c4dff863d2b4d P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:48.480983 11149 raft_consensus.cc:740] T 6f46a5102fd64b45bf1c4dff863d2b4d P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 873357742d5843579556ee8166f23f47, State: Initialized, Role: FOLLOWER
I20260709 13:03:48.482055 11149 consensus_queue.cc:260] T 6f46a5102fd64b45bf1c4dff863d2b4d P 873357742d5843579556ee8166f23f47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:48.484661 11149 ts_tablet_manager.cc:1434] T 6f46a5102fd64b45bf1c4dff863d2b4d P 873357742d5843579556ee8166f23f47: Time spent starting tablet: real 0.040s	user 0.006s	sys 0.000s
I20260709 13:03:48.497696 11152 tablet_bootstrap.cc:654] T c51782cd9d064ec0949e84e24dd4de67 P 0172fed22b6b4890a7fdc953fa128af4: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:48.497368 11089 tablet_service.cc:1511] Processing CreateTablet for tablet b67eff0634e540eab91cf8cee795969f (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260709 13:03:48.508443 11089 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b67eff0634e540eab91cf8cee795969f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:48.525542 11162 raft_consensus.cc:493] T 7709387f4d3d4f9992d97a0fbca39a40 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:48.541474 11162 raft_consensus.cc:515] T 7709387f4d3d4f9992d97a0fbca39a40 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:48.534768 11016 tablet_service.cc:1511] Processing CreateTablet for tablet 401a9cf11eac479aa445ec8c1d6e33b2 (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260709 13:03:48.543398 11016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 401a9cf11eac479aa445ec8c1d6e33b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:48.543959 11162 leader_election.cc:290] T 7709387f4d3d4f9992d97a0fbca39a40 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:03:48.544463 10778 catalog_manager.cc:5697] T 57b6a2418c1342e7924195e9c1dbd357 P 873357742d5843579556ee8166f23f47 reported cstate change: term changed from 0 to 1, leader changed from <none> to 873357742d5843579556ee8166f23f47 (127.0.107.195). New cstate: current_term: 1 leader_uuid: "873357742d5843579556ee8166f23f47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } health_report { overall_health: HEALTHY } } }
I20260709 13:03:48.540446 11091 tablet_service.cc:1511] Processing CreateTablet for tablet 401a9cf11eac479aa445ec8c1d6e33b2 (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260709 13:03:48.549036 10878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7709387f4d3d4f9992d97a0fbca39a40" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:03:48.549515 11091 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 401a9cf11eac479aa445ec8c1d6e33b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:48.535611 10865 tablet_service.cc:1511] Processing CreateTablet for tablet b67eff0634e540eab91cf8cee795969f (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260709 13:03:48.552153 10940 tablet_service.cc:1511] Processing CreateTablet for tablet b67eff0634e540eab91cf8cee795969f (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260709 13:03:48.553422 10940 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b67eff0634e540eab91cf8cee795969f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:48.550359 11149 tablet_bootstrap.cc:492] T 7839ca2d2c13431a85f44d7c458c6351 P 873357742d5843579556ee8166f23f47: Bootstrap starting.
I20260709 13:03:48.571247 10865 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b67eff0634e540eab91cf8cee795969f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:48.549868 10878 raft_consensus.cc:2468] T 7709387f4d3d4f9992d97a0fbca39a40 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 0.
I20260709 13:03:48.569293 11151 tablet_bootstrap.cc:700] T 0f9df2bc281f4866b6da5da94a0defc7 P 186b12638a87410487467df721a3cbda: Time spent opening tablet: real 0.119s	user 0.000s	sys 0.003s
I20260709 13:03:48.587841 11148 raft_consensus.cc:359] T c51782cd9d064ec0949e84e24dd4de67 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:48.588989 11148 raft_consensus.cc:385] T c51782cd9d064ec0949e84e24dd4de67 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:48.589381 11148 raft_consensus.cc:740] T c51782cd9d064ec0949e84e24dd4de67 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10ca5ee68ad64b3283b9d235352c9d23, State: Initialized, Role: FOLLOWER
I20260709 13:03:48.591328 11149 tablet_bootstrap.cc:654] T 7839ca2d2c13431a85f44d7c458c6351 P 873357742d5843579556ee8166f23f47: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:48.600242 11028 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7709387f4d3d4f9992d97a0fbca39a40" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
W20260709 13:03:48.601944 11067 leader_election.cc:343] T 7709387f4d3d4f9992d97a0fbca39a40 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 873357742d5843579556ee8166f23f47 (127.0.107.195:45807): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:03:48.603545 10944 tablet_service.cc:1511] Processing CreateTablet for tablet a977842ba4b744f4bb2febd55d286006 (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260709 13:03:48.605052 10944 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a977842ba4b744f4bb2febd55d286006. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:48.605553 11086 tablet_service.cc:1511] Processing CreateTablet for tablet a977842ba4b744f4bb2febd55d286006 (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260709 13:03:48.606945 11086 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a977842ba4b744f4bb2febd55d286006. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:48.607604 10869 tablet_service.cc:1511] Processing CreateTablet for tablet 30f628e3d7914d609e7f1d980a74c654 (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260709 13:03:48.609001 10869 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30f628e3d7914d609e7f1d980a74c654. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:48.608742 10864 tablet_service.cc:1511] Processing CreateTablet for tablet 401a9cf11eac479aa445ec8c1d6e33b2 (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260709 13:03:48.610136 10864 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 401a9cf11eac479aa445ec8c1d6e33b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:48.611370 11016 tablet_service.cc:1511] Processing CreateTablet for tablet a977842ba4b744f4bb2febd55d286006 (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260709 13:03:48.611984 10938 tablet_service.cc:1511] Processing CreateTablet for tablet 30f628e3d7914d609e7f1d980a74c654 (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260709 13:03:48.612622 11016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a977842ba4b744f4bb2febd55d286006. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:48.613278 10938 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30f628e3d7914d609e7f1d980a74c654. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:48.614918 11151 tablet_bootstrap.cc:654] T 0f9df2bc281f4866b6da5da94a0defc7 P 186b12638a87410487467df721a3cbda: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:48.615828 10941 tablet_service.cc:1511] Processing CreateTablet for tablet f34f2bc02d85444f9096e0f5b4fe4331 (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260709 13:03:48.617478 10941 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f34f2bc02d85444f9096e0f5b4fe4331. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:48.621868 11013 tablet_service.cc:1511] Processing CreateTablet for tablet 30f628e3d7914d609e7f1d980a74c654 (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260709 13:03:48.623226 11013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30f628e3d7914d609e7f1d980a74c654. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:48.629329 10870 tablet_service.cc:1511] Processing CreateTablet for tablet f34f2bc02d85444f9096e0f5b4fe4331 (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260709 13:03:48.630805 10870 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f34f2bc02d85444f9096e0f5b4fe4331. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:48.632937 11069 leader_election.cc:304] T 7709387f4d3d4f9992d97a0fbca39a40 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda; no voters: 873357742d5843579556ee8166f23f47
I20260709 13:03:48.635926 11088 tablet_service.cc:1511] Processing CreateTablet for tablet f34f2bc02d85444f9096e0f5b4fe4331 (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260709 13:03:48.647358 11088 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f34f2bc02d85444f9096e0f5b4fe4331. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:48.664740 11093 tablet_service.cc:1511] Processing CreateTablet for tablet a1ec2b50c6ad4f7c8033476208abb26d (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260709 13:03:48.651464 11162 raft_consensus.cc:2804] T 7709387f4d3d4f9992d97a0fbca39a40 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:03:48.665905 11162 raft_consensus.cc:493] T 7709387f4d3d4f9992d97a0fbca39a40 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:48.666256 11093 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a1ec2b50c6ad4f7c8033476208abb26d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:48.669917 11015 tablet_service.cc:1511] Processing CreateTablet for tablet a1ec2b50c6ad4f7c8033476208abb26d (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
W20260709 13:03:48.670331 10995 consensus_peers.cc:597] T 6a9353d1d230416296627d7cf8b1d425 P 873357742d5843579556ee8166f23f47 -> Peer 186b12638a87410487467df721a3cbda (127.0.107.193:38967): Couldn't send request to peer 186b12638a87410487467df721a3cbda. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:03:48.666287 11162 raft_consensus.cc:3060] T 7709387f4d3d4f9992d97a0fbca39a40 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:03:48.671341 11015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a1ec2b50c6ad4f7c8033476208abb26d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:48.590420 11148 consensus_queue.cc:260] T c51782cd9d064ec0949e84e24dd4de67 P 10ca5ee68ad64b3283b9d235352c9d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:48.677996 11228 consensus_queue.cc:1048] T 6a9353d1d230416296627d7cf8b1d425 P 873357742d5843579556ee8166f23f47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:03:48.683429 10867 tablet_service.cc:1511] Processing CreateTablet for tablet a1ec2b50c6ad4f7c8033476208abb26d (DEFAULT_TABLE table=test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260709 13:03:48.685238 10867 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a1ec2b50c6ad4f7c8033476208abb26d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:48.703346 11162 raft_consensus.cc:515] T 7709387f4d3d4f9992d97a0fbca39a40 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:48.708541 11028 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7709387f4d3d4f9992d97a0fbca39a40" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
W20260709 13:03:48.710239 11067 leader_election.cc:343] T 7709387f4d3d4f9992d97a0fbca39a40 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 873357742d5843579556ee8166f23f47 (127.0.107.195:45807): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:03:48.712004 11162 leader_election.cc:290] T 7709387f4d3d4f9992d97a0fbca39a40 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 1 election: Requested vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:03:48.724931 10778 catalog_manager.cc:5697] T d7ce47b9a92d43dd88d95bcda5038597 P 10ca5ee68ad64b3283b9d235352c9d23 reported cstate change: term changed from 0 to 1, leader changed from <none> to 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194). New cstate: current_term: 1 leader_uuid: "10ca5ee68ad64b3283b9d235352c9d23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } health_report { overall_health: HEALTHY } } }
I20260709 13:03:48.742647 10878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7709387f4d3d4f9992d97a0fbca39a40" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:03:48.743392 10878 raft_consensus.cc:3060] T 7709387f4d3d4f9992d97a0fbca39a40 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:03:48.747236 11148 ts_tablet_manager.cc:1434] T c51782cd9d064ec0949e84e24dd4de67 P 10ca5ee68ad64b3283b9d235352c9d23: Time spent starting tablet: real 0.310s	user 0.010s	sys 0.003s
I20260709 13:03:48.748629 11148 tablet_bootstrap.cc:492] T 57b6a2418c1342e7924195e9c1dbd357 P 10ca5ee68ad64b3283b9d235352c9d23: Bootstrap starting.
I20260709 13:03:48.752333 11149 tablet_bootstrap.cc:492] T 7839ca2d2c13431a85f44d7c458c6351 P 873357742d5843579556ee8166f23f47: No bootstrap required, opened a new log
I20260709 13:03:48.770418 11188 raft_consensus.cc:493] T c51782cd9d064ec0949e84e24dd4de67 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:48.752635 11151 tablet_bootstrap.cc:492] T 0f9df2bc281f4866b6da5da94a0defc7 P 186b12638a87410487467df721a3cbda: No bootstrap required, opened a new log
W20260709 13:03:48.776422 10920 consensus_peers.cc:597] T 525bfdbd6d934afd823f7d3ee3f9d052 P 10ca5ee68ad64b3283b9d235352c9d23 -> Peer 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507): Couldn't send request to peer 0172fed22b6b4890a7fdc953fa128af4. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:03:48.782496 11188 raft_consensus.cc:515] T c51782cd9d064ec0949e84e24dd4de67 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:48.790899 11028 raft_consensus.cc:3060] T 6f46a5102fd64b45bf1c4dff863d2b4d P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:03:48.791785 11188 leader_election.cc:290] T c51782cd9d064ec0949e84e24dd4de67 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:03:48.785334 11149 ts_tablet_manager.cc:1403] T 7839ca2d2c13431a85f44d7c458c6351 P 873357742d5843579556ee8166f23f47: Time spent bootstrapping tablet: real 0.256s	user 0.028s	sys 0.012s
I20260709 13:03:48.796095 10879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c51782cd9d064ec0949e84e24dd4de67" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:03:48.796391 11149 raft_consensus.cc:359] T 7839ca2d2c13431a85f44d7c458c6351 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:48.796851 10879 raft_consensus.cc:2468] T c51782cd9d064ec0949e84e24dd4de67 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 0.
I20260709 13:03:48.797223 11149 raft_consensus.cc:385] T 7839ca2d2c13431a85f44d7c458c6351 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:48.797608 11149 raft_consensus.cc:740] T 7839ca2d2c13431a85f44d7c458c6351 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 873357742d5843579556ee8166f23f47, State: Initialized, Role: FOLLOWER
I20260709 13:03:48.799309 10921 leader_election.cc:304] T c51782cd9d064ec0949e84e24dd4de67 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda; no voters: 
I20260709 13:03:48.800311 11188 raft_consensus.cc:2804] T c51782cd9d064ec0949e84e24dd4de67 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:03:48.800691 11188 raft_consensus.cc:493] T c51782cd9d064ec0949e84e24dd4de67 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:48.801029 11188 raft_consensus.cc:3060] T c51782cd9d064ec0949e84e24dd4de67 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Advancing to term 1
W20260709 13:03:48.781494 10763 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.0.107.195:45807, user_credentials={real_user=slave}} blocked reactor thread for 45947.4us
I20260709 13:03:48.785821 11151 ts_tablet_manager.cc:1403] T 0f9df2bc281f4866b6da5da94a0defc7 P 186b12638a87410487467df721a3cbda: Time spent bootstrapping tablet: real 0.454s	user 0.014s	sys 0.022s
I20260709 13:03:48.807907 11102 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c51782cd9d064ec0949e84e24dd4de67" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
W20260709 13:03:48.809759 10920 leader_election.cc:343] T c51782cd9d064ec0949e84e24dd4de67 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:03:48.815164 11152 tablet_bootstrap.cc:492] T c51782cd9d064ec0949e84e24dd4de67 P 0172fed22b6b4890a7fdc953fa128af4: No bootstrap required, opened a new log
I20260709 13:03:48.815781 11152 ts_tablet_manager.cc:1403] T c51782cd9d064ec0949e84e24dd4de67 P 0172fed22b6b4890a7fdc953fa128af4: Time spent bootstrapping tablet: real 0.379s	user 0.015s	sys 0.007s
I20260709 13:03:48.824536 11149 consensus_queue.cc:260] T 7839ca2d2c13431a85f44d7c458c6351 P 873357742d5843579556ee8166f23f47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:48.826628 11168 consensus_queue.cc:1048] T d7ce47b9a92d43dd88d95bcda5038597 P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:03:48.827710 11149 ts_tablet_manager.cc:1434] T 7839ca2d2c13431a85f44d7c458c6351 P 873357742d5843579556ee8166f23f47: Time spent starting tablet: real 0.035s	user 0.007s	sys 0.001s
I20260709 13:03:48.828321 11168 consensus_queue.cc:1048] T 525bfdbd6d934afd823f7d3ee3f9d052 P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:03:48.828989 11149 tablet_bootstrap.cc:492] T 7709387f4d3d4f9992d97a0fbca39a40 P 873357742d5843579556ee8166f23f47: Bootstrap starting.
I20260709 13:03:48.829739 11151 raft_consensus.cc:359] T 0f9df2bc281f4866b6da5da94a0defc7 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:48.837343 11149 tablet_bootstrap.cc:654] T 7709387f4d3d4f9992d97a0fbca39a40 P 873357742d5843579556ee8166f23f47: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:48.842660 11151 raft_consensus.cc:385] T 0f9df2bc281f4866b6da5da94a0defc7 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:48.843765 11151 raft_consensus.cc:740] T 0f9df2bc281f4866b6da5da94a0defc7 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 186b12638a87410487467df721a3cbda, State: Initialized, Role: FOLLOWER
I20260709 13:03:48.837656 11152 raft_consensus.cc:359] T c51782cd9d064ec0949e84e24dd4de67 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:48.855780 11235 consensus_queue.cc:1048] T 6f46a5102fd64b45bf1c4dff863d2b4d P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:03:48.860985 10878 raft_consensus.cc:2468] T 7709387f4d3d4f9992d97a0fbca39a40 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 1.
I20260709 13:03:48.862746 11069 leader_election.cc:304] T 7709387f4d3d4f9992d97a0fbca39a40 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda; no voters: 873357742d5843579556ee8166f23f47
I20260709 13:03:48.864323 11235 raft_consensus.cc:2804] T 7709387f4d3d4f9992d97a0fbca39a40 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:03:48.864969 11235 raft_consensus.cc:697] T 7709387f4d3d4f9992d97a0fbca39a40 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 LEADER]: Becoming Leader. State: Replica: 0172fed22b6b4890a7fdc953fa128af4, State: Running, Role: LEADER
I20260709 13:03:48.866606 11235 consensus_queue.cc:237] T 7709387f4d3d4f9992d97a0fbca39a40 P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:48.845016 11151 consensus_queue.cc:260] T 0f9df2bc281f4866b6da5da94a0defc7 P 186b12638a87410487467df721a3cbda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:48.850944 11152 raft_consensus.cc:385] T c51782cd9d064ec0949e84e24dd4de67 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:48.896486 11237 consensus_queue.cc:1048] T 6f46a5102fd64b45bf1c4dff863d2b4d P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260709 13:03:48.871894 10919 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:45807, user_credentials={real_user=slave}} blocked reactor thread for 44794.4us
I20260709 13:03:48.878233 11152 raft_consensus.cc:740] T c51782cd9d064ec0949e84e24dd4de67 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0172fed22b6b4890a7fdc953fa128af4, State: Initialized, Role: FOLLOWER
I20260709 13:03:48.915329 11168 consensus_queue.cc:1048] T d7ce47b9a92d43dd88d95bcda5038597 P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:03:48.918570 11240 raft_consensus.cc:493] T 0f9df2bc281f4866b6da5da94a0defc7 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:48.919204 11240 raft_consensus.cc:515] T 0f9df2bc281f4866b6da5da94a0defc7 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
W20260709 13:03:48.915560 10826 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 13:03:48.930554 11148 tablet_bootstrap.cc:654] T 57b6a2418c1342e7924195e9c1dbd357 P 10ca5ee68ad64b3283b9d235352c9d23: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:48.948410 10953 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f9df2bc281f4866b6da5da94a0defc7" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:03:48.948992 11240 leader_election.cc:290] T 0f9df2bc281f4866b6da5da94a0defc7 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:03:48.930462 11152 consensus_queue.cc:260] T c51782cd9d064ec0949e84e24dd4de67 P 0172fed22b6b4890a7fdc953fa128af4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:48.950066 11101 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f9df2bc281f4866b6da5da94a0defc7" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:03:48.952699 11152 ts_tablet_manager.cc:1434] T c51782cd9d064ec0949e84e24dd4de67 P 0172fed22b6b4890a7fdc953fa128af4: Time spent starting tablet: real 0.137s	user 0.011s	sys 0.000s
I20260709 13:03:48.952888 10845 leader_election.cc:304] T 0f9df2bc281f4866b6da5da94a0defc7 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 186b12638a87410487467df721a3cbda; no voters: 0172fed22b6b4890a7fdc953fa128af4, 10ca5ee68ad64b3283b9d235352c9d23
I20260709 13:03:48.965699 11152 tablet_bootstrap.cc:492] T e5ba72ccd854430baadbc636dfd17136 P 0172fed22b6b4890a7fdc953fa128af4: Bootstrap starting.
I20260709 13:03:48.963162 11240 raft_consensus.cc:2749] T 0f9df2bc281f4866b6da5da94a0defc7 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:03:48.938372 11188 raft_consensus.cc:515] T c51782cd9d064ec0949e84e24dd4de67 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:48.973869 11198 raft_consensus.cc:493] T 7839ca2d2c13431a85f44d7c458c6351 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:48.980940 11198 raft_consensus.cc:515] T 7839ca2d2c13431a85f44d7c458c6351 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:48.984755 10953 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7839ca2d2c13431a85f44d7c458c6351" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:03:48.975772 11151 ts_tablet_manager.cc:1434] T 0f9df2bc281f4866b6da5da94a0defc7 P 186b12638a87410487467df721a3cbda: Time spent starting tablet: real 0.172s	user 0.010s	sys 0.007s
I20260709 13:03:48.986126 11198 leader_election.cc:290] T 7839ca2d2c13431a85f44d7c458c6351 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:03:48.987980 10953 raft_consensus.cc:2468] T 7839ca2d2c13431a85f44d7c458c6351 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 0.
I20260709 13:03:48.989508 11149 tablet_bootstrap.cc:492] T 7709387f4d3d4f9992d97a0fbca39a40 P 873357742d5843579556ee8166f23f47: No bootstrap required, opened a new log
I20260709 13:03:48.990060 11149 ts_tablet_manager.cc:1403] T 7709387f4d3d4f9992d97a0fbca39a40 P 873357742d5843579556ee8166f23f47: Time spent bootstrapping tablet: real 0.161s	user 0.016s	sys 0.006s
I20260709 13:03:48.990787 10992 leader_election.cc:304] T 7839ca2d2c13431a85f44d7c458c6351 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:03:48.990836 10878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7839ca2d2c13431a85f44d7c458c6351" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:03:48.996717 11198 raft_consensus.cc:2804] T 7839ca2d2c13431a85f44d7c458c6351 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:03:48.997352 11198 raft_consensus.cc:493] T 7839ca2d2c13431a85f44d7c458c6351 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:48.997915 11198 raft_consensus.cc:3060] T 7839ca2d2c13431a85f44d7c458c6351 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Advancing to term 1
W20260709 13:03:48.999940 10995 leader_election.cc:343] T 7839ca2d2c13431a85f44d7c458c6351 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 186b12638a87410487467df721a3cbda (127.0.107.193:38967): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:03:49.001667 11188 leader_election.cc:290] T c51782cd9d064ec0949e84e24dd4de67 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 1 election: Requested vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:03:49.001984 10878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c51782cd9d064ec0949e84e24dd4de67" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:03:49.004462 11102 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c51782cd9d064ec0949e84e24dd4de67" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:03:49.004649 10878 raft_consensus.cc:3060] T c51782cd9d064ec0949e84e24dd4de67 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:03:49.005208 11102 raft_consensus.cc:3060] T c51782cd9d064ec0949e84e24dd4de67 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:03:49.010749 11149 raft_consensus.cc:359] T 7709387f4d3d4f9992d97a0fbca39a40 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:49.011585 11149 raft_consensus.cc:385] T 7709387f4d3d4f9992d97a0fbca39a40 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:49.011906 11149 raft_consensus.cc:740] T 7709387f4d3d4f9992d97a0fbca39a40 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 873357742d5843579556ee8166f23f47, State: Initialized, Role: FOLLOWER
I20260709 13:03:49.012701 11149 consensus_queue.cc:260] T 7709387f4d3d4f9992d97a0fbca39a40 P 873357742d5843579556ee8166f23f47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
W20260709 13:03:49.019413 10992 consensus_peers.cc:597] T 57b6a2418c1342e7924195e9c1dbd357 P 873357742d5843579556ee8166f23f47 -> Peer 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903): Couldn't send request to peer 10ca5ee68ad64b3283b9d235352c9d23. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:03:49.021479 11149 ts_tablet_manager.cc:1434] T 7709387f4d3d4f9992d97a0fbca39a40 P 873357742d5843579556ee8166f23f47: Time spent starting tablet: real 0.031s	user 0.008s	sys 0.000s
I20260709 13:03:49.022902 11149 tablet_bootstrap.cc:492] T e5ba72ccd854430baadbc636dfd17136 P 873357742d5843579556ee8166f23f47: Bootstrap starting.
I20260709 13:03:49.028587 11102 raft_consensus.cc:2468] T c51782cd9d064ec0949e84e24dd4de67 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 1.
I20260709 13:03:49.029420 10878 raft_consensus.cc:2468] T c51782cd9d064ec0949e84e24dd4de67 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 1.
I20260709 13:03:49.030335 10920 leader_election.cc:304] T c51782cd9d064ec0949e84e24dd4de67 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 10ca5ee68ad64b3283b9d235352c9d23; no voters: 
I20260709 13:03:49.032038 11238 raft_consensus.cc:2804] T c51782cd9d064ec0949e84e24dd4de67 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:03:49.032858 11238 raft_consensus.cc:697] T c51782cd9d064ec0949e84e24dd4de67 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 LEADER]: Becoming Leader. State: Replica: 10ca5ee68ad64b3283b9d235352c9d23, State: Running, Role: LEADER
I20260709 13:03:49.034160 11238 consensus_queue.cc:237] T c51782cd9d064ec0949e84e24dd4de67 P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:49.042846 11149 tablet_bootstrap.cc:654] T e5ba72ccd854430baadbc636dfd17136 P 873357742d5843579556ee8166f23f47: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:49.057704 11148 tablet_bootstrap.cc:492] T 57b6a2418c1342e7924195e9c1dbd357 P 10ca5ee68ad64b3283b9d235352c9d23: No bootstrap required, opened a new log
I20260709 13:03:49.058503 11148 ts_tablet_manager.cc:1403] T 57b6a2418c1342e7924195e9c1dbd357 P 10ca5ee68ad64b3283b9d235352c9d23: Time spent bootstrapping tablet: real 0.310s	user 0.018s	sys 0.022s
I20260709 13:03:49.061681 11228 consensus_queue.cc:1048] T 57b6a2418c1342e7924195e9c1dbd357 P 873357742d5843579556ee8166f23f47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:03:49.062891 11148 raft_consensus.cc:359] T 57b6a2418c1342e7924195e9c1dbd357 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:49.066043 11148 raft_consensus.cc:385] T 57b6a2418c1342e7924195e9c1dbd357 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:49.067785 11151 tablet_bootstrap.cc:492] T 8a69f32c53f24a50b7852d143665d7b4 P 186b12638a87410487467df721a3cbda: Bootstrap starting.
I20260709 13:03:49.066375 11148 raft_consensus.cc:740] T 57b6a2418c1342e7924195e9c1dbd357 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10ca5ee68ad64b3283b9d235352c9d23, State: Initialized, Role: FOLLOWER
I20260709 13:03:49.073321 11148 consensus_queue.cc:260] T 57b6a2418c1342e7924195e9c1dbd357 P 10ca5ee68ad64b3283b9d235352c9d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:49.079234 11148 ts_tablet_manager.cc:1434] T 57b6a2418c1342e7924195e9c1dbd357 P 10ca5ee68ad64b3283b9d235352c9d23: Time spent starting tablet: real 0.020s	user 0.009s	sys 0.000s
I20260709 13:03:49.080483 11148 tablet_bootstrap.cc:492] T e5ba72ccd854430baadbc636dfd17136 P 10ca5ee68ad64b3283b9d235352c9d23: Bootstrap starting.
I20260709 13:03:49.093006 11228 raft_consensus.cc:493] T 7709387f4d3d4f9992d97a0fbca39a40 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:49.094466 11228 raft_consensus.cc:515] T 7709387f4d3d4f9992d97a0fbca39a40 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:49.095032 10778 catalog_manager.cc:5697] T 7709387f4d3d4f9992d97a0fbca39a40 P 0172fed22b6b4890a7fdc953fa128af4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196). New cstate: current_term: 1 leader_uuid: "0172fed22b6b4890a7fdc953fa128af4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } health_report { overall_health: UNKNOWN } } }
I20260709 13:03:49.097898 11228 leader_election.cc:290] T 7709387f4d3d4f9992d97a0fbca39a40 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:03:49.098804 11102 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7709387f4d3d4f9992d97a0fbca39a40" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:03:49.100224 11148 tablet_bootstrap.cc:654] T e5ba72ccd854430baadbc636dfd17136 P 10ca5ee68ad64b3283b9d235352c9d23: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:49.101109 10877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7709387f4d3d4f9992d97a0fbca39a40" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:03:49.102095 10877 raft_consensus.cc:2393] T 7709387f4d3d4f9992d97a0fbca39a40 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 873357742d5843579556ee8166f23f47 in current term 1: Already voted for candidate 0172fed22b6b4890a7fdc953fa128af4 in this term.
I20260709 13:03:49.109558 11198 raft_consensus.cc:515] T 7839ca2d2c13431a85f44d7c458c6351 P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:49.128475 10955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7839ca2d2c13431a85f44d7c458c6351" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:03:49.129176 10955 raft_consensus.cc:3060] T 7839ca2d2c13431a85f44d7c458c6351 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:03:49.111195 11149 tablet_bootstrap.cc:492] T e5ba72ccd854430baadbc636dfd17136 P 873357742d5843579556ee8166f23f47: No bootstrap required, opened a new log
I20260709 13:03:49.106868 10995 leader_election.cc:304] T 7709387f4d3d4f9992d97a0fbca39a40 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 873357742d5843579556ee8166f23f47; no voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda
I20260709 13:03:49.121840 11151 tablet_bootstrap.cc:654] T 8a69f32c53f24a50b7852d143665d7b4 P 186b12638a87410487467df721a3cbda: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:49.123610 10777 catalog_manager.cc:5697] T c51782cd9d064ec0949e84e24dd4de67 P 10ca5ee68ad64b3283b9d235352c9d23 reported cstate change: term changed from 0 to 1, leader changed from <none> to 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194). New cstate: current_term: 1 leader_uuid: "10ca5ee68ad64b3283b9d235352c9d23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } health_report { overall_health: HEALTHY } } }
I20260709 13:03:49.133623 11234 raft_consensus.cc:3060] T 7709387f4d3d4f9992d97a0fbca39a40 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:03:49.138845 11149 ts_tablet_manager.cc:1403] T e5ba72ccd854430baadbc636dfd17136 P 873357742d5843579556ee8166f23f47: Time spent bootstrapping tablet: real 0.116s	user 0.026s	sys 0.010s
I20260709 13:03:49.142140 11149 raft_consensus.cc:359] T e5ba72ccd854430baadbc636dfd17136 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:49.152357 11149 raft_consensus.cc:385] T e5ba72ccd854430baadbc636dfd17136 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:49.152690 11149 raft_consensus.cc:740] T e5ba72ccd854430baadbc636dfd17136 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 873357742d5843579556ee8166f23f47, State: Initialized, Role: FOLLOWER
I20260709 13:03:49.154997 11149 consensus_queue.cc:260] T e5ba72ccd854430baadbc636dfd17136 P 873357742d5843579556ee8166f23f47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:49.118300 11152 tablet_bootstrap.cc:654] T e5ba72ccd854430baadbc636dfd17136 P 0172fed22b6b4890a7fdc953fa128af4: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:49.168618 10878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7839ca2d2c13431a85f44d7c458c6351" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
W20260709 13:03:49.170745 10995 leader_election.cc:343] T 7839ca2d2c13431a85f44d7c458c6351 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 186b12638a87410487467df721a3cbda (127.0.107.193:38967): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:03:49.173367 11234 raft_consensus.cc:2749] T 7709387f4d3d4f9992d97a0fbca39a40 P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:03:49.173868 11198 leader_election.cc:290] T 7839ca2d2c13431a85f44d7c458c6351 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 1 election: Requested vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:03:49.162029 10955 raft_consensus.cc:2468] T 7839ca2d2c13431a85f44d7c458c6351 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 1.
I20260709 13:03:49.178755 10992 leader_election.cc:304] T 7839ca2d2c13431a85f44d7c458c6351 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47; no voters: 186b12638a87410487467df721a3cbda
I20260709 13:03:49.180116 11198 raft_consensus.cc:2804] T 7839ca2d2c13431a85f44d7c458c6351 P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:03:49.180691 11198 raft_consensus.cc:697] T 7839ca2d2c13431a85f44d7c458c6351 P 873357742d5843579556ee8166f23f47 [term 1 LEADER]: Becoming Leader. State: Replica: 873357742d5843579556ee8166f23f47, State: Running, Role: LEADER
I20260709 13:03:49.181715 11198 consensus_queue.cc:237] T 7839ca2d2c13431a85f44d7c458c6351 P 873357742d5843579556ee8166f23f47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:49.185109 11149 ts_tablet_manager.cc:1434] T e5ba72ccd854430baadbc636dfd17136 P 873357742d5843579556ee8166f23f47: Time spent starting tablet: real 0.046s	user 0.005s	sys 0.002s
I20260709 13:03:49.186365 11149 tablet_bootstrap.cc:492] T 34064dc7bf3747dbb3876dd824ee0688 P 873357742d5843579556ee8166f23f47: Bootstrap starting.
I20260709 13:03:49.194015 11152 tablet_bootstrap.cc:492] T e5ba72ccd854430baadbc636dfd17136 P 0172fed22b6b4890a7fdc953fa128af4: No bootstrap required, opened a new log
I20260709 13:03:49.194357 11151 tablet_bootstrap.cc:492] T 8a69f32c53f24a50b7852d143665d7b4 P 186b12638a87410487467df721a3cbda: No bootstrap required, opened a new log
I20260709 13:03:49.195431 11151 ts_tablet_manager.cc:1403] T 8a69f32c53f24a50b7852d143665d7b4 P 186b12638a87410487467df721a3cbda: Time spent bootstrapping tablet: real 0.128s	user 0.005s	sys 0.021s
I20260709 13:03:49.196533 11148 tablet_bootstrap.cc:492] T e5ba72ccd854430baadbc636dfd17136 P 10ca5ee68ad64b3283b9d235352c9d23: No bootstrap required, opened a new log
I20260709 13:03:49.197120 11148 ts_tablet_manager.cc:1403] T e5ba72ccd854430baadbc636dfd17136 P 10ca5ee68ad64b3283b9d235352c9d23: Time spent bootstrapping tablet: real 0.117s	user 0.027s	sys 0.024s
I20260709 13:03:49.200281 11148 raft_consensus.cc:359] T e5ba72ccd854430baadbc636dfd17136 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:49.200379 11151 raft_consensus.cc:359] T 8a69f32c53f24a50b7852d143665d7b4 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:49.201122 11148 raft_consensus.cc:385] T e5ba72ccd854430baadbc636dfd17136 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:49.201565 11148 raft_consensus.cc:740] T e5ba72ccd854430baadbc636dfd17136 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10ca5ee68ad64b3283b9d235352c9d23, State: Initialized, Role: FOLLOWER
I20260709 13:03:49.201509 11151 raft_consensus.cc:385] T 8a69f32c53f24a50b7852d143665d7b4 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:49.202607 11148 consensus_queue.cc:260] T e5ba72ccd854430baadbc636dfd17136 P 10ca5ee68ad64b3283b9d235352c9d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:49.194650 11152 ts_tablet_manager.cc:1403] T e5ba72ccd854430baadbc636dfd17136 P 0172fed22b6b4890a7fdc953fa128af4: Time spent bootstrapping tablet: real 0.241s	user 0.015s	sys 0.002s
I20260709 13:03:49.206643 11151 raft_consensus.cc:740] T 8a69f32c53f24a50b7852d143665d7b4 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 186b12638a87410487467df721a3cbda, State: Initialized, Role: FOLLOWER
I20260709 13:03:49.206053 11149 tablet_bootstrap.cc:654] T 34064dc7bf3747dbb3876dd824ee0688 P 873357742d5843579556ee8166f23f47: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:49.209740 11152 raft_consensus.cc:359] T e5ba72ccd854430baadbc636dfd17136 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:49.210507 11152 raft_consensus.cc:385] T e5ba72ccd854430baadbc636dfd17136 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:49.210788 11152 raft_consensus.cc:740] T e5ba72ccd854430baadbc636dfd17136 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0172fed22b6b4890a7fdc953fa128af4, State: Initialized, Role: FOLLOWER
I20260709 13:03:49.212297 11151 consensus_queue.cc:260] T 8a69f32c53f24a50b7852d143665d7b4 P 186b12638a87410487467df721a3cbda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:49.215046 10778 catalog_manager.cc:5697] T 7839ca2d2c13431a85f44d7c458c6351 P 873357742d5843579556ee8166f23f47 reported cstate change: term changed from 0 to 1, leader changed from <none> to 873357742d5843579556ee8166f23f47 (127.0.107.195). New cstate: current_term: 1 leader_uuid: "873357742d5843579556ee8166f23f47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } health_report { overall_health: HEALTHY } } }
I20260709 13:03:49.218667 11148 ts_tablet_manager.cc:1434] T e5ba72ccd854430baadbc636dfd17136 P 10ca5ee68ad64b3283b9d235352c9d23: Time spent starting tablet: real 0.021s	user 0.006s	sys 0.004s
I20260709 13:03:49.219796 11148 tablet_bootstrap.cc:492] T 34064dc7bf3747dbb3876dd824ee0688 P 10ca5ee68ad64b3283b9d235352c9d23: Bootstrap starting.
I20260709 13:03:49.221306 11152 consensus_queue.cc:260] T e5ba72ccd854430baadbc636dfd17136 P 0172fed22b6b4890a7fdc953fa128af4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:49.224053 11152 ts_tablet_manager.cc:1434] T e5ba72ccd854430baadbc636dfd17136 P 0172fed22b6b4890a7fdc953fa128af4: Time spent starting tablet: real 0.017s	user 0.005s	sys 0.000s
I20260709 13:03:49.229511 11151 ts_tablet_manager.cc:1434] T 8a69f32c53f24a50b7852d143665d7b4 P 186b12638a87410487467df721a3cbda: Time spent starting tablet: real 0.034s	user 0.006s	sys 0.003s
I20260709 13:03:49.230906 11152 tablet_bootstrap.cc:492] T 525bfdbd6d934afd823f7d3ee3f9d052 P 0172fed22b6b4890a7fdc953fa128af4: Bootstrap starting.
I20260709 13:03:49.237840 11151 tablet_bootstrap.cc:492] T 34064dc7bf3747dbb3876dd824ee0688 P 186b12638a87410487467df721a3cbda: Bootstrap starting.
I20260709 13:03:49.244462 11152 tablet_bootstrap.cc:654] T 525bfdbd6d934afd823f7d3ee3f9d052 P 0172fed22b6b4890a7fdc953fa128af4: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:49.246416 11151 tablet_bootstrap.cc:654] T 34064dc7bf3747dbb3876dd824ee0688 P 186b12638a87410487467df721a3cbda: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:49.255882 11148 tablet_bootstrap.cc:654] T 34064dc7bf3747dbb3876dd824ee0688 P 10ca5ee68ad64b3283b9d235352c9d23: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:49.256866 11149 tablet_bootstrap.cc:492] T 34064dc7bf3747dbb3876dd824ee0688 P 873357742d5843579556ee8166f23f47: No bootstrap required, opened a new log
I20260709 13:03:49.257467 11149 ts_tablet_manager.cc:1403] T 34064dc7bf3747dbb3876dd824ee0688 P 873357742d5843579556ee8166f23f47: Time spent bootstrapping tablet: real 0.071s	user 0.010s	sys 0.005s
I20260709 13:03:49.257660 11151 tablet_bootstrap.cc:492] T 34064dc7bf3747dbb3876dd824ee0688 P 186b12638a87410487467df721a3cbda: No bootstrap required, opened a new log
I20260709 13:03:49.258198 11151 ts_tablet_manager.cc:1403] T 34064dc7bf3747dbb3876dd824ee0688 P 186b12638a87410487467df721a3cbda: Time spent bootstrapping tablet: real 0.021s	user 0.006s	sys 0.006s
I20260709 13:03:49.260308 11238 raft_consensus.cc:493] T 57b6a2418c1342e7924195e9c1dbd357 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:49.260819 11238 raft_consensus.cc:515] T 57b6a2418c1342e7924195e9c1dbd357 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:49.260934 11149 raft_consensus.cc:359] T 34064dc7bf3747dbb3876dd824ee0688 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:49.261780 11149 raft_consensus.cc:385] T 34064dc7bf3747dbb3876dd824ee0688 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:49.261415 11151 raft_consensus.cc:359] T 34064dc7bf3747dbb3876dd824ee0688 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:49.262950 11151 raft_consensus.cc:385] T 34064dc7bf3747dbb3876dd824ee0688 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:49.263242 11151 raft_consensus.cc:740] T 34064dc7bf3747dbb3876dd824ee0688 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 186b12638a87410487467df721a3cbda, State: Initialized, Role: FOLLOWER
I20260709 13:03:49.264127 11151 consensus_queue.cc:260] T 34064dc7bf3747dbb3876dd824ee0688 P 186b12638a87410487467df721a3cbda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:49.264770 11029 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57b6a2418c1342e7924195e9c1dbd357" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:03:49.266789 11149 raft_consensus.cc:740] T 34064dc7bf3747dbb3876dd824ee0688 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 873357742d5843579556ee8166f23f47, State: Initialized, Role: FOLLOWER
I20260709 13:03:49.267261 10877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57b6a2418c1342e7924195e9c1dbd357" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:03:49.270195 11151 ts_tablet_manager.cc:1434] T 34064dc7bf3747dbb3876dd824ee0688 P 186b12638a87410487467df721a3cbda: Time spent starting tablet: real 0.012s	user 0.004s	sys 0.002s
I20260709 13:03:49.271442 11151 tablet_bootstrap.cc:492] T 4f9eb06ed01f4ce68be3af8a8434665b P 186b12638a87410487467df721a3cbda: Bootstrap starting.
I20260709 13:03:49.267995 11149 consensus_queue.cc:260] T 34064dc7bf3747dbb3876dd824ee0688 P 873357742d5843579556ee8166f23f47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:49.275090 10921 leader_election.cc:304] T 57b6a2418c1342e7924195e9c1dbd357 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23; no voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47
I20260709 13:03:49.276090 11188 raft_consensus.cc:2749] T 57b6a2418c1342e7924195e9c1dbd357 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:03:49.278599 11151 tablet_bootstrap.cc:654] T 4f9eb06ed01f4ce68be3af8a8434665b P 186b12638a87410487467df721a3cbda: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:49.279886 11196 raft_consensus.cc:493] T e5ba72ccd854430baadbc636dfd17136 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:49.280442 11196 raft_consensus.cc:515] T e5ba72ccd854430baadbc636dfd17136 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:49.283738 11149 ts_tablet_manager.cc:1434] T 34064dc7bf3747dbb3876dd824ee0688 P 873357742d5843579556ee8166f23f47: Time spent starting tablet: real 0.026s	user 0.007s	sys 0.001s
I20260709 13:03:49.283738 10955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5ba72ccd854430baadbc636dfd17136" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:03:49.284593 10955 raft_consensus.cc:2468] T e5ba72ccd854430baadbc636dfd17136 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 0.
I20260709 13:03:49.286039 11066 leader_election.cc:304] T e5ba72ccd854430baadbc636dfd17136 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 10ca5ee68ad64b3283b9d235352c9d23; no voters: 
I20260709 13:03:49.286345 11152 tablet_bootstrap.cc:492] T 525bfdbd6d934afd823f7d3ee3f9d052 P 0172fed22b6b4890a7fdc953fa128af4: No bootstrap required, opened a new log
I20260709 13:03:49.286991 11152 ts_tablet_manager.cc:1403] T 525bfdbd6d934afd823f7d3ee3f9d052 P 0172fed22b6b4890a7fdc953fa128af4: Time spent bootstrapping tablet: real 0.056s	user 0.015s	sys 0.002s
I20260709 13:03:49.287053 11235 raft_consensus.cc:2804] T e5ba72ccd854430baadbc636dfd17136 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:03:49.287554 11235 raft_consensus.cc:493] T e5ba72ccd854430baadbc636dfd17136 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:49.287904 11235 raft_consensus.cc:3060] T e5ba72ccd854430baadbc636dfd17136 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:03:49.290483 11152 raft_consensus.cc:359] T 525bfdbd6d934afd823f7d3ee3f9d052 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:49.291241 11152 raft_consensus.cc:385] T 525bfdbd6d934afd823f7d3ee3f9d052 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:49.291319 11238 leader_election.cc:290] T 57b6a2418c1342e7924195e9c1dbd357 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:03:49.291594 11152 raft_consensus.cc:740] T 525bfdbd6d934afd823f7d3ee3f9d052 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0172fed22b6b4890a7fdc953fa128af4, State: Initialized, Role: FOLLOWER
I20260709 13:03:49.290841 11196 leader_election.cc:290] T e5ba72ccd854430baadbc636dfd17136 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:03:49.292016 11029 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5ba72ccd854430baadbc636dfd17136" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:03:49.292450 11152 consensus_queue.cc:260] T 525bfdbd6d934afd823f7d3ee3f9d052 P 0172fed22b6b4890a7fdc953fa128af4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:49.293085 11029 raft_consensus.cc:2468] T e5ba72ccd854430baadbc636dfd17136 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 0.
I20260709 13:03:49.296612 11149 tablet_bootstrap.cc:492] T d71ca3a1ba6b48478aaf62741bf3cf6a P 873357742d5843579556ee8166f23f47: Bootstrap starting.
I20260709 13:03:49.303210 11152 ts_tablet_manager.cc:1434] T 525bfdbd6d934afd823f7d3ee3f9d052 P 0172fed22b6b4890a7fdc953fa128af4: Time spent starting tablet: real 0.016s	user 0.008s	sys 0.000s
I20260709 13:03:49.304394 11149 tablet_bootstrap.cc:654] T d71ca3a1ba6b48478aaf62741bf3cf6a P 873357742d5843579556ee8166f23f47: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:49.304534 11152 tablet_bootstrap.cc:492] T 7757ecbf402e47bfbc93d0447741e26c P 0172fed22b6b4890a7fdc953fa128af4: Bootstrap starting.
I20260709 13:03:49.307587 11235 raft_consensus.cc:515] T e5ba72ccd854430baadbc636dfd17136 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:49.310995 10955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5ba72ccd854430baadbc636dfd17136" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:03:49.311619 10955 raft_consensus.cc:3060] T e5ba72ccd854430baadbc636dfd17136 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:03:49.312428 11152 tablet_bootstrap.cc:654] T 7757ecbf402e47bfbc93d0447741e26c P 0172fed22b6b4890a7fdc953fa128af4: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:49.328801 11148 tablet_bootstrap.cc:492] T 34064dc7bf3747dbb3876dd824ee0688 P 10ca5ee68ad64b3283b9d235352c9d23: No bootstrap required, opened a new log
I20260709 13:03:49.323475 11029 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5ba72ccd854430baadbc636dfd17136" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
I20260709 13:03:49.329528 11148 ts_tablet_manager.cc:1403] T 34064dc7bf3747dbb3876dd824ee0688 P 10ca5ee68ad64b3283b9d235352c9d23: Time spent bootstrapping tablet: real 0.110s	user 0.013s	sys 0.003s
I20260709 13:03:49.329643 11029 raft_consensus.cc:3060] T e5ba72ccd854430baadbc636dfd17136 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:03:49.326995 11149 tablet_bootstrap.cc:492] T d71ca3a1ba6b48478aaf62741bf3cf6a P 873357742d5843579556ee8166f23f47: No bootstrap required, opened a new log
I20260709 13:03:49.330706 11149 ts_tablet_manager.cc:1403] T d71ca3a1ba6b48478aaf62741bf3cf6a P 873357742d5843579556ee8166f23f47: Time spent bootstrapping tablet: real 0.034s	user 0.018s	sys 0.003s
I20260709 13:03:49.331552 11152 tablet_bootstrap.cc:492] T 7757ecbf402e47bfbc93d0447741e26c P 0172fed22b6b4890a7fdc953fa128af4: No bootstrap required, opened a new log
I20260709 13:03:49.332100 11152 ts_tablet_manager.cc:1403] T 7757ecbf402e47bfbc93d0447741e26c P 0172fed22b6b4890a7fdc953fa128af4: Time spent bootstrapping tablet: real 0.028s	user 0.012s	sys 0.000s
I20260709 13:03:49.316849 11151 tablet_bootstrap.cc:492] T 4f9eb06ed01f4ce68be3af8a8434665b P 186b12638a87410487467df721a3cbda: No bootstrap required, opened a new log
I20260709 13:03:49.334280 11149 raft_consensus.cc:359] T d71ca3a1ba6b48478aaf62741bf3cf6a P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:49.335014 11151 ts_tablet_manager.cc:1403] T 4f9eb06ed01f4ce68be3af8a8434665b P 186b12638a87410487467df721a3cbda: Time spent bootstrapping tablet: real 0.064s	user 0.014s	sys 0.016s
I20260709 13:03:49.335040 11149 raft_consensus.cc:385] T d71ca3a1ba6b48478aaf62741bf3cf6a P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:49.335528 11149 raft_consensus.cc:740] T d71ca3a1ba6b48478aaf62741bf3cf6a P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 873357742d5843579556ee8166f23f47, State: Initialized, Role: FOLLOWER
I20260709 13:03:49.336333 11149 consensus_queue.cc:260] T d71ca3a1ba6b48478aaf62741bf3cf6a P 873357742d5843579556ee8166f23f47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:49.323155 10955 raft_consensus.cc:2468] T e5ba72ccd854430baadbc636dfd17136 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 1.
I20260709 13:03:49.338595 11029 raft_consensus.cc:2468] T e5ba72ccd854430baadbc636dfd17136 P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 1.
I20260709 13:03:49.339051 11066 leader_election.cc:304] T e5ba72ccd854430baadbc636dfd17136 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 10ca5ee68ad64b3283b9d235352c9d23; no voters: 
I20260709 13:03:49.340282 11231 consensus_queue.cc:1048] T 7709387f4d3d4f9992d97a0fbca39a40 P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:03:49.340618 11237 raft_consensus.cc:2804] T e5ba72ccd854430baadbc636dfd17136 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:03:49.341212 11237 raft_consensus.cc:697] T e5ba72ccd854430baadbc636dfd17136 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 LEADER]: Becoming Leader. State: Replica: 0172fed22b6b4890a7fdc953fa128af4, State: Running, Role: LEADER
I20260709 13:03:49.342100 11237 consensus_queue.cc:237] T e5ba72ccd854430baadbc636dfd17136 P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:49.343493 11149 ts_tablet_manager.cc:1434] T d71ca3a1ba6b48478aaf62741bf3cf6a P 873357742d5843579556ee8166f23f47: Time spent starting tablet: real 0.013s	user 0.007s	sys 0.000s
I20260709 13:03:49.344419 11149 tablet_bootstrap.cc:492] T 7757ecbf402e47bfbc93d0447741e26c P 873357742d5843579556ee8166f23f47: Bootstrap starting.
I20260709 13:03:49.347312 11148 raft_consensus.cc:359] T 34064dc7bf3747dbb3876dd824ee0688 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:49.348032 11148 raft_consensus.cc:385] T 34064dc7bf3747dbb3876dd824ee0688 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:49.348317 11148 raft_consensus.cc:740] T 34064dc7bf3747dbb3876dd824ee0688 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10ca5ee68ad64b3283b9d235352c9d23, State: Initialized, Role: FOLLOWER
I20260709 13:03:49.322150 11235 leader_election.cc:290] T e5ba72ccd854430baadbc636dfd17136 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 1 election: Requested vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:03:49.354004 11149 tablet_bootstrap.cc:654] T 7757ecbf402e47bfbc93d0447741e26c P 873357742d5843579556ee8166f23f47: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:49.354553 11152 raft_consensus.cc:359] T 7757ecbf402e47bfbc93d0447741e26c P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:49.355304 11152 raft_consensus.cc:385] T 7757ecbf402e47bfbc93d0447741e26c P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:49.355577 11152 raft_consensus.cc:740] T 7757ecbf402e47bfbc93d0447741e26c P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0172fed22b6b4890a7fdc953fa128af4, State: Initialized, Role: FOLLOWER
I20260709 13:03:49.356295 11152 consensus_queue.cc:260] T 7757ecbf402e47bfbc93d0447741e26c P 0172fed22b6b4890a7fdc953fa128af4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:49.357759 11148 consensus_queue.cc:260] T 34064dc7bf3747dbb3876dd824ee0688 P 10ca5ee68ad64b3283b9d235352c9d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:49.360198 11148 ts_tablet_manager.cc:1434] T 34064dc7bf3747dbb3876dd824ee0688 P 10ca5ee68ad64b3283b9d235352c9d23: Time spent starting tablet: real 0.030s	user 0.003s	sys 0.004s
I20260709 13:03:49.361359 11148 tablet_bootstrap.cc:492] T 8a69f32c53f24a50b7852d143665d7b4 P 10ca5ee68ad64b3283b9d235352c9d23: Bootstrap starting.
I20260709 13:03:49.368613 11148 tablet_bootstrap.cc:654] T 8a69f32c53f24a50b7852d143665d7b4 P 10ca5ee68ad64b3283b9d235352c9d23: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:49.369504 11152 ts_tablet_manager.cc:1434] T 7757ecbf402e47bfbc93d0447741e26c P 0172fed22b6b4890a7fdc953fa128af4: Time spent starting tablet: real 0.026s	user 0.005s	sys 0.000s
I20260709 13:03:49.370893 11152 tablet_bootstrap.cc:492] T 8a69f32c53f24a50b7852d143665d7b4 P 0172fed22b6b4890a7fdc953fa128af4: Bootstrap starting.
I20260709 13:03:49.371025 11162 consensus_queue.cc:1048] T 7709387f4d3d4f9992d97a0fbca39a40 P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:03:49.374372 11149 tablet_bootstrap.cc:492] T 7757ecbf402e47bfbc93d0447741e26c P 873357742d5843579556ee8166f23f47: No bootstrap required, opened a new log
I20260709 13:03:49.374989 11149 ts_tablet_manager.cc:1403] T 7757ecbf402e47bfbc93d0447741e26c P 873357742d5843579556ee8166f23f47: Time spent bootstrapping tablet: real 0.031s	user 0.009s	sys 0.004s
I20260709 13:03:49.375315 11240 raft_consensus.cc:493] T 8a69f32c53f24a50b7852d143665d7b4 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:49.375986 11240 raft_consensus.cc:515] T 8a69f32c53f24a50b7852d143665d7b4 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:49.379398 11152 tablet_bootstrap.cc:654] T 8a69f32c53f24a50b7852d143665d7b4 P 0172fed22b6b4890a7fdc953fa128af4: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:49.379643 11102 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a69f32c53f24a50b7852d143665d7b4" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:03:49.379307 11149 raft_consensus.cc:359] T 7757ecbf402e47bfbc93d0447741e26c P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
W20260709 13:03:49.381767 10845 leader_election.cc:343] T 8a69f32c53f24a50b7852d143665d7b4 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:03:49.383982 11149 raft_consensus.cc:385] T 7757ecbf402e47bfbc93d0447741e26c P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:49.385664 11149 raft_consensus.cc:740] T 7757ecbf402e47bfbc93d0447741e26c P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 873357742d5843579556ee8166f23f47, State: Initialized, Role: FOLLOWER
I20260709 13:03:49.386662 11149 consensus_queue.cc:260] T 7757ecbf402e47bfbc93d0447741e26c P 873357742d5843579556ee8166f23f47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:49.388633 11240 leader_election.cc:290] T 8a69f32c53f24a50b7852d143665d7b4 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:03:49.389052 11149 ts_tablet_manager.cc:1434] T 7757ecbf402e47bfbc93d0447741e26c P 873357742d5843579556ee8166f23f47: Time spent starting tablet: real 0.014s	user 0.006s	sys 0.001s
I20260709 13:03:49.389863 10953 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a69f32c53f24a50b7852d143665d7b4" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:03:49.390143 11149 tablet_bootstrap.cc:492] T 9a1716c0adf34bee8a4e99341ba5788f P 873357742d5843579556ee8166f23f47: Bootstrap starting.
W20260709 13:03:49.391820 10843 leader_election.cc:343] T 8a69f32c53f24a50b7852d143665d7b4 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:03:49.392534 10843 leader_election.cc:304] T 8a69f32c53f24a50b7852d143665d7b4 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 186b12638a87410487467df721a3cbda; no voters: 0172fed22b6b4890a7fdc953fa128af4, 10ca5ee68ad64b3283b9d235352c9d23
I20260709 13:03:49.394284 11148 tablet_bootstrap.cc:492] T 8a69f32c53f24a50b7852d143665d7b4 P 10ca5ee68ad64b3283b9d235352c9d23: No bootstrap required, opened a new log
I20260709 13:03:49.394865 11148 ts_tablet_manager.cc:1403] T 8a69f32c53f24a50b7852d143665d7b4 P 10ca5ee68ad64b3283b9d235352c9d23: Time spent bootstrapping tablet: real 0.034s	user 0.013s	sys 0.000s
I20260709 13:03:49.398679 10776 catalog_manager.cc:5697] T e5ba72ccd854430baadbc636dfd17136 P 0172fed22b6b4890a7fdc953fa128af4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196). New cstate: current_term: 1 leader_uuid: "0172fed22b6b4890a7fdc953fa128af4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } health_report { overall_health: HEALTHY } } }
I20260709 13:03:49.399556 11240 raft_consensus.cc:2749] T 8a69f32c53f24a50b7852d143665d7b4 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:03:49.402798 11151 raft_consensus.cc:359] T 4f9eb06ed01f4ce68be3af8a8434665b P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:49.405337 11151 raft_consensus.cc:385] T 4f9eb06ed01f4ce68be3af8a8434665b P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:49.405782 11151 raft_consensus.cc:740] T 4f9eb06ed01f4ce68be3af8a8434665b P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 186b12638a87410487467df721a3cbda, State: Initialized, Role: FOLLOWER
I20260709 13:03:49.407030 11151 consensus_queue.cc:260] T 4f9eb06ed01f4ce68be3af8a8434665b P 186b12638a87410487467df721a3cbda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:49.411223 11151 ts_tablet_manager.cc:1434] T 4f9eb06ed01f4ce68be3af8a8434665b P 186b12638a87410487467df721a3cbda: Time spent starting tablet: real 0.076s	user 0.007s	sys 0.001s
I20260709 13:03:49.412693 11151 tablet_bootstrap.cc:492] T 7757ecbf402e47bfbc93d0447741e26c P 186b12638a87410487467df721a3cbda: Bootstrap starting.
I20260709 13:03:49.417444 10877 raft_consensus.cc:3060] T 4f9eb06ed01f4ce68be3af8a8434665b P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:03:49.398216 11148 raft_consensus.cc:359] T 8a69f32c53f24a50b7852d143665d7b4 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:49.424775 11148 raft_consensus.cc:385] T 8a69f32c53f24a50b7852d143665d7b4 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:49.425109 11148 raft_consensus.cc:740] T 8a69f32c53f24a50b7852d143665d7b4 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10ca5ee68ad64b3283b9d235352c9d23, State: Initialized, Role: FOLLOWER
I20260709 13:03:49.425909 11149 tablet_bootstrap.cc:654] T 9a1716c0adf34bee8a4e99341ba5788f P 873357742d5843579556ee8166f23f47: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:49.427561 11151 tablet_bootstrap.cc:654] T 7757ecbf402e47bfbc93d0447741e26c P 186b12638a87410487467df721a3cbda: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:49.430402 11148 consensus_queue.cc:260] T 8a69f32c53f24a50b7852d143665d7b4 P 10ca5ee68ad64b3283b9d235352c9d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:49.435771 11152 tablet_bootstrap.cc:492] T 8a69f32c53f24a50b7852d143665d7b4 P 0172fed22b6b4890a7fdc953fa128af4: No bootstrap required, opened a new log
I20260709 13:03:49.436307 11152 ts_tablet_manager.cc:1403] T 8a69f32c53f24a50b7852d143665d7b4 P 0172fed22b6b4890a7fdc953fa128af4: Time spent bootstrapping tablet: real 0.066s	user 0.022s	sys 0.022s
I20260709 13:03:49.436692 11148 ts_tablet_manager.cc:1434] T 8a69f32c53f24a50b7852d143665d7b4 P 10ca5ee68ad64b3283b9d235352c9d23: Time spent starting tablet: real 0.042s	user 0.008s	sys 0.000s
I20260709 13:03:49.438319 11148 tablet_bootstrap.cc:492] T e96e80ba9ac34d0998395515e5c8c41a P 10ca5ee68ad64b3283b9d235352c9d23: Bootstrap starting.
I20260709 13:03:49.441479 11152 raft_consensus.cc:359] T 8a69f32c53f24a50b7852d143665d7b4 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:49.442828 11152 raft_consensus.cc:385] T 8a69f32c53f24a50b7852d143665d7b4 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:49.443117 11152 raft_consensus.cc:740] T 8a69f32c53f24a50b7852d143665d7b4 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0172fed22b6b4890a7fdc953fa128af4, State: Initialized, Role: FOLLOWER
I20260709 13:03:49.445710 10878 raft_consensus.cc:3060] T 0f9df2bc281f4866b6da5da94a0defc7 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:03:49.445247 11152 consensus_queue.cc:260] T 8a69f32c53f24a50b7852d143665d7b4 P 0172fed22b6b4890a7fdc953fa128af4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:49.447009 11148 tablet_bootstrap.cc:654] T e96e80ba9ac34d0998395515e5c8c41a P 10ca5ee68ad64b3283b9d235352c9d23: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:49.448525 11152 ts_tablet_manager.cc:1434] T 8a69f32c53f24a50b7852d143665d7b4 P 0172fed22b6b4890a7fdc953fa128af4: Time spent starting tablet: real 0.012s	user 0.008s	sys 0.001s
I20260709 13:03:49.449769 11152 tablet_bootstrap.cc:492] T 836251ebc2b74f5da88b8a8b2338e2a5 P 0172fed22b6b4890a7fdc953fa128af4: Bootstrap starting.
I20260709 13:03:49.451524 11151 tablet_bootstrap.cc:492] T 7757ecbf402e47bfbc93d0447741e26c P 186b12638a87410487467df721a3cbda: No bootstrap required, opened a new log
I20260709 13:03:49.452021 11151 ts_tablet_manager.cc:1403] T 7757ecbf402e47bfbc93d0447741e26c P 186b12638a87410487467df721a3cbda: Time spent bootstrapping tablet: real 0.040s	user 0.010s	sys 0.012s
I20260709 13:03:49.455404 11151 raft_consensus.cc:359] T 7757ecbf402e47bfbc93d0447741e26c P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:49.456099 11151 raft_consensus.cc:385] T 7757ecbf402e47bfbc93d0447741e26c P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:49.456686 11151 raft_consensus.cc:740] T 7757ecbf402e47bfbc93d0447741e26c P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 186b12638a87410487467df721a3cbda, State: Initialized, Role: FOLLOWER
I20260709 13:03:49.458329 11152 tablet_bootstrap.cc:654] T 836251ebc2b74f5da88b8a8b2338e2a5 P 0172fed22b6b4890a7fdc953fa128af4: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:49.457546 11151 consensus_queue.cc:260] T 7757ecbf402e47bfbc93d0447741e26c P 186b12638a87410487467df721a3cbda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:49.464159 11151 ts_tablet_manager.cc:1434] T 7757ecbf402e47bfbc93d0447741e26c P 186b12638a87410487467df721a3cbda: Time spent starting tablet: real 0.012s	user 0.008s	sys 0.000s
I20260709 13:03:49.468253 11151 tablet_bootstrap.cc:492] T 6a9353d1d230416296627d7cf8b1d425 P 186b12638a87410487467df721a3cbda: Bootstrap starting.
I20260709 13:03:49.469689 11238 consensus_queue.cc:1048] T c51782cd9d064ec0949e84e24dd4de67 P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:03:49.477430 11148 tablet_bootstrap.cc:492] T e96e80ba9ac34d0998395515e5c8c41a P 10ca5ee68ad64b3283b9d235352c9d23: No bootstrap required, opened a new log
I20260709 13:03:49.478158 11148 ts_tablet_manager.cc:1403] T e96e80ba9ac34d0998395515e5c8c41a P 10ca5ee68ad64b3283b9d235352c9d23: Time spent bootstrapping tablet: real 0.040s	user 0.011s	sys 0.000s
I20260709 13:03:49.483999 11149 tablet_bootstrap.cc:492] T 9a1716c0adf34bee8a4e99341ba5788f P 873357742d5843579556ee8166f23f47: No bootstrap required, opened a new log
I20260709 13:03:49.486608 11148 raft_consensus.cc:359] T e96e80ba9ac34d0998395515e5c8c41a P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:49.489701 11151 tablet_bootstrap.cc:654] T 6a9353d1d230416296627d7cf8b1d425 P 186b12638a87410487467df721a3cbda: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:49.489827 11148 raft_consensus.cc:385] T e96e80ba9ac34d0998395515e5c8c41a P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:49.490352 11148 raft_consensus.cc:740] T e96e80ba9ac34d0998395515e5c8c41a P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10ca5ee68ad64b3283b9d235352c9d23, State: Initialized, Role: FOLLOWER
I20260709 13:03:49.491237 11148 consensus_queue.cc:260] T e96e80ba9ac34d0998395515e5c8c41a P 10ca5ee68ad64b3283b9d235352c9d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:49.491973 11149 ts_tablet_manager.cc:1403] T 9a1716c0adf34bee8a4e99341ba5788f P 873357742d5843579556ee8166f23f47: Time spent bootstrapping tablet: real 0.100s	user 0.014s	sys 0.008s
I20260709 13:03:49.494884 11238 consensus_queue.cc:1048] T c51782cd9d064ec0949e84e24dd4de67 P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:03:49.496606 11152 tablet_bootstrap.cc:492] T 836251ebc2b74f5da88b8a8b2338e2a5 P 0172fed22b6b4890a7fdc953fa128af4: No bootstrap required, opened a new log
I20260709 13:03:49.497195 11152 ts_tablet_manager.cc:1403] T 836251ebc2b74f5da88b8a8b2338e2a5 P 0172fed22b6b4890a7fdc953fa128af4: Time spent bootstrapping tablet: real 0.048s	user 0.024s	sys 0.004s
I20260709 13:03:49.497442 11149 raft_consensus.cc:359] T 9a1716c0adf34bee8a4e99341ba5788f P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:49.498221 11149 raft_consensus.cc:385] T 9a1716c0adf34bee8a4e99341ba5788f P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:49.498503 11149 raft_consensus.cc:740] T 9a1716c0adf34bee8a4e99341ba5788f P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 873357742d5843579556ee8166f23f47, State: Initialized, Role: FOLLOWER
I20260709 13:03:49.499233 11149 consensus_queue.cc:260] T 9a1716c0adf34bee8a4e99341ba5788f P 873357742d5843579556ee8166f23f47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:49.500851 11152 raft_consensus.cc:359] T 836251ebc2b74f5da88b8a8b2338e2a5 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:49.501581 11152 raft_consensus.cc:385] T 836251ebc2b74f5da88b8a8b2338e2a5 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:49.501850 11152 raft_consensus.cc:740] T 836251ebc2b74f5da88b8a8b2338e2a5 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0172fed22b6b4890a7fdc953fa128af4, State: Initialized, Role: FOLLOWER
I20260709 13:03:49.502604 11148 ts_tablet_manager.cc:1434] T e96e80ba9ac34d0998395515e5c8c41a P 10ca5ee68ad64b3283b9d235352c9d23: Time spent starting tablet: real 0.024s	user 0.008s	sys 0.000s
I20260709 13:03:49.502632 11152 consensus_queue.cc:260] T 836251ebc2b74f5da88b8a8b2338e2a5 P 0172fed22b6b4890a7fdc953fa128af4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:49.503875 11148 tablet_bootstrap.cc:492] T d71ca3a1ba6b48478aaf62741bf3cf6a P 10ca5ee68ad64b3283b9d235352c9d23: Bootstrap starting.
I20260709 13:03:49.505291 11152 ts_tablet_manager.cc:1434] T 836251ebc2b74f5da88b8a8b2338e2a5 P 0172fed22b6b4890a7fdc953fa128af4: Time spent starting tablet: real 0.008s	user 0.008s	sys 0.000s
I20260709 13:03:49.505954 11149 ts_tablet_manager.cc:1434] T 9a1716c0adf34bee8a4e99341ba5788f P 873357742d5843579556ee8166f23f47: Time spent starting tablet: real 0.013s	user 0.006s	sys 0.000s
I20260709 13:03:49.506280 11152 tablet_bootstrap.cc:492] T d71ca3a1ba6b48478aaf62741bf3cf6a P 0172fed22b6b4890a7fdc953fa128af4: Bootstrap starting.
I20260709 13:03:49.512472 11148 tablet_bootstrap.cc:654] T d71ca3a1ba6b48478aaf62741bf3cf6a P 10ca5ee68ad64b3283b9d235352c9d23: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:49.513738 11152 tablet_bootstrap.cc:654] T d71ca3a1ba6b48478aaf62741bf3cf6a P 0172fed22b6b4890a7fdc953fa128af4: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:49.521523 11149 tablet_bootstrap.cc:492] T e96e80ba9ac34d0998395515e5c8c41a P 873357742d5843579556ee8166f23f47: Bootstrap starting.
I20260709 13:03:49.529573 11149 tablet_bootstrap.cc:654] T e96e80ba9ac34d0998395515e5c8c41a P 873357742d5843579556ee8166f23f47: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:49.517935 11235 raft_consensus.cc:493] T 525bfdbd6d934afd823f7d3ee3f9d052 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:49.530328 11235 raft_consensus.cc:515] T 525bfdbd6d934afd823f7d3ee3f9d052 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:49.534365 11235 leader_election.cc:290] T 525bfdbd6d934afd823f7d3ee3f9d052 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:03:49.536314 10955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "525bfdbd6d934afd823f7d3ee3f9d052" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:03:49.536132 11029 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "525bfdbd6d934afd823f7d3ee3f9d052" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:03:49.540055 11067 leader_election.cc:304] T 525bfdbd6d934afd823f7d3ee3f9d052 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47
I20260709 13:03:49.541658 11151 tablet_bootstrap.cc:492] T 6a9353d1d230416296627d7cf8b1d425 P 186b12638a87410487467df721a3cbda: No bootstrap required, opened a new log
I20260709 13:03:49.542358 11151 ts_tablet_manager.cc:1403] T 6a9353d1d230416296627d7cf8b1d425 P 186b12638a87410487467df721a3cbda: Time spent bootstrapping tablet: real 0.074s	user 0.005s	sys 0.012s
I20260709 13:03:49.543125 11148 tablet_bootstrap.cc:492] T d71ca3a1ba6b48478aaf62741bf3cf6a P 10ca5ee68ad64b3283b9d235352c9d23: No bootstrap required, opened a new log
I20260709 13:03:49.543743 11148 ts_tablet_manager.cc:1403] T d71ca3a1ba6b48478aaf62741bf3cf6a P 10ca5ee68ad64b3283b9d235352c9d23: Time spent bootstrapping tablet: real 0.040s	user 0.016s	sys 0.001s
I20260709 13:03:49.541586 11235 raft_consensus.cc:2749] T 525bfdbd6d934afd823f7d3ee3f9d052 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:03:49.546830 11151 raft_consensus.cc:359] T 6a9353d1d230416296627d7cf8b1d425 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:49.549933 11151 raft_consensus.cc:385] T 6a9353d1d230416296627d7cf8b1d425 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:49.550305 11151 raft_consensus.cc:740] T 6a9353d1d230416296627d7cf8b1d425 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 186b12638a87410487467df721a3cbda, State: Initialized, Role: FOLLOWER
I20260709 13:03:49.551584 11151 consensus_queue.cc:260] T 6a9353d1d230416296627d7cf8b1d425 P 186b12638a87410487467df721a3cbda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:49.554152 11151 ts_tablet_manager.cc:1434] T 6a9353d1d230416296627d7cf8b1d425 P 186b12638a87410487467df721a3cbda: Time spent starting tablet: real 0.011s	user 0.007s	sys 0.000s
I20260709 13:03:49.555265 11151 tablet_bootstrap.cc:492] T 7839ca2d2c13431a85f44d7c458c6351 P 186b12638a87410487467df721a3cbda: Bootstrap starting.
I20260709 13:03:49.555801 11148 raft_consensus.cc:359] T d71ca3a1ba6b48478aaf62741bf3cf6a P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:49.556627 11148 raft_consensus.cc:385] T d71ca3a1ba6b48478aaf62741bf3cf6a P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:49.556972 11148 raft_consensus.cc:740] T d71ca3a1ba6b48478aaf62741bf3cf6a P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10ca5ee68ad64b3283b9d235352c9d23, State: Initialized, Role: FOLLOWER
I20260709 13:03:49.557825 11240 raft_consensus.cc:493] T 34064dc7bf3747dbb3876dd824ee0688 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:49.557770 11148 consensus_queue.cc:260] T d71ca3a1ba6b48478aaf62741bf3cf6a P 10ca5ee68ad64b3283b9d235352c9d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:49.558398 11240 raft_consensus.cc:515] T 34064dc7bf3747dbb3876dd824ee0688 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:49.561694 11240 leader_election.cc:290] T 34064dc7bf3747dbb3876dd824ee0688 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:03:49.562893 11149 tablet_bootstrap.cc:492] T e96e80ba9ac34d0998395515e5c8c41a P 873357742d5843579556ee8166f23f47: No bootstrap required, opened a new log
I20260709 13:03:49.598066 11149 ts_tablet_manager.cc:1403] T e96e80ba9ac34d0998395515e5c8c41a P 873357742d5843579556ee8166f23f47: Time spent bootstrapping tablet: real 0.091s	user 0.023s	sys 0.017s
I20260709 13:03:49.598838 10955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34064dc7bf3747dbb3876dd824ee0688" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:03:49.599646 10955 raft_consensus.cc:2468] T 34064dc7bf3747dbb3876dd824ee0688 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 0.
I20260709 13:03:49.600822 10843 leader_election.cc:304] T 34064dc7bf3747dbb3876dd824ee0688 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda; no voters: 
I20260709 13:03:49.601822 11240 raft_consensus.cc:2804] T 34064dc7bf3747dbb3876dd824ee0688 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:03:49.601642 11149 raft_consensus.cc:359] T e96e80ba9ac34d0998395515e5c8c41a P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:49.602339 11240 raft_consensus.cc:493] T 34064dc7bf3747dbb3876dd824ee0688 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:49.563868 11148 ts_tablet_manager.cc:1434] T d71ca3a1ba6b48478aaf62741bf3cf6a P 10ca5ee68ad64b3283b9d235352c9d23: Time spent starting tablet: real 0.020s	user 0.006s	sys 0.001s
I20260709 13:03:49.602865 11240 raft_consensus.cc:3060] T 34064dc7bf3747dbb3876dd824ee0688 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Advancing to term 1
W20260709 13:03:49.603502 10995 consensus_peers.cc:597] T 7839ca2d2c13431a85f44d7c458c6351 P 873357742d5843579556ee8166f23f47 -> Peer 186b12638a87410487467df721a3cbda (127.0.107.193:38967): Couldn't send request to peer 186b12638a87410487467df721a3cbda. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:03:49.603910 11148 tablet_bootstrap.cc:492] T 836251ebc2b74f5da88b8a8b2338e2a5 P 10ca5ee68ad64b3283b9d235352c9d23: Bootstrap starting.
I20260709 13:03:49.602499 11149 raft_consensus.cc:385] T e96e80ba9ac34d0998395515e5c8c41a P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:49.605000 11149 raft_consensus.cc:740] T e96e80ba9ac34d0998395515e5c8c41a P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 873357742d5843579556ee8166f23f47, State: Initialized, Role: FOLLOWER
I20260709 13:03:49.605479 11152 tablet_bootstrap.cc:492] T d71ca3a1ba6b48478aaf62741bf3cf6a P 0172fed22b6b4890a7fdc953fa128af4: No bootstrap required, opened a new log
I20260709 13:03:49.606035 11152 ts_tablet_manager.cc:1403] T d71ca3a1ba6b48478aaf62741bf3cf6a P 0172fed22b6b4890a7fdc953fa128af4: Time spent bootstrapping tablet: real 0.100s	user 0.005s	sys 0.014s
I20260709 13:03:49.605837 11149 consensus_queue.cc:260] T e96e80ba9ac34d0998395515e5c8c41a P 873357742d5843579556ee8166f23f47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:49.607589 10953 raft_consensus.cc:3060] T 57b6a2418c1342e7924195e9c1dbd357 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:03:49.609125 11152 raft_consensus.cc:359] T d71ca3a1ba6b48478aaf62741bf3cf6a P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:49.609910 11152 raft_consensus.cc:385] T d71ca3a1ba6b48478aaf62741bf3cf6a P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:49.610203 11152 raft_consensus.cc:740] T d71ca3a1ba6b48478aaf62741bf3cf6a P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0172fed22b6b4890a7fdc953fa128af4, State: Initialized, Role: FOLLOWER
I20260709 13:03:49.611186 11152 consensus_queue.cc:260] T d71ca3a1ba6b48478aaf62741bf3cf6a P 0172fed22b6b4890a7fdc953fa128af4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:49.613721 11149 ts_tablet_manager.cc:1434] T e96e80ba9ac34d0998395515e5c8c41a P 873357742d5843579556ee8166f23f47: Time spent starting tablet: real 0.015s	user 0.006s	sys 0.000s
I20260709 13:03:49.614830 11149 tablet_bootstrap.cc:492] T e89e640169e14e2d98f968f76027daa7 P 873357742d5843579556ee8166f23f47: Bootstrap starting.
I20260709 13:03:49.615267 11029 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34064dc7bf3747dbb3876dd824ee0688" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:03:49.615983 11029 raft_consensus.cc:2468] T 34064dc7bf3747dbb3876dd824ee0688 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 0.
I20260709 13:03:49.618625 11152 ts_tablet_manager.cc:1434] T d71ca3a1ba6b48478aaf62741bf3cf6a P 0172fed22b6b4890a7fdc953fa128af4: Time spent starting tablet: real 0.012s	user 0.006s	sys 0.004s
I20260709 13:03:49.619629 11152 tablet_bootstrap.cc:492] T 9a1716c0adf34bee8a4e99341ba5788f P 0172fed22b6b4890a7fdc953fa128af4: Bootstrap starting.
I20260709 13:03:49.623107 11279 raft_consensus.cc:493] T 6a9353d1d230416296627d7cf8b1d425 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:49.623653 11279 raft_consensus.cc:515] T 6a9353d1d230416296627d7cf8b1d425 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:49.624364 11148 tablet_bootstrap.cc:654] T 836251ebc2b74f5da88b8a8b2338e2a5 P 10ca5ee68ad64b3283b9d235352c9d23: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:49.626228 11279 leader_election.cc:290] T 6a9353d1d230416296627d7cf8b1d425 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:03:49.580505 11198 raft_consensus.cc:493] T 7757ecbf402e47bfbc93d0447741e26c P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:49.628492 11198 raft_consensus.cc:515] T 7757ecbf402e47bfbc93d0447741e26c P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:49.632761 11149 tablet_bootstrap.cc:654] T e89e640169e14e2d98f968f76027daa7 P 873357742d5843579556ee8166f23f47: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:49.633023 11102 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a9353d1d230416296627d7cf8b1d425" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:03:49.632928 11101 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7757ecbf402e47bfbc93d0447741e26c" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:03:49.633740 10878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7757ecbf402e47bfbc93d0447741e26c" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:03:49.634042 11101 raft_consensus.cc:2468] T 7757ecbf402e47bfbc93d0447741e26c P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 0.
I20260709 13:03:49.634462 10878 raft_consensus.cc:2468] T 7757ecbf402e47bfbc93d0447741e26c P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 0.
I20260709 13:03:49.635462 10994 leader_election.cc:304] T 7757ecbf402e47bfbc93d0447741e26c P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:03:49.636096 11198 leader_election.cc:290] T 7757ecbf402e47bfbc93d0447741e26c P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:03:49.638825 11029 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a9353d1d230416296627d7cf8b1d425" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:03:49.640265 11198 raft_consensus.cc:2804] T 7757ecbf402e47bfbc93d0447741e26c P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:03:49.641870 10844 leader_election.cc:304] T 6a9353d1d230416296627d7cf8b1d425 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 186b12638a87410487467df721a3cbda; no voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47
I20260709 13:03:49.644779 11151 tablet_bootstrap.cc:654] T 7839ca2d2c13431a85f44d7c458c6351 P 186b12638a87410487467df721a3cbda: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:49.648001 11149 tablet_bootstrap.cc:492] T e89e640169e14e2d98f968f76027daa7 P 873357742d5843579556ee8166f23f47: No bootstrap required, opened a new log
I20260709 13:03:49.648654 11149 ts_tablet_manager.cc:1403] T e89e640169e14e2d98f968f76027daa7 P 873357742d5843579556ee8166f23f47: Time spent bootstrapping tablet: real 0.034s	user 0.010s	sys 0.003s
I20260709 13:03:49.646293 11279 raft_consensus.cc:2749] T 6a9353d1d230416296627d7cf8b1d425 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:03:49.649955 11198 raft_consensus.cc:493] T 7757ecbf402e47bfbc93d0447741e26c P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:49.650379 11198 raft_consensus.cc:3060] T 7757ecbf402e47bfbc93d0447741e26c P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:03:49.651865 11149 raft_consensus.cc:359] T e89e640169e14e2d98f968f76027daa7 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:49.652577 11149 raft_consensus.cc:385] T e89e640169e14e2d98f968f76027daa7 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:49.652843 11149 raft_consensus.cc:740] T e89e640169e14e2d98f968f76027daa7 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 873357742d5843579556ee8166f23f47, State: Initialized, Role: FOLLOWER
I20260709 13:03:49.656574 11240 raft_consensus.cc:515] T 34064dc7bf3747dbb3876dd824ee0688 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:49.658099 11152 tablet_bootstrap.cc:654] T 9a1716c0adf34bee8a4e99341ba5788f P 0172fed22b6b4890a7fdc953fa128af4: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:49.659737 11149 consensus_queue.cc:260] T e89e640169e14e2d98f968f76027daa7 P 873357742d5843579556ee8166f23f47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:49.664528 11149 ts_tablet_manager.cc:1434] T e89e640169e14e2d98f968f76027daa7 P 873357742d5843579556ee8166f23f47: Time spent starting tablet: real 0.016s	user 0.009s	sys 0.001s
I20260709 13:03:49.665657 11149 tablet_bootstrap.cc:492] T a64b7e99d6c54934a7de6f6dffcf5e94 P 873357742d5843579556ee8166f23f47: Bootstrap starting.
I20260709 13:03:49.668988 10955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34064dc7bf3747dbb3876dd824ee0688" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:03:49.669735 10955 raft_consensus.cc:3060] T 34064dc7bf3747dbb3876dd824ee0688 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:03:49.678213 11149 tablet_bootstrap.cc:654] T a64b7e99d6c54934a7de6f6dffcf5e94 P 873357742d5843579556ee8166f23f47: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:49.683312 11152 tablet_bootstrap.cc:492] T 9a1716c0adf34bee8a4e99341ba5788f P 0172fed22b6b4890a7fdc953fa128af4: No bootstrap required, opened a new log
I20260709 13:03:49.683812 11152 ts_tablet_manager.cc:1403] T 9a1716c0adf34bee8a4e99341ba5788f P 0172fed22b6b4890a7fdc953fa128af4: Time spent bootstrapping tablet: real 0.064s	user 0.008s	sys 0.004s
I20260709 13:03:49.683902 11234 raft_consensus.cc:493] T d71ca3a1ba6b48478aaf62741bf3cf6a P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:49.684506 11234 raft_consensus.cc:515] T d71ca3a1ba6b48478aaf62741bf3cf6a P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:49.687937 11234 leader_election.cc:290] T d71ca3a1ba6b48478aaf62741bf3cf6a P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:03:49.688545 10955 raft_consensus.cc:2468] T 34064dc7bf3747dbb3876dd824ee0688 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 1.
I20260709 13:03:49.690235 10843 leader_election.cc:304] T 34064dc7bf3747dbb3876dd824ee0688 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda; no voters: 
I20260709 13:03:49.691296 11279 raft_consensus.cc:2804] T 34064dc7bf3747dbb3876dd824ee0688 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:03:49.693053 11101 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d71ca3a1ba6b48478aaf62741bf3cf6a" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:03:49.692978 11152 raft_consensus.cc:359] T 9a1716c0adf34bee8a4e99341ba5788f P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:49.693825 11148 tablet_bootstrap.cc:492] T 836251ebc2b74f5da88b8a8b2338e2a5 P 10ca5ee68ad64b3283b9d235352c9d23: No bootstrap required, opened a new log
I20260709 13:03:49.693866 11152 raft_consensus.cc:385] T 9a1716c0adf34bee8a4e99341ba5788f P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:49.694366 11152 raft_consensus.cc:740] T 9a1716c0adf34bee8a4e99341ba5788f P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0172fed22b6b4890a7fdc953fa128af4, State: Initialized, Role: FOLLOWER
I20260709 13:03:49.694612 11148 ts_tablet_manager.cc:1403] T 836251ebc2b74f5da88b8a8b2338e2a5 P 10ca5ee68ad64b3283b9d235352c9d23: Time spent bootstrapping tablet: real 0.091s	user 0.012s	sys 0.025s
I20260709 13:03:49.693092 10954 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d71ca3a1ba6b48478aaf62741bf3cf6a" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:03:49.696434 10954 raft_consensus.cc:2468] T d71ca3a1ba6b48478aaf62741bf3cf6a P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 0.
I20260709 13:03:49.697045 11240 leader_election.cc:290] T 34064dc7bf3747dbb3876dd824ee0688 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 1 election: Requested vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:03:49.693926 11101 raft_consensus.cc:2468] T d71ca3a1ba6b48478aaf62741bf3cf6a P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 0.
I20260709 13:03:49.697677 10992 leader_election.cc:304] T d71ca3a1ba6b48478aaf62741bf3cf6a P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:03:49.698820 11152 consensus_queue.cc:260] T 9a1716c0adf34bee8a4e99341ba5788f P 0172fed22b6b4890a7fdc953fa128af4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:49.698861 11029 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34064dc7bf3747dbb3876dd824ee0688" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
I20260709 13:03:49.700273 11029 raft_consensus.cc:3060] T 34064dc7bf3747dbb3876dd824ee0688 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:03:49.701391 11148 raft_consensus.cc:359] T 836251ebc2b74f5da88b8a8b2338e2a5 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:49.703070 11148 raft_consensus.cc:385] T 836251ebc2b74f5da88b8a8b2338e2a5 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:49.703512 11148 raft_consensus.cc:740] T 836251ebc2b74f5da88b8a8b2338e2a5 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10ca5ee68ad64b3283b9d235352c9d23, State: Initialized, Role: FOLLOWER
I20260709 13:03:49.705523 11152 ts_tablet_manager.cc:1434] T 9a1716c0adf34bee8a4e99341ba5788f P 0172fed22b6b4890a7fdc953fa128af4: Time spent starting tablet: real 0.021s	user 0.008s	sys 0.000s
I20260709 13:03:49.706745 11152 tablet_bootstrap.cc:492] T a64b7e99d6c54934a7de6f6dffcf5e94 P 0172fed22b6b4890a7fdc953fa128af4: Bootstrap starting.
I20260709 13:03:49.706176 11148 consensus_queue.cc:260] T 836251ebc2b74f5da88b8a8b2338e2a5 P 10ca5ee68ad64b3283b9d235352c9d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:49.708854 11149 tablet_bootstrap.cc:492] T a64b7e99d6c54934a7de6f6dffcf5e94 P 873357742d5843579556ee8166f23f47: No bootstrap required, opened a new log
I20260709 13:03:49.712610 11149 ts_tablet_manager.cc:1403] T a64b7e99d6c54934a7de6f6dffcf5e94 P 873357742d5843579556ee8166f23f47: Time spent bootstrapping tablet: real 0.047s	user 0.011s	sys 0.002s
I20260709 13:03:49.716279 11152 tablet_bootstrap.cc:654] T a64b7e99d6c54934a7de6f6dffcf5e94 P 0172fed22b6b4890a7fdc953fa128af4: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:49.717394 10878 raft_consensus.cc:3060] T 6a9353d1d230416296627d7cf8b1d425 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:03:49.702769 11234 raft_consensus.cc:2804] T d71ca3a1ba6b48478aaf62741bf3cf6a P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:03:49.711221 11148 ts_tablet_manager.cc:1434] T 836251ebc2b74f5da88b8a8b2338e2a5 P 10ca5ee68ad64b3283b9d235352c9d23: Time spent starting tablet: real 0.016s	user 0.007s	sys 0.000s
I20260709 13:03:49.717836 11198 raft_consensus.cc:515] T 7757ecbf402e47bfbc93d0447741e26c P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:49.723014 11198 leader_election.cc:290] T 7757ecbf402e47bfbc93d0447741e26c P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 1 election: Requested vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:03:49.719179 11279 raft_consensus.cc:697] T 34064dc7bf3747dbb3876dd824ee0688 P 186b12638a87410487467df721a3cbda [term 1 LEADER]: Becoming Leader. State: Replica: 186b12638a87410487467df721a3cbda, State: Running, Role: LEADER
I20260709 13:03:49.724360 11279 consensus_queue.cc:237] T 34064dc7bf3747dbb3876dd824ee0688 P 186b12638a87410487467df721a3cbda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:49.720659 11188 raft_consensus.cc:493] T 8a69f32c53f24a50b7852d143665d7b4 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:49.725453 11188 raft_consensus.cc:515] T 8a69f32c53f24a50b7852d143665d7b4 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:49.724563 10877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7757ecbf402e47bfbc93d0447741e26c" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:03:49.726565 11234 raft_consensus.cc:493] T d71ca3a1ba6b48478aaf62741bf3cf6a P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:49.726752 10877 raft_consensus.cc:3060] T 7757ecbf402e47bfbc93d0447741e26c P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:03:49.726995 11234 raft_consensus.cc:3060] T d71ca3a1ba6b48478aaf62741bf3cf6a P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:03:49.727763 11148 tablet_bootstrap.cc:492] T e89e640169e14e2d98f968f76027daa7 P 10ca5ee68ad64b3283b9d235352c9d23: Bootstrap starting.
I20260709 13:03:49.730968 10879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a69f32c53f24a50b7852d143665d7b4" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:03:49.729386 11188 leader_election.cc:290] T 8a69f32c53f24a50b7852d143665d7b4 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:03:49.731434 11101 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7757ecbf402e47bfbc93d0447741e26c" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:03:49.731773 10879 raft_consensus.cc:2468] T 8a69f32c53f24a50b7852d143665d7b4 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 0.
I20260709 13:03:49.732110 11101 raft_consensus.cc:3060] T 7757ecbf402e47bfbc93d0447741e26c P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:03:49.733315 10921 leader_election.cc:304] T 8a69f32c53f24a50b7852d143665d7b4 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda; no voters: 
I20260709 13:03:49.738511 11188 raft_consensus.cc:2804] T 8a69f32c53f24a50b7852d143665d7b4 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:03:49.738905 11188 raft_consensus.cc:493] T 8a69f32c53f24a50b7852d143665d7b4 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:49.739272 11188 raft_consensus.cc:3060] T 8a69f32c53f24a50b7852d143665d7b4 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:03:49.739631 10877 raft_consensus.cc:2468] T 7757ecbf402e47bfbc93d0447741e26c P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 1.
I20260709 13:03:49.730000 11149 raft_consensus.cc:359] T a64b7e99d6c54934a7de6f6dffcf5e94 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:49.740988 11149 raft_consensus.cc:385] T a64b7e99d6c54934a7de6f6dffcf5e94 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:49.741310 11149 raft_consensus.cc:740] T a64b7e99d6c54934a7de6f6dffcf5e94 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 873357742d5843579556ee8166f23f47, State: Initialized, Role: FOLLOWER
I20260709 13:03:49.741451 10995 leader_election.cc:304] T 7757ecbf402e47bfbc93d0447741e26c P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:03:49.742540 11198 raft_consensus.cc:2804] T 7757ecbf402e47bfbc93d0447741e26c P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:03:49.743105 11198 raft_consensus.cc:697] T 7757ecbf402e47bfbc93d0447741e26c P 873357742d5843579556ee8166f23f47 [term 1 LEADER]: Becoming Leader. State: Replica: 873357742d5843579556ee8166f23f47, State: Running, Role: LEADER
I20260709 13:03:49.743613 11149 consensus_queue.cc:260] T a64b7e99d6c54934a7de6f6dffcf5e94 P 873357742d5843579556ee8166f23f47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:49.744144 11198 consensus_queue.cc:237] T 7757ecbf402e47bfbc93d0447741e26c P 873357742d5843579556ee8166f23f47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:49.730898 11102 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a69f32c53f24a50b7852d143665d7b4" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:03:49.747562 11102 raft_consensus.cc:2468] T 8a69f32c53f24a50b7852d143665d7b4 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 0.
I20260709 13:03:49.748960 11101 raft_consensus.cc:2468] T 7757ecbf402e47bfbc93d0447741e26c P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 1.
I20260709 13:03:49.752278 11149 ts_tablet_manager.cc:1434] T a64b7e99d6c54934a7de6f6dffcf5e94 P 873357742d5843579556ee8166f23f47: Time spent starting tablet: real 0.033s	user 0.007s	sys 0.001s
I20260709 13:03:49.753090 11235 consensus_queue.cc:1048] T e5ba72ccd854430baadbc636dfd17136 P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:03:49.753520 11149 tablet_bootstrap.cc:492] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 873357742d5843579556ee8166f23f47: Bootstrap starting.
I20260709 13:03:49.753718 11295 consensus_queue.cc:1048] T 7839ca2d2c13431a85f44d7c458c6351 P 873357742d5843579556ee8166f23f47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:03:49.757041 11148 tablet_bootstrap.cc:654] T e89e640169e14e2d98f968f76027daa7 P 10ca5ee68ad64b3283b9d235352c9d23: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:49.763032 11152 tablet_bootstrap.cc:492] T a64b7e99d6c54934a7de6f6dffcf5e94 P 0172fed22b6b4890a7fdc953fa128af4: No bootstrap required, opened a new log
I20260709 13:03:49.697729 11151 tablet_bootstrap.cc:492] T 7839ca2d2c13431a85f44d7c458c6351 P 186b12638a87410487467df721a3cbda: No bootstrap required, opened a new log
I20260709 13:03:49.763604 11152 ts_tablet_manager.cc:1403] T a64b7e99d6c54934a7de6f6dffcf5e94 P 0172fed22b6b4890a7fdc953fa128af4: Time spent bootstrapping tablet: real 0.057s	user 0.010s	sys 0.009s
I20260709 13:03:49.763867 11151 ts_tablet_manager.cc:1403] T 7839ca2d2c13431a85f44d7c458c6351 P 186b12638a87410487467df721a3cbda: Time spent bootstrapping tablet: real 0.209s	user 0.013s	sys 0.002s
I20260709 13:03:49.719406 11029 raft_consensus.cc:2468] T 34064dc7bf3747dbb3876dd824ee0688 P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 1.
I20260709 13:03:49.772467 10778 catalog_manager.cc:5697] T 7757ecbf402e47bfbc93d0447741e26c P 873357742d5843579556ee8166f23f47 reported cstate change: term changed from 0 to 1, leader changed from <none> to 873357742d5843579556ee8166f23f47 (127.0.107.195). New cstate: current_term: 1 leader_uuid: "873357742d5843579556ee8166f23f47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } health_report { overall_health: UNKNOWN } } }
I20260709 13:03:49.775600 11151 raft_consensus.cc:359] T 7839ca2d2c13431a85f44d7c458c6351 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:49.776014 11234 raft_consensus.cc:515] T d71ca3a1ba6b48478aaf62741bf3cf6a P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:49.779573 11101 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d71ca3a1ba6b48478aaf62741bf3cf6a" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:03:49.776417 11151 raft_consensus.cc:385] T 7839ca2d2c13431a85f44d7c458c6351 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:49.780239 11101 raft_consensus.cc:3060] T d71ca3a1ba6b48478aaf62741bf3cf6a P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:03:49.780305 11151 raft_consensus.cc:740] T 7839ca2d2c13431a85f44d7c458c6351 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 186b12638a87410487467df721a3cbda, State: Initialized, Role: FOLLOWER
I20260709 13:03:49.781535 11151 consensus_queue.cc:260] T 7839ca2d2c13431a85f44d7c458c6351 P 186b12638a87410487467df721a3cbda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:49.784904 11152 raft_consensus.cc:359] T a64b7e99d6c54934a7de6f6dffcf5e94 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:49.785718 11152 raft_consensus.cc:385] T a64b7e99d6c54934a7de6f6dffcf5e94 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:49.785996 11152 raft_consensus.cc:740] T a64b7e99d6c54934a7de6f6dffcf5e94 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0172fed22b6b4890a7fdc953fa128af4, State: Initialized, Role: FOLLOWER
I20260709 13:03:49.781646 10953 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d71ca3a1ba6b48478aaf62741bf3cf6a" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:03:49.787494 11151 ts_tablet_manager.cc:1434] T 7839ca2d2c13431a85f44d7c458c6351 P 186b12638a87410487467df721a3cbda: Time spent starting tablet: real 0.023s	user 0.006s	sys 0.001s
I20260709 13:03:49.787485 10953 raft_consensus.cc:3060] T d71ca3a1ba6b48478aaf62741bf3cf6a P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:03:49.789768 11101 raft_consensus.cc:2468] T d71ca3a1ba6b48478aaf62741bf3cf6a P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 1.
I20260709 13:03:49.798806 10994 leader_election.cc:304] T d71ca3a1ba6b48478aaf62741bf3cf6a P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:03:49.799139 11188 raft_consensus.cc:515] T 8a69f32c53f24a50b7852d143665d7b4 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:49.799722 11295 raft_consensus.cc:493] T e89e640169e14e2d98f968f76027daa7 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:49.800271 11295 raft_consensus.cc:515] T e89e640169e14e2d98f968f76027daa7 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:49.801667 11257 raft_consensus.cc:2804] T d71ca3a1ba6b48478aaf62741bf3cf6a P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:03:49.802700 11295 leader_election.cc:290] T e89e640169e14e2d98f968f76027daa7 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:03:49.802493 11235 consensus_queue.cc:1048] T e5ba72ccd854430baadbc636dfd17136 P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:03:49.805214 10955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e89e640169e14e2d98f968f76027daa7" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:03:49.788940 11151 tablet_bootstrap.cc:492] T 836251ebc2b74f5da88b8a8b2338e2a5 P 186b12638a87410487467df721a3cbda: Bootstrap starting.
I20260709 13:03:49.806985 10877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e89e640169e14e2d98f968f76027daa7" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
W20260709 13:03:49.808996 10992 leader_election.cc:343] T e89e640169e14e2d98f968f76027daa7 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903): Illegal state: must be running to vote when last-logged opid is not known
W20260709 13:03:49.811007 10995 leader_election.cc:343] T e89e640169e14e2d98f968f76027daa7 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 186b12638a87410487467df721a3cbda (127.0.107.193:38967): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:03:49.814051 10995 leader_election.cc:304] T e89e640169e14e2d98f968f76027daa7 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 873357742d5843579556ee8166f23f47; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda
I20260709 13:03:49.811034 11152 consensus_queue.cc:260] T a64b7e99d6c54934a7de6f6dffcf5e94 P 0172fed22b6b4890a7fdc953fa128af4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:49.822010 11101 raft_consensus.cc:3060] T 525bfdbd6d934afd823f7d3ee3f9d052 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:03:49.828619 11235 raft_consensus.cc:493] T 836251ebc2b74f5da88b8a8b2338e2a5 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:49.837868 11235 raft_consensus.cc:515] T 836251ebc2b74f5da88b8a8b2338e2a5 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:49.778370 11234 leader_election.cc:290] T d71ca3a1ba6b48478aaf62741bf3cf6a P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 1 election: Requested vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:03:49.840505 11235 leader_election.cc:290] T 836251ebc2b74f5da88b8a8b2338e2a5 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:03:49.843629 11102 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a69f32c53f24a50b7852d143665d7b4" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:03:49.844295 11102 raft_consensus.cc:3060] T 8a69f32c53f24a50b7852d143665d7b4 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:03:49.847146 10955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "836251ebc2b74f5da88b8a8b2338e2a5" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:03:49.847880 10955 raft_consensus.cc:2468] T 836251ebc2b74f5da88b8a8b2338e2a5 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 0.
I20260709 13:03:49.849064 11066 leader_election.cc:304] T 836251ebc2b74f5da88b8a8b2338e2a5 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 10ca5ee68ad64b3283b9d235352c9d23; no voters: 
I20260709 13:03:49.849913 11235 raft_consensus.cc:2804] T 836251ebc2b74f5da88b8a8b2338e2a5 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:03:49.824780 11152 ts_tablet_manager.cc:1434] T a64b7e99d6c54934a7de6f6dffcf5e94 P 0172fed22b6b4890a7fdc953fa128af4: Time spent starting tablet: real 0.061s	user 0.008s	sys 0.000s
I20260709 13:03:49.850694 11235 raft_consensus.cc:493] T 836251ebc2b74f5da88b8a8b2338e2a5 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:49.851068 11235 raft_consensus.cc:3060] T 836251ebc2b74f5da88b8a8b2338e2a5 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Advancing to term 1
W20260709 13:03:49.851452 10844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:45807, user_credentials={real_user=slave}} blocked reactor thread for 73677.7us
I20260709 13:03:49.851840 11152 tablet_bootstrap.cc:492] T 5d439326f23747eaab3c1e7da2d6b785 P 0172fed22b6b4890a7fdc953fa128af4: Bootstrap starting.
I20260709 13:03:49.820984 11151 tablet_bootstrap.cc:654] T 836251ebc2b74f5da88b8a8b2338e2a5 P 186b12638a87410487467df721a3cbda: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:49.865847 10776 catalog_manager.cc:5697] T 34064dc7bf3747dbb3876dd824ee0688 P 186b12638a87410487467df721a3cbda reported cstate change: term changed from 0 to 1, leader changed from <none> to 186b12638a87410487467df721a3cbda (127.0.107.193). New cstate: current_term: 1 leader_uuid: "186b12638a87410487467df721a3cbda" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } health_report { overall_health: UNKNOWN } } }
I20260709 13:03:49.833166 11302 raft_consensus.cc:2749] T e89e640169e14e2d98f968f76027daa7 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:03:49.833530 11149 tablet_bootstrap.cc:654] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 873357742d5843579556ee8166f23f47: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:49.833947 11257 raft_consensus.cc:697] T d71ca3a1ba6b48478aaf62741bf3cf6a P 873357742d5843579556ee8166f23f47 [term 1 LEADER]: Becoming Leader. State: Replica: 873357742d5843579556ee8166f23f47, State: Running, Role: LEADER
I20260709 13:03:49.870515 11257 consensus_queue.cc:237] T d71ca3a1ba6b48478aaf62741bf3cf6a P 873357742d5843579556ee8166f23f47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:49.874583 11162 raft_consensus.cc:493] T 9a1716c0adf34bee8a4e99341ba5788f P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:49.875276 11162 raft_consensus.cc:515] T 9a1716c0adf34bee8a4e99341ba5788f P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
W20260709 13:03:49.886760 10995 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:38967, user_credentials={real_user=slave}} blocked reactor thread for 75824.5us
I20260709 13:03:49.901976 11235 raft_consensus.cc:515] T 836251ebc2b74f5da88b8a8b2338e2a5 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:49.907047 11237 raft_consensus.cc:493] T a64b7e99d6c54934a7de6f6dffcf5e94 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:49.907577 11237 raft_consensus.cc:515] T a64b7e99d6c54934a7de6f6dffcf5e94 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:49.888199 10878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "836251ebc2b74f5da88b8a8b2338e2a5" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:03:49.894003 11162 leader_election.cc:290] T 9a1716c0adf34bee8a4e99341ba5788f P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:03:49.892601 11102 raft_consensus.cc:2468] T 8a69f32c53f24a50b7852d143665d7b4 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 1.
I20260709 13:03:49.896016 11149 tablet_bootstrap.cc:492] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 873357742d5843579556ee8166f23f47: No bootstrap required, opened a new log
I20260709 13:03:49.916303 11149 ts_tablet_manager.cc:1403] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 873357742d5843579556ee8166f23f47: Time spent bootstrapping tablet: real 0.163s	user 0.009s	sys 0.007s
I20260709 13:03:49.919406 10878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a69f32c53f24a50b7852d143665d7b4" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:03:49.920022 11149 raft_consensus.cc:359] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:49.945341 11149 raft_consensus.cc:385] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:49.896322 11188 leader_election.cc:290] T 8a69f32c53f24a50b7852d143665d7b4 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 1 election: Requested vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:03:49.922845 11237 leader_election.cc:290] T a64b7e99d6c54934a7de6f6dffcf5e94 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:03:49.929426 10920 leader_election.cc:304] T 8a69f32c53f24a50b7852d143665d7b4 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 10ca5ee68ad64b3283b9d235352c9d23; no voters: 
I20260709 13:03:49.947710 10955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "836251ebc2b74f5da88b8a8b2338e2a5" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:03:49.948354 10955 raft_consensus.cc:3060] T 836251ebc2b74f5da88b8a8b2338e2a5 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:03:49.948760 10954 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a64b7e99d6c54934a7de6f6dffcf5e94" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:03:49.950210 11028 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a64b7e99d6c54934a7de6f6dffcf5e94" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:03:49.950932 11028 raft_consensus.cc:2468] T a64b7e99d6c54934a7de6f6dffcf5e94 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 0.
I20260709 13:03:49.952021 11239 raft_consensus.cc:2804] T 8a69f32c53f24a50b7852d143665d7b4 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:03:49.952607 11239 raft_consensus.cc:697] T 8a69f32c53f24a50b7852d143665d7b4 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 LEADER]: Becoming Leader. State: Replica: 10ca5ee68ad64b3283b9d235352c9d23, State: Running, Role: LEADER
I20260709 13:03:49.953565 11239 consensus_queue.cc:237] T 8a69f32c53f24a50b7852d143665d7b4 P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
W20260709 13:03:49.954921 10826 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 13:03:49.955145 10953 raft_consensus.cc:2468] T d71ca3a1ba6b48478aaf62741bf3cf6a P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 1.
W20260709 13:03:49.955750 11066 leader_election.cc:343] T a64b7e99d6c54934a7de6f6dffcf5e94 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:03:49.930240 11235 leader_election.cc:290] T 836251ebc2b74f5da88b8a8b2338e2a5 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 1 election: Requested vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:03:49.940619 11029 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a1716c0adf34bee8a4e99341ba5788f" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:03:49.942101 10877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a1716c0adf34bee8a4e99341ba5788f" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:03:49.942414 10879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "836251ebc2b74f5da88b8a8b2338e2a5" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
W20260709 13:03:49.942552 11069 leader_election.cc:343] T 836251ebc2b74f5da88b8a8b2338e2a5 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 186b12638a87410487467df721a3cbda (127.0.107.193:38967): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:03:49.959242 11029 raft_consensus.cc:2468] T 9a1716c0adf34bee8a4e99341ba5788f P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 0.
I20260709 13:03:49.962203 11257 raft_consensus.cc:493] T e96e80ba9ac34d0998395515e5c8c41a P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:49.962939 11257 raft_consensus.cc:515] T e96e80ba9ac34d0998395515e5c8c41a P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:49.966292 11067 leader_election.cc:304] T a64b7e99d6c54934a7de6f6dffcf5e94 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47; no voters: 10ca5ee68ad64b3283b9d235352c9d23
I20260709 13:03:49.968097 11235 raft_consensus.cc:2804] T a64b7e99d6c54934a7de6f6dffcf5e94 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:03:49.944844 11152 tablet_bootstrap.cc:654] T 5d439326f23747eaab3c1e7da2d6b785 P 0172fed22b6b4890a7fdc953fa128af4: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:49.945413 10878 raft_consensus.cc:3060] T 8a69f32c53f24a50b7852d143665d7b4 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:03:49.945906 11149 raft_consensus.cc:740] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 873357742d5843579556ee8166f23f47, State: Initialized, Role: FOLLOWER
I20260709 13:03:49.973799 11149 consensus_queue.cc:260] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 873357742d5843579556ee8166f23f47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:49.920821 11151 tablet_bootstrap.cc:492] T 836251ebc2b74f5da88b8a8b2338e2a5 P 186b12638a87410487467df721a3cbda: No bootstrap required, opened a new log
I20260709 13:03:49.976837 11151 ts_tablet_manager.cc:1403] T 836251ebc2b74f5da88b8a8b2338e2a5 P 186b12638a87410487467df721a3cbda: Time spent bootstrapping tablet: real 0.188s	user 0.012s	sys 0.003s
I20260709 13:03:49.980600 11188 raft_consensus.cc:493] T e96e80ba9ac34d0998395515e5c8c41a P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:49.981282 11188 raft_consensus.cc:515] T e96e80ba9ac34d0998395515e5c8c41a P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:49.986783 11148 tablet_bootstrap.cc:492] T e89e640169e14e2d98f968f76027daa7 P 10ca5ee68ad64b3283b9d235352c9d23: No bootstrap required, opened a new log
I20260709 13:03:49.987349 11148 ts_tablet_manager.cc:1403] T e89e640169e14e2d98f968f76027daa7 P 10ca5ee68ad64b3283b9d235352c9d23: Time spent bootstrapping tablet: real 0.260s	user 0.069s	sys 0.075s
I20260709 13:03:49.987496 10879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e96e80ba9ac34d0998395515e5c8c41a" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:03:49.988181 11149 ts_tablet_manager.cc:1434] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 873357742d5843579556ee8166f23f47: Time spent starting tablet: real 0.071s	user 0.009s	sys 0.000s
I20260709 13:03:49.988230 11067 leader_election.cc:304] T 9a1716c0adf34bee8a4e99341ba5788f P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:03:49.989398 11162 raft_consensus.cc:2804] T 9a1716c0adf34bee8a4e99341ba5788f P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:03:49.989548 11028 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e96e80ba9ac34d0998395515e5c8c41a" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:03:49.989866 11162 raft_consensus.cc:493] T 9a1716c0adf34bee8a4e99341ba5788f P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:49.990271 11162 raft_consensus.cc:3060] T 9a1716c0adf34bee8a4e99341ba5788f P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:03:49.990353 11028 raft_consensus.cc:2468] T e96e80ba9ac34d0998395515e5c8c41a P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 0.
I20260709 13:03:49.991688 10919 leader_election.cc:304] T e96e80ba9ac34d0998395515e5c8c41a P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:03:49.992903 11168 raft_consensus.cc:2804] T e96e80ba9ac34d0998395515e5c8c41a P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:03:49.993022 10955 raft_consensus.cc:2468] T 836251ebc2b74f5da88b8a8b2338e2a5 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 1.
I20260709 13:03:49.993523 11168 raft_consensus.cc:493] T e96e80ba9ac34d0998395515e5c8c41a P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:49.993952 11168 raft_consensus.cc:3060] T e96e80ba9ac34d0998395515e5c8c41a P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:03:49.990310 11148 raft_consensus.cc:359] T e89e640169e14e2d98f968f76027daa7 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:49.994495 11066 leader_election.cc:304] T 836251ebc2b74f5da88b8a8b2338e2a5 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 10ca5ee68ad64b3283b9d235352c9d23; no voters: 
I20260709 13:03:49.994782 11148 raft_consensus.cc:385] T e89e640169e14e2d98f968f76027daa7 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:49.995096 11148 raft_consensus.cc:740] T e89e640169e14e2d98f968f76027daa7 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10ca5ee68ad64b3283b9d235352c9d23, State: Initialized, Role: FOLLOWER
I20260709 13:03:49.995894 11148 consensus_queue.cc:260] T e89e640169e14e2d98f968f76027daa7 P 10ca5ee68ad64b3283b9d235352c9d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:49.996582 11196 raft_consensus.cc:2804] T 836251ebc2b74f5da88b8a8b2338e2a5 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:03:49.997407 11196 raft_consensus.cc:697] T 836251ebc2b74f5da88b8a8b2338e2a5 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 LEADER]: Becoming Leader. State: Replica: 0172fed22b6b4890a7fdc953fa128af4, State: Running, Role: LEADER
W20260709 13:03:49.990011 11069 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:38967, user_credentials={real_user=slave}} blocked reactor thread for 47555.3us
I20260709 13:03:49.998659 11148 ts_tablet_manager.cc:1434] T e89e640169e14e2d98f968f76027daa7 P 10ca5ee68ad64b3283b9d235352c9d23: Time spent starting tablet: real 0.011s	user 0.007s	sys 0.001s
I20260709 13:03:49.998298 11196 consensus_queue.cc:237] T 836251ebc2b74f5da88b8a8b2338e2a5 P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
W20260709 13:03:49.999395 10921 leader_election.cc:343] T e96e80ba9ac34d0998395515e5c8c41a P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 186b12638a87410487467df721a3cbda (127.0.107.193:38967): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:03:49.999877 11148 tablet_bootstrap.cc:492] T d270b9e5481245d9b5340a454a7cd10b P 10ca5ee68ad64b3283b9d235352c9d23: Bootstrap starting.
W20260709 13:03:50.002046 11069 leader_election.cc:343] T 9a1716c0adf34bee8a4e99341ba5788f P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 186b12638a87410487467df721a3cbda (127.0.107.193:38967): Illegal state: must be running to vote when last-logged opid is not known
W20260709 13:03:50.003153 11069 leader_election.cc:343] T 836251ebc2b74f5da88b8a8b2338e2a5 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 186b12638a87410487467df721a3cbda (127.0.107.193:38967): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:03:49.968604 11235 raft_consensus.cc:493] T a64b7e99d6c54934a7de6f6dffcf5e94 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:50.004340 11235 raft_consensus.cc:3060] T a64b7e99d6c54934a7de6f6dffcf5e94 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:03:50.005146 10955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e96e80ba9ac34d0998395515e5c8c41a" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:03:49.986093 10776 catalog_manager.cc:5697] T d71ca3a1ba6b48478aaf62741bf3cf6a P 873357742d5843579556ee8166f23f47 reported cstate change: term changed from 0 to 1, leader changed from <none> to 873357742d5843579556ee8166f23f47 (127.0.107.195). New cstate: current_term: 1 leader_uuid: "873357742d5843579556ee8166f23f47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } health_report { overall_health: UNKNOWN } } }
I20260709 13:03:49.987594 11188 leader_election.cc:290] T e96e80ba9ac34d0998395515e5c8c41a P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:03:49.986980 11151 raft_consensus.cc:359] T 836251ebc2b74f5da88b8a8b2338e2a5 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:50.006652 11151 raft_consensus.cc:385] T 836251ebc2b74f5da88b8a8b2338e2a5 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:50.006960 11151 raft_consensus.cc:740] T 836251ebc2b74f5da88b8a8b2338e2a5 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 186b12638a87410487467df721a3cbda, State: Initialized, Role: FOLLOWER
I20260709 13:03:49.989480 11149 tablet_bootstrap.cc:492] T 2567c2d84e4648d7ba5dc23a3df32817 P 873357742d5843579556ee8166f23f47: Bootstrap starting.
I20260709 13:03:50.007874 11148 tablet_bootstrap.cc:654] T d270b9e5481245d9b5340a454a7cd10b P 10ca5ee68ad64b3283b9d235352c9d23: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:50.008224 11257 leader_election.cc:290] T e96e80ba9ac34d0998395515e5c8c41a P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:03:50.012249 11151 consensus_queue.cc:260] T 836251ebc2b74f5da88b8a8b2338e2a5 P 186b12638a87410487467df721a3cbda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:50.020886 11168 raft_consensus.cc:515] T e96e80ba9ac34d0998395515e5c8c41a P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:50.022229 10955 raft_consensus.cc:2393] T e96e80ba9ac34d0998395515e5c8c41a P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 873357742d5843579556ee8166f23f47 in current term 1: Already voted for candidate 10ca5ee68ad64b3283b9d235352c9d23 in this term.
I20260709 13:03:50.023488 10879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e96e80ba9ac34d0998395515e5c8c41a" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:03:50.027942 10879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e96e80ba9ac34d0998395515e5c8c41a" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
W20260709 13:03:50.029230 10921 leader_election.cc:343] T e96e80ba9ac34d0998395515e5c8c41a P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 186b12638a87410487467df721a3cbda (127.0.107.193:38967): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:03:50.030784 11028 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e96e80ba9ac34d0998395515e5c8c41a" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
I20260709 13:03:50.031339 11151 ts_tablet_manager.cc:1434] T 836251ebc2b74f5da88b8a8b2338e2a5 P 186b12638a87410487467df721a3cbda: Time spent starting tablet: real 0.054s	user 0.011s	sys 0.000s
I20260709 13:03:50.031474 11028 raft_consensus.cc:3060] T e96e80ba9ac34d0998395515e5c8c41a P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:03:50.032652 11151 tablet_bootstrap.cc:492] T e96e80ba9ac34d0998395515e5c8c41a P 186b12638a87410487467df721a3cbda: Bootstrap starting.
I20260709 13:03:50.039695 11168 leader_election.cc:290] T e96e80ba9ac34d0998395515e5c8c41a P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 1 election: Requested vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:03:50.042790 11152 tablet_bootstrap.cc:492] T 5d439326f23747eaab3c1e7da2d6b785 P 0172fed22b6b4890a7fdc953fa128af4: No bootstrap required, opened a new log
I20260709 13:03:50.043773 11152 ts_tablet_manager.cc:1403] T 5d439326f23747eaab3c1e7da2d6b785 P 0172fed22b6b4890a7fdc953fa128af4: Time spent bootstrapping tablet: real 0.192s	user 0.011s	sys 0.002s
I20260709 13:03:50.045830 11149 tablet_bootstrap.cc:654] T 2567c2d84e4648d7ba5dc23a3df32817 P 873357742d5843579556ee8166f23f47: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:50.047118 11028 raft_consensus.cc:2468] T e96e80ba9ac34d0998395515e5c8c41a P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 1.
W20260709 13:03:50.047621 10995 leader_election.cc:343] T e96e80ba9ac34d0998395515e5c8c41a P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 186b12638a87410487467df721a3cbda (127.0.107.193:38967): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:03:50.049036 11279 raft_consensus.cc:493] T 7839ca2d2c13431a85f44d7c458c6351 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:50.049124 10995 leader_election.cc:304] T e96e80ba9ac34d0998395515e5c8c41a P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 873357742d5843579556ee8166f23f47; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda
I20260709 13:03:50.049695 11279 raft_consensus.cc:515] T 7839ca2d2c13431a85f44d7c458c6351 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:50.052042 11279 leader_election.cc:290] T 7839ca2d2c13431a85f44d7c458c6351 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:03:50.053704 11305 raft_consensus.cc:2749] T e96e80ba9ac34d0998395515e5c8c41a P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:03:50.053653 10955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7839ca2d2c13431a85f44d7c458c6351" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:03:50.056121 11162 raft_consensus.cc:515] T 9a1716c0adf34bee8a4e99341ba5788f P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:50.062248 11152 raft_consensus.cc:359] T 5d439326f23747eaab3c1e7da2d6b785 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:50.062937 11152 raft_consensus.cc:385] T 5d439326f23747eaab3c1e7da2d6b785 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:50.063220 11152 raft_consensus.cc:740] T 5d439326f23747eaab3c1e7da2d6b785 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0172fed22b6b4890a7fdc953fa128af4, State: Initialized, Role: FOLLOWER
I20260709 13:03:50.074941 11028 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a1716c0adf34bee8a4e99341ba5788f" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
I20260709 13:03:50.064059 11152 consensus_queue.cc:260] T 5d439326f23747eaab3c1e7da2d6b785 P 0172fed22b6b4890a7fdc953fa128af4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:50.066987 10919 leader_election.cc:304] T e96e80ba9ac34d0998395515e5c8c41a P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47; no voters: 186b12638a87410487467df721a3cbda
I20260709 13:03:50.075817 11028 raft_consensus.cc:3060] T 9a1716c0adf34bee8a4e99341ba5788f P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:03:50.076469 11168 raft_consensus.cc:2804] T e96e80ba9ac34d0998395515e5c8c41a P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:03:50.077066 11152 ts_tablet_manager.cc:1434] T 5d439326f23747eaab3c1e7da2d6b785 P 0172fed22b6b4890a7fdc953fa128af4: Time spent starting tablet: real 0.033s	user 0.010s	sys 0.007s
I20260709 13:03:50.076993 11168 raft_consensus.cc:697] T e96e80ba9ac34d0998395515e5c8c41a P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 LEADER]: Becoming Leader. State: Replica: 10ca5ee68ad64b3283b9d235352c9d23, State: Running, Role: LEADER
I20260709 13:03:50.078523 11152 tablet_bootstrap.cc:492] T 868e92339c2f4508af0afbd07c814d64 P 0172fed22b6b4890a7fdc953fa128af4: Bootstrap starting.
I20260709 13:03:50.078459 11168 consensus_queue.cc:237] T e96e80ba9ac34d0998395515e5c8c41a P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:50.067831 10879 raft_consensus.cc:3060] T 7839ca2d2c13431a85f44d7c458c6351 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:03:50.081684 10878 raft_consensus.cc:2468] T 8a69f32c53f24a50b7852d143665d7b4 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 1.
I20260709 13:03:50.063551 11162 leader_election.cc:290] T 9a1716c0adf34bee8a4e99341ba5788f P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 1 election: Requested vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:03:50.063316 11029 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7839ca2d2c13431a85f44d7c458c6351" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:03:50.086006 11152 tablet_bootstrap.cc:654] T 868e92339c2f4508af0afbd07c814d64 P 0172fed22b6b4890a7fdc953fa128af4: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:50.086844 11235 raft_consensus.cc:515] T a64b7e99d6c54934a7de6f6dffcf5e94 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:50.090483 10955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a64b7e99d6c54934a7de6f6dffcf5e94" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
W20260709 13:03:50.092613 11066 leader_election.cc:343] T a64b7e99d6c54934a7de6f6dffcf5e94 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:03:50.091355 10878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a1716c0adf34bee8a4e99341ba5788f" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:03:50.094255 11235 leader_election.cc:290] T a64b7e99d6c54934a7de6f6dffcf5e94 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 1 election: Requested vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:03:50.098609 10844 leader_election.cc:304] T 7839ca2d2c13431a85f44d7c458c6351 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 186b12638a87410487467df721a3cbda; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47
I20260709 13:03:50.123508 11029 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a64b7e99d6c54934a7de6f6dffcf5e94" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
I20260709 13:03:50.124500 11029 raft_consensus.cc:3060] T a64b7e99d6c54934a7de6f6dffcf5e94 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:03:50.116204 10777 catalog_manager.cc:5697] T 836251ebc2b74f5da88b8a8b2338e2a5 P 0172fed22b6b4890a7fdc953fa128af4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196). New cstate: current_term: 1 leader_uuid: "0172fed22b6b4890a7fdc953fa128af4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } health_report { overall_health: UNKNOWN } } }
I20260709 13:03:50.133388 11149 tablet_bootstrap.cc:492] T 2567c2d84e4648d7ba5dc23a3df32817 P 873357742d5843579556ee8166f23f47: No bootstrap required, opened a new log
I20260709 13:03:50.134011 11149 ts_tablet_manager.cc:1403] T 2567c2d84e4648d7ba5dc23a3df32817 P 873357742d5843579556ee8166f23f47: Time spent bootstrapping tablet: real 0.145s	user 0.014s	sys 0.011s
W20260709 13:03:50.140573 11069 leader_election.cc:343] T 9a1716c0adf34bee8a4e99341ba5788f P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 186b12638a87410487467df721a3cbda (127.0.107.193:38967): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:03:50.141762 11148 tablet_bootstrap.cc:492] T d270b9e5481245d9b5340a454a7cd10b P 10ca5ee68ad64b3283b9d235352c9d23: No bootstrap required, opened a new log
I20260709 13:03:50.142355 11148 ts_tablet_manager.cc:1403] T d270b9e5481245d9b5340a454a7cd10b P 10ca5ee68ad64b3283b9d235352c9d23: Time spent bootstrapping tablet: real 0.143s	user 0.034s	sys 0.029s
I20260709 13:03:50.145790 11148 raft_consensus.cc:359] T d270b9e5481245d9b5340a454a7cd10b P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:50.154971 11148 raft_consensus.cc:385] T d270b9e5481245d9b5340a454a7cd10b P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:50.148518 11151 tablet_bootstrap.cc:654] T e96e80ba9ac34d0998395515e5c8c41a P 186b12638a87410487467df721a3cbda: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:50.154093 11292 consensus_queue.cc:1048] T 34064dc7bf3747dbb3876dd824ee0688 P 186b12638a87410487467df721a3cbda [LEADER]: Connected to new peer: Peer: permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:03:50.155331 11148 raft_consensus.cc:740] T d270b9e5481245d9b5340a454a7cd10b P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10ca5ee68ad64b3283b9d235352c9d23, State: Initialized, Role: FOLLOWER
I20260709 13:03:50.157300 10776 catalog_manager.cc:5697] T 8a69f32c53f24a50b7852d143665d7b4 P 10ca5ee68ad64b3283b9d235352c9d23 reported cstate change: term changed from 0 to 1, leader changed from <none> to 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194). New cstate: current_term: 1 leader_uuid: "10ca5ee68ad64b3283b9d235352c9d23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } health_report { overall_health: HEALTHY } } }
I20260709 13:03:50.158838 11148 consensus_queue.cc:260] T d270b9e5481245d9b5340a454a7cd10b P 10ca5ee68ad64b3283b9d235352c9d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:50.160749 11148 ts_tablet_manager.cc:1434] T d270b9e5481245d9b5340a454a7cd10b P 10ca5ee68ad64b3283b9d235352c9d23: Time spent starting tablet: real 0.018s	user 0.005s	sys 0.000s
I20260709 13:03:50.160722 11029 raft_consensus.cc:2468] T a64b7e99d6c54934a7de6f6dffcf5e94 P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 1.
I20260709 13:03:50.157718 11028 raft_consensus.cc:2468] T 9a1716c0adf34bee8a4e99341ba5788f P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 1.
I20260709 13:03:50.163081 11148 tablet_bootstrap.cc:492] T a64b7e99d6c54934a7de6f6dffcf5e94 P 10ca5ee68ad64b3283b9d235352c9d23: Bootstrap starting.
I20260709 13:03:50.165135 11067 leader_election.cc:304] T a64b7e99d6c54934a7de6f6dffcf5e94 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47; no voters: 10ca5ee68ad64b3283b9d235352c9d23
I20260709 13:03:50.166630 11235 raft_consensus.cc:2804] T a64b7e99d6c54934a7de6f6dffcf5e94 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:03:50.167140 11235 raft_consensus.cc:697] T a64b7e99d6c54934a7de6f6dffcf5e94 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 LEADER]: Becoming Leader. State: Replica: 0172fed22b6b4890a7fdc953fa128af4, State: Running, Role: LEADER
I20260709 13:03:50.168017 11235 consensus_queue.cc:237] T a64b7e99d6c54934a7de6f6dffcf5e94 P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:50.184356 11152 tablet_bootstrap.cc:492] T 868e92339c2f4508af0afbd07c814d64 P 0172fed22b6b4890a7fdc953fa128af4: No bootstrap required, opened a new log
I20260709 13:03:50.185256 11152 ts_tablet_manager.cc:1403] T 868e92339c2f4508af0afbd07c814d64 P 0172fed22b6b4890a7fdc953fa128af4: Time spent bootstrapping tablet: real 0.107s	user 0.043s	sys 0.035s
I20260709 13:03:50.190513 11067 leader_election.cc:304] T 9a1716c0adf34bee8a4e99341ba5788f P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47; no voters: 186b12638a87410487467df721a3cbda
I20260709 13:03:50.209801 11148 tablet_bootstrap.cc:654] T a64b7e99d6c54934a7de6f6dffcf5e94 P 10ca5ee68ad64b3283b9d235352c9d23: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:50.210973 11162 raft_consensus.cc:2804] T 9a1716c0adf34bee8a4e99341ba5788f P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:03:50.211620 11162 raft_consensus.cc:697] T 9a1716c0adf34bee8a4e99341ba5788f P 0172fed22b6b4890a7fdc953fa128af4 [term 1 LEADER]: Becoming Leader. State: Replica: 0172fed22b6b4890a7fdc953fa128af4, State: Running, Role: LEADER
I20260709 13:03:50.214399 11279 raft_consensus.cc:2749] T 7839ca2d2c13431a85f44d7c458c6351 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:03:50.217728 11162 consensus_queue.cc:237] T 9a1716c0adf34bee8a4e99341ba5788f P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:50.216626 10778 catalog_manager.cc:5697] T e96e80ba9ac34d0998395515e5c8c41a P 10ca5ee68ad64b3283b9d235352c9d23 reported cstate change: term changed from 0 to 1, leader changed from <none> to 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194). New cstate: current_term: 1 leader_uuid: "10ca5ee68ad64b3283b9d235352c9d23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } health_report { overall_health: UNKNOWN } } }
I20260709 13:03:50.214119 11152 raft_consensus.cc:359] T 868e92339c2f4508af0afbd07c814d64 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:50.222330 11152 raft_consensus.cc:385] T 868e92339c2f4508af0afbd07c814d64 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:50.222663 11152 raft_consensus.cc:740] T 868e92339c2f4508af0afbd07c814d64 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0172fed22b6b4890a7fdc953fa128af4, State: Initialized, Role: FOLLOWER
I20260709 13:03:50.211787 11196 raft_consensus.cc:493] T 5d439326f23747eaab3c1e7da2d6b785 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:50.223555 11152 consensus_queue.cc:260] T 868e92339c2f4508af0afbd07c814d64 P 0172fed22b6b4890a7fdc953fa128af4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:50.224193 11196 raft_consensus.cc:515] T 5d439326f23747eaab3c1e7da2d6b785 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:50.226202 11152 ts_tablet_manager.cc:1434] T 868e92339c2f4508af0afbd07c814d64 P 0172fed22b6b4890a7fdc953fa128af4: Time spent starting tablet: real 0.041s	user 0.008s	sys 0.000s
I20260709 13:03:50.227327 11152 tablet_bootstrap.cc:492] T a610d1a71f52432e88fe56130f921fa2 P 0172fed22b6b4890a7fdc953fa128af4: Bootstrap starting.
I20260709 13:03:50.227769 10955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d439326f23747eaab3c1e7da2d6b785" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:03:50.230476 10878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d439326f23747eaab3c1e7da2d6b785" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
W20260709 13:03:50.230844 11066 leader_election.cc:343] T 5d439326f23747eaab3c1e7da2d6b785 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:03:50.234579 11196 leader_election.cc:290] T 5d439326f23747eaab3c1e7da2d6b785 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
W20260709 13:03:50.234658 11069 leader_election.cc:343] T 5d439326f23747eaab3c1e7da2d6b785 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 186b12638a87410487467df721a3cbda (127.0.107.193:38967): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:03:50.235410 11069 leader_election.cc:304] T 5d439326f23747eaab3c1e7da2d6b785 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda
I20260709 13:03:50.239180 11196 raft_consensus.cc:2749] T 5d439326f23747eaab3c1e7da2d6b785 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:03:50.240671 11152 tablet_bootstrap.cc:654] T a610d1a71f52432e88fe56130f921fa2 P 0172fed22b6b4890a7fdc953fa128af4: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:50.212081 11149 raft_consensus.cc:359] T 2567c2d84e4648d7ba5dc23a3df32817 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:50.247694 11149 raft_consensus.cc:385] T 2567c2d84e4648d7ba5dc23a3df32817 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:50.248072 11149 raft_consensus.cc:740] T 2567c2d84e4648d7ba5dc23a3df32817 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 873357742d5843579556ee8166f23f47, State: Initialized, Role: FOLLOWER
I20260709 13:03:50.249203 11149 consensus_queue.cc:260] T 2567c2d84e4648d7ba5dc23a3df32817 P 873357742d5843579556ee8166f23f47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:50.254331 11149 ts_tablet_manager.cc:1434] T 2567c2d84e4648d7ba5dc23a3df32817 P 873357742d5843579556ee8166f23f47: Time spent starting tablet: real 0.120s	user 0.008s	sys 0.001s
I20260709 13:03:50.255779 11149 tablet_bootstrap.cc:492] T a610d1a71f52432e88fe56130f921fa2 P 873357742d5843579556ee8166f23f47: Bootstrap starting.
I20260709 13:03:50.266321 11148 tablet_bootstrap.cc:492] T a64b7e99d6c54934a7de6f6dffcf5e94 P 10ca5ee68ad64b3283b9d235352c9d23: No bootstrap required, opened a new log
I20260709 13:03:50.267019 11148 ts_tablet_manager.cc:1403] T a64b7e99d6c54934a7de6f6dffcf5e94 P 10ca5ee68ad64b3283b9d235352c9d23: Time spent bootstrapping tablet: real 0.105s	user 0.014s	sys 0.006s
I20260709 13:03:50.270797 11148 raft_consensus.cc:359] T a64b7e99d6c54934a7de6f6dffcf5e94 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:50.271665 11148 raft_consensus.cc:385] T a64b7e99d6c54934a7de6f6dffcf5e94 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:50.272012 11148 raft_consensus.cc:740] T a64b7e99d6c54934a7de6f6dffcf5e94 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10ca5ee68ad64b3283b9d235352c9d23, State: Initialized, Role: FOLLOWER
I20260709 13:03:50.273231 11148 consensus_queue.cc:260] T a64b7e99d6c54934a7de6f6dffcf5e94 P 10ca5ee68ad64b3283b9d235352c9d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:50.273988 11257 raft_consensus.cc:493] T 2567c2d84e4648d7ba5dc23a3df32817 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:50.274580 11257 raft_consensus.cc:515] T 2567c2d84e4648d7ba5dc23a3df32817 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:50.278919 10955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2567c2d84e4648d7ba5dc23a3df32817" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:03:50.279757 11148 ts_tablet_manager.cc:1434] T a64b7e99d6c54934a7de6f6dffcf5e94 P 10ca5ee68ad64b3283b9d235352c9d23: Time spent starting tablet: real 0.012s	user 0.007s	sys 0.001s
I20260709 13:03:50.281177 11148 tablet_bootstrap.cc:492] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 10ca5ee68ad64b3283b9d235352c9d23: Bootstrap starting.
W20260709 13:03:50.281025 10992 leader_election.cc:343] T 2567c2d84e4648d7ba5dc23a3df32817 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:03:50.282148 11257 leader_election.cc:290] T 2567c2d84e4648d7ba5dc23a3df32817 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:03:50.289503 11151 tablet_bootstrap.cc:492] T e96e80ba9ac34d0998395515e5c8c41a P 186b12638a87410487467df721a3cbda: No bootstrap required, opened a new log
I20260709 13:03:50.290136 11151 ts_tablet_manager.cc:1403] T e96e80ba9ac34d0998395515e5c8c41a P 186b12638a87410487467df721a3cbda: Time spent bootstrapping tablet: real 0.258s	user 0.013s	sys 0.000s
I20260709 13:03:50.297979 11151 raft_consensus.cc:359] T e96e80ba9ac34d0998395515e5c8c41a P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:50.298754 11151 raft_consensus.cc:385] T e96e80ba9ac34d0998395515e5c8c41a P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:50.299068 11151 raft_consensus.cc:740] T e96e80ba9ac34d0998395515e5c8c41a P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 186b12638a87410487467df721a3cbda, State: Initialized, Role: FOLLOWER
I20260709 13:03:50.299901 11151 consensus_queue.cc:260] T e96e80ba9ac34d0998395515e5c8c41a P 186b12638a87410487467df721a3cbda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:50.303108 11151 ts_tablet_manager.cc:1434] T e96e80ba9ac34d0998395515e5c8c41a P 186b12638a87410487467df721a3cbda: Time spent starting tablet: real 0.013s	user 0.009s	sys 0.000s
I20260709 13:03:50.304802 10777 catalog_manager.cc:5697] T a64b7e99d6c54934a7de6f6dffcf5e94 P 0172fed22b6b4890a7fdc953fa128af4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196). New cstate: current_term: 1 leader_uuid: "0172fed22b6b4890a7fdc953fa128af4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } health_report { overall_health: HEALTHY } } }
I20260709 13:03:50.306284 11148 tablet_bootstrap.cc:654] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 10ca5ee68ad64b3283b9d235352c9d23: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:50.308589 11149 tablet_bootstrap.cc:654] T a610d1a71f52432e88fe56130f921fa2 P 873357742d5843579556ee8166f23f47: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:50.311739 11257 consensus_queue.cc:1048] T 7757ecbf402e47bfbc93d0447741e26c P 873357742d5843579556ee8166f23f47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:03:50.312374 11168 raft_consensus.cc:493] T e89e640169e14e2d98f968f76027daa7 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:50.313177 11168 raft_consensus.cc:515] T e89e640169e14e2d98f968f76027daa7 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:50.318526 11257 raft_consensus.cc:493] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:50.319324 11257 raft_consensus.cc:515] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:50.320153 11028 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e89e640169e14e2d98f968f76027daa7" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:03:50.321009 11028 raft_consensus.cc:2468] T e89e640169e14e2d98f968f76027daa7 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 0.
I20260709 13:03:50.320310 10879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2567c2d84e4648d7ba5dc23a3df32817" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
W20260709 13:03:50.322896 10995 leader_election.cc:343] T 2567c2d84e4648d7ba5dc23a3df32817 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 186b12638a87410487467df721a3cbda (127.0.107.193:38967): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:03:50.323256 11279 consensus_queue.cc:1048] T 34064dc7bf3747dbb3876dd824ee0688 P 186b12638a87410487467df721a3cbda [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:03:50.323437 10995 leader_election.cc:304] T 2567c2d84e4648d7ba5dc23a3df32817 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 873357742d5843579556ee8166f23f47; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda
I20260709 13:03:50.325197 11302 raft_consensus.cc:2749] T 2567c2d84e4648d7ba5dc23a3df32817 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:03:50.325446 10919 leader_election.cc:304] T e89e640169e14e2d98f968f76027daa7 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:03:50.326259 11152 tablet_bootstrap.cc:492] T a610d1a71f52432e88fe56130f921fa2 P 0172fed22b6b4890a7fdc953fa128af4: No bootstrap required, opened a new log
I20260709 13:03:50.326463 11102 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f1466124dbd4f6ebb3c4d62ca48bf96" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:03:50.326843 11152 ts_tablet_manager.cc:1403] T a610d1a71f52432e88fe56130f921fa2 P 0172fed22b6b4890a7fdc953fa128af4: Time spent bootstrapping tablet: real 0.100s	user 0.010s	sys 0.009s
I20260709 13:03:50.327929 11151 tablet_bootstrap.cc:492] T 9a1716c0adf34bee8a4e99341ba5788f P 186b12638a87410487467df721a3cbda: Bootstrap starting.
W20260709 13:03:50.328265 10994 leader_election.cc:343] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:03:50.328621 11239 raft_consensus.cc:2804] T e89e640169e14e2d98f968f76027daa7 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:03:50.328889 11257 leader_election.cc:290] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:03:50.329109 11239 raft_consensus.cc:493] T e89e640169e14e2d98f968f76027daa7 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:50.329591 11239 raft_consensus.cc:3060] T e89e640169e14e2d98f968f76027daa7 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:03:50.331808 11152 raft_consensus.cc:359] T a610d1a71f52432e88fe56130f921fa2 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:50.335646 10955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f1466124dbd4f6ebb3c4d62ca48bf96" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:03:50.335666 10877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e89e640169e14e2d98f968f76027daa7" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
W20260709 13:03:50.347704 10992 leader_election.cc:343] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:03:50.348119 11152 raft_consensus.cc:385] T a610d1a71f52432e88fe56130f921fa2 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:50.348230 10992 leader_election.cc:304] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 873357742d5843579556ee8166f23f47; no voters: 0172fed22b6b4890a7fdc953fa128af4, 10ca5ee68ad64b3283b9d235352c9d23
I20260709 13:03:50.348519 11152 raft_consensus.cc:740] T a610d1a71f52432e88fe56130f921fa2 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0172fed22b6b4890a7fdc953fa128af4, State: Initialized, Role: FOLLOWER
I20260709 13:03:50.349175 11257 raft_consensus.cc:2749] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:03:50.340554 11168 leader_election.cc:290] T e89e640169e14e2d98f968f76027daa7 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:03:50.342576 11151 tablet_bootstrap.cc:654] T 9a1716c0adf34bee8a4e99341ba5788f P 186b12638a87410487467df721a3cbda: Neither blocks nor log segments found. Creating new log.
W20260709 13:03:50.350184 10921 leader_election.cc:343] T e89e640169e14e2d98f968f76027daa7 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 186b12638a87410487467df721a3cbda (127.0.107.193:38967): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:03:50.340116 11188 raft_consensus.cc:493] T d270b9e5481245d9b5340a454a7cd10b P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:50.351213 11152 consensus_queue.cc:260] T a610d1a71f52432e88fe56130f921fa2 P 0172fed22b6b4890a7fdc953fa128af4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:50.351471 11188 raft_consensus.cc:515] T d270b9e5481245d9b5340a454a7cd10b P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:50.353729 11188 leader_election.cc:290] T d270b9e5481245d9b5340a454a7cd10b P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:03:50.355499 11102 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d270b9e5481245d9b5340a454a7cd10b" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
W20260709 13:03:50.357079 10920 leader_election.cc:343] T d270b9e5481245d9b5340a454a7cd10b P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:03:50.358546 11152 ts_tablet_manager.cc:1434] T a610d1a71f52432e88fe56130f921fa2 P 0172fed22b6b4890a7fdc953fa128af4: Time spent starting tablet: real 0.031s	user 0.008s	sys 0.001s
I20260709 13:03:50.359668 11152 tablet_bootstrap.cc:492] T d270b9e5481245d9b5340a454a7cd10b P 0172fed22b6b4890a7fdc953fa128af4: Bootstrap starting.
I20260709 13:03:50.364223 10877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d270b9e5481245d9b5340a454a7cd10b" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
W20260709 13:03:50.375763 10921 leader_election.cc:343] T d270b9e5481245d9b5340a454a7cd10b P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 186b12638a87410487467df721a3cbda (127.0.107.193:38967): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:03:50.376259 10921 leader_election.cc:304] T d270b9e5481245d9b5340a454a7cd10b P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23; no voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda
I20260709 13:03:50.377035 11188 raft_consensus.cc:2749] T d270b9e5481245d9b5340a454a7cd10b P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:03:50.378983 11257 consensus_queue.cc:1048] T 7757ecbf402e47bfbc93d0447741e26c P 873357742d5843579556ee8166f23f47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:03:50.384150 11152 tablet_bootstrap.cc:654] T d270b9e5481245d9b5340a454a7cd10b P 0172fed22b6b4890a7fdc953fa128af4: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:50.381184 11239 raft_consensus.cc:515] T e89e640169e14e2d98f968f76027daa7 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:50.388372 11239 leader_election.cc:290] T e89e640169e14e2d98f968f76027daa7 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 1 election: Requested vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:03:50.389174 10778 catalog_manager.cc:5697] T 9a1716c0adf34bee8a4e99341ba5788f P 0172fed22b6b4890a7fdc953fa128af4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196). New cstate: current_term: 1 leader_uuid: "0172fed22b6b4890a7fdc953fa128af4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } health_report { overall_health: UNKNOWN } } }
I20260709 13:03:50.390249 11028 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e89e640169e14e2d98f968f76027daa7" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
I20260709 13:03:50.390946 11028 raft_consensus.cc:3060] T e89e640169e14e2d98f968f76027daa7 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:03:50.400777 11149 tablet_bootstrap.cc:492] T a610d1a71f52432e88fe56130f921fa2 P 873357742d5843579556ee8166f23f47: No bootstrap required, opened a new log
I20260709 13:03:50.401609 11149 ts_tablet_manager.cc:1403] T a610d1a71f52432e88fe56130f921fa2 P 873357742d5843579556ee8166f23f47: Time spent bootstrapping tablet: real 0.146s	user 0.010s	sys 0.015s
I20260709 13:03:50.409121 11151 tablet_bootstrap.cc:492] T 9a1716c0adf34bee8a4e99341ba5788f P 186b12638a87410487467df721a3cbda: No bootstrap required, opened a new log
I20260709 13:03:50.413700 11151 ts_tablet_manager.cc:1403] T 9a1716c0adf34bee8a4e99341ba5788f P 186b12638a87410487467df721a3cbda: Time spent bootstrapping tablet: real 0.086s	user 0.018s	sys 0.022s
I20260709 13:03:50.417096 11151 raft_consensus.cc:359] T 9a1716c0adf34bee8a4e99341ba5788f P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:50.417832 11151 raft_consensus.cc:385] T 9a1716c0adf34bee8a4e99341ba5788f P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:50.418141 11151 raft_consensus.cc:740] T 9a1716c0adf34bee8a4e99341ba5788f P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 186b12638a87410487467df721a3cbda, State: Initialized, Role: FOLLOWER
I20260709 13:03:50.419071 11151 consensus_queue.cc:260] T 9a1716c0adf34bee8a4e99341ba5788f P 186b12638a87410487467df721a3cbda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:50.439524 11151 ts_tablet_manager.cc:1434] T 9a1716c0adf34bee8a4e99341ba5788f P 186b12638a87410487467df721a3cbda: Time spent starting tablet: real 0.025s	user 0.000s	sys 0.008s
I20260709 13:03:50.445847 11151 tablet_bootstrap.cc:492] T e89e640169e14e2d98f968f76027daa7 P 186b12638a87410487467df721a3cbda: Bootstrap starting.
I20260709 13:03:50.448163 10878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e89e640169e14e2d98f968f76027daa7" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:03:50.449846 11332 raft_consensus.cc:493] T 9a1716c0adf34bee8a4e99341ba5788f P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:03:50.450163 10921 leader_election.cc:343] T e89e640169e14e2d98f968f76027daa7 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 186b12638a87410487467df721a3cbda (127.0.107.193:38967): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:03:50.450740 11332 raft_consensus.cc:515] T 9a1716c0adf34bee8a4e99341ba5788f P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:50.452976 11151 tablet_bootstrap.cc:654] T e89e640169e14e2d98f968f76027daa7 P 186b12638a87410487467df721a3cbda: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:50.453883 11332 leader_election.cc:290] T 9a1716c0adf34bee8a4e99341ba5788f P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:03:50.458333 11102 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a1716c0adf34bee8a4e99341ba5788f" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:03:50.461167 11027 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a1716c0adf34bee8a4e99341ba5788f" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:03:50.462167 11027 raft_consensus.cc:2393] T 9a1716c0adf34bee8a4e99341ba5788f P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 186b12638a87410487467df721a3cbda in current term 1: Already voted for candidate 0172fed22b6b4890a7fdc953fa128af4 in this term.
I20260709 13:03:50.465307 10878 raft_consensus.cc:3060] T 836251ebc2b74f5da88b8a8b2338e2a5 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:03:50.473694 11149 raft_consensus.cc:359] T a610d1a71f52432e88fe56130f921fa2 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:50.475963 11149 raft_consensus.cc:385] T a610d1a71f52432e88fe56130f921fa2 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:50.476354 11149 raft_consensus.cc:740] T a610d1a71f52432e88fe56130f921fa2 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 873357742d5843579556ee8166f23f47, State: Initialized, Role: FOLLOWER
I20260709 13:03:50.480572 11149 consensus_queue.cc:260] T a610d1a71f52432e88fe56130f921fa2 P 873357742d5843579556ee8166f23f47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:50.484962 11148 tablet_bootstrap.cc:492] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 10ca5ee68ad64b3283b9d235352c9d23: No bootstrap required, opened a new log
I20260709 13:03:50.485550 11148 ts_tablet_manager.cc:1403] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 10ca5ee68ad64b3283b9d235352c9d23: Time spent bootstrapping tablet: real 0.205s	user 0.015s	sys 0.002s
I20260709 13:03:50.479714 10844 leader_election.cc:304] T 9a1716c0adf34bee8a4e99341ba5788f P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 186b12638a87410487467df721a3cbda; no voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47
I20260709 13:03:50.493305 11302 consensus_queue.cc:1048] T d71ca3a1ba6b48478aaf62741bf3cf6a P 873357742d5843579556ee8166f23f47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:03:50.501736 11148 raft_consensus.cc:359] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:50.504340 11148 raft_consensus.cc:385] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:50.504631 11148 raft_consensus.cc:740] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10ca5ee68ad64b3283b9d235352c9d23, State: Initialized, Role: FOLLOWER
I20260709 13:03:50.505390 11148 consensus_queue.cc:260] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 10ca5ee68ad64b3283b9d235352c9d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:50.502337 11149 ts_tablet_manager.cc:1434] T a610d1a71f52432e88fe56130f921fa2 P 873357742d5843579556ee8166f23f47: Time spent starting tablet: real 0.100s	user 0.007s	sys 0.004s
I20260709 13:03:50.500769 11333 raft_consensus.cc:493] T e96e80ba9ac34d0998395515e5c8c41a P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:50.508097 11149 tablet_bootstrap.cc:492] T 868e92339c2f4508af0afbd07c814d64 P 873357742d5843579556ee8166f23f47: Bootstrap starting.
I20260709 13:03:50.499881 11292 raft_consensus.cc:3060] T 9a1716c0adf34bee8a4e99341ba5788f P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:03:50.501991 11028 raft_consensus.cc:2468] T e89e640169e14e2d98f968f76027daa7 P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 1.
I20260709 13:03:50.520051 11148 ts_tablet_manager.cc:1434] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 10ca5ee68ad64b3283b9d235352c9d23: Time spent starting tablet: real 0.034s	user 0.008s	sys 0.002s
I20260709 13:03:50.521376 11148 tablet_bootstrap.cc:492] T 72b111ad7b5f4753ab1d2b5ee1114840 P 10ca5ee68ad64b3283b9d235352c9d23: Bootstrap starting.
I20260709 13:03:50.528860 11151 tablet_bootstrap.cc:492] T e89e640169e14e2d98f968f76027daa7 P 186b12638a87410487467df721a3cbda: No bootstrap required, opened a new log
I20260709 13:03:50.529645 11151 ts_tablet_manager.cc:1403] T e89e640169e14e2d98f968f76027daa7 P 186b12638a87410487467df721a3cbda: Time spent bootstrapping tablet: real 0.089s	user 0.011s	sys 0.007s
I20260709 13:03:50.507854 11333 raft_consensus.cc:515] T e96e80ba9ac34d0998395515e5c8c41a P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:50.551082 11302 consensus_queue.cc:1048] T d71ca3a1ba6b48478aaf62741bf3cf6a P 873357742d5843579556ee8166f23f47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:03:50.538527 10919 leader_election.cc:304] T e89e640169e14e2d98f968f76027daa7 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47; no voters: 186b12638a87410487467df721a3cbda
I20260709 13:03:50.555562 11239 raft_consensus.cc:2804] T e89e640169e14e2d98f968f76027daa7 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:03:50.556277 11239 raft_consensus.cc:697] T e89e640169e14e2d98f968f76027daa7 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 LEADER]: Becoming Leader. State: Replica: 10ca5ee68ad64b3283b9d235352c9d23, State: Running, Role: LEADER
I20260709 13:03:50.557443 11239 consensus_queue.cc:237] T e89e640169e14e2d98f968f76027daa7 P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:50.558327 11168 consensus_queue.cc:1048] T 8a69f32c53f24a50b7852d143665d7b4 P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:03:50.554128 11151 raft_consensus.cc:359] T e89e640169e14e2d98f968f76027daa7 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:50.564139 11151 raft_consensus.cc:385] T e89e640169e14e2d98f968f76027daa7 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:50.569446 11149 tablet_bootstrap.cc:654] T 868e92339c2f4508af0afbd07c814d64 P 873357742d5843579556ee8166f23f47: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:50.564522 11151 raft_consensus.cc:740] T e89e640169e14e2d98f968f76027daa7 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 186b12638a87410487467df721a3cbda, State: Initialized, Role: FOLLOWER
I20260709 13:03:50.572311 11151 consensus_queue.cc:260] T e89e640169e14e2d98f968f76027daa7 P 186b12638a87410487467df721a3cbda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:50.579732 11152 tablet_bootstrap.cc:492] T d270b9e5481245d9b5340a454a7cd10b P 0172fed22b6b4890a7fdc953fa128af4: No bootstrap required, opened a new log
I20260709 13:03:50.580312 11152 ts_tablet_manager.cc:1403] T d270b9e5481245d9b5340a454a7cd10b P 0172fed22b6b4890a7fdc953fa128af4: Time spent bootstrapping tablet: real 0.221s	user 0.025s	sys 0.015s
I20260709 13:03:50.581611 11148 tablet_bootstrap.cc:654] T 72b111ad7b5f4753ab1d2b5ee1114840 P 10ca5ee68ad64b3283b9d235352c9d23: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:50.581286 11188 consensus_queue.cc:1048] T e96e80ba9ac34d0998395515e5c8c41a P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:03:50.603595 11152 raft_consensus.cc:359] T d270b9e5481245d9b5340a454a7cd10b P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:50.604503 11152 raft_consensus.cc:385] T d270b9e5481245d9b5340a454a7cd10b P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:50.605038 11152 raft_consensus.cc:740] T d270b9e5481245d9b5340a454a7cd10b P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0172fed22b6b4890a7fdc953fa128af4, State: Initialized, Role: FOLLOWER
I20260709 13:03:50.608321 11151 ts_tablet_manager.cc:1434] T e89e640169e14e2d98f968f76027daa7 P 186b12638a87410487467df721a3cbda: Time spent starting tablet: real 0.078s	user 0.010s	sys 0.000s
I20260709 13:03:50.609001 10955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e96e80ba9ac34d0998395515e5c8c41a" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:03:50.609580 11151 tablet_bootstrap.cc:492] T 868e92339c2f4508af0afbd07c814d64 P 186b12638a87410487467df721a3cbda: Bootstrap starting.
I20260709 13:03:50.612905 11152 consensus_queue.cc:260] T d270b9e5481245d9b5340a454a7cd10b P 0172fed22b6b4890a7fdc953fa128af4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:50.567260 11168 consensus_queue.cc:1048] T 8a69f32c53f24a50b7852d143665d7b4 P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.036s
I20260709 13:03:50.609077 11333 leader_election.cc:290] T e96e80ba9ac34d0998395515e5c8c41a P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:03:50.616106 11196 consensus_queue.cc:1048] T 836251ebc2b74f5da88b8a8b2338e2a5 P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
I20260709 13:03:50.622951 11196 raft_consensus.cc:493] T 868e92339c2f4508af0afbd07c814d64 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:50.623454 11196 raft_consensus.cc:515] T 868e92339c2f4508af0afbd07c814d64 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:50.634662 11152 ts_tablet_manager.cc:1434] T d270b9e5481245d9b5340a454a7cd10b P 0172fed22b6b4890a7fdc953fa128af4: Time spent starting tablet: real 0.054s	user 0.012s	sys 0.000s
I20260709 13:03:50.637053 11148 tablet_bootstrap.cc:492] T 72b111ad7b5f4753ab1d2b5ee1114840 P 10ca5ee68ad64b3283b9d235352c9d23: No bootstrap required, opened a new log
I20260709 13:03:50.637795 11152 tablet_bootstrap.cc:492] T 72b111ad7b5f4753ab1d2b5ee1114840 P 0172fed22b6b4890a7fdc953fa128af4: Bootstrap starting.
I20260709 13:03:50.638016 11148 ts_tablet_manager.cc:1403] T 72b111ad7b5f4753ab1d2b5ee1114840 P 10ca5ee68ad64b3283b9d235352c9d23: Time spent bootstrapping tablet: real 0.117s	user 0.012s	sys 0.019s
I20260709 13:03:50.640422 11028 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e96e80ba9ac34d0998395515e5c8c41a" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:03:50.641402 11028 raft_consensus.cc:2393] T e96e80ba9ac34d0998395515e5c8c41a P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 186b12638a87410487467df721a3cbda in current term 1: Already voted for candidate 10ca5ee68ad64b3283b9d235352c9d23 in this term.
I20260709 13:03:50.634261 11162 consensus_queue.cc:1048] T a64b7e99d6c54934a7de6f6dffcf5e94 P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.011s
I20260709 13:03:50.647648 10844 leader_election.cc:304] T e96e80ba9ac34d0998395515e5c8c41a P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 186b12638a87410487467df721a3cbda; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47
I20260709 13:03:50.651185 11196 leader_election.cc:290] T 868e92339c2f4508af0afbd07c814d64 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:03:50.651253 11333 raft_consensus.cc:3060] T e96e80ba9ac34d0998395515e5c8c41a P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:03:50.652444 11029 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "868e92339c2f4508af0afbd07c814d64" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:03:50.652575 11148 raft_consensus.cc:359] T 72b111ad7b5f4753ab1d2b5ee1114840 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:50.653342 11148 raft_consensus.cc:385] T 72b111ad7b5f4753ab1d2b5ee1114840 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:50.653272 11332 raft_consensus.cc:493] T 836251ebc2b74f5da88b8a8b2338e2a5 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:50.653860 11148 raft_consensus.cc:740] T 72b111ad7b5f4753ab1d2b5ee1114840 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10ca5ee68ad64b3283b9d235352c9d23, State: Initialized, Role: FOLLOWER
I20260709 13:03:50.654064 11332 raft_consensus.cc:515] T 836251ebc2b74f5da88b8a8b2338e2a5 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
W20260709 13:03:50.654505 11067 leader_election.cc:343] T 868e92339c2f4508af0afbd07c814d64 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 873357742d5843579556ee8166f23f47 (127.0.107.195:45807): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:03:50.659271 11102 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "836251ebc2b74f5da88b8a8b2338e2a5" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:03:50.659618 11292 raft_consensus.cc:2749] T 9a1716c0adf34bee8a4e99341ba5788f P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:03:50.662483 10877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "868e92339c2f4508af0afbd07c814d64" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:03:50.663762 10954 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "836251ebc2b74f5da88b8a8b2338e2a5" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:03:50.669580 11148 consensus_queue.cc:260] T 72b111ad7b5f4753ab1d2b5ee1114840 P 10ca5ee68ad64b3283b9d235352c9d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:50.675484 11332 leader_election.cc:290] T 836251ebc2b74f5da88b8a8b2338e2a5 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:03:50.688947 10843 leader_election.cc:304] T 836251ebc2b74f5da88b8a8b2338e2a5 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 186b12638a87410487467df721a3cbda; no voters: 0172fed22b6b4890a7fdc953fa128af4, 10ca5ee68ad64b3283b9d235352c9d23
W20260709 13:03:50.664431 11069 leader_election.cc:343] T 868e92339c2f4508af0afbd07c814d64 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 186b12638a87410487467df721a3cbda (127.0.107.193:38967): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:03:50.691238 11069 leader_election.cc:304] T 868e92339c2f4508af0afbd07c814d64 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4; no voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47
I20260709 13:03:50.690789 11332 raft_consensus.cc:2749] T 836251ebc2b74f5da88b8a8b2338e2a5 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:03:50.708002 11148 ts_tablet_manager.cc:1434] T 72b111ad7b5f4753ab1d2b5ee1114840 P 10ca5ee68ad64b3283b9d235352c9d23: Time spent starting tablet: real 0.070s	user 0.006s	sys 0.003s
I20260709 13:03:50.711592 11148 tablet_bootstrap.cc:492] T 2567c2d84e4648d7ba5dc23a3df32817 P 10ca5ee68ad64b3283b9d235352c9d23: Bootstrap starting.
I20260709 13:03:50.701162 11162 raft_consensus.cc:2749] T 868e92339c2f4508af0afbd07c814d64 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:03:50.720257 11149 tablet_bootstrap.cc:492] T 868e92339c2f4508af0afbd07c814d64 P 873357742d5843579556ee8166f23f47: No bootstrap required, opened a new log
I20260709 13:03:50.725049 11188 raft_consensus.cc:493] T a64b7e99d6c54934a7de6f6dffcf5e94 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:50.725677 11188 raft_consensus.cc:515] T a64b7e99d6c54934a7de6f6dffcf5e94 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:50.725394 10778 catalog_manager.cc:5697] T e89e640169e14e2d98f968f76027daa7 P 10ca5ee68ad64b3283b9d235352c9d23 reported cstate change: term changed from 0 to 1, leader changed from <none> to 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194). New cstate: current_term: 1 leader_uuid: "10ca5ee68ad64b3283b9d235352c9d23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } health_report { overall_health: UNKNOWN } } }
I20260709 13:03:50.727283 11152 tablet_bootstrap.cc:654] T 72b111ad7b5f4753ab1d2b5ee1114840 P 0172fed22b6b4890a7fdc953fa128af4: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:50.728938 11188 leader_election.cc:290] T a64b7e99d6c54934a7de6f6dffcf5e94 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:03:50.731714 11102 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a64b7e99d6c54934a7de6f6dffcf5e94" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:03:50.731890 11028 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a64b7e99d6c54934a7de6f6dffcf5e94" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:03:50.732225 10953 raft_consensus.cc:3060] T a64b7e99d6c54934a7de6f6dffcf5e94 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:03:50.740821 10920 leader_election.cc:304] T a64b7e99d6c54934a7de6f6dffcf5e94 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23; no voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47
I20260709 13:03:50.742272 11151 tablet_bootstrap.cc:654] T 868e92339c2f4508af0afbd07c814d64 P 186b12638a87410487467df721a3cbda: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:50.720927 11149 ts_tablet_manager.cc:1403] T 868e92339c2f4508af0afbd07c814d64 P 873357742d5843579556ee8166f23f47: Time spent bootstrapping tablet: real 0.213s	user 0.019s	sys 0.016s
I20260709 13:03:50.761577 11188 raft_consensus.cc:2749] T a64b7e99d6c54934a7de6f6dffcf5e94 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:03:50.707878 11196 consensus_queue.cc:1048] T 836251ebc2b74f5da88b8a8b2338e2a5 P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.058s
I20260709 13:03:50.773836 11333 raft_consensus.cc:2749] T e96e80ba9ac34d0998395515e5c8c41a P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:03:50.783046 11162 raft_consensus.cc:493] T a610d1a71f52432e88fe56130f921fa2 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:50.783736 11162 raft_consensus.cc:515] T a610d1a71f52432e88fe56130f921fa2 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:50.785422 11149 raft_consensus.cc:359] T 868e92339c2f4508af0afbd07c814d64 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:50.786444 11149 raft_consensus.cc:385] T 868e92339c2f4508af0afbd07c814d64 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:50.786796 11149 raft_consensus.cc:740] T 868e92339c2f4508af0afbd07c814d64 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 873357742d5843579556ee8166f23f47, State: Initialized, Role: FOLLOWER
I20260709 13:03:50.795708 11188 consensus_queue.cc:1048] T e96e80ba9ac34d0998395515e5c8c41a P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:03:50.794795 11196 consensus_queue.cc:1048] T a64b7e99d6c54934a7de6f6dffcf5e94 P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:03:50.808609 11149 consensus_queue.cc:260] T 868e92339c2f4508af0afbd07c814d64 P 873357742d5843579556ee8166f23f47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:50.825525 11305 raft_consensus.cc:493] T a610d1a71f52432e88fe56130f921fa2 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:50.839341 11149 ts_tablet_manager.cc:1434] T 868e92339c2f4508af0afbd07c814d64 P 873357742d5843579556ee8166f23f47: Time spent starting tablet: real 0.092s	user 0.006s	sys 0.003s
I20260709 13:03:50.840417 11149 tablet_bootstrap.cc:492] T 65bf0a62f4d84ef18dd6b23e082724b8 P 873357742d5843579556ee8166f23f47: Bootstrap starting.
I20260709 13:03:50.848541 11188 raft_consensus.cc:493] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:50.849375 11188 raft_consensus.cc:515] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:50.851863 11151 tablet_bootstrap.cc:492] T 868e92339c2f4508af0afbd07c814d64 P 186b12638a87410487467df721a3cbda: No bootstrap required, opened a new log
I20260709 13:03:50.852489 11151 ts_tablet_manager.cc:1403] T 868e92339c2f4508af0afbd07c814d64 P 186b12638a87410487467df721a3cbda: Time spent bootstrapping tablet: real 0.243s	user 0.039s	sys 0.030s
W20260709 13:03:50.856587 11348 log.cc:927] Time spent T e89e640169e14e2d98f968f76027daa7 P 10ca5ee68ad64b3283b9d235352c9d23: Append to log took a long time: real 0.085s	user 0.001s	sys 0.000s
I20260709 13:03:50.862804 11151 raft_consensus.cc:359] T 868e92339c2f4508af0afbd07c814d64 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:50.863853 11151 raft_consensus.cc:385] T 868e92339c2f4508af0afbd07c814d64 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:50.864244 11151 raft_consensus.cc:740] T 868e92339c2f4508af0afbd07c814d64 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 186b12638a87410487467df721a3cbda, State: Initialized, Role: FOLLOWER
I20260709 13:03:50.867584 11151 consensus_queue.cc:260] T 868e92339c2f4508af0afbd07c814d64 P 186b12638a87410487467df721a3cbda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:50.868404 11027 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f1466124dbd4f6ebb3c4d62ca48bf96" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:03:50.869200 11027 raft_consensus.cc:2468] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 0.
I20260709 13:03:50.870533 11151 ts_tablet_manager.cc:1434] T 868e92339c2f4508af0afbd07c814d64 P 186b12638a87410487467df721a3cbda: Time spent starting tablet: real 0.018s	user 0.010s	sys 0.000s
I20260709 13:03:50.870702 10919 leader_election.cc:304] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:03:50.871778 11151 tablet_bootstrap.cc:492] T 2567c2d84e4648d7ba5dc23a3df32817 P 186b12638a87410487467df721a3cbda: Bootstrap starting.
I20260709 13:03:50.871552 11162 leader_election.cc:290] T a610d1a71f52432e88fe56130f921fa2 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
W20260709 13:03:50.870617 11349 log.cc:927] Time spent T 8a69f32c53f24a50b7852d143665d7b4 P 186b12638a87410487467df721a3cbda: Append to log took a long time: real 0.065s	user 0.004s	sys 0.000s
I20260709 13:03:50.859522 11188 leader_election.cc:290] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:03:50.878300 11152 tablet_bootstrap.cc:492] T 72b111ad7b5f4753ab1d2b5ee1114840 P 0172fed22b6b4890a7fdc953fa128af4: No bootstrap required, opened a new log
I20260709 13:03:50.879787 11152 ts_tablet_manager.cc:1403] T 72b111ad7b5f4753ab1d2b5ee1114840 P 0172fed22b6b4890a7fdc953fa128af4: Time spent bootstrapping tablet: real 0.242s	user 0.030s	sys 0.016s
I20260709 13:03:50.871752 11341 raft_consensus.cc:2804] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:03:50.881968 11239 raft_consensus.cc:493] T 72b111ad7b5f4753ab1d2b5ee1114840 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:50.884085 11148 tablet_bootstrap.cc:654] T 2567c2d84e4648d7ba5dc23a3df32817 P 10ca5ee68ad64b3283b9d235352c9d23: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:50.902786 11149 tablet_bootstrap.cc:654] T 65bf0a62f4d84ef18dd6b23e082724b8 P 873357742d5843579556ee8166f23f47: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:50.844717 11305 raft_consensus.cc:515] T a610d1a71f52432e88fe56130f921fa2 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:50.910912 11239 raft_consensus.cc:515] T 72b111ad7b5f4753ab1d2b5ee1114840 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:50.908841 11341 raft_consensus.cc:493] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:50.911669 11151 tablet_bootstrap.cc:654] T 2567c2d84e4648d7ba5dc23a3df32817 P 186b12638a87410487467df721a3cbda: Neither blocks nor log segments found. Creating new log.
W20260709 13:03:50.918417 11067 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:45807, user_credentials={real_user=slave}} blocked reactor thread for 72636.2us
I20260709 13:03:50.919826 11341 raft_consensus.cc:3060] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:03:50.926820 11333 raft_consensus.cc:493] T e89e640169e14e2d98f968f76027daa7 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:50.927560 11333 raft_consensus.cc:515] T e89e640169e14e2d98f968f76027daa7 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:50.932044 11152 raft_consensus.cc:359] T 72b111ad7b5f4753ab1d2b5ee1114840 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:50.932857 11152 raft_consensus.cc:385] T 72b111ad7b5f4753ab1d2b5ee1114840 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:50.933233 11239 leader_election.cc:290] T 72b111ad7b5f4753ab1d2b5ee1114840 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:03:50.933687 11152 raft_consensus.cc:740] T 72b111ad7b5f4753ab1d2b5ee1114840 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0172fed22b6b4890a7fdc953fa128af4, State: Initialized, Role: FOLLOWER
I20260709 13:03:50.934715 11152 consensus_queue.cc:260] T 72b111ad7b5f4753ab1d2b5ee1114840 P 0172fed22b6b4890a7fdc953fa128af4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:50.936367 11103 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72b111ad7b5f4753ab1d2b5ee1114840" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:03:50.937100 11103 raft_consensus.cc:2468] T 72b111ad7b5f4753ab1d2b5ee1114840 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 0.
I20260709 13:03:50.937453 10880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72b111ad7b5f4753ab1d2b5ee1114840" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:03:50.938856 10878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a610d1a71f52432e88fe56130f921fa2" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
W20260709 13:03:50.940568 10921 leader_election.cc:343] T 72b111ad7b5f4753ab1d2b5ee1114840 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 186b12638a87410487467df721a3cbda (127.0.107.193:38967): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:03:50.940688 10955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e89e640169e14e2d98f968f76027daa7" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:03:50.941828 10920 leader_election.cc:304] T 72b111ad7b5f4753ab1d2b5ee1114840 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 10ca5ee68ad64b3283b9d235352c9d23; no voters: 186b12638a87410487467df721a3cbda
I20260709 13:03:50.942866 11333 leader_election.cc:290] T e89e640169e14e2d98f968f76027daa7 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:03:50.944149 11305 leader_election.cc:290] T a610d1a71f52432e88fe56130f921fa2 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:03:50.944142 11029 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e89e640169e14e2d98f968f76027daa7" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:03:50.946131 11029 raft_consensus.cc:2393] T e89e640169e14e2d98f968f76027daa7 P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 186b12638a87410487467df721a3cbda in current term 1: Already voted for candidate 10ca5ee68ad64b3283b9d235352c9d23 in this term.
I20260709 13:03:50.947541 11239 raft_consensus.cc:2804] T 72b111ad7b5f4753ab1d2b5ee1114840 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:03:50.948078 11239 raft_consensus.cc:493] T 72b111ad7b5f4753ab1d2b5ee1114840 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:50.948405 10878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a610d1a71f52432e88fe56130f921fa2" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:03:50.948622 11239 raft_consensus.cc:3060] T 72b111ad7b5f4753ab1d2b5ee1114840 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:03:50.876317 11101 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f1466124dbd4f6ebb3c4d62ca48bf96" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
W20260709 13:03:50.953969 10995 leader_election.cc:343] T a610d1a71f52432e88fe56130f921fa2 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 186b12638a87410487467df721a3cbda (127.0.107.193:38967): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:03:50.954932 11027 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a610d1a71f52432e88fe56130f921fa2" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:03:50.955665 11027 raft_consensus.cc:2468] T a610d1a71f52432e88fe56130f921fa2 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 0.
I20260709 13:03:50.958560 11332 raft_consensus.cc:493] T 868e92339c2f4508af0afbd07c814d64 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:50.958305 11239 raft_consensus.cc:515] T 72b111ad7b5f4753ab1d2b5ee1114840 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:50.959162 11332 raft_consensus.cc:515] T 868e92339c2f4508af0afbd07c814d64 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:50.961206 11239 leader_election.cc:290] T 72b111ad7b5f4753ab1d2b5ee1114840 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 1 election: Requested vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:03:50.961895 11332 leader_election.cc:290] T 868e92339c2f4508af0afbd07c814d64 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:03:50.962617 10878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72b111ad7b5f4753ab1d2b5ee1114840" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
W20260709 13:03:50.964146 10921 leader_election.cc:343] T 72b111ad7b5f4753ab1d2b5ee1114840 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 186b12638a87410487467df721a3cbda (127.0.107.193:38967): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:03:50.971068 11102 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "868e92339c2f4508af0afbd07c814d64" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:03:50.971807 11102 raft_consensus.cc:2468] T 868e92339c2f4508af0afbd07c814d64 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 0.
W20260709 13:03:50.973655 11069 leader_election.cc:343] T a610d1a71f52432e88fe56130f921fa2 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 186b12638a87410487467df721a3cbda (127.0.107.193:38967): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:03:50.974885 10845 leader_election.cc:304] T 868e92339c2f4508af0afbd07c814d64 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda; no voters: 
I20260709 13:03:50.974649 11341 raft_consensus.cc:515] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:50.975937 11332 raft_consensus.cc:2804] T 868e92339c2f4508af0afbd07c814d64 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:03:50.976337 11332 raft_consensus.cc:493] T 868e92339c2f4508af0afbd07c814d64 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:50.976717 11332 raft_consensus.cc:3060] T 868e92339c2f4508af0afbd07c814d64 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:03:50.978426 11354 consensus_queue.cc:1048] T 9a1716c0adf34bee8a4e99341ba5788f P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:03:50.978883 11101 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a610d1a71f52432e88fe56130f921fa2" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:03:50.979593 11101 raft_consensus.cc:2468] T a610d1a71f52432e88fe56130f921fa2 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 0.
I20260709 13:03:50.979741 11029 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f1466124dbd4f6ebb3c4d62ca48bf96" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
I20260709 13:03:50.980041 11341 leader_election.cc:290] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 1 election: Requested vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:03:50.980527 11029 raft_consensus.cc:3060] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:03:50.995097 10994 leader_election.cc:304] T a610d1a71f52432e88fe56130f921fa2 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47; no voters: 186b12638a87410487467df721a3cbda
I20260709 13:03:50.996587 11257 raft_consensus.cc:2804] T a610d1a71f52432e88fe56130f921fa2 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:03:50.997112 11257 raft_consensus.cc:493] T a610d1a71f52432e88fe56130f921fa2 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:50.997515 11257 raft_consensus.cc:3060] T a610d1a71f52432e88fe56130f921fa2 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:03:51.014122 11257 raft_consensus.cc:515] T a610d1a71f52432e88fe56130f921fa2 P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:51.019496 11026 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "868e92339c2f4508af0afbd07c814d64" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:03:51.022313 11067 leader_election.cc:304] T a610d1a71f52432e88fe56130f921fa2 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47; no voters: 186b12638a87410487467df721a3cbda
I20260709 13:03:51.034281 11152 ts_tablet_manager.cc:1434] T 72b111ad7b5f4753ab1d2b5ee1114840 P 0172fed22b6b4890a7fdc953fa128af4: Time spent starting tablet: real 0.128s	user 0.009s	sys 0.000s
I20260709 13:03:51.037952 11029 raft_consensus.cc:2468] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 1.
I20260709 13:03:51.050741 11305 raft_consensus.cc:493] T 868e92339c2f4508af0afbd07c814d64 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:51.092713 11305 raft_consensus.cc:515] T 868e92339c2f4508af0afbd07c814d64 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:51.095206 11305 leader_election.cc:290] T 868e92339c2f4508af0afbd07c814d64 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:03:51.096060 11101 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "868e92339c2f4508af0afbd07c814d64" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:03:51.091032 11149 tablet_bootstrap.cc:492] T 65bf0a62f4d84ef18dd6b23e082724b8 P 873357742d5843579556ee8166f23f47: No bootstrap required, opened a new log
W20260709 13:03:51.096228 11067 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:45807, user_credentials={real_user=slave}} blocked reactor thread for 74145.3us
I20260709 13:03:51.096903 11101 raft_consensus.cc:2468] T 868e92339c2f4508af0afbd07c814d64 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 0.
I20260709 13:03:51.097142 11026 raft_consensus.cc:2468] T 868e92339c2f4508af0afbd07c814d64 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 0.
I20260709 13:03:51.097862 11311 raft_consensus.cc:2804] T a610d1a71f52432e88fe56130f921fa2 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:03:51.098414 11311 raft_consensus.cc:493] T a610d1a71f52432e88fe56130f921fa2 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:51.097878 11354 raft_consensus.cc:493] T d270b9e5481245d9b5340a454a7cd10b P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:51.098721 11149 ts_tablet_manager.cc:1403] T 65bf0a62f4d84ef18dd6b23e082724b8 P 873357742d5843579556ee8166f23f47: Time spent bootstrapping tablet: real 0.259s	user 0.016s	sys 0.003s
I20260709 13:03:51.098981 11311 raft_consensus.cc:3060] T a610d1a71f52432e88fe56130f921fa2 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:03:51.099164 11354 raft_consensus.cc:515] T d270b9e5481245d9b5340a454a7cd10b P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:51.101567 11354 leader_election.cc:290] T d270b9e5481245d9b5340a454a7cd10b P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:03:51.102342 11149 raft_consensus.cc:359] T 65bf0a62f4d84ef18dd6b23e082724b8 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:51.102870 11257 leader_election.cc:290] T a610d1a71f52432e88fe56130f921fa2 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 1 election: Requested vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:03:51.103094 11149 raft_consensus.cc:385] T 65bf0a62f4d84ef18dd6b23e082724b8 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:51.098117 11196 consensus_queue.cc:1048] T 9a1716c0adf34bee8a4e99341ba5788f P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
I20260709 13:03:51.103443 11149 raft_consensus.cc:740] T 65bf0a62f4d84ef18dd6b23e082724b8 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 873357742d5843579556ee8166f23f47, State: Initialized, Role: FOLLOWER
W20260709 13:03:51.086783 10920 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 145054us
I20260709 13:03:51.104215 11149 consensus_queue.cc:260] T 65bf0a62f4d84ef18dd6b23e082724b8 P 873357742d5843579556ee8166f23f47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:51.105978 10843 leader_election.cc:304] T e89e640169e14e2d98f968f76027daa7 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 186b12638a87410487467df721a3cbda; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47
I20260709 13:03:51.108621 10994 leader_election.cc:304] T 868e92339c2f4508af0afbd07c814d64 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:03:51.109618 10878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d270b9e5481245d9b5340a454a7cd10b" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:03:51.111222 11101 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a610d1a71f52432e88fe56130f921fa2" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:03:51.086129 11152 tablet_bootstrap.cc:492] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 0172fed22b6b4890a7fdc953fa128af4: Bootstrap starting.
W20260709 13:03:51.113683 11069 leader_election.cc:343] T d270b9e5481245d9b5340a454a7cd10b P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 186b12638a87410487467df721a3cbda (127.0.107.193:38967): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:03:51.114846 11301 raft_consensus.cc:2804] T 868e92339c2f4508af0afbd07c814d64 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:03:51.115278 11301 raft_consensus.cc:493] T 868e92339c2f4508af0afbd07c814d64 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:51.115621 11301 raft_consensus.cc:3060] T 868e92339c2f4508af0afbd07c814d64 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:03:51.113941 11149 ts_tablet_manager.cc:1434] T 65bf0a62f4d84ef18dd6b23e082724b8 P 873357742d5843579556ee8166f23f47: Time spent starting tablet: real 0.015s	user 0.007s	sys 0.000s
I20260709 13:03:51.116999 11149 tablet_bootstrap.cc:492] T bac7e5040f984ba2b7146bd36af7267e P 873357742d5843579556ee8166f23f47: Bootstrap starting.
I20260709 13:03:51.120191 11292 raft_consensus.cc:3060] T e89e640169e14e2d98f968f76027daa7 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:03:51.123445 11151 tablet_bootstrap.cc:492] T 2567c2d84e4648d7ba5dc23a3df32817 P 186b12638a87410487467df721a3cbda: No bootstrap required, opened a new log
I20260709 13:03:51.127131 10919 leader_election.cc:304] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:03:51.130218 11363 raft_consensus.cc:2804] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:03:51.130754 11363 raft_consensus.cc:697] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 LEADER]: Becoming Leader. State: Replica: 10ca5ee68ad64b3283b9d235352c9d23, State: Running, Role: LEADER
I20260709 13:03:51.131413 11292 raft_consensus.cc:2749] T e89e640169e14e2d98f968f76027daa7 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:03:51.131668 11363 consensus_queue.cc:237] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:51.133654 11151 ts_tablet_manager.cc:1403] T 2567c2d84e4648d7ba5dc23a3df32817 P 186b12638a87410487467df721a3cbda: Time spent bootstrapping tablet: real 0.252s	user 0.051s	sys 0.072s
I20260709 13:03:51.136862 11102 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72b111ad7b5f4753ab1d2b5ee1114840" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:03:51.137593 11102 raft_consensus.cc:3060] T 72b111ad7b5f4753ab1d2b5ee1114840 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:03:51.141335 10880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a610d1a71f52432e88fe56130f921fa2" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
W20260709 13:03:51.143987 10826 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 13:03:51.145192 11148 tablet_bootstrap.cc:492] T 2567c2d84e4648d7ba5dc23a3df32817 P 10ca5ee68ad64b3283b9d235352c9d23: No bootstrap required, opened a new log
I20260709 13:03:51.145792 11148 ts_tablet_manager.cc:1403] T 2567c2d84e4648d7ba5dc23a3df32817 P 10ca5ee68ad64b3283b9d235352c9d23: Time spent bootstrapping tablet: real 0.435s	user 0.021s	sys 0.019s
I20260709 13:03:51.149554 11148 raft_consensus.cc:359] T 2567c2d84e4648d7ba5dc23a3df32817 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:51.113742 11311 raft_consensus.cc:515] T a610d1a71f52432e88fe56130f921fa2 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:51.150336 11148 raft_consensus.cc:385] T 2567c2d84e4648d7ba5dc23a3df32817 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:51.150766 11148 raft_consensus.cc:740] T 2567c2d84e4648d7ba5dc23a3df32817 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10ca5ee68ad64b3283b9d235352c9d23, State: Initialized, Role: FOLLOWER
I20260709 13:03:51.150982 11101 raft_consensus.cc:2393] T a610d1a71f52432e88fe56130f921fa2 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 873357742d5843579556ee8166f23f47 in current term 1: Already voted for candidate 0172fed22b6b4890a7fdc953fa128af4 in this term.
I20260709 13:03:51.150862 11357 consensus_queue.cc:1048] T e89e640169e14e2d98f968f76027daa7 P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:03:51.153571 11152 tablet_bootstrap.cc:654] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 0172fed22b6b4890a7fdc953fa128af4: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:51.146148 11332 raft_consensus.cc:515] T 868e92339c2f4508af0afbd07c814d64 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:51.150279 10877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "868e92339c2f4508af0afbd07c814d64" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
W20260709 13:03:51.155867 10995 leader_election.cc:343] T a610d1a71f52432e88fe56130f921fa2 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 186b12638a87410487467df721a3cbda (127.0.107.193:38967): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:03:51.169414 10879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a610d1a71f52432e88fe56130f921fa2" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:03:51.171936 11029 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a610d1a71f52432e88fe56130f921fa2" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
W20260709 13:03:51.172281 10920 leader_election.cc:343] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:03:51.172868 11029 raft_consensus.cc:2393] T a610d1a71f52432e88fe56130f921fa2 P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 0172fed22b6b4890a7fdc953fa128af4 in current term 1: Already voted for candidate 873357742d5843579556ee8166f23f47 in this term.
I20260709 13:03:51.173636 10955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d270b9e5481245d9b5340a454a7cd10b" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:03:51.174724 10955 raft_consensus.cc:2468] T d270b9e5481245d9b5340a454a7cd10b P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 0.
I20260709 13:03:51.177448 10994 leader_election.cc:304] T a610d1a71f52432e88fe56130f921fa2 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 873357742d5843579556ee8166f23f47; no voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda
I20260709 13:03:51.178339 11257 raft_consensus.cc:2749] T a610d1a71f52432e88fe56130f921fa2 P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
W20260709 13:03:51.180685 11069 leader_election.cc:343] T a610d1a71f52432e88fe56130f921fa2 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 186b12638a87410487467df721a3cbda (127.0.107.193:38967): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:03:51.181318 11069 leader_election.cc:304] T a610d1a71f52432e88fe56130f921fa2 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4; no voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47
I20260709 13:03:51.185851 11332 leader_election.cc:290] T 868e92339c2f4508af0afbd07c814d64 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 1 election: Requested vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:03:51.182188 11354 raft_consensus.cc:2749] T a610d1a71f52432e88fe56130f921fa2 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 13:03:51.188416 11301 raft_consensus.cc:515] T 868e92339c2f4508af0afbd07c814d64 P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:51.190490 10877 raft_consensus.cc:2393] T 868e92339c2f4508af0afbd07c814d64 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 873357742d5843579556ee8166f23f47 in current term 1: Already voted for candidate 186b12638a87410487467df721a3cbda in this term.
I20260709 13:03:51.194726 11066 leader_election.cc:304] T d270b9e5481245d9b5340a454a7cd10b P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 10ca5ee68ad64b3283b9d235352c9d23; no voters: 186b12638a87410487467df721a3cbda
I20260709 13:03:51.195010 11357 consensus_queue.cc:1048] T e89e640169e14e2d98f968f76027daa7 P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:03:51.196985 11162 raft_consensus.cc:2804] T d270b9e5481245d9b5340a454a7cd10b P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:03:51.197484 11162 raft_consensus.cc:493] T d270b9e5481245d9b5340a454a7cd10b P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:51.197747 11029 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "868e92339c2f4508af0afbd07c814d64" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
I20260709 13:03:51.197896 11162 raft_consensus.cc:3060] T d270b9e5481245d9b5340a454a7cd10b P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:03:51.198686 11029 raft_consensus.cc:2393] T 868e92339c2f4508af0afbd07c814d64 P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 186b12638a87410487467df721a3cbda in current term 1: Already voted for candidate 873357742d5843579556ee8166f23f47 in this term.
I20260709 13:03:51.161659 11103 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f1466124dbd4f6ebb3c4d62ca48bf96" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:03:51.199482 11301 leader_election.cc:290] T 868e92339c2f4508af0afbd07c814d64 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 1 election: Requested vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:03:51.200105 11301 raft_consensus.cc:493] T 65bf0a62f4d84ef18dd6b23e082724b8 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:51.200578 11301 raft_consensus.cc:515] T 65bf0a62f4d84ef18dd6b23e082724b8 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:51.156733 11148 consensus_queue.cc:260] T 2567c2d84e4648d7ba5dc23a3df32817 P 10ca5ee68ad64b3283b9d235352c9d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:51.201613 11149 tablet_bootstrap.cc:654] T bac7e5040f984ba2b7146bd36af7267e P 873357742d5843579556ee8166f23f47: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:51.198527 11102 raft_consensus.cc:2468] T 72b111ad7b5f4753ab1d2b5ee1114840 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 1.
I20260709 13:03:51.206287 11151 raft_consensus.cc:359] T 2567c2d84e4648d7ba5dc23a3df32817 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:51.214690 11151 raft_consensus.cc:385] T 2567c2d84e4648d7ba5dc23a3df32817 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:51.215106 11151 raft_consensus.cc:740] T 2567c2d84e4648d7ba5dc23a3df32817 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 186b12638a87410487467df721a3cbda, State: Initialized, Role: FOLLOWER
I20260709 13:03:51.218098 11148 ts_tablet_manager.cc:1434] T 2567c2d84e4648d7ba5dc23a3df32817 P 10ca5ee68ad64b3283b9d235352c9d23: Time spent starting tablet: real 0.072s	user 0.005s	sys 0.005s
I20260709 13:03:51.217865 11151 consensus_queue.cc:260] T 2567c2d84e4648d7ba5dc23a3df32817 P 186b12638a87410487467df721a3cbda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:51.219123 11148 tablet_bootstrap.cc:492] T 5d439326f23747eaab3c1e7da2d6b785 P 10ca5ee68ad64b3283b9d235352c9d23: Bootstrap starting.
I20260709 13:03:51.167285 11311 leader_election.cc:290] T a610d1a71f52432e88fe56130f921fa2 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 1 election: Requested vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:03:51.211400 10880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65bf0a62f4d84ef18dd6b23e082724b8" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
W20260709 13:03:51.211910 10920 leader_election.cc:343] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:03:51.209499 10877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "868e92339c2f4508af0afbd07c814d64" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:03:51.216014 11101 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "868e92339c2f4508af0afbd07c814d64" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:03:51.235845 11151 ts_tablet_manager.cc:1434] T 2567c2d84e4648d7ba5dc23a3df32817 P 186b12638a87410487467df721a3cbda: Time spent starting tablet: real 0.102s	user 0.009s	sys 0.000s
I20260709 13:03:51.237716 11151 tablet_bootstrap.cc:492] T 72b111ad7b5f4753ab1d2b5ee1114840 P 186b12638a87410487467df721a3cbda: Bootstrap starting.
I20260709 13:03:51.243271 10953 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65bf0a62f4d84ef18dd6b23e082724b8" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:03:51.246400 10920 leader_election.cc:304] T 72b111ad7b5f4753ab1d2b5ee1114840 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 10ca5ee68ad64b3283b9d235352c9d23; no voters: 186b12638a87410487467df721a3cbda
I20260709 13:03:51.247488 11363 raft_consensus.cc:2804] T 72b111ad7b5f4753ab1d2b5ee1114840 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:03:51.248015 11363 raft_consensus.cc:697] T 72b111ad7b5f4753ab1d2b5ee1114840 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 LEADER]: Becoming Leader. State: Replica: 10ca5ee68ad64b3283b9d235352c9d23, State: Running, Role: LEADER
I20260709 13:03:51.237155 11292 raft_consensus.cc:493] T 2567c2d84e4648d7ba5dc23a3df32817 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:51.248874 11363 consensus_queue.cc:237] T 72b111ad7b5f4753ab1d2b5ee1114840 P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:51.249142 11292 raft_consensus.cc:515] T 2567c2d84e4648d7ba5dc23a3df32817 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:51.252111 11292 leader_election.cc:290] T 2567c2d84e4648d7ba5dc23a3df32817 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:03:51.240736 11101 raft_consensus.cc:3060] T 868e92339c2f4508af0afbd07c814d64 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:03:51.225513 11301 leader_election.cc:290] T 65bf0a62f4d84ef18dd6b23e082724b8 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:03:51.264161 10877 raft_consensus.cc:2393] T 868e92339c2f4508af0afbd07c814d64 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 873357742d5843579556ee8166f23f47 in current term 1: Already voted for candidate 186b12638a87410487467df721a3cbda in this term.
W20260709 13:03:51.270910 10995 leader_election.cc:343] T 65bf0a62f4d84ef18dd6b23e082724b8 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 186b12638a87410487467df721a3cbda (127.0.107.193:38967): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:03:51.277015 11152 tablet_bootstrap.cc:492] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 0172fed22b6b4890a7fdc953fa128af4: No bootstrap required, opened a new log
I20260709 13:03:51.277629 11152 ts_tablet_manager.cc:1403] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 0172fed22b6b4890a7fdc953fa128af4: Time spent bootstrapping tablet: real 0.192s	user 0.018s	sys 0.017s
W20260709 13:03:51.277916 10992 leader_election.cc:343] T 65bf0a62f4d84ef18dd6b23e082724b8 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:03:51.280155 10992 leader_election.cc:304] T 65bf0a62f4d84ef18dd6b23e082724b8 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 873357742d5843579556ee8166f23f47; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda
I20260709 13:03:51.281428 11305 raft_consensus.cc:2749] T 65bf0a62f4d84ef18dd6b23e082724b8 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:03:51.273350 11029 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2567c2d84e4648d7ba5dc23a3df32817" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:03:51.287212 11029 raft_consensus.cc:2468] T 2567c2d84e4648d7ba5dc23a3df32817 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 0.
I20260709 13:03:51.288595 11148 tablet_bootstrap.cc:654] T 5d439326f23747eaab3c1e7da2d6b785 P 10ca5ee68ad64b3283b9d235352c9d23: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:51.289319 10844 leader_election.cc:304] T 2567c2d84e4648d7ba5dc23a3df32817 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:03:51.290337 11332 raft_consensus.cc:2804] T 2567c2d84e4648d7ba5dc23a3df32817 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:03:51.290791 11332 raft_consensus.cc:493] T 2567c2d84e4648d7ba5dc23a3df32817 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:51.291164 11332 raft_consensus.cc:3060] T 2567c2d84e4648d7ba5dc23a3df32817 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:03:51.291503 11152 raft_consensus.cc:359] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:51.292236 11152 raft_consensus.cc:385] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:51.290611 11151 tablet_bootstrap.cc:654] T 72b111ad7b5f4753ab1d2b5ee1114840 P 186b12638a87410487467df721a3cbda: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:51.262499 11102 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "868e92339c2f4508af0afbd07c814d64" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:03:51.292647 11152 raft_consensus.cc:740] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0172fed22b6b4890a7fdc953fa128af4, State: Initialized, Role: FOLLOWER
I20260709 13:03:51.295012 10845 leader_election.cc:304] T 868e92339c2f4508af0afbd07c814d64 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 186b12638a87410487467df721a3cbda; no voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47
I20260709 13:03:51.296911 10954 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2567c2d84e4648d7ba5dc23a3df32817" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:03:51.297645 10954 raft_consensus.cc:2468] T 2567c2d84e4648d7ba5dc23a3df32817 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 0.
I20260709 13:03:51.297528 11152 consensus_queue.cc:260] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 0172fed22b6b4890a7fdc953fa128af4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:51.292618 11162 raft_consensus.cc:515] T d270b9e5481245d9b5340a454a7cd10b P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:51.300699 11162 leader_election.cc:290] T d270b9e5481245d9b5340a454a7cd10b P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 1 election: Requested vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:03:51.296927 11333 raft_consensus.cc:2749] T 868e92339c2f4508af0afbd07c814d64 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 13:03:51.305267 11152 ts_tablet_manager.cc:1434] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 0172fed22b6b4890a7fdc953fa128af4: Time spent starting tablet: real 0.027s	user 0.009s	sys 0.001s
I20260709 13:03:51.312768 11101 raft_consensus.cc:2468] T 868e92339c2f4508af0afbd07c814d64 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 1.
I20260709 13:03:51.314894 11152 tablet_bootstrap.cc:492] T bac7e5040f984ba2b7146bd36af7267e P 0172fed22b6b4890a7fdc953fa128af4: Bootstrap starting.
I20260709 13:03:51.320308 10953 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d270b9e5481245d9b5340a454a7cd10b" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:03:51.321022 10953 raft_consensus.cc:3060] T d270b9e5481245d9b5340a454a7cd10b P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:03:51.323307 10878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d270b9e5481245d9b5340a454a7cd10b" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
W20260709 13:03:51.324745 11069 leader_election.cc:343] T d270b9e5481245d9b5340a454a7cd10b P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 186b12638a87410487467df721a3cbda (127.0.107.193:38967): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:03:51.330855 11152 tablet_bootstrap.cc:654] T bac7e5040f984ba2b7146bd36af7267e P 0172fed22b6b4890a7fdc953fa128af4: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:51.334997 11151 tablet_bootstrap.cc:492] T 72b111ad7b5f4753ab1d2b5ee1114840 P 186b12638a87410487467df721a3cbda: No bootstrap required, opened a new log
I20260709 13:03:51.335640 11151 ts_tablet_manager.cc:1403] T 72b111ad7b5f4753ab1d2b5ee1114840 P 186b12638a87410487467df721a3cbda: Time spent bootstrapping tablet: real 0.098s	user 0.012s	sys 0.000s
I20260709 13:03:51.338083 10994 leader_election.cc:304] T 868e92339c2f4508af0afbd07c814d64 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47; no voters: 186b12638a87410487467df721a3cbda
I20260709 13:03:51.339176 11301 raft_consensus.cc:2804] T 868e92339c2f4508af0afbd07c814d64 P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:03:51.339792 11301 raft_consensus.cc:697] T 868e92339c2f4508af0afbd07c814d64 P 873357742d5843579556ee8166f23f47 [term 1 LEADER]: Becoming Leader. State: Replica: 873357742d5843579556ee8166f23f47, State: Running, Role: LEADER
I20260709 13:03:51.340628 11151 raft_consensus.cc:359] T 72b111ad7b5f4753ab1d2b5ee1114840 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:51.341390 11151 raft_consensus.cc:385] T 72b111ad7b5f4753ab1d2b5ee1114840 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:51.341663 11151 raft_consensus.cc:740] T 72b111ad7b5f4753ab1d2b5ee1114840 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 186b12638a87410487467df721a3cbda, State: Initialized, Role: FOLLOWER
I20260709 13:03:51.342419 11151 consensus_queue.cc:260] T 72b111ad7b5f4753ab1d2b5ee1114840 P 186b12638a87410487467df721a3cbda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:51.344014 11332 raft_consensus.cc:515] T 2567c2d84e4648d7ba5dc23a3df32817 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:51.344944 11301 consensus_queue.cc:237] T 868e92339c2f4508af0afbd07c814d64 P 873357742d5843579556ee8166f23f47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:51.347357 11149 tablet_bootstrap.cc:492] T bac7e5040f984ba2b7146bd36af7267e P 873357742d5843579556ee8166f23f47: No bootstrap required, opened a new log
I20260709 13:03:51.347925 11149 ts_tablet_manager.cc:1403] T bac7e5040f984ba2b7146bd36af7267e P 873357742d5843579556ee8166f23f47: Time spent bootstrapping tablet: real 0.231s	user 0.011s	sys 0.015s
I20260709 13:03:51.348668 10776 catalog_manager.cc:5697] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 10ca5ee68ad64b3283b9d235352c9d23 reported cstate change: term changed from 0 to 1, leader changed from <none> to 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194). New cstate: current_term: 1 leader_uuid: "10ca5ee68ad64b3283b9d235352c9d23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } health_report { overall_health: UNKNOWN } } }
I20260709 13:03:51.358314 11149 raft_consensus.cc:359] T bac7e5040f984ba2b7146bd36af7267e P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:51.359157 11149 raft_consensus.cc:385] T bac7e5040f984ba2b7146bd36af7267e P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:51.359439 11149 raft_consensus.cc:740] T bac7e5040f984ba2b7146bd36af7267e P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 873357742d5843579556ee8166f23f47, State: Initialized, Role: FOLLOWER
I20260709 13:03:51.359797 11151 ts_tablet_manager.cc:1434] T 72b111ad7b5f4753ab1d2b5ee1114840 P 186b12638a87410487467df721a3cbda: Time spent starting tablet: real 0.024s	user 0.008s	sys 0.000s
I20260709 13:03:51.360167 11149 consensus_queue.cc:260] T bac7e5040f984ba2b7146bd36af7267e P 873357742d5843579556ee8166f23f47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:51.360929 11151 tablet_bootstrap.cc:492] T a610d1a71f52432e88fe56130f921fa2 P 186b12638a87410487467df721a3cbda: Bootstrap starting.
I20260709 13:03:51.362485 11149 ts_tablet_manager.cc:1434] T bac7e5040f984ba2b7146bd36af7267e P 873357742d5843579556ee8166f23f47: Time spent starting tablet: real 0.014s	user 0.009s	sys 0.000s
I20260709 13:03:51.363535 11149 tablet_bootstrap.cc:492] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 873357742d5843579556ee8166f23f47: Bootstrap starting.
I20260709 13:03:51.368808 11151 tablet_bootstrap.cc:654] T a610d1a71f52432e88fe56130f921fa2 P 186b12638a87410487467df721a3cbda: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:51.370632 11149 tablet_bootstrap.cc:654] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 873357742d5843579556ee8166f23f47: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:51.375288 10774 catalog_manager.cc:5697] T 868e92339c2f4508af0afbd07c814d64 P 873357742d5843579556ee8166f23f47 reported cstate change: term changed from 0 to 1, leader changed from <none> to 873357742d5843579556ee8166f23f47 (127.0.107.195). New cstate: current_term: 1 leader_uuid: "873357742d5843579556ee8166f23f47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } health_report { overall_health: UNKNOWN } } }
I20260709 13:03:51.385332 11332 leader_election.cc:290] T 2567c2d84e4648d7ba5dc23a3df32817 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 1 election: Requested vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:03:51.387279 11027 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2567c2d84e4648d7ba5dc23a3df32817" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
I20260709 13:03:51.387545 10954 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2567c2d84e4648d7ba5dc23a3df32817" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:03:51.388015 11027 raft_consensus.cc:3060] T 2567c2d84e4648d7ba5dc23a3df32817 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:03:51.388203 10954 raft_consensus.cc:3060] T 2567c2d84e4648d7ba5dc23a3df32817 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:03:51.390022 11149 tablet_bootstrap.cc:492] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 873357742d5843579556ee8166f23f47: No bootstrap required, opened a new log
I20260709 13:03:51.390590 11149 ts_tablet_manager.cc:1403] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 873357742d5843579556ee8166f23f47: Time spent bootstrapping tablet: real 0.027s	user 0.008s	sys 0.011s
I20260709 13:03:51.392017 11151 tablet_bootstrap.cc:492] T a610d1a71f52432e88fe56130f921fa2 P 186b12638a87410487467df721a3cbda: No bootstrap required, opened a new log
I20260709 13:03:51.392652 11151 ts_tablet_manager.cc:1403] T a610d1a71f52432e88fe56130f921fa2 P 186b12638a87410487467df721a3cbda: Time spent bootstrapping tablet: real 0.032s	user 0.017s	sys 0.001s
I20260709 13:03:51.393896 11149 raft_consensus.cc:359] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:51.394614 11149 raft_consensus.cc:385] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:51.394869 11149 raft_consensus.cc:740] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 873357742d5843579556ee8166f23f47, State: Initialized, Role: FOLLOWER
I20260709 13:03:51.395542 11149 consensus_queue.cc:260] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 873357742d5843579556ee8166f23f47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:51.403158 10953 raft_consensus.cc:2468] T d270b9e5481245d9b5340a454a7cd10b P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 1.
I20260709 13:03:51.390614 11354 raft_consensus.cc:493] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:51.404111 11354 raft_consensus.cc:515] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:51.405026 11066 leader_election.cc:304] T d270b9e5481245d9b5340a454a7cd10b P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 10ca5ee68ad64b3283b9d235352c9d23; no voters: 186b12638a87410487467df721a3cbda
I20260709 13:03:51.407526 11149 ts_tablet_manager.cc:1434] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 873357742d5843579556ee8166f23f47: Time spent starting tablet: real 0.017s	user 0.006s	sys 0.001s
I20260709 13:03:51.409425 11149 tablet_bootstrap.cc:492] T b4cd3d7dfbbf4096908ff11e9dc57141 P 873357742d5843579556ee8166f23f47: Bootstrap starting.
I20260709 13:03:51.410910 10955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f1466124dbd4f6ebb3c4d62ca48bf96" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:03:51.406350 11354 leader_election.cc:290] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:03:51.407254 11162 raft_consensus.cc:2804] T d270b9e5481245d9b5340a454a7cd10b P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:03:51.412882 11148 tablet_bootstrap.cc:492] T 5d439326f23747eaab3c1e7da2d6b785 P 10ca5ee68ad64b3283b9d235352c9d23: No bootstrap required, opened a new log
I20260709 13:03:51.412984 11162 raft_consensus.cc:697] T d270b9e5481245d9b5340a454a7cd10b P 0172fed22b6b4890a7fdc953fa128af4 [term 1 LEADER]: Becoming Leader. State: Replica: 0172fed22b6b4890a7fdc953fa128af4, State: Running, Role: LEADER
I20260709 13:03:51.413534 11148 ts_tablet_manager.cc:1403] T 5d439326f23747eaab3c1e7da2d6b785 P 10ca5ee68ad64b3283b9d235352c9d23: Time spent bootstrapping tablet: real 0.195s	user 0.022s	sys 0.006s
I20260709 13:03:51.414072 11162 consensus_queue.cc:237] T d270b9e5481245d9b5340a454a7cd10b P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:51.417905 11148 raft_consensus.cc:359] T 5d439326f23747eaab3c1e7da2d6b785 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:51.418947 11148 raft_consensus.cc:385] T 5d439326f23747eaab3c1e7da2d6b785 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:51.419358 11148 raft_consensus.cc:740] T 5d439326f23747eaab3c1e7da2d6b785 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10ca5ee68ad64b3283b9d235352c9d23, State: Initialized, Role: FOLLOWER
I20260709 13:03:51.420310 11148 consensus_queue.cc:260] T 5d439326f23747eaab3c1e7da2d6b785 P 10ca5ee68ad64b3283b9d235352c9d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:51.420536 11151 raft_consensus.cc:359] T a610d1a71f52432e88fe56130f921fa2 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:51.422004 11151 raft_consensus.cc:385] T a610d1a71f52432e88fe56130f921fa2 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:51.423833 11151 raft_consensus.cc:740] T a610d1a71f52432e88fe56130f921fa2 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 186b12638a87410487467df721a3cbda, State: Initialized, Role: FOLLOWER
I20260709 13:03:51.406785 11027 raft_consensus.cc:2468] T 2567c2d84e4648d7ba5dc23a3df32817 P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 1.
I20260709 13:03:51.425093 11148 ts_tablet_manager.cc:1434] T 5d439326f23747eaab3c1e7da2d6b785 P 10ca5ee68ad64b3283b9d235352c9d23: Time spent starting tablet: real 0.011s	user 0.004s	sys 0.004s
I20260709 13:03:51.426244 11151 consensus_queue.cc:260] T a610d1a71f52432e88fe56130f921fa2 P 186b12638a87410487467df721a3cbda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:51.429152 11148 tablet_bootstrap.cc:492] T 65bf0a62f4d84ef18dd6b23e082724b8 P 10ca5ee68ad64b3283b9d235352c9d23: Bootstrap starting.
I20260709 13:03:51.425467 11028 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f1466124dbd4f6ebb3c4d62ca48bf96" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:03:51.430016 11028 raft_consensus.cc:2393] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0172fed22b6b4890a7fdc953fa128af4 in current term 1: Already voted for candidate 10ca5ee68ad64b3283b9d235352c9d23 in this term.
I20260709 13:03:51.430853 10844 leader_election.cc:304] T 2567c2d84e4648d7ba5dc23a3df32817 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:03:51.432793 11067 leader_election.cc:304] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47
I20260709 13:03:51.433288 11332 raft_consensus.cc:2804] T 2567c2d84e4648d7ba5dc23a3df32817 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:03:51.433830 11332 raft_consensus.cc:697] T 2567c2d84e4648d7ba5dc23a3df32817 P 186b12638a87410487467df721a3cbda [term 1 LEADER]: Becoming Leader. State: Replica: 186b12638a87410487467df721a3cbda, State: Running, Role: LEADER
I20260709 13:03:51.434763 11332 consensus_queue.cc:237] T 2567c2d84e4648d7ba5dc23a3df32817 P 186b12638a87410487467df721a3cbda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:51.435352 11354 raft_consensus.cc:3060] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:03:51.437685 11148 tablet_bootstrap.cc:654] T 65bf0a62f4d84ef18dd6b23e082724b8 P 10ca5ee68ad64b3283b9d235352c9d23: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:51.443709 11149 tablet_bootstrap.cc:654] T b4cd3d7dfbbf4096908ff11e9dc57141 P 873357742d5843579556ee8166f23f47: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:51.444916 10954 raft_consensus.cc:2468] T 2567c2d84e4648d7ba5dc23a3df32817 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 1.
I20260709 13:03:51.447814 11151 ts_tablet_manager.cc:1434] T a610d1a71f52432e88fe56130f921fa2 P 186b12638a87410487467df721a3cbda: Time spent starting tablet: real 0.055s	user 0.009s	sys 0.000s
I20260709 13:03:51.449016 11151 tablet_bootstrap.cc:492] T 5d439326f23747eaab3c1e7da2d6b785 P 186b12638a87410487467df721a3cbda: Bootstrap starting.
I20260709 13:03:51.450207 11354 raft_consensus.cc:2749] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:03:51.452719 11152 tablet_bootstrap.cc:492] T bac7e5040f984ba2b7146bd36af7267e P 0172fed22b6b4890a7fdc953fa128af4: No bootstrap required, opened a new log
I20260709 13:03:51.453687 11152 ts_tablet_manager.cc:1403] T bac7e5040f984ba2b7146bd36af7267e P 0172fed22b6b4890a7fdc953fa128af4: Time spent bootstrapping tablet: real 0.139s	user 0.019s	sys 0.008s
I20260709 13:03:51.463572 11152 raft_consensus.cc:359] T bac7e5040f984ba2b7146bd36af7267e P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:51.464476 11152 raft_consensus.cc:385] T bac7e5040f984ba2b7146bd36af7267e P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:51.464793 11152 raft_consensus.cc:740] T bac7e5040f984ba2b7146bd36af7267e P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0172fed22b6b4890a7fdc953fa128af4, State: Initialized, Role: FOLLOWER
I20260709 13:03:51.465662 11152 consensus_queue.cc:260] T bac7e5040f984ba2b7146bd36af7267e P 0172fed22b6b4890a7fdc953fa128af4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:51.468405 11363 raft_consensus.cc:493] T 5d439326f23747eaab3c1e7da2d6b785 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:51.469102 11363 raft_consensus.cc:515] T 5d439326f23747eaab3c1e7da2d6b785 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:51.472038 11363 leader_election.cc:290] T 5d439326f23747eaab3c1e7da2d6b785 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:03:51.473361 10878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d439326f23747eaab3c1e7da2d6b785" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:03:51.474687 11152 ts_tablet_manager.cc:1434] T bac7e5040f984ba2b7146bd36af7267e P 0172fed22b6b4890a7fdc953fa128af4: Time spent starting tablet: real 0.021s	user 0.009s	sys 0.000s
I20260709 13:03:51.475199 11101 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d439326f23747eaab3c1e7da2d6b785" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:03:51.476022 11101 raft_consensus.cc:2468] T 5d439326f23747eaab3c1e7da2d6b785 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 0.
W20260709 13:03:51.475343 10921 leader_election.cc:343] T 5d439326f23747eaab3c1e7da2d6b785 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 186b12638a87410487467df721a3cbda (127.0.107.193:38967): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:03:51.478137 11152 tablet_bootstrap.cc:492] T b4cd3d7dfbbf4096908ff11e9dc57141 P 0172fed22b6b4890a7fdc953fa128af4: Bootstrap starting.
I20260709 13:03:51.478309 10920 leader_election.cc:304] T 5d439326f23747eaab3c1e7da2d6b785 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 10ca5ee68ad64b3283b9d235352c9d23; no voters: 186b12638a87410487467df721a3cbda
I20260709 13:03:51.488323 11363 raft_consensus.cc:2804] T 5d439326f23747eaab3c1e7da2d6b785 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:03:51.488770 11363 raft_consensus.cc:493] T 5d439326f23747eaab3c1e7da2d6b785 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:51.491329 11151 tablet_bootstrap.cc:654] T 5d439326f23747eaab3c1e7da2d6b785 P 186b12638a87410487467df721a3cbda: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:51.489156 11363 raft_consensus.cc:3060] T 5d439326f23747eaab3c1e7da2d6b785 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:03:51.495363 11148 tablet_bootstrap.cc:492] T 65bf0a62f4d84ef18dd6b23e082724b8 P 10ca5ee68ad64b3283b9d235352c9d23: No bootstrap required, opened a new log
I20260709 13:03:51.494863 10775 catalog_manager.cc:5697] T d270b9e5481245d9b5340a454a7cd10b P 0172fed22b6b4890a7fdc953fa128af4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196). New cstate: current_term: 1 leader_uuid: "0172fed22b6b4890a7fdc953fa128af4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } health_report { overall_health: UNKNOWN } } }
I20260709 13:03:51.496059 11148 ts_tablet_manager.cc:1403] T 65bf0a62f4d84ef18dd6b23e082724b8 P 10ca5ee68ad64b3283b9d235352c9d23: Time spent bootstrapping tablet: real 0.069s	user 0.010s	sys 0.012s
I20260709 13:03:51.500519 11149 tablet_bootstrap.cc:492] T b4cd3d7dfbbf4096908ff11e9dc57141 P 873357742d5843579556ee8166f23f47: No bootstrap required, opened a new log
I20260709 13:03:51.501693 11148 raft_consensus.cc:359] T 65bf0a62f4d84ef18dd6b23e082724b8 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:51.502569 11148 raft_consensus.cc:385] T 65bf0a62f4d84ef18dd6b23e082724b8 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:51.502918 11148 raft_consensus.cc:740] T 65bf0a62f4d84ef18dd6b23e082724b8 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10ca5ee68ad64b3283b9d235352c9d23, State: Initialized, Role: FOLLOWER
I20260709 13:03:51.503561 11149 ts_tablet_manager.cc:1403] T b4cd3d7dfbbf4096908ff11e9dc57141 P 873357742d5843579556ee8166f23f47: Time spent bootstrapping tablet: real 0.094s	user 0.022s	sys 0.019s
I20260709 13:03:51.503935 11148 consensus_queue.cc:260] T 65bf0a62f4d84ef18dd6b23e082724b8 P 10ca5ee68ad64b3283b9d235352c9d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:51.506541 11148 ts_tablet_manager.cc:1434] T 65bf0a62f4d84ef18dd6b23e082724b8 P 10ca5ee68ad64b3283b9d235352c9d23: Time spent starting tablet: real 0.010s	user 0.007s	sys 0.000s
I20260709 13:03:51.507031 11149 raft_consensus.cc:359] T b4cd3d7dfbbf4096908ff11e9dc57141 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:51.507735 11148 tablet_bootstrap.cc:492] T bac7e5040f984ba2b7146bd36af7267e P 10ca5ee68ad64b3283b9d235352c9d23: Bootstrap starting.
I20260709 13:03:51.507712 11149 raft_consensus.cc:385] T b4cd3d7dfbbf4096908ff11e9dc57141 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:51.509487 11149 raft_consensus.cc:740] T b4cd3d7dfbbf4096908ff11e9dc57141 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 873357742d5843579556ee8166f23f47, State: Initialized, Role: FOLLOWER
I20260709 13:03:51.510232 11149 consensus_queue.cc:260] T b4cd3d7dfbbf4096908ff11e9dc57141 P 873357742d5843579556ee8166f23f47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:51.512923 11149 ts_tablet_manager.cc:1434] T b4cd3d7dfbbf4096908ff11e9dc57141 P 873357742d5843579556ee8166f23f47: Time spent starting tablet: real 0.009s	user 0.008s	sys 0.000s
I20260709 13:03:51.514338 11149 tablet_bootstrap.cc:492] T 63d44f6ba6434ab2870fe3190f4d7c2f P 873357742d5843579556ee8166f23f47: Bootstrap starting.
I20260709 13:03:51.518523 11354 raft_consensus.cc:493] T bac7e5040f984ba2b7146bd36af7267e P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:51.519181 11354 raft_consensus.cc:515] T bac7e5040f984ba2b7146bd36af7267e P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:51.519227 11363 raft_consensus.cc:515] T 5d439326f23747eaab3c1e7da2d6b785 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:51.521196 11148 tablet_bootstrap.cc:654] T bac7e5040f984ba2b7146bd36af7267e P 10ca5ee68ad64b3283b9d235352c9d23: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:51.522020 11149 tablet_bootstrap.cc:654] T 63d44f6ba6434ab2870fe3190f4d7c2f P 873357742d5843579556ee8166f23f47: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:51.522732 11101 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d439326f23747eaab3c1e7da2d6b785" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:03:51.523453 11101 raft_consensus.cc:3060] T 5d439326f23747eaab3c1e7da2d6b785 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:03:51.527428 11301 raft_consensus.cc:493] T bac7e5040f984ba2b7146bd36af7267e P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:51.528163 11301 raft_consensus.cc:515] T bac7e5040f984ba2b7146bd36af7267e P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:51.534152 11148 tablet_bootstrap.cc:492] T bac7e5040f984ba2b7146bd36af7267e P 10ca5ee68ad64b3283b9d235352c9d23: No bootstrap required, opened a new log
I20260709 13:03:51.534296 11301 leader_election.cc:290] T bac7e5040f984ba2b7146bd36af7267e P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:03:51.534281 11101 raft_consensus.cc:2468] T 5d439326f23747eaab3c1e7da2d6b785 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 1.
I20260709 13:03:51.535782 11148 ts_tablet_manager.cc:1403] T bac7e5040f984ba2b7146bd36af7267e P 10ca5ee68ad64b3283b9d235352c9d23: Time spent bootstrapping tablet: real 0.028s	user 0.013s	sys 0.004s
I20260709 13:03:51.536657 10920 leader_election.cc:304] T 5d439326f23747eaab3c1e7da2d6b785 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 10ca5ee68ad64b3283b9d235352c9d23; no voters: 
I20260709 13:03:51.538030 11103 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bac7e5040f984ba2b7146bd36af7267e" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:03:51.539074 11103 raft_consensus.cc:2468] T bac7e5040f984ba2b7146bd36af7267e P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 0.
I20260709 13:03:51.539430 11148 raft_consensus.cc:359] T bac7e5040f984ba2b7146bd36af7267e P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:51.540215 11148 raft_consensus.cc:385] T bac7e5040f984ba2b7146bd36af7267e P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:51.540504 11151 tablet_bootstrap.cc:492] T 5d439326f23747eaab3c1e7da2d6b785 P 186b12638a87410487467df721a3cbda: No bootstrap required, opened a new log
I20260709 13:03:51.540717 11148 raft_consensus.cc:740] T bac7e5040f984ba2b7146bd36af7267e P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10ca5ee68ad64b3283b9d235352c9d23, State: Initialized, Role: FOLLOWER
I20260709 13:03:51.541074 11151 ts_tablet_manager.cc:1403] T 5d439326f23747eaab3c1e7da2d6b785 P 186b12638a87410487467df721a3cbda: Time spent bootstrapping tablet: real 0.092s	user 0.008s	sys 0.005s
I20260709 13:03:51.541985 11148 consensus_queue.cc:260] T bac7e5040f984ba2b7146bd36af7267e P 10ca5ee68ad64b3283b9d235352c9d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:51.542402 10954 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bac7e5040f984ba2b7146bd36af7267e" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:03:51.543371 10954 raft_consensus.cc:2468] T bac7e5040f984ba2b7146bd36af7267e P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 0.
I20260709 13:03:51.544773 11151 raft_consensus.cc:359] T 5d439326f23747eaab3c1e7da2d6b785 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:51.545486 11151 raft_consensus.cc:385] T 5d439326f23747eaab3c1e7da2d6b785 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:51.545723 11151 raft_consensus.cc:740] T 5d439326f23747eaab3c1e7da2d6b785 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 186b12638a87410487467df721a3cbda, State: Initialized, Role: FOLLOWER
I20260709 13:03:51.546136 10994 leader_election.cc:304] T bac7e5040f984ba2b7146bd36af7267e P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:03:51.547580 11148 ts_tablet_manager.cc:1434] T bac7e5040f984ba2b7146bd36af7267e P 10ca5ee68ad64b3283b9d235352c9d23: Time spent starting tablet: real 0.011s	user 0.007s	sys 0.000s
I20260709 13:03:51.548206 11257 raft_consensus.cc:2804] T bac7e5040f984ba2b7146bd36af7267e P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:03:51.548385 11151 consensus_queue.cc:260] T 5d439326f23747eaab3c1e7da2d6b785 P 186b12638a87410487467df721a3cbda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:51.548830 11257 raft_consensus.cc:493] T bac7e5040f984ba2b7146bd36af7267e P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:51.549350 11257 raft_consensus.cc:3060] T bac7e5040f984ba2b7146bd36af7267e P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:03:51.551613 11148 tablet_bootstrap.cc:492] T 63d44f6ba6434ab2870fe3190f4d7c2f P 10ca5ee68ad64b3283b9d235352c9d23: Bootstrap starting.
I20260709 13:03:51.551688 11151 ts_tablet_manager.cc:1434] T 5d439326f23747eaab3c1e7da2d6b785 P 186b12638a87410487467df721a3cbda: Time spent starting tablet: real 0.010s	user 0.007s	sys 0.001s
I20260709 13:03:51.552811 11151 tablet_bootstrap.cc:492] T d270b9e5481245d9b5340a454a7cd10b P 186b12638a87410487467df721a3cbda: Bootstrap starting.
I20260709 13:03:51.538039 11341 raft_consensus.cc:2804] T 5d439326f23747eaab3c1e7da2d6b785 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:03:51.556317 11341 raft_consensus.cc:697] T 5d439326f23747eaab3c1e7da2d6b785 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 LEADER]: Becoming Leader. State: Replica: 10ca5ee68ad64b3283b9d235352c9d23, State: Running, Role: LEADER
I20260709 13:03:51.556769 10955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bac7e5040f984ba2b7146bd36af7267e" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:03:51.557526 10955 raft_consensus.cc:2468] T bac7e5040f984ba2b7146bd36af7267e P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 0.
I20260709 13:03:51.557301 11341 consensus_queue.cc:237] T 5d439326f23747eaab3c1e7da2d6b785 P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:51.559083 11066 leader_election.cc:304] T bac7e5040f984ba2b7146bd36af7267e P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 10ca5ee68ad64b3283b9d235352c9d23; no voters: 
I20260709 13:03:51.559669 11151 tablet_bootstrap.cc:654] T d270b9e5481245d9b5340a454a7cd10b P 186b12638a87410487467df721a3cbda: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:51.560037 11354 leader_election.cc:290] T bac7e5040f984ba2b7146bd36af7267e P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:03:51.560631 11162 raft_consensus.cc:2804] T bac7e5040f984ba2b7146bd36af7267e P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:03:51.561147 11162 raft_consensus.cc:493] T bac7e5040f984ba2b7146bd36af7267e P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:51.561491 11162 raft_consensus.cc:3060] T bac7e5040f984ba2b7146bd36af7267e P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:03:51.562884 11029 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bac7e5040f984ba2b7146bd36af7267e" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:03:51.570822 11357 consensus_queue.cc:1048] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:03:51.574290 11148 tablet_bootstrap.cc:654] T 63d44f6ba6434ab2870fe3190f4d7c2f P 10ca5ee68ad64b3283b9d235352c9d23: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:51.574609 11162 raft_consensus.cc:515] T bac7e5040f984ba2b7146bd36af7267e P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:51.577109 11363 leader_election.cc:290] T 5d439326f23747eaab3c1e7da2d6b785 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 1 election: Requested vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:03:51.577739 10953 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bac7e5040f984ba2b7146bd36af7267e" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:03:51.578420 10953 raft_consensus.cc:3060] T bac7e5040f984ba2b7146bd36af7267e P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:03:51.586043 10878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d439326f23747eaab3c1e7da2d6b785" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:03:51.586851 10878 raft_consensus.cc:3060] T 5d439326f23747eaab3c1e7da2d6b785 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:03:51.587101 10953 raft_consensus.cc:2468] T bac7e5040f984ba2b7146bd36af7267e P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 1.
I20260709 13:03:51.593133 11301 raft_consensus.cc:493] T b4cd3d7dfbbf4096908ff11e9dc57141 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:51.595620 11301 raft_consensus.cc:515] T b4cd3d7dfbbf4096908ff11e9dc57141 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:51.598640 11066 leader_election.cc:304] T bac7e5040f984ba2b7146bd36af7267e P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 10ca5ee68ad64b3283b9d235352c9d23; no voters: 
I20260709 13:03:51.601917 11151 tablet_bootstrap.cc:492] T d270b9e5481245d9b5340a454a7cd10b P 186b12638a87410487467df721a3cbda: No bootstrap required, opened a new log
I20260709 13:03:51.602520 11151 ts_tablet_manager.cc:1403] T d270b9e5481245d9b5340a454a7cd10b P 186b12638a87410487467df721a3cbda: Time spent bootstrapping tablet: real 0.050s	user 0.018s	sys 0.001s
I20260709 13:03:51.602186 11162 leader_election.cc:290] T bac7e5040f984ba2b7146bd36af7267e P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 1 election: Requested vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:03:51.603772 11162 raft_consensus.cc:2804] T bac7e5040f984ba2b7146bd36af7267e P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:03:51.603816 11301 leader_election.cc:290] T b4cd3d7dfbbf4096908ff11e9dc57141 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:03:51.604377 11162 raft_consensus.cc:697] T bac7e5040f984ba2b7146bd36af7267e P 0172fed22b6b4890a7fdc953fa128af4 [term 1 LEADER]: Becoming Leader. State: Replica: 0172fed22b6b4890a7fdc953fa128af4, State: Running, Role: LEADER
I20260709 13:03:51.605312 11162 consensus_queue.cc:237] T bac7e5040f984ba2b7146bd36af7267e P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:51.607806 11151 raft_consensus.cc:359] T d270b9e5481245d9b5340a454a7cd10b P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:51.608565 11151 raft_consensus.cc:385] T d270b9e5481245d9b5340a454a7cd10b P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:51.608847 11151 raft_consensus.cc:740] T d270b9e5481245d9b5340a454a7cd10b P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 186b12638a87410487467df721a3cbda, State: Initialized, Role: FOLLOWER
I20260709 13:03:51.609531 11151 consensus_queue.cc:260] T d270b9e5481245d9b5340a454a7cd10b P 186b12638a87410487467df721a3cbda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:51.611203 11151 ts_tablet_manager.cc:1434] T d270b9e5481245d9b5340a454a7cd10b P 186b12638a87410487467df721a3cbda: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.000s
I20260709 13:03:51.612213 11151 tablet_bootstrap.cc:492] T 65bf0a62f4d84ef18dd6b23e082724b8 P 186b12638a87410487467df721a3cbda: Bootstrap starting.
I20260709 13:03:51.615958 11027 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bac7e5040f984ba2b7146bd36af7267e" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
I20260709 13:03:51.601327 10774 catalog_manager.cc:5697] T 2567c2d84e4648d7ba5dc23a3df32817 P 186b12638a87410487467df721a3cbda reported cstate change: term changed from 0 to 1, leader changed from <none> to 186b12638a87410487467df721a3cbda (127.0.107.193). New cstate: current_term: 1 leader_uuid: "186b12638a87410487467df721a3cbda" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } health_report { overall_health: UNKNOWN } } }
I20260709 13:03:51.625504 11103 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4cd3d7dfbbf4096908ff11e9dc57141" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:03:51.626942 11151 tablet_bootstrap.cc:654] T 65bf0a62f4d84ef18dd6b23e082724b8 P 186b12638a87410487467df721a3cbda: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:51.627414 11148 tablet_bootstrap.cc:492] T 63d44f6ba6434ab2870fe3190f4d7c2f P 10ca5ee68ad64b3283b9d235352c9d23: No bootstrap required, opened a new log
I20260709 13:03:51.628072 11148 ts_tablet_manager.cc:1403] T 63d44f6ba6434ab2870fe3190f4d7c2f P 10ca5ee68ad64b3283b9d235352c9d23: Time spent bootstrapping tablet: real 0.077s	user 0.022s	sys 0.029s
I20260709 13:03:51.629777 10953 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4cd3d7dfbbf4096908ff11e9dc57141" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
W20260709 13:03:51.631683 10992 leader_election.cc:343] T b4cd3d7dfbbf4096908ff11e9dc57141 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:03:51.635205 11363 consensus_queue.cc:1048] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:03:51.632107 11148 raft_consensus.cc:359] T 63d44f6ba6434ab2870fe3190f4d7c2f P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:51.636487 11148 raft_consensus.cc:385] T 63d44f6ba6434ab2870fe3190f4d7c2f P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:51.636844 11148 raft_consensus.cc:740] T 63d44f6ba6434ab2870fe3190f4d7c2f P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10ca5ee68ad64b3283b9d235352c9d23, State: Initialized, Role: FOLLOWER
W20260709 13:03:51.639618 10994 leader_election.cc:343] T b4cd3d7dfbbf4096908ff11e9dc57141 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:03:51.640352 10994 leader_election.cc:304] T b4cd3d7dfbbf4096908ff11e9dc57141 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 873357742d5843579556ee8166f23f47; no voters: 0172fed22b6b4890a7fdc953fa128af4, 10ca5ee68ad64b3283b9d235352c9d23
I20260709 13:03:51.641455 11301 raft_consensus.cc:2749] T b4cd3d7dfbbf4096908ff11e9dc57141 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:03:51.637723 11148 consensus_queue.cc:260] T 63d44f6ba6434ab2870fe3190f4d7c2f P 10ca5ee68ad64b3283b9d235352c9d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:51.650484 10878 raft_consensus.cc:2468] T 5d439326f23747eaab3c1e7da2d6b785 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 1.
I20260709 13:03:51.653944 11152 tablet_bootstrap.cc:654] T b4cd3d7dfbbf4096908ff11e9dc57141 P 0172fed22b6b4890a7fdc953fa128af4: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:51.665740 10775 catalog_manager.cc:5697] T bac7e5040f984ba2b7146bd36af7267e P 0172fed22b6b4890a7fdc953fa128af4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196). New cstate: current_term: 1 leader_uuid: "0172fed22b6b4890a7fdc953fa128af4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } health_report { overall_health: HEALTHY } } }
I20260709 13:03:51.668706 11148 ts_tablet_manager.cc:1434] T 63d44f6ba6434ab2870fe3190f4d7c2f P 10ca5ee68ad64b3283b9d235352c9d23: Time spent starting tablet: real 0.040s	user 0.010s	sys 0.000s
I20260709 13:03:51.689954 11148 tablet_bootstrap.cc:492] T 11de93cd9b9d457489ff1cb80f976c9a P 10ca5ee68ad64b3283b9d235352c9d23: Bootstrap starting.
I20260709 13:03:51.671844 10776 catalog_manager.cc:5697] T 72b111ad7b5f4753ab1d2b5ee1114840 P 10ca5ee68ad64b3283b9d235352c9d23 reported cstate change: term changed from 0 to 1, leader changed from <none> to 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194). New cstate: current_term: 1 leader_uuid: "10ca5ee68ad64b3283b9d235352c9d23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } health_report { overall_health: HEALTHY } } }
I20260709 13:03:51.678959 11151 tablet_bootstrap.cc:492] T 65bf0a62f4d84ef18dd6b23e082724b8 P 186b12638a87410487467df721a3cbda: No bootstrap required, opened a new log
I20260709 13:03:51.693365 11151 ts_tablet_manager.cc:1403] T 65bf0a62f4d84ef18dd6b23e082724b8 P 186b12638a87410487467df721a3cbda: Time spent bootstrapping tablet: real 0.081s	user 0.015s	sys 0.011s
I20260709 13:03:51.696704 11151 raft_consensus.cc:359] T 65bf0a62f4d84ef18dd6b23e082724b8 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:51.697451 11151 raft_consensus.cc:385] T 65bf0a62f4d84ef18dd6b23e082724b8 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:51.697746 11151 raft_consensus.cc:740] T 65bf0a62f4d84ef18dd6b23e082724b8 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 186b12638a87410487467df721a3cbda, State: Initialized, Role: FOLLOWER
I20260709 13:03:51.698539 11151 consensus_queue.cc:260] T 65bf0a62f4d84ef18dd6b23e082724b8 P 186b12638a87410487467df721a3cbda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:51.700968 11151 ts_tablet_manager.cc:1434] T 65bf0a62f4d84ef18dd6b23e082724b8 P 186b12638a87410487467df721a3cbda: Time spent starting tablet: real 0.007s	user 0.005s	sys 0.000s
I20260709 13:03:51.701756 11148 tablet_bootstrap.cc:654] T 11de93cd9b9d457489ff1cb80f976c9a P 10ca5ee68ad64b3283b9d235352c9d23: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:51.702092 11151 tablet_bootstrap.cc:492] T 63d44f6ba6434ab2870fe3190f4d7c2f P 186b12638a87410487467df721a3cbda: Bootstrap starting.
I20260709 13:03:51.704457 11257 raft_consensus.cc:515] T bac7e5040f984ba2b7146bd36af7267e P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:51.705751 11029 raft_consensus.cc:2393] T bac7e5040f984ba2b7146bd36af7267e P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0172fed22b6b4890a7fdc953fa128af4 in current term 1: Already voted for candidate 873357742d5843579556ee8166f23f47 in this term.
I20260709 13:03:51.717080 11101 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bac7e5040f984ba2b7146bd36af7267e" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:03:51.720899 11257 leader_election.cc:290] T bac7e5040f984ba2b7146bd36af7267e P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 1 election: Requested vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:03:51.726013 11151 tablet_bootstrap.cc:654] T 63d44f6ba6434ab2870fe3190f4d7c2f P 186b12638a87410487467df721a3cbda: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:51.726897 10953 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bac7e5040f984ba2b7146bd36af7267e" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:03:51.727864 10953 raft_consensus.cc:2393] T bac7e5040f984ba2b7146bd36af7267e P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 873357742d5843579556ee8166f23f47 in current term 1: Already voted for candidate 0172fed22b6b4890a7fdc953fa128af4 in this term.
I20260709 13:03:51.729569 10992 leader_election.cc:304] T bac7e5040f984ba2b7146bd36af7267e P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 873357742d5843579556ee8166f23f47; no voters: 0172fed22b6b4890a7fdc953fa128af4, 10ca5ee68ad64b3283b9d235352c9d23
I20260709 13:03:51.732381 11149 tablet_bootstrap.cc:492] T 63d44f6ba6434ab2870fe3190f4d7c2f P 873357742d5843579556ee8166f23f47: No bootstrap required, opened a new log
I20260709 13:03:51.732985 11149 ts_tablet_manager.cc:1403] T 63d44f6ba6434ab2870fe3190f4d7c2f P 873357742d5843579556ee8166f23f47: Time spent bootstrapping tablet: real 0.219s	user 0.020s	sys 0.070s
I20260709 13:03:51.733448 11148 tablet_bootstrap.cc:492] T 11de93cd9b9d457489ff1cb80f976c9a P 10ca5ee68ad64b3283b9d235352c9d23: No bootstrap required, opened a new log
I20260709 13:03:51.734009 11148 ts_tablet_manager.cc:1403] T 11de93cd9b9d457489ff1cb80f976c9a P 10ca5ee68ad64b3283b9d235352c9d23: Time spent bootstrapping tablet: real 0.044s	user 0.003s	sys 0.007s
I20260709 13:03:51.737180 11148 raft_consensus.cc:359] T 11de93cd9b9d457489ff1cb80f976c9a P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:51.737902 11148 raft_consensus.cc:385] T 11de93cd9b9d457489ff1cb80f976c9a P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:51.738193 11148 raft_consensus.cc:740] T 11de93cd9b9d457489ff1cb80f976c9a P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10ca5ee68ad64b3283b9d235352c9d23, State: Initialized, Role: FOLLOWER
I20260709 13:03:51.739111 11148 consensus_queue.cc:260] T 11de93cd9b9d457489ff1cb80f976c9a P 10ca5ee68ad64b3283b9d235352c9d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:51.738350 11257 raft_consensus.cc:2749] T bac7e5040f984ba2b7146bd36af7267e P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 13:03:51.742327 11152 tablet_bootstrap.cc:492] T b4cd3d7dfbbf4096908ff11e9dc57141 P 0172fed22b6b4890a7fdc953fa128af4: No bootstrap required, opened a new log
I20260709 13:03:51.742901 11152 ts_tablet_manager.cc:1403] T b4cd3d7dfbbf4096908ff11e9dc57141 P 0172fed22b6b4890a7fdc953fa128af4: Time spent bootstrapping tablet: real 0.267s	user 0.013s	sys 0.001s
I20260709 13:03:51.742717 11149 raft_consensus.cc:359] T 63d44f6ba6434ab2870fe3190f4d7c2f P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:51.743693 11149 raft_consensus.cc:385] T 63d44f6ba6434ab2870fe3190f4d7c2f P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:51.744059 11149 raft_consensus.cc:740] T 63d44f6ba6434ab2870fe3190f4d7c2f P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 873357742d5843579556ee8166f23f47, State: Initialized, Role: FOLLOWER
I20260709 13:03:51.744938 11149 consensus_queue.cc:260] T 63d44f6ba6434ab2870fe3190f4d7c2f P 873357742d5843579556ee8166f23f47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:51.746114 11152 raft_consensus.cc:359] T b4cd3d7dfbbf4096908ff11e9dc57141 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:51.746974 11152 raft_consensus.cc:385] T b4cd3d7dfbbf4096908ff11e9dc57141 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:51.746688 11149 ts_tablet_manager.cc:1434] T 63d44f6ba6434ab2870fe3190f4d7c2f P 873357742d5843579556ee8166f23f47: Time spent starting tablet: real 0.013s	user 0.007s	sys 0.002s
I20260709 13:03:51.748026 11148 ts_tablet_manager.cc:1434] T 11de93cd9b9d457489ff1cb80f976c9a P 10ca5ee68ad64b3283b9d235352c9d23: Time spent starting tablet: real 0.014s	user 0.008s	sys 0.001s
I20260709 13:03:51.748569 11149 tablet_bootstrap.cc:492] T 32147c6d2d5a477aa38c597c7f78df5b P 873357742d5843579556ee8166f23f47: Bootstrap starting.
I20260709 13:03:51.749065 11148 tablet_bootstrap.cc:492] T b4cd3d7dfbbf4096908ff11e9dc57141 P 10ca5ee68ad64b3283b9d235352c9d23: Bootstrap starting.
I20260709 13:03:51.755909 11149 tablet_bootstrap.cc:654] T 32147c6d2d5a477aa38c597c7f78df5b P 873357742d5843579556ee8166f23f47: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:51.747262 11152 raft_consensus.cc:740] T b4cd3d7dfbbf4096908ff11e9dc57141 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0172fed22b6b4890a7fdc953fa128af4, State: Initialized, Role: FOLLOWER
I20260709 13:03:51.757094 11341 consensus_queue.cc:1048] T 72b111ad7b5f4753ab1d2b5ee1114840 P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:03:51.758689 10774 catalog_manager.cc:5697] T 5d439326f23747eaab3c1e7da2d6b785 P 10ca5ee68ad64b3283b9d235352c9d23 reported cstate change: term changed from 0 to 1, leader changed from <none> to 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194). New cstate: current_term: 1 leader_uuid: "10ca5ee68ad64b3283b9d235352c9d23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } health_report { overall_health: HEALTHY } } }
I20260709 13:03:51.757091 11152 consensus_queue.cc:260] T b4cd3d7dfbbf4096908ff11e9dc57141 P 0172fed22b6b4890a7fdc953fa128af4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:51.758878 11363 raft_consensus.cc:493] T 63d44f6ba6434ab2870fe3190f4d7c2f P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:51.762804 11363 raft_consensus.cc:515] T 63d44f6ba6434ab2870fe3190f4d7c2f P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:51.763617 11257 consensus_queue.cc:1048] T 868e92339c2f4508af0afbd07c814d64 P 873357742d5843579556ee8166f23f47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:03:51.771066 11363 leader_election.cc:290] T 63d44f6ba6434ab2870fe3190f4d7c2f P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:03:51.774559 11151 tablet_bootstrap.cc:492] T 63d44f6ba6434ab2870fe3190f4d7c2f P 186b12638a87410487467df721a3cbda: No bootstrap required, opened a new log
I20260709 13:03:51.775375 11151 ts_tablet_manager.cc:1403] T 63d44f6ba6434ab2870fe3190f4d7c2f P 186b12638a87410487467df721a3cbda: Time spent bootstrapping tablet: real 0.074s	user 0.012s	sys 0.001s
I20260709 13:03:51.773286 11029 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63d44f6ba6434ab2870fe3190f4d7c2f" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:03:51.780261 11029 raft_consensus.cc:2468] T 63d44f6ba6434ab2870fe3190f4d7c2f P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 0.
I20260709 13:03:51.779229 10877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63d44f6ba6434ab2870fe3190f4d7c2f" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:03:51.782003 10919 leader_election.cc:304] T 63d44f6ba6434ab2870fe3190f4d7c2f P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:03:51.783797 11148 tablet_bootstrap.cc:654] T b4cd3d7dfbbf4096908ff11e9dc57141 P 10ca5ee68ad64b3283b9d235352c9d23: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:51.784268 11341 raft_consensus.cc:2804] T 63d44f6ba6434ab2870fe3190f4d7c2f P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:03:51.785920 11341 raft_consensus.cc:493] T 63d44f6ba6434ab2870fe3190f4d7c2f P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:51.786360 11341 raft_consensus.cc:3060] T 63d44f6ba6434ab2870fe3190f4d7c2f P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:03:51.787271 11302 consensus_queue.cc:1048] T 868e92339c2f4508af0afbd07c814d64 P 873357742d5843579556ee8166f23f47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:03:51.789142 11257 raft_consensus.cc:493] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:51.789695 11257 raft_consensus.cc:515] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:51.791188 11152 ts_tablet_manager.cc:1434] T b4cd3d7dfbbf4096908ff11e9dc57141 P 0172fed22b6b4890a7fdc953fa128af4: Time spent starting tablet: real 0.048s	user 0.008s	sys 0.000s
I20260709 13:03:51.795715 11152 tablet_bootstrap.cc:492] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 0172fed22b6b4890a7fdc953fa128af4: Bootstrap starting.
I20260709 13:03:51.794631 10880 raft_consensus.cc:3060] T 72b111ad7b5f4753ab1d2b5ee1114840 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:03:51.804224 11257 leader_election.cc:290] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:03:51.805055 11103 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555217d45d9b4ee492dbb3ddaf0f7c02" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:03:51.807092 10879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555217d45d9b4ee492dbb3ddaf0f7c02" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
W20260709 13:03:51.807572 10994 leader_election.cc:343] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507): Illegal state: must be running to vote when last-logged opid is not known
W20260709 13:03:51.810388 10995 leader_election.cc:343] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 186b12638a87410487467df721a3cbda (127.0.107.193:38967): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:03:51.811081 10995 leader_election.cc:304] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 873357742d5843579556ee8166f23f47; no voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda
I20260709 13:03:51.813477 11257 raft_consensus.cc:2749] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260709 13:03:51.828338 10921 leader_election.cc:343] T 63d44f6ba6434ab2870fe3190f4d7c2f P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 186b12638a87410487467df721a3cbda (127.0.107.193:38967): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:03:51.835304 11162 raft_consensus.cc:493] T b4cd3d7dfbbf4096908ff11e9dc57141 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:51.836148 11162 raft_consensus.cc:515] T b4cd3d7dfbbf4096908ff11e9dc57141 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:51.840597 10953 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4cd3d7dfbbf4096908ff11e9dc57141" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:03:51.843976 11151 raft_consensus.cc:359] T 63d44f6ba6434ab2870fe3190f4d7c2f P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:51.844894 11360 raft_consensus.cc:493] T 11de93cd9b9d457489ff1cb80f976c9a P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:51.845849 11360 raft_consensus.cc:515] T 11de93cd9b9d457489ff1cb80f976c9a P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:51.853770 11029 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11de93cd9b9d457489ff1cb80f976c9a" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:03:51.845032 11151 raft_consensus.cc:385] T 63d44f6ba6434ab2870fe3190f4d7c2f P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:51.854660 11151 raft_consensus.cc:740] T 63d44f6ba6434ab2870fe3190f4d7c2f P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 186b12638a87410487467df721a3cbda, State: Initialized, Role: FOLLOWER
I20260709 13:03:51.855892 11151 consensus_queue.cc:260] T 63d44f6ba6434ab2870fe3190f4d7c2f P 186b12638a87410487467df721a3cbda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:51.871551 11152 tablet_bootstrap.cc:654] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 0172fed22b6b4890a7fdc953fa128af4: Neither blocks nor log segments found. Creating new log.
W20260709 13:03:51.864068 10919 leader_election.cc:343] T 11de93cd9b9d457489ff1cb80f976c9a P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 873357742d5843579556ee8166f23f47 (127.0.107.195:45807): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:03:51.872507 11332 raft_consensus.cc:493] T 72b111ad7b5f4753ab1d2b5ee1114840 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:51.863698 11360 leader_election.cc:290] T 11de93cd9b9d457489ff1cb80f976c9a P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:03:51.873136 11332 raft_consensus.cc:515] T 72b111ad7b5f4753ab1d2b5ee1114840 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:51.859126 11341 raft_consensus.cc:515] T 63d44f6ba6434ab2870fe3190f4d7c2f P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
W20260709 13:03:51.879379 11066 leader_election.cc:343] T b4cd3d7dfbbf4096908ff11e9dc57141 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:03:51.879766 11162 leader_election.cc:290] T b4cd3d7dfbbf4096908ff11e9dc57141 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:03:51.889811 11151 ts_tablet_manager.cc:1434] T 63d44f6ba6434ab2870fe3190f4d7c2f P 186b12638a87410487467df721a3cbda: Time spent starting tablet: real 0.114s	user 0.010s	sys 0.016s
I20260709 13:03:51.891086 11149 tablet_bootstrap.cc:492] T 32147c6d2d5a477aa38c597c7f78df5b P 873357742d5843579556ee8166f23f47: No bootstrap required, opened a new log
I20260709 13:03:51.891577 11149 ts_tablet_manager.cc:1403] T 32147c6d2d5a477aa38c597c7f78df5b P 873357742d5843579556ee8166f23f47: Time spent bootstrapping tablet: real 0.143s	user 0.018s	sys 0.016s
I20260709 13:03:51.892043 11333 raft_consensus.cc:493] T a610d1a71f52432e88fe56130f921fa2 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:51.893612 11151 tablet_bootstrap.cc:492] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 186b12638a87410487467df721a3cbda: Bootstrap starting.
I20260709 13:03:51.898473 11360 consensus_queue.cc:1048] T 72b111ad7b5f4753ab1d2b5ee1114840 P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:03:51.906606 10880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11de93cd9b9d457489ff1cb80f976c9a" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:03:51.908854 11341 leader_election.cc:290] T 63d44f6ba6434ab2870fe3190f4d7c2f P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 1 election: Requested vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:03:51.911825 11027 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63d44f6ba6434ab2870fe3190f4d7c2f" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
I20260709 13:03:51.912518 11027 raft_consensus.cc:3060] T 63d44f6ba6434ab2870fe3190f4d7c2f P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:03:51.913699 11149 raft_consensus.cc:359] T 32147c6d2d5a477aa38c597c7f78df5b P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:51.905663 11029 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4cd3d7dfbbf4096908ff11e9dc57141" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:03:51.915120 11029 raft_consensus.cc:2468] T b4cd3d7dfbbf4096908ff11e9dc57141 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 0.
I20260709 13:03:51.893290 11333 raft_consensus.cc:515] T a610d1a71f52432e88fe56130f921fa2 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:51.919075 11103 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a610d1a71f52432e88fe56130f921fa2" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:03:51.920037 11103 raft_consensus.cc:2393] T a610d1a71f52432e88fe56130f921fa2 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 186b12638a87410487467df721a3cbda in current term 1: Already voted for candidate 0172fed22b6b4890a7fdc953fa128af4 in this term.
I20260709 13:03:51.920116 11332 leader_election.cc:290] T 72b111ad7b5f4753ab1d2b5ee1114840 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:03:51.918359 11333 leader_election.cc:290] T a610d1a71f52432e88fe56130f921fa2 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:03:51.922999 11027 raft_consensus.cc:2468] T 63d44f6ba6434ab2870fe3190f4d7c2f P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 1.
I20260709 13:03:51.925313 11103 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72b111ad7b5f4753ab1d2b5ee1114840" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:03:51.926162 10919 leader_election.cc:304] T 63d44f6ba6434ab2870fe3190f4d7c2f P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:03:51.927268 11168 raft_consensus.cc:2804] T 63d44f6ba6434ab2870fe3190f4d7c2f P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:03:51.927532 10953 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72b111ad7b5f4753ab1d2b5ee1114840" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:03:51.927820 11168 raft_consensus.cc:697] T 63d44f6ba6434ab2870fe3190f4d7c2f P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 LEADER]: Becoming Leader. State: Replica: 10ca5ee68ad64b3283b9d235352c9d23, State: Running, Role: LEADER
I20260709 13:03:51.929616 10843 leader_election.cc:304] T 72b111ad7b5f4753ab1d2b5ee1114840 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 186b12638a87410487467df721a3cbda; no voters: 0172fed22b6b4890a7fdc953fa128af4, 10ca5ee68ad64b3283b9d235352c9d23
W20260709 13:03:51.929860 10921 leader_election.cc:343] T 11de93cd9b9d457489ff1cb80f976c9a P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 186b12638a87410487467df721a3cbda (127.0.107.193:38967): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:03:51.930370 10921 leader_election.cc:304] T 11de93cd9b9d457489ff1cb80f976c9a P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23; no voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47
I20260709 13:03:51.931203 11341 raft_consensus.cc:2749] T 11de93cd9b9d457489ff1cb80f976c9a P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:03:51.914399 11149 raft_consensus.cc:385] T 32147c6d2d5a477aa38c597c7f78df5b P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:51.932869 11149 raft_consensus.cc:740] T 32147c6d2d5a477aa38c597c7f78df5b P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 873357742d5843579556ee8166f23f47, State: Initialized, Role: FOLLOWER
I20260709 13:03:51.934579 11067 leader_election.cc:304] T b4cd3d7dfbbf4096908ff11e9dc57141 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47; no voters: 10ca5ee68ad64b3283b9d235352c9d23
I20260709 13:03:51.934648 11029 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a610d1a71f52432e88fe56130f921fa2" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:03:51.935882 11029 raft_consensus.cc:2393] T a610d1a71f52432e88fe56130f921fa2 P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 186b12638a87410487467df721a3cbda in current term 1: Already voted for candidate 873357742d5843579556ee8166f23f47 in this term.
I20260709 13:03:51.936565 10877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63d44f6ba6434ab2870fe3190f4d7c2f" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:03:51.936736 11311 raft_consensus.cc:2804] T b4cd3d7dfbbf4096908ff11e9dc57141 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:03:51.937248 11311 raft_consensus.cc:493] T b4cd3d7dfbbf4096908ff11e9dc57141 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:51.937676 11311 raft_consensus.cc:3060] T b4cd3d7dfbbf4096908ff11e9dc57141 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:03:51.939497 11168 consensus_queue.cc:237] T 63d44f6ba6434ab2870fe3190f4d7c2f P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:51.937260 10877 raft_consensus.cc:3060] T 63d44f6ba6434ab2870fe3190f4d7c2f P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:03:51.932760 11404 raft_consensus.cc:493] T d270b9e5481245d9b5340a454a7cd10b P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:51.950934 11149 consensus_queue.cc:260] T 32147c6d2d5a477aa38c597c7f78df5b P 873357742d5843579556ee8166f23f47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:51.944703 11404 raft_consensus.cc:515] T d270b9e5481245d9b5340a454a7cd10b P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:51.959496 10844 leader_election.cc:304] T a610d1a71f52432e88fe56130f921fa2 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 186b12638a87410487467df721a3cbda; no voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47
W20260709 13:03:51.984948 11403 log.cc:927] Time spent T 868e92339c2f4508af0afbd07c814d64 P 0172fed22b6b4890a7fdc953fa128af4: Append to log took a long time: real 0.085s	user 0.000s	sys 0.000s
I20260709 13:03:51.996869 10876 raft_consensus.cc:3060] T d270b9e5481245d9b5340a454a7cd10b P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:03:51.998384 11404 leader_election.cc:290] T d270b9e5481245d9b5340a454a7cd10b P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:03:51.933382 11333 consensus_queue.cc:1048] T 2567c2d84e4648d7ba5dc23a3df32817 P 186b12638a87410487467df721a3cbda [LEADER]: Connected to new peer: Peer: permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.029s
I20260709 13:03:52.004901 11162 consensus_queue.cc:1048] T d270b9e5481245d9b5340a454a7cd10b P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:03:52.005880 11151 tablet_bootstrap.cc:654] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 186b12638a87410487467df721a3cbda: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:52.021258 11149 ts_tablet_manager.cc:1434] T 32147c6d2d5a477aa38c597c7f78df5b P 873357742d5843579556ee8166f23f47: Time spent starting tablet: real 0.129s	user 0.009s	sys 0.000s
I20260709 13:03:52.082556 11149 tablet_bootstrap.cc:492] T 0e5d327630e5438b878035d28efe6c0c P 873357742d5843579556ee8166f23f47: Bootstrap starting.
W20260709 13:03:52.081310 10843 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 151820us
I20260709 13:03:52.018512 11332 raft_consensus.cc:2749] T 72b111ad7b5f4753ab1d2b5ee1114840 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:03:52.051720 11360 raft_consensus.cc:493] T 65bf0a62f4d84ef18dd6b23e082724b8 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:52.087935 11404 raft_consensus.cc:3060] T a610d1a71f52432e88fe56130f921fa2 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:03:52.014142 10877 raft_consensus.cc:2468] T 63d44f6ba6434ab2870fe3190f4d7c2f P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 1.
I20260709 13:03:52.107721 11148 tablet_bootstrap.cc:492] T b4cd3d7dfbbf4096908ff11e9dc57141 P 10ca5ee68ad64b3283b9d235352c9d23: No bootstrap required, opened a new log
I20260709 13:03:52.108958 11148 ts_tablet_manager.cc:1403] T b4cd3d7dfbbf4096908ff11e9dc57141 P 10ca5ee68ad64b3283b9d235352c9d23: Time spent bootstrapping tablet: real 0.360s	user 0.031s	sys 0.026s
I20260709 13:03:52.042187 11354 consensus_queue.cc:1048] T bac7e5040f984ba2b7146bd36af7267e P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.059s
I20260709 13:03:52.112197 11302 raft_consensus.cc:493] T 32147c6d2d5a477aa38c597c7f78df5b P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:52.116892 11302 raft_consensus.cc:515] T 32147c6d2d5a477aa38c597c7f78df5b P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:52.125573 11302 leader_election.cc:290] T 32147c6d2d5a477aa38c597c7f78df5b P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:03:52.132192 11152 tablet_bootstrap.cc:492] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 0172fed22b6b4890a7fdc953fa128af4: No bootstrap required, opened a new log
I20260709 13:03:52.132962 11152 ts_tablet_manager.cc:1403] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 0172fed22b6b4890a7fdc953fa128af4: Time spent bootstrapping tablet: real 0.340s	user 0.016s	sys 0.004s
I20260709 13:03:52.105086 11360 raft_consensus.cc:515] T 65bf0a62f4d84ef18dd6b23e082724b8 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:52.139788 10955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d270b9e5481245d9b5340a454a7cd10b" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
W20260709 13:03:52.143018 11407 log.cc:927] Time spent T 72b111ad7b5f4753ab1d2b5ee1114840 P 186b12638a87410487467df721a3cbda: Append to log took a long time: real 0.054s	user 0.000s	sys 0.001s
I20260709 13:03:52.144982 10955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32147c6d2d5a477aa38c597c7f78df5b" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:03:52.134299 11311 raft_consensus.cc:515] T b4cd3d7dfbbf4096908ff11e9dc57141 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:52.150705 10776 catalog_manager.cc:5697] T 63d44f6ba6434ab2870fe3190f4d7c2f P 10ca5ee68ad64b3283b9d235352c9d23 reported cstate change: term changed from 0 to 1, leader changed from <none> to 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194). New cstate: current_term: 1 leader_uuid: "10ca5ee68ad64b3283b9d235352c9d23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } health_report { overall_health: UNKNOWN } } }
I20260709 13:03:52.149129 11341 consensus_queue.cc:1048] T 5d439326f23747eaab3c1e7da2d6b785 P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
W20260709 13:03:52.155243 10992 leader_election.cc:343] T 32147c6d2d5a477aa38c597c7f78df5b P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:03:52.156293 11102 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d270b9e5481245d9b5340a454a7cd10b" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:03:52.128870 11409 raft_consensus.cc:493] T 65bf0a62f4d84ef18dd6b23e082724b8 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:52.148255 11311 leader_election.cc:290] T b4cd3d7dfbbf4096908ff11e9dc57141 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 1 election: Requested vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:03:52.164023 11412 consensus_queue.cc:1048] T bac7e5040f984ba2b7146bd36af7267e P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:03:52.165768 11148 raft_consensus.cc:359] T b4cd3d7dfbbf4096908ff11e9dc57141 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:52.167404 11027 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4cd3d7dfbbf4096908ff11e9dc57141" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
I20260709 13:03:52.167603 10878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65bf0a62f4d84ef18dd6b23e082724b8" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:03:52.171293 11027 raft_consensus.cc:3060] T b4cd3d7dfbbf4096908ff11e9dc57141 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:03:52.169072 11148 raft_consensus.cc:385] T b4cd3d7dfbbf4096908ff11e9dc57141 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:52.173687 11148 raft_consensus.cc:740] T b4cd3d7dfbbf4096908ff11e9dc57141 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10ca5ee68ad64b3283b9d235352c9d23, State: Initialized, Role: FOLLOWER
I20260709 13:03:52.174526 11148 consensus_queue.cc:260] T b4cd3d7dfbbf4096908ff11e9dc57141 P 10ca5ee68ad64b3283b9d235352c9d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:52.174997 10954 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4cd3d7dfbbf4096908ff11e9dc57141" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:03:52.181296 10954 raft_consensus.cc:3060] T b4cd3d7dfbbf4096908ff11e9dc57141 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:03:52.194512 11148 ts_tablet_manager.cc:1434] T b4cd3d7dfbbf4096908ff11e9dc57141 P 10ca5ee68ad64b3283b9d235352c9d23: Time spent starting tablet: real 0.085s	user 0.011s	sys 0.000s
I20260709 13:03:52.195828 11148 tablet_bootstrap.cc:492] T 87cd8f5097624c8692f1d83544d8881e P 10ca5ee68ad64b3283b9d235352c9d23: Bootstrap starting.
I20260709 13:03:52.168442 11409 raft_consensus.cc:515] T 65bf0a62f4d84ef18dd6b23e082724b8 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:52.196381 10845 leader_election.cc:304] T d270b9e5481245d9b5340a454a7cd10b P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 186b12638a87410487467df721a3cbda; no voters: 0172fed22b6b4890a7fdc953fa128af4, 10ca5ee68ad64b3283b9d235352c9d23
I20260709 13:03:52.196828 10878 raft_consensus.cc:2468] T 65bf0a62f4d84ef18dd6b23e082724b8 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 0.
I20260709 13:03:52.200260 11026 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65bf0a62f4d84ef18dd6b23e082724b8" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:03:52.201246 11026 raft_consensus.cc:2468] T 65bf0a62f4d84ef18dd6b23e082724b8 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 0.
I20260709 13:03:52.203279 10919 leader_election.cc:304] T 65bf0a62f4d84ef18dd6b23e082724b8 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:03:52.204466 11168 raft_consensus.cc:2804] T 65bf0a62f4d84ef18dd6b23e082724b8 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:03:52.204885 11168 raft_consensus.cc:493] T 65bf0a62f4d84ef18dd6b23e082724b8 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:52.205202 11168 raft_consensus.cc:3060] T 65bf0a62f4d84ef18dd6b23e082724b8 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:03:52.211649 11409 leader_election.cc:290] T 65bf0a62f4d84ef18dd6b23e082724b8 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:03:52.212460 11026 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65bf0a62f4d84ef18dd6b23e082724b8" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:03:52.213287 11026 raft_consensus.cc:2468] T 65bf0a62f4d84ef18dd6b23e082724b8 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 0.
I20260709 13:03:52.214646 10844 leader_election.cc:304] T 65bf0a62f4d84ef18dd6b23e082724b8 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:03:52.218413 10952 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65bf0a62f4d84ef18dd6b23e082724b8" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:03:52.219043 11408 raft_consensus.cc:2804] T 65bf0a62f4d84ef18dd6b23e082724b8 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:03:52.219640 11408 raft_consensus.cc:493] T 65bf0a62f4d84ef18dd6b23e082724b8 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:52.220082 11408 raft_consensus.cc:3060] T 65bf0a62f4d84ef18dd6b23e082724b8 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:03:52.215301 11413 consensus_queue.cc:1048] T 2567c2d84e4648d7ba5dc23a3df32817 P 186b12638a87410487467df721a3cbda [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.012s
I20260709 13:03:52.214747 11360 leader_election.cc:290] T 65bf0a62f4d84ef18dd6b23e082724b8 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:03:52.232419 11149 tablet_bootstrap.cc:654] T 0e5d327630e5438b878035d28efe6c0c P 873357742d5843579556ee8166f23f47: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:52.239970 11103 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32147c6d2d5a477aa38c597c7f78df5b" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:03:52.242480 11408 raft_consensus.cc:515] T 65bf0a62f4d84ef18dd6b23e082724b8 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:52.254709 11152 raft_consensus.cc:359] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:52.271047 11152 raft_consensus.cc:385] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:52.271473 11152 raft_consensus.cc:740] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0172fed22b6b4890a7fdc953fa128af4, State: Initialized, Role: FOLLOWER
I20260709 13:03:52.271605 11168 raft_consensus.cc:515] T 65bf0a62f4d84ef18dd6b23e082724b8 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:52.272526 11152 consensus_queue.cc:260] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 0172fed22b6b4890a7fdc953fa128af4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:52.273435 10952 raft_consensus.cc:2393] T 65bf0a62f4d84ef18dd6b23e082724b8 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 186b12638a87410487467df721a3cbda in current term 1: Already voted for candidate 10ca5ee68ad64b3283b9d235352c9d23 in this term.
I20260709 13:03:52.261662 11409 raft_consensus.cc:2749] T d270b9e5481245d9b5340a454a7cd10b P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:03:52.265532 11027 raft_consensus.cc:2468] T b4cd3d7dfbbf4096908ff11e9dc57141 P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 1.
I20260709 13:03:52.257225 11148 tablet_bootstrap.cc:654] T 87cd8f5097624c8692f1d83544d8881e P 10ca5ee68ad64b3283b9d235352c9d23: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:52.283493 11152 ts_tablet_manager.cc:1434] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 0172fed22b6b4890a7fdc953fa128af4: Time spent starting tablet: real 0.150s	user 0.004s	sys 0.005s
I20260709 13:03:52.286230 11152 tablet_bootstrap.cc:492] T 5de5f0581e3b41dea729363d6ca563bb P 0172fed22b6b4890a7fdc953fa128af4: Bootstrap starting.
I20260709 13:03:52.279613 10954 raft_consensus.cc:2468] T b4cd3d7dfbbf4096908ff11e9dc57141 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 1.
I20260709 13:03:52.296674 11360 consensus_queue.cc:1048] T 5d439326f23747eaab3c1e7da2d6b785 P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260709 13:03:52.290793 10994 leader_election.cc:343] T 32147c6d2d5a477aa38c597c7f78df5b P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:03:52.298576 10994 leader_election.cc:304] T 32147c6d2d5a477aa38c597c7f78df5b P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 873357742d5843579556ee8166f23f47; no voters: 0172fed22b6b4890a7fdc953fa128af4, 10ca5ee68ad64b3283b9d235352c9d23
I20260709 13:03:52.304076 11301 raft_consensus.cc:2749] T 32147c6d2d5a477aa38c597c7f78df5b P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:03:52.305979 11168 leader_election.cc:290] T 65bf0a62f4d84ef18dd6b23e082724b8 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 1 election: Requested vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:03:52.313066 11404 raft_consensus.cc:2749] T a610d1a71f52432e88fe56130f921fa2 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:03:52.318239 11066 leader_election.cc:304] T b4cd3d7dfbbf4096908ff11e9dc57141 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 10ca5ee68ad64b3283b9d235352c9d23; no voters: 
I20260709 13:03:52.319249 11354 raft_consensus.cc:2804] T b4cd3d7dfbbf4096908ff11e9dc57141 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:03:52.319741 11354 raft_consensus.cc:697] T b4cd3d7dfbbf4096908ff11e9dc57141 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 LEADER]: Becoming Leader. State: Replica: 0172fed22b6b4890a7fdc953fa128af4, State: Running, Role: LEADER
I20260709 13:03:52.325629 11354 consensus_queue.cc:237] T b4cd3d7dfbbf4096908ff11e9dc57141 P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:52.328874 11027 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65bf0a62f4d84ef18dd6b23e082724b8" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
I20260709 13:03:52.333791 11027 raft_consensus.cc:3060] T 65bf0a62f4d84ef18dd6b23e082724b8 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:03:52.356310 11151 tablet_bootstrap.cc:492] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 186b12638a87410487467df721a3cbda: No bootstrap required, opened a new log
I20260709 13:03:52.357290 11151 ts_tablet_manager.cc:1403] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 186b12638a87410487467df721a3cbda: Time spent bootstrapping tablet: real 0.466s	user 0.020s	sys 0.015s
I20260709 13:03:52.363992 10876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65bf0a62f4d84ef18dd6b23e082724b8" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:03:52.387635 11151 raft_consensus.cc:359] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:52.389254 11151 raft_consensus.cc:385] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:52.389614 11151 raft_consensus.cc:740] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 186b12638a87410487467df721a3cbda, State: Initialized, Role: FOLLOWER
I20260709 13:03:52.393985 11148 tablet_bootstrap.cc:492] T 87cd8f5097624c8692f1d83544d8881e P 10ca5ee68ad64b3283b9d235352c9d23: No bootstrap required, opened a new log
I20260709 13:03:52.394637 11148 ts_tablet_manager.cc:1403] T 87cd8f5097624c8692f1d83544d8881e P 10ca5ee68ad64b3283b9d235352c9d23: Time spent bootstrapping tablet: real 0.199s	user 0.020s	sys 0.005s
I20260709 13:03:52.412048 11151 consensus_queue.cc:260] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 186b12638a87410487467df721a3cbda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:52.416760 11408 leader_election.cc:290] T 65bf0a62f4d84ef18dd6b23e082724b8 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 1 election: Requested vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:03:52.417447 10876 raft_consensus.cc:2393] T 65bf0a62f4d84ef18dd6b23e082724b8 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 10ca5ee68ad64b3283b9d235352c9d23 in current term 1: Already voted for candidate 186b12638a87410487467df721a3cbda in this term.
I20260709 13:03:52.419373 11412 consensus_queue.cc:1048] T d270b9e5481245d9b5340a454a7cd10b P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:03:52.423179 11149 tablet_bootstrap.cc:492] T 0e5d327630e5438b878035d28efe6c0c P 873357742d5843579556ee8166f23f47: No bootstrap required, opened a new log
I20260709 13:03:52.423780 11149 ts_tablet_manager.cc:1403] T 0e5d327630e5438b878035d28efe6c0c P 873357742d5843579556ee8166f23f47: Time spent bootstrapping tablet: real 0.341s	user 0.024s	sys 0.031s
I20260709 13:03:52.419368 11148 raft_consensus.cc:359] T 87cd8f5097624c8692f1d83544d8881e P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:52.424634 11148 raft_consensus.cc:385] T 87cd8f5097624c8692f1d83544d8881e P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:52.425559 11148 raft_consensus.cc:740] T 87cd8f5097624c8692f1d83544d8881e P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10ca5ee68ad64b3283b9d235352c9d23, State: Initialized, Role: FOLLOWER
I20260709 13:03:52.426975 11148 consensus_queue.cc:260] T 87cd8f5097624c8692f1d83544d8881e P 10ca5ee68ad64b3283b9d235352c9d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:52.429379 11148 ts_tablet_manager.cc:1434] T 87cd8f5097624c8692f1d83544d8881e P 10ca5ee68ad64b3283b9d235352c9d23: Time spent starting tablet: real 0.034s	user 0.006s	sys 0.000s
I20260709 13:03:52.443934 11029 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65bf0a62f4d84ef18dd6b23e082724b8" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
I20260709 13:03:52.459391 11360 consensus_queue.cc:1048] T 63d44f6ba6434ab2870fe3190f4d7c2f P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:03:52.462514 10776 catalog_manager.cc:5697] T b4cd3d7dfbbf4096908ff11e9dc57141 P 0172fed22b6b4890a7fdc953fa128af4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196). New cstate: current_term: 1 leader_uuid: "0172fed22b6b4890a7fdc953fa128af4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } health_report { overall_health: HEALTHY } } }
I20260709 13:03:52.466825 11149 raft_consensus.cc:359] T 0e5d327630e5438b878035d28efe6c0c P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:52.469636 11152 tablet_bootstrap.cc:654] T 5de5f0581e3b41dea729363d6ca563bb P 0172fed22b6b4890a7fdc953fa128af4: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:52.472160 11149 raft_consensus.cc:385] T 0e5d327630e5438b878035d28efe6c0c P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:52.472569 11149 raft_consensus.cc:740] T 0e5d327630e5438b878035d28efe6c0c P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 873357742d5843579556ee8166f23f47, State: Initialized, Role: FOLLOWER
I20260709 13:03:52.473745 11149 consensus_queue.cc:260] T 0e5d327630e5438b878035d28efe6c0c P 873357742d5843579556ee8166f23f47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:52.438719 11148 tablet_bootstrap.cc:492] T 5de5f0581e3b41dea729363d6ca563bb P 10ca5ee68ad64b3283b9d235352c9d23: Bootstrap starting.
I20260709 13:03:52.478273 11149 ts_tablet_manager.cc:1434] T 0e5d327630e5438b878035d28efe6c0c P 873357742d5843579556ee8166f23f47: Time spent starting tablet: real 0.054s	user 0.009s	sys 0.000s
I20260709 13:03:52.478384 11151 ts_tablet_manager.cc:1434] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 186b12638a87410487467df721a3cbda: Time spent starting tablet: real 0.121s	user 0.009s	sys 0.002s
I20260709 13:03:52.482559 11151 tablet_bootstrap.cc:492] T 11de93cd9b9d457489ff1cb80f976c9a P 186b12638a87410487467df721a3cbda: Bootstrap starting.
W20260709 13:03:52.481428 10992 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 77076.5us
W20260709 13:03:52.490728 11067 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:45807, user_credentials={real_user=slave}} blocked reactor thread for 55768.3us
W20260709 13:03:52.499552 10826 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 13:03:52.503726 10954 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65bf0a62f4d84ef18dd6b23e082724b8" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:03:52.504665 10954 raft_consensus.cc:2393] T 65bf0a62f4d84ef18dd6b23e082724b8 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 186b12638a87410487467df721a3cbda in current term 1: Already voted for candidate 10ca5ee68ad64b3283b9d235352c9d23 in this term.
I20260709 13:03:52.531926 11149 tablet_bootstrap.cc:492] T 11de93cd9b9d457489ff1cb80f976c9a P 873357742d5843579556ee8166f23f47: Bootstrap starting.
I20260709 13:03:52.528419 11027 raft_consensus.cc:2468] T 65bf0a62f4d84ef18dd6b23e082724b8 P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 1.
I20260709 13:03:52.545277 11151 tablet_bootstrap.cc:654] T 11de93cd9b9d457489ff1cb80f976c9a P 186b12638a87410487467df721a3cbda: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:52.545305 10843 leader_election.cc:304] T 65bf0a62f4d84ef18dd6b23e082724b8 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 186b12638a87410487467df721a3cbda; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47
I20260709 13:03:52.561592 11409 raft_consensus.cc:2749] T 65bf0a62f4d84ef18dd6b23e082724b8 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 13:03:52.567163 11430 consensus_queue.cc:1048] T 63d44f6ba6434ab2870fe3190f4d7c2f P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:03:52.568152 10919 leader_election.cc:304] T 65bf0a62f4d84ef18dd6b23e082724b8 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47; no voters: 186b12638a87410487467df721a3cbda
I20260709 13:03:52.569666 11357 raft_consensus.cc:2804] T 65bf0a62f4d84ef18dd6b23e082724b8 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:03:52.570158 11357 raft_consensus.cc:697] T 65bf0a62f4d84ef18dd6b23e082724b8 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 LEADER]: Becoming Leader. State: Replica: 10ca5ee68ad64b3283b9d235352c9d23, State: Running, Role: LEADER
I20260709 13:03:52.571166 11357 consensus_queue.cc:237] T 65bf0a62f4d84ef18dd6b23e082724b8 P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:52.580020 11148 tablet_bootstrap.cc:654] T 5de5f0581e3b41dea729363d6ca563bb P 10ca5ee68ad64b3283b9d235352c9d23: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:52.592363 11162 raft_consensus.cc:493] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:52.593034 11162 raft_consensus.cc:515] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:52.595880 11162 leader_election.cc:290] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:03:52.598052 11149 tablet_bootstrap.cc:654] T 11de93cd9b9d457489ff1cb80f976c9a P 873357742d5843579556ee8166f23f47: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:52.598554 11029 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555217d45d9b4ee492dbb3ddaf0f7c02" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:03:52.601254 11029 raft_consensus.cc:2468] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 0.
I20260709 13:03:52.603276 11067 leader_election.cc:304] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:03:52.604264 11162 raft_consensus.cc:2804] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:03:52.604548 11428 raft_consensus.cc:493] T 0e5d327630e5438b878035d28efe6c0c P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:52.605257 11428 raft_consensus.cc:515] T 0e5d327630e5438b878035d28efe6c0c P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:52.607596 11428 leader_election.cc:290] T 0e5d327630e5438b878035d28efe6c0c P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:03:52.604753 11162 raft_consensus.cc:493] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:52.608282 11162 raft_consensus.cc:3060] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:03:52.609081 11152 tablet_bootstrap.cc:492] T 5de5f0581e3b41dea729363d6ca563bb P 0172fed22b6b4890a7fdc953fa128af4: No bootstrap required, opened a new log
I20260709 13:03:52.609674 11152 ts_tablet_manager.cc:1403] T 5de5f0581e3b41dea729363d6ca563bb P 0172fed22b6b4890a7fdc953fa128af4: Time spent bootstrapping tablet: real 0.325s	user 0.022s	sys 0.031s
I20260709 13:03:52.609828 10880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555217d45d9b4ee492dbb3ddaf0f7c02" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:03:52.610633 10880 raft_consensus.cc:2468] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 0.
I20260709 13:03:52.610846 11102 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e5d327630e5438b878035d28efe6c0c" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
W20260709 13:03:52.612495 10994 leader_election.cc:343] T 0e5d327630e5438b878035d28efe6c0c P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:03:52.614840 10880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e5d327630e5438b878035d28efe6c0c" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
W20260709 13:03:52.629797 10995 leader_election.cc:343] T 0e5d327630e5438b878035d28efe6c0c P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 186b12638a87410487467df721a3cbda (127.0.107.193:38967): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:03:52.638183 10995 leader_election.cc:304] T 0e5d327630e5438b878035d28efe6c0c P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 873357742d5843579556ee8166f23f47; no voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda
I20260709 13:03:52.639142 11428 raft_consensus.cc:2749] T 0e5d327630e5438b878035d28efe6c0c P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:03:52.651091 11162 raft_consensus.cc:515] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:52.652585 11152 raft_consensus.cc:359] T 5de5f0581e3b41dea729363d6ca563bb P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:52.653431 11152 raft_consensus.cc:385] T 5de5f0581e3b41dea729363d6ca563bb P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:52.653720 11152 raft_consensus.cc:740] T 5de5f0581e3b41dea729363d6ca563bb P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0172fed22b6b4890a7fdc953fa128af4, State: Initialized, Role: FOLLOWER
I20260709 13:03:52.654531 11152 consensus_queue.cc:260] T 5de5f0581e3b41dea729363d6ca563bb P 0172fed22b6b4890a7fdc953fa128af4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:52.654832 11029 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555217d45d9b4ee492dbb3ddaf0f7c02" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
I20260709 13:03:52.655506 11029 raft_consensus.cc:3060] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:03:52.655161 10878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555217d45d9b4ee492dbb3ddaf0f7c02" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:03:52.656589 10878 raft_consensus.cc:3060] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:03:52.656913 11152 ts_tablet_manager.cc:1434] T 5de5f0581e3b41dea729363d6ca563bb P 0172fed22b6b4890a7fdc953fa128af4: Time spent starting tablet: real 0.047s	user 0.006s	sys 0.003s
I20260709 13:03:52.660305 11148 tablet_bootstrap.cc:492] T 5de5f0581e3b41dea729363d6ca563bb P 10ca5ee68ad64b3283b9d235352c9d23: No bootstrap required, opened a new log
I20260709 13:03:52.659279 10775 catalog_manager.cc:5697] T 65bf0a62f4d84ef18dd6b23e082724b8 P 10ca5ee68ad64b3283b9d235352c9d23 reported cstate change: term changed from 0 to 1, leader changed from <none> to 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194). New cstate: current_term: 1 leader_uuid: "10ca5ee68ad64b3283b9d235352c9d23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } health_report { overall_health: UNKNOWN } } }
I20260709 13:03:52.661365 11152 tablet_bootstrap.cc:492] T 32147c6d2d5a477aa38c597c7f78df5b P 0172fed22b6b4890a7fdc953fa128af4: Bootstrap starting.
I20260709 13:03:52.665809 11029 raft_consensus.cc:2468] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 1.
I20260709 13:03:52.665753 11148 ts_tablet_manager.cc:1403] T 5de5f0581e3b41dea729363d6ca563bb P 10ca5ee68ad64b3283b9d235352c9d23: Time spent bootstrapping tablet: real 0.235s	user 0.016s	sys 0.005s
I20260709 13:03:52.667649 11067 leader_election.cc:304] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:03:52.668646 11354 raft_consensus.cc:2804] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:03:52.669178 11354 raft_consensus.cc:697] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 LEADER]: Becoming Leader. State: Replica: 0172fed22b6b4890a7fdc953fa128af4, State: Running, Role: LEADER
I20260709 13:03:52.670104 11354 consensus_queue.cc:237] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:52.673619 11149 tablet_bootstrap.cc:492] T 11de93cd9b9d457489ff1cb80f976c9a P 873357742d5843579556ee8166f23f47: No bootstrap required, opened a new log
I20260709 13:03:52.674268 11149 ts_tablet_manager.cc:1403] T 11de93cd9b9d457489ff1cb80f976c9a P 873357742d5843579556ee8166f23f47: Time spent bootstrapping tablet: real 0.143s	user 0.017s	sys 0.000s
I20260709 13:03:52.677390 11151 tablet_bootstrap.cc:492] T 11de93cd9b9d457489ff1cb80f976c9a P 186b12638a87410487467df721a3cbda: No bootstrap required, opened a new log
I20260709 13:03:52.677929 11151 ts_tablet_manager.cc:1403] T 11de93cd9b9d457489ff1cb80f976c9a P 186b12638a87410487467df721a3cbda: Time spent bootstrapping tablet: real 0.198s	user 0.012s	sys 0.008s
I20260709 13:03:52.678596 11162 leader_election.cc:290] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 1 election: Requested vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:03:52.677970 11149 raft_consensus.cc:359] T 11de93cd9b9d457489ff1cb80f976c9a P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:52.679858 11149 raft_consensus.cc:385] T 11de93cd9b9d457489ff1cb80f976c9a P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:52.680990 11151 raft_consensus.cc:359] T 11de93cd9b9d457489ff1cb80f976c9a P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:52.681744 11151 raft_consensus.cc:385] T 11de93cd9b9d457489ff1cb80f976c9a P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:52.681958 10878 raft_consensus.cc:2468] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 1.
I20260709 13:03:52.682016 11151 raft_consensus.cc:740] T 11de93cd9b9d457489ff1cb80f976c9a P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 186b12638a87410487467df721a3cbda, State: Initialized, Role: FOLLOWER
I20260709 13:03:52.680233 11149 raft_consensus.cc:740] T 11de93cd9b9d457489ff1cb80f976c9a P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 873357742d5843579556ee8166f23f47, State: Initialized, Role: FOLLOWER
I20260709 13:03:52.684540 11149 consensus_queue.cc:260] T 11de93cd9b9d457489ff1cb80f976c9a P 873357742d5843579556ee8166f23f47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:52.684983 11148 raft_consensus.cc:359] T 5de5f0581e3b41dea729363d6ca563bb P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:52.683174 11151 consensus_queue.cc:260] T 11de93cd9b9d457489ff1cb80f976c9a P 186b12638a87410487467df721a3cbda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:52.687938 11149 ts_tablet_manager.cc:1434] T 11de93cd9b9d457489ff1cb80f976c9a P 873357742d5843579556ee8166f23f47: Time spent starting tablet: real 0.013s	user 0.007s	sys 0.000s
I20260709 13:03:52.688925 11151 ts_tablet_manager.cc:1434] T 11de93cd9b9d457489ff1cb80f976c9a P 186b12638a87410487467df721a3cbda: Time spent starting tablet: real 0.011s	user 0.009s	sys 0.000s
I20260709 13:03:52.689254 11149 tablet_bootstrap.cc:492] T d371c5e27bd843469d2e2e453734cb4a P 873357742d5843579556ee8166f23f47: Bootstrap starting.
I20260709 13:03:52.690757 11151 tablet_bootstrap.cc:492] T 5de5f0581e3b41dea729363d6ca563bb P 186b12638a87410487467df721a3cbda: Bootstrap starting.
I20260709 13:03:52.685719 11148 raft_consensus.cc:385] T 5de5f0581e3b41dea729363d6ca563bb P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:52.691671 11148 raft_consensus.cc:740] T 5de5f0581e3b41dea729363d6ca563bb P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10ca5ee68ad64b3283b9d235352c9d23, State: Initialized, Role: FOLLOWER
I20260709 13:03:52.697899 11148 consensus_queue.cc:260] T 5de5f0581e3b41dea729363d6ca563bb P 10ca5ee68ad64b3283b9d235352c9d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:52.698388 11149 tablet_bootstrap.cc:654] T d371c5e27bd843469d2e2e453734cb4a P 873357742d5843579556ee8166f23f47: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:52.706576 11149 tablet_bootstrap.cc:492] T d371c5e27bd843469d2e2e453734cb4a P 873357742d5843579556ee8166f23f47: No bootstrap required, opened a new log
I20260709 13:03:52.707106 11149 ts_tablet_manager.cc:1403] T d371c5e27bd843469d2e2e453734cb4a P 873357742d5843579556ee8166f23f47: Time spent bootstrapping tablet: real 0.018s	user 0.015s	sys 0.000s
I20260709 13:03:52.708531 11152 tablet_bootstrap.cc:654] T 32147c6d2d5a477aa38c597c7f78df5b P 0172fed22b6b4890a7fdc953fa128af4: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:52.710302 11149 raft_consensus.cc:359] T d371c5e27bd843469d2e2e453734cb4a P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:52.711089 11149 raft_consensus.cc:385] T d371c5e27bd843469d2e2e453734cb4a P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:52.711366 11149 raft_consensus.cc:740] T d371c5e27bd843469d2e2e453734cb4a P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 873357742d5843579556ee8166f23f47, State: Initialized, Role: FOLLOWER
I20260709 13:03:52.712142 11149 consensus_queue.cc:260] T d371c5e27bd843469d2e2e453734cb4a P 873357742d5843579556ee8166f23f47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:52.712646 11148 ts_tablet_manager.cc:1434] T 5de5f0581e3b41dea729363d6ca563bb P 10ca5ee68ad64b3283b9d235352c9d23: Time spent starting tablet: real 0.045s	user 0.007s	sys 0.000s
I20260709 13:03:52.713780 11148 tablet_bootstrap.cc:492] T e5a5f5a608504650b48fd27f163faed6 P 10ca5ee68ad64b3283b9d235352c9d23: Bootstrap starting.
I20260709 13:03:52.714727 11149 ts_tablet_manager.cc:1434] T d371c5e27bd843469d2e2e453734cb4a P 873357742d5843579556ee8166f23f47: Time spent starting tablet: real 0.007s	user 0.008s	sys 0.000s
I20260709 13:03:52.716634 11151 tablet_bootstrap.cc:654] T 5de5f0581e3b41dea729363d6ca563bb P 186b12638a87410487467df721a3cbda: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:52.720825 11427 consensus_queue.cc:1048] T b4cd3d7dfbbf4096908ff11e9dc57141 P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:03:52.727595 11363 raft_consensus.cc:493] T 87cd8f5097624c8692f1d83544d8881e P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:52.728235 11363 raft_consensus.cc:515] T 87cd8f5097624c8692f1d83544d8881e P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:52.730386 11428 raft_consensus.cc:493] T 11de93cd9b9d457489ff1cb80f976c9a P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:52.730971 11428 raft_consensus.cc:515] T 11de93cd9b9d457489ff1cb80f976c9a P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:52.732695 11103 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87cd8f5097624c8692f1d83544d8881e" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:03:52.734951 10878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11de93cd9b9d457489ff1cb80f976c9a" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:03:52.735613 10878 raft_consensus.cc:2468] T 11de93cd9b9d457489ff1cb80f976c9a P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 0.
I20260709 13:03:52.734977 10955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11de93cd9b9d457489ff1cb80f976c9a" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:03:52.736902 10955 raft_consensus.cc:2468] T 11de93cd9b9d457489ff1cb80f976c9a P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 0.
W20260709 13:03:52.737432 10920 leader_election.cc:343] T 87cd8f5097624c8692f1d83544d8881e P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:03:52.738734 10992 leader_election.cc:304] T 11de93cd9b9d457489ff1cb80f976c9a P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:03:52.739714 11429 raft_consensus.cc:2804] T 11de93cd9b9d457489ff1cb80f976c9a P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:03:52.740235 11429 raft_consensus.cc:493] T 11de93cd9b9d457489ff1cb80f976c9a P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:52.740665 11429 raft_consensus.cc:3060] T 11de93cd9b9d457489ff1cb80f976c9a P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:03:52.743844 11428 leader_election.cc:290] T 11de93cd9b9d457489ff1cb80f976c9a P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:03:52.749517 11363 leader_election.cc:290] T 87cd8f5097624c8692f1d83544d8881e P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:03:52.751739 10878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87cd8f5097624c8692f1d83544d8881e" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:03:52.752573 11429 raft_consensus.cc:515] T 11de93cd9b9d457489ff1cb80f976c9a P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
W20260709 13:03:52.755194 10921 leader_election.cc:343] T 87cd8f5097624c8692f1d83544d8881e P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 186b12638a87410487467df721a3cbda (127.0.107.193:38967): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:03:52.758028 10921 leader_election.cc:304] T 87cd8f5097624c8692f1d83544d8881e P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23; no voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda
I20260709 13:03:52.759177 11363 raft_consensus.cc:2749] T 87cd8f5097624c8692f1d83544d8881e P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:03:52.757467 10878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11de93cd9b9d457489ff1cb80f976c9a" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:03:52.759800 10878 raft_consensus.cc:3060] T 11de93cd9b9d457489ff1cb80f976c9a P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:03:52.759999 11429 leader_election.cc:290] T 11de93cd9b9d457489ff1cb80f976c9a P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 1 election: Requested vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:03:52.768869 11429 raft_consensus.cc:493] T d371c5e27bd843469d2e2e453734cb4a P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:52.769572 11429 raft_consensus.cc:515] T d371c5e27bd843469d2e2e453734cb4a P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:52.770270 10955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11de93cd9b9d457489ff1cb80f976c9a" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:03:52.771164 10955 raft_consensus.cc:3060] T 11de93cd9b9d457489ff1cb80f976c9a P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:03:52.774793 11102 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d371c5e27bd843469d2e2e453734cb4a" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
W20260709 13:03:52.776140 10994 leader_election.cc:343] T d371c5e27bd843469d2e2e453734cb4a P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:03:52.779330 11151 tablet_bootstrap.cc:492] T 5de5f0581e3b41dea729363d6ca563bb P 186b12638a87410487467df721a3cbda: No bootstrap required, opened a new log
I20260709 13:03:52.780015 11151 ts_tablet_manager.cc:1403] T 5de5f0581e3b41dea729363d6ca563bb P 186b12638a87410487467df721a3cbda: Time spent bootstrapping tablet: real 0.090s	user 0.021s	sys 0.018s
I20260709 13:03:52.782868 10775 catalog_manager.cc:5697] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 0172fed22b6b4890a7fdc953fa128af4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196). New cstate: current_term: 1 leader_uuid: "0172fed22b6b4890a7fdc953fa128af4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } health_report { overall_health: UNKNOWN } } }
I20260709 13:03:52.784246 11429 leader_election.cc:290] T d371c5e27bd843469d2e2e453734cb4a P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:03:52.783684 11151 raft_consensus.cc:359] T 5de5f0581e3b41dea729363d6ca563bb P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:52.785331 11151 raft_consensus.cc:385] T 5de5f0581e3b41dea729363d6ca563bb P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:52.785650 11151 raft_consensus.cc:740] T 5de5f0581e3b41dea729363d6ca563bb P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 186b12638a87410487467df721a3cbda, State: Initialized, Role: FOLLOWER
I20260709 13:03:52.785799 10876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d371c5e27bd843469d2e2e453734cb4a" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:03:52.786762 11151 consensus_queue.cc:260] T 5de5f0581e3b41dea729363d6ca563bb P 186b12638a87410487467df721a3cbda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:52.789546 11151 ts_tablet_manager.cc:1434] T 5de5f0581e3b41dea729363d6ca563bb P 186b12638a87410487467df721a3cbda: Time spent starting tablet: real 0.009s	user 0.007s	sys 0.001s
I20260709 13:03:52.792870 11151 tablet_bootstrap.cc:492] T 87cd8f5097624c8692f1d83544d8881e P 186b12638a87410487467df721a3cbda: Bootstrap starting.
I20260709 13:03:52.794415 11152 tablet_bootstrap.cc:492] T 32147c6d2d5a477aa38c597c7f78df5b P 0172fed22b6b4890a7fdc953fa128af4: No bootstrap required, opened a new log
W20260709 13:03:52.790401 10995 leader_election.cc:343] T d371c5e27bd843469d2e2e453734cb4a P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 186b12638a87410487467df721a3cbda (127.0.107.193:38967): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:03:52.800294 11149 tablet_bootstrap.cc:492] T 732c4adeeff6422497aa7db4a82f535b P 873357742d5843579556ee8166f23f47: Bootstrap starting.
I20260709 13:03:52.806804 10995 leader_election.cc:304] T d371c5e27bd843469d2e2e453734cb4a P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 873357742d5843579556ee8166f23f47; no voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda
I20260709 13:03:52.814020 11152 ts_tablet_manager.cc:1403] T 32147c6d2d5a477aa38c597c7f78df5b P 0172fed22b6b4890a7fdc953fa128af4: Time spent bootstrapping tablet: real 0.153s	user 0.026s	sys 0.017s
I20260709 13:03:52.808619 11311 raft_consensus.cc:493] T 5de5f0581e3b41dea729363d6ca563bb P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:52.816067 11311 raft_consensus.cc:515] T 5de5f0581e3b41dea729363d6ca563bb P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:52.817628 11152 raft_consensus.cc:359] T 32147c6d2d5a477aa38c597c7f78df5b P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:52.819602 10880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5de5f0581e3b41dea729363d6ca563bb" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:03:52.818382 11311 leader_election.cc:290] T 5de5f0581e3b41dea729363d6ca563bb P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:03:52.820405 11151 tablet_bootstrap.cc:654] T 87cd8f5097624c8692f1d83544d8881e P 186b12638a87410487467df721a3cbda: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:52.820611 10951 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5de5f0581e3b41dea729363d6ca563bb" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:03:52.813143 10878 raft_consensus.cc:2468] T 11de93cd9b9d457489ff1cb80f976c9a P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 1.
I20260709 13:03:52.821046 10880 raft_consensus.cc:2468] T 5de5f0581e3b41dea729363d6ca563bb P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 0.
I20260709 13:03:52.824327 10951 raft_consensus.cc:2468] T 5de5f0581e3b41dea729363d6ca563bb P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 0.
I20260709 13:03:52.736925 11148 tablet_bootstrap.cc:654] T e5a5f5a608504650b48fd27f163faed6 P 10ca5ee68ad64b3283b9d235352c9d23: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:52.825377 11069 leader_election.cc:304] T 5de5f0581e3b41dea729363d6ca563bb P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda; no voters: 
I20260709 13:03:52.819868 11429 raft_consensus.cc:2749] T d371c5e27bd843469d2e2e453734cb4a P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:03:52.820731 11152 raft_consensus.cc:385] T 32147c6d2d5a477aa38c597c7f78df5b P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:52.826087 10995 leader_election.cc:304] T 11de93cd9b9d457489ff1cb80f976c9a P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:03:52.827056 11152 raft_consensus.cc:740] T 32147c6d2d5a477aa38c597c7f78df5b P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0172fed22b6b4890a7fdc953fa128af4, State: Initialized, Role: FOLLOWER
I20260709 13:03:52.827710 11429 raft_consensus.cc:2804] T 11de93cd9b9d457489ff1cb80f976c9a P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:03:52.828179 11162 raft_consensus.cc:2804] T 5de5f0581e3b41dea729363d6ca563bb P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:03:52.827916 11152 consensus_queue.cc:260] T 32147c6d2d5a477aa38c597c7f78df5b P 0172fed22b6b4890a7fdc953fa128af4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:52.828474 11429 raft_consensus.cc:697] T 11de93cd9b9d457489ff1cb80f976c9a P 873357742d5843579556ee8166f23f47 [term 1 LEADER]: Becoming Leader. State: Replica: 873357742d5843579556ee8166f23f47, State: Running, Role: LEADER
I20260709 13:03:52.828627 11162 raft_consensus.cc:493] T 5de5f0581e3b41dea729363d6ca563bb P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:52.829092 11162 raft_consensus.cc:3060] T 5de5f0581e3b41dea729363d6ca563bb P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:03:52.829531 11429 consensus_queue.cc:237] T 11de93cd9b9d457489ff1cb80f976c9a P 873357742d5843579556ee8166f23f47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:52.832527 11152 ts_tablet_manager.cc:1434] T 32147c6d2d5a477aa38c597c7f78df5b P 0172fed22b6b4890a7fdc953fa128af4: Time spent starting tablet: real 0.018s	user 0.004s	sys 0.004s
I20260709 13:03:52.827251 11425 consensus_queue.cc:1048] T b4cd3d7dfbbf4096908ff11e9dc57141 P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:03:52.835446 11152 tablet_bootstrap.cc:492] T 0e5d327630e5438b878035d28efe6c0c P 0172fed22b6b4890a7fdc953fa128af4: Bootstrap starting.
I20260709 13:03:52.861029 11151 tablet_bootstrap.cc:492] T 87cd8f5097624c8692f1d83544d8881e P 186b12638a87410487467df721a3cbda: No bootstrap required, opened a new log
I20260709 13:03:52.861675 11151 ts_tablet_manager.cc:1403] T 87cd8f5097624c8692f1d83544d8881e P 186b12638a87410487467df721a3cbda: Time spent bootstrapping tablet: real 0.069s	user 0.006s	sys 0.006s
I20260709 13:03:52.805992 10955 raft_consensus.cc:2468] T 11de93cd9b9d457489ff1cb80f976c9a P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 1.
I20260709 13:03:52.867751 11151 raft_consensus.cc:359] T 87cd8f5097624c8692f1d83544d8881e P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:52.868378 11151 raft_consensus.cc:385] T 87cd8f5097624c8692f1d83544d8881e P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:52.868571 11149 tablet_bootstrap.cc:654] T 732c4adeeff6422497aa7db4a82f535b P 873357742d5843579556ee8166f23f47: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:52.868639 11151 raft_consensus.cc:740] T 87cd8f5097624c8692f1d83544d8881e P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 186b12638a87410487467df721a3cbda, State: Initialized, Role: FOLLOWER
I20260709 13:03:52.869786 11151 consensus_queue.cc:260] T 87cd8f5097624c8692f1d83544d8881e P 186b12638a87410487467df721a3cbda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:52.872668 11151 ts_tablet_manager.cc:1434] T 87cd8f5097624c8692f1d83544d8881e P 186b12638a87410487467df721a3cbda: Time spent starting tablet: real 0.011s	user 0.011s	sys 0.001s
I20260709 13:03:52.872855 11152 tablet_bootstrap.cc:654] T 0e5d327630e5438b878035d28efe6c0c P 0172fed22b6b4890a7fdc953fa128af4: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:52.873518 11148 tablet_bootstrap.cc:492] T e5a5f5a608504650b48fd27f163faed6 P 10ca5ee68ad64b3283b9d235352c9d23: No bootstrap required, opened a new log
I20260709 13:03:52.874097 11151 tablet_bootstrap.cc:492] T 0e5d327630e5438b878035d28efe6c0c P 186b12638a87410487467df721a3cbda: Bootstrap starting.
I20260709 13:03:52.881122 11162 raft_consensus.cc:515] T 5de5f0581e3b41dea729363d6ca563bb P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:52.884482 10878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5de5f0581e3b41dea729363d6ca563bb" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:03:52.885187 10878 raft_consensus.cc:3060] T 5de5f0581e3b41dea729363d6ca563bb P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:03:52.886427 11152 tablet_bootstrap.cc:492] T 0e5d327630e5438b878035d28efe6c0c P 0172fed22b6b4890a7fdc953fa128af4: No bootstrap required, opened a new log
I20260709 13:03:52.887001 11152 ts_tablet_manager.cc:1403] T 0e5d327630e5438b878035d28efe6c0c P 0172fed22b6b4890a7fdc953fa128af4: Time spent bootstrapping tablet: real 0.052s	user 0.017s	sys 0.003s
I20260709 13:03:52.874063 11148 ts_tablet_manager.cc:1403] T e5a5f5a608504650b48fd27f163faed6 P 10ca5ee68ad64b3283b9d235352c9d23: Time spent bootstrapping tablet: real 0.161s	user 0.013s	sys 0.000s
I20260709 13:03:52.890156 11152 raft_consensus.cc:359] T 0e5d327630e5438b878035d28efe6c0c P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:52.890996 11152 raft_consensus.cc:385] T 0e5d327630e5438b878035d28efe6c0c P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:52.891276 11152 raft_consensus.cc:740] T 0e5d327630e5438b878035d28efe6c0c P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0172fed22b6b4890a7fdc953fa128af4, State: Initialized, Role: FOLLOWER
I20260709 13:03:52.895140 10878 raft_consensus.cc:2468] T 5de5f0581e3b41dea729363d6ca563bb P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 1.
I20260709 13:03:52.895340 11151 tablet_bootstrap.cc:654] T 0e5d327630e5438b878035d28efe6c0c P 186b12638a87410487467df721a3cbda: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:52.892091 11152 consensus_queue.cc:260] T 0e5d327630e5438b878035d28efe6c0c P 0172fed22b6b4890a7fdc953fa128af4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:52.901964 11069 leader_election.cc:304] T 5de5f0581e3b41dea729363d6ca563bb P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda; no voters: 
I20260709 13:03:52.897329 11162 leader_election.cc:290] T 5de5f0581e3b41dea729363d6ca563bb P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 1 election: Requested vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:03:52.902807 11162 raft_consensus.cc:2804] T 5de5f0581e3b41dea729363d6ca563bb P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:03:52.898056 11148 raft_consensus.cc:359] T e5a5f5a608504650b48fd27f163faed6 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:52.898617 10955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5de5f0581e3b41dea729363d6ca563bb" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:03:52.904141 10955 raft_consensus.cc:3060] T 5de5f0581e3b41dea729363d6ca563bb P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:03:52.903333 11162 raft_consensus.cc:697] T 5de5f0581e3b41dea729363d6ca563bb P 0172fed22b6b4890a7fdc953fa128af4 [term 1 LEADER]: Becoming Leader. State: Replica: 0172fed22b6b4890a7fdc953fa128af4, State: Running, Role: LEADER
I20260709 13:03:52.906934 11162 consensus_queue.cc:237] T 5de5f0581e3b41dea729363d6ca563bb P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:52.913946 10775 catalog_manager.cc:5697] T 11de93cd9b9d457489ff1cb80f976c9a P 873357742d5843579556ee8166f23f47 reported cstate change: term changed from 0 to 1, leader changed from <none> to 873357742d5843579556ee8166f23f47 (127.0.107.195). New cstate: current_term: 1 leader_uuid: "873357742d5843579556ee8166f23f47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } health_report { overall_health: HEALTHY } } }
I20260709 13:03:52.902355 11152 ts_tablet_manager.cc:1434] T 0e5d327630e5438b878035d28efe6c0c P 0172fed22b6b4890a7fdc953fa128af4: Time spent starting tablet: real 0.015s	user 0.006s	sys 0.000s
I20260709 13:03:52.899945 11149 tablet_bootstrap.cc:492] T 732c4adeeff6422497aa7db4a82f535b P 873357742d5843579556ee8166f23f47: No bootstrap required, opened a new log
I20260709 13:03:52.920639 11149 ts_tablet_manager.cc:1403] T 732c4adeeff6422497aa7db4a82f535b P 873357742d5843579556ee8166f23f47: Time spent bootstrapping tablet: real 0.121s	user 0.013s	sys 0.000s
I20260709 13:03:52.921060 11152 tablet_bootstrap.cc:492] T bb7d67be500947b3aef4c4245e33671c P 0172fed22b6b4890a7fdc953fa128af4: Bootstrap starting.
I20260709 13:03:52.922597 11148 raft_consensus.cc:385] T e5a5f5a608504650b48fd27f163faed6 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:52.922947 11148 raft_consensus.cc:740] T e5a5f5a608504650b48fd27f163faed6 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10ca5ee68ad64b3283b9d235352c9d23, State: Initialized, Role: FOLLOWER
I20260709 13:03:52.924302 11149 raft_consensus.cc:359] T 732c4adeeff6422497aa7db4a82f535b P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:52.925144 11149 raft_consensus.cc:385] T 732c4adeeff6422497aa7db4a82f535b P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:52.926839 10955 raft_consensus.cc:2468] T 5de5f0581e3b41dea729363d6ca563bb P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 1.
I20260709 13:03:52.929149 11152 tablet_bootstrap.cc:654] T bb7d67be500947b3aef4c4245e33671c P 0172fed22b6b4890a7fdc953fa128af4: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:52.925503 11149 raft_consensus.cc:740] T 732c4adeeff6422497aa7db4a82f535b P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 873357742d5843579556ee8166f23f47, State: Initialized, Role: FOLLOWER
I20260709 13:03:52.930052 11149 consensus_queue.cc:260] T 732c4adeeff6422497aa7db4a82f535b P 873357742d5843579556ee8166f23f47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:52.933445 11149 ts_tablet_manager.cc:1434] T 732c4adeeff6422497aa7db4a82f535b P 873357742d5843579556ee8166f23f47: Time spent starting tablet: real 0.012s	user 0.006s	sys 0.000s
I20260709 13:03:52.933266 11148 consensus_queue.cc:260] T e5a5f5a608504650b48fd27f163faed6 P 10ca5ee68ad64b3283b9d235352c9d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:52.935065 11149 tablet_bootstrap.cc:492] T bb7d67be500947b3aef4c4245e33671c P 873357742d5843579556ee8166f23f47: Bootstrap starting.
I20260709 13:03:52.937556 11148 ts_tablet_manager.cc:1434] T e5a5f5a608504650b48fd27f163faed6 P 10ca5ee68ad64b3283b9d235352c9d23: Time spent starting tablet: real 0.047s	user 0.007s	sys 0.000s
I20260709 13:03:52.940384 11148 tablet_bootstrap.cc:492] T 32147c6d2d5a477aa38c597c7f78df5b P 10ca5ee68ad64b3283b9d235352c9d23: Bootstrap starting.
I20260709 13:03:52.945282 11149 tablet_bootstrap.cc:654] T bb7d67be500947b3aef4c4245e33671c P 873357742d5843579556ee8166f23f47: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:52.948875 11148 tablet_bootstrap.cc:654] T 32147c6d2d5a477aa38c597c7f78df5b P 10ca5ee68ad64b3283b9d235352c9d23: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:52.951829 10778 catalog_manager.cc:5697] T 5de5f0581e3b41dea729363d6ca563bb P 0172fed22b6b4890a7fdc953fa128af4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196). New cstate: current_term: 1 leader_uuid: "0172fed22b6b4890a7fdc953fa128af4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } health_report { overall_health: UNKNOWN } } }
I20260709 13:03:52.964614 11151 tablet_bootstrap.cc:492] T 0e5d327630e5438b878035d28efe6c0c P 186b12638a87410487467df721a3cbda: No bootstrap required, opened a new log
I20260709 13:03:52.965337 11151 ts_tablet_manager.cc:1403] T 0e5d327630e5438b878035d28efe6c0c P 186b12638a87410487467df721a3cbda: Time spent bootstrapping tablet: real 0.092s	user 0.010s	sys 0.024s
I20260709 13:03:52.973433 11151 raft_consensus.cc:359] T 0e5d327630e5438b878035d28efe6c0c P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:52.974261 11151 raft_consensus.cc:385] T 0e5d327630e5438b878035d28efe6c0c P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:52.974603 11151 raft_consensus.cc:740] T 0e5d327630e5438b878035d28efe6c0c P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 186b12638a87410487467df721a3cbda, State: Initialized, Role: FOLLOWER
I20260709 13:03:52.975399 11151 consensus_queue.cc:260] T 0e5d327630e5438b878035d28efe6c0c P 186b12638a87410487467df721a3cbda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:52.978402 11151 ts_tablet_manager.cc:1434] T 0e5d327630e5438b878035d28efe6c0c P 186b12638a87410487467df721a3cbda: Time spent starting tablet: real 0.013s	user 0.009s	sys 0.000s
I20260709 13:03:52.981269 11151 tablet_bootstrap.cc:492] T e5a5f5a608504650b48fd27f163faed6 P 186b12638a87410487467df721a3cbda: Bootstrap starting.
I20260709 13:03:52.991360 11151 tablet_bootstrap.cc:654] T e5a5f5a608504650b48fd27f163faed6 P 186b12638a87410487467df721a3cbda: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:52.994513 11162 raft_consensus.cc:493] T 0e5d327630e5438b878035d28efe6c0c P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:52.995044 11162 raft_consensus.cc:515] T 0e5d327630e5438b878035d28efe6c0c P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:52.996881 11162 leader_election.cc:290] T 0e5d327630e5438b878035d28efe6c0c P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:03:52.998507 10876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e5d327630e5438b878035d28efe6c0c" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:03:52.999235 10876 raft_consensus.cc:2468] T 0e5d327630e5438b878035d28efe6c0c P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 0.
I20260709 13:03:53.000680 11069 leader_election.cc:304] T 0e5d327630e5438b878035d28efe6c0c P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda; no voters: 
I20260709 13:03:53.001461 11162 raft_consensus.cc:2804] T 0e5d327630e5438b878035d28efe6c0c P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:03:53.001406 11360 raft_consensus.cc:493] T e5a5f5a608504650b48fd27f163faed6 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:53.002128 11162 raft_consensus.cc:493] T 0e5d327630e5438b878035d28efe6c0c P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:53.002511 11162 raft_consensus.cc:3060] T 0e5d327630e5438b878035d28efe6c0c P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:03:53.002389 11360 raft_consensus.cc:515] T e5a5f5a608504650b48fd27f163faed6 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:53.006137 11102 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5a5f5a608504650b48fd27f163faed6" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:03:53.006973 10876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5a5f5a608504650b48fd27f163faed6" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
W20260709 13:03:53.007627 10920 leader_election.cc:343] T e5a5f5a608504650b48fd27f163faed6 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:03:53.009524 11025 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e5d327630e5438b878035d28efe6c0c" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:03:53.009680 11360 leader_election.cc:290] T e5a5f5a608504650b48fd27f163faed6 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:03:53.010222 11025 raft_consensus.cc:2468] T 0e5d327630e5438b878035d28efe6c0c P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 0.
I20260709 13:03:53.010485 11162 raft_consensus.cc:515] T 0e5d327630e5438b878035d28efe6c0c P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
W20260709 13:03:53.009557 10921 leader_election.cc:343] T e5a5f5a608504650b48fd27f163faed6 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 186b12638a87410487467df721a3cbda (127.0.107.193:38967): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:03:53.018635 10921 leader_election.cc:304] T e5a5f5a608504650b48fd27f163faed6 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23; no voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda
I20260709 13:03:53.024387 11149 tablet_bootstrap.cc:492] T bb7d67be500947b3aef4c4245e33671c P 873357742d5843579556ee8166f23f47: No bootstrap required, opened a new log
I20260709 13:03:53.024703 11148 tablet_bootstrap.cc:492] T 32147c6d2d5a477aa38c597c7f78df5b P 10ca5ee68ad64b3283b9d235352c9d23: No bootstrap required, opened a new log
I20260709 13:03:53.024999 11149 ts_tablet_manager.cc:1403] T bb7d67be500947b3aef4c4245e33671c P 873357742d5843579556ee8166f23f47: Time spent bootstrapping tablet: real 0.090s	user 0.016s	sys 0.000s
I20260709 13:03:53.025243 11148 ts_tablet_manager.cc:1403] T 32147c6d2d5a477aa38c597c7f78df5b P 10ca5ee68ad64b3283b9d235352c9d23: Time spent bootstrapping tablet: real 0.085s	user 0.014s	sys 0.001s
I20260709 13:03:53.027657 11360 raft_consensus.cc:2749] T e5a5f5a608504650b48fd27f163faed6 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:03:53.028841 11148 raft_consensus.cc:359] T 32147c6d2d5a477aa38c597c7f78df5b P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:53.029654 11148 raft_consensus.cc:385] T 32147c6d2d5a477aa38c597c7f78df5b P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:53.029928 11148 raft_consensus.cc:740] T 32147c6d2d5a477aa38c597c7f78df5b P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10ca5ee68ad64b3283b9d235352c9d23, State: Initialized, Role: FOLLOWER
I20260709 13:03:53.032727 11151 tablet_bootstrap.cc:492] T e5a5f5a608504650b48fd27f163faed6 P 186b12638a87410487467df721a3cbda: No bootstrap required, opened a new log
I20260709 13:03:53.033300 11151 ts_tablet_manager.cc:1403] T e5a5f5a608504650b48fd27f163faed6 P 186b12638a87410487467df721a3cbda: Time spent bootstrapping tablet: real 0.052s	user 0.013s	sys 0.003s
I20260709 13:03:53.012542 11162 leader_election.cc:290] T 0e5d327630e5438b878035d28efe6c0c P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 1 election: Requested vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:03:53.015946 11152 tablet_bootstrap.cc:492] T bb7d67be500947b3aef4c4245e33671c P 0172fed22b6b4890a7fdc953fa128af4: No bootstrap required, opened a new log
I20260709 13:03:53.039840 11152 ts_tablet_manager.cc:1403] T bb7d67be500947b3aef4c4245e33671c P 0172fed22b6b4890a7fdc953fa128af4: Time spent bootstrapping tablet: real 0.119s	user 0.029s	sys 0.043s
I20260709 13:03:53.040582 11151 raft_consensus.cc:359] T e5a5f5a608504650b48fd27f163faed6 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:53.041421 11151 raft_consensus.cc:385] T e5a5f5a608504650b48fd27f163faed6 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:53.041682 11151 raft_consensus.cc:740] T e5a5f5a608504650b48fd27f163faed6 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 186b12638a87410487467df721a3cbda, State: Initialized, Role: FOLLOWER
I20260709 13:03:53.042574 11152 raft_consensus.cc:359] T bb7d67be500947b3aef4c4245e33671c P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:53.017849 11025 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e5d327630e5438b878035d28efe6c0c" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
I20260709 13:03:53.043220 11152 raft_consensus.cc:385] T bb7d67be500947b3aef4c4245e33671c P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:53.043548 11152 raft_consensus.cc:740] T bb7d67be500947b3aef4c4245e33671c P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0172fed22b6b4890a7fdc953fa128af4, State: Initialized, Role: FOLLOWER
I20260709 13:03:53.043447 11025 raft_consensus.cc:3060] T 0e5d327630e5438b878035d28efe6c0c P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:03:53.016149 10876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e5d327630e5438b878035d28efe6c0c" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:03:53.043399 11148 consensus_queue.cc:260] T 32147c6d2d5a477aa38c597c7f78df5b P 10ca5ee68ad64b3283b9d235352c9d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:53.044725 10876 raft_consensus.cc:3060] T 0e5d327630e5438b878035d28efe6c0c P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:03:53.046011 11149 raft_consensus.cc:359] T bb7d67be500947b3aef4c4245e33671c P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:53.046762 11149 raft_consensus.cc:385] T bb7d67be500947b3aef4c4245e33671c P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:53.046916 11152 consensus_queue.cc:260] T bb7d67be500947b3aef4c4245e33671c P 0172fed22b6b4890a7fdc953fa128af4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:53.049523 11152 ts_tablet_manager.cc:1434] T bb7d67be500947b3aef4c4245e33671c P 0172fed22b6b4890a7fdc953fa128af4: Time spent starting tablet: real 0.009s	user 0.005s	sys 0.000s
I20260709 13:03:53.052486 11025 raft_consensus.cc:2468] T 0e5d327630e5438b878035d28efe6c0c P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 1.
I20260709 13:03:53.052896 11148 ts_tablet_manager.cc:1434] T 32147c6d2d5a477aa38c597c7f78df5b P 10ca5ee68ad64b3283b9d235352c9d23: Time spent starting tablet: real 0.027s	user 0.007s	sys 0.002s
I20260709 13:03:53.053767 11067 leader_election.cc:304] T 0e5d327630e5438b878035d28efe6c0c P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:03:53.054628 11152 tablet_bootstrap.cc:492] T 87cd8f5097624c8692f1d83544d8881e P 0172fed22b6b4890a7fdc953fa128af4: Bootstrap starting.
I20260709 13:03:53.054638 11149 raft_consensus.cc:740] T bb7d67be500947b3aef4c4245e33671c P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 873357742d5843579556ee8166f23f47, State: Initialized, Role: FOLLOWER
I20260709 13:03:53.055083 11311 raft_consensus.cc:2804] T 0e5d327630e5438b878035d28efe6c0c P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:03:53.055603 11311 raft_consensus.cc:697] T 0e5d327630e5438b878035d28efe6c0c P 0172fed22b6b4890a7fdc953fa128af4 [term 1 LEADER]: Becoming Leader. State: Replica: 0172fed22b6b4890a7fdc953fa128af4, State: Running, Role: LEADER
I20260709 13:03:53.056020 11148 tablet_bootstrap.cc:492] T bb7d67be500947b3aef4c4245e33671c P 10ca5ee68ad64b3283b9d235352c9d23: Bootstrap starting.
I20260709 13:03:53.056489 11311 consensus_queue.cc:237] T 0e5d327630e5438b878035d28efe6c0c P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:53.058727 11149 consensus_queue.cc:260] T bb7d67be500947b3aef4c4245e33671c P 873357742d5843579556ee8166f23f47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:53.061594 11149 ts_tablet_manager.cc:1434] T bb7d67be500947b3aef4c4245e33671c P 873357742d5843579556ee8166f23f47: Time spent starting tablet: real 0.036s	user 0.009s	sys 0.000s
I20260709 13:03:53.071173 11149 tablet_bootstrap.cc:492] T f371d2565dd84667b5613848604949bb P 873357742d5843579556ee8166f23f47: Bootstrap starting.
I20260709 13:03:53.076645 11148 tablet_bootstrap.cc:654] T bb7d67be500947b3aef4c4245e33671c P 10ca5ee68ad64b3283b9d235352c9d23: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:53.042426 11151 consensus_queue.cc:260] T e5a5f5a608504650b48fd27f163faed6 P 186b12638a87410487467df721a3cbda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:53.079236 11151 ts_tablet_manager.cc:1434] T e5a5f5a608504650b48fd27f163faed6 P 186b12638a87410487467df721a3cbda: Time spent starting tablet: real 0.046s	user 0.007s	sys 0.001s
I20260709 13:03:53.080212 11151 tablet_bootstrap.cc:492] T d371c5e27bd843469d2e2e453734cb4a P 186b12638a87410487467df721a3cbda: Bootstrap starting.
I20260709 13:03:53.083990 11149 tablet_bootstrap.cc:654] T f371d2565dd84667b5613848604949bb P 873357742d5843579556ee8166f23f47: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:53.086481 10876 raft_consensus.cc:2468] T 0e5d327630e5438b878035d28efe6c0c P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 1.
I20260709 13:03:53.087491 11151 tablet_bootstrap.cc:654] T d371c5e27bd843469d2e2e453734cb4a P 186b12638a87410487467df721a3cbda: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:53.093436 11162 raft_consensus.cc:493] T 32147c6d2d5a477aa38c597c7f78df5b P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:53.094226 11360 consensus_queue.cc:1048] T 65bf0a62f4d84ef18dd6b23e082724b8 P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:03:53.094179 11162 raft_consensus.cc:515] T 32147c6d2d5a477aa38c597c7f78df5b P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:53.097744 11162 leader_election.cc:290] T 32147c6d2d5a477aa38c597c7f78df5b P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:03:53.098915 11152 tablet_bootstrap.cc:654] T 87cd8f5097624c8692f1d83544d8881e P 0172fed22b6b4890a7fdc953fa128af4: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:53.099256 11029 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32147c6d2d5a477aa38c597c7f78df5b" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:03:53.099565 11151 tablet_bootstrap.cc:492] T d371c5e27bd843469d2e2e453734cb4a P 186b12638a87410487467df721a3cbda: No bootstrap required, opened a new log
I20260709 13:03:53.100039 11029 raft_consensus.cc:2468] T 32147c6d2d5a477aa38c597c7f78df5b P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 0.
I20260709 13:03:53.100296 11151 ts_tablet_manager.cc:1403] T d371c5e27bd843469d2e2e453734cb4a P 186b12638a87410487467df721a3cbda: Time spent bootstrapping tablet: real 0.020s	user 0.016s	sys 0.002s
I20260709 13:03:53.101322 11067 leader_election.cc:304] T 32147c6d2d5a477aa38c597c7f78df5b P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:03:53.102103 10951 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32147c6d2d5a477aa38c597c7f78df5b" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:03:53.102878 10951 raft_consensus.cc:2468] T 32147c6d2d5a477aa38c597c7f78df5b P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 0.
I20260709 13:03:53.104004 11151 raft_consensus.cc:359] T d371c5e27bd843469d2e2e453734cb4a P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:53.105027 11151 raft_consensus.cc:385] T d371c5e27bd843469d2e2e453734cb4a P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:53.105398 11151 raft_consensus.cc:740] T d371c5e27bd843469d2e2e453734cb4a P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 186b12638a87410487467df721a3cbda, State: Initialized, Role: FOLLOWER
I20260709 13:03:53.102257 11162 raft_consensus.cc:2804] T 32147c6d2d5a477aa38c597c7f78df5b P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:03:53.107967 11162 raft_consensus.cc:493] T 32147c6d2d5a477aa38c597c7f78df5b P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:53.108325 11162 raft_consensus.cc:3060] T 32147c6d2d5a477aa38c597c7f78df5b P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:03:53.117205 11357 consensus_queue.cc:1048] T 65bf0a62f4d84ef18dd6b23e082724b8 P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:03:53.122669 11151 consensus_queue.cc:260] T d371c5e27bd843469d2e2e453734cb4a P 186b12638a87410487467df721a3cbda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:53.125099 11162 raft_consensus.cc:515] T 32147c6d2d5a477aa38c597c7f78df5b P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:53.125764 11151 ts_tablet_manager.cc:1434] T d371c5e27bd843469d2e2e453734cb4a P 186b12638a87410487467df721a3cbda: Time spent starting tablet: real 0.025s	user 0.007s	sys 0.002s
I20260709 13:03:53.126926 11151 tablet_bootstrap.cc:492] T 732c4adeeff6422497aa7db4a82f535b P 186b12638a87410487467df721a3cbda: Bootstrap starting.
I20260709 13:03:53.133647 11149 tablet_bootstrap.cc:492] T f371d2565dd84667b5613848604949bb P 873357742d5843579556ee8166f23f47: No bootstrap required, opened a new log
I20260709 13:03:53.134279 11149 ts_tablet_manager.cc:1403] T f371d2565dd84667b5613848604949bb P 873357742d5843579556ee8166f23f47: Time spent bootstrapping tablet: real 0.063s	user 0.013s	sys 0.000s
I20260709 13:03:53.137524 10775 catalog_manager.cc:5697] T 0e5d327630e5438b878035d28efe6c0c P 0172fed22b6b4890a7fdc953fa128af4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196). New cstate: current_term: 1 leader_uuid: "0172fed22b6b4890a7fdc953fa128af4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } health_report { overall_health: UNKNOWN } } }
I20260709 13:03:53.140159 10951 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32147c6d2d5a477aa38c597c7f78df5b" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:03:53.140874 10951 raft_consensus.cc:3060] T 32147c6d2d5a477aa38c597c7f78df5b P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:03:53.143086 11152 tablet_bootstrap.cc:492] T 87cd8f5097624c8692f1d83544d8881e P 0172fed22b6b4890a7fdc953fa128af4: No bootstrap required, opened a new log
I20260709 13:03:53.142894 11354 consensus_queue.cc:1048] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:03:53.143621 11152 ts_tablet_manager.cc:1403] T 87cd8f5097624c8692f1d83544d8881e P 0172fed22b6b4890a7fdc953fa128af4: Time spent bootstrapping tablet: real 0.089s	user 0.012s	sys 0.020s
I20260709 13:03:53.149034 11354 raft_consensus.cc:493] T a610d1a71f52432e88fe56130f921fa2 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:53.149506 11354 raft_consensus.cc:515] T a610d1a71f52432e88fe56130f921fa2 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:53.150568 10951 raft_consensus.cc:2468] T 32147c6d2d5a477aa38c597c7f78df5b P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 1.
I20260709 13:03:53.151892 11066 leader_election.cc:304] T 32147c6d2d5a477aa38c597c7f78df5b P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 10ca5ee68ad64b3283b9d235352c9d23; no voters: 
I20260709 13:03:53.144027 11332 raft_consensus.cc:493] T e5a5f5a608504650b48fd27f163faed6 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:53.153036 11027 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a610d1a71f52432e88fe56130f921fa2" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:03:53.153234 11427 raft_consensus.cc:2804] T 32147c6d2d5a477aa38c597c7f78df5b P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:03:53.153779 11427 raft_consensus.cc:697] T 32147c6d2d5a477aa38c597c7f78df5b P 0172fed22b6b4890a7fdc953fa128af4 [term 1 LEADER]: Becoming Leader. State: Replica: 0172fed22b6b4890a7fdc953fa128af4, State: Running, Role: LEADER
I20260709 13:03:53.154846 11427 consensus_queue.cc:237] T 32147c6d2d5a477aa38c597c7f78df5b P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:53.144690 11151 tablet_bootstrap.cc:654] T 732c4adeeff6422497aa7db4a82f535b P 186b12638a87410487467df721a3cbda: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:53.153790 11027 raft_consensus.cc:2468] T a610d1a71f52432e88fe56130f921fa2 P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 1.
I20260709 13:03:53.159116 10876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a610d1a71f52432e88fe56130f921fa2" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:03:53.160044 11067 leader_election.cc:304] T a610d1a71f52432e88fe56130f921fa2 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:03:53.154042 11152 raft_consensus.cc:359] T 87cd8f5097624c8692f1d83544d8881e P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:53.153028 11332 raft_consensus.cc:515] T e5a5f5a608504650b48fd27f163faed6 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:53.161065 11152 raft_consensus.cc:385] T 87cd8f5097624c8692f1d83544d8881e P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:53.161240 11425 raft_consensus.cc:2804] T a610d1a71f52432e88fe56130f921fa2 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:03:53.161453 11152 raft_consensus.cc:740] T 87cd8f5097624c8692f1d83544d8881e P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0172fed22b6b4890a7fdc953fa128af4, State: Initialized, Role: FOLLOWER
I20260709 13:03:53.161748 11425 raft_consensus.cc:493] T a610d1a71f52432e88fe56130f921fa2 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:53.162225 11425 raft_consensus.cc:3060] T a610d1a71f52432e88fe56130f921fa2 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:03:53.159833 10876 raft_consensus.cc:2468] T a610d1a71f52432e88fe56130f921fa2 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 1.
I20260709 13:03:53.163599 11102 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5a5f5a608504650b48fd27f163faed6" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
W20260709 13:03:53.164979 10845 leader_election.cc:343] T e5a5f5a608504650b48fd27f163faed6 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:03:53.162554 11152 consensus_queue.cc:260] T 87cd8f5097624c8692f1d83544d8881e P 0172fed22b6b4890a7fdc953fa128af4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:53.164568 11332 leader_election.cc:290] T e5a5f5a608504650b48fd27f163faed6 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:03:53.167330 10955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5a5f5a608504650b48fd27f163faed6" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:03:53.168047 10955 raft_consensus.cc:2468] T e5a5f5a608504650b48fd27f163faed6 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 0.
I20260709 13:03:53.169005 11152 ts_tablet_manager.cc:1434] T 87cd8f5097624c8692f1d83544d8881e P 0172fed22b6b4890a7fdc953fa128af4: Time spent starting tablet: real 0.023s	user 0.008s	sys 0.000s
I20260709 13:03:53.169152 10843 leader_election.cc:304] T e5a5f5a608504650b48fd27f163faed6 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda; no voters: 0172fed22b6b4890a7fdc953fa128af4
I20260709 13:03:53.170946 11332 raft_consensus.cc:2804] T e5a5f5a608504650b48fd27f163faed6 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:03:53.175622 11332 raft_consensus.cc:493] T e5a5f5a608504650b48fd27f163faed6 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:53.176141 11332 raft_consensus.cc:3060] T e5a5f5a608504650b48fd27f163faed6 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:03:53.174760 11152 tablet_bootstrap.cc:492] T d371c5e27bd843469d2e2e453734cb4a P 0172fed22b6b4890a7fdc953fa128af4: Bootstrap starting.
I20260709 13:03:53.167193 11354 leader_election.cc:290] T a610d1a71f52432e88fe56130f921fa2 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:03:53.214742 11027 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32147c6d2d5a477aa38c597c7f78df5b" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
I20260709 13:03:53.215700 11027 raft_consensus.cc:3060] T 32147c6d2d5a477aa38c597c7f78df5b P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:03:53.222638 11151 tablet_bootstrap.cc:492] T 732c4adeeff6422497aa7db4a82f535b P 186b12638a87410487467df721a3cbda: No bootstrap required, opened a new log
I20260709 13:03:53.222889 11162 leader_election.cc:290] T 32147c6d2d5a477aa38c597c7f78df5b P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 1 election: Requested vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:03:53.223390 11151 ts_tablet_manager.cc:1403] T 732c4adeeff6422497aa7db4a82f535b P 186b12638a87410487467df721a3cbda: Time spent bootstrapping tablet: real 0.097s	user 0.024s	sys 0.037s
I20260709 13:03:53.225884 11152 tablet_bootstrap.cc:654] T d371c5e27bd843469d2e2e453734cb4a P 0172fed22b6b4890a7fdc953fa128af4: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:53.227547 11151 raft_consensus.cc:359] T 732c4adeeff6422497aa7db4a82f535b P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:53.228379 11151 raft_consensus.cc:385] T 732c4adeeff6422497aa7db4a82f535b P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:53.228648 11151 raft_consensus.cc:740] T 732c4adeeff6422497aa7db4a82f535b P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 186b12638a87410487467df721a3cbda, State: Initialized, Role: FOLLOWER
I20260709 13:03:53.229420 11151 consensus_queue.cc:260] T 732c4adeeff6422497aa7db4a82f535b P 186b12638a87410487467df721a3cbda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:53.230453 11332 raft_consensus.cc:515] T e5a5f5a608504650b48fd27f163faed6 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:53.231897 11151 ts_tablet_manager.cc:1434] T 732c4adeeff6422497aa7db4a82f535b P 186b12638a87410487467df721a3cbda: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.000s
I20260709 13:03:53.232626 11027 raft_consensus.cc:2468] T 32147c6d2d5a477aa38c597c7f78df5b P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 1.
I20260709 13:03:53.233111 11151 tablet_bootstrap.cc:492] T f371d2565dd84667b5613848604949bb P 186b12638a87410487467df721a3cbda: Bootstrap starting.
I20260709 13:03:53.234184 11332 leader_election.cc:290] T e5a5f5a608504650b48fd27f163faed6 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 1 election: Requested vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:03:53.234902 11102 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5a5f5a608504650b48fd27f163faed6" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:03:53.236538 10955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5a5f5a608504650b48fd27f163faed6" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
W20260709 13:03:53.236658 10845 leader_election.cc:343] T e5a5f5a608504650b48fd27f163faed6 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:03:53.237238 10955 raft_consensus.cc:3060] T e5a5f5a608504650b48fd27f163faed6 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:03:53.241858 11151 tablet_bootstrap.cc:654] T f371d2565dd84667b5613848604949bb P 186b12638a87410487467df721a3cbda: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:53.246407 11149 raft_consensus.cc:359] T f371d2565dd84667b5613848604949bb P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:53.247216 11149 raft_consensus.cc:385] T f371d2565dd84667b5613848604949bb P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:53.247531 11149 raft_consensus.cc:740] T f371d2565dd84667b5613848604949bb P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 873357742d5843579556ee8166f23f47, State: Initialized, Role: FOLLOWER
I20260709 13:03:53.189867 11425 raft_consensus.cc:515] T a610d1a71f52432e88fe56130f921fa2 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
W20260709 13:03:53.243305 11066 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 64829.3us
I20260709 13:03:53.252291 11148 tablet_bootstrap.cc:492] T bb7d67be500947b3aef4c4245e33671c P 10ca5ee68ad64b3283b9d235352c9d23: No bootstrap required, opened a new log
I20260709 13:03:53.250299 11149 consensus_queue.cc:260] T f371d2565dd84667b5613848604949bb P 873357742d5843579556ee8166f23f47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:53.252897 11148 ts_tablet_manager.cc:1403] T bb7d67be500947b3aef4c4245e33671c P 10ca5ee68ad64b3283b9d235352c9d23: Time spent bootstrapping tablet: real 0.197s	user 0.010s	sys 0.010s
I20260709 13:03:53.252630 11428 raft_consensus.cc:493] T bb7d67be500947b3aef4c4245e33671c P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:53.254182 11428 raft_consensus.cc:515] T bb7d67be500947b3aef4c4245e33671c P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:53.256327 11428 leader_election.cc:290] T bb7d67be500947b3aef4c4245e33671c P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:03:53.257074 10954 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb7d67be500947b3aef4c4245e33671c" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:03:53.249876 11471 raft_consensus.cc:493] T 87cd8f5097624c8692f1d83544d8881e P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:53.260982 11471 raft_consensus.cc:515] T 87cd8f5097624c8692f1d83544d8881e P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:53.263306 11471 leader_election.cc:290] T 87cd8f5097624c8692f1d83544d8881e P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:03:53.264714 10951 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87cd8f5097624c8692f1d83544d8881e" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:03:53.265262 11425 leader_election.cc:290] T a610d1a71f52432e88fe56130f921fa2 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 2 election: Requested vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:03:53.266060 11027 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a610d1a71f52432e88fe56130f921fa2" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
I20260709 13:03:53.266703 11027 raft_consensus.cc:3060] T a610d1a71f52432e88fe56130f921fa2 P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:03:53.270017 11103 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87cd8f5097624c8692f1d83544d8881e" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:03:53.270684 11103 raft_consensus.cc:2468] T 87cd8f5097624c8692f1d83544d8881e P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 0.
I20260709 13:03:53.265440 10951 raft_consensus.cc:2468] T 87cd8f5097624c8692f1d83544d8881e P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 0.
I20260709 13:03:53.271920 10845 leader_election.cc:304] T 87cd8f5097624c8692f1d83544d8881e P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda; no voters: 
I20260709 13:03:53.273344 10878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a610d1a71f52432e88fe56130f921fa2" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:03:53.273532 11148 raft_consensus.cc:359] T bb7d67be500947b3aef4c4245e33671c P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:53.280681 10878 raft_consensus.cc:3060] T a610d1a71f52432e88fe56130f921fa2 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:03:53.281138 11148 raft_consensus.cc:385] T bb7d67be500947b3aef4c4245e33671c P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:53.281579 11148 raft_consensus.cc:740] T bb7d67be500947b3aef4c4245e33671c P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10ca5ee68ad64b3283b9d235352c9d23, State: Initialized, Role: FOLLOWER
I20260709 13:03:53.285660 11471 raft_consensus.cc:2804] T 87cd8f5097624c8692f1d83544d8881e P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:03:53.286096 11471 raft_consensus.cc:493] T 87cd8f5097624c8692f1d83544d8881e P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:53.286458 11471 raft_consensus.cc:3060] T 87cd8f5097624c8692f1d83544d8881e P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:03:53.288435 11429 consensus_queue.cc:1048] T 11de93cd9b9d457489ff1cb80f976c9a P 873357742d5843579556ee8166f23f47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:03:53.302708 11148 consensus_queue.cc:260] T bb7d67be500947b3aef4c4245e33671c P 10ca5ee68ad64b3283b9d235352c9d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:53.304627 10878 raft_consensus.cc:2468] T a610d1a71f52432e88fe56130f921fa2 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 2.
I20260709 13:03:53.310702 11148 ts_tablet_manager.cc:1434] T bb7d67be500947b3aef4c4245e33671c P 10ca5ee68ad64b3283b9d235352c9d23: Time spent starting tablet: real 0.057s	user 0.002s	sys 0.007s
I20260709 13:03:53.278895 10775 catalog_manager.cc:5697] T 32147c6d2d5a477aa38c597c7f78df5b P 0172fed22b6b4890a7fdc953fa128af4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196). New cstate: current_term: 1 leader_uuid: "0172fed22b6b4890a7fdc953fa128af4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } health_report { overall_health: HEALTHY } } }
I20260709 13:03:53.308871 11069 leader_election.cc:304] T a610d1a71f52432e88fe56130f921fa2 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda; no voters: 
W20260709 13:03:53.320271 10994 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 68411.6us
I20260709 13:03:53.307938 10954 raft_consensus.cc:2468] T bb7d67be500947b3aef4c4245e33671c P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 0.
I20260709 13:03:53.319307 11427 raft_consensus.cc:2804] T a610d1a71f52432e88fe56130f921fa2 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:03:53.325670 11148 tablet_bootstrap.cc:492] T 732c4adeeff6422497aa7db4a82f535b P 10ca5ee68ad64b3283b9d235352c9d23: Bootstrap starting.
I20260709 13:03:53.310343 11469 consensus_queue.cc:1048] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.023s
I20260709 13:03:53.325073 11103 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb7d67be500947b3aef4c4245e33671c" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:03:53.333855 11103 raft_consensus.cc:2468] T bb7d67be500947b3aef4c4245e33671c P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 0.
I20260709 13:03:53.334187 11151 tablet_bootstrap.cc:492] T f371d2565dd84667b5613848604949bb P 186b12638a87410487467df721a3cbda: No bootstrap required, opened a new log
I20260709 13:03:53.334757 11151 ts_tablet_manager.cc:1403] T f371d2565dd84667b5613848604949bb P 186b12638a87410487467df721a3cbda: Time spent bootstrapping tablet: real 0.102s	user 0.021s	sys 0.007s
I20260709 13:03:53.340991 11152 tablet_bootstrap.cc:492] T d371c5e27bd843469d2e2e453734cb4a P 0172fed22b6b4890a7fdc953fa128af4: No bootstrap required, opened a new log
I20260709 13:03:53.347661 11151 raft_consensus.cc:359] T f371d2565dd84667b5613848604949bb P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:53.332348 11471 raft_consensus.cc:515] T 87cd8f5097624c8692f1d83544d8881e P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:53.348512 11151 raft_consensus.cc:385] T f371d2565dd84667b5613848604949bb P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:53.348870 11151 raft_consensus.cc:740] T f371d2565dd84667b5613848604949bb P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 186b12638a87410487467df721a3cbda, State: Initialized, Role: FOLLOWER
I20260709 13:03:53.349678 11151 consensus_queue.cc:260] T f371d2565dd84667b5613848604949bb P 186b12638a87410487467df721a3cbda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:53.350732 11152 ts_tablet_manager.cc:1403] T d371c5e27bd843469d2e2e453734cb4a P 0172fed22b6b4890a7fdc953fa128af4: Time spent bootstrapping tablet: real 0.177s	user 0.024s	sys 0.017s
I20260709 13:03:53.353019 11471 leader_election.cc:290] T 87cd8f5097624c8692f1d83544d8881e P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 1 election: Requested vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:03:53.353948 11429 raft_consensus.cc:493] T 732c4adeeff6422497aa7db4a82f535b P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:53.357239 11429 raft_consensus.cc:515] T 732c4adeeff6422497aa7db4a82f535b P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:53.358718 10992 leader_election.cc:304] T bb7d67be500947b3aef4c4245e33671c P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:03:53.360895 11148 tablet_bootstrap.cc:654] T 732c4adeeff6422497aa7db4a82f535b P 10ca5ee68ad64b3283b9d235352c9d23: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:53.324245 10955 raft_consensus.cc:2468] T e5a5f5a608504650b48fd27f163faed6 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 1.
I20260709 13:03:53.337319 11427 raft_consensus.cc:697] T a610d1a71f52432e88fe56130f921fa2 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 LEADER]: Becoming Leader. State: Replica: 0172fed22b6b4890a7fdc953fa128af4, State: Running, Role: LEADER
I20260709 13:03:53.369820 11103 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87cd8f5097624c8692f1d83544d8881e" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:03:53.379544 11103 raft_consensus.cc:3060] T 87cd8f5097624c8692f1d83544d8881e P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:03:53.384521 11152 raft_consensus.cc:359] T d371c5e27bd843469d2e2e453734cb4a P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:53.385613 11152 raft_consensus.cc:385] T d371c5e27bd843469d2e2e453734cb4a P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:53.386006 11152 raft_consensus.cc:740] T d371c5e27bd843469d2e2e453734cb4a P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0172fed22b6b4890a7fdc953fa128af4, State: Initialized, Role: FOLLOWER
I20260709 13:03:53.388727 11027 raft_consensus.cc:2468] T a610d1a71f52432e88fe56130f921fa2 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 2.
I20260709 13:03:53.397626 11103 raft_consensus.cc:2468] T 87cd8f5097624c8692f1d83544d8881e P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 1.
I20260709 13:03:53.400234 11152 consensus_queue.cc:260] T d371c5e27bd843469d2e2e453734cb4a P 0172fed22b6b4890a7fdc953fa128af4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:53.412441 11151 ts_tablet_manager.cc:1434] T f371d2565dd84667b5613848604949bb P 186b12638a87410487467df721a3cbda: Time spent starting tablet: real 0.077s	user 0.017s	sys 0.017s
I20260709 13:03:53.413782 11151 tablet_bootstrap.cc:492] T 4b37f8fca8dc42718b2a42f0883a956f P 186b12638a87410487467df721a3cbda: Bootstrap starting.
I20260709 13:03:53.416217 10845 leader_election.cc:304] T 87cd8f5097624c8692f1d83544d8881e P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda; no voters: 
I20260709 13:03:53.410140 10843 leader_election.cc:304] T e5a5f5a608504650b48fd27f163faed6 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda; no voters: 0172fed22b6b4890a7fdc953fa128af4
I20260709 13:03:53.424680 11152 ts_tablet_manager.cc:1434] T d371c5e27bd843469d2e2e453734cb4a P 0172fed22b6b4890a7fdc953fa128af4: Time spent starting tablet: real 0.073s	user 0.009s	sys 0.000s
I20260709 13:03:53.418893 11428 raft_consensus.cc:2804] T bb7d67be500947b3aef4c4245e33671c P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:03:53.433871 10951 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87cd8f5097624c8692f1d83544d8881e" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:03:53.426844 11476 raft_consensus.cc:2804] T e5a5f5a608504650b48fd27f163faed6 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:03:53.425858 11332 raft_consensus.cc:2804] T 87cd8f5097624c8692f1d83544d8881e P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:03:53.451509 11152 tablet_bootstrap.cc:492] T e5a5f5a608504650b48fd27f163faed6 P 0172fed22b6b4890a7fdc953fa128af4: Bootstrap starting.
W20260709 13:03:53.456014 10843 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 47956us
I20260709 13:03:53.436764 11428 raft_consensus.cc:493] T bb7d67be500947b3aef4c4245e33671c P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:53.461290 11149 ts_tablet_manager.cc:1434] T f371d2565dd84667b5613848604949bb P 873357742d5843579556ee8166f23f47: Time spent starting tablet: real 0.327s	user 0.020s	sys 0.010s
I20260709 13:03:53.446537 10951 raft_consensus.cc:3060] T 87cd8f5097624c8692f1d83544d8881e P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:03:53.467590 11149 tablet_bootstrap.cc:492] T a154cc8b181f460abf4880064379948b P 873357742d5843579556ee8166f23f47: Bootstrap starting.
W20260709 13:03:53.476116 11474 log.cc:927] Time spent T 11de93cd9b9d457489ff1cb80f976c9a P 10ca5ee68ad64b3283b9d235352c9d23: Append to log took a long time: real 0.104s	user 0.001s	sys 0.000s
I20260709 13:03:53.462395 11428 raft_consensus.cc:3060] T bb7d67be500947b3aef4c4245e33671c P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:03:53.493579 11151 tablet_bootstrap.cc:654] T 4b37f8fca8dc42718b2a42f0883a956f P 186b12638a87410487467df721a3cbda: Neither blocks nor log segments found. Creating new log.
W20260709 13:03:53.469115 10992 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 60967.8us
I20260709 13:03:53.508631 10951 raft_consensus.cc:2468] T 87cd8f5097624c8692f1d83544d8881e P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 1.
I20260709 13:03:53.438371 11427 consensus_queue.cc:237] T a610d1a71f52432e88fe56130f921fa2 P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:53.510586 11428 raft_consensus.cc:515] T bb7d67be500947b3aef4c4245e33671c P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:53.522653 11476 raft_consensus.cc:697] T e5a5f5a608504650b48fd27f163faed6 P 186b12638a87410487467df721a3cbda [term 1 LEADER]: Becoming Leader. State: Replica: 186b12638a87410487467df721a3cbda, State: Running, Role: LEADER
I20260709 13:03:53.511629 11429 leader_election.cc:290] T 732c4adeeff6422497aa7db4a82f535b P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
W20260709 13:03:53.516216 11067 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:45807, user_credentials={real_user=slave}} blocked reactor thread for 46396.4us
I20260709 13:03:53.585701 11332 raft_consensus.cc:697] T 87cd8f5097624c8692f1d83544d8881e P 186b12638a87410487467df721a3cbda [term 1 LEADER]: Becoming Leader. State: Replica: 186b12638a87410487467df721a3cbda, State: Running, Role: LEADER
W20260709 13:03:53.594739 11478 log.cc:927] Time spent T 555217d45d9b4ee492dbb3ddaf0f7c02 P 186b12638a87410487467df721a3cbda: Append to log took a long time: real 0.069s	user 0.001s	sys 0.000s
I20260709 13:03:53.596531 11428 leader_election.cc:290] T bb7d67be500947b3aef4c4245e33671c P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 1 election: Requested vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:03:53.605189 11149 tablet_bootstrap.cc:654] T a154cc8b181f460abf4880064379948b P 873357742d5843579556ee8166f23f47: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:53.618825 11162 raft_consensus.cc:493] T d371c5e27bd843469d2e2e453734cb4a P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:53.619292 11152 tablet_bootstrap.cc:654] T e5a5f5a608504650b48fd27f163faed6 P 0172fed22b6b4890a7fdc953fa128af4: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:53.619506 11162 raft_consensus.cc:515] T d371c5e27bd843469d2e2e453734cb4a P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:53.623533 11162 leader_election.cc:290] T d371c5e27bd843469d2e2e453734cb4a P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:03:53.605707 11476 consensus_queue.cc:237] T e5a5f5a608504650b48fd27f163faed6 P 186b12638a87410487467df721a3cbda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:53.643110 11102 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb7d67be500947b3aef4c4245e33671c" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:03:53.667378 11102 raft_consensus.cc:3060] T bb7d67be500947b3aef4c4245e33671c P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:03:53.644825 11332 consensus_queue.cc:237] T 87cd8f5097624c8692f1d83544d8881e P 186b12638a87410487467df721a3cbda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:53.677893 11148 tablet_bootstrap.cc:492] T 732c4adeeff6422497aa7db4a82f535b P 10ca5ee68ad64b3283b9d235352c9d23: No bootstrap required, opened a new log
I20260709 13:03:53.678730 11148 ts_tablet_manager.cc:1403] T 732c4adeeff6422497aa7db4a82f535b P 10ca5ee68ad64b3283b9d235352c9d23: Time spent bootstrapping tablet: real 0.353s	user 0.018s	sys 0.027s
I20260709 13:03:53.682804 11148 raft_consensus.cc:359] T 732c4adeeff6422497aa7db4a82f535b P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:53.683540 11148 raft_consensus.cc:385] T 732c4adeeff6422497aa7db4a82f535b P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:53.683830 11148 raft_consensus.cc:740] T 732c4adeeff6422497aa7db4a82f535b P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10ca5ee68ad64b3283b9d235352c9d23, State: Initialized, Role: FOLLOWER
I20260709 13:03:53.684621 11148 consensus_queue.cc:260] T 732c4adeeff6422497aa7db4a82f535b P 10ca5ee68ad64b3283b9d235352c9d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:53.687104 11148 ts_tablet_manager.cc:1434] T 732c4adeeff6422497aa7db4a82f535b P 10ca5ee68ad64b3283b9d235352c9d23: Time spent starting tablet: real 0.008s	user 0.008s	sys 0.000s
I20260709 13:03:53.688153 11151 tablet_bootstrap.cc:492] T 4b37f8fca8dc42718b2a42f0883a956f P 186b12638a87410487467df721a3cbda: No bootstrap required, opened a new log
I20260709 13:03:53.688293 11148 tablet_bootstrap.cc:492] T a154cc8b181f460abf4880064379948b P 10ca5ee68ad64b3283b9d235352c9d23: Bootstrap starting.
I20260709 13:03:53.688892 11151 ts_tablet_manager.cc:1403] T 4b37f8fca8dc42718b2a42f0883a956f P 186b12638a87410487467df721a3cbda: Time spent bootstrapping tablet: real 0.275s	user 0.029s	sys 0.021s
I20260709 13:03:53.692654 11149 tablet_bootstrap.cc:492] T a154cc8b181f460abf4880064379948b P 873357742d5843579556ee8166f23f47: No bootstrap required, opened a new log
I20260709 13:03:53.693228 11149 ts_tablet_manager.cc:1403] T a154cc8b181f460abf4880064379948b P 873357742d5843579556ee8166f23f47: Time spent bootstrapping tablet: real 0.226s	user 0.014s	sys 0.005s
I20260709 13:03:53.698904 11148 tablet_bootstrap.cc:654] T a154cc8b181f460abf4880064379948b P 10ca5ee68ad64b3283b9d235352c9d23: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:53.702749 11151 raft_consensus.cc:359] T 4b37f8fca8dc42718b2a42f0883a956f P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:53.704623 11151 raft_consensus.cc:385] T 4b37f8fca8dc42718b2a42f0883a956f P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:53.704932 11151 raft_consensus.cc:740] T 4b37f8fca8dc42718b2a42f0883a956f P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 186b12638a87410487467df721a3cbda, State: Initialized, Role: FOLLOWER
I20260709 13:03:53.705744 11151 consensus_queue.cc:260] T 4b37f8fca8dc42718b2a42f0883a956f P 186b12638a87410487467df721a3cbda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:53.707823 11151 ts_tablet_manager.cc:1434] T 4b37f8fca8dc42718b2a42f0883a956f P 186b12638a87410487467df721a3cbda: Time spent starting tablet: real 0.018s	user 0.006s	sys 0.000s
I20260709 13:03:53.708417 11149 raft_consensus.cc:359] T a154cc8b181f460abf4880064379948b P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:53.708906 11151 tablet_bootstrap.cc:492] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 186b12638a87410487467df721a3cbda: Bootstrap starting.
I20260709 13:03:53.709152 11149 raft_consensus.cc:385] T a154cc8b181f460abf4880064379948b P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:53.709437 11149 raft_consensus.cc:740] T a154cc8b181f460abf4880064379948b P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 873357742d5843579556ee8166f23f47, State: Initialized, Role: FOLLOWER
I20260709 13:03:53.710196 11149 consensus_queue.cc:260] T a154cc8b181f460abf4880064379948b P 873357742d5843579556ee8166f23f47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:53.716960 11149 ts_tablet_manager.cc:1434] T a154cc8b181f460abf4880064379948b P 873357742d5843579556ee8166f23f47: Time spent starting tablet: real 0.023s	user 0.005s	sys 0.002s
I20260709 13:03:53.717319 11151 tablet_bootstrap.cc:654] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 186b12638a87410487467df721a3cbda: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:53.718036 11149 tablet_bootstrap.cc:492] T 4b37f8fca8dc42718b2a42f0883a956f P 873357742d5843579556ee8166f23f47: Bootstrap starting.
I20260709 13:03:53.724704 11152 tablet_bootstrap.cc:492] T e5a5f5a608504650b48fd27f163faed6 P 0172fed22b6b4890a7fdc953fa128af4: No bootstrap required, opened a new log
I20260709 13:03:53.725337 11152 ts_tablet_manager.cc:1403] T e5a5f5a608504650b48fd27f163faed6 P 0172fed22b6b4890a7fdc953fa128af4: Time spent bootstrapping tablet: real 0.274s	user 0.011s	sys 0.003s
I20260709 13:03:53.727576 11148 tablet_bootstrap.cc:492] T a154cc8b181f460abf4880064379948b P 10ca5ee68ad64b3283b9d235352c9d23: No bootstrap required, opened a new log
I20260709 13:03:53.728312 11148 ts_tablet_manager.cc:1403] T a154cc8b181f460abf4880064379948b P 10ca5ee68ad64b3283b9d235352c9d23: Time spent bootstrapping tablet: real 0.040s	user 0.023s	sys 0.012s
I20260709 13:03:53.731508 11149 tablet_bootstrap.cc:654] T 4b37f8fca8dc42718b2a42f0883a956f P 873357742d5843579556ee8166f23f47: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:53.731794 11152 raft_consensus.cc:359] T e5a5f5a608504650b48fd27f163faed6 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:53.732599 11152 raft_consensus.cc:385] T e5a5f5a608504650b48fd27f163faed6 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:53.732885 11152 raft_consensus.cc:740] T e5a5f5a608504650b48fd27f163faed6 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0172fed22b6b4890a7fdc953fa128af4, State: Initialized, Role: FOLLOWER
I20260709 13:03:53.732966 11148 raft_consensus.cc:359] T a154cc8b181f460abf4880064379948b P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:53.733856 11148 raft_consensus.cc:385] T a154cc8b181f460abf4880064379948b P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:53.733716 11152 consensus_queue.cc:260] T e5a5f5a608504650b48fd27f163faed6 P 0172fed22b6b4890a7fdc953fa128af4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:53.734205 11148 raft_consensus.cc:740] T a154cc8b181f460abf4880064379948b P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10ca5ee68ad64b3283b9d235352c9d23, State: Initialized, Role: FOLLOWER
I20260709 13:03:53.739678 11152 ts_tablet_manager.cc:1434] T e5a5f5a608504650b48fd27f163faed6 P 0172fed22b6b4890a7fdc953fa128af4: Time spent starting tablet: real 0.014s	user 0.009s	sys 0.000s
I20260709 13:03:53.739338 11148 consensus_queue.cc:260] T a154cc8b181f460abf4880064379948b P 10ca5ee68ad64b3283b9d235352c9d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:53.741992 11148 ts_tablet_manager.cc:1434] T a154cc8b181f460abf4880064379948b P 10ca5ee68ad64b3283b9d235352c9d23: Time spent starting tablet: real 0.013s	user 0.006s	sys 0.000s
I20260709 13:03:53.743273 11148 tablet_bootstrap.cc:492] T 4b37f8fca8dc42718b2a42f0883a956f P 10ca5ee68ad64b3283b9d235352c9d23: Bootstrap starting.
I20260709 13:03:53.749881 11148 tablet_bootstrap.cc:654] T 4b37f8fca8dc42718b2a42f0883a956f P 10ca5ee68ad64b3283b9d235352c9d23: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:53.753659 11152 tablet_bootstrap.cc:492] T a154cc8b181f460abf4880064379948b P 0172fed22b6b4890a7fdc953fa128af4: Bootstrap starting.
I20260709 13:03:53.755614 11151 tablet_bootstrap.cc:492] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 186b12638a87410487467df721a3cbda: No bootstrap required, opened a new log
I20260709 13:03:53.756287 11151 ts_tablet_manager.cc:1403] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 186b12638a87410487467df721a3cbda: Time spent bootstrapping tablet: real 0.048s	user 0.017s	sys 0.005s
I20260709 13:03:53.759822 11151 raft_consensus.cc:359] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:53.760582 11151 raft_consensus.cc:385] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:53.760941 11151 raft_consensus.cc:740] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 186b12638a87410487467df721a3cbda, State: Initialized, Role: FOLLOWER
I20260709 13:03:53.761307 11152 tablet_bootstrap.cc:654] T a154cc8b181f460abf4880064379948b P 0172fed22b6b4890a7fdc953fa128af4: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:53.761972 11151 consensus_queue.cc:260] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 186b12638a87410487467df721a3cbda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:53.764137 11151 ts_tablet_manager.cc:1434] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 186b12638a87410487467df721a3cbda: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.000s
I20260709 13:03:53.765328 11151 tablet_bootstrap.cc:492] T a5299b8d044f41b7b68057e63d17b159 P 186b12638a87410487467df721a3cbda: Bootstrap starting.
I20260709 13:03:53.765551 11149 tablet_bootstrap.cc:492] T 4b37f8fca8dc42718b2a42f0883a956f P 873357742d5843579556ee8166f23f47: No bootstrap required, opened a new log
I20260709 13:03:53.766122 11149 ts_tablet_manager.cc:1403] T 4b37f8fca8dc42718b2a42f0883a956f P 873357742d5843579556ee8166f23f47: Time spent bootstrapping tablet: real 0.048s	user 0.018s	sys 0.001s
I20260709 13:03:53.769593 11149 raft_consensus.cc:359] T 4b37f8fca8dc42718b2a42f0883a956f P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:53.770532 11149 raft_consensus.cc:385] T 4b37f8fca8dc42718b2a42f0883a956f P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:53.770792 11149 raft_consensus.cc:740] T 4b37f8fca8dc42718b2a42f0883a956f P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 873357742d5843579556ee8166f23f47, State: Initialized, Role: FOLLOWER
I20260709 13:03:53.771410 11149 consensus_queue.cc:260] T 4b37f8fca8dc42718b2a42f0883a956f P 873357742d5843579556ee8166f23f47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:53.773029 11149 ts_tablet_manager.cc:1434] T 4b37f8fca8dc42718b2a42f0883a956f P 873357742d5843579556ee8166f23f47: Time spent starting tablet: real 0.007s	user 0.006s	sys 0.000s
I20260709 13:03:53.776010 11148 tablet_bootstrap.cc:492] T 4b37f8fca8dc42718b2a42f0883a956f P 10ca5ee68ad64b3283b9d235352c9d23: No bootstrap required, opened a new log
I20260709 13:03:53.776587 11148 ts_tablet_manager.cc:1403] T 4b37f8fca8dc42718b2a42f0883a956f P 10ca5ee68ad64b3283b9d235352c9d23: Time spent bootstrapping tablet: real 0.034s	user 0.010s	sys 0.000s
I20260709 13:03:53.779834 11148 raft_consensus.cc:359] T 4b37f8fca8dc42718b2a42f0883a956f P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:53.780476 11148 raft_consensus.cc:385] T 4b37f8fca8dc42718b2a42f0883a956f P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:53.780738 11148 raft_consensus.cc:740] T 4b37f8fca8dc42718b2a42f0883a956f P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10ca5ee68ad64b3283b9d235352c9d23, State: Initialized, Role: FOLLOWER
I20260709 13:03:53.781486 11149 tablet_bootstrap.cc:492] T ff13ae9a47c7455f962406c6d131a3ff P 873357742d5843579556ee8166f23f47: Bootstrap starting.
I20260709 13:03:53.781352 11148 consensus_queue.cc:260] T 4b37f8fca8dc42718b2a42f0883a956f P 10ca5ee68ad64b3283b9d235352c9d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:53.783818 11148 ts_tablet_manager.cc:1434] T 4b37f8fca8dc42718b2a42f0883a956f P 10ca5ee68ad64b3283b9d235352c9d23: Time spent starting tablet: real 0.007s	user 0.008s	sys 0.000s
I20260709 13:03:53.784734 11148 tablet_bootstrap.cc:492] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 10ca5ee68ad64b3283b9d235352c9d23: Bootstrap starting.
I20260709 13:03:53.791673 11148 tablet_bootstrap.cc:654] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 10ca5ee68ad64b3283b9d235352c9d23: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:53.792850 11151 tablet_bootstrap.cc:654] T a5299b8d044f41b7b68057e63d17b159 P 186b12638a87410487467df721a3cbda: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:53.796638 11149 tablet_bootstrap.cc:654] T ff13ae9a47c7455f962406c6d131a3ff P 873357742d5843579556ee8166f23f47: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:53.821355 11152 tablet_bootstrap.cc:492] T a154cc8b181f460abf4880064379948b P 0172fed22b6b4890a7fdc953fa128af4: No bootstrap required, opened a new log
I20260709 13:03:53.822165 11152 ts_tablet_manager.cc:1403] T a154cc8b181f460abf4880064379948b P 0172fed22b6b4890a7fdc953fa128af4: Time spent bootstrapping tablet: real 0.069s	user 0.021s	sys 0.018s
I20260709 13:03:53.827099 11152 raft_consensus.cc:359] T a154cc8b181f460abf4880064379948b P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:53.827893 11152 raft_consensus.cc:385] T a154cc8b181f460abf4880064379948b P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:53.828173 11152 raft_consensus.cc:740] T a154cc8b181f460abf4880064379948b P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0172fed22b6b4890a7fdc953fa128af4, State: Initialized, Role: FOLLOWER
I20260709 13:03:53.828917 11152 consensus_queue.cc:260] T a154cc8b181f460abf4880064379948b P 0172fed22b6b4890a7fdc953fa128af4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:53.831890 11152 ts_tablet_manager.cc:1434] T a154cc8b181f460abf4880064379948b P 0172fed22b6b4890a7fdc953fa128af4: Time spent starting tablet: real 0.009s	user 0.006s	sys 0.000s
I20260709 13:03:53.832990 11152 tablet_bootstrap.cc:492] T f371d2565dd84667b5613848604949bb P 0172fed22b6b4890a7fdc953fa128af4: Bootstrap starting.
W20260709 13:03:54.824019 11149 thread.cc:641] log-alloc [worker] (thread pool) Time spent creating pthread: real 1.007s	user 0.019s	sys 0.020s
W20260709 13:03:54.825461 11149 thread.cc:608] log-alloc [worker] (thread pool) Time spent starting thread: real 1.009s	user 0.019s	sys 0.020s
I20260709 13:03:55.001103 11427 raft_consensus.cc:493] T d7ce47b9a92d43dd88d95bcda5038597 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 10ca5ee68ad64b3283b9d235352c9d23)
I20260709 13:03:55.002112 11427 raft_consensus.cc:515] T d7ce47b9a92d43dd88d95bcda5038597 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:55.226976 11427 leader_election.cc:290] T d7ce47b9a92d43dd88d95bcda5038597 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:03:55.227957 11427 raft_consensus.cc:493] T 6a9353d1d230416296627d7cf8b1d425 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:03:55.228675 11427 raft_consensus.cc:515] T 6a9353d1d230416296627d7cf8b1d425 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:55.247771 11152 tablet_bootstrap.cc:700] T f371d2565dd84667b5613848604949bb P 0172fed22b6b4890a7fdc953fa128af4: Time spent opening tablet: real 0.158s	user 0.003s	sys 0.000s
I20260709 13:03:55.248795 11152 tablet_bootstrap.cc:654] T f371d2565dd84667b5613848604949bb P 0172fed22b6b4890a7fdc953fa128af4: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:55.292718 11151 tablet_bootstrap.cc:492] T a5299b8d044f41b7b68057e63d17b159 P 186b12638a87410487467df721a3cbda: No bootstrap required, opened a new log
I20260709 13:03:55.293704 11151 ts_tablet_manager.cc:1403] T a5299b8d044f41b7b68057e63d17b159 P 186b12638a87410487467df721a3cbda: Time spent bootstrapping tablet: real 1.529s	user 0.027s	sys 0.006s
I20260709 13:03:55.313057 11151 raft_consensus.cc:359] T a5299b8d044f41b7b68057e63d17b159 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:55.314378 11151 raft_consensus.cc:385] T a5299b8d044f41b7b68057e63d17b159 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:55.314841 11151 raft_consensus.cc:740] T a5299b8d044f41b7b68057e63d17b159 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 186b12638a87410487467df721a3cbda, State: Initialized, Role: FOLLOWER
I20260709 13:03:55.315891 11151 consensus_queue.cc:260] T a5299b8d044f41b7b68057e63d17b159 P 186b12638a87410487467df721a3cbda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:55.370252 11427 leader_election.cc:290] T 6a9353d1d230416296627d7cf8b1d425 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:03:55.370569 11503 raft_consensus.cc:493] T 63d44f6ba6434ab2870fe3190f4d7c2f P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 10ca5ee68ad64b3283b9d235352c9d23)
I20260709 13:03:55.380390 11503 raft_consensus.cc:515] T 63d44f6ba6434ab2870fe3190f4d7c2f P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:55.393234 11149 tablet_bootstrap.cc:492] T ff13ae9a47c7455f962406c6d131a3ff P 873357742d5843579556ee8166f23f47: No bootstrap required, opened a new log
I20260709 13:03:55.393823 11149 ts_tablet_manager.cc:1403] T ff13ae9a47c7455f962406c6d131a3ff P 873357742d5843579556ee8166f23f47: Time spent bootstrapping tablet: real 1.613s	user 0.028s	sys 0.025s
I20260709 13:03:55.391386 11505 raft_consensus.cc:493] T 34064dc7bf3747dbb3876dd824ee0688 P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 186b12638a87410487467df721a3cbda)
I20260709 13:03:55.395251 11505 raft_consensus.cc:515] T 34064dc7bf3747dbb3876dd824ee0688 P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:55.399986 11148 tablet_bootstrap.cc:492] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 10ca5ee68ad64b3283b9d235352c9d23: No bootstrap required, opened a new log
I20260709 13:03:55.400439 11148 ts_tablet_manager.cc:1403] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 10ca5ee68ad64b3283b9d235352c9d23: Time spent bootstrapping tablet: real 1.616s	user 0.017s	sys 0.030s
I20260709 13:03:55.413285 11152 tablet_bootstrap.cc:492] T f371d2565dd84667b5613848604949bb P 0172fed22b6b4890a7fdc953fa128af4: No bootstrap required, opened a new log
I20260709 13:03:55.414146 11152 ts_tablet_manager.cc:1403] T f371d2565dd84667b5613848604949bb P 0172fed22b6b4890a7fdc953fa128af4: Time spent bootstrapping tablet: real 1.581s	user 0.012s	sys 0.007s
I20260709 13:03:55.434217 11148 raft_consensus.cc:359] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:55.438995 11148 raft_consensus.cc:385] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:55.439559 11148 raft_consensus.cc:740] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10ca5ee68ad64b3283b9d235352c9d23, State: Initialized, Role: FOLLOWER
I20260709 13:03:55.441993 11148 consensus_queue.cc:260] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 10ca5ee68ad64b3283b9d235352c9d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:55.446975 11149 raft_consensus.cc:359] T ff13ae9a47c7455f962406c6d131a3ff P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:55.448441 11149 raft_consensus.cc:385] T ff13ae9a47c7455f962406c6d131a3ff P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:55.448900 11149 raft_consensus.cc:740] T ff13ae9a47c7455f962406c6d131a3ff P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 873357742d5843579556ee8166f23f47, State: Initialized, Role: FOLLOWER
I20260709 13:03:55.451823 11148 ts_tablet_manager.cc:1434] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 10ca5ee68ad64b3283b9d235352c9d23: Time spent starting tablet: real 0.051s	user 0.009s	sys 0.000s
I20260709 13:03:55.449997 11149 consensus_queue.cc:260] T ff13ae9a47c7455f962406c6d131a3ff P 873357742d5843579556ee8166f23f47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:55.453545 11148 tablet_bootstrap.cc:492] T ff13ae9a47c7455f962406c6d131a3ff P 10ca5ee68ad64b3283b9d235352c9d23: Bootstrap starting.
I20260709 13:03:55.453869 11151 ts_tablet_manager.cc:1434] T a5299b8d044f41b7b68057e63d17b159 P 186b12638a87410487467df721a3cbda: Time spent starting tablet: real 0.160s	user 0.014s	sys 0.009s
I20260709 13:03:55.457451 11151 tablet_bootstrap.cc:492] T 8c160ddfe9b34bdaa3835d0915c1e06f P 186b12638a87410487467df721a3cbda: Bootstrap starting.
I20260709 13:03:55.461031 11505 leader_election.cc:290] T 34064dc7bf3747dbb3876dd824ee0688 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:03:55.461686 11505 raft_consensus.cc:493] T 525bfdbd6d934afd823f7d3ee3f9d052 P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 10ca5ee68ad64b3283b9d235352c9d23)
I20260709 13:03:55.460366 11503 leader_election.cc:290] T 63d44f6ba6434ab2870fe3190f4d7c2f P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:03:55.467707 11503 raft_consensus.cc:493] T e5ba72ccd854430baadbc636dfd17136 P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0172fed22b6b4890a7fdc953fa128af4)
I20260709 13:03:55.468214 11503 raft_consensus.cc:515] T e5ba72ccd854430baadbc636dfd17136 P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:55.470311 11503 leader_election.cc:290] T e5ba72ccd854430baadbc636dfd17136 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:03:55.471441 11149 ts_tablet_manager.cc:1434] T ff13ae9a47c7455f962406c6d131a3ff P 873357742d5843579556ee8166f23f47: Time spent starting tablet: real 0.077s	user 0.008s	sys 0.006s
I20260709 13:03:55.471799 10878 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51704 (request call id 63) took 2120 ms. Trace:
I20260709 13:03:55.472607 11149 tablet_bootstrap.cc:492] T 8c160ddfe9b34bdaa3835d0915c1e06f P 873357742d5843579556ee8166f23f47: Bootstrap starting.
I20260709 13:03:55.462198 11505 raft_consensus.cc:515] T 525bfdbd6d934afd823f7d3ee3f9d052 P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:55.472234 10878 rpcz_store.cc:276] 0709 13:03:53.369146 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:53.370723 (+  1577us) service_pool.cc:224] Handling call
0709 13:03:53.370992 (+   269us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:53.371343 (+   351us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:53.371387 (+    44us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:53.371970 (+   583us) log.cc:844] Serialized 26 byte log entry
0709 13:03:53.435226 (+ 63256us) raft_consensus.cc:1645] Marking committed up to 1
0709 13:03:53.435651 (+   425us) log.cc:844] Serialized 14 byte log entry
0709 13:03:53.435906 (+   255us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:53.436198 (+   292us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:55.471505 (+2035307us) raft_consensus.cc:1685] finished
0709 13:03:55.471529 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:55.471692 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":634,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":37,"raft.queue_time_us":59533,"raft.run_cpu_time_us":358,"raft.run_wall_time_us":13347,"thread_start_us":62849,"threads_started":1,"wal-append.queue_time_us":141520,"wal-append.run_cpu_time_us":3176,"wal-append.run_wall_time_us":1294886}
I20260709 13:03:55.478223 11505 leader_election.cc:290] T 525bfdbd6d934afd823f7d3ee3f9d052 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:03:55.469779 11152 raft_consensus.cc:359] T f371d2565dd84667b5613848604949bb P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:55.475989 11504 raft_consensus.cc:493] T 525bfdbd6d934afd823f7d3ee3f9d052 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 10ca5ee68ad64b3283b9d235352c9d23)
I20260709 13:03:55.479096 11504 raft_consensus.cc:515] T 525bfdbd6d934afd823f7d3ee3f9d052 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:55.480187 11509 raft_consensus.cc:493] T b4cd3d7dfbbf4096908ff11e9dc57141 P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0172fed22b6b4890a7fdc953fa128af4)
I20260709 13:03:55.481478 11504 leader_election.cc:290] T 525bfdbd6d934afd823f7d3ee3f9d052 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:03:55.481035 11509 raft_consensus.cc:515] T b4cd3d7dfbbf4096908ff11e9dc57141 P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:55.476302 11427 raft_consensus.cc:493] T 868e92339c2f4508af0afbd07c814d64 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:03:55.482980 11427 raft_consensus.cc:515] T 868e92339c2f4508af0afbd07c814d64 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:55.482393 11152 raft_consensus.cc:385] T f371d2565dd84667b5613848604949bb P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:55.484179 11152 raft_consensus.cc:740] T f371d2565dd84667b5613848604949bb P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0172fed22b6b4890a7fdc953fa128af4, State: Initialized, Role: FOLLOWER
I20260709 13:03:55.485445 11510 raft_consensus.cc:493] T 65bf0a62f4d84ef18dd6b23e082724b8 P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 10ca5ee68ad64b3283b9d235352c9d23)
I20260709 13:03:55.486066 11510 raft_consensus.cc:515] T 65bf0a62f4d84ef18dd6b23e082724b8 P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:55.487119 11508 raft_consensus.cc:493] T 0f9df2bc281f4866b6da5da94a0defc7 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0172fed22b6b4890a7fdc953fa128af4)
I20260709 13:03:55.487787 11509 leader_election.cc:290] T b4cd3d7dfbbf4096908ff11e9dc57141 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:03:55.487700 11508 raft_consensus.cc:515] T 0f9df2bc281f4866b6da5da94a0defc7 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:55.489048 11510 leader_election.cc:290] T 65bf0a62f4d84ef18dd6b23e082724b8 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:03:55.488775 11152 consensus_queue.cc:260] T f371d2565dd84667b5613848604949bb P 0172fed22b6b4890a7fdc953fa128af4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:55.486517 11427 leader_election.cc:290] T 868e92339c2f4508af0afbd07c814d64 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:03:55.494910 11508 leader_election.cc:290] T 0f9df2bc281f4866b6da5da94a0defc7 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:03:55.495502 11508 raft_consensus.cc:493] T 7839ca2d2c13431a85f44d7c458c6351 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:03:55.495946 11508 raft_consensus.cc:515] T 7839ca2d2c13431a85f44d7c458c6351 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:55.497856 11508 leader_election.cc:290] T 7839ca2d2c13431a85f44d7c458c6351 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:03:55.482640 11505 raft_consensus.cc:493] T 7709387f4d3d4f9992d97a0fbca39a40 P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0172fed22b6b4890a7fdc953fa128af4)
I20260709 13:03:55.505625 11505 raft_consensus.cc:515] T 7709387f4d3d4f9992d97a0fbca39a40 P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:55.507866 11505 leader_election.cc:290] T 7709387f4d3d4f9992d97a0fbca39a40 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
W20260709 13:03:55.510054 10994 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.107.196:38507 timed out after 1.648s (SENT) [suppressed 2 similar messages]
I20260709 13:03:55.512110 11504 raft_consensus.cc:493] T 72b111ad7b5f4753ab1d2b5ee1114840 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 10ca5ee68ad64b3283b9d235352c9d23)
I20260709 13:03:55.512691 11504 raft_consensus.cc:515] T 72b111ad7b5f4753ab1d2b5ee1114840 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:55.515429 11504 leader_election.cc:290] T 72b111ad7b5f4753ab1d2b5ee1114840 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:03:55.516772 11152 ts_tablet_manager.cc:1434] T f371d2565dd84667b5613848604949bb P 0172fed22b6b4890a7fdc953fa128af4: Time spent starting tablet: real 0.102s	user 0.006s	sys 0.006s
I20260709 13:03:55.517932 11152 tablet_bootstrap.cc:492] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 0172fed22b6b4890a7fdc953fa128af4: Bootstrap starting.
I20260709 13:03:55.518721 11511 raft_consensus.cc:493] T a64b7e99d6c54934a7de6f6dffcf5e94 P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0172fed22b6b4890a7fdc953fa128af4)
I20260709 13:03:55.519289 11511 raft_consensus.cc:515] T a64b7e99d6c54934a7de6f6dffcf5e94 P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:55.519958 11503 raft_consensus.cc:493] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 10ca5ee68ad64b3283b9d235352c9d23)
I20260709 13:03:55.520529 11503 raft_consensus.cc:515] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:55.521574 11511 leader_election.cc:290] T a64b7e99d6c54934a7de6f6dffcf5e94 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:03:55.522858 11148 tablet_bootstrap.cc:654] T ff13ae9a47c7455f962406c6d131a3ff P 10ca5ee68ad64b3283b9d235352c9d23: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:55.526391 11504 raft_consensus.cc:493] T 7757ecbf402e47bfbc93d0447741e26c P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:03:55.524931 11508 raft_consensus.cc:493] T 4f9eb06ed01f4ce68be3af8a8434665b P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:03:55.527487 11511 raft_consensus.cc:493] T 2567c2d84e4648d7ba5dc23a3df32817 P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 186b12638a87410487467df721a3cbda)
I20260709 13:03:55.529438 11509 raft_consensus.cc:493] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0172fed22b6b4890a7fdc953fa128af4)
I20260709 13:03:55.529865 11511 raft_consensus.cc:515] T 2567c2d84e4648d7ba5dc23a3df32817 P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:55.530048 11509 raft_consensus.cc:515] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:55.532778 11516 raft_consensus.cc:493] T 4c2a35bb99374e93b573cf870d04faaa P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 10ca5ee68ad64b3283b9d235352c9d23)
I20260709 13:03:55.528266 11427 raft_consensus.cc:493] T 8a69f32c53f24a50b7852d143665d7b4 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 10ca5ee68ad64b3283b9d235352c9d23)
I20260709 13:03:55.533181 11511 leader_election.cc:290] T 2567c2d84e4648d7ba5dc23a3df32817 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:03:55.529047 11505 raft_consensus.cc:493] T bac7e5040f984ba2b7146bd36af7267e P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0172fed22b6b4890a7fdc953fa128af4)
I20260709 13:03:55.528501 11510 raft_consensus.cc:493] T 0e5d327630e5438b878035d28efe6c0c P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:55.533615 11516 raft_consensus.cc:515] T 4c2a35bb99374e93b573cf870d04faaa P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:55.534371 11505 raft_consensus.cc:515] T bac7e5040f984ba2b7146bd36af7267e P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:55.534394 11510 raft_consensus.cc:515] T 0e5d327630e5438b878035d28efe6c0c P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:55.533762 11427 raft_consensus.cc:515] T 8a69f32c53f24a50b7852d143665d7b4 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:55.536718 11505 leader_election.cc:290] T bac7e5040f984ba2b7146bd36af7267e P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:03:55.536826 11516 leader_election.cc:290] T 4c2a35bb99374e93b573cf870d04faaa P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:03:55.534789 11509 leader_election.cc:290] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:03:55.528949 11515 raft_consensus.cc:493] T d371c5e27bd843469d2e2e453734cb4a P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:55.548748 11516 raft_consensus.cc:493] T ff13ae9a47c7455f962406c6d131a3ff P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:55.557374 11515 raft_consensus.cc:515] T d371c5e27bd843469d2e2e453734cb4a P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:55.543013 11512 raft_consensus.cc:493] T 87cd8f5097624c8692f1d83544d8881e P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:55.538507 11504 raft_consensus.cc:515] T 7757ecbf402e47bfbc93d0447741e26c P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
W20260709 13:03:55.561523 11102 consensus_meta.cc:287] T bb7d67be500947b3aef4c4245e33671c P 0172fed22b6b4890a7fdc953fa128af4: Time spent flushing consensus metadata: real 1.888s	user 0.021s	sys 0.018s
I20260709 13:03:55.540329 11510 leader_election.cc:290] T 0e5d327630e5438b878035d28efe6c0c P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:03:55.546651 11503 leader_election.cc:290] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:03:55.543438 11514 raft_consensus.cc:493] T 9a1716c0adf34bee8a4e99341ba5788f P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0172fed22b6b4890a7fdc953fa128af4)
I20260709 13:03:55.561504 11511 raft_consensus.cc:493] T f371d2565dd84667b5613848604949bb P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:55.538640 11508 raft_consensus.cc:515] T 4f9eb06ed01f4ce68be3af8a8434665b P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:55.566847 11512 raft_consensus.cc:515] T 87cd8f5097624c8692f1d83544d8881e P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:55.576313 10950 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49424 (request call id 85) took 1985 ms. Trace:
I20260709 13:03:55.572574 11516 raft_consensus.cc:515] T ff13ae9a47c7455f962406c6d131a3ff P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:55.583068 11515 leader_election.cc:290] T d371c5e27bd843469d2e2e453734cb4a P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:03:55.576440 11511 raft_consensus.cc:515] T f371d2565dd84667b5613848604949bb P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:55.585438 11515 raft_consensus.cc:493] T 8a69f32c53f24a50b7852d143665d7b4 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 10ca5ee68ad64b3283b9d235352c9d23)
I20260709 13:03:55.585636 10955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "732c4adeeff6422497aa7db4a82f535b" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:03:55.586110 11515 raft_consensus.cc:515] T 8a69f32c53f24a50b7852d143665d7b4 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:55.586509 10955 raft_consensus.cc:2468] T 732c4adeeff6422497aa7db4a82f535b P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 0.
I20260709 13:03:55.586901 10951 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49424 (request call id 84) took 2072 ms. Trace:
I20260709 13:03:55.587086 11511 leader_election.cc:290] T f371d2565dd84667b5613848604949bb P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:03:55.587378 10951 rpcz_store.cc:276] 0709 13:03:53.526848 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:53.531140 (+  4292us) service_pool.cc:224] Handling call
0709 13:03:53.590845 (+ 59705us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:53.600559 (+  9714us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:53.601373 (+   814us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:53.602546 (+  1173us) raft_consensus.cc:1645] Marking committed up to 1
0709 13:03:53.606108 (+  3562us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:53.608037 (+  1929us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:55.586791 (+1978754us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:55.589401 11515 leader_election.cc:290] T 8a69f32c53f24a50b7852d143665d7b4 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:03:55.590019 11515 raft_consensus.cc:493] T 11de93cd9b9d457489ff1cb80f976c9a P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:55.572274 11102 raft_consensus.cc:2468] T bb7d67be500947b3aef4c4245e33671c P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 1.
I20260709 13:03:55.584601 10950 rpcz_store.cc:276] 0709 13:03:55.573471 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:55.573666 (+   195us) service_pool.cc:224] Handling call
0709 13:03:55.574092 (+   426us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:55.574320 (+   228us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:55.574349 (+    29us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:55.574397 (+    48us) raft_consensus.cc:1645] Marking committed up to 1
0709 13:03:55.574460 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:55.576054 (+  1594us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:55.576208 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:55.590600 11515 raft_consensus.cc:515] T 11de93cd9b9d457489ff1cb80f976c9a P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:55.583125 11514 raft_consensus.cc:515] T 9a1716c0adf34bee8a4e99341ba5788f P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:55.591168 10955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb7d67be500947b3aef4c4245e33671c" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:03:55.625562 11511 raft_consensus.cc:493] T 9a1716c0adf34bee8a4e99341ba5788f P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0172fed22b6b4890a7fdc953fa128af4)
I20260709 13:03:55.708261 11511 raft_consensus.cc:515] T 9a1716c0adf34bee8a4e99341ba5788f P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:55.709132 11515 leader_election.cc:290] T 11de93cd9b9d457489ff1cb80f976c9a P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:03:55.709777 11515 raft_consensus.cc:493] T 63d44f6ba6434ab2870fe3190f4d7c2f P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 10ca5ee68ad64b3283b9d235352c9d23)
W20260709 13:03:55.632787 11102 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37676 (request call id 66) took 1994 ms (client timeout 1648 ms). Trace:
W20260709 13:03:55.714325 11102 rpcz_store.cc:269] 0709 13:03:53.630819 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:53.631710 (+   891us) service_pool.cc:224] Handling call
0709 13:03:55.618480 (+1986770us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:55.710294 11515 raft_consensus.cc:515] T 63d44f6ba6434ab2870fe3190f4d7c2f P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:55.707355 10955 raft_consensus.cc:3060] T bb7d67be500947b3aef4c4245e33671c P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:03:55.715533 11512 leader_election.cc:290] T 87cd8f5097624c8692f1d83544d8881e P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:03:55.710610 11505 raft_consensus.cc:493] T e89e640169e14e2d98f968f76027daa7 P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 10ca5ee68ad64b3283b9d235352c9d23)
I20260709 13:03:55.716559 11505 raft_consensus.cc:515] T e89e640169e14e2d98f968f76027daa7 P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:55.716859 11515 leader_election.cc:290] T 63d44f6ba6434ab2870fe3190f4d7c2f P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:03:55.709173 11514 leader_election.cc:290] T 9a1716c0adf34bee8a4e99341ba5788f P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:03:55.663781 11503 raft_consensus.cc:493] T d371c5e27bd843469d2e2e453734cb4a P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:55.710824 11510 raft_consensus.cc:493] T 6f46a5102fd64b45bf1c4dff863d2b4d P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0172fed22b6b4890a7fdc953fa128af4)
I20260709 13:03:55.718781 11503 raft_consensus.cc:515] T d371c5e27bd843469d2e2e453734cb4a P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:55.719079 11510 raft_consensus.cc:515] T 6f46a5102fd64b45bf1c4dff863d2b4d P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:55.720944 11503 leader_election.cc:290] T d371c5e27bd843469d2e2e453734cb4a P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:03:55.721060 11510 leader_election.cc:290] T 6f46a5102fd64b45bf1c4dff863d2b4d P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:03:55.711066 11520 raft_consensus.cc:493] T e96e80ba9ac34d0998395515e5c8c41a P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 10ca5ee68ad64b3283b9d235352c9d23)
I20260709 13:03:55.722114 11520 raft_consensus.cc:515] T e96e80ba9ac34d0998395515e5c8c41a P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
W20260709 13:03:55.664011 10994 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 153996us
I20260709 13:03:55.724618 11520 leader_election.cc:290] T e96e80ba9ac34d0998395515e5c8c41a P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:03:55.708528 11518 raft_consensus.cc:493] T c51782cd9d064ec0949e84e24dd4de67 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 10ca5ee68ad64b3283b9d235352c9d23)
I20260709 13:03:55.713420 11519 raft_consensus.cc:493] T 7839ca2d2c13431a85f44d7c458c6351 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:03:55.828351 11518 raft_consensus.cc:515] T c51782cd9d064ec0949e84e24dd4de67 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:55.710197 11508 leader_election.cc:290] T 4f9eb06ed01f4ce68be3af8a8434665b P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:03:55.840492 11511 leader_election.cc:290] T 9a1716c0adf34bee8a4e99341ba5788f P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:03:55.711941 11516 leader_election.cc:290] T ff13ae9a47c7455f962406c6d131a3ff P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:03:55.712868 11504 leader_election.cc:290] T 7757ecbf402e47bfbc93d0447741e26c P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:03:55.713920 11521 raft_consensus.cc:493] T 6f46a5102fd64b45bf1c4dff863d2b4d P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0172fed22b6b4890a7fdc953fa128af4)
I20260709 13:03:55.813782 11512 raft_consensus.cc:493] T e5a5f5a608504650b48fd27f163faed6 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:55.888209 11505 leader_election.cc:290] T e89e640169e14e2d98f968f76027daa7 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:03:55.912199 11505 raft_consensus.cc:493] T 32147c6d2d5a477aa38c597c7f78df5b P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:55.912873 11505 raft_consensus.cc:515] T 32147c6d2d5a477aa38c597c7f78df5b P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:55.913147 10878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "732c4adeeff6422497aa7db4a82f535b" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:03:55.913836 10878 raft_consensus.cc:2468] T 732c4adeeff6422497aa7db4a82f535b P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 0.
I20260709 13:03:55.911028 10950 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63d44f6ba6434ab2870fe3190f4d7c2f" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
W20260709 13:03:55.848664 10992 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 134261us
I20260709 13:03:55.916083 10992 leader_election.cc:304] T 732c4adeeff6422497aa7db4a82f535b P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:03:55.916982 11513 raft_consensus.cc:493] T f371d2565dd84667b5613848604949bb P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:55.911006 11512 raft_consensus.cc:515] T e5a5f5a608504650b48fd27f163faed6 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:55.917543 11513 raft_consensus.cc:515] T f371d2565dd84667b5613848604949bb P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:55.919551 11510 raft_consensus.cc:2804] T 732c4adeeff6422497aa7db4a82f535b P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:03:55.920001 11510 raft_consensus.cc:493] T 732c4adeeff6422497aa7db4a82f535b P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:55.920358 11510 raft_consensus.cc:3060] T 732c4adeeff6422497aa7db4a82f535b P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:03:55.921006 11360 raft_consensus.cc:493] T 4b37f8fca8dc42718b2a42f0883a956f P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:55.921519 11360 raft_consensus.cc:515] T 4b37f8fca8dc42718b2a42f0883a956f P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:55.922607 10877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34064dc7bf3747dbb3876dd824ee0688" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:03:55.922700 10874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7709387f4d3d4f9992d97a0fbca39a40" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:03:55.923530 10874 raft_consensus.cc:2468] T 7709387f4d3d4f9992d97a0fbca39a40 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 1.
I20260709 13:03:55.924572 11505 leader_election.cc:290] T 32147c6d2d5a477aa38c597c7f78df5b P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:03:55.925016 10874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2567c2d84e4648d7ba5dc23a3df32817" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:03:55.925796 11505 raft_consensus.cc:493] T a610d1a71f52432e88fe56130f921fa2 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:55.905015 10955 raft_consensus.cc:2468] T bb7d67be500947b3aef4c4245e33671c P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 1.
I20260709 13:03:55.909890 11518 leader_election.cc:290] T c51782cd9d064ec0949e84e24dd4de67 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:03:55.929111 10878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63d44f6ba6434ab2870fe3190f4d7c2f" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:03:55.930019 10878 raft_consensus.cc:2468] T 63d44f6ba6434ab2870fe3190f4d7c2f P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 1.
I20260709 13:03:55.934505 10955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "525bfdbd6d934afd823f7d3ee3f9d052" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:03:55.937125 10950 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34064dc7bf3747dbb3876dd824ee0688" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:03:55.937188 10874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555217d45d9b4ee492dbb3ddaf0f7c02" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:03:55.937990 10950 raft_consensus.cc:2468] T 34064dc7bf3747dbb3876dd824ee0688 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 1.
I20260709 13:03:55.938074 10874 raft_consensus.cc:2468] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 1.
I20260709 13:03:55.939000 10873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e89e640169e14e2d98f968f76027daa7" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:03:55.939615 10877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e5d327630e5438b878035d28efe6c0c" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:03:55.939877 10873 raft_consensus.cc:2468] T e89e640169e14e2d98f968f76027daa7 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 1.
I20260709 13:03:55.939082 10952 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5ba72ccd854430baadbc636dfd17136" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:03:55.940323 10877 raft_consensus.cc:2468] T 0e5d327630e5438b878035d28efe6c0c P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 1.
I20260709 13:03:55.937218 10875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65bf0a62f4d84ef18dd6b23e082724b8" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:03:55.940992 10952 raft_consensus.cc:2468] T e5ba72ccd854430baadbc636dfd17136 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 1.
I20260709 13:03:55.941362 10875 raft_consensus.cc:2468] T 65bf0a62f4d84ef18dd6b23e082724b8 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 1.
I20260709 13:03:55.942265 10880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f371d2565dd84667b5613848604949bb" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:03:55.942653 11513 leader_election.cc:290] T f371d2565dd84667b5613848604949bb P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:03:55.943009 10880 raft_consensus.cc:2468] T f371d2565dd84667b5613848604949bb P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 0.
I20260709 13:03:55.944082 10878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c2a35bb99374e93b573cf870d04faaa" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:03:55.944721 10878 raft_consensus.cc:2468] T 4c2a35bb99374e93b573cf870d04faaa P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 1.
I20260709 13:03:55.945676 11024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a9353d1d230416296627d7cf8b1d425" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:03:55.946189 11427 leader_election.cc:290] T 8a69f32c53f24a50b7852d143665d7b4 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:03:55.946851 11026 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "525bfdbd6d934afd823f7d3ee3f9d052" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:03:55.947474 11022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "868e92339c2f4508af0afbd07c814d64" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:03:55.947594 11026 raft_consensus.cc:2468] T 525bfdbd6d934afd823f7d3ee3f9d052 P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 1.
I20260709 13:03:55.947512 10877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a1716c0adf34bee8a4e99341ba5788f" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:03:55.949023 10875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "732c4adeeff6422497aa7db4a82f535b" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:03:55.949260 10877 raft_consensus.cc:2468] T 9a1716c0adf34bee8a4e99341ba5788f P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 1.
I20260709 13:03:55.949723 10875 raft_consensus.cc:2468] T 732c4adeeff6422497aa7db4a82f535b P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 0.
I20260709 13:03:55.828351 11519 raft_consensus.cc:515] T 7839ca2d2c13431a85f44d7c458c6351 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:55.952047 11525 raft_consensus.cc:493] T a64b7e99d6c54934a7de6f6dffcf5e94 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0172fed22b6b4890a7fdc953fa128af4)
I20260709 13:03:55.975596 11525 raft_consensus.cc:515] T a64b7e99d6c54934a7de6f6dffcf5e94 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:55.977257 10954 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f1466124dbd4f6ebb3c4d62ca48bf96" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:03:55.958935 10774 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:59610 (request call id 76) took 2179 ms. Trace:
I20260709 13:03:55.959044 11023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7757ecbf402e47bfbc93d0447741e26c" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:03:55.978621 10774 rpcz_store.cc:276] 0709 13:03:53.779187 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:53.779360 (+   173us) service_pool.cc:224] Handling call
0709 13:03:55.958826 (+2179466us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5220,"reported_tablets":1}
I20260709 13:03:55.955811 11027 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d371c5e27bd843469d2e2e453734cb4a" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:03:55.980729 11027 raft_consensus.cc:2468] T d371c5e27bd843469d2e2e453734cb4a P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 0.
I20260709 13:03:55.933840 11512 leader_election.cc:290] T e5a5f5a608504650b48fd27f163faed6 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:03:55.959224 11510 raft_consensus.cc:515] T 732c4adeeff6422497aa7db4a82f535b P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:55.984284 11510 leader_election.cc:290] T 732c4adeeff6422497aa7db4a82f535b P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 1 election: Requested vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:03:55.960117 11524 raft_consensus.cc:493] T 34064dc7bf3747dbb3876dd824ee0688 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 186b12638a87410487467df721a3cbda)
I20260709 13:03:55.986111 10954 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2567c2d84e4648d7ba5dc23a3df32817" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:03:55.961056 11519 leader_election.cc:290] T 7839ca2d2c13431a85f44d7c458c6351 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:03:55.987092 10955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bac7e5040f984ba2b7146bd36af7267e" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:03:55.987664 11519 raft_consensus.cc:493] T bac7e5040f984ba2b7146bd36af7267e P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0172fed22b6b4890a7fdc953fa128af4)
I20260709 13:03:55.963184 11028 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f371d2565dd84667b5613848604949bb" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:03:55.960562 11360 leader_election.cc:290] T 4b37f8fca8dc42718b2a42f0883a956f P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:03:55.932515 11521 raft_consensus.cc:515] T 6f46a5102fd64b45bf1c4dff863d2b4d P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:55.988238 11519 raft_consensus.cc:515] T bac7e5040f984ba2b7146bd36af7267e P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:55.988457 11028 raft_consensus.cc:2468] T f371d2565dd84667b5613848604949bb P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 0.
I20260709 13:03:55.966199 11067 leader_election.cc:304] T 525bfdbd6d934afd823f7d3ee3f9d052 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:03:55.989336 10953 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c2a35bb99374e93b573cf870d04faaa" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:03:55.990331 11521 leader_election.cc:290] T 6f46a5102fd64b45bf1c4dff863d2b4d P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:03:55.990641 11519 leader_election.cc:290] T bac7e5040f984ba2b7146bd36af7267e P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:03:55.926339 11505 raft_consensus.cc:515] T a610d1a71f52432e88fe56130f921fa2 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:55.968904 10952 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4cd3d7dfbbf4096908ff11e9dc57141" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:03:55.969334 10950 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65bf0a62f4d84ef18dd6b23e082724b8" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:03:55.965095 11151 tablet_bootstrap.cc:654] T 8c160ddfe9b34bdaa3835d0915c1e06f P 186b12638a87410487467df721a3cbda: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:55.971633 11522 raft_consensus.cc:493] T 836251ebc2b74f5da88b8a8b2338e2a5 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0172fed22b6b4890a7fdc953fa128af4)
I20260709 13:03:55.991860 10955 raft_consensus.cc:2468] T bac7e5040f984ba2b7146bd36af7267e P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 1.
I20260709 13:03:55.993155 11522 raft_consensus.cc:515] T 836251ebc2b74f5da88b8a8b2338e2a5 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:55.970186 10951 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a64b7e99d6c54934a7de6f6dffcf5e94" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:03:55.994334 10951 raft_consensus.cc:2468] T a64b7e99d6c54934a7de6f6dffcf5e94 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 1.
I20260709 13:03:55.971908 11511 consensus_queue.cc:1048] T 11de93cd9b9d457489ff1cb80f976c9a P 873357742d5843579556ee8166f23f47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:03:55.995731 11522 leader_election.cc:290] T 836251ebc2b74f5da88b8a8b2338e2a5 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:03:55.974083 11527 raft_consensus.cc:493] T 2567c2d84e4648d7ba5dc23a3df32817 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 186b12638a87410487467df721a3cbda)
I20260709 13:03:55.997181 11527 raft_consensus.cc:515] T 2567c2d84e4648d7ba5dc23a3df32817 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:55.997982 11523 raft_consensus.cc:493] T 6a9353d1d230416296627d7cf8b1d425 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:03:55.988627 11360 raft_consensus.cc:493] T 11de93cd9b9d457489ff1cb80f976c9a P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:03:55.966907 10878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d371c5e27bd843469d2e2e453734cb4a" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:03:56.000110 10878 raft_consensus.cc:2468] T d371c5e27bd843469d2e2e453734cb4a P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 0.
I20260709 13:03:55.965898 10880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e96e80ba9ac34d0998395515e5c8c41a" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:03:55.996503 11510 raft_consensus.cc:493] T 4b37f8fca8dc42718b2a42f0883a956f P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:56.001020 11527 leader_election.cc:290] T 2567c2d84e4648d7ba5dc23a3df32817 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:03:56.001127 11510 raft_consensus.cc:515] T 4b37f8fca8dc42718b2a42f0883a956f P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:55.985955 11524 raft_consensus.cc:515] T 34064dc7bf3747dbb3876dd824ee0688 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:55.998874 11360 raft_consensus.cc:515] T 11de93cd9b9d457489ff1cb80f976c9a P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:55.991304 11425 raft_consensus.cc:2804] T 525bfdbd6d934afd823f7d3ee3f9d052 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:03:56.004520 11425 raft_consensus.cc:493] T 525bfdbd6d934afd823f7d3ee3f9d052 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 10ca5ee68ad64b3283b9d235352c9d23)
I20260709 13:03:56.004961 11425 raft_consensus.cc:3060] T 525bfdbd6d934afd823f7d3ee3f9d052 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:03:56.006079 11360 leader_election.cc:290] T 11de93cd9b9d457489ff1cb80f976c9a P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:03:56.007203 10952 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e89e640169e14e2d98f968f76027daa7" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:03:56.008596 10950 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e96e80ba9ac34d0998395515e5c8c41a" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:03:56.004197 11524 leader_election.cc:290] T 34064dc7bf3747dbb3876dd824ee0688 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:03:55.998617 11523 raft_consensus.cc:515] T 6a9353d1d230416296627d7cf8b1d425 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:56.012763 10949 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb7d67be500947b3aef4c4245e33671c" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:03:56.013329 11523 leader_election.cc:290] T 6a9353d1d230416296627d7cf8b1d425 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:03:56.013512 10949 raft_consensus.cc:2376] T bb7d67be500947b3aef4c4245e33671c P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 873357742d5843579556ee8166f23f47 in term 1. Re-sending same reply.
I20260709 13:03:56.016574 10949 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32147c6d2d5a477aa38c597c7f78df5b" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:03:56.017235 10949 raft_consensus.cc:2468] T 32147c6d2d5a477aa38c597c7f78df5b P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 1.
I20260709 13:03:55.999292 10951 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f46a5102fd64b45bf1c4dff863d2b4d" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:03:56.017885 11505 leader_election.cc:290] T a610d1a71f52432e88fe56130f921fa2 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:03:55.997208 10995 leader_election.cc:304] T 7709387f4d3d4f9992d97a0fbca39a40 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:03:56.019629 11427 raft_consensus.cc:493] T 4f9eb06ed01f4ce68be3af8a8434665b P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:03:56.020205 11427 raft_consensus.cc:515] T 4f9eb06ed01f4ce68be3af8a8434665b P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:56.020602 10995 leader_election.cc:304] T 63d44f6ba6434ab2870fe3190f4d7c2f P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:03:56.023907 10949 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "732c4adeeff6422497aa7db4a82f535b" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:03:56.024355 11511 raft_consensus.cc:2804] T 7709387f4d3d4f9992d97a0fbca39a40 P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:03:56.024580 10949 raft_consensus.cc:3060] T 732c4adeeff6422497aa7db4a82f535b P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:03:56.024799 11511 raft_consensus.cc:493] T 7709387f4d3d4f9992d97a0fbca39a40 P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0172fed22b6b4890a7fdc953fa128af4)
I20260709 13:03:56.025235 11511 raft_consensus.cc:3060] T 7709387f4d3d4f9992d97a0fbca39a40 P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:03:56.029562 11102 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f9df2bc281f4866b6da5da94a0defc7" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:03:56.031006 11520 raft_consensus.cc:2804] T 63d44f6ba6434ab2870fe3190f4d7c2f P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:03:56.031512 11520 raft_consensus.cc:493] T 63d44f6ba6434ab2870fe3190f4d7c2f P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 10ca5ee68ad64b3283b9d235352c9d23)
I20260709 13:03:56.031901 11520 raft_consensus.cc:3060] T 63d44f6ba6434ab2870fe3190f4d7c2f P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:03:56.026360 11152 tablet_bootstrap.cc:654] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 0172fed22b6b4890a7fdc953fa128af4: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:56.033624 10949 raft_consensus.cc:2468] T 732c4adeeff6422497aa7db4a82f535b P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 1.
I20260709 13:03:56.034986 10995 leader_election.cc:304] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:03:56.035244 11513 raft_consensus.cc:493] T d71ca3a1ba6b48478aaf62741bf3cf6a P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:03:56.035841 11513 raft_consensus.cc:515] T d71ca3a1ba6b48478aaf62741bf3cf6a P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:56.036824 11506 raft_consensus.cc:2804] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:03:56.036707 11103 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a69f32c53f24a50b7852d143665d7b4" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:03:56.037292 11506 raft_consensus.cc:493] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0172fed22b6b4890a7fdc953fa128af4)
I20260709 13:03:56.037730 11103 raft_consensus.cc:2468] T 8a69f32c53f24a50b7852d143665d7b4 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 1.
I20260709 13:03:56.037725 11506 raft_consensus.cc:3060] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:03:56.038944 10950 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b37f8fca8dc42718b2a42f0883a956f" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:03:56.039741 10950 raft_consensus.cc:2468] T 4b37f8fca8dc42718b2a42f0883a956f P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 0.
I20260709 13:03:56.041642 11103 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a1716c0adf34bee8a4e99341ba5788f" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:03:56.043129 11522 raft_consensus.cc:493] T 0f9df2bc281f4866b6da5da94a0defc7 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0172fed22b6b4890a7fdc953fa128af4)
I20260709 13:03:56.043462 10955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff13ae9a47c7455f962406c6d131a3ff" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:03:56.043903 11522 raft_consensus.cc:515] T 0f9df2bc281f4866b6da5da94a0defc7 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:56.045059 11103 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f9eb06ed01f4ce68be3af8a8434665b" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:03:56.045761 11103 raft_consensus.cc:2468] T 4f9eb06ed01f4ce68be3af8a8434665b P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 1.
I20260709 13:03:56.047581 11103 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c51782cd9d064ec0949e84e24dd4de67" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:03:56.047683 11519 raft_consensus.cc:493] T d270b9e5481245d9b5340a454a7cd10b P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0172fed22b6b4890a7fdc953fa128af4)
I20260709 13:03:56.048310 11519 raft_consensus.cc:515] T d270b9e5481245d9b5340a454a7cd10b P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:56.048321 11103 raft_consensus.cc:2468] T c51782cd9d064ec0949e84e24dd4de67 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 1.
I20260709 13:03:56.050288 11527 raft_consensus.cc:493] T 5de5f0581e3b41dea729363d6ca563bb P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:56.050985 11101 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a9353d1d230416296627d7cf8b1d425" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:03:56.051775 11101 raft_consensus.cc:2468] T 6a9353d1d230416296627d7cf8b1d425 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 1.
I20260709 13:03:56.050318 11523 raft_consensus.cc:493] T 5d439326f23747eaab3c1e7da2d6b785 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 10ca5ee68ad64b3283b9d235352c9d23)
I20260709 13:03:56.052713 10992 leader_election.cc:304] T 34064dc7bf3747dbb3876dd824ee0688 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47; no voters: 186b12638a87410487467df721a3cbda
I20260709 13:03:56.053046 11523 raft_consensus.cc:515] T 5d439326f23747eaab3c1e7da2d6b785 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:56.054765 11518 raft_consensus.cc:493] T 72b111ad7b5f4753ab1d2b5ee1114840 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 10ca5ee68ad64b3283b9d235352c9d23)
I20260709 13:03:56.031910 11102 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d371c5e27bd843469d2e2e453734cb4a" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:03:56.050982 11527 raft_consensus.cc:515] T 5de5f0581e3b41dea729363d6ca563bb P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:56.055816 11518 raft_consensus.cc:515] T 72b111ad7b5f4753ab1d2b5ee1114840 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:56.056537 11102 raft_consensus.cc:2468] T d371c5e27bd843469d2e2e453734cb4a P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 0.
I20260709 13:03:56.022320 11427 leader_election.cc:290] T 4f9eb06ed01f4ce68be3af8a8434665b P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:03:56.058238 11510 leader_election.cc:290] T 4b37f8fca8dc42718b2a42f0883a956f P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:03:56.058573 11527 leader_election.cc:290] T 5de5f0581e3b41dea729363d6ca563bb P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:03:56.058120 11518 leader_election.cc:290] T 72b111ad7b5f4753ab1d2b5ee1114840 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:03:56.059280 11100 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5ba72ccd854430baadbc636dfd17136" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:03:56.062109 11504 raft_consensus.cc:493] T a154cc8b181f460abf4880064379948b P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:56.064252 11504 raft_consensus.cc:515] T a154cc8b181f460abf4880064379948b P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:56.017983 10951 raft_consensus.cc:2468] T 6f46a5102fd64b45bf1c4dff863d2b4d P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 1.
I20260709 13:03:56.064997 11522 leader_election.cc:290] T 0f9df2bc281f4866b6da5da94a0defc7 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:03:56.066919 11519 leader_election.cc:290] T d270b9e5481245d9b5340a454a7cd10b P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:03:56.046154 10776 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:59608 (request call id 60) took 2445 ms. Trace:
I20260709 13:03:56.068236 10776 rpcz_store.cc:276] 0709 13:03:53.600518 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:53.775533 (+175015us) service_pool.cc:224] Handling call
0709 13:03:56.046079 (+2270546us) inbound_call.cc:177] Queueing success response
Metrics: {"reported_tablets":2}
I20260709 13:03:55.978271 11525 leader_election.cc:290] T a64b7e99d6c54934a7de6f6dffcf5e94 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:03:56.033677 11511 raft_consensus.cc:515] T 7709387f4d3d4f9992d97a0fbca39a40 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:56.074771 11149 tablet_bootstrap.cc:654] T 8c160ddfe9b34bdaa3835d0915c1e06f P 873357742d5843579556ee8166f23f47: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:56.069447 10995 leader_election.cc:304] T e89e640169e14e2d98f968f76027daa7 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:03:56.079592 11477 raft_consensus.cc:2804] T e89e640169e14e2d98f968f76027daa7 P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:03:56.080459 11477 raft_consensus.cc:493] T e89e640169e14e2d98f968f76027daa7 P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 10ca5ee68ad64b3283b9d235352c9d23)
I20260709 13:03:56.080976 11477 raft_consensus.cc:3060] T e89e640169e14e2d98f968f76027daa7 P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:03:56.079684 10995 leader_election.cc:304] T 0e5d327630e5438b878035d28efe6c0c P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:03:56.059751 11524 raft_consensus.cc:493] T e5a5f5a608504650b48fd27f163faed6 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:56.065182 11505 raft_consensus.cc:2804] T 34064dc7bf3747dbb3876dd824ee0688 P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:03:56.093107 10775 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:59598 (request call id 63) took 2596 ms. Trace:
I20260709 13:03:56.076622 10880 raft_consensus.cc:2468] T e96e80ba9ac34d0998395515e5c8c41a P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 1.
I20260709 13:03:56.054219 10992 leader_election.cc:304] T e5ba72ccd854430baadbc636dfd17136 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:03:56.096318 11151 tablet_bootstrap.cc:492] T 8c160ddfe9b34bdaa3835d0915c1e06f P 186b12638a87410487467df721a3cbda: No bootstrap required, opened a new log
I20260709 13:03:56.098412 11151 ts_tablet_manager.cc:1403] T 8c160ddfe9b34bdaa3835d0915c1e06f P 186b12638a87410487467df721a3cbda: Time spent bootstrapping tablet: real 0.641s	user 0.006s	sys 0.030s
I20260709 13:03:56.099265 11102 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4cd3d7dfbbf4096908ff11e9dc57141" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:03:56.093561 10775 rpcz_store.cc:276] 0709 13:03:53.491878 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:55.067502 (+1575624us) service_pool.cc:224] Handling call
0709 13:03:56.088090 (+1020588us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":180919,"reported_tablets":1}
I20260709 13:03:56.091543 11100 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "525bfdbd6d934afd823f7d3ee3f9d052" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:03:56.091888 11522 raft_consensus.cc:493] T 87cd8f5097624c8692f1d83544d8881e P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:56.091888 11505 raft_consensus.cc:493] T 34064dc7bf3747dbb3876dd824ee0688 P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 186b12638a87410487467df721a3cbda)
I20260709 13:03:56.111202 10774 catalog_manager.cc:5697] T a610d1a71f52432e88fe56130f921fa2 P 0172fed22b6b4890a7fdc953fa128af4 reported cstate change: term changed from 0 to 2, leader changed from <none> to 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196). New cstate: current_term: 2 leader_uuid: "0172fed22b6b4890a7fdc953fa128af4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } health_report { overall_health: UNKNOWN } } }
I20260709 13:03:56.114617 11508 raft_consensus.cc:493] T 5de5f0581e3b41dea729363d6ca563bb P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:56.115190 11508 raft_consensus.cc:515] T 5de5f0581e3b41dea729363d6ca563bb P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:56.117149 11102 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7709387f4d3d4f9992d97a0fbca39a40" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:03:56.108107 11516 raft_consensus.cc:2804] T e5ba72ccd854430baadbc636dfd17136 P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:03:56.117935 11101 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555217d45d9b4ee492dbb3ddaf0f7c02" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:03:56.117323 11508 leader_election.cc:290] T 5de5f0581e3b41dea729363d6ca563bb P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:03:56.097708 11503 raft_consensus.cc:2804] T 0e5d327630e5438b878035d28efe6c0c P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:03:56.118237 11516 raft_consensus.cc:493] T e5ba72ccd854430baadbc636dfd17136 P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0172fed22b6b4890a7fdc953fa128af4)
I20260709 13:03:56.102986 11425 raft_consensus.cc:515] T 525bfdbd6d934afd823f7d3ee3f9d052 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:56.121634 11516 raft_consensus.cc:3060] T e5ba72ccd854430baadbc636dfd17136 P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:03:56.121671 11103 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a64b7e99d6c54934a7de6f6dffcf5e94" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
W20260709 13:03:56.107476 10992 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 53366.9us
I20260709 13:03:56.124176 10992 leader_election.cc:304] T 2567c2d84e4648d7ba5dc23a3df32817 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 873357742d5843579556ee8166f23f47; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda
I20260709 13:03:56.124899 10878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7839ca2d2c13431a85f44d7c458c6351" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:03:56.126210 10876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b37f8fca8dc42718b2a42f0883a956f" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:03:56.126919 10876 raft_consensus.cc:2468] T 4b37f8fca8dc42718b2a42f0883a956f P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 0.
I20260709 13:03:56.129289 10875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34064dc7bf3747dbb3876dd824ee0688" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:03:56.129669 11026 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f46a5102fd64b45bf1c4dff863d2b4d" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:03:56.130870 11022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a64b7e99d6c54934a7de6f6dffcf5e94" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:03:56.131209 10880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2567c2d84e4648d7ba5dc23a3df32817" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:03:56.131636 10877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11de93cd9b9d457489ff1cb80f976c9a" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:03:56.132110 11024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bac7e5040f984ba2b7146bd36af7267e" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:03:56.132184 11023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b37f8fca8dc42718b2a42f0883a956f" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:03:56.132401 10877 raft_consensus.cc:2468] T 11de93cd9b9d457489ff1cb80f976c9a P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 1.
I20260709 13:03:56.132999 11023 raft_consensus.cc:2468] T 4b37f8fca8dc42718b2a42f0883a956f P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 0.
I20260709 13:03:56.120640 11503 raft_consensus.cc:493] T 0e5d327630e5438b878035d28efe6c0c P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:56.133939 11503 raft_consensus.cc:3060] T 0e5d327630e5438b878035d28efe6c0c P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Advancing to term 2
W20260709 13:03:56.133987 10843 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.597s	user 0.010s	sys 0.009s
W20260709 13:03:56.134631 10843 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.597s	user 0.010s	sys 0.009s
I20260709 13:03:56.134559 10873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5de5f0581e3b41dea729363d6ca563bb" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:03:56.135387 10873 raft_consensus.cc:2468] T 5de5f0581e3b41dea729363d6ca563bb P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 1.
I20260709 13:03:56.137688 10878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "836251ebc2b74f5da88b8a8b2338e2a5" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:03:56.138355 10878 raft_consensus.cc:2468] T 836251ebc2b74f5da88b8a8b2338e2a5 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 1.
I20260709 13:03:56.131222 11029 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7839ca2d2c13431a85f44d7c458c6351" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:03:56.140035 11021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2567c2d84e4648d7ba5dc23a3df32817" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:03:56.140758 11021 raft_consensus.cc:2468] T 2567c2d84e4648d7ba5dc23a3df32817 P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 1.
I20260709 13:03:56.142845 11100 raft_consensus.cc:2393] T 525bfdbd6d934afd823f7d3ee3f9d052 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 873357742d5843579556ee8166f23f47 in current term 2: Already voted for candidate 0172fed22b6b4890a7fdc953fa128af4 in this term.
I20260709 13:03:56.144857 10873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f9df2bc281f4866b6da5da94a0defc7" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:03:56.146502 11021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34064dc7bf3747dbb3876dd824ee0688" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:03:56.147110 11510 raft_consensus.cc:2749] T 2567c2d84e4648d7ba5dc23a3df32817 P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:03:56.131642 11022 raft_consensus.cc:2468] T a64b7e99d6c54934a7de6f6dffcf5e94 P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 1.
I20260709 13:03:56.149562 10876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d371c5e27bd843469d2e2e453734cb4a" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:03:56.150310 10876 raft_consensus.cc:2468] T d371c5e27bd843469d2e2e453734cb4a P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 0.
I20260709 13:03:56.151103 11029 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11de93cd9b9d457489ff1cb80f976c9a" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:03:56.151232 10992 leader_election.cc:304] T a64b7e99d6c54934a7de6f6dffcf5e94 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:03:56.153070 10919 leader_election.cc:304] T 4b37f8fca8dc42718b2a42f0883a956f P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:03:56.153348 10992 leader_election.cc:304] T bac7e5040f984ba2b7146bd36af7267e P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:03:56.154553 11521 raft_consensus.cc:493] T a154cc8b181f460abf4880064379948b P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:56.155081 11521 raft_consensus.cc:515] T a154cc8b181f460abf4880064379948b P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:56.155372 10992 leader_election.cc:304] T bb7d67be500947b3aef4c4245e33671c P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:03:56.155917 11360 raft_consensus.cc:2804] T 4b37f8fca8dc42718b2a42f0883a956f P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20260709 13:03:56.151232 10920 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 178070us
I20260709 13:03:56.156369 11360 raft_consensus.cc:493] T 4b37f8fca8dc42718b2a42f0883a956f P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:56.156829 11360 raft_consensus.cc:3060] T 4b37f8fca8dc42718b2a42f0883a956f P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:03:56.157629 11521 leader_election.cc:290] T a154cc8b181f460abf4880064379948b P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:03:56.158499 10919 leader_election.cc:304] T 2567c2d84e4648d7ba5dc23a3df32817 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:03:56.158788 10877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d270b9e5481245d9b5340a454a7cd10b" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:03:56.159526 10877 raft_consensus.cc:2468] T d270b9e5481245d9b5340a454a7cd10b P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 1.
I20260709 13:03:56.161355 11527 raft_consensus.cc:2804] T 2567c2d84e4648d7ba5dc23a3df32817 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:03:56.156661 11506 raft_consensus.cc:515] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:56.162916 11527 raft_consensus.cc:493] T 2567c2d84e4648d7ba5dc23a3df32817 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 186b12638a87410487467df721a3cbda)
I20260709 13:03:56.163470 11527 raft_consensus.cc:3060] T 2567c2d84e4648d7ba5dc23a3df32817 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:03:56.165580 11504 leader_election.cc:290] T a154cc8b181f460abf4880064379948b P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:03:56.160382 11522 raft_consensus.cc:515] T 87cd8f5097624c8692f1d83544d8881e P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:56.168725 11511 leader_election.cc:290] T 7709387f4d3d4f9992d97a0fbca39a40 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 2 election: Requested vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:03:56.160722 11510 raft_consensus.cc:2804] T a64b7e99d6c54934a7de6f6dffcf5e94 P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:03:56.169286 11425 leader_election.cc:290] T 525bfdbd6d934afd823f7d3ee3f9d052 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 2 election: Requested vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:03:56.169404 11510 raft_consensus.cc:493] T a64b7e99d6c54934a7de6f6dffcf5e94 P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0172fed22b6b4890a7fdc953fa128af4)
I20260709 13:03:56.169900 11510 raft_consensus.cc:3060] T a64b7e99d6c54934a7de6f6dffcf5e94 P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:03:56.170228 11101 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bac7e5040f984ba2b7146bd36af7267e" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:03:56.172150 11513 leader_election.cc:290] T d71ca3a1ba6b48478aaf62741bf3cf6a P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:03:56.172940 11100 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f1466124dbd4f6ebb3c4d62ca48bf96" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:03:56.173643 11100 raft_consensus.cc:2468] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 1.
I20260709 13:03:56.172195 11101 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e5d327630e5438b878035d28efe6c0c" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:03:56.174742 10878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d7ce47b9a92d43dd88d95bcda5038597" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:03:56.176270 11102 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f371d2565dd84667b5613848604949bb" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:03:56.176234 10919 leader_election.cc:304] T a64b7e99d6c54934a7de6f6dffcf5e94 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:03:56.176975 11522 leader_election.cc:290] T 87cd8f5097624c8692f1d83544d8881e P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:03:56.177127 11102 raft_consensus.cc:2468] T f371d2565dd84667b5613848604949bb P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 0.
I20260709 13:03:56.177620 11508 raft_consensus.cc:493] T 836251ebc2b74f5da88b8a8b2338e2a5 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0172fed22b6b4890a7fdc953fa128af4)
I20260709 13:03:56.176576 11152 tablet_bootstrap.cc:492] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 0172fed22b6b4890a7fdc953fa128af4: No bootstrap required, opened a new log
I20260709 13:03:56.178210 11508 raft_consensus.cc:515] T 836251ebc2b74f5da88b8a8b2338e2a5 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:56.179097 11152 ts_tablet_manager.cc:1403] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 0172fed22b6b4890a7fdc953fa128af4: Time spent bootstrapping tablet: real 0.661s	user 0.025s	sys 0.003s
I20260709 13:03:56.181998 11523 leader_election.cc:290] T 5d439326f23747eaab3c1e7da2d6b785 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:03:56.182914 11103 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff13ae9a47c7455f962406c6d131a3ff" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:03:56.184989 11520 raft_consensus.cc:515] T 63d44f6ba6434ab2870fe3190f4d7c2f P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:56.092330 11524 raft_consensus.cc:515] T e5a5f5a608504650b48fd27f163faed6 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:56.119349 11505 raft_consensus.cc:3060] T 34064dc7bf3747dbb3876dd824ee0688 P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:03:56.185483 11151 raft_consensus.cc:359] T 8c160ddfe9b34bdaa3835d0915c1e06f P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:56.199604 11152 raft_consensus.cc:359] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:56.199834 11151 raft_consensus.cc:385] T 8c160ddfe9b34bdaa3835d0915c1e06f P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:56.200973 11152 raft_consensus.cc:385] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:56.201710 11152 raft_consensus.cc:740] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0172fed22b6b4890a7fdc953fa128af4, State: Initialized, Role: FOLLOWER
W20260709 13:03:56.205624 10992 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 50322.1us
I20260709 13:03:56.206339 11537 raft_consensus.cc:2804] T bb7d67be500947b3aef4c4245e33671c P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:03:56.207186 11537 raft_consensus.cc:697] T bb7d67be500947b3aef4c4245e33671c P 873357742d5843579556ee8166f23f47 [term 1 LEADER]: Becoming Leader. State: Replica: 873357742d5843579556ee8166f23f47, State: Running, Role: LEADER
I20260709 13:03:56.207729 10992 leader_election.cc:304] T 32147c6d2d5a477aa38c597c7f78df5b P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:03:56.208431 11537 consensus_queue.cc:237] T bb7d67be500947b3aef4c4245e33671c P 873357742d5843579556ee8166f23f47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
W20260709 13:03:56.213052 10995 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:38967, user_credentials={real_user=slave}} blocked reactor thread for 133477us
I20260709 13:03:56.201444 11151 raft_consensus.cc:740] T 8c160ddfe9b34bdaa3835d0915c1e06f P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 186b12638a87410487467df721a3cbda, State: Initialized, Role: FOLLOWER
I20260709 13:03:56.190050 11529 raft_consensus.cc:2804] T a64b7e99d6c54934a7de6f6dffcf5e94 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:03:56.214876 11529 raft_consensus.cc:493] T a64b7e99d6c54934a7de6f6dffcf5e94 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0172fed22b6b4890a7fdc953fa128af4)
I20260709 13:03:56.215250 11529 raft_consensus.cc:3060] T a64b7e99d6c54934a7de6f6dffcf5e94 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:03:56.215302 11506 leader_election.cc:290] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 2 election: Requested vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:03:56.215353 10955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d7ce47b9a92d43dd88d95bcda5038597" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:03:56.216508 11527 raft_consensus.cc:515] T 2567c2d84e4648d7ba5dc23a3df32817 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:56.216984 11103 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f46a5102fd64b45bf1c4dff863d2b4d" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:03:56.214161 10995 leader_election.cc:304] T 65bf0a62f4d84ef18dd6b23e082724b8 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47; no voters: 10ca5ee68ad64b3283b9d235352c9d23
I20260709 13:03:56.219169 11151 consensus_queue.cc:260] T 8c160ddfe9b34bdaa3835d0915c1e06f P 186b12638a87410487467df721a3cbda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:56.223268 11527 leader_election.cc:290] T 2567c2d84e4648d7ba5dc23a3df32817 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 2 election: Requested vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:03:56.188148 11524 leader_election.cc:290] T e5a5f5a608504650b48fd27f163faed6 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:03:56.169430 11511 raft_consensus.cc:2804] T bac7e5040f984ba2b7146bd36af7267e P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:03:56.226855 11022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a154cc8b181f460abf4880064379948b" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:03:56.227310 11511 raft_consensus.cc:493] T bac7e5040f984ba2b7146bd36af7267e P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0172fed22b6b4890a7fdc953fa128af4)
I20260709 13:03:56.227638 11022 raft_consensus.cc:2468] T a154cc8b181f460abf4880064379948b P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 0.
I20260709 13:03:56.227751 11508 leader_election.cc:290] T 836251ebc2b74f5da88b8a8b2338e2a5 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:03:56.227980 11511 raft_consensus.cc:3060] T bac7e5040f984ba2b7146bd36af7267e P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:03:56.229490 11520 leader_election.cc:290] T 63d44f6ba6434ab2870fe3190f4d7c2f P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 2 election: Requested vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:03:56.232213 10994 leader_election.cc:304] T b4cd3d7dfbbf4096908ff11e9dc57141 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 873357742d5843579556ee8166f23f47; no voters: 0172fed22b6b4890a7fdc953fa128af4, 10ca5ee68ad64b3283b9d235352c9d23
I20260709 13:03:56.186153 11025 raft_consensus.cc:1240] T 63d44f6ba6434ab2870fe3190f4d7c2f P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Rejecting Update request from peer 10ca5ee68ad64b3283b9d235352c9d23 for earlier term 1. Current term is 2. Ops: []
I20260709 13:03:56.212268 11101 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d371c5e27bd843469d2e2e453734cb4a" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:03:56.220821 11152 consensus_queue.cc:260] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 0172fed22b6b4890a7fdc953fa128af4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
W20260709 13:03:56.233100 10826 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 13:03:56.221411 10995 leader_election.cc:304] T f371d2565dd84667b5613848604949bb P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:03:56.189680 11503 raft_consensus.cc:515] T 0e5d327630e5438b878035d28efe6c0c P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:56.248477 11148 tablet_bootstrap.cc:492] T ff13ae9a47c7455f962406c6d131a3ff P 10ca5ee68ad64b3283b9d235352c9d23: No bootstrap required, opened a new log
I20260709 13:03:56.264616 10878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "868e92339c2f4508af0afbd07c814d64" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:03:56.276646 11148 ts_tablet_manager.cc:1403] T ff13ae9a47c7455f962406c6d131a3ff P 10ca5ee68ad64b3283b9d235352c9d23: Time spent bootstrapping tablet: real 0.823s	user 0.036s	sys 0.003s
I20260709 13:03:56.265725 10955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "525bfdbd6d934afd823f7d3ee3f9d052" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:03:56.237663 11506 raft_consensus.cc:2804] T 65bf0a62f4d84ef18dd6b23e082724b8 P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:03:56.267769 10919 leader_election.cc:304] T a154cc8b181f460abf4880064379948b P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:03:56.268460 10951 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72b111ad7b5f4753ab1d2b5ee1114840" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:03:56.269269 10949 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a69f32c53f24a50b7852d143665d7b4" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:03:56.270205 10948 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a154cc8b181f460abf4880064379948b" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:03:56.270980 10950 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87cd8f5097624c8692f1d83544d8881e" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:03:56.271806 10952 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5a5f5a608504650b48fd27f163faed6" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:03:56.226258 11360 raft_consensus.cc:515] T 4b37f8fca8dc42718b2a42f0883a956f P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:56.242023 11101 raft_consensus.cc:2468] T d371c5e27bd843469d2e2e453734cb4a P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 0.
I20260709 13:03:56.273231 10946 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "525bfdbd6d934afd823f7d3ee3f9d052" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:03:56.274781 10954 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d71ca3a1ba6b48478aaf62741bf3cf6a" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:03:56.272483 11152 ts_tablet_manager.cc:1434] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 0172fed22b6b4890a7fdc953fa128af4: Time spent starting tablet: real 0.093s	user 0.008s	sys 0.000s
I20260709 13:03:56.283147 11151 ts_tablet_manager.cc:1434] T 8c160ddfe9b34bdaa3835d0915c1e06f P 186b12638a87410487467df721a3cbda: Time spent starting tablet: real 0.184s	user 0.003s	sys 0.006s
I20260709 13:03:56.262777 11508 raft_consensus.cc:493] T a610d1a71f52432e88fe56130f921fa2 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:56.244016 11102 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a1716c0adf34bee8a4e99341ba5788f" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:03:56.263166 11097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7709387f4d3d4f9992d97a0fbca39a40" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:03:56.293092 10845 leader_election.cc:304] T 8a69f32c53f24a50b7852d143665d7b4 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda; no voters: 
I20260709 13:03:56.278594 11506 raft_consensus.cc:493] T 65bf0a62f4d84ef18dd6b23e082724b8 P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 10ca5ee68ad64b3283b9d235352c9d23)
I20260709 13:03:56.301069 11151 tablet_bootstrap.cc:492] T c31881534c5e4f0089c82057c1746652 P 186b12638a87410487467df721a3cbda: Bootstrap starting.
I20260709 13:03:56.292449 10878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72b111ad7b5f4753ab1d2b5ee1114840" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
W20260709 13:03:56.303668 10995 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:38967, user_credentials={real_user=slave}} blocked reactor thread for 82361.6us
I20260709 13:03:56.291471 11520 raft_consensus.cc:2749] T b4cd3d7dfbbf4096908ff11e9dc57141 P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:03:56.328743 10995 leader_election.cc:304] T 4c2a35bb99374e93b573cf870d04faaa P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47; no voters: 10ca5ee68ad64b3283b9d235352c9d23
I20260709 13:03:56.329819 10876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a69f32c53f24a50b7852d143665d7b4" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:03:56.330265 10995 leader_election.cc:304] T 9a1716c0adf34bee8a4e99341ba5788f P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:03:56.331815 10995 leader_election.cc:304] T d371c5e27bd843469d2e2e453734cb4a P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:03:56.304690 11508 raft_consensus.cc:515] T a610d1a71f52432e88fe56130f921fa2 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:56.305652 11095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d439326f23747eaab3c1e7da2d6b785" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:03:56.335489 11025 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2567c2d84e4648d7ba5dc23a3df32817" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
I20260709 13:03:56.305792 11539 raft_consensus.cc:2804] T f371d2565dd84667b5613848604949bb P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:03:56.306393 10952 raft_consensus.cc:2393] T e5a5f5a608504650b48fd27f163faed6 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0172fed22b6b4890a7fdc953fa128af4 in current term 1: Already voted for candidate 186b12638a87410487467df721a3cbda in this term.
I20260709 13:03:56.308126 11098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a610d1a71f52432e88fe56130f921fa2" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:03:56.339447 10947 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a69f32c53f24a50b7852d143665d7b4" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:03:56.340215 11097 raft_consensus.cc:1240] T 525bfdbd6d934afd823f7d3ee3f9d052 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Rejecting Update request from peer 10ca5ee68ad64b3283b9d235352c9d23 for earlier term 1. Current term is 2. Ops: []
I20260709 13:03:56.286808 10877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a9353d1d230416296627d7cf8b1d425" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:03:56.309233 11102 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "836251ebc2b74f5da88b8a8b2338e2a5" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:03:56.336802 11095 raft_consensus.cc:2468] T 5d439326f23747eaab3c1e7da2d6b785 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 1.
I20260709 13:03:56.342507 11025 raft_consensus.cc:3060] T 2567c2d84e4648d7ba5dc23a3df32817 P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:03:56.290675 10948 raft_consensus.cc:2468] T a154cc8b181f460abf4880064379948b P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 0.
I20260709 13:03:56.343300 11360 leader_election.cc:290] T 4b37f8fca8dc42718b2a42f0883a956f P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 1 election: Requested vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:03:56.311937 11100 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb7d67be500947b3aef4c4245e33671c" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:03:56.336997 11539 raft_consensus.cc:493] T f371d2565dd84667b5613848604949bb P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:56.344213 11100 raft_consensus.cc:2376] T bb7d67be500947b3aef4c4245e33671c P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 873357742d5843579556ee8166f23f47 in term 1. Re-sending same reply.
W20260709 13:03:56.313522 10994 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 81420.6us
I20260709 13:03:56.313189 11094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5de5f0581e3b41dea729363d6ca563bb" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:03:56.343318 11525 consensus_queue.cc:1059] T 63d44f6ba6434ab2870fe3190f4d7c2f P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
I20260709 13:03:56.345345 11510 raft_consensus.cc:515] T a64b7e99d6c54934a7de6f6dffcf5e94 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:56.346333 11525 raft_consensus.cc:3055] T 63d44f6ba6434ab2870fe3190f4d7c2f P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:03:56.315779 10954 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11de93cd9b9d457489ff1cb80f976c9a" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:03:56.346829 11525 raft_consensus.cc:740] T 63d44f6ba6434ab2870fe3190f4d7c2f P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 10ca5ee68ad64b3283b9d235352c9d23, State: Running, Role: LEADER
I20260709 13:03:56.318004 10951 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f9df2bc281f4866b6da5da94a0defc7" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:03:56.319114 11103 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555217d45d9b4ee492dbb3ddaf0f7c02" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:03:56.347808 11525 consensus_queue.cc:260] T 63d44f6ba6434ab2870fe3190f4d7c2f P 10ca5ee68ad64b3283b9d235352c9d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:56.349124 11029 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b37f8fca8dc42718b2a42f0883a956f" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
I20260709 13:03:56.349841 11029 raft_consensus.cc:3060] T 4b37f8fca8dc42718b2a42f0883a956f P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:03:56.314738 11541 raft_consensus.cc:2804] T 32147c6d2d5a477aa38c597c7f78df5b P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:03:56.320055 10946 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c51782cd9d064ec0949e84e24dd4de67" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:03:56.350889 11541 raft_consensus.cc:493] T 32147c6d2d5a477aa38c597c7f78df5b P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:56.321969 11096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72b111ad7b5f4753ab1d2b5ee1114840" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:03:56.351310 11541 raft_consensus.cc:3060] T 32147c6d2d5a477aa38c597c7f78df5b P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:03:56.352941 10875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f9eb06ed01f4ce68be3af8a8434665b" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:03:56.353480 11503 leader_election.cc:290] T 0e5d327630e5438b878035d28efe6c0c P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 2 election: Requested vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:03:56.333595 11506 raft_consensus.cc:3060] T 65bf0a62f4d84ef18dd6b23e082724b8 P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:03:56.354154 10880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5a5f5a608504650b48fd27f163faed6" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:03:56.355015 10877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87cd8f5097624c8692f1d83544d8881e" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:03:56.295238 10950 raft_consensus.cc:2468] T 87cd8f5097624c8692f1d83544d8881e P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 1.
I20260709 13:03:56.355567 11510 leader_election.cc:290] T a64b7e99d6c54934a7de6f6dffcf5e94 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 2 election: Requested vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:03:56.355588 10873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f371d2565dd84667b5613848604949bb" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:03:56.323246 11099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32147c6d2d5a477aa38c597c7f78df5b" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:03:56.356447 10873 raft_consensus.cc:2468] T f371d2565dd84667b5613848604949bb P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 0.
I20260709 13:03:56.324652 11152 tablet_bootstrap.cc:492] T 8c160ddfe9b34bdaa3835d0915c1e06f P 0172fed22b6b4890a7fdc953fa128af4: Bootstrap starting.
I20260709 13:03:56.361368 11103 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bac7e5040f984ba2b7146bd36af7267e" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:03:56.361810 11094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "836251ebc2b74f5da88b8a8b2338e2a5" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:03:56.324766 11523 raft_consensus.cc:493] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:56.362834 11096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f46a5102fd64b45bf1c4dff863d2b4d" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:03:56.319231 10953 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63d44f6ba6434ab2870fe3190f4d7c2f" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:03:56.287614 11477 raft_consensus.cc:515] T e89e640169e14e2d98f968f76027daa7 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:56.363867 11099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a64b7e99d6c54934a7de6f6dffcf5e94" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:03:56.333910 11520 raft_consensus.cc:2804] T 4c2a35bb99374e93b573cf870d04faaa P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:03:56.365408 11024 raft_consensus.cc:1240] T e89e640169e14e2d98f968f76027daa7 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Rejecting Update request from peer 10ca5ee68ad64b3283b9d235352c9d23 for earlier term 1. Current term is 2. Ops: []
I20260709 13:03:56.337381 11148 raft_consensus.cc:359] T ff13ae9a47c7455f962406c6d131a3ff P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:56.365770 11520 raft_consensus.cc:493] T 4c2a35bb99374e93b573cf870d04faaa P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 10ca5ee68ad64b3283b9d235352c9d23)
I20260709 13:03:56.366058 11148 raft_consensus.cc:385] T ff13ae9a47c7455f962406c6d131a3ff P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:56.334632 11534 raft_consensus.cc:2804] T 9a1716c0adf34bee8a4e99341ba5788f P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:03:56.366356 11520 raft_consensus.cc:3060] T 4c2a35bb99374e93b573cf870d04faaa P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:03:56.328168 10878 raft_consensus.cc:2468] T 72b111ad7b5f4753ab1d2b5ee1114840 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 1.
I20260709 13:03:56.366866 11534 raft_consensus.cc:493] T 9a1716c0adf34bee8a4e99341ba5788f P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0172fed22b6b4890a7fdc953fa128af4)
W20260709 13:03:56.337811 10845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 44377.7us
I20260709 13:03:56.367358 11534 raft_consensus.cc:3060] T 9a1716c0adf34bee8a4e99341ba5788f P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:03:56.368199 10845 leader_election.cc:304] T 4f9eb06ed01f4ce68be3af8a8434665b P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda; no voters: 
I20260709 13:03:56.338955 11518 raft_consensus.cc:2804] T 8a69f32c53f24a50b7852d143665d7b4 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:03:56.338757 11516 raft_consensus.cc:515] T e5ba72ccd854430baadbc636dfd17136 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:56.344326 11539 raft_consensus.cc:3060] T f371d2565dd84667b5613848604949bb P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:03:56.314029 10949 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7839ca2d2c13431a85f44d7c458c6351" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:03:56.362761 11523 raft_consensus.cc:515] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:56.375473 11094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f9df2bc281f4866b6da5da94a0defc7" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:03:56.376828 10921 leader_election.cc:304] T 7839ca2d2c13431a85f44d7c458c6351 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23; no voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47
I20260709 13:03:56.346619 10994 leader_election.cc:304] T 525bfdbd6d934afd823f7d3ee3f9d052 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 873357742d5843579556ee8166f23f47; no voters: 0172fed22b6b4890a7fdc953fa128af4, 10ca5ee68ad64b3283b9d235352c9d23
I20260709 13:03:56.366245 10876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7757ecbf402e47bfbc93d0447741e26c" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:03:56.378504 11524 raft_consensus.cc:2749] T 7839ca2d2c13431a85f44d7c458c6351 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:03:56.380846 10921 leader_election.cc:304] T 11de93cd9b9d457489ff1cb80f976c9a P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda; no voters: 873357742d5843579556ee8166f23f47
I20260709 13:03:56.366456 11148 raft_consensus.cc:740] T ff13ae9a47c7455f962406c6d131a3ff P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10ca5ee68ad64b3283b9d235352c9d23, State: Initialized, Role: FOLLOWER
I20260709 13:03:56.382956 11525 raft_consensus.cc:3060] T 63d44f6ba6434ab2870fe3190f4d7c2f P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:03:56.383757 11069 leader_election.cc:304] T d371c5e27bd843469d2e2e453734cb4a P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda; no voters: 
I20260709 13:03:56.383677 11148 consensus_queue.cc:260] T ff13ae9a47c7455f962406c6d131a3ff P 10ca5ee68ad64b3283b9d235352c9d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:56.365200 11477 leader_election.cc:290] T e89e640169e14e2d98f968f76027daa7 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 2 election: Requested vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:03:56.385071 11427 consensus_queue.cc:1048] T 5de5f0581e3b41dea729363d6ca563bb P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:03:56.386322 11152 tablet_bootstrap.cc:654] T 8c160ddfe9b34bdaa3835d0915c1e06f P 0172fed22b6b4890a7fdc953fa128af4: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:56.386724 11512 raft_consensus.cc:2804] T d371c5e27bd843469d2e2e453734cb4a P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:03:56.387146 11512 raft_consensus.cc:493] T d371c5e27bd843469d2e2e453734cb4a P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:56.387495 11512 raft_consensus.cc:3060] T d371c5e27bd843469d2e2e453734cb4a P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:03:56.375353 11528 raft_consensus.cc:2804] T a154cc8b181f460abf4880064379948b P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:03:56.377969 11523 leader_election.cc:290] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:03:56.389720 11523 raft_consensus.cc:2804] T 4f9eb06ed01f4ce68be3af8a8434665b P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:03:56.390069 11523 raft_consensus.cc:493] T 4f9eb06ed01f4ce68be3af8a8434665b P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Starting leader election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:03:56.390645 11523 raft_consensus.cc:3060] T 4f9eb06ed01f4ce68be3af8a8434665b P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:03:56.391036 11528 raft_consensus.cc:493] T a154cc8b181f460abf4880064379948b P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:56.391494 11528 raft_consensus.cc:3060] T a154cc8b181f460abf4880064379948b P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Advancing to term 1
W20260709 13:03:56.286580 10992 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 79762.5us
I20260709 13:03:56.370275 11518 raft_consensus.cc:493] T 8a69f32c53f24a50b7852d143665d7b4 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Starting leader election (detected failure of leader 10ca5ee68ad64b3283b9d235352c9d23)
I20260709 13:03:56.365139 11508 leader_election.cc:290] T a610d1a71f52432e88fe56130f921fa2 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:03:56.377952 11516 leader_election.cc:290] T e5ba72ccd854430baadbc636dfd17136 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 2 election: Requested vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:03:56.397053 10878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5a5f5a608504650b48fd27f163faed6" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:03:56.400964 11148 ts_tablet_manager.cc:1434] T ff13ae9a47c7455f962406c6d131a3ff P 10ca5ee68ad64b3283b9d235352c9d23: Time spent starting tablet: real 0.124s	user 0.009s	sys 0.000s
I20260709 13:03:56.395048 11069 leader_election.cc:304] T 868e92339c2f4508af0afbd07c814d64 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4; no voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47
I20260709 13:03:56.392005 11540 raft_consensus.cc:2804] T d371c5e27bd843469d2e2e453734cb4a P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:03:56.404806 11151 tablet_bootstrap.cc:654] T c31881534c5e4f0089c82057c1746652 P 186b12638a87410487467df721a3cbda: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:56.404726 10950 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72b111ad7b5f4753ab1d2b5ee1114840" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:03:56.400763 11518 raft_consensus.cc:3060] T 8a69f32c53f24a50b7852d143665d7b4 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:03:56.419273 10877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b37f8fca8dc42718b2a42f0883a956f" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:03:56.419963 10877 raft_consensus.cc:3060] T 4b37f8fca8dc42718b2a42f0883a956f P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Advancing to term 1
W20260709 13:03:56.406893 10994 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 60391us
I20260709 13:03:56.407009 11524 raft_consensus.cc:2804] T 11de93cd9b9d457489ff1cb80f976c9a P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:03:56.392359 10992 leader_election.cc:304] T 4b37f8fca8dc42718b2a42f0883a956f P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:03:56.410946 10875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2567c2d84e4648d7ba5dc23a3df32817" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:03:56.413097 11099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d270b9e5481245d9b5340a454a7cd10b" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:03:56.431274 10876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87cd8f5097624c8692f1d83544d8881e" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:03:56.413370 11096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a154cc8b181f460abf4880064379948b" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:03:56.423609 10921 leader_election.cc:304] T 5de5f0581e3b41dea729363d6ca563bb P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda; no voters: 
I20260709 13:03:56.403148 11477 raft_consensus.cc:3060] T 525bfdbd6d934afd823f7d3ee3f9d052 P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:03:56.416606 11513 raft_consensus.cc:2749] T 868e92339c2f4508af0afbd07c814d64 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 13:03:56.431217 10992 leader_election.cc:343] T ff13ae9a47c7455f962406c6d131a3ff P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:03:56.432719 11096 raft_consensus.cc:2468] T a154cc8b181f460abf4880064379948b P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 0.
I20260709 13:03:56.434031 10992 leader_election.cc:304] T 6f46a5102fd64b45bf1c4dff863d2b4d P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:03:56.423823 11148 tablet_bootstrap.cc:492] T c31881534c5e4f0089c82057c1746652 P 10ca5ee68ad64b3283b9d235352c9d23: Bootstrap starting.
I20260709 13:03:56.436645 10946 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5de5f0581e3b41dea729363d6ca563bb" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:03:56.437076 11510 raft_consensus.cc:2804] T 6f46a5102fd64b45bf1c4dff863d2b4d P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:03:56.437424 10946 raft_consensus.cc:2468] T 5de5f0581e3b41dea729363d6ca563bb P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 1.
I20260709 13:03:56.437565 11510 raft_consensus.cc:493] T 6f46a5102fd64b45bf1c4dff863d2b4d P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0172fed22b6b4890a7fdc953fa128af4)
I20260709 13:03:56.428082 11029 raft_consensus.cc:2468] T 4b37f8fca8dc42718b2a42f0883a956f P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 1.
I20260709 13:03:56.424556 11504 consensus_queue.cc:1048] T 0e5d327630e5438b878035d28efe6c0c P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.006s
I20260709 13:03:56.438797 11527 raft_consensus.cc:2804] T 5de5f0581e3b41dea729363d6ca563bb P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:03:56.439227 11527 raft_consensus.cc:493] T 5de5f0581e3b41dea729363d6ca563bb P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:56.439646 11527 raft_consensus.cc:3060] T 5de5f0581e3b41dea729363d6ca563bb P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:03:56.433008 11103 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87cd8f5097624c8692f1d83544d8881e" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:03:56.440328 11103 raft_consensus.cc:2468] T 87cd8f5097624c8692f1d83544d8881e P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 1.
I20260709 13:03:56.424716 11524 raft_consensus.cc:493] T 11de93cd9b9d457489ff1cb80f976c9a P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:03:56.439299 10919 leader_election.cc:304] T 4b37f8fca8dc42718b2a42f0883a956f P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:03:56.438050 11510 raft_consensus.cc:3060] T 6f46a5102fd64b45bf1c4dff863d2b4d P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Advancing to term 2
W20260709 13:03:56.462693 10843 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 137863us
I20260709 13:03:56.468640 10843 leader_election.cc:304] T 0f9df2bc281f4866b6da5da94a0defc7 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 186b12638a87410487467df721a3cbda; no voters: 0172fed22b6b4890a7fdc953fa128af4, 10ca5ee68ad64b3283b9d235352c9d23
I20260709 13:03:56.469866 11508 raft_consensus.cc:2749] T 0f9df2bc281f4866b6da5da94a0defc7 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:03:56.443034 10921 leader_election.cc:304] T 836251ebc2b74f5da88b8a8b2338e2a5 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda; no voters: 
I20260709 13:03:56.445509 11511 raft_consensus.cc:515] T bac7e5040f984ba2b7146bd36af7267e P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:56.474162 11511 leader_election.cc:290] T bac7e5040f984ba2b7146bd36af7267e P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 2 election: Requested vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:03:56.474046 11508 consensus_queue.cc:1048] T e5a5f5a608504650b48fd27f163faed6 P 186b12638a87410487467df721a3cbda [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:03:56.476179 10843 leader_election.cc:304] T 5de5f0581e3b41dea729363d6ca563bb P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda; no voters: 
I20260709 13:03:56.442772 11516 raft_consensus.cc:2764] T 4b37f8fca8dc42718b2a42f0883a956f P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260709 13:03:56.446785 11094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5de5f0581e3b41dea729363d6ca563bb" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:03:56.479580 11521 raft_consensus.cc:2804] T 836251ebc2b74f5da88b8a8b2338e2a5 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:03:56.436408 10992 leader_election.cc:304] T 732c4adeeff6422497aa7db4a82f535b P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:03:56.447990 11529 raft_consensus.cc:515] T a64b7e99d6c54934a7de6f6dffcf5e94 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:56.480000 11521 raft_consensus.cc:493] T 836251ebc2b74f5da88b8a8b2338e2a5 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0172fed22b6b4890a7fdc953fa128af4)
I20260709 13:03:56.480465 11521 raft_consensus.cc:3060] T 836251ebc2b74f5da88b8a8b2338e2a5 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:03:56.482162 11529 leader_election.cc:290] T a64b7e99d6c54934a7de6f6dffcf5e94 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 2 election: Requested vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:03:56.449432 11534 raft_consensus.cc:515] T 9a1716c0adf34bee8a4e99341ba5788f P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:56.484534 11534 leader_election.cc:290] T 9a1716c0adf34bee8a4e99341ba5788f P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 2 election: Requested vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:03:56.488912 11534 raft_consensus.cc:2804] T 732c4adeeff6422497aa7db4a82f535b P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:03:56.489444 11534 raft_consensus.cc:697] T 732c4adeeff6422497aa7db4a82f535b P 873357742d5843579556ee8166f23f47 [term 1 LEADER]: Becoming Leader. State: Replica: 873357742d5843579556ee8166f23f47, State: Running, Role: LEADER
I20260709 13:03:56.490427 11534 consensus_queue.cc:237] T 732c4adeeff6422497aa7db4a82f535b P 873357742d5843579556ee8166f23f47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:56.490833 11521 raft_consensus.cc:515] T 836251ebc2b74f5da88b8a8b2338e2a5 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:56.493093 11521 leader_election.cc:290] T 836251ebc2b74f5da88b8a8b2338e2a5 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 2 election: Requested vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
W20260709 13:03:56.494787 10992 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 58462.4us
I20260709 13:03:56.503947 10921 leader_election.cc:304] T d270b9e5481245d9b5340a454a7cd10b P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda; no voters: 
I20260709 13:03:56.505189 11521 raft_consensus.cc:2804] T d270b9e5481245d9b5340a454a7cd10b P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:03:56.505769 11521 raft_consensus.cc:493] T d270b9e5481245d9b5340a454a7cd10b P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0172fed22b6b4890a7fdc953fa128af4)
I20260709 13:03:56.506276 11521 raft_consensus.cc:3060] T d270b9e5481245d9b5340a454a7cd10b P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:03:56.424036 11069 leader_election.cc:304] T 6a9353d1d230416296627d7cf8b1d425 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4; no voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47
I20260709 13:03:56.453971 11519 consensus_queue.cc:1059] T 525bfdbd6d934afd823f7d3ee3f9d052 P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:03:56.520220 11519 raft_consensus.cc:3055] T 525bfdbd6d934afd823f7d3ee3f9d052 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:03:56.520814 11519 raft_consensus.cc:740] T 525bfdbd6d934afd823f7d3ee3f9d052 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 10ca5ee68ad64b3283b9d235352c9d23, State: Running, Role: LEADER
I20260709 13:03:56.455902 10920 leader_election.cc:304] T bac7e5040f984ba2b7146bd36af7267e P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23; no voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47
W20260709 13:03:56.523375 10920 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 67571.5us
I20260709 13:03:56.525054 10920 leader_election.cc:304] T 6f46a5102fd64b45bf1c4dff863d2b4d P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23; no voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47
I20260709 13:03:56.422574 11540 raft_consensus.cc:493] T d371c5e27bd843469d2e2e453734cb4a P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:56.526726 11540 raft_consensus.cc:3060] T d371c5e27bd843469d2e2e453734cb4a P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:03:56.530725 11430 raft_consensus.cc:2749] T bac7e5040f984ba2b7146bd36af7267e P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:03:56.531807 11430 raft_consensus.cc:2749] T 6f46a5102fd64b45bf1c4dff863d2b4d P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:03:56.522018 11519 consensus_queue.cc:260] T 525bfdbd6d934afd823f7d3ee3f9d052 P 10ca5ee68ad64b3283b9d235352c9d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:56.534988 11519 raft_consensus.cc:3060] T 525bfdbd6d934afd823f7d3ee3f9d052 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Advancing to term 2
W20260709 13:03:56.541875 11124 debug-util.cc:398] Leaking SignalData structure 0x7b0800155740 after lost signal to thread 434
W20260709 13:03:56.543941 11124 debug-util.cc:398] Leaking SignalData structure 0x7b08003ac020 after lost signal to thread 10812
I20260709 13:03:56.545472 10920 leader_election.cc:304] T 0f9df2bc281f4866b6da5da94a0defc7 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23; no voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda
W20260709 13:03:56.546303 11124 debug-util.cc:398] Leaking SignalData structure 0x7b08002639a0 after lost signal to thread 10904
W20260709 13:03:56.548641 11124 debug-util.cc:398] Leaking SignalData structure 0x7b08002216e0 after lost signal to thread 10979
I20260709 13:03:56.548684 11430 raft_consensus.cc:2749] T 0f9df2bc281f4866b6da5da94a0defc7 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 13:03:56.551308 11124 debug-util.cc:398] Leaking SignalData structure 0x7b0800184fc0 after lost signal to thread 11053
W20260709 13:03:56.551752 11124 debug-util.cc:398] Leaking SignalData structure 0x7b08003ba340 after lost signal to thread 11054
W20260709 13:03:56.553669 11124 debug-util.cc:398] Leaking SignalData structure 0x7b08002cbdc0 after lost signal to thread 11127
W20260709 13:03:56.554076 11124 debug-util.cc:398] Leaking SignalData structure 0x7b08003e4f00 after lost signal to thread 11128
W20260709 13:03:56.570964 10921 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:38967, user_credentials={real_user=slave}} blocked reactor thread for 67122.9us
I20260709 13:03:56.576807 10876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "836251ebc2b74f5da88b8a8b2338e2a5" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:03:56.410301 11025 raft_consensus.cc:2468] T 2567c2d84e4648d7ba5dc23a3df32817 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 2.
W20260709 13:03:56.584586 11069 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:38967, user_credentials={real_user=slave}} blocked reactor thread for 160643us
W20260709 13:03:56.590713 10995 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:38967, user_credentials={real_user=slave}} blocked reactor thread for 258011us
I20260709 13:03:56.442600 11360 raft_consensus.cc:493] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:56.591909 10995 leader_election.cc:304] T e96e80ba9ac34d0998395515e5c8c41a P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47; no voters: 10ca5ee68ad64b3283b9d235352c9d23
I20260709 13:03:56.597255 11069 leader_election.cc:304] T f371d2565dd84667b5613848604949bb P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda; no voters: 
I20260709 13:03:56.603434 11540 raft_consensus.cc:515] T d371c5e27bd843469d2e2e453734cb4a P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:56.458942 11524 raft_consensus.cc:3060] T 11de93cd9b9d457489ff1cb80f976c9a P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:03:56.456807 11527 raft_consensus.cc:515] T 5de5f0581e3b41dea729363d6ca563bb P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:56.453810 11541 raft_consensus.cc:515] T 32147c6d2d5a477aa38c597c7f78df5b P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:56.428921 11522 consensus_queue.cc:1059] T e89e640169e14e2d98f968f76027daa7 P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.168s
I20260709 13:03:56.592527 11513 raft_consensus.cc:493] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:56.604955 11529 raft_consensus.cc:2804] T 4b37f8fca8dc42718b2a42f0883a956f P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:03:56.456637 11539 raft_consensus.cc:515] T f371d2565dd84667b5613848604949bb P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
W20260709 13:03:56.593942 10919 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:45807, user_credentials={real_user=slave}} blocked reactor thread for 154766us
I20260709 13:03:56.604977 11425 raft_consensus.cc:2749] T 6a9353d1d230416296627d7cf8b1d425 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:03:56.632570 11029 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f9eb06ed01f4ce68be3af8a8434665b" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:03:56.604787 11096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5a5f5a608504650b48fd27f163faed6" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
W20260709 13:03:56.616807 10994 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 155906us
I20260709 13:03:56.650342 10994 leader_election.cc:304] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47; no voters: 10ca5ee68ad64b3283b9d235352c9d23
W20260709 13:03:56.608525 10845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 240428us
I20260709 13:03:56.661083 11151 tablet_bootstrap.cc:492] T c31881534c5e4f0089c82057c1746652 P 186b12638a87410487467df721a3cbda: No bootstrap required, opened a new log
W20260709 13:03:56.673631 11069 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:38967, user_credentials={real_user=slave}} blocked reactor thread for 76471.5us
I20260709 13:03:56.680368 11151 ts_tablet_manager.cc:1403] T c31881534c5e4f0089c82057c1746652 P 186b12638a87410487467df721a3cbda: Time spent bootstrapping tablet: real 0.380s	user 0.044s	sys 0.058s
I20260709 13:03:56.680948 11069 leader_election.cc:304] T 72b111ad7b5f4753ab1d2b5ee1114840 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda; no voters: 
I20260709 13:03:56.612573 11360 raft_consensus.cc:515] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
W20260709 13:03:56.682762 10920 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 135083us
I20260709 13:03:56.684492 10920 leader_election.cc:304] T 87cd8f5097624c8692f1d83544d8881e P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 10ca5ee68ad64b3283b9d235352c9d23; no voters: 186b12638a87410487467df721a3cbda
I20260709 13:03:56.685817 11029 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d71ca3a1ba6b48478aaf62741bf3cf6a" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:03:56.687561 11029 raft_consensus.cc:1240] T a64b7e99d6c54934a7de6f6dffcf5e94 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Rejecting Update request from peer 0172fed22b6b4890a7fdc953fa128af4 for earlier term 1. Current term is 2. Ops: []
I20260709 13:03:56.629081 11518 raft_consensus.cc:515] T 8a69f32c53f24a50b7852d143665d7b4 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:56.674412 10951 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5ba72ccd854430baadbc636dfd17136" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:03:56.673694 11534 raft_consensus.cc:2804] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:03:56.662744 11096 raft_consensus.cc:2468] T e5a5f5a608504650b48fd27f163faed6 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 0.
I20260709 13:03:56.678088 11425 raft_consensus.cc:2804] T f371d2565dd84667b5613848604949bb P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:03:56.643810 11513 raft_consensus.cc:515] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:56.664352 11505 raft_consensus.cc:515] T 34064dc7bf3747dbb3876dd824ee0688 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:56.742311 11149 tablet_bootstrap.cc:492] T 8c160ddfe9b34bdaa3835d0915c1e06f P 873357742d5843579556ee8166f23f47: No bootstrap required, opened a new log
I20260709 13:03:56.743232 11149 ts_tablet_manager.cc:1403] T 8c160ddfe9b34bdaa3835d0915c1e06f P 873357742d5843579556ee8166f23f47: Time spent bootstrapping tablet: real 1.271s	user 0.014s	sys 0.022s
I20260709 13:03:56.728089 11027 raft_consensus.cc:1240] T 0e5d327630e5438b878035d28efe6c0c P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Rejecting Update request from peer 0172fed22b6b4890a7fdc953fa128af4 for earlier term 1. Current term is 2. Ops: []
W20260709 13:03:56.707729 11066 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 100844us
I20260709 13:03:56.688700 11427 consensus_queue.cc:1048] T 5de5f0581e3b41dea729363d6ca563bb P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.049s
I20260709 13:03:56.730273 11534 raft_consensus.cc:493] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 10ca5ee68ad64b3283b9d235352c9d23)
I20260709 13:03:56.731611 11425 raft_consensus.cc:493] T f371d2565dd84667b5613848604949bb P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:56.772887 11540 leader_election.cc:290] T d371c5e27bd843469d2e2e453734cb4a P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 1 election: Requested vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:03:56.763741 11529 raft_consensus.cc:697] T 4b37f8fca8dc42718b2a42f0883a956f P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 LEADER]: Becoming Leader. State: Replica: 10ca5ee68ad64b3283b9d235352c9d23, State: Running, Role: LEADER
I20260709 13:03:56.774555 11529 consensus_queue.cc:237] T 4b37f8fca8dc42718b2a42f0883a956f P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:56.775602 11151 raft_consensus.cc:359] T c31881534c5e4f0089c82057c1746652 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:56.776084 11541 leader_election.cc:290] T 32147c6d2d5a477aa38c597c7f78df5b P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 2 election: Requested vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:03:56.776459 11151 raft_consensus.cc:385] T c31881534c5e4f0089c82057c1746652 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:56.776840 11151 raft_consensus.cc:740] T c31881534c5e4f0089c82057c1746652 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 186b12638a87410487467df721a3cbda, State: Initialized, Role: FOLLOWER
I20260709 13:03:56.777534 11360 leader_election.cc:290] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:03:56.775696 11518 leader_election.cc:290] T 8a69f32c53f24a50b7852d143665d7b4 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 2 election: Requested vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:03:56.767251 11425 raft_consensus.cc:3060] T f371d2565dd84667b5613848604949bb P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:03:56.777637 11151 consensus_queue.cc:260] T c31881534c5e4f0089c82057c1746652 P 186b12638a87410487467df721a3cbda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:56.783581 11548 raft_consensus.cc:2804] T e96e80ba9ac34d0998395515e5c8c41a P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:03:56.790565 11548 raft_consensus.cc:493] T e96e80ba9ac34d0998395515e5c8c41a P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 10ca5ee68ad64b3283b9d235352c9d23)
I20260709 13:03:56.791257 11545 raft_consensus.cc:493] T a5299b8d044f41b7b68057e63d17b159 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:56.791898 11545 raft_consensus.cc:515] T a5299b8d044f41b7b68057e63d17b159 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:56.793103 11553 raft_consensus.cc:2804] T 87cd8f5097624c8692f1d83544d8881e P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:03:56.793534 11553 raft_consensus.cc:493] T 87cd8f5097624c8692f1d83544d8881e P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:56.793874 11553 raft_consensus.cc:3060] T 87cd8f5097624c8692f1d83544d8881e P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:03:56.794334 11527 leader_election.cc:290] T 5de5f0581e3b41dea729363d6ca563bb P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 2 election: Requested vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:03:56.761612 11534 raft_consensus.cc:3060] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:03:56.790390 11149 raft_consensus.cc:359] T 8c160ddfe9b34bdaa3835d0915c1e06f P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:56.799602 11149 raft_consensus.cc:385] T 8c160ddfe9b34bdaa3835d0915c1e06f P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:56.799957 11149 raft_consensus.cc:740] T 8c160ddfe9b34bdaa3835d0915c1e06f P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 873357742d5843579556ee8166f23f47, State: Initialized, Role: FOLLOWER
I20260709 13:03:56.800855 11149 consensus_queue.cc:260] T 8c160ddfe9b34bdaa3835d0915c1e06f P 873357742d5843579556ee8166f23f47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:56.802609 11148 tablet_bootstrap.cc:654] T c31881534c5e4f0089c82057c1746652 P 10ca5ee68ad64b3283b9d235352c9d23: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:56.807914 11505 leader_election.cc:290] T 34064dc7bf3747dbb3876dd824ee0688 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 2 election: Requested vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:03:56.809913 11513 leader_election.cc:290] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
W20260709 13:03:56.783504 11542 log.cc:927] Time spent T bb7d67be500947b3aef4c4245e33671c P 873357742d5843579556ee8166f23f47: Append to log took a long time: real 0.100s	user 0.000s	sys 0.001s
I20260709 13:03:56.812445 11555 raft_consensus.cc:2804] T 72b111ad7b5f4753ab1d2b5ee1114840 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:03:56.813063 11555 raft_consensus.cc:493] T 72b111ad7b5f4753ab1d2b5ee1114840 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 10ca5ee68ad64b3283b9d235352c9d23)
I20260709 13:03:56.813560 11555 raft_consensus.cc:3060] T 72b111ad7b5f4753ab1d2b5ee1114840 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:03:56.813328 11520 raft_consensus.cc:515] T 4c2a35bb99374e93b573cf870d04faaa P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:56.766592 11506 raft_consensus.cc:515] T 65bf0a62f4d84ef18dd6b23e082724b8 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:56.817036 11528 raft_consensus.cc:515] T a154cc8b181f460abf4880064379948b P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:56.808009 11539 leader_election.cc:290] T f371d2565dd84667b5613848604949bb P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 1 election: Requested vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:03:56.820238 11520 leader_election.cc:290] T 4c2a35bb99374e93b573cf870d04faaa P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 2 election: Requested vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:03:56.787214 11521 raft_consensus.cc:515] T d270b9e5481245d9b5340a454a7cd10b P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:56.838688 11477 raft_consensus.cc:2749] T 525bfdbd6d934afd823f7d3ee3f9d052 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:03:56.839460 11521 leader_election.cc:290] T d270b9e5481245d9b5340a454a7cd10b P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 2 election: Requested vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:03:56.841549 11523 raft_consensus.cc:515] T 4f9eb06ed01f4ce68be3af8a8434665b P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:56.791347 11548 raft_consensus.cc:3060] T e96e80ba9ac34d0998395515e5c8c41a P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:03:56.790490 11510 raft_consensus.cc:515] T 6f46a5102fd64b45bf1c4dff863d2b4d P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:56.848523 11510 leader_election.cc:290] T 6f46a5102fd64b45bf1c4dff863d2b4d P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 2 election: Requested vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:03:56.843250 11512 raft_consensus.cc:515] T d371c5e27bd843469d2e2e453734cb4a P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:56.850736 11512 leader_election.cc:290] T d371c5e27bd843469d2e2e453734cb4a P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 1 election: Requested vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:03:56.852273 11523 leader_election.cc:290] T 4f9eb06ed01f4ce68be3af8a8434665b P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 2 election: Requested vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:03:56.918620 10875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "732c4adeeff6422497aa7db4a82f535b" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:03:56.921098 10875 raft_consensus.cc:3060] T 732c4adeeff6422497aa7db4a82f535b P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:03:56.925491 10878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b37f8fca8dc42718b2a42f0883a956f" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:03:56.913188 10845 leader_election.cc:304] T c51782cd9d064ec0949e84e24dd4de67 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda; no voters: 10ca5ee68ad64b3283b9d235352c9d23
W20260709 13:03:56.932499 10994 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 282087us
I20260709 13:03:56.695456 10946 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63d44f6ba6434ab2870fe3190f4d7c2f" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:03:56.934506 10876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555217d45d9b4ee492dbb3ddaf0f7c02" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
W20260709 13:03:56.734907 11067 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:45807, user_credentials={real_user=slave}} blocked reactor thread for 117125us
I20260709 13:03:56.744463 11022 raft_consensus.cc:1240] T 32147c6d2d5a477aa38c597c7f78df5b P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Rejecting Update request from peer 0172fed22b6b4890a7fdc953fa128af4 for earlier term 1. Current term is 2. Ops: []
I20260709 13:03:56.937767 11067 leader_election.cc:304] T 7757ecbf402e47bfbc93d0447741e26c P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4; no voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47
I20260709 13:03:56.938725 11021 raft_consensus.cc:2393] T 34064dc7bf3747dbb3876dd824ee0688 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 10ca5ee68ad64b3283b9d235352c9d23 in current term 2: Already voted for candidate 873357742d5843579556ee8166f23f47 in this term.
I20260709 13:03:56.939512 11023 raft_consensus.cc:1240] T 65bf0a62f4d84ef18dd6b23e082724b8 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Rejecting Update request from peer 10ca5ee68ad64b3283b9d235352c9d23 for earlier term 1. Current term is 2. Ops: []
W20260709 13:03:56.940384 11066 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 171153us
I20260709 13:03:56.951774 11029 raft_consensus.cc:1275] T a610d1a71f52432e88fe56130f921fa2 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Refusing update from remote peer 0172fed22b6b4890a7fdc953fa128af4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260709 13:03:56.945501 11025 raft_consensus.cc:1240] T 7709387f4d3d4f9992d97a0fbca39a40 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Rejecting Update request from peer 0172fed22b6b4890a7fdc953fa128af4 for earlier term 1. Current term is 2. Ops: []
I20260709 13:03:56.945631 10953 raft_consensus.cc:2468] T 63d44f6ba6434ab2870fe3190f4d7c2f P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 2.
I20260709 13:03:56.947045 11026 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "525bfdbd6d934afd823f7d3ee3f9d052" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
I20260709 13:03:56.947093 10879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a610d1a71f52432e88fe56130f921fa2" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:03:56.969383 10879 raft_consensus.cc:2468] T a610d1a71f52432e88fe56130f921fa2 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 2.
W20260709 13:03:56.970688 11152 thread.cc:641] log-alloc [worker] (thread pool) Time spent creating pthread: real 0.551s	user 0.008s	sys 0.021s
W20260709 13:03:56.971436 11152 thread.cc:608] log-alloc [worker] (thread pool) Time spent starting thread: real 0.557s	user 0.008s	sys 0.021s
I20260709 13:03:56.972316 11528 leader_election.cc:290] T a154cc8b181f460abf4880064379948b P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 1 election: Requested vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:03:56.949155 11027 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a1716c0adf34bee8a4e99341ba5788f" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:03:56.983831 11029 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7839ca2d2c13431a85f44d7c458c6351" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:03:56.985280 11022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a9353d1d230416296627d7cf8b1d425" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:03:56.987666 11022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63d44f6ba6434ab2870fe3190f4d7c2f" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:03:56.987983 11024 raft_consensus.cc:1240] T 9a1716c0adf34bee8a4e99341ba5788f P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Rejecting Update request from peer 0172fed22b6b4890a7fdc953fa128af4 for earlier term 1. Current term is 2. Ops: []
I20260709 13:03:56.989066 11022 raft_consensus.cc:2393] T 63d44f6ba6434ab2870fe3190f4d7c2f P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 186b12638a87410487467df721a3cbda in current term 2: Already voted for candidate 873357742d5843579556ee8166f23f47 in this term.
I20260709 13:03:56.989683 11024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a64b7e99d6c54934a7de6f6dffcf5e94" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
I20260709 13:03:56.991653 10954 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bac7e5040f984ba2b7146bd36af7267e" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:03:56.988806 11027 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d371c5e27bd843469d2e2e453734cb4a" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:03:56.993155 11512 raft_consensus.cc:2749] T 7757ecbf402e47bfbc93d0447741e26c P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:03:56.985970 11029 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f9eb06ed01f4ce68be3af8a8434665b" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:03:56.992607 11151 ts_tablet_manager.cc:1434] T c31881534c5e4f0089c82057c1746652 P 186b12638a87410487467df721a3cbda: Time spent starting tablet: real 0.311s	user 0.007s	sys 0.000s
I20260709 13:03:56.983457 10876 raft_consensus.cc:3060] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:03:56.994933 10878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7709387f4d3d4f9992d97a0fbca39a40" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:03:57.001052 11518 raft_consensus.cc:493] T 8c160ddfe9b34bdaa3835d0915c1e06f P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:57.002792 11149 ts_tablet_manager.cc:1434] T 8c160ddfe9b34bdaa3835d0915c1e06f P 873357742d5843579556ee8166f23f47: Time spent starting tablet: real 0.236s	user 0.009s	sys 0.002s
W20260709 13:03:57.003298 10845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 91029.8us
I20260709 13:03:57.004927 10845 leader_election.cc:304] T 6a9353d1d230416296627d7cf8b1d425 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda; no voters: 
I20260709 13:03:56.990209 11022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a610d1a71f52432e88fe56130f921fa2" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:03:57.013075 11545 leader_election.cc:290] T a5299b8d044f41b7b68057e63d17b159 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:03:57.014662 11545 raft_consensus.cc:2804] T c51782cd9d064ec0949e84e24dd4de67 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:03:57.015316 11545 raft_consensus.cc:493] T c51782cd9d064ec0949e84e24dd4de67 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Starting leader election (detected failure of leader 10ca5ee68ad64b3283b9d235352c9d23)
I20260709 13:03:57.015887 11545 raft_consensus.cc:3060] T c51782cd9d064ec0949e84e24dd4de67 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:03:56.817080 11506 leader_election.cc:290] T 65bf0a62f4d84ef18dd6b23e082724b8 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 2 election: Requested vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:03:57.025466 11027 raft_consensus.cc:2393] T d371c5e27bd843469d2e2e453734cb4a P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 186b12638a87410487467df721a3cbda in current term 1: Already voted for candidate 873357742d5843579556ee8166f23f47 in this term.
I20260709 13:03:57.026288 11029 raft_consensus.cc:1240] T e5ba72ccd854430baadbc636dfd17136 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Rejecting Update request from peer 0172fed22b6b4890a7fdc953fa128af4 for earlier term 1. Current term is 2. Ops: []
W20260709 13:03:57.032492 10880 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.540s	user 0.012s	sys 0.004s
W20260709 13:03:57.033215 10880 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.591s	user 0.012s	sys 0.004s
I20260709 13:03:57.034822 11508 raft_consensus.cc:2804] T 5de5f0581e3b41dea729363d6ca563bb P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260709 13:03:57.034696 10994 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 100397us
I20260709 13:03:57.038054 11151 tablet_bootstrap.cc:492] T acc241e7273d49149984968596242712 P 186b12638a87410487467df721a3cbda: Bootstrap starting.
I20260709 13:03:57.041592 11523 raft_consensus.cc:2804] T 6a9353d1d230416296627d7cf8b1d425 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:03:57.034731 11022 raft_consensus.cc:2468] T a610d1a71f52432e88fe56130f921fa2 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 2.
I20260709 13:03:57.044008 10879 raft_consensus.cc:1275] T a610d1a71f52432e88fe56130f921fa2 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Refusing update from remote peer 0172fed22b6b4890a7fdc953fa128af4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
W20260709 13:03:57.044552 10994 leader_election.cc:343] T ff13ae9a47c7455f962406c6d131a3ff P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:03:57.031601 11024 raft_consensus.cc:2393] T a64b7e99d6c54934a7de6f6dffcf5e94 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 10ca5ee68ad64b3283b9d235352c9d23 in current term 2: Already voted for candidate 873357742d5843579556ee8166f23f47 in this term.
I20260709 13:03:57.002166 11518 raft_consensus.cc:515] T 8c160ddfe9b34bdaa3835d0915c1e06f P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
W20260709 13:03:57.049089 11067 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:45807, user_credentials={real_user=slave}} blocked reactor thread for 53806.2us
I20260709 13:03:57.041365 11524 raft_consensus.cc:515] T 11de93cd9b9d457489ff1cb80f976c9a P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
W20260709 13:03:57.060876 10845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 56359.6us
I20260709 13:03:57.045158 10878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63d44f6ba6434ab2870fe3190f4d7c2f" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:03:57.058931 11523 raft_consensus.cc:493] T 6a9353d1d230416296627d7cf8b1d425 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Starting leader election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:03:57.077242 11518 leader_election.cc:290] T 8c160ddfe9b34bdaa3835d0915c1e06f P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
W20260709 13:03:57.080003 10920 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 395466us
I20260709 13:03:57.081235 10879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e89e640169e14e2d98f968f76027daa7" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:03:57.063412 10994 leader_election.cc:304] T ff13ae9a47c7455f962406c6d131a3ff P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 873357742d5843579556ee8166f23f47; no voters: 0172fed22b6b4890a7fdc953fa128af4, 10ca5ee68ad64b3283b9d235352c9d23
I20260709 13:03:57.087311 10874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e5d327630e5438b878035d28efe6c0c" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:03:57.088763 11506 raft_consensus.cc:2749] T ff13ae9a47c7455f962406c6d131a3ff P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:03:57.089663 11519 raft_consensus.cc:493] T b4cd3d7dfbbf4096908ff11e9dc57141 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0172fed22b6b4890a7fdc953fa128af4)
I20260709 13:03:57.090298 11067 leader_election.cc:304] T 4f9eb06ed01f4ce68be3af8a8434665b P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4; no voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47
I20260709 13:03:57.091104 11025 raft_consensus.cc:1240] T 2567c2d84e4648d7ba5dc23a3df32817 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Rejecting Update request from peer 186b12638a87410487467df721a3cbda for earlier term 1. Current term is 2. Ops: []
I20260709 13:03:57.066095 11027 raft_consensus.cc:1240] T bac7e5040f984ba2b7146bd36af7267e P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Rejecting Update request from peer 0172fed22b6b4890a7fdc953fa128af4 for earlier term 1. Current term is 2. Ops: []
I20260709 13:03:57.093044 11506 raft_consensus.cc:493] T 8c160ddfe9b34bdaa3835d0915c1e06f P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:57.090667 11519 raft_consensus.cc:515] T b4cd3d7dfbbf4096908ff11e9dc57141 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:57.094805 11427 raft_consensus.cc:2749] T 4f9eb06ed01f4ce68be3af8a8434665b P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:03:57.094043 11506 raft_consensus.cc:515] T 8c160ddfe9b34bdaa3835d0915c1e06f P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:57.082911 10879 raft_consensus.cc:3060] T e89e640169e14e2d98f968f76027daa7 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:03:57.104616 10950 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a64b7e99d6c54934a7de6f6dffcf5e94" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
W20260709 13:03:57.106328 10994 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 62029.6us
I20260709 13:03:57.110100 10878 raft_consensus.cc:1240] T 4f9eb06ed01f4ce68be3af8a8434665b P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Rejecting Update request from peer 873357742d5843579556ee8166f23f47 for earlier term 1. Current term is 2. Ops: []
I20260709 13:03:57.071934 11548 raft_consensus.cc:515] T e96e80ba9ac34d0998395515e5c8c41a P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:57.111702 10878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65bf0a62f4d84ef18dd6b23e082724b8" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:03:57.112128 11570 consensus_queue.cc:1059] T a64b7e99d6c54934a7de6f6dffcf5e94 P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:03:57.113467 10878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34064dc7bf3747dbb3876dd824ee0688" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:03:57.115427 11554 raft_consensus.cc:3055] T a64b7e99d6c54934a7de6f6dffcf5e94 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:03:57.116094 11554 raft_consensus.cc:740] T a64b7e99d6c54934a7de6f6dffcf5e94 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0172fed22b6b4890a7fdc953fa128af4, State: Running, Role: LEADER
I20260709 13:03:57.118083 11554 consensus_queue.cc:260] T a64b7e99d6c54934a7de6f6dffcf5e94 P 0172fed22b6b4890a7fdc953fa128af4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:57.118950 11506 leader_election.cc:290] T 8c160ddfe9b34bdaa3835d0915c1e06f P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:03:57.119611 11554 raft_consensus.cc:3060] T a64b7e99d6c54934a7de6f6dffcf5e94 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:03:57.119781 11026 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11de93cd9b9d457489ff1cb80f976c9a" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:03:57.082232 10871 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a1716c0adf34bee8a4e99341ba5788f" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:03:57.135681 10871 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d371c5e27bd843469d2e2e453734cb4a" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:03:57.136593 10871 raft_consensus.cc:3060] T d371c5e27bd843469d2e2e453734cb4a P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:03:57.086052 10919 leader_election.cc:304] T 2567c2d84e4648d7ba5dc23a3df32817 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47; no voters: 186b12638a87410487467df721a3cbda
I20260709 13:03:57.124470 11523 raft_consensus.cc:3060] T 6a9353d1d230416296627d7cf8b1d425 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:03:57.139357 11360 raft_consensus.cc:3055] T e89e640169e14e2d98f968f76027daa7 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 LEADER]: Stepping down as leader of term 1
W20260709 13:03:57.139991 10919 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:45807, user_credentials={real_user=slave}} blocked reactor thread for 54548.2us
I20260709 13:03:57.125633 11028 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a154cc8b181f460abf4880064379948b" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:03:57.125962 11519 leader_election.cc:290] T b4cd3d7dfbbf4096908ff11e9dc57141 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:03:57.141247 11028 raft_consensus.cc:2468] T a154cc8b181f460abf4880064379948b P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 0.
I20260709 13:03:57.130064 11149 tablet_bootstrap.cc:492] T acc241e7273d49149984968596242712 P 873357742d5843579556ee8166f23f47: Bootstrap starting.
I20260709 13:03:57.130685 11556 consensus_queue.cc:1048] T a610d1a71f52432e88fe56130f921fa2 P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:03:57.130772 11548 leader_election.cc:290] T e96e80ba9ac34d0998395515e5c8c41a P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 2 election: Requested vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:03:57.127904 11020 raft_consensus.cc:1240] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Rejecting Update request from peer 0172fed22b6b4890a7fdc953fa128af4 for earlier term 1. Current term is 2. Ops: []
W20260709 13:03:57.065289 11066 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 111085us
I20260709 13:03:57.147073 11029 raft_consensus.cc:1240] T 34064dc7bf3747dbb3876dd824ee0688 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Rejecting Update request from peer 186b12638a87410487467df721a3cbda for earlier term 1. Current term is 2. Ops: []
I20260709 13:03:57.119802 10874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d156d6ae2ca4fa59a1e387edf0fea39" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:03:57.148569 10878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d156d6ae2ca4fa59a1e387edf0fea39" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:03:57.150398 11023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c160ddfe9b34bdaa3835d0915c1e06f" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:03:57.150712 11529 raft_consensus.cc:2804] T 2567c2d84e4648d7ba5dc23a3df32817 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:03:57.151309 11023 raft_consensus.cc:2468] T 8c160ddfe9b34bdaa3835d0915c1e06f P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 0.
I20260709 13:03:57.151376 11529 raft_consensus.cc:697] T 2567c2d84e4648d7ba5dc23a3df32817 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 LEADER]: Becoming Leader. State: Replica: 10ca5ee68ad64b3283b9d235352c9d23, State: Running, Role: LEADER
I20260709 13:03:57.152493 10880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5de5f0581e3b41dea729363d6ca563bb" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:03:57.140028 11360 raft_consensus.cc:740] T e89e640169e14e2d98f968f76027daa7 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 10ca5ee68ad64b3283b9d235352c9d23, State: Running, Role: LEADER
I20260709 13:03:57.155973 10880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f371d2565dd84667b5613848604949bb" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:03:57.156137 11529 consensus_queue.cc:237] T 2567c2d84e4648d7ba5dc23a3df32817 P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:57.150121 11427 consensus_queue.cc:1059] T 32147c6d2d5a477aa38c597c7f78df5b P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.009s
I20260709 13:03:57.159382 11427 raft_consensus.cc:3055] T 32147c6d2d5a477aa38c597c7f78df5b P 0172fed22b6b4890a7fdc953fa128af4 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:03:57.159785 11427 raft_consensus.cc:740] T 32147c6d2d5a477aa38c597c7f78df5b P 0172fed22b6b4890a7fdc953fa128af4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0172fed22b6b4890a7fdc953fa128af4, State: Running, Role: LEADER
I20260709 13:03:57.161701 10844 leader_election.cc:304] T 7839ca2d2c13431a85f44d7c458c6351 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 186b12638a87410487467df721a3cbda; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47
I20260709 13:03:57.163810 11152 tablet_bootstrap.cc:492] T 8c160ddfe9b34bdaa3835d0915c1e06f P 0172fed22b6b4890a7fdc953fa128af4: No bootstrap required, opened a new log
I20260709 13:03:57.164582 11026 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5299b8d044f41b7b68057e63d17b159" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:03:57.166857 11021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f9eb06ed01f4ce68be3af8a8434665b" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
I20260709 13:03:57.169288 11148 tablet_bootstrap.cc:492] T c31881534c5e4f0089c82057c1746652 P 10ca5ee68ad64b3283b9d235352c9d23: No bootstrap required, opened a new log
I20260709 13:03:57.170811 10874 raft_consensus.cc:2468] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 0.
I20260709 13:03:57.171445 11512 consensus_queue.cc:1059] T 0e5d327630e5438b878035d28efe6c0c P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:03:57.173676 11504 consensus_queue.cc:1048] T a610d1a71f52432e88fe56130f921fa2 P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:03:57.156823 10880 raft_consensus.cc:3060] T f371d2565dd84667b5613848604949bb P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:03:57.215732 10949 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e89e640169e14e2d98f968f76027daa7" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:03:57.216109 11512 raft_consensus.cc:3055] T 0e5d327630e5438b878035d28efe6c0c P 0172fed22b6b4890a7fdc953fa128af4 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:03:57.216691 11512 raft_consensus.cc:740] T 0e5d327630e5438b878035d28efe6c0c P 0172fed22b6b4890a7fdc953fa128af4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0172fed22b6b4890a7fdc953fa128af4, State: Running, Role: LEADER
I20260709 13:03:57.219588 11022 raft_consensus.cc:1240] T 4c2a35bb99374e93b573cf870d04faaa P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Rejecting Update request from peer 10ca5ee68ad64b3283b9d235352c9d23 for earlier term 1. Current term is 2. Ops: []
I20260709 13:03:57.219900 10874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e96e80ba9ac34d0998395515e5c8c41a" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:03:57.153112 11524 leader_election.cc:290] T 11de93cd9b9d457489ff1cb80f976c9a P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 2 election: Requested vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:03:57.190994 11508 raft_consensus.cc:493] T 5de5f0581e3b41dea729363d6ca563bb P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0172fed22b6b4890a7fdc953fa128af4)
I20260709 13:03:57.221276 11508 raft_consensus.cc:3060] T 5de5f0581e3b41dea729363d6ca563bb P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:03:57.221659 10874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c2a35bb99374e93b573cf870d04faaa" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:03:57.161432 11427 consensus_queue.cc:260] T 32147c6d2d5a477aa38c597c7f78df5b P 0172fed22b6b4890a7fdc953fa128af4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:57.180989 11556 consensus_queue.cc:1059] T 9a1716c0adf34bee8a4e99341ba5788f P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:03:57.174130 10845 leader_election.cc:304] T d371c5e27bd843469d2e2e453734cb4a P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda; no voters: 
I20260709 13:03:57.230568 11152 ts_tablet_manager.cc:1403] T 8c160ddfe9b34bdaa3835d0915c1e06f P 0172fed22b6b4890a7fdc953fa128af4: Time spent bootstrapping tablet: real 0.906s	user 0.025s	sys 0.022s
W20260709 13:03:57.240149 10994 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 95640.3us
I20260709 13:03:57.240281 10949 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32147c6d2d5a477aa38c597c7f78df5b" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:03:57.242198 10950 raft_consensus.cc:1240] T 11de93cd9b9d457489ff1cb80f976c9a P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Rejecting Update request from peer 873357742d5843579556ee8166f23f47 for earlier term 1. Current term is 2. Ops: []
I20260709 13:03:57.242326 10949 raft_consensus.cc:3060] T 32147c6d2d5a477aa38c597c7f78df5b P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:03:57.244076 11569 raft_consensus.cc:2749] T 7839ca2d2c13431a85f44d7c458c6351 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:03:57.244402 11575 consensus_queue.cc:1059] T e5ba72ccd854430baadbc636dfd17136 P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:03:57.245972 11575 raft_consensus.cc:3055] T e5ba72ccd854430baadbc636dfd17136 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:03:57.220001 11512 consensus_queue.cc:260] T 0e5d327630e5438b878035d28efe6c0c P 0172fed22b6b4890a7fdc953fa128af4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
W20260709 13:03:57.233968 10845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 80483us
I20260709 13:03:57.251590 11360 consensus_queue.cc:260] T e89e640169e14e2d98f968f76027daa7 P 10ca5ee68ad64b3283b9d235352c9d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:57.246379 11575 raft_consensus.cc:740] T e5ba72ccd854430baadbc636dfd17136 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0172fed22b6b4890a7fdc953fa128af4, State: Running, Role: LEADER
I20260709 13:03:57.255319 10774 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:59614 (request call id 63) took 1233 ms. Trace:
I20260709 13:03:57.255817 10874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c160ddfe9b34bdaa3835d0915c1e06f" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:03:57.256747 11020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a154cc8b181f460abf4880064379948b" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
I20260709 13:03:57.272253 11020 raft_consensus.cc:3060] T a154cc8b181f460abf4880064379948b P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:03:57.262666 10843 leader_election.cc:304] T 63d44f6ba6434ab2870fe3190f4d7c2f P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda; no voters: 
I20260709 13:03:57.275087 11148 ts_tablet_manager.cc:1403] T c31881534c5e4f0089c82057c1746652 P 10ca5ee68ad64b3283b9d235352c9d23: Time spent bootstrapping tablet: real 0.871s	user 0.016s	sys 0.020s
I20260709 13:03:57.276729 10878 raft_consensus.cc:2468] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 0.
I20260709 13:03:57.279924 11569 raft_consensus.cc:2804] T 63d44f6ba6434ab2870fe3190f4d7c2f P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:03:57.280095 11512 raft_consensus.cc:3060] T 0e5d327630e5438b878035d28efe6c0c P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:03:57.280452 11569 raft_consensus.cc:493] T 63d44f6ba6434ab2870fe3190f4d7c2f P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Starting leader election (detected failure of leader 10ca5ee68ad64b3283b9d235352c9d23)
I20260709 13:03:57.280894 11569 raft_consensus.cc:3060] T 63d44f6ba6434ab2870fe3190f4d7c2f P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:03:57.287125 11570 consensus_queue.cc:1059] T 7709387f4d3d4f9992d97a0fbca39a40 P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:03:57.287968 11021 raft_consensus.cc:1240] T e96e80ba9ac34d0998395515e5c8c41a P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Rejecting Update request from peer 10ca5ee68ad64b3283b9d235352c9d23 for earlier term 1. Current term is 2. Ops: []
I20260709 13:03:57.287817 11513 consensus_queue.cc:1059] T bac7e5040f984ba2b7146bd36af7267e P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:03:57.289068 11558 raft_consensus.cc:3055] T 7709387f4d3d4f9992d97a0fbca39a40 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:03:57.289547 11558 raft_consensus.cc:740] T 7709387f4d3d4f9992d97a0fbca39a40 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0172fed22b6b4890a7fdc953fa128af4, State: Running, Role: LEADER
I20260709 13:03:57.290694 11558 consensus_queue.cc:260] T 7709387f4d3d4f9992d97a0fbca39a40 P 0172fed22b6b4890a7fdc953fa128af4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:57.292198 11026 raft_consensus.cc:1240] T e89e640169e14e2d98f968f76027daa7 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Rejecting Update request from peer 10ca5ee68ad64b3283b9d235352c9d23 for earlier term 1. Current term is 2. Ops: []
I20260709 13:03:57.292884 11558 raft_consensus.cc:3060] T 7709387f4d3d4f9992d97a0fbca39a40 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:03:57.302407 10950 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65bf0a62f4d84ef18dd6b23e082724b8" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:03:57.303898 11513 raft_consensus.cc:493] T bb7d67be500947b3aef4c4245e33671c P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:57.304432 11570 raft_consensus.cc:3055] T bac7e5040f984ba2b7146bd36af7267e P 0172fed22b6b4890a7fdc953fa128af4 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:03:57.305119 11570 raft_consensus.cc:740] T bac7e5040f984ba2b7146bd36af7267e P 0172fed22b6b4890a7fdc953fa128af4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0172fed22b6b4890a7fdc953fa128af4, State: Running, Role: LEADER
I20260709 13:03:57.307010 11570 consensus_queue.cc:260] T bac7e5040f984ba2b7146bd36af7267e P 0172fed22b6b4890a7fdc953fa128af4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:57.310465 11570 raft_consensus.cc:3060] T bac7e5040f984ba2b7146bd36af7267e P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:03:57.311729 11575 consensus_queue.cc:260] T e5ba72ccd854430baadbc636dfd17136 P 0172fed22b6b4890a7fdc953fa128af4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
W20260709 13:03:57.313126 10844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:45807, user_credentials={real_user=slave}} blocked reactor thread for 151975us
I20260709 13:03:57.315515 10844 leader_election.cc:304] T 9a1716c0adf34bee8a4e99341ba5788f P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 186b12638a87410487467df721a3cbda; no voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47
I20260709 13:03:57.277979 10874 raft_consensus.cc:2468] T 8c160ddfe9b34bdaa3835d0915c1e06f P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 0.
I20260709 13:03:57.316706 11572 raft_consensus.cc:2749] T 9a1716c0adf34bee8a4e99341ba5788f P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:03:57.317474 10874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d371c5e27bd843469d2e2e453734cb4a" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:03:57.263576 11504 raft_consensus.cc:3055] T 9a1716c0adf34bee8a4e99341ba5788f P 0172fed22b6b4890a7fdc953fa128af4 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:03:57.318722 11504 raft_consensus.cc:740] T 9a1716c0adf34bee8a4e99341ba5788f P 0172fed22b6b4890a7fdc953fa128af4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0172fed22b6b4890a7fdc953fa128af4, State: Running, Role: LEADER
I20260709 13:03:57.301112 11360 raft_consensus.cc:3060] T e89e640169e14e2d98f968f76027daa7 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:03:57.320637 11504 consensus_queue.cc:260] T 9a1716c0adf34bee8a4e99341ba5788f P 0172fed22b6b4890a7fdc953fa128af4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:57.322387 11504 raft_consensus.cc:3060] T 9a1716c0adf34bee8a4e99341ba5788f P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:03:57.322959 11020 raft_consensus.cc:2468] T a154cc8b181f460abf4880064379948b P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 1.
I20260709 13:03:57.386857 11149 tablet_bootstrap.cc:654] T acc241e7273d49149984968596242712 P 873357742d5843579556ee8166f23f47: Neither blocks nor log segments found. Creating new log.
W20260709 13:03:57.342267 10995 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:38967, user_credentials={real_user=slave}} blocked reactor thread for 117229us
I20260709 13:03:57.326550 11427 raft_consensus.cc:3060] T 32147c6d2d5a477aa38c597c7f78df5b P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:03:57.325237 10844 leader_election.cc:304] T a610d1a71f52432e88fe56130f921fa2 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:03:57.423941 11151 tablet_bootstrap.cc:654] T acc241e7273d49149984968596242712 P 186b12638a87410487467df721a3cbda: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:57.413479 10878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d270b9e5481245d9b5340a454a7cd10b" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:03:57.463539 11508 raft_consensus.cc:515] T 5de5f0581e3b41dea729363d6ca563bb P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:57.427318 11545 raft_consensus.cc:515] T c51782cd9d064ec0949e84e24dd4de67 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:57.346877 11524 raft_consensus.cc:493] T ff13ae9a47c7455f962406c6d131a3ff P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:57.397895 11022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11de93cd9b9d457489ff1cb80f976c9a" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
I20260709 13:03:57.304709 11513 raft_consensus.cc:515] T bb7d67be500947b3aef4c4245e33671c P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:57.334052 11148 raft_consensus.cc:359] T c31881534c5e4f0089c82057c1746652 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:57.493103 10995 leader_election.cc:304] T a610d1a71f52432e88fe56130f921fa2 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47; no voters: 0172fed22b6b4890a7fdc953fa128af4
W20260709 13:03:57.495029 11555 consensus_meta.cc:287] T 72b111ad7b5f4753ab1d2b5ee1114840 P 0172fed22b6b4890a7fdc953fa128af4: Time spent flushing consensus metadata: real 0.681s	user 0.013s	sys 0.013s
I20260709 13:03:57.495659 11555 raft_consensus.cc:515] T 72b111ad7b5f4753ab1d2b5ee1114840 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:57.361714 11152 raft_consensus.cc:359] T 8c160ddfe9b34bdaa3835d0915c1e06f P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:57.351485 11572 raft_consensus.cc:493] T c31881534c5e4f0089c82057c1746652 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:57.400290 10775 catalog_manager.cc:5697] T e5a5f5a608504650b48fd27f163faed6 P 186b12638a87410487467df721a3cbda reported cstate change: term changed from 0 to 1, leader changed from <none> to 186b12638a87410487467df721a3cbda (127.0.107.193). New cstate: current_term: 1 leader_uuid: "186b12638a87410487467df721a3cbda" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } health_report { overall_health: UNKNOWN } } }
I20260709 13:03:57.378123 10955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e96e80ba9ac34d0998395515e5c8c41a" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
W20260709 13:03:57.385758 10919 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:57.492333 11148 raft_consensus.cc:385] T c31881534c5e4f0089c82057c1746652 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260709 13:03:57.329943 11425 consensus_meta.cc:287] T f371d2565dd84667b5613848604949bb P 0172fed22b6b4890a7fdc953fa128af4: Time spent flushing consensus metadata: real 0.547s	user 0.011s	sys 0.014s
W20260709 13:03:57.435101 11553 consensus_meta.cc:287] T 87cd8f5097624c8692f1d83544d8881e P 10ca5ee68ad64b3283b9d235352c9d23: Time spent flushing consensus metadata: real 0.641s	user 0.007s	sys 0.016s
W20260709 13:03:57.438252 11574 log.cc:927] Time spent T 11de93cd9b9d457489ff1cb80f976c9a P 186b12638a87410487467df721a3cbda: Append to log took a long time: real 0.143s	user 0.000s	sys 0.000s
W20260709 13:03:57.328521 10877 consensus_meta.cc:287] T 4b37f8fca8dc42718b2a42f0883a956f P 186b12638a87410487467df721a3cbda: Time spent flushing consensus metadata: real 0.908s	user 0.002s	sys 0.025s
I20260709 13:03:57.496943 11152 raft_consensus.cc:385] T 8c160ddfe9b34bdaa3835d0915c1e06f P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:57.410568 11099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5ba72ccd854430baadbc636dfd17136" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:03:57.402333 11096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e5d327630e5438b878035d28efe6c0c" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:03:57.363958 10879 raft_consensus.cc:2468] T e89e640169e14e2d98f968f76027daa7 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 2.
I20260709 13:03:57.375002 10947 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f46a5102fd64b45bf1c4dff863d2b4d" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:03:57.504359 11506 consensus_queue.cc:1059] T 11de93cd9b9d457489ff1cb80f976c9a P 873357742d5843579556ee8166f23f47 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:03:57.504689 10775 catalog_manager.cc:5697] T 87cd8f5097624c8692f1d83544d8881e P 186b12638a87410487467df721a3cbda reported cstate change: term changed from 0 to 1, leader changed from <none> to 186b12638a87410487467df721a3cbda (127.0.107.193). New cstate: current_term: 1 leader_uuid: "186b12638a87410487467df721a3cbda" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } health_report { overall_health: UNKNOWN } } }
I20260709 13:03:57.408597 11094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a64b7e99d6c54934a7de6f6dffcf5e94" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:03:57.412536 10875 raft_consensus.cc:2468] T 732c4adeeff6422497aa7db4a82f535b P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 1.
I20260709 13:03:57.295425 10953 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34064dc7bf3747dbb3876dd824ee0688" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:03:57.371212 10946 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d156d6ae2ca4fa59a1e387edf0fea39" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:03:57.326918 10876 raft_consensus.cc:2468] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 2.
W20260709 13:03:57.345773 11573 log.cc:927] Time spent T 0e5d327630e5438b878035d28efe6c0c P 186b12638a87410487467df721a3cbda: Append to log took a long time: real 0.067s	user 0.000s	sys 0.001s
I20260709 13:03:57.352458 11575 raft_consensus.cc:3060] T e5ba72ccd854430baadbc636dfd17136 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Advancing to term 2
W20260709 13:03:57.506232 11067 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:57.502887 11148 raft_consensus.cc:740] T c31881534c5e4f0089c82057c1746652 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10ca5ee68ad64b3283b9d235352c9d23, State: Initialized, Role: FOLLOWER
I20260709 13:03:57.374629 10948 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c2a35bb99374e93b573cf870d04faaa" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:03:57.370397 10951 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "836251ebc2b74f5da88b8a8b2338e2a5" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:03:57.571645 10948 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a69f32c53f24a50b7852d143665d7b4" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:03:57.572510 10951 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5de5f0581e3b41dea729363d6ca563bb" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:03:57.573691 10951 raft_consensus.cc:2393] T 5de5f0581e3b41dea729363d6ca563bb P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 186b12638a87410487467df721a3cbda in current term 2: Already voted for candidate 10ca5ee68ad64b3283b9d235352c9d23 in this term.
I20260709 13:03:57.574708 11102 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32147c6d2d5a477aa38c597c7f78df5b" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:03:57.574764 10955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87cd8f5097624c8692f1d83544d8881e" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:03:57.575670 11100 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d371c5e27bd843469d2e2e453734cb4a" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:03:57.576126 11098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f46a5102fd64b45bf1c4dff863d2b4d" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:03:57.577123 11100 raft_consensus.cc:2393] T d371c5e27bd843469d2e2e453734cb4a P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 873357742d5843579556ee8166f23f47 in current term 1: Already voted for candidate 0172fed22b6b4890a7fdc953fa128af4 in this term.
I20260709 13:03:57.518805 11572 raft_consensus.cc:515] T c31881534c5e4f0089c82057c1746652 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:57.579648 11095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f371d2565dd84667b5613848604949bb" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:03:57.580964 11572 leader_election.cc:290] T c31881534c5e4f0089c82057c1746652 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:03:57.516974 11510 raft_consensus.cc:2804] T a610d1a71f52432e88fe56130f921fa2 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:03:57.397279 11026 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4cd3d7dfbbf4096908ff11e9dc57141" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:03:57.582723 11510 raft_consensus.cc:493] T a610d1a71f52432e88fe56130f921fa2 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:57.583026 11026 raft_consensus.cc:2468] T b4cd3d7dfbbf4096908ff11e9dc57141 P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 1.
I20260709 13:03:57.583340 11510 raft_consensus.cc:3060] T a610d1a71f52432e88fe56130f921fa2 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:03:57.518306 11103 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bac7e5040f984ba2b7146bd36af7267e" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:03:57.469560 11524 raft_consensus.cc:515] T ff13ae9a47c7455f962406c6d131a3ff P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:57.586230 11537 raft_consensus.cc:3055] T 11de93cd9b9d457489ff1cb80f976c9a P 873357742d5843579556ee8166f23f47 [term 1 LEADER]: Stepping down as leader of term 1
W20260709 13:03:57.586740 10995 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:38967, user_credentials={real_user=slave}} blocked reactor thread for 113096us
I20260709 13:03:57.587713 10995 leader_election.cc:304] T 7709387f4d3d4f9992d97a0fbca39a40 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 873357742d5843579556ee8166f23f47; no voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda
I20260709 13:03:57.519479 11152 raft_consensus.cc:740] T 8c160ddfe9b34bdaa3835d0915c1e06f P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0172fed22b6b4890a7fdc953fa128af4, State: Initialized, Role: FOLLOWER
I20260709 13:03:57.589430 11152 consensus_queue.cc:260] T 8c160ddfe9b34bdaa3835d0915c1e06f P 0172fed22b6b4890a7fdc953fa128af4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:57.369688 10954 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d439326f23747eaab3c1e7da2d6b785" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:03:57.519773 11425 raft_consensus.cc:515] T f371d2565dd84667b5613848604949bb P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:57.586709 11537 raft_consensus.cc:740] T 11de93cd9b9d457489ff1cb80f976c9a P 873357742d5843579556ee8166f23f47 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 873357742d5843579556ee8166f23f47, State: Running, Role: LEADER
I20260709 13:03:57.505390 11548 consensus_queue.cc:1048] T 732c4adeeff6422497aa7db4a82f535b P 873357742d5843579556ee8166f23f47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:03:57.595397 10880 raft_consensus.cc:2468] T f371d2565dd84667b5613848604949bb P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 1.
I20260709 13:03:57.595275 11537 consensus_queue.cc:260] T 11de93cd9b9d457489ff1cb80f976c9a P 873357742d5843579556ee8166f23f47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:57.597084 11537 raft_consensus.cc:3060] T 11de93cd9b9d457489ff1cb80f976c9a P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:03:57.597846 11548 raft_consensus.cc:2749] T 7709387f4d3d4f9992d97a0fbca39a40 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 13:03:57.380079 11534 raft_consensus.cc:515] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:57.384081 11523 raft_consensus.cc:515] T 6a9353d1d230416296627d7cf8b1d425 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:57.600306 11534 leader_election.cc:290] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 2 election: Requested vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:03:57.327147 10845 leader_election.cc:304] T 5d439326f23747eaab3c1e7da2d6b785 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda; no voters: 
I20260709 13:03:57.534317 10947 raft_consensus.cc:3060] T 6f46a5102fd64b45bf1c4dff863d2b4d P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Advancing to term 2
W20260709 13:03:57.601576 10845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 274675us
I20260709 13:03:57.602582 10845 leader_election.cc:304] T 836251ebc2b74f5da88b8a8b2338e2a5 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 186b12638a87410487467df721a3cbda; no voters: 0172fed22b6b4890a7fdc953fa128af4, 10ca5ee68ad64b3283b9d235352c9d23
I20260709 13:03:57.603091 11513 leader_election.cc:290] T bb7d67be500947b3aef4c4245e33671c P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:03:57.603873 11425 leader_election.cc:290] T f371d2565dd84667b5613848604949bb P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 1 election: Requested vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:03:57.537487 10879 raft_consensus.cc:1240] T e89e640169e14e2d98f968f76027daa7 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Rejecting Update request from peer 10ca5ee68ad64b3283b9d235352c9d23 for earlier term 1. Current term is 2. Ops: []
I20260709 13:03:57.606132 11021 raft_consensus.cc:1240] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Rejecting Update request from peer 10ca5ee68ad64b3283b9d235352c9d23 for earlier term 1. Current term is 2. Ops: []
I20260709 13:03:57.607847 11555 leader_election.cc:290] T 72b111ad7b5f4753ab1d2b5ee1114840 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 2 election: Requested vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:03:57.605163 10845 leader_election.cc:304] T 72b111ad7b5f4753ab1d2b5ee1114840 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 186b12638a87410487467df721a3cbda; no voters: 0172fed22b6b4890a7fdc953fa128af4, 10ca5ee68ad64b3283b9d235352c9d23
I20260709 13:03:57.555787 11581 raft_consensus.cc:2804] T a610d1a71f52432e88fe56130f921fa2 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:03:57.610049 11581 raft_consensus.cc:493] T a610d1a71f52432e88fe56130f921fa2 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:57.610518 11581 raft_consensus.cc:3060] T a610d1a71f52432e88fe56130f921fa2 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:03:57.612804 11545 leader_election.cc:290] T c51782cd9d064ec0949e84e24dd4de67 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 2 election: Requested vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:03:57.531550 10877 raft_consensus.cc:2468] T 4b37f8fca8dc42718b2a42f0883a956f P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 1.
I20260709 13:03:57.613564 10947 raft_consensus.cc:2468] T 6f46a5102fd64b45bf1c4dff863d2b4d P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 2.
I20260709 13:03:57.613808 11545 raft_consensus.cc:2804] T 5d439326f23747eaab3c1e7da2d6b785 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:03:57.613822 10877 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51696 (request call id 117) took 1215 ms. Trace:
I20260709 13:03:57.561955 10946 raft_consensus.cc:2468] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 0.
I20260709 13:03:57.614845 10877 rpcz_store.cc:276] 0709 13:03:56.398246 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:56.401370 (+  3124us) service_pool.cc:224] Handling call
0709 13:03:57.613702 (+1212332us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:57.613195 11096 raft_consensus.cc:2410] T 0e5d327630e5438b878035d28efe6c0c P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 873357742d5843579556ee8166f23f47 for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260709 13:03:57.563683 11101 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a1716c0adf34bee8a4e99341ba5788f" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:03:57.527572 11508 leader_election.cc:290] T 5de5f0581e3b41dea729363d6ca563bb P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 2 election: Requested vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:03:57.441704 10949 raft_consensus.cc:2468] T 32147c6d2d5a477aa38c597c7f78df5b P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 2.
I20260709 13:03:57.617102 10871 raft_consensus.cc:2468] T d371c5e27bd843469d2e2e453734cb4a P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 1.
I20260709 13:03:57.617764 11508 raft_consensus.cc:2749] T 836251ebc2b74f5da88b8a8b2338e2a5 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:03:57.617944 11518 raft_consensus.cc:2764] T d371c5e27bd843469d2e2e453734cb4a P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260709 13:03:57.271442 10774 rpcz_store.cc:276] 0709 13:03:56.019319 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:56.019518 (+   199us) service_pool.cc:224] Handling call
0709 13:03:57.243264 (+1223746us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 13:03:56.336325 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 13:03:56.389809 (+ 53484us) write_op.cc:432] Acquiring schema lock in shared mode
0709 13:03:56.389851 (+    42us) write_op.cc:435] Acquired schema lock
0709 13:03:56.389868 (+    17us) tablet.cc:662] Decoding operations
0709 13:03:56.390227 (+   359us) write_op.cc:620] Acquiring the partition lock for write op
0709 13:03:56.390307 (+    80us) write_op.cc:641] Partition lock acquired for write op
0709 13:03:56.390338 (+    31us) tablet.cc:685] Acquiring locks for 1 operations
0709 13:03:56.390597 (+   259us) tablet.cc:701] Row locks acquired
0709 13:03:56.390628 (+    31us) write_op.cc:260] PREPARE: finished
0709 13:03:56.390829 (+   201us) write_op.cc:270] Start()
0709 13:03:56.390985 (+   156us) write_op.cc:276] Timestamp: P: 1783602236390769 usec, L: 0
0709 13:03:56.391016 (+    31us) op_driver.cc:348] REPLICATION: starting
0709 13:03:56.391914 (+   898us) log.cc:844] Serialized 540 byte log entry
0709 13:03:56.859844 (+467930us) op_driver.cc:464] REPLICATION: finished
0709 13:03:57.050331 (+190487us) write_op.cc:301] APPLY: starting
0709 13:03:57.050639 (+   308us) tablet.cc:1367] starting BulkCheckPresence
0709 13:03:57.050857 (+   218us) tablet.cc:1370] finished BulkCheckPresence
0709 13:03:57.051098 (+   241us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 13:03:57.051634 (+   536us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 13:03:57.071259 (+ 19625us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0709 13:03:57.082478 (+ 11219us) write_op.cc:312] APPLY: finished
0709 13:03:57.083048 (+   570us) log.cc:844] Serialized 22 byte log entry
0709 13:03:57.083335 (+   287us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:03:57.083574 (+   239us) write_op.cc:454] Released schema lock
0709 13:03:57.083662 (+    88us) write_op.cc:341] FINISH: Updating metrics
0709 13:03:57.084506 (+   844us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:03:57.084551 (+    45us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":36798,"reported_tablets":5,"child_traces":[["op",{"apply.queue_time_us":189994,"apply.run_cpu_time_us":4281,"apply.run_wall_time_us":34260,"num_ops":1,"prepare.queue_time_us":207669,"prepare.run_cpu_time_us":36130,"prepare.run_wall_time_us":312061,"raft.queue_time_us":40031,"raft.run_cpu_time_us":5246,"raft.run_wall_time_us":173359,"replication_time_us":468666,"thread_start_us":577808,"threads_started":4,"wal-append.queue_time_us":342917}]]}
I20260709 13:03:57.600323 11523 leader_election.cc:290] T 6a9353d1d230416296627d7cf8b1d425 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 2 election: Requested vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:03:57.619915 11095 raft_consensus.cc:2393] T f371d2565dd84667b5613848604949bb P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 873357742d5843579556ee8166f23f47 in current term 1: Already voted for candidate 0172fed22b6b4890a7fdc953fa128af4 in this term.
I20260709 13:03:57.619997 11148 consensus_queue.cc:260] T c31881534c5e4f0089c82057c1746652 P 10ca5ee68ad64b3283b9d235352c9d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:57.621268 11524 leader_election.cc:290] T ff13ae9a47c7455f962406c6d131a3ff P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:03:57.623003 10954 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c31881534c5e4f0089c82057c1746652" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:03:57.623802 10954 raft_consensus.cc:2468] T c31881534c5e4f0089c82057c1746652 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 0.
I20260709 13:03:57.624749 11148 ts_tablet_manager.cc:1434] T c31881534c5e4f0089c82057c1746652 P 10ca5ee68ad64b3283b9d235352c9d23: Time spent starting tablet: real 0.346s	user 0.012s	sys 0.008s
I20260709 13:03:57.581759 11097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c160ddfe9b34bdaa3835d0915c1e06f" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:03:57.629187 11097 raft_consensus.cc:2468] T 8c160ddfe9b34bdaa3835d0915c1e06f P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 0.
I20260709 13:03:57.298349 11066 leader_election.cc:304] T d7ce47b9a92d43dd88d95bcda5038597 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda
W20260709 13:03:57.640789 10844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:45807, user_credentials={real_user=slave}} blocked reactor thread for 315707us
I20260709 13:03:57.522666 11553 raft_consensus.cc:515] T 87cd8f5097624c8692f1d83544d8881e P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:57.574049 10948 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5299b8d044f41b7b68057e63d17b159" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:03:57.622210 11534 consensus_queue.cc:1048] T bb7d67be500947b3aef4c4245e33671c P 873357742d5843579556ee8166f23f47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.021s
I20260709 13:03:57.614356 11545 raft_consensus.cc:493] T 5d439326f23747eaab3c1e7da2d6b785 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Starting leader election (detected failure of leader 10ca5ee68ad64b3283b9d235352c9d23)
I20260709 13:03:57.667378 10843 leader_election.cc:304] T c31881534c5e4f0089c82057c1746652 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda; no voters: 
I20260709 13:03:57.679193 11512 raft_consensus.cc:2749] T d7ce47b9a92d43dd88d95bcda5038597 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:03:57.679569 11103 raft_consensus.cc:2468] T bac7e5040f984ba2b7146bd36af7267e P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 2.
I20260709 13:03:57.681121 10950 raft_consensus.cc:1240] T 87cd8f5097624c8692f1d83544d8881e P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Rejecting Update request from peer 186b12638a87410487467df721a3cbda for earlier term 1. Current term is 2. Ops: []
W20260709 13:03:57.682415 11069 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:57.666115 11545 raft_consensus.cc:3060] T 5d439326f23747eaab3c1e7da2d6b785 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:03:57.682899 10946 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c51782cd9d064ec0949e84e24dd4de67" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
W20260709 13:03:57.699332 10919 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:57.695034 10843 leader_election.cc:304] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda; no voters: 
I20260709 13:03:57.683619 11523 raft_consensus.cc:2749] T 72b111ad7b5f4753ab1d2b5ee1114840 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 13:03:57.706771 10995 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:38967, user_credentials={real_user=slave}} blocked reactor thread for 119209us
I20260709 13:03:57.692812 11508 raft_consensus.cc:2804] T c31881534c5e4f0089c82057c1746652 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:03:57.712646 10844 leader_election.cc:304] T 11de93cd9b9d457489ff1cb80f976c9a P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 186b12638a87410487467df721a3cbda; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47
I20260709 13:03:57.714951 11152 ts_tablet_manager.cc:1434] T 8c160ddfe9b34bdaa3835d0915c1e06f P 0172fed22b6b4890a7fdc953fa128af4: Time spent starting tablet: real 0.472s	user 0.014s	sys 0.003s
I20260709 13:03:57.718582 10995 leader_election.cc:304] T 63d44f6ba6434ab2870fe3190f4d7c2f P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 873357742d5843579556ee8166f23f47; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda
I20260709 13:03:57.706727 11572 consensus_queue.cc:1059] T 2567c2d84e4648d7ba5dc23a3df32817 P 186b12638a87410487467df721a3cbda [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.017s
I20260709 13:03:57.731216 11148 tablet_bootstrap.cc:492] T 75492f86fde046c3900df3eac8292fd4 P 10ca5ee68ad64b3283b9d235352c9d23: Bootstrap starting.
I20260709 13:03:57.712471 11508 raft_consensus.cc:493] T c31881534c5e4f0089c82057c1746652 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:57.697760 11569 raft_consensus.cc:515] T 63d44f6ba6434ab2870fe3190f4d7c2f P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
W20260709 13:03:57.764472 10995 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:38967, user_credentials={real_user=slave}} blocked reactor thread for 46127.5us
I20260709 13:03:57.764904 11569 leader_election.cc:290] T 63d44f6ba6434ab2870fe3190f4d7c2f P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 2 election: Requested vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:03:57.769236 11581 raft_consensus.cc:515] T a610d1a71f52432e88fe56130f921fa2 P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
W20260709 13:03:57.774879 10992 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 44477.9us
I20260709 13:03:57.775372 11152 tablet_bootstrap.cc:492] T acc241e7273d49149984968596242712 P 0172fed22b6b4890a7fdc953fa128af4: Bootstrap starting.
I20260709 13:03:57.776011 10992 leader_election.cc:304] T 6f46a5102fd64b45bf1c4dff863d2b4d P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:03:57.776886 10919 leader_election.cc:304] T 34064dc7bf3747dbb3876dd824ee0688 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23; no voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47
I20260709 13:03:57.777261 10992 leader_election.cc:304] T 32147c6d2d5a477aa38c597c7f78df5b P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:03:57.778017 11101 raft_consensus.cc:2468] T 9a1716c0adf34bee8a4e99341ba5788f P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 2.
I20260709 13:03:57.778692 11585 raft_consensus.cc:2804] T 6f46a5102fd64b45bf1c4dff863d2b4d P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:03:57.759617 11508 raft_consensus.cc:3060] T c31881534c5e4f0089c82057c1746652 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:03:57.779377 11585 raft_consensus.cc:697] T 6f46a5102fd64b45bf1c4dff863d2b4d P 873357742d5843579556ee8166f23f47 [term 2 LEADER]: Becoming Leader. State: Replica: 873357742d5843579556ee8166f23f47, State: Running, Role: LEADER
W20260709 13:03:57.727780 11066 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 493941us
W20260709 13:03:57.730036 10845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 125028us
I20260709 13:03:57.780521 11585 consensus_queue.cc:237] T 6f46a5102fd64b45bf1c4dff863d2b4d P 873357742d5843579556ee8166f23f47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
W20260709 13:03:57.782353 10843 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 58823.1us
I20260709 13:03:57.752403 11518 raft_consensus.cc:2749] T 11de93cd9b9d457489ff1cb80f976c9a P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:03:57.787348 10948 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f1466124dbd4f6ebb3c4d62ca48bf96" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
W20260709 13:03:57.798285 10843 leader_election.cc:343] T a5299b8d044f41b7b68057e63d17b159 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:03:57.794545 11553 leader_election.cc:290] T 87cd8f5097624c8692f1d83544d8881e P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 2 election: Requested vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:03:57.794965 11360 consensus_queue.cc:1059] T 65bf0a62f4d84ef18dd6b23e082724b8 P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:03:57.792951 11066 leader_election.cc:304] T 525bfdbd6d934afd823f7d3ee3f9d052 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47
I20260709 13:03:57.754387 11580 raft_consensus.cc:2804] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:03:57.804545 11103 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d156d6ae2ca4fa59a1e387edf0fea39" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:03:57.804891 11096 raft_consensus.cc:3060] T e5a5f5a608504650b48fd27f163faed6 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:03:57.805474 11518 consensus_queue.cc:1059] T 87cd8f5097624c8692f1d83544d8881e P 186b12638a87410487467df721a3cbda [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:03:57.807049 11095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a610d1a71f52432e88fe56130f921fa2" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:03:57.761974 11506 raft_consensus.cc:2749] T 63d44f6ba6434ab2870fe3190f4d7c2f P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 13:03:57.807614 10948 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63d44f6ba6434ab2870fe3190f4d7c2f" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:03:57.808573 11583 raft_consensus.cc:2804] T 32147c6d2d5a477aa38c597c7f78df5b P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:03:57.801875 11524 raft_consensus.cc:3060] T 34064dc7bf3747dbb3876dd824ee0688 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:03:57.810606 11149 tablet_bootstrap.cc:492] T acc241e7273d49149984968596242712 P 873357742d5843579556ee8166f23f47: No bootstrap required, opened a new log
I20260709 13:03:57.810920 11527 consensus_queue.cc:1059] T 4c2a35bb99374e93b573cf870d04faaa P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:03:57.838923 11518 raft_consensus.cc:3055] T 87cd8f5097624c8692f1d83544d8881e P 186b12638a87410487467df721a3cbda [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:03:57.839208 11527 raft_consensus.cc:3055] T 4c2a35bb99374e93b573cf870d04faaa P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:03:57.839560 11518 raft_consensus.cc:740] T 87cd8f5097624c8692f1d83544d8881e P 186b12638a87410487467df721a3cbda [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 186b12638a87410487467df721a3cbda, State: Running, Role: LEADER
I20260709 13:03:57.839768 11527 raft_consensus.cc:740] T 4c2a35bb99374e93b573cf870d04faaa P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 10ca5ee68ad64b3283b9d235352c9d23, State: Running, Role: LEADER
I20260709 13:03:57.840838 11518 consensus_queue.cc:260] T 87cd8f5097624c8692f1d83544d8881e P 186b12638a87410487467df721a3cbda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:57.840965 11527 consensus_queue.cc:260] T 4c2a35bb99374e93b573cf870d04faaa P 10ca5ee68ad64b3283b9d235352c9d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:57.842653 11103 raft_consensus.cc:2468] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 0.
I20260709 13:03:57.811564 11100 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c160ddfe9b34bdaa3835d0915c1e06f" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:03:57.843940 11103 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c51782cd9d064ec0949e84e24dd4de67" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:03:57.844136 11100 raft_consensus.cc:2468] T 8c160ddfe9b34bdaa3835d0915c1e06f P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 0.
I20260709 13:03:57.844655 11103 raft_consensus.cc:3060] T c51782cd9d064ec0949e84e24dd4de67 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:03:57.845433 11100 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5de5f0581e3b41dea729363d6ca563bb" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:03:57.811918 11098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f9eb06ed01f4ce68be3af8a8434665b" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:03:57.846834 11098 raft_consensus.cc:3060] T 4f9eb06ed01f4ce68be3af8a8434665b P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:03:57.847735 11583 raft_consensus.cc:697] T 32147c6d2d5a477aa38c597c7f78df5b P 873357742d5843579556ee8166f23f47 [term 2 LEADER]: Becoming Leader. State: Replica: 873357742d5843579556ee8166f23f47, State: Running, Role: LEADER
I20260709 13:03:57.813197 11512 raft_consensus.cc:2749] T 525bfdbd6d934afd823f7d3ee3f9d052 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 13:03:57.813102 11529 consensus_queue.cc:1059] T e96e80ba9ac34d0998395515e5c8c41a P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:03:57.813604 11066 leader_election.cc:304] T d71ca3a1ba6b48478aaf62741bf3cf6a P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47
I20260709 13:03:57.790220 11094 raft_consensus.cc:2468] T a64b7e99d6c54934a7de6f6dffcf5e94 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 2.
W20260709 13:03:57.832664 10995 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:38967, user_credentials={real_user=slave}} blocked reactor thread for 67359us
I20260709 13:03:57.872944 10995 leader_election.cc:304] T 65bf0a62f4d84ef18dd6b23e082724b8 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 873357742d5843579556ee8166f23f47; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda
I20260709 13:03:57.780531 11581 leader_election.cc:290] T a610d1a71f52432e88fe56130f921fa2 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 3 election: Requested vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:03:57.837117 11149 ts_tablet_manager.cc:1403] T acc241e7273d49149984968596242712 P 873357742d5843579556ee8166f23f47: Time spent bootstrapping tablet: real 0.708s	user 0.037s	sys 0.047s
I20260709 13:03:57.815589 11580 raft_consensus.cc:493] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:57.812414 11097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a69f32c53f24a50b7852d143665d7b4" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:03:57.878705 11097 raft_consensus.cc:3060] T 8a69f32c53f24a50b7852d143665d7b4 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:03:57.839567 11095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c31881534c5e4f0089c82057c1746652" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:03:57.884243 11095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f1466124dbd4f6ebb3c4d62ca48bf96" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:03:57.885118 11095 raft_consensus.cc:3060] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:03:57.886303 11580 raft_consensus.cc:3060] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Advancing to term 1
W20260709 13:03:57.887557 11587 log.cc:927] Time spent T 732c4adeeff6422497aa7db4a82f535b P 10ca5ee68ad64b3283b9d235352c9d23: Append to log took a long time: real 0.072s	user 0.000s	sys 0.000s
I20260709 13:03:57.842423 11518 raft_consensus.cc:3060] T 87cd8f5097624c8692f1d83544d8881e P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Advancing to term 2
W20260709 13:03:57.888576 11066 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 75100.6us
I20260709 13:03:57.890319 11512 raft_consensus.cc:2749] T d71ca3a1ba6b48478aaf62741bf3cf6a P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:03:57.890748 11506 consensus_queue.cc:1059] T 4f9eb06ed01f4ce68be3af8a8434665b P 873357742d5843579556ee8166f23f47 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:03:57.861194 11583 consensus_queue.cc:237] T 32147c6d2d5a477aa38c597c7f78df5b P 873357742d5843579556ee8166f23f47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:57.892168 11506 raft_consensus.cc:3055] T 4f9eb06ed01f4ce68be3af8a8434665b P 873357742d5843579556ee8166f23f47 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:03:57.892598 11506 raft_consensus.cc:740] T 4f9eb06ed01f4ce68be3af8a8434665b P 873357742d5843579556ee8166f23f47 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 873357742d5843579556ee8166f23f47, State: Running, Role: LEADER
I20260709 13:03:57.873307 10919 leader_election.cc:304] T a154cc8b181f460abf4880064379948b P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:03:57.893538 11506 consensus_queue.cc:260] T 4f9eb06ed01f4ce68be3af8a8434665b P 873357742d5843579556ee8166f23f47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:57.895659 11506 raft_consensus.cc:3060] T 4f9eb06ed01f4ce68be3af8a8434665b P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:03:57.900739 11148 tablet_bootstrap.cc:654] T 75492f86fde046c3900df3eac8292fd4 P 10ca5ee68ad64b3283b9d235352c9d23: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:57.854418 11527 raft_consensus.cc:3060] T 4c2a35bb99374e93b573cf870d04faaa P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:03:57.911937 11101 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a9353d1d230416296627d7cf8b1d425" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:03:57.915086 11151 tablet_bootstrap.cc:492] T acc241e7273d49149984968596242712 P 186b12638a87410487467df721a3cbda: No bootstrap required, opened a new log
I20260709 13:03:57.915793 11151 ts_tablet_manager.cc:1403] T acc241e7273d49149984968596242712 P 186b12638a87410487467df721a3cbda: Time spent bootstrapping tablet: real 0.879s	user 0.019s	sys 0.050s
I20260709 13:03:57.912916 11101 raft_consensus.cc:3060] T 6a9353d1d230416296627d7cf8b1d425 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:03:57.921427 11096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a610d1a71f52432e88fe56130f921fa2" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:03:57.927351 11152 tablet_bootstrap.cc:654] T acc241e7273d49149984968596242712 P 0172fed22b6b4890a7fdc953fa128af4: Neither blocks nor log segments found. Creating new log.
W20260709 13:03:57.929229 10995 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:38967, user_credentials={real_user=slave}} blocked reactor thread for 56369us
I20260709 13:03:57.929963 11592 raft_consensus.cc:2749] T 65bf0a62f4d84ef18dd6b23e082724b8 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 13:03:57.930572 10995 leader_election.cc:304] T 34064dc7bf3747dbb3876dd824ee0688 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 873357742d5843579556ee8166f23f47; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda
I20260709 13:03:57.930651 11569 raft_consensus.cc:3055] T 2567c2d84e4648d7ba5dc23a3df32817 P 186b12638a87410487467df721a3cbda [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:03:57.932301 11569 raft_consensus.cc:740] T 2567c2d84e4648d7ba5dc23a3df32817 P 186b12638a87410487467df721a3cbda [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 186b12638a87410487467df721a3cbda, State: Running, Role: LEADER
I20260709 13:03:57.932436 10995 leader_election.cc:304] T e96e80ba9ac34d0998395515e5c8c41a P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 873357742d5843579556ee8166f23f47; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda
W20260709 13:03:57.934345 10920 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 87569.1us
W20260709 13:03:57.899900 10843 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:57.937165 11548 raft_consensus.cc:2749] T e96e80ba9ac34d0998395515e5c8c41a P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 13:03:57.939134 11099 raft_consensus.cc:2468] T e5ba72ccd854430baadbc636dfd17136 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 2.
I20260709 13:03:57.941926 10995 leader_election.cc:304] T 4c2a35bb99374e93b573cf870d04faaa P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 873357742d5843579556ee8166f23f47; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda
W20260709 13:03:57.952273 10844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:45807, user_credentials={real_user=slave}} blocked reactor thread for 254041us
I20260709 13:03:57.954702 10921 leader_election.cc:304] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda; no voters: 
I20260709 13:03:57.957633 11069 leader_election.cc:304] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda; no voters: 
I20260709 13:03:57.958913 11425 raft_consensus.cc:2804] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:03:57.959463 11425 raft_consensus.cc:493] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:57.959914 11425 raft_consensus.cc:3060] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:03:57.933483 11569 consensus_queue.cc:260] T 2567c2d84e4648d7ba5dc23a3df32817 P 186b12638a87410487467df721a3cbda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:57.962348 11569 raft_consensus.cc:3060] T 2567c2d84e4648d7ba5dc23a3df32817 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:03:57.963047 11151 raft_consensus.cc:359] T acc241e7273d49149984968596242712 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:57.963873 11151 raft_consensus.cc:385] T acc241e7273d49149984968596242712 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:57.964150 11151 raft_consensus.cc:740] T acc241e7273d49149984968596242712 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 186b12638a87410487467df721a3cbda, State: Initialized, Role: FOLLOWER
I20260709 13:03:57.897660 11066 leader_election.cc:304] T 8a69f32c53f24a50b7852d143665d7b4 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda
I20260709 13:03:57.964871 11151 consensus_queue.cc:260] T acc241e7273d49149984968596242712 P 186b12638a87410487467df721a3cbda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
W20260709 13:03:57.965653 11066 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 75366.4us
I20260709 13:03:57.947638 11575 raft_consensus.cc:493] T 5d439326f23747eaab3c1e7da2d6b785 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 10ca5ee68ad64b3283b9d235352c9d23)
I20260709 13:03:57.967017 11575 raft_consensus.cc:515] T 5d439326f23747eaab3c1e7da2d6b785 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:57.967525 11151 ts_tablet_manager.cc:1434] T acc241e7273d49149984968596242712 P 186b12638a87410487467df721a3cbda: Time spent starting tablet: real 0.051s	user 0.007s	sys 0.001s
I20260709 13:03:57.940039 10948 raft_consensus.cc:2468] T 63d44f6ba6434ab2870fe3190f4d7c2f P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 2.
I20260709 13:03:57.980693 11069 leader_election.cc:304] T d371c5e27bd843469d2e2e453734cb4a P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda; no voters: 
I20260709 13:03:57.937795 11524 raft_consensus.cc:2749] T 34064dc7bf3747dbb3876dd824ee0688 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:03:57.950659 11534 raft_consensus.cc:2749] T 34064dc7bf3747dbb3876dd824ee0688 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 13:03:57.994279 10843 leader_election.cc:304] T 63d44f6ba6434ab2870fe3190f4d7c2f P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda; no voters: 
I20260709 13:03:57.994803 10879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f371d2565dd84667b5613848604949bb" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
W20260709 13:03:58.000824 10921 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:38967, user_credentials={real_user=slave}} blocked reactor thread for 46487us
W20260709 13:03:57.987627 10919 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:45807, user_credentials={real_user=slave}} blocked reactor thread for 137983us
W20260709 13:03:58.012807 10995 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:38967, user_credentials={real_user=slave}} blocked reactor thread for 70970.6us
I20260709 13:03:58.014011 10995 leader_election.cc:304] T 8c160ddfe9b34bdaa3835d0915c1e06f P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:03:57.967406 11066 leader_election.cc:304] T a154cc8b181f460abf4880064379948b P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 10ca5ee68ad64b3283b9d235352c9d23; no voters: 
I20260709 13:03:57.994335 10946 raft_consensus.cc:1240] T 2567c2d84e4648d7ba5dc23a3df32817 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 LEADER]: Rejecting Update request from peer 186b12638a87410487467df721a3cbda for earlier term 1. Current term is 2. Ops: []
I20260709 13:03:57.995741 10872 raft_consensus.cc:1240] T a610d1a71f52432e88fe56130f921fa2 P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Rejecting Update request from peer 0172fed22b6b4890a7fdc953fa128af4 for earlier term 2. Current term is 3. Ops: [2.1-2.1]
I20260709 13:03:57.986109 11098 raft_consensus.cc:2468] T 4f9eb06ed01f4ce68be3af8a8434665b P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 2.
I20260709 13:03:58.000732 11585 raft_consensus.cc:2749] T 4c2a35bb99374e93b573cf870d04faaa P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 13:03:58.023532 11149 raft_consensus.cc:359] T acc241e7273d49149984968596242712 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:58.055802 11149 raft_consensus.cc:385] T acc241e7273d49149984968596242712 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:58.056180 11149 raft_consensus.cc:740] T acc241e7273d49149984968596242712 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 873357742d5843579556ee8166f23f47, State: Initialized, Role: FOLLOWER
I20260709 13:03:58.007731 11554 raft_consensus.cc:2804] T d371c5e27bd843469d2e2e453734cb4a P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:03:58.057300 11149 consensus_queue.cc:260] T acc241e7273d49149984968596242712 P 873357742d5843579556ee8166f23f47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:58.051775 10879 raft_consensus.cc:2393] T f371d2565dd84667b5613848604949bb P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 0172fed22b6b4890a7fdc953fa128af4 in current term 1: Already voted for candidate 873357742d5843579556ee8166f23f47 in this term.
W20260709 13:03:58.063459 11069 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:38967, user_credentials={real_user=slave}} blocked reactor thread for 44149.4us
I20260709 13:03:58.026373 11524 raft_consensus.cc:2804] T a154cc8b181f460abf4880064379948b P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:03:58.022604 11591 raft_consensus.cc:2804] T 8c160ddfe9b34bdaa3835d0915c1e06f P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:03:58.054183 11529 raft_consensus.cc:2804] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:03:57.998050 10845 leader_election.cc:304] T 8c160ddfe9b34bdaa3835d0915c1e06f P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda; no voters: 
W20260709 13:03:58.062213 11067 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:58.072638 11151 tablet_bootstrap.cc:492] T 1e4f2241a94a41f38e481137f868b83d P 186b12638a87410487467df721a3cbda: Bootstrap starting.
W20260709 13:03:58.077080 11069 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:58.052268 10994 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 103617us
I20260709 13:03:58.062573 10880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72b111ad7b5f4753ab1d2b5ee1114840" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:03:58.069782 11094 raft_consensus.cc:1240] T 4f9eb06ed01f4ce68be3af8a8434665b P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Rejecting Update request from peer 873357742d5843579556ee8166f23f47 for earlier term 1. Current term is 2. Ops: []
I20260709 13:03:58.074740 11529 raft_consensus.cc:493] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:58.073531 10875 raft_consensus.cc:1240] T c51782cd9d064ec0949e84e24dd4de67 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Rejecting Update request from peer 10ca5ee68ad64b3283b9d235352c9d23 for earlier term 1. Current term is 2. Ops: []
I20260709 13:03:58.070358 11591 raft_consensus.cc:493] T 8c160ddfe9b34bdaa3835d0915c1e06f P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:03:58.111980 10995 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:38967, user_credentials={real_user=slave}} blocked reactor thread for 86766.4us
W20260709 13:03:58.072326 11066 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 105033us
I20260709 13:03:58.086449 11524 raft_consensus.cc:697] T a154cc8b181f460abf4880064379948b P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 LEADER]: Becoming Leader. State: Replica: 10ca5ee68ad64b3283b9d235352c9d23, State: Running, Role: LEADER
I20260709 13:03:58.081037 10948 raft_consensus.cc:1240] T 34064dc7bf3747dbb3876dd824ee0688 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Rejecting Update request from peer 186b12638a87410487467df721a3cbda for earlier term 1. Current term is 2. Ops: []
I20260709 13:03:58.108390 10880 raft_consensus.cc:3060] T 72b111ad7b5f4753ab1d2b5ee1114840 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:03:58.120180 11581 raft_consensus.cc:2804] T 8c160ddfe9b34bdaa3835d0915c1e06f P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:03:58.121806 11512 raft_consensus.cc:2804] T a154cc8b181f460abf4880064379948b P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:03:58.121268 11529 raft_consensus.cc:3060] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Advancing to term 1
W20260709 13:03:58.100759 10845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 114688us
I20260709 13:03:58.078876 10919 leader_election.cc:304] T b4cd3d7dfbbf4096908ff11e9dc57141 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:03:58.141508 11604 raft_consensus.cc:493] T acc241e7273d49149984968596242712 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:58.132527 11591 raft_consensus.cc:3060] T 8c160ddfe9b34bdaa3835d0915c1e06f P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:03:58.150719 10995 leader_election.cc:304] T e89e640169e14e2d98f968f76027daa7 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47; no voters: 10ca5ee68ad64b3283b9d235352c9d23
I20260709 13:03:58.152981 11066 leader_election.cc:304] T e5a5f5a608504650b48fd27f163faed6 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda
I20260709 13:03:58.136922 11523 raft_consensus.cc:493] T d7ce47b9a92d43dd88d95bcda5038597 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 10ca5ee68ad64b3283b9d235352c9d23)
I20260709 13:03:58.144979 11512 raft_consensus.cc:493] T a154cc8b181f460abf4880064379948b P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:58.145732 11581 raft_consensus.cc:493] T 8c160ddfe9b34bdaa3835d0915c1e06f P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:58.145190 11554 raft_consensus.cc:697] T d371c5e27bd843469d2e2e453734cb4a P 0172fed22b6b4890a7fdc953fa128af4 [term 1 LEADER]: Becoming Leader. State: Replica: 0172fed22b6b4890a7fdc953fa128af4, State: Running, Role: LEADER
I20260709 13:03:58.162751 11149 ts_tablet_manager.cc:1434] T acc241e7273d49149984968596242712 P 873357742d5843579556ee8166f23f47: Time spent starting tablet: real 0.289s	user 0.013s	sys 0.018s
I20260709 13:03:58.153903 11589 raft_consensus.cc:2804] T 63d44f6ba6434ab2870fe3190f4d7c2f P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:03:58.172241 11589 raft_consensus.cc:697] T 63d44f6ba6434ab2870fe3190f4d7c2f P 186b12638a87410487467df721a3cbda [term 2 LEADER]: Becoming Leader. State: Replica: 186b12638a87410487467df721a3cbda, State: Running, Role: LEADER
I20260709 13:03:58.173530 11589 consensus_queue.cc:237] T 63d44f6ba6434ab2870fe3190f4d7c2f P 186b12638a87410487467df721a3cbda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:58.164693 11548 raft_consensus.cc:2804] T e89e640169e14e2d98f968f76027daa7 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Leader election won for term 2
W20260709 13:03:58.167795 11594 log.cc:927] Time spent T 6f46a5102fd64b45bf1c4dff863d2b4d P 873357742d5843579556ee8166f23f47: Append to log took a long time: real 0.109s	user 0.001s	sys 0.000s
I20260709 13:03:58.155428 11604 raft_consensus.cc:515] T acc241e7273d49149984968596242712 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:58.155308 11597 raft_consensus.cc:3055] T e96e80ba9ac34d0998395515e5c8c41a P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:03:58.171525 11595 raft_consensus.cc:2804] T b4cd3d7dfbbf4096908ff11e9dc57141 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:03:58.162528 11581 raft_consensus.cc:3060] T 8c160ddfe9b34bdaa3835d0915c1e06f P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:03:58.139468 10994 leader_election.cc:304] T d371c5e27bd843469d2e2e453734cb4a P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 873357742d5843579556ee8166f23f47; no voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda
I20260709 13:03:58.194931 11555 raft_consensus.cc:2749] T e5a5f5a608504650b48fd27f163faed6 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260709 13:03:58.197134 11066 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 51440.5us
I20260709 13:03:58.161556 11512 raft_consensus.cc:3060] T a154cc8b181f460abf4880064379948b P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:03:58.155906 10879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87cd8f5097624c8692f1d83544d8881e" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:03:58.211406 10879 raft_consensus.cc:2410] T 87cd8f5097624c8692f1d83544d8881e P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 10ca5ee68ad64b3283b9d235352c9d23 for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
W20260709 13:03:58.157684 10919 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:45807, user_credentials={real_user=slave}} blocked reactor thread for 88125.7us
I20260709 13:03:58.215747 11066 leader_election.cc:304] T 87cd8f5097624c8692f1d83544d8881e P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 10ca5ee68ad64b3283b9d235352c9d23; no voters: 186b12638a87410487467df721a3cbda
W20260709 13:03:58.218007 11427 consensus_meta.cc:287] T 32147c6d2d5a477aa38c597c7f78df5b P 0172fed22b6b4890a7fdc953fa128af4: Time spent flushing consensus metadata: real 0.760s	user 0.020s	sys 0.000s
I20260709 13:03:58.218743 11607 raft_consensus.cc:2804] T 87cd8f5097624c8692f1d83544d8881e P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:03:58.219329 11607 raft_consensus.cc:493] T 87cd8f5097624c8692f1d83544d8881e P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:58.219700 11607 raft_consensus.cc:3060] T 87cd8f5097624c8692f1d83544d8881e P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:03:58.162863 11575 leader_election.cc:290] T 5d439326f23747eaab3c1e7da2d6b785 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:03:58.224779 11601 consensus_queue.cc:1059] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:03:58.162195 10875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d439326f23747eaab3c1e7da2d6b785" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
W20260709 13:03:58.226768 10921 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:38967, user_credentials={real_user=slave}} blocked reactor thread for 82951.2us
I20260709 13:03:58.226229 11601 consensus_queue.cc:1048] T 4b37f8fca8dc42718b2a42f0883a956f P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20260709 13:03:58.231482 10874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11de93cd9b9d457489ff1cb80f976c9a" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:03:58.239836 11102 raft_consensus.cc:2410] T 32147c6d2d5a477aa38c597c7f78df5b P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 873357742d5843579556ee8166f23f47 for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260709 13:03:58.155033 11580 raft_consensus.cc:515] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:58.176651 11095 raft_consensus.cc:2468] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 2.
I20260709 13:03:58.250584 11149 tablet_bootstrap.cc:492] T 1e4f2241a94a41f38e481137f868b83d P 873357742d5843579556ee8166f23f47: Bootstrap starting.
I20260709 13:03:58.159490 10845 leader_election.cc:304] T 5de5f0581e3b41dea729363d6ca563bb P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 186b12638a87410487467df721a3cbda; no voters: 0172fed22b6b4890a7fdc953fa128af4, 10ca5ee68ad64b3283b9d235352c9d23
I20260709 13:03:58.206499 11595 raft_consensus.cc:493] T b4cd3d7dfbbf4096908ff11e9dc57141 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0172fed22b6b4890a7fdc953fa128af4)
I20260709 13:03:58.175645 11523 raft_consensus.cc:515] T d7ce47b9a92d43dd88d95bcda5038597 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:58.203603 11597 raft_consensus.cc:740] T e96e80ba9ac34d0998395515e5c8c41a P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 10ca5ee68ad64b3283b9d235352c9d23, State: Running, Role: LEADER
I20260709 13:03:58.227767 11604 leader_election.cc:290] T acc241e7273d49149984968596242712 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:03:58.228868 11103 raft_consensus.cc:2468] T c51782cd9d064ec0949e84e24dd4de67 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 2.
I20260709 13:03:58.267892 11598 raft_consensus.cc:3055] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:03:58.268420 11598 raft_consensus.cc:740] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 10ca5ee68ad64b3283b9d235352c9d23, State: Running, Role: LEADER
I20260709 13:03:58.270673 11590 raft_consensus.cc:2749] T 5de5f0581e3b41dea729363d6ca563bb P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 13:03:58.275305 11021 raft_consensus.cc:1275] T 2567c2d84e4648d7ba5dc23a3df32817 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Refusing update from remote peer 10ca5ee68ad64b3283b9d235352c9d23: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:03:58.275586 11523 leader_election.cc:290] T d7ce47b9a92d43dd88d95bcda5038597 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:03:58.276274 11100 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "836251ebc2b74f5da88b8a8b2338e2a5" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:03:58.276489 11099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a64b7e99d6c54934a7de6f6dffcf5e94" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:03:58.277361 11099 raft_consensus.cc:2393] T a64b7e99d6c54934a7de6f6dffcf5e94 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 10ca5ee68ad64b3283b9d235352c9d23 in current term 2: Already voted for candidate 873357742d5843579556ee8166f23f47 in this term.
I20260709 13:03:58.277657 11100 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d156d6ae2ca4fa59a1e387edf0fea39" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:03:58.279675 11098 raft_consensus.cc:1240] T c51782cd9d064ec0949e84e24dd4de67 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Rejecting Update request from peer 10ca5ee68ad64b3283b9d235352c9d23 for earlier term 1. Current term is 2. Ops: []
I20260709 13:03:58.280965 10949 raft_consensus.cc:1240] T 5de5f0581e3b41dea729363d6ca563bb P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Rejecting Update request from peer 0172fed22b6b4890a7fdc953fa128af4 for earlier term 1. Current term is 2. Ops: [1.1-1.1]
I20260709 13:03:58.282104 10953 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d439326f23747eaab3c1e7da2d6b785" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:03:58.282850 10955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72b111ad7b5f4753ab1d2b5ee1114840" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:03:58.283948 10954 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d156d6ae2ca4fa59a1e387edf0fea39" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:03:58.280913 11098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a154cc8b181f460abf4880064379948b" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:03:58.282295 10949 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d7ce47b9a92d43dd88d95bcda5038597" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:03:58.282117 10948 raft_consensus.cc:1240] T 6f46a5102fd64b45bf1c4dff863d2b4d P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Rejecting Update request from peer 0172fed22b6b4890a7fdc953fa128af4 for earlier term 1. Current term is 2. Ops: []
I20260709 13:03:58.256887 11360 raft_consensus.cc:493] T e5ba72ccd854430baadbc636dfd17136 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0172fed22b6b4890a7fdc953fa128af4)
I20260709 13:03:58.285396 10880 raft_consensus.cc:2468] T 72b111ad7b5f4753ab1d2b5ee1114840 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 2.
I20260709 13:03:58.285666 11360 raft_consensus.cc:515] T e5ba72ccd854430baadbc636dfd17136 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:58.278313 11099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5de5f0581e3b41dea729363d6ca563bb" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:03:58.287195 11597 consensus_queue.cc:260] T e96e80ba9ac34d0998395515e5c8c41a P 10ca5ee68ad64b3283b9d235352c9d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:58.288319 11099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d270b9e5481245d9b5340a454a7cd10b" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:03:58.288892 11360 leader_election.cc:290] T e5ba72ccd854430baadbc636dfd17136 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:03:58.290264 10878 raft_consensus.cc:1240] T 72b111ad7b5f4753ab1d2b5ee1114840 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Rejecting Update request from peer 10ca5ee68ad64b3283b9d235352c9d23 for earlier term 1. Current term is 2. Ops: []
I20260709 13:03:58.253803 11583 raft_consensus.cc:2749] T d371c5e27bd843469d2e2e453734cb4a P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 13:03:58.253539 11504 raft_consensus.cc:493] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:58.289032 11597 raft_consensus.cc:3060] T e96e80ba9ac34d0998395515e5c8c41a P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Advancing to term 2
W20260709 13:03:58.293073 10995 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:38967, user_credentials={real_user=slave}} blocked reactor thread for 89755.1us
I20260709 13:03:58.252678 11097 raft_consensus.cc:2468] T 8a69f32c53f24a50b7852d143665d7b4 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 2.
I20260709 13:03:58.254154 11548 raft_consensus.cc:697] T e89e640169e14e2d98f968f76027daa7 P 873357742d5843579556ee8166f23f47 [term 2 LEADER]: Becoming Leader. State: Replica: 873357742d5843579556ee8166f23f47, State: Running, Role: LEADER
I20260709 13:03:58.260587 11595 raft_consensus.cc:3060] T b4cd3d7dfbbf4096908ff11e9dc57141 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:03:58.269450 11598 consensus_queue.cc:260] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 10ca5ee68ad64b3283b9d235352c9d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:58.253849 11518 consensus_queue.cc:1059] T 34064dc7bf3747dbb3876dd824ee0688 P 186b12638a87410487467df721a3cbda [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.040s
I20260709 13:03:58.267125 11101 raft_consensus.cc:2468] T 6a9353d1d230416296627d7cf8b1d425 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 2.
I20260709 13:03:58.326236 11148 tablet_bootstrap.cc:492] T 75492f86fde046c3900df3eac8292fd4 P 10ca5ee68ad64b3283b9d235352c9d23: No bootstrap required, opened a new log
I20260709 13:03:58.328171 11148 ts_tablet_manager.cc:1403] T 75492f86fde046c3900df3eac8292fd4 P 10ca5ee68ad64b3283b9d235352c9d23: Time spent bootstrapping tablet: real 0.623s	user 0.038s	sys 0.031s
I20260709 13:03:58.289973 11099 raft_consensus.cc:1240] T 72b111ad7b5f4753ab1d2b5ee1114840 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Rejecting Update request from peer 10ca5ee68ad64b3283b9d235352c9d23 for earlier term 1. Current term is 2. Ops: []
I20260709 13:03:58.307878 10995 leader_election.cc:304] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47; no voters: 0172fed22b6b4890a7fdc953fa128af4
I20260709 13:03:58.239218 11524 consensus_queue.cc:237] T a154cc8b181f460abf4880064379948b P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
W20260709 13:03:58.296417 10994 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 164353us
I20260709 13:03:58.299093 11022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff13ae9a47c7455f962406c6d131a3ff" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:03:58.323552 11598 raft_consensus.cc:3060] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:03:58.301012 11513 raft_consensus.cc:2749] T 8a69f32c53f24a50b7852d143665d7b4 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:03:58.280930 10947 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb7d67be500947b3aef4c4245e33671c" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:03:58.317538 11601 raft_consensus.cc:493] T 525bfdbd6d934afd823f7d3ee3f9d052 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:58.342813 11022 raft_consensus.cc:2468] T ff13ae9a47c7455f962406c6d131a3ff P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 0.
I20260709 13:03:58.343789 10874 raft_consensus.cc:1275] T 2567c2d84e4648d7ba5dc23a3df32817 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Refusing update from remote peer 10ca5ee68ad64b3283b9d235352c9d23: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:03:58.345636 11096 raft_consensus.cc:1240] T 6a9353d1d230416296627d7cf8b1d425 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Rejecting Update request from peer 873357742d5843579556ee8166f23f47 for earlier term 1. Current term is 2. Ops: []
I20260709 13:03:58.347903 11097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4cd3d7dfbbf4096908ff11e9dc57141" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:03:58.352540 11580 leader_election.cc:290] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 1 election: Requested vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:03:58.355365 10876 raft_consensus.cc:1240] T 8a69f32c53f24a50b7852d143665d7b4 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Rejecting Update request from peer 10ca5ee68ad64b3283b9d235352c9d23 for earlier term 1. Current term is 2. Ops: []
I20260709 13:03:58.350078 10994 leader_election.cc:304] T 0e5d327630e5438b878035d28efe6c0c P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 873357742d5843579556ee8166f23f47; no voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda
I20260709 13:03:58.354707 11604 consensus_queue.cc:1048] T bb7d67be500947b3aef4c4245e33671c P 873357742d5843579556ee8166f23f47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260709 13:03:58.358382 11537 raft_consensus.cc:2749] T 0e5d327630e5438b878035d28efe6c0c P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 13:03:58.360741 11553 raft_consensus.cc:3055] T 65bf0a62f4d84ef18dd6b23e082724b8 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:03:58.361383 11569 raft_consensus.cc:3055] T 34064dc7bf3747dbb3876dd824ee0688 P 186b12638a87410487467df721a3cbda [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:03:58.361948 11569 raft_consensus.cc:740] T 34064dc7bf3747dbb3876dd824ee0688 P 186b12638a87410487467df721a3cbda [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 186b12638a87410487467df721a3cbda, State: Running, Role: LEADER
I20260709 13:03:58.362923 11569 consensus_queue.cc:260] T 34064dc7bf3747dbb3876dd824ee0688 P 186b12638a87410487467df721a3cbda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:58.364336 11569 raft_consensus.cc:3060] T 34064dc7bf3747dbb3876dd824ee0688 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Advancing to term 2
W20260709 13:03:58.338479 11510 consensus_meta.cc:287] T a610d1a71f52432e88fe56130f921fa2 P 873357742d5843579556ee8166f23f47: Time spent flushing consensus metadata: real 0.751s	user 0.016s	sys 0.002s
I20260709 13:03:58.367044 11548 consensus_queue.cc:237] T e89e640169e14e2d98f968f76027daa7 P 873357742d5843579556ee8166f23f47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:58.367419 11510 raft_consensus.cc:515] T a610d1a71f52432e88fe56130f921fa2 P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:58.371029 11601 raft_consensus.cc:515] T 525bfdbd6d934afd823f7d3ee3f9d052 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:58.302946 11554 consensus_queue.cc:237] T d371c5e27bd843469d2e2e453734cb4a P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:58.383574 11151 tablet_bootstrap.cc:654] T 1e4f2241a94a41f38e481137f868b83d P 186b12638a87410487467df721a3cbda: Neither blocks nor log segments found. Creating new log.
W20260709 13:03:58.392058 10995 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:38967, user_credentials={real_user=slave}} blocked reactor thread for 84346.8us
W20260709 13:03:58.301676 10845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 147294us
I20260709 13:03:58.293002 11512 raft_consensus.cc:515] T a154cc8b181f460abf4880064379948b P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:58.295382 11504 raft_consensus.cc:515] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:58.412379 10995 leader_election.cc:304] T f371d2565dd84667b5613848604949bb P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:03:58.409789 10774 catalog_manager.cc:5697] T 732c4adeeff6422497aa7db4a82f535b P 10ca5ee68ad64b3283b9d235352c9d23 reported cstate change: term changed from 0 to 1, leader changed from <none> to 873357742d5843579556ee8166f23f47 (127.0.107.195). New cstate: current_term: 1 leader_uuid: "873357742d5843579556ee8166f23f47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } }
I20260709 13:03:58.418475 11069 leader_election.cc:304] T 72b111ad7b5f4753ab1d2b5ee1114840 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda; no voters: 
I20260709 13:03:58.380707 10872 raft_consensus.cc:1240] T 5de5f0581e3b41dea729363d6ca563bb P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Rejecting Update request from peer 0172fed22b6b4890a7fdc953fa128af4 for earlier term 1. Current term is 2. Ops: []
I20260709 13:03:58.382344 11583 raft_consensus.cc:2804] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Leader election won for term 2
W20260709 13:03:58.419843 11067 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:45807, user_credentials={real_user=slave}} blocked reactor thread for 155282us
I20260709 13:03:58.361404 11553 raft_consensus.cc:740] T 65bf0a62f4d84ef18dd6b23e082724b8 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 10ca5ee68ad64b3283b9d235352c9d23, State: Running, Role: LEADER
I20260709 13:03:58.374372 11601 leader_election.cc:290] T 525bfdbd6d934afd823f7d3ee3f9d052 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
W20260709 13:03:58.422400 11545 consensus_meta.cc:287] T 5d439326f23747eaab3c1e7da2d6b785 P 186b12638a87410487467df721a3cbda: Time spent flushing consensus metadata: real 0.723s	user 0.012s	sys 0.003s
I20260709 13:03:58.412469 11510 leader_election.cc:290] T a610d1a71f52432e88fe56130f921fa2 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 3 election: Requested vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
W20260709 13:03:58.406806 10994 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 56844us
W20260709 13:03:58.421342 10845 leader_election.cc:343] T c31881534c5e4f0089c82057c1746652 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:03:58.406682 11529 raft_consensus.cc:515] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
W20260709 13:03:58.447260 11508 consensus_meta.cc:287] T c31881534c5e4f0089c82057c1746652 P 186b12638a87410487467df721a3cbda: Time spent flushing consensus metadata: real 0.668s	user 0.011s	sys 0.004s
I20260709 13:03:58.444780 11575 raft_consensus.cc:2804] T 72b111ad7b5f4753ab1d2b5ee1114840 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:03:58.446244 11604 raft_consensus.cc:2804] T f371d2565dd84667b5613848604949bb P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:03:58.420773 10947 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d156d6ae2ca4fa59a1e387edf0fea39" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
W20260709 13:03:58.466291 11425 consensus_meta.cc:287] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 0172fed22b6b4890a7fdc953fa128af4: Time spent flushing consensus metadata: real 0.506s	user 0.013s	sys 0.001s
I20260709 13:03:58.452697 11583 raft_consensus.cc:697] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 873357742d5843579556ee8166f23f47 [term 2 LEADER]: Becoming Leader. State: Replica: 873357742d5843579556ee8166f23f47, State: Running, Role: LEADER
I20260709 13:03:58.500535 11152 tablet_bootstrap.cc:492] T acc241e7273d49149984968596242712 P 0172fed22b6b4890a7fdc953fa128af4: No bootstrap required, opened a new log
I20260709 13:03:58.500440 11583 consensus_queue.cc:237] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 873357742d5843579556ee8166f23f47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:58.501346 11152 ts_tablet_manager.cc:1403] T acc241e7273d49149984968596242712 P 0172fed22b6b4890a7fdc953fa128af4: Time spent bootstrapping tablet: real 0.726s	user 0.026s	sys 0.028s
I20260709 13:03:58.491153 10921 leader_election.cc:304] T 11de93cd9b9d457489ff1cb80f976c9a P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23; no voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47
I20260709 13:03:58.503170 11604 raft_consensus.cc:697] T f371d2565dd84667b5613848604949bb P 873357742d5843579556ee8166f23f47 [term 1 LEADER]: Becoming Leader. State: Replica: 873357742d5843579556ee8166f23f47, State: Running, Role: LEADER
I20260709 13:03:58.493652 11610 consensus_queue.cc:1059] T a610d1a71f52432e88fe56130f921fa2 P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.083s
I20260709 13:03:58.499081 11601 consensus_queue.cc:1048] T 4b37f8fca8dc42718b2a42f0883a956f P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:03:58.504187 11604 consensus_queue.cc:237] T f371d2565dd84667b5613848604949bb P 873357742d5843579556ee8166f23f47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:58.505843 11601 raft_consensus.cc:2749] T 11de93cd9b9d457489ff1cb80f976c9a P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 13:03:58.490513 11360 consensus_queue.cc:1059] T c51782cd9d064ec0949e84e24dd4de67 P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.016s
I20260709 13:03:58.506563 11601 consensus_queue.cc:1048] T 2567c2d84e4648d7ba5dc23a3df32817 P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:03:58.507598 11360 consensus_queue.cc:1059] T 8a69f32c53f24a50b7852d143665d7b4 P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:03:58.508641 11360 raft_consensus.cc:3055] T c51782cd9d064ec0949e84e24dd4de67 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:03:58.509011 11360 raft_consensus.cc:740] T c51782cd9d064ec0949e84e24dd4de67 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 10ca5ee68ad64b3283b9d235352c9d23, State: Running, Role: LEADER
I20260709 13:03:58.509060 11512 leader_election.cc:290] T a154cc8b181f460abf4880064379948b P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 1 election: Requested vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
W20260709 13:03:58.510072 10826 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 13:03:58.508507 11601 consensus_queue.cc:1059] T 72b111ad7b5f4753ab1d2b5ee1114840 P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
I20260709 13:03:58.510318 11553 consensus_queue.cc:260] T 65bf0a62f4d84ef18dd6b23e082724b8 P 10ca5ee68ad64b3283b9d235352c9d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:58.488184 10774 catalog_manager.cc:5697] T 4b37f8fca8dc42718b2a42f0883a956f P 10ca5ee68ad64b3283b9d235352c9d23 reported cstate change: term changed from 0 to 1, leader changed from <none> to 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194). New cstate: current_term: 1 leader_uuid: "10ca5ee68ad64b3283b9d235352c9d23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } health_report { overall_health: UNKNOWN } } }
I20260709 13:03:58.511245 11527 raft_consensus.cc:3055] T 8a69f32c53f24a50b7852d143665d7b4 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:03:58.512722 11553 raft_consensus.cc:3060] T 65bf0a62f4d84ef18dd6b23e082724b8 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:03:58.513044 11527 raft_consensus.cc:740] T 8a69f32c53f24a50b7852d143665d7b4 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 10ca5ee68ad64b3283b9d235352c9d23, State: Running, Role: LEADER
I20260709 13:03:58.491338 11098 raft_consensus.cc:2393] T a154cc8b181f460abf4880064379948b P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 10ca5ee68ad64b3283b9d235352c9d23 in current term 1: Already voted for candidate 0172fed22b6b4890a7fdc953fa128af4 in this term.
I20260709 13:03:58.514072 11527 consensus_queue.cc:260] T 8a69f32c53f24a50b7852d143665d7b4 P 10ca5ee68ad64b3283b9d235352c9d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:58.514595 10994 leader_election.cc:304] T bac7e5040f984ba2b7146bd36af7267e P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47; no voters: 10ca5ee68ad64b3283b9d235352c9d23
I20260709 13:03:58.515487 11527 raft_consensus.cc:3060] T 8a69f32c53f24a50b7852d143665d7b4 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:03:58.516917 10994 leader_election.cc:304] T a64b7e99d6c54934a7de6f6dffcf5e94 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47; no voters: 10ca5ee68ad64b3283b9d235352c9d23
I20260709 13:03:58.518499 10954 raft_consensus.cc:2393] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0172fed22b6b4890a7fdc953fa128af4 in current term 1: Already voted for candidate 10ca5ee68ad64b3283b9d235352c9d23 in this term.
I20260709 13:03:58.519568 11537 raft_consensus.cc:2804] T bac7e5040f984ba2b7146bd36af7267e P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:03:58.519896 11529 leader_election.cc:290] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 1 election: Requested vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:03:58.520140 11537 raft_consensus.cc:697] T bac7e5040f984ba2b7146bd36af7267e P 873357742d5843579556ee8166f23f47 [term 2 LEADER]: Becoming Leader. State: Replica: 873357742d5843579556ee8166f23f47, State: Running, Role: LEADER
I20260709 13:03:58.521234 11537 consensus_queue.cc:237] T bac7e5040f984ba2b7146bd36af7267e P 873357742d5843579556ee8166f23f47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:58.522658 11608 raft_consensus.cc:2804] T a64b7e99d6c54934a7de6f6dffcf5e94 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:03:58.471294 11094 raft_consensus.cc:1240] T 8a69f32c53f24a50b7852d143665d7b4 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Rejecting Update request from peer 10ca5ee68ad64b3283b9d235352c9d23 for earlier term 1. Current term is 2. Ops: []
I20260709 13:03:58.522773 11425 raft_consensus.cc:515] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:58.492136 11095 raft_consensus.cc:1240] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Rejecting Update request from peer 10ca5ee68ad64b3283b9d235352c9d23 for earlier term 1. Current term is 2. Ops: []
I20260709 13:03:58.450711 11545 raft_consensus.cc:515] T 5d439326f23747eaab3c1e7da2d6b785 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:58.540910 10775 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:59598 (request call id 64) took 2236 ms. Trace:
I20260709 13:03:58.523423 11608 raft_consensus.cc:697] T a64b7e99d6c54934a7de6f6dffcf5e94 P 873357742d5843579556ee8166f23f47 [term 2 LEADER]: Becoming Leader. State: Replica: 873357742d5843579556ee8166f23f47, State: Running, Role: LEADER
I20260709 13:03:58.489506 11508 raft_consensus.cc:515] T c31881534c5e4f0089c82057c1746652 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:58.531999 11427 raft_consensus.cc:493] T 32147c6d2d5a477aa38c597c7f78df5b P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:58.509922 11360 consensus_queue.cc:260] T c51782cd9d064ec0949e84e24dd4de67 P 10ca5ee68ad64b3283b9d235352c9d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
W20260709 13:03:58.556124 10921 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:38967, user_credentials={real_user=slave}} blocked reactor thread for 50600.3us
W20260709 13:03:58.532284 10845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 108407us
W20260709 13:03:58.552603 11506 consensus_meta.cc:287] T 4f9eb06ed01f4ce68be3af8a8434665b P 873357742d5843579556ee8166f23f47: Time spent flushing consensus metadata: real 0.651s	user 0.012s	sys 0.004s
I20260709 13:03:58.518306 10994 leader_election.cc:304] T 9a1716c0adf34bee8a4e99341ba5788f P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47; no voters: 186b12638a87410487467df721a3cbda
I20260709 13:03:58.563073 11102 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87cd8f5097624c8692f1d83544d8881e" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:03:58.563854 10878 raft_consensus.cc:1240] T 5d439326f23747eaab3c1e7da2d6b785 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Rejecting Update request from peer 10ca5ee68ad64b3283b9d235352c9d23 for earlier term 1. Current term is 2. Ops: []
I20260709 13:03:58.550982 11512 raft_consensus.cc:3055] T a610d1a71f52432e88fe56130f921fa2 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:03:58.570498 11098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff13ae9a47c7455f962406c6d131a3ff" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:03:58.578233 11504 leader_election.cc:290] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:03:58.559669 11427 raft_consensus.cc:515] T 32147c6d2d5a477aa38c597c7f78df5b P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:58.578471 11425 leader_election.cc:290] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 1 election: Requested vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:03:58.579489 11512 raft_consensus.cc:740] T a610d1a71f52432e88fe56130f921fa2 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0172fed22b6b4890a7fdc953fa128af4, State: Running, Role: LEADER
I20260709 13:03:58.580418 11427 leader_election.cc:290] T 32147c6d2d5a477aa38c597c7f78df5b P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:03:58.580755 11512 consensus_queue.cc:260] T a610d1a71f52432e88fe56130f921fa2 P 0172fed22b6b4890a7fdc953fa128af4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 2.1, Last appended by leader: 1, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:58.582162 11512 raft_consensus.cc:3060] T a610d1a71f52432e88fe56130f921fa2 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:03:58.556821 11610 consensus_queue.cc:1059] T 5de5f0581e3b41dea729363d6ca563bb P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 0, Time since last communication: 0.026s
I20260709 13:03:58.582427 11148 raft_consensus.cc:359] T 75492f86fde046c3900df3eac8292fd4 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:58.589529 11148 raft_consensus.cc:385] T 75492f86fde046c3900df3eac8292fd4 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:58.567020 11607 raft_consensus.cc:515] T 87cd8f5097624c8692f1d83544d8881e P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:58.589910 11148 raft_consensus.cc:740] T 75492f86fde046c3900df3eac8292fd4 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10ca5ee68ad64b3283b9d235352c9d23, State: Initialized, Role: FOLLOWER
I20260709 13:03:58.593034 10879 raft_consensus.cc:1240] T 6a9353d1d230416296627d7cf8b1d425 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Rejecting Update request from peer 873357742d5843579556ee8166f23f47 for earlier term 1. Current term is 2. Ops: []
W20260709 13:03:58.596426 10921 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:58.607884 10844 leader_election.cc:343] T a5299b8d044f41b7b68057e63d17b159 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 873357742d5843579556ee8166f23f47 (127.0.107.195:45807): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:03:58.608590 10844 leader_election.cc:304] T a5299b8d044f41b7b68057e63d17b159 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 186b12638a87410487467df721a3cbda; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47
I20260709 13:03:58.582969 10874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acc241e7273d49149984968596242712" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:03:58.595984 11100 raft_consensus.cc:2393] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 10ca5ee68ad64b3283b9d235352c9d23 in current term 1: Already voted for candidate 0172fed22b6b4890a7fdc953fa128af4 in this term.
I20260709 13:03:58.582402 11094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5de5f0581e3b41dea729363d6ca563bb" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:03:58.583451 11610 raft_consensus.cc:3055] T 5de5f0581e3b41dea729363d6ca563bb P 0172fed22b6b4890a7fdc953fa128af4 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:03:58.544901 10775 rpcz_store.cc:276] 0709 13:03:56.293493 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:56.293663 (+   170us) service_pool.cc:224] Handling call
0709 13:03:58.529751 (+2236088us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 13:03:57.824348 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 13:03:57.961652 (+137304us) write_op.cc:432] Acquiring schema lock in shared mode
0709 13:03:57.961736 (+    84us) write_op.cc:435] Acquired schema lock
0709 13:03:57.961757 (+    21us) tablet.cc:662] Decoding operations
0709 13:03:57.962274 (+   517us) write_op.cc:620] Acquiring the partition lock for write op
0709 13:03:57.962376 (+   102us) write_op.cc:641] Partition lock acquired for write op
0709 13:03:57.962405 (+    29us) tablet.cc:685] Acquiring locks for 2 operations
0709 13:03:57.962867 (+   462us) tablet.cc:701] Row locks acquired
0709 13:03:57.962900 (+    33us) write_op.cc:260] PREPARE: finished
0709 13:03:57.963085 (+   185us) write_op.cc:270] Start()
0709 13:03:57.963265 (+   180us) write_op.cc:276] Timestamp: P: 1783602237963015 usec, L: 0
0709 13:03:57.963287 (+    22us) op_driver.cc:348] REPLICATION: starting
0709 13:03:57.964175 (+   888us) log.cc:844] Serialized 928 byte log entry
0709 13:03:58.152277 (+188102us) op_driver.cc:464] REPLICATION: finished
0709 13:03:58.375283 (+223006us) write_op.cc:301] APPLY: starting
0709 13:03:58.375434 (+   151us) tablet.cc:1367] starting BulkCheckPresence
0709 13:03:58.375620 (+   186us) tablet.cc:1370] finished BulkCheckPresence
0709 13:03:58.375837 (+   217us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 13:03:58.376453 (+   616us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 13:03:58.376719 (+   266us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0709 13:03:58.376799 (+    80us) write_op.cc:312] APPLY: finished
0709 13:03:58.377112 (+   313us) log.cc:844] Serialized 28 byte log entry
0709 13:03:58.378345 (+  1233us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:03:58.378528 (+   183us) write_op.cc:454] Released schema lock
0709 13:03:58.378602 (+    74us) write_op.cc:341] FINISH: Updating metrics
0709 13:03:58.380205 (+  1603us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:03:58.402805 (+ 22600us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":8,"child_traces":[["op",{"apply.queue_time_us":219361,"apply.run_cpu_time_us":2917,"apply.run_wall_time_us":4042,"num_ops":2,"prepare.queue_time_us":175178,"prepare.run_cpu_time_us":4182,"prepare.run_wall_time_us":161516,"replication_time_us":174805,"thread_start_us":181674,"threads_started":2}]]}
I20260709 13:03:58.584685 10880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a610d1a71f52432e88fe56130f921fa2" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:03:58.628669 11151 tablet_bootstrap.cc:492] T 1e4f2241a94a41f38e481137f868b83d P 186b12638a87410487467df721a3cbda: No bootstrap required, opened a new log
I20260709 13:03:58.630291 11102 raft_consensus.cc:2393] T 87cd8f5097624c8692f1d83544d8881e P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 10ca5ee68ad64b3283b9d235352c9d23 in current term 2: Already voted for candidate 0172fed22b6b4890a7fdc953fa128af4 in this term.
I20260709 13:03:58.632375 11151 ts_tablet_manager.cc:1403] T 1e4f2241a94a41f38e481137f868b83d P 186b12638a87410487467df721a3cbda: Time spent bootstrapping tablet: real 0.612s	user 0.022s	sys 0.021s
I20260709 13:03:58.630333 11152 raft_consensus.cc:359] T acc241e7273d49149984968596242712 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:58.630829 11148 consensus_queue.cc:260] T 75492f86fde046c3900df3eac8292fd4 P 10ca5ee68ad64b3283b9d235352c9d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:58.633234 11152 raft_consensus.cc:385] T acc241e7273d49149984968596242712 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:58.633595 11152 raft_consensus.cc:740] T acc241e7273d49149984968596242712 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0172fed22b6b4890a7fdc953fa128af4, State: Initialized, Role: FOLLOWER
I20260709 13:03:58.604853 11506 raft_consensus.cc:2804] T 9a1716c0adf34bee8a4e99341ba5788f P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Leader election won for term 2
W20260709 13:03:58.637442 11067 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:58.626610 11592 raft_consensus.cc:493] T 4b37f8fca8dc42718b2a42f0883a956f P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:58.628942 11590 raft_consensus.cc:2749] T a5299b8d044f41b7b68057e63d17b159 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:03:58.630379 11555 raft_consensus.cc:493] T 0e5d327630e5438b878035d28efe6c0c P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:58.650915 11029 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5ba72ccd854430baadbc636dfd17136" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:03:58.627549 11575 raft_consensus.cc:697] T 72b111ad7b5f4753ab1d2b5ee1114840 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 LEADER]: Becoming Leader. State: Replica: 0172fed22b6b4890a7fdc953fa128af4, State: Running, Role: LEADER
I20260709 13:03:58.654273 11152 consensus_queue.cc:260] T acc241e7273d49149984968596242712 P 0172fed22b6b4890a7fdc953fa128af4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:58.634225 11021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63d44f6ba6434ab2870fe3190f4d7c2f" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
I20260709 13:03:58.655376 11029 raft_consensus.cc:2393] T e5ba72ccd854430baadbc636dfd17136 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 10ca5ee68ad64b3283b9d235352c9d23 in current term 2: Already voted for candidate 873357742d5843579556ee8166f23f47 in this term.
I20260709 13:03:58.631412 11610 raft_consensus.cc:740] T 5de5f0581e3b41dea729363d6ca563bb P 0172fed22b6b4890a7fdc953fa128af4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0172fed22b6b4890a7fdc953fa128af4, State: Running, Role: LEADER
I20260709 13:03:58.648713 10880 raft_consensus.cc:2393] T a610d1a71f52432e88fe56130f921fa2 P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 873357742d5843579556ee8166f23f47 in current term 3: Already voted for candidate 186b12638a87410487467df721a3cbda in this term.
I20260709 13:03:58.628216 10874 raft_consensus.cc:2468] T acc241e7273d49149984968596242712 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 0.
I20260709 13:03:58.609925 11589 consensus_queue.cc:1048] T e5a5f5a608504650b48fd27f163faed6 P 186b12638a87410487467df721a3cbda [LEADER]: Connected to new peer: Peer: permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.049s
I20260709 13:03:58.660907 11620 consensus_queue.cc:1059] T 5d439326f23747eaab3c1e7da2d6b785 P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.012s
I20260709 13:03:58.687539 11620 raft_consensus.cc:3055] T 5d439326f23747eaab3c1e7da2d6b785 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:03:58.687765 11626 raft_consensus.cc:493] T e89e640169e14e2d98f968f76027daa7 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:58.688190 11620 raft_consensus.cc:740] T 5d439326f23747eaab3c1e7da2d6b785 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 10ca5ee68ad64b3283b9d235352c9d23, State: Running, Role: LEADER
I20260709 13:03:58.688585 11626 raft_consensus.cc:515] T e89e640169e14e2d98f968f76027daa7 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
W20260709 13:03:58.655611 11609 log.cc:927] Time spent T bb7d67be500947b3aef4c4245e33671c P 10ca5ee68ad64b3283b9d235352c9d23: Append to log took a long time: real 0.329s	user 0.002s	sys 0.000s
I20260709 13:03:58.653842 11024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb7d67be500947b3aef4c4245e33671c" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:03:58.699873 11529 raft_consensus.cc:3055] T 72b111ad7b5f4753ab1d2b5ee1114840 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 LEADER]: Stepping down as leader of term 1
W20260709 13:03:58.701294 11591 consensus_meta.cc:287] T 8c160ddfe9b34bdaa3835d0915c1e06f P 873357742d5843579556ee8166f23f47: Time spent flushing consensus metadata: real 0.537s	user 0.005s	sys 0.011s
I20260709 13:03:58.713362 10950 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d439326f23747eaab3c1e7da2d6b785" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:03:58.671089 11025 raft_consensus.cc:1240] T e5ba72ccd854430baadbc636dfd17136 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Rejecting Update request from peer 0172fed22b6b4890a7fdc953fa128af4 for earlier term 1. Current term is 2. Ops: []
I20260709 13:03:58.668540 10871 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d156d6ae2ca4fa59a1e387edf0fea39" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:03:58.715301 10871 raft_consensus.cc:2393] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 0172fed22b6b4890a7fdc953fa128af4 in current term 1: Already voted for candidate 186b12638a87410487467df721a3cbda in this term.
I20260709 13:03:58.670935 11029 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f371d2565dd84667b5613848604949bb" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
I20260709 13:03:58.671730 11021 raft_consensus.cc:2393] T 63d44f6ba6434ab2870fe3190f4d7c2f P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 186b12638a87410487467df721a3cbda in current term 2: Already voted for candidate 873357742d5843579556ee8166f23f47 in this term.
I20260709 13:03:58.653152 11095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5ba72ccd854430baadbc636dfd17136" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:03:58.673517 11027 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a154cc8b181f460abf4880064379948b" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
I20260709 13:03:58.722924 10995 leader_election.cc:304] T acc241e7273d49149984968596242712 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:03:58.661564 11555 raft_consensus.cc:515] T 0e5d327630e5438b878035d28efe6c0c P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:58.731998 11027 raft_consensus.cc:2393] T a154cc8b181f460abf4880064379948b P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 0172fed22b6b4890a7fdc953fa128af4 in current term 1: Already voted for candidate 10ca5ee68ad64b3283b9d235352c9d23 in this term.
I20260709 13:03:58.674561 11548 consensus_queue.cc:1059] T 6a9353d1d230416296627d7cf8b1d425 P 873357742d5843579556ee8166f23f47 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:03:58.674868 11099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "525bfdbd6d934afd823f7d3ee3f9d052" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:03:58.733960 11555 leader_election.cc:290] T 0e5d327630e5438b878035d28efe6c0c P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:03:58.734402 11099 raft_consensus.cc:2468] T 525bfdbd6d934afd823f7d3ee3f9d052 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 2.
I20260709 13:03:58.659101 11608 consensus_queue.cc:237] T a64b7e99d6c54934a7de6f6dffcf5e94 P 873357742d5843579556ee8166f23f47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:58.734685 11510 raft_consensus.cc:2804] T acc241e7273d49149984968596242712 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:03:58.649880 11518 raft_consensus.cc:493] T 7709387f4d3d4f9992d97a0fbca39a40 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0172fed22b6b4890a7fdc953fa128af4)
I20260709 13:03:58.735221 11510 raft_consensus.cc:493] T acc241e7273d49149984968596242712 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:58.735754 11510 raft_consensus.cc:3060] T acc241e7273d49149984968596242712 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:03:58.735526 11518 raft_consensus.cc:515] T 7709387f4d3d4f9992d97a0fbca39a40 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:58.737811 11518 leader_election.cc:290] T 7709387f4d3d4f9992d97a0fbca39a40 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:03:58.708848 11028 raft_consensus.cc:1240] T a610d1a71f52432e88fe56130f921fa2 P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Rejecting Update request from peer 0172fed22b6b4890a7fdc953fa128af4 for earlier term 2. Current term is 3. Ops: [2.1-2.1]
I20260709 13:03:58.739032 11607 leader_election.cc:290] T 87cd8f5097624c8692f1d83544d8881e P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 2 election: Requested vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:03:58.705760 11360 raft_consensus.cc:3060] T c51782cd9d064ec0949e84e24dd4de67 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:03:58.743708 11099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d156d6ae2ca4fa59a1e387edf0fea39" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:03:58.744640 11099 raft_consensus.cc:2393] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 10ca5ee68ad64b3283b9d235352c9d23 in current term 1: Already voted for candidate 0172fed22b6b4890a7fdc953fa128af4 in this term.
I20260709 13:03:58.746882 10920 leader_election.cc:304] T 836251ebc2b74f5da88b8a8b2338e2a5 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23; no voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda
I20260709 13:03:58.747224 11020 raft_consensus.cc:1240] T 2567c2d84e4648d7ba5dc23a3df32817 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Rejecting Update request from peer 186b12638a87410487467df721a3cbda for earlier term 1. Current term is 2. Ops: []
I20260709 13:03:58.707926 11529 raft_consensus.cc:740] T 72b111ad7b5f4753ab1d2b5ee1114840 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 10ca5ee68ad64b3283b9d235352c9d23, State: Running, Role: LEADER
I20260709 13:03:58.748425 11524 raft_consensus.cc:2749] T 836251ebc2b74f5da88b8a8b2338e2a5 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 13:03:58.748534 10920 leader_election.cc:304] T a64b7e99d6c54934a7de6f6dffcf5e94 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23; no voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47
I20260709 13:03:58.679960 11148 ts_tablet_manager.cc:1434] T 75492f86fde046c3900df3eac8292fd4 P 10ca5ee68ad64b3283b9d235352c9d23: Time spent starting tablet: real 0.351s	user 0.004s	sys 0.010s
I20260709 13:03:58.683699 11575 consensus_queue.cc:237] T 72b111ad7b5f4753ab1d2b5ee1114840 P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:58.694319 11098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5a5f5a608504650b48fd27f163faed6" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:03:58.713632 10873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d156d6ae2ca4fa59a1e387edf0fea39" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
W20260709 13:03:58.656759 11067 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:58.639129 10994 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 120925us
I20260709 13:03:58.651039 11506 raft_consensus.cc:697] T 9a1716c0adf34bee8a4e99341ba5788f P 873357742d5843579556ee8166f23f47 [term 2 LEADER]: Becoming Leader. State: Replica: 873357742d5843579556ee8166f23f47, State: Running, Role: LEADER
I20260709 13:03:58.674121 11151 raft_consensus.cc:359] T 1e4f2241a94a41f38e481137f868b83d P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:58.767922 11151 raft_consensus.cc:385] T 1e4f2241a94a41f38e481137f868b83d P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:58.712522 11024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "525bfdbd6d934afd823f7d3ee3f9d052" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:03:58.726187 11095 raft_consensus.cc:2393] T e5ba72ccd854430baadbc636dfd17136 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 10ca5ee68ad64b3283b9d235352c9d23 in current term 2: Already voted for candidate 873357742d5843579556ee8166f23f47 in this term.
I20260709 13:03:58.726805 11601 raft_consensus.cc:493] T 57b6a2418c1342e7924195e9c1dbd357 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:03:58.709291 11022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a9353d1d230416296627d7cf8b1d425" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
I20260709 13:03:58.727489 11026 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a610d1a71f52432e88fe56130f921fa2" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
I20260709 13:03:58.707224 11619 raft_consensus.cc:493] T bac7e5040f984ba2b7146bd36af7267e P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0172fed22b6b4890a7fdc953fa128af4)
I20260709 13:03:58.698230 11595 raft_consensus.cc:515] T b4cd3d7dfbbf4096908ff11e9dc57141 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:58.713299 11591 raft_consensus.cc:515] T 8c160ddfe9b34bdaa3835d0915c1e06f P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:58.711711 11545 leader_election.cc:290] T 5d439326f23747eaab3c1e7da2d6b785 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 2 election: Requested vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:03:58.711809 11152 ts_tablet_manager.cc:1434] T acc241e7273d49149984968596242712 P 0172fed22b6b4890a7fdc953fa128af4: Time spent starting tablet: real 0.210s	user 0.005s	sys 0.004s
I20260709 13:03:58.659782 11592 raft_consensus.cc:515] T 4b37f8fca8dc42718b2a42f0883a956f P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:58.715771 11025 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a154cc8b181f460abf4880064379948b" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:03:58.653620 10845 leader_election.cc:304] T 4f9eb06ed01f4ce68be3af8a8434665b P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda; no voters: 873357742d5843579556ee8166f23f47
I20260709 13:03:58.689631 11620 consensus_queue.cc:260] T 5d439326f23747eaab3c1e7da2d6b785 P 10ca5ee68ad64b3283b9d235352c9d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:58.786983 10878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87cd8f5097624c8692f1d83544d8881e" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:03:58.748479 11529 consensus_queue.cc:260] T 72b111ad7b5f4753ab1d2b5ee1114840 P 10ca5ee68ad64b3283b9d235352c9d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:58.664981 11604 consensus_queue.cc:1048] T 732c4adeeff6422497aa7db4a82f535b P 873357742d5843579556ee8166f23f47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.114s
I20260709 13:03:58.792984 11148 tablet_bootstrap.cc:492] T a5299b8d044f41b7b68057e63d17b159 P 10ca5ee68ad64b3283b9d235352c9d23: Bootstrap starting.
W20260709 13:03:58.822894 10992 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 151336us
I20260709 13:03:58.801903 11510 raft_consensus.cc:515] T acc241e7273d49149984968596242712 P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:58.702289 11581 raft_consensus.cc:515] T 8c160ddfe9b34bdaa3835d0915c1e06f P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:58.791929 11617 consensus_queue.cc:1048] T 2567c2d84e4648d7ba5dc23a3df32817 P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.012s
I20260709 13:03:58.834471 11545 raft_consensus.cc:2804] T 4f9eb06ed01f4ce68be3af8a8434665b P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Leader election won for term 2
W20260709 13:03:58.834753 10845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 194244us
I20260709 13:03:58.835076 11545 raft_consensus.cc:697] T 4f9eb06ed01f4ce68be3af8a8434665b P 186b12638a87410487467df721a3cbda [term 2 LEADER]: Becoming Leader. State: Replica: 186b12638a87410487467df721a3cbda, State: Running, Role: LEADER
I20260709 13:03:58.835655 10845 leader_election.cc:304] T c51782cd9d064ec0949e84e24dd4de67 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda; no voters: 10ca5ee68ad64b3283b9d235352c9d23
I20260709 13:03:58.836100 11545 consensus_queue.cc:237] T 4f9eb06ed01f4ce68be3af8a8434665b P 186b12638a87410487467df721a3cbda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:58.837095 10845 leader_election.cc:304] T 8a69f32c53f24a50b7852d143665d7b4 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda; no voters: 10ca5ee68ad64b3283b9d235352c9d23
I20260709 13:03:58.838177 11518 raft_consensus.cc:2804] T c51782cd9d064ec0949e84e24dd4de67 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:03:58.798950 11626 leader_election.cc:290] T e89e640169e14e2d98f968f76027daa7 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:03:58.799381 11020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7709387f4d3d4f9992d97a0fbca39a40" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:03:58.801764 10871 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e5d327630e5438b878035d28efe6c0c" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:03:58.802877 11026 raft_consensus.cc:2393] T a610d1a71f52432e88fe56130f921fa2 P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 186b12638a87410487467df721a3cbda in current term 3: Already voted for candidate 873357742d5843579556ee8166f23f47 in this term.
I20260709 13:03:58.802591 11619 raft_consensus.cc:515] T bac7e5040f984ba2b7146bd36af7267e P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:58.768292 11151 raft_consensus.cc:740] T 1e4f2241a94a41f38e481137f868b83d P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 186b12638a87410487467df721a3cbda, State: Initialized, Role: FOLLOWER
I20260709 13:03:58.819526 10878 raft_consensus.cc:2410] T 87cd8f5097624c8692f1d83544d8881e P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 0172fed22b6b4890a7fdc953fa128af4 for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260709 13:03:58.819442 11066 leader_election.cc:304] T 5d439326f23747eaab3c1e7da2d6b785 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda
I20260709 13:03:58.777525 11098 raft_consensus.cc:2468] T e5a5f5a608504650b48fd27f163faed6 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 1.
I20260709 13:03:58.786906 11524 raft_consensus.cc:2749] T a64b7e99d6c54934a7de6f6dffcf5e94 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 13:03:58.827766 11149 tablet_bootstrap.cc:654] T 1e4f2241a94a41f38e481137f868b83d P 873357742d5843579556ee8166f23f47: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:58.796241 10920 leader_election.cc:304] T 5de5f0581e3b41dea729363d6ca563bb P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23; no voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda
I20260709 13:03:58.794524 11023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d371c5e27bd843469d2e2e453734cb4a" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
I20260709 13:03:58.828310 10873 raft_consensus.cc:2393] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 10ca5ee68ad64b3283b9d235352c9d23 in current term 1: Already voted for candidate 186b12638a87410487467df721a3cbda in this term.
I20260709 13:03:58.849562 11152 tablet_bootstrap.cc:492] T c31881534c5e4f0089c82057c1746652 P 0172fed22b6b4890a7fdc953fa128af4: Bootstrap starting.
I20260709 13:03:58.806239 11024 raft_consensus.cc:2468] T 525bfdbd6d934afd823f7d3ee3f9d052 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 2.
I20260709 13:03:58.830476 10874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e89e640169e14e2d98f968f76027daa7" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:03:58.856660 11510 leader_election.cc:290] T acc241e7273d49149984968596242712 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 1 election: Requested vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:03:58.856936 10874 raft_consensus.cc:2468] T e89e640169e14e2d98f968f76027daa7 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 2.
I20260709 13:03:58.857775 11066 leader_election.cc:304] T bb7d67be500947b3aef4c4245e33671c P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47
I20260709 13:03:58.838656 11590 raft_consensus.cc:2804] T 8a69f32c53f24a50b7852d143665d7b4 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:03:58.839102 11518 raft_consensus.cc:697] T c51782cd9d064ec0949e84e24dd4de67 P 186b12638a87410487467df721a3cbda [term 2 LEADER]: Becoming Leader. State: Replica: 186b12638a87410487467df721a3cbda, State: Running, Role: LEADER
I20260709 13:03:58.861099 11590 raft_consensus.cc:697] T 8a69f32c53f24a50b7852d143665d7b4 P 186b12638a87410487467df721a3cbda [term 2 LEADER]: Becoming Leader. State: Replica: 186b12638a87410487467df721a3cbda, State: Running, Role: LEADER
I20260709 13:03:58.862874 11023 raft_consensus.cc:2393] T d371c5e27bd843469d2e2e453734cb4a P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 0172fed22b6b4890a7fdc953fa128af4 in current term 1: Already voted for candidate 873357742d5843579556ee8166f23f47 in this term.
I20260709 13:03:58.862346 11590 consensus_queue.cc:237] T 8a69f32c53f24a50b7852d143665d7b4 P 186b12638a87410487467df721a3cbda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
W20260709 13:03:58.864622 10920 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 68530.5us
I20260709 13:03:58.865540 10920 leader_election.cc:304] T d270b9e5481245d9b5340a454a7cd10b P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23; no voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda
I20260709 13:03:58.866595 11427 raft_consensus.cc:2749] T 5d439326f23747eaab3c1e7da2d6b785 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:03:58.861905 11518 consensus_queue.cc:237] T c51782cd9d064ec0949e84e24dd4de67 P 186b12638a87410487467df721a3cbda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:58.867367 11427 raft_consensus.cc:2749] T bb7d67be500947b3aef4c4245e33671c P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:03:58.867214 11607 consensus_queue.cc:1059] T 6f46a5102fd64b45bf1c4dff863d2b4d P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:03:58.846725 11020 raft_consensus.cc:2393] T 7709387f4d3d4f9992d97a0fbca39a40 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 186b12638a87410487467df721a3cbda in current term 2: Already voted for candidate 873357742d5843579556ee8166f23f47 in this term.
I20260709 13:03:58.847844 11508 leader_election.cc:290] T c31881534c5e4f0089c82057c1746652 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 1 election: Requested vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:03:58.868943 11524 raft_consensus.cc:2749] T 5de5f0581e3b41dea729363d6ca563bb P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 13:03:58.869709 11524 raft_consensus.cc:2749] T d270b9e5481245d9b5340a454a7cd10b P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 13:03:58.840298 11591 leader_election.cc:290] T 8c160ddfe9b34bdaa3835d0915c1e06f P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 1 election: Requested vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:03:58.845054 11529 raft_consensus.cc:3060] T 72b111ad7b5f4753ab1d2b5ee1114840 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:03:58.845744 10994 leader_election.cc:304] T e5ba72ccd854430baadbc636dfd17136 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47; no voters: 10ca5ee68ad64b3283b9d235352c9d23
I20260709 13:03:58.838420 10845 leader_election.cc:304] T 6a9353d1d230416296627d7cf8b1d425 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda; no voters: 873357742d5843579556ee8166f23f47
I20260709 13:03:58.838896 11506 consensus_queue.cc:237] T 9a1716c0adf34bee8a4e99341ba5788f P 873357742d5843579556ee8166f23f47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:58.791414 11601 raft_consensus.cc:515] T 57b6a2418c1342e7924195e9c1dbd357 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:58.842989 11025 raft_consensus.cc:2393] T a154cc8b181f460abf4880064379948b P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0172fed22b6b4890a7fdc953fa128af4 in current term 1: Already voted for candidate 10ca5ee68ad64b3283b9d235352c9d23 in this term.
I20260709 13:03:58.711252 11610 consensus_queue.cc:260] T 5de5f0581e3b41dea729363d6ca563bb P 0172fed22b6b4890a7fdc953fa128af4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:58.908002 11151 consensus_queue.cc:260] T 1e4f2241a94a41f38e481137f868b83d P 186b12638a87410487467df721a3cbda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
W20260709 13:03:58.911274 10920 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 45837.8us
I20260709 13:03:58.843861 10880 raft_consensus.cc:1240] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Rejecting Update request from peer 0172fed22b6b4890a7fdc953fa128af4 for earlier term 1. Current term is 2. Ops: []
I20260709 13:03:58.844730 11595 leader_election.cc:290] T b4cd3d7dfbbf4096908ff11e9dc57141 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 2 election: Requested vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:03:58.854050 11620 raft_consensus.cc:3060] T 5d439326f23747eaab3c1e7da2d6b785 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:03:58.918818 11606 raft_consensus.cc:3055] T 6a9353d1d230416296627d7cf8b1d425 P 873357742d5843579556ee8166f23f47 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:03:58.918648 11103 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c31881534c5e4f0089c82057c1746652" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:03:58.919489 11102 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d439326f23747eaab3c1e7da2d6b785" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:03:58.919579 11094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7709387f4d3d4f9992d97a0fbca39a40" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:03:58.920392 11099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d156d6ae2ca4fa59a1e387edf0fea39" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:03:58.920560 11095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d7ce47b9a92d43dd88d95bcda5038597" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:03:58.923480 10871 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c160ddfe9b34bdaa3835d0915c1e06f" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:03:58.810199 10919 leader_election.cc:304] T ff13ae9a47c7455f962406c6d131a3ff P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:03:58.899200 11580 raft_consensus.cc:2804] T 6a9353d1d230416296627d7cf8b1d425 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:03:58.842453 11425 consensus_queue.cc:1059] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.083s
I20260709 13:03:58.947834 10844 leader_election.cc:304] T a610d1a71f52432e88fe56130f921fa2 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 186b12638a87410487467df721a3cbda; no voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47
I20260709 13:03:58.844069 11067 leader_election.cc:304] T f371d2565dd84667b5613848604949bb P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4; no voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47
I20260709 13:03:58.931165 11606 raft_consensus.cc:740] T 6a9353d1d230416296627d7cf8b1d425 P 873357742d5843579556ee8166f23f47 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 873357742d5843579556ee8166f23f47, State: Running, Role: LEADER
W20260709 13:03:58.961621 10920 leader_election.cc:343] T ff13ae9a47c7455f962406c6d131a3ff P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507): Illegal state: must be running to vote when last-logged opid is not known
W20260709 13:03:58.962538 10919 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:45807, user_credentials={real_user=slave}} blocked reactor thread for 171567us
I20260709 13:03:58.963320 11595 raft_consensus.cc:2804] T ff13ae9a47c7455f962406c6d131a3ff P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:03:58.963812 11595 raft_consensus.cc:493] T ff13ae9a47c7455f962406c6d131a3ff P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:58.963979 10919 leader_election.cc:304] T 525bfdbd6d934afd823f7d3ee3f9d052 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:03:58.964253 11595 raft_consensus.cc:3060] T ff13ae9a47c7455f962406c6d131a3ff P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:03:58.966934 10871 raft_consensus.cc:2393] T 8c160ddfe9b34bdaa3835d0915c1e06f P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 873357742d5843579556ee8166f23f47 in current term 1: Already voted for candidate 186b12638a87410487467df721a3cbda in this term.
I20260709 13:03:58.967031 11591 raft_consensus.cc:493] T 2567c2d84e4648d7ba5dc23a3df32817 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:03:58.933516 10992 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 70564.7us
I20260709 13:03:58.967710 11591 raft_consensus.cc:515] T 2567c2d84e4648d7ba5dc23a3df32817 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:58.968240 11099 raft_consensus.cc:2393] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 186b12638a87410487467df721a3cbda in current term 1: Already voted for candidate 0172fed22b6b4890a7fdc953fa128af4 in this term.
I20260709 13:03:58.972187 10952 raft_consensus.cc:1275] T 32147c6d2d5a477aa38c597c7f78df5b P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Refusing update from remote peer 873357742d5843579556ee8166f23f47: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260709 13:03:58.973186 10951 raft_consensus.cc:1275] T 6f46a5102fd64b45bf1c4dff863d2b4d P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Refusing update from remote peer 873357742d5843579556ee8166f23f47: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:03:58.970165 11591 leader_election.cc:290] T 2567c2d84e4648d7ba5dc23a3df32817 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:03:58.976104 11148 tablet_bootstrap.cc:654] T a5299b8d044f41b7b68057e63d17b159 P 10ca5ee68ad64b3283b9d235352c9d23: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:58.949643 11094 raft_consensus.cc:2468] T 7709387f4d3d4f9992d97a0fbca39a40 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 2.
I20260709 13:03:58.952175 11580 raft_consensus.cc:697] T 6a9353d1d230416296627d7cf8b1d425 P 186b12638a87410487467df721a3cbda [term 2 LEADER]: Becoming Leader. State: Replica: 186b12638a87410487467df721a3cbda, State: Running, Role: LEADER
W20260709 13:03:58.980366 11569 consensus_meta.cc:287] T 34064dc7bf3747dbb3876dd824ee0688 P 186b12638a87410487467df721a3cbda: Time spent flushing consensus metadata: real 0.611s	user 0.017s	sys 0.000s
I20260709 13:03:58.950711 11633 raft_consensus.cc:2804] T e5ba72ccd854430baadbc636dfd17136 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:03:58.981647 11545 raft_consensus.cc:2749] T a610d1a71f52432e88fe56130f921fa2 P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 13:03:58.999266 11151 ts_tablet_manager.cc:1434] T 1e4f2241a94a41f38e481137f868b83d P 186b12638a87410487467df721a3cbda: Time spent starting tablet: real 0.366s	user 0.014s	sys 0.002s
I20260709 13:03:58.987202 11607 raft_consensus.cc:2749] T f371d2565dd84667b5613848604949bb P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 13:03:58.991691 11553 raft_consensus.cc:2804] T 525bfdbd6d934afd823f7d3ee3f9d052 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260709 13:03:58.995820 11067 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:45807, user_credentials={real_user=slave}} blocked reactor thread for 151871us
W20260709 13:03:59.014832 11597 consensus_meta.cc:287] T e96e80ba9ac34d0998395515e5c8c41a P 10ca5ee68ad64b3283b9d235352c9d23: Time spent flushing consensus metadata: real 0.713s	user 0.009s	sys 0.006s
W20260709 13:03:58.927901 11555 consensus_queue.cc:1175] T 5de5f0581e3b41dea729363d6ca563bb P 0172fed22b6b4890a7fdc953fa128af4 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "10ca5ee68ad64b3283b9d235352c9d23" responder_term: 2 status { last_received { term: 0 index: 0 } last_committed_idx: 0 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 0172fed22b6b4890a7fdc953fa128af4 for earlier term 1. Current term is 2. Ops: [1.1-1.1]" } } last_received_current_leader { term: 0 index: 0 } }
W20260709 13:03:58.985181 10994 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 139566us
I20260709 13:03:59.001063 10880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acc241e7273d49149984968596242712" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:03:59.028456 11619 leader_election.cc:290] T bac7e5040f984ba2b7146bd36af7267e P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:03:59.028129 11606 consensus_queue.cc:260] T 6a9353d1d230416296627d7cf8b1d425 P 873357742d5843579556ee8166f23f47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:59.004797 11633 raft_consensus.cc:697] T e5ba72ccd854430baadbc636dfd17136 P 873357742d5843579556ee8166f23f47 [term 2 LEADER]: Becoming Leader. State: Replica: 873357742d5843579556ee8166f23f47, State: Running, Role: LEADER
I20260709 13:03:59.019261 11553 raft_consensus.cc:493] T 525bfdbd6d934afd823f7d3ee3f9d052 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:59.020262 10951 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2567c2d84e4648d7ba5dc23a3df32817" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:03:59.039167 11151 tablet_bootstrap.cc:492] T b67eff0634e540eab91cf8cee795969f P 186b12638a87410487467df721a3cbda: Bootstrap starting.
I20260709 13:03:59.029561 10880 raft_consensus.cc:3060] T acc241e7273d49149984968596242712 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:03:59.034410 11553 raft_consensus.cc:3060] T 525bfdbd6d934afd823f7d3ee3f9d052 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:03:59.049095 11601 leader_election.cc:290] T 57b6a2418c1342e7924195e9c1dbd357 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:03:59.049455 10877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2567c2d84e4648d7ba5dc23a3df32817" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:03:59.049750 11025 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e89e640169e14e2d98f968f76027daa7" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:03:59.052410 10876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b37f8fca8dc42718b2a42f0883a956f" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:03:59.056483 11149 tablet_bootstrap.cc:492] T 1e4f2241a94a41f38e481137f868b83d P 873357742d5843579556ee8166f23f47: No bootstrap required, opened a new log
I20260709 13:03:59.057088 11149 ts_tablet_manager.cc:1403] T 1e4f2241a94a41f38e481137f868b83d P 873357742d5843579556ee8166f23f47: Time spent bootstrapping tablet: real 0.856s	user 0.030s	sys 0.023s
I20260709 13:03:59.061230 11425 raft_consensus.cc:493] T acc241e7273d49149984968596242712 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:59.061832 11425 raft_consensus.cc:515] T acc241e7273d49149984968596242712 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:59.064121 11425 leader_election.cc:290] T acc241e7273d49149984968596242712 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:03:59.064888 11020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e5d327630e5438b878035d28efe6c0c" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
W20260709 13:03:59.065200 11598 consensus_meta.cc:287] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 10ca5ee68ad64b3283b9d235352c9d23: Time spent flushing consensus metadata: real 0.683s	user 0.009s	sys 0.004s
I20260709 13:03:59.065677 11020 raft_consensus.cc:2468] T 0e5d327630e5438b878035d28efe6c0c P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 2.
I20260709 13:03:59.065331 11149 raft_consensus.cc:359] T 1e4f2241a94a41f38e481137f868b83d P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:59.066170 11149 raft_consensus.cc:385] T 1e4f2241a94a41f38e481137f868b83d P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:59.066525 11149 raft_consensus.cc:740] T 1e4f2241a94a41f38e481137f868b83d P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 873357742d5843579556ee8166f23f47, State: Initialized, Role: FOLLOWER
I20260709 13:03:59.067028 11026 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32147c6d2d5a477aa38c597c7f78df5b" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:03:59.067229 11149 consensus_queue.cc:260] T 1e4f2241a94a41f38e481137f868b83d P 873357742d5843579556ee8166f23f47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:59.069401 11149 ts_tablet_manager.cc:1434] T 1e4f2241a94a41f38e481137f868b83d P 873357742d5843579556ee8166f23f47: Time spent starting tablet: real 0.012s	user 0.009s	sys 0.000s
I20260709 13:03:59.042775 11606 raft_consensus.cc:3060] T 6a9353d1d230416296627d7cf8b1d425 P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Advancing to term 2
W20260709 13:03:59.072973 10920 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 113220us
I20260709 13:03:59.074844 10872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57b6a2418c1342e7924195e9c1dbd357" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:03:59.082494 10873 raft_consensus.cc:1275] T e89e640169e14e2d98f968f76027daa7 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Refusing update from remote peer 873357742d5843579556ee8166f23f47: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:03:59.029503 10845 leader_election.cc:304] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 186b12638a87410487467df721a3cbda; no voters: 0172fed22b6b4890a7fdc953fa128af4, 10ca5ee68ad64b3283b9d235352c9d23
W20260709 13:03:59.087728 10845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 64822.6us
W20260709 13:03:59.088807 10845 leader_election.cc:343] T c31881534c5e4f0089c82057c1746652 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:03:59.089579 11545 raft_consensus.cc:2749] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 13:03:59.091046 11592 leader_election.cc:290] T 4b37f8fca8dc42718b2a42f0883a956f P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:03:59.091316 11427 raft_consensus.cc:3055] T 6f46a5102fd64b45bf1c4dff863d2b4d P 0172fed22b6b4890a7fdc953fa128af4 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:03:59.091777 11427 raft_consensus.cc:740] T 6f46a5102fd64b45bf1c4dff863d2b4d P 0172fed22b6b4890a7fdc953fa128af4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0172fed22b6b4890a7fdc953fa128af4, State: Running, Role: LEADER
I20260709 13:03:59.092272 11633 consensus_queue.cc:237] T e5ba72ccd854430baadbc636dfd17136 P 873357742d5843579556ee8166f23f47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:59.092813 11427 consensus_queue.cc:260] T 6f46a5102fd64b45bf1c4dff863d2b4d P 0172fed22b6b4890a7fdc953fa128af4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:59.097843 11149 tablet_bootstrap.cc:492] T 75492f86fde046c3900df3eac8292fd4 P 873357742d5843579556ee8166f23f47: Bootstrap starting.
I20260709 13:03:59.057803 11518 raft_consensus.cc:493] T 4c2a35bb99374e93b573cf870d04faaa P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 10ca5ee68ad64b3283b9d235352c9d23)
I20260709 13:03:59.099776 11518 raft_consensus.cc:515] T 4c2a35bb99374e93b573cf870d04faaa P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:59.101517 11545 raft_consensus.cc:493] T 87cd8f5097624c8692f1d83544d8881e P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:59.093293 11152 tablet_bootstrap.cc:654] T c31881534c5e4f0089c82057c1746652 P 0172fed22b6b4890a7fdc953fa128af4: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:59.107223 10920 leader_election.cc:304] T 87cd8f5097624c8692f1d83544d8881e P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23; no voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda
I20260709 13:03:59.026995 11595 raft_consensus.cc:515] T ff13ae9a47c7455f962406c6d131a3ff P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:59.113897 10873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acc241e7273d49149984968596242712" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:03:59.114622 10994 leader_election.cc:304] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47; no voters: 10ca5ee68ad64b3283b9d235352c9d23
I20260709 13:03:59.115815 11601 raft_consensus.cc:2749] T 87cd8f5097624c8692f1d83544d8881e P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 13:03:59.113814 10845 leader_election.cc:304] T d7ce47b9a92d43dd88d95bcda5038597 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 186b12638a87410487467df721a3cbda; no voters: 0172fed22b6b4890a7fdc953fa128af4, 10ca5ee68ad64b3283b9d235352c9d23
I20260709 13:03:59.118227 11025 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f1466124dbd4f6ebb3c4d62ca48bf96" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:03:59.118625 10845 leader_election.cc:304] T 7709387f4d3d4f9992d97a0fbca39a40 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda; no voters: 873357742d5843579556ee8166f23f47
I20260709 13:03:59.118988 11025 raft_consensus.cc:2468] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 2.
I20260709 13:03:59.057353 11548 consensus_queue.cc:1048] T 32147c6d2d5a477aa38c597c7f78df5b P 873357742d5843579556ee8166f23f47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.063s
I20260709 13:03:59.120455 11569 raft_consensus.cc:2804] T 7709387f4d3d4f9992d97a0fbca39a40 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:03:59.120729 10992 leader_election.cc:304] T 2567c2d84e4648d7ba5dc23a3df32817 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 873357742d5843579556ee8166f23f47; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda
I20260709 13:03:59.121384 11548 raft_consensus.cc:2804] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:03:59.121441 11583 consensus_queue.cc:1048] T 6f46a5102fd64b45bf1c4dff863d2b4d P 873357742d5843579556ee8166f23f47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:03:59.120919 11569 raft_consensus.cc:493] T 7709387f4d3d4f9992d97a0fbca39a40 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0172fed22b6b4890a7fdc953fa128af4)
I20260709 13:03:59.119935 11590 raft_consensus.cc:2749] T d7ce47b9a92d43dd88d95bcda5038597 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 13:03:59.122267 11066 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:59.094231 11427 raft_consensus.cc:3060] T 6f46a5102fd64b45bf1c4dff863d2b4d P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:03:59.128665 11094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acc241e7273d49149984968596242712" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:03:59.139957 11508 raft_consensus.cc:493] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:59.140823 11592 raft_consensus.cc:2749] T 2567c2d84e4648d7ba5dc23a3df32817 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:03:59.140573 11508 raft_consensus.cc:515] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:59.143056 10920 leader_election.cc:304] T e5ba72ccd854430baadbc636dfd17136 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23; no voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47
I20260709 13:03:59.034957 11580 consensus_queue.cc:237] T 6a9353d1d230416296627d7cf8b1d425 P 186b12638a87410487467df721a3cbda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:59.129901 11569 raft_consensus.cc:3060] T 7709387f4d3d4f9992d97a0fbca39a40 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:03:59.189961 10920 leader_election.cc:304] T e5a5f5a608504650b48fd27f163faed6 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 10ca5ee68ad64b3283b9d235352c9d23; no voters: 186b12638a87410487467df721a3cbda
I20260709 13:03:59.102216 11545 raft_consensus.cc:515] T 87cd8f5097624c8692f1d83544d8881e P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:59.111922 11591 consensus_queue.cc:1048] T e89e640169e14e2d98f968f76027daa7 P 873357742d5843579556ee8166f23f47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.036s
I20260709 13:03:59.141651 11642 raft_consensus.cc:493] T 65bf0a62f4d84ef18dd6b23e082724b8 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 10ca5ee68ad64b3283b9d235352c9d23)
I20260709 13:03:59.198355 11067 leader_election.cc:304] T 0e5d327630e5438b878035d28efe6c0c P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:03:59.209401 11095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acc241e7273d49149984968596242712" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:03:59.138615 11099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a610d1a71f52432e88fe56130f921fa2" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:03:59.212903 11151 tablet_bootstrap.cc:654] T b67eff0634e540eab91cf8cee795969f P 186b12638a87410487467df721a3cbda: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:59.145301 10880 raft_consensus.cc:2468] T acc241e7273d49149984968596242712 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 1.
I20260709 13:03:59.193636 11598 raft_consensus.cc:3060] T e5ba72ccd854430baadbc636dfd17136 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:03:59.145326 11094 raft_consensus.cc:2468] T acc241e7273d49149984968596242712 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 0.
I20260709 13:03:59.144987 11589 raft_consensus.cc:493] T e96e80ba9ac34d0998395515e5c8c41a P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 10ca5ee68ad64b3283b9d235352c9d23)
I20260709 13:03:59.157611 10951 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b37f8fca8dc42718b2a42f0883a956f" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:03:59.214715 11360 raft_consensus.cc:2804] T e5a5f5a608504650b48fd27f163faed6 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:03:59.198753 11590 raft_consensus.cc:493] T 0e5d327630e5438b878035d28efe6c0c P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0172fed22b6b4890a7fdc953fa128af4)
I20260709 13:03:59.237478 11026 raft_consensus.cc:1240] T 6f46a5102fd64b45bf1c4dff863d2b4d P 873357742d5843579556ee8166f23f47 [term 2 LEADER]: Rejecting Update request from peer 0172fed22b6b4890a7fdc953fa128af4 for earlier term 1. Current term is 2. Ops: []
I20260709 13:03:59.228389 11610 raft_consensus.cc:3060] T 5de5f0581e3b41dea729363d6ca563bb P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:03:59.268038 11148 tablet_bootstrap.cc:492] T a5299b8d044f41b7b68057e63d17b159 P 10ca5ee68ad64b3283b9d235352c9d23: No bootstrap required, opened a new log
I20260709 13:03:59.230160 11620 raft_consensus.cc:493] T 75492f86fde046c3900df3eac8292fd4 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:59.233135 11548 raft_consensus.cc:697] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 873357742d5843579556ee8166f23f47 [term 2 LEADER]: Becoming Leader. State: Replica: 873357742d5843579556ee8166f23f47, State: Running, Role: LEADER
I20260709 13:03:59.312747 11548 consensus_queue.cc:237] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 873357742d5843579556ee8166f23f47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:59.315886 10995 leader_election.cc:304] T acc241e7273d49149984968596242712 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:03:59.317646 11581 leader_election.cc:290] T 8c160ddfe9b34bdaa3835d0915c1e06f P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 1 election: Requested vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:03:59.271938 11360 raft_consensus.cc:493] T e5a5f5a608504650b48fd27f163faed6 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 186b12638a87410487467df721a3cbda)
I20260709 13:03:59.319568 11360 raft_consensus.cc:3060] T e5a5f5a608504650b48fd27f163faed6 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:03:59.323205 11149 tablet_bootstrap.cc:654] T 75492f86fde046c3900df3eac8292fd4 P 873357742d5843579556ee8166f23f47: Neither blocks nor log segments found. Creating new log.
W20260709 13:03:59.265161 11067 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:45807, user_credentials={real_user=slave}} blocked reactor thread for 108986us
I20260709 13:03:59.307308 11148 ts_tablet_manager.cc:1403] T a5299b8d044f41b7b68057e63d17b159 P 10ca5ee68ad64b3283b9d235352c9d23: Time spent bootstrapping tablet: real 0.517s	user 0.013s	sys 0.019s
W20260709 13:03:59.308568 10920 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 118735us
I20260709 13:03:59.267450 11555 raft_consensus.cc:2804] T 0e5d327630e5438b878035d28efe6c0c P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260709 13:03:59.295207 11066 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:59.328184 10921 leader_election.cc:304] T e89e640169e14e2d98f968f76027daa7 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda; no voters: 873357742d5843579556ee8166f23f47
I20260709 13:03:59.335759 10954 raft_consensus.cc:1240] T d270b9e5481245d9b5340a454a7cd10b P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Rejecting Update request from peer 0172fed22b6b4890a7fdc953fa128af4 for earlier term 1. Current term is 2. Ops: []
I20260709 13:03:59.336691 10921 leader_election.cc:304] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23; no voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda
I20260709 13:03:59.337173 10949 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f1466124dbd4f6ebb3c4d62ca48bf96" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:03:59.337318 10948 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a154cc8b181f460abf4880064379948b" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:03:59.337894 10949 raft_consensus.cc:2468] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 2.
I20260709 13:03:59.338254 11639 raft_consensus.cc:2804] T e89e640169e14e2d98f968f76027daa7 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:03:59.338713 11639 raft_consensus.cc:493] T e89e640169e14e2d98f968f76027daa7 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:59.339104 11639 raft_consensus.cc:3060] T e89e640169e14e2d98f968f76027daa7 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:03:59.339241 10949 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d156d6ae2ca4fa59a1e387edf0fea39" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:03:59.340154 10949 raft_consensus.cc:2393] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 0172fed22b6b4890a7fdc953fa128af4 in current term 1: Already voted for candidate 10ca5ee68ad64b3283b9d235352c9d23 in this term.
I20260709 13:03:59.335646 11555 raft_consensus.cc:493] T 0e5d327630e5438b878035d28efe6c0c P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:59.340396 10946 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87cd8f5097624c8692f1d83544d8881e" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:03:59.340569 11555 raft_consensus.cc:3060] T 0e5d327630e5438b878035d28efe6c0c P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:03:59.341310 10946 raft_consensus.cc:2393] T 87cd8f5097624c8692f1d83544d8881e P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 0172fed22b6b4890a7fdc953fa128af4 in current term 2: Already voted for candidate 10ca5ee68ad64b3283b9d235352c9d23 in this term.
I20260709 13:03:59.341706 10954 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32147c6d2d5a477aa38c597c7f78df5b" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:03:59.341331 10955 raft_consensus.cc:1240] T b4cd3d7dfbbf4096908ff11e9dc57141 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Rejecting Update request from peer 0172fed22b6b4890a7fdc953fa128af4 for earlier term 1. Current term is 2. Ops: []
I20260709 13:03:59.342633 11619 raft_consensus.cc:2749] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 13:03:59.343562 11103 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c160ddfe9b34bdaa3835d0915c1e06f" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:03:59.344218 11103 raft_consensus.cc:3060] T 8c160ddfe9b34bdaa3835d0915c1e06f P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:03:59.310544 11025 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acc241e7273d49149984968596242712" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:03:59.346344 11099 raft_consensus.cc:2410] T a610d1a71f52432e88fe56130f921fa2 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 873357742d5843579556ee8166f23f47 for term 3 because replica has last-logged OpId of term: 2 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260709 13:03:59.255410 11642 raft_consensus.cc:515] T 65bf0a62f4d84ef18dd6b23e082724b8 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:59.271564 11589 raft_consensus.cc:515] T e96e80ba9ac34d0998395515e5c8c41a P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:59.372282 11023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c160ddfe9b34bdaa3835d0915c1e06f" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
I20260709 13:03:59.369690 11583 raft_consensus.cc:2804] T acc241e7273d49149984968596242712 P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:03:59.384325 11024 raft_consensus.cc:1275] T 63d44f6ba6434ab2870fe3190f4d7c2f P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Refusing update from remote peer 186b12638a87410487467df721a3cbda: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:03:59.386902 11027 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bac7e5040f984ba2b7146bd36af7267e" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:03:59.387182 10955 raft_consensus.cc:1275] T 63d44f6ba6434ab2870fe3190f4d7c2f P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Refusing update from remote peer 186b12638a87410487467df721a3cbda: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:03:59.387805 10946 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c31881534c5e4f0089c82057c1746652" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
W20260709 13:03:59.388321 10992 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 44017.6us
I20260709 13:03:59.389282 11022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4cd3d7dfbbf4096908ff11e9dc57141" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
I20260709 13:03:59.390232 11029 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57b6a2418c1342e7924195e9c1dbd357" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:03:59.391386 11095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4cd3d7dfbbf4096908ff11e9dc57141" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:03:59.303339 11590 raft_consensus.cc:515] T 0e5d327630e5438b878035d28efe6c0c P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:59.345005 11529 raft_consensus.cc:493] T c31881534c5e4f0089c82057c1746652 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:59.393304 11508 leader_election.cc:290] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:03:59.416905 11023 raft_consensus.cc:2393] T 8c160ddfe9b34bdaa3835d0915c1e06f P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 186b12638a87410487467df721a3cbda in current term 1: Already voted for candidate 873357742d5843579556ee8166f23f47 in this term.
I20260709 13:03:59.428396 11529 raft_consensus.cc:515] T c31881534c5e4f0089c82057c1746652 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:59.429558 10946 raft_consensus.cc:3060] T c31881534c5e4f0089c82057c1746652 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:03:59.425166 11098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bac7e5040f984ba2b7146bd36af7267e" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:03:59.425462 10955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c2a35bb99374e93b573cf870d04faaa" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:03:59.393644 11513 consensus_queue.cc:1059] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.028s
W20260709 13:03:59.412453 11648 log.cc:927] Time spent T 72b111ad7b5f4753ab1d2b5ee1114840 P 0172fed22b6b4890a7fdc953fa128af4: Append to log took a long time: real 0.142s	user 0.001s	sys 0.000s
I20260709 13:03:59.404460 11028 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555217d45d9b4ee492dbb3ddaf0f7c02" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:03:59.435314 10955 raft_consensus.cc:2468] T 4c2a35bb99374e93b573cf870d04faaa P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 2.
I20260709 13:03:59.334055 11620 raft_consensus.cc:515] T 75492f86fde046c3900df3eac8292fd4 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:59.405545 11583 raft_consensus.cc:697] T acc241e7273d49149984968596242712 P 873357742d5843579556ee8166f23f47 [term 1 LEADER]: Becoming Leader. State: Replica: 873357742d5843579556ee8166f23f47, State: Running, Role: LEADER
W20260709 13:03:59.423300 11650 log.cc:927] Time spent T e5a5f5a608504650b48fd27f163faed6 P 0172fed22b6b4890a7fdc953fa128af4: Append to log took a long time: real 0.102s	user 0.001s	sys 0.000s
I20260709 13:03:59.459019 11098 raft_consensus.cc:2393] T bac7e5040f984ba2b7146bd36af7267e P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 10ca5ee68ad64b3283b9d235352c9d23 in current term 2: Already voted for candidate 873357742d5843579556ee8166f23f47 in this term.
I20260709 13:03:59.461707 11097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c160ddfe9b34bdaa3835d0915c1e06f" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
W20260709 13:03:59.463115 11649 log.cc:927] Time spent T c51782cd9d064ec0949e84e24dd4de67 P 186b12638a87410487467df721a3cbda: Append to log took a long time: real 0.159s	user 0.000s	sys 0.001s
I20260709 13:03:59.466538 10953 raft_consensus.cc:1240] T 836251ebc2b74f5da88b8a8b2338e2a5 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Rejecting Update request from peer 0172fed22b6b4890a7fdc953fa128af4 for earlier term 1. Current term is 2. Ops: []
I20260709 13:03:59.449160 11595 leader_election.cc:290] T ff13ae9a47c7455f962406c6d131a3ff P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 1 election: Requested vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:03:59.446417 11518 leader_election.cc:290] T 4c2a35bb99374e93b573cf870d04faaa P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:03:59.412971 11067 leader_election.cc:304] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:03:59.459231 11504 consensus_queue.cc:1059] T d270b9e5481245d9b5340a454a7cd10b P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.089s
I20260709 13:03:59.550836 11583 consensus_queue.cc:237] T acc241e7273d49149984968596242712 P 873357742d5843579556ee8166f23f47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:59.557251 10994 leader_election.cc:304] T a610d1a71f52432e88fe56130f921fa2 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 873357742d5843579556ee8166f23f47; no voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda
I20260709 13:03:59.558044 11553 raft_consensus.cc:515] T 525bfdbd6d934afd823f7d3ee3f9d052 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:59.496471 11598 raft_consensus.cc:2749] T e5ba72ccd854430baadbc636dfd17136 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:03:59.559808 11506 raft_consensus.cc:493] T 1e4f2241a94a41f38e481137f868b83d P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:59.560647 11553 leader_election.cc:290] T 525bfdbd6d934afd823f7d3ee3f9d052 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 3 election: Requested vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:03:59.560515 11506 raft_consensus.cc:515] T 1e4f2241a94a41f38e481137f868b83d P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:59.564677 11548 raft_consensus.cc:2749] T a610d1a71f52432e88fe56130f921fa2 P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20260709 13:03:59.565745 10995 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:38967, user_credentials={real_user=slave}} blocked reactor thread for 92469.6us
I20260709 13:03:59.566720 11628 raft_consensus.cc:493] T d71ca3a1ba6b48478aaf62741bf3cf6a P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:03:59.567350 11628 raft_consensus.cc:515] T d71ca3a1ba6b48478aaf62741bf3cf6a P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:59.568270 11591 consensus_queue.cc:1048] T f371d2565dd84667b5613848604949bb P 873357742d5843579556ee8166f23f47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:03:59.569550 11628 leader_election.cc:290] T d71ca3a1ba6b48478aaf62741bf3cf6a P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:03:59.572609 11098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555217d45d9b4ee492dbb3ddaf0f7c02" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:03:59.456594 11529 leader_election.cc:290] T c31881534c5e4f0089c82057c1746652 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:03:59.574767 11148 raft_consensus.cc:359] T a5299b8d044f41b7b68057e63d17b159 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:59.610961 11148 raft_consensus.cc:385] T a5299b8d044f41b7b68057e63d17b159 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:59.626053 11664 raft_consensus.cc:493] T 9a1716c0adf34bee8a4e99341ba5788f P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:59.626358 10946 raft_consensus.cc:2468] T c31881534c5e4f0089c82057c1746652 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 1.
I20260709 13:03:59.626842 11664 raft_consensus.cc:515] T 9a1716c0adf34bee8a4e99341ba5788f P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:59.629734 11664 leader_election.cc:290] T 9a1716c0adf34bee8a4e99341ba5788f P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:03:59.627192 11589 leader_election.cc:290] T e96e80ba9ac34d0998395515e5c8c41a P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:03:59.611508 11148 raft_consensus.cc:740] T a5299b8d044f41b7b68057e63d17b159 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10ca5ee68ad64b3283b9d235352c9d23, State: Initialized, Role: FOLLOWER
I20260709 13:03:59.639266 11148 consensus_queue.cc:260] T a5299b8d044f41b7b68057e63d17b159 P 10ca5ee68ad64b3283b9d235352c9d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:59.637645 11518 raft_consensus.cc:493] T 9a1716c0adf34bee8a4e99341ba5788f P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0172fed22b6b4890a7fdc953fa128af4)
I20260709 13:03:59.645910 11513 raft_consensus.cc:2804] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:03:59.646292 11098 raft_consensus.cc:1275] T 4f9eb06ed01f4ce68be3af8a8434665b P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Refusing update from remote peer 186b12638a87410487467df721a3cbda: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:03:59.646428 11513 raft_consensus.cc:493] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:59.646909 11513 raft_consensus.cc:3060] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:03:59.648514 11620 leader_election.cc:290] T 75492f86fde046c3900df3eac8292fd4 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:03:59.640228 11606 raft_consensus.cc:493] T 11de93cd9b9d457489ff1cb80f976c9a P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:59.656174 11151 tablet_bootstrap.cc:492] T b67eff0634e540eab91cf8cee795969f P 186b12638a87410487467df721a3cbda: No bootstrap required, opened a new log
I20260709 13:03:59.656929 11151 ts_tablet_manager.cc:1403] T b67eff0634e540eab91cf8cee795969f P 186b12638a87410487467df721a3cbda: Time spent bootstrapping tablet: real 0.627s	user 0.010s	sys 0.014s
W20260709 13:03:59.579967 10843 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 105050us
I20260709 13:03:59.664563 11029 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "525bfdbd6d934afd823f7d3ee3f9d052" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
I20260709 13:03:59.664973 11023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75492f86fde046c3900df3eac8292fd4" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:03:59.665271 11029 raft_consensus.cc:3060] T 525bfdbd6d934afd823f7d3ee3f9d052 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:03:59.668207 10874 raft_consensus.cc:1275] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Refusing update from remote peer 873357742d5843579556ee8166f23f47: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:03:59.651619 11642 leader_election.cc:290] T 65bf0a62f4d84ef18dd6b23e082724b8 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:03:59.669559 11529 raft_consensus.cc:493] T 65bf0a62f4d84ef18dd6b23e082724b8 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:59.670071 11529 raft_consensus.cc:515] T 65bf0a62f4d84ef18dd6b23e082724b8 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:59.670626 11513 raft_consensus.cc:515] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:59.646155 11518 raft_consensus.cc:515] T 9a1716c0adf34bee8a4e99341ba5788f P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:59.672348 11152 tablet_bootstrap.cc:492] T c31881534c5e4f0089c82057c1746652 P 0172fed22b6b4890a7fdc953fa128af4: No bootstrap required, opened a new log
I20260709 13:03:59.673071 11152 ts_tablet_manager.cc:1403] T c31881534c5e4f0089c82057c1746652 P 0172fed22b6b4890a7fdc953fa128af4: Time spent bootstrapping tablet: real 0.845s	user 0.028s	sys 0.066s
I20260709 13:03:59.672741 11513 leader_election.cc:290] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 3 election: Requested vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:03:59.666118 11028 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff13ae9a47c7455f962406c6d131a3ff" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
I20260709 13:03:59.608186 11508 consensus_queue.cc:1048] T 63d44f6ba6434ab2870fe3190f4d7c2f P 186b12638a87410487467df721a3cbda [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.052s
W20260709 13:03:59.632112 11067 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:45807, user_credentials={real_user=slave}} blocked reactor thread for 231085us
I20260709 13:03:59.686710 11029 raft_consensus.cc:2468] T 525bfdbd6d934afd823f7d3ee3f9d052 P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 3.
I20260709 13:03:59.650426 11606 raft_consensus.cc:515] T 11de93cd9b9d457489ff1cb80f976c9a P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:59.688644 11148 ts_tablet_manager.cc:1434] T a5299b8d044f41b7b68057e63d17b159 P 10ca5ee68ad64b3283b9d235352c9d23: Time spent starting tablet: real 0.322s	user 0.009s	sys 0.001s
I20260709 13:03:59.670626 11592 raft_consensus.cc:493] T 7709387f4d3d4f9992d97a0fbca39a40 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:59.699739 11553 consensus_queue.cc:1048] T a154cc8b181f460abf4880064379948b P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.007s
I20260709 13:03:59.698417 11027 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f1466124dbd4f6ebb3c4d62ca48bf96" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
I20260709 13:03:59.698319 11023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d71ca3a1ba6b48478aaf62741bf3cf6a" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:03:59.703328 11569 raft_consensus.cc:515] T 7709387f4d3d4f9992d97a0fbca39a40 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:03:59.694748 11518 leader_election.cc:290] T 9a1716c0adf34bee8a4e99341ba5788f P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:03:59.704574 11148 tablet_bootstrap.cc:492] T a977842ba4b744f4bb2febd55d286006 P 10ca5ee68ad64b3283b9d235352c9d23: Bootstrap starting.
I20260709 13:03:59.703938 11506 leader_election.cc:290] T 1e4f2241a94a41f38e481137f868b83d P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:03:59.698151 11664 raft_consensus.cc:493] T a64b7e99d6c54934a7de6f6dffcf5e94 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:59.710312 11664 raft_consensus.cc:515] T a64b7e99d6c54934a7de6f6dffcf5e94 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:59.710544 10874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e4f2241a94a41f38e481137f868b83d" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:03:59.698310 10919 leader_election.cc:304] T 57b6a2418c1342e7924195e9c1dbd357 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23; no voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47
W20260709 13:03:59.712285 10919 leader_election.cc:343] T 75492f86fde046c3900df3eac8292fd4 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 873357742d5843579556ee8166f23f47 (127.0.107.195:45807): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:03:59.713042 10919 leader_election.cc:304] T 525bfdbd6d934afd823f7d3ee3f9d052 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:03:59.698064 11024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a1716c0adf34bee8a4e99341ba5788f" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:03:59.706215 11592 raft_consensus.cc:515] T 7709387f4d3d4f9992d97a0fbca39a40 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
W20260709 13:03:59.689095 11427 consensus_meta.cc:287] T 6f46a5102fd64b45bf1c4dff863d2b4d P 0172fed22b6b4890a7fdc953fa128af4: Time spent flushing consensus metadata: real 0.551s	user 0.012s	sys 0.001s
I20260709 13:03:59.703280 11022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65bf0a62f4d84ef18dd6b23e082724b8" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:03:59.714931 11569 leader_election.cc:290] T 7709387f4d3d4f9992d97a0fbca39a40 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 2 election: Requested vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:03:59.719305 11102 raft_consensus.cc:1275] T 6f46a5102fd64b45bf1c4dff863d2b4d P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Refusing update from remote peer 873357742d5843579556ee8166f23f47: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:03:59.720391 11620 raft_consensus.cc:2749] T 57b6a2418c1342e7924195e9c1dbd357 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:03:59.720685 11097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a1716c0adf34bee8a4e99341ba5788f" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:03:59.721165 11620 raft_consensus.cc:2804] T 525bfdbd6d934afd823f7d3ee3f9d052 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:03:59.721376 11606 leader_election.cc:290] T 11de93cd9b9d457489ff1cb80f976c9a P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:03:59.721729 11620 raft_consensus.cc:697] T 525bfdbd6d934afd823f7d3ee3f9d052 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 LEADER]: Becoming Leader. State: Replica: 10ca5ee68ad64b3283b9d235352c9d23, State: Running, Role: LEADER
I20260709 13:03:59.704634 10843 leader_election.cc:304] T 4c2a35bb99374e93b573cf870d04faaa P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda; no voters: 
I20260709 13:03:59.722198 11099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7709387f4d3d4f9992d97a0fbca39a40" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:03:59.722780 11620 consensus_queue.cc:237] T 525bfdbd6d934afd823f7d3ee3f9d052 P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:03:59.716897 11592 leader_election.cc:290] T 7709387f4d3d4f9992d97a0fbca39a40 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:03:59.726711 11569 raft_consensus.cc:2804] T 4c2a35bb99374e93b573cf870d04faaa P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:03:59.727202 11569 raft_consensus.cc:493] T 4c2a35bb99374e93b573cf870d04faaa P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Starting leader election (detected failure of leader 10ca5ee68ad64b3283b9d235352c9d23)
W20260709 13:03:59.728413 11644 log.cc:927] Time spent T 8a69f32c53f24a50b7852d143665d7b4 P 186b12638a87410487467df721a3cbda: Append to log took a long time: real 0.504s	user 0.002s	sys 0.000s
I20260709 13:03:59.721539 11097 raft_consensus.cc:2393] T 9a1716c0adf34bee8a4e99341ba5788f P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 186b12638a87410487467df721a3cbda in current term 2: Already voted for candidate 873357742d5843579556ee8166f23f47 in this term.
I20260709 13:03:59.729672 11569 raft_consensus.cc:3060] T 4c2a35bb99374e93b573cf870d04faaa P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:03:59.731165 11553 raft_consensus.cc:493] T a5299b8d044f41b7b68057e63d17b159 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:59.731693 11553 raft_consensus.cc:515] T a5299b8d044f41b7b68057e63d17b159 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:59.732283 10872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7709387f4d3d4f9992d97a0fbca39a40" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:03:59.732968 10872 raft_consensus.cc:2468] T 7709387f4d3d4f9992d97a0fbca39a40 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 2.
I20260709 13:03:59.733798 11553 leader_election.cc:290] T a5299b8d044f41b7b68057e63d17b159 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:03:59.735327 11152 raft_consensus.cc:359] T c31881534c5e4f0089c82057c1746652 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:59.735878 11152 raft_consensus.cc:385] T c31881534c5e4f0089c82057c1746652 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:59.672426 11529 leader_election.cc:290] T 65bf0a62f4d84ef18dd6b23e082724b8 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:03:59.736128 11152 raft_consensus.cc:740] T c31881534c5e4f0089c82057c1746652 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0172fed22b6b4890a7fdc953fa128af4, State: Initialized, Role: FOLLOWER
I20260709 13:03:59.739902 10995 leader_election.cc:304] T 7709387f4d3d4f9992d97a0fbca39a40 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:03:59.735008 11151 raft_consensus.cc:359] T b67eff0634e540eab91cf8cee795969f P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:59.741678 11151 raft_consensus.cc:385] T b67eff0634e540eab91cf8cee795969f P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:03:59.742074 11151 raft_consensus.cc:740] T b67eff0634e540eab91cf8cee795969f P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 186b12638a87410487467df721a3cbda, State: Initialized, Role: FOLLOWER
I20260709 13:03:59.744463 11152 consensus_queue.cc:260] T c31881534c5e4f0089c82057c1746652 P 0172fed22b6b4890a7fdc953fa128af4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:59.746930 11151 consensus_queue.cc:260] T b67eff0634e540eab91cf8cee795969f P 186b12638a87410487467df721a3cbda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:59.690120 11028 raft_consensus.cc:3060] T ff13ae9a47c7455f962406c6d131a3ff P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:03:59.688252 11642 raft_consensus.cc:493] T 1e4f2241a94a41f38e481137f868b83d P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:59.756094 11152 ts_tablet_manager.cc:1434] T c31881534c5e4f0089c82057c1746652 P 0172fed22b6b4890a7fdc953fa128af4: Time spent starting tablet: real 0.083s	user 0.008s	sys 0.002s
W20260709 13:03:59.748174 10950 consensus_meta.cc:287] T 5d439326f23747eaab3c1e7da2d6b785 P 10ca5ee68ad64b3283b9d235352c9d23: Time spent flushing consensus metadata: real 0.514s	user 0.003s	sys 0.010s
I20260709 13:03:59.692243 11595 raft_consensus.cc:493] T 34064dc7bf3747dbb3876dd824ee0688 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:59.719156 11022 raft_consensus.cc:2468] T 65bf0a62f4d84ef18dd6b23e082724b8 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 2.
I20260709 13:03:59.718129 11103 raft_consensus.cc:2468] T 8c160ddfe9b34bdaa3835d0915c1e06f P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 1.
I20260709 13:03:59.719133 11518 consensus_queue.cc:1048] T 4f9eb06ed01f4ce68be3af8a8434665b P 186b12638a87410487467df721a3cbda [LEADER]: Connected to new peer: Peer: permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.039s
I20260709 13:03:59.718981 11654 consensus_queue.cc:1048] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 873357742d5843579556ee8166f23f47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.040s
I20260709 13:03:59.747586 11592 raft_consensus.cc:2804] T 7709387f4d3d4f9992d97a0fbca39a40 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:03:59.788256 10774 catalog_manager.cc:5697] T a154cc8b181f460abf4880064379948b P 10ca5ee68ad64b3283b9d235352c9d23 reported cstate change: term changed from 0 to 1, leader changed from <none> to 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194). New cstate: current_term: 1 leader_uuid: "10ca5ee68ad64b3283b9d235352c9d23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } health_report { overall_health: UNKNOWN } } }
I20260709 13:03:59.758273 11642 raft_consensus.cc:515] T 1e4f2241a94a41f38e481137f868b83d P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
W20260709 13:03:59.759899 10994 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 65097us
I20260709 13:03:59.796456 11152 tablet_bootstrap.cc:492] T ff13ae9a47c7455f962406c6d131a3ff P 0172fed22b6b4890a7fdc953fa128af4: Bootstrap starting.
I20260709 13:03:59.821478 11148 tablet_bootstrap.cc:654] T a977842ba4b744f4bb2febd55d286006 P 10ca5ee68ad64b3283b9d235352c9d23: Neither blocks nor log segments found. Creating new log.
I20260709 13:03:59.868623 11151 ts_tablet_manager.cc:1434] T b67eff0634e540eab91cf8cee795969f P 186b12638a87410487467df721a3cbda: Time spent starting tablet: real 0.211s	user 0.004s	sys 0.006s
I20260709 13:03:59.811421 10955 raft_consensus.cc:1275] T 8a69f32c53f24a50b7852d143665d7b4 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Refusing update from remote peer 186b12638a87410487467df721a3cbda: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:03:59.790375 10774 catalog_manager.cc:5697] T 2567c2d84e4648d7ba5dc23a3df32817 P 10ca5ee68ad64b3283b9d235352c9d23 reported cstate change: term changed from 1 to 2, leader changed from 186b12638a87410487467df721a3cbda (127.0.107.193) to 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194). New cstate: current_term: 2 leader_uuid: "10ca5ee68ad64b3283b9d235352c9d23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } health_report { overall_health: UNKNOWN } } }
I20260709 13:03:59.808408 11149 tablet_bootstrap.cc:492] T 75492f86fde046c3900df3eac8292fd4 P 873357742d5843579556ee8166f23f47: No bootstrap required, opened a new log
W20260709 13:03:59.908778 10844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:45807, user_credentials={real_user=slave}} blocked reactor thread for 64012.3us
I20260709 13:03:59.760403 10950 raft_consensus.cc:2468] T 5d439326f23747eaab3c1e7da2d6b785 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 2.
I20260709 13:03:59.975194 11149 ts_tablet_manager.cc:1403] T 75492f86fde046c3900df3eac8292fd4 P 873357742d5843579556ee8166f23f47: Time spent bootstrapping tablet: real 0.878s	user 0.022s	sys 0.019s
W20260709 13:03:59.791278 11669 log.cc:927] Time spent T 6a9353d1d230416296627d7cf8b1d425 P 186b12638a87410487467df721a3cbda: Append to log took a long time: real 0.053s	user 0.000s	sys 0.001s
I20260709 13:03:59.972568 10844 leader_election.cc:304] T 8c160ddfe9b34bdaa3835d0915c1e06f P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 186b12638a87410487467df721a3cbda; no voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47
W20260709 13:03:59.976137 11066 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 560759us
I20260709 13:03:59.807186 11580 raft_consensus.cc:493] T d371c5e27bd843469d2e2e453734cb4a P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:59.985430 11066 leader_election.cc:304] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda
I20260709 13:03:59.794832 10872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11de93cd9b9d457489ff1cb80f976c9a" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:03:59.805130 11653 raft_consensus.cc:493] T 34064dc7bf3747dbb3876dd824ee0688 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:59.796113 11026 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c2a35bb99374e93b573cf870d04faaa" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:03:59.893137 11518 raft_consensus.cc:493] T 11de93cd9b9d457489ff1cb80f976c9a P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:03:59.806643 11022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a1716c0adf34bee8a4e99341ba5788f" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:03:59.793394 11427 raft_consensus.cc:493] T bac7e5040f984ba2b7146bd36af7267e P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:59.890259 11096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87cd8f5097624c8692f1d83544d8881e" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:03:59.806257 11513 raft_consensus.cc:493] T c51782cd9d064ec0949e84e24dd4de67 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:59.985250 11102 raft_consensus.cc:1275] T 6a9353d1d230416296627d7cf8b1d425 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Refusing update from remote peer 186b12638a87410487467df721a3cbda: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:03:59.811553 10952 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87cd8f5097624c8692f1d83544d8881e" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:03:59.810473 10953 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e96e80ba9ac34d0998395515e5c8c41a" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:03:59.870316 10877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a1716c0adf34bee8a4e99341ba5788f" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:03:59.878346 10919 leader_election.cc:304] T 65bf0a62f4d84ef18dd6b23e082724b8 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:03:59.798853 11545 leader_election.cc:290] T 87cd8f5097624c8692f1d83544d8881e P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:03:59.802991 11610 raft_consensus.cc:493] T a610d1a71f52432e88fe56130f921fa2 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:59.802152 11021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65bf0a62f4d84ef18dd6b23e082724b8" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:03:59.801656 11020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e96e80ba9ac34d0998395515e5c8c41a" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:03:59.878099 11512 raft_consensus.cc:3055] T d270b9e5481245d9b5340a454a7cd10b P 0172fed22b6b4890a7fdc953fa128af4 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:03:59.990911 11066 leader_election.cc:304] T a154cc8b181f460abf4880064379948b P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47
I20260709 13:03:59.758668 11360 raft_consensus.cc:515] T e5a5f5a608504650b48fd27f163faed6 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:03:59.800037 11024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e5d327630e5438b878035d28efe6c0c" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:03:59.897591 11642 leader_election.cc:290] T 1e4f2241a94a41f38e481137f868b83d P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:03:59.870311 10946 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65bf0a62f4d84ef18dd6b23e082724b8" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:00.257587 10949 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e4f2241a94a41f38e481137f868b83d" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:00.258268 11024 raft_consensus.cc:2393] T 0e5d327630e5438b878035d28efe6c0c P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 186b12638a87410487467df721a3cbda in current term 2: Already voted for candidate 873357742d5843579556ee8166f23f47 in this term.
I20260709 13:03:59.889144 11628 raft_consensus.cc:493] T e5ba72ccd854430baadbc636dfd17136 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:59.997807 11029 raft_consensus.cc:1240] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 873357742d5843579556ee8166f23f47 [term 2 LEADER]: Rejecting Update request from peer 0172fed22b6b4890a7fdc953fa128af4 for earlier term 1. Current term is 2. Ops: []
I20260709 13:03:59.996387 10844 leader_election.cc:304] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 186b12638a87410487467df721a3cbda; no voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47
I20260709 13:03:59.784292 11664 leader_election.cc:290] T a64b7e99d6c54934a7de6f6dffcf5e94 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:03:59.884008 11658 raft_consensus.cc:493] T f371d2565dd84667b5613848604949bb P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:03:59.876650 11094 raft_consensus.cc:3055] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:03:59.905337 11099 raft_consensus.cc:2468] T 7709387f4d3d4f9992d97a0fbca39a40 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 2.
W20260709 13:03:59.891849 11668 log.cc:927] Time spent T e5ba72ccd854430baadbc636dfd17136 P 873357742d5843579556ee8166f23f47: Append to log took a long time: real 0.153s	user 0.000s	sys 0.001s
I20260709 13:03:59.905066 11590 leader_election.cc:290] T 0e5d327630e5438b878035d28efe6c0c P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:03:59.995280 10774 catalog_manager.cc:5697] T bb7d67be500947b3aef4c4245e33671c P 10ca5ee68ad64b3283b9d235352c9d23 reported cstate change: term changed from 0 to 1, leader changed from <none> to 873357742d5843579556ee8166f23f47 (127.0.107.195). New cstate: current_term: 1 leader_uuid: "873357742d5843579556ee8166f23f47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } }
I20260709 13:03:59.896209 10874 raft_consensus.cc:2468] T 1e4f2241a94a41f38e481137f868b83d P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 0.
I20260709 13:03:59.875248 10876 raft_consensus.cc:1275] T 72b111ad7b5f4753ab1d2b5ee1114840 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Refusing update from remote peer 0172fed22b6b4890a7fdc953fa128af4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:03:59.787791 11592 raft_consensus.cc:493] T 7709387f4d3d4f9992d97a0fbca39a40 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:00.017366 11151 tablet_bootstrap.cc:492] T 401a9cf11eac479aa445ec8c1d6e33b2 P 186b12638a87410487467df721a3cbda: Bootstrap starting.
I20260709 13:04:00.295570 11592 raft_consensus.cc:3060] T 7709387f4d3d4f9992d97a0fbca39a40 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:03:59.984380 11025 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5299b8d044f41b7b68057e63d17b159" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:03:59.963964 11508 consensus_queue.cc:1048] T 63d44f6ba6434ab2870fe3190f4d7c2f P 186b12638a87410487467df721a3cbda [LEADER]: Connected to new peer: Peer: permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.047s
I20260709 13:03:59.792385 11639 raft_consensus.cc:515] T e89e640169e14e2d98f968f76027daa7 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:00.301365 10948 raft_consensus.cc:1240] T e5a5f5a608504650b48fd27f163faed6 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Rejecting Update request from peer 186b12638a87410487467df721a3cbda for earlier term 1. Current term is 2. Ops: []
I20260709 13:04:00.277382 11094 raft_consensus.cc:740] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0172fed22b6b4890a7fdc953fa128af4, State: Running, Role: LEADER
I20260709 13:04:00.302639 11639 leader_election.cc:290] T e89e640169e14e2d98f968f76027daa7 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 3 election: Requested vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:03:59.801055 11027 raft_consensus.cc:1275] T 4f9eb06ed01f4ce68be3af8a8434665b P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Refusing update from remote peer 186b12638a87410487467df721a3cbda: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:04:00.303828 11094 consensus_queue.cc:260] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 0172fed22b6b4890a7fdc953fa128af4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
W20260709 13:04:00.029516 10826 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 13:04:00.008091 11607 raft_consensus.cc:2749] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 13:03:59.807847 10843 leader_election.cc:304] T c31881534c5e4f0089c82057c1746652 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda; no voters: 0172fed22b6b4890a7fdc953fa128af4
I20260709 13:04:00.040608 11102 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7709387f4d3d4f9992d97a0fbca39a40" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:00.305807 11102 raft_consensus.cc:2468] T 7709387f4d3d4f9992d97a0fbca39a40 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 2.
I20260709 13:04:00.249442 11575 raft_consensus.cc:2749] T a154cc8b181f460abf4880064379948b P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 13:03:59.831923 11595 raft_consensus.cc:515] T 34064dc7bf3747dbb3876dd824ee0688 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:00.013072 11606 raft_consensus.cc:493] T d371c5e27bd843469d2e2e453734cb4a P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:00.316944 11606 raft_consensus.cc:515] T d371c5e27bd843469d2e2e453734cb4a P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
W20260709 13:04:00.317163 10843 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 509429us
I20260709 13:04:00.319582 11606 leader_election.cc:290] T d371c5e27bd843469d2e2e453734cb4a P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:00.116865 11597 raft_consensus.cc:2804] T 65bf0a62f4d84ef18dd6b23e082724b8 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:04:00.320415 11597 raft_consensus.cc:493] T 65bf0a62f4d84ef18dd6b23e082724b8 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:00.312902 11024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a64b7e99d6c54934a7de6f6dffcf5e94" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:00.320680 11508 raft_consensus.cc:2804] T c31881534c5e4f0089c82057c1746652 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:04:00.011770 11653 raft_consensus.cc:515] T 34064dc7bf3747dbb3876dd824ee0688 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:00.323114 11595 leader_election.cc:290] T 34064dc7bf3747dbb3876dd824ee0688 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:00.038005 10952 raft_consensus.cc:2468] T 87cd8f5097624c8692f1d83544d8881e P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 2.
I20260709 13:04:00.324646 10952 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e4f2241a94a41f38e481137f868b83d" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:00.325723 11653 leader_election.cc:290] T 34064dc7bf3747dbb3876dd824ee0688 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:00.015622 11427 raft_consensus.cc:515] T bac7e5040f984ba2b7146bd36af7267e P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:00.327440 11095 raft_consensus.cc:1275] T bac7e5040f984ba2b7146bd36af7267e P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Refusing update from remote peer 873357742d5843579556ee8166f23f47: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:04:00.022785 11513 raft_consensus.cc:515] T c51782cd9d064ec0949e84e24dd4de67 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:00.016717 11518 raft_consensus.cc:515] T 11de93cd9b9d457489ff1cb80f976c9a P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:03:59.996076 11580 raft_consensus.cc:515] T d371c5e27bd843469d2e2e453734cb4a P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:00.158005 11680 raft_consensus.cc:493] T c31881534c5e4f0089c82057c1746652 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:00.084810 11610 raft_consensus.cc:515] T a610d1a71f52432e88fe56130f921fa2 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
W20260709 13:04:00.085738 10919 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:45807, user_credentials={real_user=slave}} blocked reactor thread for 207489us
I20260709 13:04:00.100508 11098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e5d327630e5438b878035d28efe6c0c" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:00.358620 11580 leader_election.cc:290] T d371c5e27bd843469d2e2e453734cb4a P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
W20260709 13:04:00.147450 11665 log.cc:927] Time spent T 2567c2d84e4648d7ba5dc23a3df32817 P 873357742d5843579556ee8166f23f47: Append to log took a long time: real 0.103s	user 0.000s	sys 0.000s
I20260709 13:04:00.358980 11097 raft_consensus.cc:1275] T c51782cd9d064ec0949e84e24dd4de67 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Refusing update from remote peer 186b12638a87410487467df721a3cbda: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:04:00.228099 10955 raft_consensus.cc:1275] T a64b7e99d6c54934a7de6f6dffcf5e94 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Refusing update from remote peer 873357742d5843579556ee8166f23f47: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:04:00.041664 10953 raft_consensus.cc:2468] T e96e80ba9ac34d0998395515e5c8c41a P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 2.
I20260709 13:04:00.027110 11096 raft_consensus.cc:2468] T 87cd8f5097624c8692f1d83544d8881e P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 2.
I20260709 13:04:00.117399 11100 raft_consensus.cc:1275] T 8a69f32c53f24a50b7852d143665d7b4 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Refusing update from remote peer 186b12638a87410487467df721a3cbda: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:04:00.371198 10955 raft_consensus.cc:1275] T e5ba72ccd854430baadbc636dfd17136 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Refusing update from remote peer 873357742d5843579556ee8166f23f47: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:04:00.372854 10955 raft_consensus.cc:1240] T e89e640169e14e2d98f968f76027daa7 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Rejecting Update request from peer 873357742d5843579556ee8166f23f47 for earlier term 2. Current term is 3. Ops: []
I20260709 13:04:00.044119 10877 raft_consensus.cc:2468] T 9a1716c0adf34bee8a4e99341ba5788f P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 1.
I20260709 13:04:00.374897 10994 leader_election.cc:304] T 8c160ddfe9b34bdaa3835d0915c1e06f P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47; no voters: 186b12638a87410487467df721a3cbda
I20260709 13:04:00.228646 11619 raft_consensus.cc:493] T c51782cd9d064ec0949e84e24dd4de67 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:00.043187 11026 raft_consensus.cc:2393] T 4c2a35bb99374e93b573cf870d04faaa P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 186b12638a87410487467df721a3cbda in current term 2: Already voted for candidate 873357742d5843579556ee8166f23f47 in this term.
W20260709 13:04:00.241202 11652 log.cc:927] Time spent T 732c4adeeff6422497aa7db4a82f535b P 186b12638a87410487467df721a3cbda: Append to log took a long time: real 0.859s	user 0.000s	sys 0.002s
I20260709 13:04:00.241320 11023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7709387f4d3d4f9992d97a0fbca39a40" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
W20260709 13:04:00.246526 11066 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 255725us
I20260709 13:04:00.336943 11664 raft_consensus.cc:493] T d7ce47b9a92d43dd88d95bcda5038597 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 10ca5ee68ad64b3283b9d235352c9d23)
I20260709 13:04:00.381446 11066 leader_election.cc:304] T 87cd8f5097624c8692f1d83544d8881e P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda
I20260709 13:04:00.257254 11642 raft_consensus.cc:493] T 4b37f8fca8dc42718b2a42f0883a956f P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 10ca5ee68ad64b3283b9d235352c9d23)
I20260709 13:04:00.336934 11020 raft_consensus.cc:2393] T e96e80ba9ac34d0998395515e5c8c41a P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 186b12638a87410487467df721a3cbda in current term 2: Already voted for candidate 873357742d5843579556ee8166f23f47 in this term.
W20260709 13:04:00.270222 10844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:45807, user_credentials={real_user=slave}} blocked reactor thread for 273960us
I20260709 13:04:00.388252 11066 leader_election.cc:304] T 32147c6d2d5a477aa38c597c7f78df5b P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47
I20260709 13:04:00.287418 10949 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11de93cd9b9d457489ff1cb80f976c9a" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:03:59.798413 11555 raft_consensus.cc:515] T 0e5d327630e5438b878035d28efe6c0c P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:00.011601 11589 raft_consensus.cc:2749] T 8c160ddfe9b34bdaa3835d0915c1e06f P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
W20260709 13:04:00.303475 11569 consensus_meta.cc:287] T 4c2a35bb99374e93b573cf870d04faaa P 186b12638a87410487467df721a3cbda: Time spent flushing consensus metadata: real 0.569s	user 0.013s	sys 0.004s
I20260709 13:04:00.383481 11553 raft_consensus.cc:493] T 8a69f32c53f24a50b7852d143665d7b4 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:00.398844 11569 raft_consensus.cc:515] T 4c2a35bb99374e93b573cf870d04faaa P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:00.399190 11553 raft_consensus.cc:515] T 8a69f32c53f24a50b7852d143665d7b4 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:00.401239 11569 leader_election.cc:290] T 4c2a35bb99374e93b573cf870d04faaa P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 2 election: Requested vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:00.402160 11654 raft_consensus.cc:2804] T 8c160ddfe9b34bdaa3835d0915c1e06f P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:04:00.387099 11575 raft_consensus.cc:2749] T 87cd8f5097624c8692f1d83544d8881e P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 13:04:00.402786 11654 raft_consensus.cc:697] T 8c160ddfe9b34bdaa3835d0915c1e06f P 873357742d5843579556ee8166f23f47 [term 1 LEADER]: Becoming Leader. State: Replica: 873357742d5843579556ee8166f23f47, State: Running, Role: LEADER
I20260709 13:04:00.403800 11654 consensus_queue.cc:237] T 8c160ddfe9b34bdaa3835d0915c1e06f P 873357742d5843579556ee8166f23f47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:00.404338 11020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d371c5e27bd843469d2e2e453734cb4a" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
W20260709 13:04:00.250309 10845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 206661us
I20260709 13:04:00.320847 11597 raft_consensus.cc:3060] T 65bf0a62f4d84ef18dd6b23e082724b8 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:04:00.123178 11021 raft_consensus.cc:2393] T 65bf0a62f4d84ef18dd6b23e082724b8 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 186b12638a87410487467df721a3cbda in current term 2: Already voted for candidate 873357742d5843579556ee8166f23f47 in this term.
I20260709 13:04:00.260020 11028 raft_consensus.cc:2468] T ff13ae9a47c7455f962406c6d131a3ff P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 1.
I20260709 13:04:00.277122 11545 raft_consensus.cc:2749] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 13:04:00.252449 10992 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 126574us
I20260709 13:04:00.358168 11513 leader_election.cc:290] T c51782cd9d064ec0949e84e24dd4de67 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:00.016305 11608 raft_consensus.cc:493] T a154cc8b181f460abf4880064379948b P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:00.326303 10952 raft_consensus.cc:3060] T bac7e5040f984ba2b7146bd36af7267e P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:04:00.375283 11653 consensus_queue.cc:1048] T 6f46a5102fd64b45bf1c4dff863d2b4d P 873357742d5843579556ee8166f23f47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:04:00.358668 11610 leader_election.cc:290] T a610d1a71f52432e88fe56130f921fa2 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:00.408310 11591 raft_consensus.cc:493] T b4cd3d7dfbbf4096908ff11e9dc57141 P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0172fed22b6b4890a7fdc953fa128af4)
I20260709 13:04:00.277478 11628 raft_consensus.cc:515] T e5ba72ccd854430baadbc636dfd17136 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:00.328348 11427 leader_election.cc:290] T bac7e5040f984ba2b7146bd36af7267e P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:00.245750 11512 raft_consensus.cc:740] T d270b9e5481245d9b5340a454a7cd10b P 0172fed22b6b4890a7fdc953fa128af4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0172fed22b6b4890a7fdc953fa128af4, State: Running, Role: LEADER
I20260709 13:04:00.361195 11094 raft_consensus.cc:3060] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:04:00.283375 11658 raft_consensus.cc:515] T f371d2565dd84667b5613848604949bb P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:00.395609 11151 tablet_bootstrap.cc:654] T 401a9cf11eac479aa445ec8c1d6e33b2 P 186b12638a87410487467df721a3cbda: Neither blocks nor log segments found. Creating new log.
I20260709 13:04:00.388411 11149 raft_consensus.cc:359] T 75492f86fde046c3900df3eac8292fd4 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:00.385296 11067 leader_election.cc:304] T acc241e7273d49149984968596242712 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4; no voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47
I20260709 13:04:00.367851 11620 raft_consensus.cc:493] T e96e80ba9ac34d0998395515e5c8c41a P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:00.371198 11096 raft_consensus.cc:1275] T a64b7e99d6c54934a7de6f6dffcf5e94 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Refusing update from remote peer 873357742d5843579556ee8166f23f47: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:04:00.415557 11425 raft_consensus.cc:2749] T 32147c6d2d5a477aa38c597c7f78df5b P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:04:00.402293 11553 leader_election.cc:290] T 8a69f32c53f24a50b7852d143665d7b4 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:00.427742 10875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5299b8d044f41b7b68057e63d17b159" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:00.428253 11639 raft_consensus.cc:493] T a64b7e99d6c54934a7de6f6dffcf5e94 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:00.428557 10879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a69f32c53f24a50b7852d143665d7b4" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:00.428808 10877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c31881534c5e4f0089c82057c1746652" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:00.434582 11672 raft_consensus.cc:493] T 8a69f32c53f24a50b7852d143665d7b4 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:00.435316 11098 raft_consensus.cc:2368] T 0e5d327630e5438b878035d28efe6c0c P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 186b12638a87410487467df721a3cbda for earlier term 2. Current term is 3.
I20260709 13:04:00.435642 10878 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51718 (request call id 132) took 1433 ms. Trace:
I20260709 13:04:00.428558 10844 leader_election.cc:304] T 9a1716c0adf34bee8a4e99341ba5788f P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 186b12638a87410487467df721a3cbda; no voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47
I20260709 13:04:00.449728 11152 tablet_bootstrap.cc:654] T ff13ae9a47c7455f962406c6d131a3ff P 0172fed22b6b4890a7fdc953fa128af4: Neither blocks nor log segments found. Creating new log.
W20260709 13:04:00.239087 11135 outbound_call.cc:321] RPC callback for RPC call kudu.master.MasterService.IsCreateTableDone -> {remote=127.0.107.254:40861, user_credentials={real_user=slave}} blocked reactor thread for 119598us
I20260709 13:04:00.456365 11024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e4f2241a94a41f38e481137f868b83d" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:00.490846 11024 raft_consensus.cc:2468] T 1e4f2241a94a41f38e481137f868b83d P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 0.
I20260709 13:04:00.491520 10879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e89e640169e14e2d98f968f76027daa7" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:04:00.412757 10949 raft_consensus.cc:2468] T 11de93cd9b9d457489ff1cb80f976c9a P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 2.
I20260709 13:04:00.496593 11148 tablet_bootstrap.cc:492] T a977842ba4b744f4bb2febd55d286006 P 10ca5ee68ad64b3283b9d235352c9d23: No bootstrap required, opened a new log
I20260709 13:04:00.497251 11148 ts_tablet_manager.cc:1403] T a977842ba4b744f4bb2febd55d286006 P 10ca5ee68ad64b3283b9d235352c9d23: Time spent bootstrapping tablet: real 0.793s	user 0.041s	sys 0.075s
I20260709 13:04:00.496963 11512 consensus_queue.cc:260] T d270b9e5481245d9b5340a454a7cd10b P 0172fed22b6b4890a7fdc953fa128af4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:00.474923 11149 raft_consensus.cc:385] T 75492f86fde046c3900df3eac8292fd4 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:04:00.497439 10879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34064dc7bf3747dbb3876dd824ee0688" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:00.497944 11149 raft_consensus.cc:740] T 75492f86fde046c3900df3eac8292fd4 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 873357742d5843579556ee8166f23f47, State: Initialized, Role: FOLLOWER
I20260709 13:04:00.369105 11580 raft_consensus.cc:493] T a5299b8d044f41b7b68057e63d17b159 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:00.498240 10879 raft_consensus.cc:2468] T 34064dc7bf3747dbb3876dd824ee0688 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 2.
I20260709 13:04:00.498482 11512 raft_consensus.cc:3060] T d270b9e5481245d9b5340a454a7cd10b P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:04:00.498873 11580 raft_consensus.cc:515] T a5299b8d044f41b7b68057e63d17b159 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:00.499974 10875 raft_consensus.cc:2468] T a5299b8d044f41b7b68057e63d17b159 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 0.
I20260709 13:04:00.501302 11148 raft_consensus.cc:359] T a977842ba4b744f4bb2febd55d286006 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:00.502188 11148 raft_consensus.cc:385] T a977842ba4b744f4bb2febd55d286006 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:04:00.499604 10879 raft_consensus.cc:3060] T 9a1716c0adf34bee8a4e99341ba5788f P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:04:00.502549 11148 raft_consensus.cc:740] T a977842ba4b744f4bb2febd55d286006 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10ca5ee68ad64b3283b9d235352c9d23, State: Initialized, Role: FOLLOWER
W20260709 13:04:00.474750 11066 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 84881.8us
I20260709 13:04:00.413311 10872 raft_consensus.cc:2468] T 11de93cd9b9d457489ff1cb80f976c9a P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 1.
I20260709 13:04:00.422890 11583 raft_consensus.cc:493] T 63d44f6ba6434ab2870fe3190f4d7c2f P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:00.503360 11148 consensus_queue.cc:260] T a977842ba4b744f4bb2febd55d286006 P 10ca5ee68ad64b3283b9d235352c9d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:00.503677 11583 raft_consensus.cc:515] T 63d44f6ba6434ab2870fe3190f4d7c2f P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:00.486860 10878 rpcz_store.cc:276] 0709 13:03:59.021310 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:59.026567 (+  5257us) service_pool.cc:224] Handling call
0709 13:03:59.036482 (+  9915us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:59.039971 (+  3489us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:59.040002 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:59.057551 (+ 17549us) log.cc:844] Serialized 26 byte log entry
0709 13:03:59.102040 (+ 44489us) spinlock_profiling.cc:243] Waited 5.59 ms on lock 0x7b180006b740. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f4263057e07 00007f426beb2ab9 00007f426307469b 00007f426307293f 00007f4263074c02 00007f4265cfc20b 00007f4265d01d23 00007f4265d020d1 00007f426611da0f
0709 13:03:59.189053 (+ 87013us) raft_consensus.cc:1645] Marking committed up to 1
0709 13:03:59.200525 (+ 11472us) log.cc:844] Serialized 14 byte log entry
0709 13:03:59.416645 (+216120us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:59.416741 (+    96us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:00.240367 (+823626us) spinlock_profiling.cc:243] Waited 12 ms on lock 0x7b4400022b4c. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f4262f9bc85 00007f4262f9c6a2 00007f4265d0e1db 00007f4265d09aaf 00007f4265cfd86e 00007f4265cfcbb6 00007f4265cfc7d8 00007f4265d06221 00007f4265d061d9 00007f4265d06169
0709 13:04:00.435353 (+194986us) raft_consensus.cc:1685] finished
0709 13:04:00.435379 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:00.435535 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":20006,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":49,"raft.queue_time_us":124385,"raft.run_cpu_time_us":275,"raft.run_wall_time_us":11500,"spinlock_wait_cycles":40386048,"thread_start_us":302514,"threads_started":2,"wal-append.queue_time_us":211590}
I20260709 13:04:00.505700 11148 ts_tablet_manager.cc:1434] T a977842ba4b744f4bb2febd55d286006 P 10ca5ee68ad64b3283b9d235352c9d23: Time spent starting tablet: real 0.008s	user 0.008s	sys 0.000s
I20260709 13:04:00.505956 11583 leader_election.cc:290] T 63d44f6ba6434ab2870fe3190f4d7c2f P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:00.384161 11619 raft_consensus.cc:515] T c51782cd9d064ec0949e84e24dd4de67 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:00.408603 11504 consensus_queue.cc:1059] T b4cd3d7dfbbf4096908ff11e9dc57141 P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.041s
I20260709 13:04:00.378757 11680 raft_consensus.cc:515] T c31881534c5e4f0089c82057c1746652 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:00.394102 11664 raft_consensus.cc:515] T d7ce47b9a92d43dd88d95bcda5038597 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:00.439438 11606 raft_consensus.cc:493] T 4c2a35bb99374e93b573cf870d04faaa P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:00.393862 11642 raft_consensus.cc:515] T 4b37f8fca8dc42718b2a42f0883a956f P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:00.441185 10880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65bf0a62f4d84ef18dd6b23e082724b8" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:00.479260 11545 consensus_queue.cc:1048] T 6a9353d1d230416296627d7cf8b1d425 P 186b12638a87410487467df721a3cbda [LEADER]: Connected to new peer: Peer: permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.036s
I20260709 13:04:00.405366 11020 raft_consensus.cc:2468] T d371c5e27bd843469d2e2e453734cb4a P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 1.
I20260709 13:04:00.473989 11585 raft_consensus.cc:493] T 4f9eb06ed01f4ce68be3af8a8434665b P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:00.495455 10844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:45807, user_credentials={real_user=slave}} blocked reactor thread for 67026.3us
I20260709 13:04:00.502279 11149 consensus_queue.cc:260] T 75492f86fde046c3900df3eac8292fd4 P 873357742d5843579556ee8166f23f47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:00.440163 11589 consensus_queue.cc:1048] T 8a69f32c53f24a50b7852d143665d7b4 P 186b12638a87410487467df721a3cbda [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.046s
W20260709 13:04:00.496526 11676 log.cc:927] Time spent T 525bfdbd6d934afd823f7d3ee3f9d052 P 10ca5ee68ad64b3283b9d235352c9d23: Append to log took a long time: real 0.130s	user 0.000s	sys 0.001s
I20260709 13:04:00.487094 11639 raft_consensus.cc:515] T a64b7e99d6c54934a7de6f6dffcf5e94 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:00.525297 11100 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff13ae9a47c7455f962406c6d131a3ff" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:04:00.525804 11664 leader_election.cc:290] T d7ce47b9a92d43dd88d95bcda5038597 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:00.451784 11608 raft_consensus.cc:515] T a154cc8b181f460abf4880064379948b P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:00.570397 11149 ts_tablet_manager.cc:1434] T 75492f86fde046c3900df3eac8292fd4 P 873357742d5843579556ee8166f23f47: Time spent starting tablet: real 0.594s	user 0.015s	sys 0.002s
I20260709 13:04:00.489339 10992 leader_election.cc:304] T 4b37f8fca8dc42718b2a42f0883a956f P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 873357742d5843579556ee8166f23f47; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda
I20260709 13:04:00.527988 11148 tablet_bootstrap.cc:492] T 1e4f2241a94a41f38e481137f868b83d P 10ca5ee68ad64b3283b9d235352c9d23: Bootstrap starting.
W20260709 13:04:00.487337 10843 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 128148us
W20260709 13:04:00.494395 10919 leader_election.cc:343] T a5299b8d044f41b7b68057e63d17b159 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 873357742d5843579556ee8166f23f47 (127.0.107.195:45807): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:04:00.436156 11510 raft_consensus.cc:493] T e96e80ba9ac34d0998395515e5c8c41a P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:00.534723 10955 raft_consensus.cc:1275] T c51782cd9d064ec0949e84e24dd4de67 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Refusing update from remote peer 186b12638a87410487467df721a3cbda: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:04:00.525897 11096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75492f86fde046c3900df3eac8292fd4" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:00.489120 11672 raft_consensus.cc:515] T 8a69f32c53f24a50b7852d143665d7b4 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:00.538853 10921 leader_election.cc:304] T 34064dc7bf3747dbb3876dd824ee0688 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda; no voters: 
I20260709 13:04:00.541124 10878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5a5f5a608504650b48fd27f163faed6" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:04:00.515693 11425 raft_consensus.cc:493] T 5de5f0581e3b41dea729363d6ca563bb P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:00.540573 10880 raft_consensus.cc:2468] T 65bf0a62f4d84ef18dd6b23e082724b8 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 1.
I20260709 13:04:00.516687 11360 leader_election.cc:290] T e5a5f5a608504650b48fd27f163faed6 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 2 election: Requested vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:00.523480 11580 leader_election.cc:290] T a5299b8d044f41b7b68057e63d17b159 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
W20260709 13:04:00.514209 11067 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:45807, user_credentials={real_user=slave}} blocked reactor thread for 146057us
I20260709 13:04:00.488229 11610 raft_consensus.cc:2749] T acc241e7273d49149984968596242712 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:04:00.528342 11508 raft_consensus.cc:697] T c31881534c5e4f0089c82057c1746652 P 186b12638a87410487467df721a3cbda [term 1 LEADER]: Becoming Leader. State: Replica: 186b12638a87410487467df721a3cbda, State: Running, Role: LEADER
I20260709 13:04:00.599371 10920 leader_election.cc:304] T bac7e5040f984ba2b7146bd36af7267e P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23; no voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47
W20260709 13:04:00.578631 10995 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:38967, user_credentials={real_user=slave}} blocked reactor thread for 66956.6us
I20260709 13:04:00.584340 10949 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f1466124dbd4f6ebb3c4d62ca48bf96" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:04:00.607767 10920 leader_election.cc:304] T b4cd3d7dfbbf4096908ff11e9dc57141 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23; no voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47
I20260709 13:04:00.543510 11585 raft_consensus.cc:515] T 4f9eb06ed01f4ce68be3af8a8434665b P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
W20260709 13:04:00.571031 11149 ts_tablet_manager.cc:1467] T 75492f86fde046c3900df3eac8292fd4 P 873357742d5843579556ee8166f23f47: Tablet startup took 595ms
I20260709 13:04:00.541604 11103 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c31881534c5e4f0089c82057c1746652" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
W20260709 13:04:00.588783 10921 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:38967, user_credentials={real_user=slave}} blocked reactor thread for 50022.4us
I20260709 13:04:00.482993 11591 raft_consensus.cc:515] T b4cd3d7dfbbf4096908ff11e9dc57141 P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:00.530545 11606 raft_consensus.cc:515] T 4c2a35bb99374e93b573cf870d04faaa P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:00.604557 11024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c2a35bb99374e93b573cf870d04faaa" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
I20260709 13:04:00.592535 11553 raft_consensus.cc:2804] T 34064dc7bf3747dbb3876dd824ee0688 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:04:00.604126 11508 consensus_queue.cc:237] T c31881534c5e4f0089c82057c1746652 P 186b12638a87410487467df721a3cbda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
W20260709 13:04:00.600867 10992 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 118059us
I20260709 13:04:00.594586 11425 raft_consensus.cc:515] T 5de5f0581e3b41dea729363d6ca563bb P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:00.606501 11685 raft_consensus.cc:493] T b67eff0634e540eab91cf8cee795969f P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:00.609058 10920 leader_election.cc:343] T ff13ae9a47c7455f962406c6d131a3ff P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:04:00.481495 11620 raft_consensus.cc:515] T e96e80ba9ac34d0998395515e5c8c41a P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:00.593150 10953 raft_consensus.cc:1275] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Refusing update from remote peer 873357742d5843579556ee8166f23f47: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260709 13:04:00.587963 10919 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:45807, user_credentials={real_user=slave}} blocked reactor thread for 98336.6us
I20260709 13:04:00.595315 10951 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d71ca3a1ba6b48478aaf62741bf3cf6a" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
W20260709 13:04:00.654294 10950 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49428 (request call id 61) took 1941 ms (client timeout 2075 ms). Trace:
W20260709 13:04:00.654830 10950 rpcz_store.cc:269] 0709 13:03:58.712849 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:58.713034 (+   185us) service_pool.cc:224] Handling call
0709 13:04:00.654066 (+1941032us) spinlock_profiling.cc:243] Waited 502 ms on lock 0x7b54008b6d54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:04:00.654198 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1154782592}
I20260709 13:04:00.595252 11633 raft_consensus.cc:2749] T 4b37f8fca8dc42718b2a42f0883a956f P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:04:00.597162 11608 leader_election.cc:290] T a154cc8b181f460abf4880064379948b P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:00.590385 11510 raft_consensus.cc:515] T e96e80ba9ac34d0998395515e5c8c41a P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
W20260709 13:04:00.609609 11679 log.cc:927] Time spent T 4f9eb06ed01f4ce68be3af8a8434665b P 0172fed22b6b4890a7fdc953fa128af4: Append to log took a long time: real 0.111s	user 0.000s	sys 0.001s
I20260709 13:04:00.657298 10951 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d7ce47b9a92d43dd88d95bcda5038597" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:00.659288 11518 leader_election.cc:290] T 11de93cd9b9d457489ff1cb80f976c9a P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:00.659392 10951 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a64b7e99d6c54934a7de6f6dffcf5e94" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:00.659600 11101 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5a5f5a608504650b48fd27f163faed6" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
W20260709 13:04:00.609992 11677 log.cc:927] Time spent T 555217d45d9b4ee492dbb3ddaf0f7c02 P 186b12638a87410487467df721a3cbda: Append to log took a long time: real 0.203s	user 0.000s	sys 0.001s
I20260709 13:04:00.622162 11553 raft_consensus.cc:493] T 34064dc7bf3747dbb3876dd824ee0688 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:00.631018 11685 raft_consensus.cc:515] T b67eff0634e540eab91cf8cee795969f P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:00.631358 10949 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c51782cd9d064ec0949e84e24dd4de67" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:00.611763 11360 raft_consensus.cc:2749] T b4cd3d7dfbbf4096908ff11e9dc57141 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 13:04:00.632663 11066 leader_election.cc:304] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47
I20260709 13:04:00.607108 11595 consensus_queue.cc:1048] T a154cc8b181f460abf4880064379948b P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.020s
I20260709 13:04:00.619716 11020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e89e640169e14e2d98f968f76027daa7" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
I20260709 13:04:00.635221 11425 leader_election.cc:290] T 5de5f0581e3b41dea729363d6ca563bb P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:00.635779 11024 raft_consensus.cc:2393] T 4c2a35bb99374e93b573cf870d04faaa P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 186b12638a87410487467df721a3cbda in current term 2: Already voted for candidate 873357742d5843579556ee8166f23f47 in this term.
I20260709 13:04:00.637667 10995 leader_election.cc:304] T 1e4f2241a94a41f38e481137f868b83d P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:04:00.673615 11360 raft_consensus.cc:493] T a977842ba4b744f4bb2febd55d286006 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:00.674088 10953 raft_consensus.cc:1275] T 72b111ad7b5f4753ab1d2b5ee1114840 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Refusing update from remote peer 0172fed22b6b4890a7fdc953fa128af4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:04:00.674216 11360 raft_consensus.cc:515] T a977842ba4b744f4bb2febd55d286006 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
W20260709 13:04:00.627279 11678 log.cc:927] Time spent T 63d44f6ba6434ab2870fe3190f4d7c2f P 10ca5ee68ad64b3283b9d235352c9d23: Append to log took a long time: real 0.204s	user 0.001s	sys 0.000s
I20260709 13:04:00.675107 11024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34064dc7bf3747dbb3876dd824ee0688" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:00.675868 11024 raft_consensus.cc:2468] T 34064dc7bf3747dbb3876dd824ee0688 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 2.
I20260709 13:04:00.677606 11575 raft_consensus.cc:493] T 6f46a5102fd64b45bf1c4dff863d2b4d P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:00.610141 11149 ts_tablet_manager.cc:1469] T 75492f86fde046c3900df3eac8292fd4 P 873357742d5843579556ee8166f23f47: Trace:
0709 13:03:48.321098 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:48.321398 (+   300us) service_pool.cc:224] Handling call
0709 13:03:48.326887 (+  5489us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 13:03:48.327002 (+   115us) ts_tablet_manager.cc:682] Creating new metadata...
0709 13:03:48.367060 (+ 40058us) tablet_metadata.cc:665] Metadata flushed
0709 13:03:48.487931 (+120871us) tablet_replica.cc:206] Creating consensus instance
0709 13:03:48.494200 (+  6269us) inbound_call.cc:177] Queueing success response
0709 13:03:59.097315 (+10603115us) ts_tablet_manager.cc:1379] Bootstrapping tablet
0709 13:03:59.975971 (+878656us) ts_tablet_manager.cc:1435] Starting tablet replica
0709 13:04:00.248117 (+272146us) tablet_replica.cc:261] Starting instrumentation
0709 13:04:00.262866 (+ 14749us) tablet_replica.cc:280] Starting consensus
Metrics: {"mutex_wait_us":1779,"raft.queue_time_us":3903,"raft.run_cpu_time_us":233,"raft.run_wall_time_us":27311,"spinlock_wait_cycles":149248,"tablet-open.queue_time_us":10603176}
I20260709 13:04:00.660503 11101 raft_consensus.cc:3060] T e5a5f5a608504650b48fd27f163faed6 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:04:00.681743 11098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "525bfdbd6d934afd823f7d3ee3f9d052" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:04:00.640046 11639 leader_election.cc:290] T a64b7e99d6c54934a7de6f6dffcf5e94 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:00.683154 11639 raft_consensus.cc:493] T 72b111ad7b5f4753ab1d2b5ee1114840 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:00.683643 11639 raft_consensus.cc:515] T 72b111ad7b5f4753ab1d2b5ee1114840 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
W20260709 13:04:00.643018 11684 log.cc:927] Time spent T 6f46a5102fd64b45bf1c4dff863d2b4d P 10ca5ee68ad64b3283b9d235352c9d23: Append to log took a long time: real 0.140s	user 0.001s	sys 0.000s
I20260709 13:04:00.691093 11680 leader_election.cc:290] T c31881534c5e4f0089c82057c1746652 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:00.693272 10954 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49434 (request call id 104) took 1321 ms. Trace:
I20260709 13:04:00.665942 11642 leader_election.cc:290] T 4b37f8fca8dc42718b2a42f0883a956f P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:00.694283 11099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a64b7e99d6c54934a7de6f6dffcf5e94" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:00.661903 11545 raft_consensus.cc:493] T a610d1a71f52432e88fe56130f921fa2 P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:00.695201 11099 raft_consensus.cc:2468] T a64b7e99d6c54934a7de6f6dffcf5e94 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 2.
I20260709 13:04:00.695434 11427 raft_consensus.cc:3055] T b4cd3d7dfbbf4096908ff11e9dc57141 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:04:00.695928 11427 raft_consensus.cc:740] T b4cd3d7dfbbf4096908ff11e9dc57141 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0172fed22b6b4890a7fdc953fa128af4, State: Running, Role: LEADER
I20260709 13:04:00.696805 11427 consensus_queue.cc:260] T b4cd3d7dfbbf4096908ff11e9dc57141 P 0172fed22b6b4890a7fdc953fa128af4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:00.697438 11672 leader_election.cc:290] T 8a69f32c53f24a50b7852d143665d7b4 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:00.697909 11427 raft_consensus.cc:3060] T b4cd3d7dfbbf4096908ff11e9dc57141 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:04:00.695467 11545 raft_consensus.cc:515] T a610d1a71f52432e88fe56130f921fa2 P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:00.699024 11680 raft_consensus.cc:2749] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 13:04:00.700874 11021 raft_consensus.cc:1275] T 525bfdbd6d934afd823f7d3ee3f9d052 P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Refusing update from remote peer 10ca5ee68ad64b3283b9d235352c9d23: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 13:04:00.678730 11592 raft_consensus.cc:515] T 7709387f4d3d4f9992d97a0fbca39a40 P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:00.596979 10843 leader_election.cc:304] T 87cd8f5097624c8692f1d83544d8881e P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda; no voters: 
I20260709 13:04:00.644166 11628 leader_election.cc:290] T e5ba72ccd854430baadbc636dfd17136 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:00.693714 10954 rpcz_store.cc:276] 0709 13:03:59.372035 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:59.376476 (+  4441us) service_pool.cc:224] Handling call
0709 13:04:00.012876 (+636400us) spinlock_profiling.cc:243] Waited 5.94 ms on lock 0x7b580001d198. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f426303db71 00007f426303da22 00007f426bf56c09 00007f426b0b0ee0 00007f426b17a0ab 00007f426b17a131 00007f426b13bfc0 00007f426b126929 00007f426b1263d2 00007f426469dbd4
0709 13:04:00.038238 (+ 25362us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:00.249244 (+211006us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:00.252715 (+  3471us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:00.257638 (+  4923us) log.cc:844] Serialized 26 byte log entry
0709 13:04:00.426680 (+169042us) raft_consensus.cc:1645] Marking committed up to 1
0709 13:04:00.426752 (+    72us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:00.450053 (+ 23301us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:00.692855 (+242802us) raft_consensus.cc:1685] finished
0709 13:04:00.692888 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:00.693178 (+   290us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":131525,"raft.run_cpu_time_us":198,"raft.run_wall_time_us":9170,"spinlock_wait_cycles":13672320,"thread_start_us":156702,"threads_started":1,"wal-append.queue_time_us":243238}
I20260709 13:04:00.703986 11592 leader_election.cc:290] T 7709387f4d3d4f9992d97a0fbca39a40 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 3 election: Requested vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:00.704488 11658 leader_election.cc:290] T f371d2565dd84667b5613848604949bb P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:00.705127 11658 raft_consensus.cc:2804] T 87cd8f5097624c8692f1d83544d8881e P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:04:00.627635 11100 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a69f32c53f24a50b7852d143665d7b4" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:00.649609 11148 tablet_bootstrap.cc:654] T 1e4f2241a94a41f38e481137f868b83d P 10ca5ee68ad64b3283b9d235352c9d23: Neither blocks nor log segments found. Creating new log.
I20260709 13:04:00.653919 11664 raft_consensus.cc:493] T 6a9353d1d230416296627d7cf8b1d425 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:00.656241 10950 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bac7e5040f984ba2b7146bd36af7267e" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:00.609550 11689 consensus_queue.cc:1059] T 836251ebc2b74f5da88b8a8b2338e2a5 P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.148s
I20260709 13:04:00.669289 11066 leader_election.cc:304] T d71ca3a1ba6b48478aaf62741bf3cf6a P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47
I20260709 13:04:00.660048 11518 raft_consensus.cc:493] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:00.672061 11619 leader_election.cc:290] T c51782cd9d064ec0949e84e24dd4de67 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:00.661984 11028 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5299b8d044f41b7b68057e63d17b159" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:00.672988 11685 leader_election.cc:290] T b67eff0634e540eab91cf8cee795969f P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:00.669477 11553 raft_consensus.cc:3060] T 34064dc7bf3747dbb3876dd824ee0688 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:04:00.617431 11555 leader_election.cc:290] T 0e5d327630e5438b878035d28efe6c0c P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 3 election: Requested vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:00.685864 11639 leader_election.cc:290] T 72b111ad7b5f4753ab1d2b5ee1114840 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:00.701083 11545 leader_election.cc:290] T a610d1a71f52432e88fe56130f921fa2 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:00.626791 11610 raft_consensus.cc:493] T 7757ecbf402e47bfbc93d0447741e26c P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 873357742d5843579556ee8166f23f47)
W20260709 13:04:00.720316 10995 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:38967, user_credentials={real_user=slave}} blocked reactor thread for 82758.9us
I20260709 13:04:00.771315 10995 leader_election.cc:304] T 11de93cd9b9d457489ff1cb80f976c9a P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:04:00.678275 11575 raft_consensus.cc:515] T 6f46a5102fd64b45bf1c4dff863d2b4d P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:00.740835 11152 tablet_bootstrap.cc:492] T ff13ae9a47c7455f962406c6d131a3ff P 0172fed22b6b4890a7fdc953fa128af4: No bootstrap required, opened a new log
I20260709 13:04:00.775806 11152 ts_tablet_manager.cc:1403] T ff13ae9a47c7455f962406c6d131a3ff P 0172fed22b6b4890a7fdc953fa128af4: Time spent bootstrapping tablet: real 0.992s	user 0.052s	sys 0.017s
W20260709 13:04:00.749461 11066 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 80318us
W20260709 13:04:00.776141 11069 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:38967, user_credentials={real_user=slave}} blocked reactor thread for 379779us
I20260709 13:04:00.628948 11103 raft_consensus.cc:2468] T c31881534c5e4f0089c82057c1746652 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 0.
I20260709 13:04:00.706907 11098 raft_consensus.cc:3060] T 525bfdbd6d934afd823f7d3ee3f9d052 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Advancing to term 3
W20260709 13:04:00.779160 10920 leader_election.cc:343] T 75492f86fde046c3900df3eac8292fd4 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:04:00.779733 10920 leader_election.cc:304] T 75492f86fde046c3900df3eac8292fd4 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23; no voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47
I20260709 13:04:00.784550 10920 leader_election.cc:304] T a64b7e99d6c54934a7de6f6dffcf5e94 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 10ca5ee68ad64b3283b9d235352c9d23; no voters: 
I20260709 13:04:00.660293 10951 raft_consensus.cc:2468] T a64b7e99d6c54934a7de6f6dffcf5e94 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 2.
W20260709 13:04:00.715752 10843 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 126012us
I20260709 13:04:00.786933 10955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5de5f0581e3b41dea729363d6ca563bb" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
W20260709 13:04:00.728284 10992 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 61850.9us
I20260709 13:04:00.764276 11023 raft_consensus.cc:2368] T 7709387f4d3d4f9992d97a0fbca39a40 P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 186b12638a87410487467df721a3cbda for earlier term 2. Current term is 3.
I20260709 13:04:00.793713 10955 raft_consensus.cc:2468] T 5de5f0581e3b41dea729363d6ca563bb P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 2.
I20260709 13:04:00.751222 11653 raft_consensus.cc:2804] T 1e4f2241a94a41f38e481137f868b83d P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:04:00.800048 11066 leader_election.cc:304] T 5de5f0581e3b41dea729363d6ca563bb P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 10ca5ee68ad64b3283b9d235352c9d23; no voters: 
I20260709 13:04:00.706064 11658 raft_consensus.cc:493] T 87cd8f5097624c8692f1d83544d8881e P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:00.797348 11069 leader_election.cc:304] T 9a1716c0adf34bee8a4e99341ba5788f P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda; no voters: 873357742d5843579556ee8166f23f47
I20260709 13:04:00.709280 11597 raft_consensus.cc:515] T 65bf0a62f4d84ef18dd6b23e082724b8 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:00.770270 11512 raft_consensus.cc:2749] T d71ca3a1ba6b48478aaf62741bf3cf6a P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:04:00.799008 11102 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c51782cd9d064ec0949e84e24dd4de67" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:00.804507 10920 leader_election.cc:304] T c31881534c5e4f0089c82057c1746652 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 10ca5ee68ad64b3283b9d235352c9d23; no voters: 
I20260709 13:04:00.791462 11619 raft_consensus.cc:2749] T 75492f86fde046c3900df3eac8292fd4 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:04:00.748154 11518 raft_consensus.cc:515] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:00.793123 11595 raft_consensus.cc:2804] T a64b7e99d6c54934a7de6f6dffcf5e94 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:04:00.745165 11100 raft_consensus.cc:2468] T 8a69f32c53f24a50b7852d143665d7b4 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 2.
I20260709 13:04:00.810189 10880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c51782cd9d064ec0949e84e24dd4de67" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:00.790825 11654 raft_consensus.cc:2804] T 11de93cd9b9d457489ff1cb80f976c9a P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:04:00.811066 10947 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a69f32c53f24a50b7852d143665d7b4" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:00.812804 10949 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c31881534c5e4f0089c82057c1746652" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:00.814718 11101 raft_consensus.cc:2468] T e5a5f5a608504650b48fd27f163faed6 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 2.
I20260709 13:04:00.816628 10879 raft_consensus.cc:1275] T 9a1716c0adf34bee8a4e99341ba5788f P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Refusing update from remote peer 873357742d5843579556ee8166f23f47: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:04:00.806134 11658 raft_consensus.cc:3060] T 87cd8f5097624c8692f1d83544d8881e P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:04:00.818337 11024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b37f8fca8dc42718b2a42f0883a956f" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:00.818576 11020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a610d1a71f52432e88fe56130f921fa2" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 4 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:00.819300 11639 raft_consensus.cc:2764] T c31881534c5e4f0089c82057c1746652 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260709 13:04:00.819533 10952 raft_consensus.cc:1275] T bac7e5040f984ba2b7146bd36af7267e P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Refusing update from remote peer 873357742d5843579556ee8166f23f47: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:04:00.819334 11022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bac7e5040f984ba2b7146bd36af7267e" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:00.819697 11510 leader_election.cc:290] T e96e80ba9ac34d0998395515e5c8c41a P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:00.820830 11021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f371d2565dd84667b5613848604949bb" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:00.824223 11152 raft_consensus.cc:359] T ff13ae9a47c7455f962406c6d131a3ff P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:00.924047 11152 raft_consensus.cc:385] T ff13ae9a47c7455f962406c6d131a3ff P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:04:00.924646 11152 raft_consensus.cc:740] T ff13ae9a47c7455f962406c6d131a3ff P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0172fed22b6b4890a7fdc953fa128af4, State: Initialized, Role: FOLLOWER
I20260709 13:04:00.826079 11606 leader_election.cc:290] T 4c2a35bb99374e93b573cf870d04faaa P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:00.828315 11680 consensus_queue.cc:1048] T d371c5e27bd843469d2e2e453734cb4a P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.039s
I20260709 13:04:00.832139 10876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d371c5e27bd843469d2e2e453734cb4a" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:00.837180 10872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63d44f6ba6434ab2870fe3190f4d7c2f" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:00.839656 10874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5de5f0581e3b41dea729363d6ca563bb" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:00.840317 11688 consensus_queue.cc:1048] T 4f9eb06ed01f4ce68be3af8a8434665b P 186b12638a87410487467df721a3cbda [LEADER]: Connected to new peer: Peer: permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:04:00.840596 10878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a610d1a71f52432e88fe56130f921fa2" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:00.841212 10871 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34064dc7bf3747dbb3876dd824ee0688" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:00.843173 11066 leader_election.cc:304] T a64b7e99d6c54934a7de6f6dffcf5e94 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 10ca5ee68ad64b3283b9d235352c9d23; no voters: 873357742d5843579556ee8166f23f47
I20260709 13:04:00.799739 11653 raft_consensus.cc:493] T 1e4f2241a94a41f38e481137f868b83d P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:00.749941 11664 raft_consensus.cc:515] T 6a9353d1d230416296627d7cf8b1d425 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
W20260709 13:04:00.852025 10921 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:38967, user_credentials={real_user=slave}} blocked reactor thread for 237931us
I20260709 13:04:00.792513 11689 consensus_queue.cc:1048] T 72b111ad7b5f4753ab1d2b5ee1114840 P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.035s
I20260709 13:04:00.804801 11628 raft_consensus.cc:2804] T 5de5f0581e3b41dea729363d6ca563bb P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:04:00.790553 11610 raft_consensus.cc:515] T 7757ecbf402e47bfbc93d0447741e26c P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
W20260709 13:04:00.805030 10843 leader_election.cc:343] T 1e4f2241a94a41f38e481137f868b83d P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:04:00.828639 11607 raft_consensus.cc:2804] T 9a1716c0adf34bee8a4e99341ba5788f P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:04:00.811265 11028 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11de93cd9b9d457489ff1cb80f976c9a" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:00.850419 11595 raft_consensus.cc:493] T a64b7e99d6c54934a7de6f6dffcf5e94 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:00.848801 11360 leader_election.cc:290] T a977842ba4b744f4bb2febd55d286006 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:00.849846 11096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72b111ad7b5f4753ab1d2b5ee1114840" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:00.849186 11585 leader_election.cc:290] T 4f9eb06ed01f4ce68be3af8a8434665b P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:00.991139 11024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e5d327630e5438b878035d28efe6c0c" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
I20260709 13:04:00.992115 11024 raft_consensus.cc:3060] T 0e5d327630e5438b878035d28efe6c0c P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:04:00.992419 11597 leader_election.cc:290] T 65bf0a62f4d84ef18dd6b23e082724b8 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 3 election: Requested vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:00.851255 10950 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5ba72ccd854430baadbc636dfd17136" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:00.974370 11555 consensus_queue.cc:1048] T 72b111ad7b5f4753ab1d2b5ee1114840 P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.024s
I20260709 13:04:00.995071 10950 raft_consensus.cc:2468] T e5ba72ccd854430baadbc636dfd17136 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 2.
I20260709 13:04:00.853412 11654 raft_consensus.cc:493] T 11de93cd9b9d457489ff1cb80f976c9a P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:01.001684 11654 raft_consensus.cc:3060] T 11de93cd9b9d457489ff1cb80f976c9a P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:04:01.003990 11610 leader_election.cc:290] T 7757ecbf402e47bfbc93d0447741e26c P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
W20260709 13:04:00.927434 11069 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:00.912824 10946 raft_consensus.cc:2368] T 65bf0a62f4d84ef18dd6b23e082724b8 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 186b12638a87410487467df721a3cbda for earlier term 2. Current term is 3.
I20260709 13:04:01.005110 10872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f9eb06ed01f4ce68be3af8a8434665b" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:00.940855 10879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e5d327630e5438b878035d28efe6c0c" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:04:01.006552 10872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7709387f4d3d4f9992d97a0fbca39a40" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:04:01.006632 11102 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f371d2565dd84667b5613848604949bb" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:01.007143 10872 raft_consensus.cc:3060] T 7709387f4d3d4f9992d97a0fbca39a40 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:04:01.007660 11101 raft_consensus.cc:1240] T e5a5f5a608504650b48fd27f163faed6 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Rejecting Update request from peer 186b12638a87410487467df721a3cbda for earlier term 1. Current term is 2. Ops: []
I20260709 13:04:01.007613 11102 raft_consensus.cc:2468] T f371d2565dd84667b5613848604949bb P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 1.
I20260709 13:04:01.011364 11103 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b67eff0634e540eab91cf8cee795969f" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:01.011847 11664 leader_election.cc:290] T 6a9353d1d230416296627d7cf8b1d425 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:00.944352 10880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d7ce47b9a92d43dd88d95bcda5038597" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:00.859463 10947 raft_consensus.cc:2468] T 8a69f32c53f24a50b7852d143665d7b4 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 2.
I20260709 13:04:00.946118 10949 raft_consensus.cc:2393] T c31881534c5e4f0089c82057c1746652 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0172fed22b6b4890a7fdc953fa128af4 in current term 1: Already voted for candidate 186b12638a87410487467df721a3cbda in this term.
I20260709 13:04:00.977458 11595 raft_consensus.cc:3060] T a64b7e99d6c54934a7de6f6dffcf5e94 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:04:01.013190 10880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e96e80ba9ac34d0998395515e5c8c41a" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:01.013902 10880 raft_consensus.cc:2468] T e96e80ba9ac34d0998395515e5c8c41a P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 1.
I20260709 13:04:00.951108 10878 raft_consensus.cc:2468] T a610d1a71f52432e88fe56130f921fa2 P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 3.
I20260709 13:04:00.962191 11628 raft_consensus.cc:493] T 5de5f0581e3b41dea729363d6ca563bb P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:00.860157 11506 consensus_queue.cc:1048] T acc241e7273d49149984968596242712 P 873357742d5843579556ee8166f23f47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.088s
I20260709 13:04:01.017551 11096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d371c5e27bd843469d2e2e453734cb4a" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:00.946215 11021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a610d1a71f52432e88fe56130f921fa2" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
W20260709 13:04:00.938459 10992 leader_election.cc:343] T 1e4f2241a94a41f38e481137f868b83d P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903): Illegal state: must be running to vote when last-logged opid is not known
W20260709 13:04:01.021039 10992 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 90452.9us
I20260709 13:04:01.021275 11021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5ba72ccd854430baadbc636dfd17136" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:00.931656 11529 raft_consensus.cc:2749] T bac7e5040f984ba2b7146bd36af7267e P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:04:00.926801 11687 raft_consensus.cc:2749] T 9a1716c0adf34bee8a4e99341ba5788f P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:04:00.947355 11639 raft_consensus.cc:493] T d71ca3a1ba6b48478aaf62741bf3cf6a P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:00.958696 11653 raft_consensus.cc:3060] T 1e4f2241a94a41f38e481137f868b83d P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:04:00.986817 11151 tablet_bootstrap.cc:492] T 401a9cf11eac479aa445ec8c1d6e33b2 P 186b12638a87410487467df721a3cbda: No bootstrap required, opened a new log
I20260709 13:04:00.952134 10876 raft_consensus.cc:2468] T d371c5e27bd843469d2e2e453734cb4a P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 1.
I20260709 13:04:00.963374 11672 raft_consensus.cc:2804] T a64b7e99d6c54934a7de6f6dffcf5e94 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:04:00.961814 10871 raft_consensus.cc:2468] T 34064dc7bf3747dbb3876dd824ee0688 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 2.
I20260709 13:04:01.041854 11639 raft_consensus.cc:515] T d71ca3a1ba6b48478aaf62741bf3cf6a P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:01.041922 11102 raft_consensus.cc:1275] T 32147c6d2d5a477aa38c597c7f78df5b P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Refusing update from remote peer 873357742d5843579556ee8166f23f47: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 1. (term mismatch)
I20260709 13:04:01.043133 11102 pending_rounds.cc:85] T 32147c6d2d5a477aa38c597c7f78df5b P 0172fed22b6b4890a7fdc953fa128af4: Aborting all ops after (but not including) 0
I20260709 13:04:01.043335 11102 pending_rounds.cc:107] T 32147c6d2d5a477aa38c597c7f78df5b P 0172fed22b6b4890a7fdc953fa128af4: Aborting uncommitted NO_OP operation due to leader change: 1.1
I20260709 13:04:01.043605 11102 raft_consensus.cc:2889] T 32147c6d2d5a477aa38c597c7f78df5b P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:04:01.043576 11705 consensus_queue.cc:1048] T f371d2565dd84667b5613848604949bb P 873357742d5843579556ee8166f23f47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:04:01.044549 11639 leader_election.cc:290] T d71ca3a1ba6b48478aaf62741bf3cf6a P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:01.045939 11608 consensus_queue.cc:1048] T bac7e5040f984ba2b7146bd36af7267e P 873357742d5843579556ee8166f23f47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:04:01.036738 11682 consensus_queue.cc:1048] T a64b7e99d6c54934a7de6f6dffcf5e94 P 873357742d5843579556ee8166f23f47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.010s
I20260709 13:04:01.047808 11705 consensus_queue.cc:1048] T a64b7e99d6c54934a7de6f6dffcf5e94 P 873357742d5843579556ee8166f23f47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:04:01.049140 11096 raft_consensus.cc:1275] T e5ba72ccd854430baadbc636dfd17136 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Refusing update from remote peer 873357742d5843579556ee8166f23f47: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:04:01.050277 11682 consensus_queue.cc:1048] T e5ba72ccd854430baadbc636dfd17136 P 873357742d5843579556ee8166f23f47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:04:01.051671 11682 consensus_queue.cc:1059] T e89e640169e14e2d98f968f76027daa7 P 873357742d5843579556ee8166f23f47 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:04:01.052062 11102 raft_consensus.cc:1240] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Rejecting Update request from peer 873357742d5843579556ee8166f23f47 for earlier term 2. Current term is 3. Ops: []
I20260709 13:04:01.052457 11656 consensus_queue.cc:1048] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 873357742d5843579556ee8166f23f47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:04:01.053184 11102 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a154cc8b181f460abf4880064379948b" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:01.053522 11682 consensus_queue.cc:1048] T bac7e5040f984ba2b7146bd36af7267e P 873357742d5843579556ee8166f23f47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:04:01.053881 11102 raft_consensus.cc:2468] T a154cc8b181f460abf4880064379948b P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 1.
I20260709 13:04:01.055013 11506 raft_consensus.cc:493] T 75492f86fde046c3900df3eac8292fd4 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:01.055555 11506 raft_consensus.cc:515] T 75492f86fde046c3900df3eac8292fd4 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:01.057999 11506 leader_election.cc:290] T 75492f86fde046c3900df3eac8292fd4 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:01.058840 10950 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34064dc7bf3747dbb3876dd824ee0688" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:01.058909 10953 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e96e80ba9ac34d0998395515e5c8c41a" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:01.059779 10953 raft_consensus.cc:2468] T e96e80ba9ac34d0998395515e5c8c41a P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 2.
I20260709 13:04:01.060084 10952 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63d44f6ba6434ab2870fe3190f4d7c2f" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
W20260709 13:04:01.060415 11094 consensus_meta.cc:287] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 0172fed22b6b4890a7fdc953fa128af4: Time spent flushing consensus metadata: real 0.583s	user 0.006s	sys 0.006s
I20260709 13:04:01.060650 11553 raft_consensus.cc:515] T 34064dc7bf3747dbb3876dd824ee0688 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:01.061144 11094 raft_consensus.cc:1275] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Refusing update from remote peer 873357742d5843579556ee8166f23f47: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:04:01.061924 11094 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37676 (request call id 115) took 1740 ms. Trace:
W20260709 13:04:01.062098 11625 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 2 current term: 2 is higher.
I20260709 13:04:01.062013 10955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a154cc8b181f460abf4880064379948b" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:01.062979 11021 raft_consensus.cc:1275] T 6a9353d1d230416296627d7cf8b1d425 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Refusing update from remote peer 186b12638a87410487467df721a3cbda: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:04:01.063747 10950 raft_consensus.cc:2393] T 34064dc7bf3747dbb3876dd824ee0688 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 873357742d5843579556ee8166f23f47 in current term 3: Already voted for candidate 10ca5ee68ad64b3283b9d235352c9d23 in this term.
I20260709 13:04:01.064396 10947 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c2a35bb99374e93b573cf870d04faaa" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:01.064952 11553 leader_election.cc:290] T 34064dc7bf3747dbb3876dd824ee0688 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 3 election: Requested vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:01.065115 10947 raft_consensus.cc:2468] T 4c2a35bb99374e93b573cf870d04faaa P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 2.
I20260709 13:04:01.068801 11148 tablet_bootstrap.cc:492] T 1e4f2241a94a41f38e481137f868b83d P 10ca5ee68ad64b3283b9d235352c9d23: No bootstrap required, opened a new log
I20260709 13:04:01.069635 11148 ts_tablet_manager.cc:1403] T 1e4f2241a94a41f38e481137f868b83d P 10ca5ee68ad64b3283b9d235352c9d23: Time spent bootstrapping tablet: real 0.542s	user 0.015s	sys 0.022s
I20260709 13:04:00.852079 10954 raft_consensus.cc:1240] T b4cd3d7dfbbf4096908ff11e9dc57141 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Rejecting Update request from peer 0172fed22b6b4890a7fdc953fa128af4 for earlier term 1. Current term is 2. Ops: []
W20260709 13:04:00.952987 11066 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 109910us
I20260709 13:04:01.071969 11610 raft_consensus.cc:493] T 5d439326f23747eaab3c1e7da2d6b785 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 10ca5ee68ad64b3283b9d235352c9d23)
I20260709 13:04:01.072149 11029 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46628 (request call id 114) took 1272 ms. Trace:
I20260709 13:04:01.072301 11096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d371c5e27bd843469d2e2e453734cb4a" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:01.006767 10879 raft_consensus.cc:3060] T 0e5d327630e5438b878035d28efe6c0c P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Advancing to term 3
I20260709 13:04:00.971730 11607 raft_consensus.cc:493] T 9a1716c0adf34bee8a4e99341ba5788f P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:00.960368 10874 raft_consensus.cc:2468] T 5de5f0581e3b41dea729363d6ca563bb P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 2.
W20260709 13:04:00.973791 10843 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 169704us
I20260709 13:04:00.932106 11020 raft_consensus.cc:2468] T a610d1a71f52432e88fe56130f921fa2 P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 3.
I20260709 13:04:01.102550 11682 consensus_queue.cc:1059] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 873357742d5843579556ee8166f23f47 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.003s
I20260709 13:04:01.029194 10878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c2a35bb99374e93b573cf870d04faaa" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:01.104185 10878 raft_consensus.cc:2468] T 4c2a35bb99374e93b573cf870d04faaa P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 2.
I20260709 13:04:01.104454 10843 leader_election.cc:304] T e96e80ba9ac34d0998395515e5c8c41a P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda; no voters: 873357742d5843579556ee8166f23f47
I20260709 13:04:01.033435 11151 ts_tablet_manager.cc:1403] T 401a9cf11eac479aa445ec8c1d6e33b2 P 186b12638a87410487467df721a3cbda: Time spent bootstrapping tablet: real 1.022s	user 0.026s	sys 0.003s
I20260709 13:04:01.105994 10843 leader_election.cc:304] T 5d439326f23747eaab3c1e7da2d6b785 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda; no voters: 0172fed22b6b4890a7fdc953fa128af4
I20260709 13:04:00.976867 11518 leader_election.cc:290] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:01.072477 11066 leader_election.cc:304] T e5ba72ccd854430baadbc636dfd17136 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 10ca5ee68ad64b3283b9d235352c9d23; no voters: 
I20260709 13:04:00.966531 11697 raft_consensus.cc:493] T 836251ebc2b74f5da88b8a8b2338e2a5 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:01.129927 11027 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a64b7e99d6c54934a7de6f6dffcf5e94" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:01.130710 11697 raft_consensus.cc:515] T 836251ebc2b74f5da88b8a8b2338e2a5 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:01.131217 11682 raft_consensus.cc:3055] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 873357742d5843579556ee8166f23f47 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:04:01.131922 11682 raft_consensus.cc:740] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 873357742d5843579556ee8166f23f47 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 873357742d5843579556ee8166f23f47, State: Running, Role: LEADER
I20260709 13:04:01.133260 11697 leader_election.cc:290] T 836251ebc2b74f5da88b8a8b2338e2a5 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:01.133098 11682 consensus_queue.cc:260] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 873357742d5843579556ee8166f23f47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:01.133728 11151 raft_consensus.cc:359] T 401a9cf11eac479aa445ec8c1d6e33b2 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:01.140059 11151 raft_consensus.cc:385] T 401a9cf11eac479aa445ec8c1d6e33b2 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:04:01.140443 11151 raft_consensus.cc:740] T 401a9cf11eac479aa445ec8c1d6e33b2 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 186b12638a87410487467df721a3cbda, State: Initialized, Role: FOLLOWER
I20260709 13:04:01.082239 11024 raft_consensus.cc:2468] T 0e5d327630e5438b878035d28efe6c0c P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 3.
I20260709 13:04:01.022696 11098 raft_consensus.cc:2468] T 525bfdbd6d934afd823f7d3ee3f9d052 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 3.
I20260709 13:04:01.085848 11702 consensus_queue.cc:1048] T 32147c6d2d5a477aa38c597c7f78df5b P 873357742d5843579556ee8166f23f47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:04:01.086104 10994 leader_election.cc:304] T a154cc8b181f460abf4880064379948b P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47; no voters: 
W20260709 13:04:01.086861 11068 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:01.039438 11672 raft_consensus.cc:493] T a64b7e99d6c54934a7de6f6dffcf5e94 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:01.072606 11029 rpcz_store.cc:276] 0709 13:03:59.799287 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:00.286661 (+487374us) service_pool.cc:224] Handling call
0709 13:04:00.650935 (+364274us) spinlock_profiling.cc:243] Waited 18.8 ms on lock 0x7b58007a2898. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f426303db71 00007f426303da22 00007f426bf56c09 00007f426b0b0ee0 00007f426b17a0ab 00007f426b17a131 00007f426b13bfc0 00007f426b126929 00007f426b1263d2 00007f426469dbd4
0709 13:04:00.703928 (+ 52993us) spinlock_profiling.cc:243] Waited 10.9 ms on lock 0x7b58007a2898. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f426303de9c 00007f426303da5b 00007f426bf56c09 00007f426b0b0ee0 00007f426b17a0ab 00007f426b17a131 00007f426b13bfc0 00007f426b126929 00007f426b1263d2 00007f426469dbd4
0709 13:04:00.704131 (+   203us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:00.704557 (+   426us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:00.704602 (+    45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:00.705430 (+   828us) log.cc:844] Serialized 26 byte log entry
0709 13:04:00.954031 (+248601us) raft_consensus.cc:1645] Marking committed up to 0
0709 13:04:00.954085 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:00.954174 (+    89us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:01.071915 (+117741us) raft_consensus.cc:1685] finished
0709 13:04:01.071944 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:01.072083 (+   139us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":359,"raft.run_cpu_time_us":101,"raft.run_wall_time_us":101,"spinlock_wait_cycles":68316288,"thread_start_us":241827,"threads_started":1,"wal-append.queue_time_us":328740}
I20260709 13:04:01.095427 11506 consensus_queue.cc:1048] T e5ba72ccd854430baadbc636dfd17136 P 873357742d5843579556ee8166f23f47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.007s
I20260709 13:04:00.947922 10845 leader_election.cc:304] T 7709387f4d3d4f9992d97a0fbca39a40 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda; no voters: 
I20260709 13:04:01.022446 11628 raft_consensus.cc:3060] T 5de5f0581e3b41dea729363d6ca563bb P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:04:01.106690 11688 raft_consensus.cc:2804] T e96e80ba9ac34d0998395515e5c8c41a P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:04:01.037766 11103 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a610d1a71f52432e88fe56130f921fa2" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 4 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
W20260709 13:04:01.135304 11066 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 62930.3us
I20260709 13:04:01.072605 11610 raft_consensus.cc:515] T 5d439326f23747eaab3c1e7da2d6b785 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:01.087234 11425 raft_consensus.cc:3055] T 836251ebc2b74f5da88b8a8b2338e2a5 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:04:01.138300 11427 raft_consensus.cc:2804] T e5ba72ccd854430baadbc636dfd17136 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:04:01.096678 11607 raft_consensus.cc:3060] T 9a1716c0adf34bee8a4e99341ba5788f P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:04:01.148023 11151 consensus_queue.cc:260] T 401a9cf11eac479aa445ec8c1d6e33b2 P 186b12638a87410487467df721a3cbda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:01.149520 11672 raft_consensus.cc:3060] T a64b7e99d6c54934a7de6f6dffcf5e94 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:04:01.155501 11066 leader_election.cc:304] T 8a69f32c53f24a50b7852d143665d7b4 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 10ca5ee68ad64b3283b9d235352c9d23; no voters: 
I20260709 13:04:01.145650 11705 raft_consensus.cc:493] T 6a9353d1d230416296627d7cf8b1d425 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:01.143440 11595 raft_consensus.cc:515] T a64b7e99d6c54934a7de6f6dffcf5e94 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:01.152941 11518 raft_consensus.cc:2804] T 5d439326f23747eaab3c1e7da2d6b785 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:04:01.174280 11595 leader_election.cc:290] T a64b7e99d6c54934a7de6f6dffcf5e94 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 3 election: Requested vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:01.160513 11425 raft_consensus.cc:740] T 836251ebc2b74f5da88b8a8b2338e2a5 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0172fed22b6b4890a7fdc953fa128af4, State: Running, Role: LEADER
I20260709 13:04:01.177241 11575 leader_election.cc:290] T 6f46a5102fd64b45bf1c4dff863d2b4d P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:01.178258 11425 consensus_queue.cc:260] T 836251ebc2b74f5da88b8a8b2338e2a5 P 0172fed22b6b4890a7fdc953fa128af4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:01.180020 11610 leader_election.cc:290] T 5d439326f23747eaab3c1e7da2d6b785 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:01.180114 11425 raft_consensus.cc:3060] T 836251ebc2b74f5da88b8a8b2338e2a5 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:04:01.181535 10948 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49428 (request call id 64) took 1800 ms. Trace:
I20260709 13:04:01.154785 11688 raft_consensus.cc:493] T e96e80ba9ac34d0998395515e5c8c41a P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Starting leader election (detected failure of leader 10ca5ee68ad64b3283b9d235352c9d23)
I20260709 13:04:01.182094 11103 raft_consensus.cc:2410] T a610d1a71f52432e88fe56130f921fa2 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 186b12638a87410487467df721a3cbda for term 4 because replica has last-logged OpId of term: 2 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260709 13:04:01.182448 11688 raft_consensus.cc:3060] T e96e80ba9ac34d0998395515e5c8c41a P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:04:01.183655 11103 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a977842ba4b744f4bb2febd55d286006" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:01.185268 11103 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7709387f4d3d4f9992d97a0fbca39a40" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:04:01.186034 11103 raft_consensus.cc:3060] T 7709387f4d3d4f9992d97a0fbca39a40 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Advancing to term 3
W20260709 13:04:01.160916 10845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 237326us
I20260709 13:04:01.186626 11705 raft_consensus.cc:515] T 6a9353d1d230416296627d7cf8b1d425 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:01.187225 11097 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37678 (request call id 53) took 1273 ms. Trace:
I20260709 13:04:01.187669 11097 rpcz_store.cc:276] 0709 13:03:59.914114 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:59.954271 (+ 40157us) service_pool.cc:224] Handling call
0709 13:03:59.994605 (+ 40334us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:00.365447 (+370842us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:01.186930 (+821483us) spinlock_profiling.cc:243] Waited 333 ms on lock 0x7b540086fe54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:04:01.187149 (+   219us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":765923712}
I20260709 13:04:01.188971 11705 leader_election.cc:290] T 6a9353d1d230416296627d7cf8b1d425 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:01.189208 10845 leader_election.cc:400] T 0e5d327630e5438b878035d28efe6c0c P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 2 pre-election: Vote denied by peer 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507) with higher term. Message: Invalid argument: T 0e5d327630e5438b878035d28efe6c0c P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 186b12638a87410487467df721a3cbda for earlier term 2. Current term is 3.
I20260709 13:04:01.189749 10845 leader_election.cc:403] T 0e5d327630e5438b878035d28efe6c0c P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:04:01.151142 11673 raft_consensus.cc:2804] T a154cc8b181f460abf4880064379948b P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:04:01.181664 11680 raft_consensus.cc:2804] T 8a69f32c53f24a50b7852d143665d7b4 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:04:01.190624 10844 leader_election.cc:304] T 1e4f2241a94a41f38e481137f868b83d P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47; no voters: 10ca5ee68ad64b3283b9d235352c9d23
I20260709 13:04:01.190383 11685 consensus_queue.cc:1048] T 8a69f32c53f24a50b7852d143665d7b4 P 186b12638a87410487467df721a3cbda [LEADER]: Connected to new peer: Peer: permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:04:01.149858 11102 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f9eb06ed01f4ce68be3af8a8434665b" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:01.076925 11094 rpcz_store.cc:276] 0709 13:03:59.321326 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:59.329523 (+  8197us) service_pool.cc:224] Handling call
0709 13:03:59.403084 (+ 73561us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:59.876525 (+473441us) spinlock_profiling.cc:243] Waited 11.8 ms on lock 0x7b5400334120. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42660d9dc0 00007f42661641c0 00007f4266165231 00007f4266161d8f 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c
0709 13:04:01.061559 (+1185034us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:01.061828 (+   269us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":27075968}
W20260709 13:04:01.149211 10994 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 60972us
I20260709 13:04:01.194247 10920 leader_election.cc:304] T 8a69f32c53f24a50b7852d143665d7b4 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 10ca5ee68ad64b3283b9d235352c9d23; no voters: 186b12638a87410487467df721a3cbda
I20260709 13:04:01.218901 10920 leader_election.cc:304] T e5a5f5a608504650b48fd27f163faed6 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 10ca5ee68ad64b3283b9d235352c9d23; no voters: 186b12638a87410487467df721a3cbda
W20260709 13:04:01.221294 10920 leader_election.cc:343] T a977842ba4b744f4bb2febd55d286006 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:04:01.206135 11029 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a977842ba4b744f4bb2febd55d286006" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:01.147272 11682 raft_consensus.cc:3060] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:04:01.189009 11097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "836251ebc2b74f5da88b8a8b2338e2a5" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:01.205950 11098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d71ca3a1ba6b48478aaf62741bf3cf6a" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:01.166162 11099 raft_consensus.cc:1275] T 525bfdbd6d934afd823f7d3ee3f9d052 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Refusing update from remote peer 10ca5ee68ad64b3283b9d235352c9d23: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 13:04:01.162294 11427 raft_consensus.cc:493] T e5ba72ccd854430baadbc636dfd17136 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:01.230886 11102 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a64b7e99d6c54934a7de6f6dffcf5e94" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:04:01.213966 10995 leader_election.cc:304] T e96e80ba9ac34d0998395515e5c8c41a P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:04:01.212297 11024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7757ecbf402e47bfbc93d0447741e26c" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:01.181684 10952 raft_consensus.cc:1240] T a64b7e99d6c54934a7de6f6dffcf5e94 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Rejecting Update request from peer 873357742d5843579556ee8166f23f47 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
W20260709 13:04:01.160912 10843 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 55003.3us
I20260709 13:04:01.209394 11680 raft_consensus.cc:493] T 8a69f32c53f24a50b7852d143665d7b4 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:01.207194 11595 raft_consensus.cc:493] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:01.240190 11029 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e96e80ba9ac34d0998395515e5c8c41a" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:01.229108 11100 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75492f86fde046c3900df3eac8292fd4" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:01.244534 11025 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46640 (request call id 126) took 1192 ms. Trace:
I20260709 13:04:01.207180 11673 raft_consensus.cc:493] T a154cc8b181f460abf4880064379948b P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 10ca5ee68ad64b3283b9d235352c9d23)
I20260709 13:04:01.245553 11673 raft_consensus.cc:3060] T a154cc8b181f460abf4880064379948b P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:04:01.229674 11697 raft_consensus.cc:2804] T 8a69f32c53f24a50b7852d143665d7b4 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:04:01.246723 11697 raft_consensus.cc:493] T 8a69f32c53f24a50b7852d143665d7b4 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:01.247068 11697 raft_consensus.cc:3060] T 8a69f32c53f24a50b7852d143665d7b4 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:04:01.248624 11654 raft_consensus.cc:515] T 11de93cd9b9d457489ff1cb80f976c9a P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:01.233671 10955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d439326f23747eaab3c1e7da2d6b785" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:01.229385 11619 raft_consensus.cc:2804] T e5a5f5a608504650b48fd27f163faed6 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:04:01.239373 11680 raft_consensus.cc:3060] T 8a69f32c53f24a50b7852d143665d7b4 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:04:01.230695 11027 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a9353d1d230416296627d7cf8b1d425" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:01.227010 10954 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f46a5102fd64b45bf1c4dff863d2b4d" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:01.250609 11152 consensus_queue.cc:260] T ff13ae9a47c7455f962406c6d131a3ff P 0172fed22b6b4890a7fdc953fa128af4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:01.249516 11607 raft_consensus.cc:515] T 9a1716c0adf34bee8a4e99341ba5788f P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:01.252449 10955 raft_consensus.cc:2393] T 5d439326f23747eaab3c1e7da2d6b785 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0172fed22b6b4890a7fdc953fa128af4 in current term 2: Already voted for candidate 186b12638a87410487467df721a3cbda in this term.
I20260709 13:04:01.238922 10995 leader_election.cc:304] T d371c5e27bd843469d2e2e453734cb4a P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:04:01.234068 11427 raft_consensus.cc:3060] T e5ba72ccd854430baadbc636dfd17136 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:04:01.232785 11151 ts_tablet_manager.cc:1434] T 401a9cf11eac479aa445ec8c1d6e33b2 P 186b12638a87410487467df721a3cbda: Time spent starting tablet: real 0.128s	user 0.009s	sys 0.001s
I20260709 13:04:01.227502 11691 consensus_queue.cc:1048] T 9a1716c0adf34bee8a4e99341ba5788f P 873357742d5843579556ee8166f23f47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.023s
I20260709 13:04:01.265300 11152 ts_tablet_manager.cc:1434] T ff13ae9a47c7455f962406c6d131a3ff P 0172fed22b6b4890a7fdc953fa128af4: Time spent starting tablet: real 0.489s	user 0.011s	sys 0.002s
I20260709 13:04:01.241598 11705 raft_consensus.cc:2804] T e96e80ba9ac34d0998395515e5c8c41a P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:04:01.235203 11098 raft_consensus.cc:2468] T d71ca3a1ba6b48478aaf62741bf3cf6a P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 1.
I20260709 13:04:01.251062 11595 raft_consensus.cc:515] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
W20260709 13:04:01.353892 10844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:45807, user_credentials={real_user=slave}} blocked reactor thread for 513802us
I20260709 13:04:01.297573 11149 tablet_bootstrap.cc:492] T a5299b8d044f41b7b68057e63d17b159 P 873357742d5843579556ee8166f23f47: Bootstrap starting.
I20260709 13:04:01.282784 10995 leader_election.cc:304] T 34064dc7bf3747dbb3876dd824ee0688 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:04:01.299396 11152 tablet_bootstrap.cc:492] T 75492f86fde046c3900df3eac8292fd4 P 0172fed22b6b4890a7fdc953fa128af4: Bootstrap starting.
I20260709 13:04:01.182031 10948 rpcz_store.cc:276] 0709 13:03:59.380809 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:59.393640 (+ 12831us) service_pool.cc:224] Handling call
0709 13:03:59.800503 (+406863us) spinlock_profiling.cc:243] Waited 905 us on lock 0x7b580001d198. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f426303db71 00007f426303da22 00007f426bf56c09 00007f426b0b0ee0 00007f426b17a0ab 00007f426b17a131 00007f426b13bfc0 00007f426b126929 00007f426b1263d2 00007f426469dbd4
0709 13:03:59.800732 (+   229us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:00.301983 (+501251us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:01.181215 (+879232us) spinlock_profiling.cc:243] Waited 477 ms on lock 0x7b54000e4554. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:04:01.181439 (+   224us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1099786240}
I20260709 13:04:01.381696 10919 leader_election.cc:304] T ff13ae9a47c7455f962406c6d131a3ff P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47; no voters: 0172fed22b6b4890a7fdc953fa128af4
I20260709 13:04:01.383277 11723 consensus_queue.cc:1048] T c51782cd9d064ec0949e84e24dd4de67 P 186b12638a87410487467df721a3cbda [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260709 13:04:01.380363 10995 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:38967, user_credentials={real_user=slave}} blocked reactor thread for 97700.9us
I20260709 13:04:01.291709 11028 raft_consensus.cc:2368] T 11de93cd9b9d457489ff1cb80f976c9a P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 186b12638a87410487467df721a3cbda for earlier term 2. Current term is 3.
I20260709 13:04:01.386197 10995 leader_election.cc:304] T 4c2a35bb99374e93b573cf870d04faaa P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:04:01.350940 11151 tablet_bootstrap.cc:492] T 30f628e3d7914d609e7f1d980a74c654 P 186b12638a87410487467df721a3cbda: Bootstrap starting.
I20260709 13:04:01.380587 10844 leader_election.cc:304] T d371c5e27bd843469d2e2e453734cb4a P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:04:01.270840 11619 raft_consensus.cc:697] T e5a5f5a608504650b48fd27f163faed6 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 LEADER]: Becoming Leader. State: Replica: 10ca5ee68ad64b3283b9d235352c9d23, State: Running, Role: LEADER
W20260709 13:04:01.393626 10844 leader_election.cc:343] T a5299b8d044f41b7b68057e63d17b159 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 873357742d5843579556ee8166f23f47 (127.0.107.195:45807): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:04:01.297731 11656 raft_consensus.cc:2804] T d371c5e27bd843469d2e2e453734cb4a P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:04:01.397258 11656 raft_consensus.cc:493] T d371c5e27bd843469d2e2e453734cb4a P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:01.397660 11656 raft_consensus.cc:3060] T d371c5e27bd843469d2e2e453734cb4a P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:04:01.386461 11633 raft_consensus.cc:2804] T 34064dc7bf3747dbb3876dd824ee0688 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:04:01.350832 11598 raft_consensus.cc:493] T d270b9e5481245d9b5340a454a7cd10b P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:01.400282 11506 consensus_queue.cc:1048] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 873357742d5843579556ee8166f23f47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:04:01.400472 10994 leader_election.cc:304] T 4f9eb06ed01f4ce68be3af8a8434665b P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 873357742d5843579556ee8166f23f47; no voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda
I20260709 13:04:01.401855 11506 raft_consensus.cc:2749] T 4f9eb06ed01f4ce68be3af8a8434665b P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:04:01.400028 11598 raft_consensus.cc:515] T d270b9e5481245d9b5340a454a7cd10b P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:01.403645 10919 leader_election.cc:304] T e89e640169e14e2d98f968f76027daa7 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23; no voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47
I20260709 13:04:01.266999 10845 leader_election.cc:304] T f371d2565dd84667b5613848604949bb P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda; no voters: 
I20260709 13:04:01.405227 11148 raft_consensus.cc:359] T 1e4f2241a94a41f38e481137f868b83d P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:01.405992 11148 raft_consensus.cc:385] T 1e4f2241a94a41f38e481137f868b83d P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:04:01.406630 11360 raft_consensus.cc:2749] T e89e640169e14e2d98f968f76027daa7 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 13:04:01.406248 11148 raft_consensus.cc:740] T 1e4f2241a94a41f38e481137f868b83d P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10ca5ee68ad64b3283b9d235352c9d23, State: Initialized, Role: FOLLOWER
I20260709 13:04:01.284210 11023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46642 (request call id 54) took 1487 ms. Trace:
I20260709 13:04:01.299767 11098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a9353d1d230416296627d7cf8b1d425" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:01.303406 11024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65bf0a62f4d84ef18dd6b23e082724b8" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
I20260709 13:04:01.482897 10920 leader_election.cc:304] T d71ca3a1ba6b48478aaf62741bf3cf6a P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 10ca5ee68ad64b3283b9d235352c9d23; no voters: 
I20260709 13:04:01.280737 11705 raft_consensus.cc:493] T e96e80ba9ac34d0998395515e5c8c41a P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:01.484246 11705 raft_consensus.cc:3060] T e96e80ba9ac34d0998395515e5c8c41a P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:04:01.399516 11633 raft_consensus.cc:493] T 34064dc7bf3747dbb3876dd824ee0688 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:01.485044 11633 raft_consensus.cc:3060] T 34064dc7bf3747dbb3876dd824ee0688 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:04:01.347608 11610 raft_consensus.cc:493] T ff13ae9a47c7455f962406c6d131a3ff P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:01.486716 11708 raft_consensus.cc:2804] T d71ca3a1ba6b48478aaf62741bf3cf6a P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:04:01.487205 11708 raft_consensus.cc:493] T d71ca3a1ba6b48478aaf62741bf3cf6a P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:01.487654 11708 raft_consensus.cc:3060] T d71ca3a1ba6b48478aaf62741bf3cf6a P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:04:01.312062 11607 leader_election.cc:290] T 9a1716c0adf34bee8a4e99341ba5788f P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 3 election: Requested vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:01.437309 11148 consensus_queue.cc:260] T 1e4f2241a94a41f38e481137f868b83d P 10ca5ee68ad64b3283b9d235352c9d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
W20260709 13:04:01.286615 10946 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49428 (request call id 67) took 1477 ms (client timeout 1514 ms). Trace:
W20260709 13:04:01.402108 10994 leader_election.cc:343] T 75492f86fde046c3900df3eac8292fd4 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:04:01.289543 10992 leader_election.cc:304] T 63d44f6ba6434ab2870fe3190f4d7c2f P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 873357742d5843579556ee8166f23f47; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda
W20260709 13:04:01.352348 10843 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 53573.7us
I20260709 13:04:01.314531 11721 raft_consensus.cc:2804] T 1e4f2241a94a41f38e481137f868b83d P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:04:01.311712 10879 raft_consensus.cc:2468] T 0e5d327630e5438b878035d28efe6c0c P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 3.
W20260709 13:04:01.445429 11069 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:01.253242 11067 leader_election.cc:304] T bac7e5040f984ba2b7146bd36af7267e P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47
I20260709 13:04:01.275873 11027 raft_consensus.cc:2468] T 6a9353d1d230416296627d7cf8b1d425 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 2.
I20260709 13:04:01.407994 11360 consensus_queue.cc:1048] T 525bfdbd6d934afd823f7d3ee3f9d052 P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:04:01.475644 11654 leader_election.cc:290] T 11de93cd9b9d457489ff1cb80f976c9a P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 3 election: Requested vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:01.274232 11518 raft_consensus.cc:697] T 5d439326f23747eaab3c1e7da2d6b785 P 186b12638a87410487467df721a3cbda [term 2 LEADER]: Becoming Leader. State: Replica: 186b12638a87410487467df721a3cbda, State: Running, Role: LEADER
I20260709 13:04:01.244973 11025 rpcz_store.cc:276] 0709 13:04:00.052511 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:00.299160 (+246649us) service_pool.cc:224] Handling call
0709 13:04:00.643627 (+344467us) spinlock_profiling.cc:243] Waited 6.43 ms on lock 0x7b58007a2898. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f426303db71 00007f426303da22 00007f426bf56c09 00007f426b0b0ee0 00007f426b17a0ab 00007f426b17a131 00007f426b13bfc0 00007f426b126929 00007f426b1263d2 00007f426469dbd4
0709 13:04:01.243602 (+599975us) spinlock_profiling.cc:243] Waited 24 ms on lock 0x7b58007a2898. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f426303de9c 00007f426303da5b 00007f426bf56c09 00007f426b0b0ee0 00007f426b17a0ab 00007f426b17a131 00007f426b13bfc0 00007f426b126929 00007f426b1263d2 00007f426469dbd4
0709 13:04:01.243820 (+   218us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:01.244052 (+   232us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:01.244079 (+    27us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:01.244227 (+   148us) raft_consensus.cc:1645] Marking committed up to 1
0709 13:04:01.244265 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:01.244336 (+    71us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:01.244466 (+   130us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":69896320}
I20260709 13:04:01.408833 11553 raft_consensus.cc:2804] T ff13ae9a47c7455f962406c6d131a3ff P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:04:01.468498 11545 raft_consensus.cc:2804] T f371d2565dd84667b5613848604949bb P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:04:01.297088 11094 raft_consensus.cc:1240] T 9a1716c0adf34bee8a4e99341ba5788f P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Rejecting Update request from peer 873357742d5843579556ee8166f23f47 for earlier term 2. Current term is 3. Ops: []
I20260709 13:04:01.483083 11639 raft_consensus.cc:493] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:01.486846 11610 raft_consensus.cc:515] T ff13ae9a47c7455f962406c6d131a3ff P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:01.304363 11022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d71ca3a1ba6b48478aaf62741bf3cf6a" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:01.304327 11026 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34064dc7bf3747dbb3876dd824ee0688" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
I20260709 13:04:01.468645 11427 raft_consensus.cc:515] T e5ba72ccd854430baadbc636dfd17136 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:01.409161 11583 raft_consensus.cc:2804] T 4c2a35bb99374e93b573cf870d04faaa P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260709 13:04:01.511751 10993 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:01.775936 11591 leader_election.cc:290] T b4cd3d7dfbbf4096908ff11e9dc57141 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
W20260709 13:04:01.511053 10946 rpcz_store.cc:269] 0709 13:03:59.808947 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:59.866884 (+ 57937us) service_pool.cc:224] Handling call
0709 13:04:01.286342 (+1419458us) spinlock_profiling.cc:243] Waited 592 ms on lock 0x7b5400415354. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:04:01.286541 (+   199us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1361045376}
I20260709 13:04:01.776378 11518 consensus_queue.cc:237] T 5d439326f23747eaab3c1e7da2d6b785 P 186b12638a87410487467df721a3cbda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:01.777732 10946 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c2a35bb99374e93b573cf870d04faaa" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:04:01.476408 10948 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75492f86fde046c3900df3eac8292fd4" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:01.752993 11425 raft_consensus.cc:2749] T bac7e5040f984ba2b7146bd36af7267e P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:04:01.782934 10948 raft_consensus.cc:2468] T 75492f86fde046c3900df3eac8292fd4 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 0.
I20260709 13:04:01.472741 11023 rpcz_store.cc:276] 0709 13:03:59.797086 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:00.240939 (+443853us) service_pool.cc:224] Handling call
0709 13:04:01.283975 (+1043036us) spinlock_profiling.cc:243] Waited 323 ms on lock 0x7b54005ff5d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:04:01.284143 (+   168us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":743246208}
W20260709 13:04:01.784228 11719 log.cc:927] Time spent T 6f46a5102fd64b45bf1c4dff863d2b4d P 873357742d5843579556ee8166f23f47: Append to log took a long time: real 0.319s	user 0.001s	sys 0.000s
I20260709 13:04:01.465543 11097 raft_consensus.cc:2468] T 836251ebc2b74f5da88b8a8b2338e2a5 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 2.
I20260709 13:04:01.660653 11721 raft_consensus.cc:493] T 1e4f2241a94a41f38e481137f868b83d P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:01.786695 11721 raft_consensus.cc:3060] T 1e4f2241a94a41f38e481137f868b83d P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Advancing to term 1
W20260709 13:04:01.789273 11697 consensus_meta.cc:287] T 8a69f32c53f24a50b7852d143665d7b4 P 10ca5ee68ad64b3283b9d235352c9d23: Time spent flushing consensus metadata: real 0.542s	user 0.010s	sys 0.000s
W20260709 13:04:01.790172 11672 consensus_meta.cc:287] T a64b7e99d6c54934a7de6f6dffcf5e94 P 0172fed22b6b4890a7fdc953fa128af4: Time spent flushing consensus metadata: real 0.600s	user 0.007s	sys 0.002s
I20260709 13:04:01.668663 11598 leader_election.cc:290] T d270b9e5481245d9b5340a454a7cd10b P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:01.669533 10953 raft_consensus.cc:1240] T e89e640169e14e2d98f968f76027daa7 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Rejecting Update request from peer 873357742d5843579556ee8166f23f47 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260709 13:04:01.645560 11024 raft_consensus.cc:3060] T 65bf0a62f4d84ef18dd6b23e082724b8 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:04:01.464928 11702 consensus_queue.cc:1048] T 8c160ddfe9b34bdaa3835d0915c1e06f P 873357742d5843579556ee8166f23f47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.188s
I20260709 13:04:01.749387 11673 raft_consensus.cc:515] T a154cc8b181f460abf4880064379948b P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:01.511637 11724 raft_consensus.cc:2804] T d371c5e27bd843469d2e2e453734cb4a P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:04:01.820096 11724 raft_consensus.cc:493] T d371c5e27bd843469d2e2e453734cb4a P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:01.820523 11724 raft_consensus.cc:3060] T d371c5e27bd843469d2e2e453734cb4a P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:04:01.820963 11673 leader_election.cc:290] T a154cc8b181f460abf4880064379948b P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 2 election: Requested vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:01.820986 10955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5299b8d044f41b7b68057e63d17b159" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:01.822510 10955 raft_consensus.cc:2468] T a5299b8d044f41b7b68057e63d17b159 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 0.
I20260709 13:04:01.663082 11020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f46a5102fd64b45bf1c4dff863d2b4d" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
W20260709 13:04:01.644445 10872 consensus_meta.cc:287] T 7709387f4d3d4f9992d97a0fbca39a40 P 186b12638a87410487467df721a3cbda: Time spent flushing consensus metadata: real 0.637s	user 0.010s	sys 0.003s
I20260709 13:04:01.807359 11721 raft_consensus.cc:515] T 1e4f2241a94a41f38e481137f868b83d P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:01.827440 10872 raft_consensus.cc:2468] T 7709387f4d3d4f9992d97a0fbca39a40 P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 3.
I20260709 13:04:01.664436 11607 raft_consensus.cc:493] T 4f9eb06ed01f4ce68be3af8a8434665b P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 186b12638a87410487467df721a3cbda)
I20260709 13:04:01.828563 11607 raft_consensus.cc:515] T 4f9eb06ed01f4ce68be3af8a8434665b P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
W20260709 13:04:01.829500 11628 consensus_meta.cc:287] T 5de5f0581e3b41dea729363d6ca563bb P 0172fed22b6b4890a7fdc953fa128af4: Time spent flushing consensus metadata: real 0.669s	user 0.012s	sys 0.001s
W20260709 13:04:01.644340 10994 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 242251us
I20260709 13:04:01.663746 11685 raft_consensus.cc:2749] T 0e5d327630e5438b878035d28efe6c0c P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507) with higher term. Message: Invalid argument: T 0e5d327630e5438b878035d28efe6c0c P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 186b12638a87410487467df721a3cbda for earlier term 2. Current term is 3.
W20260709 13:04:01.649653 10919 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:45807, user_credentials={real_user=slave}} blocked reactor thread for 243372us
W20260709 13:04:01.877548 10919 leader_election.cc:343] T a977842ba4b744f4bb2febd55d286006 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 873357742d5843579556ee8166f23f47 (127.0.107.195:45807): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:04:01.521486 11653 raft_consensus.cc:515] T 1e4f2241a94a41f38e481137f868b83d P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:01.880738 11653 leader_election.cc:290] T 1e4f2241a94a41f38e481137f868b83d P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 1 election: Requested vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:01.722606 11620 leader_election.cc:290] T e96e80ba9ac34d0998395515e5c8c41a P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:01.881987 11620 raft_consensus.cc:493] T 5d439326f23747eaab3c1e7da2d6b785 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:01.882673 11620 raft_consensus.cc:515] T 5d439326f23747eaab3c1e7da2d6b785 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:01.878204 10919 leader_election.cc:304] T a977842ba4b744f4bb2febd55d286006 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23; no voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47
W20260709 13:04:01.888078 10826 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 13:04:01.890005 11099 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37666 (request call id 118) took 1310 ms. Trace:
I20260709 13:04:01.893087 11099 rpcz_store.cc:276] 0709 13:04:00.579928 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:00.697074 (+117146us) service_pool.cc:224] Handling call
0709 13:04:01.145380 (+448306us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:01.233728 (+ 88348us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:01.889725 (+655997us) spinlock_profiling.cc:243] Waited 446 ms on lock 0x7b540063a950. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661618ad 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:04:01.889924 (+   199us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1026641280}
W20260709 13:04:01.519171 11677 log.cc:927] Time spent T 555217d45d9b4ee492dbb3ddaf0f7c02 P 186b12638a87410487467df721a3cbda: Append to log took a long time: real 0.057s	user 0.000s	sys 0.000s
W20260709 13:04:01.667433 11717 log.cc:927] Time spent T 32147c6d2d5a477aa38c597c7f78df5b P 873357742d5843579556ee8166f23f47: Append to log took a long time: real 0.288s	user 0.000s	sys 0.001s
W20260709 13:04:01.791204 11722 log.cc:927] Time spent T 32147c6d2d5a477aa38c597c7f78df5b P 10ca5ee68ad64b3283b9d235352c9d23: Append to log took a long time: real 0.075s	user 0.000s	sys 0.000s
I20260709 13:04:01.794049 11610 leader_election.cc:290] T ff13ae9a47c7455f962406c6d131a3ff P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
W20260709 13:04:01.795226 11726 log.cc:927] Time spent T 4f9eb06ed01f4ce68be3af8a8434665b P 186b12638a87410487467df721a3cbda: Append to log took a long time: real 0.064s	user 0.000s	sys 0.000s
I20260709 13:04:01.796306 11427 leader_election.cc:290] T e5ba72ccd854430baadbc636dfd17136 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 3 election: Requested vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
W20260709 13:04:01.803191 11680 consensus_meta.cc:287] T 8a69f32c53f24a50b7852d143665d7b4 P 0172fed22b6b4890a7fdc953fa128af4: Time spent flushing consensus metadata: real 0.527s	user 0.011s	sys 0.000s
I20260709 13:04:01.765130 11639 raft_consensus.cc:515] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:01.896777 11680 raft_consensus.cc:515] T 8a69f32c53f24a50b7852d143665d7b4 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:01.897451 10947 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d156d6ae2ca4fa59a1e387edf0fea39" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:01.898607 11639 leader_election.cc:290] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:01.668057 11510 raft_consensus.cc:2749] T 63d44f6ba6434ab2870fe3190f4d7c2f P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:04:01.751554 11606 raft_consensus.cc:3055] T e89e640169e14e2d98f968f76027daa7 P 873357742d5843579556ee8166f23f47 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:04:01.899753 11606 raft_consensus.cc:740] T e89e640169e14e2d98f968f76027daa7 P 873357742d5843579556ee8166f23f47 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 873357742d5843579556ee8166f23f47, State: Running, Role: LEADER
I20260709 13:04:01.900822 11606 consensus_queue.cc:260] T e89e640169e14e2d98f968f76027daa7 P 873357742d5843579556ee8166f23f47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:01.751284 11545 raft_consensus.cc:493] T f371d2565dd84667b5613848604949bb P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Starting leader election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:01.902707 11545 raft_consensus.cc:3060] T f371d2565dd84667b5613848604949bb P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:04:01.774698 11583 raft_consensus.cc:493] T 4c2a35bb99374e93b573cf870d04faaa P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:01.813376 10948 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff13ae9a47c7455f962406c6d131a3ff" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:01.817400 10953 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5ba72ccd854430baadbc636dfd17136" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:04:01.838546 11508 raft_consensus.cc:2764] T 7709387f4d3d4f9992d97a0fbca39a40 P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Leader election decision vote started in defunct term 2: won
I20260709 13:04:01.830514 11721 leader_election.cc:290] T 1e4f2241a94a41f38e481137f868b83d P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 1 election: Requested vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:01.898196 10947 raft_consensus.cc:2468] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 1.
I20260709 13:04:01.909193 11427 raft_consensus.cc:493] T b4cd3d7dfbbf4096908ff11e9dc57141 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:01.909870 11427 raft_consensus.cc:515] T b4cd3d7dfbbf4096908ff11e9dc57141 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:01.907202 11583 raft_consensus.cc:3060] T 4c2a35bb99374e93b573cf870d04faaa P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:04:01.912762 10952 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e4f2241a94a41f38e481137f868b83d" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:04:01.913421 10952 raft_consensus.cc:3060] T 1e4f2241a94a41f38e481137f868b83d P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:04:01.910028 10947 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b37f8fca8dc42718b2a42f0883a956f" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:01.914248 11028 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46642 (request call id 60) took 1121 ms. Trace:
I20260709 13:04:01.914661 11028 rpcz_store.cc:276] 0709 13:04:00.792359 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:00.794623 (+  2264us) service_pool.cc:224] Handling call
0709 13:04:01.913991 (+1119368us) spinlock_profiling.cc:243] Waited 279 ms on lock 0x7b540059fbd4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:04:01.914150 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":640700800}
I20260709 13:04:01.839495 11633 raft_consensus.cc:515] T 34064dc7bf3747dbb3876dd824ee0688 P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:01.915829 11097 raft_consensus.cc:1240] T e5ba72ccd854430baadbc636dfd17136 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Rejecting Update request from peer 873357742d5843579556ee8166f23f47 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260709 13:04:01.915990 11427 leader_election.cc:290] T b4cd3d7dfbbf4096908ff11e9dc57141 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:01.916843 10954 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b67eff0634e540eab91cf8cee795969f" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:01.917017 11097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d270b9e5481245d9b5340a454a7cd10b" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:01.917389 11633 leader_election.cc:290] T 34064dc7bf3747dbb3876dd824ee0688 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 3 election: Requested vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:01.831257 11149 tablet_bootstrap.cc:654] T a5299b8d044f41b7b68057e63d17b159 P 873357742d5843579556ee8166f23f47: Neither blocks nor log segments found. Creating new log.
I20260709 13:04:01.917466 11360 consensus_queue.cc:1048] T 525bfdbd6d934afd823f7d3ee3f9d052 P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:04:01.914088 10953 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a69f32c53f24a50b7852d143665d7b4" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:04:01.920020 11026 raft_consensus.cc:2393] T 34064dc7bf3747dbb3876dd824ee0688 P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 10ca5ee68ad64b3283b9d235352c9d23 in current term 3: Already voted for candidate 873357742d5843579556ee8166f23f47 in this term.
W20260709 13:04:01.814762 11658 consensus_meta.cc:287] T 87cd8f5097624c8692f1d83544d8881e P 186b12638a87410487467df721a3cbda: Time spent flushing consensus metadata: real 0.938s	user 0.011s	sys 0.004s
I20260709 13:04:01.921334 11658 raft_consensus.cc:515] T 87cd8f5097624c8692f1d83544d8881e P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:01.922390 11028 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f1466124dbd4f6ebb3c4d62ca48bf96" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:01.923581 11658 leader_election.cc:290] T 87cd8f5097624c8692f1d83544d8881e P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 3 election: Requested vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:01.830010 11628 raft_consensus.cc:515] T 5de5f0581e3b41dea729363d6ca563bb P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:01.925149 11620 leader_election.cc:290] T 5d439326f23747eaab3c1e7da2d6b785 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:01.839581 11607 leader_election.cc:290] T 4f9eb06ed01f4ce68be3af8a8434665b P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:01.842144 11595 leader_election.cc:290] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:01.926218 11628 leader_election.cc:290] T 5de5f0581e3b41dea729363d6ca563bb P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 3 election: Requested vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:01.927035 10878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e4f2241a94a41f38e481137f868b83d" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:04:01.899250 11680 leader_election.cc:290] T 8a69f32c53f24a50b7852d143665d7b4 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 3 election: Requested vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
W20260709 13:04:01.845814 10843 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.107.194:44903 timed out after 1.514s (SENT) [suppressed 6 similar messages]
W20260709 13:04:01.773612 11103 consensus_meta.cc:287] T 7709387f4d3d4f9992d97a0fbca39a40 P 0172fed22b6b4890a7fdc953fa128af4: Time spent flushing consensus metadata: real 0.564s	user 0.008s	sys 0.004s
I20260709 13:04:01.729204 11619 consensus_queue.cc:237] T e5a5f5a608504650b48fd27f163faed6 P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:01.940145 10879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34064dc7bf3747dbb3876dd824ee0688" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
W20260709 13:04:01.849534 11682 consensus_meta.cc:287] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 873357742d5843579556ee8166f23f47: Time spent flushing consensus metadata: real 0.619s	user 0.003s	sys 0.008s
I20260709 13:04:01.857556 10955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11de93cd9b9d457489ff1cb80f976c9a" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:01.964359 10955 raft_consensus.cc:2393] T 11de93cd9b9d457489ff1cb80f976c9a P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 186b12638a87410487467df721a3cbda in current term 2: Already voted for candidate 10ca5ee68ad64b3283b9d235352c9d23 in this term.
I20260709 13:04:01.860319 11151 tablet_bootstrap.cc:654] T 30f628e3d7914d609e7f1d980a74c654 P 186b12638a87410487467df721a3cbda: Neither blocks nor log segments found. Creating new log.
W20260709 13:04:01.861521 11688 consensus_meta.cc:287] T e96e80ba9ac34d0998395515e5c8c41a P 186b12638a87410487467df721a3cbda: Time spent flushing consensus metadata: real 0.679s	user 0.012s	sys 0.000s
I20260709 13:04:01.861814 11101 raft_consensus.cc:3060] T acc241e7273d49149984968596242712 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:04:01.891407 11020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a64b7e99d6c54934a7de6f6dffcf5e94" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
I20260709 13:04:01.892228 11598 raft_consensus.cc:2749] T a977842ba4b744f4bb2febd55d286006 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:04:01.896600 11672 raft_consensus.cc:515] T a64b7e99d6c54934a7de6f6dffcf5e94 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
W20260709 13:04:01.804113 11069 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:01.902148 11606 raft_consensus.cc:3060] T e89e640169e14e2d98f968f76027daa7 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:04:01.907939 10920 leader_election.cc:304] T 836251ebc2b74f5da88b8a8b2338e2a5 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 10ca5ee68ad64b3283b9d235352c9d23; no voters: 
W20260709 13:04:01.773912 11067 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:45807, user_credentials={real_user=slave}} blocked reactor thread for 533636us
W20260709 13:04:01.980439 10920 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 72651.7us
I20260709 13:04:01.915023 10949 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49424 (request call id 130) took 1127 ms. Trace:
I20260709 13:04:01.789866 11697 raft_consensus.cc:515] T 8a69f32c53f24a50b7852d143665d7b4 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:01.929620 10878 raft_consensus.cc:2393] T 1e4f2241a94a41f38e481137f868b83d P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 873357742d5843579556ee8166f23f47 in current term 1: Already voted for candidate 186b12638a87410487467df721a3cbda in this term.
I20260709 13:04:01.928324 11595 raft_consensus.cc:493] T 7839ca2d2c13431a85f44d7c458c6351 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:01.932324 10952 raft_consensus.cc:2468] T 1e4f2241a94a41f38e481137f868b83d P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 1.
I20260709 13:04:01.941169 11103 raft_consensus.cc:2468] T 7709387f4d3d4f9992d97a0fbca39a40 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 3.
I20260709 13:04:01.935333 10871 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11de93cd9b9d457489ff1cb80f976c9a" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:04:02.002318 11102 raft_consensus.cc:2393] T a64b7e99d6c54934a7de6f6dffcf5e94 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 10ca5ee68ad64b3283b9d235352c9d23 in current term 3: Already voted for candidate 0172fed22b6b4890a7fdc953fa128af4 in this term.
I20260709 13:04:01.841428 11152 tablet_bootstrap.cc:654] T 75492f86fde046c3900df3eac8292fd4 P 0172fed22b6b4890a7fdc953fa128af4: Neither blocks nor log segments found. Creating new log.
W20260709 13:04:01.659991 10845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 419160us
I20260709 13:04:01.973382 10879 raft_consensus.cc:3060] T 34064dc7bf3747dbb3876dd824ee0688 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:04:01.809474 11656 raft_consensus.cc:515] T d371c5e27bd843469d2e2e453734cb4a P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:01.957821 11097 raft_consensus.cc:2468] T d270b9e5481245d9b5340a454a7cd10b P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 2.
I20260709 13:04:01.984897 11680 raft_consensus.cc:493] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:01.997399 10955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4cd3d7dfbbf4096908ff11e9dc57141" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
W20260709 13:04:01.993161 10995 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:38967, user_credentials={real_user=slave}} blocked reactor thread for 69119.8us
W20260709 13:04:01.977419 10873 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.662s	user 0.000s	sys 0.014s
I20260709 13:04:01.992328 11597 raft_consensus.cc:2804] T 836251ebc2b74f5da88b8a8b2338e2a5 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260709 13:04:02.015565 11148 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.501s	user 0.002s	sys 0.013s
I20260709 13:04:01.973721 11688 raft_consensus.cc:515] T e96e80ba9ac34d0998395515e5c8c41a P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
W20260709 13:04:02.008962 11708 consensus_meta.cc:287] T d71ca3a1ba6b48478aaf62741bf3cf6a P 10ca5ee68ad64b3283b9d235352c9d23: Time spent flushing consensus metadata: real 0.521s	user 0.015s	sys 0.000s
W20260709 13:04:02.028270 10992 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 738799us
I20260709 13:04:02.043385 11680 raft_consensus.cc:515] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:02.053717 11096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a154cc8b181f460abf4880064379948b" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:04:02.054993 11094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d439326f23747eaab3c1e7da2d6b785" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:02.055620 11735 consensus_queue.cc:1059] T e5a5f5a608504650b48fd27f163faed6 P 186b12638a87410487467df721a3cbda [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:04:02.008638 11595 raft_consensus.cc:515] T 7839ca2d2c13431a85f44d7c458c6351 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:02.056399 11026 raft_consensus.cc:1240] T a154cc8b181f460abf4880064379948b P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Rejecting Update request from peer 10ca5ee68ad64b3283b9d235352c9d23 for earlier term 1. Current term is 2. Ops: []
I20260709 13:04:02.058104 11656 leader_election.cc:290] T d371c5e27bd843469d2e2e453734cb4a P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 2 election: Requested vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:02.058313 11097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f1466124dbd4f6ebb3c4d62ca48bf96" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:02.061863 10774 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:59608 (request call id 62) took 3133 ms. Trace:
I20260709 13:04:02.013288 10871 raft_consensus.cc:3060] T 11de93cd9b9d457489ff1cb80f976c9a P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Advancing to term 3
W20260709 13:04:02.054165 10845 leader_election.cc:343] T b67eff0634e540eab91cf8cee795969f P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:04:02.066293 10920 leader_election.cc:304] T d270b9e5481245d9b5340a454a7cd10b P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 10ca5ee68ad64b3283b9d235352c9d23; no voters: 
I20260709 13:04:02.067992 11610 raft_consensus.cc:493] T 8c160ddfe9b34bdaa3835d0915c1e06f P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:02.069317 10994 leader_election.cc:304] T 7709387f4d3d4f9992d97a0fbca39a40 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47; no voters: 
W20260709 13:04:02.124526 10994 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 55300.6us
W20260709 13:04:02.049527 11705 consensus_meta.cc:287] T e96e80ba9ac34d0998395515e5c8c41a P 873357742d5843579556ee8166f23f47: Time spent flushing consensus metadata: real 0.565s	user 0.018s	sys 0.004s
I20260709 13:04:02.047961 11102 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4cd3d7dfbbf4096908ff11e9dc57141" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:02.100188 11100 raft_consensus.cc:1240] T a64b7e99d6c54934a7de6f6dffcf5e94 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Rejecting Update request from peer 873357742d5843579556ee8166f23f47 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260709 13:04:02.074009 10955 raft_consensus.cc:2468] T b4cd3d7dfbbf4096908ff11e9dc57141 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 2.
I20260709 13:04:02.007499 11553 raft_consensus.cc:697] T ff13ae9a47c7455f962406c6d131a3ff P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 LEADER]: Becoming Leader. State: Replica: 10ca5ee68ad64b3283b9d235352c9d23, State: Running, Role: LEADER
I20260709 13:04:02.038199 11028 raft_consensus.cc:2410] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 10ca5ee68ad64b3283b9d235352c9d23 for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20260709 13:04:02.031913 10873 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.716s	user 0.000s	sys 0.015s
I20260709 13:04:02.014005 10953 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5de5f0581e3b41dea729363d6ca563bb" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
W20260709 13:04:02.083276 11148 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.588s	user 0.006s	sys 0.017s
I20260709 13:04:02.064026 10776 catalog_manager.cc:5697] T 6f46a5102fd64b45bf1c4dff863d2b4d P 873357742d5843579556ee8166f23f47 reported cstate change: term changed from 1 to 2, leader changed from 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196) to 873357742d5843579556ee8166f23f47 (127.0.107.195). New cstate: current_term: 2 leader_uuid: "873357742d5843579556ee8166f23f47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } health_report { overall_health: UNKNOWN } } }
I20260709 13:04:02.048902 11597 raft_consensus.cc:493] T 836251ebc2b74f5da88b8a8b2338e2a5 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:02.164368 11597 raft_consensus.cc:3060] T 836251ebc2b74f5da88b8a8b2338e2a5 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:04:02.124125 10946 raft_consensus.cc:2468] T 4c2a35bb99374e93b573cf870d04faaa P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 2.
I20260709 13:04:02.165232 11545 raft_consensus.cc:515] T f371d2565dd84667b5613848604949bb P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:02.047529 10947 raft_consensus.cc:1240] T 8a69f32c53f24a50b7852d143665d7b4 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Rejecting Update request from peer 186b12638a87410487467df721a3cbda for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260709 13:04:02.166930 11149 tablet_bootstrap.cc:492] T a5299b8d044f41b7b68057e63d17b159 P 873357742d5843579556ee8166f23f47: No bootstrap required, opened a new log
I20260709 13:04:02.167575 11149 ts_tablet_manager.cc:1403] T a5299b8d044f41b7b68057e63d17b159 P 873357742d5843579556ee8166f23f47: Time spent bootstrapping tablet: real 0.891s	user 0.030s	sys 0.043s
I20260709 13:04:02.136610 11067 leader_election.cc:304] T 0e5d327630e5438b878035d28efe6c0c P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:04:02.134999 11705 raft_consensus.cc:515] T e96e80ba9ac34d0998395515e5c8c41a P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:02.170091 11029 raft_consensus.cc:2393] T e96e80ba9ac34d0998395515e5c8c41a P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 10ca5ee68ad64b3283b9d235352c9d23 in current term 3: Already voted for candidate 873357742d5843579556ee8166f23f47 in this term.
I20260709 13:04:02.047111 11103 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d156d6ae2ca4fa59a1e387edf0fea39" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:02.013625 11510 consensus_queue.cc:1059] T e5ba72ccd854430baadbc636dfd17136 P 873357742d5843579556ee8166f23f47 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.073s
I20260709 13:04:02.170812 11103 raft_consensus.cc:2468] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 1.
I20260709 13:04:02.171079 11510 raft_consensus.cc:3055] T e5ba72ccd854430baadbc636dfd17136 P 873357742d5843579556ee8166f23f47 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:04:02.073781 10950 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49434 (request call id 116) took 1018 ms. Trace:
I20260709 13:04:02.171386 11425 raft_consensus.cc:2804] T 0e5d327630e5438b878035d28efe6c0c P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:04:02.104252 10919 leader_election.cc:304] T a64b7e99d6c54934a7de6f6dffcf5e94 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23; no voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47
I20260709 13:04:02.171907 11103 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d371c5e27bd843469d2e2e453734cb4a" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:04:02.172592 11598 raft_consensus.cc:2749] T a64b7e99d6c54934a7de6f6dffcf5e94 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20260709 13:04:02.172664 10919 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:45807, user_credentials={real_user=slave}} blocked reactor thread for 68555.6us
I20260709 13:04:02.055835 11695 raft_consensus.cc:493] T 63d44f6ba6434ab2870fe3190f4d7c2f P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 186b12638a87410487467df721a3cbda)
I20260709 13:04:01.986900 10949 rpcz_store.cc:276] 0709 13:04:00.787489 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:00.789920 (+  2431us) service_pool.cc:224] Handling call
0709 13:04:01.914820 (+1124900us) spinlock_profiling.cc:243] Waited 72.8 ms on lock 0x7b540040ea54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:04:01.914960 (+   140us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":167530368}
I20260709 13:04:02.174319 11695 raft_consensus.cc:515] T 63d44f6ba6434ab2870fe3190f4d7c2f P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:02.095793 11708 raft_consensus.cc:515] T d71ca3a1ba6b48478aaf62741bf3cf6a P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
W20260709 13:04:02.125519 10920 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 59340.3us
W20260709 13:04:02.090245 10843 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 244471us
I20260709 13:04:02.130245 11102 raft_consensus.cc:2468] T b4cd3d7dfbbf4096908ff11e9dc57141 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 2.
W20260709 13:04:02.049593 10844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:45807, user_credentials={real_user=slave}} blocked reactor thread for 653884us
I20260709 13:04:02.160050 10776 catalog_manager.cc:5697] T 32147c6d2d5a477aa38c597c7f78df5b P 873357742d5843579556ee8166f23f47 reported cstate change: term changed from 1 to 2, leader changed from 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196) to 873357742d5843579556ee8166f23f47 (127.0.107.195). New cstate: current_term: 2 leader_uuid: "873357742d5843579556ee8166f23f47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } health_report { overall_health: UNKNOWN } } }
W20260709 13:04:02.160974 10995 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:38967, user_credentials={real_user=slave}} blocked reactor thread for 69341.9us
I20260709 13:04:02.161507 11595 leader_election.cc:290] T 7839ca2d2c13431a85f44d7c458c6351 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:02.161792 11097 raft_consensus.cc:2393] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 10ca5ee68ad64b3283b9d235352c9d23 in current term 3: Already voted for candidate 0172fed22b6b4890a7fdc953fa128af4 in this term.
W20260709 13:04:02.129626 10845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 75509.7us
I20260709 13:04:02.131575 11610 raft_consensus.cc:515] T 8c160ddfe9b34bdaa3835d0915c1e06f P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:02.132400 11633 raft_consensus.cc:2804] T 7709387f4d3d4f9992d97a0fbca39a40 P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:04:02.117538 11094 raft_consensus.cc:2468] T 5d439326f23747eaab3c1e7da2d6b785 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 1.
I20260709 13:04:02.128522 11427 consensus_queue.cc:1048] T d371c5e27bd843469d2e2e453734cb4a P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.040s
I20260709 13:04:02.136686 11656 consensus_queue.cc:1059] T 9a1716c0adf34bee8a4e99341ba5788f P 873357742d5843579556ee8166f23f47 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.029s
I20260709 13:04:02.172053 11425 raft_consensus.cc:697] T 0e5d327630e5438b878035d28efe6c0c P 0172fed22b6b4890a7fdc953fa128af4 [term 3 LEADER]: Becoming Leader. State: Replica: 0172fed22b6b4890a7fdc953fa128af4, State: Running, Role: LEADER
I20260709 13:04:02.107285 10774 rpcz_store.cc:276] 0709 13:03:58.943140 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:58.950232 (+  7092us) service_pool.cc:224] Handling call
0709 13:04:02.061789 (+3111557us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 13:04:00.491894 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 13:04:00.495037 (+  3143us) write_op.cc:432] Acquiring schema lock in shared mode
0709 13:04:00.495091 (+    54us) write_op.cc:435] Acquired schema lock
0709 13:04:00.495115 (+    24us) tablet.cc:662] Decoding operations
0709 13:04:00.495770 (+   655us) write_op.cc:620] Acquiring the partition lock for write op
0709 13:04:00.495875 (+   105us) write_op.cc:641] Partition lock acquired for write op
0709 13:04:00.495920 (+    45us) tablet.cc:685] Acquiring locks for 3 operations
0709 13:04:00.496310 (+   390us) tablet.cc:701] Row locks acquired
0709 13:04:00.496338 (+    28us) write_op.cc:260] PREPARE: finished
0709 13:04:00.496546 (+   208us) write_op.cc:270] Start()
0709 13:04:00.496679 (+   133us) write_op.cc:276] Timestamp: P: 1783602240496491 usec, L: 0
0709 13:04:00.496699 (+    20us) op_driver.cc:348] REPLICATION: starting
0709 13:04:00.497490 (+   791us) log.cc:844] Serialized 1316 byte log entry
0709 13:04:00.990928 (+493438us) op_driver.cc:464] REPLICATION: finished
0709 13:04:01.151227 (+160299us) write_op.cc:301] APPLY: starting
0709 13:04:01.152879 (+  1652us) tablet.cc:1367] starting BulkCheckPresence
0709 13:04:01.193279 (+ 40400us) tablet.cc:1370] finished BulkCheckPresence
0709 13:04:01.229082 (+ 35803us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 13:04:01.309073 (+ 79991us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 13:04:01.309312 (+   239us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=3
0709 13:04:01.309362 (+    50us) write_op.cc:312] APPLY: finished
0709 13:04:01.309624 (+   262us) log.cc:844] Serialized 34 byte log entry
0709 13:04:01.309849 (+   225us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:04:01.309964 (+   115us) write_op.cc:454] Released schema lock
0709 13:04:01.310011 (+    47us) write_op.cc:341] FINISH: Updating metrics
0709 13:04:01.310475 (+   464us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:04:01.310508 (+    33us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":4270,"reported_tablets":7,"child_traces":[["op",{"apply.queue_time_us":151025,"apply.run_cpu_time_us":3212,"apply.run_wall_time_us":164552,"num_ops":3,"prepare.queue_time_us":175670,"prepare.run_cpu_time_us":21738,"prepare.run_wall_time_us":102843,"raft.queue_time_us":279351,"raft.run_cpu_time_us":8032,"raft.run_wall_time_us":65041,"replication_time_us":494045,"thread_start_us":427911,"threads_started":4,"wal-append.queue_time_us":117529}]]}
I20260709 13:04:02.092392 11628 raft_consensus.cc:493] T 87cd8f5097624c8692f1d83544d8881e P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:02.193552 11069 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:02.188766 10844 leader_election.cc:304] T a610d1a71f52432e88fe56130f921fa2 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:04:02.173987 11024 raft_consensus.cc:2468] T 65bf0a62f4d84ef18dd6b23e082724b8 P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 3.
I20260709 13:04:02.132524 11748 raft_consensus.cc:2804] T d270b9e5481245d9b5340a454a7cd10b P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:04:02.225612 11149 raft_consensus.cc:359] T a5299b8d044f41b7b68057e63d17b159 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:02.176616 11510 raft_consensus.cc:740] T e5ba72ccd854430baadbc636dfd17136 P 873357742d5843579556ee8166f23f47 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 873357742d5843579556ee8166f23f47, State: Running, Role: LEADER
I20260709 13:04:02.179629 10953 raft_consensus.cc:3060] T 5de5f0581e3b41dea729363d6ca563bb P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:04:02.229564 11029 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46628 (request call id 123) took 1002 ms. Trace:
I20260709 13:04:02.260587 11029 rpcz_store.cc:276] 0709 13:04:01.237795 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:01.239933 (+  2138us) service_pool.cc:224] Handling call
0709 13:04:02.229257 (+989324us) spinlock_profiling.cc:243] Waited 929 ms on lock 0x7b5400465854. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:04:02.229466 (+   209us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2136236416}
I20260709 13:04:02.186832 11598 raft_consensus.cc:493] T bb7d67be500947b3aef4c4245e33671c P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:02.207543 10872 raft_consensus.cc:1240] T f371d2565dd84667b5613848604949bb P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Rejecting Update request from peer 873357742d5843579556ee8166f23f47 for earlier term 1. Current term is 2. Ops: []
I20260709 13:04:02.199631 11604 raft_consensus.cc:493] T 4b37f8fca8dc42718b2a42f0883a956f P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 10ca5ee68ad64b3283b9d235352c9d23)
I20260709 13:04:02.210259 11518 raft_consensus.cc:3055] T e5a5f5a608504650b48fd27f163faed6 P 186b12638a87410487467df721a3cbda [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:04:02.237649 10994 leader_election.cc:304] T b4cd3d7dfbbf4096908ff11e9dc57141 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:04:02.240021 11148 ts_tablet_manager.cc:1434] T 1e4f2241a94a41f38e481137f868b83d P 10ca5ee68ad64b3283b9d235352c9d23: Time spent starting tablet: real 1.170s	user 0.014s	sys 0.020s
I20260709 13:04:02.231684 10874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a9353d1d230416296627d7cf8b1d425" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
W20260709 13:04:02.269552 11148 ts_tablet_manager.cc:1467] T 1e4f2241a94a41f38e481137f868b83d P 10ca5ee68ad64b3283b9d235352c9d23: Tablet startup took 1199ms
W20260709 13:04:02.240687 10844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:45807, user_credentials={real_user=slave}} blocked reactor thread for 52038.1us
I20260709 13:04:02.174263 11724 raft_consensus.cc:515] T d371c5e27bd843469d2e2e453734cb4a P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:02.275043 10879 raft_consensus.cc:2468] T 34064dc7bf3747dbb3876dd824ee0688 P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 3.
I20260709 13:04:02.275609 11724 leader_election.cc:290] T d371c5e27bd843469d2e2e453734cb4a P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 2 election: Requested vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:02.281140 11595 consensus_queue.cc:1059] T a154cc8b181f460abf4880064379948b P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:04:02.171648 10950 rpcz_store.cc:276] 0709 13:04:01.055045 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:01.058537 (+  3492us) service_pool.cc:224] Handling call
0709 13:04:02.029083 (+970546us) spinlock_profiling.cc:243] Waited 3.88 ms on lock 0x7b5400414954. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:04:02.049057 (+ 19974us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":8935296}
I20260709 13:04:02.258741 11149 raft_consensus.cc:385] T a5299b8d044f41b7b68057e63d17b159 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:04:02.283617 11149 raft_consensus.cc:740] T a5299b8d044f41b7b68057e63d17b159 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 873357742d5843579556ee8166f23f47, State: Initialized, Role: FOLLOWER
I20260709 13:04:02.233755 11633 raft_consensus.cc:697] T 7709387f4d3d4f9992d97a0fbca39a40 P 873357742d5843579556ee8166f23f47 [term 3 LEADER]: Becoming Leader. State: Replica: 873357742d5843579556ee8166f23f47, State: Running, Role: LEADER
I20260709 13:04:02.201699 11735 consensus_queue.cc:1048] T 6a9353d1d230416296627d7cf8b1d425 P 186b12638a87410487467df721a3cbda [LEADER]: Connected to new peer: Peer: permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.042s
W20260709 13:04:02.267573 10921 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:38967, user_credentials={real_user=slave}} blocked reactor thread for 195504us
I20260709 13:04:02.272234 11425 consensus_queue.cc:237] T 0e5d327630e5438b878035d28efe6c0c P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:02.287348 10844 leader_election.cc:400] T 7709387f4d3d4f9992d97a0fbca39a40 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 2 election: Vote denied by peer 873357742d5843579556ee8166f23f47 (127.0.107.195:45807) with higher term. Message: Invalid argument: T 7709387f4d3d4f9992d97a0fbca39a40 P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 186b12638a87410487467df721a3cbda for earlier term 2. Current term is 3.
W20260709 13:04:02.230505 11067 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:45807, user_credentials={real_user=slave}} blocked reactor thread for 101050us
W20260709 13:04:02.292845 10994 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 55262.8us
W20260709 13:04:02.234068 10992 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 47079.2us
I20260709 13:04:02.252524 11748 raft_consensus.cc:493] T d270b9e5481245d9b5340a454a7cd10b P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:02.284761 11149 consensus_queue.cc:260] T a5299b8d044f41b7b68057e63d17b159 P 873357742d5843579556ee8166f23f47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:02.307428 10844 leader_election.cc:400] T 11de93cd9b9d457489ff1cb80f976c9a P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 2 pre-election: Vote denied by peer 873357742d5843579556ee8166f23f47 (127.0.107.195:45807) with higher term. Message: Invalid argument: T 11de93cd9b9d457489ff1cb80f976c9a P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 186b12638a87410487467df721a3cbda for earlier term 2. Current term is 3.
I20260709 13:04:02.256142 11743 raft_consensus.cc:2804] T a610d1a71f52432e88fe56130f921fa2 P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:04:02.314028 10844 leader_election.cc:403] T 11de93cd9b9d457489ff1cb80f976c9a P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
W20260709 13:04:02.269992 11148 ts_tablet_manager.cc:1469] T 1e4f2241a94a41f38e481137f868b83d P 10ca5ee68ad64b3283b9d235352c9d23: Trace:
0709 13:03:48.449633 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:48.450344 (+   711us) service_pool.cc:224] Handling call
0709 13:03:48.456433 (+  6089us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 13:03:48.456567 (+   134us) ts_tablet_manager.cc:682] Creating new metadata...
0709 13:03:48.500115 (+ 43548us) tablet_metadata.cc:665] Metadata flushed
0709 13:03:48.700291 (+200176us) tablet_replica.cc:206] Creating consensus instance
0709 13:03:48.700825 (+   534us) inbound_call.cc:177] Queueing success response
0709 13:04:00.516397 (+11815572us) ts_tablet_manager.cc:1379] Bootstrapping tablet
0709 13:04:01.070014 (+553617us) ts_tablet_manager.cc:1435] Starting tablet replica
0709 13:04:01.379579 (+309565us) tablet_replica.cc:261] Starting instrumentation
0709 13:04:01.385470 (+  5891us) tablet_replica.cc:280] Starting consensus
Metrics: {"mutex_wait_us":27906,"raft.queue_time_us":347984,"raft.run_cpu_time_us":87,"raft.run_wall_time_us":87,"tablet-open.queue_time_us":11815691,"thread_start_us":663925,"threads_started":1}
I20260709 13:04:02.268862 11518 raft_consensus.cc:740] T e5a5f5a608504650b48fd27f163faed6 P 186b12638a87410487467df721a3cbda [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 186b12638a87410487467df721a3cbda, State: Running, Role: LEADER
I20260709 13:04:02.290123 11506 raft_consensus.cc:493] T ff13ae9a47c7455f962406c6d131a3ff P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:02.251823 11628 raft_consensus.cc:515] T 87cd8f5097624c8692f1d83544d8881e P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:02.335215 11067 leader_election.cc:304] T 6a9353d1d230416296627d7cf8b1d425 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:04:02.335536 11545 leader_election.cc:290] T f371d2565dd84667b5613848604949bb P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 2 election: Requested vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:02.335004 10879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d371c5e27bd843469d2e2e453734cb4a" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:04:02.336714 10879 raft_consensus.cc:2393] T d371c5e27bd843469d2e2e453734cb4a P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 873357742d5843579556ee8166f23f47 in current term 2: Already voted for candidate 186b12638a87410487467df721a3cbda in this term.
I20260709 13:04:02.337495 11633 consensus_queue.cc:237] T 7709387f4d3d4f9992d97a0fbca39a40 P 873357742d5843579556ee8166f23f47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:02.341569 11697 leader_election.cc:290] T 8a69f32c53f24a50b7852d143665d7b4 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 3 election: Requested vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:02.344048 11149 ts_tablet_manager.cc:1434] T a5299b8d044f41b7b68057e63d17b159 P 873357742d5843579556ee8166f23f47: Time spent starting tablet: real 0.176s	user 0.009s	sys 0.002s
I20260709 13:04:02.345017 11595 raft_consensus.cc:493] T 57b6a2418c1342e7924195e9c1dbd357 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:02.345862 11149 tablet_bootstrap.cc:492] T 401a9cf11eac479aa445ec8c1d6e33b2 P 873357742d5843579556ee8166f23f47: Bootstrap starting.
I20260709 13:04:02.345721 11595 raft_consensus.cc:515] T 57b6a2418c1342e7924195e9c1dbd357 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:02.352367 11152 tablet_bootstrap.cc:492] T 75492f86fde046c3900df3eac8292fd4 P 0172fed22b6b4890a7fdc953fa128af4: No bootstrap required, opened a new log
I20260709 13:04:02.353114 11152 ts_tablet_manager.cc:1403] T 75492f86fde046c3900df3eac8292fd4 P 0172fed22b6b4890a7fdc953fa128af4: Time spent bootstrapping tablet: real 1.054s	user 0.042s	sys 0.050s
I20260709 13:04:02.353552 11151 tablet_bootstrap.cc:492] T 30f628e3d7914d609e7f1d980a74c654 P 186b12638a87410487467df721a3cbda: No bootstrap required, opened a new log
I20260709 13:04:02.354338 11151 ts_tablet_manager.cc:1403] T 30f628e3d7914d609e7f1d980a74c654 P 186b12638a87410487467df721a3cbda: Time spent bootstrapping tablet: real 1.034s	user 0.014s	sys 0.015s
W20260709 13:04:02.356786 11741 log.cc:927] Time spent T a154cc8b181f460abf4880064379948b P 0172fed22b6b4890a7fdc953fa128af4: Append to log took a long time: real 0.194s	user 0.003s	sys 0.000s
I20260709 13:04:02.315500 11748 raft_consensus.cc:3060] T d270b9e5481245d9b5340a454a7cd10b P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:04:02.360719 11067 leader_election.cc:304] T 6f46a5102fd64b45bf1c4dff863d2b4d P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47
I20260709 13:04:02.361049 11705 leader_election.cc:290] T e96e80ba9ac34d0998395515e5c8c41a P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 3 election: Requested vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
W20260709 13:04:02.363207 11737 log.cc:927] Time spent T f371d2565dd84667b5613848604949bb P 0172fed22b6b4890a7fdc953fa128af4: Append to log took a long time: real 0.202s	user 0.002s	sys 0.000s
I20260709 13:04:02.365792 11098 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37676 (request call id 137) took 1297 ms. Trace:
I20260709 13:04:02.315579 11743 raft_consensus.cc:493] T a610d1a71f52432e88fe56130f921fa2 P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:02.367406 11743 raft_consensus.cc:3060] T a610d1a71f52432e88fe56130f921fa2 P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:04:02.366331 11098 rpcz_store.cc:276] 0709 13:04:01.068620 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:01.668309 (+599689us) service_pool.cc:224] Handling call
0709 13:04:01.668746 (+   437us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:01.669177 (+   431us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:01.669208 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:01.669841 (+   633us) log.cc:844] Serialized 26 byte log entry
0709 13:04:02.043017 (+373176us) raft_consensus.cc:1645] Marking committed up to 1
0709 13:04:02.136343 (+ 93326us) log.cc:844] Serialized 14 byte log entry
0709 13:04:02.234464 (+ 98121us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:02.243263 (+  8799us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:02.362349 (+119086us) spinlock_profiling.cc:243] Waited 25.7 ms on lock 0x7b4400022b4c. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f4262f9bc85 00007f4262f9c6a2 00007f4265d0e1db 00007f4265d09aaf 00007f4265cfd86e 00007f4265cfcbb6 00007f4265cfc7d8 00007f4265d06221 00007f4265d061d9 00007f4265d06169
0709 13:04:02.365357 (+  3008us) raft_consensus.cc:1685] finished
0709 13:04:02.365511 (+   154us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:02.365709 (+   198us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":15008,"prepare.queue_time_us":182459,"prepare.run_cpu_time_us":52,"prepare.run_wall_time_us":52,"raft.queue_time_us":12591,"raft.run_cpu_time_us":311,"raft.run_wall_time_us":10855,"spinlock_wait_cycles":59089280,"thread_start_us":484775,"threads_started":3,"wal-append.queue_time_us":445764}
I20260709 13:04:02.369004 11100 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37676 (request call id 139) took 1073 ms. Trace:
I20260709 13:04:02.286924 11604 raft_consensus.cc:515] T 4b37f8fca8dc42718b2a42f0883a956f P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:02.370800 11758 raft_consensus.cc:2804] T 6a9353d1d230416296627d7cf8b1d425 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:04:02.371384 11758 raft_consensus.cc:493] T 6a9353d1d230416296627d7cf8b1d425 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 186b12638a87410487467df721a3cbda)
I20260709 13:04:02.371840 11758 raft_consensus.cc:3060] T 6a9353d1d230416296627d7cf8b1d425 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:04:02.372650 11604 leader_election.cc:290] T 4b37f8fca8dc42718b2a42f0883a956f P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:02.373447 11510 consensus_queue.cc:260] T e5ba72ccd854430baadbc636dfd17136 P 873357742d5843579556ee8166f23f47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:02.374831 11628 leader_election.cc:290] T 87cd8f5097624c8692f1d83544d8881e P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:02.375042 11518 consensus_queue.cc:260] T e5a5f5a608504650b48fd27f163faed6 P 186b12638a87410487467df721a3cbda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:02.376488 11518 raft_consensus.cc:3060] T e5a5f5a608504650b48fd27f163faed6 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Advancing to term 2
W20260709 13:04:02.378947 10992 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 52709.7us
I20260709 13:04:02.380739 10992 leader_election.cc:304] T 75492f86fde046c3900df3eac8292fd4 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47; no voters: 0172fed22b6b4890a7fdc953fa128af4
I20260709 13:04:02.381771 11592 consensus_queue.cc:1059] T a64b7e99d6c54934a7de6f6dffcf5e94 P 873357742d5843579556ee8166f23f47 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:04:02.382336 11691 raft_consensus.cc:2804] T 75492f86fde046c3900df3eac8292fd4 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:04:02.382848 11691 raft_consensus.cc:493] T 75492f86fde046c3900df3eac8292fd4 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:02.383193 11691 raft_consensus.cc:3060] T 75492f86fde046c3900df3eac8292fd4 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:04:02.303396 11656 raft_consensus.cc:2804] T b4cd3d7dfbbf4096908ff11e9dc57141 P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260709 13:04:02.386917 10844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:45807, user_credentials={real_user=slave}} blocked reactor thread for 89497.4us
I20260709 13:04:02.387763 11148 tablet_bootstrap.cc:492] T b67eff0634e540eab91cf8cee795969f P 10ca5ee68ad64b3283b9d235352c9d23: Bootstrap starting.
I20260709 13:04:02.369596 11100 rpcz_store.cc:276] 0709 13:04:01.295848 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:01.855995 (+560147us) service_pool.cc:224] Handling call
0709 13:04:02.073002 (+217007us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:02.136922 (+ 63920us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:02.368623 (+231701us) spinlock_profiling.cc:243] Waited 192 ms on lock 0x7b54006be9d0. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661618ad 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:04:02.368898 (+   275us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":442088832}
I20260709 13:04:02.390237 11069 leader_election.cc:304] T c51782cd9d064ec0949e84e24dd4de67 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda
I20260709 13:04:02.404775 11069 leader_election.cc:304] T d7ce47b9a92d43dd88d95bcda5038597 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda
I20260709 13:04:02.267289 11598 raft_consensus.cc:515] T bb7d67be500947b3aef4c4245e33671c P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:02.320456 11739 raft_consensus.cc:3055] T a154cc8b181f460abf4880064379948b P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:04:02.322679 11680 leader_election.cc:290] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:02.320729 11724 raft_consensus.cc:493] T 401a9cf11eac479aa445ec8c1d6e33b2 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:02.411242 11095 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37666 (request call id 123) took 1623 ms. Trace:
I20260709 13:04:02.417487 11029 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e4f2241a94a41f38e481137f868b83d" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
I20260709 13:04:02.332991 10947 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87cd8f5097624c8692f1d83544d8881e" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:04:02.385277 11695 leader_election.cc:290] T 63d44f6ba6434ab2870fe3190f4d7c2f P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:02.434505 11029 raft_consensus.cc:2393] T 1e4f2241a94a41f38e481137f868b83d P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 186b12638a87410487467df721a3cbda in current term 1: Already voted for candidate 873357742d5843579556ee8166f23f47 in this term.
I20260709 13:04:02.425148 11069 leader_election.cc:304] T a610d1a71f52432e88fe56130f921fa2 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda; no voters: 873357742d5843579556ee8166f23f47
I20260709 13:04:02.430018 10919 leader_election.cc:304] T 65bf0a62f4d84ef18dd6b23e082724b8 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:04:02.416664 11024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7839ca2d2c13431a85f44d7c458c6351" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:02.407823 11628 raft_consensus.cc:2749] T c51782cd9d064ec0949e84e24dd4de67 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 13:04:02.414709 11653 consensus_queue.cc:1175] T e5ba72ccd854430baadbc636dfd17136 P 873357742d5843579556ee8166f23f47 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "10ca5ee68ad64b3283b9d235352c9d23" responder_term: 2 status { last_received { term: 2 index: 2 } last_committed_idx: 1 last_received_current_leader { term: 2 index: 2 } }
I20260709 13:04:02.438868 11672 leader_election.cc:290] T a64b7e99d6c54934a7de6f6dffcf5e94 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 3 election: Requested vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:02.440435 10843 leader_election.cc:304] T a5299b8d044f41b7b68057e63d17b159 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda; no voters: 873357742d5843579556ee8166f23f47
I20260709 13:04:02.425529 11739 raft_consensus.cc:740] T a154cc8b181f460abf4880064379948b P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 10ca5ee68ad64b3283b9d235352c9d23, State: Running, Role: LEADER
W20260709 13:04:02.442822 11736 log.cc:927] Time spent T 5d439326f23747eaab3c1e7da2d6b785 P 186b12638a87410487467df721a3cbda: Append to log took a long time: real 0.275s	user 0.004s	sys 0.002s
W20260709 13:04:02.444267 10993 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:02.396970 11656 raft_consensus.cc:493] T b4cd3d7dfbbf4096908ff11e9dc57141 P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0172fed22b6b4890a7fdc953fa128af4)
I20260709 13:04:02.445115 11023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f371d2565dd84667b5613848604949bb" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
I20260709 13:04:02.445251 10946 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11de93cd9b9d457489ff1cb80f976c9a" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:04:02.445676 10949 raft_consensus.cc:1240] T d71ca3a1ba6b48478aaf62741bf3cf6a P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Rejecting Update request from peer 873357742d5843579556ee8166f23f47 for earlier term 1. Current term is 2. Ops: []
I20260709 13:04:02.463004 11585 raft_consensus.cc:3055] T 9a1716c0adf34bee8a4e99341ba5788f P 873357742d5843579556ee8166f23f47 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:04:02.406965 11595 leader_election.cc:290] T 57b6a2418c1342e7924195e9c1dbd357 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:02.413144 11510 raft_consensus.cc:3060] T e5ba72ccd854430baadbc636dfd17136 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:04:02.417052 11729 raft_consensus.cc:493] T 87cd8f5097624c8692f1d83544d8881e P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:02.314888 11553 consensus_queue.cc:237] T ff13ae9a47c7455f962406c6d131a3ff P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:02.480720 10955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4cd3d7dfbbf4096908ff11e9dc57141" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:02.556664 10955 raft_consensus.cc:2393] T b4cd3d7dfbbf4096908ff11e9dc57141 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 873357742d5843579556ee8166f23f47 in current term 2: Already voted for candidate 10ca5ee68ad64b3283b9d235352c9d23 in this term.
I20260709 13:04:02.455565 11680 raft_consensus.cc:2749] T d7ce47b9a92d43dd88d95bcda5038597 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:04:02.394923 11658 raft_consensus.cc:493] T 0f9df2bc281f4866b6da5da94a0defc7 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0172fed22b6b4890a7fdc953fa128af4)
I20260709 13:04:02.558852 11658 raft_consensus.cc:515] T 0f9df2bc281f4866b6da5da94a0defc7 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:02.559924 11103 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37676 (request call id 143) took 1119 ms. Trace:
I20260709 13:04:02.395918 11506 raft_consensus.cc:515] T ff13ae9a47c7455f962406c6d131a3ff P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:02.560580 11103 rpcz_store.cc:276] 0709 13:04:01.440894 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:02.173062 (+732168us) service_pool.cc:224] Handling call
0709 13:04:02.173450 (+   388us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:02.173847 (+   397us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:02.173877 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:02.181664 (+  7787us) log.cc:844] Serialized 26 byte log entry
0709 13:04:02.480024 (+298360us) raft_consensus.cc:1645] Marking committed up to 1
0709 13:04:02.480087 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:02.486262 (+  6175us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:02.559569 (+ 73307us) raft_consensus.cc:1685] finished
0709 13:04:02.559614 (+    45us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:02.559821 (+   207us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":27144,"raft.queue_time_us":141952,"raft.run_cpu_time_us":228,"raft.run_wall_time_us":10485,"thread_start_us":200494,"threads_started":1,"wal-append.queue_time_us":332802}
I20260709 13:04:02.483439 11151 raft_consensus.cc:359] T 30f628e3d7914d609e7f1d980a74c654 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:02.468178 11639 raft_consensus.cc:493] T c31881534c5e4f0089c82057c1746652 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:02.463490 10845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 51960.3us
I20260709 13:04:02.563637 11151 raft_consensus.cc:385] T 30f628e3d7914d609e7f1d980a74c654 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:04:02.564103 11151 raft_consensus.cc:740] T 30f628e3d7914d609e7f1d980a74c654 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 186b12638a87410487467df721a3cbda, State: Initialized, Role: FOLLOWER
I20260709 13:04:02.565062 11151 consensus_queue.cc:260] T 30f628e3d7914d609e7f1d980a74c654 P 186b12638a87410487467df721a3cbda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:02.482590 11695 raft_consensus.cc:2804] T 65bf0a62f4d84ef18dd6b23e082724b8 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:04:02.485773 11656 raft_consensus.cc:3060] T b4cd3d7dfbbf4096908ff11e9dc57141 P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:04:02.455273 11724 raft_consensus.cc:515] T 401a9cf11eac479aa445ec8c1d6e33b2 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:02.480242 11026 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d371c5e27bd843469d2e2e453734cb4a" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
I20260709 13:04:02.518587 11735 raft_consensus.cc:2804] T a5299b8d044f41b7b68057e63d17b159 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:04:02.562147 11658 leader_election.cc:290] T 0f9df2bc281f4866b6da5da94a0defc7 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:02.523293 10946 raft_consensus.cc:3060] T 11de93cd9b9d457489ff1cb80f976c9a P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:04:02.515550 11628 raft_consensus.cc:2804] T a610d1a71f52432e88fe56130f921fa2 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:04:02.519946 11688 leader_election.cc:290] T e96e80ba9ac34d0998395515e5c8c41a P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 2 election: Requested vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:02.515121 11585 raft_consensus.cc:740] T 9a1716c0adf34bee8a4e99341ba5788f P 873357742d5843579556ee8166f23f47 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 873357742d5843579556ee8166f23f47, State: Running, Role: LEADER
I20260709 13:04:02.567124 11100 raft_consensus.cc:1240] T e5ba72ccd854430baadbc636dfd17136 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Rejecting Update request from peer 873357742d5843579556ee8166f23f47 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260709 13:04:02.578406 10871 raft_consensus.cc:2468] T 11de93cd9b9d457489ff1cb80f976c9a P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 3.
I20260709 13:04:02.597864 11705 raft_consensus.cc:493] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:02.598678 10949 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b37f8fca8dc42718b2a42f0883a956f" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:02.568674 11729 raft_consensus.cc:515] T 87cd8f5097624c8692f1d83544d8881e P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:02.574659 11639 raft_consensus.cc:515] T c31881534c5e4f0089c82057c1746652 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:02.604943 11598 leader_election.cc:290] T bb7d67be500947b3aef4c4245e33671c P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:02.610533 11149 tablet_bootstrap.cc:654] T 401a9cf11eac479aa445ec8c1d6e33b2 P 873357742d5843579556ee8166f23f47: Neither blocks nor log segments found. Creating new log.
I20260709 13:04:02.577105 11723 consensus_queue.cc:1048] T c51782cd9d064ec0949e84e24dd4de67 P 186b12638a87410487467df721a3cbda [LEADER]: Connected to new peer: Peer: permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.078s
I20260709 13:04:02.614085 11148 tablet_bootstrap.cc:654] T b67eff0634e540eab91cf8cee795969f P 10ca5ee68ad64b3283b9d235352c9d23: Neither blocks nor log segments found. Creating new log.
I20260709 13:04:02.618796 11151 ts_tablet_manager.cc:1434] T 30f628e3d7914d609e7f1d980a74c654 P 186b12638a87410487467df721a3cbda: Time spent starting tablet: real 0.264s	user 0.006s	sys 0.004s
I20260709 13:04:02.573366 11607 raft_consensus.cc:2749] T 6f46a5102fd64b45bf1c4dff863d2b4d P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:04:02.673135 10879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b37f8fca8dc42718b2a42f0883a956f" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:02.633956 11595 raft_consensus.cc:493] T e5ba72ccd854430baadbc636dfd17136 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:02.673928 10879 raft_consensus.cc:2468] T 4b37f8fca8dc42718b2a42f0883a956f P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 1.
I20260709 13:04:02.674134 11595 raft_consensus.cc:515] T e5ba72ccd854430baadbc636dfd17136 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
W20260709 13:04:02.675225 11606 consensus_meta.cc:287] T e89e640169e14e2d98f968f76027daa7 P 873357742d5843579556ee8166f23f47: Time spent flushing consensus metadata: real 0.690s	user 0.012s	sys 0.002s
I20260709 13:04:02.675253 10946 raft_consensus.cc:2468] T 11de93cd9b9d457489ff1cb80f976c9a P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 3.
I20260709 13:04:02.676707 11595 leader_election.cc:290] T e5ba72ccd854430baadbc636dfd17136 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:02.634719 11680 raft_consensus.cc:493] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:02.677649 11680 raft_consensus.cc:515] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:02.679919 11680 leader_election.cc:290] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
W20260709 13:04:02.681891 11583 consensus_meta.cc:287] T 4c2a35bb99374e93b573cf870d04faaa P 873357742d5843579556ee8166f23f47: Time spent flushing consensus metadata: real 0.771s	user 0.012s	sys 0.000s
W20260709 13:04:02.636041 11745 log.cc:927] Time spent T c51782cd9d064ec0949e84e24dd4de67 P 10ca5ee68ad64b3283b9d235352c9d23: Append to log took a long time: real 0.477s	user 0.000s	sys 0.002s
I20260709 13:04:02.636046 11102 raft_consensus.cc:1240] T 8a69f32c53f24a50b7852d143665d7b4 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Rejecting Update request from peer 186b12638a87410487467df721a3cbda for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260709 13:04:02.637558 11026 raft_consensus.cc:2393] T d371c5e27bd843469d2e2e453734cb4a P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 186b12638a87410487467df721a3cbda in current term 2: Already voted for candidate 873357742d5843579556ee8166f23f47 in this term.
I20260709 13:04:02.639660 11682 consensus_queue.cc:1059] T a64b7e99d6c54934a7de6f6dffcf5e94 P 873357742d5843579556ee8166f23f47 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:04:02.641640 11610 leader_election.cc:290] T 8c160ddfe9b34bdaa3835d0915c1e06f P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
W20260709 13:04:02.644587 11760 log.cc:927] Time spent T 63d44f6ba6434ab2870fe3190f4d7c2f P 10ca5ee68ad64b3283b9d235352c9d23: Append to log took a long time: real 0.167s	user 0.000s	sys 0.000s
I20260709 13:04:02.647228 11673 raft_consensus.cc:3055] T a64b7e99d6c54934a7de6f6dffcf5e94 P 873357742d5843579556ee8166f23f47 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:04:02.702662 11673 raft_consensus.cc:740] T a64b7e99d6c54934a7de6f6dffcf5e94 P 873357742d5843579556ee8166f23f47 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 873357742d5843579556ee8166f23f47, State: Running, Role: LEADER
I20260709 13:04:02.703699 11673 consensus_queue.cc:260] T a64b7e99d6c54934a7de6f6dffcf5e94 P 873357742d5843579556ee8166f23f47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
W20260709 13:04:02.566980 10843 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 126655us
I20260709 13:04:02.704509 11583 raft_consensus.cc:515] T 4c2a35bb99374e93b573cf870d04faaa P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:02.705291 10843 leader_election.cc:304] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda; no voters: 
I20260709 13:04:02.587105 11735 raft_consensus.cc:493] T a5299b8d044f41b7b68057e63d17b159 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:02.706507 10843 leader_election.cc:304] T 4b37f8fca8dc42718b2a42f0883a956f P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 186b12638a87410487467df721a3cbda; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47
I20260709 13:04:02.706739 11735 raft_consensus.cc:3060] T a5299b8d044f41b7b68057e63d17b159 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Advancing to term 1
W20260709 13:04:02.707891 10843 leader_election.cc:343] T b67eff0634e540eab91cf8cee795969f P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:04:02.708370 10843 leader_election.cc:304] T b67eff0634e540eab91cf8cee795969f P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 186b12638a87410487467df721a3cbda; no voters: 0172fed22b6b4890a7fdc953fa128af4, 10ca5ee68ad64b3283b9d235352c9d23
I20260709 13:04:02.708575 11658 raft_consensus.cc:2804] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:04:02.709056 11658 raft_consensus.cc:493] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:02.709414 11658 raft_consensus.cc:3060] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:04:02.589150 11695 raft_consensus.cc:697] T 65bf0a62f4d84ef18dd6b23e082724b8 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 LEADER]: Becoming Leader. State: Replica: 10ca5ee68ad64b3283b9d235352c9d23, State: Running, Role: LEADER
I20260709 13:04:02.608448 11628 raft_consensus.cc:493] T a610d1a71f52432e88fe56130f921fa2 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:02.566830 10877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e96e80ba9ac34d0998395515e5c8c41a" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:04:02.632977 11103 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d156d6ae2ca4fa59a1e387edf0fea39" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:02.710671 11688 raft_consensus.cc:2749] T 4b37f8fca8dc42718b2a42f0883a956f P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:04:02.661470 10871 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51718 (request call id 171) took 1259 ms. Trace:
I20260709 13:04:02.714541 11583 leader_election.cc:290] T 4c2a35bb99374e93b573cf870d04faaa P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 3 election: Requested vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:02.711375 11695 consensus_queue.cc:237] T 65bf0a62f4d84ef18dd6b23e082724b8 P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:02.714825 10871 rpcz_store.cc:276] 0709 13:04:01.398815 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:01.437282 (+ 38467us) service_pool.cc:224] Handling call
0709 13:04:01.935091 (+497809us) spinlock_profiling.cc:243] Waited 3.85 ms on lock 0x7b5800000098. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f426303db71 00007f426303da22 00007f426bf56c09 00007f426b0b0ee0 00007f426b17a0ab 00007f426b17a131 00007f426b1280a0 00007f426b127984 00007f42646a26d4 00007f42646a2662
0709 13:04:02.657806 (+722715us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":8847232}
I20260709 13:04:02.711570 11729 leader_election.cc:290] T 87cd8f5097624c8692f1d83544d8881e P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:02.711578 11545 raft_consensus.cc:2749] T b67eff0634e540eab91cf8cee795969f P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:04:02.711647 11628 raft_consensus.cc:3060] T a610d1a71f52432e88fe56130f921fa2 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:04:02.591105 11739 consensus_queue.cc:260] T a154cc8b181f460abf4880064379948b P 10ca5ee68ad64b3283b9d235352c9d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:02.712026 10877 raft_consensus.cc:3060] T e96e80ba9ac34d0998395515e5c8c41a P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:04:02.719530 11739 raft_consensus.cc:3060] T a154cc8b181f460abf4880064379948b P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:04:02.712947 11103 raft_consensus.cc:2468] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 1.
I20260709 13:04:02.724002 10955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e4f2241a94a41f38e481137f868b83d" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:04:02.725128 10955 raft_consensus.cc:2393] T 1e4f2241a94a41f38e481137f868b83d P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 873357742d5843579556ee8166f23f47 in current term 1: Already voted for candidate 186b12638a87410487467df721a3cbda in this term.
I20260709 13:04:02.659101 11705 raft_consensus.cc:515] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:02.589887 11758 raft_consensus.cc:515] T 6a9353d1d230416296627d7cf8b1d425 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:02.731791 11758 leader_election.cc:290] T 6a9353d1d230416296627d7cf8b1d425 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 3 election: Requested vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:02.710366 10843 leader_election.cc:304] T 1e4f2241a94a41f38e481137f868b83d P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda; no voters: 
I20260709 13:04:02.739880 11723 raft_consensus.cc:2804] T 1e4f2241a94a41f38e481137f868b83d P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:04:02.740167 10843 leader_election.cc:304] T 4c2a35bb99374e93b573cf870d04faaa P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda; no voters: 873357742d5843579556ee8166f23f47
I20260709 13:04:02.659457 10953 raft_consensus.cc:2468] T 5de5f0581e3b41dea729363d6ca563bb P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 3.
I20260709 13:04:02.682852 10950 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e96e80ba9ac34d0998395515e5c8c41a" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:04:02.634684 11585 consensus_queue.cc:260] T 9a1716c0adf34bee8a4e99341ba5788f P 873357742d5843579556ee8166f23f47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:02.685433 11151 tablet_bootstrap.cc:492] T a1ec2b50c6ad4f7c8033476208abb26d P 186b12638a87410487467df721a3cbda: Bootstrap starting.
I20260709 13:04:02.676733 10946 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34064dc7bf3747dbb3876dd824ee0688" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:04:02.691552 11506 leader_election.cc:290] T ff13ae9a47c7455f962406c6d131a3ff P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:02.692278 10875 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51718 (request call id 172) took 1293 ms. Trace:
I20260709 13:04:02.696250 11639 leader_election.cc:290] T c31881534c5e4f0089c82057c1746652 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:02.653486 11731 raft_consensus.cc:2749] T 11de93cd9b9d457489ff1cb80f976c9a P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 873357742d5843579556ee8166f23f47 (127.0.107.195:45807) with higher term. Message: Invalid argument: T 11de93cd9b9d457489ff1cb80f976c9a P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 186b12638a87410487467df721a3cbda for earlier term 2. Current term is 3.
I20260709 13:04:02.697053 10947 raft_consensus.cc:3060] T 87cd8f5097624c8692f1d83544d8881e P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:04:02.702081 11102 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87cd8f5097624c8692f1d83544d8881e" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:04:02.792269 11102 raft_consensus.cc:3060] T 87cd8f5097624c8692f1d83544d8881e P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:04:02.740603 11723 raft_consensus.cc:697] T 1e4f2241a94a41f38e481137f868b83d P 186b12638a87410487467df721a3cbda [term 1 LEADER]: Becoming Leader. State: Replica: 186b12638a87410487467df721a3cbda, State: Running, Role: LEADER
I20260709 13:04:02.433866 11095 rpcz_store.cc:276] 0709 13:04:00.788041 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:00.790023 (+  1982us) service_pool.cc:224] Handling call
0709 13:04:01.812771 (+1022748us) spinlock_profiling.cc:243] Waited 2.2 ms on lock 0x7b5800117698. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f426303db71 00007f426303da22 00007f426bf56c09 00007f426b0b0ee0 00007f426b17a0ab 00007f426b17a131 00007f426b13bfc0 00007f426b126929 00007f426b1263d2 00007f426469dbd4
0709 13:04:01.813074 (+   303us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:01.813599 (+   525us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:01.813633 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:01.816769 (+  3136us) log.cc:844] Serialized 26 byte log entry
0709 13:04:02.012452 (+195683us) raft_consensus.cc:1645] Marking committed up to 0
0709 13:04:02.029833 (+ 17381us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:02.045590 (+ 15757us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:02.356114 (+310524us) spinlock_profiling.cc:243] Waited 32.9 ms on lock 0x7b4400022b4c. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f4262f9bc85 00007f4262f9c6a2 00007f4265d0e1db 00007f4265d09aaf 00007f4265cfd86e 00007f4265cfcbb6 00007f4265cfc7d8 00007f4265d06221 00007f4265d061d9 00007f4265d06169
0709 13:04:02.395839 (+ 39725us) raft_consensus.cc:1685] finished
0709 13:04:02.405928 (+ 10089us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:02.411146 (+  5218us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":618,"raft.run_cpu_time_us":106,"raft.run_wall_time_us":106,"spinlock_wait_cycles":80761600,"thread_start_us":177538,"threads_started":1,"wal-append.queue_time_us":319922}
W20260709 13:04:02.654213 11022 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.516s	user 0.016s	sys 0.024s
W20260709 13:04:02.775696 11066 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.592s	user 0.008s	sys 0.036s
W20260709 13:04:02.795089 11066 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.612s	user 0.008s	sys 0.036s
W20260709 13:04:02.795514 11066 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 612152us
I20260709 13:04:02.796371 11066 leader_election.cc:304] T b4cd3d7dfbbf4096908ff11e9dc57141 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 10ca5ee68ad64b3283b9d235352c9d23; no voters: 
I20260709 13:04:02.797829 11066 leader_election.cc:304] T 5de5f0581e3b41dea729363d6ca563bb P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 10ca5ee68ad64b3283b9d235352c9d23; no voters: 
I20260709 13:04:02.777082 10950 raft_consensus.cc:3060] T e96e80ba9ac34d0998395515e5c8c41a P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Advancing to term 3
W20260709 13:04:02.789978 11619 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.615s	user 0.018s	sys 0.019s
I20260709 13:04:02.779706 11688 raft_consensus.cc:2804] T 4c2a35bb99374e93b573cf870d04faaa P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Leader election won for term 2
W20260709 13:04:02.806591 10994 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 346505us
W20260709 13:04:02.793277 10919 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:45807, user_credentials={real_user=slave}} blocked reactor thread for 363403us
W20260709 13:04:02.802381 11022 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.668s	user 0.017s	sys 0.024s
I20260709 13:04:02.814466 10920 leader_election.cc:304] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 10ca5ee68ad64b3283b9d235352c9d23; no voters: 
I20260709 13:04:02.830229 10920 leader_election.cc:304] T 5d439326f23747eaab3c1e7da2d6b785 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 10ca5ee68ad64b3283b9d235352c9d23; no voters: 
I20260709 13:04:02.794550 11585 raft_consensus.cc:3060] T 9a1716c0adf34bee8a4e99341ba5788f P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:04:02.831554 11595 raft_consensus.cc:2804] T 5d439326f23747eaab3c1e7da2d6b785 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:04:02.832085 11595 raft_consensus.cc:493] T 5d439326f23747eaab3c1e7da2d6b785 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:02.832566 11595 raft_consensus.cc:3060] T 5d439326f23747eaab3c1e7da2d6b785 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:04:02.809620 11610 raft_consensus.cc:2804] T b4cd3d7dfbbf4096908ff11e9dc57141 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:04:02.843807 11610 raft_consensus.cc:493] T b4cd3d7dfbbf4096908ff11e9dc57141 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:02.844228 11610 raft_consensus.cc:3060] T b4cd3d7dfbbf4096908ff11e9dc57141 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:04:02.845090 11545 raft_consensus.cc:493] T 57b6a2418c1342e7924195e9c1dbd357 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:02.845325 10954 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87cd8f5097624c8692f1d83544d8881e" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:02.845741 11545 raft_consensus.cc:515] T 57b6a2418c1342e7924195e9c1dbd357 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:02.846676 11688 raft_consensus.cc:697] T 4c2a35bb99374e93b573cf870d04faaa P 186b12638a87410487467df721a3cbda [term 2 LEADER]: Becoming Leader. State: Replica: 186b12638a87410487467df721a3cbda, State: Running, Role: LEADER
I20260709 13:04:02.847597 10919 leader_election.cc:304] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23; no voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47
I20260709 13:04:02.847872 11688 consensus_queue.cc:237] T 4c2a35bb99374e93b573cf870d04faaa P 186b12638a87410487467df721a3cbda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:02.848387 11545 leader_election.cc:290] T 57b6a2418c1342e7924195e9c1dbd357 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:02.811619 10946 raft_consensus.cc:2393] T 34064dc7bf3747dbb3876dd824ee0688 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 873357742d5843579556ee8166f23f47 in current term 3: Already voted for candidate 10ca5ee68ad64b3283b9d235352c9d23 in this term.
W20260709 13:04:02.851951 11742 log.cc:927] Time spent T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 10ca5ee68ad64b3283b9d235352c9d23: Append to log took a long time: real 0.690s	user 0.003s	sys 0.000s
I20260709 13:04:02.852734 11639 raft_consensus.cc:3060] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:04:02.853257 10951 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49434 (request call id 125) took 1562 ms. Trace:
I20260709 13:04:02.853700 10951 rpcz_store.cc:276] 0709 13:04:01.290332 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:01.438205 (+147873us) service_pool.cc:224] Handling call
0709 13:04:01.883201 (+444996us) spinlock_profiling.cc:243] Waited 3.71 ms on lock 0x7b580001d198. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f426303db71 00007f426303deee 0000000000375939 00007f426b0b11e0 00007f426b17a0c4 00007f426b17a131 00007f426b13bfc0 00007f426b126929 00007f426b1263d2 00007f426469dbd4
0709 13:04:01.883457 (+   256us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:01.883916 (+   459us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:01.883951 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:01.884628 (+   677us) log.cc:844] Serialized 26 byte log entry
0709 13:04:02.007364 (+122736us) raft_consensus.cc:1645] Marking committed up to 1
0709 13:04:02.014565 (+  7201us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:02.038141 (+ 23576us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:02.812542 (+774401us) spinlock_profiling.cc:243] Waited 17.9 ms on lock 0x7b4400022b4c. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f4262f9bc85 00007f4262f9c6a2 00007f4265d0e1db 00007f4265d09aaf 00007f4265cfd86e 00007f4265cfcbb6 00007f4265cfc7d8 00007f4265d06221 00007f4265d061d9 00007f4265d06169
0709 13:04:02.853003 (+ 40461us) raft_consensus.cc:1685] finished
0709 13:04:02.853024 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:02.853183 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1249,"raft.run_cpu_time_us":154,"raft.run_wall_time_us":155,"spinlock_wait_cycles":49778432,"thread_start_us":102101,"threads_started":1,"wal-append.queue_time_us":225650}
I20260709 13:04:02.779520 10875 rpcz_store.cc:276] 0709 13:04:01.399137 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:01.437326 (+ 38189us) service_pool.cc:224] Handling call
0709 13:04:01.473835 (+ 36509us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:01.895396 (+421561us) spinlock_profiling.cc:243] Waited 14.3 ms on lock 0x7b4400022b4c. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f4262f9bc85 00007f4262f9c6a2 00007f4265d0e1db 00007f4265d09aaf 00007f42630580ac 00007f426beb2ab9 00007f426307469b 00007f426307293f 00007f4263072455 00007f426615a060
0709 13:04:02.084424 (+189028us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:02.093866 (+  9442us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:02.159592 (+ 65726us) log.cc:844] Serialized 26 byte log entry
0709 13:04:02.479908 (+320316us) raft_consensus.cc:1645] Marking committed up to 1
0709 13:04:02.486508 (+  6600us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:02.516231 (+ 29723us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:02.652386 (+136155us) raft_consensus.cc:1685] finished
0709 13:04:02.652416 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:02.692158 (+ 39742us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1861,"raft.queue_time_us":159268,"raft.run_cpu_time_us":178,"raft.run_wall_time_us":4159,"spinlock_wait_cycles":32860544,"thread_start_us":617238,"threads_started":2,"wal-append.queue_time_us":398032}
W20260709 13:04:02.859112 11101 consensus_meta.cc:287] T acc241e7273d49149984968596242712 P 0172fed22b6b4890a7fdc953fa128af4: Time spent flushing consensus metadata: real 0.883s	user 0.007s	sys 0.011s
I20260709 13:04:02.860366 11101 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37676 (request call id 134) took 1792 ms. Trace:
I20260709 13:04:02.860834 11101 rpcz_store.cc:276] 0709 13:04:01.067530 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:01.233507 (+165977us) service_pool.cc:224] Handling call
0709 13:04:01.861282 (+627775us) spinlock_profiling.cc:243] Waited 8.3 ms on lock 0x7b5800117698. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f426303db71 00007f426303da22 00007f426bf56c09 00007f426b0b0ee0 00007f426b17a0ab 00007f426b17a131 00007f426b13bfc0 00007f426b126929 00007f426b1263d2 00007f426469dbd4
0709 13:04:01.861616 (+   334us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:02.860013 (+998397us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:02.860287 (+   274us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":19078912}
W20260709 13:04:02.803053 10843 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 63021.2us
I20260709 13:04:02.844099 10949 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f1466124dbd4f6ebb3c4d62ca48bf96" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
W20260709 13:04:02.862996 11069 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:02.811820 11619 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.654s	user 0.019s	sys 0.020s
I20260709 13:04:02.840238 10955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a64b7e99d6c54934a7de6f6dffcf5e94" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:04:02.861737 11691 raft_consensus.cc:515] T 75492f86fde046c3900df3eac8292fd4 P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:02.815821 11758 raft_consensus.cc:2804] T 5de5f0581e3b41dea729363d6ca563bb P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:04:02.965103 10955 raft_consensus.cc:2393] T a64b7e99d6c54934a7de6f6dffcf5e94 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 0172fed22b6b4890a7fdc953fa128af4 in current term 3: Already voted for candidate 10ca5ee68ad64b3283b9d235352c9d23 in this term.
W20260709 13:04:02.874210 11743 consensus_meta.cc:287] T a610d1a71f52432e88fe56130f921fa2 P 186b12638a87410487467df721a3cbda: Time spent flushing consensus metadata: real 0.506s	user 0.008s	sys 0.008s
W20260709 13:04:02.866107 11748 consensus_meta.cc:287] T d270b9e5481245d9b5340a454a7cd10b P 10ca5ee68ad64b3283b9d235352c9d23: Time spent flushing consensus metadata: real 0.507s	user 0.012s	sys 0.000s
I20260709 13:04:02.867627 11639 raft_consensus.cc:2749] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:04:02.864172 11734 consensus_queue.cc:1048] T acc241e7273d49149984968596242712 P 873357742d5843579556ee8166f23f47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.115s
I20260709 13:04:02.841020 11757 raft_consensus.cc:493] T d71ca3a1ba6b48478aaf62741bf3cf6a P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:02.863303 11705 leader_election.cc:290] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
W20260709 13:04:02.992575 10995 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.538s	user 0.004s	sys 0.008s
I20260709 13:04:02.988667 10873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65bf0a62f4d84ef18dd6b23e082724b8" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:04:02.866099 11656 raft_consensus.cc:515] T b4cd3d7dfbbf4096908ff11e9dc57141 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:02.830381 11103 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d371c5e27bd843469d2e2e453734cb4a" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:04:02.873418 11096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb7d67be500947b3aef4c4245e33671c" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:02.984521 10878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34064dc7bf3747dbb3876dd824ee0688" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:04:02.866273 11754 raft_consensus.cc:493] T acc241e7273d49149984968596242712 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:02.994356 10873 raft_consensus.cc:3060] T 65bf0a62f4d84ef18dd6b23e082724b8 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Advancing to term 3
W20260709 13:04:02.873695 11597 consensus_meta.cc:287] T 836251ebc2b74f5da88b8a8b2338e2a5 P 10ca5ee68ad64b3283b9d235352c9d23: Time spent flushing consensus metadata: real 0.709s	user 0.011s	sys 0.003s
I20260709 13:04:02.877533 11729 raft_consensus.cc:2804] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:04:02.862375 11101 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a69f32c53f24a50b7852d143665d7b4" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
W20260709 13:04:02.992986 10995 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.538s	user 0.004s	sys 0.008s
I20260709 13:04:03.020686 10878 raft_consensus.cc:2393] T 34064dc7bf3747dbb3876dd824ee0688 P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 10ca5ee68ad64b3283b9d235352c9d23 in current term 3: Already voted for candidate 873357742d5843579556ee8166f23f47 in this term.
I20260709 13:04:02.989464 11724 leader_election.cc:290] T 401a9cf11eac479aa445ec8c1d6e33b2 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:02.984719 11758 raft_consensus.cc:697] T 5de5f0581e3b41dea729363d6ca563bb P 0172fed22b6b4890a7fdc953fa128af4 [term 3 LEADER]: Becoming Leader. State: Replica: 0172fed22b6b4890a7fdc953fa128af4, State: Running, Role: LEADER
I20260709 13:04:02.995471 10879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72b111ad7b5f4753ab1d2b5ee1114840" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:02.875566 11518 consensus_queue.cc:1059] T 8a69f32c53f24a50b7852d143665d7b4 P 186b12638a87410487467df721a3cbda [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.153s
I20260709 13:04:02.874275 11723 consensus_queue.cc:237] T 1e4f2241a94a41f38e481137f868b83d P 186b12638a87410487467df721a3cbda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:03.089607 10878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7839ca2d2c13431a85f44d7c458c6351" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:03.059188 11639 raft_consensus.cc:493] T 732c4adeeff6422497aa7db4a82f535b P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:02.984972 11743 raft_consensus.cc:515] T a610d1a71f52432e88fe56130f921fa2 P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:03.068977 11508 raft_consensus.cc:493] T d7ce47b9a92d43dd88d95bcda5038597 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 10ca5ee68ad64b3283b9d235352c9d23)
I20260709 13:04:03.064623 11779 raft_consensus.cc:493] T 32147c6d2d5a477aa38c597c7f78df5b P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:03.058647 11729 raft_consensus.cc:493] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:03.096791 10995 leader_election.cc:304] T 34064dc7bf3747dbb3876dd824ee0688 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47; no voters: 
W20260709 13:04:03.105822 11069 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:38967, user_credentials={real_user=slave}} blocked reactor thread for 149813us
I20260709 13:04:03.124938 10995 leader_election.cc:304] T d371c5e27bd843469d2e2e453734cb4a P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 873357742d5843579556ee8166f23f47; no voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda
I20260709 13:04:03.108904 11101 raft_consensus.cc:2393] T 8a69f32c53f24a50b7852d143665d7b4 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 10ca5ee68ad64b3283b9d235352c9d23 in current term 3: Already voted for candidate 0172fed22b6b4890a7fdc953fa128af4 in this term.
I20260709 13:04:03.123186 11152 raft_consensus.cc:359] T 75492f86fde046c3900df3eac8292fd4 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:03.094376 11724 raft_consensus.cc:493] T 2567c2d84e4648d7ba5dc23a3df32817 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 10ca5ee68ad64b3283b9d235352c9d23)
I20260709 13:04:03.120105 10878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63d44f6ba6434ab2870fe3190f4d7c2f" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
W20260709 13:04:03.104315 11510 consensus_meta.cc:287] T e5ba72ccd854430baadbc636dfd17136 P 873357742d5843579556ee8166f23f47: Time spent flushing consensus metadata: real 0.549s	user 0.008s	sys 0.011s
I20260709 13:04:03.065604 11597 raft_consensus.cc:515] T 836251ebc2b74f5da88b8a8b2338e2a5 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:03.129585 11152 raft_consensus.cc:385] T 75492f86fde046c3900df3eac8292fd4 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:04:03.134465 11152 raft_consensus.cc:740] T 75492f86fde046c3900df3eac8292fd4 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0172fed22b6b4890a7fdc953fa128af4, State: Initialized, Role: FOLLOWER
I20260709 13:04:03.128743 11425 raft_consensus.cc:493] T e5a5f5a608504650b48fd27f163faed6 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:02.992676 11748 raft_consensus.cc:515] T d270b9e5481245d9b5340a454a7cd10b P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:03.115046 11508 raft_consensus.cc:515] T d7ce47b9a92d43dd88d95bcda5038597 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:03.136205 11425 raft_consensus.cc:515] T e5a5f5a608504650b48fd27f163faed6 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:03.098237 11777 raft_consensus.cc:493] T 6f46a5102fd64b45bf1c4dff863d2b4d P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:03.022159 11757 raft_consensus.cc:515] T d71ca3a1ba6b48478aaf62741bf3cf6a P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:03.090519 10879 raft_consensus.cc:2468] T 72b111ad7b5f4753ab1d2b5ee1114840 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 2.
I20260709 13:04:03.103886 11780 raft_consensus.cc:493] T a5299b8d044f41b7b68057e63d17b159 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:03.135627 11096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c31881534c5e4f0089c82057c1746652" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:03.135380 11152 consensus_queue.cc:260] T 75492f86fde046c3900df3eac8292fd4 P 0172fed22b6b4890a7fdc953fa128af4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:03.122229 11729 raft_consensus.cc:3060] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:04:03.054661 11754 raft_consensus.cc:515] T acc241e7273d49149984968596242712 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
W20260709 13:04:03.139448 11770 log.cc:927] Time spent T e5a5f5a608504650b48fd27f163faed6 P 10ca5ee68ad64b3283b9d235352c9d23: Append to log took a long time: real 0.073s	user 0.000s	sys 0.000s
I20260709 13:04:03.180787 10878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a69f32c53f24a50b7852d143665d7b4" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
W20260709 13:04:03.139448 11766 log.cc:927] Time spent T 525bfdbd6d934afd823f7d3ee3f9d052 P 873357742d5843579556ee8166f23f47: Append to log took a long time: real 0.074s	user 0.000s	sys 0.000s
I20260709 13:04:03.107707 11639 raft_consensus.cc:515] T 732c4adeeff6422497aa7db4a82f535b P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:03.138269 11748 leader_election.cc:290] T d270b9e5481245d9b5340a454a7cd10b P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 3 election: Requested vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:03.143692 11583 raft_consensus.cc:493] T 2567c2d84e4648d7ba5dc23a3df32817 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 10ca5ee68ad64b3283b9d235352c9d23)
I20260709 13:04:03.141192 11101 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f1466124dbd4f6ebb3c4d62ca48bf96" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:03.140060 11026 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63d44f6ba6434ab2870fe3190f4d7c2f" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:03.279176 11101 raft_consensus.cc:2468] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 3.
I20260709 13:04:03.280627 11101 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d270b9e5481245d9b5340a454a7cd10b" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:04:03.281456 11101 raft_consensus.cc:3060] T d270b9e5481245d9b5340a454a7cd10b P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Advancing to term 3
W20260709 13:04:03.170421 10843 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 305763us
I20260709 13:04:03.232024 11680 raft_consensus.cc:493] T f371d2565dd84667b5613848604949bb P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:03.282922 11680 raft_consensus.cc:515] T f371d2565dd84667b5613848604949bb P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:03.171731 11777 raft_consensus.cc:515] T 6f46a5102fd64b45bf1c4dff863d2b4d P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:03.286506 11777 leader_election.cc:290] T 6f46a5102fd64b45bf1c4dff863d2b4d P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:03.232568 10878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d156d6ae2ca4fa59a1e387edf0fea39" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:03.232806 11639 leader_election.cc:290] T 732c4adeeff6422497aa7db4a82f535b P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:03.233263 10879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d270b9e5481245d9b5340a454a7cd10b" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:03.236054 11151 tablet_bootstrap.cc:654] T a1ec2b50c6ad4f7c8033476208abb26d P 186b12638a87410487467df721a3cbda: Neither blocks nor log segments found. Creating new log.
W20260709 13:04:03.238332 10993 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:03.238591 11705 consensus_queue.cc:1048] T 8c160ddfe9b34bdaa3835d0915c1e06f P 873357742d5843579556ee8166f23f47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.011s
I20260709 13:04:03.143239 11758 consensus_queue.cc:237] T 5de5f0581e3b41dea729363d6ca563bb P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:03.240789 11691 leader_election.cc:290] T 75492f86fde046c3900df3eac8292fd4 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 1 election: Requested vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:03.166245 11724 raft_consensus.cc:515] T 2567c2d84e4648d7ba5dc23a3df32817 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:03.242748 11656 leader_election.cc:290] T b4cd3d7dfbbf4096908ff11e9dc57141 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 2 election: Requested vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:03.244637 11148 tablet_bootstrap.cc:492] T b67eff0634e540eab91cf8cee795969f P 10ca5ee68ad64b3283b9d235352c9d23: No bootstrap required, opened a new log
I20260709 13:04:03.327004 11148 ts_tablet_manager.cc:1403] T b67eff0634e540eab91cf8cee795969f P 10ca5ee68ad64b3283b9d235352c9d23: Time spent bootstrapping tablet: real 0.940s	user 0.016s	sys 0.016s
W20260709 13:04:03.245061 11735 consensus_meta.cc:287] T a5299b8d044f41b7b68057e63d17b159 P 186b12638a87410487467df721a3cbda: Time spent flushing consensus metadata: real 0.538s	user 0.009s	sys 0.004s
I20260709 13:04:03.238587 10947 raft_consensus.cc:2468] T 87cd8f5097624c8692f1d83544d8881e P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 3.
W20260709 13:04:03.246050 11658 consensus_meta.cc:287] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 186b12638a87410487467df721a3cbda: Time spent flushing consensus metadata: real 0.536s	user 0.025s	sys 0.000s
I20260709 13:04:03.175417 11780 raft_consensus.cc:515] T a5299b8d044f41b7b68057e63d17b159 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:03.121062 11779 raft_consensus.cc:515] T 32147c6d2d5a477aa38c597c7f78df5b P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:03.191841 11734 raft_consensus.cc:2804] T 34064dc7bf3747dbb3876dd824ee0688 P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:04:03.211318 11510 raft_consensus.cc:2749] T d371c5e27bd843469d2e2e453734cb4a P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 13:04:03.189214 11096 raft_consensus.cc:2468] T c31881534c5e4f0089c82057c1746652 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 0.
W20260709 13:04:03.289413 10994 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 162617us
I20260709 13:04:03.273514 11583 raft_consensus.cc:515] T 2567c2d84e4648d7ba5dc23a3df32817 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:03.240082 11097 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37676 (request call id 144) took 1229 ms. Trace:
W20260709 13:04:03.295851 11102 consensus_meta.cc:287] T 87cd8f5097624c8692f1d83544d8881e P 0172fed22b6b4890a7fdc953fa128af4: Time spent flushing consensus metadata: real 0.501s	user 0.013s	sys 0.000s
I20260709 13:04:03.343523 11102 raft_consensus.cc:2468] T 87cd8f5097624c8692f1d83544d8881e P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 3.
I20260709 13:04:03.299372 11508 leader_election.cc:290] T d7ce47b9a92d43dd88d95bcda5038597 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:03.302583 11597 leader_election.cc:290] T 836251ebc2b74f5da88b8a8b2338e2a5 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 3 election: Requested vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
W20260709 13:04:03.303571 10826 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 13:04:03.303853 10879 raft_consensus.cc:2468] T d270b9e5481245d9b5340a454a7cd10b P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 1.
I20260709 13:04:03.305497 11425 leader_election.cc:290] T e5a5f5a608504650b48fd27f163faed6 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
W20260709 13:04:03.375464 10995 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:38967, user_credentials={real_user=slave}} blocked reactor thread for 250658us
I20260709 13:04:03.376519 10995 leader_election.cc:304] T 6a9353d1d230416296627d7cf8b1d425 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 873357742d5843579556ee8166f23f47; no voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda
I20260709 13:04:03.376994 10994 leader_election.cc:304] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:04:03.336206 10949 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5a5f5a608504650b48fd27f163faed6" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:03.339936 11518 raft_consensus.cc:3055] T 8a69f32c53f24a50b7852d143665d7b4 P 186b12638a87410487467df721a3cbda [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:04:03.382601 11772 raft_consensus.cc:493] T a154cc8b181f460abf4880064379948b P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 10ca5ee68ad64b3283b9d235352c9d23)
I20260709 13:04:03.385285 11152 ts_tablet_manager.cc:1434] T 75492f86fde046c3900df3eac8292fd4 P 0172fed22b6b4890a7fdc953fa128af4: Time spent starting tablet: real 1.032s	user 0.014s	sys 0.008s
W20260709 13:04:03.385649 11152 ts_tablet_manager.cc:1467] T 75492f86fde046c3900df3eac8292fd4 P 0172fed22b6b4890a7fdc953fa128af4: Tablet startup took 1032ms
I20260709 13:04:03.389539 11102 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "836251ebc2b74f5da88b8a8b2338e2a5" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:04:03.344904 11734 raft_consensus.cc:697] T 34064dc7bf3747dbb3876dd824ee0688 P 873357742d5843579556ee8166f23f47 [term 3 LEADER]: Becoming Leader. State: Replica: 873357742d5843579556ee8166f23f47, State: Running, Role: LEADER
I20260709 13:04:03.333022 11658 raft_consensus.cc:515] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:03.333881 11735 raft_consensus.cc:515] T a5299b8d044f41b7b68057e63d17b159 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:03.394279 10920 leader_election.cc:304] T c31881534c5e4f0089c82057c1746652 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 10ca5ee68ad64b3283b9d235352c9d23; no voters: 
I20260709 13:04:03.380088 11656 raft_consensus.cc:493] T e89e640169e14e2d98f968f76027daa7 P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:03.380312 11680 leader_election.cc:290] T f371d2565dd84667b5613848604949bb P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
W20260709 13:04:03.383543 11595 consensus_meta.cc:287] T 5d439326f23747eaab3c1e7da2d6b785 P 10ca5ee68ad64b3283b9d235352c9d23: Time spent flushing consensus metadata: real 0.550s	user 0.009s	sys 0.007s
I20260709 13:04:03.387952 11425 raft_consensus.cc:493] T 836251ebc2b74f5da88b8a8b2338e2a5 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:03.383724 11510 raft_consensus.cc:2749] T 6a9353d1d230416296627d7cf8b1d425 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:04:03.398169 10995 leader_election.cc:304] T 11de93cd9b9d457489ff1cb80f976c9a P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:04:03.384958 11518 raft_consensus.cc:740] T 8a69f32c53f24a50b7852d143665d7b4 P 186b12638a87410487467df721a3cbda [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 186b12638a87410487467df721a3cbda, State: Running, Role: LEADER
I20260709 13:04:03.405982 11796 raft_consensus.cc:493] T 836251ebc2b74f5da88b8a8b2338e2a5 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0172fed22b6b4890a7fdc953fa128af4)
I20260709 13:04:03.412083 11425 raft_consensus.cc:515] T 836251ebc2b74f5da88b8a8b2338e2a5 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:03.387116 11508 raft_consensus.cc:493] T e89e640169e14e2d98f968f76027daa7 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:03.414721 11508 raft_consensus.cc:515] T e89e640169e14e2d98f968f76027daa7 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:03.415868 11425 leader_election.cc:290] T 836251ebc2b74f5da88b8a8b2338e2a5 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:03.384711 11022 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46628 (request call id 128) took 1458 ms. Trace:
W20260709 13:04:03.386121 11152 ts_tablet_manager.cc:1469] T 75492f86fde046c3900df3eac8292fd4 P 0172fed22b6b4890a7fdc953fa128af4: Trace:
0709 13:03:48.325585 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:48.327233 (+  1648us) service_pool.cc:224] Handling call
0709 13:03:48.337905 (+ 10672us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 13:03:48.338013 (+   108us) ts_tablet_manager.cc:682] Creating new metadata...
0709 13:03:48.453894 (+115881us) tablet_metadata.cc:665] Metadata flushed
0709 13:03:48.735317 (+281423us) tablet_replica.cc:206] Creating consensus instance
0709 13:03:48.737010 (+  1693us) inbound_call.cc:177] Queueing success response
0709 13:04:01.270952 (+12533942us) ts_tablet_manager.cc:1379] Bootstrapping tablet
0709 13:04:02.353431 (+1082479us) ts_tablet_manager.cc:1435] Starting tablet replica
0709 13:04:03.062038 (+708607us) tablet_replica.cc:261] Starting instrumentation
0709 13:04:03.075213 (+ 13175us) tablet_replica.cc:280] Starting consensus
Metrics: {"mutex_wait_us":700705,"raft.queue_time_us":58029,"raft.run_cpu_time_us":261,"raft.run_wall_time_us":264,"tablet-open.queue_time_us":12533988,"thread_start_us":159271,"threads_started":1}
I20260709 13:04:03.409775 11780 leader_election.cc:290] T a5299b8d044f41b7b68057e63d17b159 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:03.403179 11597 raft_consensus.cc:2804] T c31881534c5e4f0089c82057c1746652 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:04:03.433894 11029 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb7d67be500947b3aef4c4245e33671c" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:03.434294 11597 raft_consensus.cc:493] T c31881534c5e4f0089c82057c1746652 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:03.434755 11597 raft_consensus.cc:3060] T c31881534c5e4f0089c82057c1746652 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:04:03.434855 11027 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "732c4adeeff6422497aa7db4a82f535b" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:03.392644 11800 raft_consensus.cc:2804] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:04:03.417726 10878 raft_consensus.cc:2393] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 10ca5ee68ad64b3283b9d235352c9d23 in current term 2: Already voted for candidate 186b12638a87410487467df721a3cbda in this term.
I20260709 13:04:03.418887 11102 raft_consensus.cc:3060] T 836251ebc2b74f5da88b8a8b2338e2a5 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:04:03.417026 11148 raft_consensus.cc:359] T b67eff0634e540eab91cf8cee795969f P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:03.390857 11772 raft_consensus.cc:515] T a154cc8b181f460abf4880064379948b P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:03.425911 11096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f46a5102fd64b45bf1c4dff863d2b4d" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:03.426714 11425 raft_consensus.cc:493] T 75492f86fde046c3900df3eac8292fd4 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:03.427776 11774 raft_consensus.cc:2804] T 11de93cd9b9d457489ff1cb80f976c9a P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:04:03.461992 11774 raft_consensus.cc:697] T 11de93cd9b9d457489ff1cb80f976c9a P 873357742d5843579556ee8166f23f47 [term 3 LEADER]: Becoming Leader. State: Replica: 873357742d5843579556ee8166f23f47, State: Running, Role: LEADER
I20260709 13:04:03.463133 11026 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d71ca3a1ba6b48478aaf62741bf3cf6a" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
I20260709 13:04:03.463044 11774 consensus_queue.cc:237] T 11de93cd9b9d457489ff1cb80f976c9a P 873357742d5843579556ee8166f23f47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:03.464607 11103 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5ba72ccd854430baadbc636dfd17136" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:03.464871 11680 raft_consensus.cc:493] T 868e92339c2f4508af0afbd07c814d64 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:03.465462 11103 raft_consensus.cc:2393] T e5ba72ccd854430baadbc636dfd17136 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 10ca5ee68ad64b3283b9d235352c9d23 in current term 3: Already voted for candidate 0172fed22b6b4890a7fdc953fa128af4 in this term.
I20260709 13:04:03.428613 10874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57b6a2418c1342e7924195e9c1dbd357" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:03.465456 11680 raft_consensus.cc:515] T 868e92339c2f4508af0afbd07c814d64 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:03.430388 11658 leader_election.cc:290] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 2 election: Requested vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:03.468103 11658 raft_consensus.cc:493] T 8c160ddfe9b34bdaa3835d0915c1e06f P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:03.468588 11098 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37666 (request call id 134) took 1399 ms. Trace:
I20260709 13:04:03.432158 11025 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5ba72ccd854430baadbc636dfd17136" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:03.468966 10919 leader_election.cc:304] T bb7d67be500947b3aef4c4245e33671c P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23; no voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47
I20260709 13:04:03.469342 11025 raft_consensus.cc:2468] T e5ba72ccd854430baadbc636dfd17136 P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 3.
I20260709 13:04:03.469931 11772 leader_election.cc:290] T a154cc8b181f460abf4880064379948b P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:03.432997 11023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f46a5102fd64b45bf1c4dff863d2b4d" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:03.471720 11680 leader_election.cc:290] T 868e92339c2f4508af0afbd07c814d64 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:03.445282 11148 raft_consensus.cc:385] T b67eff0634e540eab91cf8cee795969f P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:04:03.471518 11639 raft_consensus.cc:2749] T bb7d67be500947b3aef4c4245e33671c P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:04:03.472512 11148 raft_consensus.cc:740] T b67eff0634e540eab91cf8cee795969f P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10ca5ee68ad64b3283b9d235352c9d23, State: Initialized, Role: FOLLOWER
W20260709 13:04:03.472823 10992 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:03.417528 10879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d439326f23747eaab3c1e7da2d6b785" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:03.473341 11425 raft_consensus.cc:515] T 75492f86fde046c3900df3eac8292fd4 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:03.474360 10879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5de5f0581e3b41dea729363d6ca563bb" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:04:03.475028 10879 raft_consensus.cc:3060] T 5de5f0581e3b41dea729363d6ca563bb P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:04:03.475579 11425 leader_election.cc:290] T 75492f86fde046c3900df3eac8292fd4 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:03.469007 11098 rpcz_store.cc:276] 0709 13:04:02.069416 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:02.407292 (+337876us) service_pool.cc:224] Handling call
0709 13:04:02.466707 (+ 59415us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:02.578134 (+111427us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:02.583284 (+  5150us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:02.588996 (+  5712us) raft_consensus.cc:1645] Marking committed up to 1
0709 13:04:02.603684 (+ 14688us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:03.468324 (+864640us) spinlock_profiling.cc:243] Waited 100 ms on lock 0x7b54001054d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:04:03.468388 (+    64us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:03.468521 (+   133us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":229976832}
I20260709 13:04:03.473485 11148 consensus_queue.cc:260] T b67eff0634e540eab91cf8cee795969f P 10ca5ee68ad64b3283b9d235352c9d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:03.468555 11658 raft_consensus.cc:515] T 8c160ddfe9b34bdaa3835d0915c1e06f P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:03.479156 11724 leader_election.cc:290] T 2567c2d84e4648d7ba5dc23a3df32817 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:03.481099 11779 leader_election.cc:290] T 32147c6d2d5a477aa38c597c7f78df5b P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:03.395823 11592 consensus_queue.cc:1059] T f371d2565dd84667b5613848604949bb P 873357742d5843579556ee8166f23f47 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.029s
I20260709 13:04:03.482072 10955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65bf0a62f4d84ef18dd6b23e082724b8" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
W20260709 13:04:03.446949 10921 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:03.416680 10873 raft_consensus.cc:2468] T 65bf0a62f4d84ef18dd6b23e082724b8 P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 3.
I20260709 13:04:03.411129 11656 raft_consensus.cc:515] T e89e640169e14e2d98f968f76027daa7 P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:03.452803 11800 raft_consensus.cc:493] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:03.456333 11806 consensus_queue.cc:1059] T 8a69f32c53f24a50b7852d143665d7b4 P 186b12638a87410487467df721a3cbda [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:04:03.413883 11796 raft_consensus.cc:515] T 836251ebc2b74f5da88b8a8b2338e2a5 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:03.456995 11096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4cd3d7dfbbf4096908ff11e9dc57141" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
W20260709 13:04:03.459568 11628 consensus_meta.cc:287] T a610d1a71f52432e88fe56130f921fa2 P 0172fed22b6b4890a7fdc953fa128af4: Time spent flushing consensus metadata: real 0.741s	user 0.004s	sys 0.019s
I20260709 13:04:03.495097 11633 raft_consensus.cc:3055] T f371d2565dd84667b5613848604949bb P 873357742d5843579556ee8166f23f47 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:04:03.495275 11628 raft_consensus.cc:515] T a610d1a71f52432e88fe56130f921fa2 P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:03.496541 11796 leader_election.cc:290] T 836251ebc2b74f5da88b8a8b2338e2a5 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:03.497548 11656 leader_election.cc:290] T e89e640169e14e2d98f968f76027daa7 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:03.497679 11628 leader_election.cc:290] T a610d1a71f52432e88fe56130f921fa2 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 4 election: Requested vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:03.491791 11800 raft_consensus.cc:3060] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:04:03.498468 11098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d7ce47b9a92d43dd88d95bcda5038597" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:03.499991 11098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "401a9cf11eac479aa445ec8c1d6e33b2" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:03.501171 11098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75492f86fde046c3900df3eac8292fd4" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:04:03.501334 11148 ts_tablet_manager.cc:1434] T b67eff0634e540eab91cf8cee795969f P 10ca5ee68ad64b3283b9d235352c9d23: Time spent starting tablet: real 0.174s	user 0.003s	sys 0.005s
I20260709 13:04:03.501760 11098 raft_consensus.cc:3060] T 75492f86fde046c3900df3eac8292fd4 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:04:03.444949 11708 leader_election.cc:290] T d71ca3a1ba6b48478aaf62741bf3cf6a P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 2 election: Requested vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:03.459322 11729 raft_consensus.cc:515] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
W20260709 13:04:03.461239 11069 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:03.461148 11149 tablet_bootstrap.cc:492] T 401a9cf11eac479aa445ec8c1d6e33b2 P 873357742d5843579556ee8166f23f47: No bootstrap required, opened a new log
I20260709 13:04:03.466809 10874 raft_consensus.cc:2468] T 57b6a2418c1342e7924195e9c1dbd357 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 1.
I20260709 13:04:03.506366 11149 ts_tablet_manager.cc:1403] T 401a9cf11eac479aa445ec8c1d6e33b2 P 873357742d5843579556ee8166f23f47: Time spent bootstrapping tablet: real 1.161s	user 0.029s	sys 0.013s
W20260709 13:04:03.463045 11744 log.cc:927] Time spent T 32147c6d2d5a477aa38c597c7f78df5b P 0172fed22b6b4890a7fdc953fa128af4: Append to log took a long time: real 1.299s	user 0.004s	sys 0.001s
I20260709 13:04:03.424098 10995 leader_election.cc:304] T 4b37f8fca8dc42718b2a42f0883a956f P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:04:03.430506 11583 leader_election.cc:290] T 2567c2d84e4648d7ba5dc23a3df32817 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:03.412621 11595 raft_consensus.cc:515] T 5d439326f23747eaab3c1e7da2d6b785 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:03.407048 11782 raft_consensus.cc:493] T 7839ca2d2c13431a85f44d7c458c6351 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:03.411420 11100 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f371d2565dd84667b5613848604949bb" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:04:03.493446 11102 raft_consensus.cc:2468] T 836251ebc2b74f5da88b8a8b2338e2a5 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 3.
I20260709 13:04:03.481294 11658 leader_election.cc:290] T 8c160ddfe9b34bdaa3835d0915c1e06f P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:03.481256 10876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d439326f23747eaab3c1e7da2d6b785" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:03.495492 10873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555217d45d9b4ee492dbb3ddaf0f7c02" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:03.495582 11633 raft_consensus.cc:740] T f371d2565dd84667b5613848604949bb P 873357742d5843579556ee8166f23f47 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 873357742d5843579556ee8166f23f47, State: Running, Role: LEADER
I20260709 13:04:03.416745 11022 rpcz_store.cc:276] 0709 13:04:01.915147 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:01.915745 (+   598us) service_pool.cc:224] Handling call
0709 13:04:01.916137 (+   392us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:01.977593 (+ 61456us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:01.979740 (+  2147us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:02.082579 (+102839us) log.cc:844] Serialized 26 byte log entry
0709 13:04:02.957504 (+874925us) raft_consensus.cc:1645] Marking committed up to 1
0709 13:04:02.965986 (+  8482us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:02.975858 (+  9872us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:03.339238 (+363380us) raft_consensus.cc:1685] finished
0709 13:04:03.339263 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:03.373889 (+ 34626us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":24271,"raft.queue_time_us":57214,"raft.run_cpu_time_us":382,"raft.run_wall_time_us":20168,"thread_start_us":696155,"threads_started":1,"wal-append.queue_time_us":808007}
W20260709 13:04:03.481972 10845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 43764.4us
I20260709 13:04:03.509914 10919 leader_election.cc:304] T e5ba72ccd854430baadbc636dfd17136 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47; no voters: 
W20260709 13:04:03.490018 11585 consensus_meta.cc:287] T 9a1716c0adf34bee8a4e99341ba5788f P 873357742d5843579556ee8166f23f47: Time spent flushing consensus metadata: real 0.659s	user 0.015s	sys 0.003s
I20260709 13:04:03.523010 10955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d7ce47b9a92d43dd88d95bcda5038597" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:03.527487 11152 tablet_bootstrap.cc:492] T b67eff0634e540eab91cf8cee795969f P 0172fed22b6b4890a7fdc953fa128af4: Bootstrap starting.
W20260709 13:04:03.508788 10995 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:38967, user_credentials={real_user=slave}} blocked reactor thread for 84826.2us
W20260709 13:04:03.522753 11610 consensus_meta.cc:287] T b4cd3d7dfbbf4096908ff11e9dc57141 P 0172fed22b6b4890a7fdc953fa128af4: Time spent flushing consensus metadata: real 0.678s	user 0.015s	sys 0.000s
I20260709 13:04:03.521289 11780 raft_consensus.cc:2804] T 4b37f8fca8dc42718b2a42f0883a956f P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:04:03.528002 11100 raft_consensus.cc:3060] T f371d2565dd84667b5613848604949bb P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:04:03.536621 11639 raft_consensus.cc:2804] T e5ba72ccd854430baadbc636dfd17136 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:04:03.547217 11029 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f9eb06ed01f4ce68be3af8a8434665b" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:03.553067 11069 leader_election.cc:304] T 5d439326f23747eaab3c1e7da2d6b785 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda
I20260709 13:04:03.539124 11780 raft_consensus.cc:493] T 4b37f8fca8dc42718b2a42f0883a956f P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 10ca5ee68ad64b3283b9d235352c9d23)
I20260709 13:04:03.541980 10873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "836251ebc2b74f5da88b8a8b2338e2a5" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
W20260709 13:04:03.578337 10921 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:38967, user_credentials={real_user=slave}} blocked reactor thread for 94472us
I20260709 13:04:03.517189 11734 consensus_queue.cc:237] T 34064dc7bf3747dbb3876dd824ee0688 P 873357742d5843579556ee8166f23f47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:03.526680 11782 raft_consensus.cc:515] T 7839ca2d2c13431a85f44d7c458c6351 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:03.546775 10952 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a154cc8b181f460abf4880064379948b" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:04:03.580693 10921 leader_election.cc:304] T 34064dc7bf3747dbb3876dd824ee0688 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23; no voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47
I20260709 13:04:03.586081 11148 tablet_bootstrap.cc:492] T 30f628e3d7914d609e7f1d980a74c654 P 10ca5ee68ad64b3283b9d235352c9d23: Bootstrap starting.
I20260709 13:04:03.585532 10921 leader_election.cc:304] T 7839ca2d2c13431a85f44d7c458c6351 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23; no voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47
I20260709 13:04:03.570467 11607 raft_consensus.cc:3060] T 5d439326f23747eaab3c1e7da2d6b785 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:04:03.574676 10874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "868e92339c2f4508af0afbd07c814d64" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:03.569270 11639 raft_consensus.cc:493] T e5ba72ccd854430baadbc636dfd17136 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:03.582763 10876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "836251ebc2b74f5da88b8a8b2338e2a5" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:04:03.586992 10873 raft_consensus.cc:2468] T 836251ebc2b74f5da88b8a8b2338e2a5 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 1.
W20260709 13:04:03.606809 10950 consensus_meta.cc:287] T e96e80ba9ac34d0998395515e5c8c41a P 10ca5ee68ad64b3283b9d235352c9d23: Time spent flushing consensus metadata: real 0.779s	user 0.016s	sys 0.000s
I20260709 13:04:03.578171 11780 raft_consensus.cc:3060] T 4b37f8fca8dc42718b2a42f0883a956f P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:04:03.550055 10845 leader_election.cc:304] T d371c5e27bd843469d2e2e453734cb4a P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 186b12638a87410487467df721a3cbda; no voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47
I20260709 13:04:03.647253 10950 raft_consensus.cc:2468] T e96e80ba9ac34d0998395515e5c8c41a P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 3.
I20260709 13:04:03.647751 10873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d270b9e5481245d9b5340a454a7cd10b" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
W20260709 13:04:03.647945 10845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 107051us
I20260709 13:04:03.580798 11598 raft_consensus.cc:493] T bac7e5040f984ba2b7146bd36af7267e P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:03.604612 11729 leader_election.cc:290] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 2 election: Requested vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:03.591008 11529 raft_consensus.cc:2749] T 7839ca2d2c13431a85f44d7c458c6351 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:04:03.582554 11099 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37676 (request call id 141) took 2279 ms. Trace:
I20260709 13:04:03.649859 11807 raft_consensus.cc:2749] T d371c5e27bd843469d2e2e453734cb4a P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 13:04:03.581413 11797 raft_consensus.cc:493] T 9a1716c0adf34bee8a4e99341ba5788f P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:03.619544 10952 raft_consensus.cc:2468] T a154cc8b181f460abf4880064379948b P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 2.
I20260709 13:04:03.650676 11797 raft_consensus.cc:515] T 9a1716c0adf34bee8a4e99341ba5788f P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:03.651796 10952 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49434 (request call id 134) took 1183 ms. Trace:
I20260709 13:04:03.652212 10952 rpcz_store.cc:276] 0709 13:04:02.468320 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:02.471524 (+  3204us) service_pool.cc:224] Handling call
0709 13:04:03.545208 (+1073684us) spinlock_profiling.cc:243] Waited 3.85 ms on lock 0x7b580001d198. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f426303db71 00007f426303deee 0000000000375939 00007f426b0b11e0 00007f426b17a0c4 00007f426b17a131 00007f426b1280a0 00007f426b127984 00007f42646a26d4 00007f42646a2662
0709 13:04:03.651534 (+106326us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":8865664}
I20260709 13:04:03.653008 11023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a1716c0adf34bee8a4e99341ba5788f" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
I20260709 13:04:03.653381 11797 leader_election.cc:290] T 9a1716c0adf34bee8a4e99341ba5788f P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:03.654106 11023 raft_consensus.cc:2410] T 9a1716c0adf34bee8a4e99341ba5788f P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 0172fed22b6b4890a7fdc953fa128af4 for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 13:04:03.630195 10954 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e96e80ba9ac34d0998395515e5c8c41a" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:04:03.589205 10921 leader_election.cc:304] T 63d44f6ba6434ab2870fe3190f4d7c2f P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23; no voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47
I20260709 13:04:03.656078 11023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e89e640169e14e2d98f968f76027daa7" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
W20260709 13:04:03.656275 10921 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:38967, user_credentials={real_user=slave}} blocked reactor thread for 67220.7us
I20260709 13:04:03.656826 11023 raft_consensus.cc:2468] T e89e640169e14e2d98f968f76027daa7 P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 3.
I20260709 13:04:03.657155 10921 leader_election.cc:304] T 72b111ad7b5f4753ab1d2b5ee1114840 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda; no voters: 0172fed22b6b4890a7fdc953fa128af4
I20260709 13:04:03.658036 11553 raft_consensus.cc:2749] T 63d44f6ba6434ab2870fe3190f4d7c2f P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:04:03.658798 10921 leader_election.cc:304] T 8a69f32c53f24a50b7852d143665d7b4 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23; no voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda
I20260709 13:04:03.659135 10946 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57b6a2418c1342e7924195e9c1dbd357" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:03.659710 11023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57b6a2418c1342e7924195e9c1dbd357" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:03.661055 11023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2567c2d84e4648d7ba5dc23a3df32817" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:03.661271 10921 leader_election.cc:304] T 57b6a2418c1342e7924195e9c1dbd357 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda; no voters: 
I20260709 13:04:03.644378 11708 raft_consensus.cc:493] T b4cd3d7dfbbf4096908ff11e9dc57141 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:03.541525 11610 raft_consensus.cc:515] T b4cd3d7dfbbf4096908ff11e9dc57141 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:03.648460 10950 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49434 (request call id 138) took 1133 ms. Trace:
I20260709 13:04:03.649235 10876 raft_consensus.cc:3060] T 836251ebc2b74f5da88b8a8b2338e2a5 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Advancing to term 3
I20260709 13:04:03.650499 11781 raft_consensus.cc:493] T 525bfdbd6d934afd823f7d3ee3f9d052 P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 10ca5ee68ad64b3283b9d235352c9d23)
I20260709 13:04:03.664124 10992 leader_election.cc:304] T 1e4f2241a94a41f38e481137f868b83d P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 873357742d5843579556ee8166f23f47; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda
W20260709 13:04:03.611797 10993 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:03.658993 11553 raft_consensus.cc:2804] T 72b111ad7b5f4753ab1d2b5ee1114840 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:04:03.571137 11633 consensus_queue.cc:260] T f371d2565dd84667b5613848604949bb P 873357742d5843579556ee8166f23f47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:03.580964 11027 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4cd3d7dfbbf4096908ff11e9dc57141" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:03.648831 10845 leader_election.cc:304] T 87cd8f5097624c8692f1d83544d8881e P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda; no voters: 
I20260709 13:04:03.663451 11069 leader_election.cc:304] T 836251ebc2b74f5da88b8a8b2338e2a5 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda; no voters: 
I20260709 13:04:03.343290 11097 rpcz_store.cc:276] 0709 13:04:02.010851 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:02.251487 (+240636us) service_pool.cc:224] Handling call
0709 13:04:02.276220 (+ 24733us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:02.483844 (+207624us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:02.486407 (+  2563us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:02.612951 (+126544us) log.cc:844] Serialized 26 byte log entry
0709 13:04:03.103552 (+490601us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:04:03.136475 (+ 32923us) log.cc:844] Serialized 14 byte log entry
0709 13:04:03.136748 (+   273us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:03.136844 (+    96us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:03.239803 (+102959us) raft_consensus.cc:1685] finished
0709 13:04:03.239826 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:03.239998 (+   172us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":10665,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":46,"raft.queue_time_us":84362,"raft.run_cpu_time_us":252,"raft.run_wall_time_us":255,"spinlock_wait_cycles":51968,"thread_start_us":637494,"threads_started":2,"wal-append.queue_time_us":544367}
I20260709 13:04:03.610716 11804 consensus_queue.cc:1059] T d71ca3a1ba6b48478aaf62741bf3cf6a P 873357742d5843579556ee8166f23f47 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.084s
I20260709 13:04:03.607153 10953 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e89e640169e14e2d98f968f76027daa7" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:03.694885 11097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff13ae9a47c7455f962406c6d131a3ff" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:03.695817 11097 raft_consensus.cc:2468] T ff13ae9a47c7455f962406c6d131a3ff P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 0.
I20260709 13:04:03.697031 11097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87cd8f5097624c8692f1d83544d8881e" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:03.585052 11748 raft_consensus.cc:2749] T 34064dc7bf3747dbb3876dd824ee0688 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 13:04:03.606666 11151 tablet_bootstrap.cc:492] T a1ec2b50c6ad4f7c8033476208abb26d P 186b12638a87410487467df721a3cbda: No bootstrap required, opened a new log
I20260709 13:04:03.652251 11029 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "868e92339c2f4508af0afbd07c814d64" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:03.606078 11149 raft_consensus.cc:359] T 401a9cf11eac479aa445ec8c1d6e33b2 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:03.698738 11628 raft_consensus.cc:2764] T 836251ebc2b74f5da88b8a8b2338e2a5 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260709 13:04:03.699016 11803 raft_consensus.cc:493] T bac7e5040f984ba2b7146bd36af7267e P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:03.672569 11101 raft_consensus.cc:2468] T d270b9e5481245d9b5340a454a7cd10b P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 3.
I20260709 13:04:03.575325 11723 raft_consensus.cc:493] T 868e92339c2f4508af0afbd07c814d64 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:03.650153 11639 raft_consensus.cc:3060] T e5ba72ccd854430baadbc636dfd17136 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:04:03.700022 11096 raft_consensus.cc:2368] T b4cd3d7dfbbf4096908ff11e9dc57141 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 873357742d5843579556ee8166f23f47 for earlier term 2. Current term is 3.
I20260709 13:04:03.700451 11723 raft_consensus.cc:515] T 868e92339c2f4508af0afbd07c814d64 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:03.703156 11723 leader_election.cc:290] T 868e92339c2f4508af0afbd07c814d64 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:03.703492 11029 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c160ddfe9b34bdaa3835d0915c1e06f" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:03.699826 11803 raft_consensus.cc:515] T bac7e5040f984ba2b7146bd36af7267e P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:03.632251 11021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32147c6d2d5a477aa38c597c7f78df5b" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:03.697970 11097 raft_consensus.cc:2393] T 87cd8f5097624c8692f1d83544d8881e P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 10ca5ee68ad64b3283b9d235352c9d23 in current term 3: Already voted for candidate 186b12638a87410487467df721a3cbda in this term.
I20260709 13:04:03.698668 11151 ts_tablet_manager.cc:1403] T a1ec2b50c6ad4f7c8033476208abb26d P 186b12638a87410487467df721a3cbda: Time spent bootstrapping tablet: real 1.014s	user 0.019s	sys 0.008s
I20260709 13:04:03.706717 11803 leader_election.cc:290] T bac7e5040f984ba2b7146bd36af7267e P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:03.707468 11743 leader_election.cc:290] T a610d1a71f52432e88fe56130f921fa2 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 4 election: Requested vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:03.707975 10994 leader_election.cc:304] T ff13ae9a47c7455f962406c6d131a3ff P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:04:03.580003 11777 raft_consensus.cc:493] T e89e640169e14e2d98f968f76027daa7 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:03.709002 11653 raft_consensus.cc:2804] T ff13ae9a47c7455f962406c6d131a3ff P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:04:03.709501 11653 raft_consensus.cc:493] T ff13ae9a47c7455f962406c6d131a3ff P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:03.613597 11508 leader_election.cc:290] T e89e640169e14e2d98f968f76027daa7 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:03.703867 10874 raft_consensus.cc:2468] T 868e92339c2f4508af0afbd07c814d64 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 1.
I20260709 13:04:03.699203 11149 raft_consensus.cc:385] T 401a9cf11eac479aa445ec8c1d6e33b2 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:04:03.710589 11149 raft_consensus.cc:740] T 401a9cf11eac479aa445ec8c1d6e33b2 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 873357742d5843579556ee8166f23f47, State: Initialized, Role: FOLLOWER
I20260709 13:04:03.674315 11598 raft_consensus.cc:515] T bac7e5040f984ba2b7146bd36af7267e P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:03.711601 11149 consensus_queue.cc:260] T 401a9cf11eac479aa445ec8c1d6e33b2 P 873357742d5843579556ee8166f23f47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:03.714067 11598 leader_election.cc:290] T bac7e5040f984ba2b7146bd36af7267e P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:03.714687 11149 ts_tablet_manager.cc:1434] T 401a9cf11eac479aa445ec8c1d6e33b2 P 873357742d5843579556ee8166f23f47: Time spent starting tablet: real 0.208s	user 0.009s	sys 0.000s
I20260709 13:04:03.629040 11673 raft_consensus.cc:3060] T a64b7e99d6c54934a7de6f6dffcf5e94 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:04:03.665709 10844 leader_election.cc:304] T e89e640169e14e2d98f968f76027daa7 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:04:03.678623 11729 raft_consensus.cc:2749] T 8a69f32c53f24a50b7852d143665d7b4 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 13:04:03.709998 11653 raft_consensus.cc:3060] T ff13ae9a47c7455f962406c6d131a3ff P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:04:03.612325 11782 leader_election.cc:290] T 7839ca2d2c13431a85f44d7c458c6351 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
W20260709 13:04:03.734303 10844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:45807, user_credentials={real_user=slave}} blocked reactor thread for 68681.2us
I20260709 13:04:03.737010 11069 leader_election.cc:304] T 868e92339c2f4508af0afbd07c814d64 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda; no voters: 873357742d5843579556ee8166f23f47
I20260709 13:04:03.629369 11595 leader_election.cc:290] T 5d439326f23747eaab3c1e7da2d6b785 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 3 election: Requested vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:03.715915 11029 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a154cc8b181f460abf4880064379948b" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:03.648515 10873 raft_consensus.cc:3060] T d270b9e5481245d9b5340a454a7cd10b P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Advancing to term 3
I20260709 13:04:03.749025 11803 raft_consensus.cc:2804] T 868e92339c2f4508af0afbd07c814d64 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:04:03.749678 11803 raft_consensus.cc:493] T 868e92339c2f4508af0afbd07c814d64 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:03.662106 11708 raft_consensus.cc:515] T b4cd3d7dfbbf4096908ff11e9dc57141 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:03.750185 11803 raft_consensus.cc:3060] T 868e92339c2f4508af0afbd07c814d64 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:04:03.753656 11708 leader_election.cc:290] T b4cd3d7dfbbf4096908ff11e9dc57141 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:03.714677 11027 raft_consensus.cc:2468] T b4cd3d7dfbbf4096908ff11e9dc57141 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 2.
I20260709 13:04:03.576244 11597 raft_consensus.cc:515] T c31881534c5e4f0089c82057c1746652 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:03.755259 11508 raft_consensus.cc:2804] T e89e640169e14e2d98f968f76027daa7 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:04:03.755744 11508 raft_consensus.cc:493] T e89e640169e14e2d98f968f76027daa7 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Starting leader election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:03.756126 11508 raft_consensus.cc:3060] T e89e640169e14e2d98f968f76027daa7 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:04:03.756162 11597 leader_election.cc:290] T c31881534c5e4f0089c82057c1746652 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 2 election: Requested vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:03.661374 11724 raft_consensus.cc:493] T 7757ecbf402e47bfbc93d0447741e26c P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:03.757131 11724 raft_consensus.cc:515] T 7757ecbf402e47bfbc93d0447741e26c P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:03.637005 11028 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57b6a2418c1342e7924195e9c1dbd357" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:03.759629 11724 leader_election.cc:290] T 7757ecbf402e47bfbc93d0447741e26c P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:03.618178 11022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e96e80ba9ac34d0998395515e5c8c41a" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
I20260709 13:04:03.760589 11022 raft_consensus.cc:2368] T e96e80ba9ac34d0998395515e5c8c41a P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 186b12638a87410487467df721a3cbda for earlier term 2. Current term is 3.
I20260709 13:04:03.709736 11151 raft_consensus.cc:359] T a1ec2b50c6ad4f7c8033476208abb26d P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:03.761876 11151 raft_consensus.cc:385] T a1ec2b50c6ad4f7c8033476208abb26d P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:04:03.762208 11151 raft_consensus.cc:740] T a1ec2b50c6ad4f7c8033476208abb26d P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 186b12638a87410487467df721a3cbda, State: Initialized, Role: FOLLOWER
I20260709 13:04:03.762689 11021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a610d1a71f52432e88fe56130f921fa2" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
I20260709 13:04:03.763479 11021 raft_consensus.cc:3060] T a610d1a71f52432e88fe56130f921fa2 P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:04:03.763695 11803 raft_consensus.cc:515] T 868e92339c2f4508af0afbd07c814d64 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:03.766858 10874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d439326f23747eaab3c1e7da2d6b785" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:03.767175 10873 raft_consensus.cc:2468] T d270b9e5481245d9b5340a454a7cd10b P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 3.
I20260709 13:04:03.767546 11803 leader_election.cc:290] T 868e92339c2f4508af0afbd07c814d64 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 2 election: Requested vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:03.755262 10955 raft_consensus.cc:1240] T c31881534c5e4f0089c82057c1746652 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Rejecting Update request from peer 186b12638a87410487467df721a3cbda for earlier term 1. Current term is 2. Ops: []
I20260709 13:04:03.682147 11774 raft_consensus.cc:2749] T 1e4f2241a94a41f38e481137f868b83d P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 13:04:03.709571 11777 raft_consensus.cc:515] T e89e640169e14e2d98f968f76027daa7 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:03.763055 11151 consensus_queue.cc:260] T a1ec2b50c6ad4f7c8033476208abb26d P 186b12638a87410487467df721a3cbda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:03.699885 11748 raft_consensus.cc:2804] T 57b6a2418c1342e7924195e9c1dbd357 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:04:03.682055 11553 raft_consensus.cc:493] T 72b111ad7b5f4753ab1d2b5ee1114840 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 0172fed22b6b4890a7fdc953fa128af4)
W20260709 13:04:03.774631 10921 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:38967, user_credentials={real_user=slave}} blocked reactor thread for 113454us
I20260709 13:04:03.775229 11029 raft_consensus.cc:2393] T a154cc8b181f460abf4880064379948b P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0172fed22b6b4890a7fdc953fa128af4 in current term 2: Already voted for candidate 873357742d5843579556ee8166f23f47 in this term.
W20260709 13:04:03.728976 11744 log.cc:927] Time spent T 32147c6d2d5a477aa38c597c7f78df5b P 0172fed22b6b4890a7fdc953fa128af4: Append to log took a long time: real 0.061s	user 0.000s	sys 0.000s
I20260709 13:04:03.739086 11735 leader_election.cc:290] T a5299b8d044f41b7b68057e63d17b159 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 1 election: Requested vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:03.796288 10844 leader_election.cc:400] T e96e80ba9ac34d0998395515e5c8c41a P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 2 election: Vote denied by peer 873357742d5843579556ee8166f23f47 (127.0.107.195:45807) with higher term. Message: Invalid argument: T e96e80ba9ac34d0998395515e5c8c41a P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 186b12638a87410487467df721a3cbda for earlier term 2. Current term is 3.
I20260709 13:04:03.678838 10950 rpcz_store.cc:276] 0709 13:04:02.515211 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:02.682528 (+167317us) service_pool.cc:224] Handling call
0709 13:04:03.648377 (+965849us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:03.781548 11152 tablet_bootstrap.cc:654] T b67eff0634e540eab91cf8cee795969f P 0172fed22b6b4890a7fdc953fa128af4: Neither blocks nor log segments found. Creating new log.
I20260709 13:04:03.769030 11743 raft_consensus.cc:2804] T 87cd8f5097624c8692f1d83544d8881e P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:04:03.768332 10874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a9353d1d230416296627d7cf8b1d425" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:04:03.809201 10992 leader_election.cc:304] T a154cc8b181f460abf4880064379948b P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47; no voters: 0172fed22b6b4890a7fdc953fa128af4
I20260709 13:04:03.768268 10873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c31881534c5e4f0089c82057c1746652" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
W20260709 13:04:03.744545 10845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 95842.7us
I20260709 13:04:03.770255 11633 raft_consensus.cc:3060] T f371d2565dd84667b5613848604949bb P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:04:03.782267 10952 raft_consensus.cc:1240] T 5d439326f23747eaab3c1e7da2d6b785 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Rejecting Update request from peer 186b12638a87410487467df721a3cbda for earlier term 2. Current term is 3. Ops: []
I20260709 13:04:03.689697 11781 raft_consensus.cc:515] T 525bfdbd6d934afd823f7d3ee3f9d052 P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:03.823693 10921 leader_election.cc:304] T d270b9e5481245d9b5340a454a7cd10b P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda; no voters: 
I20260709 13:04:03.782903 11748 raft_consensus.cc:493] T 57b6a2418c1342e7924195e9c1dbd357 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:03.807884 11553 raft_consensus.cc:3060] T 72b111ad7b5f4753ab1d2b5ee1114840 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Advancing to term 3
W20260709 13:04:03.827899 11688 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.685s	user 0.007s	sys 0.037s
I20260709 13:04:03.842084 10992 leader_election.cc:304] T e96e80ba9ac34d0998395515e5c8c41a P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:04:03.831362 10844 leader_election.cc:403] T e96e80ba9ac34d0998395515e5c8c41a P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
W20260709 13:04:03.844242 10919 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:45807, user_credentials={real_user=slave}} blocked reactor thread for 232914us
I20260709 13:04:03.847641 11029 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7757ecbf402e47bfbc93d0447741e26c" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:03.851715 10874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a1716c0adf34bee8a4e99341ba5788f" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:04:03.852401 10874 raft_consensus.cc:3060] T 9a1716c0adf34bee8a4e99341ba5788f P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:04:03.852805 10876 raft_consensus.cc:2468] T 836251ebc2b74f5da88b8a8b2338e2a5 P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 3.
I20260709 13:04:03.839035 11774 raft_consensus.cc:2804] T a154cc8b181f460abf4880064379948b P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:04:03.860353 11151 ts_tablet_manager.cc:1434] T a1ec2b50c6ad4f7c8033476208abb26d P 186b12638a87410487467df721a3cbda: Time spent starting tablet: real 0.154s	user 0.008s	sys 0.000s
I20260709 13:04:03.836844 11743 raft_consensus.cc:697] T 87cd8f5097624c8692f1d83544d8881e P 186b12638a87410487467df721a3cbda [term 3 LEADER]: Becoming Leader. State: Replica: 186b12638a87410487467df721a3cbda, State: Running, Role: LEADER
I20260709 13:04:03.856185 10953 raft_consensus.cc:2393] T e89e640169e14e2d98f968f76027daa7 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 186b12638a87410487467df721a3cbda in current term 3: Already voted for candidate 10ca5ee68ad64b3283b9d235352c9d23 in this term.
W20260709 13:04:03.855192 11688 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.713s	user 0.008s	sys 0.038s
W20260709 13:04:03.860021 10993 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:03.854921 11103 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f9df2bc281f4866b6da5da94a0defc7" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
W20260709 13:04:03.836230 11067 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:45807, user_credentials={real_user=slave}} blocked reactor thread for 52809.9us
I20260709 13:04:03.849987 11595 raft_consensus.cc:2804] T d270b9e5481245d9b5340a454a7cd10b P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:04:03.846585 11748 raft_consensus.cc:3060] T 57b6a2418c1342e7924195e9c1dbd357 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:04:03.845448 10879 raft_consensus.cc:2468] T 5de5f0581e3b41dea729363d6ca563bb P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 3.
I20260709 13:04:03.857478 11804 raft_consensus.cc:2804] T e96e80ba9ac34d0998395515e5c8c41a P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Leader election won for term 3
W20260709 13:04:03.862370 10844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:45807, user_credentials={real_user=slave}} blocked reactor thread for 62468.2us
I20260709 13:04:03.856518 10950 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2567c2d84e4648d7ba5dc23a3df32817" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:03.863279 11597 consensus_queue.cc:1048] T ff13ae9a47c7455f962406c6d131a3ff P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.019s
I20260709 13:04:03.883100 11029 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "401a9cf11eac479aa445ec8c1d6e33b2" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:03.898020 11029 raft_consensus.cc:2468] T 401a9cf11eac479aa445ec8c1d6e33b2 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 0.
I20260709 13:04:03.883287 11743 consensus_queue.cc:237] T 87cd8f5097624c8692f1d83544d8881e P 186b12638a87410487467df721a3cbda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:03.888109 11151 tablet_bootstrap.cc:492] T f34f2bc02d85444f9096e0f5b4fe4331 P 186b12638a87410487467df721a3cbda: Bootstrap starting.
I20260709 13:04:03.891212 10921 leader_election.cc:304] T 836251ebc2b74f5da88b8a8b2338e2a5 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda; no voters: 
I20260709 13:04:03.868407 10876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f9eb06ed01f4ce68be3af8a8434665b" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:03.866963 11774 raft_consensus.cc:697] T a154cc8b181f460abf4880064379948b P 873357742d5843579556ee8166f23f47 [term 2 LEADER]: Becoming Leader. State: Replica: 873357742d5843579556ee8166f23f47, State: Running, Role: LEADER
I20260709 13:04:03.897559 11688 raft_consensus.cc:493] T 30f628e3d7914d609e7f1d980a74c654 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:03.897948 11803 raft_consensus.cc:493] T 7709387f4d3d4f9992d97a0fbca39a40 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:03.896384 10845 leader_election.cc:343] T 401a9cf11eac479aa445ec8c1d6e33b2 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:04:03.896091 11758 raft_consensus.cc:493] T bb7d67be500947b3aef4c4245e33671c P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:03.895676 11595 raft_consensus.cc:697] T d270b9e5481245d9b5340a454a7cd10b P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 LEADER]: Becoming Leader. State: Replica: 10ca5ee68ad64b3283b9d235352c9d23, State: Running, Role: LEADER
I20260709 13:04:03.922782 11777 leader_election.cc:290] T e89e640169e14e2d98f968f76027daa7 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:03.650281 11099 rpcz_store.cc:276] 0709 13:04:01.302904 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:01.894230 (+591326us) service_pool.cc:224] Handling call
0709 13:04:01.894676 (+   446us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:01.901493 (+  6817us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:01.901531 (+    38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:01.902159 (+   628us) log.cc:844] Serialized 26 byte log entry
0709 13:04:02.027405 (+125246us) raft_consensus.cc:1645] Marking committed up to 1
0709 13:04:02.133970 (+106565us) log.cc:844] Serialized 14 byte log entry
0709 13:04:02.245234 (+111264us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:02.264267 (+ 19033us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:02.645220 (+380953us) spinlock_profiling.cc:243] Waited 30.8 ms on lock 0x7b180025b040. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f42630432a6 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f4262fefc0a 00007f426c72f219
0709 13:04:03.019333 (+374113us) spinlock_profiling.cc:243] Waited 882 us on lock 0x7b180025b040. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f4262fefc0a 00007f426c72f219 00007f426a03658f 00007f4263075571 00007f42630777c1 00007f4263077779 00007f4263077709 00007f42630776d1 00007f42630769cd
0709 13:04:03.462251 (+442918us) spinlock_profiling.cc:243] Waited 23.2 ms on lock 0x7b4400022b4c. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f4262f9bc85 00007f4262f9c6a2 00007f4265d0e1db 00007f4265d09aaf 00007f4265cfd86e 00007f4265cfcbb6 00007f4265cfc7d8 00007f4265d06221 00007f4265d061d9 00007f4265d06169
0709 13:04:03.569752 (+107501us) raft_consensus.cc:1685] finished
0709 13:04:03.573048 (+  3296us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:03.580267 (+  7219us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":45461,"prepare.queue_time_us":207527,"prepare.run_cpu_time_us":59,"prepare.run_wall_time_us":60,"raft.queue_time_us":146557,"raft.run_cpu_time_us":393,"raft.run_wall_time_us":3069,"spinlock_wait_cycles":126190336,"thread_start_us":229032,"threads_started":3,"wal-append.queue_time_us":228890}
I20260709 13:04:03.926877 10879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5299b8d044f41b7b68057e63d17b159" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:03.914306 10952 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "836251ebc2b74f5da88b8a8b2338e2a5" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:03.929956 11029 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a610d1a71f52432e88fe56130f921fa2" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 4 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
I20260709 13:04:03.963128 11029 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c160ddfe9b34bdaa3835d0915c1e06f" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:03.933852 11585 raft_consensus.cc:3055] T d71ca3a1ba6b48478aaf62741bf3cf6a P 873357742d5843579556ee8166f23f47 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:04:03.896629 11100 raft_consensus.cc:2468] T f371d2565dd84667b5613848604949bb P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 2.
I20260709 13:04:03.897648 10873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c31881534c5e4f0089c82057c1746652" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:03.888594 11780 raft_consensus.cc:515] T 4b37f8fca8dc42718b2a42f0883a956f P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:03.908862 11695 raft_consensus.cc:2804] T 836251ebc2b74f5da88b8a8b2338e2a5 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:04:03.904573 11804 raft_consensus.cc:697] T e96e80ba9ac34d0998395515e5c8c41a P 873357742d5843579556ee8166f23f47 [term 3 LEADER]: Becoming Leader. State: Replica: 873357742d5843579556ee8166f23f47, State: Running, Role: LEADER
I20260709 13:04:03.979497 11152 tablet_bootstrap.cc:492] T b67eff0634e540eab91cf8cee795969f P 0172fed22b6b4890a7fdc953fa128af4: No bootstrap required, opened a new log
I20260709 13:04:03.980237 11152 ts_tablet_manager.cc:1403] T b67eff0634e540eab91cf8cee795969f P 0172fed22b6b4890a7fdc953fa128af4: Time spent bootstrapping tablet: real 0.453s	user 0.013s	sys 0.012s
I20260709 13:04:03.945498 11781 leader_election.cc:290] T 525bfdbd6d934afd823f7d3ee3f9d052 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:03.945037 11774 consensus_queue.cc:237] T a154cc8b181f460abf4880064379948b P 873357742d5843579556ee8166f23f47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:03.940251 11803 raft_consensus.cc:515] T 7709387f4d3d4f9992d97a0fbca39a40 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:03.937897 11688 raft_consensus.cc:515] T 30f628e3d7914d609e7f1d980a74c654 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:03.947125 10950 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5299b8d044f41b7b68057e63d17b159" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:03.915377 11610 leader_election.cc:290] T b4cd3d7dfbbf4096908ff11e9dc57141 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 3 election: Requested vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:03.955039 10876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7757ecbf402e47bfbc93d0447741e26c" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:03.954900 11748 raft_consensus.cc:515] T 57b6a2418c1342e7924195e9c1dbd357 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:03.987037 11796 raft_consensus.cc:493] T a1ec2b50c6ad4f7c8033476208abb26d P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:03.936954 10877 consensus_meta.cc:287] T e96e80ba9ac34d0998395515e5c8c41a P 186b12638a87410487467df721a3cbda: Time spent flushing consensus metadata: real 1.217s	user 0.013s	sys 0.014s
I20260709 13:04:03.938278 11673 raft_consensus.cc:493] T 4f9eb06ed01f4ce68be3af8a8434665b P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 186b12638a87410487467df721a3cbda)
I20260709 13:04:03.976966 11100 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37678 (request call id 67) took 1144 ms. Trace:
I20260709 13:04:04.007630 11673 raft_consensus.cc:515] T 4f9eb06ed01f4ce68be3af8a8434665b P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:04.007833 11804 consensus_queue.cc:237] T e96e80ba9ac34d0998395515e5c8c41a P 873357742d5843579556ee8166f23f47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:04.009491 11688 leader_election.cc:290] T 30f628e3d7914d609e7f1d980a74c654 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
W20260709 13:04:03.977128 10947 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49428 (request call id 83) took 1678 ms (client timeout 1711 ms). Trace:
I20260709 13:04:03.977309 11029 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75492f86fde046c3900df3eac8292fd4" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:03.967465 11585 raft_consensus.cc:740] T d71ca3a1ba6b48478aaf62741bf3cf6a P 873357742d5843579556ee8166f23f47 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 873357742d5843579556ee8166f23f47, State: Running, Role: LEADER
I20260709 13:04:03.977921 11695 raft_consensus.cc:697] T 836251ebc2b74f5da88b8a8b2338e2a5 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 LEADER]: Becoming Leader. State: Replica: 10ca5ee68ad64b3283b9d235352c9d23, State: Running, Role: LEADER
I20260709 13:04:03.975057 10878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a610d1a71f52432e88fe56130f921fa2" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:04:03.954232 10949 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4cd3d7dfbbf4096908ff11e9dc57141" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:04:03.957420 11758 raft_consensus.cc:515] T bb7d67be500947b3aef4c4245e33671c P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:03.943873 11021 raft_consensus.cc:2468] T a610d1a71f52432e88fe56130f921fa2 P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 4.
W20260709 13:04:03.942812 10845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 52730.6us
I20260709 13:04:03.969702 11103 raft_consensus.cc:1240] T 6a9353d1d230416296627d7cf8b1d425 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Rejecting Update request from peer 186b12638a87410487467df721a3cbda for earlier term 2. Current term is 3. Ops: []
I20260709 13:04:03.983305 11777 raft_consensus.cc:493] T 1e4f2241a94a41f38e481137f868b83d P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:03.997834 10879 raft_consensus.cc:2393] T a5299b8d044f41b7b68057e63d17b159 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 10ca5ee68ad64b3283b9d235352c9d23 in current term 1: Already voted for candidate 186b12638a87410487467df721a3cbda in this term.
I20260709 13:04:03.956681 11148 tablet_bootstrap.cc:654] T 30f628e3d7914d609e7f1d980a74c654 P 10ca5ee68ad64b3283b9d235352c9d23: Neither blocks nor log segments found. Creating new log.
I20260709 13:04:04.016615 11029 raft_consensus.cc:2393] T 75492f86fde046c3900df3eac8292fd4 P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0172fed22b6b4890a7fdc953fa128af4 in current term 1: Already voted for candidate 873357742d5843579556ee8166f23f47 in this term.
I20260709 13:04:04.016878 10878 raft_consensus.cc:2393] T a610d1a71f52432e88fe56130f921fa2 P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 0172fed22b6b4890a7fdc953fa128af4 in current term 4: Already voted for candidate 186b12638a87410487467df721a3cbda in this term.
I20260709 13:04:04.001178 11099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32147c6d2d5a477aa38c597c7f78df5b" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:04.010218 11673 leader_election.cc:290] T 4f9eb06ed01f4ce68be3af8a8434665b P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:04.003895 10950 raft_consensus.cc:2468] T a5299b8d044f41b7b68057e63d17b159 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 0.
I20260709 13:04:03.998262 11800 raft_consensus.cc:515] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:04.011354 10876 raft_consensus.cc:2468] T 7757ecbf402e47bfbc93d0447741e26c P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 1.
W20260709 13:04:04.012739 10947 rpcz_store.cc:269] 0709 13:04:02.298403 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:02.303856 (+  5453us) service_pool.cc:224] Handling call
0709 13:04:03.976904 (+1673048us) spinlock_profiling.cc:243] Waited 232 ms on lock 0x7b5400415854. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:04:03.977048 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":533690624}
I20260709 13:04:04.057673 10871 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e96e80ba9ac34d0998395515e5c8c41a" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:04.010406 10877 raft_consensus.cc:2468] T e96e80ba9ac34d0998395515e5c8c41a P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 3.
I20260709 13:04:04.058270 10876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a1716c0adf34bee8a4e99341ba5788f" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:04:04.058616 10871 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51696 (request call id 156) took 1101 ms. Trace:
I20260709 13:04:04.058874 11658 raft_consensus.cc:2749] T e96e80ba9ac34d0998395515e5c8c41a P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 873357742d5843579556ee8166f23f47 (127.0.107.195:45807) with higher term. Message: Invalid argument: T e96e80ba9ac34d0998395515e5c8c41a P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 186b12638a87410487467df721a3cbda for earlier term 2. Current term is 3.
I20260709 13:04:04.059103 10871 rpcz_store.cc:276] 0709 13:04:02.956981 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:02.960884 (+  3903us) service_pool.cc:224] Handling call
0709 13:04:04.057483 (+1096599us) spinlock_profiling.cc:243] Waited 1.65 ms on lock 0x7b5800000098. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f426303db71 00007f426303da22 00007f426bf56c09 00007f426b0b0ee0 00007f426b17a0ab 00007f426b17a131 00007f426b1280a0 00007f426b127984 00007f42646a26d4 00007f42646a2662
0709 13:04:04.058546 (+  1063us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3792256}
I20260709 13:04:04.059612 10876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5299b8d044f41b7b68057e63d17b159" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:04.003075 11508 raft_consensus.cc:515] T e89e640169e14e2d98f968f76027daa7 P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
W20260709 13:04:04.021587 10843 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 103544us
I20260709 13:04:04.047205 10950 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7839ca2d2c13431a85f44d7c458c6351" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:04.048338 11103 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d156d6ae2ca4fa59a1e387edf0fea39" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:04:04.049983 11748 leader_election.cc:290] T 57b6a2418c1342e7924195e9c1dbd357 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 2 election: Requested vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:04.051647 11780 leader_election.cc:290] T 4b37f8fca8dc42718b2a42f0883a956f P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 2 election: Requested vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
W20260709 13:04:04.045758 11098 consensus_meta.cc:287] T 75492f86fde046c3900df3eac8292fd4 P 0172fed22b6b4890a7fdc953fa128af4: Time spent flushing consensus metadata: real 0.540s	user 0.017s	sys 0.000s
I20260709 13:04:04.019367 11595 consensus_queue.cc:237] T d270b9e5481245d9b5340a454a7cd10b P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:04.052896 10878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c31881534c5e4f0089c82057c1746652" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:04.007632 11100 rpcz_store.cc:276] 0709 13:04:02.841377 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:03.372398 (+531021us) service_pool.cc:224] Handling call
0709 13:04:03.972261 (+599863us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:04.052525 11585 consensus_queue.cc:260] T d71ca3a1ba6b48478aaf62741bf3cf6a P 873357742d5843579556ee8166f23f47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:04.054160 11782 raft_consensus.cc:493] T 0e5d327630e5438b878035d28efe6c0c P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:04.055971 11795 raft_consensus.cc:493] T 7709387f4d3d4f9992d97a0fbca39a40 P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:04.056098 11029 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a1716c0adf34bee8a4e99341ba5788f" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:04.032668 11777 raft_consensus.cc:515] T 1e4f2241a94a41f38e481137f868b83d P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:04.017674 10880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "836251ebc2b74f5da88b8a8b2338e2a5" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:04.066303 10921 leader_election.cc:304] T a5299b8d044f41b7b68057e63d17b159 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23; no voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47
I20260709 13:04:04.012646 11796 raft_consensus.cc:515] T a1ec2b50c6ad4f7c8033476208abb26d P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:04.067868 11069 leader_election.cc:304] T 9a1716c0adf34bee8a4e99341ba5788f P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4; no voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47
I20260709 13:04:04.051581 11803 leader_election.cc:290] T 7709387f4d3d4f9992d97a0fbca39a40 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:04.063175 10949 raft_consensus.cc:2393] T b4cd3d7dfbbf4096908ff11e9dc57141 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 873357742d5843579556ee8166f23f47 in current term 2: Already voted for candidate 10ca5ee68ad64b3283b9d235352c9d23 in this term.
I20260709 13:04:04.078374 11800 leader_election.cc:290] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 4 election: Requested vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:04.041404 11607 raft_consensus.cc:2749] T 5d439326f23747eaab3c1e7da2d6b785 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:04:04.090525 11777 leader_election.cc:290] T 1e4f2241a94a41f38e481137f868b83d P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:04.058502 10947 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2567c2d84e4648d7ba5dc23a3df32817" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:04.091141 10872 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51696 (request call id 157) took 1133 ms. Trace:
I20260709 13:04:04.091842 10946 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d156d6ae2ca4fa59a1e387edf0fea39" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:04:04.092257 10947 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "525bfdbd6d934afd823f7d3ee3f9d052" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:04.092871 10946 raft_consensus.cc:2393] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 186b12638a87410487467df721a3cbda in current term 2: Already voted for candidate 10ca5ee68ad64b3283b9d235352c9d23 in this term.
I20260709 13:04:04.093869 10946 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e89e640169e14e2d98f968f76027daa7" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:04.063331 10871 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7757ecbf402e47bfbc93d0447741e26c" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:04.094870 10871 raft_consensus.cc:2468] T 7757ecbf402e47bfbc93d0447741e26c P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 1.
I20260709 13:04:04.096292 10871 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c31881534c5e4f0089c82057c1746652" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:04.097837 10871 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c160ddfe9b34bdaa3835d0915c1e06f" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:04.058774 10877 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51718 (request call id 182) took 1623 ms. Trace:
I20260709 13:04:04.098297 10946 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "30f628e3d7914d609e7f1d980a74c654" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:04.098809 10877 rpcz_store.cc:276] 0709 13:04:02.476908 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:02.484765 (+  7857us) service_pool.cc:224] Handling call
0709 13:04:04.058701 (+1573936us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:04.100260 10946 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c2a35bb99374e93b573cf870d04faaa" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:04:04.100437 10877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a9353d1d230416296627d7cf8b1d425" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:04.100908 10946 raft_consensus.cc:3060] T 4c2a35bb99374e93b573cf870d04faaa P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:04:04.060369 11758 leader_election.cc:290] T bb7d67be500947b3aef4c4245e33671c P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:04.045329 11743 raft_consensus.cc:493] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:04.102807 11743 raft_consensus.cc:515] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:04.103586 11152 raft_consensus.cc:359] T b67eff0634e540eab91cf8cee795969f P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:04.104337 11152 raft_consensus.cc:385] T b67eff0634e540eab91cf8cee795969f P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:04:04.073797 11029 raft_consensus.cc:2468] T 9a1716c0adf34bee8a4e99341ba5788f P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 3.
I20260709 13:04:04.077086 10879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f371d2565dd84667b5613848604949bb" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:04.077421 11585 raft_consensus.cc:3060] T d71ca3a1ba6b48478aaf62741bf3cf6a P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:04:04.067137 10950 raft_consensus.cc:2468] T 7839ca2d2c13431a85f44d7c458c6351 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 1.
I20260709 13:04:04.075834 11795 raft_consensus.cc:515] T 7709387f4d3d4f9992d97a0fbca39a40 P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:04.119678 10871 raft_consensus.cc:2410] T 8c160ddfe9b34bdaa3835d0915c1e06f P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0172fed22b6b4890a7fdc953fa128af4 for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260709 13:04:04.131083 10871 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d156d6ae2ca4fa59a1e387edf0fea39" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:04:04.132879 10871 raft_consensus.cc:2393] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 10ca5ee68ad64b3283b9d235352c9d23 in current term 2: Already voted for candidate 186b12638a87410487467df721a3cbda in this term.
I20260709 13:04:04.134239 10871 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5a5f5a608504650b48fd27f163faed6" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:04.080963 10876 raft_consensus.cc:2393] T a5299b8d044f41b7b68057e63d17b159 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 873357742d5843579556ee8166f23f47 in current term 1: Already voted for candidate 186b12638a87410487467df721a3cbda in this term.
I20260709 13:04:04.138919 11094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f9eb06ed01f4ce68be3af8a8434665b" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:04.139704 10876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2567c2d84e4648d7ba5dc23a3df32817" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:04.140292 11094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d439326f23747eaab3c1e7da2d6b785" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
W20260709 13:04:04.140627 11824 log.cc:927] Time spent T bac7e5040f984ba2b7146bd36af7267e P 0172fed22b6b4890a7fdc953fa128af4: Append to log took a long time: real 0.130s	user 0.001s	sys 0.000s
I20260709 13:04:04.140980 11094 raft_consensus.cc:3060] T 5d439326f23747eaab3c1e7da2d6b785 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:04:04.141470 10876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "868e92339c2f4508af0afbd07c814d64" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:04:04.070672 11021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "868e92339c2f4508af0afbd07c814d64" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
I20260709 13:04:04.082388 10843 leader_election.cc:304] T 65bf0a62f4d84ef18dd6b23e082724b8 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 186b12638a87410487467df721a3cbda; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47
I20260709 13:04:04.143381 11028 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bac7e5040f984ba2b7146bd36af7267e" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
W20260709 13:04:04.144263 10843 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 61976.8us
I20260709 13:04:04.144345 11021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57b6a2418c1342e7924195e9c1dbd357" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
I20260709 13:04:04.145211 10843 leader_election.cc:304] T d7ce47b9a92d43dd88d95bcda5038597 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 186b12638a87410487467df721a3cbda; no voters: 0172fed22b6b4890a7fdc953fa128af4, 10ca5ee68ad64b3283b9d235352c9d23
I20260709 13:04:04.145515 11151 tablet_bootstrap.cc:654] T f34f2bc02d85444f9096e0f5b4fe4331 P 186b12638a87410487467df721a3cbda: Neither blocks nor log segments found. Creating new log.
I20260709 13:04:04.082650 10880 raft_consensus.cc:2376] T 836251ebc2b74f5da88b8a8b2338e2a5 P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 3. Re-sending same reply.
I20260709 13:04:04.070254 11098 raft_consensus.cc:2468] T 75492f86fde046c3900df3eac8292fd4 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 1.
I20260709 13:04:04.146246 10880 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51696 (request call id 160) took 1188 ms. Trace:
I20260709 13:04:04.146656 10843 leader_election.cc:304] T 57b6a2418c1342e7924195e9c1dbd357 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 186b12638a87410487467df721a3cbda; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47
I20260709 13:04:04.146803 11098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a1716c0adf34bee8a4e99341ba5788f" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:04.147641 11688 raft_consensus.cc:2749] T 65bf0a62f4d84ef18dd6b23e082724b8 P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:04:04.148356 11729 raft_consensus.cc:493] T a977842ba4b744f4bb2febd55d286006 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:04.148747 10843 leader_election.cc:304] T 2567c2d84e4648d7ba5dc23a3df32817 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 186b12638a87410487467df721a3cbda; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47
I20260709 13:04:04.149070 11797 consensus_queue.cc:1059] T 5d439326f23747eaab3c1e7da2d6b785 P 186b12638a87410487467df721a3cbda [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:04:04.149677 11724 raft_consensus.cc:2749] T 2567c2d84e4648d7ba5dc23a3df32817 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:04:04.083618 11100 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7757ecbf402e47bfbc93d0447741e26c" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:04.150619 11100 raft_consensus.cc:2468] T 7757ecbf402e47bfbc93d0447741e26c P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 1.
I20260709 13:04:04.145661 11021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bac7e5040f984ba2b7146bd36af7267e" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:04.151655 11100 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a610d1a71f52432e88fe56130f921fa2" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 4 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
W20260709 13:04:04.151714 10843 leader_election.cc:343] T 30f628e3d7914d609e7f1d980a74c654 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:04:04.152563 11100 raft_consensus.cc:2393] T a610d1a71f52432e88fe56130f921fa2 P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 186b12638a87410487467df721a3cbda in current term 4: Already voted for candidate 0172fed22b6b4890a7fdc953fa128af4 in this term.
I20260709 13:04:04.085331 11754 leader_election.cc:290] T acc241e7273d49149984968596242712 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:04.048389 11695 consensus_queue.cc:237] T 836251ebc2b74f5da88b8a8b2338e2a5 P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:04.146684 10880 rpcz_store.cc:276] 0709 13:04:02.958327 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:02.961420 (+  3093us) service_pool.cc:224] Handling call
0709 13:04:04.006255 (+1044835us) spinlock_profiling.cc:243] Waited 1.43 ms on lock 0x7b5800000098. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f426303db71 00007f426303da22 00007f426bf56c09 00007f426b0b0ee0 00007f426b17a0ab 00007f426b17a131 00007f426b1280a0 00007f426b127984 00007f42646a26d4 00007f42646a2662
0709 13:04:04.146149 (+139894us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3282688}
I20260709 13:04:04.153481 11026 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5ba72ccd854430baadbc636dfd17136" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
I20260709 13:04:04.154024 10880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87cd8f5097624c8692f1d83544d8881e" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:04.154450 11026 raft_consensus.cc:2410] T e5ba72ccd854430baadbc636dfd17136 P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 0172fed22b6b4890a7fdc953fa128af4 for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 13:04:04.041565 11639 raft_consensus.cc:515] T e5ba72ccd854430baadbc636dfd17136 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:04.155457 10880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e96e80ba9ac34d0998395515e5c8c41a" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:04.076157 11658 consensus_queue.cc:1059] T 6a9353d1d230416296627d7cf8b1d425 P 186b12638a87410487467df721a3cbda [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.011s
I20260709 13:04:04.104599 11152 raft_consensus.cc:740] T b67eff0634e540eab91cf8cee795969f P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0172fed22b6b4890a7fdc953fa128af4, State: Initialized, Role: FOLLOWER
I20260709 13:04:04.150094 11797 raft_consensus.cc:3055] T 5d439326f23747eaab3c1e7da2d6b785 P 186b12638a87410487467df721a3cbda [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:04:04.156587 11653 raft_consensus.cc:515] T ff13ae9a47c7455f962406c6d131a3ff P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:04.156857 11658 raft_consensus.cc:3055] T 6a9353d1d230416296627d7cf8b1d425 P 186b12638a87410487467df721a3cbda [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:04:04.157812 11658 raft_consensus.cc:740] T 6a9353d1d230416296627d7cf8b1d425 P 186b12638a87410487467df721a3cbda [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 186b12638a87410487467df721a3cbda, State: Running, Role: LEADER
I20260709 13:04:04.157538 11152 consensus_queue.cc:260] T b67eff0634e540eab91cf8cee795969f P 0172fed22b6b4890a7fdc953fa128af4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:04.157912 11022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f371d2565dd84667b5613848604949bb" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:04.070281 11103 raft_consensus.cc:3060] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:04:04.158845 11658 consensus_queue.cc:260] T 6a9353d1d230416296627d7cf8b1d425 P 186b12638a87410487467df721a3cbda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:04.160176 11658 raft_consensus.cc:3060] T 6a9353d1d230416296627d7cf8b1d425 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:04:04.108394 11029 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a64b7e99d6c54934a7de6f6dffcf5e94" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
I20260709 13:04:04.161357 11029 raft_consensus.cc:2410] T a64b7e99d6c54934a7de6f6dffcf5e94 P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 0172fed22b6b4890a7fdc953fa128af4 for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 13:04:04.157120 11797 raft_consensus.cc:740] T 5d439326f23747eaab3c1e7da2d6b785 P 186b12638a87410487467df721a3cbda [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 186b12638a87410487467df721a3cbda, State: Running, Role: LEADER
I20260709 13:04:04.162372 11029 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f1466124dbd4f6ebb3c4d62ca48bf96" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:04.091605 10872 rpcz_store.cc:276] 0709 13:04:02.957285 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:02.960959 (+  3674us) service_pool.cc:224] Handling call
0709 13:04:04.090215 (+1129256us) spinlock_profiling.cc:243] Waited 4.95 ms on lock 0x7b5800000098. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f426303db71 00007f426303da22 00007f426bf56c09 00007f426b0b0ee0 00007f426b17a0ab 00007f426b17a131 00007f426b13bfc0 00007f426b126929 00007f426b1263d2 00007f426469dbd4
0709 13:04:04.090519 (+   304us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:04.090781 (+   262us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:04.090811 (+    30us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:04.090863 (+    52us) raft_consensus.cc:1645] Marking committed up to 1
0709 13:04:04.090903 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:04.090964 (+    61us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:04.091078 (+   114us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":11378432}
I20260709 13:04:04.162995 10951 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f9df2bc281f4866b6da5da94a0defc7" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:04.163196 11797 consensus_queue.cc:260] T 5d439326f23747eaab3c1e7da2d6b785 P 186b12638a87410487467df721a3cbda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:04.164204 10872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "732c4adeeff6422497aa7db4a82f535b" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:04.164628 11797 raft_consensus.cc:3060] T 5d439326f23747eaab3c1e7da2d6b785 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:04:04.118235 11743 leader_election.cc:290] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:04.117551 11758 raft_consensus.cc:2749] T 9a1716c0adf34bee8a4e99341ba5788f P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 13:04:04.164898 10872 raft_consensus.cc:2468] T 732c4adeeff6422497aa7db4a82f535b P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 1.
I20260709 13:04:04.148384 11792 raft_consensus.cc:2749] T d7ce47b9a92d43dd88d95bcda5038597 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:04:04.090539 10949 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f1466124dbd4f6ebb3c4d62ca48bf96" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:04.166826 10872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57b6a2418c1342e7924195e9c1dbd357" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
W20260709 13:04:04.146489 10948 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49424 (request call id 136) took 2339 ms (client timeout 1778 ms). Trace:
I20260709 13:04:04.094019 10947 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75492f86fde046c3900df3eac8292fd4" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
W20260709 13:04:04.167470 10948 rpcz_store.cc:269] 0709 13:04:01.807134 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:01.809998 (+  2864us) service_pool.cc:224] Handling call
0709 13:04:04.146252 (+2336254us) spinlock_profiling.cc:243] Waited 1.21 s on lock 0x7b540078a354. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:04:04.146383 (+   131us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2791979776}
I20260709 13:04:04.125720 11023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7839ca2d2c13431a85f44d7c458c6351" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:04.160187 11152 ts_tablet_manager.cc:1434] T b67eff0634e540eab91cf8cee795969f P 0172fed22b6b4890a7fdc953fa128af4: Time spent starting tablet: real 0.180s	user 0.007s	sys 0.002s
I20260709 13:04:04.163252 11029 raft_consensus.cc:2393] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0172fed22b6b4890a7fdc953fa128af4 in current term 4: Already voted for candidate 873357742d5843579556ee8166f23f47 in this term.
I20260709 13:04:04.169698 11029 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e89e640169e14e2d98f968f76027daa7" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:04.127205 11025 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff13ae9a47c7455f962406c6d131a3ff" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:04.170821 11029 raft_consensus.cc:2410] T e89e640169e14e2d98f968f76027daa7 P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 10ca5ee68ad64b3283b9d235352c9d23 for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 13:04:04.170852 11025 raft_consensus.cc:2368] T ff13ae9a47c7455f962406c6d131a3ff P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0172fed22b6b4890a7fdc953fa128af4 for earlier term 1. Current term is 2.
I20260709 13:04:04.144677 11028 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e4f2241a94a41f38e481137f868b83d" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:04.171761 11757 leader_election.cc:290] T d71ca3a1ba6b48478aaf62741bf3cf6a P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:04.172194 11029 raft_consensus.cc:1240] T 4b37f8fca8dc42718b2a42f0883a956f P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Rejecting Update request from peer 10ca5ee68ad64b3283b9d235352c9d23 for earlier term 1. Current term is 2. Ops: []
I20260709 13:04:04.149089 11729 raft_consensus.cc:515] T a977842ba4b744f4bb2febd55d286006 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:04.152740 10843 leader_election.cc:304] T 7839ca2d2c13431a85f44d7c458c6351 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda; no voters: 
I20260709 13:04:04.174053 11597 consensus_queue.cc:1059] T 4b37f8fca8dc42718b2a42f0883a956f P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:04:04.174821 11029 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5ba72ccd854430baadbc636dfd17136" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
I20260709 13:04:04.174959 11020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555217d45d9b4ee492dbb3ddaf0f7c02" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:04.175283 11597 raft_consensus.cc:3055] T 4b37f8fca8dc42718b2a42f0883a956f P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:04:04.175514 11729 leader_election.cc:290] T a977842ba4b744f4bb2febd55d286006 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:04.171962 11025 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4cd3d7dfbbf4096908ff11e9dc57141" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:04.128103 11096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c160ddfe9b34bdaa3835d0915c1e06f" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:04.135591 11508 leader_election.cc:290] T e89e640169e14e2d98f968f76027daa7 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 3 election: Requested vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:04.135516 11099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bac7e5040f984ba2b7146bd36af7267e" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:04.169638 10873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a9353d1d230416296627d7cf8b1d425" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:04.176654 11020 raft_consensus.cc:1240] T d371c5e27bd843469d2e2e453734cb4a P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Rejecting Update request from peer 0172fed22b6b4890a7fdc953fa128af4 for earlier term 1. Current term is 2. Ops: [1.1-1.1]
I20260709 13:04:04.198040 11099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5ba72ccd854430baadbc636dfd17136" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:04:04.198273 11022 raft_consensus.cc:2468] T f371d2565dd84667b5613848604949bb P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 2.
I20260709 13:04:04.198971 11099 raft_consensus.cc:2393] T e5ba72ccd854430baadbc636dfd17136 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 10ca5ee68ad64b3283b9d235352c9d23 in current term 3: Already voted for candidate 0172fed22b6b4890a7fdc953fa128af4 in this term.
I20260709 13:04:04.199560 11020 raft_consensus.cc:1275] T 0e5d327630e5438b878035d28efe6c0c P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Refusing update from remote peer 0172fed22b6b4890a7fdc953fa128af4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 13:04:04.157369 11639 leader_election.cc:290] T e5ba72ccd854430baadbc636dfd17136 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 3 election: Requested vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:04.175769 11597 raft_consensus.cc:740] T 4b37f8fca8dc42718b2a42f0883a956f P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 10ca5ee68ad64b3283b9d235352c9d23, State: Running, Role: LEADER
I20260709 13:04:04.188195 11067 leader_election.cc:304] T a610d1a71f52432e88fe56130f921fa2 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47; no voters: 186b12638a87410487467df721a3cbda
I20260709 13:04:04.137004 11102 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "836251ebc2b74f5da88b8a8b2338e2a5" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:04.080761 10874 raft_consensus.cc:2410] T 9a1716c0adf34bee8a4e99341ba5788f P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 0172fed22b6b4890a7fdc953fa128af4 for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 13:04:04.076964 11782 raft_consensus.cc:515] T 0e5d327630e5438b878035d28efe6c0c P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:04.084719 11097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "525bfdbd6d934afd823f7d3ee3f9d052" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:04.178364 11025 raft_consensus.cc:2468] T b4cd3d7dfbbf4096908ff11e9dc57141 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 2.
I20260709 13:04:04.164564 10951 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff13ae9a47c7455f962406c6d131a3ff" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:04.094100 11748 raft_consensus.cc:3060] T a5299b8d044f41b7b68057e63d17b159 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:04:04.167230 10949 raft_consensus.cc:2468] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 3.
I20260709 13:04:04.210196 10871 raft_consensus.cc:2468] T e5a5f5a608504650b48fd27f163faed6 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 2.
I20260709 13:04:04.231204 11782 leader_election.cc:290] T 0e5d327630e5438b878035d28efe6c0c P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:04.231384 10949 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e89e640169e14e2d98f968f76027daa7" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:04:04.231719 10871 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f9eb06ed01f4ce68be3af8a8434665b" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:04.156848 10880 raft_consensus.cc:2393] T e96e80ba9ac34d0998395515e5c8c41a P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 10ca5ee68ad64b3283b9d235352c9d23 in current term 3: Already voted for candidate 873357742d5843579556ee8166f23f47 in this term.
I20260709 13:04:04.167960 10947 raft_consensus.cc:3060] T 75492f86fde046c3900df3eac8292fd4 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:04:04.169821 11023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5299b8d044f41b7b68057e63d17b159" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
I20260709 13:04:04.174307 11545 raft_consensus.cc:2804] T 7839ca2d2c13431a85f44d7c458c6351 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:04:04.176961 11777 raft_consensus.cc:493] T b67eff0634e540eab91cf8cee795969f P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:04.112648 11069 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:38967, user_credentials={real_user=slave}} blocked reactor thread for 45096.4us
W20260709 13:04:04.245939 11069 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:04.181285 10921 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:04.247452 10921 leader_election.cc:304] T 732c4adeeff6422497aa7db4a82f535b P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda; no voters: 873357742d5843579556ee8166f23f47
I20260709 13:04:04.211572 11633 raft_consensus.cc:493] T 63d44f6ba6434ab2870fe3190f4d7c2f P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 186b12638a87410487467df721a3cbda)
I20260709 13:04:04.127115 10879 raft_consensus.cc:2393] T f371d2565dd84667b5613848604949bb P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0172fed22b6b4890a7fdc953fa128af4 in current term 2: Already voted for candidate 186b12638a87410487467df721a3cbda in this term.
I20260709 13:04:04.119442 10950 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5299b8d044f41b7b68057e63d17b159" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:04:04.271302 10879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c31881534c5e4f0089c82057c1746652" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:04:04.272671 10879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d439326f23747eaab3c1e7da2d6b785" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:04:04.239495 11097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a977842ba4b744f4bb2febd55d286006" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:04.128934 11796 leader_election.cc:290] T a1ec2b50c6ad4f7c8033476208abb26d P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:04.121559 11795 leader_election.cc:290] T 7709387f4d3d4f9992d97a0fbca39a40 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:04.182065 10920 leader_election.cc:304] T 6f46a5102fd64b45bf1c4dff863d2b4d P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23; no voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47
I20260709 13:04:04.274616 11095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d71ca3a1ba6b48478aaf62741bf3cf6a" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
W20260709 13:04:04.275548 10920 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 93595.5us
I20260709 13:04:04.255851 10946 raft_consensus.cc:2468] T 4c2a35bb99374e93b573cf870d04faaa P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 3.
I20260709 13:04:04.277015 11152 tablet_bootstrap.cc:492] T 401a9cf11eac479aa445ec8c1d6e33b2 P 0172fed22b6b4890a7fdc953fa128af4: Bootstrap starting.
I20260709 13:04:04.277843 10946 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f1466124dbd4f6ebb3c4d62ca48bf96" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:04:04.176776 10843 leader_election.cc:304] T 0f9df2bc281f4866b6da5da94a0defc7 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 186b12638a87410487467df721a3cbda; no voters: 0172fed22b6b4890a7fdc953fa128af4, 10ca5ee68ad64b3283b9d235352c9d23
I20260709 13:04:04.185046 10875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c51782cd9d064ec0949e84e24dd4de67" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:04.278625 11779 raft_consensus.cc:2749] T 6f46a5102fd64b45bf1c4dff863d2b4d P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:04:04.278780 10953 raft_consensus.cc:1240] T 4c2a35bb99374e93b573cf870d04faaa P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Rejecting Update request from peer 186b12638a87410487467df721a3cbda for earlier term 2. Current term is 3. Ops: []
I20260709 13:04:04.279029 10875 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51696 (request call id 154) took 1322 ms. Trace:
W20260709 13:04:04.279220 10843 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 102557us
I20260709 13:04:04.279531 10875 rpcz_store.cc:276] 0709 13:04:02.956292 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:02.961051 (+  4759us) service_pool.cc:224] Handling call
0709 13:04:04.184748 (+1223697us) spinlock_profiling.cc:243] Waited 4.82 ms on lock 0x7b5800000098. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f426303db71 00007f426303da22 00007f426bf56c09 00007f426b0b0ee0 00007f426b17a0ab 00007f426b17a131 00007f426b1280a0 00007f426b127984 00007f42646a26d4 00007f42646a2662
0709 13:04:04.278955 (+ 94207us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":11082496}
I20260709 13:04:04.186333 11149 tablet_bootstrap.cc:492] T a977842ba4b744f4bb2febd55d286006 P 873357742d5843579556ee8166f23f47: Bootstrap starting.
I20260709 13:04:04.280802 10875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b37f8fca8dc42718b2a42f0883a956f" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:04:04.281464 10875 raft_consensus.cc:3060] T 4b37f8fca8dc42718b2a42f0883a956f P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:04:04.281970 11795 raft_consensus.cc:2749] T 0f9df2bc281f4866b6da5da94a0defc7 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:04:04.283442 11069 leader_election.cc:304] T 7757ecbf402e47bfbc93d0447741e26c P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda; no voters: 873357742d5843579556ee8166f23f47
I20260709 13:04:04.284354 11610 raft_consensus.cc:2804] T 7757ecbf402e47bfbc93d0447741e26c P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:04:04.284765 11610 raft_consensus.cc:493] T 7757ecbf402e47bfbc93d0447741e26c P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:04.285179 11610 raft_consensus.cc:3060] T 7757ecbf402e47bfbc93d0447741e26c P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:04:04.189003 10992 leader_election.cc:304] T a5299b8d044f41b7b68057e63d17b159 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47; no voters: 
W20260709 13:04:04.286875 10992 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 98011.2us
I20260709 13:04:04.191642 10844 leader_election.cc:304] T 401a9cf11eac479aa445ec8c1d6e33b2 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47; no voters: 0172fed22b6b4890a7fdc953fa128af4
I20260709 13:04:04.197778 10954 raft_consensus.cc:2368] T e96e80ba9ac34d0998395515e5c8c41a P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 186b12638a87410487467df721a3cbda for earlier term 2. Current term is 3.
I20260709 13:04:04.148430 11688 raft_consensus.cc:2749] T 57b6a2418c1342e7924195e9c1dbd357 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:04:04.202394 11102 raft_consensus.cc:2368] T 836251ebc2b74f5da88b8a8b2338e2a5 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 186b12638a87410487467df721a3cbda for earlier term 2. Current term is 3.
I20260709 13:04:04.172019 11028 raft_consensus.cc:2468] T 1e4f2241a94a41f38e481137f868b83d P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 1.
I20260709 13:04:04.142336 10876 raft_consensus.cc:3060] T 868e92339c2f4508af0afbd07c814d64 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:04:04.204463 11792 raft_consensus.cc:493] T acc241e7273d49149984968596242712 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:04.204825 11597 consensus_queue.cc:260] T 4b37f8fca8dc42718b2a42f0883a956f P 10ca5ee68ad64b3283b9d235352c9d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:04.110710 10877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5de5f0581e3b41dea729363d6ca563bb" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:04:04.205672 11096 raft_consensus.cc:2468] T 8c160ddfe9b34bdaa3835d0915c1e06f P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 1.
I20260709 13:04:04.206607 11653 leader_election.cc:290] T ff13ae9a47c7455f962406c6d131a3ff P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 2 election: Requested vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:04.212175 11098 raft_consensus.cc:2393] T 9a1716c0adf34bee8a4e99341ba5788f P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 186b12638a87410487467df721a3cbda in current term 3: Already voted for candidate 0172fed22b6b4890a7fdc953fa128af4 in this term.
I20260709 13:04:04.155431 11026 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "868e92339c2f4508af0afbd07c814d64" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
W20260709 13:04:04.288223 10844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:45807, user_credentials={real_user=slave}} blocked reactor thread for 96721.3us
I20260709 13:04:04.313844 10775 catalog_manager.cc:5697] T 63d44f6ba6434ab2870fe3190f4d7c2f P 186b12638a87410487467df721a3cbda reported cstate change: term changed from 1 to 2, leader changed from 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194) to 186b12638a87410487467df721a3cbda (127.0.107.193). New cstate: current_term: 2 leader_uuid: "186b12638a87410487467df721a3cbda" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } health_report { overall_health: UNKNOWN } } }
W20260709 13:04:04.292821 10995 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:38967, user_credentials={real_user=slave}} blocked reactor thread for 78898us
I20260709 13:04:04.211055 11067 leader_election.cc:304] T a64b7e99d6c54934a7de6f6dffcf5e94 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47
I20260709 13:04:04.086706 10878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c2a35bb99374e93b573cf870d04faaa" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:04:04.214389 10955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87cd8f5097624c8692f1d83544d8881e" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:04:04.212795 10874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e4f2241a94a41f38e481137f868b83d" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:04.338572 10878 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51718 (request call id 185) took 1252 ms. Trace:
I20260709 13:04:04.339071 10878 rpcz_store.cc:276] 0709 13:04:03.108197 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:04.086415 (+978218us) service_pool.cc:224] Handling call
0709 13:04:04.338496 (+252081us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:04.199916 11099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "868e92339c2f4508af0afbd07c814d64" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:04.332809 10844 leader_election.cc:304] T 9a1716c0adf34bee8a4e99341ba5788f P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:04:04.288040 11796 raft_consensus.cc:2804] T 401a9cf11eac479aa445ec8c1d6e33b2 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:04:04.351814 10921 leader_election.cc:304] T e96e80ba9ac34d0998395515e5c8c41a P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23; no voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47
I20260709 13:04:04.281383 10953 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b37f8fca8dc42718b2a42f0883a956f" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:04:04.236608 11029 raft_consensus.cc:2468] T e5ba72ccd854430baadbc636dfd17136 P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 3.
I20260709 13:04:04.214329 11757 raft_consensus.cc:2804] T a610d1a71f52432e88fe56130f921fa2 P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:04:04.242700 10919 leader_election.cc:304] T b4cd3d7dfbbf4096908ff11e9dc57141 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47; no voters: 
W20260709 13:04:04.402666 10919 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:45807, user_credentials={real_user=slave}} blocked reactor thread for 160071us
I20260709 13:04:04.403995 10919 leader_election.cc:304] T 1e4f2241a94a41f38e481137f868b83d P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:04:04.255760 10951 raft_consensus.cc:1240] T 8a69f32c53f24a50b7852d143665d7b4 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Rejecting Update request from peer 186b12638a87410487467df721a3cbda for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260709 13:04:04.256444 11729 raft_consensus.cc:2804] T 732c4adeeff6422497aa7db4a82f535b P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:04:04.256803 11518 consensus_queue.cc:260] T 8a69f32c53f24a50b7852d143665d7b4 P 186b12638a87410487467df721a3cbda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:04.237298 11021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a977842ba4b744f4bb2febd55d286006" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:04.258569 11777 raft_consensus.cc:515] T b67eff0634e540eab91cf8cee795969f P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:04.259163 11633 raft_consensus.cc:515] T 63d44f6ba6434ab2870fe3190f4d7c2f P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:04.259462 10880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a69f32c53f24a50b7852d143665d7b4" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:04:04.241806 11545 raft_consensus.cc:493] T 7839ca2d2c13431a85f44d7c458c6351 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Starting leader election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:04.260170 10845 leader_election.cc:304] T f371d2565dd84667b5613848604949bb P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda; no voters: 873357742d5843579556ee8166f23f47
I20260709 13:04:04.266481 11748 raft_consensus.cc:2749] T a5299b8d044f41b7b68057e63d17b159 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:04:04.238927 11027 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a9353d1d230416296627d7cf8b1d425" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
I20260709 13:04:04.267870 10994 leader_election.cc:400] T b4cd3d7dfbbf4096908ff11e9dc57141 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 2 election: Vote denied by peer 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507) with higher term. Message: Invalid argument: T b4cd3d7dfbbf4096908ff11e9dc57141 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 873357742d5843579556ee8166f23f47 for earlier term 2. Current term is 3.
I20260709 13:04:04.422971 10994 leader_election.cc:403] T b4cd3d7dfbbf4096908ff11e9dc57141 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20260709 13:04:04.408355 11817 raft_consensus.cc:2804] T b4cd3d7dfbbf4096908ff11e9dc57141 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:04:04.252007 11023 raft_consensus.cc:3060] T a5299b8d044f41b7b68057e63d17b159 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:04:04.271054 11101 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37676 (request call id 155) took 1360 ms. Trace:
I20260709 13:04:04.408949 11729 raft_consensus.cc:493] T 732c4adeeff6422497aa7db4a82f535b P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:04.084596 11553 raft_consensus.cc:515] T 72b111ad7b5f4753ab1d2b5ee1114840 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:04.312595 10920 leader_election.cc:304] T 32147c6d2d5a477aa38c597c7f78df5b P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23; no voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47
W20260709 13:04:04.335048 11067 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:45807, user_credentials={real_user=slave}} blocked reactor thread for 124096us
I20260709 13:04:04.434894 11067 leader_election.cc:304] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47
I20260709 13:04:04.287533 11069 leader_election.cc:304] T c31881534c5e4f0089c82057c1746652 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda
W20260709 13:04:04.436270 11069 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:38967, user_credentials={real_user=slave}} blocked reactor thread for 148816us
I20260709 13:04:04.437664 11069 leader_election.cc:304] T 8c160ddfe9b34bdaa3835d0915c1e06f P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4; no voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47
I20260709 13:04:04.334547 10955 raft_consensus.cc:2376] T 87cd8f5097624c8692f1d83544d8881e P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 186b12638a87410487467df721a3cbda in term 3. Re-sending same reply.
W20260709 13:04:04.434922 10920 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 122440us
I20260709 13:04:04.440038 10920 leader_election.cc:304] T bac7e5040f984ba2b7146bd36af7267e P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23; no voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47
I20260709 13:04:04.153515 11100 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c31881534c5e4f0089c82057c1746652" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:04:04.339720 10776 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:59610 (request call id 77) took 5007 ms. Trace:
I20260709 13:04:04.340466 10946 raft_consensus.cc:3060] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:04:04.320180 11095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f1466124dbd4f6ebb3c4d62ca48bf96" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:04:04.362875 10878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e89e640169e14e2d98f968f76027daa7" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:04.366259 10874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f9eb06ed01f4ce68be3af8a8434665b" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:04.370508 10876 raft_consensus.cc:2468] T 868e92339c2f4508af0afbd07c814d64 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 2.
I20260709 13:04:04.443758 10776 rpcz_store.cc:276] 0709 13:03:59.311983 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:59.312361 (+   378us) service_pool.cc:224] Handling call
0709 13:04:04.317400 (+5005039us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 13:04:02.438202 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 13:04:02.438936 (+   734us) write_op.cc:432] Acquiring schema lock in shared mode
0709 13:04:02.438982 (+    46us) write_op.cc:435] Acquired schema lock
0709 13:04:02.439006 (+    24us) tablet.cc:662] Decoding operations
0709 13:04:02.439635 (+   629us) write_op.cc:620] Acquiring the partition lock for write op
0709 13:04:02.439738 (+   103us) write_op.cc:641] Partition lock acquired for write op
0709 13:04:02.439772 (+    34us) tablet.cc:685] Acquiring locks for 2 operations
0709 13:04:02.440130 (+   358us) tablet.cc:701] Row locks acquired
0709 13:04:02.440145 (+    15us) write_op.cc:260] PREPARE: finished
0709 13:04:02.440324 (+   179us) write_op.cc:270] Start()
0709 13:04:02.440453 (+   129us) write_op.cc:276] Timestamp: P: 1783602242440266 usec, L: 0
0709 13:04:02.440472 (+    19us) op_driver.cc:348] REPLICATION: starting
0709 13:04:02.441360 (+   888us) log.cc:844] Serialized 928 byte log entry
0709 13:04:03.504691 (+1063331us) op_driver.cc:464] REPLICATION: finished
0709 13:04:03.690011 (+185320us) write_op.cc:301] APPLY: starting
0709 13:04:03.691439 (+  1428us) tablet.cc:1367] starting BulkCheckPresence
0709 13:04:03.747783 (+ 56344us) tablet.cc:1370] finished BulkCheckPresence
0709 13:04:03.830476 (+ 82693us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 13:04:03.864547 (+ 34071us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 13:04:03.868233 (+  3686us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0709 13:04:03.879278 (+ 11045us) write_op.cc:312] APPLY: finished
0709 13:04:03.901208 (+ 21930us) log.cc:844] Serialized 28 byte log entry
0709 13:04:03.907632 (+  6424us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:04:03.907806 (+   174us) write_op.cc:454] Released schema lock
0709 13:04:03.907876 (+    70us) write_op.cc:341] FINISH: Updating metrics
0709 13:04:03.930751 (+ 22875us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:04:03.930799 (+    48us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":136232,"reported_tablets":10,"child_traces":[["op",{"apply.queue_time_us":82203,"apply.run_cpu_time_us":3839,"apply.run_wall_time_us":254927,"num_ops":2,"prepare.queue_time_us":171359,"prepare.run_cpu_time_us":32479,"prepare.run_wall_time_us":343515,"raft.queue_time_us":203015,"raft.run_cpu_time_us":17138,"raft.run_wall_time_us":55063,"replication_time_us":1064032,"thread_start_us":556383,"threads_started":4,"wal-append.queue_time_us":771315}]]}
I20260709 13:04:04.338825 11754 raft_consensus.cc:2749] T a64b7e99d6c54934a7de6f6dffcf5e94 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 13:04:04.319692 11758 raft_consensus.cc:493] T d7ce47b9a92d43dd88d95bcda5038597 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 10ca5ee68ad64b3283b9d235352c9d23)
I20260709 13:04:04.352027 11796 raft_consensus.cc:493] T 401a9cf11eac479aa445ec8c1d6e33b2 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:04.347026 11607 raft_consensus.cc:493] T b67eff0634e540eab91cf8cee795969f P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:04.373222 11149 tablet_bootstrap.cc:654] T a977842ba4b744f4bb2febd55d286006 P 873357742d5843579556ee8166f23f47: Neither blocks nor log segments found. Creating new log.
I20260709 13:04:04.377770 10947 raft_consensus.cc:2468] T 75492f86fde046c3900df3eac8292fd4 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 1.
I20260709 13:04:04.460201 10952 raft_consensus.cc:1275] T 11de93cd9b9d457489ff1cb80f976c9a P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Refusing update from remote peer 873357742d5843579556ee8166f23f47: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 13:04:04.342017 10877 raft_consensus.cc:2376] T 5de5f0581e3b41dea729363d6ca563bb P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 3. Re-sending same reply.
I20260709 13:04:04.358385 11024 raft_consensus.cc:1240] T a154cc8b181f460abf4880064379948b P 873357742d5843579556ee8166f23f47 [term 2 LEADER]: Rejecting Update request from peer 10ca5ee68ad64b3283b9d235352c9d23 for earlier term 1. Current term is 2. Ops: []
I20260709 13:04:04.386250 11782 raft_consensus.cc:493] T 11de93cd9b9d457489ff1cb80f976c9a P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:04.359114 11597 raft_consensus.cc:3060] T 4b37f8fca8dc42718b2a42f0883a956f P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:04:04.391101 11152 tablet_bootstrap.cc:654] T 401a9cf11eac479aa445ec8c1d6e33b2 P 0172fed22b6b4890a7fdc953fa128af4: Neither blocks nor log segments found. Creating new log.
I20260709 13:04:04.391963 11094 raft_consensus.cc:2468] T 5d439326f23747eaab3c1e7da2d6b785 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 3.
I20260709 13:04:04.392385 10879 raft_consensus.cc:2410] T 5d439326f23747eaab3c1e7da2d6b785 P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 10ca5ee68ad64b3283b9d235352c9d23 for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 13:04:04.393066 10875 raft_consensus.cc:2468] T 4b37f8fca8dc42718b2a42f0883a956f P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 2.
I20260709 13:04:04.392959 11028 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555217d45d9b4ee492dbb3ddaf0f7c02" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:04.382118 11098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7709387f4d3d4f9992d97a0fbca39a40" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:04.406359 11102 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1ec2b50c6ad4f7c8033476208abb26d" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:04.348027 11795 raft_consensus.cc:2764] T 9a1716c0adf34bee8a4e99341ba5788f P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260709 13:04:04.404932 10919 leader_election.cc:304] T e5ba72ccd854430baadbc636dfd17136 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:04:04.354229 10954 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49428 (request call id 85) took 1066 ms. Trace:
I20260709 13:04:04.356228 11639 raft_consensus.cc:2749] T e96e80ba9ac34d0998395515e5c8c41a P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:04:04.418231 11029 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4cd3d7dfbbf4096908ff11e9dc57141" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
I20260709 13:04:04.233511 10871 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65bf0a62f4d84ef18dd6b23e082724b8" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:04:04.407166 11734 raft_consensus.cc:493] T a64b7e99d6c54934a7de6f6dffcf5e94 P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:04.425028 10994 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 171375us
I20260709 13:04:04.511601 10994 leader_election.cc:304] T 4f9eb06ed01f4ce68be3af8a8434665b P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 873357742d5843579556ee8166f23f47; no voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda
I20260709 13:04:04.512959 10994 leader_election.cc:304] T 75492f86fde046c3900df3eac8292fd4 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:04:04.408555 11779 raft_consensus.cc:2804] T 1e4f2241a94a41f38e481137f868b83d P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:04:04.417663 11026 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5ba72ccd854430baadbc636dfd17136" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
I20260709 13:04:04.410605 10872 raft_consensus.cc:3060] T 57b6a2418c1342e7924195e9c1dbd357 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Advancing to term 2
W20260709 13:04:04.428138 10992 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 140681us
I20260709 13:04:04.514925 10992 leader_election.cc:304] T 2567c2d84e4648d7ba5dc23a3df32817 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 873357742d5843579556ee8166f23f47; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda
I20260709 13:04:04.515002 10995 leader_election.cc:304] T 4b37f8fca8dc42718b2a42f0883a956f P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:04:04.428018 11101 rpcz_store.cc:276] 0709 13:04:02.910377 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:03.702042 (+791665us) service_pool.cc:224] Handling call
0709 13:04:03.702351 (+   309us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:03.707509 (+  5158us) raft_consensus.cc:1534] Early marking committed up to index 2
0709 13:04:03.875768 (+168259us) log.cc:844] Serialized 14 byte log entry
0709 13:04:03.919741 (+ 43973us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:03.919807 (+    66us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:04:03.919870 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:04.168237 (+248367us) spinlock_profiling.cc:243] Waited 8.91 ms on lock 0x7b18001142e0. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f4263075522 00007f42630777c1 00007f4263077779 00007f4263077709 00007f42630776d1 00007f42630769cd 000000000038f354 000000000038f199 00007f426305883a
0709 13:04:04.270791 (+102554us) spinlock_profiling.cc:243] Waited 4.89 ms on lock 0x7b54007c2c54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:04:04.270852 (+    61us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:04.270987 (+   135us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":209320,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":52,"raft.queue_time_us":186,"raft.run_cpu_time_us":152,"raft.run_wall_time_us":163,"spinlock_wait_cycles":31742464,"thread_start_us":205037,"threads_started":2,"wal-append.queue_time_us":111227}
I20260709 13:04:04.516304 11101 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4cd3d7dfbbf4096908ff11e9dc57141" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:04.517196 11101 raft_consensus.cc:2393] T b4cd3d7dfbbf4096908ff11e9dc57141 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 10ca5ee68ad64b3283b9d235352c9d23 in current term 3: Already voted for candidate 0172fed22b6b4890a7fdc953fa128af4 in this term.
I20260709 13:04:04.274677 11097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555217d45d9b4ee492dbb3ddaf0f7c02" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:04.430336 11148 tablet_bootstrap.cc:492] T 30f628e3d7914d609e7f1d980a74c654 P 10ca5ee68ad64b3283b9d235352c9d23: No bootstrap required, opened a new log
I20260709 13:04:04.527804 11148 ts_tablet_manager.cc:1403] T 30f628e3d7914d609e7f1d980a74c654 P 10ca5ee68ad64b3283b9d235352c9d23: Time spent bootstrapping tablet: real 0.955s	user 0.010s	sys 0.017s
I20260709 13:04:04.343479 11792 raft_consensus.cc:515] T acc241e7273d49149984968596242712 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:04.415455 11757 raft_consensus.cc:697] T a610d1a71f52432e88fe56130f921fa2 P 0172fed22b6b4890a7fdc953fa128af4 [term 4 LEADER]: Becoming Leader. State: Replica: 0172fed22b6b4890a7fdc953fa128af4, State: Running, Role: LEADER
I20260709 13:04:04.441515 11069 leader_election.cc:304] T 87cd8f5097624c8692f1d83544d8881e P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda
I20260709 13:04:04.496292 11553 leader_election.cc:290] T 72b111ad7b5f4753ab1d2b5ee1114840 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 3 election: Requested vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:04.370884 11099 raft_consensus.cc:2393] T 868e92339c2f4508af0afbd07c814d64 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 186b12638a87410487467df721a3cbda in current term 2: Already voted for candidate 0172fed22b6b4890a7fdc953fa128af4 in this term.
I20260709 13:04:04.508605 11819 raft_consensus.cc:2804] T e5ba72ccd854430baadbc636dfd17136 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:04:04.420050 11545 raft_consensus.cc:3060] T 7839ca2d2c13431a85f44d7c458c6351 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:04:04.420171 11518 raft_consensus.cc:3060] T 8a69f32c53f24a50b7852d143665d7b4 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:04:04.451370 10878 raft_consensus.cc:2468] T e89e640169e14e2d98f968f76027daa7 P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 3.
W20260709 13:04:04.448163 10826 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 13:04:04.462241 10955 raft_consensus.cc:1240] T d71ca3a1ba6b48478aaf62741bf3cf6a P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Rejecting Update request from peer 873357742d5843579556ee8166f23f47 for earlier term 1. Current term is 2. Ops: []
I20260709 13:04:04.429181 11817 raft_consensus.cc:493] T b4cd3d7dfbbf4096908ff11e9dc57141 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:04.431875 11804 raft_consensus.cc:3060] T b4cd3d7dfbbf4096908ff11e9dc57141 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:04:04.435130 11688 raft_consensus.cc:2804] T f371d2565dd84667b5613848604949bb P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:04:04.435791 11729 raft_consensus.cc:3060] T 732c4adeeff6422497aa7db4a82f535b P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:04:04.379009 11096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff13ae9a47c7455f962406c6d131a3ff" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:04:04.459013 11628 raft_consensus.cc:3060] T c31881534c5e4f0089c82057c1746652 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:04:04.460889 11748 raft_consensus.cc:2749] T 32147c6d2d5a477aa38c597c7f78df5b P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:04:04.450937 11095 raft_consensus.cc:3060] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:04:04.455924 11027 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555217d45d9b4ee492dbb3ddaf0f7c02" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:04.454032 10876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a69f32c53f24a50b7852d143665d7b4" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
W20260709 13:04:04.473482 10993 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:04.599474 11151 tablet_bootstrap.cc:492] T f34f2bc02d85444f9096e0f5b4fe4331 P 186b12638a87410487467df721a3cbda: No bootstrap required, opened a new log
I20260709 13:04:04.586606 11026 raft_consensus.cc:2393] T e5ba72ccd854430baadbc636dfd17136 P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 0172fed22b6b4890a7fdc953fa128af4 in current term 3: Already voted for candidate 10ca5ee68ad64b3283b9d235352c9d23 in this term.
I20260709 13:04:04.452960 10874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c51782cd9d064ec0949e84e24dd4de67" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:04.446372 11025 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d71ca3a1ba6b48478aaf62741bf3cf6a" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:04.463814 11797 raft_consensus.cc:493] T b67eff0634e540eab91cf8cee795969f P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:04.463244 11508 raft_consensus.cc:493] T 34064dc7bf3747dbb3876dd824ee0688 P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:04.463564 10845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 203517us
I20260709 13:04:04.474887 11796 raft_consensus.cc:3060] T 401a9cf11eac479aa445ec8c1d6e33b2 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:04:04.472961 10951 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff13ae9a47c7455f962406c6d131a3ff" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:04:04.500291 11094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e5d327630e5438b878035d28efe6c0c" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:04.479691 11607 raft_consensus.cc:515] T b67eff0634e540eab91cf8cee795969f P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:04.493026 11024 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46628 (request call id 137) took 1062 ms. Trace:
I20260709 13:04:04.501648 10873 raft_consensus.cc:2410] T 6a9353d1d230416296627d7cf8b1d425 P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0172fed22b6b4890a7fdc953fa128af4 for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 13:04:04.487044 11021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff13ae9a47c7455f962406c6d131a3ff" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
W20260709 13:04:04.503681 10919 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:45807, user_credentials={real_user=slave}} blocked reactor thread for 98815.7us
W20260709 13:04:04.507682 10920 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 66460.4us
I20260709 13:04:04.475211 11758 raft_consensus.cc:515] T d7ce47b9a92d43dd88d95bcda5038597 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:04.494331 11777 leader_election.cc:290] T b67eff0634e540eab91cf8cee795969f P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
W20260709 13:04:04.536230 11069 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:38967, user_credentials={real_user=slave}} blocked reactor thread for 94798us
I20260709 13:04:04.412081 11103 raft_consensus.cc:2468] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 2.
I20260709 13:04:04.502177 10877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b67eff0634e540eab91cf8cee795969f" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
W20260709 13:04:04.608251 11832 log.cc:927] Time spent T a154cc8b181f460abf4880064379948b P 873357742d5843579556ee8166f23f47: Append to log took a long time: real 0.166s	user 0.000s	sys 0.001s
I20260709 13:04:04.619060 11022 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46642 (request call id 70) took 1063 ms. Trace:
I20260709 13:04:04.630685 11025 raft_consensus.cc:2468] T d71ca3a1ba6b48478aaf62741bf3cf6a P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 2.
I20260709 13:04:04.630144 10880 raft_consensus.cc:2410] T 8a69f32c53f24a50b7852d143665d7b4 P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 0172fed22b6b4890a7fdc953fa128af4 for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 13:04:04.646576 11024 rpcz_store.cc:276] 0709 13:04:03.430619 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:03.444187 (+ 13568us) service_pool.cc:224] Handling call
0709 13:04:03.545157 (+100970us) spinlock_profiling.cc:243] Waited 9.46 ms on lock 0x7b58007a2898. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f426303db71 00007f426303da22 00007f426bf56c09 00007f426b0b0ee0 00007f426b17a0ab 00007f426b17a131 00007f426b13bfc0 00007f426b126929 00007f426b1263d2 00007f426469dbd4
0709 13:04:04.185152 (+639995us) spinlock_profiling.cc:243] Waited 1.3 ms on lock 0x7b58007a2898. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f426303df8c 00007f426303def6 0000000000375939 00007f426b0b11e0 00007f426b17a0c4 00007f426b17a131 00007f426b13bfc0 00007f426b126929 00007f426b1263d2 00007f426469dbd4
0709 13:04:04.185311 (+   159us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:04.485833 (+300522us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:04.489757 (+  3924us) spinlock_profiling.cc:243] Waited 162 ms on lock 0x7b540065ed54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:04:04.492943 (+  3186us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":398024832}
I20260709 13:04:04.648240 11758 leader_election.cc:290] T d7ce47b9a92d43dd88d95bcda5038597 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:04.648294 11024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e5d327630e5438b878035d28efe6c0c" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:04.648536 11069 leader_election.cc:304] T e5a5f5a608504650b48fd27f163faed6 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda; no voters: 10ca5ee68ad64b3283b9d235352c9d23
I20260709 13:04:04.648895 11758 raft_consensus.cc:2749] T 87cd8f5097624c8692f1d83544d8881e P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:04:04.649153 11024 raft_consensus.cc:2468] T 0e5d327630e5438b878035d28efe6c0c P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 3.
I20260709 13:04:04.649741 11758 raft_consensus.cc:493] T 6f46a5102fd64b45bf1c4dff863d2b4d P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:04.649887 11069 leader_election.cc:304] T 4f9eb06ed01f4ce68be3af8a8434665b P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4; no voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47
I20260709 13:04:04.604723 10876 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51704 (request call id 121) took 1361 ms. Trace:
I20260709 13:04:04.650797 10876 rpcz_store.cc:276] 0709 13:04:03.242704 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:04.453647 (+1210943us) service_pool.cc:224] Handling call
0709 13:04:04.603682 (+150035us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:04.607685 11797 raft_consensus.cc:515] T b67eff0634e540eab91cf8cee795969f P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:04.651926 10876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e89e640169e14e2d98f968f76027daa7" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:04.652838 10876 raft_consensus.cc:2410] T e89e640169e14e2d98f968f76027daa7 P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 10ca5ee68ad64b3283b9d235352c9d23 for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 13:04:04.653581 11797 leader_election.cc:290] T b67eff0634e540eab91cf8cee795969f P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:04.653753 10876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87cd8f5097624c8692f1d83544d8881e" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:04.654359 10877 raft_consensus.cc:2468] T b67eff0634e540eab91cf8cee795969f P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 0.
I20260709 13:04:04.619621 11099 raft_consensus.cc:1240] T f371d2565dd84667b5613848604949bb P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Rejecting Update request from peer 873357742d5843579556ee8166f23f47 for earlier term 1. Current term is 2. Ops: []
I20260709 13:04:04.655548 11099 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37676 (request call id 159) took 1267 ms. Trace:
I20260709 13:04:04.655932 11099 rpcz_store.cc:276] 0709 13:04:03.387905 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:04.600501 (+1212596us) service_pool.cc:224] Handling call
0709 13:04:04.619340 (+ 18839us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:04.655258 (+ 35918us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:04.655484 (+   226us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:04.656339 10877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acc241e7273d49149984968596242712" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:04.614616 11028 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1ec2b50c6ad4f7c8033476208abb26d" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:04.564705 11102 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d156d6ae2ca4fa59a1e387edf0fea39" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:04:04.658182 11102 raft_consensus.cc:2393] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 186b12638a87410487467df721a3cbda in current term 2: Already voted for candidate 10ca5ee68ad64b3283b9d235352c9d23 in this term.
W20260709 13:04:04.612118 11830 log.cc:927] Time spent T d270b9e5481245d9b5340a454a7cd10b P 10ca5ee68ad64b3283b9d235352c9d23: Append to log took a long time: real 0.058s	user 0.000s	sys 0.001s
I20260709 13:04:04.659343 11102 raft_consensus.cc:1240] T 5d439326f23747eaab3c1e7da2d6b785 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Rejecting Update request from peer 186b12638a87410487467df721a3cbda for earlier term 2. Current term is 3. Ops: []
I20260709 13:04:04.659583 11607 leader_election.cc:290] T b67eff0634e540eab91cf8cee795969f P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:04.637370 11026 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "30f628e3d7914d609e7f1d980a74c654" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:04.660301 11607 raft_consensus.cc:2804] T e5a5f5a608504650b48fd27f163faed6 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:04:04.660697 11607 raft_consensus.cc:493] T e5a5f5a608504650b48fd27f163faed6 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 10ca5ee68ad64b3283b9d235352c9d23)
I20260709 13:04:04.661070 11607 raft_consensus.cc:3060] T e5a5f5a608504650b48fd27f163faed6 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Advancing to term 3
W20260709 13:04:04.569114 10992 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 54275us
I20260709 13:04:04.661511 10880 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51704 (request call id 131) took 1416 ms. Trace:
I20260709 13:04:04.610144 11027 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7709387f4d3d4f9992d97a0fbca39a40" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:04.511150 11098 raft_consensus.cc:2468] T 7709387f4d3d4f9992d97a0fbca39a40 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 3.
I20260709 13:04:04.662425 10873 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51704 (request call id 125) took 1418 ms. Trace:
I20260709 13:04:04.663031 10873 rpcz_store.cc:276] 0709 13:04:03.243885 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:03.993545 (+749660us) service_pool.cc:224] Handling call
0709 13:04:04.169505 (+175960us) spinlock_profiling.cc:243] Waited 4.35 ms on lock 0x7b5800000098. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f426303db71 00007f426303deee 0000000000375939 00007f426b0b11e0 00007f426b17a0c4 00007f426b17a131 00007f426b1280a0 00007f426b127984 00007f42646a26d4 00007f42646a2662
0709 13:04:04.662214 (+492709us) spinlock_profiling.cc:243] Waited 287 ms on lock 0x7b54007f17d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:04:04.662361 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":669952128}
W20260709 13:04:04.610827 11834 log.cc:927] Time spent T 836251ebc2b74f5da88b8a8b2338e2a5 P 10ca5ee68ad64b3283b9d235352c9d23: Append to log took a long time: real 0.057s	user 0.001s	sys 0.000s
W20260709 13:04:04.555998 10846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:04.644246 11022 rpcz_store.cc:276] 0709 13:04:03.556066 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:04.200023 (+643957us) service_pool.cc:224] Handling call
0709 13:04:04.200423 (+   400us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:04.200853 (+   430us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:04.200896 (+    43us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:04.212783 (+ 11887us) log.cc:844] Serialized 26 byte log entry
0709 13:04:04.558970 (+346187us) raft_consensus.cc:1645] Marking committed up to 1
0709 13:04:04.570886 (+ 11916us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:04.579131 (+  8245us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:04.618716 (+ 39585us) raft_consensus.cc:1685] finished
0709 13:04:04.618750 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:04.618967 (+   217us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":11852,"raft.queue_time_us":1894,"raft.run_cpu_time_us":173,"raft.run_wall_time_us":5834,"thread_start_us":316525,"threads_started":1,"wal-append.queue_time_us":374850}
I20260709 13:04:04.523890 11779 raft_consensus.cc:493] T 1e4f2241a94a41f38e481137f868b83d P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:04.573586 11583 raft_consensus.cc:2749] T 2567c2d84e4648d7ba5dc23a3df32817 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:04:04.572597 11101 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37676 (request call id 158) took 1192 ms. Trace:
I20260709 13:04:04.677651 11101 rpcz_store.cc:276] 0709 13:04:03.380121 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:04.557808 (+1177687us) service_pool.cc:224] Handling call
0709 13:04:04.558325 (+   517us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:04.558628 (+   303us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:04.558667 (+    39us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:04.558734 (+    67us) raft_consensus.cc:1645] Marking committed up to 1
0709 13:04:04.558779 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:04.558855 (+    76us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:04.572478 (+ 13623us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:04.567131 11653 raft_consensus.cc:2804] T 4b37f8fca8dc42718b2a42f0883a956f P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:04:04.531773 10871 raft_consensus.cc:2376] T 65bf0a62f4d84ef18dd6b23e082724b8 P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 3. Re-sending same reply.
I20260709 13:04:04.556710 11097 raft_consensus.cc:2468] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 2.
I20260709 13:04:04.572369 10875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72b111ad7b5f4753ab1d2b5ee1114840" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:04:04.528836 11734 raft_consensus.cc:515] T a64b7e99d6c54934a7de6f6dffcf5e94 P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:04.597167 10878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7709387f4d3d4f9992d97a0fbca39a40" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:04.570606 11817 raft_consensus.cc:3060] T b4cd3d7dfbbf4096908ff11e9dc57141 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:04:04.600188 11151 ts_tablet_manager.cc:1403] T f34f2bc02d85444f9096e0f5b4fe4331 P 186b12638a87410487467df721a3cbda: Time spent bootstrapping tablet: real 0.732s	user 0.027s	sys 0.016s
I20260709 13:04:04.579746 11819 raft_consensus.cc:697] T e5ba72ccd854430baadbc636dfd17136 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 LEADER]: Becoming Leader. State: Replica: 10ca5ee68ad64b3283b9d235352c9d23, State: Running, Role: LEADER
I20260709 13:04:04.459113 11803 raft_consensus.cc:2749] T 8c160ddfe9b34bdaa3835d0915c1e06f P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:04:04.584522 11688 raft_consensus.cc:697] T f371d2565dd84667b5613848604949bb P 186b12638a87410487467df721a3cbda [term 2 LEADER]: Becoming Leader. State: Replica: 186b12638a87410487467df721a3cbda, State: Running, Role: LEADER
I20260709 13:04:04.505748 10954 rpcz_store.cc:276] 0709 13:04:03.284044 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:03.284599 (+   555us) service_pool.cc:224] Handling call
0709 13:04:03.629953 (+345354us) spinlock_profiling.cc:243] Waited 7.45 ms on lock 0x7b580001d198. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f426303db71 00007f426303da22 00007f426bf56c09 00007f426b0b0ee0 00007f426b17a0ab 00007f426b17a131 00007f426b1280a0 00007f426b127984 00007f42646a26d4 00007f42646a2662
0709 13:04:04.197373 (+567420us) spinlock_profiling.cc:243] Waited 9.46 ms on lock 0x7b540069e028. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf3f924 00007f426b128227 00007f426b127ab8 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6
0709 13:04:04.349943 (+152570us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":38892672}
I20260709 13:04:04.584985 11633 leader_election.cc:290] T 63d44f6ba6434ab2870fe3190f4d7c2f P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:04.596437 11096 raft_consensus.cc:3060] T ff13ae9a47c7455f962406c6d131a3ff P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Advancing to term 2
I20260709 13:04:04.657505 10877 raft_consensus.cc:2410] T acc241e7273d49149984968596242712 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0172fed22b6b4890a7fdc953fa128af4 for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260709 13:04:04.659912 11102 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37678 (request call id 71) took 1206 ms. Trace:
I20260709 13:04:04.490195 11782 raft_consensus.cc:515] T 11de93cd9b9d457489ff1cb80f976c9a P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:04.661190 11028 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e89e640169e14e2d98f968f76027daa7" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
I20260709 13:04:04.662333 10880 rpcz_store.cc:276] 0709 13:04:03.245597 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:04.259123 (+1013526us) service_pool.cc:224] Handling call
0709 13:04:04.660697 (+401574us) spinlock_profiling.cc:243] Waited 212 ms on lock 0x7b54005a0854. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:04:04.661440 (+   743us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":486804224}
I20260709 13:04:04.662276 10992 leader_election.cc:304] T 525bfdbd6d934afd823f7d3ee3f9d052 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 873357742d5843579556ee8166f23f47; no voters: 0172fed22b6b4890a7fdc953fa128af4, 10ca5ee68ad64b3283b9d235352c9d23
I20260709 13:04:04.529340 11585 raft_consensus.cc:2804] T 75492f86fde046c3900df3eac8292fd4 P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Leader election won for term 1
W20260709 13:04:04.666142 10919 leader_election.cc:343] T a977842ba4b744f4bb2febd55d286006 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 873357742d5843579556ee8166f23f47 (127.0.107.195:45807): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:04:04.711783 10876 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51696 (request call id 172) took 1469 ms. Trace:
I20260709 13:04:04.739163 11021 raft_consensus.cc:2368] T ff13ae9a47c7455f962406c6d131a3ff P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0172fed22b6b4890a7fdc953fa128af4 for earlier term 1. Current term is 2.
I20260709 13:04:04.667694 11069 leader_election.cc:304] T 868e92339c2f4508af0afbd07c814d64 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda; no voters: 873357742d5843579556ee8166f23f47
W20260709 13:04:04.757978 10920 leader_election.cc:343] T a977842ba4b744f4bb2febd55d286006 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507): Illegal state: must be running to vote when last-logged opid is not known
W20260709 13:04:04.539193 10844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:04.760663 10878 raft_consensus.cc:1240] T d371c5e27bd843469d2e2e453734cb4a P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Rejecting Update request from peer 0172fed22b6b4890a7fdc953fa128af4 for earlier term 1. Current term is 2. Ops: [1.1-1.1]
I20260709 13:04:04.761569 10878 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51704 (request call id 122) took 1518 ms. Trace:
I20260709 13:04:04.665738 10995 leader_election.cc:304] T e89e640169e14e2d98f968f76027daa7 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47; no voters: 
W20260709 13:04:04.688560 11067 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:45807, user_credentials={real_user=slave}} blocked reactor thread for 253764us
W20260709 13:04:04.694288 10921 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:04.763561 11067 leader_election.cc:304] T f371d2565dd84667b5613848604949bb P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47; no voters: 186b12638a87410487467df721a3cbda
I20260709 13:04:04.621091 11792 leader_election.cc:290] T acc241e7273d49149984968596242712 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:04.650277 11024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb7d67be500947b3aef4c4245e33671c" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:04.772002 11842 raft_consensus.cc:2804] T 868e92339c2f4508af0afbd07c814d64 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:04:04.772675 11842 raft_consensus.cc:697] T 868e92339c2f4508af0afbd07c814d64 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 LEADER]: Becoming Leader. State: Replica: 0172fed22b6b4890a7fdc953fa128af4, State: Running, Role: LEADER
I20260709 13:04:04.774832 10880 raft_consensus.cc:1240] T 836251ebc2b74f5da88b8a8b2338e2a5 P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Rejecting Update request from peer 0172fed22b6b4890a7fdc953fa128af4 for earlier term 1. Current term is 3. Ops: []
I20260709 13:04:04.775620 10880 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51704 (request call id 123) took 1532 ms. Trace:
I20260709 13:04:04.775002 11819 consensus_queue.cc:237] T e5ba72ccd854430baadbc636dfd17136 P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:04.775990 10880 rpcz_store.cc:276] 0709 13:04:03.243318 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:04.774119 (+1530801us) service_pool.cc:224] Handling call
0709 13:04:04.774535 (+   416us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:04.775330 (+   795us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:04.775543 (+   213us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:04.777868 10880 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51704 (request call id 126) took 1533 ms. Trace:
I20260709 13:04:04.765187 11803 raft_consensus.cc:2749] T 4f9eb06ed01f4ce68be3af8a8434665b P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:04:04.779448 11803 consensus_queue.cc:1059] T d371c5e27bd843469d2e2e453734cb4a P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:04:04.780328 11803 raft_consensus.cc:2764] T f371d2565dd84667b5613848604949bb P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Leader pre-election decision vote started in defunct term 1: won
I20260709 13:04:04.781122 11803 raft_consensus.cc:3055] T d371c5e27bd843469d2e2e453734cb4a P 0172fed22b6b4890a7fdc953fa128af4 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:04:04.781483 11803 raft_consensus.cc:740] T d371c5e27bd843469d2e2e453734cb4a P 0172fed22b6b4890a7fdc953fa128af4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0172fed22b6b4890a7fdc953fa128af4, State: Running, Role: LEADER
I20260709 13:04:04.781826 10871 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63d44f6ba6434ab2870fe3190f4d7c2f" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:04.773872 11842 consensus_queue.cc:237] T 868e92339c2f4508af0afbd07c814d64 P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:04.783161 10921 leader_election.cc:304] T c51782cd9d064ec0949e84e24dd4de67 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23; no voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda
I20260709 13:04:04.784209 11639 raft_consensus.cc:2749] T c51782cd9d064ec0949e84e24dd4de67 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:04:04.784678 10921 leader_election.cc:304] T e89e640169e14e2d98f968f76027daa7 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23; no voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47
I20260709 13:04:04.744910 11729 raft_consensus.cc:515] T 732c4adeeff6422497aa7db4a82f535b P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:04.661278 11026 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acc241e7273d49149984968596242712" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:04.786085 11020 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46642 (request call id 77) took 1176 ms. Trace:
I20260709 13:04:04.787339 11729 leader_election.cc:290] T 732c4adeeff6422497aa7db4a82f535b P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 2 election: Requested vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:04.787842 11026 raft_consensus.cc:1240] T ff13ae9a47c7455f962406c6d131a3ff P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Rejecting Update request from peer 10ca5ee68ad64b3283b9d235352c9d23 for earlier term 1. Current term is 2. Ops: [1.1-1.1]
I20260709 13:04:04.787997 11729 raft_consensus.cc:2749] T e89e640169e14e2d98f968f76027daa7 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:04:04.788735 11774 raft_consensus.cc:2749] T 4f9eb06ed01f4ce68be3af8a8434665b P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:04:04.615410 10874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a69f32c53f24a50b7852d143665d7b4" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:04.791242 10874 raft_consensus.cc:2410] T 8a69f32c53f24a50b7852d143665d7b4 P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0172fed22b6b4890a7fdc953fa128af4 for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 13:04:04.792527 10874 raft_consensus.cc:1275] T e5a5f5a608504650b48fd27f163faed6 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Refusing update from remote peer 10ca5ee68ad64b3283b9d235352c9d23: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:04:04.793090 10874 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51696 (request call id 173) took 1346 ms. Trace:
I20260709 13:04:04.793412 10874 rpcz_store.cc:276] 0709 13:04:03.446603 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:04.791849 (+1345246us) service_pool.cc:224] Handling call
0709 13:04:04.792192 (+   343us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:04.792867 (+   675us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:04.793040 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:04.630275 10879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "836251ebc2b74f5da88b8a8b2338e2a5" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:04.646369 11025 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7709387f4d3d4f9992d97a0fbca39a40" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:04.794988 10879 raft_consensus.cc:2376] T 836251ebc2b74f5da88b8a8b2338e2a5 P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 3. Re-sending same reply.
I20260709 13:04:04.795575 11734 leader_election.cc:290] T a64b7e99d6c54934a7de6f6dffcf5e94 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:04.636324 10946 raft_consensus.cc:2468] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 4.
I20260709 13:04:04.737402 10948 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49428 (request call id 96) took 1041 ms. Trace:
I20260709 13:04:04.673874 11779 raft_consensus.cc:3060] T 1e4f2241a94a41f38e481137f868b83d P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:04:04.796288 10948 rpcz_store.cc:276] 0709 13:04:03.689616 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:04.168777 (+479161us) service_pool.cc:224] Handling call
0709 13:04:04.169149 (+   372us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:04.177951 (+  8802us) raft_consensus.cc:1534] Early marking committed up to index 2
0709 13:04:04.427804 (+249853us) log.cc:844] Serialized 14 byte log entry
0709 13:04:04.704926 (+277122us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:04.711311 (+  6385us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:04:04.718395 (+  7084us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:04.730569 (+ 12174us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:04.730772 (+   203us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1922,"prepare.queue_time_us":374856,"prepare.run_cpu_time_us":61,"prepare.run_wall_time_us":62,"raft.queue_time_us":23142,"raft.run_cpu_time_us":165,"raft.run_wall_time_us":10707,"thread_start_us":465329,"threads_started":2,"wal-append.queue_time_us":237925}
W20260709 13:04:04.798556 10843 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:04.627148 10950 raft_consensus.cc:2468] T a5299b8d044f41b7b68057e63d17b159 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 1.
I20260709 13:04:04.607479 11508 raft_consensus.cc:515] T 34064dc7bf3747dbb3876dd824ee0688 P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:04.650239 11758 raft_consensus.cc:515] T 6f46a5102fd64b45bf1c4dff863d2b4d P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:04.700136 11023 raft_consensus.cc:2468] T a5299b8d044f41b7b68057e63d17b159 P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 1.
I20260709 13:04:04.747403 11102 rpcz_store.cc:276] 0709 13:04:03.453327 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:04.658788 (+1205461us) service_pool.cc:224] Handling call
0709 13:04:04.659146 (+   358us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:04.659677 (+   531us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:04.659859 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:04.703562 10872 raft_consensus.cc:2468] T 57b6a2418c1342e7924195e9c1dbd357 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 2.
W20260709 13:04:04.756563 10919 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:45807, user_credentials={real_user=slave}} blocked reactor thread for 90462us
I20260709 13:04:04.627976 11148 raft_consensus.cc:359] T 30f628e3d7914d609e7f1d980a74c654 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:04.761942 10878 rpcz_store.cc:276] 0709 13:04:03.243003 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:04.742678 (+1499675us) service_pool.cc:224] Handling call
0709 13:04:04.760308 (+ 17630us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:04.761211 (+   903us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:04.761497 (+   286us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:04.614178 10845 leader_election.cc:304] T 7757ecbf402e47bfbc93d0447741e26c P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda; no voters: 873357742d5843579556ee8166f23f47
I20260709 13:04:04.765846 10920 leader_election.cc:304] T a977842ba4b744f4bb2febd55d286006 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23; no voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47
I20260709 13:04:04.713426 11653 raft_consensus.cc:697] T 4b37f8fca8dc42718b2a42f0883a956f P 873357742d5843579556ee8166f23f47 [term 2 LEADER]: Becoming Leader. State: Replica: 873357742d5843579556ee8166f23f47, State: Running, Role: LEADER
I20260709 13:04:04.790274 11026 raft_consensus.cc:1240] T 4c2a35bb99374e93b573cf870d04faaa P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Rejecting Update request from peer 186b12638a87410487467df721a3cbda for earlier term 2. Current term is 3. Ops: []
I20260709 13:04:04.813159 11653 consensus_queue.cc:237] T 4b37f8fca8dc42718b2a42f0883a956f P 873357742d5843579556ee8166f23f47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:04.814653 11695 raft_consensus.cc:2749] T a977842ba4b744f4bb2febd55d286006 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:04:04.720331 10875 raft_consensus.cc:3060] T 72b111ad7b5f4753ab1d2b5ee1114840 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:04:04.820657 11025 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "732c4adeeff6422497aa7db4a82f535b" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
I20260709 13:04:04.824990 11796 raft_consensus.cc:515] T 401a9cf11eac479aa445ec8c1d6e33b2 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:04.825498 11095 raft_consensus.cc:2468] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 4.
I20260709 13:04:04.605211 11757 consensus_queue.cc:237] T a610d1a71f52432e88fe56130f921fa2 P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 2.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:04.826588 11754 raft_consensus.cc:2749] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:04:04.766992 10844 leader_election.cc:304] T 0e5d327630e5438b878035d28efe6c0c P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47; no voters: 
W20260709 13:04:04.836771 10844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:45807, user_credentials={real_user=slave}} blocked reactor thread for 69879.6us
I20260709 13:04:04.767997 11748 raft_consensus.cc:493] T 5de5f0581e3b41dea729363d6ca563bb P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:04.664821 10873 raft_consensus.cc:1275] T 7709387f4d3d4f9992d97a0fbca39a40 P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Refusing update from remote peer 873357742d5843579556ee8166f23f47: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20260709 13:04:04.837685 10844 leader_election.cc:343] T a1ec2b50c6ad4f7c8033476208abb26d P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 873357742d5843579556ee8166f23f47 (127.0.107.195:45807): Illegal state: must be running to vote when last-logged opid is not known
W20260709 13:04:04.839630 10844 leader_election.cc:343] T 30f628e3d7914d609e7f1d980a74c654 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 873357742d5843579556ee8166f23f47 (127.0.107.195:45807): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:04:04.840173 10844 leader_election.cc:304] T 30f628e3d7914d609e7f1d980a74c654 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 186b12638a87410487467df721a3cbda; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47
I20260709 13:04:04.786027 10921 leader_election.cc:304] T b67eff0634e540eab91cf8cee795969f P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda; no voters: 
W20260709 13:04:04.841480 10921 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:38967, user_credentials={real_user=slave}} blocked reactor thread for 55533.6us
W20260709 13:04:04.804795 10995 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:38967, user_credentials={real_user=slave}} blocked reactor thread for 139152us
W20260709 13:04:04.805850 10992 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 143675us
I20260709 13:04:04.810350 11148 raft_consensus.cc:385] T 30f628e3d7914d609e7f1d980a74c654 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:04:04.849699 11148 raft_consensus.cc:740] T 30f628e3d7914d609e7f1d980a74c654 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10ca5ee68ad64b3283b9d235352c9d23, State: Initialized, Role: FOLLOWER
W20260709 13:04:04.811792 10920 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 144831us
I20260709 13:04:04.818905 11102 raft_consensus.cc:1275] T 7709387f4d3d4f9992d97a0fbca39a40 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Refusing update from remote peer 873357742d5843579556ee8166f23f47: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 13:04:04.850965 10920 leader_election.cc:304] T d71ca3a1ba6b48478aaf62741bf3cf6a P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23; no voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47
I20260709 13:04:04.851034 11102 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37676 (request call id 166) took 1157 ms. Trace:
I20260709 13:04:04.851718 11102 rpcz_store.cc:276] 0709 13:04:03.693310 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:04.802274 (+1108964us) service_pool.cc:224] Handling call
0709 13:04:04.802586 (+   312us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:04.850644 (+ 48058us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:04.850955 (+   311us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:04.852581 10920 leader_election.cc:304] T 5d439326f23747eaab3c1e7da2d6b785 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 10ca5ee68ad64b3283b9d235352c9d23; no voters: 186b12638a87410487467df721a3cbda
I20260709 13:04:04.861388 11025 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acc241e7273d49149984968596242712" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:04.786718 11020 rpcz_store.cc:276] 0709 13:04:03.609279 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:04.200382 (+591103us) service_pool.cc:224] Handling call
0709 13:04:04.200695 (+   313us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:04.200916 (+   221us) raft_consensus.cc:1534] Early marking committed up to index 2
0709 13:04:04.461038 (+260122us) log.cc:844] Serialized 14 byte log entry
0709 13:04:04.762658 (+301620us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:04.771171 (+  8513us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:04:04.785697 (+ 14526us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:04.785821 (+   124us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:04.786000 (+   179us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2886,"prepare.queue_time_us":323732,"prepare.run_cpu_time_us":116,"prepare.run_wall_time_us":6414,"raft.queue_time_us":13164,"raft.run_cpu_time_us":180,"raft.run_wall_time_us":33951,"thread_start_us":484716,"threads_started":2}
I20260709 13:04:04.865695 11151 raft_consensus.cc:359] T f34f2bc02d85444f9096e0f5b4fe4331 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:04.867311 11151 raft_consensus.cc:385] T f34f2bc02d85444f9096e0f5b4fe4331 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:04:04.867699 11151 raft_consensus.cc:740] T f34f2bc02d85444f9096e0f5b4fe4331 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 186b12638a87410487467df721a3cbda, State: Initialized, Role: FOLLOWER
I20260709 13:04:04.778220 10880 rpcz_store.cc:276] 0709 13:04:03.244160 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:04.776913 (+1532753us) service_pool.cc:224] Handling call
0709 13:04:04.777267 (+   354us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:04.777482 (+   215us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:04.777514 (+    32us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:04.777567 (+    53us) raft_consensus.cc:1645] Marking committed up to 1
0709 13:04:04.777624 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:04.777687 (+    63us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:04.777812 (+   125us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:04.811255 11506 raft_consensus.cc:2764] T a5299b8d044f41b7b68057e63d17b159 P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260709 13:04:04.808990 11585 raft_consensus.cc:697] T 75492f86fde046c3900df3eac8292fd4 P 873357742d5843579556ee8166f23f47 [term 1 LEADER]: Becoming Leader. State: Replica: 873357742d5843579556ee8166f23f47, State: Running, Role: LEADER
I20260709 13:04:04.765060 10876 rpcz_store.cc:276] 0709 13:04:03.242027 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:04.653465 (+1411438us) service_pool.cc:224] Handling call
0709 13:04:04.704678 (+ 51213us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:04.829564 11023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46642 (request call id 82) took 1079 ms. Trace:
I20260709 13:04:04.765976 11610 raft_consensus.cc:515] T 7757ecbf402e47bfbc93d0447741e26c P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:04.885396 11148 consensus_queue.cc:260] T 30f628e3d7914d609e7f1d980a74c654 P 10ca5ee68ad64b3283b9d235352c9d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:04.869318 11553 raft_consensus.cc:2804] T b67eff0634e540eab91cf8cee795969f P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:04:04.864053 11748 raft_consensus.cc:515] T 5de5f0581e3b41dea729363d6ca563bb P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:04.869649 11792 raft_consensus.cc:2749] T 30f628e3d7914d609e7f1d980a74c654 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:04:04.878222 11795 consensus_queue.cc:1059] T 4c2a35bb99374e93b573cf870d04faaa P 186b12638a87410487467df721a3cbda [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.038s
I20260709 13:04:04.876410 11518 raft_consensus.cc:2804] T 7757ecbf402e47bfbc93d0447741e26c P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:04:04.923036 11518 raft_consensus.cc:493] T 7757ecbf402e47bfbc93d0447741e26c P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Starting leader election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:04.923151 11795 raft_consensus.cc:3055] T 4c2a35bb99374e93b573cf870d04faaa P 186b12638a87410487467df721a3cbda [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:04:04.923527 11518 raft_consensus.cc:3060] T 7757ecbf402e47bfbc93d0447741e26c P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:04:04.923593 11795 raft_consensus.cc:740] T 4c2a35bb99374e93b573cf870d04faaa P 186b12638a87410487467df721a3cbda [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 186b12638a87410487467df721a3cbda, State: Running, Role: LEADER
I20260709 13:04:04.924670 11795 consensus_queue.cc:260] T 4c2a35bb99374e93b573cf870d04faaa P 186b12638a87410487467df721a3cbda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:04.925872 11795 raft_consensus.cc:3060] T 4c2a35bb99374e93b573cf870d04faaa P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:04:04.898394 11095 raft_consensus.cc:1240] T 8a69f32c53f24a50b7852d143665d7b4 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Rejecting Update request from peer 186b12638a87410487467df721a3cbda for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260709 13:04:04.926613 11094 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37676 (request call id 165) took 1233 ms. Trace:
I20260709 13:04:04.927441 11508 leader_election.cc:290] T 34064dc7bf3747dbb3876dd824ee0688 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:04.928141 11748 leader_election.cc:290] T 5de5f0581e3b41dea729363d6ca563bb P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
W20260709 13:04:04.888087 10845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 274021us
I20260709 13:04:04.929324 10845 leader_election.cc:304] T a610d1a71f52432e88fe56130f921fa2 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 186b12638a87410487467df721a3cbda; no voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47
I20260709 13:04:04.929544 11748 consensus_queue.cc:1048] T e5a5f5a608504650b48fd27f163faed6 P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:04:04.930558 11658 raft_consensus.cc:2749] T a610d1a71f52432e88fe56130f921fa2 P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 13:04:04.932552 11103 raft_consensus.cc:1240] T 7757ecbf402e47bfbc93d0447741e26c P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Rejecting Update request from peer 873357742d5843579556ee8166f23f47 for earlier term 1. Current term is 2. Ops: []
I20260709 13:04:04.934494 11777 raft_consensus.cc:2804] T 5d439326f23747eaab3c1e7da2d6b785 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:04:04.935105 11777 raft_consensus.cc:697] T 5d439326f23747eaab3c1e7da2d6b785 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 LEADER]: Becoming Leader. State: Replica: 10ca5ee68ad64b3283b9d235352c9d23, State: Running, Role: LEADER
I20260709 13:04:04.912709 11151 consensus_queue.cc:260] T f34f2bc02d85444f9096e0f5b4fe4331 P 186b12638a87410487467df721a3cbda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:04.782320 11803 consensus_queue.cc:260] T d371c5e27bd843469d2e2e453734cb4a P 0172fed22b6b4890a7fdc953fa128af4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:04.876629 11797 consensus_queue.cc:1048] T 1e4f2241a94a41f38e481137f868b83d P 186b12638a87410487467df721a3cbda [LEADER]: Connected to new peer: Peer: permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.034s
W20260709 13:04:04.944110 11069 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:38967, user_credentials={real_user=slave}} blocked reactor thread for 276495us
I20260709 13:04:04.887210 10874 raft_consensus.cc:1275] T 0e5d327630e5438b878035d28efe6c0c P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Refusing update from remote peer 0172fed22b6b4890a7fdc953fa128af4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 13:04:04.872175 11639 raft_consensus.cc:2749] T d71ca3a1ba6b48478aaf62741bf3cf6a P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 13:04:04.883495 10873 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51718 (request call id 186) took 1756 ms. Trace:
I20260709 13:04:04.891593 11688 consensus_queue.cc:237] T f371d2565dd84667b5613848604949bb P 186b12638a87410487467df721a3cbda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:04.920655 11796 leader_election.cc:290] T 401a9cf11eac479aa445ec8c1d6e33b2 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 1 election: Requested vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:04.976146 10874 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51704 (request call id 145) took 1464 ms. Trace:
I20260709 13:04:04.860158 11695 consensus_queue.cc:1059] T ff13ae9a47c7455f962406c6d131a3ff P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.058s
I20260709 13:04:04.976645 10874 rpcz_store.cc:276] 0709 13:04:03.512016 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:04.794256 (+1282240us) service_pool.cc:224] Handling call
0709 13:04:04.819526 (+ 25270us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:04.975740 (+156214us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:04.976065 (+   325us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:04.977355 11069 leader_election.cc:304] T 8a69f32c53f24a50b7852d143665d7b4 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda
I20260709 13:04:04.976163 10873 rpcz_store.cc:276] 0709 13:04:03.171033 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:04.664109 (+1493076us) service_pool.cc:224] Handling call
0709 13:04:04.664423 (+   314us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:04.857670 (+193247us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:04.879322 (+ 21652us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:04.982297 11780 consensus_queue.cc:1048] T 7709387f4d3d4f9992d97a0fbca39a40 P 873357742d5843579556ee8166f23f47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:04:04.918044 11553 raft_consensus.cc:493] T b67eff0634e540eab91cf8cee795969f P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:05.014175 11628 raft_consensus.cc:2749] T c31881534c5e4f0089c82057c1746652 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:04:05.020895 11151 ts_tablet_manager.cc:1434] T f34f2bc02d85444f9096e0f5b4fe4331 P 186b12638a87410487467df721a3cbda: Time spent starting tablet: real 0.304s	user 0.009s	sys 0.000s
I20260709 13:04:05.020669 11518 raft_consensus.cc:515] T 7757ecbf402e47bfbc93d0447741e26c P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:05.022755 10874 raft_consensus.cc:1240] T 7757ecbf402e47bfbc93d0447741e26c P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Rejecting Update request from peer 873357742d5843579556ee8166f23f47 for earlier term 1. Current term is 2. Ops: []
I20260709 13:04:05.018625 11553 raft_consensus.cc:3060] T b67eff0634e540eab91cf8cee795969f P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:04:04.936093 11777 consensus_queue.cc:237] T 5d439326f23747eaab3c1e7da2d6b785 P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:05.030941 11758 leader_election.cc:290] T 6f46a5102fd64b45bf1c4dff863d2b4d P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:05.031718 11782 leader_election.cc:290] T 11de93cd9b9d457489ff1cb80f976c9a P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:04.992473 11804 raft_consensus.cc:2749] T b4cd3d7dfbbf4096908ff11e9dc57141 P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507) with higher term. Message: Invalid argument: T b4cd3d7dfbbf4096908ff11e9dc57141 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 873357742d5843579556ee8166f23f47 for earlier term 2. Current term is 3.
I20260709 13:04:05.033051 11095 raft_consensus.cc:3060] T a154cc8b181f460abf4880064379948b P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:04:05.031190 11518 leader_election.cc:290] T 7757ecbf402e47bfbc93d0447741e26c P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 2 election: Requested vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:05.038152 11100 raft_consensus.cc:3060] T c31881534c5e4f0089c82057c1746652 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:04:05.046865 10876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "732c4adeeff6422497aa7db4a82f535b" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:04:05.047950 10876 raft_consensus.cc:3060] T 732c4adeeff6422497aa7db4a82f535b P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:04:04.994875 11723 raft_consensus.cc:2804] T 0e5d327630e5438b878035d28efe6c0c P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:04:05.052949 10845 leader_election.cc:400] T 836251ebc2b74f5da88b8a8b2338e2a5 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 2 pre-election: Vote denied by peer 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507) with higher term. Message: Invalid argument: T 836251ebc2b74f5da88b8a8b2338e2a5 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 186b12638a87410487467df721a3cbda for earlier term 2. Current term is 3.
I20260709 13:04:05.053500 10845 leader_election.cc:403] T 836251ebc2b74f5da88b8a8b2338e2a5 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:04:05.054390 11518 raft_consensus.cc:2749] T 836251ebc2b74f5da88b8a8b2338e2a5 P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507) with higher term. Message: Invalid argument: T 836251ebc2b74f5da88b8a8b2338e2a5 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 186b12638a87410487467df721a3cbda for earlier term 2. Current term is 3.
I20260709 13:04:05.055264 10845 leader_election.cc:304] T 8c160ddfe9b34bdaa3835d0915c1e06f P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda; no voters: 873357742d5843579556ee8166f23f47
I20260709 13:04:05.060194 11152 tablet_bootstrap.cc:492] T 401a9cf11eac479aa445ec8c1d6e33b2 P 0172fed22b6b4890a7fdc953fa128af4: No bootstrap required, opened a new log
I20260709 13:04:05.054378 11029 raft_consensus.cc:2468] T b4cd3d7dfbbf4096908ff11e9dc57141 P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 3.
I20260709 13:04:05.061038 11152 ts_tablet_manager.cc:1403] T 401a9cf11eac479aa445ec8c1d6e33b2 P 0172fed22b6b4890a7fdc953fa128af4: Time spent bootstrapping tablet: real 0.784s	user 0.035s	sys 0.001s
I20260709 13:04:05.060782 11553 raft_consensus.cc:515] T b67eff0634e540eab91cf8cee795969f P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:05.062279 11095 raft_consensus.cc:1275] T a154cc8b181f460abf4880064379948b P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Refusing update from remote peer 873357742d5843579556ee8166f23f47: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:04:05.066238 11095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "401a9cf11eac479aa445ec8c1d6e33b2" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:04:05.073092 11149 tablet_bootstrap.cc:492] T a977842ba4b744f4bb2febd55d286006 P 873357742d5843579556ee8166f23f47: No bootstrap required, opened a new log
I20260709 13:04:05.073940 11149 ts_tablet_manager.cc:1403] T a977842ba4b744f4bb2febd55d286006 P 873357742d5843579556ee8166f23f47: Time spent bootstrapping tablet: real 0.888s	user 0.025s	sys 0.017s
W20260709 13:04:05.078270 10919 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:45807, user_credentials={real_user=slave}} blocked reactor thread for 61808.5us
I20260709 13:04:04.912209 11023 rpcz_store.cc:276] 0709 13:04:03.738090 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:04.169231 (+431141us) service_pool.cc:224] Handling call
0709 13:04:04.817274 (+648043us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:05.012694 11779 raft_consensus.cc:515] T 1e4f2241a94a41f38e481137f868b83d P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:05.056581 11673 raft_consensus.cc:2749] T 525bfdbd6d934afd823f7d3ee3f9d052 P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260709 13:04:05.097504 11067 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:45807, user_credentials={real_user=slave}} blocked reactor thread for 334020us
W20260709 13:04:05.101748 10992 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 247835us
I20260709 13:04:05.109330 11103 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37676 (request call id 162) took 1702 ms. Trace:
I20260709 13:04:05.112907 11103 rpcz_store.cc:276] 0709 13:04:03.407107 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:04.647115 (+1240008us) service_pool.cc:224] Handling call
0709 13:04:04.647517 (+   402us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:04.932998 (+285481us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:05.109057 (+176059us) spinlock_profiling.cc:243] Waited 285 ms on lock 0x7b540074d954. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:04:05.109248 (+   191us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":654981376}
I20260709 13:04:05.114593 11103 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acc241e7273d49149984968596242712" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:05.116299 11103 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7757ecbf402e47bfbc93d0447741e26c" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:04:05.117247 11103 raft_consensus.cc:2393] T 7757ecbf402e47bfbc93d0447741e26c P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 186b12638a87410487467df721a3cbda in current term 2: Already voted for candidate 0172fed22b6b4890a7fdc953fa128af4 in this term.
I20260709 13:04:05.109333 10994 leader_election.cc:304] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:04:05.104657 10992 leader_election.cc:304] T 4c2a35bb99374e93b573cf870d04faaa P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47; no voters: 186b12638a87410487467df721a3cbda
I20260709 13:04:04.927112 11094 rpcz_store.cc:276] 0709 13:04:03.692968 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:04.757367 (+1064399us) service_pool.cc:224] Handling call
0709 13:04:04.789800 (+ 32433us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:04.874512 (+ 84712us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:04.876900 (+  2388us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:04.884651 (+  7751us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:04:04.888964 (+  4313us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:04.926303 (+ 37339us) spinlock_profiling.cc:243] Waited 68.3 ms on lock 0x7b54000e51d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:04:04.926366 (+    63us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:04.926542 (+   176us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":157182720}
I20260709 13:04:05.075917 11729 raft_consensus.cc:493] T 0f9df2bc281f4866b6da5da94a0defc7 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0172fed22b6b4890a7fdc953fa128af4)
I20260709 13:04:05.017659 11849 raft_consensus.cc:2749] T 8a69f32c53f24a50b7852d143665d7b4 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20260709 13:04:05.137782 10846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:05.141539 11103 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a64b7e99d6c54934a7de6f6dffcf5e94" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:05.153982 11103 raft_consensus.cc:2468] T a64b7e99d6c54934a7de6f6dffcf5e94 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 3.
W20260709 13:04:05.154213 11067 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:45807, user_credentials={real_user=slave}} blocked reactor thread for 55608.6us
I20260709 13:04:05.155380 11067 leader_election.cc:304] T 6a9353d1d230416296627d7cf8b1d425 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4; no voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47
I20260709 13:04:05.156145 11103 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b67eff0634e540eab91cf8cee795969f" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:05.156996 11067 leader_election.cc:304] T e5ba72ccd854430baadbc636dfd17136 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47
I20260709 13:04:05.157130 11103 raft_consensus.cc:2468] T b67eff0634e540eab91cf8cee795969f P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 0.
W20260709 13:04:05.133368 10920 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 280744us
I20260709 13:04:05.159941 11067 leader_election.cc:304] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4; no voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47
I20260709 13:04:05.159878 10920 leader_election.cc:304] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 10ca5ee68ad64b3283b9d235352c9d23; no voters: 186b12638a87410487467df721a3cbda
I20260709 13:04:05.163030 11553 leader_election.cc:290] T b67eff0634e540eab91cf8cee795969f P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 1 election: Requested vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
W20260709 13:04:05.175258 10921 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:38967, user_credentials={real_user=slave}} blocked reactor thread for 296449us
I20260709 13:04:05.173167 11067 leader_election.cc:304] T d71ca3a1ba6b48478aaf62741bf3cf6a P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:04:05.174852 11628 raft_consensus.cc:2749] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:04:05.173771 11152 raft_consensus.cc:359] T 401a9cf11eac479aa445ec8c1d6e33b2 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:05.176645 11152 raft_consensus.cc:385] T 401a9cf11eac479aa445ec8c1d6e33b2 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:04:05.176949 11152 raft_consensus.cc:740] T 401a9cf11eac479aa445ec8c1d6e33b2 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0172fed22b6b4890a7fdc953fa128af4, State: Initialized, Role: FOLLOWER
I20260709 13:04:05.177862 11152 consensus_queue.cc:260] T 401a9cf11eac479aa445ec8c1d6e33b2 P 0172fed22b6b4890a7fdc953fa128af4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:05.123966 11097 raft_consensus.cc:1240] T 868e92339c2f4508af0afbd07c814d64 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 LEADER]: Rejecting Update request from peer 873357742d5843579556ee8166f23f47 for earlier term 1. Current term is 2. Ops: []
I20260709 13:04:05.133350 11734 raft_consensus.cc:2804] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Leader election won for term 4
W20260709 13:04:05.177845 10921 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:05.184553 11067 leader_election.cc:304] T acc241e7273d49149984968596242712 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4; no voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47
I20260709 13:04:05.204497 11148 ts_tablet_manager.cc:1434] T 30f628e3d7914d609e7f1d980a74c654 P 10ca5ee68ad64b3283b9d235352c9d23: Time spent starting tablet: real 0.676s	user 0.027s	sys 0.015s
W20260709 13:04:05.205091 11148 ts_tablet_manager.cc:1467] T 30f628e3d7914d609e7f1d980a74c654 P 10ca5ee68ad64b3283b9d235352c9d23: Tablet startup took 676ms
I20260709 13:04:05.164973 10873 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51696 (request call id 179) took 1260 ms. Trace:
I20260709 13:04:05.169004 11842 raft_consensus.cc:2749] T 6a9353d1d230416296627d7cf8b1d425 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 13:04:05.166257 11028 raft_consensus.cc:2468] T e89e640169e14e2d98f968f76027daa7 P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 3.
W20260709 13:04:05.164860 10845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 109701us
W20260709 13:04:05.205696 11148 ts_tablet_manager.cc:1469] T 30f628e3d7914d609e7f1d980a74c654 P 10ca5ee68ad64b3283b9d235352c9d23: Trace:
0709 13:03:48.610485 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:48.611075 (+   590us) service_pool.cc:224] Handling call
0709 13:03:48.612809 (+  1734us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 13:03:48.612906 (+    97us) ts_tablet_manager.cc:682] Creating new metadata...
0709 13:03:48.835119 (+222213us) tablet_metadata.cc:665] Metadata flushed
0709 13:03:48.892274 (+ 57155us) tablet_replica.cc:206] Creating consensus instance
0709 13:03:48.894269 (+  1995us) inbound_call.cc:177] Queueing success response
0709 13:04:03.547611 (+14653342us) ts_tablet_manager.cc:1379] Bootstrapping tablet
0709 13:04:04.528248 (+980637us) ts_tablet_manager.cc:1435] Starting tablet replica
0709 13:04:04.625257 (+ 97009us) tablet_replica.cc:261] Starting instrumentation
0709 13:04:04.626003 (+   746us) tablet_replica.cc:280] Starting consensus
Metrics: {"mutex_wait_us":13994,"raft.queue_time_us":20884,"raft.run_cpu_time_us":235,"raft.run_wall_time_us":49799,"tablet-open.queue_time_us":14653799,"thread_start_us":265450,"threads_started":1}
I20260709 13:04:05.220816 10950 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49428 (request call id 95) took 1534 ms. Trace:
I20260709 13:04:05.221313 10950 rpcz_store.cc:276] 0709 13:04:03.686236 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:04.119153 (+432917us) service_pool.cc:224] Handling call
0709 13:04:05.220607 (+1101454us) spinlock_profiling.cc:243] Waited 157 ms on lock 0x7b5400416754. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:04:05.220745 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":360162560}
I20260709 13:04:05.222514 11853 raft_consensus.cc:493] T 525bfdbd6d934afd823f7d3ee3f9d052 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 10ca5ee68ad64b3283b9d235352c9d23)
I20260709 13:04:05.223148 11853 raft_consensus.cc:515] T 525bfdbd6d934afd823f7d3ee3f9d052 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:05.225426 11853 leader_election.cc:290] T 525bfdbd6d934afd823f7d3ee3f9d052 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
W20260709 13:04:05.225776 10844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:05.228533 10874 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51718 (request call id 192) took 1361 ms. Trace:
I20260709 13:04:05.228986 10874 rpcz_store.cc:276] 0709 13:04:03.888648 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:04.977631 (+1088983us) service_pool.cc:224] Handling call
0709 13:04:04.978100 (+   469us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:05.023323 (+ 45223us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:05.228255 (+204932us) spinlock_profiling.cc:243] Waited 44.3 ms on lock 0x7b540060f4d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:04:05.228459 (+   204us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":101953920}
I20260709 13:04:05.230631 10946 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "836251ebc2b74f5da88b8a8b2338e2a5" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:05.230973 11025 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34064dc7bf3747dbb3876dd824ee0688" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:05.231654 11020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7757ecbf402e47bfbc93d0447741e26c" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
I20260709 13:04:05.207758 10873 rpcz_store.cc:276] 0709 13:04:03.892766 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:04.980089 (+1087323us) service_pool.cc:224] Handling call
0709 13:04:04.980497 (+   408us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:05.074866 (+ 94369us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:05.081338 (+  6472us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:05.094051 (+ 12713us) raft_consensus.cc:1645] Marking committed up to 1
0709 13:04:05.096768 (+  2717us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:05.109811 (+ 13043us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:05.152734 (+ 42923us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":33551,"raft.queue_time_us":35818,"raft.run_cpu_time_us":217,"raft.run_wall_time_us":39029}
I20260709 13:04:05.234689 11096 raft_consensus.cc:2468] T ff13ae9a47c7455f962406c6d131a3ff P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 2.
I20260709 13:04:05.235620 11102 raft_consensus.cc:1240] T ff13ae9a47c7455f962406c6d131a3ff P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Rejecting Update request from peer 10ca5ee68ad64b3283b9d235352c9d23 for earlier term 1. Current term is 2. Ops: []
I20260709 13:04:05.236142 10951 raft_consensus.cc:1240] T 72b111ad7b5f4753ab1d2b5ee1114840 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Rejecting Update request from peer 0172fed22b6b4890a7fdc953fa128af4 for earlier term 2. Current term is 3. Ops: []
I20260709 13:04:05.236967 11102 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37666 (request call id 152) took 1047 ms. Trace:
I20260709 13:04:05.237358 11102 rpcz_store.cc:276] 0709 13:04:04.189178 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:04.853996 (+664818us) service_pool.cc:224] Handling call
0709 13:04:05.235423 (+381427us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:05.235968 (+   545us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:05.236773 (+   805us) spinlock_profiling.cc:243] Waited 381 ms on lock 0x7b5400155ed0. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661618ad 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:04:05.236904 (+   131us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":876392704}
I20260709 13:04:05.237372 10948 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d71ca3a1ba6b48478aaf62741bf3cf6a" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:05.205885 11780 consensus_queue.cc:1048] T a154cc8b181f460abf4880064379948b P 873357742d5843579556ee8166f23f47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.033s
I20260709 13:04:05.239212 10948 raft_consensus.cc:2393] T d71ca3a1ba6b48478aaf62741bf3cf6a P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0172fed22b6b4890a7fdc953fa128af4 in current term 2: Already voted for candidate 10ca5ee68ad64b3283b9d235352c9d23 in this term.
I20260709 13:04:05.239786 11734 raft_consensus.cc:697] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 873357742d5843579556ee8166f23f47 [term 4 LEADER]: Becoming Leader. State: Replica: 873357742d5843579556ee8166f23f47, State: Running, Role: LEADER
I20260709 13:04:05.240804 11734 consensus_queue.cc:237] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 873357742d5843579556ee8166f23f47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:05.242785 11800 raft_consensus.cc:2804] T 4c2a35bb99374e93b573cf870d04faaa P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Leader election won for term 3
W20260709 13:04:05.243101 11844 log.cc:927] Time spent T bac7e5040f984ba2b7146bd36af7267e P 10ca5ee68ad64b3283b9d235352c9d23: Append to log took a long time: real 0.053s	user 0.000s	sys 0.000s
I20260709 13:04:05.229993 11024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "401a9cf11eac479aa445ec8c1d6e33b2" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
I20260709 13:04:05.244187 11024 raft_consensus.cc:3060] T 401a9cf11eac479aa445ec8c1d6e33b2 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:04:05.245047 11633 raft_consensus.cc:2804] T e89e640169e14e2d98f968f76027daa7 P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:04:05.246199 11633 raft_consensus.cc:493] T e89e640169e14e2d98f968f76027daa7 P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:05.204641 10875 raft_consensus.cc:2468] T 72b111ad7b5f4753ab1d2b5ee1114840 P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 3.
I20260709 13:04:05.243541 11800 raft_consensus.cc:697] T 4c2a35bb99374e93b573cf870d04faaa P 873357742d5843579556ee8166f23f47 [term 3 LEADER]: Becoming Leader. State: Replica: 873357742d5843579556ee8166f23f47, State: Running, Role: LEADER
I20260709 13:04:05.246608 11633 raft_consensus.cc:3060] T e89e640169e14e2d98f968f76027daa7 P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:04:05.246906 11695 raft_consensus.cc:493] T a64b7e99d6c54934a7de6f6dffcf5e94 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:05.247825 11094 raft_consensus.cc:1275] T 87cd8f5097624c8692f1d83544d8881e P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Refusing update from remote peer 186b12638a87410487467df721a3cbda: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 13:04:05.247522 11695 raft_consensus.cc:515] T a64b7e99d6c54934a7de6f6dffcf5e94 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:05.247578 11800 consensus_queue.cc:237] T 4c2a35bb99374e93b573cf870d04faaa P 873357742d5843579556ee8166f23f47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:05.250097 11695 leader_election.cc:290] T a64b7e99d6c54934a7de6f6dffcf5e94 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:05.234577 11817 raft_consensus.cc:515] T b4cd3d7dfbbf4096908ff11e9dc57141 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:05.191684 11628 raft_consensus.cc:2804] T d71ca3a1ba6b48478aaf62741bf3cf6a P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:04:05.179571 11851 raft_consensus.cc:2749] T e5ba72ccd854430baadbc636dfd17136 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 13:04:05.102308 11585 consensus_queue.cc:237] T 75492f86fde046c3900df3eac8292fd4 P 873357742d5843579556ee8166f23f47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:05.258971 10844 leader_election.cc:304] T a5299b8d044f41b7b68057e63d17b159 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:04:05.200318 11553 raft_consensus.cc:2804] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:04:05.306947 11553 raft_consensus.cc:697] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 LEADER]: Becoming Leader. State: Replica: 10ca5ee68ad64b3283b9d235352c9d23, State: Running, Role: LEADER
I20260709 13:04:05.308393 11553 consensus_queue.cc:237] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:05.310874 10874 raft_consensus.cc:1240] T 868e92339c2f4508af0afbd07c814d64 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Rejecting Update request from peer 873357742d5843579556ee8166f23f47 for earlier term 1. Current term is 2. Ops: []
I20260709 13:04:05.165535 11729 raft_consensus.cc:515] T 0f9df2bc281f4866b6da5da94a0defc7 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:05.316063 11729 leader_election.cc:290] T 0f9df2bc281f4866b6da5da94a0defc7 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:05.164276 11845 consensus_queue.cc:1048] T 7709387f4d3d4f9992d97a0fbca39a40 P 873357742d5843579556ee8166f23f47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.078s
I20260709 13:04:05.169914 11758 consensus_queue.cc:1048] T 0e5d327630e5438b878035d28efe6c0c P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.043s
I20260709 13:04:05.257895 10921 leader_election.cc:304] T 87cd8f5097624c8692f1d83544d8881e P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23; no voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda
I20260709 13:04:05.255460 11862 consensus_queue.cc:1059] T 7757ecbf402e47bfbc93d0447741e26c P 873357742d5843579556ee8166f23f47 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.102s
W20260709 13:04:05.318805 10921 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:38967, user_credentials={real_user=slave}} blocked reactor thread for 63498.3us
I20260709 13:04:05.319156 11852 consensus_queue.cc:1048] T 0e5d327630e5438b878035d28efe6c0c P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:04:05.319638 11729 raft_consensus.cc:2749] T 87cd8f5097624c8692f1d83544d8881e P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:04:05.262121 11152 ts_tablet_manager.cc:1434] T 401a9cf11eac479aa445ec8c1d6e33b2 P 0172fed22b6b4890a7fdc953fa128af4: Time spent starting tablet: real 0.200s	user 0.013s	sys 0.000s
I20260709 13:04:05.320725 10874 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51718 (request call id 193) took 1415 ms. Trace:
I20260709 13:04:05.263355 11658 raft_consensus.cc:2804] T a5299b8d044f41b7b68057e63d17b159 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:04:05.289534 10875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b67eff0634e540eab91cf8cee795969f" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:04:05.298843 11024 raft_consensus.cc:2468] T 401a9cf11eac479aa445ec8c1d6e33b2 P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 1.
W20260709 13:04:05.301291 10845 leader_election.cc:343] T a1ec2b50c6ad4f7c8033476208abb26d P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:04:05.261638 10844 leader_election.cc:304] T e89e640169e14e2d98f968f76027daa7 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:04:05.310742 10871 raft_consensus.cc:1240] T 72b111ad7b5f4753ab1d2b5ee1114840 P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Rejecting Update request from peer 0172fed22b6b4890a7fdc953fa128af4 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260709 13:04:05.305508 11817 leader_election.cc:290] T b4cd3d7dfbbf4096908ff11e9dc57141 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 3 election: Requested vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:05.260792 11628 raft_consensus.cc:493] T d71ca3a1ba6b48478aaf62741bf3cf6a P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:05.311010 11595 raft_consensus.cc:2749] T bac7e5040f984ba2b7146bd36af7267e P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:04:05.311699 10949 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49434 (request call id 143) took 1491 ms. Trace:
I20260709 13:04:05.295953 10953 raft_consensus.cc:2468] T 4b37f8fca8dc42718b2a42f0883a956f P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 2.
I20260709 13:04:05.313418 11026 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11de93cd9b9d457489ff1cb80f976c9a" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:05.321365 11152 tablet_bootstrap.cc:492] T a977842ba4b744f4bb2febd55d286006 P 0172fed22b6b4890a7fdc953fa128af4: Bootstrap starting.
I20260709 13:04:05.215287 11757 raft_consensus.cc:2749] T acc241e7273d49149984968596242712 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:04:05.323009 10955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb7d67be500947b3aef4c4245e33671c" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:05.255864 11067 leader_election.cc:400] T ff13ae9a47c7455f962406c6d131a3ff P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 1 pre-election: Vote denied by peer 873357742d5843579556ee8166f23f47 (127.0.107.195:45807) with higher term. Message: Invalid argument: T ff13ae9a47c7455f962406c6d131a3ff P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0172fed22b6b4890a7fdc953fa128af4 for earlier term 1. Current term is 2.
W20260709 13:04:05.329337 10844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:45807, user_credentials={real_user=slave}} blocked reactor thread for 67770.7us
W20260709 13:04:05.271140 10843 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 380008us
I20260709 13:04:05.303972 11098 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37678 (request call id 72) took 1843 ms. Trace:
I20260709 13:04:05.350574 10921 leader_election.cc:304] T 72b111ad7b5f4753ab1d2b5ee1114840 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda; no voters: 
I20260709 13:04:05.328270 10875 raft_consensus.cc:3060] T b67eff0634e540eab91cf8cee795969f P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Advancing to term 1
W20260709 13:04:05.346153 10992 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 241636us
W20260709 13:04:05.346985 10994 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 221719us
I20260709 13:04:05.347764 10955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d7ce47b9a92d43dd88d95bcda5038597" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:05.346410 10844 leader_election.cc:304] T 401a9cf11eac479aa445ec8c1d6e33b2 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:04:05.333287 11688 raft_consensus.cc:2804] T e89e640169e14e2d98f968f76027daa7 P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:04:05.328763 10948 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff13ae9a47c7455f962406c6d131a3ff" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:05.335162 11628 raft_consensus.cc:3060] T d71ca3a1ba6b48478aaf62741bf3cf6a P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:04:05.372598 10878 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51718 (request call id 190) took 1593 ms. Trace:
I20260709 13:04:05.364966 10994 leader_election.cc:304] T a64b7e99d6c54934a7de6f6dffcf5e94 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:04:05.368108 11067 leader_election.cc:403] T ff13ae9a47c7455f962406c6d131a3ff P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 1 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:04:05.332872 11658 raft_consensus.cc:697] T a5299b8d044f41b7b68057e63d17b159 P 186b12638a87410487467df721a3cbda [term 1 LEADER]: Becoming Leader. State: Replica: 186b12638a87410487467df721a3cbda, State: Running, Role: LEADER
I20260709 13:04:05.373147 10878 rpcz_store.cc:276] 0709 13:04:03.826155 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:04.810830 (+984675us) service_pool.cc:224] Handling call
0709 13:04:04.856644 (+ 45814us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:04.856896 (+   252us) raft_consensus.cc:1534] Early marking committed up to index 1
0709 13:04:05.162939 (+306043us) log.cc:844] Serialized 14 byte log entry
0709 13:04:05.372213 (+209274us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:05.372275 (+    62us) raft_consensus.cc:1645] Marking committed up to 1
0709 13:04:05.372323 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:05.372401 (+    78us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:05.372531 (+   130us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":431169,"prepare.run_cpu_time_us":54,"prepare.run_wall_time_us":55,"thread_start_us":488938,"threads_started":2}
I20260709 13:04:05.374716 10955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b67eff0634e540eab91cf8cee795969f" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:05.338390 10947 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75492f86fde046c3900df3eac8292fd4" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:05.349750 11518 raft_consensus.cc:493] T f34f2bc02d85444f9096e0f5b4fe4331 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:05.379370 10955 raft_consensus.cc:2393] T b67eff0634e540eab91cf8cee795969f P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0172fed22b6b4890a7fdc953fa128af4 in current term 1: Already voted for candidate 10ca5ee68ad64b3283b9d235352c9d23 in this term.
I20260709 13:04:05.413975 11148 tablet_bootstrap.cc:492] T f34f2bc02d85444f9096e0f5b4fe4331 P 10ca5ee68ad64b3283b9d235352c9d23: Bootstrap starting.
I20260709 13:04:05.333843 10845 leader_election.cc:304] T a1ec2b50c6ad4f7c8033476208abb26d P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 186b12638a87410487467df721a3cbda; no voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47
I20260709 13:04:05.414758 10955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f46a5102fd64b45bf1c4dff863d2b4d" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
W20260709 13:04:05.378937 11067 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:45807, user_credentials={real_user=slave}} blocked reactor thread for 126654us
I20260709 13:04:05.378069 11149 raft_consensus.cc:359] T a977842ba4b744f4bb2febd55d286006 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:05.418371 11149 raft_consensus.cc:385] T a977842ba4b744f4bb2febd55d286006 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:04:05.418365 11067 leader_election.cc:304] T 7709387f4d3d4f9992d97a0fbca39a40 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4; no voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47
I20260709 13:04:05.422386 10878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d7ce47b9a92d43dd88d95bcda5038597" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:05.419062 11149 raft_consensus.cc:740] T a977842ba4b744f4bb2febd55d286006 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 873357742d5843579556ee8166f23f47, State: Initialized, Role: FOLLOWER
I20260709 13:04:05.428825 11149 consensus_queue.cc:260] T a977842ba4b744f4bb2febd55d286006 P 873357742d5843579556ee8166f23f47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:05.324371 10874 rpcz_store.cc:276] 0709 13:04:03.920345 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:05.230217 (+1309872us) service_pool.cc:224] Handling call
0709 13:04:05.261036 (+ 30819us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:05.320405 (+ 59369us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:05.320655 (+   250us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:05.407599 11688 raft_consensus.cc:697] T e89e640169e14e2d98f968f76027daa7 P 186b12638a87410487467df721a3cbda [term 3 LEADER]: Becoming Leader. State: Replica: 186b12638a87410487467df721a3cbda, State: Running, Role: LEADER
I20260709 13:04:05.448325 10878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b67eff0634e540eab91cf8cee795969f" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:05.378424 11508 raft_consensus.cc:2804] T 8c160ddfe9b34bdaa3835d0915c1e06f P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:04:05.377349 11817 raft_consensus.cc:2804] T 72b111ad7b5f4753ab1d2b5ee1114840 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:04:05.379127 11782 raft_consensus.cc:2804] T 401a9cf11eac479aa445ec8c1d6e33b2 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:04:05.405324 11758 raft_consensus.cc:2749] T ff13ae9a47c7455f962406c6d131a3ff P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Leader pre-election lost for term 1. Reason: Vote denied by peer 873357742d5843579556ee8166f23f47 (127.0.107.195:45807) with higher term. Message: Invalid argument: T ff13ae9a47c7455f962406c6d131a3ff P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0172fed22b6b4890a7fdc953fa128af4 for earlier term 1. Current term is 2.
I20260709 13:04:05.418605 11757 raft_consensus.cc:493] T 836251ebc2b74f5da88b8a8b2338e2a5 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:05.369372 11780 consensus_queue.cc:1059] T 7757ecbf402e47bfbc93d0447741e26c P 873357742d5843579556ee8166f23f47 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.084s
W20260709 13:04:05.468314 10992 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 61481.7us
I20260709 13:04:05.472400 10955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "525bfdbd6d934afd823f7d3ee3f9d052" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:05.404904 11804 consensus_queue.cc:1048] T 11de93cd9b9d457489ff1cb80f976c9a P 873357742d5843579556ee8166f23f47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.046s
I20260709 13:04:05.427011 10952 raft_consensus.cc:1275] T 5de5f0581e3b41dea729363d6ca563bb P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Refusing update from remote peer 0172fed22b6b4890a7fdc953fa128af4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20260709 13:04:05.447695 11607 consensus_meta.cc:287] T e5a5f5a608504650b48fd27f163faed6 P 0172fed22b6b4890a7fdc953fa128af4: Time spent flushing consensus metadata: real 0.748s	user 0.008s	sys 0.007s
I20260709 13:04:05.423848 11871 raft_consensus.cc:2804] T a64b7e99d6c54934a7de6f6dffcf5e94 P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260709 13:04:05.478163 10994 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 97967.5us
I20260709 13:04:05.492208 10878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5de5f0581e3b41dea729363d6ca563bb" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 4 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:05.472913 10947 raft_consensus.cc:2393] T 75492f86fde046c3900df3eac8292fd4 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0172fed22b6b4890a7fdc953fa128af4 in current term 1: Already voted for candidate 873357742d5843579556ee8166f23f47 in this term.
I20260709 13:04:05.449677 11796 raft_consensus.cc:2749] T a1ec2b50c6ad4f7c8033476208abb26d P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:04:05.508278 10994 leader_election.cc:304] T ff13ae9a47c7455f962406c6d131a3ff P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47; no voters: 10ca5ee68ad64b3283b9d235352c9d23
I20260709 13:04:05.505576 11021 raft_consensus.cc:3055] T 868e92339c2f4508af0afbd07c814d64 P 873357742d5843579556ee8166f23f47 [term 1 LEADER]: Stepping down as leader of term 1
W20260709 13:04:05.506634 11795 consensus_meta.cc:287] T 4c2a35bb99374e93b573cf870d04faaa P 186b12638a87410487467df721a3cbda: Time spent flushing consensus metadata: real 0.580s	user 0.013s	sys 0.004s
I20260709 13:04:05.419305 11518 raft_consensus.cc:515] T f34f2bc02d85444f9096e0f5b4fe4331 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:05.505079 10878 raft_consensus.cc:2410] T 5de5f0581e3b41dea729363d6ca563bb P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 10ca5ee68ad64b3283b9d235352c9d23 for term 4 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
W20260709 13:04:05.480187 11545 consensus_meta.cc:287] T 7839ca2d2c13431a85f44d7c458c6351 P 186b12638a87410487467df721a3cbda: Time spent flushing consensus metadata: real 0.902s	user 0.014s	sys 0.001s
W20260709 13:04:05.475895 10845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 174700us
I20260709 13:04:05.580508 10871 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51704 (request call id 129) took 2335 ms. Trace:
I20260709 13:04:05.581053 10845 leader_election.cc:304] T 868e92339c2f4508af0afbd07c814d64 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 186b12638a87410487467df721a3cbda; no voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47
I20260709 13:04:05.581007 10871 rpcz_store.cc:276] 0709 13:04:03.245020 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:04.782944 (+1537924us) service_pool.cc:224] Handling call
0709 13:04:05.257782 (+474838us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:05.335983 (+ 78201us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:05.580246 (+244263us) spinlock_profiling.cc:243] Waited 468 ms on lock 0x7b5400370fd0. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661618ad 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:04:05.580426 (+   180us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1076326784}
I20260709 13:04:05.582660 10845 leader_election.cc:304] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 186b12638a87410487467df721a3cbda; no voters: 0172fed22b6b4890a7fdc953fa128af4, 10ca5ee68ad64b3283b9d235352c9d23
I20260709 13:04:05.584288 10845 leader_election.cc:304] T 7709387f4d3d4f9992d97a0fbca39a40 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda; no voters: 873357742d5843579556ee8166f23f47
W20260709 13:04:05.561797 10920 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 49552.9us
I20260709 13:04:05.557457 11782 raft_consensus.cc:697] T 401a9cf11eac479aa445ec8c1d6e33b2 P 186b12638a87410487467df721a3cbda [term 1 LEADER]: Becoming Leader. State: Replica: 186b12638a87410487467df721a3cbda, State: Running, Role: LEADER
I20260709 13:04:05.510313 11021 raft_consensus.cc:740] T 868e92339c2f4508af0afbd07c814d64 P 873357742d5843579556ee8166f23f47 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 873357742d5843579556ee8166f23f47, State: Running, Role: LEADER
I20260709 13:04:05.479373 11817 raft_consensus.cc:697] T 72b111ad7b5f4753ab1d2b5ee1114840 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 LEADER]: Becoming Leader. State: Replica: 10ca5ee68ad64b3283b9d235352c9d23, State: Running, Role: LEADER
I20260709 13:04:05.507491 11607 raft_consensus.cc:515] T e5a5f5a608504650b48fd27f163faed6 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:05.578863 10874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e4f2241a94a41f38e481137f868b83d" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:04:05.575273 11695 consensus_queue.cc:1059] T ff13ae9a47c7455f962406c6d131a3ff P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20260709 13:04:05.579627 11545 raft_consensus.cc:515] T 7839ca2d2c13431a85f44d7c458c6351 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:05.337229 10949 rpcz_store.cc:276] 0709 13:04:03.820638 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:04.232583 (+411945us) service_pool.cc:224] Handling call
0709 13:04:04.232947 (+   364us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:04.233228 (+   281us) raft_consensus.cc:1534] Early marking committed up to index 2
0709 13:04:04.513335 (+280107us) log.cc:844] Serialized 14 byte log entry
0709 13:04:05.310683 (+797348us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:05.310749 (+    66us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:04:05.310807 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:05.311454 (+   647us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:05.311625 (+   171us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":347470,"prepare.run_cpu_time_us":54,"prepare.run_wall_time_us":57,"raft.queue_time_us":33260,"raft.run_cpu_time_us":112,"raft.run_wall_time_us":112,"thread_start_us":1059895,"threads_started":3,"wal-append.queue_time_us":610036}
I20260709 13:04:05.503921 11871 raft_consensus.cc:493] T a64b7e99d6c54934a7de6f6dffcf5e94 P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:05.571781 11149 ts_tablet_manager.cc:1434] T a977842ba4b744f4bb2febd55d286006 P 873357742d5843579556ee8166f23f47: Time spent starting tablet: real 0.497s	user 0.007s	sys 0.005s
I20260709 13:04:05.507344 10948 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63d44f6ba6434ab2870fe3190f4d7c2f" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:05.593719 10878 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51718 (request call id 196) took 1646 ms. Trace:
I20260709 13:04:05.487073 11757 raft_consensus.cc:515] T 836251ebc2b74f5da88b8a8b2338e2a5 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
W20260709 13:04:05.601660 11149 ts_tablet_manager.cc:1467] T a977842ba4b744f4bb2febd55d286006 P 873357742d5843579556ee8166f23f47: Tablet startup took 527ms
W20260709 13:04:05.594477 10843 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:05.578884 10952 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34064dc7bf3747dbb3876dd824ee0688" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
W20260709 13:04:05.612673 11872 log.cc:927] Time spent T 8c160ddfe9b34bdaa3835d0915c1e06f P 186b12638a87410487467df721a3cbda: Append to log took a long time: real 0.105s	user 0.000s	sys 0.000s
W20260709 13:04:05.613000 10826 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 13:04:05.612854 10951 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bac7e5040f984ba2b7146bd36af7267e" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:05.591732 11796 raft_consensus.cc:2749] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 13:04:05.585186 10947 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a64b7e99d6c54934a7de6f6dffcf5e94" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
W20260709 13:04:05.599488 10876 consensus_meta.cc:287] T 732c4adeeff6422497aa7db4a82f535b P 186b12638a87410487467df721a3cbda: Time spent flushing consensus metadata: real 0.550s	user 0.017s	sys 0.000s
I20260709 13:04:05.590098 11795 raft_consensus.cc:2749] T 868e92339c2f4508af0afbd07c814d64 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:04:05.487231 11658 consensus_queue.cc:237] T a5299b8d044f41b7b68057e63d17b159 P 186b12638a87410487467df721a3cbda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:05.594732 11729 raft_consensus.cc:493] T 11de93cd9b9d457489ff1cb80f976c9a P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:05.611212 11723 raft_consensus.cc:493] T 0e5d327630e5438b878035d28efe6c0c P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:05.611883 10948 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11de93cd9b9d457489ff1cb80f976c9a" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
W20260709 13:04:05.599414 11100 consensus_meta.cc:287] T c31881534c5e4f0089c82057c1746652 P 0172fed22b6b4890a7fdc953fa128af4: Time spent flushing consensus metadata: real 0.560s	user 0.015s	sys 0.000s
I20260709 13:04:05.604586 10878 rpcz_store.cc:276] 0709 13:04:03.947376 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:05.592846 (+1645470us) service_pool.cc:224] Handling call
0709 13:04:05.593175 (+   329us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:05.593381 (+   206us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:05.593407 (+    26us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:05.593456 (+    49us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:04:05.593492 (+    36us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:05.593551 (+    59us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:05.593659 (+   108us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:05.605417 11871 raft_consensus.cc:3060] T a64b7e99d6c54934a7de6f6dffcf5e94 P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:04:05.598101 11758 consensus_queue.cc:1059] T 72b111ad7b5f4753ab1d2b5ee1114840 P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.024s
I20260709 13:04:05.605225 11879 raft_consensus.cc:2804] T ff13ae9a47c7455f962406c6d131a3ff P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Leader election won for term 2
W20260709 13:04:05.606953 11149 ts_tablet_manager.cc:1469] T a977842ba4b744f4bb2febd55d286006 P 873357742d5843579556ee8166f23f47: Trace:
0709 13:03:48.595967 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:48.610526 (+ 14559us) service_pool.cc:224] Handling call
0709 13:03:48.612175 (+  1649us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 13:03:48.612263 (+    88us) ts_tablet_manager.cc:682] Creating new metadata...
0709 13:03:48.643470 (+ 31207us) tablet_metadata.cc:665] Metadata flushed
0709 13:03:48.730094 (+ 86624us) tablet_replica.cc:206] Creating consensus instance
0709 13:03:48.732277 (+  2183us) inbound_call.cc:177] Queueing success response
0709 13:04:04.185883 (+15453606us) ts_tablet_manager.cc:1379] Bootstrapping tablet
0709 13:04:05.074308 (+888425us) ts_tablet_manager.cc:1435] Starting tablet replica
0709 13:04:05.299316 (+225008us) tablet_replica.cc:261] Starting instrumentation
0709 13:04:05.300171 (+   855us) tablet_replica.cc:280] Starting consensus
Metrics: {"mutex_wait_us":38131,"raft.queue_time_us":74029,"raft.run_cpu_time_us":10801,"raft.run_wall_time_us":61886,"tablet-open.queue_time_us":15453655}
I20260709 13:04:05.618562 11021 consensus_queue.cc:260] T 868e92339c2f4508af0afbd07c814d64 P 873357742d5843579556ee8166f23f47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
W20260709 13:04:05.632315 11881 log.cc:927] Time spent T 4c2a35bb99374e93b573cf870d04faaa P 873357742d5843579556ee8166f23f47: Append to log took a long time: real 0.058s	user 0.000s	sys 0.000s
W20260709 13:04:05.632787 10843 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:05.633345 10953 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49434 (request call id 155) took 1357 ms (client timeout 1777 ms). Trace:
W20260709 13:04:05.634284 10872 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51696 (request call id 188) took 1577 ms (client timeout 1705 ms). Trace:
I20260709 13:04:05.634478 10879 raft_consensus.cc:1240] T 57b6a2418c1342e7924195e9c1dbd357 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Rejecting Update request from peer 873357742d5843579556ee8166f23f47 for earlier term 1. Current term is 2. Ops: []
I20260709 13:04:05.635052 11817 consensus_queue.cc:237] T 72b111ad7b5f4753ab1d2b5ee1114840 P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:05.636046 11777 raft_consensus.cc:493] T 8a69f32c53f24a50b7852d143665d7b4 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:05.637804 11729 raft_consensus.cc:515] T 11de93cd9b9d457489ff1cb80f976c9a P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:05.638290 11610 leader_election.cc:290] T 7757ecbf402e47bfbc93d0447741e26c P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 2 election: Requested vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:05.629314 10877 raft_consensus.cc:1240] T 7839ca2d2c13431a85f44d7c458c6351 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Rejecting Update request from peer 873357742d5843579556ee8166f23f47 for earlier term 1. Current term is 2. Ops: []
I20260709 13:04:05.639966 11607 leader_election.cc:290] T e5a5f5a608504650b48fd27f163faed6 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 3 election: Requested vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:05.641957 11025 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4cd3d7dfbbf4096908ff11e9dc57141" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
I20260709 13:04:05.642990 11028 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a64b7e99d6c54934a7de6f6dffcf5e94" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:05.607995 11883 raft_consensus.cc:2804] T 7709387f4d3d4f9992d97a0fbca39a40 P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:04:05.616240 11757 leader_election.cc:290] T 836251ebc2b74f5da88b8a8b2338e2a5 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:05.612874 10952 raft_consensus.cc:2468] T 34064dc7bf3747dbb3876dd824ee0688 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 3.
I20260709 13:04:05.652405 10878 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51696 (request call id 187) took 1600 ms. Trace:
I20260709 13:04:05.692147 10878 rpcz_store.cc:276] 0709 13:04:04.052476 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:05.651669 (+1599193us) service_pool.cc:224] Handling call
0709 13:04:05.651944 (+   275us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:05.652116 (+   172us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:05.652131 (+    15us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:05.652165 (+    34us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:04:05.652184 (+    19us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:05.652218 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:05.652340 (+   122us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:05.666826 11879 raft_consensus.cc:697] T ff13ae9a47c7455f962406c6d131a3ff P 873357742d5843579556ee8166f23f47 [term 2 LEADER]: Becoming Leader. State: Replica: 873357742d5843579556ee8166f23f47, State: Running, Role: LEADER
I20260709 13:04:05.624363 10876 raft_consensus.cc:2468] T 732c4adeeff6422497aa7db4a82f535b P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 2.
I20260709 13:04:05.651223 11723 raft_consensus.cc:3060] T 0e5d327630e5438b878035d28efe6c0c P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:04:05.627861 10947 raft_consensus.cc:2468] T a64b7e99d6c54934a7de6f6dffcf5e94 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 3.
I20260709 13:04:05.657492 10951 raft_consensus.cc:1240] T 57b6a2418c1342e7924195e9c1dbd357 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Rejecting Update request from peer 873357742d5843579556ee8166f23f47 for earlier term 1. Current term is 2. Ops: []
I20260709 13:04:05.669438 11021 raft_consensus.cc:3060] T 868e92339c2f4508af0afbd07c814d64 P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:04:05.349562 11098 rpcz_store.cc:276] 0709 13:04:03.460405 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:04.662936 (+1202531us) service_pool.cc:224] Handling call
0709 13:04:04.663363 (+   427us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:04.663778 (+   415us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:04.663816 (+    38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:04.664431 (+   615us) log.cc:844] Serialized 26 byte log entry
0709 13:04:04.979193 (+314762us) raft_consensus.cc:1645] Marking committed up to 1
0709 13:04:04.979259 (+    66us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:04.979374 (+   115us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:05.303693 (+324319us) raft_consensus.cc:1685] finished
0709 13:04:05.303721 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:05.303893 (+   172us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":93200,"raft.run_cpu_time_us":206,"raft.run_wall_time_us":5683,"thread_start_us":314206,"threads_started":1,"wal-append.queue_time_us":536805}
I20260709 13:04:05.608891 11688 consensus_queue.cc:237] T e89e640169e14e2d98f968f76027daa7 P 186b12638a87410487467df721a3cbda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:05.694016 10878 raft_consensus.cc:1275] T 11de93cd9b9d457489ff1cb80f976c9a P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Refusing update from remote peer 873357742d5843579556ee8166f23f47: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 13:04:05.669880 11149 tablet_bootstrap.cc:492] T a1ec2b50c6ad4f7c8033476208abb26d P 873357742d5843579556ee8166f23f47: Bootstrap starting.
I20260709 13:04:05.661630 11100 raft_consensus.cc:2468] T c31881534c5e4f0089c82057c1746652 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 2.
W20260709 13:04:05.667066 10872 rpcz_store.cc:269] 0709 13:04:04.056774 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:04.166527 (+109753us) service_pool.cc:224] Handling call
0709 13:04:05.634051 (+1467524us) spinlock_profiling.cc:243] Waited 241 ms on lock 0x7b54003219d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:04:05.634217 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":554943872}
I20260709 13:04:05.684079 11777 raft_consensus.cc:515] T 8a69f32c53f24a50b7852d143665d7b4 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:05.649358 11782 consensus_queue.cc:237] T 401a9cf11eac479aa445ec8c1d6e33b2 P 186b12638a87410487467df721a3cbda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:05.690043 11883 raft_consensus.cc:493] T 7709387f4d3d4f9992d97a0fbca39a40 P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:05.704286 10878 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51718 (request call id 197) took 1521 ms. Trace:
I20260709 13:04:05.723173 10878 rpcz_store.cc:276] 0709 13:04:04.183378 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:05.693314 (+1509936us) service_pool.cc:224] Handling call
0709 13:04:05.693687 (+   373us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:05.704016 (+ 10329us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:05.704232 (+   216us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:05.725461 10878 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51718 (request call id 200) took 1313 ms. Trace:
W20260709 13:04:05.667202 10953 rpcz_store.cc:269] 0709 13:04:04.275437 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:04.281074 (+  5637us) service_pool.cc:224] Handling call
0709 13:04:05.633122 (+1352048us) spinlock_profiling.cc:243] Waited 582 ms on lock 0x7b5400416254. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:04:05.633271 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1338173824}
I20260709 13:04:05.702580 11880 raft_consensus.cc:493] T 6a9353d1d230416296627d7cf8b1d425 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 186b12638a87410487467df721a3cbda)
I20260709 13:04:05.712468 10871 raft_consensus.cc:1240] T 732c4adeeff6422497aa7db4a82f535b P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Rejecting Update request from peer 873357742d5843579556ee8166f23f47 for earlier term 1. Current term is 2. Ops: []
I20260709 13:04:05.716444 11152 tablet_bootstrap.cc:654] T a977842ba4b744f4bb2febd55d286006 P 0172fed22b6b4890a7fdc953fa128af4: Neither blocks nor log segments found. Creating new log.
I20260709 13:04:05.717779 11027 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46628 (request call id 145) took 1827 ms. Trace:
I20260709 13:04:05.719537 10876 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51696 (request call id 190) took 1317 ms. Trace:
I20260709 13:04:05.725961 10878 rpcz_store.cc:276] 0709 13:04:04.423765 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:05.724303 (+1300538us) service_pool.cc:224] Handling call
0709 13:04:05.724743 (+   440us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:05.725018 (+   275us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:05.725081 (+    63us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:05.725145 (+    64us) raft_consensus.cc:1645] Marking committed up to 1
0709 13:04:05.725190 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:05.725260 (+    70us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:05.725386 (+   126us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:05.729626 11021 raft_consensus.cc:1275] T 868e92339c2f4508af0afbd07c814d64 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Refusing update from remote peer 0172fed22b6b4890a7fdc953fa128af4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:04:05.694953 10948 raft_consensus.cc:2468] T 11de93cd9b9d457489ff1cb80f976c9a P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 3.
I20260709 13:04:05.739290 10878 raft_consensus.cc:1275] T d270b9e5481245d9b5340a454a7cd10b P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Refusing update from remote peer 10ca5ee68ad64b3283b9d235352c9d23: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 13:04:05.727231 10874 raft_consensus.cc:1275] T 5de5f0581e3b41dea729363d6ca563bb P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Refusing update from remote peer 0172fed22b6b4890a7fdc953fa128af4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20260709 13:04:05.730400 11877 log.cc:927] Time spent T 5d156d6ae2ca4fa59a1e387edf0fea39 P 10ca5ee68ad64b3283b9d235352c9d23: Append to log took a long time: real 0.156s	user 0.000s	sys 0.001s
I20260709 13:04:05.726786 11883 raft_consensus.cc:3060] T 7709387f4d3d4f9992d97a0fbca39a40 P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:04:05.731812 11027 rpcz_store.cc:276] 0709 13:04:03.909214 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:04.663103 (+753889us) service_pool.cc:224] Handling call
0709 13:04:04.663403 (+   300us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:04.733070 (+ 69667us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:04.743497 (+ 10427us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:04.746547 (+  3050us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:04:04.755687 (+  9140us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:05.717499 (+961812us) spinlock_profiling.cc:243] Waited 47.8 ms on lock 0x7b54005ffad4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:04:05.717560 (+    61us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:05.717707 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":109991168}
I20260709 13:04:05.749056 11729 leader_election.cc:290] T 11de93cd9b9d457489ff1cb80f976c9a P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:05.751639 10875 raft_consensus.cc:2468] T b67eff0634e540eab91cf8cee795969f P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 1.
I20260709 13:04:05.753633 11148 tablet_bootstrap.cc:654] T f34f2bc02d85444f9096e0f5b4fe4331 P 10ca5ee68ad64b3283b9d235352c9d23: Neither blocks nor log segments found. Creating new log.
I20260709 13:04:05.753772 10875 raft_consensus.cc:1275] T 836251ebc2b74f5da88b8a8b2338e2a5 P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Refusing update from remote peer 10ca5ee68ad64b3283b9d235352c9d23: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 13:04:05.735883 10947 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e96e80ba9ac34d0998395515e5c8c41a" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:04:05.735253 11518 leader_election.cc:290] T f34f2bc02d85444f9096e0f5b4fe4331 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:05.743501 10953 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7839ca2d2c13431a85f44d7c458c6351" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:04:05.732071 11880 raft_consensus.cc:515] T 6a9353d1d230416296627d7cf8b1d425 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:05.765985 11149 tablet_bootstrap.cc:654] T a1ec2b50c6ad4f7c8033476208abb26d P 873357742d5843579556ee8166f23f47: Neither blocks nor log segments found. Creating new log.
I20260709 13:04:05.755678 10872 raft_consensus.cc:1275] T e96e80ba9ac34d0998395515e5c8c41a P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Refusing update from remote peer 873357742d5843579556ee8166f23f47: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 13:04:05.757311 11849 raft_consensus.cc:2749] T 7709387f4d3d4f9992d97a0fbca39a40 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:04:05.756403 11887 raft_consensus.cc:493] T 8a69f32c53f24a50b7852d143665d7b4 P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:05.767155 10947 raft_consensus.cc:2368] T e96e80ba9ac34d0998395515e5c8c41a P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 186b12638a87410487467df721a3cbda for earlier term 2. Current term is 3.
I20260709 13:04:05.745369 11633 raft_consensus.cc:515] T e89e640169e14e2d98f968f76027daa7 P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:05.764405 10874 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51704 (request call id 153) took 1699 ms. Trace:
I20260709 13:04:05.733232 10876 rpcz_store.cc:276] 0709 13:04:04.404167 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:05.718496 (+1314329us) service_pool.cc:224] Handling call
0709 13:04:05.718919 (+   423us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:05.719154 (+   235us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:05.719179 (+    25us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:05.719225 (+    46us) raft_consensus.cc:1645] Marking committed up to 1
0709 13:04:05.719259 (+    34us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:05.719328 (+    69us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:05.719460 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:05.783481 11518 consensus_queue.cc:1059] T c31881534c5e4f0089c82057c1746652 P 186b12638a87410487467df721a3cbda [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20260709 13:04:05.792814 11777 leader_election.cc:290] T 8a69f32c53f24a50b7852d143665d7b4 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:05.784024 11887 raft_consensus.cc:515] T 8a69f32c53f24a50b7852d143665d7b4 P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
W20260709 13:04:05.810088 11875 log.cc:927] Time spent T 1e4f2241a94a41f38e481137f868b83d P 873357742d5843579556ee8166f23f47: Append to log took a long time: real 0.231s	user 0.003s	sys 0.000s
I20260709 13:04:05.795825 10872 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51718 (request call id 202) took 1119 ms. Trace:
W20260709 13:04:05.830021 10921 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:38967, user_credentials={real_user=slave}} blocked reactor thread for 70015.9us
W20260709 13:04:05.829970 11067 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:45807, user_credentials={real_user=slave}} blocked reactor thread for 211543us
I20260709 13:04:05.832343 11757 raft_consensus.cc:493] T 401a9cf11eac479aa445ec8c1d6e33b2 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:05.833171 11757 raft_consensus.cc:515] T 401a9cf11eac479aa445ec8c1d6e33b2 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:05.762801 11879 consensus_queue.cc:237] T ff13ae9a47c7455f962406c6d131a3ff P 873357742d5843579556ee8166f23f47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:05.845912 11757 leader_election.cc:290] T 401a9cf11eac479aa445ec8c1d6e33b2 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:05.847869 11887 leader_election.cc:290] T 8a69f32c53f24a50b7852d143665d7b4 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:05.851228 11633 leader_election.cc:290] T e89e640169e14e2d98f968f76027daa7 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 4 election: Requested vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:05.830896 10872 rpcz_store.cc:276] 0709 13:04:04.665267 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:05.732863 (+1067596us) service_pool.cc:224] Handling call
0709 13:04:05.737945 (+  5082us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:05.776254 (+ 38309us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:05.784153 (+  7899us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:04:05.864938 10992 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:05.863670 11607 raft_consensus.cc:493] T d270b9e5481245d9b5340a454a7cd10b P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:05.870922 11607 raft_consensus.cc:515] T d270b9e5481245d9b5340a454a7cd10b P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:05.873256 11152 tablet_bootstrap.cc:492] T a977842ba4b744f4bb2febd55d286006 P 0172fed22b6b4890a7fdc953fa128af4: No bootstrap required, opened a new log
I20260709 13:04:05.873579 11607 leader_election.cc:290] T d270b9e5481245d9b5340a454a7cd10b P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:05.874225 11152 ts_tablet_manager.cc:1403] T a977842ba4b744f4bb2febd55d286006 P 0172fed22b6b4890a7fdc953fa128af4: Time spent bootstrapping tablet: real 0.553s	user 0.016s	sys 0.014s
W20260709 13:04:05.875944 11100 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37666 (request call id 151) took 1742 ms (client timeout 2000 ms). Trace:
I20260709 13:04:05.876526 10953 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e89e640169e14e2d98f968f76027daa7" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:04:05.877437 10953 raft_consensus.cc:3060] T e89e640169e14e2d98f968f76027daa7 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:04:05.877799 11152 raft_consensus.cc:359] T a977842ba4b744f4bb2febd55d286006 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:05.878544 11152 raft_consensus.cc:385] T a977842ba4b744f4bb2febd55d286006 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:04:05.878819 11152 raft_consensus.cc:740] T a977842ba4b744f4bb2febd55d286006 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0172fed22b6b4890a7fdc953fa128af4, State: Initialized, Role: FOLLOWER
I20260709 13:04:05.879580 11152 consensus_queue.cc:260] T a977842ba4b744f4bb2febd55d286006 P 0172fed22b6b4890a7fdc953fa128af4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:05.881040 10955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e89e640169e14e2d98f968f76027daa7" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:05.864053 11880 leader_election.cc:290] T 6a9353d1d230416296627d7cf8b1d425 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:05.881832 11887 consensus_queue.cc:1048] T 1e4f2241a94a41f38e481137f868b83d P 186b12638a87410487467df721a3cbda [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:04:05.795161 10874 rpcz_store.cc:276] 0709 13:04:04.062117 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:05.667723 (+1605606us) service_pool.cc:224] Handling call
0709 13:04:05.695220 (+ 27497us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:05.742363 (+ 47143us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:05.761740 (+ 19377us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:05.884059 10952 raft_consensus.cc:1275] T 34064dc7bf3747dbb3876dd824ee0688 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Refusing update from remote peer 873357742d5843579556ee8166f23f47: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 13:04:05.884339 10843 leader_election.cc:304] T 34064dc7bf3747dbb3876dd824ee0688 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda; no voters: 873357742d5843579556ee8166f23f47
I20260709 13:04:05.885609 10952 raft_consensus.cc:1275] T a154cc8b181f460abf4880064379948b P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Refusing update from remote peer 873357742d5843579556ee8166f23f47: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:04:05.886726 11099 raft_consensus.cc:1240] T c31881534c5e4f0089c82057c1746652 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Rejecting Update request from peer 186b12638a87410487467df721a3cbda for earlier term 1. Current term is 2. Ops: []
I20260709 13:04:05.887140 10951 raft_consensus.cc:1275] T e96e80ba9ac34d0998395515e5c8c41a P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Refusing update from remote peer 873357742d5843579556ee8166f23f47: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 13:04:05.888689 10951 raft_consensus.cc:1275] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Refusing update from remote peer 873357742d5843579556ee8166f23f47: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 13:04:05.888938 11099 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37678 (request call id 70) took 2447 ms. Trace:
I20260709 13:04:05.894950 11102 raft_consensus.cc:1240] T 7757ecbf402e47bfbc93d0447741e26c P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Rejecting Update request from peer 873357742d5843579556ee8166f23f47 for earlier term 1. Current term is 2. Ops: []
I20260709 13:04:05.897154 11098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a9353d1d230416296627d7cf8b1d425" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:05.898078 11098 raft_consensus.cc:2393] T 6a9353d1d230416296627d7cf8b1d425 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 873357742d5843579556ee8166f23f47 in current term 3: Already voted for candidate 0172fed22b6b4890a7fdc953fa128af4 in this term.
I20260709 13:04:05.895768 11887 raft_consensus.cc:2804] T 34064dc7bf3747dbb3876dd824ee0688 P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:04:05.899029 11887 raft_consensus.cc:493] T 34064dc7bf3747dbb3876dd824ee0688 P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:05.899246 11094 raft_consensus.cc:1275] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Refusing update from remote peer 873357742d5843579556ee8166f23f47: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 13:04:05.900375 11095 raft_consensus.cc:3060] T 401a9cf11eac479aa445ec8c1d6e33b2 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:04:05.899457 11887 raft_consensus.cc:3060] T 34064dc7bf3747dbb3876dd824ee0688 P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:04:05.885847 10843 leader_election.cc:400] T e96e80ba9ac34d0998395515e5c8c41a P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 2 election: Vote denied by peer 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903) with higher term. Message: Invalid argument: T e96e80ba9ac34d0998395515e5c8c41a P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 186b12638a87410487467df721a3cbda for earlier term 2. Current term is 3.
I20260709 13:04:05.862493 11846 raft_consensus.cc:493] T a977842ba4b744f4bb2febd55d286006 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:05.903564 11846 raft_consensus.cc:515] T a977842ba4b744f4bb2febd55d286006 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:05.906605 11777 raft_consensus.cc:493] T a154cc8b181f460abf4880064379948b P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:05.907051 10951 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "836251ebc2b74f5da88b8a8b2338e2a5" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:05.907718 10946 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f34f2bc02d85444f9096e0f5b4fe4331" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:05.907338 11777 raft_consensus.cc:515] T a154cc8b181f460abf4880064379948b P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:05.889700 11099 rpcz_store.cc:276] 0709 13:04:03.441250 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:04.656787 (+1215537us) service_pool.cc:224] Handling call
0709 13:04:05.886504 (+1229717us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:05.887162 (+   658us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:05.888723 (+  1561us) spinlock_profiling.cc:243] Waited 1.22 s on lock 0x7b540058c5d0. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661618ad 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:04:05.888871 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2814346624}
I20260709 13:04:05.915261 11152 ts_tablet_manager.cc:1434] T a977842ba4b744f4bb2febd55d286006 P 0172fed22b6b4890a7fdc953fa128af4: Time spent starting tablet: real 0.041s	user 0.007s	sys 0.000s
I20260709 13:04:05.916592 11152 tablet_bootstrap.cc:492] T a1ec2b50c6ad4f7c8033476208abb26d P 0172fed22b6b4890a7fdc953fa128af4: Bootstrap starting.
I20260709 13:04:05.921761 10954 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a154cc8b181f460abf4880064379948b" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:05.925132 11777 leader_election.cc:290] T a154cc8b181f460abf4880064379948b P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:05.926720 11846 leader_election.cc:290] T a977842ba4b744f4bb2febd55d286006 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:05.927870 10952 raft_consensus.cc:1275] T 4c2a35bb99374e93b573cf870d04faaa P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Refusing update from remote peer 873357742d5843579556ee8166f23f47: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 13:04:05.928532 11871 raft_consensus.cc:515] T a64b7e99d6c54934a7de6f6dffcf5e94 P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:05.928951 10946 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a977842ba4b744f4bb2febd55d286006" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:05.929147 10952 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a69f32c53f24a50b7852d143665d7b4" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
W20260709 13:04:05.929289 10843 leader_election.cc:343] T f34f2bc02d85444f9096e0f5b4fe4331 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:04:05.929762 10946 raft_consensus.cc:2468] T a977842ba4b744f4bb2febd55d286006 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 0.
I20260709 13:04:05.929909 10952 raft_consensus.cc:2468] T 8a69f32c53f24a50b7852d143665d7b4 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 3.
W20260709 13:04:05.914757 11628 consensus_meta.cc:287] T d71ca3a1ba6b48478aaf62741bf3cf6a P 0172fed22b6b4890a7fdc953fa128af4: Time spent flushing consensus metadata: real 0.506s	user 0.013s	sys 0.003s
I20260709 13:04:05.930950 10947 raft_consensus.cc:1275] T 87cd8f5097624c8692f1d83544d8881e P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Refusing update from remote peer 186b12638a87410487467df721a3cbda: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 13:04:05.931370 10946 raft_consensus.cc:1240] T 1e4f2241a94a41f38e481137f868b83d P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Rejecting Update request from peer 186b12638a87410487467df721a3cbda for earlier term 1. Current term is 2. Ops: [1.1-1.1]
I20260709 13:04:05.931227 11628 raft_consensus.cc:515] T d71ca3a1ba6b48478aaf62741bf3cf6a P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:05.932972 11845 raft_consensus.cc:3055] T 7757ecbf402e47bfbc93d0447741e26c P 873357742d5843579556ee8166f23f47 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:04:05.933395 11845 raft_consensus.cc:740] T 7757ecbf402e47bfbc93d0447741e26c P 873357742d5843579556ee8166f23f47 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 873357742d5843579556ee8166f23f47, State: Running, Role: LEADER
I20260709 13:04:05.933564 11896 consensus_queue.cc:1048] T 87cd8f5097624c8692f1d83544d8881e P 186b12638a87410487467df721a3cbda [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:04:05.933983 10954 raft_consensus.cc:2393] T a154cc8b181f460abf4880064379948b P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0172fed22b6b4890a7fdc953fa128af4 in current term 2: Already voted for candidate 873357742d5843579556ee8166f23f47 in this term.
W20260709 13:04:05.876406 11100 rpcz_store.cc:269] 0709 13:04:04.133635 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:04.153257 (+ 19622us) service_pool.cc:224] Handling call
0709 13:04:05.875701 (+1722444us) spinlock_profiling.cc:243] Waited 594 ms on lock 0x7b540058c5d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:04:05.875848 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1367197440}
I20260709 13:04:05.920235 11782 raft_consensus.cc:493] T d270b9e5481245d9b5340a454a7cd10b P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:05.922235 11148 tablet_bootstrap.cc:492] T f34f2bc02d85444f9096e0f5b4fe4331 P 10ca5ee68ad64b3283b9d235352c9d23: No bootstrap required, opened a new log
I20260709 13:04:05.936168 11896 consensus_queue.cc:1059] T 1e4f2241a94a41f38e481137f868b83d P 186b12638a87410487467df721a3cbda [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.009s
I20260709 13:04:05.925191 11097 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37676 (request call id 164) took 2232 ms. Trace:
I20260709 13:04:05.946113 11097 rpcz_store.cc:276] 0709 13:04:03.692626 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:04.737876 (+1045250us) service_pool.cc:224] Handling call
0709 13:04:04.858945 (+121069us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:05.196853 (+337908us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:05.924839 (+727986us) spinlock_profiling.cc:243] Waited 230 ms on lock 0x7b540073ffd4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:04:05.925115 (+   276us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":528750976}
I20260709 13:04:05.946956 11871 leader_election.cc:290] T a64b7e99d6c54934a7de6f6dffcf5e94 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 4 election: Requested vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:05.949698 10947 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a64b7e99d6c54934a7de6f6dffcf5e94" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:04:05.934386 11845 consensus_queue.cc:260] T 7757ecbf402e47bfbc93d0447741e26c P 873357742d5843579556ee8166f23f47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:05.951186 11028 raft_consensus.cc:2393] T a64b7e99d6c54934a7de6f6dffcf5e94 P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 10ca5ee68ad64b3283b9d235352c9d23 in current term 4: Already voted for candidate 873357742d5843579556ee8166f23f47 in this term.
I20260709 13:04:05.951548 11845 raft_consensus.cc:3060] T 7757ecbf402e47bfbc93d0447741e26c P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:04:05.952289 11026 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f46a5102fd64b45bf1c4dff863d2b4d" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:05.952067 11887 raft_consensus.cc:515] T 34064dc7bf3747dbb3876dd824ee0688 P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:05.942373 11148 ts_tablet_manager.cc:1403] T f34f2bc02d85444f9096e0f5b4fe4331 P 10ca5ee68ad64b3283b9d235352c9d23: Time spent bootstrapping tablet: real 0.559s	user 0.021s	sys 0.002s
I20260709 13:04:05.957619 10843 leader_election.cc:304] T 8a69f32c53f24a50b7852d143665d7b4 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda; no voters: 
I20260709 13:04:05.937587 11099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a977842ba4b744f4bb2febd55d286006" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:05.958793 11099 raft_consensus.cc:2468] T a977842ba4b744f4bb2febd55d286006 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 0.
I20260709 13:04:05.954922 10947 raft_consensus.cc:3060] T a64b7e99d6c54934a7de6f6dffcf5e94 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:04:05.959431 11842 raft_consensus.cc:3055] T 72b111ad7b5f4753ab1d2b5ee1114840 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:04:05.959717 11896 raft_consensus.cc:3055] T 1e4f2241a94a41f38e481137f868b83d P 186b12638a87410487467df721a3cbda [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:04:05.960219 11896 raft_consensus.cc:740] T 1e4f2241a94a41f38e481137f868b83d P 186b12638a87410487467df721a3cbda [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 186b12638a87410487467df721a3cbda, State: Running, Role: LEADER
W20260709 13:04:05.960974 10992 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:05.959308 11658 raft_consensus.cc:2804] T 8a69f32c53f24a50b7852d143665d7b4 P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:04:05.961755 11658 raft_consensus.cc:493] T 8a69f32c53f24a50b7852d143665d7b4 P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:05.962110 11658 raft_consensus.cc:3060] T 8a69f32c53f24a50b7852d143665d7b4 P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:04:05.960074 11842 raft_consensus.cc:740] T 72b111ad7b5f4753ab1d2b5ee1114840 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0172fed22b6b4890a7fdc953fa128af4, State: Running, Role: LEADER
I20260709 13:04:05.938311 11723 raft_consensus.cc:515] T 0e5d327630e5438b878035d28efe6c0c P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:05.936479 11628 leader_election.cc:290] T d71ca3a1ba6b48478aaf62741bf3cf6a P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 2 election: Requested vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:05.937007 11782 raft_consensus.cc:515] T d270b9e5481245d9b5340a454a7cd10b P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:05.971073 11508 raft_consensus.cc:493] T 8c160ddfe9b34bdaa3835d0915c1e06f P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Starting leader election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:05.971606 11508 raft_consensus.cc:3060] T 8c160ddfe9b34bdaa3835d0915c1e06f P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:04:05.971316 11842 consensus_queue.cc:260] T 72b111ad7b5f4753ab1d2b5ee1114840 P 0172fed22b6b4890a7fdc953fa128af4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:05.972609 11782 leader_election.cc:290] T d270b9e5481245d9b5340a454a7cd10b P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:05.972890 11842 raft_consensus.cc:3060] T 72b111ad7b5f4753ab1d2b5ee1114840 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:04:05.974699 11095 raft_consensus.cc:2468] T 401a9cf11eac479aa445ec8c1d6e33b2 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 1.
I20260709 13:04:05.976351 11020 raft_consensus.cc:1275] T e5ba72ccd854430baadbc636dfd17136 P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Refusing update from remote peer 10ca5ee68ad64b3283b9d235352c9d23: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:04:05.978786 11658 raft_consensus.cc:515] T 8a69f32c53f24a50b7852d143665d7b4 P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:05.978668 11896 consensus_queue.cc:260] T 1e4f2241a94a41f38e481137f868b83d P 186b12638a87410487467df721a3cbda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:05.986020 10948 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d270b9e5481245d9b5340a454a7cd10b" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:05.979521 11883 raft_consensus.cc:515] T 7709387f4d3d4f9992d97a0fbca39a40 P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:05.980334 10873 raft_consensus.cc:1240] T 4b37f8fca8dc42718b2a42f0883a956f P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Rejecting Update request from peer 10ca5ee68ad64b3283b9d235352c9d23 for earlier term 1. Current term is 2. Ops: []
I20260709 13:04:05.987166 10873 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51696 (request call id 184) took 2081 ms. Trace:
I20260709 13:04:05.987520 10873 rpcz_store.cc:276] 0709 13:04:03.905833 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:05.233180 (+1327347us) service_pool.cc:224] Handling call
0709 13:04:05.979865 (+746685us) spinlock_profiling.cc:243] Waited 22.6 ms on lock 0x7b5800000098. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f426303db71 00007f426303da22 00007f426bf56c09 00007f426b0b0ee0 00007f426b17a0ab 00007f426b17a131 00007f426b13bfc0 00007f426b126929 00007f426b1263d2 00007f426469dbd4
0709 13:04:05.980117 (+   252us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:05.986871 (+  6754us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:05.987103 (+   232us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":52059520}
I20260709 13:04:05.988554 10843 leader_election.cc:304] T 11de93cd9b9d457489ff1cb80f976c9a P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda; no voters: 873357742d5843579556ee8166f23f47
I20260709 13:04:05.989737 11887 leader_election.cc:290] T 34064dc7bf3747dbb3876dd824ee0688 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 4 election: Requested vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:05.992228 11883 leader_election.cc:290] T 7709387f4d3d4f9992d97a0fbca39a40 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 4 election: Requested vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:05.964246 10953 raft_consensus.cc:2468] T e89e640169e14e2d98f968f76027daa7 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 4.
I20260709 13:04:05.981889 11553 raft_consensus.cc:493] T 63d44f6ba6434ab2870fe3190f4d7c2f P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 186b12638a87410487467df721a3cbda)
I20260709 13:04:05.994861 11553 raft_consensus.cc:515] T 63d44f6ba6434ab2870fe3190f4d7c2f P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:05.990345 11797 raft_consensus.cc:2804] T 11de93cd9b9d457489ff1cb80f976c9a P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:04:05.997088 11658 leader_election.cc:290] T 8a69f32c53f24a50b7852d143665d7b4 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 4 election: Requested vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:05.997283 11797 raft_consensus.cc:493] T 11de93cd9b9d457489ff1cb80f976c9a P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:05.997750 11797 raft_consensus.cc:3060] T 11de93cd9b9d457489ff1cb80f976c9a P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:04:05.999256 11842 consensus_queue.cc:1048] T 868e92339c2f4508af0afbd07c814d64 P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:04:05.992228 10948 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34064dc7bf3747dbb3876dd824ee0688" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:04:05.990562 10878 raft_consensus.cc:1240] T 7709387f4d3d4f9992d97a0fbca39a40 P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Rejecting Update request from peer 873357742d5843579556ee8166f23f47 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260709 13:04:05.966128 11518 raft_consensus.cc:493] T e5a5f5a608504650b48fd27f163faed6 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:05.981065 11723 leader_election.cc:290] T 0e5d327630e5438b878035d28efe6c0c P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 4 election: Requested vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
W20260709 13:04:06.013470 10992 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 44509.6us
I20260709 13:04:06.019738 10992 leader_election.cc:304] T 63d44f6ba6434ab2870fe3190f4d7c2f P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 873357742d5843579556ee8166f23f47; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda
I20260709 13:04:05.997089 11553 leader_election.cc:290] T 63d44f6ba6434ab2870fe3190f4d7c2f P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:06.014173 11097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a64b7e99d6c54934a7de6f6dffcf5e94" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:04:06.023058 11027 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d71ca3a1ba6b48478aaf62741bf3cf6a" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
I20260709 13:04:06.016482 10948 raft_consensus.cc:3060] T 34064dc7bf3747dbb3876dd824ee0688 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:04:06.023339 11583 raft_consensus.cc:2749] T 63d44f6ba6434ab2870fe3190f4d7c2f P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:04:06.018116 11518 raft_consensus.cc:515] T e5a5f5a608504650b48fd27f163faed6 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:06.048385 11885 consensus_queue.cc:1048] T 4c2a35bb99374e93b573cf870d04faaa P 873357742d5843579556ee8166f23f47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:04:06.050948 11518 leader_election.cc:290] T e5a5f5a608504650b48fd27f163faed6 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:06.047183 11633 consensus_queue.cc:1048] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 873357742d5843579556ee8166f23f47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.004s
I20260709 13:04:06.038101 11021 raft_consensus.cc:1275] T a610d1a71f52432e88fe56130f921fa2 P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Refusing update from remote peer 0172fed22b6b4890a7fdc953fa128af4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20260709 13:04:06.038286 11804 consensus_queue.cc:1048] T 34064dc7bf3747dbb3876dd824ee0688 P 873357742d5843579556ee8166f23f47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.015s
I20260709 13:04:06.045204 11879 consensus_queue.cc:1048] T e96e80ba9ac34d0998395515e5c8c41a P 873357742d5843579556ee8166f23f47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.009s
I20260709 13:04:06.054628 11024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "525bfdbd6d934afd823f7d3ee3f9d052" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:06.047120 11097 raft_consensus.cc:3060] T a64b7e99d6c54934a7de6f6dffcf5e94 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:04:06.049929 11873 consensus_queue.cc:1048] T a154cc8b181f460abf4880064379948b P 873357742d5843579556ee8166f23f47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.006s
I20260709 13:04:06.044544 11020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7757ecbf402e47bfbc93d0447741e26c" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
I20260709 13:04:06.046005 10992 leader_election.cc:304] T a977842ba4b744f4bb2febd55d286006 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:04:06.057430 10950 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4cd3d7dfbbf4096908ff11e9dc57141" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:04:06.058334 10950 raft_consensus.cc:2393] T b4cd3d7dfbbf4096908ff11e9dc57141 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 0172fed22b6b4890a7fdc953fa128af4 in current term 3: Already voted for candidate 10ca5ee68ad64b3283b9d235352c9d23 in this term.
I20260709 13:04:06.025341 11896 raft_consensus.cc:3060] T 1e4f2241a94a41f38e481137f868b83d P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:04:06.063019 10953 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a69f32c53f24a50b7852d143665d7b4" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:04:06.066416 10953 raft_consensus.cc:3060] T 8a69f32c53f24a50b7852d143665d7b4 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:04:06.068665 10871 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51718 (request call id 195) took 2121 ms. Trace:
I20260709 13:04:06.069250 10871 rpcz_store.cc:276] 0709 13:04:03.947020 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:05.582382 (+1635362us) service_pool.cc:224] Handling call
0709 13:04:05.698540 (+116158us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:05.733012 (+ 34472us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:06.068370 (+335358us) spinlock_profiling.cc:243] Waited 100 ms on lock 0x7b540071d9d0. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661618ad 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:04:06.068577 (+   207us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":230697344}
W20260709 13:04:06.079139 11029 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46640 (request call id 154) took 1741 ms (client timeout 1615 ms). Trace:
W20260709 13:04:06.079860 11029 rpcz_store.cc:269] 0709 13:04:04.337537 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:04.400677 (+ 63140us) service_pool.cc:224] Handling call
0709 13:04:06.078849 (+1678172us) spinlock_profiling.cc:243] Waited 528 ms on lock 0x7b540059f6d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:04:06.079030 (+   181us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1214099200}
W20260709 13:04:06.098737 10992 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 52782.4us
I20260709 13:04:06.101051 10921 leader_election.cc:304] T 732c4adeeff6422497aa7db4a82f535b P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda; no voters: 873357742d5843579556ee8166f23f47
W20260709 13:04:06.102734 10844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:06.105110 11029 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7839ca2d2c13431a85f44d7c458c6351" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
I20260709 13:04:06.106781 11029 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7709387f4d3d4f9992d97a0fbca39a40" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
I20260709 13:04:06.106383 10953 raft_consensus.cc:2468] T 8a69f32c53f24a50b7852d143665d7b4 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 4.
I20260709 13:04:06.109485 11021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e5d327630e5438b878035d28efe6c0c" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
I20260709 13:04:06.109764 11023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34064dc7bf3747dbb3876dd824ee0688" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
I20260709 13:04:06.107074 10952 raft_consensus.cc:1240] T 732c4adeeff6422497aa7db4a82f535b P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Rejecting Update request from peer 873357742d5843579556ee8166f23f47 for earlier term 1. Current term is 2. Ops: []
I20260709 13:04:06.111675 10880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f9df2bc281f4866b6da5da94a0defc7" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:06.111951 10921 leader_election.cc:304] T b67eff0634e540eab91cf8cee795969f P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda; no voters: 
I20260709 13:04:06.099999 10992 leader_election.cc:304] T e89e640169e14e2d98f968f76027daa7 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47; no voters: 
W20260709 13:04:06.112540 11870 log.cc:927] Time spent T 8c160ddfe9b34bdaa3835d0915c1e06f P 0172fed22b6b4890a7fdc953fa128af4: Append to log took a long time: real 0.635s	user 0.005s	sys 0.000s
I20260709 13:04:06.034832 11846 consensus_queue.cc:1059] T 57b6a2418c1342e7924195e9c1dbd357 P 873357742d5843579556ee8166f23f47 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.083s
I20260709 13:04:06.120805 11673 raft_consensus.cc:2804] T e89e640169e14e2d98f968f76027daa7 P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:04:06.131886 11152 tablet_bootstrap.cc:654] T a1ec2b50c6ad4f7c8033476208abb26d P 0172fed22b6b4890a7fdc953fa128af4: Neither blocks nor log segments found. Creating new log.
I20260709 13:04:06.107865 11795 raft_consensus.cc:493] T 5de5f0581e3b41dea729363d6ca563bb P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:06.109520 11873 raft_consensus.cc:2804] T a977842ba4b744f4bb2febd55d286006 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:04:06.114787 10949 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b67eff0634e540eab91cf8cee795969f" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:06.123360 11817 raft_consensus.cc:2804] T b67eff0634e540eab91cf8cee795969f P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:04:06.120381 11553 raft_consensus.cc:2804] T 732c4adeeff6422497aa7db4a82f535b P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Leader election won for term 2
W20260709 13:04:06.108395 10845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 524202us
I20260709 13:04:06.149205 10845 leader_election.cc:304] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda; no voters: 873357742d5843579556ee8166f23f47
I20260709 13:04:06.150695 11723 raft_consensus.cc:2804] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:04:06.151093 11723 raft_consensus.cc:493] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Starting leader election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:06.151451 11723 raft_consensus.cc:3060] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:04:06.128756 10950 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5a5f5a608504650b48fd27f163faed6" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:06.153623 10950 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b37f8fca8dc42718b2a42f0883a956f" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:04:06.154253 10950 raft_consensus.cc:2376] T 4b37f8fca8dc42718b2a42f0883a956f P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 873357742d5843579556ee8166f23f47 in term 2. Re-sending same reply.
I20260709 13:04:06.156996 10845 leader_election.cc:304] T acc241e7273d49149984968596242712 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 186b12638a87410487467df721a3cbda; no voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47
I20260709 13:04:06.126634 11748 consensus_queue.cc:1048] T 836251ebc2b74f5da88b8a8b2338e2a5 P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.037s
I20260709 13:04:06.160833 11729 raft_consensus.cc:493] T bb7d67be500947b3aef4c4245e33671c P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:06.161463 11729 raft_consensus.cc:515] T bb7d67be500947b3aef4c4245e33671c P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:06.162142 11518 raft_consensus.cc:2749] T acc241e7273d49149984968596242712 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:04:06.163076 11508 raft_consensus.cc:515] T 8c160ddfe9b34bdaa3835d0915c1e06f P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:06.136003 11780 consensus_queue.cc:1048] T 11de93cd9b9d457489ff1cb80f976c9a P 873357742d5843579556ee8166f23f47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.031s
I20260709 13:04:06.165684 11148 raft_consensus.cc:359] T f34f2bc02d85444f9096e0f5b4fe4331 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:06.168593 11148 raft_consensus.cc:385] T f34f2bc02d85444f9096e0f5b4fe4331 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:04:06.168926 11148 raft_consensus.cc:740] T f34f2bc02d85444f9096e0f5b4fe4331 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10ca5ee68ad64b3283b9d235352c9d23, State: Initialized, Role: FOLLOWER
I20260709 13:04:06.115401 11754 consensus_queue.cc:1048] T 5de5f0581e3b41dea729363d6ca563bb P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.029s
W20260709 13:04:06.181571 10995 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:38967, user_credentials={real_user=slave}} blocked reactor thread for 52713.5us
I20260709 13:04:06.117065 11628 consensus_queue.cc:1048] T a610d1a71f52432e88fe56130f921fa2 P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.052s
I20260709 13:04:06.123445 11867 consensus_queue.cc:1048] T d270b9e5481245d9b5340a454a7cd10b P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.051s
I20260709 13:04:06.142663 11873 raft_consensus.cc:493] T a977842ba4b744f4bb2febd55d286006 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:06.184988 10844 leader_election.cc:304] T 7839ca2d2c13431a85f44d7c458c6351 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 186b12638a87410487467df721a3cbda; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47
I20260709 13:04:06.144775 11639 raft_consensus.cc:3055] T ff13ae9a47c7455f962406c6d131a3ff P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:04:06.186918 11148 consensus_queue.cc:260] T f34f2bc02d85444f9096e0f5b4fe4331 P 10ca5ee68ad64b3283b9d235352c9d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:06.164572 11729 leader_election.cc:290] T bb7d67be500947b3aef4c4245e33671c P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:06.166123 11553 raft_consensus.cc:697] T 732c4adeeff6422497aa7db4a82f535b P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 LEADER]: Becoming Leader. State: Replica: 10ca5ee68ad64b3283b9d235352c9d23, State: Running, Role: LEADER
I20260709 13:04:06.172874 11842 raft_consensus.cc:493] T c51782cd9d064ec0949e84e24dd4de67 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 186b12638a87410487467df721a3cbda)
I20260709 13:04:06.171010 10875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63d44f6ba6434ab2870fe3190f4d7c2f" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:06.129742 10843 leader_election.cc:304] T 8a69f32c53f24a50b7852d143665d7b4 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda; no voters: 
I20260709 13:04:06.198799 11149 tablet_bootstrap.cc:492] T a1ec2b50c6ad4f7c8033476208abb26d P 873357742d5843579556ee8166f23f47: No bootstrap required, opened a new log
I20260709 13:04:06.208904 11149 ts_tablet_manager.cc:1403] T a1ec2b50c6ad4f7c8033476208abb26d P 873357742d5843579556ee8166f23f47: Time spent bootstrapping tablet: real 0.539s	user 0.019s	sys 0.035s
W20260709 13:04:06.202740 10993 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:06.177958 11817 raft_consensus.cc:697] T b67eff0634e540eab91cf8cee795969f P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 LEADER]: Becoming Leader. State: Replica: 10ca5ee68ad64b3283b9d235352c9d23, State: Running, Role: LEADER
I20260709 13:04:06.179513 11508 leader_election.cc:290] T 8c160ddfe9b34bdaa3835d0915c1e06f P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 2 election: Requested vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:06.181175 11673 raft_consensus.cc:697] T e89e640169e14e2d98f968f76027daa7 P 873357742d5843579556ee8166f23f47 [term 4 LEADER]: Becoming Leader. State: Replica: 873357742d5843579556ee8166f23f47, State: Running, Role: LEADER
I20260709 13:04:06.192246 11853 raft_consensus.cc:493] T 9a1716c0adf34bee8a4e99341ba5788f P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:06.195041 11518 raft_consensus.cc:2749] T 7839ca2d2c13431a85f44d7c458c6351 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 13:04:06.205196 11021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c160ddfe9b34bdaa3835d0915c1e06f" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
I20260709 13:04:06.196938 11859 raft_consensus.cc:493] T 75492f86fde046c3900df3eac8292fd4 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:06.195940 11639 raft_consensus.cc:740] T ff13ae9a47c7455f962406c6d131a3ff P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 10ca5ee68ad64b3283b9d235352c9d23, State: Running, Role: LEADER
I20260709 13:04:06.192332 11506 raft_consensus.cc:493] T f371d2565dd84667b5613848604949bb P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:06.171396 11795 raft_consensus.cc:515] T 5de5f0581e3b41dea729363d6ca563bb P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:06.206765 11545 leader_election.cc:290] T 7839ca2d2c13431a85f44d7c458c6351 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 2 election: Requested vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:06.206877 11553 consensus_queue.cc:237] T 732c4adeeff6422497aa7db4a82f535b P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:06.181411 11871 consensus_queue.cc:1059] T 732c4adeeff6422497aa7db4a82f535b P 873357742d5843579556ee8166f23f47 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.026s
I20260709 13:04:06.207743 11022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "401a9cf11eac479aa445ec8c1d6e33b2" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:06.193193 11873 raft_consensus.cc:3060] T a977842ba4b744f4bb2febd55d286006 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:04:06.163859 10845 leader_election.cc:304] T 7757ecbf402e47bfbc93d0447741e26c P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 186b12638a87410487467df721a3cbda; no voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47
I20260709 13:04:06.223212 11853 raft_consensus.cc:515] T 9a1716c0adf34bee8a4e99341ba5788f P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
W20260709 13:04:06.255175 10845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 91426us
I20260709 13:04:06.256101 10845 leader_election.cc:304] T b67eff0634e540eab91cf8cee795969f P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda; no voters: 
I20260709 13:04:06.256699 11853 leader_election.cc:290] T 9a1716c0adf34bee8a4e99341ba5788f P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:06.193661 11780 consensus_queue.cc:1048] T e96e80ba9ac34d0998395515e5c8c41a P 873357742d5843579556ee8166f23f47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.022s
I20260709 13:04:06.257623 11845 raft_consensus.cc:493] T 9a1716c0adf34bee8a4e99341ba5788f P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:06.258586 11887 raft_consensus.cc:2764] T b67eff0634e540eab91cf8cee795969f P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260709 13:04:06.258354 11845 raft_consensus.cc:515] T 9a1716c0adf34bee8a4e99341ba5788f P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:06.259356 11887 consensus_queue.cc:1048] T 87cd8f5097624c8692f1d83544d8881e P 186b12638a87410487467df721a3cbda [LEADER]: Connected to new peer: Peer: permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:04:06.262883 11095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a1716c0adf34bee8a4e99341ba5788f" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:06.263789 11095 raft_consensus.cc:2468] T 9a1716c0adf34bee8a4e99341ba5788f P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 3.
I20260709 13:04:06.268016 11803 raft_consensus.cc:3060] T d371c5e27bd843469d2e2e453734cb4a P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:04:06.268597 11845 leader_election.cc:290] T 9a1716c0adf34bee8a4e99341ba5788f P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:06.262223 11099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f34f2bc02d85444f9096e0f5b4fe4331" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:06.234300 10947 raft_consensus.cc:2468] T a64b7e99d6c54934a7de6f6dffcf5e94 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 4.
I20260709 13:04:06.239869 11859 raft_consensus.cc:515] T 75492f86fde046c3900df3eac8292fd4 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
W20260709 13:04:06.223906 10843 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 99909us
I20260709 13:04:06.211653 11842 raft_consensus.cc:515] T c51782cd9d064ec0949e84e24dd4de67 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:06.234764 11688 raft_consensus.cc:3055] T c31881534c5e4f0089c82057c1746652 P 186b12638a87410487467df721a3cbda [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:04:06.236404 11639 consensus_queue.cc:260] T ff13ae9a47c7455f962406c6d131a3ff P 10ca5ee68ad64b3283b9d235352c9d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:06.237341 11779 leader_election.cc:290] T 1e4f2241a94a41f38e481137f868b83d P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 2 election: Requested vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:06.226080 11883 raft_consensus.cc:2804] T 8a69f32c53f24a50b7852d143665d7b4 P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:04:06.237733 11673 consensus_queue.cc:237] T e89e640169e14e2d98f968f76027daa7 P 873357742d5843579556ee8166f23f47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:06.238350 11101 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37676 (request call id 163) took 2546 ms. Trace:
I20260709 13:04:06.238396 10878 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51718 (request call id 203) took 1253 ms. Trace:
I20260709 13:04:06.317203 10878 rpcz_store.cc:276] 0709 13:04:04.985262 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:05.740223 (+754961us) service_pool.cc:224] Handling call
0709 13:04:05.740457 (+   234us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:06.008062 (+267605us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:06.238105 (+230043us) spinlock_profiling.cc:243] Waited 250 ms on lock 0x7b540045f454. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:04:06.238308 (+   203us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":574598912}
I20260709 13:04:06.319147 10878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "836251ebc2b74f5da88b8a8b2338e2a5" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:06.320920 10878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5a5f5a608504650b48fd27f163faed6" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:04:06.235455 11097 raft_consensus.cc:2468] T a64b7e99d6c54934a7de6f6dffcf5e94 P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 4.
I20260709 13:04:06.244626 11148 ts_tablet_manager.cc:1434] T f34f2bc02d85444f9096e0f5b4fe4331 P 10ca5ee68ad64b3283b9d235352c9d23: Time spent starting tablet: real 0.288s	user 0.007s	sys 0.003s
I20260709 13:04:06.250725 11022 raft_consensus.cc:2393] T 401a9cf11eac479aa445ec8c1d6e33b2 P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0172fed22b6b4890a7fdc953fa128af4 in current term 1: Already voted for candidate 186b12638a87410487467df721a3cbda in this term.
I20260709 13:04:06.253485 11795 leader_election.cc:290] T 5de5f0581e3b41dea729363d6ca563bb P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:06.253726 11797 raft_consensus.cc:515] T 11de93cd9b9d457489ff1cb80f976c9a P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:06.255259 11518 raft_consensus.cc:2749] T 7757ecbf402e47bfbc93d0447741e26c P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 13:04:06.229687 11506 raft_consensus.cc:515] T f371d2565dd84667b5613848604949bb P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:06.288635 11066 leader_election.cc:304] T bb7d67be500947b3aef4c4245e33671c P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47
I20260709 13:04:06.291858 10992 leader_election.cc:304] T a64b7e99d6c54934a7de6f6dffcf5e94 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:04:06.308982 10994 leader_election.cc:304] T 9a1716c0adf34bee8a4e99341ba5788f P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47; no voters: 
W20260709 13:04:06.341533 11066 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 52966us
I20260709 13:04:06.342725 11066 leader_election.cc:304] T d7ce47b9a92d43dd88d95bcda5038597 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda
I20260709 13:04:06.322283 10878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "401a9cf11eac479aa445ec8c1d6e33b2" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:06.315881 11101 rpcz_store.cc:276] 0709 13:04:03.692241 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:04.702508 (+1010267us) service_pool.cc:224] Handling call
0709 13:04:04.703049 (+   541us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:05.095600 (+392551us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:05.095662 (+    62us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:05.096709 (+  1047us) log.cc:844] Serialized 26 byte log entry
0709 13:04:05.334628 (+237919us) raft_consensus.cc:1645] Marking committed up to 0
0709 13:04:05.334683 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:05.976782 (+642099us) spinlock_profiling.cc:243] Waited 41.4 ms on lock 0x7b54005f28d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:04:05.976838 (+    56us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:06.111887 (+135049us) spinlock_profiling.cc:243] Waited 2.47 ms on lock 0x7b4400022b4c. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f4262f9bc85 00007f4262f9c6a2 00007f4265d0e1db 00007f4265d09aaf 00007f4265cfd86e 00007f4265cfcbb6 00007f4265cfc7d8 00007f4265d06221 00007f4265d061d9 00007f4265d06169
0709 13:04:06.238086 (+126199us) raft_consensus.cc:1685] finished
0709 13:04:06.238111 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:06.238279 (+   168us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":13884,"raft.queue_time_us":34240,"raft.run_cpu_time_us":91,"raft.run_wall_time_us":91,"spinlock_wait_cycles":100963328,"thread_start_us":573855,"threads_started":2,"wal-append.queue_time_us":313532}
I20260709 13:04:06.245097 11027 raft_consensus.cc:2468] T d71ca3a1ba6b48478aaf62741bf3cf6a P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 2.
I20260709 13:04:06.282486 11859 leader_election.cc:290] T 75492f86fde046c3900df3eac8292fd4 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:06.307905 11688 raft_consensus.cc:740] T c31881534c5e4f0089c82057c1746652 P 186b12638a87410487467df721a3cbda [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 186b12638a87410487467df721a3cbda, State: Running, Role: LEADER
I20260709 13:04:06.328722 11024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a1716c0adf34bee8a4e99341ba5788f" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:06.285993 11723 raft_consensus.cc:515] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
W20260709 13:04:06.347600 10992 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 55852us
I20260709 13:04:06.333112 11094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a69f32c53f24a50b7852d143665d7b4" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
W20260709 13:04:06.377480 10920 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:06.434849 10878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d270b9e5481245d9b5340a454a7cd10b" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:06.337198 11639 raft_consensus.cc:3060] T ff13ae9a47c7455f962406c6d131a3ff P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:04:06.329131 10947 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "836251ebc2b74f5da88b8a8b2338e2a5" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:06.313520 11878 consensus_queue.cc:1048] T 75492f86fde046c3900df3eac8292fd4 P 873357742d5843579556ee8166f23f47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.027s
I20260709 13:04:06.350220 11066 leader_election.cc:304] T b67eff0634e540eab91cf8cee795969f P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda
I20260709 13:04:06.307756 11804 consensus_queue.cc:1048] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 873357742d5843579556ee8166f23f47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.031s
W20260709 13:04:06.441694 11066 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 91617.3us
I20260709 13:04:06.348817 11851 raft_consensus.cc:2749] T bb7d67be500947b3aef4c4245e33671c P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:04:06.436043 10878 raft_consensus.cc:2468] T d270b9e5481245d9b5340a454a7cd10b P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 3.
I20260709 13:04:06.451648 10878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a1716c0adf34bee8a4e99341ba5788f" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:06.356803 11885 raft_consensus.cc:2804] T a64b7e99d6c54934a7de6f6dffcf5e94 P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:04:06.354125 11583 raft_consensus.cc:2804] T 9a1716c0adf34bee8a4e99341ba5788f P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:04:06.353008 11853 raft_consensus.cc:2749] T d7ce47b9a92d43dd88d95bcda5038597 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:04:06.446152 11066 leader_election.cc:304] T 6f46a5102fd64b45bf1c4dff863d2b4d P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47
I20260709 13:04:06.461170 11066 leader_election.cc:304] T 525bfdbd6d934afd823f7d3ee3f9d052 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47
I20260709 13:04:06.445170 11754 raft_consensus.cc:493] T 4f9eb06ed01f4ce68be3af8a8434665b P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 186b12638a87410487467df721a3cbda)
I20260709 13:04:06.452575 10878 raft_consensus.cc:2410] T 9a1716c0adf34bee8a4e99341ba5788f P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0172fed22b6b4890a7fdc953fa128af4 for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 13:04:06.464843 11885 raft_consensus.cc:697] T a64b7e99d6c54934a7de6f6dffcf5e94 P 873357742d5843579556ee8166f23f47 [term 4 LEADER]: Becoming Leader. State: Replica: 873357742d5843579556ee8166f23f47, State: Running, Role: LEADER
I20260709 13:04:06.470427 11885 consensus_queue.cc:237] T a64b7e99d6c54934a7de6f6dffcf5e94 P 873357742d5843579556ee8166f23f47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
W20260709 13:04:06.471473 11910 log.cc:927] Time spent T e96e80ba9ac34d0998395515e5c8c41a P 10ca5ee68ad64b3283b9d235352c9d23: Append to log took a long time: real 0.125s	user 0.000s	sys 0.000s
I20260709 13:04:06.472888 11842 leader_election.cc:290] T c51782cd9d064ec0949e84e24dd4de67 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:06.473367 10950 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5a5f5a608504650b48fd27f163faed6" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:04:06.473511 11842 raft_consensus.cc:2749] T 525bfdbd6d934afd823f7d3ee3f9d052 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:04:06.442993 11508 raft_consensus.cc:493] T 4c2a35bb99374e93b573cf870d04faaa P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:06.443773 10874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7757ecbf402e47bfbc93d0447741e26c" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:04:06.449146 11628 raft_consensus.cc:3060] T b67eff0634e540eab91cf8cee795969f P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:04:06.332192 11817 consensus_queue.cc:237] T b67eff0634e540eab91cf8cee795969f P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:06.448287 11094 raft_consensus.cc:2468] T 8a69f32c53f24a50b7852d143665d7b4 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 3.
I20260709 13:04:06.458267 11583 raft_consensus.cc:493] T 9a1716c0adf34bee8a4e99341ba5788f P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:06.474177 11099 raft_consensus.cc:1275] T f371d2565dd84667b5613848604949bb P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Refusing update from remote peer 186b12638a87410487467df721a3cbda: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:04:06.466089 11859 raft_consensus.cc:493] T a5299b8d044f41b7b68057e63d17b159 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:06.466476 11849 raft_consensus.cc:2749] T 6f46a5102fd64b45bf1c4dff863d2b4d P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 13:04:06.503872 10992 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 72570.9us
I20260709 13:04:06.459955 11845 consensus_queue.cc:1048] T 75492f86fde046c3900df3eac8292fd4 P 873357742d5843579556ee8166f23f47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.037s
I20260709 13:04:06.520051 11658 raft_consensus.cc:493] T d371c5e27bd843469d2e2e453734cb4a P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:06.520805 11658 raft_consensus.cc:515] T d371c5e27bd843469d2e2e453734cb4a P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:06.522508 10874 raft_consensus.cc:2393] T 7757ecbf402e47bfbc93d0447741e26c P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 0172fed22b6b4890a7fdc953fa128af4 in current term 2: Already voted for candidate 186b12638a87410487467df721a3cbda in this term.
I20260709 13:04:06.523540 11723 leader_election.cc:290] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 3 election: Requested vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:06.524080 10874 raft_consensus.cc:1275] T a610d1a71f52432e88fe56130f921fa2 P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Refusing update from remote peer 0172fed22b6b4890a7fdc953fa128af4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20260709 13:04:06.529043 11020 raft_consensus.cc:2468] T 7757ecbf402e47bfbc93d0447741e26c P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 2.
I20260709 13:04:06.503479 11797 leader_election.cc:290] T 11de93cd9b9d457489ff1cb80f976c9a P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 4 election: Requested vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:06.488445 11508 raft_consensus.cc:515] T 4c2a35bb99374e93b573cf870d04faaa P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:06.531744 11508 leader_election.cc:290] T 4c2a35bb99374e93b573cf870d04faaa P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:06.514206 11518 raft_consensus.cc:493] T 5d439326f23747eaab3c1e7da2d6b785 P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:06.532454 11518 raft_consensus.cc:515] T 5d439326f23747eaab3c1e7da2d6b785 P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:06.506137 11583 raft_consensus.cc:3060] T 9a1716c0adf34bee8a4e99341ba5788f P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:04:06.533747 11658 leader_election.cc:290] T d371c5e27bd843469d2e2e453734cb4a P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:06.534900 10874 raft_consensus.cc:1275] T 868e92339c2f4508af0afbd07c814d64 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Refusing update from remote peer 0172fed22b6b4890a7fdc953fa128af4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:04:06.537773 10874 raft_consensus.cc:1275] T 72b111ad7b5f4753ab1d2b5ee1114840 P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Refusing update from remote peer 10ca5ee68ad64b3283b9d235352c9d23: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:04:06.538774 11867 raft_consensus.cc:493] T 30f628e3d7914d609e7f1d980a74c654 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:06.539361 11867 raft_consensus.cc:515] T 30f628e3d7914d609e7f1d980a74c654 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:06.539788 10879 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51718 (request call id 187) took 2995 ms. Trace:
I20260709 13:04:06.478157 11754 raft_consensus.cc:515] T 4f9eb06ed01f4ce68be3af8a8434665b P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:06.540222 10879 rpcz_store.cc:276] 0709 13:04:03.573691 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:04.795762 (+1222071us) service_pool.cc:224] Handling call
0709 13:04:04.924238 (+128476us) spinlock_profiling.cc:243] Waited 10.9 ms on lock 0x7b5800000098. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f426303db71 00007f426303da22 00007f426bf56c09 00007f426b0b0ee0 00007f426b17a0ab 00007f426b17a131 00007f426b13bfc0 00007f426b126929 00007f426b1263d2 00007f426469dbd4
0709 13:04:05.634232 (+709994us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:05.670310 (+ 36078us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:06.539584 (+869274us) spinlock_profiling.cc:243] Waited 710 ms on lock 0x7b54003219d0. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661618ad 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:04:06.539727 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1657368576}
I20260709 13:04:06.551000 10879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e89e640169e14e2d98f968f76027daa7" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:04:06.552668 10879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a9353d1d230416296627d7cf8b1d425" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:06.553470 10879 raft_consensus.cc:2468] T 6a9353d1d230416296627d7cf8b1d425 P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 3.
I20260709 13:04:06.563971 10879 raft_consensus.cc:1240] T 0e5d327630e5438b878035d28efe6c0c P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Rejecting Update request from peer 0172fed22b6b4890a7fdc953fa128af4 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260709 13:04:06.481802 11688 consensus_queue.cc:260] T c31881534c5e4f0089c82057c1746652 P 186b12638a87410487467df721a3cbda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:06.515437 11859 raft_consensus.cc:515] T a5299b8d044f41b7b68057e63d17b159 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
W20260709 13:04:06.610466 11913 log.cc:927] Time spent T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 10ca5ee68ad64b3283b9d235352c9d23: Append to log took a long time: real 0.172s	user 0.001s	sys 0.000s
W20260709 13:04:06.645396 10994 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 296811us
I20260709 13:04:06.557533 11658 raft_consensus.cc:493] T e96e80ba9ac34d0998395515e5c8c41a P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:06.606881 11849 raft_consensus.cc:493] T 5d439326f23747eaab3c1e7da2d6b785 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:06.674299 11067 leader_election.cc:304] T d71ca3a1ba6b48478aaf62741bf3cf6a P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47; no voters: 
W20260709 13:04:06.656380 11066 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 195325us
I20260709 13:04:06.657691 11152 tablet_bootstrap.cc:492] T a1ec2b50c6ad4f7c8033476208abb26d P 0172fed22b6b4890a7fdc953fa128af4: No bootstrap required, opened a new log
I20260709 13:04:06.665475 11880 raft_consensus.cc:493] T 2567c2d84e4648d7ba5dc23a3df32817 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 10ca5ee68ad64b3283b9d235352c9d23)
W20260709 13:04:06.680223 10919 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:45807, user_credentials={real_user=slave}} blocked reactor thread for 139647us
I20260709 13:04:06.621866 10946 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d270b9e5481245d9b5340a454a7cd10b" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:06.681898 10946 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5de5f0581e3b41dea729363d6ca563bb" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:06.682756 10946 raft_consensus.cc:2468] T 5de5f0581e3b41dea729363d6ca563bb P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 3.
I20260709 13:04:06.683946 10946 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11de93cd9b9d457489ff1cb80f976c9a" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:04:06.670898 11506 leader_election.cc:290] T f371d2565dd84667b5613848604949bb P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:06.679989 11880 raft_consensus.cc:515] T 2567c2d84e4648d7ba5dc23a3df32817 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:06.689241 11880 leader_election.cc:290] T 2567c2d84e4648d7ba5dc23a3df32817 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:06.661796 11149 raft_consensus.cc:359] T a1ec2b50c6ad4f7c8033476208abb26d P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:06.690120 11149 raft_consensus.cc:385] T a1ec2b50c6ad4f7c8033476208abb26d P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:04:06.690510 11149 raft_consensus.cc:740] T a1ec2b50c6ad4f7c8033476208abb26d P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 873357742d5843579556ee8166f23f47, State: Initialized, Role: FOLLOWER
I20260709 13:04:06.691411 11149 consensus_queue.cc:260] T a1ec2b50c6ad4f7c8033476208abb26d P 873357742d5843579556ee8166f23f47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:06.695065 11152 ts_tablet_manager.cc:1403] T a1ec2b50c6ad4f7c8033476208abb26d P 0172fed22b6b4890a7fdc953fa128af4: Time spent bootstrapping tablet: real 0.779s	user 0.026s	sys 0.008s
W20260709 13:04:06.623761 10845 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:06.722563 11066 leader_election.cc:304] T 75492f86fde046c3900df3eac8292fd4 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47
I20260709 13:04:06.726756 11066 leader_election.cc:304] T bac7e5040f984ba2b7146bd36af7267e P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47
I20260709 13:04:06.673308 11688 raft_consensus.cc:3060] T c31881534c5e4f0089c82057c1746652 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:04:06.671684 11595 consensus_queue.cc:1048] T e5ba72ccd854430baadbc636dfd17136 P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.048s
I20260709 13:04:06.753135 11149 ts_tablet_manager.cc:1434] T a1ec2b50c6ad4f7c8033476208abb26d P 873357742d5843579556ee8166f23f47: Time spent starting tablet: real 0.544s	user 0.005s	sys 0.006s
W20260709 13:04:06.753720 11149 ts_tablet_manager.cc:1467] T a1ec2b50c6ad4f7c8033476208abb26d P 873357742d5843579556ee8166f23f47: Tablet startup took 544ms
I20260709 13:04:06.720647 11795 raft_consensus.cc:493] T 30f628e3d7914d609e7f1d980a74c654 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:06.724985 11754 leader_election.cc:290] T 4f9eb06ed01f4ce68be3af8a8434665b P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:06.779909 11754 raft_consensus.cc:2749] T bac7e5040f984ba2b7146bd36af7267e P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:04:06.721249 11817 raft_consensus.cc:493] T 6f46a5102fd64b45bf1c4dff863d2b4d P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:06.780839 11817 raft_consensus.cc:515] T 6f46a5102fd64b45bf1c4dff863d2b4d P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:06.783473 11817 leader_election.cc:290] T 6f46a5102fd64b45bf1c4dff863d2b4d P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:06.783526 11871 consensus_queue.cc:1059] T 732c4adeeff6422497aa7db4a82f535b P 873357742d5843579556ee8166f23f47 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:04:06.785120 11871 consensus_queue.cc:1059] T 7709387f4d3d4f9992d97a0fbca39a40 P 873357742d5843579556ee8166f23f47 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:04:06.707571 11658 raft_consensus.cc:515] T e96e80ba9ac34d0998395515e5c8c41a P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:06.787942 11658 leader_election.cc:290] T e96e80ba9ac34d0998395515e5c8c41a P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:06.789945 10775 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:59598 (request call id 65) took 7578 ms. Trace:
W20260709 13:04:06.754086 11149 ts_tablet_manager.cc:1469] T a1ec2b50c6ad4f7c8033476208abb26d P 873357742d5843579556ee8166f23f47: Trace:
0709 13:03:48.668669 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:48.668859 (+   190us) service_pool.cc:224] Handling call
0709 13:03:48.670841 (+  1982us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 13:03:48.670944 (+   103us) ts_tablet_manager.cc:682] Creating new metadata...
0709 13:03:48.724424 (+ 53480us) tablet_metadata.cc:665] Metadata flushed
0709 13:03:48.786235 (+ 61811us) tablet_replica.cc:206] Creating consensus instance
0709 13:03:48.811791 (+ 25556us) inbound_call.cc:177] Queueing success response
0709 13:04:05.665214 (+16853423us) ts_tablet_manager.cc:1379] Bootstrapping tablet
0709 13:04:06.209224 (+544010us) ts_tablet_manager.cc:1435] Starting tablet replica
0709 13:04:06.531466 (+322242us) tablet_replica.cc:261] Starting instrumentation
0709 13:04:06.532111 (+   645us) tablet_replica.cc:280] Starting consensus
Metrics: {"mutex_wait_us":91331,"tablet-open.queue_time_us":16850069}
I20260709 13:04:06.723943 11842 raft_consensus.cc:2804] T d71ca3a1ba6b48478aaf62741bf3cf6a P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:04:06.719234 11849 raft_consensus.cc:515] T 5d439326f23747eaab3c1e7da2d6b785 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:06.748262 11025 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f9eb06ed01f4ce68be3af8a8434665b" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:06.794296 10955 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49434 (request call id 166) took 1184 ms. Trace:
I20260709 13:04:06.816282 10955 rpcz_store.cc:276] 0709 13:04:05.610409 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:05.882176 (+271767us) service_pool.cc:224] Handling call
0709 13:04:05.980888 (+ 98712us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:05.981323 (+   435us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:05.981355 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:05.981962 (+   607us) log.cc:844] Serialized 26 byte log entry
0709 13:04:06.034413 (+ 52451us) raft_consensus.cc:1645] Marking committed up to 1
0709 13:04:06.034476 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:06.034559 (+    83us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:06.114360 (+ 79801us) raft_consensus.cc:1685] finished
0709 13:04:06.114383 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:06.794028 (+679645us) spinlock_profiling.cc:243] Waited 88.9 ms on lock 0x7b54004925d0. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661618ad 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:04:06.794223 (+   195us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1262,"raft.run_cpu_time_us":159,"raft.run_wall_time_us":4549,"spinlock_wait_cycles":204460288,"thread_start_us":52043,"threads_started":1,"wal-append.queue_time_us":130678}
I20260709 13:04:06.803126 11101 raft_consensus.cc:1275] T ff13ae9a47c7455f962406c6d131a3ff P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Refusing update from remote peer 873357742d5843579556ee8166f23f47: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260709 13:04:06.820394 11867 leader_election.cc:290] T 30f628e3d7914d609e7f1d980a74c654 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:06.823966 11926 raft_consensus.cc:493] T 6a9353d1d230416296627d7cf8b1d425 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:06.824694 11926 raft_consensus.cc:515] T 6a9353d1d230416296627d7cf8b1d425 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:06.830156 10874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a1716c0adf34bee8a4e99341ba5788f" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:06.830629 11859 leader_election.cc:290] T a5299b8d044f41b7b68057e63d17b159 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:06.831010 10874 raft_consensus.cc:2468] T 9a1716c0adf34bee8a4e99341ba5788f P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 3.
I20260709 13:04:06.832463 10874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f371d2565dd84667b5613848604949bb" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:06.804153 11754 raft_consensus.cc:493] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:06.836776 11754 raft_consensus.cc:515] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:06.838618 11842 raft_consensus.cc:697] T d71ca3a1ba6b48478aaf62741bf3cf6a P 0172fed22b6b4890a7fdc953fa128af4 [term 2 LEADER]: Becoming Leader. State: Replica: 0172fed22b6b4890a7fdc953fa128af4, State: Running, Role: LEADER
I20260709 13:04:06.839118 11754 leader_election.cc:290] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:06.839859 11842 consensus_queue.cc:237] T d71ca3a1ba6b48478aaf62741bf3cf6a P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:06.742702 11851 consensus_queue.cc:1048] T 5de5f0581e3b41dea729363d6ca563bb P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.100s
I20260709 13:04:06.803653 11853 raft_consensus.cc:2749] T 75492f86fde046c3900df3eac8292fd4 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:04:06.803332 11795 raft_consensus.cc:515] T 30f628e3d7914d609e7f1d980a74c654 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:06.846776 10955 raft_consensus.cc:1240] T e89e640169e14e2d98f968f76027daa7 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Rejecting Update request from peer 186b12638a87410487467df721a3cbda for earlier term 3. Current term is 4. Ops: []
I20260709 13:04:06.848587 10955 raft_consensus.cc:1275] T 4b37f8fca8dc42718b2a42f0883a956f P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Refusing update from remote peer 873357742d5843579556ee8166f23f47: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260709 13:04:06.851016 11873 consensus_meta.cc:287] T a977842ba4b744f4bb2febd55d286006 P 873357742d5843579556ee8166f23f47: Time spent flushing consensus metadata: real 0.601s	user 0.012s	sys 0.000s
I20260709 13:04:06.851660 11873 raft_consensus.cc:515] T a977842ba4b744f4bb2febd55d286006 P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:06.856762 10872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11de93cd9b9d457489ff1cb80f976c9a" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:06.857426 11849 leader_election.cc:290] T 5d439326f23747eaab3c1e7da2d6b785 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:06.801072 11069 leader_election.cc:304] T 401a9cf11eac479aa445ec8c1d6e33b2 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4; no voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47
I20260709 13:04:06.812530 11097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f371d2565dd84667b5613848604949bb" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:06.814046 11149 tablet_bootstrap.cc:492] T 30f628e3d7914d609e7f1d980a74c654 P 873357742d5843579556ee8166f23f47: Bootstrap starting.
W20260709 13:04:06.814738 10948 consensus_meta.cc:287] T 34064dc7bf3747dbb3876dd824ee0688 P 10ca5ee68ad64b3283b9d235352c9d23: Time spent flushing consensus metadata: real 0.751s	user 0.007s	sys 0.017s
I20260709 13:04:06.857718 10872 raft_consensus.cc:2393] T 11de93cd9b9d457489ff1cb80f976c9a P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 10ca5ee68ad64b3283b9d235352c9d23 in current term 4: Already voted for candidate 186b12638a87410487467df721a3cbda in this term.
W20260709 13:04:06.862040 11069 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:38967, user_credentials={real_user=slave}} blocked reactor thread for 61262.6us
I20260709 13:04:06.863020 11069 leader_election.cc:304] T d270b9e5481245d9b5340a454a7cd10b P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda; no voters: 
W20260709 13:04:06.776839 11916 log.cc:927] Time spent T 836251ebc2b74f5da88b8a8b2338e2a5 P 186b12638a87410487467df721a3cbda: Append to log took a long time: real 0.167s	user 0.001s	sys 0.000s
I20260709 13:04:06.867579 11101 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d270b9e5481245d9b5340a454a7cd10b" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:06.868300 11101 raft_consensus.cc:2468] T d270b9e5481245d9b5340a454a7cd10b P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 3.
W20260709 13:04:06.787957 11906 log.cc:927] Time spent T e5a5f5a608504650b48fd27f163faed6 P 186b12638a87410487467df721a3cbda: Append to log took a long time: real 0.448s	user 0.003s	sys 0.000s
I20260709 13:04:06.868709 10955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c2a35bb99374e93b573cf870d04faaa" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
W20260709 13:04:06.816442 11803 consensus_meta.cc:287] T d371c5e27bd843469d2e2e453734cb4a P 0172fed22b6b4890a7fdc953fa128af4: Time spent flushing consensus metadata: real 0.548s	user 0.014s	sys 0.000s
I20260709 13:04:06.875999 11026 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4cd3d7dfbbf4096908ff11e9dc57141" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
I20260709 13:04:06.876293 11852 raft_consensus.cc:493] T d371c5e27bd843469d2e2e453734cb4a P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:06.876840 11026 raft_consensus.cc:2376] T b4cd3d7dfbbf4096908ff11e9dc57141 P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 3. Re-sending same reply.
I20260709 13:04:06.877022 11852 raft_consensus.cc:515] T d371c5e27bd843469d2e2e453734cb4a P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:06.888410 10875 raft_consensus.cc:1240] T 34064dc7bf3747dbb3876dd824ee0688 P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Rejecting Update request from peer 873357742d5843579556ee8166f23f47 for earlier term 3. Current term is 4. Ops: []
I20260709 13:04:06.888432 11933 raft_consensus.cc:493] T 65bf0a62f4d84ef18dd6b23e082724b8 P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 10ca5ee68ad64b3283b9d235352c9d23)
I20260709 13:04:06.890851 11097 raft_consensus.cc:2468] T f371d2565dd84667b5613848604949bb P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 2.
I20260709 13:04:06.888461 11849 raft_consensus.cc:2804] T d270b9e5481245d9b5340a454a7cd10b P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:04:06.893549 11849 raft_consensus.cc:493] T d270b9e5481245d9b5340a454a7cd10b P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:06.893951 11849 raft_consensus.cc:3060] T d270b9e5481245d9b5340a454a7cd10b P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:04:06.894814 11846 raft_consensus.cc:493] T 1e4f2241a94a41f38e481137f868b83d P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 186b12638a87410487467df721a3cbda)
I20260709 13:04:06.895588 11846 raft_consensus.cc:515] T 1e4f2241a94a41f38e481137f868b83d P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:06.890286 11933 raft_consensus.cc:515] T 65bf0a62f4d84ef18dd6b23e082724b8 P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:06.900444 11933 leader_election.cc:290] T 65bf0a62f4d84ef18dd6b23e082724b8 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:06.873521 11853 raft_consensus.cc:2749] T 401a9cf11eac479aa445ec8c1d6e33b2 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:04:06.868181 10948 raft_consensus.cc:2468] T 34064dc7bf3747dbb3876dd824ee0688 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 4.
I20260709 13:04:06.903805 10872 raft_consensus.cc:1275] T 4c2a35bb99374e93b573cf870d04faaa P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Refusing update from remote peer 873357742d5843579556ee8166f23f47: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20260709 13:04:06.904273 10951 raft_consensus.cc:1240] T 34064dc7bf3747dbb3876dd824ee0688 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Rejecting Update request from peer 873357742d5843579556ee8166f23f47 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260709 13:04:06.906661 11795 leader_election.cc:290] T 30f628e3d7914d609e7f1d980a74c654 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:06.907946 10948 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "30f628e3d7914d609e7f1d980a74c654" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:06.908052 11583 raft_consensus.cc:515] T 9a1716c0adf34bee8a4e99341ba5788f P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:06.908951 10948 raft_consensus.cc:2468] T 30f628e3d7914d609e7f1d980a74c654 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 0.
I20260709 13:04:06.910900 11024 raft_consensus.cc:2393] T 9a1716c0adf34bee8a4e99341ba5788f P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0172fed22b6b4890a7fdc953fa128af4 in current term 4: Already voted for candidate 873357742d5843579556ee8166f23f47 in this term.
I20260709 13:04:06.916265 10950 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d71ca3a1ba6b48478aaf62741bf3cf6a" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:04:06.904402 10872 pending_rounds.cc:85] T 4c2a35bb99374e93b573cf870d04faaa P 186b12638a87410487467df721a3cbda: Aborting all ops after (but not including) 1
I20260709 13:04:06.790416 10775 rpcz_store.cc:276] 0709 13:03:59.211747 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:59.212142 (+   395us) service_pool.cc:224] Handling call
0709 13:04:06.789853 (+7577711us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 13:04:04.594976 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 13:04:04.650836 (+ 55860us) write_op.cc:432] Acquiring schema lock in shared mode
0709 13:04:04.650883 (+    47us) write_op.cc:435] Acquired schema lock
0709 13:04:04.650914 (+    31us) tablet.cc:662] Decoding operations
0709 13:04:04.651347 (+   433us) write_op.cc:620] Acquiring the partition lock for write op
0709 13:04:04.651417 (+    70us) write_op.cc:641] Partition lock acquired for write op
0709 13:04:04.651446 (+    29us) tablet.cc:685] Acquiring locks for 1 operations
0709 13:04:04.651651 (+   205us) tablet.cc:701] Row locks acquired
0709 13:04:04.651671 (+    20us) write_op.cc:260] PREPARE: finished
0709 13:04:04.651848 (+   177us) write_op.cc:270] Start()
0709 13:04:04.651981 (+   133us) write_op.cc:276] Timestamp: P: 1783602244651801 usec, L: 0
0709 13:04:04.652002 (+    21us) op_driver.cc:348] REPLICATION: starting
0709 13:04:04.652844 (+   842us) log.cc:844] Serialized 540 byte log entry
0709 13:04:05.051119 (+398275us) op_driver.cc:464] REPLICATION: finished
0709 13:04:05.416937 (+365818us) write_op.cc:301] APPLY: starting
0709 13:04:05.424316 (+  7379us) tablet.cc:1367] starting BulkCheckPresence
0709 13:04:05.446079 (+ 21763us) tablet.cc:1370] finished BulkCheckPresence
0709 13:04:05.480120 (+ 34041us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 13:04:05.513088 (+ 32968us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 13:04:05.588914 (+ 75826us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0709 13:04:05.595272 (+  6358us) write_op.cc:312] APPLY: finished
0709 13:04:05.604998 (+  9726us) log.cc:844] Serialized 22 byte log entry
0709 13:04:05.619200 (+ 14202us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:04:05.624120 (+  4920us) write_op.cc:454] Released schema lock
0709 13:04:05.626151 (+  2031us) write_op.cc:341] FINISH: Updating metrics
0709 13:04:05.659529 (+ 33378us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:04:05.665420 (+  5891us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":362757,"reported_tablets":10,"child_traces":[["op",{"apply.queue_time_us":323032,"apply.run_cpu_time_us":4288,"apply.run_wall_time_us":240881,"num_ops":1,"prepare.queue_time_us":232310,"prepare.run_cpu_time_us":3329,"prepare.run_wall_time_us":71848,"replication_time_us":398902,"thread_start_us":322683,"threads_started":2}]]}
I20260709 13:04:06.838251 11628 raft_consensus.cc:2749] T b67eff0634e540eab91cf8cee795969f P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:04:06.876166 10955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e96e80ba9ac34d0998395515e5c8c41a" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:06.926666 10872 pending_rounds.cc:107] T 4c2a35bb99374e93b573cf870d04faaa P 186b12638a87410487467df721a3cbda: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20260709 13:04:06.898440 11846 leader_election.cc:290] T 1e4f2241a94a41f38e481137f868b83d P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
W20260709 13:04:06.952769 10826 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 13:04:06.957449 11102 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c160ddfe9b34bdaa3835d0915c1e06f" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:04:06.959452 10953 raft_consensus.cc:1275] T ff13ae9a47c7455f962406c6d131a3ff P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Refusing update from remote peer 873357742d5843579556ee8166f23f47: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 1. (term mismatch)
I20260709 13:04:06.962566 10953 pending_rounds.cc:85] T ff13ae9a47c7455f962406c6d131a3ff P 10ca5ee68ad64b3283b9d235352c9d23: Aborting all ops after (but not including) 0
I20260709 13:04:06.962656 11102 raft_consensus.cc:3060] T 8c160ddfe9b34bdaa3835d0915c1e06f P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:04:06.962998 10953 pending_rounds.cc:107] T ff13ae9a47c7455f962406c6d131a3ff P 10ca5ee68ad64b3283b9d235352c9d23: Aborting uncommitted NO_OP operation due to leader change: 1.1
I20260709 13:04:06.963418 10953 raft_consensus.cc:2889] T ff13ae9a47c7455f962406c6d131a3ff P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:04:06.941800 11152 raft_consensus.cc:359] T a1ec2b50c6ad4f7c8033476208abb26d P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:06.964318 11152 raft_consensus.cc:385] T a1ec2b50c6ad4f7c8033476208abb26d P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:04:06.964687 11152 raft_consensus.cc:740] T a1ec2b50c6ad4f7c8033476208abb26d P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0172fed22b6b4890a7fdc953fa128af4, State: Initialized, Role: FOLLOWER
I20260709 13:04:06.972198 11103 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5de5f0581e3b41dea729363d6ca563bb" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:06.973712 11103 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555217d45d9b4ee492dbb3ddaf0f7c02" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:04:06.973984 11096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d439326f23747eaab3c1e7da2d6b785" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:06.974961 11096 raft_consensus.cc:2468] T 5d439326f23747eaab3c1e7da2d6b785 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 3.
I20260709 13:04:06.975708 11103 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d371c5e27bd843469d2e2e453734cb4a" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:06.976617 11103 raft_consensus.cc:2410] T d371c5e27bd843469d2e2e453734cb4a P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 186b12638a87410487467df721a3cbda for term 3 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
W20260709 13:04:06.926280 10845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 43878.3us
I20260709 13:04:06.995841 11101 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e5d327630e5438b878035d28efe6c0c" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:04:06.967514 11152 consensus_queue.cc:260] T a1ec2b50c6ad4f7c8033476208abb26d P 0172fed22b6b4890a7fdc953fa128af4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:06.944039 11583 leader_election.cc:290] T 9a1716c0adf34bee8a4e99341ba5788f P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 4 election: Requested vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:06.960584 10872 raft_consensus.cc:2889] T 4c2a35bb99374e93b573cf870d04faaa P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 13:04:07.020808 10920 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.107.196:38507 timed out after 1.531s (SENT) [suppressed 26 similar messages]
I20260709 13:04:06.994515 10950 raft_consensus.cc:2393] T d71ca3a1ba6b48478aaf62741bf3cf6a P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 0172fed22b6b4890a7fdc953fa128af4 in current term 2: Already voted for candidate 10ca5ee68ad64b3283b9d235352c9d23 in this term.
I20260709 13:04:07.006928 11639 raft_consensus.cc:493] T c51782cd9d064ec0949e84e24dd4de67 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 186b12638a87410487467df721a3cbda)
I20260709 13:04:07.011859 11094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a69f32c53f24a50b7852d143665d7b4" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
W20260709 13:04:07.033524 11066 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 306923us
W20260709 13:04:07.006999 10995 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:38967, user_credentials={real_user=slave}} blocked reactor thread for 330188us
I20260709 13:04:07.043720 11066 leader_election.cc:304] T a154cc8b181f460abf4880064379948b P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47
I20260709 13:04:07.049341 11103 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72b111ad7b5f4753ab1d2b5ee1114840" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:04:07.049762 11102 raft_consensus.cc:2468] T 8c160ddfe9b34bdaa3835d0915c1e06f P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 2.
W20260709 13:04:07.032562 11883 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.695s	user 0.007s	sys 0.006s
I20260709 13:04:07.031263 11097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7709387f4d3d4f9992d97a0fbca39a40" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:04:07.030130 11873 leader_election.cc:290] T a977842ba4b744f4bb2febd55d286006 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 1 election: Requested vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:07.065672 11102 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b67eff0634e540eab91cf8cee795969f" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:07.066325 11101 raft_consensus.cc:1240] T e5a5f5a608504650b48fd27f163faed6 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Rejecting Update request from peer 10ca5ee68ad64b3283b9d235352c9d23 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260709 13:04:07.066560 11102 raft_consensus.cc:2468] T b67eff0634e540eab91cf8cee795969f P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 1.
I20260709 13:04:07.035303 11098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5a5f5a608504650b48fd27f163faed6" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:07.047006 11780 raft_consensus.cc:3055] T 7709387f4d3d4f9992d97a0fbca39a40 P 873357742d5843579556ee8166f23f47 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:04:07.074287 10995 leader_election.cc:304] T 6a9353d1d230416296627d7cf8b1d425 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47; no voters: 0172fed22b6b4890a7fdc953fa128af4
I20260709 13:04:07.046773 11094 raft_consensus.cc:3060] T 8a69f32c53f24a50b7852d143665d7b4 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:04:07.047484 10877 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51718 (request call id 188) took 3458 ms. Trace:
I20260709 13:04:07.077813 11102 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b67eff0634e540eab91cf8cee795969f" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:04:07.088626 11101 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a64b7e99d6c54934a7de6f6dffcf5e94" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:07.098018 11152 ts_tablet_manager.cc:1434] T a1ec2b50c6ad4f7c8033476208abb26d P 0172fed22b6b4890a7fdc953fa128af4: Time spent starting tablet: real 0.400s	user 0.006s	sys 0.006s
W20260709 13:04:07.056010 11883 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.722s	user 0.008s	sys 0.006s
I20260709 13:04:07.058178 11853 raft_consensus.cc:2749] T a154cc8b181f460abf4880064379948b P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:04:07.086280 11067 leader_election.cc:304] T 7757ecbf402e47bfbc93d0447741e26c P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47; no voters: 
W20260709 13:04:07.091702 10920 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 70948.1us
I20260709 13:04:07.093425 11103 raft_consensus.cc:2468] T 72b111ad7b5f4753ab1d2b5ee1114840 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 3.
I20260709 13:04:07.095469 10877 rpcz_store.cc:276] 0709 13:04:03.666231 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:04.815830 (+1149599us) service_pool.cc:224] Handling call
0709 13:04:04.895289 (+ 79459us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:05.685191 (+789902us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:07.029054 (+1343863us) spinlock_profiling.cc:243] Waited 726 ms on lock 0x7b54001d83d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:04:07.036365 (+  7311us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1669344512}
I20260709 13:04:07.079169 11780 raft_consensus.cc:740] T 7709387f4d3d4f9992d97a0fbca39a40 P 873357742d5843579556ee8166f23f47 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 873357742d5843579556ee8166f23f47, State: Running, Role: LEADER
W20260709 13:04:07.176163 11067 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:45807, user_credentials={real_user=slave}} blocked reactor thread for 90034.8us
I20260709 13:04:07.135594 11098 raft_consensus.cc:2393] T e5a5f5a608504650b48fd27f163faed6 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 186b12638a87410487467df721a3cbda in current term 3: Already voted for candidate 0172fed22b6b4890a7fdc953fa128af4 in this term.
I20260709 13:04:07.076449 11926 leader_election.cc:290] T 6a9353d1d230416296627d7cf8b1d425 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:07.177949 11098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f9df2bc281f4866b6da5da94a0defc7" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:07.177608 11780 consensus_queue.cc:260] T 7709387f4d3d4f9992d97a0fbca39a40 P 873357742d5843579556ee8166f23f47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:07.179389 11780 raft_consensus.cc:3060] T 7709387f4d3d4f9992d97a0fbca39a40 P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:04:07.179703 11098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4cd3d7dfbbf4096908ff11e9dc57141" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:04:07.180755 11098 raft_consensus.cc:2393] T b4cd3d7dfbbf4096908ff11e9dc57141 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 10ca5ee68ad64b3283b9d235352c9d23 in current term 3: Already voted for candidate 0172fed22b6b4890a7fdc953fa128af4 in this term.
I20260709 13:04:07.181749 11098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5de5f0581e3b41dea729363d6ca563bb" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 4 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:07.135933 11937 raft_consensus.cc:493] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:07.043166 11639 raft_consensus.cc:515] T c51782cd9d064ec0949e84e24dd4de67 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:07.183324 11937 raft_consensus.cc:515] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:07.185276 11639 leader_election.cc:290] T c51782cd9d064ec0949e84e24dd4de67 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:07.097157 11101 raft_consensus.cc:2393] T a64b7e99d6c54934a7de6f6dffcf5e94 P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 10ca5ee68ad64b3283b9d235352c9d23 in current term 4: Already voted for candidate 873357742d5843579556ee8166f23f47 in this term.
I20260709 13:04:07.100504 11883 raft_consensus.cc:697] T 8a69f32c53f24a50b7852d143665d7b4 P 186b12638a87410487467df721a3cbda [term 4 LEADER]: Becoming Leader. State: Replica: 186b12638a87410487467df721a3cbda, State: Running, Role: LEADER
I20260709 13:04:07.186501 11937 leader_election.cc:290] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:07.187193 11101 raft_consensus.cc:1275] T 836251ebc2b74f5da88b8a8b2338e2a5 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Refusing update from remote peer 10ca5ee68ad64b3283b9d235352c9d23: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 13:04:07.186962 11883 consensus_queue.cc:237] T 8a69f32c53f24a50b7852d143665d7b4 P 186b12638a87410487467df721a3cbda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:07.188398 11067 leader_election.cc:304] T b4cd3d7dfbbf4096908ff11e9dc57141 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47; no voters: 10ca5ee68ad64b3283b9d235352c9d23
I20260709 13:04:07.189971 11020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555217d45d9b4ee492dbb3ddaf0f7c02" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:07.139117 11936 consensus_queue.cc:1048] T 4c2a35bb99374e93b573cf870d04faaa P 873357742d5843579556ee8166f23f47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:04:07.190775 11098 raft_consensus.cc:1275] T e5ba72ccd854430baadbc636dfd17136 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Refusing update from remote peer 10ca5ee68ad64b3283b9d235352c9d23: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:04:07.071139 11097 raft_consensus.cc:3060] T 7709387f4d3d4f9992d97a0fbca39a40 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:04:07.174226 11102 raft_consensus.cc:2468] T b67eff0634e540eab91cf8cee795969f P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 1.
I20260709 13:04:07.195554 11103 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a69f32c53f24a50b7852d143665d7b4" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:07.094097 11873 consensus_queue.cc:1059] T 57b6a2418c1342e7924195e9c1dbd357 P 873357742d5843579556ee8166f23f47 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.104s
I20260709 13:04:07.198640 11152 tablet_bootstrap.cc:492] T f34f2bc02d85444f9096e0f5b4fe4331 P 0172fed22b6b4890a7fdc953fa128af4: Bootstrap starting.
I20260709 13:04:07.198752 11937 raft_consensus.cc:2804] T b4cd3d7dfbbf4096908ff11e9dc57141 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:04:07.140863 11944 raft_consensus.cc:2804] T 6a9353d1d230416296627d7cf8b1d425 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:04:07.313612 11102 raft_consensus.cc:1275] T 5d439326f23747eaab3c1e7da2d6b785 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Refusing update from remote peer 10ca5ee68ad64b3283b9d235352c9d23: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 13:04:07.158710 11920 raft_consensus.cc:493] T a5299b8d044f41b7b68057e63d17b159 P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:07.382221 11920 raft_consensus.cc:515] T a5299b8d044f41b7b68057e63d17b159 P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:07.141597 11885 consensus_queue.cc:1059] T 34064dc7bf3747dbb3876dd824ee0688 P 873357742d5843579556ee8166f23f47 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.210s
I20260709 13:04:07.388005 11937 raft_consensus.cc:697] T b4cd3d7dfbbf4096908ff11e9dc57141 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 LEADER]: Becoming Leader. State: Replica: 0172fed22b6b4890a7fdc953fa128af4, State: Running, Role: LEADER
I20260709 13:04:07.390234 11943 raft_consensus.cc:493] T a1ec2b50c6ad4f7c8033476208abb26d P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:07.391096 11943 raft_consensus.cc:515] T a1ec2b50c6ad4f7c8033476208abb26d P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:07.174973 11845 raft_consensus.cc:493] T 525bfdbd6d934afd823f7d3ee3f9d052 P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 10ca5ee68ad64b3283b9d235352c9d23)
I20260709 13:04:07.393822 11845 raft_consensus.cc:515] T 525bfdbd6d934afd823f7d3ee3f9d052 P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
W20260709 13:04:07.282934 11903 log.cc:927] Time spent T 87cd8f5097624c8692f1d83544d8881e P 10ca5ee68ad64b3283b9d235352c9d23: Append to log took a long time: real 0.079s	user 0.001s	sys 0.000s
I20260709 13:04:07.199700 11853 raft_consensus.cc:2804] T 7757ecbf402e47bfbc93d0447741e26c P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:04:07.399641 11853 raft_consensus.cc:697] T 7757ecbf402e47bfbc93d0447741e26c P 0172fed22b6b4890a7fdc953fa128af4 [term 2 LEADER]: Becoming Leader. State: Replica: 0172fed22b6b4890a7fdc953fa128af4, State: Running, Role: LEADER
I20260709 13:04:07.389465 11937 consensus_queue.cc:237] T b4cd3d7dfbbf4096908ff11e9dc57141 P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:07.400877 11853 consensus_queue.cc:237] T 7757ecbf402e47bfbc93d0447741e26c P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:07.282785 11926 raft_consensus.cc:493] T 32147c6d2d5a477aa38c597c7f78df5b P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:07.403053 11926 raft_consensus.cc:515] T 32147c6d2d5a477aa38c597c7f78df5b P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:07.314841 11779 raft_consensus.cc:493] T 7839ca2d2c13431a85f44d7c458c6351 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:07.405323 11779 raft_consensus.cc:515] T 7839ca2d2c13431a85f44d7c458c6351 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:07.409276 11885 raft_consensus.cc:3055] T 34064dc7bf3747dbb3876dd824ee0688 P 873357742d5843579556ee8166f23f47 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:04:07.409890 11885 raft_consensus.cc:740] T 34064dc7bf3747dbb3876dd824ee0688 P 873357742d5843579556ee8166f23f47 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 873357742d5843579556ee8166f23f47, State: Running, Role: LEADER
I20260709 13:04:07.352329 11944 raft_consensus.cc:493] T 6a9353d1d230416296627d7cf8b1d425 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 186b12638a87410487467df721a3cbda)
I20260709 13:04:07.411794 11944 raft_consensus.cc:3060] T 6a9353d1d230416296627d7cf8b1d425 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:04:07.319409 11103 raft_consensus.cc:1275] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Refusing update from remote peer 10ca5ee68ad64b3283b9d235352c9d23: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260709 13:04:07.162133 11800 raft_consensus.cc:493] T e5ba72ccd854430baadbc636dfd17136 P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:07.354538 11102 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a1716c0adf34bee8a4e99341ba5788f" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:04:07.417891 11102 raft_consensus.cc:3060] T 9a1716c0adf34bee8a4e99341ba5788f P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Advancing to term 4
W20260709 13:04:07.380868 11941 log.cc:927] Time spent T e96e80ba9ac34d0998395515e5c8c41a P 873357742d5843579556ee8166f23f47: Append to log took a long time: real 0.279s	user 0.000s	sys 0.000s
I20260709 13:04:07.415967 11103 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a154cc8b181f460abf4880064379948b" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:07.423341 11103 raft_consensus.cc:2410] T a154cc8b181f460abf4880064379948b P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 10ca5ee68ad64b3283b9d235352c9d23 for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 13:04:07.424788 11103 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a977842ba4b744f4bb2febd55d286006" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:04:07.427852 11920 leader_election.cc:290] T a5299b8d044f41b7b68057e63d17b159 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:07.438964 11103 raft_consensus.cc:3060] T a977842ba4b744f4bb2febd55d286006 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:04:07.487290 11102 raft_consensus.cc:2468] T 9a1716c0adf34bee8a4e99341ba5788f P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 4.
W20260709 13:04:07.467728 11842 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.551s	user 0.004s	sys 0.009s
I20260709 13:04:07.440372 11800 raft_consensus.cc:515] T e5ba72ccd854430baadbc636dfd17136 P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:07.548450 11800 leader_election.cc:290] T e5ba72ccd854430baadbc636dfd17136 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:07.489068 11885 consensus_queue.cc:260] T 34064dc7bf3747dbb3876dd824ee0688 P 873357742d5843579556ee8166f23f47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:07.550118 11885 raft_consensus.cc:3060] T 34064dc7bf3747dbb3876dd824ee0688 P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:04:07.520658 11943 leader_election.cc:290] T a1ec2b50c6ad4f7c8033476208abb26d P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:07.523317 11926 leader_election.cc:290] T 32147c6d2d5a477aa38c597c7f78df5b P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:07.523042 11845 leader_election.cc:290] T 525bfdbd6d934afd823f7d3ee3f9d052 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:07.526396 11779 leader_election.cc:290] T 7839ca2d2c13431a85f44d7c458c6351 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
W20260709 13:04:07.541453 11849 consensus_meta.cc:287] T d270b9e5481245d9b5340a454a7cd10b P 0172fed22b6b4890a7fdc953fa128af4: Time spent flushing consensus metadata: real 0.647s	user 0.009s	sys 0.006s
I20260709 13:04:07.560380 11849 raft_consensus.cc:515] T d270b9e5481245d9b5340a454a7cd10b P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
W20260709 13:04:07.562065 11842 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.645s	user 0.004s	sys 0.009s
W20260709 13:04:07.543239 11688 consensus_meta.cc:287] T c31881534c5e4f0089c82057c1746652 P 186b12638a87410487467df721a3cbda: Time spent flushing consensus metadata: real 0.783s	user 0.013s	sys 0.003s
I20260709 13:04:07.570672 11849 leader_election.cc:290] T d270b9e5481245d9b5340a454a7cd10b P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 4 election: Requested vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:07.134851 10873 raft_consensus.cc:1240] T 11de93cd9b9d457489ff1cb80f976c9a P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Rejecting Update request from peer 873357742d5843579556ee8166f23f47 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
W20260709 13:04:07.141136 10845 leader_election.cc:343] T f34f2bc02d85444f9096e0f5b4fe4331 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:04:07.691771 10845 leader_election.cc:304] T f34f2bc02d85444f9096e0f5b4fe4331 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 186b12638a87410487467df721a3cbda; no voters: 0172fed22b6b4890a7fdc953fa128af4, 10ca5ee68ad64b3283b9d235352c9d23
I20260709 13:04:07.619601 10872 raft_consensus.cc:1240] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Rejecting Update request from peer 873357742d5843579556ee8166f23f47 for earlier term 2. Current term is 3. Ops: []
I20260709 13:04:07.158643 10874 raft_consensus.cc:1240] T 8c160ddfe9b34bdaa3835d0915c1e06f P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Rejecting Update request from peer 873357742d5843579556ee8166f23f47 for earlier term 1. Current term is 2. Ops: []
I20260709 13:04:07.625653 10879 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51704 (request call id 166) took 1332 ms. Trace:
W20260709 13:04:07.627532 10992 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.757s	user 0.074s	sys 0.097s
I20260709 13:04:07.640563 11020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32147c6d2d5a477aa38c597c7f78df5b" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:07.647258 11101 raft_consensus.cc:1240] T d270b9e5481245d9b5340a454a7cd10b P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Rejecting Update request from peer 10ca5ee68ad64b3283b9d235352c9d23 for earlier term 3. Current term is 4. Ops: []
I20260709 13:04:07.649197 11096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c31881534c5e4f0089c82057c1746652" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:04:07.649521 11102 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb7d67be500947b3aef4c4245e33671c" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:07.653213 10946 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c51782cd9d064ec0949e84e24dd4de67" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:07.698474 10946 raft_consensus.cc:2468] T c51782cd9d064ec0949e84e24dd4de67 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 2.
I20260709 13:04:07.698552 10872 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51718 (request call id 213) took 1022 ms. Trace:
I20260709 13:04:07.653777 10950 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d156d6ae2ca4fa59a1e387edf0fea39" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:07.656945 11026 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1ec2b50c6ad4f7c8033476208abb26d" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:07.699523 10874 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51718 (request call id 212) took 1024 ms. Trace:
I20260709 13:04:07.700357 11026 raft_consensus.cc:2468] T a1ec2b50c6ad4f7c8033476208abb26d P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 0.
I20260709 13:04:07.700136 10874 rpcz_store.cc:276] 0709 13:04:06.675491 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:07.041861 (+366370us) service_pool.cc:224] Handling call
0709 13:04:07.076106 (+ 34245us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:07.698859 (+622753us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:07.699283 (+   424us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:07.702466 11102 raft_consensus.cc:1275] T 72b111ad7b5f4753ab1d2b5ee1114840 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Refusing update from remote peer 10ca5ee68ad64b3283b9d235352c9d23: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:04:07.703812 10874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a1716c0adf34bee8a4e99341ba5788f" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:04:07.704794 10874 raft_consensus.cc:3060] T 9a1716c0adf34bee8a4e99341ba5788f P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:04:07.500090 11152 tablet_bootstrap.cc:654] T f34f2bc02d85444f9096e0f5b4fe4331 P 0172fed22b6b4890a7fdc953fa128af4: Neither blocks nor log segments found. Creating new log.
I20260709 13:04:07.492444 11149 tablet_bootstrap.cc:654] T 30f628e3d7914d609e7f1d980a74c654 P 873357742d5843579556ee8166f23f47: Neither blocks nor log segments found. Creating new log.
I20260709 13:04:07.699180 10872 rpcz_store.cc:276] 0709 13:04:06.675922 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:07.070921 (+394999us) service_pool.cc:224] Handling call
0709 13:04:07.157198 (+ 86277us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:07.697955 (+540757us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:07.698471 (+   516us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:07.710479 10872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e4f2241a94a41f38e481137f868b83d" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:07.573884 11944 raft_consensus.cc:515] T 6a9353d1d230416296627d7cf8b1d425 P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:07.665130 11883 raft_consensus.cc:493] T 868e92339c2f4508af0afbd07c814d64 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:07.666396 10951 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d439326f23747eaab3c1e7da2d6b785" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:07.713881 11936 consensus_queue.cc:1048] T ff13ae9a47c7455f962406c6d131a3ff P 873357742d5843579556ee8166f23f47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:04:07.715163 11883 raft_consensus.cc:515] T 868e92339c2f4508af0afbd07c814d64 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:07.719558 11020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11de93cd9b9d457489ff1cb80f976c9a" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
I20260709 13:04:07.666663 11943 raft_consensus.cc:493] T c31881534c5e4f0089c82057c1746652 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:07.676694 11027 raft_consensus.cc:1275] T d71ca3a1ba6b48478aaf62741bf3cf6a P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Refusing update from remote peer 0172fed22b6b4890a7fdc953fa128af4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:04:07.679196 11920 consensus_queue.cc:1059] T 7839ca2d2c13431a85f44d7c458c6351 P 873357742d5843579556ee8166f23f47 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:04:07.681075 11639 raft_consensus.cc:493] T bac7e5040f984ba2b7146bd36af7267e P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:07.682106 11926 raft_consensus.cc:493] T ff13ae9a47c7455f962406c6d131a3ff P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:07.684219 10871 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a69f32c53f24a50b7852d143665d7b4" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:07.668432 11947 raft_consensus.cc:493] T 1e4f2241a94a41f38e481137f868b83d P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:07.688350 11779 consensus_queue.cc:1059] T e5a5f5a608504650b48fd27f163faed6 P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:04:07.692095 10873 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51718 (request call id 209) took 1142 ms. Trace:
I20260709 13:04:07.622747 10877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65bf0a62f4d84ef18dd6b23e082724b8" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
W20260709 13:04:07.696717 10992 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.826s	user 0.076s	sys 0.097s
I20260709 13:04:07.704746 11067 leader_election.cc:304] T a1ec2b50c6ad4f7c8033476208abb26d P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:04:07.670185 11780 raft_consensus.cc:493] T 401a9cf11eac479aa445ec8c1d6e33b2 P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:07.723286 11026 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555217d45d9b4ee492dbb3ddaf0f7c02" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
I20260709 13:04:07.723475 11069 leader_election.cc:304] T 9a1716c0adf34bee8a4e99341ba5788f P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4; no voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47
W20260709 13:04:07.726831 10995 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:07.728797 11020 raft_consensus.cc:1275] T f371d2565dd84667b5613848604949bb P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Refusing update from remote peer 186b12638a87410487467df721a3cbda: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:04:07.729518 10994 leader_election.cc:304] T f371d2565dd84667b5613848604949bb P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47; no voters: 186b12638a87410487467df721a3cbda
I20260709 13:04:07.730616 11098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75492f86fde046c3900df3eac8292fd4" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:07.694948 10879 rpcz_store.cc:276] 0709 13:04:06.292807 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:06.569157 (+276350us) service_pool.cc:224] Handling call
0709 13:04:06.569607 (+   450us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:06.598504 (+ 28897us) raft_consensus.cc:1534] Early marking committed up to index 1
0709 13:04:06.900276 (+301772us) spinlock_profiling.cc:243] Waited 8.29 ms on lock 0x7b180042e6e0. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f4263057e07 00007f426beb2ab9 00007f426307469b 00007f426307293f 00007f4263072455 00007f4269b5a3c7 00007f426616c9a7 00007f426617c0bf 00007f426617c038
0709 13:04:07.042250 (+141974us) log.cc:844] Serialized 14 byte log entry
0709 13:04:07.157150 (+114900us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:07.157764 (+   614us) log.cc:844] Serialized 26 byte log entry
0709 13:04:07.157942 (+   178us) raft_consensus.cc:1645] Marking committed up to 1
0709 13:04:07.157990 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:07.158069 (+    79us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:07.624491 (+466422us) raft_consensus.cc:1685] finished
0709 13:04:07.624776 (+   285us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:07.625328 (+   552us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":88883,"prepare.queue_time_us":80811,"prepare.run_cpu_time_us":80,"prepare.run_wall_time_us":5212,"raft.queue_time_us":30397,"raft.run_cpu_time_us":401,"raft.run_wall_time_us":46342,"spinlock_wait_cycles":19067776,"thread_start_us":251451,"threads_started":2,"wal-append.queue_time_us":331571}
I20260709 13:04:07.700474 10949 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49428 (request call id 105) took 2189 ms. Trace:
I20260709 13:04:07.741816 10949 rpcz_store.cc:276] 0709 13:04:05.510920 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:05.609445 (+ 98525us) service_pool.cc:224] Handling call
0709 13:04:06.114587 (+505142us) spinlock_profiling.cc:243] Waited 2.2 ms on lock 0x7b580001d198. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f426303db71 00007f426303da22 00007f426bf56c09 00007f426b0b0ee0 00007f426b17a0ab 00007f426b17a131 00007f426b1280a0 00007f426b127984 00007f42646a26d4 00007f42646a2662
0709 13:04:07.699908 (+1585321us) spinlock_profiling.cc:243] Waited 529 ms on lock 0x7b54004b6c54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:04:07.700215 (+   307us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1220912256}
I20260709 13:04:07.742383 11639 raft_consensus.cc:515] T bac7e5040f984ba2b7146bd36af7267e P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
W20260709 13:04:07.744199 11029 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.806s	user 0.002s	sys 0.016s
I20260709 13:04:07.747745 11947 raft_consensus.cc:515] T 1e4f2241a94a41f38e481137f868b83d P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:07.748199 10876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57b6a2418c1342e7924195e9c1dbd357" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:04:07.748494 11849 raft_consensus.cc:2804] T a1ec2b50c6ad4f7c8033476208abb26d P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:04:07.750871 11926 raft_consensus.cc:515] T ff13ae9a47c7455f962406c6d131a3ff P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:07.752135 11943 raft_consensus.cc:515] T c31881534c5e4f0089c82057c1746652 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:07.753235 11780 raft_consensus.cc:515] T 401a9cf11eac479aa445ec8c1d6e33b2 P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:07.756428 10877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2567c2d84e4648d7ba5dc23a3df32817" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:07.756866 11944 leader_election.cc:290] T 6a9353d1d230416296627d7cf8b1d425 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 3 election: Requested vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:07.758893 11920 raft_consensus.cc:3055] T 7839ca2d2c13431a85f44d7c458c6351 P 873357742d5843579556ee8166f23f47 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:04:07.787370 11920 raft_consensus.cc:740] T 7839ca2d2c13431a85f44d7c458c6351 P 873357742d5843579556ee8166f23f47 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 873357742d5843579556ee8166f23f47, State: Running, Role: LEADER
I20260709 13:04:07.788570 10872 raft_consensus.cc:2468] T 1e4f2241a94a41f38e481137f868b83d P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 2.
I20260709 13:04:07.789153 11920 consensus_queue.cc:260] T 7839ca2d2c13431a85f44d7c458c6351 P 873357742d5843579556ee8166f23f47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:07.789882 10954 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49434 (request call id 176) took 1668 ms. Trace:
I20260709 13:04:07.790463 10954 rpcz_store.cc:276] 0709 13:04:06.120934 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:06.122463 (+  1529us) service_pool.cc:224] Handling call
0709 13:04:07.752603 (+1630140us) spinlock_profiling.cc:243] Waited 30.3 ms on lock 0x7b580001d198. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f426303db71 00007f426303da22 00007f426bf56c09 00007f426b0b0ee0 00007f426b17a0ab 00007f426b17a131 00007f426b13bfc0 00007f426b126929 00007f426b1263d2 00007f426469dbd4
0709 13:04:07.753204 (+   601us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:07.753788 (+   584us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:07.753848 (+    60us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:07.770569 (+ 16721us) raft_consensus.cc:1645] Marking committed up to 1
0709 13:04:07.785478 (+ 14909us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:07.789428 (+  3950us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:07.789623 (+   195us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":69669248}
I20260709 13:04:07.759701 10874 raft_consensus.cc:2468] T 9a1716c0adf34bee8a4e99341ba5788f P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 4.
I20260709 13:04:07.761770 11953 raft_consensus.cc:2804] T f371d2565dd84667b5613848604949bb P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260709 13:04:07.762470 10994 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 52343.3us
I20260709 13:04:07.766163 10879 raft_consensus.cc:1240] T 7709387f4d3d4f9992d97a0fbca39a40 P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Rejecting Update request from peer 873357742d5843579556ee8166f23f47 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260709 13:04:07.734575 10873 rpcz_store.cc:276] 0709 13:04:06.549857 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:07.034704 (+484847us) service_pool.cc:224] Handling call
0709 13:04:07.064401 (+ 29697us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:07.691263 (+626862us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:07.691883 (+   620us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:07.752153 10953 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49434 (request call id 182) took 1238 ms. Trace:
W20260709 13:04:07.775225 11029 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.842s	user 0.002s	sys 0.017s
I20260709 13:04:07.781186 11937 raft_consensus.cc:2749] T 9a1716c0adf34bee8a4e99341ba5788f P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:04:07.773945 11849 raft_consensus.cc:493] T a1ec2b50c6ad4f7c8033476208abb26d P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:07.781901 10876 raft_consensus.cc:2376] T 57b6a2418c1342e7924195e9c1dbd357 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 2. Re-sending same reply.
W20260709 13:04:07.771958 10871 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51696 (request call id 203) took 1662 ms (client timeout 2010 ms). Trace:
I20260709 13:04:07.790156 11947 leader_election.cc:290] T 1e4f2241a94a41f38e481137f868b83d P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
W20260709 13:04:07.800721 10992 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:07.795853 11953 raft_consensus.cc:493] T f371d2565dd84667b5613848604949bb P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:07.772729 11936 raft_consensus.cc:493] T d371c5e27bd843469d2e2e453734cb4a P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:07.808597 11953 raft_consensus.cc:3060] T f371d2565dd84667b5613848604949bb P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:04:07.816671 11096 raft_consensus.cc:2376] T c31881534c5e4f0089c82057c1746652 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 2. Re-sending same reply.
I20260709 13:04:07.817888 11098 raft_consensus.cc:2468] T 75492f86fde046c3900df3eac8292fd4 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 1.
I20260709 13:04:07.818845 11936 raft_consensus.cc:515] T d371c5e27bd843469d2e2e453734cb4a P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:07.822165 11633 consensus_queue.cc:1059] T 8c160ddfe9b34bdaa3835d0915c1e06f P 873357742d5843579556ee8166f23f47 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:04:07.822614 11734 consensus_queue.cc:1059] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 873357742d5843579556ee8166f23f47 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:04:07.823318 11846 raft_consensus.cc:493] T a610d1a71f52432e88fe56130f921fa2 P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0172fed22b6b4890a7fdc953fa128af4)
I20260709 13:04:07.823798 11879 raft_consensus.cc:3055] T 57b6a2418c1342e7924195e9c1dbd357 P 873357742d5843579556ee8166f23f47 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:04:07.824245 11633 consensus_queue.cc:1059] T 11de93cd9b9d457489ff1cb80f976c9a P 873357742d5843579556ee8166f23f47 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:04:07.824167 11846 raft_consensus.cc:515] T a610d1a71f52432e88fe56130f921fa2 P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:07.824868 11921 raft_consensus.cc:3055] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 873357742d5843579556ee8166f23f47 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:04:07.825532 11921 raft_consensus.cc:740] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 873357742d5843579556ee8166f23f47 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 873357742d5843579556ee8166f23f47, State: Running, Role: LEADER
I20260709 13:04:07.826900 11095 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37676 (request call id 190) took 1534 ms. Trace:
I20260709 13:04:07.827416 11921 consensus_queue.cc:260] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 873357742d5843579556ee8166f23f47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:07.828799 11926 leader_election.cc:290] T ff13ae9a47c7455f962406c6d131a3ff P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:07.829854 11921 raft_consensus.cc:3060] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:04:07.830037 11962 raft_consensus.cc:493] T e96e80ba9ac34d0998395515e5c8c41a P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:07.824322 11734 raft_consensus.cc:3055] T 8c160ddfe9b34bdaa3835d0915c1e06f P 873357742d5843579556ee8166f23f47 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:04:07.832023 11734 raft_consensus.cc:740] T 8c160ddfe9b34bdaa3835d0915c1e06f P 873357742d5843579556ee8166f23f47 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 873357742d5843579556ee8166f23f47, State: Running, Role: LEADER
I20260709 13:04:07.831714 11962 raft_consensus.cc:515] T e96e80ba9ac34d0998395515e5c8c41a P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
W20260709 13:04:07.832654 11927 log.cc:927] Time spent T e89e640169e14e2d98f968f76027daa7 P 873357742d5843579556ee8166f23f47: Append to log took a long time: real 0.806s	user 0.013s	sys 0.040s
I20260709 13:04:07.824281 11879 raft_consensus.cc:740] T 57b6a2418c1342e7924195e9c1dbd357 P 873357742d5843579556ee8166f23f47 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 873357742d5843579556ee8166f23f47, State: Running, Role: LEADER
I20260709 13:04:07.833607 11734 consensus_queue.cc:260] T 8c160ddfe9b34bdaa3835d0915c1e06f P 873357742d5843579556ee8166f23f47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:07.798036 10953 rpcz_store.cc:276] 0709 13:04:06.513693 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:06.774576 (+260883us) service_pool.cc:224] Handling call
0709 13:04:06.796881 (+ 22305us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:06.963969 (+167088us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:07.751425 (+787456us) spinlock_profiling.cc:243] Waited 151 ms on lock 0x7b540078a354. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:04:07.751813 (+   388us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":346860928}
W20260709 13:04:07.811164 10876 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51696 (request call id 205) took 1687 ms (client timeout 1705 ms). Trace:
I20260709 13:04:07.836164 10873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5299b8d044f41b7b68057e63d17b159" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:07.836858 11734 raft_consensus.cc:3060] T 8c160ddfe9b34bdaa3835d0915c1e06f P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:04:07.837533 10995 leader_election.cc:304] T 9a1716c0adf34bee8a4e99341ba5788f P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:04:07.835772 11879 consensus_queue.cc:260] T 57b6a2418c1342e7924195e9c1dbd357 P 873357742d5843579556ee8166f23f47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:07.804334 11849 raft_consensus.cc:3060] T a1ec2b50c6ad4f7c8033476208abb26d P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:04:07.840343 11852 leader_election.cc:290] T d371c5e27bd843469d2e2e453734cb4a P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:07.840955 11962 leader_election.cc:290] T e96e80ba9ac34d0998395515e5c8c41a P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:07.838331 10873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a9353d1d230416296627d7cf8b1d425" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:04:07.842742 11883 leader_election.cc:290] T 868e92339c2f4508af0afbd07c814d64 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
W20260709 13:04:07.863636 10920 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 87554us
I20260709 13:04:07.876194 11025 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a9353d1d230416296627d7cf8b1d425" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:07.851883 11639 leader_election.cc:290] T bac7e5040f984ba2b7146bd36af7267e P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
W20260709 13:04:07.835534 10876 rpcz_store.cc:269] 0709 13:04:06.123249 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:06.125292 (+  2043us) service_pool.cc:224] Handling call
0709 13:04:07.747721 (+1622429us) spinlock_profiling.cc:243] Waited 17.9 ms on lock 0x7b5800000098. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f426303db71 00007f426303da22 00007f426bf56c09 00007f426b0b0ee0 00007f426b17a0ab 00007f426b17a131 00007f426b1280a0 00007f426b127984 00007f42646a26d4 00007f42646a2662
0709 13:04:07.809106 (+ 61385us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":41263488}
I20260709 13:04:07.857493 11947 raft_consensus.cc:2749] T f34f2bc02d85444f9096e0f5b4fe4331 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:04:07.857795 11936 leader_election.cc:290] T d371c5e27bd843469d2e2e453734cb4a P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
W20260709 13:04:07.803577 10871 rpcz_store.cc:269] 0709 13:04:06.109090 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:06.117635 (+  8545us) service_pool.cc:224] Handling call
0709 13:04:07.063940 (+946305us) spinlock_profiling.cc:243] Waited 5.85 ms on lock 0x7b5800000098. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f426303db71 00007f426303da22 00007f426bf56c09 00007f426b0b0ee0 00007f426b17a0ab 00007f426b17a131 00007f426b1280a0 00007f426b127984 00007f42646a26d4 00007f42646a2662
0709 13:04:07.683621 (+619681us) spinlock_profiling.cc:243] Waited 5.29 ms on lock 0x7b5800000098. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f426303de9c 00007f426303da5b 00007f426bf56c09 00007f426b0b0ee0 00007f426b17a0ab 00007f426b17a131 00007f426b1280a0 00007f426b127984 00007f42646a26d4 00007f42646a2662
0709 13:04:07.771820 (+ 88199us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":25611520}
I20260709 13:04:07.894515 11780 leader_election.cc:290] T 401a9cf11eac479aa445ec8c1d6e33b2 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:07.896647 11633 raft_consensus.cc:2804] T 9a1716c0adf34bee8a4e99341ba5788f P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:04:07.879161 11748 raft_consensus.cc:3055] T e5a5f5a608504650b48fd27f163faed6 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:04:07.882292 11920 raft_consensus.cc:3060] T 7839ca2d2c13431a85f44d7c458c6351 P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:04:07.877691 11025 raft_consensus.cc:2468] T 6a9353d1d230416296627d7cf8b1d425 P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 3.
I20260709 13:04:07.827466 11095 rpcz_store.cc:276] 0709 13:04:06.292024 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:06.305457 (+ 13433us) service_pool.cc:224] Handling call
0709 13:04:07.743910 (+1438453us) spinlock_profiling.cc:243] Waited 4.13 ms on lock 0x7b5800117698. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f426303db71 00007f426303da22 00007f426bf56c09 00007f426b0b0ee0 00007f426b17a0ab 00007f426b17a131 00007f426b13bfc0 00007f426b126929 00007f426b1263d2 00007f426469dbd4
0709 13:04:07.744416 (+   506us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:07.745085 (+   669us) raft_consensus.cc:1534] Early marking committed up to index 2
0709 13:04:07.748687 (+  3602us) log.cc:844] Serialized 14 byte log entry
0709 13:04:07.825801 (+ 77114us) log.cc:844] Serialized 14 byte log entry
0709 13:04:07.826059 (+   258us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:07.826128 (+    69us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:04:07.826210 (+    82us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:07.826303 (+    93us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:07.826578 (+   275us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":4039,"prepare.run_cpu_time_us":107,"prepare.run_wall_time_us":109,"raft.queue_time_us":15506,"raft.run_cpu_time_us":341,"raft.run_wall_time_us":342,"spinlock_wait_cycles":9498496,"thread_start_us":54463,"threads_started":1}
W20260709 13:04:07.923758 11094 consensus_meta.cc:287] T 8a69f32c53f24a50b7852d143665d7b4 P 0172fed22b6b4890a7fdc953fa128af4: Time spent flushing consensus metadata: real 0.790s	user 0.011s	sys 0.016s
I20260709 13:04:07.925822 11094 raft_consensus.cc:2468] T 8a69f32c53f24a50b7852d143665d7b4 P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 4.
I20260709 13:04:07.926028 10877 raft_consensus.cc:1240] T 57b6a2418c1342e7924195e9c1dbd357 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Rejecting Update request from peer 873357742d5843579556ee8166f23f47 for earlier term 1. Current term is 2. Ops: []
I20260709 13:04:07.891144 11879 raft_consensus.cc:3060] T 57b6a2418c1342e7924195e9c1dbd357 P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:04:07.891498 11026 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11de93cd9b9d457489ff1cb80f976c9a" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:07.902020 11748 raft_consensus.cc:740] T e5a5f5a608504650b48fd27f163faed6 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 10ca5ee68ad64b3283b9d235352c9d23, State: Running, Role: LEADER
I20260709 13:04:07.954659 11936 raft_consensus.cc:493] T 63d44f6ba6434ab2870fe3190f4d7c2f P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 186b12638a87410487467df721a3cbda)
I20260709 13:04:07.955757 11936 raft_consensus.cc:515] T 63d44f6ba6434ab2870fe3190f4d7c2f P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
W20260709 13:04:07.956532 11069 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:38967, user_credentials={real_user=slave}} blocked reactor thread for 233519us
W20260709 13:04:07.957386 11942 log.cc:927] Time spent T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 0172fed22b6b4890a7fdc953fa128af4: Append to log took a long time: real 0.980s	user 0.001s	sys 0.002s
I20260709 13:04:07.960389 11748 consensus_queue.cc:260] T e5a5f5a608504650b48fd27f163faed6 P 10ca5ee68ad64b3283b9d235352c9d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:07.962035 11748 raft_consensus.cc:3060] T e5a5f5a608504650b48fd27f163faed6 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:04:07.963939 11673 raft_consensus.cc:3055] T 11de93cd9b9d457489ff1cb80f976c9a P 873357742d5843579556ee8166f23f47 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:04:07.964480 11673 raft_consensus.cc:740] T 11de93cd9b9d457489ff1cb80f976c9a P 873357742d5843579556ee8166f23f47 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 873357742d5843579556ee8166f23f47, State: Running, Role: LEADER
I20260709 13:04:07.966765 11025 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a154cc8b181f460abf4880064379948b" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
W20260709 13:04:07.969364 10995 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:38967, user_credentials={real_user=slave}} blocked reactor thread for 132121us
I20260709 13:04:07.970901 11673 consensus_queue.cc:260] T 11de93cd9b9d457489ff1cb80f976c9a P 873357742d5843579556ee8166f23f47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
W20260709 13:04:07.972157 10994 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 72007.4us
I20260709 13:04:07.975760 11028 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb7d67be500947b3aef4c4245e33671c" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:07.977396 11028 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e4f2241a94a41f38e481137f868b83d" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
I20260709 13:04:07.978271 11028 raft_consensus.cc:3060] T 1e4f2241a94a41f38e481137f868b83d P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Advancing to term 2
W20260709 13:04:07.982971 10845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 881765us
I20260709 13:04:07.972886 11022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff13ae9a47c7455f962406c6d131a3ff" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:07.984534 11936 leader_election.cc:290] T 63d44f6ba6434ab2870fe3190f4d7c2f P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:07.997668 10920 leader_election.cc:304] T 0f9df2bc281f4866b6da5da94a0defc7 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23; no voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda
I20260709 13:04:07.997803 11022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "30f628e3d7914d609e7f1d980a74c654" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:07.999886 11027 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f46a5102fd64b45bf1c4dff863d2b4d" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:08.000270 11962 raft_consensus.cc:2749] T 0f9df2bc281f4866b6da5da94a0defc7 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:04:07.984683 11021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63d44f6ba6434ab2870fe3190f4d7c2f" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:07.985339 11639 raft_consensus.cc:493] T ff13ae9a47c7455f962406c6d131a3ff P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:07.962287 11944 consensus_queue.cc:1048] T 4b37f8fca8dc42718b2a42f0883a956f P 873357742d5843579556ee8166f23f47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.018s
I20260709 13:04:07.972545 11518 leader_election.cc:290] T 5d439326f23747eaab3c1e7da2d6b785 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:07.988142 11943 leader_election.cc:290] T c31881534c5e4f0089c82057c1746652 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:07.989228 11026 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d371c5e27bd843469d2e2e453734cb4a" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:07.948185 10879 raft_consensus.cc:1240] T 732c4adeeff6422497aa7db4a82f535b P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Rejecting Update request from peer 873357742d5843579556ee8166f23f47 for earlier term 1. Current term is 2. Ops: []
I20260709 13:04:07.993248 11673 raft_consensus.cc:3060] T 11de93cd9b9d457489ff1cb80f976c9a P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Advancing to term 4
W20260709 13:04:08.008836 10920 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 74985.3us
I20260709 13:04:08.013877 11962 raft_consensus.cc:493] T f34f2bc02d85444f9096e0f5b4fe4331 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:08.031347 11029 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46640 (request call id 173) took 1820 ms. Trace:
I20260709 13:04:08.016929 11067 leader_election.cc:304] T 6a9353d1d230416296627d7cf8b1d425 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:04:08.035333 11029 rpcz_store.cc:276] 0709 13:04:06.210637 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:06.210838 (+   201us) service_pool.cc:224] Handling call
0709 13:04:06.211279 (+   441us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:04:06.216451 (+  5172us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:06.216503 (+    52us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:04:06.217317 (+   814us) log.cc:844] Serialized 52 byte log entry
0709 13:04:06.932739 (+715422us) spinlock_profiling.cc:243] Waited 2.43 ms on lock 0x7b1800382120. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f4263057e07 00007f426beb2ab9 00007f426307469b 00007f426307293f 00007f4263074c02 00007f4265cfc20b 00007f4265d01d23 00007f4265d020d1 00007f426611da0f
0709 13:04:07.812915 (+880176us) raft_consensus.cc:1645] Marking committed up to 0
0709 13:04:07.813018 (+   103us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:07.813152 (+   134us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:08.012770 (+199618us) raft_consensus.cc:1685] finished
0709 13:04:08.012788 (+    18us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:08.031215 (+ 18427us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":602,"raft.run_cpu_time_us":212,"raft.run_wall_time_us":6158,"spinlock_wait_cycles":5594368,"thread_start_us":868843,"threads_started":1,"wal-append.queue_time_us":1558514}
I20260709 13:04:08.000310 11022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75492f86fde046c3900df3eac8292fd4" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:08.015790 11027 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5299b8d044f41b7b68057e63d17b159" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
W20260709 13:04:08.061079 10845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 65174.9us
I20260709 13:04:08.061782 11027 raft_consensus.cc:2468] T a5299b8d044f41b7b68057e63d17b159 P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 1.
I20260709 13:04:08.015264 11852 raft_consensus.cc:493] T e5ba72ccd854430baadbc636dfd17136 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:08.063252 11027 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7839ca2d2c13431a85f44d7c458c6351" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:08.063514 11852 raft_consensus.cc:515] T e5ba72ccd854430baadbc636dfd17136 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
W20260709 13:04:08.058559 11908 log.cc:927] Time spent T 4c2a35bb99374e93b573cf870d04faaa P 10ca5ee68ad64b3283b9d235352c9d23: Append to log took a long time: real 1.686s	user 0.002s	sys 0.002s
I20260709 13:04:08.066715 10952 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49434 (request call id 174) took 1948 ms. Trace:
I20260709 13:04:08.067178 10952 rpcz_store.cc:276] 0709 13:04:06.118224 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:06.119113 (+   889us) service_pool.cc:224] Handling call
0709 13:04:06.130928 (+ 11815us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:06.131433 (+   505us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:06.131465 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:06.131949 (+   484us) log.cc:844] Serialized 26 byte log entry
0709 13:04:06.199320 (+ 67371us) raft_consensus.cc:1645] Marking committed up to 1
0709 13:04:06.204125 (+  4805us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:06.213366 (+  9241us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:08.025342 (+1811976us) spinlock_profiling.cc:243] Waited 8.04 ms on lock 0x7b4400022b4c. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f4262f9bc85 00007f4262f9c6a2 00007f4265d0e1db 00007f4265d09aaf 00007f4265cfd86e 00007f4265cfcbb6 00007f4265cfc7d8 00007f4265d06221 00007f4265d061d9 00007f4265d06169
0709 13:04:08.066088 (+ 40746us) raft_consensus.cc:1685] finished
0709 13:04:08.066123 (+    35us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:08.066454 (+   331us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":11051,"raft.queue_time_us":3926,"raft.run_cpu_time_us":211,"raft.run_wall_time_us":13283,"spinlock_wait_cycles":18490240,"thread_start_us":57404,"threads_started":1,"wal-append.queue_time_us":115259}
I20260709 13:04:08.069953 11852 leader_election.cc:290] T e5ba72ccd854430baadbc636dfd17136 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:08.041440 11026 raft_consensus.cc:2468] T d371c5e27bd843469d2e2e453734cb4a P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 2.
I20260709 13:04:08.079984 11099 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37676 (request call id 192) took 1286 ms. Trace:
W20260709 13:04:08.081305 11911 log.cc:927] Time spent T 868e92339c2f4508af0afbd07c814d64 P 873357742d5843579556ee8166f23f47: Append to log took a long time: real 1.581s	user 0.028s	sys 0.026s
I20260709 13:04:08.081063 11024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d371c5e27bd843469d2e2e453734cb4a" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:08.082099 11024 raft_consensus.cc:2468] T d371c5e27bd843469d2e2e453734cb4a P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 2.
I20260709 13:04:08.014328 11842 consensus_queue.cc:1048] T a610d1a71f52432e88fe56130f921fa2 P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.070s
I20260709 13:04:08.076268 11639 raft_consensus.cc:515] T ff13ae9a47c7455f962406c6d131a3ff P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:08.060642 11022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e96e80ba9ac34d0998395515e5c8c41a" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:08.020757 11774 raft_consensus.cc:3055] T 732c4adeeff6422497aa7db4a82f535b P 873357742d5843579556ee8166f23f47 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:04:08.017341 11853 consensus_queue.cc:1048] T d71ca3a1ba6b48478aaf62741bf3cf6a P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.042s
I20260709 13:04:08.060124 10920 leader_election.cc:304] T b4cd3d7dfbbf4096908ff11e9dc57141 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23; no voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47
I20260709 13:04:08.036748 11021 raft_consensus.cc:2468] T 63d44f6ba6434ab2870fe3190f4d7c2f P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 2.
I20260709 13:04:08.038995 11936 raft_consensus.cc:493] T 4f9eb06ed01f4ce68be3af8a8434665b P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 186b12638a87410487467df721a3cbda)
I20260709 13:04:08.083536 11023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46640 (request call id 172) took 1895 ms. Trace:
W20260709 13:04:08.093621 11067 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:45807, user_credentials={real_user=slave}} blocked reactor thread for 76884us
I20260709 13:04:08.086570 11025 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c2a35bb99374e93b573cf870d04faaa" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:08.122020 11028 raft_consensus.cc:2410] T 1e4f2241a94a41f38e481137f868b83d P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 10ca5ee68ad64b3283b9d235352c9d23 for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260709 13:04:08.089274 11774 raft_consensus.cc:740] T 732c4adeeff6422497aa7db4a82f535b P 873357742d5843579556ee8166f23f47 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 873357742d5843579556ee8166f23f47, State: Running, Role: LEADER
I20260709 13:04:08.115023 11518 raft_consensus.cc:493] T 732c4adeeff6422497aa7db4a82f535b P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:08.135228 10880 raft_consensus.cc:1275] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Refusing update from remote peer 10ca5ee68ad64b3283b9d235352c9d23: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260709 13:04:08.135483 11518 raft_consensus.cc:515] T 732c4adeeff6422497aa7db4a82f535b P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:08.136251 10880 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51696 (request call id 204) took 2014 ms. Trace:
I20260709 13:04:08.136688 10880 rpcz_store.cc:276] 0709 13:04:06.122131 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:06.125579 (+  3448us) service_pool.cc:224] Handling call
0709 13:04:07.990545 (+1864966us) spinlock_profiling.cc:243] Waited 19.7 ms on lock 0x7b5800000098. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f426303db71 00007f426303da22 00007f426bf56c09 00007f426b0b0ee0 00007f426b17a0ab 00007f426b17a131 00007f426b13bfc0 00007f426b126929 00007f426b1263d2 00007f426469dbd4
0709 13:04:07.991001 (+   456us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:08.135883 (+144882us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:08.136185 (+   302us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":45212928}
I20260709 13:04:08.138105 11152 tablet_bootstrap.cc:492] T f34f2bc02d85444f9096e0f5b4fe4331 P 0172fed22b6b4890a7fdc953fa128af4: No bootstrap required, opened a new log
I20260709 13:04:08.139014 10879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c51782cd9d064ec0949e84e24dd4de67" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:08.139420 11518 leader_election.cc:290] T 732c4adeeff6422497aa7db4a82f535b P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:08.140962 11887 consensus_queue.cc:1048] T f371d2565dd84667b5613848604949bb P 186b12638a87410487467df721a3cbda [LEADER]: Connected to new peer: Peer: permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.012s
I20260709 13:04:08.099519 11846 leader_election.cc:290] T a610d1a71f52432e88fe56130f921fa2 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:08.122022 11067 leader_election.cc:304] T d371c5e27bd843469d2e2e453734cb4a P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:04:08.139020 11152 ts_tablet_manager.cc:1403] T f34f2bc02d85444f9096e0f5b4fe4331 P 0172fed22b6b4890a7fdc953fa128af4: Time spent bootstrapping tablet: real 0.941s	user 0.040s	sys 0.006s
I20260709 13:04:08.153362 11878 raft_consensus.cc:493] T d71ca3a1ba6b48478aaf62741bf3cf6a P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:08.154270 11967 log.cc:927] Time spent T 5de5f0581e3b41dea729363d6ca563bb P 10ca5ee68ad64b3283b9d235352c9d23: Append to log took a long time: real 0.052s	user 0.000s	sys 0.001s
I20260709 13:04:08.154100 11878 raft_consensus.cc:515] T d71ca3a1ba6b48478aaf62741bf3cf6a P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:08.156697 11878 leader_election.cc:290] T d71ca3a1ba6b48478aaf62741bf3cf6a P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:08.162730 10995 leader_election.cc:304] T 1e4f2241a94a41f38e481137f868b83d P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:04:08.163165 10879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f9eb06ed01f4ce68be3af8a8434665b" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:08.083559 11962 raft_consensus.cc:515] T f34f2bc02d85444f9096e0f5b4fe4331 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
W20260709 13:04:08.116730 10845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 54643.6us
I20260709 13:04:08.181568 10879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d156d6ae2ca4fa59a1e387edf0fea39" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:08.169567 11943 raft_consensus.cc:2804] T 6a9353d1d230416296627d7cf8b1d425 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:04:08.182405 10879 raft_consensus.cc:2468] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 2.
W20260709 13:04:08.191345 11067 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:45807, user_credentials={real_user=slave}} blocked reactor thread for 69387.1us
I20260709 13:04:08.176113 11846 raft_consensus.cc:2764] T 1e4f2241a94a41f38e481137f868b83d P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Leader pre-election decision vote started in defunct term 1: won
W20260709 13:04:08.195776 10919 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:45807, user_credentials={real_user=slave}} blocked reactor thread for 115976us
I20260709 13:04:08.174755 11633 raft_consensus.cc:697] T 9a1716c0adf34bee8a4e99341ba5788f P 873357742d5843579556ee8166f23f47 [term 4 LEADER]: Becoming Leader. State: Replica: 873357742d5843579556ee8166f23f47, State: Running, Role: LEADER
I20260709 13:04:08.150974 11936 raft_consensus.cc:515] T 4f9eb06ed01f4ce68be3af8a8434665b P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:08.192080 11943 raft_consensus.cc:493] T 6a9353d1d230416296627d7cf8b1d425 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:04:08.212530 10992 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 255439us
I20260709 13:04:08.219502 10879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555217d45d9b4ee492dbb3ddaf0f7c02" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:08.206849 11022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e96e80ba9ac34d0998395515e5c8c41a" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
W20260709 13:04:08.221771 10918 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:08.221517 10843 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 390080us
I20260709 13:04:08.080597 11099 rpcz_store.cc:276] 0709 13:04:06.793356 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:06.793527 (+   171us) service_pool.cc:224] Handling call
0709 13:04:06.793981 (+   454us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:04:06.794427 (+   446us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:06.794476 (+    49us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:04:06.795303 (+   827us) log.cc:844] Serialized 52 byte log entry
0709 13:04:06.977952 (+182649us) raft_consensus.cc:1645] Marking committed up to 1
0709 13:04:06.978024 (+    72us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:06.978128 (+   104us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:07.956152 (+978024us) spinlock_profiling.cc:243] Waited 58.9 ms on lock 0x7b4400022b4c. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f4262f9bc85 00007f4262f9c6a2 00007f4265d0e1db 00007f4265d09aaf 00007f4265cfd86e 00007f4265cfcbb6 00007f4265cfc7d8 00007f4265d06221 00007f4265d061d9 00007f4265d06169
0709 13:04:08.079240 (+123088us) raft_consensus.cc:1685] finished
0709 13:04:08.079272 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:08.079724 (+   452us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":29236,"raft.run_cpu_time_us":135,"raft.run_wall_time_us":135,"spinlock_wait_cycles":135521024,"thread_start_us":182118,"threads_started":1,"wal-append.queue_time_us":181432}
I20260709 13:04:08.222927 10845 leader_election.cc:304] T d270b9e5481245d9b5340a454a7cd10b P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda; no voters: 10ca5ee68ad64b3283b9d235352c9d23
W20260709 13:04:08.224069 10920 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 164130us
I20260709 13:04:08.226677 10872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "401a9cf11eac479aa445ec8c1d6e33b2" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:08.227823 11639 leader_election.cc:290] T ff13ae9a47c7455f962406c6d131a3ff P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:08.232637 11962 leader_election.cc:290] T f34f2bc02d85444f9096e0f5b4fe4331 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:08.233139 11937 raft_consensus.cc:2804] T d371c5e27bd843469d2e2e453734cb4a P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260709 13:04:08.234493 11097 consensus_meta.cc:287] T 7709387f4d3d4f9992d97a0fbca39a40 P 0172fed22b6b4890a7fdc953fa128af4: Time spent flushing consensus metadata: real 0.910s	user 0.018s	sys 0.008s
W20260709 13:04:08.267768 10919 leader_election.cc:343] T 30f628e3d7914d609e7f1d980a74c654 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 873357742d5843579556ee8166f23f47 (127.0.107.195:45807): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:04:08.221624 11782 raft_consensus.cc:493] T 0f9df2bc281f4866b6da5da94a0defc7 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0172fed22b6b4890a7fdc953fa128af4)
I20260709 13:04:08.245908 10879 raft_consensus.cc:2393] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0172fed22b6b4890a7fdc953fa128af4 in current term 3: Already voted for candidate 186b12638a87410487467df721a3cbda in this term.
I20260709 13:04:08.100805 11023 rpcz_store.cc:276] 0709 13:04:06.188410 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:06.191183 (+  2773us) service_pool.cc:224] Handling call
0709 13:04:06.220106 (+ 28923us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:06.228546 (+  8440us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:06.228585 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:06.229190 (+   605us) log.cc:844] Serialized 26 byte log entry
0709 13:04:06.328499 (+ 99309us) raft_consensus.cc:1645] Marking committed up to 1
0709 13:04:06.334126 (+  5627us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:06.340764 (+  6638us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:08.021593 (+1680829us) spinlock_profiling.cc:243] Waited 8.24 ms on lock 0x7b4400022b4c. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f4262f9bc85 00007f4262f9c6a2 00007f4265d0e1db 00007f4265d09aaf 00007f4265cfd86e 00007f4265cfcbb6 00007f4265cfc7d8 00007f4265d06221 00007f4265d061d9 00007f4265d06169
0709 13:04:08.082896 (+ 61303us) raft_consensus.cc:1685] finished
0709 13:04:08.082933 (+    37us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:08.083283 (+   350us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":775,"raft.run_cpu_time_us":131,"raft.run_wall_time_us":129,"spinlock_wait_cycles":18951936,"thread_start_us":76226,"threads_started":1,"wal-append.queue_time_us":208273}
W20260709 13:04:08.245167 11103 consensus_meta.cc:287] T a977842ba4b744f4bb2febd55d286006 P 0172fed22b6b4890a7fdc953fa128af4: Time spent flushing consensus metadata: real 0.780s	user 0.010s	sys 0.016s
I20260709 13:04:08.214592 10877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63d44f6ba6434ab2870fe3190f4d7c2f" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:08.250574 11937 raft_consensus.cc:493] T d371c5e27bd843469d2e2e453734cb4a P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:08.240526 11774 consensus_queue.cc:260] T 732c4adeeff6422497aa7db4a82f535b P 873357742d5843579556ee8166f23f47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:08.267503 11025 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "30f628e3d7914d609e7f1d980a74c654" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
W20260709 13:04:08.262493 11969 log.cc:927] Time spent T e96e80ba9ac34d0998395515e5c8c41a P 186b12638a87410487467df721a3cbda: Append to log took a long time: real 0.053s	user 0.000s	sys 0.000s
I20260709 13:04:08.259763 10872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d270b9e5481245d9b5340a454a7cd10b" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
W20260709 13:04:08.263470 11885 consensus_meta.cc:287] T 34064dc7bf3747dbb3876dd824ee0688 P 873357742d5843579556ee8166f23f47: Time spent flushing consensus metadata: real 0.713s	user 0.014s	sys 0.009s
I20260709 13:04:08.260449 11748 raft_consensus.cc:2749] T b4cd3d7dfbbf4096908ff11e9dc57141 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20260709 13:04:08.276069 10995 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:08.236007 11943 raft_consensus.cc:3060] T 6a9353d1d230416296627d7cf8b1d425 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:04:08.217994 10880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d439326f23747eaab3c1e7da2d6b785" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:08.326593 10880 raft_consensus.cc:2410] T 5d439326f23747eaab3c1e7da2d6b785 P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0172fed22b6b4890a7fdc953fa128af4 for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 13:04:08.327756 10880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "30f628e3d7914d609e7f1d980a74c654" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:08.328605 10880 raft_consensus.cc:2468] T 30f628e3d7914d609e7f1d980a74c654 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 0.
I20260709 13:04:08.329095 10874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d371c5e27bd843469d2e2e453734cb4a" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:08.329694 10880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7839ca2d2c13431a85f44d7c458c6351" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:08.216109 11953 raft_consensus.cc:515] T f371d2565dd84667b5613848604949bb P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:08.276958 11097 raft_consensus.cc:2410] T 7709387f4d3d4f9992d97a0fbca39a40 P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 186b12638a87410487467df721a3cbda for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 13:04:08.317210 11103 raft_consensus.cc:2468] T a977842ba4b744f4bb2febd55d286006 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 1.
I20260709 13:04:08.322809 11795 raft_consensus.cc:2804] T d270b9e5481245d9b5340a454a7cd10b P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:04:08.295056 11782 raft_consensus.cc:515] T 0f9df2bc281f4866b6da5da94a0defc7 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:08.324339 11972 consensus_queue.cc:1059] T 732c4adeeff6422497aa7db4a82f535b P 873357742d5843579556ee8166f23f47 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.006s
I20260709 13:04:08.312786 11639 raft_consensus.cc:493] T d71ca3a1ba6b48478aaf62741bf3cf6a P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:08.325189 10920 leader_election.cc:304] T a64b7e99d6c54934a7de6f6dffcf5e94 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23; no voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47
I20260709 13:04:08.279333 10844 leader_election.cc:304] T d371c5e27bd843469d2e2e453734cb4a P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:04:08.278010 11887 consensus_queue.cc:1048] T f371d2565dd84667b5613848604949bb P 186b12638a87410487467df721a3cbda [LEADER]: Connected to new peer: Peer: permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.059s
I20260709 13:04:08.345685 11639 raft_consensus.cc:515] T d71ca3a1ba6b48478aaf62741bf3cf6a P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
W20260709 13:04:08.346333 10844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:45807, user_credentials={real_user=slave}} blocked reactor thread for 67342.3us
I20260709 13:04:08.278410 11960 consensus_queue.cc:1048] T a5299b8d044f41b7b68057e63d17b159 P 186b12638a87410487467df721a3cbda [LEADER]: Connected to new peer: Peer: permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.063s
I20260709 13:04:08.348135 11639 leader_election.cc:290] T d71ca3a1ba6b48478aaf62741bf3cf6a P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:08.332711 10879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c51782cd9d064ec0949e84e24dd4de67" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:08.351316 10879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5299b8d044f41b7b68057e63d17b159" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:08.351608 11953 leader_election.cc:290] T f371d2565dd84667b5613848604949bb P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 3 election: Requested vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:08.352669 10879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c31881534c5e4f0089c82057c1746652" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:08.353057 11099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5ba72ccd854430baadbc636dfd17136" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:08.353614 10879 raft_consensus.cc:2410] T c31881534c5e4f0089c82057c1746652 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0172fed22b6b4890a7fdc953fa128af4 for term 3 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260709 13:04:08.353875 11099 raft_consensus.cc:2468] T e5ba72ccd854430baadbc636dfd17136 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 3.
I20260709 13:04:08.354827 11103 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37676 (request call id 199) took 1032 ms. Trace:
I20260709 13:04:08.355240 11103 rpcz_store.cc:276] 0709 13:04:07.322550 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:07.424421 (+101871us) service_pool.cc:224] Handling call
0709 13:04:08.338587 (+914166us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:08.314030 11022 raft_consensus.cc:1240] T e89e640169e14e2d98f968f76027daa7 P 873357742d5843579556ee8166f23f47 [term 4 LEADER]: Rejecting Update request from peer 186b12638a87410487467df721a3cbda for earlier term 3. Current term is 4. Ops: []
I20260709 13:04:08.312326 11937 raft_consensus.cc:3060] T d371c5e27bd843469d2e2e453734cb4a P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:04:08.354840 11553 raft_consensus.cc:493] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 873357742d5843579556ee8166f23f47)
W20260709 13:04:08.358845 11097 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37678 (request call id 98) took 1411 ms (client timeout 1711 ms). Trace:
W20260709 13:04:08.359361 11097 rpcz_store.cc:269] 0709 13:04:06.947863 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:06.960086 (+ 12223us) service_pool.cc:224] Handling call
0709 13:04:08.358709 (+1398623us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:08.360488 11628 raft_consensus.cc:493] T 7709387f4d3d4f9992d97a0fbca39a40 P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:08.360728 11918 raft_consensus.cc:493] T 7839ca2d2c13431a85f44d7c458c6351 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:08.361066 11628 raft_consensus.cc:515] T 7709387f4d3d4f9992d97a0fbca39a40 P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:08.361397 11918 raft_consensus.cc:515] T 7839ca2d2c13431a85f44d7c458c6351 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:08.274343 11152 raft_consensus.cc:359] T f34f2bc02d85444f9096e0f5b4fe4331 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:08.362694 11152 raft_consensus.cc:385] T f34f2bc02d85444f9096e0f5b4fe4331 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:04:08.362959 11100 raft_consensus.cc:1240] T 7709387f4d3d4f9992d97a0fbca39a40 P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Rejecting Update request from peer 873357742d5843579556ee8166f23f47 for earlier term 3. Current term is 4. Ops: []
I20260709 13:04:08.364001 11918 leader_election.cc:290] T 7839ca2d2c13431a85f44d7c458c6351 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:08.364607 11100 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37676 (request call id 191) took 2065 ms. Trace:
I20260709 13:04:08.365494 11020 raft_consensus.cc:1240] T ff13ae9a47c7455f962406c6d131a3ff P 873357742d5843579556ee8166f23f47 [term 2 LEADER]: Rejecting Update request from peer 10ca5ee68ad64b3283b9d235352c9d23 for earlier term 1. Current term is 2. Ops: [1.1-1.1]
I20260709 13:04:08.362982 11152 raft_consensus.cc:740] T f34f2bc02d85444f9096e0f5b4fe4331 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0172fed22b6b4890a7fdc953fa128af4, State: Initialized, Role: FOLLOWER
I20260709 13:04:08.337553 11795 raft_consensus.cc:493] T d270b9e5481245d9b5340a454a7cd10b P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Starting leader election (detected failure of leader 10ca5ee68ad64b3283b9d235352c9d23)
I20260709 13:04:08.366914 11795 raft_consensus.cc:3060] T d270b9e5481245d9b5340a454a7cd10b P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:04:08.366969 11152 consensus_queue.cc:260] T f34f2bc02d85444f9096e0f5b4fe4331 P 0172fed22b6b4890a7fdc953fa128af4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:08.369127 11026 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bac7e5040f984ba2b7146bd36af7267e" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:08.369992 11152 ts_tablet_manager.cc:1434] T f34f2bc02d85444f9096e0f5b4fe4331 P 0172fed22b6b4890a7fdc953fa128af4: Time spent starting tablet: real 0.200s	user 0.004s	sys 0.004s
I20260709 13:04:08.365907 11553 raft_consensus.cc:515] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:08.373416 11774 raft_consensus.cc:3060] T 732c4adeeff6422497aa7db4a82f535b P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:04:08.373776 11553 leader_election.cc:290] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:08.374653 10873 raft_consensus.cc:2468] T 6a9353d1d230416296627d7cf8b1d425 P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 3.
I20260709 13:04:08.341480 11101 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "525bfdbd6d934afd823f7d3ee3f9d052" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:08.329905 10874 raft_consensus.cc:2468] T d371c5e27bd843469d2e2e453734cb4a P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 2.
I20260709 13:04:08.324517 10919 leader_election.cc:304] T a5299b8d044f41b7b68057e63d17b159 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:04:08.376647 11782 leader_election.cc:290] T 0f9df2bc281f4866b6da5da94a0defc7 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:08.344779 10876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a9353d1d230416296627d7cf8b1d425" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:08.389308 10879 raft_consensus.cc:1275] T 5d439326f23747eaab3c1e7da2d6b785 P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Refusing update from remote peer 10ca5ee68ad64b3283b9d235352c9d23: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20260709 13:04:08.348875 10875 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51718 (request call id 210) took 1798 ms. Trace:
I20260709 13:04:08.389860 10876 raft_consensus.cc:2468] T 6a9353d1d230416296627d7cf8b1d425 P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 3.
I20260709 13:04:08.349439 10877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d371c5e27bd843469d2e2e453734cb4a" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
W20260709 13:04:08.344313 10845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 121814us
I20260709 13:04:08.390182 10875 rpcz_store.cc:276] 0709 13:04:06.550162 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:07.689618 (+1139456us) service_pool.cc:224] Handling call
0709 13:04:07.697309 (+  7691us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:07.807292 (+109983us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:07.810400 (+  3108us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:07.838303 (+ 27903us) log.cc:844] Serialized 26 byte log entry
0709 13:04:08.038489 (+200186us) raft_consensus.cc:1645] Marking committed up to 1
0709 13:04:08.039385 (+   896us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:08.043352 (+  3967us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:08.273207 (+229855us) spinlock_profiling.cc:243] Waited 22 ms on lock 0x7b48002655d8. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426b08baa1 00007f426b082b37 00007f426b0829ca 00007f426b17a26f 00007f426b179f37 00007f426b18e55c 00007f426b18e508 00007f426b18e488 00007f426b18e443 00007f426b18d6ec 00007f426a0b7a96
0709 13:04:08.345782 (+ 72575us) raft_consensus.cc:1685] finished
0709 13:04:08.345803 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:08.348625 (+  2822us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5109,"raft.queue_time_us":161129,"raft.run_cpu_time_us":2358,"raft.run_wall_time_us":412242,"spinlock_wait_cycles":50596736,"thread_start_us":274432,"threads_started":2,"wal-append.queue_time_us":291868}
I20260709 13:04:08.295820 11936 leader_election.cc:290] T 4f9eb06ed01f4ce68be3af8a8434665b P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:08.392843 10845 leader_election.cc:304] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 186b12638a87410487467df721a3cbda; no voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47
I20260709 13:04:08.394663 10845 leader_election.cc:304] T 5d439326f23747eaab3c1e7da2d6b785 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda; no voters: 
I20260709 13:04:08.394896 11633 consensus_queue.cc:237] T 9a1716c0adf34bee8a4e99341ba5788f P 873357742d5843579556ee8166f23f47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:08.395882 11545 raft_consensus.cc:2749] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 13:04:08.397878 10845 leader_election.cc:304] T 0e5d327630e5438b878035d28efe6c0c P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 186b12638a87410487467df721a3cbda; no voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47
I20260709 13:04:08.398470 10880 raft_consensus.cc:2393] T 7839ca2d2c13431a85f44d7c458c6351 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 10ca5ee68ad64b3283b9d235352c9d23 in current term 2: Already voted for candidate 186b12638a87410487467df721a3cbda in this term.
I20260709 13:04:08.389856 10879 pending_rounds.cc:85] T 5d439326f23747eaab3c1e7da2d6b785 P 186b12638a87410487467df721a3cbda: Aborting all ops after (but not including) 1
I20260709 13:04:08.398998 10879 pending_rounds.cc:107] T 5d439326f23747eaab3c1e7da2d6b785 P 186b12638a87410487467df721a3cbda: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20260709 13:04:08.399787 11960 raft_consensus.cc:2749] T 0e5d327630e5438b878035d28efe6c0c P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 13:04:08.364993 11100 rpcz_store.cc:276] 0709 13:04:06.298956 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:06.305749 (+  6793us) service_pool.cc:224] Handling call
0709 13:04:07.966119 (+1660370us) spinlock_profiling.cc:243] Waited 3.91 ms on lock 0x7b5800117698. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f426303db71 00007f426303da22 00007f426bf56c09 00007f426b0b0ee0 00007f426b17a0ab 00007f426b17a131 00007f426b13bfc0 00007f426b126929 00007f426b1263d2 00007f426469dbd4
0709 13:04:08.362595 (+396476us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:08.363432 (+   837us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:08.364346 (+   914us) spinlock_profiling.cc:243] Waited 396 ms on lock 0x7b54001545d0. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661618ad 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:04:08.364518 (+   172us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":919557248}
I20260709 13:04:08.401831 10845 leader_election.cc:304] T 8c160ddfe9b34bdaa3835d0915c1e06f P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda; no voters: 873357742d5843579556ee8166f23f47
I20260709 13:04:08.399372 10879 raft_consensus.cc:2889] T 5d439326f23747eaab3c1e7da2d6b785 P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:04:08.406149 11628 leader_election.cc:290] T 7709387f4d3d4f9992d97a0fbca39a40 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:08.407312 11959 raft_consensus.cc:2804] T 5d439326f23747eaab3c1e7da2d6b785 P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:04:08.408763 10845 leader_election.cc:304] T e5a5f5a608504650b48fd27f163faed6 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 186b12638a87410487467df721a3cbda; no voters: 0172fed22b6b4890a7fdc953fa128af4, 10ca5ee68ad64b3283b9d235352c9d23
I20260709 13:04:08.409881 11960 raft_consensus.cc:2804] T 8c160ddfe9b34bdaa3835d0915c1e06f P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Leader election won for term 2
W20260709 13:04:08.410488 10920 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 85467.3us
I20260709 13:04:08.410775 11748 raft_consensus.cc:2749] T a64b7e99d6c54934a7de6f6dffcf5e94 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:04:08.410691 11782 raft_consensus.cc:3060] T e5a5f5a608504650b48fd27f163faed6 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:04:08.412238 10920 leader_election.cc:304] T 5de5f0581e3b41dea729363d6ca563bb P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23; no voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda
I20260709 13:04:08.412484 11639 consensus_queue.cc:1048] T 836251ebc2b74f5da88b8a8b2338e2a5 P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:04:08.410537 11960 raft_consensus.cc:697] T 8c160ddfe9b34bdaa3835d0915c1e06f P 186b12638a87410487467df721a3cbda [term 2 LEADER]: Becoming Leader. State: Replica: 186b12638a87410487467df721a3cbda, State: Running, Role: LEADER
I20260709 13:04:08.415553 11960 consensus_queue.cc:237] T 8c160ddfe9b34bdaa3835d0915c1e06f P 186b12638a87410487467df721a3cbda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:08.419409 11149 tablet_bootstrap.cc:492] T 30f628e3d7914d609e7f1d980a74c654 P 873357742d5843579556ee8166f23f47: No bootstrap required, opened a new log
I20260709 13:04:08.420324 11149 ts_tablet_manager.cc:1403] T 30f628e3d7914d609e7f1d980a74c654 P 873357742d5843579556ee8166f23f47: Time spent bootstrapping tablet: real 1.607s	user 0.020s	sys 0.015s
I20260709 13:04:08.387149 10878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1ec2b50c6ad4f7c8033476208abb26d" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:08.426512 10878 raft_consensus.cc:2468] T a1ec2b50c6ad4f7c8033476208abb26d P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 0.
I20260709 13:04:08.427606 11518 raft_consensus.cc:493] T 57b6a2418c1342e7924195e9c1dbd357 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:08.428110 11963 raft_consensus.cc:493] T 72b111ad7b5f4753ab1d2b5ee1114840 P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:08.428262 11518 raft_consensus.cc:515] T 57b6a2418c1342e7924195e9c1dbd357 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:08.428791 11688 raft_consensus.cc:493] T d7ce47b9a92d43dd88d95bcda5038597 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 10ca5ee68ad64b3283b9d235352c9d23)
I20260709 13:04:08.428922 11963 raft_consensus.cc:515] T 72b111ad7b5f4753ab1d2b5ee1114840 P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:08.429503 11688 raft_consensus.cc:515] T d7ce47b9a92d43dd88d95bcda5038597 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:08.430641 11545 raft_consensus.cc:493] T c31881534c5e4f0089c82057c1746652 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:08.431232 11545 raft_consensus.cc:515] T c31881534c5e4f0089c82057c1746652 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:08.433056 11796 raft_consensus.cc:493] T 7757ecbf402e47bfbc93d0447741e26c P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:08.433670 11796 raft_consensus.cc:515] T 7757ecbf402e47bfbc93d0447741e26c P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:08.428268 11797 raft_consensus.cc:493] T 836251ebc2b74f5da88b8a8b2338e2a5 P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 10ca5ee68ad64b3283b9d235352c9d23)
I20260709 13:04:08.434944 11797 raft_consensus.cc:515] T 836251ebc2b74f5da88b8a8b2338e2a5 P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:08.436147 11796 leader_election.cc:290] T 7757ecbf402e47bfbc93d0447741e26c P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
W20260709 13:04:08.439541 11849 consensus_meta.cc:287] T a1ec2b50c6ad4f7c8033476208abb26d P 0172fed22b6b4890a7fdc953fa128af4: Time spent flushing consensus metadata: real 0.574s	user 0.013s	sys 0.002s
W20260709 13:04:08.445642 10992 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 129890us
W20260709 13:04:08.459712 10919 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:45807, user_credentials={real_user=slave}} blocked reactor thread for 135335us
I20260709 13:04:08.463618 11100 raft_consensus.cc:1275] T a64b7e99d6c54934a7de6f6dffcf5e94 P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Refusing update from remote peer 873357742d5843579556ee8166f23f47: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 13:04:08.465369 11748 raft_consensus.cc:2804] T a5299b8d044f41b7b68057e63d17b159 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:04:08.465971 11748 raft_consensus.cc:493] T a5299b8d044f41b7b68057e63d17b159 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:08.466423 11748 raft_consensus.cc:3060] T a5299b8d044f41b7b68057e63d17b159 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:04:08.467469 11795 raft_consensus.cc:515] T d270b9e5481245d9b5340a454a7cd10b P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
W20260709 13:04:08.449259 11920 consensus_meta.cc:287] T 7839ca2d2c13431a85f44d7c458c6351 P 873357742d5843579556ee8166f23f47: Time spent flushing consensus metadata: real 0.511s	user 0.004s	sys 0.012s
I20260709 13:04:08.387790 11887 raft_consensus.cc:2804] T d371c5e27bd843469d2e2e453734cb4a P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:04:08.490157 10919 leader_election.cc:304] T 63d44f6ba6434ab2870fe3190f4d7c2f P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47; no voters: 186b12638a87410487467df721a3cbda
I20260709 13:04:08.491554 11959 raft_consensus.cc:493] T 5d439326f23747eaab3c1e7da2d6b785 P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:08.491364 11026 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5ba72ccd854430baadbc636dfd17136" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:08.492143 11959 raft_consensus.cc:3060] T 5d439326f23747eaab3c1e7da2d6b785 P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:04:08.493382 11782 raft_consensus.cc:2749] T e5a5f5a608504650b48fd27f163faed6 P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:04:08.440116 11849 raft_consensus.cc:515] T a1ec2b50c6ad4f7c8033476208abb26d P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
W20260709 13:04:08.487416 11921 consensus_meta.cc:287] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 873357742d5843579556ee8166f23f47: Time spent flushing consensus metadata: real 0.646s	user 0.010s	sys 0.008s
I20260709 13:04:08.504572 10953 raft_consensus.cc:1240] T 57b6a2418c1342e7924195e9c1dbd357 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Rejecting Update request from peer 873357742d5843579556ee8166f23f47 for earlier term 1. Current term is 2. Ops: []
I20260709 13:04:08.535087 10872 raft_consensus.cc:2393] T d270b9e5481245d9b5340a454a7cd10b P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 0172fed22b6b4890a7fdc953fa128af4 in current term 4: Already voted for candidate 186b12638a87410487467df721a3cbda in this term.
I20260709 13:04:08.535586 11936 raft_consensus.cc:493] T b4cd3d7dfbbf4096908ff11e9dc57141 P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:08.536345 11936 raft_consensus.cc:515] T b4cd3d7dfbbf4096908ff11e9dc57141 P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:08.538347 11795 leader_election.cc:290] T d270b9e5481245d9b5340a454a7cd10b P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 4 election: Requested vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:08.540627 11963 leader_election.cc:290] T 72b111ad7b5f4753ab1d2b5ee1114840 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
W20260709 13:04:08.538919 10994 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:08.542333 10919 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:45807, user_credentials={real_user=slave}} blocked reactor thread for 54205.7us
I20260709 13:04:08.543430 10919 leader_election.cc:304] T 1e4f2241a94a41f38e481137f868b83d P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23; no voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47
I20260709 13:04:08.534852 11518 leader_election.cc:290] T 57b6a2418c1342e7924195e9c1dbd357 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:08.506565 11628 raft_consensus.cc:493] T 87cd8f5097624c8692f1d83544d8881e P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 186b12638a87410487467df721a3cbda)
I20260709 13:04:08.494939 11639 raft_consensus.cc:493] T 87cd8f5097624c8692f1d83544d8881e P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 186b12638a87410487467df721a3cbda)
I20260709 13:04:08.505134 10954 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a977842ba4b744f4bb2febd55d286006" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:04:08.545059 11981 raft_consensus.cc:2804] T 63d44f6ba6434ab2870fe3190f4d7c2f P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:04:08.544968 11628 raft_consensus.cc:515] T 87cd8f5097624c8692f1d83544d8881e P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:08.547528 11981 raft_consensus.cc:493] T 63d44f6ba6434ab2870fe3190f4d7c2f P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 186b12638a87410487467df721a3cbda)
I20260709 13:04:08.548028 11981 raft_consensus.cc:3060] T 63d44f6ba6434ab2870fe3190f4d7c2f P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:04:08.549594 11628 leader_election.cc:290] T 87cd8f5097624c8692f1d83544d8881e P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
W20260709 13:04:08.554831 10920 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 58792.1us
I20260709 13:04:08.556416 11978 consensus_queue.cc:1048] T 5d439326f23747eaab3c1e7da2d6b785 P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:04:08.556856 10920 leader_election.cc:304] T a154cc8b181f460abf4880064379948b P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23; no voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47
I20260709 13:04:08.558120 11974 raft_consensus.cc:2749] T a154cc8b181f460abf4880064379948b P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:04:08.559306 11101 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a9353d1d230416296627d7cf8b1d425" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:04:08.559641 11936 leader_election.cc:290] T b4cd3d7dfbbf4096908ff11e9dc57141 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:08.560813 11103 raft_consensus.cc:1240] T 8c160ddfe9b34bdaa3835d0915c1e06f P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Rejecting Update request from peer 873357742d5843579556ee8166f23f47 for earlier term 1. Current term is 2. Ops: []
I20260709 13:04:08.561327 11094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a610d1a71f52432e88fe56130f921fa2" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:08.556669 11962 consensus_queue.cc:1048] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:04:08.500339 10952 raft_consensus.cc:1240] T 732c4adeeff6422497aa7db4a82f535b P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 LEADER]: Rejecting Update request from peer 873357742d5843579556ee8166f23f47 for earlier term 1. Current term is 2. Ops: []
I20260709 13:04:08.583015 11097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d371c5e27bd843469d2e2e453734cb4a" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:08.593497 11099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "401a9cf11eac479aa445ec8c1d6e33b2" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:08.593885 10949 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e4f2241a94a41f38e481137f868b83d" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:08.546190 10954 raft_consensus.cc:3060] T a977842ba4b744f4bb2febd55d286006 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Advancing to term 1
W20260709 13:04:08.594635 10826 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 13:04:08.596060 10953 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65bf0a62f4d84ef18dd6b23e082724b8" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:08.597682 10953 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "525bfdbd6d934afd823f7d3ee3f9d052" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:08.598886 10920 leader_election.cc:304] T bb7d67be500947b3aef4c4245e33671c P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23; no voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47
I20260709 13:04:08.599522 10950 raft_consensus.cc:1275] T a64b7e99d6c54934a7de6f6dffcf5e94 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Refusing update from remote peer 873357742d5843579556ee8166f23f47: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 13:04:08.600814 10950 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5ba72ccd854430baadbc636dfd17136" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:08.601257 11779 consensus_queue.cc:1048] T e5ba72ccd854430baadbc636dfd17136 P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:04:08.601552 11978 raft_consensus.cc:2749] T bb7d67be500947b3aef4c4245e33671c P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:04:08.602105 11974 consensus_queue.cc:1048] T 72b111ad7b5f4753ab1d2b5ee1114840 P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:04:08.605603 11022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f1466124dbd4f6ebb3c4d62ca48bf96" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:08.499539 11961 raft_consensus.cc:2749] T 5de5f0581e3b41dea729363d6ca563bb P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:04:08.497186 10871 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a610d1a71f52432e88fe56130f921fa2" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:08.614583 11098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d71ca3a1ba6b48478aaf62741bf3cf6a" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:08.614856 10871 raft_consensus.cc:2468] T a610d1a71f52432e88fe56130f921fa2 P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 4.
I20260709 13:04:08.545819 11983 raft_consensus.cc:2749] T 1e4f2241a94a41f38e481137f868b83d P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260709 13:04:08.608613 11066 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 121616us
I20260709 13:04:08.605975 11026 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff13ae9a47c7455f962406c6d131a3ff" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:08.613207 11023 raft_consensus.cc:1275] T 7757ecbf402e47bfbc93d0447741e26c P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Refusing update from remote peer 0172fed22b6b4890a7fdc953fa128af4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:04:08.613940 11094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f9eb06ed01f4ce68be3af8a8434665b" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:08.617641 11029 raft_consensus.cc:1275] T b4cd3d7dfbbf4096908ff11e9dc57141 P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Refusing update from remote peer 0172fed22b6b4890a7fdc953fa128af4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 13:04:08.496973 11896 raft_consensus.cc:493] T 2567c2d84e4648d7ba5dc23a3df32817 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 10ca5ee68ad64b3283b9d235352c9d23)
I20260709 13:04:08.630841 11849 leader_election.cc:290] T a1ec2b50c6ad4f7c8033476208abb26d P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 1 election: Requested vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:08.631374 11096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f371d2565dd84667b5613848604949bb" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:04:08.632191 11096 raft_consensus.cc:3060] T f371d2565dd84667b5613848604949bb P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:04:08.499503 11027 raft_consensus.cc:2468] T 7839ca2d2c13431a85f44d7c458c6351 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 2.
I20260709 13:04:08.502519 10877 raft_consensus.cc:2468] T d371c5e27bd843469d2e2e453734cb4a P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 2.
I20260709 13:04:08.624061 11094 raft_consensus.cc:2468] T 4f9eb06ed01f4ce68be3af8a8434665b P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 2.
I20260709 13:04:08.649924 10920 leader_election.cc:304] T 75492f86fde046c3900df3eac8292fd4 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 10ca5ee68ad64b3283b9d235352c9d23; no voters: 873357742d5843579556ee8166f23f47
I20260709 13:04:08.624189 11149 raft_consensus.cc:359] T 30f628e3d7914d609e7f1d980a74c654 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:08.651599 11149 raft_consensus.cc:385] T 30f628e3d7914d609e7f1d980a74c654 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:04:08.625466 11688 leader_election.cc:290] T d7ce47b9a92d43dd88d95bcda5038597 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:08.626988 11797 leader_election.cc:290] T 836251ebc2b74f5da88b8a8b2338e2a5 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:08.545511 11639 raft_consensus.cc:515] T 87cd8f5097624c8692f1d83544d8881e P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:08.652064 11149 raft_consensus.cc:740] T 30f628e3d7914d609e7f1d980a74c654 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 873357742d5843579556ee8166f23f47, State: Initialized, Role: FOLLOWER
I20260709 13:04:08.594631 11099 raft_consensus.cc:2468] T 401a9cf11eac479aa445ec8c1d6e33b2 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 1.
I20260709 13:04:08.606024 10948 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5299b8d044f41b7b68057e63d17b159" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:08.639039 11849 raft_consensus.cc:493] T acc241e7273d49149984968596242712 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:08.659528 10950 raft_consensus.cc:1275] T e89e640169e14e2d98f968f76027daa7 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Refusing update from remote peer 873357742d5843579556ee8166f23f47: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 13:04:08.661700 11020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d71ca3a1ba6b48478aaf62741bf3cf6a" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:08.669011 10949 raft_consensus.cc:2393] T 1e4f2241a94a41f38e481137f868b83d P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 873357742d5843579556ee8166f23f47 in current term 2: Already voted for candidate 10ca5ee68ad64b3283b9d235352c9d23 in this term.
I20260709 13:04:08.676102 11149 consensus_queue.cc:260] T 30f628e3d7914d609e7f1d980a74c654 P 873357742d5843579556ee8166f23f47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:08.630856 11896 raft_consensus.cc:515] T 2567c2d84e4648d7ba5dc23a3df32817 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
W20260709 13:04:08.673736 10918 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:08.687924 10955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2567c2d84e4648d7ba5dc23a3df32817" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:08.669891 11943 raft_consensus.cc:515] T 6a9353d1d230416296627d7cf8b1d425 P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:08.699741 10920 leader_election.cc:304] T c31881534c5e4f0089c82057c1746652 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 10ca5ee68ad64b3283b9d235352c9d23; no voters: 186b12638a87410487467df721a3cbda
I20260709 13:04:08.691967 11961 raft_consensus.cc:2804] T 75492f86fde046c3900df3eac8292fd4 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:04:08.711108 11896 leader_election.cc:290] T 2567c2d84e4648d7ba5dc23a3df32817 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:08.713737 11553 raft_consensus.cc:2804] T c31881534c5e4f0089c82057c1746652 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:04:08.703449 11024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7709387f4d3d4f9992d97a0fbca39a40" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:08.714519 11553 raft_consensus.cc:697] T c31881534c5e4f0089c82057c1746652 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 LEADER]: Becoming Leader. State: Replica: 10ca5ee68ad64b3283b9d235352c9d23, State: Running, Role: LEADER
I20260709 13:04:08.694202 10952 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63d44f6ba6434ab2870fe3190f4d7c2f" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:08.689761 11962 consensus_queue.cc:1059] T d270b9e5481245d9b5340a454a7cd10b P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.018s
I20260709 13:04:08.683869 11849 raft_consensus.cc:515] T acc241e7273d49149984968596242712 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:08.731276 11101 raft_consensus.cc:2368] T 6a9353d1d230416296627d7cf8b1d425 P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 873357742d5843579556ee8166f23f47 for earlier term 3. Current term is 4.
I20260709 13:04:08.684774 11959 raft_consensus.cc:515] T 5d439326f23747eaab3c1e7da2d6b785 P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:08.732338 10953 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d71ca3a1ba6b48478aaf62741bf3cf6a" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:08.733486 11628 raft_consensus.cc:493] T d7ce47b9a92d43dd88d95bcda5038597 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 10ca5ee68ad64b3283b9d235352c9d23)
I20260709 13:04:08.733736 11023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1ec2b50c6ad4f7c8033476208abb26d" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
I20260709 13:04:08.736805 11849 leader_election.cc:290] T acc241e7273d49149984968596242712 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:08.738998 11149 ts_tablet_manager.cc:1434] T 30f628e3d7914d609e7f1d980a74c654 P 873357742d5843579556ee8166f23f47: Time spent starting tablet: real 0.318s	user 0.004s	sys 0.008s
I20260709 13:04:08.752238 11936 consensus_queue.cc:1048] T ff13ae9a47c7455f962406c6d131a3ff P 873357742d5843579556ee8166f23f47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:04:08.752875 10992 leader_election.cc:304] T 65bf0a62f4d84ef18dd6b23e082724b8 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 873357742d5843579556ee8166f23f47; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda
I20260709 13:04:08.739948 10871 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e96e80ba9ac34d0998395515e5c8c41a" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:08.758657 11972 consensus_queue.cc:1048] T a64b7e99d6c54934a7de6f6dffcf5e94 P 873357742d5843579556ee8166f23f47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:04:08.715201 11024 raft_consensus.cc:2468] T 7709387f4d3d4f9992d97a0fbca39a40 P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 4.
I20260709 13:04:08.741873 10955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4cd3d7dfbbf4096908ff11e9dc57141" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:08.758317 11936 raft_consensus.cc:2749] T 65bf0a62f4d84ef18dd6b23e082724b8 P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:04:08.761003 10955 raft_consensus.cc:2468] T b4cd3d7dfbbf4096908ff11e9dc57141 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 3.
I20260709 13:04:08.762243 10995 leader_election.cc:304] T 6a9353d1d230416296627d7cf8b1d425 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:04:08.763839 10995 leader_election.cc:304] T d371c5e27bd843469d2e2e453734cb4a P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:04:08.733938 11553 consensus_queue.cc:237] T c31881534c5e4f0089c82057c1746652 P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:08.764956 11804 raft_consensus.cc:2804] T 6a9353d1d230416296627d7cf8b1d425 P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:04:08.765574 11804 raft_consensus.cc:697] T 6a9353d1d230416296627d7cf8b1d425 P 873357742d5843579556ee8166f23f47 [term 3 LEADER]: Becoming Leader. State: Replica: 873357742d5843579556ee8166f23f47, State: Running, Role: LEADER
I20260709 13:04:08.766635 11926 raft_consensus.cc:493] T 525bfdbd6d934afd823f7d3ee3f9d052 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 10ca5ee68ad64b3283b9d235352c9d23)
I20260709 13:04:08.767328 11926 raft_consensus.cc:515] T 525bfdbd6d934afd823f7d3ee3f9d052 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:08.768240 11673 raft_consensus.cc:2804] T d371c5e27bd843469d2e2e453734cb4a P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:04:08.769099 10995 leader_election.cc:304] T a610d1a71f52432e88fe56130f921fa2 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:04:08.744987 10953 raft_consensus.cc:2468] T d71ca3a1ba6b48478aaf62741bf3cf6a P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 2.
I20260709 13:04:08.750795 11943 leader_election.cc:290] T 6a9353d1d230416296627d7cf8b1d425 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 4 election: Requested vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:08.751108 11023 raft_consensus.cc:3060] T a1ec2b50c6ad4f7c8033476208abb26d P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:04:08.750818 11103 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4cd3d7dfbbf4096908ff11e9dc57141" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:08.742359 11937 raft_consensus.cc:515] T d371c5e27bd843469d2e2e453734cb4a P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
W20260709 13:04:08.783783 10920 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 88064us
W20260709 13:04:08.822526 10992 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 52740.1us
I20260709 13:04:08.775002 11673 raft_consensus.cc:493] T d371c5e27bd843469d2e2e453734cb4a P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:08.783313 11849 raft_consensus.cc:493] T b67eff0634e540eab91cf8cee795969f P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:08.774811 10872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f34f2bc02d85444f9096e0f5b4fe4331" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:08.798278 11972 raft_consensus.cc:2804] T a610d1a71f52432e88fe56130f921fa2 P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260709 13:04:08.846029 10844 leader_election.cc:343] T 30f628e3d7914d609e7f1d980a74c654 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 873357742d5843579556ee8166f23f47 (127.0.107.195:45807): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:04:08.832195 11953 raft_consensus.cc:493] T 7757ecbf402e47bfbc93d0447741e26c P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:08.780138 11981 raft_consensus.cc:515] T 63d44f6ba6434ab2870fe3190f4d7c2f P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:08.853091 10880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f371d2565dd84667b5613848604949bb" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:04:08.850713 11097 raft_consensus.cc:2393] T d371c5e27bd843469d2e2e453734cb4a P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 873357742d5843579556ee8166f23f47 in current term 3: Already voted for candidate 0172fed22b6b4890a7fdc953fa128af4 in this term.
I20260709 13:04:08.837451 11673 raft_consensus.cc:3060] T d371c5e27bd843469d2e2e453734cb4a P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:04:08.830327 11628 raft_consensus.cc:515] T d7ce47b9a92d43dd88d95bcda5038597 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:08.877578 11981 leader_election.cc:290] T 63d44f6ba6434ab2870fe3190f4d7c2f P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 3 election: Requested vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:08.878300 11028 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7839ca2d2c13431a85f44d7c458c6351" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:08.847918 11972 raft_consensus.cc:493] T a610d1a71f52432e88fe56130f921fa2 P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 0172fed22b6b4890a7fdc953fa128af4)
I20260709 13:04:08.878950 11029 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "732c4adeeff6422497aa7db4a82f535b" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:08.847404 11896 raft_consensus.cc:493] T 65bf0a62f4d84ef18dd6b23e082724b8 P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 10ca5ee68ad64b3283b9d235352c9d23)
I20260709 13:04:08.879817 11028 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7757ecbf402e47bfbc93d0447741e26c" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:08.834291 11852 consensus_queue.cc:1048] T 868e92339c2f4508af0afbd07c814d64 P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.025s
W20260709 13:04:08.878417 11774 consensus_meta.cc:287] T 732c4adeeff6422497aa7db4a82f535b P 873357742d5843579556ee8166f23f47: Time spent flushing consensus metadata: real 0.504s	user 0.017s	sys 0.000s
I20260709 13:04:08.890909 11029 raft_consensus.cc:2468] T 732c4adeeff6422497aa7db4a82f535b P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 2.
I20260709 13:04:08.890650 11067 leader_election.cc:304] T 7709387f4d3d4f9992d97a0fbca39a40 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:04:08.853557 10872 raft_consensus.cc:2468] T f34f2bc02d85444f9096e0f5b4fe4331 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 0.
W20260709 13:04:08.878988 11956 log.cc:927] Time spent T a610d1a71f52432e88fe56130f921fa2 P 873357742d5843579556ee8166f23f47: Append to log took a long time: real 0.057s	user 0.000s	sys 0.000s
I20260709 13:04:08.888410 10843 leader_election.cc:304] T 11de93cd9b9d457489ff1cb80f976c9a P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 186b12638a87410487467df721a3cbda; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47
I20260709 13:04:08.868646 11962 raft_consensus.cc:3055] T d270b9e5481245d9b5340a454a7cd10b P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:04:08.857369 11887 raft_consensus.cc:493] T d371c5e27bd843469d2e2e453734cb4a P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:08.854097 11842 consensus_queue.cc:1048] T b4cd3d7dfbbf4096908ff11e9dc57141 P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.034s
I20260709 13:04:08.880169 11020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e4f2241a94a41f38e481137f868b83d" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:08.884876 11972 raft_consensus.cc:3060] T a610d1a71f52432e88fe56130f921fa2 P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:04:08.880949 11926 leader_election.cc:290] T 525bfdbd6d934afd823f7d3ee3f9d052 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
W20260709 13:04:08.907158 10921 leader_election.cc:336] T 57b6a2418c1342e7924195e9c1dbd357 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 186b12638a87410487467df721a3cbda (127.0.107.193:38967): Timed out: RequestConsensusVote RPC to 127.0.107.193:38967 timed out after 1.705s (SENT)
I20260709 13:04:08.853991 11849 raft_consensus.cc:515] T b67eff0634e540eab91cf8cee795969f P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:08.832532 11804 consensus_queue.cc:237] T 6a9353d1d230416296627d7cf8b1d425 P 873357742d5843579556ee8166f23f47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:08.888456 11639 leader_election.cc:290] T 87cd8f5097624c8692f1d83544d8881e P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:08.922971 10921 leader_election.cc:304] T 57b6a2418c1342e7924195e9c1dbd357 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23; no voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47
W20260709 13:04:08.926723 11994 log.cc:927] Time spent T a610d1a71f52432e88fe56130f921fa2 P 0172fed22b6b4890a7fdc953fa128af4: Append to log took a long time: real 0.057s	user 0.000s	sys 0.001s
I20260709 13:04:08.927109 11849 leader_election.cc:290] T b67eff0634e540eab91cf8cee795969f P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:08.928120 11849 raft_consensus.cc:2804] T 7709387f4d3d4f9992d97a0fbca39a40 P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:04:08.928294 10954 raft_consensus.cc:2468] T a977842ba4b744f4bb2febd55d286006 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 1.
I20260709 13:04:08.928588 11849 raft_consensus.cc:493] T 7709387f4d3d4f9992d97a0fbca39a40 P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:08.929092 11849 raft_consensus.cc:3060] T 7709387f4d3d4f9992d97a0fbca39a40 P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:04:08.907491 11688 raft_consensus.cc:493] T b67eff0634e540eab91cf8cee795969f P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:08.910180 11022 raft_consensus.cc:1240] T f371d2565dd84667b5613848604949bb P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Rejecting Update request from peer 186b12638a87410487467df721a3cbda for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260709 13:04:08.880975 11921 consensus_queue.cc:1048] T e89e640169e14e2d98f968f76027daa7 P 873357742d5843579556ee8166f23f47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.035s
I20260709 13:04:08.874200 11943 consensus_queue.cc:1048] T 7757ecbf402e47bfbc93d0447741e26c P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.043s
I20260709 13:04:08.873315 11748 raft_consensus.cc:515] T a5299b8d044f41b7b68057e63d17b159 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:08.873979 11953 raft_consensus.cc:515] T 7757ecbf402e47bfbc93d0447741e26c P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:08.911854 11887 raft_consensus.cc:3060] T d371c5e27bd843469d2e2e453734cb4a P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:04:08.908066 11962 raft_consensus.cc:740] T d270b9e5481245d9b5340a454a7cd10b P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 10ca5ee68ad64b3283b9d235352c9d23, State: Running, Role: LEADER
I20260709 13:04:08.923296 10952 raft_consensus.cc:2393] T 63d44f6ba6434ab2870fe3190f4d7c2f P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 873357742d5843579556ee8166f23f47 in current term 3: Already voted for candidate 10ca5ee68ad64b3283b9d235352c9d23 in this term.
I20260709 13:04:08.898576 11896 raft_consensus.cc:515] T 65bf0a62f4d84ef18dd6b23e082724b8 P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:08.926178 11026 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2567c2d84e4648d7ba5dc23a3df32817" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:08.911067 11024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "868e92339c2f4508af0afbd07c814d64" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:08.905112 11797 consensus_queue.cc:1059] T e89e640169e14e2d98f968f76027daa7 P 186b12638a87410487467df721a3cbda [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.033s
W20260709 13:04:08.965046 10843 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 83635.5us
I20260709 13:04:08.966337 11028 raft_consensus.cc:2468] T 7757ecbf402e47bfbc93d0447741e26c P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 2.
I20260709 13:04:08.922844 11020 raft_consensus.cc:2468] T 1e4f2241a94a41f38e481137f868b83d P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 2.
I20260709 13:04:08.942081 11796 raft_consensus.cc:2749] T 11de93cd9b9d457489ff1cb80f976c9a P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20260709 13:04:08.972581 10994 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:08.939476 11960 consensus_queue.cc:1048] T 401a9cf11eac479aa445ec8c1d6e33b2 P 186b12638a87410487467df721a3cbda [LEADER]: Connected to new peer: Peer: permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.032s
I20260709 13:04:08.948652 11639 raft_consensus.cc:2749] T 57b6a2418c1342e7924195e9c1dbd357 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260709 13:04:08.976665 11067 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:45807, user_credentials={real_user=slave}} blocked reactor thread for 86291us
W20260709 13:04:08.977463 10992 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 97361.7us
I20260709 13:04:08.936302 11926 consensus_queue.cc:1059] T 0e5d327630e5438b878035d28efe6c0c P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 0, Time since last communication: 0.039s
I20260709 13:04:08.971065 11937 leader_election.cc:290] T d371c5e27bd843469d2e2e453734cb4a P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 3 election: Requested vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:08.971321 11959 leader_election.cc:290] T 5d439326f23747eaab3c1e7da2d6b785 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 4 election: Requested vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:08.942085 11688 raft_consensus.cc:515] T b67eff0634e540eab91cf8cee795969f P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
W20260709 13:04:08.987637 10921 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:38967, user_credentials={real_user=slave}} blocked reactor thread for 204347us
I20260709 13:04:09.009043 10948 raft_consensus.cc:2393] T a5299b8d044f41b7b68057e63d17b159 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 873357742d5843579556ee8166f23f47 in current term 2: Already voted for candidate 10ca5ee68ad64b3283b9d235352c9d23 in this term.
I20260709 13:04:08.997782 11963 consensus_queue.cc:1048] T a5299b8d044f41b7b68057e63d17b159 P 186b12638a87410487467df721a3cbda [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:04:09.037469 11025 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a9353d1d230416296627d7cf8b1d425" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
I20260709 13:04:08.980398 11797 consensus_queue.cc:1243] T e89e640169e14e2d98f968f76027daa7 P 186b12638a87410487467df721a3cbda [LEADER]: Peer 873357742d5843579556ee8166f23f47 log is divergent from this leader: its last log entry 4.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260709 13:04:09.038976 11025 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acc241e7273d49149984968596242712" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
W20260709 13:04:09.010720 11069 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:38967, user_credentials={real_user=slave}} blocked reactor thread for 159112us
I20260709 13:04:08.996855 10843 leader_election.cc:304] T 30f628e3d7914d609e7f1d980a74c654 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda; no voters: 873357742d5843579556ee8166f23f47
I20260709 13:04:09.022764 11879 raft_consensus.cc:493] T 34064dc7bf3747dbb3876dd824ee0688 P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:09.026695 12000 log.cc:927] Time spent T a154cc8b181f460abf4880064379948b P 10ca5ee68ad64b3283b9d235352c9d23: Append to log took a long time: real 0.106s	user 0.000s	sys 0.001s
I20260709 13:04:09.027750 11953 leader_election.cc:290] T 7757ecbf402e47bfbc93d0447741e26c P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:09.028508 11024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63d44f6ba6434ab2870fe3190f4d7c2f" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
I20260709 13:04:09.029094 11748 leader_election.cc:290] T a5299b8d044f41b7b68057e63d17b159 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 2 election: Requested vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:09.033114 11688 leader_election.cc:290] T b67eff0634e540eab91cf8cee795969f P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:09.008674 11962 consensus_queue.cc:260] T d270b9e5481245d9b5340a454a7cd10b P 10ca5ee68ad64b3283b9d235352c9d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:09.034255 11896 leader_election.cc:290] T 65bf0a62f4d84ef18dd6b23e082724b8 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:09.040707 12003 raft_consensus.cc:3055] T 0e5d327630e5438b878035d28efe6c0c P 0172fed22b6b4890a7fdc953fa128af4 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:04:09.060806 11797 raft_consensus.cc:2804] T 30f628e3d7914d609e7f1d980a74c654 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:04:09.043270 11782 consensus_queue.cc:1059] T e89e640169e14e2d98f968f76027daa7 P 186b12638a87410487467df721a3cbda [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.049s
W20260709 13:04:09.099174 10992 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 110571us
W20260709 13:04:09.097491 10843 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 100768us
W20260709 13:04:09.100628 10992 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:09.101161 10992 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:09.086767 12003 raft_consensus.cc:740] T 0e5d327630e5438b878035d28efe6c0c P 0172fed22b6b4890a7fdc953fa128af4 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 0172fed22b6b4890a7fdc953fa128af4, State: Running, Role: LEADER
I20260709 13:04:09.044859 11879 raft_consensus.cc:515] T 34064dc7bf3747dbb3876dd824ee0688 P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
W20260709 13:04:09.105928 10921 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:38967, user_credentials={real_user=slave}} blocked reactor thread for 71502us
I20260709 13:04:09.091459 10954 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e4f2241a94a41f38e481137f868b83d" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:09.096550 11797 raft_consensus.cc:493] T 30f628e3d7914d609e7f1d980a74c654 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:09.095803 11022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57b6a2418c1342e7924195e9c1dbd357" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:09.099257 11545 leader_election.cc:290] T c31881534c5e4f0089c82057c1746652 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:09.134049 11024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d371c5e27bd843469d2e2e453734cb4a" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
I20260709 13:04:09.136509 10946 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c2a35bb99374e93b573cf870d04faaa" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:09.137337 11673 raft_consensus.cc:515] T d371c5e27bd843469d2e2e453734cb4a P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:09.138686 11024 raft_consensus.cc:2393] T d371c5e27bd843469d2e2e453734cb4a P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 0172fed22b6b4890a7fdc953fa128af4 in current term 3: Already voted for candidate 873357742d5843579556ee8166f23f47 in this term.
I20260709 13:04:09.139266 12003 consensus_queue.cc:260] T 0e5d327630e5438b878035d28efe6c0c P 0172fed22b6b4890a7fdc953fa128af4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:09.108968 11953 raft_consensus.cc:493] T 30f628e3d7914d609e7f1d980a74c654 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:09.141188 11953 raft_consensus.cc:515] T 30f628e3d7914d609e7f1d980a74c654 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:09.141397 12003 raft_consensus.cc:3060] T 0e5d327630e5438b878035d28efe6c0c P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:04:09.142712 11023 raft_consensus.cc:2468] T a1ec2b50c6ad4f7c8033476208abb26d P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 1.
I20260709 13:04:09.119678 11069 leader_election.cc:304] T 4f9eb06ed01f4ce68be3af8a8434665b P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4; no voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47
I20260709 13:04:09.144905 11069 leader_election.cc:304] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda; no voters: 
I20260709 13:04:09.146076 11673 leader_election.cc:290] T d371c5e27bd843469d2e2e453734cb4a P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 3 election: Requested vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:09.146759 12004 raft_consensus.cc:2749] T 4f9eb06ed01f4ce68be3af8a8434665b P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:04:09.147616 12004 raft_consensus.cc:2804] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:04:09.147643 11067 leader_election.cc:304] T a1ec2b50c6ad4f7c8033476208abb26d P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:04:09.148125 12004 raft_consensus.cc:493] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:09.148337 11879 leader_election.cc:290] T 34064dc7bf3747dbb3876dd824ee0688 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:09.148475 12004 raft_consensus.cc:3060] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:04:09.149672 11023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5299b8d044f41b7b68057e63d17b159" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
I20260709 13:04:09.150831 12007 raft_consensus.cc:2804] T a1ec2b50c6ad4f7c8033476208abb26d P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Leader election won for term 1
W20260709 13:04:09.151350 10995 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:09.155948 11022 raft_consensus.cc:2468] T 57b6a2418c1342e7924195e9c1dbd357 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 2.
I20260709 13:04:09.156260 11553 consensus_queue.cc:1048] T 72b111ad7b5f4753ab1d2b5ee1114840 P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:04:09.158663 10776 catalog_manager.cc:5697] T e5a5f5a608504650b48fd27f163faed6 P 0172fed22b6b4890a7fdc953fa128af4 reported cstate change: term changed from 1 to 2, leader changed from 186b12638a87410487467df721a3cbda (127.0.107.193) to 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194). New cstate: current_term: 2 leader_uuid: "10ca5ee68ad64b3283b9d235352c9d23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } }
I20260709 13:04:09.160844 11023 raft_consensus.cc:1275] T 8c160ddfe9b34bdaa3835d0915c1e06f P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Refusing update from remote peer 186b12638a87410487467df721a3cbda: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:04:09.161283 10955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57b6a2418c1342e7924195e9c1dbd357" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:09.161078 10776 catalog_manager.cc:5697] T 72b111ad7b5f4753ab1d2b5ee1114840 P 0172fed22b6b4890a7fdc953fa128af4 reported cstate change: term changed from 1 to 2, leader changed from 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194) to 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196). New cstate: current_term: 2 leader_uuid: "0172fed22b6b4890a7fdc953fa128af4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } health_report { overall_health: HEALTHY } } }
I20260709 13:04:09.146011 11069 leader_election.cc:304] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4; no voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47
W20260709 13:04:09.170781 10992 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 43937.4us
I20260709 13:04:09.110600 10951 raft_consensus.cc:1275] T 8a69f32c53f24a50b7852d143665d7b4 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Refusing update from remote peer 186b12638a87410487467df721a3cbda: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 13:04:09.129560 11797 raft_consensus.cc:3060] T 30f628e3d7914d609e7f1d980a74c654 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:04:09.110286 11025 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "525bfdbd6d934afd823f7d3ee3f9d052" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:09.157712 11953 leader_election.cc:290] T 30f628e3d7914d609e7f1d980a74c654 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:09.165390 10946 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34064dc7bf3747dbb3876dd824ee0688" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:04:09.194916 10776 catalog_manager.cc:5697] T d371c5e27bd843469d2e2e453734cb4a P 0172fed22b6b4890a7fdc953fa128af4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196). New cstate: current_term: 1 leader_uuid: "0172fed22b6b4890a7fdc953fa128af4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } health_report { overall_health: UNKNOWN } } }
I20260709 13:04:09.235703 10776 catalog_manager.cc:5697] T 5de5f0581e3b41dea729363d6ca563bb P 0172fed22b6b4890a7fdc953fa128af4 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "0172fed22b6b4890a7fdc953fa128af4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } health_report { overall_health: UNKNOWN } } }
I20260709 13:04:09.177434 11972 raft_consensus.cc:515] T a610d1a71f52432e88fe56130f921fa2 P 873357742d5843579556ee8166f23f47 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:09.244266 10921 leader_election.cc:304] T 8a69f32c53f24a50b7852d143665d7b4 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23; no voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda
I20260709 13:04:09.250628 11972 leader_election.cc:290] T a610d1a71f52432e88fe56130f921fa2 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 5 election: Requested vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:09.253965 10954 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d270b9e5481245d9b5340a454a7cd10b" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:04:09.256326 11926 raft_consensus.cc:3060] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:04:09.260795 11804 raft_consensus.cc:493] T 11de93cd9b9d457489ff1cb80f976c9a P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:09.261452 11804 raft_consensus.cc:515] T 11de93cd9b9d457489ff1cb80f976c9a P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:09.264895 10948 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "836251ebc2b74f5da88b8a8b2338e2a5" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:09.265352 11804 leader_election.cc:290] T 11de93cd9b9d457489ff1cb80f976c9a P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:09.267944 11961 raft_consensus.cc:493] T 75492f86fde046c3900df3eac8292fd4 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:09.268748 11961 raft_consensus.cc:3060] T 75492f86fde046c3900df3eac8292fd4 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:04:09.269752 10951 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5ba72ccd854430baadbc636dfd17136" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:09.274609 11628 leader_election.cc:290] T d7ce47b9a92d43dd88d95bcda5038597 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:09.271330 10951 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7839ca2d2c13431a85f44d7c458c6351" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:09.276618 10951 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f9df2bc281f4866b6da5da94a0defc7" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:09.277256 10951 raft_consensus.cc:2468] T 0f9df2bc281f4866b6da5da94a0defc7 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 1.
I20260709 13:04:09.278184 10951 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32147c6d2d5a477aa38c597c7f78df5b" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:09.279385 10951 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff13ae9a47c7455f962406c6d131a3ff" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:09.279976 10951 raft_consensus.cc:2468] T ff13ae9a47c7455f962406c6d131a3ff P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 2.
I20260709 13:04:09.281543 12004 raft_consensus.cc:515] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:09.282789 12009 raft_consensus.cc:2749] T 8a69f32c53f24a50b7852d143665d7b4 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:04:09.284107 10921 leader_election.cc:304] T 30f628e3d7914d609e7f1d980a74c654 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda; no voters: 873357742d5843579556ee8166f23f47
I20260709 13:04:09.285068 11639 raft_consensus.cc:2804] T 30f628e3d7914d609e7f1d980a74c654 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:04:09.284940 11748 consensus_queue.cc:1048] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:04:09.285547 11639 raft_consensus.cc:493] T 30f628e3d7914d609e7f1d980a74c654 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:09.285954 11639 raft_consensus.cc:3060] T 30f628e3d7914d609e7f1d980a74c654 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:04:09.286849 10955 raft_consensus.cc:2468] T 57b6a2418c1342e7924195e9c1dbd357 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 2.
I20260709 13:04:09.288619 11983 consensus_queue.cc:1048] T 5d439326f23747eaab3c1e7da2d6b785 P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260709 13:04:09.289729 11096 consensus_meta.cc:287] T f371d2565dd84667b5613848604949bb P 0172fed22b6b4890a7fdc953fa128af4: Time spent flushing consensus metadata: real 0.657s	user 0.006s	sys 0.011s
I20260709 13:04:09.288652 10955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c31881534c5e4f0089c82057c1746652" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:09.291530 10951 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b67eff0634e540eab91cf8cee795969f" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:09.293324 10951 raft_consensus.cc:1275] T b4cd3d7dfbbf4096908ff11e9dc57141 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Refusing update from remote peer 0172fed22b6b4890a7fdc953fa128af4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 13:04:09.245124 10946 raft_consensus.cc:2376] T 34064dc7bf3747dbb3876dd824ee0688 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 186b12638a87410487467df721a3cbda in term 4. Re-sending same reply.
I20260709 13:04:09.307823 10871 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f9eb06ed01f4ce68be3af8a8434665b" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:09.279778 10950 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d439326f23747eaab3c1e7da2d6b785" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:09.245360 12007 raft_consensus.cc:697] T a1ec2b50c6ad4f7c8033476208abb26d P 0172fed22b6b4890a7fdc953fa128af4 [term 1 LEADER]: Becoming Leader. State: Replica: 0172fed22b6b4890a7fdc953fa128af4, State: Running, Role: LEADER
I20260709 13:04:09.312384 12007 consensus_queue.cc:237] T a1ec2b50c6ad4f7c8033476208abb26d P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:09.306093 11066 leader_election.cc:304] T c51782cd9d064ec0949e84e24dd4de67 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 10ca5ee68ad64b3283b9d235352c9d23; no voters: 186b12638a87410487467df721a3cbda
I20260709 13:04:09.315281 10921 leader_election.cc:304] T e96e80ba9ac34d0998395515e5c8c41a P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23; no voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47
I20260709 13:04:09.320868 10921 leader_election.cc:304] T f34f2bc02d85444f9096e0f5b4fe4331 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda; no voters: 
I20260709 13:04:09.324038 10992 leader_election.cc:304] T b4cd3d7dfbbf4096908ff11e9dc57141 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:04:09.290282 11096 raft_consensus.cc:2468] T f371d2565dd84667b5613848604949bb P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 3.
I20260709 13:04:09.336374 10992 leader_election.cc:304] T d71ca3a1ba6b48478aaf62741bf3cf6a P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:04:09.358150 11926 raft_consensus.cc:2749] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:04:09.360638 11887 raft_consensus.cc:515] T d371c5e27bd843469d2e2e453734cb4a P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:09.364380 11879 raft_consensus.cc:2804] T d71ca3a1ba6b48478aaf62741bf3cf6a P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:04:09.364903 11981 raft_consensus.cc:493] T 34064dc7bf3747dbb3876dd824ee0688 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:09.365516 11981 raft_consensus.cc:515] T 34064dc7bf3747dbb3876dd824ee0688 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:09.368494 11981 leader_election.cc:290] T 34064dc7bf3747dbb3876dd824ee0688 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:09.369055 11028 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34064dc7bf3747dbb3876dd824ee0688" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:09.370007 11028 raft_consensus.cc:2410] T 34064dc7bf3747dbb3876dd824ee0688 P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 10ca5ee68ad64b3283b9d235352c9d23 for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 13:04:09.371308 12004 leader_election.cc:290] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 3 election: Requested vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:09.335227 11983 raft_consensus.cc:2804] T f34f2bc02d85444f9096e0f5b4fe4331 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:04:09.362546 11748 raft_consensus.cc:2749] T e96e80ba9ac34d0998395515e5c8c41a P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:04:09.363083 12019 raft_consensus.cc:2804] T c51782cd9d064ec0949e84e24dd4de67 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260709 13:04:09.369422 11066 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 63487.7us
W20260709 13:04:09.374903 10921 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:09.377897 10992 leader_election.cc:304] T a5299b8d044f41b7b68057e63d17b159 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 873357742d5843579556ee8166f23f47; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda
I20260709 13:04:09.367941 11887 leader_election.cc:290] T d371c5e27bd843469d2e2e453734cb4a P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 3 election: Requested vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:09.327344 11553 consensus_queue.cc:1048] T b67eff0634e540eab91cf8cee795969f P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.047s
W20260709 13:04:09.386607 11069 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:38967, user_credentials={real_user=slave}} blocked reactor thread for 240749us
I20260709 13:04:09.387995 11936 raft_consensus.cc:3060] T a5299b8d044f41b7b68057e63d17b159 P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:04:09.398738 10955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34064dc7bf3747dbb3876dd824ee0688" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:09.388891 10951 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a977842ba4b744f4bb2febd55d286006" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:04:09.389992 11972 raft_consensus.cc:2804] T b4cd3d7dfbbf4096908ff11e9dc57141 P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:04:09.384868 12019 raft_consensus.cc:493] T c51782cd9d064ec0949e84e24dd4de67 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 186b12638a87410487467df721a3cbda)
I20260709 13:04:09.376699 11983 raft_consensus.cc:493] T f34f2bc02d85444f9096e0f5b4fe4331 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:09.399646 10955 raft_consensus.cc:2468] T 34064dc7bf3747dbb3876dd824ee0688 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 4.
I20260709 13:04:09.374332 11879 raft_consensus.cc:493] T d71ca3a1ba6b48478aaf62741bf3cf6a P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 0172fed22b6b4890a7fdc953fa128af4)
I20260709 13:04:09.410856 11066 leader_election.cc:304] T 5d439326f23747eaab3c1e7da2d6b785 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda
I20260709 13:04:09.412282 11879 raft_consensus.cc:3060] T d71ca3a1ba6b48478aaf62741bf3cf6a P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:04:09.420011 10955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11de93cd9b9d457489ff1cb80f976c9a" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:09.399238 11849 raft_consensus.cc:515] T 7709387f4d3d4f9992d97a0fbca39a40 P 0172fed22b6b4890a7fdc953fa128af4 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:09.424296 10873 raft_consensus.cc:3055] T e89e640169e14e2d98f968f76027daa7 P 186b12638a87410487467df721a3cbda [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:04:09.424750 10873 raft_consensus.cc:740] T e89e640169e14e2d98f968f76027daa7 P 186b12638a87410487467df721a3cbda [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 186b12638a87410487467df721a3cbda, State: Running, Role: LEADER
I20260709 13:04:09.425034 11849 leader_election.cc:290] T 7709387f4d3d4f9992d97a0fbca39a40 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 5 election: Requested vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:09.428753 11027 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46628 (request call id 175) took 1395 ms. Trace:
I20260709 13:04:09.429225 11027 rpcz_store.cc:276] 0709 13:04:08.032947 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:08.062829 (+ 29882us) service_pool.cc:224] Handling call
0709 13:04:09.428533 (+1365704us) spinlock_profiling.cc:243] Waited 435 ms on lock 0x7b5400035cd4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:04:09.428662 (+   129us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1001167104}
I20260709 13:04:09.408334 12019 raft_consensus.cc:3060] T c51782cd9d064ec0949e84e24dd4de67 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:04:09.434077 10919 leader_election.cc:304] T 7839ca2d2c13431a85f44d7c458c6351 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47; no voters: 186b12638a87410487467df721a3cbda
I20260709 13:04:09.435667 10946 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "30f628e3d7914d609e7f1d980a74c654" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:09.436043 11027 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7709387f4d3d4f9992d97a0fbca39a40" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
W20260709 13:04:09.437103 11995 log.cc:927] Time spent T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 10ca5ee68ad64b3283b9d235352c9d23: Append to log took a long time: real 0.593s	user 0.008s	sys 0.000s
I20260709 13:04:09.437525 11943 raft_consensus.cc:2749] T 5d439326f23747eaab3c1e7da2d6b785 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:04:09.438225 12024 raft_consensus.cc:493] T 8c160ddfe9b34bdaa3835d0915c1e06f P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:09.425691 10873 consensus_queue.cc:260] T e89e640169e14e2d98f968f76027daa7 P 186b12638a87410487467df721a3cbda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:09.438802 12024 raft_consensus.cc:515] T 8c160ddfe9b34bdaa3835d0915c1e06f P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:09.439824 10873 raft_consensus.cc:3060] T e89e640169e14e2d98f968f76027daa7 P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:04:09.410283 11983 raft_consensus.cc:3060] T f34f2bc02d85444f9096e0f5b4fe4331 P 10ca5ee68ad64b3283b9d235352c9d23 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:04:09.455554 10872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7757ecbf402e47bfbc93d0447741e26c" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:09.455838 10878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d371c5e27bd843469d2e2e453734cb4a" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:04:09.456337 10872 raft_consensus.cc:2468] T 7757ecbf402e47bfbc93d0447741e26c P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 2.
I20260709 13:04:09.457075 10871 raft_consensus.cc:1275] T 9a1716c0adf34bee8a4e99341ba5788f P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Refusing update from remote peer 873357742d5843579556ee8166f23f47: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 13:04:09.457535 10879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34064dc7bf3747dbb3876dd824ee0688" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:09.458276 10879 raft_consensus.cc:2468] T 34064dc7bf3747dbb3876dd824ee0688 P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 4.
I20260709 13:04:09.458498 10880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "30f628e3d7914d609e7f1d980a74c654" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:09.441229 12024 leader_election.cc:290] T 8c160ddfe9b34bdaa3835d0915c1e06f P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:09.460222 10879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a610d1a71f52432e88fe56130f921fa2" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:04:09.452744 10955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2567c2d84e4648d7ba5dc23a3df32817" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:09.460793 10879 raft_consensus.cc:3060] T a610d1a71f52432e88fe56130f921fa2 P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:04:09.463016 10871 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11de93cd9b9d457489ff1cb80f976c9a" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:09.454735 11974 raft_consensus.cc:2804] T 7839ca2d2c13431a85f44d7c458c6351 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:04:09.463982 10871 raft_consensus.cc:2468] T 11de93cd9b9d457489ff1cb80f976c9a P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 4.
I20260709 13:04:09.464193 11974 raft_consensus.cc:493] T 7839ca2d2c13431a85f44d7c458c6351 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:09.437247 11027 raft_consensus.cc:3060] T 7709387f4d3d4f9992d97a0fbca39a40 P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:04:09.465273 11974 raft_consensus.cc:3060] T 7839ca2d2c13431a85f44d7c458c6351 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:04:09.465700 10995 leader_election.cc:304] T 7757ecbf402e47bfbc93d0447741e26c P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:04:09.425908 11639 raft_consensus.cc:515] T 30f628e3d7914d609e7f1d980a74c654 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:09.467013 10995 leader_election.cc:304] T 34064dc7bf3747dbb3876dd824ee0688 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:04:09.467741 10946 raft_consensus.cc:2393] T 30f628e3d7914d609e7f1d980a74c654 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 873357742d5843579556ee8166f23f47 in current term 1: Already voted for candidate 10ca5ee68ad64b3283b9d235352c9d23 in this term.
I20260709 13:04:09.410593 10951 raft_consensus.cc:2376] T a977842ba4b744f4bb2febd55d286006 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 873357742d5843579556ee8166f23f47 in term 1. Re-sending same reply.
I20260709 13:04:09.467924 11804 raft_consensus.cc:2804] T 7757ecbf402e47bfbc93d0447741e26c P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:04:09.468266 11673 consensus_queue.cc:1048] T 9a1716c0adf34bee8a4e99341ba5788f P 873357742d5843579556ee8166f23f47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:04:09.468835 11639 leader_election.cc:290] T 30f628e3d7914d609e7f1d980a74c654 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 1 election: Requested vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:09.468886 11804 raft_consensus.cc:493] T 7757ecbf402e47bfbc93d0447741e26c P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 0172fed22b6b4890a7fdc953fa128af4)
I20260709 13:04:09.469587 11804 raft_consensus.cc:3060] T 7757ecbf402e47bfbc93d0447741e26c P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:04:09.469907 11920 raft_consensus.cc:493] T 8c160ddfe9b34bdaa3835d0915c1e06f P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:09.470476 11920 raft_consensus.cc:515] T 8c160ddfe9b34bdaa3835d0915c1e06f P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:09.468461 11953 raft_consensus.cc:2804] T 34064dc7bf3747dbb3876dd824ee0688 P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260709 13:04:09.462844 10992 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 51786.4us
I20260709 13:04:09.471765 10995 leader_election.cc:304] T 11de93cd9b9d457489ff1cb80f976c9a P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47; no voters: 
W20260709 13:04:09.483991 11069 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:38967, user_credentials={real_user=slave}} blocked reactor thread for 73422.6us
I20260709 13:04:09.485873 11028 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "30f628e3d7914d609e7f1d980a74c654" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
I20260709 13:04:09.486856 11028 raft_consensus.cc:3060] T 30f628e3d7914d609e7f1d980a74c654 P 873357742d5843579556ee8166f23f47 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:04:09.470477 11096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f46a5102fd64b45bf1c4dff863d2b4d" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:09.476429 11953 raft_consensus.cc:493] T 34064dc7bf3747dbb3876dd824ee0688 P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:09.499154 11097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b67eff0634e540eab91cf8cee795969f" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:09.522013 11021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c160ddfe9b34bdaa3835d0915c1e06f" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:09.524930 10879 raft_consensus.cc:2468] T a610d1a71f52432e88fe56130f921fa2 P 186b12638a87410487467df721a3cbda [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 5.
I20260709 13:04:09.502393 11879 raft_consensus.cc:515] T d71ca3a1ba6b48478aaf62741bf3cf6a P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:09.527737 11021 raft_consensus.cc:2468] T 8c160ddfe9b34bdaa3835d0915c1e06f P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 2.
I20260709 13:04:09.527185 10995 leader_election.cc:304] T a610d1a71f52432e88fe56130f921fa2 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:04:09.499667 11094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bac7e5040f984ba2b7146bd36af7267e" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:09.508621 11849 raft_consensus.cc:493] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:09.528784 11920 leader_election.cc:290] T 8c160ddfe9b34bdaa3835d0915c1e06f P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:09.529022 12024 raft_consensus.cc:493] T a64b7e99d6c54934a7de6f6dffcf5e94 P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:09.492545 11673 raft_consensus.cc:2804] T 11de93cd9b9d457489ff1cb80f976c9a P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:04:09.492326 11101 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72b111ad7b5f4753ab1d2b5ee1114840" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:04:09.534099 10879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c160ddfe9b34bdaa3835d0915c1e06f" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:09.492116 11953 raft_consensus.cc:3060] T 34064dc7bf3747dbb3876dd824ee0688 P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:04:09.528558 10877 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51704 (request call id 176) took 1293 ms. Trace:
I20260709 13:04:09.538848 11103 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f34f2bc02d85444f9096e0f5b4fe4331" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:09.546182 11103 raft_consensus.cc:2468] T f34f2bc02d85444f9096e0f5b4fe4331 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 0.
I20260709 13:04:09.547250 11103 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f1466124dbd4f6ebb3c4d62ca48bf96" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:09.497422 10873 raft_consensus.cc:1275] T e89e640169e14e2d98f968f76027daa7 P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Refusing update from remote peer 873357742d5843579556ee8166f23f47: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 3. (term mismatch)
I20260709 13:04:09.499423 11095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b67eff0634e540eab91cf8cee795969f" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:04:09.539805 11102 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d71ca3a1ba6b48478aaf62741bf3cf6a" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:09.541872 11028 raft_consensus.cc:2468] T 30f628e3d7914d609e7f1d980a74c654 P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 1.
I20260709 13:04:09.560842 10919 leader_election.cc:304] T 30f628e3d7914d609e7f1d980a74c654 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:04:09.551116 10994 leader_election.cc:304] T e5ba72ccd854430baadbc636dfd17136 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47; no voters: 10ca5ee68ad64b3283b9d235352c9d23
I20260709 13:04:09.542038 11097 raft_consensus.cc:2376] T b67eff0634e540eab91cf8cee795969f P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 1. Re-sending same reply.
I20260709 13:04:09.552642 11101 raft_consensus.cc:2376] T 72b111ad7b5f4753ab1d2b5ee1114840 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 3. Re-sending same reply.
I20260709 13:04:09.548941 10877 rpcz_store.cc:276] 0709 13:04:08.234765 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:08.349075 (+114310us) service_pool.cc:224] Handling call
0709 13:04:09.528358 (+1179283us) spinlock_profiling.cc:243] Waited 97.6 ms on lock 0x7b54003719d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:04:09.528485 (+   127us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":224466304}
I20260709 13:04:09.574024 11920 raft_consensus.cc:2804] T e5ba72ccd854430baadbc636dfd17136 P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:04:09.574747 11639 raft_consensus.cc:2804] T 30f628e3d7914d609e7f1d980a74c654 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:04:09.574831 11095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87cd8f5097624c8692f1d83544d8881e" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:09.575671 11639 raft_consensus.cc:697] T 30f628e3d7914d609e7f1d980a74c654 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 LEADER]: Becoming Leader. State: Replica: 10ca5ee68ad64b3283b9d235352c9d23, State: Running, Role: LEADER
I20260709 13:04:09.554563 10873 pending_rounds.cc:85] T e89e640169e14e2d98f968f76027daa7 P 186b12638a87410487467df721a3cbda: Aborting all ops after (but not including) 2
I20260709 13:04:09.550047 11673 raft_consensus.cc:493] T 11de93cd9b9d457489ff1cb80f976c9a P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:09.575852 11095 raft_consensus.cc:2468] T 87cd8f5097624c8692f1d83544d8881e P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 3.
I20260709 13:04:09.584228 11653 raft_consensus.cc:2804] T a610d1a71f52432e88fe56130f921fa2 P 873357742d5843579556ee8166f23f47 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:04:09.585242 11097 raft_consensus.cc:1240] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Rejecting Update request from peer 10ca5ee68ad64b3283b9d235352c9d23 for earlier term 2. Current term is 3. Ops: [2.1-2.1]
I20260709 13:04:09.585505 10873 pending_rounds.cc:107] T e89e640169e14e2d98f968f76027daa7 P 186b12638a87410487467df721a3cbda: Aborting uncommitted NO_OP operation due to leader change: 3.3
I20260709 13:04:09.538962 11100 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff13ae9a47c7455f962406c6d131a3ff" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:09.517048 11961 raft_consensus.cc:515] T 75492f86fde046c3900df3eac8292fd4 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:09.507800 12019 raft_consensus.cc:515] T c51782cd9d064ec0949e84e24dd4de67 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:09.576280 11920 raft_consensus.cc:493] T e5ba72ccd854430baadbc636dfd17136 P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 10ca5ee68ad64b3283b9d235352c9d23)
I20260709 13:04:09.546211 11849 raft_consensus.cc:515] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:09.779553 11920 raft_consensus.cc:3060] T e5ba72ccd854430baadbc636dfd17136 P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:04:09.781337 11849 leader_election.cc:290] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:09.601097 10873 raft_consensus.cc:2889] T e89e640169e14e2d98f968f76027daa7 P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:04:09.593871 11673 raft_consensus.cc:3060] T 11de93cd9b9d457489ff1cb80f976c9a P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Advancing to term 5
W20260709 13:04:09.786494 11795 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 4 current term: 4 is higher.
I20260709 13:04:09.603868 11653 raft_consensus.cc:697] T a610d1a71f52432e88fe56130f921fa2 P 873357742d5843579556ee8166f23f47 [term 5 LEADER]: Becoming Leader. State: Replica: 873357742d5843579556ee8166f23f47, State: Running, Role: LEADER
I20260709 13:04:09.553719 12024 raft_consensus.cc:515] T a64b7e99d6c54934a7de6f6dffcf5e94 P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
W20260709 13:04:09.636057 10919 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:09.681083 10992 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:09.620236 10872 raft_consensus.cc:1275] T 6a9353d1d230416296627d7cf8b1d425 P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Refusing update from remote peer 873357742d5843579556ee8166f23f47: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:04:09.598845 11553 raft_consensus.cc:493] T e5a5f5a608504650b48fd27f163faed6 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:09.755280 11804 raft_consensus.cc:515] T 7757ecbf402e47bfbc93d0447741e26c P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:09.757905 11961 leader_election.cc:290] T 75492f86fde046c3900df3eac8292fd4 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 2 election: Requested vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:09.756126 11027 raft_consensus.cc:2468] T 7709387f4d3d4f9992d97a0fbca39a40 P 873357742d5843579556ee8166f23f47 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 5.
I20260709 13:04:09.759155 11974 raft_consensus.cc:515] T 7839ca2d2c13431a85f44d7c458c6351 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
W20260709 13:04:09.760421 11797 consensus_meta.cc:287] T 30f628e3d7914d609e7f1d980a74c654 P 186b12638a87410487467df721a3cbda: Time spent flushing consensus metadata: real 0.563s	user 0.018s	sys 0.004s
I20260709 13:04:09.623476 11639 consensus_queue.cc:237] T 30f628e3d7914d609e7f1d980a74c654 P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:09.768219 11936 raft_consensus.cc:2749] T a5299b8d044f41b7b68057e63d17b159 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:04:09.771507 11879 leader_election.cc:290] T d71ca3a1ba6b48478aaf62741bf3cf6a P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 3 election: Requested vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
W20260709 13:04:09.778956 10994 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 227974us
I20260709 13:04:09.788110 11103 raft_consensus.cc:2468] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 4.
W20260709 13:04:09.793159 10992 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:09.776244 12019 leader_election.cc:290] T c51782cd9d064ec0949e84e24dd4de67 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 3 election: Requested vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:09.787637 11962 raft_consensus.cc:3060] T d270b9e5481245d9b5340a454a7cd10b P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:04:09.825985 10878 raft_consensus.cc:1275] T 7757ecbf402e47bfbc93d0447741e26c P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Refusing update from remote peer 0172fed22b6b4890a7fdc953fa128af4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:04:09.830488 11849 raft_consensus.cc:493] T bac7e5040f984ba2b7146bd36af7267e P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:09.831168 11849 raft_consensus.cc:515] T bac7e5040f984ba2b7146bd36af7267e P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:09.831954 11879 raft_consensus.cc:493] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:09.832561 11879 raft_consensus.cc:515] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:09.833587 11849 leader_election.cc:290] T bac7e5040f984ba2b7146bd36af7267e P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:09.834010 10953 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "732c4adeeff6422497aa7db4a82f535b" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:09.834450 11020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75492f86fde046c3900df3eac8292fd4" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
I20260709 13:04:09.834831 11879 leader_election.cc:290] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:09.835479 10953 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d71ca3a1ba6b48478aaf62741bf3cf6a" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:04:09.836081 10953 raft_consensus.cc:3060] T d71ca3a1ba6b48478aaf62741bf3cf6a P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:04:09.836895 12019 raft_consensus.cc:493] T 72b111ad7b5f4753ab1d2b5ee1114840 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:09.837457 12019 raft_consensus.cc:515] T 72b111ad7b5f4753ab1d2b5ee1114840 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:09.838001 11027 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65bf0a62f4d84ef18dd6b23e082724b8" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:09.840030 12019 leader_election.cc:290] T 72b111ad7b5f4753ab1d2b5ee1114840 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:09.843537 12024 leader_election.cc:290] T a64b7e99d6c54934a7de6f6dffcf5e94 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:09.816123 11953 raft_consensus.cc:515] T 34064dc7bf3747dbb3876dd824ee0688 P 873357742d5843579556ee8166f23f47 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
W20260709 13:04:09.850060 10845 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:09.849934 10875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87cd8f5097624c8692f1d83544d8881e" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:09.850955 10952 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49434 (request call id 203) took 1217 ms. Trace:
I20260709 13:04:09.851626 10875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555217d45d9b4ee492dbb3ddaf0f7c02" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:09.852434 10875 raft_consensus.cc:2468] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 3.
I20260709 13:04:09.853922 10877 raft_consensus.cc:1240] T a610d1a71f52432e88fe56130f921fa2 P 186b12638a87410487467df721a3cbda [term 5 FOLLOWER]: Rejecting Update request from peer 0172fed22b6b4890a7fdc953fa128af4 for earlier term 4. Current term is 5. Ops: [2.1-4.2]
I20260709 13:04:09.854624 11067 leader_election.cc:304] T 8c160ddfe9b34bdaa3835d0915c1e06f P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:04:09.855429 10877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d7ce47b9a92d43dd88d95bcda5038597" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:09.855895 11067 leader_election.cc:304] T 7709387f4d3d4f9992d97a0fbca39a40 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:04:09.856732 12019 raft_consensus.cc:2804] T 8c160ddfe9b34bdaa3835d0915c1e06f P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:04:09.857152 12019 raft_consensus.cc:493] T 8c160ddfe9b34bdaa3835d0915c1e06f P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:09.857539 11937 raft_consensus.cc:2804] T 7709387f4d3d4f9992d97a0fbca39a40 P 0172fed22b6b4890a7fdc953fa128af4 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:04:09.858088 11937 raft_consensus.cc:697] T 7709387f4d3d4f9992d97a0fbca39a40 P 0172fed22b6b4890a7fdc953fa128af4 [term 5 LEADER]: Becoming Leader. State: Replica: 0172fed22b6b4890a7fdc953fa128af4, State: Running, Role: LEADER
I20260709 13:04:09.859285 11804 leader_election.cc:290] T 7757ecbf402e47bfbc93d0447741e26c P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 3 election: Requested vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:09.859143 11937 consensus_queue.cc:237] T 7709387f4d3d4f9992d97a0fbca39a40 P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:09.862703 11653 consensus_queue.cc:237] T a610d1a71f52432e88fe56130f921fa2 P 873357742d5843579556ee8166f23f47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:09.807129 10873 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51718 (request call id 231) took 1028 ms. Trace:
I20260709 13:04:09.866364 10873 rpcz_store.cc:276] 0709 13:04:08.777330 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:08.797804 (+ 20474us) service_pool.cc:224] Handling call
0709 13:04:09.423924 (+626120us) spinlock_profiling.cc:243] Waited 3.41 ms on lock 0x7b5800000098. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f426303db71 00007f426303deee 0000000000375939 00007f426b0b11e0 00007f426b17a0c4 00007f426b17a131 00007f426b13bfc0 00007f426b126929 00007f426b1263d2 00007f426469dbd4
0709 13:04:09.424116 (+   192us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:09.782305 (+358189us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:09.805201 (+ 22896us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":7840128}
I20260709 13:04:09.852852 11972 raft_consensus.cc:493] T b4cd3d7dfbbf4096908ff11e9dc57141 P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 0172fed22b6b4890a7fdc953fa128af4)
I20260709 13:04:09.867811 11972 raft_consensus.cc:3060] T b4cd3d7dfbbf4096908ff11e9dc57141 P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:04:09.867903 10873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7709387f4d3d4f9992d97a0fbca39a40" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:04:09.868642 10873 raft_consensus.cc:3060] T 7709387f4d3d4f9992d97a0fbca39a40 P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:04:09.851459 10952 rpcz_store.cc:276] 0709 13:04:08.635531 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:08.680998 (+ 45467us) service_pool.cc:224] Handling call
0709 13:04:09.850732 (+1169734us) spinlock_profiling.cc:243] Waited 160 ms on lock 0x7b540040e054. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:04:09.850883 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":368591744}
I20260709 13:04:09.872392 10876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1ec2b50c6ad4f7c8033476208abb26d" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:04:09.873188 10876 raft_consensus.cc:3060] T a1ec2b50c6ad4f7c8033476208abb26d P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:04:09.876032 10879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a9353d1d230416296627d7cf8b1d425" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:04:09.876755 10879 raft_consensus.cc:3060] T 6a9353d1d230416296627d7cf8b1d425 P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:04:09.877863 11024 raft_consensus.cc:1240] T e89e640169e14e2d98f968f76027daa7 P 873357742d5843579556ee8166f23f47 [term 4 LEADER]: Rejecting Update request from peer 186b12638a87410487467df721a3cbda for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260709 13:04:09.878240 10843 leader_election.cc:304] T 4c2a35bb99374e93b573cf870d04faaa P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 186b12638a87410487467df721a3cbda; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47
I20260709 13:04:09.856621 10877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d371c5e27bd843469d2e2e453734cb4a" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:04:09.888150 10843 leader_election.cc:304] T 7839ca2d2c13431a85f44d7c458c6351 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 186b12638a87410487467df721a3cbda; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47
I20260709 13:04:09.889932 10877 raft_consensus.cc:2393] T d371c5e27bd843469d2e2e453734cb4a P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 0172fed22b6b4890a7fdc953fa128af4 in current term 3: Already voted for candidate 186b12638a87410487467df721a3cbda in this term.
I20260709 13:04:09.891196 11066 leader_election.cc:304] T 32147c6d2d5a477aa38c597c7f78df5b P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47
I20260709 13:04:09.893306 11022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d371c5e27bd843469d2e2e453734cb4a" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
I20260709 13:04:09.894191 11022 raft_consensus.cc:2393] T d371c5e27bd843469d2e2e453734cb4a P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 186b12638a87410487467df721a3cbda in current term 3: Already voted for candidate 873357742d5843579556ee8166f23f47 in this term.
I20260709 13:04:09.895100 11027 raft_consensus.cc:1240] T d71ca3a1ba6b48478aaf62741bf3cf6a P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Rejecting Update request from peer 0172fed22b6b4890a7fdc953fa128af4 for earlier term 2. Current term is 3. Ops: []
I20260709 13:04:09.895282 11022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a64b7e99d6c54934a7de6f6dffcf5e94" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:09.896153 11849 raft_consensus.cc:2749] T 32147c6d2d5a477aa38c597c7f78df5b P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:04:09.893692 11066 leader_election.cc:304] T ff13ae9a47c7455f962406c6d131a3ff P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 10ca5ee68ad64b3283b9d235352c9d23; no voters: 873357742d5843579556ee8166f23f47
I20260709 13:04:09.898276 11849 raft_consensus.cc:2804] T ff13ae9a47c7455f962406c6d131a3ff P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:04:09.898743 11849 raft_consensus.cc:493] T ff13ae9a47c7455f962406c6d131a3ff P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:09.898984 11023 raft_consensus.cc:1240] T a610d1a71f52432e88fe56130f921fa2 P 873357742d5843579556ee8166f23f47 [term 5 LEADER]: Rejecting Update request from peer 0172fed22b6b4890a7fdc953fa128af4 for earlier term 4. Current term is 5. Ops: []
I20260709 13:04:09.899111 11849 raft_consensus.cc:3060] T ff13ae9a47c7455f962406c6d131a3ff P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:04:09.899482 11028 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f1466124dbd4f6ebb3c4d62ca48bf96" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:09.808924 11553 raft_consensus.cc:515] T e5a5f5a608504650b48fd27f163faed6 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:09.900919 11026 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bac7e5040f984ba2b7146bd36af7267e" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:09.857527 10872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b67eff0634e540eab91cf8cee795969f" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:09.891978 10874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7709387f4d3d4f9992d97a0fbca39a40" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:09.906946 11066 leader_election.cc:304] T c31881534c5e4f0089c82057c1746652 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda
I20260709 13:04:09.905453 10878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acc241e7273d49149984968596242712" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:09.913128 11102 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "868e92339c2f4508af0afbd07c814d64" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:09.837942 10948 raft_consensus.cc:1240] T 75492f86fde046c3900df3eac8292fd4 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Rejecting Update request from peer 873357742d5843579556ee8166f23f47 for earlier term 1. Current term is 2. Ops: []
I20260709 13:04:09.913159 10879 raft_consensus.cc:2468] T 6a9353d1d230416296627d7cf8b1d425 P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 4.
I20260709 13:04:09.804453 11983 raft_consensus.cc:515] T f34f2bc02d85444f9096e0f5b4fe4331 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:09.928352 10955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5ba72ccd854430baadbc636dfd17136" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:09.910684 11095 raft_consensus.cc:1240] T c51782cd9d064ec0949e84e24dd4de67 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Rejecting Update request from peer 186b12638a87410487467df721a3cbda for earlier term 2. Current term is 3. Ops: []
I20260709 13:04:09.809496 11797 raft_consensus.cc:515] T 30f628e3d7914d609e7f1d980a74c654 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:09.930886 11066 leader_election.cc:304] T 836251ebc2b74f5da88b8a8b2338e2a5 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda
I20260709 13:04:09.938747 10879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d156d6ae2ca4fa59a1e387edf0fea39" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:04:09.857525 12019 raft_consensus.cc:3060] T 8c160ddfe9b34bdaa3835d0915c1e06f P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:04:09.879737 11795 raft_consensus.cc:2749] T 4c2a35bb99374e93b573cf870d04faaa P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:04:09.941083 11103 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7757ecbf402e47bfbc93d0447741e26c" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:09.852823 11953 leader_election.cc:290] T 34064dc7bf3747dbb3876dd824ee0688 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 5 election: Requested vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:09.946265 10878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7757ecbf402e47bfbc93d0447741e26c" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:04:09.961565 10878 raft_consensus.cc:3060] T 7757ecbf402e47bfbc93d0447741e26c P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:04:09.924041 11852 raft_consensus.cc:2749] T c31881534c5e4f0089c82057c1746652 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:04:09.948544 10877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34064dc7bf3747dbb3876dd824ee0688" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:04:09.962759 10877 raft_consensus.cc:3060] T 34064dc7bf3747dbb3876dd824ee0688 P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:04:09.902351 11097 raft_consensus.cc:1240] T f371d2565dd84667b5613848604949bb P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Rejecting Update request from peer 186b12638a87410487467df721a3cbda for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260709 13:04:09.953002 10879 raft_consensus.cc:3060] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:04:09.960382 10994 leader_election.cc:304] T 525bfdbd6d934afd823f7d3ee3f9d052 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 873357742d5843579556ee8166f23f47; no voters: 0172fed22b6b4890a7fdc953fa128af4, 10ca5ee68ad64b3283b9d235352c9d23
I20260709 13:04:09.965408 10955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d270b9e5481245d9b5340a454a7cd10b" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:04:09.911511 11887 consensus_queue.cc:1048] T 8a69f32c53f24a50b7852d143665d7b4 P 186b12638a87410487467df721a3cbda [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.049s
I20260709 13:04:09.970005 10880 raft_consensus.cc:2393] T 30f628e3d7914d609e7f1d980a74c654 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 873357742d5843579556ee8166f23f47 in current term 1: Already voted for candidate 186b12638a87410487467df721a3cbda in this term.
W20260709 13:04:09.970103 10843 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 82081.9us
I20260709 13:04:09.977051 11103 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "401a9cf11eac479aa445ec8c1d6e33b2" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:09.977813 11103 raft_consensus.cc:2468] T 401a9cf11eac479aa445ec8c1d6e33b2 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 1.
I20260709 13:04:09.982357 10953 raft_consensus.cc:2468] T d71ca3a1ba6b48478aaf62741bf3cf6a P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 3.
I20260709 13:04:09.985280 11102 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d71ca3a1ba6b48478aaf62741bf3cf6a" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:04:09.922986 11633 consensus_queue.cc:1048] T 6a9353d1d230416296627d7cf8b1d425 P 873357742d5843579556ee8166f23f47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.044s
I20260709 13:04:09.927846 10872 raft_consensus.cc:2468] T b67eff0634e540eab91cf8cee795969f P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 1.
I20260709 13:04:09.991024 11103 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a610d1a71f52432e88fe56130f921fa2" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:09.957464 12004 raft_consensus.cc:2749] T 836251ebc2b74f5da88b8a8b2338e2a5 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:04:09.952955 11921 raft_consensus.cc:493] T 0e5d327630e5438b878035d28efe6c0c P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 0172fed22b6b4890a7fdc953fa128af4)
I20260709 13:04:10.012691 10953 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d7ce47b9a92d43dd88d95bcda5038597" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:09.943176 12007 consensus_queue.cc:1048] T b4cd3d7dfbbf4096908ff11e9dc57141 P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.046s
I20260709 13:04:10.017041 10955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "525bfdbd6d934afd823f7d3ee3f9d052" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
W20260709 13:04:10.017419 10994 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 57134.8us
I20260709 13:04:09.953645 11926 consensus_queue.cc:1059] T d71ca3a1ba6b48478aaf62741bf3cf6a P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.047s
I20260709 13:04:09.962181 11982 consensus_queue.cc:1048] T a1ec2b50c6ad4f7c8033476208abb26d P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.052s
I20260709 13:04:09.994366 11795 raft_consensus.cc:2749] T 7839ca2d2c13431a85f44d7c458c6351 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:04:09.969110 12020 consensus_queue.cc:1059] T a610d1a71f52432e88fe56130f921fa2 P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.055s
I20260709 13:04:10.033419 10920 leader_election.cc:304] T 6f46a5102fd64b45bf1c4dff863d2b4d P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23; no voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47
W20260709 13:04:10.034301 10992 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 218825us
I20260709 13:04:10.037537 10992 leader_election.cc:304] T 2567c2d84e4648d7ba5dc23a3df32817 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 873357742d5843579556ee8166f23f47; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda
I20260709 13:04:10.048945 10955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c51782cd9d064ec0949e84e24dd4de67" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:04:09.963023 12040 consensus_queue.cc:1048] T e89e640169e14e2d98f968f76027daa7 P 873357742d5843579556ee8166f23f47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.074s
I20260709 13:04:10.048877 10879 raft_consensus.cc:2468] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 3.
I20260709 13:04:10.055431 11748 raft_consensus.cc:2749] T 6f46a5102fd64b45bf1c4dff863d2b4d P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:04:10.035745 10874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c51782cd9d064ec0949e84e24dd4de67" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:04:10.051692 11102 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a9353d1d230416296627d7cf8b1d425" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:04:10.066211 11102 raft_consensus.cc:2368] T 6a9353d1d230416296627d7cf8b1d425 P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 873357742d5843579556ee8166f23f47 for earlier term 3. Current term is 4.
I20260709 13:04:10.069756 11102 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a977842ba4b744f4bb2febd55d286006" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:04:10.070989 10874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63d44f6ba6434ab2870fe3190f4d7c2f" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:04:10.075621 11102 raft_consensus.cc:2376] T a977842ba4b744f4bb2febd55d286006 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 873357742d5843579556ee8166f23f47 in term 1. Re-sending same reply.
I20260709 13:04:10.076694 10874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5a5f5a608504650b48fd27f163faed6" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:10.076828 10872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72b111ad7b5f4753ab1d2b5ee1114840" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:10.077416 10947 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f1466124dbd4f6ebb3c4d62ca48bf96" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:10.077566 10874 raft_consensus.cc:2468] T e5a5f5a608504650b48fd27f163faed6 P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 3.
I20260709 13:04:10.077649 10872 raft_consensus.cc:2468] T 72b111ad7b5f4753ab1d2b5ee1114840 P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 3.
I20260709 13:04:10.078142 12020 consensus_queue.cc:1243] T a610d1a71f52432e88fe56130f921fa2 P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Peer 873357742d5843579556ee8166f23f47 log is divergent from this leader: its last log entry 5.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260709 13:04:10.071673 11026 raft_consensus.cc:1275] T 732c4adeeff6422497aa7db4a82f535b P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Refusing update from remote peer 10ca5ee68ad64b3283b9d235352c9d23: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:04:10.078789 10947 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d270b9e5481245d9b5340a454a7cd10b" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:10.074652 11795 consensus_queue.cc:1059] T f371d2565dd84667b5613848604949bb P 186b12638a87410487467df721a3cbda [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:04:10.080006 11795 raft_consensus.cc:3055] T f371d2565dd84667b5613848604949bb P 186b12638a87410487467df721a3cbda [term 2 LEADER]: Stepping down as leader of term 2
W20260709 13:04:10.079319 12042 log.cc:927] Time spent T e5ba72ccd854430baadbc636dfd17136 P 0172fed22b6b4890a7fdc953fa128af4: Append to log took a long time: real 0.086s	user 0.001s	sys 0.000s
I20260709 13:04:10.080154 11797 leader_election.cc:290] T 30f628e3d7914d609e7f1d980a74c654 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 1 election: Requested vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
W20260709 13:04:10.077426 12035 log.cc:927] Time spent T 9a1716c0adf34bee8a4e99341ba5788f P 186b12638a87410487467df721a3cbda: Append to log took a long time: real 0.118s	user 0.000s	sys 0.001s
W20260709 13:04:10.082643 12036 log.cc:927] Time spent T b4cd3d7dfbbf4096908ff11e9dc57141 P 0172fed22b6b4890a7fdc953fa128af4: Append to log took a long time: real 0.051s	user 0.001s	sys 0.000s
I20260709 13:04:10.082558 11100 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d270b9e5481245d9b5340a454a7cd10b" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:04:10.048429 10948 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5a5f5a608504650b48fd27f163faed6" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:04:10.083213 10871 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c160ddfe9b34bdaa3835d0915c1e06f" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:10.083735 11100 raft_consensus.cc:2393] T d270b9e5481245d9b5340a454a7cd10b P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 186b12638a87410487467df721a3cbda in current term 4: Already voted for candidate 0172fed22b6b4890a7fdc953fa128af4 in this term.
I20260709 13:04:10.084112 10948 raft_consensus.cc:2410] T e5a5f5a608504650b48fd27f163faed6 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 0172fed22b6b4890a7fdc953fa128af4 for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 13:04:10.084681 10871 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7839ca2d2c13431a85f44d7c458c6351" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:04:10.085141 10948 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d156d6ae2ca4fa59a1e387edf0fea39" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:04:10.085567 10871 raft_consensus.cc:2393] T 7839ca2d2c13431a85f44d7c458c6351 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 10ca5ee68ad64b3283b9d235352c9d23 in current term 2: Already voted for candidate 186b12638a87410487467df721a3cbda in this term.
I20260709 13:04:10.084681 11100 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d439326f23747eaab3c1e7da2d6b785" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:04:10.085364 10874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "30f628e3d7914d609e7f1d980a74c654" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:04:10.080099 10947 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b67eff0634e540eab91cf8cee795969f" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:10.078799 10872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11de93cd9b9d457489ff1cb80f976c9a" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:10.087229 10871 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34064dc7bf3747dbb3876dd824ee0688" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:10.089473 10874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5299b8d044f41b7b68057e63d17b159" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:04:10.087868 10872 raft_consensus.cc:2393] T 11de93cd9b9d457489ff1cb80f976c9a P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 10ca5ee68ad64b3283b9d235352c9d23 in current term 4: Already voted for candidate 186b12638a87410487467df721a3cbda in this term.
I20260709 13:04:10.088789 11100 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c160ddfe9b34bdaa3835d0915c1e06f" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:10.090725 11795 raft_consensus.cc:740] T f371d2565dd84667b5613848604949bb P 186b12638a87410487467df721a3cbda [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 186b12638a87410487467df721a3cbda, State: Running, Role: LEADER
I20260709 13:04:10.076753 11102 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "836251ebc2b74f5da88b8a8b2338e2a5" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:10.026552 11921 raft_consensus.cc:515] T 0e5d327630e5438b878035d28efe6c0c P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:10.099735 10955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d71ca3a1ba6b48478aaf62741bf3cf6a" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:04:10.063762 11673 raft_consensus.cc:515] T 11de93cd9b9d457489ff1cb80f976c9a P 873357742d5843579556ee8166f23f47 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:10.105528 11795 consensus_queue.cc:260] T f371d2565dd84667b5613848604949bb P 186b12638a87410487467df721a3cbda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:10.108593 11795 raft_consensus.cc:3060] T f371d2565dd84667b5613848604949bb P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Advancing to term 3
W20260709 13:04:10.103526 10920 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 70417.3us
I20260709 13:04:10.105140 11981 consensus_queue.cc:1048] T 732c4adeeff6422497aa7db4a82f535b P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.007s
I20260709 13:04:10.097244 11983 leader_election.cc:290] T f34f2bc02d85444f9096e0f5b4fe4331 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 1 election: Requested vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:10.074072 10844 leader_election.cc:304] T 1e4f2241a94a41f38e481137f868b83d P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47; no voters: 
W20260709 13:04:10.076887 12038 log.cc:927] Time spent T 836251ebc2b74f5da88b8a8b2338e2a5 P 0172fed22b6b4890a7fdc953fa128af4: Append to log took a long time: real 0.102s	user 0.000s	sys 0.000s
I20260709 13:04:10.109510 11102 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a610d1a71f52432e88fe56130f921fa2" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:04:10.100539 10879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87cd8f5097624c8692f1d83544d8881e" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:10.122618 10877 raft_consensus.cc:2468] T 34064dc7bf3747dbb3876dd824ee0688 P 186b12638a87410487467df721a3cbda [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 5.
I20260709 13:04:10.113360 10920 leader_election.cc:304] T bac7e5040f984ba2b7146bd36af7267e P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23; no voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47
I20260709 13:04:10.127698 10948 raft_consensus.cc:1240] T d71ca3a1ba6b48478aaf62741bf3cf6a P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Rejecting Update request from peer 0172fed22b6b4890a7fdc953fa128af4 for earlier term 2. Current term is 3. Ops: []
W20260709 13:04:10.124889 10844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:45807, user_credentials={real_user=slave}} blocked reactor thread for 50901.9us
I20260709 13:04:10.140956 10844 leader_election.cc:304] T 868e92339c2f4508af0afbd07c814d64 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 186b12638a87410487467df721a3cbda; no voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47
I20260709 13:04:10.067075 12044 raft_consensus.cc:2749] T 525bfdbd6d934afd823f7d3ee3f9d052 P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:04:10.146529 10844 leader_election.cc:304] T 57b6a2418c1342e7924195e9c1dbd357 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:04:10.166266 10879 raft_consensus.cc:1240] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Rejecting Update request from peer 10ca5ee68ad64b3283b9d235352c9d23 for earlier term 2. Current term is 3. Ops: [2.1-2.1]
I20260709 13:04:10.169369 11102 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f9df2bc281f4866b6da5da94a0defc7" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
W20260709 13:04:10.170799 10843 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:10.171089 10844 leader_election.cc:304] T 7757ecbf402e47bfbc93d0447741e26c P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:04:10.171833 10843 leader_election.cc:304] T 0f9df2bc281f4866b6da5da94a0defc7 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda; no voters: 
I20260709 13:04:10.131278 11887 raft_consensus.cc:2804] T 1e4f2241a94a41f38e481137f868b83d P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:04:10.140220 11983 raft_consensus.cc:2749] T bac7e5040f984ba2b7146bd36af7267e P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:04:10.105834 11920 raft_consensus.cc:515] T e5ba72ccd854430baadbc636dfd17136 P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:10.111378 11972 raft_consensus.cc:515] T b4cd3d7dfbbf4096908ff11e9dc57141 P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:10.181692 10879 raft_consensus.cc:1240] T 5d439326f23747eaab3c1e7da2d6b785 P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Rejecting Update request from peer 10ca5ee68ad64b3283b9d235352c9d23 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260709 13:04:10.144344 11936 raft_consensus.cc:2749] T 2567c2d84e4648d7ba5dc23a3df32817 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:04:10.174067 11797 raft_consensus.cc:2804] T 57b6a2418c1342e7924195e9c1dbd357 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:04:10.247748 11797 raft_consensus.cc:493] T 57b6a2418c1342e7924195e9c1dbd357 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:10.248004 11926 raft_consensus.cc:493] T 6f46a5102fd64b45bf1c4dff863d2b4d P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 873357742d5843579556ee8166f23f47)
W20260709 13:04:10.248257 10920 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 119485us
I20260709 13:04:10.249007 11926 raft_consensus.cc:515] T 6f46a5102fd64b45bf1c4dff863d2b4d P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:10.249688 10920 leader_election.cc:304] T d71ca3a1ba6b48478aaf62741bf3cf6a P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23; no voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47
I20260709 13:04:10.248287 11797 raft_consensus.cc:3060] T 57b6a2418c1342e7924195e9c1dbd357 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:04:10.250660 11983 raft_consensus.cc:2749] T d71ca3a1ba6b48478aaf62741bf3cf6a P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:04:10.252070 11102 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c31881534c5e4f0089c82057c1746652" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:10.252767 11102 raft_consensus.cc:2468] T c31881534c5e4f0089c82057c1746652 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 2.
I20260709 13:04:10.253788 11102 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d7ce47b9a92d43dd88d95bcda5038597" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:10.180078 11887 raft_consensus.cc:493] T 1e4f2241a94a41f38e481137f868b83d P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:04:10.175211 10994 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 109473us
I20260709 13:04:10.139840 11953 consensus_queue.cc:1048] T a64b7e99d6c54934a7de6f6dffcf5e94 P 873357742d5843579556ee8166f23f47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.042s
I20260709 13:04:10.264531 11101 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75492f86fde046c3900df3eac8292fd4" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:04:10.285440 11101 raft_consensus.cc:3060] T 75492f86fde046c3900df3eac8292fd4 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:04:10.257597 10994 leader_election.cc:304] T 4f9eb06ed01f4ce68be3af8a8434665b P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47; no voters: 186b12638a87410487467df721a3cbda
I20260709 13:04:10.281109 10920 leader_election.cc:304] T ff13ae9a47c7455f962406c6d131a3ff P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23; no voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47
W20260709 13:04:10.293819 10845 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:10.254415 11102 raft_consensus.cc:2468] T d7ce47b9a92d43dd88d95bcda5038597 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 1.
I20260709 13:04:10.249926 11849 raft_consensus.cc:515] T ff13ae9a47c7455f962406c6d131a3ff P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:10.309031 10920 leader_election.cc:304] T 87cd8f5097624c8692f1d83544d8881e P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 10ca5ee68ad64b3283b9d235352c9d23; no voters: 
W20260709 13:04:10.363471 10994 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 72595.1us
I20260709 13:04:10.372100 11921 leader_election.cc:290] T 0e5d327630e5438b878035d28efe6c0c P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:10.258574 11020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f46a5102fd64b45bf1c4dff863d2b4d" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:10.373531 12050 consensus_queue.cc:1048] T 8c160ddfe9b34bdaa3835d0915c1e06f P 186b12638a87410487467df721a3cbda [LEADER]: Connected to new peer: Peer: permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:04:10.373594 11102 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d371c5e27bd843469d2e2e453734cb4a" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:04:10.374642 11983 raft_consensus.cc:2749] T ff13ae9a47c7455f962406c6d131a3ff P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:04:10.184523 12020 raft_consensus.cc:493] T bb7d67be500947b3aef4c4245e33671c P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:10.400475 11102 raft_consensus.cc:2393] T d371c5e27bd843469d2e2e453734cb4a P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 186b12638a87410487467df721a3cbda in current term 3: Already voted for candidate 0172fed22b6b4890a7fdc953fa128af4 in this term.
I20260709 13:04:10.255174 11887 raft_consensus.cc:3060] T 1e4f2241a94a41f38e481137f868b83d P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:04:10.257779 11926 leader_election.cc:290] T 6f46a5102fd64b45bf1c4dff863d2b4d P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:10.386896 10953 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87cd8f5097624c8692f1d83544d8881e" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:10.298187 11653 raft_consensus.cc:2804] T 4f9eb06ed01f4ce68be3af8a8434665b P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:04:10.390910 11103 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72b111ad7b5f4753ab1d2b5ee1114840" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:10.407581 11103 raft_consensus.cc:2410] T 72b111ad7b5f4753ab1d2b5ee1114840 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 186b12638a87410487467df721a3cbda for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20260709 13:04:10.392302 10920 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 83401.9us
I20260709 13:04:10.378276 11025 raft_consensus.cc:1240] T e5ba72ccd854430baadbc636dfd17136 P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Rejecting Update request from peer 10ca5ee68ad64b3283b9d235352c9d23 for earlier term 3. Current term is 4. Ops: []
W20260709 13:04:10.410372 10843 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 238657us
W20260709 13:04:10.412894 10992 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 375400us
I20260709 13:04:10.412859 10994 leader_election.cc:304] T f371d2565dd84667b5613848604949bb P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47; no voters: 186b12638a87410487467df721a3cbda
I20260709 13:04:10.414194 10992 leader_election.cc:304] T a977842ba4b744f4bb2febd55d286006 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:04:10.419373 10994 leader_election.cc:304] T 401a9cf11eac479aa445ec8c1d6e33b2 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47; no voters: 186b12638a87410487467df721a3cbda
I20260709 13:04:10.420166 12040 raft_consensus.cc:2804] T a977842ba4b744f4bb2febd55d286006 P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:04:10.421839 10994 leader_election.cc:400] T 6a9353d1d230416296627d7cf8b1d425 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 3 election: Vote denied by peer 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507) with higher term. Message: Invalid argument: T 6a9353d1d230416296627d7cf8b1d425 P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 873357742d5843579556ee8166f23f47 for earlier term 3. Current term is 4.
I20260709 13:04:10.423902 12043 raft_consensus.cc:2804] T 401a9cf11eac479aa445ec8c1d6e33b2 P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:04:10.424335 12043 raft_consensus.cc:493] T 401a9cf11eac479aa445ec8c1d6e33b2 P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 186b12638a87410487467df721a3cbda)
I20260709 13:04:10.424681 12043 raft_consensus.cc:3060] T 401a9cf11eac479aa445ec8c1d6e33b2 P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:04:10.402176 12007 raft_consensus.cc:493] T a977842ba4b744f4bb2febd55d286006 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:10.402602 12049 raft_consensus.cc:2804] T 87cd8f5097624c8692f1d83544d8881e P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:04:10.454156 11102 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7757ecbf402e47bfbc93d0447741e26c" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:10.411835 11653 raft_consensus.cc:493] T 4f9eb06ed01f4ce68be3af8a8434665b P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 186b12638a87410487467df721a3cbda)
W20260709 13:04:10.469673 10844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:45807, user_credentials={real_user=slave}} blocked reactor thread for 298724us
I20260709 13:04:10.430833 11953 raft_consensus.cc:2804] T f371d2565dd84667b5613848604949bb P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:04:10.428651 11099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b67eff0634e540eab91cf8cee795969f" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:10.488092 12040 raft_consensus.cc:697] T a977842ba4b744f4bb2febd55d286006 P 873357742d5843579556ee8166f23f47 [term 1 LEADER]: Becoming Leader. State: Replica: 873357742d5843579556ee8166f23f47, State: Running, Role: LEADER
W20260709 13:04:10.489117 10920 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 58199.7us
I20260709 13:04:10.490203 10920 leader_election.cc:304] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 10ca5ee68ad64b3283b9d235352c9d23; no voters: 873357742d5843579556ee8166f23f47
W20260709 13:04:10.492305 10992 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 70955us
I20260709 13:04:10.414291 11748 consensus_queue.cc:1059] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.054s
I20260709 13:04:10.406754 12020 raft_consensus.cc:515] T bb7d67be500947b3aef4c4245e33671c P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:10.469314 11653 raft_consensus.cc:3060] T 4f9eb06ed01f4ce68be3af8a8434665b P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:04:10.505788 11102 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d371c5e27bd843469d2e2e453734cb4a" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:04:10.509709 11102 raft_consensus.cc:2393] T d371c5e27bd843469d2e2e453734cb4a P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 873357742d5843579556ee8166f23f47 in current term 3: Already voted for candidate 0172fed22b6b4890a7fdc953fa128af4 in this term.
I20260709 13:04:10.456588 12049 raft_consensus.cc:493] T 87cd8f5097624c8692f1d83544d8881e P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 186b12638a87410487467df721a3cbda)
I20260709 13:04:10.515998 11101 raft_consensus.cc:2468] T 75492f86fde046c3900df3eac8292fd4 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 2.
W20260709 13:04:10.517568 10843 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 105696us
I20260709 13:04:10.521888 10843 leader_election.cc:304] T 34064dc7bf3747dbb3876dd824ee0688 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda; no voters: 873357742d5843579556ee8166f23f47
I20260709 13:04:10.464656 12007 raft_consensus.cc:515] T a977842ba4b744f4bb2febd55d286006 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:10.500327 11099 raft_consensus.cc:2468] T b67eff0634e540eab91cf8cee795969f P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 1.
I20260709 13:04:10.498746 12052 consensus_queue.cc:1048] T b67eff0634e540eab91cf8cee795969f P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.055s
I20260709 13:04:10.537334 11020 raft_consensus.cc:1240] T b4cd3d7dfbbf4096908ff11e9dc57141 P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Rejecting Update request from peer 0172fed22b6b4890a7fdc953fa128af4 for earlier term 3. Current term is 4. Ops: []
I20260709 13:04:10.528218 11981 raft_consensus.cc:2804] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:04:10.538394 11981 raft_consensus.cc:493] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:10.538897 11981 raft_consensus.cc:3060] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:04:10.540114 11094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7709387f4d3d4f9992d97a0fbca39a40" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:04:10.541889 11094 raft_consensus.cc:1275] T 9a1716c0adf34bee8a4e99341ba5788f P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Refusing update from remote peer 873357742d5843579556ee8166f23f47: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 13:04:10.543967 12064 raft_consensus.cc:493] T a154cc8b181f460abf4880064379948b P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:10.544443 11096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "836251ebc2b74f5da88b8a8b2338e2a5" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:10.537701 11098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c51782cd9d064ec0949e84e24dd4de67" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:10.544744 12064 raft_consensus.cc:515] T a154cc8b181f460abf4880064379948b P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
W20260709 13:04:10.553465 10845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 159776us
I20260709 13:04:10.555109 11102 raft_consensus.cc:1275] T 8a69f32c53f24a50b7852d143665d7b4 P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Refusing update from remote peer 186b12638a87410487467df721a3cbda: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20260709 13:04:10.555581 10992 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 62594.1us
I20260709 13:04:10.561488 10992 leader_election.cc:304] T 63d44f6ba6434ab2870fe3190f4d7c2f P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 873357742d5843579556ee8166f23f47; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda
I20260709 13:04:10.562705 10845 leader_election.cc:304] T c31881534c5e4f0089c82057c1746652 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda; no voters: 
W20260709 13:04:10.559908 10920 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 68326.6us
I20260709 13:04:10.565455 10992 leader_election.cc:304] T d71ca3a1ba6b48478aaf62741bf3cf6a P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47; no voters: 0172fed22b6b4890a7fdc953fa128af4
I20260709 13:04:10.522648 12049 raft_consensus.cc:3060] T 87cd8f5097624c8692f1d83544d8881e P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Advancing to term 4
W20260709 13:04:10.581406 10878 consensus_meta.cc:287] T 7757ecbf402e47bfbc93d0447741e26c P 186b12638a87410487467df721a3cbda: Time spent flushing consensus metadata: real 0.616s	user 0.010s	sys 0.003s
I20260709 13:04:10.587647 10878 raft_consensus.cc:2468] T 7757ecbf402e47bfbc93d0447741e26c P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 3.
I20260709 13:04:10.544010 11553 leader_election.cc:290] T e5a5f5a608504650b48fd27f163faed6 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:10.537400 11849 leader_election.cc:290] T ff13ae9a47c7455f962406c6d131a3ff P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 3 election: Requested vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:10.546053 12061 raft_consensus.cc:2804] T 0f9df2bc281f4866b6da5da94a0defc7 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:04:10.656456 10878 raft_consensus.cc:1275] T c31881534c5e4f0089c82057c1746652 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Refusing update from remote peer 10ca5ee68ad64b3283b9d235352c9d23: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 1. (term mismatch)
I20260709 13:04:10.575361 12052 raft_consensus.cc:3055] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 LEADER]: Stepping down as leader of term 2
W20260709 13:04:10.674705 10844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:45807, user_credentials={real_user=slave}} blocked reactor thread for 184583us
I20260709 13:04:10.668439 10878 pending_rounds.cc:85] T c31881534c5e4f0089c82057c1746652 P 186b12638a87410487467df721a3cbda: Aborting all ops after (but not including) 0
I20260709 13:04:10.679872 10878 pending_rounds.cc:107] T c31881534c5e4f0089c82057c1746652 P 186b12638a87410487467df721a3cbda: Aborting uncommitted NO_OP operation due to leader change: 1.1
I20260709 13:04:10.680228 10878 raft_consensus.cc:2889] T c31881534c5e4f0089c82057c1746652 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:04:10.626636 12050 raft_consensus.cc:2764] T 7757ecbf402e47bfbc93d0447741e26c P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260709 13:04:10.682036 10844 leader_election.cc:304] T 732c4adeeff6422497aa7db4a82f535b P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:04:10.676468 11020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff13ae9a47c7455f962406c6d131a3ff" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
I20260709 13:04:10.676767 11098 raft_consensus.cc:2393] T c51782cd9d064ec0949e84e24dd4de67 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 10ca5ee68ad64b3283b9d235352c9d23 in current term 3: Already voted for candidate 0172fed22b6b4890a7fdc953fa128af4 in this term.
W20260709 13:04:10.708328 10845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 145697us
I20260709 13:04:10.646867 11972 leader_election.cc:290] T b4cd3d7dfbbf4096908ff11e9dc57141 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 4 election: Requested vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:10.669490 11920 leader_election.cc:290] T e5ba72ccd854430baadbc636dfd17136 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 4 election: Requested vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:10.712908 10845 leader_election.cc:304] T d7ce47b9a92d43dd88d95bcda5038597 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda; no voters: 
I20260709 13:04:10.675884 12040 consensus_queue.cc:237] T a977842ba4b744f4bb2febd55d286006 P 873357742d5843579556ee8166f23f47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:10.673233 12061 raft_consensus.cc:493] T 0f9df2bc281f4866b6da5da94a0defc7 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0172fed22b6b4890a7fdc953fa128af4)
I20260709 13:04:10.718415 10878 raft_consensus.cc:1240] T 6a9353d1d230416296627d7cf8b1d425 P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Rejecting Update request from peer 873357742d5843579556ee8166f23f47 for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260709 13:04:10.723096 11066 leader_election.cc:304] T e5ba72ccd854430baadbc636dfd17136 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47
I20260709 13:04:10.684072 10948 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d7ce47b9a92d43dd88d95bcda5038597" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:10.672335 12052 raft_consensus.cc:740] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 10ca5ee68ad64b3283b9d235352c9d23, State: Running, Role: LEADER
W20260709 13:04:10.712875 11962 consensus_meta.cc:287] T d270b9e5481245d9b5340a454a7cd10b P 10ca5ee68ad64b3283b9d235352c9d23: Time spent flushing consensus metadata: real 0.887s	user 0.012s	sys 0.003s
I20260709 13:04:10.719542 11096 raft_consensus.cc:1240] T 6a9353d1d230416296627d7cf8b1d425 P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Rejecting Update request from peer 873357742d5843579556ee8166f23f47 for earlier term 3. Current term is 4. Ops: []
W20260709 13:04:10.835714 11066 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 112713us
I20260709 13:04:10.664449 12063 consensus_queue.cc:1059] T c51782cd9d064ec0949e84e24dd4de67 P 186b12638a87410487467df721a3cbda [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.079s
I20260709 13:04:10.683558 11797 raft_consensus.cc:515] T 57b6a2418c1342e7924195e9c1dbd357 P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:10.843047 11066 leader_election.cc:304] T d270b9e5481245d9b5340a454a7cd10b P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda
I20260709 13:04:10.727350 10946 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d270b9e5481245d9b5340a454a7cd10b" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:04:10.776057 12061 raft_consensus.cc:3060] T 0f9df2bc281f4866b6da5da94a0defc7 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:04:10.859336 12064 leader_election.cc:290] T a154cc8b181f460abf4880064379948b P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:10.834614 11926 raft_consensus.cc:2749] T e5ba72ccd854430baadbc636dfd17136 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:04:10.834487 12009 raft_consensus.cc:493] T d270b9e5481245d9b5340a454a7cd10b P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:10.832716 11098 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37666 (request call id 177) took 1395 ms (client timeout 1659 ms). Trace:
I20260709 13:04:10.870611 11066 leader_election.cc:304] T 525bfdbd6d934afd823f7d3ee3f9d052 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47
W20260709 13:04:10.865932 10843 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 179807us
I20260709 13:04:10.873106 11066 leader_election.cc:304] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47
I20260709 13:04:10.851279 11094 raft_consensus.cc:1275] T c31881534c5e4f0089c82057c1746652 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Refusing update from remote peer 10ca5ee68ad64b3283b9d235352c9d23: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
W20260709 13:04:10.855352 11795 consensus_meta.cc:287] T f371d2565dd84667b5613848604949bb P 186b12638a87410487467df721a3cbda: Time spent flushing consensus metadata: real 0.745s	user 0.015s	sys 0.000s
I20260709 13:04:10.866122 11920 raft_consensus.cc:3060] T 63d44f6ba6434ab2870fe3190f4d7c2f P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Advancing to term 3
W20260709 13:04:10.889235 10845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 176463us
I20260709 13:04:10.866739 11096 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37676 (request call id 226) took 1001 ms. Trace:
I20260709 13:04:10.868515 12020 leader_election.cc:290] T bb7d67be500947b3aef4c4245e33671c P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:10.887015 12087 raft_consensus.cc:2804] T d71ca3a1ba6b48478aaf62741bf3cf6a P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:04:10.883172 11633 raft_consensus.cc:493] T 7839ca2d2c13431a85f44d7c458c6351 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:10.896888 12057 log.cc:927] Time spent T 72b111ad7b5f4753ab1d2b5ee1114840 P 186b12638a87410487467df721a3cbda: Append to log took a long time: real 0.253s	user 0.001s	sys 0.000s
I20260709 13:04:10.897809 12070 raft_consensus.cc:493] T 1e4f2241a94a41f38e481137f868b83d P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:10.876791 12064 raft_consensus.cc:2749] T 525bfdbd6d934afd823f7d3ee3f9d052 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260709 13:04:10.867038 11098 rpcz_store.cc:269] 0709 13:04:09.437184 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:09.446636 (+  9452us) service_pool.cc:224] Handling call
0709 13:04:09.848473 (+401837us) spinlock_profiling.cc:243] Waited 4.15 ms on lock 0x7b5800117698. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f426303db71 00007f426303da22 00007f426bf56c09 00007f426b0b0ee0 00007f426b17a0ab 00007f426b17a131 00007f426b1280a0 00007f426b127984 00007f42646a26d4 00007f42646a2662
0709 13:04:09.897424 (+ 48951us) spinlock_profiling.cc:243] Waited 37.8 ms on lock 0x7b5800117698. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f426303de9c 00007f426303da5b 00007f426bf56c09 00007f426b0b0ee0 00007f426b17a0ab 00007f426b17a131 00007f426b1280a0 00007f426b127984 00007f42646a26d4 00007f42646a2662
0709 13:04:10.537554 (+640130us) spinlock_profiling.cc:243] Waited 20.7 ms on lock 0x7b5800117698. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f426303db71 00007f426303deee 0000000000375939 00007f426b0b11e0 00007f426b17a0c4 00007f426b17a131 00007f426b1280a0 00007f426b127984 00007f42646a26d4 00007f42646a2662
0709 13:04:10.788063 (+250509us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":144211968}
W20260709 13:04:10.899441 12054 log.cc:927] Time spent T 00000000000000000000000000000000 P c669bee8ad644d0d956c148b33886b51: Append to log took a long time: real 0.329s	user 0.001s	sys 0.000s
W20260709 13:04:10.899916 12072 log.cc:927] Time spent T 5de5f0581e3b41dea729363d6ca563bb P 186b12638a87410487467df721a3cbda: Append to log took a long time: real 0.160s	user 0.001s	sys 0.000s
I20260709 13:04:10.877786 12066 raft_consensus.cc:2749] T d270b9e5481245d9b5340a454a7cd10b P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 13:04:10.870747 12086 raft_consensus.cc:2764] T 34064dc7bf3747dbb3876dd824ee0688 P 186b12638a87410487467df721a3cbda [term 5 FOLLOWER]: Leader election decision vote started in defunct term 4: won
W20260709 13:04:10.892534 12055 log.cc:927] Time spent T 5de5f0581e3b41dea729363d6ca563bb P 10ca5ee68ad64b3283b9d235352c9d23: Append to log took a long time: real 0.341s	user 0.001s	sys 0.001s
I20260709 13:04:10.903296 12007 leader_election.cc:290] T a977842ba4b744f4bb2febd55d286006 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:10.918591 11066 leader_election.cc:304] T e5a5f5a608504650b48fd27f163faed6 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda
W20260709 13:04:10.920948 10992 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 355585us
I20260709 13:04:10.871318 12009 raft_consensus.cc:515] T d270b9e5481245d9b5340a454a7cd10b P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:10.909056 10948 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e4f2241a94a41f38e481137f868b83d" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:10.925830 12087 raft_consensus.cc:697] T d71ca3a1ba6b48478aaf62741bf3cf6a P 873357742d5843579556ee8166f23f47 [term 3 LEADER]: Becoming Leader. State: Replica: 873357742d5843579556ee8166f23f47, State: Running, Role: LEADER
W20260709 13:04:10.929582 10876 consensus_meta.cc:287] T a1ec2b50c6ad4f7c8033476208abb26d P 186b12638a87410487467df721a3cbda: Time spent flushing consensus metadata: real 1.056s	user 0.012s	sys 0.004s
I20260709 13:04:10.930794 10994 leader_election.cc:304] T d371c5e27bd843469d2e2e453734cb4a P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 873357742d5843579556ee8166f23f47; no voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda
I20260709 13:04:10.875007 12052 consensus_queue.cc:260] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 10ca5ee68ad64b3283b9d235352c9d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 2.1, Last appended by leader: 1, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:10.908418 11633 raft_consensus.cc:515] T 7839ca2d2c13431a85f44d7c458c6351 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:10.913012 10946 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "732c4adeeff6422497aa7db4a82f535b" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
W20260709 13:04:10.939865 10874 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.503s	user 0.000s	sys 0.024s
I20260709 13:04:10.937637 12007 raft_consensus.cc:2749] T e5a5f5a608504650b48fd27f163faed6 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20260709 13:04:10.947116 12024 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.921s	user 0.008s	sys 0.021s
I20260709 13:04:10.892477 11096 rpcz_store.cc:276] 0709 13:04:09.861999 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:10.553207 (+691208us) service_pool.cc:224] Handling call
0709 13:04:10.697524 (+144317us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:10.846614 (+149090us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:10.863307 (+ 16693us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:04:10.964555 10826 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 13:04:10.937058 10876 raft_consensus.cc:2468] T a1ec2b50c6ad4f7c8033476208abb26d P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 1.
I20260709 13:04:10.954401 10920 leader_election.cc:304] T 75492f86fde046c3900df3eac8292fd4 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 10ca5ee68ad64b3283b9d235352c9d23; no voters: 873357742d5843579556ee8166f23f47
I20260709 13:04:10.943208 11926 raft_consensus.cc:493] T 0e5d327630e5438b878035d28efe6c0c P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:10.962617 12071 raft_consensus.cc:493] T 4c2a35bb99374e93b573cf870d04faaa P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:10.964475 11553 raft_consensus.cc:493] T 0f9df2bc281f4866b6da5da94a0defc7 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0172fed22b6b4890a7fdc953fa128af4)
I20260709 13:04:10.948881 12075 raft_consensus.cc:493] T a64b7e99d6c54934a7de6f6dffcf5e94 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 873357742d5843579556ee8166f23f47)
W20260709 13:04:10.966670 12024 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.950s	user 0.009s	sys 0.021s
I20260709 13:04:10.907199 12070 raft_consensus.cc:515] T 1e4f2241a94a41f38e481137f868b83d P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:10.962157 12063 raft_consensus.cc:2804] T c31881534c5e4f0089c82057c1746652 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260709 13:04:10.954715 10874 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.522s	user 0.000s	sys 0.025s
I20260709 13:04:10.982306 11962 raft_consensus.cc:2804] T 75492f86fde046c3900df3eac8292fd4 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Leader election won for term 2
W20260709 13:04:10.991572 11066 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 77894.3us
I20260709 13:04:10.943432 10948 raft_consensus.cc:2468] T 1e4f2241a94a41f38e481137f868b83d P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 2.
I20260709 13:04:10.982227 12087 consensus_queue.cc:237] T d71ca3a1ba6b48478aaf62741bf3cf6a P 873357742d5843579556ee8166f23f47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
W20260709 13:04:11.007939 10845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 117799us
I20260709 13:04:10.982004 12074 raft_consensus.cc:2804] T 732c4adeeff6422497aa7db4a82f535b P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:04:10.994872 12088 raft_consensus.cc:3055] T c51782cd9d064ec0949e84e24dd4de67 P 186b12638a87410487467df721a3cbda [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:04:10.985642 10876 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51704 (request call id 183) took 1190 ms. Trace:
I20260709 13:04:10.982788 12076 raft_consensus.cc:2749] T d371c5e27bd843469d2e2e453734cb4a P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 13:04:11.013355 10845 leader_election.cc:304] T b67eff0634e540eab91cf8cee795969f P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda; no voters: 
I20260709 13:04:10.996318 10954 raft_consensus.cc:2410] T d270b9e5481245d9b5340a454a7cd10b P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 0172fed22b6b4890a7fdc953fa128af4 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 13:04:11.019291 10949 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e96e80ba9ac34d0998395515e5c8c41a" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:10.995759 12063 raft_consensus.cc:493] T c31881534c5e4f0089c82057c1746652 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:10.986837 11094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7757ecbf402e47bfbc93d0447741e26c" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:04:11.022365 10845 leader_election.cc:304] T 7709387f4d3d4f9992d97a0fbca39a40 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 186b12638a87410487467df721a3cbda; no voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47
I20260709 13:04:11.011540 12088 raft_consensus.cc:740] T c51782cd9d064ec0949e84e24dd4de67 P 186b12638a87410487467df721a3cbda [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 186b12638a87410487467df721a3cbda, State: Running, Role: LEADER
W20260709 13:04:11.029726 12082 log.cc:927] Time spent T 732c4adeeff6422497aa7db4a82f535b P 873357742d5843579556ee8166f23f47: Append to log took a long time: real 0.176s	user 0.000s	sys 0.001s
I20260709 13:04:11.030668 11094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5ba72ccd854430baadbc636dfd17136" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:04:11.030575 12088 consensus_queue.cc:260] T c51782cd9d064ec0949e84e24dd4de67 P 186b12638a87410487467df721a3cbda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:11.031621 12007 consensus_queue.cc:1059] T d71ca3a1ba6b48478aaf62741bf3cf6a P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.007s
I20260709 13:04:11.031996 12088 raft_consensus.cc:3060] T c51782cd9d064ec0949e84e24dd4de67 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:04:11.032549 12069 consensus_queue.cc:1059] T 75492f86fde046c3900df3eac8292fd4 P 873357742d5843579556ee8166f23f47 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 0, Time since last communication: 0.089s
I20260709 13:04:10.982573 12071 raft_consensus.cc:515] T 4c2a35bb99374e93b573cf870d04faaa P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:10.985271 12052 raft_consensus.cc:3060] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:04:10.981405 11953 raft_consensus.cc:697] T f371d2565dd84667b5613848604949bb P 873357742d5843579556ee8166f23f47 [term 3 LEADER]: Becoming Leader. State: Replica: 873357742d5843579556ee8166f23f47, State: Running, Role: LEADER
I20260709 13:04:11.027545 11099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d371c5e27bd843469d2e2e453734cb4a" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:11.028369 12063 raft_consensus.cc:3060] T c31881534c5e4f0089c82057c1746652 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:04:11.029088 10948 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a64b7e99d6c54934a7de6f6dffcf5e94" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:11.029652 12070 leader_election.cc:290] T 1e4f2241a94a41f38e481137f868b83d P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:10.982774 11553 raft_consensus.cc:515] T 0f9df2bc281f4866b6da5da94a0defc7 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:11.008136 11983 raft_consensus.cc:493] T 1e4f2241a94a41f38e481137f868b83d P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:10.982383 11926 raft_consensus.cc:515] T 0e5d327630e5438b878035d28efe6c0c P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:10.972155 12089 consensus_queue.cc:1048] T 9a1716c0adf34bee8a4e99341ba5788f P 873357742d5843579556ee8166f23f47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.065s
I20260709 13:04:11.012632 12074 raft_consensus.cc:493] T 732c4adeeff6422497aa7db4a82f535b P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Starting leader election (detected failure of leader 10ca5ee68ad64b3283b9d235352c9d23)
I20260709 13:04:11.012439 12095 raft_consensus.cc:2804] T d7ce47b9a92d43dd88d95bcda5038597 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260709 13:04:11.024479 10873 consensus_meta.cc:287] T 7709387f4d3d4f9992d97a0fbca39a40 P 186b12638a87410487467df721a3cbda: Time spent flushing consensus metadata: real 1.152s	user 0.018s	sys 0.002s
I20260709 13:04:11.024240 11962 raft_consensus.cc:697] T 75492f86fde046c3900df3eac8292fd4 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 LEADER]: Becoming Leader. State: Replica: 10ca5ee68ad64b3283b9d235352c9d23, State: Running, Role: LEADER
I20260709 13:04:11.024047 11879 raft_consensus.cc:493] T a1ec2b50c6ad4f7c8033476208abb26d P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:11.055891 10946 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2567c2d84e4648d7ba5dc23a3df32817" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
W20260709 13:04:11.048038 10844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:45807, user_credentials={real_user=slave}} blocked reactor thread for 372036us
I20260709 13:04:11.077945 10844 leader_election.cc:304] T d371c5e27bd843469d2e2e453734cb4a P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 186b12638a87410487467df721a3cbda; no voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47
I20260709 13:04:11.068535 10955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "836251ebc2b74f5da88b8a8b2338e2a5" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:11.049593 12105 consensus_queue.cc:1048] T c31881534c5e4f0089c82057c1746652 P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:04:10.996430 12075 raft_consensus.cc:515] T a64b7e99d6c54934a7de6f6dffcf5e94 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:11.050388 12048 raft_consensus.cc:2749] T 868e92339c2f4508af0afbd07c814d64 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:04:11.006611 11981 raft_consensus.cc:515] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 10ca5ee68ad64b3283b9d235352c9d23 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:11.051542 11099 raft_consensus.cc:2393] T d371c5e27bd843469d2e2e453734cb4a P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 873357742d5843579556ee8166f23f47 in current term 3: Already voted for candidate 0172fed22b6b4890a7fdc953fa128af4 in this term.
I20260709 13:04:11.027658 11097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4cd3d7dfbbf4096908ff11e9dc57141" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:04:11.029492 11633 leader_election.cc:290] T 7839ca2d2c13431a85f44d7c458c6351 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:11.037447 11096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e5d327630e5438b878035d28efe6c0c" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:11.015151 10876 rpcz_store.cc:276] 0709 13:04:09.794734 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:09.805422 (+ 10688us) service_pool.cc:224] Handling call
0709 13:04:10.985554 (+1180132us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:11.061007 10949 raft_consensus.cc:2410] T e96e80ba9ac34d0998395515e5c8c41a P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 186b12638a87410487467df721a3cbda for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20260709 13:04:11.093606 12043 consensus_meta.cc:287] T 401a9cf11eac479aa445ec8c1d6e33b2 P 873357742d5843579556ee8166f23f47: Time spent flushing consensus metadata: real 0.669s	user 0.016s	sys 0.000s
I20260709 13:04:11.037263 12083 consensus_queue.cc:1048] T 401a9cf11eac479aa445ec8c1d6e33b2 P 186b12638a87410487467df721a3cbda [LEADER]: Connected to new peer: Peer: permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.035s
I20260709 13:04:11.069352 10872 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51696 (request call id 225) took 1013 ms. Trace:
W20260709 13:04:11.060416 12019 consensus_meta.cc:287] T 8c160ddfe9b34bdaa3835d0915c1e06f P 0172fed22b6b4890a7fdc953fa128af4: Time spent flushing consensus metadata: real 1.094s	user 0.012s	sys 0.004s
I20260709 13:04:11.058281 12095 raft_consensus.cc:493] T d7ce47b9a92d43dd88d95bcda5038597 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Starting leader election (detected failure of leader 10ca5ee68ad64b3283b9d235352c9d23)
I20260709 13:04:11.059201 10874 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51704 (request call id 192) took 1141 ms. Trace:
I20260709 13:04:11.136247 10874 rpcz_store.cc:276] 0709 13:04:09.916376 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:10.093016 (+176640us) service_pool.cc:224] Handling call
0709 13:04:10.093451 (+   435us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:10.093994 (+   543us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:10.094040 (+    46us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:10.118344 (+ 24304us) log.cc:844] Serialized 26 byte log entry
0709 13:04:10.386803 (+268459us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:04:11.013268 (+626465us) log.cc:844] Serialized 14 byte log entry
0709 13:04:11.035489 (+ 22221us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:11.039988 (+  4499us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:11.044586 (+  4598us) raft_consensus.cc:1685] finished
0709 13:04:11.044598 (+    12us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:11.057169 (+ 12571us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5454,"prepare.queue_time_us":192417,"prepare.run_cpu_time_us":80,"prepare.run_wall_time_us":82,"raft.queue_time_us":26816,"raft.run_cpu_time_us":368,"raft.run_wall_time_us":9241,"thread_start_us":820111,"threads_started":2,"wal-append.queue_time_us":315710}
I20260709 13:04:11.143949 12075 leader_election.cc:290] T a64b7e99d6c54934a7de6f6dffcf5e94 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:11.112396 10955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a977842ba4b744f4bb2febd55d286006" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:11.114058 10877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11de93cd9b9d457489ff1cb80f976c9a" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:04:11.058393 12074 raft_consensus.cc:3060] T 732c4adeeff6422497aa7db4a82f535b P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:04:11.147825 10877 raft_consensus.cc:3060] T 11de93cd9b9d457489ff1cb80f976c9a P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:04:11.060071 10873 raft_consensus.cc:2468] T 7709387f4d3d4f9992d97a0fbca39a40 P 186b12638a87410487467df721a3cbda [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 5.
I20260709 13:04:11.122072 11097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5a5f5a608504650b48fd27f163faed6" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:11.058145 11983 raft_consensus.cc:515] T 1e4f2241a94a41f38e481137f868b83d P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
W20260709 13:04:11.129769 11066 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 80955.5us
I20260709 13:04:11.123035 10948 raft_consensus.cc:2410] T a64b7e99d6c54934a7de6f6dffcf5e94 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0172fed22b6b4890a7fdc953fa128af4 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 13:04:11.132486 12019 raft_consensus.cc:515] T 8c160ddfe9b34bdaa3835d0915c1e06f P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:11.089039 11879 raft_consensus.cc:515] T a1ec2b50c6ad4f7c8033476208abb26d P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:11.067751 11962 consensus_queue.cc:237] T 75492f86fde046c3900df3eac8292fd4 P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:11.135833 12095 raft_consensus.cc:3060] T d7ce47b9a92d43dd88d95bcda5038597 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:04:11.148860 10955 raft_consensus.cc:2468] T a977842ba4b744f4bb2febd55d286006 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 1.
I20260709 13:04:11.066150 12102 consensus_queue.cc:1048] T 8a69f32c53f24a50b7852d143665d7b4 P 186b12638a87410487467df721a3cbda [LEADER]: Connected to new peer: Peer: permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.062s
I20260709 13:04:11.169709 11102 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37676 (request call id 227) took 1307 ms. Trace:
I20260709 13:04:11.189649 10949 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f46a5102fd64b45bf1c4dff863d2b4d" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:11.172186 11849 raft_consensus.cc:2749] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:04:11.143976 11981 leader_election.cc:290] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 5 election: Requested vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:11.196298 10955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d439326f23747eaab3c1e7da2d6b785" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:04:11.129680 11096 raft_consensus.cc:2468] T 0e5d327630e5438b878035d28efe6c0c P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 4.
I20260709 13:04:11.133929 11953 consensus_queue.cc:237] T f371d2565dd84667b5613848604949bb P 873357742d5843579556ee8166f23f47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:11.198275 11066 leader_election.cc:304] T a977842ba4b744f4bb2febd55d286006 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 10ca5ee68ad64b3283b9d235352c9d23; no voters: 
W20260709 13:04:11.226873 10845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 175305us
I20260709 13:04:11.196914 11920 raft_consensus.cc:2749] T 63d44f6ba6434ab2870fe3190f4d7c2f P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:04:11.149447 12105 raft_consensus.cc:493] T e96e80ba9ac34d0998395515e5c8c41a P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:11.146232 11673 leader_election.cc:290] T 11de93cd9b9d457489ff1cb80f976c9a P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 5 election: Requested vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:11.208070 11099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f34f2bc02d85444f9096e0f5b4fe4331" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:04:11.144580 12063 raft_consensus.cc:515] T c31881534c5e4f0089c82057c1746652 P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
W20260709 13:04:11.215324 10844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:45807, user_credentials={real_user=slave}} blocked reactor thread for 137493us
I20260709 13:04:11.165800 10873 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51704 (request call id 191) took 1337 ms. Trace:
I20260709 13:04:11.211020 12083 raft_consensus.cc:2749] T d371c5e27bd843469d2e2e453734cb4a P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 13:04:11.173091 11926 leader_election.cc:290] T 0e5d327630e5438b878035d28efe6c0c P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:11.169569 11097 raft_consensus.cc:2468] T e5a5f5a608504650b48fd27f163faed6 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 3.
I20260709 13:04:11.130759 12043 raft_consensus.cc:515] T 401a9cf11eac479aa445ec8c1d6e33b2 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:11.183537 10876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e5d327630e5438b878035d28efe6c0c" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:11.221755 11102 rpcz_store.cc:276] 0709 13:04:09.862312 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:10.568039 (+705727us) service_pool.cc:224] Handling call
0709 13:04:10.640498 (+ 72459us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:10.640825 (+   327us) raft_consensus.cc:1534] Early marking committed up to index 3
0709 13:04:11.021236 (+380411us) log.cc:844] Serialized 14 byte log entry
0709 13:04:11.168827 (+147591us) log.cc:844] Serialized 14 byte log entry
0709 13:04:11.169174 (+   347us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:11.169246 (+    72us) raft_consensus.cc:1645] Marking committed up to 3
0709 13:04:11.169314 (+    68us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:11.169438 (+   124us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:11.169620 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":385795,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":45,"raft.queue_time_us":5005,"raft.run_wall_time_us":17698,"spinlock_wait_cycles":73173888,"thread_start_us":427831,"threads_started":2}
I20260709 13:04:11.154918 12108 raft_consensus.cc:2749] T 7709387f4d3d4f9992d97a0fbca39a40 P 186b12638a87410487467df721a3cbda [term 5 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 13:04:11.218266 10946 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff13ae9a47c7455f962406c6d131a3ff" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:04:11.239324 10995 leader_election.cc:304] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:04:11.231007 10949 raft_consensus.cc:2468] T 6f46a5102fd64b45bf1c4dff863d2b4d P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 2.
I20260709 13:04:11.237192 11099 raft_consensus.cc:3060] T f34f2bc02d85444f9096e0f5b4fe4331 P 0172fed22b6b4890a7fdc953fa128af4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:04:11.244812 11102 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f1466124dbd4f6ebb3c4d62ca48bf96" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
W20260709 13:04:11.247009 10921 leader_election.cc:336] T 11de93cd9b9d457489ff1cb80f976c9a P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 186b12638a87410487467df721a3cbda (127.0.107.193:38967): Timed out: RequestConsensusVote RPC to 127.0.107.193:38967 timed out after 1.828s (SENT)
I20260709 13:04:11.257464 10921 leader_election.cc:304] T 11de93cd9b9d457489ff1cb80f976c9a P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23; no voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47
I20260709 13:04:11.235942 11100 raft_consensus.cc:2393] T 8c160ddfe9b34bdaa3835d0915c1e06f P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 873357742d5843579556ee8166f23f47 in current term 3: Already voted for candidate 0172fed22b6b4890a7fdc953fa128af4 in this term.
I20260709 13:04:11.242345 10877 raft_consensus.cc:2468] T 11de93cd9b9d457489ff1cb80f976c9a P 186b12638a87410487467df721a3cbda [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 5.
I20260709 13:04:11.249953 11920 raft_consensus.cc:2804] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260709 13:04:11.250592 10918 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:11.250409 10946 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34064dc7bf3747dbb3876dd824ee0688" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:04:11.251031 10949 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72b111ad7b5f4753ab1d2b5ee1114840" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:11.251812 12071 leader_election.cc:290] T 4c2a35bb99374e93b573cf870d04faaa P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:11.249106 11779 raft_consensus.cc:493] T b4cd3d7dfbbf4096908ff11e9dc57141 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:11.253593 10955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65bf0a62f4d84ef18dd6b23e082724b8" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:11.230715 12096 raft_consensus.cc:2804] T a977842ba4b744f4bb2febd55d286006 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:04:11.268855 11981 raft_consensus.cc:2749] T 11de93cd9b9d457489ff1cb80f976c9a P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:04:11.276453 12111 raft_consensus.cc:493] T bb7d67be500947b3aef4c4245e33671c P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:11.278294 10949 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7839ca2d2c13431a85f44d7c458c6351" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:11.279219 11097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a64b7e99d6c54934a7de6f6dffcf5e94" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:11.280184 11102 raft_consensus.cc:1240] T ff13ae9a47c7455f962406c6d131a3ff P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Rejecting Update request from peer 873357742d5843579556ee8166f23f47 for earlier term 2. Current term is 3. Ops: []
I20260709 13:04:11.280316 10950 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a154cc8b181f460abf4880064379948b" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:11.272035 11920 raft_consensus.cc:493] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:11.247627 10875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7839ca2d2c13431a85f44d7c458c6351" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:11.232165 10873 rpcz_store.cc:276] 0709 13:04:09.828701 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:09.867542 (+ 38841us) service_pool.cc:224] Handling call
0709 13:04:11.154649 (+1287107us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1280}
I20260709 13:04:11.301661 10873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f34f2bc02d85444f9096e0f5b4fe4331" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:04:11.302490 10873 raft_consensus.cc:3060] T f34f2bc02d85444f9096e0f5b4fe4331 P 186b12638a87410487467df721a3cbda [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:04:11.284857 12096 raft_consensus.cc:493] T a977842ba4b744f4bb2febd55d286006 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:11.307188 12096 raft_consensus.cc:3060] T a977842ba4b744f4bb2febd55d286006 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:04:11.285277 10877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e4f2241a94a41f38e481137f868b83d" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:11.230643 12105 raft_consensus.cc:515] T e96e80ba9ac34d0998395515e5c8c41a P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:11.271494 10874 raft_consensus.cc:1275] T a610d1a71f52432e88fe56130f921fa2 P 186b12638a87410487467df721a3cbda [term 5 FOLLOWER]: Refusing update from remote peer 873357742d5843579556ee8166f23f47: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260709 13:04:11.282651 10876 raft_consensus.cc:2393] T 0e5d327630e5438b878035d28efe6c0c P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 873357742d5843579556ee8166f23f47 in current term 4: Already voted for candidate 186b12638a87410487467df721a3cbda in this term.
I20260709 13:04:11.313122 10875 raft_consensus.cc:2468] T 7839ca2d2c13431a85f44d7c458c6351 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 2.
I20260709 13:04:11.289309 10955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e4f2241a94a41f38e481137f868b83d" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:11.297504 10921 leader_election.cc:304] T e5a5f5a608504650b48fd27f163faed6 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda; no voters: 
I20260709 13:04:11.291224 10995 leader_election.cc:304] T 34064dc7bf3747dbb3876dd824ee0688 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:04:11.291072 12111 raft_consensus.cc:515] T bb7d67be500947b3aef4c4245e33671c P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:11.314491 10878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87cd8f5097624c8692f1d83544d8881e" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:11.292279 11095 raft_consensus.cc:1240] T 75492f86fde046c3900df3eac8292fd4 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Rejecting Update request from peer 873357742d5843579556ee8166f23f47 for earlier term 1. Current term is 2. Ops: []
I20260709 13:04:11.292434 10949 raft_consensus.cc:2468] T 7839ca2d2c13431a85f44d7c458c6351 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 2.
I20260709 13:04:11.321671 12111 leader_election.cc:290] T bb7d67be500947b3aef4c4245e33671c P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:11.322311 10949 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bac7e5040f984ba2b7146bd36af7267e" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:11.322717 10995 leader_election.cc:304] T 7757ecbf402e47bfbc93d0447741e26c P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:04:11.288822 11879 leader_election.cc:290] T a1ec2b50c6ad4f7c8033476208abb26d P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:11.274410 10946 raft_consensus.cc:3060] T 34064dc7bf3747dbb3876dd824ee0688 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:04:11.299798 11974 leader_election.cc:290] T 7839ca2d2c13431a85f44d7c458c6351 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 2 election: Requested vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:11.308940 10952 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4cd3d7dfbbf4096908ff11e9dc57141" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
W20260709 13:04:11.287264 11653 consensus_meta.cc:287] T 4f9eb06ed01f4ce68be3af8a8434665b P 873357742d5843579556ee8166f23f47: Time spent flushing consensus metadata: real 0.763s	user 0.014s	sys 0.002s
I20260709 13:04:11.314117 10955 raft_consensus.cc:2468] T 1e4f2241a94a41f38e481137f868b83d P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 2.
I20260709 13:04:11.296036 12053 raft_consensus.cc:3055] T 75492f86fde046c3900df3eac8292fd4 P 873357742d5843579556ee8166f23f47 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:04:11.320992 10950 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11de93cd9b9d457489ff1cb80f976c9a" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:04:11.331784 10995 leader_election.cc:304] T 11de93cd9b9d457489ff1cb80f976c9a P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:04:11.338701 10921 leader_election.cc:304] T 34064dc7bf3747dbb3876dd824ee0688 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23; no voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47
I20260709 13:04:11.339040 10876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c2a35bb99374e93b573cf870d04faaa" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:11.339593 10955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb7d67be500947b3aef4c4245e33671c" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:11.340186 10921 leader_election.cc:304] T a5299b8d044f41b7b68057e63d17b159 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23; no voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47
I20260709 13:04:11.340332 10955 raft_consensus.cc:2468] T bb7d67be500947b3aef4c4245e33671c P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 1.
I20260709 13:04:11.341365 11981 raft_consensus.cc:2804] T e5a5f5a608504650b48fd27f163faed6 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:04:11.341421 10955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b67eff0634e540eab91cf8cee795969f" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:11.341672 12063 leader_election.cc:290] T c31881534c5e4f0089c82057c1746652 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 3 election: Requested vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:11.338918 10874 raft_consensus.cc:1275] T 7709387f4d3d4f9992d97a0fbca39a40 P 186b12638a87410487467df721a3cbda [term 5 FOLLOWER]: Refusing update from remote peer 0172fed22b6b4890a7fdc953fa128af4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260709 13:04:11.341904 11981 raft_consensus.cc:493] T e5a5f5a608504650b48fd27f163faed6 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:11.346303 10955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5de5f0581e3b41dea729363d6ca563bb" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:11.345656 10950 raft_consensus.cc:3060] T 11de93cd9b9d457489ff1cb80f976c9a P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Advancing to term 5
I20260709 13:04:11.344993 10995 leader_election.cc:304] T 7839ca2d2c13431a85f44d7c458c6351 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:04:11.358083 11028 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "30f628e3d7914d609e7f1d980a74c654" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
I20260709 13:04:11.333700 12044 raft_consensus.cc:2804] T 34064dc7bf3747dbb3876dd824ee0688 P 873357742d5843579556ee8166f23f47 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:04:11.359912 11020 raft_consensus.cc:1240] T 401a9cf11eac479aa445ec8c1d6e33b2 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Rejecting Update request from peer 186b12638a87410487467df721a3cbda for earlier term 1. Current term is 2. Ops: []
I20260709 13:04:11.361703 11025 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46628 (request call id 188) took 1821 ms. Trace:
I20260709 13:04:11.288714 11779 raft_consensus.cc:515] T b4cd3d7dfbbf4096908ff11e9dc57141 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:11.333115 12053 raft_consensus.cc:740] T 75492f86fde046c3900df3eac8292fd4 P 873357742d5843579556ee8166f23f47 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 873357742d5843579556ee8166f23f47, State: Running, Role: LEADER
I20260709 13:04:11.364321 10952 raft_consensus.cc:3060] T b4cd3d7dfbbf4096908ff11e9dc57141 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:04:11.364969 12103 raft_consensus.cc:2804] T 7839ca2d2c13431a85f44d7c458c6351 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:04:11.365468 12103 raft_consensus.cc:493] T 7839ca2d2c13431a85f44d7c458c6351 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:11.365808 12103 raft_consensus.cc:3060] T 7839ca2d2c13431a85f44d7c458c6351 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:04:11.343433 12059 raft_consensus.cc:2804] T 11de93cd9b9d457489ff1cb80f976c9a P 873357742d5843579556ee8166f23f47 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:04:11.366655 12071 consensus_queue.cc:1059] T 5d439326f23747eaab3c1e7da2d6b785 P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:04:11.344223 11673 consensus_queue.cc:1059] T 6a9353d1d230416296627d7cf8b1d425 P 873357742d5843579556ee8166f23f47 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.032s
I20260709 13:04:11.347884 11974 raft_consensus.cc:493] T a154cc8b181f460abf4880064379948b P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:11.323115 10949 raft_consensus.cc:2468] T bac7e5040f984ba2b7146bd36af7267e P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 2.
I20260709 13:04:11.369248 10955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5ba72ccd854430baadbc636dfd17136" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:04:11.363893 12053 consensus_queue.cc:260] T 75492f86fde046c3900df3eac8292fd4 P 873357742d5843579556ee8166f23f47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:11.298102 11920 raft_consensus.cc:3060] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:04:11.378496 12063 consensus_queue.cc:1059] T 401a9cf11eac479aa445ec8c1d6e33b2 P 186b12638a87410487467df721a3cbda [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:04:11.333966 11879 raft_consensus.cc:2804] T 7757ecbf402e47bfbc93d0447741e26c P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:04:11.385620 10955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "30f628e3d7914d609e7f1d980a74c654" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:04:11.346195 11639 raft_consensus.cc:2749] T a5299b8d044f41b7b68057e63d17b159 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 13:04:11.383225 12059 raft_consensus.cc:697] T 11de93cd9b9d457489ff1cb80f976c9a P 873357742d5843579556ee8166f23f47 [term 5 LEADER]: Becoming Leader. State: Replica: 873357742d5843579556ee8166f23f47, State: Running, Role: LEADER
I20260709 13:04:11.387274 11673 raft_consensus.cc:3055] T 6a9353d1d230416296627d7cf8b1d425 P 873357742d5843579556ee8166f23f47 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:04:11.305998 12061 raft_consensus.cc:515] T 0f9df2bc281f4866b6da5da94a0defc7 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:11.348879 11981 raft_consensus.cc:3060] T e5a5f5a608504650b48fd27f163faed6 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:04:11.387745 11673 raft_consensus.cc:740] T 6a9353d1d230416296627d7cf8b1d425 P 873357742d5843579556ee8166f23f47 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 873357742d5843579556ee8166f23f47, State: Running, Role: LEADER
I20260709 13:04:11.352628 12019 leader_election.cc:290] T 8c160ddfe9b34bdaa3835d0915c1e06f P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 3 election: Requested vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
W20260709 13:04:11.383972 11887 consensus_meta.cc:287] T 1e4f2241a94a41f38e481137f868b83d P 186b12638a87410487467df721a3cbda: Time spent flushing consensus metadata: real 0.970s	user 0.014s	sys 0.005s
W20260709 13:04:11.406395 10880 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51718 (request call id 242) took 1950 ms (client timeout 2085 ms). Trace:
W20260709 13:04:11.408493 10880 rpcz_store.cc:269] 0709 13:04:09.456099 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:09.458182 (+  2083us) service_pool.cc:224] Handling call
0709 13:04:11.406150 (+1947968us) spinlock_profiling.cc:243] Waited 511 ms on lock 0x7b540033a254. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:04:11.406325 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1174462208}
I20260709 13:04:11.370632 12043 leader_election.cc:290] T 401a9cf11eac479aa445ec8c1d6e33b2 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 2 election: Requested vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:11.333678 11653 raft_consensus.cc:515] T 4f9eb06ed01f4ce68be3af8a8434665b P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:11.392460 11879 raft_consensus.cc:697] T 7757ecbf402e47bfbc93d0447741e26c P 873357742d5843579556ee8166f23f47 [term 3 LEADER]: Becoming Leader. State: Replica: 873357742d5843579556ee8166f23f47, State: Running, Role: LEADER
I20260709 13:04:11.370170 10871 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51704 (request call id 189) took 1540 ms. Trace:
I20260709 13:04:11.131007 10872 rpcz_store.cc:276] 0709 13:04:10.056196 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:10.095095 (+ 38899us) service_pool.cc:224] Handling call
0709 13:04:10.095599 (+   504us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:04:10.096199 (+   600us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:10.096264 (+    65us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:04:10.097096 (+   832us) log.cc:844] Serialized 52 byte log entry
0709 13:04:10.399821 (+302725us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:04:10.775996 (+376175us) log.cc:844] Serialized 14 byte log entry
0709 13:04:10.928744 (+152748us) spinlock_profiling.cc:243] Waited 9.94 ms on lock 0x7b1800152860. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f4263075522 00007f42630777c1 00007f4263077779 00007f4263077709 00007f42630776d1 00007f42630769cd 000000000038f354 000000000038f199 00007f426305883a
0709 13:04:11.037877 (+109133us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:11.043287 (+  5410us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:11.043893 (+   606us) raft_consensus.cc:1685] finished
0709 13:04:11.059060 (+ 15167us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:11.062652 (+  3592us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":108495,"prepare.queue_time_us":263272,"prepare.run_cpu_time_us":63,"prepare.run_wall_time_us":63,"raft.queue_time_us":149512,"raft.run_cpu_time_us":334,"raft.run_wall_time_us":20495,"spinlock_wait_cycles":22867200,"thread_start_us":567150,"threads_started":3,"wal-append.queue_time_us":421935}
I20260709 13:04:11.384461 12044 raft_consensus.cc:697] T 34064dc7bf3747dbb3876dd824ee0688 P 873357742d5843579556ee8166f23f47 [term 5 LEADER]: Becoming Leader. State: Replica: 873357742d5843579556ee8166f23f47, State: Running, Role: LEADER
I20260709 13:04:11.362141 11025 rpcz_store.cc:276] 0709 13:04:09.540587 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:09.543452 (+  2865us) service_pool.cc:224] Handling call
0709 13:04:09.601739 (+ 58287us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:10.428985 (+827246us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:11.361442 (+932457us) spinlock_profiling.cc:243] Waited 688 ms on lock 0x7b5400465354. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:04:11.361635 (+   193us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1582192384}
I20260709 13:04:11.371759 11974 raft_consensus.cc:515] T a154cc8b181f460abf4880064379948b P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:11.400480 12053 raft_consensus.cc:3060] T 75492f86fde046c3900df3eac8292fd4 P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:04:11.455372 10880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "401a9cf11eac479aa445ec8c1d6e33b2" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:04:11.407003 12101 raft_consensus.cc:493] T e89e640169e14e2d98f968f76027daa7 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:11.413408 10876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1ec2b50c6ad4f7c8033476208abb26d" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
W20260709 13:04:11.461938 10921 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:38967, user_credentials={real_user=slave}} blocked reactor thread for 111242us
I20260709 13:04:11.413483 11887 raft_consensus.cc:515] T 1e4f2241a94a41f38e481137f868b83d P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:11.406749 11673 consensus_queue.cc:260] T 6a9353d1d230416296627d7cf8b1d425 P 873357742d5843579556ee8166f23f47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:11.406940 12098 consensus_queue.cc:1048] T a610d1a71f52432e88fe56130f921fa2 P 873357742d5843579556ee8166f23f47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 0, Time since last communication: 0.099s
I20260709 13:04:11.387766 12059 consensus_queue.cc:237] T 11de93cd9b9d457489ff1cb80f976c9a P 873357742d5843579556ee8166f23f47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:11.500303 11096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c51782cd9d064ec0949e84e24dd4de67" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:11.501444 11096 raft_consensus.cc:2393] T c51782cd9d064ec0949e84e24dd4de67 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 10ca5ee68ad64b3283b9d235352c9d23 in current term 3: Already voted for candidate 0172fed22b6b4890a7fdc953fa128af4 in this term.
I20260709 13:04:11.472898 10876 raft_consensus.cc:2468] T a1ec2b50c6ad4f7c8033476208abb26d P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 1.
I20260709 13:04:11.512040 10877 raft_consensus.cc:2393] T 1e4f2241a94a41f38e481137f868b83d P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 873357742d5843579556ee8166f23f47 in current term 3: Already voted for candidate 186b12638a87410487467df721a3cbda in this term.
I20260709 13:04:11.468701 12101 raft_consensus.cc:515] T e89e640169e14e2d98f968f76027daa7 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:11.518668 12074 raft_consensus.cc:515] T 732c4adeeff6422497aa7db4a82f535b P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:11.519789 12101 leader_election.cc:290] T e89e640169e14e2d98f968f76027daa7 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:11.523188 11983 leader_election.cc:290] T 1e4f2241a94a41f38e481137f868b83d P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:11.525538 12044 consensus_queue.cc:237] T 34064dc7bf3747dbb3876dd824ee0688 P 873357742d5843579556ee8166f23f47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:11.485517 10949 raft_consensus.cc:1240] T a5299b8d044f41b7b68057e63d17b159 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Rejecting Update request from peer 186b12638a87410487467df721a3cbda for earlier term 1. Current term is 2. Ops: [1.1-1.1]
I20260709 13:04:11.529501 11673 raft_consensus.cc:3060] T 6a9353d1d230416296627d7cf8b1d425 P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:04:11.530249 11887 leader_election.cc:290] T 1e4f2241a94a41f38e481137f868b83d P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 3 election: Requested vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:11.534413 12074 leader_election.cc:290] T 732c4adeeff6422497aa7db4a82f535b P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 3 election: Requested vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:11.477584 10874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "30f628e3d7914d609e7f1d980a74c654" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:11.527621 11653 leader_election.cc:290] T 4f9eb06ed01f4ce68be3af8a8434665b P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 3 election: Requested vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:11.504034 11879 consensus_queue.cc:237] T 7757ecbf402e47bfbc93d0447741e26c P 873357742d5843579556ee8166f23f47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:11.550292 11852 raft_consensus.cc:3055] T d71ca3a1ba6b48478aaf62741bf3cf6a P 0172fed22b6b4890a7fdc953fa128af4 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:04:11.550830 11852 raft_consensus.cc:740] T d71ca3a1ba6b48478aaf62741bf3cf6a P 0172fed22b6b4890a7fdc953fa128af4 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0172fed22b6b4890a7fdc953fa128af4, State: Running, Role: LEADER
I20260709 13:04:11.551064 12096 raft_consensus.cc:515] T a977842ba4b744f4bb2febd55d286006 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:11.551812 11852 consensus_queue.cc:260] T d71ca3a1ba6b48478aaf62741bf3cf6a P 0172fed22b6b4890a7fdc953fa128af4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:11.552840 11852 raft_consensus.cc:3060] T d71ca3a1ba6b48478aaf62741bf3cf6a P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:04:11.554117 11974 leader_election.cc:290] T a154cc8b181f460abf4880064379948b P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:11.555126 11926 raft_consensus.cc:493] T f371d2565dd84667b5613848604949bb P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:11.555765 11099 raft_consensus.cc:2468] T f34f2bc02d85444f9096e0f5b4fe4331 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 1.
I20260709 13:04:11.555720 11926 raft_consensus.cc:515] T f371d2565dd84667b5613848604949bb P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:11.557683 12096 leader_election.cc:290] T a977842ba4b744f4bb2febd55d286006 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 2 election: Requested vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:11.562053 10949 raft_consensus.cc:1240] T 63d44f6ba6434ab2870fe3190f4d7c2f P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Rejecting Update request from peer 186b12638a87410487467df721a3cbda for earlier term 2. Current term is 3. Ops: []
I20260709 13:04:11.547644 10946 raft_consensus.cc:2468] T 34064dc7bf3747dbb3876dd824ee0688 P 10ca5ee68ad64b3283b9d235352c9d23 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 5.
W20260709 13:04:11.586172 12126 log.cc:927] Time spent T 9a1716c0adf34bee8a4e99341ba5788f P 186b12638a87410487467df721a3cbda: Append to log took a long time: real 0.070s	user 0.000s	sys 0.001s
I20260709 13:04:11.588565 11926 leader_election.cc:290] T f371d2565dd84667b5613848604949bb P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:11.616629 10874 raft_consensus.cc:2393] T 30f628e3d7914d609e7f1d980a74c654 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 873357742d5843579556ee8166f23f47 in current term 1: Already voted for candidate 186b12638a87410487467df721a3cbda in this term.
I20260709 13:04:11.561192 12101 consensus_queue.cc:1048] T c31881534c5e4f0089c82057c1746652 P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.061s
I20260709 13:04:11.622711 12111 raft_consensus.cc:2749] T 34064dc7bf3747dbb3876dd824ee0688 P 10ca5ee68ad64b3283b9d235352c9d23 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:04:11.627272 12061 leader_election.cc:290] T 0f9df2bc281f4866b6da5da94a0defc7 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 2 election: Requested vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:11.619655 11974 raft_consensus.cc:493] T 5de5f0581e3b41dea729363d6ca563bb P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 0172fed22b6b4890a7fdc953fa128af4)
I20260709 13:04:11.641075 12063 raft_consensus.cc:493] T 6a9353d1d230416296627d7cf8b1d425 P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:11.632061 10952 raft_consensus.cc:2468] T b4cd3d7dfbbf4096908ff11e9dc57141 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 4.
I20260709 13:04:11.650133 10949 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49428 (request call id 155) took 1006 ms. Trace:
W20260709 13:04:11.662066 12088 consensus_meta.cc:287] T c51782cd9d064ec0949e84e24dd4de67 P 186b12638a87410487467df721a3cbda: Time spent flushing consensus metadata: real 0.630s	user 0.012s	sys 0.009s
W20260709 13:04:11.670526 12131 log.cc:927] Time spent T b67eff0634e540eab91cf8cee795969f P 0172fed22b6b4890a7fdc953fa128af4: Append to log took a long time: real 0.152s	user 0.000s	sys 0.002s
I20260709 13:04:11.416332 10871 rpcz_store.cc:276] 0709 13:04:09.811745 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:10.090187 (+278442us) service_pool.cc:224] Handling call
0709 13:04:10.091844 (+  1657us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:10.092146 (+   302us) raft_consensus.cc:1534] Early marking committed up to index 2
0709 13:04:10.377677 (+285531us) log.cc:844] Serialized 14 byte log entry
0709 13:04:11.054107 (+676430us) spinlock_profiling.cc:243] Waited 11.8 ms on lock 0x7b1800284260. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f4265d0a4bc 00007f426305848b 00007f426beb2ab9 00007f426307469b 00007f426307293f 00007f4263074c02 00007f4265cfc20b 00007f4265d01d23 00007f4265d0223a
0709 13:04:11.337558 (+283451us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:11.337633 (+    75us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:04:11.337687 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:11.337789 (+   102us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:11.351739 (+ 13950us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":340754,"prepare.run_cpu_time_us":66,"prepare.run_wall_time_us":68,"raft.queue_time_us":140691,"raft.run_cpu_time_us":131,"raft.run_wall_time_us":132,"spinlock_wait_cycles":27057920,"thread_start_us":527914,"threads_started":3,"wal-append.queue_time_us":277699}
I20260709 13:04:11.674669 12128 raft_consensus.cc:493] T 57b6a2418c1342e7924195e9c1dbd357 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:11.681695 10873 raft_consensus.cc:2468] T f34f2bc02d85444f9096e0f5b4fe4331 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 1.
I20260709 13:04:11.683849 12128 raft_consensus.cc:515] T 57b6a2418c1342e7924195e9c1dbd357 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:11.692643 12128 leader_election.cc:290] T 57b6a2418c1342e7924195e9c1dbd357 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:11.662899 11974 raft_consensus.cc:515] T 5de5f0581e3b41dea729363d6ca563bb P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:11.668992 12095 raft_consensus.cc:515] T d7ce47b9a92d43dd88d95bcda5038597 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:11.695369 11974 leader_election.cc:290] T 5de5f0581e3b41dea729363d6ca563bb P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:11.697389 11981 raft_consensus.cc:515] T e5a5f5a608504650b48fd27f163faed6 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:11.698211 11795 raft_consensus.cc:3055] T 401a9cf11eac479aa445ec8c1d6e33b2 P 186b12638a87410487467df721a3cbda [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:04:11.698586 10950 raft_consensus.cc:2468] T 11de93cd9b9d457489ff1cb80f976c9a P 10ca5ee68ad64b3283b9d235352c9d23 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 5.
I20260709 13:04:11.699769 11981 leader_election.cc:290] T e5a5f5a608504650b48fd27f163faed6 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 4 election: Requested vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:11.698671 11795 raft_consensus.cc:740] T 401a9cf11eac479aa445ec8c1d6e33b2 P 186b12638a87410487467df721a3cbda [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 186b12638a87410487467df721a3cbda, State: Running, Role: LEADER
I20260709 13:04:11.700286 11098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555217d45d9b4ee492dbb3ddaf0f7c02" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:11.701480 12103 raft_consensus.cc:515] T 7839ca2d2c13431a85f44d7c458c6351 P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:11.701933 11098 raft_consensus.cc:2468] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 3.
I20260709 13:04:11.701898 11795 consensus_queue.cc:260] T 401a9cf11eac479aa445ec8c1d6e33b2 P 186b12638a87410487467df721a3cbda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:11.703337 11795 raft_consensus.cc:3060] T 401a9cf11eac479aa445ec8c1d6e33b2 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:04:11.703871 12103 leader_election.cc:290] T 7839ca2d2c13431a85f44d7c458c6351 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 3 election: Requested vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:11.706890 12004 raft_consensus.cc:3055] T a610d1a71f52432e88fe56130f921fa2 P 0172fed22b6b4890a7fdc953fa128af4 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:04:11.707329 12004 raft_consensus.cc:740] T a610d1a71f52432e88fe56130f921fa2 P 0172fed22b6b4890a7fdc953fa128af4 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 0172fed22b6b4890a7fdc953fa128af4, State: Running, Role: LEADER
I20260709 13:04:11.708278 12004 consensus_queue.cc:260] T a610d1a71f52432e88fe56130f921fa2 P 0172fed22b6b4890a7fdc953fa128af4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:11.709455 12004 raft_consensus.cc:3060] T a610d1a71f52432e88fe56130f921fa2 P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:04:11.710170 10950 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49428 (request call id 159) took 1007 ms. Trace:
I20260709 13:04:11.710623 10950 rpcz_store.cc:276] 0709 13:04:10.702508 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:11.699547 (+997039us) service_pool.cc:224] Handling call
0709 13:04:11.699962 (+   415us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:11.700216 (+   254us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:11.700248 (+    32us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:11.700311 (+    63us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:04:11.709879 (+  9568us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:11.709972 (+    93us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:11.710101 (+   129us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:11.665132 12063 raft_consensus.cc:515] T 6a9353d1d230416296627d7cf8b1d425 P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:11.711431 12050 raft_consensus.cc:493] T 0e5d327630e5438b878035d28efe6c0c P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:11.711954 12050 raft_consensus.cc:515] T 0e5d327630e5438b878035d28efe6c0c P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:11.714143 12050 leader_election.cc:290] T 0e5d327630e5438b878035d28efe6c0c P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:11.715065 12095 leader_election.cc:290] T d7ce47b9a92d43dd88d95bcda5038597 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 2 election: Requested vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
W20260709 13:04:11.715801 11069 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:11.716248 11069 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:11.716432 10879 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51696 (request call id 235) took 1644 ms. Trace:
W20260709 13:04:11.718746 10921 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:38967, user_credentials={real_user=slave}} blocked reactor thread for 239997us
I20260709 13:04:11.721396 10878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e4f2241a94a41f38e481137f868b83d" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
W20260709 13:04:11.722606 10994 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:11.725190 10921 leader_election.cc:304] T f34f2bc02d85444f9096e0f5b4fe4331 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda; no voters: 
W20260709 13:04:11.673700 12049 consensus_meta.cc:287] T 87cd8f5097624c8692f1d83544d8881e P 10ca5ee68ad64b3283b9d235352c9d23: Time spent flushing consensus metadata: real 1.028s	user 0.016s	sys 0.002s
I20260709 13:04:11.727280 10876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e89e640169e14e2d98f968f76027daa7" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:11.728874 10872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57b6a2418c1342e7924195e9c1dbd357" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:11.729007 10873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e5d327630e5438b878035d28efe6c0c" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:11.727258 10871 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c160ddfe9b34bdaa3835d0915c1e06f" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:04:11.729743 10873 raft_consensus.cc:2468] T 0e5d327630e5438b878035d28efe6c0c P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 4.
I20260709 13:04:11.731684 10994 leader_election.cc:304] T 0e5d327630e5438b878035d28efe6c0c P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47; no voters: 186b12638a87410487467df721a3cbda
I20260709 13:04:11.727880 10875 raft_consensus.cc:1240] T 732c4adeeff6422497aa7db4a82f535b P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Rejecting Update request from peer 10ca5ee68ad64b3283b9d235352c9d23 for earlier term 2. Current term is 3. Ops: []
I20260709 13:04:11.728354 10880 raft_consensus.cc:1240] T c31881534c5e4f0089c82057c1746652 P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Rejecting Update request from peer 10ca5ee68ad64b3283b9d235352c9d23 for earlier term 2. Current term is 3. Ops: [2.1-2.1]
I20260709 13:04:11.728205 10874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f371d2565dd84667b5613848604949bb" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:11.733282 10874 raft_consensus.cc:2410] T f371d2565dd84667b5613848604949bb P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0172fed22b6b4890a7fdc953fa128af4 for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 13:04:11.669143 10949 rpcz_store.cc:276] 0709 13:04:10.637871 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:11.528807 (+890936us) service_pool.cc:224] Handling call
0709 13:04:11.529202 (+   395us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:11.627671 (+ 98469us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:11.643774 (+ 16103us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:11.727074 12091 raft_consensus.cc:2804] T b67eff0634e540eab91cf8cee795969f P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:04:11.729739 10872 raft_consensus.cc:2393] T 57b6a2418c1342e7924195e9c1dbd357 P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 10ca5ee68ad64b3283b9d235352c9d23 in current term 3: Already voted for candidate 186b12638a87410487467df721a3cbda in this term.
I20260709 13:04:11.727469 12075 consensus_queue.cc:1048] T 30f628e3d7914d609e7f1d980a74c654 P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:04:11.726418 11069 leader_election.cc:304] T d7ce47b9a92d43dd88d95bcda5038597 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda
I20260709 13:04:11.727922 10878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5de5f0581e3b41dea729363d6ca563bb" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:11.716848 10879 rpcz_store.cc:276] 0709 13:04:10.071879 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:10.536584 (+464705us) service_pool.cc:224] Handling call
0709 13:04:10.537017 (+   433us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:10.669999 (+132982us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:10.670036 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:10.681900 (+ 11864us) log.cc:844] Serialized 26 byte log entry
0709 13:04:11.715771 (+1033871us) spinlock_profiling.cc:243] Waited 11.5 ms on lock 0x7b18003dda40. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f4265d0a4bc 00007f426305848b 00007f426beb2ab9 00007f426307469b 00007f426307293f 00007f4263074c02 00007f4265cfc20b 00007f4265d01d23 00007f4265d020d1
0709 13:04:11.715940 (+   169us) raft_consensus.cc:1645] Marking committed up to 0
0709 13:04:11.715979 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:11.716064 (+    85us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:11.716113 (+    49us) raft_consensus.cc:1685] finished
0709 13:04:11.716125 (+    12us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:11.716366 (+   241us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":26331,"raft.queue_time_us":418884,"raft.run_cpu_time_us":153,"raft.run_wall_time_us":5080,"spinlock_wait_cycles":26541312,"thread_start_us":198902,"threads_started":2,"wal-append.queue_time_us":88377}
I20260709 13:04:11.747267 10878 raft_consensus.cc:2468] T 5de5f0581e3b41dea729363d6ca563bb P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 3.
I20260709 13:04:11.745635 11974 raft_consensus.cc:2804] T f34f2bc02d85444f9096e0f5b4fe4331 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:04:11.736491 12049 raft_consensus.cc:515] T 87cd8f5097624c8692f1d83544d8881e P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:11.750517 10953 raft_consensus.cc:2393] T 87cd8f5097624c8692f1d83544d8881e P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0172fed22b6b4890a7fdc953fa128af4 in current term 4: Already voted for candidate 10ca5ee68ad64b3283b9d235352c9d23 in this term.
I20260709 13:04:11.750907 12128 consensus_queue.cc:1059] T 732c4adeeff6422497aa7db4a82f535b P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:04:11.740728 12087 raft_consensus.cc:2804] T 0e5d327630e5438b878035d28efe6c0c P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:04:11.752079 12087 raft_consensus.cc:493] T 0e5d327630e5438b878035d28efe6c0c P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 0172fed22b6b4890a7fdc953fa128af4)
I20260709 13:04:11.752485 12087 raft_consensus.cc:3060] T 0e5d327630e5438b878035d28efe6c0c P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:04:11.752394 10949 raft_consensus.cc:1240] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 10ca5ee68ad64b3283b9d235352c9d23 [term 5 FOLLOWER]: Rejecting Update request from peer 873357742d5843579556ee8166f23f47 for earlier term 4. Current term is 5. Ops: []
W20260709 13:04:11.753060 12060 log.cc:927] Time spent T 7757ecbf402e47bfbc93d0447741e26c P 0172fed22b6b4890a7fdc953fa128af4: Append to log took a long time: real 0.823s	user 0.004s	sys 0.000s
I20260709 13:04:11.755537 10874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5a5f5a608504650b48fd27f163faed6" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:04:11.755565 11974 raft_consensus.cc:697] T f34f2bc02d85444f9096e0f5b4fe4331 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 LEADER]: Becoming Leader. State: Replica: 10ca5ee68ad64b3283b9d235352c9d23, State: Running, Role: LEADER
I20260709 13:04:11.756244 10874 raft_consensus.cc:3060] T e5a5f5a608504650b48fd27f163faed6 P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:04:11.757138 11974 consensus_queue.cc:237] T f34f2bc02d85444f9096e0f5b4fe4331 P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:11.758167 12116 raft_consensus.cc:3055] T 5d439326f23747eaab3c1e7da2d6b785 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:04:11.758589 12116 raft_consensus.cc:740] T 5d439326f23747eaab3c1e7da2d6b785 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 10ca5ee68ad64b3283b9d235352c9d23, State: Running, Role: LEADER
I20260709 13:04:11.759598 12116 consensus_queue.cc:260] T 5d439326f23747eaab3c1e7da2d6b785 P 10ca5ee68ad64b3283b9d235352c9d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:11.760954 12116 raft_consensus.cc:3060] T 5d439326f23747eaab3c1e7da2d6b785 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:04:11.755129 11981 consensus_queue.cc:1059] T c31881534c5e4f0089c82057c1746652 P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.006s
I20260709 13:04:11.725082 12063 leader_election.cc:290] T 6a9353d1d230416296627d7cf8b1d425 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:11.764514 11069 leader_election.cc:304] T d371c5e27bd843469d2e2e453734cb4a P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4; no voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47
I20260709 13:04:11.762916 10921 leader_election.cc:304] T 5de5f0581e3b41dea729363d6ca563bb P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda; no voters: 
I20260709 13:04:11.769213 11943 raft_consensus.cc:2749] T d7ce47b9a92d43dd88d95bcda5038597 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:04:11.769948 10920 leader_election.cc:304] T c51782cd9d064ec0949e84e24dd4de67 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23; no voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda
I20260709 13:04:11.742867 12044 consensus_queue.cc:1059] T ff13ae9a47c7455f962406c6d131a3ff P 873357742d5843579556ee8166f23f47 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 }, Status: INVALID_TERM, Last received: 2.1, Next index: 2, Last known committed idx: 0, Time since last communication: 0.027s
I20260709 13:04:11.770849 11961 raft_consensus.cc:493] T 8a69f32c53f24a50b7852d143665d7b4 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 186b12638a87410487467df721a3cbda)
I20260709 13:04:11.770990 12044 raft_consensus.cc:3055] T ff13ae9a47c7455f962406c6d131a3ff P 873357742d5843579556ee8166f23f47 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:04:11.771541 12044 raft_consensus.cc:740] T ff13ae9a47c7455f962406c6d131a3ff P 873357742d5843579556ee8166f23f47 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 873357742d5843579556ee8166f23f47, State: Running, Role: LEADER
I20260709 13:04:11.771435 11961 raft_consensus.cc:515] T 8a69f32c53f24a50b7852d143665d7b4 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:11.774780 11639 raft_consensus.cc:3060] T c51782cd9d064ec0949e84e24dd4de67 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:04:11.776860 12127 raft_consensus.cc:2804] T 5de5f0581e3b41dea729363d6ca563bb P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:04:11.777273 12127 raft_consensus.cc:493] T 5de5f0581e3b41dea729363d6ca563bb P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 0172fed22b6b4890a7fdc953fa128af4)
I20260709 13:04:11.777653 12127 raft_consensus.cc:3060] T 5de5f0581e3b41dea729363d6ca563bb P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:04:11.725768 11852 consensus_queue.cc:1048] T 7757ecbf402e47bfbc93d0447741e26c P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.037s
I20260709 13:04:11.770002 11982 raft_consensus.cc:2749] T d371c5e27bd843469d2e2e453734cb4a P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 13:04:11.787392 11069 leader_election.cc:304] T acc241e7273d49149984968596242712 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4; no voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47
I20260709 13:04:11.752399 12049 leader_election.cc:290] T 87cd8f5097624c8692f1d83544d8881e P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 4 election: Requested vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:11.772737 12044 consensus_queue.cc:260] T ff13ae9a47c7455f962406c6d131a3ff P 873357742d5843579556ee8166f23f47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:11.790624 11069 leader_election.cc:304] T 6a9353d1d230416296627d7cf8b1d425 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda; no voters: 873357742d5843579556ee8166f23f47
I20260709 13:04:11.783031 11981 raft_consensus.cc:3055] T c31881534c5e4f0089c82057c1746652 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:04:11.789961 11982 raft_consensus.cc:2749] T acc241e7273d49149984968596242712 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:04:11.791071 10875 raft_consensus.cc:1240] T d7ce47b9a92d43dd88d95bcda5038597 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Rejecting Update request from peer 10ca5ee68ad64b3283b9d235352c9d23 for earlier term 1. Current term is 2. Ops: []
I20260709 13:04:11.801797 10872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87cd8f5097624c8692f1d83544d8881e" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:04:11.799957 11981 raft_consensus.cc:740] T c31881534c5e4f0089c82057c1746652 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 10ca5ee68ad64b3283b9d235352c9d23, State: Running, Role: LEADER
I20260709 13:04:11.805097 11852 raft_consensus.cc:2804] T 6a9353d1d230416296627d7cf8b1d425 P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:04:11.806100 11069 leader_election.cc:304] T b67eff0634e540eab91cf8cee795969f P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda; no voters: 10ca5ee68ad64b3283b9d235352c9d23
I20260709 13:04:11.806665 12044 raft_consensus.cc:3060] T ff13ae9a47c7455f962406c6d131a3ff P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:04:11.820884 11069 leader_election.cc:304] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda; no voters: 10ca5ee68ad64b3283b9d235352c9d23
I20260709 13:04:11.825944 11069 leader_election.cc:304] T c51782cd9d064ec0949e84e24dd4de67 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda
I20260709 13:04:11.826970 11926 raft_consensus.cc:2804] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:04:11.829773 11974 raft_consensus.cc:493] T 4b37f8fca8dc42718b2a42f0883a956f P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:11.830299 11926 raft_consensus.cc:697] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 LEADER]: Becoming Leader. State: Replica: 0172fed22b6b4890a7fdc953fa128af4, State: Running, Role: LEADER
I20260709 13:04:11.830865 12145 raft_consensus.cc:493] T 32147c6d2d5a477aa38c597c7f78df5b P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:11.831686 12049 consensus_queue.cc:1059] T d7ce47b9a92d43dd88d95bcda5038597 P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:04:11.831472 11926 consensus_queue.cc:237] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
W20260709 13:04:11.832535 10953 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49424 (request call id 180) took 1974 ms (client timeout 2194 ms). Trace:
I20260709 13:04:11.832870 12049 raft_consensus.cc:3055] T d7ce47b9a92d43dd88d95bcda5038597 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:04:11.833374 12049 raft_consensus.cc:740] T d7ce47b9a92d43dd88d95bcda5038597 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 10ca5ee68ad64b3283b9d235352c9d23, State: Running, Role: LEADER
I20260709 13:04:11.833899 10992 leader_election.cc:304] T 1e4f2241a94a41f38e481137f868b83d P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:04:11.822072 11982 raft_consensus.cc:2804] T b67eff0634e540eab91cf8cee795969f P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:04:11.833909 10955 raft_consensus.cc:1240] T 87cd8f5097624c8692f1d83544d8881e P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Rejecting Update request from peer 186b12638a87410487467df721a3cbda for earlier term 3. Current term is 4. Ops: []
I20260709 13:04:11.835322 11982 raft_consensus.cc:493] T b67eff0634e540eab91cf8cee795969f P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 10ca5ee68ad64b3283b9d235352c9d23)
I20260709 13:04:11.835813 11982 raft_consensus.cc:3060] T b67eff0634e540eab91cf8cee795969f P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:04:11.837510 10955 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49428 (request call id 154) took 1200 ms. Trace:
I20260709 13:04:11.829551 11943 raft_consensus.cc:2749] T c51782cd9d064ec0949e84e24dd4de67 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 13:04:11.822664 11852 raft_consensus.cc:697] T 6a9353d1d230416296627d7cf8b1d425 P 0172fed22b6b4890a7fdc953fa128af4 [term 4 LEADER]: Becoming Leader. State: Replica: 0172fed22b6b4890a7fdc953fa128af4, State: Running, Role: LEADER
I20260709 13:04:11.837893 10955 rpcz_store.cc:276] 0709 13:04:10.637576 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:11.395317 (+757741us) service_pool.cc:224] Handling call
0709 13:04:11.833641 (+438324us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:11.835550 (+  1909us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:11.837321 (+  1771us) spinlock_profiling.cc:243] Waited 438 ms on lock 0x7b5400415850. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661618ad 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:04:11.837447 (+   126us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1006899968}
W20260709 13:04:11.832973 10953 rpcz_store.cc:269] 0709 13:04:09.857694 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:10.036296 (+178602us) service_pool.cc:224] Handling call
0709 13:04:10.386667 (+350371us) spinlock_profiling.cc:243] Waited 12.5 ms on lock 0x7b580001d198. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f426303db71 00007f426303da22 00007f426bf56c09 00007f426b0b0ee0 00007f426b17a0ab 00007f426b17a131 00007f426b1280a0 00007f426b127984 00007f42646a26d4 00007f42646a2662
0709 13:04:11.832293 (+1445626us) spinlock_profiling.cc:243] Waited 1.34 s on lock 0x7b5400415854. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:04:11.832466 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3119357184}
I20260709 13:04:11.844359 10955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a64b7e99d6c54934a7de6f6dffcf5e94" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:11.830556 11974 raft_consensus.cc:515] T 4b37f8fca8dc42718b2a42f0883a956f P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:11.834338 12049 consensus_queue.cc:260] T d7ce47b9a92d43dd88d95bcda5038597 P 10ca5ee68ad64b3283b9d235352c9d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:11.852403 12049 raft_consensus.cc:3060] T d7ce47b9a92d43dd88d95bcda5038597 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Advancing to term 2
W20260709 13:04:11.854378 10843 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:11.854859 10871 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7839ca2d2c13431a85f44d7c458c6351" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:04:11.855847 10871 raft_consensus.cc:3060] T 7839ca2d2c13431a85f44d7c458c6351 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:04:11.856056 10953 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7839ca2d2c13431a85f44d7c458c6351" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:04:11.857064 10953 raft_consensus.cc:3060] T 7839ca2d2c13431a85f44d7c458c6351 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:04:11.831485 12145 raft_consensus.cc:515] T 32147c6d2d5a477aa38c597c7f78df5b P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:11.864856 10992 leader_election.cc:304] T e5ba72ccd854430baadbc636dfd17136 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 873357742d5843579556ee8166f23f47; no voters: 0172fed22b6b4890a7fdc953fa128af4, 10ca5ee68ad64b3283b9d235352c9d23
I20260709 13:04:11.866083 11069 leader_election.cc:304] T 72b111ad7b5f4753ab1d2b5ee1114840 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda; no voters: 
W20260709 13:04:11.865021 10843 leader_election.cc:336] T 1e4f2241a94a41f38e481137f868b83d P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903): Timed out: RequestConsensusVote RPC to 127.0.107.194:44903 timed out after 1.557s (SENT)
I20260709 13:04:11.840845 11879 raft_consensus.cc:493] T 732c4adeeff6422497aa7db4a82f535b P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 10ca5ee68ad64b3283b9d235352c9d23)
I20260709 13:04:11.868626 11879 raft_consensus.cc:515] T 732c4adeeff6422497aa7db4a82f535b P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:11.869623 12145 leader_election.cc:290] T 32147c6d2d5a477aa38c597c7f78df5b P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:11.870262 10951 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c31881534c5e4f0089c82057c1746652" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:11.870970 11879 leader_election.cc:290] T 732c4adeeff6422497aa7db4a82f535b P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:11.871565 11943 raft_consensus.cc:2804] T 72b111ad7b5f4753ab1d2b5ee1114840 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:04:11.872009 11943 raft_consensus.cc:493] T 72b111ad7b5f4753ab1d2b5ee1114840 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 10ca5ee68ad64b3283b9d235352c9d23)
I20260709 13:04:11.872391 11943 raft_consensus.cc:3060] T 72b111ad7b5f4753ab1d2b5ee1114840 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:04:11.873193 11025 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e4f2241a94a41f38e481137f868b83d" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
I20260709 13:04:11.873847 11025 raft_consensus.cc:3060] T 1e4f2241a94a41f38e481137f868b83d P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:04:11.875114 11023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "732c4adeeff6422497aa7db4a82f535b" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
I20260709 13:04:11.875785 11023 raft_consensus.cc:3060] T 732c4adeeff6422497aa7db4a82f535b P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:04:11.877308 11028 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a9353d1d230416296627d7cf8b1d425" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:11.878572 11028 raft_consensus.cc:2410] T 6a9353d1d230416296627d7cf8b1d425 P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 186b12638a87410487467df721a3cbda for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260709 13:04:11.880008 11020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e5d327630e5438b878035d28efe6c0c" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:11.877527 12043 raft_consensus.cc:493] T 7709387f4d3d4f9992d97a0fbca39a40 P 873357742d5843579556ee8166f23f47 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:11.881835 12043 raft_consensus.cc:515] T 7709387f4d3d4f9992d97a0fbca39a40 P 873357742d5843579556ee8166f23f47 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:11.883239 12103 raft_consensus.cc:493] T 2567c2d84e4648d7ba5dc23a3df32817 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 10ca5ee68ad64b3283b9d235352c9d23)
I20260709 13:04:11.883842 12103 raft_consensus.cc:515] T 2567c2d84e4648d7ba5dc23a3df32817 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:11.859831 11981 consensus_queue.cc:260] T c31881534c5e4f0089c82057c1746652 P 10ca5ee68ad64b3283b9d235352c9d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 2.1, Last appended by leader: 1, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
W20260709 13:04:11.885710 10954 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49424 (request call id 173) took 2754 ms (client timeout 1748 ms). Trace:
W20260709 13:04:11.886207 10954 rpcz_store.cc:269] 0709 13:04:09.130905 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:09.235251 (+104346us) service_pool.cc:224] Handling call
0709 13:04:11.885426 (+2650175us) spinlock_profiling.cc:243] Waited 1.69 s on lock 0x7b54000b6154. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:04:11.885593 (+   167us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3898078464}
I20260709 13:04:11.886632 11981 raft_consensus.cc:3060] T c31881534c5e4f0089c82057c1746652 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:04:11.887530 10954 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57b6a2418c1342e7924195e9c1dbd357" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:04:11.892517 10955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "836251ebc2b74f5da88b8a8b2338e2a5" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
W20260709 13:04:11.895315 10843 leader_election.cc:336] T e96e80ba9ac34d0998395515e5c8c41a P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903): Timed out: RequestConsensusVote RPC to 127.0.107.194:44903 timed out after 1.877s (SENT)
I20260709 13:04:11.896143 10843 leader_election.cc:304] T e96e80ba9ac34d0998395515e5c8c41a P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 186b12638a87410487467df721a3cbda; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47
I20260709 13:04:11.896154 10955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2567c2d84e4648d7ba5dc23a3df32817" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:11.874104 10995 leader_election.cc:304] T a1ec2b50c6ad4f7c8033476208abb26d P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:04:11.906916 10995 leader_election.cc:304] T 30f628e3d7914d609e7f1d980a74c654 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 873357742d5843579556ee8166f23f47; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda
I20260709 13:04:11.907224 10955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e4f2241a94a41f38e481137f868b83d" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:04:11.908551 10955 raft_consensus.cc:3060] T 1e4f2241a94a41f38e481137f868b83d P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:04:11.888181 10954 raft_consensus.cc:3060] T 57b6a2418c1342e7924195e9c1dbd357 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:04:11.829043 11920 raft_consensus.cc:515] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:11.912654 11069 leader_election.cc:304] T 0e5d327630e5438b878035d28efe6c0c P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda; no voters: 
W20260709 13:04:11.919020 10992 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 54327us
I20260709 13:04:11.865763 10872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f9eb06ed01f4ce68be3af8a8434665b" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
W20260709 13:04:11.933641 10843 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 65538.4us
I20260709 13:04:11.926190 10953 raft_consensus.cc:2468] T 7839ca2d2c13431a85f44d7c458c6351 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 3.
I20260709 13:04:11.911348 12096 consensus_queue.cc:1048] T a1ec2b50c6ad4f7c8033476208abb26d P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.022s
I20260709 13:04:11.936654 10953 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f9df2bc281f4866b6da5da94a0defc7" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
W20260709 13:04:11.939426 11067 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:11.939931 10953 raft_consensus.cc:3060] T 0f9df2bc281f4866b6da5da94a0defc7 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:04:11.904420 11961 leader_election.cc:290] T 8a69f32c53f24a50b7852d143665d7b4 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:11.904908 11795 raft_consensus.cc:2749] T e96e80ba9ac34d0998395515e5c8c41a P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:04:11.944490 11028 raft_consensus.cc:1275] T 7709387f4d3d4f9992d97a0fbca39a40 P 873357742d5843579556ee8166f23f47 [term 5 FOLLOWER]: Refusing update from remote peer 0172fed22b6b4890a7fdc953fa128af4: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260709 13:04:11.921448 11921 raft_consensus.cc:2804] T a1ec2b50c6ad4f7c8033476208abb26d P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:04:11.911298 12076 raft_consensus.cc:2749] T e5ba72ccd854430baadbc636dfd17136 P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 13:04:11.948885 11102 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f9df2bc281f4866b6da5da94a0defc7" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
W20260709 13:04:11.888044 12073 consensus_queue.cc:1175] T c31881534c5e4f0089c82057c1746652 P 10ca5ee68ad64b3283b9d235352c9d23 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "0172fed22b6b4890a7fdc953fa128af4" responder_term: 2 status { last_received { term: 2 index: 1 } last_committed_idx: 0 last_received_current_leader { term: 2 index: 1 } }
I20260709 13:04:11.920817 12059 raft_consensus.cc:2749] T 30f628e3d7914d609e7f1d980a74c654 P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:04:11.925916 11639 raft_consensus.cc:2749] T c51782cd9d064ec0949e84e24dd4de67 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:04:11.921653 12091 raft_consensus.cc:493] T b67eff0634e540eab91cf8cee795969f P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Starting leader election (detected failure of leader 10ca5ee68ad64b3283b9d235352c9d23)
I20260709 13:04:11.951570 10953 raft_consensus.cc:2468] T 0f9df2bc281f4866b6da5da94a0defc7 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 2.
I20260709 13:04:11.938931 12019 raft_consensus.cc:2804] T 0e5d327630e5438b878035d28efe6c0c P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:04:11.966967 10953 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72b111ad7b5f4753ab1d2b5ee1114840" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:11.920274 12066 consensus_queue.cc:1048] T 7709387f4d3d4f9992d97a0fbca39a40 P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.031s
I20260709 13:04:11.979171 10953 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "732c4adeeff6422497aa7db4a82f535b" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:11.980635 10953 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "732c4adeeff6422497aa7db4a82f535b" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:04:11.949301 10873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a69f32c53f24a50b7852d143665d7b4" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:11.987743 10953 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a977842ba4b744f4bb2febd55d286006" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:04:11.990211 11029 raft_consensus.cc:1240] T a610d1a71f52432e88fe56130f921fa2 P 873357742d5843579556ee8166f23f47 [term 5 LEADER]: Rejecting Update request from peer 0172fed22b6b4890a7fdc953fa128af4 for earlier term 4. Current term is 5. Ops: []
I20260709 13:04:11.957218 11921 raft_consensus.cc:493] T a1ec2b50c6ad4f7c8033476208abb26d P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:11.958112 11099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d7ce47b9a92d43dd88d95bcda5038597" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:04:11.995297 10953 raft_consensus.cc:3060] T a977842ba4b744f4bb2febd55d286006 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:04:11.996938 11102 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1ec2b50c6ad4f7c8033476208abb26d" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
W20260709 13:04:11.998575 11067 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:45807, user_credentials={real_user=slave}} blocked reactor thread for 57934.5us
I20260709 13:04:11.993005 10880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "732c4adeeff6422497aa7db4a82f535b" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:12.003530 11102 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f9eb06ed01f4ce68be3af8a8434665b" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:04:11.949586 12089 consensus_queue.cc:1059] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 873357742d5843579556ee8166f23f47 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.038s
I20260709 13:04:11.974673 12019 raft_consensus.cc:493] T 0e5d327630e5438b878035d28efe6c0c P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:11.972585 12091 raft_consensus.cc:3060] T b67eff0634e540eab91cf8cee795969f P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:04:12.007586 10880 raft_consensus.cc:2393] T 732c4adeeff6422497aa7db4a82f535b P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 873357742d5843579556ee8166f23f47 in current term 3: Already voted for candidate 186b12638a87410487467df721a3cbda in this term.
W20260709 13:04:12.010831 10843 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 76199.5us
I20260709 13:04:12.012570 11102 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a9353d1d230416296627d7cf8b1d425" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:11.987581 11021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb7d67be500947b3aef4c4245e33671c" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:12.020175 11029 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e5d327630e5438b878035d28efe6c0c" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:12.014350 10843 leader_election.cc:304] T 5d439326f23747eaab3c1e7da2d6b785 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 186b12638a87410487467df721a3cbda; no voters: 0172fed22b6b4890a7fdc953fa128af4, 10ca5ee68ad64b3283b9d235352c9d23
I20260709 13:04:12.001402 11026 raft_consensus.cc:1240] T d71ca3a1ba6b48478aaf62741bf3cf6a P 873357742d5843579556ee8166f23f47 [term 3 LEADER]: Rejecting Update request from peer 0172fed22b6b4890a7fdc953fa128af4 for earlier term 2. Current term is 3. Ops: []
I20260709 13:04:12.000860 11921 raft_consensus.cc:3060] T a1ec2b50c6ad4f7c8033476208abb26d P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:04:11.953002 11852 consensus_queue.cc:237] T 6a9353d1d230416296627d7cf8b1d425 P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:12.032861 11102 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c160ddfe9b34bdaa3835d0915c1e06f" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:12.012457 12019 raft_consensus.cc:3060] T 0e5d327630e5438b878035d28efe6c0c P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:04:12.023375 11022 raft_consensus.cc:1240] T 7757ecbf402e47bfbc93d0447741e26c P 873357742d5843579556ee8166f23f47 [term 3 LEADER]: Rejecting Update request from peer 0172fed22b6b4890a7fdc953fa128af4 for earlier term 2. Current term is 3. Ops: []
W20260709 13:04:12.058027 10843 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.107.194:44903 timed out after 2.020s (SENT) [suppressed 39 similar messages]
I20260709 13:04:12.072873 11102 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e5d327630e5438b878035d28efe6c0c" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:12.070987 11027 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c160ddfe9b34bdaa3835d0915c1e06f" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
I20260709 13:04:12.077260 11021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a977842ba4b744f4bb2febd55d286006" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
W20260709 13:04:12.077790 12156 log.cc:927] Time spent T 5d156d6ae2ca4fa59a1e387edf0fea39 P 0172fed22b6b4890a7fdc953fa128af4: Append to log took a long time: real 0.050s	user 0.000s	sys 0.000s
I20260709 13:04:12.083253 12103 leader_election.cc:290] T 2567c2d84e4648d7ba5dc23a3df32817 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
W20260709 13:04:12.084380 12151 log.cc:927] Time spent T 30f628e3d7914d609e7f1d980a74c654 P 186b12638a87410487467df721a3cbda: Append to log took a long time: real 0.073s	user 0.000s	sys 0.000s
W20260709 13:04:12.087879 12150 log.cc:927] Time spent T 836251ebc2b74f5da88b8a8b2338e2a5 P 0172fed22b6b4890a7fdc953fa128af4: Append to log took a long time: real 0.095s	user 0.001s	sys 0.000s
I20260709 13:04:12.028636 12050 raft_consensus.cc:2749] T 5d439326f23747eaab3c1e7da2d6b785 P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 13:04:12.092618 10953 raft_consensus.cc:2468] T a977842ba4b744f4bb2febd55d286006 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 2.
I20260709 13:04:12.099615 10953 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f46a5102fd64b45bf1c4dff863d2b4d" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:12.102625 12043 leader_election.cc:290] T 7709387f4d3d4f9992d97a0fbca39a40 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:12.103790 10874 raft_consensus.cc:2468] T e5a5f5a608504650b48fd27f163faed6 P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 4.
I20260709 13:04:12.104130 11097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7709387f4d3d4f9992d97a0fbca39a40" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:12.107286 11920 leader_election.cc:290] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 4 election: Requested vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:12.011781 11926 consensus_queue.cc:1059] T b4cd3d7dfbbf4096908ff11e9dc57141 P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.064s
I20260709 13:04:12.106037 10953 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87cd8f5097624c8692f1d83544d8881e" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:12.122335 10953 raft_consensus.cc:2393] T 87cd8f5097624c8692f1d83544d8881e P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0172fed22b6b4890a7fdc953fa128af4 in current term 4: Already voted for candidate 10ca5ee68ad64b3283b9d235352c9d23 in this term.
I20260709 13:04:12.108772 11926 raft_consensus.cc:3055] T b4cd3d7dfbbf4096908ff11e9dc57141 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 LEADER]: Stepping down as leader of term 3
W20260709 13:04:12.113735 12144 log.cc:927] Time spent T 34064dc7bf3747dbb3876dd824ee0688 P 873357742d5843579556ee8166f23f47: Append to log took a long time: real 0.331s	user 0.000s	sys 0.001s
I20260709 13:04:12.127311 11926 raft_consensus.cc:740] T b4cd3d7dfbbf4096908ff11e9dc57141 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 0172fed22b6b4890a7fdc953fa128af4, State: Running, Role: LEADER
I20260709 13:04:12.127374 10947 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72b111ad7b5f4753ab1d2b5ee1114840" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:12.128748 10947 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49428 (request call id 142) took 1556 ms. Trace:
I20260709 13:04:12.128568 11926 consensus_queue.cc:260] T b4cd3d7dfbbf4096908ff11e9dc57141 P 0172fed22b6b4890a7fdc953fa128af4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:12.129343 10947 rpcz_store.cc:276] 0709 13:04:10.572352 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:10.632064 (+ 59712us) service_pool.cc:224] Handling call
0709 13:04:12.127167 (+1495103us) spinlock_profiling.cc:243] Waited 22.4 ms on lock 0x7b580001d198. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f426303db71 00007f426303deee 0000000000375939 00007f426b0b11e0 00007f426b17a0c4 00007f426b17a131 00007f426b1280a0 00007f426b127984 00007f42646a26d4 00007f42646a2662
0709 13:04:12.128675 (+  1508us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":51432192}
I20260709 13:04:12.126149 11974 leader_election.cc:290] T 4b37f8fca8dc42718b2a42f0883a956f P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:12.131748 11553 leader_election.cc:290] T 0f9df2bc281f4866b6da5da94a0defc7 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:12.129894 10953 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c31881534c5e4f0089c82057c1746652" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
W20260709 13:04:12.133992 12080 log.cc:927] Time spent T 836251ebc2b74f5da88b8a8b2338e2a5 P 186b12638a87410487467df721a3cbda: Append to log took a long time: real 1.221s	user 0.000s	sys 0.002s
I20260709 13:04:12.135993 11066 leader_election.cc:304] T ff13ae9a47c7455f962406c6d131a3ff P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47
I20260709 13:04:12.136538 11023 raft_consensus.cc:2468] T 732c4adeeff6422497aa7db4a82f535b P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 3.
I20260709 13:04:12.137881 11066 leader_election.cc:304] T bb7d67be500947b3aef4c4245e33671c P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 10ca5ee68ad64b3283b9d235352c9d23; no voters: 
I20260709 13:04:12.138700 12096 raft_consensus.cc:2749] T ff13ae9a47c7455f962406c6d131a3ff P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 13:04:12.139768 12096 raft_consensus.cc:2804] T bb7d67be500947b3aef4c4245e33671c P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:04:12.140107 12096 raft_consensus.cc:493] T bb7d67be500947b3aef4c4245e33671c P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:12.140468 12096 raft_consensus.cc:3060] T bb7d67be500947b3aef4c4245e33671c P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:04:12.140939 11025 raft_consensus.cc:2468] T 1e4f2241a94a41f38e481137f868b83d P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 3.
I20260709 13:04:12.141551 11953 raft_consensus.cc:2764] T 1e4f2241a94a41f38e481137f868b83d P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260709 13:04:12.144122 10952 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49434 (request call id 229) took 1152 ms. Trace:
I20260709 13:04:12.144572 10952 rpcz_store.cc:276] 0709 13:04:10.991814 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:11.308529 (+316715us) service_pool.cc:224] Handling call
0709 13:04:12.143912 (+835383us) spinlock_profiling.cc:243] Waited 34.1 ms on lock 0x7b5400789954. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:04:12.144040 (+   128us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":78373632}
W20260709 13:04:12.148111 11066 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:12.095571 11029 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f371d2565dd84667b5613848604949bb" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:12.151721 10845 leader_election.cc:304] T 6a9353d1d230416296627d7cf8b1d425 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 186b12638a87410487467df721a3cbda; no voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47
I20260709 13:04:12.166141 11066 leader_election.cc:304] T a64b7e99d6c54934a7de6f6dffcf5e94 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47
W20260709 13:04:12.168926 10921 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:12.183018 10843 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 104350us
W20260709 13:04:12.188153 10843 leader_election.cc:336] T 5de5f0581e3b41dea729363d6ca563bb P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903): Timed out: RequestConsensusVote RPC to 127.0.107.194:44903 timed out after 2.541s (SENT)
I20260709 13:04:12.188452 11066 leader_election.cc:304] T a977842ba4b744f4bb2febd55d286006 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 10ca5ee68ad64b3283b9d235352c9d23; no voters: 
I20260709 13:04:12.188764 10843 leader_election.cc:304] T 5de5f0581e3b41dea729363d6ca563bb P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 186b12638a87410487467df721a3cbda; no voters: 0172fed22b6b4890a7fdc953fa128af4, 10ca5ee68ad64b3283b9d235352c9d23
I20260709 13:04:12.189579 11066 leader_election.cc:304] T 6f46a5102fd64b45bf1c4dff863d2b4d P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47
I20260709 13:04:12.131791 11926 raft_consensus.cc:3060] T b4cd3d7dfbbf4096908ff11e9dc57141 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:04:12.206952 11067 leader_election.cc:304] T 8c160ddfe9b34bdaa3835d0915c1e06f P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4; no voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47
I20260709 13:04:12.168352 12004 raft_consensus.cc:493] T 9a1716c0adf34bee8a4e99341ba5788f P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:12.202401 12153 log.cc:927] Time spent T 72b111ad7b5f4753ab1d2b5ee1114840 P 10ca5ee68ad64b3283b9d235352c9d23: Append to log took a long time: real 0.143s	user 0.000s	sys 0.001s
I20260709 13:04:12.184984 12048 raft_consensus.cc:2749] T 6a9353d1d230416296627d7cf8b1d425 P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:04:12.233551 10992 leader_election.cc:304] T 7839ca2d2c13431a85f44d7c458c6351 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:04:12.241590 10843 leader_election.cc:304] T 30f628e3d7914d609e7f1d980a74c654 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 186b12638a87410487467df721a3cbda; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47
I20260709 13:04:12.195677 12007 raft_consensus.cc:2749] T a64b7e99d6c54934a7de6f6dffcf5e94 P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:04:12.242993 10992 leader_election.cc:304] T b4cd3d7dfbbf4096908ff11e9dc57141 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47; no voters: 0172fed22b6b4890a7fdc953fa128af4
I20260709 13:04:12.215641 12086 raft_consensus.cc:2749] T 5de5f0581e3b41dea729363d6ca563bb P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:04:12.218247 10950 raft_consensus.cc:1240] T b4cd3d7dfbbf4096908ff11e9dc57141 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Rejecting Update request from peer 0172fed22b6b4890a7fdc953fa128af4 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260709 13:04:12.232105 11972 raft_consensus.cc:493] T a5299b8d044f41b7b68057e63d17b159 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:12.227736 11653 raft_consensus.cc:493] T 868e92339c2f4508af0afbd07c814d64 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0172fed22b6b4890a7fdc953fa128af4)
I20260709 13:04:12.235545 12089 raft_consensus.cc:493] T 525bfdbd6d934afd823f7d3ee3f9d052 P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 10ca5ee68ad64b3283b9d235352c9d23)
I20260709 13:04:12.196944 12064 consensus_queue.cc:1048] T 7709387f4d3d4f9992d97a0fbca39a40 P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.047s
I20260709 13:04:12.197518 11982 raft_consensus.cc:515] T b67eff0634e540eab91cf8cee795969f P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:12.237166 12066 raft_consensus.cc:2804] T a977842ba4b744f4bb2febd55d286006 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:04:12.250072 11025 raft_consensus.cc:1240] T 63d44f6ba6434ab2870fe3190f4d7c2f P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Rejecting Update request from peer 186b12638a87410487467df721a3cbda for earlier term 2. Current term is 3. Ops: []
I20260709 13:04:12.243844 10947 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bac7e5040f984ba2b7146bd36af7267e" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:12.249564 12053 raft_consensus.cc:2804] T b4cd3d7dfbbf4096908ff11e9dc57141 P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:04:12.289945 12053 raft_consensus.cc:697] T b4cd3d7dfbbf4096908ff11e9dc57141 P 873357742d5843579556ee8166f23f47 [term 4 LEADER]: Becoming Leader. State: Replica: 873357742d5843579556ee8166f23f47, State: Running, Role: LEADER
I20260709 13:04:12.224850 10953 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d7ce47b9a92d43dd88d95bcda5038597" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:04:12.290930 10951 raft_consensus.cc:2410] T c31881534c5e4f0089c82057c1746652 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 186b12638a87410487467df721a3cbda for term 3 because replica has last-logged OpId of term: 2 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 13:04:12.290992 10947 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2567c2d84e4648d7ba5dc23a3df32817" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:12.291535 12053 consensus_queue.cc:237] T b4cd3d7dfbbf4096908ff11e9dc57141 P 873357742d5843579556ee8166f23f47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:12.293300 10947 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49434 (request call id 236) took 1232 ms. Trace:
I20260709 13:04:12.293665 10947 rpcz_store.cc:276] 0709 13:04:11.061144 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:12.292396 (+1231252us) service_pool.cc:224] Handling call
0709 13:04:12.292747 (+   351us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:12.292964 (+   217us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:12.292990 (+    26us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:12.293043 (+    53us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:04:12.293072 (+    29us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:12.293125 (+    53us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:12.293228 (+   103us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:12.246538 12103 raft_consensus.cc:2804] T 7839ca2d2c13431a85f44d7c458c6351 P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:04:12.265028 12048 raft_consensus.cc:2749] T 30f628e3d7914d609e7f1d980a74c654 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 13:04:12.281673 12066 raft_consensus.cc:697] T a977842ba4b744f4bb2febd55d286006 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 LEADER]: Becoming Leader. State: Replica: 0172fed22b6b4890a7fdc953fa128af4, State: Running, Role: LEADER
I20260709 13:04:12.296676 12050 consensus_queue.cc:1059] T 87cd8f5097624c8692f1d83544d8881e P 186b12638a87410487467df721a3cbda [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:04:12.297744 12066 consensus_queue.cc:237] T a977842ba4b744f4bb2febd55d286006 P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:12.296705 12103 raft_consensus.cc:697] T 7839ca2d2c13431a85f44d7c458c6351 P 873357742d5843579556ee8166f23f47 [term 3 LEADER]: Becoming Leader. State: Replica: 873357742d5843579556ee8166f23f47, State: Running, Role: LEADER
I20260709 13:04:12.300153 12103 consensus_queue.cc:237] T 7839ca2d2c13431a85f44d7c458c6351 P 873357742d5843579556ee8166f23f47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
W20260709 13:04:12.284617 12087 consensus_meta.cc:287] T 0e5d327630e5438b878035d28efe6c0c P 873357742d5843579556ee8166f23f47: Time spent flushing consensus metadata: real 0.531s	user 0.014s	sys 0.000s
I20260709 13:04:12.301564 12087 raft_consensus.cc:515] T 0e5d327630e5438b878035d28efe6c0c P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:12.302776 12048 raft_consensus.cc:3055] T 87cd8f5097624c8692f1d83544d8881e P 186b12638a87410487467df721a3cbda [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:04:12.303448 12048 raft_consensus.cc:740] T 87cd8f5097624c8692f1d83544d8881e P 186b12638a87410487467df721a3cbda [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 186b12638a87410487467df721a3cbda, State: Running, Role: LEADER
I20260709 13:04:12.304190 12087 leader_election.cc:290] T 0e5d327630e5438b878035d28efe6c0c P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 4 election: Requested vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:12.286235 11852 raft_consensus.cc:2749] T 8c160ddfe9b34bdaa3835d0915c1e06f P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 13:04:12.287138 11653 raft_consensus.cc:515] T 868e92339c2f4508af0afbd07c814d64 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:12.304764 12048 consensus_queue.cc:260] T 87cd8f5097624c8692f1d83544d8881e P 186b12638a87410487467df721a3cbda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:12.306901 11653 leader_election.cc:290] T 868e92339c2f4508af0afbd07c814d64 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:12.288280 11097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f9df2bc281f4866b6da5da94a0defc7" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:12.308706 11097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb7d67be500947b3aef4c4245e33671c" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:12.308904 10954 raft_consensus.cc:2468] T 57b6a2418c1342e7924195e9c1dbd357 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 3.
I20260709 13:04:12.288398 12086 consensus_queue.cc:1059] T 63d44f6ba6434ab2870fe3190f4d7c2f P 186b12638a87410487467df721a3cbda [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:04:12.310261 12086 raft_consensus.cc:3055] T 63d44f6ba6434ab2870fe3190f4d7c2f P 186b12638a87410487467df721a3cbda [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:04:12.310631 12086 raft_consensus.cc:740] T 63d44f6ba6434ab2870fe3190f4d7c2f P 186b12638a87410487467df721a3cbda [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 186b12638a87410487467df721a3cbda, State: Running, Role: LEADER
I20260709 13:04:12.311115 10871 raft_consensus.cc:2468] T 7839ca2d2c13431a85f44d7c458c6351 P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 3.
I20260709 13:04:12.311587 12086 consensus_queue.cc:260] T 63d44f6ba6434ab2870fe3190f4d7c2f P 186b12638a87410487467df721a3cbda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
W20260709 13:04:12.312951 12127 consensus_meta.cc:287] T 5de5f0581e3b41dea729363d6ca563bb P 10ca5ee68ad64b3283b9d235352c9d23: Time spent flushing consensus metadata: real 0.535s	user 0.010s	sys 0.007s
I20260709 13:04:12.313429 12127 raft_consensus.cc:515] T 5de5f0581e3b41dea729363d6ca563bb P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:12.312975 12086 raft_consensus.cc:3060] T 63d44f6ba6434ab2870fe3190f4d7c2f P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Advancing to term 3
W20260709 13:04:12.315176 12155 log.cc:927] Time spent T a610d1a71f52432e88fe56130f921fa2 P 186b12638a87410487467df721a3cbda: Append to log took a long time: real 0.051s	user 0.000s	sys 0.000s
I20260709 13:04:12.319247 10843 leader_election.cc:304] T 836251ebc2b74f5da88b8a8b2338e2a5 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 186b12638a87410487467df721a3cbda; no voters: 0172fed22b6b4890a7fdc953fa128af4, 10ca5ee68ad64b3283b9d235352c9d23
I20260709 13:04:12.320819 10843 leader_election.cc:304] T 2567c2d84e4648d7ba5dc23a3df32817 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 186b12638a87410487467df721a3cbda; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47
I20260709 13:04:12.245080 12158 raft_consensus.cc:2749] T 6f46a5102fd64b45bf1c4dff863d2b4d P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:04:12.322116 10843 leader_election.cc:304] T 0f9df2bc281f4866b6da5da94a0defc7 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda; no voters: 0172fed22b6b4890a7fdc953fa128af4
W20260709 13:04:12.323231 11066 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 133728us
I20260709 13:04:12.236774 12004 raft_consensus.cc:515] T 9a1716c0adf34bee8a4e99341ba5788f P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:12.335269 10955 raft_consensus.cc:2468] T 1e4f2241a94a41f38e481137f868b83d P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 3.
I20260709 13:04:12.337409 10955 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49434 (request call id 239) took 1052 ms. Trace:
I20260709 13:04:12.337852 10955 rpcz_store.cc:276] 0709 13:04:11.284826 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:12.336286 (+1051460us) service_pool.cc:224] Handling call
0709 13:04:12.336749 (+   463us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:12.336974 (+   225us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:12.337012 (+    38us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:12.337076 (+    64us) raft_consensus.cc:1645] Marking committed up to 1
0709 13:04:12.337121 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:12.337201 (+    80us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:12.337335 (+   134us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:12.340606 10955 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49434 (request call id 240) took 1052 ms. Trace:
W20260709 13:04:12.340771 11067 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:45807, user_credentials={real_user=slave}} blocked reactor thread for 133985us
I20260709 13:04:12.341172 10955 rpcz_store.cc:276] 0709 13:04:11.288654 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:12.339427 (+1050773us) service_pool.cc:224] Handling call
0709 13:04:12.339925 (+   498us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:12.340187 (+   262us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:12.340223 (+    36us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:12.340270 (+    47us) raft_consensus.cc:1645] Marking committed up to 1
0709 13:04:12.340333 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:12.340402 (+    69us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:12.340536 (+   134us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:04:12.333088 12044 consensus_meta.cc:287] T ff13ae9a47c7455f962406c6d131a3ff P 873357742d5843579556ee8166f23f47: Time spent flushing consensus metadata: real 0.508s	user 0.015s	sys 0.000s
I20260709 13:04:12.343246 10954 raft_consensus.cc:1240] T 5de5f0581e3b41dea729363d6ca563bb P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Rejecting Update request from peer 0172fed22b6b4890a7fdc953fa128af4 for earlier term 3. Current term is 4. Ops: []
I20260709 13:04:12.324182 12063 raft_consensus.cc:2749] T 836251ebc2b74f5da88b8a8b2338e2a5 P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:04:12.275621 12088 consensus_queue.cc:1059] T a5299b8d044f41b7b68057e63d17b159 P 186b12638a87410487467df721a3cbda [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.073s
I20260709 13:04:12.265244 11972 raft_consensus.cc:515] T a5299b8d044f41b7b68057e63d17b159 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:12.367894 11020 raft_consensus.cc:2410] T 0e5d327630e5438b878035d28efe6c0c P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 186b12638a87410487467df721a3cbda for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 13:04:12.381873 11067 leader_election.cc:304] T f371d2565dd84667b5613848604949bb P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4; no voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47
I20260709 13:04:12.279270 12089 raft_consensus.cc:515] T 525bfdbd6d934afd823f7d3ee3f9d052 P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
W20260709 13:04:12.366928 12068 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.394s	user 0.014s	sys 0.019s
W20260709 13:04:12.417629 10843 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 95607.2us
I20260709 13:04:12.383884 11066 leader_election.cc:304] T 87cd8f5097624c8692f1d83544d8881e P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda
I20260709 13:04:12.380862 12063 raft_consensus.cc:2749] T 2567c2d84e4648d7ba5dc23a3df32817 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:04:12.399605 12167 raft_consensus.cc:2749] T f371d2565dd84667b5613848604949bb P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260709 13:04:12.394948 12068 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.425s	user 0.014s	sys 0.020s
I20260709 13:04:12.452207 12170 raft_consensus.cc:493] T 7757ecbf402e47bfbc93d0447741e26c P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:12.455080 12170 raft_consensus.cc:515] T 7757ecbf402e47bfbc93d0447741e26c P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:12.455596 12089 leader_election.cc:290] T 525bfdbd6d934afd823f7d3ee3f9d052 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
W20260709 13:04:12.456321 11069 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:12.457060 10843 leader_election.cc:304] T 57b6a2418c1342e7924195e9c1dbd357 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda; no voters: 
I20260709 13:04:12.457528 12170 leader_election.cc:290] T 7757ecbf402e47bfbc93d0447741e26c P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:12.458091 12170 raft_consensus.cc:2804] T 57b6a2418c1342e7924195e9c1dbd357 P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:04:12.453248 11937 raft_consensus.cc:3060] T 87cd8f5097624c8692f1d83544d8881e P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Advancing to term 4
W20260709 13:04:12.458794 12116 consensus_meta.cc:287] T 5d439326f23747eaab3c1e7da2d6b785 P 10ca5ee68ad64b3283b9d235352c9d23: Time spent flushing consensus metadata: real 0.697s	user 0.015s	sys 0.004s
I20260709 13:04:12.458860 12004 leader_election.cc:290] T 9a1716c0adf34bee8a4e99341ba5788f P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
W20260709 13:04:12.450531 11066 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 66779.4us
W20260709 13:04:12.418125 11943 consensus_meta.cc:287] T 72b111ad7b5f4753ab1d2b5ee1114840 P 0172fed22b6b4890a7fdc953fa128af4: Time spent flushing consensus metadata: real 0.538s	user 0.012s	sys 0.002s
I20260709 13:04:12.443753 12127 leader_election.cc:290] T 5de5f0581e3b41dea729363d6ca563bb P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 4 election: Requested vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:12.466135 11943 raft_consensus.cc:515] T 72b111ad7b5f4753ab1d2b5ee1114840 P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
W20260709 13:04:12.461622 12162 log.cc:927] Time spent T 6a9353d1d230416296627d7cf8b1d425 P 0172fed22b6b4890a7fdc953fa128af4: Append to log took a long time: real 0.086s	user 0.000s	sys 0.001s
I20260709 13:04:12.461102 10843 leader_election.cc:304] T 1e4f2241a94a41f38e481137f868b83d P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda; no voters: 
W20260709 13:04:12.478885 10843 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:12.462357 12170 raft_consensus.cc:697] T 57b6a2418c1342e7924195e9c1dbd357 P 186b12638a87410487467df721a3cbda [term 3 LEADER]: Becoming Leader. State: Replica: 186b12638a87410487467df721a3cbda, State: Running, Role: LEADER
I20260709 13:04:12.480475 12063 raft_consensus.cc:2804] T 1e4f2241a94a41f38e481137f868b83d P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:04:12.485599 12009 leader_election.cc:290] T d270b9e5481245d9b5340a454a7cd10b P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:12.463150 11982 leader_election.cc:290] T b67eff0634e540eab91cf8cee795969f P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 2 election: Requested vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:12.453856 11921 raft_consensus.cc:515] T a1ec2b50c6ad4f7c8033476208abb26d P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:12.465157 12019 raft_consensus.cc:515] T 0e5d327630e5438b878035d28efe6c0c P 0172fed22b6b4890a7fdc953fa128af4 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:12.445057 12163 raft_consensus.cc:2804] T 0f9df2bc281f4866b6da5da94a0defc7 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Leader election won for term 2
W20260709 13:04:12.469799 12160 log.cc:927] Time spent T 8a69f32c53f24a50b7852d143665d7b4 P 0172fed22b6b4890a7fdc953fa128af4: Append to log took a long time: real 0.078s	user 0.000s	sys 0.000s
I20260709 13:04:12.472854 10994 leader_election.cc:304] T 4f9eb06ed01f4ce68be3af8a8434665b P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 873357742d5843579556ee8166f23f47; no voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda
I20260709 13:04:12.500221 10994 leader_election.cc:304] T 8c160ddfe9b34bdaa3835d0915c1e06f P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 873357742d5843579556ee8166f23f47; no voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda
I20260709 13:04:12.422160 11553 raft_consensus.cc:493] T d71ca3a1ba6b48478aaf62741bf3cf6a P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:12.406661 10949 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.540s	user 0.020s	sys 0.042s
I20260709 13:04:12.460871 11066 leader_election.cc:304] T bac7e5040f984ba2b7146bd36af7267e P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47
I20260709 13:04:12.497555 11027 raft_consensus.cc:1240] T a1ec2b50c6ad4f7c8033476208abb26d P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Rejecting Update request from peer 0172fed22b6b4890a7fdc953fa128af4 for earlier term 1. Current term is 2. Ops: [1.1-1.1]
I20260709 13:04:12.394311 12064 consensus_queue.cc:1059] T 7757ecbf402e47bfbc93d0447741e26c P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.099s
W20260709 13:04:12.505220 10994 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:12.507197 11066 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 46452.2us
I20260709 13:04:12.512466 11102 raft_consensus.cc:2393] T 0e5d327630e5438b878035d28efe6c0c P 0172fed22b6b4890a7fdc953fa128af4 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 186b12638a87410487467df721a3cbda in current term 5: Already voted for candidate 0172fed22b6b4890a7fdc953fa128af4 in this term.
I20260709 13:04:12.493007 10947 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49434 (request call id 237) took 1405 ms. Trace:
I20260709 13:04:12.505689 11653 raft_consensus.cc:2749] T 4f9eb06ed01f4ce68be3af8a8434665b P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20260709 13:04:12.506410 10949 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.658s	user 0.032s	sys 0.062s
I20260709 13:04:12.506621 12089 raft_consensus.cc:2749] T 8c160ddfe9b34bdaa3835d0915c1e06f P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:04:12.508718 12158 raft_consensus.cc:2749] T bac7e5040f984ba2b7146bd36af7267e P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:04:12.516132 10947 rpcz_store.cc:276] 0709 13:04:11.087408 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:12.294739 (+1207331us) service_pool.cc:224] Handling call
0709 13:04:12.295055 (+   316us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:12.295325 (+   270us) raft_consensus.cc:1534] Early marking committed up to index 3
0709 13:04:12.295879 (+   554us) log.cc:844] Serialized 14 byte log entry
0709 13:04:12.443252 (+147373us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:12.443944 (+   692us) raft_consensus.cc:1645] Marking committed up to 3
0709 13:04:12.447470 (+  3526us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:12.459975 (+ 12505us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:12.488793 (+ 28818us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":133,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":40,"raft.queue_time_us":3333,"raft.run_cpu_time_us":154,"raft.run_wall_time_us":1821,"thread_start_us":112856,"threads_started":1,"wal-append.queue_time_us":147842}
I20260709 13:04:12.549482 10947 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b67eff0634e540eab91cf8cee795969f" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:04:12.552311 11921 leader_election.cc:290] T a1ec2b50c6ad4f7c8033476208abb26d P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 2 election: Requested vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:12.494995 12091 raft_consensus.cc:515] T b67eff0634e540eab91cf8cee795969f P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:12.534085 12004 raft_consensus.cc:493] T 32147c6d2d5a477aa38c597c7f78df5b P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 873357742d5843579556ee8166f23f47)
W20260709 13:04:12.534382 12161 log.cc:927] Time spent T a64b7e99d6c54934a7de6f6dffcf5e94 P 873357742d5843579556ee8166f23f47: Append to log took a long time: real 0.090s	user 0.001s	sys 0.000s
I20260709 13:04:12.539155 11972 leader_election.cc:290] T a5299b8d044f41b7b68057e63d17b159 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:12.508440 12063 raft_consensus.cc:697] T 1e4f2241a94a41f38e481137f868b83d P 186b12638a87410487467df721a3cbda [term 3 LEADER]: Becoming Leader. State: Replica: 186b12638a87410487467df721a3cbda, State: Running, Role: LEADER
I20260709 13:04:12.508165 11553 raft_consensus.cc:515] T d71ca3a1ba6b48478aaf62741bf3cf6a P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:12.545204 11095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5de5f0581e3b41dea729363d6ca563bb" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:12.521806 12064 consensus_queue.cc:1243] T 7757ecbf402e47bfbc93d0447741e26c P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Peer 873357742d5843579556ee8166f23f47 log is divergent from this leader: its last log entry 3.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260709 13:04:12.540123 12019 leader_election.cc:290] T 0e5d327630e5438b878035d28efe6c0c P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 5 election: Requested vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:12.591142 12009 raft_consensus.cc:493] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:12.567498 11066 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:12.521212 12170 consensus_queue.cc:237] T 57b6a2418c1342e7924195e9c1dbd357 P 186b12638a87410487467df721a3cbda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:12.630576 12063 consensus_queue.cc:237] T 1e4f2241a94a41f38e481137f868b83d P 186b12638a87410487467df721a3cbda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:12.636999 12171 raft_consensus.cc:3055] T a5299b8d044f41b7b68057e63d17b159 P 186b12638a87410487467df721a3cbda [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:04:12.637529 12171 raft_consensus.cc:740] T a5299b8d044f41b7b68057e63d17b159 P 186b12638a87410487467df721a3cbda [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 186b12638a87410487467df721a3cbda, State: Running, Role: LEADER
I20260709 13:04:12.634547 10949 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49434 (request call id 235) took 1587 ms. Trace:
I20260709 13:04:12.640215 12096 raft_consensus.cc:515] T bb7d67be500947b3aef4c4245e33671c P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:12.641919 11553 leader_election.cc:290] T d71ca3a1ba6b48478aaf62741bf3cf6a P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:12.639214 10949 rpcz_store.cc:276] 0709 13:04:11.046843 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:11.754743 (+707900us) service_pool.cc:224] Handling call
0709 13:04:11.759114 (+  4371us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:11.759382 (+   268us) raft_consensus.cc:1534] Early marking committed up to index 1
0709 13:04:11.823601 (+ 64219us) log.cc:844] Serialized 14 byte log entry
0709 13:04:12.589147 (+765546us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:12.592591 (+  3444us) raft_consensus.cc:1645] Marking committed up to 1
0709 13:04:12.625634 (+ 33043us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:12.628951 (+  3317us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:12.633678 (+  4727us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":31668,"prepare.run_cpu_time_us":933,"prepare.run_wall_time_us":4795,"raft.queue_time_us":67647,"raft.run_cpu_time_us":170,"raft.run_wall_time_us":1717,"thread_start_us":723697,"threads_started":2,"wal-append.queue_time_us":582538}
I20260709 13:04:12.643862 10949 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65bf0a62f4d84ef18dd6b23e082724b8" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:12.644753 12091 leader_election.cc:290] T b67eff0634e540eab91cf8cee795969f P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 2 election: Requested vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:12.645303 10949 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c31881534c5e4f0089c82057c1746652" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:12.642877 11097 raft_consensus.cc:2393] T bb7d67be500947b3aef4c4245e33671c P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 10ca5ee68ad64b3283b9d235352c9d23 in current term 2: Already voted for candidate 0172fed22b6b4890a7fdc953fa128af4 in this term.
I20260709 13:04:12.647540 10843 leader_election.cc:304] T 65bf0a62f4d84ef18dd6b23e082724b8 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 186b12638a87410487467df721a3cbda; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47
I20260709 13:04:12.592357 12004 raft_consensus.cc:515] T 32147c6d2d5a477aa38c597c7f78df5b P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
W20260709 13:04:12.652853 10826 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 13:04:12.629859 12172 raft_consensus.cc:493] T e89e640169e14e2d98f968f76027daa7 P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:12.634392 10947 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d270b9e5481245d9b5340a454a7cd10b" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:04:12.648408 12019 raft_consensus.cc:493] T 75492f86fde046c3900df3eac8292fd4 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:12.632890 12009 raft_consensus.cc:515] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:12.658682 10778 rpcz_store.cc:275] Call kudu.master.MasterService.IsCreateTableDone from 127.0.0.1:59616 (request call id 43) took 1356 ms. Trace:
I20260709 13:04:12.659114 10778 rpcz_store.cc:276] 0709 13:04:11.295418 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:11.295721 (+   303us) service_pool.cc:224] Handling call
0709 13:04:11.296096 (+   375us) catalog_manager.cc:2419] Looking up, locking, and authorizing table
0709 13:04:12.142526 (+846430us) catalog_manager.cc:2317] Verify if the table creation is in progress for test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]
0709 13:04:12.652049 (+509523us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1354135}
I20260709 13:04:12.660665 12009 leader_election.cc:290] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
F20260709 13:04:12.661491   431 test_workload.cc:382] Timed out: Timed out waiting for Table Creation
*** Check failure stack trace: ***
I20260709 13:04:12.661737 11095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87cd8f5097624c8692f1d83544d8881e" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
*** Aborted at 1783602252 (unix time) try "date -d @1783602252" if you are using GNU date ***
I20260709 13:04:12.664922 11937 raft_consensus.cc:2749] T 87cd8f5097624c8692f1d83544d8881e P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:04:12.666698 12019 raft_consensus.cc:515] T 75492f86fde046c3900df3eac8292fd4 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:12.835178 11098 raft_consensus.cc:1240] T 87cd8f5097624c8692f1d83544d8881e P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Rejecting Update request from peer 186b12638a87410487467df721a3cbda for earlier term 3. Current term is 4. Ops: []
I20260709 13:04:12.836581 12019 leader_election.cc:290] T 75492f86fde046c3900df3eac8292fd4 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:12.638490 12171 consensus_queue.cc:260] T a5299b8d044f41b7b68057e63d17b159 P 186b12638a87410487467df721a3cbda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:12.651630 12091 raft_consensus.cc:2749] T 65bf0a62f4d84ef18dd6b23e082724b8 P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:04:12.837203 11095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a154cc8b181f460abf4880064379948b" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:12.671990 12004 leader_election.cc:290] T 32147c6d2d5a477aa38c597c7f78df5b P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
W20260709 13:04:12.674139 12049 consensus_meta.cc:287] T d7ce47b9a92d43dd88d95bcda5038597 P 10ca5ee68ad64b3283b9d235352c9d23: Time spent flushing consensus metadata: real 0.821s	user 0.013s	sys 0.004s
I20260709 13:04:12.657541 12172 raft_consensus.cc:515] T e89e640169e14e2d98f968f76027daa7 P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:12.680349 10952 raft_consensus.cc:1240] T 0f9df2bc281f4866b6da5da94a0defc7 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Rejecting Update request from peer 0172fed22b6b4890a7fdc953fa128af4 for earlier term 1. Current term is 2. Ops: []
I20260709 13:04:12.839715 10952 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49424 (request call id 202) took 1790 ms. Trace:
I20260709 13:04:12.840126 10952 rpcz_store.cc:276] 0709 13:04:11.049054 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:12.679674 (+1630620us) service_pool.cc:224] Handling call
0709 13:04:12.680127 (+   453us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:12.839371 (+159244us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:12.839643 (+   272us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:12.840691 12172 leader_election.cc:290] T e89e640169e14e2d98f968f76027daa7 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:12.672056 10776 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:59614 (request call id 64) took 7661 ms. Trace:
I20260709 13:04:12.841610 10952 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb7d67be500947b3aef4c4245e33671c" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
W20260709 13:04:12.680354 10877 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51718 (request call id 259) took 1453 ms (client timeout 1912 ms). Trace:
I20260709 13:04:12.654048 12096 leader_election.cc:290] T bb7d67be500947b3aef4c4245e33671c P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 2 election: Requested vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:12.653335 12163 raft_consensus.cc:697] T 0f9df2bc281f4866b6da5da94a0defc7 P 186b12638a87410487467df721a3cbda [term 2 LEADER]: Becoming Leader. State: Replica: 186b12638a87410487467df721a3cbda, State: Running, Role: LEADER
I20260709 13:04:12.843053 11102 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d270b9e5481245d9b5340a454a7cd10b" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
W20260709 13:04:12.842406 10877 rpcz_store.cc:269] 0709 13:04:11.226670 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:11.266960 (+ 40290us) service_pool.cc:224] Handling call
0709 13:04:12.679997 (+1413037us) spinlock_profiling.cc:243] Waited 194 ms on lock 0x7b5400290254. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:04:12.680247 (+   250us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":445498112}
I20260709 13:04:12.683211 11972 raft_consensus.cc:493] T 65bf0a62f4d84ef18dd6b23e082724b8 P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 10ca5ee68ad64b3283b9d235352c9d23)
I20260709 13:04:12.686601 10947 raft_consensus.cc:2468] T d270b9e5481245d9b5340a454a7cd10b P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 4.
W20260709 13:04:12.687647 10993 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:12.687918 11633 raft_consensus.cc:493] T 57b6a2418c1342e7924195e9c1dbd357 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:12.689702 11094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c31881534c5e4f0089c82057c1746652" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:04:12.690742 12158 raft_consensus.cc:493] T d270b9e5481245d9b5340a454a7cd10b P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:12.695204 12086 raft_consensus.cc:493] T 34064dc7bf3747dbb3876dd824ee0688 P 186b12638a87410487467df721a3cbda [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:12.695061 11099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "401a9cf11eac479aa445ec8c1d6e33b2" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:04:12.685537 11797 leader_election.cc:290] T 57b6a2418c1342e7924195e9c1dbd357 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 3 election: Requested vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
W20260709 13:04:12.718883 10921 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:12.718832 11553 consensus_queue.cc:1059] T e5ba72ccd854430baadbc636dfd17136 P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
W20260709 13:04:12.720305 12092 log.cc:927] Time spent T 72b111ad7b5f4753ab1d2b5ee1114840 P 0172fed22b6b4890a7fdc953fa128af4: Append to log took a long time: real 1.738s	user 0.002s	sys 0.008s
W20260709 13:04:12.720319 11100 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37676 (request call id 230) took 2856 ms (client timeout 2107 ms). Trace:
I20260709 13:04:12.722812 11103 raft_consensus.cc:1240] T 8c160ddfe9b34bdaa3835d0915c1e06f P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Rejecting Update request from peer 186b12638a87410487467df721a3cbda for earlier term 2. Current term is 3. Ops: []
I20260709 13:04:12.724393 11779 leader_election.cc:290] T b4cd3d7dfbbf4096908ff11e9dc57141 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:12.727278 11982 raft_consensus.cc:3055] T 7757ecbf402e47bfbc93d0447741e26c P 0172fed22b6b4890a7fdc953fa128af4 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:04:12.734267 10950 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49424 (request call id 196) took 1791 ms. Trace:
I20260709 13:04:12.736336 11024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a154cc8b181f460abf4880064379948b" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:12.742086 10775 catalog_manager.cc:5697] T a154cc8b181f460abf4880064379948b P 10ca5ee68ad64b3283b9d235352c9d23 reported cstate change: term changed from 1 to 2, leader changed from 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194) to 873357742d5843579556ee8166f23f47 (127.0.107.195). New cstate: current_term: 2 leader_uuid: "873357742d5843579556ee8166f23f47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } }
W20260709 13:04:12.749188 12052 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.980s	user 0.014s	sys 0.000s
I20260709 13:04:12.890347 10775 catalog_manager.cc:5697] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 10ca5ee68ad64b3283b9d235352c9d23 reported cstate change: term changed from 0 to 2, leader changed from <none> to 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194). New cstate: current_term: 2 leader_uuid: "10ca5ee68ad64b3283b9d235352c9d23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } health_report { overall_health: HEALTHY } } }
I20260709 13:04:12.760528 11097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5a5f5a608504650b48fd27f163faed6" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:04:12.790339 11025 raft_consensus.cc:1240] T 4f9eb06ed01f4ce68be3af8a8434665b P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Rejecting Update request from peer 186b12638a87410487467df721a3cbda for earlier term 2. Current term is 3. Ops: []
W20260709 13:04:12.791469 10951 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49428 (request call id 141) took 2219 ms (client timeout 2213 ms). Trace:
I20260709 13:04:12.791813 11023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7757ecbf402e47bfbc93d0447741e26c" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:12.792681 10995 leader_election.cc:304] T 732c4adeeff6422497aa7db4a82f535b P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 873357742d5843579556ee8166f23f47; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda
I20260709 13:04:12.792693 11022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57b6a2418c1342e7924195e9c1dbd357" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
I20260709 13:04:12.788036 12048 raft_consensus.cc:3060] T 87cd8f5097624c8692f1d83544d8881e P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:04:12.794603 11795 raft_consensus.cc:493] T f371d2565dd84667b5613848604949bb P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:12.794874 12050 raft_consensus.cc:493] T 868e92339c2f4508af0afbd07c814d64 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0172fed22b6b4890a7fdc953fa128af4)
I20260709 13:04:12.794660 11028 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a977842ba4b744f4bb2febd55d286006" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:12.795995 10844 leader_election.cc:304] T 732c4adeeff6422497aa7db4a82f535b P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47; no voters: 10ca5ee68ad64b3283b9d235352c9d23
W20260709 13:04:12.796017 10846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:12.802806 11943 leader_election.cc:290] T 72b111ad7b5f4753ab1d2b5ee1114840 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 4 election: Requested vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:12.889497 11021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c2a35bb99374e93b573cf870d04faaa" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:12.943933 12049 raft_consensus.cc:3055] T e5ba72ccd854430baadbc636dfd17136 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:04:12.944504 12049 raft_consensus.cc:740] T e5ba72ccd854430baadbc636dfd17136 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 10ca5ee68ad64b3283b9d235352c9d23, State: Running, Role: LEADER
I20260709 13:04:12.944908 11021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57b6a2418c1342e7924195e9c1dbd357" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:12.946039 10873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5299b8d044f41b7b68057e63d17b159" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
W20260709 13:04:12.821563 12147 log.cc:927] Time spent T 8a69f32c53f24a50b7852d143665d7b4 P 10ca5ee68ad64b3283b9d235352c9d23: Append to log took a long time: real 1.078s	user 0.006s	sys 0.000s
I20260709 13:04:12.822767 10954 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49424 (request call id 203) took 1588 ms. Trace:
I20260709 13:04:12.827095 12105 leader_election.cc:290] T e96e80ba9ac34d0998395515e5c8c41a P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:12.655922 10949 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72b111ad7b5f4753ab1d2b5ee1114840" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:12.843497 10920 leader_election.cc:304] T 87cd8f5097624c8692f1d83544d8881e P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23; no voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda
I20260709 13:04:12.654471 12071 raft_consensus.cc:3055] T 732c4adeeff6422497aa7db4a82f535b P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:04:12.857167 11972 raft_consensus.cc:515] T 65bf0a62f4d84ef18dd6b23e082724b8 P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:12.862251 11029 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a64b7e99d6c54934a7de6f6dffcf5e94" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:12.876103 10953 raft_consensus.cc:2468] T d7ce47b9a92d43dd88d95bcda5038597 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 2.
I20260709 13:04:12.882776 11026 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f1466124dbd4f6ebb3c4d62ca48bf96" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
I20260709 13:04:12.951808 10919 leader_election.cc:304] T 57b6a2418c1342e7924195e9c1dbd357 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23; no voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47
I20260709 13:04:12.844130 12163 consensus_queue.cc:237] T 0f9df2bc281f4866b6da5da94a0defc7 P 186b12638a87410487467df721a3cbda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:12.768013 12070 raft_consensus.cc:3055] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 873357742d5843579556ee8166f23f47 [term 4 LEADER]: Stepping down as leader of term 4
W20260709 13:04:12.895893 10995 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:38967, user_credentials={real_user=slave}} blocked reactor thread for 103346us
W20260709 13:04:12.891525 12052 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.122s	user 0.015s	sys 0.000s
I20260709 13:04:12.893116 10775 catalog_manager.cc:5697] T e5ba72ccd854430baadbc636dfd17136 P 10ca5ee68ad64b3283b9d235352c9d23 reported cstate change: term changed from 1 to 3, leader changed from 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196) to 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194). New cstate: current_term: 3 leader_uuid: "10ca5ee68ad64b3283b9d235352c9d23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } health_report { overall_health: UNKNOWN } } }
I20260709 13:04:12.865662 12096 consensus_queue.cc:1059] T 0f9df2bc281f4866b6da5da94a0defc7 P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.026s
I20260709 13:04:12.865614 11633 raft_consensus.cc:515] T 57b6a2418c1342e7924195e9c1dbd357 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:12.910552 10878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2567c2d84e4648d7ba5dc23a3df32817" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
W20260709 13:04:12.814424 10948 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49424 (request call id 195) took 1872 ms (client timeout 1623 ms). Trace:
I20260709 13:04:12.886723 11982 raft_consensus.cc:740] T 7757ecbf402e47bfbc93d0447741e26c P 0172fed22b6b4890a7fdc953fa128af4 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0172fed22b6b4890a7fdc953fa128af4, State: Running, Role: LEADER
I20260709 13:04:12.872567 12086 raft_consensus.cc:515] T 34064dc7bf3747dbb3876dd824ee0688 P 186b12638a87410487467df721a3cbda [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:12.898550 11879 raft_consensus.cc:2749] T 732c4adeeff6422497aa7db4a82f535b P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 13:04:12.885656 11100 rpcz_store.cc:269] 0709 13:04:09.863350 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:10.088526 (+225176us) service_pool.cc:224] Handling call
0709 13:04:12.719962 (+2631436us) spinlock_profiling.cc:243] Waited 1.11 s on lock 0x7b54005f28d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:04:12.720173 (+   211us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2556662144}
I20260709 13:04:12.932217 11103 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37678 (request call id 125) took 3085 ms. Trace:
I20260709 13:04:12.978880 11103 rpcz_store.cc:276] 0709 13:04:09.846422 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:10.539269 (+692847us) service_pool.cc:224] Handling call
0709 13:04:12.722581 (+2183312us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:12.886515 (+163934us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:12.931924 (+ 45409us) spinlock_profiling.cc:243] Waited 2.18 s on lock 0x7b54005f28d0. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661618ad 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:04:12.932096 (+   172us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5020515200}
I20260709 13:04:12.980167 11103 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b67eff0634e540eab91cf8cee795969f" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:04:12.900118 10952 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e89e640169e14e2d98f968f76027daa7" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:12.932793 12091 raft_consensus.cc:2804] T 732c4adeeff6422497aa7db4a82f535b P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:04:12.948165 10954 rpcz_store.cc:276] 0709 13:04:11.234245 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:12.309886 (+1075641us) service_pool.cc:224] Handling call
0709 13:04:12.310274 (+   388us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:12.361109 (+ 50835us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:12.822375 (+461266us) spinlock_profiling.cc:243] Waited 14.9 ms on lock 0x7b54003bcc54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:04:12.822665 (+   290us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":34243456}
I20260709 13:04:12.989711 12091 raft_consensus.cc:697] T 732c4adeeff6422497aa7db4a82f535b P 186b12638a87410487467df721a3cbda [term 3 LEADER]: Becoming Leader. State: Replica: 186b12638a87410487467df721a3cbda, State: Running, Role: LEADER
I20260709 13:04:12.990559 10954 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72b111ad7b5f4753ab1d2b5ee1114840" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:04:12.949160 10872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555217d45d9b4ee492dbb3ddaf0f7c02" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:04:12.990857 12091 consensus_queue.cc:237] T 732c4adeeff6422497aa7db4a82f535b P 186b12638a87410487467df721a3cbda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:12.991995 10954 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b67eff0634e540eab91cf8cee795969f" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:12.992334 11025 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f1466124dbd4f6ebb3c4d62ca48bf96" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
I20260709 13:04:12.992861 11103 raft_consensus.cc:2393] T b67eff0634e540eab91cf8cee795969f P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 186b12638a87410487467df721a3cbda in current term 2: Already voted for candidate 0172fed22b6b4890a7fdc953fa128af4 in this term.
I20260709 13:04:12.993219 10954 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "525bfdbd6d934afd823f7d3ee3f9d052" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:12.993772 11103 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1ec2b50c6ad4f7c8033476208abb26d" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:04:12.934796 10871 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7709387f4d3d4f9992d97a0fbca39a40" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
W20260709 13:04:12.978273 10948 rpcz_store.cc:269] 0709 13:04:10.942324 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:11.028774 (+ 86450us) service_pool.cc:224] Handling call
0709 13:04:12.814071 (+1785297us) spinlock_profiling.cc:243] Waited 71.5 ms on lock 0x7b54008a5f54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:04:12.814312 (+   241us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":164457216}
I20260709 13:04:12.994643 10871 raft_consensus.cc:2468] T 7709387f4d3d4f9992d97a0fbca39a40 P 186b12638a87410487467df721a3cbda [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 5.
I20260709 13:04:12.994994 11103 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d156d6ae2ca4fa59a1e387edf0fea39" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:12.841367 10776 rpcz_store.cc:276] 0709 13:04:05.010261 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:05.010524 (+   263us) service_pool.cc:224] Handling call
0709 13:04:12.671955 (+7661431us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 13:04:09.693218 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 13:04:09.766835 (+ 73617us) write_op.cc:432] Acquiring schema lock in shared mode
0709 13:04:09.766933 (+    98us) write_op.cc:435] Acquired schema lock
0709 13:04:09.766970 (+    37us) tablet.cc:662] Decoding operations
0709 13:04:09.768115 (+  1145us) write_op.cc:620] Acquiring the partition lock for write op
0709 13:04:09.768300 (+   185us) write_op.cc:641] Partition lock acquired for write op
0709 13:04:09.768383 (+    83us) tablet.cc:685] Acquiring locks for 4 operations
0709 13:04:09.769182 (+   799us) tablet.cc:701] Row locks acquired
0709 13:04:09.769419 (+   237us) write_op.cc:260] PREPARE: finished
0709 13:04:09.769690 (+   271us) write_op.cc:270] Start()
0709 13:04:09.790484 (+ 20794us) write_op.cc:276] Timestamp: P: 1783602249769607 usec, L: 0
0709 13:04:09.803491 (+ 13007us) op_driver.cc:348] REPLICATION: starting
0709 13:04:09.999259 (+195768us) log.cc:844] Serialized 1705 byte log entry
0709 13:04:11.045586 (+1046327us) op_driver.cc:464] REPLICATION: finished
0709 13:04:11.050191 (+  4605us) write_op.cc:301] APPLY: starting
0709 13:04:11.051080 (+   889us) tablet.cc:1367] starting BulkCheckPresence
0709 13:04:11.051274 (+   194us) tablet.cc:1370] finished BulkCheckPresence
0709 13:04:11.051441 (+   167us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 13:04:11.067478 (+ 16037us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 13:04:11.067877 (+   399us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=4
0709 13:04:11.067971 (+    94us) write_op.cc:312] APPLY: finished
0709 13:04:11.068388 (+   417us) log.cc:844] Serialized 40 byte log entry
0709 13:04:11.068655 (+   267us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:04:11.068805 (+   150us) write_op.cc:454] Released schema lock
0709 13:04:11.068911 (+   106us) write_op.cc:341] FINISH: Updating metrics
0709 13:04:11.095889 (+ 26978us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:04:11.095926 (+    37us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":14,"child_traces":[["op",{"apply.queue_time_us":4193,"apply.run_cpu_time_us":3654,"apply.run_wall_time_us":36878,"num_ops":4,"prepare.queue_time_us":320027,"prepare.run_cpu_time_us":34946,"prepare.run_wall_time_us":693393,"raft.queue_time_us":62043,"raft.run_cpu_time_us":2364,"raft.run_wall_time_us":4481,"replication_time_us":1241836,"thread_start_us":471286,"threads_started":4,"wal-append.queue_time_us":480230,"wal-append.run_cpu_time_us":12913,"wal-append.run_wall_time_us":1649078}]]}
I20260709 13:04:12.932236 11095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7757ecbf402e47bfbc93d0447741e26c" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:12.995471 11067 leader_election.cc:304] T a154cc8b181f460abf4880064379948b P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47
I20260709 13:04:12.912202 11937 raft_consensus.cc:493] T 5d439326f23747eaab3c1e7da2d6b785 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 10ca5ee68ad64b3283b9d235352c9d23)
I20260709 13:04:12.996408 11937 raft_consensus.cc:515] T 5d439326f23747eaab3c1e7da2d6b785 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:12.997099 11103 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d71ca3a1ba6b48478aaf62741bf3cf6a" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:12.997563 12004 consensus_queue.cc:1059] T a1ec2b50c6ad4f7c8033476208abb26d P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:04:12.938094 11028 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b37f8fca8dc42718b2a42f0883a956f" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:12.990423 10952 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32147c6d2d5a477aa38c597c7f78df5b" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:12.998530 12004 raft_consensus.cc:2749] T a154cc8b181f460abf4880064379948b P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:04:12.899130 11097 raft_consensus.cc:3060] T e5a5f5a608504650b48fd27f163faed6 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:04:12.999298 12004 raft_consensus.cc:3055] T a1ec2b50c6ad4f7c8033476208abb26d P 0172fed22b6b4890a7fdc953fa128af4 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:04:12.999330 11028 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7839ca2d2c13431a85f44d7c458c6351" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
I20260709 13:04:12.999816 12004 raft_consensus.cc:740] T a1ec2b50c6ad4f7c8033476208abb26d P 0172fed22b6b4890a7fdc953fa128af4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0172fed22b6b4890a7fdc953fa128af4, State: Running, Role: LEADER
I20260709 13:04:12.950598 12105 raft_consensus.cc:2749] T 87cd8f5097624c8692f1d83544d8881e P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 13:04:13.000655 11028 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32147c6d2d5a477aa38c597c7f78df5b" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:12.950315 12071 raft_consensus.cc:740] T 732c4adeeff6422497aa7db4a82f535b P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 10ca5ee68ad64b3283b9d235352c9d23, State: Running, Role: LEADER
I20260709 13:04:13.001813 11028 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e89e640169e14e2d98f968f76027daa7" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:12.939141 11099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32147c6d2d5a477aa38c597c7f78df5b" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:13.001907 12071 consensus_queue.cc:260] T 732c4adeeff6422497aa7db4a82f535b P 10ca5ee68ad64b3283b9d235352c9d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:13.002789 11099 raft_consensus.cc:2468] T 32147c6d2d5a477aa38c597c7f78df5b P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 2.
I20260709 13:04:13.003039 12071 raft_consensus.cc:3060] T 732c4adeeff6422497aa7db4a82f535b P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:04:13.003856 11099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5de5f0581e3b41dea729363d6ca563bb" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:04:13.004083 11633 leader_election.cc:290] T 57b6a2418c1342e7924195e9c1dbd357 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:12.930553 10876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e5d327630e5438b878035d28efe6c0c" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:04:12.931192 11024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb7d67be500947b3aef4c4245e33671c" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:12.912889 11797 raft_consensus.cc:493] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:12.938941 10874 raft_consensus.cc:1275] T 7757ecbf402e47bfbc93d0447741e26c P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Refusing update from remote peer 873357742d5843579556ee8166f23f47: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
W20260709 13:04:12.913800 10844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:45807, user_credentials={real_user=slave}} blocked reactor thread for 117930us
I20260709 13:04:12.949366 10949 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b67eff0634e540eab91cf8cee795969f" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:04:12.942795 10877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "868e92339c2f4508af0afbd07c814d64" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
W20260709 13:04:12.894781 10951 rpcz_store.cc:269] 0709 13:04:10.571822 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:10.584211 (+ 12389us) service_pool.cc:224] Handling call
0709 13:04:11.870117 (+1285906us) spinlock_profiling.cc:243] Waited 32.6 ms on lock 0x7b580001d198. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f426303db71 00007f426303deee 0000000000375939 00007f426b0b11e0 00007f426b17a0c4 00007f426b17a131 00007f426b1280a0 00007f426b127984 00007f42646a26d4 00007f42646a2662
0709 13:04:12.791182 (+921065us) spinlock_profiling.cc:243] Waited 420 ms on lock 0x7b540040ea54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:04:12.791366 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1039832576}
I20260709 13:04:12.953088 10994 leader_election.cc:304] T 401a9cf11eac479aa445ec8c1d6e33b2 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 873357742d5843579556ee8166f23f47; no voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda
W20260709 13:04:12.947283 10843 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:13.016754 10951 raft_consensus.cc:1240] T 75492f86fde046c3900df3eac8292fd4 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 LEADER]: Rejecting Update request from peer 873357742d5843579556ee8166f23f47 for earlier term 1. Current term is 2. Ops: []
I20260709 13:04:13.017590 10951 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49434 (request call id 243) took 1178 ms. Trace:
I20260709 13:04:12.949859 11101 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37666 (request call id 193) took 3021 ms. Trace:
W20260709 13:04:12.950141 10920 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 106756us
I20260709 13:04:13.018638 11797 raft_consensus.cc:515] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:13.019840 11024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4cd3d7dfbbf4096908ff11e9dc57141" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
W20260709 13:04:13.016711 10994 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 63730.4us
W20260709 13:04:13.021903 11069 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:12.966674 10775 catalog_manager.cc:5697] T 5d439326f23747eaab3c1e7da2d6b785 P 10ca5ee68ad64b3283b9d235352c9d23 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "10ca5ee68ad64b3283b9d235352c9d23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } health_report { overall_health: HEALTHY } } }
I20260709 13:04:13.022290 10844 leader_election.cc:304] T 0e5d327630e5438b878035d28efe6c0c P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 186b12638a87410487467df721a3cbda; no voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47
I20260709 13:04:12.968118 10953 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49428 (request call id 167) took 1071 ms. Trace:
I20260709 13:04:12.871294 12158 raft_consensus.cc:515] T d270b9e5481245d9b5340a454a7cd10b P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:12.969909 11029 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c2a35bb99374e93b573cf870d04faaa" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:12.975365 10878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1ec2b50c6ad4f7c8033476208abb26d" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:04:12.965400 12070 raft_consensus.cc:740] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 873357742d5843579556ee8166f23f47 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 873357742d5843579556ee8166f23f47, State: Running, Role: LEADER
I20260709 13:04:13.025182 10878 raft_consensus.cc:3060] T a1ec2b50c6ad4f7c8033476208abb26d P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:04:13.025256 10843 leader_election.cc:304] T d270b9e5481245d9b5340a454a7cd10b P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda; no voters: 0172fed22b6b4890a7fdc953fa128af4
I20260709 13:04:12.977056 10919 leader_election.cc:304] T a64b7e99d6c54934a7de6f6dffcf5e94 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23; no voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47
I20260709 13:04:13.018599 11101 rpcz_store.cc:276] 0709 13:04:09.928124 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:10.569722 (+641598us) service_pool.cc:224] Handling call
0709 13:04:10.654533 (+ 84811us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:10.658821 (+  4288us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:10.658867 (+    46us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:10.691071 (+ 32204us) log.cc:844] Serialized 26 byte log entry
0709 13:04:10.876192 (+185121us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:04:10.876315 (+   123us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:10.876516 (+   201us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:12.719552 (+1843036us) spinlock_profiling.cc:243] Waited 9.34 ms on lock 0x7b4400022b4c. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f4262f9bc85 00007f4262f9c6a2 00007f4265d0e1db 00007f4265d09aaf 00007f4265cfd86e 00007f4265cfcbb6 00007f4265cfc7d8 00007f4265d06221 00007f4265d061d9 00007f4265d06169
0709 13:04:12.949493 (+229941us) raft_consensus.cc:1685] finished
0709 13:04:12.949524 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:12.949794 (+   270us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3680,"raft.queue_time_us":53723,"raft.run_cpu_time_us":161,"raft.run_wall_time_us":5665,"spinlock_wait_cycles":21490048,"thread_start_us":142491,"threads_started":1,"wal-append.queue_time_us":230541}
I20260709 13:04:12.970966 10921 leader_election.cc:304] T e5a5f5a608504650b48fd27f163faed6 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda; no voters: 
I20260709 13:04:12.950405 11023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d71ca3a1ba6b48478aaf62741bf3cf6a" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:12.965534 11553 raft_consensus.cc:2749] T 57b6a2418c1342e7924195e9c1dbd357 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:04:12.983356 11022 raft_consensus.cc:3060] T 57b6a2418c1342e7924195e9c1dbd357 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:04:13.029603 10843 leader_election.cc:304] T 72b111ad7b5f4753ab1d2b5ee1114840 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 186b12638a87410487467df721a3cbda; no voters: 0172fed22b6b4890a7fdc953fa128af4, 10ca5ee68ad64b3283b9d235352c9d23
I20260709 13:04:12.984150 12096 raft_consensus.cc:3055] T 0f9df2bc281f4866b6da5da94a0defc7 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:04:12.985708 12086 leader_election.cc:290] T 34064dc7bf3747dbb3876dd824ee0688 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:12.953819 11795 raft_consensus.cc:515] T f371d2565dd84667b5613848604949bb P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:12.987305 11094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a69f32c53f24a50b7852d143665d7b4" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:12.987998 11982 consensus_queue.cc:260] T 7757ecbf402e47bfbc93d0447741e26c P 0172fed22b6b4890a7fdc953fa128af4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:12.988821 11100 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e5d327630e5438b878035d28efe6c0c" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:04:12.969107 12050 raft_consensus.cc:515] T 868e92339c2f4508af0afbd07c814d64 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:12.948763 10946 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49428 (request call id 157) took 2287 ms. Trace:
I20260709 13:04:13.003895 11028 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e4f2241a94a41f38e481137f868b83d" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:12.946391 11021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a154cc8b181f460abf4880064379948b" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:13.023284 10953 rpcz_store.cc:276] 0709 13:04:11.896410 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:12.186624 (+290214us) service_pool.cc:224] Handling call
0709 13:04:12.967866 (+781242us) spinlock_profiling.cc:243] Waited 548 ms on lock 0x7b54001e23d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:04:12.968028 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1259262208}
W20260709 13:04:13.027884 10919 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:45807, user_credentials={real_user=slave}} blocked reactor thread for 50928.4us
W20260709 13:04:13.028505 10921 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:38967, user_credentials={real_user=slave}} blocked reactor thread for 57791.2us
I20260709 13:04:12.969601 11926 raft_consensus.cc:493] T e5ba72ccd854430baadbc636dfd17136 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 10ca5ee68ad64b3283b9d235352c9d23)
I20260709 13:04:13.064347 10871 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f9df2bc281f4866b6da5da94a0defc7" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:13.064291 11926 raft_consensus.cc:515] T e5ba72ccd854430baadbc636dfd17136 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:13.065234 10949 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57b6a2418c1342e7924195e9c1dbd357" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:13.065881 10871 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "30f628e3d7914d609e7f1d980a74c654" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:04:13.026575 10920 leader_election.cc:304] T 32147c6d2d5a477aa38c597c7f78df5b P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 10ca5ee68ad64b3283b9d235352c9d23; no voters: 
I20260709 13:04:13.003189 11937 leader_election.cc:290] T 5d439326f23747eaab3c1e7da2d6b785 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:13.031406 11779 raft_consensus.cc:2749] T a64b7e99d6c54934a7de6f6dffcf5e94 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:04:13.067442 11100 raft_consensus.cc:2368] T 0e5d327630e5438b878035d28efe6c0c P 0172fed22b6b4890a7fdc953fa128af4 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 873357742d5843579556ee8166f23f47 for earlier term 4. Current term is 5.
I20260709 13:04:13.068506 11100 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "525bfdbd6d934afd823f7d3ee3f9d052" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:13.068764 11926 leader_election.cc:290] T e5ba72ccd854430baadbc636dfd17136 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:13.069223 11100 raft_consensus.cc:2468] T 525bfdbd6d934afd823f7d3ee3f9d052 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 3.
I20260709 13:04:13.069387 11098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555217d45d9b4ee492dbb3ddaf0f7c02" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:13.029640 11029 raft_consensus.cc:1275] T 75492f86fde046c3900df3eac8292fd4 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Refusing update from remote peer 10ca5ee68ad64b3283b9d235352c9d23: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:04:13.060719 11982 raft_consensus.cc:3060] T 7757ecbf402e47bfbc93d0447741e26c P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:04:13.017932 10951 rpcz_store.cc:276] 0709 13:04:11.839100 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:13.016201 (+1177101us) service_pool.cc:224] Handling call
0709 13:04:13.016544 (+   343us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:13.017262 (+   718us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:13.017525 (+   263us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:13.075225 10951 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75492f86fde046c3900df3eac8292fd4" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:13.076745 10951 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34064dc7bf3747dbb3876dd824ee0688" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:13.077080 11972 leader_election.cc:290] T 65bf0a62f4d84ef18dd6b23e082724b8 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:13.077410 10951 raft_consensus.cc:2468] T 34064dc7bf3747dbb3876dd824ee0688 P 10ca5ee68ad64b3283b9d235352c9d23 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 5.
I20260709 13:04:12.993772 11025 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e96e80ba9ac34d0998395515e5c8c41a" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:13.005178 11099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555217d45d9b4ee492dbb3ddaf0f7c02" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:04:13.034374 12096 raft_consensus.cc:740] T 0f9df2bc281f4866b6da5da94a0defc7 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0172fed22b6b4890a7fdc953fa128af4, State: Running, Role: LEADER
I20260709 13:04:13.033888 12104 raft_consensus.cc:2804] T d270b9e5481245d9b5340a454a7cd10b P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:04:13.079383 11029 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a1716c0adf34bee8a4e99341ba5788f" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:13.079854 12104 raft_consensus.cc:697] T d270b9e5481245d9b5340a454a7cd10b P 186b12638a87410487467df721a3cbda [term 4 LEADER]: Becoming Leader. State: Replica: 186b12638a87410487467df721a3cbda, State: Running, Role: LEADER
I20260709 13:04:13.033946 10775 catalog_manager.cc:5697] T b67eff0634e540eab91cf8cee795969f P 10ca5ee68ad64b3283b9d235352c9d23 reported cstate change: term changed from 0 to 1, leader changed from <none> to 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194). New cstate: current_term: 1 leader_uuid: "10ca5ee68ad64b3283b9d235352c9d23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } health_report { overall_health: HEALTHY } } }
I20260709 13:04:13.081004 12104 consensus_queue.cc:237] T d270b9e5481245d9b5340a454a7cd10b P 186b12638a87410487467df721a3cbda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:13.081744 10875 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51718 (request call id 266) took 1243 ms. Trace:
I20260709 13:04:13.066237 10949 raft_consensus.cc:2393] T 57b6a2418c1342e7924195e9c1dbd357 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 873357742d5843579556ee8166f23f47 in current term 3: Already voted for candidate 186b12638a87410487467df721a3cbda in this term.
I20260709 13:04:13.082481 10775 catalog_manager.cc:5697] T 4c2a35bb99374e93b573cf870d04faaa P 10ca5ee68ad64b3283b9d235352c9d23 reported cstate change: term changed from 1 to 3, leader changed from 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194) to 873357742d5843579556ee8166f23f47 (127.0.107.195). New cstate: current_term: 3 leader_uuid: "873357742d5843579556ee8166f23f47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } }
I20260709 13:04:13.084223 10949 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49434 (request call id 247) took 1136 ms. Trace:
I20260709 13:04:13.084408 11094 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37676 (request call id 247) took 1141 ms. Trace:
I20260709 13:04:13.085212 10952 raft_consensus.cc:1240] T d71ca3a1ba6b48478aaf62741bf3cf6a P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Rejecting Update request from peer 0172fed22b6b4890a7fdc953fa128af4 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260709 13:04:13.084933 11094 rpcz_store.cc:276] 0709 13:04:11.942715 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:13.083370 (+1140655us) service_pool.cc:224] Handling call
0709 13:04:13.083794 (+   424us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:13.084040 (+   246us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:13.084071 (+    31us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:13.084127 (+    56us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:04:13.084169 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:13.084231 (+    62us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:13.084337 (+   106us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:13.086115 10952 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49424 (request call id 205) took 1313 ms. Trace:
I20260709 13:04:12.997962 11103 raft_consensus.cc:2410] T d71ca3a1ba6b48478aaf62741bf3cf6a P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 10ca5ee68ad64b3283b9d235352c9d23 for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 13:04:13.044229 11102 raft_consensus.cc:2468] T d270b9e5481245d9b5340a454a7cd10b P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 4.
I20260709 13:04:13.025712 10872 raft_consensus.cc:3060] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:04:13.060416 11028 raft_consensus.cc:2393] T 1e4f2241a94a41f38e481137f868b83d P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 10ca5ee68ad64b3283b9d235352c9d23 in current term 3: Already voted for candidate 186b12638a87410487467df721a3cbda in this term.
I20260709 13:04:13.061097 11797 leader_election.cc:290] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:13.061769 10919 leader_election.cc:304] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23; no voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47
I20260709 13:04:13.028352 12103 raft_consensus.cc:2749] T 401a9cf11eac479aa445ec8c1d6e33b2 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 13:04:13.030990 12172 raft_consensus.cc:493] T c51782cd9d064ec0949e84e24dd4de67 P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:13.116691 10919 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:45807, user_credentials={real_user=slave}} blocked reactor thread for 55040.6us
I20260709 13:04:13.066797 10880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b37f8fca8dc42718b2a42f0883a956f" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:13.021189 10876 raft_consensus.cc:2393] T 0e5d327630e5438b878035d28efe6c0c P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 873357742d5843579556ee8166f23f47 in current term 4: Already voted for candidate 186b12638a87410487467df721a3cbda in this term.
I20260709 13:04:13.070101 11098 raft_consensus.cc:2468] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 3.
I20260709 13:04:13.086822 11937 raft_consensus.cc:493] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:13.084729 10775 catalog_manager.cc:5697] T 525bfdbd6d934afd823f7d3ee3f9d052 P 10ca5ee68ad64b3283b9d235352c9d23 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "10ca5ee68ad64b3283b9d235352c9d23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } health_report { overall_health: HEALTHY } } }
I20260709 13:04:13.087836 10874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d270b9e5481245d9b5340a454a7cd10b" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:13.032712 12066 raft_consensus.cc:493] T 836251ebc2b74f5da88b8a8b2338e2a5 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 10ca5ee68ad64b3283b9d235352c9d23)
W20260709 13:04:13.088600 11066 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:13.015298 10954 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5299b8d044f41b7b68057e63d17b159" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
W20260709 13:04:13.090085 10920 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 63637.4us
I20260709 13:04:13.111487 11779 raft_consensus.cc:2804] T 32147c6d2d5a477aa38c597c7f78df5b P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:04:13.090207 10994 leader_election.cc:400] T 0e5d327630e5438b878035d28efe6c0c P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 4 election: Vote denied by peer 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507) with higher term. Message: Invalid argument: T 0e5d327630e5438b878035d28efe6c0c P 0172fed22b6b4890a7fdc953fa128af4 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 873357742d5843579556ee8166f23f47 for earlier term 4. Current term is 5.
I20260709 13:04:13.146302 10994 leader_election.cc:403] T 0e5d327630e5438b878035d28efe6c0c P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
W20260709 13:04:13.146929 10994 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 56927.3us
I20260709 13:04:13.147441 10921 leader_election.cc:304] T 0f9df2bc281f4866b6da5da94a0defc7 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23; no voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda
I20260709 13:04:13.148407 10994 leader_election.cc:304] T 525bfdbd6d934afd823f7d3ee3f9d052 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47; no voters: 10ca5ee68ad64b3283b9d235352c9d23
I20260709 13:04:12.889007 10950 rpcz_store.cc:276] 0709 13:04:10.942619 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:11.711840 (+769221us) service_pool.cc:224] Handling call
0709 13:04:12.169457 (+457617us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:12.257040 (+ 87583us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:12.733970 (+476930us) spinlock_profiling.cc:243] Waited 436 ms on lock 0x7b5400789950. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661618ad 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:04:12.734172 (+   202us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1002124544}
I20260709 13:04:13.033344 12105 raft_consensus.cc:2804] T e5a5f5a608504650b48fd27f163faed6 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:04:13.095265 10879 raft_consensus.cc:1275] T f371d2565dd84667b5613848604949bb P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Refusing update from remote peer 873357742d5843579556ee8166f23f47: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20260709 13:04:13.096629 11101 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "868e92339c2f4508af0afbd07c814d64" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:13.151367 10879 pending_rounds.cc:85] T f371d2565dd84667b5613848604949bb P 186b12638a87410487467df721a3cbda: Aborting all ops after (but not including) 1
I20260709 13:04:13.061617 10953 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d439326f23747eaab3c1e7da2d6b785" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:13.151755 10879 pending_rounds.cc:107] T f371d2565dd84667b5613848604949bb P 186b12638a87410487467df721a3cbda: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20260709 13:04:13.152123 10879 raft_consensus.cc:2889] T f371d2565dd84667b5613848604949bb P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:04:13.152402 10953 raft_consensus.cc:2468] T 5d439326f23747eaab3c1e7da2d6b785 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 4.
I20260709 13:04:13.104137 11021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32147c6d2d5a477aa38c597c7f78df5b" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:13.152316 10878 raft_consensus.cc:2468] T a1ec2b50c6ad4f7c8033476208abb26d P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 2.
I20260709 13:04:13.034368 12004 consensus_queue.cc:260] T a1ec2b50c6ad4f7c8033476208abb26d P 0172fed22b6b4890a7fdc953fa128af4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:13.106139 12158 leader_election.cc:290] T d270b9e5481245d9b5340a454a7cd10b P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:13.106895 11025 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb7d67be500947b3aef4c4245e33671c" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
I20260709 13:04:13.160948 10921 leader_election.cc:304] T 4b37f8fca8dc42718b2a42f0883a956f P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23; no voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47
I20260709 13:04:13.107945 11103 raft_consensus.cc:1240] T bb7d67be500947b3aef4c4245e33671c P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Rejecting Update request from peer 873357742d5843579556ee8166f23f47 for earlier term 1. Current term is 2. Ops: []
I20260709 13:04:13.108193 11020 raft_consensus.cc:1275] T 6a9353d1d230416296627d7cf8b1d425 P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Refusing update from remote peer 0172fed22b6b4890a7fdc953fa128af4: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 3. (term mismatch)
I20260709 13:04:13.109004 11023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75492f86fde046c3900df3eac8292fd4" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:13.109651 12019 raft_consensus.cc:493] T d71ca3a1ba6b48478aaf62741bf3cf6a P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:13.047528 12095 raft_consensus.cc:2749] T 72b111ad7b5f4753ab1d2b5ee1114840 P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:04:13.129976 11553 raft_consensus.cc:2749] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 10ca5ee68ad64b3283b9d235352c9d23 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 13:04:13.027660 11943 consensus_queue.cc:1059] T 5de5f0581e3b41dea729363d6ca563bb P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.064s
I20260709 13:04:13.111552 11028 raft_consensus.cc:1217] T 7757ecbf402e47bfbc93d0447741e26c P 873357742d5843579556ee8166f23f47 [term 3 LEADER]: Deduplicated request from leader. Original: 1.1->[2.2-2.2]   Dedup: 2.2->[]
I20260709 13:04:13.118081 10919 leader_election.cc:304] T 7839ca2d2c13431a85f44d7c458c6351 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23; no voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47
I20260709 13:04:13.121666 11024 raft_consensus.cc:3055] T a977842ba4b744f4bb2febd55d286006 P 873357742d5843579556ee8166f23f47 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:04:13.175622 11028 raft_consensus.cc:1240] T 7757ecbf402e47bfbc93d0447741e26c P 873357742d5843579556ee8166f23f47 [term 3 LEADER]: Rejecting Update request from peer 0172fed22b6b4890a7fdc953fa128af4 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260709 13:04:13.124673 10995 leader_election.cc:304] T 2567c2d84e4648d7ba5dc23a3df32817 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 873357742d5843579556ee8166f23f47; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda
I20260709 13:04:13.070344 11100 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4cd3d7dfbbf4096908ff11e9dc57141" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:13.133107 10775 catalog_manager.cc:5697] T 836251ebc2b74f5da88b8a8b2338e2a5 P 10ca5ee68ad64b3283b9d235352c9d23 reported cstate change: term changed from 1 to 3, leader changed from 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196) to 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194). New cstate: current_term: 3 leader_uuid: "10ca5ee68ad64b3283b9d235352c9d23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } health_report { overall_health: HEALTHY } } }
I20260709 13:04:13.163225 11020 pending_rounds.cc:85] T 6a9353d1d230416296627d7cf8b1d425 P 873357742d5843579556ee8166f23f47: Aborting all ops after (but not including) 2
I20260709 13:04:13.184439 10775 catalog_manager.cc:5697] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 10ca5ee68ad64b3283b9d235352c9d23 reported cstate change: term changed from 1 to 4, leader changed from 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194) to 873357742d5843579556ee8166f23f47 (127.0.107.195). New cstate: current_term: 4 leader_uuid: "873357742d5843579556ee8166f23f47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } }
I20260709 13:04:13.099968 10952 rpcz_store.cc:276] 0709 13:04:11.772950 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:13.084501 (+1311551us) service_pool.cc:224] Handling call
0709 13:04:13.084908 (+   407us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:13.085807 (+   899us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:13.086044 (+   237us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:13.091313 10845 leader_election.cc:304] T c31881534c5e4f0089c82057c1746652 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 186b12638a87410487467df721a3cbda; no voters: 0172fed22b6b4890a7fdc953fa128af4, 10ca5ee68ad64b3283b9d235352c9d23
I20260709 13:04:13.103924 11029 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e5d327630e5438b878035d28efe6c0c" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
I20260709 13:04:13.154402 10953 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65bf0a62f4d84ef18dd6b23e082724b8" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:13.082252 10875 rpcz_store.cc:276] 0709 13:04:11.838015 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:11.841063 (+  3048us) service_pool.cc:224] Handling call
0709 13:04:12.777378 (+936315us) spinlock_profiling.cc:243] Waited 1.26 ms on lock 0x7b5800000098. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f426303db71 00007f426303deee 0000000000375939 00007f426b0b11e0 00007f426b17a0c4 00007f426b17a131 00007f426b13bfc0 00007f426b126929 00007f426b1263d2 00007f426469dbd4
0709 13:04:12.777770 (+   392us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:12.857270 (+ 79500us) raft_consensus.cc:1534] Early marking committed up to index 3
0709 13:04:12.857845 (+   575us) log.cc:844] Serialized 14 byte log entry
0709 13:04:13.005686 (+147841us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:13.010759 (+  5073us) raft_consensus.cc:1645] Marking committed up to 3
0709 13:04:13.013075 (+  2316us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:13.081390 (+ 68315us) spinlock_profiling.cc:243] Waited 61.3 ms on lock 0x7b54003223d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:04:13.081498 (+   108us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:13.081662 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1807,"prepare.queue_time_us":184,"prepare.run_cpu_time_us":111,"prepare.run_wall_time_us":112,"raft.queue_time_us":13210,"raft.run_cpu_time_us":147,"raft.run_wall_time_us":7849,"spinlock_wait_cycles":143919744,"thread_start_us":93790,"threads_started":1}
I20260709 13:04:13.156425 10878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d439326f23747eaab3c1e7da2d6b785" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:13.154201 10920 leader_election.cc:304] T 8a69f32c53f24a50b7852d143665d7b4 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23; no voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda
I20260709 13:04:13.163111 11103 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37676 (request call id 251) took 1192 ms. Trace:
I20260709 13:04:13.218058 11103 rpcz_store.cc:276] 0709 13:04:11.970964 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:13.107336 (+1136372us) service_pool.cc:224] Handling call
0709 13:04:13.107701 (+   365us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:13.162808 (+ 55107us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:13.163041 (+   233us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:13.111055 10877 raft_consensus.cc:2468] T 868e92339c2f4508af0afbd07c814d64 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 2.
I20260709 13:04:13.116350 11926 raft_consensus.cc:493] T c31881534c5e4f0089c82057c1746652 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 10ca5ee68ad64b3283b9d235352c9d23)
I20260709 13:04:13.167047 10880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63d44f6ba6434ab2870fe3190f4d7c2f" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:04:13.156708 12066 raft_consensus.cc:515] T 836251ebc2b74f5da88b8a8b2338e2a5 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:13.169601 11101 raft_consensus.cc:1240] T a977842ba4b744f4bb2febd55d286006 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 LEADER]: Rejecting Update request from peer 873357742d5843579556ee8166f23f47 for earlier term 1. Current term is 2. Ops: []
W20260709 13:04:13.169761 12183 log.cc:927] Time spent T 401a9cf11eac479aa445ec8c1d6e33b2 P 0172fed22b6b4890a7fdc953fa128af4: Append to log took a long time: real 0.080s	user 0.000s	sys 0.001s
I20260709 13:04:13.173444 11795 leader_election.cc:290] T f371d2565dd84667b5613848604949bb P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:13.174016 11023 raft_consensus.cc:2468] T 75492f86fde046c3900df3eac8292fd4 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 2.
I20260709 13:04:13.142189 12088 raft_consensus.cc:3060] T 0e5d327630e5438b878035d28efe6c0c P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Advancing to term 5
W20260709 13:04:13.177315 10919 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:45807, user_credentials={real_user=slave}} blocked reactor thread for 59364.4us
I20260709 13:04:13.205710 11094 raft_consensus.cc:1275] T a610d1a71f52432e88fe56130f921fa2 P 0172fed22b6b4890a7fdc953fa128af4 [term 5 FOLLOWER]: Refusing update from remote peer 873357742d5843579556ee8166f23f47: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260709 13:04:13.237187 10919 leader_election.cc:304] T e89e640169e14e2d98f968f76027daa7 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23; no voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47
I20260709 13:04:13.230496 10953 raft_consensus.cc:1240] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 10ca5ee68ad64b3283b9d235352c9d23 [term 5 FOLLOWER]: Rejecting Update request from peer 873357742d5843579556ee8166f23f47 for earlier term 4. Current term is 5. Ops: []
I20260709 13:04:13.165195 11974 raft_consensus.cc:2749] T 0f9df2bc281f4866b6da5da94a0defc7 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:04:13.146049 11779 raft_consensus.cc:493] T 32147c6d2d5a477aa38c597c7f78df5b P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:13.175149 10950 raft_consensus.cc:1275] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Refusing update from remote peer 0172fed22b6b4890a7fdc953fa128af4: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 1. (term mismatch)
I20260709 13:04:13.165901 12019 raft_consensus.cc:515] T d71ca3a1ba6b48478aaf62741bf3cf6a P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:13.215798 10878 raft_consensus.cc:2393] T 5d439326f23747eaab3c1e7da2d6b785 P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0172fed22b6b4890a7fdc953fa128af4 in current term 4: Already voted for candidate 186b12638a87410487467df721a3cbda in this term.
I20260709 13:04:13.079413 12096 consensus_queue.cc:260] T 0f9df2bc281f4866b6da5da94a0defc7 P 0172fed22b6b4890a7fdc953fa128af4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:13.163146 11653 raft_consensus.cc:2804] T 525bfdbd6d934afd823f7d3ee3f9d052 P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:04:13.175179 11025 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5ba72ccd854430baadbc636dfd17136" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:13.084659 10949 rpcz_store.cc:276] 0709 13:04:11.947829 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:13.083212 (+1135383us) service_pool.cc:224] Handling call
0709 13:04:13.083605 (+   393us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:13.083849 (+   244us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:13.083879 (+    30us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:13.083935 (+    56us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:04:13.083974 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:13.084043 (+    69us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:13.084154 (+   111us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:13.274329 12127 raft_consensus.cc:2749] T e89e640169e14e2d98f968f76027daa7 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:04:13.275300 12127 raft_consensus.cc:493] T ff13ae9a47c7455f962406c6d131a3ff P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:13.275866 12127 raft_consensus.cc:515] T ff13ae9a47c7455f962406c6d131a3ff P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:13.278371 12127 leader_election.cc:290] T ff13ae9a47c7455f962406c6d131a3ff P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:13.254406 11553 raft_consensus.cc:2749] T 8a69f32c53f24a50b7852d143665d7b4 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260709 13:04:13.279585 12186 log.cc:927] Time spent T 30f628e3d7914d609e7f1d980a74c654 P 873357742d5843579556ee8166f23f47: Append to log took a long time: real 0.055s	user 0.001s	sys 0.000s
I20260709 13:04:13.282835 10949 raft_consensus.cc:1275] T b4cd3d7dfbbf4096908ff11e9dc57141 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Refusing update from remote peer 873357742d5843579556ee8166f23f47: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 13:04:13.160442 11953 raft_consensus.cc:3060] T 0e5d327630e5438b878035d28efe6c0c P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:04:13.129704 12172 raft_consensus.cc:515] T c51782cd9d064ec0949e84e24dd4de67 P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:13.163328 10954 raft_consensus.cc:2468] T a5299b8d044f41b7b68057e63d17b159 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 2.
I20260709 13:04:13.142195 11937 raft_consensus.cc:515] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:13.166498 12105 raft_consensus.cc:697] T e5a5f5a608504650b48fd27f163faed6 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 LEADER]: Becoming Leader. State: Replica: 10ca5ee68ad64b3283b9d235352c9d23, State: Running, Role: LEADER
I20260709 13:04:13.224351 11101 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37676 (request call id 255) took 1192 ms. Trace:
I20260709 13:04:13.293673 11101 rpcz_store.cc:276] 0709 13:04:12.032033 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:13.168980 (+1136947us) service_pool.cc:224] Handling call
0709 13:04:13.169381 (+   401us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:13.224069 (+ 54688us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:13.224283 (+   214us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:04:13.226576 10845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 135365us
I20260709 13:04:13.295552 11103 raft_consensus.cc:1275] T d71ca3a1ba6b48478aaf62741bf3cf6a P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Refusing update from remote peer 873357742d5843579556ee8166f23f47: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:04:13.186295 10775 catalog_manager.cc:5697] T d270b9e5481245d9b5340a454a7cd10b P 10ca5ee68ad64b3283b9d235352c9d23 reported cstate change: term changed from 1 to 3, leader changed from 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196) to 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194). New cstate: current_term: 3 leader_uuid: "10ca5ee68ad64b3283b9d235352c9d23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } health_report { overall_health: HEALTHY } } }
I20260709 13:04:13.164011 10951 raft_consensus.cc:1275] T d71ca3a1ba6b48478aaf62741bf3cf6a P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Refusing update from remote peer 873357742d5843579556ee8166f23f47: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:04:13.205145 11020 pending_rounds.cc:107] T 6a9353d1d230416296627d7cf8b1d425 P 873357742d5843579556ee8166f23f47: Aborting uncommitted NO_OP operation due to leader change: 3.3
I20260709 13:04:13.180687 11639 raft_consensus.cc:2749] T 4b37f8fca8dc42718b2a42f0883a956f P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:04:13.184655 11961 raft_consensus.cc:2749] T 7839ca2d2c13431a85f44d7c458c6351 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 13:04:13.176081 11024 raft_consensus.cc:740] T a977842ba4b744f4bb2febd55d286006 P 873357742d5843579556ee8166f23f47 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 873357742d5843579556ee8166f23f47, State: Running, Role: LEADER
I20260709 13:04:13.216145 12004 raft_consensus.cc:3060] T a1ec2b50c6ad4f7c8033476208abb26d P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:04:13.214862 12098 raft_consensus.cc:2749] T 2567c2d84e4648d7ba5dc23a3df32817 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 13:04:13.247718 10920 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 93663.2us
I20260709 13:04:13.286638 11653 raft_consensus.cc:493] T 525bfdbd6d934afd823f7d3ee3f9d052 P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 10ca5ee68ad64b3283b9d235352c9d23)
W20260709 13:04:13.185727 10995 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:38967, user_credentials={real_user=slave}} blocked reactor thread for 61167us
I20260709 13:04:13.311407 10995 leader_election.cc:304] T 7709387f4d3d4f9992d97a0fbca39a40 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47; no voters: 0172fed22b6b4890a7fdc953fa128af4
I20260709 13:04:13.141956 12070 consensus_queue.cc:260] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 873357742d5843579556ee8166f23f47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:13.233511 12095 raft_consensus.cc:2749] T c31881534c5e4f0089c82057c1746652 P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 13:04:13.306921 11025 raft_consensus.cc:2393] T e5ba72ccd854430baadbc636dfd17136 P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0172fed22b6b4890a7fdc953fa128af4 in current term 4: Already voted for candidate 873357742d5843579556ee8166f23f47 in this term.
I20260709 13:04:13.259569 10919 leader_election.cc:304] T bb7d67be500947b3aef4c4245e33671c P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23; no voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47
I20260709 13:04:13.250468 11100 raft_consensus.cc:2410] T b4cd3d7dfbbf4096908ff11e9dc57141 P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 10ca5ee68ad64b3283b9d235352c9d23 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 13:04:13.248551 11779 raft_consensus.cc:3060] T 32147c6d2d5a477aa38c597c7f78df5b P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:04:13.296305 10845 leader_election.cc:304] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda; no voters: 
W20260709 13:04:13.320585 12189 log.cc:927] Time spent T 732c4adeeff6422497aa7db4a82f535b P 186b12638a87410487467df721a3cbda: Append to log took a long time: real 0.101s	user 0.000s	sys 0.000s
I20260709 13:04:13.230630 11926 raft_consensus.cc:515] T c31881534c5e4f0089c82057c1746652 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:13.254637 10876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d270b9e5481245d9b5340a454a7cd10b" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:13.248582 10950 pending_rounds.cc:85] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 10ca5ee68ad64b3283b9d235352c9d23: Aborting all ops after (but not including) 0
I20260709 13:04:13.284487 10875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d156d6ae2ca4fa59a1e387edf0fea39" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:13.309790 10880 raft_consensus.cc:2468] T 63d44f6ba6434ab2870fe3190f4d7c2f P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 3.
I20260709 13:04:13.223075 10871 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b67eff0634e540eab91cf8cee795969f" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:04:13.286085 11094 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37676 (request call id 248) took 1343 ms. Trace:
I20260709 13:04:13.313278 11102 raft_consensus.cc:1240] T b67eff0634e540eab91cf8cee795969f P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Rejecting Update request from peer 10ca5ee68ad64b3283b9d235352c9d23 for earlier term 1. Current term is 2. Ops: []
I20260709 13:04:13.268771 10878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72b111ad7b5f4753ab1d2b5ee1114840" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:04:13.350986 10878 raft_consensus.cc:3060] T 72b111ad7b5f4753ab1d2b5ee1114840 P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:04:13.293483 11943 raft_consensus.cc:493] T f34f2bc02d85444f9096e0f5b4fe4331 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:13.319893 10954 raft_consensus.cc:1275] T a977842ba4b744f4bb2febd55d286006 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Refusing update from remote peer 0172fed22b6b4890a7fdc953fa128af4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260709 13:04:13.319628 11024 consensus_queue.cc:260] T a977842ba4b744f4bb2febd55d286006 P 873357742d5843579556ee8166f23f47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
W20260709 13:04:13.329619 10920 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:13.300659 11020 raft_consensus.cc:2889] T 6a9353d1d230416296627d7cf8b1d425 P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:04:13.298026 10775 catalog_manager.cc:5697] T 87cd8f5097624c8692f1d83544d8881e P 10ca5ee68ad64b3283b9d235352c9d23 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "186b12638a87410487467df721a3cbda" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } }
I20260709 13:04:13.366614 10775 catalog_manager.cc:5697] T bac7e5040f984ba2b7146bd36af7267e P 10ca5ee68ad64b3283b9d235352c9d23 reported cstate change: term changed from 1 to 2, leader changed from 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196) to 873357742d5843579556ee8166f23f47 (127.0.107.195). New cstate: current_term: 2 leader_uuid: "873357742d5843579556ee8166f23f47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } }
I20260709 13:04:13.307626 10952 raft_consensus.cc:1275] T 34064dc7bf3747dbb3876dd824ee0688 P 10ca5ee68ad64b3283b9d235352c9d23 [term 5 FOLLOWER]: Refusing update from remote peer 873357742d5843579556ee8166f23f47: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
W20260709 13:04:13.337220 10919 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:45807, user_credentials={real_user=slave}} blocked reactor thread for 77768.3us
I20260709 13:04:13.375795 10919 leader_election.cc:304] T 4c2a35bb99374e93b573cf870d04faaa P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23; no voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47
I20260709 13:04:13.316340 11653 raft_consensus.cc:3060] T 525bfdbd6d934afd823f7d3ee3f9d052 P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:04:13.357949 11961 raft_consensus.cc:3060] T bb7d67be500947b3aef4c4245e33671c P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:04:13.381778 10919 leader_election.cc:304] T d71ca3a1ba6b48478aaf62741bf3cf6a P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23; no voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47
I20260709 13:04:13.315622 12052 raft_consensus.cc:493] T c31881534c5e4f0089c82057c1746652 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:13.356161 11102 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37666 (request call id 213) took 1009 ms. Trace:
I20260709 13:04:13.419697 12052 raft_consensus.cc:515] T c31881534c5e4f0089c82057c1746652 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:13.419891 11102 rpcz_store.cc:276] 0709 13:04:12.346407 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:13.292585 (+946178us) service_pool.cc:224] Handling call
0709 13:04:13.310273 (+ 17688us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:13.355835 (+ 45562us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:13.356084 (+   249us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:13.421231 11102 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "868e92339c2f4508af0afbd07c814d64" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:13.422487 12052 leader_election.cc:290] T c31881534c5e4f0089c82057c1746652 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:13.366670 12171 raft_consensus.cc:3060] T a5299b8d044f41b7b68057e63d17b159 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:04:13.304479 11795 consensus_queue.cc:1059] T 8c160ddfe9b34bdaa3835d0915c1e06f P 186b12638a87410487467df721a3cbda [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.036s
I20260709 13:04:13.424665 11795 raft_consensus.cc:3055] T 8c160ddfe9b34bdaa3835d0915c1e06f P 186b12638a87410487467df721a3cbda [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:04:13.425117 11795 raft_consensus.cc:740] T 8c160ddfe9b34bdaa3835d0915c1e06f P 186b12638a87410487467df721a3cbda [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 186b12638a87410487467df721a3cbda, State: Running, Role: LEADER
I20260709 13:04:13.425500 12098 raft_consensus.cc:493] T ff13ae9a47c7455f962406c6d131a3ff P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:13.351207 12195 consensus_queue.cc:1059] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 873357742d5843579556ee8166f23f47 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.083s
I20260709 13:04:13.358557 10921 leader_election.cc:304] T 63d44f6ba6434ab2870fe3190f4d7c2f P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda; no voters: 873357742d5843579556ee8166f23f47
I20260709 13:04:13.322968 10951 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49434 (request call id 244) took 1483 ms. Trace:
I20260709 13:04:13.369030 10775 catalog_manager.cc:5697] T 65bf0a62f4d84ef18dd6b23e082724b8 P 10ca5ee68ad64b3283b9d235352c9d23 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "10ca5ee68ad64b3283b9d235352c9d23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } health_report { overall_health: HEALTHY } } }
I20260709 13:04:13.427701 12195 consensus_queue.cc:1059] T bb7d67be500947b3aef4c4245e33671c P 873357742d5843579556ee8166f23f47 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260709 13:04:13.428056 10921 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:38967, user_credentials={real_user=slave}} blocked reactor thread for 69675.6us
I20260709 13:04:13.428956 12195 raft_consensus.cc:3055] T bb7d67be500947b3aef4c4245e33671c P 873357742d5843579556ee8166f23f47 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:04:13.426277 11795 consensus_queue.cc:260] T 8c160ddfe9b34bdaa3835d0915c1e06f P 186b12638a87410487467df721a3cbda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:13.429148 10775 catalog_manager.cc:5697] T 72b111ad7b5f4753ab1d2b5ee1114840 P 10ca5ee68ad64b3283b9d235352c9d23 reported cstate change: term changed from 2 to 3, leader changed from 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196) to 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194). New cstate: current_term: 3 leader_uuid: "10ca5ee68ad64b3283b9d235352c9d23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } health_report { overall_health: HEALTHY } } }
I20260709 13:04:13.430742 12052 raft_consensus.cc:2804] T 63d44f6ba6434ab2870fe3190f4d7c2f P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:04:13.350410 11094 rpcz_store.cc:276] 0709 13:04:11.943081 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:13.102662 (+1159581us) service_pool.cc:224] Handling call
0709 13:04:13.145015 (+ 42353us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:13.239116 (+ 94101us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:13.270788 (+ 31672us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:04:13.348057 10992 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 109401us
W20260709 13:04:13.384374 11067 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:13.356333 11098 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37676 (request call id 254) took 1318 ms. Trace:
I20260709 13:04:13.311872 10874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e96e80ba9ac34d0998395515e5c8c41a" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
W20260709 13:04:13.373183 12185 log.cc:927] Time spent T e89e640169e14e2d98f968f76027daa7 P 186b12638a87410487467df721a3cbda: Append to log took a long time: real 0.210s	user 0.000s	sys 0.001s
I20260709 13:04:13.053030 10946 rpcz_store.cc:276] 0709 13:04:10.661930 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:11.652670 (+990740us) service_pool.cc:224] Handling call
0709 13:04:11.670389 (+ 17719us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:04:11.730543 (+ 60154us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:11.730582 (+    39us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:04:11.731316 (+   734us) log.cc:844] Serialized 52 byte log entry
0709 13:04:11.768167 (+ 36851us) raft_consensus.cc:1645] Marking committed up to 1
0709 13:04:11.768225 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:11.768326 (+   101us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:12.820784 (+1052458us) spinlock_profiling.cc:243] Waited 23.4 ms on lock 0x7b4400022b4c. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f4262f9bc85 00007f4262f9c6a2 00007f4265d0e1db 00007f4265d09aaf 00007f4265cfd86e 00007f4265cfcbb6 00007f4265cfc7d8 00007f4265d06221 00007f4265d061d9 00007f4265d06169
0709 13:04:12.948353 (+127569us) raft_consensus.cc:1685] finished
0709 13:04:12.948390 (+    37us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:12.948680 (+   290us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1138,"raft.queue_time_us":7581,"raft.run_cpu_time_us":124,"raft.run_wall_time_us":124,"spinlock_wait_cycles":53896448,"thread_start_us":14873,"threads_started":1,"wal-append.queue_time_us":12082}
I20260709 13:04:13.403096 10871 raft_consensus.cc:2393] T b67eff0634e540eab91cf8cee795969f P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 0172fed22b6b4890a7fdc953fa128af4 in current term 2: Already voted for candidate 186b12638a87410487467df721a3cbda in this term.
I20260709 13:04:13.365899 11943 raft_consensus.cc:515] T f34f2bc02d85444f9096e0f5b4fe4331 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:13.403890 11095 raft_consensus.cc:2410] T 7757ecbf402e47bfbc93d0447741e26c P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 186b12638a87410487467df721a3cbda for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 13:04:13.404603 10954 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5ba72ccd854430baadbc636dfd17136" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:13.405755 10876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e5d327630e5438b878035d28efe6c0c" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:04:13.454991 12202 consensus_queue.cc:1048] T a610d1a71f52432e88fe56130f921fa2 P 873357742d5843579556ee8166f23f47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.006s
I20260709 13:04:13.455250 10871 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c31881534c5e4f0089c82057c1746652" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:13.456179 10871 raft_consensus.cc:2393] T c31881534c5e4f0089c82057c1746652 P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0172fed22b6b4890a7fdc953fa128af4 in current term 3: Already voted for candidate 186b12638a87410487467df721a3cbda in this term.
I20260709 13:04:13.457170 10871 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "836251ebc2b74f5da88b8a8b2338e2a5" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:13.457805 10871 raft_consensus.cc:2468] T 836251ebc2b74f5da88b8a8b2338e2a5 P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 3.
I20260709 13:04:13.459869 11094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c51782cd9d064ec0949e84e24dd4de67" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:13.460774 11094 raft_consensus.cc:2468] T c51782cd9d064ec0949e84e24dd4de67 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 3.
W20260709 13:04:13.462529 11069 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:13.406018 11926 leader_election.cc:290] T c31881534c5e4f0089c82057c1746652 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:13.407264 12066 leader_election.cc:290] T 836251ebc2b74f5da88b8a8b2338e2a5 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:13.407644 11097 raft_consensus.cc:2468] T e5a5f5a608504650b48fd27f163faed6 P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 4.
I20260709 13:04:13.407868 11100 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f371d2565dd84667b5613848604949bb" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:13.377496 12105 consensus_queue.cc:237] T e5a5f5a608504650b48fd27f163faed6 P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:13.475283 11097 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37666 (request call id 211) took 1142 ms. Trace:
I20260709 13:04:13.475455 11100 raft_consensus.cc:2468] T f371d2565dd84667b5613848604949bb P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 3.
I20260709 13:04:13.475849 11097 rpcz_store.cc:276] 0709 13:04:12.332452 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:12.760103 (+427651us) service_pool.cc:224] Handling call
0709 13:04:13.475195 (+715092us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:13.366832 12193 raft_consensus.cc:493] T d7ce47b9a92d43dd88d95bcda5038597 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 10ca5ee68ad64b3283b9d235352c9d23)
W20260709 13:04:13.408720 12190 log.cc:927] Time spent T b67eff0634e540eab91cf8cee795969f P 10ca5ee68ad64b3283b9d235352c9d23: Append to log took a long time: real 0.107s	user 0.000s	sys 0.000s
I20260709 13:04:13.409616 10949 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d270b9e5481245d9b5340a454a7cd10b" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:13.379748 11024 raft_consensus.cc:3060] T a977842ba4b744f4bb2febd55d286006 P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:04:13.442655 12073 raft_consensus.cc:2749] T d71ca3a1ba6b48478aaf62741bf3cf6a P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:04:13.400393 12009 raft_consensus.cc:2749] T 4c2a35bb99374e93b573cf870d04faaa P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:04:13.410899 11022 raft_consensus.cc:2468] T 57b6a2418c1342e7924195e9c1dbd357 P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 3.
I20260709 13:04:13.415321 11096 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37678 (request call id 128) took 1485 ms. Trace:
I20260709 13:04:13.417368 12172 leader_election.cc:290] T c51782cd9d064ec0949e84e24dd4de67 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:13.418099 11953 raft_consensus.cc:2749] T 0e5d327630e5438b878035d28efe6c0c P 873357742d5843579556ee8166f23f47 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507) with higher term. Message: Invalid argument: T 0e5d327630e5438b878035d28efe6c0c P 0172fed22b6b4890a7fdc953fa128af4 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 873357742d5843579556ee8166f23f47 for earlier term 4. Current term is 5.
I20260709 13:04:13.445073 12052 raft_consensus.cc:697] T 63d44f6ba6434ab2870fe3190f4d7c2f P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 LEADER]: Becoming Leader. State: Replica: 10ca5ee68ad64b3283b9d235352c9d23, State: Running, Role: LEADER
I20260709 13:04:13.515391 10879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5de5f0581e3b41dea729363d6ca563bb" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:04:13.418202 11099 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37676 (request call id 246) took 1480 ms. Trace:
I20260709 13:04:13.375231 12050 leader_election.cc:290] T 868e92339c2f4508af0afbd07c814d64 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:13.375193 11921 consensus_queue.cc:1048] T 7757ecbf402e47bfbc93d0447741e26c P 873357742d5843579556ee8166f23f47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.044s
W20260709 13:04:13.419108 11068 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:13.485337 11026 raft_consensus.cc:1240] T 732c4adeeff6422497aa7db4a82f535b P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Rejecting Update request from peer 10ca5ee68ad64b3283b9d235352c9d23 for earlier term 2. Current term is 3. Ops: []
I20260709 13:04:13.355546 10875 raft_consensus.cc:2468] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 3.
I20260709 13:04:13.430792 11937 leader_election.cc:290] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:13.427451 10951 rpcz_store.cc:276] 0709 13:04:11.839406 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:13.078279 (+1238873us) service_pool.cc:224] Handling call
0709 13:04:13.116541 (+ 38262us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:13.303263 (+186722us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:13.319358 (+ 16095us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:13.346899 10950 pending_rounds.cc:107] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 10ca5ee68ad64b3283b9d235352c9d23: Aborting uncommitted NO_OP operation due to leader change: 2.1
I20260709 13:04:13.431362 10775 catalog_manager.cc:5697] T 732c4adeeff6422497aa7db4a82f535b P 10ca5ee68ad64b3283b9d235352c9d23 reported cstate change: term changed from 1 to 2, leader changed from 873357742d5843579556ee8166f23f47 (127.0.107.195) to 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194). New cstate: current_term: 2 leader_uuid: "10ca5ee68ad64b3283b9d235352c9d23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } health_report { overall_health: UNKNOWN } } }
W20260709 13:04:13.409688 12194 log.cc:927] Time spent T a64b7e99d6c54934a7de6f6dffcf5e94 P 0172fed22b6b4890a7fdc953fa128af4: Append to log took a long time: real 0.069s	user 0.000s	sys 0.000s
I20260709 13:04:13.478403 11097 raft_consensus.cc:1275] T f371d2565dd84667b5613848604949bb P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Refusing update from remote peer 873357742d5843579556ee8166f23f47: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 13:04:13.401851 10878 raft_consensus.cc:2468] T 72b111ad7b5f4753ab1d2b5ee1114840 P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 4.
W20260709 13:04:13.441195 10919 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:45807, user_credentials={real_user=slave}} blocked reactor thread for 59605.9us
I20260709 13:04:13.354910 10995 leader_election.cc:304] T a1ec2b50c6ad4f7c8033476208abb26d P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47; no voters: 0172fed22b6b4890a7fdc953fa128af4
W20260709 13:04:13.446813 10846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:13.540607 10919 leader_election.cc:304] T a154cc8b181f460abf4880064379948b P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23; no voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47
I20260709 13:04:13.426120 12098 raft_consensus.cc:515] T ff13ae9a47c7455f962406c6d131a3ff P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:13.429587 12195 raft_consensus.cc:740] T bb7d67be500947b3aef4c4245e33671c P 873357742d5843579556ee8166f23f47 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 873357742d5843579556ee8166f23f47, State: Running, Role: LEADER
I20260709 13:04:13.464105 11069 leader_election.cc:304] T b67eff0634e540eab91cf8cee795969f P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda
I20260709 13:04:13.453409 10992 leader_election.cc:304] T a5299b8d044f41b7b68057e63d17b159 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47; no voters: 
W20260709 13:04:13.547411 10992 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 94084.8us
I20260709 13:04:13.463936 10845 leader_election.cc:304] T c51782cd9d064ec0949e84e24dd4de67 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda; no voters: 
PC: @                0x0 (unknown)
I20260709 13:04:13.468184 12196 consensus_queue.cc:1048] T b4cd3d7dfbbf4096908ff11e9dc57141 P 873357742d5843579556ee8166f23f47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.070s
I20260709 13:04:13.497053 11103 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37676 (request call id 256) took 1448 ms. Trace:
I20260709 13:04:13.521988 10877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f34f2bc02d85444f9096e0f5b4fe4331" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:13.481386 10949 raft_consensus.cc:2410] T d270b9e5481245d9b5340a454a7cd10b P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0172fed22b6b4890a7fdc953fa128af4 for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20260709 13:04:13.480123 12048 consensus_meta.cc:287] T 87cd8f5097624c8692f1d83544d8881e P 186b12638a87410487467df721a3cbda: Time spent flushing consensus metadata: real 0.567s	user 0.007s	sys 0.007s
I20260709 13:04:13.423614 10952 raft_consensus.cc:1240] T a977842ba4b744f4bb2febd55d286006 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Rejecting Update request from peer 873357742d5843579556ee8166f23f47 for earlier term 1. Current term is 2. Ops: []
I20260709 13:04:13.419652 11779 raft_consensus.cc:515] T 32147c6d2d5a477aa38c597c7f78df5b P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:13.488145 11100 raft_consensus.cc:1275] T b4cd3d7dfbbf4096908ff11e9dc57141 P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Refusing update from remote peer 873357742d5843579556ee8166f23f47: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 13:04:13.535912 10775 catalog_manager.cc:5697] T e96e80ba9ac34d0998395515e5c8c41a P 10ca5ee68ad64b3283b9d235352c9d23 reported cstate change: term changed from 1 to 3, leader changed from 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194) to 873357742d5843579556ee8166f23f47 (127.0.107.195). New cstate: current_term: 3 leader_uuid: "873357742d5843579556ee8166f23f47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } }
I20260709 13:04:13.515549 12103 raft_consensus.cc:2804] T 7709387f4d3d4f9992d97a0fbca39a40 P 873357742d5843579556ee8166f23f47 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260709 13:04:13.544441 10995 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:38967, user_credentials={real_user=slave}} blocked reactor thread for 189682us
I20260709 13:04:13.548182 10921 leader_election.cc:304] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda; no voters: 0172fed22b6b4890a7fdc953fa128af4
I20260709 13:04:13.516696 10879 raft_consensus.cc:3060] T 5de5f0581e3b41dea729363d6ca563bb P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:04:13.549733 10874 raft_consensus.cc:1240] T 72b111ad7b5f4753ab1d2b5ee1114840 P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Rejecting Update request from peer 10ca5ee68ad64b3283b9d235352c9d23 for earlier term 3. Current term is 4. Ops: []
I20260709 13:04:13.550508 12071 raft_consensus.cc:2749] T a154cc8b181f460abf4880064379948b P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:04:13.586169 12063 raft_consensus.cc:2804] T c51782cd9d064ec0949e84e24dd4de67 P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:04:13.591528 10952 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff13ae9a47c7455f962406c6d131a3ff" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:13.592350 12063 raft_consensus.cc:493] T c51782cd9d064ec0949e84e24dd4de67 P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:13.592679 10952 raft_consensus.cc:2468] T ff13ae9a47c7455f962406c6d131a3ff P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 2.
I20260709 13:04:13.580595 10946 raft_consensus.cc:1275] T 11de93cd9b9d457489ff1cb80f976c9a P 10ca5ee68ad64b3283b9d235352c9d23 [term 5 FOLLOWER]: Refusing update from remote peer 873357742d5843579556ee8166f23f47: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260709 13:04:13.595571 10949 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "836251ebc2b74f5da88b8a8b2338e2a5" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:13.599709 11067 leader_election.cc:304] T bb7d67be500947b3aef4c4245e33671c P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47
I20260709 13:04:13.600236 10952 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f1466124dbd4f6ebb3c4d62ca48bf96" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:13.601735 11067 leader_election.cc:304] T 32147c6d2d5a477aa38c597c7f78df5b P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47
I20260709 13:04:13.604012 10952 raft_consensus.cc:2393] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 10ca5ee68ad64b3283b9d235352c9d23 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0172fed22b6b4890a7fdc953fa128af4 in current term 5: Already voted for candidate 10ca5ee68ad64b3283b9d235352c9d23 in this term.
I20260709 13:04:13.605729 11961 raft_consensus.cc:2749] T bb7d67be500947b3aef4c4245e33671c P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:04:13.551085 11953 raft_consensus.cc:2804] T a5299b8d044f41b7b68057e63d17b159 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:04:13.610551 11953 raft_consensus.cc:493] T a5299b8d044f41b7b68057e63d17b159 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:13.611061 11953 raft_consensus.cc:3060] T a5299b8d044f41b7b68057e63d17b159 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:04:13.567946 10875 raft_consensus.cc:1240] T 732c4adeeff6422497aa7db4a82f535b P 186b12638a87410487467df721a3cbda [term 3 LEADER]: Rejecting Update request from peer 10ca5ee68ad64b3283b9d235352c9d23 for earlier term 2. Current term is 3. Ops: []
I20260709 13:04:13.612744 10875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c31881534c5e4f0089c82057c1746652" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:13.551632 12195 consensus_queue.cc:260] T bb7d67be500947b3aef4c4245e33671c P 873357742d5843579556ee8166f23f47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:13.552767 12070 raft_consensus.cc:3060] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:04:13.569926 11943 leader_election.cc:290] T f34f2bc02d85444f9096e0f5b4fe4331 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:13.556056 12009 consensus_queue.cc:1048] T 75492f86fde046c3900df3eac8292fd4 P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:04:13.533314 12088 raft_consensus.cc:2749] T 0e5d327630e5438b878035d28efe6c0c P 186b12638a87410487467df721a3cbda [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:04:13.534948 10950 raft_consensus.cc:2889] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:04:13.557073 10948 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49434 (request call id 242) took 1833 ms. Trace:
I20260709 13:04:13.559006 10843 leader_election.cc:304] T d7ce47b9a92d43dd88d95bcda5038597 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda; no voters: 0172fed22b6b4890a7fdc953fa128af4
I20260709 13:04:13.479950 12193 raft_consensus.cc:515] T d7ce47b9a92d43dd88d95bcda5038597 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:13.561344 10951 raft_consensus.cc:1275] T 1e4f2241a94a41f38e481137f868b83d P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Refusing update from remote peer 186b12638a87410487467df721a3cbda: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 13:04:13.485548 11653 raft_consensus.cc:515] T 525bfdbd6d934afd823f7d3ee3f9d052 P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:13.519351 11099 rpcz_store.cc:276] 0709 13:04:11.942149 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:13.079215 (+1137066us) service_pool.cc:224] Handling call
0709 13:04:13.079695 (+   480us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:04:13.080175 (+   480us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:13.080213 (+    38us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:04:13.080965 (+   752us) log.cc:844] Serialized 52 byte log entry
0709 13:04:13.224048 (+143083us) raft_consensus.cc:1645] Marking committed up to 1
0709 13:04:13.232845 (+  8797us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:13.239136 (+  6291us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:13.417881 (+178745us) raft_consensus.cc:1685] finished
0709 13:04:13.417927 (+    46us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:13.418126 (+   199us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":6381,"raft.run_cpu_time_us":126,"raft.run_wall_time_us":131,"thread_start_us":127464,"threads_started":1,"wal-append.queue_time_us":226099}
I20260709 13:04:13.546044 10919 leader_election.cc:304] T e96e80ba9ac34d0998395515e5c8c41a P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23; no voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47
I20260709 13:04:13.564150 12203 raft_consensus.cc:2804] T a1ec2b50c6ad4f7c8033476208abb26d P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:04:13.565275 10920 leader_election.cc:304] T d270b9e5481245d9b5340a454a7cd10b P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 10ca5ee68ad64b3283b9d235352c9d23; no voters: 186b12638a87410487467df721a3cbda
I20260709 13:04:13.653012 10879 raft_consensus.cc:2468] T 5de5f0581e3b41dea729363d6ca563bb P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 4.
W20260709 13:04:13.653734 10920 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 88600.6us
I20260709 13:04:13.592789 12063 raft_consensus.cc:3060] T c51782cd9d064ec0949e84e24dd4de67 P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:04:13.567893 10947 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c31881534c5e4f0089c82057c1746652" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:13.664992 10947 raft_consensus.cc:2468] T c31881534c5e4f0089c82057c1746652 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 3.
I20260709 13:04:13.665190 10920 leader_election.cc:304] T b4cd3d7dfbbf4096908ff11e9dc57141 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23; no voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47
W20260709 13:04:13.570190 11069 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:38967, user_credentials={real_user=slave}} blocked reactor thread for 106193us
I20260709 13:04:13.657068 12172 consensus_queue.cc:1059] T 4f9eb06ed01f4ce68be3af8a8434665b P 186b12638a87410487467df721a3cbda [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.010s
I20260709 13:04:13.666702 12073 raft_consensus.cc:2749] T b4cd3d7dfbbf4096908ff11e9dc57141 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:04:13.667582 12073 consensus_queue.cc:1059] T b67eff0634e540eab91cf8cee795969f P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:04:13.667909 12048 raft_consensus.cc:3055] T 4f9eb06ed01f4ce68be3af8a8434665b P 186b12638a87410487467df721a3cbda [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:04:13.668394 12048 raft_consensus.cc:740] T 4f9eb06ed01f4ce68be3af8a8434665b P 186b12638a87410487467df721a3cbda [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 186b12638a87410487467df721a3cbda, State: Running, Role: LEADER
I20260709 13:04:13.668964 12073 raft_consensus.cc:3055] T b67eff0634e540eab91cf8cee795969f P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:04:13.669385 12073 raft_consensus.cc:740] T b67eff0634e540eab91cf8cee795969f P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 10ca5ee68ad64b3283b9d235352c9d23, State: Running, Role: LEADER
W20260709 13:04:13.571589 10992 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:13.571914 10877 raft_consensus.cc:2468] T f34f2bc02d85444f9096e0f5b4fe4331 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 1.
I20260709 13:04:13.670356 12073 consensus_queue.cc:260] T b67eff0634e540eab91cf8cee795969f P 10ca5ee68ad64b3283b9d235352c9d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:13.671736 12073 raft_consensus.cc:3060] T b67eff0634e540eab91cf8cee795969f P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:04:13.672891 10992 leader_election.cc:304] T ff13ae9a47c7455f962406c6d131a3ff P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:04:13.673903 12202 consensus_queue.cc:1048] T 34064dc7bf3747dbb3876dd824ee0688 P 873357742d5843579556ee8166f23f47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:04:13.674604 12044 consensus_queue.cc:1048] T d71ca3a1ba6b48478aaf62741bf3cf6a P 873357742d5843579556ee8166f23f47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:04:13.669412 12048 consensus_queue.cc:260] T 4f9eb06ed01f4ce68be3af8a8434665b P 186b12638a87410487467df721a3cbda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:13.675645 12202 raft_consensus.cc:2804] T ff13ae9a47c7455f962406c6d131a3ff P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:04:13.676105 12202 raft_consensus.cc:493] T ff13ae9a47c7455f962406c6d131a3ff P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:13.613811 12052 consensus_queue.cc:237] T 63d44f6ba6434ab2870fe3190f4d7c2f P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:13.676479 12202 raft_consensus.cc:3060] T ff13ae9a47c7455f962406c6d131a3ff P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:04:13.676867 12044 consensus_queue.cc:1048] T 11de93cd9b9d457489ff1cb80f976c9a P 873357742d5843579556ee8166f23f47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20260709 13:04:13.573333 11066 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:13.679363 11066 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:13.679714 10872 consensus_meta.cc:287] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 186b12638a87410487467df721a3cbda: Time spent flushing consensus metadata: real 0.571s	user 0.012s	sys 0.004s
W20260709 13:04:13.582813 10993 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:13.680064 12103 raft_consensus.cc:493] T 7709387f4d3d4f9992d97a0fbca39a40 P 873357742d5843579556ee8166f23f47 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 0172fed22b6b4890a7fdc953fa128af4)
I20260709 13:04:13.680614 12103 raft_consensus.cc:3060] T 7709387f4d3d4f9992d97a0fbca39a40 P 873357742d5843579556ee8166f23f47 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:04:13.579597 11937 raft_consensus.cc:2749] T b67eff0634e540eab91cf8cee795969f P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 13:04:13.580329 11797 raft_consensus.cc:493] T acc241e7273d49149984968596242712 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:13.682070 11797 raft_consensus.cc:515] T acc241e7273d49149984968596242712 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:13.684626 11797 leader_election.cc:290] T acc241e7273d49149984968596242712 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:13.587523 10775 catalog_manager.cc:5697] T 11de93cd9b9d457489ff1cb80f976c9a P 10ca5ee68ad64b3283b9d235352c9d23 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "873357742d5843579556ee8166f23f47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } }
I20260709 13:04:13.543054 10871 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a1716c0adf34bee8a4e99341ba5788f" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:13.609063 11926 raft_consensus.cc:2749] T bb7d67be500947b3aef4c4245e33671c P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 13:04:13.609579 10949 raft_consensus.cc:1275] T 7839ca2d2c13431a85f44d7c458c6351 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Refusing update from remote peer 873357742d5843579556ee8166f23f47: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 13:04:13.555498 12158 raft_consensus.cc:3055] T 5de5f0581e3b41dea729363d6ca563bb P 0172fed22b6b4890a7fdc953fa128af4 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:04:13.496987 11096 rpcz_store.cc:276] 0709 13:04:11.929691 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:11.933482 (+  3791us) service_pool.cc:224] Handling call
0709 13:04:12.817505 (+884023us) spinlock_profiling.cc:243] Waited 4.3 ms on lock 0x7b5800117698. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f426303db71 00007f426303da22 00007f426bf56c09 00007f426b0b0ee0 00007f426b17a0ab 00007f426b17a131 00007f426b13bfc0 00007f426b126929 00007f426b1263d2 00007f426469dbd4
0709 13:04:12.817814 (+   309us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:12.826791 (+  8977us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:12.826836 (+    45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:12.827567 (+   731us) log.cc:844] Serialized 26 byte log entry
0709 13:04:12.981265 (+153698us) raft_consensus.cc:1645] Marking committed up to 0
0709 13:04:13.012127 (+ 30862us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:13.022772 (+ 10645us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:13.380128 (+357356us) raft_consensus.cc:1685] finished
0709 13:04:13.380158 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:13.415223 (+ 35065us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8304,"raft.queue_time_us":9946,"raft.run_cpu_time_us":147,"raft.run_wall_time_us":148,"spinlock_wait_cycles":9888512,"thread_start_us":123016,"threads_started":1,"wal-append.queue_time_us":261408}
I20260709 13:04:13.620038 11067 leader_election.cc:304] T 75492f86fde046c3900df3eac8292fd4 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:04:13.555428 12098 leader_election.cc:290] T ff13ae9a47c7455f962406c6d131a3ff P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:13.613729 11852 raft_consensus.cc:493] T d371c5e27bd843469d2e2e453734cb4a P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:13.634407 10843 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 75524.6us
I20260709 13:04:13.636667 11779 leader_election.cc:290] T 32147c6d2d5a477aa38c597c7f78df5b P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 3 election: Requested vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:13.637248 12193 leader_election.cc:290] T d7ce47b9a92d43dd88d95bcda5038597 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:13.439437 11098 rpcz_store.cc:276] 0709 13:04:12.031556 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:13.153474 (+1121918us) service_pool.cc:224] Handling call
0709 13:04:13.172747 (+ 19273us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:13.173050 (+   303us) raft_consensus.cc:1534] Early marking committed up to index 3
0709 13:04:13.173431 (+   381us) log.cc:844] Serialized 14 byte log entry
0709 13:04:13.293463 (+120032us) log.cc:844] Serialized 14 byte log entry
0709 13:04:13.312331 (+ 18868us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:13.317661 (+  5330us) raft_consensus.cc:1645] Marking committed up to 3
0709 13:04:13.321354 (+  3693us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:13.336429 (+ 15075us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:13.350233 (+ 13804us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":25411,"prepare.run_cpu_time_us":59,"prepare.run_wall_time_us":60,"raft.queue_time_us":61584,"raft.run_cpu_time_us":411,"raft.run_wall_time_us":57560,"spinlock_wait_cycles":3964800,"thread_start_us":70810,"threads_started":1,"wal-append.queue_time_us":127565}
I20260709 13:04:13.639063 12198 raft_consensus.cc:493] T e5ba72ccd854430baadbc636dfd17136 P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:13.641004 11020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34064dc7bf3747dbb3876dd824ee0688" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:13.642073 11025 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555217d45d9b4ee492dbb3ddaf0f7c02" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:13.643083 11023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f371d2565dd84667b5613848604949bb" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:13.644131 11022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e89e640169e14e2d98f968f76027daa7" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:13.648455 10880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57b6a2418c1342e7924195e9c1dbd357" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:13.635028 12050 raft_consensus.cc:2804] T d7ce47b9a92d43dd88d95bcda5038597 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:04:13.655050 10921 leader_election.cc:304] T 5de5f0581e3b41dea729363d6ca563bb P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda; no voters: 0172fed22b6b4890a7fdc953fa128af4
I20260709 13:04:13.566278 12004 raft_consensus.cc:493] T a610d1a71f52432e88fe56130f921fa2 P 0172fed22b6b4890a7fdc953fa128af4 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:13.658124 11027 raft_consensus.cc:1275] T 1e4f2241a94a41f38e481137f868b83d P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Refusing update from remote peer 186b12638a87410487467df721a3cbda: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20260709 13:04:13.658150 10919 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:45807, user_credentials={real_user=slave}} blocked reactor thread for 112203us
I20260709 13:04:13.702380 11096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32147c6d2d5a477aa38c597c7f78df5b" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:04:13.689774 12158 raft_consensus.cc:740] T 5de5f0581e3b41dea729363d6ca563bb P 0172fed22b6b4890a7fdc953fa128af4 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 0172fed22b6b4890a7fdc953fa128af4, State: Running, Role: LEADER
I20260709 13:04:13.661067 10954 raft_consensus.cc:1240] T 32147c6d2d5a477aa38c597c7f78df5b P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Rejecting Update request from peer 873357742d5843579556ee8166f23f47 for earlier term 2. Current term is 3. Ops: []
I20260709 13:04:13.661347 12071 raft_consensus.cc:2749] T e96e80ba9ac34d0998395515e5c8c41a P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:04:13.621949 10875 raft_consensus.cc:2468] T c31881534c5e4f0089c82057c1746652 P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 3.
I20260709 13:04:13.711226 10871 raft_consensus.cc:1240] T a1ec2b50c6ad4f7c8033476208abb26d P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Rejecting Update request from peer 0172fed22b6b4890a7fdc953fa128af4 for earlier term 1. Current term is 2. Ops: [1.1-1.1]
I20260709 13:04:13.712258 12019 leader_election.cc:290] T d71ca3a1ba6b48478aaf62741bf3cf6a P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:13.712921 11639 raft_consensus.cc:2804] T 5de5f0581e3b41dea729363d6ca563bb P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:04:13.713526 11639 raft_consensus.cc:697] T 5de5f0581e3b41dea729363d6ca563bb P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 LEADER]: Becoming Leader. State: Replica: 10ca5ee68ad64b3283b9d235352c9d23, State: Running, Role: LEADER
I20260709 13:04:13.632272 10948 rpcz_store.cc:276] 0709 13:04:11.723604 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:13.555857 (+1832253us) service_pool.cc:224] Handling call
0709 13:04:13.556332 (+   475us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:13.556600 (+   268us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:13.556640 (+    40us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:13.556708 (+    68us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:04:13.556748 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:13.556818 (+    70us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:13.556973 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:13.662231 10950 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49424 (request call id 212) took 1144 ms. Trace:
I20260709 13:04:13.714864 11023 raft_consensus.cc:1275] T 57b6a2418c1342e7924195e9c1dbd357 P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Refusing update from remote peer 186b12638a87410487467df721a3cbda: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 13:04:13.714504 11639 consensus_queue.cc:237] T 5de5f0581e3b41dea729363d6ca563bb P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:13.714978 10950 rpcz_store.cc:276] 0709 13:04:12.517901 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:13.167379 (+649478us) service_pool.cc:224] Handling call
0709 13:04:13.174732 (+  7353us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:13.658309 (+483577us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:13.662149 (+  3840us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:13.717892 11025 raft_consensus.cc:2393] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 186b12638a87410487467df721a3cbda in current term 4: Already voted for candidate 873357742d5843579556ee8166f23f47 in this term.
I20260709 13:04:13.718340 12158 consensus_queue.cc:260] T 5de5f0581e3b41dea729363d6ca563bb P 0172fed22b6b4890a7fdc953fa128af4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:13.719897 12158 raft_consensus.cc:3060] T 5de5f0581e3b41dea729363d6ca563bb P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:04:13.720466 10880 raft_consensus.cc:1240] T 5de5f0581e3b41dea729363d6ca563bb P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Rejecting Update request from peer 0172fed22b6b4890a7fdc953fa128af4 for earlier term 3. Current term is 4. Ops: []
I20260709 13:04:13.708501 11099 raft_consensus.cc:1240] T 72b111ad7b5f4753ab1d2b5ee1114840 P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Rejecting Update request from peer 10ca5ee68ad64b3283b9d235352c9d23 for earlier term 3. Current term is 4. Ops: []
I20260709 13:04:13.722015 10880 raft_consensus.cc:1275] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Refusing update from remote peer 0172fed22b6b4890a7fdc953fa128af4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 3 index: 1. (index mismatch)
I20260709 13:04:13.708735 12004 raft_consensus.cc:515] T a610d1a71f52432e88fe56130f921fa2 P 0172fed22b6b4890a7fdc953fa128af4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:13.702601 11100 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c31881534c5e4f0089c82057c1746652" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:13.723009 11100 raft_consensus.cc:2468] T c31881534c5e4f0089c82057c1746652 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 2.
I20260709 13:04:13.702798 12198 raft_consensus.cc:515] T e5ba72ccd854430baadbc636dfd17136 P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:13.724251 12004 leader_election.cc:290] T a610d1a71f52432e88fe56130f921fa2 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:13.724931 10878 raft_consensus.cc:1275] T 6a9353d1d230416296627d7cf8b1d425 P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Refusing update from remote peer 0172fed22b6b4890a7fdc953fa128af4: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 13:04:13.704525 11027 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f1466124dbd4f6ebb3c4d62ca48bf96" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:13.706903 12050 raft_consensus.cc:697] T d7ce47b9a92d43dd88d95bcda5038597 P 186b12638a87410487467df721a3cbda [term 2 LEADER]: Becoming Leader. State: Replica: 186b12638a87410487467df721a3cbda, State: Running, Role: LEADER
I20260709 13:04:13.728863 12050 consensus_queue.cc:237] T d7ce47b9a92d43dd88d95bcda5038597 P 186b12638a87410487467df721a3cbda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:13.662509 10995 leader_election.cc:304] T 868e92339c2f4508af0afbd07c814d64 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47; no voters: 0172fed22b6b4890a7fdc953fa128af4
I20260709 13:04:13.662876 10873 raft_consensus.cc:2468] T a5299b8d044f41b7b68057e63d17b159 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 2.
I20260709 13:04:13.662806 11879 consensus_queue.cc:1048] T f371d2565dd84667b5613848604949bb P 873357742d5843579556ee8166f23f47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:04:13.663558 11974 raft_consensus.cc:2804] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:04:13.661115 12105 raft_consensus.cc:2804] T d270b9e5481245d9b5340a454a7cd10b P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:04:13.742601 12105 raft_consensus.cc:493] T d270b9e5481245d9b5340a454a7cd10b P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:13.743003 12105 raft_consensus.cc:3060] T d270b9e5481245d9b5340a454a7cd10b P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:04:13.663798 12049 consensus_queue.cc:260] T e5ba72ccd854430baadbc636dfd17136 P 10ca5ee68ad64b3283b9d235352c9d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:13.666661 11069 leader_election.cc:304] T 836251ebc2b74f5da88b8a8b2338e2a5 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda; no voters: 
W20260709 13:04:13.570870 10845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 107130us
W20260709 13:04:13.746125 11069 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:38967, user_credentials={real_user=slave}} blocked reactor thread for 79545.5us
I20260709 13:04:13.749521 11069 leader_election.cc:304] T 72b111ad7b5f4753ab1d2b5ee1114840 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda; no voters: 10ca5ee68ad64b3283b9d235352c9d23
I20260709 13:04:13.676237 12048 raft_consensus.cc:3060] T 4f9eb06ed01f4ce68be3af8a8434665b P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:04:13.679101 10947 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f34f2bc02d85444f9096e0f5b4fe4331" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:13.684523 11795 raft_consensus.cc:3060] T 8c160ddfe9b34bdaa3835d0915c1e06f P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:04:13.759712 11953 raft_consensus.cc:515] T a5299b8d044f41b7b68057e63d17b159 P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:13.761162 11028 raft_consensus.cc:1240] T a5299b8d044f41b7b68057e63d17b159 P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Rejecting Update request from peer 186b12638a87410487467df721a3cbda for earlier term 1. Current term is 3. Ops: []
I20260709 13:04:13.599848 11024 raft_consensus.cc:1275] T a977842ba4b744f4bb2febd55d286006 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Refusing update from remote peer 0172fed22b6b4890a7fdc953fa128af4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 1. (term mismatch)
I20260709 13:04:13.753808 11972 raft_consensus.cc:493] T 63d44f6ba6434ab2870fe3190f4d7c2f P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:13.763854 11024 pending_rounds.cc:85] T a977842ba4b744f4bb2febd55d286006 P 873357742d5843579556ee8166f23f47: Aborting all ops after (but not including) 0
I20260709 13:04:13.756322 12019 raft_consensus.cc:2804] T 836251ebc2b74f5da88b8a8b2338e2a5 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:04:13.764305 11024 pending_rounds.cc:107] T a977842ba4b744f4bb2febd55d286006 P 873357742d5843579556ee8166f23f47: Aborting uncommitted NO_OP operation due to leader change: 1.1
I20260709 13:04:13.764480 12019 raft_consensus.cc:493] T 836251ebc2b74f5da88b8a8b2338e2a5 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 10ca5ee68ad64b3283b9d235352c9d23)
I20260709 13:04:13.764665 11024 raft_consensus.cc:2889] T a977842ba4b744f4bb2febd55d286006 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:04:13.756134 12004 raft_consensus.cc:2804] T 72b111ad7b5f4753ab1d2b5ee1114840 P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:04:13.622149 12200 raft_consensus.cc:2749] T 32147c6d2d5a477aa38c597c7f78df5b P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:04:13.765523 12004 raft_consensus.cc:697] T 72b111ad7b5f4753ab1d2b5ee1114840 P 0172fed22b6b4890a7fdc953fa128af4 [term 4 LEADER]: Becoming Leader. State: Replica: 0172fed22b6b4890a7fdc953fa128af4, State: Running, Role: LEADER
I20260709 13:04:13.572010 11103 rpcz_store.cc:276] 0709 13:04:12.062318 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:13.219009 (+1156691us) service_pool.cc:224] Handling call
0709 13:04:13.235005 (+ 15996us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:13.295960 (+ 60955us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:13.496646 (+200686us) spinlock_profiling.cc:243] Waited 56.7 ms on lock 0x7b5400800cd4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:04:13.496966 (+   320us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":130513280}
I20260709 13:04:13.685911 10775 catalog_manager.cc:5697] T c51782cd9d064ec0949e84e24dd4de67 P 10ca5ee68ad64b3283b9d235352c9d23 reported cstate change: term changed from 1 to 2, leader changed from 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194) to 186b12638a87410487467df721a3cbda (127.0.107.193). New cstate: current_term: 2 leader_uuid: "186b12638a87410487467df721a3cbda" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } }
I20260709 13:04:13.766651 12004 consensus_queue.cc:237] T 72b111ad7b5f4753ab1d2b5ee1114840 P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:13.768280 11972 raft_consensus.cc:515] T 63d44f6ba6434ab2870fe3190f4d7c2f P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:13.643946 11653 leader_election.cc:290] T 525bfdbd6d934afd823f7d3ee3f9d052 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 4 election: Requested vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
W20260709 13:04:13.690315 11067 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:45807, user_credentials={real_user=slave}} blocked reactor thread for 70363us
I20260709 13:04:13.763600 11953 leader_election.cc:290] T a5299b8d044f41b7b68057e63d17b159 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 3 election: Requested vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
W20260709 13:04:13.772950 11067 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:13.755856 12098 raft_consensus.cc:2804] T 868e92339c2f4508af0afbd07c814d64 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:04:13.774657 12098 raft_consensus.cc:493] T 868e92339c2f4508af0afbd07c814d64 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 0172fed22b6b4890a7fdc953fa128af4)
I20260709 13:04:13.775071 12098 raft_consensus.cc:3060] T 868e92339c2f4508af0afbd07c814d64 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:04:13.577225 11029 raft_consensus.cc:2468] T 0e5d327630e5438b878035d28efe6c0c P 873357742d5843579556ee8166f23f47 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 5.
I20260709 13:04:13.690342 10955 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49434 (request call id 241) took 1967 ms. Trace:
I20260709 13:04:13.778968 10955 rpcz_store.cc:276] 0709 13:04:11.723340 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:12.342854 (+619514us) service_pool.cc:224] Handling call
0709 13:04:12.823436 (+480582us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:12.823734 (+   298us) raft_consensus.cc:1534] Early marking committed up to index 3
0709 13:04:12.824125 (+   391us) log.cc:844] Serialized 14 byte log entry
0709 13:04:13.009799 (+185674us) log.cc:844] Serialized 14 byte log entry
0709 13:04:13.023839 (+ 14040us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:13.023900 (+    61us) raft_consensus.cc:1645] Marking committed up to 3
0709 13:04:13.023947 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:13.024021 (+    74us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:13.275615 (+251594us) spinlock_profiling.cc:243] Waited 2.65 ms on lock 0x7b1800199d20. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f42630753f3 00007f42630777c1 00007f4263077779 00007f4263077709 00007f42630776d1 00007f42630769cd 000000000038f354 000000000038f199 00007f426305883a
0709 13:04:13.690143 (+414528us) spinlock_profiling.cc:243] Waited 480 ms on lock 0x7b54008a5f50. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661618ad 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:04:13.690267 (+   124us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":2769,"prepare.run_cpu_time_us":92,"prepare.run_wall_time_us":94,"raft.queue_time_us":5196,"raft.run_cpu_time_us":242,"raft.run_wall_time_us":7755,"spinlock_wait_cycles":1109288448,"thread_start_us":184879,"threads_started":1,"wal-append.queue_time_us":297123}
I20260709 13:04:13.689152 10877 raft_consensus.cc:1240] T 7757ecbf402e47bfbc93d0447741e26c P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Rejecting Update request from peer 0172fed22b6b4890a7fdc953fa128af4 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260709 13:04:13.694859 10843 leader_election.cc:304] T b67eff0634e540eab91cf8cee795969f P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 186b12638a87410487467df721a3cbda; no voters: 0172fed22b6b4890a7fdc953fa128af4, 10ca5ee68ad64b3283b9d235352c9d23
I20260709 13:04:13.675529 11021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "868e92339c2f4508af0afbd07c814d64" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:13.692360 12195 raft_consensus.cc:3060] T bb7d67be500947b3aef4c4245e33671c P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:04:13.690330 12203 raft_consensus.cc:697] T a1ec2b50c6ad4f7c8033476208abb26d P 873357742d5843579556ee8166f23f47 [term 2 LEADER]: Becoming Leader. State: Replica: 873357742d5843579556ee8166f23f47, State: Running, Role: LEADER
I20260709 13:04:13.786814 12193 consensus_queue.cc:1048] T 6a9353d1d230416296627d7cf8b1d425 P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:04:13.787369 12203 consensus_queue.cc:237] T a1ec2b50c6ad4f7c8033476208abb26d P 873357742d5843579556ee8166f23f47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:13.789598 10877 raft_consensus.cc:1275] T 11de93cd9b9d457489ff1cb80f976c9a P 186b12638a87410487467df721a3cbda [term 5 FOLLOWER]: Refusing update from remote peer 873357742d5843579556ee8166f23f47: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260709 13:04:13.764988 12019 raft_consensus.cc:3060] T 836251ebc2b74f5da88b8a8b2338e2a5 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:04:13.790786 12202 raft_consensus.cc:515] T ff13ae9a47c7455f962406c6d131a3ff P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:13.680114 10872 raft_consensus.cc:2468] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 4.
I20260709 13:04:13.796537 11027 raft_consensus.cc:2468] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 873357742d5843579556ee8166f23f47 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 5.
I20260709 13:04:13.704397 10919 leader_election.cc:304] T 1e4f2241a94a41f38e481137f868b83d P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23; no voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47
I20260709 13:04:13.659584 10879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65bf0a62f4d84ef18dd6b23e082724b8" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
W20260709 13:04:13.705742 10921 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:38967, user_credentials={real_user=slave}} blocked reactor thread for 50811.7us
W20260709 13:04:13.797891 10919 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:45807, user_credentials={real_user=slave}} blocked reactor thread for 93629.8us
I20260709 13:04:13.694087 11937 raft_consensus.cc:2804] T 75492f86fde046c3900df3eac8292fd4 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:04:13.732535 10876 raft_consensus.cc:2468] T 0e5d327630e5438b878035d28efe6c0c P 186b12638a87410487467df721a3cbda [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 5.
W20260709 13:04:13.739214 10995 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:38967, user_credentials={real_user=slave}} blocked reactor thread for 76849.8us
I20260709 13:04:13.740518 10880 raft_consensus.cc:1240] T b67eff0634e540eab91cf8cee795969f P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Rejecting Update request from peer 10ca5ee68ad64b3283b9d235352c9d23 for earlier term 1. Current term is 2. Ops: []
I20260709 13:04:13.698426 11852 raft_consensus.cc:515] T d371c5e27bd843469d2e2e453734cb4a P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
W20260709 13:04:13.742154 11066 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 52490.4us
I20260709 13:04:13.804816 11066 leader_election.cc:304] T 5d439326f23747eaab3c1e7da2d6b785 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 10ca5ee68ad64b3283b9d235352c9d23; no voters: 186b12638a87410487467df721a3cbda
I20260709 13:04:13.751968 11069 leader_election.cc:304] T f34f2bc02d85444f9096e0f5b4fe4331 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda; no voters: 
I20260709 13:04:13.726289 12198 leader_election.cc:290] T e5ba72ccd854430baadbc636dfd17136 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
W20260709 13:04:13.783192 10843 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 88457.3us
W20260709 13:04:13.813091 11069 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:38967, user_credentials={real_user=slave}} blocked reactor thread for 61240.9us
I20260709 13:04:13.793413 10877 raft_consensus.cc:1275] T 34064dc7bf3747dbb3876dd824ee0688 P 186b12638a87410487467df721a3cbda [term 5 FOLLOWER]: Refusing update from remote peer 873357742d5843579556ee8166f23f47: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260709 13:04:13.808282 10953 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5299b8d044f41b7b68057e63d17b159" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:04:13.814920 10953 raft_consensus.cc:3060] T a5299b8d044f41b7b68057e63d17b159 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:04:13.816001 10877 raft_consensus.cc:1275] T 7839ca2d2c13431a85f44d7c458c6351 P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Refusing update from remote peer 873357742d5843579556ee8166f23f47: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 13:04:13.814194 11069 leader_election.cc:304] T 9a1716c0adf34bee8a4e99341ba5788f P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4; no voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47
W20260709 13:04:13.809993 10826 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 13:04:13.813658 10843 leader_election.cc:304] T 34064dc7bf3747dbb3876dd824ee0688 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda; no voters: 
I20260709 13:04:13.824810 10877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63d44f6ba6434ab2870fe3190f4d7c2f" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:13.795845 12004 raft_consensus.cc:493] T b4cd3d7dfbbf4096908ff11e9dc57141 P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:13.832746 10995 leader_election.cc:304] T 57b6a2418c1342e7924195e9c1dbd357 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 873357742d5843579556ee8166f23f47; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda
I20260709 13:04:13.829176 10877 raft_consensus.cc:2468] T 63d44f6ba6434ab2870fe3190f4d7c2f P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 3.
I20260709 13:04:13.797219 12068 raft_consensus.cc:2764] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20260709 13:04:13.803992 12088 raft_consensus.cc:2749] T b67eff0634e540eab91cf8cee795969f P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 13:04:13.812675 12009 raft_consensus.cc:2749] T 1e4f2241a94a41f38e481137f868b83d P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:04:13.805482 11937 raft_consensus.cc:493] T 75492f86fde046c3900df3eac8292fd4 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:13.855891 10880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5299b8d044f41b7b68057e63d17b159" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:04:13.858351 11066 leader_election.cc:304] T e5ba72ccd854430baadbc636dfd17136 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47
I20260709 13:04:13.860205 11066 leader_election.cc:304] T d270b9e5481245d9b5340a454a7cd10b P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda
I20260709 13:04:13.876762 10953 raft_consensus.cc:2468] T a5299b8d044f41b7b68057e63d17b159 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 3.
I20260709 13:04:13.827554 12044 raft_consensus.cc:493] T 6a9353d1d230416296627d7cf8b1d425 P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:13.878988 10920 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:13.819274 12158 raft_consensus.cc:2804] T 5d439326f23747eaab3c1e7da2d6b785 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:04:13.828969 11926 raft_consensus.cc:2749] T 9a1716c0adf34bee8a4e99341ba5788f P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260709 13:04:13.891907 10921 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:38967, user_credentials={real_user=slave}} blocked reactor thread for 82998.6us
I20260709 13:04:13.813766 11653 consensus_queue.cc:1048] T 7839ca2d2c13431a85f44d7c458c6351 P 873357742d5843579556ee8166f23f47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.066s
I20260709 13:04:13.880555 10995 leader_election.cc:304] T 65bf0a62f4d84ef18dd6b23e082724b8 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 873357742d5843579556ee8166f23f47; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda
I20260709 13:04:13.893006 10921 leader_election.cc:304] T c31881534c5e4f0089c82057c1746652 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda; no voters: 
I20260709 13:04:13.854935 12095 raft_consensus.cc:2804] T 34064dc7bf3747dbb3876dd824ee0688 P 186b12638a87410487467df721a3cbda [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:04:13.849231 11972 leader_election.cc:290] T 63d44f6ba6434ab2870fe3190f4d7c2f P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:13.899595 10995 leader_election.cc:304] T 63d44f6ba6434ab2870fe3190f4d7c2f P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:04:13.847661 10952 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5ba72ccd854430baadbc636dfd17136" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:13.830099 11797 consensus_queue.cc:1048] T 1e4f2241a94a41f38e481137f868b83d P 186b12638a87410487467df721a3cbda [LEADER]: Connected to new peer: Peer: permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.060s
I20260709 13:04:13.823697 12193 raft_consensus.cc:2804] T f34f2bc02d85444f9096e0f5b4fe4331 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:04:13.903576 10953 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63d44f6ba6434ab2870fe3190f4d7c2f" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:13.867594 11937 raft_consensus.cc:3060] T 75492f86fde046c3900df3eac8292fd4 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:04:13.828279 12064 consensus_queue.cc:1048] T a977842ba4b744f4bb2febd55d286006 P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.062s
W20260709 13:04:13.901266 11069 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:38967, user_credentials={real_user=slave}} blocked reactor thread for 63840.1us
I20260709 13:04:13.891939 12044 raft_consensus.cc:515] T 6a9353d1d230416296627d7cf8b1d425 P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:13.836460 12171 consensus_queue.cc:1048] T 57b6a2418c1342e7924195e9c1dbd357 P 186b12638a87410487467df721a3cbda [LEADER]: Connected to new peer: Peer: permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.064s
I20260709 13:04:13.884519 12198 raft_consensus.cc:2749] T 57b6a2418c1342e7924195e9c1dbd357 P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:04:13.862886 11982 consensus_queue.cc:1048] T 6a9353d1d230416296627d7cf8b1d425 P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.052s
I20260709 13:04:13.922544 10992 leader_election.cc:304] T a5299b8d044f41b7b68057e63d17b159 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:04:13.925017 10845 leader_election.cc:304] T f371d2565dd84667b5613848604949bb P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda; no voters: 873357742d5843579556ee8166f23f47
I20260709 13:04:13.879354 11943 consensus_queue.cc:1048] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.046s
I20260709 13:04:13.893556 12158 raft_consensus.cc:493] T 5d439326f23747eaab3c1e7da2d6b785 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 10ca5ee68ad64b3283b9d235352c9d23)
I20260709 13:04:13.855289 12004 raft_consensus.cc:515] T b4cd3d7dfbbf4096908ff11e9dc57141 P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:13.912091 11673 raft_consensus.cc:2749] T 65bf0a62f4d84ef18dd6b23e082724b8 P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260709 13:04:13.929423 11069 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:13.899323 12095 raft_consensus.cc:493] T 34064dc7bf3747dbb3876dd824ee0688 P 186b12638a87410487467df721a3cbda [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:13.912540 10875 raft_consensus.cc:1240] T 0f9df2bc281f4866b6da5da94a0defc7 P 186b12638a87410487467df721a3cbda [term 2 LEADER]: Rejecting Update request from peer 0172fed22b6b4890a7fdc953fa128af4 for earlier term 1. Current term is 2. Ops: []
I20260709 13:04:13.889540 12070 consensus_queue.cc:1048] T 11de93cd9b9d457489ff1cb80f976c9a P 873357742d5843579556ee8166f23f47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.038s
I20260709 13:04:13.911973 11020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a610d1a71f52432e88fe56130f921fa2" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
W20260709 13:04:13.944152 12206 log.cc:927] Time spent T a610d1a71f52432e88fe56130f921fa2 P 873357742d5843579556ee8166f23f47: Append to log took a long time: real 0.205s	user 0.000s	sys 0.001s
I20260709 13:04:13.919214 12193 raft_consensus.cc:493] T f34f2bc02d85444f9096e0f5b4fe4331 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:04:13.968207 10843 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 123274us
I20260709 13:04:13.939656 12215 raft_consensus.cc:493] T c51782cd9d064ec0949e84e24dd4de67 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:13.968966 12095 raft_consensus.cc:3060] T 34064dc7bf3747dbb3876dd824ee0688 P 186b12638a87410487467df721a3cbda [term 5 FOLLOWER]: Advancing to term 6
*** SIGABRT (@0x3e8000001af) received by PID 431 (TID 0x7f425eca8840) from PID 431; stack trace: ***
I20260709 13:04:13.977416 11103 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acc241e7273d49149984968596242712" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:13.978263 12009 raft_consensus.cc:2804] T c31881534c5e4f0089c82057c1746652 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:04:13.978390 11100 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff13ae9a47c7455f962406c6d131a3ff" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:13.978554 11097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff13ae9a47c7455f962406c6d131a3ff" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:13.979156 11096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5ba72ccd854430baadbc636dfd17136" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:13.979709 11099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "525bfdbd6d934afd823f7d3ee3f9d052" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
    @           0x2d8530 __tsan::CallUserSignalHandler() at /apache/dev/git/kudu/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1915
I20260709 13:04:14.006122 12198 raft_consensus.cc:2804] T a5299b8d044f41b7b68057e63d17b159 P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:04:13.954282 12158 raft_consensus.cc:3060] T 5d439326f23747eaab3c1e7da2d6b785 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Advancing to term 4
W20260709 13:04:13.964097 10921 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:14.102352 10921 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:13.940025 11879 raft_consensus.cc:2804] T 63d44f6ba6434ab2870fe3190f4d7c2f P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:04:13.922572 12053 consensus_queue.cc:1048] T 34064dc7bf3747dbb3876dd824ee0688 P 873357742d5843579556ee8166f23f47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.057s
    @           0x2daa13 rtl_sigaction() at /apache/dev/git/kudu/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:2005
W20260709 13:04:14.012727 10843 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:14.026541 12050 consensus_queue.cc:1048] T 1e4f2241a94a41f38e481137f868b83d P 186b12638a87410487467df721a3cbda [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:04:14.028420 11020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32147c6d2d5a477aa38c597c7f78df5b" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
I20260709 13:04:14.029402 11025 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acc241e7273d49149984968596242712" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:14.032297 10871 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a9353d1d230416296627d7cf8b1d425" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:14.034881 12066 raft_consensus.cc:493] T 8c160ddfe9b34bdaa3835d0915c1e06f P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:14.038244 12049 raft_consensus.cc:3060] T e5ba72ccd854430baadbc636dfd17136 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:04:13.942620 12220 raft_consensus.cc:3060] T e5ba72ccd854430baadbc636dfd17136 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:04:13.966156 12091 raft_consensus.cc:2804] T f371d2565dd84667b5613848604949bb P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260709 13:04:14.044004 11066 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 183939us
W20260709 13:04:13.984581 11068 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:13.944674 11953 consensus_queue.cc:1048] T 7839ca2d2c13431a85f44d7c458c6351 P 873357742d5843579556ee8166f23f47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.074s
I20260709 13:04:13.981550 12044 leader_election.cc:290] T 6a9353d1d230416296627d7cf8b1d425 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:14.016323 12193 raft_consensus.cc:3060] T f34f2bc02d85444f9096e0f5b4fe4331 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:04:14.028514 11974 raft_consensus.cc:493] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:13.983274 12219 raft_consensus.cc:2749] T d270b9e5481245d9b5340a454a7cd10b P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:04:14.038930 12218 consensus_queue.cc:1048] T f34f2bc02d85444f9096e0f5b4fe4331 P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.056s
I20260709 13:04:14.128443 11067 leader_election.cc:304] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47; no voters: 
W20260709 13:04:14.125025 10994 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 185925us
I20260709 13:04:14.128981 11027 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d371c5e27bd843469d2e2e453734cb4a" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:14.312722 11027 raft_consensus.cc:2468] T d371c5e27bd843469d2e2e453734cb4a P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 3.
I20260709 13:04:14.057183 12009 raft_consensus.cc:493] T c31881534c5e4f0089c82057c1746652 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:13.945816 11926 consensus_queue.cc:1048] T a977842ba4b744f4bb2febd55d286006 P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.191s
I20260709 13:04:14.119807 12201 raft_consensus.cc:493] T 1e4f2241a94a41f38e481137f868b83d P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:14.119180 11879 raft_consensus.cc:493] T 63d44f6ba6434ab2870fe3190f4d7c2f P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:14.100893 11100 raft_consensus.cc:2468] T ff13ae9a47c7455f962406c6d131a3ff P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 3.
I20260709 13:04:14.134065 12171 raft_consensus.cc:493] T 87cd8f5097624c8692f1d83544d8881e P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:14.322916 12233 consensus_queue.cc:1048] T f371d2565dd84667b5613848604949bb P 873357742d5843579556ee8166f23f47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.004s
I20260709 13:04:14.326360 12171 raft_consensus.cc:515] T 87cd8f5097624c8692f1d83544d8881e P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:14.329151 12171 leader_election.cc:290] T 87cd8f5097624c8692f1d83544d8881e P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:14.283051 11020 raft_consensus.cc:1240] T 525bfdbd6d934afd823f7d3ee3f9d052 P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Rejecting Update request from peer 10ca5ee68ad64b3283b9d235352c9d23 for earlier term 3. Current term is 4. Ops: []
I20260709 13:04:14.329493 11026 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46628 (request call id 213) took 1395 ms. Trace:
I20260709 13:04:14.330261 11026 rpcz_store.cc:276] 0709 13:04:12.934124 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:13.042876 (+108752us) service_pool.cc:224] Handling call
0709 13:04:13.113664 (+ 70788us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:13.537151 (+423487us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:14.329024 (+791873us) spinlock_profiling.cc:243] Waited 371 ms on lock 0x7b54006c25d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:04:14.329389 (+   365us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":854202112}
I20260709 13:04:14.323770 12224 raft_consensus.cc:2804] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260709 13:04:14.284123 12212 log.cc:927] Time spent T 30f628e3d7914d609e7f1d980a74c654 P 186b12638a87410487467df721a3cbda: Append to log took a long time: real 0.277s	user 0.000s	sys 0.002s
I20260709 13:04:14.141767 12066 raft_consensus.cc:515] T 8c160ddfe9b34bdaa3835d0915c1e06f P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:14.336265 12224 raft_consensus.cc:493] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:14.336735 12224 raft_consensus.cc:3060] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:04:14.142572 10871 raft_consensus.cc:2468] T 6a9353d1d230416296627d7cf8b1d425 P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 4.
I20260709 13:04:14.141113 12091 raft_consensus.cc:493] T f371d2565dd84667b5613848604949bb P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:14.149789 10874 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51696 (request call id 261) took 1093 ms. Trace:
I20260709 13:04:14.136776 12198 raft_consensus.cc:697] T a5299b8d044f41b7b68057e63d17b159 P 873357742d5843579556ee8166f23f47 [term 3 LEADER]: Becoming Leader. State: Replica: 873357742d5843579556ee8166f23f47, State: Running, Role: LEADER
I20260709 13:04:14.338315 12066 leader_election.cc:290] T 8c160ddfe9b34bdaa3835d0915c1e06f P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:14.339815 12198 consensus_queue.cc:237] T a5299b8d044f41b7b68057e63d17b159 P 873357742d5843579556ee8166f23f47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:14.324256 11879 raft_consensus.cc:3060] T 63d44f6ba6434ab2870fe3190f4d7c2f P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:04:14.138949 12221 raft_consensus.cc:493] T e5a5f5a608504650b48fd27f163faed6 P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:14.321311 12044 consensus_queue.cc:1048] T b4cd3d7dfbbf4096908ff11e9dc57141 P 873357742d5843579556ee8166f23f47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.022s
I20260709 13:04:14.342736 12221 raft_consensus.cc:515] T e5a5f5a608504650b48fd27f163faed6 P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:14.338809 12091 raft_consensus.cc:3060] T f371d2565dd84667b5613848604949bb P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:04:14.314873 12009 raft_consensus.cc:3060] T c31881534c5e4f0089c82057c1746652 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Advancing to term 4
W20260709 13:04:14.285877 12063 consensus_meta.cc:287] T c51782cd9d064ec0949e84e24dd4de67 P 186b12638a87410487467df721a3cbda: Time spent flushing consensus metadata: real 0.605s	user 0.012s	sys 0.000s
I20260709 13:04:14.338953 10874 rpcz_store.cc:276] 0709 13:04:13.053331 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:13.482921 (+429590us) service_pool.cc:224] Handling call
0709 13:04:13.540865 (+ 57944us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:13.600579 (+ 59714us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:14.136189 (+535610us) spinlock_profiling.cc:243] Waited 56.7 ms on lock 0x7b5400370fd0. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661618ad 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:04:14.146710 (+ 10521us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":130475264}
I20260709 13:04:14.346280 12063 raft_consensus.cc:515] T c51782cd9d064ec0949e84e24dd4de67 P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:14.348625 12063 leader_election.cc:290] T c51782cd9d064ec0949e84e24dd4de67 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 4 election: Requested vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
W20260709 13:04:14.289709 12103 consensus_meta.cc:287] T 7709387f4d3d4f9992d97a0fbca39a40 P 873357742d5843579556ee8166f23f47: Time spent flushing consensus metadata: real 0.609s	user 0.012s	sys 0.000s
I20260709 13:04:14.348816 12234 consensus_queue.cc:1048] T d71ca3a1ba6b48478aaf62741bf3cf6a P 873357742d5843579556ee8166f23f47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:04:14.349429 12103 raft_consensus.cc:515] T 7709387f4d3d4f9992d97a0fbca39a40 P 873357742d5843579556ee8166f23f47 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
W20260709 13:04:14.280588 12098 consensus_meta.cc:287] T 868e92339c2f4508af0afbd07c814d64 P 873357742d5843579556ee8166f23f47: Time spent flushing consensus metadata: real 0.505s	user 0.012s	sys 0.001s
I20260709 13:04:14.135362 12223 raft_consensus.cc:493] T 401a9cf11eac479aa445ec8c1d6e33b2 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:14.350769 12098 raft_consensus.cc:515] T 868e92339c2f4508af0afbd07c814d64 P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:14.350898 12223 raft_consensus.cc:515] T 401a9cf11eac479aa445ec8c1d6e33b2 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:14.056867 12215 raft_consensus.cc:515] T c51782cd9d064ec0949e84e24dd4de67 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:14.352778 11021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff13ae9a47c7455f962406c6d131a3ff" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
W20260709 13:04:14.152118 10992 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 229695us
I20260709 13:04:14.353672 11021 raft_consensus.cc:2368] T ff13ae9a47c7455f962406c6d131a3ff P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 10ca5ee68ad64b3283b9d235352c9d23 for earlier term 3. Current term is 4.
I20260709 13:04:14.282544 11974 raft_consensus.cc:3060] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:04:14.356559 12098 leader_election.cc:290] T 868e92339c2f4508af0afbd07c814d64 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 3 election: Requested vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
W20260709 13:04:14.301540 12105 consensus_meta.cc:287] T d270b9e5481245d9b5340a454a7cd10b P 10ca5ee68ad64b3283b9d235352c9d23: Time spent flushing consensus metadata: real 0.558s	user 0.016s	sys 0.003s
I20260709 13:04:14.358742 12105 raft_consensus.cc:515] T d270b9e5481245d9b5340a454a7cd10b P 10ca5ee68ad64b3283b9d235352c9d23 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:14.360121 12103 leader_election.cc:290] T 7709387f4d3d4f9992d97a0fbca39a40 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 6 election: Requested vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:14.360852 12221 leader_election.cc:290] T e5a5f5a608504650b48fd27f163faed6 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:14.361084 12105 leader_election.cc:290] T d270b9e5481245d9b5340a454a7cd10b P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 5 election: Requested vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
W20260709 13:04:14.302165 12019 consensus_meta.cc:287] T 836251ebc2b74f5da88b8a8b2338e2a5 P 0172fed22b6b4890a7fdc953fa128af4: Time spent flushing consensus metadata: real 0.510s	user 0.018s	sys 0.000s
I20260709 13:04:14.361924 12019 raft_consensus.cc:515] T 836251ebc2b74f5da88b8a8b2338e2a5 P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:14.362910 11099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87cd8f5097624c8692f1d83544d8881e" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:14.363647 11099 raft_consensus.cc:2468] T 87cd8f5097624c8692f1d83544d8881e P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 4.
I20260709 13:04:14.364389 12019 leader_election.cc:290] T 836251ebc2b74f5da88b8a8b2338e2a5 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 4 election: Requested vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:14.364837 10952 raft_consensus.cc:2468] T e5ba72ccd854430baadbc636dfd17136 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 4.
I20260709 13:04:14.365761 12215 leader_election.cc:290] T c51782cd9d064ec0949e84e24dd4de67 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
W20260709 13:04:14.301667 12073 consensus_meta.cc:287] T b67eff0634e540eab91cf8cee795969f P 10ca5ee68ad64b3283b9d235352c9d23: Time spent flushing consensus metadata: real 0.629s	user 0.007s	sys 0.011s
I20260709 13:04:14.304183 12004 leader_election.cc:290] T b4cd3d7dfbbf4096908ff11e9dc57141 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:14.305310 10876 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51704 (request call id 210) took 1189 ms. Trace:
I20260709 13:04:14.307410 10873 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51718 (request call id 283) took 1361 ms. Trace:
I20260709 13:04:14.366693 10876 rpcz_store.cc:276] 0709 13:04:13.116839 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:13.405410 (+288571us) service_pool.cc:224] Handling call
0709 13:04:14.305094 (+899684us) spinlock_profiling.cc:243] Waited 190 ms on lock 0x7b5400385c54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:04:14.305218 (+   124us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":437201280}
W20260709 13:04:14.308545 10844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:45807, user_credentials={real_user=slave}} blocked reactor thread for 481395us
I20260709 13:04:14.368168 10844 leader_election.cc:304] T e89e640169e14e2d98f968f76027daa7 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 186b12638a87410487467df721a3cbda; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47
I20260709 13:04:14.368989 12223 leader_election.cc:290] T 401a9cf11eac479aa445ec8c1d6e33b2 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:14.369611 12223 raft_consensus.cc:2749] T e89e640169e14e2d98f968f76027daa7 P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:04:14.370850 10845 leader_election.cc:304] T 87cd8f5097624c8692f1d83544d8881e P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda; no voters: 
I20260709 13:04:14.370934 11098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "868e92339c2f4508af0afbd07c814d64" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:04:14.371800 12223 raft_consensus.cc:2804] T 87cd8f5097624c8692f1d83544d8881e P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:04:14.372175 12223 raft_consensus.cc:493] T 87cd8f5097624c8692f1d83544d8881e P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:14.372537 12223 raft_consensus.cc:3060] T 87cd8f5097624c8692f1d83544d8881e P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:04:14.372910 11103 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5a5f5a608504650b48fd27f163faed6" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:14.373572 11103 raft_consensus.cc:2468] T e5a5f5a608504650b48fd27f163faed6 P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 4.
W20260709 13:04:14.219895 12222 log.cc:927] Time spent T a610d1a71f52432e88fe56130f921fa2 P 186b12638a87410487467df721a3cbda: Append to log took a long time: real 0.129s	user 0.001s	sys 0.000s
I20260709 13:04:14.323385 12201 raft_consensus.cc:515] T 1e4f2241a94a41f38e481137f868b83d P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:14.183856 11852 leader_election.cc:290] T d371c5e27bd843469d2e2e453734cb4a P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:14.323560 12216 raft_consensus.cc:493] T 11de93cd9b9d457489ff1cb80f976c9a P 10ca5ee68ad64b3283b9d235352c9d23 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:14.370146 10844 leader_election.cc:304] T 868e92339c2f4508af0afbd07c814d64 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 186b12638a87410487467df721a3cbda; no voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47
I20260709 13:04:14.374791 12216 raft_consensus.cc:515] T 11de93cd9b9d457489ff1cb80f976c9a P 10ca5ee68ad64b3283b9d235352c9d23 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
W20260709 13:04:14.290496 11795 consensus_meta.cc:287] T 8c160ddfe9b34bdaa3835d0915c1e06f P 186b12638a87410487467df721a3cbda: Time spent flushing consensus metadata: real 0.532s	user 0.013s	sys 0.001s
I20260709 13:04:14.305977 11029 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46640 (request call id 229) took 1231 ms. Trace:
I20260709 13:04:14.375941 11102 raft_consensus.cc:1275] T d270b9e5481245d9b5340a454a7cd10b P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Refusing update from remote peer 186b12638a87410487467df721a3cbda: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 13:04:14.376462 10845 leader_election.cc:304] T e5a5f5a608504650b48fd27f163faed6 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda; no voters: 
I20260709 13:04:14.376900 11795 raft_consensus.cc:2749] T 868e92339c2f4508af0afbd07c814d64 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:04:14.376089 11029 rpcz_store.cc:276] 0709 13:04:13.092998 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:13.103592 (+ 10594us) service_pool.cc:224] Handling call
0709 13:04:14.305785 (+1202193us) spinlock_profiling.cc:243] Waited 299 ms on lock 0x7b540082fad4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:04:14.305905 (+   120us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":688556800}
I20260709 13:04:14.377866 11795 raft_consensus.cc:2804] T e5a5f5a608504650b48fd27f163faed6 P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:04:14.378321 11795 raft_consensus.cc:493] T e5a5f5a608504650b48fd27f163faed6 P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:14.378693 11795 raft_consensus.cc:3060] T e5a5f5a608504650b48fd27f163faed6 P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Advancing to term 5
W20260709 13:04:14.316855 11067 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:45807, user_credentials={real_user=slave}} blocked reactor thread for 188507us
I20260709 13:04:14.379335 12216 leader_election.cc:290] T 11de93cd9b9d457489ff1cb80f976c9a P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:14.366788 10873 rpcz_store.cc:276] 0709 13:04:12.945626 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:12.945757 (+   131us) service_pool.cc:224] Handling call
0709 13:04:14.307208 (+1361451us) spinlock_profiling.cc:243] Waited 716 ms on lock 0x7b5400131ad4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:04:14.307344 (+   136us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1646366080}
I20260709 13:04:14.361829 11100 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c51782cd9d064ec0949e84e24dd4de67" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:04:14.372368 11098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7709387f4d3d4f9992d97a0fbca39a40" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:04:14.382063 10844 leader_election.cc:304] T acc241e7273d49149984968596242712 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 186b12638a87410487467df721a3cbda; no voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47
I20260709 13:04:14.358927 11653 consensus_queue.cc:1059] T 32147c6d2d5a477aa38c597c7f78df5b P 873357742d5843579556ee8166f23f47 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 }, Status: INVALID_TERM, Last received: 2.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:04:14.400027 10992 leader_election.cc:304] T e5ba72ccd854430baadbc636dfd17136 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47; no voters: 0172fed22b6b4890a7fdc953fa128af4
I20260709 13:04:14.410665 10877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c51782cd9d064ec0949e84e24dd4de67" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:14.411270 10878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11de93cd9b9d457489ff1cb80f976c9a" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:14.411674 10877 raft_consensus.cc:2393] T c51782cd9d064ec0949e84e24dd4de67 P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 10ca5ee68ad64b3283b9d235352c9d23 in current term 4: Already voted for candidate 186b12638a87410487467df721a3cbda in this term.
I20260709 13:04:14.411556 10879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d7ce47b9a92d43dd88d95bcda5038597" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:14.413081 10878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a610d1a71f52432e88fe56130f921fa2" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:14.402143 11069 leader_election.cc:304] T 0e5d327630e5438b878035d28efe6c0c P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda; no voters: 
I20260709 13:04:14.418865 10876 raft_consensus.cc:1275] T 5de5f0581e3b41dea729363d6ca563bb P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Refusing update from remote peer 10ca5ee68ad64b3283b9d235352c9d23: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 13:04:14.381132 12201 leader_election.cc:290] T 1e4f2241a94a41f38e481137f868b83d P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:14.427394 11103 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "401a9cf11eac479aa445ec8c1d6e33b2" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:14.396214 11100 raft_consensus.cc:3060] T c51782cd9d064ec0949e84e24dd4de67 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:04:14.428200 11103 raft_consensus.cc:2468] T 401a9cf11eac479aa445ec8c1d6e33b2 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 1.
I20260709 13:04:14.426587 11099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a9353d1d230416296627d7cf8b1d425" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
W20260709 13:04:14.430729 11095 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37678 (request call id 137) took 1971 ms (client timeout 1982 ms). Trace:
I20260709 13:04:14.391016 11982 raft_consensus.cc:493] T a1ec2b50c6ad4f7c8033476208abb26d P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:14.431262 11095 rpcz_store.cc:269] 0709 13:04:12.459082 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:12.894727 (+435645us) service_pool.cc:224] Handling call
0709 13:04:14.422632 (+1527905us) spinlock_profiling.cc:243] Waited 407 ms on lock 0x7b540074d954. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:04:14.430659 (+  8027us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":936817536}
I20260709 13:04:14.431734 11982 raft_consensus.cc:515] T a1ec2b50c6ad4f7c8033476208abb26d P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:14.432188 10845 leader_election.cc:304] T 401a9cf11eac479aa445ec8c1d6e33b2 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda; no voters: 
W20260709 13:04:14.433902 11069 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:14.435645 10872 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51718 (request call id 278) took 1523 ms. Trace:
I20260709 13:04:14.436089 10872 rpcz_store.cc:276] 0709 13:04:12.911880 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:12.912557 (+   677us) service_pool.cc:224] Handling call
0709 13:04:14.435431 (+1522874us) spinlock_profiling.cc:243] Waited 29.9 ms on lock 0x7b5400339d54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:04:14.435572 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":68788608}
I20260709 13:04:14.436697 11653 raft_consensus.cc:3055] T 32147c6d2d5a477aa38c597c7f78df5b P 873357742d5843579556ee8166f23f47 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:04:14.437134 11653 raft_consensus.cc:740] T 32147c6d2d5a477aa38c597c7f78df5b P 873357742d5843579556ee8166f23f47 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 873357742d5843579556ee8166f23f47, State: Running, Role: LEADER
I20260709 13:04:14.437295 10872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c160ddfe9b34bdaa3835d0915c1e06f" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:14.434128 11982 leader_election.cc:290] T a1ec2b50c6ad4f7c8033476208abb26d P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:14.438956 10876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1ec2b50c6ad4f7c8033476208abb26d" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:14.444795 11069 leader_election.cc:304] T a610d1a71f52432e88fe56130f921fa2 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4; no voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47
I20260709 13:04:14.465529 10876 raft_consensus.cc:2468] T a1ec2b50c6ad4f7c8033476208abb26d P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 2.
W20260709 13:04:14.445863 10992 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 46006.6us
I20260709 13:04:14.448230 12044 raft_consensus.cc:493] T 30f628e3d7914d609e7f1d980a74c654 P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 10ca5ee68ad64b3283b9d235352c9d23)
I20260709 13:04:14.448968 11673 raft_consensus.cc:493] T 8c160ddfe9b34bdaa3835d0915c1e06f P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 186b12638a87410487467df721a3cbda)
W20260709 13:04:14.472636 10919 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:45807, user_credentials={real_user=slave}} blocked reactor thread for 89536.2us
W20260709 13:04:14.472877 10921 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:38967, user_credentials={real_user=slave}} blocked reactor thread for 68706.4us
I20260709 13:04:14.449980 11921 raft_consensus.cc:2804] T e5ba72ccd854430baadbc636dfd17136 P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:04:14.399672 12171 raft_consensus.cc:2749] T acc241e7273d49149984968596242712 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:04:14.415287 12004 raft_consensus.cc:2804] T 0e5d327630e5438b878035d28efe6c0c P 0172fed22b6b4890a7fdc953fa128af4 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:04:14.438354 10872 raft_consensus.cc:2410] T 8c160ddfe9b34bdaa3835d0915c1e06f P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0172fed22b6b4890a7fdc953fa128af4 for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 13:04:14.469590 11069 leader_election.cc:304] T a1ec2b50c6ad4f7c8033476208abb26d P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda; no voters: 
I20260709 13:04:14.478046 10955 raft_consensus.cc:1275] T 732c4adeeff6422497aa7db4a82f535b P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Refusing update from remote peer 186b12638a87410487467df721a3cbda: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:04:14.393450 12063 consensus_queue.cc:1048] T d270b9e5481245d9b5340a454a7cd10b P 186b12638a87410487467df721a3cbda [LEADER]: Connected to new peer: Peer: permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.066s
W20260709 13:04:14.524701 11069 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:38967, user_credentials={real_user=slave}} blocked reactor thread for 55227.5us
I20260709 13:04:14.438035 11653 consensus_queue.cc:260] T 32147c6d2d5a477aa38c597c7f78df5b P 873357742d5843579556ee8166f23f47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:14.527571 12216 consensus_queue.cc:1059] T 525bfdbd6d934afd823f7d3ee3f9d052 P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20260709 13:04:14.529148 12048 consensus_meta.cc:287] T 4f9eb06ed01f4ce68be3af8a8434665b P 186b12638a87410487467df721a3cbda: Time spent flushing consensus metadata: real 0.767s	user 0.014s	sys 0.000s
I20260709 13:04:14.530128 12088 raft_consensus.cc:2804] T 401a9cf11eac479aa445ec8c1d6e33b2 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:04:14.530853 12088 raft_consensus.cc:493] T 401a9cf11eac479aa445ec8c1d6e33b2 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:14.531337 12088 raft_consensus.cc:3060] T 401a9cf11eac479aa445ec8c1d6e33b2 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:04:14.532858 10953 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87cd8f5097624c8692f1d83544d8881e" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:14.533613 10953 raft_consensus.cc:2468] T 87cd8f5097624c8692f1d83544d8881e P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 4.
I20260709 13:04:14.459262 11797 raft_consensus.cc:493] T 5d439326f23747eaab3c1e7da2d6b785 P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:14.538272 12216 raft_consensus.cc:3055] T 525bfdbd6d934afd823f7d3ee3f9d052 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:04:14.539501 10952 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c51782cd9d064ec0949e84e24dd4de67" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:14.532253 12073 consensus_queue.cc:1059] T 72b111ad7b5f4753ab1d2b5ee1114840 P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.008s
I20260709 13:04:14.539741 11795 raft_consensus.cc:515] T e5a5f5a608504650b48fd27f163faed6 P 186b12638a87410487467df721a3cbda [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:14.546590 12004 raft_consensus.cc:697] T 0e5d327630e5438b878035d28efe6c0c P 0172fed22b6b4890a7fdc953fa128af4 [term 5 LEADER]: Becoming Leader. State: Replica: 0172fed22b6b4890a7fdc953fa128af4, State: Running, Role: LEADER
I20260709 13:04:14.551537 10872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d371c5e27bd843469d2e2e453734cb4a" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:14.555253 12095 raft_consensus.cc:515] T 34064dc7bf3747dbb3876dd824ee0688 P 186b12638a87410487467df721a3cbda [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:14.470582 12066 raft_consensus.cc:2749] T a610d1a71f52432e88fe56130f921fa2 P 0172fed22b6b4890a7fdc953fa128af4 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:04:14.487653 11921 raft_consensus.cc:493] T e5ba72ccd854430baadbc636dfd17136 P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:14.585057 10919 leader_election.cc:400] T ff13ae9a47c7455f962406c6d131a3ff P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 873357742d5843579556ee8166f23f47 (127.0.107.195:45807) with higher term. Message: Invalid argument: T ff13ae9a47c7455f962406c6d131a3ff P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 10ca5ee68ad64b3283b9d235352c9d23 for earlier term 3. Current term is 4.
I20260709 13:04:14.587671 11797 raft_consensus.cc:515] T 5d439326f23747eaab3c1e7da2d6b785 P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:14.469794 10871 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d270b9e5481245d9b5340a454a7cd10b" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:04:14.595824 10919 leader_election.cc:403] T ff13ae9a47c7455f962406c6d131a3ff P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:04:14.532008 10950 raft_consensus.cc:1275] T 57b6a2418c1342e7924195e9c1dbd357 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Refusing update from remote peer 186b12638a87410487467df721a3cbda: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 13:04:14.438658 12091 raft_consensus.cc:515] T f371d2565dd84667b5613848604949bb P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:14.557482 11943 raft_consensus.cc:2804] T a1ec2b50c6ad4f7c8033476208abb26d P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260709 13:04:14.598694 12195 consensus_meta.cc:287] T bb7d67be500947b3aef4c4245e33671c P 873357742d5843579556ee8166f23f47: Time spent flushing consensus metadata: real 0.812s	user 0.012s	sys 0.003s
I20260709 13:04:14.534540 11094 raft_consensus.cc:1275] T 7757ecbf402e47bfbc93d0447741e26c P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Refusing update from remote peer 873357742d5843579556ee8166f23f47: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:04:14.622663 10953 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c51782cd9d064ec0949e84e24dd4de67" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:04:14.585168 12216 raft_consensus.cc:740] T 525bfdbd6d934afd823f7d3ee3f9d052 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 10ca5ee68ad64b3283b9d235352c9d23, State: Running, Role: LEADER
W20260709 13:04:14.624300 10919 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:45807, user_credentials={real_user=slave}} blocked reactor thread for 58515.8us
I20260709 13:04:14.625172 10880 raft_consensus.cc:1240] T 34064dc7bf3747dbb3876dd824ee0688 P 186b12638a87410487467df721a3cbda [term 6 FOLLOWER]: Rejecting Update request from peer 873357742d5843579556ee8166f23f47 for earlier term 5. Current term is 6. Ops: [3.2-5.3]
I20260709 13:04:14.487411 12044 raft_consensus.cc:515] T 30f628e3d7914d609e7f1d980a74c654 P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:14.642851 10955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5a5f5a608504650b48fd27f163faed6" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:14.681408 10955 raft_consensus.cc:1275] T d7ce47b9a92d43dd88d95bcda5038597 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Refusing update from remote peer 186b12638a87410487467df721a3cbda: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:04:14.600760 11921 raft_consensus.cc:3060] T e5ba72ccd854430baadbc636dfd17136 P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Advancing to term 5
W20260709 13:04:14.684544 10995 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:38967, user_credentials={real_user=slave}} blocked reactor thread for 394532us
I20260709 13:04:14.595386 10952 raft_consensus.cc:2468] T c51782cd9d064ec0949e84e24dd4de67 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 3.
I20260709 13:04:14.604688 12052 raft_consensus.cc:493] T 732c4adeeff6422497aa7db4a82f535b P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:14.621975 11943 raft_consensus.cc:493] T a1ec2b50c6ad4f7c8033476208abb26d P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:14.604205 11021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11de93cd9b9d457489ff1cb80f976c9a" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:14.709810 11067 leader_election.cc:304] T d371c5e27bd843469d2e2e453734cb4a P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:04:14.643227 12225 raft_consensus.cc:3060] T ff13ae9a47c7455f962406c6d131a3ff P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Advancing to term 4
I20260709 13:04:14.646166 10875 raft_consensus.cc:1240] T f371d2565dd84667b5613848604949bb P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Rejecting Update request from peer 873357742d5843579556ee8166f23f47 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260709 13:04:14.604681 10872 raft_consensus.cc:2468] T d371c5e27bd843469d2e2e453734cb4a P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 3.
I20260709 13:04:14.626351 12220 raft_consensus.cc:2749] T e5ba72ccd854430baadbc636dfd17136 P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:04:14.639372 11639 raft_consensus.cc:3055] T 72b111ad7b5f4753ab1d2b5ee1114840 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:04:14.643947 11797 leader_election.cc:290] T 5d439326f23747eaab3c1e7da2d6b785 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:14.640450 10947 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "525bfdbd6d934afd823f7d3ee3f9d052" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:04:14.564890 12215 consensus_queue.cc:1048] T 5de5f0581e3b41dea729363d6ca563bb P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.145s
I20260709 13:04:14.643656 12095 leader_election.cc:290] T 34064dc7bf3747dbb3876dd824ee0688 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 6 election: Requested vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:14.645200 10950 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34064dc7bf3747dbb3876dd824ee0688" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:04:14.491448 11673 raft_consensus.cc:515] T 8c160ddfe9b34bdaa3835d0915c1e06f P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:14.696484 10946 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d439326f23747eaab3c1e7da2d6b785" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:14.711365 11943 raft_consensus.cc:3060] T a1ec2b50c6ad4f7c8033476208abb26d P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Advancing to term 3
W20260709 13:04:14.773561 12245 log.cc:927] Time spent T f371d2565dd84667b5613848604949bb P 0172fed22b6b4890a7fdc953fa128af4: Append to log took a long time: real 0.069s	user 0.000s	sys 0.000s
I20260709 13:04:14.717514 12004 consensus_queue.cc:237] T 0e5d327630e5438b878035d28efe6c0c P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:14.716032 12066 raft_consensus.cc:2804] T d371c5e27bd843469d2e2e453734cb4a P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260709 13:04:14.715142 11937 consensus_meta.cc:287] T 75492f86fde046c3900df3eac8292fd4 P 0172fed22b6b4890a7fdc953fa128af4: Time spent flushing consensus metadata: real 0.776s	user 0.016s	sys 0.002s
I20260709 13:04:14.652207 11879 raft_consensus.cc:515] T 63d44f6ba6434ab2870fe3190f4d7c2f P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:14.778853 12066 raft_consensus.cc:493] T d371c5e27bd843469d2e2e453734cb4a P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:14.779318 12066 raft_consensus.cc:3060] T d371c5e27bd843469d2e2e453734cb4a P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:04:14.780036 11029 raft_consensus.cc:1240] T 63d44f6ba6434ab2870fe3190f4d7c2f P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Rejecting Update request from peer 10ca5ee68ad64b3283b9d235352c9d23 for earlier term 3. Current term is 4. Ops: []
W20260709 13:04:14.731302 11068 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:14.785225 11937 raft_consensus.cc:515] T 75492f86fde046c3900df3eac8292fd4 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:14.642037 12091 leader_election.cc:290] T f371d2565dd84667b5613848604949bb P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 4 election: Requested vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:14.761081 11639 raft_consensus.cc:740] T 72b111ad7b5f4753ab1d2b5ee1114840 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 10ca5ee68ad64b3283b9d235352c9d23, State: Running, Role: LEADER
I20260709 13:04:14.803074 10919 leader_election.cc:304] T 11de93cd9b9d457489ff1cb80f976c9a P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23; no voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47
W20260709 13:04:14.768991 12193 consensus_meta.cc:287] T f34f2bc02d85444f9096e0f5b4fe4331 P 0172fed22b6b4890a7fdc953fa128af4: Time spent flushing consensus metadata: real 0.613s	user 0.014s	sys 0.000s
I20260709 13:04:14.806941 12196 raft_consensus.cc:493] T a977842ba4b744f4bb2febd55d286006 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:14.813143 12193 raft_consensus.cc:515] T f34f2bc02d85444f9096e0f5b4fe4331 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:14.813365 12196 raft_consensus.cc:515] T a977842ba4b744f4bb2febd55d286006 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:14.815433 12193 leader_election.cc:290] T f34f2bc02d85444f9096e0f5b4fe4331 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 2 election: Requested vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:14.817365 12196 leader_election.cc:290] T a977842ba4b744f4bb2febd55d286006 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:14.834920 12096 raft_consensus.cc:3060] T 0f9df2bc281f4866b6da5da94a0defc7 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:04:14.837450 10875 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51718 (request call id 296) took 1028 ms. Trace:
I20260709 13:04:14.838133 10875 rpcz_store.cc:276] 0709 13:04:13.808603 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:14.035747 (+227144us) service_pool.cc:224] Handling call
0709 13:04:14.045179 (+  9432us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:14.730841 (+685662us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:14.836985 (+106144us) spinlock_profiling.cc:243] Waited 600 ms on lock 0x7b5400386154. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:04:14.837349 (+   364us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1381092736}
I20260709 13:04:14.838974 10872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "868e92339c2f4508af0afbd07c814d64" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:04:14.712466 12052 raft_consensus.cc:515] T 732c4adeeff6422497aa7db4a82f535b P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:14.803097 12216 consensus_queue.cc:260] T 525bfdbd6d934afd823f7d3ee3f9d052 P 10ca5ee68ad64b3283b9d235352c9d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:14.844997 12216 raft_consensus.cc:3060] T 525bfdbd6d934afd823f7d3ee3f9d052 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:04:14.811131 11937 leader_election.cc:290] T 75492f86fde046c3900df3eac8292fd4 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 3 election: Requested vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:14.810838 11639 consensus_queue.cc:260] T 72b111ad7b5f4753ab1d2b5ee1114840 P 10ca5ee68ad64b3283b9d235352c9d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:14.855479 11639 raft_consensus.cc:3060] T 72b111ad7b5f4753ab1d2b5ee1114840 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:04:14.845418 12215 raft_consensus.cc:493] T a64b7e99d6c54934a7de6f6dffcf5e94 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:14.858784 12215 raft_consensus.cc:515] T a64b7e99d6c54934a7de6f6dffcf5e94 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:14.861203 11779 raft_consensus.cc:493] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 10ca5ee68ad64b3283b9d235352c9d23 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:14.862066 11779 raft_consensus.cc:515] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 10ca5ee68ad64b3283b9d235352c9d23 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:14.864969 11779 leader_election.cc:290] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:14.812520 12105 raft_consensus.cc:2749] T 11de93cd9b9d457489ff1cb80f976c9a P 10ca5ee68ad64b3283b9d235352c9d23 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:04:14.767218 12073 raft_consensus.cc:493] T 5d439326f23747eaab3c1e7da2d6b785 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:14.865957 10954 raft_consensus.cc:1240] T d270b9e5481245d9b5340a454a7cd10b P 10ca5ee68ad64b3283b9d235352c9d23 [term 5 FOLLOWER]: Rejecting Update request from peer 186b12638a87410487467df721a3cbda for earlier term 4. Current term is 5. Ops: []
I20260709 13:04:14.866376 12073 raft_consensus.cc:515] T 5d439326f23747eaab3c1e7da2d6b785 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:14.868582 11795 leader_election.cc:290] T e5a5f5a608504650b48fd27f163faed6 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 5 election: Requested vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:14.757357 12195 raft_consensus.cc:493] T 4f9eb06ed01f4ce68be3af8a8434665b P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:14.871137 10946 raft_consensus.cc:2410] T 5d439326f23747eaab3c1e7da2d6b785 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 186b12638a87410487467df721a3cbda for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 13:04:14.869675 12195 raft_consensus.cc:515] T 4f9eb06ed01f4ce68be3af8a8434665b P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:14.873301 11094 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37676 (request call id 261) took 2074 ms. Trace:
I20260709 13:04:14.873955 12195 leader_election.cc:290] T 4f9eb06ed01f4ce68be3af8a8434665b P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:14.870852 11022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d71ca3a1ba6b48478aaf62741bf3cf6a" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:14.874958 11023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46640 (request call id 241) took 1042 ms. Trace:
I20260709 13:04:14.875150 11022 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46640 (request call id 239) took 1043 ms. Trace:
I20260709 13:04:14.875828 11022 rpcz_store.cc:276] 0709 13:04:13.831359 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:13.840094 (+  8735us) service_pool.cc:224] Handling call
0709 13:04:14.870643 (+1030549us) spinlock_profiling.cc:243] Waited 2.71 ms on lock 0x7b58007a2898. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f426303db71 00007f426303da22 00007f426bf56c09 00007f426b0b0ee0 00007f426b17a0ab 00007f426b17a131 00007f426b1280a0 00007f426b127984 00007f42646a26d4 00007f42646a2662
0709 13:04:14.875052 (+  4409us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":6228224}
I20260709 13:04:14.865054 12044 leader_election.cc:290] T 30f628e3d7914d609e7f1d980a74c654 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:14.878551 10873 raft_consensus.cc:1240] T e5a5f5a608504650b48fd27f163faed6 P 186b12638a87410487467df721a3cbda [term 5 FOLLOWER]: Rejecting Update request from peer 10ca5ee68ad64b3283b9d235352c9d23 for earlier term 4. Current term is 5. Ops: []
I20260709 13:04:14.876538 10950 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5a5f5a608504650b48fd27f163faed6" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:04:14.879151 12052 leader_election.cc:290] T 732c4adeeff6422497aa7db4a82f535b P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:14.881386 10873 raft_consensus.cc:1275] T 63d44f6ba6434ab2870fe3190f4d7c2f P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Refusing update from remote peer 10ca5ee68ad64b3283b9d235352c9d23: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:04:14.881649 12073 leader_election.cc:290] T 5d439326f23747eaab3c1e7da2d6b785 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:14.809159 10871 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7709387f4d3d4f9992d97a0fbca39a40" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
W20260709 13:04:14.890782 10995 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:38967, user_credentials={real_user=slave}} blocked reactor thread for 129726us
I20260709 13:04:14.891600 11024 raft_consensus.cc:1275] T 732c4adeeff6422497aa7db4a82f535b P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Refusing update from remote peer 186b12638a87410487467df721a3cbda: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:04:14.892357 11024 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46642 (request call id 156) took 1062 ms. Trace:
I20260709 13:04:14.892688 11024 rpcz_store.cc:276] 0709 13:04:13.830486 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:13.832478 (+  1992us) service_pool.cc:224] Handling call
0709 13:04:14.891003 (+1058525us) spinlock_profiling.cc:243] Waited 9.62 ms on lock 0x7b58007a2898. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f426303db71 00007f426303deee 0000000000375939 00007f426b0b11e0 00007f426b17a0c4 00007f426b17a131 00007f426b13bfc0 00007f426b126929 00007f426b1263d2 00007f426469dbd4
0709 13:04:14.891198 (+   195us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:14.892036 (+   838us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:14.892285 (+   249us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":22119680}
I20260709 13:04:14.894704 10878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d439326f23747eaab3c1e7da2d6b785" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:14.895390 10878 raft_consensus.cc:2468] T 5d439326f23747eaab3c1e7da2d6b785 P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 4.
I20260709 13:04:14.896468 10878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "732c4adeeff6422497aa7db4a82f535b" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:14.898589 12202 leader_election.cc:290] T ff13ae9a47c7455f962406c6d131a3ff P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 4 election: Requested vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:14.900195 10950 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff13ae9a47c7455f962406c6d131a3ff" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:04:14.904104 10949 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e4f2241a94a41f38e481137f868b83d" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:14.904810 11879 leader_election.cc:290] T 63d44f6ba6434ab2870fe3190f4d7c2f P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 4 election: Requested vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:14.811652 11943 raft_consensus.cc:515] T a1ec2b50c6ad4f7c8033476208abb26d P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:14.907377 11943 leader_election.cc:290] T a1ec2b50c6ad4f7c8033476208abb26d P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 3 election: Requested vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:14.873786 11094 rpcz_store.cc:276] 0709 13:04:12.798562 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:13.473791 (+675229us) service_pool.cc:224] Handling call
0709 13:04:14.450890 (+977099us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:14.683121 (+232231us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:14.873029 (+189908us) spinlock_profiling.cc:243] Waited 965 ms on lock 0x7b540074d950. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661618ad 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:04:14.873221 (+   192us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2219581696}
I20260709 13:04:14.875399 11023 rpcz_store.cc:276] 0709 13:04:13.832266 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:13.836338 (+  4072us) service_pool.cc:224] Handling call
0709 13:04:13.895653 (+ 59315us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:14.030536 (+134883us) raft_consensus.cc:1534] Early marking committed up to index 1
0709 13:04:14.030602 (+    66us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:14.102366 (+ 71764us) log.cc:844] Serialized 26 byte log entry
0709 13:04:14.438774 (+336408us) raft_consensus.cc:1645] Marking committed up to 1
0709 13:04:14.438854 (+    80us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:14.874469 (+435615us) spinlock_profiling.cc:243] Waited 67 ms on lock 0x7b540083fc54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:04:14.874537 (+    68us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:14.874594 (+    57us) raft_consensus.cc:1685] finished
0709 13:04:14.874614 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:14.874887 (+   273us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":25824,"raft.queue_time_us":50203,"raft.run_cpu_time_us":109,"raft.run_wall_time_us":108,"spinlock_wait_cycles":154002816,"thread_start_us":274096,"threads_started":1,"wal-append.queue_time_us":407851}
I20260709 13:04:14.861366 12215 leader_election.cc:290] T a64b7e99d6c54934a7de6f6dffcf5e94 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
W20260709 13:04:14.912490 11066 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 771963us
I20260709 13:04:14.913552 11066 leader_election.cc:304] T c31881534c5e4f0089c82057c1746652 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 10ca5ee68ad64b3283b9d235352c9d23; no voters: 186b12638a87410487467df721a3cbda
I20260709 13:04:14.916658 10995 leader_election.cc:304] T 6a9353d1d230416296627d7cf8b1d425 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47; no voters: 0172fed22b6b4890a7fdc953fa128af4
I20260709 13:04:14.919976 11024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "732c4adeeff6422497aa7db4a82f535b" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:14.920861 11024 raft_consensus.cc:2468] T 732c4adeeff6422497aa7db4a82f535b P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 3.
I20260709 13:04:14.921008 12202 raft_consensus.cc:2804] T 6a9353d1d230416296627d7cf8b1d425 P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:04:14.921459 12202 raft_consensus.cc:493] T 6a9353d1d230416296627d7cf8b1d425 P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 0172fed22b6b4890a7fdc953fa128af4)
I20260709 13:04:14.921875 12202 raft_consensus.cc:3060] T 6a9353d1d230416296627d7cf8b1d425 P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:04:14.922955 11943 raft_consensus.cc:2804] T c31881534c5e4f0089c82057c1746652 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:04:14.923543 11943 raft_consensus.cc:493] T c31881534c5e4f0089c82057c1746652 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 10ca5ee68ad64b3283b9d235352c9d23)
I20260709 13:04:14.924014 11943 raft_consensus.cc:3060] T c31881534c5e4f0089c82057c1746652 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:04:14.926935 11022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a64b7e99d6c54934a7de6f6dffcf5e94" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:14.929457 12091 raft_consensus.cc:493] T 4f9eb06ed01f4ce68be3af8a8434665b P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:14.930022 12091 raft_consensus.cc:515] T 4f9eb06ed01f4ce68be3af8a8434665b P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:14.931213 11926 consensus_queue.cc:1048] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:04:14.932348 12091 leader_election.cc:290] T 4f9eb06ed01f4ce68be3af8a8434665b P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:14.875494 10872 raft_consensus.cc:1275] T 72b111ad7b5f4753ab1d2b5ee1114840 P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Refusing update from remote peer 0172fed22b6b4890a7fdc953fa128af4: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 13:04:14.934640 10920 leader_election.cc:304] T 32147c6d2d5a477aa38c597c7f78df5b P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23; no voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47
I20260709 13:04:14.934985 10921 leader_election.cc:304] T 5d439326f23747eaab3c1e7da2d6b785 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda; no voters: 
W20260709 13:04:14.888090 12224 consensus_meta.cc:287] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 0172fed22b6b4890a7fdc953fa128af4: Time spent flushing consensus metadata: real 0.549s	user 0.009s	sys 0.005s
W20260709 13:04:14.934147 11100 consensus_meta.cc:287] T c51782cd9d064ec0949e84e24dd4de67 P 0172fed22b6b4890a7fdc953fa128af4: Time spent flushing consensus metadata: real 0.506s	user 0.009s	sys 0.005s
W20260709 13:04:14.944298 10994 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:14.944547 10953 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d71ca3a1ba6b48478aaf62741bf3cf6a" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:14.870961 11921 raft_consensus.cc:515] T e5ba72ccd854430baadbc636dfd17136 P 873357742d5843579556ee8166f23f47 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:14.959506 10880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e4f2241a94a41f38e481137f868b83d" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:14.968307 11921 leader_election.cc:290] T e5ba72ccd854430baadbc636dfd17136 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 5 election: Requested vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:14.970546 11098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff13ae9a47c7455f962406c6d131a3ff" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:04:14.971513 11098 raft_consensus.cc:3060] T ff13ae9a47c7455f962406c6d131a3ff P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:04:14.972121 10876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "836251ebc2b74f5da88b8a8b2338e2a5" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:04:14.976090 11101 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f9eb06ed01f4ce68be3af8a8434665b" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:14.977560 11673 leader_election.cc:290] T 8c160ddfe9b34bdaa3835d0915c1e06f P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:14.940297 11095 raft_consensus.cc:1240] T a1ec2b50c6ad4f7c8033476208abb26d P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Rejecting Update request from peer 873357742d5843579556ee8166f23f47 for earlier term 2. Current term is 3. Ops: []
W20260709 13:04:14.979403 11974 consensus_meta.cc:287] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 10ca5ee68ad64b3283b9d235352c9d23: Time spent flushing consensus metadata: real 0.623s	user 0.007s	sys 0.007s
I20260709 13:04:14.979946 11974 raft_consensus.cc:515] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:14.982825 11974 leader_election.cc:290] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 4 election: Requested vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
W20260709 13:04:14.984629 12158 consensus_meta.cc:287] T 5d439326f23747eaab3c1e7da2d6b785 P 0172fed22b6b4890a7fdc953fa128af4: Time spent flushing consensus metadata: real 0.858s	user 0.012s	sys 0.001s
I20260709 13:04:14.947558 12224 raft_consensus.cc:515] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 0172fed22b6b4890a7fdc953fa128af4 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:14.987383 12219 raft_consensus.cc:493] T 4f9eb06ed01f4ce68be3af8a8434665b P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 186b12638a87410487467df721a3cbda)
I20260709 13:04:14.988135 12219 raft_consensus.cc:515] T 4f9eb06ed01f4ce68be3af8a8434665b P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:14.988745 12224 leader_election.cc:290] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 5 election: Requested vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:14.980044 11026 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f1466124dbd4f6ebb3c4d62ca48bf96" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:14.988667 12088 raft_consensus.cc:515] T 401a9cf11eac479aa445ec8c1d6e33b2 P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:14.991322 11103 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5ba72ccd854430baadbc636dfd17136" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:04:14.992091 11103 raft_consensus.cc:3060] T e5ba72ccd854430baadbc636dfd17136 P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:04:14.985092 12158 raft_consensus.cc:515] T 5d439326f23747eaab3c1e7da2d6b785 P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
W20260709 13:04:14.997624 10921 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:38967, user_credentials={real_user=slave}} blocked reactor thread for 62733.4us
I20260709 13:04:14.949414 11100 raft_consensus.cc:2468] T c51782cd9d064ec0949e84e24dd4de67 P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 4.
W20260709 13:04:14.997867 10920 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 63325.5us
W20260709 13:04:15.005545 10919 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:45807, user_credentials={real_user=slave}} blocked reactor thread for 84879.9us
I20260709 13:04:15.005877 10953 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d7ce47b9a92d43dd88d95bcda5038597" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:15.006816 10919 leader_election.cc:304] T 732c4adeeff6422497aa7db4a82f535b P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47; no voters: 186b12638a87410487467df721a3cbda
I20260709 13:04:15.004459 10955 raft_consensus.cc:1240] T 0f9df2bc281f4866b6da5da94a0defc7 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Rejecting Update request from peer 0172fed22b6b4890a7fdc953fa128af4 for earlier term 1. Current term is 2. Ops: []
I20260709 13:04:15.011168 10955 raft_consensus.cc:1275] T 72b111ad7b5f4753ab1d2b5ee1114840 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Refusing update from remote peer 0172fed22b6b4890a7fdc953fa128af4: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
W20260709 13:04:15.004005 10992 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 62285.2us
I20260709 13:04:15.013526 10953 raft_consensus.cc:2393] T d7ce47b9a92d43dd88d95bcda5038597 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0172fed22b6b4890a7fdc953fa128af4 in current term 2: Already voted for candidate 186b12638a87410487467df721a3cbda in this term.
I20260709 13:04:14.938750 11779 consensus_queue.cc:1059] T e5a5f5a608504650b48fd27f163faed6 P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.063s
I20260709 13:04:15.021237 11101 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c160ddfe9b34bdaa3835d0915c1e06f" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:15.021624 10955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "836251ebc2b74f5da88b8a8b2338e2a5" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:04:15.022418 11101 raft_consensus.cc:2393] T 8c160ddfe9b34bdaa3835d0915c1e06f P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 873357742d5843579556ee8166f23f47 in current term 3: Already voted for candidate 0172fed22b6b4890a7fdc953fa128af4 in this term.
W20260709 13:04:15.024209 10845 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.571s	user 0.017s	sys 0.030s
W20260709 13:04:15.024708 10845 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.572s	user 0.017s	sys 0.030s
W20260709 13:04:15.025059 10845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 572688us
I20260709 13:04:14.937077 12073 consensus_queue.cc:1048] T 63d44f6ba6434ab2870fe3190f4d7c2f P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.073s
I20260709 13:04:15.037235 10955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a977842ba4b744f4bb2febd55d286006" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:15.038221 10955 raft_consensus.cc:2468] T a977842ba4b744f4bb2febd55d286006 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 2.
I20260709 13:04:15.039274 10955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5ba72ccd854430baadbc636dfd17136" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:04:15.040019 10955 raft_consensus.cc:3060] T e5ba72ccd854430baadbc636dfd17136 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:04:15.040971 10876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d156d6ae2ca4fa59a1e387edf0fea39" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 4 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
W20260709 13:04:15.044436 10995 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:38967, user_credentials={real_user=slave}} blocked reactor thread for 127866us
I20260709 13:04:14.989615 11026 raft_consensus.cc:2468] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 873357742d5843579556ee8166f23f47 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 5.
I20260709 13:04:15.003371 12232 raft_consensus.cc:2749] T 32147c6d2d5a477aa38c597c7f78df5b P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 13:04:15.053119 12236 consensus_queue.cc:1059] T 63d44f6ba6434ab2870fe3190f4d7c2f P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.008s
I20260709 13:04:15.057467 10949 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "30f628e3d7914d609e7f1d980a74c654" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:15.053435 10946 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4cd3d7dfbbf4096908ff11e9dc57141" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:15.058560 12088 leader_election.cc:290] T 401a9cf11eac479aa445ec8c1d6e33b2 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 3 election: Requested vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:15.058946 10949 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75492f86fde046c3900df3eac8292fd4" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:04:15.059303 10946 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63d44f6ba6434ab2870fe3190f4d7c2f" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:04:15.060209 10949 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f34f2bc02d85444f9096e0f5b4fe4331" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:04:15.062704 11097 raft_consensus.cc:1240] T 75492f86fde046c3900df3eac8292fd4 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Rejecting Update request from peer 10ca5ee68ad64b3283b9d235352c9d23 for earlier term 2. Current term is 3. Ops: []
I20260709 13:04:15.019492 11094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a977842ba4b744f4bb2febd55d286006" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:15.067047 11779 raft_consensus.cc:3055] T e5a5f5a608504650b48fd27f163faed6 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:04:15.067502 11779 raft_consensus.cc:740] T e5a5f5a608504650b48fd27f163faed6 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 10ca5ee68ad64b3283b9d235352c9d23, State: Running, Role: LEADER
I20260709 13:04:15.068420 11779 consensus_queue.cc:260] T e5a5f5a608504650b48fd27f163faed6 P 10ca5ee68ad64b3283b9d235352c9d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:15.005355 12105 raft_consensus.cc:2804] T 5d439326f23747eaab3c1e7da2d6b785 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:04:15.073096 10992 leader_election.cc:304] T a977842ba4b744f4bb2febd55d286006 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47; no voters: 
W20260709 13:04:15.034909 12223 consensus_meta.cc:287] T 87cd8f5097624c8692f1d83544d8881e P 186b12638a87410487467df721a3cbda: Time spent flushing consensus metadata: real 0.662s	user 0.015s	sys 0.000s
I20260709 13:04:15.005086 12064 raft_consensus.cc:493] T 7757ecbf402e47bfbc93d0447741e26c P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:15.005295 11926 raft_consensus.cc:493] T 8a69f32c53f24a50b7852d143665d7b4 P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 186b12638a87410487467df721a3cbda)
I20260709 13:04:15.072192 10953 raft_consensus.cc:1275] T a5299b8d044f41b7b68057e63d17b159 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Refusing update from remote peer 873357742d5843579556ee8166f23f47: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20260709 13:04:15.070753 10919 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:45807, user_credentials={real_user=slave}} blocked reactor thread for 64055.8us
W20260709 13:04:15.056731 12009 consensus_meta.cc:287] T c31881534c5e4f0089c82057c1746652 P 10ca5ee68ad64b3283b9d235352c9d23: Time spent flushing consensus metadata: real 0.711s	user 0.014s	sys 0.000s
I20260709 13:04:15.087543 10919 leader_election.cc:304] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:04:15.007835 12224 consensus_queue.cc:1048] T 72b111ad7b5f4753ab1d2b5ee1114840 P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.064s
I20260709 13:04:15.035567 11937 raft_consensus.cc:493] T 525bfdbd6d934afd823f7d3ee3f9d052 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 10ca5ee68ad64b3283b9d235352c9d23)
I20260709 13:04:15.091575 11103 raft_consensus.cc:2468] T e5ba72ccd854430baadbc636dfd17136 P 0172fed22b6b4890a7fdc953fa128af4 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 5.
I20260709 13:04:15.069804 11779 raft_consensus.cc:3060] T e5a5f5a608504650b48fd27f163faed6 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:04:15.083927 12127 raft_consensus.cc:2804] T 732c4adeeff6422497aa7db4a82f535b P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260709 13:04:15.168079 10920 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 111976us
I20260709 13:04:15.095769 10955 raft_consensus.cc:2468] T e5ba72ccd854430baadbc636dfd17136 P 10ca5ee68ad64b3283b9d235352c9d23 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 5.
I20260709 13:04:15.171643 11102 raft_consensus.cc:3060] T d7ce47b9a92d43dd88d95bcda5038597 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:04:15.080650 11974 raft_consensus.cc:493] T d7ce47b9a92d43dd88d95bcda5038597 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:15.079453 12105 raft_consensus.cc:493] T 5d439326f23747eaab3c1e7da2d6b785 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:04:15.182513 10995 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:38967, user_credentials={real_user=slave}} blocked reactor thread for 84506us
I20260709 13:04:15.173630 10876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f34f2bc02d85444f9096e0f5b4fe4331" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:04:15.183624 10995 leader_election.cc:304] T 868e92339c2f4508af0afbd07c814d64 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 873357742d5843579556ee8166f23f47; no voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda
I20260709 13:04:15.174760 10880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1ec2b50c6ad4f7c8033476208abb26d" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:04:15.184670 10880 raft_consensus.cc:3060] T a1ec2b50c6ad4f7c8033476208abb26d P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Advancing to term 3
W20260709 13:04:15.191707 10919 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:45807, user_credentials={real_user=slave}} blocked reactor thread for 104242us
I20260709 13:04:15.093596 12103 raft_consensus.cc:2804] T a977842ba4b744f4bb2febd55d286006 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:04:15.202252 10955 raft_consensus.cc:1240] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Rejecting Update request from peer 0172fed22b6b4890a7fdc953fa128af4 for earlier term 3. Current term is 4. Ops: [3.1-3.1]
W20260709 13:04:15.210417 10826 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 13:04:15.091262 10948 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49434 (request call id 270) took 1288 ms. Trace:
I20260709 13:04:15.216084 10948 rpcz_store.cc:276] 0709 13:04:13.800390 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:13.805862 (+  5472us) service_pool.cc:224] Handling call
0709 13:04:14.032523 (+226661us) spinlock_profiling.cc:243] Waited 3.08 ms on lock 0x7b580001d198. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f426303db71 00007f426303da22 00007f426bf56c09 00007f426b0b0ee0 00007f426b17a0ab 00007f426b17a131 00007f426b13bfc0 00007f426b126929 00007f426b1263d2 00007f426469dbd4
0709 13:04:14.032781 (+   258us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:04:14.097034 (+ 64253us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:14.106523 (+  9489us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:04:14.643587 (+537064us) spinlock_profiling.cc:243] Waited 4.23 ms on lock 0x7b2c00173848. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4264a5ba11 00007f4264a6b372 00007f426619da01 00007f426616243a 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2
0709 13:04:14.644535 (+   948us) log.cc:844] Serialized 52 byte log entry
0709 13:04:14.926298 (+281763us) spinlock_profiling.cc:243] Waited 2.87 ms on lock 0x7b1800229a80. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f42630753f3 00007f42630777c1 00007f4263077779 00007f4263077709 00007f42630776d1 00007f42630769cd 000000000038f354 000000000038f199 00007f426305883a
0709 13:04:14.961070 (+ 34772us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:04:14.962088 (+  1018us) log.cc:844] Serialized 14 byte log entry
0709 13:04:15.028999 (+ 66911us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:15.052636 (+ 23637us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:15.070892 (+ 18256us) raft_consensus.cc:1685] finished
0709 13:04:15.079223 (+  8331us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:15.088968 (+  9745us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":166886,"prepare.queue_time_us":441,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":41,"raft.queue_time_us":155883,"raft.run_cpu_time_us":353,"raft.run_wall_time_us":20893,"spinlock_wait_cycles":23427200,"thread_start_us":293816,"threads_started":1,"wal-append.queue_time_us":372478}
I20260709 13:04:15.213541 11066 leader_election.cc:304] T d7ce47b9a92d43dd88d95bcda5038597 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda
    @     0x7f426656a980 (unknown) at ??:0
I20260709 13:04:15.175653 12127 raft_consensus.cc:493] T 732c4adeeff6422497aa7db4a82f535b P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:15.265403 12127 raft_consensus.cc:3060] T 732c4adeeff6422497aa7db4a82f535b P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:04:15.269610 10845 leader_election.cc:304] T c51782cd9d064ec0949e84e24dd4de67 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda; no voters: 10ca5ee68ad64b3283b9d235352c9d23
I20260709 13:04:15.269999 10992 leader_election.cc:304] T e5ba72ccd854430baadbc636dfd17136 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:04:15.184763 12193 raft_consensus.cc:493] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:15.220690 11795 consensus_queue.cc:1048] T d7ce47b9a92d43dd88d95bcda5038597 P 186b12638a87410487467df721a3cbda [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:04:15.271662 12193 raft_consensus.cc:515] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:15.228511 12091 consensus_queue.cc:1059] T d270b9e5481245d9b5340a454a7cd10b P 186b12638a87410487467df721a3cbda [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:04:15.271559 11972 consensus_queue.cc:1048] T a5299b8d044f41b7b68057e63d17b159 P 873357742d5843579556ee8166f23f47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:04:15.273110 12091 raft_consensus.cc:3055] T d270b9e5481245d9b5340a454a7cd10b P 186b12638a87410487467df721a3cbda [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:04:15.273492 12091 raft_consensus.cc:740] T d270b9e5481245d9b5340a454a7cd10b P 186b12638a87410487467df721a3cbda [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 186b12638a87410487467df721a3cbda, State: Running, Role: LEADER
I20260709 13:04:15.274396 12091 consensus_queue.cc:260] T d270b9e5481245d9b5340a454a7cd10b P 186b12638a87410487467df721a3cbda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:15.275743 12091 raft_consensus.cc:3060] T d270b9e5481245d9b5340a454a7cd10b P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Advancing to term 5
W20260709 13:04:15.229919 12259 log.cc:927] Time spent T 5d156d6ae2ca4fa59a1e387edf0fea39 P 186b12638a87410487467df721a3cbda: Append to log took a long time: real 0.181s	user 0.000s	sys 0.001s
I20260709 13:04:15.091917 12064 raft_consensus.cc:515] T 7757ecbf402e47bfbc93d0447741e26c P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:15.221649 12172 consensus_queue.cc:1048] T 57b6a2418c1342e7924195e9c1dbd357 P 186b12638a87410487467df721a3cbda [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.023s
I20260709 13:04:15.237445 10947 raft_consensus.cc:2468] T 525bfdbd6d934afd823f7d3ee3f9d052 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 4.
W20260709 13:04:15.238893 10919 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:15.239856 12073 consensus_queue.cc:1059] T 75492f86fde046c3900df3eac8292fd4 P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:04:15.241649 10948 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f1466124dbd4f6ebb3c4d62ca48bf96" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:04:15.244426 11103 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5a5f5a608504650b48fd27f163faed6" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:04:15.301364 11103 raft_consensus.cc:3060] T e5a5f5a608504650b48fd27f163faed6 P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:04:15.247136 12158 leader_election.cc:290] T 5d439326f23747eaab3c1e7da2d6b785 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 4 election: Requested vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:15.255640 12240 raft_consensus.cc:3055] T 63d44f6ba6434ab2870fe3190f4d7c2f P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:04:15.188381 12105 raft_consensus.cc:3060] T 5d439326f23747eaab3c1e7da2d6b785 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:04:15.259873 11653 raft_consensus.cc:3060] T 32147c6d2d5a477aa38c597c7f78df5b P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:04:15.097265 12070 consensus_queue.cc:1059] T 34064dc7bf3747dbb3876dd824ee0688 P 873357742d5843579556ee8166f23f47 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.130s
I20260709 13:04:15.202895 11953 consensus_queue.cc:1059] T f371d2565dd84667b5613848604949bb P 873357742d5843579556ee8166f23f47 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.060s
I20260709 13:04:15.089560 12223 raft_consensus.cc:515] T 87cd8f5097624c8692f1d83544d8881e P 186b12638a87410487467df721a3cbda [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:15.189421 12071 consensus_queue.cc:1048] T f34f2bc02d85444f9096e0f5b4fe4331 P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.074s
I20260709 13:04:15.207826 12215 raft_consensus.cc:2804] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 10ca5ee68ad64b3283b9d235352c9d23 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:04:15.309130 11972 raft_consensus.cc:493] T bb7d67be500947b3aef4c4245e33671c P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:15.175004 11937 raft_consensus.cc:515] T 525bfdbd6d934afd823f7d3ee3f9d052 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:15.264778 11095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f371d2565dd84667b5613848604949bb" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:04:15.271145 12068 raft_consensus.cc:2804] T c51782cd9d064ec0949e84e24dd4de67 P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:04:15.271984 12198 raft_consensus.cc:2804] T e5ba72ccd854430baadbc636dfd17136 P 873357742d5843579556ee8166f23f47 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:04:15.430500 10877 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51704 (request call id 220) took 1017 ms. Trace:
I20260709 13:04:15.229385 12220 consensus_queue.cc:1048] T 72b111ad7b5f4753ab1d2b5ee1114840 P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
W20260709 13:04:15.278826 10995 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:38967, user_credentials={real_user=slave}} blocked reactor thread for 95343.4us
I20260709 13:04:15.223430 11066 leader_election.cc:304] T 836251ebc2b74f5da88b8a8b2338e2a5 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda
I20260709 13:04:15.282471 10880 raft_consensus.cc:2468] T a1ec2b50c6ad4f7c8033476208abb26d P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 3.
I20260709 13:04:15.289448 10878 raft_consensus.cc:1275] T 0e5d327630e5438b878035d28efe6c0c P 186b12638a87410487467df721a3cbda [term 5 FOLLOWER]: Refusing update from remote peer 0172fed22b6b4890a7fdc953fa128af4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260709 13:04:15.293438 12064 leader_election.cc:290] T 7757ecbf402e47bfbc93d0447741e26c P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:15.231639 12219 leader_election.cc:290] T 4f9eb06ed01f4ce68be3af8a8434665b P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:15.213150 11943 raft_consensus.cc:515] T c31881534c5e4f0089c82057c1746652 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:15.210314 11097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d439326f23747eaab3c1e7da2d6b785" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:15.274199 12193 leader_election.cc:290] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:15.209923 12103 raft_consensus.cc:493] T a977842ba4b744f4bb2febd55d286006 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:15.091746 11926 raft_consensus.cc:515] T 8a69f32c53f24a50b7852d143665d7b4 P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:15.191655 12088 consensus_queue.cc:1048] T 732c4adeeff6422497aa7db4a82f535b P 186b12638a87410487467df721a3cbda [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.105s
I20260709 13:04:15.286474 11879 raft_consensus.cc:2749] T 868e92339c2f4508af0afbd07c814d64 P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20260709 13:04:15.503753 12066 consensus_meta.cc:287] T d371c5e27bd843469d2e2e453734cb4a P 0172fed22b6b4890a7fdc953fa128af4: Time spent flushing consensus metadata: real 0.699s	user 0.014s	sys 0.000s
I20260709 13:04:15.210978 11974 raft_consensus.cc:515] T d7ce47b9a92d43dd88d95bcda5038597 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:15.295288 12048 raft_consensus.cc:493] T 8c160ddfe9b34bdaa3835d0915c1e06f P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:15.313439 11100 raft_consensus.cc:1240] T 836251ebc2b74f5da88b8a8b2338e2a5 P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Rejecting Update request from peer 10ca5ee68ad64b3283b9d235352c9d23 for earlier term 3. Current term is 4. Ops: []
I20260709 13:04:15.375574 11103 raft_consensus.cc:2468] T e5a5f5a608504650b48fd27f163faed6 P 0172fed22b6b4890a7fdc953fa128af4 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 5.
I20260709 13:04:15.550489 11102 raft_consensus.cc:1275] T d7ce47b9a92d43dd88d95bcda5038597 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Refusing update from remote peer 186b12638a87410487467df721a3cbda: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:04:15.316534 10872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d439326f23747eaab3c1e7da2d6b785" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:04:15.093585 12009 raft_consensus.cc:515] T c31881534c5e4f0089c82057c1746652 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:15.558749 11926 leader_election.cc:290] T 8a69f32c53f24a50b7852d143665d7b4 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
W20260709 13:04:15.412664 10994 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 223863us
I20260709 13:04:15.308411 12240 raft_consensus.cc:740] T 63d44f6ba6434ab2870fe3190f4d7c2f P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 10ca5ee68ad64b3283b9d235352c9d23, State: Running, Role: LEADER
I20260709 13:04:15.560853 12240 consensus_queue.cc:260] T 63d44f6ba6434ab2870fe3190f4d7c2f P 10ca5ee68ad64b3283b9d235352c9d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:15.562090 12240 raft_consensus.cc:3060] T 63d44f6ba6434ab2870fe3190f4d7c2f P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:04:15.328122 12250 raft_consensus.cc:493] T a610d1a71f52432e88fe56130f921fa2 P 186b12638a87410487467df721a3cbda [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:15.565538 12250 raft_consensus.cc:515] T a610d1a71f52432e88fe56130f921fa2 P 186b12638a87410487467df721a3cbda [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:15.554268 12004 raft_consensus.cc:2749] T d7ce47b9a92d43dd88d95bcda5038597 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:04:15.567852 11020 raft_consensus.cc:1240] T 63d44f6ba6434ab2870fe3190f4d7c2f P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Rejecting Update request from peer 10ca5ee68ad64b3283b9d235352c9d23 for earlier term 3. Current term is 4. Ops: [3.3-3.3]
W20260709 13:04:15.415899 12096 consensus_meta.cc:287] T 0f9df2bc281f4866b6da5da94a0defc7 P 0172fed22b6b4890a7fdc953fa128af4: Time spent flushing consensus metadata: real 0.580s	user 0.012s	sys 0.000s
I20260709 13:04:15.430994 10877 rpcz_store.cc:276] 0709 13:04:14.413060 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:14.413251 (+   191us) service_pool.cc:224] Handling call
0709 13:04:14.413596 (+   345us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:14.624308 (+210712us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:14.624352 (+    44us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:14.648379 (+ 24027us) log.cc:844] Serialized 26 byte log entry
0709 13:04:15.006658 (+358279us) raft_consensus.cc:1645] Marking committed up to 0
0709 13:04:15.006714 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:15.006793 (+    79us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:15.430218 (+423425us) raft_consensus.cc:1685] finished
0709 13:04:15.430241 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:15.430396 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5165,"raft.queue_time_us":47421,"raft.run_cpu_time_us":190,"raft.run_wall_time_us":3983,"thread_start_us":490158,"threads_started":2,"wal-append.queue_time_us":320417}
I20260709 13:04:15.426853 12196 raft_consensus.cc:3055] T 34064dc7bf3747dbb3876dd824ee0688 P 873357742d5843579556ee8166f23f47 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:04:15.440413 11095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d156d6ae2ca4fa59a1e387edf0fea39" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 4 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:04:15.383013 12215 raft_consensus.cc:493] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 10ca5ee68ad64b3283b9d235352c9d23 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:15.478699 10878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7757ecbf402e47bfbc93d0447741e26c" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:15.602607 10878 raft_consensus.cc:2410] T 7757ecbf402e47bfbc93d0447741e26c P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0172fed22b6b4890a7fdc953fa128af4 for term 4 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260709 13:04:15.486608 10995 leader_election.cc:304] T 7709387f4d3d4f9992d97a0fbca39a40 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 873357742d5843579556ee8166f23f47; no voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda
I20260709 13:04:15.500010 10880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555217d45d9b4ee492dbb3ddaf0f7c02" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:15.416783 12172 raft_consensus.cc:493] T 30f628e3d7914d609e7f1d980a74c654 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 10ca5ee68ad64b3283b9d235352c9d23)
W20260709 13:04:15.506086 11066 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 282735us
I20260709 13:04:15.423499 12224 raft_consensus.cc:493] T 5de5f0581e3b41dea729363d6ca563bb P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:15.426126 12095 raft_consensus.cc:493] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:15.312350 12225 consensus_meta.cc:287] T ff13ae9a47c7455f962406c6d131a3ff P 10ca5ee68ad64b3283b9d235352c9d23: Time spent flushing consensus metadata: real 0.582s	user 0.008s	sys 0.005s
I20260709 13:04:15.480823 12071 raft_consensus.cc:493] T bb7d67be500947b3aef4c4245e33671c P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:15.529759 10876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f9eb06ed01f4ce68be3af8a8434665b" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:15.522428 12103 raft_consensus.cc:3060] T a977842ba4b744f4bb2febd55d286006 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:04:15.510015 12219 raft_consensus.cc:493] T 32147c6d2d5a477aa38c597c7f78df5b P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:15.450069 10874 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51696 (request call id 278) took 1022 ms. Trace:
I20260709 13:04:15.505291 12064 raft_consensus.cc:493] T bb7d67be500947b3aef4c4245e33671c P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:15.551899 11102 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d270b9e5481245d9b5340a454a7cd10b" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:04:15.426369 11972 raft_consensus.cc:515] T bb7d67be500947b3aef4c4245e33671c P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:15.508284 12198 raft_consensus.cc:697] T e5ba72ccd854430baadbc636dfd17136 P 873357742d5843579556ee8166f23f47 [term 5 LEADER]: Becoming Leader. State: Replica: 873357742d5843579556ee8166f23f47, State: Running, Role: LEADER
I20260709 13:04:15.532893 12073 raft_consensus.cc:493] T b67eff0634e540eab91cf8cee795969f P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:15.605010 12215 raft_consensus.cc:3060] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 10ca5ee68ad64b3283b9d235352c9d23 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:04:15.558877 12009 leader_election.cc:290] T c31881534c5e4f0089c82057c1746652 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 4 election: Requested vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:15.314132 11795 raft_consensus.cc:493] T c31881534c5e4f0089c82057c1746652 P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:15.356424 12267 log.cc:927] Time spent T 11de93cd9b9d457489ff1cb80f976c9a P 873357742d5843579556ee8166f23f47: Append to log took a long time: real 0.050s	user 0.001s	sys 0.000s
W20260709 13:04:15.538918 12271 log.cc:927] Time spent T 11de93cd9b9d457489ff1cb80f976c9a P 10ca5ee68ad64b3283b9d235352c9d23: Append to log took a long time: real 0.115s	user 0.000s	sys 0.000s
W20260709 13:04:15.563205 12202 consensus_meta.cc:287] T 6a9353d1d230416296627d7cf8b1d425 P 873357742d5843579556ee8166f23f47: Time spent flushing consensus metadata: real 0.641s	user 0.006s	sys 0.008s
I20260709 13:04:15.564452 11103 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f9eb06ed01f4ce68be3af8a8434665b" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:15.659111 11103 raft_consensus.cc:2468] T 4f9eb06ed01f4ce68be3af8a8434665b P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 2.
I20260709 13:04:15.660362 11103 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f1466124dbd4f6ebb3c4d62ca48bf96" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:15.661032 11103 raft_consensus.cc:2468] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 0172fed22b6b4890a7fdc953fa128af4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 5.
I20260709 13:04:15.661895 11103 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a64b7e99d6c54934a7de6f6dffcf5e94" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
W20260709 13:04:15.551321 12255 log.cc:927] Time spent T 00000000000000000000000000000000 P c669bee8ad644d0d956c148b33886b51: Append to log took a long time: real 0.301s	user 0.003s	sys 0.000s
I20260709 13:04:15.606173 11066 leader_election.cc:304] T f34f2bc02d85444f9096e0f5b4fe4331 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda
W20260709 13:04:15.668779 11066 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 62705.3us
W20260709 13:04:15.578663 12263 log.cc:927] Time spent T 0e5d327630e5438b878035d28efe6c0c P 0172fed22b6b4890a7fdc953fa128af4: Append to log took a long time: real 0.103s	user 0.000s	sys 0.004s
I20260709 13:04:15.672740 12282 raft_consensus.cc:2749] T f34f2bc02d85444f9096e0f5b4fe4331 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 13:04:15.673800 12282 consensus_queue.cc:1059] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:04:15.674991 12282 raft_consensus.cc:3055] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:04:15.675397 12282 raft_consensus.cc:740] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 0172fed22b6b4890a7fdc953fa128af4, State: Running, Role: LEADER
I20260709 13:04:15.676405 12282 consensus_queue.cc:260] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 0172fed22b6b4890a7fdc953fa128af4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 3.1, Last appended by leader: 1, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:15.677837 12282 raft_consensus.cc:3060] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Advancing to term 4
W20260709 13:04:15.598359 12266 log.cc:927] Time spent T 30f628e3d7914d609e7f1d980a74c654 P 873357742d5843579556ee8166f23f47: Append to log took a long time: real 0.183s	user 0.001s	sys 0.000s
I20260709 13:04:15.569399 12200 raft_consensus.cc:493] T 0f9df2bc281f4866b6da5da94a0defc7 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:15.679865 12200 raft_consensus.cc:515] T 0f9df2bc281f4866b6da5da94a0defc7 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:15.555081 12104 raft_consensus.cc:493] T 4c2a35bb99374e93b573cf870d04faaa P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:15.682744 12104 raft_consensus.cc:515] T 4c2a35bb99374e93b573cf870d04faaa P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:15.685287 12104 leader_election.cc:290] T 4c2a35bb99374e93b573cf870d04faaa P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:15.598243 12193 raft_consensus.cc:2749] T 836251ebc2b74f5da88b8a8b2338e2a5 P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 13:04:15.624243 10876 raft_consensus.cc:2468] T 4f9eb06ed01f4ce68be3af8a8434665b P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 3.
I20260709 13:04:15.606981 11097 raft_consensus.cc:2410] T 5d439326f23747eaab3c1e7da2d6b785 P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 186b12638a87410487467df721a3cbda for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 13:04:15.531838 12066 raft_consensus.cc:515] T d371c5e27bd843469d2e2e453734cb4a P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:15.576503 12196 raft_consensus.cc:740] T 34064dc7bf3747dbb3876dd824ee0688 P 873357742d5843579556ee8166f23f47 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 873357742d5843579556ee8166f23f47, State: Running, Role: LEADER
W20260709 13:04:15.620811 10995 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:38967, user_credentials={real_user=slave}} blocked reactor thread for 134376us
I20260709 13:04:15.568099 12250 leader_election.cc:290] T a610d1a71f52432e88fe56130f921fa2 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:15.685817 12104 raft_consensus.cc:493] T b67eff0634e540eab91cf8cee795969f P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:15.700659 12098 raft_consensus.cc:3055] T f371d2565dd84667b5613848604949bb P 873357742d5843579556ee8166f23f47 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:04:15.700459 12104 raft_consensus.cc:515] T b67eff0634e540eab91cf8cee795969f P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:15.701241 12098 raft_consensus.cc:740] T f371d2565dd84667b5613848604949bb P 873357742d5843579556ee8166f23f47 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 873357742d5843579556ee8166f23f47, State: Running, Role: LEADER
I20260709 13:04:15.555768 12048 raft_consensus.cc:515] T 8c160ddfe9b34bdaa3835d0915c1e06f P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:15.639772 11102 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d439326f23747eaab3c1e7da2d6b785" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:15.702896 11102 raft_consensus.cc:2468] T 5d439326f23747eaab3c1e7da2d6b785 P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 4.
I20260709 13:04:15.685979 12068 raft_consensus.cc:697] T c51782cd9d064ec0949e84e24dd4de67 P 186b12638a87410487467df721a3cbda [term 4 LEADER]: Becoming Leader. State: Replica: 186b12638a87410487467df721a3cbda, State: Running, Role: LEADER
I20260709 13:04:15.704340 11102 raft_consensus.cc:1240] T e5a5f5a608504650b48fd27f163faed6 P 0172fed22b6b4890a7fdc953fa128af4 [term 5 FOLLOWER]: Rejecting Update request from peer 10ca5ee68ad64b3283b9d235352c9d23 for earlier term 4. Current term is 5. Ops: []
I20260709 13:04:15.556800 10872 raft_consensus.cc:2393] T 5d439326f23747eaab3c1e7da2d6b785 P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 0172fed22b6b4890a7fdc953fa128af4 in current term 4: Already voted for candidate 186b12638a87410487467df721a3cbda in this term.
I20260709 13:04:15.628942 12225 raft_consensus.cc:2749] T ff13ae9a47c7455f962406c6d131a3ff P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer 873357742d5843579556ee8166f23f47 (127.0.107.195:45807) with higher term. Message: Invalid argument: T ff13ae9a47c7455f962406c6d131a3ff P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 10ca5ee68ad64b3283b9d235352c9d23 for earlier term 3. Current term is 4.
W20260709 13:04:15.540040 10947 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49434 (request call id 269) took 1740 ms (client timeout 2121 ms). Trace:
I20260709 13:04:15.634769 11974 leader_election.cc:290] T d7ce47b9a92d43dd88d95bcda5038597 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:15.632256 11633 raft_consensus.cc:2749] T 7709387f4d3d4f9992d97a0fbca39a40 P 873357742d5843579556ee8166f23f47 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20260709 13:04:15.706355 10947 rpcz_store.cc:269] 0709 13:04:13.800036 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:13.805147 (+  5111us) service_pool.cc:224] Handling call
0709 13:04:14.640173 (+835026us) spinlock_profiling.cc:243] Waited 9.22 ms on lock 0x7b580001d198. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f426303db71 00007f426303da22 00007f426bf56c09 00007f426b0b0ee0 00007f426b17a0ab 00007f426b17a131 00007f426b1280a0 00007f426b127984 00007f42646a26d4 00007f42646a2662
0709 13:04:15.539832 (+899659us) spinlock_profiling.cc:243] Waited 307 ms on lock 0x7b54001f4fd4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:04:15.539969 (+   137us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":726155904}
W20260709 13:04:15.661895 10920 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:15.710867 12048 leader_election.cc:290] T 8c160ddfe9b34bdaa3835d0915c1e06f P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:15.711993 11974 consensus_queue.cc:1059] T 836251ebc2b74f5da88b8a8b2338e2a5 P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20260709 13:04:15.610567 10875 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.769s	user 0.036s	sys 0.061s
W20260709 13:04:15.716269 10875 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.874s	user 0.037s	sys 0.061s
I20260709 13:04:15.716790 11974 raft_consensus.cc:3055] T 836251ebc2b74f5da88b8a8b2338e2a5 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:04:15.717211 10875 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51704 (request call id 221) took 1303 ms. Trace:
I20260709 13:04:15.717288 11974 raft_consensus.cc:740] T 836251ebc2b74f5da88b8a8b2338e2a5 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 10ca5ee68ad64b3283b9d235352c9d23, State: Running, Role: LEADER
I20260709 13:04:15.687048 11101 raft_consensus.cc:1275] T 0f9df2bc281f4866b6da5da94a0defc7 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Refusing update from remote peer 186b12638a87410487467df721a3cbda: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:04:15.718240 11099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a610d1a71f52432e88fe56130f921fa2" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:15.705049 12068 consensus_queue.cc:237] T c51782cd9d064ec0949e84e24dd4de67 P 186b12638a87410487467df721a3cbda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:15.718652 11974 consensus_queue.cc:260] T 836251ebc2b74f5da88b8a8b2338e2a5 P 10ca5ee68ad64b3283b9d235352c9d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:15.694478 12066 leader_election.cc:290] T d371c5e27bd843469d2e2e453734cb4a P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 4 election: Requested vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:15.720000 11974 raft_consensus.cc:3060] T 836251ebc2b74f5da88b8a8b2338e2a5 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:04:15.697651 12250 raft_consensus.cc:493] T 11de93cd9b9d457489ff1cb80f976c9a P 186b12638a87410487467df721a3cbda [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:15.720407 11101 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87cd8f5097624c8692f1d83544d8881e" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:04:15.721100 11101 raft_consensus.cc:3060] T 87cd8f5097624c8692f1d83544d8881e P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:04:15.720757 12250 raft_consensus.cc:515] T 11de93cd9b9d457489ff1cb80f976c9a P 186b12638a87410487467df721a3cbda [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:15.695461 10995 leader_election.cc:304] T 1e4f2241a94a41f38e481137f868b83d P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 873357742d5843579556ee8166f23f47; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda
I20260709 13:04:15.710878 10845 leader_election.cc:304] T e5a5f5a608504650b48fd27f163faed6 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda; no voters: 10ca5ee68ad64b3283b9d235352c9d23
I20260709 13:04:15.725469 11069 leader_election.cc:304] T a1ec2b50c6ad4f7c8033476208abb26d P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda; no voters: 
I20260709 13:04:15.613358 12070 consensus_queue.cc:1048] T 7757ecbf402e47bfbc93d0447741e26c P 873357742d5843579556ee8166f23f47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.040s
I20260709 13:04:15.610996 12172 raft_consensus.cc:515] T 30f628e3d7914d609e7f1d980a74c654 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
W20260709 13:04:15.695202 11066 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:15.733402 11102 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d7ce47b9a92d43dd88d95bcda5038597" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:15.734261 11102 raft_consensus.cc:2468] T d7ce47b9a92d43dd88d95bcda5038597 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 2.
I20260709 13:04:15.715126 12104 leader_election.cc:290] T b67eff0634e540eab91cf8cee795969f P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:15.695290 12196 consensus_queue.cc:260] T 34064dc7bf3747dbb3876dd824ee0688 P 873357742d5843579556ee8166f23f47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:15.686291 12292 raft_consensus.cc:493] T e96e80ba9ac34d0998395515e5c8c41a P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:15.711526 12048 raft_consensus.cc:493] T a5299b8d044f41b7b68057e63d17b159 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:15.709775 10920 leader_election.cc:304] T d270b9e5481245d9b5340a454a7cd10b P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23; no voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda
I20260709 13:04:15.752241 11103 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c51782cd9d064ec0949e84e24dd4de67" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:15.725441 10845 leader_election.cc:304] T 4f9eb06ed01f4ce68be3af8a8434665b P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda; no voters: 
I20260709 13:04:15.753368 11103 raft_consensus.cc:2393] T c51782cd9d064ec0949e84e24dd4de67 P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 10ca5ee68ad64b3283b9d235352c9d23 in current term 4: Already voted for candidate 186b12638a87410487467df721a3cbda in this term.
I20260709 13:04:15.754318 10845 leader_election.cc:304] T 5d439326f23747eaab3c1e7da2d6b785 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 186b12638a87410487467df721a3cbda; no voters: 0172fed22b6b4890a7fdc953fa128af4, 10ca5ee68ad64b3283b9d235352c9d23
I20260709 13:04:15.614616 12224 raft_consensus.cc:515] T 5de5f0581e3b41dea729363d6ca563bb P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:15.624783 12095 raft_consensus.cc:515] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:15.748031 11102 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c31881534c5e4f0089c82057c1746652" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:04:15.761911 11102 raft_consensus.cc:3060] T c31881534c5e4f0089c82057c1746652 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:04:15.753382 10920 leader_election.cc:304] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23; no voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda
I20260709 13:04:15.657215 11100 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7757ecbf402e47bfbc93d0447741e26c" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:15.632196 12071 raft_consensus.cc:515] T bb7d67be500947b3aef4c4245e33671c P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:15.644769 12064 raft_consensus.cc:515] T bb7d67be500947b3aef4c4245e33671c P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:15.776459 11027 raft_consensus.cc:1240] T 868e92339c2f4508af0afbd07c814d64 P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Rejecting Update request from peer 0172fed22b6b4890a7fdc953fa128af4 for earlier term 2. Current term is 3. Ops: []
I20260709 13:04:15.640689 12219 raft_consensus.cc:515] T 32147c6d2d5a477aa38c597c7f78df5b P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:15.725085 12088 raft_consensus.cc:493] T 4b37f8fca8dc42718b2a42f0883a956f P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:15.769510 10879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a69f32c53f24a50b7852d143665d7b4" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:15.786331 11102 raft_consensus.cc:2468] T c31881534c5e4f0089c82057c1746652 P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 4.
I20260709 13:04:15.770952 10920 leader_election.cc:304] T a64b7e99d6c54934a7de6f6dffcf5e94 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23; no voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47
I20260709 13:04:15.745671 11921 raft_consensus.cc:2749] T 1e4f2241a94a41f38e481137f868b83d P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:04:15.656081 12053 consensus_queue.cc:1059] T a1ec2b50c6ad4f7c8033476208abb26d P 873357742d5843579556ee8166f23f47 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.108s
I20260709 13:04:15.763715 11069 leader_election.cc:304] T 4f9eb06ed01f4ce68be3af8a8434665b P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda; no voters: 
I20260709 13:04:15.750075 11101 raft_consensus.cc:2468] T 87cd8f5097624c8692f1d83544d8881e P 0172fed22b6b4890a7fdc953fa128af4 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 5.
I20260709 13:04:15.782275 10878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c31881534c5e4f0089c82057c1746652" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:04:15.648079 12073 raft_consensus.cc:515] T b67eff0634e540eab91cf8cee795969f P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:15.656176 11795 raft_consensus.cc:515] T c31881534c5e4f0089c82057c1746652 P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:15.757588 12104 raft_consensus.cc:2804] T e5a5f5a608504650b48fd27f163faed6 P 186b12638a87410487467df721a3cbda [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:04:15.785589 10880 raft_consensus.cc:2393] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0172fed22b6b4890a7fdc953fa128af4 in current term 4: Already voted for candidate 873357742d5843579556ee8166f23f47 in this term.
W20260709 13:04:15.750349 11098 consensus_meta.cc:287] T ff13ae9a47c7455f962406c6d131a3ff P 0172fed22b6b4890a7fdc953fa128af4: Time spent flushing consensus metadata: real 0.778s	user 0.013s	sys 0.003s
I20260709 13:04:15.811623 10845 leader_election.cc:304] T 87cd8f5097624c8692f1d83544d8881e P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda; no voters: 
W20260709 13:04:15.780520 10992 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 85474.2us
I20260709 13:04:15.751253 12066 consensus_queue.cc:1048] T 0e5d327630e5438b878035d28efe6c0c P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 0, Time since last communication: 0.038s
I20260709 13:04:15.761022 12004 raft_consensus.cc:2804] T a1ec2b50c6ad4f7c8033476208abb26d P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:04:15.791412 11028 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c160ddfe9b34bdaa3835d0915c1e06f" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:15.785666 11095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "401a9cf11eac479aa445ec8c1d6e33b2" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:04:15.792595 10920 leader_election.cc:304] T d7ce47b9a92d43dd88d95bcda5038597 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 10ca5ee68ad64b3283b9d235352c9d23; no voters: 
I20260709 13:04:15.761147 12048 raft_consensus.cc:515] T a5299b8d044f41b7b68057e63d17b159 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:15.793007 11027 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4cd3d7dfbbf4096908ff11e9dc57141" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:15.805778 10879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c31881534c5e4f0089c82057c1746652" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:04:15.811875 11101 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c160ddfe9b34bdaa3835d0915c1e06f" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:15.927336 11101 raft_consensus.cc:2468] T 8c160ddfe9b34bdaa3835d0915c1e06f P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 3.
I20260709 13:04:15.928752 11101 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b67eff0634e540eab91cf8cee795969f" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:15.929553 11101 raft_consensus.cc:2468] T b67eff0634e540eab91cf8cee795969f P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 2.
I20260709 13:04:15.766757 11022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f371d2565dd84667b5613848604949bb" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
I20260709 13:04:15.805181 11021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a610d1a71f52432e88fe56130f921fa2" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:15.761194 12225 raft_consensus.cc:2749] T d270b9e5481245d9b5340a454a7cd10b P 10ca5ee68ad64b3283b9d235352c9d23 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 13:04:15.770246 12049 raft_consensus.cc:493] T a5299b8d044f41b7b68057e63d17b159 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:15.767999 12223 leader_election.cc:290] T 87cd8f5097624c8692f1d83544d8881e P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 5 election: Requested vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:15.832564 11028 raft_consensus.cc:2410] T 8c160ddfe9b34bdaa3835d0915c1e06f P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0172fed22b6b4890a7fdc953fa128af4 for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 13:04:15.832765 10876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d371c5e27bd843469d2e2e453734cb4a" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
W20260709 13:04:15.772380 10918 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:15.796126 12088 raft_consensus.cc:515] T 4b37f8fca8dc42718b2a42f0883a956f P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:15.783907 12196 raft_consensus.cc:3060] T 34064dc7bf3747dbb3876dd824ee0688 P 873357742d5843579556ee8166f23f47 [term 5 FOLLOWER]: Advancing to term 6
W20260709 13:04:15.863996 10919 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:15.826285 11098 raft_consensus.cc:2468] T ff13ae9a47c7455f962406c6d131a3ff P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 4.
W20260709 13:04:15.904553 10920 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 112041us
I20260709 13:04:15.727192 12202 raft_consensus.cc:515] T 6a9353d1d230416296627d7cf8b1d425 P 873357742d5843579556ee8166f23f47 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:15.812973 12198 consensus_queue.cc:237] T e5ba72ccd854430baadbc636dfd17136 P 873357742d5843579556ee8166f23f47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:15.794333 12250 leader_election.cc:290] T 11de93cd9b9d457489ff1cb80f976c9a P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
W20260709 13:04:15.962102 12279 log.cc:927] Time spent T 7709387f4d3d4f9992d97a0fbca39a40 P 186b12638a87410487467df721a3cbda: Append to log took a long time: real 0.200s	user 0.001s	sys 0.000s
I20260709 13:04:15.938468 11099 raft_consensus.cc:1240] T f34f2bc02d85444f9096e0f5b4fe4331 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Rejecting Update request from peer 10ca5ee68ad64b3283b9d235352c9d23 for earlier term 1. Current term is 2. Ops: [1.1-1.1]
I20260709 13:04:15.926311 10951 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49434 (request call id 273) took 2094 ms. Trace:
I20260709 13:04:15.931468 12004 raft_consensus.cc:697] T a1ec2b50c6ad4f7c8033476208abb26d P 0172fed22b6b4890a7fdc953fa128af4 [term 3 LEADER]: Becoming Leader. State: Replica: 0172fed22b6b4890a7fdc953fa128af4, State: Running, Role: LEADER
I20260709 13:04:15.936211 11027 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75492f86fde046c3900df3eac8292fd4" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
I20260709 13:04:15.811460 12193 raft_consensus.cc:2804] T 4f9eb06ed01f4ce68be3af8a8434665b P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:04:15.943810 10876 raft_consensus.cc:3060] T d371c5e27bd843469d2e2e453734cb4a P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:04:15.931352 10845 leader_election.cc:304] T 8c160ddfe9b34bdaa3835d0915c1e06f P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda; no voters: 
I20260709 13:04:15.923389 11100 raft_consensus.cc:2410] T 7757ecbf402e47bfbc93d0447741e26c P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 186b12638a87410487467df721a3cbda for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 13:04:15.830529 12288 raft_consensus.cc:2749] T a64b7e99d6c54934a7de6f6dffcf5e94 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:04:15.905288 12053 consensus_queue.cc:1243] T a1ec2b50c6ad4f7c8033476208abb26d P 873357742d5843579556ee8166f23f47 [LEADER]: Peer 0172fed22b6b4890a7fdc953fa128af4 log is divergent from this leader: its last log entry 1.1 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 0
I20260709 13:04:15.955148 11028 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1ec2b50c6ad4f7c8033476208abb26d" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:15.925763 11097 raft_consensus.cc:1275] T 5de5f0581e3b41dea729363d6ca563bb P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Refusing update from remote peer 10ca5ee68ad64b3283b9d235352c9d23: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 13:04:15.962320 12202 leader_election.cc:290] T 6a9353d1d230416296627d7cf8b1d425 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 5 election: Requested vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:15.793850 12283 raft_consensus.cc:2749] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 13:04:15.932097 10878 raft_consensus.cc:2393] T c31881534c5e4f0089c82057c1746652 P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 0172fed22b6b4890a7fdc953fa128af4 in current term 3: Already voted for candidate 186b12638a87410487467df721a3cbda in this term.
I20260709 13:04:15.963153 10947 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d439326f23747eaab3c1e7da2d6b785" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:04:15.964147 10953 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11de93cd9b9d457489ff1cb80f976c9a" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:15.901942 11926 raft_consensus.cc:493] T 9a1716c0adf34bee8a4e99341ba5788f P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:15.764946 12292 raft_consensus.cc:515] T e96e80ba9ac34d0998395515e5c8c41a P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:15.940200 11095 raft_consensus.cc:3060] T 401a9cf11eac479aa445ec8c1d6e33b2 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Advancing to term 3
W20260709 13:04:15.952915 12272 log.cc:927] Time spent T 7839ca2d2c13431a85f44d7c458c6351 P 186b12638a87410487467df721a3cbda: Append to log took a long time: real 0.423s	user 0.000s	sys 0.001s
I20260709 13:04:15.975195 11098 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37676 (request call id 292) took 1006 ms. Trace:
I20260709 13:04:15.953490 12049 raft_consensus.cc:515] T a5299b8d044f41b7b68057e63d17b159 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:15.994957 11027 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f1466124dbd4f6ebb3c4d62ca48bf96" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
I20260709 13:04:15.958520 11937 leader_election.cc:290] T 525bfdbd6d934afd823f7d3ee3f9d052 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:15.999327 11028 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1ec2b50c6ad4f7c8033476208abb26d" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
I20260709 13:04:15.717847 10875 rpcz_store.cc:276] 0709 13:04:14.413339 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:14.839648 (+426309us) service_pool.cc:224] Handling call
0709 13:04:14.840170 (+   522us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:14.840718 (+   548us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:14.840754 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:14.841537 (+   783us) log.cc:844] Serialized 26 byte log entry
0709 13:04:15.716771 (+875234us) raft_consensus.cc:1645] Marking committed up to 1
0709 13:04:15.716839 (+    68us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:15.716930 (+    91us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:15.716980 (+    50us) raft_consensus.cc:1685] finished
0709 13:04:15.716995 (+    15us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:15.717146 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":123,"raft.run_cpu_time_us":112,"raft.run_wall_time_us":113,"thread_start_us":874663,"threads_started":1,"wal-append.queue_time_us":844380}
I20260709 13:04:15.959165 12068 raft_consensus.cc:2749] T 5d439326f23747eaab3c1e7da2d6b785 P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:04:16.011183 10953 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87cd8f5097624c8692f1d83544d8881e" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:04:15.937669 12295 consensus_queue.cc:1048] T d7ce47b9a92d43dd88d95bcda5038597 P 186b12638a87410487467df721a3cbda [LEADER]: Connected to new peer: Peer: permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.159s
W20260709 13:04:15.985110 10845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 53887.5us
I20260709 13:04:15.934630 10879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d7ce47b9a92d43dd88d95bcda5038597" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:15.980583 12223 raft_consensus.cc:2804] T 87cd8f5097624c8692f1d83544d8881e P 186b12638a87410487467df721a3cbda [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:04:15.963235 12250 consensus_queue.cc:1048] T 0f9df2bc281f4866b6da5da94a0defc7 P 186b12638a87410487467df721a3cbda [LEADER]: Connected to new peer: Peer: permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.040s
I20260709 13:04:15.977939 12299 raft_consensus.cc:2804] T 4f9eb06ed01f4ce68be3af8a8434665b P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:04:15.984028 12193 raft_consensus.cc:493] T 4f9eb06ed01f4ce68be3af8a8434665b P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 186b12638a87410487467df721a3cbda)
I20260709 13:04:15.988852 12200 leader_election.cc:290] T 0f9df2bc281f4866b6da5da94a0defc7 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
W20260709 13:04:16.021787 12277 log.cc:927] Time spent T a977842ba4b744f4bb2febd55d286006 P 10ca5ee68ad64b3283b9d235352c9d23: Append to log took a long time: real 0.379s	user 0.001s	sys 0.001s
W20260709 13:04:16.034798 10992 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 217738us
W20260709 13:04:16.029695 11067 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:45807, user_credentials={real_user=slave}} blocked reactor thread for 136940us
I20260709 13:04:16.059610 11028 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7757ecbf402e47bfbc93d0447741e26c" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
W20260709 13:04:16.039971 10993 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:16.063903 10953 raft_consensus.cc:3060] T 87cd8f5097624c8692f1d83544d8881e P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:04:16.066885 10845 leader_election.cc:304] T b67eff0634e540eab91cf8cee795969f P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda; no voters: 
W20260709 13:04:16.062758 10846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:15.633307 10874 rpcz_store.cc:276] 0709 13:04:14.409505 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:14.420673 (+ 11168us) service_pool.cc:224] Handling call
0709 13:04:14.564401 (+143728us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:14.620394 (+ 55993us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:14.628853 (+  8459us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:14.757578 (+128725us) log.cc:844] Serialized 26 byte log entry
0709 13:04:15.299615 (+542037us) raft_consensus.cc:1645] Marking committed up to 0
0709 13:04:15.306490 (+  6875us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:15.327173 (+ 20683us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:15.330166 (+  2993us) raft_consensus.cc:1685] finished
0709 13:04:15.332447 (+  2281us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:15.431453 (+ 99006us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":4595,"raft.run_cpu_time_us":182,"raft.run_wall_time_us":10284,"thread_start_us":497564,"threads_started":1,"wal-append.queue_time_us":431285}
I20260709 13:04:16.029686 12286 raft_consensus.cc:2804] T d7ce47b9a92d43dd88d95bcda5038597 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:04:16.031491 11972 leader_election.cc:290] T bb7d67be500947b3aef4c4245e33671c P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:16.127048 10876 raft_consensus.cc:2468] T d371c5e27bd843469d2e2e453734cb4a P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 4.
I20260709 13:04:16.128882 10876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f9df2bc281f4866b6da5da94a0defc7" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:16.167979 11027 raft_consensus.cc:2468] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 873357742d5843579556ee8166f23f47 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 5.
I20260709 13:04:16.188657 11027 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555217d45d9b4ee492dbb3ddaf0f7c02" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
W20260709 13:04:16.062371 12218 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.585s	user 0.006s	sys 0.006s
I20260709 13:04:16.200906 11028 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d371c5e27bd843469d2e2e453734cb4a" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
I20260709 13:04:16.026234 11926 raft_consensus.cc:515] T 9a1716c0adf34bee8a4e99341ba5788f P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:16.061448 11937 raft_consensus.cc:493] T 6f46a5102fd64b45bf1c4dff863d2b4d P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 873357742d5843579556ee8166f23f47)
W20260709 13:04:16.201740 10992 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 77064.4us
I20260709 13:04:16.311208 10871 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51718 (request call id 305) took 1591 ms. Trace:
I20260709 13:04:16.075203 12066 raft_consensus.cc:493] T f371d2565dd84667b5613848604949bb P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:16.072073 11021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f9eb06ed01f4ce68be3af8a8434665b" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:16.074381 12104 raft_consensus.cc:697] T e5a5f5a608504650b48fd27f163faed6 P 186b12638a87410487467df721a3cbda [term 5 LEADER]: Becoming Leader. State: Replica: 186b12638a87410487467df721a3cbda, State: Running, Role: LEADER
I20260709 13:04:16.111023 12193 raft_consensus.cc:3060] T 4f9eb06ed01f4ce68be3af8a8434665b P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:04:16.109470 12299 raft_consensus.cc:493] T 4f9eb06ed01f4ce68be3af8a8434665b P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:16.029033 11098 rpcz_store.cc:276] 0709 13:04:14.968845 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:14.970106 (+  1261us) service_pool.cc:224] Handling call
0709 13:04:15.971303 (+1001197us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:16.119319 12223 raft_consensus.cc:697] T 87cd8f5097624c8692f1d83544d8881e P 186b12638a87410487467df721a3cbda [term 5 LEADER]: Becoming Leader. State: Replica: 186b12638a87410487467df721a3cbda, State: Running, Role: LEADER
W20260709 13:04:16.256822 12284 log.cc:927] Time spent T 0f9df2bc281f4866b6da5da94a0defc7 P 10ca5ee68ad64b3283b9d235352c9d23: Append to log took a long time: real 0.451s	user 0.000s	sys 0.002s
I20260709 13:04:16.199182 12296 raft_consensus.cc:2804] T 8c160ddfe9b34bdaa3835d0915c1e06f P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:04:16.212716 10873 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51704 (request call id 228) took 1503 ms. Trace:
I20260709 13:04:16.221354 12068 raft_consensus.cc:493] T e89e640169e14e2d98f968f76027daa7 P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:16.279018 12172 leader_election.cc:290] T 30f628e3d7914d609e7f1d980a74c654 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
W20260709 13:04:16.343714 10994 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.710s	user 0.011s	sys 0.010s
W20260709 13:04:16.417023 10994 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.783s	user 0.011s	sys 0.011s
I20260709 13:04:16.378883 12073 leader_election.cc:290] T b67eff0634e540eab91cf8cee795969f P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:16.130915 12004 consensus_queue.cc:237] T a1ec2b50c6ad4f7c8033476208abb26d P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:16.304752 12288 raft_consensus.cc:493] T 8a69f32c53f24a50b7852d143665d7b4 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 186b12638a87410487467df721a3cbda)
I20260709 13:04:16.372913 11028 raft_consensus.cc:3060] T d371c5e27bd843469d2e2e453734cb4a P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Advancing to term 4
W20260709 13:04:16.201751 10877 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.597s	user 0.038s	sys 0.081s
W20260709 13:04:16.347040 12269 log.cc:927] Time spent T 5de5f0581e3b41dea729363d6ca563bb P 186b12638a87410487467df721a3cbda: Append to log took a long time: real 0.057s	user 0.001s	sys 0.000s
W20260709 13:04:16.422742 10877 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.821s	user 0.038s	sys 0.081s
I20260709 13:04:16.318776 12064 leader_election.cc:290] T bb7d67be500947b3aef4c4245e33671c P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:16.343134 12224 leader_election.cc:290] T 5de5f0581e3b41dea729363d6ca563bb P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:16.340660 12096 raft_consensus.cc:493] T acc241e7273d49149984968596242712 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 873357742d5843579556ee8166f23f47)
W20260709 13:04:16.378391 12218 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.904s	user 0.007s	sys 0.006s
W20260709 13:04:16.415798 12293 log.cc:927] Time spent T 57b6a2418c1342e7924195e9c1dbd357 P 10ca5ee68ad64b3283b9d235352c9d23: Append to log took a long time: real 0.429s	user 0.001s	sys 0.000s
I20260709 13:04:16.402158 11027 raft_consensus.cc:2393] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0172fed22b6b4890a7fdc953fa128af4 in current term 4: Already voted for candidate 873357742d5843579556ee8166f23f47 in this term.
I20260709 13:04:16.367434 11795 leader_election.cc:290] T c31881534c5e4f0089c82057c1746652 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:16.357224 12295 raft_consensus.cc:493] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 0172fed22b6b4890a7fdc953fa128af4)
I20260709 13:04:16.367134 12048 leader_election.cc:290] T a5299b8d044f41b7b68057e63d17b159 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:16.367087 12071 leader_election.cc:290] T bb7d67be500947b3aef4c4245e33671c P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:16.413622 10946 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49424 (request call id 233) took 1358 ms. Trace:
I20260709 13:04:16.414142 12223 consensus_queue.cc:237] T 87cd8f5097624c8692f1d83544d8881e P 186b12638a87410487467df721a3cbda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:16.405483 12296 raft_consensus.cc:493] T 8c160ddfe9b34bdaa3835d0915c1e06f P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:16.397579 12299 raft_consensus.cc:3060] T 4f9eb06ed01f4ce68be3af8a8434665b P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:04:16.381181 11921 raft_consensus.cc:493] T 0e5d327630e5438b878035d28efe6c0c P 873357742d5843579556ee8166f23f47 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:16.396080 12302 log.cc:927] Time spent T d270b9e5481245d9b5340a454a7cd10b P 0172fed22b6b4890a7fdc953fa128af4: Append to log took a long time: real 0.393s	user 0.001s	sys 0.001s
I20260709 13:04:16.300078 12095 leader_election.cc:290] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:16.300575 10874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5de5f0581e3b41dea729363d6ca563bb" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:16.432139 10952 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49428 (request call id 194) took 1134 ms. Trace:
W20260709 13:04:16.417578 10994 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 784187us
I20260709 13:04:16.406822 12219 leader_election.cc:290] T 32147c6d2d5a477aa38c597c7f78df5b P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:16.437059 11921 raft_consensus.cc:515] T 0e5d327630e5438b878035d28efe6c0c P 873357742d5843579556ee8166f23f47 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:16.433139 12295 raft_consensus.cc:515] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:16.462565 11094 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37678 (request call id 152) took 1376 ms. Trace:
I20260709 13:04:16.503589 11094 rpcz_store.cc:276] 0709 13:04:15.088054 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:15.090636 (+  2582us) service_pool.cc:224] Handling call
0709 13:04:15.261831 (+171195us) spinlock_profiling.cc:243] Waited 11.4 ms on lock 0x7b5800117698. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f426303db71 00007f426303deee 0000000000375939 00007f426b0b11e0 00007f426b17a0c4 00007f426b17a131 00007f426b13bfc0 00007f426b126929 00007f426b1263d2 00007f426469dbd4
0709 13:04:15.262040 (+   209us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:04:15.521439 (+259399us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:15.523219 (+  1780us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:04:15.652266 (+129047us) log.cc:844] Serialized 52 byte log entry
0709 13:04:16.022473 (+370207us) raft_consensus.cc:1645] Marking committed up to 1
0709 13:04:16.028235 (+  5762us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:16.030959 (+  2724us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:16.462142 (+431183us) raft_consensus.cc:1685] finished
0709 13:04:16.462178 (+    36us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:16.462480 (+   302us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":8434,"raft.run_cpu_time_us":130,"raft.run_wall_time_us":129,"spinlock_wait_cycles":26214784,"thread_start_us":404814,"threads_started":2,"wal-append.queue_time_us":225767}
I20260709 13:04:16.505203 12295 leader_election.cc:290] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:16.432737 11795 raft_consensus.cc:493] T f34f2bc02d85444f9096e0f5b4fe4331 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 10ca5ee68ad64b3283b9d235352c9d23)
I20260709 13:04:16.506155 11795 raft_consensus.cc:515] T f34f2bc02d85444f9096e0f5b4fe4331 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:16.433534 10953 raft_consensus.cc:2468] T 87cd8f5097624c8692f1d83544d8881e P 10ca5ee68ad64b3283b9d235352c9d23 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 5.
I20260709 13:04:16.519321 10953 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c2a35bb99374e93b573cf870d04faaa" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:16.393711 12066 raft_consensus.cc:515] T f371d2565dd84667b5613848604949bb P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:16.491828 10955 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49428 (request call id 191) took 1251 ms. Trace:
I20260709 13:04:16.427438 12068 raft_consensus.cc:515] T e89e640169e14e2d98f968f76027daa7 P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:16.440032 12288 raft_consensus.cc:515] T 8a69f32c53f24a50b7852d143665d7b4 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:16.382547 11937 raft_consensus.cc:515] T 6f46a5102fd64b45bf1c4dff863d2b4d P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:16.433872 10946 rpcz_store.cc:276] 0709 13:04:15.054849 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:15.060274 (+  5425us) service_pool.cc:224] Handling call
0709 13:04:15.060630 (+   356us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:15.236688 (+176058us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:15.236742 (+    54us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:15.237348 (+   606us) log.cc:844] Serialized 26 byte log entry
0709 13:04:15.474159 (+236811us) raft_consensus.cc:1645] Marking committed up to 0
0709 13:04:15.477298 (+  3139us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:15.507162 (+ 29864us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:15.563838 (+ 56676us) spinlock_profiling.cc:243] Waited 45.5 ms on lock 0x7b48002004d8. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426b08baa1 00007f426b082b37 00007f426b0829ca 00007f426b17a26f 00007f426b179f37 00007f426b18e55c 00007f426b18e508 00007f426b18e488 00007f426b18e443 00007f426b18d6ec 00007f426a0b7a96
0709 13:04:16.401246 (+837408us) raft_consensus.cc:1685] finished
0709 13:04:16.401282 (+    36us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:16.413510 (+ 12228us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":151407,"raft.queue_time_us":177375,"raft.run_cpu_time_us":3216,"raft.run_wall_time_us":325523,"spinlock_wait_cycles":104701824,"thread_start_us":184236,"threads_started":1,"wal-append.queue_time_us":322822}
I20260709 13:04:16.410272 10873 rpcz_store.cc:276] 0709 13:04:14.709761 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:14.882209 (+172448us) service_pool.cc:224] Handling call
0709 13:04:14.938171 (+ 55962us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:14.951930 (+ 13759us) raft_consensus.cc:1534] Early marking committed up to index 3
0709 13:04:15.238289 (+286359us) log.cc:844] Serialized 14 byte log entry
0709 13:04:15.502738 (+264449us) log.cc:844] Serialized 14 byte log entry
0709 13:04:15.557371 (+ 54633us) spinlock_profiling.cc:243] Waited 17.5 ms on lock 0x7b180017efc0. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f42630753f3 00007f42630777c1 00007f4263077779 00007f4263077709 00007f42630776d1 00007f42630769cd 000000000038f354 000000000038f199 00007f426305883a
0709 13:04:15.568979 (+ 11608us) spinlock_profiling.cc:243] Waited 13.8 ms on lock 0x7b180017efc0. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f4263057e07 00007f426beb2ab9 00007f426307469b 00007f426307293f 00007f4263072455 00007f426615a060 00007f426616c716 00007f426617d287 00007f426617d208
0709 13:04:16.060857 (+491878us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:16.060969 (+   112us) raft_consensus.cc:1645] Marking committed up to 3
0709 13:04:16.074243 (+ 13274us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:16.107876 (+ 33633us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:16.206841 (+ 98965us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":258645,"prepare.run_cpu_time_us":82,"prepare.run_wall_time_us":82,"raft.queue_time_us":25920,"raft.run_cpu_time_us":308,"raft.run_wall_time_us":53765,"spinlock_wait_cycles":72137216,"thread_start_us":955628,"threads_started":3,"wal-append.queue_time_us":411774}
I20260709 13:04:16.464395 10952 rpcz_store.cc:276] 0709 13:04:15.297905 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:15.539636 (+241731us) service_pool.cc:224] Handling call
0709 13:04:15.540109 (+   473us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:15.540514 (+   405us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:15.540548 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:15.541201 (+   653us) log.cc:844] Serialized 26 byte log entry
0709 13:04:15.789899 (+248698us) raft_consensus.cc:1645] Marking committed up to 1
0709 13:04:15.805541 (+ 15642us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:15.815217 (+  9676us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:16.431888 (+616671us) raft_consensus.cc:1685] finished
0709 13:04:16.431909 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:16.432067 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1030,"raft.run_cpu_time_us":97,"raft.run_wall_time_us":97,"spinlock_wait_cycles":24448,"thread_start_us":232820,"threads_started":1,"wal-append.queue_time_us":388528}
I20260709 13:04:16.520665 12292 leader_election.cc:290] T e96e80ba9ac34d0998395515e5c8c41a P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:16.499804 10994 leader_election.cc:304] T ff13ae9a47c7455f962406c6d131a3ff P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47; no voters: 
W20260709 13:04:16.476711 12127 consensus_meta.cc:287] T 732c4adeeff6422497aa7db4a82f535b P 10ca5ee68ad64b3283b9d235352c9d23: Time spent flushing consensus metadata: real 1.211s	user 0.012s	sys 0.005s
I20260709 13:04:16.463518 12104 consensus_queue.cc:237] T e5a5f5a608504650b48fd27f163faed6 P 186b12638a87410487467df721a3cbda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:16.521025 10953 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "525bfdbd6d934afd823f7d3ee3f9d052" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
W20260709 13:04:16.430660 12301 log.cc:927] Time spent T c51782cd9d064ec0949e84e24dd4de67 P 186b12638a87410487467df721a3cbda: Append to log took a long time: real 0.211s	user 0.000s	sys 0.001s
I20260709 13:04:16.441056 11021 raft_consensus.cc:2393] T 4f9eb06ed01f4ce68be3af8a8434665b P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0172fed22b6b4890a7fdc953fa128af4 in current term 3: Already voted for candidate 873357742d5843579556ee8166f23f47 in this term.
I20260709 13:04:15.977486 10951 rpcz_store.cc:276] 0709 13:04:13.807061 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:13.812616 (+  5555us) service_pool.cc:224] Handling call
0709 13:04:14.898656 (+1086040us) spinlock_profiling.cc:243] Waited 7.91 ms on lock 0x7b580001d198. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f426303db71 00007f426303da22 00007f426bf56c09 00007f426b0b0ee0 00007f426b17a0ab 00007f426b17a131 00007f426b13bfc0 00007f426b126929 00007f426b1263d2 00007f426469dbd4
0709 13:04:14.901223 (+  2567us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:14.940166 (+ 38943us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:14.942333 (+  2167us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:14.946611 (+  4278us) log.cc:844] Serialized 26 byte log entry
0709 13:04:15.214422 (+267811us) raft_consensus.cc:1645] Marking committed up to 1
0709 13:04:15.215027 (+   605us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:15.215135 (+   108us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:15.797939 (+582804us) raft_consensus.cc:1685] finished
0709 13:04:15.805609 (+  7670us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:15.901349 (+ 95740us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":34074,"raft.queue_time_us":37416,"raft.run_cpu_time_us":153,"raft.run_wall_time_us":4775,"spinlock_wait_cycles":18189696,"thread_start_us":267460,"threads_started":1,"wal-append.queue_time_us":368196}
I20260709 13:04:16.422415 12283 raft_consensus.cc:493] T e89e640169e14e2d98f968f76027daa7 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:16.429844 12096 raft_consensus.cc:515] T acc241e7273d49149984968596242712 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:16.525431 10955 rpcz_store.cc:276] 0709 13:04:15.240442 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:15.280502 (+ 40060us) service_pool.cc:224] Handling call
0709 13:04:15.280903 (+   401us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:15.281160 (+   257us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:15.281184 (+    24us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:15.281659 (+   475us) log.cc:844] Serialized 26 byte log entry
0709 13:04:15.548290 (+266631us) raft_consensus.cc:1645] Marking committed up to 1
0709 13:04:15.548328 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:15.548408 (+    80us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:16.491539 (+943131us) raft_consensus.cc:1685] finished
0709 13:04:16.491573 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:16.491748 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":221453,"threads_started":1,"wal-append.queue_time_us":339419}
I20260709 13:04:16.423063 12088 leader_election.cc:290] T 4b37f8fca8dc42718b2a42f0883a956f P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:16.432003 12218 raft_consensus.cc:3055] T 75492f86fde046c3900df3eac8292fd4 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 LEADER]: Stepping down as leader of term 2
W20260709 13:04:16.547346 10918 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:16.447621 11027 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "525bfdbd6d934afd823f7d3ee3f9d052" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:16.434829 12296 raft_consensus.cc:3060] T 8c160ddfe9b34bdaa3835d0915c1e06f P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:04:16.542953 11028 raft_consensus.cc:2468] T d371c5e27bd843469d2e2e453734cb4a P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 4.
I20260709 13:04:16.424230 12313 raft_consensus.cc:493] T 4b37f8fca8dc42718b2a42f0883a956f P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:16.423709 12310 raft_consensus.cc:2804] T b67eff0634e540eab91cf8cee795969f P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:04:16.505244 11921 leader_election.cc:290] T 0e5d327630e5438b878035d28efe6c0c P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:16.508620 12250 raft_consensus.cc:493] T 63d44f6ba6434ab2870fe3190f4d7c2f P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 10ca5ee68ad64b3283b9d235352c9d23)
W20260709 13:04:16.443959 12282 consensus_meta.cc:287] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 0172fed22b6b4890a7fdc953fa128af4: Time spent flushing consensus metadata: real 0.766s	user 0.019s	sys 0.000s
W20260709 13:04:16.564883 10993 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:16.484524 11926 leader_election.cc:290] T 9a1716c0adf34bee8a4e99341ba5788f P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:16.517937 12071 raft_consensus.cc:493] T 34064dc7bf3747dbb3876dd824ee0688 P 10ca5ee68ad64b3283b9d235352c9d23 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:16.518608 12219 raft_consensus.cc:493] T a610d1a71f52432e88fe56130f921fa2 P 0172fed22b6b4890a7fdc953fa128af4 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 873357742d5843579556ee8166f23f47)
W20260709 13:04:16.577723 10994 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 78995us
I20260709 13:04:16.603960 10953 raft_consensus.cc:2393] T 525bfdbd6d934afd823f7d3ee3f9d052 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0172fed22b6b4890a7fdc953fa128af4 in current term 4: Already voted for candidate 873357742d5843579556ee8166f23f47 in this term.
I20260709 13:04:16.623309 10955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b67eff0634e540eab91cf8cee795969f" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:16.704797 10955 raft_consensus.cc:2468] T b67eff0634e540eab91cf8cee795969f P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 2.
I20260709 13:04:16.625289 11027 raft_consensus.cc:2393] T 525bfdbd6d934afd823f7d3ee3f9d052 P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0172fed22b6b4890a7fdc953fa128af4 in current term 4: Already voted for candidate 873357742d5843579556ee8166f23f47 in this term.
I20260709 13:04:16.661552 11028 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb7d67be500947b3aef4c4245e33671c" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:16.709108 11028 raft_consensus.cc:2468] T bb7d67be500947b3aef4c4245e33671c P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 2.
I20260709 13:04:16.710507 11028 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d71ca3a1ba6b48478aaf62741bf3cf6a" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
W20260709 13:04:16.568060 10845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 501293us
I20260709 13:04:16.711920 11028 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32147c6d2d5a477aa38c597c7f78df5b" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:16.712158 10845 leader_election.cc:304] T 7757ecbf402e47bfbc93d0447741e26c P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 186b12638a87410487467df721a3cbda; no voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47
W20260709 13:04:16.673597 10844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:45807, user_credentials={real_user=slave}} blocked reactor thread for 301712us
I20260709 13:04:16.755234 12088 consensus_queue.cc:1048] T 732c4adeeff6422497aa7db4a82f535b P 186b12638a87410487467df721a3cbda [LEADER]: Connected to new peer: Peer: permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:04:16.757808 12088 raft_consensus.cc:2749] T 7757ecbf402e47bfbc93d0447741e26c P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:04:16.755254 10844 leader_election.cc:304] T a610d1a71f52432e88fe56130f921fa2 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 186b12638a87410487467df721a3cbda; no voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47
W20260709 13:04:16.674314 12240 consensus_meta.cc:287] T 63d44f6ba6434ab2870fe3190f4d7c2f P 10ca5ee68ad64b3283b9d235352c9d23: Time spent flushing consensus metadata: real 1.108s	user 0.017s	sys 0.000s
W20260709 13:04:16.614671 11974 consensus_meta.cc:287] T 836251ebc2b74f5da88b8a8b2338e2a5 P 10ca5ee68ad64b3283b9d235352c9d23: Time spent flushing consensus metadata: real 0.894s	user 0.011s	sys 0.006s
W20260709 13:04:16.673514 12215 consensus_meta.cc:287] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 10ca5ee68ad64b3283b9d235352c9d23: Time spent flushing consensus metadata: real 1.017s	user 0.006s	sys 0.011s
W20260709 13:04:16.641124 12103 consensus_meta.cc:287] T a977842ba4b744f4bb2febd55d286006 P 873357742d5843579556ee8166f23f47: Time spent flushing consensus metadata: real 1.000s	user 0.016s	sys 0.000s
W20260709 13:04:16.633556 11653 consensus_meta.cc:287] T 32147c6d2d5a477aa38c597c7f78df5b P 873357742d5843579556ee8166f23f47: Time spent flushing consensus metadata: real 1.303s	user 0.017s	sys 0.000s
W20260709 13:04:16.613783 12105 consensus_meta.cc:287] T 5d439326f23747eaab3c1e7da2d6b785 P 10ca5ee68ad64b3283b9d235352c9d23: Time spent flushing consensus metadata: real 1.281s	user 0.013s	sys 0.003s
I20260709 13:04:16.644694 10874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f9eb06ed01f4ce68be3af8a8434665b" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
W20260709 13:04:16.779656 12196 consensus_meta.cc:287] T 34064dc7bf3747dbb3876dd824ee0688 P 873357742d5843579556ee8166f23f47: Time spent flushing consensus metadata: real 0.808s	user 0.008s	sys 0.008s
I20260709 13:04:16.780357 12317 raft_consensus.cc:493] T 2567c2d84e4648d7ba5dc23a3df32817 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 10ca5ee68ad64b3283b9d235352c9d23)
I20260709 13:04:16.781190 12317 raft_consensus.cc:515] T 2567c2d84e4648d7ba5dc23a3df32817 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:16.784005 12317 leader_election.cc:290] T 2567c2d84e4648d7ba5dc23a3df32817 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:16.785249 11961 raft_consensus.cc:493] T 32147c6d2d5a477aa38c597c7f78df5b P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:16.785585 12308 raft_consensus.cc:493] T 65bf0a62f4d84ef18dd6b23e082724b8 P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 10ca5ee68ad64b3283b9d235352c9d23)
I20260709 13:04:16.785772 12240 raft_consensus.cc:493] T e5ba72ccd854430baadbc636dfd17136 P 10ca5ee68ad64b3283b9d235352c9d23 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:16.786032 11961 raft_consensus.cc:515] T 32147c6d2d5a477aa38c597c7f78df5b P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:16.786715 11974 raft_consensus.cc:493] T 7839ca2d2c13431a85f44d7c458c6351 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:16.378149 10871 rpcz_store.cc:276] 0709 13:04:14.714186 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:14.883395 (+169209us) service_pool.cc:224] Handling call
0709 13:04:14.883783 (+   388us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:14.884204 (+   421us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:14.884236 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:14.884853 (+   617us) log.cc:844] Serialized 26 byte log entry
0709 13:04:15.330683 (+445830us) raft_consensus.cc:1645] Marking committed up to 1
0709 13:04:15.381608 (+ 50925us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:15.417536 (+ 35928us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:16.262796 (+845260us) raft_consensus.cc:1685] finished
0709 13:04:16.262836 (+    40us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:16.305200 (+ 42364us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1579,"raft.run_cpu_time_us":203,"raft.run_wall_time_us":51462,"thread_start_us":415314,"threads_started":1,"wal-append.queue_time_us":535590}
I20260709 13:04:16.787338 11974 raft_consensus.cc:515] T 7839ca2d2c13431a85f44d7c458c6351 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:16.788617 10871 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c160ddfe9b34bdaa3835d0915c1e06f" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
W20260709 13:04:16.744712 12091 consensus_meta.cc:287] T d270b9e5481245d9b5340a454a7cd10b P 186b12638a87410487467df721a3cbda: Time spent flushing consensus metadata: real 1.469s	user 0.013s	sys 0.005s
W20260709 13:04:16.793471 10992 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 182204us
I20260709 13:04:16.619189 11795 leader_election.cc:290] T f34f2bc02d85444f9096e0f5b4fe4331 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:16.630337 12066 leader_election.cc:290] T f371d2565dd84667b5613848604949bb P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:16.622072 12096 leader_election.cc:290] T acc241e7273d49149984968596242712 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:16.789988 10877 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51696 (request call id 290) took 1505 ms. Trace:
I20260709 13:04:16.695928 12073 raft_consensus.cc:493] T e5a5f5a608504650b48fd27f163faed6 P 10ca5ee68ad64b3283b9d235352c9d23 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:16.623459 10952 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a69f32c53f24a50b7852d143665d7b4" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:16.645164 10873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "30f628e3d7914d609e7f1d980a74c654" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:16.623554 12202 raft_consensus.cc:2804] T ff13ae9a47c7455f962406c6d131a3ff P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:04:16.780537 12196 raft_consensus.cc:493] T 401a9cf11eac479aa445ec8c1d6e33b2 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:16.672211 12310 raft_consensus.cc:493] T b67eff0634e540eab91cf8cee795969f P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:16.620012 12292 raft_consensus.cc:493] T 6a9353d1d230416296627d7cf8b1d425 P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0172fed22b6b4890a7fdc953fa128af4)
I20260709 13:04:16.645599 10872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555217d45d9b4ee492dbb3ddaf0f7c02" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:04:16.612465 12068 leader_election.cc:290] T e89e640169e14e2d98f968f76027daa7 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:16.624102 10946 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb7d67be500947b3aef4c4245e33671c" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:16.796455 11028 raft_consensus.cc:2468] T 32147c6d2d5a477aa38c597c7f78df5b P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 3.
I20260709 13:04:16.786238 12308 raft_consensus.cc:515] T 65bf0a62f4d84ef18dd6b23e082724b8 P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:16.789160 11961 leader_election.cc:290] T 32147c6d2d5a477aa38c597c7f78df5b P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:16.617360 12288 leader_election.cc:290] T 8a69f32c53f24a50b7852d143665d7b4 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:16.623782 12218 raft_consensus.cc:740] T 75492f86fde046c3900df3eac8292fd4 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 10ca5ee68ad64b3283b9d235352c9d23, State: Running, Role: LEADER
I20260709 13:04:16.614064 11937 leader_election.cc:290] T 6f46a5102fd64b45bf1c4dff863d2b4d P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:16.607159 11021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c160ddfe9b34bdaa3835d0915c1e06f" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:16.738757 12305 raft_consensus.cc:493] T 9a1716c0adf34bee8a4e99341ba5788f P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:16.810912 10877 rpcz_store.cc:276] 0709 13:04:15.284651 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:15.569308 (+284657us) service_pool.cc:224] Handling call
0709 13:04:15.576200 (+  6892us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:16.423305 (+847105us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:16.423343 (+    38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:16.423993 (+   650us) log.cc:844] Serialized 26 byte log entry
0709 13:04:16.508027 (+ 84034us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:04:16.508085 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:16.508199 (+   114us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:16.789712 (+281513us) raft_consensus.cc:1685] finished
0709 13:04:16.789741 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:16.789914 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":95405,"raft.run_cpu_time_us":99,"raft.run_wall_time_us":99,"thread_start_us":896898,"threads_started":2,"wal-append.queue_time_us":307664}
I20260709 13:04:16.838258 10877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63d44f6ba6434ab2870fe3190f4d7c2f" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
W20260709 13:04:16.815452 10844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:45807, user_credentials={real_user=slave}} blocked reactor thread for 60311.2us
I20260709 13:04:16.674801 12313 raft_consensus.cc:515] T 4b37f8fca8dc42718b2a42f0883a956f P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:16.638639 12127 raft_consensus.cc:515] T 732c4adeeff6422497aa7db4a82f535b P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:16.686131 12071 raft_consensus.cc:515] T 34064dc7bf3747dbb3876dd824ee0688 P 10ca5ee68ad64b3283b9d235352c9d23 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:16.675489 12219 raft_consensus.cc:515] T a610d1a71f52432e88fe56130f921fa2 P 0172fed22b6b4890a7fdc953fa128af4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:16.805997 11639 raft_consensus.cc:493] T 6f46a5102fd64b45bf1c4dff863d2b4d P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:16.676330 12250 raft_consensus.cc:515] T 63d44f6ba6434ab2870fe3190f4d7c2f P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:16.795984 12215 raft_consensus.cc:515] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 10ca5ee68ad64b3283b9d235352c9d23 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:16.810909 11974 leader_election.cc:290] T 7839ca2d2c13431a85f44d7c458c6351 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:16.608434 12283 raft_consensus.cc:515] T e89e640169e14e2d98f968f76027daa7 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:16.809643 11653 raft_consensus.cc:493] T 75492f86fde046c3900df3eac8292fd4 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 10ca5ee68ad64b3283b9d235352c9d23)
I20260709 13:04:16.786283 12240 raft_consensus.cc:515] T e5ba72ccd854430baadbc636dfd17136 P 10ca5ee68ad64b3283b9d235352c9d23 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:16.796295 12105 raft_consensus.cc:515] T 5d439326f23747eaab3c1e7da2d6b785 P 10ca5ee68ad64b3283b9d235352c9d23 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:16.819729 12317 raft_consensus.cc:2749] T a610d1a71f52432e88fe56130f921fa2 P 186b12638a87410487467df721a3cbda [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:04:16.795516 12103 raft_consensus.cc:515] T a977842ba4b744f4bb2febd55d286006 P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:16.861541 10877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acc241e7273d49149984968596242712" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:16.835328 10946 raft_consensus.cc:2468] T bb7d67be500947b3aef4c4245e33671c P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 2.
I20260709 13:04:16.863272 12103 leader_election.cc:290] T a977842ba4b744f4bb2febd55d286006 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 3 election: Requested vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
W20260709 13:04:16.822127 12307 thread.cc:641] apply [worker] (thread pool) Time spent creating pthread: real 0.854s	user 0.025s	sys 0.023s
I20260709 13:04:16.863905 12105 leader_election.cc:290] T 5d439326f23747eaab3c1e7da2d6b785 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 5 election: Requested vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:16.864614 11027 raft_consensus.cc:1240] T a977842ba4b744f4bb2febd55d286006 P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Rejecting Update request from peer 0172fed22b6b4890a7fdc953fa128af4 for earlier term 2. Current term is 3. Ops: [2.1-2.1]
I20260709 13:04:16.824417 12310 raft_consensus.cc:3060] T b67eff0634e540eab91cf8cee795969f P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:04:16.865228 10947 raft_consensus.cc:2368] T 5d439326f23747eaab3c1e7da2d6b785 P 10ca5ee68ad64b3283b9d235352c9d23 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 0172fed22b6b4890a7fdc953fa128af4 for earlier term 4. Current term is 5.
I20260709 13:04:16.867463 12105 raft_consensus.cc:493] T bac7e5040f984ba2b7146bd36af7267e P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:16.867995 12215 leader_election.cc:290] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 6 election: Requested vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:16.868146 12105 raft_consensus.cc:515] T bac7e5040f984ba2b7146bd36af7267e P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:16.868947 10948 raft_consensus.cc:2368] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 10ca5ee68ad64b3283b9d235352c9d23 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 0172fed22b6b4890a7fdc953fa128af4 for earlier term 5. Current term is 6.
I20260709 13:04:16.869417 10875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7839ca2d2c13431a85f44d7c458c6351" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:16.817381 12073 raft_consensus.cc:515] T e5a5f5a608504650b48fd27f163faed6 P 10ca5ee68ad64b3283b9d235352c9d23 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:16.871358 12105 leader_election.cc:290] T bac7e5040f984ba2b7146bd36af7267e P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
W20260709 13:04:16.871538 10948 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49424 (request call id 235) took 1631 ms (client timeout 1836 ms). Trace:
I20260709 13:04:16.871845 11098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb7d67be500947b3aef4c4245e33671c" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:16.829607 12292 raft_consensus.cc:515] T 6a9353d1d230416296627d7cf8b1d425 P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
W20260709 13:04:16.864028 12307 thread.cc:608] apply [worker] (thread pool) Time spent starting thread: real 0.898s	user 0.026s	sys 0.023s
I20260709 13:04:16.872391 12073 leader_election.cc:290] T e5a5f5a608504650b48fd27f163faed6 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:16.875360 10879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f371d2565dd84667b5613848604949bb" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:16.909260 10879 raft_consensus.cc:2393] T f371d2565dd84667b5613848604949bb P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0172fed22b6b4890a7fdc953fa128af4 in current term 4: Already voted for candidate 186b12638a87410487467df721a3cbda in this term.
I20260709 13:04:16.828392 12196 raft_consensus.cc:515] T 401a9cf11eac479aa445ec8c1d6e33b2 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:16.862373 10877 raft_consensus.cc:2468] T acc241e7273d49149984968596242712 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 1.
I20260709 13:04:16.855154 11101 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555217d45d9b4ee492dbb3ddaf0f7c02" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:16.879364 11069 leader_election.cc:304] T d371c5e27bd843469d2e2e453734cb4a P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda; no voters: 
W20260709 13:04:16.912432 12327 log.cc:927] Time spent T 0e5d327630e5438b878035d28efe6c0c P 186b12638a87410487467df721a3cbda: Append to log took a long time: real 0.069s	user 0.000s	sys 0.000s
I20260709 13:04:16.875981 11653 raft_consensus.cc:515] T 75492f86fde046c3900df3eac8292fd4 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
W20260709 13:04:16.872051 10948 rpcz_store.cc:269] 0709 13:04:15.239698 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:15.241309 (+  1611us) service_pool.cc:224] Handling call
0709 13:04:16.871311 (+1630002us) spinlock_profiling.cc:243] Waited 1.57 s on lock 0x7b540040db54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:04:16.871466 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3605212544}
I20260709 13:04:16.890223 11021 raft_consensus.cc:2410] T 8c160ddfe9b34bdaa3835d0915c1e06f P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0172fed22b6b4890a7fdc953fa128af4 for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 13:04:16.915835 12127 leader_election.cc:290] T 732c4adeeff6422497aa7db4a82f535b P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 4 election: Requested vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:16.917465 10879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b67eff0634e540eab91cf8cee795969f" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:16.922021 11101 raft_consensus.cc:2468] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 3.
I20260709 13:04:16.905586 11921 raft_consensus.cc:493] T 57b6a2418c1342e7924195e9c1dbd357 P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:16.874187 12240 leader_election.cc:290] T e5ba72ccd854430baadbc636dfd17136 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
W20260709 13:04:16.925714 11069 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:38967, user_credentials={real_user=slave}} blocked reactor thread for 46513.2us
I20260709 13:04:16.932957 10877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5a5f5a608504650b48fd27f163faed6" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:16.841146 12305 raft_consensus.cc:515] T 9a1716c0adf34bee8a4e99341ba5788f P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:16.910733 10872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a1716c0adf34bee8a4e99341ba5788f" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:16.934033 10845 leader_election.cc:304] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda; no voters: 
I20260709 13:04:16.905884 11098 raft_consensus.cc:2468] T bb7d67be500947b3aef4c4245e33671c P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 2.
I20260709 13:04:16.853704 11639 raft_consensus.cc:515] T 6f46a5102fd64b45bf1c4dff863d2b4d P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:16.931560 11937 raft_consensus.cc:2804] T d371c5e27bd843469d2e2e453734cb4a P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:04:16.916273 12283 leader_election.cc:290] T e89e640169e14e2d98f968f76027daa7 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:16.930384 11101 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d156d6ae2ca4fa59a1e387edf0fea39" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 4 candidate_status { last_received { term: 3 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:16.999182 11101 raft_consensus.cc:2468] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 4.
I20260709 13:04:16.999260 10877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a69f32c53f24a50b7852d143665d7b4" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:16.996172 12202 raft_consensus.cc:697] T ff13ae9a47c7455f962406c6d131a3ff P 873357742d5843579556ee8166f23f47 [term 4 LEADER]: Becoming Leader. State: Replica: 873357742d5843579556ee8166f23f47, State: Running, Role: LEADER
I20260709 13:04:16.997660 11098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a9353d1d230416296627d7cf8b1d425" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:04:17.001211 10845 leader_election.cc:304] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda; no voters: 
I20260709 13:04:17.002312 11101 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f34f2bc02d85444f9096e0f5b4fe4331" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:17.008826 11101 raft_consensus.cc:2393] T f34f2bc02d85444f9096e0f5b4fe4331 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 186b12638a87410487467df721a3cbda in current term 2: Already voted for candidate 0172fed22b6b4890a7fdc953fa128af4 in this term.
W20260709 13:04:17.002070 12327 log.cc:927] Time spent T 0e5d327630e5438b878035d28efe6c0c P 186b12638a87410487467df721a3cbda: Append to log took a long time: real 0.070s	user 0.000s	sys 0.000s
I20260709 13:04:16.939114 11921 raft_consensus.cc:515] T 57b6a2418c1342e7924195e9c1dbd357 P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:16.989241 12332 raft_consensus.cc:493] T a154cc8b181f460abf4880064379948b P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:17.013052 11101 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e5d327630e5438b878035d28efe6c0c" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:17.013372 12332 raft_consensus.cc:515] T a154cc8b181f460abf4880064379948b P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:17.013940 10876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e5d327630e5438b878035d28efe6c0c" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:17.014722 11101 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c31881534c5e4f0089c82057c1746652" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 4 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:17.015583 11101 raft_consensus.cc:2393] T c31881534c5e4f0089c82057c1746652 P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 186b12638a87410487467df721a3cbda in current term 4: Already voted for candidate 10ca5ee68ad64b3283b9d235352c9d23 in this term.
I20260709 13:04:17.015797 12332 leader_election.cc:290] T a154cc8b181f460abf4880064379948b P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:17.016716 10877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a9353d1d230416296627d7cf8b1d425" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:04:17.017414 10877 raft_consensus.cc:3060] T 6a9353d1d230416296627d7cf8b1d425 P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:04:17.006958 10872 raft_consensus.cc:2468] T 9a1716c0adf34bee8a4e99341ba5788f P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 4.
I20260709 13:04:17.021426 12098 consensus_queue.cc:260] T f371d2565dd84667b5613848604949bb P 873357742d5843579556ee8166f23f47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:17.022616 12098 raft_consensus.cc:3060] T f371d2565dd84667b5613848604949bb P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:04:17.031840 11974 raft_consensus.cc:493] T 63d44f6ba6434ab2870fe3190f4d7c2f P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:17.032588 11974 raft_consensus.cc:515] T 63d44f6ba6434ab2870fe3190f4d7c2f P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:17.035218 11653 leader_election.cc:290] T 75492f86fde046c3900df3eac8292fd4 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:17.036007 11921 leader_election.cc:290] T 57b6a2418c1342e7924195e9c1dbd357 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:17.037942 11937 raft_consensus.cc:697] T d371c5e27bd843469d2e2e453734cb4a P 0172fed22b6b4890a7fdc953fa128af4 [term 4 LEADER]: Becoming Leader. State: Replica: 0172fed22b6b4890a7fdc953fa128af4, State: Running, Role: LEADER
I20260709 13:04:17.038456 11974 leader_election.cc:290] T 63d44f6ba6434ab2870fe3190f4d7c2f P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:16.986673 12104 raft_consensus.cc:2804] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:04:16.977907 10875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d439326f23747eaab3c1e7da2d6b785" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:04:17.049854 10954 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49424 (request call id 237) took 1092 ms. Trace:
I20260709 13:04:17.005424 10992 leader_election.cc:304] T bb7d67be500947b3aef4c4245e33671c P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:04:17.050450 10954 rpcz_store.cc:276] 0709 13:04:15.957757 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:15.963739 (+  5982us) service_pool.cc:224] Handling call
0709 13:04:15.964096 (+   357us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:15.964521 (+   425us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:15.964558 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:16.872854 (+908296us) spinlock_profiling.cc:243] Waited 3.99 ms on lock 0x7b2c00173848. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4264a5ba11 00007f4264a6b372 00007f426619da01 00007f426616243a 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2
0709 13:04:16.873487 (+   633us) log.cc:844] Serialized 26 byte log entry
0709 13:04:16.907623 (+ 34136us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:04:16.907685 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:16.907795 (+   110us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:17.039222 (+131427us) raft_consensus.cc:1685] finished
0709 13:04:17.039264 (+    42us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:17.049738 (+ 10474us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":903,"raft.run_cpu_time_us":108,"raft.run_wall_time_us":109,"spinlock_wait_cycles":9175936,"thread_start_us":33691,"threads_started":1,"wal-append.queue_time_us":162372}
W20260709 13:04:17.052397 10992 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 47075.5us
I20260709 13:04:16.888582 12218 consensus_queue.cc:260] T 75492f86fde046c3900df3eac8292fd4 P 10ca5ee68ad64b3283b9d235352c9d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:17.046303 12317 raft_consensus.cc:2804] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:04:17.047315 12104 raft_consensus.cc:493] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:17.057655 11921 raft_consensus.cc:2804] T bb7d67be500947b3aef4c4245e33671c P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:04:17.059759 10875 raft_consensus.cc:3060] T 5d439326f23747eaab3c1e7da2d6b785 P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Advancing to term 5
    @     0x7f425f968e87 gsignal at ??:0
I20260709 13:04:17.073020 10877 raft_consensus.cc:2468] T 6a9353d1d230416296627d7cf8b1d425 P 186b12638a87410487467df721a3cbda [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 5.
I20260709 13:04:17.074331 12336 raft_consensus.cc:493] T 65bf0a62f4d84ef18dd6b23e082724b8 P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 10ca5ee68ad64b3283b9d235352c9d23)
W20260709 13:04:17.081888 12299 consensus_meta.cc:287] T 4f9eb06ed01f4ce68be3af8a8434665b P 186b12638a87410487467df721a3cbda: Time spent flushing consensus metadata: real 0.645s	user 0.011s	sys 0.005s
I20260709 13:04:17.087184 12219 leader_election.cc:290] T a610d1a71f52432e88fe56130f921fa2 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:17.039218 11937 consensus_queue.cc:237] T d371c5e27bd843469d2e2e453734cb4a P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:17.072331 12104 raft_consensus.cc:3060] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Advancing to term 5
W20260709 13:04:17.087931 10993 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:17.089109 11029 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "401a9cf11eac479aa445ec8c1d6e33b2" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
I20260709 13:04:17.057849 11653 raft_consensus.cc:493] T 2567c2d84e4648d7ba5dc23a3df32817 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 10ca5ee68ad64b3283b9d235352c9d23)
W20260709 13:04:17.093040 10919 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:17.069242 12317 raft_consensus.cc:493] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Starting leader election (detected failure of leader 0172fed22b6b4890a7fdc953fa128af4)
I20260709 13:04:17.097759 12336 raft_consensus.cc:515] T 65bf0a62f4d84ef18dd6b23e082724b8 P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:17.099097 12292 leader_election.cc:290] T 6a9353d1d230416296627d7cf8b1d425 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:17.102670 11029 raft_consensus.cc:3060] T 401a9cf11eac479aa445ec8c1d6e33b2 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:04:17.073361 12218 raft_consensus.cc:3060] T 75492f86fde046c3900df3eac8292fd4 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Advancing to term 3
W20260709 13:04:17.130157 10919 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:17.094285 10877 raft_consensus.cc:1240] T a1ec2b50c6ad4f7c8033476208abb26d P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Rejecting Update request from peer 873357742d5843579556ee8166f23f47 for earlier term 2. Current term is 3. Ops: []
I20260709 13:04:17.095602 12308 leader_election.cc:290] T 65bf0a62f4d84ef18dd6b23e082724b8 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
W20260709 13:04:17.079886 10950 consensus_meta.cc:287] T ff13ae9a47c7455f962406c6d131a3ff P 10ca5ee68ad64b3283b9d235352c9d23: Time spent flushing consensus metadata: real 1.329s	user 0.012s	sys 0.005s
I20260709 13:04:17.074331 11921 raft_consensus.cc:493] T bb7d67be500947b3aef4c4245e33671c P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:17.141829 12317 raft_consensus.cc:3060] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Advancing to term 4
W20260709 13:04:17.080124 12323 log.cc:927] Time spent T 72b111ad7b5f4753ab1d2b5ee1114840 P 0172fed22b6b4890a7fdc953fa128af4: Append to log took a long time: real 0.235s	user 0.001s	sys 0.000s
I20260709 13:04:17.155424 11974 raft_consensus.cc:493] T b4cd3d7dfbbf4096908ff11e9dc57141 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:17.169993 10995 leader_election.cc:304] T 63d44f6ba6434ab2870fe3190f4d7c2f P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 873357742d5843579556ee8166f23f47; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda
I20260709 13:04:17.163965 11029 raft_consensus.cc:2468] T 401a9cf11eac479aa445ec8c1d6e33b2 P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 3.
I20260709 13:04:17.080592 12300 raft_consensus.cc:3055] T a1ec2b50c6ad4f7c8033476208abb26d P 873357742d5843579556ee8166f23f47 [term 2 LEADER]: Stepping down as leader of term 2
W20260709 13:04:17.171330 11066 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 132647us
I20260709 13:04:17.176331 11029 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46642 (request call id 164) took 1424 ms. Trace:
I20260709 13:04:17.176959 11029 rpcz_store.cc:276] 0709 13:04:15.751847 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:15.761494 (+  9647us) service_pool.cc:224] Handling call
0709 13:04:17.088869 (+1327375us) spinlock_profiling.cc:243] Waited 7.2 ms on lock 0x7b58007a2898. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f426303db71 00007f426303da22 00007f426bf56c09 00007f426b0b0ee0 00007f426b17a0ab 00007f426b17a131 00007f426b1280a0 00007f426b127984 00007f42646a26d4 00007f42646a2662
0709 13:04:17.176234 (+ 87365us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":16566144}
W20260709 13:04:17.171561 12193 consensus_meta.cc:287] T 4f9eb06ed01f4ce68be3af8a8434665b P 0172fed22b6b4890a7fdc953fa128af4: Time spent flushing consensus metadata: real 0.755s	user 0.014s	sys 0.000s
I20260709 13:04:17.164760 12305 leader_election.cc:290] T 9a1716c0adf34bee8a4e99341ba5788f P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:17.183084 11029 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5299b8d044f41b7b68057e63d17b159" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:17.142378 11653 raft_consensus.cc:515] T 2567c2d84e4648d7ba5dc23a3df32817 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:17.211735 11029 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c160ddfe9b34bdaa3835d0915c1e06f" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
W20260709 13:04:17.174181 12321 log.cc:927] Time spent T a1ec2b50c6ad4f7c8033476208abb26d P 0172fed22b6b4890a7fdc953fa128af4: Append to log took a long time: real 0.258s	user 0.001s	sys 0.000s
I20260709 13:04:17.179224 12300 raft_consensus.cc:740] T a1ec2b50c6ad4f7c8033476208abb26d P 873357742d5843579556ee8166f23f47 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 873357742d5843579556ee8166f23f47, State: Running, Role: LEADER
I20260709 13:04:17.161495 10950 raft_consensus.cc:2468] T ff13ae9a47c7455f962406c6d131a3ff P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 4.
I20260709 13:04:17.206691 12333 raft_consensus.cc:2749] T 63d44f6ba6434ab2870fe3190f4d7c2f P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 13:04:17.168951 11921 raft_consensus.cc:3060] T bb7d67be500947b3aef4c4245e33671c P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Advancing to term 3
W20260709 13:04:17.169875 12296 consensus_meta.cc:287] T 8c160ddfe9b34bdaa3835d0915c1e06f P 186b12638a87410487467df721a3cbda: Time spent flushing consensus metadata: real 0.524s	user 0.015s	sys 0.000s
I20260709 13:04:17.107136 12299 raft_consensus.cc:515] T 4f9eb06ed01f4ce68be3af8a8434665b P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:17.212862 11029 raft_consensus.cc:2468] T 8c160ddfe9b34bdaa3835d0915c1e06f P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 2.
W20260709 13:04:17.209393 11095 consensus_meta.cc:287] T 401a9cf11eac479aa445ec8c1d6e33b2 P 0172fed22b6b4890a7fdc953fa128af4: Time spent flushing consensus metadata: real 1.176s	user 0.013s	sys 0.001s
I20260709 13:04:17.100054 12202 consensus_queue.cc:237] T ff13ae9a47c7455f962406c6d131a3ff P 873357742d5843579556ee8166f23f47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:17.277522 11027 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46640 (request call id 248) took 1374 ms. Trace:
I20260709 13:04:17.277907 11029 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "30f628e3d7914d609e7f1d980a74c654" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:17.199994 12336 leader_election.cc:290] T 65bf0a62f4d84ef18dd6b23e082724b8 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:17.279084 11029 raft_consensus.cc:2468] T 30f628e3d7914d609e7f1d980a74c654 P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 1.
I20260709 13:04:17.278262 11027 rpcz_store.cc:276] 0709 13:04:15.902961 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:16.742593 (+839632us) service_pool.cc:224] Handling call
0709 13:04:16.743111 (+   518us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:16.865092 (+121981us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:17.277063 (+411971us) spinlock_profiling.cc:243] Waited 121 ms on lock 0x7b540074efd4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:04:17.277414 (+   351us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":278702464}
W20260709 13:04:17.260488 12323 log.cc:927] Time spent T 72b111ad7b5f4753ab1d2b5ee1114840 P 0172fed22b6b4890a7fdc953fa128af4: Append to log took a long time: real 0.071s	user 0.001s	sys 0.000s
I20260709 13:04:17.278563 11095 raft_consensus.cc:2468] T 401a9cf11eac479aa445ec8c1d6e33b2 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 3.
I20260709 13:04:17.286145 11943 leader_election.cc:290] T c31881534c5e4f0089c82057c1746652 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 3 election: Requested vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:17.287204 12300 consensus_queue.cc:260] T a1ec2b50c6ad4f7c8033476208abb26d P 873357742d5843579556ee8166f23f47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 2.1, Last appended by leader: 1, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:17.290985 11102 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75492f86fde046c3900df3eac8292fd4" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
W20260709 13:04:17.292527 10880 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51704 (request call id 235) took 2008 ms (client timeout 2347 ms). Trace:
I20260709 13:04:17.296214 12049 leader_election.cc:290] T a5299b8d044f41b7b68057e63d17b159 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:17.297737 11097 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37666 (request call id 234) took 2099 ms. Trace:
W20260709 13:04:17.300491 10845 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.107.196:38507 timed out after 2.216s (SENT) [suppressed 16 similar messages]
I20260709 13:04:17.300474 11028 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555217d45d9b4ee492dbb3ddaf0f7c02" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:17.338703 11028 raft_consensus.cc:2468] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 4.
I20260709 13:04:17.226073 12286 raft_consensus.cc:493] T d7ce47b9a92d43dd88d95bcda5038597 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 186b12638a87410487467df721a3cbda)
I20260709 13:04:17.185607 12196 leader_election.cc:290] T 401a9cf11eac479aa445ec8c1d6e33b2 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:17.180542 11974 raft_consensus.cc:515] T b4cd3d7dfbbf4096908ff11e9dc57141 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:17.262037 12313 leader_election.cc:290] T 4b37f8fca8dc42718b2a42f0883a956f P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:17.309027 10874 raft_consensus.cc:2393] T 4f9eb06ed01f4ce68be3af8a8434665b P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 873357742d5843579556ee8166f23f47 in current term 4: Already voted for candidate 186b12638a87410487467df721a3cbda in this term.
I20260709 13:04:17.337446 11097 rpcz_store.cc:276] 0709 13:04:15.198390 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:15.691891 (+493501us) service_pool.cc:224] Handling call
0709 13:04:15.692251 (+   360us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:16.000446 (+308195us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:17.297308 (+1296862us) spinlock_profiling.cc:243] Waited 93.8 ms on lock 0x7b54003246d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:04:17.297618 (+   310us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":215759616}
W20260709 13:04:17.339704 12274 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.948s	user 0.002s	sys 0.016s
I20260709 13:04:17.227149 12310 raft_consensus.cc:515] T b67eff0634e540eab91cf8cee795969f P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:17.301393 11639 leader_election.cc:290] T 6f46a5102fd64b45bf1c4dff863d2b4d P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:17.335297 11102 raft_consensus.cc:2393] T 75492f86fde046c3900df3eac8292fd4 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 873357742d5843579556ee8166f23f47 in current term 3: Already voted for candidate 0172fed22b6b4890a7fdc953fa128af4 in this term.
W20260709 13:04:17.335657 10880 rpcz_store.cc:269] 0709 13:04:15.284052 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:15.499580 (+215528us) service_pool.cc:224] Handling call
0709 13:04:17.292084 (+1792504us) spinlock_profiling.cc:243] Waited 182 ms on lock 0x7b5400339d54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:04:17.292413 (+   329us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":418033024}
W20260709 13:04:17.331418 12223 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.831s	user 0.012s	sys 0.023s
I20260709 13:04:17.271456 12296 raft_consensus.cc:515] T 8c160ddfe9b34bdaa3835d0915c1e06f P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:17.342397 12300 raft_consensus.cc:3060] T a1ec2b50c6ad4f7c8033476208abb26d P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:04:17.347746 11101 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a977842ba4b744f4bb2febd55d286006" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:04:17.351166 11022 raft_consensus.cc:2410] T f371d2565dd84667b5613848604949bb P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 186b12638a87410487467df721a3cbda for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20260709 13:04:17.334928 11095 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37678 (request call id 159) took 2164 ms (client timeout 2216 ms). Trace:
I20260709 13:04:17.352011 12299 leader_election.cc:290] T 4f9eb06ed01f4ce68be3af8a8434665b P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 4 election: Requested vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:17.341558 12049 raft_consensus.cc:493] T 525bfdbd6d934afd823f7d3ee3f9d052 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:17.353225 11028 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e96e80ba9ac34d0998395515e5c8c41a" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:17.353840 12286 raft_consensus.cc:3060] T d7ce47b9a92d43dd88d95bcda5038597 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:04:17.353855 12305 raft_consensus.cc:493] T 868e92339c2f4508af0afbd07c814d64 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0172fed22b6b4890a7fdc953fa128af4)
I20260709 13:04:17.211737 12193 raft_consensus.cc:515] T 4f9eb06ed01f4ce68be3af8a8434665b P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:17.324290 11029 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11de93cd9b9d457489ff1cb80f976c9a" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:17.361901 10880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65bf0a62f4d84ef18dd6b23e082724b8" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:17.408371 10880 raft_consensus.cc:2468] T 65bf0a62f4d84ef18dd6b23e082724b8 P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 3.
W20260709 13:04:17.359368 12274 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.972s	user 0.002s	sys 0.017s
W20260709 13:04:17.375325 12264 log.cc:927] Time spent T 8a69f32c53f24a50b7852d143665d7b4 P 10ca5ee68ad64b3283b9d235352c9d23: Append to log took a long time: real 1.755s	user 0.021s	sys 0.043s
I20260709 13:04:17.376940 12296 leader_election.cc:290] T 8c160ddfe9b34bdaa3835d0915c1e06f P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 4 election: Requested vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:17.379892 10879 raft_consensus.cc:2393] T b67eff0634e540eab91cf8cee795969f P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 10ca5ee68ad64b3283b9d235352c9d23 in current term 3: Already voted for candidate 186b12638a87410487467df721a3cbda in this term.
W20260709 13:04:17.363140 12223 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.871s	user 0.013s	sys 0.024s
I20260709 13:04:17.358556 12308 raft_consensus.cc:493] T a1ec2b50c6ad4f7c8033476208abb26d P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:17.392758 12317 raft_consensus.cc:515] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:17.396029 10873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57b6a2418c1342e7924195e9c1dbd357" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:17.400832 11028 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f1466124dbd4f6ebb3c4d62ca48bf96" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
I20260709 13:04:17.425619 11028 raft_consensus.cc:3060] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 873357742d5843579556ee8166f23f47 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:04:17.401325 10877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "401a9cf11eac479aa445ec8c1d6e33b2" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
W20260709 13:04:17.404502 10993 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:17.363600 11653 leader_election.cc:290] T 2567c2d84e4648d7ba5dc23a3df32817 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:17.411288 11102 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a9353d1d230416296627d7cf8b1d425" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:17.396152 12104 raft_consensus.cc:515] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 186b12638a87410487467df721a3cbda [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:17.414866 10880 raft_consensus.cc:1240] T 34064dc7bf3747dbb3876dd824ee0688 P 186b12638a87410487467df721a3cbda [term 6 FOLLOWER]: Rejecting Update request from peer 873357742d5843579556ee8166f23f47 for earlier term 5. Current term is 6. Ops: [3.2-5.3]
I20260709 13:04:17.430482 10877 raft_consensus.cc:2393] T 401a9cf11eac479aa445ec8c1d6e33b2 P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 873357742d5843579556ee8166f23f47 in current term 3: Already voted for candidate 186b12638a87410487467df721a3cbda in this term.
I20260709 13:04:17.401932 11097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a1716c0adf34bee8a4e99341ba5788f" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:17.421638 10871 raft_consensus.cc:2368] T 8c160ddfe9b34bdaa3835d0915c1e06f P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 873357742d5843579556ee8166f23f47 for earlier term 3. Current term is 4.
I20260709 13:04:17.434070 10880 raft_consensus.cc:1240] T f371d2565dd84667b5613848604949bb P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Rejecting Update request from peer 873357742d5843579556ee8166f23f47 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260709 13:04:17.438585 11102 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c160ddfe9b34bdaa3835d0915c1e06f" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:04:17.441190 11102 raft_consensus.cc:3060] T 8c160ddfe9b34bdaa3835d0915c1e06f P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:04:17.447646 10877 raft_consensus.cc:3060] T a5299b8d044f41b7b68057e63d17b159 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Advancing to term 3
W20260709 13:04:17.380505 11095 rpcz_store.cc:269] 0709 13:04:15.168677 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:15.726048 (+557371us) service_pool.cc:224] Handling call
0709 13:04:17.332581 (+1606533us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:17.399834 12049 raft_consensus.cc:515] T 525bfdbd6d934afd823f7d3ee3f9d052 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:17.459048 11028 raft_consensus.cc:2468] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 873357742d5843579556ee8166f23f47 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 6.
I20260709 13:04:17.431381 12296 raft_consensus.cc:493] T d270b9e5481245d9b5340a454a7cd10b P 186b12638a87410487467df721a3cbda [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:17.469467 11028 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7839ca2d2c13431a85f44d7c458c6351" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:17.469816 10919 leader_election.cc:304] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:04:17.411554 12305 raft_consensus.cc:515] T 868e92339c2f4508af0afbd07c814d64 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:17.472115 10877 raft_consensus.cc:1275] T a5299b8d044f41b7b68057e63d17b159 P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Refusing update from remote peer 873357742d5843579556ee8166f23f47: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 13:04:17.478407 11028 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c2a35bb99374e93b573cf870d04faaa" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:17.479233 11102 raft_consensus.cc:2468] T 8c160ddfe9b34bdaa3835d0915c1e06f P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 4.
    @     0x7f425f96a7f1 abort at ??:0
    @           0x2de526 abort at /apache/dev/git/kudu/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:263
I20260709 13:04:17.453228 11097 raft_consensus.cc:2468] T 9a1716c0adf34bee8a4e99341ba5788f P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 4.
I20260709 13:04:17.499812 12193 leader_election.cc:290] T 4f9eb06ed01f4ce68be3af8a8434665b P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 3 election: Requested vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:17.500676 11097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "868e92339c2f4508af0afbd07c814d64" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:17.502480 11097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "401a9cf11eac479aa445ec8c1d6e33b2" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:17.504362 10845 leader_election.cc:304] T 9a1716c0adf34bee8a4e99341ba5788f P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda; no voters: 
I20260709 13:04:17.505878 12068 raft_consensus.cc:2804] T 9a1716c0adf34bee8a4e99341ba5788f P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:04:17.506330 12068 raft_consensus.cc:493] T 9a1716c0adf34bee8a4e99341ba5788f P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Starting leader election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:17.506780 12068 raft_consensus.cc:3060] T 9a1716c0adf34bee8a4e99341ba5788f P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:04:17.429139 12308 raft_consensus.cc:515] T a1ec2b50c6ad4f7c8033476208abb26d P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:17.509889 12308 leader_election.cc:290] T a1ec2b50c6ad4f7c8033476208abb26d P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:17.462023 12223 raft_consensus.cc:493] T 7839ca2d2c13431a85f44d7c458c6351 P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 873357742d5843579556ee8166f23f47)
W20260709 13:04:17.511063 10878 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51704 (request call id 240) took 1729 ms (client timeout 1887 ms). Trace:
I20260709 13:04:17.510936 12223 raft_consensus.cc:515] T 7839ca2d2c13431a85f44d7c458c6351 P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
W20260709 13:04:17.511622 10878 rpcz_store.cc:269] 0709 13:04:15.781756 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:15.782000 (+   244us) service_pool.cc:224] Handling call
0709 13:04:17.510821 (+1728821us) spinlock_profiling.cc:243] Waited 115 ms on lock 0x7b54009ccf54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:04:17.510990 (+   169us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":263567616}
I20260709 13:04:17.513917 12223 leader_election.cc:290] T 7839ca2d2c13431a85f44d7c458c6351 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:17.483863 12310 leader_election.cc:290] T b67eff0634e540eab91cf8cee795969f P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 3 election: Requested vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:17.485520 11028 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2567c2d84e4648d7ba5dc23a3df32817" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
W20260709 13:04:17.490916 12349 log.cc:927] Time spent T 0f9df2bc281f4866b6da5da94a0defc7 P 0172fed22b6b4890a7fdc953fa128af4: Append to log took a long time: real 0.056s	user 0.000s	sys 0.000s
I20260709 13:04:17.492400 11974 leader_election.cc:290] T b4cd3d7dfbbf4096908ff11e9dc57141 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:17.499179 12305 leader_election.cc:290] T 868e92339c2f4508af0afbd07c814d64 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:17.518329 11028 raft_consensus.cc:2468] T 2567c2d84e4648d7ba5dc23a3df32817 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 2.
I20260709 13:04:17.523679 11028 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32147c6d2d5a477aa38c597c7f78df5b" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:17.477814 11639 raft_consensus.cc:2804] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 10ca5ee68ad64b3283b9d235352c9d23 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 13:04:17.524391 11028 raft_consensus.cc:2468] T 32147c6d2d5a477aa38c597c7f78df5b P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 3.
I20260709 13:04:17.521234 10845 leader_election.cc:304] T 8c160ddfe9b34bdaa3835d0915c1e06f P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda; no voters: 
I20260709 13:04:17.525422 11028 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63d44f6ba6434ab2870fe3190f4d7c2f" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:17.526002 11028 raft_consensus.cc:2468] T 63d44f6ba6434ab2870fe3190f4d7c2f P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 4.
I20260709 13:04:17.525583 10919 leader_election.cc:304] T 32147c6d2d5a477aa38c597c7f78df5b P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:04:17.503559 11097 raft_consensus.cc:2393] T 401a9cf11eac479aa445ec8c1d6e33b2 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 873357742d5843579556ee8166f23f47 in current term 3: Already voted for candidate 186b12638a87410487467df721a3cbda in this term.
I20260709 13:04:17.529456 12104 leader_election.cc:290] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 5 election: Requested vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:17.530035 11097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1ec2b50c6ad4f7c8033476208abb26d" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 4 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:17.476686 12296 raft_consensus.cc:515] T d270b9e5481245d9b5340a454a7cd10b P 186b12638a87410487467df721a3cbda [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:17.528649 11028 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b37f8fca8dc42718b2a42f0883a956f" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:17.533363 11028 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f46a5102fd64b45bf1c4dff863d2b4d" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:17.527585 10919 leader_election.cc:304] T 63d44f6ba6434ab2870fe3190f4d7c2f P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47; no voters: 
W20260709 13:04:17.530393 12309 log.cc:927] Time spent T 5de5f0581e3b41dea729363d6ca563bb P 10ca5ee68ad64b3283b9d235352c9d23: Append to log took a long time: real 1.119s	user 0.003s	sys 0.010s
I20260709 13:04:17.541002 11028 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5ba72ccd854430baadbc636dfd17136" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:17.533027 11102 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "401a9cf11eac479aa445ec8c1d6e33b2" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:04:17.530119 12049 leader_election.cc:290] T 525bfdbd6d934afd823f7d3ee3f9d052 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:17.543088 11102 raft_consensus.cc:2376] T 401a9cf11eac479aa445ec8c1d6e33b2 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 186b12638a87410487467df721a3cbda in term 3. Re-sending same reply.
I20260709 13:04:17.547593 11028 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5299b8d044f41b7b68057e63d17b159" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:17.523010 11095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f9eb06ed01f4ce68be3af8a8434665b" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:04:17.532989 11974 raft_consensus.cc:2804] T 32147c6d2d5a477aa38c597c7f78df5b P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:04:17.532665 11639 raft_consensus.cc:697] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 10ca5ee68ad64b3283b9d235352c9d23 [term 6 LEADER]: Becoming Leader. State: Replica: 10ca5ee68ad64b3283b9d235352c9d23, State: Running, Role: LEADER
I20260709 13:04:17.542483 12223 raft_consensus.cc:2804] T 8c160ddfe9b34bdaa3835d0915c1e06f P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:04:17.557224 10845 leader_election.cc:304] T 401a9cf11eac479aa445ec8c1d6e33b2 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda; no voters: 
I20260709 13:04:17.557257 11028 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "401a9cf11eac479aa445ec8c1d6e33b2" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:17.544767 12215 raft_consensus.cc:2804] T 63d44f6ba6434ab2870fe3190f4d7c2f P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:04:17.556026 11095 raft_consensus.cc:3060] T 4f9eb06ed01f4ce68be3af8a8434665b P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:04:17.565133 11102 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b67eff0634e540eab91cf8cee795969f" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:04:17.568754 11102 raft_consensus.cc:3060] T b67eff0634e540eab91cf8cee795969f P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:04:17.564535 11028 raft_consensus.cc:2376] T 401a9cf11eac479aa445ec8c1d6e33b2 P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 186b12638a87410487467df721a3cbda in term 3. Re-sending same reply.
I20260709 13:04:17.570950 11028 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a154cc8b181f460abf4880064379948b" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:17.555996 12193 raft_consensus.cc:493] T 836251ebc2b74f5da88b8a8b2338e2a5 P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:17.557726 12049 raft_consensus.cc:493] T a64b7e99d6c54934a7de6f6dffcf5e94 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:17.556656 11974 raft_consensus.cc:493] T 32147c6d2d5a477aa38c597c7f78df5b P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:17.565863 12215 raft_consensus.cc:493] T 63d44f6ba6434ab2870fe3190f4d7c2f P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:17.577258 11028 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "732c4adeeff6422497aa7db4a82f535b" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
I20260709 13:04:17.570835 11094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555217d45d9b4ee492dbb3ddaf0f7c02" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
W20260709 13:04:17.605772 10919 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:17.605365 11102 raft_consensus.cc:2468] T b67eff0634e540eab91cf8cee795969f P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 3.
I20260709 13:04:17.610392 10951 raft_consensus.cc:1240] T 732c4adeeff6422497aa7db4a82f535b P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Rejecting Update request from peer 186b12638a87410487467df721a3cbda for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260709 13:04:17.573061 12356 raft_consensus.cc:2804] T 401a9cf11eac479aa445ec8c1d6e33b2 P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:04:17.575996 12223 raft_consensus.cc:697] T 8c160ddfe9b34bdaa3835d0915c1e06f P 186b12638a87410487467df721a3cbda [term 4 LEADER]: Becoming Leader. State: Replica: 186b12638a87410487467df721a3cbda, State: Running, Role: LEADER
I20260709 13:04:17.583490 11974 raft_consensus.cc:3060] T 32147c6d2d5a477aa38c597c7f78df5b P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:04:17.616540 12223 consensus_queue.cc:237] T 8c160ddfe9b34bdaa3835d0915c1e06f P 186b12638a87410487467df721a3cbda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:17.576767 12193 raft_consensus.cc:515] T 836251ebc2b74f5da88b8a8b2338e2a5 P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:17.610852 12317 leader_election.cc:290] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 4 election: Requested vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:17.585312 12215 raft_consensus.cc:3060] T 63d44f6ba6434ab2870fe3190f4d7c2f P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:04:17.623755 12193 leader_election.cc:290] T 836251ebc2b74f5da88b8a8b2338e2a5 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:17.626881 11025 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46642 (request call id 158) took 1902 ms. Trace:
I20260709 13:04:17.630728 11025 rpcz_store.cc:276] 0709 13:04:15.724219 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:15.728979 (+  4760us) service_pool.cc:224] Handling call
0709 13:04:17.410792 (+1681813us) spinlock_profiling.cc:243] Waited 8.79 ms on lock 0x7b58007a2898. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f426303db71 00007f426303da22 00007f426bf56c09 00007f426b0b0ee0 00007f426b17a0ab 00007f426b17a131 00007f426b13bfc0 00007f426b126929 00007f426b1263d2 00007f426469dbd4
0709 13:04:17.431994 (+ 21202us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:17.482590 (+ 50596us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:17.482675 (+    85us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:17.483504 (+   829us) log.cc:844] Serialized 26 byte log entry
0709 13:04:17.584777 (+101273us) raft_consensus.cc:1645] Marking committed up to 0
0709 13:04:17.608656 (+ 23879us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:17.612114 (+  3458us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:17.626486 (+ 14372us) raft_consensus.cc:1685] finished
0709 13:04:17.626523 (+    37us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:17.626803 (+   280us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":2593,"raft.run_cpu_time_us":214,"raft.run_wall_time_us":25888,"spinlock_wait_cycles":20222720,"thread_start_us":93412,"threads_started":1,"wal-append.queue_time_us":122583}
I20260709 13:04:17.633093 11025 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e89e640169e14e2d98f968f76027daa7" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:17.612532 11094 raft_consensus.cc:3060] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Advancing to term 5
I20260709 13:04:17.627034 11027 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b37f8fca8dc42718b2a42f0883a956f" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:17.614301 10845 leader_election.cc:304] T b67eff0634e540eab91cf8cee795969f P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda; no voters: 
I20260709 13:04:17.638827 11027 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4cd3d7dfbbf4096908ff11e9dc57141" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:17.627731 12356 raft_consensus.cc:697] T 401a9cf11eac479aa445ec8c1d6e33b2 P 186b12638a87410487467df721a3cbda [term 3 LEADER]: Becoming Leader. State: Replica: 186b12638a87410487467df721a3cbda, State: Running, Role: LEADER
I20260709 13:04:17.628434 11097 raft_consensus.cc:1275] T c51782cd9d064ec0949e84e24dd4de67 P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Refusing update from remote peer 186b12638a87410487467df721a3cbda: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 13:04:17.640205 11027 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb7d67be500947b3aef4c4245e33671c" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:17.579656 10875 raft_consensus.cc:2468] T 5d439326f23747eaab3c1e7da2d6b785 P 186b12638a87410487467df721a3cbda [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 5.
I20260709 13:04:17.657347 11102 raft_consensus.cc:1240] T a1ec2b50c6ad4f7c8033476208abb26d P 0172fed22b6b4890a7fdc953fa128af4 [term 3 LEADER]: Rejecting Update request from peer 873357742d5843579556ee8166f23f47 for earlier term 2. Current term is 3. Ops: [2.1-2.1]
I20260709 13:04:17.646814 10995 leader_election.cc:304] T 30f628e3d7914d609e7f1d980a74c654 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 873357742d5843579556ee8166f23f47; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda
I20260709 13:04:17.660748 10995 leader_election.cc:304] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 873357742d5843579556ee8166f23f47; no voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda
I20260709 13:04:17.586194 11028 raft_consensus.cc:3060] T 732c4adeeff6422497aa7db4a82f535b P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:04:17.649878 12286 raft_consensus.cc:515] T d7ce47b9a92d43dd88d95bcda5038597 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:17.577996 12049 raft_consensus.cc:515] T a64b7e99d6c54934a7de6f6dffcf5e94 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:17.653558 11921 raft_consensus.cc:515] T bb7d67be500947b3aef4c4245e33671c P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:17.654392 11025 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bac7e5040f984ba2b7146bd36af7267e" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:17.651170 10872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2567c2d84e4648d7ba5dc23a3df32817" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:17.663617 10995 leader_election.cc:304] T 0e5d327630e5438b878035d28efe6c0c P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 873357742d5843579556ee8166f23f47; no voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda
I20260709 13:04:17.639940 11098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d156d6ae2ca4fa59a1e387edf0fea39" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 4 candidate_status { last_received { term: 3 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:04:17.655320 12355 raft_consensus.cc:2804] T b67eff0634e540eab91cf8cee795969f P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:04:17.698958 12355 raft_consensus.cc:697] T b67eff0634e540eab91cf8cee795969f P 186b12638a87410487467df721a3cbda [term 3 LEADER]: Becoming Leader. State: Replica: 186b12638a87410487467df721a3cbda, State: Running, Role: LEADER
I20260709 13:04:17.700062 12355 consensus_queue.cc:237] T b67eff0634e540eab91cf8cee795969f P 186b12638a87410487467df721a3cbda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:17.701691 11021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e89e640169e14e2d98f968f76027daa7" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:17.666177 12070 raft_consensus.cc:2749] T 30f628e3d7914d609e7f1d980a74c654 P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:04:17.666370 12053 raft_consensus.cc:2749] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 13:04:17.671124 10951 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49428 (request call id 195) took 1712 ms. Trace:
I20260709 13:04:17.681862 11633 raft_consensus.cc:2749] T 0e5d327630e5438b878035d28efe6c0c P 873357742d5843579556ee8166f23f47 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:04:17.665633 11639 consensus_queue.cc:237] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:17.703341 10951 rpcz_store.cc:276] 0709 13:04:15.958801 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:17.608372 (+1649571us) service_pool.cc:224] Handling call
0709 13:04:17.610020 (+  1648us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:17.615245 (+  5225us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:17.667105 (+ 51860us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:17.703899 11021 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46628 (request call id 233) took 1712 ms. Trace:
I20260709 13:04:17.704373 11021 rpcz_store.cc:276] 0709 13:04:15.991479 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:17.702910 (+1711431us) service_pool.cc:224] Handling call
0709 13:04:17.703256 (+   346us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:17.703505 (+   249us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:17.703537 (+    32us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:17.703583 (+    46us) raft_consensus.cc:1645] Marking committed up to 1
0709 13:04:17.703625 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:17.703705 (+    80us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:17.703832 (+   127us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:17.705740 11921 leader_election.cc:290] T bb7d67be500947b3aef4c4245e33671c P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 3 election: Requested vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:17.682925 11025 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "525bfdbd6d934afd823f7d3ee3f9d052" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:17.684191 12068 raft_consensus.cc:515] T 9a1716c0adf34bee8a4e99341ba5788f P 186b12638a87410487467df721a3cbda [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:17.680105 10995 leader_election.cc:304] T 6a9353d1d230416296627d7cf8b1d425 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47; no voters: 0172fed22b6b4890a7fdc953fa128af4
I20260709 13:04:17.707058 11103 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb7d67be500947b3aef4c4245e33671c" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:04:17.694401 12356 consensus_queue.cc:237] T 401a9cf11eac479aa445ec8c1d6e33b2 P 186b12638a87410487467df721a3cbda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:17.718755 11103 raft_consensus.cc:3060] T bb7d67be500947b3aef4c4245e33671c P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:04:17.718173 11972 raft_consensus.cc:2804] T 6a9353d1d230416296627d7cf8b1d425 P 873357742d5843579556ee8166f23f47 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:04:17.697371 12286 leader_election.cc:290] T d7ce47b9a92d43dd88d95bcda5038597 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 3 election: Requested vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:17.667097 11974 raft_consensus.cc:515] T 32147c6d2d5a477aa38c597c7f78df5b P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:17.707595 10880 raft_consensus.cc:1240] T 9a1716c0adf34bee8a4e99341ba5788f P 186b12638a87410487467df721a3cbda [term 5 FOLLOWER]: Rejecting Update request from peer 873357742d5843579556ee8166f23f47 for earlier term 4. Current term is 5. Ops: []
W20260709 13:04:17.717828 10993 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:17.707362 11027 raft_consensus.cc:2393] T bb7d67be500947b3aef4c4245e33671c P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 10ca5ee68ad64b3283b9d235352c9d23 in current term 3: Already voted for candidate 873357742d5843579556ee8166f23f47 in this term.
I20260709 13:04:17.717820 10995 leader_election.cc:304] T 65bf0a62f4d84ef18dd6b23e082724b8 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47; no voters: 
W20260709 13:04:17.723901 10919 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:17.732951 11028 raft_consensus.cc:2468] T 732c4adeeff6422497aa7db4a82f535b P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 4.
I20260709 13:04:17.734807 10919 leader_election.cc:304] T 732c4adeeff6422497aa7db4a82f535b P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:04:17.737257 11103 raft_consensus.cc:2468] T bb7d67be500947b3aef4c4245e33671c P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 3.
I20260709 13:04:17.738979 10994 leader_election.cc:304] T bb7d67be500947b3aef4c4245e33671c P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47; no voters: 
W20260709 13:04:17.712877 12218 consensus_meta.cc:287] T 75492f86fde046c3900df3eac8292fd4 P 10ca5ee68ad64b3283b9d235352c9d23: Time spent flushing consensus metadata: real 0.554s	user 0.015s	sys 0.000s
I20260709 13:04:17.756269 11028 raft_consensus.cc:1240] T 7709387f4d3d4f9992d97a0fbca39a40 P 873357742d5843579556ee8166f23f47 [term 6 FOLLOWER]: Rejecting Update request from peer 0172fed22b6b4890a7fdc953fa128af4 for earlier term 5. Current term is 6. Ops: []
I20260709 13:04:17.757385 11028 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46640 (request call id 255) took 1495 ms. Trace:
I20260709 13:04:17.757851 11028 rpcz_store.cc:276] 0709 13:04:16.261855 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:17.755585 (+1493730us) service_pool.cc:224] Handling call
0709 13:04:17.756031 (+   446us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:17.757072 (+  1041us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:17.757315 (+   243us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:17.709008 12068 leader_election.cc:290] T 9a1716c0adf34bee8a4e99341ba5788f P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 5 election: Requested vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:17.708182 11100 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37676 (request call id 300) took 1003 ms. Trace:
I20260709 13:04:17.706409 11021 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46628 (request call id 234) took 1677 ms. Trace:
I20260709 13:04:17.714320 11094 raft_consensus.cc:2468] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 0172fed22b6b4890a7fdc953fa128af4 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 5.
W20260709 13:04:17.709227 10950 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49434 (request call id 278) took 2809 ms (client timeout 1502 ms). Trace:
I20260709 13:04:17.763101 10995 leader_election.cc:304] T 401a9cf11eac479aa445ec8c1d6e33b2 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 873357742d5843579556ee8166f23f47; no voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda
I20260709 13:04:17.761662 11028 raft_consensus.cc:1275] T 0e5d327630e5438b878035d28efe6c0c P 873357742d5843579556ee8166f23f47 [term 5 FOLLOWER]: Refusing update from remote peer 0172fed22b6b4890a7fdc953fa128af4: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260709 13:04:17.707204 11025 raft_consensus.cc:2468] T 525bfdbd6d934afd823f7d3ee3f9d052 P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 4.
I20260709 13:04:17.773496 11028 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46640 (request call id 256) took 1397 ms. Trace:
I20260709 13:04:17.738128 11972 raft_consensus.cc:697] T 6a9353d1d230416296627d7cf8b1d425 P 873357742d5843579556ee8166f23f47 [term 5 LEADER]: Becoming Leader. State: Replica: 873357742d5843579556ee8166f23f47, State: Running, Role: LEADER
I20260709 13:04:17.743798 12313 raft_consensus.cc:2804] T 732c4adeeff6422497aa7db4a82f535b P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:04:17.775067 11028 rpcz_store.cc:276] 0709 13:04:16.376387 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:17.759223 (+1382836us) service_pool.cc:224] Handling call
0709 13:04:17.759681 (+   458us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:17.773166 (+ 13485us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:17.773434 (+   268us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:17.762176 11633 raft_consensus.cc:2804] T 65bf0a62f4d84ef18dd6b23e082724b8 P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:04:17.769860 12334 raft_consensus.cc:2804] T bb7d67be500947b3aef4c4245e33671c P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:04:17.788371 11028 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a64b7e99d6c54934a7de6f6dffcf5e94" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:17.788479 10919 leader_election.cc:304] T 525bfdbd6d934afd823f7d3ee3f9d052 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:04:17.791970 10995 leader_election.cc:304] T 4f9eb06ed01f4ce68be3af8a8434665b P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 873357742d5843579556ee8166f23f47; no voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda
I20260709 13:04:17.802239 11028 raft_consensus.cc:1240] T 732c4adeeff6422497aa7db4a82f535b P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Rejecting Update request from peer 186b12638a87410487467df721a3cbda for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260709 13:04:17.775950 11921 raft_consensus.cc:2749] T 401a9cf11eac479aa445ec8c1d6e33b2 P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:04:17.768105 11100 rpcz_store.cc:276] 0709 13:04:16.705080 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:16.742096 (+ 37016us) service_pool.cc:224] Handling call
0709 13:04:17.707292 (+965196us) spinlock_profiling.cc:243] Waited 4.89 ms on lock 0x7b5800117698. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f426303db71 00007f426303da22 00007f426bf56c09 00007f426b0b0ee0 00007f426b17a0ab 00007f426b17a131 00007f426b13bfc0 00007f426b126929 00007f426b1263d2 00007f426469dbd4
0709 13:04:17.707549 (+   257us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:17.707753 (+   204us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:17.707780 (+    27us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:17.707835 (+    55us) raft_consensus.cc:1645] Marking committed up to 1
0709 13:04:17.707868 (+    33us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:17.707971 (+   103us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:17.708103 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":11252992}
I20260709 13:04:17.811611 12049 leader_election.cc:290] T a64b7e99d6c54934a7de6f6dffcf5e94 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:17.813082 12338 raft_consensus.cc:493] T f371d2565dd84667b5613848604949bb P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:17.813645 12338 raft_consensus.cc:515] T f371d2565dd84667b5613848604949bb P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:17.817456 12334 raft_consensus.cc:697] T bb7d67be500947b3aef4c4245e33671c P 873357742d5843579556ee8166f23f47 [term 3 LEADER]: Becoming Leader. State: Replica: 873357742d5843579556ee8166f23f47, State: Running, Role: LEADER
I20260709 13:04:17.818694 12334 consensus_queue.cc:237] T bb7d67be500947b3aef4c4245e33671c P 873357742d5843579556ee8166f23f47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:17.819832 11972 consensus_queue.cc:237] T 6a9353d1d230416296627d7cf8b1d425 P 873357742d5843579556ee8166f23f47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:17.821059 12196 raft_consensus.cc:493] T 7709387f4d3d4f9992d97a0fbca39a40 P 873357742d5843579556ee8166f23f47 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:17.821630 12196 raft_consensus.cc:515] T 7709387f4d3d4f9992d97a0fbca39a40 P 873357742d5843579556ee8166f23f47 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:17.786254 11633 raft_consensus.cc:493] T 65bf0a62f4d84ef18dd6b23e082724b8 P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 10ca5ee68ad64b3283b9d235352c9d23)
I20260709 13:04:17.822497 11633 raft_consensus.cc:3060] T 65bf0a62f4d84ef18dd6b23e082724b8 P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:04:17.823928 12196 leader_election.cc:290] T 7709387f4d3d4f9992d97a0fbca39a40 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:17.824770 11024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34064dc7bf3747dbb3876dd824ee0688" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
I20260709 13:04:17.825801 11024 raft_consensus.cc:2410] T 34064dc7bf3747dbb3876dd824ee0688 P 873357742d5843579556ee8166f23f47 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 186b12638a87410487467df721a3cbda for term 6 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20260709 13:04:17.826282 11024 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46642 (request call id 162) took 2098 ms (client timeout 2029 ms). Trace:
W20260709 13:04:17.775017 10950 rpcz_store.cc:269] 0709 13:04:14.899700 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:14.899874 (+   174us) service_pool.cc:224] Handling call
0709 13:04:17.709020 (+2809146us) spinlock_profiling.cc:243] Waited 825 ms on lock 0x7b540078a354. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:04:17.709143 (+   123us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1898425728}
I20260709 13:04:17.827224 11100 raft_consensus.cc:1275] T ff13ae9a47c7455f962406c6d131a3ff P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Refusing update from remote peer 873357742d5843579556ee8166f23f47: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20260709 13:04:17.829553 11095 raft_consensus.cc:2468] T 4f9eb06ed01f4ce68be3af8a8434665b P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 4.
I20260709 13:04:17.829944 12198 consensus_queue.cc:1048] T ff13ae9a47c7455f962406c6d131a3ff P 873357742d5843579556ee8166f23f47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:04:17.830888 11094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7709387f4d3d4f9992d97a0fbca39a40" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
W20260709 13:04:17.826713 11024 rpcz_store.cc:269] 0709 13:04:15.727489 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:15.730742 (+  3253us) service_pool.cc:224] Handling call
0709 13:04:17.824532 (+2093790us) spinlock_profiling.cc:243] Waited 6.12 ms on lock 0x7b58007a2898. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f426303db71 00007f426303da22 00007f426bf56c09 00007f426b0b0ee0 00007f426b17a0ab 00007f426b17a131 00007f426b1280a0 00007f426b127984 00007f42646a26d4 00007f42646a2662
0709 13:04:17.826176 (+  1644us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":14082816}
W20260709 13:04:17.834586 10947 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49424 (request call id 236) took 1877 ms (client timeout 1680 ms). Trace:
I20260709 13:04:17.834317 12071 leader_election.cc:290] T 34064dc7bf3747dbb3876dd824ee0688 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
W20260709 13:04:17.835222 10947 rpcz_store.cc:269] 0709 13:04:15.957388 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:15.962816 (+  5428us) service_pool.cc:224] Handling call
0709 13:04:17.834302 (+1871486us) spinlock_profiling.cc:243] Waited 855 ms on lock 0x7b54008b6d54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:04:17.834499 (+   197us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1965583744}
I20260709 13:04:17.838259 11974 leader_election.cc:290] T 32147c6d2d5a477aa38c597c7f78df5b P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 4 election: Requested vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:17.838460 10949 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49428 (request call id 193) took 2549 ms. Trace:
I20260709 13:04:17.848841 11020 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46642 (request call id 159) took 2124 ms. Trace:
I20260709 13:04:17.806344 12202 raft_consensus.cc:3060] T 4f9eb06ed01f4ce68be3af8a8434665b P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:04:17.850241 12296 leader_election.cc:290] T d270b9e5481245d9b5340a454a7cd10b P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:17.805724 12329 consensus_queue.cc:1048] T a5299b8d044f41b7b68057e63d17b159 P 873357742d5843579556ee8166f23f47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.048s
I20260709 13:04:17.806183 12218 raft_consensus.cc:2804] T 525bfdbd6d934afd823f7d3ee3f9d052 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260709 13:04:17.855501 11022 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46642 (request call id 161) took 2128 ms (client timeout 1721 ms). Trace:
I20260709 13:04:17.855734 12218 raft_consensus.cc:493] T 525bfdbd6d934afd823f7d3ee3f9d052 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:17.856401 12218 raft_consensus.cc:3060] T 525bfdbd6d934afd823f7d3ee3f9d052 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Advancing to term 5
W20260709 13:04:17.856180 11022 rpcz_store.cc:269] 0709 13:04:15.727000 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:15.730015 (+  3015us) service_pool.cc:224] Handling call
0709 13:04:17.855162 (+2125147us) spinlock_profiling.cc:243] Waited 1.41 s on lock 0x7b54005c3854. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:04:17.855393 (+   231us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3250458880}
I20260709 13:04:17.791793 12313 raft_consensus.cc:697] T 732c4adeeff6422497aa7db4a82f535b P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 LEADER]: Becoming Leader. State: Replica: 10ca5ee68ad64b3283b9d235352c9d23, State: Running, Role: LEADER
I20260709 13:04:17.860806 12313 consensus_queue.cc:237] T 732c4adeeff6422497aa7db4a82f535b P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:17.769630 11021 rpcz_store.cc:276] 0709 13:04:16.029406 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:17.705413 (+1676007us) service_pool.cc:224] Handling call
0709 13:04:17.705771 (+   358us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:17.705999 (+   228us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:17.706031 (+    32us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:17.706082 (+    51us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:04:17.706122 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:17.706191 (+    69us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:17.706352 (+   161us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:17.864683 11026 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "401a9cf11eac479aa445ec8c1d6e33b2" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:17.865655 11026 raft_consensus.cc:2376] T 401a9cf11eac479aa445ec8c1d6e33b2 P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 186b12638a87410487467df721a3cbda in term 3. Re-sending same reply.
I20260709 13:04:17.869495 12329 raft_consensus.cc:493] T 34064dc7bf3747dbb3876dd824ee0688 P 873357742d5843579556ee8166f23f47 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:17.870368 12314 raft_consensus.cc:493] T 868e92339c2f4508af0afbd07c814d64 P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:17.871781 11021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1ec2b50c6ad4f7c8033476208abb26d" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:17.872287 12314 raft_consensus.cc:515] T 868e92339c2f4508af0afbd07c814d64 P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:17.873013 11021 raft_consensus.cc:2410] T a1ec2b50c6ad4f7c8033476208abb26d P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0172fed22b6b4890a7fdc953fa128af4 for term 3 because replica has last-logged OpId of term: 2 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 13:04:17.874505 11025 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a1716c0adf34bee8a4e99341ba5788f" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:17.870085 12329 raft_consensus.cc:515] T 34064dc7bf3747dbb3876dd824ee0688 P 873357742d5843579556ee8166f23f47 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:17.876945 11024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f371d2565dd84667b5613848604949bb" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:17.877650 11024 raft_consensus.cc:2468] T f371d2565dd84667b5613848604949bb P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 4.
I20260709 13:04:17.877874 12338 leader_election.cc:290] T f371d2565dd84667b5613848604949bb P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:17.861380 11029 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f371d2565dd84667b5613848604949bb" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
I20260709 13:04:17.867537 11023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f9eb06ed01f4ce68be3af8a8434665b" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
    @     0x7f426273ad84 google::LogMessage::SendToLog() at ??:0
    @     0x7f426273b930 google::LogMessage::Flush() at ??:0
I20260709 13:04:17.899679 11025 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f9eb06ed01f4ce68be3af8a8434665b" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
I20260709 13:04:17.898237 11099 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37676 (request call id 301) took 1189 ms. Trace:
I20260709 13:04:18.047703 11103 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b67eff0634e540eab91cf8cee795969f" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:18.049692 11103 raft_consensus.cc:2393] T b67eff0634e540eab91cf8cee795969f P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 10ca5ee68ad64b3283b9d235352c9d23 in current term 3: Already voted for candidate 186b12638a87410487467df721a3cbda in this term.
W20260709 13:04:17.979905 10843 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 175458us
I20260709 13:04:18.051111 11974 raft_consensus.cc:493] T 57b6a2418c1342e7924195e9c1dbd357 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 186b12638a87410487467df721a3cbda)
I20260709 13:04:18.052544 11974 raft_consensus.cc:515] T 57b6a2418c1342e7924195e9c1dbd357 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:17.883462 11094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f371d2565dd84667b5613848604949bb" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:17.999980 10871 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51718 (request call id 315) took 1348 ms. Trace:
I20260709 13:04:18.005721 10873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34064dc7bf3747dbb3876dd824ee0688" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:18.057165 11094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d270b9e5481245d9b5340a454a7cd10b" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:18.058121 10873 raft_consensus.cc:2468] T 34064dc7bf3747dbb3876dd824ee0688 P 186b12638a87410487467df721a3cbda [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 6.
I20260709 13:04:18.007440 10874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7709387f4d3d4f9992d97a0fbca39a40" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:18.060734 10874 raft_consensus.cc:2468] T 7709387f4d3d4f9992d97a0fbca39a40 P 186b12638a87410487467df721a3cbda [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 5.
I20260709 13:04:18.008733 10875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f371d2565dd84667b5613848604949bb" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:18.061554 11100 raft_consensus.cc:1240] T 75492f86fde046c3900df3eac8292fd4 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Rejecting Update request from peer 10ca5ee68ad64b3283b9d235352c9d23 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260709 13:04:17.885450 11102 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "868e92339c2f4508af0afbd07c814d64" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:18.013444 10879 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51696 (request call id 296) took 1169 ms. Trace:
W20260709 13:04:18.023195 10919 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:18.030774 10993 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:17.849423 11020 rpcz_store.cc:276] 0709 13:04:15.724541 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:15.729019 (+  4478us) service_pool.cc:224] Handling call
0709 13:04:17.628289 (+1899270us) spinlock_profiling.cc:243] Waited 8.65 ms on lock 0x7b58007a2898. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f426303db71 00007f426303da22 00007f426bf56c09 00007f426b0b0ee0 00007f426b17a0ab 00007f426b17a131 00007f426b13bfc0 00007f426b126929 00007f426b1263d2 00007f426469dbd4
0709 13:04:17.628561 (+   272us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:17.629066 (+   505us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:17.629104 (+    38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:17.629717 (+   613us) log.cc:844] Serialized 26 byte log entry
0709 13:04:17.759774 (+130057us) raft_consensus.cc:1645] Marking committed up to 1
0709 13:04:17.759847 (+    73us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:17.759978 (+   131us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:17.848320 (+ 88342us) raft_consensus.cc:1685] finished
0709 13:04:17.848364 (+    44us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:17.848698 (+   334us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":103,"raft.run_cpu_time_us":124,"raft.run_wall_time_us":123,"spinlock_wait_cycles":19888000,"thread_start_us":108781,"threads_started":1,"wal-append.queue_time_us":157390}
I20260709 13:04:17.879702 11024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a610d1a71f52432e88fe56130f921fa2" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:17.889182 11027 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f46a5102fd64b45bf1c4dff863d2b4d" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
W20260709 13:04:18.106686 10921 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:18.108131 11027 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c160ddfe9b34bdaa3835d0915c1e06f" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
I20260709 13:04:18.108438 11639 raft_consensus.cc:493] T 72b111ad7b5f4753ab1d2b5ee1114840 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0172fed22b6b4890a7fdc953fa128af4)
I20260709 13:04:18.109308 11027 raft_consensus.cc:3060] T 8c160ddfe9b34bdaa3835d0915c1e06f P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Advancing to term 4
I20260709 13:04:18.109412 11639 raft_consensus.cc:515] T 72b111ad7b5f4753ab1d2b5ee1114840 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:17.900146 11028 raft_consensus.cc:1275] T d371c5e27bd843469d2e2e453734cb4a P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Refusing update from remote peer 0172fed22b6b4890a7fdc953fa128af4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20260709 13:04:18.111413 10921 leader_election.cc:304] T 7839ca2d2c13431a85f44d7c458c6351 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23; no voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47
I20260709 13:04:18.113873 11028 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65bf0a62f4d84ef18dd6b23e082724b8" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:17.842664 10949 rpcz_store.cc:276] 0709 13:04:15.288577 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:15.532023 (+243446us) service_pool.cc:224] Handling call
0709 13:04:15.542294 (+ 10271us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:15.607873 (+ 65579us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:15.610942 (+  3069us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:15.740082 (+129140us) log.cc:844] Serialized 26 byte log entry
0709 13:04:16.002263 (+262181us) raft_consensus.cc:1645] Marking committed up to 1
0709 13:04:16.002326 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:17.837909 (+1835583us) spinlock_profiling.cc:243] Waited 1.78 ms on lock 0x7b54001e23d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:04:17.837987 (+    78us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:17.838085 (+    98us) raft_consensus.cc:1685] finished
0709 13:04:17.838108 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:17.838345 (+   237us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":7902,"raft.queue_time_us":296,"raft.run_cpu_time_us":207,"raft.run_wall_time_us":6238,"spinlock_wait_cycles":4099968,"thread_start_us":195460,"threads_started":1,"wal-append.queue_time_us":524629,"wal-append.run_cpu_time_us":2521,"wal-append.run_wall_time_us":1117988}
I20260709 13:04:18.041543 11021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f9eb06ed01f4ce68be3af8a8434665b" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:18.125170 11021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a9353d1d230416296627d7cf8b1d425" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:18.125349 11098 raft_consensus.cc:2468] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 4.
I20260709 13:04:18.125670 11974 leader_election.cc:290] T 57b6a2418c1342e7924195e9c1dbd357 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:18.041887 11022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acc241e7273d49149984968596242712" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
W20260709 13:04:17.897756 11067 connection.cc:256] RPC call timeout handler was delayed by -1.3924719749999213s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18200000000000002s for a response
I20260709 13:04:18.012974 10878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "868e92339c2f4508af0afbd07c814d64" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:18.127884 10878 raft_consensus.cc:2468] T 868e92339c2f4508af0afbd07c814d64 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 2.
I20260709 13:04:17.913928 12250 leader_election.cc:290] T 63d44f6ba6434ab2870fe3190f4d7c2f P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:17.880440 12314 leader_election.cc:290] T 868e92339c2f4508af0afbd07c814d64 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:17.878394 12329 leader_election.cc:290] T 34064dc7bf3747dbb3876dd824ee0688 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:17.992690 10845 leader_election.cc:304] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda; no voters: 
I20260709 13:04:18.131490 12240 raft_consensus.cc:2749] T 7839ca2d2c13431a85f44d7c458c6351 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:04:18.133394 12299 raft_consensus.cc:2804] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 186b12638a87410487467df721a3cbda [term 5 FOLLOWER]: Leader election won for term 5
W20260709 13:04:18.131529 10845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 139273us
I20260709 13:04:18.135366 12298 consensus_queue.cc:1059] T 9a1716c0adf34bee8a4e99341ba5788f P 873357742d5843579556ee8166f23f47 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 13:04:18.135756 10845 leader_election.cc:304] T 4f9eb06ed01f4ce68be3af8a8434665b P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda; no voters: 
I20260709 13:04:18.137332 12298 raft_consensus.cc:3055] T 9a1716c0adf34bee8a4e99341ba5788f P 873357742d5843579556ee8166f23f47 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:04:18.137629 11025 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34064dc7bf3747dbb3876dd824ee0688" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:18.083515 10875 raft_consensus.cc:2468] T f371d2565dd84667b5613848604949bb P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 4.
I20260709 13:04:18.056778 10871 rpcz_store.cc:276] 0709 13:04:16.651925 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:16.788258 (+136333us) service_pool.cc:224] Handling call
0709 13:04:17.999342 (+1211084us) spinlock_profiling.cc:243] Waited 590 ms on lock 0x7b5400386654. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:04:17.999725 (+   383us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1356768640}
I20260709 13:04:18.104709 12223 consensus_queue.cc:1048] T c51782cd9d064ec0949e84e24dd4de67 P 186b12638a87410487467df721a3cbda [LEADER]: Connected to new peer: Peer: permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:04:17.888360 11029 raft_consensus.cc:2410] T f371d2565dd84667b5613848604949bb P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 186b12638a87410487467df721a3cbda for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20260709 13:04:17.866375 11026 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46642 (request call id 160) took 2141 ms (client timeout 2026 ms). Trace:
I20260709 13:04:18.146694 11022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32147c6d2d5a477aa38c597c7f78df5b" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
W20260709 13:04:18.146641 11026 rpcz_store.cc:269] 0709 13:04:15.724866 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:15.726624 (+  1758us) service_pool.cc:224] Handling call
0709 13:04:17.851027 (+2124403us) spinlock_profiling.cc:243] Waited 11.1 ms on lock 0x7b58007a2898. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f426303db71 00007f426303deee 0000000000375939 00007f426b0b11e0 00007f426b17a0c4 00007f426b17a131 00007f426b1280a0 00007f426b127984 00007f42646a26d4 00007f42646a2662
0709 13:04:17.864534 (+ 13507us) spinlock_profiling.cc:243] Waited 11.5 ms on lock 0x7b58007a2898. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f426303df8c 00007f426303def6 0000000000375939 00007f426b0b11e0 00007f426b17a0c4 00007f426b17a131 00007f426b1280a0 00007f426b127984 00007f42646a26d4 00007f42646a2662
0709 13:04:17.866277 (+  1743us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":51852928}
I20260709 13:04:18.147929 11022 raft_consensus.cc:3060] T 32147c6d2d5a477aa38c597c7f78df5b P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Advancing to term 4
W20260709 13:04:18.148931 11067 connection.cc:256] RPC call timeout handler was delayed by -1.267380548999995s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18520000000000003s for a response
I20260709 13:04:18.152163 11639 leader_election.cc:290] T 72b111ad7b5f4753ab1d2b5ee1114840 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:18.120649 12215 raft_consensus.cc:515] T 63d44f6ba6434ab2870fe3190f4d7c2f P 10ca5ee68ad64b3283b9d235352c9d23 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:18.154806 12250 raft_consensus.cc:493] T 0e5d327630e5438b878035d28efe6c0c P 186b12638a87410487467df721a3cbda [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 0172fed22b6b4890a7fdc953fa128af4)
I20260709 13:04:18.155187 11097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb7d67be500947b3aef4c4245e33671c" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:18.155905 12250 raft_consensus.cc:515] T 0e5d327630e5438b878035d28efe6c0c P 186b12638a87410487467df721a3cbda [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:18.157132 12215 leader_election.cc:290] T 63d44f6ba6434ab2870fe3190f4d7c2f P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 5 election: Requested vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:18.119061 11103 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a1716c0adf34bee8a4e99341ba5788f" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:04:18.118338 11020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a1716c0adf34bee8a4e99341ba5788f" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:18.090536 10879 rpcz_store.cc:276] 0709 13:04:16.843455 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:16.917081 (+ 73626us) service_pool.cc:224] Handling call
0709 13:04:18.012892 (+1095811us) spinlock_profiling.cc:243] Waited 450 ms on lock 0x7b5400bac554. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:04:18.013198 (+   306us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1036055424}
I20260709 13:04:18.159646 11069 leader_election.cc:304] T acc241e7273d49149984968596242712 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda; no voters: 
I20260709 13:04:18.161634 12250 leader_election.cc:290] T 0e5d327630e5438b878035d28efe6c0c P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:18.163650 12193 raft_consensus.cc:2804] T acc241e7273d49149984968596242712 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260709 13:04:18.152233 11067 connection.cc:256] RPC call timeout handler was delayed by -1.2010394569999789s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1836s for a response
I20260709 13:04:18.164316 12193 raft_consensus.cc:493] T acc241e7273d49149984968596242712 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:18.164836 12193 raft_consensus.cc:3060] T acc241e7273d49149984968596242712 P 0172fed22b6b4890a7fdc953fa128af4 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:04:18.047502 11099 rpcz_store.cc:276] 0709 13:04:16.708756 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:16.713451 (+  4695us) service_pool.cc:224] Handling call
0709 13:04:17.633113 (+919662us) spinlock_profiling.cc:243] Waited 4.7 ms on lock 0x7b5800117698. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f426303db71 00007f426303da22 00007f426bf56c09 00007f426b0b0ee0 00007f426b17a0ab 00007f426b17a131 00007f426b13bfc0 00007f426b126929 00007f426b1263d2 00007f426469dbd4
0709 13:04:17.815517 (+182404us) spinlock_profiling.cc:243] Waited 6.95 ms on lock 0x7b5800117698. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f426303de9c 00007f426303da5b 00007f426bf56c09 00007f426b0b0ee0 00007f426b17a0ab 00007f426b17a131 00007f426b13bfc0 00007f426b126929 00007f426b1263d2 00007f426469dbd4
0709 13:04:17.815790 (+   273us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:17.816048 (+   258us) raft_consensus.cc:1534] Early marking committed up to index 2
0709 13:04:17.816430 (+   382us) log.cc:844] Serialized 14 byte log entry
0709 13:04:17.897767 (+ 81337us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:17.897847 (+    80us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:04:17.897900 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:17.897995 (+    95us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:17.898152 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":467,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":39,"raft.queue_time_us":104,"raft.run_cpu_time_us":92,"raft.run_wall_time_us":92,"spinlock_wait_cycles":26795136,"thread_start_us":80505,"threads_started":1,"wal-append.queue_time_us":74656}
I20260709 13:04:18.097047 11094 raft_consensus.cc:2468] T d270b9e5481245d9b5340a454a7cd10b P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 4.
I20260709 13:04:18.169828 11937 raft_consensus.cc:493] T f34f2bc02d85444f9096e0f5b4fe4331 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:18.170938 11029 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46642 (request call id 177) took 1208 ms. Trace:
I20260709 13:04:18.170830 11937 raft_consensus.cc:515] T f34f2bc02d85444f9096e0f5b4fe4331 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:18.171478 11029 rpcz_store.cc:276] 0709 13:04:16.936042 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:17.417441 (+481399us) service_pool.cc:224] Handling call
0709 13:04:17.861190 (+443749us) spinlock_profiling.cc:243] Waited 3.38 ms on lock 0x7b58007a2898. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f426303db71 00007f426303da22 00007f426bf56c09 00007f426b0b0ee0 00007f426b17a0ab 00007f426b17a131 00007f426b1280a0 00007f426b127984 00007f42646a26d4 00007f42646a2662
0709 13:04:18.144776 (+283586us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":7776640}
I20260709 13:04:18.170849 12355 raft_consensus.cc:2804] T 4f9eb06ed01f4ce68be3af8a8434665b P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:04:18.156529 11097 raft_consensus.cc:2393] T bb7d67be500947b3aef4c4245e33671c P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 10ca5ee68ad64b3283b9d235352c9d23 in current term 3: Already voted for candidate 873357742d5843579556ee8166f23f47 in this term.
I20260709 13:04:18.137867 12298 raft_consensus.cc:740] T 9a1716c0adf34bee8a4e99341ba5788f P 873357742d5843579556ee8166f23f47 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 873357742d5843579556ee8166f23f47, State: Running, Role: LEADER
I20260709 13:04:18.146767 11024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57b6a2418c1342e7924195e9c1dbd357" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:18.183724 11937 leader_election.cc:290] T f34f2bc02d85444f9096e0f5b4fe4331 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:18.176760 12355 raft_consensus.cc:697] T 4f9eb06ed01f4ce68be3af8a8434665b P 186b12638a87410487467df721a3cbda [term 4 LEADER]: Becoming Leader. State: Replica: 186b12638a87410487467df721a3cbda, State: Running, Role: LEADER
I20260709 13:04:18.184801 11937 raft_consensus.cc:493] T c51782cd9d064ec0949e84e24dd4de67 P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:18.134397 12299 raft_consensus.cc:697] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 186b12638a87410487467df721a3cbda [term 5 LEADER]: Becoming Leader. State: Replica: 186b12638a87410487467df721a3cbda, State: Running, Role: LEADER
I20260709 13:04:18.187873 12355 consensus_queue.cc:237] T 4f9eb06ed01f4ce68be3af8a8434665b P 186b12638a87410487467df721a3cbda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
W20260709 13:04:18.191717 10919 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:18.192276 12298 consensus_queue.cc:260] T 9a1716c0adf34bee8a4e99341ba5788f P 873357742d5843579556ee8166f23f47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:18.180347 11021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f9eb06ed01f4ce68be3af8a8434665b" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
W20260709 13:04:18.200371 12373 log.cc:927] Time spent T b67eff0634e540eab91cf8cee795969f P 186b12638a87410487467df721a3cbda: Append to log took a long time: real 0.085s	user 0.001s	sys 0.000s
I20260709 13:04:18.174955 11069 leader_election.cc:304] T 9a1716c0adf34bee8a4e99341ba5788f P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda; no voters: 
I20260709 13:04:18.205394 10845 leader_election.cc:304] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda; no voters: 
I20260709 13:04:18.206214 10995 leader_election.cc:304] T 34064dc7bf3747dbb3876dd824ee0688 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:04:18.207945 12356 raft_consensus.cc:2804] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:04:18.208483 10845 leader_election.cc:304] T d270b9e5481245d9b5340a454a7cd10b P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda; no voters: 
I20260709 13:04:18.208866 12356 raft_consensus.cc:697] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 186b12638a87410487467df721a3cbda [term 4 LEADER]: Becoming Leader. State: Replica: 186b12638a87410487467df721a3cbda, State: Running, Role: LEADER
I20260709 13:04:18.207918 10995 leader_election.cc:304] T 7709387f4d3d4f9992d97a0fbca39a40 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47; no voters: 0172fed22b6b4890a7fdc953fa128af4
I20260709 13:04:18.211221 12247 raft_consensus.cc:2804] T d270b9e5481245d9b5340a454a7cd10b P 186b12638a87410487467df721a3cbda [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:04:18.211720 12247 raft_consensus.cc:493] T d270b9e5481245d9b5340a454a7cd10b P 186b12638a87410487467df721a3cbda [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:18.212178 12247 raft_consensus.cc:3060] T d270b9e5481245d9b5340a454a7cd10b P 186b12638a87410487467df721a3cbda [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:04:18.213696 11025 raft_consensus.cc:2468] T 34064dc7bf3747dbb3876dd824ee0688 P 873357742d5843579556ee8166f23f47 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 6.
I20260709 13:04:18.214691 11972 raft_consensus.cc:2804] T 34064dc7bf3747dbb3876dd824ee0688 P 873357742d5843579556ee8166f23f47 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:04:18.217336 11972 raft_consensus.cc:493] T 34064dc7bf3747dbb3876dd824ee0688 P 873357742d5843579556ee8166f23f47 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:18.217916 11972 raft_consensus.cc:3060] T 34064dc7bf3747dbb3876dd824ee0688 P 873357742d5843579556ee8166f23f47 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:04:18.221101 11020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63d44f6ba6434ab2870fe3190f4d7c2f" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
I20260709 13:04:18.226620 12356 consensus_queue.cc:237] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 186b12638a87410487467df721a3cbda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 3.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:18.229965 10921 leader_election.cc:304] T 5d439326f23747eaab3c1e7da2d6b785 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda; no voters: 
I20260709 13:04:18.231329 10919 leader_election.cc:304] T 34064dc7bf3747dbb3876dd824ee0688 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:04:18.200067 12298 raft_consensus.cc:3060] T 9a1716c0adf34bee8a4e99341ba5788f P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:04:18.244166 12329 raft_consensus.cc:2804] T 7709387f4d3d4f9992d97a0fbca39a40 P 873357742d5843579556ee8166f23f47 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:04:18.244853 12329 raft_consensus.cc:493] T 7709387f4d3d4f9992d97a0fbca39a40 P 873357742d5843579556ee8166f23f47 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:18.245229 12329 raft_consensus.cc:3060] T 7709387f4d3d4f9992d97a0fbca39a40 P 873357742d5843579556ee8166f23f47 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:04:18.246624 11099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e5d327630e5438b878035d28efe6c0c" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:18.252351 12386 raft_consensus.cc:2804] T 9a1716c0adf34bee8a4e99341ba5788f P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:04:18.260898 12386 raft_consensus.cc:493] T 9a1716c0adf34bee8a4e99341ba5788f P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:18.261940 12386 raft_consensus.cc:3060] T 9a1716c0adf34bee8a4e99341ba5788f P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:04:18.189692 12299 consensus_queue.cc:237] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 186b12638a87410487467df721a3cbda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:18.271154 11633 raft_consensus.cc:515] T 65bf0a62f4d84ef18dd6b23e082724b8 P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:18.276502 11028 raft_consensus.cc:2393] T 65bf0a62f4d84ef18dd6b23e082724b8 P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 186b12638a87410487467df721a3cbda in current term 4: Already voted for candidate 873357742d5843579556ee8166f23f47 in this term.
I20260709 13:04:18.278752 12223 raft_consensus.cc:493] T 836251ebc2b74f5da88b8a8b2338e2a5 P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 10ca5ee68ad64b3283b9d235352c9d23)
I20260709 13:04:18.279726 12223 raft_consensus.cc:515] T 836251ebc2b74f5da88b8a8b2338e2a5 P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:18.283828 12223 leader_election.cc:290] T 836251ebc2b74f5da88b8a8b2338e2a5 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:18.283838 11633 leader_election.cc:290] T 65bf0a62f4d84ef18dd6b23e082724b8 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 4 election: Requested vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:18.290266 11972 raft_consensus.cc:515] T 34064dc7bf3747dbb3876dd824ee0688 P 873357742d5843579556ee8166f23f47 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:18.292758 12329 raft_consensus.cc:515] T 7709387f4d3d4f9992d97a0fbca39a40 P 873357742d5843579556ee8166f23f47 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:18.296336 11972 leader_election.cc:290] T 34064dc7bf3747dbb3876dd824ee0688 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 7 election: Requested vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:18.300024 12329 leader_election.cc:290] T 7709387f4d3d4f9992d97a0fbca39a40 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 7 election: Requested vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:18.299000 12386 raft_consensus.cc:515] T 9a1716c0adf34bee8a4e99341ba5788f P 0172fed22b6b4890a7fdc953fa128af4 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:18.304214 12386 leader_election.cc:290] T 9a1716c0adf34bee8a4e99341ba5788f P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 5 election: Requested vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:18.306739 11020 raft_consensus.cc:3060] T 63d44f6ba6434ab2870fe3190f4d7c2f P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:04:18.185907 11937 raft_consensus.cc:515] T c51782cd9d064ec0949e84e24dd4de67 P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:18.320055 11020 raft_consensus.cc:2468] T 63d44f6ba6434ab2870fe3190f4d7c2f P 873357742d5843579556ee8166f23f47 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 5.
W20260709 13:04:18.323355 10919 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:45807, user_credentials={real_user=slave}} blocked reactor thread for 92259.7us
W20260709 13:04:18.325438 11066 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 92705.5us
W20260709 13:04:18.325781 10921 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:38967, user_credentials={real_user=slave}} blocked reactor thread for 96148.8us
I20260709 13:04:18.326836 12049 raft_consensus.cc:2804] T 5d439326f23747eaab3c1e7da2d6b785 P 10ca5ee68ad64b3283b9d235352c9d23 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:04:18.327629 12049 raft_consensus.cc:697] T 5d439326f23747eaab3c1e7da2d6b785 P 10ca5ee68ad64b3283b9d235352c9d23 [term 5 LEADER]: Becoming Leader. State: Replica: 10ca5ee68ad64b3283b9d235352c9d23, State: Running, Role: LEADER
I20260709 13:04:18.326761 12286 raft_consensus.cc:2804] T 34064dc7bf3747dbb3876dd824ee0688 P 10ca5ee68ad64b3283b9d235352c9d23 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:04:18.321645 11937 leader_election.cc:290] T c51782cd9d064ec0949e84e24dd4de67 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:18.337558 10995 leader_election.cc:304] T 868e92339c2f4508af0afbd07c814d64 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47; no voters: 0172fed22b6b4890a7fdc953fa128af4
I20260709 13:04:18.339744 10919 leader_election.cc:304] T 63d44f6ba6434ab2870fe3190f4d7c2f P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:04:18.347499 11103 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "836251ebc2b74f5da88b8a8b2338e2a5" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:18.348871 11103 raft_consensus.cc:2393] T 836251ebc2b74f5da88b8a8b2338e2a5 P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 186b12638a87410487467df721a3cbda in current term 4: Already voted for candidate 0172fed22b6b4890a7fdc953fa128af4 in this term.
I20260709 13:04:18.350943 11097 raft_consensus.cc:1275] T 8c160ddfe9b34bdaa3835d0915c1e06f P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Refusing update from remote peer 186b12638a87410487467df721a3cbda: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 13:04:18.355376 10953 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75492f86fde046c3900df3eac8292fd4" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:18.356370 10953 raft_consensus.cc:2468] T 75492f86fde046c3900df3eac8292fd4 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 3.
I20260709 13:04:18.358215 10947 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "525bfdbd6d934afd823f7d3ee3f9d052" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:04:18.358364 10955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57b6a2418c1342e7924195e9c1dbd357" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:18.358374 11022 raft_consensus.cc:2468] T 32147c6d2d5a477aa38c597c7f78df5b P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 4.
I20260709 13:04:18.359511 10955 raft_consensus.cc:2410] T 57b6a2418c1342e7924195e9c1dbd357 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 873357742d5843579556ee8166f23f47 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 13:04:18.361635 10951 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff13ae9a47c7455f962406c6d131a3ff" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:04:18.362629 10951 raft_consensus.cc:2376] T ff13ae9a47c7455f962406c6d131a3ff P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 873357742d5843579556ee8166f23f47 in term 4. Re-sending same reply.
I20260709 13:04:18.364538 10844 leader_election.cc:304] T 30f628e3d7914d609e7f1d980a74c654 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:04:18.331637 12286 raft_consensus.cc:493] T 34064dc7bf3747dbb3876dd824ee0688 P 10ca5ee68ad64b3283b9d235352c9d23 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:18.379930 12292 raft_consensus.cc:493] T 5de5f0581e3b41dea729363d6ca563bb P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 10ca5ee68ad64b3283b9d235352c9d23)
I20260709 13:04:18.379382 10774 catalog_manager.cc:5697] T 401a9cf11eac479aa445ec8c1d6e33b2 P 186b12638a87410487467df721a3cbda reported cstate change: term changed from 0 to 1, leader changed from <none> to 186b12638a87410487467df721a3cbda (127.0.107.193). New cstate: current_term: 1 leader_uuid: "186b12638a87410487467df721a3cbda" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } health_report { overall_health: HEALTHY } } }
I20260709 13:04:18.380831 12292 raft_consensus.cc:515] T 5de5f0581e3b41dea729363d6ca563bb P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:18.383639 11097 raft_consensus.cc:1275] T e5ba72ccd854430baadbc636dfd17136 P 0172fed22b6b4890a7fdc953fa128af4 [term 5 FOLLOWER]: Refusing update from remote peer 873357742d5843579556ee8166f23f47: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260709 13:04:18.383769 12215 raft_consensus.cc:2804] T 63d44f6ba6434ab2870fe3190f4d7c2f P 10ca5ee68ad64b3283b9d235352c9d23 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:04:18.384608 12215 raft_consensus.cc:697] T 63d44f6ba6434ab2870fe3190f4d7c2f P 10ca5ee68ad64b3283b9d235352c9d23 [term 5 LEADER]: Becoming Leader. State: Replica: 10ca5ee68ad64b3283b9d235352c9d23, State: Running, Role: LEADER
W20260709 13:04:18.385917 10826 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 13:04:18.328796 12049 consensus_queue.cc:237] T 5d439326f23747eaab3c1e7da2d6b785 P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:18.382683 10774 catalog_manager.cc:5697] T 8c160ddfe9b34bdaa3835d0915c1e06f P 186b12638a87410487467df721a3cbda reported cstate change: term changed from 0 to 2, leader changed from <none> to 186b12638a87410487467df721a3cbda (127.0.107.193). New cstate: current_term: 2 leader_uuid: "186b12638a87410487467df721a3cbda" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } health_report { overall_health: HEALTHY } } }
I20260709 13:04:18.395107 11100 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d439326f23747eaab3c1e7da2d6b785" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:04:18.396521 11100 raft_consensus.cc:3060] T 5d439326f23747eaab3c1e7da2d6b785 P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:04:18.398528 12286 raft_consensus.cc:3060] T 34064dc7bf3747dbb3876dd824ee0688 P 10ca5ee68ad64b3283b9d235352c9d23 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:04:18.398789 11103 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7709387f4d3d4f9992d97a0fbca39a40" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:04:18.399067 11098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f1466124dbd4f6ebb3c4d62ca48bf96" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:04:18.387463 10954 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a977842ba4b744f4bb2febd55d286006" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
W20260709 13:04:18.382333 10994 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 45783.7us
I20260709 13:04:18.408470 11099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a69f32c53f24a50b7852d143665d7b4" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:18.394659 10774 catalog_manager.cc:5697] T 4f9eb06ed01f4ce68be3af8a8434665b P 186b12638a87410487467df721a3cbda reported cstate change: term changed from 1 to 2, leader changed from 873357742d5843579556ee8166f23f47 (127.0.107.195) to 186b12638a87410487467df721a3cbda (127.0.107.193). New cstate: current_term: 2 leader_uuid: "186b12638a87410487467df721a3cbda" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } health_report { overall_health: HEALTHY } } }
I20260709 13:04:18.400144 11098 raft_consensus.cc:3060] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 0172fed22b6b4890a7fdc953fa128af4 [term 5 FOLLOWER]: Advancing to term 6
W20260709 13:04:18.422837 10844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:45807, user_credentials={real_user=slave}} blocked reactor thread for 140680us
I20260709 13:04:18.429316 10954 raft_consensus.cc:3060] T a977842ba4b744f4bb2febd55d286006 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:04:18.430176 10919 leader_election.cc:304] T 32147c6d2d5a477aa38c597c7f78df5b P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:04:18.428588 11027 raft_consensus.cc:2468] T 8c160ddfe9b34bdaa3835d0915c1e06f P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 4.
W20260709 13:04:18.430383 11067 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:45807, user_credentials={real_user=slave}} blocked reactor thread for 189284us
I20260709 13:04:18.441843 11102 raft_consensus.cc:1275] T 87cd8f5097624c8692f1d83544d8881e P 0172fed22b6b4890a7fdc953fa128af4 [term 5 FOLLOWER]: Refusing update from remote peer 186b12638a87410487467df721a3cbda: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260709 13:04:18.386055 12215 consensus_queue.cc:237] T 63d44f6ba6434ab2870fe3190f4d7c2f P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
W20260709 13:04:18.460175 10919 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:45807, user_credentials={real_user=slave}} blocked reactor thread for 84445.8us
I20260709 13:04:18.439035 12329 raft_consensus.cc:2804] T 868e92339c2f4508af0afbd07c814d64 P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:04:18.468991 10774 catalog_manager.cc:5697] T e89e640169e14e2d98f968f76027daa7 P 186b12638a87410487467df721a3cbda reported cstate change: term changed from 1 to 3, leader changed from 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194) to 186b12638a87410487467df721a3cbda (127.0.107.193). New cstate: current_term: 3 leader_uuid: "186b12638a87410487467df721a3cbda" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } health_report { overall_health: UNKNOWN } } }
I20260709 13:04:18.478817 12296 raft_consensus.cc:2804] T 30f628e3d7914d609e7f1d980a74c654 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:04:18.479389 12296 raft_consensus.cc:493] T 30f628e3d7914d609e7f1d980a74c654 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Starting leader election (detected failure of leader 10ca5ee68ad64b3283b9d235352c9d23)
I20260709 13:04:18.479830 12296 raft_consensus.cc:3060] T 30f628e3d7914d609e7f1d980a74c654 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Advancing to term 2
W20260709 13:04:18.481402 10995 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:38967, user_credentials={real_user=slave}} blocked reactor thread for 239273us
I20260709 13:04:18.464808 12329 raft_consensus.cc:493] T 868e92339c2f4508af0afbd07c814d64 P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:18.487373 10951 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65bf0a62f4d84ef18dd6b23e082724b8" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:18.461251 11097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32147c6d2d5a477aa38c597c7f78df5b" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:18.496652 11103 raft_consensus.cc:1275] T bb7d67be500947b3aef4c4245e33671c P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Refusing update from remote peer 873357742d5843579556ee8166f23f47: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 13:04:18.470536 11099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bac7e5040f984ba2b7146bd36af7267e" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:18.499745 11102 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5a5f5a608504650b48fd27f163faed6" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:18.494283 10774 catalog_manager.cc:5697] T 4b37f8fca8dc42718b2a42f0883a956f P 186b12638a87410487467df721a3cbda reported cstate change: term changed from 1 to 2, leader changed from 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194) to 873357742d5843579556ee8166f23f47 (127.0.107.195). New cstate: current_term: 2 leader_uuid: "873357742d5843579556ee8166f23f47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } }
I20260709 13:04:18.495327 12329 raft_consensus.cc:3060] T 868e92339c2f4508af0afbd07c814d64 P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Advancing to term 4
W20260709 13:04:18.516695 10844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:45807, user_credentials={real_user=slave}} blocked reactor thread for 88561.3us
I20260709 13:04:18.520700 11103 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5ba72ccd854430baadbc636dfd17136" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:18.521484 11103 raft_consensus.cc:2468] T e5ba72ccd854430baadbc636dfd17136 P 0172fed22b6b4890a7fdc953fa128af4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 5.
I20260709 13:04:18.512688 12292 leader_election.cc:290] T 5de5f0581e3b41dea729363d6ca563bb P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:18.527650 10774 catalog_manager.cc:5697] T f371d2565dd84667b5613848604949bb P 186b12638a87410487467df721a3cbda reported cstate change: term changed from 0 to 2, leader changed from <none> to 186b12638a87410487467df721a3cbda (127.0.107.193). New cstate: current_term: 2 leader_uuid: "186b12638a87410487467df721a3cbda" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } health_report { overall_health: HEALTHY } } }
I20260709 13:04:18.517166 11972 consensus_queue.cc:1048] T bb7d67be500947b3aef4c4245e33671c P 873357742d5843579556ee8166f23f47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.018s
I20260709 13:04:18.506301 12220 consensus_queue.cc:1059] T 868e92339c2f4508af0afbd07c814d64 P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.019s
I20260709 13:04:18.538512 12313 raft_consensus.cc:2804] T 32147c6d2d5a477aa38c597c7f78df5b P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:04:18.538906 11103 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5de5f0581e3b41dea729363d6ca563bb" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:18.540032 11103 raft_consensus.cc:2468] T 5de5f0581e3b41dea729363d6ca563bb P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 4.
I20260709 13:04:18.541483 11103 raft_consensus.cc:3055] T 6a9353d1d230416296627d7cf8b1d425 P 0172fed22b6b4890a7fdc953fa128af4 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:04:18.541795 11103 raft_consensus.cc:740] T 6a9353d1d230416296627d7cf8b1d425 P 0172fed22b6b4890a7fdc953fa128af4 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 0172fed22b6b4890a7fdc953fa128af4, State: Running, Role: LEADER
I20260709 13:04:18.517547 12326 consensus_queue.cc:1048] T e5ba72ccd854430baadbc636dfd17136 P 873357742d5843579556ee8166f23f47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.113s
I20260709 13:04:18.529963 11102 raft_consensus.cc:2468] T e5a5f5a608504650b48fd27f163faed6 P 0172fed22b6b4890a7fdc953fa128af4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 5.
I20260709 13:04:18.546042 11099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a154cc8b181f460abf4880064379948b" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:18.546077 12292 consensus_queue.cc:1048] T 8c160ddfe9b34bdaa3835d0915c1e06f P 186b12638a87410487467df721a3cbda [LEADER]: Connected to new peer: Peer: permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:04:18.547634 11099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f46a5102fd64b45bf1c4dff863d2b4d" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:18.549022 12292 raft_consensus.cc:493] T 72b111ad7b5f4753ab1d2b5ee1114840 P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0172fed22b6b4890a7fdc953fa128af4)
I20260709 13:04:18.549886 12292 raft_consensus.cc:515] T 72b111ad7b5f4753ab1d2b5ee1114840 P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:18.552085 12247 raft_consensus.cc:515] T d270b9e5481245d9b5340a454a7cd10b P 186b12638a87410487467df721a3cbda [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:18.553231 12292 leader_election.cc:290] T 72b111ad7b5f4753ab1d2b5ee1114840 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:18.523196 11097 raft_consensus.cc:2468] T 32147c6d2d5a477aa38c597c7f78df5b P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 2.
I20260709 13:04:18.554729 10845 leader_election.cc:304] T 5de5f0581e3b41dea729363d6ca563bb P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda; no voters: 
I20260709 13:04:18.536144 11067 leader_election.cc:304] T 75492f86fde046c3900df3eac8292fd4 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47
I20260709 13:04:18.557538 10995 leader_election.cc:400] T 8c160ddfe9b34bdaa3835d0915c1e06f P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 186b12638a87410487467df721a3cbda (127.0.107.193:38967) with higher term. Message: Invalid argument: T 8c160ddfe9b34bdaa3835d0915c1e06f P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 873357742d5843579556ee8166f23f47 for earlier term 3. Current term is 4.
I20260709 13:04:18.558025 10995 leader_election.cc:403] T 8c160ddfe9b34bdaa3835d0915c1e06f P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:04:18.556463 11100 raft_consensus.cc:2468] T 5d439326f23747eaab3c1e7da2d6b785 P 0172fed22b6b4890a7fdc953fa128af4 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 5.
I20260709 13:04:18.564141 11103 consensus_queue.cc:260] T 6a9353d1d230416296627d7cf8b1d425 P 0172fed22b6b4890a7fdc953fa128af4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:18.566452 11103 raft_consensus.cc:3060] T 6a9353d1d230416296627d7cf8b1d425 P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:04:18.565217 12292 consensus_queue.cc:1059] T 732c4adeeff6422497aa7db4a82f535b P 186b12638a87410487467df721a3cbda [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:04:18.575801 10954 raft_consensus.cc:2410] T a977842ba4b744f4bb2febd55d286006 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 873357742d5843579556ee8166f23f47 for term 3 because replica has last-logged OpId of term: 2 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260709 13:04:18.540081 11096 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37676 (request call id 302) took 1808 ms. Trace:
I20260709 13:04:18.562281 12313 raft_consensus.cc:697] T 32147c6d2d5a477aa38c597c7f78df5b P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 LEADER]: Becoming Leader. State: Replica: 10ca5ee68ad64b3283b9d235352c9d23, State: Running, Role: LEADER
I20260709 13:04:18.588268 10951 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2567c2d84e4648d7ba5dc23a3df32817" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
W20260709 13:04:18.594483 12218 consensus_meta.cc:287] T 525bfdbd6d934afd823f7d3ee3f9d052 P 10ca5ee68ad64b3283b9d235352c9d23: Time spent flushing consensus metadata: real 0.643s	user 0.022s	sys 0.005s
I20260709 13:04:18.620826 11102 raft_consensus.cc:1275] T 401a9cf11eac479aa445ec8c1d6e33b2 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Refusing update from remote peer 186b12638a87410487467df721a3cbda: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 13:04:18.620733 12218 raft_consensus.cc:515] T 525bfdbd6d934afd823f7d3ee3f9d052 P 10ca5ee68ad64b3283b9d235352c9d23 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:18.624082 10947 raft_consensus.cc:2368] T 525bfdbd6d934afd823f7d3ee3f9d052 P 10ca5ee68ad64b3283b9d235352c9d23 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 873357742d5843579556ee8166f23f47 for earlier term 4. Current term is 5.
I20260709 13:04:18.624935 12397 raft_consensus.cc:2804] T 5de5f0581e3b41dea729363d6ca563bb P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:04:18.625469 12397 raft_consensus.cc:493] T 5de5f0581e3b41dea729363d6ca563bb P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Starting leader election (detected failure of leader 10ca5ee68ad64b3283b9d235352c9d23)
I20260709 13:04:18.625869 12397 raft_consensus.cc:3060] T 5de5f0581e3b41dea729363d6ca563bb P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:04:18.579838 11096 rpcz_store.cc:276] 0709 13:04:16.709174 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:16.713508 (+  4334us) service_pool.cc:224] Handling call
0709 13:04:17.693416 (+979908us) spinlock_profiling.cc:243] Waited 13.9 ms on lock 0x7b5800117698. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f426303db71 00007f426303deee 0000000000375939 00007f426b0b11e0 00007f426b17a0c4 00007f426b17a131 00007f426b13bfc0 00007f426b126929 00007f426b1263d2 00007f426469dbd4
0709 13:04:17.693612 (+   196us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:17.694087 (+   475us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:17.694123 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:17.694795 (+   672us) log.cc:844] Serialized 26 byte log entry
0709 13:04:17.909183 (+214388us) raft_consensus.cc:1645] Marking committed up to 3
0709 13:04:18.145914 (+236731us) log.cc:844] Serialized 14 byte log entry
0709 13:04:18.452888 (+306974us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:18.471385 (+ 18497us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:18.477904 (+  6519us) raft_consensus.cc:1685] finished
0709 13:04:18.497331 (+ 19427us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:18.517489 (+ 20158us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":83432,"prepare.run_cpu_time_us":72,"prepare.run_wall_time_us":73,"raft.queue_time_us":85371,"raft.run_cpu_time_us":374,"raft.run_wall_time_us":372,"spinlock_wait_cycles":31855488,"thread_start_us":427662,"threads_started":3,"wal-append.queue_time_us":232000}
W20260709 13:04:18.635628 11067 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:45807, user_credentials={real_user=slave}} blocked reactor thread for 115562us
I20260709 13:04:18.637863 11067 leader_election.cc:304] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:04:18.639032 11101 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d7ce47b9a92d43dd88d95bcda5038597" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:04:18.640573 11097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32147c6d2d5a477aa38c597c7f78df5b" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:04:18.641319 11097 raft_consensus.cc:3060] T 32147c6d2d5a477aa38c597c7f78df5b P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Advancing to term 4
I20260709 13:04:18.594911 11100 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4cd3d7dfbbf4096908ff11e9dc57141" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
W20260709 13:04:18.595836 12392 log.cc:927] Time spent T 0e5d327630e5438b878035d28efe6c0c P 0172fed22b6b4890a7fdc953fa128af4: Append to log took a long time: real 0.059s	user 0.000s	sys 0.000s
I20260709 13:04:18.599030 10844 leader_election.cc:304] T 2567c2d84e4648d7ba5dc23a3df32817 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:04:18.583292 12247 leader_election.cc:290] T d270b9e5481245d9b5340a454a7cd10b P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 6 election: Requested vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
W20260709 13:04:18.602602 12202 consensus_meta.cc:287] T 4f9eb06ed01f4ce68be3af8a8434665b P 873357742d5843579556ee8166f23f47: Time spent flushing consensus metadata: real 0.706s	user 0.016s	sys 0.011s
I20260709 13:04:18.582957 11972 raft_consensus.cc:2749] T 8c160ddfe9b34bdaa3835d0915c1e06f P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer 186b12638a87410487467df721a3cbda (127.0.107.193:38967) with higher term. Message: Invalid argument: T 8c160ddfe9b34bdaa3835d0915c1e06f P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 873357742d5843579556ee8166f23f47 for earlier term 3. Current term is 4.
I20260709 13:04:18.643124 12202 raft_consensus.cc:2749] T 4f9eb06ed01f4ce68be3af8a8434665b P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:04:18.644143 12292 raft_consensus.cc:3055] T 732c4adeeff6422497aa7db4a82f535b P 186b12638a87410487467df721a3cbda [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:04:18.644337 11023 raft_consensus.cc:2468] T 4f9eb06ed01f4ce68be3af8a8434665b P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 4.
I20260709 13:04:18.644606 12292 raft_consensus.cc:740] T 732c4adeeff6422497aa7db4a82f535b P 186b12638a87410487467df721a3cbda [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 186b12638a87410487467df721a3cbda, State: Running, Role: LEADER
I20260709 13:04:18.645476 11020 raft_consensus.cc:1240] T 65bf0a62f4d84ef18dd6b23e082724b8 P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Rejecting Update request from peer 10ca5ee68ad64b3283b9d235352c9d23 for earlier term 3. Current term is 4. Ops: []
W20260709 13:04:18.647053 10844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:45807, user_credentials={real_user=slave}} blocked reactor thread for 48170.1us
W20260709 13:04:18.647637 11023 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46642 (request call id 163) took 2919 ms (client timeout 1611 ms). Trace:
I20260709 13:04:18.603646 12385 raft_consensus.cc:2749] T 75492f86fde046c3900df3eac8292fd4 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20260709 13:04:18.611652 10995 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:38967, user_credentials={real_user=slave}} blocked reactor thread for 93462.6us
W20260709 13:04:18.648300 12395 log.cc:927] Time spent T a5299b8d044f41b7b68057e63d17b159 P 10ca5ee68ad64b3283b9d235352c9d23: Append to log took a long time: real 0.069s	user 0.000s	sys 0.000s
I20260709 13:04:18.576654 12355 consensus_queue.cc:1048] T 87cd8f5097624c8692f1d83544d8881e P 186b12638a87410487467df721a3cbda [LEADER]: Connected to new peer: Peer: permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.076s
I20260709 13:04:18.654036 10995 leader_election.cc:304] T f371d2565dd84667b5613848604949bb P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47; no voters: 0172fed22b6b4890a7fdc953fa128af4
I20260709 13:04:18.653813 12193 raft_consensus.cc:515] T acc241e7273d49149984968596242712 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
W20260709 13:04:18.656250 10919 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:18.648367 12223 raft_consensus.cc:2804] T 2567c2d84e4648d7ba5dc23a3df32817 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:04:18.665459 11639 consensus_queue.cc:1059] T 65bf0a62f4d84ef18dd6b23e082724b8 P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.010s
I20260709 13:04:18.675046 11102 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a64b7e99d6c54934a7de6f6dffcf5e94" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:18.645859 12292 consensus_queue.cc:260] T 732c4adeeff6422497aa7db4a82f535b P 186b12638a87410487467df721a3cbda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:18.679303 11067 leader_election.cc:304] T 7757ecbf402e47bfbc93d0447741e26c P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4; no voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47
I20260709 13:04:18.680980 11639 raft_consensus.cc:3055] T 65bf0a62f4d84ef18dd6b23e082724b8 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:04:18.682262 11639 raft_consensus.cc:740] T 65bf0a62f4d84ef18dd6b23e082724b8 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 10ca5ee68ad64b3283b9d235352c9d23, State: Running, Role: LEADER
W20260709 13:04:18.682615 10919 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:18.648447 11023 rpcz_store.cc:269] 0709 13:04:15.727942 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:15.730208 (+  2266us) service_pool.cc:224] Handling call
0709 13:04:17.847161 (+2116953us) spinlock_profiling.cc:243] Waited 16.1 ms on lock 0x7b58007a2898. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f426303db71 00007f426303da22 00007f426bf56c09 00007f426b0b0ee0 00007f426b17a0ab 00007f426b17a131 00007f426b1280a0 00007f426b127984 00007f42646a26d4 00007f42646a2662
0709 13:04:17.867351 (+ 20190us) spinlock_profiling.cc:243] Waited 19.2 ms on lock 0x7b58007a2898. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f426303de9c 00007f426303da5b 00007f426bf56c09 00007f426b0b0ee0 00007f426b17a0ab 00007f426b17a131 00007f426b1280a0 00007f426b127984 00007f42646a26d4 00007f42646a2662
0709 13:04:18.647268 (+779917us) spinlock_profiling.cc:243] Waited 601 ms on lock 0x7b54004e02d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:04:18.647470 (+   202us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1463438336}
I20260709 13:04:18.713829 11103 raft_consensus.cc:1275] T 6a9353d1d230416296627d7cf8b1d425 P 0172fed22b6b4890a7fdc953fa128af4 [term 5 FOLLOWER]: Refusing update from remote peer 873357742d5843579556ee8166f23f47: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260709 13:04:18.674458 12223 raft_consensus.cc:493] T 2567c2d84e4648d7ba5dc23a3df32817 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Starting leader election (detected failure of leader 10ca5ee68ad64b3283b9d235352c9d23)
I20260709 13:04:18.678623 11098 raft_consensus.cc:2468] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 0172fed22b6b4890a7fdc953fa128af4 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 6.
I20260709 13:04:18.719049 12220 raft_consensus.cc:2749] T 7757ecbf402e47bfbc93d0447741e26c P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260709 13:04:18.719751 11067 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:45807, user_credentials={real_user=slave}} blocked reactor thread for 56693.8us
I20260709 13:04:18.715492 12223 raft_consensus.cc:3060] T 2567c2d84e4648d7ba5dc23a3df32817 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Advancing to term 3
W20260709 13:04:18.730743 10844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:45807, user_credentials={real_user=slave}} blocked reactor thread for 82853.8us
W20260709 13:04:18.733132 10995 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:38967, user_credentials={real_user=slave}} blocked reactor thread for 79252.3us
I20260709 13:04:18.712878 11096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "525bfdbd6d934afd823f7d3ee3f9d052" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
    @     0x7f4262740a6b google::LogMessageFatal::~LogMessageFatal() at ??:0
I20260709 13:04:18.716207 11094 raft_consensus.cc:1240] T acc241e7273d49149984968596242712 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Rejecting Update request from peer 873357742d5843579556ee8166f23f47 for earlier term 1. Current term is 2. Ops: []
I20260709 13:04:18.727471 12053 raft_consensus.cc:2804] T f371d2565dd84667b5613848604949bb P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:04:18.724488 12326 raft_consensus.cc:493] T a1ec2b50c6ad4f7c8033476208abb26d P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:18.737772 11639 consensus_queue.cc:260] T 65bf0a62f4d84ef18dd6b23e082724b8 P 10ca5ee68ad64b3283b9d235352c9d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:18.727563 12215 raft_consensus.cc:493] T 0f9df2bc281f4866b6da5da94a0defc7 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 186b12638a87410487467df721a3cbda)
I20260709 13:04:18.752679 12390 raft_consensus.cc:2764] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 0172fed22b6b4890a7fdc953fa128af4 [term 6 FOLLOWER]: Leader election decision vote started in defunct term 5: won
W20260709 13:04:18.746843 12298 consensus_meta.cc:287] T 9a1716c0adf34bee8a4e99341ba5788f P 873357742d5843579556ee8166f23f47: Time spent flushing consensus metadata: real 0.505s	user 0.002s	sys 0.014s
I20260709 13:04:18.742542 12053 raft_consensus.cc:493] T f371d2565dd84667b5613848604949bb P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:18.762936 12403 raft_consensus.cc:493] T 87cd8f5097624c8692f1d83544d8881e P 10ca5ee68ad64b3283b9d235352c9d23 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:18.763846 12403 raft_consensus.cc:515] T 87cd8f5097624c8692f1d83544d8881e P 10ca5ee68ad64b3283b9d235352c9d23 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:18.766499 10949 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34064dc7bf3747dbb3876dd824ee0688" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:18.767946 12215 raft_consensus.cc:515] T 0f9df2bc281f4866b6da5da94a0defc7 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:18.747269 12049 raft_consensus.cc:493] T 11de93cd9b9d457489ff1cb80f976c9a P 10ca5ee68ad64b3283b9d235352c9d23 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:18.740077 12313 consensus_queue.cc:237] T 32147c6d2d5a477aa38c597c7f78df5b P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:18.741786 11096 raft_consensus.cc:2468] T 525bfdbd6d934afd823f7d3ee3f9d052 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 3.
I20260709 13:04:18.738469 12286 raft_consensus.cc:515] T 34064dc7bf3747dbb3876dd824ee0688 P 10ca5ee68ad64b3283b9d235352c9d23 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:18.749166 12292 raft_consensus.cc:3060] T 732c4adeeff6422497aa7db4a82f535b P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:04:18.750564 12218 leader_election.cc:290] T 525bfdbd6d934afd823f7d3ee3f9d052 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 5 election: Requested vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:18.802170 12240 raft_consensus.cc:493] T a977842ba4b744f4bb2febd55d286006 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:18.804452 10874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a610d1a71f52432e88fe56130f921fa2" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:18.783442 12053 raft_consensus.cc:3060] T f371d2565dd84667b5613848604949bb P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:04:18.810755 10953 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65bf0a62f4d84ef18dd6b23e082724b8" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:04:18.811862 10875 raft_consensus.cc:1240] T 6a9353d1d230416296627d7cf8b1d425 P 186b12638a87410487467df721a3cbda [term 5 FOLLOWER]: Rejecting Update request from peer 0172fed22b6b4890a7fdc953fa128af4 for earlier term 4. Current term is 5. Ops: []
I20260709 13:04:18.813697 12215 leader_election.cc:290] T 0f9df2bc281f4866b6da5da94a0defc7 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:18.819684 12385 raft_consensus.cc:493] T 87cd8f5097624c8692f1d83544d8881e P 0172fed22b6b4890a7fdc953fa128af4 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:18.820055 11067 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:45807, user_credentials={real_user=slave}} blocked reactor thread for 96783.2us
I20260709 13:04:18.755708 12326 raft_consensus.cc:515] T a1ec2b50c6ad4f7c8033476208abb26d P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:18.820397 12385 raft_consensus.cc:515] T 87cd8f5097624c8692f1d83544d8881e P 0172fed22b6b4890a7fdc953fa128af4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:18.821167 11067 leader_election.cc:304] T bb7d67be500947b3aef4c4245e33671c P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:04:18.824445 10774 catalog_manager.cc:5697] T acc241e7273d49149984968596242712 P 186b12638a87410487467df721a3cbda reported cstate change: term changed from 0 to 1, leader changed from <none> to 873357742d5843579556ee8166f23f47 (127.0.107.195). New cstate: current_term: 1 leader_uuid: "873357742d5843579556ee8166f23f47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } }
I20260709 13:04:18.826392 12329 raft_consensus.cc:515] T 868e92339c2f4508af0afbd07c814d64 P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:18.828922 12329 leader_election.cc:290] T 868e92339c2f4508af0afbd07c814d64 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 4 election: Requested vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:18.822738 10949 raft_consensus.cc:2468] T 34064dc7bf3747dbb3876dd824ee0688 P 10ca5ee68ad64b3283b9d235352c9d23 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 6.
I20260709 13:04:18.830353 10774 catalog_manager.cc:5697] T 8a69f32c53f24a50b7852d143665d7b4 P 186b12638a87410487467df721a3cbda reported cstate change: term changed from 1 to 4, leader changed from 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194) to 186b12638a87410487467df721a3cbda (127.0.107.193). New cstate: current_term: 4 leader_uuid: "186b12638a87410487467df721a3cbda" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } health_report { overall_health: UNKNOWN } } }
I20260709 13:04:18.832079 12286 leader_election.cc:290] T 34064dc7bf3747dbb3876dd824ee0688 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 6 election: Requested vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:18.833679 11020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34064dc7bf3747dbb3876dd824ee0688" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
I20260709 13:04:18.834637 11020 raft_consensus.cc:2368] T 34064dc7bf3747dbb3876dd824ee0688 P 873357742d5843579556ee8166f23f47 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 10ca5ee68ad64b3283b9d235352c9d23 for earlier term 6. Current term is 7.
I20260709 13:04:18.808905 12240 raft_consensus.cc:515] T a977842ba4b744f4bb2febd55d286006 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:18.835857 12403 leader_election.cc:290] T 87cd8f5097624c8692f1d83544d8881e P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:18.836499 10919 leader_election.cc:400] T 34064dc7bf3747dbb3876dd824ee0688 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 6 election: Vote denied by peer 873357742d5843579556ee8166f23f47 (127.0.107.195:45807) with higher term. Message: Invalid argument: T 34064dc7bf3747dbb3876dd824ee0688 P 873357742d5843579556ee8166f23f47 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 10ca5ee68ad64b3283b9d235352c9d23 for earlier term 6. Current term is 7.
I20260709 13:04:18.836843 10919 leader_election.cc:403] T 34064dc7bf3747dbb3876dd824ee0688 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 6 election: Cancelling election due to peer responding with higher term
I20260709 13:04:18.837275 12240 leader_election.cc:290] T a977842ba4b744f4bb2febd55d286006 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:18.837357 10952 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34064dc7bf3747dbb3876dd824ee0688" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:04:18.839781 10946 raft_consensus.cc:3060] T bb7d67be500947b3aef4c4245e33671c P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:04:18.841712 10992 leader_election.cc:304] T 75492f86fde046c3900df3eac8292fd4 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47; no voters: 0172fed22b6b4890a7fdc953fa128af4
I20260709 13:04:18.833777 12326 leader_election.cc:290] T a1ec2b50c6ad4f7c8033476208abb26d P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:18.834861 12385 leader_election.cc:290] T 87cd8f5097624c8692f1d83544d8881e P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:18.844601 10992 leader_election.cc:304] T 57b6a2418c1342e7924195e9c1dbd357 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 873357742d5843579556ee8166f23f47; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda
I20260709 13:04:18.845675 11972 raft_consensus.cc:2749] T 57b6a2418c1342e7924195e9c1dbd357 P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:04:18.846311 12397 raft_consensus.cc:515] T 5de5f0581e3b41dea729363d6ca563bb P 186b12638a87410487467df721a3cbda [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:18.847313 12193 leader_election.cc:290] T acc241e7273d49149984968596242712 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 2 election: Requested vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:18.848825 12397 leader_election.cc:290] T 5de5f0581e3b41dea729363d6ca563bb P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 5 election: Requested vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:18.853969 10872 raft_consensus.cc:1275] T a1ec2b50c6ad4f7c8033476208abb26d P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Refusing update from remote peer 0172fed22b6b4890a7fdc953fa128af4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 13:04:18.806380 12049 raft_consensus.cc:515] T 11de93cd9b9d457489ff1cb80f976c9a P 10ca5ee68ad64b3283b9d235352c9d23 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:18.855216 11097 raft_consensus.cc:2468] T 32147c6d2d5a477aa38c597c7f78df5b P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 4.
I20260709 13:04:18.858177 12049 leader_election.cc:290] T 11de93cd9b9d457489ff1cb80f976c9a P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:18.860354 10874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "836251ebc2b74f5da88b8a8b2338e2a5" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:18.883695 10874 raft_consensus.cc:2468] T 836251ebc2b74f5da88b8a8b2338e2a5 P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 3.
I20260709 13:04:18.860685 10871 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c31881534c5e4f0089c82057c1746652" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:04:18.802300 12296 raft_consensus.cc:515] T 30f628e3d7914d609e7f1d980a74c654 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:18.862279 10875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555217d45d9b4ee492dbb3ddaf0f7c02" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:18.866660 11633 consensus_queue.cc:1048] T 6a9353d1d230416296627d7cf8b1d425 P 873357742d5843579556ee8166f23f47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 13:04:18.844161 12326 raft_consensus.cc:2804] T 75492f86fde046c3900df3eac8292fd4 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:04:18.868896 11099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "868e92339c2f4508af0afbd07c814d64" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:04:18.898293 12326 raft_consensus.cc:493] T 75492f86fde046c3900df3eac8292fd4 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 10ca5ee68ad64b3283b9d235352c9d23)
I20260709 13:04:18.869488 10872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f9eb06ed01f4ce68be3af8a8434665b" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:04:18.898792 12326 raft_consensus.cc:3060] T 75492f86fde046c3900df3eac8292fd4 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:04:18.822372 11067 leader_election.cc:304] T d71ca3a1ba6b48478aaf62741bf3cf6a P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47
I20260709 13:04:18.838191 10774 catalog_manager.cc:5697] T a5299b8d044f41b7b68057e63d17b159 P 186b12638a87410487467df721a3cbda reported cstate change: term changed from 0 to 1, leader changed from <none> to 186b12638a87410487467df721a3cbda (127.0.107.193). New cstate: current_term: 1 leader_uuid: "186b12638a87410487467df721a3cbda" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } health_report { overall_health: UNKNOWN } } }
I20260709 13:04:18.878216 10879 raft_consensus.cc:1275] T d371c5e27bd843469d2e2e453734cb4a P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Refusing update from remote peer 0172fed22b6b4890a7fdc953fa128af4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20260709 13:04:18.879592 11096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1ec2b50c6ad4f7c8033476208abb26d" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:18.880998 10955 raft_consensus.cc:1275] T ff13ae9a47c7455f962406c6d131a3ff P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Refusing update from remote peer 873357742d5843579556ee8166f23f47: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20260709 13:04:18.804811 10951 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb7d67be500947b3aef4c4245e33671c" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:04:18.886025 11639 raft_consensus.cc:3060] T 65bf0a62f4d84ef18dd6b23e082724b8 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Advancing to term 4
W20260709 13:04:18.889842 10919 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:45807, user_credentials={real_user=slave}} blocked reactor thread for 53548us
W20260709 13:04:18.891696 11069 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:18.895864 12394 raft_consensus.cc:3055] T 868e92339c2f4508af0afbd07c814d64 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:04:18.895987 12071 raft_consensus.cc:493] T 1e4f2241a94a41f38e481137f868b83d P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 186b12638a87410487467df721a3cbda)
I20260709 13:04:18.903524 10874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65bf0a62f4d84ef18dd6b23e082724b8" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:04:18.903816 10878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34064dc7bf3747dbb3876dd824ee0688" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:04:18.907897 10878 raft_consensus.cc:3060] T 34064dc7bf3747dbb3876dd824ee0688 P 186b12638a87410487467df721a3cbda [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:04:18.899361 12240 raft_consensus.cc:3060] T 34064dc7bf3747dbb3876dd824ee0688 P 10ca5ee68ad64b3283b9d235352c9d23 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:04:18.910953 10880 raft_consensus.cc:1275] T 6a9353d1d230416296627d7cf8b1d425 P 186b12638a87410487467df721a3cbda [term 5 FOLLOWER]: Refusing update from remote peer 873357742d5843579556ee8166f23f47: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260709 13:04:18.904397 10873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7709387f4d3d4f9992d97a0fbca39a40" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:04:18.913288 11020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a977842ba4b744f4bb2febd55d286006" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:18.914233 11020 raft_consensus.cc:2468] T a977842ba4b744f4bb2febd55d286006 P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 3.
I20260709 13:04:18.915648 11023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11de93cd9b9d457489ff1cb80f976c9a" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:18.916517 12296 leader_election.cc:290] T 30f628e3d7914d609e7f1d980a74c654 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 2 election: Requested vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:18.916761 10775 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:59608 (request call id 63) took 10178 ms. Trace:
I20260709 13:04:18.918392 11027 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "525bfdbd6d934afd823f7d3ee3f9d052" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
W20260709 13:04:18.919227 11069 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:18.919989 10920 connection.cc:256] RPC call timeout handler was delayed by -1.0033869419999064s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2916s for a response
I20260709 13:04:18.920782 12193 raft_consensus.cc:2749] T d71ca3a1ba6b48478aaf62741bf3cf6a P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260709 13:04:18.921227 10845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 366774us
I20260709 13:04:18.921885 10871 raft_consensus.cc:2393] T c31881534c5e4f0089c82057c1746652 P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 0172fed22b6b4890a7fdc953fa128af4 in current term 3: Already voted for candidate 186b12638a87410487467df721a3cbda in this term.
I20260709 13:04:18.922081 10919 leader_election.cc:304] T a977842ba4b744f4bb2febd55d286006 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47; no voters: 
W20260709 13:04:18.920799 11067 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:45807, user_credentials={real_user=slave}} blocked reactor thread for 98588.1us
I20260709 13:04:18.924329 11067 leader_election.cc:304] T 8c160ddfe9b34bdaa3835d0915c1e06f P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4; no voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47
I20260709 13:04:18.926041 12071 raft_consensus.cc:515] T 1e4f2241a94a41f38e481137f868b83d P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:18.926749 12390 raft_consensus.cc:2749] T 8c160ddfe9b34bdaa3835d0915c1e06f P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:04:18.929131 12071 leader_election.cc:290] T 1e4f2241a94a41f38e481137f868b83d P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:18.929736 12394 raft_consensus.cc:740] T 868e92339c2f4508af0afbd07c814d64 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0172fed22b6b4890a7fdc953fa128af4, State: Running, Role: LEADER
I20260709 13:04:18.931017 12394 consensus_queue.cc:260] T 868e92339c2f4508af0afbd07c814d64 P 0172fed22b6b4890a7fdc953fa128af4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:18.933063 12394 raft_consensus.cc:3060] T 868e92339c2f4508af0afbd07c814d64 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:04:18.935896 12223 raft_consensus.cc:515] T 2567c2d84e4648d7ba5dc23a3df32817 P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:18.938993 10874 raft_consensus.cc:3060] T 65bf0a62f4d84ef18dd6b23e082724b8 P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:04:18.939921 10946 raft_consensus.cc:1275] T bb7d67be500947b3aef4c4245e33671c P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Refusing update from remote peer 873357742d5843579556ee8166f23f47: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 13:04:18.919176 11027 raft_consensus.cc:3060] T 525bfdbd6d934afd823f7d3ee3f9d052 P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:04:18.926095 11067 leader_election.cc:304] T 32147c6d2d5a477aa38c597c7f78df5b P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:04:18.964504 10872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f34f2bc02d85444f9096e0f5b4fe4331" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:18.966022 10872 raft_consensus.cc:2410] T f34f2bc02d85444f9096e0f5b4fe4331 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0172fed22b6b4890a7fdc953fa128af4 for term 3 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
W20260709 13:04:18.966662 10919 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:45807, user_credentials={real_user=slave}} blocked reactor thread for 44700.8us
I20260709 13:04:18.968349 11096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72b111ad7b5f4753ab1d2b5ee1114840" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:18.969381 10951 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f9df2bc281f4866b6da5da94a0defc7" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:18.970621 11974 raft_consensus.cc:493] T 836251ebc2b74f5da88b8a8b2338e2a5 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:18.971100 10955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5de5f0581e3b41dea729363d6ca563bb" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:18.971230 11974 raft_consensus.cc:515] T 836251ebc2b74f5da88b8a8b2338e2a5 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:18.972668 10955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d439326f23747eaab3c1e7da2d6b785" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:04:18.973017 10954 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a69f32c53f24a50b7852d143665d7b4" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:18.973857 10954 raft_consensus.cc:2468] T 8a69f32c53f24a50b7852d143665d7b4 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 4.
I20260709 13:04:18.973937 11974 leader_election.cc:290] T 836251ebc2b74f5da88b8a8b2338e2a5 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:18.974293 10949 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f46a5102fd64b45bf1c4dff863d2b4d" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
W20260709 13:04:18.975050 11067 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:45807, user_credentials={real_user=slave}} blocked reactor thread for 49129.8us
I20260709 13:04:18.975127 10952 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f1466124dbd4f6ebb3c4d62ca48bf96" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:04:18.970223 11101 raft_consensus.cc:1275] T e5a5f5a608504650b48fd27f163faed6 P 0172fed22b6b4890a7fdc953fa128af4 [term 5 FOLLOWER]: Refusing update from remote peer 186b12638a87410487467df721a3cbda: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260709 13:04:18.976620 11097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d270b9e5481245d9b5340a454a7cd10b" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:04:18.974628 12403 raft_consensus.cc:2804] T a977842ba4b744f4bb2febd55d286006 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:04:18.977322 12403 raft_consensus.cc:493] T a977842ba4b744f4bb2febd55d286006 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:18.977736 12403 raft_consensus.cc:3060] T a977842ba4b744f4bb2febd55d286006 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:04:18.977988 12193 raft_consensus.cc:2764] T 32147c6d2d5a477aa38c597c7f78df5b P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260709 13:04:18.979132 11022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e4f2241a94a41f38e481137f868b83d" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:18.979938 11022 raft_consensus.cc:2468] T 1e4f2241a94a41f38e481137f868b83d P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 3.
I20260709 13:04:18.977340 11097 raft_consensus.cc:3060] T d270b9e5481245d9b5340a454a7cd10b P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Advancing to term 6
I20260709 13:04:18.955924 11937 raft_consensus.cc:2764] T bb7d67be500947b3aef4c4245e33671c P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260709 13:04:18.973867 10955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "836251ebc2b74f5da88b8a8b2338e2a5" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:18.986868 10878 raft_consensus.cc:2468] T 34064dc7bf3747dbb3876dd824ee0688 P 186b12638a87410487467df721a3cbda [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 7.
I20260709 13:04:18.986717 10919 leader_election.cc:304] T 1e4f2241a94a41f38e481137f868b83d P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:04:18.987584 10955 raft_consensus.cc:2468] T 836251ebc2b74f5da88b8a8b2338e2a5 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 4.
W20260709 13:04:18.996557 10994 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 130482us
I20260709 13:04:18.998333 11639 raft_consensus.cc:2804] T 1e4f2241a94a41f38e481137f868b83d P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:04:18.998941 11639 raft_consensus.cc:493] T 1e4f2241a94a41f38e481137f868b83d P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 186b12638a87410487467df721a3cbda)
I20260709 13:04:18.999686 11639 raft_consensus.cc:3060] T 1e4f2241a94a41f38e481137f868b83d P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:04:18.970397 10951 raft_consensus.cc:2410] T 0f9df2bc281f4866b6da5da94a0defc7 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0172fed22b6b4890a7fdc953fa128af4 for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 13:04:18.996781 10874 raft_consensus.cc:2468] T 65bf0a62f4d84ef18dd6b23e082724b8 P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 4.
I20260709 13:04:18.971019 10947 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb7d67be500947b3aef4c4245e33671c" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:19.005330 11101 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5de5f0581e3b41dea729363d6ca563bb" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:04:19.008762 11101 raft_consensus.cc:3060] T 5de5f0581e3b41dea729363d6ca563bb P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:04:19.013104 10947 raft_consensus.cc:2468] T bb7d67be500947b3aef4c4245e33671c P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 3.
I20260709 13:04:19.013648 10992 leader_election.cc:304] T a977842ba4b744f4bb2febd55d286006 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 873357742d5843579556ee8166f23f47; no voters: 0172fed22b6b4890a7fdc953fa128af4, 10ca5ee68ad64b3283b9d235352c9d23
I20260709 13:04:19.015360 10992 leader_election.cc:304] T 2567c2d84e4648d7ba5dc23a3df32817 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 873357742d5843579556ee8166f23f47; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda
I20260709 13:04:19.016114 10950 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32147c6d2d5a477aa38c597c7f78df5b" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:19.017102 10992 leader_election.cc:400] T 525bfdbd6d934afd823f7d3ee3f9d052 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 4 election: Vote denied by peer 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903) with higher term. Message: Invalid argument: T 525bfdbd6d934afd823f7d3ee3f9d052 P 10ca5ee68ad64b3283b9d235352c9d23 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 873357742d5843579556ee8166f23f47 for earlier term 4. Current term is 5.
I20260709 13:04:19.017450 10992 leader_election.cc:403] T 525bfdbd6d934afd823f7d3ee3f9d052 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
I20260709 13:04:19.018097 10955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "525bfdbd6d934afd823f7d3ee3f9d052" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:19.029443 10955 raft_consensus.cc:2368] T 525bfdbd6d934afd823f7d3ee3f9d052 P 10ca5ee68ad64b3283b9d235352c9d23 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0172fed22b6b4890a7fdc953fa128af4 for earlier term 4. Current term is 5.
I20260709 13:04:19.022007 11097 raft_consensus.cc:2468] T d270b9e5481245d9b5340a454a7cd10b P 0172fed22b6b4890a7fdc953fa128af4 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 6.
I20260709 13:04:19.018016 10875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a1716c0adf34bee8a4e99341ba5788f" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:04:19.045765 11027 raft_consensus.cc:2468] T 525bfdbd6d934afd823f7d3ee3f9d052 P 873357742d5843579556ee8166f23f47 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 5.
I20260709 13:04:19.003113 12397 consensus_queue.cc:1048] T 401a9cf11eac479aa445ec8c1d6e33b2 P 186b12638a87410487467df721a3cbda [LEADER]: Connected to new peer: Peer: permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.037s
I20260709 13:04:19.039187 10845 leader_election.cc:304] T d270b9e5481245d9b5340a454a7cd10b P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda; no voters: 
I20260709 13:04:18.995321 12390 consensus_queue.cc:1059] T a977842ba4b744f4bb2febd55d286006 P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.042s
I20260709 13:04:19.035508 11633 raft_consensus.cc:2749] T a977842ba4b744f4bb2febd55d286006 P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 13:04:19.029629 10946 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c31881534c5e4f0089c82057c1746652" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:04:19.055711 10875 raft_consensus.cc:2393] T 9a1716c0adf34bee8a4e99341ba5788f P 186b12638a87410487467df721a3cbda [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 0172fed22b6b4890a7fdc953fa128af4 in current term 5: Already voted for candidate 186b12638a87410487467df721a3cbda in this term.
I20260709 13:04:19.069631 10874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c51782cd9d064ec0949e84e24dd4de67" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:19.071130 12387 raft_consensus.cc:3055] T a977842ba4b744f4bb2febd55d286006 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:04:19.071677 12387 raft_consensus.cc:740] T a977842ba4b744f4bb2febd55d286006 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0172fed22b6b4890a7fdc953fa128af4, State: Running, Role: LEADER
I20260709 13:04:19.072716 12387 consensus_queue.cc:260] T a977842ba4b744f4bb2febd55d286006 P 0172fed22b6b4890a7fdc953fa128af4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:18.917219 10775 rpcz_store.cc:276] 0709 13:04:08.738378 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:08.750967 (+ 12589us) service_pool.cc:224] Handling call
0709 13:04:18.916465 (+10165498us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 13:04:14.350119 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 13:04:14.350862 (+   743us) write_op.cc:432] Acquiring schema lock in shared mode
0709 13:04:14.350906 (+    44us) write_op.cc:435] Acquired schema lock
0709 13:04:14.350927 (+    21us) tablet.cc:662] Decoding operations
0709 13:04:14.352660 (+  1733us) write_op.cc:620] Acquiring the partition lock for write op
0709 13:04:14.352727 (+    67us) write_op.cc:641] Partition lock acquired for write op
0709 13:04:14.352757 (+    30us) tablet.cc:685] Acquiring locks for 18 operations
0709 13:04:14.354258 (+  1501us) tablet.cc:701] Row locks acquired
0709 13:04:14.354282 (+    24us) write_op.cc:260] PREPARE: finished
0709 13:04:14.391679 (+ 37397us) write_op.cc:270] Start()
0709 13:04:14.400168 (+  8489us) write_op.cc:276] Timestamp: P: 1783602254381717 usec, L: 0
0709 13:04:14.400198 (+    30us) op_driver.cc:348] REPLICATION: starting
0709 13:04:14.531345 (+131147us) log.cc:844] Serialized 7137 byte log entry
0709 13:04:15.249730 (+718385us) spinlock_profiling.cc:243] Waited 18.5 ms on lock 0x7b18001362a0. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f42630753f3 00007f42630777c1 00007f4263077779 00007f4263077709 00007f42630776d1 00007f42630769cd 000000000038f354 000000000038f199 00007f426305883a
0709 13:04:15.966063 (+716333us) op_driver.cc:464] REPLICATION: finished
0709 13:04:17.868437 (+1902374us) spinlock_profiling.cc:243] Waited 27.1 ms on lock 0x7b18001362a0. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f42630753f3 00007f42630777c1 00007f4263077779 00007f4263077709 00007f42630776d1 00007f42630769cd 000000000038f354 000000000038f199 00007f426305883a
0709 13:04:17.868652 (+   215us) write_op.cc:301] APPLY: starting
0709 13:04:17.868773 (+   121us) tablet.cc:1367] starting BulkCheckPresence
0709 13:04:17.869284 (+   511us) tablet.cc:1370] finished BulkCheckPresence
0709 13:04:17.869317 (+    33us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 13:04:17.919077 (+ 49760us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 13:04:17.930771 (+ 11694us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=18
0709 13:04:17.930865 (+    94us) write_op.cc:312] APPLY: finished
0709 13:04:17.931511 (+   646us) log.cc:844] Serialized 124 byte log entry
0709 13:04:18.306813 (+375302us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:04:18.307355 (+   542us) write_op.cc:454] Released schema lock
0709 13:04:18.307523 (+   168us) write_op.cc:341] FINISH: Updating metrics
0709 13:04:18.318327 (+ 10804us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:04:18.318362 (+    35us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":32,"child_traces":[["op",{"apply.queue_time_us":924756,"apply.run_cpu_time_us":24100,"apply.run_wall_time_us":449571,"num_ops":18,"prepare.queue_time_us":439008,"prepare.run_cpu_time_us":59651,"prepare.run_wall_time_us":678370,"raft.queue_time_us":151813,"raft.run_cpu_time_us":49803,"raft.run_wall_time_us":939604,"replication_time_us":1555329,"spinlock_wait_cycles":104818816,"thread_start_us":2045956,"threads_started":5,"wal-append.queue_time_us":764914,"wal-append.run_cpu_time_us":6800,"wal-append.run_wall_time_us":1726872}]]}
I20260709 13:04:19.077432 12274 raft_consensus.cc:2749] T 2567c2d84e4648d7ba5dc23a3df32817 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:04:19.078733 11633 raft_consensus.cc:2749] T 525bfdbd6d934afd823f7d3ee3f9d052 P 873357742d5843579556ee8166f23f47 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903) with higher term. Message: Invalid argument: T 525bfdbd6d934afd823f7d3ee3f9d052 P 10ca5ee68ad64b3283b9d235352c9d23 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 873357742d5843579556ee8166f23f47 for earlier term 4. Current term is 5.
I20260709 13:04:19.079568 10946 raft_consensus.cc:2368] T c31881534c5e4f0089c82057c1746652 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 0172fed22b6b4890a7fdc953fa128af4 for earlier term 3. Current term is 4.
I20260709 13:04:19.056792 11639 raft_consensus.cc:515] T 1e4f2241a94a41f38e481137f868b83d P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:19.059357 10919 leader_election.cc:304] T 525bfdbd6d934afd823f7d3ee3f9d052 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:04:19.038084 12223 leader_election.cc:290] T 2567c2d84e4648d7ba5dc23a3df32817 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 3 election: Requested vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:19.083040 12223 raft_consensus.cc:2804] T d270b9e5481245d9b5340a454a7cd10b P 186b12638a87410487467df721a3cbda [term 6 FOLLOWER]: Leader election won for term 6
I20260709 13:04:19.083657 12223 raft_consensus.cc:697] T d270b9e5481245d9b5340a454a7cd10b P 186b12638a87410487467df721a3cbda [term 6 LEADER]: Becoming Leader. State: Replica: 186b12638a87410487467df721a3cbda, State: Running, Role: LEADER
I20260709 13:04:19.084887 12223 consensus_queue.cc:237] T d270b9e5481245d9b5340a454a7cd10b P 186b12638a87410487467df721a3cbda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:19.060895 10955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c51782cd9d064ec0949e84e24dd4de67" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:19.087587 10955 raft_consensus.cc:2468] T c51782cd9d064ec0949e84e24dd4de67 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 3.
I20260709 13:04:19.086764 12286 raft_consensus.cc:2804] T 525bfdbd6d934afd823f7d3ee3f9d052 P 10ca5ee68ad64b3283b9d235352c9d23 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:04:19.029127 12250 consensus_queue.cc:1048] T e5a5f5a608504650b48fd27f163faed6 P 186b12638a87410487467df721a3cbda [LEADER]: Connected to new peer: Peer: permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.022s
I20260709 13:04:19.043282 12385 consensus_queue.cc:1059] T 7709387f4d3d4f9992d97a0fbca39a40 P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.022s
I20260709 13:04:19.092803 12385 raft_consensus.cc:3055] T 7709387f4d3d4f9992d97a0fbca39a40 P 0172fed22b6b4890a7fdc953fa128af4 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:04:19.093171 12385 raft_consensus.cc:740] T 7709387f4d3d4f9992d97a0fbca39a40 P 0172fed22b6b4890a7fdc953fa128af4 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 0172fed22b6b4890a7fdc953fa128af4, State: Running, Role: LEADER
I20260709 13:04:19.094651 10950 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f34f2bc02d85444f9096e0f5b4fe4331" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:19.097122 11639 leader_election.cc:290] T 1e4f2241a94a41f38e481137f868b83d P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 4 election: Requested vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:19.098464 12240 raft_consensus.cc:2749] T 34064dc7bf3747dbb3876dd824ee0688 P 10ca5ee68ad64b3283b9d235352c9d23 [term 7 FOLLOWER]: Leader election lost for term 6. Reason: Vote denied by peer 873357742d5843579556ee8166f23f47 (127.0.107.195:45807) with higher term. Message: Invalid argument: T 34064dc7bf3747dbb3876dd824ee0688 P 873357742d5843579556ee8166f23f47 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 10ca5ee68ad64b3283b9d235352c9d23 for earlier term 6. Current term is 7.
I20260709 13:04:19.102928 12403 raft_consensus.cc:515] T a977842ba4b744f4bb2febd55d286006 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:19.090803 12286 raft_consensus.cc:697] T 525bfdbd6d934afd823f7d3ee3f9d052 P 10ca5ee68ad64b3283b9d235352c9d23 [term 5 LEADER]: Becoming Leader. State: Replica: 10ca5ee68ad64b3283b9d235352c9d23, State: Running, Role: LEADER
I20260709 13:04:19.111073 12286 consensus_queue.cc:237] T 525bfdbd6d934afd823f7d3ee3f9d052 P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:19.103446 11067 leader_election.cc:304] T f371d2565dd84667b5613848604949bb P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47; no voters: 186b12638a87410487467df721a3cbda
I20260709 13:04:19.058671 12220 consensus_queue.cc:1048] T 0e5d327630e5438b878035d28efe6c0c P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.047s
I20260709 13:04:19.116137 10951 raft_consensus.cc:1240] T a977842ba4b744f4bb2febd55d286006 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Rejecting Update request from peer 0172fed22b6b4890a7fdc953fa128af4 for earlier term 2. Current term is 4. Ops: []
I20260709 13:04:19.115741 12326 raft_consensus.cc:515] T 75492f86fde046c3900df3eac8292fd4 P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:19.120182 11027 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e4f2241a94a41f38e481137f868b83d" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
I20260709 13:04:19.106010 10953 raft_consensus.cc:2468] T 65bf0a62f4d84ef18dd6b23e082724b8 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 4.
I20260709 13:04:19.121748 11027 raft_consensus.cc:3060] T 1e4f2241a94a41f38e481137f868b83d P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:04:19.126672 10874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87cd8f5097624c8692f1d83544d8881e" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
W20260709 13:04:19.139351 10921 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:38967, user_credentials={real_user=slave}} blocked reactor thread for 254673us
I20260709 13:04:19.094151 12385 consensus_queue.cc:260] T 7709387f4d3d4f9992d97a0fbca39a40 P 0172fed22b6b4890a7fdc953fa128af4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:19.130970 12387 raft_consensus.cc:3060] T a977842ba4b744f4bb2febd55d286006 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Advancing to term 3
W20260709 13:04:19.154687 11067 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:45807, user_credentials={real_user=slave}} blocked reactor thread for 51381.5us
I20260709 13:04:19.137179 12390 raft_consensus.cc:2804] T f371d2565dd84667b5613848604949bb P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:04:19.138916 12326 leader_election.cc:290] T 75492f86fde046c3900df3eac8292fd4 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 3 election: Requested vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:19.173079 10874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "732c4adeeff6422497aa7db4a82f535b" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:04:19.157107 11101 raft_consensus.cc:2468] T 5de5f0581e3b41dea729363d6ca563bb P 0172fed22b6b4890a7fdc953fa128af4 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 5.
W20260709 13:04:19.173949 11069 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:38967, user_credentials={real_user=slave}} blocked reactor thread for 93177.2us
I20260709 13:04:19.178038 10878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e89e640169e14e2d98f968f76027daa7" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
W20260709 13:04:19.201087 10919 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:19.205135 10845 leader_election.cc:304] T 5de5f0581e3b41dea729363d6ca563bb P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda; no voters: 
I20260709 13:04:19.164222 12240 raft_consensus.cc:493] T d270b9e5481245d9b5340a454a7cd10b P 10ca5ee68ad64b3283b9d235352c9d23 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:19.223073 11633 consensus_queue.cc:1048] T ff13ae9a47c7455f962406c6d131a3ff P 873357742d5843579556ee8166f23f47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:04:19.223090 12053 raft_consensus.cc:515] T f371d2565dd84667b5613848604949bb P 873357742d5843579556ee8166f23f47 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:19.223134 12240 raft_consensus.cc:515] T d270b9e5481245d9b5340a454a7cd10b P 10ca5ee68ad64b3283b9d235352c9d23 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:19.229300 12240 leader_election.cc:290] T d270b9e5481245d9b5340a454a7cd10b P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:19.229436 10950 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87cd8f5097624c8692f1d83544d8881e" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:19.230289 10950 raft_consensus.cc:2468] T 87cd8f5097624c8692f1d83544d8881e P 10ca5ee68ad64b3283b9d235352c9d23 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 5.
W20260709 13:04:19.231323 10919 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:19.232681 12298 consensus_queue.cc:1048] T bb7d67be500947b3aef4c4245e33671c P 873357742d5843579556ee8166f23f47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:04:19.236069 10992 leader_election.cc:304] T 65bf0a62f4d84ef18dd6b23e082724b8 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:04:19.217737 11027 raft_consensus.cc:2468] T 1e4f2241a94a41f38e481137f868b83d P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 4.
I20260709 13:04:19.164543 12193 raft_consensus.cc:493] T 401a9cf11eac479aa445ec8c1d6e33b2 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:19.244376 10919 leader_election.cc:304] T 1e4f2241a94a41f38e481137f868b83d P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:04:19.164125 12390 raft_consensus.cc:493] T f371d2565dd84667b5613848604949bb P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:19.252682 12390 raft_consensus.cc:3060] T f371d2565dd84667b5613848604949bb P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:04:19.254187 12193 raft_consensus.cc:515] T 401a9cf11eac479aa445ec8c1d6e33b2 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:19.237737 12397 raft_consensus.cc:2804] T 5de5f0581e3b41dea729363d6ca563bb P 186b12638a87410487467df721a3cbda [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:04:19.256703 12313 raft_consensus.cc:2804] T 1e4f2241a94a41f38e481137f868b83d P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:04:19.256963 12397 raft_consensus.cc:697] T 5de5f0581e3b41dea729363d6ca563bb P 186b12638a87410487467df721a3cbda [term 5 LEADER]: Becoming Leader. State: Replica: 186b12638a87410487467df721a3cbda, State: Running, Role: LEADER
I20260709 13:04:19.257647 12313 raft_consensus.cc:697] T 1e4f2241a94a41f38e481137f868b83d P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 LEADER]: Becoming Leader. State: Replica: 10ca5ee68ad64b3283b9d235352c9d23, State: Running, Role: LEADER
I20260709 13:04:19.258507 12397 consensus_queue.cc:237] T 5de5f0581e3b41dea729363d6ca563bb P 186b12638a87410487467df721a3cbda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:19.259145 12313 consensus_queue.cc:237] T 1e4f2241a94a41f38e481137f868b83d P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:19.205513 12403 leader_election.cc:290] T a977842ba4b744f4bb2febd55d286006 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 4 election: Requested vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:19.262603 12329 consensus_queue.cc:1059] T acc241e7273d49149984968596242712 P 873357742d5843579556ee8166f23f47 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:04:19.264096 12329 raft_consensus.cc:3055] T acc241e7273d49149984968596242712 P 873357742d5843579556ee8166f23f47 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:04:19.264539 12329 raft_consensus.cc:740] T acc241e7273d49149984968596242712 P 873357742d5843579556ee8166f23f47 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 873357742d5843579556ee8166f23f47, State: Running, Role: LEADER
I20260709 13:04:19.158895 11022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a977842ba4b744f4bb2febd55d286006" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
I20260709 13:04:19.267000 11022 raft_consensus.cc:3060] T a977842ba4b744f4bb2febd55d286006 P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:04:19.171595 10875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acc241e7273d49149984968596242712" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:04:19.249070 10920 leader_election.cc:304] T c51782cd9d064ec0949e84e24dd4de67 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23; no voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda
W20260709 13:04:19.271003 10920 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 58842.5us
I20260709 13:04:19.272061 10920 leader_election.cc:304] T c31881534c5e4f0089c82057c1746652 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 10ca5ee68ad64b3283b9d235352c9d23; no voters: 186b12638a87410487467df721a3cbda
I20260709 13:04:19.273005 12240 raft_consensus.cc:3060] T c51782cd9d064ec0949e84e24dd4de67 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:04:19.243983 12053 leader_election.cc:290] T f371d2565dd84667b5613848604949bb P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 5 election: Requested vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:19.275928 11639 raft_consensus.cc:2804] T c31881534c5e4f0089c82057c1746652 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:04:19.276620 11639 raft_consensus.cc:697] T c31881534c5e4f0089c82057c1746652 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 LEADER]: Becoming Leader. State: Replica: 10ca5ee68ad64b3283b9d235352c9d23, State: Running, Role: LEADER
I20260709 13:04:19.277593 11639 consensus_queue.cc:237] T c31881534c5e4f0089c82057c1746652 P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 2.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:19.278751 12049 consensus_queue.cc:1059] T f34f2bc02d85444f9096e0f5b4fe4331 P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:04:19.279129 10871 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b37f8fca8dc42718b2a42f0883a956f" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:19.280409 10877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d7ce47b9a92d43dd88d95bcda5038597" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:04:19.281061 10879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5299b8d044f41b7b68057e63d17b159" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:19.281924 10873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34064dc7bf3747dbb3876dd824ee0688" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:19.282833 10873 raft_consensus.cc:2368] T 34064dc7bf3747dbb3876dd824ee0688 P 186b12638a87410487467df721a3cbda [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 10ca5ee68ad64b3283b9d235352c9d23 for earlier term 6. Current term is 7.
I20260709 13:04:19.282812 12193 leader_election.cc:290] T 401a9cf11eac479aa445ec8c1d6e33b2 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:19.283854 12071 raft_consensus.cc:3055] T f34f2bc02d85444f9096e0f5b4fe4331 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:04:19.284296 12071 raft_consensus.cc:740] T f34f2bc02d85444f9096e0f5b4fe4331 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 10ca5ee68ad64b3283b9d235352c9d23, State: Running, Role: LEADER
I20260709 13:04:19.285354 12385 raft_consensus.cc:3060] T 7709387f4d3d4f9992d97a0fbca39a40 P 0172fed22b6b4890a7fdc953fa128af4 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:04:19.285284 12071 consensus_queue.cc:260] T f34f2bc02d85444f9096e0f5b4fe4331 P 10ca5ee68ad64b3283b9d235352c9d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:19.287369 10875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63d44f6ba6434ab2870fe3190f4d7c2f" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:19.288154 10875 raft_consensus.cc:2468] T 63d44f6ba6434ab2870fe3190f4d7c2f P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 3.
W20260709 13:04:19.289172 10918 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:19.265694 12329 consensus_queue.cc:260] T acc241e7273d49149984968596242712 P 873357742d5843579556ee8166f23f47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:19.290948 12329 raft_consensus.cc:3060] T acc241e7273d49149984968596242712 P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:04:19.295316 10920 leader_election.cc:304] T bb7d67be500947b3aef4c4245e33671c P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23; no voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47
W20260709 13:04:19.298404 10921 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:38967, user_credentials={real_user=slave}} blocked reactor thread for 100961us
I20260709 13:04:19.237615 12417 raft_consensus.cc:2804] T 65bf0a62f4d84ef18dd6b23e082724b8 P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:04:19.180441 11937 consensus_queue.cc:1048] T d371c5e27bd843469d2e2e453734cb4a P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.119s
I20260709 13:04:19.308773 12218 raft_consensus.cc:2749] T bb7d67be500947b3aef4c4245e33671c P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:04:19.334957 10920 leader_election.cc:304] T 8a69f32c53f24a50b7852d143665d7b4 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23; no voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda
I20260709 13:04:19.337615 11027 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "868e92339c2f4508af0afbd07c814d64" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:19.339622 10920 leader_election.cc:304] T bac7e5040f984ba2b7146bd36af7267e P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23; no voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47
I20260709 13:04:19.340325 11974 raft_consensus.cc:2749] T 8a69f32c53f24a50b7852d143665d7b4 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:04:19.338490 11027 raft_consensus.cc:2368] T 868e92339c2f4508af0afbd07c814d64 P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 186b12638a87410487467df721a3cbda for earlier term 3. Current term is 4.
I20260709 13:04:19.305220 12403 consensus_queue.cc:1048] T 5de5f0581e3b41dea729363d6ca563bb P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.029s
I20260709 13:04:19.327721 12417 raft_consensus.cc:697] T 65bf0a62f4d84ef18dd6b23e082724b8 P 873357742d5843579556ee8166f23f47 [term 4 LEADER]: Becoming Leader. State: Replica: 873357742d5843579556ee8166f23f47, State: Running, Role: LEADER
I20260709 13:04:19.362663 10873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63d44f6ba6434ab2870fe3190f4d7c2f" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:04:19.363687 10873 raft_consensus.cc:3060] T 63d44f6ba6434ab2870fe3190f4d7c2f P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Advancing to term 5
I20260709 13:04:19.369168 11633 raft_consensus.cc:493] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 873357742d5843579556ee8166f23f47 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:19.369915 11633 raft_consensus.cc:515] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 873357742d5843579556ee8166f23f47 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:19.371887 10955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75492f86fde046c3900df3eac8292fd4" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:04:19.378312 11028 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555217d45d9b4ee492dbb3ddaf0f7c02" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
I20260709 13:04:19.353960 10875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57b6a2418c1342e7924195e9c1dbd357" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:19.398298 11028 raft_consensus.cc:3060] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:04:19.411732 11097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75492f86fde046c3900df3eac8292fd4" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:04:19.412598 10879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1ec2b50c6ad4f7c8033476208abb26d" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:19.413082 11097 raft_consensus.cc:2393] T 75492f86fde046c3900df3eac8292fd4 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 873357742d5843579556ee8166f23f47 in current term 3: Already voted for candidate 0172fed22b6b4890a7fdc953fa128af4 in this term.
I20260709 13:04:19.373783 11024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a1716c0adf34bee8a4e99341ba5788f" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
W20260709 13:04:19.411407 11066 leader_election.cc:336] T 5d439326f23747eaab3c1e7da2d6b785 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903): Timed out: RequestConsensusVote RPC to 127.0.107.194:44903 timed out after 1.680s (SENT)
I20260709 13:04:19.423422 11066 leader_election.cc:304] T 5d439326f23747eaab3c1e7da2d6b785 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda
I20260709 13:04:19.425879 10879 raft_consensus.cc:2468] T a1ec2b50c6ad4f7c8033476208abb26d P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 3.
I20260709 13:04:19.385757 12286 raft_consensus.cc:2749] T bac7e5040f984ba2b7146bd36af7267e P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:04:19.406800 11025 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63d44f6ba6434ab2870fe3190f4d7c2f" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
W20260709 13:04:19.452299 10921 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:19.438030 10872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "868e92339c2f4508af0afbd07c814d64" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:04:19.488638 11029 raft_consensus.cc:1275] T a1ec2b50c6ad4f7c8033476208abb26d P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Refusing update from remote peer 0172fed22b6b4890a7fdc953fa128af4: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 13:04:19.494009 10951 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f1466124dbd4f6ebb3c4d62ca48bf96" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:19.452823 12220 raft_consensus.cc:2749] T 5d439326f23747eaab3c1e7da2d6b785 P 0172fed22b6b4890a7fdc953fa128af4 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 13:04:19.495959 11025 raft_consensus.cc:2368] T 63d44f6ba6434ab2870fe3190f4d7c2f P 873357742d5843579556ee8166f23f47 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 186b12638a87410487467df721a3cbda for earlier term 4. Current term is 5.
W20260709 13:04:19.513260 11068 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:19.488169 11026 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4cd3d7dfbbf4096908ff11e9dc57141" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:19.514529 12223 raft_consensus.cc:493] T acc241e7273d49149984968596242712 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 873357742d5843579556ee8166f23f47)
W20260709 13:04:19.514757 12412 log.cc:927] Time spent T 8c160ddfe9b34bdaa3835d0915c1e06f P 0172fed22b6b4890a7fdc953fa128af4: Append to log took a long time: real 0.051s	user 0.001s	sys 0.000s
W20260709 13:04:19.510994 10995 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:38967, user_credentials={real_user=slave}} blocked reactor thread for 60835.7us
W20260709 13:04:19.500042 12292 consensus_meta.cc:287] T 732c4adeeff6422497aa7db4a82f535b P 186b12638a87410487467df721a3cbda: Time spent flushing consensus metadata: real 0.692s	user 0.027s	sys 0.005s
W20260709 13:04:19.512496 12413 log.cc:927] Time spent T 87cd8f5097624c8692f1d83544d8881e P 0172fed22b6b4890a7fdc953fa128af4: Append to log took a long time: real 0.051s	user 0.000s	sys 0.000s
I20260709 13:04:19.496181 10871 raft_consensus.cc:1240] T 30f628e3d7914d609e7f1d980a74c654 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Rejecting Update request from peer 10ca5ee68ad64b3283b9d235352c9d23 for earlier term 1. Current term is 2. Ops: []
I20260709 13:04:19.525637 11029 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a1716c0adf34bee8a4e99341ba5788f" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
W20260709 13:04:19.536263 12419 log.cc:927] Time spent T 6a9353d1d230416296627d7cf8b1d425 P 0172fed22b6b4890a7fdc953fa128af4: Append to log took a long time: real 0.121s	user 0.000s	sys 0.001s
W20260709 13:04:19.536669 12430 log.cc:927] Time spent T f34f2bc02d85444f9096e0f5b4fe4331 P 186b12638a87410487467df721a3cbda: Append to log took a long time: real 0.148s	user 0.001s	sys 0.000s
I20260709 13:04:19.536563 12223 raft_consensus.cc:515] T acc241e7273d49149984968596242712 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:19.531020 10875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f371d2565dd84667b5613848604949bb" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:04:19.592495 10875 raft_consensus.cc:3060] T f371d2565dd84667b5613848604949bb P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:04:19.537365 11023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1ec2b50c6ad4f7c8033476208abb26d" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 4 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:19.537576 11026 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acc241e7273d49149984968596242712" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
I20260709 13:04:19.538647 11021 raft_consensus.cc:1275] T 32147c6d2d5a477aa38c597c7f78df5b P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Refusing update from remote peer 10ca5ee68ad64b3283b9d235352c9d23: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20260709 13:04:19.539665 10879 raft_consensus.cc:1240] T 2567c2d84e4648d7ba5dc23a3df32817 P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Rejecting Update request from peer 10ca5ee68ad64b3283b9d235352c9d23 for earlier term 2. Current term is 3. Ops: []
I20260709 13:04:19.540750 11020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7839ca2d2c13431a85f44d7c458c6351" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:19.547664 10872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d270b9e5481245d9b5340a454a7cd10b" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
W20260709 13:04:19.548659 11066 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 167120us
I20260709 13:04:19.555714 10874 raft_consensus.cc:2410] T 732c4adeeff6422497aa7db4a82f535b P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 10ca5ee68ad64b3283b9d235352c9d23 for term 4 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
W20260709 13:04:19.556610 10843 leader_election.cc:336] T a5299b8d044f41b7b68057e63d17b159 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903): Timed out: RequestConsensusVote RPC to 127.0.107.194:44903 timed out after 1.504s (SENT)
W20260709 13:04:19.564872 12432 log.cc:927] Time spent T 525bfdbd6d934afd823f7d3ee3f9d052 P 10ca5ee68ad64b3283b9d235352c9d23: Append to log took a long time: real 0.075s	user 0.000s	sys 0.000s
W20260709 13:04:19.566874 12439 log.cc:927] Time spent T 8c160ddfe9b34bdaa3835d0915c1e06f P 186b12638a87410487467df721a3cbda: Append to log took a long time: real 0.066s	user 0.001s	sys 0.000s
I20260709 13:04:19.568447 10871 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e4f2241a94a41f38e481137f868b83d" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:19.489076 12417 consensus_queue.cc:237] T 65bf0a62f4d84ef18dd6b23e082724b8 P 873357742d5843579556ee8166f23f47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:19.570119 12240 raft_consensus.cc:2749] T c51782cd9d064ec0949e84e24dd4de67 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260709 13:04:19.582091 11067 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:19.582762 12425 log.cc:927] Time spent T 72b111ad7b5f4753ab1d2b5ee1114840 P 10ca5ee68ad64b3283b9d235352c9d23: Append to log took a long time: real 0.240s	user 0.000s	sys 0.002s
W20260709 13:04:19.602864 11066 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:19.599864 10879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34064dc7bf3747dbb3876dd824ee0688" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:04:19.595489 10876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63d44f6ba6434ab2870fe3190f4d7c2f" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:19.624430 10879 raft_consensus.cc:2368] T 34064dc7bf3747dbb3876dd824ee0688 P 186b12638a87410487467df721a3cbda [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 10ca5ee68ad64b3283b9d235352c9d23 for earlier term 6. Current term is 7.
I20260709 13:04:19.624316 11028 raft_consensus.cc:2468] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 873357742d5843579556ee8166f23f47 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 5.
I20260709 13:04:19.626538 10955 raft_consensus.cc:2468] T 75492f86fde046c3900df3eac8292fd4 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 3.
I20260709 13:04:19.603615 10843 leader_election.cc:304] T a5299b8d044f41b7b68057e63d17b159 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 186b12638a87410487467df721a3cbda; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47
I20260709 13:04:19.623231 10995 leader_election.cc:304] T 7709387f4d3d4f9992d97a0fbca39a40 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 873357742d5843579556ee8166f23f47; no voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda
W20260709 13:04:19.629491 10992 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 170600us
I20260709 13:04:19.631181 11103 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acc241e7273d49149984968596242712" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:19.632093 11103 raft_consensus.cc:2393] T acc241e7273d49149984968596242712 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 186b12638a87410487467df721a3cbda in current term 2: Already voted for candidate 0172fed22b6b4890a7fdc953fa128af4 in this term.
I20260709 13:04:19.596001 11023 raft_consensus.cc:2410] T a1ec2b50c6ad4f7c8033476208abb26d P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 186b12638a87410487467df721a3cbda for term 4 because replica has last-logged OpId of term: 2 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260709 13:04:19.637501 10879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f9df2bc281f4866b6da5da94a0defc7" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
W20260709 13:04:19.605603 12429 log.cc:927] Time spent T 0e5d327630e5438b878035d28efe6c0c P 186b12638a87410487467df721a3cbda: Append to log took a long time: real 0.165s	user 0.000s	sys 0.001s
I20260709 13:04:19.634210 10995 leader_election.cc:304] T 34064dc7bf3747dbb3876dd824ee0688 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47; no voters: 10ca5ee68ad64b3283b9d235352c9d23
I20260709 13:04:19.642345 10880 raft_consensus.cc:1240] T 5de5f0581e3b41dea729363d6ca563bb P 186b12638a87410487467df721a3cbda [term 5 LEADER]: Rejecting Update request from peer 10ca5ee68ad64b3283b9d235352c9d23 for earlier term 4. Current term is 5. Ops: []
I20260709 13:04:19.594266 12223 leader_election.cc:290] T acc241e7273d49149984968596242712 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:19.607422 12071 raft_consensus.cc:3060] T f34f2bc02d85444f9096e0f5b4fe4331 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:04:19.601522 10872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87cd8f5097624c8692f1d83544d8881e" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:19.651005 10879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "732c4adeeff6422497aa7db4a82f535b" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:04:19.597816 11025 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "401a9cf11eac479aa445ec8c1d6e33b2" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:19.609790 11100 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f371d2565dd84667b5613848604949bb" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
W20260709 13:04:19.652288 11066 leader_election.cc:336] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903): Timed out: RequestConsensusVote RPC to 127.0.107.194:44903 timed out after 1.836s (SENT)
I20260709 13:04:19.606328 10871 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b67eff0634e540eab91cf8cee795969f" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
W20260709 13:04:19.657205 10843 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 100391us
W20260709 13:04:19.658881 10843 leader_election.cc:336] T 87cd8f5097624c8692f1d83544d8881e P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903): Timed out: RequestConsensusVote RPC to 127.0.107.194:44903 timed out after 1.548s (SENT)
I20260709 13:04:19.640676 12202 raft_consensus.cc:2749] T 7709387f4d3d4f9992d97a0fbca39a40 P 873357742d5843579556ee8166f23f47 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 13:04:19.668381 10876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e4f2241a94a41f38e481137f868b83d" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:04:19.598286 12390 raft_consensus.cc:515] T f371d2565dd84667b5613848604949bb P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:19.668309 10879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11de93cd9b9d457489ff1cb80f976c9a" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
W20260709 13:04:19.671239 10843 leader_election.cc:336] T 30f628e3d7914d609e7f1d980a74c654 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903): Timed out: RequestConsensusVote RPC to 127.0.107.194:44903 timed out after 1.563s (SENT)
W20260709 13:04:19.672420 10843 leader_election.cc:336] T 11de93cd9b9d457489ff1cb80f976c9a P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903): Timed out: RequestConsensusVote RPC to 127.0.107.194:44903 timed out after 1.595s (SENT)
I20260709 13:04:19.672797 10843 leader_election.cc:304] T 11de93cd9b9d457489ff1cb80f976c9a P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 186b12638a87410487467df721a3cbda; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47
I20260709 13:04:19.673794 12397 raft_consensus.cc:2749] T 11de93cd9b9d457489ff1cb80f976c9a P 186b12638a87410487467df721a3cbda [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:04:19.648872 12292 raft_consensus.cc:2749] T a5299b8d044f41b7b68057e63d17b159 P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:04:19.595618 12441 consensus_queue.cc:1048] T 6a9353d1d230416296627d7cf8b1d425 P 873357742d5843579556ee8166f23f47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.068s
I20260709 13:04:19.688392 11024 raft_consensus.cc:2468] T 9a1716c0adf34bee8a4e99341ba5788f P 873357742d5843579556ee8166f23f47 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 5.
I20260709 13:04:19.697293 10876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e89e640169e14e2d98f968f76027daa7" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
W20260709 13:04:19.699621 11066 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 47416.6us
I20260709 13:04:19.667400 11025 raft_consensus.cc:2468] T 401a9cf11eac479aa445ec8c1d6e33b2 P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 3.
I20260709 13:04:19.708856 10879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "836251ebc2b74f5da88b8a8b2338e2a5" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
W20260709 13:04:19.716069 10995 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:38967, user_credentials={real_user=slave}} blocked reactor thread for 81968.1us
I20260709 13:04:19.718080 10995 leader_election.cc:304] T a1ec2b50c6ad4f7c8033476208abb26d P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47; no voters: 0172fed22b6b4890a7fdc953fa128af4
I20260709 13:04:19.714953 10879 raft_consensus.cc:2468] T 836251ebc2b74f5da88b8a8b2338e2a5 P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 3.
I20260709 13:04:19.719897 10995 leader_election.cc:304] T 868e92339c2f4508af0afbd07c814d64 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 873357742d5843579556ee8166f23f47; no voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda
W20260709 13:04:19.719985 10920 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 380520us
I20260709 13:04:19.721058 12418 raft_consensus.cc:2804] T 34064dc7bf3747dbb3876dd824ee0688 P 873357742d5843579556ee8166f23f47 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:04:19.723765 10878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72b111ad7b5f4753ab1d2b5ee1114840" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:19.774564 10878 raft_consensus.cc:2468] T 72b111ad7b5f4753ab1d2b5ee1114840 P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 4.
W20260709 13:04:19.775204 10995 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:38967, user_credentials={real_user=slave}} blocked reactor thread for 55396.8us
I20260709 13:04:19.704723 11100 raft_consensus.cc:3060] T f371d2565dd84667b5613848604949bb P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:04:19.776917 10873 raft_consensus.cc:2468] T 63d44f6ba6434ab2870fe3190f4d7c2f P 186b12638a87410487467df721a3cbda [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 5.
W20260709 13:04:19.777727 10921 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:38967, user_credentials={real_user=slave}} blocked reactor thread for 64951.7us
I20260709 13:04:19.781157 10872 raft_consensus.cc:1240] T 65bf0a62f4d84ef18dd6b23e082724b8 P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Rejecting Update request from peer 10ca5ee68ad64b3283b9d235352c9d23 for earlier term 3. Current term is 4. Ops: []
I20260709 13:04:19.788584 10920 leader_election.cc:304] T e5ba72ccd854430baadbc636dfd17136 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 10ca5ee68ad64b3283b9d235352c9d23; no voters: 873357742d5843579556ee8166f23f47
I20260709 13:04:19.796481 11067 leader_election.cc:304] T b4cd3d7dfbbf4096908ff11e9dc57141 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47
I20260709 13:04:19.798964 12385 consensus_queue.cc:1048] T a1ec2b50c6ad4f7c8033476208abb26d P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.015s
I20260709 13:04:19.811743 12385 consensus_queue.cc:1243] T a1ec2b50c6ad4f7c8033476208abb26d P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Peer 873357742d5843579556ee8166f23f47 log is divergent from this leader: its last log entry 2.1 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 0
W20260709 13:04:19.815258 10844 connection.cc:256] RPC call timeout handler was delayed by -1.2423400849999098s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17210000000000003s for a response
I20260709 13:04:19.801936 10879 raft_consensus.cc:1275] T 63d44f6ba6434ab2870fe3190f4d7c2f P 186b12638a87410487467df721a3cbda [term 5 FOLLOWER]: Refusing update from remote peer 10ca5ee68ad64b3283b9d235352c9d23: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
    @     0x7f426c732a9e kudu::TestWorkload::Setup() at ??:0
I20260709 13:04:19.786705 12329 raft_consensus.cc:2749] T 868e92339c2f4508af0afbd07c814d64 P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 13:04:19.789407 12202 raft_consensus.cc:2804] T a1ec2b50c6ad4f7c8033476208abb26d P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:04:19.810626 11067 leader_election.cc:304] T 401a9cf11eac479aa445ec8c1d6e33b2 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:04:19.792060 12418 raft_consensus.cc:697] T 34064dc7bf3747dbb3876dd824ee0688 P 873357742d5843579556ee8166f23f47 [term 7 LEADER]: Becoming Leader. State: Replica: 873357742d5843579556ee8166f23f47, State: Running, Role: LEADER
    @           0x365578 kudu::master::LeaderRebalancerTest::CreateWorkloadTable() at /apache/dev/git/kudu/src/kudu/master/auto_leader_rebalancer-test.cc:112
W20260709 13:04:19.794612 11022 consensus_meta.cc:287] T a977842ba4b744f4bb2febd55d286006 P 873357742d5843579556ee8166f23f47: Time spent flushing consensus metadata: real 0.524s	user 0.019s	sys 0.000s
I20260709 13:04:19.814355 10871 raft_consensus.cc:1275] T 5d439326f23747eaab3c1e7da2d6b785 P 186b12638a87410487467df721a3cbda [term 5 FOLLOWER]: Refusing update from remote peer 10ca5ee68ad64b3283b9d235352c9d23: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
W20260709 13:04:19.827666 10843 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 153119us
I20260709 13:04:19.806663 12403 raft_consensus.cc:2804] T e5ba72ccd854430baadbc636dfd17136 P 10ca5ee68ad64b3283b9d235352c9d23 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:04:19.834197 11069 leader_election.cc:304] T 836251ebc2b74f5da88b8a8b2338e2a5 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda; no voters: 
I20260709 13:04:19.831212 10921 leader_election.cc:304] T 57b6a2418c1342e7924195e9c1dbd357 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23; no voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47
I20260709 13:04:19.809870 12193 raft_consensus.cc:2749] T b4cd3d7dfbbf4096908ff11e9dc57141 P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
    @           0x35a423 kudu::master::LeaderRebalancerTest_RestartTserver_Test::TestBody() at /apache/dev/git/kudu/src/kudu/master/auto_leader_rebalancer-test.cc:412
I20260709 13:04:19.835040 10844 leader_election.cc:304] T 6a9353d1d230416296627d7cf8b1d425 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 186b12638a87410487467df721a3cbda; no voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47
I20260709 13:04:19.825501 12202 raft_consensus.cc:493] T a1ec2b50c6ad4f7c8033476208abb26d P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:19.844834 10921 leader_election.cc:400] T 34064dc7bf3747dbb3876dd824ee0688 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 6 election: Vote denied by peer 186b12638a87410487467df721a3cbda (127.0.107.193:38967) with higher term. Message: Invalid argument: T 34064dc7bf3747dbb3876dd824ee0688 P 186b12638a87410487467df721a3cbda [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 10ca5ee68ad64b3283b9d235352c9d23 for earlier term 6. Current term is 7.
I20260709 13:04:19.833374 12220 consensus_queue.cc:1048] T a1ec2b50c6ad4f7c8033476208abb26d P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.013s
I20260709 13:04:19.845714 12071 consensus_queue.cc:1059] T 30f628e3d7914d609e7f1d980a74c654 P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.005s
I20260709 13:04:19.855516 12071 raft_consensus.cc:3055] T 30f628e3d7914d609e7f1d980a74c654 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:04:19.855567 11069 leader_election.cc:304] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4; no voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47
I20260709 13:04:19.856105 12071 raft_consensus.cc:740] T 30f628e3d7914d609e7f1d980a74c654 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 10ca5ee68ad64b3283b9d235352c9d23, State: Running, Role: LEADER
I20260709 13:04:19.856168 12418 consensus_queue.cc:237] T 34064dc7bf3747dbb3876dd824ee0688 P 873357742d5843579556ee8166f23f47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:19.834589 12403 raft_consensus.cc:493] T e5ba72ccd854430baadbc636dfd17136 P 10ca5ee68ad64b3283b9d235352c9d23 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:19.858668 12403 raft_consensus.cc:3060] T e5ba72ccd854430baadbc636dfd17136 P 10ca5ee68ad64b3283b9d235352c9d23 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:04:19.858965 12220 raft_consensus.cc:2749] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 0172fed22b6b4890a7fdc953fa128af4 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260709 13:04:19.859359 10843 leader_election.cc:336] T 4c2a35bb99374e93b573cf870d04faaa P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903): Timed out: RequestConsensusVote RPC to 127.0.107.194:44903 timed out after 1.800s (SENT)
I20260709 13:04:19.859885 10843 leader_election.cc:304] T 4c2a35bb99374e93b573cf870d04faaa P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 186b12638a87410487467df721a3cbda; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47
I20260709 13:04:19.860421 12387 raft_consensus.cc:2804] T 401a9cf11eac479aa445ec8c1d6e33b2 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:04:19.860597 12250 raft_consensus.cc:2749] T 6a9353d1d230416296627d7cf8b1d425 P 186b12638a87410487467df721a3cbda [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:04:19.840153 12202 raft_consensus.cc:3060] T a1ec2b50c6ad4f7c8033476208abb26d P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:04:19.843879 12385 raft_consensus.cc:2804] T 836251ebc2b74f5da88b8a8b2338e2a5 P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:04:19.853626 12240 consensus_queue.cc:1059] T 2567c2d84e4648d7ba5dc23a3df32817 P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:04:19.842391 12448 raft_consensus.cc:2749] T 57b6a2418c1342e7924195e9c1dbd357 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:04:19.850518 12193 consensus_queue.cc:1048] T d371c5e27bd843469d2e2e453734cb4a P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:04:19.873485 11025 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e5d327630e5438b878035d28efe6c0c" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:19.857175 12071 consensus_queue.cc:260] T 30f628e3d7914d609e7f1d980a74c654 P 10ca5ee68ad64b3283b9d235352c9d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:19.830994 11022 raft_consensus.cc:2468] T a977842ba4b744f4bb2febd55d286006 P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 4.
I20260709 13:04:19.874853 11027 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34064dc7bf3747dbb3876dd824ee0688" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
I20260709 13:04:19.876284 12071 raft_consensus.cc:3060] T 30f628e3d7914d609e7f1d980a74c654 P 10ca5ee68ad64b3283b9d235352c9d23 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:04:19.860623 11069 leader_election.cc:304] T 9a1716c0adf34bee8a4e99341ba5788f P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4; no voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47
I20260709 13:04:19.866266 12385 raft_consensus.cc:493] T 836251ebc2b74f5da88b8a8b2338e2a5 P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:19.872817 12240 raft_consensus.cc:3055] T 2567c2d84e4648d7ba5dc23a3df32817 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:04:19.887290 11029 raft_consensus.cc:1240] T 6a9353d1d230416296627d7cf8b1d425 P 873357742d5843579556ee8166f23f47 [term 5 LEADER]: Rejecting Update request from peer 0172fed22b6b4890a7fdc953fa128af4 for earlier term 4. Current term is 5. Ops: []
W20260709 13:04:19.888657 10920 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 167924us
I20260709 13:04:19.893815 10920 leader_election.cc:304] T e5a5f5a608504650b48fd27f163faed6 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 10ca5ee68ad64b3283b9d235352c9d23; no voters: 186b12638a87410487467df721a3cbda
W20260709 13:04:19.891350 11069 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:19.899667 11028 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2567c2d84e4648d7ba5dc23a3df32817" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
I20260709 13:04:19.903481 11097 raft_consensus.cc:1275] T 5de5f0581e3b41dea729363d6ca563bb P 0172fed22b6b4890a7fdc953fa128af4 [term 5 FOLLOWER]: Refusing update from remote peer 186b12638a87410487467df721a3cbda: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260709 13:04:19.909343 11027 raft_consensus.cc:1275] T 8c160ddfe9b34bdaa3835d0915c1e06f P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Refusing update from remote peer 186b12638a87410487467df721a3cbda: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20260709 13:04:19.912612 11066 leader_election.cc:336] T 525bfdbd6d934afd823f7d3ee3f9d052 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903): Timed out: RequestConsensusVote RPC to 127.0.107.194:44903 timed out after 1.999s (SENT)
I20260709 13:04:19.913221 11066 leader_election.cc:304] T 525bfdbd6d934afd823f7d3ee3f9d052 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47
I20260709 13:04:19.878757 12385 raft_consensus.cc:3060] T 836251ebc2b74f5da88b8a8b2338e2a5 P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:04:19.917647 11103 raft_consensus.cc:1275] T b67eff0634e540eab91cf8cee795969f P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Refusing update from remote peer 186b12638a87410487467df721a3cbda: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 13:04:19.888077 11025 raft_consensus.cc:1275] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 873357742d5843579556ee8166f23f47 [term 5 FOLLOWER]: Refusing update from remote peer 186b12638a87410487467df721a3cbda: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260709 13:04:19.880133 12193 raft_consensus.cc:2749] T 9a1716c0adf34bee8a4e99341ba5788f P 0172fed22b6b4890a7fdc953fa128af4 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 13:04:19.920614 10920 leader_election.cc:304] T a154cc8b181f460abf4880064379948b P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23; no voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47
I20260709 13:04:19.886232 11022 raft_consensus.cc:1275] T 401a9cf11eac479aa445ec8c1d6e33b2 P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Refusing update from remote peer 186b12638a87410487467df721a3cbda: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 13:04:19.880293 12240 raft_consensus.cc:740] T 2567c2d84e4648d7ba5dc23a3df32817 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 10ca5ee68ad64b3283b9d235352c9d23, State: Running, Role: LEADER
I20260709 13:04:19.878387 11026 raft_consensus.cc:2468] T acc241e7273d49149984968596242712 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 2.
I20260709 13:04:19.900404 12390 leader_election.cc:290] T f371d2565dd84667b5613848604949bb P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 4 election: Requested vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:19.927471 11639 raft_consensus.cc:2749] T a154cc8b181f460abf4880064379948b P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:04:19.908806 12223 raft_consensus.cc:2749] T 4c2a35bb99374e93b573cf870d04faaa P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260709 13:04:19.924666 10843 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 65370.2us
W20260709 13:04:19.929273 10992 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 292326us
I20260709 13:04:19.927043 11024 raft_consensus.cc:1240] T 1e4f2241a94a41f38e481137f868b83d P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Rejecting Update request from peer 186b12638a87410487467df721a3cbda for earlier term 3. Current term is 4. Ops: []
I20260709 13:04:19.933724 10920 leader_election.cc:304] T 6f46a5102fd64b45bf1c4dff863d2b4d P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23; no voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47
I20260709 13:04:19.937283 12240 consensus_queue.cc:260] T 2567c2d84e4648d7ba5dc23a3df32817 P 10ca5ee68ad64b3283b9d235352c9d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:19.939252 12240 raft_consensus.cc:3060] T 2567c2d84e4648d7ba5dc23a3df32817 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:04:19.922961 12299 consensus_queue.cc:1048] T 5de5f0581e3b41dea729363d6ca563bb P 186b12638a87410487467df721a3cbda [LEADER]: Connected to new peer: Peer: permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.020s
I20260709 13:04:19.942942 11094 raft_consensus.cc:1275] T d270b9e5481245d9b5340a454a7cd10b P 0172fed22b6b4890a7fdc953fa128af4 [term 6 FOLLOWER]: Refusing update from remote peer 186b12638a87410487467df721a3cbda: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 13:04:19.948599 10875 raft_consensus.cc:2468] T f371d2565dd84667b5613848604949bb P 186b12638a87410487467df721a3cbda [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 5.
I20260709 13:04:19.948828 12202 raft_consensus.cc:515] T a1ec2b50c6ad4f7c8033476208abb26d P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:19.923161 12220 raft_consensus.cc:3060] T 525bfdbd6d934afd823f7d3ee3f9d052 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:04:19.923059 12448 raft_consensus.cc:2804] T e5a5f5a608504650b48fd27f163faed6 P 10ca5ee68ad64b3283b9d235352c9d23 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:04:19.954121 12448 raft_consensus.cc:493] T e5a5f5a608504650b48fd27f163faed6 P 10ca5ee68ad64b3283b9d235352c9d23 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:19.928369 11020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acc241e7273d49149984968596242712" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
W20260709 13:04:19.967679 11066 leader_election.cc:336] T 8a69f32c53f24a50b7852d143665d7b4 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903): Timed out: RequestConsensusVote RPC to 127.0.107.194:44903 timed out after 2.059s (SENT)
I20260709 13:04:19.968242 11066 leader_election.cc:304] T 8a69f32c53f24a50b7852d143665d7b4 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda
I20260709 13:04:19.971472 12394 raft_consensus.cc:493] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:19.964373 12448 raft_consensus.cc:3060] T e5a5f5a608504650b48fd27f163faed6 P 10ca5ee68ad64b3283b9d235352c9d23 [term 5 FOLLOWER]: Advancing to term 6
W20260709 13:04:19.977159 10844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:45807, user_credentials={real_user=slave}} blocked reactor thread for 142239us
I20260709 13:04:19.976112 10992 leader_election.cc:304] T 75492f86fde046c3900df3eac8292fd4 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47; no voters: 0172fed22b6b4890a7fdc953fa128af4
I20260709 13:04:19.967048 12049 raft_consensus.cc:2749] T 6f46a5102fd64b45bf1c4dff863d2b4d P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:04:19.979607 11020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f371d2565dd84667b5613848604949bb" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
I20260709 13:04:19.980506 11020 raft_consensus.cc:2368] T f371d2565dd84667b5613848604949bb P 873357742d5843579556ee8166f23f47 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 0172fed22b6b4890a7fdc953fa128af4 for earlier term 4. Current term is 5.
I20260709 13:04:19.975683 12394 raft_consensus.cc:515] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:19.984553 12394 leader_election.cc:290] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:19.989292 11067 leader_election.cc:400] T f371d2565dd84667b5613848604949bb P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 4 election: Vote denied by peer 873357742d5843579556ee8166f23f47 (127.0.107.195:45807) with higher term. Message: Invalid argument: T f371d2565dd84667b5613848604949bb P 873357742d5843579556ee8166f23f47 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 0172fed22b6b4890a7fdc953fa128af4 for earlier term 4. Current term is 5.
I20260709 13:04:19.989694 11067 leader_election.cc:403] T f371d2565dd84667b5613848604949bb P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
W20260709 13:04:19.996363 10920 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 62802.9us
W20260709 13:04:19.996022 10918 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:20.001051 12403 raft_consensus.cc:515] T e5ba72ccd854430baadbc636dfd17136 P 10ca5ee68ad64b3283b9d235352c9d23 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:20.003593 11100 raft_consensus.cc:2468] T f371d2565dd84667b5613848604949bb P 0172fed22b6b4890a7fdc953fa128af4 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 5.
I20260709 13:04:20.005208 10844 leader_election.cc:304] T f371d2565dd84667b5613848604949bb P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 186b12638a87410487467df721a3cbda; no voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47
I20260709 13:04:19.991247 12417 raft_consensus.cc:2804] T 75492f86fde046c3900df3eac8292fd4 P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:04:19.991525 12394 raft_consensus.cc:2749] T 8a69f32c53f24a50b7852d143665d7b4 P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:04:20.006269 12460 raft_consensus.cc:2749] T f371d2565dd84667b5613848604949bb P 186b12638a87410487467df721a3cbda [term 5 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 13:04:20.017064 10995 leader_election.cc:304] T f371d2565dd84667b5613848604949bb P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:04:20.011751 12390 raft_consensus.cc:2749] T f371d2565dd84667b5613848604949bb P 0172fed22b6b4890a7fdc953fa128af4 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer 873357742d5843579556ee8166f23f47 (127.0.107.195:45807) with higher term. Message: Invalid argument: T f371d2565dd84667b5613848604949bb P 873357742d5843579556ee8166f23f47 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 0172fed22b6b4890a7fdc953fa128af4 for earlier term 4. Current term is 5.
I20260709 13:04:20.023423 11020 raft_consensus.cc:1240] T a1ec2b50c6ad4f7c8033476208abb26d P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Rejecting Update request from peer 0172fed22b6b4890a7fdc953fa128af4 for earlier term 3. Current term is 4. Ops: [1.1-3.2]
I20260709 13:04:20.033186 11099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1ec2b50c6ad4f7c8033476208abb26d" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:04:20.027783 12457 raft_consensus.cc:2804] T f371d2565dd84667b5613848604949bb P 873357742d5843579556ee8166f23f47 [term 5 FOLLOWER]: Leader election won for term 5
W20260709 13:04:20.037356 12443 log.cc:927] Time spent T 5de5f0581e3b41dea729363d6ca563bb P 186b12638a87410487467df721a3cbda: Append to log took a long time: real 0.234s	user 0.001s	sys 0.002s
I20260709 13:04:20.037892 10920 leader_election.cc:304] T d7ce47b9a92d43dd88d95bcda5038597 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23; no voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda
I20260709 13:04:20.039418 10920 leader_election.cc:304] T b4cd3d7dfbbf4096908ff11e9dc57141 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23; no voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47
I20260709 13:04:20.026964 12417 raft_consensus.cc:697] T 75492f86fde046c3900df3eac8292fd4 P 873357742d5843579556ee8166f23f47 [term 3 LEADER]: Becoming Leader. State: Replica: 873357742d5843579556ee8166f23f47, State: Running, Role: LEADER
I20260709 13:04:20.041958 12385 raft_consensus.cc:515] T 836251ebc2b74f5da88b8a8b2338e2a5 P 0172fed22b6b4890a7fdc953fa128af4 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:20.050331 12448 raft_consensus.cc:515] T e5a5f5a608504650b48fd27f163faed6 P 10ca5ee68ad64b3283b9d235352c9d23 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:20.060873 12403 leader_election.cc:290] T e5ba72ccd854430baadbc636dfd17136 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 6 election: Requested vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:20.061287 11633 leader_election.cc:290] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:20.061087 12385 leader_election.cc:290] T 836251ebc2b74f5da88b8a8b2338e2a5 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 5 election: Requested vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:20.050853 12460 raft_consensus.cc:493] T 732c4adeeff6422497aa7db4a82f535b P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:20.047020 12313 raft_consensus.cc:2749] T d7ce47b9a92d43dd88d95bcda5038597 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 13:04:20.079346 10919 leader_election.cc:304] T a977842ba4b744f4bb2febd55d286006 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47; no voters: 
W20260709 13:04:20.081219 10992 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 84326.5us
I20260709 13:04:20.055078 12049 raft_consensus.cc:2749] T b4cd3d7dfbbf4096908ff11e9dc57141 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:04:20.083124 11094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f1466124dbd4f6ebb3c4d62ca48bf96" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:20.088042 11094 raft_consensus.cc:2468] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 0172fed22b6b4890a7fdc953fa128af4 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 6.
I20260709 13:04:20.086581 10953 raft_consensus.cc:1240] T e5ba72ccd854430baadbc636dfd17136 P 10ca5ee68ad64b3283b9d235352c9d23 [term 6 FOLLOWER]: Rejecting Update request from peer 873357742d5843579556ee8166f23f47 for earlier term 5. Current term is 6. Ops: []
I20260709 13:04:20.090621 11639 raft_consensus.cc:2804] T a977842ba4b744f4bb2febd55d286006 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:04:20.094290 11639 raft_consensus.cc:697] T a977842ba4b744f4bb2febd55d286006 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 LEADER]: Becoming Leader. State: Replica: 10ca5ee68ad64b3283b9d235352c9d23, State: Running, Role: LEADER
W20260709 13:04:20.094316 10995 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:38967, user_credentials={real_user=slave}} blocked reactor thread for 63503.8us
I20260709 13:04:20.083735 12202 leader_election.cc:290] T a1ec2b50c6ad4f7c8033476208abb26d P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 4 election: Requested vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
W20260709 13:04:20.101871 10920 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 62536.1us
I20260709 13:04:20.081099 12071 consensus_queue.cc:1048] T 32147c6d2d5a477aa38c597c7f78df5b P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.018s
I20260709 13:04:20.109294 11639 consensus_queue.cc:237] T a977842ba4b744f4bb2febd55d286006 P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 2.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:20.124487 11025 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5ba72ccd854430baadbc636dfd17136" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
I20260709 13:04:20.112576 12385 consensus_queue.cc:1059] T a1ec2b50c6ad4f7c8033476208abb26d P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.014s
I20260709 13:04:20.126572 12385 consensus_queue.cc:1243] T a1ec2b50c6ad4f7c8033476208abb26d P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Peer 873357742d5843579556ee8166f23f47 log is divergent from this leader: its last log entry 2.1 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 0
I20260709 13:04:20.127254 10875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1ec2b50c6ad4f7c8033476208abb26d" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:04:20.127910 10875 raft_consensus.cc:3060] T a1ec2b50c6ad4f7c8033476208abb26d P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:04:20.134200 11023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "30f628e3d7914d609e7f1d980a74c654" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
W20260709 13:04:20.134815 11066 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:20.107365 11020 raft_consensus.cc:1275] T 63d44f6ba6434ab2870fe3190f4d7c2f P 873357742d5843579556ee8166f23f47 [term 5 FOLLOWER]: Refusing update from remote peer 10ca5ee68ad64b3283b9d235352c9d23: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260709 13:04:20.063504 12460 raft_consensus.cc:515] T 732c4adeeff6422497aa7db4a82f535b P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:20.152046 10994 leader_election.cc:304] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47; no voters: 10ca5ee68ad64b3283b9d235352c9d23
I20260709 13:04:20.171221 10920 leader_election.cc:304] T a64b7e99d6c54934a7de6f6dffcf5e94 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23; no voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47
I20260709 13:04:20.100456 12448 leader_election.cc:290] T e5a5f5a608504650b48fd27f163faed6 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 6 election: Requested vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
W20260709 13:04:20.178504 10919 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:45807, user_credentials={real_user=slave}} blocked reactor thread for 104581us
I20260709 13:04:20.180542 10844 leader_election.cc:400] T 868e92339c2f4508af0afbd07c814d64 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 3 pre-election: Vote denied by peer 873357742d5843579556ee8166f23f47 (127.0.107.195:45807) with higher term. Message: Invalid argument: T 868e92339c2f4508af0afbd07c814d64 P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 186b12638a87410487467df721a3cbda for earlier term 3. Current term is 4.
I20260709 13:04:20.180972 10844 leader_election.cc:403] T 868e92339c2f4508af0afbd07c814d64 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:04:20.199699 12071 consensus_queue.cc:1048] T 63d44f6ba6434ab2870fe3190f4d7c2f P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:04:20.201332 12460 leader_election.cc:290] T 732c4adeeff6422497aa7db4a82f535b P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:20.192857 12385 raft_consensus.cc:493] T 75492f86fde046c3900df3eac8292fd4 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:20.168690 12417 consensus_queue.cc:237] T 75492f86fde046c3900df3eac8292fd4 P 873357742d5843579556ee8166f23f47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:20.206799 12385 raft_consensus.cc:515] T 75492f86fde046c3900df3eac8292fd4 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:20.209132 10877 raft_consensus.cc:1275] T 65bf0a62f4d84ef18dd6b23e082724b8 P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Refusing update from remote peer 873357742d5843579556ee8166f23f47: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 13:04:20.209290 12385 leader_election.cc:290] T 75492f86fde046c3900df3eac8292fd4 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:20.194120 12240 raft_consensus.cc:2749] T a64b7e99d6c54934a7de6f6dffcf5e94 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:04:20.216360 10875 raft_consensus.cc:2468] T a1ec2b50c6ad4f7c8033476208abb26d P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 4.
W20260709 13:04:20.177382 12451 log.cc:927] Time spent T 0e5d327630e5438b878035d28efe6c0c P 873357742d5843579556ee8166f23f47: Append to log took a long time: real 0.366s	user 0.003s	sys 0.000s
I20260709 13:04:20.238736 11027 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75492f86fde046c3900df3eac8292fd4" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:20.241879 10995 leader_election.cc:304] T a1ec2b50c6ad4f7c8033476208abb26d P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47; no voters: 0172fed22b6b4890a7fdc953fa128af4
I20260709 13:04:20.194761 11633 raft_consensus.cc:2804] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 873357742d5843579556ee8166f23f47 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:04:20.195204 11022 raft_consensus.cc:1275] T 1e4f2241a94a41f38e481137f868b83d P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Refusing update from remote peer 10ca5ee68ad64b3283b9d235352c9d23: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 13:04:20.247735 11633 raft_consensus.cc:493] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 873357742d5843579556ee8166f23f47 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:20.240715 12460 raft_consensus.cc:3060] T 868e92339c2f4508af0afbd07c814d64 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Advancing to term 4
W20260709 13:04:20.255823 10920 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 138911us
I20260709 13:04:20.247831 12467 raft_consensus.cc:2804] T a1ec2b50c6ad4f7c8033476208abb26d P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Leader election won for term 4
W20260709 13:04:20.274093 10844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:45807, user_credentials={real_user=slave}} blocked reactor thread for 104656us
I20260709 13:04:20.275769 10844 leader_election.cc:400] T 63d44f6ba6434ab2870fe3190f4d7c2f P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 4 pre-election: Vote denied by peer 873357742d5843579556ee8166f23f47 (127.0.107.195:45807) with higher term. Message: Invalid argument: T 63d44f6ba6434ab2870fe3190f4d7c2f P 873357742d5843579556ee8166f23f47 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 186b12638a87410487467df721a3cbda for earlier term 4. Current term is 5.
I20260709 13:04:20.276144 10844 leader_election.cc:403] T 63d44f6ba6434ab2870fe3190f4d7c2f P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:04:20.248293 12385 raft_consensus.cc:493] T e5ba72ccd854430baadbc636dfd17136 P 0172fed22b6b4890a7fdc953fa128af4 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:20.257102 11633 raft_consensus.cc:3060] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 873357742d5843579556ee8166f23f47 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:04:20.248999 12426 consensus_queue.cc:1048] T 65bf0a62f4d84ef18dd6b23e082724b8 P 873357742d5843579556ee8166f23f47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.029s
I20260709 13:04:20.249264 12299 consensus_queue.cc:1048] T b67eff0634e540eab91cf8cee795969f P 186b12638a87410487467df721a3cbda [LEADER]: Connected to new peer: Peer: permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.029s
I20260709 13:04:20.258977 12250 consensus_queue.cc:1048] T d270b9e5481245d9b5340a454a7cd10b P 186b12638a87410487467df721a3cbda [LEADER]: Connected to new peer: Peer: permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.027s
I20260709 13:04:20.299649 12387 raft_consensus.cc:493] T 401a9cf11eac479aa445ec8c1d6e33b2 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 186b12638a87410487467df721a3cbda)
I20260709 13:04:20.300278 12387 raft_consensus.cc:3060] T 401a9cf11eac479aa445ec8c1d6e33b2 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:04:20.284730 12385 raft_consensus.cc:515] T e5ba72ccd854430baadbc636dfd17136 P 0172fed22b6b4890a7fdc953fa128af4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:20.305478 12385 leader_election.cc:290] T e5ba72ccd854430baadbc636dfd17136 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:20.312211 12390 raft_consensus.cc:3055] T a1ec2b50c6ad4f7c8033476208abb26d P 0172fed22b6b4890a7fdc953fa128af4 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:04:20.312971 12390 raft_consensus.cc:740] T a1ec2b50c6ad4f7c8033476208abb26d P 0172fed22b6b4890a7fdc953fa128af4 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 0172fed22b6b4890a7fdc953fa128af4, State: Running, Role: LEADER
I20260709 13:04:20.297000 12296 raft_consensus.cc:2749] T 63d44f6ba6434ab2870fe3190f4d7c2f P 186b12638a87410487467df721a3cbda [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer 873357742d5843579556ee8166f23f47 (127.0.107.195:45807) with higher term. Message: Invalid argument: T 63d44f6ba6434ab2870fe3190f4d7c2f P 873357742d5843579556ee8166f23f47 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 186b12638a87410487467df721a3cbda for earlier term 4. Current term is 5.
I20260709 13:04:20.288901 12467 raft_consensus.cc:697] T a1ec2b50c6ad4f7c8033476208abb26d P 873357742d5843579556ee8166f23f47 [term 4 LEADER]: Becoming Leader. State: Replica: 873357742d5843579556ee8166f23f47, State: Running, Role: LEADER
I20260709 13:04:20.317835 12467 consensus_queue.cc:237] T a1ec2b50c6ad4f7c8033476208abb26d P 873357742d5843579556ee8166f23f47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 2.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:20.325826 10951 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5299b8d044f41b7b68057e63d17b159" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
W20260709 13:04:20.328930 10826 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 13:04:20.329341 10953 raft_consensus.cc:1240] T 1e4f2241a94a41f38e481137f868b83d P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 LEADER]: Rejecting Update request from peer 186b12638a87410487467df721a3cbda for earlier term 3. Current term is 4. Ops: []
I20260709 13:04:20.329141 10844 leader_election.cc:304] T a1ec2b50c6ad4f7c8033476208abb26d P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 186b12638a87410487467df721a3cbda; no voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47
W20260709 13:04:20.329843 10845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 77039.3us
I20260709 13:04:20.326306 12220 raft_consensus.cc:2749] T 525bfdbd6d934afd823f7d3ee3f9d052 P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:04:20.329696 12299 raft_consensus.cc:493] T 7709387f4d3d4f9992d97a0fbca39a40 P 186b12638a87410487467df721a3cbda [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 0172fed22b6b4890a7fdc953fa128af4)
I20260709 13:04:20.338675 10953 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b37f8fca8dc42718b2a42f0883a956f" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:20.331861 12299 raft_consensus.cc:515] T 7709387f4d3d4f9992d97a0fbca39a40 P 186b12638a87410487467df721a3cbda [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:20.342744 10844 leader_election.cc:304] T 9a1716c0adf34bee8a4e99341ba5788f P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47; no voters: 0172fed22b6b4890a7fdc953fa128af4
I20260709 13:04:20.343727 12299 leader_election.cc:290] T 7709387f4d3d4f9992d97a0fbca39a40 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:20.344384 10844 leader_election.cc:304] T 0e5d327630e5438b878035d28efe6c0c P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 186b12638a87410487467df721a3cbda; no voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47
I20260709 13:04:20.344606 12299 raft_consensus.cc:2749] T a1ec2b50c6ad4f7c8033476208abb26d P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:04:20.345803 10844 leader_election.cc:304] T 34064dc7bf3747dbb3876dd824ee0688 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 186b12638a87410487467df721a3cbda; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47
I20260709 13:04:20.369746 10952 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f34f2bc02d85444f9096e0f5b4fe4331" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:20.369976 11639 consensus_queue.cc:1048] T 1e4f2241a94a41f38e481137f868b83d P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:04:20.371349 10949 raft_consensus.cc:1275] T c51782cd9d064ec0949e84e24dd4de67 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Refusing update from remote peer 186b12638a87410487467df721a3cbda: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20260709 13:04:20.374549 12456 log.cc:927] Time spent T 401a9cf11eac479aa445ec8c1d6e33b2 P 0172fed22b6b4890a7fdc953fa128af4: Append to log took a long time: real 0.567s	user 0.003s	sys 0.001s
I20260709 13:04:20.376770 12250 raft_consensus.cc:2749] T 0e5d327630e5438b878035d28efe6c0c P 186b12638a87410487467df721a3cbda [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:04:20.377686 10949 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87cd8f5097624c8692f1d83544d8881e" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:04:20.378413 10949 raft_consensus.cc:2376] T 87cd8f5097624c8692f1d83544d8881e P 10ca5ee68ad64b3283b9d235352c9d23 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 186b12638a87410487467df721a3cbda in term 5. Re-sending same reply.
I20260709 13:04:20.378511 11096 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37678 (request call id 204) took 1076 ms. Trace:
I20260709 13:04:20.379921 10949 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7839ca2d2c13431a85f44d7c458c6351" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:20.381470 10949 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b67eff0634e540eab91cf8cee795969f" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:04:20.387588 10949 raft_consensus.cc:3060] T b67eff0634e540eab91cf8cee795969f P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:04:20.349714 10947 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "30f628e3d7914d609e7f1d980a74c654" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
W20260709 13:04:20.390849 12449 log.cc:927] Time spent T 65bf0a62f4d84ef18dd6b23e082724b8 P 873357742d5843579556ee8166f23f47: Append to log took a long time: real 0.590s	user 0.004s	sys 0.000s
I20260709 13:04:20.391160 10947 raft_consensus.cc:2468] T 30f628e3d7914d609e7f1d980a74c654 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 2.
I20260709 13:04:20.392197 11067 leader_election.cc:304] T acc241e7273d49149984968596242712 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47; no voters: 186b12638a87410487467df721a3cbda
W20260709 13:04:20.392800 12440 log.cc:927] Time spent T bb7d67be500947b3aef4c4245e33671c P 873357742d5843579556ee8166f23f47: Append to log took a long time: real 0.723s	user 0.000s	sys 0.004s
I20260709 13:04:20.393822 12474 raft_consensus.cc:2804] T acc241e7273d49149984968596242712 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:04:20.357645 11021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5ba72ccd854430baadbc636dfd17136" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:20.394488 12474 raft_consensus.cc:697] T acc241e7273d49149984968596242712 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 LEADER]: Becoming Leader. State: Replica: 0172fed22b6b4890a7fdc953fa128af4, State: Running, Role: LEADER
I20260709 13:04:20.384636 12479 raft_consensus.cc:2749] T 34064dc7bf3747dbb3876dd824ee0688 P 186b12638a87410487467df721a3cbda [term 7 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 13:04:20.395516 12474 consensus_queue.cc:237] T acc241e7273d49149984968596242712 P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:20.373006 10952 raft_consensus.cc:2468] T f34f2bc02d85444f9096e0f5b4fe4331 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 2.
W20260709 13:04:20.383065 12433 log.cc:927] Time spent T c31881534c5e4f0089c82057c1746652 P 10ca5ee68ad64b3283b9d235352c9d23: Append to log took a long time: real 0.868s	user 0.000s	sys 0.005s
I20260709 13:04:20.333240 12220 raft_consensus.cc:493] T 868e92339c2f4508af0afbd07c814d64 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:20.398047 12220 raft_consensus.cc:515] T 868e92339c2f4508af0afbd07c814d64 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:20.400529 12220 leader_election.cc:290] T 868e92339c2f4508af0afbd07c814d64 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:20.314321 12390 consensus_queue.cc:260] T a1ec2b50c6ad4f7c8033476208abb26d P 0172fed22b6b4890a7fdc953fa128af4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:20.379218 11096 rpcz_store.cc:276] 0709 13:04:19.302926 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:19.306192 (+  3266us) service_pool.cc:224] Handling call
0709 13:04:19.357660 (+ 51468us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:19.414200 (+ 56540us) raft_consensus.cc:1534] Early marking committed up to index 1
0709 13:04:19.650179 (+235979us) log.cc:844] Serialized 14 byte log entry
0709 13:04:19.804570 (+154391us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:19.859893 (+ 55323us) log.cc:844] Serialized 26 byte log entry
0709 13:04:19.860139 (+   246us) raft_consensus.cc:1645] Marking committed up to 1
0709 13:04:19.860182 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:19.860292 (+   110us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:20.373832 (+513540us) spinlock_profiling.cc:243] Waited 23.5 ms on lock 0x7b4400022b4c. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f4262f9bc85 00007f4262f9c6a2 00007f4265d0e1db 00007f4265d09aaf 00007f4265cfd86e 00007f4265cfcbb6 00007f4265cfc7d8 00007f4265d06221 00007f4265d061d9 00007f4265d06169
0709 13:04:20.378147 (+  4315us) raft_consensus.cc:1685] finished
0709 13:04:20.378261 (+   114us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:20.378416 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":137313,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":41,"raft.queue_time_us":19550,"raft.run_cpu_time_us":321,"raft.run_wall_time_us":17709,"spinlock_wait_cycles":54000512,"thread_start_us":278394,"threads_started":2,"wal-append.queue_time_us":128362}
I20260709 13:04:20.371938 12296 raft_consensus.cc:2804] T 9a1716c0adf34bee8a4e99341ba5788f P 186b12638a87410487467df721a3cbda [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:04:20.402141 12296 raft_consensus.cc:697] T 9a1716c0adf34bee8a4e99341ba5788f P 186b12638a87410487467df721a3cbda [term 5 LEADER]: Becoming Leader. State: Replica: 186b12638a87410487467df721a3cbda, State: Running, Role: LEADER
I20260709 13:04:20.403195 12296 consensus_queue.cc:237] T 9a1716c0adf34bee8a4e99341ba5788f P 186b12638a87410487467df721a3cbda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:20.374630 10954 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e89e640169e14e2d98f968f76027daa7" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:20.375695 10955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65bf0a62f4d84ef18dd6b23e082724b8" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:20.375144 10950 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2567c2d84e4648d7ba5dc23a3df32817" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:20.373505 10951 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e96e80ba9ac34d0998395515e5c8c41a" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:20.426781 10950 raft_consensus.cc:2468] T 2567c2d84e4648d7ba5dc23a3df32817 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 3.
I20260709 13:04:20.428023 10955 raft_consensus.cc:2393] T 65bf0a62f4d84ef18dd6b23e082724b8 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 186b12638a87410487467df721a3cbda in current term 4: Already voted for candidate 873357742d5843579556ee8166f23f47 in this term.
I20260709 13:04:20.439646 10953 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c31881534c5e4f0089c82057c1746652" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 4 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:20.389968 11633 raft_consensus.cc:515] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 873357742d5843579556ee8166f23f47 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:20.454082 10953 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11de93cd9b9d457489ff1cb80f976c9a" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
W20260709 13:04:20.456311 10843 leader_election.cc:336] T b67eff0634e540eab91cf8cee795969f P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903): Timed out: RequestConsensusVote RPC to 127.0.107.194:44903 timed out after 2.087s (SENT)
W20260709 13:04:20.433514 10918 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:20.374085 10946 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d156d6ae2ca4fa59a1e387edf0fea39" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 4 candidate_status { last_received { term: 3 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:20.461640 10954 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d156d6ae2ca4fa59a1e387edf0fea39" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 4 candidate_status { last_received { term: 3 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:04:20.461889 10946 raft_consensus.cc:2393] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 186b12638a87410487467df721a3cbda in current term 4: Already voted for candidate 10ca5ee68ad64b3283b9d235352c9d23 in this term.
I20260709 13:04:20.463624 10950 raft_consensus.cc:1275] T 65bf0a62f4d84ef18dd6b23e082724b8 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Refusing update from remote peer 873357742d5843579556ee8166f23f47: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20260709 13:04:20.472152 10844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:45807, user_credentials={real_user=slave}} blocked reactor thread for 126453us
I20260709 13:04:20.437285 12220 raft_consensus.cc:493] T 7709387f4d3d4f9992d97a0fbca39a40 P 0172fed22b6b4890a7fdc953fa128af4 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:20.484658 10953 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c2a35bb99374e93b573cf870d04faaa" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
W20260709 13:04:20.508239 11069 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:20.510063 10843 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 53819.9us
I20260709 13:04:20.516010 10955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "30f628e3d7914d609e7f1d980a74c654" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:20.516799 10955 raft_consensus.cc:2468] T 30f628e3d7914d609e7f1d980a74c654 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 2.
I20260709 13:04:20.488729 11020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "868e92339c2f4508af0afbd07c814d64" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:20.520188 10953 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5299b8d044f41b7b68057e63d17b159" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:20.481516 12250 consensus_queue.cc:1059] T 1e4f2241a94a41f38e481137f868b83d P 186b12638a87410487467df721a3cbda [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 0, Time since last communication: 0.030s
I20260709 13:04:20.481001 12479 consensus_queue.cc:1048] T 8c160ddfe9b34bdaa3835d0915c1e06f P 186b12638a87410487467df721a3cbda [LEADER]: Connected to new peer: Peer: permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.040s
I20260709 13:04:20.493155 12220 raft_consensus.cc:515] T 7709387f4d3d4f9992d97a0fbca39a40 P 0172fed22b6b4890a7fdc953fa128af4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:20.513852 12457 raft_consensus.cc:697] T f371d2565dd84667b5613848604949bb P 873357742d5843579556ee8166f23f47 [term 5 LEADER]: Becoming Leader. State: Replica: 873357742d5843579556ee8166f23f47, State: Running, Role: LEADER
I20260709 13:04:20.551671 10955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d270b9e5481245d9b5340a454a7cd10b" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:20.553767 10955 raft_consensus.cc:2468] T d270b9e5481245d9b5340a454a7cd10b P 10ca5ee68ad64b3283b9d235352c9d23 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 5.
I20260709 13:04:20.555228 10953 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63d44f6ba6434ab2870fe3190f4d7c2f" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:20.506726 12292 consensus_queue.cc:1048] T 401a9cf11eac479aa445ec8c1d6e33b2 P 186b12638a87410487467df721a3cbda [LEADER]: Connected to new peer: Peer: permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.039s
W20260709 13:04:20.556746 10844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:45807, user_credentials={real_user=slave}} blocked reactor thread for 82634.1us
W20260709 13:04:20.560200 10992 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 48451.1us
I20260709 13:04:20.560604 10844 leader_election.cc:304] T acc241e7273d49149984968596242712 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 186b12638a87410487467df721a3cbda; no voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47
I20260709 13:04:20.563856 10879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "401a9cf11eac479aa445ec8c1d6e33b2" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:20.565966 10873 raft_consensus.cc:1240] T a1ec2b50c6ad4f7c8033476208abb26d P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Rejecting Update request from peer 0172fed22b6b4890a7fdc953fa128af4 for earlier term 3. Current term is 4. Ops: [1.1-3.2]
I20260709 13:04:20.567165 10873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f371d2565dd84667b5613848604949bb" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:04:20.567416 10953 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b67eff0634e540eab91cf8cee795969f" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:20.567855 10873 raft_consensus.cc:2368] T f371d2565dd84667b5613848604949bb P 186b12638a87410487467df721a3cbda [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 0172fed22b6b4890a7fdc953fa128af4 for earlier term 4. Current term is 5.
I20260709 13:04:20.569517 10873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "868e92339c2f4508af0afbd07c814d64" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:20.572134 10876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f9eb06ed01f4ce68be3af8a8434665b" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:04:20.576567 10879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "836251ebc2b74f5da88b8a8b2338e2a5" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:04:20.589936 10879 raft_consensus.cc:3060] T 836251ebc2b74f5da88b8a8b2338e2a5 P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Advancing to term 5
I20260709 13:04:20.527663 12387 raft_consensus.cc:515] T 401a9cf11eac479aa445ec8c1d6e33b2 P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:20.569399 11020 raft_consensus.cc:2393] T 868e92339c2f4508af0afbd07c814d64 P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0172fed22b6b4890a7fdc953fa128af4 in current term 4: Already voted for candidate 873357742d5843579556ee8166f23f47 in this term.
I20260709 13:04:20.561447 12460 raft_consensus.cc:2749] T 868e92339c2f4508af0afbd07c814d64 P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer 873357742d5843579556ee8166f23f47 (127.0.107.195:45807) with higher term. Message: Invalid argument: T 868e92339c2f4508af0afbd07c814d64 P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 186b12638a87410487467df721a3cbda for earlier term 3. Current term is 4.
    @     0x7f42632a5e10 testing::internal::HandleExceptionsInMethodIfSupported<>() at ??:0
I20260709 13:04:20.599787 10953 raft_consensus.cc:1240] T e5a5f5a608504650b48fd27f163faed6 P 10ca5ee68ad64b3283b9d235352c9d23 [term 6 FOLLOWER]: Rejecting Update request from peer 186b12638a87410487467df721a3cbda for earlier term 5. Current term is 6. Ops: []
I20260709 13:04:20.599967 11024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "732c4adeeff6422497aa7db4a82f535b" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:20.604117 12424 consensus_queue.cc:1048] T 65bf0a62f4d84ef18dd6b23e082724b8 P 873357742d5843579556ee8166f23f47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:04:20.604396 11633 leader_election.cc:290] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 7 election: Requested vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:20.599999 11025 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7709387f4d3d4f9992d97a0fbca39a40" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:20.606274 11028 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7709387f4d3d4f9992d97a0fbca39a40" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:20.616468 10953 raft_consensus.cc:1275] T 87cd8f5097624c8692f1d83544d8881e P 10ca5ee68ad64b3283b9d235352c9d23 [term 5 FOLLOWER]: Refusing update from remote peer 186b12638a87410487467df721a3cbda: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260709 13:04:20.625903 11028 raft_consensus.cc:2368] T 7709387f4d3d4f9992d97a0fbca39a40 P 873357742d5843579556ee8166f23f47 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 186b12638a87410487467df721a3cbda for earlier term 6. Current term is 7.
I20260709 13:04:20.571606 12397 raft_consensus.cc:3055] T 1e4f2241a94a41f38e481137f868b83d P 186b12638a87410487467df721a3cbda [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:04:20.619359 11103 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f1466124dbd4f6ebb3c4d62ca48bf96" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:04:20.628296 11103 raft_consensus.cc:3060] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 0172fed22b6b4890a7fdc953fa128af4 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:04:20.577085 10871 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a610d1a71f52432e88fe56130f921fa2" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:20.621994 11094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7709387f4d3d4f9992d97a0fbca39a40" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:20.631919 10873 raft_consensus.cc:2468] T 868e92339c2f4508af0afbd07c814d64 P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 4.
I20260709 13:04:20.593319 12250 raft_consensus.cc:3060] T acc241e7273d49149984968596242712 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:04:20.573814 10872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d156d6ae2ca4fa59a1e387edf0fea39" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 5 candidate_status { last_received { term: 3 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:20.638836 11099 raft_consensus.cc:1240] T 401a9cf11eac479aa445ec8c1d6e33b2 P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Rejecting Update request from peer 186b12638a87410487467df721a3cbda for earlier term 3. Current term is 4. Ops: []
I20260709 13:04:20.633199 10953 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "836251ebc2b74f5da88b8a8b2338e2a5" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:20.639734 10953 raft_consensus.cc:2468] T 836251ebc2b74f5da88b8a8b2338e2a5 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 4.
I20260709 13:04:20.630993 11094 raft_consensus.cc:2468] T 7709387f4d3d4f9992d97a0fbca39a40 P 0172fed22b6b4890a7fdc953fa128af4 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 6.
I20260709 13:04:20.578712 12462 consensus_queue.cc:1059] T e5ba72ccd854430baadbc636dfd17136 P 873357742d5843579556ee8166f23f47 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.153s
I20260709 13:04:20.627108 12397 raft_consensus.cc:740] T 1e4f2241a94a41f38e481137f868b83d P 186b12638a87410487467df721a3cbda [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 186b12638a87410487467df721a3cbda, State: Running, Role: LEADER
I20260709 13:04:20.657981 11028 raft_consensus.cc:1275] T a977842ba4b744f4bb2febd55d286006 P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Refusing update from remote peer 10ca5ee68ad64b3283b9d235352c9d23: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20260709 13:04:20.635802 11024 raft_consensus.cc:2468] T 732c4adeeff6422497aa7db4a82f535b P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 4.
I20260709 13:04:20.661185 11025 raft_consensus.cc:2393] T 7709387f4d3d4f9992d97a0fbca39a40 P 873357742d5843579556ee8166f23f47 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0172fed22b6b4890a7fdc953fa128af4 in current term 7: Already voted for candidate 873357742d5843579556ee8166f23f47 in this term.
W20260709 13:04:20.702244 11066 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:20.572240 12457 consensus_queue.cc:237] T f371d2565dd84667b5613848604949bb P 873357742d5843579556ee8166f23f47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:20.711666 11639 consensus_queue.cc:1048] T a977842ba4b744f4bb2febd55d286006 P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:04:20.645138 12476 consensus_queue.cc:1048] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 186b12638a87410487467df721a3cbda [LEADER]: Connected to new peer: Peer: permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.192s
I20260709 13:04:20.717931 10951 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5de5f0581e3b41dea729363d6ca563bb" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:20.720583 10880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34064dc7bf3747dbb3876dd824ee0688" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:20.716413 10876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5299b8d044f41b7b68057e63d17b159" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:20.736783 10880 raft_consensus.cc:2368] T 34064dc7bf3747dbb3876dd824ee0688 P 186b12638a87410487467df721a3cbda [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 10ca5ee68ad64b3283b9d235352c9d23 for earlier term 6. Current term is 7.
I20260709 13:04:20.745177 11069 leader_election.cc:400] T f371d2565dd84667b5613848604949bb P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 4 election: Vote denied by peer 186b12638a87410487467df721a3cbda (127.0.107.193:38967) with higher term. Message: Invalid argument: T f371d2565dd84667b5613848604949bb P 186b12638a87410487467df721a3cbda [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 0172fed22b6b4890a7fdc953fa128af4 for earlier term 4. Current term is 5.
I20260709 13:04:20.752349 10951 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72b111ad7b5f4753ab1d2b5ee1114840" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:20.718936 12474 raft_consensus.cc:493] T a977842ba4b744f4bb2febd55d286006 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:20.752434 11103 raft_consensus.cc:2468] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 0172fed22b6b4890a7fdc953fa128af4 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 7.
I20260709 13:04:20.753410 10951 raft_consensus.cc:2468] T 72b111ad7b5f4753ab1d2b5ee1114840 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 4.
W20260709 13:04:20.765684 10845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 101078us
W20260709 13:04:20.742153 12461 log.cc:927] Time spent T ff13ae9a47c7455f962406c6d131a3ff P 10ca5ee68ad64b3283b9d235352c9d23: Append to log took a long time: real 0.561s	user 0.004s	sys 0.002s
I20260709 13:04:20.740603 12053 raft_consensus.cc:493] T 8c160ddfe9b34bdaa3835d0915c1e06f P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:20.781993 10951 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d270b9e5481245d9b5340a454a7cd10b" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:04:20.783980 10951 raft_consensus.cc:3060] T d270b9e5481245d9b5340a454a7cd10b P 10ca5ee68ad64b3283b9d235352c9d23 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:04:20.771327 12474 raft_consensus.cc:515] T a977842ba4b744f4bb2febd55d286006 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:20.791445 10876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5a5f5a608504650b48fd27f163faed6" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:04:20.753376 10872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b37f8fca8dc42718b2a42f0883a956f" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:20.796241 10951 raft_consensus.cc:2468] T d270b9e5481245d9b5340a454a7cd10b P 10ca5ee68ad64b3283b9d235352c9d23 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 6.
I20260709 13:04:20.824712 10950 raft_consensus.cc:1275] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Refusing update from remote peer 186b12638a87410487467df721a3cbda: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
W20260709 13:04:20.844444 12480 log.cc:927] Time spent T a1ec2b50c6ad4f7c8033476208abb26d P 873357742d5843579556ee8166f23f47: Append to log took a long time: real 0.252s	user 0.001s	sys 0.000s
I20260709 13:04:20.846903 10954 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2567c2d84e4648d7ba5dc23a3df32817" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:04:20.847975 10947 raft_consensus.cc:1275] T d270b9e5481245d9b5340a454a7cd10b P 10ca5ee68ad64b3283b9d235352c9d23 [term 6 FOLLOWER]: Refusing update from remote peer 186b12638a87410487467df721a3cbda: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 13:04:20.848745 10951 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "30f628e3d7914d609e7f1d980a74c654" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:04:20.794701 12053 raft_consensus.cc:515] T 8c160ddfe9b34bdaa3835d0915c1e06f P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:20.783226 10949 raft_consensus.cc:2468] T b67eff0634e540eab91cf8cee795969f P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 3.
I20260709 13:04:20.868253 10946 raft_consensus.cc:1275] T b67eff0634e540eab91cf8cee795969f P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Refusing update from remote peer 186b12638a87410487467df721a3cbda: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 13:04:20.863940 10951 raft_consensus.cc:2468] T 30f628e3d7914d609e7f1d980a74c654 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 2.
I20260709 13:04:20.872299 10876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7709387f4d3d4f9992d97a0fbca39a40" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:20.798305 11098 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37676 (request call id 353) took 1197 ms. Trace:
I20260709 13:04:20.877462 10954 raft_consensus.cc:2468] T 2567c2d84e4648d7ba5dc23a3df32817 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 3.
I20260709 13:04:20.873291 10876 raft_consensus.cc:2468] T 7709387f4d3d4f9992d97a0fbca39a40 P 186b12638a87410487467df721a3cbda [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 5.
I20260709 13:04:20.891976 10950 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5de5f0581e3b41dea729363d6ca563bb" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
W20260709 13:04:20.848017 12486 log.cc:927] Time spent T 401a9cf11eac479aa445ec8c1d6e33b2 P 186b12638a87410487467df721a3cbda: Append to log took a long time: real 0.072s	user 0.000s	sys 0.000s
W20260709 13:04:20.853690 12481 log.cc:927] Time spent T 65bf0a62f4d84ef18dd6b23e082724b8 P 186b12638a87410487467df721a3cbda: Append to log took a long time: real 0.278s	user 0.002s	sys 0.000s
I20260709 13:04:20.839325 12387 leader_election.cc:290] T 401a9cf11eac479aa445ec8c1d6e33b2 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 4 election: Requested vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:20.909785 10950 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7839ca2d2c13431a85f44d7c458c6351" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:20.923511 12474 leader_election.cc:290] T a977842ba4b744f4bb2febd55d286006 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:20.927021 12053 leader_election.cc:290] T 8c160ddfe9b34bdaa3835d0915c1e06f P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:20.793157 12397 consensus_queue.cc:260] T 1e4f2241a94a41f38e481137f868b83d P 186b12638a87410487467df721a3cbda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:20.933554 12397 raft_consensus.cc:3060] T 1e4f2241a94a41f38e481137f868b83d P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:04:20.939292 10948 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49434 (request call id 339) took 1564 ms. Trace:
I20260709 13:04:20.941573 10948 rpcz_store.cc:276] 0709 13:04:19.374273 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:19.379121 (+  4848us) service_pool.cc:224] Handling call
0709 13:04:19.781303 (+402182us) spinlock_profiling.cc:243] Waited 1.32 ms on lock 0x7b580001d198. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f426303db71 00007f426303deee 0000000000375939 00007f426b0b11e0 00007f426b17a0c4 00007f426b17a131 00007f426b13bfc0 00007f426b126929 00007f426b1263d2 00007f426469dbd4
0709 13:04:19.793988 (+ 12685us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:19.831407 (+ 37419us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:19.833356 (+  1949us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:19.854887 (+ 21531us) log.cc:844] Serialized 26 byte log entry
0709 13:04:19.974958 (+120071us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:04:20.058156 (+ 83198us) log.cc:844] Serialized 14 byte log entry
0709 13:04:20.058832 (+   676us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:20.058939 (+   107us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:20.495799 (+436860us) spinlock_profiling.cc:243] Waited 5.56 ms on lock 0x7b4400022b4c. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f4262f9bc85 00007f4262f9c6a2 00007f4265d0e1db 00007f4265d09aaf 00007f4265cfd86e 00007f4265cfcbb6 00007f4265cfc7d8 00007f4265d06221 00007f4265d061d9 00007f4265d06169
0709 13:04:20.938593 (+442794us) raft_consensus.cc:1685] finished
0709 13:04:20.938667 (+    74us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:20.938929 (+   262us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":9000,"prepare.queue_time_us":79955,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":48,"raft.queue_time_us":17907,"raft.run_cpu_time_us":261,"raft.run_wall_time_us":3658,"spinlock_wait_cycles":15891200,"thread_start_us":124811,"threads_started":2,"wal-append.queue_time_us":160519}
I20260709 13:04:20.890031 11098 rpcz_store.cc:276] 0709 13:04:19.598947 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:19.599139 (+   192us) service_pool.cc:224] Handling call
0709 13:04:20.362665 (+763526us) spinlock_profiling.cc:243] Waited 10.7 ms on lock 0x7b5800117698. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f426303db71 00007f426303da22 00007f426bf56c09 00007f426b0b0ee0 00007f426b17a0ab 00007f426b17a131 00007f426b13bfc0 00007f426b126929 00007f426b1263d2 00007f426469dbd4
0709 13:04:20.362992 (+   327us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:20.363279 (+   287us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 13:04:20.363701 (+   422us) log.cc:844] Serialized 14 byte log entry
0709 13:04:20.752206 (+388505us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:20.768885 (+ 16679us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:04:20.771004 (+  2119us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:20.778750 (+  7746us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:20.796470 (+ 17720us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":3090,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":47,"raft.queue_time_us":2417,"raft.run_cpu_time_us":208,"raft.run_wall_time_us":6011,"spinlock_wait_cycles":24499584,"thread_start_us":368089,"threads_started":2,"wal-append.queue_time_us":200752}
I20260709 13:04:20.897578 12193 raft_consensus.cc:493] T b67eff0634e540eab91cf8cee795969f P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 186b12638a87410487467df721a3cbda)
I20260709 13:04:20.953728 12193 raft_consensus.cc:515] T b67eff0634e540eab91cf8cee795969f P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:20.957639 10948 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e96e80ba9ac34d0998395515e5c8c41a" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:20.961351 12329 raft_consensus.cc:3055] T e5ba72ccd854430baadbc636dfd17136 P 873357742d5843579556ee8166f23f47 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:04:20.961941 12329 raft_consensus.cc:740] T e5ba72ccd854430baadbc636dfd17136 P 873357742d5843579556ee8166f23f47 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 873357742d5843579556ee8166f23f47, State: Running, Role: LEADER
I20260709 13:04:20.959045 10950 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63d44f6ba6434ab2870fe3190f4d7c2f" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:20.960103 10954 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2567c2d84e4648d7ba5dc23a3df32817" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:20.963169 12329 consensus_queue.cc:260] T e5ba72ccd854430baadbc636dfd17136 P 873357742d5843579556ee8166f23f47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:20.959103 10951 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d270b9e5481245d9b5340a454a7cd10b" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:20.964632 10951 raft_consensus.cc:2376] T d270b9e5481245d9b5340a454a7cd10b P 10ca5ee68ad64b3283b9d235352c9d23 [term 6 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 186b12638a87410487467df721a3cbda in term 6. Re-sending same reply.
I20260709 13:04:20.963738 10954 raft_consensus.cc:2376] T 2567c2d84e4648d7ba5dc23a3df32817 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 186b12638a87410487467df721a3cbda in term 3. Re-sending same reply.
I20260709 13:04:20.959777 10949 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d156d6ae2ca4fa59a1e387edf0fea39" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 4 candidate_status { last_received { term: 3 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:20.989176 10949 raft_consensus.cc:2393] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 186b12638a87410487467df721a3cbda in current term 4: Already voted for candidate 10ca5ee68ad64b3283b9d235352c9d23 in this term.
I20260709 13:04:21.005205 10947 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65bf0a62f4d84ef18dd6b23e082724b8" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:21.006693 10947 raft_consensus.cc:2393] T 65bf0a62f4d84ef18dd6b23e082724b8 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 186b12638a87410487467df721a3cbda in current term 4: Already voted for candidate 873357742d5843579556ee8166f23f47 in this term.
W20260709 13:04:21.013823 10918 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:21.032621 10875 raft_consensus.cc:1275] T 34064dc7bf3747dbb3876dd824ee0688 P 186b12638a87410487467df721a3cbda [term 7 FOLLOWER]: Refusing update from remote peer 873357742d5843579556ee8166f23f47: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260709 13:04:21.084946 12471 raft_consensus.cc:493] T bb7d67be500947b3aef4c4245e33671c P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:21.085390 10871 raft_consensus.cc:3060] T c31881534c5e4f0089c82057c1746652 P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:04:21.085839 12471 raft_consensus.cc:515] T bb7d67be500947b3aef4c4245e33671c P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:21.087092 12250 raft_consensus.cc:2749] T acc241e7273d49149984968596242712 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:04:21.088495 11021 raft_consensus.cc:1275] T 525bfdbd6d934afd823f7d3ee3f9d052 P 873357742d5843579556ee8166f23f47 [term 5 FOLLOWER]: Refusing update from remote peer 10ca5ee68ad64b3283b9d235352c9d23: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
W20260709 13:04:21.103741 12485 log.cc:927] Time spent T 9a1716c0adf34bee8a4e99341ba5788f P 186b12638a87410487467df721a3cbda: Append to log took a long time: real 0.462s	user 0.006s	sys 0.000s
I20260709 13:04:21.116844 10946 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "732c4adeeff6422497aa7db4a82f535b" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
W20260709 13:04:21.120569 10844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:21.116887 12471 leader_election.cc:290] T bb7d67be500947b3aef4c4245e33671c P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:21.130892 10946 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b37f8fca8dc42718b2a42f0883a956f" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
W20260709 13:04:21.128407 11069 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:38967, user_credentials={real_user=slave}} blocked reactor thread for 362566us
W20260709 13:04:21.132769 11069 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:21.135020 10947 raft_consensus.cc:3055] T 5de5f0581e3b41dea729363d6ca563bb P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:04:21.135447 10947 raft_consensus.cc:740] T 5de5f0581e3b41dea729363d6ca563bb P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 10ca5ee68ad64b3283b9d235352c9d23, State: Running, Role: LEADER
I20260709 13:04:21.140349 11101 raft_consensus.cc:1275] T 75492f86fde046c3900df3eac8292fd4 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Refusing update from remote peer 873357742d5843579556ee8166f23f47: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:04:21.143208 10844 leader_election.cc:400] T 7709387f4d3d4f9992d97a0fbca39a40 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 6 pre-election: Vote denied by peer 873357742d5843579556ee8166f23f47 (127.0.107.195:45807) with higher term. Message: Invalid argument: T 7709387f4d3d4f9992d97a0fbca39a40 P 873357742d5843579556ee8166f23f47 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 186b12638a87410487467df721a3cbda for earlier term 6. Current term is 7.
I20260709 13:04:21.143574 10844 leader_election.cc:403] T 7709387f4d3d4f9992d97a0fbca39a40 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 6 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:04:21.144678 10844 leader_election.cc:304] T 732c4adeeff6422497aa7db4a82f535b P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47; no voters: 
W20260709 13:04:21.154548 10879 consensus_meta.cc:287] T 836251ebc2b74f5da88b8a8b2338e2a5 P 186b12638a87410487467df721a3cbda: Time spent flushing consensus metadata: real 0.560s	user 0.012s	sys 0.003s
I20260709 13:04:21.159092 10879 raft_consensus.cc:2468] T 836251ebc2b74f5da88b8a8b2338e2a5 P 186b12638a87410487467df721a3cbda [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 5.
I20260709 13:04:21.160775 10946 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f34f2bc02d85444f9096e0f5b4fe4331" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:21.161585 10946 raft_consensus.cc:2468] T f34f2bc02d85444f9096e0f5b4fe4331 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 2.
I20260709 13:04:21.163458 11098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c160ddfe9b34bdaa3835d0915c1e06f" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:21.166266 10879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c160ddfe9b34bdaa3835d0915c1e06f" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:21.136358 10947 consensus_queue.cc:260] T 5de5f0581e3b41dea729363d6ca563bb P 10ca5ee68ad64b3283b9d235352c9d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
W20260709 13:04:21.155196 11068 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.501s	user 0.022s	sys 0.060s
I20260709 13:04:21.172114 10947 raft_consensus.cc:3060] T 5de5f0581e3b41dea729363d6ca563bb P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Advancing to term 5
W20260709 13:04:21.198076 10921 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:38967, user_credentials={real_user=slave}} blocked reactor thread for 229500us
I20260709 13:04:21.166339 12424 raft_consensus.cc:493] T 525bfdbd6d934afd823f7d3ee3f9d052 P 873357742d5843579556ee8166f23f47 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:21.155862 12299 raft_consensus.cc:2804] T 732c4adeeff6422497aa7db4a82f535b P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260709 13:04:21.175387 11068 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.529s	user 0.022s	sys 0.061s
W20260709 13:04:21.167074 11023 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.584s	user 0.039s	sys 0.054s
I20260709 13:04:21.154273 12500 raft_consensus.cc:3060] T 7709387f4d3d4f9992d97a0fbca39a40 P 186b12638a87410487467df721a3cbda [term 5 FOLLOWER]: Advancing to term 7
I20260709 13:04:21.206684 10946 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5de5f0581e3b41dea729363d6ca563bb" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:21.176098 10949 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d156d6ae2ca4fa59a1e387edf0fea39" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 4 candidate_status { last_received { term: 3 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:04:21.214738 10954 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f1466124dbd4f6ebb3c4d62ca48bf96" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:04:21.221207 10879 raft_consensus.cc:1275] T a1ec2b50c6ad4f7c8033476208abb26d P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Refusing update from remote peer 873357742d5843579556ee8166f23f47: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20260709 13:04:21.223173 10953 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72b111ad7b5f4753ab1d2b5ee1114840" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
W20260709 13:04:21.214184 11023 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.631s	user 0.039s	sys 0.055s
I20260709 13:04:21.223919 10953 raft_consensus.cc:2468] T 72b111ad7b5f4753ab1d2b5ee1114840 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 4.
I20260709 13:04:21.203595 12434 raft_consensus.cc:493] T 0e5d327630e5438b878035d28efe6c0c P 873357742d5843579556ee8166f23f47 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 0172fed22b6b4890a7fdc953fa128af4)
I20260709 13:04:21.207315 12299 raft_consensus.cc:493] T 732c4adeeff6422497aa7db4a82f535b P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:21.228725 10951 raft_consensus.cc:1275] T 34064dc7bf3747dbb3876dd824ee0688 P 10ca5ee68ad64b3283b9d235352c9d23 [term 7 FOLLOWER]: Refusing update from remote peer 873357742d5843579556ee8166f23f47: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260709 13:04:21.232033 10950 raft_consensus.cc:1275] T 75492f86fde046c3900df3eac8292fd4 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Refusing update from remote peer 873357742d5843579556ee8166f23f47: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:04:21.236182 10952 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "836251ebc2b74f5da88b8a8b2338e2a5" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:21.236841 10952 raft_consensus.cc:2468] T 836251ebc2b74f5da88b8a8b2338e2a5 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 4.
I20260709 13:04:21.239269 10949 raft_consensus.cc:2393] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 186b12638a87410487467df721a3cbda in current term 4: Already voted for candidate 10ca5ee68ad64b3283b9d235352c9d23 in this term.
I20260709 13:04:21.240338 12418 raft_consensus.cc:493] T 30f628e3d7914d609e7f1d980a74c654 P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 10ca5ee68ad64b3283b9d235352c9d23)
I20260709 13:04:21.241720 10953 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5de5f0581e3b41dea729363d6ca563bb" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:04:21.246414 10871 raft_consensus.cc:1275] T c31881534c5e4f0089c82057c1746652 P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Refusing update from remote peer 10ca5ee68ad64b3283b9d235352c9d23: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20260709 13:04:21.240940 12418 raft_consensus.cc:515] T 30f628e3d7914d609e7f1d980a74c654 P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:21.249100 11024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a977842ba4b744f4bb2febd55d286006" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
W20260709 13:04:21.251320 10919 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:21.251533 11028 raft_consensus.cc:1275] T acc241e7273d49149984968596242712 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Refusing update from remote peer 0172fed22b6b4890a7fdc953fa128af4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:04:21.253499 11028 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb7d67be500947b3aef4c4245e33671c" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:21.208236 12424 raft_consensus.cc:515] T 525bfdbd6d934afd823f7d3ee3f9d052 P 873357742d5843579556ee8166f23f47 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:21.255808 11021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "401a9cf11eac479aa445ec8c1d6e33b2" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
I20260709 13:04:21.256456 11021 raft_consensus.cc:3060] T 401a9cf11eac479aa445ec8c1d6e33b2 P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:04:21.262063 11025 raft_consensus.cc:1240] T a1ec2b50c6ad4f7c8033476208abb26d P 873357742d5843579556ee8166f23f47 [term 4 LEADER]: Rejecting Update request from peer 0172fed22b6b4890a7fdc953fa128af4 for earlier term 3. Current term is 4. Ops: [1.1-3.2]
I20260709 13:04:21.236709 12299 raft_consensus.cc:3060] T 732c4adeeff6422497aa7db4a82f535b P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:04:21.268251 10951 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c31881534c5e4f0089c82057c1746652" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 4 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:21.269843 10950 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b67eff0634e540eab91cf8cee795969f" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:04:21.269958 10947 raft_consensus.cc:1275] T 5de5f0581e3b41dea729363d6ca563bb P 10ca5ee68ad64b3283b9d235352c9d23 [term 5 FOLLOWER]: Refusing update from remote peer 186b12638a87410487467df721a3cbda: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260709 13:04:21.270607 10950 raft_consensus.cc:2376] T b67eff0634e540eab91cf8cee795969f P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 186b12638a87410487467df721a3cbda in term 3. Re-sending same reply.
I20260709 13:04:21.271219 10953 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2567c2d84e4648d7ba5dc23a3df32817" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:04:21.271706 10953 raft_consensus.cc:2376] T 2567c2d84e4648d7ba5dc23a3df32817 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 186b12638a87410487467df721a3cbda in term 3. Re-sending same reply.
I20260709 13:04:21.284822 11099 raft_consensus.cc:1240] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 0172fed22b6b4890a7fdc953fa128af4 [term 7 FOLLOWER]: Rejecting Update request from peer 10ca5ee68ad64b3283b9d235352c9d23 for earlier term 6. Current term is 7. Ops: []
W20260709 13:04:21.283962 10921 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:38967, user_credentials={real_user=slave}} blocked reactor thread for 46565.5us
I20260709 13:04:21.288303 11099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72b111ad7b5f4753ab1d2b5ee1114840" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:21.292411 11099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "525bfdbd6d934afd823f7d3ee3f9d052" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:04:21.288353 10921 leader_election.cc:304] T 11de93cd9b9d457489ff1cb80f976c9a P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23; no voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47
I20260709 13:04:21.293282 11099 raft_consensus.cc:3060] T 525bfdbd6d934afd823f7d3ee3f9d052 P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:04:21.294639 11639 raft_consensus.cc:2749] T 11de93cd9b9d457489ff1cb80f976c9a P 10ca5ee68ad64b3283b9d235352c9d23 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260709 13:04:21.295740 10995 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:38967, user_credentials={real_user=slave}} blocked reactor thread for 51159.3us
I20260709 13:04:21.295831 10955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "30f628e3d7914d609e7f1d980a74c654" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:04:21.296514 11101 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f9df2bc281f4866b6da5da94a0defc7" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:21.297413 11103 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87cd8f5097624c8692f1d83544d8881e" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:21.296664 10955 raft_consensus.cc:2376] T 30f628e3d7914d609e7f1d980a74c654 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 186b12638a87410487467df721a3cbda in term 2. Re-sending same reply.
I20260709 13:04:21.299503 11102 raft_consensus.cc:1275] T 9a1716c0adf34bee8a4e99341ba5788f P 0172fed22b6b4890a7fdc953fa128af4 [term 5 FOLLOWER]: Refusing update from remote peer 186b12638a87410487467df721a3cbda: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260709 13:04:21.241497 12434 raft_consensus.cc:515] T 0e5d327630e5438b878035d28efe6c0c P 873357742d5843579556ee8166f23f47 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:21.303752 10921 leader_election.cc:304] T e89e640169e14e2d98f968f76027daa7 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23; no voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47
I20260709 13:04:21.305325 11101 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d270b9e5481245d9b5340a454a7cd10b" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:21.283883 10948 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d270b9e5481245d9b5340a454a7cd10b" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:04:21.288079 12518 raft_consensus.cc:493] T 34064dc7bf3747dbb3876dd824ee0688 P 10ca5ee68ad64b3283b9d235352c9d23 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:21.308086 11101 raft_consensus.cc:1217] T 5de5f0581e3b41dea729363d6ca563bb P 0172fed22b6b4890a7fdc953fa128af4 [term 5 FOLLOWER]: Deduplicated request from leader. Original: 3.2->[4.3-4.3]   Dedup: 4.3->[]
I20260709 13:04:21.311085 11101 raft_consensus.cc:1240] T 5de5f0581e3b41dea729363d6ca563bb P 0172fed22b6b4890a7fdc953fa128af4 [term 5 FOLLOWER]: Rejecting Update request from peer 10ca5ee68ad64b3283b9d235352c9d23 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260709 13:04:21.308225 11100 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a977842ba4b744f4bb2febd55d286006" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:21.310124 11102 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a977842ba4b744f4bb2febd55d286006" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:04:21.292044 12471 consensus_queue.cc:1048] T 525bfdbd6d934afd823f7d3ee3f9d052 P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.021s
I20260709 13:04:21.311870 11100 raft_consensus.cc:2468] T a977842ba4b744f4bb2febd55d286006 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 3.
I20260709 13:04:21.312562 10843 leader_election.cc:304] T e89e640169e14e2d98f968f76027daa7 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 186b12638a87410487467df721a3cbda; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47
I20260709 13:04:21.315781 11097 raft_consensus.cc:1275] T c31881534c5e4f0089c82057c1746652 P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Refusing update from remote peer 10ca5ee68ad64b3283b9d235352c9d23: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20260709 13:04:21.317093 11103 raft_consensus.cc:1275] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Refusing update from remote peer 186b12638a87410487467df721a3cbda: Log matching property violated. Preceding OpId in replica: term: 3 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20260709 13:04:21.334515 11023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46628 (request call id 274) took 1094 ms. Trace:
I20260709 13:04:21.336361 12487 raft_consensus.cc:2749] T e89e640169e14e2d98f968f76027daa7 P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:04:21.334379 11095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "836251ebc2b74f5da88b8a8b2338e2a5" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:21.337348 11103 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5ba72ccd854430baadbc636dfd17136" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:04:21.338119 11097 raft_consensus.cc:1275] T 4f9eb06ed01f4ce68be3af8a8434665b P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Refusing update from remote peer 186b12638a87410487467df721a3cbda: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 13:04:21.338716 11100 raft_consensus.cc:1275] T 32147c6d2d5a477aa38c597c7f78df5b P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Refusing update from remote peer 10ca5ee68ad64b3283b9d235352c9d23: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20260709 13:04:21.337577 11095 raft_consensus.cc:2393] T 836251ebc2b74f5da88b8a8b2338e2a5 P 0172fed22b6b4890a7fdc953fa128af4 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 10ca5ee68ad64b3283b9d235352c9d23 in current term 5: Already voted for candidate 0172fed22b6b4890a7fdc953fa128af4 in this term.
I20260709 13:04:21.348692 10880 raft_consensus.cc:1275] T 1e4f2241a94a41f38e481137f868b83d P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Refusing update from remote peer 10ca5ee68ad64b3283b9d235352c9d23: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 13:04:21.343546 11069 leader_election.cc:304] T 4f9eb06ed01f4ce68be3af8a8434665b P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4; no voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47
I20260709 13:04:21.318058 10948 raft_consensus.cc:2376] T d270b9e5481245d9b5340a454a7cd10b P 10ca5ee68ad64b3283b9d235352c9d23 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 186b12638a87410487467df721a3cbda in term 6. Re-sending same reply.
I20260709 13:04:21.319979 12479 consensus_queue.cc:1048] T c51782cd9d064ec0949e84e24dd4de67 P 186b12638a87410487467df721a3cbda [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.030s
I20260709 13:04:21.323036 12519 consensus_queue.cc:1048] T 34064dc7bf3747dbb3876dd824ee0688 P 873357742d5843579556ee8166f23f47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.041s
I20260709 13:04:21.350998 12500 raft_consensus.cc:2749] T 7709387f4d3d4f9992d97a0fbca39a40 P 186b12638a87410487467df721a3cbda [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: Vote denied by peer 873357742d5843579556ee8166f23f47 (127.0.107.195:45807) with higher term. Message: Invalid argument: T 7709387f4d3d4f9992d97a0fbca39a40 P 873357742d5843579556ee8166f23f47 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 186b12638a87410487467df721a3cbda for earlier term 6. Current term is 7.
I20260709 13:04:21.373864 11069 leader_election.cc:304] T a610d1a71f52432e88fe56130f921fa2 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4; no voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47
I20260709 13:04:21.370566 11103 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5a5f5a608504650b48fd27f163faed6" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
W20260709 13:04:21.379424 11066 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:21.381227 11066 leader_election.cc:304] T c51782cd9d064ec0949e84e24dd4de67 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 10ca5ee68ad64b3283b9d235352c9d23; no voters: 186b12638a87410487467df721a3cbda
I20260709 13:04:21.387297 11069 leader_election.cc:304] T 7709387f4d3d4f9992d97a0fbca39a40 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda; no voters: 873357742d5843579556ee8166f23f47
I20260709 13:04:21.315121 12518 raft_consensus.cc:515] T 34064dc7bf3747dbb3876dd824ee0688 P 10ca5ee68ad64b3283b9d235352c9d23 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
W20260709 13:04:21.389542 10995 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:38967, user_credentials={real_user=slave}} blocked reactor thread for 74523.6us
I20260709 13:04:21.345091 12487 consensus_queue.cc:1059] T e5a5f5a608504650b48fd27f163faed6 P 186b12638a87410487467df721a3cbda [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.030s
I20260709 13:04:21.392829 11099 raft_consensus.cc:2468] T 525bfdbd6d934afd823f7d3ee3f9d052 P 0172fed22b6b4890a7fdc953fa128af4 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 5.
I20260709 13:04:21.386030 11021 raft_consensus.cc:2468] T 401a9cf11eac479aa445ec8c1d6e33b2 P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 4.
I20260709 13:04:21.352675 12397 consensus_queue.cc:1048] T 87cd8f5097624c8692f1d83544d8881e P 186b12638a87410487467df721a3cbda [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.030s
I20260709 13:04:21.394544 11102 raft_consensus.cc:1275] T 525bfdbd6d934afd823f7d3ee3f9d052 P 0172fed22b6b4890a7fdc953fa128af4 [term 5 FOLLOWER]: Refusing update from remote peer 10ca5ee68ad64b3283b9d235352c9d23: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260709 13:04:21.395380 10874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "401a9cf11eac479aa445ec8c1d6e33b2" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:04:21.393858 10994 leader_election.cc:304] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:04:21.372444 12511 raft_consensus.cc:2749] T 4f9eb06ed01f4ce68be3af8a8434665b P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 13:04:21.403172 11066 leader_election.cc:304] T f34f2bc02d85444f9096e0f5b4fe4331 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda
I20260709 13:04:21.393332 12512 raft_consensus.cc:2749] T a610d1a71f52432e88fe56130f921fa2 P 0172fed22b6b4890a7fdc953fa128af4 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:04:21.412344 12496 raft_consensus.cc:2804] T c51782cd9d064ec0949e84e24dd4de67 P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:04:21.395147 12505 raft_consensus.cc:2804] T 7709387f4d3d4f9992d97a0fbca39a40 P 0172fed22b6b4890a7fdc953fa128af4 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:04:21.414021 12394 raft_consensus.cc:2749] T f34f2bc02d85444f9096e0f5b4fe4331 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 13:04:21.494535 10921 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:38967, user_credentials={real_user=slave}} blocked reactor thread for 197709us
I20260709 13:04:21.459851 11095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb7d67be500947b3aef4c4245e33671c" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:21.413959 12385 consensus_queue.cc:1048] T acc241e7273d49149984968596242712 P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.073s
W20260709 13:04:21.497128 10845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 108551us
I20260709 13:04:21.497933 10879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "30f628e3d7914d609e7f1d980a74c654" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:21.495565 10921 leader_election.cc:304] T 836251ebc2b74f5da88b8a8b2338e2a5 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda; no voters: 
I20260709 13:04:21.498826 10879 raft_consensus.cc:2393] T 30f628e3d7914d609e7f1d980a74c654 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 873357742d5843579556ee8166f23f47 in current term 2: Already voted for candidate 186b12638a87410487467df721a3cbda in this term.
I20260709 13:04:21.464498 12496 raft_consensus.cc:493] T c51782cd9d064ec0949e84e24dd4de67 P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 186b12638a87410487467df721a3cbda)
I20260709 13:04:21.486558 12505 raft_consensus.cc:493] T 7709387f4d3d4f9992d97a0fbca39a40 P 0172fed22b6b4890a7fdc953fa128af4 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:21.467159 12525 raft_consensus.cc:2749] T e89e640169e14e2d98f968f76027daa7 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:04:21.487466 12527 raft_consensus.cc:2804] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 873357742d5843579556ee8166f23f47 [term 7 FOLLOWER]: Leader election won for term 7
W20260709 13:04:21.504971 10994 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 106045us
I20260709 13:04:21.509140 10875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e5d327630e5438b878035d28efe6c0c" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:21.491479 10871 raft_consensus.cc:1275] T acc241e7273d49149984968596242712 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Refusing update from remote peer 0172fed22b6b4890a7fdc953fa128af4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:04:21.426679 12250 consensus_queue.cc:1059] T 401a9cf11eac479aa445ec8c1d6e33b2 P 186b12638a87410487467df721a3cbda [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.072s
I20260709 13:04:21.502349 11067 leader_election.cc:304] T 401a9cf11eac479aa445ec8c1d6e33b2 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:04:21.493518 12298 raft_consensus.cc:493] T 2567c2d84e4648d7ba5dc23a3df32817 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 10ca5ee68ad64b3283b9d235352c9d23)
I20260709 13:04:21.519600 10921 leader_election.cc:304] T b67eff0634e540eab91cf8cee795969f P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23; no voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda
I20260709 13:04:21.504215 12496 raft_consensus.cc:3060] T c51782cd9d064ec0949e84e24dd4de67 P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:04:21.544250 10954 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c31881534c5e4f0089c82057c1746652" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:04:21.504815 12434 leader_election.cc:290] T 0e5d327630e5438b878035d28efe6c0c P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:21.546904 10954 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75492f86fde046c3900df3eac8292fd4" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:21.546787 10947 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5de5f0581e3b41dea729363d6ca563bb" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:21.547871 10954 raft_consensus.cc:2410] T 75492f86fde046c3900df3eac8292fd4 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0172fed22b6b4890a7fdc953fa128af4 for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 13:04:21.547972 10947 raft_consensus.cc:2410] T 5de5f0581e3b41dea729363d6ca563bb P 10ca5ee68ad64b3283b9d235352c9d23 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0172fed22b6b4890a7fdc953fa128af4 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260709 13:04:21.549075 10950 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d156d6ae2ca4fa59a1e387edf0fea39" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 5 candidate_status { last_received { term: 3 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:21.551139 10955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f46a5102fd64b45bf1c4dff863d2b4d" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:21.549851 10950 raft_consensus.cc:2468] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 4.
I20260709 13:04:21.535061 12387 raft_consensus.cc:2804] T 401a9cf11eac479aa445ec8c1d6e33b2 P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:04:21.553123 10946 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f9df2bc281f4866b6da5da94a0defc7" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:21.539043 12487 consensus_queue.cc:1243] T e5a5f5a608504650b48fd27f163faed6 P 186b12638a87410487467df721a3cbda [LEADER]: Peer 10ca5ee68ad64b3283b9d235352c9d23 log is divergent from this leader: its last log entry 4.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260709 13:04:21.512856 12505 raft_consensus.cc:3060] T 7709387f4d3d4f9992d97a0fbca39a40 P 0172fed22b6b4890a7fdc953fa128af4 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:04:21.554600 10994 leader_election.cc:304] T 8c160ddfe9b34bdaa3835d0915c1e06f P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 873357742d5843579556ee8166f23f47; no voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda
W20260709 13:04:21.556385 11066 leader_election.cc:336] T bb7d67be500947b3aef4c4245e33671c P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903): Timed out: RequestConsensusVote RPC to 127.0.107.194:44903 timed out after 1.787s (SENT)
I20260709 13:04:21.559309 10952 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87cd8f5097624c8692f1d83544d8881e" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
W20260709 13:04:21.551062 10918 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:21.562354 10950 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a977842ba4b744f4bb2febd55d286006" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:21.560039 10952 raft_consensus.cc:2468] T 87cd8f5097624c8692f1d83544d8881e P 10ca5ee68ad64b3283b9d235352c9d23 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 5.
I20260709 13:04:21.542640 12527 raft_consensus.cc:697] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 873357742d5843579556ee8166f23f47 [term 7 LEADER]: Becoming Leader. State: Replica: 873357742d5843579556ee8166f23f47, State: Running, Role: LEADER
I20260709 13:04:21.549799 12526 raft_consensus.cc:2804] T 836251ebc2b74f5da88b8a8b2338e2a5 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:04:21.412181 12409 consensus_queue.cc:1048] T a1ec2b50c6ad4f7c8033476208abb26d P 873357742d5843579556ee8166f23f47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.145s
I20260709 13:04:21.553543 12525 raft_consensus.cc:493] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:21.538611 12298 raft_consensus.cc:515] T 2567c2d84e4648d7ba5dc23a3df32817 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:21.499307 12522 consensus_queue.cc:1048] T 75492f86fde046c3900df3eac8292fd4 P 873357742d5843579556ee8166f23f47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 0, Time since last communication: 0.069s
I20260709 13:04:21.575748 11066 leader_election.cc:304] T c31881534c5e4f0089c82057c1746652 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda
I20260709 13:04:21.558938 10948 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb7d67be500947b3aef4c4245e33671c" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:21.562645 12387 raft_consensus.cc:697] T 401a9cf11eac479aa445ec8c1d6e33b2 P 0172fed22b6b4890a7fdc953fa128af4 [term 4 LEADER]: Becoming Leader. State: Replica: 0172fed22b6b4890a7fdc953fa128af4, State: Running, Role: LEADER
I20260709 13:04:21.580956 10920 leader_election.cc:304] T 0f9df2bc281f4866b6da5da94a0defc7 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23; no voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda
W20260709 13:04:21.584141 10843 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 246703us
I20260709 13:04:21.594758 12385 raft_consensus.cc:2749] T c31881534c5e4f0089c82057c1746652 P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 13:04:21.595541 12418 leader_election.cc:290] T 30f628e3d7914d609e7f1d980a74c654 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:21.565948 12526 raft_consensus.cc:493] T 836251ebc2b74f5da88b8a8b2338e2a5 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:21.598817 12526 raft_consensus.cc:3060] T 836251ebc2b74f5da88b8a8b2338e2a5 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:04:21.599680 10953 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32147c6d2d5a477aa38c597c7f78df5b" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:21.600503 11066 leader_election.cc:304] T 5de5f0581e3b41dea729363d6ca563bb P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda
I20260709 13:04:21.601794 10843 leader_election.cc:304] T 7839ca2d2c13431a85f44d7c458c6351 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 186b12638a87410487467df721a3cbda; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47
I20260709 13:04:21.568558 12518 leader_election.cc:290] T 34064dc7bf3747dbb3876dd824ee0688 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:21.335131 11023 rpcz_store.cc:276] 0709 13:04:20.239887 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:20.241234 (+  1347us) service_pool.cc:224] Handling call
0709 13:04:20.253200 (+ 11966us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:04:20.300643 (+ 47443us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:20.300700 (+    57us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:04:20.301609 (+   909us) log.cc:844] Serialized 52 byte log entry
0709 13:04:20.583265 (+281656us) spinlock_profiling.cc:243] Waited 24.4 ms on lock 0x7b180028f540. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f4263057e07 00007f426beb2ab9 00007f426307469b 00007f426307293f 00007f4263074c02 00007f4265cfc20b 00007f4265d01d23 00007f4265d020d1 00007f426611da0f
0709 13:04:21.231685 (+648420us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:04:21.231735 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:21.231835 (+   100us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:21.332935 (+101100us) raft_consensus.cc:1685] finished
0709 13:04:21.334106 (+  1171us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:21.334395 (+   289us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5174,"raft.queue_time_us":12806,"raft.run_cpu_time_us":138,"raft.run_wall_time_us":140,"spinlock_wait_cycles":56147840,"thread_start_us":644552,"threads_started":1,"wal-append.queue_time_us":933476}
I20260709 13:04:21.578343 10949 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5ba72ccd854430baadbc636dfd17136" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:21.607307 11101 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e5d327630e5438b878035d28efe6c0c" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:21.608206 12385 raft_consensus.cc:2749] T 5de5f0581e3b41dea729363d6ca563bb P 0172fed22b6b4890a7fdc953fa128af4 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:04:21.628896 10949 raft_consensus.cc:2393] T e5ba72ccd854430baadbc636dfd17136 P 10ca5ee68ad64b3283b9d235352c9d23 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0172fed22b6b4890a7fdc953fa128af4 in current term 6: Already voted for candidate 10ca5ee68ad64b3283b9d235352c9d23 in this term.
I20260709 13:04:21.602417 12512 raft_consensus.cc:493] T 8c160ddfe9b34bdaa3835d0915c1e06f P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 186b12638a87410487467df721a3cbda)
W20260709 13:04:21.631234 10845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 91416us
I20260709 13:04:21.612844 12471 raft_consensus.cc:2749] T 0f9df2bc281f4866b6da5da94a0defc7 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:04:21.603659 12532 raft_consensus.cc:2749] T b67eff0634e540eab91cf8cee795969f P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:04:21.636245 10843 leader_election.cc:304] T e96e80ba9ac34d0998395515e5c8c41a P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 186b12638a87410487467df721a3cbda; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47
I20260709 13:04:21.615254 12220 leader_election.cc:290] T 7709387f4d3d4f9992d97a0fbca39a40 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
W20260709 13:04:21.646220 10920 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 65364.5us
I20260709 13:04:21.647187 10920 leader_election.cc:304] T d270b9e5481245d9b5340a454a7cd10b P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23; no voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda
I20260709 13:04:21.633188 12536 raft_consensus.cc:2749] T 8c160ddfe9b34bdaa3835d0915c1e06f P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:04:21.571410 12525 raft_consensus.cc:515] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
W20260709 13:04:21.653548 10921 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:38967, user_credentials={real_user=slave}} blocked reactor thread for 134063us
I20260709 13:04:21.670492 10921 leader_election.cc:304] T 72b111ad7b5f4753ab1d2b5ee1114840 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda; no voters: 0172fed22b6b4890a7fdc953fa128af4
I20260709 13:04:21.653986 10920 leader_election.cc:304] T e5ba72ccd854430baadbc636dfd17136 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23; no voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47
I20260709 13:04:21.614902 12529 consensus_queue.cc:1048] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 186b12638a87410487467df721a3cbda [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.033s
W20260709 13:04:21.683701 11066 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 83311.2us
I20260709 13:04:21.685039 11066 leader_election.cc:304] T 75492f86fde046c3900df3eac8292fd4 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47
I20260709 13:04:21.687697 11066 leader_election.cc:304] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 10ca5ee68ad64b3283b9d235352c9d23; no voters: 186b12638a87410487467df721a3cbda
W20260709 13:04:21.690488 10845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 47654.5us
I20260709 13:04:21.690820 11066 leader_election.cc:304] T 6f46a5102fd64b45bf1c4dff863d2b4d P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47
I20260709 13:04:21.696585 11028 raft_consensus.cc:1275] T 732c4adeeff6422497aa7db4a82f535b P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Refusing update from remote peer 10ca5ee68ad64b3283b9d235352c9d23: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 13:04:21.650061 12518 raft_consensus.cc:2749] T d270b9e5481245d9b5340a454a7cd10b P 10ca5ee68ad64b3283b9d235352c9d23 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:04:21.632982 12250 consensus_queue.cc:1048] T d270b9e5481245d9b5340a454a7cd10b P 186b12638a87410487467df721a3cbda [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.049s
W20260709 13:04:21.708528 10843 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 72397.8us
I20260709 13:04:21.694960 12532 raft_consensus.cc:2804] T 72b111ad7b5f4753ab1d2b5ee1114840 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:04:21.716040 11066 leader_election.cc:304] T 0f9df2bc281f4866b6da5da94a0defc7 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda
I20260709 13:04:21.697084 12474 raft_consensus.cc:2804] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260709 13:04:21.705246 12520 log.cc:927] Time spent T bb7d67be500947b3aef4c4245e33671c P 10ca5ee68ad64b3283b9d235352c9d23: Append to log took a long time: real 0.178s	user 0.000s	sys 0.001s
I20260709 13:04:21.704972 12511 raft_consensus.cc:2749] T 6f46a5102fd64b45bf1c4dff863d2b4d P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 13:04:21.753983 10920 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 100095us
I20260709 13:04:21.757311 10920 leader_election.cc:304] T bb7d67be500947b3aef4c4245e33671c P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23; no voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47
I20260709 13:04:21.695403 12471 consensus_queue.cc:1048] T 5d439326f23747eaab3c1e7da2d6b785 P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.028s
W20260709 13:04:21.761595 10921 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:38967, user_credentials={real_user=slave}} blocked reactor thread for 88690.3us
I20260709 13:04:21.645203 12512 raft_consensus.cc:515] T 8c160ddfe9b34bdaa3835d0915c1e06f P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:21.704978 12544 raft_consensus.cc:2749] T e5ba72ccd854430baadbc636dfd17136 P 10ca5ee68ad64b3283b9d235352c9d23 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20260709 13:04:21.763912 11069 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:21.716660 11023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34064dc7bf3747dbb3876dd824ee0688" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:21.724668 12532 raft_consensus.cc:493] T 72b111ad7b5f4753ab1d2b5ee1114840 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 0172fed22b6b4890a7fdc953fa128af4)
I20260709 13:04:21.676663 12545 consensus_queue.cc:1048] T c31881534c5e4f0089c82057c1746652 P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.075s
I20260709 13:04:21.686236 12387 consensus_queue.cc:237] T 401a9cf11eac479aa445ec8c1d6e33b2 P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:21.642473 12527 consensus_queue.cc:237] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 873357742d5843579556ee8166f23f47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:21.738644 12474 raft_consensus.cc:493] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:04:21.774634 11066 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 58677.2us
I20260709 13:04:21.778740 11066 leader_election.cc:304] T 87cd8f5097624c8692f1d83544d8881e P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 10ca5ee68ad64b3283b9d235352c9d23; no voters: 186b12638a87410487467df721a3cbda
I20260709 13:04:21.780584 11066 leader_election.cc:304] T a977842ba4b744f4bb2febd55d286006 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47
I20260709 13:04:21.783468 11101 raft_consensus.cc:1275] T f371d2565dd84667b5613848604949bb P 0172fed22b6b4890a7fdc953fa128af4 [term 5 FOLLOWER]: Refusing update from remote peer 873357742d5843579556ee8166f23f47: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260709 13:04:21.777808 12474 raft_consensus.cc:3060] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:04:21.794670 12513 raft_consensus.cc:493] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 0172fed22b6b4890a7fdc953fa128af4 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:21.774799 12532 raft_consensus.cc:3060] T 72b111ad7b5f4753ab1d2b5ee1114840 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:04:21.763935 12524 consensus_queue.cc:1059] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.172s
I20260709 13:04:21.780160 12394 raft_consensus.cc:2804] T 87cd8f5097624c8692f1d83544d8881e P 0172fed22b6b4890a7fdc953fa128af4 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:04:21.808669 11066 leader_election.cc:304] T e5ba72ccd854430baadbc636dfd17136 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47
I20260709 13:04:21.785336 12511 raft_consensus.cc:2749] T a977842ba4b744f4bb2febd55d286006 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260709 13:04:21.824193 11066 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:21.786218 12298 leader_election.cc:290] T 2567c2d84e4648d7ba5dc23a3df32817 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
W20260709 13:04:21.836607 10843 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 67874.2us
I20260709 13:04:21.837790 10843 leader_election.cc:304] T 65bf0a62f4d84ef18dd6b23e082724b8 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 186b12638a87410487467df721a3cbda; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47
I20260709 13:04:21.841048 10843 leader_election.cc:304] T 4b37f8fca8dc42718b2a42f0883a956f P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 186b12638a87410487467df721a3cbda; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47
W20260709 13:04:21.842540 10921 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:38967, user_credentials={real_user=slave}} blocked reactor thread for 75649.7us
I20260709 13:04:21.798776 12541 raft_consensus.cc:2749] T 7839ca2d2c13431a85f44d7c458c6351 P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:04:21.822585 12394 raft_consensus.cc:493] T 87cd8f5097624c8692f1d83544d8881e P 0172fed22b6b4890a7fdc953fa128af4 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 186b12638a87410487467df721a3cbda)
I20260709 13:04:21.843819 10921 leader_election.cc:304] T a5299b8d044f41b7b68057e63d17b159 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23; no voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47
W20260709 13:04:21.847321 10845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 133039us
I20260709 13:04:21.832376 12387 raft_consensus.cc:3060] T e5ba72ccd854430baadbc636dfd17136 P 0172fed22b6b4890a7fdc953fa128af4 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:04:21.790815 12544 consensus_queue.cc:1048] T 32147c6d2d5a477aa38c597c7f78df5b P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.052s
I20260709 13:04:21.853003 10843 leader_election.cc:304] T f34f2bc02d85444f9096e0f5b4fe4331 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda; no voters: 0172fed22b6b4890a7fdc953fa128af4
I20260709 13:04:21.854076 12529 raft_consensus.cc:2749] T 4b37f8fca8dc42718b2a42f0883a956f P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:04:21.858162 12529 consensus_queue.cc:1048] T 4f9eb06ed01f4ce68be3af8a8434665b P 186b12638a87410487467df721a3cbda [LEADER]: Connected to new peer: Peer: permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:04:21.854261 12546 consensus_queue.cc:1048] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 186b12638a87410487467df721a3cbda [LEADER]: Connected to new peer: Peer: permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:04:21.836026 12298 raft_consensus.cc:493] T 9a1716c0adf34bee8a4e99341ba5788f P 873357742d5843579556ee8166f23f47 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:21.847486 12530 raft_consensus.cc:2749] T e96e80ba9ac34d0998395515e5c8c41a P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:04:21.850093 12394 raft_consensus.cc:3060] T 87cd8f5097624c8692f1d83544d8881e P 0172fed22b6b4890a7fdc953fa128af4 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:04:21.857244 12511 raft_consensus.cc:493] T 5d439326f23747eaab3c1e7da2d6b785 P 0172fed22b6b4890a7fdc953fa128af4 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:21.848042 12250 consensus_queue.cc:1048] T b67eff0634e540eab91cf8cee795969f P 186b12638a87410487467df721a3cbda [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.028s
I20260709 13:04:21.818074 12513 raft_consensus.cc:515] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 0172fed22b6b4890a7fdc953fa128af4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:21.890551 12512 leader_election.cc:290] T 8c160ddfe9b34bdaa3835d0915c1e06f P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:21.891403 12512 raft_consensus.cc:493] T 8a69f32c53f24a50b7852d143665d7b4 P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 186b12638a87410487467df721a3cbda)
I20260709 13:04:21.892241 12513 leader_election.cc:290] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:21.892095 12512 raft_consensus.cc:515] T 8a69f32c53f24a50b7852d143665d7b4 P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:21.878324 12541 raft_consensus.cc:2804] T f34f2bc02d85444f9096e0f5b4fe4331 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:04:21.897807 11094 raft_consensus.cc:1275] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 0172fed22b6b4890a7fdc953fa128af4 [term 5 FOLLOWER]: Refusing update from remote peer 186b12638a87410487467df721a3cbda: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260709 13:04:21.882964 12525 leader_election.cc:290] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:21.910115 12524 raft_consensus.cc:3055] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 10ca5ee68ad64b3283b9d235352c9d23 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:04:21.909986 12550 consensus_queue.cc:1048] T 732c4adeeff6422497aa7db4a82f535b P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.124s
I20260709 13:04:21.910985 12554 raft_consensus.cc:493] T 9a1716c0adf34bee8a4e99341ba5788f P 0172fed22b6b4890a7fdc953fa128af4 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:21.911235 12390 raft_consensus.cc:3060] T a1ec2b50c6ad4f7c8033476208abb26d P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:04:21.911908 12518 raft_consensus.cc:2749] T a5299b8d044f41b7b68057e63d17b159 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260709 13:04:21.931579 10921 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:38967, user_credentials={real_user=slave}} blocked reactor thread for 87919.8us
I20260709 13:04:21.914552 10951 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "836251ebc2b74f5da88b8a8b2338e2a5" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
W20260709 13:04:21.916731 10843 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 63845.3us
I20260709 13:04:21.867285 12471 consensus_queue.cc:1048] T 63d44f6ba6434ab2870fe3190f4d7c2f P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.030s
I20260709 13:04:21.874382 12298 raft_consensus.cc:515] T 9a1716c0adf34bee8a4e99341ba5788f P 873357742d5843579556ee8166f23f47 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:21.912323 12512 leader_election.cc:290] T 8a69f32c53f24a50b7852d143665d7b4 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:21.900091 12530 raft_consensus.cc:493] T 5d439326f23747eaab3c1e7da2d6b785 P 186b12638a87410487467df721a3cbda [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:21.935293 10843 leader_election.cc:304] T 72b111ad7b5f4753ab1d2b5ee1114840 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda; no voters: 0172fed22b6b4890a7fdc953fa128af4
I20260709 13:04:21.936445 10921 leader_election.cc:400] T 34064dc7bf3747dbb3876dd824ee0688 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 6 pre-election: Vote denied by peer 186b12638a87410487467df721a3cbda (127.0.107.193:38967) with higher term. Message: Invalid argument: T 34064dc7bf3747dbb3876dd824ee0688 P 186b12638a87410487467df721a3cbda [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 10ca5ee68ad64b3283b9d235352c9d23 for earlier term 6. Current term is 7.
W20260709 13:04:22.007579 10921 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:38967, user_credentials={real_user=slave}} blocked reactor thread for 73818.8us
I20260709 13:04:22.008673 10921 leader_election.cc:304] T e5a5f5a608504650b48fd27f163faed6 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23; no voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda
W20260709 13:04:21.937098 12561 log.cc:927] Time spent T 401a9cf11eac479aa445ec8c1d6e33b2 P 0172fed22b6b4890a7fdc953fa128af4: Append to log took a long time: real 0.063s	user 0.000s	sys 0.000s
W20260709 13:04:22.016993 10844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:45807, user_credentials={real_user=slave}} blocked reactor thread for 372588us
I20260709 13:04:21.940644 12535 raft_consensus.cc:2749] T 65bf0a62f4d84ef18dd6b23e082724b8 P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:04:21.894196 12511 raft_consensus.cc:515] T 5d439326f23747eaab3c1e7da2d6b785 P 0172fed22b6b4890a7fdc953fa128af4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:21.946465 12555 raft_consensus.cc:2749] T bb7d67be500947b3aef4c4245e33671c P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:04:21.903739 12541 raft_consensus.cc:493] T f34f2bc02d85444f9096e0f5b4fe4331 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Starting leader election (detected failure of leader 10ca5ee68ad64b3283b9d235352c9d23)
I20260709 13:04:21.897258 12553 raft_consensus.cc:2749] T 0f9df2bc281f4866b6da5da94a0defc7 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:04:21.934587 12544 raft_consensus.cc:493] T a154cc8b181f460abf4880064379948b P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:22.002045 12479 raft_consensus.cc:3055] T 401a9cf11eac479aa445ec8c1d6e33b2 P 186b12638a87410487467df721a3cbda [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:04:21.920547 12524 raft_consensus.cc:740] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 10ca5ee68ad64b3283b9d235352c9d23 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 10ca5ee68ad64b3283b9d235352c9d23, State: Running, Role: LEADER
W20260709 13:04:21.942216 12552 log.cc:927] Time spent T b67eff0634e540eab91cf8cee795969f P 186b12638a87410487467df721a3cbda: Append to log took a long time: real 0.114s	user 0.000s	sys 0.001s
W20260709 13:04:22.023160 10843 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 87961.9us
I20260709 13:04:22.043125 10843 leader_election.cc:304] T 836251ebc2b74f5da88b8a8b2338e2a5 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda; no voters: 0172fed22b6b4890a7fdc953fa128af4
W20260709 13:04:22.031409 10992 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:21.882305 12543 consensus_queue.cc:1048] T 9a1716c0adf34bee8a4e99341ba5788f P 186b12638a87410487467df721a3cbda [LEADER]: Connected to new peer: Peer: permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.157s
I20260709 13:04:22.015254 12518 raft_consensus.cc:493] T 6f46a5102fd64b45bf1c4dff863d2b4d P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:22.049613 10877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87cd8f5097624c8692f1d83544d8881e" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:22.036489 12489 raft_consensus.cc:2804] T 72b111ad7b5f4753ab1d2b5ee1114840 P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:04:22.024061 12525 raft_consensus.cc:493] T a64b7e99d6c54934a7de6f6dffcf5e94 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:22.052273 10843 leader_election.cc:304] T c31881534c5e4f0089c82057c1746652 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 186b12638a87410487467df721a3cbda; no voters: 0172fed22b6b4890a7fdc953fa128af4, 10ca5ee68ad64b3283b9d235352c9d23
I20260709 13:04:22.055034 11101 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a1716c0adf34bee8a4e99341ba5788f" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:22.036370 12541 raft_consensus.cc:3060] T f34f2bc02d85444f9096e0f5b4fe4331 P 186b12638a87410487467df721a3cbda [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:04:22.043931 12479 raft_consensus.cc:740] T 401a9cf11eac479aa445ec8c1d6e33b2 P 186b12638a87410487467df721a3cbda [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 186b12638a87410487467df721a3cbda, State: Running, Role: LEADER
I20260709 13:04:22.044262 12250 raft_consensus.cc:493] T 4c2a35bb99374e93b573cf870d04faaa P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:22.048745 12535 raft_consensus.cc:2764] T 836251ebc2b74f5da88b8a8b2338e2a5 P 186b12638a87410487467df721a3cbda [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20260709 13:04:22.058929 10877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d156d6ae2ca4fa59a1e387edf0fea39" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 5 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:22.062000 10955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2567c2d84e4648d7ba5dc23a3df32817" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:22.045337 12298 leader_election.cc:290] T 9a1716c0adf34bee8a4e99341ba5788f P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:22.071130 10921 leader_election.cc:304] T 4b37f8fca8dc42718b2a42f0883a956f P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23; no voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47
I20260709 13:04:21.934526 12554 raft_consensus.cc:515] T 9a1716c0adf34bee8a4e99341ba5788f P 0172fed22b6b4890a7fdc953fa128af4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:22.076120 10950 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "30f628e3d7914d609e7f1d980a74c654" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:22.055327 12558 raft_consensus.cc:2749] T e5a5f5a608504650b48fd27f163faed6 P 10ca5ee68ad64b3283b9d235352c9d23 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 13:04:22.080150 10955 raft_consensus.cc:2393] T 2567c2d84e4648d7ba5dc23a3df32817 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 873357742d5843579556ee8166f23f47 in current term 3: Already voted for candidate 186b12638a87410487467df721a3cbda in this term.
I20260709 13:04:22.057291 12555 raft_consensus.cc:493] T b4cd3d7dfbbf4096908ff11e9dc57141 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:22.082697 12569 consensus_queue.cc:1048] T 34064dc7bf3747dbb3876dd824ee0688 P 873357742d5843579556ee8166f23f47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.014s
I20260709 13:04:22.084069 12524 consensus_queue.cc:260] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 10ca5ee68ad64b3283b9d235352c9d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:22.024304 12530 raft_consensus.cc:515] T 5d439326f23747eaab3c1e7da2d6b785 P 186b12638a87410487467df721a3cbda [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:22.088989 10843 leader_election.cc:304] T 2567c2d84e4648d7ba5dc23a3df32817 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda; no voters: 873357742d5843579556ee8166f23f47
I20260709 13:04:22.090852 12565 consensus_queue.cc:1048] T 5de5f0581e3b41dea729363d6ca563bb P 186b12638a87410487467df721a3cbda [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 13:04:22.091220 10948 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "525bfdbd6d934afd823f7d3ee3f9d052" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:22.053194 12489 raft_consensus.cc:493] T 72b111ad7b5f4753ab1d2b5ee1114840 P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Starting leader election (detected failure of leader 0172fed22b6b4890a7fdc953fa128af4)
W20260709 13:04:22.093722 12299 consensus_meta.cc:287] T 732c4adeeff6422497aa7db4a82f535b P 186b12638a87410487467df721a3cbda: Time spent flushing consensus metadata: real 0.824s	user 0.007s	sys 0.013s
I20260709 13:04:22.063786 10871 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34064dc7bf3747dbb3876dd824ee0688" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:22.100907 10950 raft_consensus.cc:2393] T 30f628e3d7914d609e7f1d980a74c654 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 873357742d5843579556ee8166f23f47 in current term 2: Already voted for candidate 186b12638a87410487467df721a3cbda in this term.
I20260709 13:04:22.113322 10843 leader_election.cc:304] T 30f628e3d7914d609e7f1d980a74c654 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda; no voters: 873357742d5843579556ee8166f23f47
I20260709 13:04:22.119308 11101 raft_consensus.cc:2468] T 9a1716c0adf34bee8a4e99341ba5788f P 0172fed22b6b4890a7fdc953fa128af4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 5.
I20260709 13:04:22.079674 10872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63d44f6ba6434ab2870fe3190f4d7c2f" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:04:22.045022 12544 raft_consensus.cc:515] T a154cc8b181f460abf4880064379948b P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:22.076593 12250 raft_consensus.cc:515] T 4c2a35bb99374e93b573cf870d04faaa P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:22.064675 12518 raft_consensus.cc:515] T 6f46a5102fd64b45bf1c4dff863d2b4d P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:22.134291 11100 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d156d6ae2ca4fa59a1e387edf0fea39" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 5 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:22.134536 12476 raft_consensus.cc:2804] T 30f628e3d7914d609e7f1d980a74c654 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:04:22.063846 12525 raft_consensus.cc:515] T a64b7e99d6c54934a7de6f6dffcf5e94 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:22.137730 10872 raft_consensus.cc:2376] T 63d44f6ba6434ab2870fe3190f4d7c2f P 186b12638a87410487467df721a3cbda [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 5. Re-sending same reply.
I20260709 13:04:22.073040 12528 consensus_queue.cc:1048] T 75492f86fde046c3900df3eac8292fd4 P 873357742d5843579556ee8166f23f47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.055s
I20260709 13:04:22.121558 12489 raft_consensus.cc:3060] T 72b111ad7b5f4753ab1d2b5ee1114840 P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:04:22.118229 12535 raft_consensus.cc:2804] T 2567c2d84e4648d7ba5dc23a3df32817 P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Leader election won for term 3
W20260709 13:04:22.146236 10921 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:38967, user_credentials={real_user=slave}} blocked reactor thread for 75307.3us
I20260709 13:04:22.164189 12525 leader_election.cc:290] T a64b7e99d6c54934a7de6f6dffcf5e94 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:22.146953 12474 raft_consensus.cc:515] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 0172fed22b6b4890a7fdc953fa128af4 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:22.140033 12571 raft_consensus.cc:2749] T c31881534c5e4f0089c82057c1746652 P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:04:22.151343 12476 raft_consensus.cc:697] T 30f628e3d7914d609e7f1d980a74c654 P 186b12638a87410487467df721a3cbda [term 2 LEADER]: Becoming Leader. State: Replica: 186b12638a87410487467df721a3cbda, State: Running, Role: LEADER
I20260709 13:04:22.136942 12299 raft_consensus.cc:515] T 732c4adeeff6422497aa7db4a82f535b P 186b12638a87410487467df721a3cbda [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:22.109964 12555 raft_consensus.cc:515] T b4cd3d7dfbbf4096908ff11e9dc57141 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
W20260709 13:04:22.192665 10992 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 97340.6us
I20260709 13:04:22.172161 12527 raft_consensus.cc:493] T 868e92339c2f4508af0afbd07c814d64 P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:22.174844 12575 raft_consensus.cc:493] T d71ca3a1ba6b48478aaf62741bf3cf6a P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:22.177506 12530 leader_election.cc:290] T 5d439326f23747eaab3c1e7da2d6b785 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:22.268414 11098 raft_consensus.cc:1275] T a1ec2b50c6ad4f7c8033476208abb26d P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Refusing update from remote peer 873357742d5843579556ee8166f23f47: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 2. (term mismatch)
W20260709 13:04:22.253743 12577 log.cc:927] Time spent T d270b9e5481245d9b5340a454a7cd10b P 0172fed22b6b4890a7fdc953fa128af4: Append to log took a long time: real 0.068s	user 0.002s	sys 0.001s
I20260709 13:04:22.164280 12544 leader_election.cc:290] T a154cc8b181f460abf4880064379948b P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
W20260709 13:04:22.189822 12526 consensus_meta.cc:287] T 836251ebc2b74f5da88b8a8b2338e2a5 P 10ca5ee68ad64b3283b9d235352c9d23: Time spent flushing consensus metadata: real 0.590s	user 0.016s	sys 0.000s
I20260709 13:04:22.082736 12479 consensus_queue.cc:260] T 401a9cf11eac479aa445ec8c1d6e33b2 P 186b12638a87410487467df721a3cbda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:22.173326 12559 raft_consensus.cc:493] T 1e4f2241a94a41f38e481137f868b83d P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:22.173571 12566 raft_consensus.cc:493] T 75492f86fde046c3900df3eac8292fd4 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:22.232009 11097 raft_consensus.cc:1240] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 0172fed22b6b4890a7fdc953fa128af4 [term 5 FOLLOWER]: Rejecting Update request from peer 186b12638a87410487467df721a3cbda for earlier term 4. Current term is 5. Ops: [4.2-4.2]
I20260709 13:04:22.173053 12578 raft_consensus.cc:2749] T 4b37f8fca8dc42718b2a42f0883a956f P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:04:22.270332 11098 pending_rounds.cc:85] T a1ec2b50c6ad4f7c8033476208abb26d P 0172fed22b6b4890a7fdc953fa128af4: Aborting all ops after (but not including) 1
I20260709 13:04:22.273099 10875 raft_consensus.cc:1240] T 732c4adeeff6422497aa7db4a82f535b P 186b12638a87410487467df721a3cbda [term 5 FOLLOWER]: Rejecting Update request from peer 10ca5ee68ad64b3283b9d235352c9d23 for earlier term 4. Current term is 5. Ops: []
I20260709 13:04:22.221899 12535 raft_consensus.cc:697] T 2567c2d84e4648d7ba5dc23a3df32817 P 186b12638a87410487467df721a3cbda [term 3 LEADER]: Becoming Leader. State: Replica: 186b12638a87410487467df721a3cbda, State: Running, Role: LEADER
W20260709 13:04:22.351898 10921 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:38967, user_credentials={real_user=slave}} blocked reactor thread for 199439us
I20260709 13:04:22.358053 10921 leader_election.cc:304] T 87cd8f5097624c8692f1d83544d8881e P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23; no voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda
I20260709 13:04:22.347962 11098 pending_rounds.cc:107] T a1ec2b50c6ad4f7c8033476208abb26d P 0172fed22b6b4890a7fdc953fa128af4: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20260709 13:04:22.368736 11098 raft_consensus.cc:2889] T a1ec2b50c6ad4f7c8033476208abb26d P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 13:04:22.327544 10992 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 53269.8us
I20260709 13:04:22.383188 10921 leader_election.cc:304] T 34064dc7bf3747dbb3876dd824ee0688 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23; no voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47
I20260709 13:04:22.264391 12583 consensus_queue.cc:1048] T c31881534c5e4f0089c82057c1746652 P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.094s
W20260709 13:04:22.373562 12387 consensus_meta.cc:287] T e5ba72ccd854430baadbc636dfd17136 P 0172fed22b6b4890a7fdc953fa128af4: Time spent flushing consensus metadata: real 0.510s	user 0.012s	sys 0.009s
W20260709 13:04:22.406211 11066 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:22.418238 12518 leader_election.cc:290] T 6f46a5102fd64b45bf1c4dff863d2b4d P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:22.419540 12555 leader_election.cc:290] T b4cd3d7dfbbf4096908ff11e9dc57141 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:22.420971 12581 raft_consensus.cc:493] T 57b6a2418c1342e7924195e9c1dbd357 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 186b12638a87410487467df721a3cbda)
I20260709 13:04:22.406481 12554 leader_election.cc:290] T 9a1716c0adf34bee8a4e99341ba5788f P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:22.388347 12544 raft_consensus.cc:2749] T 87cd8f5097624c8692f1d83544d8881e P 10ca5ee68ad64b3283b9d235352c9d23 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:04:22.433079 12582 raft_consensus.cc:2749] T 34064dc7bf3747dbb3876dd824ee0688 P 10ca5ee68ad64b3283b9d235352c9d23 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:04:22.388389 12534 raft_consensus.cc:493] T 7709387f4d3d4f9992d97a0fbca39a40 P 873357742d5843579556ee8166f23f47 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:22.434631 12534 raft_consensus.cc:515] T 7709387f4d3d4f9992d97a0fbca39a40 P 873357742d5843579556ee8166f23f47 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:22.436641 12299 leader_election.cc:290] T 732c4adeeff6422497aa7db4a82f535b P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 5 election: Requested vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:22.439044 12535 consensus_queue.cc:237] T 2567c2d84e4648d7ba5dc23a3df32817 P 186b12638a87410487467df721a3cbda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
W20260709 13:04:22.440927 12505 consensus_meta.cc:287] T 7709387f4d3d4f9992d97a0fbca39a40 P 0172fed22b6b4890a7fdc953fa128af4: Time spent flushing consensus metadata: real 0.876s	user 0.012s	sys 0.009s
I20260709 13:04:22.407781 12578 raft_consensus.cc:493] T d7ce47b9a92d43dd88d95bcda5038597 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:22.441721 12505 raft_consensus.cc:515] T 7709387f4d3d4f9992d97a0fbca39a40 P 0172fed22b6b4890a7fdc953fa128af4 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:22.354678 12566 raft_consensus.cc:515] T 75492f86fde046c3900df3eac8292fd4 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:22.443794 12474 leader_election.cc:290] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 5 election: Requested vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:22.447325 12505 leader_election.cc:290] T 7709387f4d3d4f9992d97a0fbca39a40 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 7 election: Requested vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:22.276131 12527 raft_consensus.cc:515] T 868e92339c2f4508af0afbd07c814d64 P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:22.449469 10948 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a69f32c53f24a50b7852d143665d7b4" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:22.449841 12566 leader_election.cc:290] T 75492f86fde046c3900df3eac8292fd4 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:22.450912 10948 raft_consensus.cc:2468] T 8a69f32c53f24a50b7852d143665d7b4 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 4.
I20260709 13:04:22.451797 12527 leader_election.cc:290] T 868e92339c2f4508af0afbd07c814d64 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:22.443217 12578 raft_consensus.cc:515] T d7ce47b9a92d43dd88d95bcda5038597 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:22.397244 12479 raft_consensus.cc:3060] T 401a9cf11eac479aa445ec8c1d6e33b2 P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:04:22.306953 12526 raft_consensus.cc:515] T 836251ebc2b74f5da88b8a8b2338e2a5 P 10ca5ee68ad64b3283b9d235352c9d23 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
W20260709 13:04:22.416764 12394 consensus_meta.cc:287] T 87cd8f5097624c8692f1d83544d8881e P 0172fed22b6b4890a7fdc953fa128af4: Time spent flushing consensus metadata: real 0.528s	user 0.017s	sys 0.004s
I20260709 13:04:22.503908 10992 leader_election.cc:304] T 30f628e3d7914d609e7f1d980a74c654 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 873357742d5843579556ee8166f23f47; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda
I20260709 13:04:22.277206 12575 raft_consensus.cc:515] T d71ca3a1ba6b48478aaf62741bf3cf6a P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:22.506879 11066 leader_election.cc:304] T 8a69f32c53f24a50b7852d143665d7b4 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 10ca5ee68ad64b3283b9d235352c9d23; no voters: 
I20260709 13:04:22.414734 12387 raft_consensus.cc:2749] T e5ba72ccd854430baadbc636dfd17136 P 0172fed22b6b4890a7fdc953fa128af4 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:04:22.511893 10950 raft_consensus.cc:1240] T d7ce47b9a92d43dd88d95bcda5038597 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Rejecting Update request from peer 186b12638a87410487467df721a3cbda for earlier term 2. Current term is 3. Ops: []
I20260709 13:04:22.344488 12559 raft_consensus.cc:515] T 1e4f2241a94a41f38e481137f868b83d P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:22.526263 10951 raft_consensus.cc:2393] T 836251ebc2b74f5da88b8a8b2338e2a5 P 10ca5ee68ad64b3283b9d235352c9d23 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 0172fed22b6b4890a7fdc953fa128af4 in current term 5: Already voted for candidate 10ca5ee68ad64b3283b9d235352c9d23 in this term.
W20260709 13:04:22.532147 12562 log.cc:927] Time spent T 75492f86fde046c3900df3eac8292fd4 P 0172fed22b6b4890a7fdc953fa128af4: Append to log took a long time: real 0.075s	user 0.000s	sys 0.002s
I20260709 13:04:22.534013 12585 raft_consensus.cc:493] T 868e92339c2f4508af0afbd07c814d64 P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:22.535916 12585 raft_consensus.cc:515] T 868e92339c2f4508af0afbd07c814d64 P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:22.505693 12542 raft_consensus.cc:493] T 4f9eb06ed01f4ce68be3af8a8434665b P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:22.515283 12534 leader_election.cc:290] T 7709387f4d3d4f9992d97a0fbca39a40 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
W20260709 13:04:22.563493 12496 consensus_meta.cc:287] T c51782cd9d064ec0949e84e24dd4de67 P 0172fed22b6b4890a7fdc953fa128af4: Time spent flushing consensus metadata: real 1.011s	user 0.018s	sys 0.013s
I20260709 13:04:22.559149 12250 leader_election.cc:290] T 4c2a35bb99374e93b573cf870d04faaa P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:22.647719 12590 raft_consensus.cc:493] T 7757ecbf402e47bfbc93d0447741e26c P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:22.411350 12476 consensus_queue.cc:237] T 30f628e3d7914d609e7f1d980a74c654 P 186b12638a87410487467df721a3cbda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:22.659519 12589 raft_consensus.cc:3060] T 30f628e3d7914d609e7f1d980a74c654 P 873357742d5843579556ee8166f23f47 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:04:22.550804 12575 leader_election.cc:290] T d71ca3a1ba6b48478aaf62741bf3cf6a P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:22.601568 10948 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d439326f23747eaab3c1e7da2d6b785" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:22.649595 12590 raft_consensus.cc:515] T 7757ecbf402e47bfbc93d0447741e26c P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:22.657217 12526 leader_election.cc:290] T 836251ebc2b74f5da88b8a8b2338e2a5 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 5 election: Requested vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:22.517551 12394 raft_consensus.cc:515] T 87cd8f5097624c8692f1d83544d8881e P 0172fed22b6b4890a7fdc953fa128af4 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:22.675189 12578 leader_election.cc:290] T d7ce47b9a92d43dd88d95bcda5038597 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:22.422183 12581 raft_consensus.cc:515] T 57b6a2418c1342e7924195e9c1dbd357 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
W20260709 13:04:22.733039 12541 consensus_meta.cc:287] T f34f2bc02d85444f9096e0f5b4fe4331 P 186b12638a87410487467df721a3cbda: Time spent flushing consensus metadata: real 0.659s	user 0.020s	sys 0.006s
W20260709 13:04:22.767431 10992 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:22.639961 12496 raft_consensus.cc:515] T c51782cd9d064ec0949e84e24dd4de67 P 0172fed22b6b4890a7fdc953fa128af4 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:22.585984 12542 raft_consensus.cc:515] T 4f9eb06ed01f4ce68be3af8a8434665b P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:22.746766 12193 leader_election.cc:290] T b67eff0634e540eab91cf8cee795969f P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:22.757431 12559 leader_election.cc:290] T 1e4f2241a94a41f38e481137f868b83d P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
W20260709 13:04:22.820901 12593 log.cc:927] Time spent T 5d439326f23747eaab3c1e7da2d6b785 P 186b12638a87410487467df721a3cbda: Append to log took a long time: real 0.115s	user 0.001s	sys 0.002s
I20260709 13:04:22.869899 11100 raft_consensus.cc:1240] T c51782cd9d064ec0949e84e24dd4de67 P 0172fed22b6b4890a7fdc953fa128af4 [term 5 FOLLOWER]: Rejecting Update request from peer 186b12638a87410487467df721a3cbda for earlier term 4. Current term is 5. Ops: []
I20260709 13:04:22.870220 11098 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37676 (request call id 382) took 1659 ms. Trace:
I20260709 13:04:22.809499 12250 raft_consensus.cc:493] T acc241e7273d49149984968596242712 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:22.851694 11024 raft_consensus.cc:1275] T 4f9eb06ed01f4ce68be3af8a8434665b P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Refusing update from remote peer 186b12638a87410487467df721a3cbda: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 13:04:22.851694 12394 leader_election.cc:290] T 87cd8f5097624c8692f1d83544d8881e P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 6 election: Requested vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:22.830155 12590 leader_election.cc:290] T 7757ecbf402e47bfbc93d0447741e26c P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:22.904644 11098 rpcz_store.cc:276] 0709 13:04:21.210486 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:21.210613 (+   127us) service_pool.cc:224] Handling call
0709 13:04:21.223786 (+ 13173us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:22.382353 (+1158567us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:22.869330 (+486977us) spinlock_profiling.cc:243] Waited 1.01 s on lock 0x7b5400483fd4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:04:22.869970 (+   640us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2323552128}
I20260709 13:04:22.774691 12541 raft_consensus.cc:515] T f34f2bc02d85444f9096e0f5b4fe4331 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
W20260709 13:04:22.905395 12595 log.cc:927] Time spent T c51782cd9d064ec0949e84e24dd4de67 P 10ca5ee68ad64b3283b9d235352c9d23: Append to log took a long time: real 0.079s	user 0.005s	sys 0.000s
I20260709 13:04:22.936035 11098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a64b7e99d6c54934a7de6f6dffcf5e94" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
W20260709 13:04:22.942194 12594 log.cc:927] Time spent T d270b9e5481245d9b5340a454a7cd10b P 10ca5ee68ad64b3283b9d235352c9d23: Append to log took a long time: real 0.118s	user 0.002s	sys 0.002s
    @     0x7f4263283a12 testing::Test::Run() at ??:0
I20260709 13:04:22.938527 11098 raft_consensus.cc:2468] T a64b7e99d6c54934a7de6f6dffcf5e94 P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 4.
I20260709 13:04:22.953042 11098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75492f86fde046c3900df3eac8292fd4" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:22.914434 12250 raft_consensus.cc:515] T acc241e7273d49149984968596242712 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:22.981984 12585 leader_election.cc:290] T 868e92339c2f4508af0afbd07c814d64 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:22.983604 11098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f46a5102fd64b45bf1c4dff863d2b4d" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:22.984465 11098 raft_consensus.cc:2468] T 6f46a5102fd64b45bf1c4dff863d2b4d P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 2.
I20260709 13:04:22.986106 11098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a154cc8b181f460abf4880064379948b" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:22.986411 12500 raft_consensus.cc:3055] T e5a5f5a608504650b48fd27f163faed6 P 186b12638a87410487467df721a3cbda [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:04:22.965842 12542 leader_election.cc:290] T 4f9eb06ed01f4ce68be3af8a8434665b P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:22.997382 11098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4cd3d7dfbbf4096908ff11e9dc57141" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:22.998418 11098 raft_consensus.cc:2468] T b4cd3d7dfbbf4096908ff11e9dc57141 P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 4.
I20260709 13:04:22.987357 11096 raft_consensus.cc:1275] T 5d439326f23747eaab3c1e7da2d6b785 P 0172fed22b6b4890a7fdc953fa128af4 [term 5 FOLLOWER]: Refusing update from remote peer 10ca5ee68ad64b3283b9d235352c9d23: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260709 13:04:22.994607 12500 raft_consensus.cc:740] T e5a5f5a608504650b48fd27f163faed6 P 186b12638a87410487467df721a3cbda [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 186b12638a87410487467df721a3cbda, State: Running, Role: LEADER
I20260709 13:04:23.002959 12581 leader_election.cc:290] T 57b6a2418c1342e7924195e9c1dbd357 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
W20260709 13:04:23.004117 12532 consensus_meta.cc:287] T 72b111ad7b5f4753ab1d2b5ee1114840 P 10ca5ee68ad64b3283b9d235352c9d23: Time spent flushing consensus metadata: real 1.176s	user 0.025s	sys 0.012s
W20260709 13:04:23.003408 12592 log.cc:927] Time spent T 525bfdbd6d934afd823f7d3ee3f9d052 P 10ca5ee68ad64b3283b9d235352c9d23: Append to log took a long time: real 0.073s	user 0.002s	sys 0.000s
I20260709 13:04:23.010946 12424 leader_election.cc:290] T 525bfdbd6d934afd823f7d3ee3f9d052 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:23.011445 12532 raft_consensus.cc:515] T 72b111ad7b5f4753ab1d2b5ee1114840 P 10ca5ee68ad64b3283b9d235352c9d23 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:23.007356 12496 leader_election.cc:290] T c51782cd9d064ec0949e84e24dd4de67 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 5 election: Requested vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:23.015700 12541 leader_election.cc:290] T f34f2bc02d85444f9096e0f5b4fe4331 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 2 election: Requested vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:23.030746 12532 leader_election.cc:290] T 72b111ad7b5f4753ab1d2b5ee1114840 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 5 election: Requested vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:23.035970 12511 leader_election.cc:290] T 5d439326f23747eaab3c1e7da2d6b785 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:23.015925 12500 consensus_queue.cc:260] T e5a5f5a608504650b48fd27f163faed6 P 186b12638a87410487467df721a3cbda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:23.040244 12500 raft_consensus.cc:3060] T e5a5f5a608504650b48fd27f163faed6 P 186b12638a87410487467df721a3cbda [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:04:23.007496 11096 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37666 (request call id 267) took 1707 ms. Trace:
I20260709 13:04:23.042721 10952 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "836251ebc2b74f5da88b8a8b2338e2a5" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:23.043774 10952 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49424 (request call id 268) took 1480 ms. Trace:
I20260709 13:04:23.044108 10952 rpcz_store.cc:276] 0709 13:04:21.563554 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:21.563688 (+   134us) service_pool.cc:224] Handling call
0709 13:04:23.042497 (+1478809us) spinlock_profiling.cc:243] Waited 5.2 ms on lock 0x7b580001d198. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f426303db71 00007f426303da22 00007f426bf56c09 00007f426b0b0ee0 00007f426b17a0ab 00007f426b17a131 00007f426b1280a0 00007f426b127984 00007f42646a26d4 00007f42646a2662
0709 13:04:23.043673 (+  1176us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":11968896}
W20260709 13:04:23.036180 12489 consensus_meta.cc:287] T 72b111ad7b5f4753ab1d2b5ee1114840 P 186b12638a87410487467df721a3cbda: Time spent flushing consensus metadata: real 0.877s	user 0.021s	sys 0.009s
I20260709 13:04:23.047338 12489 raft_consensus.cc:515] T 72b111ad7b5f4753ab1d2b5ee1114840 P 186b12638a87410487467df721a3cbda [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:23.048723 10879 raft_consensus.cc:1240] T 72b111ad7b5f4753ab1d2b5ee1114840 P 186b12638a87410487467df721a3cbda [term 5 FOLLOWER]: Rejecting Update request from peer 0172fed22b6b4890a7fdc953fa128af4 for earlier term 4. Current term is 5. Ops: []
I20260709 13:04:23.051208 12489 leader_election.cc:290] T 72b111ad7b5f4753ab1d2b5ee1114840 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 5 election: Requested vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:23.042481 11096 rpcz_store.cc:276] 0709 13:04:21.300109 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:21.302485 (+  2376us) service_pool.cc:224] Handling call
0709 13:04:22.942638 (+1640153us) spinlock_profiling.cc:243] Waited 3.53 ms on lock 0x7b5800117698. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f426303db71 00007f426303deee 0000000000375939 00007f426b0b11e0 00007f426b17a0c4 00007f426b17a131 00007f426b13bfc0 00007f426b126929 00007f426b1263d2 00007f426469dbd4
0709 13:04:22.950978 (+  8340us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:23.001892 (+ 50914us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:23.006621 (+  4729us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":8127744}
I20260709 13:04:23.055864 11096 raft_consensus.cc:3060] T a977842ba4b744f4bb2febd55d286006 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:04:23.021741 12250 leader_election.cc:290] T acc241e7273d49149984968596242712 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
W20260709 13:04:23.059326 10844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:23.070078 10873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d7ce47b9a92d43dd88d95bcda5038597" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:23.072336 10872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "836251ebc2b74f5da88b8a8b2338e2a5" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:04:23.073824 10872 raft_consensus.cc:2393] T 836251ebc2b74f5da88b8a8b2338e2a5 P 186b12638a87410487467df721a3cbda [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 10ca5ee68ad64b3283b9d235352c9d23 in current term 5: Already voted for candidate 0172fed22b6b4890a7fdc953fa128af4 in this term.
I20260709 13:04:23.074661 11099 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37676 (request call id 383) took 1670 ms. Trace:
I20260709 13:04:23.075165 11099 rpcz_store.cc:276] 0709 13:04:21.404633 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:21.404792 (+   159us) service_pool.cc:224] Handling call
0709 13:04:21.405118 (+   326us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:21.491899 (+ 86781us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:21.491939 (+    40us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:21.491999 (+    60us) raft_consensus.cc:1645] Marking committed up to 3
0709 13:04:21.492035 (+    36us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:23.074377 (+1582342us) spinlock_profiling.cc:243] Waited 86.5 ms on lock 0x7b5400154ad4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:04:23.074457 (+    80us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:23.074593 (+   136us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":198985088}
I20260709 13:04:23.084264 12298 raft_consensus.cc:493] T 57b6a2418c1342e7924195e9c1dbd357 P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 186b12638a87410487467df721a3cbda)
I20260709 13:04:23.077682 12250 raft_consensus.cc:493] T a5299b8d044f41b7b68057e63d17b159 P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 873357742d5843579556ee8166f23f47)
W20260709 13:04:23.083694 12479 consensus_meta.cc:287] T 401a9cf11eac479aa445ec8c1d6e33b2 P 186b12638a87410487467df721a3cbda: Time spent flushing consensus metadata: real 0.580s	user 0.023s	sys 0.004s
I20260709 13:04:23.083142 12462 raft_consensus.cc:493] T d371c5e27bd843469d2e2e453734cb4a P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0172fed22b6b4890a7fdc953fa128af4)
I20260709 13:04:23.090876 12462 raft_consensus.cc:515] T d371c5e27bd843469d2e2e453734cb4a P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:23.092267 12534 raft_consensus.cc:493] T 732c4adeeff6422497aa7db4a82f535b P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:23.092862 12534 raft_consensus.cc:515] T 732c4adeeff6422497aa7db4a82f535b P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:23.093369 12202 raft_consensus.cc:493] T a977842ba4b744f4bb2febd55d286006 P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 10ca5ee68ad64b3283b9d235352c9d23)
I20260709 13:04:23.094177 12202 raft_consensus.cc:515] T a977842ba4b744f4bb2febd55d286006 P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:23.096108 11095 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37676 (request call id 386) took 1488 ms. Trace:
I20260709 13:04:23.097024 12202 leader_election.cc:290] T a977842ba4b744f4bb2febd55d286006 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:23.088136 12250 raft_consensus.cc:515] T a5299b8d044f41b7b68057e63d17b159 P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
W20260709 13:04:23.098652 11069 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:38967, user_credentials={real_user=slave}} blocked reactor thread for 1.07949e+06us
W20260709 13:04:23.099491 11069 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.107.193:38967 timed out after 1.690s (SENT) [suppressed 51 similar messages]
I20260709 13:04:23.099825 11026 raft_consensus.cc:1275] T 9a1716c0adf34bee8a4e99341ba5788f P 873357742d5843579556ee8166f23f47 [term 5 FOLLOWER]: Refusing update from remote peer 186b12638a87410487467df721a3cbda: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260709 13:04:23.100196 12250 leader_election.cc:290] T a5299b8d044f41b7b68057e63d17b159 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:23.100497 12220 raft_consensus.cc:2749] T 75492f86fde046c3900df3eac8292fd4 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:04:23.102306 11020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "732c4adeeff6422497aa7db4a82f535b" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
I20260709 13:04:23.103305 11020 raft_consensus.cc:3060] T 732c4adeeff6422497aa7db4a82f535b P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:04:23.104928 12572 raft_consensus.cc:493] T a1ec2b50c6ad4f7c8033476208abb26d P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:23.105482 12572 raft_consensus.cc:515] T a1ec2b50c6ad4f7c8033476208abb26d P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:23.106652 12553 raft_consensus.cc:493] T bac7e5040f984ba2b7146bd36af7267e P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:23.107084 11023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c2a35bb99374e93b573cf870d04faaa" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:23.107163 12553 raft_consensus.cc:515] T bac7e5040f984ba2b7146bd36af7267e P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:23.109725 11021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e4f2241a94a41f38e481137f868b83d" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:23.110230 12570 raft_consensus.cc:493] T d7ce47b9a92d43dd88d95bcda5038597 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 186b12638a87410487467df721a3cbda)
I20260709 13:04:23.088917 12548 raft_consensus.cc:493] T 1e4f2241a94a41f38e481137f868b83d P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 10ca5ee68ad64b3283b9d235352c9d23)
I20260709 13:04:23.110841 12570 raft_consensus.cc:515] T d7ce47b9a92d43dd88d95bcda5038597 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:23.111093 12548 raft_consensus.cc:515] T 1e4f2241a94a41f38e481137f868b83d P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:23.085114 12298 raft_consensus.cc:515] T 57b6a2418c1342e7924195e9c1dbd357 P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:23.113348 12570 leader_election.cc:290] T d7ce47b9a92d43dd88d95bcda5038597 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:23.113904 12570 raft_consensus.cc:493] T f371d2565dd84667b5613848604949bb P 0172fed22b6b4890a7fdc953fa128af4 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:23.114972 12298 leader_election.cc:290] T 57b6a2418c1342e7924195e9c1dbd357 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:23.114001 12548 leader_election.cc:290] T 1e4f2241a94a41f38e481137f868b83d P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
W20260709 13:04:23.116153 11066 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 609468us
I20260709 13:04:23.118568 11029 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acc241e7273d49149984968596242712" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:23.123704 12394 raft_consensus.cc:493] T 7757ecbf402e47bfbc93d0447741e26c P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:23.124017 11028 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c160ddfe9b34bdaa3835d0915c1e06f" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:23.124554 12394 raft_consensus.cc:515] T 7757ecbf402e47bfbc93d0447741e26c P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:23.125768 11028 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555217d45d9b4ee492dbb3ddaf0f7c02" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:23.127166 11021 raft_consensus.cc:2410] T 1e4f2241a94a41f38e481137f868b83d P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 186b12638a87410487467df721a3cbda for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260709 13:04:23.128144 11026 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "868e92339c2f4508af0afbd07c814d64" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:23.128805 12563 raft_consensus.cc:493] T a610d1a71f52432e88fe56130f921fa2 P 0172fed22b6b4890a7fdc953fa128af4 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:23.128916 11026 raft_consensus.cc:2468] T 868e92339c2f4508af0afbd07c814d64 P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 4.
I20260709 13:04:23.130244 11029 raft_consensus.cc:2410] T acc241e7273d49149984968596242712 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 186b12638a87410487467df721a3cbda for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 13:04:23.087710 12589 raft_consensus.cc:2749] T 30f628e3d7914d609e7f1d980a74c654 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:04:23.158901 10879 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51704 (request call id 278) took 1108 ms. Trace:
I20260709 13:04:23.159521 10879 rpcz_store.cc:276] 0709 13:04:22.041848 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:22.041995 (+   147us) service_pool.cc:224] Handling call
0709 13:04:22.042407 (+   412us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:23.049208 (+1006801us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:23.149417 (+100209us) spinlock_profiling.cc:243] Waited 1.01 s on lock 0x7b5400370fd4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:04:23.149709 (+   292us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2314029312}
I20260709 13:04:23.161181 11103 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37678 (request call id 235) took 1275 ms. Trace:
I20260709 13:04:23.109874 12553 leader_election.cc:290] T bac7e5040f984ba2b7146bd36af7267e P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:23.120721 12387 raft_consensus.cc:2804] T 8a69f32c53f24a50b7852d143665d7b4 P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260709 13:04:23.172745 10992 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:23.175129 10844 leader_election.cc:304] T 868e92339c2f4508af0afbd07c814d64 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:04:23.161650 11103 rpcz_store.cc:276] 0709 13:04:21.885522 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:21.885782 (+   260us) service_pool.cc:224] Handling call
0709 13:04:21.886181 (+   399us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:21.886516 (+   335us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 13:04:22.047249 (+160733us) log.cc:844] Serialized 14 byte log entry
0709 13:04:23.149932 (+1102683us) log.cc:844] Serialized 14 byte log entry
0709 13:04:23.159371 (+  9439us) log.cc:844] Serialized 14 byte log entry
0709 13:04:23.160636 (+  1265us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:23.160802 (+   166us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:04:23.160868 (+    66us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:23.160968 (+   100us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:23.161095 (+   127us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1005689,"prepare.queue_time_us":34231,"prepare.run_cpu_time_us":60,"prepare.run_wall_time_us":61,"raft.queue_time_us":1006875,"raft.run_cpu_time_us":107,"raft.run_wall_time_us":107,"thread_start_us":255296,"threads_started":3,"wal-append.queue_time_us":109577}
I20260709 13:04:23.181730 11103 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d71ca3a1ba6b48478aaf62741bf3cf6a" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:23.182658 11103 raft_consensus.cc:2468] T d71ca3a1ba6b48478aaf62741bf3cf6a P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 3.
I20260709 13:04:23.184441 11103 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "836251ebc2b74f5da88b8a8b2338e2a5" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:04:23.185458 11103 raft_consensus.cc:2393] T 836251ebc2b74f5da88b8a8b2338e2a5 P 0172fed22b6b4890a7fdc953fa128af4 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 10ca5ee68ad64b3283b9d235352c9d23 in current term 5: Already voted for candidate 0172fed22b6b4890a7fdc953fa128af4 in this term.
I20260709 13:04:23.181492 12537 raft_consensus.cc:2804] T 868e92339c2f4508af0afbd07c814d64 P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:04:23.193289 11101 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37676 (request call id 390) took 1179 ms. Trace:
I20260709 13:04:23.096575 11095 rpcz_store.cc:276] 0709 13:04:21.607295 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:21.616895 (+  9600us) service_pool.cc:224] Handling call
0709 13:04:21.644374 (+ 27479us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:04:21.706823 (+ 62449us) raft_consensus.cc:1534] Early marking committed up to index 1
0709 13:04:21.826805 (+119982us) log.cc:844] Serialized 14 byte log entry
0709 13:04:21.866376 (+ 39571us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:04:22.002511 (+136135us) log.cc:844] Serialized 52 byte log entry
0709 13:04:22.002772 (+   261us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:04:22.003101 (+   329us) log.cc:844] Serialized 14 byte log entry
0709 13:04:22.421035 (+417934us) spinlock_profiling.cc:243] Waited 6.52 ms on lock 0x7b180004b6a0. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f42630753f3 00007f42630777c1 00007f4263077779 00007f4263077709 00007f42630776d1 00007f42630769cd 000000000038f354 000000000038f199 00007f426305883a
0709 13:04:23.091697 (+670662us) spinlock_profiling.cc:243] Waited 1.68 ms on lock 0x7b180004b6a0. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f4262fefc0a 00007f426c72f219 00007f426a03658f 00007f4263075571 00007f42630777c1 00007f4263077779 00007f4263077709 00007f42630776d1 00007f42630769cd
0709 13:04:23.095640 (+  3943us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:23.095764 (+   124us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:23.095821 (+    57us) raft_consensus.cc:1685] finished
0709 13:04:23.095836 (+    15us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:23.096019 (+   183us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1098350,"prepare.queue_time_us":112174,"prepare.run_cpu_time_us":93,"prepare.run_wall_time_us":96,"raft.queue_time_us":1136802,"raft.run_cpu_time_us":524,"raft.run_wall_time_us":16391,"spinlock_wait_cycles":18862080,"thread_start_us":83658,"threads_started":2,"wal-append.queue_time_us":71245}
I20260709 13:04:23.193742 11101 rpcz_store.cc:276] 0709 13:04:22.014233 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:22.054765 (+ 40532us) service_pool.cc:224] Handling call
0709 13:04:23.193071 (+1138306us) spinlock_profiling.cc:243] Waited 63.1 ms on lock 0x7b540063ae54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:04:23.193195 (+   124us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":145050880}
I20260709 13:04:23.195418 10880 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51696 (request call id 337) took 2451 ms. Trace:
I20260709 13:04:23.195842 10880 rpcz_store.cc:276] 0709 13:04:20.744230 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:20.845041 (+100811us) service_pool.cc:224] Handling call
0709 13:04:20.845443 (+   402us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:21.349164 (+503721us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:23.195122 (+1845958us) spinlock_profiling.cc:243] Waited 496 ms on lock 0x7b5400290254. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:04:23.195328 (+   206us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1141234944}
I20260709 13:04:23.189363 12537 raft_consensus.cc:493] T 868e92339c2f4508af0afbd07c814d64 P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:23.197345 12537 raft_consensus.cc:3060] T 868e92339c2f4508af0afbd07c814d64 P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:04:23.198957 12329 raft_consensus.cc:3060] T e5ba72ccd854430baadbc636dfd17136 P 873357742d5843579556ee8166f23f47 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:04:23.203958 10946 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49434 (request call id 379) took 1142 ms. Trace:
I20260709 13:04:23.204561 10946 rpcz_store.cc:276] 0709 13:04:22.061188 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:22.061326 (+   138us) service_pool.cc:224] Handling call
0709 13:04:22.075293 (+ 13967us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:22.293302 (+218009us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:22.293458 (+   156us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:22.293831 (+   373us) raft_consensus.cc:1645] Marking committed up to 3
0709 13:04:22.294027 (+   196us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:23.203513 (+909486us) spinlock_profiling.cc:243] Waited 217 ms on lock 0x7b5400789954. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:04:23.203638 (+   125us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:23.203851 (+   213us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":498940672}
I20260709 13:04:23.211051 10951 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49424 (request call id 263) took 1664 ms. Trace:
I20260709 13:04:23.211632 10951 rpcz_store.cc:276] 0709 13:04:21.546163 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:21.546540 (+   377us) service_pool.cc:224] Handling call
0709 13:04:21.914333 (+367793us) spinlock_profiling.cc:243] Waited 4.4 ms on lock 0x7b580001d198. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f426303db71 00007f426303da22 00007f426bf56c09 00007f426b0b0ee0 00007f426b17a0ab 00007f426b17a131 00007f426b1280a0 00007f426b127984 00007f42646a26d4 00007f42646a2662
0709 13:04:23.210752 (+1296419us) spinlock_profiling.cc:243] Waited 592 ms on lock 0x7b54005cfdd4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:04:23.210959 (+   207us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1371243008}
I20260709 13:04:23.216050 12462 leader_election.cc:290] T d371c5e27bd843469d2e2e453734cb4a P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:23.218387 11026 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5299b8d044f41b7b68057e63d17b159" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:23.219573 11094 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37678 (request call id 233) took 1336 ms. Trace:
I20260709 13:04:23.220120 11094 rpcz_store.cc:276] 0709 13:04:21.883729 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:21.884137 (+   408us) service_pool.cc:224] Handling call
0709 13:04:21.884493 (+   356us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:21.905063 (+ 20570us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:23.219247 (+1314184us) spinlock_profiling.cc:243] Waited 10.5 ms on lock 0x7b540058bbd4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:04:23.219484 (+   237us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":24207104}
I20260709 13:04:23.222108 11972 raft_consensus.cc:493] T acc241e7273d49149984968596242712 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0172fed22b6b4890a7fdc953fa128af4)
I20260709 13:04:23.222920 11972 raft_consensus.cc:515] T acc241e7273d49149984968596242712 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:23.226801 11102 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37666 (request call id 273) took 1903 ms. Trace:
I20260709 13:04:23.227443 11102 rpcz_store.cc:276] 0709 13:04:21.323426 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:21.323756 (+   330us) service_pool.cc:224] Handling call
0709 13:04:21.393542 (+ 69786us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:21.397080 (+  3538us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:23.226494 (+1829414us) spinlock_profiling.cc:243] Waited 53.6 ms on lock 0x7b540063a950. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661618ad 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:04:23.226703 (+   209us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":123214720}
I20260709 13:04:23.229473 11097 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37678 (request call id 237) took 1340 ms. Trace:
I20260709 13:04:23.230010 11097 rpcz_store.cc:276] 0709 13:04:21.889096 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:21.889240 (+   144us) service_pool.cc:224] Handling call
0709 13:04:21.904560 (+ 15320us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:22.354901 (+450341us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:23.229087 (+874186us) spinlock_profiling.cc:243] Waited 326 ms on lock 0x7b54001559d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:04:23.229382 (+   295us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":750273920}
I20260709 13:04:23.236020 10995 leader_election.cc:304] T 0e5d327630e5438b878035d28efe6c0c P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 873357742d5843579556ee8166f23f47; no voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda
I20260709 13:04:23.238595 12434 raft_consensus.cc:2749] T 0e5d327630e5438b878035d28efe6c0c P 873357742d5843579556ee8166f23f47 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:04:23.245138 12535 raft_consensus.cc:493] T d371c5e27bd843469d2e2e453734cb4a P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0172fed22b6b4890a7fdc953fa128af4)
I20260709 13:04:23.245951 12535 raft_consensus.cc:515] T d371c5e27bd843469d2e2e453734cb4a P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:23.114377 12570 raft_consensus.cc:515] T f371d2565dd84667b5613848604949bb P 0172fed22b6b4890a7fdc953fa128af4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:23.250982 12487 raft_consensus.cc:493] T f371d2565dd84667b5613848604949bb P 186b12638a87410487467df721a3cbda [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:23.251641 12487 raft_consensus.cc:515] T f371d2565dd84667b5613848604949bb P 186b12638a87410487467df721a3cbda [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:23.252238 12570 leader_election.cc:290] T f371d2565dd84667b5613848604949bb P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:23.129501 12563 raft_consensus.cc:515] T a610d1a71f52432e88fe56130f921fa2 P 0172fed22b6b4890a7fdc953fa128af4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:23.254966 12541 raft_consensus.cc:493] T a610d1a71f52432e88fe56130f921fa2 P 186b12638a87410487467df721a3cbda [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:23.255174 12563 leader_election.cc:290] T a610d1a71f52432e88fe56130f921fa2 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:23.184608 12527 raft_consensus.cc:493] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 873357742d5843579556ee8166f23f47 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 186b12638a87410487467df721a3cbda)
I20260709 13:04:23.255746 12541 raft_consensus.cc:515] T a610d1a71f52432e88fe56130f921fa2 P 186b12638a87410487467df721a3cbda [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:23.256680 12527 raft_consensus.cc:515] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 873357742d5843579556ee8166f23f47 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:23.259119 12572 leader_election.cc:290] T a1ec2b50c6ad4f7c8033476208abb26d P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:23.259626 12479 raft_consensus.cc:493] T e89e640169e14e2d98f968f76027daa7 P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:23.261076 12479 raft_consensus.cc:515] T e89e640169e14e2d98f968f76027daa7 P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:23.263602 12479 leader_election.cc:290] T e89e640169e14e2d98f968f76027daa7 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:23.180745 12250 consensus_queue.cc:1048] T 9a1716c0adf34bee8a4e99341ba5788f P 186b12638a87410487467df721a3cbda [LEADER]: Connected to new peer: Peer: permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.089s
I20260709 13:04:23.270089 10879 raft_consensus.cc:1275] T f371d2565dd84667b5613848604949bb P 186b12638a87410487467df721a3cbda [term 5 FOLLOWER]: Refusing update from remote peer 873357742d5843579556ee8166f23f47: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260709 13:04:23.271349 10873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2567c2d84e4648d7ba5dc23a3df32817" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:23.170938 12387 raft_consensus.cc:493] T 8a69f32c53f24a50b7852d143665d7b4 P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 186b12638a87410487467df721a3cbda)
I20260709 13:04:23.272891 12387 raft_consensus.cc:3060] T 8a69f32c53f24a50b7852d143665d7b4 P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:04:23.277464 12487 leader_election.cc:290] T f371d2565dd84667b5613848604949bb P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:23.281186 11021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f371d2565dd84667b5613848604949bb" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:23.282694 11021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a610d1a71f52432e88fe56130f921fa2" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:23.283910 12394 leader_election.cc:290] T 7757ecbf402e47bfbc93d0447741e26c P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:23.188156 11023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7757ecbf402e47bfbc93d0447741e26c" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:23.187119 12534 leader_election.cc:290] T 732c4adeeff6422497aa7db4a82f535b P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:23.287272 11096 raft_consensus.cc:1275] T a977842ba4b744f4bb2febd55d286006 P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Refusing update from remote peer 10ca5ee68ad64b3283b9d235352c9d23: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20260709 13:04:23.288142 11096 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37666 (request call id 279) took 1231 ms. Trace:
I20260709 13:04:23.284065 11021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e89e640169e14e2d98f968f76027daa7" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:23.290237 12535 leader_election.cc:290] T d371c5e27bd843469d2e2e453734cb4a P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:23.282227 11972 leader_election.cc:290] T acc241e7273d49149984968596242712 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:23.291464 11022 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46642 (request call id 206) took 1891 ms. Trace:
I20260709 13:04:23.292034 11022 rpcz_store.cc:276] 0709 13:04:21.399627 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:21.402345 (+  2718us) service_pool.cc:224] Handling call
0709 13:04:21.466370 (+ 64025us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:21.532810 (+ 66440us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:21.535968 (+  3158us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:21.596188 (+ 60220us) spinlock_profiling.cc:243] Waited 4.23 ms on lock 0x7b18003d1200. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f42630753f3 00007f42630777c1 00007f4263077779 00007f4263077709 00007f42630776d1 00007f42630769cd 000000000038f354 000000000038f199 00007f426305883a
0709 13:04:23.072307 (+1476119us) spinlock_profiling.cc:243] Waited 1.99 ms on lock 0x7b2c0010d988. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4264a5ba11 00007f4264a6b372 00007f426619da01 00007f426616243a 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2
0709 13:04:23.073004 (+   697us) log.cc:844] Serialized 26 byte log entry
0709 13:04:23.170655 (+ 97651us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:04:23.176919 (+  6264us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:23.183911 (+  6992us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:23.291127 (+107216us) raft_consensus.cc:1685] finished
0709 13:04:23.291169 (+    42us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:23.291378 (+   209us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":11981,"raft.queue_time_us":45321,"raft.run_cpu_time_us":195,"raft.run_wall_time_us":11878,"spinlock_wait_cycles":14298880,"thread_start_us":78949,"threads_started":1,"wal-append.queue_time_us":212739}
I20260709 13:04:23.295369 10873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a1716c0adf34bee8a4e99341ba5788f" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:23.298274 11029 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a1716c0adf34bee8a4e99341ba5788f" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:23.298652 12513 raft_consensus.cc:493] T a154cc8b181f460abf4880064379948b P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:23.299037 11029 raft_consensus.cc:2468] T 9a1716c0adf34bee8a4e99341ba5788f P 873357742d5843579556ee8166f23f47 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 5.
I20260709 13:04:23.299220 12513 raft_consensus.cc:515] T a154cc8b181f460abf4880064379948b P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:23.301513 11026 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d371c5e27bd843469d2e2e453734cb4a" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:23.302932 10874 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51696 (request call id 342) took 1256 ms. Trace:
I20260709 13:04:23.303714 11100 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37678 (request call id 238) took 1295 ms. Trace:
I20260709 13:04:23.302922 11026 raft_consensus.cc:2468] T d371c5e27bd843469d2e2e453734cb4a P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 4.
I20260709 13:04:23.304329 11024 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46642 (request call id 205) took 1905 ms. Trace:
I20260709 13:04:23.305676 10879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "868e92339c2f4508af0afbd07c814d64" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
W20260709 13:04:23.308286 11069 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:38967, user_credentials={real_user=slave}} blocked reactor thread for 208845us
I20260709 13:04:23.310978 11103 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d7ce47b9a92d43dd88d95bcda5038597" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:23.312099 11103 raft_consensus.cc:2468] T d7ce47b9a92d43dd88d95bcda5038597 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 2.
I20260709 13:04:23.315266 10844 leader_election.cc:304] T d371c5e27bd843469d2e2e453734cb4a P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:04:23.316473 11024 rpcz_store.cc:276] 0709 13:04:21.399285 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:21.402257 (+  2972us) service_pool.cc:224] Handling call
0709 13:04:22.581490 (+1179233us) spinlock_profiling.cc:243] Waited 2.85 ms on lock 0x7b58007a2898. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f426303db71 00007f426303da22 00007f426bf56c09 00007f426b0b0ee0 00007f426b17a0ab 00007f426b17a131 00007f426b13bfc0 00007f426b126929 00007f426b1263d2 00007f426469dbd4
0709 13:04:22.623626 (+ 42136us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:22.916010 (+292384us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:23.304059 (+388049us) spinlock_profiling.cc:243] Waited 223 ms on lock 0x7b54004e02d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:04:23.304250 (+   191us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":520125184}
I20260709 13:04:23.317598 10873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7709387f4d3d4f9992d97a0fbca39a40" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:23.304179 11100 rpcz_store.cc:276] 0709 13:04:22.008553 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:22.151256 (+142703us) service_pool.cc:224] Handling call
0709 13:04:22.163795 (+ 12539us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:22.871768 (+707973us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:23.303439 (+431671us) spinlock_profiling.cc:243] Waited 706 ms on lock 0x7b540086fe54. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:04:23.303640 (+   201us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1622915840}
I20260709 13:04:23.319331 10875 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51696 (request call id 341) took 1273 ms. Trace:
I20260709 13:04:23.303478 10874 rpcz_store.cc:276] 0709 13:04:22.046380 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:22.051942 (+  5562us) service_pool.cc:224] Handling call
0709 13:04:22.058864 (+  6922us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:04:22.234874 (+176010us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:22.235006 (+   132us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:04:22.316613 (+ 81607us) log.cc:844] Serialized 52 byte log entry
0709 13:04:22.499884 (+183271us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:04:22.776705 (+276821us) log.cc:844] Serialized 14 byte log entry
0709 13:04:22.777564 (+   859us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:23.302464 (+524900us) spinlock_profiling.cc:243] Waited 69.4 ms on lock 0x7b54007f21d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:04:23.302527 (+    63us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:23.302577 (+    50us) raft_consensus.cc:1685] finished
0709 13:04:23.302690 (+   113us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:23.302862 (+   172us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":12019,"prepare.queue_time_us":402969,"prepare.run_cpu_time_us":215,"prepare.run_wall_time_us":6290,"raft.queue_time_us":40421,"raft.run_cpu_time_us":626,"raft.run_wall_time_us":20387,"spinlock_wait_cycles":159712640,"thread_start_us":409327,"threads_started":3,"wal-append.queue_time_us":299606}
I20260709 13:04:23.324070 10877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f9eb06ed01f4ce68be3af8a8434665b" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:23.319921 10875 rpcz_store.cc:276] 0709 13:04:22.046009 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:22.046136 (+   127us) service_pool.cc:224] Handling call
0709 13:04:22.053300 (+  7164us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:22.349790 (+296490us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:23.318986 (+969196us) spinlock_profiling.cc:243] Waited 219 ms on lock 0x7b540071d9d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:04:23.319239 (+   253us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":503733120}
I20260709 13:04:23.318328 10873 raft_consensus.cc:2468] T 7709387f4d3d4f9992d97a0fbca39a40 P 186b12638a87410487467df721a3cbda [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 7.
I20260709 13:04:23.326148 11094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "868e92339c2f4508af0afbd07c814d64" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:23.327099 11094 raft_consensus.cc:2468] T 868e92339c2f4508af0afbd07c814d64 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 3.
I20260709 13:04:23.302806 12513 leader_election.cc:290] T a154cc8b181f460abf4880064379948b P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:23.329429 10877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "732c4adeeff6422497aa7db4a82f535b" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
W20260709 13:04:23.331573 10843 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:23.288578 11096 rpcz_store.cc:276] 0709 13:04:22.056780 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:23.055366 (+998586us) service_pool.cc:224] Handling call
0709 13:04:23.055698 (+   332us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:23.287797 (+232099us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:23.288049 (+   252us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:23.338943 10873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e4f2241a94a41f38e481137f868b83d" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:23.332994 10877 raft_consensus.cc:2393] T 732c4adeeff6422497aa7db4a82f535b P 186b12638a87410487467df721a3cbda [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 873357742d5843579556ee8166f23f47 in current term 5: Already voted for candidate 186b12638a87410487467df721a3cbda in this term.
I20260709 13:04:23.340149 10873 raft_consensus.cc:2468] T 1e4f2241a94a41f38e481137f868b83d P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 4.
I20260709 13:04:23.338483 11097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7709387f4d3d4f9992d97a0fbca39a40" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
W20260709 13:04:23.341562 10843 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:23.341457 11097 raft_consensus.cc:2468] T 7709387f4d3d4f9992d97a0fbca39a40 P 0172fed22b6b4890a7fdc953fa128af4 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 7.
I20260709 13:04:23.326965 12586 raft_consensus.cc:2804] T d371c5e27bd843469d2e2e453734cb4a P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:04:23.319142 12220 raft_consensus.cc:493] T bb7d67be500947b3aef4c4245e33671c P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:23.354173 10877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d371c5e27bd843469d2e2e453734cb4a" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:23.327600 12394 raft_consensus.cc:493] T ff13ae9a47c7455f962406c6d131a3ff P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:23.327945 12572 raft_consensus.cc:493] T 5de5f0581e3b41dea729363d6ca563bb P 0172fed22b6b4890a7fdc953fa128af4 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 186b12638a87410487467df721a3cbda)
I20260709 13:04:23.354625 10871 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57b6a2418c1342e7924195e9c1dbd357" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:23.347295 12535 raft_consensus.cc:493] T 6a9353d1d230416296627d7cf8b1d425 P 186b12638a87410487467df721a3cbda [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:23.354599 12487 raft_consensus.cc:493] T 11de93cd9b9d457489ff1cb80f976c9a P 186b12638a87410487467df721a3cbda [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:23.353149 12586 raft_consensus.cc:493] T d371c5e27bd843469d2e2e453734cb4a P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Starting leader election (detected failure of leader 0172fed22b6b4890a7fdc953fa128af4)
I20260709 13:04:23.377632 11097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "525bfdbd6d934afd823f7d3ee3f9d052" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:23.378521 11097 raft_consensus.cc:2468] T 525bfdbd6d934afd823f7d3ee3f9d052 P 0172fed22b6b4890a7fdc953fa128af4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 5.
I20260709 13:04:23.375602 11067 leader_election.cc:304] T 9a1716c0adf34bee8a4e99341ba5788f P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:04:23.377712 11029 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a64b7e99d6c54934a7de6f6dffcf5e94" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:23.360033 12250 consensus_queue.cc:1048] T 4f9eb06ed01f4ce68be3af8a8434665b P 186b12638a87410487467df721a3cbda [LEADER]: Connected to new peer: Peer: permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.021s
I20260709 13:04:23.365644 12513 raft_consensus.cc:493] T e5a5f5a608504650b48fd27f163faed6 P 0172fed22b6b4890a7fdc953fa128af4 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 186b12638a87410487467df721a3cbda)
I20260709 13:04:23.387427 12553 raft_consensus.cc:493] T d270b9e5481245d9b5340a454a7cd10b P 0172fed22b6b4890a7fdc953fa128af4 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 186b12638a87410487467df721a3cbda)
I20260709 13:04:23.387306 12513 raft_consensus.cc:515] T e5a5f5a608504650b48fd27f163faed6 P 0172fed22b6b4890a7fdc953fa128af4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:23.359934 12220 raft_consensus.cc:515] T bb7d67be500947b3aef4c4245e33671c P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:23.388134 12553 raft_consensus.cc:515] T d270b9e5481245d9b5340a454a7cd10b P 0172fed22b6b4890a7fdc953fa128af4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:23.389184 11098 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37666 (request call id 278) took 1332 ms. Trace:
I20260709 13:04:23.389813 12513 leader_election.cc:290] T e5a5f5a608504650b48fd27f163faed6 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:23.390077 12220 leader_election.cc:290] T bb7d67be500947b3aef4c4245e33671c P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:23.383747 11021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a154cc8b181f460abf4880064379948b" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:23.380730 11096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f9eb06ed01f4ce68be3af8a8434665b" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:23.391966 11029 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a610d1a71f52432e88fe56130f921fa2" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:23.389590 11098 rpcz_store.cc:276] 0709 13:04:22.056492 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:23.009959 (+953467us) service_pool.cc:224] Handling call
0709 13:04:23.010297 (+   338us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:23.128604 (+118307us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:23.128646 (+    42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:23.129142 (+   496us) log.cc:844] Serialized 26 byte log entry
0709 13:04:23.325140 (+195998us) raft_consensus.cc:1645] Marking committed up to 0
0709 13:04:23.325184 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:23.325262 (+    78us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:23.388962 (+ 63700us) raft_consensus.cc:1685] finished
0709 13:04:23.388984 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:23.389116 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":124287,"raft.queue_time_us":118939,"raft.run_cpu_time_us":176,"raft.run_wall_time_us":11525,"thread_start_us":195588,"threads_started":1,"wal-append.queue_time_us":239016}
I20260709 13:04:23.402324 11029 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a154cc8b181f460abf4880064379948b" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:23.381155 12586 raft_consensus.cc:3060] T d371c5e27bd843469d2e2e453734cb4a P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:04:23.390347 12553 leader_election.cc:290] T d270b9e5481245d9b5340a454a7cd10b P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:23.407403 11029 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bac7e5040f984ba2b7146bd36af7267e" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:23.410259 11029 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75492f86fde046c3900df3eac8292fd4" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:23.363785 12394 raft_consensus.cc:515] T ff13ae9a47c7455f962406c6d131a3ff P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:23.382485 12390 raft_consensus.cc:2804] T 9a1716c0adf34bee8a4e99341ba5788f P 0172fed22b6b4890a7fdc953fa128af4 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:04:23.411957 11029 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1ec2b50c6ad4f7c8033476208abb26d" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:23.416034 11027 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d71ca3a1ba6b48478aaf62741bf3cf6a" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:23.427734 10880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acc241e7273d49149984968596242712" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:23.428591 10880 raft_consensus.cc:2468] T acc241e7273d49149984968596242712 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 2.
I20260709 13:04:23.365217 12572 raft_consensus.cc:515] T 5de5f0581e3b41dea729363d6ca563bb P 0172fed22b6b4890a7fdc953fa128af4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:23.432682 11027 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57b6a2418c1342e7924195e9c1dbd357" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:23.427477 10995 leader_election.cc:304] T 2567c2d84e4648d7ba5dc23a3df32817 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 873357742d5843579556ee8166f23f47; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda
I20260709 13:04:23.374516 12535 raft_consensus.cc:515] T 6a9353d1d230416296627d7cf8b1d425 P 186b12638a87410487467df721a3cbda [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:23.559330 11029 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb7d67be500947b3aef4c4245e33671c" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:23.410372 12527 leader_election.cc:290] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:23.562279 11027 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f371d2565dd84667b5613848604949bb" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:23.569950 10995 leader_election.cc:304] T 7709387f4d3d4f9992d97a0fbca39a40 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:04:23.575816 11021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7757ecbf402e47bfbc93d0447741e26c" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:23.422904 12390 raft_consensus.cc:493] T 9a1716c0adf34bee8a4e99341ba5788f P 0172fed22b6b4890a7fdc953fa128af4 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:23.631001 12390 raft_consensus.cc:3060] T 9a1716c0adf34bee8a4e99341ba5788f P 0172fed22b6b4890a7fdc953fa128af4 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:04:23.423906 12553 raft_consensus.cc:493] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 0172fed22b6b4890a7fdc953fa128af4 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:23.636920 10995 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:38967, user_credentials={real_user=slave}} blocked reactor thread for 67106.3us
I20260709 13:04:23.637225 11027 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4cd3d7dfbbf4096908ff11e9dc57141" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:23.638128 10995 leader_election.cc:304] T 1e4f2241a94a41f38e481137f868b83d P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:04:23.629226 11025 raft_consensus.cc:1240] T 401a9cf11eac479aa445ec8c1d6e33b2 P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Rejecting Update request from peer 186b12638a87410487467df721a3cbda for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260709 13:04:23.386078 12487 raft_consensus.cc:515] T 11de93cd9b9d457489ff1cb80f976c9a P 186b12638a87410487467df721a3cbda [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:23.677410 12487 leader_election.cc:290] T 11de93cd9b9d457489ff1cb80f976c9a P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:23.426735 11023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f46a5102fd64b45bf1c4dff863d2b4d" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:23.558344 10871 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555217d45d9b4ee492dbb3ddaf0f7c02" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:23.642797 12424 raft_consensus.cc:2804] T 7709387f4d3d4f9992d97a0fbca39a40 P 873357742d5843579556ee8166f23f47 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:04:23.678673 12424 raft_consensus.cc:493] T 7709387f4d3d4f9992d97a0fbca39a40 P 873357742d5843579556ee8166f23f47 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:23.679128 12424 raft_consensus.cc:3060] T 7709387f4d3d4f9992d97a0fbca39a40 P 873357742d5843579556ee8166f23f47 [term 7 FOLLOWER]: Advancing to term 8
W20260709 13:04:23.643249 12614 log.cc:927] Time spent T 32147c6d2d5a477aa38c597c7f78df5b P 0172fed22b6b4890a7fdc953fa128af4: Append to log took a long time: real 0.224s	user 0.000s	sys 0.000s
I20260709 13:04:23.682950 12394 leader_election.cc:290] T ff13ae9a47c7455f962406c6d131a3ff P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:23.645275 11021 raft_consensus.cc:1240] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 873357742d5843579556ee8166f23f47 [term 7 LEADER]: Rejecting Update request from peer 10ca5ee68ad64b3283b9d235352c9d23 for earlier term 6. Current term is 7. Ops: []
I20260709 13:04:23.685565 12572 leader_election.cc:290] T 5de5f0581e3b41dea729363d6ca563bb P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:23.662715 12298 raft_consensus.cc:2804] T 1e4f2241a94a41f38e481137f868b83d P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:04:23.686388 12298 raft_consensus.cc:493] T 1e4f2241a94a41f38e481137f868b83d P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 10ca5ee68ad64b3283b9d235352c9d23)
I20260709 13:04:23.686874 12298 raft_consensus.cc:3060] T 1e4f2241a94a41f38e481137f868b83d P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:04:23.685724 11099 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37666 (request call id 280) took 1628 ms. Trace:
I20260709 13:04:23.690829 11096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a977842ba4b744f4bb2febd55d286006" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:23.674307 12584 raft_consensus.cc:493] T e5ba72ccd854430baadbc636dfd17136 P 873357742d5843579556ee8166f23f47 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:23.692384 12584 raft_consensus.cc:515] T e5ba72ccd854430baadbc636dfd17136 P 873357742d5843579556ee8166f23f47 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:23.691708 11096 raft_consensus.cc:2468] T a977842ba4b744f4bb2febd55d286006 P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 4.
I20260709 13:04:23.693868 12535 leader_election.cc:290] T 6a9353d1d230416296627d7cf8b1d425 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:23.558871 12534 raft_consensus.cc:3060] T 2567c2d84e4648d7ba5dc23a3df32817 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:04:23.691119 11099 rpcz_store.cc:276] 0709 13:04:22.056966 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:23.076304 (+1019338us) service_pool.cc:224] Handling call
0709 13:04:23.076760 (+   456us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:23.127093 (+ 50333us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:23.127191 (+    98us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:23.127927 (+   736us) log.cc:844] Serialized 26 byte log entry
0709 13:04:23.318893 (+190966us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:04:23.344039 (+ 25146us) log.cc:844] Serialized 14 byte log entry
0709 13:04:23.350739 (+  6700us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:23.350876 (+   137us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:23.405573 (+ 54697us) spinlock_profiling.cc:243] Waited 3.65 ms on lock 0x7b180039d540. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f4263075522 00007f42630777c1 00007f4263077779 00007f4263077709 00007f42630776d1 00007f42630769cd 000000000038f354 000000000038f199 00007f426305883a
0709 13:04:23.685358 (+279785us) raft_consensus.cc:1685] finished
0709 13:04:23.685486 (+   128us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:23.685649 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":58008,"prepare.queue_time_us":18447,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":47,"raft.queue_time_us":54873,"raft.run_cpu_time_us":322,"raft.run_wall_time_us":2364,"spinlock_wait_cycles":8402304,"thread_start_us":190500,"threads_started":1,"wal-append.queue_time_us":244106}
I20260709 13:04:23.578697 12505 raft_consensus.cc:493] T 4f9eb06ed01f4ce68be3af8a8434665b P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 186b12638a87410487467df721a3cbda)
I20260709 13:04:23.699434 12505 raft_consensus.cc:515] T 4f9eb06ed01f4ce68be3af8a8434665b P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:23.430334 12450 consensus_queue.cc:1048] T f371d2565dd84667b5613848604949bb P 873357742d5843579556ee8166f23f47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.273s
I20260709 13:04:23.704636 12553 raft_consensus.cc:515] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 0172fed22b6b4890a7fdc953fa128af4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:23.715154 11099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acc241e7273d49149984968596242712" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:23.716074 11100 raft_consensus.cc:1275] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 0172fed22b6b4890a7fdc953fa128af4 [term 7 FOLLOWER]: Refusing update from remote peer 873357742d5843579556ee8166f23f47: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260709 13:04:23.720855 12424 raft_consensus.cc:515] T 7709387f4d3d4f9992d97a0fbca39a40 P 873357742d5843579556ee8166f23f47 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:23.723148 12424 leader_election.cc:290] T 7709387f4d3d4f9992d97a0fbca39a40 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 8 election: Requested vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:23.728199 11029 raft_consensus.cc:1275] T 401a9cf11eac479aa445ec8c1d6e33b2 P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Refusing update from remote peer 0172fed22b6b4890a7fdc953fa128af4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 13:04:23.729167 11099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555217d45d9b4ee492dbb3ddaf0f7c02" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:23.729918 11099 raft_consensus.cc:2468] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 0172fed22b6b4890a7fdc953fa128af4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 5.
W20260709 13:04:23.736279 10843 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:23.737496 10992 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:23.740990 10949 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49434 (request call id 375) took 1680 ms. Trace:
I20260709 13:04:23.743510 10949 rpcz_store.cc:276] 0709 13:04:22.060087 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:22.063257 (+  3170us) service_pool.cc:224] Handling call
0709 13:04:23.711397 (+1648140us) spinlock_profiling.cc:243] Waited 11.6 ms on lock 0x7b580001d198. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f426303db71 00007f426303da22 00007f426bf56c09 00007f426b0b0ee0 00007f426b17a0ab 00007f426b17a131 00007f426b13bfc0 00007f426b126929 00007f426b1263d2 00007f426469dbd4
0709 13:04:23.717341 (+  5944us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:23.731858 (+ 14517us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:23.731919 (+    61us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:23.732003 (+    84us) raft_consensus.cc:1645] Marking committed up to 3
0709 13:04:23.733237 (+  1234us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:23.734895 (+  1658us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:23.740894 (+  5999us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":26643840}
I20260709 13:04:23.746140 11028 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7709387f4d3d4f9992d97a0fbca39a40" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
I20260709 13:04:23.754423 11028 raft_consensus.cc:2368] T 7709387f4d3d4f9992d97a0fbca39a40 P 873357742d5843579556ee8166f23f47 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 0172fed22b6b4890a7fdc953fa128af4 for earlier term 7. Current term is 8.
I20260709 13:04:23.735770 11021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff13ae9a47c7455f962406c6d131a3ff" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:23.747200 10995 leader_election.cc:304] T acc241e7273d49149984968596242712 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:04:23.739892 11025 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46642 (request call id 203) took 2334 ms. Trace:
I20260709 13:04:23.733999 12586 raft_consensus.cc:515] T d371c5e27bd843469d2e2e453734cb4a P 186b12638a87410487467df721a3cbda [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
W20260709 13:04:23.761039 12537 consensus_meta.cc:287] T 868e92339c2f4508af0afbd07c814d64 P 186b12638a87410487467df721a3cbda: Time spent flushing consensus metadata: real 0.560s	user 0.012s	sys 0.003s
I20260709 13:04:23.770766 11103 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d439326f23747eaab3c1e7da2d6b785" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:23.773066 11067 leader_election.cc:400] T 7709387f4d3d4f9992d97a0fbca39a40 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 7 election: Vote denied by peer 873357742d5843579556ee8166f23f47 (127.0.107.195:45807) with higher term. Message: Invalid argument: T 7709387f4d3d4f9992d97a0fbca39a40 P 873357742d5843579556ee8166f23f47 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 0172fed22b6b4890a7fdc953fa128af4 for earlier term 7. Current term is 8.
I20260709 13:04:23.785172 11067 leader_election.cc:403] T 7709387f4d3d4f9992d97a0fbca39a40 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 7 election: Cancelling election due to peer responding with higher term
W20260709 13:04:23.787009 10843 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:23.760517 11025 rpcz_store.cc:276] 0709 13:04:21.398472 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:21.402232 (+  3760us) service_pool.cc:224] Handling call
0709 13:04:23.628107 (+2225875us) spinlock_profiling.cc:243] Waited 32.7 ms on lock 0x7b58007a2898. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f426303db71 00007f426303da22 00007f426bf56c09 00007f426b0b0ee0 00007f426b17a0ab 00007f426b17a131 00007f426b13bfc0 00007f426b126929 00007f426b1263d2 00007f426469dbd4
0709 13:04:23.628621 (+   514us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:23.715889 (+ 87268us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:23.732657 (+ 16768us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":75329792}
I20260709 13:04:23.746266 12387 raft_consensus.cc:515] T 8a69f32c53f24a50b7852d143665d7b4 P 0172fed22b6b4890a7fdc953fa128af4 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:23.782990 10877 raft_consensus.cc:2393] T d371c5e27bd843469d2e2e453734cb4a P 186b12638a87410487467df721a3cbda [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 873357742d5843579556ee8166f23f47 in current term 5: Already voted for candidate 186b12638a87410487467df721a3cbda in this term.
I20260709 13:04:23.794827 11098 raft_consensus.cc:1240] T 87cd8f5097624c8692f1d83544d8881e P 0172fed22b6b4890a7fdc953fa128af4 [term 6 FOLLOWER]: Rejecting Update request from peer 186b12638a87410487467df721a3cbda for earlier term 5. Current term is 6. Ops: []
W20260709 13:04:23.773715 11020 consensus_meta.cc:287] T 732c4adeeff6422497aa7db4a82f535b P 873357742d5843579556ee8166f23f47: Time spent flushing consensus metadata: real 0.670s	user 0.014s	sys 0.004s
I20260709 13:04:23.782197 12586 leader_election.cc:290] T d371c5e27bd843469d2e2e453734cb4a P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 5 election: Requested vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:23.801970 11103 raft_consensus.cc:2410] T 5d439326f23747eaab3c1e7da2d6b785 P 0172fed22b6b4890a7fdc953fa128af4 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 186b12638a87410487467df721a3cbda for term 6 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 13:04:23.791083 12611 raft_consensus.cc:493] T f34f2bc02d85444f9096e0f5b4fe4331 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:23.793857 12394 raft_consensus.cc:3060] T 7709387f4d3d4f9992d97a0fbca39a40 P 0172fed22b6b4890a7fdc953fa128af4 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:04:23.812436 11098 raft_consensus.cc:1240] T 8a69f32c53f24a50b7852d143665d7b4 P 0172fed22b6b4890a7fdc953fa128af4 [term 5 FOLLOWER]: Rejecting Update request from peer 186b12638a87410487467df721a3cbda for earlier term 4. Current term is 5. Ops: []
I20260709 13:04:23.816285 11028 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f9eb06ed01f4ce68be3af8a8434665b" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:23.816995 11028 raft_consensus.cc:2468] T 4f9eb06ed01f4ce68be3af8a8434665b P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 4.
I20260709 13:04:23.818783 11103 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7757ecbf402e47bfbc93d0447741e26c" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:23.784469 12537 raft_consensus.cc:515] T 868e92339c2f4508af0afbd07c814d64 P 186b12638a87410487467df721a3cbda [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:23.800450 11020 raft_consensus.cc:2468] T 732c4adeeff6422497aa7db4a82f535b P 873357742d5843579556ee8166f23f47 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 5.
I20260709 13:04:23.806624 12505 leader_election.cc:290] T 4f9eb06ed01f4ce68be3af8a8434665b P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:23.827422 11102 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "868e92339c2f4508af0afbd07c814d64" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:23.828241 11102 raft_consensus.cc:2468] T 868e92339c2f4508af0afbd07c814d64 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 3.
I20260709 13:04:23.818626 12611 raft_consensus.cc:515] T f34f2bc02d85444f9096e0f5b4fe4331 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:23.831533 11103 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f34f2bc02d85444f9096e0f5b4fe4331" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:04:23.832507 11103 raft_consensus.cc:2393] T f34f2bc02d85444f9096e0f5b4fe4331 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 186b12638a87410487467df721a3cbda in current term 2: Already voted for candidate 0172fed22b6b4890a7fdc953fa128af4 in this term.
I20260709 13:04:23.833563 10879 raft_consensus.cc:2393] T 868e92339c2f4508af0afbd07c814d64 P 186b12638a87410487467df721a3cbda [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 873357742d5843579556ee8166f23f47 in current term 5: Already voted for candidate 186b12638a87410487467df721a3cbda in this term.
W20260709 13:04:23.835229 12622 log.cc:927] Time spent T 30f628e3d7914d609e7f1d980a74c654 P 186b12638a87410487467df721a3cbda: Append to log took a long time: real 0.063s	user 0.000s	sys 0.001s
I20260709 13:04:23.836941 10953 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49428 (request call id 261) took 1429 ms. Trace:
I20260709 13:04:23.838320 12558 raft_consensus.cc:493] T 4c2a35bb99374e93b573cf870d04faaa P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:23.837338 10953 rpcz_store.cc:276] 0709 13:04:22.407568 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:22.407814 (+   246us) service_pool.cc:224] Handling call
0709 13:04:22.423790 (+ 15976us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:22.424616 (+   826us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:22.424682 (+    66us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:22.425751 (+  1069us) log.cc:844] Serialized 26 byte log entry
0709 13:04:22.588270 (+162519us) raft_consensus.cc:1645] Marking committed up to 3
0709 13:04:22.692402 (+104132us) log.cc:844] Serialized 14 byte log entry
0709 13:04:23.836586 (+1144184us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:23.836672 (+    86us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:23.836721 (+    49us) raft_consensus.cc:1685] finished
0709 13:04:23.836737 (+    16us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:23.836870 (+   133us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1147199,"prepare.queue_time_us":47488,"prepare.run_cpu_time_us":139,"prepare.run_wall_time_us":146,"raft.queue_time_us":1144277,"raft.run_cpu_time_us":267,"raft.run_wall_time_us":270,"thread_start_us":125252,"threads_started":1,"wal-append.queue_time_us":338134}
I20260709 13:04:23.838913 12558 raft_consensus.cc:515] T 4c2a35bb99374e93b573cf870d04faaa P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:23.840972 12500 consensus_queue.cc:1059] T d7ce47b9a92d43dd88d95bcda5038597 P 186b12638a87410487467df721a3cbda [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:04:23.841325 12558 leader_election.cc:290] T 4c2a35bb99374e93b573cf870d04faaa P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:23.841940 12558 raft_consensus.cc:493] T bb7d67be500947b3aef4c4245e33671c P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:23.833038 12450 raft_consensus.cc:2804] T acc241e7273d49149984968596242712 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:04:23.842911 12578 raft_consensus.cc:493] T 2567c2d84e4648d7ba5dc23a3df32817 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:23.843588 10949 raft_consensus.cc:1217] T e5a5f5a608504650b48fd27f163faed6 P 10ca5ee68ad64b3283b9d235352c9d23 [term 6 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[2.2-5.3]   Dedup: 2.2->[5.3-5.3]
I20260709 13:04:23.843436 12578 raft_consensus.cc:515] T 2567c2d84e4648d7ba5dc23a3df32817 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:23.844122 10949 raft_consensus.cc:1240] T e5a5f5a608504650b48fd27f163faed6 P 10ca5ee68ad64b3283b9d235352c9d23 [term 6 FOLLOWER]: Rejecting Update request from peer 186b12638a87410487467df721a3cbda for earlier term 5. Current term is 6. Ops: [2.2-5.3]
I20260709 13:04:23.845252 10949 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72b111ad7b5f4753ab1d2b5ee1114840" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:04:23.845757 12583 raft_consensus.cc:493] T a5299b8d044f41b7b68057e63d17b159 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:23.846657 10951 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "732c4adeeff6422497aa7db4a82f535b" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:04:23.846843 12557 raft_consensus.cc:493] T e96e80ba9ac34d0998395515e5c8c41a P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:23.847446 12557 raft_consensus.cc:515] T e96e80ba9ac34d0998395515e5c8c41a P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:23.848038 10951 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f34f2bc02d85444f9096e0f5b4fe4331" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:04:23.848500 12581 raft_consensus.cc:493] T f34f2bc02d85444f9096e0f5b4fe4331 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:23.849016 12581 raft_consensus.cc:515] T f34f2bc02d85444f9096e0f5b4fe4331 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:23.849195 12298 raft_consensus.cc:515] T 1e4f2241a94a41f38e481137f868b83d P 873357742d5843579556ee8166f23f47 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:23.849802 12557 leader_election.cc:290] T e96e80ba9ac34d0998395515e5c8c41a P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:23.850417 11027 raft_consensus.cc:1240] T 1e4f2241a94a41f38e481137f868b83d P 873357742d5843579556ee8166f23f47 [term 5 FOLLOWER]: Rejecting Update request from peer 10ca5ee68ad64b3283b9d235352c9d23 for earlier term 4. Current term is 5. Ops: []
I20260709 13:04:23.851260 12581 leader_election.cc:290] T f34f2bc02d85444f9096e0f5b4fe4331 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:23.852499 10952 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49434 (request call id 388) took 1104 ms. Trace:
I20260709 13:04:23.853037 10953 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c2a35bb99374e93b573cf870d04faaa" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:23.849056 10946 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e4f2241a94a41f38e481137f868b83d" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:23.857884 10920 leader_election.cc:304] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23; no voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda
I20260709 13:04:23.845999 10949 raft_consensus.cc:2393] T 72b111ad7b5f4753ab1d2b5ee1114840 P 10ca5ee68ad64b3283b9d235352c9d23 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 186b12638a87410487467df721a3cbda in current term 5: Already voted for candidate 10ca5ee68ad64b3283b9d235352c9d23 in this term.
I20260709 13:04:23.852856 10952 rpcz_store.cc:276] 0709 13:04:22.748934 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:23.053901 (+304967us) service_pool.cc:224] Handling call
0709 13:04:23.054215 (+   314us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:23.054429 (+   214us) raft_consensus.cc:1534] Early marking committed up to index 3
0709 13:04:23.054809 (+   380us) log.cc:844] Serialized 14 byte log entry
0709 13:04:23.852156 (+797347us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:23.852217 (+    61us) raft_consensus.cc:1645] Marking committed up to 3
0709 13:04:23.852264 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:23.852328 (+    64us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:23.852438 (+   110us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":763030,"prepare.queue_time_us":3945,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":44,"raft.queue_time_us":763538,"raft.run_cpu_time_us":84,"raft.run_wall_time_us":84,"thread_start_us":33902,"threads_started":1,"wal-append.queue_time_us":127418}
I20260709 13:04:23.875451 11103 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72b111ad7b5f4753ab1d2b5ee1114840" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:04:23.874826 11027 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f1466124dbd4f6ebb3c4d62ca48bf96" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:23.865918 10949 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11de93cd9b9d457489ff1cb80f976c9a" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:23.884579 11103 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72b111ad7b5f4753ab1d2b5ee1114840" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:04:23.842038 12500 raft_consensus.cc:3055] T d7ce47b9a92d43dd88d95bcda5038597 P 186b12638a87410487467df721a3cbda [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:04:23.826927 12586 raft_consensus.cc:493] T 0e5d327630e5438b878035d28efe6c0c P 186b12638a87410487467df721a3cbda [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 0172fed22b6b4890a7fdc953fa128af4)
I20260709 13:04:23.892172 10946 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e89e640169e14e2d98f968f76027daa7" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:23.863363 11026 raft_consensus.cc:1240] T 732c4adeeff6422497aa7db4a82f535b P 873357742d5843579556ee8166f23f47 [term 5 FOLLOWER]: Rejecting Update request from peer 10ca5ee68ad64b3283b9d235352c9d23 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260709 13:04:23.846671 12544 raft_consensus.cc:493] T 7839ca2d2c13431a85f44d7c458c6351 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:23.850338 12557 raft_consensus.cc:493] T 11de93cd9b9d457489ff1cb80f976c9a P 10ca5ee68ad64b3283b9d235352c9d23 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:23.896505 10953 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5299b8d044f41b7b68057e63d17b159" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:23.897266 10920 leader_election.cc:304] T a64b7e99d6c54934a7de6f6dffcf5e94 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 10ca5ee68ad64b3283b9d235352c9d23; no voters: 
I20260709 13:04:23.846705 12387 leader_election.cc:290] T 8a69f32c53f24a50b7852d143665d7b4 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 5 election: Requested vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:23.870889 12591 raft_consensus.cc:493] T 30f628e3d7914d609e7f1d980a74c654 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:23.868938 12553 leader_election.cc:290] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:23.871548 12298 leader_election.cc:290] T 1e4f2241a94a41f38e481137f868b83d P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 5 election: Requested vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:23.869454 11096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acc241e7273d49149984968596242712" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:23.837352 12611 leader_election.cc:290] T f34f2bc02d85444f9096e0f5b4fe4331 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:23.888971 12500 raft_consensus.cc:740] T d7ce47b9a92d43dd88d95bcda5038597 P 186b12638a87410487467df721a3cbda [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 186b12638a87410487467df721a3cbda, State: Running, Role: LEADER
I20260709 13:04:23.914732 10951 raft_consensus.cc:2468] T f34f2bc02d85444f9096e0f5b4fe4331 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 2.
I20260709 13:04:23.846411 12583 raft_consensus.cc:515] T a5299b8d044f41b7b68057e63d17b159 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:23.887902 12578 leader_election.cc:290] T 2567c2d84e4648d7ba5dc23a3df32817 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:23.932060 10920 leader_election.cc:304] T 6f46a5102fd64b45bf1c4dff863d2b4d P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 10ca5ee68ad64b3283b9d235352c9d23; no voters: 
I20260709 13:04:23.857379 12558 raft_consensus.cc:515] T bb7d67be500947b3aef4c4245e33671c P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:23.942891 11094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a610d1a71f52432e88fe56130f921fa2" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:23.949942 11094 raft_consensus.cc:2468] T a610d1a71f52432e88fe56130f921fa2 P 0172fed22b6b4890a7fdc953fa128af4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 5.
I20260709 13:04:23.943912 11103 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f371d2565dd84667b5613848604949bb" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:23.919275 12517 raft_consensus.cc:2749] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:04:23.953017 11103 raft_consensus.cc:2410] T f371d2565dd84667b5613848604949bb P 0172fed22b6b4890a7fdc953fa128af4 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 186b12638a87410487467df721a3cbda for term 6 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 13:04:23.956020 10919 leader_election.cc:304] T 75492f86fde046c3900df3eac8292fd4 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23; no voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47
I20260709 13:04:23.956553 10953 raft_consensus.cc:2468] T a5299b8d044f41b7b68057e63d17b159 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 3.
I20260709 13:04:23.901566 12586 raft_consensus.cc:515] T 0e5d327630e5438b878035d28efe6c0c P 186b12638a87410487467df721a3cbda [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:23.969594 10920 leader_election.cc:304] T b4cd3d7dfbbf4096908ff11e9dc57141 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 10ca5ee68ad64b3283b9d235352c9d23; no voters: 
I20260709 13:04:23.929477 12591 raft_consensus.cc:515] T 30f628e3d7914d609e7f1d980a74c654 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:23.972127 12587 raft_consensus.cc:2804] T 6f46a5102fd64b45bf1c4dff863d2b4d P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:04:23.972664 12587 raft_consensus.cc:493] T 6f46a5102fd64b45bf1c4dff863d2b4d P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:23.973078 12587 raft_consensus.cc:3060] T 6f46a5102fd64b45bf1c4dff863d2b4d P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:04:23.959187 12578 raft_consensus.cc:493] T e5ba72ccd854430baadbc636dfd17136 P 10ca5ee68ad64b3283b9d235352c9d23 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:23.975414 12578 raft_consensus.cc:515] T e5ba72ccd854430baadbc636dfd17136 P 10ca5ee68ad64b3283b9d235352c9d23 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:23.977876 12578 leader_election.cc:290] T e5ba72ccd854430baadbc636dfd17136 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:23.974200 12591 leader_election.cc:290] T 30f628e3d7914d609e7f1d980a74c654 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:23.978744 12582 consensus_queue.cc:1059] T 732c4adeeff6422497aa7db4a82f535b P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:04:23.979431 12578 raft_consensus.cc:2804] T b4cd3d7dfbbf4096908ff11e9dc57141 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:04:23.979825 12582 consensus_queue.cc:1243] T 732c4adeeff6422497aa7db4a82f535b P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Peer 186b12638a87410487467df721a3cbda log is divergent from this leader: its last log entry 3.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260709 13:04:23.980149 12578 raft_consensus.cc:493] T b4cd3d7dfbbf4096908ff11e9dc57141 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:23.980729 12582 raft_consensus.cc:3055] T 732c4adeeff6422497aa7db4a82f535b P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:04:23.981220 12582 raft_consensus.cc:740] T 732c4adeeff6422497aa7db4a82f535b P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 10ca5ee68ad64b3283b9d235352c9d23, State: Running, Role: LEADER
I20260709 13:04:23.982640 10946 raft_consensus.cc:1275] T 30f628e3d7914d609e7f1d980a74c654 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Refusing update from remote peer 186b12638a87410487467df721a3cbda: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
    @     0x7f4263284d29 testing::TestInfo::Run() at ??:0
I20260709 13:04:23.985777 10919 leader_election.cc:304] T a154cc8b181f460abf4880064379948b P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23; no voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47
I20260709 13:04:23.987179 12488 raft_consensus.cc:493] T c31881534c5e4f0089c82057c1746652 P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 10ca5ee68ad64b3283b9d235352c9d23)
I20260709 13:04:23.988765 10954 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49434 (request call id 382) took 1513 ms. Trace:
I20260709 13:04:23.988906 12488 raft_consensus.cc:515] T c31881534c5e4f0089c82057c1746652 P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:23.989995 12564 raft_consensus.cc:2749] T a154cc8b181f460abf4880064379948b P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:04:23.993309 12488 leader_election.cc:290] T c31881534c5e4f0089c82057c1746652 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:23.987509 12582 consensus_queue.cc:260] T 732c4adeeff6422497aa7db4a82f535b P 10ca5ee68ad64b3283b9d235352c9d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:23.989187 10954 rpcz_store.cc:276] 0709 13:04:22.475814 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:22.989709 (+513895us) service_pool.cc:224] Handling call
0709 13:04:22.990120 (+   411us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:23.847265 (+857145us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:23.847302 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:23.847857 (+   555us) log.cc:844] Serialized 26 byte log entry
0709 13:04:23.897815 (+ 49958us) raft_consensus.cc:1645] Marking committed up to 3
0709 13:04:23.897864 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:23.897946 (+    82us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:23.988485 (+ 90539us) raft_consensus.cc:1685] finished
0709 13:04:23.988512 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:23.988694 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":856605,"raft.queue_time_us":861210,"raft.run_cpu_time_us":92,"raft.run_wall_time_us":92,"thread_start_us":47920,"threads_started":1,"wal-append.queue_time_us":106901}
I20260709 13:04:23.994565 12582 raft_consensus.cc:3060] T 732c4adeeff6422497aa7db4a82f535b P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:04:23.995682 10954 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e4f2241a94a41f38e481137f868b83d" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:23.997247 10954 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57b6a2418c1342e7924195e9c1dbd357" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:23.998150 10948 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49428 (request call id 264) took 1582 ms. Trace:
I20260709 13:04:23.998775 12566 consensus_queue.cc:1059] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 10ca5ee68ad64b3283b9d235352c9d23 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 13:04:23.999924 12566 consensus_queue.cc:1243] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 10ca5ee68ad64b3283b9d235352c9d23 [NON_LEADER]: Peer 873357742d5843579556ee8166f23f47 log is divergent from this leader: its last log entry 7.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20260709 13:04:23.998668 10948 rpcz_store.cc:276] 0709 13:04:22.415372 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:22.676074 (+260702us) service_pool.cc:224] Handling call
0709 13:04:22.677390 (+  1316us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:23.835500 (+1158110us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:23.835543 (+    43us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:23.836306 (+   763us) log.cc:844] Serialized 26 byte log entry
0709 13:04:23.885138 (+ 48832us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:04:23.924011 (+ 38873us) log.cc:844] Serialized 14 byte log entry
0709 13:04:23.960903 (+ 36892us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:23.997798 (+ 36895us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:23.997868 (+    70us) raft_consensus.cc:1685] finished
0709 13:04:23.997887 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:23.998074 (+   187us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1190280,"raft.queue_time_us":1194837,"raft.run_cpu_time_us":216,"raft.run_wall_time_us":216,"spinlock_wait_cycles":60160,"thread_start_us":64377,"threads_started":2,"wal-append.queue_time_us":48789}
I20260709 13:04:24.001255 10948 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a977842ba4b744f4bb2febd55d286006" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:24.003964 11023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e96e80ba9ac34d0998395515e5c8c41a" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:24.004467 11024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "30f628e3d7914d609e7f1d980a74c654" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:24.005424 11024 raft_consensus.cc:2468] T 30f628e3d7914d609e7f1d980a74c654 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 2.
I20260709 13:04:24.011214 12534 raft_consensus.cc:2749] T 2567c2d84e4648d7ba5dc23a3df32817 P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:04:23.957875 12471 raft_consensus.cc:2804] T a64b7e99d6c54934a7de6f6dffcf5e94 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:04:24.012585 12471 raft_consensus.cc:493] T a64b7e99d6c54934a7de6f6dffcf5e94 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:24.013176 12471 raft_consensus.cc:3060] T a64b7e99d6c54934a7de6f6dffcf5e94 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:04:23.958953 12526 consensus_queue.cc:1048] T 1e4f2241a94a41f38e481137f868b83d P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.059s
I20260709 13:04:23.918066 12544 raft_consensus.cc:515] T 7839ca2d2c13431a85f44d7c458c6351 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:23.989554 11103 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d371c5e27bd843469d2e2e453734cb4a" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
W20260709 13:04:24.018390 10844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:23.925710 12557 raft_consensus.cc:515] T 11de93cd9b9d457489ff1cb80f976c9a P 10ca5ee68ad64b3283b9d235352c9d23 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
W20260709 13:04:24.023967 10920 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 54498.2us
W20260709 13:04:24.043567 10994 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 135769us
I20260709 13:04:24.044230 11094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d371c5e27bd843469d2e2e453734cb4a" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:04:24.028996 10919 leader_election.cc:304] T 30f628e3d7914d609e7f1d980a74c654 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:04:24.033411 10992 leader_election.cc:304] T 57b6a2418c1342e7924195e9c1dbd357 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 873357742d5843579556ee8166f23f47; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda
I20260709 13:04:24.038018 10879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57b6a2418c1342e7924195e9c1dbd357" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:23.957679 11022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c2a35bb99374e93b573cf870d04faaa" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:24.023037 10954 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e4f2241a94a41f38e481137f868b83d" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
W20260709 13:04:24.016449 12627 log.cc:927] Time spent T b67eff0634e540eab91cf8cee795969f P 10ca5ee68ad64b3283b9d235352c9d23: Append to log took a long time: real 0.071s	user 0.001s	sys 0.000s
I20260709 13:04:24.041548 10873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2567c2d84e4648d7ba5dc23a3df32817" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:23.952790 12525 raft_consensus.cc:493] T ff13ae9a47c7455f962406c6d131a3ff P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:24.040007 10872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72b111ad7b5f4753ab1d2b5ee1114840" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:04:24.049602 10879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e96e80ba9ac34d0998395515e5c8c41a" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:24.040596 10871 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c2a35bb99374e93b573cf870d04faaa" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:24.051519 10921 leader_election.cc:304] T 57b6a2418c1342e7924195e9c1dbd357 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23; no voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47
I20260709 13:04:24.056605 11094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c31881534c5e4f0089c82057c1746652" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:23.980746 12578 raft_consensus.cc:3060] T b4cd3d7dfbbf4096908ff11e9dc57141 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:04:24.060417 10920 leader_election.cc:304] T d71ca3a1ba6b48478aaf62741bf3cf6a P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 10ca5ee68ad64b3283b9d235352c9d23; no voters: 873357742d5843579556ee8166f23f47
I20260709 13:04:24.073068 10873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f34f2bc02d85444f9096e0f5b4fe4331" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:24.073671 10949 raft_consensus.cc:2468] T 11de93cd9b9d457489ff1cb80f976c9a P 10ca5ee68ad64b3283b9d235352c9d23 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 5.
I20260709 13:04:24.071085 10879 raft_consensus.cc:2468] T e96e80ba9ac34d0998395515e5c8c41a P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 3.
I20260709 13:04:24.084786 10879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5299b8d044f41b7b68057e63d17b159" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:24.084856 10921 leader_election.cc:304] T e96e80ba9ac34d0998395515e5c8c41a P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda; no voters: 873357742d5843579556ee8166f23f47
I20260709 13:04:24.077075 12583 leader_election.cc:290] T a5299b8d044f41b7b68057e63d17b159 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:24.077597 11096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a9353d1d230416296627d7cf8b1d425" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:24.071849 12525 raft_consensus.cc:515] T ff13ae9a47c7455f962406c6d131a3ff P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:24.079593 10872 raft_consensus.cc:2393] T 72b111ad7b5f4753ab1d2b5ee1114840 P 186b12638a87410487467df721a3cbda [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 10ca5ee68ad64b3283b9d235352c9d23 in current term 5: Already voted for candidate 186b12638a87410487467df721a3cbda in this term.
I20260709 13:04:24.055670 12537 leader_election.cc:290] T 868e92339c2f4508af0afbd07c814d64 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 5 election: Requested vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:24.061932 12202 raft_consensus.cc:2749] T 57b6a2418c1342e7924195e9c1dbd357 P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:04:24.088133 10879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "30f628e3d7914d609e7f1d980a74c654" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:24.176508 10953 raft_consensus.cc:1240] T e5ba72ccd854430baadbc636dfd17136 P 10ca5ee68ad64b3283b9d235352c9d23 [term 6 FOLLOWER]: Rejecting Update request from peer 873357742d5843579556ee8166f23f47 for earlier term 5. Current term is 6. Ops: []
W20260709 13:04:24.150791 10921 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:38967, user_credentials={real_user=slave}} blocked reactor thread for 66058.2us
W20260709 13:04:24.137766 10920 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 77622.2us
I20260709 13:04:24.035202 12566 consensus_queue.cc:1059] T 1e4f2241a94a41f38e481137f868b83d P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.047s
I20260709 13:04:24.153277 11103 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "868e92339c2f4508af0afbd07c814d64" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:04:24.183072 11103 raft_consensus.cc:3060] T 868e92339c2f4508af0afbd07c814d64 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Advancing to term 5
I20260709 13:04:24.183089 10920 leader_election.cc:304] T 836251ebc2b74f5da88b8a8b2338e2a5 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23; no voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda
I20260709 13:04:24.154060 12545 raft_consensus.cc:2804] T d71ca3a1ba6b48478aaf62741bf3cf6a P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:04:24.155406 12526 raft_consensus.cc:2804] T e96e80ba9ac34d0998395515e5c8c41a P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:04:24.156550 12517 raft_consensus.cc:2804] T 30f628e3d7914d609e7f1d980a74c654 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:04:24.157172 11028 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a9353d1d230416296627d7cf8b1d425" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:24.161501 11022 raft_consensus.cc:1275] T 2567c2d84e4648d7ba5dc23a3df32817 P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Refusing update from remote peer 186b12638a87410487467df721a3cbda: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:04:24.088692 12564 consensus_queue.cc:1048] T 5d439326f23747eaab3c1e7da2d6b785 P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.082s
I20260709 13:04:24.165014 12585 raft_consensus.cc:493] T 63d44f6ba6434ab2870fe3190f4d7c2f P 186b12638a87410487467df721a3cbda [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 10ca5ee68ad64b3283b9d235352c9d23)
I20260709 13:04:24.165386 12390 raft_consensus.cc:515] T 9a1716c0adf34bee8a4e99341ba5788f P 0172fed22b6b4890a7fdc953fa128af4 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:24.166994 10947 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49434 (request call id 384) took 1685 ms. Trace:
I20260709 13:04:24.087141 10948 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "732c4adeeff6422497aa7db4a82f535b" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:24.199329 11097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d371c5e27bd843469d2e2e453734cb4a" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:24.170699 12586 leader_election.cc:290] T 0e5d327630e5438b878035d28efe6c0c P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:24.171590 12525 leader_election.cc:290] T ff13ae9a47c7455f962406c6d131a3ff P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:24.172384 12544 leader_election.cc:290] T 7839ca2d2c13431a85f44d7c458c6351 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:24.172871 12555 raft_consensus.cc:493] T 5de5f0581e3b41dea729363d6ca563bb P 10ca5ee68ad64b3283b9d235352c9d23 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 186b12638a87410487467df721a3cbda)
I20260709 13:04:24.173938 12450 raft_consensus.cc:493] T acc241e7273d49149984968596242712 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 0172fed22b6b4890a7fdc953fa128af4)
I20260709 13:04:24.174880 12557 leader_election.cc:290] T 11de93cd9b9d457489ff1cb80f976c9a P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:24.175648 11023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11de93cd9b9d457489ff1cb80f976c9a" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
W20260709 13:04:24.150553 10919 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:45807, user_credentials={real_user=slave}} blocked reactor thread for 121648us
W20260709 13:04:24.038300 10832 catalog_manager.cc:6160] T 00000000000000000000000000000000 P c669bee8ad644d0d956c148b33886b51: Tablet ff13ae9a47c7455f962406c6d131a3ff (table test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]) was not created within the allowed timeout. Replacing with a new tablet 2725460e311143fcbdb427a04c064f5a
I20260709 13:04:24.083537 10873 raft_consensus.cc:2468] T f34f2bc02d85444f9096e0f5b4fe4331 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 2.
I20260709 13:04:24.182296 10879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7839ca2d2c13431a85f44d7c458c6351" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
W20260709 13:04:24.220921 10918 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:24.095739 11069 leader_election.cc:304] T 836251ebc2b74f5da88b8a8b2338e2a5 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda; no voters: 
I20260709 13:04:24.061916 12591 raft_consensus.cc:2749] T 75492f86fde046c3900df3eac8292fd4 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:04:24.062350 11020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5ba72ccd854430baadbc636dfd17136" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:24.202905 11028 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "868e92339c2f4508af0afbd07c814d64" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
W20260709 13:04:24.224224 11069 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:38967, user_credentials={real_user=slave}} blocked reactor thread for 128644us
I20260709 13:04:24.225787 11028 raft_consensus.cc:3060] T 868e92339c2f4508af0afbd07c814d64 P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:04:24.199846 10947 rpcz_store.cc:276] 0709 13:04:22.481420 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:23.010413 (+528993us) service_pool.cc:224] Handling call
0709 13:04:23.024092 (+ 13679us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:23.873849 (+849757us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:23.873884 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:23.874328 (+   444us) log.cc:844] Serialized 26 byte log entry
0709 13:04:24.061069 (+186741us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:04:24.061121 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:24.061200 (+    79us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:24.166627 (+105427us) raft_consensus.cc:1685] finished
0709 13:04:24.166662 (+    35us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:24.166902 (+   240us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":876362,"raft.queue_time_us":856714,"raft.run_cpu_time_us":171,"raft.run_wall_time_us":4404,"thread_start_us":130019,"threads_started":1,"wal-append.queue_time_us":246647}
I20260709 13:04:24.172438 12584 leader_election.cc:290] T e5ba72ccd854430baadbc636dfd17136 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:24.088363 12582 raft_consensus.cc:493] T d270b9e5481245d9b5340a454a7cd10b P 10ca5ee68ad64b3283b9d235352c9d23 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 186b12638a87410487467df721a3cbda)
I20260709 13:04:24.080508 12518 raft_consensus.cc:2749] T 57b6a2418c1342e7924195e9c1dbd357 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:04:24.185915 12537 raft_consensus.cc:493] T 836251ebc2b74f5da88b8a8b2338e2a5 P 186b12638a87410487467df721a3cbda [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:24.185873 12583 raft_consensus.cc:2749] T 836251ebc2b74f5da88b8a8b2338e2a5 P 10ca5ee68ad64b3283b9d235352c9d23 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 13:04:24.186132 12545 raft_consensus.cc:493] T d71ca3a1ba6b48478aaf62741bf3cf6a P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:24.194723 12517 raft_consensus.cc:493] T 30f628e3d7914d609e7f1d980a74c654 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:24.192390 12526 raft_consensus.cc:493] T e96e80ba9ac34d0998395515e5c8c41a P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:24.190722 12532 raft_consensus.cc:3055] T 1e4f2241a94a41f38e481137f868b83d P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:04:24.219036 11099 raft_consensus.cc:1240] T 9a1716c0adf34bee8a4e99341ba5788f P 0172fed22b6b4890a7fdc953fa128af4 [term 6 FOLLOWER]: Rejecting Update request from peer 186b12638a87410487467df721a3cbda for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20260709 13:04:24.198804 10872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11de93cd9b9d457489ff1cb80f976c9a" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:24.323690 12524 raft_consensus.cc:3060] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 10ca5ee68ad64b3283b9d235352c9d23 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:04:24.324113 10872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a69f32c53f24a50b7852d143665d7b4" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
W20260709 13:04:24.326985 12634 log.cc:927] Time spent T acc241e7273d49149984968596242712 P 0172fed22b6b4890a7fdc953fa128af4: Append to log took a long time: real 0.112s	user 0.000s	sys 0.000s
I20260709 13:04:24.318624 12526 raft_consensus.cc:3060] T e96e80ba9ac34d0998395515e5c8c41a P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:04:24.330971 12390 leader_election.cc:290] T 9a1716c0adf34bee8a4e99341ba5788f P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 6 election: Requested vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:24.218837 12555 raft_consensus.cc:515] T 5de5f0581e3b41dea729363d6ca563bb P 10ca5ee68ad64b3283b9d235352c9d23 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:24.280504 12570 consensus_queue.cc:1059] T 72b111ad7b5f4753ab1d2b5ee1114840 P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.052s
I20260709 13:04:24.333500 12570 raft_consensus.cc:3055] T 72b111ad7b5f4753ab1d2b5ee1114840 P 0172fed22b6b4890a7fdc953fa128af4 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:04:24.333892 12570 raft_consensus.cc:740] T 72b111ad7b5f4753ab1d2b5ee1114840 P 0172fed22b6b4890a7fdc953fa128af4 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 0172fed22b6b4890a7fdc953fa128af4, State: Running, Role: LEADER
I20260709 13:04:24.184947 10920 leader_election.cc:304] T d7ce47b9a92d43dd88d95bcda5038597 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 10ca5ee68ad64b3283b9d235352c9d23; no voters: 186b12638a87410487467df721a3cbda
I20260709 13:04:24.334854 12570 consensus_queue.cc:260] T 72b111ad7b5f4753ab1d2b5ee1114840 P 0172fed22b6b4890a7fdc953fa128af4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
W20260709 13:04:24.336232 10920 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 151339us
I20260709 13:04:24.207260 12326 raft_consensus.cc:493] T 401a9cf11eac479aa445ec8c1d6e33b2 P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:24.337121 12326 raft_consensus.cc:515] T 401a9cf11eac479aa445ec8c1d6e33b2 P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:24.338871 10955 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49428 (request call id 267) took 1922 ms. Trace:
I20260709 13:04:24.188637 12575 consensus_queue.cc:1048] T 525bfdbd6d934afd823f7d3ee3f9d052 P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.026s
I20260709 13:04:24.339509 12326 leader_election.cc:290] T 401a9cf11eac479aa445ec8c1d6e33b2 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:24.340698 10874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a1716c0adf34bee8a4e99341ba5788f" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:24.341022 12575 consensus_queue.cc:1048] T a977842ba4b744f4bb2febd55d286006 P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:04:24.278815 10873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c160ddfe9b34bdaa3835d0915c1e06f" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:24.343364 10873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c51782cd9d064ec0949e84e24dd4de67" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:04:24.344475 10873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d439326f23747eaab3c1e7da2d6b785" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:24.345319 10873 raft_consensus.cc:2410] T 5d439326f23747eaab3c1e7da2d6b785 P 186b12638a87410487467df721a3cbda [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0172fed22b6b4890a7fdc953fa128af4 for term 6 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260709 13:04:24.346065 10873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f371d2565dd84667b5613848604949bb" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
W20260709 13:04:24.279282 10845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 47451.1us
I20260709 13:04:24.347066 10876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7709387f4d3d4f9992d97a0fbca39a40" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:04:24.347901 10876 raft_consensus.cc:3060] T 7709387f4d3d4f9992d97a0fbca39a40 P 186b12638a87410487467df721a3cbda [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:04:24.317359 12517 raft_consensus.cc:3060] T 30f628e3d7914d609e7f1d980a74c654 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:04:24.218995 12450 raft_consensus.cc:3060] T acc241e7273d49149984968596242712 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Advancing to term 3
W20260709 13:04:24.351586 11068 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:24.282347 11020 raft_consensus.cc:2468] T e5ba72ccd854430baadbc636dfd17136 P 873357742d5843579556ee8166f23f47 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 6.
I20260709 13:04:24.288800 12434 consensus_queue.cc:1048] T f371d2565dd84667b5613848604949bb P 873357742d5843579556ee8166f23f47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.065s
I20260709 13:04:24.288554 12537 raft_consensus.cc:515] T 836251ebc2b74f5da88b8a8b2338e2a5 P 186b12638a87410487467df721a3cbda [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:24.356091 12397 consensus_queue.cc:1048] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 186b12638a87410487467df721a3cbda [LEADER]: Connected to new peer: Peer: permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:04:24.357159 12537 leader_election.cc:290] T 836251ebc2b74f5da88b8a8b2338e2a5 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:24.357595 12397 consensus_queue.cc:1059] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 186b12638a87410487467df721a3cbda [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 }, Status: INVALID_TERM, Last received: 3.1, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:04:24.357781 12537 consensus_queue.cc:1059] T c51782cd9d064ec0949e84e24dd4de67 P 186b12638a87410487467df721a3cbda [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:04:24.281355 12612 consensus_queue.cc:1048] T acc241e7273d49149984968596242712 P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.070s
I20260709 13:04:24.358644 12250 raft_consensus.cc:3055] T c51782cd9d064ec0949e84e24dd4de67 P 186b12638a87410487467df721a3cbda [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:04:24.359028 12250 raft_consensus.cc:740] T c51782cd9d064ec0949e84e24dd4de67 P 186b12638a87410487467df721a3cbda [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 186b12638a87410487467df721a3cbda, State: Running, Role: LEADER
I20260709 13:04:24.359308 10919 leader_election.cc:304] T 4c2a35bb99374e93b573cf870d04faaa P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23; no voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47
I20260709 13:04:24.359926 12250 consensus_queue.cc:260] T c51782cd9d064ec0949e84e24dd4de67 P 186b12638a87410487467df721a3cbda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:24.285212 10879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a610d1a71f52432e88fe56130f921fa2" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:24.286046 11103 raft_consensus.cc:2468] T 868e92339c2f4508af0afbd07c814d64 P 0172fed22b6b4890a7fdc953fa128af4 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 5.
I20260709 13:04:24.361052 10919 leader_election.cc:304] T e5ba72ccd854430baadbc636dfd17136 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:04:24.230034 12513 raft_consensus.cc:2804] T 836251ebc2b74f5da88b8a8b2338e2a5 P 0172fed22b6b4890a7fdc953fa128af4 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:04:24.292550 12558 leader_election.cc:290] T bb7d67be500947b3aef4c4245e33671c P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:24.362203 12513 raft_consensus.cc:697] T 836251ebc2b74f5da88b8a8b2338e2a5 P 0172fed22b6b4890a7fdc953fa128af4 [term 5 LEADER]: Becoming Leader. State: Replica: 0172fed22b6b4890a7fdc953fa128af4, State: Running, Role: LEADER
I20260709 13:04:24.362283 10879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b67eff0634e540eab91cf8cee795969f" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:24.363559 10879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7709387f4d3d4f9992d97a0fbca39a40" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:04:24.362298 12558 raft_consensus.cc:2749] T 4c2a35bb99374e93b573cf870d04faaa P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:04:24.364737 10879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d7ce47b9a92d43dd88d95bcda5038597" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:24.364763 12558 raft_consensus.cc:2804] T e5ba72ccd854430baadbc636dfd17136 P 10ca5ee68ad64b3283b9d235352c9d23 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:04:24.299317 11021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e5d327630e5438b878035d28efe6c0c" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:24.365578 12558 raft_consensus.cc:493] T e5ba72ccd854430baadbc636dfd17136 P 10ca5ee68ad64b3283b9d235352c9d23 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:24.365908 11021 raft_consensus.cc:2468] T 0e5d327630e5438b878035d28efe6c0c P 873357742d5843579556ee8166f23f47 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 5.
W20260709 13:04:24.299983 10843 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:24.366514 11026 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46628 (request call id 290) took 1007 ms. Trace:
W20260709 13:04:24.367415 10843 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:24.301328 11024 raft_consensus.cc:1275] T 30f628e3d7914d609e7f1d980a74c654 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Refusing update from remote peer 186b12638a87410487467df721a3cbda: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:04:24.208106 12585 raft_consensus.cc:515] T 63d44f6ba6434ab2870fe3190f4d7c2f P 186b12638a87410487467df721a3cbda [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:24.308104 11067 leader_election.cc:304] T 4f9eb06ed01f4ce68be3af8a8434665b P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:04:24.339300 10955 rpcz_store.cc:276] 0709 13:04:22.416776 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:22.803612 (+386836us) service_pool.cc:224] Handling call
0709 13:04:22.807571 (+  3959us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:23.873347 (+1065776us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:23.873386 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:23.873954 (+   568us) log.cc:844] Serialized 26 byte log entry
0709 13:04:24.081493 (+207539us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:04:24.150811 (+ 69318us) log.cc:844] Serialized 14 byte log entry
0709 13:04:24.151087 (+   276us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:24.151173 (+    86us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:24.338609 (+187436us) raft_consensus.cc:1685] finished
0709 13:04:24.338633 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:24.338782 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1064828,"prepare.queue_time_us":69858,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":39,"raft.queue_time_us":1070997,"raft.run_cpu_time_us":249,"raft.run_wall_time_us":248,"thread_start_us":207148,"threads_started":1,"wal-append.queue_time_us":332647}
W20260709 13:04:24.370873 11067 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:45807, user_credentials={real_user=slave}} blocked reactor thread for 62877us
I20260709 13:04:24.371318 12541 leader_election.cc:290] T a610d1a71f52432e88fe56130f921fa2 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:24.372373 12612 raft_consensus.cc:2804] T 4f9eb06ed01f4ce68be3af8a8434665b P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:04:24.372828 12612 raft_consensus.cc:493] T 4f9eb06ed01f4ce68be3af8a8434665b P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 186b12638a87410487467df721a3cbda)
I20260709 13:04:24.373188 12612 raft_consensus.cc:3060] T 4f9eb06ed01f4ce68be3af8a8434665b P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:04:24.373723 12541 raft_consensus.cc:493] T 401a9cf11eac479aa445ec8c1d6e33b2 P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:24.374221 10877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87cd8f5097624c8692f1d83544d8881e" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:04:24.374276 12541 raft_consensus.cc:515] T 401a9cf11eac479aa445ec8c1d6e33b2 P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:24.375133 12611 consensus_queue.cc:1048] T 401a9cf11eac479aa445ec8c1d6e33b2 P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:04:24.375648 10877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7757ecbf402e47bfbc93d0447741e26c" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:24.376852 10877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555217d45d9b4ee492dbb3ddaf0f7c02" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:24.301824 12578 raft_consensus.cc:515] T b4cd3d7dfbbf4096908ff11e9dc57141 P 10ca5ee68ad64b3283b9d235352c9d23 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:24.287926 10948 raft_consensus.cc:2410] T 732c4adeeff6422497aa7db4a82f535b P 10ca5ee68ad64b3283b9d235352c9d23 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 873357742d5843579556ee8166f23f47 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260709 13:04:24.303529 12545 raft_consensus.cc:3060] T d71ca3a1ba6b48478aaf62741bf3cf6a P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:04:24.321393 11098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7709387f4d3d4f9992d97a0fbca39a40" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:04:24.287067 11022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d371c5e27bd843469d2e2e453734cb4a" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
I20260709 13:04:24.379069 12578 leader_election.cc:290] T b4cd3d7dfbbf4096908ff11e9dc57141 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 5 election: Requested vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:24.321794 10875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d156d6ae2ca4fa59a1e387edf0fea39" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 5 candidate_status { last_received { term: 3 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:04:24.318285 12532 raft_consensus.cc:740] T 1e4f2241a94a41f38e481137f868b83d P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 10ca5ee68ad64b3283b9d235352c9d23, State: Running, Role: LEADER
I20260709 13:04:24.383605 10875 raft_consensus.cc:1240] T d371c5e27bd843469d2e2e453734cb4a P 186b12638a87410487467df721a3cbda [term 5 FOLLOWER]: Rejecting Update request from peer 0172fed22b6b4890a7fdc953fa128af4 for earlier term 4. Current term is 5. Ops: []
I20260709 13:04:24.383277 12532 consensus_queue.cc:260] T 1e4f2241a94a41f38e481137f868b83d P 10ca5ee68ad64b3283b9d235352c9d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:24.384640 11029 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2567c2d84e4648d7ba5dc23a3df32817" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:24.385326 11029 raft_consensus.cc:2468] T 2567c2d84e4648d7ba5dc23a3df32817 P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 3.
I20260709 13:04:24.385869 11028 raft_consensus.cc:2468] T 868e92339c2f4508af0afbd07c814d64 P 873357742d5843579556ee8166f23f47 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 5.
W20260709 13:04:24.386843 12394 consensus_meta.cc:287] T 7709387f4d3d4f9992d97a0fbca39a40 P 0172fed22b6b4890a7fdc953fa128af4: Time spent flushing consensus metadata: real 0.570s	user 0.013s	sys 0.001s
I20260709 13:04:24.387444 12394 raft_consensus.cc:2749] T 7709387f4d3d4f9992d97a0fbca39a40 P 0172fed22b6b4890a7fdc953fa128af4 [term 8 FOLLOWER]: Leader election lost for term 7. Reason: Vote denied by peer 873357742d5843579556ee8166f23f47 (127.0.107.195:45807) with higher term. Message: Invalid argument: T 7709387f4d3d4f9992d97a0fbca39a40 P 873357742d5843579556ee8166f23f47 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 0172fed22b6b4890a7fdc953fa128af4 for earlier term 7. Current term is 8.
I20260709 13:04:24.388192 12585 leader_election.cc:290] T 63d44f6ba6434ab2870fe3190f4d7c2f P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
W20260709 13:04:24.393082 10992 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:24.363296 12513 consensus_queue.cc:237] T 836251ebc2b74f5da88b8a8b2338e2a5 P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:24.294971 12329 consensus_queue.cc:1048] T a1ec2b50c6ad4f7c8033476208abb26d P 873357742d5843579556ee8166f23f47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:04:24.405695 10877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "401a9cf11eac479aa445ec8c1d6e33b2" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:24.342235 10874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e4f2241a94a41f38e481137f868b83d" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:04:24.406873 10919 leader_election.cc:304] T 2567c2d84e4648d7ba5dc23a3df32817 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47; no voters: 186b12638a87410487467df721a3cbda
I20260709 13:04:24.407466 10877 raft_consensus.cc:2410] T 401a9cf11eac479aa445ec8c1d6e33b2 P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 873357742d5843579556ee8166f23f47 for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 13:04:24.283650 12582 raft_consensus.cc:515] T d270b9e5481245d9b5340a454a7cd10b P 10ca5ee68ad64b3283b9d235352c9d23 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:24.366015 12558 raft_consensus.cc:3060] T e5ba72ccd854430baadbc636dfd17136 P 10ca5ee68ad64b3283b9d235352c9d23 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:04:24.336283 12570 raft_consensus.cc:3060] T 72b111ad7b5f4753ab1d2b5ee1114840 P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:04:24.338181 10920 leader_election.cc:304] T 72b111ad7b5f4753ab1d2b5ee1114840 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23; no voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda
W20260709 13:04:24.413097 10920 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 75032.3us
I20260709 13:04:24.399151 10921 leader_election.cc:304] T f34f2bc02d85444f9096e0f5b4fe4331 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda; no voters: 
I20260709 13:04:24.418047 10951 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c51782cd9d064ec0949e84e24dd4de67" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:04:24.418720 10951 raft_consensus.cc:3060] T c51782cd9d064ec0949e84e24dd4de67 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Advancing to term 5
W20260709 13:04:24.418236 10994 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 97110.3us
I20260709 13:04:24.401110 11027 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5299b8d044f41b7b68057e63d17b159" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:24.402972 11029 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7839ca2d2c13431a85f44d7c458c6351" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:24.403713 11098 raft_consensus.cc:2468] T 7709387f4d3d4f9992d97a0fbca39a40 P 0172fed22b6b4890a7fdc953fa128af4 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 8.
W20260709 13:04:24.414292 10919 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:24.414815 10954 raft_consensus.cc:1240] T 72b111ad7b5f4753ab1d2b5ee1114840 P 10ca5ee68ad64b3283b9d235352c9d23 [term 5 FOLLOWER]: Rejecting Update request from peer 0172fed22b6b4890a7fdc953fa128af4 for earlier term 4. Current term is 5. Ops: []
I20260709 13:04:24.416651 10949 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b67eff0634e540eab91cf8cee795969f" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:24.063606 12500 consensus_queue.cc:260] T d7ce47b9a92d43dd88d95bcda5038597 P 186b12638a87410487467df721a3cbda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:24.419660 10994 leader_election.cc:304] T 868e92339c2f4508af0afbd07c814d64 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47; no voters: 186b12638a87410487467df721a3cbda
I20260709 13:04:24.421782 10919 leader_election.cc:304] T a5299b8d044f41b7b68057e63d17b159 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23; no voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47
I20260709 13:04:24.366904 11026 rpcz_store.cc:276] 0709 13:04:23.358751 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:23.360317 (+  1566us) service_pool.cc:224] Handling call
0709 13:04:23.797514 (+437197us) spinlock_profiling.cc:243] Waited 4.21 ms on lock 0x7b58007a2898. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f426303db71 00007f426303deee 0000000000375939 00007f426b0b11e0 00007f426b17a0c4 00007f426b17a131 00007f426b13bfc0 00007f426b126929 00007f426b1263d2 00007f426469dbd4
0709 13:04:23.824592 (+ 27078us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:23.900883 (+ 76291us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:24.366260 (+465377us) spinlock_profiling.cc:243] Waited 20.1 ms on lock 0x7b54006c25d0. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661618ad 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:04:24.366418 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":55914112}
I20260709 13:04:24.423403 11024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb7d67be500947b3aef4c4245e33671c" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:24.424145 11103 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e5d327630e5438b878035d28efe6c0c" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:24.416030 10950 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d439326f23747eaab3c1e7da2d6b785" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:24.426344 10954 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5ba72ccd854430baadbc636dfd17136" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:24.431388 10955 raft_consensus.cc:1240] T b4cd3d7dfbbf4096908ff11e9dc57141 P 10ca5ee68ad64b3283b9d235352c9d23 [term 5 FOLLOWER]: Rejecting Update request from peer 873357742d5843579556ee8166f23f47 for earlier term 4. Current term is 5. Ops: []
I20260709 13:04:24.437008 10950 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bac7e5040f984ba2b7146bd36af7267e" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:24.438828 10955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d7ce47b9a92d43dd88d95bcda5038597" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:24.377092 12541 leader_election.cc:290] T 401a9cf11eac479aa445ec8c1d6e33b2 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:24.384647 12532 raft_consensus.cc:3060] T 1e4f2241a94a41f38e481137f868b83d P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:04:24.441296 10952 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d156d6ae2ca4fa59a1e387edf0fea39" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 5 candidate_status { last_received { term: 3 index: 1 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:04:24.450203 10952 raft_consensus.cc:3060] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:04:24.338195 12550 raft_consensus.cc:2804] T d7ce47b9a92d43dd88d95bcda5038597 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:04:24.400030 12397 raft_consensus.cc:3055] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 186b12638a87410487467df721a3cbda [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:04:24.436184 11100 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5ba72ccd854430baadbc636dfd17136" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:24.443773 10873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d270b9e5481245d9b5340a454a7cd10b" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:24.436064 12557 raft_consensus.cc:2749] T a5299b8d044f41b7b68057e63d17b159 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:04:24.451249 10992 leader_election.cc:304] T 732c4adeeff6422497aa7db4a82f535b P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 873357742d5843579556ee8166f23f47; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda
I20260709 13:04:24.399824 10872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1ec2b50c6ad4f7c8033476208abb26d" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:24.411118 12329 consensus_queue.cc:1243] T a1ec2b50c6ad4f7c8033476208abb26d P 873357742d5843579556ee8166f23f47 [LEADER]: Peer 0172fed22b6b4890a7fdc953fa128af4 log is divergent from this leader: its last log entry 1.1 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 0
I20260709 13:04:24.456156 10875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5a5f5a608504650b48fd27f163faed6" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:24.437428 12525 raft_consensus.cc:2804] T f34f2bc02d85444f9096e0f5b4fe4331 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:04:24.462716 10951 raft_consensus.cc:2468] T c51782cd9d064ec0949e84e24dd4de67 P 10ca5ee68ad64b3283b9d235352c9d23 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 5.
W20260709 13:04:24.348176 10832 catalog_manager.cc:6160] T 00000000000000000000000000000000 P c669bee8ad644d0d956c148b33886b51: Tablet c31881534c5e4f0089c82057c1746652 (table test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]) was not created within the allowed timeout. Replacing with a new tablet 15e2677516d34fcaa77bf45d5584d668
I20260709 13:04:24.535597 11027 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a1716c0adf34bee8a4e99341ba5788f" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
W20260709 13:04:24.544590 10826 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 13:04:24.545456 11098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb7d67be500947b3aef4c4245e33671c" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:24.490120 11099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5ba72ccd854430baadbc636dfd17136" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:24.425343 12578 raft_consensus.cc:2804] T 2567c2d84e4648d7ba5dc23a3df32817 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260709 13:04:24.558681 10995 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:38967, user_credentials={real_user=slave}} blocked reactor thread for 112295us
W20260709 13:04:24.559424 10992 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 108330us
I20260709 13:04:24.559597 10995 leader_election.cc:304] T 1e4f2241a94a41f38e481137f868b83d P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 873357742d5843579556ee8166f23f47; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda
I20260709 13:04:24.509449 10843 leader_election.cc:304] T 4c2a35bb99374e93b573cf870d04faaa P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 186b12638a87410487467df721a3cbda; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47
I20260709 13:04:24.433439 12564 raft_consensus.cc:2749] T 72b111ad7b5f4753ab1d2b5ee1114840 P 10ca5ee68ad64b3283b9d235352c9d23 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 13:04:24.509904 10873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "868e92339c2f4508af0afbd07c814d64" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:24.507263 10875 raft_consensus.cc:2468] T e5a5f5a608504650b48fd27f163faed6 P 186b12638a87410487467df721a3cbda [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 6.
I20260709 13:04:24.506217 11100 raft_consensus.cc:2468] T e5ba72ccd854430baadbc636dfd17136 P 0172fed22b6b4890a7fdc953fa128af4 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 6.
I20260709 13:04:24.457101 12550 raft_consensus.cc:493] T d7ce47b9a92d43dd88d95bcda5038597 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:24.516155 11097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff13ae9a47c7455f962406c6d131a3ff" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:24.510654 11103 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f34f2bc02d85444f9096e0f5b4fe4331" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:24.573064 11103 raft_consensus.cc:2468] T f34f2bc02d85444f9096e0f5b4fe4331 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 2.
I20260709 13:04:24.458828 12434 raft_consensus.cc:2764] T 868e92339c2f4508af0afbd07c814d64 P 873357742d5843579556ee8166f23f47 [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260709 13:04:24.555239 10951 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a154cc8b181f460abf4880064379948b" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:24.581801 10951 raft_consensus.cc:2468] T a154cc8b181f460abf4880064379948b P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 2.
W20260709 13:04:24.556191 10919 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:45807, user_credentials={real_user=slave}} blocked reactor thread for 47750.4us
I20260709 13:04:24.566401 10877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5de5f0581e3b41dea729363d6ca563bb" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
W20260709 13:04:24.507336 12587 consensus_meta.cc:287] T 6f46a5102fd64b45bf1c4dff863d2b4d P 10ca5ee68ad64b3283b9d235352c9d23: Time spent flushing consensus metadata: real 0.534s	user 0.008s	sys 0.005s
I20260709 13:04:24.570006 10873 raft_consensus.cc:2368] T 868e92339c2f4508af0afbd07c814d64 P 186b12638a87410487467df721a3cbda [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0172fed22b6b4890a7fdc953fa128af4 for earlier term 4. Current term is 5.
I20260709 13:04:24.515760 10949 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d270b9e5481245d9b5340a454a7cd10b" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:24.526315 12555 leader_election.cc:290] T 5de5f0581e3b41dea729363d6ca563bb P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:24.458081 12397 raft_consensus.cc:740] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 186b12638a87410487467df721a3cbda [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 186b12638a87410487467df721a3cbda, State: Running, Role: LEADER
I20260709 13:04:24.454629 11022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11de93cd9b9d457489ff1cb80f976c9a" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:24.585372 10952 raft_consensus.cc:2410] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 10ca5ee68ad64b3283b9d235352c9d23 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 0172fed22b6b4890a7fdc953fa128af4 for term 5 because replica has last-logged OpId of term: 4 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 1.
I20260709 13:04:24.603240 11098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4cd3d7dfbbf4096908ff11e9dc57141" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
W20260709 13:04:24.597432 10832 catalog_manager.cc:6160] T 00000000000000000000000000000000 P c669bee8ad644d0d956c148b33886b51: Tablet 75492f86fde046c3900df3eac8292fd4 (table test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]) was not created within the allowed timeout. Replacing with a new tablet bd6a96d732d640d692e3c1985d466525
W20260709 13:04:24.565858 12640 log.cc:927] Time spent T 4f9eb06ed01f4ce68be3af8a8434665b P 873357742d5843579556ee8166f23f47: Append to log took a long time: real 0.063s	user 0.000s	sys 0.001s
I20260709 13:04:24.589021 10949 raft_consensus.cc:2468] T d270b9e5481245d9b5340a454a7cd10b P 10ca5ee68ad64b3283b9d235352c9d23 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 6.
I20260709 13:04:24.460317 10948 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87cd8f5097624c8692f1d83544d8881e" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:04:24.556422 12525 raft_consensus.cc:493] T f34f2bc02d85444f9096e0f5b4fe4331 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:24.586468 12548 raft_consensus.cc:2749] T 1e4f2241a94a41f38e481137f868b83d P 873357742d5843579556ee8166f23f47 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20260709 13:04:24.577198 10994 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 119736us
W20260709 13:04:24.568511 12618 log.cc:927] Time spent T a1ec2b50c6ad4f7c8033476208abb26d P 186b12638a87410487467df721a3cbda: Append to log took a long time: real 0.114s	user 0.001s	sys 0.000s
I20260709 13:04:24.565598 12578 raft_consensus.cc:493] T 2567c2d84e4648d7ba5dc23a3df32817 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 186b12638a87410487467df721a3cbda)
I20260709 13:04:24.608346 10919 leader_election.cc:304] T 7839ca2d2c13431a85f44d7c458c6351 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23; no voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47
W20260709 13:04:24.611466 10843 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 102108us
I20260709 13:04:24.611938 12518 raft_consensus.cc:493] T e89e640169e14e2d98f968f76027daa7 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:24.612301 12586 raft_consensus.cc:2749] T 4c2a35bb99374e93b573cf870d04faaa P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:04:24.583156 12550 raft_consensus.cc:3060] T d7ce47b9a92d43dd88d95bcda5038597 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:04:24.615257 12397 consensus_queue.cc:260] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 186b12638a87410487467df721a3cbda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:24.568925 12394 raft_consensus.cc:493] T 6a9353d1d230416296627d7cf8b1d425 P 0172fed22b6b4890a7fdc953fa128af4 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:24.505599 12471 raft_consensus.cc:515] T a64b7e99d6c54934a7de6f6dffcf5e94 P 10ca5ee68ad64b3283b9d235352c9d23 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
W20260709 13:04:24.623039 10845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 273375us
I20260709 13:04:24.586057 12418 raft_consensus.cc:2749] T 732c4adeeff6422497aa7db4a82f535b P 873357742d5843579556ee8166f23f47 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:04:24.624044 10873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f9eb06ed01f4ce68be3af8a8434665b" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:24.622129 10920 leader_election.cc:304] T bb7d67be500947b3aef4c4245e33671c P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23; no voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47
I20260709 13:04:24.625798 10952 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f1466124dbd4f6ebb3c4d62ca48bf96" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:24.515205 11023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4cd3d7dfbbf4096908ff11e9dc57141" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
I20260709 13:04:24.662096 10875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a1716c0adf34bee8a4e99341ba5788f" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:04:24.664192 12471 leader_election.cc:290] T a64b7e99d6c54934a7de6f6dffcf5e94 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 5 election: Requested vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:24.627542 11099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5de5f0581e3b41dea729363d6ca563bb" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:24.631767 11066 leader_election.cc:304] T bac7e5040f984ba2b7146bd36af7267e P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47
I20260709 13:04:24.635077 10949 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb7d67be500947b3aef4c4245e33671c" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:24.636353 10948 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c31881534c5e4f0089c82057c1746652" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:24.636952 10871 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f34f2bc02d85444f9096e0f5b4fe4331" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:24.631477 12525 raft_consensus.cc:3060] T f34f2bc02d85444f9096e0f5b4fe4331 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:04:24.642151 10843 leader_election.cc:304] T 1e4f2241a94a41f38e481137f868b83d P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 186b12638a87410487467df721a3cbda; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47
I20260709 13:04:24.642138 10877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a69f32c53f24a50b7852d143665d7b4" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:04:24.645474 10994 leader_election.cc:304] T 525bfdbd6d934afd823f7d3ee3f9d052 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47; no voters: 10ca5ee68ad64b3283b9d235352c9d23
I20260709 13:04:24.647687 10951 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5a5f5a608504650b48fd27f163faed6" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:24.604223 12564 raft_consensus.cc:493] T e5a5f5a608504650b48fd27f163faed6 P 10ca5ee68ad64b3283b9d235352c9d23 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:24.643755 12555 raft_consensus.cc:2749] T 7839ca2d2c13431a85f44d7c458c6351 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:04:24.647435 12518 raft_consensus.cc:515] T e89e640169e14e2d98f968f76027daa7 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:24.660543 10845 leader_election.cc:304] T 5d439326f23747eaab3c1e7da2d6b785 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 186b12638a87410487467df721a3cbda; no voters: 0172fed22b6b4890a7fdc953fa128af4, 10ca5ee68ad64b3283b9d235352c9d23
I20260709 13:04:24.686877 10880 raft_consensus.cc:1275] T 401a9cf11eac479aa445ec8c1d6e33b2 P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Refusing update from remote peer 0172fed22b6b4890a7fdc953fa128af4: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 13:04:24.687417 10845 leader_election.cc:304] T 7757ecbf402e47bfbc93d0447741e26c P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 186b12638a87410487467df721a3cbda; no voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47
I20260709 13:04:24.594236 12587 raft_consensus.cc:515] T 6f46a5102fd64b45bf1c4dff863d2b4d P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:24.602520 12569 consensus_queue.cc:1059] T b4cd3d7dfbbf4096908ff11e9dc57141 P 873357742d5843579556ee8166f23f47 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.050s
I20260709 13:04:24.645792 12578 raft_consensus.cc:3060] T 2567c2d84e4648d7ba5dc23a3df32817 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:04:24.631829 12611 raft_consensus.cc:493] T 32147c6d2d5a477aa38c597c7f78df5b P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 10ca5ee68ad64b3283b9d235352c9d23)
I20260709 13:04:24.668614 11029 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63d44f6ba6434ab2870fe3190f4d7c2f" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
W20260709 13:04:24.670292 10920 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 48294.7us
I20260709 13:04:24.667210 10921 leader_election.cc:304] T 11de93cd9b9d457489ff1cb80f976c9a P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23; no voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47
I20260709 13:04:24.665318 10953 raft_consensus.cc:1240] T a64b7e99d6c54934a7de6f6dffcf5e94 P 10ca5ee68ad64b3283b9d235352c9d23 [term 5 FOLLOWER]: Rejecting Update request from peer 873357742d5843579556ee8166f23f47 for earlier term 4. Current term is 5. Ops: []
I20260709 13:04:24.680888 10949 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "836251ebc2b74f5da88b8a8b2338e2a5" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:24.682961 10843 leader_election.cc:304] T e89e640169e14e2d98f968f76027daa7 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 186b12638a87410487467df721a3cbda; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47
I20260709 13:04:24.703903 10949 raft_consensus.cc:2468] T 836251ebc2b74f5da88b8a8b2338e2a5 P 10ca5ee68ad64b3283b9d235352c9d23 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 5.
I20260709 13:04:24.685456 10952 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff13ae9a47c7455f962406c6d131a3ff" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:24.707195 10994 leader_election.cc:304] T 4f9eb06ed01f4ce68be3af8a8434665b P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 873357742d5843579556ee8166f23f47; no voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda
I20260709 13:04:24.646332 12545 raft_consensus.cc:515] T d71ca3a1ba6b48478aaf62741bf3cf6a P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:24.680876 12575 raft_consensus.cc:2749] T bb7d67be500947b3aef4c4245e33671c P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:04:24.692026 10876 raft_consensus.cc:2468] T 7709387f4d3d4f9992d97a0fbca39a40 P 186b12638a87410487467df721a3cbda [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 8.
I20260709 13:04:24.695851 12479 raft_consensus.cc:2749] T 1e4f2241a94a41f38e481137f868b83d P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:04:24.719899 10994 leader_election.cc:304] T a977842ba4b744f4bb2febd55d286006 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47; no voters: 10ca5ee68ad64b3283b9d235352c9d23
I20260709 13:04:24.721165 10946 raft_consensus.cc:1275] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 10ca5ee68ad64b3283b9d235352c9d23 [term 7 FOLLOWER]: Refusing update from remote peer 873357742d5843579556ee8166f23f47: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 4. (term mismatch)
I20260709 13:04:24.721590 10946 pending_rounds.cc:85] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 10ca5ee68ad64b3283b9d235352c9d23: Aborting all ops after (but not including) 3
I20260709 13:04:24.721796 10946 pending_rounds.cc:107] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 10ca5ee68ad64b3283b9d235352c9d23: Aborting uncommitted NO_OP operation due to leader change: 6.4
I20260709 13:04:24.722045 10946 raft_consensus.cc:2889] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 10ca5ee68ad64b3283b9d235352c9d23 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:04:24.723465 10845 leader_election.cc:304] T 72b111ad7b5f4753ab1d2b5ee1114840 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 186b12638a87410487467df721a3cbda; no voters: 0172fed22b6b4890a7fdc953fa128af4, 10ca5ee68ad64b3283b9d235352c9d23
I20260709 13:04:24.696851 12397 raft_consensus.cc:3060] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:04:24.733135 12526 raft_consensus.cc:515] T e96e80ba9ac34d0998395515e5c8c41a P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:24.699018 12548 raft_consensus.cc:2804] T 525bfdbd6d934afd823f7d3ee3f9d052 P 873357742d5843579556ee8166f23f47 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:04:24.660564 12394 raft_consensus.cc:515] T 6a9353d1d230416296627d7cf8b1d425 P 0172fed22b6b4890a7fdc953fa128af4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:24.738013 10950 raft_consensus.cc:1240] T d71ca3a1ba6b48478aaf62741bf3cf6a P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Rejecting Update request from peer 873357742d5843579556ee8166f23f47 for earlier term 3. Current term is 4. Ops: []
I20260709 13:04:24.725025 10994 leader_election.cc:304] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47; no voters: 186b12638a87410487467df721a3cbda
I20260709 13:04:24.685323 12586 consensus_queue.cc:1059] T 87cd8f5097624c8692f1d83544d8881e P 186b12638a87410487467df721a3cbda [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.040s
I20260709 13:04:24.697289 12564 raft_consensus.cc:515] T e5a5f5a608504650b48fd27f163faed6 P 10ca5ee68ad64b3283b9d235352c9d23 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:24.702520 11022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "401a9cf11eac479aa445ec8c1d6e33b2" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:24.709707 12387 raft_consensus.cc:2749] T bac7e5040f984ba2b7146bd36af7267e P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:04:24.709158 12555 raft_consensus.cc:493] T 65bf0a62f4d84ef18dd6b23e082724b8 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:24.747639 10994 leader_election.cc:304] T d371c5e27bd843469d2e2e453734cb4a P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 873357742d5843579556ee8166f23f47; no voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda
I20260709 13:04:24.749548 11069 leader_election.cc:304] T 8c160ddfe9b34bdaa3835d0915c1e06f P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4; no voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47
I20260709 13:04:24.709375 12644 raft_consensus.cc:2749] T 11de93cd9b9d457489ff1cb80f976c9a P 10ca5ee68ad64b3283b9d235352c9d23 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:04:24.849135 10952 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a69f32c53f24a50b7852d143665d7b4" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:04:24.721803 12585 raft_consensus.cc:2749] T 5d439326f23747eaab3c1e7da2d6b785 P 186b12638a87410487467df721a3cbda [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:04:24.722080 12462 raft_consensus.cc:2749] T 4f9eb06ed01f4ce68be3af8a8434665b P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:04:24.764003 10949 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63d44f6ba6434ab2870fe3190f4d7c2f" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:24.709169 12517 raft_consensus.cc:515] T 30f628e3d7914d609e7f1d980a74c654 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:24.762113 12513 raft_consensus.cc:493] T 75492f86fde046c3900df3eac8292fd4 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:24.773403 12526 leader_election.cc:290] T e96e80ba9ac34d0998395515e5c8c41a P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 4 election: Requested vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:24.775239 12518 leader_election.cc:290] T e89e640169e14e2d98f968f76027daa7 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:24.759815 12488 raft_consensus.cc:2749] T 72b111ad7b5f4753ab1d2b5ee1114840 P 186b12638a87410487467df721a3cbda [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 13:04:24.776000 12479 raft_consensus.cc:2749] T e89e640169e14e2d98f968f76027daa7 P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:04:24.776772 10919 leader_election.cc:304] T b4cd3d7dfbbf4096908ff11e9dc57141 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23; no voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47
I20260709 13:04:24.777683 11025 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a64b7e99d6c54934a7de6f6dffcf5e94" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
I20260709 13:04:24.757779 12582 leader_election.cc:290] T d270b9e5481245d9b5340a454a7cd10b P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:24.782823 10947 raft_consensus.cc:1240] T e96e80ba9ac34d0998395515e5c8c41a P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Rejecting Update request from peer 873357742d5843579556ee8166f23f47 for earlier term 3. Current term is 4. Ops: []
I20260709 13:04:24.784400 10946 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5de5f0581e3b41dea729363d6ca563bb" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
W20260709 13:04:24.870836 10919 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:45807, user_credentials={real_user=slave}} blocked reactor thread for 94168.6us
I20260709 13:04:24.735829 12202 raft_consensus.cc:2804] T a977842ba4b744f4bb2febd55d286006 P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:04:24.736631 12591 raft_consensus.cc:493] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 10ca5ee68ad64b3283b9d235352c9d23 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:24.737468 10871 raft_consensus.cc:2410] T f34f2bc02d85444f9096e0f5b4fe4331 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0172fed22b6b4890a7fdc953fa128af4 for term 3 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260709 13:04:24.792627 10878 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51718 (request call id 412) took 1043 ms. Trace:
I20260709 13:04:24.806263 12545 leader_election.cc:290] T d71ca3a1ba6b48478aaf62741bf3cf6a P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 4 election: Requested vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:24.786453 12534 raft_consensus.cc:3055] T b4cd3d7dfbbf4096908ff11e9dc57141 P 873357742d5843579556ee8166f23f47 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:04:24.761159 12450 raft_consensus.cc:515] T acc241e7273d49149984968596242712 P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:24.809654 12250 raft_consensus.cc:3060] T c51782cd9d064ec0949e84e24dd4de67 P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Advancing to term 5
W20260709 13:04:24.811012 12648 log.cc:927] Time spent T 5d439326f23747eaab3c1e7da2d6b785 P 10ca5ee68ad64b3283b9d235352c9d23: Append to log took a long time: real 0.053s	user 0.000s	sys 0.000s
I20260709 13:04:24.812237 12587 leader_election.cc:290] T 6f46a5102fd64b45bf1c4dff863d2b4d P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 3 election: Requested vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:24.760351 12476 raft_consensus.cc:2749] T 7757ecbf402e47bfbc93d0447741e26c P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:04:24.709830 12611 raft_consensus.cc:515] T 32147c6d2d5a477aa38c597c7f78df5b P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:24.901299 12611 leader_election.cc:290] T 32147c6d2d5a477aa38c597c7f78df5b P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
W20260709 13:04:24.795468 12646 log.cc:927] Time spent T a1ec2b50c6ad4f7c8033476208abb26d P 873357742d5843579556ee8166f23f47: Append to log took a long time: real 0.163s	user 0.001s	sys 0.000s
I20260709 13:04:24.899323 11026 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f46a5102fd64b45bf1c4dff863d2b4d" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
I20260709 13:04:24.744756 12548 raft_consensus.cc:493] T 525bfdbd6d934afd823f7d3ee3f9d052 P 873357742d5843579556ee8166f23f47 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 10ca5ee68ad64b3283b9d235352c9d23)
I20260709 13:04:24.903003 12548 raft_consensus.cc:3060] T 525bfdbd6d934afd823f7d3ee3f9d052 P 873357742d5843579556ee8166f23f47 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:04:24.817153 10950 raft_consensus.cc:1240] T 6f46a5102fd64b45bf1c4dff863d2b4d P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Rejecting Update request from peer 873357742d5843579556ee8166f23f47 for earlier term 2. Current term is 3. Ops: []
W20260709 13:04:24.819119 11066 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 187461us
I20260709 13:04:24.905534 11066 leader_election.cc:304] T c51782cd9d064ec0949e84e24dd4de67 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 10ca5ee68ad64b3283b9d235352c9d23; no voters: 
W20260709 13:04:24.824985 11069 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:38967, user_credentials={real_user=slave}} blocked reactor thread for 75579.4us
I20260709 13:04:24.907447 11069 leader_election.cc:304] T 5d439326f23747eaab3c1e7da2d6b785 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda
I20260709 13:04:24.760852 12569 raft_consensus.cc:2804] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 873357742d5843579556ee8166f23f47 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:04:24.907989 12450 leader_election.cc:290] T acc241e7273d49149984968596242712 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 3 election: Requested vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:24.908465 12569 raft_consensus.cc:493] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 873357742d5843579556ee8166f23f47 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 186b12638a87410487467df721a3cbda)
I20260709 13:04:24.908762 11027 raft_consensus.cc:1240] T acc241e7273d49149984968596242712 P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Rejecting Update request from peer 0172fed22b6b4890a7fdc953fa128af4 for earlier term 2. Current term is 3. Ops: []
I20260709 13:04:24.909093 12569 raft_consensus.cc:3060] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 873357742d5843579556ee8166f23f47 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:04:24.896179 12591 raft_consensus.cc:515] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 10ca5ee68ad64b3283b9d235352c9d23 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:24.889055 12531 consensus_queue.cc:1059] T d71ca3a1ba6b48478aaf62741bf3cf6a P 873357742d5843579556ee8166f23f47 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.024s
I20260709 13:04:24.881691 12581 raft_consensus.cc:2749] T b4cd3d7dfbbf4096908ff11e9dc57141 P 10ca5ee68ad64b3283b9d235352c9d23 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 13:04:24.913096 11023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e96e80ba9ac34d0998395515e5c8c41a" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
    @     0x7f4263285d05 testing::TestSuite::Run() at ??:0
I20260709 13:04:24.881445 12424 raft_consensus.cc:3060] T d371c5e27bd843469d2e2e453734cb4a P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:04:24.863229 10951 raft_consensus.cc:2393] T e5a5f5a608504650b48fd27f163faed6 P 10ca5ee68ad64b3283b9d235352c9d23 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0172fed22b6b4890a7fdc953fa128af4 in current term 6: Already voted for candidate 10ca5ee68ad64b3283b9d235352c9d23 in this term.
I20260709 13:04:24.802626 12541 raft_consensus.cc:3055] T 87cd8f5097624c8692f1d83544d8881e P 186b12638a87410487467df721a3cbda [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:04:24.815349 11021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff13ae9a47c7455f962406c6d131a3ff" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:24.814574 10948 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "836251ebc2b74f5da88b8a8b2338e2a5" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
W20260709 13:04:24.785653 10905 heartbeater.cc:646] Failed to heartbeat to 127.0.107.254:40861 (0 consecutive failures): Timed out: Failed to send heartbeat to master: TSHeartbeat RPC to 127.0.107.254:40861 timed out after 15.000s (SENT)
W20260709 13:04:24.799762 10832 catalog_manager.cc:6160] T 00000000000000000000000000000000 P c669bee8ad644d0d956c148b33886b51: Tablet 1e4f2241a94a41f38e481137f868b83d (table test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]) was not created within the allowed timeout. Replacing with a new tablet 19e6d6a833594546a23ba5290786550e
W20260709 13:04:24.873730 10994 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 126197us
I20260709 13:04:24.876982 10878 rpcz_store.cc:276] 0709 13:04:23.748859 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:23.750420 (+  1561us) service_pool.cc:224] Handling call
0709 13:04:23.750885 (+   465us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:04:23.757963 (+  7078us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:23.758010 (+    47us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:04:23.762722 (+  4712us) log.cc:844] Serialized 52 byte log entry
0709 13:04:24.372831 (+610109us) spinlock_profiling.cc:243] Waited 1.43 ms on lock 0x7b180013d200. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f426bf3dde1 00007f426bf4a5b1 00007f4263057e07 00007f426beb2ab9 00007f426307469b 00007f426307293f 00007f4263074c02 00007f4265cfc20b 00007f4265d01d23 00007f4265d020d1 00007f426611da0f
0709 13:04:24.684072 (+311241us) raft_consensus.cc:1645] Marking committed up to 1
0709 13:04:24.684142 (+    70us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:24.684256 (+   114us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:24.792211 (+107955us) raft_consensus.cc:1685] finished
0709 13:04:24.792237 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:24.792540 (+   303us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1041,"raft.queue_time_us":9615,"raft.run_cpu_time_us":171,"raft.run_wall_time_us":3543,"spinlock_wait_cycles":3282176,"thread_start_us":310955,"threads_started":1,"wal-append.queue_time_us":976171}
W20260709 13:04:24.882203 10843 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 199336us
I20260709 13:04:24.881206 12534 raft_consensus.cc:740] T b4cd3d7dfbbf4096908ff11e9dc57141 P 873357742d5843579556ee8166f23f47 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 873357742d5843579556ee8166f23f47, State: Running, Role: LEADER
I20260709 13:04:24.925978 10843 leader_election.cc:304] T a5299b8d044f41b7b68057e63d17b159 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda; no voters: 873357742d5843579556ee8166f23f47
I20260709 13:04:24.867223 12434 consensus_queue.cc:1048] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 873357742d5843579556ee8166f23f47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.050s
I20260709 13:04:24.929116 11066 leader_election.cc:304] T a154cc8b181f460abf4880064379948b P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 10ca5ee68ad64b3283b9d235352c9d23; no voters: 873357742d5843579556ee8166f23f47
I20260709 13:04:24.867666 12584 consensus_queue.cc:1059] T a64b7e99d6c54934a7de6f6dffcf5e94 P 873357742d5843579556ee8166f23f47 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.050s
W20260709 13:04:24.889062 10845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 153982us
I20260709 13:04:24.892652 11028 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d71ca3a1ba6b48478aaf62741bf3cf6a" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
I20260709 13:04:24.938050 10845 leader_election.cc:304] T a610d1a71f52432e88fe56130f921fa2 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda; no voters: 873357742d5843579556ee8166f23f47
I20260709 13:04:24.855476 12654 raft_consensus.cc:2749] T 8c160ddfe9b34bdaa3835d0915c1e06f P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:04:24.854638 12390 raft_consensus.cc:493] T 6f46a5102fd64b45bf1c4dff863d2b4d P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:24.894189 10995 leader_election.cc:304] T 7709387f4d3d4f9992d97a0fbca39a40 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:04:24.876677 12202 raft_consensus.cc:493] T a977842ba4b744f4bb2febd55d286006 P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 10ca5ee68ad64b3283b9d235352c9d23)
I20260709 13:04:24.794334 12612 raft_consensus.cc:515] T 4f9eb06ed01f4ce68be3af8a8434665b P 0172fed22b6b4890a7fdc953fa128af4 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:24.862840 10953 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f34f2bc02d85444f9096e0f5b4fe4331" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:24.906036 11020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e89e640169e14e2d98f968f76027daa7" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:24.919064 12541 raft_consensus.cc:740] T 87cd8f5097624c8692f1d83544d8881e P 186b12638a87410487467df721a3cbda [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 186b12638a87410487467df721a3cbda, State: Running, Role: LEADER
I20260709 13:04:24.858217 12555 raft_consensus.cc:515] T 65bf0a62f4d84ef18dd6b23e082724b8 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:24.983630 11972 consensus_queue.cc:1048] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 873357742d5843579556ee8166f23f47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 13:04:24.931517 11066 leader_election.cc:304] T d270b9e5481245d9b5340a454a7cd10b P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 10ca5ee68ad64b3283b9d235352c9d23; no voters: 
I20260709 13:04:24.986788 12394 leader_election.cc:290] T 6a9353d1d230416296627d7cf8b1d425 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
W20260709 13:04:24.986953 11066 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 55510us
I20260709 13:04:24.987730 12394 raft_consensus.cc:2804] T d270b9e5481245d9b5340a454a7cd10b P 0172fed22b6b4890a7fdc953fa128af4 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:04:24.988132 12394 raft_consensus.cc:493] T d270b9e5481245d9b5340a454a7cd10b P 0172fed22b6b4890a7fdc953fa128af4 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 186b12638a87410487467df721a3cbda)
I20260709 13:04:24.988513 12394 raft_consensus.cc:3060] T d270b9e5481245d9b5340a454a7cd10b P 0172fed22b6b4890a7fdc953fa128af4 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:04:24.988337 11066 leader_election.cc:304] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47
I20260709 13:04:24.887809 10952 raft_consensus.cc:1240] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 10ca5ee68ad64b3283b9d235352c9d23 [term 5 FOLLOWER]: Rejecting Update request from peer 186b12638a87410487467df721a3cbda for earlier term 4. Current term is 5. Ops: []
I20260709 13:04:24.936213 10876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5de5f0581e3b41dea729363d6ca563bb" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:24.992095 12578 raft_consensus.cc:515] T 2567c2d84e4648d7ba5dc23a3df32817 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:24.939070 10877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e96e80ba9ac34d0998395515e5c8c41a" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:04:24.998805 12555 leader_election.cc:290] T 65bf0a62f4d84ef18dd6b23e082724b8 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:24.939705 10880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e89e640169e14e2d98f968f76027daa7" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
W20260709 13:04:24.942849 10995 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:38967, user_credentials={real_user=slave}} blocked reactor thread for 48775.3us
W20260709 13:04:24.939319 10992 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 53201.5us
I20260709 13:04:24.942770 10844 leader_election.cc:304] T 732c4adeeff6422497aa7db4a82f535b P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47; no voters: 10ca5ee68ad64b3283b9d235352c9d23
W20260709 13:04:25.005985 10844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:45807, user_credentials={real_user=slave}} blocked reactor thread for 63322.8us
I20260709 13:04:25.006240 11022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65bf0a62f4d84ef18dd6b23e082724b8" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:25.008725 12531 consensus_queue.cc:1059] T 6f46a5102fd64b45bf1c4dff863d2b4d P 873357742d5843579556ee8166f23f47 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:04:25.009047 10844 leader_election.cc:304] T 0e5d327630e5438b878035d28efe6c0c P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:04:25.009243 11027 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f1466124dbd4f6ebb3c4d62ca48bf96" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:25.010054 10844 leader_election.cc:304] T 868e92339c2f4508af0afbd07c814d64 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:04:24.950295 11102 raft_consensus.cc:1240] T 4f9eb06ed01f4ce68be3af8a8434665b P 0172fed22b6b4890a7fdc953fa128af4 [term 5 FOLLOWER]: Rejecting Update request from peer 186b12638a87410487467df721a3cbda for earlier term 4. Current term is 5. Ops: []
I20260709 13:04:24.950513 10845 leader_election.cc:304] T f371d2565dd84667b5613848604949bb P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 186b12638a87410487467df721a3cbda; no voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47
I20260709 13:04:24.957535 12612 leader_election.cc:290] T 4f9eb06ed01f4ce68be3af8a8434665b P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 5 election: Requested vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:24.945240 12488 raft_consensus.cc:2804] T a5299b8d044f41b7b68057e63d17b159 P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:04:24.959792 10873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d270b9e5481245d9b5340a454a7cd10b" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:24.961361 12586 raft_consensus.cc:2804] T a610d1a71f52432e88fe56130f921fa2 P 186b12638a87410487467df721a3cbda [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:04:24.951153 12476 consensus_queue.cc:1048] T 30f628e3d7914d609e7f1d980a74c654 P 186b12638a87410487467df721a3cbda [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.011s
I20260709 13:04:24.951345 12441 raft_consensus.cc:2804] T 7709387f4d3d4f9992d97a0fbca39a40 P 873357742d5843579556ee8166f23f47 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 13:04:24.951594 12202 raft_consensus.cc:3060] T a977842ba4b744f4bb2febd55d286006 P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Advancing to term 5
W20260709 13:04:25.046872 10845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 96476.6us
I20260709 13:04:25.048563 12612 raft_consensus.cc:2749] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 0172fed22b6b4890a7fdc953fa128af4 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:04:25.051076 12612 consensus_queue.cc:1059] T acc241e7273d49149984968596242712 P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:04:25.054355 11021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "30f628e3d7914d609e7f1d980a74c654" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
I20260709 13:04:25.055188 11021 raft_consensus.cc:3060] T 30f628e3d7914d609e7f1d980a74c654 P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:04:24.965097 12659 raft_consensus.cc:493] T b4cd3d7dfbbf4096908ff11e9dc57141 P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:25.056977 12659 raft_consensus.cc:515] T b4cd3d7dfbbf4096908ff11e9dc57141 P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:25.058077 12450 raft_consensus.cc:3055] T d71ca3a1ba6b48478aaf62741bf3cf6a P 873357742d5843579556ee8166f23f47 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:04:25.058583 12450 raft_consensus.cc:740] T d71ca3a1ba6b48478aaf62741bf3cf6a P 873357742d5843579556ee8166f23f47 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 873357742d5843579556ee8166f23f47, State: Running, Role: LEADER
I20260709 13:04:24.967617 12534 consensus_queue.cc:260] T b4cd3d7dfbbf4096908ff11e9dc57141 P 873357742d5843579556ee8166f23f47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:24.968693 12390 raft_consensus.cc:515] T 6f46a5102fd64b45bf1c4dff863d2b4d P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:25.059545 12450 consensus_queue.cc:260] T d71ca3a1ba6b48478aaf62741bf3cf6a P 873357742d5843579556ee8166f23f47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:24.935381 12387 raft_consensus.cc:2804] T c51782cd9d064ec0949e84e24dd4de67 P 0172fed22b6b4890a7fdc953fa128af4 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:04:25.060734 12387 raft_consensus.cc:697] T c51782cd9d064ec0949e84e24dd4de67 P 0172fed22b6b4890a7fdc953fa128af4 [term 5 LEADER]: Becoming Leader. State: Replica: 0172fed22b6b4890a7fdc953fa128af4, State: Running, Role: LEADER
I20260709 13:04:24.970450 12584 raft_consensus.cc:3055] T a64b7e99d6c54934a7de6f6dffcf5e94 P 873357742d5843579556ee8166f23f47 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:04:25.061694 12660 raft_consensus.cc:2749] T f371d2565dd84667b5613848604949bb P 186b12638a87410487467df721a3cbda [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:04:24.925765 12581 raft_consensus.cc:493] T 0f9df2bc281f4866b6da5da94a0defc7 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 186b12638a87410487467df721a3cbda)
I20260709 13:04:25.063059 12581 raft_consensus.cc:515] T 0f9df2bc281f4866b6da5da94a0defc7 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:25.063685 10871 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acc241e7273d49149984968596242712" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:04:25.065393 10871 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5a5f5a608504650b48fd27f163faed6" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:24.964571 12517 leader_election.cc:290] T 30f628e3d7914d609e7f1d980a74c654 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 3 election: Requested vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:24.978621 10948 raft_consensus.cc:2393] T 836251ebc2b74f5da88b8a8b2338e2a5 P 10ca5ee68ad64b3283b9d235352c9d23 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 0172fed22b6b4890a7fdc953fa128af4 in current term 5: Already voted for candidate 10ca5ee68ad64b3283b9d235352c9d23 in this term.
W20260709 13:04:24.983001 10994 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 49368.1us
I20260709 13:04:24.974743 12541 consensus_queue.cc:260] T 87cd8f5097624c8692f1d83544d8881e P 186b12638a87410487467df721a3cbda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:25.013434 12662 raft_consensus.cc:2804] T 732c4adeeff6422497aa7db4a82f535b P 186b12638a87410487467df721a3cbda [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:04:25.069199 10874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65bf0a62f4d84ef18dd6b23e082724b8" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:25.069486 12541 raft_consensus.cc:3060] T 87cd8f5097624c8692f1d83544d8881e P 186b12638a87410487467df721a3cbda [term 5 FOLLOWER]: Advancing to term 6
W20260709 13:04:25.001415 12532 consensus_meta.cc:287] T 1e4f2241a94a41f38e481137f868b83d P 10ca5ee68ad64b3283b9d235352c9d23: Time spent flushing consensus metadata: real 0.541s	user 0.008s	sys 0.004s
I20260709 13:04:24.871799 12513 raft_consensus.cc:515] T 75492f86fde046c3900df3eac8292fd4 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:24.945618 10843 leader_election.cc:304] T f34f2bc02d85444f9096e0f5b4fe4331 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda; no voters: 0172fed22b6b4890a7fdc953fa128af4
I20260709 13:04:24.944128 12611 raft_consensus.cc:2804] T a154cc8b181f460abf4880064379948b P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:04:25.010401 12462 raft_consensus.cc:3055] T 6f46a5102fd64b45bf1c4dff863d2b4d P 873357742d5843579556ee8166f23f47 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:04:25.088439 12611 raft_consensus.cc:493] T a154cc8b181f460abf4880064379948b P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:25.015733 12487 raft_consensus.cc:2804] T 0e5d327630e5438b878035d28efe6c0c P 186b12638a87410487467df721a3cbda [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:04:25.089164 12611 raft_consensus.cc:3060] T a154cc8b181f460abf4880064379948b P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:04:25.089268 12487 raft_consensus.cc:493] T 0e5d327630e5438b878035d28efe6c0c P 186b12638a87410487467df721a3cbda [term 5 FOLLOWER]: Starting leader election (detected failure of leader 0172fed22b6b4890a7fdc953fa128af4)
I20260709 13:04:25.089886 12487 raft_consensus.cc:3060] T 0e5d327630e5438b878035d28efe6c0c P 186b12638a87410487467df721a3cbda [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:04:24.990166 11066 leader_election.cc:304] T bb7d67be500947b3aef4c4245e33671c P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47
I20260709 13:04:25.021973 11027 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32147c6d2d5a477aa38c597c7f78df5b" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:25.023012 10947 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49434 (request call id 399) took 1046 ms. Trace:
I20260709 13:04:25.092506 10947 rpcz_store.cc:276] 0709 13:04:23.976511 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:24.226923 (+250412us) service_pool.cc:224] Handling call
0709 13:04:24.287556 (+ 60633us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:24.870757 (+583201us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:25.022758 (+152001us) spinlock_profiling.cc:243] Waited 478 ms on lock 0x7b540069e1d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f4266162e01 00007f426616183e 00007f426b126419 00007f426469dbd4 00007f426469db62 00007f426469da91 00007f426469da0c 00007f426469ccc2 00007f426449d48c 00007f426449c8c6
0709 13:04:25.022944 (+   186us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1100243328}
W20260709 13:04:25.012951 12656 log.cc:927] Time spent T 525bfdbd6d934afd823f7d3ee3f9d052 P 0172fed22b6b4890a7fdc953fa128af4: Append to log took a long time: real 0.100s	user 0.000s	sys 0.002s
I20260709 13:04:25.034173 10921 leader_election.cc:304] T e96e80ba9ac34d0998395515e5c8c41a P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23; no voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47
I20260709 13:04:25.037128 10877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "30f628e3d7914d609e7f1d980a74c654" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
W20260709 13:04:25.101055 10921 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:38967, user_credentials={real_user=slave}} blocked reactor thread for 66983us
I20260709 13:04:25.035796 12649 raft_consensus.cc:2804] T 868e92339c2f4508af0afbd07c814d64 P 186b12638a87410487467df721a3cbda [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:04:25.054176 12479 consensus_queue.cc:1048] T 30f628e3d7914d609e7f1d980a74c654 P 186b12638a87410487467df721a3cbda [LEADER]: Connected to new peer: Peer: permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:04:25.036240 12586 raft_consensus.cc:493] T a610d1a71f52432e88fe56130f921fa2 P 186b12638a87410487467df721a3cbda [term 5 FOLLOWER]: Starting leader election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:24.999598 12564 leader_election.cc:290] T e5a5f5a608504650b48fd27f163faed6 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:25.000346 12591 leader_election.cc:290] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:24.977433 12500 raft_consensus.cc:3060] T d7ce47b9a92d43dd88d95bcda5038597 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:04:25.047813 10845 leader_election.cc:304] T acc241e7273d49149984968596242712 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 186b12638a87410487467df721a3cbda; no voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47
I20260709 13:04:25.072665 10994 leader_election.cc:304] T e5ba72ccd854430baadbc636dfd17136 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47; no voters: 
W20260709 13:04:25.075168 10995 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:38967, user_credentials={real_user=slave}} blocked reactor thread for 52558us
W20260709 13:04:25.074555 10992 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 63859.3us
I20260709 13:04:25.083509 12450 raft_consensus.cc:3060] T d71ca3a1ba6b48478aaf62741bf3cf6a P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:04:25.061738 12387 consensus_queue.cc:237] T c51782cd9d064ec0949e84e24dd4de67 P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:25.083665 10873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2567c2d84e4648d7ba5dc23a3df32817" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:04:25.135982 10871 raft_consensus.cc:2410] T e5a5f5a608504650b48fd27f163faed6 P 186b12638a87410487467df721a3cbda [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 10ca5ee68ad64b3283b9d235352c9d23 for term 7 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260709 13:04:25.083839 12441 raft_consensus.cc:697] T 7709387f4d3d4f9992d97a0fbca39a40 P 873357742d5843579556ee8166f23f47 [term 8 LEADER]: Becoming Leader. State: Replica: 873357742d5843579556ee8166f23f47, State: Running, Role: LEADER
I20260709 13:04:25.038771 12488 raft_consensus.cc:493] T a5299b8d044f41b7b68057e63d17b159 P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Starting leader election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:25.086707 12513 leader_election.cc:290] T 75492f86fde046c3900df3eac8292fd4 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:25.087538 12660 raft_consensus.cc:2804] T f34f2bc02d85444f9096e0f5b4fe4331 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Leader election won for term 2
W20260709 13:04:25.087786 10843 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 142271us
I20260709 13:04:25.105559 12532 raft_consensus.cc:2749] T e96e80ba9ac34d0998395515e5c8c41a P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20260709 13:04:25.091265 11066 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 101212us
I20260709 13:04:25.001937 10919 leader_election.cc:304] T ff13ae9a47c7455f962406c6d131a3ff P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23; no voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47
W20260709 13:04:25.197682 10919 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:45807, user_credentials={real_user=slave}} blocked reactor thread for 195921us
I20260709 13:04:25.014942 12585 consensus_queue.cc:1048] T 2567c2d84e4648d7ba5dc23a3df32817 P 186b12638a87410487467df721a3cbda [LEADER]: Connected to new peer: Peer: permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.061s
I20260709 13:04:25.061903 12584 raft_consensus.cc:740] T a64b7e99d6c54934a7de6f6dffcf5e94 P 873357742d5843579556ee8166f23f47 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 873357742d5843579556ee8166f23f47, State: Running, Role: LEADER
I20260709 13:04:25.146972 12649 raft_consensus.cc:697] T 868e92339c2f4508af0afbd07c814d64 P 186b12638a87410487467df721a3cbda [term 5 LEADER]: Becoming Leader. State: Replica: 186b12638a87410487467df721a3cbda, State: Running, Role: LEADER
W20260709 13:04:24.986680 10832 catalog_manager.cc:6160] T 00000000000000000000000000000000 P c669bee8ad644d0d956c148b33886b51: Tablet a977842ba4b744f4bb2febd55d286006 (table test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]) was not created within the allowed timeout. Replacing with a new tablet 290a8fadcbdd4b3dace26237d9157717
W20260709 13:04:25.144203 10994 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 71665.3us
I20260709 13:04:25.099217 12534 raft_consensus.cc:3060] T b4cd3d7dfbbf4096908ff11e9dc57141 P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:04:25.157313 11028 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75492f86fde046c3900df3eac8292fd4" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:25.168443 10843 leader_election.cc:304] T 11de93cd9b9d457489ff1cb80f976c9a P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda; no voters: 873357742d5843579556ee8166f23f47
I20260709 13:04:25.088506 12462 raft_consensus.cc:740] T 6f46a5102fd64b45bf1c4dff863d2b4d P 873357742d5843579556ee8166f23f47 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 873357742d5843579556ee8166f23f47, State: Running, Role: LEADER
I20260709 13:04:25.136292 12517 raft_consensus.cc:493] T 732c4adeeff6422497aa7db4a82f535b P 10ca5ee68ad64b3283b9d235352c9d23 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:25.167191 12488 raft_consensus.cc:3060] T a5299b8d044f41b7b68057e63d17b159 P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:04:25.146322 12586 raft_consensus.cc:3060] T a610d1a71f52432e88fe56130f921fa2 P 186b12638a87410487467df721a3cbda [term 5 FOLLOWER]: Advancing to term 6
W20260709 13:04:25.191488 11069 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:38967, user_credentials={real_user=slave}} blocked reactor thread for 284139us
I20260709 13:04:25.199647 10919 leader_election.cc:304] T e89e640169e14e2d98f968f76027daa7 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23; no voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47
I20260709 13:04:25.173517 12513 raft_consensus.cc:3055] T acc241e7273d49149984968596242712 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:04:25.239745 10919 leader_election.cc:304] T 65bf0a62f4d84ef18dd6b23e082724b8 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23; no voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47
I20260709 13:04:25.204344 12564 raft_consensus.cc:2749] T ff13ae9a47c7455f962406c6d131a3ff P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:04:25.246819 12441 consensus_queue.cc:237] T 7709387f4d3d4f9992d97a0fbca39a40 P 873357742d5843579556ee8166f23f47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:25.247676 12424 raft_consensus.cc:2749] T d371c5e27bd843469d2e2e453734cb4a P 873357742d5843579556ee8166f23f47 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:04:25.157572 12053 raft_consensus.cc:2804] T e5ba72ccd854430baadbc636dfd17136 P 873357742d5843579556ee8166f23f47 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:04:25.069319 12662 raft_consensus.cc:697] T 732c4adeeff6422497aa7db4a82f535b P 186b12638a87410487467df721a3cbda [term 5 LEADER]: Becoming Leader. State: Replica: 186b12638a87410487467df721a3cbda, State: Running, Role: LEADER
I20260709 13:04:25.158217 12578 leader_election.cc:290] T 2567c2d84e4648d7ba5dc23a3df32817 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 4 election: Requested vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
W20260709 13:04:25.235457 10843 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 67139.4us
I20260709 13:04:25.270686 10843 leader_election.cc:304] T 836251ebc2b74f5da88b8a8b2338e2a5 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda; no voters: 
I20260709 13:04:25.167254 12298 consensus_queue.cc:1059] T e96e80ba9ac34d0998395515e5c8c41a P 873357742d5843579556ee8166f23f47 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.069s
I20260709 13:04:25.259761 12462 consensus_queue.cc:260] T 6f46a5102fd64b45bf1c4dff863d2b4d P 873357742d5843579556ee8166f23f47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:25.263696 12202 raft_consensus.cc:515] T a977842ba4b744f4bb2febd55d286006 P 873357742d5843579556ee8166f23f47 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:25.241588 12513 raft_consensus.cc:740] T acc241e7273d49149984968596242712 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0172fed22b6b4890a7fdc953fa128af4, State: Running, Role: LEADER
I20260709 13:04:25.264513 12672 raft_consensus.cc:2749] T bb7d67be500947b3aef4c4245e33671c P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260709 13:04:25.272396 12525 consensus_meta.cc:287] T f34f2bc02d85444f9096e0f5b4fe4331 P 10ca5ee68ad64b3283b9d235352c9d23: Time spent flushing consensus metadata: real 0.574s	user 0.003s	sys 0.009s
I20260709 13:04:25.277014 10843 leader_election.cc:304] T 63d44f6ba6434ab2870fe3190f4d7c2f P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 186b12638a87410487467df721a3cbda; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47
W20260709 13:04:25.267810 12550 consensus_meta.cc:287] T d7ce47b9a92d43dd88d95bcda5038597 P 10ca5ee68ad64b3283b9d235352c9d23: Time spent flushing consensus metadata: real 0.615s	user 0.008s	sys 0.006s
I20260709 13:04:25.238680 11026 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2567c2d84e4648d7ba5dc23a3df32817" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
I20260709 13:04:25.265444 12591 raft_consensus.cc:2749] T e89e640169e14e2d98f968f76027daa7 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260709 13:04:25.276319 12570 consensus_meta.cc:287] T 72b111ad7b5f4753ab1d2b5ee1114840 P 0172fed22b6b4890a7fdc953fa128af4: Time spent flushing consensus metadata: real 0.842s	user 0.013s	sys 0.002s
I20260709 13:04:25.273078 12053 raft_consensus.cc:493] T e5ba72ccd854430baadbc636dfd17136 P 873357742d5843579556ee8166f23f47 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:04:25.302397 10919 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:45807, user_credentials={real_user=slave}} blocked reactor thread for 62774.7us
W20260709 13:04:25.305608 10995 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:38967, user_credentials={real_user=slave}} blocked reactor thread for 141776us
I20260709 13:04:25.237694 12517 raft_consensus.cc:515] T 732c4adeeff6422497aa7db4a82f535b P 10ca5ee68ad64b3283b9d235352c9d23 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:25.334865 12581 leader_election.cc:290] T 0f9df2bc281f4866b6da5da94a0defc7 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:25.336325 12517 leader_election.cc:290] T 732c4adeeff6422497aa7db4a82f535b P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:25.336812 12424 raft_consensus.cc:3055] T e96e80ba9ac34d0998395515e5c8c41a P 873357742d5843579556ee8166f23f47 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:04:25.337246 12424 raft_consensus.cc:740] T e96e80ba9ac34d0998395515e5c8c41a P 873357742d5843579556ee8166f23f47 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 873357742d5843579556ee8166f23f47, State: Running, Role: LEADER
I20260709 13:04:25.337636 12585 raft_consensus.cc:2804] T 11de93cd9b9d457489ff1cb80f976c9a P 186b12638a87410487467df721a3cbda [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:04:25.338156 12585 raft_consensus.cc:493] T 11de93cd9b9d457489ff1cb80f976c9a P 186b12638a87410487467df721a3cbda [term 5 FOLLOWER]: Starting leader election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:25.311993 11026 raft_consensus.cc:3060] T 2567c2d84e4648d7ba5dc23a3df32817 P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:04:25.296207 12525 raft_consensus.cc:515] T f34f2bc02d85444f9096e0f5b4fe4331 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:25.317370 12584 consensus_queue.cc:260] T a64b7e99d6c54934a7de6f6dffcf5e94 P 873357742d5843579556ee8166f23f47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:25.343609 10953 raft_consensus.cc:2393] T f34f2bc02d85444f9096e0f5b4fe4331 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0172fed22b6b4890a7fdc953fa128af4 in current term 3: Already voted for candidate 10ca5ee68ad64b3283b9d235352c9d23 in this term.
I20260709 13:04:25.317148 11069 leader_election.cc:304] T a610d1a71f52432e88fe56130f921fa2 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4; no voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47
I20260709 13:04:25.316262 12053 raft_consensus.cc:3060] T e5ba72ccd854430baadbc636dfd17136 P 873357742d5843579556ee8166f23f47 [term 6 FOLLOWER]: Advancing to term 7
W20260709 13:04:25.303123 12558 consensus_meta.cc:287] T e5ba72ccd854430baadbc636dfd17136 P 10ca5ee68ad64b3283b9d235352c9d23: Time spent flushing consensus metadata: real 0.866s	user 0.008s	sys 0.007s
I20260709 13:04:25.302845 12578 raft_consensus.cc:2749] T 65bf0a62f4d84ef18dd6b23e082724b8 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:04:25.322396 12662 consensus_queue.cc:237] T 732c4adeeff6422497aa7db4a82f535b P 186b12638a87410487467df721a3cbda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:25.280498 12659 leader_election.cc:290] T b4cd3d7dfbbf4096908ff11e9dc57141 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:25.326566 12649 consensus_queue.cc:237] T 868e92339c2f4508af0afbd07c814d64 P 186b12638a87410487467df721a3cbda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:25.328452 12513 consensus_queue.cc:260] T acc241e7273d49149984968596242712 P 0172fed22b6b4890a7fdc953fa128af4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:25.338581 12585 raft_consensus.cc:3060] T 11de93cd9b9d457489ff1cb80f976c9a P 186b12638a87410487467df721a3cbda [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:04:25.333582 12394 raft_consensus.cc:515] T d270b9e5481245d9b5340a454a7cd10b P 0172fed22b6b4890a7fdc953fa128af4 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:25.202088 11066 leader_election.cc:304] T ff13ae9a47c7455f962406c6d131a3ff P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47
W20260709 13:04:25.360204 10992 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 188649us
I20260709 13:04:25.317804 12462 raft_consensus.cc:3060] T 6f46a5102fd64b45bf1c4dff863d2b4d P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:04:25.343971 12584 raft_consensus.cc:3060] T a64b7e99d6c54934a7de6f6dffcf5e94 P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:04:25.297977 12550 raft_consensus.cc:515] T d7ce47b9a92d43dd88d95bcda5038597 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:25.341473 12479 raft_consensus.cc:2749] T 63d44f6ba6434ab2870fe3190f4d7c2f P 186b12638a87410487467df721a3cbda [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:04:25.384682 10952 raft_consensus.cc:1240] T a64b7e99d6c54934a7de6f6dffcf5e94 P 10ca5ee68ad64b3283b9d235352c9d23 [term 5 FOLLOWER]: Rejecting Update request from peer 873357742d5843579556ee8166f23f47 for earlier term 4. Current term is 5. Ops: []
W20260709 13:04:25.385056 11066 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 183127us
I20260709 13:04:25.382494 11029 raft_consensus.cc:1240] T a977842ba4b744f4bb2febd55d286006 P 873357742d5843579556ee8166f23f47 [term 5 FOLLOWER]: Rejecting Update request from peer 10ca5ee68ad64b3283b9d235352c9d23 for earlier term 4. Current term is 5. Ops: []
I20260709 13:04:25.338382 12488 raft_consensus.cc:515] T a5299b8d044f41b7b68057e63d17b159 P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
W20260709 13:04:25.375041 12397 consensus_meta.cc:287] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 186b12638a87410487467df721a3cbda: Time spent flushing consensus metadata: real 0.639s	user 0.011s	sys 0.003s
I20260709 13:04:25.340655 12250 raft_consensus.cc:2804] T 836251ebc2b74f5da88b8a8b2338e2a5 P 186b12638a87410487467df721a3cbda [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260709 13:04:25.408118 12548 consensus_meta.cc:287] T 525bfdbd6d934afd823f7d3ee3f9d052 P 873357742d5843579556ee8166f23f47: Time spent flushing consensus metadata: real 0.505s	user 0.011s	sys 0.002s
I20260709 13:04:25.408677 12548 raft_consensus.cc:515] T 525bfdbd6d934afd823f7d3ee3f9d052 P 873357742d5843579556ee8166f23f47 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:25.409839 11025 raft_consensus.cc:1240] T 525bfdbd6d934afd823f7d3ee3f9d052 P 873357742d5843579556ee8166f23f47 [term 6 FOLLOWER]: Rejecting Update request from peer 10ca5ee68ad64b3283b9d235352c9d23 for earlier term 5. Current term is 6. Ops: []
I20260709 13:04:25.411809 12548 leader_election.cc:290] T 525bfdbd6d934afd823f7d3ee3f9d052 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 6 election: Requested vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:25.387398 12672 raft_consensus.cc:2749] T ff13ae9a47c7455f962406c6d131a3ff P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:04:25.414288 10879 raft_consensus.cc:2468] T d7ce47b9a92d43dd88d95bcda5038597 P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 3.
I20260709 13:04:25.381012 12677 raft_consensus.cc:2749] T acc241e7273d49149984968596242712 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:04:25.384356 12659 raft_consensus.cc:2749] T a610d1a71f52432e88fe56130f921fa2 P 0172fed22b6b4890a7fdc953fa128af4 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:04:25.417207 10955 raft_consensus.cc:2368] T d7ce47b9a92d43dd88d95bcda5038597 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0172fed22b6b4890a7fdc953fa128af4 for earlier term 3. Current term is 4.
I20260709 13:04:25.363765 12558 raft_consensus.cc:515] T e5ba72ccd854430baadbc636dfd17136 P 10ca5ee68ad64b3283b9d235352c9d23 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:25.421906 10880 raft_consensus.cc:1240] T a5299b8d044f41b7b68057e63d17b159 P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Rejecting Update request from peer 873357742d5843579556ee8166f23f47 for earlier term 3. Current term is 4. Ops: []
I20260709 13:04:25.402284 12513 raft_consensus.cc:3060] T acc241e7273d49149984968596242712 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:04:25.433014 12390 leader_election.cc:290] T 6f46a5102fd64b45bf1c4dff863d2b4d P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:25.422072 12660 raft_consensus.cc:697] T f34f2bc02d85444f9096e0f5b4fe4331 P 186b12638a87410487467df721a3cbda [term 2 LEADER]: Becoming Leader. State: Replica: 186b12638a87410487467df721a3cbda, State: Running, Role: LEADER
W20260709 13:04:25.381636 11069 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:38967, user_credentials={real_user=slave}} blocked reactor thread for 64611.2us
I20260709 13:04:25.404381 12250 raft_consensus.cc:493] T 836251ebc2b74f5da88b8a8b2338e2a5 P 186b12638a87410487467df721a3cbda [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:04:25.439143 10992 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:25.338164 12424 consensus_queue.cc:260] T e96e80ba9ac34d0998395515e5c8c41a P 873357742d5843579556ee8166f23f47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:25.424923 12517 consensus_queue.cc:1059] T a977842ba4b744f4bb2febd55d286006 P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:04:25.418690 12654 raft_consensus.cc:2749] T 5d439326f23747eaab3c1e7da2d6b785 P 0172fed22b6b4890a7fdc953fa128af4 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:04:25.393954 11029 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "732c4adeeff6422497aa7db4a82f535b" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:25.449841 11029 raft_consensus.cc:2468] T 732c4adeeff6422497aa7db4a82f535b P 873357742d5843579556ee8166f23f47 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 5.
W20260709 13:04:25.434854 11067 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:25.451241 11097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a64b7e99d6c54934a7de6f6dffcf5e94" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:04:25.451346 10954 raft_consensus.cc:2393] T e5ba72ccd854430baadbc636dfd17136 P 10ca5ee68ad64b3283b9d235352c9d23 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 873357742d5843579556ee8166f23f47 in current term 7: Already voted for candidate 10ca5ee68ad64b3283b9d235352c9d23 in this term.
W20260709 13:04:25.389065 10844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:45807, user_credentials={real_user=slave}} blocked reactor thread for 240517us
W20260709 13:04:25.435750 10995 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:38967, user_credentials={real_user=slave}} blocked reactor thread for 59904.2us
I20260709 13:04:25.437578 11069 leader_election.cc:304] T b67eff0634e540eab91cf8cee795969f P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda
I20260709 13:04:25.397486 10920 leader_election.cc:304] T 5de5f0581e3b41dea729363d6ca563bb P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23; no voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda
W20260709 13:04:25.422083 10845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 374384us
I20260709 13:04:25.387358 12569 raft_consensus.cc:515] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 873357742d5843579556ee8166f23f47 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:25.423313 12202 leader_election.cc:290] T a977842ba4b744f4bb2febd55d286006 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 5 election: Requested vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:25.457603 11097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d71ca3a1ba6b48478aaf62741bf3cf6a" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:04:25.477159 11097 raft_consensus.cc:3060] T d71ca3a1ba6b48478aaf62741bf3cf6a P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:04:25.458925 11094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f46a5102fd64b45bf1c4dff863d2b4d" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:04:25.478319 11094 raft_consensus.cc:3060] T 6f46a5102fd64b45bf1c4dff863d2b4d P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:04:25.445562 12250 raft_consensus.cc:3060] T 836251ebc2b74f5da88b8a8b2338e2a5 P 186b12638a87410487467df721a3cbda [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:04:25.466954 10919 leader_election.cc:304] T 732c4adeeff6422497aa7db4a82f535b P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47; no voters: 
W20260709 13:04:25.471534 10920 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 74208.3us
I20260709 13:04:25.449298 12672 raft_consensus.cc:493] T 0f9df2bc281f4866b6da5da94a0defc7 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 186b12638a87410487467df721a3cbda)
I20260709 13:04:25.472358 11069 leader_election.cc:304] T 87cd8f5097624c8692f1d83544d8881e P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda
I20260709 13:04:25.480878 10845 leader_election.cc:304] T d371c5e27bd843469d2e2e453734cb4a P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 186b12638a87410487467df721a3cbda; no voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47
I20260709 13:04:25.483530 10949 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75492f86fde046c3900df3eac8292fd4" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:25.484830 10952 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32147c6d2d5a477aa38c597c7f78df5b" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:25.488045 10946 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "525bfdbd6d934afd823f7d3ee3f9d052" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:04:25.455497 12591 raft_consensus.cc:493] T b67eff0634e540eab91cf8cee795969f P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 186b12638a87410487467df721a3cbda)
I20260709 13:04:25.489465 11069 leader_election.cc:304] T 7757ecbf402e47bfbc93d0447741e26c P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4; no voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47
I20260709 13:04:25.483958 10872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f9df2bc281f4866b6da5da94a0defc7" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
W20260709 13:04:25.495749 10993 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:25.491866 11102 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5a5f5a608504650b48fd27f163faed6" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:25.497404 12659 raft_consensus.cc:2749] T 87cd8f5097624c8692f1d83544d8881e P 0172fed22b6b4890a7fdc953fa128af4 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 13:04:25.501963 10875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "732c4adeeff6422497aa7db4a82f535b" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:25.502463 11028 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f46a5102fd64b45bf1c4dff863d2b4d" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:25.479434 12517 raft_consensus.cc:2749] T 5de5f0581e3b41dea729363d6ca563bb P 10ca5ee68ad64b3283b9d235352c9d23 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:04:25.502686 10947 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4cd3d7dfbbf4096908ff11e9dc57141" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:25.491942 12564 raft_consensus.cc:2804] T 732c4adeeff6422497aa7db4a82f535b P 10ca5ee68ad64b3283b9d235352c9d23 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:04:25.512789 12564 raft_consensus.cc:493] T 732c4adeeff6422497aa7db4a82f535b P 10ca5ee68ad64b3283b9d235352c9d23 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:25.513195 12564 raft_consensus.cc:3060] T 732c4adeeff6422497aa7db4a82f535b P 10ca5ee68ad64b3283b9d235352c9d23 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:04:25.503667 11024 raft_consensus.cc:1240] T d371c5e27bd843469d2e2e453734cb4a P 873357742d5843579556ee8166f23f47 [term 5 FOLLOWER]: Rejecting Update request from peer 0172fed22b6b4890a7fdc953fa128af4 for earlier term 4. Current term is 5. Ops: []
I20260709 13:04:25.504822 11029 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4cd3d7dfbbf4096908ff11e9dc57141" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:25.508275 11102 raft_consensus.cc:2410] T e5a5f5a608504650b48fd27f163faed6 P 0172fed22b6b4890a7fdc953fa128af4 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 10ca5ee68ad64b3283b9d235352c9d23 for term 7 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260709 13:04:25.504571 12672 raft_consensus.cc:515] T 0f9df2bc281f4866b6da5da94a0defc7 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:25.509093 11069 leader_election.cc:304] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda
I20260709 13:04:25.494277 10950 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a977842ba4b744f4bb2febd55d286006" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:04:25.512444 10947 raft_consensus.cc:2393] T b4cd3d7dfbbf4096908ff11e9dc57141 P 10ca5ee68ad64b3283b9d235352c9d23 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0172fed22b6b4890a7fdc953fa128af4 in current term 5: Already voted for candidate 10ca5ee68ad64b3283b9d235352c9d23 in this term.
I20260709 13:04:25.480026 12387 raft_consensus.cc:2749] T b67eff0634e540eab91cf8cee795969f P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:04:25.523757 11101 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f1466124dbd4f6ebb3c4d62ca48bf96" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:25.527500 11101 raft_consensus.cc:2468] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 0172fed22b6b4890a7fdc953fa128af4 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 7.
I20260709 13:04:25.520279 12669 raft_consensus.cc:2749] T 7757ecbf402e47bfbc93d0447741e26c P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:04:25.518296 12672 leader_election.cc:290] T 0f9df2bc281f4866b6da5da94a0defc7 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:25.547320 11069 leader_election.cc:304] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4; no voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47
I20260709 13:04:25.590343 10955 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49424 (request call id 277) took 1152 ms. Trace:
I20260709 13:04:25.590950 11069 leader_election.cc:304] T f371d2565dd84667b5613848604949bb P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4; no voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47
I20260709 13:04:25.525933 12591 raft_consensus.cc:515] T b67eff0634e540eab91cf8cee795969f P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:25.539288 12679 raft_consensus.cc:2749] T d371c5e27bd843469d2e2e453734cb4a P 186b12638a87410487467df721a3cbda [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 13:04:25.560312 12558 leader_election.cc:290] T e5ba72ccd854430baadbc636dfd17136 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 7 election: Requested vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:25.488658 12661 consensus_queue.cc:1059] T 8a69f32c53f24a50b7852d143665d7b4 P 186b12638a87410487467df721a3cbda [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.051s
I20260709 13:04:25.534634 12660 consensus_queue.cc:237] T f34f2bc02d85444f9096e0f5b4fe4331 P 186b12638a87410487467df721a3cbda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:25.529618 10948 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f46a5102fd64b45bf1c4dff863d2b4d" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:25.566573 11029 raft_consensus.cc:2468] T b4cd3d7dfbbf4096908ff11e9dc57141 P 873357742d5843579556ee8166f23f47 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 5.
I20260709 13:04:25.529394 12394 leader_election.cc:290] T d270b9e5481245d9b5340a454a7cd10b P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 7 election: Requested vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
W20260709 13:04:25.569993 11021 consensus_meta.cc:287] T 30f628e3d7914d609e7f1d980a74c654 P 873357742d5843579556ee8166f23f47: Time spent flushing consensus metadata: real 0.514s	user 0.010s	sys 0.002s
I20260709 13:04:25.571165 12611 raft_consensus.cc:515] T a154cc8b181f460abf4880064379948b P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:25.525498 12570 raft_consensus.cc:2749] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 0172fed22b6b4890a7fdc953fa128af4 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 13:04:25.525724 12517 raft_consensus.cc:493] T 8a69f32c53f24a50b7852d143665d7b4 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 186b12638a87410487467df721a3cbda)
I20260709 13:04:25.519641 11096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d270b9e5481245d9b5340a454a7cd10b" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:25.640942 12679 raft_consensus.cc:3055] T 8a69f32c53f24a50b7852d143665d7b4 P 186b12638a87410487467df721a3cbda [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:04:25.573004 12550 leader_election.cc:290] T d7ce47b9a92d43dd88d95bcda5038597 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 4 election: Requested vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:25.641391 12679 raft_consensus.cc:740] T 8a69f32c53f24a50b7852d143665d7b4 P 186b12638a87410487467df721a3cbda [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 186b12638a87410487467df721a3cbda, State: Running, Role: LEADER
I20260709 13:04:25.573393 12053 raft_consensus.cc:515] T e5ba72ccd854430baadbc636dfd17136 P 873357742d5843579556ee8166f23f47 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:25.642263 12517 raft_consensus.cc:515] T 8a69f32c53f24a50b7852d143665d7b4 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:25.644098 12053 leader_election.cc:290] T e5ba72ccd854430baadbc636dfd17136 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 7 election: Requested vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:25.578469 12581 raft_consensus.cc:493] T 1e4f2241a94a41f38e481137f868b83d P 10ca5ee68ad64b3283b9d235352c9d23 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:25.644616 12517 leader_election.cc:290] T 8a69f32c53f24a50b7852d143665d7b4 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:25.644943 12581 raft_consensus.cc:515] T 1e4f2241a94a41f38e481137f868b83d P 10ca5ee68ad64b3283b9d235352c9d23 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:25.578503 12534 raft_consensus.cc:493] T 8c160ddfe9b34bdaa3835d0915c1e06f P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 186b12638a87410487467df721a3cbda)
I20260709 13:04:25.646771 12534 raft_consensus.cc:515] T 8c160ddfe9b34bdaa3835d0915c1e06f P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:25.627902 12672 raft_consensus.cc:2749] T f371d2565dd84667b5613848604949bb P 0172fed22b6b4890a7fdc953fa128af4 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:04:25.579046 12488 leader_election.cc:290] T a5299b8d044f41b7b68057e63d17b159 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 4 election: Requested vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:25.529021 12390 consensus_queue.cc:1059] T d371c5e27bd843469d2e2e453734cb4a P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.051s
I20260709 13:04:25.540423 12487 raft_consensus.cc:515] T 0e5d327630e5438b878035d28efe6c0c P 186b12638a87410487467df721a3cbda [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:25.581104 11101 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f34f2bc02d85444f9096e0f5b4fe4331" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:04:25.657418 11101 raft_consensus.cc:3060] T f34f2bc02d85444f9096e0f5b4fe4331 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:04:25.583477 11103 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f9df2bc281f4866b6da5da94a0defc7" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:25.585192 10872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555217d45d9b4ee492dbb3ddaf0f7c02" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:04:25.587255 11026 raft_consensus.cc:2468] T 2567c2d84e4648d7ba5dc23a3df32817 P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 4.
I20260709 13:04:25.660976 11026 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5299b8d044f41b7b68057e63d17b159" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
I20260709 13:04:25.588222 11097 raft_consensus.cc:2468] T d71ca3a1ba6b48478aaf62741bf3cf6a P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 4.
    @     0x7f426329a7f5 testing::internal::UnitTestImpl::RunAllTests() at ??:0
I20260709 13:04:25.590790 10955 rpcz_store.cc:276] 0709 13:04:24.438111 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:24.438589 (+   478us) service_pool.cc:224] Handling call
0709 13:04:25.590128 (+1151539us) spinlock_profiling.cc:243] Waited 978 ms on lock 0x7b54001e23d4. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:04:25.590270 (+   142us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2248417664}
I20260709 13:04:25.548290 11066 leader_election.cc:304] T 75492f86fde046c3900df3eac8292fd4 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47
W20260709 13:04:25.668299 11066 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 120117us
I20260709 13:04:25.669137 11066 leader_election.cc:304] T 32147c6d2d5a477aa38c597c7f78df5b P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47
I20260709 13:04:25.669391 11029 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e5d327630e5438b878035d28efe6c0c" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
W20260709 13:04:25.601572 10921 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:38967, user_credentials={real_user=slave}} blocked reactor thread for 65026.8us
I20260709 13:04:25.638640 11022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f9eb06ed01f4ce68be3af8a8434665b" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
I20260709 13:04:25.615087 10950 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d270b9e5481245d9b5340a454a7cd10b" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:04:25.615145 10920 leader_election.cc:304] T a64b7e99d6c54934a7de6f6dffcf5e94 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23; no voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47
I20260709 13:04:25.618062 11069 leader_election.cc:304] T a1ec2b50c6ad4f7c8033476208abb26d P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4; no voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47
I20260709 13:04:25.663444 11027 raft_consensus.cc:1240] T 2567c2d84e4648d7ba5dc23a3df32817 P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Rejecting Update request from peer 186b12638a87410487467df721a3cbda for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260709 13:04:25.634884 11021 raft_consensus.cc:2468] T 30f628e3d7914d609e7f1d980a74c654 P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 3.
I20260709 13:04:25.626267 10947 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f9df2bc281f4866b6da5da94a0defc7" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:25.578572 12569 leader_election.cc:290] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 6 election: Requested vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:25.624099 12387 raft_consensus.cc:2749] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 0172fed22b6b4890a7fdc953fa128af4 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:04:25.658610 11103 raft_consensus.cc:2468] T 0f9df2bc281f4866b6da5da94a0defc7 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 2.
I20260709 13:04:25.662098 12487 leader_election.cc:290] T 0e5d327630e5438b878035d28efe6c0c P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 6 election: Requested vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:25.641395 11096 raft_consensus.cc:2393] T d270b9e5481245d9b5340a454a7cd10b P 0172fed22b6b4890a7fdc953fa128af4 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 10ca5ee68ad64b3283b9d235352c9d23 in current term 7: Already voted for candidate 0172fed22b6b4890a7fdc953fa128af4 in this term.
W20260709 13:04:25.603024 10845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 122227us
W20260709 13:04:25.624115 10843 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 347471us
I20260709 13:04:25.665721 10919 leader_election.cc:304] T 2567c2d84e4648d7ba5dc23a3df32817 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47; no voters: 186b12638a87410487467df721a3cbda
I20260709 13:04:25.675428 10871 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b67eff0634e540eab91cf8cee795969f" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:25.662397 11026 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5ba72ccd854430baadbc636dfd17136" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
I20260709 13:04:25.667218 11067 leader_election.cc:304] T b4cd3d7dfbbf4096908ff11e9dc57141 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:04:25.623190 12250 raft_consensus.cc:515] T 836251ebc2b74f5da88b8a8b2338e2a5 P 186b12638a87410487467df721a3cbda [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
W20260709 13:04:25.675377 10920 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 60378.4us
I20260709 13:04:25.663242 11025 raft_consensus.cc:1240] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 873357742d5843579556ee8166f23f47 [term 6 FOLLOWER]: Rejecting Update request from peer 186b12638a87410487467df721a3cbda for earlier term 5. Current term is 6. Ops: []
W20260709 13:04:25.679564 11069 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:38967, user_credentials={real_user=slave}} blocked reactor thread for 61640us
I20260709 13:04:25.684527 11024 raft_consensus.cc:1240] T 30f628e3d7914d609e7f1d980a74c654 P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Rejecting Update request from peer 186b12638a87410487467df721a3cbda for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260709 13:04:25.625083 12585 raft_consensus.cc:515] T 11de93cd9b9d457489ff1cb80f976c9a P 186b12638a87410487467df721a3cbda [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:25.676086 10878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a69f32c53f24a50b7852d143665d7b4" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:25.679121 12691 raft_consensus.cc:2749] T 32147c6d2d5a477aa38c597c7f78df5b P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:04:25.700819 12558 raft_consensus.cc:493] T 87cd8f5097624c8692f1d83544d8881e P 10ca5ee68ad64b3283b9d235352c9d23 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 186b12638a87410487467df721a3cbda)
I20260709 13:04:25.709070 11026 raft_consensus.cc:2393] T e5ba72ccd854430baadbc636dfd17136 P 873357742d5843579556ee8166f23f47 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 10ca5ee68ad64b3283b9d235352c9d23 in current term 7: Already voted for candidate 873357742d5843579556ee8166f23f47 in this term.
I20260709 13:04:25.708930 12558 raft_consensus.cc:515] T 87cd8f5097624c8692f1d83544d8881e P 10ca5ee68ad64b3283b9d235352c9d23 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
W20260709 13:04:25.709532 10919 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:45807, user_credentials={real_user=slave}} blocked reactor thread for 43930.6us
I20260709 13:04:25.690814 12669 consensus_queue.cc:1059] T d371c5e27bd843469d2e2e453734cb4a P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 0, Time since last communication: 0.019s
I20260709 13:04:25.710675 12644 raft_consensus.cc:2804] T 2567c2d84e4648d7ba5dc23a3df32817 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:04:25.666508 12534 leader_election.cc:290] T 8c160ddfe9b34bdaa3835d0915c1e06f P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:25.646771 12591 leader_election.cc:290] T b67eff0634e540eab91cf8cee795969f P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:25.632318 12586 raft_consensus.cc:515] T a610d1a71f52432e88fe56130f921fa2 P 186b12638a87410487467df721a3cbda [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:25.689730 10875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f34f2bc02d85444f9096e0f5b4fe4331" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:04:25.672175 10948 raft_consensus.cc:2393] T 6f46a5102fd64b45bf1c4dff863d2b4d P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0172fed22b6b4890a7fdc953fa128af4 in current term 3: Already voted for candidate 10ca5ee68ad64b3283b9d235352c9d23 in this term.
I20260709 13:04:25.692020 10876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d7ce47b9a92d43dd88d95bcda5038597" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:04:25.692209 10873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e4f2241a94a41f38e481137f868b83d" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:25.679030 12517 raft_consensus.cc:2749] T a64b7e99d6c54934a7de6f6dffcf5e94 P 10ca5ee68ad64b3283b9d235352c9d23 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 13:04:25.706730 11028 raft_consensus.cc:2468] T 6f46a5102fd64b45bf1c4dff863d2b4d P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 3.
W20260709 13:04:25.711962 11067 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:45807, user_credentials={real_user=slave}} blocked reactor thread for 44856.7us
I20260709 13:04:25.712507 11069 leader_election.cc:304] T e5a5f5a608504650b48fd27f163faed6 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda; no voters: 10ca5ee68ad64b3283b9d235352c9d23
I20260709 13:04:25.722465 11069 leader_election.cc:400] T 868e92339c2f4508af0afbd07c814d64 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 4 pre-election: Vote denied by peer 186b12638a87410487467df721a3cbda (127.0.107.193:38967) with higher term. Message: Invalid argument: T 868e92339c2f4508af0afbd07c814d64 P 186b12638a87410487467df721a3cbda [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0172fed22b6b4890a7fdc953fa128af4 for earlier term 4. Current term is 5.
I20260709 13:04:25.722761 11069 leader_election.cc:403] T 868e92339c2f4508af0afbd07c814d64 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:04:25.723884 11069 leader_election.cc:304] T 5de5f0581e3b41dea729363d6ca563bb P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda
I20260709 13:04:25.725126 11069 leader_election.cc:304] T 9a1716c0adf34bee8a4e99341ba5788f P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4; no voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47
I20260709 13:04:25.725813 11101 raft_consensus.cc:2468] T f34f2bc02d85444f9096e0f5b4fe4331 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 3.
I20260709 13:04:25.688742 10946 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5ba72ccd854430baadbc636dfd17136" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:04:25.697845 10845 leader_election.cc:304] T c31881534c5e4f0089c82057c1746652 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 186b12638a87410487467df721a3cbda; no voters: 0172fed22b6b4890a7fdc953fa128af4, 10ca5ee68ad64b3283b9d235352c9d23
I20260709 13:04:25.673386 12672 raft_consensus.cc:2749] T 75492f86fde046c3900df3eac8292fd4 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:04:25.664615 12525 leader_election.cc:290] T f34f2bc02d85444f9096e0f5b4fe4331 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 3 election: Requested vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:25.683030 12394 raft_consensus.cc:2749] T a1ec2b50c6ad4f7c8033476208abb26d P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:04:25.718010 10873 raft_consensus.cc:2468] T 1e4f2241a94a41f38e481137f868b83d P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 4.
I20260709 13:04:25.666002 12581 leader_election.cc:290] T 1e4f2241a94a41f38e481137f868b83d P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:25.736528 11025 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e4f2241a94a41f38e481137f868b83d" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:25.729429 11069 leader_election.cc:304] T 8a69f32c53f24a50b7852d143665d7b4 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda
I20260709 13:04:25.737812 10919 leader_election.cc:304] T 30f628e3d7914d609e7f1d980a74c654 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47; no voters: 186b12638a87410487467df721a3cbda
I20260709 13:04:25.699919 12611 leader_election.cc:290] T a154cc8b181f460abf4880064379948b P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 3 election: Requested vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:25.741575 10877 raft_consensus.cc:1240] T a610d1a71f52432e88fe56130f921fa2 P 186b12638a87410487467df721a3cbda [term 6 FOLLOWER]: Rejecting Update request from peer 873357742d5843579556ee8166f23f47 for earlier term 5. Current term is 6. Ops: []
I20260709 13:04:25.717788 12654 raft_consensus.cc:2804] T b4cd3d7dfbbf4096908ff11e9dc57141 P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:04:25.746224 12654 raft_consensus.cc:493] T b4cd3d7dfbbf4096908ff11e9dc57141 P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:25.746726 12654 raft_consensus.cc:3060] T b4cd3d7dfbbf4096908ff11e9dc57141 P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:04:25.742856 11025 raft_consensus.cc:2468] T 1e4f2241a94a41f38e481137f868b83d P 873357742d5843579556ee8166f23f47 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 5.
I20260709 13:04:25.731819 12659 raft_consensus.cc:2749] T 5de5f0581e3b41dea729363d6ca563bb P 0172fed22b6b4890a7fdc953fa128af4 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:04:25.737490 12644 raft_consensus.cc:697] T 2567c2d84e4648d7ba5dc23a3df32817 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 LEADER]: Becoming Leader. State: Replica: 10ca5ee68ad64b3283b9d235352c9d23, State: Running, Role: LEADER
I20260709 13:04:25.736797 12387 raft_consensus.cc:2804] T e5a5f5a608504650b48fd27f163faed6 P 0172fed22b6b4890a7fdc953fa128af4 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:04:25.731058 12691 raft_consensus.cc:2749] T 9a1716c0adf34bee8a4e99341ba5788f P 0172fed22b6b4890a7fdc953fa128af4 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 13:04:25.755986 10875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c160ddfe9b34bdaa3835d0915c1e06f" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:25.762192 11094 raft_consensus.cc:2468] T 6f46a5102fd64b45bf1c4dff863d2b4d P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 3.
I20260709 13:04:25.742707 12390 raft_consensus.cc:2749] T 868e92339c2f4508af0afbd07c814d64 P 0172fed22b6b4890a7fdc953fa128af4 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer 186b12638a87410487467df721a3cbda (127.0.107.193:38967) with higher term. Message: Invalid argument: T 868e92339c2f4508af0afbd07c814d64 P 186b12638a87410487467df721a3cbda [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0172fed22b6b4890a7fdc953fa128af4 for earlier term 4. Current term is 5.
I20260709 13:04:25.756593 12387 raft_consensus.cc:493] T e5a5f5a608504650b48fd27f163faed6 P 0172fed22b6b4890a7fdc953fa128af4 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 186b12638a87410487467df721a3cbda)
I20260709 13:04:25.753790 12672 raft_consensus.cc:2749] T 8a69f32c53f24a50b7852d143665d7b4 P 0172fed22b6b4890a7fdc953fa128af4 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 13:04:25.732929 12550 consensus_queue.cc:1059] T 525bfdbd6d934afd823f7d3ee3f9d052 P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.043s
I20260709 13:04:25.752868 11021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a154cc8b181f460abf4880064379948b" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
I20260709 13:04:25.750353 12703 raft_consensus.cc:2804] T 30f628e3d7914d609e7f1d980a74c654 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:04:25.756240 11023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a9353d1d230416296627d7cf8b1d425" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:25.729346 12671 consensus_queue.cc:1048] T 401a9cf11eac479aa445ec8c1d6e33b2 P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.039s
I20260709 13:04:25.777534 11025 raft_consensus.cc:1240] T a977842ba4b744f4bb2febd55d286006 P 873357742d5843579556ee8166f23f47 [term 5 FOLLOWER]: Rejecting Update request from peer 10ca5ee68ad64b3283b9d235352c9d23 for earlier term 4. Current term is 5. Ops: []
I20260709 13:04:25.771342 12387 raft_consensus.cc:3060] T e5a5f5a608504650b48fd27f163faed6 P 0172fed22b6b4890a7fdc953fa128af4 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:04:25.782094 12703 raft_consensus.cc:697] T 30f628e3d7914d609e7f1d980a74c654 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 LEADER]: Becoming Leader. State: Replica: 10ca5ee68ad64b3283b9d235352c9d23, State: Running, Role: LEADER
I20260709 13:04:25.792793 12575 raft_consensus.cc:3055] T a977842ba4b744f4bb2febd55d286006 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:04:25.793350 12575 raft_consensus.cc:740] T a977842ba4b744f4bb2febd55d286006 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 10ca5ee68ad64b3283b9d235352c9d23, State: Running, Role: LEADER
I20260709 13:04:25.792951 12703 consensus_queue.cc:237] T 30f628e3d7914d609e7f1d980a74c654 P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:25.794970 12250 leader_election.cc:290] T 836251ebc2b74f5da88b8a8b2338e2a5 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 6 election: Requested vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:25.795460 12585 leader_election.cc:290] T 11de93cd9b9d457489ff1cb80f976c9a P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 6 election: Requested vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:25.777448 12558 leader_election.cc:290] T 87cd8f5097624c8692f1d83544d8881e P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:25.804677 10876 raft_consensus.cc:1240] T 11de93cd9b9d457489ff1cb80f976c9a P 186b12638a87410487467df721a3cbda [term 6 FOLLOWER]: Rejecting Update request from peer 873357742d5843579556ee8166f23f47 for earlier term 5. Current term is 6. Ops: []
I20260709 13:04:25.787199 12644 consensus_queue.cc:237] T 2567c2d84e4648d7ba5dc23a3df32817 P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:25.807297 12586 leader_election.cc:290] T a610d1a71f52432e88fe56130f921fa2 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 6 election: Requested vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:25.809346 12424 raft_consensus.cc:3060] T e96e80ba9ac34d0998395515e5c8c41a P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:04:25.811141 11067 leader_election.cc:304] T 6f46a5102fd64b45bf1c4dff863d2b4d P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:04:25.812423 12513 raft_consensus.cc:2764] T 6f46a5102fd64b45bf1c4dff863d2b4d P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260709 13:04:25.812469 10953 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49424 (request call id 286) took 1343 ms. Trace:
I20260709 13:04:25.813019 10953 rpcz_store.cc:276] 0709 13:04:24.468650 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:24.813147 (+344497us) service_pool.cc:224] Handling call
0709 13:04:24.862624 (+ 49477us) spinlock_profiling.cc:243] Waited 4.29 ms on lock 0x7b580001d198. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f4262efddb1 00007f426303db71 00007f426303da22 00007f426bf56c09 00007f426b0b0ee0 00007f426b17a0ab 00007f426b17a131 00007f426b1280a0 00007f426b127984 00007f42646a26d4 00007f42646a2662
0709 13:04:25.812240 (+949616us) spinlock_profiling.cc:243] Waited 397 ms on lock 0x7b540040b354. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:04:25.812382 (+   142us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":924014208}
I20260709 13:04:25.816390 10919 leader_election.cc:304] T 1e4f2241a94a41f38e481137f868b83d P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:04:25.817286 12578 raft_consensus.cc:2804] T 1e4f2241a94a41f38e481137f868b83d P 10ca5ee68ad64b3283b9d235352c9d23 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:04:25.817672 12578 raft_consensus.cc:493] T 1e4f2241a94a41f38e481137f868b83d P 10ca5ee68ad64b3283b9d235352c9d23 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:25.818015 12578 raft_consensus.cc:3060] T 1e4f2241a94a41f38e481137f868b83d P 10ca5ee68ad64b3283b9d235352c9d23 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:04:25.794349 12575 consensus_queue.cc:260] T a977842ba4b744f4bb2febd55d286006 P 10ca5ee68ad64b3283b9d235352c9d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:25.818358 12679 consensus_queue.cc:260] T 8a69f32c53f24a50b7852d143665d7b4 P 186b12638a87410487467df721a3cbda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:25.819666 12575 raft_consensus.cc:3060] T a977842ba4b744f4bb2febd55d286006 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:04:25.819888 12679 raft_consensus.cc:3060] T 8a69f32c53f24a50b7852d143665d7b4 P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:04:25.823715 10879 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51704 (request call id 291) took 1541 ms. Trace:
I20260709 13:04:25.824148 10879 rpcz_store.cc:276] 0709 13:04:24.282258 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:24.364490 (+ 82232us) service_pool.cc:224] Handling call
0709 13:04:25.823520 (+1459030us) spinlock_profiling.cc:243] Waited 1.05 s on lock 0x7b54000e3154. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:04:25.823638 (+   118us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2412334336}
I20260709 13:04:25.828200 12570 raft_consensus.cc:3055] T d371c5e27bd843469d2e2e453734cb4a P 0172fed22b6b4890a7fdc953fa128af4 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:04:25.828668 12570 raft_consensus.cc:740] T d371c5e27bd843469d2e2e453734cb4a P 0172fed22b6b4890a7fdc953fa128af4 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 0172fed22b6b4890a7fdc953fa128af4, State: Running, Role: LEADER
I20260709 13:04:25.829197 12578 raft_consensus.cc:515] T 1e4f2241a94a41f38e481137f868b83d P 10ca5ee68ad64b3283b9d235352c9d23 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:25.829690 12570 consensus_queue.cc:260] T d371c5e27bd843469d2e2e453734cb4a P 0172fed22b6b4890a7fdc953fa128af4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:25.864688 12578 leader_election.cc:290] T 1e4f2241a94a41f38e481137f868b83d P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 6 election: Requested vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:25.865154 12570 raft_consensus.cc:3060] T d371c5e27bd843469d2e2e453734cb4a P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Advancing to term 5
W20260709 13:04:25.831905 10954 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49434 (request call id 404) took 1406 ms (client timeout 1590 ms). Trace:
W20260709 13:04:25.866711 10954 rpcz_store.cc:269] 0709 13:04:24.425962 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:24.426092 (+   130us) service_pool.cc:224] Handling call
0709 13:04:25.831672 (+1405580us) spinlock_profiling.cc:243] Waited 1.01 s on lock 0x7b5400626954. stack: 00007f4263043343 00007f4263043240 00007f42629d21d4 00007f426c710304 00007f426c710299 00007f42660cb571 00007f42661662fa 00007f426b127b17 00007f42646a26d4 00007f42646a2662 00007f42646a2591 00007f42646a250c 00007f42646a17c2 00007f426449d48c 00007f426449c8c6 00007f426449c26f
0709 13:04:25.831832 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2332424576}
I20260709 13:04:25.802695 12564 raft_consensus.cc:515] T 732c4adeeff6422497aa7db4a82f535b P 10ca5ee68ad64b3283b9d235352c9d23 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:25.868659 11024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e4f2241a94a41f38e481137f868b83d" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
I20260709 13:04:25.869536 11024 raft_consensus.cc:3060] T 1e4f2241a94a41f38e481137f868b83d P 873357742d5843579556ee8166f23f47 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:04:25.832975 11028 raft_consensus.cc:1240] T 525bfdbd6d934afd823f7d3ee3f9d052 P 873357742d5843579556ee8166f23f47 [term 6 FOLLOWER]: Rejecting Update request from peer 10ca5ee68ad64b3283b9d235352c9d23 for earlier term 5. Current term is 6. Ops: []
W20260709 13:04:25.340119 10832 catalog_manager.cc:6160] T 00000000000000000000000000000000 P c669bee8ad644d0d956c148b33886b51: Tablet 30f628e3d7914d609e7f1d980a74c654 (table test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]) was not created within the allowed timeout. Replacing with a new tablet 841df6d41a0b4f3aa4a268afb8430e6b
W20260709 13:04:25.847746 10843 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 134653us
W20260709 13:04:25.848343 10844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:45807, user_credentials={real_user=slave}} blocked reactor thread for 134714us
W20260709 13:04:25.875923 11067 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:25.849112 12649 consensus_queue.cc:1059] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 186b12638a87410487467df721a3cbda [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:04:25.850137 12397 raft_consensus.cc:2749] T c31881534c5e4f0089c82057c1746652 P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:04:25.853082 12517 raft_consensus.cc:3055] T 525bfdbd6d934afd823f7d3ee3f9d052 P 10ca5ee68ad64b3283b9d235352c9d23 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:04:25.878832 10880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e4f2241a94a41f38e481137f868b83d" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:04:25.878966 12649 raft_consensus.cc:3055] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 186b12638a87410487467df721a3cbda [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:04:25.879184 12517 raft_consensus.cc:740] T 525bfdbd6d934afd823f7d3ee3f9d052 P 10ca5ee68ad64b3283b9d235352c9d23 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 10ca5ee68ad64b3283b9d235352c9d23, State: Running, Role: LEADER
I20260709 13:04:25.879518 12649 raft_consensus.cc:740] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 186b12638a87410487467df721a3cbda [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 186b12638a87410487467df721a3cbda, State: Running, Role: LEADER
W20260709 13:04:25.862653 11066 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 192331us
W20260709 13:04:25.880257 10832 catalog_manager.cc:6160] T 00000000000000000000000000000000 P c669bee8ad644d0d956c148b33886b51: Tablet f34f2bc02d85444f9096e0f5b4fe4331 (table test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]) was not created within the allowed timeout. Replacing with a new tablet 7b2274007e414d59b2e02843d65e60cb
I20260709 13:04:25.880478 12649 consensus_queue.cc:260] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 186b12638a87410487467df721a3cbda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:25.881273 12654 raft_consensus.cc:515] T b4cd3d7dfbbf4096908ff11e9dc57141 P 0172fed22b6b4890a7fdc953fa128af4 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
W20260709 13:04:25.881605 10832 catalog_manager.cc:6160] T 00000000000000000000000000000000 P c669bee8ad644d0d956c148b33886b51: Tablet a1ec2b50c6ad4f7c8033476208abb26d (table test-workload [id=eb7b42c4a8a94433b508f75e823ed2e9]) was not created within the allowed timeout. Replacing with a new tablet e9d521c599a546b19a94bf921ff92fb7
I20260709 13:04:25.881865 12649 raft_consensus.cc:3060] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 186b12638a87410487467df721a3cbda [term 5 FOLLOWER]: Advancing to term 6
W20260709 13:04:25.862921 10845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 165195us
I20260709 13:04:25.879319 12532 consensus_queue.cc:1059] T 525bfdbd6d934afd823f7d3ee3f9d052 P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:04:25.883090 10845 leader_election.cc:304] T 6a9353d1d230416296627d7cf8b1d425 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 186b12638a87410487467df721a3cbda; no voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47
I20260709 13:04:25.885073 12679 raft_consensus.cc:2749] T 6a9353d1d230416296627d7cf8b1d425 P 186b12638a87410487467df721a3cbda [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:04:25.878168 10877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87cd8f5097624c8692f1d83544d8881e" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:25.886322 12679 consensus_queue.cc:1059] T 9a1716c0adf34bee8a4e99341ba5788f P 186b12638a87410487467df721a3cbda [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:04:25.886776 10877 raft_consensus.cc:2468] T 87cd8f5097624c8692f1d83544d8881e P 186b12638a87410487467df721a3cbda [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 6.
I20260709 13:04:25.887151 12679 raft_consensus.cc:3055] T 9a1716c0adf34bee8a4e99341ba5788f P 186b12638a87410487467df721a3cbda [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:04:25.886994 10873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "732c4adeeff6422497aa7db4a82f535b" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:04:25.887568 12679 raft_consensus.cc:740] T 9a1716c0adf34bee8a4e99341ba5788f P 186b12638a87410487467df721a3cbda [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 186b12638a87410487467df721a3cbda, State: Running, Role: LEADER
I20260709 13:04:25.888573 12585 consensus_queue.cc:1059] T 2567c2d84e4648d7ba5dc23a3df32817 P 186b12638a87410487467df721a3cbda [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:04:25.889411 12585 consensus_queue.cc:1059] T 30f628e3d7914d609e7f1d980a74c654 P 186b12638a87410487467df721a3cbda [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:04:25.879519 10880 raft_consensus.cc:3060] T 1e4f2241a94a41f38e481137f868b83d P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Advancing to term 6
I20260709 13:04:25.890197 12585 raft_consensus.cc:3055] T 2567c2d84e4648d7ba5dc23a3df32817 P 186b12638a87410487467df721a3cbda [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:04:25.890583 12585 raft_consensus.cc:740] T 2567c2d84e4648d7ba5dc23a3df32817 P 186b12638a87410487467df721a3cbda [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 186b12638a87410487467df721a3cbda, State: Running, Role: LEADER
I20260709 13:04:25.891280 12586 raft_consensus.cc:3055] T 30f628e3d7914d609e7f1d980a74c654 P 186b12638a87410487467df721a3cbda [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:04:25.891666 12586 raft_consensus.cc:740] T 30f628e3d7914d609e7f1d980a74c654 P 186b12638a87410487467df721a3cbda [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 186b12638a87410487467df721a3cbda, State: Running, Role: LEADER
I20260709 13:04:25.892534 12586 consensus_queue.cc:260] T 30f628e3d7914d609e7f1d980a74c654 P 186b12638a87410487467df721a3cbda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:25.893747 12586 raft_consensus.cc:3060] T 30f628e3d7914d609e7f1d980a74c654 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:04:25.893874 12654 leader_election.cc:290] T b4cd3d7dfbbf4096908ff11e9dc57141 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 5 election: Requested vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:25.894718 12528 consensus_queue.cc:1059] T 11de93cd9b9d457489ff1cb80f976c9a P 873357742d5843579556ee8166f23f47 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 13:04:25.895280 11024 raft_consensus.cc:2468] T 1e4f2241a94a41f38e481137f868b83d P 873357742d5843579556ee8166f23f47 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 6.
I20260709 13:04:25.894855 12679 consensus_queue.cc:260] T 9a1716c0adf34bee8a4e99341ba5788f P 186b12638a87410487467df721a3cbda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:25.895982 12528 consensus_queue.cc:1059] T a5299b8d044f41b7b68057e63d17b159 P 873357742d5843579556ee8166f23f47 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:04:25.896565 10919 leader_election.cc:304] T 1e4f2241a94a41f38e481137f868b83d P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:04:25.896767 12053 consensus_queue.cc:1059] T a610d1a71f52432e88fe56130f921fa2 P 873357742d5843579556ee8166f23f47 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 13:04:25.897573 12434 raft_consensus.cc:3055] T a5299b8d044f41b7b68057e63d17b159 P 873357742d5843579556ee8166f23f47 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:04:25.897979 12434 raft_consensus.cc:740] T a5299b8d044f41b7b68057e63d17b159 P 873357742d5843579556ee8166f23f47 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 873357742d5843579556ee8166f23f47, State: Running, Role: LEADER
I20260709 13:04:25.898845 12532 raft_consensus.cc:2804] T 1e4f2241a94a41f38e481137f868b83d P 10ca5ee68ad64b3283b9d235352c9d23 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 13:04:25.899405 12532 raft_consensus.cc:697] T 1e4f2241a94a41f38e481137f868b83d P 10ca5ee68ad64b3283b9d235352c9d23 [term 6 LEADER]: Becoming Leader. State: Replica: 10ca5ee68ad64b3283b9d235352c9d23, State: Running, Role: LEADER
I20260709 13:04:25.899528 12662 raft_consensus.cc:493] T c51782cd9d064ec0949e84e24dd4de67 P 186b12638a87410487467df721a3cbda [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:25.899576 11028 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4cd3d7dfbbf4096908ff11e9dc57141" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
I20260709 13:04:25.898880 12434 consensus_queue.cc:260] T a5299b8d044f41b7b68057e63d17b159 P 873357742d5843579556ee8166f23f47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:25.900259 12662 raft_consensus.cc:515] T c51782cd9d064ec0949e84e24dd4de67 P 186b12638a87410487467df721a3cbda [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:25.900656 12679 raft_consensus.cc:3060] T 9a1716c0adf34bee8a4e99341ba5788f P 186b12638a87410487467df721a3cbda [term 5 FOLLOWER]: Advancing to term 6
W20260709 13:04:25.838212 10774 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:59598 (request call id 66) took 16330 ms (client timeout 15000 ms). Trace:
I20260709 13:04:25.902606 12532 consensus_queue.cc:237] T 1e4f2241a94a41f38e481137f868b83d P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
W20260709 13:04:25.902837 10774 rpcz_store.cc:269] 0709 13:04:09.508217 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:09.508469 (+   252us) service_pool.cc:224] Handling call
0709 13:04:25.838066 (+16329597us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 13:04:19.360761 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 13:04:19.367296 (+  6535us) write_op.cc:432] Acquiring schema lock in shared mode
0709 13:04:19.367355 (+    59us) write_op.cc:435] Acquired schema lock
0709 13:04:19.367379 (+    24us) tablet.cc:662] Decoding operations
0709 13:04:19.545591 (+178212us) write_op.cc:620] Acquiring the partition lock for write op
0709 13:04:19.545733 (+   142us) write_op.cc:641] Partition lock acquired for write op
0709 13:04:19.545764 (+    31us) tablet.cc:685] Acquiring locks for 9 operations
0709 13:04:19.546729 (+   965us) tablet.cc:701] Row locks acquired
0709 13:04:19.546776 (+    47us) write_op.cc:260] PREPARE: finished
0709 13:04:19.547008 (+   232us) write_op.cc:270] Start()
0709 13:04:19.547234 (+   226us) write_op.cc:276] Timestamp: P: 1783602259546935 usec, L: 0
0709 13:04:19.547271 (+    37us) op_driver.cc:348] REPLICATION: starting
0709 13:04:19.548279 (+  1008us) log.cc:844] Serialized 3645 byte log entry
0709 13:04:19.716370 (+168091us) op_driver.cc:464] REPLICATION: finished
0709 13:04:19.920116 (+203746us) write_op.cc:301] APPLY: starting
0709 13:04:19.920236 (+   120us) tablet.cc:1367] starting BulkCheckPresence
0709 13:04:19.920508 (+   272us) tablet.cc:1370] finished BulkCheckPresence
0709 13:04:19.920675 (+   167us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 13:04:19.932888 (+ 12213us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 13:04:19.933269 (+   381us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=9
0709 13:04:19.933484 (+   215us) write_op.cc:312] APPLY: finished
0709 13:04:19.933932 (+   448us) log.cc:844] Serialized 70 byte log entry
0709 13:04:19.934149 (+   217us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:04:19.934293 (+   144us) write_op.cc:454] Released schema lock
0709 13:04:19.934349 (+    56us) write_op.cc:341] FINISH: Updating metrics
0709 13:04:19.934857 (+   508us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:04:19.934890 (+    33us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":357576,"reported_tablets":31,"child_traces":[["op",{"apply.queue_time_us":182638,"apply.run_cpu_time_us":4263,"apply.run_wall_time_us":16701,"num_ops":9,"prepare.queue_time_us":136590,"prepare.run_cpu_time_us":15097,"prepare.run_wall_time_us":325533,"raft.queue_time_us":59708,"raft.run_cpu_time_us":21198,"raft.run_wall_time_us":120451,"replication_time_us":168902,"thread_start_us":292622,"threads_started":4,"wal-append.queue_time_us":105328,"wal-append.run_cpu_time_us":12578,"wal-append.run_wall_time_us":1282661}]]}
I20260709 13:04:25.898001 12584 raft_consensus.cc:3055] T a610d1a71f52432e88fe56130f921fa2 P 873357742d5843579556ee8166f23f47 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:04:25.906916 12584 raft_consensus.cc:740] T a610d1a71f52432e88fe56130f921fa2 P 873357742d5843579556ee8166f23f47 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 873357742d5843579556ee8166f23f47, State: Running, Role: LEADER
I20260709 13:04:25.907761 12434 raft_consensus.cc:3060] T a5299b8d044f41b7b68057e63d17b159 P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:04:25.907805 12584 consensus_queue.cc:260] T a610d1a71f52432e88fe56130f921fa2 P 873357742d5843579556ee8166f23f47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:25.910507 12584 raft_consensus.cc:3060] T a610d1a71f52432e88fe56130f921fa2 P 873357742d5843579556ee8166f23f47 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:04:25.884629 12517 consensus_queue.cc:260] T 525bfdbd6d934afd823f7d3ee3f9d052 P 10ca5ee68ad64b3283b9d235352c9d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:25.912717 12517 raft_consensus.cc:3060] T 525bfdbd6d934afd823f7d3ee3f9d052 P 10ca5ee68ad64b3283b9d235352c9d23 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:04:25.897078 12528 raft_consensus.cc:3055] T 11de93cd9b9d457489ff1cb80f976c9a P 873357742d5843579556ee8166f23f47 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:04:25.914011 12528 raft_consensus.cc:740] T 11de93cd9b9d457489ff1cb80f976c9a P 873357742d5843579556ee8166f23f47 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 873357742d5843579556ee8166f23f47, State: Running, Role: LEADER
I20260709 13:04:25.896533 10921 leader_election.cc:304] T 87cd8f5097624c8692f1d83544d8881e P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda; no voters: 
I20260709 13:04:25.915035 12578 raft_consensus.cc:2804] T 87cd8f5097624c8692f1d83544d8881e P 10ca5ee68ad64b3283b9d235352c9d23 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:04:25.914958 12528 consensus_queue.cc:260] T 11de93cd9b9d457489ff1cb80f976c9a P 873357742d5843579556ee8166f23f47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:25.916389 12662 leader_election.cc:290] T c51782cd9d064ec0949e84e24dd4de67 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:25.916703 12528 raft_consensus.cc:3060] T 11de93cd9b9d457489ff1cb80f976c9a P 873357742d5843579556ee8166f23f47 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:04:25.921986 10880 raft_consensus.cc:2468] T 1e4f2241a94a41f38e481137f868b83d P 186b12638a87410487467df721a3cbda [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 6.
I20260709 13:04:25.915485 12578 raft_consensus.cc:493] T 87cd8f5097624c8692f1d83544d8881e P 10ca5ee68ad64b3283b9d235352c9d23 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 186b12638a87410487467df721a3cbda)
I20260709 13:04:25.940205 11027 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a610d1a71f52432e88fe56130f921fa2" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
I20260709 13:04:25.940598 12578 raft_consensus.cc:3060] T 87cd8f5097624c8692f1d83544d8881e P 10ca5ee68ad64b3283b9d235352c9d23 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:04:25.941900 11023 raft_consensus.cc:1240] T 2567c2d84e4648d7ba5dc23a3df32817 P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Rejecting Update request from peer 186b12638a87410487467df721a3cbda for earlier term 3. Current term is 4. Ops: [3.3-3.3]
    @     0x7f42632a6cd0 testing::internal::HandleExceptionsInMethodIfSupported<>() at ??:0
I20260709 13:04:25.944231 12564 leader_election.cc:290] T 732c4adeeff6422497aa7db4a82f535b P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 6 election: Requested vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:25.945474 11023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "732c4adeeff6422497aa7db4a82f535b" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
I20260709 13:04:25.923859 12585 consensus_queue.cc:260] T 2567c2d84e4648d7ba5dc23a3df32817 P 186b12638a87410487467df721a3cbda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:25.935425 12558 raft_consensus.cc:493] T bac7e5040f984ba2b7146bd36af7267e P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:25.946818 12558 raft_consensus.cc:515] T bac7e5040f984ba2b7146bd36af7267e P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:25.947139 10919 leader_election.cc:304] T 732c4adeeff6422497aa7db4a82f535b P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23; no voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47
I20260709 13:04:25.949108 12558 leader_election.cc:290] T bac7e5040f984ba2b7146bd36af7267e P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:25.949648 12558 raft_consensus.cc:2749] T 732c4adeeff6422497aa7db4a82f535b P 10ca5ee68ad64b3283b9d235352c9d23 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 13:04:25.937654 11024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11de93cd9b9d457489ff1cb80f976c9a" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
I20260709 13:04:25.938669 11026 raft_consensus.cc:1275] T 732c4adeeff6422497aa7db4a82f535b P 873357742d5843579556ee8166f23f47 [term 5 FOLLOWER]: Refusing update from remote peer 186b12638a87410487467df721a3cbda: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260709 13:04:25.951156 11025 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bac7e5040f984ba2b7146bd36af7267e" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:25.951354 12585 raft_consensus.cc:3060] T 2567c2d84e4648d7ba5dc23a3df32817 P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:04:25.957733 12397 consensus_queue.cc:1048] T 732c4adeeff6422497aa7db4a82f535b P 186b12638a87410487467df721a3cbda [LEADER]: Connected to new peer: Peer: permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.000s
W20260709 13:04:25.963487 10992 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 166167us
W20260709 13:04:25.964995 10920 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 253719us
I20260709 13:04:25.973903 10920 leader_election.cc:304] T e5a5f5a608504650b48fd27f163faed6 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23; no voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda
W20260709 13:04:25.980237 10995 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:25.981926 12705 raft_consensus.cc:2749] T e5a5f5a608504650b48fd27f163faed6 P 10ca5ee68ad64b3283b9d235352c9d23 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:04:25.983045 12578 raft_consensus.cc:515] T 87cd8f5097624c8692f1d83544d8881e P 10ca5ee68ad64b3283b9d235352c9d23 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:25.983604 10920 leader_election.cc:304] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 10ca5ee68ad64b3283b9d235352c9d23; no voters: 873357742d5843579556ee8166f23f47
W20260709 13:04:25.985626 11069 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:38967, user_credentials={real_user=slave}} blocked reactor thread for 256299us
I20260709 13:04:25.987274 10880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87cd8f5097624c8692f1d83544d8881e" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:04:25.988888 11028 raft_consensus.cc:2468] T b4cd3d7dfbbf4096908ff11e9dc57141 P 873357742d5843579556ee8166f23f47 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 5.
I20260709 13:04:25.989805 12387 raft_consensus.cc:515] T e5a5f5a608504650b48fd27f163faed6 P 0172fed22b6b4890a7fdc953fa128af4 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:25.991267 11067 leader_election.cc:304] T b4cd3d7dfbbf4096908ff11e9dc57141 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:04:25.992834 12387 leader_election.cc:290] T e5a5f5a608504650b48fd27f163faed6 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 6 election: Requested vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:25.993528 12387 raft_consensus.cc:2804] T b4cd3d7dfbbf4096908ff11e9dc57141 P 0172fed22b6b4890a7fdc953fa128af4 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:04:25.994012 12387 raft_consensus.cc:697] T b4cd3d7dfbbf4096908ff11e9dc57141 P 0172fed22b6b4890a7fdc953fa128af4 [term 5 LEADER]: Becoming Leader. State: Replica: 0172fed22b6b4890a7fdc953fa128af4, State: Running, Role: LEADER
I20260709 13:04:25.996574 10877 raft_consensus.cc:1275] T 7709387f4d3d4f9992d97a0fbca39a40 P 186b12638a87410487467df721a3cbda [term 8 FOLLOWER]: Refusing update from remote peer 873357742d5843579556ee8166f23f47: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
I20260709 13:04:25.999469 11069 leader_election.cc:304] T d7ce47b9a92d43dd88d95bcda5038597 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda; no voters: 
I20260709 13:04:26.001281 10877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f9eb06ed01f4ce68be3af8a8434665b" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:04:26.002760 12705 raft_consensus.cc:2804] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 10ca5ee68ad64b3283b9d235352c9d23 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:04:26.003257 12705 raft_consensus.cc:493] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 10ca5ee68ad64b3283b9d235352c9d23 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:26.003165 12531 consensus_queue.cc:1048] T 7709387f4d3d4f9992d97a0fbca39a40 P 873357742d5843579556ee8166f23f47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 13:04:26.003815 12705 raft_consensus.cc:3060] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 10ca5ee68ad64b3283b9d235352c9d23 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:04:26.003681 10876 raft_consensus.cc:1240] T 0e5d327630e5438b878035d28efe6c0c P 186b12638a87410487467df721a3cbda [term 6 FOLLOWER]: Rejecting Update request from peer 0172fed22b6b4890a7fdc953fa128af4 for earlier term 5. Current term is 6. Ops: []
W20260709 13:04:26.004827 10845 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:26.006085 10920 leader_election.cc:304] T d71ca3a1ba6b48478aaf62741bf3cf6a P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 10ca5ee68ad64b3283b9d235352c9d23; no voters: 873357742d5843579556ee8166f23f47
I20260709 13:04:26.006963 12644 raft_consensus.cc:2804] T d71ca3a1ba6b48478aaf62741bf3cf6a P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:04:26.007681 12644 raft_consensus.cc:697] T d71ca3a1ba6b48478aaf62741bf3cf6a P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 LEADER]: Becoming Leader. State: Replica: 10ca5ee68ad64b3283b9d235352c9d23, State: Running, Role: LEADER
I20260709 13:04:26.008615 12644 consensus_queue.cc:237] T d71ca3a1ba6b48478aaf62741bf3cf6a P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
W20260709 13:04:26.010600 10844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:26.012580 12654 raft_consensus.cc:2804] T d7ce47b9a92d43dd88d95bcda5038597 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:04:26.014938 10954 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5ba72ccd854430baadbc636dfd17136" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:26.015168 10920 leader_election.cc:304] T 0f9df2bc281f4866b6da5da94a0defc7 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 10ca5ee68ad64b3283b9d235352c9d23; no voters: 186b12638a87410487467df721a3cbda
I20260709 13:04:26.020539 11024 raft_consensus.cc:2468] T 11de93cd9b9d457489ff1cb80f976c9a P 873357742d5843579556ee8166f23f47 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 6.
I20260709 13:04:26.021284 10877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d270b9e5481245d9b5340a454a7cd10b" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:04:26.021919 10873 raft_consensus.cc:1240] T 836251ebc2b74f5da88b8a8b2338e2a5 P 186b12638a87410487467df721a3cbda [term 6 FOLLOWER]: Rejecting Update request from peer 0172fed22b6b4890a7fdc953fa128af4 for earlier term 5. Current term is 6. Ops: []
I20260709 13:04:26.025063 10871 raft_consensus.cc:1275] T c51782cd9d064ec0949e84e24dd4de67 P 186b12638a87410487467df721a3cbda [term 5 FOLLOWER]: Refusing update from remote peer 0172fed22b6b4890a7fdc953fa128af4: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260709 13:04:26.028195 10880 raft_consensus.cc:2468] T 87cd8f5097624c8692f1d83544d8881e P 186b12638a87410487467df721a3cbda [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 6.
I20260709 13:04:26.019816 10876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a9353d1d230416296627d7cf8b1d425" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:26.030683 12578 leader_election.cc:290] T 87cd8f5097624c8692f1d83544d8881e P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 6 election: Requested vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:26.031152 10954 raft_consensus.cc:2393] T e5ba72ccd854430baadbc636dfd17136 P 10ca5ee68ad64b3283b9d235352c9d23 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 873357742d5843579556ee8166f23f47 in current term 7: Already voted for candidate 10ca5ee68ad64b3283b9d235352c9d23 in this term.
I20260709 13:04:26.034061 11094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "836251ebc2b74f5da88b8a8b2338e2a5" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:26.036811 10920 leader_election.cc:304] T d270b9e5481245d9b5340a454a7cd10b P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23; no voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda
I20260709 13:04:26.041941 10920 leader_election.cc:304] T f34f2bc02d85444f9096e0f5b4fe4331 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 10ca5ee68ad64b3283b9d235352c9d23; no voters: 186b12638a87410487467df721a3cbda
I20260709 13:04:26.063678 12387 consensus_queue.cc:237] T b4cd3d7dfbbf4096908ff11e9dc57141 P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:26.062686 11094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "401a9cf11eac479aa445ec8c1d6e33b2" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:26.067725 10921 leader_election.cc:304] T 87cd8f5097624c8692f1d83544d8881e P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda; no voters: 
I20260709 13:04:26.062747 11103 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e5d327630e5438b878035d28efe6c0c" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:04:26.060915 10872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5a5f5a608504650b48fd27f163faed6" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:04:26.061040 10875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f9df2bc281f4866b6da5da94a0defc7" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:26.067126 10844 leader_election.cc:304] T 11de93cd9b9d457489ff1cb80f976c9a P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:04:26.070384 11103 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "836251ebc2b74f5da88b8a8b2338e2a5" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
W20260709 13:04:26.080547 10918 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:26.016024 12654 raft_consensus.cc:493] T d7ce47b9a92d43dd88d95bcda5038597 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 186b12638a87410487467df721a3cbda)
I20260709 13:04:26.120364 11027 raft_consensus.cc:2468] T a610d1a71f52432e88fe56130f921fa2 P 873357742d5843579556ee8166f23f47 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 6.
W20260709 13:04:26.060187 11066 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 177864us
W20260709 13:04:26.102898 10920 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 61068.2us
I20260709 13:04:26.061276 12578 raft_consensus.cc:2804] T 0f9df2bc281f4866b6da5da94a0defc7 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:04:26.109426 11094 raft_consensus.cc:1240] T d270b9e5481245d9b5340a454a7cd10b P 0172fed22b6b4890a7fdc953fa128af4 [term 7 FOLLOWER]: Rejecting Update request from peer 186b12638a87410487467df721a3cbda for earlier term 6. Current term is 7. Ops: []
I20260709 13:04:26.043955 12709 raft_consensus.cc:493] T 72b111ad7b5f4753ab1d2b5ee1114840 P 0172fed22b6b4890a7fdc953fa128af4 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:26.136374 11099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5ba72ccd854430baadbc636dfd17136" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:04:26.137014 11099 raft_consensus.cc:3060] T e5ba72ccd854430baadbc636dfd17136 P 0172fed22b6b4890a7fdc953fa128af4 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:04:26.118795 12612 raft_consensus.cc:493] T a977842ba4b744f4bb2febd55d286006 P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 10ca5ee68ad64b3283b9d235352c9d23)
I20260709 13:04:26.144556 11100 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "401a9cf11eac479aa445ec8c1d6e33b2" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:26.121959 11103 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a610d1a71f52432e88fe56130f921fa2" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:04:26.122990 11097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c51782cd9d064ec0949e84e24dd4de67" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:26.141445 11094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a1716c0adf34bee8a4e99341ba5788f" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:26.128646 10920 leader_election.cc:304] T 6f46a5102fd64b45bf1c4dff863d2b4d P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 10ca5ee68ad64b3283b9d235352c9d23; no voters: 873357742d5843579556ee8166f23f47
W20260709 13:04:26.102932 10992 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 64476us
I20260709 13:04:26.110694 12703 raft_consensus.cc:2804] T f34f2bc02d85444f9096e0f5b4fe4331 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:04:26.111493 12532 raft_consensus.cc:2804] T 87cd8f5097624c8692f1d83544d8881e P 10ca5ee68ad64b3283b9d235352c9d23 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 13:04:26.110854 12694 raft_consensus.cc:2804] T 11de93cd9b9d457489ff1cb80f976c9a P 186b12638a87410487467df721a3cbda [term 6 FOLLOWER]: Leader election won for term 6
I20260709 13:04:26.147568 11100 pending_rounds.cc:85] T a1ec2b50c6ad4f7c8033476208abb26d P 0172fed22b6b4890a7fdc953fa128af4: Aborting all ops after (but not including) 0
I20260709 13:04:26.155478 11100 pending_rounds.cc:107] T a1ec2b50c6ad4f7c8033476208abb26d P 0172fed22b6b4890a7fdc953fa128af4: Aborting uncommitted NO_OP operation due to leader change: 1.1
I20260709 13:04:26.148458 11103 raft_consensus.cc:3060] T a610d1a71f52432e88fe56130f921fa2 P 0172fed22b6b4890a7fdc953fa128af4 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:04:26.061322 12575 raft_consensus.cc:3060] T d270b9e5481245d9b5340a454a7cd10b P 10ca5ee68ad64b3283b9d235352c9d23 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:04:26.147004 11098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d7ce47b9a92d43dd88d95bcda5038597" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:04:26.155795 11100 raft_consensus.cc:2889] T a1ec2b50c6ad4f7c8033476208abb26d P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:04:26.119741 12669 raft_consensus.cc:493] T 525bfdbd6d934afd823f7d3ee3f9d052 P 0172fed22b6b4890a7fdc953fa128af4 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 10ca5ee68ad64b3283b9d235352c9d23)
I20260709 13:04:26.164500 11094 raft_consensus.cc:2393] T 9a1716c0adf34bee8a4e99341ba5788f P 0172fed22b6b4890a7fdc953fa128af4 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 873357742d5843579556ee8166f23f47 in current term 6: Already voted for candidate 0172fed22b6b4890a7fdc953fa128af4 in this term.
I20260709 13:04:26.167668 11094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a69f32c53f24a50b7852d143665d7b4" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:26.168630 11094 raft_consensus.cc:2393] T 8a69f32c53f24a50b7852d143665d7b4 P 0172fed22b6b4890a7fdc953fa128af4 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 10ca5ee68ad64b3283b9d235352c9d23 in current term 5: Already voted for candidate 0172fed22b6b4890a7fdc953fa128af4 in this term.
I20260709 13:04:26.168743 11097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b67eff0634e540eab91cf8cee795969f" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:26.169451 11097 raft_consensus.cc:2468] T b67eff0634e540eab91cf8cee795969f P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 3.
W20260709 13:04:26.171967 10992 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:26.131078 12578 raft_consensus.cc:493] T 0f9df2bc281f4866b6da5da94a0defc7 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 186b12638a87410487467df721a3cbda)
I20260709 13:04:26.177176 11066 leader_election.cc:304] T f34f2bc02d85444f9096e0f5b4fe4331 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda
I20260709 13:04:26.127403 12654 raft_consensus.cc:3060] T d7ce47b9a92d43dd88d95bcda5038597 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:04:26.119419 12397 consensus_queue.cc:1059] T 4f9eb06ed01f4ce68be3af8a8434665b P 186b12638a87410487467df721a3cbda [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.042s
I20260709 13:04:26.152858 12564 raft_consensus.cc:2804] T 6f46a5102fd64b45bf1c4dff863d2b4d P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:04:26.206298 11103 raft_consensus.cc:2468] T a610d1a71f52432e88fe56130f921fa2 P 0172fed22b6b4890a7fdc953fa128af4 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 6.
I20260709 13:04:26.184526 11099 raft_consensus.cc:2468] T e5ba72ccd854430baadbc636dfd17136 P 0172fed22b6b4890a7fdc953fa128af4 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 7.
I20260709 13:04:26.144984 12672 raft_consensus.cc:493] T a64b7e99d6c54934a7de6f6dffcf5e94 P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:26.181007 12691 raft_consensus.cc:2749] T f34f2bc02d85444f9096e0f5b4fe4331 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:04:26.183228 12578 raft_consensus.cc:3060] T 0f9df2bc281f4866b6da5da94a0defc7 P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:04:26.151170 12612 raft_consensus.cc:515] T a977842ba4b744f4bb2febd55d286006 P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:26.147738 12709 raft_consensus.cc:515] T 72b111ad7b5f4753ab1d2b5ee1114840 P 0172fed22b6b4890a7fdc953fa128af4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:26.212497 12672 raft_consensus.cc:515] T a64b7e99d6c54934a7de6f6dffcf5e94 P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
W20260709 13:04:26.263311 10845 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:26.176059 12694 raft_consensus.cc:697] T 11de93cd9b9d457489ff1cb80f976c9a P 186b12638a87410487467df721a3cbda [term 6 LEADER]: Becoming Leader. State: Replica: 186b12638a87410487467df721a3cbda, State: Running, Role: LEADER
I20260709 13:04:26.175832 12532 raft_consensus.cc:697] T 87cd8f5097624c8692f1d83544d8881e P 10ca5ee68ad64b3283b9d235352c9d23 [term 6 LEADER]: Becoming Leader. State: Replica: 10ca5ee68ad64b3283b9d235352c9d23, State: Running, Role: LEADER
W20260709 13:04:26.279081 10846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:26.249411 12564 raft_consensus.cc:697] T 6f46a5102fd64b45bf1c4dff863d2b4d P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 LEADER]: Becoming Leader. State: Replica: 10ca5ee68ad64b3283b9d235352c9d23, State: Running, Role: LEADER
I20260709 13:04:26.284286 12534 raft_consensus.cc:493] T d71ca3a1ba6b48478aaf62741bf3cf6a P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:26.306208 10953 raft_consensus.cc:1240] T 2567c2d84e4648d7ba5dc23a3df32817 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 LEADER]: Rejecting Update request from peer 186b12638a87410487467df721a3cbda for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260709 13:04:26.310689 10844 leader_election.cc:304] T a610d1a71f52432e88fe56130f921fa2 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:04:26.204387 12703 raft_consensus.cc:697] T f34f2bc02d85444f9096e0f5b4fe4331 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 LEADER]: Becoming Leader. State: Replica: 10ca5ee68ad64b3283b9d235352c9d23, State: Running, Role: LEADER
I20260709 13:04:26.212822 12679 raft_consensus.cc:3055] T 4f9eb06ed01f4ce68be3af8a8434665b P 186b12638a87410487467df721a3cbda [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:04:26.323459 10949 raft_consensus.cc:1240] T 732c4adeeff6422497aa7db4a82f535b P 10ca5ee68ad64b3283b9d235352c9d23 [term 6 FOLLOWER]: Rejecting Update request from peer 186b12638a87410487467df721a3cbda for earlier term 5. Current term is 6. Ops: []
I20260709 13:04:26.308158 11102 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bac7e5040f984ba2b7146bd36af7267e" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:26.324235 11103 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87cd8f5097624c8692f1d83544d8881e" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:26.324502 11102 raft_consensus.cc:2468] T bac7e5040f984ba2b7146bd36af7267e P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 2.
I20260709 13:04:26.325318 11103 raft_consensus.cc:2393] T 87cd8f5097624c8692f1d83544d8881e P 0172fed22b6b4890a7fdc953fa128af4 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 10ca5ee68ad64b3283b9d235352c9d23 in current term 6: Already voted for candidate 0172fed22b6b4890a7fdc953fa128af4 in this term.
I20260709 13:04:26.313488 10953 raft_consensus.cc:1240] T 30f628e3d7914d609e7f1d980a74c654 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 LEADER]: Rejecting Update request from peer 186b12638a87410487467df721a3cbda for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260709 13:04:26.315042 10952 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5299b8d044f41b7b68057e63d17b159" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:04:26.316493 10946 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11de93cd9b9d457489ff1cb80f976c9a" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:04:26.318532 10955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c51782cd9d064ec0949e84e24dd4de67" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:26.326769 10955 raft_consensus.cc:2468] T c51782cd9d064ec0949e84e24dd4de67 P 10ca5ee68ad64b3283b9d235352c9d23 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 5.
I20260709 13:04:26.326802 10953 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4cd3d7dfbbf4096908ff11e9dc57141" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:04:26.327656 10955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a154cc8b181f460abf4880064379948b" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:04:26.327872 10953 raft_consensus.cc:2393] T b4cd3d7dfbbf4096908ff11e9dc57141 P 10ca5ee68ad64b3283b9d235352c9d23 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 0172fed22b6b4890a7fdc953fa128af4 in current term 5: Already voted for candidate 10ca5ee68ad64b3283b9d235352c9d23 in this term.
I20260709 13:04:26.330096 10953 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5a5f5a608504650b48fd27f163faed6" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:04:26.331296 10953 raft_consensus.cc:2393] T e5a5f5a608504650b48fd27f163faed6 P 10ca5ee68ad64b3283b9d235352c9d23 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 0172fed22b6b4890a7fdc953fa128af4 in current term 6: Already voted for candidate 10ca5ee68ad64b3283b9d235352c9d23 in this term.
I20260709 13:04:26.332149 11102 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87cd8f5097624c8692f1d83544d8881e" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:04:26.175390 12669 raft_consensus.cc:515] T 525bfdbd6d934afd823f7d3ee3f9d052 P 0172fed22b6b4890a7fdc953fa128af4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:26.333200 10872 raft_consensus.cc:2468] T e5a5f5a608504650b48fd27f163faed6 P 186b12638a87410487467df721a3cbda [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 6.
I20260709 13:04:26.339425 10951 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "836251ebc2b74f5da88b8a8b2338e2a5" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:04:26.340840 11102 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acc241e7273d49149984968596242712" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:04:26.340871 10951 raft_consensus.cc:3060] T 836251ebc2b74f5da88b8a8b2338e2a5 P 10ca5ee68ad64b3283b9d235352c9d23 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:04:26.347925 10843 leader_election.cc:304] T c51782cd9d064ec0949e84e24dd4de67 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda; no voters: 
I20260709 13:04:26.350374 10843 leader_election.cc:304] T a5299b8d044f41b7b68057e63d17b159 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 186b12638a87410487467df721a3cbda; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47
I20260709 13:04:26.321744 12679 raft_consensus.cc:740] T 4f9eb06ed01f4ce68be3af8a8434665b P 186b12638a87410487467df721a3cbda [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 186b12638a87410487467df721a3cbda, State: Running, Role: LEADER
I20260709 13:04:26.342727 12662 raft_consensus.cc:2804] T a610d1a71f52432e88fe56130f921fa2 P 186b12638a87410487467df721a3cbda [term 6 FOLLOWER]: Leader election won for term 6
I20260709 13:04:26.329970 12534 raft_consensus.cc:515] T d71ca3a1ba6b48478aaf62741bf3cf6a P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:26.362867 12537 raft_consensus.cc:2804] T c51782cd9d064ec0949e84e24dd4de67 P 186b12638a87410487467df721a3cbda [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:04:26.433362 12537 raft_consensus.cc:493] T c51782cd9d064ec0949e84e24dd4de67 P 186b12638a87410487467df721a3cbda [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:26.434007 12537 raft_consensus.cc:3060] T c51782cd9d064ec0949e84e24dd4de67 P 186b12638a87410487467df721a3cbda [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:04:26.436414 11103 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c160ddfe9b34bdaa3835d0915c1e06f" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:26.437650 11103 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5ba72ccd854430baadbc636dfd17136" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:04:26.373273 12488 consensus_queue.cc:1059] T 732c4adeeff6422497aa7db4a82f535b P 186b12638a87410487467df721a3cbda [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.022s
I20260709 13:04:26.394798 12534 leader_election.cc:290] T d71ca3a1ba6b48478aaf62741bf3cf6a P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
W20260709 13:04:26.399080 12716 log.cc:927] Time spent T 401a9cf11eac479aa445ec8c1d6e33b2 P 186b12638a87410487467df721a3cbda: Append to log took a long time: real 0.059s	user 0.000s	sys 0.000s
I20260709 13:04:26.438791 12488 consensus_queue.cc:1243] T 732c4adeeff6422497aa7db4a82f535b P 186b12638a87410487467df721a3cbda [LEADER]: Peer 10ca5ee68ad64b3283b9d235352c9d23 log is divergent from this leader: its last log entry 4.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260709 13:04:26.360846 12644 raft_consensus.cc:493] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 10ca5ee68ad64b3283b9d235352c9d23 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:26.439669 12644 raft_consensus.cc:515] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 10ca5ee68ad64b3283b9d235352c9d23 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:26.399784 12678 raft_consensus.cc:2749] T a5299b8d044f41b7b68057e63d17b159 P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 13:04:26.402679 12662 raft_consensus.cc:697] T a610d1a71f52432e88fe56130f921fa2 P 186b12638a87410487467df721a3cbda [term 6 LEADER]: Becoming Leader. State: Replica: 186b12638a87410487467df721a3cbda, State: Running, Role: LEADER
I20260709 13:04:26.442855 11094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d71ca3a1ba6b48478aaf62741bf3cf6a" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:26.442620 12662 consensus_queue.cc:237] T a610d1a71f52432e88fe56130f921fa2 P 186b12638a87410487467df721a3cbda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:26.443535 11094 raft_consensus.cc:2468] T d71ca3a1ba6b48478aaf62741bf3cf6a P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 4.
I20260709 13:04:26.444442 11094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555217d45d9b4ee492dbb3ddaf0f7c02" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:04:26.445551 11094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "525bfdbd6d934afd823f7d3ee3f9d052" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:04:26.402740 12564 consensus_queue.cc:237] T 6f46a5102fd64b45bf1c4dff863d2b4d P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:26.404021 12703 consensus_queue.cc:237] T f34f2bc02d85444f9096e0f5b4fe4331 P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:26.449535 10920 leader_election.cc:304] T 8a69f32c53f24a50b7852d143665d7b4 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23; no voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda
I20260709 13:04:26.450405 12558 raft_consensus.cc:3060] T 8a69f32c53f24a50b7852d143665d7b4 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:04:26.401368 12709 leader_election.cc:290] T 72b111ad7b5f4753ab1d2b5ee1114840 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:26.412050 12669 leader_election.cc:290] T 525bfdbd6d934afd823f7d3ee3f9d052 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:26.412957 11023 raft_consensus.cc:1275] T 868e92339c2f4508af0afbd07c814d64 P 873357742d5843579556ee8166f23f47 [term 5 FOLLOWER]: Refusing update from remote peer 186b12638a87410487467df721a3cbda: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260709 13:04:26.413834 11020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "525bfdbd6d934afd823f7d3ee3f9d052" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:26.416411 12578 raft_consensus.cc:515] T 0f9df2bc281f4866b6da5da94a0defc7 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:26.419744 11069 leader_election.cc:304] T 4f9eb06ed01f4ce68be3af8a8434665b P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4; no voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47
I20260709 13:04:26.420463 11099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a977842ba4b744f4bb2febd55d286006" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:04:26.427022 12679 consensus_queue.cc:260] T 4f9eb06ed01f4ce68be3af8a8434665b P 186b12638a87410487467df721a3cbda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:26.429950 12612 leader_election.cc:290] T a977842ba4b744f4bb2febd55d286006 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:26.430371 10872 raft_consensus.cc:1275] T 2567c2d84e4648d7ba5dc23a3df32817 P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Refusing update from remote peer 10ca5ee68ad64b3283b9d235352c9d23: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 3. (term mismatch)
I20260709 13:04:26.477152 10872 pending_rounds.cc:85] T 2567c2d84e4648d7ba5dc23a3df32817 P 186b12638a87410487467df721a3cbda: Aborting all ops after (but not including) 2
I20260709 13:04:26.477407 10872 pending_rounds.cc:107] T 2567c2d84e4648d7ba5dc23a3df32817 P 186b12638a87410487467df721a3cbda: Aborting uncommitted NO_OP operation due to leader change: 3.3
I20260709 13:04:26.477628 10872 raft_consensus.cc:2889] T 2567c2d84e4648d7ba5dc23a3df32817 P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:04:26.430843 10955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d71ca3a1ba6b48478aaf62741bf3cf6a" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:26.433466 12672 leader_election.cc:290] T a64b7e99d6c54934a7de6f6dffcf5e94 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
W20260709 13:04:26.498284 10995 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:26.407636 10994 leader_election.cc:304] T 401a9cf11eac479aa445ec8c1d6e33b2 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 873357742d5843579556ee8166f23f47; no voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda
W20260709 13:04:26.499589 10994 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 92101.9us
I20260709 13:04:26.496074 10873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f9df2bc281f4866b6da5da94a0defc7" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:04:26.501144 10994 leader_election.cc:304] T 9a1716c0adf34bee8a4e99341ba5788f P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 873357742d5843579556ee8166f23f47; no voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda
I20260709 13:04:26.450973 10920 leader_election.cc:304] T b67eff0634e540eab91cf8cee795969f P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 10ca5ee68ad64b3283b9d235352c9d23; no voters: 186b12638a87410487467df721a3cbda
I20260709 13:04:26.454317 10871 raft_consensus.cc:1275] T 1e4f2241a94a41f38e481137f868b83d P 186b12638a87410487467df721a3cbda [term 6 FOLLOWER]: Refusing update from remote peer 10ca5ee68ad64b3283b9d235352c9d23: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 13:04:26.455148 10875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72b111ad7b5f4753ab1d2b5ee1114840" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:26.459959 11024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a977842ba4b744f4bb2febd55d286006" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:26.503458 10875 raft_consensus.cc:2468] T 72b111ad7b5f4753ab1d2b5ee1114840 P 186b12638a87410487467df721a3cbda [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 5.
I20260709 13:04:26.503654 11024 raft_consensus.cc:2393] T a977842ba4b744f4bb2febd55d286006 P 873357742d5843579556ee8166f23f47 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0172fed22b6b4890a7fdc953fa128af4 in current term 5: Already voted for candidate 873357742d5843579556ee8166f23f47 in this term.
I20260709 13:04:26.466589 11025 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a64b7e99d6c54934a7de6f6dffcf5e94" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
W20260709 13:04:26.505138 10844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:26.504756 11025 raft_consensus.cc:2468] T a64b7e99d6c54934a7de6f6dffcf5e94 P 873357742d5843579556ee8166f23f47 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 5.
W20260709 13:04:26.508761 10920 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 57910.9us
I20260709 13:04:26.438354 11103 raft_consensus.cc:2393] T e5ba72ccd854430baadbc636dfd17136 P 0172fed22b6b4890a7fdc953fa128af4 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 873357742d5843579556ee8166f23f47 in current term 7: Already voted for candidate 10ca5ee68ad64b3283b9d235352c9d23 in this term.
I20260709 13:04:26.474246 11099 raft_consensus.cc:3060] T a977842ba4b744f4bb2febd55d286006 P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:04:26.475483 11027 raft_consensus.cc:1275] T 1e4f2241a94a41f38e481137f868b83d P 873357742d5843579556ee8166f23f47 [term 6 FOLLOWER]: Refusing update from remote peer 10ca5ee68ad64b3283b9d235352c9d23: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 13:04:26.445724 12654 raft_consensus.cc:515] T d7ce47b9a92d43dd88d95bcda5038597 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:26.520069 12517 raft_consensus.cc:2804] T b67eff0634e540eab91cf8cee795969f P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:04:26.520555 12517 raft_consensus.cc:493] T b67eff0634e540eab91cf8cee795969f P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 186b12638a87410487467df721a3cbda)
I20260709 13:04:26.520924 12517 raft_consensus.cc:3060] T b67eff0634e540eab91cf8cee795969f P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:04:26.521616 12654 leader_election.cc:290] T d7ce47b9a92d43dd88d95bcda5038597 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 3 election: Requested vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:26.502786 12434 raft_consensus.cc:2749] T 401a9cf11eac479aa445ec8c1d6e33b2 P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:04:26.478113 11066 leader_election.cc:304] T a154cc8b181f460abf4880064379948b P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47
I20260709 13:04:26.482332 11020 raft_consensus.cc:2393] T 525bfdbd6d934afd823f7d3ee3f9d052 P 873357742d5843579556ee8166f23f47 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0172fed22b6b4890a7fdc953fa128af4 in current term 6: Already voted for candidate 873357742d5843579556ee8166f23f47 in this term.
I20260709 13:04:26.483114 12578 leader_election.cc:290] T 0f9df2bc281f4866b6da5da94a0defc7 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 3 election: Requested vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:26.483593 12581 consensus_queue.cc:1048] T 2567c2d84e4648d7ba5dc23a3df32817 P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:04:26.484287 12644 leader_election.cc:290] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:26.402935 12532 consensus_queue.cc:237] T 87cd8f5097624c8692f1d83544d8881e P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:26.469978 12705 raft_consensus.cc:515] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 10ca5ee68ad64b3283b9d235352c9d23 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:26.486579 10878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d156d6ae2ca4fa59a1e387edf0fea39" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:26.491268 11021 raft_consensus.cc:1275] T 2567c2d84e4648d7ba5dc23a3df32817 P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Refusing update from remote peer 10ca5ee68ad64b3283b9d235352c9d23: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 13:04:26.542706 10878 raft_consensus.cc:2468] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 186b12638a87410487467df721a3cbda [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 5.
I20260709 13:04:26.491320 11094 raft_consensus.cc:1240] T 6f46a5102fd64b45bf1c4dff863d2b4d P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Rejecting Update request from peer 873357742d5843579556ee8166f23f47 for earlier term 2. Current term is 3. Ops: []
I20260709 13:04:26.493827 10947 raft_consensus.cc:1240] T 0f9df2bc281f4866b6da5da94a0defc7 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Rejecting Update request from peer 186b12638a87410487467df721a3cbda for earlier term 2. Current term is 3. Ops: []
I20260709 13:04:26.496771 11102 raft_consensus.cc:2468] T acc241e7273d49149984968596242712 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 3.
I20260709 13:04:26.497077 11028 raft_consensus.cc:1275] T 30f628e3d7914d609e7f1d980a74c654 P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Refusing update from remote peer 10ca5ee68ad64b3283b9d235352c9d23: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20260709 13:04:26.515954 11069 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:38967, user_credentials={real_user=slave}} blocked reactor thread for 96328us
I20260709 13:04:26.522254 11098 raft_consensus.cc:3060] T d7ce47b9a92d43dd88d95bcda5038597 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:04:26.386041 10951 raft_consensus.cc:2468] T 836251ebc2b74f5da88b8a8b2338e2a5 P 10ca5ee68ad64b3283b9d235352c9d23 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 6.
I20260709 13:04:26.525014 11067 leader_election.cc:304] T a64b7e99d6c54934a7de6f6dffcf5e94 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:04:26.530659 12053 raft_consensus.cc:3060] T 9a1716c0adf34bee8a4e99341ba5788f P 873357742d5843579556ee8166f23f47 [term 5 FOLLOWER]: Advancing to term 6
W20260709 13:04:26.534114 11066 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 56083.1us
I20260709 13:04:26.382905 12694 consensus_queue.cc:237] T 11de93cd9b9d457489ff1cb80f976c9a P 186b12638a87410487467df721a3cbda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:26.534693 12654 raft_consensus.cc:2749] T a154cc8b181f460abf4880064379948b P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 13:04:26.537770 11099 raft_consensus.cc:2468] T a977842ba4b744f4bb2febd55d286006 P 0172fed22b6b4890a7fdc953fa128af4 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 5.
I20260709 13:04:26.495390 12390 raft_consensus.cc:493] T c31881534c5e4f0089c82057c1746652 P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 10ca5ee68ad64b3283b9d235352c9d23)
I20260709 13:04:26.550132 11069 leader_election.cc:304] T d270b9e5481245d9b5340a454a7cd10b P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda
I20260709 13:04:26.550910 12669 consensus_queue.cc:1059] T 0e5d327630e5438b878035d28efe6c0c P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 13:04:26.552043 10920 leader_election.cc:304] T e5ba72ccd854430baadbc636dfd17136 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 10ca5ee68ad64b3283b9d235352c9d23; no voters: 873357742d5843579556ee8166f23f47
I20260709 13:04:26.523180 12702 consensus_queue.cc:1048] T 1e4f2241a94a41f38e481137f868b83d P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.012s
W20260709 13:04:26.555879 10826 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 13:04:26.558509 11094 raft_consensus.cc:1240] T a610d1a71f52432e88fe56130f921fa2 P 0172fed22b6b4890a7fdc953fa128af4 [term 6 FOLLOWER]: Rejecting Update request from peer 873357742d5843579556ee8166f23f47 for earlier term 5. Current term is 6. Ops: []
I20260709 13:04:26.524165 12691 raft_consensus.cc:2749] T 4f9eb06ed01f4ce68be3af8a8434665b P 0172fed22b6b4890a7fdc953fa128af4 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 13:04:26.532255 12679 raft_consensus.cc:3060] T 4f9eb06ed01f4ce68be3af8a8434665b P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:04:26.569388 10921 leader_election.cc:304] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda; no voters: 
I20260709 13:04:26.589072 11103 raft_consensus.cc:1240] T a154cc8b181f460abf4880064379948b P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Rejecting Update request from peer 873357742d5843579556ee8166f23f47 for earlier term 2. Current term is 3. Ops: []
I20260709 13:04:26.571768 11102 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d156d6ae2ca4fa59a1e387edf0fea39" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:26.574716 10954 raft_consensus.cc:1240] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 10ca5ee68ad64b3283b9d235352c9d23 [term 8 FOLLOWER]: Rejecting Update request from peer 873357742d5843579556ee8166f23f47 for earlier term 7. Current term is 8. Ops: []
I20260709 13:04:26.588943 11069 leader_election.cc:304] T 6a9353d1d230416296627d7cf8b1d425 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4; no voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47
I20260709 13:04:26.591517 11102 raft_consensus.cc:2468] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 0172fed22b6b4890a7fdc953fa128af4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 5.
I20260709 13:04:26.571065 11099 raft_consensus.cc:1275] T 7709387f4d3d4f9992d97a0fbca39a40 P 0172fed22b6b4890a7fdc953fa128af4 [term 8 FOLLOWER]: Refusing update from remote peer 873357742d5843579556ee8166f23f47: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
I20260709 13:04:26.523131 12678 consensus_queue.cc:1048] T 868e92339c2f4508af0afbd07c814d64 P 186b12638a87410487467df721a3cbda [LEADER]: Connected to new peer: Peer: permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.050s
I20260709 13:04:26.596745 11069 leader_election.cc:304] T 0f9df2bc281f4866b6da5da94a0defc7 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda
I20260709 13:04:26.606652 11098 raft_consensus.cc:2468] T d7ce47b9a92d43dd88d95bcda5038597 P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 4.
W20260709 13:04:26.609795 10992 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:26.567974 12581 raft_consensus.cc:493] T 72b111ad7b5f4753ab1d2b5ee1114840 P 10ca5ee68ad64b3283b9d235352c9d23 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:26.617522 11069 leader_election.cc:304] T e5a5f5a608504650b48fd27f163faed6 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda; no voters: 10ca5ee68ad64b3283b9d235352c9d23
I20260709 13:04:26.569506 12709 raft_consensus.cc:2804] T a64b7e99d6c54934a7de6f6dffcf5e94 P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:04:26.618748 12537 raft_consensus.cc:515] T c51782cd9d064ec0949e84e24dd4de67 P 186b12638a87410487467df721a3cbda [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:26.626713 11069 leader_election.cc:304] T 72b111ad7b5f4753ab1d2b5ee1114840 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda; no voters: 
I20260709 13:04:26.585091 12390 raft_consensus.cc:515] T c31881534c5e4f0089c82057c1746652 P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:26.627424 12513 raft_consensus.cc:2804] T e5a5f5a608504650b48fd27f163faed6 P 0172fed22b6b4890a7fdc953fa128af4 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 13:04:26.607745 12550 consensus_queue.cc:1048] T 1e4f2241a94a41f38e481137f868b83d P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.064s
I20260709 13:04:26.672964 12390 leader_election.cc:290] T c31881534c5e4f0089c82057c1746652 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:26.575657 12578 consensus_queue.cc:1048] T 30f628e3d7914d609e7f1d980a74c654 P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.055s
I20260709 13:04:26.673501 12550 raft_consensus.cc:2804] T e5ba72ccd854430baadbc636dfd17136 P 10ca5ee68ad64b3283b9d235352c9d23 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:04:26.675393 12578 raft_consensus.cc:2804] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 10ca5ee68ad64b3283b9d235352c9d23 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:04:26.675907 12578 raft_consensus.cc:493] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 10ca5ee68ad64b3283b9d235352c9d23 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:26.676297 12578 raft_consensus.cc:3060] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 10ca5ee68ad64b3283b9d235352c9d23 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:04:26.677299 12537 leader_election.cc:290] T c51782cd9d064ec0949e84e24dd4de67 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 6 election: Requested vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:26.607733 12654 raft_consensus.cc:2749] T 6a9353d1d230416296627d7cf8b1d425 P 0172fed22b6b4890a7fdc953fa128af4 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:04:26.572755 12644 consensus_queue.cc:1048] T 2567c2d84e4648d7ba5dc23a3df32817 P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.053s
W20260709 13:04:26.684626 10843 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 334340us
W20260709 13:04:26.649411 10918 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:26.692227 11069 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:38967, user_credentials={real_user=slave}} blocked reactor thread for 65636.5us
I20260709 13:04:26.596736 12387 raft_consensus.cc:2749] T d270b9e5481245d9b5340a454a7cd10b P 0172fed22b6b4890a7fdc953fa128af4 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 13:04:26.701097 11102 raft_consensus.cc:1275] T 868e92339c2f4508af0afbd07c814d64 P 0172fed22b6b4890a7fdc953fa128af4 [term 5 FOLLOWER]: Refusing update from remote peer 186b12638a87410487467df721a3cbda: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260709 13:04:26.702673 10871 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d7ce47b9a92d43dd88d95bcda5038597" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda"
I20260709 13:04:26.620594 12671 raft_consensus.cc:2749] T 0f9df2bc281f4866b6da5da94a0defc7 P 0172fed22b6b4890a7fdc953fa128af4 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:04:26.679286 12575 raft_consensus.cc:2749] T d270b9e5481245d9b5340a454a7cd10b P 10ca5ee68ad64b3283b9d235352c9d23 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:04:26.625247 12709 raft_consensus.cc:493] T a64b7e99d6c54934a7de6f6dffcf5e94 P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:26.615393 12570 consensus_queue.cc:1059] T 836251ebc2b74f5da88b8a8b2338e2a5 P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.114s
I20260709 13:04:26.737659 10878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c31881534c5e4f0089c82057c1746652" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:26.674248 12550 raft_consensus.cc:697] T e5ba72ccd854430baadbc636dfd17136 P 10ca5ee68ad64b3283b9d235352c9d23 [term 7 LEADER]: Becoming Leader. State: Replica: 10ca5ee68ad64b3283b9d235352c9d23, State: Running, Role: LEADER
I20260709 13:04:26.625041 12612 consensus_queue.cc:1048] T c51782cd9d064ec0949e84e24dd4de67 P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.119s
I20260709 13:04:26.722484 12390 raft_consensus.cc:2804] T 72b111ad7b5f4753ab1d2b5ee1114840 P 0172fed22b6b4890a7fdc953fa128af4 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:04:26.749943 10950 raft_consensus.cc:1240] T d270b9e5481245d9b5340a454a7cd10b P 10ca5ee68ad64b3283b9d235352c9d23 [term 7 FOLLOWER]: Rejecting Update request from peer 186b12638a87410487467df721a3cbda for earlier term 6. Current term is 7. Ops: []
I20260709 13:04:26.627698 12581 raft_consensus.cc:515] T 72b111ad7b5f4753ab1d2b5ee1114840 P 10ca5ee68ad64b3283b9d235352c9d23 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:26.737006 10871 raft_consensus.cc:2468] T d7ce47b9a92d43dd88d95bcda5038597 P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 3.
I20260709 13:04:26.744419 12709 raft_consensus.cc:3060] T a64b7e99d6c54934a7de6f6dffcf5e94 P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Advancing to term 5
W20260709 13:04:26.758852 12718 log.cc:927] Time spent T d71ca3a1ba6b48478aaf62741bf3cf6a P 10ca5ee68ad64b3283b9d235352c9d23: Append to log took a long time: real 0.054s	user 0.001s	sys 0.000s
I20260709 13:04:26.741747 12537 raft_consensus.cc:493] T e96e80ba9ac34d0998395515e5c8c41a P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:26.797371 10949 raft_consensus.cc:1240] T f34f2bc02d85444f9096e0f5b4fe4331 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 LEADER]: Rejecting Update request from peer 186b12638a87410487467df721a3cbda for earlier term 2. Current term is 3. Ops: []
I20260709 13:04:26.760809 12390 raft_consensus.cc:493] T 72b111ad7b5f4753ab1d2b5ee1114840 P 0172fed22b6b4890a7fdc953fa128af4 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:26.807687 11099 raft_consensus.cc:1240] T f34f2bc02d85444f9096e0f5b4fe4331 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Rejecting Update request from peer 186b12638a87410487467df721a3cbda for earlier term 2. Current term is 3. Ops: []
I20260709 13:04:26.810159 11099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c51782cd9d064ec0949e84e24dd4de67" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:04:26.815918 10845 leader_election.cc:304] T 401a9cf11eac479aa445ec8c1d6e33b2 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 186b12638a87410487467df721a3cbda; no voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47
I20260709 13:04:26.822834 10845 leader_election.cc:304] T 0e5d327630e5438b878035d28efe6c0c P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 186b12638a87410487467df721a3cbda; no voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47
I20260709 13:04:26.804795 12390 raft_consensus.cc:3060] T 72b111ad7b5f4753ab1d2b5ee1114840 P 0172fed22b6b4890a7fdc953fa128af4 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:04:26.803601 12537 raft_consensus.cc:515] T e96e80ba9ac34d0998395515e5c8c41a P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:26.830662 12487 raft_consensus.cc:2749] T 401a9cf11eac479aa445ec8c1d6e33b2 P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:04:26.840021 12488 raft_consensus.cc:2749] T 0e5d327630e5438b878035d28efe6c0c P 186b12638a87410487467df721a3cbda [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 13:04:26.811223 12550 consensus_queue.cc:237] T e5ba72ccd854430baadbc636dfd17136 P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:26.849684 12585 consensus_queue.cc:1059] T d270b9e5481245d9b5340a454a7cd10b P 186b12638a87410487467df721a3cbda [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.029s
I20260709 13:04:26.880751 12709 raft_consensus.cc:515] T a64b7e99d6c54934a7de6f6dffcf5e94 P 0172fed22b6b4890a7fdc953fa128af4 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:26.885560 12705 leader_election.cc:290] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 8 election: Requested vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:26.889541 12660 consensus_queue.cc:1059] T f34f2bc02d85444f9096e0f5b4fe4331 P 186b12638a87410487467df721a3cbda [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:04:26.895460 12558 raft_consensus.cc:2749] T 8a69f32c53f24a50b7852d143665d7b4 P 10ca5ee68ad64b3283b9d235352c9d23 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:04:26.905164 12488 raft_consensus.cc:3055] T d270b9e5481245d9b5340a454a7cd10b P 186b12638a87410487467df721a3cbda [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:04:26.906965 10878 raft_consensus.cc:1240] T c51782cd9d064ec0949e84e24dd4de67 P 186b12638a87410487467df721a3cbda [term 6 FOLLOWER]: Rejecting Update request from peer 0172fed22b6b4890a7fdc953fa128af4 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
W20260709 13:04:26.909999 10921 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:38967, user_credentials={real_user=slave}} blocked reactor thread for 340691us
I20260709 13:04:26.916394 10875 raft_consensus.cc:1240] T 0e5d327630e5438b878035d28efe6c0c P 186b12638a87410487467df721a3cbda [term 6 FOLLOWER]: Rejecting Update request from peer 0172fed22b6b4890a7fdc953fa128af4 for earlier term 5. Current term is 6. Ops: []
W20260709 13:04:26.907089 12720 log.cc:927] Time spent T a1ec2b50c6ad4f7c8033476208abb26d P 0172fed22b6b4890a7fdc953fa128af4: Append to log took a long time: real 0.202s	user 0.000s	sys 0.002s
I20260709 13:04:26.905762 12488 raft_consensus.cc:740] T d270b9e5481245d9b5340a454a7cd10b P 186b12638a87410487467df721a3cbda [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 186b12638a87410487467df721a3cbda, State: Running, Role: LEADER
I20260709 13:04:26.913991 12513 raft_consensus.cc:697] T e5a5f5a608504650b48fd27f163faed6 P 0172fed22b6b4890a7fdc953fa128af4 [term 6 LEADER]: Becoming Leader. State: Replica: 0172fed22b6b4890a7fdc953fa128af4, State: Running, Role: LEADER
I20260709 13:04:26.928647 10994 leader_election.cc:304] T 8c160ddfe9b34bdaa3835d0915c1e06f P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 873357742d5843579556ee8166f23f47; no voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda
I20260709 13:04:26.930249 10994 leader_election.cc:304] T d71ca3a1ba6b48478aaf62741bf3cf6a P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47; no voters: 10ca5ee68ad64b3283b9d235352c9d23
I20260709 13:04:26.931618 12450 raft_consensus.cc:2749] T 8c160ddfe9b34bdaa3835d0915c1e06f P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260709 13:04:26.931877 12723 log.cc:927] Time spent T f371d2565dd84667b5613848604949bb P 0172fed22b6b4890a7fdc953fa128af4: Append to log took a long time: real 0.122s	user 0.000s	sys 0.000s
I20260709 13:04:26.932241 10994 leader_election.cc:304] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 873357742d5843579556ee8166f23f47; no voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda
I20260709 13:04:26.932456 12450 raft_consensus.cc:2804] T d71ca3a1ba6b48478aaf62741bf3cf6a P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:04:26.931941 12488 consensus_queue.cc:260] T d270b9e5481245d9b5340a454a7cd10b P 186b12638a87410487467df721a3cbda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:26.932936 12450 raft_consensus.cc:493] T d71ca3a1ba6b48478aaf62741bf3cf6a P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:26.935845 10994 leader_election.cc:304] T 525bfdbd6d934afd823f7d3ee3f9d052 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 873357742d5843579556ee8166f23f47; no voters: 0172fed22b6b4890a7fdc953fa128af4, 10ca5ee68ad64b3283b9d235352c9d23
I20260709 13:04:26.868865 12662 consensus_queue.cc:1048] T 868e92339c2f4508af0afbd07c814d64 P 186b12638a87410487467df721a3cbda [LEADER]: Connected to new peer: Peer: permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.055s
I20260709 13:04:26.908963 12517 raft_consensus.cc:515] T b67eff0634e540eab91cf8cee795969f P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:26.933498 12450 raft_consensus.cc:3060] T d71ca3a1ba6b48478aaf62741bf3cf6a P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:04:26.927716 12558 raft_consensus.cc:493] T b4cd3d7dfbbf4096908ff11e9dc57141 P 10ca5ee68ad64b3283b9d235352c9d23 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:26.962528 10994 leader_election.cc:304] T e5ba72ccd854430baadbc636dfd17136 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 873357742d5843579556ee8166f23f47; no voters: 0172fed22b6b4890a7fdc953fa128af4, 10ca5ee68ad64b3283b9d235352c9d23
I20260709 13:04:26.965854 12418 raft_consensus.cc:2749] T 525bfdbd6d934afd823f7d3ee3f9d052 P 873357742d5843579556ee8166f23f47 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20260709 13:04:26.976594 10920 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 424640us
I20260709 13:04:26.933568 12488 raft_consensus.cc:3060] T d270b9e5481245d9b5340a454a7cd10b P 186b12638a87410487467df721a3cbda [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:04:26.982016 10920 leader_election.cc:304] T bac7e5040f984ba2b7146bd36af7267e P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 10ca5ee68ad64b3283b9d235352c9d23; no voters: 873357742d5843579556ee8166f23f47
I20260709 13:04:26.929116 12397 raft_consensus.cc:3055] T f34f2bc02d85444f9096e0f5b4fe4331 P 186b12638a87410487467df721a3cbda [term 2 LEADER]: Stepping down as leader of term 2
W20260709 13:04:26.963896 12719 log.cc:927] Time spent T 7709387f4d3d4f9992d97a0fbca39a40 P 186b12638a87410487467df721a3cbda: Append to log took a long time: real 0.205s	user 0.000s	sys 0.001s
I20260709 13:04:26.988220 10994 leader_election.cc:304] T acc241e7273d49149984968596242712 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47; no voters: 186b12638a87410487467df721a3cbda
I20260709 13:04:26.927253 12584 consensus_queue.cc:1059] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 873357742d5843579556ee8166f23f47 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.056s
I20260709 13:04:26.954586 12528 raft_consensus.cc:2749] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 873357742d5843579556ee8166f23f47 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 13:04:26.992270 10994 leader_election.cc:304] T a977842ba4b744f4bb2febd55d286006 P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47; no voters: 10ca5ee68ad64b3283b9d235352c9d23
I20260709 13:04:26.988293 11027 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f1466124dbd4f6ebb3c4d62ca48bf96" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
I20260709 13:04:26.981716 12569 raft_consensus.cc:2749] T e5ba72ccd854430baadbc636dfd17136 P 873357742d5843579556ee8166f23f47 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 13:04:27.010869 12584 raft_consensus.cc:3055] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 873357742d5843579556ee8166f23f47 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:04:26.996821 12534 raft_consensus.cc:2804] T acc241e7273d49149984968596242712 P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:04:27.011329 12584 raft_consensus.cc:740] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 873357742d5843579556ee8166f23f47 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 873357742d5843579556ee8166f23f47, State: Running, Role: LEADER
I20260709 13:04:27.011866 12534 raft_consensus.cc:697] T acc241e7273d49149984968596242712 P 873357742d5843579556ee8166f23f47 [term 3 LEADER]: Becoming Leader. State: Replica: 873357742d5843579556ee8166f23f47, State: Running, Role: LEADER
I20260709 13:04:27.011490 12513 consensus_queue.cc:237] T e5a5f5a608504650b48fd27f163faed6 P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:27.012349 12584 consensus_queue.cc:260] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 873357742d5843579556ee8166f23f47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:27.012864 12534 consensus_queue.cc:237] T acc241e7273d49149984968596242712 P 873357742d5843579556ee8166f23f47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:27.013783 12584 raft_consensus.cc:3060] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 873357742d5843579556ee8166f23f47 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:04:27.016897 10949 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d7ce47b9a92d43dd88d95bcda5038597" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:27.017711 10949 raft_consensus.cc:2368] T d7ce47b9a92d43dd88d95bcda5038597 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0172fed22b6b4890a7fdc953fa128af4 for earlier term 3. Current term is 4.
W20260709 13:04:27.002903 12731 log.cc:927] Time spent T f34f2bc02d85444f9096e0f5b4fe4331 P 10ca5ee68ad64b3283b9d235352c9d23: Append to log took a long time: real 0.074s	user 0.000s	sys 0.001s
W20260709 13:04:27.004235 12734 log.cc:927] Time spent T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 0172fed22b6b4890a7fdc953fa128af4: Append to log took a long time: real 0.051s	user 0.000s	sys 0.001s
I20260709 13:04:27.007323 12537 leader_election.cc:290] T e96e80ba9ac34d0998395515e5c8c41a P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:27.008563 12053 raft_consensus.cc:2749] T 9a1716c0adf34bee8a4e99341ba5788f P 873357742d5843579556ee8166f23f47 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:04:27.009133 12738 raft_consensus.cc:2804] T bac7e5040f984ba2b7146bd36af7267e P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:04:27.009553 12671 raft_consensus.cc:3055] T 836251ebc2b74f5da88b8a8b2338e2a5 P 0172fed22b6b4890a7fdc953fa128af4 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:04:26.989187 12397 raft_consensus.cc:740] T f34f2bc02d85444f9096e0f5b4fe4331 P 186b12638a87410487467df721a3cbda [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 186b12638a87410487467df721a3cbda, State: Running, Role: LEADER
I20260709 13:04:27.019865 10951 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a64b7e99d6c54934a7de6f6dffcf5e94" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:27.040789 10951 raft_consensus.cc:2393] T a64b7e99d6c54934a7de6f6dffcf5e94 P 10ca5ee68ad64b3283b9d235352c9d23 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0172fed22b6b4890a7fdc953fa128af4 in current term 5: Already voted for candidate 10ca5ee68ad64b3283b9d235352c9d23 in this term.
I20260709 13:04:27.021934 10947 raft_consensus.cc:1240] T 836251ebc2b74f5da88b8a8b2338e2a5 P 10ca5ee68ad64b3283b9d235352c9d23 [term 6 FOLLOWER]: Rejecting Update request from peer 0172fed22b6b4890a7fdc953fa128af4 for earlier term 5. Current term is 6. Ops: []
I20260709 13:04:27.023952 10955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72b111ad7b5f4753ab1d2b5ee1114840" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:27.041739 10843 leader_election.cc:304] T 836251ebc2b74f5da88b8a8b2338e2a5 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda; no voters: 0172fed22b6b4890a7fdc953fa128af4
I20260709 13:04:27.041996 10955 raft_consensus.cc:2468] T 72b111ad7b5f4753ab1d2b5ee1114840 P 10ca5ee68ad64b3283b9d235352c9d23 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 5.
I20260709 13:04:27.024947 10949 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "525bfdbd6d934afd823f7d3ee3f9d052" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:27.044417 11101 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f1466124dbd4f6ebb3c4d62ca48bf96" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:04:27.025828 10946 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d7ce47b9a92d43dd88d95bcda5038597" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:04:27.026747 10953 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c31881534c5e4f0089c82057c1746652" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:27.045897 10946 raft_consensus.cc:2368] T d7ce47b9a92d43dd88d95bcda5038597 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 0172fed22b6b4890a7fdc953fa128af4 for earlier term 3. Current term is 4.
I20260709 13:04:27.044672 10949 raft_consensus.cc:2468] T 525bfdbd6d934afd823f7d3ee3f9d052 P 10ca5ee68ad64b3283b9d235352c9d23 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 6.
I20260709 13:04:27.048200 10948 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a977842ba4b744f4bb2febd55d286006" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:27.048635 12517 leader_election.cc:290] T b67eff0634e540eab91cf8cee795969f P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 4 election: Requested vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:27.049162 10948 raft_consensus.cc:2468] T a977842ba4b744f4bb2febd55d286006 P 10ca5ee68ad64b3283b9d235352c9d23 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 5.
I20260709 13:04:26.986001 12558 raft_consensus.cc:515] T b4cd3d7dfbbf4096908ff11e9dc57141 P 10ca5ee68ad64b3283b9d235352c9d23 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:27.055104 10954 raft_consensus.cc:1275] T c51782cd9d064ec0949e84e24dd4de67 P 10ca5ee68ad64b3283b9d235352c9d23 [term 5 FOLLOWER]: Refusing update from remote peer 0172fed22b6b4890a7fdc953fa128af4: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260709 13:04:27.052695 11066 leader_election.cc:400] T d7ce47b9a92d43dd88d95bcda5038597 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903) with higher term. Message: Invalid argument: T d7ce47b9a92d43dd88d95bcda5038597 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0172fed22b6b4890a7fdc953fa128af4 for earlier term 3. Current term is 4.
I20260709 13:04:27.057633 11101 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72b111ad7b5f4753ab1d2b5ee1114840" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:27.037534 12740 raft_consensus.cc:2804] T a977842ba4b744f4bb2febd55d286006 P 873357742d5843579556ee8166f23f47 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:04:27.065399 11097 raft_consensus.cc:1275] T d71ca3a1ba6b48478aaf62741bf3cf6a P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Refusing update from remote peer 10ca5ee68ad64b3283b9d235352c9d23: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 13:04:27.065752 12740 raft_consensus.cc:697] T a977842ba4b744f4bb2febd55d286006 P 873357742d5843579556ee8166f23f47 [term 5 LEADER]: Becoming Leader. State: Replica: 873357742d5843579556ee8166f23f47, State: Running, Role: LEADER
I20260709 13:04:27.054694 12531 consensus_queue.cc:1048] T 7709387f4d3d4f9992d97a0fbca39a40 P 873357742d5843579556ee8166f23f47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.012s
I20260709 13:04:27.066908 12740 consensus_queue.cc:237] T a977842ba4b744f4bb2febd55d286006 P 873357742d5843579556ee8166f23f47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:27.068389 12450 raft_consensus.cc:515] T d71ca3a1ba6b48478aaf62741bf3cf6a P 873357742d5843579556ee8166f23f47 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:27.069795 11021 raft_consensus.cc:1240] T d71ca3a1ba6b48478aaf62741bf3cf6a P 873357742d5843579556ee8166f23f47 [term 5 FOLLOWER]: Rejecting Update request from peer 10ca5ee68ad64b3283b9d235352c9d23 for earlier term 4. Current term is 5. Ops: []
    @     0x7f4263299dad testing::UnitTest::Run() at ??:0
I20260709 13:04:27.038300 12738 raft_consensus.cc:493] T bac7e5040f984ba2b7146bd36af7267e P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:27.073978 12738 raft_consensus.cc:3060] T bac7e5040f984ba2b7146bd36af7267e P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Advancing to term 3
    @           0x38f0bc RUN_ALL_TESTS() at /apache/dev/git/kudu/thirdparty/installed/tsan/include/gtest/gtest.h:?
I20260709 13:04:27.058739 12662 raft_consensus.cc:2804] T 836251ebc2b74f5da88b8a8b2338e2a5 P 186b12638a87410487467df721a3cbda [term 6 FOLLOWER]: Leader election won for term 6
I20260709 13:04:27.075990 12662 raft_consensus.cc:697] T 836251ebc2b74f5da88b8a8b2338e2a5 P 186b12638a87410487467df721a3cbda [term 6 LEADER]: Becoming Leader. State: Replica: 186b12638a87410487467df721a3cbda, State: Running, Role: LEADER
    @           0x38dcbd main at /apache/dev/git/kudu/src/kudu/util/test_main.cc:?
I20260709 13:04:27.076550 12390 raft_consensus.cc:515] T 72b111ad7b5f4753ab1d2b5ee1114840 P 0172fed22b6b4890a7fdc953fa128af4 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:27.077071 12662 consensus_queue.cc:237] T 836251ebc2b74f5da88b8a8b2338e2a5 P 186b12638a87410487467df721a3cbda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:27.038651 12671 raft_consensus.cc:740] T 836251ebc2b74f5da88b8a8b2338e2a5 P 0172fed22b6b4890a7fdc953fa128af4 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 0172fed22b6b4890a7fdc953fa128af4, State: Running, Role: LEADER
I20260709 13:04:27.050552 12569 consensus_queue.cc:1059] T a154cc8b181f460abf4880064379948b P 873357742d5843579556ee8166f23f47 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.029s
I20260709 13:04:27.079686 12671 consensus_queue.cc:260] T 836251ebc2b74f5da88b8a8b2338e2a5 P 0172fed22b6b4890a7fdc953fa128af4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:27.080307 12569 raft_consensus.cc:3055] T a154cc8b181f460abf4880064379948b P 873357742d5843579556ee8166f23f47 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:04:27.080904 12569 raft_consensus.cc:740] T a154cc8b181f460abf4880064379948b P 873357742d5843579556ee8166f23f47 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 873357742d5843579556ee8166f23f47, State: Running, Role: LEADER
I20260709 13:04:27.061959 11096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f9df2bc281f4866b6da5da94a0defc7" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:04:27.083040 12558 leader_election.cc:290] T b4cd3d7dfbbf4096908ff11e9dc57141 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:27.058399 12728 consensus_queue.cc:1059] T 0f9df2bc281f4866b6da5da94a0defc7 P 186b12638a87410487467df721a3cbda [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.025s
I20260709 13:04:27.083590 12450 leader_election.cc:290] T d71ca3a1ba6b48478aaf62741bf3cf6a P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 5 election: Requested vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:27.083106 10952 raft_consensus.cc:1275] T b4cd3d7dfbbf4096908ff11e9dc57141 P 10ca5ee68ad64b3283b9d235352c9d23 [term 5 FOLLOWER]: Refusing update from remote peer 0172fed22b6b4890a7fdc953fa128af4: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260709 13:04:27.087191 11101 raft_consensus.cc:2393] T 72b111ad7b5f4753ab1d2b5ee1114840 P 0172fed22b6b4890a7fdc953fa128af4 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 10ca5ee68ad64b3283b9d235352c9d23 in current term 6: Already voted for candidate 0172fed22b6b4890a7fdc953fa128af4 in this term.
I20260709 13:04:27.081789 12569 consensus_queue.cc:260] T a154cc8b181f460abf4880064379948b P 873357742d5843579556ee8166f23f47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:27.092861 10951 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a64b7e99d6c54934a7de6f6dffcf5e94" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:04:27.093737 10951 raft_consensus.cc:2393] T a64b7e99d6c54934a7de6f6dffcf5e94 P 10ca5ee68ad64b3283b9d235352c9d23 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 0172fed22b6b4890a7fdc953fa128af4 in current term 5: Already voted for candidate 10ca5ee68ad64b3283b9d235352c9d23 in this term.
I20260709 13:04:27.097762 11096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d71ca3a1ba6b48478aaf62741bf3cf6a" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:04:27.098699 11096 raft_consensus.cc:3060] T d71ca3a1ba6b48478aaf62741bf3cf6a P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:04:27.084303 12694 raft_consensus.cc:3055] T 0f9df2bc281f4866b6da5da94a0defc7 P 186b12638a87410487467df721a3cbda [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:04:27.100064 12694 raft_consensus.cc:740] T 0f9df2bc281f4866b6da5da94a0defc7 P 186b12638a87410487467df721a3cbda [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 186b12638a87410487467df721a3cbda, State: Running, Role: LEADER
I20260709 13:04:27.100998 12694 consensus_queue.cc:260] T 0f9df2bc281f4866b6da5da94a0defc7 P 186b12638a87410487467df721a3cbda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:27.103073 12694 raft_consensus.cc:3060] T 0f9df2bc281f4866b6da5da94a0defc7 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:04:27.104142 11100 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b67eff0634e540eab91cf8cee795969f" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
W20260709 13:04:27.118530 10843 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:27.062397 12660 consensus_queue.cc:1059] T f34f2bc02d85444f9096e0f5b4fe4331 P 186b12638a87410487467df721a3cbda [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.048s
I20260709 13:04:27.133193 11021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4cd3d7dfbbf4096908ff11e9dc57141" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:27.147390 11021 raft_consensus.cc:2468] T b4cd3d7dfbbf4096908ff11e9dc57141 P 873357742d5843579556ee8166f23f47 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 5.
I20260709 13:04:27.151973 11096 raft_consensus.cc:2468] T d71ca3a1ba6b48478aaf62741bf3cf6a P 0172fed22b6b4890a7fdc953fa128af4 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 5.
I20260709 13:04:27.134666 12672 raft_consensus.cc:3055] T 0e5d327630e5438b878035d28efe6c0c P 0172fed22b6b4890a7fdc953fa128af4 [term 5 LEADER]: Stepping down as leader of term 5
W20260709 13:04:27.136409 12736 log.cc:927] Time spent T 2567c2d84e4648d7ba5dc23a3df32817 P 186b12638a87410487467df721a3cbda: Append to log took a long time: real 0.082s	user 0.001s	sys 0.000s
I20260709 13:04:27.170710 10919 leader_election.cc:304] T b4cd3d7dfbbf4096908ff11e9dc57141 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:04:27.169490 12660 consensus_queue.cc:1243] T f34f2bc02d85444f9096e0f5b4fe4331 P 186b12638a87410487467df721a3cbda [LEADER]: Peer 10ca5ee68ad64b3283b9d235352c9d23 log is divergent from this leader: its last log entry 3.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260709 13:04:27.172411 12513 raft_consensus.cc:493] T 87cd8f5097624c8692f1d83544d8881e P 0172fed22b6b4890a7fdc953fa128af4 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:27.167061 12542 raft_consensus.cc:493] T e96e80ba9ac34d0998395515e5c8c41a P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:27.210857 12542 raft_consensus.cc:515] T e96e80ba9ac34d0998395515e5c8c41a P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:27.172215 12672 raft_consensus.cc:740] T 0e5d327630e5438b878035d28efe6c0c P 0172fed22b6b4890a7fdc953fa128af4 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 0172fed22b6b4890a7fdc953fa128af4, State: Running, Role: LEADER
I20260709 13:04:27.219290 11098 raft_consensus.cc:1275] T a610d1a71f52432e88fe56130f921fa2 P 0172fed22b6b4890a7fdc953fa128af4 [term 6 FOLLOWER]: Refusing update from remote peer 186b12638a87410487467df721a3cbda: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 13:04:27.198334 10950 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e96e80ba9ac34d0998395515e5c8c41a" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:27.198689 10955 raft_consensus.cc:1240] T b67eff0634e540eab91cf8cee795969f P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Rejecting Update request from peer 186b12638a87410487467df721a3cbda for earlier term 3. Current term is 4. Ops: []
I20260709 13:04:27.201999 10949 raft_consensus.cc:1240] T 732c4adeeff6422497aa7db4a82f535b P 10ca5ee68ad64b3283b9d235352c9d23 [term 6 FOLLOWER]: Rejecting Update request from peer 186b12638a87410487467df721a3cbda for earlier term 5. Current term is 6. Ops: [3.3-5.4]
W20260709 13:04:27.203476 12578 consensus_meta.cc:287] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 10ca5ee68ad64b3283b9d235352c9d23: Time spent flushing consensus metadata: real 0.527s	user 0.015s	sys 0.003s
I20260709 13:04:27.182971 12564 raft_consensus.cc:493] T a5299b8d044f41b7b68057e63d17b159 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:27.180394 12558 raft_consensus.cc:2804] T b4cd3d7dfbbf4096908ff11e9dc57141 P 10ca5ee68ad64b3283b9d235352c9d23 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:04:27.177431 12550 consensus_queue.cc:1059] T d71ca3a1ba6b48478aaf62741bf3cf6a P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.031s
I20260709 13:04:27.197010 10920 leader_election.cc:304] T d7ce47b9a92d43dd88d95bcda5038597 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 10ca5ee68ad64b3283b9d235352c9d23; no voters: 186b12638a87410487467df721a3cbda
W20260709 13:04:27.228178 10995 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:38967, user_credentials={real_user=slave}} blocked reactor thread for 54468.2us
I20260709 13:04:27.224288 10954 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c51782cd9d064ec0949e84e24dd4de67" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:04:27.185094 12569 raft_consensus.cc:3060] T a154cc8b181f460abf4880064379948b P 873357742d5843579556ee8166f23f47 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:04:27.231946 10954 raft_consensus.cc:3060] T c51782cd9d064ec0949e84e24dd4de67 P 10ca5ee68ad64b3283b9d235352c9d23 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:04:27.221086 10950 raft_consensus.cc:2393] T e96e80ba9ac34d0998395515e5c8c41a P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 186b12638a87410487467df721a3cbda in current term 4: Already voted for candidate 10ca5ee68ad64b3283b9d235352c9d23 in this term.
I20260709 13:04:27.225590 12558 raft_consensus.cc:493] T b4cd3d7dfbbf4096908ff11e9dc57141 P 10ca5ee68ad64b3283b9d235352c9d23 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:27.239975 12558 raft_consensus.cc:3060] T b4cd3d7dfbbf4096908ff11e9dc57141 P 10ca5ee68ad64b3283b9d235352c9d23 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:04:27.242823 12550 raft_consensus.cc:3055] T d71ca3a1ba6b48478aaf62741bf3cf6a P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:04:27.243305 12550 raft_consensus.cc:740] T d71ca3a1ba6b48478aaf62741bf3cf6a P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 10ca5ee68ad64b3283b9d235352c9d23, State: Running, Role: LEADER
I20260709 13:04:27.244257 12550 consensus_queue.cc:260] T d71ca3a1ba6b48478aaf62741bf3cf6a P 10ca5ee68ad64b3283b9d235352c9d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:27.245499 12550 raft_consensus.cc:3060] T d71ca3a1ba6b48478aaf62741bf3cf6a P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:04:27.225898 12578 raft_consensus.cc:515] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 10ca5ee68ad64b3283b9d235352c9d23 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:27.253917 12578 leader_election.cc:290] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 6 election: Requested vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
W20260709 13:04:27.256291 10920 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 59417.3us
I20260709 13:04:27.257233 10920 leader_election.cc:304] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23; no voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47
I20260709 13:04:27.258934 10920 leader_election.cc:304] T 0f9df2bc281f4866b6da5da94a0defc7 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23; no voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda
I20260709 13:04:27.260026 12737 raft_consensus.cc:2749] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 10ca5ee68ad64b3283b9d235352c9d23 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 13:04:27.261391 12517 raft_consensus.cc:2749] T 0f9df2bc281f4866b6da5da94a0defc7 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 13:04:27.256433 12702 raft_consensus.cc:2804] T d7ce47b9a92d43dd88d95bcda5038597 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:04:27.264747 11066 leader_election.cc:304] T c31881534c5e4f0089c82057c1746652 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda
I20260709 13:04:27.267287 11066 leader_election.cc:304] T 525bfdbd6d934afd823f7d3ee3f9d052 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 10ca5ee68ad64b3283b9d235352c9d23; no voters: 873357742d5843579556ee8166f23f47
I20260709 13:04:27.271104 10950 raft_consensus.cc:3060] T 11de93cd9b9d457489ff1cb80f976c9a P 10ca5ee68ad64b3283b9d235352c9d23 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:04:27.271445 10955 raft_consensus.cc:1240] T f34f2bc02d85444f9096e0f5b4fe4331 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 LEADER]: Rejecting Update request from peer 186b12638a87410487467df721a3cbda for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260709 13:04:27.275885 11066 leader_election.cc:400] T d7ce47b9a92d43dd88d95bcda5038597 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 3 election: Vote denied by peer 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903) with higher term. Message: Invalid argument: T d7ce47b9a92d43dd88d95bcda5038597 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 0172fed22b6b4890a7fdc953fa128af4 for earlier term 3. Current term is 4.
I20260709 13:04:27.276358 11066 leader_election.cc:403] T d7ce47b9a92d43dd88d95bcda5038597 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20260709 13:04:27.277272 10954 raft_consensus.cc:2468] T c51782cd9d064ec0949e84e24dd4de67 P 10ca5ee68ad64b3283b9d235352c9d23 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 6.
W20260709 13:04:27.236589 12679 consensus_meta.cc:287] T 4f9eb06ed01f4ce68be3af8a8434665b P 186b12638a87410487467df721a3cbda: Time spent flushing consensus metadata: real 0.626s	user 0.007s	sys 0.010s
I20260709 13:04:27.279043 10843 leader_election.cc:304] T c51782cd9d064ec0949e84e24dd4de67 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 186b12638a87410487467df721a3cbda; no voters: 0172fed22b6b4890a7fdc953fa128af4
I20260709 13:04:27.236114 12696 raft_consensus.cc:3055] T 732c4adeeff6422497aa7db4a82f535b P 186b12638a87410487467df721a3cbda [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:04:27.218554 12513 raft_consensus.cc:515] T 87cd8f5097624c8692f1d83544d8881e P 0172fed22b6b4890a7fdc953fa128af4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:27.292901 11066 leader_election.cc:304] T a977842ba4b744f4bb2febd55d286006 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 10ca5ee68ad64b3283b9d235352c9d23; no voters: 873357742d5843579556ee8166f23f47
W20260709 13:04:27.296015 12722 log.cc:927] Time spent T b4cd3d7dfbbf4096908ff11e9dc57141 P 0172fed22b6b4890a7fdc953fa128af4: Append to log took a long time: real 0.463s	user 0.003s	sys 0.000s
I20260709 13:04:27.279064 12669 consensus_queue.cc:1048] T c51782cd9d064ec0949e84e24dd4de67 P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.018s
I20260709 13:04:27.289963 12678 consensus_queue.cc:1059] T f34f2bc02d85444f9096e0f5b4fe4331 P 186b12638a87410487467df721a3cbda [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.009s
I20260709 13:04:27.298825 12678 consensus_queue.cc:1243] T f34f2bc02d85444f9096e0f5b4fe4331 P 186b12638a87410487467df721a3cbda [LEADER]: Peer 10ca5ee68ad64b3283b9d235352c9d23 log is divergent from this leader: its last log entry 3.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260709 13:04:27.299255 12558 raft_consensus.cc:515] T b4cd3d7dfbbf4096908ff11e9dc57141 P 10ca5ee68ad64b3283b9d235352c9d23 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:27.299906 12738 raft_consensus.cc:515] T bac7e5040f984ba2b7146bd36af7267e P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:27.272045 12739 raft_consensus.cc:2749] T c31881534c5e4f0089c82057c1746652 P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:04:27.285194 12612 raft_consensus.cc:2804] T 525bfdbd6d934afd823f7d3ee3f9d052 P 0172fed22b6b4890a7fdc953fa128af4 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:04:27.229151 12564 raft_consensus.cc:515] T a5299b8d044f41b7b68057e63d17b159 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:27.284083 12702 raft_consensus.cc:697] T d7ce47b9a92d43dd88d95bcda5038597 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 LEADER]: Becoming Leader. State: Replica: 10ca5ee68ad64b3283b9d235352c9d23, State: Running, Role: LEADER
I20260709 13:04:27.289007 12488 raft_consensus.cc:2804] T c51782cd9d064ec0949e84e24dd4de67 P 186b12638a87410487467df721a3cbda [term 6 FOLLOWER]: Leader election won for term 6
I20260709 13:04:27.285076 12696 raft_consensus.cc:740] T 732c4adeeff6422497aa7db4a82f535b P 186b12638a87410487467df721a3cbda [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 186b12638a87410487467df721a3cbda, State: Running, Role: LEADER
I20260709 13:04:27.448355 12678 raft_consensus.cc:493] T 8a69f32c53f24a50b7852d143665d7b4 P 186b12638a87410487467df721a3cbda [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:27.312098 12570 raft_consensus.cc:2749] T d7ce47b9a92d43dd88d95bcda5038597 P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903) with higher term. Message: Invalid argument: T d7ce47b9a92d43dd88d95bcda5038597 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 0172fed22b6b4890a7fdc953fa128af4 for earlier term 3. Current term is 4.
I20260709 13:04:27.287379 12728 consensus_queue.cc:1059] T b67eff0634e540eab91cf8cee795969f P 186b12638a87410487467df721a3cbda [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.172s
I20260709 13:04:27.465819 11024 raft_consensus.cc:1275] T b4cd3d7dfbbf4096908ff11e9dc57141 P 873357742d5843579556ee8166f23f47 [term 5 FOLLOWER]: Refusing update from remote peer 0172fed22b6b4890a7fdc953fa128af4: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260709 13:04:27.477856 10878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e96e80ba9ac34d0998395515e5c8c41a" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:27.482764 10878 raft_consensus.cc:2468] T e96e80ba9ac34d0998395515e5c8c41a P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 3.
W20260709 13:04:27.458027 11066 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 165254us
I20260709 13:04:27.284370 12660 consensus_queue.cc:1059] T 732c4adeeff6422497aa7db4a82f535b P 186b12638a87410487467df721a3cbda [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.172s
I20260709 13:04:27.456108 12550 raft_consensus.cc:493] T e96e80ba9ac34d0998395515e5c8c41a P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:27.458701 12654 raft_consensus.cc:2764] T a977842ba4b744f4bb2febd55d286006 P 0172fed22b6b4890a7fdc953fa128af4 [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260709 13:04:27.302214 12738 leader_election.cc:290] T bac7e5040f984ba2b7146bd36af7267e P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 3 election: Requested vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:27.458984 12649 raft_consensus.cc:493] T 9a1716c0adf34bee8a4e99341ba5788f P 186b12638a87410487467df721a3cbda [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:27.280716 12672 consensus_queue.cc:260] T 0e5d327630e5438b878035d28efe6c0c P 0172fed22b6b4890a7fdc953fa128af4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:27.455475 12612 raft_consensus.cc:493] T 525bfdbd6d934afd823f7d3ee3f9d052 P 0172fed22b6b4890a7fdc953fa128af4 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 10ca5ee68ad64b3283b9d235352c9d23)
I20260709 13:04:27.459750 12739 raft_consensus.cc:493] T d371c5e27bd843469d2e2e453734cb4a P 0172fed22b6b4890a7fdc953fa128af4 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:27.480149 12250 raft_consensus.cc:493] T acc241e7273d49149984968596242712 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0172fed22b6b4890a7fdc953fa128af4)
I20260709 13:04:27.475183 11020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bac7e5040f984ba2b7146bd36af7267e" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
I20260709 13:04:27.493512 12678 raft_consensus.cc:515] T 8a69f32c53f24a50b7852d143665d7b4 P 186b12638a87410487467df721a3cbda [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:27.550119 12660 consensus_queue.cc:1243] T 732c4adeeff6422497aa7db4a82f535b P 186b12638a87410487467df721a3cbda [LEADER]: Peer 10ca5ee68ad64b3283b9d235352c9d23 log is divergent from this leader: its last log entry 4.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260709 13:04:27.559854 11023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a64b7e99d6c54934a7de6f6dffcf5e94" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
I20260709 13:04:27.594838 12612 raft_consensus.cc:3060] T 525bfdbd6d934afd823f7d3ee3f9d052 P 0172fed22b6b4890a7fdc953fa128af4 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:04:27.611330 12662 raft_consensus.cc:493] T d371c5e27bd843469d2e2e453734cb4a P 186b12638a87410487467df721a3cbda [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:27.642244 11066 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 157363us
W20260709 13:04:27.643963 11067 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:45807, user_credentials={real_user=slave}} blocked reactor thread for 158214us
I20260709 13:04:27.587011 12570 consensus_queue.cc:1048] T b4cd3d7dfbbf4096908ff11e9dc57141 P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.047s
I20260709 13:04:27.559379 12550 raft_consensus.cc:515] T e96e80ba9ac34d0998395515e5c8c41a P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
W20260709 13:04:27.678279 11066 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:27.589123 12649 raft_consensus.cc:515] T 9a1716c0adf34bee8a4e99341ba5788f P 186b12638a87410487467df721a3cbda [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:27.686877 11097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d156d6ae2ca4fa59a1e387edf0fea39" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:04:27.692569 11097 raft_consensus.cc:3060] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 0172fed22b6b4890a7fdc953fa128af4 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:04:27.637974 12578 raft_consensus.cc:493] T 732c4adeeff6422497aa7db4a82f535b P 10ca5ee68ad64b3283b9d235352c9d23 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:27.697084 11096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4cd3d7dfbbf4096908ff11e9dc57141" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:27.610333 12250 raft_consensus.cc:515] T acc241e7273d49149984968596242712 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:27.605193 12739 raft_consensus.cc:515] T d371c5e27bd843469d2e2e453734cb4a P 0172fed22b6b4890a7fdc953fa128af4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
W20260709 13:04:27.713310 11069 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:38967, user_credentials={real_user=slave}} blocked reactor thread for 151844us
I20260709 13:04:27.712805 11100 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bac7e5040f984ba2b7146bd36af7267e" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:04:27.688601 12756 raft_consensus.cc:493] T 6f46a5102fd64b45bf1c4dff863d2b4d P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:27.731146 10995 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:38967, user_credentials={real_user=slave}} blocked reactor thread for 251058us
I20260709 13:04:27.672883 12662 raft_consensus.cc:515] T d371c5e27bd843469d2e2e453734cb4a P 186b12638a87410487467df721a3cbda [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:27.756464 10873 raft_consensus.cc:3060] T acc241e7273d49149984968596242712 P 186b12638a87410487467df721a3cbda [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:04:27.728351 12558 leader_election.cc:290] T b4cd3d7dfbbf4096908ff11e9dc57141 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 6 election: Requested vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:27.764228 11024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e96e80ba9ac34d0998395515e5c8c41a" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:27.764930 10995 leader_election.cc:304] T e96e80ba9ac34d0998395515e5c8c41a P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:04:27.766664 11097 raft_consensus.cc:2468] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 0172fed22b6b4890a7fdc953fa128af4 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 6.
W20260709 13:04:27.765501 10921 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:38967, user_credentials={real_user=slave}} blocked reactor thread for 69724.9us
W20260709 13:04:27.776436 12759 log.cc:927] Time spent T 87cd8f5097624c8692f1d83544d8881e P 10ca5ee68ad64b3283b9d235352c9d23: Append to log took a long time: real 0.057s	user 0.000s	sys 0.001s
I20260709 13:04:27.777122 12739 leader_election.cc:290] T d371c5e27bd843469d2e2e453734cb4a P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
W20260709 13:04:27.779388 12768 log.cc:927] Time spent T 868e92339c2f4508af0afbd07c814d64 P 873357742d5843579556ee8166f23f47: Append to log took a long time: real 0.052s	user 0.000s	sys 0.000s
I20260709 13:04:27.779572 12662 leader_election.cc:290] T d371c5e27bd843469d2e2e453734cb4a P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:27.774399 11024 raft_consensus.cc:2468] T e96e80ba9ac34d0998395515e5c8c41a P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 4.
I20260709 13:04:27.713608 12578 raft_consensus.cc:515] T 732c4adeeff6422497aa7db4a82f535b P 10ca5ee68ad64b3283b9d235352c9d23 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:27.782424 11097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d371c5e27bd843469d2e2e453734cb4a" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:27.765514 11021 raft_consensus.cc:1275] T a610d1a71f52432e88fe56130f921fa2 P 873357742d5843579556ee8166f23f47 [term 6 FOLLOWER]: Refusing update from remote peer 186b12638a87410487467df721a3cbda: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 13:04:27.786358 11097 raft_consensus.cc:2468] T d371c5e27bd843469d2e2e453734cb4a P 0172fed22b6b4890a7fdc953fa128af4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 5.
I20260709 13:04:27.634040 12702 consensus_queue.cc:237] T d7ce47b9a92d43dd88d95bcda5038597 P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:27.734328 12488 raft_consensus.cc:697] T c51782cd9d064ec0949e84e24dd4de67 P 186b12638a87410487467df721a3cbda [term 6 LEADER]: Becoming Leader. State: Replica: 186b12638a87410487467df721a3cbda, State: Running, Role: LEADER
I20260709 13:04:27.762380 12564 leader_election.cc:290] T a5299b8d044f41b7b68057e63d17b159 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
W20260709 13:04:27.800029 12760 log.cc:927] Time spent T 836251ebc2b74f5da88b8a8b2338e2a5 P 186b12638a87410487467df721a3cbda: Append to log took a long time: real 0.067s	user 0.000s	sys 0.000s
I20260709 13:04:27.802596 12250 leader_election.cc:290] T acc241e7273d49149984968596242712 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:27.738495 12756 raft_consensus.cc:515] T 6f46a5102fd64b45bf1c4dff863d2b4d P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
W20260709 13:04:27.805270 12569 consensus_meta.cc:287] T a154cc8b181f460abf4880064379948b P 873357742d5843579556ee8166f23f47: Time spent flushing consensus metadata: real 0.567s	user 0.010s	sys 0.001s
I20260709 13:04:27.765503 11022 raft_consensus.cc:1275] T 11de93cd9b9d457489ff1cb80f976c9a P 873357742d5843579556ee8166f23f47 [term 6 FOLLOWER]: Refusing update from remote peer 186b12638a87410487467df721a3cbda: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 13:04:27.806588 12756 leader_election.cc:290] T 6f46a5102fd64b45bf1c4dff863d2b4d P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:27.807451 12550 leader_election.cc:290] T e96e80ba9ac34d0998395515e5c8c41a P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:27.808187 12488 consensus_queue.cc:237] T c51782cd9d064ec0949e84e24dd4de67 P 186b12638a87410487467df721a3cbda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:27.808990 12649 leader_election.cc:290] T 9a1716c0adf34bee8a4e99341ba5788f P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:27.784262 12517 raft_consensus.cc:493] T e89e640169e14e2d98f968f76027daa7 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:27.810288 12517 raft_consensus.cc:515] T e89e640169e14e2d98f968f76027daa7 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:27.810734 12581 leader_election.cc:290] T 72b111ad7b5f4753ab1d2b5ee1114840 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:27.760486 12654 consensus_queue.cc:1059] T c51782cd9d064ec0949e84e24dd4de67 P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.061s
I20260709 13:04:27.787606 11101 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a69f32c53f24a50b7852d143665d7b4" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
W20260709 13:04:27.761121 12694 consensus_meta.cc:287] T 0f9df2bc281f4866b6da5da94a0defc7 P 186b12638a87410487467df721a3cbda: Time spent flushing consensus metadata: real 0.646s	user 0.014s	sys 0.011s
I20260709 13:04:27.812189 11101 raft_consensus.cc:2468] T 8a69f32c53f24a50b7852d143665d7b4 P 0172fed22b6b4890a7fdc953fa128af4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 5.
I20260709 13:04:27.813622 11100 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acc241e7273d49149984968596242712" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:27.814395 11096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a1716c0adf34bee8a4e99341ba5788f" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:27.840689 11096 raft_consensus.cc:2468] T 9a1716c0adf34bee8a4e99341ba5788f P 0172fed22b6b4890a7fdc953fa128af4 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 6.
I20260709 13:04:27.814313 11097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f46a5102fd64b45bf1c4dff863d2b4d" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:27.830340 12654 raft_consensus.cc:3055] T c51782cd9d064ec0949e84e24dd4de67 P 0172fed22b6b4890a7fdc953fa128af4 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:04:27.849164 10994 leader_election.cc:304] T d71ca3a1ba6b48478aaf62741bf3cf6a P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47; no voters: 
W20260709 13:04:27.851879 10950 consensus_meta.cc:287] T 11de93cd9b9d457489ff1cb80f976c9a P 10ca5ee68ad64b3283b9d235352c9d23: Time spent flushing consensus metadata: real 0.568s	user 0.023s	sys 0.001s
I20260709 13:04:27.852977 10950 raft_consensus.cc:1275] T 11de93cd9b9d457489ff1cb80f976c9a P 10ca5ee68ad64b3283b9d235352c9d23 [term 6 FOLLOWER]: Refusing update from remote peer 186b12638a87410487467df721a3cbda: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
W20260709 13:04:27.826810 10921 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:38967, user_credentials={real_user=slave}} blocked reactor thread for 49609.1us
I20260709 13:04:27.831136 12671 raft_consensus.cc:3060] T 836251ebc2b74f5da88b8a8b2338e2a5 P 0172fed22b6b4890a7fdc953fa128af4 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:04:27.831667 11022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d371c5e27bd843469d2e2e453734cb4a" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:27.829434 12770 consensus_queue.cc:1048] T b4cd3d7dfbbf4096908ff11e9dc57141 P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.004s
I20260709 13:04:27.832810 10844 leader_election.cc:304] T e96e80ba9ac34d0998395515e5c8c41a P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 186b12638a87410487467df721a3cbda, 873357742d5843579556ee8166f23f47; no voters: 10ca5ee68ad64b3283b9d235352c9d23
I20260709 13:04:27.813766 12739 raft_consensus.cc:493] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 0172fed22b6b4890a7fdc953fa128af4 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 186b12638a87410487467df721a3cbda)
I20260709 13:04:27.833379 12649 raft_consensus.cc:493] T 4f9eb06ed01f4ce68be3af8a8434665b P 186b12638a87410487467df721a3cbda [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:27.863526 12649 raft_consensus.cc:515] T 4f9eb06ed01f4ce68be3af8a8434665b P 186b12638a87410487467df721a3cbda [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
W20260709 13:04:27.834880 11067 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:27.845003 11097 raft_consensus.cc:2468] T 6f46a5102fd64b45bf1c4dff863d2b4d P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 873357742d5843579556ee8166f23f47 in term 3.
I20260709 13:04:27.785063 12053 raft_consensus.cc:2804] T e96e80ba9ac34d0998395515e5c8c41a P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:04:27.806180 12777 raft_consensus.cc:493] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 186b12638a87410487467df721a3cbda [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:27.786783 12513 leader_election.cc:290] T 87cd8f5097624c8692f1d83544d8881e P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:27.861995 12779 raft_consensus.cc:2804] T d71ca3a1ba6b48478aaf62741bf3cf6a P 873357742d5843579556ee8166f23f47 [term 5 FOLLOWER]: Leader election won for term 5
W20260709 13:04:27.913121 10995 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:38967, user_credentials={real_user=slave}} blocked reactor thread for 54005.7us
W20260709 13:04:27.918048 10844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:45807, user_credentials={real_user=slave}} blocked reactor thread for 49826.6us
I20260709 13:04:27.848367 12654 raft_consensus.cc:740] T c51782cd9d064ec0949e84e24dd4de67 P 0172fed22b6b4890a7fdc953fa128af4 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 0172fed22b6b4890a7fdc953fa128af4, State: Running, Role: LEADER
I20260709 13:04:27.924890 11021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4cd3d7dfbbf4096908ff11e9dc57141" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47"
I20260709 13:04:27.925788 11021 raft_consensus.cc:3060] T b4cd3d7dfbbf4096908ff11e9dc57141 P 873357742d5843579556ee8166f23f47 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:04:27.917191 12250 raft_consensus.cc:2804] T e96e80ba9ac34d0998395515e5c8c41a P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:04:27.929944 12250 raft_consensus.cc:493] T e96e80ba9ac34d0998395515e5c8c41a P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Starting leader election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:27.866715 11022 raft_consensus.cc:2468] T d371c5e27bd843469d2e2e453734cb4a P 873357742d5843579556ee8166f23f47 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 5.
I20260709 13:04:27.865854 12769 raft_consensus.cc:493] T 11de93cd9b9d457489ff1cb80f976c9a P 10ca5ee68ad64b3283b9d235352c9d23 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:27.862304 12678 leader_election.cc:290] T 8a69f32c53f24a50b7852d143665d7b4 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
W20260709 13:04:27.933614 11028 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.669s	user 0.007s	sys 0.017s
I20260709 13:04:27.935118 11024 raft_consensus.cc:1275] T e5ba72ccd854430baadbc636dfd17136 P 873357742d5843579556ee8166f23f47 [term 7 FOLLOWER]: Refusing update from remote peer 10ca5ee68ad64b3283b9d235352c9d23: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
W20260709 13:04:27.934319 11028 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.670s	user 0.007s	sys 0.017s
I20260709 13:04:27.913296 12669 raft_consensus.cc:493] T 75492f86fde046c3900df3eac8292fd4 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:27.935470 12434 raft_consensus.cc:493] T 4f9eb06ed01f4ce68be3af8a8434665b P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 186b12638a87410487467df721a3cbda)
I20260709 13:04:27.922825 12777 raft_consensus.cc:515] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 186b12638a87410487467df721a3cbda [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:27.961232 11021 raft_consensus.cc:2468] T b4cd3d7dfbbf4096908ff11e9dc57141 P 873357742d5843579556ee8166f23f47 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 6.
I20260709 13:04:27.939429 11096 raft_consensus.cc:1275] T 6f46a5102fd64b45bf1c4dff863d2b4d P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Refusing update from remote peer 10ca5ee68ad64b3283b9d235352c9d23: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:04:27.946040 11100 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4cd3d7dfbbf4096908ff11e9dc57141" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4"
I20260709 13:04:27.952535 11028 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5299b8d044f41b7b68057e63d17b159" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:27.962714 11067 leader_election.cc:304] T d371c5e27bd843469d2e2e453734cb4a P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:04:27.963241 11028 raft_consensus.cc:2468] T a5299b8d044f41b7b68057e63d17b159 P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 4.
I20260709 13:04:27.963480 10950 raft_consensus.cc:1275] T 836251ebc2b74f5da88b8a8b2338e2a5 P 10ca5ee68ad64b3283b9d235352c9d23 [term 6 FOLLOWER]: Refusing update from remote peer 186b12638a87410487467df721a3cbda: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
W20260709 13:04:27.952905 10845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:38507, user_credentials={real_user=slave}} blocked reactor thread for 108517us
I20260709 13:04:27.916599 12513 raft_consensus.cc:493] T e5ba72ccd854430baadbc636dfd17136 P 0172fed22b6b4890a7fdc953fa128af4 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:27.959059 11026 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e96e80ba9ac34d0998395515e5c8c41a" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:27.928287 12517 leader_election.cc:290] T e89e640169e14e2d98f968f76027daa7 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
W20260709 13:04:27.999677 10826 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 13:04:27.965368 10845 leader_election.cc:304] T d371c5e27bd843469d2e2e453734cb4a P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda; no voters: 
I20260709 13:04:27.968984 10946 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a69f32c53f24a50b7852d143665d7b4" candidate_uuid: "186b12638a87410487467df721a3cbda" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:27.969913 10920 leader_election.cc:304] T bac7e5040f984ba2b7146bd36af7267e P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23; no voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47
I20260709 13:04:27.970227 10949 raft_consensus.cc:1240] T 732c4adeeff6422497aa7db4a82f535b P 10ca5ee68ad64b3283b9d235352c9d23 [term 6 FOLLOWER]: Rejecting Update request from peer 186b12638a87410487467df721a3cbda for earlier term 5. Current term is 6. Ops: [3.3-5.4]
I20260709 13:04:27.921767 12779 raft_consensus.cc:697] T d71ca3a1ba6b48478aaf62741bf3cf6a P 873357742d5843579556ee8166f23f47 [term 5 LEADER]: Becoming Leader. State: Replica: 873357742d5843579556ee8166f23f47, State: Running, Role: LEADER
I20260709 13:04:27.932497 12250 raft_consensus.cc:3060] T e96e80ba9ac34d0998395515e5c8c41a P 186b12638a87410487467df721a3cbda [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:04:27.972390 11026 raft_consensus.cc:2468] T e96e80ba9ac34d0998395515e5c8c41a P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 4.
I20260709 13:04:28.001569 10946 raft_consensus.cc:2468] T 8a69f32c53f24a50b7852d143665d7b4 P 10ca5ee68ad64b3283b9d235352c9d23 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 186b12638a87410487467df721a3cbda in term 5.
I20260709 13:04:28.007091 10920 leader_election.cc:304] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 10ca5ee68ad64b3283b9d235352c9d23; no voters: 
I20260709 13:04:28.002405 10845 leader_election.cc:304] T 8a69f32c53f24a50b7852d143665d7b4 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 186b12638a87410487467df721a3cbda; no voters: 
W20260709 13:04:28.009879 10919 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:45807, user_credentials={real_user=slave}} blocked reactor thread for 47844.9us
I20260709 13:04:27.937340 12649 leader_election.cc:290] T 4f9eb06ed01f4ce68be3af8a8434665b P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:27.876533 12739 raft_consensus.cc:515] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 0172fed22b6b4890a7fdc953fa128af4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:27.919703 12772 consensus_queue.cc:1048] T a610d1a71f52432e88fe56130f921fa2 P 186b12638a87410487467df721a3cbda [LEADER]: Connected to new peer: Peer: permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.085s
I20260709 13:04:28.012307 11026 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e89e640169e14e2d98f968f76027daa7" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:28.011593 10919 leader_election.cc:304] T b4cd3d7dfbbf4096908ff11e9dc57141 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47; no voters: 
I20260709 13:04:28.016675 10950 raft_consensus.cc:1240] T c51782cd9d064ec0949e84e24dd4de67 P 10ca5ee68ad64b3283b9d235352c9d23 [term 6 FOLLOWER]: Rejecting Update request from peer 0172fed22b6b4890a7fdc953fa128af4 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20260709 13:04:28.016819 10955 raft_consensus.cc:1240] T b4cd3d7dfbbf4096908ff11e9dc57141 P 10ca5ee68ad64b3283b9d235352c9d23 [term 6 FOLLOWER]: Rejecting Update request from peer 0172fed22b6b4890a7fdc953fa128af4 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20260709 13:04:28.016897 10946 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87cd8f5097624c8692f1d83544d8881e" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:28.017663 10949 raft_consensus.cc:1275] T e5a5f5a608504650b48fd27f163faed6 P 10ca5ee68ad64b3283b9d235352c9d23 [term 6 FOLLOWER]: Refusing update from remote peer 0172fed22b6b4890a7fdc953fa128af4: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
W20260709 13:04:28.018141 12780 log.cc:927] Time spent T 30f628e3d7914d609e7f1d980a74c654 P 10ca5ee68ad64b3283b9d235352c9d23: Append to log took a long time: real 0.090s	user 0.000s	sys 0.000s
I20260709 13:04:27.950616 12769 raft_consensus.cc:515] T 11de93cd9b9d457489ff1cb80f976c9a P 10ca5ee68ad64b3283b9d235352c9d23 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:27.958098 12702 raft_consensus.cc:493] T 4b37f8fca8dc42718b2a42f0883a956f P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:27.957329 12705 raft_consensus.cc:493] T a64b7e99d6c54934a7de6f6dffcf5e94 P 10ca5ee68ad64b3283b9d235352c9d23 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:28.012724 12584 consensus_meta.cc:287] T 8f1466124dbd4f6ebb3c4d62ca48bf96 P 873357742d5843579556ee8166f23f47: Time spent flushing consensus metadata: real 0.998s	user 0.019s	sys 0.000s
I20260709 13:04:28.031147 10954 raft_consensus.cc:1240] T f34f2bc02d85444f9096e0f5b4fe4331 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 LEADER]: Rejecting Update request from peer 186b12638a87410487467df721a3cbda for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260709 13:04:28.043175 11024 raft_consensus.cc:1240] T b4cd3d7dfbbf4096908ff11e9dc57141 P 873357742d5843579556ee8166f23f47 [term 6 FOLLOWER]: Rejecting Update request from peer 0172fed22b6b4890a7fdc953fa128af4 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
W20260709 13:04:28.044600 12755 log.cc:927] Time spent T 1e4f2241a94a41f38e481137f868b83d P 873357742d5843579556ee8166f23f47: Append to log took a long time: real 0.064s	user 0.001s	sys 0.000s
I20260709 13:04:28.046317 12390 leader_election.cc:290] T 72b111ad7b5f4753ab1d2b5ee1114840 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 6 election: Requested vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903)
I20260709 13:04:28.048364 12771 consensus_queue.cc:1048] T e5a5f5a608504650b48fd27f163faed6 P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.012s
I20260709 13:04:28.049135 12771 consensus_queue.cc:1243] T e5a5f5a608504650b48fd27f163faed6 P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Peer 10ca5ee68ad64b3283b9d235352c9d23 log is divergent from this leader: its last log entry 4.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260709 13:04:28.049431 12672 raft_consensus.cc:3060] T 0e5d327630e5438b878035d28efe6c0c P 0172fed22b6b4890a7fdc953fa128af4 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:04:27.957206 12678 consensus_queue.cc:1048] T 11de93cd9b9d457489ff1cb80f976c9a P 186b12638a87410487467df721a3cbda [LEADER]: Connected to new peer: Peer: permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.069s
I20260709 13:04:28.050782 12779 consensus_queue.cc:237] T d71ca3a1ba6b48478aaf62741bf3cf6a P 873357742d5843579556ee8166f23f47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
W20260709 13:04:28.052492 12741 log.cc:927] Time spent T 75492f86fde046c3900df3eac8292fd4 P 0172fed22b6b4890a7fdc953fa128af4: Append to log took a long time: real 0.946s	user 0.006s	sys 0.008s
I20260709 13:04:28.053195 12578 leader_election.cc:290] T 732c4adeeff6422497aa7db4a82f535b P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 186b12638a87410487467df721a3cbda (127.0.107.193:38967), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:28.054735 12705 raft_consensus.cc:515] T a64b7e99d6c54934a7de6f6dffcf5e94 P 10ca5ee68ad64b3283b9d235352c9d23 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:28.014272 12654 consensus_queue.cc:260] T c51782cd9d064ec0949e84e24dd4de67 P 0172fed22b6b4890a7fdc953fa128af4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:27.957962 12669 raft_consensus.cc:515] T 75492f86fde046c3900df3eac8292fd4 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:28.010982 12513 raft_consensus.cc:515] T e5ba72ccd854430baadbc636dfd17136 P 0172fed22b6b4890a7fdc953fa128af4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:28.061077 12669 leader_election.cc:290] T 75492f86fde046c3900df3eac8292fd4 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:28.062016 12709 leader_election.cc:290] T a64b7e99d6c54934a7de6f6dffcf5e94 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 5 election: Requested vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:28.013597 12550 raft_consensus.cc:2749] T bac7e5040f984ba2b7146bd36af7267e P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 13:04:28.036648 12728 consensus_queue.cc:1048] T a610d1a71f52432e88fe56130f921fa2 P 186b12638a87410487467df721a3cbda [LEADER]: Connected to new peer: Peer: permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.031s
I20260709 13:04:28.058389 12654 raft_consensus.cc:3060] T c51782cd9d064ec0949e84e24dd4de67 P 0172fed22b6b4890a7fdc953fa128af4 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:04:28.075717 11024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75492f86fde046c3900df3eac8292fd4" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:28.076149 11026 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5ba72ccd854430baadbc636dfd17136" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:28.077208 11026 raft_consensus.cc:2468] T e5ba72ccd854430baadbc636dfd17136 P 873357742d5843579556ee8166f23f47 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 7.
I20260709 13:04:28.089789 11095 raft_consensus.cc:1275] T e5ba72ccd854430baadbc636dfd17136 P 0172fed22b6b4890a7fdc953fa128af4 [term 7 FOLLOWER]: Refusing update from remote peer 10ca5ee68ad64b3283b9d235352c9d23: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260709 13:04:28.061307 12513 leader_election.cc:290] T e5ba72ccd854430baadbc636dfd17136 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:28.003628 12581 consensus_queue.cc:1048] T e5ba72ccd854430baadbc636dfd17136 P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.064s
I20260709 13:04:28.115147 12705 leader_election.cc:290] T a64b7e99d6c54934a7de6f6dffcf5e94 P 10ca5ee68ad64b3283b9d235352c9d23 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:28.019034 12784 raft_consensus.cc:2804] T d371c5e27bd843469d2e2e453734cb4a P 0172fed22b6b4890a7fdc953fa128af4 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:04:28.135558 12784 raft_consensus.cc:493] T d371c5e27bd843469d2e2e453734cb4a P 0172fed22b6b4890a7fdc953fa128af4 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:28.136003 12784 raft_consensus.cc:3060] T d371c5e27bd843469d2e2e453734cb4a P 0172fed22b6b4890a7fdc953fa128af4 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:04:28.067590 12764 raft_consensus.cc:2804] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 10ca5ee68ad64b3283b9d235352c9d23 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 13:04:27.962634 12434 raft_consensus.cc:515] T 4f9eb06ed01f4ce68be3af8a8434665b P 873357742d5843579556ee8166f23f47 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:28.138543 12764 raft_consensus.cc:697] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 10ca5ee68ad64b3283b9d235352c9d23 [term 6 LEADER]: Becoming Leader. State: Replica: 10ca5ee68ad64b3283b9d235352c9d23, State: Running, Role: LEADER
I20260709 13:04:28.138670 11023 raft_consensus.cc:2468] T a64b7e99d6c54934a7de6f6dffcf5e94 P 873357742d5843579556ee8166f23f47 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0172fed22b6b4890a7fdc953fa128af4 in term 5.
I20260709 13:04:28.139763 12764 consensus_queue.cc:237] T 5d156d6ae2ca4fa59a1e387edf0fea39 P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 4.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:28.140486 12434 leader_election.cc:290] T 4f9eb06ed01f4ce68be3af8a8434665b P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:28.112373 11067 leader_election.cc:304] T e5ba72ccd854430baadbc636dfd17136 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47; no voters: 
W20260709 13:04:28.115913 11066 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:28.116513 12558 consensus_queue.cc:1048] T 6f46a5102fd64b45bf1c4dff863d2b4d P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:04:28.118988 12575 raft_consensus.cc:493] T ff13ae9a47c7455f962406c6d131a3ff P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:28.119665 12564 raft_consensus.cc:2804] T b4cd3d7dfbbf4096908ff11e9dc57141 P 10ca5ee68ad64b3283b9d235352c9d23 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 13:04:28.036152 12702 raft_consensus.cc:515] T 4b37f8fca8dc42718b2a42f0883a956f P 10ca5ee68ad64b3283b9d235352c9d23 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:28.088567 12649 raft_consensus.cc:2804] T d371c5e27bd843469d2e2e453734cb4a P 186b12638a87410487467df721a3cbda [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:04:28.068122 12662 raft_consensus.cc:493] T c31881534c5e4f0089c82057c1746652 P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 10ca5ee68ad64b3283b9d235352c9d23)
I20260709 13:04:28.161626 11100 raft_consensus.cc:1275] T 87cd8f5097624c8692f1d83544d8881e P 0172fed22b6b4890a7fdc953fa128af4 [term 6 FOLLOWER]: Refusing update from remote peer 10ca5ee68ad64b3283b9d235352c9d23: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
W20260709 13:04:28.127045 12747 log.cc:927] Time spent T 2567c2d84e4648d7ba5dc23a3df32817 P 873357742d5843579556ee8166f23f47: Append to log took a long time: real 0.093s	user 0.000s	sys 0.001s
I20260709 13:04:28.129380 12739 leader_election.cc:290] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:28.089669 12694 consensus_queue.cc:1048] T 11de93cd9b9d457489ff1cb80f976c9a P 186b12638a87410487467df721a3cbda [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.060s
I20260709 13:04:28.131035 12488 consensus_queue.cc:1059] T 732c4adeeff6422497aa7db4a82f535b P 186b12638a87410487467df721a3cbda [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:04:28.131551 12777 leader_election.cc:290] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 0172fed22b6b4890a7fdc953fa128af4 (127.0.107.196:38507)
I20260709 13:04:28.132314 12390 raft_consensus.cc:493] T 7757ecbf402e47bfbc93d0447741e26c P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 873357742d5843579556ee8166f23f47)
W20260709 13:04:28.144098 10919 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:45807, user_credentials={real_user=slave}} blocked reactor thread for 132606us
I20260709 13:04:28.146232 10878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f9eb06ed01f4ce68be3af8a8434665b" candidate_uuid: "873357742d5843579556ee8166f23f47" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "186b12638a87410487467df721a3cbda" is_pre_election: true
I20260709 13:04:28.147130 11067 leader_election.cc:304] T a64b7e99d6c54934a7de6f6dffcf5e94 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47; no voters: 10ca5ee68ad64b3283b9d235352c9d23
I20260709 13:04:28.146068 10994 leader_election.cc:304] T 6f46a5102fd64b45bf1c4dff863d2b4d P 873357742d5843579556ee8166f23f47 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4, 873357742d5843579556ee8166f23f47; no voters: 
W20260709 13:04:28.149377 10843 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:44903, user_credentials={real_user=slave}} blocked reactor thread for 129976us
I20260709 13:04:28.032171 12691 consensus_queue.cc:1059] T b4cd3d7dfbbf4096908ff11e9dc57141 P 0172fed22b6b4890a7fdc953fa128af4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.080s
I20260709 13:04:28.151367 11097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a64b7e99d6c54934a7de6f6dffcf5e94" candidate_uuid: "10ca5ee68ad64b3283b9d235352c9d23" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0172fed22b6b4890a7fdc953fa128af4" is_pre_election: true
I20260709 13:04:28.154106 10950 raft_consensus.cc:1217] T e5a5f5a608504650b48fd27f163faed6 P 10ca5ee68ad64b3283b9d235352c9d23 [term 6 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[2.2-6.4]   Dedup: 2.2->[5.3-6.4]
I20260709 13:04:28.154314 10949 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75492f86fde046c3900df3eac8292fd4" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:28.191072 11097 raft_consensus.cc:2468] T a64b7e99d6c54934a7de6f6dffcf5e94 P 0172fed22b6b4890a7fdc953fa128af4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ca5ee68ad64b3283b9d235352c9d23 in term 5.
I20260709 13:04:28.154398 10946 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5ba72ccd854430baadbc636dfd17136" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23" is_pre_election: true
I20260709 13:04:28.114461 12678 raft_consensus.cc:2804] T 8a69f32c53f24a50b7852d143665d7b4 P 186b12638a87410487467df721a3cbda [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:04:28.155997 12644 raft_consensus.cc:493] T 65bf0a62f4d84ef18dd6b23e082724b8 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:28.166301 11023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555217d45d9b4ee492dbb3ddaf0f7c02" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "873357742d5843579556ee8166f23f47" is_pre_election: true
I20260709 13:04:28.195530 11023 raft_consensus.cc:2393] T 555217d45d9b4ee492dbb3ddaf0f7c02 P 873357742d5843579556ee8166f23f47 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0172fed22b6b4890a7fdc953fa128af4 in current term 6: Already voted for candidate 873357742d5843579556ee8166f23f47 in this term.
I20260709 13:04:28.192421 12796 consensus_queue.cc:1048] T 87cd8f5097624c8692f1d83544d8881e P 10ca5ee68ad64b3283b9d235352c9d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.009s
I20260709 13:04:28.195703 12728 raft_consensus.cc:493] T 2567c2d84e4648d7ba5dc23a3df32817 P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 10ca5ee68ad64b3283b9d235352c9d23)
I20260709 13:04:28.196338 12678 raft_consensus.cc:493] T 8a69f32c53f24a50b7852d143665d7b4 P 186b12638a87410487467df721a3cbda [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:28.196812 12678 raft_consensus.cc:3060] T 8a69f32c53f24a50b7852d143665d7b4 P 186b12638a87410487467df721a3cbda [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:04:28.161598 12649 raft_consensus.cc:493] T d371c5e27bd843469d2e2e453734cb4a P 186b12638a87410487467df721a3cbda [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:28.170928 12662 raft_consensus.cc:515] T c31881534c5e4f0089c82057c1746652 P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } }
I20260709 13:04:28.115362 12771 raft_consensus.cc:493] T 32147c6d2d5a477aa38c597c7f78df5b P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 10ca5ee68ad64b3283b9d235352c9d23)
W20260709 13:04:28.207111 11067 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:45807, user_credentials={real_user=slave}} blocked reactor thread for 60098.9us
I20260709 13:04:28.178741 10954 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72b111ad7b5f4753ab1d2b5ee1114840" candidate_uuid: "0172fed22b6b4890a7fdc953fa128af4" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "10ca5ee68ad64b3283b9d235352c9d23"
I20260709 13:04:28.208537 10954 raft_consensus.cc:3060] T 72b111ad7b5f4753ab1d2b5ee1114840 P 10ca5ee68ad64b3283b9d235352c9d23 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:04:28.179965 11100 raft_consensus.cc:1275] T f34f2bc02d85444f9096e0f5b4fe4331 P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Refusing update from remote peer 10ca5ee68ad64b3283b9d235352c9d23: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 13:04:28.162276 12564 raft_consensus.cc:697] T b4cd3d7dfbbf4096908ff11e9dc57141 P 10ca5ee68ad64b3283b9d235352c9d23 [term 6 LEADER]: Becoming Leader. State: Replica: 10ca5ee68ad64b3283b9d235352c9d23, State: Running, Role: LEADER
I20260709 13:04:28.160092 12781 raft_consensus.cc:2804] T e5ba72ccd854430baadbc636dfd17136 P 0172fed22b6b4890a7fdc953fa128af4 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:04:28.156968 12575 raft_consensus.cc:515] T ff13ae9a47c7455f962406c6d131a3ff P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:28.152539 12705 raft_consensus.cc:493] T 7839ca2d2c13431a85f44d7c458c6351 P 10ca5ee68ad64b3283b9d235352c9d23 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 873357742d5843579556ee8166f23f47)
I20260709 13:04:28.194219 11066 leader_election.cc:304] T 75492f86fde046c3900df3eac8292fd4 P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0172fed22b6b4890a7fdc953fa128af4; no voters: 10ca5ee68ad64b3283b9d235352c9d23, 873357742d5843579556ee8166f23f47
I20260709 13:04:28.184655 12488 consensus_queue.cc:1243] T 732c4adeeff6422497aa7db4a82f535b P 186b12638a87410487467df721a3cbda [LEADER]: Peer 10ca5ee68ad64b3283b9d235352c9d23 log is divergent from this leader: its last log entry 4.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260709 13:04:28.196440 12728 raft_consensus.cc:515] T 2567c2d84e4648d7ba5dc23a3df32817 P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:28.195338 12800 raft_consensus.cc:493] T a5299b8d044f41b7b68057e63d17b159 P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:28.204612 12649 raft_consensus.cc:3060] T d371c5e27bd843469d2e2e453734cb4a P 186b12638a87410487467df721a3cbda [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:04:28.189036 12390 raft_consensus.cc:515] T 7757ecbf402e47bfbc93d0447741e26c P 0172fed22b6b4890a7fdc953fa128af4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:28.214987 12709 raft_consensus.cc:3055] T b4cd3d7dfbbf4096908ff11e9dc57141 P 0172fed22b6b4890a7fdc953fa128af4 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:04:28.195024 12644 raft_consensus.cc:515] T 65bf0a62f4d84ef18dd6b23e082724b8 P 10ca5ee68ad64b3283b9d235352c9d23 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:28.209131 12799 raft_consensus.cc:2804] T 6f46a5102fd64b45bf1c4dff863d2b4d P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:04:28.225651 12691 raft_consensus.cc:2804] T a64b7e99d6c54934a7de6f6dffcf5e94 P 0172fed22b6b4890a7fdc953fa128af4 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:04:28.207360 12772 consensus_queue.cc:1059] T f34f2bc02d85444f9096e0f5b4fe4331 P 186b12638a87410487467df721a3cbda [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.033s
I20260709 13:04:28.217890 12781 raft_consensus.cc:493] T e5ba72ccd854430baadbc636dfd17136 P 0172fed22b6b4890a7fdc953fa128af4 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:28.245177 12709 raft_consensus.cc:740] T b4cd3d7dfbbf4096908ff11e9dc57141 P 0172fed22b6b4890a7fdc953fa128af4 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 0172fed22b6b4890a7fdc953fa128af4, State: Running, Role: LEADER
I20260709 13:04:28.249737 12728 leader_election.cc:290] T 2567c2d84e4648d7ba5dc23a3df32817 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:28.215615 12771 raft_consensus.cc:515] T 32147c6d2d5a477aa38c597c7f78df5b P 0172fed22b6b4890a7fdc953fa128af4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } }
I20260709 13:04:28.246313 12799 raft_consensus.cc:493] T 6f46a5102fd64b45bf1c4dff863d2b4d P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:28.214565 12793 consensus_queue.cc:1048] T 836251ebc2b74f5da88b8a8b2338e2a5 P 186b12638a87410487467df721a3cbda [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.051s
I20260709 13:04:28.254595 12772 consensus_queue.cc:1243] T f34f2bc02d85444f9096e0f5b4fe4331 P 186b12638a87410487467df721a3cbda [LEADER]: Peer 10ca5ee68ad64b3283b9d235352c9d23 log is divergent from this leader: its last log entry 3.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260709 13:04:28.253054 12781 raft_consensus.cc:3060] T e5ba72ccd854430baadbc636dfd17136 P 0172fed22b6b4890a7fdc953fa128af4 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:04:28.240016 12800 raft_consensus.cc:515] T a5299b8d044f41b7b68057e63d17b159 P 186b12638a87410487467df721a3cbda [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } } peers { permanent_uuid: "10ca5ee68ad64b3283b9d235352c9d23" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 44903 } } peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } }
I20260709 13:04:28.259606 12799 raft_consensus.cc:3060] T 6f46a5102fd64b45bf1c4dff863d2b4d P 873357742d5843579556ee8166f23f47 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:04:28.262786 12784 raft_consensus.cc:515] T d371c5e27bd843469d2e2e453734cb4a P 0172fed22b6b4890a7fdc953fa128af4 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873357742d5843579556ee8166f23f47" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45807 } } peers { permanent_uuid: "0172fed22b6b4890a7fdc953fa128af4" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 38507 } } peers { permanent_uuid: "186b12638a87410487467df721a3cbda" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38967 } }
I20260709 13:04:28.264448 12771 leader_election.cc:290] T 32147c6d2d5a477aa38c597c7f78df5b P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
I20260709 13:04:28.265146 12784 leader_election.cc:290] T d371c5e27bd843469d2e2e453734cb4a P 0172fed22b6b4890a7fdc953fa128af4 [CANDIDATE]: Term 6 election: Requested vote from peers 873357742d5843579556ee8166f23f47 (127.0.107.195:45807), 186b12638a87410487467df721a3cbda (127.0.107.193:38967)
I20260709 13:04:28.266309 12800 leader_election.cc:290] T a5299b8d044f41b7b68057e63d17b159 P 186b12638a87410487467df721a3cbda [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 10ca5ee68ad64b3283b9d235352c9d23 (127.0.107.194:44903), 873357742d5843579556ee8166f23f47 (127.0.107.195:45807)
    @     0x7f425f94bc87 __libc_start_main at ??:0
    @           0x2a338a _start at ??:?
